2023-10-16T13:58:10.054 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T13:58:10.055 DEBUG:teuthology.run:Teuthology command: teuthology --name rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --owner scheduled_rishabh@teuthology --verbose --archive /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037 -- /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/orig.config.yaml 2023-10-16T13:58:10.134 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T13:58:10.218 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037 branch: rishabh-2023oct13 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: null first_in_suite: false job_id: '7429037' kernel: client: branch: testing flavor: default kdb: true sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-2023oct13 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-2023oct13-b2 sha1: 33be745ec4531df08cd1bb70b4c9c1f89be58078 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6006 sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 sleep_before_teardown: 0 subset: 402/512 suite: fs suite_branch: rishabh-2023oct13-b2 suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_33be745ec4531df08cd1bb70b4c9c1f89be58078/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 33be745ec4531df08cd1bb70b4c9c1f89be58078 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLovRJaWv6bFJ6tOwzKzcWOli1s8H6A6heFn15YRWNmDHD1HBnGtgWuQuDz9pU15qe12OyQtW/A9zhZwnw3C9bg= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOBtmsYnh/Ohhod1nwajWDpTORU7ptsmilduWzGZaP1RVkfWpkWMiYu1Q5wefsop6hhXLZARMmrAzXyTGKqqYw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_12:17:50 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-16T13:58:10.219 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_33be745ec4531df08cd1bb70b4c9c1f89be58078/qa; will attempt to use it 2023-10-16T13:58:10.219 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_33be745ec4531df08cd1bb70b4c9c1f89be58078/qa/tasks 2023-10-16T13:58:10.220 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-16T13:58:10.222 INFO:teuthology.task.internal:Checking packages... 2023-10-16T13:58:10.245 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0' 2023-10-16T13:58:10.245 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-16T13:58:10.246 INFO:teuthology.packaging:ref: None 2023-10-16T13:58:10.246 INFO:teuthology.packaging:tag: None 2023-10-16T13:58:10.246 INFO:teuthology.packaging:branch: rishabh-2023oct13 2023-10-16T13:58:10.247 INFO:teuthology.packaging:sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T13:58:10.247 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-2023oct13 2023-10-16T13:58:10.498 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6730.g4ba8a190 2023-10-16T13:58:10.498 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-16T13:58:10.508 INFO:teuthology.task.internal:no buildpackages task found 2023-10-16T13:58:10.508 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-16T13:58:10.535 INFO:teuthology.task.internal:Saving configuration 2023-10-16T13:58:10.557 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-16T13:58:10.564 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T13:58:10.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 13:53:02.388477', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLovRJaWv6bFJ6tOwzKzcWOli1s8H6A6heFn15YRWNmDHD1HBnGtgWuQuDz9pU15qe12OyQtW/A9zhZwnw3C9bg='} 2023-10-16T13:58:10.613 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 13:53:02.386821', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOBtmsYnh/Ohhod1nwajWDpTORU7ptsmilduWzGZaP1RVkfWpkWMiYu1Q5wefsop6hhXLZARMmrAzXyTGKqqYw='} 2023-10-16T13:58:10.613 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-16T13:58:10.620 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-16T13:58:10.621 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-16T13:58:10.621 INFO:teuthology.run_tasks:Running task console_log... 2023-10-16T13:58:10.697 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb1a520a60>, signals=[15]) 2023-10-16T13:58:10.698 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-16T13:58:10.705 INFO:teuthology.task.internal:Opening connections... 2023-10-16T13:58:10.705 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T13:58:10.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T13:58:10.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T13:58:10.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T13:58:10.890 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-16T13:58:10.897 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-10-16T13:58:10.915 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-10-16T13:58:10.916 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-10-16T13:58:10.973 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-10-16T13:58:10.973 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2023-10-16T13:58:10.974 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-10-16T13:58:10.974 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-10-16T13:58:10.974 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2023-10-16T13:58:10.974 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2023-10-16T13:58:10.974 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-16T13:58:10.975 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-10-16T13:58:10.975 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2023-10-16T13:58:10.975 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-16T13:58:10.975 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-10-16T13:58:10.975 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T13:58:10.975 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-16T13:58:10.976 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-16T13:58:10.976 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-10-16T13:58:11.000 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-10-16T13:58:11.017 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-10-16T13:58:11.018 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-10-16T13:58:11.074 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2023-10-16T13:58:11.074 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2023-10-16T13:58:11.075 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2023-10-16T13:58:11.075 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2023-10-16T13:58:11.075 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2023-10-16T13:58:11.075 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2023-10-16T13:58:11.075 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-16T13:58:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-10-16T13:58:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2023-10-16T13:58:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-16T13:58:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2023-10-16T13:58:11.076 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T13:58:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-16T13:58:11.077 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-16T13:58:11.077 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-10-16T13:58:11.100 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-16T13:58:11.109 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-16T13:58:11.158 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-16T13:58:11.159 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-10-16T13:58:11.161 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-10-16T13:58:11.177 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-16T13:58:11.184 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-16T13:58:11.184 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-10-16T13:58:11.218 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-10-16T13:58:11.311 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-16T13:58:11.363 INFO:teuthology.run_tasks:Running task kernel... 2023-10-16T13:58:11.515 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'} 2023-10-16T13:58:11.516 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2023-10-16T13:58:11.516 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2023-10-16T13:58:11.516 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T13:58:11.517 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T13:58:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:58:11.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-10-16T13:58:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:58:11.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-10-16T13:58:11.693 DEBUG:teuthology.task.kernel:sha1 for client.1 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T13:58:11.693 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-16T13:58:11.693 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-16T13:58:11.699 DEBUG:teuthology.task.kernel:sha1 for client.0 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T13:58:11.700 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-16T13:58:11.700 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-10-16T13:58:11.710 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2023-10-16T13:58:11.711 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T13:58:11.711 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-16T13:58:11.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum upgrade -y linux-firmware 2023-10-16T13:58:11.718 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2023-10-16T13:58:11.718 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T13:58:11.718 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-16T13:58:11.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum upgrade -y linux-firmware 2023-10-16T13:58:13.781 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-16T13:58:14.846 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-16T13:58:15.081 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2023-10-16T13:58:16.291 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2023-10-16T13:58:17.005 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.9 MB 00:01 2023-10-16T13:58:18.948 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 19 MB/s | 18 MB 00:00 2023-10-16T13:58:21.057 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 7.9 MB 00:04 2023-10-16T13:58:23.557 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.7 MB 00:01 2023-10-16T13:58:24.087 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-10-16T13:58:25.024 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 14 kB 00:00 2023-10-16T13:58:28.986 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 19 MB 00:03 2023-10-16T13:58:29.111 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.7 MB 00:01 2023-10-16T13:58:30.589 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 14 kB 00:00 2023-10-16T13:58:32.350 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 12 MB/s | 19 MB 00:01 2023-10-16T13:58:33.508 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2023-10-16T13:58:35.704 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-16T13:58:35.712 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-16T13:58:35.713 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-16T13:58:35.713 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-16T13:58:35.713 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-10-16T13:58:35.713 INFO:teuthology.orchestra.run.smithi170.stdout: linux-firmware noarch 20230926-140.el9 CentOS-BaseOS 326 M 2023-10-16T13:58:35.713 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T13:58:35.714 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-16T13:58:35.714 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-10-16T13:58:35.714 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 1 Package 2023-10-16T13:58:35.714 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T13:58:35.714 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 326 M 2023-10-16T13:58:35.715 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-16T13:58:36.976 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2023-10-16T13:58:39.321 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-10-16T13:58:39.337 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-16T13:58:39.338 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-10-16T13:58:39.338 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-16T13:58:39.338 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-10-16T13:58:39.338 INFO:teuthology.orchestra.run.smithi029.stdout: linux-firmware noarch 20230926-140.el9 CentOS-BaseOS 326 M 2023-10-16T13:58:39.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T13:58:39.339 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-10-16T13:58:39.339 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-16T13:58:39.339 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 1 Package 2023-10-16T13:58:39.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T13:58:39.339 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 326 M 2023-10-16T13:58:39.340 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-10-16T13:58:42.276 INFO:teuthology.orchestra.run.smithi170.stdout:linux-firmware-20230926-140.el9.noarch.rpm 50 MB/s | 326 MB 00:06 2023-10-16T13:58:42.277 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-16T13:58:42.278 INFO:teuthology.orchestra.run.smithi170.stdout:Total 50 MB/s | 326 MB 00:06 2023-10-16T13:58:42.278 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-16T13:58:42.368 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-16T13:58:42.368 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-16T13:58:44.456 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-16T13:58:44.456 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-16T13:58:50.175 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-16T13:58:50.623 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T13:58:50.707 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T13:58:51.539 INFO:teuthology.orchestra.run.smithi029.stdout:linux-firmware-20230926-140.el9.noarch.rpm 27 MB/s | 326 MB 00:12 2023-10-16T13:58:51.540 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-10-16T13:58:51.540 INFO:teuthology.orchestra.run.smithi029.stdout:Total 27 MB/s | 326 MB 00:12 2023-10-16T13:58:51.540 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-10-16T13:58:51.644 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-10-16T13:58:51.644 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-10-16T13:58:51.744 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T13:58:51.744 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T13:58:52.159 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T13:58:52.159 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T13:58:52.159 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-10-16T13:58:52.159 INFO:teuthology.orchestra.run.smithi170.stdout: linux-firmware-20230926-140.el9.noarch 2023-10-16T13:58:52.160 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T13:58:52.160 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-16T13:58:52.471 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.1... 2023-10-16T13:58:52.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T13:58:52.700 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ 2023-10-16T13:58:52.700 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ --input-file=- 2023-10-16T13:58:52.703 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi170.front.sepia.ceph.com to complete... 2023-10-16T13:58:53.819 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-10-16T13:58:53.820 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-10-16T13:59:01.678 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-10-16T13:59:02.024 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T13:59:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T13:59:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T13:59:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T13:59:03.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16 13:59:03 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [320683515/320683515] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-16T13:59:03.572 INFO:teuthology.orchestra.run.smithi170.stderr:FINISHED --2023-10-16 13:59:03-- 2023-10-16T13:59:03.572 INFO:teuthology.orchestra.run.smithi170.stderr:Total wall clock time: 11s 2023-10-16T13:59:03.572 INFO:teuthology.orchestra.run.smithi170.stderr:Downloaded: 1 files, 306M in 11s (28.9 MB/s) 2023-10-16T13:59:03.573 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.1... 2023-10-16T13:59:03.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-16T13:59:03.881 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T13:59:03.882 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T13:59:03.882 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-10-16T13:59:03.882 INFO:teuthology.orchestra.run.smithi029.stdout: linux-firmware-20230926-140.el9.noarch 2023-10-16T13:59:03.882 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T13:59:03.882 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-10-16T13:59:04.255 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-16T13:59:04.255 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T13:59:04.476 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ 2023-10-16T13:59:04.476 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ --input-file=- 2023-10-16T13:59:04.479 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi029.front.sepia.ceph.com to complete... 2023-10-16T13:59:05.413 INFO:teuthology.orchestra.run.smithi170.stdout:Verifying... ######################################## 2023-10-16T13:59:05.485 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing... ######################################## 2023-10-16T13:59:05.564 INFO:teuthology.orchestra.run.smithi170.stdout:Updating / installing... 2023-10-16T13:59:12.592 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-16T13:59:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16 13:59:15 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [320683515/320683515] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-16T13:59:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:FINISHED --2023-10-16 13:59:15-- 2023-10-16T13:59:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:Total wall clock time: 11s 2023-10-16T13:59:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:Downloaded: 1 files, 306M in 11s (28.9 MB/s) 2023-10-16T13:59:15.393 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-16T13:59:15.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-16T13:59:17.337 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying... ######################################## 2023-10-16T13:59:17.417 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing... ######################################## 2023-10-16T13:59:17.501 INFO:teuthology.orchestra.run.smithi029.stdout:Updating / installing... 2023-10-16T13:59:24.885 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-16T13:59:27.972 INFO:teuthology.orchestra.run.smithi170.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-16T13:59:49.217 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-16T13:59:49.217 DEBUG:teuthology.orchestra.run.smithi170:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-16T13:59:49.297 INFO:teuthology.orchestra.run.smithi170.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.298 INFO:teuthology.orchestra.run.smithi170.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.298 INFO:teuthology.orchestra.run.smithi170.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.298 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.298 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-16T13:59:49.299 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-16T13:59:49.299 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-16T13:59:49.299 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-16T13:59:49.299 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-16T13:59:49.299 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-16T13:59:49.300 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-16T13:59:49.300 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-16T13:59:49.300 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-16T13:59:49.300 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-16T13:59:49.300 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-16T13:59:49.301 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-16T13:59:49.301 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-16T13:59:49.301 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-16T13:59:49.301 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-16T13:59:49.301 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-16T13:59:49.301 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-16T13:59:49.302 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-16T13:59:49.302 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-16T13:59:49.302 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-16T13:59:49.303 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-16T13:59:49.303 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-16T13:59:49.303 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-16T13:59:49.303 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-16T13:59:49.304 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-16T13:59:49.304 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-16T13:59:49.304 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-16T13:59:49.304 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-16T13:59:49.304 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-16T13:59:49.305 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-16T13:59:49.305 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-16T13:59:49.305 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-16T13:59:49.305 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-16T13:59:49.305 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-16T13:59:49.305 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-16T13:59:49.306 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-16T13:59:49.306 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-16T13:59:49.306 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-16T13:59:49.306 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-16T13:59:49.306 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-16T13:59:49.307 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-16T13:59:49.307 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-16T13:59:49.307 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-16T13:59:49.307 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-16T13:59:49.307 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-16T13:59:49.308 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-16T13:59:49.308 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-16T13:59:49.308 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-16T13:59:49.308 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-16T13:59:49.308 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-16T13:59:49.309 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-16T13:59:49.309 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-16T13:59:49.309 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-16T13:59:49.309 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-16T13:59:49.309 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-16T13:59:49.310 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-16T13:59:49.310 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-16T13:59:49.310 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-16T13:59:49.310 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-16T13:59:49.310 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-16T13:59:49.311 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-16T13:59:49.311 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-16T13:59:49.311 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-16T13:59:49.311 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-16T13:59:49.311 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-16T13:59:49.311 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-16T13:59:49.312 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-16T13:59:49.312 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-16T13:59:49.312 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-16T13:59:49.312 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-16T13:59:49.312 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-16T13:59:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-16T13:59:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-16T13:59:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-16T13:59:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-16T13:59:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-16T13:59:49.313 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-16T13:59:49.314 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-16T13:59:49.314 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-16T13:59:49.314 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-16T13:59:49.314 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-16T13:59:49.314 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-16T13:59:49.315 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-16T13:59:49.315 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-16T13:59:49.315 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-16T13:59:49.315 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-16T13:59:49.315 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-16T13:59:49.315 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-16T13:59:49.316 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-16T13:59:49.316 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-16T13:59:49.316 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-16T13:59:49.316 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-16T13:59:49.316 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-16T13:59:49.316 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-16T13:59:49.317 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-16T13:59:49.317 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-16T13:59:49.317 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-16T13:59:49.317 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-16T13:59:49.317 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-16T13:59:49.317 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-16T13:59:49.318 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-16T13:59:49.318 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-16T13:59:49.318 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-16T13:59:49.318 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-16T13:59:49.318 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-16T13:59:49.319 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-16T13:59:49.319 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-16T13:59:49.319 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-16T13:59:49.319 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-16T13:59:49.319 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-16T13:59:49.319 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-16T13:59:49.320 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-16T13:59:49.320 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-16T13:59:49.320 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-16T13:59:49.320 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-16T13:59:49.320 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-16T13:59:49.321 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-16T13:59:49.321 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-16T13:59:49.321 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-16T13:59:49.321 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-16T13:59:49.321 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-16T13:59:49.322 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-16T13:59:49.322 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-16T13:59:49.322 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-16T13:59:49.322 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-16T13:59:49.322 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-16T13:59:49.323 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-16T13:59:49.323 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-16T13:59:49.323 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-16T13:59:49.323 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-16T13:59:49.323 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-16T13:59:49.324 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-16T13:59:49.324 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-16T13:59:49.324 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-16T13:59:49.324 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-16T13:59:49.324 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-16T13:59:49.324 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-16T13:59:49.325 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-16T13:59:49.325 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-16T13:59:49.325 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-16T13:59:49.325 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-16T13:59:49.325 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-16T13:59:49.326 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-16T13:59:49.326 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-16T13:59:49.326 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-16T13:59:49.326 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-16T13:59:49.326 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-16T13:59:49.327 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-16T13:59:49.327 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-16T13:59:49.327 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-16T13:59:49.327 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-16T13:59:49.327 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-16T13:59:49.328 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-16T13:59:49.328 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-16T13:59:49.328 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-16T13:59:49.328 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-16T13:59:49.329 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-16T13:59:49.329 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-16T13:59:49.329 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-16T13:59:49.329 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-16T13:59:49.329 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-16T13:59:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-16T13:59:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-16T13:59:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-16T13:59:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-16T13:59:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-16T13:59:49.331 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-16T13:59:49.331 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-16T13:59:49.331 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-16T13:59:49.331 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-16T13:59:49.331 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-16T13:59:49.332 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-16T13:59:49.332 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-16T13:59:49.332 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-16T13:59:49.332 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-16T13:59:49.332 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-16T13:59:49.333 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-16T13:59:49.333 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-16T13:59:49.333 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-16T13:59:49.333 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-16T13:59:49.333 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-16T13:59:49.334 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-16T13:59:49.334 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-16T13:59:49.334 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-16T13:59:49.334 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-16T13:59:49.334 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-16T13:59:49.335 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-16T13:59:49.335 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-16T13:59:49.335 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-16T13:59:49.335 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-16T13:59:49.335 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-16T13:59:49.335 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-16T13:59:49.336 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-16T13:59:49.336 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-16T13:59:49.336 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-16T13:59:49.336 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-16T13:59:49.336 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-16T13:59:49.337 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-16T13:59:49.337 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-16T13:59:49.337 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-16T13:59:49.337 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-16T13:59:49.337 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-16T13:59:49.337 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-16T13:59:49.338 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-16T13:59:49.338 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-16T13:59:49.338 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-16T13:59:49.338 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-16T13:59:49.338 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-16T13:59:49.339 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-16T13:59:49.339 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-16T13:59:49.339 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-16T13:59:49.339 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-16T13:59:49.339 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-16T13:59:49.340 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-16T13:59:49.340 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-16T13:59:49.340 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-16T13:59:49.340 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-16T13:59:49.340 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-16T13:59:49.340 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-16T13:59:49.341 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-16T13:59:49.341 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-16T13:59:49.341 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-16T13:59:49.341 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-16T13:59:49.341 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-16T13:59:49.342 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-16T13:59:49.342 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-16T13:59:49.342 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-16T13:59:49.342 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-16T13:59:49.342 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-16T13:59:49.343 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-16T13:59:49.343 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-16T13:59:49.343 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-16T13:59:49.343 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-16T13:59:49.343 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-16T13:59:49.344 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-16T13:59:49.344 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-16T13:59:49.344 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-16T13:59:49.344 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-16T13:59:49.344 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-16T13:59:49.345 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-16T13:59:49.345 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-16T13:59:49.345 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-16T13:59:49.345 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-16T13:59:49.346 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-16T13:59:49.346 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-16T13:59:49.346 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-16T13:59:49.346 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-16T13:59:49.346 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-16T13:59:49.347 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-16T13:59:49.347 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-16T13:59:49.347 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-16T13:59:49.347 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-16T13:59:49.347 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-16T13:59:49.347 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-16T13:59:49.348 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-16T13:59:49.348 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-16T13:59:49.348 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-16T13:59:49.348 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-16T13:59:49.348 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-16T13:59:49.349 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-16T13:59:49.349 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-16T13:59:49.349 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-16T13:59:49.349 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-16T13:59:49.349 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-16T13:59:49.350 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-16T13:59:49.350 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-16T13:59:49.350 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-16T13:59:49.350 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-16T13:59:49.350 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-16T13:59:49.350 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-16T13:59:49.351 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-16T13:59:49.351 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-16T13:59:49.351 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-16T13:59:49.351 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-16T13:59:49.351 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-16T13:59:49.352 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-16T13:59:49.352 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-16T13:59:49.352 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-16T13:59:49.352 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-16T13:59:49.352 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-16T13:59:49.352 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-16T13:59:49.353 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-16T13:59:49.353 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-16T13:59:49.353 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-16T13:59:49.353 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-16T13:59:49.353 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-16T13:59:49.354 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-16T13:59:49.354 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-16T13:59:49.354 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-16T13:59:49.354 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-16T13:59:49.354 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-16T13:59:49.355 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-16T13:59:49.355 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-16T13:59:49.355 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-16T13:59:49.355 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-16T13:59:49.355 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-16T13:59:49.355 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-16T13:59:49.356 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-16T13:59:49.356 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-16T13:59:49.356 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-16T13:59:49.356 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-16T13:59:49.356 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-16T13:59:49.357 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-16T13:59:49.357 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-16T13:59:49.357 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-16T13:59:49.357 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-16T13:59:49.357 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-16T13:59:49.357 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-16T13:59:49.358 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-16T13:59:49.358 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-16T13:59:49.358 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-16T13:59:49.358 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-16T13:59:49.358 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-16T13:59:49.359 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-16T13:59:49.359 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-16T13:59:49.359 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-16T13:59:49.359 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-16T13:59:49.359 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-16T13:59:49.359 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-16T13:59:49.360 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-16T13:59:49.360 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-16T13:59:49.360 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-16T13:59:49.360 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-16T13:59:49.360 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-16T13:59:49.360 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-16T13:59:49.361 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-16T13:59:49.361 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-16T13:59:49.361 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-16T13:59:49.361 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-16T13:59:49.362 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-16T13:59:49.362 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-16T13:59:49.362 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-16T13:59:49.362 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-16T13:59:49.362 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-16T13:59:49.363 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-16T13:59:49.363 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-16T13:59:49.363 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-16T13:59:49.363 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-16T13:59:49.363 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-16T13:59:49.363 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-16T13:59:49.364 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-16T13:59:49.364 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-16T13:59:49.364 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-16T13:59:49.364 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-16T13:59:49.364 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-16T13:59:49.365 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-16T13:59:49.365 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-16T13:59:49.365 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-16T13:59:49.365 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-16T13:59:49.365 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-16T13:59:49.365 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-16T13:59:49.366 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-16T13:59:49.366 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-16T13:59:49.366 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-16T13:59:49.366 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-16T13:59:49.366 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-16T13:59:49.367 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-16T13:59:49.367 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-16T13:59:49.367 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-16T13:59:49.367 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-16T13:59:49.367 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-16T13:59:49.367 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-16T13:59:49.368 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-16T13:59:49.368 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-16T13:59:49.368 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-16T13:59:49.368 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-16T13:59:49.368 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-16T13:59:49.368 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-16T13:59:49.369 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-16T13:59:49.369 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-16T13:59:49.369 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-16T13:59:49.369 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-16T13:59:49.369 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-16T13:59:49.370 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-16T13:59:49.370 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-16T13:59:49.370 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-16T13:59:49.370 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-16T13:59:49.370 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-16T13:59:49.371 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-16T13:59:49.371 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-16T13:59:49.371 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-16T13:59:49.371 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-16T13:59:49.371 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-16T13:59:49.372 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-16T13:59:49.372 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-16T13:59:49.372 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-16T13:59:49.372 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-16T13:59:49.372 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-16T13:59:49.372 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-16T13:59:49.373 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-16T13:59:49.373 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-16T13:59:49.373 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-16T13:59:49.373 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-16T13:59:49.373 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-16T13:59:49.373 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-16T13:59:49.374 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-16T13:59:49.374 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-16T13:59:49.374 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-16T13:59:49.374 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-16T13:59:49.374 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-16T13:59:49.374 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-16T13:59:49.375 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-16T13:59:49.375 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-16T13:59:49.375 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-16T13:59:49.375 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-16T13:59:49.375 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-16T13:59:49.375 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-16T13:59:49.376 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-16T13:59:49.376 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-16T13:59:49.376 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-16T13:59:49.376 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-16T13:59:49.376 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-16T13:59:49.377 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-16T13:59:49.377 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-16T13:59:49.377 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-16T13:59:49.377 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-16T13:59:49.377 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-16T13:59:49.377 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-16T13:59:49.378 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-16T13:59:49.378 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-16T13:59:49.378 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-16T13:59:49.378 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-16T13:59:49.378 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-16T13:59:49.379 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-16T13:59:49.379 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-16T13:59:49.379 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-16T13:59:49.379 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-16T13:59:49.379 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-16T13:59:49.379 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-16T13:59:49.380 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-16T13:59:49.380 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-16T13:59:49.380 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-16T13:59:49.380 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-16T13:59:49.380 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-16T13:59:49.381 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-16T13:59:49.381 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-16T13:59:49.381 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-16T13:59:49.381 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-16T13:59:49.381 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-16T13:59:49.381 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-16T13:59:49.382 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-16T13:59:49.382 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-16T13:59:49.382 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-16T13:59:49.382 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-16T13:59:49.382 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-16T13:59:49.383 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-16T13:59:49.383 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-16T13:59:49.383 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-16T13:59:49.383 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-16T13:59:49.383 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-16T13:59:49.383 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-16T13:59:49.384 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-16T13:59:49.384 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-16T13:59:49.384 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-16T13:59:49.384 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-16T13:59:49.384 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-16T13:59:49.384 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-16T13:59:49.385 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-16T13:59:49.385 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-16T13:59:49.385 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-16T13:59:49.385 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-16T13:59:49.385 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-16T13:59:49.385 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-16T13:59:49.386 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-16T13:59:49.386 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-16T13:59:49.386 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-16T13:59:49.386 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-16T13:59:49.386 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-16T13:59:49.387 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-16T13:59:49.387 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-16T13:59:49.387 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-16T13:59:49.387 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-16T13:59:49.387 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-16T13:59:49.388 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-16T13:59:49.388 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-16T13:59:49.388 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-16T13:59:49.388 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-16T13:59:49.388 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-16T13:59:49.388 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-16T13:59:49.389 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-16T13:59:49.389 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-16T13:59:49.389 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-16T13:59:49.389 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-16T13:59:49.389 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-16T13:59:49.389 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-16T13:59:49.390 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-16T13:59:49.390 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-16T13:59:49.390 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-16T13:59:49.390 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-16T13:59:49.390 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-16T13:59:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-16T13:59:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-16T13:59:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-16T13:59:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-16T13:59:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-16T13:59:49.391 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-16T13:59:49.392 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-16T13:59:49.392 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-16T13:59:49.392 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-16T13:59:49.392 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-16T13:59:49.392 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-16T13:59:49.393 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-16T13:59:49.393 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-16T13:59:49.393 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-16T13:59:49.393 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-16T13:59:49.393 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-16T13:59:49.393 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-16T13:59:49.394 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-16T13:59:49.394 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-16T13:59:49.394 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-16T13:59:49.394 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-16T13:59:49.394 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-16T13:59:49.395 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-16T13:59:49.395 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-16T13:59:49.395 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-16T13:59:49.395 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-16T13:59:49.395 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-16T13:59:49.395 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-16T13:59:49.396 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-16T13:59:49.396 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-16T13:59:49.396 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-16T13:59:49.396 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-16T13:59:49.396 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-16T13:59:49.397 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-16T13:59:49.397 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-16T13:59:49.397 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-16T13:59:49.397 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-16T13:59:49.397 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-16T13:59:49.397 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-16T13:59:49.398 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-16T13:59:49.398 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-16T13:59:49.398 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-16T13:59:49.398 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-16T13:59:49.398 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-16T13:59:49.399 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-16T13:59:49.399 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-16T13:59:49.399 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-16T13:59:49.399 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-16T13:59:49.399 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-16T13:59:49.399 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-16T13:59:49.400 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-16T13:59:49.400 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-16T13:59:49.400 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-16T13:59:49.400 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-16T13:59:49.400 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-16T13:59:49.401 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-16T13:59:49.401 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-16T13:59:49.401 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-16T13:59:49.401 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-16T13:59:49.401 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-16T13:59:49.402 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-16T13:59:49.402 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-16T13:59:49.402 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-16T13:59:49.402 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-16T13:59:49.402 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-16T13:59:49.402 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-16T13:59:49.403 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-16T13:59:49.403 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-16T13:59:49.403 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-16T13:59:49.403 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-16T13:59:49.404 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-16T13:59:49.404 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-16T13:59:49.404 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-16T13:59:49.404 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-16T13:59:49.404 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-16T13:59:49.405 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-16T13:59:49.405 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-16T13:59:49.405 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-16T13:59:49.405 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-16T13:59:49.405 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-16T13:59:49.406 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-16T13:59:49.406 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-16T13:59:49.406 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-16T13:59:49.406 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-16T13:59:49.406 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-16T13:59:49.406 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-16T13:59:49.407 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-16T13:59:49.407 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-16T13:59:49.407 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-16T13:59:49.407 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-16T13:59:49.407 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-16T13:59:49.408 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-16T13:59:49.408 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-16T13:59:49.408 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-16T13:59:49.408 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-16T13:59:49.408 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-16T13:59:49.408 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-16T13:59:49.409 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-16T13:59:49.409 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-16T13:59:49.409 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-16T13:59:49.409 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-16T13:59:49.409 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-16T13:59:49.409 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-16T13:59:49.410 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-16T13:59:49.410 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-16T13:59:49.410 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-16T13:59:49.410 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-16T13:59:49.410 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-16T13:59:49.411 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-16T13:59:49.411 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-16T13:59:49.411 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-16T13:59:49.411 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-16T13:59:49.411 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-16T13:59:49.412 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-16T13:59:49.412 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-16T13:59:49.412 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-16T13:59:49.412 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-16T13:59:49.412 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-16T13:59:49.412 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-16T13:59:49.413 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-16T13:59:49.413 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-16T13:59:49.413 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-16T13:59:49.413 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-16T13:59:49.413 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-16T13:59:49.414 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-16T13:59:49.414 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-16T13:59:49.414 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-16T13:59:49.414 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-16T13:59:49.414 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-16T13:59:49.414 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-16T13:59:49.415 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-16T13:59:49.415 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-16T13:59:49.415 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-16T13:59:49.415 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-16T13:59:49.415 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-16T13:59:49.416 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-16T13:59:49.416 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-16T13:59:49.416 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-16T13:59:49.416 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-16T13:59:49.416 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-16T13:59:49.416 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-16T13:59:49.417 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-16T13:59:49.417 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-16T13:59:49.417 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-16T13:59:49.417 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-16T13:59:49.417 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-16T13:59:49.418 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-16T13:59:49.418 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-16T13:59:49.418 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-16T13:59:49.418 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-16T13:59:49.418 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-16T13:59:49.418 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-16T13:59:49.419 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-16T13:59:49.419 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-16T13:59:49.419 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-16T13:59:49.419 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-16T13:59:49.419 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-16T13:59:49.420 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-16T13:59:49.420 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-16T13:59:49.420 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-16T13:59:49.420 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-16T13:59:49.420 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-16T13:59:49.420 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-16T13:59:49.421 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-16T13:59:49.421 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-16T13:59:49.421 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-16T13:59:49.421 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-16T13:59:49.421 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-16T13:59:49.422 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-16T13:59:49.422 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-16T13:59:49.422 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-16T13:59:49.422 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-16T13:59:49.422 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-16T13:59:49.423 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-16T13:59:49.423 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-16T13:59:49.423 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-16T13:59:49.423 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-16T13:59:49.423 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-16T13:59:49.424 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-16T13:59:49.424 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-16T13:59:49.424 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-16T13:59:49.424 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-16T13:59:49.424 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-16T13:59:49.425 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-16T13:59:49.425 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-16T13:59:49.425 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-16T13:59:49.425 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-16T13:59:49.425 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-16T13:59:49.425 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-16T13:59:49.426 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-16T13:59:49.426 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-16T13:59:49.426 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-16T13:59:49.426 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-16T13:59:49.426 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-16T13:59:49.427 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-16T13:59:49.427 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-16T13:59:49.427 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-16T13:59:49.427 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-16T13:59:49.427 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-16T13:59:49.428 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-16T13:59:49.428 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-16T13:59:49.428 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-16T13:59:49.428 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-16T13:59:49.428 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-16T13:59:49.429 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-16T13:59:49.429 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-16T13:59:49.429 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-16T13:59:49.429 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-16T13:59:49.429 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-16T13:59:49.430 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-16T13:59:49.430 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-16T13:59:49.430 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-16T13:59:49.430 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-16T13:59:49.430 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-16T13:59:49.431 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-16T13:59:49.431 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-16T13:59:49.431 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-16T13:59:49.431 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-16T13:59:49.431 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-16T13:59:49.432 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-16T13:59:49.432 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-16T13:59:49.432 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-16T13:59:49.432 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-16T13:59:49.432 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-16T13:59:49.433 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-16T13:59:49.433 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-16T13:59:49.433 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-16T13:59:49.433 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-16T13:59:49.433 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-16T13:59:49.433 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-16T13:59:49.434 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-16T13:59:49.434 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-16T13:59:49.434 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-16T13:59:49.435 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-16T13:59:49.435 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-16T13:59:49.435 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-16T13:59:49.435 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-16T13:59:49.436 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-16T13:59:49.436 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-16T13:59:49.436 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-16T13:59:49.436 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-16T13:59:49.436 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-16T13:59:49.436 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-16T13:59:49.437 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-16T13:59:49.437 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-16T13:59:49.437 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-16T13:59:49.437 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-16T13:59:49.438 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-16T13:59:49.438 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-16T13:59:49.438 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-16T13:59:49.439 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-16T13:59:49.439 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-16T13:59:49.439 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-16T13:59:49.440 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-16T13:59:49.440 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-16T13:59:49.440 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-16T13:59:49.441 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-16T13:59:49.441 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-16T13:59:49.441 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-16T13:59:49.442 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-16T13:59:49.442 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-16T13:59:49.442 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-16T13:59:49.443 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-16T13:59:49.443 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-16T13:59:49.443 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-16T13:59:49.444 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-16T13:59:49.444 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-16T13:59:49.445 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-16T13:59:49.445 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-16T13:59:49.445 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-16T13:59:49.446 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-16T13:59:49.446 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-16T13:59:49.446 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-16T13:59:49.446 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-16T13:59:49.447 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-16T13:59:49.447 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-16T13:59:49.447 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-16T13:59:49.447 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-16T13:59:49.448 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-16T13:59:49.448 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-16T13:59:49.448 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-16T13:59:49.448 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-16T13:59:49.448 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-16T13:59:49.448 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-16T13:59:49.449 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-16T13:59:49.449 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-16T13:59:49.449 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-16T13:59:49.449 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-16T13:59:49.449 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-16T13:59:49.450 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-16T13:59:49.450 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-16T13:59:49.450 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-16T13:59:49.450 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-16T13:59:49.450 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-16T13:59:49.451 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-16T13:59:49.451 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-16T13:59:49.451 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-16T13:59:49.451 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-16T13:59:49.451 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-16T13:59:49.451 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-16T13:59:49.452 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-16T13:59:49.452 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-16T13:59:49.452 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-16T13:59:49.452 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-16T13:59:49.452 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-16T13:59:49.453 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-16T13:59:49.453 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-16T13:59:49.453 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-16T13:59:49.453 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-16T13:59:49.453 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-16T13:59:49.453 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-16T13:59:49.454 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-16T13:59:49.454 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-16T13:59:49.454 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-16T13:59:49.454 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-16T13:59:49.454 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-16T13:59:49.455 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-16T13:59:49.455 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-16T13:59:49.455 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-16T13:59:49.455 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-16T13:59:49.455 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-16T13:59:49.455 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-16T13:59:49.456 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-16T13:59:49.456 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-16T13:59:49.456 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-16T13:59:49.456 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-16T13:59:49.456 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-16T13:59:49.456 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-16T13:59:49.457 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-16T13:59:49.457 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-16T13:59:49.457 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-16T13:59:49.457 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-16T13:59:49.457 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-16T13:59:49.457 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-16T13:59:49.458 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-16T13:59:49.458 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-16T13:59:49.458 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-16T13:59:49.458 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-16T13:59:49.458 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-16T13:59:49.459 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-16T13:59:49.459 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-16T13:59:49.459 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-16T13:59:49.459 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-16T13:59:49.459 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-16T13:59:49.459 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-16T13:59:49.460 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-16T13:59:49.460 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-16T13:59:49.460 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-16T13:59:49.460 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-16T13:59:49.460 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-16T13:59:49.460 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-16T13:59:49.461 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-16T13:59:49.461 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-16T13:59:49.461 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-16T13:59:49.461 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-16T13:59:49.461 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-16T13:59:49.462 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-16T13:59:49.462 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-16T13:59:49.462 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-16T13:59:49.462 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-16T13:59:49.462 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-16T13:59:49.463 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-16T13:59:49.463 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-16T13:59:49.463 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-16T13:59:49.463 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-16T13:59:49.463 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-16T13:59:49.464 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-16T13:59:49.464 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-16T13:59:49.464 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-16T13:59:49.464 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-16T13:59:49.465 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-16T13:59:49.465 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-16T13:59:49.465 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-16T13:59:49.466 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-16T13:59:49.466 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-16T13:59:49.466 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-16T13:59:49.466 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-16T13:59:49.467 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-16T13:59:49.467 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-16T13:59:49.467 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-16T13:59:49.467 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-16T13:59:49.468 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-16T13:59:49.468 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-16T13:59:49.468 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-16T13:59:49.469 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-16T13:59:49.469 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-16T13:59:49.469 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-16T13:59:49.469 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-16T13:59:49.469 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-16T13:59:49.469 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-16T13:59:49.470 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-16T13:59:49.470 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-16T13:59:49.470 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-16T13:59:49.470 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-16T13:59:49.471 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-16T13:59:49.471 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-16T13:59:49.471 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-16T13:59:49.471 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-16T13:59:49.471 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-16T13:59:49.472 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-16T13:59:49.472 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-16T13:59:49.472 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-16T13:59:49.472 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-16T13:59:49.472 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-16T13:59:49.472 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-16T13:59:49.473 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-16T13:59:49.473 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-16T13:59:49.473 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-16T13:59:49.473 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-16T13:59:49.473 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-16T13:59:49.474 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-16T13:59:49.474 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-16T13:59:49.474 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-16T13:59:49.474 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-16T13:59:49.474 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-16T13:59:49.475 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-16T13:59:49.475 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-16T13:59:49.475 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-16T13:59:49.475 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-16T13:59:49.475 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-16T13:59:49.476 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-16T13:59:49.476 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-16T13:59:49.476 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-16T13:59:49.476 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-16T13:59:49.476 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-16T13:59:49.476 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-16T13:59:49.477 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-16T13:59:49.477 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-16T13:59:49.477 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-16T13:59:49.477 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-16T13:59:49.478 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-16T13:59:49.478 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-16T13:59:49.478 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-16T13:59:49.478 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-16T13:59:49.478 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-16T13:59:49.479 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-16T13:59:49.479 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-16T13:59:49.479 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-16T13:59:49.479 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-16T13:59:49.479 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-16T13:59:49.480 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-16T13:59:49.480 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-16T13:59:49.480 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-16T13:59:49.480 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-16T13:59:49.480 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-16T13:59:49.481 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-16T13:59:49.481 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-16T13:59:49.481 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-16T13:59:49.481 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-16T13:59:49.481 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-16T13:59:49.481 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-16T13:59:49.482 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-16T13:59:49.482 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-16T13:59:49.482 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-16T13:59:49.482 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-16T13:59:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-16T13:59:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-16T13:59:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-16T13:59:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-16T13:59:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-16T13:59:49.483 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-16T13:59:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-16T13:59:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-16T13:59:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-16T13:59:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-16T13:59:49.484 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-16T13:59:49.485 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-16T13:59:49.485 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-16T13:59:49.485 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-16T13:59:49.485 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-16T13:59:49.485 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-16T13:59:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-16T13:59:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-16T13:59:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-16T13:59:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-16T13:59:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-16T13:59:49.487 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-16T13:59:49.487 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-16T13:59:49.487 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-16T13:59:49.487 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-16T13:59:49.487 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-16T13:59:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-16T13:59:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-16T13:59:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-16T13:59:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-16T13:59:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-16T13:59:49.489 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-16T13:59:49.489 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-16T13:59:49.489 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-16T13:59:49.489 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-16T13:59:49.490 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-16T13:59:49.490 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-16T13:59:49.490 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-16T13:59:49.490 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-16T13:59:49.490 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-16T13:59:49.491 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-16T13:59:49.491 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-16T13:59:49.491 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-16T13:59:49.491 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-16T13:59:49.491 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-16T13:59:49.492 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-16T13:59:49.492 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-16T13:59:49.492 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-16T13:59:49.492 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-16T13:59:49.492 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-16T13:59:49.493 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-16T13:59:49.493 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-16T13:59:49.493 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-16T13:59:49.493 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-16T13:59:49.493 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-16T13:59:49.494 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-16T13:59:49.494 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-16T13:59:49.494 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-16T13:59:49.494 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-16T13:59:49.495 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-16T13:59:49.495 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-16T13:59:49.495 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-16T13:59:49.495 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-16T13:59:49.495 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-16T13:59:49.495 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-16T13:59:49.496 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-16T13:59:49.496 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-16T13:59:49.496 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-16T13:59:49.496 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-16T13:59:49.496 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-16T13:59:49.497 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-16T13:59:49.497 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-16T13:59:49.497 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-16T13:59:49.497 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-16T13:59:49.497 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-16T13:59:49.498 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-16T13:59:49.498 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-16T13:59:49.498 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-16T13:59:49.498 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-16T13:59:49.498 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-16T13:59:49.499 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-16T13:59:49.499 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-16T13:59:49.499 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-16T13:59:49.499 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-16T13:59:49.499 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-16T13:59:49.500 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-16T13:59:49.500 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-16T13:59:49.500 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-16T13:59:49.500 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-16T13:59:49.500 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-16T13:59:49.501 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-16T13:59:49.501 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-16T13:59:49.501 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-16T13:59:49.501 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-16T13:59:49.501 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-16T13:59:49.502 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-16T13:59:49.502 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-16T13:59:49.502 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-16T13:59:49.502 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-16T13:59:49.502 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-16T13:59:49.503 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-16T13:59:49.503 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-16T13:59:49.503 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-16T13:59:49.503 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-16T13:59:49.503 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-16T13:59:49.504 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-16T13:59:49.504 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-16T13:59:49.504 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-16T13:59:49.504 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-16T13:59:49.504 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-16T13:59:49.505 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-16T13:59:49.505 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-16T13:59:49.505 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-16T13:59:49.505 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-16T13:59:49.505 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-16T13:59:49.505 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-16T13:59:49.506 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-16T13:59:49.506 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-16T13:59:49.506 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-16T13:59:49.506 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-16T13:59:49.507 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-16T13:59:49.507 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-16T13:59:49.507 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-16T13:59:49.507 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-16T13:59:49.507 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-16T13:59:49.507 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-16T13:59:49.508 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-16T13:59:49.508 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-16T13:59:49.508 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-16T13:59:49.508 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-16T13:59:49.508 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-16T13:59:49.508 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-16T13:59:49.509 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-16T13:59:49.509 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-16T13:59:49.509 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-16T13:59:49.509 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-16T13:59:49.509 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-16T13:59:49.509 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-16T13:59:49.510 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-16T13:59:49.510 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-16T13:59:49.510 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-16T13:59:49.510 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-16T13:59:49.510 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-16T13:59:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-16T13:59:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-16T13:59:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-16T13:59:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-16T13:59:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-16T13:59:49.512 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-16T13:59:49.512 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-16T13:59:49.512 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-16T13:59:49.512 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-16T13:59:49.512 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-16T13:59:49.513 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-16T13:59:49.513 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-16T13:59:49.513 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-16T13:59:49.513 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-16T13:59:49.513 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-16T13:59:49.514 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-16T13:59:49.514 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-16T13:59:49.514 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-16T13:59:49.514 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-16T13:59:49.514 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-16T13:59:49.515 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-16T13:59:49.515 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-16T13:59:49.515 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-16T13:59:49.515 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-16T13:59:49.515 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-16T13:59:49.516 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-16T13:59:49.516 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-16T13:59:49.516 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-16T13:59:49.516 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-16T13:59:49.516 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-16T13:59:49.516 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-16T13:59:49.517 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-16T13:59:49.517 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-16T13:59:49.517 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-16T13:59:49.517 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-16T13:59:49.517 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-16T13:59:49.518 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-16T13:59:49.518 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-16T13:59:49.518 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-16T13:59:49.518 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-16T13:59:49.519 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-16T13:59:49.519 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-16T13:59:49.519 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-16T13:59:49.519 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-16T13:59:49.519 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-16T13:59:49.519 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-16T13:59:49.520 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-16T13:59:49.520 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-16T13:59:49.520 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-16T13:59:49.520 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-16T13:59:49.520 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-16T13:59:49.521 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-16T13:59:49.521 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-16T13:59:49.521 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-16T13:59:49.521 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-16T13:59:49.521 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-16T13:59:49.522 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-16T13:59:49.522 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-16T13:59:49.522 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-16T13:59:49.522 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-16T13:59:49.522 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-16T13:59:49.523 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-16T13:59:49.523 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-16T13:59:49.523 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-16T13:59:49.523 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-16T13:59:49.523 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-16T13:59:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-16T13:59:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-16T13:59:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-16T13:59:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-16T13:59:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-16T13:59:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-16T13:59:49.525 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-16T13:59:49.525 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-16T13:59:49.525 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-16T13:59:49.525 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-16T13:59:49.526 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-16T13:59:49.526 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-16T13:59:49.526 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-16T13:59:49.526 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-16T13:59:49.526 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-16T13:59:49.526 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-16T13:59:49.527 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-16T13:59:49.527 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-16T13:59:49.527 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-16T13:59:49.527 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-16T13:59:49.527 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-16T13:59:49.528 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-16T13:59:49.528 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-16T13:59:49.528 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-16T13:59:49.528 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-16T13:59:49.528 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-16T13:59:49.529 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-16T13:59:49.529 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-16T13:59:49.529 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-16T13:59:49.529 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-16T13:59:49.529 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-16T13:59:49.530 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-16T13:59:49.530 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-16T13:59:49.530 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-16T13:59:49.530 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-16T13:59:49.530 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-16T13:59:49.531 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-16T13:59:49.531 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-16T13:59:49.531 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-16T13:59:49.531 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-16T13:59:49.531 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-16T13:59:49.532 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-16T13:59:49.532 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-16T13:59:49.532 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-16T13:59:49.532 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-16T13:59:49.532 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-16T13:59:49.532 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-16T13:59:49.533 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-16T13:59:49.533 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-16T13:59:49.533 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-16T13:59:49.533 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-16T13:59:49.533 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-16T13:59:49.534 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-16T13:59:49.534 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-16T13:59:49.534 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-16T13:59:49.534 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-16T13:59:49.534 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-16T13:59:49.535 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-16T13:59:49.535 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-16T13:59:49.535 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-16T13:59:49.535 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-16T13:59:49.535 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-16T13:59:49.536 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-16T13:59:49.536 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-16T13:59:49.536 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-16T13:59:49.536 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-16T13:59:49.536 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-16T13:59:49.537 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-16T13:59:49.537 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-16T13:59:49.537 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-16T13:59:49.537 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-16T13:59:49.537 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-16T13:59:49.538 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-16T13:59:49.538 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-16T13:59:49.538 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-16T13:59:49.538 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-16T13:59:49.538 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-16T13:59:49.539 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-16T13:59:49.539 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-16T13:59:49.539 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-16T13:59:49.539 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-16T13:59:49.539 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-16T13:59:49.539 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-16T13:59:49.540 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-16T13:59:49.540 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-16T13:59:49.540 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-16T13:59:49.540 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-16T13:59:49.540 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-16T13:59:49.541 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-16T13:59:49.541 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-16T13:59:49.541 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-16T13:59:49.541 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-16T13:59:49.541 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-16T13:59:49.542 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-16T13:59:49.542 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-16T13:59:49.542 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-16T13:59:49.542 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-16T13:59:49.542 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-16T13:59:49.543 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-16T13:59:49.543 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-16T13:59:49.543 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-16T13:59:49.543 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-16T13:59:49.543 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-16T13:59:49.543 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-16T13:59:49.544 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-16T13:59:49.544 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-16T13:59:49.544 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-16T13:59:49.544 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-16T13:59:49.544 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-16T13:59:49.545 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-16T13:59:49.545 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-16T13:59:49.545 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-16T13:59:49.545 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-16T13:59:49.545 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-16T13:59:49.546 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-16T13:59:49.546 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-16T13:59:49.546 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-16T13:59:49.546 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-16T13:59:49.546 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-16T13:59:49.546 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-16T13:59:49.547 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-16T13:59:49.547 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-16T13:59:49.547 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-16T13:59:49.547 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-16T13:59:49.547 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-16T13:59:49.548 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-16T13:59:49.548 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-16T13:59:49.548 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-16T13:59:49.548 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-16T13:59:49.548 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-16T13:59:49.548 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-16T13:59:49.549 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-16T13:59:49.549 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-16T13:59:49.549 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-16T13:59:49.549 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-16T13:59:49.549 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-16T13:59:49.550 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-16T13:59:49.550 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-16T13:59:49.550 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-16T13:59:49.550 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-16T13:59:49.550 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-16T13:59:49.551 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-16T13:59:49.551 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-16T13:59:49.551 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-16T13:59:49.551 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-16T13:59:49.551 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-16T13:59:49.552 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-16T13:59:49.552 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-16T13:59:49.552 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-16T13:59:49.552 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-16T13:59:49.552 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-16T13:59:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-16T13:59:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-16T13:59:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-16T13:59:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-16T13:59:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-16T13:59:49.553 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-16T13:59:49.554 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-16T13:59:49.554 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-16T13:59:49.554 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-16T13:59:49.554 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-16T13:59:49.554 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-16T13:59:49.555 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-16T13:59:49.555 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-16T13:59:49.555 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-16T13:59:49.555 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-16T13:59:49.555 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-16T13:59:49.556 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-16T13:59:49.556 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-16T13:59:49.556 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-16T13:59:49.556 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-16T13:59:49.556 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-16T13:59:49.556 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-16T13:59:49.557 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-16T13:59:49.557 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-16T13:59:49.557 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-16T13:59:49.557 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-16T13:59:49.557 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-16T13:59:49.558 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-16T13:59:49.558 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-16T13:59:49.558 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-16T13:59:49.558 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-16T13:59:49.558 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-16T13:59:49.558 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-16T13:59:49.559 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-16T13:59:49.559 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-16T13:59:49.559 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-16T13:59:49.559 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-16T13:59:49.559 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-16T13:59:49.560 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-16T13:59:49.560 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-16T13:59:49.560 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-16T13:59:49.560 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-16T13:59:49.560 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-16T13:59:49.560 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-16T13:59:49.561 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-16T13:59:49.561 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-16T13:59:49.561 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-16T13:59:49.561 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-16T13:59:49.562 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-16T13:59:49.562 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-16T13:59:49.562 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-16T13:59:49.562 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-16T13:59:49.562 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-16T13:59:49.562 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-16T13:59:49.563 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-16T13:59:49.563 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-16T13:59:49.563 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-16T13:59:49.563 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-16T13:59:49.563 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-16T13:59:49.564 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-16T13:59:49.564 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-16T13:59:49.564 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-16T13:59:49.564 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-16T13:59:49.564 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-16T13:59:49.565 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-16T13:59:49.565 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-16T13:59:49.565 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-16T13:59:49.565 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-16T13:59:49.565 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-16T13:59:49.565 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-16T13:59:49.566 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-16T13:59:49.566 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-16T13:59:49.566 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-16T13:59:49.566 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-16T13:59:49.566 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-16T13:59:49.566 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-16T13:59:49.567 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-16T13:59:49.567 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-16T13:59:49.567 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-16T13:59:49.567 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-16T13:59:49.567 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-16T13:59:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-16T13:59:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-16T13:59:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-16T13:59:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-16T13:59:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-16T13:59:49.568 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-16T13:59:49.569 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-16T13:59:49.569 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-16T13:59:49.569 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-16T13:59:49.569 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-16T13:59:49.569 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-16T13:59:49.570 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-16T13:59:49.570 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-16T13:59:49.570 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-16T13:59:49.570 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-16T13:59:49.570 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-16T13:59:49.571 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-16T13:59:49.571 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-16T13:59:49.571 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-16T13:59:49.571 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-16T13:59:49.571 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-16T13:59:49.571 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-16T13:59:49.572 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-16T13:59:49.572 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-16T13:59:49.572 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-16T13:59:49.572 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-16T13:59:49.572 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-16T13:59:49.573 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-16T13:59:49.573 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-16T13:59:49.573 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-16T13:59:49.573 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-16T13:59:49.573 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-16T13:59:49.573 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-16T13:59:49.574 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-16T13:59:49.574 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-16T13:59:49.574 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-16T13:59:49.574 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-16T13:59:49.574 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-16T13:59:49.575 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-16T13:59:49.575 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-16T13:59:49.575 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-16T13:59:49.575 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-16T13:59:49.575 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-16T13:59:49.575 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-16T13:59:49.576 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-16T13:59:49.576 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-16T13:59:49.576 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-16T13:59:49.576 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-16T13:59:49.576 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-16T13:59:49.577 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-16T13:59:49.577 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-16T13:59:49.577 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-16T13:59:49.577 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-16T13:59:49.577 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-16T13:59:49.578 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-16T13:59:49.578 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-16T13:59:49.578 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-16T13:59:49.578 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-16T13:59:49.578 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-16T13:59:49.579 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-16T13:59:49.579 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-16T13:59:49.579 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-16T13:59:49.579 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-16T13:59:49.579 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-16T13:59:49.580 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-16T13:59:49.580 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-16T13:59:49.580 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-16T13:59:49.580 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-16T13:59:49.580 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-16T13:59:49.581 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-16T13:59:49.581 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-16T13:59:49.581 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-16T13:59:49.581 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-16T13:59:49.581 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-16T13:59:49.581 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-16T13:59:49.582 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-16T13:59:49.582 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-16T13:59:49.582 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-16T13:59:49.582 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-16T13:59:49.582 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-16T13:59:49.583 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-16T13:59:49.583 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-16T13:59:49.583 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-16T13:59:49.583 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-16T13:59:49.583 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-16T13:59:49.584 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-16T13:59:49.584 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-16T13:59:49.584 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-16T13:59:49.584 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-16T13:59:49.584 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-16T13:59:49.584 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-16T13:59:49.585 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-16T13:59:49.585 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-16T13:59:49.585 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-16T13:59:49.585 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-16T13:59:49.585 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-16T13:59:49.585 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-16T13:59:49.586 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-16T13:59:49.586 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-16T13:59:49.586 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-16T13:59:49.586 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-16T13:59:49.586 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-16T13:59:49.587 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-16T13:59:49.587 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-16T13:59:49.587 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-16T13:59:49.587 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-16T13:59:49.587 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-16T13:59:49.587 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-16T13:59:49.588 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-16T13:59:49.588 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-16T13:59:49.588 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-16T13:59:49.588 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-16T13:59:49.588 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-16T13:59:49.589 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-16T13:59:49.589 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-16T13:59:49.589 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-16T13:59:49.589 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-16T13:59:49.589 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-16T13:59:49.589 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-16T13:59:49.590 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-16T13:59:49.590 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-16T13:59:49.590 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-16T13:59:49.590 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-16T13:59:49.590 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-16T13:59:49.591 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-16T13:59:49.591 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-16T13:59:49.591 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-16T13:59:49.591 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-16T13:59:49.591 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-16T13:59:49.591 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-16T13:59:49.592 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-16T13:59:49.592 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-16T13:59:49.592 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-16T13:59:49.592 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-16T13:59:49.592 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-16T13:59:49.593 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-16T13:59:49.593 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-16T13:59:49.593 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-16T13:59:49.593 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-16T13:59:49.593 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-16T13:59:49.593 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-16T13:59:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-16T13:59:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-16T13:59:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-16T13:59:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-16T13:59:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-16T13:59:49.594 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-16T13:59:49.595 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-16T13:59:49.595 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-16T13:59:49.595 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-16T13:59:49.595 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-16T13:59:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-16T13:59:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-16T13:59:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-16T13:59:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-16T13:59:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-16T13:59:49.596 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-16T13:59:49.597 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-16T13:59:49.597 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-16T13:59:49.597 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-16T13:59:49.597 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-16T13:59:49.597 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-16T13:59:49.598 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-16T13:59:49.598 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-16T13:59:49.598 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-16T13:59:49.598 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-16T13:59:49.598 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-16T13:59:49.599 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-16T13:59:49.599 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-16T13:59:49.599 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-16T13:59:49.599 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-16T13:59:49.599 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-16T13:59:49.600 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-16T13:59:49.600 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-16T13:59:49.600 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-16T13:59:49.600 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-16T13:59:49.600 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-16T13:59:49.601 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-16T13:59:49.601 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-16T13:59:49.601 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-16T13:59:49.601 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-16T13:59:49.601 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-16T13:59:49.602 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-16T13:59:49.602 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-16T13:59:49.602 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-16T13:59:49.602 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-16T13:59:49.602 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-16T13:59:49.603 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-16T13:59:49.603 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-16T13:59:49.603 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-16T13:59:49.603 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-16T13:59:49.603 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-16T13:59:49.604 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-16T13:59:49.604 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-16T13:59:49.604 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-16T13:59:49.604 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-16T13:59:49.605 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-16T13:59:49.605 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-16T13:59:49.605 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-16T13:59:49.605 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-16T13:59:49.605 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-16T13:59:49.605 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-16T13:59:49.606 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-16T13:59:49.606 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-16T13:59:49.606 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-16T13:59:49.606 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-16T13:59:49.606 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-16T13:59:49.607 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-16T13:59:49.607 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-16T13:59:49.607 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-16T13:59:49.607 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-16T13:59:49.607 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-16T13:59:49.607 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-16T13:59:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-16T13:59:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-16T13:59:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-16T13:59:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-16T13:59:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-16T13:59:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-16T13:59:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-16T13:59:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-16T13:59:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-16T13:59:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-16T13:59:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-16T13:59:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-16T13:59:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-16T13:59:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-16T13:59:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-16T13:59:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-16T13:59:49.611 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-16T13:59:49.611 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-16T13:59:49.611 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-16T13:59:49.611 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-16T13:59:49.611 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-16T13:59:49.612 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-16T13:59:49.612 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-16T13:59:49.612 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-16T13:59:49.612 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-16T13:59:49.612 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-16T13:59:49.612 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-16T13:59:49.613 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-16T13:59:49.613 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-16T13:59:49.613 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-16T13:59:49.613 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-16T13:59:49.613 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-16T13:59:49.614 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-16T13:59:49.614 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-16T13:59:49.614 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-16T13:59:49.614 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-16T13:59:49.614 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-16T13:59:49.615 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-16T13:59:49.615 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-16T13:59:49.615 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-16T13:59:49.615 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-16T13:59:49.615 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-16T13:59:49.616 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-16T13:59:49.616 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-16T13:59:49.616 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-16T13:59:49.616 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-16T13:59:49.616 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-16T13:59:49.617 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-16T13:59:49.617 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-16T13:59:49.617 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-16T13:59:49.617 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-16T13:59:49.617 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-16T13:59:49.618 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-16T13:59:49.619 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-16T13:59:49.620 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-16T13:59:49.621 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-16T13:59:49.621 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-16T13:59:49.621 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-16T13:59:49.621 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-16T13:59:49.621 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-16T13:59:49.622 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-16T13:59:49.622 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-16T13:59:49.622 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-16T13:59:49.622 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-16T13:59:49.623 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-16T13:59:49.623 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-16T13:59:49.623 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-16T13:59:49.623 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-16T13:59:49.623 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-16T13:59:49.624 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-16T13:59:49.624 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-16T13:59:49.624 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-16T13:59:49.624 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-16T13:59:49.624 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-16T13:59:49.625 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-16T13:59:49.625 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-16T13:59:49.625 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-16T13:59:49.625 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-16T13:59:49.626 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-16T13:59:49.626 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-16T13:59:49.626 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-16T13:59:49.626 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-16T13:59:49.627 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-16T13:59:49.627 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-16T13:59:49.627 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-16T13:59:49.627 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-16T13:59:49.627 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-16T13:59:49.628 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-16T13:59:49.628 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-16T13:59:49.628 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-16T13:59:49.628 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-16T13:59:49.628 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-16T13:59:49.628 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-16T13:59:49.629 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-16T13:59:49.629 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-16T13:59:49.629 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-16T13:59:49.629 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-16T13:59:49.629 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-16T13:59:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-16T13:59:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-16T13:59:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-16T13:59:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-16T13:59:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-16T13:59:49.631 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-16T13:59:49.631 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-16T13:59:49.631 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-16T13:59:49.631 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-16T13:59:49.631 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-16T13:59:49.632 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-16T13:59:49.632 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-16T13:59:49.632 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-16T13:59:49.632 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-16T13:59:49.632 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-16T13:59:49.633 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-16T13:59:49.633 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-16T13:59:49.633 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-16T13:59:49.633 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-16T13:59:49.633 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-16T13:59:49.634 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-16T13:59:49.634 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-16T13:59:49.634 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-16T13:59:49.634 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-16T13:59:49.635 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-16T13:59:49.635 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-16T13:59:49.635 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-16T13:59:49.635 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-16T13:59:49.635 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-16T13:59:49.635 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-16T13:59:49.636 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-16T13:59:49.636 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-16T13:59:49.636 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-16T13:59:49.636 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-16T13:59:49.636 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-16T13:59:49.637 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-16T13:59:49.637 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-16T13:59:49.637 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-16T13:59:49.637 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-16T13:59:49.637 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-16T13:59:49.638 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-16T13:59:49.638 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-16T13:59:49.638 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-16T13:59:49.638 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-16T13:59:49.639 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-16T13:59:49.639 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-16T13:59:49.639 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-16T13:59:49.639 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-16T13:59:49.639 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-16T13:59:49.640 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-16T13:59:49.640 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-16T13:59:49.640 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-16T13:59:49.640 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-16T13:59:49.640 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-16T13:59:49.640 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-16T13:59:49.641 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-16T13:59:49.641 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-16T13:59:49.641 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-16T13:59:49.641 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-16T13:59:49.642 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-16T13:59:49.642 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-16T13:59:49.642 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-16T13:59:49.642 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-16T13:59:49.642 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-16T13:59:49.643 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-16T13:59:49.643 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-16T13:59:49.643 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-16T13:59:49.643 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-16T13:59:49.643 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-16T13:59:49.643 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-16T13:59:49.644 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-16T13:59:49.644 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-16T13:59:49.644 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-16T13:59:49.644 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-16T13:59:49.644 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-16T13:59:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-16T13:59:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-16T13:59:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-16T13:59:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-16T13:59:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-16T13:59:49.646 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-16T13:59:49.646 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-16T13:59:49.646 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-16T13:59:49.646 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-16T13:59:49.647 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-16T13:59:49.647 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-16T13:59:49.647 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-16T13:59:49.647 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-16T13:59:49.647 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-16T13:59:49.647 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-16T13:59:49.648 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-16T13:59:49.648 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-16T13:59:49.648 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-16T13:59:49.648 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-16T13:59:49.648 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-16T13:59:49.649 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-16T13:59:49.649 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-16T13:59:49.649 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-16T13:59:49.649 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-16T13:59:49.649 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-16T13:59:49.650 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-16T13:59:49.650 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-16T13:59:49.650 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-16T13:59:49.650 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-16T13:59:49.650 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-16T13:59:49.651 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-16T13:59:49.651 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-16T13:59:49.651 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-16T13:59:49.651 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-16T13:59:49.651 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-16T13:59:49.651 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-16T13:59:49.652 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-16T13:59:49.652 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-16T13:59:49.652 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-16T13:59:49.652 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-16T13:59:49.653 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-16T13:59:49.653 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-16T13:59:49.653 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-16T13:59:49.653 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-16T13:59:49.653 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-16T13:59:49.653 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-16T13:59:49.654 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-16T13:59:49.654 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-16T13:59:49.654 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-16T13:59:49.654 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-16T13:59:49.654 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-16T13:59:49.655 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-16T13:59:49.655 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-16T13:59:49.655 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-16T13:59:49.655 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-16T13:59:49.655 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-16T13:59:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-16T13:59:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-16T13:59:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-16T13:59:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-16T13:59:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-16T13:59:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-16T13:59:49.657 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-16T13:59:49.657 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-16T13:59:49.657 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-16T13:59:49.657 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-16T13:59:49.657 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-16T13:59:49.658 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-16T13:59:49.658 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-16T13:59:49.658 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-16T13:59:49.658 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-16T13:59:49.658 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-16T13:59:49.658 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-16T13:59:49.659 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-16T13:59:49.659 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-16T13:59:49.659 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-16T13:59:49.659 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-16T13:59:49.659 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-16T13:59:49.660 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-16T13:59:49.660 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-16T13:59:49.660 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-16T13:59:49.660 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-16T13:59:49.661 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-16T13:59:49.661 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-16T13:59:49.661 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-16T13:59:49.661 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-16T13:59:49.661 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-16T13:59:49.662 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-16T13:59:49.662 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-16T13:59:49.662 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-16T13:59:49.662 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-16T13:59:49.662 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-16T13:59:49.663 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-16T13:59:49.663 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-16T13:59:49.663 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-16T13:59:49.663 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-16T13:59:49.663 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-16T13:59:49.664 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-16T13:59:49.664 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-16T13:59:49.664 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-16T13:59:49.664 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-16T13:59:49.664 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-16T13:59:49.664 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-16T13:59:49.665 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-16T13:59:49.665 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-16T13:59:49.665 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-16T13:59:49.665 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-16T13:59:49.665 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-16T13:59:49.666 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-16T13:59:49.666 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-16T13:59:49.666 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-16T13:59:49.666 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-16T13:59:49.666 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-16T13:59:49.667 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-16T13:59:49.667 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-16T13:59:49.667 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-16T13:59:49.667 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-16T13:59:49.667 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-16T13:59:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-16T13:59:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-16T13:59:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-16T13:59:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-16T13:59:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-16T13:59:49.669 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-16T13:59:49.669 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-16T13:59:49.669 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-16T13:59:49.669 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-16T13:59:49.669 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-16T13:59:49.670 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-16T13:59:49.670 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-16T13:59:49.670 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-16T13:59:49.670 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-16T13:59:49.670 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-16T13:59:49.671 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-16T13:59:49.671 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-16T13:59:49.671 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-16T13:59:49.671 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-16T13:59:49.671 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-16T13:59:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-16T13:59:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-16T13:59:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-16T13:59:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-16T13:59:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-16T13:59:49.672 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-16T13:59:49.673 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-16T13:59:49.673 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-16T13:59:49.673 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-16T13:59:49.673 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-16T13:59:49.673 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-16T13:59:49.673 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-16T13:59:49.674 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-16T13:59:49.674 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-16T13:59:49.674 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-16T13:59:49.674 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-16T13:59:49.674 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-16T13:59:49.675 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-16T13:59:49.675 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-16T13:59:49.675 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-16T13:59:49.675 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-16T13:59:49.675 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-16T13:59:49.675 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-16T13:59:49.676 INFO:teuthology.orchestra.run.smithi170.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-16T13:59:49.676 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.677 DEBUG:teuthology.orchestra.run.smithi170:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-16T13:59:49.701 INFO:teuthology.orchestra.run.smithi170.stdout:postinstall scriptlet (using /bin/sh): 2023-10-16T13:59:49.701 INFO:teuthology.orchestra.run.smithi170.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-16T13:59:49.701 INFO:teuthology.orchestra.run.smithi170.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-16T13:59:49.702 INFO:teuthology.orchestra.run.smithi170.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T13:59:49.702 INFO:teuthology.orchestra.run.smithi170.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.702 INFO:teuthology.orchestra.run.smithi170.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T13:59:49.702 INFO:teuthology.orchestra.run.smithi170.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T13:59:49.702 INFO:teuthology.orchestra.run.smithi170.stdout:fi 2023-10-16T13:59:49.703 INFO:teuthology.orchestra.run.smithi170.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-16T13:59:49.703 INFO:teuthology.orchestra.run.smithi170.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-16T13:59:49.703 INFO:teuthology.orchestra.run.smithi170.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-16T13:59:49.703 INFO:teuthology.orchestra.run.smithi170.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-16T13:59:49.703 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.703 INFO:teuthology.orchestra.run.smithi170.stdout:fi 2023-10-16T13:59:49.704 INFO:teuthology.orchestra.run.smithi170.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-16T13:59:49.704 INFO:teuthology.orchestra.run.smithi170.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-16T13:59:49.704 INFO:teuthology.orchestra.run.smithi170.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.704 INFO:teuthology.orchestra.run.smithi170.stdout:fi 2023-10-16T13:59:49.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2023-10-16T13:59:49.782 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2023-10-16T13:59:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2023-10-16T13:59:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2023-10-16T13:59:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2023-10-16T13:59:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2023-10-16T13:59:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-16T13:59:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2023-10-16T13:59:49.784 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2023-10-16T13:59:49.784 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2023-10-16T13:59:49.784 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-16T13:59:49.784 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-16T13:59:49.784 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-16T13:59:49.784 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-16T13:59:49.785 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2023-10-16T13:59:49.785 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2023-10-16T13:59:49.785 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-16T13:59:49.785 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2023-10-16T13:59:49.785 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2023-10-16T13:59:49.786 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T13:59:49.786 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-16T13:59:49.786 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-16T13:59:49.786 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-16T13:59:49.786 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2023-10-16T13:59:49.787 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T13:59:49.787 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2023-10-16T13:59:49.788 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-16T13:59:49.788 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 6.6.0-rc5-g47672119149b 2023-10-16T13:59:49.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-16T13:59:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-10-16T13:59:51.845 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-16T13:59:51.894 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-10-16T13:59:51.896 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-16T13:59:51.912 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.DSULet3ptL 2023-10-16T13:59:51.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DSULet3ptL 2023-10-16T13:59:51.980 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.DSULet3ptL 2023-10-16T13:59:52.122 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.DSULet3ptL is 6KB 2023-10-16T13:59:52.171 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.DSULet3ptL 2023-10-16T13:59:52.185 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2023-10-16T13:59:52.251 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-16T13:59:52.251 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-16T13:59:52.252 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-16T13:59:52.252 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b.conf 2023-10-16T13:59:52.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b 2023-10-16T13:59:52.416 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-10-16T13:59:54.821 INFO:teuthology.orchestra.run.smithi029.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-16T14:00:18.469 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-16T14:00:18.470 DEBUG:teuthology.orchestra.run.smithi029:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-16T14:00:18.556 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.556 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.557 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-16T14:00:18.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-16T14:00:18.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-16T14:00:18.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-16T14:00:18.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-16T14:00:18.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-16T14:00:18.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-16T14:00:18.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-16T14:00:18.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-16T14:00:18.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-16T14:00:18.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-16T14:00:18.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-16T14:00:18.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-16T14:00:18.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-16T14:00:18.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-16T14:00:18.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-16T14:00:18.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-16T14:00:18.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-16T14:00:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-16T14:00:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-16T14:00:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-16T14:00:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-16T14:00:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-16T14:00:18.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-16T14:00:18.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-16T14:00:18.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-16T14:00:18.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-16T14:00:18.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-16T14:00:18.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-16T14:00:18.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-16T14:00:18.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-16T14:00:18.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-16T14:00:18.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-16T14:00:18.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-16T14:00:18.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-16T14:00:18.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-16T14:00:18.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-16T14:00:18.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-16T14:00:18.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-16T14:00:18.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-16T14:00:18.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-16T14:00:18.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-16T14:00:18.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-16T14:00:18.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-16T14:00:18.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-16T14:00:18.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-16T14:00:18.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-16T14:00:18.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-16T14:00:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-16T14:00:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-16T14:00:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-16T14:00:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-16T14:00:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-16T14:00:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-16T14:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-16T14:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-16T14:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-16T14:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-16T14:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-16T14:00:18.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-16T14:00:18.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-16T14:00:18.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-16T14:00:18.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-16T14:00:18.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-16T14:00:18.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-16T14:00:18.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-16T14:00:18.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-16T14:00:18.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-16T14:00:18.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-16T14:00:18.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-16T14:00:18.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-16T14:00:18.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-16T14:00:18.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-16T14:00:18.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-16T14:00:18.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-16T14:00:18.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-16T14:00:18.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-16T14:00:18.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-16T14:00:18.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-16T14:00:18.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-16T14:00:18.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-16T14:00:18.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-16T14:00:18.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-16T14:00:18.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-16T14:00:18.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-16T14:00:18.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-16T14:00:18.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-16T14:00:18.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-16T14:00:18.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-16T14:00:18.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-16T14:00:18.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-16T14:00:18.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-16T14:00:18.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-16T14:00:18.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-16T14:00:18.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-16T14:00:18.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-16T14:00:18.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-16T14:00:18.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-16T14:00:18.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-16T14:00:18.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-16T14:00:18.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-16T14:00:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-16T14:00:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-16T14:00:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-16T14:00:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-16T14:00:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-16T14:00:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-16T14:00:18.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-16T14:00:18.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-16T14:00:18.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-16T14:00:18.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-16T14:00:18.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-16T14:00:18.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-16T14:00:18.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-16T14:00:18.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-16T14:00:18.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-16T14:00:18.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-16T14:00:18.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-16T14:00:18.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-16T14:00:18.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-16T14:00:18.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-16T14:00:18.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-16T14:00:18.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-16T14:00:18.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-16T14:00:18.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-16T14:00:18.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-16T14:00:18.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-16T14:00:18.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-16T14:00:18.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-16T14:00:18.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-16T14:00:18.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-16T14:00:18.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-16T14:00:18.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-16T14:00:18.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-16T14:00:18.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-16T14:00:18.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-16T14:00:18.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-16T14:00:18.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-16T14:00:18.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-16T14:00:18.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-16T14:00:18.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-16T14:00:18.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-16T14:00:18.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-16T14:00:18.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-16T14:00:18.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-16T14:00:18.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-16T14:00:18.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-16T14:00:18.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-16T14:00:18.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-16T14:00:18.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-16T14:00:18.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-16T14:00:18.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-16T14:00:18.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-16T14:00:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-16T14:00:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-16T14:00:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-16T14:00:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-16T14:00:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-16T14:00:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-16T14:00:18.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-16T14:00:18.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-16T14:00:18.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-16T14:00:18.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-16T14:00:18.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-16T14:00:18.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-16T14:00:18.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-16T14:00:18.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-16T14:00:18.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-16T14:00:18.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-16T14:00:18.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-16T14:00:18.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-16T14:00:18.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-16T14:00:18.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-16T14:00:18.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-16T14:00:18.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-16T14:00:18.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-16T14:00:18.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-16T14:00:18.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-16T14:00:18.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-16T14:00:18.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-16T14:00:18.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-16T14:00:18.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-16T14:00:18.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-16T14:00:18.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-16T14:00:18.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-16T14:00:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-16T14:00:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-16T14:00:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-16T14:00:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-16T14:00:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-16T14:00:18.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-16T14:00:18.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-16T14:00:18.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-16T14:00:18.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-16T14:00:18.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-16T14:00:18.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-16T14:00:18.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-16T14:00:18.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-16T14:00:18.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-16T14:00:18.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-16T14:00:18.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-16T14:00:18.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-16T14:00:18.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-16T14:00:18.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-16T14:00:18.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-16T14:00:18.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-16T14:00:18.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-16T14:00:18.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-16T14:00:18.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-16T14:00:18.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-16T14:00:18.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-16T14:00:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-16T14:00:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-16T14:00:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-16T14:00:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-16T14:00:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-16T14:00:18.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-16T14:00:18.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-16T14:00:18.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-16T14:00:18.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-16T14:00:18.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-16T14:00:18.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-16T14:00:18.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-16T14:00:18.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-16T14:00:18.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-16T14:00:18.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-16T14:00:18.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-16T14:00:18.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-16T14:00:18.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-16T14:00:18.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-16T14:00:18.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-16T14:00:18.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-16T14:00:18.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-16T14:00:18.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-16T14:00:18.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-16T14:00:18.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-16T14:00:18.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-16T14:00:18.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-16T14:00:18.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-16T14:00:18.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-16T14:00:18.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-16T14:00:18.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-16T14:00:18.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-16T14:00:18.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-16T14:00:18.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-16T14:00:18.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-16T14:00:18.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-16T14:00:18.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-16T14:00:18.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-16T14:00:18.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-16T14:00:18.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-16T14:00:18.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-16T14:00:18.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-16T14:00:18.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-16T14:00:18.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-16T14:00:18.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-16T14:00:18.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-16T14:00:18.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-16T14:00:18.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-16T14:00:18.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-16T14:00:18.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-16T14:00:18.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-16T14:00:18.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-16T14:00:18.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-16T14:00:18.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-16T14:00:18.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-16T14:00:18.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-16T14:00:18.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-16T14:00:18.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-16T14:00:18.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-16T14:00:18.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-16T14:00:18.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-16T14:00:18.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-16T14:00:18.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-16T14:00:18.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-16T14:00:18.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-16T14:00:18.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-16T14:00:18.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-16T14:00:18.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-16T14:00:18.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-16T14:00:18.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-16T14:00:18.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-16T14:00:18.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-16T14:00:18.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-16T14:00:18.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-16T14:00:18.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-16T14:00:18.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-16T14:00:18.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-16T14:00:18.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-16T14:00:18.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-16T14:00:18.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-16T14:00:18.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-16T14:00:18.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-16T14:00:18.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-16T14:00:18.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-16T14:00:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-16T14:00:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-16T14:00:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-16T14:00:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-16T14:00:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-16T14:00:18.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-16T14:00:18.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-16T14:00:18.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-16T14:00:18.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-16T14:00:18.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-16T14:00:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-16T14:00:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-16T14:00:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-16T14:00:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-16T14:00:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-16T14:00:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-16T14:00:18.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-16T14:00:18.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-16T14:00:18.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-16T14:00:18.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-16T14:00:18.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-16T14:00:18.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-16T14:00:18.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-16T14:00:18.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-16T14:00:18.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-16T14:00:18.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-16T14:00:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-16T14:00:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-16T14:00:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-16T14:00:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-16T14:00:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-16T14:00:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-16T14:00:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-16T14:00:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-16T14:00:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-16T14:00:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-16T14:00:18.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-16T14:00:18.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-16T14:00:18.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-16T14:00:18.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-16T14:00:18.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-16T14:00:18.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-16T14:00:18.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-16T14:00:18.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-16T14:00:18.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-16T14:00:18.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-16T14:00:18.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-16T14:00:18.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-16T14:00:18.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-16T14:00:18.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-16T14:00:18.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-16T14:00:18.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-16T14:00:18.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-16T14:00:18.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-16T14:00:18.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-16T14:00:18.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-16T14:00:18.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-16T14:00:18.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-16T14:00:18.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-16T14:00:18.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-16T14:00:18.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-16T14:00:18.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-16T14:00:18.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-16T14:00:18.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-16T14:00:18.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-16T14:00:18.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-16T14:00:18.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-16T14:00:18.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-16T14:00:18.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-16T14:00:18.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-16T14:00:18.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-16T14:00:18.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-16T14:00:18.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-16T14:00:18.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-16T14:00:18.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-16T14:00:18.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-16T14:00:18.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-16T14:00:18.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-16T14:00:18.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-16T14:00:18.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-16T14:00:18.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-16T14:00:18.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-16T14:00:18.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-16T14:00:18.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-16T14:00:18.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-16T14:00:18.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-16T14:00:18.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-16T14:00:18.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-16T14:00:18.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-16T14:00:18.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-16T14:00:18.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-16T14:00:18.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-16T14:00:18.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-16T14:00:18.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-16T14:00:18.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-16T14:00:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-16T14:00:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-16T14:00:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-16T14:00:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-16T14:00:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-16T14:00:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-16T14:00:18.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-16T14:00:18.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-16T14:00:18.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-16T14:00:18.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-16T14:00:18.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-16T14:00:18.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-16T14:00:18.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-16T14:00:18.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-16T14:00:18.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-16T14:00:18.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-16T14:00:18.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-16T14:00:18.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-16T14:00:18.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-16T14:00:18.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-16T14:00:18.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-16T14:00:18.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-16T14:00:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-16T14:00:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-16T14:00:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-16T14:00:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-16T14:00:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-16T14:00:18.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-16T14:00:18.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-16T14:00:18.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-16T14:00:18.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-16T14:00:18.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-16T14:00:18.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-16T14:00:18.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-16T14:00:18.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-16T14:00:18.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-16T14:00:18.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-16T14:00:18.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-16T14:00:18.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-16T14:00:18.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-16T14:00:18.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-16T14:00:18.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-16T14:00:18.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-16T14:00:18.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-16T14:00:18.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-16T14:00:18.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-16T14:00:18.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-16T14:00:18.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-16T14:00:18.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-16T14:00:18.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-16T14:00:18.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-16T14:00:18.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-16T14:00:18.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-16T14:00:18.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-16T14:00:18.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-16T14:00:18.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-16T14:00:18.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-16T14:00:18.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-16T14:00:18.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-16T14:00:18.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-16T14:00:18.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-16T14:00:18.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-16T14:00:18.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-16T14:00:18.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-16T14:00:18.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-16T14:00:18.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-16T14:00:18.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-16T14:00:18.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-16T14:00:18.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-16T14:00:18.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-16T14:00:18.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-16T14:00:18.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-16T14:00:18.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-16T14:00:18.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-16T14:00:18.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-16T14:00:18.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-16T14:00:18.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-16T14:00:18.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-16T14:00:18.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-16T14:00:18.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-16T14:00:18.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-16T14:00:18.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-16T14:00:18.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-16T14:00:18.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-16T14:00:18.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-16T14:00:18.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-16T14:00:18.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-16T14:00:18.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-16T14:00:18.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-16T14:00:18.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-16T14:00:18.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-16T14:00:18.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-16T14:00:18.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-16T14:00:18.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-16T14:00:18.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-16T14:00:18.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-16T14:00:18.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-16T14:00:18.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-16T14:00:18.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-16T14:00:18.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-16T14:00:18.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-16T14:00:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-16T14:00:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-16T14:00:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-16T14:00:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-16T14:00:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-16T14:00:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-16T14:00:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-16T14:00:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-16T14:00:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-16T14:00:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-16T14:00:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-16T14:00:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-16T14:00:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-16T14:00:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-16T14:00:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-16T14:00:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-16T14:00:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-16T14:00:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-16T14:00:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-16T14:00:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-16T14:00:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-16T14:00:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-16T14:00:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-16T14:00:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-16T14:00:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-16T14:00:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-16T14:00:18.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-16T14:00:18.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-16T14:00:18.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-16T14:00:18.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-16T14:00:18.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-16T14:00:18.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-16T14:00:18.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-16T14:00:18.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-16T14:00:18.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-16T14:00:18.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-16T14:00:18.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-16T14:00:18.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-16T14:00:18.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-16T14:00:18.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-16T14:00:18.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-16T14:00:18.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-16T14:00:18.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-16T14:00:18.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-16T14:00:18.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-16T14:00:18.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-16T14:00:18.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-16T14:00:18.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-16T14:00:18.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-16T14:00:18.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-16T14:00:18.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-16T14:00:18.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-16T14:00:18.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-16T14:00:18.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-16T14:00:18.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-16T14:00:18.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-16T14:00:18.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-16T14:00:18.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-16T14:00:18.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-16T14:00:18.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-16T14:00:18.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-16T14:00:18.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-16T14:00:18.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-16T14:00:18.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-16T14:00:18.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-16T14:00:18.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-16T14:00:18.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-16T14:00:18.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-16T14:00:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-16T14:00:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-16T14:00:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-16T14:00:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-16T14:00:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-16T14:00:18.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-16T14:00:18.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-16T14:00:18.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-16T14:00:18.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-16T14:00:18.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-16T14:00:18.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-16T14:00:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-16T14:00:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-16T14:00:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-16T14:00:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-16T14:00:18.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-16T14:00:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-16T14:00:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-16T14:00:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-16T14:00:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-16T14:00:18.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-16T14:00:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-16T14:00:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-16T14:00:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-16T14:00:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-16T14:00:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-16T14:00:18.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-16T14:00:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-16T14:00:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-16T14:00:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-16T14:00:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-16T14:00:18.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-16T14:00:18.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-16T14:00:18.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-16T14:00:18.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-16T14:00:18.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-16T14:00:18.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-16T14:00:18.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-16T14:00:18.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-16T14:00:18.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-16T14:00:18.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-16T14:00:18.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-16T14:00:18.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-16T14:00:18.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-16T14:00:18.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-16T14:00:18.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-16T14:00:18.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-16T14:00:18.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-16T14:00:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-16T14:00:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-16T14:00:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-16T14:00:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-16T14:00:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-16T14:00:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-16T14:00:18.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-16T14:00:18.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-16T14:00:18.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-16T14:00:18.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-16T14:00:18.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-16T14:00:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-16T14:00:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-16T14:00:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-16T14:00:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-16T14:00:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-16T14:00:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-16T14:00:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-16T14:00:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-16T14:00:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-16T14:00:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-16T14:00:18.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-16T14:00:18.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-16T14:00:18.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-16T14:00:18.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-16T14:00:18.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-16T14:00:18.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-16T14:00:18.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-16T14:00:18.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-16T14:00:18.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-16T14:00:18.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-16T14:00:18.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-16T14:00:18.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-16T14:00:18.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-16T14:00:18.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-16T14:00:18.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-16T14:00:18.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-16T14:00:18.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-16T14:00:18.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-16T14:00:18.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-16T14:00:18.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-16T14:00:18.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-16T14:00:18.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-16T14:00:18.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-16T14:00:18.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-16T14:00:18.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-16T14:00:18.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-16T14:00:18.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-16T14:00:18.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-16T14:00:18.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-16T14:00:18.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-16T14:00:18.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-16T14:00:18.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-16T14:00:18.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-16T14:00:18.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-16T14:00:18.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-16T14:00:18.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-16T14:00:18.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-16T14:00:18.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-16T14:00:18.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-16T14:00:18.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-16T14:00:18.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-16T14:00:18.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-16T14:00:18.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-16T14:00:18.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-16T14:00:18.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-16T14:00:18.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-16T14:00:18.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-16T14:00:18.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-16T14:00:18.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-16T14:00:18.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-16T14:00:18.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-16T14:00:18.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-16T14:00:18.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-16T14:00:18.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-16T14:00:18.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-16T14:00:18.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-16T14:00:18.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-16T14:00:18.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-16T14:00:18.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-16T14:00:18.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-16T14:00:18.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-16T14:00:18.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-16T14:00:18.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-16T14:00:18.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-16T14:00:18.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-16T14:00:18.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-16T14:00:18.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-16T14:00:18.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-16T14:00:18.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-16T14:00:18.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-16T14:00:18.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-16T14:00:18.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-16T14:00:18.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-16T14:00:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-16T14:00:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-16T14:00:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-16T14:00:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-16T14:00:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-16T14:00:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-16T14:00:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-16T14:00:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-16T14:00:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-16T14:00:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-16T14:00:18.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-16T14:00:18.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-16T14:00:18.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-16T14:00:18.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-16T14:00:18.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-16T14:00:18.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-16T14:00:18.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-16T14:00:18.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-16T14:00:18.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-16T14:00:18.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-16T14:00:18.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-16T14:00:18.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-16T14:00:18.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-16T14:00:18.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-16T14:00:18.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-16T14:00:18.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-16T14:00:18.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-16T14:00:18.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-16T14:00:18.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-16T14:00:18.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-16T14:00:18.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-16T14:00:18.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-16T14:00:18.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-16T14:00:18.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-16T14:00:18.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-16T14:00:18.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-16T14:00:18.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-16T14:00:18.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-16T14:00:18.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-16T14:00:18.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-16T14:00:18.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-16T14:00:18.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-16T14:00:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-16T14:00:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-16T14:00:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-16T14:00:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-16T14:00:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-16T14:00:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-16T14:00:18.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-16T14:00:18.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-16T14:00:18.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-16T14:00:18.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-16T14:00:18.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-16T14:00:18.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-16T14:00:18.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-16T14:00:18.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-16T14:00:18.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-16T14:00:18.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-16T14:00:18.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-16T14:00:18.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-16T14:00:18.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-16T14:00:18.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-16T14:00:18.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-16T14:00:18.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-16T14:00:18.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-16T14:00:18.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-16T14:00:18.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-16T14:00:18.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-16T14:00:18.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-16T14:00:18.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-16T14:00:18.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-16T14:00:18.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-16T14:00:18.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-16T14:00:18.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-16T14:00:18.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-16T14:00:18.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-16T14:00:18.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-16T14:00:18.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-16T14:00:18.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-16T14:00:18.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-16T14:00:18.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-16T14:00:18.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-16T14:00:18.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-16T14:00:18.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-16T14:00:18.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-16T14:00:18.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-16T14:00:18.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-16T14:00:18.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-16T14:00:18.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-16T14:00:18.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-16T14:00:18.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-16T14:00:18.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-16T14:00:18.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-16T14:00:18.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-16T14:00:18.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-16T14:00:18.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-16T14:00:18.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-16T14:00:18.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-16T14:00:18.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-16T14:00:18.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-16T14:00:18.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-16T14:00:18.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-16T14:00:18.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-16T14:00:18.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-16T14:00:18.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-16T14:00:18.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-16T14:00:18.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-16T14:00:18.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-16T14:00:18.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-16T14:00:18.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-16T14:00:18.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-16T14:00:18.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-16T14:00:18.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-16T14:00:18.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-16T14:00:18.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-16T14:00:18.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-16T14:00:18.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-16T14:00:18.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-16T14:00:18.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-16T14:00:18.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-16T14:00:18.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-16T14:00:18.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-16T14:00:18.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-16T14:00:18.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-16T14:00:18.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-16T14:00:18.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-16T14:00:18.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-16T14:00:18.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-16T14:00:18.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-16T14:00:18.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-16T14:00:18.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-16T14:00:18.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-16T14:00:18.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-16T14:00:18.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-16T14:00:18.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-16T14:00:18.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-16T14:00:18.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-16T14:00:18.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-16T14:00:18.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-16T14:00:18.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-16T14:00:18.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-16T14:00:18.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-16T14:00:18.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-16T14:00:18.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-16T14:00:18.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-16T14:00:18.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-16T14:00:18.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-16T14:00:18.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-16T14:00:18.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-16T14:00:18.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-16T14:00:18.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-16T14:00:18.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-16T14:00:18.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-16T14:00:18.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-16T14:00:18.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-16T14:00:18.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-16T14:00:18.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-16T14:00:18.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-16T14:00:18.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-16T14:00:18.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-16T14:00:18.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-16T14:00:18.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-16T14:00:18.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-16T14:00:18.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-16T14:00:18.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-16T14:00:18.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-16T14:00:18.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-16T14:00:18.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-16T14:00:18.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-16T14:00:18.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-16T14:00:18.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-16T14:00:18.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-16T14:00:18.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-16T14:00:18.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-16T14:00:18.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-16T14:00:18.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-16T14:00:18.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-16T14:00:18.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-16T14:00:18.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-16T14:00:18.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-16T14:00:18.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-16T14:00:18.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-16T14:00:18.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-16T14:00:18.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-16T14:00:18.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-16T14:00:18.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-16T14:00:18.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-16T14:00:18.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-16T14:00:18.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-16T14:00:18.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-16T14:00:18.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-16T14:00:18.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-16T14:00:18.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-16T14:00:18.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-16T14:00:18.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-16T14:00:18.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-16T14:00:18.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-16T14:00:18.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-16T14:00:18.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-16T14:00:18.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-16T14:00:18.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-16T14:00:18.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-16T14:00:18.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-16T14:00:18.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-16T14:00:18.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-16T14:00:18.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-16T14:00:18.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-16T14:00:18.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-16T14:00:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-16T14:00:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-16T14:00:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-16T14:00:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-16T14:00:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-16T14:00:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-16T14:00:18.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-16T14:00:18.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-16T14:00:18.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-16T14:00:18.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-16T14:00:18.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-16T14:00:18.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-16T14:00:18.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-16T14:00:18.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-16T14:00:18.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-16T14:00:18.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-16T14:00:18.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-16T14:00:18.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-16T14:00:18.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-16T14:00:18.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-16T14:00:18.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-16T14:00:18.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-16T14:00:18.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-16T14:00:18.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-16T14:00:18.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-16T14:00:18.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-16T14:00:18.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-16T14:00:18.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-16T14:00:18.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-16T14:00:18.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-16T14:00:18.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-16T14:00:18.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-16T14:00:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-16T14:00:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-16T14:00:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-16T14:00:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-16T14:00:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-16T14:00:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-16T14:00:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-16T14:00:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-16T14:00:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-16T14:00:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-16T14:00:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-16T14:00:18.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-16T14:00:18.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-16T14:00:18.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-16T14:00:18.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-16T14:00:18.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-16T14:00:18.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-16T14:00:18.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-16T14:00:18.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-16T14:00:18.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-16T14:00:18.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-16T14:00:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-16T14:00:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-16T14:00:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-16T14:00:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-16T14:00:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-16T14:00:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-16T14:00:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-16T14:00:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-16T14:00:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-16T14:00:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-16T14:00:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-16T14:00:18.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-16T14:00:18.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-16T14:00:18.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-16T14:00:18.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-16T14:00:18.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-16T14:00:18.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-16T14:00:18.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-16T14:00:18.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-16T14:00:18.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-16T14:00:18.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-16T14:00:18.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-16T14:00:18.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-16T14:00:18.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-16T14:00:18.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-16T14:00:18.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-16T14:00:18.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-16T14:00:18.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-16T14:00:18.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-16T14:00:18.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-16T14:00:18.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-16T14:00:18.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-16T14:00:18.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-16T14:00:18.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-16T14:00:18.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-16T14:00:18.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-16T14:00:18.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-16T14:00:18.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-16T14:00:18.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-16T14:00:18.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-16T14:00:18.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-16T14:00:18.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-16T14:00:18.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-16T14:00:18.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-16T14:00:18.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-16T14:00:18.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-16T14:00:18.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-16T14:00:18.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-16T14:00:18.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-16T14:00:18.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-16T14:00:18.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-16T14:00:18.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-16T14:00:18.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-16T14:00:18.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-16T14:00:18.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-16T14:00:18.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-16T14:00:18.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-16T14:00:18.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-16T14:00:18.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-16T14:00:18.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-16T14:00:18.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-16T14:00:18.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-16T14:00:18.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-16T14:00:18.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-16T14:00:18.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-16T14:00:18.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-16T14:00:18.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-16T14:00:18.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-16T14:00:18.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-16T14:00:18.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-16T14:00:18.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-16T14:00:18.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-16T14:00:18.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-16T14:00:18.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-16T14:00:18.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-16T14:00:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-16T14:00:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-16T14:00:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-16T14:00:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-16T14:00:18.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-16T14:00:18.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-16T14:00:18.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-16T14:00:18.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-16T14:00:18.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-16T14:00:18.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-16T14:00:18.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-16T14:00:18.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-16T14:00:18.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-16T14:00:18.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-16T14:00:18.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-16T14:00:18.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-16T14:00:18.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-16T14:00:18.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-16T14:00:18.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-16T14:00:18.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-16T14:00:18.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-16T14:00:18.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-16T14:00:18.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-16T14:00:18.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-16T14:00:18.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-16T14:00:18.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-16T14:00:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-16T14:00:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-16T14:00:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-16T14:00:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-16T14:00:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-16T14:00:18.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-16T14:00:18.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-16T14:00:18.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-16T14:00:18.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-16T14:00:18.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-16T14:00:18.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-16T14:00:18.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-16T14:00:18.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-16T14:00:18.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-16T14:00:18.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-16T14:00:18.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-16T14:00:18.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-16T14:00:18.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-16T14:00:18.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-16T14:00:18.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-16T14:00:18.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-16T14:00:18.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-16T14:00:18.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-16T14:00:18.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-16T14:00:18.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-16T14:00:18.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-16T14:00:18.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-16T14:00:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-16T14:00:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-16T14:00:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-16T14:00:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-16T14:00:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-16T14:00:18.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-16T14:00:18.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-16T14:00:18.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-16T14:00:18.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-16T14:00:18.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-16T14:00:18.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-16T14:00:18.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-16T14:00:18.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-16T14:00:18.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-16T14:00:18.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-16T14:00:18.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-16T14:00:18.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-16T14:00:18.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-16T14:00:18.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-16T14:00:18.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-16T14:00:18.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-16T14:00:18.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-16T14:00:18.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-16T14:00:18.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-16T14:00:18.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-16T14:00:18.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-16T14:00:18.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-16T14:00:18.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-16T14:00:18.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-16T14:00:18.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-16T14:00:18.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-16T14:00:18.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-16T14:00:18.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-16T14:00:18.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-16T14:00:18.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-16T14:00:18.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-16T14:00:18.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-16T14:00:18.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-16T14:00:18.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-16T14:00:18.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-16T14:00:18.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-16T14:00:18.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-16T14:00:18.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-16T14:00:18.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-16T14:00:18.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-16T14:00:18.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-16T14:00:18.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-16T14:00:18.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-16T14:00:18.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-16T14:00:18.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-16T14:00:18.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-16T14:00:18.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-16T14:00:18.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-16T14:00:18.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-16T14:00:18.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-16T14:00:18.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-16T14:00:18.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-16T14:00:18.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-16T14:00:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-16T14:00:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-16T14:00:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-16T14:00:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-16T14:00:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-16T14:00:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-16T14:00:18.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-16T14:00:18.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-16T14:00:18.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-16T14:00:18.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-16T14:00:18.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-16T14:00:18.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-16T14:00:18.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-16T14:00:18.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-16T14:00:18.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-16T14:00:18.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-16T14:00:18.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-16T14:00:18.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-16T14:00:18.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-16T14:00:18.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-16T14:00:18.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-16T14:00:18.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-16T14:00:18.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-16T14:00:18.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-16T14:00:18.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-16T14:00:18.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-16T14:00:18.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-16T14:00:18.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-16T14:00:18.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-16T14:00:18.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-16T14:00:18.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-16T14:00:18.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-16T14:00:18.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-16T14:00:18.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-16T14:00:18.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-16T14:00:18.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-16T14:00:18.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-16T14:00:18.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-16T14:00:18.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-16T14:00:18.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-16T14:00:18.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-16T14:00:18.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-16T14:00:18.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-16T14:00:18.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-16T14:00:18.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-16T14:00:18.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-16T14:00:18.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-16T14:00:18.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-16T14:00:18.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-16T14:00:18.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-16T14:00:18.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-16T14:00:18.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-16T14:00:18.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-16T14:00:18.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-16T14:00:18.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-16T14:00:18.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-16T14:00:18.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-16T14:00:18.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-16T14:00:18.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-16T14:00:18.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-16T14:00:18.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-16T14:00:18.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-16T14:00:18.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-16T14:00:18.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-16T14:00:18.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-16T14:00:18.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-16T14:00:18.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-16T14:00:18.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-16T14:00:18.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-16T14:00:18.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-16T14:00:18.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-16T14:00:18.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-16T14:00:18.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-16T14:00:18.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-16T14:00:18.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-16T14:00:18.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-16T14:00:18.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-16T14:00:18.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-16T14:00:18.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-16T14:00:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-16T14:00:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-16T14:00:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-16T14:00:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-16T14:00:18.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-16T14:00:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-16T14:00:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-16T14:00:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-16T14:00:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-16T14:00:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-16T14:00:18.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-16T14:00:18.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-16T14:00:18.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-16T14:00:18.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-16T14:00:18.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-16T14:00:18.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-16T14:00:18.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-16T14:00:18.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-16T14:00:18.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-16T14:00:18.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-16T14:00:18.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-16T14:00:18.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-16T14:00:18.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-16T14:00:18.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-16T14:00:18.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-16T14:00:18.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-16T14:00:18.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-16T14:00:18.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-16T14:00:18.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-16T14:00:18.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-16T14:00:18.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-16T14:00:18.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-16T14:00:18.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-16T14:00:18.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-16T14:00:18.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-16T14:00:18.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-16T14:00:18.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-16T14:00:18.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-16T14:00:18.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-16T14:00:18.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-16T14:00:18.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-16T14:00:18.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-16T14:00:18.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-16T14:00:18.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-16T14:00:18.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-16T14:00:18.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-16T14:00:18.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-16T14:00:18.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-16T14:00:18.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-16T14:00:18.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-16T14:00:18.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-16T14:00:18.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-16T14:00:18.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-16T14:00:18.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-16T14:00:18.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-16T14:00:18.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-16T14:00:18.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-16T14:00:18.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-16T14:00:18.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-16T14:00:18.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-16T14:00:18.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-16T14:00:18.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-16T14:00:18.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-16T14:00:18.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-16T14:00:18.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-16T14:00:18.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-16T14:00:18.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-16T14:00:18.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-16T14:00:18.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-16T14:00:18.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-16T14:00:18.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-16T14:00:18.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-16T14:00:18.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-16T14:00:18.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-16T14:00:18.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-16T14:00:18.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-16T14:00:18.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-16T14:00:18.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-16T14:00:18.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-16T14:00:18.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-16T14:00:18.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-16T14:00:18.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-16T14:00:18.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-16T14:00:18.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-16T14:00:18.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-16T14:00:18.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-16T14:00:18.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-16T14:00:18.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-16T14:00:18.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-16T14:00:18.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-16T14:00:18.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-16T14:00:18.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-16T14:00:18.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-16T14:00:18.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-16T14:00:18.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-16T14:00:18.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-16T14:00:18.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-16T14:00:18.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-16T14:00:18.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-16T14:00:18.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-16T14:00:18.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-16T14:00:18.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-16T14:00:18.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-16T14:00:18.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-16T14:00:18.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-16T14:00:18.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-16T14:00:18.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-16T14:00:18.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-16T14:00:18.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-16T14:00:18.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-16T14:00:18.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-16T14:00:18.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-16T14:00:18.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-16T14:00:18.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-16T14:00:18.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-16T14:00:18.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-16T14:00:18.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-16T14:00:18.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-16T14:00:18.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-16T14:00:18.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-16T14:00:18.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-16T14:00:18.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-16T14:00:18.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-16T14:00:18.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-16T14:00:18.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-16T14:00:18.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-16T14:00:18.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-16T14:00:18.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-16T14:00:18.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-16T14:00:18.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-16T14:00:18.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-16T14:00:18.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-16T14:00:18.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-16T14:00:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-16T14:00:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-16T14:00:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-16T14:00:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-16T14:00:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-16T14:00:18.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-16T14:00:18.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-16T14:00:18.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-16T14:00:18.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-16T14:00:18.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-16T14:00:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-16T14:00:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-16T14:00:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-16T14:00:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-16T14:00:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-16T14:00:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-16T14:00:18.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-16T14:00:18.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-16T14:00:18.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-16T14:00:18.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-16T14:00:18.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-16T14:00:18.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-16T14:00:18.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-16T14:00:18.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-16T14:00:18.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-16T14:00:18.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-16T14:00:18.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-16T14:00:18.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-16T14:00:18.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-16T14:00:18.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-16T14:00:18.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-16T14:00:18.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-16T14:00:18.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-16T14:00:18.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-16T14:00:18.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-16T14:00:18.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-16T14:00:18.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-16T14:00:18.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-16T14:00:18.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-16T14:00:18.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-16T14:00:18.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-16T14:00:18.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-16T14:00:18.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-16T14:00:18.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-16T14:00:18.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-16T14:00:18.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-16T14:00:18.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-16T14:00:18.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-16T14:00:18.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-16T14:00:18.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-16T14:00:18.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-16T14:00:18.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-16T14:00:18.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-16T14:00:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-16T14:00:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-16T14:00:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-16T14:00:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-16T14:00:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-16T14:00:18.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-16T14:00:18.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-16T14:00:18.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-16T14:00:18.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-16T14:00:18.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-16T14:00:18.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-16T14:00:18.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-16T14:00:18.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-16T14:00:18.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-16T14:00:18.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-16T14:00:18.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-16T14:00:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-16T14:00:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-16T14:00:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-16T14:00:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-16T14:00:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-16T14:00:18.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-16T14:00:18.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-16T14:00:18.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-16T14:00:18.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-16T14:00:18.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-16T14:00:18.849 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-16T14:00:18.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-16T14:00:18.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-16T14:00:18.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-16T14:00:18.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-16T14:00:18.850 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-16T14:00:18.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-16T14:00:18.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-16T14:00:18.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-16T14:00:18.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-16T14:00:18.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-16T14:00:18.851 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-16T14:00:18.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-16T14:00:18.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-16T14:00:18.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-16T14:00:18.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-16T14:00:18.852 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-16T14:00:18.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-16T14:00:18.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-16T14:00:18.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-16T14:00:18.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-16T14:00:18.853 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-16T14:00:18.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-16T14:00:18.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-16T14:00:18.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-16T14:00:18.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-16T14:00:18.854 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-16T14:00:18.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-16T14:00:18.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-16T14:00:18.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-16T14:00:18.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-16T14:00:18.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-16T14:00:18.855 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-16T14:00:18.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-16T14:00:18.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-16T14:00:18.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-16T14:00:18.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-16T14:00:18.856 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-16T14:00:18.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-16T14:00:18.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-16T14:00:18.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-16T14:00:18.857 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-16T14:00:18.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-16T14:00:18.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-16T14:00:18.858 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-16T14:00:18.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-16T14:00:18.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-16T14:00:18.859 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-16T14:00:18.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-16T14:00:18.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-16T14:00:18.860 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-16T14:00:18.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-16T14:00:18.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-16T14:00:18.861 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-16T14:00:18.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-16T14:00:18.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-16T14:00:18.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-16T14:00:18.862 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-16T14:00:18.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-16T14:00:18.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-16T14:00:18.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-16T14:00:18.863 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-16T14:00:18.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-16T14:00:18.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-16T14:00:18.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-16T14:00:18.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-16T14:00:18.864 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-16T14:00:18.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-16T14:00:18.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-16T14:00:18.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-16T14:00:18.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-16T14:00:18.865 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-16T14:00:18.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-16T14:00:18.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-16T14:00:18.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-16T14:00:18.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-16T14:00:18.866 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-16T14:00:18.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-16T14:00:18.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-16T14:00:18.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-16T14:00:18.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-16T14:00:18.867 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-16T14:00:18.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-16T14:00:18.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-16T14:00:18.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-16T14:00:18.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-16T14:00:18.868 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-16T14:00:18.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-16T14:00:18.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-16T14:00:18.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-16T14:00:18.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-16T14:00:18.869 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-16T14:00:18.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-16T14:00:18.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-16T14:00:18.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-16T14:00:18.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-16T14:00:18.870 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-16T14:00:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-16T14:00:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-16T14:00:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-16T14:00:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-16T14:00:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-16T14:00:18.871 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-16T14:00:18.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-16T14:00:18.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-16T14:00:18.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-16T14:00:18.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-16T14:00:18.872 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-16T14:00:18.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-16T14:00:18.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-16T14:00:18.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-16T14:00:18.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-16T14:00:18.873 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-16T14:00:18.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-16T14:00:18.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-16T14:00:18.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-16T14:00:18.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-16T14:00:18.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-16T14:00:18.874 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-16T14:00:18.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-16T14:00:18.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-16T14:00:18.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-16T14:00:18.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-16T14:00:18.875 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-16T14:00:18.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-16T14:00:18.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-16T14:00:18.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-16T14:00:18.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-16T14:00:18.876 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-16T14:00:18.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-16T14:00:18.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-16T14:00:18.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-16T14:00:18.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-16T14:00:18.877 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-16T14:00:18.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-16T14:00:18.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-16T14:00:18.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-16T14:00:18.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-16T14:00:18.878 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-16T14:00:18.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-16T14:00:18.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-16T14:00:18.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-16T14:00:18.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-16T14:00:18.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-16T14:00:18.879 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-16T14:00:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-16T14:00:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-16T14:00:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-16T14:00:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-16T14:00:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-16T14:00:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-16T14:00:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-16T14:00:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-16T14:00:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-16T14:00:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-16T14:00:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-16T14:00:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-16T14:00:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-16T14:00:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-16T14:00:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-16T14:00:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-16T14:00:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-16T14:00:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-16T14:00:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-16T14:00:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-16T14:00:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-16T14:00:18.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-16T14:00:18.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-16T14:00:18.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-16T14:00:18.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-16T14:00:18.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-16T14:00:18.884 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-16T14:00:18.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-16T14:00:18.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-16T14:00:18.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-16T14:00:18.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-16T14:00:18.885 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-16T14:00:18.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-16T14:00:18.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-16T14:00:18.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-16T14:00:18.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-16T14:00:18.886 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-16T14:00:18.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-16T14:00:18.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-16T14:00:18.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-16T14:00:18.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-16T14:00:18.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-16T14:00:18.887 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-16T14:00:18.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-16T14:00:18.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-16T14:00:18.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-16T14:00:18.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-16T14:00:18.888 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-16T14:00:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-16T14:00:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-16T14:00:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-16T14:00:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-16T14:00:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-16T14:00:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-16T14:00:18.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-16T14:00:18.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-16T14:00:18.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-16T14:00:18.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-16T14:00:18.890 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-16T14:00:18.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-16T14:00:18.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-16T14:00:18.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-16T14:00:18.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-16T14:00:18.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-16T14:00:18.891 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-16T14:00:18.892 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-16T14:00:18.892 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-16T14:00:18.892 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-16T14:00:18.892 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-16T14:00:18.892 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-16T14:00:18.892 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-16T14:00:18.893 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-16T14:00:18.893 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-16T14:00:18.893 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-16T14:00:18.893 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-16T14:00:18.893 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-16T14:00:18.894 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-16T14:00:18.894 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-16T14:00:18.894 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-16T14:00:18.894 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-16T14:00:18.894 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-16T14:00:18.895 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-16T14:00:18.895 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-16T14:00:18.895 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-16T14:00:18.895 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-16T14:00:18.895 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-16T14:00:18.896 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-16T14:00:18.896 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-16T14:00:18.896 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-16T14:00:18.896 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-16T14:00:18.896 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-16T14:00:18.897 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-16T14:00:18.897 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-16T14:00:18.897 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-16T14:00:18.897 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-16T14:00:18.897 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-16T14:00:18.897 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-16T14:00:18.898 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-16T14:00:18.898 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-16T14:00:18.898 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-16T14:00:18.898 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-16T14:00:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-16T14:00:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-16T14:00:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-16T14:00:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-16T14:00:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-16T14:00:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-16T14:00:18.900 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-16T14:00:18.900 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-16T14:00:18.900 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-16T14:00:18.900 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-16T14:00:18.900 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-16T14:00:18.901 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-16T14:00:18.901 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-16T14:00:18.901 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-16T14:00:18.901 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-16T14:00:18.901 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-16T14:00:18.902 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-16T14:00:18.902 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-16T14:00:18.902 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-16T14:00:18.902 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-16T14:00:18.902 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-16T14:00:18.902 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-16T14:00:18.903 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-16T14:00:18.903 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-16T14:00:18.903 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-16T14:00:18.903 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-16T14:00:18.903 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-16T14:00:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-16T14:00:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-16T14:00:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-16T14:00:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-16T14:00:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-16T14:00:18.904 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-16T14:00:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-16T14:00:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-16T14:00:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-16T14:00:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-16T14:00:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-16T14:00:18.906 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-16T14:00:18.906 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-16T14:00:18.906 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-16T14:00:18.906 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-16T14:00:18.906 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-16T14:00:18.907 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-16T14:00:18.907 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-16T14:00:18.907 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-16T14:00:18.907 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-16T14:00:18.907 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-16T14:00:18.908 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-16T14:00:18.908 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-16T14:00:18.908 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-16T14:00:18.908 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-16T14:00:18.908 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-16T14:00:18.908 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-16T14:00:18.909 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-16T14:00:18.909 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-16T14:00:18.909 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-16T14:00:18.909 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-16T14:00:18.909 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-16T14:00:18.910 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-16T14:00:18.910 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-16T14:00:18.910 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-16T14:00:18.910 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-16T14:00:18.910 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-16T14:00:18.911 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-16T14:00:18.911 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-16T14:00:18.911 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-16T14:00:18.911 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-16T14:00:18.911 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-16T14:00:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-16T14:00:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-16T14:00:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-16T14:00:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-16T14:00:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-16T14:00:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-16T14:00:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-16T14:00:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-16T14:00:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-16T14:00:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-16T14:00:18.913 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-16T14:00:18.914 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-16T14:00:18.914 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-16T14:00:18.914 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-16T14:00:18.914 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-16T14:00:18.914 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-16T14:00:18.915 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-16T14:00:18.915 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-16T14:00:18.915 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-16T14:00:18.915 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-16T14:00:18.915 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-16T14:00:18.915 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-16T14:00:18.916 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-16T14:00:18.916 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-16T14:00:18.916 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-16T14:00:18.916 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-16T14:00:18.916 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-16T14:00:18.916 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-16T14:00:18.917 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-16T14:00:18.917 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-16T14:00:18.917 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-16T14:00:18.917 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-16T14:00:18.917 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-16T14:00:18.918 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-16T14:00:18.918 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-16T14:00:18.918 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-16T14:00:18.918 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-16T14:00:18.918 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-16T14:00:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-16T14:00:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-16T14:00:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-16T14:00:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-16T14:00:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-16T14:00:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-16T14:00:18.920 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-16T14:00:18.920 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-16T14:00:18.920 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-16T14:00:18.920 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-16T14:00:18.920 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-16T14:00:18.921 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-16T14:00:18.921 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-16T14:00:18.921 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-16T14:00:18.921 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-16T14:00:18.921 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-16T14:00:18.921 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-16T14:00:18.922 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-16T14:00:18.922 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-16T14:00:18.922 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-16T14:00:18.922 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-16T14:00:18.922 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-16T14:00:18.923 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-16T14:00:18.923 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-16T14:00:18.923 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-16T14:00:18.923 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-16T14:00:18.923 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-16T14:00:18.923 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-16T14:00:18.924 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-16T14:00:18.924 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-16T14:00:18.924 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-16T14:00:18.924 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-16T14:00:18.924 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-16T14:00:18.925 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-16T14:00:18.925 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-16T14:00:18.925 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-16T14:00:18.925 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-16T14:00:18.925 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-16T14:00:18.926 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-16T14:00:18.926 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-16T14:00:18.926 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-16T14:00:18.926 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-16T14:00:18.926 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-16T14:00:18.926 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-16T14:00:18.927 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-16T14:00:18.927 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-16T14:00:18.927 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-16T14:00:18.927 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-16T14:00:18.927 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-16T14:00:18.928 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-16T14:00:18.928 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-16T14:00:18.928 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-16T14:00:18.928 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-16T14:00:18.928 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-16T14:00:18.929 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-16T14:00:18.929 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-16T14:00:18.929 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-16T14:00:18.929 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-16T14:00:18.929 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-16T14:00:18.930 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-16T14:00:18.930 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-16T14:00:18.930 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-16T14:00:18.930 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-16T14:00:18.930 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-16T14:00:18.930 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-16T14:00:18.931 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-16T14:00:18.931 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-16T14:00:18.931 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-16T14:00:18.931 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-16T14:00:18.931 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-16T14:00:18.932 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-16T14:00:18.932 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-16T14:00:18.932 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-16T14:00:18.932 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-16T14:00:18.933 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-16T14:00:18.933 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-16T14:00:18.933 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-16T14:00:18.933 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-16T14:00:18.933 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-16T14:00:18.934 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-16T14:00:18.934 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-16T14:00:18.934 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-16T14:00:18.934 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-16T14:00:18.934 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-16T14:00:18.935 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-16T14:00:18.935 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-16T14:00:18.935 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-16T14:00:18.935 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-16T14:00:18.935 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-16T14:00:18.936 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-16T14:00:18.936 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-16T14:00:18.936 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-16T14:00:18.936 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-16T14:00:18.936 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-16T14:00:18.937 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-16T14:00:18.937 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-16T14:00:18.937 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-16T14:00:18.937 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-16T14:00:18.937 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-16T14:00:18.938 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-16T14:00:18.938 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-16T14:00:18.938 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-16T14:00:18.938 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-16T14:00:18.938 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-16T14:00:18.938 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-16T14:00:18.939 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-16T14:00:18.939 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-16T14:00:18.939 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-16T14:00:18.939 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-16T14:00:18.939 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-16T14:00:18.940 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-16T14:00:18.940 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-16T14:00:18.940 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-16T14:00:18.940 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-16T14:00:18.940 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-16T14:00:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-16T14:00:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-16T14:00:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-16T14:00:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-16T14:00:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-16T14:00:18.941 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-16T14:00:18.942 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-16T14:00:18.942 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-16T14:00:18.942 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-16T14:00:18.942 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-16T14:00:18.942 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-16T14:00:18.943 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-16T14:00:18.943 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-16T14:00:18.943 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-16T14:00:18.943 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-16T14:00:18.943 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-16T14:00:18.944 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-16T14:00:18.944 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-16T14:00:18.944 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-16T14:00:18.944 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-16T14:00:18.944 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-16T14:00:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-16T14:00:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-16T14:00:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-16T14:00:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-16T14:00:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-16T14:00:18.946 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-16T14:00:18.946 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-16T14:00:18.946 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-16T14:00:18.946 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-16T14:00:18.946 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-16T14:00:18.946 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-16T14:00:18.947 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-16T14:00:18.947 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-16T14:00:18.947 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-16T14:00:18.947 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-16T14:00:18.947 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-16T14:00:18.948 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-16T14:00:18.948 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-16T14:00:18.949 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.949 DEBUG:teuthology.orchestra.run.smithi029:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-16T14:00:18.975 INFO:teuthology.orchestra.run.smithi029.stdout:postinstall scriptlet (using /bin/sh): 2023-10-16T14:00:18.975 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-16T14:00:18.975 INFO:teuthology.orchestra.run.smithi029.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-16T14:00:18.976 INFO:teuthology.orchestra.run.smithi029.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T14:00:18.976 INFO:teuthology.orchestra.run.smithi029.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.976 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T14:00:18.976 INFO:teuthology.orchestra.run.smithi029.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T14:00:18.976 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-16T14:00:18.977 INFO:teuthology.orchestra.run.smithi029.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-16T14:00:18.977 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-16T14:00:18.977 INFO:teuthology.orchestra.run.smithi029.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-16T14:00:18.977 INFO:teuthology.orchestra.run.smithi029.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-16T14:00:18.978 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.978 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-16T14:00:18.978 INFO:teuthology.orchestra.run.smithi029.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-16T14:00:18.978 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-16T14:00:18.978 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-16T14:00:18.979 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-16T14:00:18.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-10-16T14:00:19.062 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-10-16T14:00:19.062 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-10-16T14:00:19.062 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2023-10-16T14:00:19.063 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2023-10-16T14:00:19.063 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-10-16T14:00:19.063 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-16T14:00:19.063 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2023-10-16T14:00:19.063 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2023-10-16T14:00:19.064 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-10-16T14:00:19.064 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-16T14:00:19.064 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-16T14:00:19.064 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-16T14:00:19.065 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-16T14:00:19.065 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2023-10-16T14:00:19.065 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-10-16T14:00:19.065 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-16T14:00:19.065 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-10-16T14:00:19.066 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-10-16T14:00:19.066 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:00:19.066 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-16T14:00:19.066 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-16T14:00:19.066 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-16T14:00:19.067 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-10-16T14:00:19.067 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:00:19.067 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-10-16T14:00:19.068 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-16T14:00:19.068 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 6.6.0-rc5-g47672119149b 2023-10-16T14:00:19.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-16T14:00:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-10-16T14:00:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-16T14:00:21.425 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-10-16T14:00:21.427 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-10-16T14:00:21.442 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.uM0jGiEcce 2023-10-16T14:00:21.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uM0jGiEcce 2023-10-16T14:00:21.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.uM0jGiEcce 2023-10-16T14:00:21.689 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.uM0jGiEcce is 6KB 2023-10-16T14:00:21.709 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.uM0jGiEcce 2023-10-16T14:00:21.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-10-16T14:00:21.798 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-16T14:00:21.799 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-16T14:00:21.799 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-16T14:00:21.799 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b.conf 2023-10-16T14:00:21.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b 2023-10-16T14:00:21.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-10-16T14:00:22.421 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-16T14:00:22.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:00:22.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:00:40.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-10-16T14:00:49.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:00:49.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:00:51.962 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-16T14:00:51.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-16T14:00:51.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:01:08.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-10-16T14:01:10.522 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2023-10-16T14:01:19.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-16T14:01:19.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:01:20.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:01:20.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:01:23.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-10-16T14:01:38.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:01:38.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:01:38.631 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-16T14:01:39.252 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:01:39.252 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.6.0-rc5-g47672119149b"... 2023-10-16T14:01:39.253 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-16T14:01:39.292 INFO:teuthology.orchestra.run.smithi170.stdout:6.6.0-rc5-g47672119149b 2023-10-16T14:01:39.293 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-16T14:01:39.293 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T14:01:39.293 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-16T14:01:40.295 DEBUG:teuthology.parallel:result is None 2023-10-16T14:02:19.533 DEBUG:teuthology.orchestra.remote:timed out 2023-10-16T14:02:31.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-16T14:02:31.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:02:32.148 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-10-16T14:02:33.676 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-16T14:02:33.676 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g47672119149b"... 2023-10-16T14:02:33.677 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-10-16T14:02:33.736 INFO:teuthology.orchestra.run.smithi029.stdout:6.6.0-rc5-g47672119149b 2023-10-16T14:02:33.737 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-16T14:02:33.737 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T14:02:33.737 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-16T14:02:34.738 DEBUG:teuthology.parallel:result is None 2023-10-16T14:02:34.739 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-16T14:02:34.749 INFO:teuthology.task.internal:Creating test directory... 2023-10-16T14:02:34.749 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T14:02:34.758 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T14:02:34.820 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-16T14:02:34.829 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-16T14:02:34.836 INFO:teuthology.task.internal:Creating archive directory... 2023-10-16T14:02:34.836 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T14:02:34.845 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T14:02:34.918 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-16T14:02:34.926 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-16T14:02:34.927 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T14:02:34.935 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T14:02:35.083 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T14:02:35.136 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T14:02:35.162 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T14:02:35.198 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T14:02:35.203 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-16T14:02:35.210 INFO:teuthology.task.internal:Configuring sudo... 2023-10-16T14:02:35.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T14:02:35.220 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T14:02:35.324 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-16T14:02:35.333 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-16T14:02:35.334 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T14:02:35.343 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T14:02:35.400 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T14:02:35.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T14:02:35.606 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T14:02:35.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T14:02:35.802 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:02:35.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T14:02:35.910 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T14:02:35.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T14:02:36.104 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T14:02:36.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T14:02:36.289 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:02:36.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T14:02:36.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-10-16T14:02:36.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-10-16T14:02:36.527 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T14:02:36.567 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T14:02:37.004 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-16T14:02:37.012 INFO:teuthology.task.internal:Starting timer... 2023-10-16T14:02:37.013 INFO:teuthology.run_tasks:Running task pcp... 2023-10-16T14:02:37.043 INFO:teuthology.run_tasks:Running task selinux... 2023-10-16T14:02:37.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-10-16T14:02:37.251 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-10-16T14:02:37.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2023-10-16T14:02:37.418 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2023-10-16T14:02:37.428 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-16T14:02:37.428 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-10-16T14:02:37.492 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-10-16T14:02:37.492 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2023-10-16T14:02:37.557 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2023-10-16T14:02:37.558 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2023-10-16T14:02:37.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T14:02:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:02:37.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T14:02:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:02:37.770 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-16T14:02:37.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-10-16T14:02:37.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2023-10-16T14:02:37.996 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-16T14:02:38.005 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-16T14:02:38.011 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-16T14:02:38.131 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-16T14:02:38.147 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-16T14:02:38.149 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-16T14:13:01.028 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2023-10-16T14:13:01.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-16T14:13:01.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:13:01.173 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-10-16T14:13:01.335 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-16T14:13:01.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:13:01.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T14:13:01.476 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-16T14:13:01.628 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-16T14:13:01.629 INFO:teuthology.run_tasks:Running task clock... 2023-10-16T14:13:01.639 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-16T14:13:01.639 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T14:13:01.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T14:13:01.649 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T14:13:01.649 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T14:13:01.750 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T14:13:01.754 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T14:13:01.807 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T14:13:01.807 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T14:13:01.900 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2023-10-16T14:13:01.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-10-16T14:13:01.941 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2023-10-16T14:13:01.954 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-10-16T14:13:01.994 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T14:13:02.014 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T14:13:02.045 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T14:13:02.071 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T14:13:02.172 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2023-10-16T14:13:02.180 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T14:13:02.181 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-10-16T14:13:02.181 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.181 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.181 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.181 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.202 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-10-16T14:13:02.212 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T14:13:02.212 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-10-16T14:13:02.212 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.212 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.213 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.213 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:13:02.214 INFO:teuthology.run_tasks:Running task install... 2023-10-16T14:13:02.222 DEBUG:teuthology.task.install:project ceph 2023-10-16T14:13:02.223 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}} 2023-10-16T14:13:02.223 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'} 2023-10-16T14:13:02.223 INFO:teuthology.task.install:Using flavor: default 2023-10-16T14:13:02.229 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-16T14:13:02.229 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-16T14:13:02.230 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'tag': None, 'wait_for_package': False} 2023-10-16T14:13:02.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T14:13:02.232 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'tag': None, 'wait_for_package': False} 2023-10-16T14:13:02.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T14:13:02.377 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/ 2023-10-16T14:13:02.378 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6730.g4ba8a190 2023-10-16T14:13:02.471 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/ 2023-10-16T14:13:02.471 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6730.g4ba8a190 2023-10-16T14:13:02.579 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T14:13:02.580 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:13:02.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T14:13:02.601 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T14:13:02.601 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:13:02.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T14:13:02.684 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T14:13:02.685 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T14:13:02.710 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T14:13:02.710 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T14:13:02.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T14:13:02.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T14:13:02.961 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T14:13:03.005 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T14:13:03.054 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2023-10-16T14:13:03.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2023-10-16T14:13:03.108 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-10-16T14:13:03.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-10-16T14:13:03.404 INFO:teuthology.orchestra.run.smithi170.stdout:72 files removed 2023-10-16T14:13:03.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T14:13:03.481 INFO:teuthology.orchestra.run.smithi029.stdout:72 files removed 2023-10-16T14:13:03.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T14:13:04.426 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 163 kB/s | 78 kB 00:00 2023-10-16T14:13:04.650 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 128 kB/s | 78 kB 00:00 2023-10-16T14:13:04.885 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-10-16T14:13:05.302 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-10-16T14:13:05.527 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2023-10-16T14:13:06.095 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2023-10-16T14:13:06.639 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.3 MB 00:01 2023-10-16T14:13:07.058 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.3 MB 00:00 2023-10-16T14:13:08.491 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2023-10-16T14:13:08.962 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2023-10-16T14:13:10.259 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 7.9 MB 00:00 2023-10-16T14:13:10.657 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 7.9 MB 00:01 2023-10-16T14:13:12.561 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 16 MB/s | 18 MB 00:01 2023-10-16T14:13:17.982 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.7 MB 00:01 2023-10-16T14:13:19.692 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 14 kB 00:00 2023-10-16T14:13:21.927 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 19 MB 00:02 2023-10-16T14:13:27.582 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 9.0 kB/s | 1.7 kB 00:00 2023-10-16T14:13:30.328 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T14:13:30.329 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T14:13:30.353 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-16T14:13:30.354 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-16T14:13:30.356 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-16T14:13:30.357 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-16T14:13:30.360 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-16T14:13:30.364 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-16T14:13:30.365 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-16T14:13:30.451 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-10-16T14:13:30.475 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-10-16T14:13:30.475 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-10-16T14:13:30.476 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-10-16T14:13:30.476 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-10-16T14:13:30.476 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-16T14:13:30.476 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 6.5 k 2023-10-16T14:13:30.476 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 5.2 M 2023-10-16T14:13:30.477 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 855 k 2023-10-16T14:13:30.477 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 141 k 2023-10-16T14:13:30.477 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 1.5 M 2023-10-16T14:13:30.477 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 132 k 2023-10-16T14:13:30.477 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 1.8 M 2023-10-16T14:13:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 7.4 M 2023-10-16T14:13:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 47 k 2023-10-16T14:13:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 8.6 M 2023-10-16T14:13:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 41 M 2023-10-16T14:13:30.478 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 262 k 2023-10-16T14:13:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 269 k 2023-10-16T14:13:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 218 k 2023-10-16T14:13:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 18 k 2023-10-16T14:13:30.479 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-16T14:13:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-16T14:13:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-16T14:13:30.480 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 31 k 2023-10-16T14:13:30.481 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 715 k 2023-10-16T14:13:30.481 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 127 k 2023-10-16T14:13:30.481 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-16T14:13:30.481 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-16T14:13:30.482 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-16T14:13:30.482 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-16T14:13:30.482 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-16T14:13:30.482 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-16T14:13:30.483 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 162 k 2023-10-16T14:13:30.483 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 321 k 2023-10-16T14:13:30.483 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 298 k 2023-10-16T14:13:30.483 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 99 k 2023-10-16T14:13:30.483 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 85 k 2023-10-16T14:13:30.483 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 3.0 M 2023-10-16T14:13:30.484 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 169 k 2023-10-16T14:13:30.484 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-16T14:13:30.484 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-10-16T14:13:30.484 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-16T14:13:30.485 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-16T14:13:30.485 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-16T14:13:30.485 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-16T14:13:30.486 INFO:teuthology.orchestra.run.smithi029.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-16T14:13:30.486 INFO:teuthology.orchestra.run.smithi029.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-16T14:13:30.486 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-16T14:13:30.486 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 3.3 M 2023-10-16T14:13:30.487 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 3.0 M 2023-10-16T14:13:30.487 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-16T14:13:30.487 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-10-16T14:13:30.487 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-16T14:13:30.487 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-16T14:13:30.488 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 19 M 2023-10-16T14:13:30.488 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 23 k 2023-10-16T14:13:30.488 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 2.1 M 2023-10-16T14:13:30.489 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 242 k 2023-10-16T14:13:30.489 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 4.4 M 2023-10-16T14:13:30.489 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 18 M 2023-10-16T14:13:30.489 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 15 k 2023-10-16T14:13:30.490 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 24 k 2023-10-16T14:13:30.490 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-16T14:13:30.490 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-16T14:13:30.490 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-16T14:13:30.491 INFO:teuthology.orchestra.run.smithi029.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-16T14:13:30.491 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-16T14:13:30.491 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-16T14:13:30.492 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-16T14:13:30.492 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-16T14:13:30.492 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-16T14:13:30.492 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 163 k 2023-10-16T14:13:30.493 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-16T14:13:30.493 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-16T14:13:30.493 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-16T14:13:30.493 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-16T14:13:30.493 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-16T14:13:30.494 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 472 k 2023-10-16T14:13:30.494 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-16T14:13:30.494 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 4.8 M 2023-10-16T14:13:30.494 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-16T14:13:30.495 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-16T14:13:30.495 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-16T14:13:30.495 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-16T14:13:30.495 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-16T14:13:30.496 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-16T14:13:30.496 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-16T14:13:30.496 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-16T14:13:30.496 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-16T14:13:30.496 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-16T14:13:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-16T14:13:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-16T14:13:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 45 k 2023-10-16T14:13:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 127 k 2023-10-16T14:13:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-16T14:13:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-16T14:13:30.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-16T14:13:30.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-16T14:13:30.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-16T14:13:30.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-16T14:13:30.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-16T14:13:30.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-16T14:13:30.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-16T14:13:30.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-16T14:13:30.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-16T14:13:30.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-16T14:13:30.500 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-16T14:13:30.500 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-16T14:13:30.500 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-16T14:13:30.500 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-16T14:13:30.500 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-16T14:13:30.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-16T14:13:30.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-16T14:13:30.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-16T14:13:30.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-16T14:13:30.502 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-16T14:13:30.502 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-16T14:13:30.502 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-16T14:13:30.503 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-16T14:13:30.503 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-16T14:13:30.503 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-16T14:13:30.503 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-16T14:13:30.504 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-16T14:13:30.504 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-16T14:13:30.504 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-16T14:13:30.504 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-16T14:13:30.504 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-16T14:13:30.505 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-16T14:13:30.505 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-16T14:13:30.505 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-16T14:13:30.505 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-16T14:13:30.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-16T14:13:30.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-16T14:13:30.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-16T14:13:30.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-16T14:13:30.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-16T14:13:30.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-16T14:13:30.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-16T14:13:30.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-16T14:13:30.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-16T14:13:30.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-16T14:13:30.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-16T14:13:30.510 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-16T14:13:30.510 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-16T14:13:30.511 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-16T14:13:30.511 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-16T14:13:30.511 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-16T14:13:30.512 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-16T14:13:30.512 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2023-10-16T14:13:30.512 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-16T14:13:30.513 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:13:30.513 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-10-16T14:13:30.514 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-10-16T14:13:30.514 INFO:teuthology.orchestra.run.smithi029.stdout:Install 133 Packages 2023-10-16T14:13:30.514 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 10 Packages 2023-10-16T14:13:30.515 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:13:30.516 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 199 M 2023-10-16T14:13:30.516 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-10-16T14:13:31.135 INFO:teuthology.orchestra.run.smithi029.stdout:(1/143): ceph-18.0.0-6730.g4ba8a190.el9.x86_64. 22 kB/s | 6.5 kB 00:00 2023-10-16T14:13:31.394 INFO:teuthology.orchestra.run.smithi029.stdout:(2/143): ceph-fuse-18.0.0-6730.g4ba8a190.el9.x8 3.2 MB/s | 855 kB 00:00 2023-10-16T14:13:31.494 INFO:teuthology.orchestra.run.smithi029.stdout:(3/143): ceph-immutable-object-cache-18.0.0-673 1.4 MB/s | 141 kB 00:00 2023-10-16T14:13:31.701 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 916 kB/s | 18 MB 00:19 2023-10-16T14:13:31.953 INFO:teuthology.orchestra.run.smithi029.stdout:(4/143): ceph-mds-18.0.0-6730.g4ba8a190.el9.x86 4.5 MB/s | 2.1 MB 00:00 2023-10-16T14:13:32.362 INFO:teuthology.orchestra.run.smithi029.stdout:(5/143): ceph-common-18.0.0-6730.g4ba8a190.el9. 12 MB/s | 19 MB 00:01 2023-10-16T14:13:32.454 INFO:teuthology.orchestra.run.smithi029.stdout:(6/143): ceph-mgr-18.0.0-6730.g4ba8a190.el9.x86 2.9 MB/s | 1.5 MB 00:00 2023-10-16T14:13:32.671 INFO:teuthology.orchestra.run.smithi029.stdout:(7/143): ceph-mon-18.0.0-6730.g4ba8a190.el9.x86 14 MB/s | 4.4 MB 00:00 2023-10-16T14:13:33.305 INFO:teuthology.orchestra.run.smithi029.stdout:(8/143): ceph-radosgw-18.0.0-6730.g4ba8a190.el9 14 MB/s | 8.6 MB 00:00 2023-10-16T14:13:33.422 INFO:teuthology.orchestra.run.smithi029.stdout:(9/143): ceph-selinux-18.0.0-6730.g4ba8a190.el9 208 kB/s | 24 kB 00:00 2023-10-16T14:13:33.582 INFO:teuthology.orchestra.run.smithi029.stdout:(10/143): ceph-base-18.0.0-6730.g4ba8a190.el9.x 1.9 MB/s | 5.2 MB 00:02 2023-10-16T14:13:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:(11/143): cephfs-mirror-18.0.0-6730.g4ba8a190.e 1.4 MB/s | 218 kB 00:00 2023-10-16T14:13:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:(12/143): libcephfs-devel-18.0.0-6730.g4ba8a190 198 kB/s | 31 kB 00:00 2023-10-16T14:13:34.201 INFO:teuthology.orchestra.run.smithi029.stdout:(13/143): libcephfs2-18.0.0-6730.g4ba8a190.el9. 2.3 MB/s | 715 kB 00:00 2023-10-16T14:13:34.326 INFO:teuthology.orchestra.run.smithi029.stdout:(14/143): libcephsqlite-18.0.0-6730.g4ba8a190.e 1.3 MB/s | 163 kB 00:00 2023-10-16T14:13:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:(15/143): librados-devel-18.0.0-6730.g4ba8a190. 1.1 MB/s | 127 kB 00:00 2023-10-16T14:13:34.661 INFO:teuthology.orchestra.run.smithi029.stdout:(16/143): libradosstriper1-18.0.0-6730.g4ba8a19 2.1 MB/s | 472 kB 00:00 2023-10-16T14:13:35.695 INFO:teuthology.orchestra.run.smithi029.stdout:(17/143): ceph-osd-18.0.0-6730.g4ba8a190.el9.x8 5.5 MB/s | 18 MB 00:03 2023-10-16T14:13:35.795 INFO:teuthology.orchestra.run.smithi029.stdout:(18/143): python3-ceph-argparse-18.0.0-6730.g4b 456 kB/s | 45 kB 00:00 2023-10-16T14:13:36.521 INFO:teuthology.orchestra.run.smithi029.stdout:(19/143): ceph-test-18.0.0-6730.g4ba8a190.el9.x 13 MB/s | 41 MB 00:03 2023-10-16T14:13:36.554 INFO:teuthology.orchestra.run.smithi029.stdout:(20/143): python3-ceph-common-18.0.0-6730.g4ba8 167 kB/s | 127 kB 00:00 2023-10-16T14:13:36.597 INFO:teuthology.orchestra.run.smithi029.stdout:(21/143): python3-cephfs-18.0.0-6730.g4ba8a190. 2.1 MB/s | 162 kB 00:00 2023-10-16T14:13:36.730 INFO:teuthology.orchestra.run.smithi029.stdout:(22/143): librgw2-18.0.0-6730.g4ba8a190.el9.x86 2.3 MB/s | 4.8 MB 00:02 2023-10-16T14:13:36.773 INFO:teuthology.orchestra.run.smithi029.stdout:(23/143): python3-rados-18.0.0-6730.g4ba8a190.e 1.4 MB/s | 321 kB 00:00 2023-10-16T14:13:36.806 INFO:teuthology.orchestra.run.smithi029.stdout:(24/143): python3-rbd-18.0.0-6730.g4ba8a190.el9 1.4 MB/s | 298 kB 00:00 2023-10-16T14:13:36.832 INFO:teuthology.orchestra.run.smithi029.stdout:(25/143): rbd-fuse-18.0.0-6730.g4ba8a190.el9.x8 1.4 MB/s | 85 kB 00:00 2023-10-16T14:13:36.876 INFO:teuthology.orchestra.run.smithi029.stdout:(26/143): python3-rgw-18.0.0-6730.g4ba8a190.el9 690 kB/s | 99 kB 00:00 2023-10-16T14:13:36.910 INFO:teuthology.orchestra.run.smithi029.stdout:(27/143): rbd-nbd-18.0.0-6730.g4ba8a190.el9.x86 2.2 MB/s | 169 kB 00:00 2023-10-16T14:13:36.960 INFO:teuthology.orchestra.run.smithi029.stdout:(28/143): ceph-grafana-dashboards-18.0.0-6730.g 279 kB/s | 23 kB 00:00 2023-10-16T14:13:36.994 INFO:teuthology.orchestra.run.smithi029.stdout:(29/143): ceph-mgr-cephadm-18.0.0-6730.g4ba8a19 1.6 MB/s | 132 kB 00:00 2023-10-16T14:13:37.111 INFO:teuthology.orchestra.run.smithi029.stdout:(30/143): rbd-mirror-18.0.0-6730.g4ba8a190.el9. 10 MB/s | 3.0 MB 00:00 2023-10-16T14:13:37.178 INFO:teuthology.orchestra.run.smithi029.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6730.g4b 3.6 MB/s | 242 kB 00:00 2023-10-16T14:13:37.246 INFO:teuthology.orchestra.run.smithi029.stdout:(32/143): ceph-mgr-rook-18.0.0-6730.g4ba8a190.e 719 kB/s | 47 kB 00:00 2023-10-16T14:13:37.321 INFO:teuthology.orchestra.run.smithi029.stdout:(33/143): ceph-mgr-dashboard-18.0.0-6730.g4ba8a 4.9 MB/s | 1.8 MB 00:00 2023-10-16T14:13:37.355 INFO:teuthology.orchestra.run.smithi029.stdout:(34/143): ceph-prometheus-alerts-18.0.0-6730.g4 140 kB/s | 15 kB 00:00 2023-10-16T14:13:37.439 INFO:teuthology.orchestra.run.smithi029.stdout:(35/143): ceph-volume-18.0.0-6730.g4ba8a190.el9 2.2 MB/s | 262 kB 00:00 2023-10-16T14:13:37.481 INFO:teuthology.orchestra.run.smithi029.stdout:(36/143): cephadm-18.0.0-6730.g4ba8a190.el9.noa 2.1 MB/s | 269 kB 00:00 2023-10-16T14:13:37.515 INFO:teuthology.orchestra.run.smithi029.stdout:(37/143): cephfs-top-18.0.0-6730.g4ba8a190.el9. 243 kB/s | 18 kB 00:00 2023-10-16T14:13:37.526 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.7 MB 00:02 2023-10-16T14:13:37.720 INFO:teuthology.orchestra.run.smithi029.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 518 kB/s | 106 kB 00:00 2023-10-16T14:13:37.779 INFO:teuthology.orchestra.run.smithi029.stdout:(39/143): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 920 kB 00:00 2023-10-16T14:13:37.813 INFO:teuthology.orchestra.run.smithi029.stdout:(40/143): flex-2.6.4-9.el9.x86_64.rpm 3.4 MB/s | 307 kB 00:00 2023-10-16T14:13:37.847 INFO:teuthology.orchestra.run.smithi029.stdout:(41/143): flexiblas-3.0.4-8.el9.x86_64.rpm 496 kB/s | 32 kB 00:00 2023-10-16T14:13:37.964 INFO:teuthology.orchestra.run.smithi029.stdout:(42/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2023-10-16T14:13:37.999 INFO:teuthology.orchestra.run.smithi029.stdout:(43/143): flexiblas-openblas-openmp-3.0.4-8.el9 126 kB/s | 17 kB 00:00 2023-10-16T14:13:38.026 INFO:teuthology.orchestra.run.smithi029.stdout:(44/143): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 10 kB 00:00 2023-10-16T14:13:38.059 INFO:teuthology.orchestra.run.smithi029.stdout:(45/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2023-10-16T14:13:38.110 INFO:teuthology.orchestra.run.smithi029.stdout:(46/143): librdkafka-1.6.1-102.el9.x86_64.rpm 7.9 MB/s | 661 kB 00:00 2023-10-16T14:13:38.152 INFO:teuthology.orchestra.run.smithi029.stdout:(47/143): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2023-10-16T14:13:38.178 INFO:teuthology.orchestra.run.smithi029.stdout:(48/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 755 kB/s | 46 kB 00:00 2023-10-16T14:13:38.212 INFO:teuthology.orchestra.run.smithi029.stdout:(49/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-16T14:13:38.246 INFO:teuthology.orchestra.run.smithi029.stdout:(50/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-10-16T14:13:38.271 INFO:teuthology.orchestra.run.smithi029.stdout:(51/143): openblas-0.3.21-2.el9.x86_64.rpm 609 kB/s | 34 kB 00:00 2023-10-16T14:13:38.430 INFO:teuthology.orchestra.run.smithi029.stdout:(52/143): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2023-10-16T14:13:38.625 INFO:teuthology.orchestra.run.smithi029.stdout:(53/143): postgresql-13.11-1.el9.x86_64.rpm 4.5 MB/s | 1.5 MB 00:00 2023-10-16T14:13:38.682 INFO:teuthology.orchestra.run.smithi029.stdout:(54/143): postgresql-contrib-13.11-1.el9.x86_64 3.2 MB/s | 811 kB 00:00 2023-10-16T14:13:38.707 INFO:teuthology.orchestra.run.smithi029.stdout:(55/143): postgresql-private-libs-13.11-1.el9.x 1.8 MB/s | 134 kB 00:00 2023-10-16T14:13:38.933 INFO:teuthology.orchestra.run.smithi029.stdout:(56/143): ceph-mgr-diskprediction-local-18.0.0- 3.8 MB/s | 7.4 MB 00:01 2023-10-16T14:13:39.091 INFO:teuthology.orchestra.run.smithi029.stdout:(57/143): postgresql-server-13.11-1.el9.x86_64. 14 MB/s | 5.7 MB 00:00 2023-10-16T14:13:39.267 INFO:teuthology.orchestra.run.smithi029.stdout:(58/143): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2023-10-16T14:13:39.302 INFO:teuthology.orchestra.run.smithi029.stdout:(59/143): python3-devel-3.9.17-2.el9.x86_64.rpm 579 kB/s | 205 kB 00:00 2023-10-16T14:13:39.336 INFO:teuthology.orchestra.run.smithi029.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 992 kB/s | 228 kB 00:00 2023-10-16T14:13:39.361 INFO:teuthology.orchestra.run.smithi029.stdout:(61/143): python3-jsonpatch-1.21-16.el9.noarch. 437 kB/s | 24 kB 00:00 2023-10-16T14:13:39.428 INFO:teuthology.orchestra.run.smithi029.stdout:(62/143): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2023-10-16T14:13:39.462 INFO:teuthology.orchestra.run.smithi029.stdout:(63/143): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2023-10-16T14:13:39.488 INFO:teuthology.orchestra.run.smithi029.stdout:(64/143): python3-markupsafe-1.1.1-12.el9.x86_6 556 kB/s | 32 kB 00:00 2023-10-16T14:13:39.722 INFO:teuthology.orchestra.run.smithi029.stdout:(65/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2023-10-16T14:13:39.747 INFO:teuthology.orchestra.run.smithi029.stdout:(66/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 767 kB/s | 217 kB 00:00 2023-10-16T14:13:39.781 INFO:teuthology.orchestra.run.smithi029.stdout:(67/143): python3-oauthlib-3.1.1-5.el9.noarch.r 630 kB/s | 174 kB 00:00 2023-10-16T14:13:39.815 INFO:teuthology.orchestra.run.smithi029.stdout:(68/143): python3-prettytable-0.7.2-27.el9.noar 629 kB/s | 41 kB 00:00 2023-10-16T14:13:39.840 INFO:teuthology.orchestra.run.smithi029.stdout:(69/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2023-10-16T14:13:39.874 INFO:teuthology.orchestra.run.smithi029.stdout:(70/143): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 211 kB 00:00 2023-10-16T14:13:39.916 INFO:teuthology.orchestra.run.smithi029.stdout:(71/143): python3-requests-oauthlib-1.3.0-12.el 588 kB/s | 43 kB 00:00 2023-10-16T14:13:40.258 INFO:teuthology.orchestra.run.smithi029.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 101 kB/s | 39 kB 00:00 2023-10-16T14:13:40.300 INFO:teuthology.orchestra.run.smithi029.stdout:(73/143): socat-1.7.4.1-5.el9.x86_64.rpm 796 kB/s | 300 kB 00:00 2023-10-16T14:13:40.485 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 8.5 kB/s | 14 kB 00:01 2023-10-16T14:13:40.626 INFO:teuthology.orchestra.run.smithi029.stdout:(74/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2023-10-16T14:13:40.651 INFO:teuthology.orchestra.run.smithi029.stdout:(75/143): uuid-1.6.2-55.el9.x86_64.rpm 145 kB/s | 56 kB 00:00 2023-10-16T14:13:40.677 INFO:teuthology.orchestra.run.smithi029.stdout:(76/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 173 kB/s | 63 kB 00:00 2023-10-16T14:13:40.721 INFO:teuthology.orchestra.run.smithi029.stdout:(77/143): fuse-2.9.9-15.el9.x86_64.rpm 1.1 MB/s | 79 kB 00:00 2023-10-16T14:13:40.771 INFO:teuthology.orchestra.run.smithi029.stdout:(78/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2023-10-16T14:13:40.796 INFO:teuthology.orchestra.run.smithi029.stdout:(79/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2023-10-16T14:13:40.830 INFO:teuthology.orchestra.run.smithi029.stdout:(80/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2023-10-16T14:13:40.864 INFO:teuthology.orchestra.run.smithi029.stdout:(81/143): python3-pycparser-2.20-6.el9.noarch.r 1.9 MB/s | 124 kB 00:00 2023-10-16T14:13:40.898 INFO:teuthology.orchestra.run.smithi029.stdout:(82/143): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2023-10-16T14:13:40.923 INFO:teuthology.orchestra.run.smithi029.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 1.2 MB/s | 113 kB 00:00 2023-10-16T14:13:40.957 INFO:teuthology.orchestra.run.smithi029.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 2.0 MB/s | 188 kB 00:00 2023-10-16T14:13:41.074 INFO:teuthology.orchestra.run.smithi029.stdout:(85/143): elfutils-libelf-devel-0.189-3.el9.x86 191 kB/s | 22 kB 00:00 2023-10-16T14:13:41.183 INFO:teuthology.orchestra.run.smithi029.stdout:(86/143): jq-1.6-15.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-10-16T14:13:41.242 INFO:teuthology.orchestra.run.smithi029.stdout:(87/143): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2023-10-16T14:13:41.276 INFO:teuthology.orchestra.run.smithi029.stdout:(88/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2023-10-16T14:13:41.318 INFO:teuthology.orchestra.run.smithi029.stdout:(89/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 342 kB/s | 26 kB 00:00 2023-10-16T14:13:41.427 INFO:teuthology.orchestra.run.smithi029.stdout:(90/143): python3-cryptography-36.0.1-4.el9.x86 2.4 MB/s | 1.2 MB 00:00 2023-10-16T14:13:41.486 INFO:teuthology.orchestra.run.smithi029.stdout:(91/143): python3-pytz-2021.1-5.el9.noarch.rpm 306 kB/s | 51 kB 00:00 2023-10-16T14:13:41.520 INFO:teuthology.orchestra.run.smithi029.stdout:(92/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 941 kB/s | 86 kB 00:00 2023-10-16T14:13:41.647 INFO:teuthology.orchestra.run.smithi029.stdout:(93/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 282 kB/s | 45 kB 00:00 2023-10-16T14:13:41.726 INFO:teuthology.orchestra.run.smithi029.stdout:(94/143): fmt-8.1.1-5.el9.x86_64.rpm 540 kB/s | 111 kB 00:00 2023-10-16T14:13:41.777 INFO:teuthology.orchestra.run.smithi029.stdout:(95/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 2.4 MB/s | 309 kB 00:00 2023-10-16T14:13:41.952 INFO:teuthology.orchestra.run.smithi029.stdout:(96/143): libarrow-9.0.0-9.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2023-10-16T14:13:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:(97/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 127 kB/s | 24 kB 00:00 2023-10-16T14:13:42.022 INFO:teuthology.orchestra.run.smithi029.stdout:(98/143): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2023-10-16T14:13:42.047 INFO:teuthology.orchestra.run.smithi029.stdout:(99/143): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2023-10-16T14:13:42.106 INFO:teuthology.orchestra.run.smithi029.stdout:(100/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 9.9 MB/s | 837 kB 00:00 2023-10-16T14:13:42.148 INFO:teuthology.orchestra.run.smithi029.stdout:(101/143): python3-asyncssh-2.13.2-3.el9.noarch 5.8 MB/s | 548 kB 00:00 2023-10-16T14:13:42.173 INFO:teuthology.orchestra.run.smithi029.stdout:(102/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 706 kB/s | 43 kB 00:00 2023-10-16T14:13:42.199 INFO:teuthology.orchestra.run.smithi029.stdout:(103/143): python3-cachetools-4.2.4-1.el9.noarc 650 kB/s | 32 kB 00:00 2023-10-16T14:13:42.224 INFO:teuthology.orchestra.run.smithi029.stdout:(104/143): python3-certifi-2021.10.8-2.el9.noar 302 kB/s | 15 kB 00:00 2023-10-16T14:13:42.258 INFO:teuthology.orchestra.run.smithi029.stdout:(105/143): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2023-10-16T14:13:42.301 INFO:teuthology.orchestra.run.smithi029.stdout:(106/143): python3-cherrypy-18.6.1-2.el9.noarch 4.8 MB/s | 358 kB 00:00 2023-10-16T14:13:42.334 INFO:teuthology.orchestra.run.smithi029.stdout:(107/143): python3-google-auth-2.23.2-1.el9.noa 2.7 MB/s | 196 kB 00:00 2023-10-16T14:13:42.368 INFO:teuthology.orchestra.run.smithi029.stdout:(108/143): python3-jaraco-8.2.1-3.el9.noarch.rp 165 kB/s | 11 kB 00:00 2023-10-16T14:13:42.402 INFO:teuthology.orchestra.run.smithi029.stdout:(109/143): python3-jaraco-classes-3.2.1-5.el9.n 266 kB/s | 18 kB 00:00 2023-10-16T14:13:42.436 INFO:teuthology.orchestra.run.smithi029.stdout:(110/143): python3-jaraco-collections-3.0.0-8.e 349 kB/s | 23 kB 00:00 2023-10-16T14:13:42.470 INFO:teuthology.orchestra.run.smithi029.stdout:(111/143): python3-jaraco-functools-3.5.0-2.el9 292 kB/s | 19 kB 00:00 2023-10-16T14:13:42.505 INFO:teuthology.orchestra.run.smithi029.stdout:(112/143): python3-jaraco-text-3.2.0-6.el9.noar 289 kB/s | 20 kB 00:00 2023-10-16T14:13:42.531 INFO:teuthology.orchestra.run.smithi029.stdout:(113/143): python3-jwt+crypto-2.4.0-1.el9.noarc 152 kB/s | 9.0 kB 00:00 2023-10-16T14:13:42.556 INFO:teuthology.orchestra.run.smithi029.stdout:(114/143): python3-jwt-2.4.0-1.el9.noarch.rpm 822 kB/s | 41 kB 00:00 2023-10-16T14:13:42.615 INFO:teuthology.orchestra.run.smithi029.stdout:(115/143): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2023-10-16T14:13:42.641 INFO:teuthology.orchestra.run.smithi029.stdout:(116/143): python3-logutils-0.3.5-21.el9.noarch 629 kB/s | 46 kB 00:00 2023-10-16T14:13:42.675 INFO:teuthology.orchestra.run.smithi029.stdout:(117/143): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2023-10-16T14:13:42.700 INFO:teuthology.orchestra.run.smithi029.stdout:(118/143): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2023-10-16T14:13:42.743 INFO:teuthology.orchestra.run.smithi029.stdout:(119/143): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2023-10-16T14:13:42.768 INFO:teuthology.orchestra.run.smithi029.stdout:(120/143): python3-portend-3.1.0-2.el9.noarch.r 256 kB/s | 16 kB 00:00 2023-10-16T14:13:42.802 INFO:teuthology.orchestra.run.smithi029.stdout:(121/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2023-10-16T14:13:42.828 INFO:teuthology.orchestra.run.smithi029.stdout:(122/143): python3-repoze-lru-0.7-16.el9.noarch 535 kB/s | 31 kB 00:00 2023-10-16T14:13:42.861 INFO:teuthology.orchestra.run.smithi029.stdout:(123/143): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2023-10-16T14:13:42.887 INFO:teuthology.orchestra.run.smithi029.stdout:(124/143): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2023-10-16T14:13:42.913 INFO:teuthology.orchestra.run.smithi029.stdout:(125/143): python3-tempora-5.0.0-2.el9.noarch.r 723 kB/s | 36 kB 00:00 2023-10-16T14:13:42.938 INFO:teuthology.orchestra.run.smithi029.stdout:(126/143): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2023-10-16T14:13:42.974 INFO:teuthology.orchestra.run.smithi029.stdout:(127/143): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-10-16T14:13:42.999 INFO:teuthology.orchestra.run.smithi029.stdout:(128/143): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2023-10-16T14:13:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:(129/143): python3-werkzeug-2.0.3-3.el9.1.noarc 7.3 MB/s | 427 kB 00:00 2023-10-16T14:13:43.034 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 19 MB 00:02 2023-10-16T14:13:43.067 INFO:teuthology.orchestra.run.smithi029.stdout:(130/143): python3-zc-lockfile-2.0-10.el9.noarc 318 kB/s | 20 kB 00:00 2023-10-16T14:13:43.101 INFO:teuthology.orchestra.run.smithi029.stdout:(131/143): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2023-10-16T14:13:43.176 INFO:teuthology.orchestra.run.smithi029.stdout:(132/143): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-10-16T14:13:43.310 INFO:teuthology.orchestra.run.smithi029.stdout:(133/143): openssl-devel-3.0.7-24.el9.x86_64.rp 2.0 MB/s | 4.1 MB 00:02 2023-10-16T14:13:43.394 INFO:teuthology.orchestra.run.smithi029.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 819 kB/s | 68 kB 00:00 2023-10-16T14:13:43.537 INFO:teuthology.orchestra.run.smithi029.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 16 MB/s | 2.2 MB 00:00 2023-10-16T14:13:43.846 INFO:teuthology.orchestra.run.smithi029.stdout:(136/143): librados2-18.0.0-6730.g4ba8a190.el9. 4.5 MB/s | 3.3 MB 00:00 2023-10-16T14:13:43.921 INFO:teuthology.orchestra.run.smithi029.stdout:(137/143): NetworkManager-libnm-1.45.4-1.el9.x8 4.6 MB/s | 1.8 MB 00:00 2023-10-16T14:13:43.947 INFO:teuthology.orchestra.run.smithi029.stdout:(138/143): NetworkManager-team-1.45.4-1.el9.x86 425 kB/s | 41 kB 00:00 2023-10-16T14:13:43.981 INFO:teuthology.orchestra.run.smithi029.stdout:(139/143): NetworkManager-tui-1.45.4-1.el9.x86_ 4.1 MB/s | 244 kB 00:00 2023-10-16T14:13:44.015 INFO:teuthology.orchestra.run.smithi029.stdout:(140/143): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2023-10-16T14:13:44.042 INFO:teuthology.orchestra.run.smithi029.stdout:(141/143): libacl-2.3.1-4.el9.x86_64.rpm 371 kB/s | 22 kB 00:00 2023-10-16T14:13:44.075 INFO:teuthology.orchestra.run.smithi029.stdout:(142/143): xfsdump-3.1.12-4.el9.x86_64.rpm 5.3 MB/s | 321 kB 00:00 2023-10-16T14:13:44.192 INFO:teuthology.orchestra.run.smithi029.stdout:(143/143): librbd1-18.0.0-6730.g4ba8a190.el9.x8 3.0 MB/s | 3.0 MB 00:01 2023-10-16T14:13:44.205 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-10-16T14:13:44.205 INFO:teuthology.orchestra.run.smithi029.stdout:Total 15 MB/s | 199 MB 00:13 2023-10-16T14:13:44.670 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-10-16T14:13:44.886 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-10-16T14:13:44.887 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-10-16T14:13:47.197 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-10-16T14:13:47.198 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-10-16T14:13:48.211 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2023-10-16T14:13:50.719 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T14:13:50.721 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T14:13:50.742 INFO:teuthology.orchestra.run.smithi170.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-16T14:13:50.743 INFO:teuthology.orchestra.run.smithi170.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-16T14:13:50.744 INFO:teuthology.orchestra.run.smithi170.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-16T14:13:50.746 INFO:teuthology.orchestra.run.smithi170.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-16T14:13:50.748 INFO:teuthology.orchestra.run.smithi170.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-16T14:13:50.752 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-16T14:13:50.753 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-16T14:13:50.829 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-10-16T14:13:50.850 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-10-16T14:13:50.850 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-10-16T14:13:50.850 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-10-16T14:13:50.850 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-10-16T14:13:50.851 INFO:teuthology.orchestra.run.smithi170.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-16T14:13:50.851 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 6.5 k 2023-10-16T14:13:50.851 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 5.2 M 2023-10-16T14:13:50.851 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 855 k 2023-10-16T14:13:50.851 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 141 k 2023-10-16T14:13:50.852 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 1.5 M 2023-10-16T14:13:50.852 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 132 k 2023-10-16T14:13:50.852 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 1.8 M 2023-10-16T14:13:50.852 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 7.4 M 2023-10-16T14:13:50.852 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 47 k 2023-10-16T14:13:50.853 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 8.6 M 2023-10-16T14:13:50.853 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 41 M 2023-10-16T14:13:50.853 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 262 k 2023-10-16T14:13:50.853 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 269 k 2023-10-16T14:13:50.853 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 218 k 2023-10-16T14:13:50.854 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 18 k 2023-10-16T14:13:50.854 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-16T14:13:50.854 INFO:teuthology.orchestra.run.smithi170.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-16T14:13:50.854 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-16T14:13:50.854 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 31 k 2023-10-16T14:13:50.855 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 715 k 2023-10-16T14:13:50.855 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 127 k 2023-10-16T14:13:50.855 INFO:teuthology.orchestra.run.smithi170.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-16T14:13:50.855 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-16T14:13:50.855 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-16T14:13:50.856 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-16T14:13:50.856 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-16T14:13:50.856 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-16T14:13:50.856 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 162 k 2023-10-16T14:13:50.856 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 321 k 2023-10-16T14:13:50.857 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 298 k 2023-10-16T14:13:50.857 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 99 k 2023-10-16T14:13:50.857 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 85 k 2023-10-16T14:13:50.857 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 3.0 M 2023-10-16T14:13:50.858 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 169 k 2023-10-16T14:13:50.858 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-16T14:13:50.858 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-10-16T14:13:50.858 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-16T14:13:50.858 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-16T14:13:50.859 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-16T14:13:50.859 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-16T14:13:50.859 INFO:teuthology.orchestra.run.smithi170.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-16T14:13:50.859 INFO:teuthology.orchestra.run.smithi170.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-16T14:13:50.859 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-16T14:13:50.860 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 3.3 M 2023-10-16T14:13:50.860 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 3.0 M 2023-10-16T14:13:50.860 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-16T14:13:50.860 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-10-16T14:13:50.860 INFO:teuthology.orchestra.run.smithi170.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-16T14:13:50.860 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-16T14:13:50.861 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 19 M 2023-10-16T14:13:50.861 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 23 k 2023-10-16T14:13:50.861 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 2.1 M 2023-10-16T14:13:50.861 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 242 k 2023-10-16T14:13:50.861 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 4.4 M 2023-10-16T14:13:50.862 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 18 M 2023-10-16T14:13:50.862 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6730.g4ba8a190.el9 ceph-noarch 15 k 2023-10-16T14:13:50.862 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 24 k 2023-10-16T14:13:50.862 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-16T14:13:50.862 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-16T14:13:50.862 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-16T14:13:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-16T14:13:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-16T14:13:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-16T14:13:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-16T14:13:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-16T14:13:50.863 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-16T14:13:50.864 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 163 k 2023-10-16T14:13:50.864 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-16T14:13:50.864 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-16T14:13:50.864 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-16T14:13:50.864 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-16T14:13:50.865 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-16T14:13:50.865 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 472 k 2023-10-16T14:13:50.865 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-16T14:13:50.865 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 4.8 M 2023-10-16T14:13:50.865 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-16T14:13:50.865 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-16T14:13:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-16T14:13:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-16T14:13:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-16T14:13:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-16T14:13:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-16T14:13:50.866 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-16T14:13:50.867 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-16T14:13:50.867 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-16T14:13:50.867 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-16T14:13:50.867 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-16T14:13:50.869 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 45 k 2023-10-16T14:13:50.869 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:18.0.0-6730.g4ba8a190.el9 ceph 127 k 2023-10-16T14:13:50.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-16T14:13:50.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-16T14:13:50.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-16T14:13:50.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-16T14:13:50.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-16T14:13:50.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-16T14:13:50.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-16T14:13:50.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-16T14:13:50.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-16T14:13:50.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-16T14:13:50.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-16T14:13:50.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-16T14:13:50.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-16T14:13:50.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-16T14:13:50.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-16T14:13:50.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-16T14:13:50.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-16T14:13:50.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-16T14:13:50.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-16T14:13:50.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-16T14:13:50.874 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-16T14:13:50.874 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-16T14:13:50.874 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-16T14:13:50.874 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-16T14:13:50.874 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-16T14:13:50.876 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-16T14:13:50.876 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-16T14:13:50.876 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-16T14:13:50.876 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-16T14:13:50.876 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-16T14:13:50.877 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-16T14:13:50.877 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-16T14:13:50.877 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-16T14:13:50.877 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-16T14:13:50.877 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi170.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-16T14:13:50.880 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-16T14:13:50.880 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-16T14:13:50.880 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2023-10-16T14:13:50.880 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-16T14:13:50.880 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:13:50.881 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-10-16T14:13:50.881 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-10-16T14:13:50.881 INFO:teuthology.orchestra.run.smithi170.stdout:Install 133 Packages 2023-10-16T14:13:50.881 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 10 Packages 2023-10-16T14:13:50.881 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:13:50.882 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 199 M 2023-10-16T14:13:50.882 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-10-16T14:13:51.863 INFO:teuthology.orchestra.run.smithi170.stdout:(1/143): ceph-18.0.0-6730.g4ba8a190.el9.x86_64. 18 kB/s | 6.5 kB 00:00 2023-10-16T14:13:52.222 INFO:teuthology.orchestra.run.smithi170.stdout:(2/143): ceph-base-18.0.0-6730.g4ba8a190.el9.x8 7.2 MB/s | 5.2 MB 00:00 2023-10-16T14:13:52.288 INFO:teuthology.orchestra.run.smithi170.stdout:(3/143): ceph-immutable-object-cache-18.0.0-673 2.1 MB/s | 141 kB 00:00 2023-10-16T14:13:52.573 INFO:teuthology.orchestra.run.smithi170.stdout:(4/143): ceph-mds-18.0.0-6730.g4ba8a190.el9.x86 7.3 MB/s | 2.1 MB 00:00 2023-10-16T14:13:52.747 INFO:teuthology.orchestra.run.smithi170.stdout:(5/143): ceph-mgr-18.0.0-6730.g4ba8a190.el9.x86 8.5 MB/s | 1.5 MB 00:00 2023-10-16T14:13:53.181 INFO:teuthology.orchestra.run.smithi170.stdout:(6/143): ceph-common-18.0.0-6730.g4ba8a190.el9. 11 MB/s | 19 MB 00:01 2023-10-16T14:13:53.231 INFO:teuthology.orchestra.run.smithi170.stdout:(7/143): ceph-fuse-18.0.0-6730.g4ba8a190.el9.x8 626 kB/s | 855 kB 00:01 2023-10-16T14:13:53.341 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-10-16T14:13:53.380 INFO:teuthology.orchestra.run.smithi170.stdout:(8/143): ceph-mon-18.0.0-6730.g4ba8a190.el9.x86 6.9 MB/s | 4.4 MB 00:00 2023-10-16T14:13:53.427 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-16T14:13:53.448 INFO:teuthology.orchestra.run.smithi170.stdout:(9/143): ceph-selinux-18.0.0-6730.g4ba8a190.el9 359 kB/s | 24 kB 00:00 2023-10-16T14:13:53.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-16T14:13:53.494 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-16T14:13:53.605 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-16T14:13:54.047 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-16T14:13:54.178 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 6/153 2023-10-16T14:13:54.347 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 6/153 2023-10-16T14:13:54.380 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 7/153 2023-10-16T14:13:54.507 INFO:teuthology.orchestra.run.smithi170.stdout:(10/143): ceph-osd-18.0.0-6730.g4ba8a190.el9.x8 13 MB/s | 18 MB 00:01 2023-10-16T14:13:54.590 INFO:teuthology.orchestra.run.smithi170.stdout:(11/143): cephfs-mirror-18.0.0-6730.g4ba8a190.e 2.6 MB/s | 218 kB 00:00 2023-10-16T14:13:54.604 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 7/153 2023-10-16T14:13:54.649 INFO:teuthology.orchestra.run.smithi170.stdout:(12/143): libcephfs-devel-18.0.0-6730.g4ba8a190 543 kB/s | 31 kB 00:00 2023-10-16T14:13:54.688 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:18.0.0-6730.g4ba8a190.el9.x86_64 8/153 2023-10-16T14:13:54.735 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-16T14:13:54.782 INFO:teuthology.orchestra.run.smithi170.stdout:(13/143): libcephfs2-18.0.0-6730.g4ba8a190.el9. 5.3 MB/s | 715 kB 00:00 2023-10-16T14:13:54.858 INFO:teuthology.orchestra.run.smithi170.stdout:(14/143): libcephsqlite-18.0.0-6730.g4ba8a190.e 2.2 MB/s | 163 kB 00:00 2023-10-16T14:13:54.933 INFO:teuthology.orchestra.run.smithi170.stdout:(15/143): librados-devel-18.0.0-6730.g4ba8a190. 1.7 MB/s | 127 kB 00:00 2023-10-16T14:13:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-16T14:13:55.033 INFO:teuthology.orchestra.run.smithi170.stdout:(16/143): libradosstriper1-18.0.0-6730.g4ba8a19 4.7 MB/s | 472 kB 00:00 2023-10-16T14:13:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-16T14:13:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 12/153 2023-10-16T14:13:55.243 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 12/153 2023-10-16T14:13:55.317 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86 13/153 2023-10-16T14:13:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86 13/153 2023-10-16T14:13:55.542 INFO:teuthology.orchestra.run.smithi170.stdout:(17/143): librgw2-18.0.0-6730.g4ba8a190.el9.x86 9.4 MB/s | 4.8 MB 00:00 2023-10-16T14:13:55.617 INFO:teuthology.orchestra.run.smithi170.stdout:(18/143): python3-ceph-argparse-18.0.0-6730.g4b 611 kB/s | 45 kB 00:00 2023-10-16T14:13:55.701 INFO:teuthology.orchestra.run.smithi170.stdout:(19/143): python3-ceph-common-18.0.0-6730.g4ba8 1.5 MB/s | 127 kB 00:00 2023-10-16T14:13:55.916 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-16T14:13:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T14:13:56.143 INFO:teuthology.orchestra.run.smithi170.stdout:(20/143): ceph-test-18.0.0-6730.g4ba8a190.el9.x 15 MB/s | 41 MB 00:02 2023-10-16T14:13:56.209 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T14:13:56.210 INFO:teuthology.orchestra.run.smithi170.stdout:(21/143): python3-cephfs-18.0.0-6730.g4ba8a190. 319 kB/s | 162 kB 00:00 2023-10-16T14:13:56.251 INFO:teuthology.orchestra.run.smithi170.stdout:(22/143): python3-rados-18.0.0-6730.g4ba8a190.e 2.9 MB/s | 321 kB 00:00 2023-10-16T14:13:56.285 INFO:teuthology.orchestra.run.smithi170.stdout:(23/143): python3-rbd-18.0.0-6730.g4ba8a190.el9 3.9 MB/s | 298 kB 00:00 2023-10-16T14:13:56.310 INFO:teuthology.orchestra.run.smithi170.stdout:(24/143): python3-rgw-18.0.0-6730.g4ba8a190.el9 1.7 MB/s | 99 kB 00:00 2023-10-16T14:13:56.336 INFO:teuthology.orchestra.run.smithi170.stdout:(25/143): rbd-fuse-18.0.0-6730.g4ba8a190.el9.x8 1.7 MB/s | 85 kB 00:00 2023-10-16T14:13:56.407 INFO:teuthology.orchestra.run.smithi170.stdout:(26/143): rbd-nbd-18.0.0-6730.g4ba8a190.el9.x86 2.3 MB/s | 169 kB 00:00 2023-10-16T14:13:56.481 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T14:13:56.508 INFO:teuthology.orchestra.run.smithi170.stdout:(27/143): rbd-mirror-18.0.0-6730.g4ba8a190.el9. 15 MB/s | 3.0 MB 00:00 2023-10-16T14:13:56.533 INFO:teuthology.orchestra.run.smithi170.stdout:(28/143): ceph-grafana-dashboards-18.0.0-6730.g 186 kB/s | 23 kB 00:00 2023-10-16T14:13:56.535 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T14:13:56.608 INFO:teuthology.orchestra.run.smithi170.stdout:(29/143): ceph-mgr-cephadm-18.0.0-6730.g4ba8a19 1.7 MB/s | 132 kB 00:00 2023-10-16T14:13:56.684 INFO:teuthology.orchestra.run.smithi170.stdout:(30/143): ceph-mgr-dashboard-18.0.0-6730.g4ba8a 12 MB/s | 1.8 MB 00:00 2023-10-16T14:13:56.759 INFO:teuthology.orchestra.run.smithi170.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6730.g4b 3.2 MB/s | 242 kB 00:00 2023-10-16T14:13:56.834 INFO:teuthology.orchestra.run.smithi170.stdout:(32/143): ceph-mgr-rook-18.0.0-6730.g4ba8a190.e 641 kB/s | 47 kB 00:00 2023-10-16T14:13:56.901 INFO:teuthology.orchestra.run.smithi170.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6730.g4 229 kB/s | 15 kB 00:00 2023-10-16T14:13:57.017 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T14:13:57.058 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-16T14:13:57.060 INFO:teuthology.orchestra.run.smithi170.stdout:(34/143): ceph-mgr-diskprediction-local-18.0.0- 16 MB/s | 7.4 MB 00:00 2023-10-16T14:13:57.093 INFO:teuthology.orchestra.run.smithi170.stdout:(35/143): ceph-volume-18.0.0-6730.g4ba8a190.el9 1.3 MB/s | 262 kB 00:00 2023-10-16T14:13:57.115 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-16T14:13:57.127 INFO:teuthology.orchestra.run.smithi170.stdout:(36/143): cephadm-18.0.0-6730.g4ba8a190.el9.noa 4.0 MB/s | 269 kB 00:00 2023-10-16T14:13:57.152 INFO:teuthology.orchestra.run.smithi170.stdout:(37/143): cephfs-top-18.0.0-6730.g4ba8a190.el9. 315 kB/s | 18 kB 00:00 2023-10-16T14:13:57.244 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-16T14:13:57.333 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-16T14:13:57.353 INFO:teuthology.orchestra.run.smithi170.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 531 kB/s | 106 kB 00:00 2023-10-16T14:13:57.375 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-16T14:13:57.419 INFO:teuthology.orchestra.run.smithi170.stdout:(39/143): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 920 kB 00:00 2023-10-16T14:13:57.446 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-16T14:13:57.453 INFO:teuthology.orchestra.run.smithi170.stdout:(40/143): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 307 kB 00:00 2023-10-16T14:13:57.478 INFO:teuthology.orchestra.run.smithi170.stdout:(41/143): flexiblas-3.0.4-8.el9.x86_64.rpm 571 kB/s | 32 kB 00:00 2023-10-16T14:13:57.567 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-16T14:13:57.587 INFO:teuthology.orchestra.run.smithi170.stdout:(42/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2023-10-16T14:13:57.612 INFO:teuthology.orchestra.run.smithi170.stdout:(43/143): flexiblas-openblas-openmp-3.0.4-8.el9 144 kB/s | 17 kB 00:00 2023-10-16T14:13:57.622 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-16T14:13:57.656 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-16T14:13:57.754 INFO:teuthology.orchestra.run.smithi170.stdout:(44/143): libaio-devel-0.3.111-13.el9.x86_64.rp 61 kB/s | 10 kB 00:00 2023-10-16T14:13:57.775 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-16T14:13:57.804 INFO:teuthology.orchestra.run.smithi170.stdout:(45/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 833 kB/s | 159 kB 00:00 2023-10-16T14:13:57.812 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el9. 27/153 2023-10-16T14:13:57.846 INFO:teuthology.orchestra.run.smithi170.stdout:(46/143): librdkafka-1.6.1-102.el9.x86_64.rpm 7.2 MB/s | 661 kB 00:00 2023-10-16T14:13:57.863 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 28/153 2023-10-16T14:13:57.889 INFO:teuthology.orchestra.run.smithi170.stdout:(47/143): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2023-10-16T14:13:57.914 INFO:teuthology.orchestra.run.smithi170.stdout:(48/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 758 kB/s | 46 kB 00:00 2023-10-16T14:13:57.947 INFO:teuthology.orchestra.run.smithi170.stdout:(49/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-16T14:13:57.981 INFO:teuthology.orchestra.run.smithi170.stdout:(50/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-10-16T14:13:57.985 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:18.0.0-6730.g4ba8a190.el9.x86_6 29/153 2023-10-16T14:13:58.006 INFO:teuthology.orchestra.run.smithi170.stdout:(51/143): openblas-0.3.21-2.el9.x86_64.rpm 611 kB/s | 34 kB 00:00 2023-10-16T14:13:58.178 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-16T14:13:58.215 INFO:teuthology.orchestra.run.smithi170.stdout:(52/143): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2023-10-16T14:13:58.241 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-16T14:13:58.290 INFO:teuthology.orchestra.run.smithi170.stdout:(53/143): postgresql-13.11-1.el9.x86_64.rpm 5.7 MB/s | 1.5 MB 00:00 2023-10-16T14:13:58.341 INFO:teuthology.orchestra.run.smithi170.stdout:(54/143): postgresql-contrib-13.11-1.el9.x86_64 7.1 MB/s | 811 kB 00:00 2023-10-16T14:13:58.366 INFO:teuthology.orchestra.run.smithi170.stdout:(55/143): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2023-10-16T14:13:58.724 INFO:teuthology.orchestra.run.smithi170.stdout:(56/143): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-10-16T14:13:58.850 INFO:teuthology.orchestra.run.smithi170.stdout:(57/143): postgresql-server-13.11-1.el9.x86_64. 11 MB/s | 5.7 MB 00:00 2023-10-16T14:13:58.883 INFO:teuthology.orchestra.run.smithi170.stdout:(58/143): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 205 kB 00:00 2023-10-16T14:13:58.942 INFO:teuthology.orchestra.run.smithi170.stdout:(59/143): python3-jinja2-2.11.3-4.el9.noarch.rp 2.5 MB/s | 228 kB 00:00 2023-10-16T14:13:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-16T14:13:59.009 INFO:teuthology.orchestra.run.smithi170.stdout:(60/143): python3-jsonpatch-1.21-16.el9.noarch. 195 kB/s | 24 kB 00:00 2023-10-16T14:13:59.042 INFO:teuthology.orchestra.run.smithi170.stdout:(61/143): python3-jsonpointer-2.0-4.el9.noarch. 182 kB/s | 18 kB 00:00 2023-10-16T14:13:59.068 INFO:teuthology.orchestra.run.smithi170.stdout:(62/143): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2023-10-16T14:13:59.093 INFO:teuthology.orchestra.run.smithi170.stdout:(63/143): python3-markupsafe-1.1.1-12.el9.x86_6 655 kB/s | 32 kB 00:00 2023-10-16T14:13:59.302 INFO:teuthology.orchestra.run.smithi170.stdout:(64/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2023-10-16T14:13:59.335 INFO:teuthology.orchestra.run.smithi170.stdout:(65/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 936 kB/s | 217 kB 00:00 2023-10-16T14:13:59.361 INFO:teuthology.orchestra.run.smithi170.stdout:(66/143): python3-oauthlib-3.1.1-5.el9.noarch.r 3.0 MB/s | 174 kB 00:00 2023-10-16T14:13:59.411 INFO:teuthology.orchestra.run.smithi170.stdout:(67/143): python3-prettytable-0.7.2-27.el9.noar 564 kB/s | 41 kB 00:00 2023-10-16T14:13:59.436 INFO:teuthology.orchestra.run.smithi170.stdout:(68/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-16T14:13:59.470 INFO:teuthology.orchestra.run.smithi170.stdout:(69/143): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2023-10-16T14:13:59.495 INFO:teuthology.orchestra.run.smithi170.stdout:(70/143): python3-requests-oauthlib-1.3.0-12.el 765 kB/s | 43 kB 00:00 2023-10-16T14:13:59.792 INFO:teuthology.orchestra.run.smithi170.stdout:(71/143): ceph-radosgw-18.0.0-6730.g4ba8a190.el 1.3 MB/s | 8.6 MB 00:06 2023-10-16T14:13:59.854 INFO:teuthology.orchestra.run.smithi170.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 110 kB/s | 39 kB 00:00 2023-10-16T14:14:00.079 INFO:teuthology.orchestra.run.smithi170.stdout:(73/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2023-10-16T14:14:00.104 INFO:teuthology.orchestra.run.smithi170.stdout:(74/143): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 300 kB 00:00 2023-10-16T14:14:00.114 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-16T14:14:00.129 INFO:teuthology.orchestra.run.smithi170.stdout:(75/143): uuid-1.6.2-55.el9.x86_64.rpm 217 kB/s | 56 kB 00:00 2023-10-16T14:14:00.167 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-16T14:14:00.195 INFO:teuthology.orchestra.run.smithi170.stdout:(76/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 979 kB/s | 63 kB 00:00 2023-10-16T14:14:00.216 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:18.0.0-6730.g4ba8a190.el9.x86_6 35/153 2023-10-16T14:14:00.238 INFO:teuthology.orchestra.run.smithi170.stdout:(77/143): fuse-2.9.9-15.el9.x86_64.rpm 734 kB/s | 79 kB 00:00 2023-10-16T14:14:00.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-16T14:14:00.281 INFO:teuthology.orchestra.run.smithi170.stdout:(78/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 5.2 MB/s | 803 kB 00:00 2023-10-16T14:14:00.315 INFO:teuthology.orchestra.run.smithi170.stdout:(79/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2023-10-16T14:14:00.332 INFO:teuthology.orchestra.run.smithi170.stdout:(80/143): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2023-10-16T14:14:00.366 INFO:teuthology.orchestra.run.smithi170.stdout:(81/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.9 MB/s | 241 kB 00:00 2023-10-16T14:14:00.390 INFO:teuthology.orchestra.run.smithi170.stdout:(82/143): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2023-10-16T14:14:00.416 INFO:teuthology.orchestra.run.smithi170.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 1.4 MB/s | 113 kB 00:00 2023-10-16T14:14:00.443 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-16T14:14:00.458 INFO:teuthology.orchestra.run.smithi170.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 2.8 MB/s | 188 kB 00:00 2023-10-16T14:14:00.491 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-16T14:14:00.517 INFO:teuthology.orchestra.run.smithi170.stdout:(85/143): elfutils-libelf-devel-0.189-3.el9.x86 388 kB/s | 22 kB 00:00 2023-10-16T14:14:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-16T14:14:00.676 INFO:teuthology.orchestra.run.smithi170.stdout:(86/143): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2023-10-16T14:14:00.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-16T14:14:00.709 INFO:teuthology.orchestra.run.smithi170.stdout:(87/143): jq-1.6-15.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-10-16T14:14:00.727 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-16T14:14:00.768 INFO:teuthology.orchestra.run.smithi170.stdout:(88/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 531 kB/s | 45 kB 00:00 2023-10-16T14:14:00.776 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-16T14:14:00.793 INFO:teuthology.orchestra.run.smithi170.stdout:(89/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 309 kB/s | 26 kB 00:00 2023-10-16T14:14:00.860 INFO:teuthology.orchestra.run.smithi170.stdout:(90/143): python3-pytz-2021.1-5.el9.noarch.rpm 774 kB/s | 51 kB 00:00 2023-10-16T14:14:00.861 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-16T14:14:00.930 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-16T14:14:00.935 INFO:teuthology.orchestra.run.smithi170.stdout:(91/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.1 MB/s | 86 kB 00:00 2023-10-16T14:14:00.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-16T14:14:01.019 INFO:teuthology.orchestra.run.smithi170.stdout:(92/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 548 kB/s | 45 kB 00:00 2023-10-16T14:14:01.198 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-16T14:14:01.219 INFO:teuthology.orchestra.run.smithi170.stdout:(93/143): openssl-devel-3.0.7-24.el9.x86_64.rpm 9.2 MB/s | 4.1 MB 00:00 2023-10-16T14:14:01.247 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-16T14:14:01.278 INFO:teuthology.orchestra.run.smithi170.stdout:(94/143): fmt-8.1.1-5.el9.x86_64.rpm 430 kB/s | 111 kB 00:00 2023-10-16T14:14:01.309 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-16T14:14:01.345 INFO:teuthology.orchestra.run.smithi170.stdout:(95/143): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2023-10-16T14:14:01.455 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-16T14:14:01.470 INFO:teuthology.orchestra.run.smithi170.stdout:(96/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.2 MB/s | 309 kB 00:00 2023-10-16T14:14:01.495 INFO:teuthology.orchestra.run.smithi170.stdout:(97/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 163 kB/s | 24 kB 00:00 2023-10-16T14:14:01.529 INFO:teuthology.orchestra.run.smithi170.stdout:(98/143): liboath-2.6.7-2.el9.x86_64.rpm 840 kB/s | 49 kB 00:00 2023-10-16T14:14:01.556 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 50/153 2023-10-16T14:14:01.622 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-16T14:14:01.661 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-16T14:14:01.663 INFO:teuthology.orchestra.run.smithi170.stdout:(99/143): libarrow-9.0.0-9.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2023-10-16T14:14:01.688 INFO:teuthology.orchestra.run.smithi170.stdout:(100/143): libunwind-1.6.2-1.el9.x86_64.rpm 352 kB/s | 67 kB 00:00 2023-10-16T14:14:01.702 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-16T14:14:01.738 INFO:teuthology.orchestra.run.smithi170.stdout:(101/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2023-10-16T14:14:01.744 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-16T14:14:01.763 INFO:teuthology.orchestra.run.smithi170.stdout:(102/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 581 kB/s | 43 kB 00:00 2023-10-16T14:14:01.808 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-16T14:14:01.822 INFO:teuthology.orchestra.run.smithi170.stdout:(103/143): python3-asyncssh-2.13.2-3.el9.noarch 3.4 MB/s | 548 kB 00:00 2023-10-16T14:14:01.847 INFO:teuthology.orchestra.run.smithi170.stdout:(104/143): python3-certifi-2021.10.8-2.el9.noar 181 kB/s | 15 kB 00:00 2023-10-16T14:14:01.872 INFO:teuthology.orchestra.run.smithi170.stdout:(105/143): python3-cachetools-4.2.4-1.el9.noarc 242 kB/s | 32 kB 00:00 2023-10-16T14:14:01.898 INFO:teuthology.orchestra.run.smithi170.stdout:(106/143): python3-cheroot-8.6.0-4.el9.noarch.r 2.3 MB/s | 172 kB 00:00 2023-10-16T14:14:01.939 INFO:teuthology.orchestra.run.smithi170.stdout:(107/143): python3-google-auth-2.23.2-1.el9.noa 2.9 MB/s | 196 kB 00:00 2023-10-16T14:14:01.965 INFO:teuthology.orchestra.run.smithi170.stdout:(108/143): python3-jaraco-8.2.1-3.el9.noarch.rp 161 kB/s | 11 kB 00:00 2023-10-16T14:14:01.977 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-16T14:14:01.999 INFO:teuthology.orchestra.run.smithi170.stdout:(109/143): python3-jaraco-classes-3.2.1-5.el9.n 305 kB/s | 18 kB 00:00 2023-10-16T14:14:02.024 INFO:teuthology.orchestra.run.smithi170.stdout:(110/143): python3-jaraco-collections-3.0.0-8.e 400 kB/s | 23 kB 00:00 2023-10-16T14:14:02.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-16T14:14:02.057 INFO:teuthology.orchestra.run.smithi170.stdout:(111/143): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2023-10-16T14:14:02.083 INFO:teuthology.orchestra.run.smithi170.stdout:(112/143): python3-jaraco-functools-3.5.0-2.el9 235 kB/s | 19 kB 00:00 2023-10-16T14:14:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-16T14:14:02.116 INFO:teuthology.orchestra.run.smithi170.stdout:(113/143): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2023-10-16T14:14:02.133 INFO:teuthology.orchestra.run.smithi170.stdout:(114/143): python3-jwt+crypto-2.4.0-1.el9.noarc 120 kB/s | 9.0 kB 00:00 2023-10-16T14:14:02.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-16T14:14:02.196 INFO:teuthology.orchestra.run.smithi170.stdout:(115/143): python3-jwt-2.4.0-1.el9.noarch.rpm 522 kB/s | 41 kB 00:00 2023-10-16T14:14:02.217 INFO:teuthology.orchestra.run.smithi170.stdout:(116/143): python3-logutils-0.3.5-21.el9.noarch 557 kB/s | 46 kB 00:00 2023-10-16T14:14:02.251 INFO:teuthology.orchestra.run.smithi170.stdout:(117/143): python3-more-itertools-8.12.0-2.el9. 1.4 MB/s | 79 kB 00:00 2023-10-16T14:14:02.267 INFO:teuthology.orchestra.run.smithi170.stdout:(118/143): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2023-10-16T14:14:02.309 INFO:teuthology.orchestra.run.smithi170.stdout:(119/143): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2023-10-16T14:14:02.327 INFO:teuthology.orchestra.run.smithi170.stdout:(120/143): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2023-10-16T14:14:02.360 INFO:teuthology.orchestra.run.smithi170.stdout:(121/143): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2023-10-16T14:14:02.385 INFO:teuthology.orchestra.run.smithi170.stdout:(122/143): python3-repoze-lru-0.7-16.el9.noarch 530 kB/s | 31 kB 00:00 2023-10-16T14:14:02.419 INFO:teuthology.orchestra.run.smithi170.stdout:(123/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 829 kB/s | 90 kB 00:00 2023-10-16T14:14:02.444 INFO:teuthology.orchestra.run.smithi170.stdout:(124/143): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2023-10-16T14:14:02.461 INFO:teuthology.orchestra.run.smithi170.stdout:(125/143): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2023-10-16T14:14:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-16T14:14:02.486 INFO:teuthology.orchestra.run.smithi170.stdout:(126/143): python3-tempora-5.0.0-2.el9.noarch.r 538 kB/s | 36 kB 00:00 2023-10-16T14:14:02.511 INFO:teuthology.orchestra.run.smithi170.stdout:(127/143): python3-typing-extensions-4.4.0-2.el 763 kB/s | 51 kB 00:00 2023-10-16T14:14:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-16T14:14:02.545 INFO:teuthology.orchestra.run.smithi170.stdout:(128/143): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-10-16T14:14:02.591 INFO:teuthology.orchestra.run.smithi170.stdout:(129/143): python3-websocket-client-1.2.3-2.el9 865 kB/s | 90 kB 00:00 2023-10-16T14:14:02.629 INFO:teuthology.orchestra.run.smithi170.stdout:(130/143): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2023-10-16T14:14:02.654 INFO:teuthology.orchestra.run.smithi170.stdout:(131/143): re2-20211101-3.el9.x86_64.rpm 3.0 MB/s | 192 kB 00:00 2023-10-16T14:14:02.688 INFO:teuthology.orchestra.run.smithi170.stdout:(132/143): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2023-10-16T14:14:02.708 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-16T14:14:02.755 INFO:teuthology.orchestra.run.smithi170.stdout:(133/143): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-10-16T14:14:02.766 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-16T14:14:02.797 INFO:teuthology.orchestra.run.smithi170.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2023-10-16T14:14:02.799 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-16T14:14:02.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-16T14:14:02.914 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-16T14:14:02.931 INFO:teuthology.orchestra.run.smithi170.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 17 MB/s | 2.2 MB 00:00 2023-10-16T14:14:02.976 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-16T14:14:03.031 INFO:teuthology.orchestra.run.smithi170.stdout:(136/143): NetworkManager-libnm-1.45.4-1.el9.x8 18 MB/s | 1.8 MB 00:00 2023-10-16T14:14:03.055 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-16T14:14:03.106 INFO:teuthology.orchestra.run.smithi170.stdout:(137/143): librbd1-18.0.0-6730.g4ba8a190.el9.x8 7.3 MB/s | 3.0 MB 00:00 2023-10-16T14:14:03.132 INFO:teuthology.orchestra.run.smithi170.stdout:(138/143): NetworkManager-team-1.45.4-1.el9.x86 415 kB/s | 41 kB 00:00 2023-10-16T14:14:03.147 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-16T14:14:03.165 INFO:teuthology.orchestra.run.smithi170.stdout:(139/143): NetworkManager-tui-1.45.4-1.el9.x86_ 4.2 MB/s | 244 kB 00:00 2023-10-16T14:14:03.182 INFO:teuthology.orchestra.run.smithi170.stdout:(140/143): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2023-10-16T14:14:03.199 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-16T14:14:03.216 INFO:teuthology.orchestra.run.smithi170.stdout:(141/143): libacl-2.3.1-4.el9.x86_64.rpm 450 kB/s | 22 kB 00:00 2023-10-16T14:14:03.249 INFO:teuthology.orchestra.run.smithi170.stdout:(142/143): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2023-10-16T14:14:03.250 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-16T14:14:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-16T14:14:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-16T14:14:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-16T14:14:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-16T14:14:03.631 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-16T14:14:03.796 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-16T14:14:03.837 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-16T14:14:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-16T14:14:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-16T14:14:04.091 INFO:teuthology.orchestra.run.smithi170.stdout:(143/143): librados2-18.0.0-6730.g4ba8a190.el9. 2.3 MB/s | 3.3 MB 00:01 2023-10-16T14:14:04.101 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-10-16T14:14:04.101 INFO:teuthology.orchestra.run.smithi170.stdout:Total 15 MB/s | 199 MB 00:13 2023-10-16T14:14:04.544 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-10-16T14:14:04.746 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-10-16T14:14:04.746 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-10-16T14:14:05.092 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-16T14:14:05.193 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-16T14:14:05.238 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-16T14:14:05.389 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-16T14:14:05.480 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-16T14:14:05.755 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-16T14:14:05.786 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-16T14:14:05.893 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-16T14:14:06.755 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-16T14:14:06.906 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-10-16T14:14:06.907 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-10-16T14:14:06.914 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-16T14:14:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-16T14:14:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-16T14:14:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-16T14:14:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-16T14:14:08.810 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-16T14:14:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-16T14:14:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 97/153 2023-10-16T14:14:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 97/153 2023-10-16T14:14:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 98/153 2023-10-16T14:14:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-16T14:14:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-16T14:14:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 101/153 2023-10-16T14:14:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 101/153 2023-10-16T14:14:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 102/153 2023-10-16T14:14:12.734 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 103/153 2023-10-16T14:14:12.778 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 103/153 2023-10-16T14:14:12.956 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-10-16T14:14:13.301 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-16T14:14:13.339 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-16T14:14:13.377 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-16T14:14:13.433 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 103/153 2023-10-16T14:14:13.486 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 104/153 2023-10-16T14:14:13.529 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-16T14:14:13.669 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 104/153 2023-10-16T14:14:13.669 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-16T14:14:13.670 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-16T14:14:13.670 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:13.705 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 105/153 2023-10-16T14:14:13.971 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-16T14:14:14.013 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 6/153 2023-10-16T14:14:14.168 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 6/153 2023-10-16T14:14:14.234 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 7/153 2023-10-16T14:14:14.356 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 7/153 2023-10-16T14:14:14.429 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:18.0.0-6730.g4ba8a190.el9.x86_64 8/153 2023-10-16T14:14:14.471 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-16T14:14:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-16T14:14:14.783 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-16T14:14:14.809 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 12/153 2023-10-16T14:14:14.956 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 12/153 2023-10-16T14:14:14.993 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86 13/153 2023-10-16T14:14:15.093 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86 13/153 2023-10-16T14:14:15.481 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-16T14:14:15.632 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T14:14:15.720 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T14:14:15.937 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T14:14:15.988 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T14:14:16.418 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T14:14:16.458 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-16T14:14:16.509 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-16T14:14:16.620 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-16T14:14:16.689 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-16T14:14:16.729 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-16T14:14:16.784 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-16T14:14:16.892 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-16T14:14:16.940 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-16T14:14:16.982 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-16T14:14:17.120 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-16T14:14:17.162 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el9. 27/153 2023-10-16T14:14:17.212 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 28/153 2023-10-16T14:14:17.319 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:18.0.0-6730.g4ba8a190.el9.x86_6 29/153 2023-10-16T14:14:17.500 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-16T14:14:17.572 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-16T14:14:18.291 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-16T14:14:18.385 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-16T14:14:18.431 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-16T14:14:18.470 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:18.0.0-6730.g4ba8a190.el9.x86_6 35/153 2023-10-16T14:14:18.509 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-16T14:14:18.659 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-16T14:14:18.876 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-16T14:14:18.965 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-16T14:14:19.078 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-16T14:14:19.122 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-16T14:14:19.161 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-16T14:14:19.230 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-16T14:14:19.286 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-16T14:14:19.345 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-16T14:14:19.527 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-16T14:14:19.569 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-16T14:14:19.614 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-16T14:14:19.745 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-16T14:14:19.840 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 50/153 2023-10-16T14:14:19.909 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-16T14:14:19.945 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-16T14:14:19.970 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-16T14:14:20.009 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-16T14:14:20.061 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-16T14:14:20.217 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-16T14:14:20.278 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-16T14:14:20.359 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-16T14:14:20.448 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-16T14:14:20.714 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-16T14:14:20.780 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-16T14:14:20.957 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-16T14:14:21.016 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-16T14:14:21.065 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-16T14:14:21.106 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-16T14:14:21.159 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-16T14:14:21.208 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-16T14:14:21.261 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-16T14:14:21.344 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-16T14:14:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-16T14:14:21.466 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-16T14:14:21.599 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-16T14:14:21.638 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-16T14:14:21.674 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-16T14:14:21.738 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-16T14:14:21.777 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-16T14:14:21.948 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-16T14:14:21.991 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-16T14:14:22.134 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-16T14:14:22.209 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-16T14:14:23.216 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-16T14:14:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-16T14:14:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-16T14:14:23.456 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-16T14:14:23.508 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-16T14:14:23.741 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-16T14:14:23.789 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-16T14:14:23.883 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-16T14:14:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-16T14:14:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-16T14:14:26.424 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-16T14:14:26.518 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-16T14:14:26.556 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-16T14:14:26.587 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-16T14:14:26.628 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-16T14:14:27.010 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-16T14:14:27.067 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 97/153 2023-10-16T14:14:27.187 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 97/153 2023-10-16T14:14:27.223 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 98/153 2023-10-16T14:14:27.279 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-16T14:14:27.773 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-16T14:14:27.803 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 101/153 2023-10-16T14:14:27.903 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 101/153 2023-10-16T14:14:27.930 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 102/153 2023-10-16T14:14:29.998 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 105/153 2023-10-16T14:14:29.999 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-10-16T14:14:29.999 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-10-16T14:14:29.999 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-10-16T14:14:29.999 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-10-16T14:14:30.000 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-10-16T14:14:30.000 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2023-10-16T14:14:30.000 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-10-16T14:14:30.000 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:30.225 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 106/153 2023-10-16T14:14:30.273 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 106/153 2023-10-16T14:14:30.273 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:30.273 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-16T14:14:30.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:30.911 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 103/153 2023-10-16T14:14:30.952 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 103/153 2023-10-16T14:14:31.567 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 103/153 2023-10-16T14:14:31.642 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 104/153 2023-10-16T14:14:31.822 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 104/153 2023-10-16T14:14:31.822 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-16T14:14:31.822 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-16T14:14:31.823 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:31.864 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 105/153 2023-10-16T14:14:32.049 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 107/153 2023-10-16T14:14:32.136 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 107/153 2023-10-16T14:14:32.136 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:32.136 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-16T14:14:32.137 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T14:14:32.137 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T14:14:32.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:32.383 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 108/153 2023-10-16T14:14:32.460 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 108/153 2023-10-16T14:14:32.460 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:32.460 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-16T14:14:32.461 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T14:14:32.461 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T14:14:32.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:32.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 109/153 2023-10-16T14:14:33.070 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 109/153 2023-10-16T14:14:33.071 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:33.071 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-16T14:14:33.071 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T14:14:33.072 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T14:14:33.072 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:33.795 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 110/153 2023-10-16T14:14:33.836 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 110/153 2023-10-16T14:14:33.885 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 111/153 2023-10-16T14:14:34.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 112/153 2023-10-16T14:14:34.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noa 113/153 2023-10-16T14:14:34.380 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noa 113/153 2023-10-16T14:14:34.425 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.n 114/153 2023-10-16T14:14:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.n 114/153 2023-10-16T14:14:35.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 115/153 2023-10-16T14:14:35.793 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 115/153 2023-10-16T14:14:35.968 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 116/153 2023-10-16T14:14:35.996 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 117/153 2023-10-16T14:14:36.102 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 117/153 2023-10-16T14:14:36.103 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:36.103 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-16T14:14:36.103 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T14:14:36.103 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T14:14:36.103 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:36.157 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 118/153 2023-10-16T14:14:36.200 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 118/153 2023-10-16T14:14:37.117 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:18.0.0-6730.g4ba8a190.el9.x86_64 119/153 2023-10-16T14:14:37.151 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 120/153 2023-10-16T14:14:37.254 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 120/153 2023-10-16T14:14:37.254 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:37.255 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-16T14:14:37.255 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T14:14:37.255 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T14:14:37.255 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:37.302 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 121/153 2023-10-16T14:14:37.406 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 121/153 2023-10-16T14:14:37.406 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:37.407 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-16T14:14:37.407 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:37.479 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 122/153 2023-10-16T14:14:37.594 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 122/153 2023-10-16T14:14:37.594 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:37.595 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-16T14:14:37.595 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T14:14:37.595 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T14:14:37.595 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:37.980 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 123/153 2023-10-16T14:14:38.060 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 123/153 2023-10-16T14:14:38.060 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:38.060 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-16T14:14:38.060 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T14:14:38.061 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T14:14:38.061 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:42.409 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:18.0.0-6730.g4ba8a190.el9.x86_64 124/153 2023-10-16T14:14:42.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 125/153 2023-10-16T14:14:42.507 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 126/153 2023-10-16T14:14:42.801 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-16T14:14:42.943 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-16T14:14:43.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 129/153 2023-10-16T14:14:43.060 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-16T14:14:43.106 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-16T14:14:43.137 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el9.x86_ 132/153 2023-10-16T14:14:43.182 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-16T14:14:43.296 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-16T14:14:43.364 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-16T14:14:43.403 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-16T14:14:44.405 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T14:14:44.461 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T14:14:44.534 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T14:14:44.573 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 105/153 2023-10-16T14:14:44.573 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2023-10-16T14:14:44.573 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2023-10-16T14:14:44.573 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2023-10-16T14:14:44.574 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2023-10-16T14:14:44.574 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2023-10-16T14:14:44.574 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2023-10-16T14:14:44.574 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2023-10-16T14:14:44.574 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:44.618 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:18.0.0-6730.g4ba8a190.el9.noarch 138/153 2023-10-16T14:14:44.794 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 106/153 2023-10-16T14:14:44.847 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 106/153 2023-10-16T14:14:44.847 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:44.847 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-16T14:14:44.848 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:46.152 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-16T14:14:46.274 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-16T14:14:46.335 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-16T14:14:46.470 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-16T14:14:46.513 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-16T14:14:46.553 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-16T14:14:46.553 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T14:14:46.569 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 107/153 2023-10-16T14:14:46.652 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 107/153 2023-10-16T14:14:46.652 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:46.652 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-16T14:14:46.653 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T14:14:46.653 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T14:14:46.653 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T14:14:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-16T14:14:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-16T14:14:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-16T14:14:46.877 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T14:14:46.878 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T14:14:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 108/153 2023-10-16T14:14:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 108/153 2023-10-16T14:14:46.980 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:46.981 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-16T14:14:46.981 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T14:14:46.982 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T14:14:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:47.230 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T14:14:47.231 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T14:14:47.361 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T14:14:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T14:14:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T14:14:47.479 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 109/153 2023-10-16T14:14:47.541 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T14:14:47.550 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 109/153 2023-10-16T14:14:47.550 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:47.550 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-16T14:14:47.551 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T14:14:47.551 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T14:14:47.551 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:47.679 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T14:14:47.948 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T14:14:47.949 INFO:teuthology.orchestra.run.smithi029.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-16T14:14:47.949 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:48.239 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 110/153 2023-10-16T14:14:48.273 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 110/153 2023-10-16T14:14:48.312 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 111/153 2023-10-16T14:14:48.414 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 112/153 2023-10-16T14:14:48.442 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noa 113/153 2023-10-16T14:14:48.760 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noa 113/153 2023-10-16T14:14:48.794 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.n 114/153 2023-10-16T14:14:49.943 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.n 114/153 2023-10-16T14:14:49.977 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 115/153 2023-10-16T14:14:50.120 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 115/153 2023-10-16T14:14:50.293 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 116/153 2023-10-16T14:14:50.321 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 117/153 2023-10-16T14:14:50.397 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 117/153 2023-10-16T14:14:50.397 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:50.398 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-16T14:14:50.398 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T14:14:50.398 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T14:14:50.398 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:50.447 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 118/153 2023-10-16T14:14:50.486 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 118/153 2023-10-16T14:14:51.368 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:18.0.0-6730.g4ba8a190.el9.x86_64 119/153 2023-10-16T14:14:51.414 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 120/153 2023-10-16T14:14:51.484 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 120/153 2023-10-16T14:14:51.484 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:51.484 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-16T14:14:51.484 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T14:14:51.485 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T14:14:51.485 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:51.532 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 121/153 2023-10-16T14:14:51.605 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 121/153 2023-10-16T14:14:51.605 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:51.606 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-16T14:14:51.606 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:51.674 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 122/153 2023-10-16T14:14:51.752 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 122/153 2023-10-16T14:14:51.753 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:51.753 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-16T14:14:51.753 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T14:14:51.753 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T14:14:51.754 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:52.091 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 123/153 2023-10-16T14:14:52.164 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 123/153 2023-10-16T14:14:52.164 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T14:14:52.165 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-16T14:14:52.165 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T14:14:52.165 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T14:14:52.165 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:14:56.156 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:18.0.0-6730.g4ba8a190.el9.x86_64 1/153 2023-10-16T14:14:56.157 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 2/153 2023-10-16T14:14:56.157 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 3/153 2023-10-16T14:14:56.157 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 4/153 2023-10-16T14:14:56.157 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 5/153 2023-10-16T14:14:56.157 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 6/153 2023-10-16T14:14:56.158 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 7/153 2023-10-16T14:14:56.158 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 8/153 2023-10-16T14:14:56.158 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 9/153 2023-10-16T14:14:56.158 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 10/153 2023-10-16T14:14:56.158 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 11/153 2023-10-16T14:14:56.159 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:18.0.0-6730.g4ba8a190.el9.x86_64 12/153 2023-10-16T14:14:56.159 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 13/153 2023-10-16T14:14:56.159 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el9.x86_ 14/153 2023-10-16T14:14:56.159 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 15/153 2023-10-16T14:14:56.159 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 16/153 2023-10-16T14:14:56.160 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:18.0.0-6730.g4ba8a190.el9.x86_6 17/153 2023-10-16T14:14:56.160 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86 18/153 2023-10-16T14:14:56.160 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 19/153 2023-10-16T14:14:56.160 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 20/153 2023-10-16T14:14:56.160 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el9. 21/153 2023-10-16T14:14:56.161 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:18.0.0-6730.g4ba8a190.el9.x86_6 22/153 2023-10-16T14:14:56.161 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:18.0.0-6730.g4ba8a190.el9.x86_64 23/153 2023-10-16T14:14:56.161 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 24/153 2023-10-16T14:14:56.161 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 25/153 2023-10-16T14:14:56.172 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 26/153 2023-10-16T14:14:56.172 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 27/153 2023-10-16T14:14:56.173 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 28/153 2023-10-16T14:14:56.173 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 29/153 2023-10-16T14:14:56.173 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noa 30/153 2023-10-16T14:14:56.173 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.n 31/153 2023-10-16T14:14:56.174 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 32/153 2023-10-16T14:14:56.174 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 33/153 2023-10-16T14:14:56.174 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 34/153 2023-10-16T14:14:56.174 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 35/153 2023-10-16T14:14:56.174 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 36/153 2023-10-16T14:14:56.175 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 37/153 2023-10-16T14:14:56.175 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:18.0.0-6730.g4ba8a190.el9.noarch 38/153 2023-10-16T14:14:56.175 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-16T14:14:56.175 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-16T14:14:56.175 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-16T14:14:56.175 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-16T14:14:56.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-16T14:14:56.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-16T14:14:56.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-16T14:14:56.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-16T14:14:56.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-16T14:14:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-16T14:14:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-16T14:14:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-16T14:14:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-16T14:14:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-16T14:14:56.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-16T14:14:56.178 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-16T14:14:56.178 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-16T14:14:56.178 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-16T14:14:56.178 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-16T14:14:56.178 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-16T14:14:56.179 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-16T14:14:56.179 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-16T14:14:56.179 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-16T14:14:56.179 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-16T14:14:56.179 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-16T14:14:56.179 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-16T14:14:56.180 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-16T14:14:56.180 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-16T14:14:56.180 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-16T14:14:56.180 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-16T14:14:56.180 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-16T14:14:56.180 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-16T14:14:56.181 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-16T14:14:56.181 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-16T14:14:56.181 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-16T14:14:56.181 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-16T14:14:56.181 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-16T14:14:56.181 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-16T14:14:56.182 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-16T14:14:56.182 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-16T14:14:56.182 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-16T14:14:56.182 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-16T14:14:56.182 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-16T14:14:56.182 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-16T14:14:56.183 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-16T14:14:56.183 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-16T14:14:56.183 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-16T14:14:56.183 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-16T14:14:56.183 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-16T14:14:56.183 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-16T14:14:56.184 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-16T14:14:56.184 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-16T14:14:56.184 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-16T14:14:56.184 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-16T14:14:56.184 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-16T14:14:56.184 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-16T14:14:56.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-16T14:14:56.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-16T14:14:56.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-16T14:14:56.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-16T14:14:56.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-16T14:14:56.185 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-16T14:14:56.186 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-16T14:14:56.186 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-16T14:14:56.186 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-16T14:14:56.186 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-16T14:14:56.186 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-16T14:14:56.187 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-16T14:14:56.187 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-16T14:14:56.187 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 108/153 2023-10-16T14:14:56.187 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-16T14:14:56.187 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-16T14:14:56.187 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-16T14:14:56.188 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-16T14:14:56.188 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-16T14:14:56.188 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-16T14:14:56.188 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-16T14:14:56.188 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-16T14:14:56.188 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-16T14:14:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-16T14:14:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-16T14:14:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-16T14:14:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-16T14:14:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-16T14:14:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-16T14:14:56.190 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-16T14:14:56.190 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-16T14:14:56.190 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-16T14:14:56.190 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-16T14:14:56.190 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-16T14:14:56.191 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-16T14:14:56.191 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-16T14:14:56.191 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-16T14:14:56.191 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-16T14:14:56.191 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-16T14:14:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 134/153 2023-10-16T14:14:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-16T14:14:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 136/153 2023-10-16T14:14:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-16T14:14:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-16T14:14:56.192 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-16T14:14:56.193 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-16T14:14:56.193 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-16T14:14:56.193 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-16T14:14:56.193 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-16T14:14:56.193 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-16T14:14:56.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-16T14:14:56.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-16T14:14:56.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-16T14:14:56.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-16T14:14:56.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-16T14:14:56.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-16T14:14:56.195 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T14:14:56.195 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-16T14:14:56.221 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:18.0.0-6730.g4ba8a190.el9.x86_64 124/153 2023-10-16T14:14:56.266 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 125/153 2023-10-16T14:14:56.302 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 126/153 2023-10-16T14:14:56.564 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-16T14:14:56.686 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-16T14:14:56.755 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 129/153 2023-10-16T14:14:56.805 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-16T14:14:56.835 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-16T14:14:56.865 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el9.x86_ 132/153 2023-10-16T14:14:56.901 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-16T14:14:57.032 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-16T14:14:57.105 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-16T14:14:57.152 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-16T14:14:58.130 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T14:14:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T14:14:58.233 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T14:14:58.302 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-top-2:18.0.0-6730.g4ba8a190.el9.noarch 138/153 2023-10-16T14:14:58.323 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T14:14:58.323 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:58.323 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-10-16T14:14:58.324 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-16T14:14:58.324 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-16T14:14:58.324 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-16T14:14:58.324 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-16T14:14:58.324 INFO:teuthology.orchestra.run.smithi029.stdout: acl-2.3.1-4.el9.x86_64 2023-10-16T14:14:58.325 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-16T14:14:58.325 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-16T14:14:58.325 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.325 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.325 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-16T14:14:58.326 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-10-16T14:14:58.326 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2023-10-16T14:14:58.326 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2023-10-16T14:14:58.326 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-16T14:14:58.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.327 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.328 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.328 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.328 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.328 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.330 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.330 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.330 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.330 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.330 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.331 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.331 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.331 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.331 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:14:58.332 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-16T14:14:58.332 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2023-10-16T14:14:58.332 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-16T14:14:58.332 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-16T14:14:58.332 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-16T14:14:58.333 INFO:teuthology.orchestra.run.smithi029.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-16T14:14:58.333 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-16T14:14:58.333 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-16T14:14:58.333 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-15.el9.x86_64 2023-10-16T14:14:58.333 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-16T14:14:58.334 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-16T14:14:58.334 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-16T14:14:58.334 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.334 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.334 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.335 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-16T14:14:58.335 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-16T14:14:58.335 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-16T14:14:58.335 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-16T14:14:58.335 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-16T14:14:58.336 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.336 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.336 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-16T14:14:58.337 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.337 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-16T14:14:58.337 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-16T14:14:58.337 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-16T14:14:58.337 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-16T14:14:58.338 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-16T14:14:58.338 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-16T14:14:58.338 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-16T14:14:58.338 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-16T14:14:58.338 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-16T14:14:58.339 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-16T14:14:58.339 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.11-1.el9.x86_64 2023-10-16T14:14:58.339 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-16T14:14:58.339 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-16T14:14:58.340 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-16T14:14:58.340 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-16T14:14:58.340 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-16T14:14:58.340 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-16T14:14:58.340 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-16T14:14:58.341 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.341 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.341 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.341 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-16T14:14:58.341 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-16T14:14:58.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-16T14:14:58.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-16T14:14:58.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-16T14:14:58.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-16T14:14:58.342 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-16T14:14:58.343 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-16T14:14:58.343 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-16T14:14:58.343 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-16T14:14:58.343 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-16T14:14:58.343 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-16T14:14:58.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-16T14:14:58.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-16T14:14:58.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-16T14:14:58.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-16T14:14:58.345 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-16T14:14:58.345 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-16T14:14:58.345 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-16T14:14:58.345 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-16T14:14:58.345 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-16T14:14:58.346 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-16T14:14:58.346 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-16T14:14:58.346 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-16T14:14:58.346 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-16T14:14:58.346 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-16T14:14:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-16T14:14:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2023-10-16T14:14:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-16T14:14:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-16T14:14:58.347 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-16T14:14:58.348 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-16T14:14:58.348 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-16T14:14:58.348 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-16T14:14:58.348 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-16T14:14:58.348 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.349 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.349 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-16T14:14:58.349 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-16T14:14:58.349 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-16T14:14:58.349 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-16T14:14:58.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-16T14:14:58.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-16T14:14:58.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-16T14:14:58.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-16T14:14:58.350 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-16T14:14:58.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-16T14:14:58.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-16T14:14:58.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-16T14:14:58.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-16T14:14:58.351 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-16T14:14:58.352 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.352 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.352 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:14:58.352 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2023-10-16T14:14:58.352 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-16T14:14:58.353 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-16T14:14:58.353 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-16T14:14:58.353 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-16T14:14:58.353 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-16T14:14:58.354 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-16T14:14:58.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:14:58.354 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-10-16T14:14:59.055 DEBUG:teuthology.parallel:result is None 2023-10-16T14:14:59.690 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-16T14:14:59.807 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-16T14:14:59.851 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-16T14:14:59.976 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-16T14:15:00.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-16T14:15:00.055 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-16T14:15:00.056 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T14:15:00.242 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T14:15:00.273 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-16T14:15:00.305 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-16T14:15:00.338 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-16T14:15:00.376 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T14:15:00.377 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T14:15:00.704 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T14:15:00.705 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T14:15:00.844 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T14:15:00.874 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T14:15:00.875 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T14:15:01.046 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T14:15:01.170 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T14:15:01.433 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T14:15:01.433 INFO:teuthology.orchestra.run.smithi170.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-16T14:15:01.433 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:18.0.0-6730.g4ba8a190.el9.x86_64 1/153 2023-10-16T14:15:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 2/153 2023-10-16T14:15:06.117 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 3/153 2023-10-16T14:15:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 4/153 2023-10-16T14:15:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a 5/153 2023-10-16T14:15:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 6/153 2023-10-16T14:15:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 7/153 2023-10-16T14:15:06.118 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 8/153 2023-10-16T14:15:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 9/153 2023-10-16T14:15:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 10/153 2023-10-16T14:15:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 11/153 2023-10-16T14:15:06.119 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:18.0.0-6730.g4ba8a190.el9.x86_64 12/153 2023-10-16T14:15:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 13/153 2023-10-16T14:15:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:18.0.0-6730.g4ba8a190.el9.x86_ 14/153 2023-10-16T14:15:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 15/153 2023-10-16T14:15:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 16/153 2023-10-16T14:15:06.120 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:18.0.0-6730.g4ba8a190.el9.x86_6 17/153 2023-10-16T14:15:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86 18/153 2023-10-16T14:15:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 19/153 2023-10-16T14:15:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el 20/153 2023-10-16T14:15:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:18.0.0-6730.g4ba8a190.el9. 21/153 2023-10-16T14:15:06.121 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:18.0.0-6730.g4ba8a190.el9.x86_6 22/153 2023-10-16T14:15:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:18.0.0-6730.g4ba8a190.el9.x86_64 23/153 2023-10-16T14:15:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 24/153 2023-10-16T14:15:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 25/153 2023-10-16T14:15:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 26/153 2023-10-16T14:15:06.122 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 27/153 2023-10-16T14:15:06.134 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 28/153 2023-10-16T14:15:06.134 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190. 29/153 2023-10-16T14:15:06.134 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noa 30/153 2023-10-16T14:15:06.134 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.n 31/153 2023-10-16T14:15:06.134 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba 32/153 2023-10-16T14:15:06.135 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el 33/153 2023-10-16T14:15:06.135 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 34/153 2023-10-16T14:15:06.135 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.e 35/153 2023-10-16T14:15:06.135 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 36/153 2023-10-16T14:15:06.135 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 37/153 2023-10-16T14:15:06.136 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-top-2:18.0.0-6730.g4ba8a190.el9.noarch 38/153 2023-10-16T14:15:06.136 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-16T14:15:06.136 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-16T14:15:06.136 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-16T14:15:06.136 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-16T14:15:06.137 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-16T14:15:06.137 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-16T14:15:06.137 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-16T14:15:06.137 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-16T14:15:06.137 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-16T14:15:06.138 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-16T14:15:06.138 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-16T14:15:06.138 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-16T14:15:06.138 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-16T14:15:06.138 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-16T14:15:06.139 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-16T14:15:06.139 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-16T14:15:06.139 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-16T14:15:06.139 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-16T14:15:06.139 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-16T14:15:06.140 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-16T14:15:06.140 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-16T14:15:06.140 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-16T14:15:06.140 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-16T14:15:06.140 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-16T14:15:06.141 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-16T14:15:06.141 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-16T14:15:06.141 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-16T14:15:06.141 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-16T14:15:06.141 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-16T14:15:06.142 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-16T14:15:06.142 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-16T14:15:06.142 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-16T14:15:06.142 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-16T14:15:06.142 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-16T14:15:06.143 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-16T14:15:06.143 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-16T14:15:06.143 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-16T14:15:06.143 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-16T14:15:06.143 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-16T14:15:06.143 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-16T14:15:06.144 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-16T14:15:06.144 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-16T14:15:06.144 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-16T14:15:06.144 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-16T14:15:06.144 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-16T14:15:06.145 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-16T14:15:06.145 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-16T14:15:06.145 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-16T14:15:06.145 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-16T14:15:06.145 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-16T14:15:06.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-16T14:15:06.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-16T14:15:06.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-16T14:15:06.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-16T14:15:06.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-16T14:15:06.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-16T14:15:06.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-16T14:15:06.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-16T14:15:06.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-16T14:15:06.147 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-16T14:15:06.148 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-16T14:15:06.148 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-16T14:15:06.148 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-16T14:15:06.148 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-16T14:15:06.148 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-16T14:15:06.148 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-16T14:15:06.149 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-16T14:15:06.149 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-16T14:15:06.149 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-16T14:15:06.149 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 108/153 2023-10-16T14:15:06.149 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-16T14:15:06.150 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-16T14:15:06.150 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-16T14:15:06.150 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-16T14:15:06.150 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-16T14:15:06.150 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-16T14:15:06.151 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-16T14:15:06.151 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-16T14:15:06.151 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-16T14:15:06.151 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-16T14:15:06.151 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-16T14:15:06.152 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-16T14:15:06.152 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-16T14:15:06.152 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-16T14:15:06.152 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-16T14:15:06.152 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-16T14:15:06.153 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-16T14:15:06.153 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-16T14:15:06.153 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-16T14:15:06.153 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-16T14:15:06.153 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-16T14:15:06.154 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-16T14:15:06.154 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-16T14:15:06.154 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-16T14:15:06.154 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-16T14:15:06.154 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 134/153 2023-10-16T14:15:06.155 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-16T14:15:06.155 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 136/153 2023-10-16T14:15:06.155 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-16T14:15:06.155 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-16T14:15:06.155 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-16T14:15:06.155 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-16T14:15:06.156 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-16T14:15:06.156 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-16T14:15:06.156 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-16T14:15:06.156 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-16T14:15:06.156 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-16T14:15:06.157 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-16T14:15:06.157 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-16T14:15:06.157 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-16T14:15:06.157 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-16T14:15:06.157 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-16T14:15:06.157 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T14:15:06.158 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-16T14:15:08.343 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T14:15:08.343 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:08.343 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-10-16T14:15:08.344 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-16T14:15:08.344 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-16T14:15:08.344 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-16T14:15:08.344 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-16T14:15:08.344 INFO:teuthology.orchestra.run.smithi170.stdout: acl-2.3.1-4.el9.x86_64 2023-10-16T14:15:08.345 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-16T14:15:08.345 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-16T14:15:08.345 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.345 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.345 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-16T14:15:08.346 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-10-16T14:15:08.346 INFO:teuthology.orchestra.run.smithi170.stdout: automake-1.16.2-8.el9.noarch 2023-10-16T14:15:08.346 INFO:teuthology.orchestra.run.smithi170.stdout: bison-3.7.4-5.el9.x86_64 2023-10-16T14:15:08.346 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-16T14:15:08.346 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.347 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.347 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.347 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.347 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.348 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.348 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.348 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.348 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.348 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.349 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.349 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.349 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.349 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.349 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.349 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.350 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.350 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.350 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.350 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.350 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.351 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.351 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top-2:18.0.0-6730.g4ba8a190.el9.noarch 2023-10-16T14:15:08.351 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-16T14:15:08.351 INFO:teuthology.orchestra.run.smithi170.stdout: flex-2.6.4-9.el9.x86_64 2023-10-16T14:15:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-16T14:15:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-16T14:15:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-16T14:15:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-16T14:15:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-16T14:15:08.352 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-16T14:15:08.353 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-15.el9.x86_64 2023-10-16T14:15:08.353 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-16T14:15:08.353 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-16T14:15:08.353 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-16T14:15:08.353 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.354 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.354 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.354 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-16T14:15:08.354 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-16T14:15:08.354 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-16T14:15:08.355 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-16T14:15:08.355 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-16T14:15:08.355 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.355 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.355 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-16T14:15:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-16T14:15:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-16T14:15:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-16T14:15:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-16T14:15:08.356 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-16T14:15:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-16T14:15:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-16T14:15:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-16T14:15:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-16T14:15:08.357 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-16T14:15:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-13.11-1.el9.x86_64 2023-10-16T14:15:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-16T14:15:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-16T14:15:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-16T14:15:08.358 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-16T14:15:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-16T14:15:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-16T14:15:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-16T14:15:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.359 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.360 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.360 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-16T14:15:08.360 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-16T14:15:08.360 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-16T14:15:08.360 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-16T14:15:08.361 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-16T14:15:08.361 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-16T14:15:08.361 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-16T14:15:08.361 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-16T14:15:08.361 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-16T14:15:08.361 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-16T14:15:08.362 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-16T14:15:08.362 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-16T14:15:08.362 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-16T14:15:08.362 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-16T14:15:08.363 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-16T14:15:08.363 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-16T14:15:08.363 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-16T14:15:08.363 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-16T14:15:08.363 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-16T14:15:08.364 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-16T14:15:08.364 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-16T14:15:08.364 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-16T14:15:08.364 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-16T14:15:08.364 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-16T14:15:08.365 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-16T14:15:08.365 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-16T14:15:08.365 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-16T14:15:08.365 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2023-10-16T14:15:08.365 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-16T14:15:08.366 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-16T14:15:08.366 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-16T14:15:08.366 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-16T14:15:08.366 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-16T14:15:08.366 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-16T14:15:08.367 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-16T14:15:08.367 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.367 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.367 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-16T14:15:08.367 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-16T14:15:08.368 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-16T14:15:08.368 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.368 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-16T14:15:08.368 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-16T14:15:08.368 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-16T14:15:08.368 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-16T14:15:08.369 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-16T14:15:08.369 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-16T14:15:08.369 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-16T14:15:08.369 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-16T14:15:08.369 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-16T14:15:08.370 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-16T14:15:08.370 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-16T14:15:08.370 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.370 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.370 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:18.0.0-6730.g4ba8a190.el9.x86_64 2023-10-16T14:15:08.371 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2023-10-16T14:15:08.371 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-16T14:15:08.371 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-16T14:15:08.371 INFO:teuthology.orchestra.run.smithi170.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-16T14:15:08.371 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-16T14:15:08.372 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-16T14:15:08.372 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-16T14:15:08.373 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:08.373 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-10-16T14:15:08.918 DEBUG:teuthology.parallel:result is None 2023-10-16T14:15:08.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T14:15:09.142 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T14:15:09.222 INFO:teuthology.orchestra.run.smithi029.stdout:18.0.0-6730.g4ba8a190.el9 2023-10-16T14:15:09.222 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6730.g4ba8a190.el9 2023-10-16T14:15:09.223 INFO:teuthology.task.install:The correct ceph version 18.0.0-6730.g4ba8a190 is installed. 2023-10-16T14:15:09.223 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T14:15:09.447 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T14:15:09.505 INFO:teuthology.orchestra.run.smithi170.stdout:18.0.0-6730.g4ba8a190.el9 2023-10-16T14:15:09.505 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6730.g4ba8a190.el9 2023-10-16T14:15:09.506 INFO:teuthology.task.install:The correct ceph version 18.0.0-6730.g4ba8a190 is installed. 2023-10-16T14:15:09.507 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-16T14:15:09.508 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:09.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T14:15:09.613 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:09.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T14:15:09.695 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-16T14:15:09.695 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:09.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T14:15:09.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T14:15:09.905 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:09.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T14:15:09.988 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T14:15:10.102 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-16T14:15:10.102 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:10.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T14:15:10.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T14:15:10.354 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:10.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T14:15:10.445 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T14:15:10.561 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-16T14:15:10.561 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:10.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T14:15:10.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T14:15:10.773 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:10.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T14:15:10.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T14:15:10.970 INFO:teuthology.run_tasks:Running task ceph... 2023-10-16T14:15:11.041 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-16T14:15:11.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2023-10-16T14:15:11.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2023-10-16T14:15:11.152 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-16T14:15:11.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T14:15:11.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T14:15:11.248 INFO:tasks.ceph:Creating extra log directories... 2023-10-16T14:15:11.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T14:15:11.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T14:15:11.349 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-16T14:15:11.349 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-16T14:15:11.350 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-16T14:15:11.350 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:11.350 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T14:15:11.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T14:15:11.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T14:15:11.798 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:11.799 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T14:15:11.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T14:15:12.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T14:15:12.139 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-16T14:15:12.139 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-16T14:15:12.140 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037', 'branch': 'rishabh-2023oct13', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': None, 'first_in_suite': False, 'job_id': '7429037', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-2023oct13'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-2023oct13-b2', 'sha1': '33be745ec4531df08cd1bb70b4c9c1f89be58078'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6006, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'sleep_before_teardown': 0, 'subset': '402/512', 'suite': 'fs', 'suite_branch': 'rishabh-2023oct13-b2', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_33be745ec4531df08cd1bb70b4c9c1f89be58078/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '33be745ec4531df08cd1bb70b4c9c1f89be58078', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLovRJaWv6bFJ6tOwzKzcWOli1s8H6A6heFn15YRWNmDHD1HBnGtgWuQuDz9pU15qe12OyQtW/A9zhZwnw3C9bg=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOBtmsYnh/Ohhod1nwajWDpTORU7ptsmilduWzGZaP1RVkfWpkWMiYu1Q5wefsop6hhXLZARMmrAzXyTGKqqYw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_12:17:50', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-16T14:15:12.141 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T14:15:12.150 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T14:15:12.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T14:15:12.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T14:15:12.309 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:12.310 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T14:15:12.367 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T14:15:12.368 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2023-10-16T14:15:12.422 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T14:15:12.423 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:12.423 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 633 Links: 1 2023-10-16T14:15:12.423 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:12.423 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:12.424 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-16 14:14:48.397547724 +0000 2023-10-16T14:15:12.424 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-16 14:12:16.836436293 +0000 2023-10-16T14:15:12.424 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-16 14:12:16.836436293 +0000 2023-10-16T14:15:12.424 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-16 14:12:16.836436293 +0000 2023-10-16T14:15:12.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T14:15:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-16T14:15:12.520 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-16T14:15:12.520 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000232161 s, 2.2 MB/s 2023-10-16T14:15:12.523 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T14:15:12.584 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2023-10-16T14:15:12.637 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T14:15:12.638 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:12.638 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 650 Links: 1 2023-10-16T14:15:12.638 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:12.638 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:12.639 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-16 14:14:48.397547724 +0000 2023-10-16T14:15:12.639 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-16 14:12:17.940418128 +0000 2023-10-16T14:15:12.639 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-16 14:12:17.940418128 +0000 2023-10-16T14:15:12.639 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-16 14:12:17.940418128 +0000 2023-10-16T14:15:12.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T14:15:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-16T14:15:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-16T14:15:12.731 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000217038 s, 2.4 MB/s 2023-10-16T14:15:12.734 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T14:15:12.793 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2023-10-16T14:15:12.846 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T14:15:12.846 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:12.847 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 669 Links: 1 2023-10-16T14:15:12.847 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:12.847 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:12.847 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-16 14:14:48.398547708 +0000 2023-10-16T14:15:12.847 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-16 14:12:19.055399783 +0000 2023-10-16T14:15:12.848 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-16 14:12:19.055399783 +0000 2023-10-16T14:15:12.848 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-16 14:12:19.055399783 +0000 2023-10-16T14:15:12.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T14:15:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-16T14:15:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-16T14:15:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000259902 s, 2.0 MB/s 2023-10-16T14:15:12.947 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T14:15:13.010 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2023-10-16T14:15:13.063 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T14:15:13.063 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:13.063 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 690 Links: 1 2023-10-16T14:15:13.063 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:13.063 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:13.064 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-16 14:14:48.399547692 +0000 2023-10-16T14:15:13.064 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-16 14:12:20.095382671 +0000 2023-10-16T14:15:13.064 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-16 14:12:20.095382671 +0000 2023-10-16T14:15:13.064 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-16 14:12:20.095382671 +0000 2023-10-16T14:15:13.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T14:15:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-16T14:15:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-16T14:15:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000255303 s, 2.0 MB/s 2023-10-16T14:15:13.160 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T14:15:13.219 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:13.220 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:13.220 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T14:15:13.277 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T14:15:13.278 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2023-10-16T14:15:13.326 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T14:15:13.326 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:13.326 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 633 Links: 1 2023-10-16T14:15:13.327 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:13.327 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:13.327 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-16 14:15:01.902873578 +0000 2023-10-16T14:15:13.327 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-16 14:11:47.274573837 +0000 2023-10-16T14:15:13.327 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-16 14:11:47.274573837 +0000 2023-10-16T14:15:13.328 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-16 14:11:47.274573837 +0000 2023-10-16T14:15:13.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T14:15:13.415 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-16T14:15:13.415 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-16T14:15:13.415 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000297787 s, 1.7 MB/s 2023-10-16T14:15:13.418 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T14:15:13.472 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2023-10-16T14:15:13.516 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T14:15:13.517 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:13.517 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 653 Links: 1 2023-10-16T14:15:13.517 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:13.517 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:13.517 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-16 14:15:01.903873555 +0000 2023-10-16T14:15:13.518 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-16 14:11:48.284549742 +0000 2023-10-16T14:15:13.518 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-16 14:11:48.284549742 +0000 2023-10-16T14:15:13.518 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-16 14:11:48.284549742 +0000 2023-10-16T14:15:13.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T14:15:13.653 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-16T14:15:13.653 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-16T14:15:13.653 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000281298 s, 1.8 MB/s 2023-10-16T14:15:13.657 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T14:15:13.712 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2023-10-16T14:15:13.761 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T14:15:13.761 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:13.761 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 669 Links: 1 2023-10-16T14:15:13.762 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:13.762 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:13.762 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-16 14:15:01.904873532 +0000 2023-10-16T14:15:13.762 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-16 14:11:49.284525886 +0000 2023-10-16T14:15:13.763 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-16 14:11:49.284525886 +0000 2023-10-16T14:15:13.763 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-16 14:11:49.284525886 +0000 2023-10-16T14:15:13.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T14:15:13.847 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-16T14:15:13.847 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-16T14:15:13.848 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000235936 s, 2.2 MB/s 2023-10-16T14:15:13.851 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T14:15:13.909 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2023-10-16T14:15:13.954 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T14:15:13.954 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T14:15:13.954 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 689 Links: 1 2023-10-16T14:15:13.954 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T14:15:13.955 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T14:15:13.955 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-16 14:15:01.905873509 +0000 2023-10-16T14:15:13.955 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-16 14:11:50.280502125 +0000 2023-10-16T14:15:13.955 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-16 14:11:50.281502101 +0000 2023-10-16T14:15:13.955 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2023-10-16 14:11:50.280502125 +0000 2023-10-16T14:15:13.956 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T14:15:14.086 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-16T14:15:14.086 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-16T14:15:14.087 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00022017 s, 2.3 MB/s 2023-10-16T14:15:14.089 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T14:15:14.136 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:14.136 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T14:15:14.136 INFO:tasks.ceph:Generating config... 2023-10-16T14:15:14.138 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-16T14:15:14.138 INFO:tasks.ceph:[client] debug client = 20 2023-10-16T14:15:14.138 INFO:tasks.ceph:[client] debug ms = 1 2023-10-16T14:15:14.139 INFO:tasks.ceph:[client] pid file = 2023-10-16T14:15:14.139 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-16T14:15:14.139 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-16T14:15:14.140 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-16T14:15:14.140 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-16T14:15:14.140 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-16T14:15:14.140 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-16T14:15:14.141 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-16T14:15:14.141 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-16T14:15:14.141 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-16T14:15:14.142 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-16T14:15:14.142 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-16T14:15:14.142 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-16T14:15:14.142 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-16T14:15:14.143 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-16T14:15:14.143 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-16T14:15:14.143 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-16T14:15:14.144 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-16T14:15:14.144 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-16T14:15:14.144 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-16T14:15:14.144 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-16T14:15:14.145 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-16T14:15:14.145 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-16T14:15:14.145 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-16T14:15:14.146 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-16T14:15:14.146 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-16T14:15:14.146 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-16T14:15:14.146 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-16T14:15:14.147 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-16T14:15:14.147 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-16T14:15:14.147 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-16T14:15:14.148 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-16T14:15:14.148 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-16T14:15:14.148 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-16T14:15:14.148 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-16T14:15:14.149 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-16T14:15:14.149 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-16T14:15:14.149 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-16T14:15:14.150 INFO:tasks.ceph:Setting up mon.a... 2023-10-16T14:15:14.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-16T14:15:14.316 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2023-10-16T14:15:14.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-16T14:15:14.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T14:15:14.574 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29'), ('mon.b', '172.21.15.170'), ('mon.c', '[v2:172.21.15.170:3301,v1:172.21.15.170:6790]')] 2023-10-16T14:15:14.575 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29,172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-16T14:15:14.576 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:14.576 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T14:15:14.636 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --add b 172.21.15.170 --addv c '[v2:172.21.15.170:3301,v1:172.21.15.170:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:14.752 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:14.752 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid aff38114-d596-4938-a779-5e1b7478d87b 2023-10-16T14:15:14.753 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = pacific 2023-10-16T14:15:14.753 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2023-10-16T14:15:14.753 INFO:teuthology.orchestra.run.smithi029.stdout:fsid aff38114-d596-4938-a779-5e1b7478d87b 2023-10-16T14:15:14.753 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2023-10-16T14:15:14.750055+0000 2023-10-16T14:15:14.753 INFO:teuthology.orchestra.run.smithi029.stdout:created 2023-10-16T14:15:14.750055+0000 2023-10-16T14:15:14.754 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 16 (pacific) 2023-10-16T14:15:14.754 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2023-10-16T14:15:14.754 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-10-16T14:15:14.754 INFO:teuthology.orchestra.run.smithi029.stdout:1: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2023-10-16T14:15:14.755 INFO:teuthology.orchestra.run.smithi029.stdout:2: [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] mon.c 2023-10-16T14:15:14.755 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-16T14:15:14.756 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T14:15:14.813 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID aff38114-d596-4938-a779-5e1b7478d87b... 2023-10-16T14:15:14.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T14:15:14.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T14:15:14.995 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-10-16T14:15:14.995 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-10-16T14:15:14.995 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T14:15:14.995 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-10-16T14:15:14.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.996 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-10-16T14:15:14.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.996 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-10-16T14:15:14.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.996 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-10-16T14:15:14.997 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-10-16T14:15:14.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.997 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-10-16T14:15:14.997 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-10-16T14:15:14.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.998 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T14:15:14.998 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-10-16T14:15:14.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.998 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T14:15:14.998 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T14:15:14.999 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.999 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-10-16T14:15:14.999 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:14.999 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-10-16T14:15:14.999 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-10-16T14:15:14.999 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-10-16T14:15:15.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-10-16T14:15:15.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-10-16T14:15:15.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-10-16T14:15:15.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-10-16T14:15:15.000 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T14:15:15.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T14:15:15.001 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-10-16T14:15:15.001 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.001 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T14:15:15.001 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.001 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-10-16T14:15:15.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.002 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-10-16T14:15:15.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.002 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-10-16T14:15:15.002 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-10-16T14:15:15.003 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-10-16T14:15:15.003 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "172.21.15.29,172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790]" 2023-10-16T14:15:15.003 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on skipped message = False 2023-10-16T14:15:15.003 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = aff38114-d596-4938-a779-5e1b7478d87b 2023-10-16T14:15:15.003 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.004 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-10-16T14:15:15.004 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-10-16T14:15:15.004 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.004 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-10-16T14:15:15.005 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-10-16T14:15:15.005 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-10-16T14:15:15.005 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.005 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-10-16T14:15:15.005 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-10-16T14:15:15.006 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.006 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-10-16T14:15:15.006 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-10-16T14:15:15.006 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-10-16T14:15:15.006 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.007 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-10-16T14:15:15.007 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-10-16T14:15:15.007 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.007 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-10-16T14:15:15.007 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.007 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-10-16T14:15:15.008 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.008 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-10-16T14:15:15.008 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.008 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-10-16T14:15:15.008 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-10-16T14:15:15.009 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.009 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-10-16T14:15:15.009 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-10-16T14:15:15.009 INFO:teuthology.orchestra.run.smithi029.stdout: bdev async discard = True 2023-10-16T14:15:15.009 INFO:teuthology.orchestra.run.smithi029.stdout: bdev enable discard = True 2023-10-16T14:15:15.009 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = bitmap 2023-10-16T14:15:15.010 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-10-16T14:15:15.010 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-10-16T14:15:15.010 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1/20 2023-10-16T14:15:15.010 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1/20 2023-10-16T14:15:15.010 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-16T14:15:15.011 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-10-16T14:15:15.011 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 4/10 2023-10-16T14:15:15.011 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T14:15:15.011 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-10-16T14:15:15.011 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-10-16T14:15:15.011 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-10-16T14:15:15.012 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-10-16T14:15:15.012 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2023-10-16T14:15:15.012 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.012 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-10-16T14:15:15.012 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-16T14:15:15.013 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-10-16T14:15:15.013 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-10-16T14:15:15.013 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-10-16T14:15:15.013 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-10-16T14:15:15.013 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-10-16T14:15:15.013 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-10-16T14:15:15.014 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2023-10-16T14:15:15.014 INFO:teuthology.orchestra.run.smithi029.stdout: debug finisher = 20 2023-10-16T14:15:15.014 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.014 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-10-16T14:15:15.014 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-16T14:15:15.015 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-10-16T14:15:15.015 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-10-16T14:15:15.015 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-10-16T14:15:15.015 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-10-16T14:15:15.015 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-10-16T14:15:15.016 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-10-16T14:15:15.016 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-10-16T14:15:15.016 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-10-16T14:15:15.016 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-10-16T14:15:15.016 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.016 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T14:15:15.017 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T14:15:15.017 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T14:15:15.017 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.017 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T14:15:15.017 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T14:15:15.018 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T14:15:15.018 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.018 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-10-16T14:15:15.018 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-10-16T14:15:15.018 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.019 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-10-16T14:15:15.019 INFO:teuthology.orchestra.run.smithi029.stdout: mon op complaint time = 120 2023-10-16T14:15:15.019 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:15:15.019 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-10-16T14:15:15.019 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-10-16T14:15:15.019 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-10-16T14:15:15.020 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-10-16T14:15:15.020 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T14:15:15.020 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T14:15:15.020 INFO:teuthology.orchestra.run.smithi029.stdout: client mount timeout = 600 2023-10-16T14:15:15.020 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2023-10-16T14:15:15.021 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-16T14:15:15.021 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = "" 2023-10-16T14:15:15.021 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2023-10-16T14:15:15.021 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2023-10-16T14:15:15.021 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-10-16T14:15:15.021 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2023-10-16T14:15:15.022 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2023-10-16T14:15:15.022 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2023-10-16T14:15:15.022 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2023-10-16T14:15:15.022 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2023-10-16T14:15:15.022 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2023-10-16T14:15:15.023 INFO:teuthology.orchestra.run.smithi029.stdout:[mds] 2023-10-16T14:15:15.023 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds = 20 2023-10-16T14:15:15.023 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds balancer = 20 2023-10-16T14:15:15.023 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-16T14:15:15.023 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug frag = True 2023-10-16T14:15:15.023 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug scatterstat = True 2023-10-16T14:15:15.024 INFO:teuthology.orchestra.run.smithi029.stdout: mds op complaint time = 180 2023-10-16T14:15:15.024 INFO:teuthology.orchestra.run.smithi029.stdout: mds verify scatter = True 2023-10-16T14:15:15.024 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2023-10-16T14:15:15.024 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2023-10-16T14:15:15.024 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2023-10-16T14:15:15.025 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-10-16T14:15:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-10-16T14:15:15.025 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T14:15:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-10-16T14:15:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-10-16T14:15:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-10-16T14:15:15.026 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-10-16T14:15:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-10-16T14:15:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-10-16T14:15:15.027 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-10-16T14:15:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T14:15:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-10-16T14:15:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T14:15:15.028 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T14:15:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-10-16T14:15:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-10-16T14:15:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-10-16T14:15:15.029 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-10-16T14:15:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-10-16T14:15:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-10-16T14:15:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-10-16T14:15:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-10-16T14:15:15.030 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T14:15:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T14:15:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-10-16T14:15:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T14:15:15.031 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-10-16T14:15:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-10-16T14:15:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-10-16T14:15:15.032 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-10-16T14:15:15.033 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-10-16T14:15:15.033 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = "172.21.15.29,172.21.15.170,[v2:172.21.15.170:3301,v1:172.21.15.170:6790]" 2023-10-16T14:15:15.033 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on skipped message = False 2023-10-16T14:15:15.033 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = aff38114-d596-4938-a779-5e1b7478d87b 2023-10-16T14:15:15.033 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.033 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-10-16T14:15:15.034 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-10-16T14:15:15.034 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.034 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-10-16T14:15:15.034 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-10-16T14:15:15.034 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2023-10-16T14:15:15.035 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.035 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-10-16T14:15:15.035 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-10-16T14:15:15.035 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.035 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-10-16T14:15:15.036 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-10-16T14:15:15.036 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-10-16T14:15:15.036 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.036 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-10-16T14:15:15.036 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-10-16T14:15:15.036 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.037 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-10-16T14:15:15.037 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.037 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-10-16T14:15:15.037 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.037 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-10-16T14:15:15.038 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.038 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-10-16T14:15:15.038 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-10-16T14:15:15.038 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.038 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-10-16T14:15:15.039 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-10-16T14:15:15.039 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2023-10-16T14:15:15.039 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2023-10-16T14:15:15.039 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore allocator = bitmap 2023-10-16T14:15:15.039 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2023-10-16T14:15:15.039 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2023-10-16T14:15:15.040 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 1/20 2023-10-16T14:15:15.040 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 1/20 2023-10-16T14:15:15.040 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-16T14:15:15.040 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-10-16T14:15:15.040 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 4/10 2023-10-16T14:15:15.041 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T14:15:15.041 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2023-10-16T14:15:15.041 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2023-10-16T14:15:15.041 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2023-10-16T14:15:15.041 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2023-10-16T14:15:15.041 INFO:teuthology.orchestra.run.smithi170.stdout: osd op complaint time = 180 2023-10-16T14:15:15.042 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.042 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-10-16T14:15:15.042 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-16T14:15:15.042 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-10-16T14:15:15.042 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-16T14:15:15.043 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-16T14:15:15.043 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-16T14:15:15.043 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-16T14:15:15.043 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-10-16T14:15:15.043 INFO:teuthology.orchestra.run.smithi170.stdout: debug client = 20 2023-10-16T14:15:15.043 INFO:teuthology.orchestra.run.smithi170.stdout: debug finisher = 20 2023-10-16T14:15:15.044 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.044 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-10-16T14:15:15.044 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-16T14:15:15.044 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-16T14:15:15.044 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-10-16T14:15:15.045 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-16T14:15:15.045 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-10-16T14:15:15.045 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-10-16T14:15:15.045 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-16T14:15:15.045 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-10-16T14:15:15.045 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-10-16T14:15:15.046 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-16T14:15:15.046 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.046 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T14:15:15.046 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T14:15:15.046 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T14:15:15.047 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.047 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T14:15:15.047 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T14:15:15.047 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T14:15:15.047 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.047 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2023-10-16T14:15:15.048 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2023-10-16T14:15:15.048 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.048 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2023-10-16T14:15:15.048 INFO:teuthology.orchestra.run.smithi170.stdout: mon op complaint time = 120 2023-10-16T14:15:15.048 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-16T14:15:15.049 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-10-16T14:15:15.049 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-10-16T14:15:15.049 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-10-16T14:15:15.049 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-10-16T14:15:15.049 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T14:15:15.049 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T14:15:15.050 INFO:teuthology.orchestra.run.smithi170.stdout: client mount timeout = 600 2023-10-16T14:15:15.050 INFO:teuthology.orchestra.run.smithi170.stdout: debug client = 20 2023-10-16T14:15:15.050 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-16T14:15:15.050 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = "" 2023-10-16T14:15:15.050 INFO:teuthology.orchestra.run.smithi170.stdout: rados mon op timeout = 900 2023-10-16T14:15:15.051 INFO:teuthology.orchestra.run.smithi170.stdout: rados osd op timeout = 900 2023-10-16T14:15:15.051 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-10-16T14:15:15.051 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.b] 2023-10-16T14:15:15.051 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.c] 2023-10-16T14:15:15.051 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.a] 2023-10-16T14:15:15.051 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.c] 2023-10-16T14:15:15.052 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.b] 2023-10-16T14:15:15.052 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.d] 2023-10-16T14:15:15.052 INFO:teuthology.orchestra.run.smithi170.stdout:[mds] 2023-10-16T14:15:15.052 INFO:teuthology.orchestra.run.smithi170.stdout: debug mds = 20 2023-10-16T14:15:15.053 INFO:teuthology.orchestra.run.smithi170.stdout: debug mds balancer = 20 2023-10-16T14:15:15.053 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-16T14:15:15.053 INFO:teuthology.orchestra.run.smithi170.stdout: mds debug frag = True 2023-10-16T14:15:15.053 INFO:teuthology.orchestra.run.smithi170.stdout: mds debug scatterstat = True 2023-10-16T14:15:15.053 INFO:teuthology.orchestra.run.smithi170.stdout: mds op complaint time = 180 2023-10-16T14:15:15.054 INFO:teuthology.orchestra.run.smithi170.stdout: mds verify scatter = True 2023-10-16T14:15:15.054 INFO:teuthology.orchestra.run.smithi170.stdout: osd op complaint time = 180 2023-10-16T14:15:15.054 INFO:teuthology.orchestra.run.smithi170.stdout: rados mon op timeout = 900 2023-10-16T14:15:15.054 INFO:teuthology.orchestra.run.smithi170.stdout: rados osd op timeout = 900 2023-10-16T14:15:15.056 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-16T14:15:15.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-16T14:15:15.221 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-16T14:15:15.222 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:15.222 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-16T14:15:15.281 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:15.281 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-16T14:15:15.337 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:15:15.337 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:15.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T14:15:15.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T14:15:15.498 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:15.499 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:15.601 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:15:15.601 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:15.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T14:15:15.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T14:15:15.760 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:15.761 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:15.859 INFO:tasks.ceph:Setting up mon nodes... 2023-10-16T14:15:15.859 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-16T14:15:15.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T14:15:16.068 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T14:15:16.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T14:15:16.282 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T14:15:16.294 INFO:tasks.ceph:Setting up mds nodes... 2023-10-16T14:15:16.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T14:15:16.500 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T14:15:16.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-16T14:15:16.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T14:15:16.813 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T14:15:16.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-16T14:15:16.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T14:15:17.107 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T14:15:17.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-16T14:15:17.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T14:15:17.400 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T14:15:17.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-16T14:15:17.502 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-16T14:15:17.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-16T14:15:17.646 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-16T14:15:17.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-16T14:15:17.846 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-16T14:15:17.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-16T14:15:18.049 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-16T14:15:18.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-16T14:15:18.240 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-16T14:15:18.287 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-16T14:15:18.287 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T14:15:18.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-16T14:15:18.396 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:18.397 INFO:tasks.ceph:role: osd.0 2023-10-16T14:15:18.397 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:15:18.397 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T14:15:18.499 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:18.500 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:18.500 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:18.500 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:18.500 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:18.501 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:18.501 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:18.501 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:18.501 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:18.501 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:18.534 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-16T14:15:18.539 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-16T14:15:18.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-16T14:15:18.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-16T14:15:18.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-16T14:15:18.820 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:18.821 INFO:tasks.ceph:role: osd.1 2023-10-16T14:15:18.821 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:15:18.821 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T14:15:18.910 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:18.911 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:18.911 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:18.911 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:18.911 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:18.911 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:18.912 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:18.912 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-16T14:15:18.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-16T14:15:18.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-16T14:15:19.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-16T14:15:19.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-16T14:15:19.212 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:19.213 INFO:tasks.ceph:role: osd.2 2023-10-16T14:15:19.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:15:19.213 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T14:15:19.310 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:19.310 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:19.310 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:19.311 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:19.311 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:19.311 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:19.311 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:19.311 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:19.311 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:19.312 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:19.348 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-16T14:15:19.353 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-16T14:15:19.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-16T14:15:19.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-16T14:15:19.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-16T14:15:19.613 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:19.613 INFO:tasks.ceph:role: osd.3 2023-10-16T14:15:19.614 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:15:19.614 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T14:15:19.699 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:19.699 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:19.700 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:19.702 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:19.704 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:19.707 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:19.712 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:19.712 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:19.715 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:19.715 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:19.732 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-16T14:15:19.737 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-16T14:15:19.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-16T14:15:19.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-16T14:15:19.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:20.059+0000 7fce89a14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-16T14:15:20.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:20.060+0000 7fce89a14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-16T14:15:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:20.060+0000 7fce89a14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T14:15:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:20.060+0000 7fce89a14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T14:15:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:20.060+0000 7fce89a14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T14:15:20.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:20.062+0000 7fce89a14500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-16T14:15:23.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-16T14:15:23.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:23.496+0000 7ff087411500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-16T14:15:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:23.496+0000 7ff087411500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-16T14:15:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:23.496+0000 7ff087411500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T14:15:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:23.496+0000 7ff087411500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T14:15:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:23.496+0000 7ff087411500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T14:15:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:23.498+0000 7ff087411500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-16T14:15:26.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-16T14:15:26.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:26.927+0000 7ffa97e14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-16T14:15:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:26.927+0000 7ffa97e14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-16T14:15:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:26.928+0000 7ffa97e14500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T14:15:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:26.928+0000 7ffa97e14500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T14:15:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:26.928+0000 7ffa97e14500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T14:15:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:26.930+0000 7ffa97e14500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-16T14:15:30.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-16T14:15:30.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:30.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:30.361+0000 7f93da6a1500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-16T14:15:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:30.362+0000 7f93da6a1500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-16T14:15:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:30.362+0000 7f93da6a1500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T14:15:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:30.362+0000 7f93da6a1500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T14:15:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:30.362+0000 7f93da6a1500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T14:15:30.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:30.364+0000 7f93da6a1500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-16T14:15:33.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-16T14:15:33.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-16T14:15:34.001 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:34.002 INFO:tasks.ceph:role: osd.4 2023-10-16T14:15:34.002 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:15:34.002 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T14:15:34.098 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:34.098 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:34.098 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:34.098 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:34.099 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:34.099 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:34.099 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:34.099 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:34.099 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:34.100 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:34.135 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-16T14:15:34.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-16T14:15:34.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-16T14:15:34.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-16T14:15:34.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-16T14:15:34.426 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:34.427 INFO:tasks.ceph:role: osd.5 2023-10-16T14:15:34.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:15:34.427 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T14:15:34.505 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:34.505 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:34.506 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:34.506 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:34.507 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:34.507 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:34.507 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:34.507 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:34.544 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-16T14:15:34.549 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-16T14:15:34.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-16T14:15:34.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-16T14:15:34.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-16T14:15:34.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:34.807 INFO:tasks.ceph:role: osd.6 2023-10-16T14:15:34.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:15:34.808 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T14:15:34.884 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:34.884 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:34.884 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:34.884 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:34.885 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:34.885 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:34.885 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:34.885 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:34.885 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:34.885 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:34.926 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-16T14:15:34.933 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-16T14:15:34.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-16T14:15:35.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-16T14:15:35.109 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-16T14:15:35.189 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T14:15:35.189 INFO:tasks.ceph:role: osd.7 2023-10-16T14:15:35.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:15:35.190 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T14:15:35.267 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T14:15:35.268 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T14:15:35.268 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T14:15:35.268 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T14:15:35.268 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T14:15:35.269 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-16T14:15:35.269 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T14:15:35.269 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T14:15:35.269 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T14:15:35.269 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T14:15:35.309 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-16T14:15:35.314 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-16T14:15:35.315 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-16T14:15:35.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-16T14:15:35.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:35.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:35.641+0000 7fec74e69500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-16T14:15:35.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:35.641+0000 7fec74e69500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-16T14:15:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:35.641+0000 7fec74e69500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T14:15:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:35.641+0000 7fec74e69500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T14:15:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:35.641+0000 7fec74e69500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T14:15:35.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:35.644+0000 7fec74e69500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-16T14:15:38.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-16T14:15:38.668 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:38.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:38.797+0000 7f7ce08fa500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-16T14:15:38.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:38.797+0000 7f7ce08fa500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-16T14:15:38.801 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:38.797+0000 7f7ce08fa500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T14:15:38.801 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:38.797+0000 7f7ce08fa500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T14:15:38.801 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:38.797+0000 7f7ce08fa500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T14:15:38.802 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:38.799+0000 7f7ce08fa500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-16T14:15:41.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-16T14:15:42.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:42.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:15:42.151 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:15:42.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:42.209+0000 7f247ad2b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-16T14:15:42.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:42.210+0000 7f247ad2b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-16T14:15:42.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:42.210+0000 7f247ad2b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T14:15:42.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:42.210+0000 7f247ad2b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T14:15:42.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:42.210+0000 7f247ad2b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T14:15:42.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:42.212+0000 7f247ad2b500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-16T14:15:45.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-16T14:15:45.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:45.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:45.618+0000 7fef4081a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-16T14:15:45.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:45.618+0000 7fef4081a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-16T14:15:45.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:45.618+0000 7fef4081a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T14:15:45.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:45.618+0000 7fef4081a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T14:15:45.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:45.618+0000 7fef4081a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T14:15:45.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:15:45.620+0000 7fef4081a500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-16T14:15:48.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-16T14:15:48.891 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-16T14:15:48.891 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:48.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-16T14:15:49.003 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:49.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-16T14:15:49.112 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:49.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-16T14:15:49.215 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:49.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-16T14:15:49.318 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:49.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-16T14:15:49.424 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:49.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-16T14:15:49.528 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:49.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-16T14:15:49.632 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:49.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-16T14:15:49.734 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:49.734 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-16T14:15:49.830 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:49.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-16T14:15:49.925 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:49.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-16T14:15:50.023 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:50.023 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-16T14:15:50.121 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:50.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-16T14:15:50.217 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:50.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-16T14:15:50.314 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:50.315 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-16T14:15:50.379 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:15:50.379 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-16T14:15:50.440 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:50.440 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-16T14:15:50.498 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:15:50.498 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-16T14:15:50.551 INFO:tasks.ceph:Adding keys to all mons... 2023-10-16T14:15:50.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T14:15:50.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T14:15:50.655 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2023-10-16T14:15:50.656 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA== 2023-10-16T14:15:50.656 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2023-10-16T14:15:50.656 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg== 2023-10-16T14:15:50.656 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2023-10-16T14:15:50.657 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g== 2023-10-16T14:15:50.657 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2023-10-16T14:15:50.657 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA== 2023-10-16T14:15:50.657 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2023-10-16T14:15:50.657 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w== 2023-10-16T14:15:50.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2023-10-16T14:15:50.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw== 2023-10-16T14:15:50.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2023-10-16T14:15:50.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw== 2023-10-16T14:15:50.658 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2023-10-16T14:15:50.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw== 2023-10-16T14:15:50.659 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2023-10-16T14:15:50.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug== 2023-10-16T14:15:50.659 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2023-10-16T14:15:50.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ== 2023-10-16T14:15:50.660 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2023-10-16T14:15:50.660 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g== 2023-10-16T14:15:50.660 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2023-10-16T14:15:50.661 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw== 2023-10-16T14:15:50.661 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2023-10-16T14:15:50.661 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw== 2023-10-16T14:15:50.661 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2023-10-16T14:15:50.662 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g== 2023-10-16T14:15:50.662 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:15:50.662 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:15:50.663 INFO:teuthology.orchestra.run.smithi029.stdout:[client.2] 2023-10-16T14:15:50.663 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ== 2023-10-16T14:15:50.663 INFO:teuthology.orchestra.run.smithi029.stdout:[client.1] 2023-10-16T14:15:50.664 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw== 2023-10-16T14:15:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:[client.3] 2023-10-16T14:15:50.665 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg== 2023-10-16T14:15:50.667 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.y] 2023-10-16T14:15:50.667 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA== 2023-10-16T14:15:50.668 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.a] 2023-10-16T14:15:50.668 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg== 2023-10-16T14:15:50.668 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.c] 2023-10-16T14:15:50.669 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g== 2023-10-16T14:15:50.669 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-10-16T14:15:50.670 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA== 2023-10-16T14:15:50.670 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-10-16T14:15:50.670 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w== 2023-10-16T14:15:50.670 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-10-16T14:15:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw== 2023-10-16T14:15:50.671 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2023-10-16T14:15:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw== 2023-10-16T14:15:50.671 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2023-10-16T14:15:50.671 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw== 2023-10-16T14:15:50.672 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.b] 2023-10-16T14:15:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug== 2023-10-16T14:15:50.672 INFO:teuthology.orchestra.run.smithi170.stdout:[mds.d] 2023-10-16T14:15:50.672 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ== 2023-10-16T14:15:50.672 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.4] 2023-10-16T14:15:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g== 2023-10-16T14:15:50.673 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.5] 2023-10-16T14:15:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw== 2023-10-16T14:15:50.673 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.6] 2023-10-16T14:15:50.673 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw== 2023-10-16T14:15:50.674 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.7] 2023-10-16T14:15:50.674 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g== 2023-10-16T14:15:50.674 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2023-10-16T14:15:50.675 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:15:50.675 INFO:teuthology.orchestra.run.smithi170.stdout:[client.2] 2023-10-16T14:15:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ== 2023-10-16T14:15:50.676 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2023-10-16T14:15:50.676 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw== 2023-10-16T14:15:50.677 INFO:teuthology.orchestra.run.smithi170.stdout:[client.3] 2023-10-16T14:15:50.677 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg== 2023-10-16T14:15:50.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T14:15:50.686 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T14:15:50.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:50.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:50.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:51.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:51.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.468 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.626 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:51.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T14:15:51.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T14:15:51.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:51.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:52.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:52.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T14:15:52.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.539 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T14:15:52.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:52.846 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:52.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:53.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:53.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:53.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:53.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:53.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T14:15:53.486 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-16T14:15:53.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-16T14:15:53.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T14:15:53.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-16T14:15:53.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-16T14:15:53.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T14:15:54.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-16T14:15:54.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-16T14:15:54.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T14:15:54.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-16T14:15:54.515 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:54.525 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T14:15:54.580 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-16T14:15:54.580 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-16T14:15:54.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-16T14:15:54.587 INFO:tasks.ceph.mon.a:Started 2023-10-16T14:15:54.588 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-16T14:15:54.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-16T14:15:54.596 INFO:tasks.ceph.mon.b:Started 2023-10-16T14:15:54.597 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-16T14:15:54.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-16T14:15:54.605 INFO:tasks.ceph.mon.c:Started 2023-10-16T14:15:54.605 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-16T14:15:54.606 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T14:15:54.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T14:15:54.615 INFO:tasks.ceph.mgr.y:Started 2023-10-16T14:15:54.615 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T14:15:54.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T14:15:54.621 INFO:tasks.ceph.mgr.x:Started 2023-10-16T14:15:54.621 INFO:tasks.ceph:Setting crush tunables to default 2023-10-16T14:15:54.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2023-10-16T14:15:54.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.806+0000 7f733669e640 1 Processor -- start 2023-10-16T14:15:54.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.806+0000 7f733669e640 1 -- start start 2023-10-16T14:15:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.807+0000 7f733669e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:15:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.807+0000 7f733669e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:15:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.807+0000 7f733669e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300d0970 con 0x7f73300ce960 2023-10-16T14:15:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.807+0000 7f733669e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300d0ea0 con 0x7f73300b2080 2023-10-16T14:15:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.807+0000 7f733669e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300d0bc0 con 0x7f73300b1ca0 2023-10-16T14:15:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7334e9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f733569c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7335e9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f73300ce960 0x7f73300b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:43016/0 (socket says 172.21.15.29:43016) 2023-10-16T14:15:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7335e9d640 1 -- 172.21.15.29:0/2748017778 learned_addr learned my addr 172.21.15.29:0/2748017778 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:15:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T14:15:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:15:54.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:54.808+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:15:55.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.009+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:55.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.009+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:55.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T14:15:55.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:15:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.010+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:15:55.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.411+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:55.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.411+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:55.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T14:15:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:15:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:55.412+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:15:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.213+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T14:15:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:15:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:56.214+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:15:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.807+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 msgr2=0x7f73300d36b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T14:15:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.807+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300b1ca0 0x7f73300d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:15:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.807+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f73300ce420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T14:15:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.807+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f73300ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:15:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.807+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v1:172.21.15.170:6789/0 conn(0x7f73300ce960 legacy=0x7f73300b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:15:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.808+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:15:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.808+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300cd940 con 0x7f7314006200 2023-10-16T14:15:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.808+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300d0ea0 con 0x7f73140036a0 2023-10-16T14:15:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.808+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.809+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.809+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.809+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:57.809+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:15:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.010+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.011+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.011+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.011+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.011+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:15:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.412+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.413+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.413+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.413+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:58.413+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:15:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:59.215+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:15:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:59.215+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:15:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:59.215+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:15:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:59.215+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:15:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:15:59.215+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:16:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.808+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 msgr2=0x7f7314005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T14:16:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.808+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f73140036a0 0x7f7314005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.808+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v1:172.21.15.29:6789/0 conn(0x7f7314006200 legacy=0x7f73140086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:00.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.808+0000 7f7335e9d640 1 -- 172.21.15.29:0/2748017778 reap_dead start 2023-10-16T14:16:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.808+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731400af80 con 0x7f7314005b50 2023-10-16T14:16:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731400ab10 con 0x7f731400a540 2023-10-16T14:16:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300d0ea0 con 0x7f73300b2080 2023-10-16T14:16:00.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.809+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:00.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.810+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:00.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.810+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:00.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.810+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:00.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:00.810+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:16:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.011+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.012+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.012+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:01.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.012+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:01.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.012+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:16:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.413+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.414+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.414+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.414+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:01.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:01.414+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:16:02.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:02.215+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:02.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:02.216+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:02.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:02.216+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:02.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:02.216+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:02.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:02.216+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:16:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 msgr2=0x7f731400a000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T14:16:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.29:3300/0 conn(0x7f73300b2080 0x7f731400a000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v1:172.21.15.170:6789/0 conn(0x7f731400a540 legacy=0x7f731400d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 >> v1:172.21.15.29:6789/0 conn(0x7f7314005b50 legacy=0x7f73140086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300d0ea0 con 0x7f731400f650 2023-10-16T14:16:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.809+0000 7f7326ffd640 1 -- 172.21.15.29:0/2748017778 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300d0bc0 con 0x7f731400eb60 2023-10-16T14:16:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.810+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.810+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.810+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.810+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.810+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.811+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 msgr2=0x7f731400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T14:16:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.811+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 msgr2=0x7f731400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.811+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:03.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.811+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:16:03.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:03.811+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:16:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.012+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:16:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 msgr2=0x7f731400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T14:16:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 msgr2=0x7f731400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.013+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:16:04.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.414+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T14:16:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f733569c640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:16:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 msgr2=0x7f731400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T14:16:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 msgr2=0x7f731400fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T14:16:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T14:16:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:04.415+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:16:05.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.216+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> v2:172.21.15.170:3300/0 conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:05.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.217+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:05.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.217+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 msgr2=0x7f731400ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:05.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.217+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.217+0000 7f7334e9b640 1 -- 172.21.15.29:0/2748017778 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731400ad50 con 0x7f731400f650 2023-10-16T14:16:05.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.217+0000 7f733569c640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f731400eb60 0x7f731400ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:05.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.218+0000 7f7334e9b640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f731400fb00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7320006a90 tx=0x7f732002d050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc2afa4922836813 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.219+0000 7f7327fff640 1 -- 172.21.15.29:0/2748017778 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732002d970 con 0x7f731400f650 2023-10-16T14:16:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.219+0000 7f7327fff640 1 -- 172.21.15.29:0/2748017778 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7320031070 con 0x7f731400f650 2023-10-16T14:16:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.219+0000 7f7327fff640 1 -- 172.21.15.29:0/2748017778 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732002dcf0 con 0x7f731400f650 2023-10-16T14:16:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.220+0000 7f733669e640 1 -- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 msgr2=0x7f731400fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.220+0000 7f733669e640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f731400fb00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7320006a90 tx=0x7f732002d050 comp rx=0 tx=0).stop 2023-10-16T14:16:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.220+0000 7f7335e9d640 1 -- 172.21.15.29:0/2748017778 reap_dead start 2023-10-16T14:16:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.221+0000 7f733669e640 1 -- 172.21.15.29:0/2748017778 shutdown_connections 2023-10-16T14:16:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.221+0000 7f733669e640 1 --2- 172.21.15.29:0/2748017778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f731400fb00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.221+0000 7f733669e640 1 -- 172.21.15.29:0/2748017778 >> 172.21.15.29:0/2748017778 conn(0x7f73300d2b90 msgr2=0x7f73300d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:05.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.221+0000 7f733669e640 1 -- 172.21.15.29:0/2748017778 shutdown_connections 2023-10-16T14:16:05.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.221+0000 7f733669e640 1 -- 172.21.15.29:0/2748017778 wait complete. 2023-10-16T14:16:05.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.224+0000 7f733669e640 1 Processor -- start 2023-10-16T14:16:05.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.224+0000 7f733669e640 1 -- start start 2023-10-16T14:16:05.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.225+0000 7f733669e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f733015a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.225+0000 7f733669e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300cc470 0x7f73300cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.225+0000 7f733669e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f733015dff0 0x7f733015e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.225+0000 7f733669e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733015e9b0 con 0x7f733015dff0 2023-10-16T14:16:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.225+0000 7f733669e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300ccfc0 con 0x7f731400f650 2023-10-16T14:16:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.225+0000 7f733669e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733015ac10 con 0x7f73300cc470 2023-10-16T14:16:05.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.226+0000 7f733569c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f733015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.226+0000 7f733569c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f733015a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:45594/0 (socket says 172.21.15.29:45594) 2023-10-16T14:16:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.226+0000 7f733569c640 1 -- 172.21.15.29:0/545744394 learned_addr learned my addr 172.21.15.29:0/545744394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.226+0000 7f7335e9d640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f733015dff0 0x7f733015e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.226+0000 7f7334e9b640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300cc470 0x7f73300cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7335e9d640 1 -- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300cc470 msgr2=0x7f73300cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7335e9d640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300cc470 0x7f73300cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7335e9d640 1 -- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 msgr2=0x7f733015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7335e9d640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f733015a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7335e9d640 1 -- 172.21.15.29:0/545744394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731400ab10 con 0x7f733015dff0 2023-10-16T14:16:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f733569c640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f733015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7335e9d640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f733015dff0 0x7f733015e470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7330067300 tx=0x7f732800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732800a940 con 0x7f733015dff0 2023-10-16T14:16:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732800aaa0 con 0x7f733015dff0 2023-10-16T14:16:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733015aeb0 con 0x7f733015dff0 2023-10-16T14:16:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.227+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732800ae30 con 0x7f733015dff0 2023-10-16T14:16:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.228+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733015b470 con 0x7f733015dff0 2023-10-16T14:16:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.228+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f73280073d0 con 0x7f733015dff0 2023-10-16T14:16:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.228+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7328012aa0 con 0x7f733015dff0 2023-10-16T14:16:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.229+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733015b660 con 0x7f733015dff0 2023-10-16T14:16:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.235+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7328017070 con 0x7f733015dff0 2023-10-16T14:16:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.309+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f733004e9b0 con 0x7f733015dff0 2023-10-16T14:16:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.312+0000 7f7325ffb640 1 -- 172.21.15.29:0/545744394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7328007810 con 0x7f733015dff0 2023-10-16T14:16:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-10-16T14:16:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f733015dff0 msgr2=0x7f733015e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f733015dff0 0x7f733015e470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7330067300 tx=0x7f732800cc70 comp rx=0 tx=0).stop 2023-10-16T14:16:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 shutdown_connections 2023-10-16T14:16:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f733015dff0 0x7f733015e470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73300cc470 0x7f73300cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 --2- 172.21.15.29:0/545744394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f731400f650 0x7f733015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 >> 172.21.15.29:0/545744394 conn(0x7f73300d2b90 msgr2=0x7f73300d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 shutdown_connections 2023-10-16T14:16:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:05.314+0000 7f733669e640 1 -- 172.21.15.29:0/545744394 wait complete. 2023-10-16T14:16:05.335 INFO:tasks.ceph:check_enable_crimson: False 2023-10-16T14:16:05.336 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-16T14:16:05.336 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:16:05.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-16T14:16:05.416 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:16:05.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-16T14:16:05.475 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:05.471+0000 7f9df102c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:16:05.540 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:05.536+0000 7f9df102c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:16:05.549 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:16:05.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-16T14:16:05.639 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:16:05.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-16T14:16:05.726 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:16:05.726 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-16T14:16:05.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:05.800+0000 7f7fcea2e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:16:05.824 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:16:05.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-16T14:16:05.870 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:05.867+0000 7f7fcea2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:16:05.910 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:16:05.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-16T14:16:06.037 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:16:06.037 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-16T14:16:06.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 87107ae3-91be-4d1c-a15f-09a9db4e483a 0 2023-10-16T14:16:06.176 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.171+0000 7f9df102c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:16:06.239 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.235+0000 7f9df102c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:16:06.293 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.289+0000 7f9df102c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:16:06.362 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.358+0000 7f9df102c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:16:06.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.363+0000 7f47f3fff640 1 Processor -- start 2023-10-16T14:16:06.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.363+0000 7f47f3fff640 1 -- start start 2023-10-16T14:16:06.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47f40d0be0 0x7f47f40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f3fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415d150 con 0x7f47f415cd70 2023-10-16T14:16:06.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f40d1180 con 0x7f47f40d0be0 2023-10-16T14:16:06.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40d1460 con 0x7f47f40b8480 2023-10-16T14:16:06.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f27fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f27fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46022/0 (socket says 172.21.15.170:46022) 2023-10-16T14:16:06.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.365+0000 7f47f27fc640 1 -- 172.21.15.170:0/1159303433 learned_addr learned my addr 172.21.15.170:0/1159303433 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:06.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.366+0000 7f47f37fe640 1 --2- 172.21.15.170:0/1159303433 >> v2:172.21.15.29:3300/0 conn(0x7f47f40d0be0 0x7f47f40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.366+0000 7f47f27fc640 1 -- 172.21.15.170:0/1159303433 >> v2:172.21.15.29:3300/0 conn(0x7f47f40d0be0 msgr2=0x7f47f40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:06.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.366+0000 7f47f27fc640 1 --2- 172.21.15.170:0/1159303433 >> v2:172.21.15.29:3300/0 conn(0x7f47f40d0be0 0x7f47f40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.366+0000 7f47f27fc640 1 -- 172.21.15.170:0/1159303433 >> v1:172.21.15.29:6789/0 conn(0x7f47f415cd70 legacy=0x7f47f40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:06.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.366+0000 7f47f27fc640 1 -- 172.21.15.170:0/1159303433 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f40d1740 con 0x7f47f40b8480 2023-10-16T14:16:06.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.367+0000 7f47f27fc640 1 --2- 172.21.15.170:0/1159303433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40cd7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009870 tx=0x7f47dc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b513e05f8395d68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:06.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.367+0000 7f47f1ffb640 1 -- 172.21.15.170:0/1159303433 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc031640 con 0x7f47f40b8480 2023-10-16T14:16:06.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.367+0000 7f47f1ffb640 1 -- 172.21.15.170:0/1159303433 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc0317a0 con 0x7f47f40b8480 2023-10-16T14:16:06.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.367+0000 7f47f1ffb640 1 -- 172.21.15.170:0/1159303433 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc031b30 con 0x7f47f40b8480 2023-10-16T14:16:06.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 -- 172.21.15.170:0/1159303433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 msgr2=0x7f47f40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1159303433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40cd7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009870 tx=0x7f47dc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 -- 172.21.15.170:0/1159303433 shutdown_connections 2023-10-16T14:16:06.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1159303433 >> v2:172.21.15.29:3300/0 conn(0x7f47f40d0be0 0x7f47f40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1159303433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40cd7c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 -- 172.21.15.170:0/1159303433 >> 172.21.15.170:0/1159303433 conn(0x7f47f40cf570 msgr2=0x7f47f40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:06.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 -- 172.21.15.170:0/1159303433 shutdown_connections 2023-10-16T14:16:06.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.368+0000 7f47f3fff640 1 -- 172.21.15.170:0/1159303433 wait complete. 2023-10-16T14:16:06.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.370+0000 7f47f3fff640 1 Processor -- start 2023-10-16T14:16:06.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.370+0000 7f47f3fff640 1 -- start start 2023-10-16T14:16:06.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f40d0be0 0x7f47f40c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47f415cd70 0x7f47f415d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40c7920 con 0x7f47f40d0be0 2023-10-16T14:16:06.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f3fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40cb2c0 con 0x7f47f415cd70 2023-10-16T14:16:06.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40c7ba0 con 0x7f47f40b8480 2023-10-16T14:16:06.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46024/0 (socket says 172.21.15.170:46024) 2023-10-16T14:16:06.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f37fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47f415cd70 0x7f47f415d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 -- 172.21.15.170:0/1553816312 learned_addr learned my addr 172.21.15.170:0/1553816312 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:06.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f27fc640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f40d0be0 0x7f47f40c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 -- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47f415cd70 msgr2=0x7f47f415d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47f415cd70 0x7f47f415d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 -- 172.21.15.170:0/1553816312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f40d0be0 msgr2=0x7f47f40c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f40d0be0 0x7f47f40c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.371+0000 7f47f2ffd640 1 -- 172.21.15.170:0/1553816312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0094d0 con 0x7f47f40b8480 2023-10-16T14:16:06.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47f2ffd640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40c6450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f47e400c430 tx=0x7f47e400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:06.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4011790 con 0x7f47f40b8480 2023-10-16T14:16:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e40118f0 con 0x7f47f40b8480 2023-10-16T14:16:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4011ce0 con 0x7f47f40b8480 2023-10-16T14:16:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f40cb560 con 0x7f47f40b8480 2023-10-16T14:16:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f40cbb20 con 0x7f47f40b8480 2023-10-16T14:16:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f47e401c760 con 0x7f47f40b8480 2023-10-16T14:16:06.382 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.372+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f47e4012820 con 0x7f47f40b8480 2023-10-16T14:16:06.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.373+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f40d28f0 con 0x7f47f40b8480 2023-10-16T14:16:06.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.376+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f47e4011e40 con 0x7f47f40b8480 2023-10-16T14:16:06.415 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.411+0000 7f9df102c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:16:06.457 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.453+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd new", "uuid": "87107ae3-91be-4d1c-a15f-09a9db4e483a", "id": 0} v 0) v1 -- 0x7f47f404e9b0 con 0x7f47f40b8480 2023-10-16T14:16:06.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.459+0000 7f47eb7fe640 1 -- 172.21.15.170:0/1553816312 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87107ae3-91be-4d1c-a15f-09a9db4e483a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f47e401cbf0 con 0x7f47f40b8480 2023-10-16T14:16:06.464 INFO:teuthology.orchestra.run.smithi170.stdout:0 2023-10-16T14:16:06.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 msgr2=0x7f47f40c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40c6450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f47e400c430 tx=0x7f47e400c8f0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 shutdown_connections 2023-10-16T14:16:06.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f40d0be0 0x7f47f40c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47f415cd70 0x7f47f415d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 --2- 172.21.15.170:0/1553816312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47f40b8480 0x7f47f40c6450 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 >> 172.21.15.170:0/1553816312 conn(0x7f47f40cf570 msgr2=0x7f47f40d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:06.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 shutdown_connections 2023-10-16T14:16:06.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.461+0000 7f47f3fff640 1 -- 172.21.15.170:0/1553816312 wait complete. 2023-10-16T14:16:06.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 096d82fa-2713-4df9-ba03-9533658a0d6b 1 2023-10-16T14:16:06.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.521+0000 7f7fcea2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:16:06.534 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.531+0000 7f9df102c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:16:06.589 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.586+0000 7f7fcea2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:16:06.618 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.615+0000 7f9df102c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:16:06.643 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.640+0000 7f7fcea2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:16:06.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.667+0000 7fd07ccec640 1 Processor -- start 2023-10-16T14:16:06.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.667+0000 7fd07ccec640 1 -- start start 2023-10-16T14:16:06.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.668+0000 7fd07ccec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 0x7fd0780ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.668+0000 7fd07ccec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd07806c140 0x7fd07806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.668+0000 7fd07ccec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd078070360 con 0x7fd078159ab0 2023-10-16T14:16:06.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.668+0000 7fd07ccec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07806cb10 con 0x7fd07806c140 2023-10-16T14:16:06.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.668+0000 7fd07ccec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07815d890 con 0x7fd078159e90 2023-10-16T14:16:06.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 0x7fd0780ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 0x7fd0780ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46040/0 (socket says 172.21.15.170:46040) 2023-10-16T14:16:06.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 -- 172.21.15.170:0/1745806947 learned_addr learned my addr 172.21.15.170:0/1745806947 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:06.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd076d9d640 1 --2- 172.21.15.170:0/1745806947 >> v2:172.21.15.29:3300/0 conn(0x7fd07806c140 0x7fd07806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 -- 172.21.15.170:0/1745806947 >> v2:172.21.15.29:3300/0 conn(0x7fd07806c140 msgr2=0x7fd07806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:06.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 --2- 172.21.15.170:0/1745806947 >> v2:172.21.15.29:3300/0 conn(0x7fd07806c140 0x7fd07806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 -- 172.21.15.170:0/1745806947 >> v1:172.21.15.29:6789/0 conn(0x7fd078159ab0 legacy=0x7fd07806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd06ffff640 1 -- 172.21.15.170:0/1745806947 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780cd1f0 con 0x7fd078159e90 2023-10-16T14:16:06.675 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.669+0000 7fd076d9d640 1 --2- 172.21.15.170:0/1745806947 >> v2:172.21.15.29:3300/0 conn(0x7fd07806c140 0x7fd07806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:06.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.671+0000 7fd06ffff640 1 --2- 172.21.15.170:0/1745806947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 0x7fd0780ce370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd06000bb40 tx=0x7fd06002e020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf95afc5fa39f2f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:06.676 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.671+0000 7fd075d9b640 1 -- 172.21.15.170:0/1745806947 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06002edd0 con 0x7fd078159e90 2023-10-16T14:16:06.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.671+0000 7fd075d9b640 1 -- 172.21.15.170:0/1745806947 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060036440 con 0x7fd078159e90 2023-10-16T14:16:06.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.671+0000 7fd075d9b640 1 -- 172.21.15.170:0/1745806947 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600367d0 con 0x7fd078159e90 2023-10-16T14:16:06.677 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.672+0000 7f9df102c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:16:06.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 -- 172.21.15.170:0/1745806947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 msgr2=0x7fd0780ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 --2- 172.21.15.170:0/1745806947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 0x7fd0780ce370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd06000bb40 tx=0x7fd06002e020 comp rx=0 tx=0).stop 2023-10-16T14:16:06.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 -- 172.21.15.170:0/1745806947 shutdown_connections 2023-10-16T14:16:06.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 --2- 172.21.15.170:0/1745806947 >> v2:172.21.15.29:3300/0 conn(0x7fd07806c140 0x7fd07806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 --2- 172.21.15.170:0/1745806947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd078159e90 0x7fd0780ce370 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 -- 172.21.15.170:0/1745806947 >> 172.21.15.170:0/1745806947 conn(0x7fd0780ce9b0 msgr2=0x7fd0780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:06.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 -- 172.21.15.170:0/1745806947 shutdown_connections 2023-10-16T14:16:06.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.672+0000 7fd07ccec640 1 -- 172.21.15.170:0/1745806947 wait complete. 2023-10-16T14:16:06.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.674+0000 7fd07ccec640 1 Processor -- start 2023-10-16T14:16:06.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.674+0000 7fd07ccec640 1 -- start start 2023-10-16T14:16:06.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07ccec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 0x7fd0781f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07ccec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078159ab0 0x7fd0780c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07ccec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd078159e90 0x7fd07815cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07ccec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07815c770 con 0x7fd078159e90 2023-10-16T14:16:06.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07ccec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0781f25c0 con 0x7fd078159ab0 2023-10-16T14:16:06.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07ccec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07815bdd0 con 0x7fd07806c140 2023-10-16T14:16:06.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 0x7fd0781f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd06ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078159ab0 0x7fd0780c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 0x7fd0781f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46054/0 (socket says 172.21.15.170:46054) 2023-10-16T14:16:06.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.675+0000 7fd07659c640 1 -- 172.21.15.170:0/465262578 learned_addr learned my addr 172.21.15.170:0/465262578 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:06.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd076d9d640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd078159e90 0x7fd07815cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06ffff640 1 -- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 msgr2=0x7fd0781f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06ffff640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 0x7fd0781f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06ffff640 1 -- 172.21.15.170:0/465262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd078159e90 msgr2=0x7fd07815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06ffff640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd078159e90 0x7fd07815cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06ffff640 1 -- 172.21.15.170:0/465262578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06000b770 con 0x7fd078159ab0 2023-10-16T14:16:06.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd07659c640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 0x7fd0781f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:06.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd076d9d640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd078159e90 0x7fd07815cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:06.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06ffff640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078159ab0 0x7fd0780c24e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd060035000 tx=0x7fd0600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:06.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060034750 con 0x7fd078159ab0 2023-10-16T14:16:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0600348b0 con 0x7fd078159ab0 2023-10-16T14:16:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060034c20 con 0x7fd078159ab0 2023-10-16T14:16:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.676+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07815c0b0 con 0x7fd078159ab0 2023-10-16T14:16:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.677+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07815b9f0 con 0x7fd078159ab0 2023-10-16T14:16:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.677+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd060041460 con 0x7fd078159ab0 2023-10-16T14:16:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.677+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fd06004a7a0 con 0x7fd078159ab0 2023-10-16T14:16:06.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.678+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07815bb80 con 0x7fd078159ab0 2023-10-16T14:16:06.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.683+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd060046070 con 0x7fd078159ab0 2023-10-16T14:16:06.709 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.706+0000 7f7fcea2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:16:06.740 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.737+0000 7f9df102c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:16:06.762 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.759+0000 7f7fcea2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:16:06.764 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.761+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "096d82fa-2713-4df9-ba03-9533658a0d6b", "id": 1} v 0) v1 -- 0x7fd07804e160 con 0x7fd078159ab0 2023-10-16T14:16:06.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.767+0000 7fd06effd640 1 -- 172.21.15.170:0/465262578 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "096d82fa-2713-4df9-ba03-9533658a0d6b", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd06003b070 con 0x7fd078159ab0 2023-10-16T14:16:06.771 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-10-16T14:16:06.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078159ab0 msgr2=0x7fd0780c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078159ab0 0x7fd0780c24e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd060035000 tx=0x7fd0600046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 shutdown_connections 2023-10-16T14:16:06.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd078159e90 0x7fd07815cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd078159ab0 0x7fd0780c24e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 --2- 172.21.15.170:0/465262578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd07806c140 0x7fd0781f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 >> 172.21.15.170:0/465262578 conn(0x7fd0780ce9b0 msgr2=0x7fd0780b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:06.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 shutdown_connections 2023-10-16T14:16:06.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.768+0000 7fd07ccec640 1 -- 172.21.15.170:0/465262578 wait complete. 2023-10-16T14:16:06.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 121bf1ec-267b-4a07-aec1-e63cd130ccea 2 2023-10-16T14:16:06.799 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:06.796+0000 7f9df102c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:16:06.883 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.880+0000 7f7fcea2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:16:06.968 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:06.964+0000 7f7fcea2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:16:06.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.973+0000 7f52188f2640 1 Processor -- start 2023-10-16T14:16:06.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.974+0000 7f52188f2640 1 -- start start 2023-10-16T14:16:06.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.975+0000 7f52188f2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5214070830 0x7f521406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.975+0000 7f52188f2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f52140cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.975+0000 7f52188f2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5214161740 con 0x7f52140cf5f0 2023-10-16T14:16:06.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.975+0000 7f52188f2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52140cdbd0 con 0x7f5214070830 2023-10-16T14:16:06.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.975+0000 7f52188f2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521406b0f0 con 0x7f5214070c10 2023-10-16T14:16:06.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5212d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f52140cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5211d9b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f52140cf5f0 0x7f521406c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:55978/0 (socket says 172.21.15.170:55978) 2023-10-16T14:16:06.981 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5211d9b640 1 -- 172.21.15.170:0/3555503015 learned_addr learned my addr 172.21.15.170:0/3555503015 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-16T14:16:06.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f521259c640 1 --2- 172.21.15.170:0/3555503015 >> v2:172.21.15.29:3300/0 conn(0x7f5214070830 0x7f521406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5212d9d640 1 -- 172.21.15.170:0/3555503015 >> v2:172.21.15.29:3300/0 conn(0x7f5214070830 msgr2=0x7f521406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:06.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5212d9d640 1 --2- 172.21.15.170:0/3555503015 >> v2:172.21.15.29:3300/0 conn(0x7f5214070830 0x7f521406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5212d9d640 1 -- 172.21.15.170:0/3555503015 >> v1:172.21.15.170:6789/0 conn(0x7f52140cf5f0 legacy=0x7f521406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.976+0000 7f5212d9d640 1 -- 172.21.15.170:0/3555503015 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52141619b0 con 0x7f5214070c10 2023-10-16T14:16:06.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.977+0000 7f5212d9d640 1 --2- 172.21.15.170:0/3555503015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f52140cd690 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f520c009d90 tx=0x7f520c02bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d88fd254324d44b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:06.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.977+0000 7f521159a640 1 -- 172.21.15.170:0/3555503015 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c02cb40 con 0x7f5214070c10 2023-10-16T14:16:06.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.977+0000 7f521159a640 1 -- 172.21.15.170:0/3555503015 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c02cca0 con 0x7f5214070c10 2023-10-16T14:16:06.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.977+0000 7f521159a640 1 -- 172.21.15.170:0/3555503015 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c034660 con 0x7f5214070c10 2023-10-16T14:16:06.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 -- 172.21.15.170:0/3555503015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 msgr2=0x7f52140cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 --2- 172.21.15.170:0/3555503015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f52140cd690 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f520c009d90 tx=0x7f520c02bd90 comp rx=0 tx=0).stop 2023-10-16T14:16:06.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 -- 172.21.15.170:0/3555503015 shutdown_connections 2023-10-16T14:16:06.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 --2- 172.21.15.170:0/3555503015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f52140cd690 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 --2- 172.21.15.170:0/3555503015 >> v2:172.21.15.29:3300/0 conn(0x7f5214070830 0x7f521406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 -- 172.21.15.170:0/3555503015 >> 172.21.15.170:0/3555503015 conn(0x7f52140cfac0 msgr2=0x7f52140cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:06.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 -- 172.21.15.170:0/3555503015 shutdown_connections 2023-10-16T14:16:06.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.978+0000 7f52188f2640 1 -- 172.21.15.170:0/3555503015 wait complete. 2023-10-16T14:16:06.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.980+0000 7f52188f2640 1 Processor -- start 2023-10-16T14:16:06.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.980+0000 7f52188f2640 1 -- start start 2023-10-16T14:16:06.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f52188f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5214070830 0x7f52141f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f52188f2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f521415b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f52188f2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 0x7f52140cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:06.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f52188f2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140c4c70 con 0x7f5214070830 2023-10-16T14:16:06.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f52188f2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140c35e0 con 0x7f52140cf5f0 2023-10-16T14:16:06.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f52188f2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52141f3e80 con 0x7f5214070c10 2023-10-16T14:16:06.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f5212d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 0x7f52140cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f5211d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f521415b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f5212d9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 0x7f52140cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33706/0 (socket says 172.21.15.170:33706) 2023-10-16T14:16:06.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f5212d9d640 1 -- 172.21.15.170:0/4182591828 learned_addr learned my addr 172.21.15.170:0/4182591828 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:06.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.981+0000 7f521259c640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5214070830 0x7f52141f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:06.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5212d9d640 1 -- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 msgr2=0x7f521415b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5212d9d640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f521415b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5212d9d640 1 -- 172.21.15.170:0/4182591828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5214070830 msgr2=0x7f52141f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:06.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5212d9d640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5214070830 0x7f52141f3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:06.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5212d9d640 1 -- 172.21.15.170:0/4182591828 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c009a70 con 0x7f52140cf5f0 2023-10-16T14:16:06.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5211d9b640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f521415b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:06.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5212d9d640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 0x7f52140cb0e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f520c009d90 tx=0x7f520c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:06.992 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c040650 con 0x7f52140cf5f0 2023-10-16T14:16:06.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c0407d0 con 0x7f52140cf5f0 2023-10-16T14:16:06.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c040b80 con 0x7f52140cf5f0 2023-10-16T14:16:06.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52140c54d0 con 0x7f52140cf5f0 2023-10-16T14:16:06.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.982+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521415bc40 con 0x7f52140cf5f0 2023-10-16T14:16:06.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.983+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f520c049400 con 0x7f52140cf5f0 2023-10-16T14:16:06.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.983+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f520c049ad0 con 0x7f52140cf5f0 2023-10-16T14:16:06.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.984+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52140c5660 con 0x7f52140cf5f0 2023-10-16T14:16:06.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:06.986+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f520c067b40 con 0x7f52140cf5f0 2023-10-16T14:16:07.024 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.021+0000 7f7fcea2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:16:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.063+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "121bf1ec-267b-4a07-aec1-e63cd130ccea", "id": 2} v 0) v1 -- 0x7f521404e160 con 0x7f52140cf5f0 2023-10-16T14:16:07.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.070+0000 7f5202ffd640 1 -- 172.21.15.170:0/4182591828 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "121bf1ec-267b-4a07-aec1-e63cd130ccea", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f520c056030 con 0x7f52140cf5f0 2023-10-16T14:16:07.074 INFO:teuthology.orchestra.run.smithi170.stdout:2 2023-10-16T14:16:07.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 msgr2=0x7f52140cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 0x7f52140cb0e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f520c009d90 tx=0x7f520c004a40 comp rx=0 tx=0).stop 2023-10-16T14:16:07.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 shutdown_connections 2023-10-16T14:16:07.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52140cf5f0 0x7f52140cb0e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5214070c10 0x7f521415b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 --2- 172.21.15.170:0/4182591828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5214070830 0x7f52141f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.071+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 >> 172.21.15.170:0/4182591828 conn(0x7f52140cfac0 msgr2=0x7f5214160180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:07.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.072+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 shutdown_connections 2023-10-16T14:16:07.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.072+0000 7f52188f2640 1 -- 172.21.15.170:0/4182591828 wait complete. 2023-10-16T14:16:07.088 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.085+0000 7f7fcea2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:16:07.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 71979c26-ad9d-4dd4-bb21-3f766e254965 3 2023-10-16T14:16:07.146 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.143+0000 7f7fcea2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:16:07.275 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:07.271+0000 7f9df102c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:16:07.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.320+0000 7f8928ca3640 1 Processor -- start 2023-10-16T14:16:07.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.320+0000 7f8928ca3640 1 -- start start 2023-10-16T14:16:07.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.321+0000 7f8928ca3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f89240b8480 0x7f89240cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.325 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.321+0000 7f8928ca3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 0x7f89240cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.321+0000 7f8928ca3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892415d150 con 0x7f892415cd70 2023-10-16T14:16:07.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.321+0000 7f8928ca3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89240d1180 con 0x7f89240b8480 2023-10-16T14:16:07.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.321+0000 7f8928ca3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240d1460 con 0x7f89240d0be0 2023-10-16T14:16:07.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8923fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 0x7f89240cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.326 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8922ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f89240b8480 0x7f89240cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8923fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 0x7f89240cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46086/0 (socket says 172.21.15.170:46086) 2023-10-16T14:16:07.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8923fff640 1 -- 172.21.15.170:0/797151345 learned_addr learned my addr 172.21.15.170:0/797151345 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:07.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8922ffd640 1 -- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 msgr2=0x7f89240cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:07.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8922ffd640 1 --2- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 0x7f89240cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8922ffd640 1 -- 172.21.15.170:0/797151345 >> v1:172.21.15.29:6789/0 conn(0x7f892415cd70 legacy=0x7f89240d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:07.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8922ffd640 1 -- 172.21.15.170:0/797151345 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240d1740 con 0x7f89240b8480 2023-10-16T14:16:07.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.322+0000 7f8923fff640 1 --2- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 0x7f89240cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:07.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.323+0000 7f8922ffd640 1 --2- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89240b8480 0x7f89240cd7c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f89180097a0 tx=0x7f891802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1b88b08aebecf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.323+0000 7f89227fc640 1 -- 172.21.15.170:0/797151345 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918031780 con 0x7f89240b8480 2023-10-16T14:16:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.323+0000 7f89227fc640 1 -- 172.21.15.170:0/797151345 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89180318e0 con 0x7f89240b8480 2023-10-16T14:16:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.323+0000 7f89227fc640 1 -- 172.21.15.170:0/797151345 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918031c70 con 0x7f89240b8480 2023-10-16T14:16:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 -- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89240b8480 msgr2=0x7f89240cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 --2- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89240b8480 0x7f89240cd7c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f89180097a0 tx=0x7f891802b820 comp rx=0 tx=0).stop 2023-10-16T14:16:07.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 -- 172.21.15.170:0/797151345 shutdown_connections 2023-10-16T14:16:07.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 --2- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240d0be0 0x7f89240cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 --2- 172.21.15.170:0/797151345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f89240b8480 0x7f89240cd7c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 -- 172.21.15.170:0/797151345 >> 172.21.15.170:0/797151345 conn(0x7f89240cf570 msgr2=0x7f89240cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:07.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 -- 172.21.15.170:0/797151345 shutdown_connections 2023-10-16T14:16:07.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.324+0000 7f8928ca3640 1 -- 172.21.15.170:0/797151345 wait complete. 2023-10-16T14:16:07.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.326+0000 7f8928ca3640 1 Processor -- start 2023-10-16T14:16:07.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.326+0000 7f8928ca3640 1 -- start start 2023-10-16T14:16:07.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8928ca3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 0x7f89240cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8928ca3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89240d0be0 0x7f8924156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8928ca3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f892415cd70 0x7f8924152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8928ca3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240cc550 con 0x7f89240d0be0 2023-10-16T14:16:07.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8928ca3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240cb940 con 0x7f892415cd70 2023-10-16T14:16:07.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8928ca3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240cbc20 con 0x7f89240b8480 2023-10-16T14:16:07.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f89237fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 0x7f89240cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8923fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f892415cd70 0x7f8924152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f89237fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 0x7f89240cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46096/0 (socket says 172.21.15.170:46096) 2023-10-16T14:16:07.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f89237fe640 1 -- 172.21.15.170:0/3361392909 learned_addr learned my addr 172.21.15.170:0/3361392909 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:07.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.327+0000 7f8922ffd640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89240d0be0 0x7f8924156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f89237fe640 1 -- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f892415cd70 msgr2=0x7f8924152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f89237fe640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f892415cd70 0x7f8924152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f89237fe640 1 -- 172.21.15.170:0/3361392909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89240d0be0 msgr2=0x7f8924156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f89237fe640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89240d0be0 0x7f8924156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f89237fe640 1 -- 172.21.15.170:0/3361392909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c009b30 con 0x7f89240b8480 2023-10-16T14:16:07.337 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f8923fff640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f892415cd70 0x7f8924152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:07.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f89237fe640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 0x7f89240cb360 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f891400b3f0 tx=0x7f891400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:07.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914011830 con 0x7f89240b8480 2023-10-16T14:16:07.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8914011990 con 0x7f89240b8480 2023-10-16T14:16:07.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914011d40 con 0x7f89240b8480 2023-10-16T14:16:07.338 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8918009400 con 0x7f89240b8480 2023-10-16T14:16:07.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.328+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89240cc1e0 con 0x7f89240b8480 2023-10-16T14:16:07.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.329+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8914004320 con 0x7f89240b8480 2023-10-16T14:16:07.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.329+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f8914003a80 con 0x7f89240b8480 2023-10-16T14:16:07.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.329+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89240d1a00 con 0x7f89240b8480 2023-10-16T14:16:07.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.332+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f8914038b40 con 0x7f89240b8480 2023-10-16T14:16:07.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.409+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd new", "uuid": "71979c26-ad9d-4dd4-bb21-3f766e254965", "id": 3} v 0) v1 -- 0x7f892404e9b0 con 0x7f89240b8480 2023-10-16T14:16:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.415+0000 7f890bfff640 1 -- 172.21.15.170:0/3361392909 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "71979c26-ad9d-4dd4-bb21-3f766e254965", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8914027070 con 0x7f89240b8480 2023-10-16T14:16:07.419 INFO:teuthology.orchestra.run.smithi170.stdout:3 2023-10-16T14:16:07.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 msgr2=0x7f89240cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 0x7f89240cb360 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f891400b3f0 tx=0x7f891400b8b0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 shutdown_connections 2023-10-16T14:16:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89240d0be0 0x7f8924156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f892415cd70 0x7f8924152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 --2- 172.21.15.170:0/3361392909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89240b8480 0x7f89240cb360 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.416+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 >> 172.21.15.170:0/3361392909 conn(0x7f89240cf570 msgr2=0x7f89240b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.417+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 shutdown_connections 2023-10-16T14:16:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.417+0000 7f8928ca3640 1 -- 172.21.15.170:0/3361392909 wait complete. 2023-10-16T14:16:07.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new d2efc192-e765-4786-bd6f-141672cf5d53 4 2023-10-16T14:16:07.445 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:07.442+0000 7f9df102c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:16:07.584 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:07.580+0000 7f9df102c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:16:07.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.618+0000 7ff2d7247640 1 Processor -- start 2023-10-16T14:16:07.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.619+0000 7ff2d7247640 1 -- start start 2023-10-16T14:16:07.623 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.619+0000 7f7fcea2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:16:07.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.620+0000 7ff2d7247640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2d00b8480 0x7ff2d00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.620+0000 7ff2d7247640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d015d150 con 0x7ff2d015cd70 2023-10-16T14:16:07.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.620+0000 7ff2d7247640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d00d1180 con 0x7ff2d00d0be0 2023-10-16T14:16:07.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.620+0000 7ff2d7247640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d00d1460 con 0x7ff2d00b8480 2023-10-16T14:16:07.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d5a44640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2d00b8480 0x7ff2d00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d5a44640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2d00b8480 0x7ff2d00cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33742/0 (socket says 172.21.15.170:33742) 2023-10-16T14:16:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d5a44640 1 -- 172.21.15.170:0/3632013420 learned_addr learned my addr 172.21.15.170:0/3632013420 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d6a46640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7ff2d00d0be0 0x7ff2d00b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:55984/0 (socket says 172.21.15.170:55984) 2023-10-16T14:16:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d5a44640 1 -- 172.21.15.170:0/3632013420 >> v1:172.21.15.170:6789/0 conn(0x7ff2d00d0be0 legacy=0x7ff2d00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d5a44640 1 -- 172.21.15.170:0/3632013420 >> v1:172.21.15.29:6789/0 conn(0x7ff2d015cd70 legacy=0x7ff2d00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.621+0000 7ff2d5a44640 1 -- 172.21.15.170:0/3632013420 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00d1740 con 0x7ff2d00b8480 2023-10-16T14:16:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.622+0000 7ff2d5a44640 1 --2- 172.21.15.170:0/3632013420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cd7c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009870 tx=0x7ff2c002b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=666b2695efb99aa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.622+0000 7ff2d5243640 1 -- 172.21.15.170:0/3632013420 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0030850 con 0x7ff2d00b8480 2023-10-16T14:16:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.622+0000 7ff2d5243640 1 -- 172.21.15.170:0/3632013420 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c00309b0 con 0x7ff2d00b8480 2023-10-16T14:16:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.622+0000 7ff2d5243640 1 -- 172.21.15.170:0/3632013420 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0030d40 con 0x7ff2d00b8480 2023-10-16T14:16:07.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 -- 172.21.15.170:0/3632013420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00b8480 msgr2=0x7ff2d00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 --2- 172.21.15.170:0/3632013420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cd7c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009870 tx=0x7ff2c002b990 comp rx=0 tx=0).stop 2023-10-16T14:16:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 -- 172.21.15.170:0/3632013420 shutdown_connections 2023-10-16T14:16:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 --2- 172.21.15.170:0/3632013420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cd7c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 -- 172.21.15.170:0/3632013420 >> 172.21.15.170:0/3632013420 conn(0x7ff2d00cf570 msgr2=0x7ff2d00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 -- 172.21.15.170:0/3632013420 shutdown_connections 2023-10-16T14:16:07.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.623+0000 7ff2d7247640 1 -- 172.21.15.170:0/3632013420 wait complete. 2023-10-16T14:16:07.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.625+0000 7ff2d7247640 1 Processor -- start 2023-10-16T14:16:07.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.625+0000 7ff2d7247640 1 -- start start 2023-10-16T14:16:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d7247640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d7247640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 0x7ff2d01efa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d7247640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2d015cd70 0x7ff2d01f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d7247640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d0155dc0 con 0x7ff2d00b8480 2023-10-16T14:16:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d7247640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d0154ed0 con 0x7ff2d00d0be0 2023-10-16T14:16:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d7247640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d01551b0 con 0x7ff2d015cd70 2023-10-16T14:16:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d5a44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 0x7ff2d01efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d6a46640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2d015cd70 0x7ff2d01f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d5a44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 0x7ff2d01efa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33758/0 (socket says 172.21.15.170:33758) 2023-10-16T14:16:07.634 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d5a44640 1 -- 172.21.15.170:0/2365892850 learned_addr learned my addr 172.21.15.170:0/2365892850 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.626+0000 7ff2d6245640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6a46640 1 -- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 msgr2=0x7ff2d01efa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6a46640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 0x7ff2d01efa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6a46640 1 -- 172.21.15.170:0/2365892850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2d00b8480 msgr2=0x7ff2d00cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6a46640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6a46640 1 -- 172.21.15.170:0/2365892850 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c00094d0 con 0x7ff2d015cd70 2023-10-16T14:16:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d5a44640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 0x7ff2d01efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6245640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d6a46640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2d015cd70 0x7ff2d01f5cb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800c730 tx=0x7ff2c800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8004d60 con 0x7ff2d015cd70 2023-10-16T14:16:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c800a640 con 0x7ff2d015cd70 2023-10-16T14:16:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d0155490 con 0x7ff2d015cd70 2023-10-16T14:16:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800aa10 con 0x7ff2d015cd70 2023-10-16T14:16:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d0155a50 con 0x7ff2d015cd70 2023-10-16T14:16:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.627+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff2c8007070 con 0x7ff2d015cd70 2023-10-16T14:16:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.628+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7ff2c8007830 con 0x7ff2d015cd70 2023-10-16T14:16:07.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.628+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d004e9b0 con 0x7ff2d015cd70 2023-10-16T14:16:07.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.631+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff2c8018070 con 0x7ff2d015cd70 2023-10-16T14:16:07.641 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:07.637+0000 7f9df102c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:16:07.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.708+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d2efc192-e765-4786-bd6f-141672cf5d53", "id": 4} v 0) v1 -- 0x7ff2d00b8210 con 0x7ff2d015cd70 2023-10-16T14:16:07.719 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.715+0000 7ff2c6ffd640 1 -- 172.21.15.170:0/2365892850 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d2efc192-e765-4786-bd6f-141672cf5d53", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff2c8007320 con 0x7ff2d015cd70 2023-10-16T14:16:07.719 INFO:teuthology.orchestra.run.smithi170.stdout:4 2023-10-16T14:16:07.721 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2d015cd70 msgr2=0x7ff2d01f5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.721 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2d015cd70 0x7ff2d01f5cb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800c730 tx=0x7ff2c800cbf0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.721 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 shutdown_connections 2023-10-16T14:16:07.721 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2d015cd70 0x7ff2d01f5cb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.722 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2d00d0be0 0x7ff2d01efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.722 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 --2- 172.21.15.170:0/2365892850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2d00b8480 0x7ff2d00cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.722 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.716+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 >> 172.21.15.170:0/2365892850 conn(0x7ff2d00cf570 msgr2=0x7ff2d00b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:07.722 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.717+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 shutdown_connections 2023-10-16T14:16:07.722 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.717+0000 7ff2d7247640 1 -- 172.21.15.170:0/2365892850 wait complete. 2023-10-16T14:16:07.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 6cf2f5c6-ec40-4efe-b2c2-7890a56f513f 5 2023-10-16T14:16:07.754 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:07.750+0000 7f9df102c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:16:07.792 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.789+0000 7f7fcea2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:16:07.872 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:07.868+0000 7f9df102c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:16:07.933 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.929+0000 7f7fcea2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:16:07.950 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.946+0000 7f7389593640 1 Processor -- start 2023-10-16T14:16:07.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.946+0000 7f7389593640 1 -- start start 2023-10-16T14:16:07.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f7389593640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73840acf90 0x7f738406f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f7389593640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7384160730 0x7f738406fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f7389593640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7384160cb0 con 0x7f738415ca70 2023-10-16T14:16:07.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f7389593640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7384160f90 con 0x7f7384160730 2023-10-16T14:16:07.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f7389593640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7384161270 con 0x7f73840acf90 2023-10-16T14:16:07.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f73837fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73840acf90 0x7f738406f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f73837fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f73840acf90 0x7f738406f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33774/0 (socket says 172.21.15.170:33774) 2023-10-16T14:16:07.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.948+0000 7f73837fe640 1 -- 172.21.15.170:0/3162755356 learned_addr learned my addr 172.21.15.170:0/3162755356 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:07.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.949+0000 7f73837fe640 1 -- 172.21.15.170:0/3162755356 >> v2:172.21.15.29:3300/0 conn(0x7f7384160730 msgr2=0x7f738406fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:07.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.949+0000 7f73837fe640 1 --2- 172.21.15.170:0/3162755356 >> v2:172.21.15.29:3300/0 conn(0x7f7384160730 0x7f738406fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.949+0000 7f73837fe640 1 -- 172.21.15.170:0/3162755356 >> v1:172.21.15.29:6789/0 conn(0x7f738415ca70 legacy=0x7f73840ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.949+0000 7f73837fe640 1 -- 172.21.15.170:0/3162755356 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7384161550 con 0x7f73840acf90 2023-10-16T14:16:07.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.950+0000 7f73837fe640 1 --2- 172.21.15.170:0/3162755356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f738406f6f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f73840aeef0 tx=0x7f736c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3157bff82a9a1b99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:07.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.950+0000 7f7382ffd640 1 -- 172.21.15.170:0/3162755356 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c031870 con 0x7f73840acf90 2023-10-16T14:16:07.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.950+0000 7f7382ffd640 1 -- 172.21.15.170:0/3162755356 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c0319d0 con 0x7f73840acf90 2023-10-16T14:16:07.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.950+0000 7f7382ffd640 1 -- 172.21.15.170:0/3162755356 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c031d60 con 0x7f73840acf90 2023-10-16T14:16:07.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 -- 172.21.15.170:0/3162755356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 msgr2=0x7f738406f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 --2- 172.21.15.170:0/3162755356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f738406f6f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f73840aeef0 tx=0x7f736c02b910 comp rx=0 tx=0).stop 2023-10-16T14:16:07.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 -- 172.21.15.170:0/3162755356 shutdown_connections 2023-10-16T14:16:07.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 --2- 172.21.15.170:0/3162755356 >> v2:172.21.15.29:3300/0 conn(0x7f7384160730 0x7f738406fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 --2- 172.21.15.170:0/3162755356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f738406f6f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.957 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 -- 172.21.15.170:0/3162755356 >> 172.21.15.170:0/3162755356 conn(0x7f7384069da0 msgr2=0x7f738406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:07.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 -- 172.21.15.170:0/3162755356 shutdown_connections 2023-10-16T14:16:07.958 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.951+0000 7f7389593640 1 -- 172.21.15.170:0/3162755356 wait complete. 2023-10-16T14:16:07.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.953+0000 7f7389593640 1 Processor -- start 2023-10-16T14:16:07.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.953+0000 7f7389593640 1 -- start start 2023-10-16T14:16:07.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7389593640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f73841f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.962 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7389593640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f738415ca70 0x7f738414ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7389593640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7384160730 0x7f738415e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:07.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7389593640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738415f9b0 con 0x7f738415ca70 2023-10-16T14:16:07.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7389593640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841f64c0 con 0x7f73840acf90 2023-10-16T14:16:07.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7389593640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73841f1f70 con 0x7f7384160730 2023-10-16T14:16:07.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7383fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f73841f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.963 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7388d92640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7384160730 0x7f738415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7383fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f73841f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33784/0 (socket says 172.21.15.170:33784) 2023-10-16T14:16:07.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.954+0000 7f7383fff640 1 -- 172.21.15.170:0/3136114651 learned_addr learned my addr 172.21.15.170:0/3136114651 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:07.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f73837fe640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f738415ca70 0x7f738414ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:07.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7383fff640 1 -- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7384160730 msgr2=0x7f738415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.964 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7383fff640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7384160730 0x7f738415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7383fff640 1 -- 172.21.15.170:0/3136114651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f738415ca70 msgr2=0x7f738414ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:07.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7383fff640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f738415ca70 0x7f738414ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:07.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7383fff640 1 -- 172.21.15.170:0/3136114651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c0094d0 con 0x7f73840acf90 2023-10-16T14:16:07.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7388d92640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7384160730 0x7f738415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:07.965 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7383fff640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f73841f5f80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f737400c870 tx=0x7f737400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:07.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374003b70 con 0x7f73840acf90 2023-10-16T14:16:07.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374003cf0 con 0x7f73840acf90 2023-10-16T14:16:07.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738415fc50 con 0x7f73840acf90 2023-10-16T14:16:07.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.955+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400aa90 con 0x7f73840acf90 2023-10-16T14:16:07.966 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.956+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73840b22b0 con 0x7f73840acf90 2023-10-16T14:16:07.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.956+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f73740050d0 con 0x7f73840acf90 2023-10-16T14:16:07.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.956+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f7374012510 con 0x7f73840acf90 2023-10-16T14:16:07.967 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.957+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73841f2140 con 0x7f73840acf90 2023-10-16T14:16:07.968 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:07.959+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7374005380 con 0x7f73840acf90 2023-10-16T14:16:07.989 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:07.985+0000 7f7fcea2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:16:08.038 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:08.034+0000 7f9df102c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:16:08.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.036+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6cf2f5c6-ec40-4efe-b2c2-7890a56f513f", "id": 5} v 0) v1 -- 0x7f738404e9b0 con 0x7f73840acf90 2023-10-16T14:16:08.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.042+0000 7f7380ff9640 1 -- 172.21.15.170:0/3136114651 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6cf2f5c6-ec40-4efe-b2c2-7890a56f513f", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7374005520 con 0x7f73840acf90 2023-10-16T14:16:08.047 INFO:teuthology.orchestra.run.smithi170.stdout:5 2023-10-16T14:16:08.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.043+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 msgr2=0x7f73841f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.043+0000 7f7389593640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f73841f5f80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f737400c870 tx=0x7f737400cd30 comp rx=0 tx=0).stop 2023-10-16T14:16:08.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 shutdown_connections 2023-10-16T14:16:08.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f738415ca70 0x7f738414ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7384160730 0x7f738415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 --2- 172.21.15.170:0/3136114651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f73840acf90 0x7f73841f5f80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 >> 172.21.15.170:0/3136114651 conn(0x7f7384069da0 msgr2=0x7f73841562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:08.049 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 shutdown_connections 2023-10-16T14:16:08.050 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.044+0000 7f7389593640 1 -- 172.21.15.170:0/3136114651 wait complete. 2023-10-16T14:16:08.064 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 886d6f1f-a25b-4b98-a2c8-056eab1d62dc 6 2023-10-16T14:16:08.091 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:08.088+0000 7f9df102c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:16:08.104 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:08.101+0000 7f7fcea2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:16:08.221 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:08.218+0000 7f7fcea2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:16:08.256 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:08.252+0000 7f9df102c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:16:08.259 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.256+0000 7fd6c4a59640 1 Processor -- start 2023-10-16T14:16:08.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.256+0000 7fd6c4a59640 1 -- start start 2023-10-16T14:16:08.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.257+0000 7fd6c4a59640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c015cd70 0x7fd6c00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.257+0000 7fd6c4a59640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd6c00b8480 0x7fd6c00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6c4a59640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c015d150 con 0x7fd6c00d0be0 2023-10-16T14:16:08.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6c4a59640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c00d1180 con 0x7fd6c00b8480 2023-10-16T14:16:08.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6c4a59640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00d1460 con 0x7fd6c015cd70 2023-10-16T14:16:08.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6bf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c015cd70 0x7fd6c00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6b7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd6c00b8480 0x7fd6c00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6bf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c015cd70 0x7fd6c00d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46106/0 (socket says 172.21.15.170:46106) 2023-10-16T14:16:08.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6bf7fe640 1 -- 172.21.15.170:0/2527300919 learned_addr learned my addr 172.21.15.170:0/2527300919 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:08.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6b7fff640 1 -- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c015cd70 msgr2=0x7fd6c00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:08.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6b7fff640 1 --2- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c015cd70 0x7fd6c00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6b7fff640 1 -- 172.21.15.170:0/2527300919 >> v1:172.21.15.29:6789/0 conn(0x7fd6c00d0be0 legacy=0x7fd6c00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:08.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.258+0000 7fd6b7fff640 1 -- 172.21.15.170:0/2527300919 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c00d1740 con 0x7fd6c00b8480 2023-10-16T14:16:08.265 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.259+0000 7fd6b7fff640 1 --2- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c00cd7c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80097a0 tx=0x7fd6a802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8012f4f6b8d741ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:08.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6beffd640 1 -- 172.21.15.170:0/2527300919 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8031780 con 0x7fd6c00b8480 2023-10-16T14:16:08.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6beffd640 1 -- 172.21.15.170:0/2527300919 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a80318e0 con 0x7fd6c00b8480 2023-10-16T14:16:08.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6beffd640 1 -- 172.21.15.170:0/2527300919 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8031c70 con 0x7fd6c00b8480 2023-10-16T14:16:08.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 msgr2=0x7fd6c00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c00cd7c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80097a0 tx=0x7fd6a802b820 comp rx=0 tx=0).stop 2023-10-16T14:16:08.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2527300919 shutdown_connections 2023-10-16T14:16:08.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c00cd7c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2527300919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c015cd70 0x7fd6c00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2527300919 >> 172.21.15.170:0/2527300919 conn(0x7fd6c00cf570 msgr2=0x7fd6c00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:08.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2527300919 shutdown_connections 2023-10-16T14:16:08.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.260+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2527300919 wait complete. 2023-10-16T14:16:08.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.262+0000 7fd6c4a59640 1 Processor -- start 2023-10-16T14:16:08.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.262+0000 7fd6c4a59640 1 -- start start 2023-10-16T14:16:08.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6c4a59640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c01f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6c4a59640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 0x7fd6c00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6c4a59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c015cd70 0x7fd6c00b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6c4a59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c015d3f0 con 0x7fd6c015cd70 2023-10-16T14:16:08.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6c4a59640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00cc0f0 con 0x7fd6c00b8480 2023-10-16T14:16:08.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6c4a59640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00cc3d0 con 0x7fd6c00d0be0 2023-10-16T14:16:08.271 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6b7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 0x7fd6c00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6bf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c01f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6bf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c01f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33802/0 (socket says 172.21.15.170:33802) 2023-10-16T14:16:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6b7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 0x7fd6c00b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.170:46122/0 (socket says 172.21.15.170:46122) 2023-10-16T14:16:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.263+0000 7fd6bf7fe640 1 -- 172.21.15.170:0/2859717231 learned_addr learned my addr 172.21.15.170:0/2859717231 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bffff640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c015cd70 0x7fd6c00b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bf7fe640 1 -- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 msgr2=0x7fd6c00b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bf7fe640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 0x7fd6c00b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bf7fe640 1 -- 172.21.15.170:0/2859717231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c015cd70 msgr2=0x7fd6c00b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bf7fe640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c015cd70 0x7fd6c00b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bf7fe640 1 -- 172.21.15.170:0/2859717231 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8009400 con 0x7fd6c00b8480 2023-10-16T14:16:08.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6b7fff640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 0x7fd6c00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:08.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bf7fe640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c01f3130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009fd0 tx=0x7fd6b000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:08.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b000c940 con 0x7fd6c00b8480 2023-10-16T14:16:08.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b000caa0 con 0x7fd6c00b8480 2023-10-16T14:16:08.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c00cc6b0 con 0x7fd6c00b8480 2023-10-16T14:16:08.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.264+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b000ce30 con 0x7fd6c00b8480 2023-10-16T14:16:08.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.265+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c00cb3c0 con 0x7fd6c00b8480 2023-10-16T14:16:08.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.265+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd6b0005c90 con 0x7fd6c00b8480 2023-10-16T14:16:08.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.265+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fd6b0006840 con 0x7fd6c00b8480 2023-10-16T14:16:08.275 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.266+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c00d1cb0 con 0x7fd6c00b8480 2023-10-16T14:16:08.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.268+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd6b0021060 con 0x7fd6c00b8480 2023-10-16T14:16:08.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.346+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "886d6f1f-a25b-4b98-a2c8-056eab1d62dc", "id": 6} v 0) v1 -- 0x7fd6c004e9b0 con 0x7fd6c00b8480 2023-10-16T14:16:08.355 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.352+0000 7fd6bcff9640 1 -- 172.21.15.170:0/2859717231 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "886d6f1f-a25b-4b98-a2c8-056eab1d62dc", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd6b0017070 con 0x7fd6c00b8480 2023-10-16T14:16:08.356 INFO:teuthology.orchestra.run.smithi170.stdout:6 2023-10-16T14:16:08.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 msgr2=0x7fd6c01f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c01f3130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009fd0 tx=0x7fd6b000ed40 comp rx=0 tx=0).stop 2023-10-16T14:16:08.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 shutdown_connections 2023-10-16T14:16:08.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c015cd70 0x7fd6c00b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00d0be0 0x7fd6c00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 --2- 172.21.15.170:0/2859717231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00b8480 0x7fd6c01f3130 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 >> 172.21.15.170:0/2859717231 conn(0x7fd6c00cf570 msgr2=0x7fd6c00cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:08.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 shutdown_connections 2023-10-16T14:16:08.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.353+0000 7fd6c4a59640 1 -- 172.21.15.170:0/2859717231 wait complete. 2023-10-16T14:16:08.373 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 324071bd-38bc-42a6-bde4-8b9546db1433 7 2023-10-16T14:16:08.390 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:08.386+0000 7f7fcea2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:16:08.442 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:08.439+0000 7f7fcea2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:16:08.608 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:08.604+0000 7f7fcea2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:16:08.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.605+0000 7fb905421640 1 Processor -- start 2023-10-16T14:16:08.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.605+0000 7fb905421640 1 -- start start 2023-10-16T14:16:08.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.606+0000 7fb905421640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb900065ab0 0x7fb9000c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.606+0000 7fb905421640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9000ce280 0x7fb9000ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.606+0000 7fb905421640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9000acd70 con 0x7fb90006d220 2023-10-16T14:16:08.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.606+0000 7fb905421640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9000ac4a0 con 0x7fb9000ce280 2023-10-16T14:16:08.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.606+0000 7fb905421640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb900065f40 con 0x7fb900065ab0 2023-10-16T14:16:08.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb900065ab0 0x7fb9000c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8fffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb90006d220 0x7fb9000cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.170:55988/0 (socket says 172.21.15.170:55988) 2023-10-16T14:16:08.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8fffff640 1 -- 172.21.15.170:0/4050909448 learned_addr learned my addr 172.21.15.170:0/4050909448 (peer_addr_for_me v1:172.21.15.170:0/0) 2023-10-16T14:16:08.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb904c20640 1 --2- 172.21.15.170:0/4050909448 >> v2:172.21.15.29:3300/0 conn(0x7fb9000ce280 0x7fb9000ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/4050909448 >> v2:172.21.15.29:3300/0 conn(0x7fb9000ce280 msgr2=0x7fb9000ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:08.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8ff7fe640 1 --2- 172.21.15.170:0/4050909448 >> v2:172.21.15.29:3300/0 conn(0x7fb9000ce280 0x7fb9000ad200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/4050909448 >> v1:172.21.15.170:6789/0 conn(0x7fb90006d220 legacy=0x7fb9000cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/4050909448 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000ac720 con 0x7fb900065ab0 2023-10-16T14:16:08.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb904c20640 1 --2- 172.21.15.170:0/4050909448 >> v2:172.21.15.29:3300/0 conn(0x7fb9000ce280 0x7fb9000ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:08.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.607+0000 7fb8feffd640 1 -- 172.21.15.170:0/4050909448 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981634494 0 0) 0x7fb9000acd70 con 0x7fb90006d220 2023-10-16T14:16:08.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.608+0000 7fb8ff7fe640 1 --2- 172.21.15.170:0/4050909448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb900065ab0 0x7fb9000c95e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40097a0 tx=0x7fb8f402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c485517107849a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:08.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.608+0000 7fb8feffd640 1 -- 172.21.15.170:0/4050909448 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f402d040 con 0x7fb900065ab0 2023-10-16T14:16:08.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb8feffd640 1 -- 172.21.15.170:0/4050909448 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f4031810 con 0x7fb900065ab0 2023-10-16T14:16:08.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb8feffd640 1 -- 172.21.15.170:0/4050909448 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4031ba0 con 0x7fb900065ab0 2023-10-16T14:16:08.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 -- 172.21.15.170:0/4050909448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb900065ab0 msgr2=0x7fb9000c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 --2- 172.21.15.170:0/4050909448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb900065ab0 0x7fb9000c95e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40097a0 tx=0x7fb8f402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 -- 172.21.15.170:0/4050909448 shutdown_connections 2023-10-16T14:16:08.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 --2- 172.21.15.170:0/4050909448 >> v2:172.21.15.29:3300/0 conn(0x7fb9000ce280 0x7fb9000ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.616 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 --2- 172.21.15.170:0/4050909448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb900065ab0 0x7fb9000c95e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 -- 172.21.15.170:0/4050909448 >> 172.21.15.170:0/4050909448 conn(0x7fb9000b0a80 msgr2=0x7fb9000b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:08.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 -- 172.21.15.170:0/4050909448 shutdown_connections 2023-10-16T14:16:08.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.609+0000 7fb905421640 1 -- 172.21.15.170:0/4050909448 wait complete. 2023-10-16T14:16:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.611+0000 7fb905421640 1 Processor -- start 2023-10-16T14:16:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.611+0000 7fb905421640 1 -- start start 2023-10-16T14:16:08.619 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb905421640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb90006d220 0x7fb900157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb905421640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 0x7fb9000c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb905421640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9000c6fb0 0x7fb90015e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:08.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb905421640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000b1d50 con 0x7fb90006d220 2023-10-16T14:16:08.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb905421640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000c7530 con 0x7fb9000ce280 2023-10-16T14:16:08.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb905421640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000c7810 con 0x7fb9000c6fb0 2023-10-16T14:16:08.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 0x7fb9000c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb904c20640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9000c6fb0 0x7fb90015e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 0x7fb9000c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:33806/0 (socket says 172.21.15.170:33806) 2023-10-16T14:16:08.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/2946622943 learned_addr learned my addr 172.21.15.170:0/2946622943 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-10-16T14:16:08.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.612+0000 7fb8fffff640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb90006d220 0x7fb900157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9000c6fb0 msgr2=0x7fb90015e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8ff7fe640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9000c6fb0 0x7fb90015e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/2946622943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb90006d220 msgr2=0x7fb900157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8ff7fe640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb90006d220 0x7fb900157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8ff7fe640 1 -- 172.21.15.170:0/2946622943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4009400 con 0x7fb9000ce280 2023-10-16T14:16:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb904c20640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9000c6fb0 0x7fb90015e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8ff7fe640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 0x7fb9000c6a70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000c00 tx=0x7fb8f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f403ad60 con 0x7fb9000ce280 2023-10-16T14:16:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f40443f0 con 0x7fb9000ce280 2023-10-16T14:16:08.623 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40447a0 con 0x7fb9000ce280 2023-10-16T14:16:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9000c7af0 con 0x7fb9000ce280 2023-10-16T14:16:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.613+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9000c80b0 con 0x7fb9000ce280 2023-10-16T14:16:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.614+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb8f4044c00 con 0x7fb9000ce280 2023-10-16T14:16:08.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.614+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb8f404d450 con 0x7fb9000ce280 2023-10-16T14:16:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.614+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9000c82a0 con 0x7fb9000ce280 2023-10-16T14:16:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.617+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb8f4035020 con 0x7fb9000ce280 2023-10-16T14:16:08.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.694+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd new", "uuid": "324071bd-38bc-42a6-bde4-8b9546db1433", "id": 7} v 0) v1 -- 0x7fb90004e9b0 con 0x7fb9000ce280 2023-10-16T14:16:08.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.700+0000 7fb8fcff9640 1 -- 172.21.15.170:0/2946622943 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "324071bd-38bc-42a6-bde4-8b9546db1433", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb8f404d6f0 con 0x7fb9000ce280 2023-10-16T14:16:08.704 INFO:teuthology.orchestra.run.smithi170.stdout:7 2023-10-16T14:16:08.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.701+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 msgr2=0x7fb9000c6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:08.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.701+0000 7fb905421640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 0x7fb9000c6a70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000c00 tx=0x7fb8f40046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.701+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 shutdown_connections 2023-10-16T14:16:08.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.701+0000 7fb905421640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9000c6fb0 0x7fb90015e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.701+0000 7fb905421640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9000ce280 0x7fb9000c6a70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.701+0000 7fb905421640 1 --2- 172.21.15.170:0/2946622943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb90006d220 0x7fb900157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:08.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.702+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 >> 172.21.15.170:0/2946622943 conn(0x7fb9000b0a80 msgr2=0x7fb9000afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:08.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.702+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 shutdown_connections 2023-10-16T14:16:08.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-16T14:16:08.702+0000 7fb905421640 1 -- 172.21.15.170:0/2946622943 wait complete. 2023-10-16T14:16:08.721 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-16T14:16:08.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-16T14:16:08.730 INFO:tasks.ceph.osd.0:Started 2023-10-16T14:16:08.731 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-16T14:16:08.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-16T14:16:08.740 INFO:tasks.ceph.osd.1:Started 2023-10-16T14:16:08.740 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-16T14:16:08.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-16T14:16:08.747 INFO:tasks.ceph.osd.2:Started 2023-10-16T14:16:08.747 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-16T14:16:08.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-16T14:16:08.755 INFO:tasks.ceph.osd.3:Started 2023-10-16T14:16:08.756 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-16T14:16:08.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-16T14:16:08.764 INFO:tasks.ceph.osd.4:Started 2023-10-16T14:16:08.765 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-16T14:16:08.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-16T14:16:08.773 INFO:tasks.ceph.osd.5:Started 2023-10-16T14:16:08.773 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-16T14:16:08.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-16T14:16:08.781 INFO:tasks.ceph.osd.6:Started 2023-10-16T14:16:08.781 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-16T14:16:08.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-16T14:16:08.787 INFO:tasks.ceph.osd.7:Started 2023-10-16T14:16:08.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:09.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.001+0000 7f12fcfd0640 1 Processor -- start 2023-10-16T14:16:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.002+0000 7f12fcfd0640 1 -- start start 2023-10-16T14:16:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12fcfd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80adb50 0x7f12f806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:09.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12fcfd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f8071450 con 0x7f12f8070fd0 2023-10-16T14:16:09.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12fcfd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f80cc600 con 0x7f12f80b0b20 2023-10-16T14:16:09.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12fcfd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80cc8e0 con 0x7f12f80adb50 2023-10-16T14:16:09.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f12f8070fd0 0x7f12f80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36916/0 (socket says 172.21.15.29:36916) 2023-10-16T14:16:09.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12effff640 1 -- 172.21.15.29:0/641809215 learned_addr learned my addr 172.21.15.29:0/641809215 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:09.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12f659c640 1 --2- 172.21.15.29:0/641809215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80adb50 0x7f12f806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:09.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12f659c640 1 -- 172.21.15.29:0/641809215 >> v1:172.21.15.170:6789/0 conn(0x7f12f80b0b20 legacy=0x7f12f8164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:09.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.003+0000 7f12f659c640 1 -- 172.21.15.29:0/641809215 >> v1:172.21.15.29:6789/0 conn(0x7f12f8070fd0 legacy=0x7f12f80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:09.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.004+0000 7f12f659c640 1 -- 172.21.15.29:0/641809215 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f80ccbc0 con 0x7f12f80adb50 2023-10-16T14:16:09.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12f659c640 1 --2- 172.21.15.29:0/641809215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80adb50 0x7f12f806f550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f12e80097d0 tx=0x7f12e802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5541ae15c112f269 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:09.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12f5d9b640 1 -- 172.21.15.29:0/641809215 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8030610 con 0x7f12f80adb50 2023-10-16T14:16:09.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12f5d9b640 1 -- 172.21.15.29:0/641809215 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e8030770 con 0x7f12f80adb50 2023-10-16T14:16:09.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12f5d9b640 1 -- 172.21.15.29:0/641809215 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8030b40 con 0x7f12f80adb50 2023-10-16T14:16:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 -- 172.21.15.29:0/641809215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80adb50 msgr2=0x7f12f806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 --2- 172.21.15.29:0/641809215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80adb50 0x7f12f806f550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f12e80097d0 tx=0x7f12e802b960 comp rx=0 tx=0).stop 2023-10-16T14:16:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 -- 172.21.15.29:0/641809215 shutdown_connections 2023-10-16T14:16:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 --2- 172.21.15.29:0/641809215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80adb50 0x7f12f806f550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:09.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 -- 172.21.15.29:0/641809215 >> 172.21.15.29:0/641809215 conn(0x7f12f80cb410 msgr2=0x7f12f80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:09.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 -- 172.21.15.29:0/641809215 shutdown_connections 2023-10-16T14:16:09.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.005+0000 7f12fcfd0640 1 -- 172.21.15.29:0/641809215 wait complete. 2023-10-16T14:16:09.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.007+0000 7f12fcfd0640 1 Processor -- start 2023-10-16T14:16:09.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.007+0000 7f12fcfd0640 1 -- start start 2023-10-16T14:16:09.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12fcfd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f8070fd0 0x7f12f80b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12fcfd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 0x7f12f80b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12fcfd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80b0b20 0x7f12f80b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12fcfd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80b1e90 con 0x7f12f80adb50 2023-10-16T14:16:09.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12fcfd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80b2170 con 0x7f12f8070fd0 2023-10-16T14:16:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12fcfd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80b2450 con 0x7f12f80b0b20 2023-10-16T14:16:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 0x7f12f80b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 0x7f12f80b39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51274/0 (socket says 172.21.15.29:51274) 2023-10-16T14:16:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12effff640 1 -- 172.21.15.29:0/379257871 learned_addr learned my addr 172.21.15.29:0/379257871 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12f659c640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f8070fd0 0x7f12f80b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.008+0000 7f12f6d9d640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80b0b20 0x7f12f80b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12effff640 1 -- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80b0b20 msgr2=0x7f12f80b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12effff640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80b0b20 0x7f12f80b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12effff640 1 -- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f8070fd0 msgr2=0x7f12f80b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12effff640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f8070fd0 0x7f12f80b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:09.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12effff640 1 -- 172.21.15.29:0/379257871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e8009400 con 0x7f12f80adb50 2023-10-16T14:16:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12f659c640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f8070fd0 0x7f12f80b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12f6d9d640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80b0b20 0x7f12f80b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.009+0000 7f12effff640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 0x7f12f80b39a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c480 tx=0x7f12e000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0005ce0 con 0x7f12f80adb50 2023-10-16T14:16:09.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0005e40 con 0x7f12f80adb50 2023-10-16T14:16:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e00072f0 con 0x7f12f80adb50 2023-10-16T14:16:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f80b2730 con 0x7f12f80adb50 2023-10-16T14:16:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f80b2f30 con 0x7f12f80adb50 2023-10-16T14:16:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f12e001c440 con 0x7f12f80adb50 2023-10-16T14:16:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.010+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f12e0012ad0 con 0x7f12f80adb50 2023-10-16T14:16:09.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.011+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f80b2c50 con 0x7f12f80adb50 2023-10-16T14:16:09.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.013+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f12f80b2c50 con 0x7f12f80adb50 2023-10-16T14:16:09.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.078+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f804e160 con 0x7f12f80adb50 2023-10-16T14:16:09.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.079+0000 7f12eeffd640 1 -- 172.21.15.29:0/379257871 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f12e002a090 con 0x7f12f80adb50 2023-10-16T14:16:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:09.082 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:08.697085+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 msgr2=0x7f12f80b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:09.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 0x7f12f80b39a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c480 tx=0x7f12e000c940 comp rx=0 tx=0).stop 2023-10-16T14:16:09.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 shutdown_connections 2023-10-16T14:16:09.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12f80adb50 0x7f12f80b39a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:09.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12f80b0b20 0x7f12f80b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:09.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 --2- 172.21.15.29:0/379257871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12f8070fd0 0x7f12f80b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:09.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 >> 172.21.15.29:0/379257871 conn(0x7f12f80cb410 msgr2=0x7f12f80aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:09.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.080+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 shutdown_connections 2023-10-16T14:16:09.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:09.081+0000 7f12fcfd0640 1 -- 172.21.15.29:0/379257871 wait complete. 2023-10-16T14:16:09.098 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-16T14:16:09.099 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-16T14:16:09.311 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:09.307+0000 7f9df102c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:16:09.440 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-16T14:16:09.437+0000 7f24e5613500 -1 Falling back to public interface 2023-10-16T14:16:09.449 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-10-16T14:16:09.445+0000 7fae56c0f500 -1 Falling back to public interface 2023-10-16T14:16:09.454 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-10-16T14:16:09.450+0000 7f9f40f1a500 -1 Falling back to public interface 2023-10-16T14:16:09.461 INFO:tasks.ceph.osd.6.smithi170.stderr:2023-10-16T14:16:09.457+0000 7f99fd6fa500 -1 Falling back to public interface 2023-10-16T14:16:09.467 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-16T14:16:09.463+0000 7fa2a9c0e500 -1 Falling back to public interface 2023-10-16T14:16:09.480 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-16T14:16:09.477+0000 7f36ce49e500 -1 Falling back to public interface 2023-10-16T14:16:09.481 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-16T14:16:09.477+0000 7fa7b2c69500 -1 Falling back to public interface 2023-10-16T14:16:09.486 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-16T14:16:09.484+0000 7fee62ea5500 -1 Falling back to public interface 2023-10-16T14:16:09.616 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:09.612+0000 7f9df102c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:16:09.675 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:09.671+0000 7f9df102c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:16:09.791 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:09.787+0000 7f9df102c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:16:09.850 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:09.847+0000 7f7fcea2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:16:09.904 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:10.076 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.073+0000 7f859dd39640 1 Processor -- start 2023-10-16T14:16:10.077 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.073+0000 7f859dd39640 1 -- start start 2023-10-16T14:16:10.078 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.074+0000 7f859dd39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85980cd510 0x7f85980cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:10.079 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.074+0000 7f859dd39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85980ce7f0 0x7f85980cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:10.079 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f859dd39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980cc940 con 0x7f859815f220 2023-10-16T14:16:10.079 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f859dd39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980d1560 con 0x7f85980cd510 2023-10-16T14:16:10.079 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f859dd39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980cbd30 con 0x7f85980ce7f0 2023-10-16T14:16:10.079 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85980cd510 0x7f85980cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:10.080 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85980cd510 0x7f85980cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51276/0 (socket says 172.21.15.29:51276) 2023-10-16T14:16:10.080 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 -- 172.21.15.29:0/2407126872 learned_addr learned my addr 172.21.15.29:0/2407126872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:10.080 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f8597fff640 1 --2- 172.21.15.29:0/2407126872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85980ce7f0 0x7f85980cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:10.080 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 -- 172.21.15.29:0/2407126872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85980ce7f0 msgr2=0x7f85980cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:10.080 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 --2- 172.21.15.29:0/2407126872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85980ce7f0 0x7f85980cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.081 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 -- 172.21.15.29:0/2407126872 >> v1:172.21.15.170:6789/0 conn(0x7f859815f220 legacy=0x7f85980cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:10.081 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.075+0000 7f85977fe640 1 -- 172.21.15.29:0/2407126872 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980cc010 con 0x7f85980cd510 2023-10-16T14:16:10.082 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.077+0000 7f85977fe640 1 --2- 172.21.15.29:0/2407126872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980cd510 0x7f85980cb730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8588009850 tx=0x7f858802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480a8a3eb7cb420f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:10.082 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.077+0000 7f85967fc640 1 -- 172.21.15.29:0/2407126872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588004230 con 0x7f85980cd510 2023-10-16T14:16:10.083 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.077+0000 7f85967fc640 1 -- 172.21.15.29:0/2407126872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8588004390 con 0x7f85980cd510 2023-10-16T14:16:10.083 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.077+0000 7f85967fc640 1 -- 172.21.15.29:0/2407126872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588003810 con 0x7f85980cd510 2023-10-16T14:16:10.083 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.077+0000 7f859dd39640 1 -- 172.21.15.29:0/2407126872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980cd510 msgr2=0x7f85980cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:10.083 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.077+0000 7f859dd39640 1 --2- 172.21.15.29:0/2407126872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980cd510 0x7f85980cb730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8588009850 tx=0x7f858802b960 comp rx=0 tx=0).stop 2023-10-16T14:16:10.083 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.078+0000 7f859dd39640 1 -- 172.21.15.29:0/2407126872 shutdown_connections 2023-10-16T14:16:10.084 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.078+0000 7f859dd39640 1 --2- 172.21.15.29:0/2407126872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85980ce7f0 0x7f85980cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.084 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.078+0000 7f859dd39640 1 --2- 172.21.15.29:0/2407126872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980cd510 0x7f85980cb730 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.084 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.078+0000 7f859dd39640 1 -- 172.21.15.29:0/2407126872 >> 172.21.15.29:0/2407126872 conn(0x7f8598099ea0 msgr2=0x7f85980d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:10.084 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.078+0000 7f859dd39640 1 -- 172.21.15.29:0/2407126872 shutdown_connections 2023-10-16T14:16:10.084 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.078+0000 7f859dd39640 1 -- 172.21.15.29:0/2407126872 wait complete. 2023-10-16T14:16:10.085 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.079+0000 7f859dd39640 1 Processor -- start 2023-10-16T14:16:10.085 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.080+0000 7f859dd39640 1 -- start start 2023-10-16T14:16:10.087 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.080+0000 7f859dd39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85980cd510 0x7f859814b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:10.087 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.080+0000 7f859dd39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 0x7f859806bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:10.087 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f859dd39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f859815f220 0x7f85980b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:10.088 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f859dd39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859806bb60 con 0x7f85980ce7f0 2023-10-16T14:16:10.088 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f859dd39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859814ba60 con 0x7f85980cd510 2023-10-16T14:16:10.088 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f859dd39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859806af50 con 0x7f859815f220 2023-10-16T14:16:10.088 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 0x7f859806bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:10.088 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 0x7f859806bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51292/0 (socket says 172.21.15.29:51292) 2023-10-16T14:16:10.089 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 -- 172.21.15.29:0/791055445 learned_addr learned my addr 172.21.15.29:0/791055445 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:10.089 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f85977fe640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85980cd510 0x7f859814b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:10.089 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8597fff640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f859815f220 0x7f85980b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:10.089 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 -- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f859815f220 msgr2=0x7f85980b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:10.089 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f859815f220 0x7f85980b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.090 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 -- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85980cd510 msgr2=0x7f859814b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:10.090 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85980cd510 0x7f859814b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.090 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.081+0000 7f8596ffd640 1 -- 172.21.15.29:0/791055445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588009460 con 0x7f85980ce7f0 2023-10-16T14:16:10.090 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.082+0000 7f8596ffd640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 0x7f859806bff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f858000c730 tx=0x7f858000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:10.091 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.082+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580003b70 con 0x7f85980ce7f0 2023-10-16T14:16:10.091 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.082+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8580003cf0 con 0x7f85980ce7f0 2023-10-16T14:16:10.091 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.082+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859806b230 con 0x7f85980ce7f0 2023-10-16T14:16:10.091 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.082+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580003ea0 con 0x7f85980ce7f0 2023-10-16T14:16:10.091 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.082+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859806b7f0 con 0x7f85980ce7f0 2023-10-16T14:16:10.091 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.083+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859806bd10 con 0x7f85980ce7f0 2023-10-16T14:16:10.092 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.084+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f858000aa90 con 0x7f85980ce7f0 2023-10-16T14:16:10.092 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.084+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f85800493e0 con 0x7f85980ce7f0 2023-10-16T14:16:10.092 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.086+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f858000ce40 con 0x7f85980ce7f0 2023-10-16T14:16:10.155 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.152+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85980cb730 con 0x7f85980ce7f0 2023-10-16T14:16:10.156 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.153+0000 7f858ffff640 1 -- 172.21.15.29:0/791055445 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8580011380 con 0x7f85980ce7f0 2023-10-16T14:16:10.156 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-16T14:16:10.157 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:08.697085+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:10.158 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 msgr2=0x7f859806bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:10.158 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 0x7f859806bff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f858000c730 tx=0x7f858000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.158 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 shutdown_connections 2023-10-16T14:16:10.158 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ce7f0 0x7f859806bff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.158 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f859815f220 0x7f85980b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.159 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 --2- 172.21.15.29:0/791055445 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85980cd510 0x7f859814b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:10.159 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 >> 172.21.15.29:0/791055445 conn(0x7f8598099ea0 msgr2=0x7f859815e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:10.159 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.154+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 shutdown_connections 2023-10-16T14:16:10.159 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:10.155+0000 7f859dd39640 1 -- 172.21.15.29:0/791055445 wait complete. 2023-10-16T14:16:10.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-16T14:16:10.196 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:10.193+0000 7f7fcea2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:16:10.259 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:10.256+0000 7f7fcea2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:16:10.379 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:16:10.376+0000 7f7fcea2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:16:10.827 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:10.824+0000 7f9d91bab640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-16T14:16:10.833 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:16:10.831+0000 7f9d91bab640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-16T14:16:12.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:16:12.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:16:13.013 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-10-16T14:16:13.010+0000 7f9f40f1a500 -1 osd.4 0 log_to_monitors true 2023-10-16T14:16:13.050 INFO:tasks.ceph.osd.6.smithi170.stderr:2023-10-16T14:16:13.048+0000 7f99fd6fa500 -1 osd.6 0 log_to_monitors true 2023-10-16T14:16:13.054 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-16T14:16:13.051+0000 7fee62ea5500 -1 osd.3 0 log_to_monitors true 2023-10-16T14:16:13.062 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-16T14:16:13.060+0000 7fa2a9c0e500 -1 osd.7 0 log_to_monitors true 2023-10-16T14:16:13.066 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-16T14:16:13.063+0000 7fa7b2c69500 -1 osd.2 0 log_to_monitors true 2023-10-16T14:16:13.213 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-10-16T14:16:13.210+0000 7fae56c0f500 -1 osd.5 0 log_to_monitors true 2023-10-16T14:16:13.240 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-16T14:16:13.237+0000 7f24e5613500 -1 osd.1 0 log_to_monitors true 2023-10-16T14:16:13.320 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-16T14:16:13.317+0000 7f36ce49e500 -1 osd.0 0 log_to_monitors true 2023-10-16T14:16:16.590 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-16T14:16:16.587+0000 7f24e1f68640 -1 osd.1 0 waiting for initial osdmap 2023-10-16T14:16:16.606 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-16T14:16:16.603+0000 7f24d8d60640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.639 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-16T14:16:16.636+0000 7fa7ae3d4640 -1 osd.2 0 waiting for initial osdmap 2023-10-16T14:16:16.653 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-16T14:16:16.650+0000 7fa7a6a06640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.659 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-16T14:16:16.656+0000 7fee5e60e640 -1 osd.3 0 waiting for initial osdmap 2023-10-16T14:16:16.673 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-16T14:16:16.670+0000 7fee56c40640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.697 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-16T14:16:16.694+0000 7f36c9c07640 -1 osd.0 0 waiting for initial osdmap 2023-10-16T14:16:16.698 INFO:tasks.ceph.osd.6.smithi170.stderr:2023-10-16T14:16:16.695+0000 7f99f8e63640 -1 osd.6 0 waiting for initial osdmap 2023-10-16T14:16:16.705 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-10-16T14:16:16.703+0000 7f9f3c683640 -1 osd.4 0 waiting for initial osdmap 2023-10-16T14:16:16.710 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-16T14:16:16.707+0000 7f36c2239640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.721 INFO:tasks.ceph.osd.6.smithi170.stderr:2023-10-16T14:16:16.718+0000 7f99f1495640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.722 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-10-16T14:16:16.719+0000 7f9f34cb5640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.754 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-10-16T14:16:16.752+0000 7fae535e2640 -1 osd.5 0 waiting for initial osdmap 2023-10-16T14:16:16.756 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-16T14:16:16.753+0000 7fa2a6568640 -1 osd.7 0 waiting for initial osdmap 2023-10-16T14:16:16.769 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-10-16T14:16:16.766+0000 7fae4a3da640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.778 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-16T14:16:16.775+0000 7fa29d360640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T14:16:16.975 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:17.167 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.164+0000 7fe42de1a640 1 Processor -- start 2023-10-16T14:16:17.168 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.164+0000 7fe42de1a640 1 -- start start 2023-10-16T14:16:17.169 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.166+0000 7fe42de1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe428159950 0x7fe42815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.169 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.166+0000 7fe42de1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42815aef0 con 0x7fe42806c810 2023-10-16T14:16:17.169 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.166+0000 7fe42de1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42815d910 con 0x7fe428070450 2023-10-16T14:16:17.170 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.166+0000 7fe42de1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42815e490 con 0x7fe428159950 2023-10-16T14:16:17.170 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.166+0000 7fe42d619640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe42806c810 0x7fe4280c9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52216/0 (socket says 172.21.15.29:52216) 2023-10-16T14:16:17.170 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.166+0000 7fe42d619640 1 -- 172.21.15.29:0/3541106212 learned_addr learned my addr 172.21.15.29:0/3541106212 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:17.172 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269494108 0 0) 0x7fe420001490 con 0x7fe42806c810 2023-10-16T14:16:17.173 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe42ce18640 1 --2- 172.21.15.29:0/3541106212 >> v2:172.21.15.170:3300/0 conn(0x7fe428159950 0x7fe42815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:17.173 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe418005730 con 0x7fe42806c810 2023-10-16T14:16:17.173 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3964107561 0 0) 0x7fe418005730 con 0x7fe42806c810 2023-10-16T14:16:17.173 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 >> v2:172.21.15.170:3300/0 conn(0x7fe428159950 msgr2=0x7fe42815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:17.173 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 --2- 172.21.15.29:0/3541106212 >> v2:172.21.15.170:3300/0 conn(0x7fe428159950 0x7fe42815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.174 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 >> v1:172.21.15.170:6789/0 conn(0x7fe428070450 legacy=0x7fe42806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.174 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42815dbf0 con 0x7fe42806c810 2023-10-16T14:16:17.174 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe420001b80 con 0x7fe42806c810 2023-10-16T14:16:17.174 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 >> v1:172.21.15.29:6789/0 conn(0x7fe42806c810 legacy=0x7fe4280c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.175 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe418005b80 0x7fe418005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.175 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4180069d0 0x7fe418006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.175 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180074b0 0x7fe41800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.175 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42815e490 con 0x7fe4180069d0 2023-10-16T14:16:17.175 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41800e380 con 0x7fe4180074b0 2023-10-16T14:16:17.175 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.167+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41800e540 con 0x7fe418005b80 2023-10-16T14:16:17.176 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe427fff640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4180069d0 0x7fe418006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:17.176 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe42ce18640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe418005b80 0x7fe418005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:17.176 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe42d619640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180074b0 0x7fe41800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:17.176 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe427fff640 1 -- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe418005b80 msgr2=0x7fe418005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:17.176 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe427fff640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe418005b80 0x7fe418005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.177 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe427fff640 1 -- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180074b0 msgr2=0x7fe41800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:17.177 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe427fff640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180074b0 0x7fe41800db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.177 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe427fff640 1 -- 172.21.15.29:0/3541106212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418005730 con 0x7fe4180069d0 2023-10-16T14:16:17.177 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe42d619640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180074b0 0x7fe41800db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:17.178 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe42ce18640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe418005b80 0x7fe418005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:17.178 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.168+0000 7fe42d619640 1 -- 172.21.15.29:0/3541106212 reap_dead start 2023-10-16T14:16:17.178 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.169+0000 7fe427fff640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4180069d0 0x7fe418006e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe414001720 tx=0x7fe418007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f7dbbe991ab906 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:17.178 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.169+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4140052d0 con 0x7fe4180069d0 2023-10-16T14:16:17.179 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.169+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41402e050 con 0x7fe4180069d0 2023-10-16T14:16:17.179 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.169+0000 7fe4277fe640 1 -- 172.21.15.29:0/3541106212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414003830 con 0x7fe4180069d0 2023-10-16T14:16:17.179 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 -- 172.21.15.29:0/3541106212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4180069d0 msgr2=0x7fe418006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.179 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4180069d0 0x7fe418006e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe414001720 tx=0x7fe418007a00 comp rx=0 tx=0).stop 2023-10-16T14:16:17.182 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 -- 172.21.15.29:0/3541106212 shutdown_connections 2023-10-16T14:16:17.182 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 --2- 172.21.15.29:0/3541106212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4180069d0 0x7fe418006e50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.182 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 -- 172.21.15.29:0/3541106212 >> 172.21.15.29:0/3541106212 conn(0x7fe4280ce6a0 msgr2=0x7fe4280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:17.183 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 -- 172.21.15.29:0/3541106212 shutdown_connections 2023-10-16T14:16:17.183 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.170+0000 7fe42de1a640 1 -- 172.21.15.29:0/3541106212 wait complete. 2023-10-16T14:16:17.183 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.172+0000 7fe42de1a640 1 Processor -- start 2023-10-16T14:16:17.183 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.172+0000 7fe42de1a640 1 -- start start 2023-10-16T14:16:17.183 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe42de1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180069d0 0x7fe428155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.184 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe42de1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 0x7fe4280aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.184 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe42de1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4280c2940 0x7fe4280c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.184 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe42de1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4280c3300 con 0x7fe4280aea80 2023-10-16T14:16:17.184 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe42de1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe428156480 con 0x7fe4180069d0 2023-10-16T14:16:17.184 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe42de1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42815b7a0 con 0x7fe4280c2940 2023-10-16T14:16:17.185 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 0x7fe4280aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:17.185 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 0x7fe4280aeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42866/0 (socket says 172.21.15.29:42866) 2023-10-16T14:16:17.185 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 -- 172.21.15.29:0/2100835301 learned_addr learned my addr 172.21.15.29:0/2100835301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:17.185 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 -- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4280c2940 msgr2=0x7fe4280c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.185 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4280c2940 0x7fe4280c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.186 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 -- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180069d0 msgr2=0x7fe428155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.186 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180069d0 0x7fe428155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.186 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.173+0000 7fe427fff640 1 -- 172.21.15.29:0/2100835301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4280af500 con 0x7fe4280aea80 2023-10-16T14:16:17.186 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.174+0000 7fe427fff640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 0x7fe4280aeee0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe414005040 tx=0x7fe414001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:17.186 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.174+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4140076c0 con 0x7fe4280aea80 2023-10-16T14:16:17.187 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.174+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe414007a30 con 0x7fe4280aea80 2023-10-16T14:16:17.187 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.174+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414006be0 con 0x7fe4280aea80 2023-10-16T14:16:17.187 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.174+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4280af7e0 con 0x7fe4280aea80 2023-10-16T14:16:17.187 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.174+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42815bcd0 con 0x7fe4280aea80 2023-10-16T14:16:17.187 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.175+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42804e9b0 con 0x7fe4280aea80 2023-10-16T14:16:17.188 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.176+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fe414006d40 con 0x7fe4280aea80 2023-10-16T14:16:17.188 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.177+0000 7fe4257fa640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe40c06ae10 0x7fe40c06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:17.188 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.177+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe4140aa3e0 con 0x7fe4280aea80 2023-10-16T14:16:17.188 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.178+0000 7fe42ce18640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe40c06ae10 0x7fe40c06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:17.189 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.179+0000 7fe42ce18640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe40c06ae10 0x7fe40c06d2d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe428072d70 tx=0x7fe41c009340 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:17.189 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.182+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4140762f0 con 0x7fe4280aea80 2023-10-16T14:16:17.370 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.367+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4280b4620 con 0x7fe4280aea80 2023-10-16T14:16:17.371 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.368+0000 7fe4257fa640 1 -- 172.21.15.29:0/2100835301 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe414041090 con 0x7fe4280aea80 2023-10-16T14:16:17.371 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-16T14:16:17.372 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":12,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:14.878600+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:17.373 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe40c06ae10 msgr2=0x7fe40c06d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.374 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe40c06ae10 0x7fe40c06d2d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe428072d70 tx=0x7fe41c009340 comp rx=0 tx=0).stop 2023-10-16T14:16:17.374 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 msgr2=0x7fe4280aeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:17.374 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 0x7fe4280aeee0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe414005040 tx=0x7fe414001120 comp rx=0 tx=0).stop 2023-10-16T14:16:17.374 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 shutdown_connections 2023-10-16T14:16:17.375 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe40c06ae10 0x7fe40c06d2d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.375 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4280aea80 0x7fe4280aeee0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.375 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4280c2940 0x7fe4280c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.375 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 --2- 172.21.15.29:0/2100835301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4180069d0 0x7fe428155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:17.375 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 >> 172.21.15.29:0/2100835301 conn(0x7fe4280ce6a0 msgr2=0x7fe42806bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:17.376 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 shutdown_connections 2023-10-16T14:16:17.376 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:17.370+0000 7fe42de1a640 1 -- 172.21.15.29:0/2100835301 wait complete. 2023-10-16T14:16:17.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-16T14:16:24.193 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:24.391 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.388+0000 7f42c759e640 1 Processor -- start 2023-10-16T14:16:24.391 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.388+0000 7f42c759e640 1 -- start start 2023-10-16T14:16:24.393 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.389+0000 7f42c759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42c815cfc0 0x7f42c80b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.393 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.389+0000 7f42c759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42c80d2640 0x7f42c80b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.393 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.389+0000 7f42c759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c80d15a0 con 0x7f42c815cbe0 2023-10-16T14:16:24.393 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.389+0000 7f42c759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c80d1880 con 0x7f42c80d2640 2023-10-16T14:16:24.394 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.389+0000 7f42c759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c80d1b60 con 0x7f42c815cfc0 2023-10-16T14:16:24.394 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42c80d2640 0x7f42c80b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.394 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42c80d2640 0x7f42c80b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60544/0 (socket says 172.21.15.29:60544) 2023-10-16T14:16:24.394 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 -- 172.21.15.29:0/1336532009 learned_addr learned my addr 172.21.15.29:0/1336532009 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:24.394 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c5d9b640 1 --2- 172.21.15.29:0/1336532009 >> v2:172.21.15.170:3300/0 conn(0x7f42c815cfc0 0x7f42c80b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.396 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 -- 172.21.15.29:0/1336532009 >> v2:172.21.15.170:3300/0 conn(0x7f42c815cfc0 msgr2=0x7f42c80b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:24.396 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 --2- 172.21.15.29:0/1336532009 >> v2:172.21.15.170:3300/0 conn(0x7f42c815cfc0 0x7f42c80b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.397 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 -- 172.21.15.29:0/1336532009 >> v1:172.21.15.170:6789/0 conn(0x7f42c815cbe0 legacy=0x7f42c80d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:24.397 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.390+0000 7f42c6d9d640 1 -- 172.21.15.29:0/1336532009 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c80d1e40 con 0x7f42c80d2640 2023-10-16T14:16:24.397 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.391+0000 7f42c6d9d640 1 --2- 172.21.15.29:0/1336532009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80b4ab0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f42c80cf4b0 tx=0x7f42c002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e0549e7e28ecc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:24.397 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c559a640 1 -- 172.21.15.29:0/1336532009 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c002cb20 con 0x7f42c80d2640 2023-10-16T14:16:24.397 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c559a640 1 -- 172.21.15.29:0/1336532009 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c002cc80 con 0x7f42c80d2640 2023-10-16T14:16:24.398 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c559a640 1 -- 172.21.15.29:0/1336532009 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c00343f0 con 0x7f42c80d2640 2023-10-16T14:16:24.398 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c759e640 1 -- 172.21.15.29:0/1336532009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 msgr2=0x7f42c80b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.398 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c759e640 1 --2- 172.21.15.29:0/1336532009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80b4ab0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f42c80cf4b0 tx=0x7f42c002bf80 comp rx=0 tx=0).stop 2023-10-16T14:16:24.398 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c759e640 1 -- 172.21.15.29:0/1336532009 shutdown_connections 2023-10-16T14:16:24.398 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c759e640 1 --2- 172.21.15.29:0/1336532009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80b4ab0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.399 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c759e640 1 --2- 172.21.15.29:0/1336532009 >> v2:172.21.15.170:3300/0 conn(0x7f42c815cfc0 0x7f42c80b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.399 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.392+0000 7f42c759e640 1 -- 172.21.15.29:0/1336532009 >> 172.21.15.29:0/1336532009 conn(0x7f42c80cf570 msgr2=0x7f42c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:24.399 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.393+0000 7f42c759e640 1 -- 172.21.15.29:0/1336532009 shutdown_connections 2023-10-16T14:16:24.399 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.393+0000 7f42c759e640 1 -- 172.21.15.29:0/1336532009 wait complete. 2023-10-16T14:16:24.402 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.395+0000 7f42c759e640 1 Processor -- start 2023-10-16T14:16:24.402 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.395+0000 7f42c759e640 1 -- start start 2023-10-16T14:16:24.402 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.402 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42c815cbe0 0x7f42c80b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.402 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42c815cfc0 0x7f42c8156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.403 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80c6fe0 con 0x7f42c80d2640 2023-10-16T14:16:24.403 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80b74c0 con 0x7f42c815cfc0 2023-10-16T14:16:24.403 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80c7ac0 con 0x7f42c815cbe0 2023-10-16T14:16:24.403 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.403 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60546/0 (socket says 172.21.15.29:60546) 2023-10-16T14:16:24.404 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c659c640 1 -- 172.21.15.29:0/1516006506 learned_addr learned my addr 172.21.15.29:0/1516006506 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:24.404 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c659c640 1 -- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42c815cbe0 msgr2=0x7f42c80b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.404 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.396+0000 7f42c6d9d640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42c815cfc0 0x7f42c8156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.404 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c659c640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42c815cbe0 0x7f42c80b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.404 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c659c640 1 -- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42c815cfc0 msgr2=0x7f42c8156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.405 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c659c640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42c815cfc0 0x7f42c8156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.405 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c659c640 1 -- 172.21.15.29:0/1516006506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c0009b30 con 0x7f42c80d2640 2023-10-16T14:16:24.405 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c6d9d640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42c815cfc0 0x7f42c8156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:24.405 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c659c640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80cc630 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c460 tx=0x7f42b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:24.405 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8012830 con 0x7f42c80d2640 2023-10-16T14:16:24.406 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b80129d0 con 0x7f42c80d2640 2023-10-16T14:16:24.406 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8012d80 con 0x7f42c80d2640 2023-10-16T14:16:24.406 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c80c71d0 con 0x7f42c80d2640 2023-10-16T14:16:24.406 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.397+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c80c7750 con 0x7f42c80d2640 2023-10-16T14:16:24.406 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.398+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c80d11e0 con 0x7f42c80d2640 2023-10-16T14:16:24.407 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.400+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f42b801a3b0 con 0x7f42c80d2640 2023-10-16T14:16:24.408 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.400+0000 7f42b6ffd640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f42a806aea0 0x7f42a806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.408 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.401+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f42b8088fa0 con 0x7f42c80d2640 2023-10-16T14:16:24.408 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.401+0000 7f42c5d9b640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f42a806aea0 0x7f42a806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.408 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.402+0000 7f42c5d9b640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f42a806aea0 0x7f42a806d360 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f42bc006fd0 tx=0x7f42bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:24.409 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.406+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42b8054300 con 0x7f42c80d2640 2023-10-16T14:16:24.604 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.600+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42c804e9b0 con 0x7f42c80d2640 2023-10-16T14:16:24.606 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.603+0000 7f42b6ffd640 1 -- 172.21.15.29:0/1516006506 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f42b8057920 con 0x7f42c80d2640 2023-10-16T14:16:24.606 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-16T14:16:24.606 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":16,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:20.609906+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:24.608 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f42a806aea0 msgr2=0x7f42a806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.609 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f42a806aea0 0x7f42a806d360 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f42bc006fd0 tx=0x7f42bc008040 comp rx=0 tx=0).stop 2023-10-16T14:16:24.609 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 msgr2=0x7f42c80cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.609 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80cc630 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c460 tx=0x7f42b800c920 comp rx=0 tx=0).stop 2023-10-16T14:16:24.609 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 shutdown_connections 2023-10-16T14:16:24.610 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f42a806aea0 0x7f42a806d360 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.610 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42c815cfc0 0x7f42c8156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.610 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42c815cbe0 0x7f42c80b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.610 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 --2- 172.21.15.29:0/1516006506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42c80d2640 0x7f42c80cc630 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.610 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 >> 172.21.15.29:0/1516006506 conn(0x7f42c80cf570 msgr2=0x7f42c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:24.611 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.605+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 shutdown_connections 2023-10-16T14:16:24.611 INFO:teuthology.misc.health.smithi029.stderr:2023-10-16T14:16:24.606+0000 7f42c759e640 1 -- 172.21.15.29:0/1516006506 wait complete. 2023-10-16T14:16:24.623 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-16T14:16:24.624 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-16T14:16:24.624 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:16:24.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:16:24.633 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:16:24.633 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:16:24.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:16:24.641 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:16:24.642 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:16:24.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:16:24.652 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:16:24.652 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:16:24.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:16:24.663 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:16:24.663 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-16T14:16:24.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:24.794 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:16:24.802 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:16:24.819 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:16:24.820 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:16:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.853+0000 7f3b32687640 1 Processor -- start 2023-10-16T14:16:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.854+0000 7f3b32687640 1 -- start start 2023-10-16T14:16:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b32687640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3b2c06c3f0 0x7f3b2c06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b32687640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0cd630 con 0x7f3b2c15e0c0 2023-10-16T14:16:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b32687640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0ce1b0 con 0x7f3b2c15e540 2023-10-16T14:16:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b32687640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c06cd10 con 0x7f3b2c06c3f0 2023-10-16T14:16:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b2b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3b2c15e0c0 0x7f3b2c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36664/0 (socket says 172.21.15.29:36664) 2023-10-16T14:16:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b2b7fe640 1 -- 172.21.15.29:0/2065892973 learned_addr learned my addr 172.21.15.29:0/2065892973 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.855+0000 7f3b2bfff640 1 --2- 172.21.15.29:0/2065892973 >> v2:172.21.15.170:3300/0 conn(0x7f3b2c06c3f0 0x7f3b2c06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931541480 0 0) 0x7f3b2c0cd630 con 0x7f3b2c15e0c0 2023-10-16T14:16:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b14003520 con 0x7f3b2c15e0c0 2023-10-16T14:16:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876713809 0 0) 0x7f3b2c0ce1b0 con 0x7f3b2c15e540 2023-10-16T14:16:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c0cd630 con 0x7f3b2c15e540 2023-10-16T14:16:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2bfff640 1 -- 172.21.15.29:0/2065892973 >> v1:172.21.15.170:6789/0 conn(0x7f3b2c15e540 legacy=0x7f3b2c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2bfff640 1 -- 172.21.15.29:0/2065892973 >> v1:172.21.15.29:6789/0 conn(0x7f3b2c15e0c0 legacy=0x7f3b2c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.856+0000 7f3b2bfff640 1 -- 172.21.15.29:0/2065892973 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0d2170 con 0x7f3b2c06c3f0 2023-10-16T14:16:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.857+0000 7f3b2bfff640 1 --2- 172.21.15.29:0/2065892973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c06c7d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0097a0 tx=0x7f3b1c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b8dedad97be724 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.857+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c030610 con 0x7f3b2c06c3f0 2023-10-16T14:16:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.857+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c030770 con 0x7f3b2c06c3f0 2023-10-16T14:16:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.857+0000 7f3b2affd640 1 -- 172.21.15.29:0/2065892973 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c030ae0 con 0x7f3b2c06c3f0 2023-10-16T14:16:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 -- 172.21.15.29:0/2065892973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 msgr2=0x7f3b2c06c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 --2- 172.21.15.29:0/2065892973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c06c7d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0097a0 tx=0x7f3b1c02b960 comp rx=0 tx=0).stop 2023-10-16T14:16:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 -- 172.21.15.29:0/2065892973 shutdown_connections 2023-10-16T14:16:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 --2- 172.21.15.29:0/2065892973 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c06c7d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 -- 172.21.15.29:0/2065892973 >> 172.21.15.29:0/2065892973 conn(0x7f3b2c0d2cb0 msgr2=0x7f3b2c0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 -- 172.21.15.29:0/2065892973 shutdown_connections 2023-10-16T14:16:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.858+0000 7f3b32687640 1 -- 172.21.15.29:0/2065892973 wait complete. 2023-10-16T14:16:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.860+0000 7f3b32687640 1 Processor -- start 2023-10-16T14:16:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.860+0000 7f3b32687640 1 -- start start 2023-10-16T14:16:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b32687640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b32687640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c15e0c0 0x7f3b2c1f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b32687640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 0x7f3b2c15fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b32687640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c159da0 con 0x7f3b2c15e540 2023-10-16T14:16:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b32687640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c159120 con 0x7f3b2c06c3f0 2023-10-16T14:16:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b32687640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c159400 con 0x7f3b2c15e0c0 2023-10-16T14:16:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b30c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 0x7f3b2c15fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b30c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 0x7f3b2c15fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60586/0 (socket says 172.21.15.29:60586) 2023-10-16T14:16:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.861+0000 7f3b30c24640 1 -- 172.21.15.29:0/466657042 learned_addr learned my addr 172.21.15.29:0/466657042 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b2bfff640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b2b7fe640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c15e0c0 0x7f3b2c1f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b30c24640 1 -- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c15e0c0 msgr2=0x7f3b2c1f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b30c24640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c15e0c0 0x7f3b2c1f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b30c24640 1 -- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 msgr2=0x7f3b2c154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b30c24640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b30c24640 1 -- 172.21.15.29:0/466657042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c009400 con 0x7f3b2c15e540 2023-10-16T14:16:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b2b7fe640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c15e0c0 0x7f3b2c1f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b30c24640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 0x7f3b2c15fd40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400c790 tx=0x7f3b2400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400cea0 con 0x7f3b2c15e540 2023-10-16T14:16:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24003b70 con 0x7f3b2c15e540 2023-10-16T14:16:24.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.862+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400a850 con 0x7f3b2c15e540 2023-10-16T14:16:24.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.863+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c1596e0 con 0x7f3b2c15e540 2023-10-16T14:16:24.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.863+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c1f2a50 con 0x7f3b2c15e540 2023-10-16T14:16:24.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.864+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c159ee0 con 0x7f3b2c15e540 2023-10-16T14:16:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.865+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3b2400a9b0 con 0x7f3b2c15e540 2023-10-16T14:16:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.866+0000 7f3b28ff9640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3b1806ae70 0x7f3b1806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.866+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b24089160 con 0x7f3b2c15e540 2023-10-16T14:16:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.867+0000 7f3b2bfff640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3b1806ae70 0x7f3b1806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.867+0000 7f3b2bfff640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3b1806ae70 0x7f3b1806d330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c036000 tx=0x7f3b1c03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:24.871+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3b2c159ee0 con 0x7f3b2c15e540 2023-10-16T14:16:25.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.062+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b2c159ee0 con 0x7f3b2c15e540 2023-10-16T14:16:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.063+0000 7f3b28ff9640 1 -- 172.21.15.29:0/466657042 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f3b2c159ee0 con 0x7f3b2c15e540 2023-10-16T14:16:25.066 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:25.066 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:20.609906+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.065+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3b1806ae70 msgr2=0x7f3b1806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.065+0000 7f3b32687640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3b1806ae70 0x7f3b1806d330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c036000 tx=0x7f3b1c03b040 comp rx=0 tx=0).stop 2023-10-16T14:16:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.065+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 msgr2=0x7f3b2c15fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.065+0000 7f3b32687640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 0x7f3b2c15fd40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400c790 tx=0x7f3b2400cc50 comp rx=0 tx=0).stop 2023-10-16T14:16:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 shutdown_connections 2023-10-16T14:16:25.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3b1806ae70 0x7f3b1806d330 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b2c15e540 0x7f3b2c15fd40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b2c15e0c0 0x7f3b2c1f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 --2- 172.21.15.29:0/466657042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b2c06c3f0 0x7f3b2c154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 >> 172.21.15.29:0/466657042 conn(0x7f3b2c0d2cb0 msgr2=0x7f3b2c0d19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:25.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 shutdown_connections 2023-10-16T14:16:25.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.066+0000 7f3b32687640 1 -- 172.21.15.29:0/466657042 wait complete. 2023-10-16T14:16:25.086 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:16:25.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:16:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.299+0000 7f4447c64640 1 Processor -- start 2023-10-16T14:16:25.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.299+0000 7f4447c64640 1 -- start start 2023-10-16T14:16:25.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f4447c64640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f444015ad70 0x7f4440069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f4447c64640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44400c9ca0 0x7f444006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f4447c64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f44400ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f4447c64640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44400cad80 con 0x7f44400c9ca0 2023-10-16T14:16:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f4447c64640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44400cb030 con 0x7f444015ad70 2023-10-16T14:16:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f4447c64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44400c91c0 con 0x7f44400ca3b0 2023-10-16T14:16:25.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f44451ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44400c9ca0 0x7f444006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f44451ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44400c9ca0 0x7f444006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60592/0 (socket says 172.21.15.29:60592) 2023-10-16T14:16:25.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.301+0000 7f44451ff640 1 -- 172.21.15.29:0/1198692631 learned_addr learned my addr 172.21.15.29:0/1198692631 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:25.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f4445a00640 1 --2- 172.21.15.29:0/1198692631 >> v2:172.21.15.170:3300/0 conn(0x7f444015ad70 0x7f4440069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f4446201640 1 --2- 172.21.15.29:0/1198692631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f44400ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f44451ff640 1 -- 172.21.15.29:0/1198692631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 msgr2=0x7f44400ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f44451ff640 1 --2- 172.21.15.29:0/1198692631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f44400ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f44451ff640 1 -- 172.21.15.29:0/1198692631 >> v2:172.21.15.170:3300/0 conn(0x7f444015ad70 msgr2=0x7f4440069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:25.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f44451ff640 1 --2- 172.21.15.29:0/1198692631 >> v2:172.21.15.170:3300/0 conn(0x7f444015ad70 0x7f4440069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.302+0000 7f44451ff640 1 -- 172.21.15.29:0/1198692631 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44400c94a0 con 0x7f44400c9ca0 2023-10-16T14:16:25.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.303+0000 7f44451ff640 1 --2- 172.21.15.29:0/1198692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f444006a2e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44400cf780 tx=0x7f443002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc655a9252bf4a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:25.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.303+0000 7f44449fe640 1 -- 172.21.15.29:0/1198692631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430031870 con 0x7f44400c9ca0 2023-10-16T14:16:25.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.303+0000 7f44449fe640 1 -- 172.21.15.29:0/1198692631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44300319d0 con 0x7f44400c9ca0 2023-10-16T14:16:25.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.303+0000 7f44449fe640 1 -- 172.21.15.29:0/1198692631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430031d60 con 0x7f44400c9ca0 2023-10-16T14:16:25.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 -- 172.21.15.29:0/1198692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 msgr2=0x7f444006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 --2- 172.21.15.29:0/1198692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f444006a2e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44400cf780 tx=0x7f443002b910 comp rx=0 tx=0).stop 2023-10-16T14:16:25.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 -- 172.21.15.29:0/1198692631 shutdown_connections 2023-10-16T14:16:25.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 --2- 172.21.15.29:0/1198692631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f44400ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 --2- 172.21.15.29:0/1198692631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f444006a2e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 --2- 172.21.15.29:0/1198692631 >> v2:172.21.15.170:3300/0 conn(0x7f444015ad70 0x7f4440069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 -- 172.21.15.29:0/1198692631 >> 172.21.15.29:0/1198692631 conn(0x7f44400ce9b0 msgr2=0x7f44400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 -- 172.21.15.29:0/1198692631 shutdown_connections 2023-10-16T14:16:25.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.304+0000 7f4447c64640 1 -- 172.21.15.29:0/1198692631 wait complete. 2023-10-16T14:16:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.306+0000 7f4447c64640 1 Processor -- start 2023-10-16T14:16:25.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.307+0000 7f4447c64640 1 -- start start 2023-10-16T14:16:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4447c64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f44401f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4447c64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f444015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4447c64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f444015ad70 0x7f444015da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4447c64640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44400a87f0 con 0x7f44400c9ca0 2023-10-16T14:16:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4447c64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44400c74e0 con 0x7f444015ad70 2023-10-16T14:16:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4447c64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44401f62a0 con 0x7f44400ca3b0 2023-10-16T14:16:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4445a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f44401f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.308+0000 7f4445a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f44401f5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60606/0 (socket says 172.21.15.29:60606) 2023-10-16T14:16:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 -- 172.21.15.29:0/2990910235 learned_addr learned my addr 172.21.15.29:0/2990910235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f44451ff640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f444015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 -- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 msgr2=0x7f444015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f444015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 -- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f444015ad70 msgr2=0x7f444015da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f444015ad70 0x7f444015da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 -- 172.21.15.29:0/2990910235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300094d0 con 0x7f44400c9ca0 2023-10-16T14:16:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4445a00640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f44401f5d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f443c00d5f0 tx=0x7f443c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0125f0 con 0x7f44400c9ca0 2023-10-16T14:16:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c012750 con 0x7f44400c9ca0 2023-10-16T14:16:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c012b00 con 0x7f44400c9ca0 2023-10-16T14:16:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.309+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44400c7730 con 0x7f44400c9ca0 2023-10-16T14:16:25.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.310+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44400a8480 con 0x7f44400c9ca0 2023-10-16T14:16:25.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.311+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444004e160 con 0x7f44400c9ca0 2023-10-16T14:16:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.312+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f443c012c60 con 0x7f44400c9ca0 2023-10-16T14:16:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.312+0000 7f44367fc640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f442406ae70 0x7f442406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.313+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f443c089850 con 0x7f44400c9ca0 2023-10-16T14:16:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.313+0000 7f44451ff640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f442406ae70 0x7f442406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.315+0000 7f44451ff640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f442406ae70 0x7f442406d330 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4430036000 tx=0x7f443003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:25.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.317+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f443c054bb0 con 0x7f44400c9ca0 2023-10-16T14:16:25.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.508+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f444006a9d0 con 0x7f44400c9ca0 2023-10-16T14:16:25.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.509+0000 7f44367fc640 1 -- 172.21.15.29:0/2990910235 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f443c0581d0 con 0x7f44400c9ca0 2023-10-16T14:16:25.513 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:16:25.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f442406ae70 msgr2=0x7f442406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f442406ae70 0x7f442406d330 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4430036000 tx=0x7f443003c040 comp rx=0 tx=0).stop 2023-10-16T14:16:25.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 msgr2=0x7f44401f5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f44401f5d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f443c00d5f0 tx=0x7f443c00dab0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 shutdown_connections 2023-10-16T14:16:25.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f442406ae70 0x7f442406d330 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f444015ad70 0x7f444015da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44400ca3b0 0x7f444015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 --2- 172.21.15.29:0/2990910235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44400c9ca0 0x7f44401f5d60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 >> 172.21.15.29:0/2990910235 conn(0x7f44400ce9b0 msgr2=0x7f44400cdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.511+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 shutdown_connections 2023-10-16T14:16:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.512+0000 7f4447c64640 1 -- 172.21.15.29:0/2990910235 wait complete. 2023-10-16T14:16:25.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:16:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.756+0000 7f3036b5e640 1 Processor -- start 2023-10-16T14:16:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.756+0000 7f3036b5e640 1 -- start start 2023-10-16T14:16:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f3036b5e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f303015aaa0 0x7f303015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f3036b5e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f303015ae80 0x7f30300cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f3036b5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f3030070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f3036b5e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303006b810 con 0x7f303015ae80 2023-10-16T14:16:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f3036b5e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303015ed60 con 0x7f303015aaa0 2023-10-16T14:16:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f3036b5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300707e0 con 0x7f30300ccaa0 2023-10-16T14:16:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f302ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f303015ae80 0x7f30300cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f302ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f303015ae80 0x7f30300cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60620/0 (socket says 172.21.15.29:60620) 2023-10-16T14:16:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f302ffff640 1 -- 172.21.15.29:0/3260614210 learned_addr learned my addr 172.21.15.29:0/3260614210 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f302ffff640 1 -- 172.21.15.29:0/3260614210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 msgr2=0x7f3030070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f30350fb640 1 --2- 172.21.15.29:0/3260614210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f3030070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f30348fa640 1 --2- 172.21.15.29:0/3260614210 >> v2:172.21.15.170:3300/0 conn(0x7f303015aaa0 0x7f303015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f302ffff640 1 --2- 172.21.15.29:0/3260614210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f3030070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.758+0000 7f302ffff640 1 -- 172.21.15.29:0/3260614210 >> v2:172.21.15.170:3300/0 conn(0x7f303015aaa0 msgr2=0x7f303015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.759+0000 7f302ffff640 1 --2- 172.21.15.29:0/3260614210 >> v2:172.21.15.170:3300/0 conn(0x7f303015aaa0 0x7f303015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.759+0000 7f302ffff640 1 -- 172.21.15.29:0/3260614210 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303015ef90 con 0x7f303015ae80 2023-10-16T14:16:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.759+0000 7f30348fa640 1 --2- 172.21.15.29:0/3260614210 >> v2:172.21.15.170:3300/0 conn(0x7f303015aaa0 0x7f303015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.760+0000 7f302ffff640 1 --2- 172.21.15.29:0/3260614210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015ae80 0x7f30300cc560 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30300cd990 tx=0x7f301c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c692d9888bd9cf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.760+0000 7f302f7fe640 1 -- 172.21.15.29:0/3260614210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c031870 con 0x7f303015ae80 2023-10-16T14:16:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.760+0000 7f302f7fe640 1 -- 172.21.15.29:0/3260614210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c0319d0 con 0x7f303015ae80 2023-10-16T14:16:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.760+0000 7f302f7fe640 1 -- 172.21.15.29:0/3260614210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c031d60 con 0x7f303015ae80 2023-10-16T14:16:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.760+0000 7f3036b5e640 1 -- 172.21.15.29:0/3260614210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015ae80 msgr2=0x7f30300cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.760+0000 7f3036b5e640 1 --2- 172.21.15.29:0/3260614210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015ae80 0x7f30300cc560 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30300cd990 tx=0x7f301c02b910 comp rx=0 tx=0).stop 2023-10-16T14:16:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 -- 172.21.15.29:0/3260614210 shutdown_connections 2023-10-16T14:16:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 --2- 172.21.15.29:0/3260614210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f3030070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 --2- 172.21.15.29:0/3260614210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015ae80 0x7f30300cc560 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 --2- 172.21.15.29:0/3260614210 >> v2:172.21.15.170:3300/0 conn(0x7f303015aaa0 0x7f303015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 -- 172.21.15.29:0/3260614210 >> 172.21.15.29:0/3260614210 conn(0x7f30300cb410 msgr2=0x7f30300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:25.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 -- 172.21.15.29:0/3260614210 shutdown_connections 2023-10-16T14:16:25.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.761+0000 7f3036b5e640 1 -- 172.21.15.29:0/3260614210 wait complete. 2023-10-16T14:16:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.762+0000 7f3036b5e640 1 Processor -- start 2023-10-16T14:16:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.763+0000 7f3036b5e640 1 -- start start 2023-10-16T14:16:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f3036b5e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f30301f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f3036b5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 0x7f303015c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f3036b5e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f303015ae80 0x7f303015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f3036b5e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30301f6d70 con 0x7f303015aaa0 2023-10-16T14:16:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f3036b5e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303015ce70 con 0x7f303015ae80 2023-10-16T14:16:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f3036b5e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303015d150 con 0x7f30300ccaa0 2023-10-16T14:16:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 0x7f303015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 0x7f303015c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60636/0 (socket says 172.21.15.29:60636) 2023-10-16T14:16:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 -- 172.21.15.29:0/787426211 learned_addr learned my addr 172.21.15.29:0/787426211 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f30348fa640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f30301f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f30350fb640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f303015ae80 0x7f303015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 -- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 msgr2=0x7f30301f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f30301f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 -- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f303015ae80 msgr2=0x7f303015c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f303015ae80 0x7f303015c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f302ffff640 1 -- 172.21.15.29:0/787426211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c0094d0 con 0x7f303015aaa0 2023-10-16T14:16:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.764+0000 7f30350fb640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f303015ae80 0x7f303015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.765+0000 7f302ffff640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 0x7f303015c300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f301c02b950 tx=0x7f301c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.765+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c031860 con 0x7f303015aaa0 2023-10-16T14:16:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.765+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c0443f0 con 0x7f303015aaa0 2023-10-16T14:16:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.765+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0447a0 con 0x7f303015aaa0 2023-10-16T14:16:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.765+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303015d430 con 0x7f303015aaa0 2023-10-16T14:16:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.765+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303015d9f0 con 0x7f303015aaa0 2023-10-16T14:16:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.766+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303015dbe0 con 0x7f303015aaa0 2023-10-16T14:16:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.767+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f301c04d450 con 0x7f303015aaa0 2023-10-16T14:16:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.768+0000 7f302d7fa640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f301406ae70 0x7f301406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.768+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f301c02d040 con 0x7f303015aaa0 2023-10-16T14:16:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.769+0000 7f30348fa640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f301406ae70 0x7f301406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.770+0000 7f30348fa640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f301406ae70 0x7f301406d330 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f302400a990 tx=0x7f3024008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:25.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.772+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f301c076e90 con 0x7f303015aaa0 2023-10-16T14:16:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.997+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f303004e160 con 0x7f303015aaa0 2023-10-16T14:16:26.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:25.998+0000 7f302d7fa640 1 -- 172.21.15.29:0/787426211 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7f301c035070 con 0x7f303015aaa0 2023-10-16T14:16:26.002 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:26.003 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-16T14:16:26.003 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2023-10-16T14:16:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f301406ae70 msgr2=0x7f301406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f301406ae70 0x7f301406d330 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f302400a990 tx=0x7f3024008040 comp rx=0 tx=0).stop 2023-10-16T14:16:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 msgr2=0x7f303015c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 0x7f303015c300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f301c02b950 tx=0x7f301c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 shutdown_connections 2023-10-16T14:16:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f301406ae70 0x7f301406d330 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303015aaa0 0x7f303015c300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f303015ae80 0x7f303015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 --2- 172.21.15.29:0/787426211 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30300ccaa0 0x7f30301f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 >> 172.21.15.29:0/787426211 conn(0x7f30300cb410 msgr2=0x7f303006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 shutdown_connections 2023-10-16T14:16:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.001+0000 7f3036b5e640 1 -- 172.21.15.29:0/787426211 wait complete. 2023-10-16T14:16:26.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:26.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.238+0000 7f2f3e090640 1 Processor -- start 2023-10-16T14:16:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.238+0000 7f2f3e090640 1 -- start start 2023-10-16T14:16:26.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3e090640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f3815c9f0 0x7f2f380c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3e090640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2f380c9230 0x7f2f380709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3e090640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f38070f20 0x7f2f380ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3e090640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f380cdec0 con 0x7f2f3815c9f0 2023-10-16T14:16:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3e090640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f38160a90 con 0x7f2f380c9230 2023-10-16T14:16:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3e090640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380cd900 con 0x7f2f38070f20 2023-10-16T14:16:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3d08e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f3815c9f0 0x7f2f380c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3d08e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f3815c9f0 0x7f2f380c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60648/0 (socket says 172.21.15.29:60648) 2023-10-16T14:16:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3d08e640 1 -- 172.21.15.29:0/696195933 learned_addr learned my addr 172.21.15.29:0/696195933 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3c88d640 1 --2- 172.21.15.29:0/696195933 >> v2:172.21.15.170:3300/0 conn(0x7f2f380c9230 0x7f2f380709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3d88f640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f38070f20 0x7f2f380ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.240+0000 7f2f3d08e640 1 -- 172.21.15.29:0/696195933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f38070f20 msgr2=0x7f2f380ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:26.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.241+0000 7f2f3d08e640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f38070f20 0x7f2f380ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.241+0000 7f2f3d08e640 1 -- 172.21.15.29:0/696195933 >> v2:172.21.15.170:3300/0 conn(0x7f2f380c9230 msgr2=0x7f2f380709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:26.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.241+0000 7f2f3d08e640 1 --2- 172.21.15.29:0/696195933 >> v2:172.21.15.170:3300/0 conn(0x7f2f380c9230 0x7f2f380709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.241+0000 7f2f3d08e640 1 -- 172.21.15.29:0/696195933 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f380cdbe0 con 0x7f2f3815c9f0 2023-10-16T14:16:26.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.241+0000 7f2f3d88f640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f38070f20 0x7f2f380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.242+0000 7f2f3d08e640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3815c9f0 0x7f2f380c8cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f380b1260 tx=0x7f2f3402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d9d9de937997d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:26.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.242+0000 7f2f2ffff640 1 -- 172.21.15.29:0/696195933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031860 con 0x7f2f3815c9f0 2023-10-16T14:16:26.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.242+0000 7f2f2ffff640 1 -- 172.21.15.29:0/696195933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f340319c0 con 0x7f2f3815c9f0 2023-10-16T14:16:26.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.242+0000 7f2f2ffff640 1 -- 172.21.15.29:0/696195933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031d50 con 0x7f2f3815c9f0 2023-10-16T14:16:26.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.242+0000 7f2f3e090640 1 -- 172.21.15.29:0/696195933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3815c9f0 msgr2=0x7f2f380c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.242+0000 7f2f3e090640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3815c9f0 0x7f2f380c8cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f380b1260 tx=0x7f2f3402b900 comp rx=0 tx=0).stop 2023-10-16T14:16:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 -- 172.21.15.29:0/696195933 shutdown_connections 2023-10-16T14:16:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f38070f20 0x7f2f380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 --2- 172.21.15.29:0/696195933 >> v2:172.21.15.170:3300/0 conn(0x7f2f380c9230 0x7f2f380709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 --2- 172.21.15.29:0/696195933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3815c9f0 0x7f2f380c8cf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 -- 172.21.15.29:0/696195933 >> 172.21.15.29:0/696195933 conn(0x7f2f380ce6a0 msgr2=0x7f2f380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 -- 172.21.15.29:0/696195933 shutdown_connections 2023-10-16T14:16:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.243+0000 7f2f3e090640 1 -- 172.21.15.29:0/696195933 wait complete. 2023-10-16T14:16:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.245+0000 7f2f3e090640 1 Processor -- start 2023-10-16T14:16:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.245+0000 7f2f3e090640 1 -- start start 2023-10-16T14:16:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3e090640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 0x7f2f380c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3e090640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f380c9230 0x7f2f380c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3e090640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3815c9f0 0x7f2f381f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3e090640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f381f6bb0 con 0x7f2f38070f20 2023-10-16T14:16:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3e090640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f381f6100 con 0x7f2f3815c9f0 2023-10-16T14:16:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3e090640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3815e140 con 0x7f2f380c9230 2023-10-16T14:16:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 0x7f2f380c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 0x7f2f380c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60650/0 (socket says 172.21.15.29:60650) 2023-10-16T14:16:26.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 -- 172.21.15.29:0/3097675539 learned_addr learned my addr 172.21.15.29:0/3097675539 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:26.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3c88d640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f380c9230 0x7f2f380c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 -- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f380c9230 msgr2=0x7f2f380c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f380c9230 0x7f2f380c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 -- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3815c9f0 msgr2=0x7f2f381f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3815c9f0 0x7f2f381f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.246+0000 7f2f3d08e640 1 -- 172.21.15.29:0/3097675539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340094d0 con 0x7f2f38070f20 2023-10-16T14:16:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.247+0000 7f2f3d08e640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 0x7f2f380c5770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.247+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3403ad30 con 0x7f2f38070f20 2023-10-16T14:16:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.247+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3403ae90 con 0x7f2f38070f20 2023-10-16T14:16:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.247+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34044630 con 0x7f2f38070f20 2023-10-16T14:16:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.247+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3815de10 con 0x7f2f38070f20 2023-10-16T14:16:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.247+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f381f65f0 con 0x7f2f38070f20 2023-10-16T14:16:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.248+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f38160030 con 0x7f2f38070f20 2023-10-16T14:16:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.249+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2f3404d450 con 0x7f2f38070f20 2023-10-16T14:16:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.250+0000 7f2f2dffb640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2f1c06aea0 0x7f2f1c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.250+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2f340ab7b0 con 0x7f2f38070f20 2023-10-16T14:16:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.251+0000 7f2f3c88d640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2f1c06aea0 0x7f2f1c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.252+0000 7f2f3c88d640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2f1c06aea0 0x7f2f1c06d360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f381f5b20 tx=0x7f2f28008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.255+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2f34076c40 con 0x7f2f38070f20 2023-10-16T14:16:26.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.449+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f3804e9b0 con 0x7f2f38070f20 2023-10-16T14:16:26.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.451+0000 7f2f2dffb640 1 -- 172.21.15.29:0/3097675539 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f2f3407a260 con 0x7f2f38070f20 2023-10-16T14:16:26.455 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:20.609906+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:26.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.453+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2f1c06aea0 msgr2=0x7f2f1c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.453+0000 7f2f3e090640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2f1c06aea0 0x7f2f1c06d360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f381f5b20 tx=0x7f2f28008040 comp rx=0 tx=0).stop 2023-10-16T14:16:26.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.453+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 msgr2=0x7f2f380c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.453+0000 7f2f3e090640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 0x7f2f380c5770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f340046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 shutdown_connections 2023-10-16T14:16:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2f1c06aea0 0x7f2f1c06d360 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3815c9f0 0x7f2f381f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f380c9230 0x7f2f380c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 --2- 172.21.15.29:0/3097675539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f38070f20 0x7f2f380c5770 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 >> 172.21.15.29:0/3097675539 conn(0x7f2f380ce6a0 msgr2=0x7f2f380ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:26.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 shutdown_connections 2023-10-16T14:16:26.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.454+0000 7f2f3e090640 1 -- 172.21.15.29:0/3097675539 wait complete. 2023-10-16T14:16:26.478 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:16:26.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:16:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.696+0000 7ff1b149e640 1 Processor -- start 2023-10-16T14:16:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.696+0000 7ff1b149e640 1 -- start start 2023-10-16T14:16:26.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.698+0000 7ff1b149e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff1ac0b1ca0 0x7ff1ac0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.698+0000 7ff1b149e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0cfc40 0x7ff1ac0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.698+0000 7ff1b149e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac0d0970 con 0x7ff1ac0b2080 2023-10-16T14:16:26.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.698+0000 7ff1b149e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac0d0ea0 con 0x7ff1ac0b1ca0 2023-10-16T14:16:26.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.698+0000 7ff1b149e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac0d0bc0 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.698+0000 7ff1abfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff1ac0b1ca0 0x7ff1ac0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1abfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff1ac0b1ca0 0x7ff1ac0d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43566/0 (socket says 172.21.15.29:43566) 2023-10-16T14:16:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1abfff640 1 -- 172.21.15.29:0/1897333455 learned_addr learned my addr 172.21.15.29:0/1897333455 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1b0c9d640 1 --2- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0cfc40 0x7ff1ac0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1abfff640 1 -- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0cfc40 msgr2=0x7ff1ac0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1abfff640 1 --2- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0cfc40 0x7ff1ac0b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1abfff640 1 -- 172.21.15.29:0/1897333455 >> v1:172.21.15.170:6789/0 conn(0x7ff1ac0b2080 legacy=0x7ff1ac0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.699+0000 7ff1abfff640 1 -- 172.21.15.29:0/1897333455 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0cd940 con 0x7ff1ac0b1ca0 2023-10-16T14:16:26.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.700+0000 7ff1abfff640 1 --2- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac0d36b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009e50 tx=0x7ff19c02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a864e94c50900df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:26.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.701+0000 7ff1ab7fe640 1 -- 172.21.15.29:0/1897333455 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c02cb90 con 0x7ff1ac0b1ca0 2023-10-16T14:16:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.701+0000 7ff1ab7fe640 1 -- 172.21.15.29:0/1897333455 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c02ccf0 con 0x7ff1ac0b1ca0 2023-10-16T14:16:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.701+0000 7ff1ab7fe640 1 -- 172.21.15.29:0/1897333455 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c034610 con 0x7ff1ac0b1ca0 2023-10-16T14:16:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 -- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 msgr2=0x7ff1ac0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 --2- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac0d36b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009e50 tx=0x7ff19c02bff0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 -- 172.21.15.29:0/1897333455 shutdown_connections 2023-10-16T14:16:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 --2- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0cfc40 0x7ff1ac0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 --2- 172.21.15.29:0/1897333455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac0d36b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 -- 172.21.15.29:0/1897333455 >> 172.21.15.29:0/1897333455 conn(0x7ff1ac0d2b90 msgr2=0x7ff1ac0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 -- 172.21.15.29:0/1897333455 shutdown_connections 2023-10-16T14:16:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.702+0000 7ff1b149e640 1 -- 172.21.15.29:0/1897333455 wait complete. 2023-10-16T14:16:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.705+0000 7ff1b149e640 1 Processor -- start 2023-10-16T14:16:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.705+0000 7ff1b149e640 1 -- start start 2023-10-16T14:16:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.706+0000 7ff1b149e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.706+0000 7ff1b149e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0b2080 0x7ff1ac0c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.706+0000 7ff1b149e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 0x7ff1ac0c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.706+0000 7ff1b149e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac0cadc0 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.706+0000 7ff1b149e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac1f2db0 con 0x7ff1ac0b1ca0 2023-10-16T14:16:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.706+0000 7ff1b149e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac0c9890 con 0x7ff1ac0b2080 2023-10-16T14:16:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 0x7ff1ac0c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 0x7ff1ac0c9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60666/0 (socket says 172.21.15.29:60666) 2023-10-16T14:16:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 -- 172.21.15.29:0/569151961 learned_addr learned my addr 172.21.15.29:0/569151961 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1abfff640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 -- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0b2080 msgr2=0x7ff1ac0c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0b2080 0x7ff1ac0c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 -- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 msgr2=0x7ff1ac153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 -- 172.21.15.29:0/569151961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c009b30 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.707+0000 7ff1b0c9d640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 0x7ff1ac0c9280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e820 tx=0x7ff1a400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.708+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4009510 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.708+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a400ee70 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.708+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac0cafd0 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.708+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400cb80 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.708+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac0cb590 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.709+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac0cdc00 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.710+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff1a400cce0 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.711+0000 7ff1a97fa640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff18c06ae70 0x7ff18c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.711+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff1a4089470 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.712+0000 7ff1abfff640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff18c06ae70 0x7ff18c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.713+0000 7ff1abfff640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff18c06ae70 0x7ff18c06d330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff19c035000 tx=0x7ff19c030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.715+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1ac0cdc00 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.907+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1ac0cdc00 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.908+0000 7ff1a97fa640 1 -- 172.21.15.29:0/569151961 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1ac0cdc00 con 0x7ff1ac0cfc40 2023-10-16T14:16:26.912 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:16:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff18c06ae70 msgr2=0x7ff18c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff18c06ae70 0x7ff18c06d330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff19c035000 tx=0x7ff19c030000 comp rx=0 tx=0).stop 2023-10-16T14:16:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 msgr2=0x7ff1ac0c9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 0x7ff1ac0c9280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e820 tx=0x7ff1a400ec20 comp rx=0 tx=0).stop 2023-10-16T14:16:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 shutdown_connections 2023-10-16T14:16:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff18c06ae70 0x7ff18c06d330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac0cfc40 0x7ff1ac0c9280 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1ac0b2080 0x7ff1ac0c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 --2- 172.21.15.29:0/569151961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1ac0b1ca0 0x7ff1ac153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 >> 172.21.15.29:0/569151961 conn(0x7ff1ac0d2b90 msgr2=0x7ff1ac0d30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 shutdown_connections 2023-10-16T14:16:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:26.910+0000 7ff1b149e640 1 -- 172.21.15.29:0/569151961 wait complete. 2023-10-16T14:16:26.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:16:27.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.150+0000 7ff29f59e640 1 Processor -- start 2023-10-16T14:16:27.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.150+0000 7ff29f59e640 1 -- start start 2023-10-16T14:16:27.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2a015cbe0 0x7ff2a00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a00b4c80 0x7ff2a00b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00d15a0 con 0x7ff2a015cfc0 2023-10-16T14:16:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a00d1880 con 0x7ff2a015cbe0 2023-10-16T14:16:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00d1b60 con 0x7ff2a00b4c80 2023-10-16T14:16:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2a015cbe0 0x7ff2a00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2a015cbe0 0x7ff2a00d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60680/0 (socket says 172.21.15.29:60680) 2023-10-16T14:16:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.151+0000 7ff29e59c640 1 -- 172.21.15.29:0/445525711 learned_addr learned my addr 172.21.15.29:0/445525711 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.152+0000 7ff29e59c640 1 -- 172.21.15.29:0/445525711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a00b4c80 msgr2=0x7ff2a00b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.152+0000 7ff29e59c640 1 --2- 172.21.15.29:0/445525711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a00b4c80 0x7ff2a00b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.152+0000 7ff29e59c640 1 -- 172.21.15.29:0/445525711 >> v1:172.21.15.170:6789/0 conn(0x7ff2a015cfc0 legacy=0x7ff2a00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.152+0000 7ff29e59c640 1 -- 172.21.15.29:0/445525711 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00d1e40 con 0x7ff2a015cbe0 2023-10-16T14:16:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.153+0000 7ff29e59c640 1 --2- 172.21.15.29:0/445525711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a015cbe0 0x7ff2a00d0b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff290009870 tx=0x7ff29002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90f6f24a4597ed6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.153+0000 7ff29d59a640 1 -- 172.21.15.29:0/445525711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290030840 con 0x7ff2a015cbe0 2023-10-16T14:16:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.153+0000 7ff29d59a640 1 -- 172.21.15.29:0/445525711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2900309a0 con 0x7ff2a015cbe0 2023-10-16T14:16:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.153+0000 7ff29d59a640 1 -- 172.21.15.29:0/445525711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290030d30 con 0x7ff2a015cbe0 2023-10-16T14:16:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 -- 172.21.15.29:0/445525711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a015cbe0 msgr2=0x7ff2a00d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 --2- 172.21.15.29:0/445525711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a015cbe0 0x7ff2a00d0b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff290009870 tx=0x7ff29002b980 comp rx=0 tx=0).stop 2023-10-16T14:16:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 -- 172.21.15.29:0/445525711 shutdown_connections 2023-10-16T14:16:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 --2- 172.21.15.29:0/445525711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a00b4c80 0x7ff2a00b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 --2- 172.21.15.29:0/445525711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a015cbe0 0x7ff2a00d0b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 -- 172.21.15.29:0/445525711 >> 172.21.15.29:0/445525711 conn(0x7ff2a00cf570 msgr2=0x7ff2a00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 -- 172.21.15.29:0/445525711 shutdown_connections 2023-10-16T14:16:27.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.154+0000 7ff29f59e640 1 -- 172.21.15.29:0/445525711 wait complete. 2023-10-16T14:16:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.156+0000 7ff29f59e640 1 Processor -- start 2023-10-16T14:16:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.156+0000 7ff29f59e640 1 -- start start 2023-10-16T14:16:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 0x7ff2a00c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a015cbe0 0x7ff2a00c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2a015cfc0 0x7ff2a01f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a01531e0 con 0x7ff2a00b4c80 2023-10-16T14:16:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00c9460 con 0x7ff2a015cfc0 2023-10-16T14:16:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0152c20 con 0x7ff2a015cbe0 2023-10-16T14:16:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 0x7ff2a00c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 0x7ff2a00c6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60696/0 (socket says 172.21.15.29:60696) 2023-10-16T14:16:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.157+0000 7ff29e59c640 1 -- 172.21.15.29:0/3925281751 learned_addr learned my addr 172.21.15.29:0/3925281751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29ed9d640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2a015cfc0 0x7ff2a01f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29dd9b640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a015cbe0 0x7ff2a00c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29e59c640 1 -- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a015cbe0 msgr2=0x7ff2a00c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29e59c640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a015cbe0 0x7ff2a00c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29e59c640 1 -- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2a015cfc0 msgr2=0x7ff2a01f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29e59c640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2a015cfc0 0x7ff2a01f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29e59c640 1 -- 172.21.15.29:0/3925281751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2900094d0 con 0x7ff2a00b4c80 2023-10-16T14:16:27.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29ed9d640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2a015cfc0 0x7ff2a01f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:27.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29dd9b640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a015cbe0 0x7ff2a00c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:27.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff29e59c640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 0x7ff2a00c6690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff290009870 tx=0x7ff290004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:27.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290039d60 con 0x7ff2a00b4c80 2023-10-16T14:16:27.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff290043400 con 0x7ff2a00b4c80 2023-10-16T14:16:27.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.158+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2900437b0 con 0x7ff2a00b4c80 2023-10-16T14:16:27.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.159+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a0152f00 con 0x7ff2a00b4c80 2023-10-16T14:16:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.159+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a00c7b10 con 0x7ff2a00b4c80 2023-10-16T14:16:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.160+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a004e9b0 con 0x7ff2a00b4c80 2023-10-16T14:16:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.161+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff290043910 con 0x7ff2a00b4c80 2023-10-16T14:16:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.161+0000 7ff28effd640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff28006ae70 0x7ff28006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.162+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff29003e070 con 0x7ff2a00b4c80 2023-10-16T14:16:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.162+0000 7ff29dd9b640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff28006ae70 0x7ff28006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.163+0000 7ff29dd9b640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff28006ae70 0x7ff28006d330 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00d3700 tx=0x7ff2940073a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:27.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.166+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2900778d0 con 0x7ff2a00b4c80 2023-10-16T14:16:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.359+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff2a00d3450 con 0x7ff2a00b4c80 2023-10-16T14:16:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.360+0000 7ff28effd640 1 -- 172.21.15.29:0/3925281751 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff2900ac070 con 0x7ff2a00b4c80 2023-10-16T14:16:27.364 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:27.364 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:16:27.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:16:27.365 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:16:27.365 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:16:27.365 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:16:27.365 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:16:27.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff28006ae70 msgr2=0x7ff28006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff28006ae70 0x7ff28006d330 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00d3700 tx=0x7ff2940073a0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 msgr2=0x7ff2a00c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 0x7ff2a00c6690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff290009870 tx=0x7ff290004730 comp rx=0 tx=0).stop 2023-10-16T14:16:27.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 shutdown_connections 2023-10-16T14:16:27.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff28006ae70 0x7ff28006d330 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2a015cfc0 0x7ff2a01f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2a015cbe0 0x7ff2a00c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 --2- 172.21.15.29:0/3925281751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a00b4c80 0x7ff2a00c6690 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.362+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 >> 172.21.15.29:0/3925281751 conn(0x7ff2a00cf570 msgr2=0x7ff2a00d2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:27.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.363+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 shutdown_connections 2023-10-16T14:16:27.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.363+0000 7ff29f59e640 1 -- 172.21.15.29:0/3925281751 wait complete. 2023-10-16T14:16:27.387 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:16:27.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:16:27.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.602+0000 7fd56359e640 1 Processor -- start 2023-10-16T14:16:27.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.603+0000 7fd56359e640 1 -- start start 2023-10-16T14:16:27.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.604+0000 7fd56359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd56406b3e0 0x7fd56406cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.604+0000 7fd56359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd56406d2c0 0x7fd564070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.604+0000 7fd56359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56415a810 con 0x7fd564159bd0 2023-10-16T14:16:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.604+0000 7fd56359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5640c5740 con 0x7fd56406d2c0 2023-10-16T14:16:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.604+0000 7fd56359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56406d750 con 0x7fd56406b3e0 2023-10-16T14:16:27.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.605+0000 7fd56259c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd564159bd0 0x7fd564159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:59642/0 (socket says 172.21.15.29:59642) 2023-10-16T14:16:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.605+0000 7fd56259c640 1 -- 172.21.15.29:0/3607418776 learned_addr learned my addr 172.21.15.29:0/3607418776 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.605+0000 7fd561d9b640 1 --2- 172.21.15.29:0/3607418776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd56406b3e0 0x7fd56406cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.605+0000 7fd562d9d640 1 --2- 172.21.15.29:0/3607418776 >> v2:172.21.15.170:3300/0 conn(0x7fd56406d2c0 0x7fd564070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.606+0000 7fd56159a640 1 -- 172.21.15.29:0/3607418776 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176139351 0 0) 0x7fd56415a810 con 0x7fd564159bd0 2023-10-16T14:16:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.606+0000 7fd56159a640 1 -- 172.21.15.29:0/3607418776 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c005730 con 0x7fd564159bd0 2023-10-16T14:16:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.606+0000 7fd561d9b640 1 -- 172.21.15.29:0/3607418776 >> v2:172.21.15.170:3300/0 conn(0x7fd56406d2c0 msgr2=0x7fd564070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.606+0000 7fd561d9b640 1 --2- 172.21.15.29:0/3607418776 >> v2:172.21.15.170:3300/0 conn(0x7fd56406d2c0 0x7fd564070070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.606+0000 7fd561d9b640 1 -- 172.21.15.29:0/3607418776 >> v1:172.21.15.170:6789/0 conn(0x7fd564159bd0 legacy=0x7fd564159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.606+0000 7fd561d9b640 1 -- 172.21.15.29:0/3607418776 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56415aaf0 con 0x7fd56406b3e0 2023-10-16T14:16:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.607+0000 7fd561d9b640 1 --2- 172.21.15.29:0/3607418776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd56406b3e0 0x7fd56406cd80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5580097a0 tx=0x7fd55802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7db83c5e9e27d9aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.608+0000 7fd56159a640 1 -- 172.21.15.29:0/3607418776 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558031780 con 0x7fd56406b3e0 2023-10-16T14:16:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.608+0000 7fd56159a640 1 -- 172.21.15.29:0/3607418776 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5580318e0 con 0x7fd56406b3e0 2023-10-16T14:16:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.608+0000 7fd56159a640 1 -- 172.21.15.29:0/3607418776 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558031c70 con 0x7fd56406b3e0 2023-10-16T14:16:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 -- 172.21.15.29:0/3607418776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd56406b3e0 msgr2=0x7fd56406cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 --2- 172.21.15.29:0/3607418776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd56406b3e0 0x7fd56406cd80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5580097a0 tx=0x7fd55802b820 comp rx=0 tx=0).stop 2023-10-16T14:16:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 -- 172.21.15.29:0/3607418776 shutdown_connections 2023-10-16T14:16:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 --2- 172.21.15.29:0/3607418776 >> v2:172.21.15.170:3300/0 conn(0x7fd56406d2c0 0x7fd564070070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 --2- 172.21.15.29:0/3607418776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd56406b3e0 0x7fd56406cd80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 -- 172.21.15.29:0/3607418776 >> 172.21.15.29:0/3607418776 conn(0x7fd5640ce820 msgr2=0x7fd5640cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 -- 172.21.15.29:0/3607418776 shutdown_connections 2023-10-16T14:16:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.609+0000 7fd56359e640 1 -- 172.21.15.29:0/3607418776 wait complete. 2023-10-16T14:16:27.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.611+0000 7fd56359e640 1 Processor -- start 2023-10-16T14:16:27.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.611+0000 7fd56359e640 1 -- start start 2023-10-16T14:16:27.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd56406b3e0 0x7fd5640c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 0x7fd5640c2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd564159bd0 0x7fd5640b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640c4730 con 0x7fd56406d2c0 2023-10-16T14:16:27.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640c4a10 con 0x7fd56406b3e0 2023-10-16T14:16:27.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640b24b0 con 0x7fd564159bd0 2023-10-16T14:16:27.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd561d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 0x7fd5640c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd561d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 0x7fd5640c2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60704/0 (socket says 172.21.15.29:60704) 2023-10-16T14:16:27.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd561d9b640 1 -- 172.21.15.29:0/949632827 learned_addr learned my addr 172.21.15.29:0/949632827 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:27.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd56259c640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd56406b3e0 0x7fd5640c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.612+0000 7fd562d9d640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd564159bd0 0x7fd5640b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd561d9b640 1 -- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd564159bd0 msgr2=0x7fd5640b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd561d9b640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd564159bd0 0x7fd5640b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd561d9b640 1 -- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd56406b3e0 msgr2=0x7fd5640c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:27.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd561d9b640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd56406b3e0 0x7fd5640c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:27.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd561d9b640 1 -- 172.21.15.29:0/949632827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558009400 con 0x7fd56406d2c0 2023-10-16T14:16:27.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd561d9b640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 0x7fd5640c2330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd558034040 tx=0x7fd5580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:27.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.613+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558032820 con 0x7fd56406d2c0 2023-10-16T14:16:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.614+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558032980 con 0x7fd56406d2c0 2023-10-16T14:16:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.614+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5640b2790 con 0x7fd56406d2c0 2023-10-16T14:16:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.614+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558032d30 con 0x7fd56406d2c0 2023-10-16T14:16:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.614+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5640b2160 con 0x7fd56406d2c0 2023-10-16T14:16:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.615+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56404e9b0 con 0x7fd56406d2c0 2023-10-16T14:16:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.616+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd55804e440 con 0x7fd56406d2c0 2023-10-16T14:16:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.617+0000 7fd552ffd640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd54406aea0 0x7fd54406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.617+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd5580ab240 con 0x7fd56406d2c0 2023-10-16T14:16:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.617+0000 7fd56259c640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd54406aea0 0x7fd54406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:27.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.618+0000 7fd56259c640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd54406aea0 0x7fd54406d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd554001560 tx=0x7fd554001a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:27.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.621+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd558076630 con 0x7fd56406d2c0 2023-10-16T14:16:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.805+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd5640cf730 con 0x7fd56406d2c0 2023-10-16T14:16:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:27.999+0000 7fd552ffd640 1 -- 172.21.15.29:0/949632827 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd558079c50 con 0x7fd56406d2c0 2023-10-16T14:16:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:16:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd54406aea0 msgr2=0x7fd54406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd54406aea0 0x7fd54406d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd554001560 tx=0x7fd554001a90 comp rx=0 tx=0).stop 2023-10-16T14:16:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 msgr2=0x7fd5640c2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 0x7fd5640c2330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd558034040 tx=0x7fd5580046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 shutdown_connections 2023-10-16T14:16:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd54406aea0 0x7fd54406d360 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd56406d2c0 0x7fd5640c2330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd564159bd0 0x7fd5640b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 --2- 172.21.15.29:0/949632827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd56406b3e0 0x7fd5640c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 >> 172.21.15.29:0/949632827 conn(0x7fd5640ce820 msgr2=0x7fd564158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 shutdown_connections 2023-10-16T14:16:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.002+0000 7fd56359e640 1 -- 172.21.15.29:0/949632827 wait complete. 2023-10-16T14:16:28.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:16:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.239+0000 7fe8a58b6640 1 Processor -- start 2023-10-16T14:16:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.239+0000 7fe8a58b6640 1 -- start start 2023-10-16T14:16:28.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.240+0000 7fe8a58b6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe8a015cb50 0x7fe8a0161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:28.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.240+0000 7fe8a58b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8a00b1690 0x7fe8a00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.240+0000 7fe8a58b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a00b4f00 con 0x7fe8a00b4ab0 2023-10-16T14:16:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.240+0000 7fe8a58b6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a01616c0 con 0x7fe8a00b1690 2023-10-16T14:16:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.240+0000 7fe8a58b6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a00cdbf0 con 0x7fe8a015cb50 2023-10-16T14:16:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.241+0000 7fe89ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8a00b1690 0x7fe8a00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.241+0000 7fe8a48b4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe8a015cb50 0x7fe8a0161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.241+0000 7fe8a48b4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe8a015cb50 0x7fe8a0161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43604/0 (socket says 172.21.15.29:43604) 2023-10-16T14:16:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.241+0000 7fe89ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8a00b1690 0x7fe8a00b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60706/0 (socket says 172.21.15.29:60706) 2023-10-16T14:16:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.241+0000 7fe8a48b4640 1 -- 172.21.15.29:0/575964384 learned_addr learned my addr 172.21.15.29:0/575964384 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.242+0000 7fe89ffff640 1 -- 172.21.15.29:0/575964384 >> v2:172.21.15.170:3300/0 conn(0x7fe8a015cb50 msgr2=0x7fe8a0161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.242+0000 7fe89ffff640 1 --2- 172.21.15.29:0/575964384 >> v2:172.21.15.170:3300/0 conn(0x7fe8a015cb50 0x7fe8a0161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.242+0000 7fe89ffff640 1 -- 172.21.15.29:0/575964384 >> v1:172.21.15.170:6789/0 conn(0x7fe8a00b4ab0 legacy=0x7fe8a006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.242+0000 7fe89ffff640 1 -- 172.21.15.29:0/575964384 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a00cde90 con 0x7fe8a00b1690 2023-10-16T14:16:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.243+0000 7fe89ffff640 1 --2- 172.21.15.29:0/575964384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a00b4570 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00724b0 tx=0x7fe89402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9405dccd49b15f48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.244+0000 7fe89f7fe640 1 -- 172.21.15.29:0/575964384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894031710 con 0x7fe8a00b1690 2023-10-16T14:16:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.244+0000 7fe89f7fe640 1 -- 172.21.15.29:0/575964384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89402f050 con 0x7fe8a00b1690 2023-10-16T14:16:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.244+0000 7fe89f7fe640 1 -- 172.21.15.29:0/575964384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894031a90 con 0x7fe8a00b1690 2023-10-16T14:16:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.244+0000 7fe8a58b6640 1 -- 172.21.15.29:0/575964384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 msgr2=0x7fe8a00b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.244+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/575964384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a00b4570 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00724b0 tx=0x7fe89402b7b0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.245+0000 7fe8a58b6640 1 -- 172.21.15.29:0/575964384 shutdown_connections 2023-10-16T14:16:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.245+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/575964384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a00b4570 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.245+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/575964384 >> v2:172.21.15.170:3300/0 conn(0x7fe8a015cb50 0x7fe8a0161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.245+0000 7fe8a58b6640 1 -- 172.21.15.29:0/575964384 >> 172.21.15.29:0/575964384 conn(0x7fe8a00cf570 msgr2=0x7fe8a00cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.245+0000 7fe8a58b6640 1 -- 172.21.15.29:0/575964384 shutdown_connections 2023-10-16T14:16:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.245+0000 7fe8a58b6640 1 -- 172.21.15.29:0/575964384 wait complete. 2023-10-16T14:16:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.246+0000 7fe8a58b6640 1 Processor -- start 2023-10-16T14:16:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.247+0000 7fe8a58b6640 1 -- start start 2023-10-16T14:16:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a58b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a015e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a58b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a00b4ab0 0x7fe8a015fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a58b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a015cb50 0x7fe8a0157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a58b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a0158390 con 0x7fe8a00b1690 2023-10-16T14:16:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a58b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a015f5b0 con 0x7fe8a015cb50 2023-10-16T14:16:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a58b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a015e9a0 con 0x7fe8a00b4ab0 2023-10-16T14:16:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a015e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a015e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60722/0 (socket says 172.21.15.29:60722) 2023-10-16T14:16:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 -- 172.21.15.29:0/1105692899 learned_addr learned my addr 172.21.15.29:0/1105692899 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a50b5640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a015cb50 0x7fe8a0157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe89ffff640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a00b4ab0 0x7fe8a015fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 -- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a00b4ab0 msgr2=0x7fe8a015fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a00b4ab0 0x7fe8a015fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 -- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a015cb50 msgr2=0x7fe8a0157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a015cb50 0x7fe8a0157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.248+0000 7fe8a48b4640 1 -- 172.21.15.29:0/1105692899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894009400 con 0x7fe8a00b1690 2023-10-16T14:16:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.249+0000 7fe8a48b4640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a015e330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe89000efc0 tx=0x7fe89000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.249+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89000cd40 con 0x7fe8a00b1690 2023-10-16T14:16:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.249+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89000f040 con 0x7fe8a00b1690 2023-10-16T14:16:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.249+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89000cea0 con 0x7fe8a00b1690 2023-10-16T14:16:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.249+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a015ec80 con 0x7fe8a00b1690 2023-10-16T14:16:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.249+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a015f240 con 0x7fe8a00b1690 2023-10-16T14:16:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.250+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a004e9b0 con 0x7fe8a00b1690 2023-10-16T14:16:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.251+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe890014680 con 0x7fe8a00b1690 2023-10-16T14:16:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.252+0000 7fe89d7fa640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe88c06ae70 0x7fe88c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.252+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe89008a600 con 0x7fe8a00b1690 2023-10-16T14:16:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.253+0000 7fe89ffff640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe88c06ae70 0x7fe88c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.254+0000 7fe89ffff640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe88c06ae70 0x7fe88c06d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe894035000 tx=0x7fe894008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.256+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe890055900 con 0x7fe8a00b1690 2023-10-16T14:16:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:28.452+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe8a00d0790 con 0x7fe8a00b1690 2023-10-16T14:16:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.013+0000 7fe89d7fa640 1 -- 172.21.15.29:0/1105692899 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe890058f20 con 0x7fe8a00b1690 2023-10-16T14:16:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:16:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe88c06ae70 msgr2=0x7fe88c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe88c06ae70 0x7fe88c06d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe894035000 tx=0x7fe894008030 comp rx=0 tx=0).stop 2023-10-16T14:16:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 msgr2=0x7fe8a015e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a015e330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe89000efc0 tx=0x7fe89000c490 comp rx=0 tx=0).stop 2023-10-16T14:16:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 shutdown_connections 2023-10-16T14:16:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe88c06ae70 0x7fe88c06d330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8a015cb50 0x7fe8a0157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8a00b4ab0 0x7fe8a015fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 --2- 172.21.15.29:0/1105692899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a00b1690 0x7fe8a015e330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.017+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 >> 172.21.15.29:0/1105692899 conn(0x7fe8a00cf570 msgr2=0x7fe8a00d1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.018+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 shutdown_connections 2023-10-16T14:16:29.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.018+0000 7fe8a58b6640 1 -- 172.21.15.29:0/1105692899 wait complete. 2023-10-16T14:16:29.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:16:29.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.248+0000 7f42add6c640 1 Processor -- start 2023-10-16T14:16:29.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.249+0000 7f42add6c640 1 -- start start 2023-10-16T14:16:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.250+0000 7f42add6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42a80ae260 0x7f42a8160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.250+0000 7f42add6c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a8161900 con 0x7f42a8160ad0 2023-10-16T14:16:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.250+0000 7f42add6c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a81613f0 con 0x7f42a806a780 2023-10-16T14:16:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.250+0000 7f42add6c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a80ce8a0 con 0x7f42a80ae260 2023-10-16T14:16:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.250+0000 7f42a6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42a80ae260 0x7f42a8160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.251+0000 7f42a77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42a8160ad0 0x7f42a80ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36678/0 (socket says 172.21.15.29:36678) 2023-10-16T14:16:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.251+0000 7f42a77fe640 1 -- 172.21.15.29:0/406813079 learned_addr learned my addr 172.21.15.29:0/406813079 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.251+0000 7f42a6ffd640 1 -- 172.21.15.29:0/406813079 >> v1:172.21.15.170:6789/0 conn(0x7f42a806a780 legacy=0x7f42a806ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.251+0000 7f42a6ffd640 1 -- 172.21.15.29:0/406813079 >> v1:172.21.15.29:6789/0 conn(0x7f42a8160ad0 legacy=0x7f42a80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.251+0000 7f42a6ffd640 1 -- 172.21.15.29:0/406813079 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a80ceb80 con 0x7f42a80ae260 2023-10-16T14:16:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.252+0000 7f42a6ffd640 1 --2- 172.21.15.29:0/406813079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a80ae260 0x7f42a8160eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4290009d90 tx=0x7f429002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66061f96a02bd11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.253+0000 7f42a67fc640 1 -- 172.21.15.29:0/406813079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429002cc60 con 0x7f42a80ae260 2023-10-16T14:16:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.253+0000 7f42a67fc640 1 -- 172.21.15.29:0/406813079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429002cdc0 con 0x7f42a80ae260 2023-10-16T14:16:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.253+0000 7f42a67fc640 1 -- 172.21.15.29:0/406813079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290034660 con 0x7f42a80ae260 2023-10-16T14:16:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.253+0000 7f42add6c640 1 -- 172.21.15.29:0/406813079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a80ae260 msgr2=0x7f42a8160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:29.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.253+0000 7f42add6c640 1 --2- 172.21.15.29:0/406813079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a80ae260 0x7f42a8160eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4290009d90 tx=0x7f429002beb0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.253+0000 7f42add6c640 1 -- 172.21.15.29:0/406813079 shutdown_connections 2023-10-16T14:16:29.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.254+0000 7f42add6c640 1 --2- 172.21.15.29:0/406813079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a80ae260 0x7f42a8160eb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.254+0000 7f42add6c640 1 -- 172.21.15.29:0/406813079 >> 172.21.15.29:0/406813079 conn(0x7f42a806bc10 msgr2=0x7f42a806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:29.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.254+0000 7f42add6c640 1 -- 172.21.15.29:0/406813079 shutdown_connections 2023-10-16T14:16:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.254+0000 7f42add6c640 1 -- 172.21.15.29:0/406813079 wait complete. 2023-10-16T14:16:29.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.255+0000 7f42add6c640 1 Processor -- start 2023-10-16T14:16:29.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.256+0000 7f42add6c640 1 -- start start 2023-10-16T14:16:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.256+0000 7f42add6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 0x7f42a8155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.256+0000 7f42add6c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a80ae260 0x7f42a80b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.256+0000 7f42add6c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a8160ad0 0x7f42a80b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42add6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c50e0 con 0x7f42a806a780 2023-10-16T14:16:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42add6c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c53c0 con 0x7f42a80ae260 2023-10-16T14:16:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42add6c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80c56a0 con 0x7f42a8160ad0 2023-10-16T14:16:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 0x7f42a8155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 0x7f42a8155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60740/0 (socket says 172.21.15.29:60740) 2023-10-16T14:16:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 -- 172.21.15.29:0/4027268827 learned_addr learned my addr 172.21.15.29:0/4027268827 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a6ffd640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a80ae260 0x7f42a80b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a7fff640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a8160ad0 0x7f42a80b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 -- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a8160ad0 msgr2=0x7f42a80b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a8160ad0 0x7f42a80b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 -- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a80ae260 msgr2=0x7f42a80b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a80ae260 0x7f42a80b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a77fe640 1 -- 172.21.15.29:0/4027268827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4290009a70 con 0x7f42a806a780 2023-10-16T14:16:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.257+0000 7f42a6ffd640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a80ae260 0x7f42a80b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.258+0000 7f42a77fe640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 0x7f42a8155dd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f429800c400 tx=0x7f429800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.258+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298005ce0 con 0x7f42a806a780 2023-10-16T14:16:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.258+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298005e40 con 0x7f42a806a780 2023-10-16T14:16:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.258+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980072d0 con 0x7f42a806a780 2023-10-16T14:16:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.258+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a80c5980 con 0x7f42a806a780 2023-10-16T14:16:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.258+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a8156310 con 0x7f42a806a780 2023-10-16T14:16:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.259+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a8156500 con 0x7f42a806a780 2023-10-16T14:16:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.260+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f42980074f0 con 0x7f42a806a780 2023-10-16T14:16:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.261+0000 7f429ffff640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f428806ae70 0x7f428806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.262+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4298088b80 con 0x7f42a806a780 2023-10-16T14:16:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.262+0000 7f42a6ffd640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f428806ae70 0x7f428806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.263+0000 7f42a6ffd640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f428806ae70 0x7f428806d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4290036000 tx=0x7f4290030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:29.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.266+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4298053d20 con 0x7f42a806a780 2023-10-16T14:16:29.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:29.500+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f42a804e160 con 0x7f42a806a780 2023-10-16T14:16:30.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.017+0000 7f429ffff640 1 -- 172.21.15.29:0/4027268827 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4298017020 con 0x7f42a806a780 2023-10-16T14:16:30.021 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2023-10-16T14:16:30.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f428806ae70 msgr2=0x7f428806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f428806ae70 0x7f428806d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4290036000 tx=0x7f4290030000 comp rx=0 tx=0).stop 2023-10-16T14:16:30.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 msgr2=0x7f42a8155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 0x7f42a8155dd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f429800c400 tx=0x7f429800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 shutdown_connections 2023-10-16T14:16:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f428806ae70 0x7f428806d330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42a8160ad0 0x7f42a80b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42a80ae260 0x7f42a80b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 --2- 172.21.15.29:0/4027268827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42a806a780 0x7f42a8155dd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 >> 172.21.15.29:0/4027268827 conn(0x7f42a806bc10 msgr2=0x7f42a80af340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:30.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 shutdown_connections 2023-10-16T14:16:30.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.022+0000 7f42add6c640 1 -- 172.21.15.29:0/4027268827 wait complete. 2023-10-16T14:16:30.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:30.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.236+0000 7fb620947640 1 Processor -- start 2023-10-16T14:16:30.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.237+0000 7fb620947640 1 -- start start 2023-10-16T14:16:30.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.238+0000 7fb620947640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 0x7fb61c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.238+0000 7fb620947640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0b8480 0x7fb61c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.238+0000 7fb620947640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c15d150 con 0x7fb61c0d0be0 2023-10-16T14:16:30.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.238+0000 7fb620947640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61c0d1180 con 0x7fb61c15cd70 2023-10-16T14:16:30.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.238+0000 7fb620947640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c0d1460 con 0x7fb61c0b8480 2023-10-16T14:16:30.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.239+0000 7fb61b7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 0x7fb61c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.239+0000 7fb61affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0b8480 0x7fb61c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.239+0000 7fb61b7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 0x7fb61c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:43626/0 (socket says 172.21.15.29:43626) 2023-10-16T14:16:30.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.239+0000 7fb61b7fe640 1 -- 172.21.15.29:0/3047267277 learned_addr learned my addr 172.21.15.29:0/3047267277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:30.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.240+0000 7fb61affd640 1 -- 172.21.15.29:0/3047267277 >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 msgr2=0x7fb61c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:30.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.240+0000 7fb61affd640 1 --2- 172.21.15.29:0/3047267277 >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 0x7fb61c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.240+0000 7fb61affd640 1 -- 172.21.15.29:0/3047267277 >> v1:172.21.15.170:6789/0 conn(0x7fb61c0d0be0 legacy=0x7fb61c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.240+0000 7fb61affd640 1 -- 172.21.15.29:0/3047267277 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c0d1740 con 0x7fb61c0b8480 2023-10-16T14:16:30.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.240+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/3047267277 >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 0x7fb61c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:30.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.241+0000 7fb61affd640 1 --2- 172.21.15.29:0/3047267277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0b8480 0x7fb61c0cd7c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb6040097a0 tx=0x7fb60402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4814d1b57d822cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:30.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.241+0000 7fb61a7fc640 1 -- 172.21.15.29:0/3047267277 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60402d040 con 0x7fb61c0b8480 2023-10-16T14:16:30.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.241+0000 7fb61a7fc640 1 -- 172.21.15.29:0/3047267277 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb604031810 con 0x7fb61c0b8480 2023-10-16T14:16:30.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.241+0000 7fb61a7fc640 1 -- 172.21.15.29:0/3047267277 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604031ba0 con 0x7fb61c0b8480 2023-10-16T14:16:30.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 -- 172.21.15.29:0/3047267277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0b8480 msgr2=0x7fb61c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 --2- 172.21.15.29:0/3047267277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0b8480 0x7fb61c0cd7c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb6040097a0 tx=0x7fb60402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 -- 172.21.15.29:0/3047267277 shutdown_connections 2023-10-16T14:16:30.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 --2- 172.21.15.29:0/3047267277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0b8480 0x7fb61c0cd7c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 --2- 172.21.15.29:0/3047267277 >> v2:172.21.15.170:3300/0 conn(0x7fb61c15cd70 0x7fb61c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 -- 172.21.15.29:0/3047267277 >> 172.21.15.29:0/3047267277 conn(0x7fb61c0cf570 msgr2=0x7fb61c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:30.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 -- 172.21.15.29:0/3047267277 shutdown_connections 2023-10-16T14:16:30.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.242+0000 7fb620947640 1 -- 172.21.15.29:0/3047267277 wait complete. 2023-10-16T14:16:30.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.244+0000 7fb620947640 1 Processor -- start 2023-10-16T14:16:30.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.244+0000 7fb620947640 1 -- start start 2023-10-16T14:16:30.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb620947640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c0b8480 0x7fb61c1560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb620947640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0d0be0 0x7fb61c0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb620947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 0x7fb61c151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb620947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c0cb570 con 0x7fb61c15cd70 2023-10-16T14:16:30.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb620947640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c157920 con 0x7fb61c0b8480 2023-10-16T14:16:30.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb620947640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c0cafb0 con 0x7fb61c0d0be0 2023-10-16T14:16:30.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb61bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 0x7fb61c151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb61bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 0x7fb61c151900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60748/0 (socket says 172.21.15.29:60748) 2023-10-16T14:16:30.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.245+0000 7fb61bfff640 1 -- 172.21.15.29:0/827104626 learned_addr learned my addr 172.21.15.29:0/827104626 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:30.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c0b8480 0x7fb61c1560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61affd640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0d0be0 0x7fb61c0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61bfff640 1 -- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0d0be0 msgr2=0x7fb61c0ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61bfff640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0d0be0 0x7fb61c0ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61bfff640 1 -- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c0b8480 msgr2=0x7fb61c1560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61bfff640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c0b8480 0x7fb61c1560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61bfff640 1 -- 172.21.15.29:0/827104626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c009b30 con 0x7fb61c15cd70 2023-10-16T14:16:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61affd640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0d0be0 0x7fb61c0ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb61bfff640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 0x7fb61c151900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c3c0 tx=0x7fb61400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6140052d0 con 0x7fb61c15cd70 2023-10-16T14:16:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb614005430 con 0x7fb61c15cd70 2023-10-16T14:16:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb604009400 con 0x7fb61c15cd70 2023-10-16T14:16:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.246+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614007260 con 0x7fb61c15cd70 2023-10-16T14:16:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.247+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c157bc0 con 0x7fb61c15cd70 2023-10-16T14:16:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.248+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61c04e9b0 con 0x7fb61c15cd70 2023-10-16T14:16:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.249+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb6140073c0 con 0x7fb61c15cd70 2023-10-16T14:16:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.249+0000 7fb613fff640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb5fc06ae70 0x7fb5fc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.250+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb614088800 con 0x7fb61c15cd70 2023-10-16T14:16:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.250+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb5fc06ae70 0x7fb5fc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.251+0000 7fb61b7fe640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb5fc06ae70 0x7fb5fc06d330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009320 tx=0x7fb60c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:30.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.254+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb614053960 con 0x7fb61c15cd70 2023-10-16T14:16:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.449+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb61c15c300 con 0x7fb61c15cd70 2023-10-16T14:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.451+0000 7fb613fff640 1 -- 172.21.15.29:0/827104626 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7fb614056f80 con 0x7fb61c15cd70 2023-10-16T14:16:30.455 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:30.456 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:29.502933+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.453+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb5fc06ae70 msgr2=0x7fb5fc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.453+0000 7fb620947640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb5fc06ae70 0x7fb5fc06d330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009320 tx=0x7fb60c00b040 comp rx=0 tx=0).stop 2023-10-16T14:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.453+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 msgr2=0x7fb61c151900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.453+0000 7fb620947640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 0x7fb61c151900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c3c0 tx=0x7fb61400c880 comp rx=0 tx=0).stop 2023-10-16T14:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 shutdown_connections 2023-10-16T14:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb5fc06ae70 0x7fb5fc06d330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb61c15cd70 0x7fb61c151900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb61c0d0be0 0x7fb61c0ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 --2- 172.21.15.29:0/827104626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb61c0b8480 0x7fb61c1560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 >> 172.21.15.29:0/827104626 conn(0x7fb61c0cf570 msgr2=0x7fb61c0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 shutdown_connections 2023-10-16T14:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.454+0000 7fb620947640 1 -- 172.21.15.29:0/827104626 wait complete. 2023-10-16T14:16:30.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:30.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.688+0000 7fcec11f5640 1 Processor -- start 2023-10-16T14:16:30.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.688+0000 7fcec11f5640 1 -- start start 2023-10-16T14:16:30.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.689+0000 7fcec11f5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcebc15aaa0 0x7fcebc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.689+0000 7fcec11f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc15ae80 0x7fcebc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.689+0000 7fcec11f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcebc0ccaa0 0x7fcebc070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.689+0000 7fcec11f5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc06b810 con 0x7fcebc0ccaa0 2023-10-16T14:16:30.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.689+0000 7fcec11f5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc15ed60 con 0x7fcebc15aaa0 2023-10-16T14:16:30.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.689+0000 7fcec11f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0707e0 con 0x7fcebc15ae80 2023-10-16T14:16:30.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcebc0ccaa0 0x7fcebc070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcebc0ccaa0 0x7fcebc070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60760/0 (socket says 172.21.15.29:60760) 2023-10-16T14:16:30.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 -- 172.21.15.29:0/3259082934 learned_addr learned my addr 172.21.15.29:0/3259082934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:30.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebad9d640 1 --2- 172.21.15.29:0/3259082934 >> v2:172.21.15.170:3300/0 conn(0x7fcebc15aaa0 0x7fcebc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 -- 172.21.15.29:0/3259082934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc15ae80 msgr2=0x7fcebc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:30.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fceb3fff640 1 --2- 172.21.15.29:0/3259082934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc15ae80 0x7fcebc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 --2- 172.21.15.29:0/3259082934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc15ae80 0x7fcebc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 -- 172.21.15.29:0/3259082934 >> v2:172.21.15.170:3300/0 conn(0x7fcebc15aaa0 msgr2=0x7fcebc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:30.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 --2- 172.21.15.29:0/3259082934 >> v2:172.21.15.170:3300/0 conn(0x7fcebc15aaa0 0x7fcebc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.690+0000 7fcebb59e640 1 -- 172.21.15.29:0/3259082934 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc15ef90 con 0x7fcebc0ccaa0 2023-10-16T14:16:30.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.691+0000 7fcebb59e640 1 --2- 172.21.15.29:0/3259082934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc0ccaa0 0x7fcebc070f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0cd990 tx=0x7fceb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad93d8b0d4ca79e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:30.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fceba59c640 1 -- 172.21.15.29:0/3259082934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb402ca70 con 0x7fcebc0ccaa0 2023-10-16T14:16:30.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fceba59c640 1 -- 172.21.15.29:0/3259082934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb402cbd0 con 0x7fcebc0ccaa0 2023-10-16T14:16:30.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fceba59c640 1 -- 172.21.15.29:0/3259082934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40343f0 con 0x7fcebc0ccaa0 2023-10-16T14:16:30.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 -- 172.21.15.29:0/3259082934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc0ccaa0 msgr2=0x7fcebc070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 --2- 172.21.15.29:0/3259082934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc0ccaa0 0x7fcebc070f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0cd990 tx=0x7fceb402bed0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 -- 172.21.15.29:0/3259082934 shutdown_connections 2023-10-16T14:16:30.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 --2- 172.21.15.29:0/3259082934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc0ccaa0 0x7fcebc070f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 --2- 172.21.15.29:0/3259082934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc15ae80 0x7fcebc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 --2- 172.21.15.29:0/3259082934 >> v2:172.21.15.170:3300/0 conn(0x7fcebc15aaa0 0x7fcebc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.692+0000 7fcec11f5640 1 -- 172.21.15.29:0/3259082934 >> 172.21.15.29:0/3259082934 conn(0x7fcebc0cb410 msgr2=0x7fcebc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:30.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.693+0000 7fcec11f5640 1 -- 172.21.15.29:0/3259082934 shutdown_connections 2023-10-16T14:16:30.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.693+0000 7fcec11f5640 1 -- 172.21.15.29:0/3259082934 wait complete. 2023-10-16T14:16:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.694+0000 7fcec11f5640 1 Processor -- start 2023-10-16T14:16:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.694+0000 7fcec11f5640 1 -- start start 2023-10-16T14:16:30.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.695+0000 7fcec11f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc0ccaa0 0x7fcebc0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.695+0000 7fcec11f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 0x7fcebc0c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.695+0000 7fcec11f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcebc15ae80 0x7fcebc0c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.695+0000 7fcec11f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0c7830 con 0x7fcebc15aaa0 2023-10-16T14:16:30.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.695+0000 7fcec11f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0c5b90 con 0x7fcebc15ae80 2023-10-16T14:16:30.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.695+0000 7fcec11f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0c5e70 con 0x7fcebc0ccaa0 2023-10-16T14:16:30.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 0x7fcebc0c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 0x7fcebc0c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60772/0 (socket says 172.21.15.29:60772) 2023-10-16T14:16:30.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 -- 172.21.15.29:0/2445099644 learned_addr learned my addr 172.21.15.29:0/2445099644 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:30.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fcebad9d640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc0ccaa0 0x7fcebc0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 -- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc0ccaa0 msgr2=0x7fcebc0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fcebb59e640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcebc15ae80 0x7fcebc0c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc0ccaa0 0x7fcebc0ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 -- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcebc15ae80 msgr2=0x7fcebc0c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcebc15ae80 0x7fcebc0c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 -- 172.21.15.29:0/2445099644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb4009b30 con 0x7fcebc15aaa0 2023-10-16T14:16:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb3fff640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 0x7fcebc0c6db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcea400c7f0 tx=0x7fcea400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.696+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4004220 con 0x7fcebc15aaa0 2023-10-16T14:16:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.697+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4004380 con 0x7fcebc15aaa0 2023-10-16T14:16:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.697+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc0c6150 con 0x7fcebc15aaa0 2023-10-16T14:16:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.697+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4007840 con 0x7fcebc15aaa0 2023-10-16T14:16:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.697+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc0c6710 con 0x7fcebc15aaa0 2023-10-16T14:16:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.698+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc04e160 con 0x7fcebc15aaa0 2023-10-16T14:16:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.699+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcea4007bb0 con 0x7fcebc15aaa0 2023-10-16T14:16:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.700+0000 7fceb37fe640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fce9c06ae70 0x7fce9c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.700+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcea4089470 con 0x7fcebc15aaa0 2023-10-16T14:16:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.701+0000 7fcebad9d640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fce9c06ae70 0x7fce9c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.702+0000 7fcebad9d640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fce9c06ae70 0x7fce9c06d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fceac006fd0 tx=0x7fceac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.705+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcea40545b0 con 0x7fcebc15aaa0 2023-10-16T14:16:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.894+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcebc0cdc60 con 0x7fcebc15aaa0 2023-10-16T14:16:30.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.896+0000 7fceb37fe640 1 -- 172.21.15.29:0/2445099644 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fcea4007e60 con 0x7fcebc15aaa0 2023-10-16T14:16:30.901 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:30.901 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:30.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.898+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fce9c06ae70 msgr2=0x7fce9c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.898+0000 7fcec11f5640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fce9c06ae70 0x7fce9c06d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fceac006fd0 tx=0x7fceac008040 comp rx=0 tx=0).stop 2023-10-16T14:16:30.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.898+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 msgr2=0x7fcebc0c6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:30.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.898+0000 7fcec11f5640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 0x7fcebc0c6db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcea400c7f0 tx=0x7fcea400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.898+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 shutdown_connections 2023-10-16T14:16:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fce9c06ae70 0x7fce9c06d330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcebc15aaa0 0x7fcebc0c6db0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcebc15ae80 0x7fcebc0c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 --2- 172.21.15.29:0/2445099644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcebc0ccaa0 0x7fcebc0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:30.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 >> 172.21.15.29:0/2445099644 conn(0x7fcebc0cb410 msgr2=0x7fcebc0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:30.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 shutdown_connections 2023-10-16T14:16:30.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:30.899+0000 7fcec11f5640 1 -- 172.21.15.29:0/2445099644 wait complete. 2023-10-16T14:16:30.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:16:31.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.136+0000 7fb24e4d7640 1 Processor -- start 2023-10-16T14:16:31.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.137+0000 7fb24e4d7640 1 -- start start 2023-10-16T14:16:31.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.138+0000 7fb24e4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248070b60 0x7fb24815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:31.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.138+0000 7fb24e4d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb24806c100 0x7fb24806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:31.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.138+0000 7fb24e4d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24815aef0 con 0x7fb248159950 2023-10-16T14:16:31.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.138+0000 7fb24e4d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24815e490 con 0x7fb24806c100 2023-10-16T14:16:31.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.138+0000 7fb24e4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24815d910 con 0x7fb248070b60 2023-10-16T14:16:31.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.138+0000 7fb24d4d5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb248159950 0x7fb248070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36692/0 (socket says 172.21.15.29:36692) 2023-10-16T14:16:31.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24d4d5640 1 -- 172.21.15.29:0/1608716060 learned_addr learned my addr 172.21.15.29:0/1608716060 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:31.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24dcd6640 1 --2- 172.21.15.29:0/1608716060 >> v2:172.21.15.29:3300/0 conn(0x7fb24806c100 0x7fb24806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:31.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24ccd4640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248070b60 0x7fb24815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:31.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb23ffff640 1 -- 172.21.15.29:0/1608716060 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242514732 0 0) 0x7fb244001110 con 0x7fb248159950 2023-10-16T14:16:31.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb23ffff640 1 -- 172.21.15.29:0/1608716060 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234003520 con 0x7fb248159950 2023-10-16T14:16:31.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb23ffff640 1 -- 172.21.15.29:0/1608716060 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4184480417 0 0) 0x7fb234003520 con 0x7fb248159950 2023-10-16T14:16:31.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24dcd6640 1 -- 172.21.15.29:0/1608716060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248070b60 msgr2=0x7fb24815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:31.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24dcd6640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248070b60 0x7fb24815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:31.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24dcd6640 1 -- 172.21.15.29:0/1608716060 >> v1:172.21.15.29:6789/0 conn(0x7fb248159950 legacy=0x7fb248070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:31.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24dcd6640 1 -- 172.21.15.29:0/1608716060 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24815dbf0 con 0x7fb24806c100 2023-10-16T14:16:31.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.139+0000 7fb24ccd4640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248070b60 0x7fb24815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:31.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.141+0000 7fb24dcd6640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb24806c550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb240009d90 tx=0x7fb24002bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02d30d9ad12c0fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:31.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.141+0000 7fb23ffff640 1 -- 172.21.15.29:0/1608716060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244001800 con 0x7fb24806c100 2023-10-16T14:16:31.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.141+0000 7fb23ffff640 1 -- 172.21.15.29:0/1608716060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24002c810 con 0x7fb24806c100 2023-10-16T14:16:31.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.141+0000 7fb23ffff640 1 -- 172.21.15.29:0/1608716060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24002cba0 con 0x7fb24806c100 2023-10-16T14:16:31.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 -- 172.21.15.29:0/1608716060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 msgr2=0x7fb24806c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:31.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb24806c550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb240009d90 tx=0x7fb24002bce0 comp rx=0 tx=0).stop 2023-10-16T14:16:31.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 -- 172.21.15.29:0/1608716060 shutdown_connections 2023-10-16T14:16:31.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb24806c550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:31.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/1608716060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248070b60 0x7fb24815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:31.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 -- 172.21.15.29:0/1608716060 >> 172.21.15.29:0/1608716060 conn(0x7fb2480ce6a0 msgr2=0x7fb2480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:31.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 -- 172.21.15.29:0/1608716060 shutdown_connections 2023-10-16T14:16:31.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.142+0000 7fb24e4d7640 1 -- 172.21.15.29:0/1608716060 wait complete. 2023-10-16T14:16:31.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.144+0000 7fb24e4d7640 1 Processor -- start 2023-10-16T14:16:31.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.144+0000 7fb24e4d7640 1 -- start start 2023-10-16T14:16:31.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24e4d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb2480c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:31.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24e4d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb248070b60 0x7fb24814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:31.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24e4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248159950 0x7fb2481ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:31.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24e4d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480c6760 con 0x7fb24806c100 2023-10-16T14:16:31.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24e4d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248152a00 con 0x7fb248070b60 2023-10-16T14:16:31.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24e4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480c6530 con 0x7fb248159950 2023-10-16T14:16:31.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24d4d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb2480c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24d4d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb2480c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60790/0 (socket says 172.21.15.29:60790) 2023-10-16T14:16:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24d4d5640 1 -- 172.21.15.29:0/2750005681 learned_addr learned my addr 172.21.15.29:0/2750005681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24ccd4640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb248070b60 0x7fb24814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24dcd6640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248159950 0x7fb2481ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24d4d5640 1 -- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248159950 msgr2=0x7fb2481ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.145+0000 7fb24d4d5640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248159950 0x7fb2481ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24d4d5640 1 -- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb248070b60 msgr2=0x7fb24814e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24d4d5640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb248070b60 0x7fb24814e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24d4d5640 1 -- 172.21.15.29:0/2750005681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240009a70 con 0x7fb24806c100 2023-10-16T14:16:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24ccd4640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb248070b60 0x7fb24814e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24d4d5640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb2480c5ff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb24400c750 tx=0x7fb24400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:31.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244005ad0 con 0x7fb24806c100 2023-10-16T14:16:31.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb244005c30 con 0x7fb24806c100 2023-10-16T14:16:31.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb248152c10 con 0x7fb24806c100 2023-10-16T14:16:31.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244007070 con 0x7fb24806c100 2023-10-16T14:16:31.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.146+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2481531d0 con 0x7fb24806c100 2023-10-16T14:16:31.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.147+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24806bf20 con 0x7fb24806c100 2023-10-16T14:16:31.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.148+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb2440071d0 con 0x7fb24806c100 2023-10-16T14:16:31.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.149+0000 7fb23dffb640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb22c06ae70 0x7fb22c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:31.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.149+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2440892d0 con 0x7fb24806c100 2023-10-16T14:16:31.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.150+0000 7fb24ccd4640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb22c06ae70 0x7fb22c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:31.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.151+0000 7fb24ccd4640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb22c06ae70 0x7fb22c06d330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb238006fd0 tx=0x7fb238008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:31.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.154+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb244054250 con 0x7fb24806c100 2023-10-16T14:16:31.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:31.382+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb24804e9b0 con 0x7fb24806c100 2023-10-16T14:16:32.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.047+0000 7fb23dffb640 1 -- 172.21.15.29:0/2750005681 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb244057870 con 0x7fb24806c100 2023-10-16T14:16:32.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.050+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb22c06ae70 msgr2=0x7fb22c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.050+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb22c06ae70 0x7fb22c06d330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb238006fd0 tx=0x7fb238008040 comp rx=0 tx=0).stop 2023-10-16T14:16:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.050+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 msgr2=0x7fb2480c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.050+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb2480c5ff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb24400c750 tx=0x7fb24400cc10 comp rx=0 tx=0).stop 2023-10-16T14:16:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 shutdown_connections 2023-10-16T14:16:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb22c06ae70 0x7fb22c06d330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb248159950 0x7fb2481ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb248070b60 0x7fb24814e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 --2- 172.21.15.29:0/2750005681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24806c100 0x7fb2480c5ff0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 >> 172.21.15.29:0/2750005681 conn(0x7fb2480ce6a0 msgr2=0x7fb248070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 shutdown_connections 2023-10-16T14:16:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.051+0000 7fb24e4d7640 1 -- 172.21.15.29:0/2750005681 wait complete. 2023-10-16T14:16:32.075 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:16:32.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:16:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.300+0000 7f333a87c640 1 Processor -- start 2023-10-16T14:16:32.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.300+0000 7f333a87c640 1 -- start start 2023-10-16T14:16:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.302+0000 7f333a87c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33340c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.302+0000 7f333a87c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340cc6b0 con 0x7f33340c9690 2023-10-16T14:16:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.302+0000 7f333a87c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33341599b0 con 0x7f333406b660 2023-10-16T14:16:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.302+0000 7f333a87c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340c9b20 con 0x7f33341428c0 2023-10-16T14:16:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.302+0000 7f3338e19640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f33340c9690 0x7f33340ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36704/0 (socket says 172.21.15.29:36704) 2023-10-16T14:16:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.302+0000 7f3338e19640 1 -- 172.21.15.29:0/1957772024 learned_addr learned my addr 172.21.15.29:0/1957772024 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:32.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f332b7fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33340c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322026559 0 0) 0x7f33340cc6b0 con 0x7f33340c9690 2023-10-16T14:16:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3320003520 con 0x7f33340c9690 2023-10-16T14:16:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761932024 0 0) 0x7f3320003520 con 0x7f33340c9690 2023-10-16T14:16:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 msgr2=0x7f33340c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33340c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 >> v1:172.21.15.170:6789/0 conn(0x7f333406b660 legacy=0x7f333406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333406c290 con 0x7f33340c9690 2023-10-16T14:16:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f332c001670 con 0x7f33340c9690 2023-10-16T14:16:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 >> v1:172.21.15.29:6789/0 conn(0x7f33340c9690 legacy=0x7f33340ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33200039a0 0x7f3320003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3320004d00 0x7f3320005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340c9b20 con 0x7f3320004d00 2023-10-16T14:16:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332000ba60 con 0x7f33200039a0 2023-10-16T14:16:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.303+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332000bc20 con 0x7f33341428c0 2023-10-16T14:16:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3338e19640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3320004d00 0x7f3320005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3333fff640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33200039a0 0x7f3320003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f332b7fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3338e19640 1 -- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 msgr2=0x7f33200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3338e19640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3338e19640 1 -- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33200039a0 msgr2=0x7f3320003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3338e19640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33200039a0 0x7f3320003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3338e19640 1 -- 172.21.15.29:0/1957772024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320003520 con 0x7f3320004d00 2023-10-16T14:16:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f332b7fe640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f33341428c0 0x7f33200047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.304+0000 7f3333fff640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33200039a0 0x7f3320003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.305+0000 7f3338e19640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3320004d00 0x7f3320005180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f332c001eb0 tx=0x7f33200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ed267d85432939 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.305+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c02ebd0 con 0x7f3320004d00 2023-10-16T14:16:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.305+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332c02ed30 con 0x7f3320004d00 2023-10-16T14:16:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.305+0000 7f33337fe640 1 -- 172.21.15.29:0/1957772024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c005b10 con 0x7f3320004d00 2023-10-16T14:16:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f333a87c640 1 -- 172.21.15.29:0/1957772024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3320004d00 msgr2=0x7f3320005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f333a87c640 1 --2- 172.21.15.29:0/1957772024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3320004d00 0x7f3320005180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f332c001eb0 tx=0x7f33200057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f3338e19640 1 -- 172.21.15.29:0/1957772024 reap_dead start 2023-10-16T14:16:32.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f333a87c640 1 -- 172.21.15.29:0/1957772024 shutdown_connections 2023-10-16T14:16:32.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f333a87c640 1 -- 172.21.15.29:0/1957772024 >> 172.21.15.29:0/1957772024 conn(0x7f33340ce9b0 msgr2=0x7f33340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:32.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f333a87c640 1 -- 172.21.15.29:0/1957772024 shutdown_connections 2023-10-16T14:16:32.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.306+0000 7f333a87c640 1 -- 172.21.15.29:0/1957772024 wait complete. 2023-10-16T14:16:32.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.308+0000 7f333a87c640 1 Processor -- start 2023-10-16T14:16:32.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.308+0000 7f333a87c640 1 -- start start 2023-10-16T14:16:32.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f333a87c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3320004d00 0x7f3334153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f333a87c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 0x7f333415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f333a87c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33340af690 0x7f33340afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f333a87c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333415ba60 con 0x7f333415b0a0 2023-10-16T14:16:32.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f333a87c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340c5a80 con 0x7f33340af690 2023-10-16T14:16:32.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f333a87c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b03c0 con 0x7f3320004d00 2023-10-16T14:16:32.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f332b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 0x7f333415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f332b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 0x7f333415b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60806/0 (socket says 172.21.15.29:60806) 2023-10-16T14:16:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.309+0000 7f332b7fe640 1 -- 172.21.15.29:0/1382499685 learned_addr learned my addr 172.21.15.29:0/1382499685 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f3333fff640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3320004d00 0x7f3334153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f3338e19640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33340af690 0x7f33340afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f332b7fe640 1 -- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3320004d00 msgr2=0x7f3334153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f332b7fe640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3320004d00 0x7f3334153d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f332b7fe640 1 -- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33340af690 msgr2=0x7f33340afb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f332b7fe640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33340af690 0x7f33340afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f332b7fe640 1 -- 172.21.15.29:0/1382499685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340b06a0 con 0x7f333415b0a0 2023-10-16T14:16:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f332b7fe640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 0x7f333415b520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f331c0108b0 tx=0x7f331c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c00eb60 con 0x7f333415b0a0 2023-10-16T14:16:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c00ecc0 con 0x7f333415b0a0 2023-10-16T14:16:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c01c400 con 0x7f333415b0a0 2023-10-16T14:16:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.310+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33340b0980 con 0x7f333415b0a0 2023-10-16T14:16:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.311+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3334154270 con 0x7f333415b0a0 2023-10-16T14:16:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.312+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333404e160 con 0x7f333415b0a0 2023-10-16T14:16:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.313+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f331c01c560 con 0x7f333415b0a0 2023-10-16T14:16:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.313+0000 7f33317fa640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f331406aea0 0x7f331406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.314+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f331c089c60 con 0x7f333415b0a0 2023-10-16T14:16:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.314+0000 7f3333fff640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f331406aea0 0x7f331406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.315+0000 7f3333fff640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f331406aea0 0x7f331406d360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3324007fd0 tx=0x7f3324009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.318+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f331c054d20 con 0x7f333415b0a0 2023-10-16T14:16:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:32.551+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f333406ac60 con 0x7f333415b0a0 2023-10-16T14:16:33.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.058+0000 7f33317fa640 1 -- 172.21.15.29:0/1382499685 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f331c058340 con 0x7f333415b0a0 2023-10-16T14:16:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f331406aea0 msgr2=0x7f331406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f331406aea0 0x7f331406d360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3324007fd0 tx=0x7f3324009040 comp rx=0 tx=0).stop 2023-10-16T14:16:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 msgr2=0x7f333415b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 0x7f333415b520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f331c0108b0 tx=0x7f331c010d70 comp rx=0 tx=0).stop 2023-10-16T14:16:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 shutdown_connections 2023-10-16T14:16:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f331406aea0 0x7f331406d360 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f333415b0a0 0x7f333415b520 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f33340af690 0x7f33340afb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 --2- 172.21.15.29:0/1382499685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3320004d00 0x7f3334153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.061+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 >> 172.21.15.29:0/1382499685 conn(0x7f33340ce9b0 msgr2=0x7f33340c5010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.062+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 shutdown_connections 2023-10-16T14:16:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.062+0000 7f333a87c640 1 -- 172.21.15.29:0/1382499685 wait complete. 2023-10-16T14:16:33.089 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:16:33.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:16:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.302+0000 7f869d840640 1 Processor -- start 2023-10-16T14:16:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.302+0000 7f869d840640 1 -- start start 2023-10-16T14:16:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f869d840640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86980b4c80 0x7f8698167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f869d840640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86980d15a0 con 0x7f869815cfc0 2023-10-16T14:16:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f869d840640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86980d1880 con 0x7f869815cbe0 2023-10-16T14:16:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f869d840640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86980d1b60 con 0x7f86980b4c80 2023-10-16T14:16:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f869d03f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86980b4c80 0x7f8698167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f8697fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f869815cfc0 0x7f86980b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36710/0 (socket says 172.21.15.29:36710) 2023-10-16T14:16:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.303+0000 7f8697fff640 1 -- 172.21.15.29:0/527401375 learned_addr learned my addr 172.21.15.29:0/527401375 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.304+0000 7f869d03f640 1 -- 172.21.15.29:0/527401375 >> v1:172.21.15.170:6789/0 conn(0x7f869815cbe0 legacy=0x7f86980d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.304+0000 7f869d03f640 1 -- 172.21.15.29:0/527401375 >> v1:172.21.15.29:6789/0 conn(0x7f869815cfc0 legacy=0x7f86980b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.304+0000 7f869d03f640 1 -- 172.21.15.29:0/527401375 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86980d1e40 con 0x7f86980b4c80 2023-10-16T14:16:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.304+0000 7f86977fe640 1 -- 172.21.15.29:0/527401375 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699717832 0 0) 0x7f86980d15a0 con 0x7f869815cfc0 2023-10-16T14:16:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.305+0000 7f869d03f640 1 --2- 172.21.15.29:0/527401375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86980b4c80 0x7f8698167910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f86900097a0 tx=0x7f869002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290d31c71e49e5ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.305+0000 7f86977fe640 1 -- 172.21.15.29:0/527401375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690030610 con 0x7f86980b4c80 2023-10-16T14:16:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.305+0000 7f86977fe640 1 -- 172.21.15.29:0/527401375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8690030770 con 0x7f86980b4c80 2023-10-16T14:16:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.305+0000 7f86977fe640 1 -- 172.21.15.29:0/527401375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690030b00 con 0x7f86980b4c80 2023-10-16T14:16:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 -- 172.21.15.29:0/527401375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86980b4c80 msgr2=0x7f8698167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 --2- 172.21.15.29:0/527401375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86980b4c80 0x7f8698167910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f86900097a0 tx=0x7f869002b960 comp rx=0 tx=0).stop 2023-10-16T14:16:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 -- 172.21.15.29:0/527401375 shutdown_connections 2023-10-16T14:16:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 --2- 172.21.15.29:0/527401375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86980b4c80 0x7f8698167910 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 -- 172.21.15.29:0/527401375 >> 172.21.15.29:0/527401375 conn(0x7f86980cf570 msgr2=0x7f86980cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 -- 172.21.15.29:0/527401375 shutdown_connections 2023-10-16T14:16:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.306+0000 7f869d840640 1 -- 172.21.15.29:0/527401375 wait complete. 2023-10-16T14:16:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.308+0000 7f869d840640 1 Processor -- start 2023-10-16T14:16:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.308+0000 7f869d840640 1 -- start start 2023-10-16T14:16:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86980b4c80 0x7f86980aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869815cbe0 0x7f86981f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d840640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 0x7f86981f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d840640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86980ca4c0 con 0x7f869815cfc0 2023-10-16T14:16:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86980ca7a0 con 0x7f86980b4c80 2023-10-16T14:16:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86980caa80 con 0x7f869815cbe0 2023-10-16T14:16:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d03f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 0x7f86981f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d03f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 0x7f86981f4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60824/0 (socket says 172.21.15.29:60824) 2023-10-16T14:16:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.309+0000 7f869d03f640 1 -- 172.21.15.29:0/3477297120 learned_addr learned my addr 172.21.15.29:0/3477297120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f8697fff640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869815cbe0 0x7f86981f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869c83e640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86980b4c80 0x7f86980aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d03f640 1 -- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869815cbe0 msgr2=0x7f86981f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d03f640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869815cbe0 0x7f86981f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d03f640 1 -- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86980b4c80 msgr2=0x7f86980aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d03f640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86980b4c80 0x7f86980aafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d03f640 1 -- 172.21.15.29:0/3477297120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8690009400 con 0x7f869815cfc0 2023-10-16T14:16:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f8697fff640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869815cbe0 0x7f86981f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d03f640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 0x7f86981f4db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f86900098c0 tx=0x7f8690004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869003e040 con 0x7f869815cfc0 2023-10-16T14:16:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8690039c10 con 0x7f869815cfc0 2023-10-16T14:16:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690030e10 con 0x7f869815cfc0 2023-10-16T14:16:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.310+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86980cad60 con 0x7f869815cfc0 2023-10-16T14:16:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.311+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86981f52f0 con 0x7f869815cfc0 2023-10-16T14:16:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.312+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86980cb280 con 0x7f869815cfc0 2023-10-16T14:16:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.313+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f869004d3b0 con 0x7f869815cfc0 2023-10-16T14:16:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.313+0000 7f86957fa640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f868406aea0 0x7f868406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.314+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86900abcb0 con 0x7f869815cfc0 2023-10-16T14:16:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.314+0000 7f869c83e640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f868406aea0 0x7f868406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.315+0000 7f869c83e640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f868406aea0 0x7f868406d360 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8688007fd0 tx=0x7f8688009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.318+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86980cb280 con 0x7f869815cfc0 2023-10-16T14:16:33.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.553+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86980cb280 con 0x7f869815cfc0 2023-10-16T14:16:33.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.553+0000 7f86957fa640 1 -- 172.21.15.29:0/3477297120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f86980cb280 con 0x7f869815cfc0 2023-10-16T14:16:33.558 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:33.558 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:16:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:16:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.556+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f868406aea0 msgr2=0x7f868406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.556+0000 7f869d840640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f868406aea0 0x7f868406d360 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8688007fd0 tx=0x7f8688009040 comp rx=0 tx=0).stop 2023-10-16T14:16:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.556+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 msgr2=0x7f86981f4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.556+0000 7f869d840640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 0x7f86981f4db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f86900098c0 tx=0x7f8690004730 comp rx=0 tx=0).stop 2023-10-16T14:16:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 shutdown_connections 2023-10-16T14:16:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f868406aea0 0x7f868406d360 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869815cfc0 0x7f86981f4db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869815cbe0 0x7f86981f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 --2- 172.21.15.29:0/3477297120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86980b4c80 0x7f86980aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 >> 172.21.15.29:0/3477297120 conn(0x7f86980cf570 msgr2=0x7f86980b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 shutdown_connections 2023-10-16T14:16:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.557+0000 7f869d840640 1 -- 172.21.15.29:0/3477297120 wait complete. 2023-10-16T14:16:33.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:33.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.804+0000 7f9b0dabb640 1 Processor -- start 2023-10-16T14:16:33.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.804+0000 7f9b0dabb640 1 -- start start 2023-10-16T14:16:33.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.806+0000 7f9b0dabb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b080c9060 0x7f9b0806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.806+0000 7f9b0dabb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b080ca560 con 0x7f9b0806c9e0 2023-10-16T14:16:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.806+0000 7f9b0dabb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b08165b70 con 0x7f9b080c9440 2023-10-16T14:16:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.806+0000 7f9b0dabb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b080c9950 con 0x7f9b080c9060 2023-10-16T14:16:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b07fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b0806c9e0 0x7f9b08164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36712/0 (socket says 172.21.15.29:36712) 2023-10-16T14:16:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b0cab9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b080c9060 0x7f9b0806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b07fff640 1 -- 172.21.15.29:0/2001538271 learned_addr learned my addr 172.21.15.29:0/2001538271 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b0cab9640 1 -- 172.21.15.29:0/2001538271 >> v1:172.21.15.170:6789/0 conn(0x7f9b080c9440 legacy=0x7f9b08165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:33.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b0cab9640 1 -- 172.21.15.29:0/2001538271 >> v1:172.21.15.29:6789/0 conn(0x7f9b0806c9e0 legacy=0x7f9b08164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b077fe640 1 -- 172.21.15.29:0/2001538271 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37100573 0 0) 0x7f9b080ca560 con 0x7f9b0806c9e0 2023-10-16T14:16:33.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.807+0000 7f9b0cab9640 1 -- 172.21.15.29:0/2001538271 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b080c9c30 con 0x7f9b080c9060 2023-10-16T14:16:33.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.808+0000 7f9b0cab9640 1 --2- 172.21.15.29:0/2001538271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b0806f440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0097a0 tx=0x7f9afc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d546fa90fc8549 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:33.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b077fe640 1 -- 172.21.15.29:0/2001538271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc030610 con 0x7f9b080c9060 2023-10-16T14:16:33.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b077fe640 1 -- 172.21.15.29:0/2001538271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc030770 con 0x7f9b080c9060 2023-10-16T14:16:33.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b077fe640 1 -- 172.21.15.29:0/2001538271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc030b00 con 0x7f9b080c9060 2023-10-16T14:16:33.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b0dabb640 1 -- 172.21.15.29:0/2001538271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 msgr2=0x7f9b0806f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/2001538271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b0806f440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0097a0 tx=0x7f9afc02b960 comp rx=0 tx=0).stop 2023-10-16T14:16:33.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b0dabb640 1 -- 172.21.15.29:0/2001538271 shutdown_connections 2023-10-16T14:16:33.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/2001538271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b0806f440 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.809+0000 7f9b0dabb640 1 -- 172.21.15.29:0/2001538271 >> 172.21.15.29:0/2001538271 conn(0x7f9b080ce6a0 msgr2=0x7f9b080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:33.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.810+0000 7f9b0dabb640 1 -- 172.21.15.29:0/2001538271 shutdown_connections 2023-10-16T14:16:33.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.810+0000 7f9b0dabb640 1 -- 172.21.15.29:0/2001538271 wait complete. 2023-10-16T14:16:33.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.812+0000 7f9b0dabb640 1 Processor -- start 2023-10-16T14:16:33.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.812+0000 7f9b0dabb640 1 -- start start 2023-10-16T14:16:33.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.813+0000 7f9b0dabb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b0806c9e0 0x7f9b081f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.813+0000 7f9b0dabb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b081f9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.813+0000 7f9b0dabb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b080c9440 0x7f9b0815b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.813+0000 7f9b0dabb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b081f7e40 con 0x7f9b080c9060 2023-10-16T14:16:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.813+0000 7f9b0dabb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0815ba30 con 0x7f9b0806c9e0 2023-10-16T14:16:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.813+0000 7f9b0dabb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0815bc90 con 0x7f9b080c9440 2023-10-16T14:16:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b081f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b081f9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60834/0 (socket says 172.21.15.29:60834) 2023-10-16T14:16:33.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 -- 172.21.15.29:0/3888158512 learned_addr learned my addr 172.21.15.29:0/3888158512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 -- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b080c9440 msgr2=0x7f9b0815b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b080c9440 0x7f9b0815b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 -- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b0806c9e0 msgr2=0x7f9b081f7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b0806c9e0 0x7f9b081f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 -- 172.21.15.29:0/3888158512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc009400 con 0x7f9b080c9060 2023-10-16T14:16:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.814+0000 7f9b07fff640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b081f9890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9af400c3c0 tx=0x7f9af400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.815+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40074e0 con 0x7f9b080c9060 2023-10-16T14:16:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.815+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4007640 con 0x7f9b080c9060 2023-10-16T14:16:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.815+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4004a20 con 0x7f9b080c9060 2023-10-16T14:16:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.815+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0815bf70 con 0x7f9b080c9060 2023-10-16T14:16:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.815+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0815c530 con 0x7f9b080c9060 2023-10-16T14:16:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.816+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0815b100 con 0x7f9b080c9060 2023-10-16T14:16:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.817+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9af4004b80 con 0x7f9b080c9060 2023-10-16T14:16:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.818+0000 7f9b057fa640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9aec06ae70 0x7f9aec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.818+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9af4089460 con 0x7f9b080c9060 2023-10-16T14:16:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.819+0000 7f9b0cab9640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9aec06ae70 0x7f9aec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.820+0000 7f9b0cab9640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9aec06ae70 0x7f9aec06d330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9afc036000 tx=0x7f9afc03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:33.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:33.823+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9af4054500 con 0x7f9b080c9060 2023-10-16T14:16:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.020+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b0804e9b0 con 0x7f9b080c9060 2023-10-16T14:16:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.023+0000 7f9b057fa640 1 -- 172.21.15.29:0/3888158512 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f9af4057b20 con 0x7f9b080c9060 2023-10-16T14:16:34.027 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:34.028 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9aec06ae70 msgr2=0x7f9aec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9aec06ae70 0x7f9aec06d330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9afc036000 tx=0x7f9afc03b040 comp rx=0 tx=0).stop 2023-10-16T14:16:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 msgr2=0x7f9b081f9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b081f9890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9af400c3c0 tx=0x7f9af400c880 comp rx=0 tx=0).stop 2023-10-16T14:16:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 shutdown_connections 2023-10-16T14:16:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9aec06ae70 0x7f9aec06d330 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b080c9060 0x7f9b081f9890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b080c9440 0x7f9b0815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 --2- 172.21.15.29:0/3888158512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b0806c9e0 0x7f9b081f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.025+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 >> 172.21.15.29:0/3888158512 conn(0x7f9b080ce6a0 msgr2=0x7f9b0806a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.026+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 shutdown_connections 2023-10-16T14:16:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.026+0000 7f9b0dabb640 1 -- 172.21.15.29:0/3888158512 wait complete. 2023-10-16T14:16:34.049 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:16:34.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:16:34.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.260+0000 7f8006c1a640 1 Processor -- start 2023-10-16T14:16:34.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.260+0000 7f8006c1a640 1 -- start start 2023-10-16T14:16:34.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006c1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f800006c040 0x7f800006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006c1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f800015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006c1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800006c960 con 0x7f800015ebc0 2023-10-16T14:16:34.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006c1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800015b9d0 con 0x7f800006c040 2023-10-16T14:16:34.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006c1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800015bcb0 con 0x7f80000ad970 2023-10-16T14:16:34.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8005c18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f800006c040 0x7f800006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006419640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f800015ebc0 0x7f80000cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46296/0 (socket says 172.21.15.29:46296) 2023-10-16T14:16:34.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8006419640 1 -- 172.21.15.29:0/3381352992 learned_addr learned my addr 172.21.15.29:0/3381352992 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:34.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8005c18640 1 -- 172.21.15.29:0/3381352992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 msgr2=0x7f800015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:34.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8005c18640 1 --2- 172.21.15.29:0/3381352992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f800015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.262+0000 7f8005c18640 1 -- 172.21.15.29:0/3381352992 >> v1:172.21.15.29:6789/0 conn(0x7f800015ebc0 legacy=0x7f80000cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.263+0000 7f8005c18640 1 -- 172.21.15.29:0/3381352992 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800015bf90 con 0x7f800006c040 2023-10-16T14:16:34.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.263+0000 7f8004c16640 1 -- 172.21.15.29:0/3381352992 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170434473 0 0) 0x7f800006c960 con 0x7f800015ebc0 2023-10-16T14:16:34.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.264+0000 7f8005c18640 1 --2- 172.21.15.29:0/3381352992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f800006c420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0097d0 tx=0x7f7ffc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b3297642025502 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:34.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.264+0000 7f8004c16640 1 -- 172.21.15.29:0/3381352992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc031800 con 0x7f800006c040 2023-10-16T14:16:34.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.264+0000 7f8004c16640 1 -- 172.21.15.29:0/3381352992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc031960 con 0x7f800006c040 2023-10-16T14:16:34.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.264+0000 7f8004c16640 1 -- 172.21.15.29:0/3381352992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc031cf0 con 0x7f800006c040 2023-10-16T14:16:34.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 -- 172.21.15.29:0/3381352992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 msgr2=0x7f800006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 --2- 172.21.15.29:0/3381352992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f800006c420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0097d0 tx=0x7f7ffc02b8a0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 -- 172.21.15.29:0/3381352992 shutdown_connections 2023-10-16T14:16:34.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 --2- 172.21.15.29:0/3381352992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f800015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 --2- 172.21.15.29:0/3381352992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f800006c420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 -- 172.21.15.29:0/3381352992 >> 172.21.15.29:0/3381352992 conn(0x7f80000ce6a0 msgr2=0x7f80000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:34.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 -- 172.21.15.29:0/3381352992 shutdown_connections 2023-10-16T14:16:34.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.265+0000 7f8006c1a640 1 -- 172.21.15.29:0/3381352992 wait complete. 2023-10-16T14:16:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.267+0000 7f8006c1a640 1 Processor -- start 2023-10-16T14:16:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.267+0000 7f8006c1a640 1 -- start start 2023-10-16T14:16:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f80001f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006c1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f80001f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006c1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f800015ebc0 0x7f8000156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006c1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80000ca010 con 0x7f800006c040 2023-10-16T14:16:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006c1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001f37f0 con 0x7f800015ebc0 2023-10-16T14:16:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006c1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80000c5c80 con 0x7f80000ad970 2023-10-16T14:16:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8005c18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f80001f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8005c18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f80001f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50610/0 (socket says 172.21.15.29:50610) 2023-10-16T14:16:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8005c18640 1 -- 172.21.15.29:0/900324772 learned_addr learned my addr 172.21.15.29:0/900324772 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8005417640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f80001f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8006419640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f800015ebc0 0x7f8000156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8005c18640 1 -- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 msgr2=0x7f80001f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.268+0000 7f8005c18640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f80001f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8005c18640 1 -- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f800015ebc0 msgr2=0x7f8000156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8005c18640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f800015ebc0 0x7f8000156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8005c18640 1 -- 172.21.15.29:0/900324772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc009400 con 0x7f800006c040 2023-10-16T14:16:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8006419640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f800015ebc0 0x7f8000156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8005c18640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f80001f32b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0097d0 tx=0x7f7ffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc03ad70 con 0x7f800006c040 2023-10-16T14:16:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc0443f0 con 0x7f800006c040 2023-10-16T14:16:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0447a0 con 0x7f800006c040 2023-10-16T14:16:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80000c5f60 con 0x7f800006c040 2023-10-16T14:16:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.269+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80000c6520 con 0x7f800006c040 2023-10-16T14:16:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.271+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800004e9b0 con 0x7f800006c040 2023-10-16T14:16:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.271+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ffc04d450 con 0x7f800006c040 2023-10-16T14:16:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.273+0000 7f7ff67fc640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7fe406ae70 0x7f7fe406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.273+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ffc02e070 con 0x7f800006c040 2023-10-16T14:16:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.274+0000 7f8005417640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7fe406ae70 0x7f7fe406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.275+0000 7f8005417640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7fe406ae70 0x7f7fe406d330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f800006caa0 tx=0x7f7ff0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.277+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ffc076b50 con 0x7f800006c040 2023-10-16T14:16:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.469+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80000cc550 con 0x7f7fe406ae70 2023-10-16T14:16:34.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.495+0000 7f7ff67fc640 1 -- 172.21.15.29:0/900324772 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264434 (secure 0 0 0) 0x7f7ff000bdc0 con 0x7f7fe406ae70 2023-10-16T14:16:34.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:16:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7fe406ae70 msgr2=0x7f7fe406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7fe406ae70 0x7f7fe406d330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f800006caa0 tx=0x7f7ff0008040 comp rx=0 tx=0).stop 2023-10-16T14:16:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 msgr2=0x7f80001f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f80001f32b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0097d0 tx=0x7f7ffc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 shutdown_connections 2023-10-16T14:16:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7fe406ae70 0x7f7fe406d330 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f800015ebc0 0x7f8000156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80000ad970 0x7f80001f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 --2- 172.21.15.29:0/900324772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800006c040 0x7f80001f32b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.501+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 >> 172.21.15.29:0/900324772 conn(0x7f80000ce6a0 msgr2=0x7f800015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.502+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 shutdown_connections 2023-10-16T14:16:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.502+0000 7f8006c1a640 1 -- 172.21.15.29:0/900324772 wait complete. 2023-10-16T14:16:34.525 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-16T14:16:33.811369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252284,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722436,"statfs":{"total":773094113280,"available":772835774464,"internally_reserved":0,"allocated":2891776,"data_stored":1752740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":255315513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.808518"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799225+0000","last_change":"2023-10-16T14:16:29.830118+0000","last_active":"2023-10-16T14:16:30.799225+0000","last_peered":"2023-10-16T14:16:30.799225+0000","last_clean":"2023-10-16T14:16:30.799225+0000","last_became_active":"2023-10-16T14:16:29.829372+0000","last_became_peered":"2023-10-16T14:16:29.829372+0000","last_unstale":"2023-10-16T14:16:30.799225+0000","last_undegraded":"2023-10-16T14:16:30.799225+0000","last_fullsized":"2023-10-16T14:16:30.799225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794763+0000","last_change":"2023-10-16T14:16:29.818444+0000","last_active":"2023-10-16T14:16:30.794763+0000","last_peered":"2023-10-16T14:16:30.794763+0000","last_clean":"2023-10-16T14:16:30.794763+0000","last_became_active":"2023-10-16T14:16:29.816215+0000","last_became_peered":"2023-10-16T14:16:29.816215+0000","last_unstale":"2023-10-16T14:16:30.794763+0000","last_undegraded":"2023-10-16T14:16:30.794763+0000","last_fullsized":"2023-10-16T14:16:30.794763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326440+0000","last_change":"2023-10-16T14:16:29.817304+0000","last_active":"2023-10-16T14:16:32.326440+0000","last_peered":"2023-10-16T14:16:32.326440+0000","last_clean":"2023-10-16T14:16:32.326440+0000","last_became_active":"2023-10-16T14:16:29.817037+0000","last_became_peered":"2023-10-16T14:16:29.817037+0000","last_unstale":"2023-10-16T14:16:32.326440+0000","last_undegraded":"2023-10-16T14:16:32.326440+0000","last_fullsized":"2023-10-16T14:16:32.326440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827089+0000","last_change":"2023-10-16T14:16:29.816180+0000","last_active":"2023-10-16T14:16:31.827089+0000","last_peered":"2023-10-16T14:16:31.827089+0000","last_clean":"2023-10-16T14:16:31.827089+0000","last_became_active":"2023-10-16T14:16:29.815841+0000","last_became_peered":"2023-10-16T14:16:29.815841+0000","last_unstale":"2023-10-16T14:16:31.827089+0000","last_undegraded":"2023-10-16T14:16:31.827089+0000","last_fullsized":"2023-10-16T14:16:31.827089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798716+0000","last_change":"2023-10-16T14:16:29.813648+0000","last_active":"2023-10-16T14:16:30.798716+0000","last_peered":"2023-10-16T14:16:30.798716+0000","last_clean":"2023-10-16T14:16:30.798716+0000","last_became_active":"2023-10-16T14:16:29.813365+0000","last_became_peered":"2023-10-16T14:16:29.813365+0000","last_unstale":"2023-10-16T14:16:30.798716+0000","last_undegraded":"2023-10-16T14:16:30.798716+0000","last_fullsized":"2023-10-16T14:16:30.798716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825920+0000","last_change":"2023-10-16T14:16:29.818850+0000","last_active":"2023-10-16T14:16:31.825920+0000","last_peered":"2023-10-16T14:16:31.825920+0000","last_clean":"2023-10-16T14:16:31.825920+0000","last_became_active":"2023-10-16T14:16:29.818486+0000","last_became_peered":"2023-10-16T14:16:29.818486+0000","last_unstale":"2023-10-16T14:16:31.825920+0000","last_undegraded":"2023-10-16T14:16:31.825920+0000","last_fullsized":"2023-10-16T14:16:31.825920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796628+0000","last_change":"2023-10-16T14:16:29.814386+0000","last_active":"2023-10-16T14:16:30.796628+0000","last_peered":"2023-10-16T14:16:30.796628+0000","last_clean":"2023-10-16T14:16:30.796628+0000","last_became_active":"2023-10-16T14:16:29.812379+0000","last_became_peered":"2023-10-16T14:16:29.812379+0000","last_unstale":"2023-10-16T14:16:30.796628+0000","last_undegraded":"2023-10-16T14:16:30.796628+0000","last_fullsized":"2023-10-16T14:16:30.796628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667250+0000","last_change":"2023-10-16T14:16:29.823474+0000","last_active":"2023-10-16T14:16:31.667250+0000","last_peered":"2023-10-16T14:16:31.667250+0000","last_clean":"2023-10-16T14:16:31.667250+0000","last_became_active":"2023-10-16T14:16:29.823172+0000","last_became_peered":"2023-10-16T14:16:29.823172+0000","last_unstale":"2023-10-16T14:16:31.667250+0000","last_undegraded":"2023-10-16T14:16:31.667250+0000","last_fullsized":"2023-10-16T14:16:31.667250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799207+0000","last_change":"2023-10-16T14:16:29.830964+0000","last_active":"2023-10-16T14:16:30.799207+0000","last_peered":"2023-10-16T14:16:30.799207+0000","last_clean":"2023-10-16T14:16:30.799207+0000","last_became_active":"2023-10-16T14:16:29.829757+0000","last_became_peered":"2023-10-16T14:16:29.829757+0000","last_unstale":"2023-10-16T14:16:30.799207+0000","last_undegraded":"2023-10-16T14:16:30.799207+0000","last_fullsized":"2023-10-16T14:16:30.799207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961588+0000","last_change":"2023-10-16T14:16:29.822512+0000","last_active":"2023-10-16T14:16:30.961588+0000","last_peered":"2023-10-16T14:16:30.961588+0000","last_clean":"2023-10-16T14:16:30.961588+0000","last_became_active":"2023-10-16T14:16:29.822050+0000","last_became_peered":"2023-10-16T14:16:29.822050+0000","last_unstale":"2023-10-16T14:16:30.961588+0000","last_undegraded":"2023-10-16T14:16:30.961588+0000","last_fullsized":"2023-10-16T14:16:30.961588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961278+0000","last_change":"2023-10-16T14:16:29.819724+0000","last_active":"2023-10-16T14:16:30.961278+0000","last_peered":"2023-10-16T14:16:30.961278+0000","last_clean":"2023-10-16T14:16:30.961278+0000","last_became_active":"2023-10-16T14:16:29.819467+0000","last_became_peered":"2023-10-16T14:16:29.819467+0000","last_unstale":"2023-10-16T14:16:30.961278+0000","last_undegraded":"2023-10-16T14:16:30.961278+0000","last_fullsized":"2023-10-16T14:16:30.961278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797593+0000","last_change":"2023-10-16T14:16:29.819218+0000","last_active":"2023-10-16T14:16:30.797593+0000","last_peered":"2023-10-16T14:16:30.797593+0000","last_clean":"2023-10-16T14:16:30.797593+0000","last_became_active":"2023-10-16T14:16:29.818622+0000","last_became_peered":"2023-10-16T14:16:29.818622+0000","last_unstale":"2023-10-16T14:16:30.797593+0000","last_undegraded":"2023-10-16T14:16:30.797593+0000","last_fullsized":"2023-10-16T14:16:30.797593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668213+0000","last_change":"2023-10-16T14:16:29.817208+0000","last_active":"2023-10-16T14:16:31.668213+0000","last_peered":"2023-10-16T14:16:31.668213+0000","last_clean":"2023-10-16T14:16:31.668213+0000","last_became_active":"2023-10-16T14:16:29.816175+0000","last_became_peered":"2023-10-16T14:16:29.816175+0000","last_unstale":"2023-10-16T14:16:31.668213+0000","last_undegraded":"2023-10-16T14:16:31.668213+0000","last_fullsized":"2023-10-16T14:16:31.668213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325899+0000","last_change":"2023-10-16T14:16:29.812726+0000","last_active":"2023-10-16T14:16:32.325899+0000","last_peered":"2023-10-16T14:16:32.325899+0000","last_clean":"2023-10-16T14:16:32.325899+0000","last_became_active":"2023-10-16T14:16:29.812358+0000","last_became_peered":"2023-10-16T14:16:29.812358+0000","last_unstale":"2023-10-16T14:16:32.325899+0000","last_undegraded":"2023-10-16T14:16:32.325899+0000","last_fullsized":"2023-10-16T14:16:32.325899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:59.947139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794118+0000","last_change":"2023-10-16T14:16:29.820374+0000","last_active":"2023-10-16T14:16:30.794118+0000","last_peered":"2023-10-16T14:16:30.794118+0000","last_clean":"2023-10-16T14:16:30.794118+0000","last_became_active":"2023-10-16T14:16:29.820053+0000","last_became_peered":"2023-10-16T14:16:29.820053+0000","last_unstale":"2023-10-16T14:16:30.794118+0000","last_undegraded":"2023-10-16T14:16:30.794118+0000","last_fullsized":"2023-10-16T14:16:30.794118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825735+0000","last_change":"2023-10-16T14:16:29.819714+0000","last_active":"2023-10-16T14:16:31.825735+0000","last_peered":"2023-10-16T14:16:31.825735+0000","last_clean":"2023-10-16T14:16:31.825735+0000","last_became_active":"2023-10-16T14:16:29.819444+0000","last_became_peered":"2023-10-16T14:16:29.819444+0000","last_unstale":"2023-10-16T14:16:31.825735+0000","last_undegraded":"2023-10-16T14:16:31.825735+0000","last_fullsized":"2023-10-16T14:16:31.825735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826922+0000","last_change":"2023-10-16T14:16:29.817726+0000","last_active":"2023-10-16T14:16:31.826922+0000","last_peered":"2023-10-16T14:16:31.826922+0000","last_clean":"2023-10-16T14:16:31.826922+0000","last_became_active":"2023-10-16T14:16:29.813107+0000","last_became_peered":"2023-10-16T14:16:29.813107+0000","last_unstale":"2023-10-16T14:16:31.826922+0000","last_undegraded":"2023-10-16T14:16:31.826922+0000","last_fullsized":"2023-10-16T14:16:31.826922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325856+0000","last_change":"2023-10-16T14:16:29.825557+0000","last_active":"2023-10-16T14:16:32.325856+0000","last_peered":"2023-10-16T14:16:32.325856+0000","last_clean":"2023-10-16T14:16:32.325856+0000","last_became_active":"2023-10-16T14:16:29.825131+0000","last_became_peered":"2023-10-16T14:16:29.825131+0000","last_unstale":"2023-10-16T14:16:32.325856+0000","last_undegraded":"2023-10-16T14:16:32.325856+0000","last_fullsized":"2023-10-16T14:16:32.325856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798289+0000","last_change":"2023-10-16T14:16:29.813672+0000","last_active":"2023-10-16T14:16:30.798289+0000","last_peered":"2023-10-16T14:16:30.798289+0000","last_clean":"2023-10-16T14:16:30.798289+0000","last_became_active":"2023-10-16T14:16:29.813271+0000","last_became_peered":"2023-10-16T14:16:29.813271+0000","last_unstale":"2023-10-16T14:16:30.798289+0000","last_undegraded":"2023-10-16T14:16:30.798289+0000","last_fullsized":"2023-10-16T14:16:30.798289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961292+0000","last_change":"2023-10-16T14:16:29.821916+0000","last_active":"2023-10-16T14:16:30.961292+0000","last_peered":"2023-10-16T14:16:30.961292+0000","last_clean":"2023-10-16T14:16:30.961292+0000","last_became_active":"2023-10-16T14:16:29.821561+0000","last_became_peered":"2023-10-16T14:16:29.821561+0000","last_unstale":"2023-10-16T14:16:30.961292+0000","last_undegraded":"2023-10-16T14:16:30.961292+0000","last_fullsized":"2023-10-16T14:16:30.961292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326088+0000","last_change":"2023-10-16T14:16:29.809106+0000","last_active":"2023-10-16T14:16:32.326088+0000","last_peered":"2023-10-16T14:16:32.326088+0000","last_clean":"2023-10-16T14:16:32.326088+0000","last_became_active":"2023-10-16T14:16:29.808677+0000","last_became_peered":"2023-10-16T14:16:29.808677+0000","last_unstale":"2023-10-16T14:16:32.326088+0000","last_undegraded":"2023-10-16T14:16:32.326088+0000","last_fullsized":"2023-10-16T14:16:32.326088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:02.431928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.814365+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.814022+0000","last_became_peered":"2023-10-16T14:16:29.814022+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667249+0000","last_change":"2023-10-16T14:16:29.826061+0000","last_active":"2023-10-16T14:16:31.667249+0000","last_peered":"2023-10-16T14:16:31.667249+0000","last_clean":"2023-10-16T14:16:31.667249+0000","last_became_active":"2023-10-16T14:16:29.825662+0000","last_became_peered":"2023-10-16T14:16:29.825662+0000","last_unstale":"2023-10-16T14:16:31.667249+0000","last_undegraded":"2023-10-16T14:16:31.667249+0000","last_fullsized":"2023-10-16T14:16:31.667249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826173+0000","last_change":"2023-10-16T14:16:29.822279+0000","last_active":"2023-10-16T14:16:31.826173+0000","last_peered":"2023-10-16T14:16:31.826173+0000","last_clean":"2023-10-16T14:16:31.826173+0000","last_became_active":"2023-10-16T14:16:29.822022+0000","last_became_peered":"2023-10-16T14:16:29.822022+0000","last_unstale":"2023-10-16T14:16:31.826173+0000","last_undegraded":"2023-10-16T14:16:31.826173+0000","last_fullsized":"2023-10-16T14:16:31.826173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668113+0000","last_change":"2023-10-16T14:16:29.823044+0000","last_active":"2023-10-16T14:16:31.668113+0000","last_peered":"2023-10-16T14:16:31.668113+0000","last_clean":"2023-10-16T14:16:31.668113+0000","last_became_active":"2023-10-16T14:16:29.822705+0000","last_became_peered":"2023-10-16T14:16:29.822705+0000","last_unstale":"2023-10-16T14:16:31.668113+0000","last_undegraded":"2023-10-16T14:16:31.668113+0000","last_fullsized":"2023-10-16T14:16:31.668113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961559+0000","last_change":"2023-10-16T14:16:29.822259+0000","last_active":"2023-10-16T14:16:30.961559+0000","last_peered":"2023-10-16T14:16:30.961559+0000","last_clean":"2023-10-16T14:16:30.961559+0000","last_became_active":"2023-10-16T14:16:29.821719+0000","last_became_peered":"2023-10-16T14:16:29.821719+0000","last_unstale":"2023-10-16T14:16:30.961559+0000","last_undegraded":"2023-10-16T14:16:30.961559+0000","last_fullsized":"2023-10-16T14:16:30.961559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794706+0000","last_change":"2023-10-16T14:16:29.821655+0000","last_active":"2023-10-16T14:16:30.794706+0000","last_peered":"2023-10-16T14:16:30.794706+0000","last_clean":"2023-10-16T14:16:30.794706+0000","last_became_active":"2023-10-16T14:16:29.821357+0000","last_became_peered":"2023-10-16T14:16:29.821357+0000","last_unstale":"2023-10-16T14:16:30.794706+0000","last_undegraded":"2023-10-16T14:16:30.794706+0000","last_fullsized":"2023-10-16T14:16:30.794706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668435+0000","last_change":"2023-10-16T14:16:29.815493+0000","last_active":"2023-10-16T14:16:31.668435+0000","last_peered":"2023-10-16T14:16:31.668435+0000","last_clean":"2023-10-16T14:16:31.668435+0000","last_became_active":"2023-10-16T14:16:29.815203+0000","last_became_peered":"2023-10-16T14:16:29.815203+0000","last_unstale":"2023-10-16T14:16:31.668435+0000","last_undegraded":"2023-10-16T14:16:31.668435+0000","last_fullsized":"2023-10-16T14:16:31.668435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326367+0000","last_change":"2023-10-16T14:16:29.809001+0000","last_active":"2023-10-16T14:16:32.326367+0000","last_peered":"2023-10-16T14:16:32.326367+0000","last_clean":"2023-10-16T14:16:32.326367+0000","last_became_active":"2023-10-16T14:16:29.808169+0000","last_became_peered":"2023-10-16T14:16:29.808169+0000","last_unstale":"2023-10-16T14:16:32.326367+0000","last_undegraded":"2023-10-16T14:16:32.326367+0000","last_fullsized":"2023-10-16T14:16:32.326367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826340+0000","last_change":"2023-10-16T14:16:29.812984+0000","last_active":"2023-10-16T14:16:31.826340+0000","last_peered":"2023-10-16T14:16:31.826340+0000","last_clean":"2023-10-16T14:16:31.826340+0000","last_became_active":"2023-10-16T14:16:29.812569+0000","last_became_peered":"2023-10-16T14:16:29.812569+0000","last_unstale":"2023-10-16T14:16:31.826340+0000","last_undegraded":"2023-10-16T14:16:31.826340+0000","last_fullsized":"2023-10-16T14:16:31.826340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325543+0000","last_change":"2023-10-16T14:16:29.821793+0000","last_active":"2023-10-16T14:16:32.325543+0000","last_peered":"2023-10-16T14:16:32.325543+0000","last_clean":"2023-10-16T14:16:32.325543+0000","last_became_active":"2023-10-16T14:16:29.819170+0000","last_became_peered":"2023-10-16T14:16:29.819170+0000","last_unstale":"2023-10-16T14:16:32.325543+0000","last_undegraded":"2023-10-16T14:16:32.325543+0000","last_fullsized":"2023-10-16T14:16:32.325543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961200+0000","last_change":"2023-10-16T14:16:29.821091+0000","last_active":"2023-10-16T14:16:30.961200+0000","last_peered":"2023-10-16T14:16:30.961200+0000","last_clean":"2023-10-16T14:16:30.961200+0000","last_became_active":"2023-10-16T14:16:29.818555+0000","last_became_peered":"2023-10-16T14:16:29.818555+0000","last_unstale":"2023-10-16T14:16:30.961200+0000","last_undegraded":"2023-10-16T14:16:30.961200+0000","last_fullsized":"2023-10-16T14:16:30.961200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797715+0000","last_change":"2023-10-16T14:16:29.812942+0000","last_active":"2023-10-16T14:16:30.797715+0000","last_peered":"2023-10-16T14:16:30.797715+0000","last_clean":"2023-10-16T14:16:30.797715+0000","last_became_active":"2023-10-16T14:16:29.811742+0000","last_became_peered":"2023-10-16T14:16:29.811742+0000","last_unstale":"2023-10-16T14:16:30.797715+0000","last_undegraded":"2023-10-16T14:16:30.797715+0000","last_fullsized":"2023-10-16T14:16:30.797715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668347+0000","last_change":"2023-10-16T14:16:29.050890+0000","last_active":"2023-10-16T14:16:31.668347+0000","last_peered":"2023-10-16T14:16:31.668347+0000","last_clean":"2023-10-16T14:16:31.668347+0000","last_became_active":"2023-10-16T14:16:29.049249+0000","last_became_peered":"2023-10-16T14:16:29.049249+0000","last_unstale":"2023-10-16T14:16:31.668347+0000","last_undegraded":"2023-10-16T14:16:31.668347+0000","last_fullsized":"2023-10-16T14:16:31.668347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326412+0000","last_change":"2023-10-16T14:16:29.820123+0000","last_active":"2023-10-16T14:16:32.326412+0000","last_peered":"2023-10-16T14:16:32.326412+0000","last_clean":"2023-10-16T14:16:32.326412+0000","last_became_active":"2023-10-16T14:16:29.819857+0000","last_became_peered":"2023-10-16T14:16:29.819857+0000","last_unstale":"2023-10-16T14:16:32.326412+0000","last_undegraded":"2023-10-16T14:16:32.326412+0000","last_fullsized":"2023-10-16T14:16:32.326412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960647+0000","last_change":"2023-10-16T14:16:29.053825+0000","last_active":"2023-10-16T14:16:30.960647+0000","last_peered":"2023-10-16T14:16:30.960647+0000","last_clean":"2023-10-16T14:16:30.960647+0000","last_became_active":"2023-10-16T14:16:29.051055+0000","last_became_peered":"2023-10-16T14:16:29.051055+0000","last_unstale":"2023-10-16T14:16:30.960647+0000","last_undegraded":"2023-10-16T14:16:30.960647+0000","last_fullsized":"2023-10-16T14:16:30.960647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667594+0000","last_change":"2023-10-16T14:16:29.055594+0000","last_active":"2023-10-16T14:16:31.667594+0000","last_peered":"2023-10-16T14:16:31.667594+0000","last_clean":"2023-10-16T14:16:31.667594+0000","last_became_active":"2023-10-16T14:16:29.055260+0000","last_became_peered":"2023-10-16T14:16:29.055260+0000","last_unstale":"2023-10-16T14:16:31.667594+0000","last_undegraded":"2023-10-16T14:16:31.667594+0000","last_fullsized":"2023-10-16T14:16:31.667594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667384+0000","last_change":"2023-10-16T14:16:29.817463+0000","last_active":"2023-10-16T14:16:31.667384+0000","last_peered":"2023-10-16T14:16:31.667384+0000","last_clean":"2023-10-16T14:16:31.667384+0000","last_became_active":"2023-10-16T14:16:29.817155+0000","last_became_peered":"2023-10-16T14:16:29.817155+0000","last_unstale":"2023-10-16T14:16:31.667384+0000","last_undegraded":"2023-10-16T14:16:31.667384+0000","last_fullsized":"2023-10-16T14:16:31.667384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.047697+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.047399+0000","last_became_peered":"2023-10-16T14:16:29.047399+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794058+0000","last_change":"2023-10-16T14:16:29.811517+0000","last_active":"2023-10-16T14:16:30.794058+0000","last_peered":"2023-10-16T14:16:30.794058+0000","last_clean":"2023-10-16T14:16:30.794058+0000","last_became_active":"2023-10-16T14:16:29.811178+0000","last_became_peered":"2023-10-16T14:16:29.811178+0000","last_unstale":"2023-10-16T14:16:30.794058+0000","last_undegraded":"2023-10-16T14:16:30.794058+0000","last_fullsized":"2023-10-16T14:16:30.794058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961080+0000","last_change":"2023-10-16T14:16:29.061919+0000","last_active":"2023-10-16T14:16:30.961080+0000","last_peered":"2023-10-16T14:16:30.961080+0000","last_clean":"2023-10-16T14:16:30.961080+0000","last_became_active":"2023-10-16T14:16:29.061130+0000","last_became_peered":"2023-10-16T14:16:29.061130+0000","last_unstale":"2023-10-16T14:16:30.961080+0000","last_undegraded":"2023-10-16T14:16:30.961080+0000","last_fullsized":"2023-10-16T14:16:30.961080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826710+0000","last_change":"2023-10-16T14:16:29.817592+0000","last_active":"2023-10-16T14:16:31.826710+0000","last_peered":"2023-10-16T14:16:31.826710+0000","last_clean":"2023-10-16T14:16:31.826710+0000","last_became_active":"2023-10-16T14:16:29.817291+0000","last_became_peered":"2023-10-16T14:16:29.817291+0000","last_unstale":"2023-10-16T14:16:31.826710+0000","last_undegraded":"2023-10-16T14:16:31.826710+0000","last_fullsized":"2023-10-16T14:16:31.826710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826905+0000","last_change":"2023-10-16T14:16:29.059877+0000","last_active":"2023-10-16T14:16:31.826905+0000","last_peered":"2023-10-16T14:16:31.826905+0000","last_clean":"2023-10-16T14:16:31.826905+0000","last_became_active":"2023-10-16T14:16:29.059080+0000","last_became_peered":"2023-10-16T14:16:29.059080+0000","last_unstale":"2023-10-16T14:16:31.826905+0000","last_undegraded":"2023-10-16T14:16:31.826905+0000","last_fullsized":"2023-10-16T14:16:31.826905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:40.688481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326692+0000","last_change":"2023-10-16T14:16:29.827311+0000","last_active":"2023-10-16T14:16:32.326692+0000","last_peered":"2023-10-16T14:16:32.326692+0000","last_clean":"2023-10-16T14:16:32.326692+0000","last_became_active":"2023-10-16T14:16:29.827004+0000","last_became_peered":"2023-10-16T14:16:29.827004+0000","last_unstale":"2023-10-16T14:16:32.326692+0000","last_undegraded":"2023-10-16T14:16:32.326692+0000","last_fullsized":"2023-10-16T14:16:32.326692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794725+0000","last_change":"2023-10-16T14:16:29.059819+0000","last_active":"2023-10-16T14:16:30.794725+0000","last_peered":"2023-10-16T14:16:30.794725+0000","last_clean":"2023-10-16T14:16:30.794725+0000","last_became_active":"2023-10-16T14:16:29.059474+0000","last_became_peered":"2023-10-16T14:16:29.059474+0000","last_unstale":"2023-10-16T14:16:30.794725+0000","last_undegraded":"2023-10-16T14:16:30.794725+0000","last_fullsized":"2023-10-16T14:16:30.794725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797712+0000","last_change":"2023-10-16T14:16:29.813588+0000","last_active":"2023-10-16T14:16:30.797712+0000","last_peered":"2023-10-16T14:16:30.797712+0000","last_clean":"2023-10-16T14:16:30.797712+0000","last_became_active":"2023-10-16T14:16:29.813342+0000","last_became_peered":"2023-10-16T14:16:29.813342+0000","last_unstale":"2023-10-16T14:16:30.797712+0000","last_undegraded":"2023-10-16T14:16:30.797712+0000","last_fullsized":"2023-10-16T14:16:30.797712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:36.779577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961265+0000","last_change":"2023-10-16T14:16:29.057707+0000","last_active":"2023-10-16T14:16:30.961265+0000","last_peered":"2023-10-16T14:16:30.961265+0000","last_clean":"2023-10-16T14:16:30.961265+0000","last_became_active":"2023-10-16T14:16:29.054590+0000","last_became_peered":"2023-10-16T14:16:29.054590+0000","last_unstale":"2023-10-16T14:16:30.961265+0000","last_undegraded":"2023-10-16T14:16:30.961265+0000","last_fullsized":"2023-10-16T14:16:30.961265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797570+0000","last_change":"2023-10-16T14:16:29.810243+0000","last_active":"2023-10-16T14:16:30.797570+0000","last_peered":"2023-10-16T14:16:30.797570+0000","last_clean":"2023-10-16T14:16:30.797570+0000","last_became_active":"2023-10-16T14:16:29.809877+0000","last_became_peered":"2023-10-16T14:16:29.809877+0000","last_unstale":"2023-10-16T14:16:30.797570+0000","last_undegraded":"2023-10-16T14:16:30.797570+0000","last_fullsized":"2023-10-16T14:16:30.797570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326441+0000","last_change":"2023-10-16T14:16:29.046872+0000","last_active":"2023-10-16T14:16:32.326441+0000","last_peered":"2023-10-16T14:16:32.326441+0000","last_clean":"2023-10-16T14:16:32.326441+0000","last_became_active":"2023-10-16T14:16:29.046411+0000","last_became_peered":"2023-10-16T14:16:29.046411+0000","last_unstale":"2023-10-16T14:16:32.326441+0000","last_undegraded":"2023-10-16T14:16:32.326441+0000","last_fullsized":"2023-10-16T14:16:32.326441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:48.638693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326040+0000","last_change":"2023-10-16T14:16:29.810247+0000","last_active":"2023-10-16T14:16:32.326040+0000","last_peered":"2023-10-16T14:16:32.326040+0000","last_clean":"2023-10-16T14:16:32.326040+0000","last_became_active":"2023-10-16T14:16:29.809830+0000","last_became_peered":"2023-10-16T14:16:29.809830+0000","last_unstale":"2023-10-16T14:16:32.326040+0000","last_undegraded":"2023-10-16T14:16:32.326040+0000","last_fullsized":"2023-10-16T14:16:32.326040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:26.623468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826189+0000","last_change":"2023-10-16T14:16:29.049519+0000","last_active":"2023-10-16T14:16:31.826189+0000","last_peered":"2023-10-16T14:16:31.826189+0000","last_clean":"2023-10-16T14:16:31.826189+0000","last_became_active":"2023-10-16T14:16:29.049203+0000","last_became_peered":"2023-10-16T14:16:29.049203+0000","last_unstale":"2023-10-16T14:16:31.826189+0000","last_undegraded":"2023-10-16T14:16:31.826189+0000","last_fullsized":"2023-10-16T14:16:31.826189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:38.962308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798901+0000","last_change":"2023-10-16T14:16:29.824625+0000","last_active":"2023-10-16T14:16:30.798901+0000","last_peered":"2023-10-16T14:16:30.798901+0000","last_clean":"2023-10-16T14:16:30.798901+0000","last_became_active":"2023-10-16T14:16:29.818988+0000","last_became_peered":"2023-10-16T14:16:29.818988+0000","last_unstale":"2023-10-16T14:16:30.798901+0000","last_undegraded":"2023-10-16T14:16:30.798901+0000","last_fullsized":"2023-10-16T14:16:30.798901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960552+0000","last_change":"2023-10-16T14:16:29.054389+0000","last_active":"2023-10-16T14:16:30.960552+0000","last_peered":"2023-10-16T14:16:30.960552+0000","last_clean":"2023-10-16T14:16:30.960552+0000","last_became_active":"2023-10-16T14:16:29.053542+0000","last_became_peered":"2023-10-16T14:16:29.053542+0000","last_unstale":"2023-10-16T14:16:30.960552+0000","last_undegraded":"2023-10-16T14:16:30.960552+0000","last_fullsized":"2023-10-16T14:16:30.960552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960683+0000","last_change":"2023-10-16T14:16:29.822533+0000","last_active":"2023-10-16T14:16:30.960683+0000","last_peered":"2023-10-16T14:16:30.960683+0000","last_clean":"2023-10-16T14:16:30.960683+0000","last_became_active":"2023-10-16T14:16:29.822130+0000","last_became_peered":"2023-10-16T14:16:29.822130+0000","last_unstale":"2023-10-16T14:16:30.960683+0000","last_undegraded":"2023-10-16T14:16:30.960683+0000","last_fullsized":"2023-10-16T14:16:30.960683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325417+0000","last_change":"2023-10-16T14:16:29.044471+0000","last_active":"2023-10-16T14:16:32.325417+0000","last_peered":"2023-10-16T14:16:32.325417+0000","last_clean":"2023-10-16T14:16:32.325417+0000","last_became_active":"2023-10-16T14:16:29.042866+0000","last_became_peered":"2023-10-16T14:16:29.042866+0000","last_unstale":"2023-10-16T14:16:32.325417+0000","last_undegraded":"2023-10-16T14:16:32.325417+0000","last_fullsized":"2023-10-16T14:16:32.325417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:01.067871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799086+0000","last_change":"2023-10-16T14:16:29.826425+0000","last_active":"2023-10-16T14:16:30.799086+0000","last_peered":"2023-10-16T14:16:30.799086+0000","last_clean":"2023-10-16T14:16:30.799086+0000","last_became_active":"2023-10-16T14:16:29.820157+0000","last_became_peered":"2023-10-16T14:16:29.820157+0000","last_unstale":"2023-10-16T14:16:30.799086+0000","last_undegraded":"2023-10-16T14:16:30.799086+0000","last_fullsized":"2023-10-16T14:16:30.799086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797731+0000","last_change":"2023-10-16T14:16:29.049608+0000","last_active":"2023-10-16T14:16:30.797731+0000","last_peered":"2023-10-16T14:16:30.797731+0000","last_clean":"2023-10-16T14:16:30.797731+0000","last_became_active":"2023-10-16T14:16:29.049125+0000","last_became_peered":"2023-10-16T14:16:29.049125+0000","last_unstale":"2023-10-16T14:16:30.797731+0000","last_undegraded":"2023-10-16T14:16:30.797731+0000","last_fullsized":"2023-10-16T14:16:30.797731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798834+0000","last_change":"2023-10-16T14:16:29.809718+0000","last_active":"2023-10-16T14:16:30.798834+0000","last_peered":"2023-10-16T14:16:30.798834+0000","last_clean":"2023-10-16T14:16:30.798834+0000","last_became_active":"2023-10-16T14:16:29.809294+0000","last_became_peered":"2023-10-16T14:16:29.809294+0000","last_unstale":"2023-10-16T14:16:30.798834+0000","last_undegraded":"2023-10-16T14:16:30.798834+0000","last_fullsized":"2023-10-16T14:16:30.798834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325704+0000","last_change":"2023-10-16T14:16:29.034974+0000","last_active":"2023-10-16T14:16:32.325704+0000","last_peered":"2023-10-16T14:16:32.325704+0000","last_clean":"2023-10-16T14:16:32.325704+0000","last_became_active":"2023-10-16T14:16:29.034651+0000","last_became_peered":"2023-10-16T14:16:29.034651+0000","last_unstale":"2023-10-16T14:16:32.325704+0000","last_undegraded":"2023-10-16T14:16:32.325704+0000","last_fullsized":"2023-10-16T14:16:32.325704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:04.878575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799415+0000","last_change":"2023-10-16T14:16:29.829055+0000","last_active":"2023-10-16T14:16:30.799415+0000","last_peered":"2023-10-16T14:16:30.799415+0000","last_clean":"2023-10-16T14:16:30.799415+0000","last_became_active":"2023-10-16T14:16:29.820785+0000","last_became_peered":"2023-10-16T14:16:29.820785+0000","last_unstale":"2023-10-16T14:16:30.799415+0000","last_undegraded":"2023-10-16T14:16:30.799415+0000","last_fullsized":"2023-10-16T14:16:30.799415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794821+0000","last_change":"2023-10-16T14:16:29.048896+0000","last_active":"2023-10-16T14:16:30.794821+0000","last_peered":"2023-10-16T14:16:30.794821+0000","last_clean":"2023-10-16T14:16:30.794821+0000","last_became_active":"2023-10-16T14:16:29.048513+0000","last_became_peered":"2023-10-16T14:16:29.048513+0000","last_unstale":"2023-10-16T14:16:30.794821+0000","last_undegraded":"2023-10-16T14:16:30.794821+0000","last_fullsized":"2023-10-16T14:16:30.794821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827198+0000","last_change":"2023-10-16T14:16:29.814236+0000","last_active":"2023-10-16T14:16:31.827198+0000","last_peered":"2023-10-16T14:16:31.827198+0000","last_clean":"2023-10-16T14:16:31.827198+0000","last_became_active":"2023-10-16T14:16:29.813943+0000","last_became_peered":"2023-10-16T14:16:29.813943+0000","last_unstale":"2023-10-16T14:16:31.827198+0000","last_undegraded":"2023-10-16T14:16:31.827198+0000","last_fullsized":"2023-10-16T14:16:31.827198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325524+0000","last_change":"2023-10-16T14:16:29.052741+0000","last_active":"2023-10-16T14:16:32.325524+0000","last_peered":"2023-10-16T14:16:32.325524+0000","last_clean":"2023-10-16T14:16:32.325524+0000","last_became_active":"2023-10-16T14:16:29.052376+0000","last_became_peered":"2023-10-16T14:16:29.052376+0000","last_unstale":"2023-10-16T14:16:32.325524+0000","last_undegraded":"2023-10-16T14:16:32.325524+0000","last_fullsized":"2023-10-16T14:16:32.325524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:40.014521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797919+0000","last_change":"2023-10-16T14:16:29.828744+0000","last_active":"2023-10-16T14:16:30.797919+0000","last_peered":"2023-10-16T14:16:30.797919+0000","last_clean":"2023-10-16T14:16:30.797919+0000","last_became_active":"2023-10-16T14:16:29.823064+0000","last_became_peered":"2023-10-16T14:16:29.823064+0000","last_unstale":"2023-10-16T14:16:30.797919+0000","last_undegraded":"2023-10-16T14:16:30.797919+0000","last_fullsized":"2023-10-16T14:16:30.797919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-16T14:16:29.790600+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:16:29.790600+0000","last_peered":"2023-10-16T14:16:29.790600+0000","last_clean":"2023-10-16T14:16:29.790600+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:16:29.790600+0000","last_undegraded":"2023-10-16T14:16:29.790600+0000","last_fullsized":"2023-10-16T14:16:29.790600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825962+0000","last_change":"2023-10-16T14:16:29.051990+0000","last_active":"2023-10-16T14:16:31.825962+0000","last_peered":"2023-10-16T14:16:31.825962+0000","last_clean":"2023-10-16T14:16:31.825962+0000","last_became_active":"2023-10-16T14:16:29.051723+0000","last_became_peered":"2023-10-16T14:16:29.051723+0000","last_unstale":"2023-10-16T14:16:31.825962+0000","last_undegraded":"2023-10-16T14:16:31.825962+0000","last_fullsized":"2023-10-16T14:16:31.825962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667924+0000","last_change":"2023-10-16T14:16:29.823986+0000","last_active":"2023-10-16T14:16:31.667924+0000","last_peered":"2023-10-16T14:16:31.667924+0000","last_clean":"2023-10-16T14:16:31.667924+0000","last_became_active":"2023-10-16T14:16:29.823102+0000","last_became_peered":"2023-10-16T14:16:29.823102+0000","last_unstale":"2023-10-16T14:16:31.667924+0000","last_undegraded":"2023-10-16T14:16:31.667924+0000","last_fullsized":"2023-10-16T14:16:31.667924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795970+0000","last_change":"2023-10-16T14:16:29.058003+0000","last_active":"2023-10-16T14:16:30.795970+0000","last_peered":"2023-10-16T14:16:30.795970+0000","last_clean":"2023-10-16T14:16:30.795970+0000","last_became_active":"2023-10-16T14:16:29.057686+0000","last_became_peered":"2023-10-16T14:16:29.057686+0000","last_unstale":"2023-10-16T14:16:30.795970+0000","last_undegraded":"2023-10-16T14:16:30.795970+0000","last_fullsized":"2023-10-16T14:16:30.795970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325936+0000","last_change":"2023-10-16T14:16:29.812594+0000","last_active":"2023-10-16T14:16:32.325936+0000","last_peered":"2023-10-16T14:16:32.325936+0000","last_clean":"2023-10-16T14:16:32.325936+0000","last_became_active":"2023-10-16T14:16:29.812256+0000","last_became_peered":"2023-10-16T14:16:29.812256+0000","last_unstale":"2023-10-16T14:16:32.325936+0000","last_undegraded":"2023-10-16T14:16:32.325936+0000","last_fullsized":"2023-10-16T14:16:32.325936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:48.787869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325633+0000","last_change":"2023-10-16T14:16:29.042443+0000","last_active":"2023-10-16T14:16:32.325633+0000","last_peered":"2023-10-16T14:16:32.325633+0000","last_clean":"2023-10-16T14:16:32.325633+0000","last_became_active":"2023-10-16T14:16:29.042189+0000","last_became_peered":"2023-10-16T14:16:29.042189+0000","last_unstale":"2023-10-16T14:16:32.325633+0000","last_undegraded":"2023-10-16T14:16:32.325633+0000","last_fullsized":"2023-10-16T14:16:32.325633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794175+0000","last_change":"2023-10-16T14:16:29.816645+0000","last_active":"2023-10-16T14:16:30.794175+0000","last_peered":"2023-10-16T14:16:30.794175+0000","last_clean":"2023-10-16T14:16:30.794175+0000","last_became_active":"2023-10-16T14:16:29.814792+0000","last_became_peered":"2023-10-16T14:16:29.814792+0000","last_unstale":"2023-10-16T14:16:30.794175+0000","last_undegraded":"2023-10-16T14:16:30.794175+0000","last_fullsized":"2023-10-16T14:16:30.794175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826471+0000","last_change":"2023-10-16T14:16:29.055439+0000","last_active":"2023-10-16T14:16:31.826471+0000","last_peered":"2023-10-16T14:16:31.826471+0000","last_clean":"2023-10-16T14:16:31.826471+0000","last_became_active":"2023-10-16T14:16:29.054422+0000","last_became_peered":"2023-10-16T14:16:29.054422+0000","last_unstale":"2023-10-16T14:16:31.826471+0000","last_undegraded":"2023-10-16T14:16:31.826471+0000","last_fullsized":"2023-10-16T14:16:31.826471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797290+0000","last_change":"2023-10-16T14:16:29.826562+0000","last_active":"2023-10-16T14:16:30.797290+0000","last_peered":"2023-10-16T14:16:30.797290+0000","last_clean":"2023-10-16T14:16:30.797290+0000","last_became_active":"2023-10-16T14:16:29.825184+0000","last_became_peered":"2023-10-16T14:16:29.825184+0000","last_unstale":"2023-10-16T14:16:30.797290+0000","last_undegraded":"2023-10-16T14:16:30.797290+0000","last_fullsized":"2023-10-16T14:16:30.797290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:49.651724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794587+0000","last_change":"2023-10-16T14:16:29.048497+0000","last_active":"2023-10-16T14:16:30.794587+0000","last_peered":"2023-10-16T14:16:30.794587+0000","last_clean":"2023-10-16T14:16:30.794587+0000","last_became_active":"2023-10-16T14:16:29.048194+0000","last_became_peered":"2023-10-16T14:16:29.048194+0000","last_unstale":"2023-10-16T14:16:30.794587+0000","last_undegraded":"2023-10-16T14:16:30.794587+0000","last_fullsized":"2023-10-16T14:16:30.794587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826954+0000","last_change":"2023-10-16T14:16:29.819205+0000","last_active":"2023-10-16T14:16:31.826954+0000","last_peered":"2023-10-16T14:16:31.826954+0000","last_clean":"2023-10-16T14:16:31.826954+0000","last_became_active":"2023-10-16T14:16:29.818921+0000","last_became_peered":"2023-10-16T14:16:29.818921+0000","last_unstale":"2023-10-16T14:16:31.826954+0000","last_undegraded":"2023-10-16T14:16:31.826954+0000","last_fullsized":"2023-10-16T14:16:31.826954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798586+0000","last_change":"2023-10-16T14:16:29.053513+0000","last_active":"2023-10-16T14:16:30.798586+0000","last_peered":"2023-10-16T14:16:30.798586+0000","last_clean":"2023-10-16T14:16:30.798586+0000","last_became_active":"2023-10-16T14:16:29.053178+0000","last_became_peered":"2023-10-16T14:16:29.053178+0000","last_unstale":"2023-10-16T14:16:30.798586+0000","last_undegraded":"2023-10-16T14:16:30.798586+0000","last_fullsized":"2023-10-16T14:16:30.798586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794789+0000","last_change":"2023-10-16T14:16:29.817442+0000","last_active":"2023-10-16T14:16:30.794789+0000","last_peered":"2023-10-16T14:16:30.794789+0000","last_clean":"2023-10-16T14:16:30.794789+0000","last_became_active":"2023-10-16T14:16:29.814899+0000","last_became_peered":"2023-10-16T14:16:29.814899+0000","last_unstale":"2023-10-16T14:16:30.794789+0000","last_undegraded":"2023-10-16T14:16:30.794789+0000","last_fullsized":"2023-10-16T14:16:30.794789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794674+0000","last_change":"2023-10-16T14:16:29.059226+0000","last_active":"2023-10-16T14:16:30.794674+0000","last_peered":"2023-10-16T14:16:30.794674+0000","last_clean":"2023-10-16T14:16:30.794674+0000","last_became_active":"2023-10-16T14:16:29.058896+0000","last_became_peered":"2023-10-16T14:16:29.058896+0000","last_unstale":"2023-10-16T14:16:30.794674+0000","last_undegraded":"2023-10-16T14:16:30.794674+0000","last_fullsized":"2023-10-16T14:16:30.794674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668348+0000","last_change":"2023-10-16T14:16:29.807423+0000","last_active":"2023-10-16T14:16:31.668348+0000","last_peered":"2023-10-16T14:16:31.668348+0000","last_clean":"2023-10-16T14:16:31.668348+0000","last_became_active":"2023-10-16T14:16:29.806278+0000","last_became_peered":"2023-10-16T14:16:29.806278+0000","last_unstale":"2023-10-16T14:16:31.668348+0000","last_undegraded":"2023-10-16T14:16:31.668348+0000","last_fullsized":"2023-10-16T14:16:31.668348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961275+0000","last_change":"2023-10-16T14:16:29.057876+0000","last_active":"2023-10-16T14:16:30.961275+0000","last_peered":"2023-10-16T14:16:30.961275+0000","last_clean":"2023-10-16T14:16:30.961275+0000","last_became_active":"2023-10-16T14:16:29.053426+0000","last_became_peered":"2023-10-16T14:16:29.053426+0000","last_unstale":"2023-10-16T14:16:30.961275+0000","last_undegraded":"2023-10-16T14:16:30.961275+0000","last_fullsized":"2023-10-16T14:16:30.961275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:01.418897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326675+0000","last_change":"2023-10-16T14:16:29.816014+0000","last_active":"2023-10-16T14:16:32.326675+0000","last_peered":"2023-10-16T14:16:32.326675+0000","last_clean":"2023-10-16T14:16:32.326675+0000","last_became_active":"2023-10-16T14:16:29.815725+0000","last_became_peered":"2023-10-16T14:16:29.815725+0000","last_unstale":"2023-10-16T14:16:32.326675+0000","last_undegraded":"2023-10-16T14:16:32.326675+0000","last_fullsized":"2023-10-16T14:16:32.326675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795715+0000","last_change":"2023-10-16T14:16:29.050583+0000","last_active":"2023-10-16T14:16:30.795715+0000","last_peered":"2023-10-16T14:16:30.795715+0000","last_clean":"2023-10-16T14:16:30.795715+0000","last_became_active":"2023-10-16T14:16:29.050281+0000","last_became_peered":"2023-10-16T14:16:29.050281+0000","last_unstale":"2023-10-16T14:16:30.795715+0000","last_undegraded":"2023-10-16T14:16:30.795715+0000","last_fullsized":"2023-10-16T14:16:30.795715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:18.882395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796790+0000","last_change":"2023-10-16T14:16:29.808419+0000","last_active":"2023-10-16T14:16:30.796790+0000","last_peered":"2023-10-16T14:16:30.796790+0000","last_clean":"2023-10-16T14:16:30.796790+0000","last_became_active":"2023-10-16T14:16:29.808038+0000","last_became_peered":"2023-10-16T14:16:29.808038+0000","last_unstale":"2023-10-16T14:16:30.796790+0000","last_undegraded":"2023-10-16T14:16:30.796790+0000","last_fullsized":"2023-10-16T14:16:30.796790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:05.891605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794294+0000","last_change":"2023-10-16T14:16:29.049120+0000","last_active":"2023-10-16T14:16:30.794294+0000","last_peered":"2023-10-16T14:16:30.794294+0000","last_clean":"2023-10-16T14:16:30.794294+0000","last_became_active":"2023-10-16T14:16:29.048782+0000","last_became_peered":"2023-10-16T14:16:29.048782+0000","last_unstale":"2023-10-16T14:16:30.794294+0000","last_undegraded":"2023-10-16T14:16:30.794294+0000","last_fullsized":"2023-10-16T14:16:30.794294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667807+0000","last_change":"2023-10-16T14:16:29.820694+0000","last_active":"2023-10-16T14:16:31.667807+0000","last_peered":"2023-10-16T14:16:31.667807+0000","last_clean":"2023-10-16T14:16:31.667807+0000","last_became_active":"2023-10-16T14:16:29.820341+0000","last_became_peered":"2023-10-16T14:16:29.820341+0000","last_unstale":"2023-10-16T14:16:31.667807+0000","last_undegraded":"2023-10-16T14:16:31.667807+0000","last_fullsized":"2023-10-16T14:16:31.667807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794452+0000","last_change":"2023-10-16T14:16:29.043057+0000","last_active":"2023-10-16T14:16:30.794452+0000","last_peered":"2023-10-16T14:16:30.794452+0000","last_clean":"2023-10-16T14:16:30.794452+0000","last_became_active":"2023-10-16T14:16:29.042783+0000","last_became_peered":"2023-10-16T14:16:29.042783+0000","last_unstale":"2023-10-16T14:16:30.794452+0000","last_undegraded":"2023-10-16T14:16:30.794452+0000","last_fullsized":"2023-10-16T14:16:30.794452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668050+0000","last_change":"2023-10-16T14:16:29.816920+0000","last_active":"2023-10-16T14:16:31.668050+0000","last_peered":"2023-10-16T14:16:31.668050+0000","last_clean":"2023-10-16T14:16:31.668050+0000","last_became_active":"2023-10-16T14:16:29.816639+0000","last_became_peered":"2023-10-16T14:16:29.816639+0000","last_unstale":"2023-10-16T14:16:31.668050+0000","last_undegraded":"2023-10-16T14:16:31.668050+0000","last_fullsized":"2023-10-16T14:16:31.668050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798966+0000","last_change":"2023-10-16T14:16:29.056773+0000","last_active":"2023-10-16T14:16:30.798966+0000","last_peered":"2023-10-16T14:16:30.798966+0000","last_clean":"2023-10-16T14:16:30.798966+0000","last_became_active":"2023-10-16T14:16:29.056477+0000","last_became_peered":"2023-10-16T14:16:29.056477+0000","last_unstale":"2023-10-16T14:16:30.798966+0000","last_undegraded":"2023-10-16T14:16:30.798966+0000","last_fullsized":"2023-10-16T14:16:30.798966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:17.696119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960869+0000","last_change":"2023-10-16T14:16:29.821383+0000","last_active":"2023-10-16T14:16:30.960869+0000","last_peered":"2023-10-16T14:16:30.960869+0000","last_clean":"2023-10-16T14:16:30.960869+0000","last_became_active":"2023-10-16T14:16:29.821018+0000","last_became_peered":"2023-10-16T14:16:29.821018+0000","last_unstale":"2023-10-16T14:16:30.960869+0000","last_undegraded":"2023-10-16T14:16:30.960869+0000","last_fullsized":"2023-10-16T14:16:30.960869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826784+0000","last_change":"2023-10-16T14:16:29.829570+0000","last_active":"2023-10-16T14:16:31.826784+0000","last_peered":"2023-10-16T14:16:31.826784+0000","last_clean":"2023-10-16T14:16:31.826784+0000","last_became_active":"2023-10-16T14:16:29.828707+0000","last_became_peered":"2023-10-16T14:16:29.828707+0000","last_unstale":"2023-10-16T14:16:31.826784+0000","last_undegraded":"2023-10-16T14:16:31.826784+0000","last_fullsized":"2023-10-16T14:16:31.826784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798241+0000","last_change":"2023-10-16T14:16:29.063205+0000","last_active":"2023-10-16T14:16:30.798241+0000","last_peered":"2023-10-16T14:16:30.798241+0000","last_clean":"2023-10-16T14:16:30.798241+0000","last_became_active":"2023-10-16T14:16:29.062657+0000","last_became_peered":"2023-10-16T14:16:29.062657+0000","last_unstale":"2023-10-16T14:16:30.798241+0000","last_undegraded":"2023-10-16T14:16:30.798241+0000","last_fullsized":"2023-10-16T14:16:30.798241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:43.103258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826847+0000","last_change":"2023-10-16T14:16:29.830412+0000","last_active":"2023-10-16T14:16:31.826847+0000","last_peered":"2023-10-16T14:16:31.826847+0000","last_clean":"2023-10-16T14:16:31.826847+0000","last_became_active":"2023-10-16T14:16:29.825817+0000","last_became_peered":"2023-10-16T14:16:29.825817+0000","last_unstale":"2023-10-16T14:16:31.826847+0000","last_undegraded":"2023-10-16T14:16:31.826847+0000","last_fullsized":"2023-10-16T14:16:31.826847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798099+0000","last_change":"2023-10-16T14:16:29.061570+0000","last_active":"2023-10-16T14:16:30.798099+0000","last_peered":"2023-10-16T14:16:30.798099+0000","last_clean":"2023-10-16T14:16:30.798099+0000","last_became_active":"2023-10-16T14:16:29.061024+0000","last_became_peered":"2023-10-16T14:16:29.061024+0000","last_unstale":"2023-10-16T14:16:30.798099+0000","last_undegraded":"2023-10-16T14:16:30.798099+0000","last_fullsized":"2023-10-16T14:16:30.798099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326481+0000","last_change":"2023-10-16T14:16:29.827404+0000","last_active":"2023-10-16T14:16:32.326481+0000","last_peered":"2023-10-16T14:16:32.326481+0000","last_clean":"2023-10-16T14:16:32.326481+0000","last_became_active":"2023-10-16T14:16:29.826687+0000","last_became_peered":"2023-10-16T14:16:29.826687+0000","last_unstale":"2023-10-16T14:16:32.326481+0000","last_undegraded":"2023-10-16T14:16:32.326481+0000","last_fullsized":"2023-10-16T14:16:32.326481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668592+0000","last_change":"2023-10-16T14:16:29.050871+0000","last_active":"2023-10-16T14:16:31.668592+0000","last_peered":"2023-10-16T14:16:31.668592+0000","last_clean":"2023-10-16T14:16:31.668592+0000","last_became_active":"2023-10-16T14:16:29.050025+0000","last_became_peered":"2023-10-16T14:16:29.050025+0000","last_unstale":"2023-10-16T14:16:31.668592+0000","last_undegraded":"2023-10-16T14:16:31.668592+0000","last_fullsized":"2023-10-16T14:16:31.668592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668230+0000","last_change":"2023-10-16T14:16:29.825092+0000","last_active":"2023-10-16T14:16:31.668230+0000","last_peered":"2023-10-16T14:16:31.668230+0000","last_clean":"2023-10-16T14:16:31.668230+0000","last_became_active":"2023-10-16T14:16:29.824815+0000","last_became_peered":"2023-10-16T14:16:29.824815+0000","last_unstale":"2023-10-16T14:16:31.668230+0000","last_undegraded":"2023-10-16T14:16:31.668230+0000","last_fullsized":"2023-10-16T14:16:31.668230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326685+0000","last_change":"2023-10-16T14:16:29.053310+0000","last_active":"2023-10-16T14:16:32.326685+0000","last_peered":"2023-10-16T14:16:32.326685+0000","last_clean":"2023-10-16T14:16:32.326685+0000","last_became_active":"2023-10-16T14:16:29.052871+0000","last_became_peered":"2023-10-16T14:16:29.052871+0000","last_unstale":"2023-10-16T14:16:32.326685+0000","last_undegraded":"2023-10-16T14:16:32.326685+0000","last_fullsized":"2023-10-16T14:16:32.326685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9550000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4529999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3159999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1760000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.484}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6030000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2519999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:16:34.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:16:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.792+0000 7fd1b93e9640 1 Processor -- start 2023-10-16T14:16:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.792+0000 7fd1b93e9640 1 -- start start 2023-10-16T14:16:34.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b93e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1b40cd0c0 0x7fd1b41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b93e9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1b40b0410 0x7fd1b40b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b93e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b415e3e0 con 0x7fd1b415d3f0 2023-10-16T14:16:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b93e9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b40b0de0 con 0x7fd1b40b0410 2023-10-16T14:16:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b93e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b40cd4a0 con 0x7fd1b40cd0c0 2023-10-16T14:16:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1b40b0410 0x7fd1b40b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1b415d3f0 0x7fd1b415d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46300/0 (socket says 172.21.15.29:46300) 2023-10-16T14:16:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.794+0000 7fd1b37fe640 1 -- 172.21.15.29:0/2311459588 learned_addr learned my addr 172.21.15.29:0/2311459588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.795+0000 7fd1b2ffd640 1 --2- 172.21.15.29:0/2311459588 >> v2:172.21.15.170:3300/0 conn(0x7fd1b40cd0c0 0x7fd1b41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.795+0000 7fd1b27fc640 1 -- 172.21.15.29:0/2311459588 >> v2:172.21.15.170:3300/0 conn(0x7fd1b40cd0c0 msgr2=0x7fd1b41615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.795+0000 7fd1b27fc640 1 --2- 172.21.15.29:0/2311459588 >> v2:172.21.15.170:3300/0 conn(0x7fd1b40cd0c0 0x7fd1b41615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.795+0000 7fd1b27fc640 1 -- 172.21.15.29:0/2311459588 >> v1:172.21.15.29:6789/0 conn(0x7fd1b415d3f0 legacy=0x7fd1b415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.795+0000 7fd1b27fc640 1 -- 172.21.15.29:0/2311459588 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b415e100 con 0x7fd1b40b0410 2023-10-16T14:16:34.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.796+0000 7fd1b27fc640 1 --2- 172.21.15.29:0/2311459588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b40b0410 0x7fd1b40b08a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cc4e0 tx=0x7fd1a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d7d31c896d5725 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:34.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.796+0000 7fd1b1ffb640 1 -- 172.21.15.29:0/2311459588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a802e070 con 0x7fd1b40b0410 2023-10-16T14:16:34.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.796+0000 7fd1b1ffb640 1 -- 172.21.15.29:0/2311459588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a80315a0 con 0x7fd1b40b0410 2023-10-16T14:16:34.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.796+0000 7fd1b1ffb640 1 -- 172.21.15.29:0/2311459588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8031930 con 0x7fd1b40b0410 2023-10-16T14:16:34.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 -- 172.21.15.29:0/2311459588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b40b0410 msgr2=0x7fd1b40b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/2311459588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b40b0410 0x7fd1b40b08a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cc4e0 tx=0x7fd1a802b840 comp rx=0 tx=0).stop 2023-10-16T14:16:34.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 -- 172.21.15.29:0/2311459588 shutdown_connections 2023-10-16T14:16:34.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/2311459588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b40b0410 0x7fd1b40b08a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/2311459588 >> v2:172.21.15.170:3300/0 conn(0x7fd1b40cd0c0 0x7fd1b41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 -- 172.21.15.29:0/2311459588 >> 172.21.15.29:0/2311459588 conn(0x7fd1b40cb410 msgr2=0x7fd1b40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:34.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 -- 172.21.15.29:0/2311459588 shutdown_connections 2023-10-16T14:16:34.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.797+0000 7fd1b93e9640 1 -- 172.21.15.29:0/2311459588 wait complete. 2023-10-16T14:16:34.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.799+0000 7fd1b93e9640 1 Processor -- start 2023-10-16T14:16:34.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.799+0000 7fd1b93e9640 1 -- start start 2023-10-16T14:16:34.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b93e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1b40b0410 0x7fd1b415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b93e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1b40cd0c0 0x7fd1b41527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b93e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 0x7fd1b40ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b93e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40c6720 con 0x7fd1b415d3f0 2023-10-16T14:16:34.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b93e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b415edc0 con 0x7fd1b40b0410 2023-10-16T14:16:34.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b93e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b415f0a0 con 0x7fd1b40cd0c0 2023-10-16T14:16:34.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 0x7fd1b40ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 0x7fd1b40ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50634/0 (socket says 172.21.15.29:50634) 2023-10-16T14:16:34.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.800+0000 7fd1b37fe640 1 -- 172.21.15.29:0/473151030 learned_addr learned my addr 172.21.15.29:0/473151030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:34.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b2ffd640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1b40b0410 0x7fd1b415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b27fc640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1b40cd0c0 0x7fd1b41527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b37fe640 1 -- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1b40cd0c0 msgr2=0x7fd1b41527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b37fe640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1b40cd0c0 0x7fd1b41527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b37fe640 1 -- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1b40b0410 msgr2=0x7fd1b415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:34.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b37fe640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1b40b0410 0x7fd1b415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:34.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b37fe640 1 -- 172.21.15.29:0/473151030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8009400 con 0x7fd1b415d3f0 2023-10-16T14:16:34.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b2ffd640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1b40b0410 0x7fd1b415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:34.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b37fe640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 0x7fd1b40ae3d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00cbc0 tx=0x7fd1ac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:34.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac00ad40 con 0x7fd1b415d3f0 2023-10-16T14:16:34.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac012400 con 0x7fd1b415d3f0 2023-10-16T14:16:34.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac00aea0 con 0x7fd1b415d3f0 2023-10-16T14:16:34.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b415f380 con 0x7fd1b415d3f0 2023-10-16T14:16:34.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.801+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b415ead0 con 0x7fd1b415d3f0 2023-10-16T14:16:34.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.803+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b40cbb20 con 0x7fd1b415d3f0 2023-10-16T14:16:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.803+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd1ac007070 con 0x7fd1b415d3f0 2023-10-16T14:16:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.804+0000 7fd19b7fe640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd1a006ae70 0x7fd1a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.804+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1ac089500 con 0x7fd1b415d3f0 2023-10-16T14:16:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.805+0000 7fd1b2ffd640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd1a006ae70 0x7fd1a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.806+0000 7fd1b2ffd640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd1a006ae70 0x7fd1a006d330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cf620 tx=0x7fd1a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:34.809+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd1ac054650 con 0x7fd1b415d3f0 2023-10-16T14:16:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.001+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1b4160df0 con 0x7fd1a006ae70 2023-10-16T14:16:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.022+0000 7fd19b7fe640 1 -- 172.21.15.29:0/473151030 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264434 (secure 0 0 0) 0x7fd1a400c040 con 0x7fd1a006ae70 2023-10-16T14:16:35.030 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:16:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd1a006ae70 msgr2=0x7fd1a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd1a006ae70 0x7fd1a006d330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cf620 tx=0x7fd1a4008040 comp rx=0 tx=0).stop 2023-10-16T14:16:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 msgr2=0x7fd1b40ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 0x7fd1b40ae3d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00cbc0 tx=0x7fd1ac00a430 comp rx=0 tx=0).stop 2023-10-16T14:16:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 shutdown_connections 2023-10-16T14:16:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd1a006ae70 0x7fd1a006d330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b415d3f0 0x7fd1b40ae3d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1b40cd0c0 0x7fd1b41527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 --2- 172.21.15.29:0/473151030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1b40b0410 0x7fd1b415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 >> 172.21.15.29:0/473151030 conn(0x7fd1b40cb410 msgr2=0x7fd1b415cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 shutdown_connections 2023-10-16T14:16:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.027+0000 7fd1b93e9640 1 -- 172.21.15.29:0/473151030 wait complete. 2023-10-16T14:16:35.051 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-16T14:16:33.811369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252284,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722436,"statfs":{"total":773094113280,"available":772835774464,"internally_reserved":0,"allocated":2891776,"data_stored":1752740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":255315513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.808518"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799225+0000","last_change":"2023-10-16T14:16:29.830118+0000","last_active":"2023-10-16T14:16:30.799225+0000","last_peered":"2023-10-16T14:16:30.799225+0000","last_clean":"2023-10-16T14:16:30.799225+0000","last_became_active":"2023-10-16T14:16:29.829372+0000","last_became_peered":"2023-10-16T14:16:29.829372+0000","last_unstale":"2023-10-16T14:16:30.799225+0000","last_undegraded":"2023-10-16T14:16:30.799225+0000","last_fullsized":"2023-10-16T14:16:30.799225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794763+0000","last_change":"2023-10-16T14:16:29.818444+0000","last_active":"2023-10-16T14:16:30.794763+0000","last_peered":"2023-10-16T14:16:30.794763+0000","last_clean":"2023-10-16T14:16:30.794763+0000","last_became_active":"2023-10-16T14:16:29.816215+0000","last_became_peered":"2023-10-16T14:16:29.816215+0000","last_unstale":"2023-10-16T14:16:30.794763+0000","last_undegraded":"2023-10-16T14:16:30.794763+0000","last_fullsized":"2023-10-16T14:16:30.794763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326440+0000","last_change":"2023-10-16T14:16:29.817304+0000","last_active":"2023-10-16T14:16:32.326440+0000","last_peered":"2023-10-16T14:16:32.326440+0000","last_clean":"2023-10-16T14:16:32.326440+0000","last_became_active":"2023-10-16T14:16:29.817037+0000","last_became_peered":"2023-10-16T14:16:29.817037+0000","last_unstale":"2023-10-16T14:16:32.326440+0000","last_undegraded":"2023-10-16T14:16:32.326440+0000","last_fullsized":"2023-10-16T14:16:32.326440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827089+0000","last_change":"2023-10-16T14:16:29.816180+0000","last_active":"2023-10-16T14:16:31.827089+0000","last_peered":"2023-10-16T14:16:31.827089+0000","last_clean":"2023-10-16T14:16:31.827089+0000","last_became_active":"2023-10-16T14:16:29.815841+0000","last_became_peered":"2023-10-16T14:16:29.815841+0000","last_unstale":"2023-10-16T14:16:31.827089+0000","last_undegraded":"2023-10-16T14:16:31.827089+0000","last_fullsized":"2023-10-16T14:16:31.827089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798716+0000","last_change":"2023-10-16T14:16:29.813648+0000","last_active":"2023-10-16T14:16:30.798716+0000","last_peered":"2023-10-16T14:16:30.798716+0000","last_clean":"2023-10-16T14:16:30.798716+0000","last_became_active":"2023-10-16T14:16:29.813365+0000","last_became_peered":"2023-10-16T14:16:29.813365+0000","last_unstale":"2023-10-16T14:16:30.798716+0000","last_undegraded":"2023-10-16T14:16:30.798716+0000","last_fullsized":"2023-10-16T14:16:30.798716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825920+0000","last_change":"2023-10-16T14:16:29.818850+0000","last_active":"2023-10-16T14:16:31.825920+0000","last_peered":"2023-10-16T14:16:31.825920+0000","last_clean":"2023-10-16T14:16:31.825920+0000","last_became_active":"2023-10-16T14:16:29.818486+0000","last_became_peered":"2023-10-16T14:16:29.818486+0000","last_unstale":"2023-10-16T14:16:31.825920+0000","last_undegraded":"2023-10-16T14:16:31.825920+0000","last_fullsized":"2023-10-16T14:16:31.825920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796628+0000","last_change":"2023-10-16T14:16:29.814386+0000","last_active":"2023-10-16T14:16:30.796628+0000","last_peered":"2023-10-16T14:16:30.796628+0000","last_clean":"2023-10-16T14:16:30.796628+0000","last_became_active":"2023-10-16T14:16:29.812379+0000","last_became_peered":"2023-10-16T14:16:29.812379+0000","last_unstale":"2023-10-16T14:16:30.796628+0000","last_undegraded":"2023-10-16T14:16:30.796628+0000","last_fullsized":"2023-10-16T14:16:30.796628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667250+0000","last_change":"2023-10-16T14:16:29.823474+0000","last_active":"2023-10-16T14:16:31.667250+0000","last_peered":"2023-10-16T14:16:31.667250+0000","last_clean":"2023-10-16T14:16:31.667250+0000","last_became_active":"2023-10-16T14:16:29.823172+0000","last_became_peered":"2023-10-16T14:16:29.823172+0000","last_unstale":"2023-10-16T14:16:31.667250+0000","last_undegraded":"2023-10-16T14:16:31.667250+0000","last_fullsized":"2023-10-16T14:16:31.667250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799207+0000","last_change":"2023-10-16T14:16:29.830964+0000","last_active":"2023-10-16T14:16:30.799207+0000","last_peered":"2023-10-16T14:16:30.799207+0000","last_clean":"2023-10-16T14:16:30.799207+0000","last_became_active":"2023-10-16T14:16:29.829757+0000","last_became_peered":"2023-10-16T14:16:29.829757+0000","last_unstale":"2023-10-16T14:16:30.799207+0000","last_undegraded":"2023-10-16T14:16:30.799207+0000","last_fullsized":"2023-10-16T14:16:30.799207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961588+0000","last_change":"2023-10-16T14:16:29.822512+0000","last_active":"2023-10-16T14:16:30.961588+0000","last_peered":"2023-10-16T14:16:30.961588+0000","last_clean":"2023-10-16T14:16:30.961588+0000","last_became_active":"2023-10-16T14:16:29.822050+0000","last_became_peered":"2023-10-16T14:16:29.822050+0000","last_unstale":"2023-10-16T14:16:30.961588+0000","last_undegraded":"2023-10-16T14:16:30.961588+0000","last_fullsized":"2023-10-16T14:16:30.961588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961278+0000","last_change":"2023-10-16T14:16:29.819724+0000","last_active":"2023-10-16T14:16:30.961278+0000","last_peered":"2023-10-16T14:16:30.961278+0000","last_clean":"2023-10-16T14:16:30.961278+0000","last_became_active":"2023-10-16T14:16:29.819467+0000","last_became_peered":"2023-10-16T14:16:29.819467+0000","last_unstale":"2023-10-16T14:16:30.961278+0000","last_undegraded":"2023-10-16T14:16:30.961278+0000","last_fullsized":"2023-10-16T14:16:30.961278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797593+0000","last_change":"2023-10-16T14:16:29.819218+0000","last_active":"2023-10-16T14:16:30.797593+0000","last_peered":"2023-10-16T14:16:30.797593+0000","last_clean":"2023-10-16T14:16:30.797593+0000","last_became_active":"2023-10-16T14:16:29.818622+0000","last_became_peered":"2023-10-16T14:16:29.818622+0000","last_unstale":"2023-10-16T14:16:30.797593+0000","last_undegraded":"2023-10-16T14:16:30.797593+0000","last_fullsized":"2023-10-16T14:16:30.797593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668213+0000","last_change":"2023-10-16T14:16:29.817208+0000","last_active":"2023-10-16T14:16:31.668213+0000","last_peered":"2023-10-16T14:16:31.668213+0000","last_clean":"2023-10-16T14:16:31.668213+0000","last_became_active":"2023-10-16T14:16:29.816175+0000","last_became_peered":"2023-10-16T14:16:29.816175+0000","last_unstale":"2023-10-16T14:16:31.668213+0000","last_undegraded":"2023-10-16T14:16:31.668213+0000","last_fullsized":"2023-10-16T14:16:31.668213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325899+0000","last_change":"2023-10-16T14:16:29.812726+0000","last_active":"2023-10-16T14:16:32.325899+0000","last_peered":"2023-10-16T14:16:32.325899+0000","last_clean":"2023-10-16T14:16:32.325899+0000","last_became_active":"2023-10-16T14:16:29.812358+0000","last_became_peered":"2023-10-16T14:16:29.812358+0000","last_unstale":"2023-10-16T14:16:32.325899+0000","last_undegraded":"2023-10-16T14:16:32.325899+0000","last_fullsized":"2023-10-16T14:16:32.325899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:59.947139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794118+0000","last_change":"2023-10-16T14:16:29.820374+0000","last_active":"2023-10-16T14:16:30.794118+0000","last_peered":"2023-10-16T14:16:30.794118+0000","last_clean":"2023-10-16T14:16:30.794118+0000","last_became_active":"2023-10-16T14:16:29.820053+0000","last_became_peered":"2023-10-16T14:16:29.820053+0000","last_unstale":"2023-10-16T14:16:30.794118+0000","last_undegraded":"2023-10-16T14:16:30.794118+0000","last_fullsized":"2023-10-16T14:16:30.794118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825735+0000","last_change":"2023-10-16T14:16:29.819714+0000","last_active":"2023-10-16T14:16:31.825735+0000","last_peered":"2023-10-16T14:16:31.825735+0000","last_clean":"2023-10-16T14:16:31.825735+0000","last_became_active":"2023-10-16T14:16:29.819444+0000","last_became_peered":"2023-10-16T14:16:29.819444+0000","last_unstale":"2023-10-16T14:16:31.825735+0000","last_undegraded":"2023-10-16T14:16:31.825735+0000","last_fullsized":"2023-10-16T14:16:31.825735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826922+0000","last_change":"2023-10-16T14:16:29.817726+0000","last_active":"2023-10-16T14:16:31.826922+0000","last_peered":"2023-10-16T14:16:31.826922+0000","last_clean":"2023-10-16T14:16:31.826922+0000","last_became_active":"2023-10-16T14:16:29.813107+0000","last_became_peered":"2023-10-16T14:16:29.813107+0000","last_unstale":"2023-10-16T14:16:31.826922+0000","last_undegraded":"2023-10-16T14:16:31.826922+0000","last_fullsized":"2023-10-16T14:16:31.826922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325856+0000","last_change":"2023-10-16T14:16:29.825557+0000","last_active":"2023-10-16T14:16:32.325856+0000","last_peered":"2023-10-16T14:16:32.325856+0000","last_clean":"2023-10-16T14:16:32.325856+0000","last_became_active":"2023-10-16T14:16:29.825131+0000","last_became_peered":"2023-10-16T14:16:29.825131+0000","last_unstale":"2023-10-16T14:16:32.325856+0000","last_undegraded":"2023-10-16T14:16:32.325856+0000","last_fullsized":"2023-10-16T14:16:32.325856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798289+0000","last_change":"2023-10-16T14:16:29.813672+0000","last_active":"2023-10-16T14:16:30.798289+0000","last_peered":"2023-10-16T14:16:30.798289+0000","last_clean":"2023-10-16T14:16:30.798289+0000","last_became_active":"2023-10-16T14:16:29.813271+0000","last_became_peered":"2023-10-16T14:16:29.813271+0000","last_unstale":"2023-10-16T14:16:30.798289+0000","last_undegraded":"2023-10-16T14:16:30.798289+0000","last_fullsized":"2023-10-16T14:16:30.798289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961292+0000","last_change":"2023-10-16T14:16:29.821916+0000","last_active":"2023-10-16T14:16:30.961292+0000","last_peered":"2023-10-16T14:16:30.961292+0000","last_clean":"2023-10-16T14:16:30.961292+0000","last_became_active":"2023-10-16T14:16:29.821561+0000","last_became_peered":"2023-10-16T14:16:29.821561+0000","last_unstale":"2023-10-16T14:16:30.961292+0000","last_undegraded":"2023-10-16T14:16:30.961292+0000","last_fullsized":"2023-10-16T14:16:30.961292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326088+0000","last_change":"2023-10-16T14:16:29.809106+0000","last_active":"2023-10-16T14:16:32.326088+0000","last_peered":"2023-10-16T14:16:32.326088+0000","last_clean":"2023-10-16T14:16:32.326088+0000","last_became_active":"2023-10-16T14:16:29.808677+0000","last_became_peered":"2023-10-16T14:16:29.808677+0000","last_unstale":"2023-10-16T14:16:32.326088+0000","last_undegraded":"2023-10-16T14:16:32.326088+0000","last_fullsized":"2023-10-16T14:16:32.326088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:02.431928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.814365+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.814022+0000","last_became_peered":"2023-10-16T14:16:29.814022+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667249+0000","last_change":"2023-10-16T14:16:29.826061+0000","last_active":"2023-10-16T14:16:31.667249+0000","last_peered":"2023-10-16T14:16:31.667249+0000","last_clean":"2023-10-16T14:16:31.667249+0000","last_became_active":"2023-10-16T14:16:29.825662+0000","last_became_peered":"2023-10-16T14:16:29.825662+0000","last_unstale":"2023-10-16T14:16:31.667249+0000","last_undegraded":"2023-10-16T14:16:31.667249+0000","last_fullsized":"2023-10-16T14:16:31.667249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826173+0000","last_change":"2023-10-16T14:16:29.822279+0000","last_active":"2023-10-16T14:16:31.826173+0000","last_peered":"2023-10-16T14:16:31.826173+0000","last_clean":"2023-10-16T14:16:31.826173+0000","last_became_active":"2023-10-16T14:16:29.822022+0000","last_became_peered":"2023-10-16T14:16:29.822022+0000","last_unstale":"2023-10-16T14:16:31.826173+0000","last_undegraded":"2023-10-16T14:16:31.826173+0000","last_fullsized":"2023-10-16T14:16:31.826173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668113+0000","last_change":"2023-10-16T14:16:29.823044+0000","last_active":"2023-10-16T14:16:31.668113+0000","last_peered":"2023-10-16T14:16:31.668113+0000","last_clean":"2023-10-16T14:16:31.668113+0000","last_became_active":"2023-10-16T14:16:29.822705+0000","last_became_peered":"2023-10-16T14:16:29.822705+0000","last_unstale":"2023-10-16T14:16:31.668113+0000","last_undegraded":"2023-10-16T14:16:31.668113+0000","last_fullsized":"2023-10-16T14:16:31.668113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961559+0000","last_change":"2023-10-16T14:16:29.822259+0000","last_active":"2023-10-16T14:16:30.961559+0000","last_peered":"2023-10-16T14:16:30.961559+0000","last_clean":"2023-10-16T14:16:30.961559+0000","last_became_active":"2023-10-16T14:16:29.821719+0000","last_became_peered":"2023-10-16T14:16:29.821719+0000","last_unstale":"2023-10-16T14:16:30.961559+0000","last_undegraded":"2023-10-16T14:16:30.961559+0000","last_fullsized":"2023-10-16T14:16:30.961559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794706+0000","last_change":"2023-10-16T14:16:29.821655+0000","last_active":"2023-10-16T14:16:30.794706+0000","last_peered":"2023-10-16T14:16:30.794706+0000","last_clean":"2023-10-16T14:16:30.794706+0000","last_became_active":"2023-10-16T14:16:29.821357+0000","last_became_peered":"2023-10-16T14:16:29.821357+0000","last_unstale":"2023-10-16T14:16:30.794706+0000","last_undegraded":"2023-10-16T14:16:30.794706+0000","last_fullsized":"2023-10-16T14:16:30.794706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668435+0000","last_change":"2023-10-16T14:16:29.815493+0000","last_active":"2023-10-16T14:16:31.668435+0000","last_peered":"2023-10-16T14:16:31.668435+0000","last_clean":"2023-10-16T14:16:31.668435+0000","last_became_active":"2023-10-16T14:16:29.815203+0000","last_became_peered":"2023-10-16T14:16:29.815203+0000","last_unstale":"2023-10-16T14:16:31.668435+0000","last_undegraded":"2023-10-16T14:16:31.668435+0000","last_fullsized":"2023-10-16T14:16:31.668435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326367+0000","last_change":"2023-10-16T14:16:29.809001+0000","last_active":"2023-10-16T14:16:32.326367+0000","last_peered":"2023-10-16T14:16:32.326367+0000","last_clean":"2023-10-16T14:16:32.326367+0000","last_became_active":"2023-10-16T14:16:29.808169+0000","last_became_peered":"2023-10-16T14:16:29.808169+0000","last_unstale":"2023-10-16T14:16:32.326367+0000","last_undegraded":"2023-10-16T14:16:32.326367+0000","last_fullsized":"2023-10-16T14:16:32.326367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826340+0000","last_change":"2023-10-16T14:16:29.812984+0000","last_active":"2023-10-16T14:16:31.826340+0000","last_peered":"2023-10-16T14:16:31.826340+0000","last_clean":"2023-10-16T14:16:31.826340+0000","last_became_active":"2023-10-16T14:16:29.812569+0000","last_became_peered":"2023-10-16T14:16:29.812569+0000","last_unstale":"2023-10-16T14:16:31.826340+0000","last_undegraded":"2023-10-16T14:16:31.826340+0000","last_fullsized":"2023-10-16T14:16:31.826340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325543+0000","last_change":"2023-10-16T14:16:29.821793+0000","last_active":"2023-10-16T14:16:32.325543+0000","last_peered":"2023-10-16T14:16:32.325543+0000","last_clean":"2023-10-16T14:16:32.325543+0000","last_became_active":"2023-10-16T14:16:29.819170+0000","last_became_peered":"2023-10-16T14:16:29.819170+0000","last_unstale":"2023-10-16T14:16:32.325543+0000","last_undegraded":"2023-10-16T14:16:32.325543+0000","last_fullsized":"2023-10-16T14:16:32.325543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961200+0000","last_change":"2023-10-16T14:16:29.821091+0000","last_active":"2023-10-16T14:16:30.961200+0000","last_peered":"2023-10-16T14:16:30.961200+0000","last_clean":"2023-10-16T14:16:30.961200+0000","last_became_active":"2023-10-16T14:16:29.818555+0000","last_became_peered":"2023-10-16T14:16:29.818555+0000","last_unstale":"2023-10-16T14:16:30.961200+0000","last_undegraded":"2023-10-16T14:16:30.961200+0000","last_fullsized":"2023-10-16T14:16:30.961200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797715+0000","last_change":"2023-10-16T14:16:29.812942+0000","last_active":"2023-10-16T14:16:30.797715+0000","last_peered":"2023-10-16T14:16:30.797715+0000","last_clean":"2023-10-16T14:16:30.797715+0000","last_became_active":"2023-10-16T14:16:29.811742+0000","last_became_peered":"2023-10-16T14:16:29.811742+0000","last_unstale":"2023-10-16T14:16:30.797715+0000","last_undegraded":"2023-10-16T14:16:30.797715+0000","last_fullsized":"2023-10-16T14:16:30.797715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668347+0000","last_change":"2023-10-16T14:16:29.050890+0000","last_active":"2023-10-16T14:16:31.668347+0000","last_peered":"2023-10-16T14:16:31.668347+0000","last_clean":"2023-10-16T14:16:31.668347+0000","last_became_active":"2023-10-16T14:16:29.049249+0000","last_became_peered":"2023-10-16T14:16:29.049249+0000","last_unstale":"2023-10-16T14:16:31.668347+0000","last_undegraded":"2023-10-16T14:16:31.668347+0000","last_fullsized":"2023-10-16T14:16:31.668347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326412+0000","last_change":"2023-10-16T14:16:29.820123+0000","last_active":"2023-10-16T14:16:32.326412+0000","last_peered":"2023-10-16T14:16:32.326412+0000","last_clean":"2023-10-16T14:16:32.326412+0000","last_became_active":"2023-10-16T14:16:29.819857+0000","last_became_peered":"2023-10-16T14:16:29.819857+0000","last_unstale":"2023-10-16T14:16:32.326412+0000","last_undegraded":"2023-10-16T14:16:32.326412+0000","last_fullsized":"2023-10-16T14:16:32.326412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960647+0000","last_change":"2023-10-16T14:16:29.053825+0000","last_active":"2023-10-16T14:16:30.960647+0000","last_peered":"2023-10-16T14:16:30.960647+0000","last_clean":"2023-10-16T14:16:30.960647+0000","last_became_active":"2023-10-16T14:16:29.051055+0000","last_became_peered":"2023-10-16T14:16:29.051055+0000","last_unstale":"2023-10-16T14:16:30.960647+0000","last_undegraded":"2023-10-16T14:16:30.960647+0000","last_fullsized":"2023-10-16T14:16:30.960647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667594+0000","last_change":"2023-10-16T14:16:29.055594+0000","last_active":"2023-10-16T14:16:31.667594+0000","last_peered":"2023-10-16T14:16:31.667594+0000","last_clean":"2023-10-16T14:16:31.667594+0000","last_became_active":"2023-10-16T14:16:29.055260+0000","last_became_peered":"2023-10-16T14:16:29.055260+0000","last_unstale":"2023-10-16T14:16:31.667594+0000","last_undegraded":"2023-10-16T14:16:31.667594+0000","last_fullsized":"2023-10-16T14:16:31.667594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667384+0000","last_change":"2023-10-16T14:16:29.817463+0000","last_active":"2023-10-16T14:16:31.667384+0000","last_peered":"2023-10-16T14:16:31.667384+0000","last_clean":"2023-10-16T14:16:31.667384+0000","last_became_active":"2023-10-16T14:16:29.817155+0000","last_became_peered":"2023-10-16T14:16:29.817155+0000","last_unstale":"2023-10-16T14:16:31.667384+0000","last_undegraded":"2023-10-16T14:16:31.667384+0000","last_fullsized":"2023-10-16T14:16:31.667384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.047697+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.047399+0000","last_became_peered":"2023-10-16T14:16:29.047399+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794058+0000","last_change":"2023-10-16T14:16:29.811517+0000","last_active":"2023-10-16T14:16:30.794058+0000","last_peered":"2023-10-16T14:16:30.794058+0000","last_clean":"2023-10-16T14:16:30.794058+0000","last_became_active":"2023-10-16T14:16:29.811178+0000","last_became_peered":"2023-10-16T14:16:29.811178+0000","last_unstale":"2023-10-16T14:16:30.794058+0000","last_undegraded":"2023-10-16T14:16:30.794058+0000","last_fullsized":"2023-10-16T14:16:30.794058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961080+0000","last_change":"2023-10-16T14:16:29.061919+0000","last_active":"2023-10-16T14:16:30.961080+0000","last_peered":"2023-10-16T14:16:30.961080+0000","last_clean":"2023-10-16T14:16:30.961080+0000","last_became_active":"2023-10-16T14:16:29.061130+0000","last_became_peered":"2023-10-16T14:16:29.061130+0000","last_unstale":"2023-10-16T14:16:30.961080+0000","last_undegraded":"2023-10-16T14:16:30.961080+0000","last_fullsized":"2023-10-16T14:16:30.961080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826710+0000","last_change":"2023-10-16T14:16:29.817592+0000","last_active":"2023-10-16T14:16:31.826710+0000","last_peered":"2023-10-16T14:16:31.826710+0000","last_clean":"2023-10-16T14:16:31.826710+0000","last_became_active":"2023-10-16T14:16:29.817291+0000","last_became_peered":"2023-10-16T14:16:29.817291+0000","last_unstale":"2023-10-16T14:16:31.826710+0000","last_undegraded":"2023-10-16T14:16:31.826710+0000","last_fullsized":"2023-10-16T14:16:31.826710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826905+0000","last_change":"2023-10-16T14:16:29.059877+0000","last_active":"2023-10-16T14:16:31.826905+0000","last_peered":"2023-10-16T14:16:31.826905+0000","last_clean":"2023-10-16T14:16:31.826905+0000","last_became_active":"2023-10-16T14:16:29.059080+0000","last_became_peered":"2023-10-16T14:16:29.059080+0000","last_unstale":"2023-10-16T14:16:31.826905+0000","last_undegraded":"2023-10-16T14:16:31.826905+0000","last_fullsized":"2023-10-16T14:16:31.826905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:40.688481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326692+0000","last_change":"2023-10-16T14:16:29.827311+0000","last_active":"2023-10-16T14:16:32.326692+0000","last_peered":"2023-10-16T14:16:32.326692+0000","last_clean":"2023-10-16T14:16:32.326692+0000","last_became_active":"2023-10-16T14:16:29.827004+0000","last_became_peered":"2023-10-16T14:16:29.827004+0000","last_unstale":"2023-10-16T14:16:32.326692+0000","last_undegraded":"2023-10-16T14:16:32.326692+0000","last_fullsized":"2023-10-16T14:16:32.326692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794725+0000","last_change":"2023-10-16T14:16:29.059819+0000","last_active":"2023-10-16T14:16:30.794725+0000","last_peered":"2023-10-16T14:16:30.794725+0000","last_clean":"2023-10-16T14:16:30.794725+0000","last_became_active":"2023-10-16T14:16:29.059474+0000","last_became_peered":"2023-10-16T14:16:29.059474+0000","last_unstale":"2023-10-16T14:16:30.794725+0000","last_undegraded":"2023-10-16T14:16:30.794725+0000","last_fullsized":"2023-10-16T14:16:30.794725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797712+0000","last_change":"2023-10-16T14:16:29.813588+0000","last_active":"2023-10-16T14:16:30.797712+0000","last_peered":"2023-10-16T14:16:30.797712+0000","last_clean":"2023-10-16T14:16:30.797712+0000","last_became_active":"2023-10-16T14:16:29.813342+0000","last_became_peered":"2023-10-16T14:16:29.813342+0000","last_unstale":"2023-10-16T14:16:30.797712+0000","last_undegraded":"2023-10-16T14:16:30.797712+0000","last_fullsized":"2023-10-16T14:16:30.797712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:36.779577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961265+0000","last_change":"2023-10-16T14:16:29.057707+0000","last_active":"2023-10-16T14:16:30.961265+0000","last_peered":"2023-10-16T14:16:30.961265+0000","last_clean":"2023-10-16T14:16:30.961265+0000","last_became_active":"2023-10-16T14:16:29.054590+0000","last_became_peered":"2023-10-16T14:16:29.054590+0000","last_unstale":"2023-10-16T14:16:30.961265+0000","last_undegraded":"2023-10-16T14:16:30.961265+0000","last_fullsized":"2023-10-16T14:16:30.961265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797570+0000","last_change":"2023-10-16T14:16:29.810243+0000","last_active":"2023-10-16T14:16:30.797570+0000","last_peered":"2023-10-16T14:16:30.797570+0000","last_clean":"2023-10-16T14:16:30.797570+0000","last_became_active":"2023-10-16T14:16:29.809877+0000","last_became_peered":"2023-10-16T14:16:29.809877+0000","last_unstale":"2023-10-16T14:16:30.797570+0000","last_undegraded":"2023-10-16T14:16:30.797570+0000","last_fullsized":"2023-10-16T14:16:30.797570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326441+0000","last_change":"2023-10-16T14:16:29.046872+0000","last_active":"2023-10-16T14:16:32.326441+0000","last_peered":"2023-10-16T14:16:32.326441+0000","last_clean":"2023-10-16T14:16:32.326441+0000","last_became_active":"2023-10-16T14:16:29.046411+0000","last_became_peered":"2023-10-16T14:16:29.046411+0000","last_unstale":"2023-10-16T14:16:32.326441+0000","last_undegraded":"2023-10-16T14:16:32.326441+0000","last_fullsized":"2023-10-16T14:16:32.326441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:48.638693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326040+0000","last_change":"2023-10-16T14:16:29.810247+0000","last_active":"2023-10-16T14:16:32.326040+0000","last_peered":"2023-10-16T14:16:32.326040+0000","last_clean":"2023-10-16T14:16:32.326040+0000","last_became_active":"2023-10-16T14:16:29.809830+0000","last_became_peered":"2023-10-16T14:16:29.809830+0000","last_unstale":"2023-10-16T14:16:32.326040+0000","last_undegraded":"2023-10-16T14:16:32.326040+0000","last_fullsized":"2023-10-16T14:16:32.326040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:26.623468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826189+0000","last_change":"2023-10-16T14:16:29.049519+0000","last_active":"2023-10-16T14:16:31.826189+0000","last_peered":"2023-10-16T14:16:31.826189+0000","last_clean":"2023-10-16T14:16:31.826189+0000","last_became_active":"2023-10-16T14:16:29.049203+0000","last_became_peered":"2023-10-16T14:16:29.049203+0000","last_unstale":"2023-10-16T14:16:31.826189+0000","last_undegraded":"2023-10-16T14:16:31.826189+0000","last_fullsized":"2023-10-16T14:16:31.826189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:38.962308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798901+0000","last_change":"2023-10-16T14:16:29.824625+0000","last_active":"2023-10-16T14:16:30.798901+0000","last_peered":"2023-10-16T14:16:30.798901+0000","last_clean":"2023-10-16T14:16:30.798901+0000","last_became_active":"2023-10-16T14:16:29.818988+0000","last_became_peered":"2023-10-16T14:16:29.818988+0000","last_unstale":"2023-10-16T14:16:30.798901+0000","last_undegraded":"2023-10-16T14:16:30.798901+0000","last_fullsized":"2023-10-16T14:16:30.798901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960552+0000","last_change":"2023-10-16T14:16:29.054389+0000","last_active":"2023-10-16T14:16:30.960552+0000","last_peered":"2023-10-16T14:16:30.960552+0000","last_clean":"2023-10-16T14:16:30.960552+0000","last_became_active":"2023-10-16T14:16:29.053542+0000","last_became_peered":"2023-10-16T14:16:29.053542+0000","last_unstale":"2023-10-16T14:16:30.960552+0000","last_undegraded":"2023-10-16T14:16:30.960552+0000","last_fullsized":"2023-10-16T14:16:30.960552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960683+0000","last_change":"2023-10-16T14:16:29.822533+0000","last_active":"2023-10-16T14:16:30.960683+0000","last_peered":"2023-10-16T14:16:30.960683+0000","last_clean":"2023-10-16T14:16:30.960683+0000","last_became_active":"2023-10-16T14:16:29.822130+0000","last_became_peered":"2023-10-16T14:16:29.822130+0000","last_unstale":"2023-10-16T14:16:30.960683+0000","last_undegraded":"2023-10-16T14:16:30.960683+0000","last_fullsized":"2023-10-16T14:16:30.960683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325417+0000","last_change":"2023-10-16T14:16:29.044471+0000","last_active":"2023-10-16T14:16:32.325417+0000","last_peered":"2023-10-16T14:16:32.325417+0000","last_clean":"2023-10-16T14:16:32.325417+0000","last_became_active":"2023-10-16T14:16:29.042866+0000","last_became_peered":"2023-10-16T14:16:29.042866+0000","last_unstale":"2023-10-16T14:16:32.325417+0000","last_undegraded":"2023-10-16T14:16:32.325417+0000","last_fullsized":"2023-10-16T14:16:32.325417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:01.067871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799086+0000","last_change":"2023-10-16T14:16:29.826425+0000","last_active":"2023-10-16T14:16:30.799086+0000","last_peered":"2023-10-16T14:16:30.799086+0000","last_clean":"2023-10-16T14:16:30.799086+0000","last_became_active":"2023-10-16T14:16:29.820157+0000","last_became_peered":"2023-10-16T14:16:29.820157+0000","last_unstale":"2023-10-16T14:16:30.799086+0000","last_undegraded":"2023-10-16T14:16:30.799086+0000","last_fullsized":"2023-10-16T14:16:30.799086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797731+0000","last_change":"2023-10-16T14:16:29.049608+0000","last_active":"2023-10-16T14:16:30.797731+0000","last_peered":"2023-10-16T14:16:30.797731+0000","last_clean":"2023-10-16T14:16:30.797731+0000","last_became_active":"2023-10-16T14:16:29.049125+0000","last_became_peered":"2023-10-16T14:16:29.049125+0000","last_unstale":"2023-10-16T14:16:30.797731+0000","last_undegraded":"2023-10-16T14:16:30.797731+0000","last_fullsized":"2023-10-16T14:16:30.797731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798834+0000","last_change":"2023-10-16T14:16:29.809718+0000","last_active":"2023-10-16T14:16:30.798834+0000","last_peered":"2023-10-16T14:16:30.798834+0000","last_clean":"2023-10-16T14:16:30.798834+0000","last_became_active":"2023-10-16T14:16:29.809294+0000","last_became_peered":"2023-10-16T14:16:29.809294+0000","last_unstale":"2023-10-16T14:16:30.798834+0000","last_undegraded":"2023-10-16T14:16:30.798834+0000","last_fullsized":"2023-10-16T14:16:30.798834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325704+0000","last_change":"2023-10-16T14:16:29.034974+0000","last_active":"2023-10-16T14:16:32.325704+0000","last_peered":"2023-10-16T14:16:32.325704+0000","last_clean":"2023-10-16T14:16:32.325704+0000","last_became_active":"2023-10-16T14:16:29.034651+0000","last_became_peered":"2023-10-16T14:16:29.034651+0000","last_unstale":"2023-10-16T14:16:32.325704+0000","last_undegraded":"2023-10-16T14:16:32.325704+0000","last_fullsized":"2023-10-16T14:16:32.325704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:04.878575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799415+0000","last_change":"2023-10-16T14:16:29.829055+0000","last_active":"2023-10-16T14:16:30.799415+0000","last_peered":"2023-10-16T14:16:30.799415+0000","last_clean":"2023-10-16T14:16:30.799415+0000","last_became_active":"2023-10-16T14:16:29.820785+0000","last_became_peered":"2023-10-16T14:16:29.820785+0000","last_unstale":"2023-10-16T14:16:30.799415+0000","last_undegraded":"2023-10-16T14:16:30.799415+0000","last_fullsized":"2023-10-16T14:16:30.799415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794821+0000","last_change":"2023-10-16T14:16:29.048896+0000","last_active":"2023-10-16T14:16:30.794821+0000","last_peered":"2023-10-16T14:16:30.794821+0000","last_clean":"2023-10-16T14:16:30.794821+0000","last_became_active":"2023-10-16T14:16:29.048513+0000","last_became_peered":"2023-10-16T14:16:29.048513+0000","last_unstale":"2023-10-16T14:16:30.794821+0000","last_undegraded":"2023-10-16T14:16:30.794821+0000","last_fullsized":"2023-10-16T14:16:30.794821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827198+0000","last_change":"2023-10-16T14:16:29.814236+0000","last_active":"2023-10-16T14:16:31.827198+0000","last_peered":"2023-10-16T14:16:31.827198+0000","last_clean":"2023-10-16T14:16:31.827198+0000","last_became_active":"2023-10-16T14:16:29.813943+0000","last_became_peered":"2023-10-16T14:16:29.813943+0000","last_unstale":"2023-10-16T14:16:31.827198+0000","last_undegraded":"2023-10-16T14:16:31.827198+0000","last_fullsized":"2023-10-16T14:16:31.827198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325524+0000","last_change":"2023-10-16T14:16:29.052741+0000","last_active":"2023-10-16T14:16:32.325524+0000","last_peered":"2023-10-16T14:16:32.325524+0000","last_clean":"2023-10-16T14:16:32.325524+0000","last_became_active":"2023-10-16T14:16:29.052376+0000","last_became_peered":"2023-10-16T14:16:29.052376+0000","last_unstale":"2023-10-16T14:16:32.325524+0000","last_undegraded":"2023-10-16T14:16:32.325524+0000","last_fullsized":"2023-10-16T14:16:32.325524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:40.014521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797919+0000","last_change":"2023-10-16T14:16:29.828744+0000","last_active":"2023-10-16T14:16:30.797919+0000","last_peered":"2023-10-16T14:16:30.797919+0000","last_clean":"2023-10-16T14:16:30.797919+0000","last_became_active":"2023-10-16T14:16:29.823064+0000","last_became_peered":"2023-10-16T14:16:29.823064+0000","last_unstale":"2023-10-16T14:16:30.797919+0000","last_undegraded":"2023-10-16T14:16:30.797919+0000","last_fullsized":"2023-10-16T14:16:30.797919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-16T14:16:29.790600+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:16:29.790600+0000","last_peered":"2023-10-16T14:16:29.790600+0000","last_clean":"2023-10-16T14:16:29.790600+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:16:29.790600+0000","last_undegraded":"2023-10-16T14:16:29.790600+0000","last_fullsized":"2023-10-16T14:16:29.790600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825962+0000","last_change":"2023-10-16T14:16:29.051990+0000","last_active":"2023-10-16T14:16:31.825962+0000","last_peered":"2023-10-16T14:16:31.825962+0000","last_clean":"2023-10-16T14:16:31.825962+0000","last_became_active":"2023-10-16T14:16:29.051723+0000","last_became_peered":"2023-10-16T14:16:29.051723+0000","last_unstale":"2023-10-16T14:16:31.825962+0000","last_undegraded":"2023-10-16T14:16:31.825962+0000","last_fullsized":"2023-10-16T14:16:31.825962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667924+0000","last_change":"2023-10-16T14:16:29.823986+0000","last_active":"2023-10-16T14:16:31.667924+0000","last_peered":"2023-10-16T14:16:31.667924+0000","last_clean":"2023-10-16T14:16:31.667924+0000","last_became_active":"2023-10-16T14:16:29.823102+0000","last_became_peered":"2023-10-16T14:16:29.823102+0000","last_unstale":"2023-10-16T14:16:31.667924+0000","last_undegraded":"2023-10-16T14:16:31.667924+0000","last_fullsized":"2023-10-16T14:16:31.667924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795970+0000","last_change":"2023-10-16T14:16:29.058003+0000","last_active":"2023-10-16T14:16:30.795970+0000","last_peered":"2023-10-16T14:16:30.795970+0000","last_clean":"2023-10-16T14:16:30.795970+0000","last_became_active":"2023-10-16T14:16:29.057686+0000","last_became_peered":"2023-10-16T14:16:29.057686+0000","last_unstale":"2023-10-16T14:16:30.795970+0000","last_undegraded":"2023-10-16T14:16:30.795970+0000","last_fullsized":"2023-10-16T14:16:30.795970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325936+0000","last_change":"2023-10-16T14:16:29.812594+0000","last_active":"2023-10-16T14:16:32.325936+0000","last_peered":"2023-10-16T14:16:32.325936+0000","last_clean":"2023-10-16T14:16:32.325936+0000","last_became_active":"2023-10-16T14:16:29.812256+0000","last_became_peered":"2023-10-16T14:16:29.812256+0000","last_unstale":"2023-10-16T14:16:32.325936+0000","last_undegraded":"2023-10-16T14:16:32.325936+0000","last_fullsized":"2023-10-16T14:16:32.325936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:48.787869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325633+0000","last_change":"2023-10-16T14:16:29.042443+0000","last_active":"2023-10-16T14:16:32.325633+0000","last_peered":"2023-10-16T14:16:32.325633+0000","last_clean":"2023-10-16T14:16:32.325633+0000","last_became_active":"2023-10-16T14:16:29.042189+0000","last_became_peered":"2023-10-16T14:16:29.042189+0000","last_unstale":"2023-10-16T14:16:32.325633+0000","last_undegraded":"2023-10-16T14:16:32.325633+0000","last_fullsized":"2023-10-16T14:16:32.325633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794175+0000","last_change":"2023-10-16T14:16:29.816645+0000","last_active":"2023-10-16T14:16:30.794175+0000","last_peered":"2023-10-16T14:16:30.794175+0000","last_clean":"2023-10-16T14:16:30.794175+0000","last_became_active":"2023-10-16T14:16:29.814792+0000","last_became_peered":"2023-10-16T14:16:29.814792+0000","last_unstale":"2023-10-16T14:16:30.794175+0000","last_undegraded":"2023-10-16T14:16:30.794175+0000","last_fullsized":"2023-10-16T14:16:30.794175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826471+0000","last_change":"2023-10-16T14:16:29.055439+0000","last_active":"2023-10-16T14:16:31.826471+0000","last_peered":"2023-10-16T14:16:31.826471+0000","last_clean":"2023-10-16T14:16:31.826471+0000","last_became_active":"2023-10-16T14:16:29.054422+0000","last_became_peered":"2023-10-16T14:16:29.054422+0000","last_unstale":"2023-10-16T14:16:31.826471+0000","last_undegraded":"2023-10-16T14:16:31.826471+0000","last_fullsized":"2023-10-16T14:16:31.826471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797290+0000","last_change":"2023-10-16T14:16:29.826562+0000","last_active":"2023-10-16T14:16:30.797290+0000","last_peered":"2023-10-16T14:16:30.797290+0000","last_clean":"2023-10-16T14:16:30.797290+0000","last_became_active":"2023-10-16T14:16:29.825184+0000","last_became_peered":"2023-10-16T14:16:29.825184+0000","last_unstale":"2023-10-16T14:16:30.797290+0000","last_undegraded":"2023-10-16T14:16:30.797290+0000","last_fullsized":"2023-10-16T14:16:30.797290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:49.651724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794587+0000","last_change":"2023-10-16T14:16:29.048497+0000","last_active":"2023-10-16T14:16:30.794587+0000","last_peered":"2023-10-16T14:16:30.794587+0000","last_clean":"2023-10-16T14:16:30.794587+0000","last_became_active":"2023-10-16T14:16:29.048194+0000","last_became_peered":"2023-10-16T14:16:29.048194+0000","last_unstale":"2023-10-16T14:16:30.794587+0000","last_undegraded":"2023-10-16T14:16:30.794587+0000","last_fullsized":"2023-10-16T14:16:30.794587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826954+0000","last_change":"2023-10-16T14:16:29.819205+0000","last_active":"2023-10-16T14:16:31.826954+0000","last_peered":"2023-10-16T14:16:31.826954+0000","last_clean":"2023-10-16T14:16:31.826954+0000","last_became_active":"2023-10-16T14:16:29.818921+0000","last_became_peered":"2023-10-16T14:16:29.818921+0000","last_unstale":"2023-10-16T14:16:31.826954+0000","last_undegraded":"2023-10-16T14:16:31.826954+0000","last_fullsized":"2023-10-16T14:16:31.826954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798586+0000","last_change":"2023-10-16T14:16:29.053513+0000","last_active":"2023-10-16T14:16:30.798586+0000","last_peered":"2023-10-16T14:16:30.798586+0000","last_clean":"2023-10-16T14:16:30.798586+0000","last_became_active":"2023-10-16T14:16:29.053178+0000","last_became_peered":"2023-10-16T14:16:29.053178+0000","last_unstale":"2023-10-16T14:16:30.798586+0000","last_undegraded":"2023-10-16T14:16:30.798586+0000","last_fullsized":"2023-10-16T14:16:30.798586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794789+0000","last_change":"2023-10-16T14:16:29.817442+0000","last_active":"2023-10-16T14:16:30.794789+0000","last_peered":"2023-10-16T14:16:30.794789+0000","last_clean":"2023-10-16T14:16:30.794789+0000","last_became_active":"2023-10-16T14:16:29.814899+0000","last_became_peered":"2023-10-16T14:16:29.814899+0000","last_unstale":"2023-10-16T14:16:30.794789+0000","last_undegraded":"2023-10-16T14:16:30.794789+0000","last_fullsized":"2023-10-16T14:16:30.794789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794674+0000","last_change":"2023-10-16T14:16:29.059226+0000","last_active":"2023-10-16T14:16:30.794674+0000","last_peered":"2023-10-16T14:16:30.794674+0000","last_clean":"2023-10-16T14:16:30.794674+0000","last_became_active":"2023-10-16T14:16:29.058896+0000","last_became_peered":"2023-10-16T14:16:29.058896+0000","last_unstale":"2023-10-16T14:16:30.794674+0000","last_undegraded":"2023-10-16T14:16:30.794674+0000","last_fullsized":"2023-10-16T14:16:30.794674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668348+0000","last_change":"2023-10-16T14:16:29.807423+0000","last_active":"2023-10-16T14:16:31.668348+0000","last_peered":"2023-10-16T14:16:31.668348+0000","last_clean":"2023-10-16T14:16:31.668348+0000","last_became_active":"2023-10-16T14:16:29.806278+0000","last_became_peered":"2023-10-16T14:16:29.806278+0000","last_unstale":"2023-10-16T14:16:31.668348+0000","last_undegraded":"2023-10-16T14:16:31.668348+0000","last_fullsized":"2023-10-16T14:16:31.668348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961275+0000","last_change":"2023-10-16T14:16:29.057876+0000","last_active":"2023-10-16T14:16:30.961275+0000","last_peered":"2023-10-16T14:16:30.961275+0000","last_clean":"2023-10-16T14:16:30.961275+0000","last_became_active":"2023-10-16T14:16:29.053426+0000","last_became_peered":"2023-10-16T14:16:29.053426+0000","last_unstale":"2023-10-16T14:16:30.961275+0000","last_undegraded":"2023-10-16T14:16:30.961275+0000","last_fullsized":"2023-10-16T14:16:30.961275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:01.418897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326675+0000","last_change":"2023-10-16T14:16:29.816014+0000","last_active":"2023-10-16T14:16:32.326675+0000","last_peered":"2023-10-16T14:16:32.326675+0000","last_clean":"2023-10-16T14:16:32.326675+0000","last_became_active":"2023-10-16T14:16:29.815725+0000","last_became_peered":"2023-10-16T14:16:29.815725+0000","last_unstale":"2023-10-16T14:16:32.326675+0000","last_undegraded":"2023-10-16T14:16:32.326675+0000","last_fullsized":"2023-10-16T14:16:32.326675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795715+0000","last_change":"2023-10-16T14:16:29.050583+0000","last_active":"2023-10-16T14:16:30.795715+0000","last_peered":"2023-10-16T14:16:30.795715+0000","last_clean":"2023-10-16T14:16:30.795715+0000","last_became_active":"2023-10-16T14:16:29.050281+0000","last_became_peered":"2023-10-16T14:16:29.050281+0000","last_unstale":"2023-10-16T14:16:30.795715+0000","last_undegraded":"2023-10-16T14:16:30.795715+0000","last_fullsized":"2023-10-16T14:16:30.795715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:18.882395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796790+0000","last_change":"2023-10-16T14:16:29.808419+0000","last_active":"2023-10-16T14:16:30.796790+0000","last_peered":"2023-10-16T14:16:30.796790+0000","last_clean":"2023-10-16T14:16:30.796790+0000","last_became_active":"2023-10-16T14:16:29.808038+0000","last_became_peered":"2023-10-16T14:16:29.808038+0000","last_unstale":"2023-10-16T14:16:30.796790+0000","last_undegraded":"2023-10-16T14:16:30.796790+0000","last_fullsized":"2023-10-16T14:16:30.796790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:05.891605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794294+0000","last_change":"2023-10-16T14:16:29.049120+0000","last_active":"2023-10-16T14:16:30.794294+0000","last_peered":"2023-10-16T14:16:30.794294+0000","last_clean":"2023-10-16T14:16:30.794294+0000","last_became_active":"2023-10-16T14:16:29.048782+0000","last_became_peered":"2023-10-16T14:16:29.048782+0000","last_unstale":"2023-10-16T14:16:30.794294+0000","last_undegraded":"2023-10-16T14:16:30.794294+0000","last_fullsized":"2023-10-16T14:16:30.794294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667807+0000","last_change":"2023-10-16T14:16:29.820694+0000","last_active":"2023-10-16T14:16:31.667807+0000","last_peered":"2023-10-16T14:16:31.667807+0000","last_clean":"2023-10-16T14:16:31.667807+0000","last_became_active":"2023-10-16T14:16:29.820341+0000","last_became_peered":"2023-10-16T14:16:29.820341+0000","last_unstale":"2023-10-16T14:16:31.667807+0000","last_undegraded":"2023-10-16T14:16:31.667807+0000","last_fullsized":"2023-10-16T14:16:31.667807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794452+0000","last_change":"2023-10-16T14:16:29.043057+0000","last_active":"2023-10-16T14:16:30.794452+0000","last_peered":"2023-10-16T14:16:30.794452+0000","last_clean":"2023-10-16T14:16:30.794452+0000","last_became_active":"2023-10-16T14:16:29.042783+0000","last_became_peered":"2023-10-16T14:16:29.042783+0000","last_unstale":"2023-10-16T14:16:30.794452+0000","last_undegraded":"2023-10-16T14:16:30.794452+0000","last_fullsized":"2023-10-16T14:16:30.794452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668050+0000","last_change":"2023-10-16T14:16:29.816920+0000","last_active":"2023-10-16T14:16:31.668050+0000","last_peered":"2023-10-16T14:16:31.668050+0000","last_clean":"2023-10-16T14:16:31.668050+0000","last_became_active":"2023-10-16T14:16:29.816639+0000","last_became_peered":"2023-10-16T14:16:29.816639+0000","last_unstale":"2023-10-16T14:16:31.668050+0000","last_undegraded":"2023-10-16T14:16:31.668050+0000","last_fullsized":"2023-10-16T14:16:31.668050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798966+0000","last_change":"2023-10-16T14:16:29.056773+0000","last_active":"2023-10-16T14:16:30.798966+0000","last_peered":"2023-10-16T14:16:30.798966+0000","last_clean":"2023-10-16T14:16:30.798966+0000","last_became_active":"2023-10-16T14:16:29.056477+0000","last_became_peered":"2023-10-16T14:16:29.056477+0000","last_unstale":"2023-10-16T14:16:30.798966+0000","last_undegraded":"2023-10-16T14:16:30.798966+0000","last_fullsized":"2023-10-16T14:16:30.798966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:17.696119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960869+0000","last_change":"2023-10-16T14:16:29.821383+0000","last_active":"2023-10-16T14:16:30.960869+0000","last_peered":"2023-10-16T14:16:30.960869+0000","last_clean":"2023-10-16T14:16:30.960869+0000","last_became_active":"2023-10-16T14:16:29.821018+0000","last_became_peered":"2023-10-16T14:16:29.821018+0000","last_unstale":"2023-10-16T14:16:30.960869+0000","last_undegraded":"2023-10-16T14:16:30.960869+0000","last_fullsized":"2023-10-16T14:16:30.960869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826784+0000","last_change":"2023-10-16T14:16:29.829570+0000","last_active":"2023-10-16T14:16:31.826784+0000","last_peered":"2023-10-16T14:16:31.826784+0000","last_clean":"2023-10-16T14:16:31.826784+0000","last_became_active":"2023-10-16T14:16:29.828707+0000","last_became_peered":"2023-10-16T14:16:29.828707+0000","last_unstale":"2023-10-16T14:16:31.826784+0000","last_undegraded":"2023-10-16T14:16:31.826784+0000","last_fullsized":"2023-10-16T14:16:31.826784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798241+0000","last_change":"2023-10-16T14:16:29.063205+0000","last_active":"2023-10-16T14:16:30.798241+0000","last_peered":"2023-10-16T14:16:30.798241+0000","last_clean":"2023-10-16T14:16:30.798241+0000","last_became_active":"2023-10-16T14:16:29.062657+0000","last_became_peered":"2023-10-16T14:16:29.062657+0000","last_unstale":"2023-10-16T14:16:30.798241+0000","last_undegraded":"2023-10-16T14:16:30.798241+0000","last_fullsized":"2023-10-16T14:16:30.798241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:43.103258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826847+0000","last_change":"2023-10-16T14:16:29.830412+0000","last_active":"2023-10-16T14:16:31.826847+0000","last_peered":"2023-10-16T14:16:31.826847+0000","last_clean":"2023-10-16T14:16:31.826847+0000","last_became_active":"2023-10-16T14:16:29.825817+0000","last_became_peered":"2023-10-16T14:16:29.825817+0000","last_unstale":"2023-10-16T14:16:31.826847+0000","last_undegraded":"2023-10-16T14:16:31.826847+0000","last_fullsized":"2023-10-16T14:16:31.826847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798099+0000","last_change":"2023-10-16T14:16:29.061570+0000","last_active":"2023-10-16T14:16:30.798099+0000","last_peered":"2023-10-16T14:16:30.798099+0000","last_clean":"2023-10-16T14:16:30.798099+0000","last_became_active":"2023-10-16T14:16:29.061024+0000","last_became_peered":"2023-10-16T14:16:29.061024+0000","last_unstale":"2023-10-16T14:16:30.798099+0000","last_undegraded":"2023-10-16T14:16:30.798099+0000","last_fullsized":"2023-10-16T14:16:30.798099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326481+0000","last_change":"2023-10-16T14:16:29.827404+0000","last_active":"2023-10-16T14:16:32.326481+0000","last_peered":"2023-10-16T14:16:32.326481+0000","last_clean":"2023-10-16T14:16:32.326481+0000","last_became_active":"2023-10-16T14:16:29.826687+0000","last_became_peered":"2023-10-16T14:16:29.826687+0000","last_unstale":"2023-10-16T14:16:32.326481+0000","last_undegraded":"2023-10-16T14:16:32.326481+0000","last_fullsized":"2023-10-16T14:16:32.326481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668592+0000","last_change":"2023-10-16T14:16:29.050871+0000","last_active":"2023-10-16T14:16:31.668592+0000","last_peered":"2023-10-16T14:16:31.668592+0000","last_clean":"2023-10-16T14:16:31.668592+0000","last_became_active":"2023-10-16T14:16:29.050025+0000","last_became_peered":"2023-10-16T14:16:29.050025+0000","last_unstale":"2023-10-16T14:16:31.668592+0000","last_undegraded":"2023-10-16T14:16:31.668592+0000","last_fullsized":"2023-10-16T14:16:31.668592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668230+0000","last_change":"2023-10-16T14:16:29.825092+0000","last_active":"2023-10-16T14:16:31.668230+0000","last_peered":"2023-10-16T14:16:31.668230+0000","last_clean":"2023-10-16T14:16:31.668230+0000","last_became_active":"2023-10-16T14:16:29.824815+0000","last_became_peered":"2023-10-16T14:16:29.824815+0000","last_unstale":"2023-10-16T14:16:31.668230+0000","last_undegraded":"2023-10-16T14:16:31.668230+0000","last_fullsized":"2023-10-16T14:16:31.668230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326685+0000","last_change":"2023-10-16T14:16:29.053310+0000","last_active":"2023-10-16T14:16:32.326685+0000","last_peered":"2023-10-16T14:16:32.326685+0000","last_clean":"2023-10-16T14:16:32.326685+0000","last_became_active":"2023-10-16T14:16:29.052871+0000","last_became_peered":"2023-10-16T14:16:29.052871+0000","last_unstale":"2023-10-16T14:16:32.326685+0000","last_undegraded":"2023-10-16T14:16:32.326685+0000","last_fullsized":"2023-10-16T14:16:32.326685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9550000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4529999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3159999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1760000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.484}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6030000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2519999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.885}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:16:35.057 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:16:35.057 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-16T14:16:35.058 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-16T14:16:35.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T14:16:35.058 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-16T14:16:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.319+0000 7f7cce0d7640 1 Processor -- start 2023-10-16T14:16:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.319+0000 7f7cce0d7640 1 -- start start 2023-10-16T14:16:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7cce0d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7cc815cbe0 0x7f7cc80d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7cce0d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cc815cfc0 0x7f7cc80b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7cce0d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc80d1090 0x7f7cc80b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7cce0d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc80d15a0 con 0x7f7cc815cfc0 2023-10-16T14:16:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7cce0d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc80d1880 con 0x7f7cc815cbe0 2023-10-16T14:16:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7cce0d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80d1b60 con 0x7f7cc80d1090 2023-10-16T14:16:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7ccc8d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cc815cfc0 0x7f7cc80b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7ccc8d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cc815cfc0 0x7f7cc80b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50648/0 (socket says 172.21.15.29:50648) 2023-10-16T14:16:35.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.320+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/831978503 learned_addr learned my addr 172.21.15.29:0/831978503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccd8d6640 1 --2- 172.21.15.29:0/831978503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc80d1090 0x7f7cc80b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/831978503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc80d1090 msgr2=0x7f7cc80b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccc8d4640 1 --2- 172.21.15.29:0/831978503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc80d1090 0x7f7cc80b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/831978503 >> v2:172.21.15.170:3300/0 conn(0x7f7cc815cbe0 msgr2=0x7f7cc80d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccd0d5640 1 --2- 172.21.15.29:0/831978503 >> v2:172.21.15.170:3300/0 conn(0x7f7cc815cbe0 0x7f7cc80d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccc8d4640 1 --2- 172.21.15.29:0/831978503 >> v2:172.21.15.170:3300/0 conn(0x7f7cc815cbe0 0x7f7cc80d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.321+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/831978503 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc80d1e40 con 0x7f7cc815cfc0 2023-10-16T14:16:35.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.322+0000 7f7ccc8d4640 1 --2- 172.21.15.29:0/831978503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cfc0 0x7f7cc80b4570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80b53f0 tx=0x7f7cb802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa6575472c93ec1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:35.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.322+0000 7f7cbffff640 1 -- 172.21.15.29:0/831978503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8030850 con 0x7f7cc815cfc0 2023-10-16T14:16:35.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.322+0000 7f7cbffff640 1 -- 172.21.15.29:0/831978503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb80309b0 con 0x7f7cc815cfc0 2023-10-16T14:16:35.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.322+0000 7f7cbffff640 1 -- 172.21.15.29:0/831978503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8030d40 con 0x7f7cc815cfc0 2023-10-16T14:16:35.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 -- 172.21.15.29:0/831978503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cfc0 msgr2=0x7f7cc80b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/831978503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cfc0 0x7f7cc80b4570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80b53f0 tx=0x7f7cb802b990 comp rx=0 tx=0).stop 2023-10-16T14:16:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 -- 172.21.15.29:0/831978503 shutdown_connections 2023-10-16T14:16:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/831978503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc80d1090 0x7f7cc80b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/831978503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cfc0 0x7f7cc80b4570 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/831978503 >> v2:172.21.15.170:3300/0 conn(0x7f7cc815cbe0 0x7f7cc80d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 -- 172.21.15.29:0/831978503 >> 172.21.15.29:0/831978503 conn(0x7f7cc80cf570 msgr2=0x7f7cc80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 -- 172.21.15.29:0/831978503 shutdown_connections 2023-10-16T14:16:35.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.323+0000 7f7cce0d7640 1 -- 172.21.15.29:0/831978503 wait complete. 2023-10-16T14:16:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.325+0000 7f7cce0d7640 1 Processor -- start 2023-10-16T14:16:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.325+0000 7f7cce0d7640 1 -- start start 2023-10-16T14:16:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7cce0d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cc80d1090 0x7f7cc80c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7cce0d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 0x7f7cc80cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7cce0d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc815cfc0 0x7f7cc81f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7cce0d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80c82b0 con 0x7f7cc815cbe0 2023-10-16T14:16:35.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7cce0d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80c8590 con 0x7f7cc80d1090 2023-10-16T14:16:35.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7cce0d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80c8870 con 0x7f7cc815cfc0 2023-10-16T14:16:35.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7ccc8d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 0x7f7cc80cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7ccc8d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 0x7f7cc80cc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50650/0 (socket says 172.21.15.29:50650) 2023-10-16T14:16:35.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.326+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/2711419729 learned_addr learned my addr 172.21.15.29:0/2711419729 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:35.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccd0d5640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cc80d1090 0x7f7cc80c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc815cfc0 msgr2=0x7f7cc81f20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccc8d4640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc815cfc0 0x7f7cc81f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cc80d1090 msgr2=0x7f7cc80c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccc8d4640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cc80d1090 0x7f7cc80c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccc8d4640 1 -- 172.21.15.29:0/2711419729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80094d0 con 0x7f7cc815cbe0 2023-10-16T14:16:35.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccd0d5640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cc80d1090 0x7f7cc80c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:35.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7ccc8d4640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 0x7f7cc80cc240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7cb802b9d0 tx=0x7f7cb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:35.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8030840 con 0x7f7cc815cbe0 2023-10-16T14:16:35.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb8039d90 con 0x7f7cc815cbe0 2023-10-16T14:16:35.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8043650 con 0x7f7cc815cbe0 2023-10-16T14:16:35.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.327+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc80c8b50 con 0x7f7cc815cbe0 2023-10-16T14:16:35.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.328+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc80c7f80 con 0x7f7cc815cbe0 2023-10-16T14:16:35.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.329+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc80c9070 con 0x7f7cc815cbe0 2023-10-16T14:16:35.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.329+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7cb802e070 con 0x7f7cc815cbe0 2023-10-16T14:16:35.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.330+0000 7f7cbdffb640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7cac06ae70 0x7f7cac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.330+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cb80ab700 con 0x7f7cc815cbe0 2023-10-16T14:16:35.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.331+0000 7f7ccd0d5640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7cac06ae70 0x7f7cac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.332+0000 7f7ccd0d5640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7cac06ae70 0x7f7cac06d330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400b840 tx=0x7f7cc4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:35.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.335+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7cc80c9070 con 0x7f7cc815cbe0 2023-10-16T14:16:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.582+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7cc80c9070 con 0x7f7cc815cbe0 2023-10-16T14:16:35.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.591+0000 7f7cbdffb640 1 -- 172.21.15.29:0/2711419729 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171036 (secure 0 0 0) 0x7f7cc80c9070 con 0x7f7cc815cbe0 2023-10-16T14:16:35.599 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7cac06ae70 msgr2=0x7f7cac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7cac06ae70 0x7f7cac06d330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400b840 tx=0x7f7cc4008040 comp rx=0 tx=0).stop 2023-10-16T14:16:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 msgr2=0x7f7cc80cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 0x7f7cc80cc240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7cb802b9d0 tx=0x7f7cb8004730 comp rx=0 tx=0).stop 2023-10-16T14:16:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 shutdown_connections 2023-10-16T14:16:35.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7cac06ae70 0x7f7cac06d330 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cc815cbe0 0x7f7cc80cc240 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cc815cfc0 0x7f7cc81f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 --2- 172.21.15.29:0/2711419729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cc80d1090 0x7f7cc80c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 >> 172.21.15.29:0/2711419729 conn(0x7f7cc80cf570 msgr2=0x7f7cc815c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 shutdown_connections 2023-10-16T14:16:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.596+0000 7f7cce0d7640 1 -- 172.21.15.29:0/2711419729 wait complete. 2023-10-16T14:16:35.619 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1268688927},{"type":"v1","addr":"172.21.15.170:6833","nonce":1268688927}]},"active_addr":"172.21.15.170:6833/1268688927","active_change":"2023-10-16T14:16:09.792244+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":176252655}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":1855634044}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":3736763821}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":3139659877}]}]} 2023-10-16T14:16:35.624 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-16T14:16:35.624 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-16T14:16:35.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:35.822 INFO:tasks.ceph.mds.d.smithi170.stderr:2023-10-16T14:16:35.819+0000 7f8dc7243640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:16:35.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.844+0000 7fb0c4452640 1 Processor -- start 2023-10-16T14:16:35.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.844+0000 7fb0c4452640 1 -- start start 2023-10-16T14:16:35.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.845+0000 7fb0c4452640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0bc15aaa0 0x7fb0bc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.845+0000 7fb0c4452640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0bc15ae80 0x7fb0bc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c4452640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc15ed60 con 0x7fb0bc0ccaa0 2023-10-16T14:16:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c4452640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc06b810 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c4452640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc0707e0 con 0x7fb0bc15ae80 2023-10-16T14:16:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0bc15aaa0 0x7fb0bc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0bc15aaa0 0x7fb0bc15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50666/0 (socket says 172.21.15.29:50666) 2023-10-16T14:16:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 -- 172.21.15.29:0/2492239714 learned_addr learned my addr 172.21.15.29:0/2492239714 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 -- 172.21.15.29:0/2492239714 >> v2:172.21.15.170:3300/0 conn(0x7fb0bc15ae80 msgr2=0x7fb0bc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 --2- 172.21.15.29:0/2492239714 >> v2:172.21.15.170:3300/0 conn(0x7fb0bc15ae80 0x7fb0bc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 -- 172.21.15.29:0/2492239714 >> v1:172.21.15.170:6789/0 conn(0x7fb0bc0ccaa0 legacy=0x7fb0bc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.846+0000 7fb0c21ee640 1 -- 172.21.15.29:0/2492239714 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc15ef90 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.847+0000 7fb0c21ee640 1 --2- 172.21.15.29:0/2492239714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc15e820 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0c59c0 tx=0x7fb0b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b045a41a9735173 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c11ec640 1 -- 172.21.15.29:0/2492239714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8031870 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c11ec640 1 -- 172.21.15.29:0/2492239714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b80319d0 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c11ec640 1 -- 172.21.15.29:0/2492239714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8031d60 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 -- 172.21.15.29:0/2492239714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 msgr2=0x7fb0bc15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 --2- 172.21.15.29:0/2492239714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc15e820 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0c59c0 tx=0x7fb0b802b910 comp rx=0 tx=0).stop 2023-10-16T14:16:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 -- 172.21.15.29:0/2492239714 shutdown_connections 2023-10-16T14:16:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 --2- 172.21.15.29:0/2492239714 >> v2:172.21.15.170:3300/0 conn(0x7fb0bc15ae80 0x7fb0bc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 --2- 172.21.15.29:0/2492239714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc15e820 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 -- 172.21.15.29:0/2492239714 >> 172.21.15.29:0/2492239714 conn(0x7fb0bc0cb410 msgr2=0x7fb0bc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 -- 172.21.15.29:0/2492239714 shutdown_connections 2023-10-16T14:16:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.848+0000 7fb0c4452640 1 -- 172.21.15.29:0/2492239714 wait complete. 2023-10-16T14:16:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.850+0000 7fb0c4452640 1 Processor -- start 2023-10-16T14:16:35.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.850+0000 7fb0c4452640 1 -- start start 2023-10-16T14:16:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c4452640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0bc0ccaa0 0x7fb0bc15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c4452640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c4452640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0bc15ae80 0x7fb0bc0c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c4452640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc155ca0 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c4452640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c4680 con 0x7fb0bc0ccaa0 2023-10-16T14:16:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c4452640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c37f0 con 0x7fb0bc15ae80 2023-10-16T14:16:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c19ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c19ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50674/0 (socket says 172.21.15.29:50674) 2023-10-16T14:16:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.851+0000 7fb0c19ed640 1 -- 172.21.15.29:0/1610929782 learned_addr learned my addr 172.21.15.29:0/1610929782 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c21ee640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0bc0ccaa0 0x7fb0bc15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c19ed640 1 -- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0bc15ae80 msgr2=0x7fb0bc0c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c19ed640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0bc15ae80 0x7fb0bc0c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c19ed640 1 -- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0bc0ccaa0 msgr2=0x7fb0bc15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c19ed640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0bc0ccaa0 0x7fb0bc15dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c19ed640 1 -- 172.21.15.29:0/1610929782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b80094d0 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c21ee640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0bc0ccaa0 0x7fb0bc15dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c19ed640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc153bc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b410 tx=0x7fb0ac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac011830 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac0119b0 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac011d60 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.852+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc0c4100 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.853+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc15b4c0 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.854+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc15b320 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.855+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb0ac01a400 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.856+0000 7fb0b2ffd640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb0a006aea0 0x7fb0a006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.856+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0ac089720 con 0x7fb0bc15aaa0 2023-10-16T14:16:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.856+0000 7fb0c21ee640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb0a006aea0 0x7fb0a006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.857+0000 7fb0c21ee640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb0a006aea0 0x7fb0a006d360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8036000 tx=0x7fb0b803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:35.860+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0bc15b320 con 0x7fb0bc15aaa0 2023-10-16T14:16:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.057+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0bc15b320 con 0x7fb0bc15aaa0 2023-10-16T14:16:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.060+0000 7fb0b2ffd640 1 -- 172.21.15.29:0/1610929782 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fb0bc15b320 con 0x7fb0bc15aaa0 2023-10-16T14:16:36.064 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:36.065 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.062+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb0a006aea0 msgr2=0x7fb0a006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.062+0000 7fb0c4452640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb0a006aea0 0x7fb0a006d360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8036000 tx=0x7fb0b803c040 comp rx=0 tx=0).stop 2023-10-16T14:16:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.062+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 msgr2=0x7fb0bc153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.062+0000 7fb0c4452640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc153bc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b410 tx=0x7fb0ac00b8d0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 shutdown_connections 2023-10-16T14:16:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb0a006aea0 0x7fb0a006d360 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0bc15aaa0 0x7fb0bc153bc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0bc15ae80 0x7fb0bc0c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 --2- 172.21.15.29:0/1610929782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0bc0ccaa0 0x7fb0bc15dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 >> 172.21.15.29:0/1610929782 conn(0x7fb0bc0cb410 msgr2=0x7fb0bc0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 shutdown_connections 2023-10-16T14:16:36.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.063+0000 7fb0c4452640 1 -- 172.21.15.29:0/1610929782 wait complete. 2023-10-16T14:16:36.087 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-16T14:16:36.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.309+0000 7febf0a40640 1 Processor -- start 2023-10-16T14:16:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.309+0000 7febf0a40640 1 -- start start 2023-10-16T14:16:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febf0a40640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7febec15cbe0 0x7febec0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febf0a40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7febec0b4c80 0x7febec0b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febf0a40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0d15a0 con 0x7febec15cfc0 2023-10-16T14:16:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febf0a40640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec0d1880 con 0x7febec15cbe0 2023-10-16T14:16:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febf0a40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec0d1b60 con 0x7febec0b4c80 2023-10-16T14:16:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febeb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7febec15cbe0 0x7febec0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febeb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7febec15cbe0 0x7febec0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50676/0 (socket says 172.21.15.29:50676) 2023-10-16T14:16:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.310+0000 7febeb7fe640 1 -- 172.21.15.29:0/2746276653 learned_addr learned my addr 172.21.15.29:0/2746276653 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.311+0000 7febebfff640 1 --2- 172.21.15.29:0/2746276653 >> v2:172.21.15.170:3300/0 conn(0x7febec0b4c80 0x7febec0b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.311+0000 7febeb7fe640 1 -- 172.21.15.29:0/2746276653 >> v2:172.21.15.170:3300/0 conn(0x7febec0b4c80 msgr2=0x7febec0b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.311+0000 7febeb7fe640 1 --2- 172.21.15.29:0/2746276653 >> v2:172.21.15.170:3300/0 conn(0x7febec0b4c80 0x7febec0b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.311+0000 7febeb7fe640 1 -- 172.21.15.29:0/2746276653 >> v1:172.21.15.170:6789/0 conn(0x7febec15cfc0 legacy=0x7febec0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.311+0000 7febeb7fe640 1 -- 172.21.15.29:0/2746276653 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec0d1e40 con 0x7febec15cbe0 2023-10-16T14:16:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.312+0000 7febeb7fe640 1 --2- 172.21.15.29:0/2746276653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cbe0 0x7febec0d0b50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7febec0b5b70 tx=0x7febdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2598611dd7ebb56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.312+0000 7febea7fc640 1 -- 172.21.15.29:0/2746276653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc031870 con 0x7febec15cbe0 2023-10-16T14:16:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febea7fc640 1 -- 172.21.15.29:0/2746276653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc0319d0 con 0x7febec15cbe0 2023-10-16T14:16:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febea7fc640 1 -- 172.21.15.29:0/2746276653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc031d60 con 0x7febec15cbe0 2023-10-16T14:16:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 -- 172.21.15.29:0/2746276653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cbe0 msgr2=0x7febec0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 --2- 172.21.15.29:0/2746276653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cbe0 0x7febec0d0b50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7febec0b5b70 tx=0x7febdc02b910 comp rx=0 tx=0).stop 2023-10-16T14:16:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 -- 172.21.15.29:0/2746276653 shutdown_connections 2023-10-16T14:16:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 --2- 172.21.15.29:0/2746276653 >> v2:172.21.15.170:3300/0 conn(0x7febec0b4c80 0x7febec0b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 --2- 172.21.15.29:0/2746276653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cbe0 0x7febec0d0b50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 -- 172.21.15.29:0/2746276653 >> 172.21.15.29:0/2746276653 conn(0x7febec0cf570 msgr2=0x7febec0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 -- 172.21.15.29:0/2746276653 shutdown_connections 2023-10-16T14:16:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.313+0000 7febf0a40640 1 -- 172.21.15.29:0/2746276653 wait complete. 2023-10-16T14:16:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.315+0000 7febf0a40640 1 Processor -- start 2023-10-16T14:16:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.315+0000 7febf0a40640 1 -- start start 2023-10-16T14:16:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febf0a40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febec0b4c80 0x7febec1f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febf0a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febec15cbe0 0x7febec0c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febf0a40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 0x7febec15f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febf0a40640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec1f22a0 con 0x7febec15cfc0 2023-10-16T14:16:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febf0a40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec15f1d0 con 0x7febec0b4c80 2023-10-16T14:16:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febf0a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febec15e5c0 con 0x7febec15cbe0 2023-10-16T14:16:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 0x7febec15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.316+0000 7febebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 0x7febec15f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50692/0 (socket says 172.21.15.29:50692) 2023-10-16T14:16:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 -- 172.21.15.29:0/4208953068 learned_addr learned my addr 172.21.15.29:0/4208953068 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 -- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febec15cbe0 msgr2=0x7febec0c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febeb7fe640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febec0b4c80 0x7febec1f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febeaffd640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febec15cbe0 0x7febec0c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febec15cbe0 0x7febec0c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 -- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febec0b4c80 msgr2=0x7febec1f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febec0b4c80 0x7febec1f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 -- 172.21.15.29:0/4208953068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0094d0 con 0x7febec15cfc0 2023-10-16T14:16:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febeb7fe640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febec0b4c80 0x7febec1f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febebfff640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 0x7febec15f660 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7febe400d580 tx=0x7febe400da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe40125f0 con 0x7febec15cfc0 2023-10-16T14:16:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe4012770 con 0x7febec15cfc0 2023-10-16T14:16:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.317+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe4012b20 con 0x7febec15cfc0 2023-10-16T14:16:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.318+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec15e8a0 con 0x7febec15cfc0 2023-10-16T14:16:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.318+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febec15ee60 con 0x7febec15cfc0 2023-10-16T14:16:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.319+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febec15f380 con 0x7febec15cfc0 2023-10-16T14:16:36.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.320+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7febe4012c80 con 0x7febec15cfc0 2023-10-16T14:16:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.321+0000 7febe3fff640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7febcc06ae70 0x7febcc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.321+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febe4089b30 con 0x7febec15cfc0 2023-10-16T14:16:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.321+0000 7febeb7fe640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7febcc06ae70 0x7febcc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.322+0000 7febeb7fe640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7febcc06ae70 0x7febcc06d330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7febdc036000 tx=0x7febdc03c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.325+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7febec15f380 con 0x7febec15cfc0 2023-10-16T14:16:36.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.524+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febec15f340 con 0x7febec15cfc0 2023-10-16T14:16:36.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.527+0000 7febe3fff640 1 -- 172.21.15.29:0/4208953068 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7febe4054b50 con 0x7febec15cfc0 2023-10-16T14:16:36.532 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:36.532 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:36.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.530+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7febcc06ae70 msgr2=0x7febcc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.530+0000 7febf0a40640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7febcc06ae70 0x7febcc06d330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7febdc036000 tx=0x7febdc03c040 comp rx=0 tx=0).stop 2023-10-16T14:16:36.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.530+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 msgr2=0x7febec15f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.530+0000 7febf0a40640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 0x7febec15f660 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7febe400d580 tx=0x7febe400da40 comp rx=0 tx=0).stop 2023-10-16T14:16:36.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 shutdown_connections 2023-10-16T14:16:36.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7febcc06ae70 0x7febcc06d330 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febec15cfc0 0x7febec15f660 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7febec15cbe0 0x7febec0c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 --2- 172.21.15.29:0/4208953068 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7febec0b4c80 0x7febec1f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 >> 172.21.15.29:0/4208953068 conn(0x7febec0cf570 msgr2=0x7febec15c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 shutdown_connections 2023-10-16T14:16:36.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.531+0000 7febf0a40640 1 -- 172.21.15.29:0/4208953068 wait complete. 2023-10-16T14:16:36.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-16T14:16:36.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-16T14:16:36.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-16T14:16:36.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-16T14:16:36.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-16T14:16:36.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-16T14:16:36.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-16T14:16:36.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-16T14:16:36.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.864+0000 7f081353c640 1 Processor -- start 2023-10-16T14:16:36.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.865+0000 7f081353c640 1 -- start start 2023-10-16T14:16:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.866+0000 7f081353c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f080c158eb0 0x7f080c06a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.866+0000 7f081353c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f080c06a670 0x7f080c15c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.867+0000 7f081353c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c15d1a0 con 0x7f080c15cdc0 2023-10-16T14:16:36.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.867+0000 7f081353c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c0cc2b0 con 0x7f080c158eb0 2023-10-16T14:16:36.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.867+0000 7f081353c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c0cc590 con 0x7f080c06a670 2023-10-16T14:16:36.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.867+0000 7f08112d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f080c158eb0 0x7f080c06a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.867+0000 7f08112d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f080c158eb0 0x7f080c06a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50702/0 (socket says 172.21.15.29:50702) 2023-10-16T14:16:36.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.867+0000 7f08112d8640 1 -- 172.21.15.29:0/464343378 learned_addr learned my addr 172.21.15.29:0/464343378 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.868+0000 7f08112d8640 1 -- 172.21.15.29:0/464343378 >> v2:172.21.15.170:3300/0 conn(0x7f080c06a670 msgr2=0x7f080c15c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.868+0000 7f08112d8640 1 --2- 172.21.15.29:0/464343378 >> v2:172.21.15.170:3300/0 conn(0x7f080c06a670 0x7f080c15c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.868+0000 7f08112d8640 1 -- 172.21.15.29:0/464343378 >> v1:172.21.15.170:6789/0 conn(0x7f080c15cdc0 legacy=0x7f080c0cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.868+0000 7f08112d8640 1 -- 172.21.15.29:0/464343378 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c0cc870 con 0x7f080c158eb0 2023-10-16T14:16:36.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.870+0000 7f08112d8640 1 --2- 172.21.15.29:0/464343378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c158eb0 0x7f080c06a130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f080c06aa50 tx=0x7f07fc02db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d71e3eeca909689 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.871+0000 7f0803fff640 1 -- 172.21.15.29:0/464343378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc02f8e0 con 0x7f080c158eb0 2023-10-16T14:16:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.871+0000 7f0803fff640 1 -- 172.21.15.29:0/464343378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc02fa40 con 0x7f080c158eb0 2023-10-16T14:16:36.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.871+0000 7f0803fff640 1 -- 172.21.15.29:0/464343378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc02f8e0 con 0x7f080c158eb0 2023-10-16T14:16:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.872+0000 7f081353c640 1 -- 172.21.15.29:0/464343378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c158eb0 msgr2=0x7f080c06a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.872+0000 7f081353c640 1 --2- 172.21.15.29:0/464343378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c158eb0 0x7f080c06a130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f080c06aa50 tx=0x7f07fc02db20 comp rx=0 tx=0).stop 2023-10-16T14:16:36.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.873+0000 7f30d38c2640 1 Processor -- start 2023-10-16T14:16:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.873+0000 7f081353c640 1 -- 172.21.15.29:0/464343378 shutdown_connections 2023-10-16T14:16:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.873+0000 7f30d38c2640 1 -- start start 2023-10-16T14:16:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.873+0000 7f081353c640 1 --2- 172.21.15.29:0/464343378 >> v2:172.21.15.170:3300/0 conn(0x7f080c06a670 0x7f080c15c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.873+0000 7f081353c640 1 --2- 172.21.15.29:0/464343378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c158eb0 0x7f080c06a130 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.873+0000 7f081353c640 1 -- 172.21.15.29:0/464343378 >> 172.21.15.29:0/464343378 conn(0x7f080c0ce6e0 msgr2=0x7f080c0ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.874+0000 7f081353c640 1 -- 172.21.15.29:0/464343378 shutdown_connections 2023-10-16T14:16:36.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.874+0000 7f081353c640 1 -- 172.21.15.29:0/464343378 wait complete. 2023-10-16T14:16:36.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.876+0000 7f30d38c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30cc159110 0x7f30cc15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.876+0000 7f30d38c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d38c2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f30cc06a2e0 0x7f30cc06b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d165e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30cc159110 0x7f30cc15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d165e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30cc159110 0x7f30cc15cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50712/0 (socket says 172.21.15.29:50712) 2023-10-16T14:16:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d38c2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc06a770 con 0x7f30cc159110 2023-10-16T14:16:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d38c2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc1594f0 con 0x7f30cc06a2e0 2023-10-16T14:16:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d38c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc15d130 con 0x7f30cc0ade50 2023-10-16T14:16:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.877+0000 7f30d165e640 1 -- 172.21.15.29:0/3999657155 learned_addr learned my addr 172.21.15.29:0/3999657155 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.878+0000 7f081353c640 1 Processor -- start 2023-10-16T14:16:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.878+0000 7f30d0e5d640 1 --2- 172.21.15.29:0/3999657155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.878+0000 7f30d1e5f640 1 --2- 172.21.15.29:0/3999657155 >> v2:172.21.15.170:3300/0 conn(0x7f30cc06a2e0 0x7f30cc06b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.879+0000 7f30d165e640 1 -- 172.21.15.29:0/3999657155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 msgr2=0x7f30cc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.879+0000 7f30d165e640 1 --2- 172.21.15.29:0/3999657155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.879+0000 7f30d165e640 1 -- 172.21.15.29:0/3999657155 >> v2:172.21.15.170:3300/0 conn(0x7f30cc06a2e0 msgr2=0x7f30cc06b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.879+0000 7f30d165e640 1 --2- 172.21.15.29:0/3999657155 >> v2:172.21.15.170:3300/0 conn(0x7f30cc06a2e0 0x7f30cc06b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.879+0000 7f30d165e640 1 -- 172.21.15.29:0/3999657155 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc15d400 con 0x7f30cc159110 2023-10-16T14:16:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f0fde11a640 1 Processor -- start 2023-10-16T14:16:36.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.879+0000 7f081353c640 1 -- start start 2023-10-16T14:16:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f081353c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f080c06a670 0x7f080c153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f081353c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c15cdc0 0x7f080c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f081353c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c06c480 0x7f080c154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f081353c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c156340 con 0x7f080c06c480 2023-10-16T14:16:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f081353c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c154210 con 0x7f080c15cdc0 2023-10-16T14:16:36.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f081353c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c06c910 con 0x7f080c06a670 2023-10-16T14:16:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.880+0000 7f30d165e640 1 --2- 172.21.15.29:0/3999657155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc159110 0x7f30cc15cbf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f30cc15cb30 tx=0x7f30c802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665e574addf20c48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.881+0000 7f0810ad7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c15cdc0 0x7f080c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.881+0000 7f0810ad7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c15cdc0 0x7f080c06bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44358/0 (socket says 172.21.15.29:44358) 2023-10-16T14:16:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.881+0000 7f0810ad7640 1 -- 172.21.15.29:0/94185299 learned_addr learned my addr 172.21.15.29:0/94185299 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f30c3fff640 1 -- 172.21.15.29:0/3999657155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8030800 con 0x7f30cc159110 2023-10-16T14:16:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f30c3fff640 1 -- 172.21.15.29:0/3999657155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c8030960 con 0x7f30cc159110 2023-10-16T14:16:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f30c3fff640 1 -- 172.21.15.29:0/3999657155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8030cd0 con 0x7f30cc159110 2023-10-16T14:16:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f08112d8640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f080c06a670 0x7f080c153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f0810ad7640 1 -- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f080c06a670 msgr2=0x7f080c153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f0810ad7640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f080c06a670 0x7f080c153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f0810ad7640 1 -- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c06c480 msgr2=0x7f080c154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f0810ad7640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c06c480 0x7f080c154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.882+0000 7f0810ad7640 1 -- 172.21.15.29:0/94185299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0094d0 con 0x7f080c15cdc0 2023-10-16T14:16:36.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.881+0000 7f0fde11a640 1 -- start start 2023-10-16T14:16:36.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f0fde11a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fd80b8480 0x7f0fd80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f0fde11a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f0fde11a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd815d150 con 0x7f0fd815cd70 2023-10-16T14:16:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f0fde11a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80d1180 con 0x7f0fd80b8480 2023-10-16T14:16:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f0fde11a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80d1460 con 0x7f0fd80d0be0 2023-10-16T14:16:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f30d38c2640 1 -- 172.21.15.29:0/3999657155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc159110 msgr2=0x7f30cc15cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.883+0000 7f30d38c2640 1 --2- 172.21.15.29:0/3999657155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc159110 0x7f30cc15cbf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f30cc15cb30 tx=0x7f30c802b940 comp rx=0 tx=0).stop 2023-10-16T14:16:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0fdd118640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f0fd815cd70 0x7f0fd80d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:48380/0 (socket says 172.21.15.29:48380) 2023-10-16T14:16:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0fdd118640 1 -- 172.21.15.29:0/4133755952 learned_addr learned my addr 172.21.15.29:0/4133755952 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0810ad7640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c15cdc0 0x7f080c06bf40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f080800b4a0 tx=0x7f080800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0fdd919640 1 --2- 172.21.15.29:0/4133755952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f30d38c2640 1 -- 172.21.15.29:0/3999657155 shutdown_connections 2023-10-16T14:16:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f30d38c2640 1 --2- 172.21.15.29:0/3999657155 >> v2:172.21.15.170:3300/0 conn(0x7f30cc06a2e0 0x7f30cc06b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f30d38c2640 1 --2- 172.21.15.29:0/3999657155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f30d38c2640 1 --2- 172.21.15.29:0/3999657155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc159110 0x7f30cc15cbf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f30d38c2640 1 -- 172.21.15.29:0/3999657155 >> 172.21.15.29:0/3999657155 conn(0x7f30cc0ce9b0 msgr2=0x7f30cc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0fdc917640 1 --2- 172.21.15.29:0/4133755952 >> v2:172.21.15.29:3300/0 conn(0x7f0fd80b8480 0x7f0fd80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f30d38c2640 1 -- 172.21.15.29:0/3999657155 shutdown_connections 2023-10-16T14:16:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808010610 con 0x7f080c15cdc0 2023-10-16T14:16:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.884+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808010770 con 0x7f080c15cdc0 2023-10-16T14:16:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.885+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808010610 con 0x7f080c15cdc0 2023-10-16T14:16:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.885+0000 7f0fcffff640 1 -- 172.21.15.29:0/4133755952 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150281908 0 0) 0x7f0fd815d150 con 0x7f0fd815cd70 2023-10-16T14:16:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.885+0000 7f0fcffff640 1 -- 172.21.15.29:0/4133755952 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc4003520 con 0x7f0fd815cd70 2023-10-16T14:16:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.885+0000 7f30d38c2640 1 -- 172.21.15.29:0/3999657155 wait complete. 2023-10-16T14:16:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c1556c0 con 0x7f080c15cdc0 2023-10-16T14:16:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f0fcffff640 1 -- 172.21.15.29:0/4133755952 <== mon.1 v1:172.21.15.170:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070153860 0 0) 0x7f0fc4003520 con 0x7f0fd815cd70 2023-10-16T14:16:36.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c155c80 con 0x7f080c15cdc0 2023-10-16T14:16:36.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f0fdd919640 1 -- 172.21.15.29:0/4133755952 >> v2:172.21.15.29:3300/0 conn(0x7f0fd80b8480 msgr2=0x7f0fd80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f0fdd919640 1 --2- 172.21.15.29:0/4133755952 >> v2:172.21.15.29:3300/0 conn(0x7f0fd80b8480 0x7f0fd80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f0fdd919640 1 -- 172.21.15.29:0/4133755952 >> v1:172.21.15.170:6789/0 conn(0x7f0fd815cd70 legacy=0x7f0fd80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.886+0000 7f0fdd919640 1 -- 172.21.15.29:0/4133755952 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80d1740 con 0x7f0fd80d0be0 2023-10-16T14:16:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.887+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f080c0cce30 con 0x7f080c15cdc0 2023-10-16T14:16:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.888+0000 7f30d38c2640 1 Processor -- start 2023-10-16T14:16:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.888+0000 7f30d38c2640 1 -- start start 2023-10-16T14:16:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.889+0000 7f0fdd919640 1 --2- 172.21.15.29:0/4133755952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd80cdd00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009d90 tx=0x7f0fd002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b14482674c8b6b71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.890+0000 7f0fcffff640 1 -- 172.21.15.29:0/4133755952 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4001650 con 0x7f0fd80d0be0 2023-10-16T14:16:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.890+0000 7f0fcffff640 1 -- 172.21.15.29:0/4133755952 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd002c8a0 con 0x7f0fd80d0be0 2023-10-16T14:16:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.890+0000 7f0fcffff640 1 -- 172.21.15.29:0/4133755952 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4001650 con 0x7f0fd80d0be0 2023-10-16T14:16:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0fde11a640 1 -- 172.21.15.29:0/4133755952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 msgr2=0x7f0fd80cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0fde11a640 1 --2- 172.21.15.29:0/4133755952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd80cdd00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009d90 tx=0x7f0fd002bd70 comp rx=0 tx=0).stop 2023-10-16T14:16:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0fde11a640 1 -- 172.21.15.29:0/4133755952 shutdown_connections 2023-10-16T14:16:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0fde11a640 1 --2- 172.21.15.29:0/4133755952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd80cdd00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0fde11a640 1 --2- 172.21.15.29:0/4133755952 >> v2:172.21.15.29:3300/0 conn(0x7f0fd80b8480 0x7f0fd80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f30d38c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc06a2e0 0x7f30cc155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f30d38c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc15c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f30d38c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc159110 0x7f30cc159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f30d38c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc155d80 con 0x7f30cc06a2e0 2023-10-16T14:16:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f30d38c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0b2fe0 con 0x7f30cc159110 2023-10-16T14:16:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f30d38c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0b3b60 con 0x7f30cc0ade50 2023-10-16T14:16:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0fde11a640 1 -- 172.21.15.29:0/4133755952 >> 172.21.15.29:0/4133755952 conn(0x7f0fd80cf570 msgr2=0x7f0fd80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d165e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc06a2e0 0x7f30cc155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d0e5d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc15c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d0e5d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc15c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59214/0 (socket says 172.21.15.29:59214) 2023-10-16T14:16:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d1e5f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc159110 0x7f30cc159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d1e5f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc159110 0x7f30cc159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44374/0 (socket says 172.21.15.29:44374) 2023-10-16T14:16:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d1e5f640 1 -- 172.21.15.29:0/442770525 learned_addr learned my addr 172.21.15.29:0/442770525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f30d165e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc06a2e0 0x7f30cc155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50744/0 (socket says 172.21.15.29:50744) 2023-10-16T14:16:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f0fde11a640 1 -- 172.21.15.29:0/4133755952 shutdown_connections 2023-10-16T14:16:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.891+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f08080108e0 con 0x7f080c15cdc0 2023-10-16T14:16:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f0801ffb640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f07f006af00 0x7f07f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.892+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0808088d80 con 0x7f080c15cdc0 2023-10-16T14:16:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f0801ffb640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7f07f00730a0 0x7f07f00754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 --> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07f0075b70 con 0x7f07f00730a0 2023-10-16T14:16:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f080808b3e0 con 0x7f080c15cdc0 2023-10-16T14:16:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f30d165e640 1 -- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 msgr2=0x7f30cc15c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f0fde11a640 1 -- 172.21.15.29:0/4133755952 wait complete. 2023-10-16T14:16:36.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f30d165e640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc15c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f30d165e640 1 -- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc159110 msgr2=0x7f30cc159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f30d165e640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc159110 0x7f30cc159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f30d165e640 1 -- 172.21.15.29:0/442770525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80094d0 con 0x7f30cc06a2e0 2023-10-16T14:16:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.894+0000 7f30d0e5d640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc0ade50 0x7f30cc15c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:36.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.893+0000 7f08112d8640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f07f006af00 0x7f07f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.894+0000 7f0811ad9640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7f07f00730a0 0x7f07f00754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.894+0000 7f30d165e640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc06a2e0 0x7f30cc155840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f30c8033000 tx=0x7f30c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.895+0000 7f0811ad9640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7f07f00730a0 0x7f07f00754c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f0fde11a640 1 Processor -- start 2023-10-16T14:16:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c803e070 con 0x7f30cc06a2e0 2023-10-16T14:16:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c80043f0 con 0x7f30cc06a2e0 2023-10-16T14:16:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== osd.1 v2:172.21.15.29:6800/3296543673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f07f0075b70 con 0x7f07f00730a0 2023-10-16T14:16:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8003810 con 0x7f30cc06a2e0 2023-10-16T14:16:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f30d38c2640 1 -- 172.21.15.29:0/442770525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc0b3230 con 0x7f30cc06a2e0 2023-10-16T14:16:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.896+0000 7f30d38c2640 1 -- 172.21.15.29:0/442770525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc0b37f0 con 0x7f30cc06a2e0 2023-10-16T14:16:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.898+0000 7f08112d8640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f07f006af00 0x7f07f006d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f07fc038000 tx=0x7f07fc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.898+0000 7f0fde11a640 1 -- start start 2023-10-16T14:16:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.898+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30a4000fc0 con 0x7f30cc06a2e0 2023-10-16T14:16:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.900+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f30c8042070 con 0x7f30cc06a2e0 2023-10-16T14:16:36.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f0fde11a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd80b8480 0x7f0fd815fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f0fde11a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f0fde11a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd815cd70 0x7f0fd815d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f0fde11a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80b2f90 con 0x7f0fd815cd70 2023-10-16T14:16:36.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f0fde11a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd815e580 con 0x7f0fd80b8480 2023-10-16T14:16:36.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f0fde11a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80cb9c0 con 0x7f0fd80d0be0 2023-10-16T14:16:36.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0442c7c640 1 Processor -- start 2023-10-16T14:16:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.901+0000 7f8ac8e4f640 1 Processor -- start 2023-10-16T14:16:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0fdd919640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd815cd70 0x7f0fd815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0fdd919640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd815cd70 0x7f0fd815d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50754/0 (socket says 172.21.15.29:50754) 2023-10-16T14:16:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0fdd919640 1 -- 172.21.15.29:0/210473300 learned_addr learned my addr 172.21.15.29:0/210473300 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0fdd118640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd80b8480 0x7f0fd815fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0fdc917640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f8ac8e4f640 1 -- start start 2023-10-16T14:16:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f30c1ffb640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f30b006aec0 0x7f30b006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30c80ab380 con 0x7f30cc06a2e0 2023-10-16T14:16:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f30c1ffb640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7f30b00730e0 0x7f30b0075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 --> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30b0075bb0 con 0x7f30b00730e0 2023-10-16T14:16:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30c80ad960 con 0x7f30cc06a2e0 2023-10-16T14:16:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f30d1e5f640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7f30b00730e0 0x7f30b0075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f0fdd919640 1 -- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 msgr2=0x7f0fd815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f0fdd919640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd80d0be0 0x7f0fd815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f0fdd919640 1 -- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd80b8480 msgr2=0x7f0fd815fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f0fdd919640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd80b8480 0x7f0fd815fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f0fdd919640 1 -- 172.21.15.29:0/210473300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd0009a70 con 0x7f0fd815cd70 2023-10-16T14:16:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.903+0000 7f0fdd919640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd815cd70 0x7f0fd815d880 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009eb0 tx=0x7f0fd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4001650 con 0x7f0fd815cd70 2023-10-16T14:16:36.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f728f7e9640 1 Processor -- start 2023-10-16T14:16:36.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f8ac8e4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac41625d0 0x7f8ac40cac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f8ac8e4f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac406caa0 con 0x7f8ac40b0fc0 2023-10-16T14:16:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f8ac8e4f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac41629b0 con 0x7f8ac4071a80 2023-10-16T14:16:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f8ac8e4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac406ccf0 con 0x7f8ac41625d0 2023-10-16T14:16:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f8ac2d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8ac40b0fc0 0x7f8ac41650a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46320/0 (socket says 172.21.15.29:46320) 2023-10-16T14:16:36.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f8ac2d9d640 1 -- 172.21.15.29:0/2363943808 learned_addr learned my addr 172.21.15.29:0/2363943808 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f728f7e9640 1 -- start start 2023-10-16T14:16:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f728f7e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72880cb500 0x7f72880ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f728f7e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728806f860 con 0x7f728806f3f0 2023-10-16T14:16:36.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f728f7e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72880caba0 con 0x7f72880cb8e0 2023-10-16T14:16:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.904+0000 7f728f7e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72880cae80 con 0x7f72880cb500 2023-10-16T14:16:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f728e587640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f728806f3f0 0x7f7288160af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46304/0 (socket says 172.21.15.29:46304) 2023-10-16T14:16:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f728e587640 1 -- 172.21.15.29:0/1962557869 learned_addr learned my addr 172.21.15.29:0/1962557869 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f8ac259c640 1 --2- 172.21.15.29:0/2363943808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac41625d0 0x7f8ac40cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.902+0000 7f0442c7c640 1 -- start start 2023-10-16T14:16:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f728d585640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72880cb500 0x7f72880ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd0009450 con 0x7f0fd815cd70 2023-10-16T14:16:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd003e410 con 0x7f0fd815cd70 2023-10-16T14:16:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd80cbca0 con 0x7f0fd815cd70 2023-10-16T14:16:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd80cc260 con 0x7f0fd815cd70 2023-10-16T14:16:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f0442c7c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f043c06a4b0 0x7f043c0ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f0442c7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c06cf50 con 0x7f043c0cf4d0 2023-10-16T14:16:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f0442c7c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c15ccf0 con 0x7f043c0ce1c0 2023-10-16T14:16:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f0442c7c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c15d970 con 0x7f043c06a4b0 2023-10-16T14:16:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553538214 0 0) 0x7f728806f860 con 0x7f728806f3f0 2023-10-16T14:16:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.905+0000 7f30d1e5f640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7f30b00730e0 0x7f30b0075500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f30d0e5d640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f30b006aec0 0x7f30b006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.906+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7274003520 con 0x7f728806f3f0 2023-10-16T14:16:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712929221 0 0) 0x7f8ac41629b0 con 0x7f8ac4071a80 2023-10-16T14:16:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab0003520 con 0x7f8ac4071a80 2023-10-16T14:16:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617393616 0 0) 0x7f8ac406caa0 con 0x7f8ac40b0fc0 2023-10-16T14:16:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac41629b0 con 0x7f8ac40b0fc0 2023-10-16T14:16:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f0440a18640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f043c0cf4d0 0x7f043c158ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46330/0 (socket says 172.21.15.29:46330) 2023-10-16T14:16:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f0440a18640 1 -- 172.21.15.29:0/2770934974 learned_addr learned my addr 172.21.15.29:0/2770934974 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7efd6ba10640 1 Processor -- start 2023-10-16T14:16:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2542856267 0 0) 0x7f7274003520 con 0x7f728806f3f0 2023-10-16T14:16:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72880cb500 msgr2=0x7f72880ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== osd.0 v2:172.21.15.29:6808/1553131287 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f30b0075bb0 con 0x7f30b00730e0 2023-10-16T14:16:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7efd6ba10640 1 -- start start 2023-10-16T14:16:36.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f0441219640 1 --2- 172.21.15.29:0/2770934974 >> v2:172.21.15.170:3300/0 conn(0x7f043c06a4b0 0x7f043c0ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72880cb500 0x7f72880ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 >> v1:172.21.15.170:6789/0 conn(0x7f72880cb8e0 legacy=0x7f72880cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72880cc5d0 con 0x7f728806f3f0 2023-10-16T14:16:36.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7280001890 con 0x7f728806f3f0 2023-10-16T14:16:36.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 >> v1:172.21.15.29:6789/0 conn(0x7f728806f3f0 legacy=0x7f7288160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72740039a0 0x7f7274003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72740047c0 0x7f7274004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7274005180 0x7f727400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72880cae80 con 0x7f7274005180 2023-10-16T14:16:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727400c050 con 0x7f72740047c0 2023-10-16T14:16:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac259c640 1 -- 172.21.15.29:0/2363943808 >> v1:172.21.15.170:6789/0 conn(0x7f8ac4071a80 legacy=0x7f8ac40b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac259c640 1 -- 172.21.15.29:0/2363943808 >> v1:172.21.15.29:6789/0 conn(0x7f8ac40b0fc0 legacy=0x7f8ac41650a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.907+0000 7f8ac259c640 1 -- 172.21.15.29:0/2363943808 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac406cfd0 con 0x7f8ac41625d0 2023-10-16T14:16:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.908+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727400c210 con 0x7f72740039a0 2023-10-16T14:16:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.909+0000 7f8ac259c640 1 --2- 172.21.15.29:0/2363943808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac41625d0 0x7f8ac40cac60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0097d0 tx=0x7f8aac02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dfe9712b2e72762 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.909+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0fd003b050 con 0x7f0fd815cd70 2023-10-16T14:16:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.909+0000 7f728e587640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7274005180 0x7f727400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.909+0000 7f728cd84640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72740047c0 0x7f7274004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.909+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564997713 0 0) 0x7f043c06cf50 con 0x7f043c0cf4d0 2023-10-16T14:16:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042c003520 con 0x7f043c0cf4d0 2023-10-16T14:16:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459842493 0 0) 0x7f042c003520 con 0x7f043c0cf4d0 2023-10-16T14:16:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 >> v2:172.21.15.170:3300/0 conn(0x7f043c06a4b0 msgr2=0x7f043c0ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f043b7fe640 1 --2- 172.21.15.29:0/2770934974 >> v2:172.21.15.170:3300/0 conn(0x7f043c06a4b0 0x7f043c0ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 >> v1:172.21.15.170:6789/0 conn(0x7f043c0ce1c0 legacy=0x7f043c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7efd6ba10640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd640b3420 0x7efd640b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f728e587640 1 -- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72740039a0 msgr2=0x7f7274003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f728e587640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72740039a0 0x7f7274003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f728e587640 1 -- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72740047c0 msgr2=0x7f7274004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f728e587640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72740047c0 0x7f7274004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7f728e587640 1 -- 172.21.15.29:0/1962557869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72880cc5d0 con 0x7f7274005180 2023-10-16T14:16:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7efd6ba10640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640d3980 con 0x7efd640cf710 2023-10-16T14:16:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7efd6ba10640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640ce630 con 0x7efd640cfaf0 2023-10-16T14:16:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7efd6ba10640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd640d3c60 con 0x7efd640b3420 2023-10-16T14:16:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7efd697ac640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efd640cf710 0x7efd640b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46334/0 (socket says 172.21.15.29:46334) 2023-10-16T14:16:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.910+0000 7efd697ac640 1 -- 172.21.15.29:0/1012769802 learned_addr learned my addr 172.21.15.29:0/1012769802 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.911+0000 7efd68fab640 1 --2- 172.21.15.29:0/1012769802 >> v2:172.21.15.29:3300/0 conn(0x7efd640b3420 0x7efd640b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.911+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296390928 0 0) 0x7efd640d3980 con 0x7efd640cf710 2023-10-16T14:16:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.911+0000 7f30d0e5d640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f30b006aec0 0x7f30b006d380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f30cc159950 tx=0x7f30bc00c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.911+0000 7f728e587640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7274005180 0x7f727400b980 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f728001b710 tx=0x7f72740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00218df735850d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.911+0000 7f0fcdffb640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0fbc06aec0 0x7f0fbc06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac031770 con 0x7f8ac41625d0 2023-10-16T14:16:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.911+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c003520 con 0x7efd640cf710 2023-10-16T14:16:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443823737 0 0) 0x7efd640ce630 con 0x7efd640cfaf0 2023-10-16T14:16:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640d3980 con 0x7efd640cfaf0 2023-10-16T14:16:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c15cfd0 con 0x7f043c0cf4d0 2023-10-16T14:16:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0430001670 con 0x7f043c0cf4d0 2023-10-16T14:16:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 >> v1:172.21.15.29:6789/0 conn(0x7f043c0cf4d0 legacy=0x7f043c158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0039c0 0x7f042c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042c0047c0 0x7f042c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7efd68fab640 1 -- 172.21.15.29:0/1012769802 >> v1:172.21.15.170:6789/0 conn(0x7efd640cfaf0 legacy=0x7efd640b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7efd68fab640 1 -- 172.21.15.29:0/1012769802 >> v1:172.21.15.29:6789/0 conn(0x7efd640cf710 legacy=0x7efd640b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f042c0053d0 0x7f042c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c15d970 con 0x7f042c0047c0 2023-10-16T14:16:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c00c310 con 0x7f042c0039c0 2023-10-16T14:16:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c00c4d0 con 0x7f042c0053d0 2023-10-16T14:16:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7efd68fab640 1 -- 172.21.15.29:0/1012769802 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640ce910 con 0x7efd640b3420 2023-10-16T14:16:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac0318d0 con 0x7f8ac41625d0 2023-10-16T14:16:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f8ac1d9b640 1 -- 172.21.15.29:0/2363943808 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac031c40 con 0x7f8ac41625d0 2023-10-16T14:16:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0fd80d1d00 con 0x7f0fd815cd70 2023-10-16T14:16:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f0fdd118640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0fbc06aec0 0x7f0fbc06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280046640 con 0x7f7274005180 2023-10-16T14:16:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fd00ab120 con 0x7f0fd815cd70 2023-10-16T14:16:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.912+0000 7f727ffff640 1 -- 172.21.15.29:0/1962557869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7280040070 con 0x7f7274005180 2023-10-16T14:16:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f728f7e9640 1 -- 172.21.15.29:0/1962557869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7274005180 msgr2=0x7f727400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f728f7e9640 1 --2- 172.21.15.29:0/1962557869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7274005180 0x7f727400b980 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f728001b710 tx=0x7f72740057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/2363943808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac41625d0 msgr2=0x7f8ac40cac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f8ac8e4f640 1 --2- 172.21.15.29:0/2363943808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac41625d0 0x7f8ac40cac60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0097d0 tx=0x7f8aac02b810 comp rx=0 tx=0).stop 2023-10-16T14:16:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/2363943808 shutdown_connections 2023-10-16T14:16:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f8ac8e4f640 1 --2- 172.21.15.29:0/2363943808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac41625d0 0x7f8ac40cac60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/2363943808 >> 172.21.15.29:0/2363943808 conn(0x7f8ac40cb8b0 msgr2=0x7f8ac40cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/2363943808 shutdown_connections 2023-10-16T14:16:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f043bfff640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042c0047c0 0x7f042c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f0440a18640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0039c0 0x7f042c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f0441219640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f042c0053d0 0x7f042c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f043bfff640 1 -- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f042c0053d0 msgr2=0x7f042c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/2363943808 wait complete. 2023-10-16T14:16:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f728e587640 1 -- 172.21.15.29:0/1962557869 reap_dead start 2023-10-16T14:16:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7efd68fab640 1 --2- 172.21.15.29:0/1012769802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640b3420 0x7efd640b5cc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bb6ff6467be7f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f728f7e9640 1 -- 172.21.15.29:0/1962557869 shutdown_connections 2023-10-16T14:16:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f728f7e9640 1 -- 172.21.15.29:0/1962557869 >> 172.21.15.29:0/1962557869 conn(0x7f72880b0570 msgr2=0x7f72880b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f728f7e9640 1 -- 172.21.15.29:0/1962557869 shutdown_connections 2023-10-16T14:16:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.913+0000 7f0fcdffb640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7f0fbc073060 0x7f0fbc075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 --> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fbc075b30 con 0x7f0fbc073060 2023-10-16T14:16:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fd0079d80 con 0x7f0fd815cd70 2023-10-16T14:16:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5402cc80 con 0x7efd640b3420 2023-10-16T14:16:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5402cde0 con 0x7efd640b3420 2023-10-16T14:16:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7efd63fff640 1 -- 172.21.15.29:0/1012769802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54034660 con 0x7efd640b3420 2023-10-16T14:16:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.915+0000 7f0fdc917640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7f0fbc073060 0x7f0fbc075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f043bfff640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f042c0053d0 0x7f042c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f043bfff640 1 -- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0039c0 msgr2=0x7f042c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f043bfff640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0039c0 0x7f042c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.915+0000 7efd6ba10640 1 -- 172.21.15.29:0/1012769802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640b3420 msgr2=0x7efd640b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.915+0000 7efd6ba10640 1 --2- 172.21.15.29:0/1012769802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640b3420 0x7efd640b5cc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd5402bed0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.915+0000 7f0fdd118640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0fbc06aec0 0x7f0fbc06d380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4001eb0 tx=0x7f0fd4001c40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.915+0000 7f728f7e9640 1 -- 172.21.15.29:0/1962557869 wait complete. 2023-10-16T14:16:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f043bfff640 1 -- 172.21.15.29:0/2770934974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c003520 con 0x7f042c0047c0 2023-10-16T14:16:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f0440a18640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0039c0 0x7f042c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f0441219640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f042c0053d0 0x7f042c00ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.914+0000 7f0441219640 1 -- 172.21.15.29:0/2770934974 reap_dead start 2023-10-16T14:16:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f043bfff640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042c0047c0 0x7f042c004c40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0428003db0 tx=0x7f042c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cb8ec3d4e8ff86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f0fdc917640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7f0fbc073060 0x7f0fbc075480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7efd6ba10640 1 -- 172.21.15.29:0/1012769802 shutdown_connections 2023-10-16T14:16:36.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7efd6ba10640 1 --2- 172.21.15.29:0/1012769802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640b3420 0x7efd640b5cc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7efd6ba10640 1 -- 172.21.15.29:0/1012769802 >> 172.21.15.29:0/1012769802 conn(0x7efd640d2cc0 msgr2=0x7efd640d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f728f7e9640 1 Processor -- start 2023-10-16T14:16:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f728f7e9640 1 -- start start 2023-10-16T14:16:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f8ac8e4f640 1 Processor -- start 2023-10-16T14:16:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428032600 con 0x7f042c0047c0 2023-10-16T14:16:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042802e050 con 0x7f042c0047c0 2023-10-16T14:16:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f043b7fe640 1 -- 172.21.15.29:0/2770934974 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04280329e0 con 0x7f042c0047c0 2023-10-16T14:16:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7efd6ba10640 1 -- 172.21.15.29:0/1012769802 shutdown_connections 2023-10-16T14:16:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.916+0000 7f8ac8e4f640 1 -- start start 2023-10-16T14:16:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f8ac8e4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac4071a80 0x7f8ac41f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f8ac8e4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac40b0fc0 0x7f8ac415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f8ac8e4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ac41625d0 0x7f8ac4160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f8ac8e4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40c6e60 con 0x7f8ac41625d0 2023-10-16T14:16:36.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f8ac8e4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40c3440 con 0x7f8ac4071a80 2023-10-16T14:16:36.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f8ac8e4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40c2aa0 con 0x7f8ac40b0fc0 2023-10-16T14:16:36.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== osd.2 v2:172.21.15.29:6809/4164234701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0fbc075b30 con 0x7f0fbc073060 2023-10-16T14:16:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f8ac2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ac41625d0 0x7f8ac4160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f8ac2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ac41625d0 0x7f8ac4160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50806/0 (socket says 172.21.15.29:50806) 2023-10-16T14:16:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.918+0000 7f8ac2d9d640 1 -- 172.21.15.29:0/3066558983 learned_addr learned my addr 172.21.15.29:0/3066558983 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f0442c7c640 1 -- 172.21.15.29:0/2770934974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042c0047c0 msgr2=0x7f042c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7f0442c7c640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042c0047c0 0x7f042c004c40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0428003db0 tx=0x7f042c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.917+0000 7efd6ba10640 1 -- 172.21.15.29:0/1012769802 wait complete. 2023-10-16T14:16:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7efd6ba10640 1 Processor -- start 2023-10-16T14:16:36.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f728f7e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7274005180 0x7f7288155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f728f7e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72881472c0 0x7f728814b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f728f7e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f728814bef0 0x7f7288148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f8ab9d9b640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac40b0fc0 0x7f8ac415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f728f7e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288155740 con 0x7f728814bef0 2023-10-16T14:16:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f728f7e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288155ff0 con 0x7f7274005180 2023-10-16T14:16:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7f728f7e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7288154d70 con 0x7f72881472c0 2023-10-16T14:16:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.919+0000 7efd6ba10640 1 -- start start 2023-10-16T14:16:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ac2d9d640 1 -- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac40b0fc0 msgr2=0x7f8ac415fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ac2d9d640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac40b0fc0 0x7f8ac415fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ac2d9d640 1 -- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac4071a80 msgr2=0x7f8ac41f66d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:16:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ac2d9d640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ac4071a80 0x7f8ac41f66d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ac2d9d640 1 -- 172.21.15.29:0/3066558983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac009400 con 0x7f8ac41625d0 2023-10-16T14:16:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ab9d9b640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ac40b0fc0 0x7f8ac415fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:36.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f728e587640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f728814bef0 0x7f7288148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f728e587640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f728814bef0 0x7f7288148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50822/0 (socket says 172.21.15.29:50822) 2023-10-16T14:16:36.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f728e587640 1 -- 172.21.15.29:0/282812685 learned_addr learned my addr 172.21.15.29:0/282812685 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f0442c7c640 1 -- 172.21.15.29:0/2770934974 shutdown_connections 2023-10-16T14:16:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f0442c7c640 1 --2- 172.21.15.29:0/2770934974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042c0047c0 0x7f042c004c40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f0442c7c640 1 -- 172.21.15.29:0/2770934974 >> 172.21.15.29:0/2770934974 conn(0x7f043c0ce9b0 msgr2=0x7f043c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f728d585640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7274005180 0x7f7288155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.920+0000 7f8ac2d9d640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ac41625d0 0x7f8ac4160260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00c7a0 tx=0x7f8abc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc00a640 con 0x7f8ac41625d0 2023-10-16T14:16:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f0442c7c640 1 -- 172.21.15.29:0/2770934974 shutdown_connections 2023-10-16T14:16:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7efd6ba10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd640b3420 0x7efd641f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f0442c7c640 1 -- 172.21.15.29:0/2770934974 wait complete. 2023-10-16T14:16:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f728e587640 1 -- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72881472c0 msgr2=0x7f728814b9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:16:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f728e587640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72881472c0 0x7f728814b9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f728e587640 1 -- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7274005180 msgr2=0x7f7288155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f728e587640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7274005180 0x7f7288155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7f728e587640 1 -- 172.21.15.29:0/282812685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7288156240 con 0x7f728814bef0 2023-10-16T14:16:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7efd6ba10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd640cf710 0x7efd640cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.921+0000 7efd6ba10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640cfaf0 0x7efd640cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd6ba10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd640caea0 con 0x7efd640cfaf0 2023-10-16T14:16:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd6ba10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd640ca290 con 0x7efd640b3420 2023-10-16T14:16:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd6ba10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd640ca570 con 0x7efd640cf710 2023-10-16T14:16:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7f728e587640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f728814bef0 0x7f7288148d40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f728003da30 tx=0x7f7280001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd68fab640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd640cf710 0x7efd640cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd68fab640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd640cf710 0x7efd640cb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59290/0 (socket says 172.21.15.29:59290) 2023-10-16T14:16:36.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd68fab640 1 -- 172.21.15.29:0/2658480532 learned_addr learned my addr 172.21.15.29:0/2658480532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728004a030 con 0x7f728814bef0 2023-10-16T14:16:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7280046650 con 0x7f728814bef0 2023-10-16T14:16:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728004a030 con 0x7f728814bef0 2023-10-16T14:16:36.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.922+0000 7efd69fad640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640cfaf0 0x7efd640cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd68fab640 1 -- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd640b3420 msgr2=0x7efd641f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd697ac640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd640b3420 0x7efd641f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd68fab640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd640b3420 0x7efd641f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd68fab640 1 -- 172.21.15.29:0/2658480532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640cfaf0 msgr2=0x7efd640cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd68fab640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640cfaf0 0x7efd640cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd68fab640 1 -- 172.21.15.29:0/2658480532 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54009a70 con 0x7efd640cf710 2023-10-16T14:16:36.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7efd69fad640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd640cfaf0 0x7efd640cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:36.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.923+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7288156520 con 0x7f728814bef0 2023-10-16T14:16:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7efd68fab640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd640cf710 0x7efd640cb330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd54004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5403e610 con 0x7efd640cf710 2023-10-16T14:16:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd640ca850 con 0x7efd640cf710 2023-10-16T14:16:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.925+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6415f4c0 con 0x7efd640cf710 2023-10-16T14:16:36.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.925+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54035e50 con 0x7efd640cf710 2023-10-16T14:16:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.925+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5403e610 con 0x7efd640cf710 2023-10-16T14:16:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc00a7a0 con 0x7f8ac41625d0 2023-10-16T14:16:36.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc00ab50 con 0x7f8ac41625d0 2023-10-16T14:16:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac40c2d80 con 0x7f8ac41625d0 2023-10-16T14:16:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.924+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac41f6c10 con 0x7f8ac41625d0 2023-10-16T14:16:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.925+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ac406d590 con 0x7f8ac41625d0 2023-10-16T14:16:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.925+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728814a170 con 0x7f728814bef0 2023-10-16T14:16:36.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.927+0000 7f0442c7c640 1 Processor -- start 2023-10-16T14:16:36.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.928+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd44000fc0 con 0x7efd640cf710 2023-10-16T14:16:36.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.927+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f72800467e0 con 0x7f728814bef0 2023-10-16T14:16:36.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.928+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72880cc8b0 con 0x7f728814bef0 2023-10-16T14:16:36.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f727dffb640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f726c06aed0 0x7f726c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f728004b070 con 0x7f728814bef0 2023-10-16T14:16:36.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f727dffb640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7f726c073070 0x7f726c075490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:36.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 --> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f726c075b40 con 0x7f726c073070 2023-10-16T14:16:36.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f728d585640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f726c06aed0 0x7f726c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72800bee50 con 0x7f728814bef0 2023-10-16T14:16:36.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f728cd84640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7f726c073070 0x7f726c075490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:36.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f728cd84640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7f726c073070 0x7f726c075490 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:36.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 --> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f080c0ccb50 con 0x7f07f00730a0 2023-10-16T14:16:37.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f728d585640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f726c06aed0 0x7f726c06d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72840047b0 tx=0x7f7284008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8abc00acb0 con 0x7f8ac41625d0 2023-10-16T14:16:37.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f76804bb640 1 Processor -- start 2023-10-16T14:16:37.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f8abb7fe640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8aa406ae70 0x7f8aa406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8abc088e80 con 0x7f8ac41625d0 2023-10-16T14:16:37.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f8abb7fe640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7f8aa4073090 0x7f8aa40754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 --> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8aa4075b60 con 0x7f8aa4073090 2023-10-16T14:16:37.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8abc089230 con 0x7f8ac41625d0 2023-10-16T14:16:37.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.928+0000 7f0442c7c640 1 -- start start 2023-10-16T14:16:37.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f0442c7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0047c0 0x7f043c1f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f0442c7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c1f5c50 0x7f043c0af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f0442c7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043c1f60d0 0x7f043c0af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f76804bb640 1 -- start start 2023-10-16T14:16:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7f0801ffb640 1 -- 172.21.15.29:0/94185299 <== osd.1 v2:172.21.15.29:6800/3296543673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f080400a720 con 0x7f07f00730a0 2023-10-16T14:16:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f0442c7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c7340 con 0x7f043c1f5c50 2023-10-16T14:16:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f0442c7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c15a970 con 0x7f042c0047c0 2023-10-16T14:16:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.929+0000 7f0442c7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c7060 con 0x7f043c1f60d0 2023-10-16T14:16:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f0441219640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043c1f60d0 0x7f043c0af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f043bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c1f5c50 0x7f043c0af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f0441219640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043c1f60d0 0x7f043c0af770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59296/0 (socket says 172.21.15.29:59296) 2023-10-16T14:16:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.930+0000 7f0441219640 1 -- 172.21.15.29:0/2742603095 learned_addr learned my addr 172.21.15.29:0/2742603095 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f0441219640 1 -- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0047c0 msgr2=0x7f043c1f7640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:16:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f0441219640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f042c0047c0 0x7f043c1f7640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f0441219640 1 -- 172.21.15.29:0/2742603095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c1f5c50 msgr2=0x7f043c0af230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f0441219640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043c1f5c50 0x7f043c0af230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.931+0000 7f0441219640 1 -- 172.21.15.29:0/2742603095 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c15abc0 con 0x7f043c1f60d0 2023-10-16T14:16:37.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.932+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efd5403b050 con 0x7efd640cf710 2023-10-16T14:16:37.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f8ab9d9b640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7f8aa4073090 0x7f8aa40754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7f07f00730a0 msgr2=0x7f07f00754c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7f07f00730a0 0x7f07f00754c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f8ac259c640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8aa406ae70 0x7f8aa406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f0441219640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043c1f60d0 0x7f043c0af770 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f043400d6f0 tx=0x7f043400dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f07f006af00 msgr2=0x7f07f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f07f006af00 0x7f07f006d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f07fc038000 tx=0x7f07fc008030 comp rx=0 tx=0).stop 2023-10-16T14:16:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== osd.3 v2:172.21.15.29:6819/1457722234 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f726c075b40 con 0x7f726c073070 2023-10-16T14:16:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f76804bb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f767815d410 0x7f76780d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f76804bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76780cd260 0x7f76780cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f76804bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780b5d90 con 0x7f767815d030 2023-10-16T14:16:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f76804bb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76780d2b50 con 0x7f767815d410 2023-10-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f76804bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76780cdbf0 con 0x7f76780cd260 2023-10-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c15cdc0 msgr2=0x7f080c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 --2- 172.21.15.29:0/94185299 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c15cdc0 0x7f080c06bf40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f080800b4a0 tx=0x7f080800b960 comp rx=0 tx=0).stop 2023-10-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f0811ad9640 1 -- 172.21.15.29:0/94185299 reap_dead start 2023-10-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 shutdown_connections 2023-10-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.933+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 >> 172.21.15.29:0/94185299 conn(0x7f080c0ce6e0 msgr2=0x7f080c0c21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f767e257640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f767815d030 0x7f7678161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46344/0 (socket says 172.21.15.29:46344) 2023-10-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f767e257640 1 -- 172.21.15.29:0/1796568364 learned_addr learned my addr 172.21.15.29:0/1796568364 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 shutdown_connections 2023-10-16T14:16:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f081353c640 1 -- 172.21.15.29:0/94185299 wait complete. 2023-10-16T14:16:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043400bae0 con 0x7f043c1f60d0 2023-10-16T14:16:37.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043400bc40 con 0x7f043c1f60d0 2023-10-16T14:16:37.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434039470 con 0x7f043c1f60d0 2023-10-16T14:16:37.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f767da56640 1 --2- 172.21.15.29:0/1796568364 >> v2:172.21.15.170:3300/0 conn(0x7f767815d410 0x7f76780d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f8ab9d9b640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7f8aa4073090 0x7f8aa40754b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7f767ea58640 1 --2- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76780cd260 0x7f76780cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f767d255640 1 -- 172.21.15.29:0/1796568364 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520880031 0 0) 0x7f76780b5d90 con 0x7f767815d030 2023-10-16T14:16:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f767d255640 1 -- 172.21.15.29:0/1796568364 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7664003520 con 0x7f767815d030 2023-10-16T14:16:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f767da56640 1 -- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76780cd260 msgr2=0x7f76780cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f767da56640 1 --2- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76780cd260 0x7f76780cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f767da56640 1 -- 172.21.15.29:0/1796568364 >> v1:172.21.15.29:6789/0 conn(0x7f767815d030 legacy=0x7f7678161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f767da56640 1 -- 172.21.15.29:0/1796568364 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678160fc0 con 0x7f767815d410 2023-10-16T14:16:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.934+0000 7efd61ffb640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efd5006af30 0x7efd5006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd540ab690 con 0x7efd640cf710 2023-10-16T14:16:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7efd61ffb640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7efd500730d0 0x7efd500754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.936+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c15aea0 con 0x7f043c1f60d0 2023-10-16T14:16:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7f8ac259c640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8aa406ae70 0x7f8aa406d330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8aac036000 tx=0x7f8aac008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.936+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c0c6330 con 0x7f043c1f60d0 2023-10-16T14:16:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 --> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd50075ba0 con 0x7efd500730d0 2023-10-16T14:16:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.935+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd5403e610 con 0x7efd640cf710 2023-10-16T14:16:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.936+0000 7efd697ac640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efd5006af30 0x7efd5006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.936+0000 7efd69fad640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7efd500730d0 0x7efd500754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7f767da56640 1 --2- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f767815d410 0x7f76780d1630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76680097a0 tx=0x7f766802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22489d3e91751608 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f043c15d590 con 0x7f043c1f60d0 2023-10-16T14:16:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== osd.4 v2:172.21.15.170:6808/1474743223 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8aa4075b60 con 0x7f8aa4073090 2023-10-16T14:16:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7f767d255640 1 -- 172.21.15.29:0/1796568364 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668030800 con 0x7f767815d410 2023-10-16T14:16:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7efd697ac640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efd5006af30 0x7efd5006d3f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd58004870 tx=0x7efd58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7f767d255640 1 -- 172.21.15.29:0/1796568364 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7668030960 con 0x7f767815d410 2023-10-16T14:16:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7f767d255640 1 -- 172.21.15.29:0/1796568364 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668030cd0 con 0x7f767815d410 2023-10-16T14:16:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.937+0000 7efd69fad640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7efd500730d0 0x7efd500754f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.938+0000 7f76804bb640 1 -- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f767815d410 msgr2=0x7f76780d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.938+0000 7f76804bb640 1 --2- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f767815d410 0x7f76780d1630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76680097a0 tx=0x7f766802b940 comp rx=0 tx=0).stop 2023-10-16T14:16:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.939+0000 7f76804bb640 1 -- 172.21.15.29:0/1796568364 shutdown_connections 2023-10-16T14:16:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.939+0000 7f76804bb640 1 --2- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76780cd260 0x7f76780cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.939+0000 7f76804bb640 1 --2- 172.21.15.29:0/1796568364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f767815d410 0x7f76780d1630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.939+0000 7f76804bb640 1 -- 172.21.15.29:0/1796568364 >> 172.21.15.29:0/1796568364 conn(0x7f76780cf720 msgr2=0x7f76780cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.939+0000 7f76804bb640 1 -- 172.21.15.29:0/1796568364 shutdown_connections 2023-10-16T14:16:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.939+0000 7f76804bb640 1 -- 172.21.15.29:0/1796568364 wait complete. 2023-10-16T14:16:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.941+0000 7f76804bb640 1 Processor -- start 2023-10-16T14:16:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.941+0000 7f76804bb640 1 -- start start 2023-10-16T14:16:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== osd.7 v2:172.21.15.170:6821/3840642819 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7efd50075ba0 con 0x7efd500730d0 2023-10-16T14:16:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.940+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0434036070 con 0x7f043c1f60d0 2023-10-16T14:16:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f76804bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76780cd260 0x7f767815c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f76804bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f767815d030 0x7f767815e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f76804bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815d410 0x7f76780cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f76804bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767815e770 con 0x7f767815d410 2023-10-16T14:16:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f76804bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76780c7ae0 con 0x7f76780cd260 2023-10-16T14:16:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f76804bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767815efb0 con 0x7f767815d030 2023-10-16T14:16:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f04397fa640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f042006aef0 0x7f042006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 --> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f30a4002cb0 con 0x7f30b00730e0 2023-10-16T14:16:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f767ea58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815d410 0x7f76780cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815d410 0x7f76780cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50852/0 (socket says 172.21.15.29:50852) 2023-10-16T14:16:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767e257640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76780cd260 0x7f767815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 -- 172.21.15.29:0/1099680719 learned_addr learned my addr 172.21.15.29:0/1099680719 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767da56640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f767815d030 0x7f767815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.942+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f043407b110 con 0x7f043c1f60d0 2023-10-16T14:16:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f0440a18640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f042006aef0 0x7f042006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 -- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f767815d030 msgr2=0x7f767815e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f767815d030 0x7f767815e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 -- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76780cd260 msgr2=0x7f767815c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76780cd260 0x7f767815c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767ea58640 1 -- 172.21.15.29:0/1099680719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668009400 con 0x7f767815d410 2023-10-16T14:16:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.943+0000 7f767e257640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76780cd260 0x7f767815c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f30c1ffb640 1 -- 172.21.15.29:0/442770525 <== osd.0 v2:172.21.15.29:6808/1553131287 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f30a4002cb0 con 0x7f30b00730e0 2023-10-16T14:16:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f04397fa640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7f0420073df0 0x7f0420076290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 --> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0420076960 con 0x7f0420073df0 2023-10-16T14:16:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04340049e0 con 0x7f043c1f60d0 2023-10-16T14:16:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f767ea58640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815d410 0x7f76780cb540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f767000efc0 tx=0x7f767000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670009ba0 con 0x7f767815d410 2023-10-16T14:16:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767000f040 con 0x7f767815d410 2023-10-16T14:16:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76700146c0 con 0x7f767815d410 2023-10-16T14:16:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.945+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7f30b00730e0 msgr2=0x7f30b0075500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.945+0000 7f30b77fe640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7f30b00730e0 0x7f30b0075500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767815e9f0 con 0x7f767815d410 2023-10-16T14:16:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.944+0000 7f0440a18640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f042006aef0 0x7f042006d3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f043c0afdb0 tx=0x7f0430000f50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.945+0000 7f043bfff640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7f0420073df0 0x7f0420076290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f30b006aec0 msgr2=0x7f30b006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30b77fe640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f30b006aec0 0x7f30b006d380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f30cc159950 tx=0x7f30bc00c040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc06a2e0 msgr2=0x7f30cc155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30b77fe640 1 --2- 172.21.15.29:0/442770525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc06a2e0 0x7f30cc155840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f30c8033000 tx=0x7f30c80046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30d1e5f640 1 -- 172.21.15.29:0/442770525 reap_dead start 2023-10-16T14:16:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 shutdown_connections 2023-10-16T14:16:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 >> 172.21.15.29:0/442770525 conn(0x7f30cc0ce9b0 msgr2=0x7f30cc0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.945+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76780c7d80 con 0x7f767815d410 2023-10-16T14:16:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.946+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76780d2290 con 0x7f767815d410 2023-10-16T14:16:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.948+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7670014820 con 0x7f767815d410 2023-10-16T14:16:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.948+0000 7f043bfff640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7f0420073df0 0x7f0420076290 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 shutdown_connections 2023-10-16T14:16:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f30b77fe640 1 -- 172.21.15.29:0/442770525 wait complete. 2023-10-16T14:16:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f766effd640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f765c06aec0 0x7f765c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f767008a080 con 0x7f767815d410 2023-10-16T14:16:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f766effd640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7f765c0730e0 0x7f765c075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 --> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f765c075bb0 con 0x7f765c0730e0 2023-10-16T14:16:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.949+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f767008c600 con 0x7f767815d410 2023-10-16T14:16:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.950+0000 7f767da56640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7f765c0730e0 0x7f765c075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.950+0000 7f767e257640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f765c06aec0 0x7f765c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.951+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 --> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0fd80d1a20 con 0x7f0fbc073060 2023-10-16T14:16:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.951+0000 7f767da56640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7f765c0730e0 0x7f765c075500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.951+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== osd.5 v2:172.21.15.170:6800/3780367638 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0420076960 con 0x7f0420073df0 2023-10-16T14:16:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.951+0000 7f767e257640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f765c06aec0 0x7f765c06d380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7678202d90 tx=0x7f7674000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.951+0000 7f0fcdffb640 1 -- 172.21.15.29:0/210473300 <== osd.2 v2:172.21.15.29:6809/4164234701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0fc80096a0 con 0x7f0fbc073060 2023-10-16T14:16:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.952+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7f0fbc073060 msgr2=0x7f0fbc075480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.952+0000 7f0fde11a640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7f0fbc073060 0x7f0fbc075480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.953+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== osd.6 v2:172.21.15.170:6816/690425284 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f765c075bb0 con 0x7f765c0730e0 2023-10-16T14:16:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.954+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0fbc06aec0 msgr2=0x7f0fbc06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.954+0000 7f0fde11a640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0fbc06aec0 0x7f0fbc06d380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4001eb0 tx=0x7f0fd4001c40 comp rx=0 tx=0).stop 2023-10-16T14:16:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.954+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd815cd70 msgr2=0x7f0fd815d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.954+0000 7f0fde11a640 1 --2- 172.21.15.29:0/210473300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd815cd70 0x7f0fd815d880 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009eb0 tx=0x7f0fd0004a40 comp rx=0 tx=0).stop 2023-10-16T14:16:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.958+0000 7f0fdd919640 1 -- 172.21.15.29:0/210473300 reap_dead start 2023-10-16T14:16:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.958+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 shutdown_connections 2023-10-16T14:16:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.958+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 >> 172.21.15.29:0/210473300 conn(0x7f0fd80cf570 msgr2=0x7f0fd80b74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.958+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 shutdown_connections 2023-10-16T14:16:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.958+0000 7f0fde11a640 1 -- 172.21.15.29:0/210473300 wait complete. 2023-10-16T14:16:37.047 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T14:16:37.048 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T14:16:37.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.970+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 --> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ac406d2b0 con 0x7f8aa4073090 2023-10-16T14:16:37.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.971+0000 7f8abb7fe640 1 -- 172.21.15.29:0/3066558983 <== osd.4 v2:172.21.15.170:6808/1474743223 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8ab40013f0 con 0x7f8aa4073090 2023-10-16T14:16:37.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.971+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 --> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7288154860 con 0x7f726c073070 2023-10-16T14:16:37.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.972+0000 7f727dffb640 1 -- 172.21.15.29:0/282812685 <== osd.3 v2:172.21.15.29:6819/1457722234 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7288154860 con 0x7f726c073070 2023-10-16T14:16:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.972+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7f8aa4073090 msgr2=0x7f8aa40754b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.972+0000 7f8ac8e4f640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7f8aa4073090 0x7f8aa40754b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7f726c073070 msgr2=0x7f726c075490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7f726c073070 0x7f726c075490 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8aa406ae70 msgr2=0x7f8aa406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8aa406ae70 0x7f8aa406d330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8aac036000 tx=0x7f8aac008030 comp rx=0 tx=0).stop 2023-10-16T14:16:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ac41625d0 msgr2=0x7f8ac4160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 --2- 172.21.15.29:0/3066558983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ac41625d0 0x7f8ac4160260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00c7a0 tx=0x7f8abc00cc60 comp rx=0 tx=0).stop 2023-10-16T14:16:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac2d9d640 1 -- 172.21.15.29:0/3066558983 reap_dead start 2023-10-16T14:16:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 shutdown_connections 2023-10-16T14:16:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 >> 172.21.15.29:0/3066558983 conn(0x7f8ac40cb8b0 msgr2=0x7f8ac40ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f726c06aed0 msgr2=0x7f726c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f726c06aed0 0x7f726c06d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72840047b0 tx=0x7f7284008040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 shutdown_connections 2023-10-16T14:16:37.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f728814bef0 msgr2=0x7f7288148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 --2- 172.21.15.29:0/282812685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f728814bef0 0x7f7288148d40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f728003da30 tx=0x7f7280001b50 comp rx=0 tx=0).stop 2023-10-16T14:16:37.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728e587640 1 -- 172.21.15.29:0/282812685 reap_dead start 2023-10-16T14:16:37.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 shutdown_connections 2023-10-16T14:16:37.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 >> 172.21.15.29:0/282812685 conn(0x7f72880b0570 msgr2=0x7f728814d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.973+0000 7f8ac8e4f640 1 -- 172.21.15.29:0/3066558983 wait complete. 2023-10-16T14:16:37.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.974+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 shutdown_connections 2023-10-16T14:16:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.975+0000 7f728f7e9640 1 -- 172.21.15.29:0/282812685 wait complete. 2023-10-16T14:16:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.976+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 --> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd44002cb0 con 0x7efd500730d0 2023-10-16T14:16:37.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.977+0000 7efd61ffb640 1 -- 172.21.15.29:0/2658480532 <== osd.7 v2:172.21.15.170:6821/3840642819 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efd44002cb0 con 0x7efd500730d0 2023-10-16T14:16:37.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.979+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 --> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f043c15d2b0 con 0x7f0420073df0 2023-10-16T14:16:37.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7efd500730d0 msgr2=0x7efd500754f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd6ba10640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7efd500730d0 0x7efd500754f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efd5006af30 msgr2=0x7efd5006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd6ba10640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efd5006af30 0x7efd5006d3f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd58004870 tx=0x7efd58009040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd640cf710 msgr2=0x7efd640cb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd6ba10640 1 --2- 172.21.15.29:0/2658480532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd640cf710 0x7efd640cb330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd54009d90 tx=0x7efd54004a40 comp rx=0 tx=0).stop 2023-10-16T14:16:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7f04397fa640 1 -- 172.21.15.29:0/2742603095 <== osd.5 v2:172.21.15.170:6800/3780367638 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0428032400 con 0x7f0420073df0 2023-10-16T14:16:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.980+0000 7efd69fad640 1 -- 172.21.15.29:0/2658480532 reap_dead start 2023-10-16T14:16:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 shutdown_connections 2023-10-16T14:16:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 >> 172.21.15.29:0/2658480532 conn(0x7efd640d2cc0 msgr2=0x7efd640cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 shutdown_connections 2023-10-16T14:16:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7efd6ba10640 1 -- 172.21.15.29:0/2658480532 wait complete. 2023-10-16T14:16:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7f0420073df0 msgr2=0x7f0420076290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7f0420073df0 0x7f0420076290 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f042006aef0 msgr2=0x7f042006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f042006aef0 0x7f042006d3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f043c0afdb0 tx=0x7f0430000f50 comp rx=0 tx=0).stop 2023-10-16T14:16:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043c1f60d0 msgr2=0x7f043c0af770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 --2- 172.21.15.29:0/2742603095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043c1f60d0 0x7f043c0af770 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f043400d6f0 tx=0x7f043400dbb0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0441219640 1 -- 172.21.15.29:0/2742603095 reap_dead start 2023-10-16T14:16:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.981+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 shutdown_connections 2023-10-16T14:16:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.982+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 >> 172.21.15.29:0/2742603095 conn(0x7f043c0ce9b0 msgr2=0x7f043c070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.982+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 shutdown_connections 2023-10-16T14:16:37.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.982+0000 7f0442c7c640 1 -- 172.21.15.29:0/2742603095 wait complete. 2023-10-16T14:16:37.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.983+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 --> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76780d25d0 con 0x7f765c0730e0 2023-10-16T14:16:37.065 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T14:16:37.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f766effd640 1 -- 172.21.15.29:0/1099680719 <== osd.6 v2:172.21.15.170:6816/690425284 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7668030ad0 con 0x7f765c0730e0 2023-10-16T14:16:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7f765c0730e0 msgr2=0x7f765c075500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f76804bb640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7f765c0730e0 0x7f765c075500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f765c06aec0 msgr2=0x7f765c06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f76804bb640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f765c06aec0 0x7f765c06d380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7678202d90 tx=0x7f7674000bc0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815d410 msgr2=0x7f76780cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.984+0000 7f76804bb640 1 --2- 172.21.15.29:0/1099680719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f767815d410 0x7f76780cb540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f767000efc0 tx=0x7f767000c490 comp rx=0 tx=0).stop 2023-10-16T14:16:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.985+0000 7f767ea58640 1 -- 172.21.15.29:0/1099680719 reap_dead start 2023-10-16T14:16:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.985+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 shutdown_connections 2023-10-16T14:16:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.985+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 >> 172.21.15.29:0/1099680719 conn(0x7f76780cf720 msgr2=0x7f76780b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.985+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 shutdown_connections 2023-10-16T14:16:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:36.985+0000 7f76804bb640 1 -- 172.21.15.29:0/1099680719 wait complete. 2023-10-16T14:16:37.069 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T14:16:37.070 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T14:16:37.071 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T14:16:37.072 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T14:16:37.074 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T14:16:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.388+0000 7f9fe5943640 1 Processor -- start 2023-10-16T14:16:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.389+0000 7f9fe5943640 1 -- start start 2023-10-16T14:16:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.391+0000 7f9fe5943640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e1cd0 0x7f9fd80eb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.391+0000 7f9fe5943640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9fd8038630 0x7f9fd80eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.391+0000 7f9fe5943640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd8006b20 con 0x7f9fd80e18f0 2023-10-16T14:16:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.391+0000 7f9fe5943640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd8006e00 con 0x7f9fd8038630 2023-10-16T14:16:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.391+0000 7f9fe5943640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd80070e0 con 0x7f9fd80e1cd0 2023-10-16T14:16:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.392+0000 7f9fe4941640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f9fd80e18f0 0x7f9fd80062d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:48422/0 (socket says 172.21.15.29:48422) 2023-10-16T14:16:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.392+0000 7f9fe4941640 1 -- 172.21.15.29:0/850707170 learned_addr learned my addr 172.21.15.29:0/850707170 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.392+0000 7f9fe5142640 1 --2- 172.21.15.29:0/850707170 >> v2:172.21.15.29:3300/0 conn(0x7f9fd8038630 0x7f9fd80eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.392+0000 7f9fd7fff640 1 --2- 172.21.15.29:0/850707170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e1cd0 0x7f9fd80eb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.394+0000 7f9fd77fe640 1 -- 172.21.15.29:0/850707170 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954856387 0 0) 0x7f9fd8006b20 con 0x7f9fd80e18f0 2023-10-16T14:16:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.394+0000 7f9fe5142640 1 -- 172.21.15.29:0/850707170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e1cd0 msgr2=0x7f9fd80eb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.394+0000 7f9fe5142640 1 --2- 172.21.15.29:0/850707170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e1cd0 0x7f9fd80eb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.394+0000 7f9fe5142640 1 -- 172.21.15.29:0/850707170 >> v1:172.21.15.170:6789/0 conn(0x7f9fd80e18f0 legacy=0x7f9fd80062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.394+0000 7f9fe5142640 1 -- 172.21.15.29:0/850707170 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd80073c0 con 0x7f9fd8038630 2023-10-16T14:16:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.396+0000 7f9fe5142640 1 --2- 172.21.15.29:0/850707170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd80eb6b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9fe004fa60 tx=0x7f9fe00a5cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b7f4e6d58162d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.397+0000 7f9fd77fe640 1 -- 172.21.15.29:0/850707170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00a6920 con 0x7f9fd8038630 2023-10-16T14:16:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.397+0000 7f9fd77fe640 1 -- 172.21.15.29:0/850707170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe00a6ac0 con 0x7f9fd8038630 2023-10-16T14:16:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.397+0000 7f9fd77fe640 1 -- 172.21.15.29:0/850707170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00a6920 con 0x7f9fd8038630 2023-10-16T14:16:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.398+0000 7f9fe5943640 1 -- 172.21.15.29:0/850707170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 msgr2=0x7f9fd80eb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.399+0000 7f9fe5943640 1 --2- 172.21.15.29:0/850707170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd80eb6b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9fe004fa60 tx=0x7f9fe00a5cc0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.400+0000 7fbf7be26640 1 Processor -- start 2023-10-16T14:16:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.399+0000 7f9fe5943640 1 -- 172.21.15.29:0/850707170 shutdown_connections 2023-10-16T14:16:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.400+0000 7f9fe5943640 1 --2- 172.21.15.29:0/850707170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd80eb6b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.400+0000 7f9fe5943640 1 --2- 172.21.15.29:0/850707170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e1cd0 0x7f9fd80eb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.400+0000 7f9fe5943640 1 -- 172.21.15.29:0/850707170 >> 172.21.15.29:0/850707170 conn(0x7f9fd80590c0 msgr2=0x7f9fd80594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.400+0000 7fbf7be26640 1 -- start start 2023-10-16T14:16:37.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.401+0000 7f731dc75640 1 Processor -- start 2023-10-16T14:16:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.401+0000 7f9fe5943640 1 -- 172.21.15.29:0/850707170 shutdown_connections 2023-10-16T14:16:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.402+0000 7f9fe5943640 1 -- 172.21.15.29:0/850707170 wait complete. 2023-10-16T14:16:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.402+0000 7fbf7be26640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf740caa80 0x7fbf74069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.403+0000 7fbf7be26640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf740caec0 con 0x7fbf740addb0 2023-10-16T14:16:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.403+0000 7fbf7be26640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf740cb140 con 0x7fbf7406a2e0 2023-10-16T14:16:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.403+0000 7fbf7be26640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74161a50 con 0x7fbf740caa80 2023-10-16T14:16:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.402+0000 7f731dc75640 1 -- start start 2023-10-16T14:16:37.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.403+0000 7fbf79bc2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf740addb0 0x7fbf740c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46352/0 (socket says 172.21.15.29:46352) 2023-10-16T14:16:37.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7fbf7a3c3640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fbf7406a2e0 0x7fbf741650d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:48432/0 (socket says 172.21.15.29:48432) 2023-10-16T14:16:37.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7fbf7a3c3640 1 -- 172.21.15.29:0/3790603240 learned_addr learned my addr 172.21.15.29:0/3790603240 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7f9fe5943640 1 Processor -- start 2023-10-16T14:16:37.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.405+0000 7f9fe5943640 1 -- start start 2023-10-16T14:16:37.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7f731dc75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73100e77a0 0x7f731004f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7fbf793c1640 1 --2- 172.21.15.29:0/3790603240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf740caa80 0x7fbf74069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.405+0000 7fbf78bc0640 1 -- 172.21.15.29:0/3790603240 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802962593 0 0) 0x7fbf740caec0 con 0x7fbf740addb0 2023-10-16T14:16:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.405+0000 7fbf78bc0640 1 -- 172.21.15.29:0/3790603240 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf60003520 con 0x7fbf740addb0 2023-10-16T14:16:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7f731dc75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100eb1b0 0x7f73100eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7f731dc75640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7310050230 con 0x7f7310052ed0 2023-10-16T14:16:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7f731dc75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7310006370 con 0x7f73100e77a0 2023-10-16T14:16:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.404+0000 7f731dc75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73100ebb30 con 0x7f73100eb1b0 2023-10-16T14:16:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.405+0000 7fbf793c1640 1 -- 172.21.15.29:0/3790603240 >> v1:172.21.15.170:6789/0 conn(0x7fbf7406a2e0 legacy=0x7fbf741650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.405+0000 7fbf793c1640 1 -- 172.21.15.29:0/3790603240 >> v1:172.21.15.29:6789/0 conn(0x7fbf740addb0 legacy=0x7fbf740c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.405+0000 7fbf793c1640 1 -- 172.21.15.29:0/3790603240 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7406b030 con 0x7fbf740caa80 2023-10-16T14:16:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f9fe5943640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd81811a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f9fe5943640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e18f0 0x7f9fd80dcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f9fe4941640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd81811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f9fe4941640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd81811a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50886/0 (socket says 172.21.15.29:50886) 2023-10-16T14:16:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f9fe4941640 1 -- 172.21.15.29:0/3741110072 learned_addr learned my addr 172.21.15.29:0/3741110072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fd80e1cd0 0x7f9fd80ea500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f73177fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73100e77a0 0x7f731004f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f73177fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73100e77a0 0x7f731004f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50876/0 (socket says 172.21.15.29:50876) 2023-10-16T14:16:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f73177fe640 1 -- 172.21.15.29:0/2360717801 learned_addr learned my addr 172.21.15.29:0/2360717801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.407+0000 7f7316ffd640 1 --2- 172.21.15.29:0/2360717801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100eb1b0 0x7f73100eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd81816e0 con 0x7f9fd8038630 2023-10-16T14:16:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd803eaf0 con 0x7f9fd80e1cd0 2023-10-16T14:16:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd80516d0 con 0x7f9fd80e18f0 2023-10-16T14:16:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fd7fff640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e18f0 0x7f9fd80dcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe4941640 1 -- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e18f0 msgr2=0x7f9fd80dcb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe4941640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e18f0 0x7f9fd80dcb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe4941640 1 -- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fd80e1cd0 msgr2=0x7f9fd80ea500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe5142640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fd80e1cd0 0x7f9fd80ea500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f9fe4941640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fd80e1cd0 0x7f9fd80ea500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.408+0000 7f7316ffd640 1 -- 172.21.15.29:0/2360717801 >> v2:172.21.15.29:3300/0 conn(0x7f73100e77a0 msgr2=0x7f731004f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.409+0000 7fbf793c1640 1 --2- 172.21.15.29:0/3790603240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf740caa80 0x7fbf74069da0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf640097d0 tx=0x7fbf6402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a2d7defd8a41e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.409+0000 7f9fe4941640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe004f050 con 0x7f9fd8038630 2023-10-16T14:16:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.409+0000 7f9fe4941640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd81811a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00c4a0 tx=0x7f9fdc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.410+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0074e0 con 0x7f9fd8038630 2023-10-16T14:16:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.410+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc007640 con 0x7f9fd8038630 2023-10-16T14:16:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.410+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc004a00 con 0x7f9fd8038630 2023-10-16T14:16:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.410+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd80506a0 con 0x7f9fd8038630 2023-10-16T14:16:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.410+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd8050c60 con 0x7f9fd8038630 2023-10-16T14:16:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.409+0000 7f7316ffd640 1 --2- 172.21.15.29:0/2360717801 >> v2:172.21.15.29:3300/0 conn(0x7f73100e77a0 0x7f731004f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.409+0000 7f7316ffd640 1 -- 172.21.15.29:0/2360717801 >> v1:172.21.15.170:6789/0 conn(0x7f7310052ed0 legacy=0x7f7310053360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.409+0000 7f7316ffd640 1 -- 172.21.15.29:0/2360717801 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7310007810 con 0x7f73100eb1b0 2023-10-16T14:16:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.411+0000 7f7316ffd640 1 --2- 172.21.15.29:0/2360717801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100eb1b0 0x7f73100eb5f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f73040097a0 tx=0x7f730402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a98689ecdd40a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7f73167fc640 1 -- 172.21.15.29:0/2360717801 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730402e070 con 0x7f73100eb1b0 2023-10-16T14:16:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7fbf78bc0640 1 -- 172.21.15.29:0/3790603240 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64030610 con 0x7fbf740caa80 2023-10-16T14:16:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7f73167fc640 1 -- 172.21.15.29:0/2360717801 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73040315a0 con 0x7f73100eb1b0 2023-10-16T14:16:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7f73167fc640 1 -- 172.21.15.29:0/2360717801 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304031970 con 0x7f73100eb1b0 2023-10-16T14:16:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd8005490 con 0x7f9fd8038630 2023-10-16T14:16:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7fbf78bc0640 1 -- 172.21.15.29:0/3790603240 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf64030770 con 0x7fbf740caa80 2023-10-16T14:16:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.412+0000 7fbf78bc0640 1 -- 172.21.15.29:0/3790603240 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64030b40 con 0x7fbf740caa80 2023-10-16T14:16:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.413+0000 7f1a2bfff640 1 Processor -- start 2023-10-16T14:16:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.413+0000 7f731dc75640 1 -- 172.21.15.29:0/2360717801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100eb1b0 msgr2=0x7f73100eb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.413+0000 7f1a2bfff640 1 -- start start 2023-10-16T14:16:37.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f731dc75640 1 --2- 172.21.15.29:0/2360717801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100eb1b0 0x7f73100eb5f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f73040097a0 tx=0x7f730402b840 comp rx=0 tx=0).stop 2023-10-16T14:16:37.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7fbf7be26640 1 -- 172.21.15.29:0/3790603240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf740caa80 msgr2=0x7fbf74069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f731dc75640 1 -- 172.21.15.29:0/2360717801 shutdown_connections 2023-10-16T14:16:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f731dc75640 1 --2- 172.21.15.29:0/2360717801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100eb1b0 0x7f73100eb5f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f731dc75640 1 --2- 172.21.15.29:0/2360717801 >> v2:172.21.15.29:3300/0 conn(0x7f73100e77a0 0x7f731004f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f731dc75640 1 -- 172.21.15.29:0/2360717801 >> 172.21.15.29:0/2360717801 conn(0x7f7310054b00 msgr2=0x7f7310054ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7fbf7be26640 1 --2- 172.21.15.29:0/3790603240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf740caa80 0x7fbf74069da0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf640097d0 tx=0x7fbf6402b960 comp rx=0 tx=0).stop 2023-10-16T14:16:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.415+0000 7f731dc75640 1 -- 172.21.15.29:0/2360717801 shutdown_connections 2023-10-16T14:16:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9fdc004b60 con 0x7f9fd8038630 2023-10-16T14:16:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f9fd57fa640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9fcc06ae70 0x7f9fcc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fdc088820 con 0x7f9fd8038630 2023-10-16T14:16:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.415+0000 7f731dc75640 1 -- 172.21.15.29:0/2360717801 wait complete. 2023-10-16T14:16:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.414+0000 7f9a37d97640 1 Processor -- start 2023-10-16T14:16:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.415+0000 7f9a37d97640 1 -- start start 2023-10-16T14:16:37.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f1a2bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a2c15cd70 0x7f1a2c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c0cded0 0x7f1a2c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f1a2bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c15d150 con 0x7f1a2c0b8480 2023-10-16T14:16:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f1a2bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0d1180 con 0x7f1a2c15cd70 2023-10-16T14:16:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f1a2bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c0d1460 con 0x7f1a2c0cded0 2023-10-16T14:16:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f9a37d97640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9a280e11e0 0x7f9a28038790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.416+0000 7f9a37d97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a280542b0 0x7f9a28038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f9a37d97640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a28055490 con 0x7f9a280e15c0 2023-10-16T14:16:37.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f9a37d97640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a28056930 con 0x7f9a280e11e0 2023-10-16T14:16:37.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f9a37d97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280552a0 con 0x7f9a280542b0 2023-10-16T14:16:37.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f9fd7fff640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9fcc06ae70 0x7f9fcc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f1a227fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a2c0b8480 0x7f1a2c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46368/0 (socket says 172.21.15.29:46368) 2023-10-16T14:16:37.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f1a227fc640 1 -- 172.21.15.29:0/1125482425 learned_addr learned my addr 172.21.15.29:0/1125482425 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7fbf7be26640 1 -- 172.21.15.29:0/3790603240 shutdown_connections 2023-10-16T14:16:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7fbf7be26640 1 --2- 172.21.15.29:0/3790603240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf740caa80 0x7fbf74069da0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7fbf7be26640 1 -- 172.21.15.29:0/3790603240 >> 172.21.15.29:0/3790603240 conn(0x7fbf740ce9b0 msgr2=0x7fbf740cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f1a2affd640 1 --2- 172.21.15.29:0/1125482425 >> v2:172.21.15.29:3300/0 conn(0x7f1a2c15cd70 0x7f1a2c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7fbf7be26640 1 -- 172.21.15.29:0/3790603240 shutdown_connections 2023-10-16T14:16:37.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7fbf7be26640 1 -- 172.21.15.29:0/3790603240 wait complete. 2023-10-16T14:16:37.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7f9fd7fff640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9fcc06ae70 0x7f9fcc06d330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8052d80 tx=0x7f9fd0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.417+0000 7f9a36334640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f9a280e15c0 0x7f9a28051690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:48442/0 (socket says 172.21.15.29:48442) 2023-10-16T14:16:37.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7f9a35b33640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9a280e11e0 0x7f9a28038790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.419+0000 7f9a35b33640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9a280e11e0 0x7f9a28038790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44480/0 (socket says 172.21.15.29:44480) 2023-10-16T14:16:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.419+0000 7f9a35b33640 1 -- 172.21.15.29:0/780915215 learned_addr learned my addr 172.21.15.29:0/780915215 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7f1a2affd640 1 -- 172.21.15.29:0/1125482425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c0cded0 msgr2=0x7f1a2c0b5c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T14:16:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7f1a2affd640 1 --2- 172.21.15.29:0/1125482425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c0cded0 0x7f1a2c0b5c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7f1a2affd640 1 -- 172.21.15.29:0/1125482425 >> v1:172.21.15.29:6789/0 conn(0x7f1a2c0b8480 legacy=0x7f1a2c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.418+0000 7f1a2affd640 1 -- 172.21.15.29:0/1125482425 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0d1740 con 0x7f1a2c15cd70 2023-10-16T14:16:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a35332640 1 --2- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a280542b0 0x7f9a28038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a34b31640 1 -- 172.21.15.29:0/780915215 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143544089 0 0) 0x7f9a28055490 con 0x7f9a280e15c0 2023-10-16T14:16:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a34b31640 1 -- 172.21.15.29:0/780915215 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a1c003520 con 0x7f9a280e15c0 2023-10-16T14:16:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f1a2affd640 1 --2- 172.21.15.29:0/1125482425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c15cd70 0x7f1a2c0d2220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0b56a0 tx=0x7f1a1c02baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722d185f148365d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f731dc75640 1 Processor -- start 2023-10-16T14:16:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a35b33640 1 -- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a280542b0 msgr2=0x7f9a28038cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a35b33640 1 --2- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a280542b0 0x7f9a28038cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a35b33640 1 -- 172.21.15.29:0/780915215 >> v1:172.21.15.170:6789/0 conn(0x7f9a280e15c0 legacy=0x7f9a28051690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f9a35b33640 1 -- 172.21.15.29:0/780915215 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a28056550 con 0x7f9a280e11e0 2023-10-16T14:16:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7f731dc75640 1 -- start start 2023-10-16T14:16:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.420+0000 7fbf7be26640 1 Processor -- start 2023-10-16T14:16:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.421+0000 7fbf7be26640 1 -- start start 2023-10-16T14:16:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f9a35b33640 1 --2- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a28038790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9a30068a60 tx=0x7f9a300a5c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=652b59d16d7fdb14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9fdc053830 con 0x7f9fd8038630 2023-10-16T14:16:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f731dc75640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7310052ed0 0x7f73100dc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f1a2a7fc640 1 -- 172.21.15.29:0/1125482425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c02d650 con 0x7f1a2c15cd70 2023-10-16T14:16:37.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f1a2a7fc640 1 -- 172.21.15.29:0/1125482425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c02d7b0 con 0x7f1a2c15cd70 2023-10-16T14:16:37.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f1a2a7fc640 1 -- 172.21.15.29:0/1125482425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c02db80 con 0x7f1a2c15cd70 2023-10-16T14:16:37.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f731dc75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100e77a0 0x7f73100492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f731dc75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 0x7f731004d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f731dc75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731004aec0 con 0x7f73100eb1b0 2023-10-16T14:16:37.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f731dc75640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731004a2b0 con 0x7f7310052ed0 2023-10-16T14:16:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f731dc75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731004a590 con 0x7f73100e77a0 2023-10-16T14:16:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.422+0000 7f9a34b31640 1 -- 172.21.15.29:0/780915215 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300a6aa0 con 0x7f9a280e11e0 2023-10-16T14:16:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f9a34b31640 1 -- 172.21.15.29:0/780915215 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a300a6c40 con 0x7f9a280e11e0 2023-10-16T14:16:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f9a34b31640 1 -- 172.21.15.29:0/780915215 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300ae640 con 0x7f9a280e11e0 2023-10-16T14:16:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7be26640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7406a2e0 0x7fbf74155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7be26640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740addb0 0x7fbf740c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7be26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 0x7fbf740c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7be26640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74156370 con 0x7fbf740caa80 2023-10-16T14:16:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7be26640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740b27a0 con 0x7fbf740addb0 2023-10-16T14:16:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7be26640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740b2a80 con 0x7fbf7406a2e0 2023-10-16T14:16:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f7317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 0x7f731004d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f7317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 0x7f731004d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50898/0 (socket says 172.21.15.29:50898) 2023-10-16T14:16:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f7317fff640 1 -- 172.21.15.29:0/2892484549 learned_addr learned my addr 172.21.15.29:0/2892484549 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f49b4421640 1 Processor -- start 2023-10-16T14:16:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf7a3c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 0x7fbf740c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7fbf79bc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7406a2e0 0x7fbf74155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 0x7fbf740c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50904/0 (socket says 172.21.15.29:50904) 2023-10-16T14:16:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 -- 172.21.15.29:0/2018859970 learned_addr learned my addr 172.21.15.29:0/2018859970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf79bc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7406a2e0 0x7fbf74155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59388/0 (socket says 172.21.15.29:59388) 2023-10-16T14:16:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f9a37d97640 1 -- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 msgr2=0x7f9a28038790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f9a37d97640 1 --2- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a28038790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9a30068a60 tx=0x7f9a300a5c30 comp rx=0 tx=0).stop 2023-10-16T14:16:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.423+0000 7f49b4421640 1 -- start start 2023-10-16T14:16:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f73177fe640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7310052ed0 0x7f73100dc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf793c1640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740addb0 0x7fbf740c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 -- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7406a2e0 msgr2=0x7fbf74155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7406a2e0 0x7fbf74155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 -- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740addb0 msgr2=0x7fbf740c2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740addb0 0x7fbf740c2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7fbf7a3c3640 1 -- 172.21.15.29:0/2018859970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf64009400 con 0x7fbf740caa80 2023-10-16T14:16:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7fbf7a3c3640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 0x7fbf740c7900 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00cbc0 tx=0x7fbf6c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00ad40 con 0x7fbf740caa80 2023-10-16T14:16:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf740b2d60 con 0x7fbf740caa80 2023-10-16T14:16:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f7316ffd640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100e77a0 0x7f73100492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f7317fff640 1 -- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100e77a0 msgr2=0x7f73100492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f7317fff640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100e77a0 0x7f73100492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f7317fff640 1 -- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7310052ed0 msgr2=0x7f73100dc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f7317fff640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7310052ed0 0x7f73100dc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f7317fff640 1 -- 172.21.15.29:0/2892484549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7304009400 con 0x7f73100eb1b0 2023-10-16T14:16:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f7317fff640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 0x7f731004d3d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7318061980 tx=0x7f73180671b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318090070 con 0x7f73100eb1b0 2023-10-16T14:16:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731004a870 con 0x7f73100eb1b0 2023-10-16T14:16:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731004b350 con 0x7f73100eb1b0 2023-10-16T14:16:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73180901d0 con 0x7f73100eb1b0 2023-10-16T14:16:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318090070 con 0x7f73100eb1b0 2023-10-16T14:16:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f1a2bfff640 1 -- 172.21.15.29:0/1125482425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c15cd70 msgr2=0x7f1a2c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/1125482425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c15cd70 0x7f1a2c0d2220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0b56a0 tx=0x7f1a1c02baa0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f1a2bfff640 1 -- 172.21.15.29:0/1125482425 shutdown_connections 2023-10-16T14:16:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/1125482425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c0cded0 0x7f1a2c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/1125482425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c15cd70 0x7f1a2c0d2220 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7f1a2bfff640 1 -- 172.21.15.29:0/1125482425 >> 172.21.15.29:0/1125482425 conn(0x7f1a2c0cf570 msgr2=0x7f1a2c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7f1a2bfff640 1 -- 172.21.15.29:0/1125482425 shutdown_connections 2023-10-16T14:16:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f9a37d97640 1 -- 172.21.15.29:0/780915215 shutdown_connections 2023-10-16T14:16:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f9a37d97640 1 --2- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a280542b0 0x7f9a28038cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f9a37d97640 1 --2- 172.21.15.29:0/780915215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a28038790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f9a37d97640 1 -- 172.21.15.29:0/780915215 >> 172.21.15.29:0/780915215 conn(0x7f9a28053790 msgr2=0x7f9a28053ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f9a37d97640 1 -- 172.21.15.29:0/780915215 shutdown_connections 2023-10-16T14:16:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f49b4421640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49ac15d030 0x7f49ac0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f49b4421640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac15d410 0x7f49ac161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f49b4421640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0b5d90 con 0x7f49ac161990 2023-10-16T14:16:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f49b4421640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac0d2b50 con 0x7f49ac15d030 2023-10-16T14:16:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f49b4421640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac160fc0 con 0x7f49ac15d410 2023-10-16T14:16:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f6da8c5d640 1 Processor -- start 2023-10-16T14:16:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c012400 con 0x7fbf740caa80 2023-10-16T14:16:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf740b3320 con 0x7fbf740caa80 2023-10-16T14:16:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00ad40 con 0x7fbf740caa80 2023-10-16T14:16:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7f49b21bd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49ac15d030 0x7f49ac0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.427+0000 7f49b19bc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac15d410 0x7f49ac161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.427+0000 7f49b29be640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f49ac161990 0x7f49ac0cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:48452/0 (socket says 172.21.15.29:48452) 2023-10-16T14:16:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.427+0000 7f49b29be640 1 -- 172.21.15.29:0/3152614805 learned_addr learned my addr 172.21.15.29:0/3152614805 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.424+0000 7f6da8c5d640 1 -- start start 2023-10-16T14:16:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.425+0000 7f9a37d97640 1 -- 172.21.15.29:0/780915215 wait complete. 2023-10-16T14:16:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f9a37d97640 1 Processor -- start 2023-10-16T14:16:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c005250 con 0x7fbf740caa80 2023-10-16T14:16:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8005250 con 0x7f73100eb1b0 2023-10-16T14:16:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f9a37d97640 1 -- start start 2023-10-16T14:16:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbf6c007070 con 0x7fbf740caa80 2023-10-16T14:16:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.427+0000 7f1a2bfff640 1 -- 172.21.15.29:0/1125482425 wait complete. 2023-10-16T14:16:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f1a2bfff640 1 Processor -- start 2023-10-16T14:16:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f1a2bfff640 1 -- start start 2023-10-16T14:16:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f9a37d97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a280335c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f9a37d97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a280e15c0 0x7f9a2803b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f9a37d97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a28179400 0x7f9a280dcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f9a37d97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28034180 con 0x7f9a280e15c0 2023-10-16T14:16:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f9a37d97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28033ea0 con 0x7f9a280e11e0 2023-10-16T14:16:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f9a37d97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28178920 con 0x7f9a28179400 2023-10-16T14:16:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f9a35b33640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a280335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f9a35b33640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a280335c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44510/0 (socket says 172.21.15.29:44510) 2023-10-16T14:16:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f9a35b33640 1 -- 172.21.15.29:0/1892949547 learned_addr learned my addr 172.21.15.29:0/1892949547 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f9a35332640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a280e15c0 0x7f9a2803b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f49b19bc640 1 -- 172.21.15.29:0/3152614805 >> v2:172.21.15.29:3300/0 conn(0x7f49ac15d030 msgr2=0x7f49ac0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f49b19bc640 1 --2- 172.21.15.29:0/3152614805 >> v2:172.21.15.29:3300/0 conn(0x7f49ac15d030 0x7f49ac0d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f49b19bc640 1 -- 172.21.15.29:0/3152614805 >> v1:172.21.15.170:6789/0 conn(0x7f49ac161990 legacy=0x7f49ac0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.428+0000 7f49b19bc640 1 -- 172.21.15.29:0/3152614805 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0d25d0 con 0x7f49ac15d410 2023-10-16T14:16:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.426+0000 7f6da8c5d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40caa80 0x7f6da4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f6da8c5d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6da406a2e0 0x7f6da41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f6da8c5d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da40caec0 con 0x7f6da40addb0 2023-10-16T14:16:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f6da8c5d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da4161a50 con 0x7f6da406a2e0 2023-10-16T14:16:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f6da8c5d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da406a750 con 0x7f6da40caa80 2023-10-16T14:16:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f6da259c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f6da40addb0 0x7f6da40c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:48460/0 (socket says 172.21.15.29:48460) 2023-10-16T14:16:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.429+0000 7f6da259c640 1 -- 172.21.15.29:0/1934679209 learned_addr learned my addr 172.21.15.29:0/1934679209 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7318095410 con 0x7f73100eb1b0 2023-10-16T14:16:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f6da1d9b640 1 --2- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40caa80 0x7f6da4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/1934679209 >> v2:172.21.15.170:3300/0 conn(0x7f6da406a2e0 0x7f6da41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7fbf6a7fc640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbf5806aed0 0x7fbf5806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.430+0000 7f6da159a640 1 -- 172.21.15.29:0/1934679209 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172855862 0 0) 0x7f6da40caec0 con 0x7f6da40addb0 2023-10-16T14:16:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f6da159a640 1 -- 172.21.15.29:0/1934679209 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c003520 con 0x7f6da40addb0 2023-10-16T14:16:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f9a36334640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a28179400 0x7f9a280dcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf6c089120 con 0x7fbf740caa80 2023-10-16T14:16:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f9a35b33640 1 -- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a28179400 msgr2=0x7f9a280dcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f9a35b33640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a28179400 0x7f9a280dcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f9a35b33640 1 -- 172.21.15.29:0/1892949547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a280e15c0 msgr2=0x7f9a2803b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f9a35b33640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a280e15c0 0x7f9a2803b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f9a35b33640 1 -- 172.21.15.29:0/1892949547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3004e6d0 con 0x7f9a280e11e0 2023-10-16T14:16:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f49b19bc640 1 --2- 172.21.15.29:0/3152614805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac15d410 0x7f49ac161450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f499c0097a0 tx=0x7f499c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=693957bac2797f96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7f7303fff640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f72fc07bee0 0x7f72fc07e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7fbf79bc2640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbf5806aed0 0x7fbf5806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7f9a35b33640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a280335c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a3004fa60 tx=0x7f9a30067b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7318102670 con 0x7f73100eb1b0 2023-10-16T14:16:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7f49b11bb640 1 -- 172.21.15.29:0/3152614805 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c030610 con 0x7f49ac15d410 2023-10-16T14:16:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7f49b11bb640 1 -- 172.21.15.29:0/3152614805 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c030770 con 0x7f49ac15d410 2023-10-16T14:16:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7f49b11bb640 1 -- 172.21.15.29:0/3152614805 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c030b40 con 0x7f49ac15d410 2023-10-16T14:16:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.432+0000 7efca7be9640 1 Processor -- start 2023-10-16T14:16:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f73177fe640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f72fc07bee0 0x7f72fc07e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f6da2d9d640 1 -- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40caa80 msgr2=0x7f6da4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40caa80 0x7f6da4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f6da2d9d640 1 -- 172.21.15.29:0/1934679209 >> v1:172.21.15.170:6789/0 conn(0x7f6da40addb0 legacy=0x7f6da40c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.431+0000 7f6da2d9d640 1 -- 172.21.15.29:0/1934679209 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da40cb140 con 0x7f6da406a2e0 2023-10-16T14:16:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300be070 con 0x7f9a280e11e0 2023-10-16T14:16:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a300b7410 con 0x7f9a280e11e0 2023-10-16T14:16:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300b7800 con 0x7f9a280e11e0 2023-10-16T14:16:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a28178c00 con 0x7f9a280e11e0 2023-10-16T14:16:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2c0b8480 0x7f1a2c1f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f49b4421640 1 -- 172.21.15.29:0/3152614805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac15d410 msgr2=0x7f49ac161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f49b4421640 1 --2- 172.21.15.29:0/3152614805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac15d410 0x7f49ac161450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f499c0097a0 tx=0x7f499c02b960 comp rx=0 tx=0).stop 2023-10-16T14:16:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a28033b00 con 0x7f9a280e11e0 2023-10-16T14:16:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da41650d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009dc0 tx=0x7f6d9c02be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d77a09a5bc3a2980 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7efca7be9640 1 -- start start 2023-10-16T14:16:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.433+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 0x7f1a2c070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7fbf79bc2640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbf5806aed0 0x7fbf5806d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf700011e0 tx=0x7fbf70007cd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7f1a227fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 0x7f1a2c070330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7f1a227fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 0x7f1a2c070330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50942/0 (socket says 172.21.15.29:50942) 2023-10-16T14:16:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7f1a227fc640 1 -- 172.21.15.29:0/3460384280 learned_addr learned my addr 172.21.15.29:0/3460384280 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7f6da159a640 1 -- 172.21.15.29:0/1934679209 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c02cc40 con 0x7f6da406a2e0 2023-10-16T14:16:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7efca7be9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efca015af80 0x7efca00cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7efca7be9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efca006faf0 0x7efca015f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.434+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c15cd70 0x7f1a2c070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c154c40 con 0x7f1a2c0cded0 2023-10-16T14:16:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c154030 con 0x7f1a2c0b8480 2023-10-16T14:16:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c154310 con 0x7f1a2c15cd70 2023-10-16T14:16:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f6da159a640 1 -- 172.21.15.29:0/1934679209 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c02cda0 con 0x7f6da406a2e0 2023-10-16T14:16:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f1a2b7fe640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c15cd70 0x7f1a2c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f6da159a640 1 -- 172.21.15.29:0/1934679209 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c034660 con 0x7f6da406a2e0 2023-10-16T14:16:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7f73177fe640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f72fc07bee0 0x7f72fc07e3a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f73100df7a0 tx=0x7f730c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a2affd640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2c0b8480 0x7f1a2c1f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a28178760 con 0x7f9a280e11e0 2023-10-16T14:16:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a2b7fe640 1 -- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2c0b8480 msgr2=0x7f1a2c1f5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a2b7fe640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2c0b8480 0x7f1a2c1f5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a2b7fe640 1 -- 172.21.15.29:0/3460384280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 msgr2=0x7f1a2c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a2b7fe640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 0x7f1a2c070330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a2b7fe640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c009400 con 0x7f1a2c15cd70 2023-10-16T14:16:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7f1a227fc640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 0x7f1a2c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:16:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f49b4421640 1 -- 172.21.15.29:0/3152614805 shutdown_connections 2023-10-16T14:16:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f49b4421640 1 --2- 172.21.15.29:0/3152614805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac15d410 0x7f49ac161450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f49b4421640 1 --2- 172.21.15.29:0/3152614805 >> v2:172.21.15.29:3300/0 conn(0x7f49ac15d030 0x7f49ac0d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f49b4421640 1 -- 172.21.15.29:0/3152614805 >> 172.21.15.29:0/3152614805 conn(0x7f49ac0cf720 msgr2=0x7f49ac0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7efca7be9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca015eff0 con 0x7efca0065ab0 2023-10-16T14:16:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7efca7be9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00d0930 con 0x7efca015af80 2023-10-16T14:16:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7efca7be9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca015ecc0 con 0x7efca006faf0 2023-10-16T14:16:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7efca6186640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efca0065ab0 0x7efca00ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46372/0 (socket says 172.21.15.29:46372) 2023-10-16T14:16:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.435+0000 7efca6186640 1 -- 172.21.15.29:0/5946686 learned_addr learned my addr 172.21.15.29:0/5946686 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.436+0000 7efca5985640 1 --2- 172.21.15.29:0/5946686 >> v2:172.21.15.29:3300/0 conn(0x7efca015af80 0x7efca00cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7efca5985640 1 -- 172.21.15.29:0/5946686 >> v2:172.21.15.170:3300/0 conn(0x7efca006faf0 msgr2=0x7efca015f480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T14:16:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f6da8c5d640 1 -- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 msgr2=0x7f6da41650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da41650d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009dc0 tx=0x7f6d9c02be90 comp rx=0 tx=0).stop 2023-10-16T14:16:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7efca5985640 1 --2- 172.21.15.29:0/5946686 >> v2:172.21.15.170:3300/0 conn(0x7efca006faf0 0x7efca015f480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f49b4421640 1 -- 172.21.15.29:0/3152614805 shutdown_connections 2023-10-16T14:16:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7efca5985640 1 -- 172.21.15.29:0/5946686 >> v1:172.21.15.29:6789/0 conn(0x7efca0065ab0 legacy=0x7efca00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7f49b4421640 1 -- 172.21.15.29:0/3152614805 wait complete. 2023-10-16T14:16:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.437+0000 7efca5985640 1 -- 172.21.15.29:0/5946686 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca015b360 con 0x7efca015af80 2023-10-16T14:16:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.439+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf6c0541e0 con 0x7fbf740caa80 2023-10-16T14:16:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f1a2b7fe640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c15cd70 0x7f1a2c070870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400bdf0 tx=0x7f1a2400bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400cd70 con 0x7f1a2c15cd70 2023-10-16T14:16:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24014440 con 0x7f1a2c15cd70 2023-10-16T14:16:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24014830 con 0x7f1a2c15cd70 2023-10-16T14:16:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c1545f0 con 0x7f1a2c15cd70 2023-10-16T14:16:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.439+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c153c00 con 0x7f1a2c15cd70 2023-10-16T14:16:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9a300b5040 con 0x7f9a280e11e0 2023-10-16T14:16:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f6da8c5d640 1 -- 172.21.15.29:0/1934679209 shutdown_connections 2023-10-16T14:16:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da41650d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/1934679209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40caa80 0x7f6da4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f6da8c5d640 1 -- 172.21.15.29:0/1934679209 >> 172.21.15.29:0/1934679209 conn(0x7f6da40ce9b0 msgr2=0x7f6da40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f6da8c5d640 1 -- 172.21.15.29:0/1934679209 shutdown_connections 2023-10-16T14:16:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.438+0000 7f6da8c5d640 1 -- 172.21.15.29:0/1934679209 wait complete. 2023-10-16T14:16:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.439+0000 7efca5985640 1 --2- 172.21.15.29:0/5946686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca015af80 0x7efca00cb1c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efca00ce490 tx=0x7efc9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b511e0ee7d4bb02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.439+0000 7f9a227fc640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9a1406af70 0x7f9a1406d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.440+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a30124d50 con 0x7f9a280e11e0 2023-10-16T14:16:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.440+0000 7f9a35332640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9a1406af70 0x7f9a1406d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.440+0000 7f6da8c5d640 1 Processor -- start 2023-10-16T14:16:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.441+0000 7f6da8c5d640 1 -- start start 2023-10-16T14:16:37.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.441+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10005250 con 0x7f1a2c15cd70 2023-10-16T14:16:37.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.441+0000 7f49b4421640 1 Processor -- start 2023-10-16T14:16:37.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.441+0000 7f49b4421640 1 -- start start 2023-10-16T14:16:37.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.442+0000 7f9a35332640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9a1406af70 0x7f9a1406d430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a2817bc80 tx=0x7f9a2c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.442+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1a24014990 con 0x7f1a2c15cd70 2023-10-16T14:16:37.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f6da8c5d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da415e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f6da8c5d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40addb0 0x7f6da41f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f49b4421640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ac15d030 0x7f49ac15a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f49b4421640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49ac15d410 0x7f49ac0c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f49b4421640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac161990 0x7f49ac0c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f49b4421640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0cb4e0 con 0x7f49ac15d030 2023-10-16T14:16:37.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f49b4421640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0caed0 con 0x7f49ac15d410 2023-10-16T14:16:37.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f49b4421640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c7b80 con 0x7f49ac161990 2023-10-16T14:16:37.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f6da8c5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da40caa80 0x7f6da40b0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f1a23fff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a0c06af40 0x7f1a0c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f6da8c5d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41f7f20 con 0x7f6da40caa80 2023-10-16T14:16:37.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.443+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a24089840 con 0x7f1a2c15cd70 2023-10-16T14:16:37.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da8c5d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41f7d00 con 0x7f6da406a2e0 2023-10-16T14:16:37.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da415e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da8c5d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da415e710 con 0x7f6da40addb0 2023-10-16T14:16:37.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da259c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da415e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44558/0 (socket says 172.21.15.29:44558) 2023-10-16T14:16:37.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da259c640 1 -- 172.21.15.29:0/560680775 learned_addr learned my addr 172.21.15.29:0/560680775 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da40caa80 0x7f6da40b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f6da1d9b640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40addb0 0x7f6da41f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f1a2affd640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a0c06af40 0x7f1a0c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f49b19bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49ac15d410 0x7f49ac0c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f49b19bc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49ac15d410 0x7f49ac0c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:44564/0 (socket says 172.21.15.29:44564) 2023-10-16T14:16:37.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f49b19bc640 1 -- 172.21.15.29:0/905364047 learned_addr learned my addr 172.21.15.29:0/905364047 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f49b29be640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac161990 0x7f49ac0c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.444+0000 7f49b21bd640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ac15d030 0x7f49ac15a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f49b21bd640 1 -- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac161990 msgr2=0x7f49ac0c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f49b21bd640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac161990 0x7f49ac0c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f49b21bd640 1 -- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49ac15d410 msgr2=0x7f49ac0c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f49b21bd640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49ac15d410 0x7f49ac0c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f49b21bd640 1 -- 172.21.15.29:0/905364047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c009400 con 0x7f49ac15d030 2023-10-16T14:16:37.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f49b21bd640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ac15d030 0x7f49ac15a750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c5c0 tx=0x7f49a800ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f6da2d9d640 1 -- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40addb0 msgr2=0x7f6da41f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40addb0 0x7f6da41f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f6da2d9d640 1 -- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 msgr2=0x7f6da415e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da415e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f6da2d9d640 1 -- 172.21.15.29:0/560680775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c009a70 con 0x7f6da40caa80 2023-10-16T14:16:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.445+0000 7f6da2d9d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da40caa80 0x7f6da40b0460 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c033000 tx=0x7f6d9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.446+0000 7f1a2affd640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a0c06af40 0x7f1a0c06d400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c036000 tx=0x7f1a1c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8004220 con 0x7f49ac15d030 2023-10-16T14:16:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8004380 con 0x7f49ac15d030 2023-10-16T14:16:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8003830 con 0x7f49ac15d030 2023-10-16T14:16:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac0c78a0 con 0x7f49ac15d030 2023-10-16T14:16:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c040650 con 0x7f6da40caa80 2023-10-16T14:16:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f73180d11a0 con 0x7f73100eb1b0 2023-10-16T14:16:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da41f7740 con 0x7f6da40caa80 2023-10-16T14:16:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da40c7060 con 0x7f6da40caa80 2023-10-16T14:16:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c0407b0 con 0x7f6da40caa80 2023-10-16T14:16:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.448+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c040b60 con 0x7f6da40caa80 2023-10-16T14:16:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.448+0000 7efca4983640 1 -- 172.21.15.29:0/5946686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c031640 con 0x7efca015af80 2023-10-16T14:16:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.448+0000 7efca4983640 1 -- 172.21.15.29:0/5946686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c0317a0 con 0x7efca015af80 2023-10-16T14:16:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.448+0000 7efca4983640 1 -- 172.21.15.29:0/5946686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c031640 con 0x7efca015af80 2023-10-16T14:16:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.449+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da4066de0 con 0x7f6da40caa80 2023-10-16T14:16:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.447+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac15ac90 con 0x7f49ac15d030 2023-10-16T14:16:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.449+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac0cad30 con 0x7f49ac15d030 2023-10-16T14:16:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.451+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49a8003990 con 0x7f49ac15d030 2023-10-16T14:16:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.452+0000 7f49a2ffd640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f499006af40 0x7f499006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.450+0000 7efca7be9640 1 -- 172.21.15.29:0/5946686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca015af80 msgr2=0x7efca00cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.450+0000 7efca7be9640 1 --2- 172.21.15.29:0/5946686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca015af80 0x7efca00cb1c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efca00ce490 tx=0x7efc9c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.452+0000 7efca7be9640 1 -- 172.21.15.29:0/5946686 shutdown_connections 2023-10-16T14:16:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.452+0000 7efca7be9640 1 --2- 172.21.15.29:0/5946686 >> v2:172.21.15.170:3300/0 conn(0x7efca006faf0 0x7efca015f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.452+0000 7efca7be9640 1 --2- 172.21.15.29:0/5946686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca015af80 0x7efca00cb1c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.452+0000 7efca7be9640 1 -- 172.21.15.29:0/5946686 >> 172.21.15.29:0/5946686 conn(0x7efca00d0ab0 msgr2=0x7efca00d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.453+0000 7efca7be9640 1 -- 172.21.15.29:0/5946686 shutdown_connections 2023-10-16T14:16:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.453+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49a8089600 con 0x7f49ac15d030 2023-10-16T14:16:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.453+0000 7efca7be9640 1 -- 172.21.15.29:0/5946686 wait complete. 2023-10-16T14:16:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.453+0000 7f49b19bc640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f499006af40 0x7f499006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.453+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6d9c040cc0 con 0x7f6da40caa80 2023-10-16T14:16:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.454+0000 7efca7be9640 1 Processor -- start 2023-10-16T14:16:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.455+0000 7f6d92ffd640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6d8406aec0 0x7f6d8406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.455+0000 7f49b19bc640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f499006af40 0x7f499006d400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0cca80 tx=0x7f499c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7f6da259c640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6d8406aec0 0x7f6d8406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.455+0000 7efca7be9640 1 -- start start 2023-10-16T14:16:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca7be9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca0065ab0 0x7efca00c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca7be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca006faf0 0x7efca00c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca7be9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 0x7efca00c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca7be9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00c98f0 con 0x7efca006faf0 2023-10-16T14:16:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca7be9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00c71d0 con 0x7efca0065ab0 2023-10-16T14:16:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca7be9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca00c8a40 con 0x7efca015af80 2023-10-16T14:16:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca5985640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca0065ab0 0x7efca00c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca5184640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca006faf0 0x7efca00c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca6186640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 0x7efca00c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca6186640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 0x7efca00c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59458/0 (socket says 172.21.15.29:59458) 2023-10-16T14:16:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.456+0000 7efca6186640 1 -- 172.21.15.29:0/3797080747 learned_addr learned my addr 172.21.15.29:0/3797080747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.457+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9a300eff90 con 0x7f9a280e11e0 2023-10-16T14:16:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.457+0000 7efca6186640 1 -- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca0065ab0 msgr2=0x7efca00c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.457+0000 7efca6186640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca0065ab0 0x7efca00c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.457+0000 7efca6186640 1 -- 172.21.15.29:0/3797080747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca006faf0 msgr2=0x7efca00c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.457+0000 7efca6186640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca006faf0 0x7efca00c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.457+0000 7efca6186640 1 -- 172.21.15.29:0/3797080747 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c009400 con 0x7efca015af80 2023-10-16T14:16:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.458+0000 7efca6186640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 0x7efca00c9d10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc9800ca80 tx=0x7efc9800cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.460+0000 7f6da259c640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6d8406aec0 0x7f6d8406d380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d940011e0 tx=0x7f6d94009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.461+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800ac30 con 0x7efca015af80 2023-10-16T14:16:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.461+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800ad90 con 0x7efca015af80 2023-10-16T14:16:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.461+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98012600 con 0x7efca015af80 2023-10-16T14:16:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.461+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a24054a10 con 0x7f1a2c15cd70 2023-10-16T14:16:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.461+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca00c8c70 con 0x7efca015af80 2023-10-16T14:16:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.461+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca00c9230 con 0x7efca015af80 2023-10-16T14:16:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.463+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49a80547d0 con 0x7f49ac15d030 2023-10-16T14:16:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.463+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d9c07b670 con 0x7f6da40caa80 2023-10-16T14:16:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.464+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca00ca250 con 0x7efca015af80 2023-10-16T14:16:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.465+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efc98007070 con 0x7efca015af80 2023-10-16T14:16:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.466+0000 7efc967fc640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efc8406aec0 0x7efc8406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:37.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.467+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc98089440 con 0x7efca015af80 2023-10-16T14:16:37.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.468+0000 7efca5985640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efc8406aec0 0x7efc8406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:37.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.470+0000 7efca5985640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efc8406aec0 0x7efc8406d380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efc9c036000 tx=0x7efc9c03c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:37.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.470+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d9c0779b0 con 0x7f6da40caa80 2023-10-16T14:16:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.480+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efca00ca250 con 0x7efca015af80 2023-10-16T14:16:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.849+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9fd8038250 con 0x7f9fd8038630 2023-10-16T14:16:37.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.850+0000 7f9fd57fa640 1 -- 172.21.15.29:0/3741110072 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9fdc018090 con 0x7f9fd8038630 2023-10-16T14:16:37.854 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.853+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9fcc06ae70 msgr2=0x7f9fcc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.853+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9fcc06ae70 0x7f9fcc06d330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8052d80 tx=0x7f9fd0008040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.854+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 msgr2=0x7f9fd81811a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.854+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd81811a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00c4a0 tx=0x7f9fdc00c960 comp rx=0 tx=0).stop 2023-10-16T14:16:37.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 shutdown_connections 2023-10-16T14:16:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9fcc06ae70 0x7f9fcc06d330 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9fd80e1cd0 0x7f9fd80ea500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9fd80e18f0 0x7f9fd80dcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 --2- 172.21.15.29:0/3741110072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9fd8038630 0x7f9fd81811a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 >> 172.21.15.29:0/3741110072 conn(0x7f9fd80590c0 msgr2=0x7f9fd80df7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.855+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 shutdown_connections 2023-10-16T14:16:37.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.856+0000 7f9fe5943640 1 -- 172.21.15.29:0/3741110072 wait complete. 2023-10-16T14:16:37.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.866+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbf5c005500 con 0x7fbf740caa80 2023-10-16T14:16:37.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.867+0000 7fbf6a7fc640 1 -- 172.21.15.29:0/2018859970 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbf6c001be0 con 0x7fbf740caa80 2023-10-16T14:16:37.878 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.877+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbf5806aed0 msgr2=0x7fbf5806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.877+0000 7fbf7be26640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbf5806aed0 0x7fbf5806d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf700011e0 tx=0x7fbf70007cd0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.877+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 msgr2=0x7fbf740c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.877+0000 7fbf7be26640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 0x7fbf740c7900 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00cbc0 tx=0x7fbf6c00a430 comp rx=0 tx=0).stop 2023-10-16T14:16:37.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 shutdown_connections 2023-10-16T14:16:37.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f72f8005500 con 0x7f73100eb1b0 2023-10-16T14:16:37.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7fbf7be26640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbf5806aed0 0x7fbf5806d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7fbf7be26640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf740caa80 0x7fbf740c7900 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7fbf7be26640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740addb0 0x7fbf740c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7fbf7be26640 1 --2- 172.21.15.29:0/2018859970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7406a2e0 0x7fbf74155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.878+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 >> 172.21.15.29:0/2018859970 conn(0x7fbf740ce9b0 msgr2=0x7fbf740ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.879+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 shutdown_connections 2023-10-16T14:16:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.879+0000 7fbf7be26640 1 -- 172.21.15.29:0/2018859970 wait complete. 2023-10-16T14:16:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.880+0000 7f7303fff640 1 -- 172.21.15.29:0/2892484549 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f73180d0d30 con 0x7f73100eb1b0 2023-10-16T14:16:37.885 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.888 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-10-16T14:16:37.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f72fc07bee0 msgr2=0x7f72fc07e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f72fc07bee0 0x7f72fc07e3a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f73100df7a0 tx=0x7f730c008040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 msgr2=0x7f731004d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 0x7f731004d3d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7318061980 tx=0x7f73180671b0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 shutdown_connections 2023-10-16T14:16:37.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f72fc07bee0 0x7f72fc07e3a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73100eb1b0 0x7f731004d3d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f73100e77a0 0x7f73100492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 --2- 172.21.15.29:0/2892484549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7310052ed0 0x7f73100dc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.884+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 >> 172.21.15.29:0/2892484549 conn(0x7f7310054b00 msgr2=0x7f731004bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.885+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 shutdown_connections 2023-10-16T14:16:37.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.885+0000 7f731dc75640 1 -- 172.21.15.29:0/2892484549 wait complete. 2023-10-16T14:16:37.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.886+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9a28005790 con 0x7f9a280e11e0 2023-10-16T14:16:37.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.887+0000 7f9a227fc640 1 -- 172.21.15.29:0/1892949547 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a300b3020 con 0x7f9a280e11e0 2023-10-16T14:16:37.892 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.889+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f49ac0cad30 con 0x7f49ac15d030 2023-10-16T14:16:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.890+0000 7f49a2ffd640 1 -- 172.21.15.29:0/905364047 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f49ac0cad30 con 0x7f49ac15d030 2023-10-16T14:16:37.894 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.891+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9a1406af70 msgr2=0x7f9a1406d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.891+0000 7f9a37d97640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9a1406af70 0x7f9a1406d430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a2817bc80 tx=0x7f9a2c008040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 msgr2=0x7f9a280335c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a280335c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a3004fa60 tx=0x7f9a30067b80 comp rx=0 tx=0).stop 2023-10-16T14:16:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 shutdown_connections 2023-10-16T14:16:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9a1406af70 0x7f9a1406d430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a280e15c0 0x7f9a2803b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a28179400 0x7f9a280dcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 --2- 172.21.15.29:0/1892949547 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a280e11e0 0x7f9a280335c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 >> 172.21.15.29:0/1892949547 conn(0x7f9a28053790 msgr2=0x7f9a2803be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 shutdown_connections 2023-10-16T14:16:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.892+0000 7f9a37d97640 1 -- 172.21.15.29:0/1892949547 wait complete. 2023-10-16T14:16:37.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.893+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f499006af40 msgr2=0x7f499006d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.893+0000 7f49b4421640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f499006af40 0x7f499006d400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0cca80 tx=0x7f499c008030 comp rx=0 tx=0).stop 2023-10-16T14:16:37.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.893+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ac15d030 msgr2=0x7f49ac15a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.893+0000 7f49b4421640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ac15d030 0x7f49ac15a750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c5c0 tx=0x7f49a800ca80 comp rx=0 tx=0).stop 2023-10-16T14:16:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 shutdown_connections 2023-10-16T14:16:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f499006af40 0x7f499006d400 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ac161990 0x7f49ac0c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49ac15d410 0x7f49ac0c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 --2- 172.21.15.29:0/905364047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ac15d030 0x7f49ac15a750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 >> 172.21.15.29:0/905364047 conn(0x7f49ac0cf720 msgr2=0x7f49ac0cd240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 shutdown_connections 2023-10-16T14:16:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.894+0000 7f49b4421640 1 -- 172.21.15.29:0/905364047 wait complete. 2023-10-16T14:16:37.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.900+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1a100057d0 con 0x7f1a2c15cd70 2023-10-16T14:16:37.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.901+0000 7f1a23fff640 1 -- 172.21.15.29:0/3460384280 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1a24058030 con 0x7f1a2c15cd70 2023-10-16T14:16:37.905 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.903+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6da404e160 con 0x7f6da40caa80 2023-10-16T14:16:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a0c06af40 msgr2=0x7f1a0c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a0c06af40 0x7f1a0c06d400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c036000 tx=0x7f1a1c008030 comp rx=0 tx=0).stop 2023-10-16T14:16:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c15cd70 msgr2=0x7f1a2c070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c15cd70 0x7f1a2c070870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400bdf0 tx=0x7f1a2400bee0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 shutdown_connections 2023-10-16T14:16:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a0c06af40 0x7f1a0c06d400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a2c0cded0 0x7f1a2c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a2c15cd70 0x7f1a2c070870 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 --2- 172.21.15.29:0/3460384280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2c0b8480 0x7f1a2c1f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 >> 172.21.15.29:0/3460384280 conn(0x7f1a2c0cf570 msgr2=0x7f1a2c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 shutdown_connections 2023-10-16T14:16:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.905+0000 7f1a2bfff640 1 -- 172.21.15.29:0/3460384280 wait complete. 2023-10-16T14:16:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.907+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efca00c9a30 con 0x7efca015af80 2023-10-16T14:16:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.909+0000 7f6d92ffd640 1 -- 172.21.15.29:0/560680775 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6d9c039090 con 0x7f6da40caa80 2023-10-16T14:16:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.915 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2023-10-16T14:16:37.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.911+0000 7efc967fc640 1 -- 172.21.15.29:0/3797080747 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc98017020 con 0x7efca015af80 2023-10-16T14:16:37.916 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-16T14:16:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6d8406aec0 msgr2=0x7f6d8406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6d8406aec0 0x7f6d8406d380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d940011e0 tx=0x7f6d94009040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da40caa80 msgr2=0x7f6da40b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da40caa80 0x7f6da40b0460 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c033000 tx=0x7f6d9c004a40 comp rx=0 tx=0).stop 2023-10-16T14:16:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 shutdown_connections 2023-10-16T14:16:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6d8406aec0 0x7f6d8406d380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6da40caa80 0x7f6da40b0460 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6da40addb0 0x7f6da41f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 --2- 172.21.15.29:0/560680775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6da406a2e0 0x7f6da415e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 >> 172.21.15.29:0/560680775 conn(0x7f6da40ce9b0 msgr2=0x7f6da40cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.913+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 shutdown_connections 2023-10-16T14:16:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.914+0000 7f6da8c5d640 1 -- 172.21.15.29:0/560680775 wait complete. 2023-10-16T14:16:37.921 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-10-16T14:16:37.921 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efc8406aec0 msgr2=0x7efc8406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efc8406aec0 0x7efc8406d380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efc9c036000 tx=0x7efc9c03c040 comp rx=0 tx=0).stop 2023-10-16T14:16:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 msgr2=0x7efca00c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 0x7efca00c9d10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc9800ca80 tx=0x7efc9800cf40 comp rx=0 tx=0).stop 2023-10-16T14:16:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 shutdown_connections 2023-10-16T14:16:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7efc8406aec0 0x7efc8406d380 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca006faf0 0x7efca00c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca015af80 0x7efca00c9d10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 --2- 172.21.15.29:0/3797080747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca0065ab0 0x7efca00c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 >> 172.21.15.29:0/3797080747 conn(0x7efca00d0ab0 msgr2=0x7efca015be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 shutdown_connections 2023-10-16T14:16:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:37.915+0000 7efca7be9640 1 -- 172.21.15.29:0/3797080747 wait complete. 2023-10-16T14:16:37.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2023-10-16T14:16:37.925 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:37.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2023-10-16T14:16:37.926 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:37.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-10-16T14:16:37.931 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:37.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2023-10-16T14:16:37.937 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:37.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2023-10-16T14:16:37.938 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:38.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T14:16:38.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T14:16:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.119+0000 7fe162075640 1 Processor -- start 2023-10-16T14:16:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.119+0000 7fe162075640 1 -- start start 2023-10-16T14:16:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe162075640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cd70 0x7fe15c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe162075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe15c0b8480 0x7fe15c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe162075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c0d0be0 0x7fe15c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe162075640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c15d150 con 0x7fe15c0b8480 2023-10-16T14:16:39.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe162075640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c0d1180 con 0x7fe15c15cd70 2023-10-16T14:16:39.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe162075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0d1460 con 0x7fe15c0d0be0 2023-10-16T14:16:39.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe160872640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe15c0b8480 0x7fe15c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe160872640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe15c0b8480 0x7fe15c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50978/0 (socket says 172.21.15.29:50978) 2023-10-16T14:16:39.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe160872640 1 -- 172.21.15.29:0/3315811577 learned_addr learned my addr 172.21.15.29:0/3315811577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:39.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe161073640 1 --2- 172.21.15.29:0/3315811577 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cd70 0x7fe15c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.121+0000 7fe161874640 1 --2- 172.21.15.29:0/3315811577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c0d0be0 0x7fe15c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.122+0000 7fe160872640 1 -- 172.21.15.29:0/3315811577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c0d0be0 msgr2=0x7fe15c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.122+0000 7fe160872640 1 --2- 172.21.15.29:0/3315811577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c0d0be0 0x7fe15c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.122+0000 7fe160872640 1 -- 172.21.15.29:0/3315811577 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cd70 msgr2=0x7fe15c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.122+0000 7fe160872640 1 --2- 172.21.15.29:0/3315811577 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cd70 0x7fe15c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.122+0000 7fe160872640 1 -- 172.21.15.29:0/3315811577 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0d1740 con 0x7fe15c0b8480 2023-10-16T14:16:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.123+0000 7fe160872640 1 --2- 172.21.15.29:0/3315811577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b8480 0x7fe15c0cd7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0b56a0 tx=0x7fe14c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c0d287c97b3cfd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.123+0000 7fe153fff640 1 -- 172.21.15.29:0/3315811577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c031870 con 0x7fe15c0b8480 2023-10-16T14:16:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.123+0000 7fe153fff640 1 -- 172.21.15.29:0/3315811577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c0319d0 con 0x7fe15c0b8480 2023-10-16T14:16:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.123+0000 7fe153fff640 1 -- 172.21.15.29:0/3315811577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c031d60 con 0x7fe15c0b8480 2023-10-16T14:16:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 -- 172.21.15.29:0/3315811577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b8480 msgr2=0x7fe15c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 --2- 172.21.15.29:0/3315811577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b8480 0x7fe15c0cd7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0b56a0 tx=0x7fe14c02b910 comp rx=0 tx=0).stop 2023-10-16T14:16:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 -- 172.21.15.29:0/3315811577 shutdown_connections 2023-10-16T14:16:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 --2- 172.21.15.29:0/3315811577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c0d0be0 0x7fe15c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 --2- 172.21.15.29:0/3315811577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b8480 0x7fe15c0cd7c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 --2- 172.21.15.29:0/3315811577 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cd70 0x7fe15c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 -- 172.21.15.29:0/3315811577 >> 172.21.15.29:0/3315811577 conn(0x7fe15c0cf570 msgr2=0x7fe15c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 -- 172.21.15.29:0/3315811577 shutdown_connections 2023-10-16T14:16:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.124+0000 7fe162075640 1 -- 172.21.15.29:0/3315811577 wait complete. 2023-10-16T14:16:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.126+0000 7fe162075640 1 Processor -- start 2023-10-16T14:16:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.126+0000 7fe162075640 1 -- start start 2023-10-16T14:16:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe162075640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c0b8480 0x7fe15c155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe162075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 0x7fe15c15ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe162075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cd70 0x7fe15c15f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe162075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1f2e70 con 0x7fe15c0d0be0 2023-10-16T14:16:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe162075640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1f55b0 con 0x7fe15c0b8480 2023-10-16T14:16:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe162075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1f5890 con 0x7fe15c15cd70 2023-10-16T14:16:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe160872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 0x7fe15c15ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe160872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 0x7fe15c15ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50980/0 (socket says 172.21.15.29:50980) 2023-10-16T14:16:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe160872640 1 -- 172.21.15.29:0/3099837450 learned_addr learned my addr 172.21.15.29:0/3099837450 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe161073640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c0b8480 0x7fe15c155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.127+0000 7fe161874640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cd70 0x7fe15c15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe160872640 1 -- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cd70 msgr2=0x7fe15c15f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe160872640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cd70 0x7fe15c15f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe160872640 1 -- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c0b8480 msgr2=0x7fe15c155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe160872640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c0b8480 0x7fe15c155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe160872640 1 -- 172.21.15.29:0/3099837450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c0094d0 con 0x7fe15c0d0be0 2023-10-16T14:16:39.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe161874640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cd70 0x7fe15c15f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:39.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe160872640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 0x7fe15c15ece0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe14c02b950 tx=0x7fe14c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c03ac60 con 0x7fe15c0d0be0 2023-10-16T14:16:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c03adc0 con 0x7fe15c0d0be0 2023-10-16T14:16:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c0445f0 con 0x7fe15c0d0be0 2023-10-16T14:16:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.128+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c1f5b70 con 0x7fe15c0d0be0 2023-10-16T14:16:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.129+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c1f6370 con 0x7fe15c0d0be0 2023-10-16T14:16:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.130+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c1f6090 con 0x7fe15c0d0be0 2023-10-16T14:16:39.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.131+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe14c04d450 con 0x7fe15c0d0be0 2023-10-16T14:16:39.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.131+0000 7fe151ffb640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe14006ae70 0x7fe14006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.131+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe14c02d040 con 0x7fe15c0d0be0 2023-10-16T14:16:39.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.132+0000 7fe161073640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe14006ae70 0x7fe14006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.133+0000 7fe161073640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe14006ae70 0x7fe14006d330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe158006fd0 tx=0x7fe158008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.136+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe15c1f6090 con 0x7fe15c0d0be0 2023-10-16T14:16:39.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.155+0000 7f7d5b59e640 1 Processor -- start 2023-10-16T14:16:39.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.155+0000 7f7d5b59e640 1 -- start start 2023-10-16T14:16:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d5b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7d5c0b8480 0x7f7d5c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d5b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c15d150 con 0x7f7d5c0d0be0 2023-10-16T14:16:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d5b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c0d1180 con 0x7f7d5c15cd70 2023-10-16T14:16:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d5b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5c0d1460 con 0x7f7d5c0b8480 2023-10-16T14:16:39.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d5ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7d5c0d0be0 0x7f7d5c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46382/0 (socket says 172.21.15.29:46382) 2023-10-16T14:16:39.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d5ad9d640 1 -- 172.21.15.29:0/1892402500 learned_addr learned my addr 172.21.15.29:0/1892402500 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:39.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d51d9b640 1 --2- 172.21.15.29:0/1892402500 >> v2:172.21.15.170:3300/0 conn(0x7f7d5c0b8480 0x7f7d5c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8950302 0 0) 0x7f7d5c15d150 con 0x7f7d5c0d0be0 2023-10-16T14:16:39.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.157+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48003520 con 0x7f7d5c0d0be0 2023-10-16T14:16:39.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3862622545 0 0) 0x7f7d48003520 con 0x7f7d5c0d0be0 2023-10-16T14:16:39.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 >> v2:172.21.15.170:3300/0 conn(0x7f7d5c0b8480 msgr2=0x7f7d5c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 --2- 172.21.15.29:0/1892402500 >> v2:172.21.15.170:3300/0 conn(0x7f7d5c0b8480 0x7f7d5c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 >> v1:172.21.15.170:6789/0 conn(0x7f7d5c15cd70 legacy=0x7f7d5c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c0d1740 con 0x7f7d5c0d0be0 2023-10-16T14:16:39.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7d54001670 con 0x7f7d5c0d0be0 2023-10-16T14:16:39.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 >> v1:172.21.15.29:6789/0 conn(0x7f7d5c0d0be0 legacy=0x7f7d5c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d480039c0 0x7f7d48003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d480047c0 0x7f7d48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480051d0 0x7f7d4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c0d1460 con 0x7f7d480039c0 2023-10-16T14:16:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4800c120 con 0x7f7d480047c0 2023-10-16T14:16:39.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4800c2e0 con 0x7f7d480051d0 2023-10-16T14:16:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d480039c0 0x7f7d48003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.158+0000 7f7d51d9b640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d480047c0 0x7f7d48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5ad9d640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480051d0 0x7f7d4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5a59c640 1 -- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480051d0 msgr2=0x7f7d4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480051d0 0x7f7d4800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5a59c640 1 -- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d480047c0 msgr2=0x7f7d48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d480047c0 0x7f7d48004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5a59c640 1 -- 172.21.15.29:0/1892402500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48003520 con 0x7f7d480039c0 2023-10-16T14:16:39.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.159+0000 7f7d5ad9d640 1 -- 172.21.15.29:0/1892402500 reap_dead start 2023-10-16T14:16:39.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.160+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d480039c0 0x7f7d48003de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c001e20 tx=0x7f7d480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8d93361f41ce25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.160+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c003b70 con 0x7f7d480039c0 2023-10-16T14:16:39.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.160+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c003cd0 con 0x7f7d480039c0 2023-10-16T14:16:39.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.160+0000 7f7d59d9b640 1 -- 172.21.15.29:0/1892402500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c005310 con 0x7f7d480039c0 2023-10-16T14:16:39.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.160+0000 7f7d5b59e640 1 -- 172.21.15.29:0/1892402500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d480039c0 msgr2=0x7f7d48003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.160+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d480039c0 0x7f7d48003de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c001e20 tx=0x7f7d480057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.161+0000 7f7d5b59e640 1 -- 172.21.15.29:0/1892402500 shutdown_connections 2023-10-16T14:16:39.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.161+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/1892402500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d480039c0 0x7f7d48003de0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.161+0000 7f7d5b59e640 1 -- 172.21.15.29:0/1892402500 >> 172.21.15.29:0/1892402500 conn(0x7f7d5c0cf570 msgr2=0x7f7d5c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:39.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.161+0000 7f7d5b59e640 1 -- 172.21.15.29:0/1892402500 shutdown_connections 2023-10-16T14:16:39.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.161+0000 7f7d5b59e640 1 -- 172.21.15.29:0/1892402500 wait complete. 2023-10-16T14:16:39.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.162+0000 7f7d5b59e640 1 Processor -- start 2023-10-16T14:16:39.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.163+0000 7f7d5b59e640 1 -- start start 2023-10-16T14:16:39.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.163+0000 7f7d5b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480039c0 0x7f7d5c0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.163+0000 7f7d5b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d5c15fc70 0x7f7d5c146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.163+0000 7f7d5b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 0x7f7d5c0b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c15ebd0 con 0x7f7d5c1600f0 2023-10-16T14:16:39.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c15eeb0 con 0x7f7d5c15fc70 2023-10-16T14:16:39.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c15f190 con 0x7f7d480039c0 2023-10-16T14:16:39.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 0x7f7d5c0b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 0x7f7d5c0b6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51000/0 (socket says 172.21.15.29:51000) 2023-10-16T14:16:39.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 -- 172.21.15.29:0/3057002734 learned_addr learned my addr 172.21.15.29:0/3057002734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:39.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480039c0 0x7f7d5c0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d51d9b640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d5c15fc70 0x7f7d5c146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 -- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480039c0 msgr2=0x7f7d5c0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480039c0 0x7f7d5c0c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 -- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d5c15fc70 msgr2=0x7f7d5c146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d5c15fc70 0x7f7d5c146b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 -- 172.21.15.29:0/3057002734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c15f470 con 0x7f7d5c1600f0 2023-10-16T14:16:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d51d9b640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d5c15fc70 0x7f7d5c146b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.164+0000 7f7d5ad9d640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 0x7f7d5c0b6ca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0cd0f0 tx=0x7f7d5400e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.165+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54014480 con 0x7f7d5c1600f0 2023-10-16T14:16:39.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.165+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54014600 con 0x7f7d5c1600f0 2023-10-16T14:16:39.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.165+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d540149b0 con 0x7f7d5c1600f0 2023-10-16T14:16:39.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.165+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c15f750 con 0x7f7d5c1600f0 2023-10-16T14:16:39.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.165+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c147040 con 0x7f7d5c1600f0 2023-10-16T14:16:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.166+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5c04e9b0 con 0x7f7d5c1600f0 2023-10-16T14:16:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.167+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7d540114f0 con 0x7f7d5c1600f0 2023-10-16T14:16:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.168+0000 7f7d537fe640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7d3c06ae70 0x7f7d3c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.168+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d54088c70 con 0x7f7d5c1600f0 2023-10-16T14:16:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.168+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7d3c06ae70 0x7f7d3c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.170+0000 7f7d5a59c640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7d3c06ae70 0x7f7d3c06d330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c034000 tx=0x7f7d4c009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.172+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7d54053d10 con 0x7f7d5c1600f0 2023-10-16T14:16:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.333+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe15c1f6090 con 0x7fe15c0d0be0 2023-10-16T14:16:39.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.333+0000 7fe151ffb640 1 -- 172.21.15.29:0/3099837450 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe15c1f6090 con 0x7fe15c0d0be0 2023-10-16T14:16:39.338 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:39.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.335+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe14006ae70 msgr2=0x7fe14006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.335+0000 7fe162075640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe14006ae70 0x7fe14006d330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe158006fd0 tx=0x7fe158008040 comp rx=0 tx=0).stop 2023-10-16T14:16:39.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.335+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 msgr2=0x7fe15c15ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 0x7fe15c15ece0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe14c02b950 tx=0x7fe14c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 shutdown_connections 2023-10-16T14:16:39.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe14006ae70 0x7fe14006d330 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0d0be0 0x7fe15c15ece0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cd70 0x7fe15c15f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 --2- 172.21.15.29:0/3099837450 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c0b8480 0x7fe15c155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 >> 172.21.15.29:0/3099837450 conn(0x7fe15c0cf570 msgr2=0x7fe15c0b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:39.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 shutdown_connections 2023-10-16T14:16:39.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.336+0000 7fe162075640 1 -- 172.21.15.29:0/3099837450 wait complete. 2023-10-16T14:16:39.357 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-10-16T14:16:39.357 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:39.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.361+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7d5c0b4a60 con 0x7f7d5c1600f0 2023-10-16T14:16:39.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.361+0000 7f7d537fe640 1 -- 172.21.15.29:0/3057002734 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d54019070 con 0x7f7d5c1600f0 2023-10-16T14:16:39.365 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-16T14:16:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.363+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7d3c06ae70 msgr2=0x7f7d3c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.363+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7d3c06ae70 0x7f7d3c06d330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c034000 tx=0x7f7d4c009030 comp rx=0 tx=0).stop 2023-10-16T14:16:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.363+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 msgr2=0x7f7d5c0b6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.363+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 0x7f7d5c0b6ca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0cd0f0 tx=0x7f7d5400e890 comp rx=0 tx=0).stop 2023-10-16T14:16:39.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 shutdown_connections 2023-10-16T14:16:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7d3c06ae70 0x7f7d3c06d330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d5c1600f0 0x7f7d5c0b6ca0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d5c15fc70 0x7f7d5c146b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 --2- 172.21.15.29:0/3057002734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d480039c0 0x7f7d5c0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 >> 172.21.15.29:0/3057002734 conn(0x7f7d5c0cf570 msgr2=0x7f7d5c0ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 shutdown_connections 2023-10-16T14:16:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.364+0000 7f7d5b59e640 1 -- 172.21.15.29:0/3057002734 wait complete. 2023-10-16T14:16:39.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-10-16T14:16:39.384 DEBUG:teuthology.parallel:result is None 2023-10-16T14:16:39.384 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T14:16:39.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:16:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.643+0000 7f8c5b13c640 1 Processor -- start 2023-10-16T14:16:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.644+0000 7f8c5b13c640 1 -- start start 2023-10-16T14:16:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.645+0000 7f8c5b13c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c540cd0c0 0x7f8c541615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.645+0000 7f8c5b13c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8c540b0410 0x7f8c540b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.645+0000 7f8c5b13c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c5415d3f0 0x7f8c5415d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c5b13c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c540b0de0 con 0x7f8c540cd0c0 2023-10-16T14:16:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c5b13c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5415e3e0 con 0x7f8c540b0410 2023-10-16T14:16:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c5b13c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c540cd4a0 con 0x7f8c5415d3f0 2023-10-16T14:16:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c540cd0c0 0x7f8c541615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c540cd0c0 0x7f8c541615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51006/0 (socket says 172.21.15.29:51006) 2023-10-16T14:16:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 -- 172.21.15.29:0/1687503673 learned_addr learned my addr 172.21.15.29:0/1687503673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 -- 172.21.15.29:0/1687503673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c5415d3f0 msgr2=0x7f8c5415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c596d9640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c5415d3f0 0x7f8c5415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c53fff640 1 --2- 172.21.15.29:0/1687503673 >> v2:172.21.15.170:3300/0 conn(0x7f8c540b0410 0x7f8c540b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c5415d3f0 0x7f8c5415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 -- 172.21.15.29:0/1687503673 >> v2:172.21.15.170:3300/0 conn(0x7f8c540b0410 msgr2=0x7f8c540b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 --2- 172.21.15.29:0/1687503673 >> v2:172.21.15.170:3300/0 conn(0x7f8c540b0410 0x7f8c540b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c58ed8640 1 -- 172.21.15.29:0/1687503673 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5415e100 con 0x7f8c540cd0c0 2023-10-16T14:16:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.646+0000 7f8c596d9640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c5415d3f0 0x7f8c5415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.647+0000 7f8c58ed8640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c541615f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8c540cdad0 tx=0x7f8c4802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4787ceefac27afb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.648+0000 7f8c537fe640 1 -- 172.21.15.29:0/1687503673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c480317e0 con 0x7f8c540cd0c0 2023-10-16T14:16:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.648+0000 7f8c537fe640 1 -- 172.21.15.29:0/1687503673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c48031940 con 0x7f8c540cd0c0 2023-10-16T14:16:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.648+0000 7f8c537fe640 1 -- 172.21.15.29:0/1687503673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c48031cd0 con 0x7f8c540cd0c0 2023-10-16T14:16:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.648+0000 7f8c5b13c640 1 -- 172.21.15.29:0/1687503673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 msgr2=0x7f8c541615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.648+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c541615f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8c540cdad0 tx=0x7f8c4802b880 comp rx=0 tx=0).stop 2023-10-16T14:16:39.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 -- 172.21.15.29:0/1687503673 shutdown_connections 2023-10-16T14:16:39.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c5415d3f0 0x7f8c5415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/1687503673 >> v2:172.21.15.170:3300/0 conn(0x7f8c540b0410 0x7f8c540b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/1687503673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c541615f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 -- 172.21.15.29:0/1687503673 >> 172.21.15.29:0/1687503673 conn(0x7f8c540cb410 msgr2=0x7f8c540cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:39.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 -- 172.21.15.29:0/1687503673 shutdown_connections 2023-10-16T14:16:39.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.649+0000 7f8c5b13c640 1 -- 172.21.15.29:0/1687503673 wait complete. 2023-10-16T14:16:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.650+0000 7f8c5b13c640 1 Processor -- start 2023-10-16T14:16:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.650+0000 7f8c5b13c640 1 -- start start 2023-10-16T14:16:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.651+0000 7f8c5b13c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c540b0410 0x7f8c541563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.651+0000 7f8c5b13c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c54159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.651+0000 7f8c5b13c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c5415d3f0 0x7f8c54159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.651+0000 7f8c5b13c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c540b1550 con 0x7f8c540cd0c0 2023-10-16T14:16:39.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.651+0000 7f8c5b13c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5415a2a0 con 0x7f8c5415d3f0 2023-10-16T14:16:39.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.651+0000 7f8c5b13c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c54156930 con 0x7f8c540b0410 2023-10-16T14:16:39.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c54159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c54159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51020/0 (socket says 172.21.15.29:51020) 2023-10-16T14:16:39.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 -- 172.21.15.29:0/4291093391 learned_addr learned my addr 172.21.15.29:0/4291093391 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:39.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c596d9640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c5415d3f0 0x7f8c54159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c58ed8640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c540b0410 0x7f8c541563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 -- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c540b0410 msgr2=0x7f8c541563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c540b0410 0x7f8c541563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 -- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c5415d3f0 msgr2=0x7f8c54159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c5415d3f0 0x7f8c54159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 -- 172.21.15.29:0/4291093391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c480094d0 con 0x7f8c540cd0c0 2023-10-16T14:16:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c53fff640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c54159820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000c570 tx=0x7f8c4000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c40011800 con 0x7f8c540cd0c0 2023-10-16T14:16:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c40011960 con 0x7f8c540cd0c0 2023-10-16T14:16:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c40011d10 con 0x7f8c540cd0c0 2023-10-16T14:16:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.652+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5406ced0 con 0x7f8c540cd0c0 2023-10-16T14:16:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.653+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c540b1760 con 0x7f8c540cd0c0 2023-10-16T14:16:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.654+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5404e160 con 0x7f8c540cd0c0 2023-10-16T14:16:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.655+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c40018360 con 0x7f8c540cd0c0 2023-10-16T14:16:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.655+0000 7f8c517fa640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c3806ae70 0x7f8c3806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.656+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c400895e0 con 0x7f8c540cd0c0 2023-10-16T14:16:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.656+0000 7f8c58ed8640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c3806ae70 0x7f8c3806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.657+0000 7f8c58ed8640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c3806ae70 0x7f8c3806d330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c48036000 tx=0x7f8c4803d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.660+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c40054690 con 0x7f8c540cd0c0 2023-10-16T14:16:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.851+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c5406d3f0 con 0x7f8c3806ae70 2023-10-16T14:16:39.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.874+0000 7f8c517fa640 1 -- 172.21.15.29:0/4291093391 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264435 (secure 0 0 0) 0x7f8c5406d3f0 con 0x7f8c3806ae70 2023-10-16T14:16:39.880 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:16:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c3806ae70 msgr2=0x7f8c3806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c3806ae70 0x7f8c3806d330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c48036000 tx=0x7f8c4803d040 comp rx=0 tx=0).stop 2023-10-16T14:16:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 msgr2=0x7f8c54159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c54159820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000c570 tx=0x7f8c4000ca30 comp rx=0 tx=0).stop 2023-10-16T14:16:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 shutdown_connections 2023-10-16T14:16:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c3806ae70 0x7f8c3806d330 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c540cd0c0 0x7f8c54159820 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c5415d3f0 0x7f8c54159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 --2- 172.21.15.29:0/4291093391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c540b0410 0x7f8c541563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:39.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 >> 172.21.15.29:0/4291093391 conn(0x7f8c540cb410 msgr2=0x7f8c540af250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:39.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 shutdown_connections 2023-10-16T14:16:39.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:39.880+0000 7f8c5b13c640 1 -- 172.21.15.29:0/4291093391 wait complete. 2023-10-16T14:16:39.907 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T14:16:39.815072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252284,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722436,"statfs":{"total":773094113280,"available":772835774464,"internally_reserved":0,"allocated":2891776,"data_stored":1752740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":255315513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005076"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799225+0000","last_change":"2023-10-16T14:16:29.830118+0000","last_active":"2023-10-16T14:16:30.799225+0000","last_peered":"2023-10-16T14:16:30.799225+0000","last_clean":"2023-10-16T14:16:30.799225+0000","last_became_active":"2023-10-16T14:16:29.829372+0000","last_became_peered":"2023-10-16T14:16:29.829372+0000","last_unstale":"2023-10-16T14:16:30.799225+0000","last_undegraded":"2023-10-16T14:16:30.799225+0000","last_fullsized":"2023-10-16T14:16:30.799225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794763+0000","last_change":"2023-10-16T14:16:29.818444+0000","last_active":"2023-10-16T14:16:30.794763+0000","last_peered":"2023-10-16T14:16:30.794763+0000","last_clean":"2023-10-16T14:16:30.794763+0000","last_became_active":"2023-10-16T14:16:29.816215+0000","last_became_peered":"2023-10-16T14:16:29.816215+0000","last_unstale":"2023-10-16T14:16:30.794763+0000","last_undegraded":"2023-10-16T14:16:30.794763+0000","last_fullsized":"2023-10-16T14:16:30.794763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326440+0000","last_change":"2023-10-16T14:16:29.817304+0000","last_active":"2023-10-16T14:16:32.326440+0000","last_peered":"2023-10-16T14:16:32.326440+0000","last_clean":"2023-10-16T14:16:32.326440+0000","last_became_active":"2023-10-16T14:16:29.817037+0000","last_became_peered":"2023-10-16T14:16:29.817037+0000","last_unstale":"2023-10-16T14:16:32.326440+0000","last_undegraded":"2023-10-16T14:16:32.326440+0000","last_fullsized":"2023-10-16T14:16:32.326440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827089+0000","last_change":"2023-10-16T14:16:29.816180+0000","last_active":"2023-10-16T14:16:31.827089+0000","last_peered":"2023-10-16T14:16:31.827089+0000","last_clean":"2023-10-16T14:16:31.827089+0000","last_became_active":"2023-10-16T14:16:29.815841+0000","last_became_peered":"2023-10-16T14:16:29.815841+0000","last_unstale":"2023-10-16T14:16:31.827089+0000","last_undegraded":"2023-10-16T14:16:31.827089+0000","last_fullsized":"2023-10-16T14:16:31.827089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798716+0000","last_change":"2023-10-16T14:16:29.813648+0000","last_active":"2023-10-16T14:16:30.798716+0000","last_peered":"2023-10-16T14:16:30.798716+0000","last_clean":"2023-10-16T14:16:30.798716+0000","last_became_active":"2023-10-16T14:16:29.813365+0000","last_became_peered":"2023-10-16T14:16:29.813365+0000","last_unstale":"2023-10-16T14:16:30.798716+0000","last_undegraded":"2023-10-16T14:16:30.798716+0000","last_fullsized":"2023-10-16T14:16:30.798716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825920+0000","last_change":"2023-10-16T14:16:29.818850+0000","last_active":"2023-10-16T14:16:31.825920+0000","last_peered":"2023-10-16T14:16:31.825920+0000","last_clean":"2023-10-16T14:16:31.825920+0000","last_became_active":"2023-10-16T14:16:29.818486+0000","last_became_peered":"2023-10-16T14:16:29.818486+0000","last_unstale":"2023-10-16T14:16:31.825920+0000","last_undegraded":"2023-10-16T14:16:31.825920+0000","last_fullsized":"2023-10-16T14:16:31.825920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796628+0000","last_change":"2023-10-16T14:16:29.814386+0000","last_active":"2023-10-16T14:16:30.796628+0000","last_peered":"2023-10-16T14:16:30.796628+0000","last_clean":"2023-10-16T14:16:30.796628+0000","last_became_active":"2023-10-16T14:16:29.812379+0000","last_became_peered":"2023-10-16T14:16:29.812379+0000","last_unstale":"2023-10-16T14:16:30.796628+0000","last_undegraded":"2023-10-16T14:16:30.796628+0000","last_fullsized":"2023-10-16T14:16:30.796628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667250+0000","last_change":"2023-10-16T14:16:29.823474+0000","last_active":"2023-10-16T14:16:31.667250+0000","last_peered":"2023-10-16T14:16:31.667250+0000","last_clean":"2023-10-16T14:16:31.667250+0000","last_became_active":"2023-10-16T14:16:29.823172+0000","last_became_peered":"2023-10-16T14:16:29.823172+0000","last_unstale":"2023-10-16T14:16:31.667250+0000","last_undegraded":"2023-10-16T14:16:31.667250+0000","last_fullsized":"2023-10-16T14:16:31.667250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799207+0000","last_change":"2023-10-16T14:16:29.830964+0000","last_active":"2023-10-16T14:16:30.799207+0000","last_peered":"2023-10-16T14:16:30.799207+0000","last_clean":"2023-10-16T14:16:30.799207+0000","last_became_active":"2023-10-16T14:16:29.829757+0000","last_became_peered":"2023-10-16T14:16:29.829757+0000","last_unstale":"2023-10-16T14:16:30.799207+0000","last_undegraded":"2023-10-16T14:16:30.799207+0000","last_fullsized":"2023-10-16T14:16:30.799207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961588+0000","last_change":"2023-10-16T14:16:29.822512+0000","last_active":"2023-10-16T14:16:30.961588+0000","last_peered":"2023-10-16T14:16:30.961588+0000","last_clean":"2023-10-16T14:16:30.961588+0000","last_became_active":"2023-10-16T14:16:29.822050+0000","last_became_peered":"2023-10-16T14:16:29.822050+0000","last_unstale":"2023-10-16T14:16:30.961588+0000","last_undegraded":"2023-10-16T14:16:30.961588+0000","last_fullsized":"2023-10-16T14:16:30.961588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961278+0000","last_change":"2023-10-16T14:16:29.819724+0000","last_active":"2023-10-16T14:16:30.961278+0000","last_peered":"2023-10-16T14:16:30.961278+0000","last_clean":"2023-10-16T14:16:30.961278+0000","last_became_active":"2023-10-16T14:16:29.819467+0000","last_became_peered":"2023-10-16T14:16:29.819467+0000","last_unstale":"2023-10-16T14:16:30.961278+0000","last_undegraded":"2023-10-16T14:16:30.961278+0000","last_fullsized":"2023-10-16T14:16:30.961278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797593+0000","last_change":"2023-10-16T14:16:29.819218+0000","last_active":"2023-10-16T14:16:30.797593+0000","last_peered":"2023-10-16T14:16:30.797593+0000","last_clean":"2023-10-16T14:16:30.797593+0000","last_became_active":"2023-10-16T14:16:29.818622+0000","last_became_peered":"2023-10-16T14:16:29.818622+0000","last_unstale":"2023-10-16T14:16:30.797593+0000","last_undegraded":"2023-10-16T14:16:30.797593+0000","last_fullsized":"2023-10-16T14:16:30.797593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668213+0000","last_change":"2023-10-16T14:16:29.817208+0000","last_active":"2023-10-16T14:16:31.668213+0000","last_peered":"2023-10-16T14:16:31.668213+0000","last_clean":"2023-10-16T14:16:31.668213+0000","last_became_active":"2023-10-16T14:16:29.816175+0000","last_became_peered":"2023-10-16T14:16:29.816175+0000","last_unstale":"2023-10-16T14:16:31.668213+0000","last_undegraded":"2023-10-16T14:16:31.668213+0000","last_fullsized":"2023-10-16T14:16:31.668213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325899+0000","last_change":"2023-10-16T14:16:29.812726+0000","last_active":"2023-10-16T14:16:32.325899+0000","last_peered":"2023-10-16T14:16:32.325899+0000","last_clean":"2023-10-16T14:16:32.325899+0000","last_became_active":"2023-10-16T14:16:29.812358+0000","last_became_peered":"2023-10-16T14:16:29.812358+0000","last_unstale":"2023-10-16T14:16:32.325899+0000","last_undegraded":"2023-10-16T14:16:32.325899+0000","last_fullsized":"2023-10-16T14:16:32.325899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:59.947139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794118+0000","last_change":"2023-10-16T14:16:29.820374+0000","last_active":"2023-10-16T14:16:30.794118+0000","last_peered":"2023-10-16T14:16:30.794118+0000","last_clean":"2023-10-16T14:16:30.794118+0000","last_became_active":"2023-10-16T14:16:29.820053+0000","last_became_peered":"2023-10-16T14:16:29.820053+0000","last_unstale":"2023-10-16T14:16:30.794118+0000","last_undegraded":"2023-10-16T14:16:30.794118+0000","last_fullsized":"2023-10-16T14:16:30.794118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825735+0000","last_change":"2023-10-16T14:16:29.819714+0000","last_active":"2023-10-16T14:16:31.825735+0000","last_peered":"2023-10-16T14:16:31.825735+0000","last_clean":"2023-10-16T14:16:31.825735+0000","last_became_active":"2023-10-16T14:16:29.819444+0000","last_became_peered":"2023-10-16T14:16:29.819444+0000","last_unstale":"2023-10-16T14:16:31.825735+0000","last_undegraded":"2023-10-16T14:16:31.825735+0000","last_fullsized":"2023-10-16T14:16:31.825735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826922+0000","last_change":"2023-10-16T14:16:29.817726+0000","last_active":"2023-10-16T14:16:31.826922+0000","last_peered":"2023-10-16T14:16:31.826922+0000","last_clean":"2023-10-16T14:16:31.826922+0000","last_became_active":"2023-10-16T14:16:29.813107+0000","last_became_peered":"2023-10-16T14:16:29.813107+0000","last_unstale":"2023-10-16T14:16:31.826922+0000","last_undegraded":"2023-10-16T14:16:31.826922+0000","last_fullsized":"2023-10-16T14:16:31.826922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325856+0000","last_change":"2023-10-16T14:16:29.825557+0000","last_active":"2023-10-16T14:16:32.325856+0000","last_peered":"2023-10-16T14:16:32.325856+0000","last_clean":"2023-10-16T14:16:32.325856+0000","last_became_active":"2023-10-16T14:16:29.825131+0000","last_became_peered":"2023-10-16T14:16:29.825131+0000","last_unstale":"2023-10-16T14:16:32.325856+0000","last_undegraded":"2023-10-16T14:16:32.325856+0000","last_fullsized":"2023-10-16T14:16:32.325856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798289+0000","last_change":"2023-10-16T14:16:29.813672+0000","last_active":"2023-10-16T14:16:30.798289+0000","last_peered":"2023-10-16T14:16:30.798289+0000","last_clean":"2023-10-16T14:16:30.798289+0000","last_became_active":"2023-10-16T14:16:29.813271+0000","last_became_peered":"2023-10-16T14:16:29.813271+0000","last_unstale":"2023-10-16T14:16:30.798289+0000","last_undegraded":"2023-10-16T14:16:30.798289+0000","last_fullsized":"2023-10-16T14:16:30.798289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961292+0000","last_change":"2023-10-16T14:16:29.821916+0000","last_active":"2023-10-16T14:16:30.961292+0000","last_peered":"2023-10-16T14:16:30.961292+0000","last_clean":"2023-10-16T14:16:30.961292+0000","last_became_active":"2023-10-16T14:16:29.821561+0000","last_became_peered":"2023-10-16T14:16:29.821561+0000","last_unstale":"2023-10-16T14:16:30.961292+0000","last_undegraded":"2023-10-16T14:16:30.961292+0000","last_fullsized":"2023-10-16T14:16:30.961292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326088+0000","last_change":"2023-10-16T14:16:29.809106+0000","last_active":"2023-10-16T14:16:32.326088+0000","last_peered":"2023-10-16T14:16:32.326088+0000","last_clean":"2023-10-16T14:16:32.326088+0000","last_became_active":"2023-10-16T14:16:29.808677+0000","last_became_peered":"2023-10-16T14:16:29.808677+0000","last_unstale":"2023-10-16T14:16:32.326088+0000","last_undegraded":"2023-10-16T14:16:32.326088+0000","last_fullsized":"2023-10-16T14:16:32.326088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:02.431928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.814365+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.814022+0000","last_became_peered":"2023-10-16T14:16:29.814022+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667249+0000","last_change":"2023-10-16T14:16:29.826061+0000","last_active":"2023-10-16T14:16:31.667249+0000","last_peered":"2023-10-16T14:16:31.667249+0000","last_clean":"2023-10-16T14:16:31.667249+0000","last_became_active":"2023-10-16T14:16:29.825662+0000","last_became_peered":"2023-10-16T14:16:29.825662+0000","last_unstale":"2023-10-16T14:16:31.667249+0000","last_undegraded":"2023-10-16T14:16:31.667249+0000","last_fullsized":"2023-10-16T14:16:31.667249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826173+0000","last_change":"2023-10-16T14:16:29.822279+0000","last_active":"2023-10-16T14:16:31.826173+0000","last_peered":"2023-10-16T14:16:31.826173+0000","last_clean":"2023-10-16T14:16:31.826173+0000","last_became_active":"2023-10-16T14:16:29.822022+0000","last_became_peered":"2023-10-16T14:16:29.822022+0000","last_unstale":"2023-10-16T14:16:31.826173+0000","last_undegraded":"2023-10-16T14:16:31.826173+0000","last_fullsized":"2023-10-16T14:16:31.826173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668113+0000","last_change":"2023-10-16T14:16:29.823044+0000","last_active":"2023-10-16T14:16:31.668113+0000","last_peered":"2023-10-16T14:16:31.668113+0000","last_clean":"2023-10-16T14:16:31.668113+0000","last_became_active":"2023-10-16T14:16:29.822705+0000","last_became_peered":"2023-10-16T14:16:29.822705+0000","last_unstale":"2023-10-16T14:16:31.668113+0000","last_undegraded":"2023-10-16T14:16:31.668113+0000","last_fullsized":"2023-10-16T14:16:31.668113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961559+0000","last_change":"2023-10-16T14:16:29.822259+0000","last_active":"2023-10-16T14:16:30.961559+0000","last_peered":"2023-10-16T14:16:30.961559+0000","last_clean":"2023-10-16T14:16:30.961559+0000","last_became_active":"2023-10-16T14:16:29.821719+0000","last_became_peered":"2023-10-16T14:16:29.821719+0000","last_unstale":"2023-10-16T14:16:30.961559+0000","last_undegraded":"2023-10-16T14:16:30.961559+0000","last_fullsized":"2023-10-16T14:16:30.961559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794706+0000","last_change":"2023-10-16T14:16:29.821655+0000","last_active":"2023-10-16T14:16:30.794706+0000","last_peered":"2023-10-16T14:16:30.794706+0000","last_clean":"2023-10-16T14:16:30.794706+0000","last_became_active":"2023-10-16T14:16:29.821357+0000","last_became_peered":"2023-10-16T14:16:29.821357+0000","last_unstale":"2023-10-16T14:16:30.794706+0000","last_undegraded":"2023-10-16T14:16:30.794706+0000","last_fullsized":"2023-10-16T14:16:30.794706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668435+0000","last_change":"2023-10-16T14:16:29.815493+0000","last_active":"2023-10-16T14:16:31.668435+0000","last_peered":"2023-10-16T14:16:31.668435+0000","last_clean":"2023-10-16T14:16:31.668435+0000","last_became_active":"2023-10-16T14:16:29.815203+0000","last_became_peered":"2023-10-16T14:16:29.815203+0000","last_unstale":"2023-10-16T14:16:31.668435+0000","last_undegraded":"2023-10-16T14:16:31.668435+0000","last_fullsized":"2023-10-16T14:16:31.668435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326367+0000","last_change":"2023-10-16T14:16:29.809001+0000","last_active":"2023-10-16T14:16:32.326367+0000","last_peered":"2023-10-16T14:16:32.326367+0000","last_clean":"2023-10-16T14:16:32.326367+0000","last_became_active":"2023-10-16T14:16:29.808169+0000","last_became_peered":"2023-10-16T14:16:29.808169+0000","last_unstale":"2023-10-16T14:16:32.326367+0000","last_undegraded":"2023-10-16T14:16:32.326367+0000","last_fullsized":"2023-10-16T14:16:32.326367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826340+0000","last_change":"2023-10-16T14:16:29.812984+0000","last_active":"2023-10-16T14:16:31.826340+0000","last_peered":"2023-10-16T14:16:31.826340+0000","last_clean":"2023-10-16T14:16:31.826340+0000","last_became_active":"2023-10-16T14:16:29.812569+0000","last_became_peered":"2023-10-16T14:16:29.812569+0000","last_unstale":"2023-10-16T14:16:31.826340+0000","last_undegraded":"2023-10-16T14:16:31.826340+0000","last_fullsized":"2023-10-16T14:16:31.826340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325543+0000","last_change":"2023-10-16T14:16:29.821793+0000","last_active":"2023-10-16T14:16:32.325543+0000","last_peered":"2023-10-16T14:16:32.325543+0000","last_clean":"2023-10-16T14:16:32.325543+0000","last_became_active":"2023-10-16T14:16:29.819170+0000","last_became_peered":"2023-10-16T14:16:29.819170+0000","last_unstale":"2023-10-16T14:16:32.325543+0000","last_undegraded":"2023-10-16T14:16:32.325543+0000","last_fullsized":"2023-10-16T14:16:32.325543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961200+0000","last_change":"2023-10-16T14:16:29.821091+0000","last_active":"2023-10-16T14:16:30.961200+0000","last_peered":"2023-10-16T14:16:30.961200+0000","last_clean":"2023-10-16T14:16:30.961200+0000","last_became_active":"2023-10-16T14:16:29.818555+0000","last_became_peered":"2023-10-16T14:16:29.818555+0000","last_unstale":"2023-10-16T14:16:30.961200+0000","last_undegraded":"2023-10-16T14:16:30.961200+0000","last_fullsized":"2023-10-16T14:16:30.961200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797715+0000","last_change":"2023-10-16T14:16:29.812942+0000","last_active":"2023-10-16T14:16:30.797715+0000","last_peered":"2023-10-16T14:16:30.797715+0000","last_clean":"2023-10-16T14:16:30.797715+0000","last_became_active":"2023-10-16T14:16:29.811742+0000","last_became_peered":"2023-10-16T14:16:29.811742+0000","last_unstale":"2023-10-16T14:16:30.797715+0000","last_undegraded":"2023-10-16T14:16:30.797715+0000","last_fullsized":"2023-10-16T14:16:30.797715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668347+0000","last_change":"2023-10-16T14:16:29.050890+0000","last_active":"2023-10-16T14:16:31.668347+0000","last_peered":"2023-10-16T14:16:31.668347+0000","last_clean":"2023-10-16T14:16:31.668347+0000","last_became_active":"2023-10-16T14:16:29.049249+0000","last_became_peered":"2023-10-16T14:16:29.049249+0000","last_unstale":"2023-10-16T14:16:31.668347+0000","last_undegraded":"2023-10-16T14:16:31.668347+0000","last_fullsized":"2023-10-16T14:16:31.668347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326412+0000","last_change":"2023-10-16T14:16:29.820123+0000","last_active":"2023-10-16T14:16:32.326412+0000","last_peered":"2023-10-16T14:16:32.326412+0000","last_clean":"2023-10-16T14:16:32.326412+0000","last_became_active":"2023-10-16T14:16:29.819857+0000","last_became_peered":"2023-10-16T14:16:29.819857+0000","last_unstale":"2023-10-16T14:16:32.326412+0000","last_undegraded":"2023-10-16T14:16:32.326412+0000","last_fullsized":"2023-10-16T14:16:32.326412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960647+0000","last_change":"2023-10-16T14:16:29.053825+0000","last_active":"2023-10-16T14:16:30.960647+0000","last_peered":"2023-10-16T14:16:30.960647+0000","last_clean":"2023-10-16T14:16:30.960647+0000","last_became_active":"2023-10-16T14:16:29.051055+0000","last_became_peered":"2023-10-16T14:16:29.051055+0000","last_unstale":"2023-10-16T14:16:30.960647+0000","last_undegraded":"2023-10-16T14:16:30.960647+0000","last_fullsized":"2023-10-16T14:16:30.960647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667594+0000","last_change":"2023-10-16T14:16:29.055594+0000","last_active":"2023-10-16T14:16:31.667594+0000","last_peered":"2023-10-16T14:16:31.667594+0000","last_clean":"2023-10-16T14:16:31.667594+0000","last_became_active":"2023-10-16T14:16:29.055260+0000","last_became_peered":"2023-10-16T14:16:29.055260+0000","last_unstale":"2023-10-16T14:16:31.667594+0000","last_undegraded":"2023-10-16T14:16:31.667594+0000","last_fullsized":"2023-10-16T14:16:31.667594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667384+0000","last_change":"2023-10-16T14:16:29.817463+0000","last_active":"2023-10-16T14:16:31.667384+0000","last_peered":"2023-10-16T14:16:31.667384+0000","last_clean":"2023-10-16T14:16:31.667384+0000","last_became_active":"2023-10-16T14:16:29.817155+0000","last_became_peered":"2023-10-16T14:16:29.817155+0000","last_unstale":"2023-10-16T14:16:31.667384+0000","last_undegraded":"2023-10-16T14:16:31.667384+0000","last_fullsized":"2023-10-16T14:16:31.667384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.047697+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.047399+0000","last_became_peered":"2023-10-16T14:16:29.047399+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794058+0000","last_change":"2023-10-16T14:16:29.811517+0000","last_active":"2023-10-16T14:16:30.794058+0000","last_peered":"2023-10-16T14:16:30.794058+0000","last_clean":"2023-10-16T14:16:30.794058+0000","last_became_active":"2023-10-16T14:16:29.811178+0000","last_became_peered":"2023-10-16T14:16:29.811178+0000","last_unstale":"2023-10-16T14:16:30.794058+0000","last_undegraded":"2023-10-16T14:16:30.794058+0000","last_fullsized":"2023-10-16T14:16:30.794058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961080+0000","last_change":"2023-10-16T14:16:29.061919+0000","last_active":"2023-10-16T14:16:30.961080+0000","last_peered":"2023-10-16T14:16:30.961080+0000","last_clean":"2023-10-16T14:16:30.961080+0000","last_became_active":"2023-10-16T14:16:29.061130+0000","last_became_peered":"2023-10-16T14:16:29.061130+0000","last_unstale":"2023-10-16T14:16:30.961080+0000","last_undegraded":"2023-10-16T14:16:30.961080+0000","last_fullsized":"2023-10-16T14:16:30.961080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826710+0000","last_change":"2023-10-16T14:16:29.817592+0000","last_active":"2023-10-16T14:16:31.826710+0000","last_peered":"2023-10-16T14:16:31.826710+0000","last_clean":"2023-10-16T14:16:31.826710+0000","last_became_active":"2023-10-16T14:16:29.817291+0000","last_became_peered":"2023-10-16T14:16:29.817291+0000","last_unstale":"2023-10-16T14:16:31.826710+0000","last_undegraded":"2023-10-16T14:16:31.826710+0000","last_fullsized":"2023-10-16T14:16:31.826710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826905+0000","last_change":"2023-10-16T14:16:29.059877+0000","last_active":"2023-10-16T14:16:31.826905+0000","last_peered":"2023-10-16T14:16:31.826905+0000","last_clean":"2023-10-16T14:16:31.826905+0000","last_became_active":"2023-10-16T14:16:29.059080+0000","last_became_peered":"2023-10-16T14:16:29.059080+0000","last_unstale":"2023-10-16T14:16:31.826905+0000","last_undegraded":"2023-10-16T14:16:31.826905+0000","last_fullsized":"2023-10-16T14:16:31.826905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:40.688481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326692+0000","last_change":"2023-10-16T14:16:29.827311+0000","last_active":"2023-10-16T14:16:32.326692+0000","last_peered":"2023-10-16T14:16:32.326692+0000","last_clean":"2023-10-16T14:16:32.326692+0000","last_became_active":"2023-10-16T14:16:29.827004+0000","last_became_peered":"2023-10-16T14:16:29.827004+0000","last_unstale":"2023-10-16T14:16:32.326692+0000","last_undegraded":"2023-10-16T14:16:32.326692+0000","last_fullsized":"2023-10-16T14:16:32.326692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794725+0000","last_change":"2023-10-16T14:16:29.059819+0000","last_active":"2023-10-16T14:16:30.794725+0000","last_peered":"2023-10-16T14:16:30.794725+0000","last_clean":"2023-10-16T14:16:30.794725+0000","last_became_active":"2023-10-16T14:16:29.059474+0000","last_became_peered":"2023-10-16T14:16:29.059474+0000","last_unstale":"2023-10-16T14:16:30.794725+0000","last_undegraded":"2023-10-16T14:16:30.794725+0000","last_fullsized":"2023-10-16T14:16:30.794725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797712+0000","last_change":"2023-10-16T14:16:29.813588+0000","last_active":"2023-10-16T14:16:30.797712+0000","last_peered":"2023-10-16T14:16:30.797712+0000","last_clean":"2023-10-16T14:16:30.797712+0000","last_became_active":"2023-10-16T14:16:29.813342+0000","last_became_peered":"2023-10-16T14:16:29.813342+0000","last_unstale":"2023-10-16T14:16:30.797712+0000","last_undegraded":"2023-10-16T14:16:30.797712+0000","last_fullsized":"2023-10-16T14:16:30.797712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:36.779577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961265+0000","last_change":"2023-10-16T14:16:29.057707+0000","last_active":"2023-10-16T14:16:30.961265+0000","last_peered":"2023-10-16T14:16:30.961265+0000","last_clean":"2023-10-16T14:16:30.961265+0000","last_became_active":"2023-10-16T14:16:29.054590+0000","last_became_peered":"2023-10-16T14:16:29.054590+0000","last_unstale":"2023-10-16T14:16:30.961265+0000","last_undegraded":"2023-10-16T14:16:30.961265+0000","last_fullsized":"2023-10-16T14:16:30.961265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797570+0000","last_change":"2023-10-16T14:16:29.810243+0000","last_active":"2023-10-16T14:16:30.797570+0000","last_peered":"2023-10-16T14:16:30.797570+0000","last_clean":"2023-10-16T14:16:30.797570+0000","last_became_active":"2023-10-16T14:16:29.809877+0000","last_became_peered":"2023-10-16T14:16:29.809877+0000","last_unstale":"2023-10-16T14:16:30.797570+0000","last_undegraded":"2023-10-16T14:16:30.797570+0000","last_fullsized":"2023-10-16T14:16:30.797570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326441+0000","last_change":"2023-10-16T14:16:29.046872+0000","last_active":"2023-10-16T14:16:32.326441+0000","last_peered":"2023-10-16T14:16:32.326441+0000","last_clean":"2023-10-16T14:16:32.326441+0000","last_became_active":"2023-10-16T14:16:29.046411+0000","last_became_peered":"2023-10-16T14:16:29.046411+0000","last_unstale":"2023-10-16T14:16:32.326441+0000","last_undegraded":"2023-10-16T14:16:32.326441+0000","last_fullsized":"2023-10-16T14:16:32.326441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:48.638693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326040+0000","last_change":"2023-10-16T14:16:29.810247+0000","last_active":"2023-10-16T14:16:32.326040+0000","last_peered":"2023-10-16T14:16:32.326040+0000","last_clean":"2023-10-16T14:16:32.326040+0000","last_became_active":"2023-10-16T14:16:29.809830+0000","last_became_peered":"2023-10-16T14:16:29.809830+0000","last_unstale":"2023-10-16T14:16:32.326040+0000","last_undegraded":"2023-10-16T14:16:32.326040+0000","last_fullsized":"2023-10-16T14:16:32.326040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:26.623468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826189+0000","last_change":"2023-10-16T14:16:29.049519+0000","last_active":"2023-10-16T14:16:31.826189+0000","last_peered":"2023-10-16T14:16:31.826189+0000","last_clean":"2023-10-16T14:16:31.826189+0000","last_became_active":"2023-10-16T14:16:29.049203+0000","last_became_peered":"2023-10-16T14:16:29.049203+0000","last_unstale":"2023-10-16T14:16:31.826189+0000","last_undegraded":"2023-10-16T14:16:31.826189+0000","last_fullsized":"2023-10-16T14:16:31.826189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:38.962308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798901+0000","last_change":"2023-10-16T14:16:29.824625+0000","last_active":"2023-10-16T14:16:30.798901+0000","last_peered":"2023-10-16T14:16:30.798901+0000","last_clean":"2023-10-16T14:16:30.798901+0000","last_became_active":"2023-10-16T14:16:29.818988+0000","last_became_peered":"2023-10-16T14:16:29.818988+0000","last_unstale":"2023-10-16T14:16:30.798901+0000","last_undegraded":"2023-10-16T14:16:30.798901+0000","last_fullsized":"2023-10-16T14:16:30.798901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960552+0000","last_change":"2023-10-16T14:16:29.054389+0000","last_active":"2023-10-16T14:16:30.960552+0000","last_peered":"2023-10-16T14:16:30.960552+0000","last_clean":"2023-10-16T14:16:30.960552+0000","last_became_active":"2023-10-16T14:16:29.053542+0000","last_became_peered":"2023-10-16T14:16:29.053542+0000","last_unstale":"2023-10-16T14:16:30.960552+0000","last_undegraded":"2023-10-16T14:16:30.960552+0000","last_fullsized":"2023-10-16T14:16:30.960552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960683+0000","last_change":"2023-10-16T14:16:29.822533+0000","last_active":"2023-10-16T14:16:30.960683+0000","last_peered":"2023-10-16T14:16:30.960683+0000","last_clean":"2023-10-16T14:16:30.960683+0000","last_became_active":"2023-10-16T14:16:29.822130+0000","last_became_peered":"2023-10-16T14:16:29.822130+0000","last_unstale":"2023-10-16T14:16:30.960683+0000","last_undegraded":"2023-10-16T14:16:30.960683+0000","last_fullsized":"2023-10-16T14:16:30.960683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325417+0000","last_change":"2023-10-16T14:16:29.044471+0000","last_active":"2023-10-16T14:16:32.325417+0000","last_peered":"2023-10-16T14:16:32.325417+0000","last_clean":"2023-10-16T14:16:32.325417+0000","last_became_active":"2023-10-16T14:16:29.042866+0000","last_became_peered":"2023-10-16T14:16:29.042866+0000","last_unstale":"2023-10-16T14:16:32.325417+0000","last_undegraded":"2023-10-16T14:16:32.325417+0000","last_fullsized":"2023-10-16T14:16:32.325417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:01.067871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799086+0000","last_change":"2023-10-16T14:16:29.826425+0000","last_active":"2023-10-16T14:16:30.799086+0000","last_peered":"2023-10-16T14:16:30.799086+0000","last_clean":"2023-10-16T14:16:30.799086+0000","last_became_active":"2023-10-16T14:16:29.820157+0000","last_became_peered":"2023-10-16T14:16:29.820157+0000","last_unstale":"2023-10-16T14:16:30.799086+0000","last_undegraded":"2023-10-16T14:16:30.799086+0000","last_fullsized":"2023-10-16T14:16:30.799086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797731+0000","last_change":"2023-10-16T14:16:29.049608+0000","last_active":"2023-10-16T14:16:30.797731+0000","last_peered":"2023-10-16T14:16:30.797731+0000","last_clean":"2023-10-16T14:16:30.797731+0000","last_became_active":"2023-10-16T14:16:29.049125+0000","last_became_peered":"2023-10-16T14:16:29.049125+0000","last_unstale":"2023-10-16T14:16:30.797731+0000","last_undegraded":"2023-10-16T14:16:30.797731+0000","last_fullsized":"2023-10-16T14:16:30.797731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798834+0000","last_change":"2023-10-16T14:16:29.809718+0000","last_active":"2023-10-16T14:16:30.798834+0000","last_peered":"2023-10-16T14:16:30.798834+0000","last_clean":"2023-10-16T14:16:30.798834+0000","last_became_active":"2023-10-16T14:16:29.809294+0000","last_became_peered":"2023-10-16T14:16:29.809294+0000","last_unstale":"2023-10-16T14:16:30.798834+0000","last_undegraded":"2023-10-16T14:16:30.798834+0000","last_fullsized":"2023-10-16T14:16:30.798834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325704+0000","last_change":"2023-10-16T14:16:29.034974+0000","last_active":"2023-10-16T14:16:32.325704+0000","last_peered":"2023-10-16T14:16:32.325704+0000","last_clean":"2023-10-16T14:16:32.325704+0000","last_became_active":"2023-10-16T14:16:29.034651+0000","last_became_peered":"2023-10-16T14:16:29.034651+0000","last_unstale":"2023-10-16T14:16:32.325704+0000","last_undegraded":"2023-10-16T14:16:32.325704+0000","last_fullsized":"2023-10-16T14:16:32.325704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:04.878575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799415+0000","last_change":"2023-10-16T14:16:29.829055+0000","last_active":"2023-10-16T14:16:30.799415+0000","last_peered":"2023-10-16T14:16:30.799415+0000","last_clean":"2023-10-16T14:16:30.799415+0000","last_became_active":"2023-10-16T14:16:29.820785+0000","last_became_peered":"2023-10-16T14:16:29.820785+0000","last_unstale":"2023-10-16T14:16:30.799415+0000","last_undegraded":"2023-10-16T14:16:30.799415+0000","last_fullsized":"2023-10-16T14:16:30.799415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794821+0000","last_change":"2023-10-16T14:16:29.048896+0000","last_active":"2023-10-16T14:16:30.794821+0000","last_peered":"2023-10-16T14:16:30.794821+0000","last_clean":"2023-10-16T14:16:30.794821+0000","last_became_active":"2023-10-16T14:16:29.048513+0000","last_became_peered":"2023-10-16T14:16:29.048513+0000","last_unstale":"2023-10-16T14:16:30.794821+0000","last_undegraded":"2023-10-16T14:16:30.794821+0000","last_fullsized":"2023-10-16T14:16:30.794821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827198+0000","last_change":"2023-10-16T14:16:29.814236+0000","last_active":"2023-10-16T14:16:31.827198+0000","last_peered":"2023-10-16T14:16:31.827198+0000","last_clean":"2023-10-16T14:16:31.827198+0000","last_became_active":"2023-10-16T14:16:29.813943+0000","last_became_peered":"2023-10-16T14:16:29.813943+0000","last_unstale":"2023-10-16T14:16:31.827198+0000","last_undegraded":"2023-10-16T14:16:31.827198+0000","last_fullsized":"2023-10-16T14:16:31.827198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325524+0000","last_change":"2023-10-16T14:16:29.052741+0000","last_active":"2023-10-16T14:16:32.325524+0000","last_peered":"2023-10-16T14:16:32.325524+0000","last_clean":"2023-10-16T14:16:32.325524+0000","last_became_active":"2023-10-16T14:16:29.052376+0000","last_became_peered":"2023-10-16T14:16:29.052376+0000","last_unstale":"2023-10-16T14:16:32.325524+0000","last_undegraded":"2023-10-16T14:16:32.325524+0000","last_fullsized":"2023-10-16T14:16:32.325524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:40.014521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797919+0000","last_change":"2023-10-16T14:16:29.828744+0000","last_active":"2023-10-16T14:16:30.797919+0000","last_peered":"2023-10-16T14:16:30.797919+0000","last_clean":"2023-10-16T14:16:30.797919+0000","last_became_active":"2023-10-16T14:16:29.823064+0000","last_became_peered":"2023-10-16T14:16:29.823064+0000","last_unstale":"2023-10-16T14:16:30.797919+0000","last_undegraded":"2023-10-16T14:16:30.797919+0000","last_fullsized":"2023-10-16T14:16:30.797919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-16T14:16:29.790600+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:16:29.790600+0000","last_peered":"2023-10-16T14:16:29.790600+0000","last_clean":"2023-10-16T14:16:29.790600+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:16:29.790600+0000","last_undegraded":"2023-10-16T14:16:29.790600+0000","last_fullsized":"2023-10-16T14:16:29.790600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825962+0000","last_change":"2023-10-16T14:16:29.051990+0000","last_active":"2023-10-16T14:16:31.825962+0000","last_peered":"2023-10-16T14:16:31.825962+0000","last_clean":"2023-10-16T14:16:31.825962+0000","last_became_active":"2023-10-16T14:16:29.051723+0000","last_became_peered":"2023-10-16T14:16:29.051723+0000","last_unstale":"2023-10-16T14:16:31.825962+0000","last_undegraded":"2023-10-16T14:16:31.825962+0000","last_fullsized":"2023-10-16T14:16:31.825962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667924+0000","last_change":"2023-10-16T14:16:29.823986+0000","last_active":"2023-10-16T14:16:31.667924+0000","last_peered":"2023-10-16T14:16:31.667924+0000","last_clean":"2023-10-16T14:16:31.667924+0000","last_became_active":"2023-10-16T14:16:29.823102+0000","last_became_peered":"2023-10-16T14:16:29.823102+0000","last_unstale":"2023-10-16T14:16:31.667924+0000","last_undegraded":"2023-10-16T14:16:31.667924+0000","last_fullsized":"2023-10-16T14:16:31.667924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795970+0000","last_change":"2023-10-16T14:16:29.058003+0000","last_active":"2023-10-16T14:16:30.795970+0000","last_peered":"2023-10-16T14:16:30.795970+0000","last_clean":"2023-10-16T14:16:30.795970+0000","last_became_active":"2023-10-16T14:16:29.057686+0000","last_became_peered":"2023-10-16T14:16:29.057686+0000","last_unstale":"2023-10-16T14:16:30.795970+0000","last_undegraded":"2023-10-16T14:16:30.795970+0000","last_fullsized":"2023-10-16T14:16:30.795970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325936+0000","last_change":"2023-10-16T14:16:29.812594+0000","last_active":"2023-10-16T14:16:32.325936+0000","last_peered":"2023-10-16T14:16:32.325936+0000","last_clean":"2023-10-16T14:16:32.325936+0000","last_became_active":"2023-10-16T14:16:29.812256+0000","last_became_peered":"2023-10-16T14:16:29.812256+0000","last_unstale":"2023-10-16T14:16:32.325936+0000","last_undegraded":"2023-10-16T14:16:32.325936+0000","last_fullsized":"2023-10-16T14:16:32.325936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:48.787869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325633+0000","last_change":"2023-10-16T14:16:29.042443+0000","last_active":"2023-10-16T14:16:32.325633+0000","last_peered":"2023-10-16T14:16:32.325633+0000","last_clean":"2023-10-16T14:16:32.325633+0000","last_became_active":"2023-10-16T14:16:29.042189+0000","last_became_peered":"2023-10-16T14:16:29.042189+0000","last_unstale":"2023-10-16T14:16:32.325633+0000","last_undegraded":"2023-10-16T14:16:32.325633+0000","last_fullsized":"2023-10-16T14:16:32.325633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794175+0000","last_change":"2023-10-16T14:16:29.816645+0000","last_active":"2023-10-16T14:16:30.794175+0000","last_peered":"2023-10-16T14:16:30.794175+0000","last_clean":"2023-10-16T14:16:30.794175+0000","last_became_active":"2023-10-16T14:16:29.814792+0000","last_became_peered":"2023-10-16T14:16:29.814792+0000","last_unstale":"2023-10-16T14:16:30.794175+0000","last_undegraded":"2023-10-16T14:16:30.794175+0000","last_fullsized":"2023-10-16T14:16:30.794175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826471+0000","last_change":"2023-10-16T14:16:29.055439+0000","last_active":"2023-10-16T14:16:31.826471+0000","last_peered":"2023-10-16T14:16:31.826471+0000","last_clean":"2023-10-16T14:16:31.826471+0000","last_became_active":"2023-10-16T14:16:29.054422+0000","last_became_peered":"2023-10-16T14:16:29.054422+0000","last_unstale":"2023-10-16T14:16:31.826471+0000","last_undegraded":"2023-10-16T14:16:31.826471+0000","last_fullsized":"2023-10-16T14:16:31.826471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797290+0000","last_change":"2023-10-16T14:16:29.826562+0000","last_active":"2023-10-16T14:16:30.797290+0000","last_peered":"2023-10-16T14:16:30.797290+0000","last_clean":"2023-10-16T14:16:30.797290+0000","last_became_active":"2023-10-16T14:16:29.825184+0000","last_became_peered":"2023-10-16T14:16:29.825184+0000","last_unstale":"2023-10-16T14:16:30.797290+0000","last_undegraded":"2023-10-16T14:16:30.797290+0000","last_fullsized":"2023-10-16T14:16:30.797290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:49.651724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794587+0000","last_change":"2023-10-16T14:16:29.048497+0000","last_active":"2023-10-16T14:16:30.794587+0000","last_peered":"2023-10-16T14:16:30.794587+0000","last_clean":"2023-10-16T14:16:30.794587+0000","last_became_active":"2023-10-16T14:16:29.048194+0000","last_became_peered":"2023-10-16T14:16:29.048194+0000","last_unstale":"2023-10-16T14:16:30.794587+0000","last_undegraded":"2023-10-16T14:16:30.794587+0000","last_fullsized":"2023-10-16T14:16:30.794587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826954+0000","last_change":"2023-10-16T14:16:29.819205+0000","last_active":"2023-10-16T14:16:31.826954+0000","last_peered":"2023-10-16T14:16:31.826954+0000","last_clean":"2023-10-16T14:16:31.826954+0000","last_became_active":"2023-10-16T14:16:29.818921+0000","last_became_peered":"2023-10-16T14:16:29.818921+0000","last_unstale":"2023-10-16T14:16:31.826954+0000","last_undegraded":"2023-10-16T14:16:31.826954+0000","last_fullsized":"2023-10-16T14:16:31.826954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798586+0000","last_change":"2023-10-16T14:16:29.053513+0000","last_active":"2023-10-16T14:16:30.798586+0000","last_peered":"2023-10-16T14:16:30.798586+0000","last_clean":"2023-10-16T14:16:30.798586+0000","last_became_active":"2023-10-16T14:16:29.053178+0000","last_became_peered":"2023-10-16T14:16:29.053178+0000","last_unstale":"2023-10-16T14:16:30.798586+0000","last_undegraded":"2023-10-16T14:16:30.798586+0000","last_fullsized":"2023-10-16T14:16:30.798586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794789+0000","last_change":"2023-10-16T14:16:29.817442+0000","last_active":"2023-10-16T14:16:30.794789+0000","last_peered":"2023-10-16T14:16:30.794789+0000","last_clean":"2023-10-16T14:16:30.794789+0000","last_became_active":"2023-10-16T14:16:29.814899+0000","last_became_peered":"2023-10-16T14:16:29.814899+0000","last_unstale":"2023-10-16T14:16:30.794789+0000","last_undegraded":"2023-10-16T14:16:30.794789+0000","last_fullsized":"2023-10-16T14:16:30.794789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.915468+0000","last_change":"2023-10-16T14:16:29.059226+0000","last_active":"2023-10-16T14:16:32.915468+0000","last_peered":"2023-10-16T14:16:32.915468+0000","last_clean":"2023-10-16T14:16:32.915468+0000","last_became_active":"2023-10-16T14:16:29.058896+0000","last_became_peered":"2023-10-16T14:16:29.058896+0000","last_unstale":"2023-10-16T14:16:32.915468+0000","last_undegraded":"2023-10-16T14:16:32.915468+0000","last_fullsized":"2023-10-16T14:16:32.915468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668348+0000","last_change":"2023-10-16T14:16:29.807423+0000","last_active":"2023-10-16T14:16:31.668348+0000","last_peered":"2023-10-16T14:16:31.668348+0000","last_clean":"2023-10-16T14:16:31.668348+0000","last_became_active":"2023-10-16T14:16:29.806278+0000","last_became_peered":"2023-10-16T14:16:29.806278+0000","last_unstale":"2023-10-16T14:16:31.668348+0000","last_undegraded":"2023-10-16T14:16:31.668348+0000","last_fullsized":"2023-10-16T14:16:31.668348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961275+0000","last_change":"2023-10-16T14:16:29.057876+0000","last_active":"2023-10-16T14:16:30.961275+0000","last_peered":"2023-10-16T14:16:30.961275+0000","last_clean":"2023-10-16T14:16:30.961275+0000","last_became_active":"2023-10-16T14:16:29.053426+0000","last_became_peered":"2023-10-16T14:16:29.053426+0000","last_unstale":"2023-10-16T14:16:30.961275+0000","last_undegraded":"2023-10-16T14:16:30.961275+0000","last_fullsized":"2023-10-16T14:16:30.961275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:01.418897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326675+0000","last_change":"2023-10-16T14:16:29.816014+0000","last_active":"2023-10-16T14:16:32.326675+0000","last_peered":"2023-10-16T14:16:32.326675+0000","last_clean":"2023-10-16T14:16:32.326675+0000","last_became_active":"2023-10-16T14:16:29.815725+0000","last_became_peered":"2023-10-16T14:16:29.815725+0000","last_unstale":"2023-10-16T14:16:32.326675+0000","last_undegraded":"2023-10-16T14:16:32.326675+0000","last_fullsized":"2023-10-16T14:16:32.326675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795715+0000","last_change":"2023-10-16T14:16:29.050583+0000","last_active":"2023-10-16T14:16:30.795715+0000","last_peered":"2023-10-16T14:16:30.795715+0000","last_clean":"2023-10-16T14:16:30.795715+0000","last_became_active":"2023-10-16T14:16:29.050281+0000","last_became_peered":"2023-10-16T14:16:29.050281+0000","last_unstale":"2023-10-16T14:16:30.795715+0000","last_undegraded":"2023-10-16T14:16:30.795715+0000","last_fullsized":"2023-10-16T14:16:30.795715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:18.882395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796790+0000","last_change":"2023-10-16T14:16:29.808419+0000","last_active":"2023-10-16T14:16:30.796790+0000","last_peered":"2023-10-16T14:16:30.796790+0000","last_clean":"2023-10-16T14:16:30.796790+0000","last_became_active":"2023-10-16T14:16:29.808038+0000","last_became_peered":"2023-10-16T14:16:29.808038+0000","last_unstale":"2023-10-16T14:16:30.796790+0000","last_undegraded":"2023-10-16T14:16:30.796790+0000","last_fullsized":"2023-10-16T14:16:30.796790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:05.891605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794294+0000","last_change":"2023-10-16T14:16:29.049120+0000","last_active":"2023-10-16T14:16:30.794294+0000","last_peered":"2023-10-16T14:16:30.794294+0000","last_clean":"2023-10-16T14:16:30.794294+0000","last_became_active":"2023-10-16T14:16:29.048782+0000","last_became_peered":"2023-10-16T14:16:29.048782+0000","last_unstale":"2023-10-16T14:16:30.794294+0000","last_undegraded":"2023-10-16T14:16:30.794294+0000","last_fullsized":"2023-10-16T14:16:30.794294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667807+0000","last_change":"2023-10-16T14:16:29.820694+0000","last_active":"2023-10-16T14:16:31.667807+0000","last_peered":"2023-10-16T14:16:31.667807+0000","last_clean":"2023-10-16T14:16:31.667807+0000","last_became_active":"2023-10-16T14:16:29.820341+0000","last_became_peered":"2023-10-16T14:16:29.820341+0000","last_unstale":"2023-10-16T14:16:31.667807+0000","last_undegraded":"2023-10-16T14:16:31.667807+0000","last_fullsized":"2023-10-16T14:16:31.667807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794452+0000","last_change":"2023-10-16T14:16:29.043057+0000","last_active":"2023-10-16T14:16:30.794452+0000","last_peered":"2023-10-16T14:16:30.794452+0000","last_clean":"2023-10-16T14:16:30.794452+0000","last_became_active":"2023-10-16T14:16:29.042783+0000","last_became_peered":"2023-10-16T14:16:29.042783+0000","last_unstale":"2023-10-16T14:16:30.794452+0000","last_undegraded":"2023-10-16T14:16:30.794452+0000","last_fullsized":"2023-10-16T14:16:30.794452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668050+0000","last_change":"2023-10-16T14:16:29.816920+0000","last_active":"2023-10-16T14:16:31.668050+0000","last_peered":"2023-10-16T14:16:31.668050+0000","last_clean":"2023-10-16T14:16:31.668050+0000","last_became_active":"2023-10-16T14:16:29.816639+0000","last_became_peered":"2023-10-16T14:16:29.816639+0000","last_unstale":"2023-10-16T14:16:31.668050+0000","last_undegraded":"2023-10-16T14:16:31.668050+0000","last_fullsized":"2023-10-16T14:16:31.668050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798966+0000","last_change":"2023-10-16T14:16:29.056773+0000","last_active":"2023-10-16T14:16:30.798966+0000","last_peered":"2023-10-16T14:16:30.798966+0000","last_clean":"2023-10-16T14:16:30.798966+0000","last_became_active":"2023-10-16T14:16:29.056477+0000","last_became_peered":"2023-10-16T14:16:29.056477+0000","last_unstale":"2023-10-16T14:16:30.798966+0000","last_undegraded":"2023-10-16T14:16:30.798966+0000","last_fullsized":"2023-10-16T14:16:30.798966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:17.696119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960869+0000","last_change":"2023-10-16T14:16:29.821383+0000","last_active":"2023-10-16T14:16:30.960869+0000","last_peered":"2023-10-16T14:16:30.960869+0000","last_clean":"2023-10-16T14:16:30.960869+0000","last_became_active":"2023-10-16T14:16:29.821018+0000","last_became_peered":"2023-10-16T14:16:29.821018+0000","last_unstale":"2023-10-16T14:16:30.960869+0000","last_undegraded":"2023-10-16T14:16:30.960869+0000","last_fullsized":"2023-10-16T14:16:30.960869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826784+0000","last_change":"2023-10-16T14:16:29.829570+0000","last_active":"2023-10-16T14:16:31.826784+0000","last_peered":"2023-10-16T14:16:31.826784+0000","last_clean":"2023-10-16T14:16:31.826784+0000","last_became_active":"2023-10-16T14:16:29.828707+0000","last_became_peered":"2023-10-16T14:16:29.828707+0000","last_unstale":"2023-10-16T14:16:31.826784+0000","last_undegraded":"2023-10-16T14:16:31.826784+0000","last_fullsized":"2023-10-16T14:16:31.826784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798241+0000","last_change":"2023-10-16T14:16:29.063205+0000","last_active":"2023-10-16T14:16:30.798241+0000","last_peered":"2023-10-16T14:16:30.798241+0000","last_clean":"2023-10-16T14:16:30.798241+0000","last_became_active":"2023-10-16T14:16:29.062657+0000","last_became_peered":"2023-10-16T14:16:29.062657+0000","last_unstale":"2023-10-16T14:16:30.798241+0000","last_undegraded":"2023-10-16T14:16:30.798241+0000","last_fullsized":"2023-10-16T14:16:30.798241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:43.103258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826847+0000","last_change":"2023-10-16T14:16:29.830412+0000","last_active":"2023-10-16T14:16:31.826847+0000","last_peered":"2023-10-16T14:16:31.826847+0000","last_clean":"2023-10-16T14:16:31.826847+0000","last_became_active":"2023-10-16T14:16:29.825817+0000","last_became_peered":"2023-10-16T14:16:29.825817+0000","last_unstale":"2023-10-16T14:16:31.826847+0000","last_undegraded":"2023-10-16T14:16:31.826847+0000","last_fullsized":"2023-10-16T14:16:31.826847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798099+0000","last_change":"2023-10-16T14:16:29.061570+0000","last_active":"2023-10-16T14:16:30.798099+0000","last_peered":"2023-10-16T14:16:30.798099+0000","last_clean":"2023-10-16T14:16:30.798099+0000","last_became_active":"2023-10-16T14:16:29.061024+0000","last_became_peered":"2023-10-16T14:16:29.061024+0000","last_unstale":"2023-10-16T14:16:30.798099+0000","last_undegraded":"2023-10-16T14:16:30.798099+0000","last_fullsized":"2023-10-16T14:16:30.798099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326481+0000","last_change":"2023-10-16T14:16:29.827404+0000","last_active":"2023-10-16T14:16:32.326481+0000","last_peered":"2023-10-16T14:16:32.326481+0000","last_clean":"2023-10-16T14:16:32.326481+0000","last_became_active":"2023-10-16T14:16:29.826687+0000","last_became_peered":"2023-10-16T14:16:29.826687+0000","last_unstale":"2023-10-16T14:16:32.326481+0000","last_undegraded":"2023-10-16T14:16:32.326481+0000","last_fullsized":"2023-10-16T14:16:32.326481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668592+0000","last_change":"2023-10-16T14:16:29.050871+0000","last_active":"2023-10-16T14:16:31.668592+0000","last_peered":"2023-10-16T14:16:31.668592+0000","last_clean":"2023-10-16T14:16:31.668592+0000","last_became_active":"2023-10-16T14:16:29.050025+0000","last_became_peered":"2023-10-16T14:16:29.050025+0000","last_unstale":"2023-10-16T14:16:31.668592+0000","last_undegraded":"2023-10-16T14:16:31.668592+0000","last_fullsized":"2023-10-16T14:16:31.668592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668230+0000","last_change":"2023-10-16T14:16:29.825092+0000","last_active":"2023-10-16T14:16:31.668230+0000","last_peered":"2023-10-16T14:16:31.668230+0000","last_clean":"2023-10-16T14:16:31.668230+0000","last_became_active":"2023-10-16T14:16:29.824815+0000","last_became_peered":"2023-10-16T14:16:29.824815+0000","last_unstale":"2023-10-16T14:16:31.668230+0000","last_undegraded":"2023-10-16T14:16:31.668230+0000","last_fullsized":"2023-10-16T14:16:31.668230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326685+0000","last_change":"2023-10-16T14:16:29.053310+0000","last_active":"2023-10-16T14:16:32.326685+0000","last_peered":"2023-10-16T14:16:32.326685+0000","last_clean":"2023-10-16T14:16:32.326685+0000","last_became_active":"2023-10-16T14:16:29.052871+0000","last_became_peered":"2023-10-16T14:16:29.052871+0000","last_unstale":"2023-10-16T14:16:32.326685+0000","last_undegraded":"2023-10-16T14:16:32.326685+0000","last_fullsized":"2023-10-16T14:16:32.326685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4830000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9930000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9830000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.165}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2650000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.653}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.391}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9369999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5779999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7610000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:16:39.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:16:40.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.178+0000 7f464c5f0640 1 Processor -- start 2023-10-16T14:16:40.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.178+0000 7f464c5f0640 1 -- start start 2023-10-16T14:16:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464c5f0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4644159f00 0x7f46440b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464c5f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 0x7f46440b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464c5f0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464406f7f0 con 0x7f4644159b20 2023-10-16T14:16:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464c5f0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464406c5e0 con 0x7f4644159f00 2023-10-16T14:16:40.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464c5f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464406c8c0 con 0x7f464415d880 2023-10-16T14:16:40.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464a38c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4644159b20 0x7f464406bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46384/0 (socket says 172.21.15.29:46384) 2023-10-16T14:16:40.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464a38c640 1 -- 172.21.15.29:0/3782447084 learned_addr learned my addr 172.21.15.29:0/3782447084 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:40.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f4649b8b640 1 --2- 172.21.15.29:0/3782447084 >> v2:172.21.15.170:3300/0 conn(0x7f4644159f00 0x7f46440b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.180+0000 7f464ab8d640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 0x7f46440b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627212039 0 0) 0x7f464406f7f0 con 0x7f4644159b20 2023-10-16T14:16:40.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4630003520 con 0x7f4644159b20 2023-10-16T14:16:40.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3104805592 0 0) 0x7f4630003520 con 0x7f4644159b20 2023-10-16T14:16:40.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 msgr2=0x7f46440b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 0x7f46440b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 >> v2:172.21.15.170:3300/0 conn(0x7f4644159f00 msgr2=0x7f46440b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 --2- 172.21.15.29:0/3782447084 >> v2:172.21.15.170:3300/0 conn(0x7f4644159f00 0x7f46440b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464406cba0 con 0x7f4644159b20 2023-10-16T14:16:40.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4640001670 con 0x7f4644159b20 2023-10-16T14:16:40.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 >> v1:172.21.15.29:6789/0 conn(0x7f4644159b20 legacy=0x7f464406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 0x7f4630003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630003f40 0x7f46300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4630004d00 0x7f4630005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464406c5e0 con 0x7f4630004d00 2023-10-16T14:16:40.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464406c8c0 con 0x7f4630003f40 2023-10-16T14:16:40.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.181+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463000b980 con 0x7f464415d880 2023-10-16T14:16:40.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464ab8d640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4630004d00 0x7f4630005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f4649b8b640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630003f40 0x7f46300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464a38c640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 0x7f4630003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464ab8d640 1 -- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 msgr2=0x7f4630003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464ab8d640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f464415d880 0x7f4630003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464ab8d640 1 -- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630003f40 msgr2=0x7f46300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464ab8d640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630003f40 0x7f46300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.182+0000 7f464ab8d640 1 -- 172.21.15.29:0/3782447084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630003520 con 0x7f4630004d00 2023-10-16T14:16:40.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.183+0000 7f464ab8d640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4630004d00 0x7f4630005180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f463c00e170 tx=0x7f46300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849313901588ab28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:40.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.183+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c02cbf0 con 0x7f4630004d00 2023-10-16T14:16:40.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.183+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c00f040 con 0x7f4630004d00 2023-10-16T14:16:40.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.183+0000 7f464938a640 1 -- 172.21.15.29:0/3782447084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c034440 con 0x7f4630004d00 2023-10-16T14:16:40.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464c5f0640 1 -- 172.21.15.29:0/3782447084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4630004d00 msgr2=0x7f4630005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464c5f0640 1 --2- 172.21.15.29:0/3782447084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4630004d00 0x7f4630005180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f463c00e170 tx=0x7f46300057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464ab8d640 1 -- 172.21.15.29:0/3782447084 reap_dead start 2023-10-16T14:16:40.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464c5f0640 1 -- 172.21.15.29:0/3782447084 shutdown_connections 2023-10-16T14:16:40.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464c5f0640 1 -- 172.21.15.29:0/3782447084 >> 172.21.15.29:0/3782447084 conn(0x7f46440cb410 msgr2=0x7f46440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:40.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464c5f0640 1 -- 172.21.15.29:0/3782447084 shutdown_connections 2023-10-16T14:16:40.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.184+0000 7f464c5f0640 1 -- 172.21.15.29:0/3782447084 wait complete. 2023-10-16T14:16:40.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.186+0000 7f464c5f0640 1 Processor -- start 2023-10-16T14:16:40.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.186+0000 7f464c5f0640 1 -- start start 2023-10-16T14:16:40.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.187+0000 7f464c5f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630004d00 0x7f4644149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.187+0000 7f464c5f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4644149e90 0x7f4644150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.187+0000 7f464c5f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 0x7f4644150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.187+0000 7f464c5f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464414db20 con 0x7f464414a310 2023-10-16T14:16:40.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.187+0000 7f464c5f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644149ca0 con 0x7f4630004d00 2023-10-16T14:16:40.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.187+0000 7f464c5f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464414a870 con 0x7f4644149e90 2023-10-16T14:16:40.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 0x7f4644150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 0x7f4644150aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51036/0 (socket says 172.21.15.29:51036) 2023-10-16T14:16:40.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 -- 172.21.15.29:0/1125753980 learned_addr learned my addr 172.21.15.29:0/1125753980 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:40.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464a38c640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630004d00 0x7f4644149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f4649b8b640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4644149e90 0x7f4644150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 -- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4644149e90 msgr2=0x7f4644150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4644149e90 0x7f4644150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 -- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630004d00 msgr2=0x7f4644149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630004d00 0x7f4644149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.188+0000 7f464ab8d640 1 -- 172.21.15.29:0/1125753980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464414ab50 con 0x7f464414a310 2023-10-16T14:16:40.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.189+0000 7f464ab8d640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 0x7f4644150aa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f463c00e170 tx=0x7f463c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:40.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.189+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c034cd0 con 0x7f464414a310 2023-10-16T14:16:40.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.189+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c034e30 con 0x7f464414a310 2023-10-16T14:16:40.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.189+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c003bd0 con 0x7f464414a310 2023-10-16T14:16:40.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.189+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464414ae30 con 0x7f464414a310 2023-10-16T14:16:40.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.189+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464414b960 con 0x7f464414a310 2023-10-16T14:16:40.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.190+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46440c3670 con 0x7f464414a310 2023-10-16T14:16:40.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.191+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f463c003d30 con 0x7f464414a310 2023-10-16T14:16:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.192+0000 7f463affd640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f462806ae70 0x7f462806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.192+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f463c0ab670 con 0x7f464414a310 2023-10-16T14:16:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.193+0000 7f464a38c640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f462806ae70 0x7f462806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.194+0000 7f464a38c640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f462806ae70 0x7f462806d330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4640001eb0 tx=0x7f4640009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:40.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.197+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f46440c3670 con 0x7f464414a310 2023-10-16T14:16:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.387+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f464406ce80 con 0x7f462806ae70 2023-10-16T14:16:40.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.407+0000 7f463affd640 1 -- 172.21.15.29:0/1125753980 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264435 (secure 0 0 0) 0x7f4640010950 con 0x7f462806ae70 2023-10-16T14:16:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:40.416 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:16:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f462806ae70 msgr2=0x7f462806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f462806ae70 0x7f462806d330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4640001eb0 tx=0x7f4640009040 comp rx=0 tx=0).stop 2023-10-16T14:16:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 msgr2=0x7f4644150aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 0x7f4644150aa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f463c00e170 tx=0x7f463c009750 comp rx=0 tx=0).stop 2023-10-16T14:16:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 shutdown_connections 2023-10-16T14:16:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f462806ae70 0x7f462806d330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f464414a310 0x7f4644150aa0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4644149e90 0x7f4644150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 --2- 172.21.15.29:0/1125753980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4630004d00 0x7f4644149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 >> 172.21.15.29:0/1125753980 conn(0x7f46440cb410 msgr2=0x7f46440cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 shutdown_connections 2023-10-16T14:16:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.413+0000 7f464c5f0640 1 -- 172.21.15.29:0/1125753980 wait complete. 2023-10-16T14:16:40.439 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T14:16:39.815072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252284,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722436,"statfs":{"total":773094113280,"available":772835774464,"internally_reserved":0,"allocated":2891776,"data_stored":1752740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":255315513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005076"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799225+0000","last_change":"2023-10-16T14:16:29.830118+0000","last_active":"2023-10-16T14:16:30.799225+0000","last_peered":"2023-10-16T14:16:30.799225+0000","last_clean":"2023-10-16T14:16:30.799225+0000","last_became_active":"2023-10-16T14:16:29.829372+0000","last_became_peered":"2023-10-16T14:16:29.829372+0000","last_unstale":"2023-10-16T14:16:30.799225+0000","last_undegraded":"2023-10-16T14:16:30.799225+0000","last_fullsized":"2023-10-16T14:16:30.799225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794763+0000","last_change":"2023-10-16T14:16:29.818444+0000","last_active":"2023-10-16T14:16:30.794763+0000","last_peered":"2023-10-16T14:16:30.794763+0000","last_clean":"2023-10-16T14:16:30.794763+0000","last_became_active":"2023-10-16T14:16:29.816215+0000","last_became_peered":"2023-10-16T14:16:29.816215+0000","last_unstale":"2023-10-16T14:16:30.794763+0000","last_undegraded":"2023-10-16T14:16:30.794763+0000","last_fullsized":"2023-10-16T14:16:30.794763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326440+0000","last_change":"2023-10-16T14:16:29.817304+0000","last_active":"2023-10-16T14:16:32.326440+0000","last_peered":"2023-10-16T14:16:32.326440+0000","last_clean":"2023-10-16T14:16:32.326440+0000","last_became_active":"2023-10-16T14:16:29.817037+0000","last_became_peered":"2023-10-16T14:16:29.817037+0000","last_unstale":"2023-10-16T14:16:32.326440+0000","last_undegraded":"2023-10-16T14:16:32.326440+0000","last_fullsized":"2023-10-16T14:16:32.326440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827089+0000","last_change":"2023-10-16T14:16:29.816180+0000","last_active":"2023-10-16T14:16:31.827089+0000","last_peered":"2023-10-16T14:16:31.827089+0000","last_clean":"2023-10-16T14:16:31.827089+0000","last_became_active":"2023-10-16T14:16:29.815841+0000","last_became_peered":"2023-10-16T14:16:29.815841+0000","last_unstale":"2023-10-16T14:16:31.827089+0000","last_undegraded":"2023-10-16T14:16:31.827089+0000","last_fullsized":"2023-10-16T14:16:31.827089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:03.796421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798716+0000","last_change":"2023-10-16T14:16:29.813648+0000","last_active":"2023-10-16T14:16:30.798716+0000","last_peered":"2023-10-16T14:16:30.798716+0000","last_clean":"2023-10-16T14:16:30.798716+0000","last_became_active":"2023-10-16T14:16:29.813365+0000","last_became_peered":"2023-10-16T14:16:29.813365+0000","last_unstale":"2023-10-16T14:16:30.798716+0000","last_undegraded":"2023-10-16T14:16:30.798716+0000","last_fullsized":"2023-10-16T14:16:30.798716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825920+0000","last_change":"2023-10-16T14:16:29.818850+0000","last_active":"2023-10-16T14:16:31.825920+0000","last_peered":"2023-10-16T14:16:31.825920+0000","last_clean":"2023-10-16T14:16:31.825920+0000","last_became_active":"2023-10-16T14:16:29.818486+0000","last_became_peered":"2023-10-16T14:16:29.818486+0000","last_unstale":"2023-10-16T14:16:31.825920+0000","last_undegraded":"2023-10-16T14:16:31.825920+0000","last_fullsized":"2023-10-16T14:16:31.825920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796628+0000","last_change":"2023-10-16T14:16:29.814386+0000","last_active":"2023-10-16T14:16:30.796628+0000","last_peered":"2023-10-16T14:16:30.796628+0000","last_clean":"2023-10-16T14:16:30.796628+0000","last_became_active":"2023-10-16T14:16:29.812379+0000","last_became_peered":"2023-10-16T14:16:29.812379+0000","last_unstale":"2023-10-16T14:16:30.796628+0000","last_undegraded":"2023-10-16T14:16:30.796628+0000","last_fullsized":"2023-10-16T14:16:30.796628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667250+0000","last_change":"2023-10-16T14:16:29.823474+0000","last_active":"2023-10-16T14:16:31.667250+0000","last_peered":"2023-10-16T14:16:31.667250+0000","last_clean":"2023-10-16T14:16:31.667250+0000","last_became_active":"2023-10-16T14:16:29.823172+0000","last_became_peered":"2023-10-16T14:16:29.823172+0000","last_unstale":"2023-10-16T14:16:31.667250+0000","last_undegraded":"2023-10-16T14:16:31.667250+0000","last_fullsized":"2023-10-16T14:16:31.667250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799207+0000","last_change":"2023-10-16T14:16:29.830964+0000","last_active":"2023-10-16T14:16:30.799207+0000","last_peered":"2023-10-16T14:16:30.799207+0000","last_clean":"2023-10-16T14:16:30.799207+0000","last_became_active":"2023-10-16T14:16:29.829757+0000","last_became_peered":"2023-10-16T14:16:29.829757+0000","last_unstale":"2023-10-16T14:16:30.799207+0000","last_undegraded":"2023-10-16T14:16:30.799207+0000","last_fullsized":"2023-10-16T14:16:30.799207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961588+0000","last_change":"2023-10-16T14:16:29.822512+0000","last_active":"2023-10-16T14:16:30.961588+0000","last_peered":"2023-10-16T14:16:30.961588+0000","last_clean":"2023-10-16T14:16:30.961588+0000","last_became_active":"2023-10-16T14:16:29.822050+0000","last_became_peered":"2023-10-16T14:16:29.822050+0000","last_unstale":"2023-10-16T14:16:30.961588+0000","last_undegraded":"2023-10-16T14:16:30.961588+0000","last_fullsized":"2023-10-16T14:16:30.961588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961278+0000","last_change":"2023-10-16T14:16:29.819724+0000","last_active":"2023-10-16T14:16:30.961278+0000","last_peered":"2023-10-16T14:16:30.961278+0000","last_clean":"2023-10-16T14:16:30.961278+0000","last_became_active":"2023-10-16T14:16:29.819467+0000","last_became_peered":"2023-10-16T14:16:29.819467+0000","last_unstale":"2023-10-16T14:16:30.961278+0000","last_undegraded":"2023-10-16T14:16:30.961278+0000","last_fullsized":"2023-10-16T14:16:30.961278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797593+0000","last_change":"2023-10-16T14:16:29.819218+0000","last_active":"2023-10-16T14:16:30.797593+0000","last_peered":"2023-10-16T14:16:30.797593+0000","last_clean":"2023-10-16T14:16:30.797593+0000","last_became_active":"2023-10-16T14:16:29.818622+0000","last_became_peered":"2023-10-16T14:16:29.818622+0000","last_unstale":"2023-10-16T14:16:30.797593+0000","last_undegraded":"2023-10-16T14:16:30.797593+0000","last_fullsized":"2023-10-16T14:16:30.797593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668213+0000","last_change":"2023-10-16T14:16:29.817208+0000","last_active":"2023-10-16T14:16:31.668213+0000","last_peered":"2023-10-16T14:16:31.668213+0000","last_clean":"2023-10-16T14:16:31.668213+0000","last_became_active":"2023-10-16T14:16:29.816175+0000","last_became_peered":"2023-10-16T14:16:29.816175+0000","last_unstale":"2023-10-16T14:16:31.668213+0000","last_undegraded":"2023-10-16T14:16:31.668213+0000","last_fullsized":"2023-10-16T14:16:31.668213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325899+0000","last_change":"2023-10-16T14:16:29.812726+0000","last_active":"2023-10-16T14:16:32.325899+0000","last_peered":"2023-10-16T14:16:32.325899+0000","last_clean":"2023-10-16T14:16:32.325899+0000","last_became_active":"2023-10-16T14:16:29.812358+0000","last_became_peered":"2023-10-16T14:16:29.812358+0000","last_unstale":"2023-10-16T14:16:32.325899+0000","last_undegraded":"2023-10-16T14:16:32.325899+0000","last_fullsized":"2023-10-16T14:16:32.325899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:59.947139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794118+0000","last_change":"2023-10-16T14:16:29.820374+0000","last_active":"2023-10-16T14:16:30.794118+0000","last_peered":"2023-10-16T14:16:30.794118+0000","last_clean":"2023-10-16T14:16:30.794118+0000","last_became_active":"2023-10-16T14:16:29.820053+0000","last_became_peered":"2023-10-16T14:16:29.820053+0000","last_unstale":"2023-10-16T14:16:30.794118+0000","last_undegraded":"2023-10-16T14:16:30.794118+0000","last_fullsized":"2023-10-16T14:16:30.794118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825735+0000","last_change":"2023-10-16T14:16:29.819714+0000","last_active":"2023-10-16T14:16:31.825735+0000","last_peered":"2023-10-16T14:16:31.825735+0000","last_clean":"2023-10-16T14:16:31.825735+0000","last_became_active":"2023-10-16T14:16:29.819444+0000","last_became_peered":"2023-10-16T14:16:29.819444+0000","last_unstale":"2023-10-16T14:16:31.825735+0000","last_undegraded":"2023-10-16T14:16:31.825735+0000","last_fullsized":"2023-10-16T14:16:31.825735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826922+0000","last_change":"2023-10-16T14:16:29.817726+0000","last_active":"2023-10-16T14:16:31.826922+0000","last_peered":"2023-10-16T14:16:31.826922+0000","last_clean":"2023-10-16T14:16:31.826922+0000","last_became_active":"2023-10-16T14:16:29.813107+0000","last_became_peered":"2023-10-16T14:16:29.813107+0000","last_unstale":"2023-10-16T14:16:31.826922+0000","last_undegraded":"2023-10-16T14:16:31.826922+0000","last_fullsized":"2023-10-16T14:16:31.826922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325856+0000","last_change":"2023-10-16T14:16:29.825557+0000","last_active":"2023-10-16T14:16:32.325856+0000","last_peered":"2023-10-16T14:16:32.325856+0000","last_clean":"2023-10-16T14:16:32.325856+0000","last_became_active":"2023-10-16T14:16:29.825131+0000","last_became_peered":"2023-10-16T14:16:29.825131+0000","last_unstale":"2023-10-16T14:16:32.325856+0000","last_undegraded":"2023-10-16T14:16:32.325856+0000","last_fullsized":"2023-10-16T14:16:32.325856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798289+0000","last_change":"2023-10-16T14:16:29.813672+0000","last_active":"2023-10-16T14:16:30.798289+0000","last_peered":"2023-10-16T14:16:30.798289+0000","last_clean":"2023-10-16T14:16:30.798289+0000","last_became_active":"2023-10-16T14:16:29.813271+0000","last_became_peered":"2023-10-16T14:16:29.813271+0000","last_unstale":"2023-10-16T14:16:30.798289+0000","last_undegraded":"2023-10-16T14:16:30.798289+0000","last_fullsized":"2023-10-16T14:16:30.798289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961292+0000","last_change":"2023-10-16T14:16:29.821916+0000","last_active":"2023-10-16T14:16:30.961292+0000","last_peered":"2023-10-16T14:16:30.961292+0000","last_clean":"2023-10-16T14:16:30.961292+0000","last_became_active":"2023-10-16T14:16:29.821561+0000","last_became_peered":"2023-10-16T14:16:29.821561+0000","last_unstale":"2023-10-16T14:16:30.961292+0000","last_undegraded":"2023-10-16T14:16:30.961292+0000","last_fullsized":"2023-10-16T14:16:30.961292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326088+0000","last_change":"2023-10-16T14:16:29.809106+0000","last_active":"2023-10-16T14:16:32.326088+0000","last_peered":"2023-10-16T14:16:32.326088+0000","last_clean":"2023-10-16T14:16:32.326088+0000","last_became_active":"2023-10-16T14:16:29.808677+0000","last_became_peered":"2023-10-16T14:16:29.808677+0000","last_unstale":"2023-10-16T14:16:32.326088+0000","last_undegraded":"2023-10-16T14:16:32.326088+0000","last_fullsized":"2023-10-16T14:16:32.326088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:02.431928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.814365+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.814022+0000","last_became_peered":"2023-10-16T14:16:29.814022+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667249+0000","last_change":"2023-10-16T14:16:29.826061+0000","last_active":"2023-10-16T14:16:31.667249+0000","last_peered":"2023-10-16T14:16:31.667249+0000","last_clean":"2023-10-16T14:16:31.667249+0000","last_became_active":"2023-10-16T14:16:29.825662+0000","last_became_peered":"2023-10-16T14:16:29.825662+0000","last_unstale":"2023-10-16T14:16:31.667249+0000","last_undegraded":"2023-10-16T14:16:31.667249+0000","last_fullsized":"2023-10-16T14:16:31.667249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826173+0000","last_change":"2023-10-16T14:16:29.822279+0000","last_active":"2023-10-16T14:16:31.826173+0000","last_peered":"2023-10-16T14:16:31.826173+0000","last_clean":"2023-10-16T14:16:31.826173+0000","last_became_active":"2023-10-16T14:16:29.822022+0000","last_became_peered":"2023-10-16T14:16:29.822022+0000","last_unstale":"2023-10-16T14:16:31.826173+0000","last_undegraded":"2023-10-16T14:16:31.826173+0000","last_fullsized":"2023-10-16T14:16:31.826173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668113+0000","last_change":"2023-10-16T14:16:29.823044+0000","last_active":"2023-10-16T14:16:31.668113+0000","last_peered":"2023-10-16T14:16:31.668113+0000","last_clean":"2023-10-16T14:16:31.668113+0000","last_became_active":"2023-10-16T14:16:29.822705+0000","last_became_peered":"2023-10-16T14:16:29.822705+0000","last_unstale":"2023-10-16T14:16:31.668113+0000","last_undegraded":"2023-10-16T14:16:31.668113+0000","last_fullsized":"2023-10-16T14:16:31.668113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961559+0000","last_change":"2023-10-16T14:16:29.822259+0000","last_active":"2023-10-16T14:16:30.961559+0000","last_peered":"2023-10-16T14:16:30.961559+0000","last_clean":"2023-10-16T14:16:30.961559+0000","last_became_active":"2023-10-16T14:16:29.821719+0000","last_became_peered":"2023-10-16T14:16:29.821719+0000","last_unstale":"2023-10-16T14:16:30.961559+0000","last_undegraded":"2023-10-16T14:16:30.961559+0000","last_fullsized":"2023-10-16T14:16:30.961559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:20.594434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794706+0000","last_change":"2023-10-16T14:16:29.821655+0000","last_active":"2023-10-16T14:16:30.794706+0000","last_peered":"2023-10-16T14:16:30.794706+0000","last_clean":"2023-10-16T14:16:30.794706+0000","last_became_active":"2023-10-16T14:16:29.821357+0000","last_became_peered":"2023-10-16T14:16:29.821357+0000","last_unstale":"2023-10-16T14:16:30.794706+0000","last_undegraded":"2023-10-16T14:16:30.794706+0000","last_fullsized":"2023-10-16T14:16:30.794706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668435+0000","last_change":"2023-10-16T14:16:29.815493+0000","last_active":"2023-10-16T14:16:31.668435+0000","last_peered":"2023-10-16T14:16:31.668435+0000","last_clean":"2023-10-16T14:16:31.668435+0000","last_became_active":"2023-10-16T14:16:29.815203+0000","last_became_peered":"2023-10-16T14:16:29.815203+0000","last_unstale":"2023-10-16T14:16:31.668435+0000","last_undegraded":"2023-10-16T14:16:31.668435+0000","last_fullsized":"2023-10-16T14:16:31.668435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326367+0000","last_change":"2023-10-16T14:16:29.809001+0000","last_active":"2023-10-16T14:16:32.326367+0000","last_peered":"2023-10-16T14:16:32.326367+0000","last_clean":"2023-10-16T14:16:32.326367+0000","last_became_active":"2023-10-16T14:16:29.808169+0000","last_became_peered":"2023-10-16T14:16:29.808169+0000","last_unstale":"2023-10-16T14:16:32.326367+0000","last_undegraded":"2023-10-16T14:16:32.326367+0000","last_fullsized":"2023-10-16T14:16:32.326367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826340+0000","last_change":"2023-10-16T14:16:29.812984+0000","last_active":"2023-10-16T14:16:31.826340+0000","last_peered":"2023-10-16T14:16:31.826340+0000","last_clean":"2023-10-16T14:16:31.826340+0000","last_became_active":"2023-10-16T14:16:29.812569+0000","last_became_peered":"2023-10-16T14:16:29.812569+0000","last_unstale":"2023-10-16T14:16:31.826340+0000","last_undegraded":"2023-10-16T14:16:31.826340+0000","last_fullsized":"2023-10-16T14:16:31.826340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325543+0000","last_change":"2023-10-16T14:16:29.821793+0000","last_active":"2023-10-16T14:16:32.325543+0000","last_peered":"2023-10-16T14:16:32.325543+0000","last_clean":"2023-10-16T14:16:32.325543+0000","last_became_active":"2023-10-16T14:16:29.819170+0000","last_became_peered":"2023-10-16T14:16:29.819170+0000","last_unstale":"2023-10-16T14:16:32.325543+0000","last_undegraded":"2023-10-16T14:16:32.325543+0000","last_fullsized":"2023-10-16T14:16:32.325543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.975338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961200+0000","last_change":"2023-10-16T14:16:29.821091+0000","last_active":"2023-10-16T14:16:30.961200+0000","last_peered":"2023-10-16T14:16:30.961200+0000","last_clean":"2023-10-16T14:16:30.961200+0000","last_became_active":"2023-10-16T14:16:29.818555+0000","last_became_peered":"2023-10-16T14:16:29.818555+0000","last_unstale":"2023-10-16T14:16:30.961200+0000","last_undegraded":"2023-10-16T14:16:30.961200+0000","last_fullsized":"2023-10-16T14:16:30.961200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797715+0000","last_change":"2023-10-16T14:16:29.812942+0000","last_active":"2023-10-16T14:16:30.797715+0000","last_peered":"2023-10-16T14:16:30.797715+0000","last_clean":"2023-10-16T14:16:30.797715+0000","last_became_active":"2023-10-16T14:16:29.811742+0000","last_became_peered":"2023-10-16T14:16:29.811742+0000","last_unstale":"2023-10-16T14:16:30.797715+0000","last_undegraded":"2023-10-16T14:16:30.797715+0000","last_fullsized":"2023-10-16T14:16:30.797715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.153414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668347+0000","last_change":"2023-10-16T14:16:29.050890+0000","last_active":"2023-10-16T14:16:31.668347+0000","last_peered":"2023-10-16T14:16:31.668347+0000","last_clean":"2023-10-16T14:16:31.668347+0000","last_became_active":"2023-10-16T14:16:29.049249+0000","last_became_peered":"2023-10-16T14:16:29.049249+0000","last_unstale":"2023-10-16T14:16:31.668347+0000","last_undegraded":"2023-10-16T14:16:31.668347+0000","last_fullsized":"2023-10-16T14:16:31.668347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326412+0000","last_change":"2023-10-16T14:16:29.820123+0000","last_active":"2023-10-16T14:16:32.326412+0000","last_peered":"2023-10-16T14:16:32.326412+0000","last_clean":"2023-10-16T14:16:32.326412+0000","last_became_active":"2023-10-16T14:16:29.819857+0000","last_became_peered":"2023-10-16T14:16:29.819857+0000","last_unstale":"2023-10-16T14:16:32.326412+0000","last_undegraded":"2023-10-16T14:16:32.326412+0000","last_fullsized":"2023-10-16T14:16:32.326412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960647+0000","last_change":"2023-10-16T14:16:29.053825+0000","last_active":"2023-10-16T14:16:30.960647+0000","last_peered":"2023-10-16T14:16:30.960647+0000","last_clean":"2023-10-16T14:16:30.960647+0000","last_became_active":"2023-10-16T14:16:29.051055+0000","last_became_peered":"2023-10-16T14:16:29.051055+0000","last_unstale":"2023-10-16T14:16:30.960647+0000","last_undegraded":"2023-10-16T14:16:30.960647+0000","last_fullsized":"2023-10-16T14:16:30.960647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667594+0000","last_change":"2023-10-16T14:16:29.055594+0000","last_active":"2023-10-16T14:16:31.667594+0000","last_peered":"2023-10-16T14:16:31.667594+0000","last_clean":"2023-10-16T14:16:31.667594+0000","last_became_active":"2023-10-16T14:16:29.055260+0000","last_became_peered":"2023-10-16T14:16:29.055260+0000","last_unstale":"2023-10-16T14:16:31.667594+0000","last_undegraded":"2023-10-16T14:16:31.667594+0000","last_fullsized":"2023-10-16T14:16:31.667594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667384+0000","last_change":"2023-10-16T14:16:29.817463+0000","last_active":"2023-10-16T14:16:31.667384+0000","last_peered":"2023-10-16T14:16:31.667384+0000","last_clean":"2023-10-16T14:16:31.667384+0000","last_became_active":"2023-10-16T14:16:29.817155+0000","last_became_peered":"2023-10-16T14:16:29.817155+0000","last_unstale":"2023-10-16T14:16:31.667384+0000","last_undegraded":"2023-10-16T14:16:31.667384+0000","last_fullsized":"2023-10-16T14:16:31.667384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825958+0000","last_change":"2023-10-16T14:16:29.047697+0000","last_active":"2023-10-16T14:16:31.825958+0000","last_peered":"2023-10-16T14:16:31.825958+0000","last_clean":"2023-10-16T14:16:31.825958+0000","last_became_active":"2023-10-16T14:16:29.047399+0000","last_became_peered":"2023-10-16T14:16:29.047399+0000","last_unstale":"2023-10-16T14:16:31.825958+0000","last_undegraded":"2023-10-16T14:16:31.825958+0000","last_fullsized":"2023-10-16T14:16:31.825958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794058+0000","last_change":"2023-10-16T14:16:29.811517+0000","last_active":"2023-10-16T14:16:30.794058+0000","last_peered":"2023-10-16T14:16:30.794058+0000","last_clean":"2023-10-16T14:16:30.794058+0000","last_became_active":"2023-10-16T14:16:29.811178+0000","last_became_peered":"2023-10-16T14:16:29.811178+0000","last_unstale":"2023-10-16T14:16:30.794058+0000","last_undegraded":"2023-10-16T14:16:30.794058+0000","last_fullsized":"2023-10-16T14:16:30.794058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:19.312745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961080+0000","last_change":"2023-10-16T14:16:29.061919+0000","last_active":"2023-10-16T14:16:30.961080+0000","last_peered":"2023-10-16T14:16:30.961080+0000","last_clean":"2023-10-16T14:16:30.961080+0000","last_became_active":"2023-10-16T14:16:29.061130+0000","last_became_peered":"2023-10-16T14:16:29.061130+0000","last_unstale":"2023-10-16T14:16:30.961080+0000","last_undegraded":"2023-10-16T14:16:30.961080+0000","last_fullsized":"2023-10-16T14:16:30.961080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826710+0000","last_change":"2023-10-16T14:16:29.817592+0000","last_active":"2023-10-16T14:16:31.826710+0000","last_peered":"2023-10-16T14:16:31.826710+0000","last_clean":"2023-10-16T14:16:31.826710+0000","last_became_active":"2023-10-16T14:16:29.817291+0000","last_became_peered":"2023-10-16T14:16:29.817291+0000","last_unstale":"2023-10-16T14:16:31.826710+0000","last_undegraded":"2023-10-16T14:16:31.826710+0000","last_fullsized":"2023-10-16T14:16:31.826710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:47.164372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826905+0000","last_change":"2023-10-16T14:16:29.059877+0000","last_active":"2023-10-16T14:16:31.826905+0000","last_peered":"2023-10-16T14:16:31.826905+0000","last_clean":"2023-10-16T14:16:31.826905+0000","last_became_active":"2023-10-16T14:16:29.059080+0000","last_became_peered":"2023-10-16T14:16:29.059080+0000","last_unstale":"2023-10-16T14:16:31.826905+0000","last_undegraded":"2023-10-16T14:16:31.826905+0000","last_fullsized":"2023-10-16T14:16:31.826905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:40.688481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326692+0000","last_change":"2023-10-16T14:16:29.827311+0000","last_active":"2023-10-16T14:16:32.326692+0000","last_peered":"2023-10-16T14:16:32.326692+0000","last_clean":"2023-10-16T14:16:32.326692+0000","last_became_active":"2023-10-16T14:16:29.827004+0000","last_became_peered":"2023-10-16T14:16:29.827004+0000","last_unstale":"2023-10-16T14:16:32.326692+0000","last_undegraded":"2023-10-16T14:16:32.326692+0000","last_fullsized":"2023-10-16T14:16:32.326692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794725+0000","last_change":"2023-10-16T14:16:29.059819+0000","last_active":"2023-10-16T14:16:30.794725+0000","last_peered":"2023-10-16T14:16:30.794725+0000","last_clean":"2023-10-16T14:16:30.794725+0000","last_became_active":"2023-10-16T14:16:29.059474+0000","last_became_peered":"2023-10-16T14:16:29.059474+0000","last_unstale":"2023-10-16T14:16:30.794725+0000","last_undegraded":"2023-10-16T14:16:30.794725+0000","last_fullsized":"2023-10-16T14:16:30.794725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797712+0000","last_change":"2023-10-16T14:16:29.813588+0000","last_active":"2023-10-16T14:16:30.797712+0000","last_peered":"2023-10-16T14:16:30.797712+0000","last_clean":"2023-10-16T14:16:30.797712+0000","last_became_active":"2023-10-16T14:16:29.813342+0000","last_became_peered":"2023-10-16T14:16:29.813342+0000","last_unstale":"2023-10-16T14:16:30.797712+0000","last_undegraded":"2023-10-16T14:16:30.797712+0000","last_fullsized":"2023-10-16T14:16:30.797712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:36.779577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961265+0000","last_change":"2023-10-16T14:16:29.057707+0000","last_active":"2023-10-16T14:16:30.961265+0000","last_peered":"2023-10-16T14:16:30.961265+0000","last_clean":"2023-10-16T14:16:30.961265+0000","last_became_active":"2023-10-16T14:16:29.054590+0000","last_became_peered":"2023-10-16T14:16:29.054590+0000","last_unstale":"2023-10-16T14:16:30.961265+0000","last_undegraded":"2023-10-16T14:16:30.961265+0000","last_fullsized":"2023-10-16T14:16:30.961265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797570+0000","last_change":"2023-10-16T14:16:29.810243+0000","last_active":"2023-10-16T14:16:30.797570+0000","last_peered":"2023-10-16T14:16:30.797570+0000","last_clean":"2023-10-16T14:16:30.797570+0000","last_became_active":"2023-10-16T14:16:29.809877+0000","last_became_peered":"2023-10-16T14:16:29.809877+0000","last_unstale":"2023-10-16T14:16:30.797570+0000","last_undegraded":"2023-10-16T14:16:30.797570+0000","last_fullsized":"2023-10-16T14:16:30.797570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:32.937773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326441+0000","last_change":"2023-10-16T14:16:29.046872+0000","last_active":"2023-10-16T14:16:32.326441+0000","last_peered":"2023-10-16T14:16:32.326441+0000","last_clean":"2023-10-16T14:16:32.326441+0000","last_became_active":"2023-10-16T14:16:29.046411+0000","last_became_peered":"2023-10-16T14:16:29.046411+0000","last_unstale":"2023-10-16T14:16:32.326441+0000","last_undegraded":"2023-10-16T14:16:32.326441+0000","last_fullsized":"2023-10-16T14:16:32.326441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:48.638693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326040+0000","last_change":"2023-10-16T14:16:29.810247+0000","last_active":"2023-10-16T14:16:32.326040+0000","last_peered":"2023-10-16T14:16:32.326040+0000","last_clean":"2023-10-16T14:16:32.326040+0000","last_became_active":"2023-10-16T14:16:29.809830+0000","last_became_peered":"2023-10-16T14:16:29.809830+0000","last_unstale":"2023-10-16T14:16:32.326040+0000","last_undegraded":"2023-10-16T14:16:32.326040+0000","last_fullsized":"2023-10-16T14:16:32.326040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:26.623468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826189+0000","last_change":"2023-10-16T14:16:29.049519+0000","last_active":"2023-10-16T14:16:31.826189+0000","last_peered":"2023-10-16T14:16:31.826189+0000","last_clean":"2023-10-16T14:16:31.826189+0000","last_became_active":"2023-10-16T14:16:29.049203+0000","last_became_peered":"2023-10-16T14:16:29.049203+0000","last_unstale":"2023-10-16T14:16:31.826189+0000","last_undegraded":"2023-10-16T14:16:31.826189+0000","last_fullsized":"2023-10-16T14:16:31.826189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:38.962308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798901+0000","last_change":"2023-10-16T14:16:29.824625+0000","last_active":"2023-10-16T14:16:30.798901+0000","last_peered":"2023-10-16T14:16:30.798901+0000","last_clean":"2023-10-16T14:16:30.798901+0000","last_became_active":"2023-10-16T14:16:29.818988+0000","last_became_peered":"2023-10-16T14:16:29.818988+0000","last_unstale":"2023-10-16T14:16:30.798901+0000","last_undegraded":"2023-10-16T14:16:30.798901+0000","last_fullsized":"2023-10-16T14:16:30.798901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960552+0000","last_change":"2023-10-16T14:16:29.054389+0000","last_active":"2023-10-16T14:16:30.960552+0000","last_peered":"2023-10-16T14:16:30.960552+0000","last_clean":"2023-10-16T14:16:30.960552+0000","last_became_active":"2023-10-16T14:16:29.053542+0000","last_became_peered":"2023-10-16T14:16:29.053542+0000","last_unstale":"2023-10-16T14:16:30.960552+0000","last_undegraded":"2023-10-16T14:16:30.960552+0000","last_fullsized":"2023-10-16T14:16:30.960552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960683+0000","last_change":"2023-10-16T14:16:29.822533+0000","last_active":"2023-10-16T14:16:30.960683+0000","last_peered":"2023-10-16T14:16:30.960683+0000","last_clean":"2023-10-16T14:16:30.960683+0000","last_became_active":"2023-10-16T14:16:29.822130+0000","last_became_peered":"2023-10-16T14:16:29.822130+0000","last_unstale":"2023-10-16T14:16:30.960683+0000","last_undegraded":"2023-10-16T14:16:30.960683+0000","last_fullsized":"2023-10-16T14:16:30.960683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:19.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325417+0000","last_change":"2023-10-16T14:16:29.044471+0000","last_active":"2023-10-16T14:16:32.325417+0000","last_peered":"2023-10-16T14:16:32.325417+0000","last_clean":"2023-10-16T14:16:32.325417+0000","last_became_active":"2023-10-16T14:16:29.042866+0000","last_became_peered":"2023-10-16T14:16:29.042866+0000","last_unstale":"2023-10-16T14:16:32.325417+0000","last_undegraded":"2023-10-16T14:16:32.325417+0000","last_fullsized":"2023-10-16T14:16:32.325417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:01.067871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799086+0000","last_change":"2023-10-16T14:16:29.826425+0000","last_active":"2023-10-16T14:16:30.799086+0000","last_peered":"2023-10-16T14:16:30.799086+0000","last_clean":"2023-10-16T14:16:30.799086+0000","last_became_active":"2023-10-16T14:16:29.820157+0000","last_became_peered":"2023-10-16T14:16:29.820157+0000","last_unstale":"2023-10-16T14:16:30.799086+0000","last_undegraded":"2023-10-16T14:16:30.799086+0000","last_fullsized":"2023-10-16T14:16:30.799086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:33.334841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797731+0000","last_change":"2023-10-16T14:16:29.049608+0000","last_active":"2023-10-16T14:16:30.797731+0000","last_peered":"2023-10-16T14:16:30.797731+0000","last_clean":"2023-10-16T14:16:30.797731+0000","last_became_active":"2023-10-16T14:16:29.049125+0000","last_became_peered":"2023-10-16T14:16:29.049125+0000","last_unstale":"2023-10-16T14:16:30.797731+0000","last_undegraded":"2023-10-16T14:16:30.797731+0000","last_fullsized":"2023-10-16T14:16:30.797731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798834+0000","last_change":"2023-10-16T14:16:29.809718+0000","last_active":"2023-10-16T14:16:30.798834+0000","last_peered":"2023-10-16T14:16:30.798834+0000","last_clean":"2023-10-16T14:16:30.798834+0000","last_became_active":"2023-10-16T14:16:29.809294+0000","last_became_peered":"2023-10-16T14:16:29.809294+0000","last_unstale":"2023-10-16T14:16:30.798834+0000","last_undegraded":"2023-10-16T14:16:30.798834+0000","last_fullsized":"2023-10-16T14:16:30.798834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325704+0000","last_change":"2023-10-16T14:16:29.034974+0000","last_active":"2023-10-16T14:16:32.325704+0000","last_peered":"2023-10-16T14:16:32.325704+0000","last_clean":"2023-10-16T14:16:32.325704+0000","last_became_active":"2023-10-16T14:16:29.034651+0000","last_became_peered":"2023-10-16T14:16:29.034651+0000","last_unstale":"2023-10-16T14:16:32.325704+0000","last_undegraded":"2023-10-16T14:16:32.325704+0000","last_fullsized":"2023-10-16T14:16:32.325704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:04.878575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.799415+0000","last_change":"2023-10-16T14:16:29.829055+0000","last_active":"2023-10-16T14:16:30.799415+0000","last_peered":"2023-10-16T14:16:30.799415+0000","last_clean":"2023-10-16T14:16:30.799415+0000","last_became_active":"2023-10-16T14:16:29.820785+0000","last_became_peered":"2023-10-16T14:16:29.820785+0000","last_unstale":"2023-10-16T14:16:30.799415+0000","last_undegraded":"2023-10-16T14:16:30.799415+0000","last_fullsized":"2023-10-16T14:16:30.799415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794821+0000","last_change":"2023-10-16T14:16:29.048896+0000","last_active":"2023-10-16T14:16:30.794821+0000","last_peered":"2023-10-16T14:16:30.794821+0000","last_clean":"2023-10-16T14:16:30.794821+0000","last_became_active":"2023-10-16T14:16:29.048513+0000","last_became_peered":"2023-10-16T14:16:29.048513+0000","last_unstale":"2023-10-16T14:16:30.794821+0000","last_undegraded":"2023-10-16T14:16:30.794821+0000","last_fullsized":"2023-10-16T14:16:30.794821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:25.610438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.827198+0000","last_change":"2023-10-16T14:16:29.814236+0000","last_active":"2023-10-16T14:16:31.827198+0000","last_peered":"2023-10-16T14:16:31.827198+0000","last_clean":"2023-10-16T14:16:31.827198+0000","last_became_active":"2023-10-16T14:16:29.813943+0000","last_became_peered":"2023-10-16T14:16:29.813943+0000","last_unstale":"2023-10-16T14:16:31.827198+0000","last_undegraded":"2023-10-16T14:16:31.827198+0000","last_fullsized":"2023-10-16T14:16:31.827198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.092441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325524+0000","last_change":"2023-10-16T14:16:29.052741+0000","last_active":"2023-10-16T14:16:32.325524+0000","last_peered":"2023-10-16T14:16:32.325524+0000","last_clean":"2023-10-16T14:16:32.325524+0000","last_became_active":"2023-10-16T14:16:29.052376+0000","last_became_peered":"2023-10-16T14:16:29.052376+0000","last_unstale":"2023-10-16T14:16:32.325524+0000","last_undegraded":"2023-10-16T14:16:32.325524+0000","last_fullsized":"2023-10-16T14:16:32.325524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:40.014521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797919+0000","last_change":"2023-10-16T14:16:29.828744+0000","last_active":"2023-10-16T14:16:30.797919+0000","last_peered":"2023-10-16T14:16:30.797919+0000","last_clean":"2023-10-16T14:16:30.797919+0000","last_became_active":"2023-10-16T14:16:29.823064+0000","last_became_peered":"2023-10-16T14:16:29.823064+0000","last_unstale":"2023-10-16T14:16:30.797919+0000","last_undegraded":"2023-10-16T14:16:30.797919+0000","last_fullsized":"2023-10-16T14:16:30.797919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-16T14:16:29.790600+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:16:29.790600+0000","last_peered":"2023-10-16T14:16:29.790600+0000","last_clean":"2023-10-16T14:16:29.790600+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:16:29.790600+0000","last_undegraded":"2023-10-16T14:16:29.790600+0000","last_fullsized":"2023-10-16T14:16:29.790600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.825962+0000","last_change":"2023-10-16T14:16:29.051990+0000","last_active":"2023-10-16T14:16:31.825962+0000","last_peered":"2023-10-16T14:16:31.825962+0000","last_clean":"2023-10-16T14:16:31.825962+0000","last_became_active":"2023-10-16T14:16:29.051723+0000","last_became_peered":"2023-10-16T14:16:29.051723+0000","last_unstale":"2023-10-16T14:16:31.825962+0000","last_undegraded":"2023-10-16T14:16:31.825962+0000","last_fullsized":"2023-10-16T14:16:31.825962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667924+0000","last_change":"2023-10-16T14:16:29.823986+0000","last_active":"2023-10-16T14:16:31.667924+0000","last_peered":"2023-10-16T14:16:31.667924+0000","last_clean":"2023-10-16T14:16:31.667924+0000","last_became_active":"2023-10-16T14:16:29.823102+0000","last_became_peered":"2023-10-16T14:16:29.823102+0000","last_unstale":"2023-10-16T14:16:31.667924+0000","last_undegraded":"2023-10-16T14:16:31.667924+0000","last_fullsized":"2023-10-16T14:16:31.667924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795970+0000","last_change":"2023-10-16T14:16:29.058003+0000","last_active":"2023-10-16T14:16:30.795970+0000","last_peered":"2023-10-16T14:16:30.795970+0000","last_clean":"2023-10-16T14:16:30.795970+0000","last_became_active":"2023-10-16T14:16:29.057686+0000","last_became_peered":"2023-10-16T14:16:29.057686+0000","last_unstale":"2023-10-16T14:16:30.795970+0000","last_undegraded":"2023-10-16T14:16:30.795970+0000","last_fullsized":"2023-10-16T14:16:30.795970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325936+0000","last_change":"2023-10-16T14:16:29.812594+0000","last_active":"2023-10-16T14:16:32.325936+0000","last_peered":"2023-10-16T14:16:32.325936+0000","last_clean":"2023-10-16T14:16:32.325936+0000","last_became_active":"2023-10-16T14:16:29.812256+0000","last_became_peered":"2023-10-16T14:16:29.812256+0000","last_unstale":"2023-10-16T14:16:32.325936+0000","last_undegraded":"2023-10-16T14:16:32.325936+0000","last_fullsized":"2023-10-16T14:16:32.325936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:48.787869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.325633+0000","last_change":"2023-10-16T14:16:29.042443+0000","last_active":"2023-10-16T14:16:32.325633+0000","last_peered":"2023-10-16T14:16:32.325633+0000","last_clean":"2023-10-16T14:16:32.325633+0000","last_became_active":"2023-10-16T14:16:29.042189+0000","last_became_peered":"2023-10-16T14:16:29.042189+0000","last_unstale":"2023-10-16T14:16:32.325633+0000","last_undegraded":"2023-10-16T14:16:32.325633+0000","last_fullsized":"2023-10-16T14:16:32.325633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:46.550305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794175+0000","last_change":"2023-10-16T14:16:29.816645+0000","last_active":"2023-10-16T14:16:30.794175+0000","last_peered":"2023-10-16T14:16:30.794175+0000","last_clean":"2023-10-16T14:16:30.794175+0000","last_became_active":"2023-10-16T14:16:29.814792+0000","last_became_peered":"2023-10-16T14:16:29.814792+0000","last_unstale":"2023-10-16T14:16:30.794175+0000","last_undegraded":"2023-10-16T14:16:30.794175+0000","last_fullsized":"2023-10-16T14:16:30.794175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826471+0000","last_change":"2023-10-16T14:16:29.055439+0000","last_active":"2023-10-16T14:16:31.826471+0000","last_peered":"2023-10-16T14:16:31.826471+0000","last_clean":"2023-10-16T14:16:31.826471+0000","last_became_active":"2023-10-16T14:16:29.054422+0000","last_became_peered":"2023-10-16T14:16:29.054422+0000","last_unstale":"2023-10-16T14:16:31.826471+0000","last_undegraded":"2023-10-16T14:16:31.826471+0000","last_fullsized":"2023-10-16T14:16:31.826471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.797290+0000","last_change":"2023-10-16T14:16:29.826562+0000","last_active":"2023-10-16T14:16:30.797290+0000","last_peered":"2023-10-16T14:16:30.797290+0000","last_clean":"2023-10-16T14:16:30.797290+0000","last_became_active":"2023-10-16T14:16:29.825184+0000","last_became_peered":"2023-10-16T14:16:29.825184+0000","last_unstale":"2023-10-16T14:16:30.797290+0000","last_undegraded":"2023-10-16T14:16:30.797290+0000","last_fullsized":"2023-10-16T14:16:30.797290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:49.651724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794587+0000","last_change":"2023-10-16T14:16:29.048497+0000","last_active":"2023-10-16T14:16:30.794587+0000","last_peered":"2023-10-16T14:16:30.794587+0000","last_clean":"2023-10-16T14:16:30.794587+0000","last_became_active":"2023-10-16T14:16:29.048194+0000","last_became_peered":"2023-10-16T14:16:29.048194+0000","last_unstale":"2023-10-16T14:16:30.794587+0000","last_undegraded":"2023-10-16T14:16:30.794587+0000","last_fullsized":"2023-10-16T14:16:30.794587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:47.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826954+0000","last_change":"2023-10-16T14:16:29.819205+0000","last_active":"2023-10-16T14:16:31.826954+0000","last_peered":"2023-10-16T14:16:31.826954+0000","last_clean":"2023-10-16T14:16:31.826954+0000","last_became_active":"2023-10-16T14:16:29.818921+0000","last_became_peered":"2023-10-16T14:16:29.818921+0000","last_unstale":"2023-10-16T14:16:31.826954+0000","last_undegraded":"2023-10-16T14:16:31.826954+0000","last_fullsized":"2023-10-16T14:16:31.826954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:26.208703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798586+0000","last_change":"2023-10-16T14:16:29.053513+0000","last_active":"2023-10-16T14:16:30.798586+0000","last_peered":"2023-10-16T14:16:30.798586+0000","last_clean":"2023-10-16T14:16:30.798586+0000","last_became_active":"2023-10-16T14:16:29.053178+0000","last_became_peered":"2023-10-16T14:16:29.053178+0000","last_unstale":"2023-10-16T14:16:30.798586+0000","last_undegraded":"2023-10-16T14:16:30.798586+0000","last_fullsized":"2023-10-16T14:16:30.798586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:18.712004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794789+0000","last_change":"2023-10-16T14:16:29.817442+0000","last_active":"2023-10-16T14:16:30.794789+0000","last_peered":"2023-10-16T14:16:30.794789+0000","last_clean":"2023-10-16T14:16:30.794789+0000","last_became_active":"2023-10-16T14:16:29.814899+0000","last_became_peered":"2023-10-16T14:16:29.814899+0000","last_unstale":"2023-10-16T14:16:30.794789+0000","last_undegraded":"2023-10-16T14:16:30.794789+0000","last_fullsized":"2023-10-16T14:16:30.794789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.915468+0000","last_change":"2023-10-16T14:16:29.059226+0000","last_active":"2023-10-16T14:16:32.915468+0000","last_peered":"2023-10-16T14:16:32.915468+0000","last_clean":"2023-10-16T14:16:32.915468+0000","last_became_active":"2023-10-16T14:16:29.058896+0000","last_became_peered":"2023-10-16T14:16:29.058896+0000","last_unstale":"2023-10-16T14:16:32.915468+0000","last_undegraded":"2023-10-16T14:16:32.915468+0000","last_fullsized":"2023-10-16T14:16:32.915468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668348+0000","last_change":"2023-10-16T14:16:29.807423+0000","last_active":"2023-10-16T14:16:31.668348+0000","last_peered":"2023-10-16T14:16:31.668348+0000","last_clean":"2023-10-16T14:16:31.668348+0000","last_became_active":"2023-10-16T14:16:29.806278+0000","last_became_peered":"2023-10-16T14:16:29.806278+0000","last_unstale":"2023-10-16T14:16:31.668348+0000","last_undegraded":"2023-10-16T14:16:31.668348+0000","last_fullsized":"2023-10-16T14:16:31.668348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:54.918621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.961275+0000","last_change":"2023-10-16T14:16:29.057876+0000","last_active":"2023-10-16T14:16:30.961275+0000","last_peered":"2023-10-16T14:16:30.961275+0000","last_clean":"2023-10-16T14:16:30.961275+0000","last_became_active":"2023-10-16T14:16:29.053426+0000","last_became_peered":"2023-10-16T14:16:29.053426+0000","last_unstale":"2023-10-16T14:16:30.961275+0000","last_undegraded":"2023-10-16T14:16:30.961275+0000","last_fullsized":"2023-10-16T14:16:30.961275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:01.418897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326675+0000","last_change":"2023-10-16T14:16:29.816014+0000","last_active":"2023-10-16T14:16:32.326675+0000","last_peered":"2023-10-16T14:16:32.326675+0000","last_clean":"2023-10-16T14:16:32.326675+0000","last_became_active":"2023-10-16T14:16:29.815725+0000","last_became_peered":"2023-10-16T14:16:29.815725+0000","last_unstale":"2023-10-16T14:16:32.326675+0000","last_undegraded":"2023-10-16T14:16:32.326675+0000","last_fullsized":"2023-10-16T14:16:32.326675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:19.895425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.795715+0000","last_change":"2023-10-16T14:16:29.050583+0000","last_active":"2023-10-16T14:16:30.795715+0000","last_peered":"2023-10-16T14:16:30.795715+0000","last_clean":"2023-10-16T14:16:30.795715+0000","last_became_active":"2023-10-16T14:16:29.050281+0000","last_became_peered":"2023-10-16T14:16:29.050281+0000","last_unstale":"2023-10-16T14:16:30.795715+0000","last_undegraded":"2023-10-16T14:16:30.795715+0000","last_fullsized":"2023-10-16T14:16:30.795715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:18.882395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.796790+0000","last_change":"2023-10-16T14:16:29.808419+0000","last_active":"2023-10-16T14:16:30.796790+0000","last_peered":"2023-10-16T14:16:30.796790+0000","last_clean":"2023-10-16T14:16:30.796790+0000","last_became_active":"2023-10-16T14:16:29.808038+0000","last_became_peered":"2023-10-16T14:16:29.808038+0000","last_unstale":"2023-10-16T14:16:30.796790+0000","last_undegraded":"2023-10-16T14:16:30.796790+0000","last_fullsized":"2023-10-16T14:16:30.796790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:05.891605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794294+0000","last_change":"2023-10-16T14:16:29.049120+0000","last_active":"2023-10-16T14:16:30.794294+0000","last_peered":"2023-10-16T14:16:30.794294+0000","last_clean":"2023-10-16T14:16:30.794294+0000","last_became_active":"2023-10-16T14:16:29.048782+0000","last_became_peered":"2023-10-16T14:16:29.048782+0000","last_unstale":"2023-10-16T14:16:30.794294+0000","last_undegraded":"2023-10-16T14:16:30.794294+0000","last_fullsized":"2023-10-16T14:16:30.794294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:36.052306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.667807+0000","last_change":"2023-10-16T14:16:29.820694+0000","last_active":"2023-10-16T14:16:31.667807+0000","last_peered":"2023-10-16T14:16:31.667807+0000","last_clean":"2023-10-16T14:16:31.667807+0000","last_became_active":"2023-10-16T14:16:29.820341+0000","last_became_peered":"2023-10-16T14:16:29.820341+0000","last_unstale":"2023-10-16T14:16:31.667807+0000","last_undegraded":"2023-10-16T14:16:31.667807+0000","last_fullsized":"2023-10-16T14:16:31.667807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.794452+0000","last_change":"2023-10-16T14:16:29.043057+0000","last_active":"2023-10-16T14:16:30.794452+0000","last_peered":"2023-10-16T14:16:30.794452+0000","last_clean":"2023-10-16T14:16:30.794452+0000","last_became_active":"2023-10-16T14:16:29.042783+0000","last_became_peered":"2023-10-16T14:16:29.042783+0000","last_unstale":"2023-10-16T14:16:30.794452+0000","last_undegraded":"2023-10-16T14:16:30.794452+0000","last_fullsized":"2023-10-16T14:16:30.794452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:35.766547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668050+0000","last_change":"2023-10-16T14:16:29.816920+0000","last_active":"2023-10-16T14:16:31.668050+0000","last_peered":"2023-10-16T14:16:31.668050+0000","last_clean":"2023-10-16T14:16:31.668050+0000","last_became_active":"2023-10-16T14:16:29.816639+0000","last_became_peered":"2023-10-16T14:16:29.816639+0000","last_unstale":"2023-10-16T14:16:31.668050+0000","last_undegraded":"2023-10-16T14:16:31.668050+0000","last_fullsized":"2023-10-16T14:16:31.668050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:18.709150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798966+0000","last_change":"2023-10-16T14:16:29.056773+0000","last_active":"2023-10-16T14:16:30.798966+0000","last_peered":"2023-10-16T14:16:30.798966+0000","last_clean":"2023-10-16T14:16:30.798966+0000","last_became_active":"2023-10-16T14:16:29.056477+0000","last_became_peered":"2023-10-16T14:16:29.056477+0000","last_unstale":"2023-10-16T14:16:30.798966+0000","last_undegraded":"2023-10-16T14:16:30.798966+0000","last_fullsized":"2023-10-16T14:16:30.798966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:17.696119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.960869+0000","last_change":"2023-10-16T14:16:29.821383+0000","last_active":"2023-10-16T14:16:30.960869+0000","last_peered":"2023-10-16T14:16:30.960869+0000","last_clean":"2023-10-16T14:16:30.960869+0000","last_became_active":"2023-10-16T14:16:29.821018+0000","last_became_peered":"2023-10-16T14:16:29.821018+0000","last_unstale":"2023-10-16T14:16:30.960869+0000","last_undegraded":"2023-10-16T14:16:30.960869+0000","last_fullsized":"2023-10-16T14:16:30.960869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:41.701511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826784+0000","last_change":"2023-10-16T14:16:29.829570+0000","last_active":"2023-10-16T14:16:31.826784+0000","last_peered":"2023-10-16T14:16:31.826784+0000","last_clean":"2023-10-16T14:16:31.826784+0000","last_became_active":"2023-10-16T14:16:29.828707+0000","last_became_peered":"2023-10-16T14:16:29.828707+0000","last_unstale":"2023-10-16T14:16:31.826784+0000","last_undegraded":"2023-10-16T14:16:31.826784+0000","last_fullsized":"2023-10-16T14:16:31.826784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:08.084157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798241+0000","last_change":"2023-10-16T14:16:29.063205+0000","last_active":"2023-10-16T14:16:30.798241+0000","last_peered":"2023-10-16T14:16:30.798241+0000","last_clean":"2023-10-16T14:16:30.798241+0000","last_became_active":"2023-10-16T14:16:29.062657+0000","last_became_peered":"2023-10-16T14:16:29.062657+0000","last_unstale":"2023-10-16T14:16:30.798241+0000","last_undegraded":"2023-10-16T14:16:30.798241+0000","last_fullsized":"2023-10-16T14:16:30.798241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:43.103258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.826847+0000","last_change":"2023-10-16T14:16:29.830412+0000","last_active":"2023-10-16T14:16:31.826847+0000","last_peered":"2023-10-16T14:16:31.826847+0000","last_clean":"2023-10-16T14:16:31.826847+0000","last_became_active":"2023-10-16T14:16:29.825817+0000","last_became_peered":"2023-10-16T14:16:29.825817+0000","last_unstale":"2023-10-16T14:16:31.826847+0000","last_undegraded":"2023-10-16T14:16:31.826847+0000","last_fullsized":"2023-10-16T14:16:31.826847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:01.070465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:30.798099+0000","last_change":"2023-10-16T14:16:29.061570+0000","last_active":"2023-10-16T14:16:30.798099+0000","last_peered":"2023-10-16T14:16:30.798099+0000","last_clean":"2023-10-16T14:16:30.798099+0000","last_became_active":"2023-10-16T14:16:29.061024+0000","last_became_peered":"2023-10-16T14:16:29.061024+0000","last_unstale":"2023-10-16T14:16:30.798099+0000","last_undegraded":"2023-10-16T14:16:30.798099+0000","last_fullsized":"2023-10-16T14:16:30.798099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326481+0000","last_change":"2023-10-16T14:16:29.827404+0000","last_active":"2023-10-16T14:16:32.326481+0000","last_peered":"2023-10-16T14:16:32.326481+0000","last_clean":"2023-10-16T14:16:32.326481+0000","last_became_active":"2023-10-16T14:16:29.826687+0000","last_became_peered":"2023-10-16T14:16:29.826687+0000","last_unstale":"2023-10-16T14:16:32.326481+0000","last_undegraded":"2023-10-16T14:16:32.326481+0000","last_fullsized":"2023-10-16T14:16:32.326481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:44.116288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668592+0000","last_change":"2023-10-16T14:16:29.050871+0000","last_active":"2023-10-16T14:16:31.668592+0000","last_peered":"2023-10-16T14:16:31.668592+0000","last_clean":"2023-10-16T14:16:31.668592+0000","last_became_active":"2023-10-16T14:16:29.050025+0000","last_became_peered":"2023-10-16T14:16:29.050025+0000","last_unstale":"2023-10-16T14:16:31.668592+0000","last_undegraded":"2023-10-16T14:16:31.668592+0000","last_fullsized":"2023-10-16T14:16:31.668592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.934109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:31.668230+0000","last_change":"2023-10-16T14:16:29.825092+0000","last_active":"2023-10-16T14:16:31.668230+0000","last_peered":"2023-10-16T14:16:31.668230+0000","last_clean":"2023-10-16T14:16:31.668230+0000","last_became_active":"2023-10-16T14:16:29.824815+0000","last_became_peered":"2023-10-16T14:16:29.824815+0000","last_unstale":"2023-10-16T14:16:31.668230+0000","last_undegraded":"2023-10-16T14:16:31.668230+0000","last_fullsized":"2023-10-16T14:16:31.668230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:29.000940+0000","last_clean_scrub_stamp":"2023-10-16T14:16:29.000940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:09.338970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-16T14:16:32.326685+0000","last_change":"2023-10-16T14:16:29.053310+0000","last_active":"2023-10-16T14:16:32.326685+0000","last_peered":"2023-10-16T14:16:32.326685+0000","last_clean":"2023-10-16T14:16:32.326685+0000","last_became_active":"2023-10-16T14:16:29.052871+0000","last_became_peered":"2023-10-16T14:16:29.052871+0000","last_unstale":"2023-10-16T14:16:32.326685+0000","last_undegraded":"2023-10-16T14:16:32.326685+0000","last_fullsized":"2023-10-16T14:16:32.326685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:27.987909+0000","last_clean_scrub_stamp":"2023-10-16T14:16:27.987909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:16.742399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4830000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9930000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9830000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.165}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2650000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.653}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.391}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9369999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5779999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7610000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:16:40.444 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T14:16:40.444 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-16T14:16:40.444 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-16T14:16:40.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T14:16:40.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.707+0000 7f2b5c190640 1 Processor -- start 2023-10-16T14:16:40.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.708+0000 7f2b5c190640 1 -- start start 2023-10-16T14:16:40.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.709+0000 7f2b5c190640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b5406a4b0 0x7f2b540c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.709+0000 7f2b5c190640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b540ac890 con 0x7f2b540ad350 2023-10-16T14:16:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.709+0000 7f2b5c190640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b540ad7a0 con 0x7f2b5415d150 2023-10-16T14:16:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.709+0000 7f2b5c190640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5415d530 con 0x7f2b5406a4b0 2023-10-16T14:16:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.709+0000 7f2b5972b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2b540ad350 0x7f2b54069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46400/0 (socket says 172.21.15.29:46400) 2023-10-16T14:16:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.709+0000 7f2b5972b640 1 -- 172.21.15.29:0/3588861156 learned_addr learned my addr 172.21.15.29:0/3588861156 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478970315 0 0) 0x7f2b540ac890 con 0x7f2b540ad350 2023-10-16T14:16:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/3588861156 >> v2:172.21.15.170:3300/0 conn(0x7f2b5406a4b0 0x7f2b540c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b40003520 con 0x7f2b540ad350 2023-10-16T14:16:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1764079288 0 0) 0x7f2b40003520 con 0x7f2b540ad350 2023-10-16T14:16:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 >> v2:172.21.15.170:3300/0 conn(0x7f2b5406a4b0 msgr2=0x7f2b540c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 --2- 172.21.15.29:0/3588861156 >> v2:172.21.15.170:3300/0 conn(0x7f2b5406a4b0 0x7f2b540c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 >> v1:172.21.15.170:6789/0 conn(0x7f2b5415d150 legacy=0x7f2b54160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b54161650 con 0x7f2b540ad350 2023-10-16T14:16:40.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2b44001670 con 0x7f2b540ad350 2023-10-16T14:16:40.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 >> v1:172.21.15.29:6789/0 conn(0x7f2b540ad350 legacy=0x7f2b54069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b40003970 0x7f2b40003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b400047c0 0x7f2b40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5415d530 con 0x7f2b40005180 2023-10-16T14:16:40.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.710+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4000c050 con 0x7f2b400047c0 2023-10-16T14:16:40.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4000c210 con 0x7f2b40003970 2023-10-16T14:16:40.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b4000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5972b640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b400047c0 0x7f2b40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b59f2c640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b40003970 0x7f2b40003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5a72d640 1 -- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b40003970 msgr2=0x7f2b40003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b40003970 0x7f2b40003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5a72d640 1 -- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b400047c0 msgr2=0x7f2b40004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:40.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b400047c0 0x7f2b40004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5a72d640 1 -- 172.21.15.29:0/3588861156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b40003520 con 0x7f2b40005180 2023-10-16T14:16:40.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b5972b640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b400047c0 0x7f2b40004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:40.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.711+0000 7f2b59f2c640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b40003970 0x7f2b40003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:40.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.712+0000 7f2b5a72d640 1 -- 172.21.15.29:0/3588861156 reap_dead start 2023-10-16T14:16:40.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.712+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b4000b980 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c007360 tx=0x7f2b400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586f0c73e37a56d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:40.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c02ccf0 con 0x7f2b40005180 2023-10-16T14:16:40.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c02ce90 con 0x7f2b40005180 2023-10-16T14:16:40.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b58f2a640 1 -- 172.21.15.29:0/3588861156 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c003bd0 con 0x7f2b40005180 2023-10-16T14:16:40.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 -- 172.21.15.29:0/3588861156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 msgr2=0x7f2b4000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b4000b980 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c007360 tx=0x7f2b400057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 -- 172.21.15.29:0/3588861156 shutdown_connections 2023-10-16T14:16:40.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 --2- 172.21.15.29:0/3588861156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b4000b980 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 -- 172.21.15.29:0/3588861156 >> 172.21.15.29:0/3588861156 conn(0x7f2b540ce9b0 msgr2=0x7f2b540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:40.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 -- 172.21.15.29:0/3588861156 shutdown_connections 2023-10-16T14:16:40.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.713+0000 7f2b5c190640 1 -- 172.21.15.29:0/3588861156 wait complete. 2023-10-16T14:16:40.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.715+0000 7f2b5c190640 1 Processor -- start 2023-10-16T14:16:40.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.715+0000 7f2b5c190640 1 -- start start 2023-10-16T14:16:40.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.716+0000 7f2b5c190640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b541f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.716+0000 7f2b5c190640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b540c36b0 0x7f2b540b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.716+0000 7f2b5c190640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b540c3b10 0x7f2b540b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.716+0000 7f2b5c190640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b540b17c0 con 0x7f2b40005180 2023-10-16T14:16:40.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.716+0000 7f2b5c190640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54150940 con 0x7f2b540c36b0 2023-10-16T14:16:40.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.716+0000 7f2b5c190640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b540b2fa0 con 0x7f2b540c3b10 2023-10-16T14:16:40.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b541f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b541f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51056/0 (socket says 172.21.15.29:51056) 2023-10-16T14:16:40.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 -- 172.21.15.29:0/1773399387 learned_addr learned my addr 172.21.15.29:0/1773399387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:40.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b5972b640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b540c36b0 0x7f2b540b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b540c3b10 0x7f2b540b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 -- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b540c3b10 msgr2=0x7f2b540b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b540c3b10 0x7f2b540b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 -- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b540c36b0 msgr2=0x7f2b540b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b540c36b0 0x7f2b540b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 -- 172.21.15.29:0/1773399387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b540c2610 con 0x7f2b40005180 2023-10-16T14:16:40.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b5a72d640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b540c3b10 0x7f2b540b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:40.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b5972b640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b540c36b0 0x7f2b540b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:40.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.717+0000 7f2b59f2c640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b541f2550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000d7f0 tx=0x7f2b5000dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:40.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.718+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50017070 con 0x7f2b40005180 2023-10-16T14:16:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.718+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b50005070 con 0x7f2b40005180 2023-10-16T14:16:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.718+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50005440 con 0x7f2b40005180 2023-10-16T14:16:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.718+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b540c28f0 con 0x7f2b40005180 2023-10-16T14:16:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.718+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b540c2eb0 con 0x7f2b40005180 2023-10-16T14:16:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.719+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5404e160 con 0x7f2b40005180 2023-10-16T14:16:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.720+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2b50007240 con 0x7f2b40005180 2023-10-16T14:16:40.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.721+0000 7f2b4a7fc640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2b3806ae70 0x7f2b3806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:40.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.721+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b50089200 con 0x7f2b40005180 2023-10-16T14:16:40.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.722+0000 7f2b5972b640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2b3806ae70 0x7f2b3806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:40.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.723+0000 7f2b5972b640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2b3806ae70 0x7f2b3806d330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2b5415c2e0 tx=0x7f2b44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:40.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.725+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b500542a0 con 0x7f2b40005180 2023-10-16T14:16:40.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.959+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2b541575b0 con 0x7f2b40005180 2023-10-16T14:16:40.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.960+0000 7f2b4a7fc640 1 -- 172.21.15.29:0/1773399387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2b500578c0 con 0x7f2b40005180 2023-10-16T14:16:40.964 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:40.964 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-16T14:16:40.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.962+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2b3806ae70 msgr2=0x7f2b3806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.962+0000 7f2b5c190640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2b3806ae70 0x7f2b3806d330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2b5415c2e0 tx=0x7f2b44009040 comp rx=0 tx=0).stop 2023-10-16T14:16:40.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.962+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 msgr2=0x7f2b541f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.962+0000 7f2b5c190640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b541f2550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000d7f0 tx=0x7f2b5000dcb0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 shutdown_connections 2023-10-16T14:16:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f2b3806ae70 0x7f2b3806d330 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b540c3b10 0x7f2b540b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b540c36b0 0x7f2b540b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 --2- 172.21.15.29:0/1773399387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b40005180 0x7f2b541f2550 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 >> 172.21.15.29:0/1773399387 conn(0x7f2b540ce9b0 msgr2=0x7f2b540ca850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 shutdown_connections 2023-10-16T14:16:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:40.963+0000 7f2b5c190640 1 -- 172.21.15.29:0/1773399387 wait complete. 2023-10-16T14:16:40.986 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-16T14:16:40.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:41.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.207+0000 7f98ed1f4640 1 Processor -- start 2023-10-16T14:16:41.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.207+0000 7f98ed1f4640 1 -- start start 2023-10-16T14:16:41.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.209+0000 7f98ed1f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 0x7f98e80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.209+0000 7f98ed1f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e815b9d0 con 0x7f98e80ccfb0 2023-10-16T14:16:41.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.209+0000 7f98ed1f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e815bcb0 con 0x7f98e806c040 2023-10-16T14:16:41.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.209+0000 7f98ed1f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e815bf90 con 0x7f98e80ad970 2023-10-16T14:16:41.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.209+0000 7f98ec9f3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f98e80ccfb0 0x7f98e80cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46412/0 (socket says 172.21.15.29:46412) 2023-10-16T14:16:41.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.209+0000 7f98ec9f3640 1 -- 172.21.15.29:0/2462594314 learned_addr learned my addr 172.21.15.29:0/2462594314 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:41.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98dffff640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 0x7f98e80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043700996 0 0) 0x7f98e815b9d0 con 0x7f98e80ccfb0 2023-10-16T14:16:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d4003520 con 0x7f98e80ccfb0 2023-10-16T14:16:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500045522 0 0) 0x7f98d4003520 con 0x7f98e80ccfb0 2023-10-16T14:16:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 msgr2=0x7f98e80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 0x7f98e80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 >> v1:172.21.15.170:6789/0 conn(0x7f98e806c040 legacy=0x7f98e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80cc270 con 0x7f98e80ccfb0 2023-10-16T14:16:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f98e00019f0 con 0x7f98e80ccfb0 2023-10-16T14:16:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 >> v1:172.21.15.29:6789/0 conn(0x7f98e80ccfb0 legacy=0x7f98e80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98d40039a0 0x7f98d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.210+0000 7f98e77fe640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d40047c0 0x7f98d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98e77fe640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 0x7f98d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e815bf90 con 0x7f98d40047c0 2023-10-16T14:16:41.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d400ba80 con 0x7f98d40039a0 2023-10-16T14:16:41.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d400bc40 con 0x7f98e80ad970 2023-10-16T14:16:41.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98dffff640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d40047c0 0x7f98d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98e7fff640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98d40039a0 0x7f98d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98ec9f3640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 0x7f98d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98dffff640 1 -- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 msgr2=0x7f98d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98dffff640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98e80ad970 0x7f98d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98dffff640 1 -- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98d40039a0 msgr2=0x7f98d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98dffff640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98d40039a0 0x7f98d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.211+0000 7f98dffff640 1 -- 172.21.15.29:0/2462594314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4003520 con 0x7f98d40047c0 2023-10-16T14:16:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.212+0000 7f98dffff640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d40047c0 0x7f98d4004c40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009ac0 tx=0x7f98d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5387b09d149ac0a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d002c6b0 con 0x7f98d40047c0 2023-10-16T14:16:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d002c810 con 0x7f98d40047c0 2023-10-16T14:16:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98e77fe640 1 -- 172.21.15.29:0/2462594314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d002cba0 con 0x7f98d40047c0 2023-10-16T14:16:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ed1f4640 1 -- 172.21.15.29:0/2462594314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d40047c0 msgr2=0x7f98d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/2462594314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98d40047c0 0x7f98d4004c40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009ac0 tx=0x7f98d40057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ec9f3640 1 -- 172.21.15.29:0/2462594314 reap_dead start 2023-10-16T14:16:41.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ed1f4640 1 -- 172.21.15.29:0/2462594314 shutdown_connections 2023-10-16T14:16:41.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ed1f4640 1 -- 172.21.15.29:0/2462594314 >> 172.21.15.29:0/2462594314 conn(0x7f98e80ce6a0 msgr2=0x7f98e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:41.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ed1f4640 1 -- 172.21.15.29:0/2462594314 shutdown_connections 2023-10-16T14:16:41.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.213+0000 7f98ed1f4640 1 -- 172.21.15.29:0/2462594314 wait complete. 2023-10-16T14:16:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.215+0000 7f98ed1f4640 1 Processor -- start 2023-10-16T14:16:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.215+0000 7f98ed1f4640 1 -- start start 2023-10-16T14:16:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.216+0000 7f98ed1f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98d40047c0 0x7f98e80b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.216+0000 7f98ed1f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 0x7f98e80af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.216+0000 7f98ed1f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98e80af880 0x7f98e80b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.216+0000 7f98ed1f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80aef20 con 0x7f98e815b1c0 2023-10-16T14:16:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.216+0000 7f98ed1f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80b0fb0 con 0x7f98e80af880 2023-10-16T14:16:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.216+0000 7f98ed1f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80ae2a0 con 0x7f98d40047c0 2023-10-16T14:16:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 0x7f98e80af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 0x7f98e80af340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51074/0 (socket says 172.21.15.29:51074) 2023-10-16T14:16:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 -- 172.21.15.29:0/3814767553 learned_addr learned my addr 172.21.15.29:0/3814767553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98e7fff640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98d40047c0 0x7f98e80b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98ec9f3640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98e80af880 0x7f98e80b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 -- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98d40047c0 msgr2=0x7f98e80b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98d40047c0 0x7f98e80b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 -- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98e80af880 msgr2=0x7f98e80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98e80af880 0x7f98e80b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 -- 172.21.15.29:0/3814767553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80cdcd0 con 0x7f98e815b1c0 2023-10-16T14:16:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.217+0000 7f98dffff640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 0x7f98e80af340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009ac0 tx=0x7f98d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.218+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0003810 con 0x7f98e815b1c0 2023-10-16T14:16:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.218+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d0003970 con 0x7f98e815b1c0 2023-10-16T14:16:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.218+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0003d00 con 0x7f98e815b1c0 2023-10-16T14:16:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.218+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e80ae580 con 0x7f98e815b1c0 2023-10-16T14:16:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.218+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e80aeb40 con 0x7f98e815b1c0 2023-10-16T14:16:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.219+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e806b330 con 0x7f98e815b1c0 2023-10-16T14:16:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.220+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f98d0040450 con 0x7f98e815b1c0 2023-10-16T14:16:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.221+0000 7f98e57fa640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f98c806aea0 0x7f98c806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.221+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98d00ab5a0 con 0x7f98e815b1c0 2023-10-16T14:16:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.222+0000 7f98e7fff640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f98c806aea0 0x7f98c806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.223+0000 7f98e7fff640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f98c806aea0 0x7f98c806d360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98d8007fd0 tx=0x7f98d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.225+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f98d0076640 con 0x7f98e815b1c0 2023-10-16T14:16:41.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.420+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98e804e9b0 con 0x7f98e815b1c0 2023-10-16T14:16:41.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.423+0000 7f98e57fa640 1 -- 172.21.15.29:0/3814767553 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f98d00044e0 con 0x7f98e815b1c0 2023-10-16T14:16:41.428 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:41.428 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f98c806aea0 msgr2=0x7f98c806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f98c806aea0 0x7f98c806d360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98d8007fd0 tx=0x7f98d8009040 comp rx=0 tx=0).stop 2023-10-16T14:16:41.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 msgr2=0x7f98e80af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 0x7f98e80af340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009ac0 tx=0x7f98d0004730 comp rx=0 tx=0).stop 2023-10-16T14:16:41.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 shutdown_connections 2023-10-16T14:16:41.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f98c806aea0 0x7f98c806d360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e815b1c0 0x7f98e80af340 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98e80af880 0x7f98e80b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 --2- 172.21.15.29:0/3814767553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98d40047c0 0x7f98e80b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 >> 172.21.15.29:0/3814767553 conn(0x7f98e80ce6a0 msgr2=0x7f98e80ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:41.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 shutdown_connections 2023-10-16T14:16:41.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.426+0000 7f98ed1f4640 1 -- 172.21.15.29:0/3814767553 wait complete. 2023-10-16T14:16:41.450 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:16:41.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:16:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.671+0000 7fa9f497b640 1 Processor -- start 2023-10-16T14:16:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.671+0000 7fa9f497b640 1 -- start start 2023-10-16T14:16:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9f497b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9f00b3ec0 0x7fa9f00cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9f497b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9f00b43c0 0x7fa9f00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9f497b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f006be30 con 0x7fa9f00b0ea0 2023-10-16T14:16:41.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9f497b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f006ca00 con 0x7fa9f00b3ec0 2023-10-16T14:16:41.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9f497b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f00ada10 con 0x7fa9f00b43c0 2023-10-16T14:16:41.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9ee59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa9f00b0ea0 0x7fa9f015a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46424/0 (socket says 172.21.15.29:46424) 2023-10-16T14:16:41.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9ee59c640 1 -- 172.21.15.29:0/2098640640 learned_addr learned my addr 172.21.15.29:0/2098640640 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:41.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9eed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa9f00b3ec0 0x7fa9f00cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9ef59e640 1 --2- 172.21.15.29:0/2098640640 >> v2:172.21.15.170:3300/0 conn(0x7fa9f00b43c0 0x7fa9f00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.673+0000 7fa9edd9b640 1 -- 172.21.15.29:0/2098640640 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485215195 0 0) 0x7fa9f006be30 con 0x7fa9f00b0ea0 2023-10-16T14:16:41.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.674+0000 7fa9edd9b640 1 -- 172.21.15.29:0/2098640640 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d8003520 con 0x7fa9f00b0ea0 2023-10-16T14:16:41.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.674+0000 7fa9eed9d640 1 -- 172.21.15.29:0/2098640640 >> v2:172.21.15.170:3300/0 conn(0x7fa9f00b43c0 msgr2=0x7fa9f00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:41.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.674+0000 7fa9eed9d640 1 --2- 172.21.15.29:0/2098640640 >> v2:172.21.15.170:3300/0 conn(0x7fa9f00b43c0 0x7fa9f00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.674+0000 7fa9eed9d640 1 -- 172.21.15.29:0/2098640640 >> v1:172.21.15.29:6789/0 conn(0x7fa9f00b0ea0 legacy=0x7fa9f015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.674+0000 7fa9eed9d640 1 -- 172.21.15.29:0/2098640640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f006c060 con 0x7fa9f00b3ec0 2023-10-16T14:16:41.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.675+0000 7fa9eed9d640 1 --2- 172.21.15.29:0/2098640640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b3ec0 0x7fa9f00cd260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0071b60 tx=0x7fa9e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d54225f080f638 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:41.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.675+0000 7fa9edd9b640 1 -- 172.21.15.29:0/2098640640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e002e070 con 0x7fa9f00b3ec0 2023-10-16T14:16:41.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.675+0000 7fa9edd9b640 1 -- 172.21.15.29:0/2098640640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e00315a0 con 0x7fa9f00b3ec0 2023-10-16T14:16:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9edd9b640 1 -- 172.21.15.29:0/2098640640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0031930 con 0x7fa9f00b3ec0 2023-10-16T14:16:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 -- 172.21.15.29:0/2098640640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b3ec0 msgr2=0x7fa9f00cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 --2- 172.21.15.29:0/2098640640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b3ec0 0x7fa9f00cd260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0071b60 tx=0x7fa9e002b840 comp rx=0 tx=0).stop 2023-10-16T14:16:41.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 -- 172.21.15.29:0/2098640640 shutdown_connections 2023-10-16T14:16:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 --2- 172.21.15.29:0/2098640640 >> v2:172.21.15.170:3300/0 conn(0x7fa9f00b43c0 0x7fa9f00cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 --2- 172.21.15.29:0/2098640640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b3ec0 0x7fa9f00cd260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 -- 172.21.15.29:0/2098640640 >> 172.21.15.29:0/2098640640 conn(0x7fa9f00b0170 msgr2=0x7fa9f00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 -- 172.21.15.29:0/2098640640 shutdown_connections 2023-10-16T14:16:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.676+0000 7fa9f497b640 1 -- 172.21.15.29:0/2098640640 wait complete. 2023-10-16T14:16:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.678+0000 7fa9f497b640 1 Processor -- start 2023-10-16T14:16:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.678+0000 7fa9f497b640 1 -- start start 2023-10-16T14:16:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9f497b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9f00b0ea0 0x7fa9f015cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9f497b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9f00b3ec0 0x7fa9f00ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9f497b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 0x7fa9f0155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9f497b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f00c6200 con 0x7fa9f00b43c0 2023-10-16T14:16:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9f497b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f00c64e0 con 0x7fa9f00b0ea0 2023-10-16T14:16:41.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9f497b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f00c67c0 con 0x7fa9f00b3ec0 2023-10-16T14:16:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9ef59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 0x7fa9f0155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.679+0000 7fa9ef59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 0x7fa9f0155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51090/0 (socket says 172.21.15.29:51090) 2023-10-16T14:16:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 -- 172.21.15.29:0/3060505228 learned_addr learned my addr 172.21.15.29:0/3060505228 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9eed9d640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9f00b0ea0 0x7fa9f015cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ee59c640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9f00b3ec0 0x7fa9f00ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 -- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9f00b3ec0 msgr2=0x7fa9f00ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9f00b3ec0 0x7fa9f00ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 -- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9f00b0ea0 msgr2=0x7fa9f015cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9f00b0ea0 0x7fa9f015cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 -- 172.21.15.29:0/3060505228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0009400 con 0x7fa9f00b43c0 2023-10-16T14:16:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.680+0000 7fa9ef59e640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 0x7fa9f0155510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8007e40 tx=0x7fa9e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.681+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8014620 con 0x7fa9f00b43c0 2023-10-16T14:16:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.681+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800ee90 con 0x7fa9f00b43c0 2023-10-16T14:16:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.681+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f00c6aa0 con 0x7fa9f00b43c0 2023-10-16T14:16:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.681+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80149e0 con 0x7fa9f00b43c0 2023-10-16T14:16:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.681+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f0155a50 con 0x7fa9f00b43c0 2023-10-16T14:16:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.682+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f0155c10 con 0x7fa9f00b43c0 2023-10-16T14:16:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.683+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa9e8014b40 con 0x7fa9f00b43c0 2023-10-16T14:16:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.684+0000 7fa9d77fe640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa9dc06ae70 0x7fa9dc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:41.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.684+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9e8089fd0 con 0x7fa9f00b43c0 2023-10-16T14:16:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.684+0000 7fa9eed9d640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa9dc06ae70 0x7fa9dc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.685+0000 7fa9eed9d640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa9dc06ae70 0x7fa9dc06d330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0036000 tx=0x7fa9e003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.688+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa9f0155c10 con 0x7fa9f00b43c0 2023-10-16T14:16:41.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.881+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9f00c6fc0 con 0x7fa9f00b43c0 2023-10-16T14:16:41.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.882+0000 7fa9d77fe640 1 -- 172.21.15.29:0/3060505228 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9f00c6fc0 con 0x7fa9f00b43c0 2023-10-16T14:16:41.886 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:16:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa9dc06ae70 msgr2=0x7fa9dc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa9dc06ae70 0x7fa9dc06d330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0036000 tx=0x7fa9e003d040 comp rx=0 tx=0).stop 2023-10-16T14:16:41.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 msgr2=0x7fa9f0155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 0x7fa9f0155510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8007e40 tx=0x7fa9e800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 shutdown_connections 2023-10-16T14:16:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa9dc06ae70 0x7fa9dc06d330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9f00b43c0 0x7fa9f0155510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9f00b3ec0 0x7fa9f00ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 --2- 172.21.15.29:0/3060505228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9f00b0ea0 0x7fa9f015cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 >> 172.21.15.29:0/3060505228 conn(0x7fa9f00b0170 msgr2=0x7fa9f00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 shutdown_connections 2023-10-16T14:16:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:41.885+0000 7fa9f497b640 1 -- 172.21.15.29:0/3060505228 wait complete. 2023-10-16T14:16:41.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:16:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.122+0000 7fd6321f4640 1 Processor -- start 2023-10-16T14:16:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.122+0000 7fd6321f4640 1 -- start start 2023-10-16T14:16:42.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.124+0000 7fd6321f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c15aa60 0x7fd62c0af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.124+0000 7fd6321f4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd62c0afd60 0x7fd62c0cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.124+0000 7fd6321f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c0b3730 con 0x7fd62c0cc820 2023-10-16T14:16:42.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.124+0000 7fd6321f4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62c0b3980 con 0x7fd62c0afd60 2023-10-16T14:16:42.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.124+0000 7fd6321f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c0b3c60 con 0x7fd62c15aa60 2023-10-16T14:16:42.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.124+0000 7fd6309f1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd62c0afd60 0x7fd62c0cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6309f1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd62c0afd60 0x7fd62c0cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51098/0 (socket says 172.21.15.29:51098) 2023-10-16T14:16:42.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6309f1640 1 -- 172.21.15.29:0/104066223 learned_addr learned my addr 172.21.15.29:0/104066223 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6311f2640 1 --2- 172.21.15.29:0/104066223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c15aa60 0x7fd62c0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6309f1640 1 -- 172.21.15.29:0/104066223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c15aa60 msgr2=0x7fd62c0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6309f1640 1 --2- 172.21.15.29:0/104066223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c15aa60 0x7fd62c0af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6309f1640 1 -- 172.21.15.29:0/104066223 >> v1:172.21.15.170:6789/0 conn(0x7fd62c0cc820 legacy=0x7fd62c0b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:42.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.125+0000 7fd6309f1640 1 -- 172.21.15.29:0/104066223 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c0b3f40 con 0x7fd62c0afd60 2023-10-16T14:16:42.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.126+0000 7fd6309f1640 1 --2- 172.21.15.29:0/104066223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c0afd60 0x7fd62c0cc2e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009870 tx=0x7fd61c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29afe5ee8f823080 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.126+0000 7fd623fff640 1 -- 172.21.15.29:0/104066223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c031640 con 0x7fd62c0afd60 2023-10-16T14:16:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.126+0000 7fd623fff640 1 -- 172.21.15.29:0/104066223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61c0317a0 con 0x7fd62c0afd60 2023-10-16T14:16:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.126+0000 7fd623fff640 1 -- 172.21.15.29:0/104066223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c031b30 con 0x7fd62c0afd60 2023-10-16T14:16:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 -- 172.21.15.29:0/104066223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c0afd60 msgr2=0x7fd62c0cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 --2- 172.21.15.29:0/104066223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c0afd60 0x7fd62c0cc2e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009870 tx=0x7fd61c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 -- 172.21.15.29:0/104066223 shutdown_connections 2023-10-16T14:16:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 --2- 172.21.15.29:0/104066223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c0afd60 0x7fd62c0cc2e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 --2- 172.21.15.29:0/104066223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c15aa60 0x7fd62c0af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 -- 172.21.15.29:0/104066223 >> 172.21.15.29:0/104066223 conn(0x7fd62c065ed0 msgr2=0x7fd62c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:42.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 -- 172.21.15.29:0/104066223 shutdown_connections 2023-10-16T14:16:42.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.127+0000 7fd6321f4640 1 -- 172.21.15.29:0/104066223 wait complete. 2023-10-16T14:16:42.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.129+0000 7fd6321f4640 1 Processor -- start 2023-10-16T14:16:42.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.129+0000 7fd6321f4640 1 -- start start 2023-10-16T14:16:42.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.130+0000 7fd6321f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c0afd60 0x7fd62c15d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.130+0000 7fd6321f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd62c0cc820 0x7fd62c06b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.130+0000 7fd6321f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 0x7fd62c14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.130+0000 7fd6321f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c1572b0 con 0x7fd62c15aa60 2023-10-16T14:16:42.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.130+0000 7fd6321f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c1502b0 con 0x7fd62c0cc820 2023-10-16T14:16:42.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.130+0000 7fd6321f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c06b7e0 con 0x7fd62c0afd60 2023-10-16T14:16:42.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 0x7fd62c14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 0x7fd62c14fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51100/0 (socket says 172.21.15.29:51100) 2023-10-16T14:16:42.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 -- 172.21.15.29:0/293487990 learned_addr learned my addr 172.21.15.29:0/293487990 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:42.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6311f2640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c0afd60 0x7fd62c15d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6309f1640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd62c0cc820 0x7fd62c06b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 -- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c0afd60 msgr2=0x7fd62c15d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c0afd60 0x7fd62c15d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 -- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd62c0cc820 msgr2=0x7fd62c06b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd62c0cc820 0x7fd62c06b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 -- 172.21.15.29:0/293487990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c0094d0 con 0x7fd62c15aa60 2023-10-16T14:16:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.131+0000 7fd6319f3640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 0x7fd62c14fd70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd62400c780 tx=0x7fd62400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.132+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624004b60 con 0x7fd62c15aa60 2023-10-16T14:16:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.132+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62400ce90 con 0x7fd62c15aa60 2023-10-16T14:16:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.132+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62400a850 con 0x7fd62c15aa60 2023-10-16T14:16:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.132+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c06bac0 con 0x7fd62c15aa60 2023-10-16T14:16:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.132+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c06c080 con 0x7fd62c15aa60 2023-10-16T14:16:42.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.133+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c04e9b0 con 0x7fd62c15aa60 2023-10-16T14:16:42.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.134+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd62400a9b0 con 0x7fd62c15aa60 2023-10-16T14:16:42.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.135+0000 7fd621ffb640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61006aec0 0x7fd61006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.135+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6240896d0 con 0x7fd62c15aa60 2023-10-16T14:16:42.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.135+0000 7fd6311f2640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61006aec0 0x7fd61006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.137+0000 7fd6311f2640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61006aec0 0x7fd61006d380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd628006fd0 tx=0x7fd628008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.139+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd624054860 con 0x7fd62c15aa60 2023-10-16T14:16:42.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.333+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd62c0cbba0 con 0x7fd62c15aa60 2023-10-16T14:16:42.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.334+0000 7fd621ffb640 1 -- 172.21.15.29:0/293487990 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd624057e80 con 0x7fd62c15aa60 2023-10-16T14:16:42.338 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:16:42.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61006aec0 msgr2=0x7fd61006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61006aec0 0x7fd61006d380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd628006fd0 tx=0x7fd628008040 comp rx=0 tx=0).stop 2023-10-16T14:16:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 msgr2=0x7fd62c14fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 0x7fd62c14fd70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd62400c780 tx=0x7fd62400cc40 comp rx=0 tx=0).stop 2023-10-16T14:16:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 shutdown_connections 2023-10-16T14:16:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61006aec0 0x7fd61006d380 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd62c15aa60 0x7fd62c14fd70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd62c0cc820 0x7fd62c06b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 --2- 172.21.15.29:0/293487990 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd62c0afd60 0x7fd62c15d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 >> 172.21.15.29:0/293487990 conn(0x7fd62c065ed0 msgr2=0x7fd62c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 shutdown_connections 2023-10-16T14:16:42.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.336+0000 7fd6321f4640 1 -- 172.21.15.29:0/293487990 wait complete. 2023-10-16T14:16:42.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:16:42.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:16:42.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:16:42.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.590+0000 7fb631f03640 1 Processor -- start 2023-10-16T14:16:42.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.590+0000 7fb631f03640 1 -- start start 2023-10-16T14:16:42.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.591+0000 7fb631f03640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb62c15cd70 0x7fb62c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb631f03640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb62c0b8480 0x7fb62c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb631f03640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c0d0be0 0x7fb62c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb631f03640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c15d150 con 0x7fb62c15cd70 2023-10-16T14:16:42.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb631f03640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c0d1180 con 0x7fb62c0b8480 2023-10-16T14:16:42.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb631f03640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0d1460 con 0x7fb62c0d0be0 2023-10-16T14:16:42.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb630f01640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb62c15cd70 0x7fb62c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb630f01640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb62c15cd70 0x7fb62c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51102/0 (socket says 172.21.15.29:51102) 2023-10-16T14:16:42.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb630f01640 1 -- 172.21.15.29:0/2145339463 learned_addr learned my addr 172.21.15.29:0/2145339463 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:42.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb631702640 1 --2- 172.21.15.29:0/2145339463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c0d0be0 0x7fb62c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb623fff640 1 --2- 172.21.15.29:0/2145339463 >> v2:172.21.15.170:3300/0 conn(0x7fb62c0b8480 0x7fb62c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.592+0000 7fb630f01640 1 -- 172.21.15.29:0/2145339463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c0d0be0 msgr2=0x7fb62c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:42.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.593+0000 7fb630f01640 1 --2- 172.21.15.29:0/2145339463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c0d0be0 0x7fb62c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.593+0000 7fb630f01640 1 -- 172.21.15.29:0/2145339463 >> v2:172.21.15.170:3300/0 conn(0x7fb62c0b8480 msgr2=0x7fb62c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:42.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.593+0000 7fb630f01640 1 --2- 172.21.15.29:0/2145339463 >> v2:172.21.15.170:3300/0 conn(0x7fb62c0b8480 0x7fb62c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.593+0000 7fb630f01640 1 -- 172.21.15.29:0/2145339463 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0d1740 con 0x7fb62c15cd70 2023-10-16T14:16:42.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.594+0000 7fb630f01640 1 --2- 172.21.15.29:0/2145339463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c15cd70 0x7fb62c0d2220 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb62c066a00 tx=0x7fb62802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6bdb7f3e847ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.594+0000 7fb6237fe640 1 -- 172.21.15.29:0/2145339463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628031870 con 0x7fb62c15cd70 2023-10-16T14:16:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.594+0000 7fb6237fe640 1 -- 172.21.15.29:0/2145339463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6280319d0 con 0x7fb62c15cd70 2023-10-16T14:16:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.594+0000 7fb6237fe640 1 -- 172.21.15.29:0/2145339463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628031d60 con 0x7fb62c15cd70 2023-10-16T14:16:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 -- 172.21.15.29:0/2145339463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c15cd70 msgr2=0x7fb62c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 --2- 172.21.15.29:0/2145339463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c15cd70 0x7fb62c0d2220 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb62c066a00 tx=0x7fb62802b910 comp rx=0 tx=0).stop 2023-10-16T14:16:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 -- 172.21.15.29:0/2145339463 shutdown_connections 2023-10-16T14:16:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 --2- 172.21.15.29:0/2145339463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c0d0be0 0x7fb62c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 --2- 172.21.15.29:0/2145339463 >> v2:172.21.15.170:3300/0 conn(0x7fb62c0b8480 0x7fb62c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 --2- 172.21.15.29:0/2145339463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c15cd70 0x7fb62c0d2220 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 -- 172.21.15.29:0/2145339463 >> 172.21.15.29:0/2145339463 conn(0x7fb62c0cf570 msgr2=0x7fb62c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 -- 172.21.15.29:0/2145339463 shutdown_connections 2023-10-16T14:16:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.595+0000 7fb631f03640 1 -- 172.21.15.29:0/2145339463 wait complete. 2023-10-16T14:16:42.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.597+0000 7fb631f03640 1 Processor -- start 2023-10-16T14:16:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.598+0000 7fb631f03640 1 -- start start 2023-10-16T14:16:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631f03640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 0x7fb62c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631f03640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb62c0d0be0 0x7fb62c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631f03640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c15cd70 0x7fb62c1f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631f03640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0ca640 con 0x7fb62c0b8480 2023-10-16T14:16:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631f03640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c152440 con 0x7fb62c0d0be0 2023-10-16T14:16:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631f03640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0ca8d0 con 0x7fb62c15cd70 2023-10-16T14:16:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb630f01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 0x7fb62c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb630f01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 0x7fb62c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51112/0 (socket says 172.21.15.29:51112) 2023-10-16T14:16:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb630f01640 1 -- 172.21.15.29:0/3964632825 learned_addr learned my addr 172.21.15.29:0/3964632825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb623fff640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb62c0d0be0 0x7fb62c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.599+0000 7fb631702640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c15cd70 0x7fb62c1f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb630f01640 1 -- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c15cd70 msgr2=0x7fb62c1f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb630f01640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c15cd70 0x7fb62c1f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb630f01640 1 -- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb62c0d0be0 msgr2=0x7fb62c0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb630f01640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb62c0d0be0 0x7fb62c0ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb630f01640 1 -- 172.21.15.29:0/3964632825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6280094d0 con 0x7fb62c0b8480 2023-10-16T14:16:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb631702640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c15cd70 0x7fb62c1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb623fff640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb62c0d0be0 0x7fb62c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb630f01640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 0x7fb62c0c9bc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb62802b950 tx=0x7fb6280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62803ad30 con 0x7fb62c0b8480 2023-10-16T14:16:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6280443f0 con 0x7fb62c0b8480 2023-10-16T14:16:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6280447a0 con 0x7fb62c0b8480 2023-10-16T14:16:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c0cabb0 con 0x7fb62c0b8480 2023-10-16T14:16:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.600+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c0cb170 con 0x7fb62c0b8480 2023-10-16T14:16:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.602+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c0cb360 con 0x7fb62c0b8480 2023-10-16T14:16:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.603+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb62804d450 con 0x7fb62c0b8480 2023-10-16T14:16:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.605+0000 7fb6217fa640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb61806ae70 0x7fb61806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.605+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb62802d040 con 0x7fb62c0b8480 2023-10-16T14:16:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.605+0000 7fb623fff640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb61806ae70 0x7fb61806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:42.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.606+0000 7fb623fff640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb61806ae70 0x7fb61806d330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0cdf50 tx=0x7fb61c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:42.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.608+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb628076c30 con 0x7fb62c0b8480 2023-10-16T14:16:42.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.800+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb62c04e9b0 con 0x7fb62c0b8480 2023-10-16T14:16:42.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.801+0000 7fb6217fa640 1 -- 172.21.15.29:0/3964632825 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb628035070 con 0x7fb62c0b8480 2023-10-16T14:16:42.806 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:16:42.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb61806ae70 msgr2=0x7fb61806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb61806ae70 0x7fb61806d330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0cdf50 tx=0x7fb61c008040 comp rx=0 tx=0).stop 2023-10-16T14:16:42.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 msgr2=0x7fb62c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:42.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 0x7fb62c0c9bc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb62802b950 tx=0x7fb6280046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 shutdown_connections 2023-10-16T14:16:42.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb61806ae70 0x7fb61806d330 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb62c15cd70 0x7fb62c1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb62c0d0be0 0x7fb62c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 --2- 172.21.15.29:0/3964632825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0b8480 0x7fb62c0c9bc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:42.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 >> 172.21.15.29:0/3964632825 conn(0x7fb62c0cf570 msgr2=0x7fb62c0d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:42.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 shutdown_connections 2023-10-16T14:16:42.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:42.804+0000 7fb631f03640 1 -- 172.21.15.29:0/3964632825 wait complete. 2023-10-16T14:16:42.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:16:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.040+0000 7f48565ce640 1 Processor -- start 2023-10-16T14:16:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.040+0000 7f48565ce640 1 -- start start 2023-10-16T14:16:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f48565ce640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4850159f00 0x7f48500b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f48565ce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f485015d880 0x7f48500b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f48565ce640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485006f7f0 con 0x7f4850159b20 2023-10-16T14:16:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f48565ce640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485006c5e0 con 0x7f485015d880 2023-10-16T14:16:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f48565ce640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485006c8c0 con 0x7f4850159f00 2023-10-16T14:16:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f484ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4850159b20 0x7f485006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46434/0 (socket says 172.21.15.29:46434) 2023-10-16T14:16:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f484ffff640 1 -- 172.21.15.29:0/1027673240 learned_addr learned my addr 172.21.15.29:0/1027673240 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.042+0000 7f4854b6b640 1 --2- 172.21.15.29:0/1027673240 >> v2:172.21.15.29:3300/0 conn(0x7f485015d880 0x7f48500b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.043+0000 7f4847fff640 1 --2- 172.21.15.29:0/1027673240 >> v2:172.21.15.170:3300/0 conn(0x7f4850159f00 0x7f48500b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.043+0000 7f484f7fe640 1 -- 172.21.15.29:0/1027673240 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512751816 0 0) 0x7f485006f7f0 con 0x7f4850159b20 2023-10-16T14:16:43.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.043+0000 7f4854b6b640 1 -- 172.21.15.29:0/1027673240 >> v2:172.21.15.170:3300/0 conn(0x7f4850159f00 msgr2=0x7f48500b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:43.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.043+0000 7f4854b6b640 1 --2- 172.21.15.29:0/1027673240 >> v2:172.21.15.170:3300/0 conn(0x7f4850159f00 0x7f48500b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.043+0000 7f4854b6b640 1 -- 172.21.15.29:0/1027673240 >> v1:172.21.15.29:6789/0 conn(0x7f4850159b20 legacy=0x7f485006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.043+0000 7f4854b6b640 1 -- 172.21.15.29:0/1027673240 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485006cba0 con 0x7f485015d880 2023-10-16T14:16:43.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.045+0000 7f4854b6b640 1 --2- 172.21.15.29:0/1027673240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f48500b0950 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f485015d770 tx=0x7f484802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86571bdd687238a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.045+0000 7f484f7fe640 1 -- 172.21.15.29:0/1027673240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484802cc80 con 0x7f485015d880 2023-10-16T14:16:43.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.045+0000 7f484f7fe640 1 -- 172.21.15.29:0/1027673240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484802cde0 con 0x7f485015d880 2023-10-16T14:16:43.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.045+0000 7f484f7fe640 1 -- 172.21.15.29:0/1027673240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848034660 con 0x7f485015d880 2023-10-16T14:16:43.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.045+0000 7f48565ce640 1 -- 172.21.15.29:0/1027673240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 msgr2=0x7f48500b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.045+0000 7f48565ce640 1 --2- 172.21.15.29:0/1027673240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f48500b0950 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f485015d770 tx=0x7f484802bed0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.046+0000 7f48565ce640 1 -- 172.21.15.29:0/1027673240 shutdown_connections 2023-10-16T14:16:43.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.046+0000 7f48565ce640 1 --2- 172.21.15.29:0/1027673240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f48500b0950 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.046+0000 7f48565ce640 1 --2- 172.21.15.29:0/1027673240 >> v2:172.21.15.170:3300/0 conn(0x7f4850159f00 0x7f48500b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.046+0000 7f48565ce640 1 -- 172.21.15.29:0/1027673240 >> 172.21.15.29:0/1027673240 conn(0x7f48500cb410 msgr2=0x7f48500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.046+0000 7f48565ce640 1 -- 172.21.15.29:0/1027673240 shutdown_connections 2023-10-16T14:16:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.046+0000 7f48565ce640 1 -- 172.21.15.29:0/1027673240 wait complete. 2023-10-16T14:16:43.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.047+0000 7f48565ce640 1 Processor -- start 2023-10-16T14:16:43.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.048+0000 7f48565ce640 1 -- start start 2023-10-16T14:16:43.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f48565ce640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4850159b20 0x7f48501f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f48565ce640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4850159f00 0x7f48500c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f48565ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f4850157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f48565ce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48501f38c0 con 0x7f485015d880 2023-10-16T14:16:43.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f48565ce640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485015b360 con 0x7f4850159b20 2023-10-16T14:16:43.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f48565ce640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500c4ff0 con 0x7f4850159f00 2023-10-16T14:16:43.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f4850157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f4850157e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51128/0 (socket says 172.21.15.29:51128) 2023-10-16T14:16:43.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 -- 172.21.15.29:0/1763010660 learned_addr learned my addr 172.21.15.29:0/1763010660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:43.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f484ffff640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4850159b20 0x7f48501f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4847fff640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4850159f00 0x7f48500c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 -- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4850159f00 msgr2=0x7f48500c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4850159f00 0x7f48500c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 -- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4850159b20 msgr2=0x7f48501f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4850159b20 0x7f48501f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4854b6b640 1 -- 172.21.15.29:0/1763010660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848009a70 con 0x7f485015d880 2023-10-16T14:16:43.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.049+0000 7f4847fff640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4850159f00 0x7f48500c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:43.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.050+0000 7f4854b6b640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f4850157e10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4848037040 tx=0x7f4848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.050+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848040650 con 0x7f485015d880 2023-10-16T14:16:43.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.050+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48480407b0 con 0x7f485015d880 2023-10-16T14:16:43.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.050+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848040b60 con 0x7f485015d880 2023-10-16T14:16:43.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.050+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48500c52d0 con 0x7f485015d880 2023-10-16T14:16:43.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.050+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485015b890 con 0x7f485015d880 2023-10-16T14:16:43.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.052+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485004e160 con 0x7f485015d880 2023-10-16T14:16:43.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.052+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4848040cc0 con 0x7f485015d880 2023-10-16T14:16:43.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.053+0000 7f484d7fa640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f483006ae70 0x7f483006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.053+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48480ac2f0 con 0x7f485015d880 2023-10-16T14:16:43.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.054+0000 7f484ffff640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f483006ae70 0x7f483006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.055+0000 7f484ffff640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f483006ae70 0x7f483006d330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4840000f60 tx=0x7f4840009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.058+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4848077390 con 0x7f485015d880 2023-10-16T14:16:43.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.280+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48500b10b0 con 0x7f485015d880 2023-10-16T14:16:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.281+0000 7f484d7fa640 1 -- 172.21.15.29:0/1763010660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f4848009450 con 0x7f485015d880 2023-10-16T14:16:43.286 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:43.286 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:16:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:16:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f483006ae70 msgr2=0x7f483006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f483006ae70 0x7f483006d330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4840000f60 tx=0x7f4840009040 comp rx=0 tx=0).stop 2023-10-16T14:16:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 msgr2=0x7f4850157e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f4850157e10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4848037040 tx=0x7f4848004a40 comp rx=0 tx=0).stop 2023-10-16T14:16:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 shutdown_connections 2023-10-16T14:16:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f483006ae70 0x7f483006d330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f485015d880 0x7f4850157e10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4850159f00 0x7f48500c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 --2- 172.21.15.29:0/1763010660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4850159b20 0x7f48501f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 >> 172.21.15.29:0/1763010660 conn(0x7f48500cb410 msgr2=0x7f48500ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:43.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.284+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 shutdown_connections 2023-10-16T14:16:43.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.285+0000 7f48565ce640 1 -- 172.21.15.29:0/1763010660 wait complete. 2023-10-16T14:16:43.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:43.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.511+0000 7f4589b73640 1 Processor -- start 2023-10-16T14:16:43.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.512+0000 7f4589b73640 1 -- start start 2023-10-16T14:16:43.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f4589b73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f458415a970 0x7f458406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f4589b73640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 0x7f4584070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f4589b73640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4584070660 con 0x7f4584071360 2023-10-16T14:16:43.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f4589b73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458415dc60 con 0x7f458415a970 2023-10-16T14:16:43.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f4589b73640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458415df40 con 0x7f458415d430 2023-10-16T14:16:43.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f458415a970 0x7f458406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f458415a970 0x7f458406ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51144/0 (socket says 172.21.15.29:51144) 2023-10-16T14:16:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 -- 172.21.15.29:0/4120750315 learned_addr learned my addr 172.21.15.29:0/4120750315 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 -- 172.21.15.29:0/4120750315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 msgr2=0x7f4584070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 --2- 172.21.15.29:0/4120750315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 0x7f4584070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 -- 172.21.15.29:0/4120750315 >> v1:172.21.15.170:6789/0 conn(0x7f4584071360 legacy=0x7f45840cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:43.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.513+0000 7f45837fe640 1 -- 172.21.15.29:0/4120750315 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458415e220 con 0x7f458415a970 2023-10-16T14:16:43.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.515+0000 7f45837fe640 1 --2- 172.21.15.29:0/4120750315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458406ecd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4574009870 tx=0x7f457402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d06b2a7715fe53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.515+0000 7f4582ffd640 1 -- 172.21.15.29:0/4120750315 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574031640 con 0x7f458415a970 2023-10-16T14:16:43.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.515+0000 7f4582ffd640 1 -- 172.21.15.29:0/4120750315 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45740317a0 con 0x7f458415a970 2023-10-16T14:16:43.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.515+0000 7f4582ffd640 1 -- 172.21.15.29:0/4120750315 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574031b30 con 0x7f458415a970 2023-10-16T14:16:43.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.515+0000 7f4589b73640 1 -- 172.21.15.29:0/4120750315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 msgr2=0x7f458406ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.515+0000 7f4589b73640 1 --2- 172.21.15.29:0/4120750315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458406ecd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4574009870 tx=0x7f457402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.516+0000 7f4589b73640 1 -- 172.21.15.29:0/4120750315 shutdown_connections 2023-10-16T14:16:43.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.516+0000 7f4589b73640 1 --2- 172.21.15.29:0/4120750315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 0x7f4584070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.516+0000 7f4589b73640 1 --2- 172.21.15.29:0/4120750315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458406ecd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.516+0000 7f4589b73640 1 -- 172.21.15.29:0/4120750315 >> 172.21.15.29:0/4120750315 conn(0x7f45840cfaa0 msgr2=0x7f45840cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.516+0000 7f4589b73640 1 -- 172.21.15.29:0/4120750315 shutdown_connections 2023-10-16T14:16:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.516+0000 7f4589b73640 1 -- 172.21.15.29:0/4120750315 wait complete. 2023-10-16T14:16:43.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.517+0000 7f4589b73640 1 Processor -- start 2023-10-16T14:16:43.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.518+0000 7f4589b73640 1 -- start start 2023-10-16T14:16:43.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f4589b73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458414cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f4589b73640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 0x7f45840b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f4589b73640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45840c65a0 0x7f45840b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f4589b73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840b1270 con 0x7f458415a970 2023-10-16T14:16:43.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f4589b73640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458414d450 con 0x7f45840c65a0 2023-10-16T14:16:43.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f4589b73640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840b14c0 con 0x7f458415d430 2023-10-16T14:16:43.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458414cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458414cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51148/0 (socket says 172.21.15.29:51148) 2023-10-16T14:16:43.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 -- 172.21.15.29:0/3067548059 learned_addr learned my addr 172.21.15.29:0/3067548059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 -- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 msgr2=0x7f45840b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 0x7f45840b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 -- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45840c65a0 msgr2=0x7f45840b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45840c65a0 0x7f45840b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.519+0000 7f45837fe640 1 -- 172.21.15.29:0/3067548059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740094d0 con 0x7f458415a970 2023-10-16T14:16:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.520+0000 7f45837fe640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458414cf10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4574009870 tx=0x7f45740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.520+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457403ac60 con 0x7f458415a970 2023-10-16T14:16:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.520+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457403adc0 con 0x7f458415a970 2023-10-16T14:16:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.520+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574044630 con 0x7f458415a970 2023-10-16T14:16:43.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.520+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45840b17a0 con 0x7f458415a970 2023-10-16T14:16:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.520+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45840b1d60 con 0x7f458415a970 2023-10-16T14:16:43.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.521+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458415d870 con 0x7f458415a970 2023-10-16T14:16:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.522+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f457402f050 con 0x7f458415a970 2023-10-16T14:16:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.523+0000 7f4580ff9640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f456406ae70 0x7f456406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.523+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45740ab590 con 0x7f458415a970 2023-10-16T14:16:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.524+0000 7f457bfff640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f456406ae70 0x7f456406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.525+0000 7f457bfff640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f456406ae70 0x7f456406d330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f458406e750 tx=0x7f456c006bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.528+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f458415d870 con 0x7f458415a970 2023-10-16T14:16:43.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.721+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f458415d870 con 0x7f458415a970 2023-10-16T14:16:43.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.722+0000 7f4580ff9640 1 -- 172.21.15.29:0/3067548059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f458415d870 con 0x7f458415a970 2023-10-16T14:16:43.727 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:43.728 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f456406ae70 msgr2=0x7f456406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f456406ae70 0x7f456406d330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f458406e750 tx=0x7f456c006bf0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 msgr2=0x7f458414cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458414cf10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4574009870 tx=0x7f45740046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 shutdown_connections 2023-10-16T14:16:43.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f456406ae70 0x7f456406d330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f45840c65a0 0x7f45840b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f458415d430 0x7f45840b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 --2- 172.21.15.29:0/3067548059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f458415a970 0x7f458414cf10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 >> 172.21.15.29:0/3067548059 conn(0x7f45840cfaa0 msgr2=0x7f458415baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:43.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 shutdown_connections 2023-10-16T14:16:43.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.725+0000 7f4589b73640 1 -- 172.21.15.29:0/3067548059 wait complete. 2023-10-16T14:16:43.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:16:43.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:16:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.970+0000 7f3248e60640 1 Processor -- start 2023-10-16T14:16:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.970+0000 7f3248e60640 1 -- start start 2023-10-16T14:16:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.971+0000 7f3248e60640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32440b2080 0x7f32440d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.971+0000 7f3248e60640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f32440cfc40 0x7f32440b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.971+0000 7f3248e60640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32440d0970 con 0x7f32440b1ca0 2023-10-16T14:16:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.971+0000 7f3248e60640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32440d0ea0 con 0x7f32440b2080 2023-10-16T14:16:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.971+0000 7f3248e60640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32440d0bc0 con 0x7f32440cfc40 2023-10-16T14:16:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3242ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32440b2080 0x7f32440d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f32437fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32440b1ca0 0x7f32440ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38950/0 (socket says 172.21.15.29:38950) 2023-10-16T14:16:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3242ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32440b2080 0x7f32440d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46086/0 (socket says 172.21.15.29:46086) 2023-10-16T14:16:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f32437fe640 1 -- 172.21.15.29:0/3506521352 learned_addr learned my addr 172.21.15.29:0/3506521352 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3243fff640 1 --2- 172.21.15.29:0/3506521352 >> v2:172.21.15.170:3300/0 conn(0x7f32440cfc40 0x7f32440b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3242ffd640 1 -- 172.21.15.29:0/3506521352 >> v2:172.21.15.170:3300/0 conn(0x7f32440cfc40 msgr2=0x7f32440b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3242ffd640 1 --2- 172.21.15.29:0/3506521352 >> v2:172.21.15.170:3300/0 conn(0x7f32440cfc40 0x7f32440b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3242ffd640 1 -- 172.21.15.29:0/3506521352 >> v1:172.21.15.29:6789/0 conn(0x7f32440b1ca0 legacy=0x7f32440ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.972+0000 7f3242ffd640 1 -- 172.21.15.29:0/3506521352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32440cd940 con 0x7f32440b2080 2023-10-16T14:16:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.974+0000 7f3242ffd640 1 --2- 172.21.15.29:0/3506521352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440b2080 0x7f32440d36b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f32440d1d30 tx=0x7f322c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716f2206a88ae5fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.974+0000 7f32427fc640 1 -- 172.21.15.29:0/3506521352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c030610 con 0x7f32440b2080 2023-10-16T14:16:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.974+0000 7f32427fc640 1 -- 172.21.15.29:0/3506521352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c030770 con 0x7f32440b2080 2023-10-16T14:16:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.974+0000 7f32427fc640 1 -- 172.21.15.29:0/3506521352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c030b00 con 0x7f32440b2080 2023-10-16T14:16:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.974+0000 7f3248e60640 1 -- 172.21.15.29:0/3506521352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440b2080 msgr2=0x7f32440d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.974+0000 7f3248e60640 1 --2- 172.21.15.29:0/3506521352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440b2080 0x7f32440d36b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f32440d1d30 tx=0x7f322c02b960 comp rx=0 tx=0).stop 2023-10-16T14:16:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.975+0000 7f3248e60640 1 -- 172.21.15.29:0/3506521352 shutdown_connections 2023-10-16T14:16:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.975+0000 7f3248e60640 1 --2- 172.21.15.29:0/3506521352 >> v2:172.21.15.170:3300/0 conn(0x7f32440cfc40 0x7f32440b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.975+0000 7f3248e60640 1 --2- 172.21.15.29:0/3506521352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440b2080 0x7f32440d36b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.975+0000 7f3248e60640 1 -- 172.21.15.29:0/3506521352 >> 172.21.15.29:0/3506521352 conn(0x7f32440d2b90 msgr2=0x7f32440d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.975+0000 7f3248e60640 1 -- 172.21.15.29:0/3506521352 shutdown_connections 2023-10-16T14:16:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.975+0000 7f3248e60640 1 -- 172.21.15.29:0/3506521352 wait complete. 2023-10-16T14:16:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.976+0000 7f3248e60640 1 Processor -- start 2023-10-16T14:16:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 -- start start 2023-10-16T14:16:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32440b1ca0 0x7f324415b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32440b2080 0x7f32440b8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 0x7f32440c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324409c8d0 con 0x7f32440cfc40 2023-10-16T14:16:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324409eab0 con 0x7f32440b1ca0 2023-10-16T14:16:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.977+0000 7f3248e60640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324409ed90 con 0x7f32440b2080 2023-10-16T14:16:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 0x7f32440c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 0x7f32440c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46092/0 (socket says 172.21.15.29:46092) 2023-10-16T14:16:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 -- 172.21.15.29:0/4068321141 learned_addr learned my addr 172.21.15.29:0/4068321141 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f32437fe640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32440b1ca0 0x7f324415b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3242ffd640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32440b2080 0x7f32440b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 -- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32440b2080 msgr2=0x7f32440b8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32440b2080 0x7f32440b8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 -- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32440b1ca0 msgr2=0x7f324415b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32440b1ca0 0x7f324415b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 -- 172.21.15.29:0/4068321141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c009400 con 0x7f32440cfc40 2023-10-16T14:16:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.978+0000 7f3243fff640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 0x7f32440c8850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f323c00d5d0 tx=0x7f323c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.979+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c0125f0 con 0x7f32440cfc40 2023-10-16T14:16:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.979+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c012770 con 0x7f32440cfc40 2023-10-16T14:16:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.979+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c012b20 con 0x7f32440cfc40 2023-10-16T14:16:43.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.979+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324409f070 con 0x7f32440cfc40 2023-10-16T14:16:43.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.979+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324409e6f0 con 0x7f32440cfc40 2023-10-16T14:16:43.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.980+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324409cd60 con 0x7f32440cfc40 2023-10-16T14:16:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.981+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f323c012c80 con 0x7f32440cfc40 2023-10-16T14:16:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.982+0000 7f323bfff640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f322406ae70 0x7f322406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:43.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.982+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f323c089b80 con 0x7f32440cfc40 2023-10-16T14:16:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.983+0000 7f32437fe640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f322406ae70 0x7f322406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.984+0000 7f32437fe640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f322406ae70 0x7f322406d330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3234005c00 tx=0x7f3234009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:43.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:43.986+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f324409cd60 con 0x7f32440cfc40 2023-10-16T14:16:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.178+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f324409cd60 con 0x7f32440cfc40 2023-10-16T14:16:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.179+0000 7f323bfff640 1 -- 172.21.15.29:0/4068321141 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f324409cd60 con 0x7f32440cfc40 2023-10-16T14:16:44.184 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:16:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f322406ae70 msgr2=0x7f322406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f322406ae70 0x7f322406d330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3234005c00 tx=0x7f3234009040 comp rx=0 tx=0).stop 2023-10-16T14:16:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 msgr2=0x7f32440c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 0x7f32440c8850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f323c00d5d0 tx=0x7f323c00da90 comp rx=0 tx=0).stop 2023-10-16T14:16:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 shutdown_connections 2023-10-16T14:16:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f322406ae70 0x7f322406d330 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32440cfc40 0x7f32440c8850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32440b2080 0x7f32440b8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 --2- 172.21.15.29:0/4068321141 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32440b1ca0 0x7f324415b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 >> 172.21.15.29:0/4068321141 conn(0x7f32440d2b90 msgr2=0x7f3244160080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 shutdown_connections 2023-10-16T14:16:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.182+0000 7f3248e60640 1 -- 172.21.15.29:0/4068321141 wait complete. 2023-10-16T14:16:44.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:16:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.434+0000 7fe67db6e640 1 Processor -- start 2023-10-16T14:16:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.434+0000 7fe67db6e640 1 -- start start 2023-10-16T14:16:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe67db6e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6780b8480 0x7fe6780cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe67db6e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe67db6e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67815d150 con 0x7fe67815cd70 2023-10-16T14:16:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe67db6e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6780d1180 con 0x7fe6780b8480 2023-10-16T14:16:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe67db6e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780d1460 con 0x7fe6780d0be0 2023-10-16T14:16:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe677fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6780b8480 0x7fe6780cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe677fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6780b8480 0x7fe6780cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46100/0 (socket says 172.21.15.29:46100) 2023-10-16T14:16:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.435+0000 7fe677fff640 1 -- 172.21.15.29:0/2489945052 learned_addr learned my addr 172.21.15.29:0/2489945052 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.436+0000 7fe67d36d640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.436+0000 7fe677fff640 1 -- 172.21.15.29:0/2489945052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 msgr2=0x7fe6780cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.436+0000 7fe677fff640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.436+0000 7fe677fff640 1 -- 172.21.15.29:0/2489945052 >> v1:172.21.15.170:6789/0 conn(0x7fe67815cd70 legacy=0x7fe6780d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.436+0000 7fe677fff640 1 -- 172.21.15.29:0/2489945052 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780d1740 con 0x7fe6780b8480 2023-10-16T14:16:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.436+0000 7fe67d36d640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.437+0000 7fe677fff640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe6780cd7c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe66c009870 tx=0x7fe66c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac41c0097080c2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.437+0000 7fe6777fe640 1 -- 172.21.15.29:0/2489945052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c030840 con 0x7fe6780b8480 2023-10-16T14:16:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.437+0000 7fe6777fe640 1 -- 172.21.15.29:0/2489945052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c0309a0 con 0x7fe6780b8480 2023-10-16T14:16:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.437+0000 7fe6777fe640 1 -- 172.21.15.29:0/2489945052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c030d30 con 0x7fe6780b8480 2023-10-16T14:16:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 -- 172.21.15.29:0/2489945052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 msgr2=0x7fe6780cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe6780cd7c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe66c009870 tx=0x7fe66c02b980 comp rx=0 tx=0).stop 2023-10-16T14:16:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 -- 172.21.15.29:0/2489945052 shutdown_connections 2023-10-16T14:16:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2489945052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe6780cd7c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 -- 172.21.15.29:0/2489945052 >> 172.21.15.29:0/2489945052 conn(0x7fe6780cf570 msgr2=0x7fe6780cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 -- 172.21.15.29:0/2489945052 shutdown_connections 2023-10-16T14:16:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.438+0000 7fe67db6e640 1 -- 172.21.15.29:0/2489945052 wait complete. 2023-10-16T14:16:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.440+0000 7fe67db6e640 1 Processor -- start 2023-10-16T14:16:44.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.440+0000 7fe67db6e640 1 -- start start 2023-10-16T14:16:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67db6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe678152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67db6e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67db6e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe67815cd70 0x7fe6780cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67db6e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67815dd20 con 0x7fe6780b8480 2023-10-16T14:16:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67db6e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678155c70 con 0x7fe67815cd70 2023-10-16T14:16:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67db6e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67815d760 con 0x7fe6780d0be0 2023-10-16T14:16:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67cb6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe678152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67cb6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe678152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46102/0 (socket says 172.21.15.29:46102) 2023-10-16T14:16:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.441+0000 7fe67cb6c640 1 -- 172.21.15.29:0/2142271277 learned_addr learned my addr 172.21.15.29:0/2142271277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe677fff640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67cb6c640 1 -- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 msgr2=0x7fe6780cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67cb6c640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67cb6c640 1 -- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe67815cd70 msgr2=0x7fe6780cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67cb6c640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe67815cd70 0x7fe6780cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67cb6c640 1 -- 172.21.15.29:0/2142271277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c0094d0 con 0x7fe6780b8480 2023-10-16T14:16:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67cb6c640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe678152af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe66800c730 tx=0x7fe66800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668003b70 con 0x7fe6780b8480 2023-10-16T14:16:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe668003cf0 con 0x7fe6780b8480 2023-10-16T14:16:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668003ea0 con 0x7fe6780b8480 2023-10-16T14:16:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.442+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67815da40 con 0x7fe6780b8480 2023-10-16T14:16:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.443+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe678153030 con 0x7fe6780b8480 2023-10-16T14:16:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.444+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6780d2220 con 0x7fe6780b8480 2023-10-16T14:16:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.445+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe66800aa90 con 0x7fe6780b8480 2023-10-16T14:16:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.445+0000 7fe6757fa640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe66406ae70 0x7fe66406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.446+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe668089450 con 0x7fe6780b8480 2023-10-16T14:16:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.446+0000 7fe677fff640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe66406ae70 0x7fe66406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.447+0000 7fe677fff640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe66406ae70 0x7fe66406d330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe66c036000 tx=0x7fe66c03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.450+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe6680544f0 con 0x7fe6780b8480 2023-10-16T14:16:44.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.643+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6780b82a0 con 0x7fe6780b8480 2023-10-16T14:16:44.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.644+0000 7fe6757fa640 1 -- 172.21.15.29:0/2142271277 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe668057b10 con 0x7fe6780b8480 2023-10-16T14:16:44.648 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:16:44.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe66406ae70 msgr2=0x7fe66406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe66406ae70 0x7fe66406d330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe66c036000 tx=0x7fe66c03b040 comp rx=0 tx=0).stop 2023-10-16T14:16:44.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 msgr2=0x7fe678152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe678152af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe66800c730 tx=0x7fe66800cbf0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 shutdown_connections 2023-10-16T14:16:44.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe66406ae70 0x7fe66406d330 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe67815cd70 0x7fe6780cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6780d0be0 0x7fe6780cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 --2- 172.21.15.29:0/2142271277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6780b8480 0x7fe678152af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 >> 172.21.15.29:0/2142271277 conn(0x7fe6780cf570 msgr2=0x7fe6780b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:44.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 shutdown_connections 2023-10-16T14:16:44.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.647+0000 7fe67db6e640 1 -- 172.21.15.29:0/2142271277 wait complete. 2023-10-16T14:16:44.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:16:44.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.895+0000 7f83f5287640 1 Processor -- start 2023-10-16T14:16:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.895+0000 7f83f5287640 1 -- start start 2023-10-16T14:16:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.897+0000 7f83f5287640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f83f0159ab0 0x7f83f00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.897+0000 7f83f5287640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83f0159e90 0x7f83f006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.897+0000 7f83f5287640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00cd1f0 con 0x7f83f006c140 2023-10-16T14:16:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.897+0000 7f83f5287640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f015d890 con 0x7f83f0159e90 2023-10-16T14:16:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.897+0000 7f83f5287640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f00cd4d0 con 0x7f83f0159ab0 2023-10-16T14:16:44.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83f0159e90 0x7f83f006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83ef7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f83f006c140 0x7f83f015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38964/0 (socket says 172.21.15.29:38964) 2023-10-16T14:16:44.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83ef7fe640 1 -- 172.21.15.29:0/3640175205 learned_addr learned my addr 172.21.15.29:0/3640175205 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:44.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83eeffd640 1 --2- 172.21.15.29:0/3640175205 >> v2:172.21.15.170:3300/0 conn(0x7f83f0159ab0 0x7f83f00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83e7fff640 1 -- 172.21.15.29:0/3640175205 >> v2:172.21.15.170:3300/0 conn(0x7f83f0159ab0 msgr2=0x7f83f00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:44.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83e7fff640 1 --2- 172.21.15.29:0/3640175205 >> v2:172.21.15.170:3300/0 conn(0x7f83f0159ab0 0x7f83f00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83e7fff640 1 -- 172.21.15.29:0/3640175205 >> v1:172.21.15.29:6789/0 conn(0x7f83f006c140 legacy=0x7f83f015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.898+0000 7f83e7fff640 1 -- 172.21.15.29:0/3640175205 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f00cd7b0 con 0x7f83f0159e90 2023-10-16T14:16:44.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.899+0000 7f83e7fff640 1 --2- 172.21.15.29:0/3640175205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159e90 0x7f83f006fc50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f83f00cfa00 tx=0x7f83d802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c297d8a5f3e9543 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:44.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.899+0000 7f83ee7fc640 1 -- 172.21.15.29:0/3640175205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8031520 con 0x7f83f0159e90 2023-10-16T14:16:44.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83ee7fc640 1 -- 172.21.15.29:0/3640175205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8031680 con 0x7f83f0159e90 2023-10-16T14:16:44.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83ee7fc640 1 -- 172.21.15.29:0/3640175205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8031a10 con 0x7f83f0159e90 2023-10-16T14:16:44.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 -- 172.21.15.29:0/3640175205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159e90 msgr2=0x7f83f006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 --2- 172.21.15.29:0/3640175205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159e90 0x7f83f006fc50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f83f00cfa00 tx=0x7f83d802b7c0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 -- 172.21.15.29:0/3640175205 shutdown_connections 2023-10-16T14:16:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 --2- 172.21.15.29:0/3640175205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159e90 0x7f83f006fc50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 --2- 172.21.15.29:0/3640175205 >> v2:172.21.15.170:3300/0 conn(0x7f83f0159ab0 0x7f83f00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 -- 172.21.15.29:0/3640175205 >> 172.21.15.29:0/3640175205 conn(0x7f83f00ce9b0 msgr2=0x7f83f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 -- 172.21.15.29:0/3640175205 shutdown_connections 2023-10-16T14:16:44.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.900+0000 7f83f5287640 1 -- 172.21.15.29:0/3640175205 wait complete. 2023-10-16T14:16:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.902+0000 7f83f5287640 1 Processor -- start 2023-10-16T14:16:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.902+0000 7f83f5287640 1 -- start start 2023-10-16T14:16:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83f5287640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83f006c140 0x7f83f00c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83f5287640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 0x7f83f0156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83f5287640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83f0159e90 0x7f83f006c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83f5287640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f00c8ff0 con 0x7f83f0159ab0 2023-10-16T14:16:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83f5287640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f0156d90 con 0x7f83f006c140 2023-10-16T14:16:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83f5287640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f00c3660 con 0x7f83f0159e90 2023-10-16T14:16:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.903+0000 7f83e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 0x7f83f0156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 0x7f83f0156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46122/0 (socket says 172.21.15.29:46122) 2023-10-16T14:16:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 -- 172.21.15.29:0/3383475491 learned_addr learned my addr 172.21.15.29:0/3383475491 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83eeffd640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83f006c140 0x7f83f00c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 -- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83f0159e90 msgr2=0x7f83f006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83f0159e90 0x7f83f006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 -- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83f006c140 msgr2=0x7f83f00c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83f006c140 0x7f83f00c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 -- 172.21.15.29:0/3383475491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8009400 con 0x7f83f0159ab0 2023-10-16T14:16:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e7fff640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 0x7f83f0156850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000c00 tx=0x7f83d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8033bd0 con 0x7f83f0159ab0 2023-10-16T14:16:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8033d30 con 0x7f83f0159ab0 2023-10-16T14:16:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.904+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8033ea0 con 0x7f83f0159ab0 2023-10-16T14:16:44.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.905+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f015afe0 con 0x7f83f0159ab0 2023-10-16T14:16:44.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.905+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f015b5a0 con 0x7f83f0159ab0 2023-10-16T14:16:44.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.906+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f004e160 con 0x7f83f0159ab0 2023-10-16T14:16:44.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.907+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f83d803a040 con 0x7f83f0159ab0 2023-10-16T14:16:44.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.908+0000 7f83e77fe640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f83d006ae70 0x7f83d006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:44.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.908+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83d80aaf60 con 0x7f83f0159ab0 2023-10-16T14:16:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.908+0000 7f83eeffd640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f83d006ae70 0x7f83d006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.909+0000 7f83eeffd640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f83d006ae70 0x7f83d006d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83e0006fd0 tx=0x7f83e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:44.912+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83d8076130 con 0x7f83f0159ab0 2023-10-16T14:16:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.105+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f83f00ccb60 con 0x7f83f0159ab0 2023-10-16T14:16:45.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.106+0000 7f83e77fe640 1 -- 172.21.15.29:0/3383475491 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f83d8006e10 con 0x7f83f0159ab0 2023-10-16T14:16:45.110 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:16:45.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.108+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f83d006ae70 msgr2=0x7f83d006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.108+0000 7f83f5287640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f83d006ae70 0x7f83d006d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83e0006fd0 tx=0x7f83e0008040 comp rx=0 tx=0).stop 2023-10-16T14:16:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.108+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 msgr2=0x7f83f0156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.108+0000 7f83f5287640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 0x7f83f0156850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000c00 tx=0x7f83d80046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 shutdown_connections 2023-10-16T14:16:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f83d006ae70 0x7f83d006d330 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83f0159ab0 0x7f83f0156850 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83f0159e90 0x7f83f006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 --2- 172.21.15.29:0/3383475491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83f006c140 0x7f83f00c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 >> 172.21.15.29:0/3383475491 conn(0x7f83f00ce9b0 msgr2=0x7f83f00ae410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 shutdown_connections 2023-10-16T14:16:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.109+0000 7f83f5287640 1 -- 172.21.15.29:0/3383475491 wait complete. 2023-10-16T14:16:45.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:16:45.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.349+0000 7f963759e640 1 Processor -- start 2023-10-16T14:16:45.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.350+0000 7f963759e640 1 -- start start 2023-10-16T14:16:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f963759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f963815cbe0 0x7f96380d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f963759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96380b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f963759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380d15a0 con 0x7f963815cfc0 2023-10-16T14:16:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f963759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380d1880 con 0x7f963815cbe0 2023-10-16T14:16:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f963759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96380d1b60 con 0x7f96380b4c80 2023-10-16T14:16:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f963659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f963815cbe0 0x7f96380d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f962ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f963815cfc0 0x7f96380b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38966/0 (socket says 172.21.15.29:38966) 2023-10-16T14:16:45.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f962ffff640 1 -- 172.21.15.29:0/3267301587 learned_addr learned my addr 172.21.15.29:0/3267301587 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:45.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.351+0000 7f9636d9d640 1 --2- 172.21.15.29:0/3267301587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96380b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.352+0000 7f963659c640 1 -- 172.21.15.29:0/3267301587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 msgr2=0x7f96380b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.352+0000 7f963659c640 1 --2- 172.21.15.29:0/3267301587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96380b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.352+0000 7f963659c640 1 -- 172.21.15.29:0/3267301587 >> v1:172.21.15.29:6789/0 conn(0x7f963815cfc0 legacy=0x7f96380b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.352+0000 7f963659c640 1 -- 172.21.15.29:0/3267301587 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380d1e40 con 0x7f963815cbe0 2023-10-16T14:16:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.352+0000 7f9635d9b640 1 -- 172.21.15.29:0/3267301587 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022224485 0 0) 0x7f96380d15a0 con 0x7f963815cfc0 2023-10-16T14:16:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.353+0000 7f963659c640 1 --2- 172.21.15.29:0/3267301587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cbe0 0x7f96380d0b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f96280097a0 tx=0x7f962802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a468a3d5121db4b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.353+0000 7f9635d9b640 1 -- 172.21.15.29:0/3267301587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628030610 con 0x7f963815cbe0 2023-10-16T14:16:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.353+0000 7f9635d9b640 1 -- 172.21.15.29:0/3267301587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9628030770 con 0x7f963815cbe0 2023-10-16T14:16:45.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.353+0000 7f9635d9b640 1 -- 172.21.15.29:0/3267301587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628030b00 con 0x7f963815cbe0 2023-10-16T14:16:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 -- 172.21.15.29:0/3267301587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cbe0 msgr2=0x7f96380d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 --2- 172.21.15.29:0/3267301587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cbe0 0x7f96380d0b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f96280097a0 tx=0x7f962802b960 comp rx=0 tx=0).stop 2023-10-16T14:16:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 -- 172.21.15.29:0/3267301587 shutdown_connections 2023-10-16T14:16:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 --2- 172.21.15.29:0/3267301587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96380b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 --2- 172.21.15.29:0/3267301587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cbe0 0x7f96380d0b50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 -- 172.21.15.29:0/3267301587 >> 172.21.15.29:0/3267301587 conn(0x7f96380cf570 msgr2=0x7f96380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 -- 172.21.15.29:0/3267301587 shutdown_connections 2023-10-16T14:16:45.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.354+0000 7f963759e640 1 -- 172.21.15.29:0/3267301587 wait complete. 2023-10-16T14:16:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.356+0000 7f963759e640 1 Processor -- start 2023-10-16T14:16:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.356+0000 7f963759e640 1 -- start start 2023-10-16T14:16:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f963759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96381f51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f963759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f963815cbe0 0x7f963815e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f963759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 0x7f96381e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f963759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963815ff50 con 0x7f963815cfc0 2023-10-16T14:16:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f963759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963815f340 con 0x7f963815cbe0 2023-10-16T14:16:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f963759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963815f620 con 0x7f96380b4c80 2023-10-16T14:16:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 0x7f96381e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 0x7f96381e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46152/0 (socket says 172.21.15.29:46152) 2023-10-16T14:16:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.357+0000 7f9636d9d640 1 -- 172.21.15.29:0/2762124659 learned_addr learned my addr 172.21.15.29:0/2762124659 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f963659c640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96381f51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f962ffff640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f963815cbe0 0x7f963815e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f9636d9d640 1 -- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 msgr2=0x7f96381f51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f9636d9d640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96381f51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f9636d9d640 1 -- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f963815cbe0 msgr2=0x7f963815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f9636d9d640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f963815cbe0 0x7f963815e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f9636d9d640 1 -- 172.21.15.29:0/2762124659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628009400 con 0x7f963815cfc0 2023-10-16T14:16:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f962ffff640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f963815cbe0 0x7f963815e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:45.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f963659c640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96381f51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f9636d9d640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 0x7f96381e5df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9630007d90 tx=0x7f963000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630014420 con 0x7f963815cfc0 2023-10-16T14:16:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300145a0 con 0x7f963815cfc0 2023-10-16T14:16:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630014950 con 0x7f963815cfc0 2023-10-16T14:16:45.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963815f900 con 0x7f963815cfc0 2023-10-16T14:16:45.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.358+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96381f5710 con 0x7f963815cfc0 2023-10-16T14:16:45.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.359+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96381600c0 con 0x7f963815cfc0 2023-10-16T14:16:45.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.360+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9630014ab0 con 0x7f963815cfc0 2023-10-16T14:16:45.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.362+0000 7f962effd640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f961806aea0 0x7f961806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.362+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9630089fc0 con 0x7f963815cfc0 2023-10-16T14:16:45.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.363+0000 7f963659c640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f961806aea0 0x7f961806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.364+0000 7f963659c640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f961806aea0 0x7f961806d360 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9628036000 tx=0x7f962803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:45.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.366+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9630054fd0 con 0x7f963815cfc0 2023-10-16T14:16:45.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.599+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f963804e9b0 con 0x7f963815cfc0 2023-10-16T14:16:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.600+0000 7f962effd640 1 -- 172.21.15.29:0/2762124659 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f96300585f0 con 0x7f963815cfc0 2023-10-16T14:16:45.605 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:45.605 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:16:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:16:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f961806aea0 msgr2=0x7f961806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f961806aea0 0x7f961806d360 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9628036000 tx=0x7f962803b040 comp rx=0 tx=0).stop 2023-10-16T14:16:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 msgr2=0x7f96381e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 0x7f96381e5df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9630007d90 tx=0x7f963000e4b0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 shutdown_connections 2023-10-16T14:16:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f961806aea0 0x7f961806d360 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f963815cfc0 0x7f96381e5df0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f963815cbe0 0x7f963815e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 --2- 172.21.15.29:0/2762124659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96380b4c80 0x7f96381f51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.603+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 >> 172.21.15.29:0/2762124659 conn(0x7f96380cf570 msgr2=0x7f963815b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.604+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 shutdown_connections 2023-10-16T14:16:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.604+0000 7f963759e640 1 -- 172.21.15.29:0/2762124659 wait complete. 2023-10-16T14:16:45.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:16:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.852+0000 7f56f2b68640 1 Processor -- start 2023-10-16T14:16:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.852+0000 7f56f2b68640 1 -- start start 2023-10-16T14:16:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f2b68640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56ec159cc0 0x7f56ec06dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f2b68640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f2b68640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec070970 con 0x7f56ec065bd0 2023-10-16T14:16:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f2b68640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec06e7f0 con 0x7f56ec159cc0 2023-10-16T14:16:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f2b68640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0ce0c0 con 0x7f56ec06b4c0 2023-10-16T14:16:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f0904640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56ec159cc0 0x7f56ec06dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f0904640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56ec159cc0 0x7f56ec06dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46154/0 (socket says 172.21.15.29:46154) 2023-10-16T14:16:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.854+0000 7f56f0904640 1 -- 172.21.15.29:0/1330736481 learned_addr learned my addr 172.21.15.29:0/1330736481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.855+0000 7f56f0904640 1 -- 172.21.15.29:0/1330736481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 msgr2=0x7f56ec0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:45.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.855+0000 7f56f1105640 1 --2- 172.21.15.29:0/1330736481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.855+0000 7f56f0904640 1 --2- 172.21.15.29:0/1330736481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.855+0000 7f56f0904640 1 -- 172.21.15.29:0/1330736481 >> v1:172.21.15.170:6789/0 conn(0x7f56ec065bd0 legacy=0x7f56ec06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:45.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.855+0000 7f56f0904640 1 -- 172.21.15.29:0/1330736481 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0ce3a0 con 0x7f56ec159cc0 2023-10-16T14:16:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.856+0000 7f56f0904640 1 --2- 172.21.15.29:0/1330736481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec06dfa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56e800bb10 tx=0x7f56e802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7976b656dbcb4f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:45.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.856+0000 7f56e37fe640 1 -- 172.21.15.29:0/1330736481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e802ee70 con 0x7f56ec159cc0 2023-10-16T14:16:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.856+0000 7f56e37fe640 1 -- 172.21.15.29:0/1330736481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e8036440 con 0x7f56ec159cc0 2023-10-16T14:16:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.856+0000 7f56e37fe640 1 -- 172.21.15.29:0/1330736481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e80367d0 con 0x7f56ec159cc0 2023-10-16T14:16:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 -- 172.21.15.29:0/1330736481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 msgr2=0x7f56ec06dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 --2- 172.21.15.29:0/1330736481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec06dfa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56e800bb10 tx=0x7f56e802e0c0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 -- 172.21.15.29:0/1330736481 shutdown_connections 2023-10-16T14:16:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 --2- 172.21.15.29:0/1330736481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 --2- 172.21.15.29:0/1330736481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec06dfa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 -- 172.21.15.29:0/1330736481 >> 172.21.15.29:0/1330736481 conn(0x7f56ec0b0d30 msgr2=0x7f56ec0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 -- 172.21.15.29:0/1330736481 shutdown_connections 2023-10-16T14:16:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.857+0000 7f56f2b68640 1 -- 172.21.15.29:0/1330736481 wait complete. 2023-10-16T14:16:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.859+0000 7f56f2b68640 1 Processor -- start 2023-10-16T14:16:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.859+0000 7f56f2b68640 1 -- start start 2023-10-16T14:16:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f2b68640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56ec065bd0 0x7f56ec0ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f2b68640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f2b68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec0caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f2b68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec15a0a0 con 0x7f56ec159cc0 2023-10-16T14:16:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f2b68640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec09d700 con 0x7f56ec065bd0 2023-10-16T14:16:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f2b68640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0c8d80 con 0x7f56ec06b4c0 2023-10-16T14:16:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f1105640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec0caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f1105640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec0caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46168/0 (socket says 172.21.15.29:46168) 2023-10-16T14:16:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.860+0000 7f56f1105640 1 -- 172.21.15.29:0/4185569720 learned_addr learned my addr 172.21.15.29:0/4185569720 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56e3fff640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f0904640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56ec065bd0 0x7f56ec0ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f1105640 1 -- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 msgr2=0x7f56ec0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f1105640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f1105640 1 -- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56ec065bd0 msgr2=0x7f56ec0ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f1105640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56ec065bd0 0x7f56ec0ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f1105640 1 -- 172.21.15.29:0/4185569720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e800b770 con 0x7f56ec159cc0 2023-10-16T14:16:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f1105640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec0caa40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f56e400cd20 tx=0x7f56e400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4003b90 con 0x7f56ec159cc0 2023-10-16T14:16:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4003cf0 con 0x7f56ec159cc0 2023-10-16T14:16:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec0c9060 con 0x7f56ec159cc0 2023-10-16T14:16:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.861+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4004240 con 0x7f56ec159cc0 2023-10-16T14:16:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.862+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec15a5d0 con 0x7f56ec159cc0 2023-10-16T14:16:45.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.863+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec04e160 con 0x7f56ec159cc0 2023-10-16T14:16:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.864+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f56e40043a0 con 0x7f56ec159cc0 2023-10-16T14:16:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.864+0000 7f56e17fa640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f56d806aea0 0x7f56d806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.865+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56e4012070 con 0x7f56ec159cc0 2023-10-16T14:16:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.865+0000 7f56f0904640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f56d806aea0 0x7f56d806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.866+0000 7f56f0904640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f56d806aea0 0x7f56d806d360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f56e8038000 tx=0x7f56e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:45.869+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f56e4055040 con 0x7f56ec159cc0 2023-10-16T14:16:46.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.064+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56ec159690 con 0x7f56ec159cc0 2023-10-16T14:16:46.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.066+0000 7f56e17fa640 1 -- 172.21.15.29:0/4185569720 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f56e4058660 con 0x7f56ec159cc0 2023-10-16T14:16:46.071 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:46.071 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:16:46.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.069+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f56d806aea0 msgr2=0x7f56d806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.069+0000 7f56f2b68640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f56d806aea0 0x7f56d806d360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f56e8038000 tx=0x7f56e8008040 comp rx=0 tx=0).stop 2023-10-16T14:16:46.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.069+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 msgr2=0x7f56ec0caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.069+0000 7f56f2b68640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec0caa40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f56e400cd20 tx=0x7f56e400a660 comp rx=0 tx=0).stop 2023-10-16T14:16:46.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 shutdown_connections 2023-10-16T14:16:46.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f56d806aea0 0x7f56d806d360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ec159cc0 0x7f56ec0caa40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56ec06b4c0 0x7f56ec0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 --2- 172.21.15.29:0/4185569720 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56ec065bd0 0x7f56ec0ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 >> 172.21.15.29:0/4185569720 conn(0x7f56ec0b0d30 msgr2=0x7f56ec15d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:46.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 shutdown_connections 2023-10-16T14:16:46.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.070+0000 7f56f2b68640 1 -- 172.21.15.29:0/4185569720 wait complete. 2023-10-16T14:16:46.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:16:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.313+0000 7f65cceca640 1 Processor -- start 2023-10-16T14:16:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.314+0000 7f65cceca640 1 -- start start 2023-10-16T14:16:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65cceca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815cd70 0x7f65c80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:46.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65cceca640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65c80cded0 0x7f65c80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:46.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65cceca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c815d150 con 0x7f65c80b8480 2023-10-16T14:16:46.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65cceca640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c80d1180 con 0x7f65c80cded0 2023-10-16T14:16:46.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65cceca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80d1460 con 0x7f65c815cd70 2023-10-16T14:16:46.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65c6ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65c80b8480 0x7f65c80cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38972/0 (socket says 172.21.15.29:38972) 2023-10-16T14:16:46.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.315+0000 7f65c6ffd640 1 -- 172.21.15.29:0/2625036545 learned_addr learned my addr 172.21.15.29:0/2625036545 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:46.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c7fff640 1 --2- 172.21.15.29:0/2625036545 >> v2:172.21.15.29:3300/0 conn(0x7f65c80cded0 0x7f65c80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c77fe640 1 --2- 172.21.15.29:0/2625036545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815cd70 0x7f65c80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c67fc640 1 -- 172.21.15.29:0/2625036545 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692701651 0 0) 0x7f65c815d150 con 0x7f65c80b8480 2023-10-16T14:16:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c67fc640 1 -- 172.21.15.29:0/2625036545 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b4003520 con 0x7f65c80b8480 2023-10-16T14:16:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c7fff640 1 -- 172.21.15.29:0/2625036545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815cd70 msgr2=0x7f65c80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c7fff640 1 --2- 172.21.15.29:0/2625036545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815cd70 0x7f65c80d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c7fff640 1 -- 172.21.15.29:0/2625036545 >> v1:172.21.15.29:6789/0 conn(0x7f65c80b8480 legacy=0x7f65c80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.316+0000 7f65c7fff640 1 -- 172.21.15.29:0/2625036545 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80d1740 con 0x7f65c80cded0 2023-10-16T14:16:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.317+0000 7f65c7fff640 1 --2- 172.21.15.29:0/2625036545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c80b5c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f65c00097a0 tx=0x7f65c002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae3180c1af6da8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65c67fc640 1 -- 172.21.15.29:0/2625036545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c002d040 con 0x7f65c80cded0 2023-10-16T14:16:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65c67fc640 1 -- 172.21.15.29:0/2625036545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c0031810 con 0x7f65c80cded0 2023-10-16T14:16:46.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65c67fc640 1 -- 172.21.15.29:0/2625036545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0031ba0 con 0x7f65c80cded0 2023-10-16T14:16:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65cceca640 1 -- 172.21.15.29:0/2625036545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 msgr2=0x7f65c80b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65cceca640 1 --2- 172.21.15.29:0/2625036545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c80b5c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f65c00097a0 tx=0x7f65c002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65cceca640 1 -- 172.21.15.29:0/2625036545 shutdown_connections 2023-10-16T14:16:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65cceca640 1 --2- 172.21.15.29:0/2625036545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c80b5c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65cceca640 1 --2- 172.21.15.29:0/2625036545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815cd70 0x7f65c80d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.318+0000 7f65cceca640 1 -- 172.21.15.29:0/2625036545 >> 172.21.15.29:0/2625036545 conn(0x7f65c80cf570 msgr2=0x7f65c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.319+0000 7f65cceca640 1 -- 172.21.15.29:0/2625036545 shutdown_connections 2023-10-16T14:16:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.319+0000 7f65cceca640 1 -- 172.21.15.29:0/2625036545 wait complete. 2023-10-16T14:16:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.320+0000 7f65cceca640 1 Processor -- start 2023-10-16T14:16:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.321+0000 7f65cceca640 1 -- start start 2023-10-16T14:16:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65cceca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c80b8480 0x7f65c815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65cceca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c8157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:46.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65cceca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c815cd70 0x7f65c815f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65cceca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c815d340 con 0x7f65c80cded0 2023-10-16T14:16:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65cceca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c815d900 con 0x7f65c815cd70 2023-10-16T14:16:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65cceca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c815fea0 con 0x7f65c80b8480 2023-10-16T14:16:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c8157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:46.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c8157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46198/0 (socket says 172.21.15.29:46198) 2023-10-16T14:16:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 -- 172.21.15.29:0/196808021 learned_addr learned my addr 172.21.15.29:0/196808021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c77fe640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c80b8480 0x7f65c815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 -- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c80b8480 msgr2=0x7f65c815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c80b8480 0x7f65c815dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 -- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c815cd70 msgr2=0x7f65c815f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c815cd70 0x7f65c815f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.322+0000 7f65c6ffd640 1 -- 172.21.15.29:0/196808021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c0009400 con 0x7f65c80cded0 2023-10-16T14:16:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.323+0000 7f65c6ffd640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c8157f50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c530 tx=0x7f65b000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.323+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0005ce0 con 0x7f65c80cded0 2023-10-16T14:16:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.323+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b0005e40 con 0x7f65c80cded0 2023-10-16T14:16:46.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.323+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00072b0 con 0x7f65c80cded0 2023-10-16T14:16:46.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.323+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c8160180 con 0x7f65c80cded0 2023-10-16T14:16:46.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.323+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c8158490 con 0x7f65c80cded0 2023-10-16T14:16:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.324+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c804e9b0 con 0x7f65c80cded0 2023-10-16T14:16:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.326+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f65b0007410 con 0x7f65c80cded0 2023-10-16T14:16:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.327+0000 7f65bffff640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65a806aea0 0x7f65a806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:46.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.327+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65b0088ca0 con 0x7f65c80cded0 2023-10-16T14:16:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.327+0000 7f65c77fe640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65a806aea0 0x7f65a806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.328+0000 7f65c77fe640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65a806aea0 0x7f65a806d360 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65b80073f0 tx=0x7f65b800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:46.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.331+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65b0053d70 con 0x7f65c80cded0 2023-10-16T14:16:46.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.560+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65c815da40 con 0x7f65c80cded0 2023-10-16T14:16:46.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.561+0000 7f65bffff640 1 -- 172.21.15.29:0/196808021 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f65b0017020 con 0x7f65c80cded0 2023-10-16T14:16:46.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:16:46.566 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:16:46.566 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:16:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.563+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65a806aea0 msgr2=0x7f65a806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.563+0000 7f65cceca640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65a806aea0 0x7f65a806d360 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65b80073f0 tx=0x7f65b800b040 comp rx=0 tx=0).stop 2023-10-16T14:16:46.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.563+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 msgr2=0x7f65c8157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.563+0000 7f65cceca640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c8157f50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c530 tx=0x7f65b000c9f0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 shutdown_connections 2023-10-16T14:16:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65a806aea0 0x7f65a806d360 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c80cded0 0x7f65c8157f50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c815cd70 0x7f65c815f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 --2- 172.21.15.29:0/196808021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c80b8480 0x7f65c815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 >> 172.21.15.29:0/196808021 conn(0x7f65c80cf570 msgr2=0x7f65c80b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 shutdown_connections 2023-10-16T14:16:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:46.564+0000 7f65cceca640 1 -- 172.21.15.29:0/196808021 wait complete. 2023-10-16T14:16:46.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:16:29.503784+0000', 'modified': '2023-10-16T14:16:32.553956+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6837/3275719351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6835', 'nonce': 3275719351}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 3275719351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:16:46.988 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:16:46.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T14:16:47.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.217+0000 7f232a555640 1 Processor -- start 2023-10-16T14:16:47.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.217+0000 7f232a555640 1 -- start start 2023-10-16T14:16:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.218+0000 7f232a555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23241618b0 0x7f2324065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.218+0000 7f232a555640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2324065ff0 0x7f23240ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.218+0000 7f232a555640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2324066420 con 0x7f232415d330 2023-10-16T14:16:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.218+0000 7f232a555640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232406b1e0 con 0x7f2324065ff0 2023-10-16T14:16:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.218+0000 7f232a555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232406cee0 con 0x7f23241618b0 2023-10-16T14:16:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2324065ff0 0x7f23240ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2324065ff0 0x7f23240ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46204/0 (socket says 172.21.15.29:46204) 2023-10-16T14:16:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 -- 172.21.15.29:0/3493864426 learned_addr learned my addr 172.21.15.29:0/3493864426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f23237fe640 1 --2- 172.21.15.29:0/3493864426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23241618b0 0x7f2324065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 -- 172.21.15.29:0/3493864426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23241618b0 msgr2=0x7f2324065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 --2- 172.21.15.29:0/3493864426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23241618b0 0x7f2324065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 -- 172.21.15.29:0/3493864426 >> v1:172.21.15.170:6789/0 conn(0x7f232415d330 legacy=0x7f232406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.219+0000 7f2328af2640 1 -- 172.21.15.29:0/3493864426 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240d0890 con 0x7f2324065ff0 2023-10-16T14:16:47.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.220+0000 7f2328af2640 1 --2- 172.21.15.29:0/3493864426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2324065ff0 0x7f23240ce9b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f231c009870 tx=0x7f231c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338c8be352369270 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f2322ffd640 1 -- 172.21.15.29:0/3493864426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c031640 con 0x7f2324065ff0 2023-10-16T14:16:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f2322ffd640 1 -- 172.21.15.29:0/3493864426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c0317a0 con 0x7f2324065ff0 2023-10-16T14:16:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f2322ffd640 1 -- 172.21.15.29:0/3493864426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c031b30 con 0x7f2324065ff0 2023-10-16T14:16:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 -- 172.21.15.29:0/3493864426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2324065ff0 msgr2=0x7f23240ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 --2- 172.21.15.29:0/3493864426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2324065ff0 0x7f23240ce9b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f231c009870 tx=0x7f231c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 -- 172.21.15.29:0/3493864426 shutdown_connections 2023-10-16T14:16:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 --2- 172.21.15.29:0/3493864426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2324065ff0 0x7f23240ce9b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 --2- 172.21.15.29:0/3493864426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23241618b0 0x7f2324065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 -- 172.21.15.29:0/3493864426 >> 172.21.15.29:0/3493864426 conn(0x7f23240d0ab0 msgr2=0x7f23240d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.221+0000 7f232a555640 1 -- 172.21.15.29:0/3493864426 shutdown_connections 2023-10-16T14:16:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.222+0000 7f232a555640 1 -- 172.21.15.29:0/3493864426 wait complete. 2023-10-16T14:16:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.223+0000 7f232a555640 1 Processor -- start 2023-10-16T14:16:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.223+0000 7f232a555640 1 -- start start 2023-10-16T14:16:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f232a555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2324065ff0 0x7f2324156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f232a555640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f232415d330 0x7f23240aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f232a555640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 0x7f23240aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f232a555640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23240cab50 con 0x7f23241618b0 2023-10-16T14:16:47.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f232a555640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23240c65e0 con 0x7f232415d330 2023-10-16T14:16:47.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f232a555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23240c68c0 con 0x7f2324065ff0 2023-10-16T14:16:47.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f2328af2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 0x7f23240aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f2328af2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 0x7f23240aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46214/0 (socket says 172.21.15.29:46214) 2023-10-16T14:16:47.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.224+0000 7f2328af2640 1 -- 172.21.15.29:0/2151427048 learned_addr learned my addr 172.21.15.29:0/2151427048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:47.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f23237fe640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f232415d330 0x7f23240aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2323fff640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2324065ff0 0x7f2324156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2328af2640 1 -- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2324065ff0 msgr2=0x7f2324156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2328af2640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2324065ff0 0x7f2324156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2328af2640 1 -- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f232415d330 msgr2=0x7f23240aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2328af2640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f232415d330 0x7f23240aea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2328af2640 1 -- 172.21.15.29:0/2151427048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0094d0 con 0x7f23241618b0 2023-10-16T14:16:47.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f23237fe640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f232415d330 0x7f23240aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:47.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2328af2640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 0x7f23240aefa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f231c009870 tx=0x7f231c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2320ff9640 1 -- 172.21.15.29:0/2151427048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c03ac60 con 0x7f23241618b0 2023-10-16T14:16:47.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2320ff9640 1 -- 172.21.15.29:0/2151427048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c03ade0 con 0x7f23241618b0 2023-10-16T14:16:47.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f2320ff9640 1 -- 172.21.15.29:0/2151427048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c044630 con 0x7f23241618b0 2023-10-16T14:16:47.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.225+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23240c6ba0 con 0x7f23241618b0 2023-10-16T14:16:47.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.226+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240c7160 con 0x7f23241618b0 2023-10-16T14:16:47.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.228+0000 7f2320ff9640 1 -- 172.21.15.29:0/2151427048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f231c02f050 con 0x7f23241618b0 2023-10-16T14:16:47.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.228+0000 7f2320ff9640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f231006ae70 0x7f231006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.229+0000 7f2320ff9640 1 -- 172.21.15.29:0/2151427048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f231c0ab4f0 con 0x7f23241618b0 2023-10-16T14:16:47.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.229+0000 7f2323fff640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f231006ae70 0x7f231006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.229+0000 7f232d1c4740 1 Processor -- start 2023-10-16T14:16:47.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.230+0000 7f232d1c4740 1 -- start start 2023-10-16T14:16:47.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.230+0000 7f2323fff640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f231006ae70 0x7f231006d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2314007fd0 tx=0x7f2314009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.230+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916bc6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.230+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf4360 0x562916bf6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 0x562916bf7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f232d1c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916ae77a0 con 0x562916bc73c0 2023-10-16T14:16:47.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916b39740 con 0x562916bf4360 2023-10-16T14:16:47.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x562916b36e30 con 0x562916bf3f80 2023-10-16T14:16:47.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 0x562916bf7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 0x562916bf7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46226/0 (socket says 172.21.15.29:46226) 2023-10-16T14:16:47.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 -- 172.21.15.29:0/3263584064 learned_addr learned my addr 172.21.15.29:0/3263584064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:47.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 -- 172.21.15.29:0/3263584064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 msgr2=0x562916bc6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916bc6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 -- 172.21.15.29:0/3263584064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf4360 msgr2=0x562916bf6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf4360 0x562916bf6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 -- 172.21.15.29:0/3263584064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562916af6910 con 0x562916bc73c0 2023-10-16T14:16:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.231+0000 7f2328af2640 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 0x562916bf7250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f231c006f40 tx=0x7f231c031e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8af161f9b0a3e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f230a7fc640 1 -- 172.21.15.29:0/3263584064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c091020 con 0x562916bc73c0 2023-10-16T14:16:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f230a7fc640 1 -- 172.21.15.29:0/3263584064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c07e430 con 0x562916bc73c0 2023-10-16T14:16:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f230a7fc640 1 -- 172.21.15.29:0/3263584064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c0825e0 con 0x562916bc73c0 2023-10-16T14:16:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 -- 172.21.15.29:0/3263584064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 msgr2=0x562916bf7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 0x562916bf7250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f231c006f40 tx=0x7f231c031e40 comp rx=0 tx=0).stop 2023-10-16T14:16:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 -- 172.21.15.29:0/3263584064 shutdown_connections 2023-10-16T14:16:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bc73c0 0x562916bf7250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf4360 0x562916bf6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3263584064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916bc6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 -- 172.21.15.29:0/3263584064 >> 172.21.15.29:0/3263584064 conn(0x562916bc9190 msgr2=0x562916c11d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:47.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 -- 172.21.15.29:0/3263584064 shutdown_connections 2023-10-16T14:16:47.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.232+0000 7f232d1c4740 1 -- 172.21.15.29:0/3263584064 wait complete. 2023-10-16T14:16:47.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.233+0000 7f232d1c4740 1 -- start start 2023-10-16T14:16:47.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.233+0000 7f232d1c4740 1 Processor -- start 2023-10-16T14:16:47.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916be4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916c15190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf31a0 0x562916c156d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916b32720 con 0x562916bf4360 2023-10-16T14:16:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916ad0970 con 0x562916bf31a0 2023-10-16T14:16:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x562916ad0610 con 0x562916bf3f80 2023-10-16T14:16:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.234+0000 7f232d1c4740 10 client.0 _ll_register_callbacks cb 0x562916af29d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:16:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916c15190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916c15190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46240/0 (socket says 172.21.15.29:46240) 2023-10-16T14:16:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 -- 172.21.15.29:0/1981010664 learned_addr learned my addr 172.21.15.29:0/1981010664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f2323fff640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916be4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f2328af2640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf31a0 0x562916c156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 -- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 msgr2=0x562916be4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916be4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 -- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf31a0 msgr2=0x562916c156d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf31a0 0x562916c156d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 -- 172.21.15.29:0/1981010664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240c7160 con 0x562916bf4360 2023-10-16T14:16:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f2328af2640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf31a0 0x562916c156d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f2323fff640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916be4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f23237fe640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916c15190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f231800e1f0 tx=0x7f231800e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb0f1334c94da82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f2302ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:16:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.235+0000 7f2302ffd640 1 -- 172.21.15.29:0/1981010664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231800c070 con 0x562916bf4360 2023-10-16T14:16:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f2302ffd640 1 -- 172.21.15.29:0/1981010664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2318013610 con 0x562916bf4360 2023-10-16T14:16:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562916b59160 con 0x562916bf4360 2023-10-16T14:16:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f2302ffd640 1 -- 172.21.15.29:0/1981010664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f231800eee0 con 0x562916bf4360 2023-10-16T14:16:47.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 10 client.4602 fetch_fsmap learned FSMap version 8 2023-10-16T14:16:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562916b30da0 con 0x562916bf4360 2023-10-16T14:16:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f2302ffd640 1 -- 172.21.15.29:0/1981010664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562916b59160 con 0x562916bf4360 2023-10-16T14:16:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f2302ffd640 1 -- 172.21.15.29:0/1981010664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f23180137d0 con 0x562916bf4360 2023-10-16T14:16:47.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 10 client.4602 fetch_fsmap finished waiting for FSMap version 8 2023-10-16T14:16:47.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 10 client.4602 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T14:16:47.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 20 client.4602 populate_metadata read hostname 'smithi029' 2023-10-16T14:16:47.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bf9890 0x562916be28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 4 client.4602 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:16:47.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.236+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 --> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562916b04cc0 con 0x562916bf9890 2023-10-16T14:16:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.237+0000 7f2323fff640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bf9890 0x562916be28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.238+0000 7f2323fff640 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bf9890 0x562916be28a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3538e7ef814e26f server_cookie=531a54a67eef0778 in_seq=0 out_seq=0 2023-10-16T14:16:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.238+0000 7f2302ffd640 10 client.4602 ms_handle_connect on v2:172.21.15.170:6835/3275719351 2023-10-16T14:16:47.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f2302ffd640 1 -- 172.21.15.29:0/1981010664 <== mds.0 v2:172.21.15.170:6835/3275719351 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562916b04cc0 con 0x562916bf9890 2023-10-16T14:16:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f2302ffd640 10 client.4602 handle_command_reply: tid=0 2023-10-16T14:16:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 client.4602 shutdown 2023-10-16T14:16:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 msgr2=0x562916c15190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916c15190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f231800e1f0 tx=0x7f231800e6b0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 shutdown_connections 2023-10-16T14:16:47.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bf9890 0x562916be28a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916c15190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf31a0 0x562916c156d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1981010664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf3f80 0x562916be4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f2302ffd640 0 client.4602 ms_handle_reset on v2:172.21.15.170:6835/3275719351 2023-10-16T14:16:47.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 >> 172.21.15.29:0/1981010664 conn(0x562916bc73c0 msgr2=0x562916c14350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:47.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 shutdown_connections 2023-10-16T14:16:47.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.240+0000 7f232d1c4740 1 -- 172.21.15.29:0/1981010664 wait complete. 2023-10-16T14:16:47.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.241+0000 7f232d1c4740 20 client.4602 trim_cache size 0 max 16384 2023-10-16T14:16:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.257+0000 7f232d1c4740 1 Processor -- start 2023-10-16T14:16:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.257+0000 7f232d1c4740 1 -- start start 2023-10-16T14:16:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 0x562916c2b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf4360 0x562916bcdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f232d1c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916896f20 con 0x562916bf31a0 2023-10-16T14:16:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916c0d040 con 0x562916bf3f80 2023-10-16T14:16:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x562916bc8bf0 con 0x562916bf4360 2023-10-16T14:16:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f2323fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 0x562916c2b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f2323fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 0x562916c2b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46254/0 (socket says 172.21.15.29:46254) 2023-10-16T14:16:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.258+0000 7f2323fff640 1 -- 172.21.15.29:0/3242452880 learned_addr learned my addr 172.21.15.29:0/3242452880 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f2323fff640 1 -- 172.21.15.29:0/3242452880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf4360 msgr2=0x562916bcdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f2323fff640 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf4360 0x562916bcdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f2323fff640 1 -- 172.21.15.29:0/3242452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 msgr2=0x562916bf7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f2323fff640 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f2323fff640 1 -- 172.21.15.29:0/3242452880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562916acfb80 con 0x562916bf31a0 2023-10-16T14:16:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f2323fff640 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 0x562916c2b560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2314003b30 tx=0x7f2314001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f77ea9655b62b1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f230affd640 1 -- 172.21.15.29:0/3242452880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314006b50 con 0x562916bf31a0 2023-10-16T14:16:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f230affd640 1 -- 172.21.15.29:0/3242452880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2314006cb0 con 0x562916bf31a0 2023-10-16T14:16:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.259+0000 7f230affd640 1 -- 172.21.15.29:0/3242452880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400d9a0 con 0x562916bf31a0 2023-10-16T14:16:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 -- 172.21.15.29:0/3242452880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 msgr2=0x562916c2b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 0x562916c2b560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2314003b30 tx=0x7f2314001250 comp rx=0 tx=0).stop 2023-10-16T14:16:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 -- 172.21.15.29:0/3242452880 shutdown_connections 2023-10-16T14:16:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf4360 0x562916bcdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 --2- 172.21.15.29:0/3242452880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf31a0 0x562916c2b560 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 -- 172.21.15.29:0/3242452880 >> 172.21.15.29:0/3242452880 conn(0x562916bc73c0 msgr2=0x562916bf2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 -- 172.21.15.29:0/3242452880 shutdown_connections 2023-10-16T14:16:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.260+0000 7f232d1c4740 1 -- 172.21.15.29:0/3242452880 wait complete. 2023-10-16T14:16:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.261+0000 7f232d1c4740 1 -- start start 2023-10-16T14:16:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.261+0000 7f232d1c4740 1 Processor -- start 2023-10-16T14:16:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf31a0 0x562916c5b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916bf1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916c7b030 con 0x562916bf4360 2023-10-16T14:16:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562916be0b80 con 0x562916bf3f80 2023-10-16T14:16:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x562916c797f0 con 0x562916bf31a0 2023-10-16T14:16:47.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.262+0000 7f232d1c4740 10 client.0 _ll_register_callbacks cb 0x7f2318009120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:16:47.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916bf1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916bf1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46262/0 (socket says 172.21.15.29:46262) 2023-10-16T14:16:47.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 -- 172.21.15.29:0/1681382772 learned_addr learned my addr 172.21.15.29:0/1681382772 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:47.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f23237fe640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2323fff640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf31a0 0x562916c5b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 -- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf31a0 msgr2=0x562916c5b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf31a0 0x562916c5b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 -- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 msgr2=0x562916bf2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 -- 172.21.15.29:0/1681382772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2324066420 con 0x562916bf4360 2023-10-16T14:16:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f23237fe640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2328af2640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916bf1190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f231c006da0 tx=0x7f231c0a9850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3b60590b78c9ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2302ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:16:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2302ffd640 1 -- 172.21.15.29:0/1681382772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c091070 con 0x562916bf4360 2023-10-16T14:16:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.263+0000 7f2302ffd640 1 -- 172.21.15.29:0/1681382772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f231c078680 con 0x562916bf4360 2023-10-16T14:16:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f231c07e430 con 0x562916bf4360 2023-10-16T14:16:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f2302ffd640 1 -- 172.21.15.29:0/1681382772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f231c07a540 con 0x562916bf4360 2023-10-16T14:16:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 10 client.4608 fetch_fsmap learned FSMap version 8 2023-10-16T14:16:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x562916af5170 con 0x562916bf4360 2023-10-16T14:16:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f2302ffd640 1 -- 172.21.15.29:0/1681382772 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c07e430 con 0x562916bf4360 2023-10-16T14:16:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f2302ffd640 1 -- 172.21.15.29:0/1681382772 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f231c0797d0 con 0x562916bf4360 2023-10-16T14:16:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 10 client.4608 fetch_fsmap finished waiting for FSMap version 8 2023-10-16T14:16:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 10 client.4608 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T14:16:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 20 client.4608 populate_metadata read hostname 'smithi029' 2023-10-16T14:16:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bc61e0 0x562916bc65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 4 client.4608 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:16:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.264+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 --> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562916b2b700 con 0x562916bc61e0 2023-10-16T14:16:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.265+0000 7f2323fff640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bc61e0 0x562916bc65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f2323fff640 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bc61e0 0x562916bc65c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd53c9f23a92990e server_cookie=1571782fc34da586 in_seq=0 out_seq=0 2023-10-16T14:16:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f2302ffd640 10 client.4608 ms_handle_connect on v2:172.21.15.170:6835/3275719351 2023-10-16T14:16:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f2302ffd640 1 -- 172.21.15.29:0/1681382772 <== mds.0 v2:172.21.15.170:6835/3275719351 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x562916b2b700 con 0x562916bc61e0 2023-10-16T14:16:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f2302ffd640 10 client.4608 handle_command_reply: tid=0 2023-10-16T14:16:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 client.4608 shutdown 2023-10-16T14:16:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 msgr2=0x562916bf1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916bf1190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f231c006da0 tx=0x7f231c0a9850 comp rx=0 tx=0).stop 2023-10-16T14:16:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 shutdown_connections 2023-10-16T14:16:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:6835/3275719351,v1:172.21.15.170:6837/3275719351] conn(0x562916bc61e0 0x562916bc65c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562916bf4360 0x562916bf1190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x562916bf3f80 0x562916bf2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 --2- 172.21.15.29:0/1681382772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x562916bf31a0 0x562916c5b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 >> 172.21.15.29:0/1681382772 conn(0x562916bc73c0 msgr2=0x562916c738b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.266+0000 7f2302ffd640 0 client.4608 ms_handle_reset on v2:172.21.15.170:6835/3275719351 2023-10-16T14:16:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 shutdown_connections 2023-10-16T14:16:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232d1c4740 1 -- 172.21.15.29:0/1681382772 wait complete. 2023-10-16T14:16:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232d1c4740 20 client.4608 trim_cache size 0 max 16384 2023-10-16T14:16:47.291 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:16:47.291 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:16:47.291 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:16:47.291 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4256, 2023-10-16T14:16:47.291 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:16:47.292 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:16:47.292 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:16:47.292 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 17.246444692000001, 2023-10-16T14:16:47.292 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 8, 2023-10-16T14:16:47.292 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 20, 2023-10-16T14:16:47.293 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 20, 2023-10-16T14:16:47.293 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 22.444689643 2023-10-16T14:16:47.293 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:16:47.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f231006ae70 msgr2=0x7f231006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f231006ae70 0x7f231006d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2314007fd0 tx=0x7f2314009040 comp rx=0 tx=0).stop 2023-10-16T14:16:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 msgr2=0x7f23240aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 0x7f23240aefa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f231c009870 tx=0x7f231c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 shutdown_connections 2023-10-16T14:16:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f231006ae70 0x7f231006d330 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23241618b0 0x7f23240aefa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f232415d330 0x7f23240aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 --2- 172.21.15.29:0/2151427048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2324065ff0 0x7f2324156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.267+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 >> 172.21.15.29:0/2151427048 conn(0x7f23240d0ab0 msgr2=0x7f232415c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.268+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 shutdown_connections 2023-10-16T14:16:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:47.268+0000 7f232a555640 1 -- 172.21.15.29:0/2151427048 wait complete. 2023-10-16T14:16:47.297 INFO:teuthology.run_tasks:Running task kclient... 2023-10-16T14:16:47.320 INFO:tasks.kclient:Mounting kernel clients... 2023-10-16T14:16:47.320 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-16T14:16:47.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.321 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:16:47.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-16T14:16:47.489 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-16T14:16:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:16:47.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.494 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:16:47.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-16T14:16:47.660 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-16T14:16:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:16:47.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.665 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2023-10-16T14:16:47.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.726 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link delete ceph-brx 2023-10-16T14:16:47.827 INFO:teuthology.orchestra.run.smithi170.stderr:Cannot find device "ceph-brx" 2023-10-16T14:16:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:16:47.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.831 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2023-10-16T14:16:47.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:47.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link delete ceph-brx 2023-10-16T14:16:47.978 INFO:teuthology.orchestra.run.smithi170.stderr:Cannot find device "ceph-brx" 2023-10-16T14:16:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:16:47.982 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-16T14:16:47.983 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T14:16:47.983 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:16:47.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:16:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.215+0000 7f14f759e640 1 Processor -- start 2023-10-16T14:16:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.215+0000 7f14f759e640 1 -- start start 2023-10-16T14:16:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.216+0000 7f14f759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14f80cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.216+0000 7f14f759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f80b53b0 con 0x7f14f80d1f80 2023-10-16T14:16:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f8160b60 con 0x7f14f80cdae0 2023-10-16T14:16:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81617e0 con 0x7f14f80cf2b0 2023-10-16T14:16:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f14f80d1f80 0x7f14f80b4ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38980/0 (socket says 172.21.15.29:38980) 2023-10-16T14:16:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f6d9d640 1 -- 172.21.15.29:0/1940498096 learned_addr learned my addr 172.21.15.29:0/1940498096 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f659c640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14f80cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28992551 0 0) 0x7f14f80b53b0 con 0x7f14f80d1f80 2023-10-16T14:16:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.217+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e4003520 con 0x7f14f80d1f80 2023-10-16T14:16:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156287223 0 0) 0x7f14e4003520 con 0x7f14f80d1f80 2023-10-16T14:16:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 msgr2=0x7f14f80cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14f80cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 >> v1:172.21.15.170:6789/0 conn(0x7f14f80cdae0 legacy=0x7f14f80b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8160e40 con 0x7f14f80d1f80 2023-10-16T14:16:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f14f0001670 con 0x7f14f80d1f80 2023-10-16T14:16:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 >> v1:172.21.15.29:6789/0 conn(0x7f14f80d1f80 legacy=0x7f14f80b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14e40039a0 0x7f14e4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14e4004d00 0x7f14e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81617e0 con 0x7f14e40039a0 2023-10-16T14:16:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e400ba30 con 0x7f14e4004d00 2023-10-16T14:16:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e400bbf0 con 0x7f14f80cf2b0 2023-10-16T14:16:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.218+0000 7f14f659c640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14e40039a0 0x7f14e4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f6d9d640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14e4004d00 0x7f14e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f5d9b640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f659c640 1 -- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 msgr2=0x7f14e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f659c640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14e40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f659c640 1 -- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14e4004d00 msgr2=0x7f14e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f659c640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14e4004d00 0x7f14e4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f659c640 1 -- 172.21.15.29:0/1940498096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e4003520 con 0x7f14e40039a0 2023-10-16T14:16:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f5d9b640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14f80cf2b0 0x7f14e40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:48.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.219+0000 7f14f6d9d640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14e4004d00 0x7f14e4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.220+0000 7f14f659c640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14e40039a0 0x7f14e4003d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009fc0 tx=0x7f14e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f9309e71ec9757 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e802cc20 con 0x7f14e40039a0 2023-10-16T14:16:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e800f040 con 0x7f14e40039a0 2023-10-16T14:16:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f559a640 1 -- 172.21.15.29:0/1940498096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8034440 con 0x7f14e40039a0 2023-10-16T14:16:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f759e640 1 -- 172.21.15.29:0/1940498096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14e40039a0 msgr2=0x7f14e4003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f759e640 1 --2- 172.21.15.29:0/1940498096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14e40039a0 0x7f14e4003d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009fc0 tx=0x7f14e40057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f6d9d640 1 -- 172.21.15.29:0/1940498096 reap_dead start 2023-10-16T14:16:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f759e640 1 -- 172.21.15.29:0/1940498096 shutdown_connections 2023-10-16T14:16:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.221+0000 7f14f759e640 1 -- 172.21.15.29:0/1940498096 >> 172.21.15.29:0/1940498096 conn(0x7f14f806bc60 msgr2=0x7f14f806c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.222+0000 7f14f759e640 1 -- 172.21.15.29:0/1940498096 shutdown_connections 2023-10-16T14:16:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.222+0000 7f14f759e640 1 -- 172.21.15.29:0/1940498096 wait complete. 2023-10-16T14:16:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.224+0000 7f14f759e640 1 Processor -- start 2023-10-16T14:16:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.224+0000 7f14f759e640 1 -- start start 2023-10-16T14:16:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14e40039a0 0x7f14f8152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 0x7f14f80c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14f80cb640 0x7f14f80cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80cc000 con 0x7f14f80c9040 2023-10-16T14:16:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81500d0 con 0x7f14f80cb640 2023-10-16T14:16:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80c8120 con 0x7f14e40039a0 2023-10-16T14:16:48.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 0x7f14f80c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 0x7f14f80c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46268/0 (socket says 172.21.15.29:46268) 2023-10-16T14:16:48.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 -- 172.21.15.29:0/3745125298 learned_addr learned my addr 172.21.15.29:0/3745125298 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:48.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f659c640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14e40039a0 0x7f14f8152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f6d9d640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14f80cb640 0x7f14f80cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 -- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14e40039a0 msgr2=0x7f14f8152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14e40039a0 0x7f14f8152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 -- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14f80cb640 msgr2=0x7f14f80cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14f80cb640 0x7f14f80cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.225+0000 7f14f5d9b640 1 -- 172.21.15.29:0/3745125298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8150390 con 0x7f14f80c9040 2023-10-16T14:16:48.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14f659c640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14e40039a0 0x7f14f8152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:48.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14f6d9d640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14f80cb640 0x7f14f80cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:48.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14f5d9b640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 0x7f14f80c7050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14e0007980 tx=0x7f14e000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:48.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e00050d0 con 0x7f14f80c9040 2023-10-16T14:16:48.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e0005230 con 0x7f14f80c9040 2023-10-16T14:16:48.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e000c660 con 0x7f14f80c9040 2023-10-16T14:16:48.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f8152980 con 0x7f14f80c9040 2023-10-16T14:16:48.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.226+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f80c7b60 con 0x7f14f80c9040 2023-10-16T14:16:48.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.227+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f815e2f0 con 0x7f14f80c9040 2023-10-16T14:16:48.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.228+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f14e000c7c0 con 0x7f14f80c9040 2023-10-16T14:16:48.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.229+0000 7f14eeffd640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f14d806ae70 0x7f14d806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:48.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.229+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14e00894f0 con 0x7f14f80c9040 2023-10-16T14:16:48.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.230+0000 7f14f659c640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f14d806ae70 0x7f14d806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:48.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.231+0000 7f14f659c640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f14d806ae70 0x7f14d806d330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f14e8036000 tx=0x7f14e802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:48.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.233+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f14e0054590 con 0x7f14f80c9040 2023-10-16T14:16:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.474+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f14f804e9b0 con 0x7f14f80c9040 2023-10-16T14:16:48.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.475+0000 7f14eeffd640 1 -- 172.21.15.29:0/3745125298 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f14e00012d0 con 0x7f14f80c9040 2023-10-16T14:16:48.480 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:16:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f14d806ae70 msgr2=0x7f14d806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f14d806ae70 0x7f14d806d330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f14e8036000 tx=0x7f14e802e000 comp rx=0 tx=0).stop 2023-10-16T14:16:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 msgr2=0x7f14f80c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 0x7f14f80c7050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14e0007980 tx=0x7f14e000ec60 comp rx=0 tx=0).stop 2023-10-16T14:16:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 shutdown_connections 2023-10-16T14:16:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f14d806ae70 0x7f14d806d330 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f80c9040 0x7f14f80c7050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f14f80cb640 0x7f14f80cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 --2- 172.21.15.29:0/3745125298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f14e40039a0 0x7f14f8152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 >> 172.21.15.29:0/3745125298 conn(0x7f14f806bc60 msgr2=0x7f14f8159c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 shutdown_connections 2023-10-16T14:16:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:48.478+0000 7f14f759e640 1 -- 172.21.15.29:0/3745125298 wait complete. 2023-10-16T14:16:48.500 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T14:16:48.501 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:16:48.501 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:16:48.501 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:16:48.501 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:16:48.502 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T14:16:48.502 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T14:16:48.502 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:16:48.502 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:16:48.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:48.503 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:16:48.554 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:16:48.555 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:16:48.555 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:16:48.555 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:48.555 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:16:48.555 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:48.556 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:48.556 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:48.556 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:16:48.556 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:48.556 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:48.556 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:16:48.557 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:16:48.557 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:48.557 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:16:48.557 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:16:48.557 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42316sec preferred_lft 42316sec 2023-10-16T14:16:48.557 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:16:48.558 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:48.558 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:48.558 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:48.558 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:16:48.559 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T14:16:48.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:48.560 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:48.560 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:48.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2023-10-16T14:16:48.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2023-10-16T14:16:48.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2023-10-16T14:16:48.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T14:16:48.562 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:49.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:49.143 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:16:49.229 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:16:49.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:49.233 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:16:49.285 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:16:49.285 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:16:49.286 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:16:49.286 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:16:49.286 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:16:49.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:49.287 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:49.287 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:49.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:16:49.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:16:49.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:16:49.289 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:50.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:50.323 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:16:50.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:50.383 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2023-10-16T14:16:50.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:50.433 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:50.433 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:50.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:16:50.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-16T14:16:50.434 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:50.571 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-16T14:16:50.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:50.571 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:50.572 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:50.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-16T14:16:50.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T14:16:50.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-16T14:16:50.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-16T14:16:50.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-16T14:16:50.573 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:50.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:50.862 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:50.863 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:50.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:16:50.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T14:16:50.864 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:51.014 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:16:51.014 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:16:51.054 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:16:51.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:16:51.054 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:16:51.150 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:16:51.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:51.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:16:51.292 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:16:51.293 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:16:51.293 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:16:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:16:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:16:51.994 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:16:51.994 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:16:51.994 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:16:52.026 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:16:52.026 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:16:52.027 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:16:52.027 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:16:52.027 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:16:52.027 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:16:52.027 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:16:52.027 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:16:52.028 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:16:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:16:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:16:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:16:52.029 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:16:52.034 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:16:52.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:52.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:16:52.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:52.134 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:52.217 INFO:teuthology.orchestra.run.smithi029.stdout:client4418 2023-10-16T14:16:52.218 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:16:52.218 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:16:52.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:52.282 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:52.365 INFO:teuthology.orchestra.run.smithi029.stdout:client4418 2023-10-16T14:16:52.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:52.366 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:52.454 INFO:teuthology.orchestra.run.smithi029.stdout:client4418 2023-10-16T14:16:52.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:52.455 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/aff38114-d596-4938-a779-5e1b7478d87b.client4418/status') 2023-10-16T14:16:52.576 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.4418 (3)172.21.15.29:0/4204804748 2023-10-16T14:16:52.576 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2023-10-16T14:16:52.576 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2023-10-16T14:16:52.576 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2023-10-16T14:16:52.577 INFO:teuthology.orchestra.run.smithi029.stderr:70 bytes copied, 8.3526e-05 s, 838 kB/s 2023-10-16T14:16:52.580 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-10-16T14:16:52.581 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T14:16:52.581 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T14:16:52.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:16:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.805+0000 7f9350797640 1 Processor -- start 2023-10-16T14:16:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.806+0000 7f9350797640 1 -- start start 2023-10-16T14:16:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.807+0000 7f9350797640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 0x7f934815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.807+0000 7f9350797640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934815ed60 con 0x7f93480ce390 2023-10-16T14:16:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.807+0000 7f9350797640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480707e0 con 0x7f934815aaa0 2023-10-16T14:16:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.807+0000 7f9350797640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934806b810 con 0x7f934815ae80 2023-10-16T14:16:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934ed34640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f93480ce390 0x7f9348070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38990/0 (socket says 172.21.15.29:38990) 2023-10-16T14:16:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934ed34640 1 -- 172.21.15.29:0/669945827 learned_addr learned my addr 172.21.15.29:0/669945827 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934dd32640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 0x7f934815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912329634 0 0) 0x7f934815ed60 con 0x7f93480ce390 2023-10-16T14:16:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334003520 con 0x7f93480ce390 2023-10-16T14:16:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410296252 0 0) 0x7f9334003520 con 0x7f93480ce390 2023-10-16T14:16:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 msgr2=0x7f934815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 0x7f934815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 >> v1:172.21.15.170:6789/0 conn(0x7f934815aaa0 legacy=0x7f93480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934815ef90 con 0x7f93480ce390 2023-10-16T14:16:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.808+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9340001670 con 0x7f93480ce390 2023-10-16T14:16:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 >> v1:172.21.15.29:6789/0 conn(0x7f93480ce390 legacy=0x7f9348070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 0x7f93340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9334003f20 0x7f93340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9334004d00 0x7f9334005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934806b810 con 0x7f9334004d00 2023-10-16T14:16:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933400ba60 con 0x7f9334003f20 2023-10-16T14:16:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933400bc20 con 0x7f934815ae80 2023-10-16T14:16:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934ed34640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9334004d00 0x7f9334005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934dd32640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9334003f20 0x7f93340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934e533640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 0x7f93340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934ed34640 1 -- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 msgr2=0x7f93340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934ed34640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934815ae80 0x7f93340039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934ed34640 1 -- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9334003f20 msgr2=0x7f93340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934ed34640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9334003f20 0x7f93340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.809+0000 7f934ed34640 1 -- 172.21.15.29:0/669945827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334003520 con 0x7f9334004d00 2023-10-16T14:16:52.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f934ed34640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9334004d00 0x7f9334005180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f934000c240 tx=0x7f93340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88569663b7febaa7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:52.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340005ad0 con 0x7f9334004d00 2023-10-16T14:16:52.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9340005c30 con 0x7f9334004d00 2023-10-16T14:16:52.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f934d531640 1 -- 172.21.15.29:0/669945827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934002caf0 con 0x7f9334004d00 2023-10-16T14:16:52.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f9350797640 1 -- 172.21.15.29:0/669945827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9334004d00 msgr2=0x7f9334005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f9350797640 1 --2- 172.21.15.29:0/669945827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9334004d00 0x7f9334005180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f934000c240 tx=0x7f93340057f0 comp rx=0 tx=0).stop 2023-10-16T14:16:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f934ed34640 1 -- 172.21.15.29:0/669945827 reap_dead start 2023-10-16T14:16:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f9350797640 1 -- 172.21.15.29:0/669945827 shutdown_connections 2023-10-16T14:16:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.811+0000 7f9350797640 1 -- 172.21.15.29:0/669945827 >> 172.21.15.29:0/669945827 conn(0x7f93480cb410 msgr2=0x7f93480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.812+0000 7f9350797640 1 -- 172.21.15.29:0/669945827 shutdown_connections 2023-10-16T14:16:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.812+0000 7f9350797640 1 -- 172.21.15.29:0/669945827 wait complete. 2023-10-16T14:16:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.814+0000 7f9350797640 1 Processor -- start 2023-10-16T14:16:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.814+0000 7f9350797640 1 -- start start 2023-10-16T14:16:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f9350797640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9334004d00 0x7f934815be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f9350797640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 0x7f9348156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f9350797640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9348156e90 0x7f934815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f9350797640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9348157310 con 0x7f93481ee200 2023-10-16T14:16:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f9350797640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c3180 con 0x7f9348156e90 2023-10-16T14:16:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f9350797640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c3460 con 0x7f9334004d00 2023-10-16T14:16:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 0x7f9348156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 0x7f9348156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46282/0 (socket says 172.21.15.29:46282) 2023-10-16T14:16:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 -- 172.21.15.29:0/326190851 learned_addr learned my addr 172.21.15.29:0/326190851 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934e533640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9334004d00 0x7f934815be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934ed34640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9348156e90 0x7f934815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 -- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9334004d00 msgr2=0x7f934815be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9334004d00 0x7f934815be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 -- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9348156e90 msgr2=0x7f934815d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9348156e90 0x7f934815d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934dd32640 1 -- 172.21.15.29:0/326190851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93480c3740 con 0x7f93481ee200 2023-10-16T14:16:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934e533640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9334004d00 0x7f934815be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.815+0000 7f934ed34640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9348156e90 0x7f934815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.816+0000 7f934dd32640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 0x7f9348156950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9338011600 tx=0x7f9338011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.816+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933800f920 con 0x7f93481ee200 2023-10-16T14:16:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.816+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933800fa80 con 0x7f93481ee200 2023-10-16T14:16:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.816+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933800fdf0 con 0x7f93481ee200 2023-10-16T14:16:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.816+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93480c3a20 con 0x7f93481ee200 2023-10-16T14:16:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.816+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934815c340 con 0x7f93481ee200 2023-10-16T14:16:52.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.818+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934804e160 con 0x7f93481ee200 2023-10-16T14:16:52.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.818+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9338015460 con 0x7f93481ee200 2023-10-16T14:16:52.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.819+0000 7f933effd640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f932c06ae70 0x7f932c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:52.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.819+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9338089a90 con 0x7f93481ee200 2023-10-16T14:16:52.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.820+0000 7f934e533640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f932c06ae70 0x7f932c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:52.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.821+0000 7f934e533640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f932c06ae70 0x7f932c06d330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9344001d60 tx=0x7f9344009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:52.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:52.824+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9338054bd0 con 0x7f93481ee200 2023-10-16T14:16:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.063+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f93480cdc60 con 0x7f93481ee200 2023-10-16T14:16:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.064+0000 7f933effd640 1 -- 172.21.15.29:0/326190851 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9338022070 con 0x7f93481ee200 2023-10-16T14:16:53.068 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:16:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f932c06ae70 msgr2=0x7f932c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f932c06ae70 0x7f932c06d330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9344001d60 tx=0x7f9344009040 comp rx=0 tx=0).stop 2023-10-16T14:16:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 msgr2=0x7f9348156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 0x7f9348156950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9338011600 tx=0x7f9338011ac0 comp rx=0 tx=0).stop 2023-10-16T14:16:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 shutdown_connections 2023-10-16T14:16:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f932c06ae70 0x7f932c06d330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93481ee200 0x7f9348156950 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9348156e90 0x7f934815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 --2- 172.21.15.29:0/326190851 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9334004d00 0x7f934815be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:53.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 >> 172.21.15.29:0/326190851 conn(0x7f93480cb410 msgr2=0x7f9348070020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:53.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 shutdown_connections 2023-10-16T14:16:53.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:53.067+0000 7f9350797640 1 -- 172.21.15.29:0/326190851 wait complete. 2023-10-16T14:16:53.088 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T14:16:53.089 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:16:53.089 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:16:53.089 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-16T14:16:53.090 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T14:16:53.090 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T14:16:53.090 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T14:16:53.090 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:16:53.091 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-16T14:16:53.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.091 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:16:53.145 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:16:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:16:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:16:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:16:53.146 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.147 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:16:53.147 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:53.147 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:16:53.148 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:16:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:16:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:16:53.148 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42312sec preferred_lft 42312sec 2023-10-16T14:16:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:16:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.149 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:53.149 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:16:53.150 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:16:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:16:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.150 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:16:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.151 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:16:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:16:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:16:53.151 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.153 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:16:53.247 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:16:53.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.251 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:16:53.308 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:16:53.308 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:16:53.309 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:16:53.309 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:16:53.309 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:16:53.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.310 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:53.310 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:53.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:16:53.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:16:53.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:16:53.311 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:53.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.514 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:16:53.568 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:16:53.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.569 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2023-10-16T14:16:53.623 INFO:teuthology.orchestra.run.smithi029.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-16T14:16:53.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.625 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:53.625 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:53.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:16:53.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-16T14:16:53.626 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:53.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T14:16:53.884 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:16:53.885 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:16:53.885 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:16:53.885 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.885 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:16:53.885 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.886 INFO:teuthology.orchestra.run.smithi029.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:16:53.886 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:c1:b3:4c:55:91 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T14:16:53.886 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T14:16:53.886 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.886 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcc1:b3ff:fe4c:5591/64 scope link 2023-10-16T14:16:53.887 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T14:16:53.991 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:16:53.991 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:16:53.992 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:16:53.992 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.992 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:16:53.992 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.992 INFO:teuthology.orchestra.run.smithi029.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether de:c1:b3:4c:55:91 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T14:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T14:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::dcc1:b3ff:fe4c:5591/64 scope link 2023-10-16T14:16:53.993 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:53.996 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-16T14:16:53.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:53.997 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:53.997 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:53.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-16T14:16:53.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T14:16:53.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-16T14:16:53.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-16T14:16:53.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-16T14:16:53.999 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:54.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:54.295 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:16:54.296 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:16:54.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.1 up 2023-10-16T14:16:54.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T14:16:54.296 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:16:54.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:16:54.480 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-16T14:16:54.531 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-16T14:16:54.532 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:16:54.532 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-16T14:16:54.586 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:16:54.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:54.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T14:16:54.736 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T14:16:54.737 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-16T14:16:54.737 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:16:54.759 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T14:16:54.759 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-16T14:16:54.760 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:16:54.760 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:16:54.760 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:16:54.789 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T14:16:54.789 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-16T14:16:54.789 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:16:54.789 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:16:54.790 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:16:54.790 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:16:54.790 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:16:54.790 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:16:54.791 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2023-10-16T14:16:54.791 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:16:54.792 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:16:54.792 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:16:54.792 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:16:54.796 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:16:54.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:54.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-16T14:16:54.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:54.896 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:54.961 INFO:teuthology.orchestra.run.smithi029.stdout:client4424 2023-10-16T14:16:54.961 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:16:54.961 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:16:55.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:55.018 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:55.079 INFO:teuthology.orchestra.run.smithi029.stdout:client4424 2023-10-16T14:16:55.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:55.080 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:55.139 INFO:teuthology.orchestra.run.smithi029.stdout:client4424 2023-10-16T14:16:55.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:55.140 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/aff38114-d596-4938-a779-5e1b7478d87b.client4424/status') 2023-10-16T14:16:55.242 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.4424 (3)172.21.15.29:0/91095246 2023-10-16T14:16:55.243 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2023-10-16T14:16:55.243 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2023-10-16T14:16:55.243 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2023-10-16T14:16:55.243 INFO:teuthology.orchestra.run.smithi029.stderr:68 bytes copied, 7.1163e-05 s, 956 kB/s 2023-10-16T14:16:55.247 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-10-16T14:16:55.247 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T14:16:55.247 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T14:16:55.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:16:55.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.476+0000 7fe94ffff640 1 Processor -- start 2023-10-16T14:16:55.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.476+0000 7fe94ffff640 1 -- start start 2023-10-16T14:16:55.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:55.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe950070340 0x7fe9500707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:55.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95006a380 con 0x7fe95015ccc0 2023-10-16T14:16:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94ffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95006a940 con 0x7fe950070340 2023-10-16T14:16:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95015e0b0 con 0x7fe950069da0 2023-10-16T14:16:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe95015ccc0 0x7fe950161100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38206/0 (socket says 172.21.15.29:38206) 2023-10-16T14:16:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94effd640 1 -- 172.21.15.29:0/1619587795 learned_addr learned my addr 172.21.15.29:0/1619587795 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:16:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe94f7fe640 1 --2- 172.21.15.29:0/1619587795 >> v2:172.21.15.29:3300/0 conn(0x7fe950070340 0x7fe9500707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.479+0000 7fe9467fc640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe94e7fc640 1 -- 172.21.15.29:0/1619587795 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029361711 0 0) 0x7fe95006a380 con 0x7fe95015ccc0 2023-10-16T14:16:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe94e7fc640 1 -- 172.21.15.29:0/1619587795 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe93c003520 con 0x7fe95015ccc0 2023-10-16T14:16:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1619587795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 msgr2=0x7fe950071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe94f7fe640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950071010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1619587795 >> v1:172.21.15.29:6789/0 conn(0x7fe95015ccc0 legacy=0x7fe950161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1619587795 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95006a660 con 0x7fe950070340 2023-10-16T14:16:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.480+0000 7fe9467fc640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.481+0000 7fe94f7fe640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe950070340 0x7fe9500707d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe948009d90 tx=0x7fe94802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1900460861ffaf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.481+0000 7fe94e7fc640 1 -- 172.21.15.29:0/1619587795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94802cce0 con 0x7fe950070340 2023-10-16T14:16:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.481+0000 7fe94e7fc640 1 -- 172.21.15.29:0/1619587795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94802ce40 con 0x7fe950070340 2023-10-16T14:16:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.481+0000 7fe94e7fc640 1 -- 172.21.15.29:0/1619587795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948034660 con 0x7fe950070340 2023-10-16T14:16:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 -- 172.21.15.29:0/1619587795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe950070340 msgr2=0x7fe9500707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:55.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe950070340 0x7fe9500707d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe948009d90 tx=0x7fe94802bf30 comp rx=0 tx=0).stop 2023-10-16T14:16:55.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 -- 172.21.15.29:0/1619587795 shutdown_connections 2023-10-16T14:16:55.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe950070340 0x7fe9500707d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1619587795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 -- 172.21.15.29:0/1619587795 >> 172.21.15.29:0/1619587795 conn(0x7fe95006aeb0 msgr2=0x7fe95006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:55.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 -- 172.21.15.29:0/1619587795 shutdown_connections 2023-10-16T14:16:55.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.482+0000 7fe94ffff640 1 -- 172.21.15.29:0/1619587795 wait complete. 2023-10-16T14:16:55.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.484+0000 7fe94ffff640 1 Processor -- start 2023-10-16T14:16:55.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.484+0000 7fe94ffff640 1 -- start start 2023-10-16T14:16:55.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:55.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe950070340 0x7fe9501535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:55.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 0x7fe9500b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95015a0f0 con 0x7fe95015ccc0 2023-10-16T14:16:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95015a3d0 con 0x7fe950070340 2023-10-16T14:16:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95015a6b0 con 0x7fe950069da0 2023-10-16T14:16:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 0x7fe9500b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 0x7fe9500b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57842/0 (socket says 172.21.15.29:57842) 2023-10-16T14:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1902867689 learned_addr learned my addr 172.21.15.29:0/1902867689 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe9467fc640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe950070340 0x7fe9501535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.485+0000 7fe94effd640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 msgr2=0x7fe950156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94f7fe640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe950070340 msgr2=0x7fe9501535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94f7fe640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe950070340 0x7fe9501535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94f7fe640 1 -- 172.21.15.29:0/1902867689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe948009a70 con 0x7fe95015ccc0 2023-10-16T14:16:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94f7fe640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 0x7fe9500b1630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe948009d90 tx=0x7fe948004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94803d600 con 0x7fe95015ccc0 2023-10-16T14:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94803d780 con 0x7fe95015ccc0 2023-10-16T14:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94803db30 con 0x7fe95015ccc0 2023-10-16T14:16:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.486+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95015a990 con 0x7fe95015ccc0 2023-10-16T14:16:55.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.487+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe950153b20 con 0x7fe95015ccc0 2023-10-16T14:16:55.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.488+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95004e9b0 con 0x7fe95015ccc0 2023-10-16T14:16:55.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.489+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe94803b040 con 0x7fe95015ccc0 2023-10-16T14:16:55.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.490+0000 7fe94cff9640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe93006aea0 0x7fe93006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:55.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.490+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9480ab8d0 con 0x7fe95015ccc0 2023-10-16T14:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.490+0000 7fe94effd640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe93006aea0 0x7fe93006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.491+0000 7fe94effd640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe93006aea0 0x7fe93006d360 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe9400011e0 tx=0x7fe940009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.495+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe9480768e0 con 0x7fe95015ccc0 2023-10-16T14:16:55.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.725+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe9500cc9f0 con 0x7fe95015ccc0 2023-10-16T14:16:55.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.726+0000 7fe94cff9640 1 -- 172.21.15.29:0/1902867689 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe948039070 con 0x7fe95015ccc0 2023-10-16T14:16:55.729 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:16:55.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe93006aea0 msgr2=0x7fe93006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:55.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe93006aea0 0x7fe93006d360 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe9400011e0 tx=0x7fe940009040 comp rx=0 tx=0).stop 2023-10-16T14:16:55.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 msgr2=0x7fe9500b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:55.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 0x7fe9500b1630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe948009d90 tx=0x7fe948004a40 comp rx=0 tx=0).stop 2023-10-16T14:16:55.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 shutdown_connections 2023-10-16T14:16:55.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fe93006aea0 0x7fe93006d360 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe95015ccc0 0x7fe9500b1630 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe950070340 0x7fe9501535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 --2- 172.21.15.29:0/1902867689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe950069da0 0x7fe950156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:55.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 >> 172.21.15.29:0/1902867689 conn(0x7fe95006aeb0 msgr2=0x7fe950156270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:55.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 shutdown_connections 2023-10-16T14:16:55.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:55.729+0000 7fe94ffff640 1 -- 172.21.15.29:0/1902867689 wait complete. 2023-10-16T14:16:55.752 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T14:16:55.752 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:16:55.752 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi170.front.sepia.ceph.com 2023-10-16T14:16:55.753 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-16T14:16:55.753 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T14:16:55.753 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T14:16:55.753 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T14:16:55.754 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:16:55.754 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-16T14:16:55.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:55.755 DEBUG:teuthology.orchestra.run.smithi170:> ip addr 2023-10-16T14:16:55.813 INFO:teuthology.orchestra.run.smithi170.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:16:55.813 INFO:teuthology.orchestra.run.smithi170.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:16:55.814 INFO:teuthology.orchestra.run.smithi170.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:16:55.814 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:55.814 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 ::1/128 scope host 2023-10-16T14:16:55.814 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:55.814 INFO:teuthology.orchestra.run.smithi170.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:55.815 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:d9:3b:54 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:55.815 INFO:teuthology.orchestra.run.smithi170.stdout: altname enp7s0f0 2023-10-16T14:16:55.815 INFO:teuthology.orchestra.run.smithi170.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:55.815 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:d9:3b:55 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:55.816 INFO:teuthology.orchestra.run.smithi170.stdout: altname enp7s0f1 2023-10-16T14:16:55.816 INFO:teuthology.orchestra.run.smithi170.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:55.816 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:88:72:54 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:55.816 INFO:teuthology.orchestra.run.smithi170.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:16:55.816 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:88:72:55 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:55.816 INFO:teuthology.orchestra.run.smithi170.stdout: inet 172.21.15.170/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T14:16:55.817 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft 42277sec preferred_lft 42277sec 2023-10-16T14:16:55.817 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::ec4:7aff:fe88:7255/64 scope link 2023-10-16T14:16:55.817 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:55.818 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T14:16:55.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:55.819 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:16:55.819 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:16:55.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link add name ceph-brx type bridge 2023-10-16T14:16:55.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip addr flush dev ceph-brx 2023-10-16T14:16:55.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set ceph-brx up 2023-10-16T14:16:55.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T14:16:55.821 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:16:56.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:56.293 DEBUG:teuthology.orchestra.run.smithi170:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:16:56.375 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-10-16T14:16:56.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:56.379 DEBUG:teuthology.orchestra.run.smithi170:> route 2023-10-16T14:16:56.438 INFO:teuthology.orchestra.run.smithi170.stdout:Kernel IP routing table 2023-10-16T14:16:56.438 INFO:teuthology.orchestra.run.smithi170.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:16:56.438 INFO:teuthology.orchestra.run.smithi170.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T14:16:56.438 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T14:16:56.439 INFO:teuthology.orchestra.run.smithi170.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:16:56.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:56.440 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:16:56.440 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:16:56.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T14:16:56.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T14:16:56.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T14:16:56.441 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:16:57.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:57.212 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2023-10-16T14:16:57.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:57.272 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list-id 2023-10-16T14:16:57.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:57.328 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:16:57.328 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:16:57.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T14:16:57.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-16T14:16:57.329 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:16:57.468 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-16T14:16:57.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:57.468 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:16:57.469 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:16:57.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-16T14:16:57.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T14:16:57.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-16T14:16:57.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-16T14:16:57.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-16T14:16:57.470 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:16:57.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:57.745 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:16:57.745 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:16:57.745 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set brx.0 up 2023-10-16T14:16:57.745 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T14:16:57.746 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:16:57.903 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:16:57.904 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-16T14:16:57.950 INFO:teuthology.orchestra.run.smithi170.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-16T14:16:57.951 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:16:57.951 DEBUG:teuthology.orchestra.run.smithi170:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-16T14:16:58.045 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:16:58.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:58.046 DEBUG:teuthology.orchestra.run.smithi170:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T14:16:58.191 INFO:teuthology.orchestra.run.smithi170.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T14:16:58.191 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-16T14:16:58.191 INFO:teuthology.orchestra.run.smithi170.stdout:invalid new device string format 2023-10-16T14:16:58.648 INFO:teuthology.orchestra.run.smithi170.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T14:16:58.648 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-16T14:16:58.649 INFO:teuthology.orchestra.run.smithi170.stdout:invalid new device string format 2023-10-16T14:16:58.649 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:16:58.649 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:16:58.680 INFO:teuthology.orchestra.run.smithi170.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T14:16:58.680 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-16T14:16:58.680 INFO:teuthology.orchestra.run.smithi170.stdout:invalid new device string format 2023-10-16T14:16:58.681 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:16:58.681 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:16:58.681 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:16:58.681 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:16:58.681 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:16:58.684 INFO:teuthology.orchestra.run.smithi170.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-10-16T14:16:58.684 INFO:teuthology.orchestra.run.smithi170.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:16:58.684 INFO:teuthology.orchestra.run.smithi170.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:16:58.685 INFO:teuthology.orchestra.run.smithi170.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:16:58.685 INFO:teuthology.orchestra.run.smithi170.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:16:58.688 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:16:58.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:58.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-16T14:16:58.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:58.774 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:58.826 INFO:teuthology.orchestra.run.smithi170.stdout:client4632 2023-10-16T14:16:58.827 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:16:58.827 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:16:58.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:58.876 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:58.981 INFO:teuthology.orchestra.run.smithi170.stdout:client4632 2023-10-16T14:16:58.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:58.982 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:16:59.040 INFO:teuthology.orchestra.run.smithi170.stdout:client4632 2023-10-16T14:16:59.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:59.041 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/aff38114-d596-4938-a779-5e1b7478d87b.client4632/status') 2023-10-16T14:16:59.136 INFO:teuthology.orchestra.run.smithi170.stdout:instance: client.4632 (3)172.21.15.170:0/1826181454 2023-10-16T14:16:59.136 INFO:teuthology.orchestra.run.smithi170.stdout:blocklisted: false 2023-10-16T14:16:59.137 INFO:teuthology.orchestra.run.smithi170.stderr:0+1 records in 2023-10-16T14:16:59.137 INFO:teuthology.orchestra.run.smithi170.stderr:0+1 records out 2023-10-16T14:16:59.137 INFO:teuthology.orchestra.run.smithi170.stderr:71 bytes copied, 6.5986e-05 s, 1.1 MB/s 2023-10-16T14:16:59.140 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-10-16T14:16:59.140 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T14:16:59.140 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T14:16:59.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:16:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.376+0000 7fc7dfb39640 1 Processor -- start 2023-10-16T14:16:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.377+0000 7fc7dfb39640 1 -- start start 2023-10-16T14:16:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.378+0000 7fc7dfb39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc7d806b8c0 0x7fc7d80cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.378+0000 7fc7dfb39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7d806bca0 0x7fc7d80b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.378+0000 7fc7dfb39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d80cf940 con 0x7fc7d80b09c0 2023-10-16T14:16:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.378+0000 7fc7dfb39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d80ce990 con 0x7fc7d806bca0 2023-10-16T14:16:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.378+0000 7fc7dfb39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d80d0310 con 0x7fc7d806b8c0 2023-10-16T14:16:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7d806bca0 0x7fc7d80b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7d806bca0 0x7fc7d80b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57844/0 (socket says 172.21.15.29:57844) 2023-10-16T14:16:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1031035556 learned_addr learned my addr 172.21.15.29:0/1031035556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd8d5640 1 --2- 172.21.15.29:0/1031035556 >> v2:172.21.15.170:3300/0 conn(0x7fc7d806b8c0 0x7fc7d80cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1031035556 >> v2:172.21.15.170:3300/0 conn(0x7fc7d806b8c0 msgr2=0x7fc7d80cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 --2- 172.21.15.29:0/1031035556 >> v2:172.21.15.170:3300/0 conn(0x7fc7d806b8c0 0x7fc7d80cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1031035556 >> v1:172.21.15.170:6789/0 conn(0x7fc7d80b09c0 legacy=0x7fc7d8160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:16:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1031035556 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d815e290 con 0x7fc7d806bca0 2023-10-16T14:16:59.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.379+0000 7fc7dd8d5640 1 --2- 172.21.15.29:0/1031035556 >> v2:172.21.15.170:3300/0 conn(0x7fc7d806b8c0 0x7fc7d80cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:59.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.380+0000 7fc7dd0d4640 1 --2- 172.21.15.29:0/1031035556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80b0480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc7d80711f0 tx=0x7fc7c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d13dc8391acffb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:59.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.381+0000 7fc7dc8d3640 1 -- 172.21.15.29:0/1031035556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8031870 con 0x7fc7d806bca0 2023-10-16T14:16:59.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.381+0000 7fc7dc8d3640 1 -- 172.21.15.29:0/1031035556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c80319d0 con 0x7fc7d806bca0 2023-10-16T14:16:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.381+0000 7fc7dc8d3640 1 -- 172.21.15.29:0/1031035556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8031d60 con 0x7fc7d806bca0 2023-10-16T14:16:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.381+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1031035556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 msgr2=0x7fc7d80b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.381+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1031035556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80b0480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc7d80711f0 tx=0x7fc7c802b910 comp rx=0 tx=0).stop 2023-10-16T14:16:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.382+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1031035556 shutdown_connections 2023-10-16T14:16:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.382+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1031035556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80b0480 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.382+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1031035556 >> v2:172.21.15.170:3300/0 conn(0x7fc7d806b8c0 0x7fc7d80cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.382+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1031035556 >> 172.21.15.29:0/1031035556 conn(0x7fc7d8099ea0 msgr2=0x7fc7d80d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.382+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1031035556 shutdown_connections 2023-10-16T14:16:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.382+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1031035556 wait complete. 2023-10-16T14:16:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.384+0000 7fc7dfb39640 1 Processor -- start 2023-10-16T14:16:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.384+0000 7fc7dfb39640 1 -- start start 2023-10-16T14:16:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.385+0000 7fc7dfb39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7d806b8c0 0x7fc7d80c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.385+0000 7fc7dfb39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.385+0000 7fc7dfb39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7d80b09c0 0x7fc7d806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.385+0000 7fc7dfb39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80c4db0 con 0x7fc7d806bca0 2023-10-16T14:16:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.385+0000 7fc7dfb39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80cbe10 con 0x7fc7d806b8c0 2023-10-16T14:16:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.385+0000 7fc7dfb39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81f3390 con 0x7fc7d80b09c0 2023-10-16T14:16:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57860/0 (socket says 172.21.15.29:57860) 2023-10-16T14:16:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1948714103 learned_addr learned my addr 172.21.15.29:0/1948714103 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:16:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd8d5640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7d806b8c0 0x7fc7d80c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7de0d6640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7d80b09c0 0x7fc7d806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7d80b09c0 msgr2=0x7fc7d806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7d80b09c0 0x7fc7d806eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7d806b8c0 msgr2=0x7fc7d80c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7d806b8c0 0x7fc7d80c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd0d4640 1 -- 172.21.15.29:0/1948714103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c80094d0 con 0x7fc7d806bca0 2023-10-16T14:16:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7dd8d5640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7d806b8c0 0x7fc7d80c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:16:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.386+0000 7fc7de0d6640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7d80b09c0 0x7fc7d806eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:16:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.387+0000 7fc7dd0d4640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80c6da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.387+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8031840 con 0x7fc7d806bca0 2023-10-16T14:16:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.387+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c80443f0 con 0x7fc7d806bca0 2023-10-16T14:16:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.387+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80447a0 con 0x7fc7d806bca0 2023-10-16T14:16:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.387+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d81f3670 con 0x7fc7d806bca0 2023-10-16T14:16:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.387+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d80c49d0 con 0x7fc7d806bca0 2023-10-16T14:16:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.388+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d804e160 con 0x7fc7d806bca0 2023-10-16T14:16:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.389+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc7c804d450 con 0x7fc7d806bca0 2023-10-16T14:16:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.391+0000 7fc7ce7fc640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc7bc06ae70 0x7fc7bc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:16:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.391+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7c802d040 con 0x7fc7d806bca0 2023-10-16T14:16:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.392+0000 7fc7dd8d5640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc7bc06ae70 0x7fc7bc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:16:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.393+0000 7fc7dd8d5640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc7bc06ae70 0x7fc7bc06d330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400aa20 tx=0x7fc7d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:16:59.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.395+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7c8076ec0 con 0x7fc7d806bca0 2023-10-16T14:16:59.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.628+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc7d80ce240 con 0x7fc7d806bca0 2023-10-16T14:16:59.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.629+0000 7fc7ce7fc640 1 -- 172.21.15.29:0/1948714103 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc7c8035070 con 0x7fc7d806bca0 2023-10-16T14:16:59.633 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:16:59.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc7bc06ae70 msgr2=0x7fc7bc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:59.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc7bc06ae70 0x7fc7bc06d330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400aa20 tx=0x7fc7d4008040 comp rx=0 tx=0).stop 2023-10-16T14:16:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 msgr2=0x7fc7d80c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:16:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80c6da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c80046c0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 shutdown_connections 2023-10-16T14:16:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc7bc06ae70 0x7fc7bc06d330 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d806bca0 0x7fc7d80c6da0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7d80b09c0 0x7fc7d806eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 --2- 172.21.15.29:0/1948714103 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7d806b8c0 0x7fc7d80c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:16:59.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 >> 172.21.15.29:0/1948714103 conn(0x7fc7d8099ea0 msgr2=0x7fc7d806d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:16:59.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 shutdown_connections 2023-10-16T14:16:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:16:59.632+0000 7fc7dfb39640 1 -- 172.21.15.29:0/1948714103 wait complete. 2023-10-16T14:16:59.653 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T14:16:59.653 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:16:59.654 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi170.front.sepia.ceph.com 2023-10-16T14:16:59.654 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-16T14:16:59.654 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T14:16:59.655 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T14:16:59.655 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T14:16:59.655 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:16:59.655 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-16T14:16:59.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:59.656 DEBUG:teuthology.orchestra.run.smithi170:> ip addr 2023-10-16T14:16:59.717 INFO:teuthology.orchestra.run.smithi170.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:16:59.717 INFO:teuthology.orchestra.run.smithi170.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:16:59.717 INFO:teuthology.orchestra.run.smithi170.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:16:59.717 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:59.718 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 ::1/128 scope host 2023-10-16T14:16:59.718 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:59.718 INFO:teuthology.orchestra.run.smithi170.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:59.718 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:d9:3b:54 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:59.718 INFO:teuthology.orchestra.run.smithi170.stdout: altname enp7s0f0 2023-10-16T14:16:59.719 INFO:teuthology.orchestra.run.smithi170.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:59.719 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:d9:3b:55 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:59.719 INFO:teuthology.orchestra.run.smithi170.stdout: altname enp7s0f1 2023-10-16T14:16:59.719 INFO:teuthology.orchestra.run.smithi170.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:16:59.719 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:88:72:54 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:59.720 INFO:teuthology.orchestra.run.smithi170.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:16:59.720 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether 0c:c4:7a:88:72:55 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:59.720 INFO:teuthology.orchestra.run.smithi170.stdout: inet 172.21.15.170/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T14:16:59.720 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft 42273sec preferred_lft 42273sec 2023-10-16T14:16:59.720 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::ec4:7aff:fe88:7255/64 scope link 2023-10-16T14:16:59.721 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:59.721 INFO:teuthology.orchestra.run.smithi170.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:16:59.721 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether c2:c4:5a:d1:19:2d brd ff:ff:ff:ff:ff:ff 2023-10-16T14:16:59.721 INFO:teuthology.orchestra.run.smithi170.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:16:59.721 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:59.721 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::c0c:50ff:fef6:ab6a/64 scope link 2023-10-16T14:16:59.722 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:59.722 INFO:teuthology.orchestra.run.smithi170.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:16:59.722 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether c2:c4:5a:d1:19:2d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T14:16:59.722 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::c0c4:5aff:fed1:192d/64 scope link 2023-10-16T14:16:59.722 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:16:59.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:59.724 DEBUG:teuthology.orchestra.run.smithi170:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:16:59.817 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-10-16T14:16:59.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:59.822 DEBUG:teuthology.orchestra.run.smithi170:> route 2023-10-16T14:16:59.873 INFO:teuthology.orchestra.run.smithi170.stdout:Kernel IP routing table 2023-10-16T14:16:59.874 INFO:teuthology.orchestra.run.smithi170.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:16:59.874 INFO:teuthology.orchestra.run.smithi170.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T14:16:59.874 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T14:16:59.874 INFO:teuthology.orchestra.run.smithi170.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:16:59.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:16:59.875 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:16:59.875 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:16:59.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T14:16:59.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T14:16:59.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T14:16:59.876 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:17:00.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.057 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2023-10-16T14:17:00.107 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T14:17:00.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.108 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list-id 2023-10-16T14:17:00.160 INFO:teuthology.orchestra.run.smithi170.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-16T14:17:00.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.162 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:17:00.162 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:17:00.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T14:17:00.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-16T14:17:00.163 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:17:00.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T14:17:00.395 INFO:teuthology.orchestra.run.smithi170.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:17:00.395 INFO:teuthology.orchestra.run.smithi170.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:17:00.395 INFO:teuthology.orchestra.run.smithi170.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:17:00.396 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.396 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 ::1/128 scope host 2023-10-16T14:17:00.396 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.396 INFO:teuthology.orchestra.run.smithi170.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:17:00.396 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether ee:05:f0:5e:67:8e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T14:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T14:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::ec05:f0ff:fe5e:678e/64 scope link 2023-10-16T14:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T14:17:00.496 INFO:teuthology.orchestra.run.smithi170.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:17:00.496 INFO:teuthology.orchestra.run.smithi170.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:17:00.497 INFO:teuthology.orchestra.run.smithi170.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:17:00.497 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.497 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 ::1/128 scope host 2023-10-16T14:17:00.497 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.497 INFO:teuthology.orchestra.run.smithi170.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:17:00.497 INFO:teuthology.orchestra.run.smithi170.stdout: link/ether ee:05:f0:5e:67:8e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T14:17:00.498 INFO:teuthology.orchestra.run.smithi170.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T14:17:00.498 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.498 INFO:teuthology.orchestra.run.smithi170.stdout: inet6 fe80::ec05:f0ff:fe5e:678e/64 scope link 2023-10-16T14:17:00.498 INFO:teuthology.orchestra.run.smithi170.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:00.501 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-16T14:17:00.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.502 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:17:00.502 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:17:00.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-16T14:17:00.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T14:17:00.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-16T14:17:00.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-16T14:17:00.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-16T14:17:00.504 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:17:00.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:00.773 DEBUG:teuthology.orchestra.run.smithi170:> (cd / && exec bash -c ' 2023-10-16T14:17:00.773 DEBUG:teuthology.orchestra.run.smithi170:> set -e 2023-10-16T14:17:00.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set brx.1 up 2023-10-16T14:17:00.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T14:17:00.774 DEBUG:teuthology.orchestra.run.smithi170:> ') 2023-10-16T14:17:00.941 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:17:00.942 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-16T14:17:00.994 INFO:teuthology.orchestra.run.smithi170.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-16T14:17:00.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:17:00.995 DEBUG:teuthology.orchestra.run.smithi170:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-16T14:17:01.044 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:17:01.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:01.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T14:17:01.185 INFO:teuthology.orchestra.run.smithi170.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T14:17:01.185 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-16T14:17:01.185 INFO:teuthology.orchestra.run.smithi170.stdout:invalid new device string format 2023-10-16T14:17:01.206 INFO:teuthology.orchestra.run.smithi170.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T14:17:01.206 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-16T14:17:01.207 INFO:teuthology.orchestra.run.smithi170.stdout:invalid new device string format 2023-10-16T14:17:01.207 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:17:01.207 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:17:01.231 INFO:teuthology.orchestra.run.smithi170.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T14:17:01.232 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-16T14:17:01.232 INFO:teuthology.orchestra.run.smithi170.stdout:invalid new device string format 2023-10-16T14:17:01.232 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:17:01.232 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:17:01.232 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:17:01.232 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:17:01.233 INFO:teuthology.orchestra.run.smithi170.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:17:01.234 INFO:teuthology.orchestra.run.smithi170.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2023-10-16T14:17:01.234 INFO:teuthology.orchestra.run.smithi170.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:17:01.234 INFO:teuthology.orchestra.run.smithi170.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:17:01.234 INFO:teuthology.orchestra.run.smithi170.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:17:01.234 INFO:teuthology.orchestra.run.smithi170.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:17:01.237 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:17:01.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:01.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-16T14:17:01.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:01.325 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:17:01.378 INFO:teuthology.orchestra.run.smithi170.stdout:client4433 2023-10-16T14:17:01.378 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:01.378 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:01.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:01.427 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:17:01.526 INFO:teuthology.orchestra.run.smithi170.stdout:client4433 2023-10-16T14:17:01.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:01.528 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:17:01.579 INFO:teuthology.orchestra.run.smithi170.stdout:client4433 2023-10-16T14:17:01.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:01.580 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/aff38114-d596-4938-a779-5e1b7478d87b.client4433/status') 2023-10-16T14:17:01.667 INFO:teuthology.orchestra.run.smithi170.stdout:instance: client.4433 (3)192.168.0.2:0/25064342 2023-10-16T14:17:01.667 INFO:teuthology.orchestra.run.smithi170.stdout:blocklisted: false 2023-10-16T14:17:01.668 INFO:teuthology.orchestra.run.smithi170.stderr:0+1 records in 2023-10-16T14:17:01.668 INFO:teuthology.orchestra.run.smithi170.stderr:0+1 records out 2023-10-16T14:17:01.668 INFO:teuthology.orchestra.run.smithi170.stderr:67 bytes copied, 6.5053e-05 s, 1.0 MB/s 2023-10-16T14:17:01.671 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-16T14:17:01.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.924+0000 7fd74c9c5640 1 Processor -- start 2023-10-16T14:17:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.924+0000 7fd74c9c5640 1 -- start start 2023-10-16T14:17:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd74c9c5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 0x7fd74815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd74c9c5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74815ed60 con 0x7fd74815ae80 2023-10-16T14:17:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd74c9c5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7480707e0 con 0x7fd74815aaa0 2023-10-16T14:17:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd74c9c5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74806b810 con 0x7fd7480ce390 2023-10-16T14:17:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd73ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd74815ae80 0x7fd748070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38208/0 (socket says 172.21.15.29:38208) 2023-10-16T14:17:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd73ffff640 1 -- 172.21.15.29:0/3133784509 learned_addr learned my addr 172.21.15.29:0/3133784509 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd746d9d640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 0x7fd74815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.926+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529596989 0 0) 0x7fd74815ed60 con 0x7fd74815ae80 2023-10-16T14:17:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734003520 con 0x7fd74815ae80 2023-10-16T14:17:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2410925181 0 0) 0x7fd734003520 con 0x7fd74815ae80 2023-10-16T14:17:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 msgr2=0x7fd74815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 0x7fd74815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 >> v1:172.21.15.170:6789/0 conn(0x7fd74815aaa0 legacy=0x7fd7480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74815ef90 con 0x7fd74815ae80 2023-10-16T14:17:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fd730001690 con 0x7fd74815ae80 2023-10-16T14:17:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 >> v1:172.21.15.29:6789/0 conn(0x7fd74815ae80 legacy=0x7fd748070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 0x7fd7340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.927+0000 7fd745d9b640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74806b810 con 0x7fd734003f70 2023-10-16T14:17:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73400bac0 con 0x7fd734004d00 2023-10-16T14:17:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73400bc80 con 0x7fd7480ce390 2023-10-16T14:17:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd73ffff640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd74659c640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 0x7fd7340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd746d9d640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd73ffff640 1 -- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 msgr2=0x7fd7340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd73ffff640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480ce390 0x7fd7340039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd73ffff640 1 -- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd734004d00 msgr2=0x7fd734005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd73ffff640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.928+0000 7fd73ffff640 1 -- 172.21.15.29:0/3133784509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd734003520 con 0x7fd734003f70 2023-10-16T14:17:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.930+0000 7fd73ffff640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7480c2490 tx=0x7fd7340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783ed7d5ea9ea972 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.931+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730005ad0 con 0x7fd734003f70 2023-10-16T14:17:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.931+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730005c30 con 0x7fd734003f70 2023-10-16T14:17:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.931+0000 7fd745d9b640 1 -- 172.21.15.29:0/3133784509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730007070 con 0x7fd734003f70 2023-10-16T14:17:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd74c9c5640 1 -- 172.21.15.29:0/3133784509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd734003f70 msgr2=0x7fd7340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/3133784509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7480c2490 tx=0x7fd7340057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd746d9d640 1 -- 172.21.15.29:0/3133784509 reap_dead start 2023-10-16T14:17:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd74c9c5640 1 -- 172.21.15.29:0/3133784509 shutdown_connections 2023-10-16T14:17:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd74c9c5640 1 -- 172.21.15.29:0/3133784509 >> 172.21.15.29:0/3133784509 conn(0x7fd7480cb410 msgr2=0x7fd7480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd74c9c5640 1 -- 172.21.15.29:0/3133784509 shutdown_connections 2023-10-16T14:17:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.932+0000 7fd74c9c5640 1 -- 172.21.15.29:0/3133784509 wait complete. 2023-10-16T14:17:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.934+0000 7fd74c9c5640 1 Processor -- start 2023-10-16T14:17:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.934+0000 7fd74c9c5640 1 -- start start 2023-10-16T14:17:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd74c9c5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd748155df0 0x7fd7480af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd74c9c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 0x7fd7480c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd74c9c5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480c6a20 0x7fd74815b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd74c9c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480afe00 con 0x7fd7480af980 2023-10-16T14:17:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd74c9c5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480c6ed0 con 0x7fd748155df0 2023-10-16T14:17:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd74c9c5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7481f4d30 con 0x7fd7480c6a20 2023-10-16T14:17:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.935+0000 7fd73ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 0x7fd7480c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 0x7fd7480c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57872/0 (socket says 172.21.15.29:57872) 2023-10-16T14:17:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 -- 172.21.15.29:0/1171420422 learned_addr learned my addr 172.21.15.29:0/1171420422 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd74659c640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd748155df0 0x7fd7480af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd746d9d640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480c6a20 0x7fd74815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 -- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480c6a20 msgr2=0x7fd74815b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480c6a20 0x7fd74815b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 -- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd748155df0 msgr2=0x7fd7480af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd748155df0 0x7fd7480af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 -- 172.21.15.29:0/1171420422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7481f3e40 con 0x7fd7480af980 2023-10-16T14:17:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.936+0000 7fd73ffff640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 0x7fd7480c64e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd73002c350 tx=0x7fd730001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.937+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73003c070 con 0x7fd7480af980 2023-10-16T14:17:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.937+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730007300 con 0x7fd7480af980 2023-10-16T14:17:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.937+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730007690 con 0x7fd7480af980 2023-10-16T14:17:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.937+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7481f4120 con 0x7fd7480af980 2023-10-16T14:17:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.937+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7481f46e0 con 0x7fd7480af980 2023-10-16T14:17:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.938+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74815ca60 con 0x7fd7480af980 2023-10-16T14:17:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.939+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd730032070 con 0x7fd7480af980 2023-10-16T14:17:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.940+0000 7fd73effd640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd72806aea0 0x7fd72806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.940+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7300aa160 con 0x7fd7480af980 2023-10-16T14:17:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.941+0000 7fd74659c640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd72806aea0 0x7fd72806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.942+0000 7fd74659c640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd72806aea0 0x7fd72806d360 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd738007fd0 tx=0x7fd738009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:01.945+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd74815ca60 con 0x7fd7480af980 2023-10-16T14:17:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.139+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd74815ca60 con 0x7fd7480af980 2023-10-16T14:17:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.142+0000 7fd73effd640 1 -- 172.21.15.29:0/1171420422 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd74815ca60 con 0x7fd7480af980 2023-10-16T14:17:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:02.147 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:02.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd72806aea0 msgr2=0x7fd72806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd72806aea0 0x7fd72806d360 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd738007fd0 tx=0x7fd738009040 comp rx=0 tx=0).stop 2023-10-16T14:17:02.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 msgr2=0x7fd7480c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 0x7fd7480c64e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd73002c350 tx=0x7fd730001950 comp rx=0 tx=0).stop 2023-10-16T14:17:02.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 shutdown_connections 2023-10-16T14:17:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd72806aea0 0x7fd72806d360 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7480af980 0x7fd7480c64e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7480c6a20 0x7fd74815b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 --2- 172.21.15.29:0/1171420422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd748155df0 0x7fd7480af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 >> 172.21.15.29:0/1171420422 conn(0x7fd7480cb410 msgr2=0x7fd7480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.145+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 shutdown_connections 2023-10-16T14:17:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.146+0000 7fd74c9c5640 1 -- 172.21.15.29:0/1171420422 wait complete. 2023-10-16T14:17:02.169 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:02.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.377+0000 7fa72759e640 1 Processor -- start 2023-10-16T14:17:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.377+0000 7fa72759e640 1 -- start start 2023-10-16T14:17:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.378+0000 7fa72759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280b1ca0 0x7fa7280d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.378+0000 7fa72759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7280b2080 0x7fa7280ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.378+0000 7fa72759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280d0970 con 0x7fa7280ce960 2023-10-16T14:17:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.378+0000 7fa72759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7280d0ea0 con 0x7fa7280b2080 2023-10-16T14:17:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.378+0000 7fa72759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280d0bc0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7280b2080 0x7fa7280ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7280b2080 0x7fa7280ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57884/0 (socket says 172.21.15.29:57884) 2023-10-16T14:17:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 -- 172.21.15.29:0/1011741607 learned_addr learned my addr 172.21.15.29:0/1011741607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa72659c640 1 --2- 172.21.15.29:0/1011741607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280b1ca0 0x7fa7280d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 -- 172.21.15.29:0/1011741607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280b1ca0 msgr2=0x7fa7280d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:02.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 --2- 172.21.15.29:0/1011741607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280b1ca0 0x7fa7280d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 -- 172.21.15.29:0/1011741607 >> v1:172.21.15.170:6789/0 conn(0x7fa7280ce960 legacy=0x7fa7280b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:02.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.379+0000 7fa71ffff640 1 -- 172.21.15.29:0/1011741607 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280cd940 con 0x7fa7280b2080 2023-10-16T14:17:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.381+0000 7fa71ffff640 1 --2- 172.21.15.29:0/1011741607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b2080 0x7fa7280ce420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa71002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e3b2e1165e3697 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.381+0000 7fa725d9b640 1 -- 172.21.15.29:0/1011741607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71002f840 con 0x7fa7280b2080 2023-10-16T14:17:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.381+0000 7fa725d9b640 1 -- 172.21.15.29:0/1011741607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71002f9a0 con 0x7fa7280b2080 2023-10-16T14:17:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.381+0000 7fa725d9b640 1 -- 172.21.15.29:0/1011741607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71002fd30 con 0x7fa7280b2080 2023-10-16T14:17:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 -- 172.21.15.29:0/1011741607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b2080 msgr2=0x7fa7280ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 --2- 172.21.15.29:0/1011741607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b2080 0x7fa7280ce420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa71002ba20 comp rx=0 tx=0).stop 2023-10-16T14:17:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 -- 172.21.15.29:0/1011741607 shutdown_connections 2023-10-16T14:17:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 --2- 172.21.15.29:0/1011741607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b2080 0x7fa7280ce420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 --2- 172.21.15.29:0/1011741607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280b1ca0 0x7fa7280d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 -- 172.21.15.29:0/1011741607 >> 172.21.15.29:0/1011741607 conn(0x7fa7280d2b90 msgr2=0x7fa7280d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:02.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 -- 172.21.15.29:0/1011741607 shutdown_connections 2023-10-16T14:17:02.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.382+0000 7fa72759e640 1 -- 172.21.15.29:0/1011741607 wait complete. 2023-10-16T14:17:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.384+0000 7fa72759e640 1 Processor -- start 2023-10-16T14:17:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.384+0000 7fa72759e640 1 -- start start 2023-10-16T14:17:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 0x7fa728150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7280b2080 0x7fa728151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280ce960 0x7fa72809c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72809dec0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72809d2b0 con 0x7fa7280b2080 2023-10-16T14:17:02.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72809d590 con 0x7fa7280ce960 2023-10-16T14:17:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 0x7fa728150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 0x7fa728150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57898/0 (socket says 172.21.15.29:57898) 2023-10-16T14:17:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa72659c640 1 -- 172.21.15.29:0/910844544 learned_addr learned my addr 172.21.15.29:0/910844544 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa726d9d640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280ce960 0x7fa72809c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.385+0000 7fa71ffff640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7280b2080 0x7fa728151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72659c640 1 -- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280ce960 msgr2=0x7fa72809c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72659c640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280ce960 0x7fa72809c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72659c640 1 -- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7280b2080 msgr2=0x7fa728151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72659c640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7280b2080 0x7fa728151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72659c640 1 -- 172.21.15.29:0/910844544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100094d0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa726d9d640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280ce960 0x7fa72809c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72659c640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 0x7fa728150fa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa71800d5f0 tx=0x7fa71800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7180125f0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa718012750 con 0x7fa7280b1ca0 2023-10-16T14:17:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718012b00 con 0x7fa7280b1ca0 2023-10-16T14:17:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72809d870 con 0x7fa7280b1ca0 2023-10-16T14:17:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.386+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7281514e0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.388+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72804e9b0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.388+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa718012c60 con 0x7fa7280b1ca0 2023-10-16T14:17:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.389+0000 7fa71effd640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa70806aec0 0x7fa70806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.390+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa718089a00 con 0x7fa7280b1ca0 2023-10-16T14:17:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.390+0000 7fa71ffff640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa70806aec0 0x7fa70806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.391+0000 7fa71ffff640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa70806aec0 0x7fa70806d380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa710036000 tx=0x7fa710039040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.394+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa718054b90 con 0x7fa7280b1ca0 2023-10-16T14:17:02.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.592+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7280b8a50 con 0x7fa7280b1ca0 2023-10-16T14:17:02.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.593+0000 7fa71effd640 1 -- 172.21.15.29:0/910844544 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa7180581b0 con 0x7fa7280b1ca0 2023-10-16T14:17:02.596 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa70806aec0 msgr2=0x7fa70806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa70806aec0 0x7fa70806d380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa710036000 tx=0x7fa710039040 comp rx=0 tx=0).stop 2023-10-16T14:17:02.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 msgr2=0x7fa728150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 0x7fa728150fa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa71800d5f0 tx=0x7fa71800dab0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 shutdown_connections 2023-10-16T14:17:02.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa70806aec0 0x7fa70806d380 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7280ce960 0x7fa72809c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7280b2080 0x7fa728151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 --2- 172.21.15.29:0/910844544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7280b1ca0 0x7fa728150fa0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 >> 172.21.15.29:0/910844544 conn(0x7fa7280d2b90 msgr2=0x7fa7280afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 shutdown_connections 2023-10-16T14:17:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.596+0000 7fa72759e640 1 -- 172.21.15.29:0/910844544 wait complete. 2023-10-16T14:17:02.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:02.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.842+0000 7f4f31490640 1 Processor -- start 2023-10-16T14:17:02.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.842+0000 7f4f31490640 1 -- start start 2023-10-16T14:17:02.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.843+0000 7f4f31490640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.843+0000 7f4f31490640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c070360 con 0x7f4f2c06c140 2023-10-16T14:17:02.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.843+0000 7f4f31490640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0cd1f0 con 0x7f4f2c159ab0 2023-10-16T14:17:02.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.843+0000 7f4f31490640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c15d890 con 0x7f4f2c159e90 2023-10-16T14:17:02.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f2b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f2c06c140 0x7f4f2c15dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38214/0 (socket says 172.21.15.29:38214) 2023-10-16T14:17:02.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f2b7fe640 1 -- 172.21.15.29:0/1604775280 learned_addr learned my addr 172.21.15.29:0/1604775280 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:02.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/1604775280 >> v2:172.21.15.170:3300/0 conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567569253 0 0) 0x7f4f2c070360 con 0x7f4f2c06c140 2023-10-16T14:17:02.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14003520 con 0x7f4f2c06c140 2023-10-16T14:17:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435403110 0 0) 0x7f4f14003520 con 0x7f4f2c06c140 2023-10-16T14:17:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 >> v2:172.21.15.170:3300/0 conn(0x7f4f2c159e90 msgr2=0x7f4f2c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 --2- 172.21.15.29:0/1604775280 >> v2:172.21.15.170:3300/0 conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 >> v1:172.21.15.170:6789/0 conn(0x7f4f2c159ab0 legacy=0x7f4f2c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0cd4d0 con 0x7f4f2c06c140 2023-10-16T14:17:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4f240019f0 con 0x7f4f2c06c140 2023-10-16T14:17:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.844+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 >> v1:172.21.15.29:6789/0 conn(0x7f4f2c06c140 legacy=0x7f4f2c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f29ffb640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f140039c0 0x7f4f14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f29ffb640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f29ffb640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f140052a0 0x7f4f1400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c15d890 con 0x7f4f140047c0 2023-10-16T14:17:02.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1400e3b0 con 0x7f4f140052a0 2023-10-16T14:17:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1400e570 con 0x7f4f140039c0 2023-10-16T14:17:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2b7fe640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f140052a0 0x7f4f1400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2affd640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f140039c0 0x7f4f14003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2a7fc640 1 -- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f140039c0 msgr2=0x7f4f14003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f140039c0 0x7f4f14003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2a7fc640 1 -- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f140052a0 msgr2=0x7f4f1400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f140052a0 0x7f4f1400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2a7fc640 1 -- 172.21.15.29:0/1604775280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14003520 con 0x7f4f140047c0 2023-10-16T14:17:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2b7fe640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f140052a0 0x7f4f1400db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.845+0000 7f4f2b7fe640 1 -- 172.21.15.29:0/1604775280 reap_dead start 2023-10-16T14:17:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.846+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f14004c40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009ac0 tx=0x7f4f140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4e4baa895b8517 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2002c6b0 con 0x7f4f140047c0 2023-10-16T14:17:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2002c810 con 0x7f4f140047c0 2023-10-16T14:17:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f29ffb640 1 -- 172.21.15.29:0/1604775280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2002cba0 con 0x7f4f140047c0 2023-10-16T14:17:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f31490640 1 -- 172.21.15.29:0/1604775280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 msgr2=0x7f4f14004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f31490640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f14004c40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009ac0 tx=0x7f4f140057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f31490640 1 -- 172.21.15.29:0/1604775280 shutdown_connections 2023-10-16T14:17:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f31490640 1 --2- 172.21.15.29:0/1604775280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f14004c40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.847+0000 7f4f31490640 1 -- 172.21.15.29:0/1604775280 >> 172.21.15.29:0/1604775280 conn(0x7f4f2c0ce9b0 msgr2=0x7f4f2c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:02.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.848+0000 7f4f31490640 1 -- 172.21.15.29:0/1604775280 shutdown_connections 2023-10-16T14:17:02.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.848+0000 7f4f31490640 1 -- 172.21.15.29:0/1604775280 wait complete. 2023-10-16T14:17:02.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.849+0000 7f4f31490640 1 Processor -- start 2023-10-16T14:17:02.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.849+0000 7f4f31490640 1 -- start start 2023-10-16T14:17:02.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.850+0000 7f4f31490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f2c154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.850+0000 7f4f31490640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f2c0c7c50 0x7f4f2c0aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.850+0000 7f4f31490640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f2c0af4b0 0x7f4f2c0c3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.850+0000 7f4f31490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0af930 con 0x7f4f140047c0 2023-10-16T14:17:02.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.850+0000 7f4f31490640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0b1fe0 con 0x7f4f2c0c7c50 2023-10-16T14:17:02.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.850+0000 7f4f31490640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0b25a0 con 0x7f4f2c0af4b0 2023-10-16T14:17:02.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f2c154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f2c154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57916/0 (socket says 172.21.15.29:57916) 2023-10-16T14:17:02.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 -- 172.21.15.29:0/345626398 learned_addr learned my addr 172.21.15.29:0/345626398 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:02.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2b7fe640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f2c0af4b0 0x7f4f2c0c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f2c0c7c50 0x7f4f2c0aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 -- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f2c0af4b0 msgr2=0x7f4f2c0c3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f2c0af4b0 0x7f4f2c0c3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 -- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f2c0c7c50 msgr2=0x7f4f2c0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f2c0c7c50 0x7f4f2c0aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.851+0000 7f4f2affd640 1 -- 172.21.15.29:0/345626398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0c4e90 con 0x7f4f140047c0 2023-10-16T14:17:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.852+0000 7f4f2affd640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f2c154fd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00d570 tx=0x7f4f1c00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.852+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c003610 con 0x7f4f140047c0 2023-10-16T14:17:02.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.852+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c003770 con 0x7f4f140047c0 2023-10-16T14:17:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.852+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c003b20 con 0x7f4f140047c0 2023-10-16T14:17:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.852+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c0c5170 con 0x7f4f140047c0 2023-10-16T14:17:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.852+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c0b2230 con 0x7f4f140047c0 2023-10-16T14:17:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.853+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c0b23c0 con 0x7f4f140047c0 2023-10-16T14:17:02.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.854+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f1c003c80 con 0x7f4f140047c0 2023-10-16T14:17:02.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.855+0000 7f4f137fe640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4f1806aea0 0x7f4f1806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:02.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.855+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f1c0894f0 con 0x7f4f140047c0 2023-10-16T14:17:02.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.856+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4f1806aea0 0x7f4f1806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:02.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.857+0000 7f4f2a7fc640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4f1806aea0 0x7f4f1806d360 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f20036000 tx=0x7f4f20030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:02.860+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f1c054650 con 0x7f4f140047c0 2023-10-16T14:17:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.053+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4f2c04e160 con 0x7f4f140047c0 2023-10-16T14:17:03.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.054+0000 7f4f137fe640 1 -- 172.21.15.29:0/345626398 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4f1c057c70 con 0x7f4f140047c0 2023-10-16T14:17:03.057 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.056+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4f1806aea0 msgr2=0x7f4f1806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.056+0000 7f4f31490640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4f1806aea0 0x7f4f1806d360 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f20036000 tx=0x7f4f20030000 comp rx=0 tx=0).stop 2023-10-16T14:17:03.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 msgr2=0x7f4f2c154fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f2c154fd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00d570 tx=0x7f4f1c00da30 comp rx=0 tx=0).stop 2023-10-16T14:17:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 shutdown_connections 2023-10-16T14:17:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4f1806aea0 0x7f4f1806d360 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f2c0af4b0 0x7f4f2c0c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f2c0c7c50 0x7f4f2c0aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 --2- 172.21.15.29:0/345626398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f140047c0 0x7f4f2c154fd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 >> 172.21.15.29:0/345626398 conn(0x7f4f2c0ce9b0 msgr2=0x7f4f2c158240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:03.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 shutdown_connections 2023-10-16T14:17:03.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.057+0000 7f4f31490640 1 -- 172.21.15.29:0/345626398 wait complete. 2023-10-16T14:17:03.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:03.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.301+0000 7f978ffff640 1 Processor -- start 2023-10-16T14:17:03.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.301+0000 7f978ffff640 1 -- start start 2023-10-16T14:17:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f979015aa60 0x7f97900af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 0x7f97900b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97900b3730 con 0x7f97900afd60 2023-10-16T14:17:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97900b3980 con 0x7f979015aa60 2023-10-16T14:17:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97900b3c60 con 0x7f97900cc9f0 2023-10-16T14:17:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978e7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f97900afd60 0x7f97900cc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38218/0 (socket says 172.21.15.29:38218) 2023-10-16T14:17:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.303+0000 7f978e7fc640 1 -- 172.21.15.29:0/1285025148 learned_addr learned my addr 172.21.15.29:0/1285025148 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978f7fe640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 0x7f97900b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978effd640 1 --2- 172.21.15.29:0/1285025148 >> v2:172.21.15.170:3300/0 conn(0x7f979015aa60 0x7f97900af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479328886 0 0) 0x7f97900b3730 con 0x7f97900afd60 2023-10-16T14:17:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f977c003520 con 0x7f97900afd60 2023-10-16T14:17:03.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1687501358 0 0) 0x7f977c003520 con 0x7f97900afd60 2023-10-16T14:17:03.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 msgr2=0x7f97900b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:03.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 0x7f97900b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 >> v2:172.21.15.170:3300/0 conn(0x7f979015aa60 msgr2=0x7f97900af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:03.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 --2- 172.21.15.29:0/1285025148 >> v2:172.21.15.170:3300/0 conn(0x7f979015aa60 0x7f97900af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900b3f40 con 0x7f97900afd60 2023-10-16T14:17:03.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f97780019f0 con 0x7f97900afd60 2023-10-16T14:17:03.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 >> v1:172.21.15.29:6789/0 conn(0x7f97900afd60 legacy=0x7f97900cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f977c003970 0x7f977c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 0x7f977c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.304+0000 7f978dffb640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f977c004d00 0x7f977c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900b3980 con 0x7f977c004d00 2023-10-16T14:17:03.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900b3c60 con 0x7f977c003970 2023-10-16T14:17:03.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c00b980 con 0x7f97900cc9f0 2023-10-16T14:17:03.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978f7fe640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f977c004d00 0x7f977c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978effd640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f977c003970 0x7f977c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978e7fc640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 0x7f977c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978f7fe640 1 -- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 msgr2=0x7f977c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:03.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978f7fe640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97900cc9f0 0x7f977c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978f7fe640 1 -- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f977c003970 msgr2=0x7f977c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:03.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978f7fe640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f977c003970 0x7f977c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.305+0000 7f978f7fe640 1 -- 172.21.15.29:0/1285025148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c003520 con 0x7f977c004d00 2023-10-16T14:17:03.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.306+0000 7f978f7fe640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f977c004d00 0x7f977c005180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9788009ac0 tx=0x7f977c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9b5373809e0f22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:03.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978802c6b0 con 0x7f977c004d00 2023-10-16T14:17:03.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978802c810 con 0x7f977c004d00 2023-10-16T14:17:03.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978dffb640 1 -- 172.21.15.29:0/1285025148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978802cba0 con 0x7f977c004d00 2023-10-16T14:17:03.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978ffff640 1 -- 172.21.15.29:0/1285025148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f977c004d00 msgr2=0x7f977c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978ffff640 1 --2- 172.21.15.29:0/1285025148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f977c004d00 0x7f977c005180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9788009ac0 tx=0x7f977c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978f7fe640 1 -- 172.21.15.29:0/1285025148 reap_dead start 2023-10-16T14:17:03.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978ffff640 1 -- 172.21.15.29:0/1285025148 shutdown_connections 2023-10-16T14:17:03.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978ffff640 1 -- 172.21.15.29:0/1285025148 >> 172.21.15.29:0/1285025148 conn(0x7f9790065ed0 msgr2=0x7f97900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:03.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.307+0000 7f978ffff640 1 -- 172.21.15.29:0/1285025148 shutdown_connections 2023-10-16T14:17:03.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.308+0000 7f978ffff640 1 -- 172.21.15.29:0/1285025148 wait complete. 2023-10-16T14:17:03.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.309+0000 7f978ffff640 1 Processor -- start 2023-10-16T14:17:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.309+0000 7f978ffff640 1 -- start start 2023-10-16T14:17:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.310+0000 7f978ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f977c004d00 0x7f97900c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.310+0000 7f978ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 0x7f979014d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.310+0000 7f978ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97900c97e0 0x7f979006bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.310+0000 7f978ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900ca8a0 con 0x7f97900ca050 2023-10-16T14:17:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.310+0000 7f978ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900c81a0 con 0x7f97900c97e0 2023-10-16T14:17:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.310+0000 7f978ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979014de40 con 0x7f977c004d00 2023-10-16T14:17:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 0x7f979014d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 0x7f979014d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57934/0 (socket says 172.21.15.29:57934) 2023-10-16T14:17:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 -- 172.21.15.29:0/3161202475 learned_addr learned my addr 172.21.15.29:0/3161202475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978effd640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f977c004d00 0x7f97900c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978f7fe640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97900c97e0 0x7f979006bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 -- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f977c004d00 msgr2=0x7f97900c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f977c004d00 0x7f97900c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 -- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97900c97e0 msgr2=0x7f979006bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97900c97e0 0x7f979006bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.311+0000 7f978e7fc640 1 -- 172.21.15.29:0/3161202475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900ca5c0 con 0x7f97900ca050 2023-10-16T14:17:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.312+0000 7f978e7fc640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 0x7f979014d900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9778007b90 tx=0x7f977800e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.312+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778004b60 con 0x7f97900ca050 2023-10-16T14:17:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.312+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9778004ce0 con 0x7f97900ca050 2023-10-16T14:17:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.312+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97780146c0 con 0x7f97900ca050 2023-10-16T14:17:03.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.312+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97900c7b50 con 0x7f97900ca050 2023-10-16T14:17:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.312+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97900c9c60 con 0x7f97900ca050 2023-10-16T14:17:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.313+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979004e9b0 con 0x7f97900ca050 2023-10-16T14:17:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.314+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9778005050 con 0x7f97900ca050 2023-10-16T14:17:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.315+0000 7f9787fff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f977006ae30 0x7f977006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.315+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9778089c90 con 0x7f97900ca050 2023-10-16T14:17:03.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.316+0000 7f978effd640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f977006ae30 0x7f977006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.317+0000 7f978effd640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f977006ae30 0x7f977006d2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9780009a10 tx=0x7f9780008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:03.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.319+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9778054ca0 con 0x7f97900ca050 2023-10-16T14:17:03.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.512+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97900cb620 con 0x7f97900ca050 2023-10-16T14:17:03.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.513+0000 7f9787fff640 1 -- 172.21.15.29:0/3161202475 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9778019070 con 0x7f97900ca050 2023-10-16T14:17:03.516 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:03.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f977006ae30 msgr2=0x7f977006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f977006ae30 0x7f977006d2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9780009a10 tx=0x7f9780008040 comp rx=0 tx=0).stop 2023-10-16T14:17:03.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 msgr2=0x7f979014d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 0x7f979014d900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9778007b90 tx=0x7f977800e5f0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 shutdown_connections 2023-10-16T14:17:03.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f977006ae30 0x7f977006d2f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97900ca050 0x7f979014d900 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97900c97e0 0x7f979006bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 --2- 172.21.15.29:0/3161202475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f977c004d00 0x7f97900c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.516+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 >> 172.21.15.29:0/3161202475 conn(0x7f9790065ed0 msgr2=0x7f97900ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.517+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 shutdown_connections 2023-10-16T14:17:03.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.517+0000 7f978ffff640 1 -- 172.21.15.29:0/3161202475 wait complete. 2023-10-16T14:17:03.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.759+0000 7fcf08bde640 1 Processor -- start 2023-10-16T14:17:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.759+0000 7fcf08bde640 1 -- start start 2023-10-16T14:17:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf08bde640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf04070b60 0x7fcf0415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf08bde640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcf0406c100 0x7fcf0406c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf08bde640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0415aef0 con 0x7fcf04159950 2023-10-16T14:17:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf08bde640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0415e490 con 0x7fcf04070b60 2023-10-16T14:17:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf08bde640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0415d910 con 0x7fcf0406c100 2023-10-16T14:17:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf04070b60 0x7fcf0415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf04070b60 0x7fcf0415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57944/0 (socket says 172.21.15.29:57944) 2023-10-16T14:17:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 -- 172.21.15.29:0/2991969191 learned_addr learned my addr 172.21.15.29:0/2991969191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf03fff640 1 --2- 172.21.15.29:0/2991969191 >> v2:172.21.15.170:3300/0 conn(0x7fcf0406c100 0x7fcf0406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 -- 172.21.15.29:0/2991969191 >> v2:172.21.15.170:3300/0 conn(0x7fcf0406c100 msgr2=0x7fcf0406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 --2- 172.21.15.29:0/2991969191 >> v2:172.21.15.170:3300/0 conn(0x7fcf0406c100 0x7fcf0406c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 -- 172.21.15.29:0/2991969191 >> v1:172.21.15.29:6789/0 conn(0x7fcf04159950 legacy=0x7fcf04070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf02ffd640 1 -- 172.21.15.29:0/2991969191 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0415dbf0 con 0x7fcf04070b60 2023-10-16T14:17:03.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.761+0000 7fcf027fc640 1 -- 172.21.15.29:0/2991969191 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753124614 0 0) 0x7fcef4001110 con 0x7fcf04159950 2023-10-16T14:17:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.763+0000 7fcf02ffd640 1 --2- 172.21.15.29:0/2991969191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf04070b60 0x7fcf0415a9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcf040ade70 tx=0x7fcef802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4de839025a77e57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.763+0000 7fcf027fc640 1 -- 172.21.15.29:0/2991969191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef802ee90 con 0x7fcf04070b60 2023-10-16T14:17:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.763+0000 7fcf027fc640 1 -- 172.21.15.29:0/2991969191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef8036440 con 0x7fcf04070b60 2023-10-16T14:17:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.763+0000 7fcf027fc640 1 -- 172.21.15.29:0/2991969191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80367d0 con 0x7fcf04070b60 2023-10-16T14:17:03.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.763+0000 7fcf08bde640 1 -- 172.21.15.29:0/2991969191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf04070b60 msgr2=0x7fcf0415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.763+0000 7fcf08bde640 1 --2- 172.21.15.29:0/2991969191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf04070b60 0x7fcf0415a9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcf040ade70 tx=0x7fcef802e0e0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.764+0000 7fcf08bde640 1 -- 172.21.15.29:0/2991969191 shutdown_connections 2023-10-16T14:17:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.764+0000 7fcf08bde640 1 --2- 172.21.15.29:0/2991969191 >> v2:172.21.15.170:3300/0 conn(0x7fcf0406c100 0x7fcf0406c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.764+0000 7fcf08bde640 1 --2- 172.21.15.29:0/2991969191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf04070b60 0x7fcf0415a9b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.764+0000 7fcf08bde640 1 -- 172.21.15.29:0/2991969191 >> 172.21.15.29:0/2991969191 conn(0x7fcf040ce6a0 msgr2=0x7fcf040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.764+0000 7fcf08bde640 1 -- 172.21.15.29:0/2991969191 shutdown_connections 2023-10-16T14:17:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.764+0000 7fcf08bde640 1 -- 172.21.15.29:0/2991969191 wait complete. 2023-10-16T14:17:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.766+0000 7fcf08bde640 1 Processor -- start 2023-10-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.766+0000 7fcf08bde640 1 -- start start 2023-10-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf08bde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 0x7fcf040c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf08bde640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf04070b60 0x7fcf041514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf08bde640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf04159950 0x7fcf040c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf08bde640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c5170 con 0x7fcf0406c100 2023-10-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf08bde640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c5450 con 0x7fcf04159950 2023-10-16T14:17:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf08bde640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c71d0 con 0x7fcf04070b60 2023-10-16T14:17:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 0x7fcf040c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 0x7fcf040c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57950/0 (socket says 172.21.15.29:57950) 2023-10-16T14:17:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 -- 172.21.15.29:0/4119273130 learned_addr learned my addr 172.21.15.29:0/4119273130 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf03fff640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf04159950 0x7fcf040c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf02ffd640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf04070b60 0x7fcf041514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 -- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf04070b60 msgr2=0x7fcf041514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf04070b60 0x7fcf041514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 -- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf04159950 msgr2=0x7fcf040c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf04159950 0x7fcf040c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.767+0000 7fcf037fe640 1 -- 172.21.15.29:0/4119273130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef800b770 con 0x7fcf0406c100 2023-10-16T14:17:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.768+0000 7fcf037fe640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 0x7fcf040c24a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c410 tx=0x7fcef400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.768+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef40035f0 con 0x7fcf0406c100 2023-10-16T14:17:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.768+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef4003770 con 0x7fcf0406c100 2023-10-16T14:17:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.768+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4003b20 con 0x7fcf0406c100 2023-10-16T14:17:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.768+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf040c74b0 con 0x7fcf0406c100 2023-10-16T14:17:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.768+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf040c7f10 con 0x7fcf0406c100 2023-10-16T14:17:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.769+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf040c52b0 con 0x7fcf0406c100 2023-10-16T14:17:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.770+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcef4003c80 con 0x7fcf0406c100 2023-10-16T14:17:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.772+0000 7fceebfff640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcef006ae70 0x7fcef006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.772+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcef4088ab0 con 0x7fcf0406c100 2023-10-16T14:17:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.772+0000 7fcf02ffd640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcef006ae70 0x7fcef006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.774+0000 7fcf02ffd640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcef006ae70 0x7fcef006d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcef8038000 tx=0x7fcef8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.776+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf040c52b0 con 0x7fcf0406c100 2023-10-16T14:17:03.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.975+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf040c52b0 con 0x7fcf0406c100 2023-10-16T14:17:03.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.978+0000 7fceebfff640 1 -- 172.21.15.29:0/4119273130 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fcf040c52b0 con 0x7fcf0406c100 2023-10-16T14:17:03.981 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:03.982 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:03.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcef006ae70 msgr2=0x7fcef006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcef006ae70 0x7fcef006d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcef8038000 tx=0x7fcef8008040 comp rx=0 tx=0).stop 2023-10-16T14:17:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 msgr2=0x7fcf040c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 0x7fcf040c24a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c410 tx=0x7fcef400c8d0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 shutdown_connections 2023-10-16T14:17:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcef006ae70 0x7fcef006d330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf04159950 0x7fcf040c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf04070b60 0x7fcf041514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 --2- 172.21.15.29:0/4119273130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf0406c100 0x7fcf040c24a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.981+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 >> 172.21.15.29:0/4119273130 conn(0x7fcf040ce6a0 msgr2=0x7fcf04070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.982+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 shutdown_connections 2023-10-16T14:17:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:03.982+0000 7fcf08bde640 1 -- 172.21.15.29:0/4119273130 wait complete. 2023-10-16T14:17:04.003 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:04.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.232+0000 7f28aa672640 1 Processor -- start 2023-10-16T14:17:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.232+0000 7f28aa672640 1 -- start start 2023-10-16T14:17:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.233+0000 7f28aa672640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28a40c9de0 0x7f28a40ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.234+0000 7f28aa672640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a40cd260 con 0x7f28a4070f30 2023-10-16T14:17:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.234+0000 7f28aa672640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a415de30 con 0x7f28a4159ad0 2023-10-16T14:17:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.234+0000 7f28aa672640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40ca790 con 0x7f28a40c9de0 2023-10-16T14:17:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.234+0000 7f28a9e71640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28a40c9de0 0x7f28a40ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.234+0000 7f28a8e6f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f28a4070f30 0x7f28a406aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51972/0 (socket says 172.21.15.29:51972) 2023-10-16T14:17:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.234+0000 7f28a8e6f640 1 -- 172.21.15.29:0/1680927157 learned_addr learned my addr 172.21.15.29:0/1680927157 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.235+0000 7f28a9e71640 1 -- 172.21.15.29:0/1680927157 >> v1:172.21.15.170:6789/0 conn(0x7f28a4159ad0 legacy=0x7f28a415d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.235+0000 7f28a9e71640 1 -- 172.21.15.29:0/1680927157 >> v1:172.21.15.29:6789/0 conn(0x7f28a4070f30 legacy=0x7f28a406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.235+0000 7f28a9e71640 1 -- 172.21.15.29:0/1680927157 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a415e110 con 0x7f28a40c9de0 2023-10-16T14:17:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.235+0000 7f289bfff640 1 -- 172.21.15.29:0/1680927157 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699770520 0 0) 0x7f28a40cd260 con 0x7f28a4070f30 2023-10-16T14:17:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.236+0000 7f28a9e71640 1 --2- 172.21.15.29:0/1680927157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40c9de0 0x7f28a40ca250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f289c0097a0 tx=0x7f289c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a55fee69e61895 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.236+0000 7f289bfff640 1 -- 172.21.15.29:0/1680927157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c030610 con 0x7f28a40c9de0 2023-10-16T14:17:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.236+0000 7f289bfff640 1 -- 172.21.15.29:0/1680927157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c030770 con 0x7f28a40c9de0 2023-10-16T14:17:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.236+0000 7f289bfff640 1 -- 172.21.15.29:0/1680927157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c030b00 con 0x7f28a40c9de0 2023-10-16T14:17:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 -- 172.21.15.29:0/1680927157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40c9de0 msgr2=0x7f28a40ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 --2- 172.21.15.29:0/1680927157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40c9de0 0x7f28a40ca250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f289c0097a0 tx=0x7f289c02b960 comp rx=0 tx=0).stop 2023-10-16T14:17:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 -- 172.21.15.29:0/1680927157 shutdown_connections 2023-10-16T14:17:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 --2- 172.21.15.29:0/1680927157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40c9de0 0x7f28a40ca250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 -- 172.21.15.29:0/1680927157 >> 172.21.15.29:0/1680927157 conn(0x7f28a40b0170 msgr2=0x7f28a40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 -- 172.21.15.29:0/1680927157 shutdown_connections 2023-10-16T14:17:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.237+0000 7f28aa672640 1 -- 172.21.15.29:0/1680927157 wait complete. 2023-10-16T14:17:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.239+0000 7f28aa672640 1 Processor -- start 2023-10-16T14:17:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.239+0000 7f28aa672640 1 -- start start 2023-10-16T14:17:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28aa672640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 0x7f28a414d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28aa672640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40c9de0 0x7f28a414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28aa672640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a4159ad0 0x7f28a415b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28aa672640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4150bb0 con 0x7f28a4070f30 2023-10-16T14:17:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28aa672640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40c5e60 con 0x7f28a4159ad0 2023-10-16T14:17:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28aa672640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a415cd90 con 0x7f28a40c9de0 2023-10-16T14:17:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 0x7f28a414d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 0x7f28a414d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43402/0 (socket says 172.21.15.29:43402) 2023-10-16T14:17:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 -- 172.21.15.29:0/1636659457 learned_addr learned my addr 172.21.15.29:0/1636659457 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 -- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40c9de0 msgr2=0x7f28a414de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40c9de0 0x7f28a414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 -- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a4159ad0 msgr2=0x7f28a415b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a4159ad0 0x7f28a415b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.240+0000 7f28a9670640 1 -- 172.21.15.29:0/1636659457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c009400 con 0x7f28a4070f30 2023-10-16T14:17:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.241+0000 7f28a9670640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 0x7f28a414d8e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c730 tx=0x7f28a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.241+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0003b70 con 0x7f28a4070f30 2023-10-16T14:17:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.241+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a0003cf0 con 0x7f28a4070f30 2023-10-16T14:17:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.241+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0003ea0 con 0x7f28a4070f30 2023-10-16T14:17:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.241+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a415c7d0 con 0x7f28a4070f30 2023-10-16T14:17:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.241+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a4150840 con 0x7f28a4070f30 2023-10-16T14:17:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.242+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a4150a00 con 0x7f28a4070f30 2023-10-16T14:17:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.243+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f28a000aa90 con 0x7f28a4070f30 2023-10-16T14:17:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.244+0000 7f2899ffb640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f288806ae70 0x7f288806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.244+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28a0089440 con 0x7f28a4070f30 2023-10-16T14:17:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.245+0000 7f28a8e6f640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f288806ae70 0x7f288806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.247+0000 7f28a8e6f640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f288806ae70 0x7f288806d330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2894007fd0 tx=0x7f2894007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.249+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28a00544f0 con 0x7f28a4070f30 2023-10-16T14:17:04.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.436+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28a40ce480 con 0x7f28a4070f30 2023-10-16T14:17:04.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.437+0000 7f2899ffb640 1 -- 172.21.15.29:0/1636659457 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28a0057b10 con 0x7f28a4070f30 2023-10-16T14:17:04.440 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f288806ae70 msgr2=0x7f288806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f288806ae70 0x7f288806d330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2894007fd0 tx=0x7f2894007900 comp rx=0 tx=0).stop 2023-10-16T14:17:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 msgr2=0x7f28a414d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 0x7f28a414d8e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c730 tx=0x7f28a000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 shutdown_connections 2023-10-16T14:17:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f288806ae70 0x7f288806d330 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a4159ad0 0x7f28a415b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40c9de0 0x7f28a414de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 --2- 172.21.15.29:0/1636659457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a4070f30 0x7f28a414d8e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 >> 172.21.15.29:0/1636659457 conn(0x7f28a40b0170 msgr2=0x7f28a40b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 shutdown_connections 2023-10-16T14:17:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.440+0000 7f28aa672640 1 -- 172.21.15.29:0/1636659457 wait complete. 2023-10-16T14:17:04.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:04.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.686+0000 7f6a2d82e640 1 Processor -- start 2023-10-16T14:17:04.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.686+0000 7f6a2d82e640 1 -- start start 2023-10-16T14:17:04.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d82e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a280b8480 0x7f6a280cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d82e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a280d0be0 0x7f6a280cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d82e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2815d150 con 0x7f6a2815cd70 2023-10-16T14:17:04.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d82e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a280d1180 con 0x7f6a280d0be0 2023-10-16T14:17:04.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d82e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280d1460 con 0x7f6a280b8480 2023-10-16T14:17:04.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d02d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a280d0be0 0x7f6a280cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d02d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a280d0be0 0x7f6a280cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43414/0 (socket says 172.21.15.29:43414) 2023-10-16T14:17:04.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.688+0000 7f6a2d02d640 1 -- 172.21.15.29:0/3516028437 learned_addr learned my addr 172.21.15.29:0/3516028437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:04.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.689+0000 7f6a27fff640 1 --2- 172.21.15.29:0/3516028437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a280b8480 0x7f6a280cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.689+0000 7f6a2d02d640 1 -- 172.21.15.29:0/3516028437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a280b8480 msgr2=0x7f6a280cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:04.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.689+0000 7f6a2d02d640 1 --2- 172.21.15.29:0/3516028437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a280b8480 0x7f6a280cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.689+0000 7f6a2d02d640 1 -- 172.21.15.29:0/3516028437 >> v1:172.21.15.170:6789/0 conn(0x7f6a2815cd70 legacy=0x7f6a280d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:04.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.689+0000 7f6a2d02d640 1 -- 172.21.15.29:0/3516028437 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a280d1740 con 0x7f6a280d0be0 2023-10-16T14:17:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.690+0000 7f6a2d02d640 1 --2- 172.21.15.29:0/3516028437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280d0be0 0x7f6a280cdd00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009870 tx=0x7f6a2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a587c556d33d057c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.690+0000 7f6a277fe640 1 -- 172.21.15.29:0/3516028437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20031640 con 0x7f6a280d0be0 2023-10-16T14:17:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.690+0000 7f6a277fe640 1 -- 172.21.15.29:0/3516028437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a200317a0 con 0x7f6a280d0be0 2023-10-16T14:17:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.690+0000 7f6a277fe640 1 -- 172.21.15.29:0/3516028437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20031b30 con 0x7f6a280d0be0 2023-10-16T14:17:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 -- 172.21.15.29:0/3516028437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280d0be0 msgr2=0x7f6a280cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/3516028437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280d0be0 0x7f6a280cdd00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009870 tx=0x7f6a2002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 -- 172.21.15.29:0/3516028437 shutdown_connections 2023-10-16T14:17:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/3516028437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280d0be0 0x7f6a280cdd00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/3516028437 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a280b8480 0x7f6a280cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 -- 172.21.15.29:0/3516028437 >> 172.21.15.29:0/3516028437 conn(0x7f6a280cf570 msgr2=0x7f6a280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:04.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 -- 172.21.15.29:0/3516028437 shutdown_connections 2023-10-16T14:17:04.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.691+0000 7f6a2d82e640 1 -- 172.21.15.29:0/3516028437 wait complete. 2023-10-16T14:17:04.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.693+0000 7f6a2d82e640 1 Processor -- start 2023-10-16T14:17:04.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.693+0000 7f6a2d82e640 1 -- start start 2023-10-16T14:17:04.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.694+0000 7f6a2d82e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 0x7f6a2815fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.694+0000 7f6a2d82e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a280d0be0 0x7f6a2814c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.694+0000 7f6a2d82e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a2815cd70 0x7f6a2815e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.694+0000 7f6a2d82e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2814d2e0 con 0x7f6a280b8480 2023-10-16T14:17:04.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.694+0000 7f6a2d82e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2815e700 con 0x7f6a280d0be0 2023-10-16T14:17:04.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.694+0000 7f6a2d82e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a28160300 con 0x7f6a2815cd70 2023-10-16T14:17:04.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 0x7f6a2815fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 0x7f6a2815fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43424/0 (socket says 172.21.15.29:43424) 2023-10-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 -- 172.21.15.29:0/1114203588 learned_addr learned my addr 172.21.15.29:0/1114203588 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2d02d640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a2815cd70 0x7f6a2815e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a27fff640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a280d0be0 0x7f6a2814c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 -- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a2815cd70 msgr2=0x7f6a2815e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a2815cd70 0x7f6a2815e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 -- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a280d0be0 msgr2=0x7f6a2814c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a280d0be0 0x7f6a2814c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.695+0000 7f6a2c82c640 1 -- 172.21.15.29:0/1114203588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200094d0 con 0x7f6a280b8480 2023-10-16T14:17:04.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.696+0000 7f6a2c82c640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 0x7f6a2815fc40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800c7c0 tx=0x7f6a1800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:04.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.696+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800aa50 con 0x7f6a280b8480 2023-10-16T14:17:04.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.696+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1800abb0 con 0x7f6a280b8480 2023-10-16T14:17:04.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.696+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18012420 con 0x7f6a280b8480 2023-10-16T14:17:04.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.696+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2815d400 con 0x7f6a280b8480 2023-10-16T14:17:04.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.696+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2815d9c0 con 0x7f6a280b8480 2023-10-16T14:17:04.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.698+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2804e9b0 con 0x7f6a280b8480 2023-10-16T14:17:04.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.699+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6a18012580 con 0x7f6a280b8480 2023-10-16T14:17:04.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.701+0000 7f6a257fa640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a1406aef0 0x7f6a1406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:04.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.701+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a18089400 con 0x7f6a280b8480 2023-10-16T14:17:04.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.702+0000 7f6a27fff640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a1406aef0 0x7f6a1406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:04.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.703+0000 7f6a27fff640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a1406aef0 0x7f6a1406d3b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6a280ceb70 tx=0x7f6a1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:04.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.707+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a18054540 con 0x7f6a280b8480 2023-10-16T14:17:04.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.907+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a28067e30 con 0x7f6a280b8480 2023-10-16T14:17:04.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.908+0000 7f6a257fa640 1 -- 172.21.15.29:0/1114203588 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a18057b60 con 0x7f6a280b8480 2023-10-16T14:17:04.912 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a1406aef0 msgr2=0x7f6a1406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a1406aef0 0x7f6a1406d3b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6a280ceb70 tx=0x7f6a1c008040 comp rx=0 tx=0).stop 2023-10-16T14:17:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 msgr2=0x7f6a2815fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 0x7f6a2815fc40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800c7c0 tx=0x7f6a1800cc80 comp rx=0 tx=0).stop 2023-10-16T14:17:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 shutdown_connections 2023-10-16T14:17:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a1406aef0 0x7f6a1406d3b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a2815cd70 0x7f6a2815e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a280d0be0 0x7f6a2814c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 --2- 172.21.15.29:0/1114203588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a280b8480 0x7f6a2815fc40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 >> 172.21.15.29:0/1114203588 conn(0x7f6a280cf570 msgr2=0x7f6a280b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 shutdown_connections 2023-10-16T14:17:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:04.911+0000 7f6a2d82e640 1 -- 172.21.15.29:0/1114203588 wait complete. 2023-10-16T14:17:04.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:05.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.156+0000 7f28f6e02640 1 Processor -- start 2023-10-16T14:17:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.157+0000 7f28f6e02640 1 -- start start 2023-10-16T14:17:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.158+0000 7f28f6e02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f28f015cd70 0x7f28f00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.158+0000 7f28f6e02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00cded0 0x7f28f00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.158+0000 7f28f6e02640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f015d150 con 0x7f28f00b8480 2023-10-16T14:17:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.158+0000 7f28f6e02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00d1180 con 0x7f28f015cd70 2023-10-16T14:17:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.158+0000 7f28f6e02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00d1460 con 0x7f28f00cded0 2023-10-16T14:17:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.159+0000 7f28f5e00640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f28f015cd70 0x7f28f00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.159+0000 7f28f55ff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f28f00b8480 0x7f28f00cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:46674/0 (socket says 172.21.15.29:46674) 2023-10-16T14:17:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.159+0000 7f28f6601640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00cded0 0x7f28f00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.159+0000 7f28f55ff640 1 -- 172.21.15.29:0/353160700 learned_addr learned my addr 172.21.15.29:0/353160700 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.160+0000 7f28f6601640 1 -- 172.21.15.29:0/353160700 >> v2:172.21.15.170:3300/0 conn(0x7f28f015cd70 msgr2=0x7f28f00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.160+0000 7f28f6601640 1 --2- 172.21.15.29:0/353160700 >> v2:172.21.15.170:3300/0 conn(0x7f28f015cd70 0x7f28f00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.160+0000 7f28f6601640 1 -- 172.21.15.29:0/353160700 >> v1:172.21.15.170:6789/0 conn(0x7f28f00b8480 legacy=0x7f28f00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.160+0000 7f28f6601640 1 -- 172.21.15.29:0/353160700 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00d1740 con 0x7f28f00cded0 2023-10-16T14:17:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.160+0000 7f28f5e00640 1 --2- 172.21.15.29:0/353160700 >> v2:172.21.15.170:3300/0 conn(0x7f28f015cd70 0x7f28f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:17:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.162+0000 7f28f6601640 1 --2- 172.21.15.29:0/353160700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00cded0 0x7f28f00b5c90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009d90 tx=0x7f28e802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=affbc5efb513876f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.162+0000 7f28f4dfe640 1 -- 172.21.15.29:0/353160700 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e802cc60 con 0x7f28f00cded0 2023-10-16T14:17:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.162+0000 7f28f4dfe640 1 -- 172.21.15.29:0/353160700 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e802cdc0 con 0x7f28f00cded0 2023-10-16T14:17:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.162+0000 7f28f4dfe640 1 -- 172.21.15.29:0/353160700 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8034660 con 0x7f28f00cded0 2023-10-16T14:17:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 -- 172.21.15.29:0/353160700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00cded0 msgr2=0x7f28f00b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 --2- 172.21.15.29:0/353160700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00cded0 0x7f28f00b5c90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009d90 tx=0x7f28e802beb0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 -- 172.21.15.29:0/353160700 shutdown_connections 2023-10-16T14:17:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 --2- 172.21.15.29:0/353160700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00cded0 0x7f28f00b5c90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 --2- 172.21.15.29:0/353160700 >> v2:172.21.15.170:3300/0 conn(0x7f28f015cd70 0x7f28f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 -- 172.21.15.29:0/353160700 >> 172.21.15.29:0/353160700 conn(0x7f28f00cf570 msgr2=0x7f28f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 -- 172.21.15.29:0/353160700 shutdown_connections 2023-10-16T14:17:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.163+0000 7f28f6e02640 1 -- 172.21.15.29:0/353160700 wait complete. 2023-10-16T14:17:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.165+0000 7f28f6e02640 1 Processor -- start 2023-10-16T14:17:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.165+0000 7f28f6e02640 1 -- start start 2023-10-16T14:17:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f6e02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00b8480 0x7f28f01431b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f6e02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 0x7f28f01594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f6e02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f015cd70 0x7f28f0142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f6e02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0159a20 con 0x7f28f00cded0 2023-10-16T14:17:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f6e02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0145520 con 0x7f28f015cd70 2023-10-16T14:17:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f6e02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0145800 con 0x7f28f00b8480 2023-10-16T14:17:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f55ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 0x7f28f01594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.166+0000 7f28f55ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 0x7f28f01594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43438/0 (socket says 172.21.15.29:43438) 2023-10-16T14:17:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 -- 172.21.15.29:0/2239410013 learned_addr learned my addr 172.21.15.29:0/2239410013 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f5e00640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00b8480 0x7f28f01431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f6601640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f015cd70 0x7f28f0142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 -- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00b8480 msgr2=0x7f28f01431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00b8480 0x7f28f01431b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 -- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f015cd70 msgr2=0x7f28f0142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f015cd70 0x7f28f0142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 -- 172.21.15.29:0/2239410013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009400 con 0x7f28f00cded0 2023-10-16T14:17:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f5e00640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00b8480 0x7f28f01431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f6601640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f015cd70 0x7f28f0142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.167+0000 7f28f55ff640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 0x7f28f01594e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f28e000c450 tx=0x7f28e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.168+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0005ce0 con 0x7f28f00cded0 2023-10-16T14:17:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.168+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0005e40 con 0x7f28f00cded0 2023-10-16T14:17:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.168+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e8009a70 con 0x7f28f00cded0 2023-10-16T14:17:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.168+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e00072b0 con 0x7f28f00cded0 2023-10-16T14:17:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.168+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f0145dc0 con 0x7f28f00cded0 2023-10-16T14:17:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.169+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f01f5a80 con 0x7f28f00cded0 2023-10-16T14:17:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.170+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f28e0007410 con 0x7f28f00cded0 2023-10-16T14:17:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.171+0000 7f28e67fc640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f28d406ae70 0x7f28d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.171+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28e0088b60 con 0x7f28f00cded0 2023-10-16T14:17:05.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.172+0000 7f28f5e00640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f28d406ae70 0x7f28d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.173+0000 7f28f5e00640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f28d406ae70 0x7f28d406d330 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009c80 tx=0x7f28ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.176+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28f01f5a80 con 0x7f28f00cded0 2023-10-16T14:17:05.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.369+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28f01f5a80 con 0x7f28f00cded0 2023-10-16T14:17:05.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.371+0000 7f28e67fc640 1 -- 172.21.15.29:0/2239410013 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f28f01f5a80 con 0x7f28f00cded0 2023-10-16T14:17:05.374 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:05.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f28d406ae70 msgr2=0x7f28d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f28d406ae70 0x7f28d406d330 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009c80 tx=0x7f28ec008040 comp rx=0 tx=0).stop 2023-10-16T14:17:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 msgr2=0x7f28f01594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 0x7f28f01594e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f28e000c450 tx=0x7f28e000c910 comp rx=0 tx=0).stop 2023-10-16T14:17:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 shutdown_connections 2023-10-16T14:17:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f28d406ae70 0x7f28d406d330 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00cded0 0x7f28f01594e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f015cd70 0x7f28f0142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 --2- 172.21.15.29:0/2239410013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00b8480 0x7f28f01431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.373+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 >> 172.21.15.29:0/2239410013 conn(0x7f28f00cf570 msgr2=0x7f28f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.374+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 shutdown_connections 2023-10-16T14:17:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.374+0000 7f28f6e02640 1 -- 172.21.15.29:0/2239410013 wait complete. 2023-10-16T14:17:05.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.609+0000 7fafcf593640 1 Processor -- start 2023-10-16T14:17:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.609+0000 7fafcf593640 1 -- start start 2023-10-16T14:17:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.611+0000 7fafcf593640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 0x7fafc80b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.611+0000 7fafcf593640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc80cac20 con 0x7fafc8070240 2023-10-16T14:17:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.611+0000 7fafcf593640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc80c9d70 con 0x7fafc80cd2f0 2023-10-16T14:17:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.611+0000 7fafcf593640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80ca010 con 0x7fafc80b3780 2023-10-16T14:17:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.611+0000 7fafce591640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fafc8070240 0x7fafc8070620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51976/0 (socket says 172.21.15.29:51976) 2023-10-16T14:17:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.611+0000 7fafce591640 1 -- 172.21.15.29:0/3464075587 learned_addr learned my addr 172.21.15.29:0/3464075587 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcdd90640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 0x7fafc80b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827430742 0 0) 0x7fafc80cac20 con 0x7fafc8070240 2023-10-16T14:17:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb4003520 con 0x7fafc8070240 2023-10-16T14:17:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3974538759 0 0) 0x7fafb4003520 con 0x7fafc8070240 2023-10-16T14:17:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 msgr2=0x7fafc80b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 0x7fafc80b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 >> v1:172.21.15.170:6789/0 conn(0x7fafc80cd2f0 legacy=0x7fafc80cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80ca2f0 con 0x7fafc8070240 2023-10-16T14:17:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fafc4001670 con 0x7fafc8070240 2023-10-16T14:17:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 >> v1:172.21.15.29:6789/0 conn(0x7fafc8070240 legacy=0x7fafc8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 0x7fafb40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.612+0000 7fafcd58f640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4003ee0 0x7fafb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafcd58f640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafb4004d00 0x7fafb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80ca010 con 0x7fafb4004d00 2023-10-16T14:17:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb400ba30 con 0x7fafb4003ee0 2023-10-16T14:17:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb400bbf0 con 0x7fafc80b3780 2023-10-16T14:17:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafced92640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafb4004d00 0x7fafb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafce591640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 0x7fafb40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafcdd90640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4003ee0 0x7fafb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafced92640 1 -- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 msgr2=0x7fafb40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafced92640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc80b3780 0x7fafb40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafced92640 1 -- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4003ee0 msgr2=0x7fafb40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafced92640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4003ee0 0x7fafb40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.613+0000 7fafced92640 1 -- 172.21.15.29:0/3464075587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4003520 con 0x7fafb4004d00 2023-10-16T14:17:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.614+0000 7fafcdd90640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4003ee0 0x7fafb40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:05.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.615+0000 7fafced92640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafb4004d00 0x7fafb4005180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafc0001f60 tx=0x7fafb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490b56556a7606bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:05.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.615+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc002d6f0 con 0x7fafb4004d00 2023-10-16T14:17:05.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.615+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc002d870 con 0x7fafb4004d00 2023-10-16T14:17:05.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.615+0000 7fafcd58f640 1 -- 172.21.15.29:0/3464075587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc002dc20 con 0x7fafb4004d00 2023-10-16T14:17:05.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafcf593640 1 -- 172.21.15.29:0/3464075587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafb4004d00 msgr2=0x7fafb4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafcf593640 1 --2- 172.21.15.29:0/3464075587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafb4004d00 0x7fafb4005180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafc0001f60 tx=0x7fafb40057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafced92640 1 -- 172.21.15.29:0/3464075587 reap_dead start 2023-10-16T14:17:05.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafcf593640 1 -- 172.21.15.29:0/3464075587 shutdown_connections 2023-10-16T14:17:05.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafcf593640 1 -- 172.21.15.29:0/3464075587 >> 172.21.15.29:0/3464075587 conn(0x7fafc80cc230 msgr2=0x7fafc80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:05.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafcf593640 1 -- 172.21.15.29:0/3464075587 shutdown_connections 2023-10-16T14:17:05.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.616+0000 7fafcf593640 1 -- 172.21.15.29:0/3464075587 wait complete. 2023-10-16T14:17:05.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.618+0000 7fafcf593640 1 Processor -- start 2023-10-16T14:17:05.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.618+0000 7fafcf593640 1 -- start start 2023-10-16T14:17:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcf593640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4004d00 0x7fafc815f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcf593640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815dea0 0x7fafc806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcf593640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 0x7fafc806cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcf593640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc806c030 con 0x7fafc815b7e0 2023-10-16T14:17:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcf593640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80c60a0 con 0x7fafb4004d00 2023-10-16T14:17:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcf593640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80c5dc0 con 0x7fafc815dea0 2023-10-16T14:17:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafced92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 0x7fafc806cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafced92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 0x7fafc806cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43458/0 (socket says 172.21.15.29:43458) 2023-10-16T14:17:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafced92640 1 -- 172.21.15.29:0/1527774231 learned_addr learned my addr 172.21.15.29:0/1527774231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.619+0000 7fafcdd90640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815dea0 0x7fafc806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafce591640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4004d00 0x7fafc815f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafced92640 1 -- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815dea0 msgr2=0x7fafc806a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafced92640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815dea0 0x7fafc806a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafced92640 1 -- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4004d00 msgr2=0x7fafc815f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafced92640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4004d00 0x7fafc815f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafced92640 1 -- 172.21.15.29:0/1527774231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc806c260 con 0x7fafc815b7e0 2023-10-16T14:17:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafce591640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4004d00 0x7fafc815f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafced92640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 0x7fafc806cd40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fafc002c080 tx=0x7fafc00015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc002dea0 con 0x7fafc815b7e0 2023-10-16T14:17:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0003b70 con 0x7fafc815b7e0 2023-10-16T14:17:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0005070 con 0x7fafc815b7e0 2023-10-16T14:17:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.620+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc806c540 con 0x7fafc815b7e0 2023-10-16T14:17:05.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.621+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc80c5a00 con 0x7fafc815b7e0 2023-10-16T14:17:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.622+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc80c5b90 con 0x7fafc815b7e0 2023-10-16T14:17:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.623+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fafc00051d0 con 0x7fafc815b7e0 2023-10-16T14:17:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.623+0000 7fafbeffd640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fafac06ae70 0x7fafac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.624+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafc0031070 con 0x7fafc815b7e0 2023-10-16T14:17:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.624+0000 7fafce591640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fafac06ae70 0x7fafac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.625+0000 7fafce591640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fafac06ae70 0x7fafac06d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fafc4007c80 tx=0x7fafc4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.628+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fafc00766a0 con 0x7fafc815b7e0 2023-10-16T14:17:05.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.847+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafc806ca60 con 0x7fafc815b7e0 2023-10-16T14:17:05.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.850+0000 7fafbeffd640 1 -- 172.21.15.29:0/1527774231 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fafc0079cc0 con 0x7fafc815b7e0 2023-10-16T14:17:05.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:05.853 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.852+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fafac06ae70 msgr2=0x7fafac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.852+0000 7fafcf593640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fafac06ae70 0x7fafac06d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fafc4007c80 tx=0x7fafc4009040 comp rx=0 tx=0).stop 2023-10-16T14:17:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.852+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 msgr2=0x7fafc806cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.852+0000 7fafcf593640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 0x7fafc806cd40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fafc002c080 tx=0x7fafc00015b0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 shutdown_connections 2023-10-16T14:17:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fafac06ae70 0x7fafac06d330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815b7e0 0x7fafc806cd40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815dea0 0x7fafc806a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 --2- 172.21.15.29:0/1527774231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafb4004d00 0x7fafc815f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 >> 172.21.15.29:0/1527774231 conn(0x7fafc80cc230 msgr2=0x7fafc80b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 shutdown_connections 2023-10-16T14:17:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:05.853+0000 7fafcf593640 1 -- 172.21.15.29:0/1527774231 wait complete. 2023-10-16T14:17:05.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:05.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:06.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.085+0000 7fad1d11d640 1 Processor -- start 2023-10-16T14:17:06.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.085+0000 7fad1d11d640 1 -- start start 2023-10-16T14:17:06.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.086+0000 7fad1d11d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad180ad350 0x7fad18069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad1d11d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad1d11d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1806a730 con 0x7fad1815d150 2023-10-16T14:17:06.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad1d11d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1815d530 con 0x7fad180ad350 2023-10-16T14:17:06.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad1d11d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad180ac890 con 0x7fad1806a2e0 2023-10-16T14:17:06.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad180ad350 0x7fad18069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad180ad350 0x7fad18069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43470/0 (socket says 172.21.15.29:43470) 2023-10-16T14:17:06.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 -- 172.21.15.29:0/3854549947 learned_addr learned my addr 172.21.15.29:0/3854549947 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:06.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad1759e640 1 --2- 172.21.15.29:0/3854549947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 -- 172.21.15.29:0/3854549947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 msgr2=0x7fad180c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 --2- 172.21.15.29:0/3854549947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 -- 172.21.15.29:0/3854549947 >> v1:172.21.15.170:6789/0 conn(0x7fad1815d150 legacy=0x7fad18160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.087+0000 7fad0ffff640 1 -- 172.21.15.29:0/3854549947 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad180ad7a0 con 0x7fad180ad350 2023-10-16T14:17:06.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.089+0000 7fad0ffff640 1 --2- 172.21.15.29:0/3854549947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad180ad350 0x7fad18069da0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fad180cf990 tx=0x7fad0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb7b1558ad95b95 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.089+0000 7fad1659c640 1 -- 172.21.15.29:0/3854549947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00031830 con 0x7fad180ad350 2023-10-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.089+0000 7fad1659c640 1 -- 172.21.15.29:0/3854549947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00031990 con 0x7fad180ad350 2023-10-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.089+0000 7fad1659c640 1 -- 172.21.15.29:0/3854549947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00031d20 con 0x7fad180ad350 2023-10-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.089+0000 7fad1d11d640 1 -- 172.21.15.29:0/3854549947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad180ad350 msgr2=0x7fad18069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.089+0000 7fad1d11d640 1 --2- 172.21.15.29:0/3854549947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad180ad350 0x7fad18069da0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fad180cf990 tx=0x7fad0002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.090+0000 7fad1d11d640 1 -- 172.21.15.29:0/3854549947 shutdown_connections 2023-10-16T14:17:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.090+0000 7fad1d11d640 1 --2- 172.21.15.29:0/3854549947 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.090+0000 7fad1d11d640 1 --2- 172.21.15.29:0/3854549947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad180ad350 0x7fad18069da0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.090+0000 7fad1d11d640 1 -- 172.21.15.29:0/3854549947 >> 172.21.15.29:0/3854549947 conn(0x7fad180ce9b0 msgr2=0x7fad180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.090+0000 7fad1d11d640 1 -- 172.21.15.29:0/3854549947 shutdown_connections 2023-10-16T14:17:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.090+0000 7fad1d11d640 1 -- 172.21.15.29:0/3854549947 wait complete. 2023-10-16T14:17:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.092+0000 7fad1d11d640 1 Processor -- start 2023-10-16T14:17:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.092+0000 7fad1d11d640 1 -- start start 2023-10-16T14:17:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1d11d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1d11d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad180ad350 0x7fad181f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1d11d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 0x7fad181f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1d11d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad180b2d80 con 0x7fad1815d150 2023-10-16T14:17:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1d11d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad180b0730 con 0x7fad180ad350 2023-10-16T14:17:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1d11d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad181f35f0 con 0x7fad1806a2e0 2023-10-16T14:17:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 0x7fad181f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 0x7fad181f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43484/0 (socket says 172.21.15.29:43484) 2023-10-16T14:17:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad1759e640 1 -- 172.21.15.29:0/583411104 learned_addr learned my addr 172.21.15.29:0/583411104 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad16d9d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.093+0000 7fad0ffff640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad180ad350 0x7fad181f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1759e640 1 -- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 msgr2=0x7fad180ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1759e640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180ae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1759e640 1 -- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad180ad350 msgr2=0x7fad181f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1759e640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad180ad350 0x7fad181f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1759e640 1 -- 172.21.15.29:0/583411104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000094d0 con 0x7fad1815d150 2023-10-16T14:17:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad16d9d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad0ffff640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad180ad350 0x7fad181f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1759e640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 0x7fad181f2f50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fad10007e60 tx=0x7fad1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10014420 con 0x7fad1815d150 2023-10-16T14:17:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad100145a0 con 0x7fad1815d150 2023-10-16T14:17:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10014950 con 0x7fad1815d150 2023-10-16T14:17:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad181f38d0 con 0x7fad1815d150 2023-10-16T14:17:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.094+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad180b29a0 con 0x7fad1815d150 2023-10-16T14:17:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.095+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad180b2ec0 con 0x7fad1815d150 2023-10-16T14:17:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.098+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fad10014ab0 con 0x7fad1815d150 2023-10-16T14:17:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.099+0000 7fad0f7fe640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7facf806ae70 0x7facf806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.099+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad10089f20 con 0x7fad1815d150 2023-10-16T14:17:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.099+0000 7fad16d9d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7facf806ae70 0x7facf806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.101+0000 7fad16d9d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7facf806ae70 0x7facf806d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fad08003eb0 tx=0x7fad08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.103+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fad10054fc0 con 0x7fad1815d150 2023-10-16T14:17:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.296+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad1804e160 con 0x7fad1815d150 2023-10-16T14:17:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.297+0000 7fad0f7fe640 1 -- 172.21.15.29:0/583411104 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad100585e0 con 0x7fad1815d150 2023-10-16T14:17:06.300 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7facf806ae70 msgr2=0x7facf806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7facf806ae70 0x7facf806d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fad08003eb0 tx=0x7fad08009040 comp rx=0 tx=0).stop 2023-10-16T14:17:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 msgr2=0x7fad181f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 0x7fad181f2f50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fad10007e60 tx=0x7fad1000e4b0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 shutdown_connections 2023-10-16T14:17:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7facf806ae70 0x7facf806d330 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad1815d150 0x7fad181f2f50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad180ad350 0x7fad181f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 --2- 172.21.15.29:0/583411104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad1806a2e0 0x7fad180ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 >> 172.21.15.29:0/583411104 conn(0x7fad180ce9b0 msgr2=0x7fad1806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:06.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 shutdown_connections 2023-10-16T14:17:06.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.300+0000 7fad1d11d640 1 -- 172.21.15.29:0/583411104 wait complete. 2023-10-16T14:17:06.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.527+0000 7f74c20d7640 1 Processor -- start 2023-10-16T14:17:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.527+0000 7f74c20d7640 1 -- start start 2023-10-16T14:17:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c20d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc1573e0 0x7f74bc0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c20d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74bc070950 0x7f74bc15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c20d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc0ca3a0 con 0x7f74bc159e70 2023-10-16T14:17:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c20d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74bc0cad70 con 0x7f74bc070950 2023-10-16T14:17:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c20d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0c9d70 con 0x7f74bc1573e0 2023-10-16T14:17:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c18d6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74bc070950 0x7f74bc15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c08d4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f74bc159e70 0x7f74bc070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51978/0 (socket says 172.21.15.29:51978) 2023-10-16T14:17:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.529+0000 7f74c08d4640 1 -- 172.21.15.29:0/454222125 learned_addr learned my addr 172.21.15.29:0/454222125 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:06.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.530+0000 7f74c10d5640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc1573e0 0x7f74bc0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.530+0000 7f74c18d6640 1 -- 172.21.15.29:0/454222125 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc1573e0 msgr2=0x7f74bc0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.530+0000 7f74c18d6640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc1573e0 0x7f74bc0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.530+0000 7f74c18d6640 1 -- 172.21.15.29:0/454222125 >> v1:172.21.15.29:6789/0 conn(0x7f74bc159e70 legacy=0x7f74bc070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.530+0000 7f74c18d6640 1 -- 172.21.15.29:0/454222125 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc1577c0 con 0x7f74bc070950 2023-10-16T14:17:06.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.530+0000 7f74c10d5640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc1573e0 0x7f74bc0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:06.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.531+0000 7f74c18d6640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc070950 0x7f74bc15a2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009d90 tx=0x7f74b402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17ce00cb4c2f2f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.531+0000 7f74b3fff640 1 -- 172.21.15.29:0/454222125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b402cc50 con 0x7f74bc070950 2023-10-16T14:17:06.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.531+0000 7f74b3fff640 1 -- 172.21.15.29:0/454222125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b402cdb0 con 0x7f74bc070950 2023-10-16T14:17:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.531+0000 7f74b3fff640 1 -- 172.21.15.29:0/454222125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4034660 con 0x7f74bc070950 2023-10-16T14:17:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 -- 172.21.15.29:0/454222125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc070950 msgr2=0x7f74bc15a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc070950 0x7f74bc15a2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009d90 tx=0x7f74b402bea0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 -- 172.21.15.29:0/454222125 shutdown_connections 2023-10-16T14:17:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc070950 0x7f74bc15a2f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 --2- 172.21.15.29:0/454222125 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc1573e0 0x7f74bc0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 -- 172.21.15.29:0/454222125 >> 172.21.15.29:0/454222125 conn(0x7f74bc0cc230 msgr2=0x7f74bc0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 -- 172.21.15.29:0/454222125 shutdown_connections 2023-10-16T14:17:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.532+0000 7f74c20d7640 1 -- 172.21.15.29:0/454222125 wait complete. 2023-10-16T14:17:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.534+0000 7f74c20d7640 1 Processor -- start 2023-10-16T14:17:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.534+0000 7f74c20d7640 1 -- start start 2023-10-16T14:17:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c20d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc070950 0x7f74bc15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c20d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 0x7f74bc14ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c20d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74bc159e70 0x7f74bc14f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c20d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0c3250 con 0x7f74bc1573e0 2023-10-16T14:17:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c20d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0c3460 con 0x7f74bc159e70 2023-10-16T14:17:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c20d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0c3740 con 0x7f74bc070950 2023-10-16T14:17:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 0x7f74bc14ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 0x7f74bc14ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43502/0 (socket says 172.21.15.29:43502) 2023-10-16T14:17:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 -- 172.21.15.29:0/235006207 learned_addr learned my addr 172.21.15.29:0/235006207 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c10d5640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc070950 0x7f74bc15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c18d6640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74bc159e70 0x7f74bc14f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 -- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc070950 msgr2=0x7f74bc15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc070950 0x7f74bc15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 -- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74bc159e70 msgr2=0x7f74bc14f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74bc159e70 0x7f74bc14f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.535+0000 7f74c08d4640 1 -- 172.21.15.29:0/235006207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4009a70 con 0x7f74bc1573e0 2023-10-16T14:17:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.536+0000 7f74c08d4640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 0x7f74bc14ee60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c4a0 tx=0x7f74ac00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.536+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac005ce0 con 0x7f74bc1573e0 2023-10-16T14:17:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.536+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac005e40 con 0x7f74bc1573e0 2023-10-16T14:17:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.536+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac007270 con 0x7f74bc1573e0 2023-10-16T14:17:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.536+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc0c3a20 con 0x7f74bc1573e0 2023-10-16T14:17:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.536+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc0c3db0 con 0x7f74bc1573e0 2023-10-16T14:17:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.537+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc0c35a0 con 0x7f74bc1573e0 2023-10-16T14:17:06.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.538+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f74ac007490 con 0x7f74bc1573e0 2023-10-16T14:17:06.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.539+0000 7f74b1ffb640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f74a006ae70 0x7f74a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.539+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74ac088b80 con 0x7f74bc1573e0 2023-10-16T14:17:06.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.540+0000 7f74c10d5640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f74a006ae70 0x7f74a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.541+0000 7f74c10d5640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f74a006ae70 0x7f74a006d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f74b800a990 tx=0x7f74b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.544+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74bc0c35a0 con 0x7f74bc1573e0 2023-10-16T14:17:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.736+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f74bc0c35a0 con 0x7f74bc1573e0 2023-10-16T14:17:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.737+0000 7f74b1ffb640 1 -- 172.21.15.29:0/235006207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f74bc0c35a0 con 0x7f74bc1573e0 2023-10-16T14:17:06.741 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f74a006ae70 msgr2=0x7f74a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f74a006ae70 0x7f74a006d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f74b800a990 tx=0x7f74b8008040 comp rx=0 tx=0).stop 2023-10-16T14:17:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 msgr2=0x7f74bc14ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 0x7f74bc14ee60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c4a0 tx=0x7f74ac00c960 comp rx=0 tx=0).stop 2023-10-16T14:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 shutdown_connections 2023-10-16T14:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f74a006ae70 0x7f74a006d330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74bc1573e0 0x7f74bc14ee60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74bc159e70 0x7f74bc14f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 --2- 172.21.15.29:0/235006207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74bc070950 0x7f74bc15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 >> 172.21.15.29:0/235006207 conn(0x7f74bc0cc230 msgr2=0x7f74bc158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 shutdown_connections 2023-10-16T14:17:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.740+0000 7f74c20d7640 1 -- 172.21.15.29:0/235006207 wait complete. 2023-10-16T14:17:06.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.971+0000 7fd3c92d7640 1 Processor -- start 2023-10-16T14:17:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.972+0000 7fd3c92d7640 1 -- start start 2023-10-16T14:17:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c92d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd3c40b3420 0x7fd3c40b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c92d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 0x7fd3c40b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c92d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c40d3980 con 0x7fd3c40cf710 2023-10-16T14:17:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c92d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c40d3c60 con 0x7fd3c40b3420 2023-10-16T14:17:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c92d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40ce630 con 0x7fd3c40cfaf0 2023-10-16T14:17:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd3c40cf710 0x7fd3c40b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51990/0 (socket says 172.21.15.29:51990) 2023-10-16T14:17:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.973+0000 7fd3c2ffd640 1 -- 172.21.15.29:0/1705340874 learned_addr learned my addr 172.21.15.29:0/1705340874 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c27fc640 1 --2- 172.21.15.29:0/1705340874 >> v2:172.21.15.170:3300/0 conn(0x7fd3c40b3420 0x7fd3c40b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c37fe640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 0x7fd3c40b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744950216 0 0) 0x7fd3c40d3980 con 0x7fd3c40cf710 2023-10-16T14:17:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b0003520 con 0x7fd3c40cf710 2023-10-16T14:17:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934239048 0 0) 0x7fd3b0003520 con 0x7fd3c40cf710 2023-10-16T14:17:06.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 msgr2=0x7fd3c40b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 0x7fd3c40b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 >> v2:172.21.15.170:3300/0 conn(0x7fd3c40b3420 msgr2=0x7fd3c40b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 --2- 172.21.15.29:0/1705340874 >> v2:172.21.15.170:3300/0 conn(0x7fd3c40b3420 0x7fd3c40b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c40ce910 con 0x7fd3c40cf710 2023-10-16T14:17:06.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fd3b40017f0 con 0x7fd3c40cf710 2023-10-16T14:17:06.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 >> v1:172.21.15.29:6789/0 conn(0x7fd3c40cf710 legacy=0x7fd3c40b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.974+0000 7fd3c1ffb640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b0003970 0x7fd3b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c1ffb640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 0x7fd3b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c1ffb640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b0004d00 0x7fd3b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40d3c60 con 0x7fd3b0004d00 2023-10-16T14:17:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40ce630 con 0x7fd3b0003970 2023-10-16T14:17:06.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b000b9b0 con 0x7fd3c40cfaf0 2023-10-16T14:17:06.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c37fe640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b0004d00 0x7fd3b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c27fc640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 0x7fd3b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c2ffd640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b0003970 0x7fd3b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c37fe640 1 -- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 msgr2=0x7fd3b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c37fe640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3c40cfaf0 0x7fd3b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c37fe640 1 -- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b0003970 msgr2=0x7fd3b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:06.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c37fe640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3b0003970 0x7fd3b0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.975+0000 7fd3c37fe640 1 -- 172.21.15.29:0/1705340874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b0003520 con 0x7fd3b0004d00 2023-10-16T14:17:06.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.976+0000 7fd3c37fe640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b0004d00 0x7fd3b0005180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009fc0 tx=0x7fd3b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbf42021813e095 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc02cbf0 con 0x7fd3b0004d00 2023-10-16T14:17:06.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc00f040 con 0x7fd3b0004d00 2023-10-16T14:17:06.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c1ffb640 1 -- 172.21.15.29:0/1705340874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc034440 con 0x7fd3b0004d00 2023-10-16T14:17:06.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c92d7640 1 -- 172.21.15.29:0/1705340874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b0004d00 msgr2=0x7fd3b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/1705340874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3b0004d00 0x7fd3b0005180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009fc0 tx=0x7fd3b00057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c37fe640 1 -- 172.21.15.29:0/1705340874 reap_dead start 2023-10-16T14:17:06.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c92d7640 1 -- 172.21.15.29:0/1705340874 shutdown_connections 2023-10-16T14:17:06.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.977+0000 7fd3c92d7640 1 -- 172.21.15.29:0/1705340874 >> 172.21.15.29:0/1705340874 conn(0x7fd3c40d2cc0 msgr2=0x7fd3c40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:06.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.978+0000 7fd3c92d7640 1 -- 172.21.15.29:0/1705340874 shutdown_connections 2023-10-16T14:17:06.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.978+0000 7fd3c92d7640 1 -- 172.21.15.29:0/1705340874 wait complete. 2023-10-16T14:17:06.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.979+0000 7fd3c92d7640 1 Processor -- start 2023-10-16T14:17:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.979+0000 7fd3c92d7640 1 -- start start 2023-10-16T14:17:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.980+0000 7fd3c92d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b0004d00 0x7fd3c40af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.980+0000 7fd3c92d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 0x7fd3c4145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.980+0000 7fd3c92d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3c4143990 0x7fd3c4143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.980+0000 7fd3c92d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40afdd0 con 0x7fd3c415a8d0 2023-10-16T14:17:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.980+0000 7fd3c92d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40b2430 con 0x7fd3c4143990 2023-10-16T14:17:06.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.980+0000 7fd3c92d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40b26d0 con 0x7fd3b0004d00 2023-10-16T14:17:06.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 0x7fd3c4145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 0x7fd3c4145c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43520/0 (socket says 172.21.15.29:43520) 2023-10-16T14:17:06.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3525100533 learned_addr learned my addr 172.21.15.29:0/3525100533 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:06.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c2ffd640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b0004d00 0x7fd3c40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c37fe640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3c4143990 0x7fd3c4143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b0004d00 msgr2=0x7fd3c40af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b0004d00 0x7fd3c40af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3c4143990 msgr2=0x7fd3c4143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:06.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3c4143990 0x7fd3c4143e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:06.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 -- 172.21.15.29:0/3525100533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c415a0d0 con 0x7fd3c415a8d0 2023-10-16T14:17:06.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c2ffd640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b0004d00 0x7fd3c40af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:06.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.981+0000 7fd3c27fc640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 0x7fd3c4145c30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0116d0 tx=0x7fd3ac011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.982+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00f9c0 con 0x7fd3c415a8d0 2023-10-16T14:17:06.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.982+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac00fb20 con 0x7fd3c415a8d0 2023-10-16T14:17:06.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.982+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c415a3b0 con 0x7fd3c415a8d0 2023-10-16T14:17:06.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.982+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00fe90 con 0x7fd3c415a8d0 2023-10-16T14:17:06.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.982+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c415ad50 con 0x7fd3c415a8d0 2023-10-16T14:17:06.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.983+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c4159f30 con 0x7fd3c415a8d0 2023-10-16T14:17:06.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.984+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd3ac015460 con 0x7fd3c415a8d0 2023-10-16T14:17:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.985+0000 7fd3bb7fe640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd3a406aea0 0x7fd3a406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.985+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3ac089f10 con 0x7fd3c415a8d0 2023-10-16T14:17:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.985+0000 7fd3c2ffd640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd3a406aea0 0x7fd3a406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.987+0000 7fd3c2ffd640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd3a406aea0 0x7fd3a406d360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000ef0 tx=0x7fd3b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:06.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:06.990+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3c4159f30 con 0x7fd3c415a8d0 2023-10-16T14:17:07.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.185+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3c4159f30 con 0x7fd3c415a8d0 2023-10-16T14:17:07.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.186+0000 7fd3bb7fe640 1 -- 172.21.15.29:0/3525100533 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3c4159f30 con 0x7fd3c415a8d0 2023-10-16T14:17:07.189 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd3a406aea0 msgr2=0x7fd3a406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd3a406aea0 0x7fd3a406d360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000ef0 tx=0x7fd3b4009040 comp rx=0 tx=0).stop 2023-10-16T14:17:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 msgr2=0x7fd3c4145c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 0x7fd3c4145c30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0116d0 tx=0x7fd3ac011b90 comp rx=0 tx=0).stop 2023-10-16T14:17:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 shutdown_connections 2023-10-16T14:17:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd3a406aea0 0x7fd3a406d360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3c415a8d0 0x7fd3c4145c30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3c4143990 0x7fd3c4143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 --2- 172.21.15.29:0/3525100533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3b0004d00 0x7fd3c40af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.188+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 >> 172.21.15.29:0/3525100533 conn(0x7fd3c40d2cc0 msgr2=0x7fd3c415cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:07.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.189+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 shutdown_connections 2023-10-16T14:17:07.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.189+0000 7fd3c92d7640 1 -- 172.21.15.29:0/3525100533 wait complete. 2023-10-16T14:17:07.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:07.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.431+0000 7fda078cf640 1 Processor -- start 2023-10-16T14:17:07.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.431+0000 7fda078cf640 1 -- start start 2023-10-16T14:17:07.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.432+0000 7fda078cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda00159b20 0x7fda0006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.432+0000 7fda078cf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fda00159f00 0x7fda000b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.432+0000 7fda078cf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.432+0000 7fda078cf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0006f7f0 con 0x7fda00159b20 2023-10-16T14:17:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.432+0000 7fda078cf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0006c5e0 con 0x7fda00159f00 2023-10-16T14:17:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.432+0000 7fda078cf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0006c8c0 con 0x7fda0006c120 2023-10-16T14:17:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda00159b20 0x7fda0006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda00159b20 0x7fda0006bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43524/0 (socket says 172.21.15.29:43524) 2023-10-16T14:17:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 -- 172.21.15.29:0/2196525415 learned_addr learned my addr 172.21.15.29:0/2196525415 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda04e6a640 1 --2- 172.21.15.29:0/2196525415 >> v2:172.21.15.170:3300/0 conn(0x7fda00159f00 0x7fda000b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda05e6c640 1 --2- 172.21.15.29:0/2196525415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 -- 172.21.15.29:0/2196525415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 msgr2=0x7fda000b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 --2- 172.21.15.29:0/2196525415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 -- 172.21.15.29:0/2196525415 >> v2:172.21.15.170:3300/0 conn(0x7fda00159f00 msgr2=0x7fda000b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 --2- 172.21.15.29:0/2196525415 >> v2:172.21.15.170:3300/0 conn(0x7fda00159f00 0x7fda000b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.433+0000 7fda0566b640 1 -- 172.21.15.29:0/2196525415 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0006cba0 con 0x7fda00159b20 2023-10-16T14:17:07.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.435+0000 7fda0566b640 1 --2- 172.21.15.29:0/2196525415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0006bbe0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fda0015d770 tx=0x7fd9fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377dcf3eeb6be16e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:07.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.435+0000 7fd9f7fff640 1 -- 172.21.15.29:0/2196525415 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc02ca70 con 0x7fda00159b20 2023-10-16T14:17:07.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.435+0000 7fd9f7fff640 1 -- 172.21.15.29:0/2196525415 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc02cbd0 con 0x7fda00159b20 2023-10-16T14:17:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.435+0000 7fd9f7fff640 1 -- 172.21.15.29:0/2196525415 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc0343f0 con 0x7fda00159b20 2023-10-16T14:17:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 -- 172.21.15.29:0/2196525415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 msgr2=0x7fda0006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 --2- 172.21.15.29:0/2196525415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0006bbe0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fda0015d770 tx=0x7fd9fc02bed0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 -- 172.21.15.29:0/2196525415 shutdown_connections 2023-10-16T14:17:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 --2- 172.21.15.29:0/2196525415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 --2- 172.21.15.29:0/2196525415 >> v2:172.21.15.170:3300/0 conn(0x7fda00159f00 0x7fda000b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 --2- 172.21.15.29:0/2196525415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0006bbe0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 -- 172.21.15.29:0/2196525415 >> 172.21.15.29:0/2196525415 conn(0x7fda000cb410 msgr2=0x7fda000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 -- 172.21.15.29:0/2196525415 shutdown_connections 2023-10-16T14:17:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.436+0000 7fda078cf640 1 -- 172.21.15.29:0/2196525415 wait complete. 2023-10-16T14:17:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.438+0000 7fda078cf640 1 Processor -- start 2023-10-16T14:17:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.438+0000 7fda078cf640 1 -- start start 2023-10-16T14:17:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda078cf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda078cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0014b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda078cf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda00159f00 0x7fda0014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda078cf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000b4410 con 0x7fda00159b20 2023-10-16T14:17:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda078cf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0014c2e0 con 0x7fda00159f00 2023-10-16T14:17:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda078cf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0015a4b0 con 0x7fda0006c120 2023-10-16T14:17:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0014b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0014b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43532/0 (socket says 172.21.15.29:43532) 2023-10-16T14:17:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 -- 172.21.15.29:0/3385667400 learned_addr learned my addr 172.21.15.29:0/3385667400 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 -- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 msgr2=0x7fda000c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 -- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda00159f00 msgr2=0x7fda0014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda00159f00 0x7fda0014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.439+0000 7fda04e6a640 1 -- 172.21.15.29:0/3385667400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc009b30 con 0x7fda00159b20 2023-10-16T14:17:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.440+0000 7fda04e6a640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0014b860 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000d7d0 tx=0x7fd9f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.440+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f000b6c0 con 0x7fda00159b20 2023-10-16T14:17:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.440+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f000b820 con 0x7fda00159b20 2023-10-16T14:17:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.440+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0015a790 con 0x7fda00159b20 2023-10-16T14:17:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.440+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f000bbb0 con 0x7fda00159b20 2023-10-16T14:17:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.440+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0015ad50 con 0x7fda00159b20 2023-10-16T14:17:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.441+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0015aee0 con 0x7fda00159b20 2023-10-16T14:17:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.442+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd9f000bd10 con 0x7fda00159b20 2023-10-16T14:17:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.444+0000 7fd9f5ffb640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd9e406ae70 0x7fd9e406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.444+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9f0089b00 con 0x7fda00159b20 2023-10-16T14:17:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.444+0000 7fda0566b640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd9e406ae70 0x7fd9e406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.445+0000 7fda0566b640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd9e406ae70 0x7fd9e406d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc035000 tx=0x7fd9fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.448+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd9f0054bb0 con 0x7fda00159b20 2023-10-16T14:17:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.646+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda0006f5f0 con 0x7fda00159b20 2023-10-16T14:17:07.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.648+0000 7fd9f5ffb640 1 -- 172.21.15.29:0/3385667400 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd9f00581d0 con 0x7fda00159b20 2023-10-16T14:17:07.651 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:07.652 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd9e406ae70 msgr2=0x7fd9e406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd9e406ae70 0x7fd9e406d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc035000 tx=0x7fd9fc02e000 comp rx=0 tx=0).stop 2023-10-16T14:17:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 msgr2=0x7fda0014b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0014b860 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000d7d0 tx=0x7fd9f000dc90 comp rx=0 tx=0).stop 2023-10-16T14:17:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 shutdown_connections 2023-10-16T14:17:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd9e406ae70 0x7fd9e406d330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda00159b20 0x7fda0014b860 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda00159f00 0x7fda0014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 --2- 172.21.15.29:0/3385667400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fda0006c120 0x7fda000c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 >> 172.21.15.29:0/3385667400 conn(0x7fda000cb410 msgr2=0x7fda000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 shutdown_connections 2023-10-16T14:17:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.651+0000 7fda078cf640 1 -- 172.21.15.29:0/3385667400 wait complete. 2023-10-16T14:17:07.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:07.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.906+0000 7fd43d5ae640 1 Processor -- start 2023-10-16T14:17:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.906+0000 7fd43d5ae640 1 -- start start 2023-10-16T14:17:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.908+0000 7fd43d5ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd4380b0ea0 0x7fd43815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.908+0000 7fd43d5ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380cd970 0x7fd43815a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.908+0000 7fd43d5ae640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43806be30 con 0x7fd4380b3ec0 2023-10-16T14:17:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.908+0000 7fd43d5ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43806ca00 con 0x7fd4380b0ea0 2023-10-16T14:17:07.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.908+0000 7fd43d5ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4380ada10 con 0x7fd4380cd970 2023-10-16T14:17:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.908+0000 7fd4377fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd4380b0ea0 0x7fd43815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd4377fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd4380b0ea0 0x7fd43815a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:34376/0 (socket says 172.21.15.29:34376) 2023-10-16T14:17:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd4377fe640 1 -- 172.21.15.29:0/3270343072 learned_addr learned my addr 172.21.15.29:0/3270343072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd43cdad640 1 --2- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380cd970 0x7fd43815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd4377fe640 1 -- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380cd970 msgr2=0x7fd43815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd4377fe640 1 --2- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380cd970 0x7fd43815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd4377fe640 1 -- 172.21.15.29:0/3270343072 >> v1:172.21.15.170:6789/0 conn(0x7fd4380b3ec0 legacy=0x7fd4380cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.909+0000 7fd4377fe640 1 -- 172.21.15.29:0/3270343072 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43806c060 con 0x7fd4380b0ea0 2023-10-16T14:17:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.911+0000 7fd4377fe640 1 --2- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380b0ea0 0x7fd43815a2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009870 tx=0x7fd42c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8840bad30252514c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.911+0000 7fd436ffd640 1 -- 172.21.15.29:0/3270343072 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c031640 con 0x7fd4380b0ea0 2023-10-16T14:17:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.911+0000 7fd436ffd640 1 -- 172.21.15.29:0/3270343072 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c0317a0 con 0x7fd4380b0ea0 2023-10-16T14:17:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.911+0000 7fd436ffd640 1 -- 172.21.15.29:0/3270343072 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c031b30 con 0x7fd4380b0ea0 2023-10-16T14:17:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380b0ea0 msgr2=0x7fd43815a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380b0ea0 0x7fd43815a2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009870 tx=0x7fd42c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3270343072 shutdown_connections 2023-10-16T14:17:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380cd970 0x7fd43815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3270343072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380b0ea0 0x7fd43815a2f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3270343072 >> 172.21.15.29:0/3270343072 conn(0x7fd4380b0170 msgr2=0x7fd4380b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3270343072 shutdown_connections 2023-10-16T14:17:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.912+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3270343072 wait complete. 2023-10-16T14:17:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.915+0000 7fd43d5ae640 1 Processor -- start 2023-10-16T14:17:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.915+0000 7fd43d5ae640 1 -- start start 2023-10-16T14:17:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd43d5ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380b0ea0 0x7fd4381f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd43d5ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 0x7fd4380ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd43d5ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380cd970 0x7fd4380acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd43d5ae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4381f6b20 con 0x7fd4380b3ec0 2023-10-16T14:17:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd43d5ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43806f780 con 0x7fd4380cd970 2023-10-16T14:17:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd43d5ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4381f6280 con 0x7fd4380b0ea0 2023-10-16T14:17:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd4377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 0x7fd4380ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd4377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 0x7fd4380ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43536/0 (socket says 172.21.15.29:43536) 2023-10-16T14:17:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.916+0000 7fd4377fe640 1 -- 172.21.15.29:0/3402544954 learned_addr learned my addr 172.21.15.29:0/3402544954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd43cdad640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380cd970 0x7fd4380acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd437fff640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380b0ea0 0x7fd4381f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd4377fe640 1 -- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380b0ea0 msgr2=0x7fd4381f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd4377fe640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380b0ea0 0x7fd4381f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd4377fe640 1 -- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380cd970 msgr2=0x7fd4380acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd4377fe640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380cd970 0x7fd4380acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd4377fe640 1 -- 172.21.15.29:0/3402544954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c0094d0 con 0x7fd4380b3ec0 2023-10-16T14:17:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.917+0000 7fd4377fe640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 0x7fd4380ac890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd42c000c00 tx=0x7fd42c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.918+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c03a930 con 0x7fd4380b3ec0 2023-10-16T14:17:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.918+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c03aa90 con 0x7fd4380b3ec0 2023-10-16T14:17:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.918+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c03ae40 con 0x7fd4380b3ec0 2023-10-16T14:17:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.918+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4381f6560 con 0x7fd4380b3ec0 2023-10-16T14:17:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.918+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd438154980 con 0x7fd4380b3ec0 2023-10-16T14:17:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.919+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4381547e0 con 0x7fd4380b3ec0 2023-10-16T14:17:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.920+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd42c02f050 con 0x7fd4380b3ec0 2023-10-16T14:17:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.921+0000 7fd434ff9640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd42406ae70 0x7fd42406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.921+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd42c0ab3a0 con 0x7fd4380b3ec0 2023-10-16T14:17:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.922+0000 7fd437fff640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd42406ae70 0x7fd42406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.923+0000 7fd437fff640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd42406ae70 0x7fd42406d330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd428000f60 tx=0x7fd428009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:07.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:07.926+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd4381547e0 con 0x7fd4380b3ec0 2023-10-16T14:17:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.123+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4381547e0 con 0x7fd4380b3ec0 2023-10-16T14:17:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.124+0000 7fd434ff9640 1 -- 172.21.15.29:0/3402544954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd4381547e0 con 0x7fd4380b3ec0 2023-10-16T14:17:08.127 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.126+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd42406ae70 msgr2=0x7fd42406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.126+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd42406ae70 0x7fd42406d330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd428000f60 tx=0x7fd428009040 comp rx=0 tx=0).stop 2023-10-16T14:17:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 msgr2=0x7fd4380ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 0x7fd4380ac890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd42c000c00 tx=0x7fd42c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 shutdown_connections 2023-10-16T14:17:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd42406ae70 0x7fd42406d330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4380b3ec0 0x7fd4380ac890 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4380cd970 0x7fd4380acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 --2- 172.21.15.29:0/3402544954 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4380b0ea0 0x7fd4381f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 >> 172.21.15.29:0/3402544954 conn(0x7fd4380b0170 msgr2=0x7fd4380b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 shutdown_connections 2023-10-16T14:17:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.127+0000 7fd43d5ae640 1 -- 172.21.15.29:0/3402544954 wait complete. 2023-10-16T14:17:08.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.381+0000 7fb37f59e640 1 Processor -- start 2023-10-16T14:17:08.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.381+0000 7fb37f59e640 1 -- start start 2023-10-16T14:17:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38006c960 con 0x7fb38015ebc0 2023-10-16T14:17:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38015b9d0 con 0x7fb3800ad970 2023-10-16T14:17:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38015bcb0 con 0x7fb38006c040 2023-10-16T14:17:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37dd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37dd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43540/0 (socket says 172.21.15.29:43540) 2023-10-16T14:17:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.382+0000 7fb37dd9b640 1 -- 172.21.15.29:0/130377386 learned_addr learned my addr 172.21.15.29:0/130377386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.383+0000 7fb37e59c640 1 --2- 172.21.15.29:0/130377386 >> v2:172.21.15.170:3300/0 conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.383+0000 7fb37dd9b640 1 -- 172.21.15.29:0/130377386 >> v2:172.21.15.170:3300/0 conn(0x7fb38006c040 msgr2=0x7fb38006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.383+0000 7fb37dd9b640 1 --2- 172.21.15.29:0/130377386 >> v2:172.21.15.170:3300/0 conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.383+0000 7fb37dd9b640 1 -- 172.21.15.29:0/130377386 >> v1:172.21.15.170:6789/0 conn(0x7fb38015ebc0 legacy=0x7fb3800cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.383+0000 7fb37dd9b640 1 -- 172.21.15.29:0/130377386 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38015bf90 con 0x7fb3800ad970 2023-10-16T14:17:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.384+0000 7fb37dd9b640 1 --2- 172.21.15.29:0/130377386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 0x7fb38015e680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb3800cf350 tx=0x7fb36802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbc30181102ef82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37d59a640 1 -- 172.21.15.29:0/130377386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36802ca80 con 0x7fb3800ad970 2023-10-16T14:17:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37d59a640 1 -- 172.21.15.29:0/130377386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36802cbe0 con 0x7fb3800ad970 2023-10-16T14:17:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37d59a640 1 -- 172.21.15.29:0/130377386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680343f0 con 0x7fb3800ad970 2023-10-16T14:17:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 -- 172.21.15.29:0/130377386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 msgr2=0x7fb38015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 --2- 172.21.15.29:0/130377386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 0x7fb38015e680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb3800cf350 tx=0x7fb36802bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 -- 172.21.15.29:0/130377386 shutdown_connections 2023-10-16T14:17:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 --2- 172.21.15.29:0/130377386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 --2- 172.21.15.29:0/130377386 >> v2:172.21.15.170:3300/0 conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 -- 172.21.15.29:0/130377386 >> 172.21.15.29:0/130377386 conn(0x7fb3800ce6a0 msgr2=0x7fb3800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 -- 172.21.15.29:0/130377386 shutdown_connections 2023-10-16T14:17:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.385+0000 7fb37f59e640 1 -- 172.21.15.29:0/130377386 wait complete. 2023-10-16T14:17:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.387+0000 7fb37f59e640 1 Processor -- start 2023-10-16T14:17:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.387+0000 7fb37f59e640 1 -- start start 2023-10-16T14:17:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.388+0000 7fb37f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.388+0000 7fb37f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.388+0000 7fb37f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb380158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.388+0000 7fb37f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3800c6810 con 0x7fb38015ebc0 2023-10-16T14:17:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.388+0000 7fb37f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015c570 con 0x7fb3800ad970 2023-10-16T14:17:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.388+0000 7fb37f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015c850 con 0x7fb38006c040 2023-10-16T14:17:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb380158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb380158780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43546/0 (socket says 172.21.15.29:43546) 2023-10-16T14:17:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 -- 172.21.15.29:0/2099057049 learned_addr learned my addr 172.21.15.29:0/2099057049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37dd9b640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37e59c640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 -- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 msgr2=0x7fb380155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 -- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 msgr2=0x7fb3800c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.389+0000 7fb37ed9d640 1 -- 172.21.15.29:0/2099057049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368009b30 con 0x7fb38015ebc0 2023-10-16T14:17:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.390+0000 7fb37ed9d640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb380158780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb37800c410 tx=0x7fb37800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.390+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb378011790 con 0x7fb38015ebc0 2023-10-16T14:17:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.390+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3780118f0 con 0x7fb38015ebc0 2023-10-16T14:17:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.390+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb378011ce0 con 0x7fb38015ebc0 2023-10-16T14:17:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.390+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38015cb30 con 0x7fb38015ebc0 2023-10-16T14:17:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.390+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38015d840 con 0x7fb38015ebc0 2023-10-16T14:17:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.391+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38006b330 con 0x7fb38015ebc0 2023-10-16T14:17:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.392+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb37801c460 con 0x7fb38015ebc0 2023-10-16T14:17:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.393+0000 7fb376ffd640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36006aea0 0x7fb36006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.393+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3780892a0 con 0x7fb38015ebc0 2023-10-16T14:17:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.394+0000 7fb37e59c640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36006aea0 0x7fb36006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.395+0000 7fb37e59c640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36006aea0 0x7fb36006d360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb370006fd0 tx=0x7fb370008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:08.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.398+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb378054340 con 0x7fb38015ebc0 2023-10-16T14:17:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.591+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3800cc550 con 0x7fb38015ebc0 2023-10-16T14:17:08.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.592+0000 7fb376ffd640 1 -- 172.21.15.29:0/2099057049 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb378057960 con 0x7fb38015ebc0 2023-10-16T14:17:08.595 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36006aea0 msgr2=0x7fb36006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36006aea0 0x7fb36006d360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb370006fd0 tx=0x7fb370008040 comp rx=0 tx=0).stop 2023-10-16T14:17:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 msgr2=0x7fb380158780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb380158780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb37800c410 tx=0x7fb37800c8d0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 shutdown_connections 2023-10-16T14:17:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36006aea0 0x7fb36006d360 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb380158780 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 --2- 172.21.15.29:0/2099057049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 >> 172.21.15.29:0/2099057049 conn(0x7fb3800ce6a0 msgr2=0x7fb3800ad220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:08.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 shutdown_connections 2023-10-16T14:17:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.594+0000 7fb37f59e640 1 -- 172.21.15.29:0/2099057049 wait complete. 2023-10-16T14:17:08.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:08.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.841+0000 7fdd5f656640 1 Processor -- start 2023-10-16T14:17:08.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.841+0000 7fdd5f656640 1 -- start start 2023-10-16T14:17:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5f656640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5815d150 0x7fdd580ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5f656640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd5806a4b0 0x7fdd58160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5f656640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd580ac890 con 0x7fdd58160b90 2023-10-16T14:17:08.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5f656640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5815d530 con 0x7fdd5806a4b0 2023-10-16T14:17:08.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5f656640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd58161650 con 0x7fdd5815d150 2023-10-16T14:17:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5dbf3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd5806a4b0 0x7fdd58160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5dbf3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd5806a4b0 0x7fdd58160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43556/0 (socket says 172.21.15.29:43556) 2023-10-16T14:17:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.843+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/837665181 learned_addr learned my addr 172.21.15.29:0/837665181 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.844+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/837665181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5815d150 msgr2=0x7fdd580ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.844+0000 7fdd5d3f2640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5815d150 0x7fdd580ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.844+0000 7fdd5dbf3640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5815d150 0x7fdd580ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.844+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/837665181 >> v1:172.21.15.170:6789/0 conn(0x7fdd58160b90 legacy=0x7fdd58069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.844+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/837665181 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd58161930 con 0x7fdd5806a4b0 2023-10-16T14:17:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.844+0000 7fdd5d3f2640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5815d150 0x7fdd580ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.845+0000 7fdd5dbf3640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5806a4b0 0x7fdd58160fb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009e50 tx=0x7fdd5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f068e059a9226f1b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.845+0000 7fdd4ffff640 1 -- 172.21.15.29:0/837665181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5002ca80 con 0x7fdd5806a4b0 2023-10-16T14:17:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.845+0000 7fdd4ffff640 1 -- 172.21.15.29:0/837665181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5002cbe0 con 0x7fdd5806a4b0 2023-10-16T14:17:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.845+0000 7fdd4ffff640 1 -- 172.21.15.29:0/837665181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500343f0 con 0x7fdd5806a4b0 2023-10-16T14:17:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 -- 172.21.15.29:0/837665181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5806a4b0 msgr2=0x7fdd58160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5806a4b0 0x7fdd58160fb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009e50 tx=0x7fdd5002bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 -- 172.21.15.29:0/837665181 shutdown_connections 2023-10-16T14:17:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5806a4b0 0x7fdd58160fb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 --2- 172.21.15.29:0/837665181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5815d150 0x7fdd580ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 -- 172.21.15.29:0/837665181 >> 172.21.15.29:0/837665181 conn(0x7fdd580ce9b0 msgr2=0x7fdd580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 -- 172.21.15.29:0/837665181 shutdown_connections 2023-10-16T14:17:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.846+0000 7fdd5f656640 1 -- 172.21.15.29:0/837665181 wait complete. 2023-10-16T14:17:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.848+0000 7fdd5f656640 1 Processor -- start 2023-10-16T14:17:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.848+0000 7fdd5f656640 1 -- start start 2023-10-16T14:17:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5f656640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5806a4b0 0x7fdd580c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5f656640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd5815d150 0x7fdd581452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5f656640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 0x7fdd5806fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5f656640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c2bd0 con 0x7fdd58160b90 2023-10-16T14:17:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5f656640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd58145b50 con 0x7fdd5815d150 2023-10-16T14:17:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5f656640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd58145e30 con 0x7fdd5806a4b0 2023-10-16T14:17:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5dbf3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 0x7fdd5806fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.849+0000 7fdd5dbf3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 0x7fdd5806fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43566/0 (socket says 172.21.15.29:43566) 2023-10-16T14:17:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/1771604149 learned_addr learned my addr 172.21.15.29:0/1771604149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5cbf1640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd5815d150 0x7fdd581452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5d3f2640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5806a4b0 0x7fdd580c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5806a4b0 msgr2=0x7fdd580c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5806a4b0 0x7fdd580c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd5815d150 msgr2=0x7fdd581452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd5815d150 0x7fdd581452c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 -- 172.21.15.29:0/1771604149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd50009b30 con 0x7fdd58160b90 2023-10-16T14:17:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.850+0000 7fdd5dbf3640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 0x7fdd5806fe50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd50036040 tx=0x7fdd50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.851+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5003e450 con 0x7fdd58160b90 2023-10-16T14:17:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.851+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5003e5d0 con 0x7fdd58160b90 2023-10-16T14:17:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.851+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5003e980 con 0x7fdd58160b90 2023-10-16T14:17:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.851+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd58146110 con 0x7fdd58160b90 2023-10-16T14:17:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.851+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd581466d0 con 0x7fdd58160b90 2023-10-16T14:17:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.852+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5804e160 con 0x7fdd58160b90 2023-10-16T14:17:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.853+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdd5002cd50 con 0x7fdd58160b90 2023-10-16T14:17:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.854+0000 7fdd4dffb640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdd3c06ae70 0x7fdd3c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.854+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd500ac2b0 con 0x7fdd58160b90 2023-10-16T14:17:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.855+0000 7fdd5d3f2640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdd3c06ae70 0x7fdd3c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.856+0000 7fdd5d3f2640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdd3c06ae70 0x7fdd3c06d330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400a990 tx=0x7fdd54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:08.858+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd50077490 con 0x7fdd58160b90 2023-10-16T14:17:09.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.052+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd580ade10 con 0x7fdd58160b90 2023-10-16T14:17:09.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.054+0000 7fdd4dffb640 1 -- 172.21.15.29:0/1771604149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd5007aab0 con 0x7fdd58160b90 2023-10-16T14:17:09.057 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:09.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.056+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdd3c06ae70 msgr2=0x7fdd3c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.056+0000 7fdd5f656640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdd3c06ae70 0x7fdd3c06d330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400a990 tx=0x7fdd54008040 comp rx=0 tx=0).stop 2023-10-16T14:17:09.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.056+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 msgr2=0x7fdd5806fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.056+0000 7fdd5f656640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 0x7fdd5806fe50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd50036040 tx=0x7fdd50004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:09.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 shutdown_connections 2023-10-16T14:17:09.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdd3c06ae70 0x7fdd3c06d330 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd58160b90 0x7fdd5806fe50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd5815d150 0x7fdd581452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 --2- 172.21.15.29:0/1771604149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd5806a4b0 0x7fdd580c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 >> 172.21.15.29:0/1771604149 conn(0x7fdd580ce9b0 msgr2=0x7fdd580c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:09.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 shutdown_connections 2023-10-16T14:17:09.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.057+0000 7fdd5f656640 1 -- 172.21.15.29:0/1771604149 wait complete. 2023-10-16T14:17:09.090 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-16T14:17:09.091 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:17:09.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:09.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.316+0000 7fd356bed640 1 Processor -- start 2023-10-16T14:17:09.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.316+0000 7fd356bed640 1 -- start start 2023-10-16T14:17:09.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd356bed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3500ad970 0x7fd3500cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd356bed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35015b9d0 con 0x7fd3500ccfb0 2023-10-16T14:17:09.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd356bed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35015bcb0 con 0x7fd35006c040 2023-10-16T14:17:09.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd356bed640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35015bf90 con 0x7fd3500ad970 2023-10-16T14:17:09.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd3553ea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3500ad970 0x7fd3500cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd3563ec640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd3500ccfb0 0x7fd3500cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52000/0 (socket says 172.21.15.29:52000) 2023-10-16T14:17:09.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.318+0000 7fd3563ec640 1 -- 172.21.15.29:0/3062411619 learned_addr learned my addr 172.21.15.29:0/3062411619 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:09.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.319+0000 7fd3553ea640 1 -- 172.21.15.29:0/3062411619 >> v1:172.21.15.170:6789/0 conn(0x7fd35006c040 legacy=0x7fd35006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:09.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.319+0000 7fd3553ea640 1 -- 172.21.15.29:0/3062411619 >> v1:172.21.15.29:6789/0 conn(0x7fd3500ccfb0 legacy=0x7fd3500cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.319+0000 7fd3553ea640 1 -- 172.21.15.29:0/3062411619 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500cc270 con 0x7fd3500ad970 2023-10-16T14:17:09.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.319+0000 7fd354be9640 1 -- 172.21.15.29:0/3062411619 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137233765 0 0) 0x7fd35015b9d0 con 0x7fd3500ccfb0 2023-10-16T14:17:09.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.320+0000 7fd3553ea640 1 --2- 172.21.15.29:0/3062411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3500ad970 0x7fd3500cca70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd3400097d0 tx=0x7fd34002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4fa6c5d15fe2b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:09.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.320+0000 7fd354be9640 1 -- 172.21.15.29:0/3062411619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340031830 con 0x7fd3500ad970 2023-10-16T14:17:09.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.320+0000 7fd354be9640 1 -- 172.21.15.29:0/3062411619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd340031990 con 0x7fd3500ad970 2023-10-16T14:17:09.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.320+0000 7fd354be9640 1 -- 172.21.15.29:0/3062411619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340031d20 con 0x7fd3500ad970 2023-10-16T14:17:09.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 -- 172.21.15.29:0/3062411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3500ad970 msgr2=0x7fd3500cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 --2- 172.21.15.29:0/3062411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3500ad970 0x7fd3500cca70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd3400097d0 tx=0x7fd34002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 -- 172.21.15.29:0/3062411619 shutdown_connections 2023-10-16T14:17:09.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 --2- 172.21.15.29:0/3062411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3500ad970 0x7fd3500cca70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 -- 172.21.15.29:0/3062411619 >> 172.21.15.29:0/3062411619 conn(0x7fd3500ce6a0 msgr2=0x7fd3500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 -- 172.21.15.29:0/3062411619 shutdown_connections 2023-10-16T14:17:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.321+0000 7fd356bed640 1 -- 172.21.15.29:0/3062411619 wait complete. 2023-10-16T14:17:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.323+0000 7fd356bed640 1 Processor -- start 2023-10-16T14:17:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.323+0000 7fd356bed640 1 -- start start 2023-10-16T14:17:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd356bed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 0x7fd350149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd356bed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3500ad970 0x7fd350159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd356bed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3500ccfb0 0x7fd350159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd356bed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35015a5d0 con 0x7fd35006c040 2023-10-16T14:17:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd356bed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350147690 con 0x7fd3500ad970 2023-10-16T14:17:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd356bed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd350147970 con 0x7fd3500ccfb0 2023-10-16T14:17:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.324+0000 7fd355beb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 0x7fd350149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 0x7fd350149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43588/0 (socket says 172.21.15.29:43588) 2023-10-16T14:17:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 -- 172.21.15.29:0/3247543138 learned_addr learned my addr 172.21.15.29:0/3247543138 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd3553ea640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3500ad970 0x7fd350159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd3563ec640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3500ccfb0 0x7fd350159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 -- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3500ccfb0 msgr2=0x7fd350159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3500ccfb0 0x7fd350159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 -- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3500ad970 msgr2=0x7fd350159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3500ad970 0x7fd350159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 -- 172.21.15.29:0/3247543138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340009400 con 0x7fd35006c040 2023-10-16T14:17:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.325+0000 7fd355beb640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 0x7fd350149c00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00b3f0 tx=0x7fd34c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.326+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c011830 con 0x7fd35006c040 2023-10-16T14:17:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.326+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c011990 con 0x7fd35006c040 2023-10-16T14:17:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.326+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c011d40 con 0x7fd35006c040 2023-10-16T14:17:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.326+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd350147c50 con 0x7fd35006c040 2023-10-16T14:17:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.326+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35014a140 con 0x7fd35006c040 2023-10-16T14:17:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.327+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35014a390 con 0x7fd35006c040 2023-10-16T14:17:09.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.328+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd34c01a400 con 0x7fd35006c040 2023-10-16T14:17:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.329+0000 7fd3467fc640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd33406ae70 0x7fd33406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.329+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd34c0894d0 con 0x7fd35006c040 2023-10-16T14:17:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.330+0000 7fd3553ea640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd33406ae70 0x7fd33406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.331+0000 7fd3553ea640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd33406ae70 0x7fd33406d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd340036000 tx=0x7fd340008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.334+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd34c0544e0 con 0x7fd35006c040 2023-10-16T14:17:09.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.532+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd35004e9b0 con 0x7fd35006c040 2023-10-16T14:17:09.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.534+0000 7fd3467fc640 1 -- 172.21.15.29:0/3247543138 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd34c057b00 con 0x7fd35006c040 2023-10-16T14:17:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:09.538 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:16:30.780379+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:09.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd33406ae70 msgr2=0x7fd33406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd33406ae70 0x7fd33406d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd340036000 tx=0x7fd340008030 comp rx=0 tx=0).stop 2023-10-16T14:17:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 msgr2=0x7fd350149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 0x7fd350149c00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00b3f0 tx=0x7fd34c00b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 shutdown_connections 2023-10-16T14:17:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd33406ae70 0x7fd33406d330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3500ccfb0 0x7fd350159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3500ad970 0x7fd350159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 --2- 172.21.15.29:0/3247543138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35006c040 0x7fd350149c00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 >> 172.21.15.29:0/3247543138 conn(0x7fd3500ce6a0 msgr2=0x7fd3500cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 shutdown_connections 2023-10-16T14:17:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.537+0000 7fd356bed640 1 -- 172.21.15.29:0/3247543138 wait complete. 2023-10-16T14:17:09.560 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:09.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:09.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.790+0000 7fa6261e2640 1 Processor -- start 2023-10-16T14:17:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.790+0000 7fa6261e2640 1 -- start start 2023-10-16T14:17:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa6261e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62006a4b0 0x7fa6200ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa6261e2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6200c9ca0 0x7fa6200ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa6261e2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200cb030 con 0x7fa62015ad70 2023-10-16T14:17:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa6261e2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6200cae30 con 0x7fa6200c9ca0 2023-10-16T14:17:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa6261e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200c91c0 con 0x7fa62006a4b0 2023-10-16T14:17:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa61f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa62015ad70 0x7fa620069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52012/0 (socket says 172.21.15.29:52012) 2023-10-16T14:17:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa61f7fe640 1 -- 172.21.15.29:0/4163851146 learned_addr learned my addr 172.21.15.29:0/4163851146 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa61ffff640 1 --2- 172.21.15.29:0/4163851146 >> v2:172.21.15.29:3300/0 conn(0x7fa6200c9ca0 0x7fa6200ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.792+0000 7fa617fff640 1 --2- 172.21.15.29:0/4163851146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62006a4b0 0x7fa6200ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.793+0000 7fa61ffff640 1 -- 172.21.15.29:0/4163851146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62006a4b0 msgr2=0x7fa6200ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:09.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.793+0000 7fa61ffff640 1 --2- 172.21.15.29:0/4163851146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62006a4b0 0x7fa6200ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.793+0000 7fa61ffff640 1 -- 172.21.15.29:0/4163851146 >> v1:172.21.15.29:6789/0 conn(0x7fa62015ad70 legacy=0x7fa620069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.793+0000 7fa61ffff640 1 -- 172.21.15.29:0/4163851146 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200c94a0 con 0x7fa6200c9ca0 2023-10-16T14:17:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.794+0000 7fa61ffff640 1 --2- 172.21.15.29:0/4163851146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6200c9ca0 0x7fa6200ca8f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa61802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8cd50427b855b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.794+0000 7fa61effd640 1 -- 172.21.15.29:0/4163851146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61802cc60 con 0x7fa6200c9ca0 2023-10-16T14:17:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.794+0000 7fa61effd640 1 -- 172.21.15.29:0/4163851146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61802cdc0 con 0x7fa6200c9ca0 2023-10-16T14:17:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.794+0000 7fa61effd640 1 -- 172.21.15.29:0/4163851146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618034660 con 0x7fa6200c9ca0 2023-10-16T14:17:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 -- 172.21.15.29:0/4163851146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6200c9ca0 msgr2=0x7fa6200ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 --2- 172.21.15.29:0/4163851146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6200c9ca0 0x7fa6200ca8f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa61802beb0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 -- 172.21.15.29:0/4163851146 shutdown_connections 2023-10-16T14:17:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 --2- 172.21.15.29:0/4163851146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6200c9ca0 0x7fa6200ca8f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 --2- 172.21.15.29:0/4163851146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa62006a4b0 0x7fa6200ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 -- 172.21.15.29:0/4163851146 >> 172.21.15.29:0/4163851146 conn(0x7fa6200ce9b0 msgr2=0x7fa6200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 -- 172.21.15.29:0/4163851146 shutdown_connections 2023-10-16T14:17:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.795+0000 7fa6261e2640 1 -- 172.21.15.29:0/4163851146 wait complete. 2023-10-16T14:17:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.797+0000 7fa6261e2640 1 Processor -- start 2023-10-16T14:17:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.797+0000 7fa6261e2640 1 -- start start 2023-10-16T14:17:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.798+0000 7fa6261e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62006a4b0 0x7fa620151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.798+0000 7fa6261e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6200c9ca0 0x7fa6201f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.798+0000 7fa6261e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 0x7fa6201f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.798+0000 7fa6261e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6201f3170 con 0x7fa62015ad70 2023-10-16T14:17:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.798+0000 7fa6261e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200c8420 con 0x7fa62006a4b0 2023-10-16T14:17:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.798+0000 7fa6261e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200c7b80 con 0x7fa6200c9ca0 2023-10-16T14:17:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 0x7fa6201f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 0x7fa6201f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43606/0 (socket says 172.21.15.29:43606) 2023-10-16T14:17:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 -- 172.21.15.29:0/931642396 learned_addr learned my addr 172.21.15.29:0/931642396 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:09.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61f7fe640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62006a4b0 0x7fa620151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa617fff640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6200c9ca0 0x7fa6201f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 -- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6200c9ca0 msgr2=0x7fa6201f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6200c9ca0 0x7fa6201f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 -- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62006a4b0 msgr2=0x7fa620151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62006a4b0 0x7fa620151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 -- 172.21.15.29:0/931642396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618009a70 con 0x7fa62015ad70 2023-10-16T14:17:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.799+0000 7fa61ffff640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 0x7fa6201f2c30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.800+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61803f420 con 0x7fa62015ad70 2023-10-16T14:17:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.800+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61803f580 con 0x7fa62015ad70 2023-10-16T14:17:09.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.800+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61803f8f0 con 0x7fa62015ad70 2023-10-16T14:17:09.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.800+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6200c7e60 con 0x7fa62015ad70 2023-10-16T14:17:09.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.800+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6200ac9c0 con 0x7fa62015ad70 2023-10-16T14:17:09.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.801+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62004e160 con 0x7fa62015ad70 2023-10-16T14:17:09.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.802+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa61803fa50 con 0x7fa62015ad70 2023-10-16T14:17:09.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.803+0000 7fa61cff9640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa60006ae70 0x7fa60006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:09.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.803+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6180ac1d0 con 0x7fa62015ad70 2023-10-16T14:17:09.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.804+0000 7fa61f7fe640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa60006ae70 0x7fa60006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:09.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.805+0000 7fa61f7fe640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa60006ae70 0x7fa60006d330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa6200c5330 tx=0x7fa610009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:09.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:09.808+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa618077390 con 0x7fa62015ad70 2023-10-16T14:17:10.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.007+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6200700e0 con 0x7fa62015ad70 2023-10-16T14:17:10.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.008+0000 7fa61cff9640 1 -- 172.21.15.29:0/931642396 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa61807a9b0 con 0x7fa62015ad70 2023-10-16T14:17:10.011 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:10.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa60006ae70 msgr2=0x7fa60006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa60006ae70 0x7fa60006d330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa6200c5330 tx=0x7fa610009040 comp rx=0 tx=0).stop 2023-10-16T14:17:10.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 msgr2=0x7fa6201f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 0x7fa6201f2c30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa618004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:10.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 shutdown_connections 2023-10-16T14:17:10.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa60006ae70 0x7fa60006d330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa62015ad70 0x7fa6201f2c30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6200c9ca0 0x7fa6201f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 --2- 172.21.15.29:0/931642396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa62006a4b0 0x7fa620151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 >> 172.21.15.29:0/931642396 conn(0x7fa6200ce9b0 msgr2=0x7fa620069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:10.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 shutdown_connections 2023-10-16T14:17:10.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.011+0000 7fa6261e2640 1 -- 172.21.15.29:0/931642396 wait complete. 2023-10-16T14:17:10.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.263+0000 7f0f0bfff640 1 Processor -- start 2023-10-16T14:17:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.263+0000 7f0f0bfff640 1 -- start start 2023-10-16T14:17:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f0c0b8480 0x7f0f0c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c15d150 con 0x7f0f0c15cd70 2023-10-16T14:17:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c0d1180 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0c0d1460 con 0x7f0f0c0b8480 2023-10-16T14:17:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f0c0b8480 0x7f0f0c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0f0c15cd70 0x7f0f0c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52014/0 (socket says 172.21.15.29:52014) 2023-10-16T14:17:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0affd640 1 -- 172.21.15.29:0/1390299198 learned_addr learned my addr 172.21.15.29:0/1390299198 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:10.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.265+0000 7f0f0a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f0c0b8480 0x7f0f0c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43618/0 (socket says 172.21.15.29:43618) 2023-10-16T14:17:10.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.266+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/1390299198 >> v1:172.21.15.170:6789/0 conn(0x7f0f0c0d0be0 legacy=0x7f0f0c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:10.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.266+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/1390299198 >> v1:172.21.15.29:6789/0 conn(0x7f0f0c15cd70 legacy=0x7f0f0c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.266+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/1390299198 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c0d1740 con 0x7f0f0c0b8480 2023-10-16T14:17:10.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.267+0000 7f0f0a7fc640 1 --2- 172.21.15.29:0/1390299198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c0cd7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40097a0 tx=0x7f0ef402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4a7b44de9c51ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:10.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.267+0000 7f0f09ffb640 1 -- 172.21.15.29:0/1390299198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef402e070 con 0x7f0f0c0b8480 2023-10-16T14:17:10.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.267+0000 7f0f09ffb640 1 -- 172.21.15.29:0/1390299198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef40315a0 con 0x7f0f0c0b8480 2023-10-16T14:17:10.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.267+0000 7f0f09ffb640 1 -- 172.21.15.29:0/1390299198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4031930 con 0x7f0f0c0b8480 2023-10-16T14:17:10.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 -- 172.21.15.29:0/1390299198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0b8480 msgr2=0x7f0f0c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/1390299198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c0cd7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40097a0 tx=0x7f0ef402b840 comp rx=0 tx=0).stop 2023-10-16T14:17:10.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 -- 172.21.15.29:0/1390299198 shutdown_connections 2023-10-16T14:17:10.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/1390299198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c0cd7c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 -- 172.21.15.29:0/1390299198 >> 172.21.15.29:0/1390299198 conn(0x7f0f0c0cf570 msgr2=0x7f0f0c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:10.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 -- 172.21.15.29:0/1390299198 shutdown_connections 2023-10-16T14:17:10.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.268+0000 7f0f0bfff640 1 -- 172.21.15.29:0/1390299198 wait complete. 2023-10-16T14:17:10.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.270+0000 7f0f0bfff640 1 Processor -- start 2023-10-16T14:17:10.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.270+0000 7f0f0bfff640 1 -- start start 2023-10-16T14:17:10.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 0x7f0f0c0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0c15cd70 0x7f0f0c1f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0ccc40 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1547d0 con 0x7f0f0c0b8480 2023-10-16T14:17:10.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c155070 con 0x7f0f0c15cd70 2023-10-16T14:17:10.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.271+0000 7f0f0a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 0x7f0f0c0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 0x7f0f0c0cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43620/0 (socket says 172.21.15.29:43620) 2023-10-16T14:17:10.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/2545778147 learned_addr learned my addr 172.21.15.29:0/2545778147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:10.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0affd640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0b7fe640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0c15cd70 0x7f0f0c1f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0c15cd70 msgr2=0x7f0f0c1f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0c15cd70 0x7f0f0c1f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f0c0b8480 msgr2=0x7f0f0c155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0a7fc640 1 -- 172.21.15.29:0/2545778147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef4009400 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.272+0000 7f0f0affd640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:10.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.273+0000 7f0f0a7fc640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 0x7f0f0c0cc700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4034040 tx=0x7f0ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:10.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.273+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4032900 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.273+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef4032a80 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.273+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4032e30 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.273+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c1f5390 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.273+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c154a20 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.274+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c0c9b60 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.275+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0ef403b3f0 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.276+0000 7f0f037fe640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0eec06ae70 0x7f0eec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.276+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ef40ab720 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.277+0000 7f0f0affd640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0eec06ae70 0x7f0eec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.278+0000 7f0f0affd640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0eec06ae70 0x7f0eec06d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005c00 tx=0x7f0efc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:10.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.281+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ef4076730 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.474+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f0c04e9b0 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.475+0000 7f0f037fe640 1 -- 172.21.15.29:0/2545778147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ef4079d50 con 0x7f0f0c0d0be0 2023-10-16T14:17:10.478 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:10.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0eec06ae70 msgr2=0x7f0eec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0eec06ae70 0x7f0eec06d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005c00 tx=0x7f0efc009040 comp rx=0 tx=0).stop 2023-10-16T14:17:10.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 msgr2=0x7f0f0c0cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 0x7f0f0c0cc700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4034040 tx=0x7f0ef40046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 shutdown_connections 2023-10-16T14:17:10.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0eec06ae70 0x7f0eec06d330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0c0d0be0 0x7f0f0c0cc700 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0c15cd70 0x7f0f0c1f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 --2- 172.21.15.29:0/2545778147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f0c0b8480 0x7f0f0c155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 >> 172.21.15.29:0/2545778147 conn(0x7f0f0c0cf570 msgr2=0x7f0f0c06bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:10.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 shutdown_connections 2023-10-16T14:17:10.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.478+0000 7f0f0bfff640 1 -- 172.21.15.29:0/2545778147 wait complete. 2023-10-16T14:17:10.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.727+0000 7f1dacafc640 1 Processor -- start 2023-10-16T14:17:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.727+0000 7f1dacafc640 1 -- start start 2023-10-16T14:17:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1dacafc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1da815cd70 0x7f1da80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1dacafc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1dacafc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da815d150 con 0x7f1da80d0be0 2023-10-16T14:17:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1dacafc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da80d1180 con 0x7f1da815cd70 2023-10-16T14:17:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1dacafc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80d1460 con 0x7f1da80b8480 2023-10-16T14:17:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1da77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1da815cd70 0x7f1da80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1da77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1da815cd70 0x7f1da80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43626/0 (socket says 172.21.15.29:43626) 2023-10-16T14:17:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.729+0000 7f1da77fe640 1 -- 172.21.15.29:0/4237530395 learned_addr learned my addr 172.21.15.29:0/4237530395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.730+0000 7f1da77fe640 1 -- 172.21.15.29:0/4237530395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 msgr2=0x7f1da80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.730+0000 7f1da77fe640 1 --2- 172.21.15.29:0/4237530395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.730+0000 7f1da77fe640 1 -- 172.21.15.29:0/4237530395 >> v1:172.21.15.29:6789/0 conn(0x7f1da80d0be0 legacy=0x7f1da80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.730+0000 7f1da77fe640 1 -- 172.21.15.29:0/4237530395 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da80d1740 con 0x7f1da815cd70 2023-10-16T14:17:10.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.731+0000 7f1da77fe640 1 --2- 172.21.15.29:0/4237530395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da80d2220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009870 tx=0x7f1d9802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73d5e60df0cbac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:10.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.731+0000 7f1da67fc640 1 -- 172.21.15.29:0/4237530395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98030840 con 0x7f1da815cd70 2023-10-16T14:17:10.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.731+0000 7f1da67fc640 1 -- 172.21.15.29:0/4237530395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d980309a0 con 0x7f1da815cd70 2023-10-16T14:17:10.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.731+0000 7f1da67fc640 1 -- 172.21.15.29:0/4237530395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98030d30 con 0x7f1da815cd70 2023-10-16T14:17:10.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 -- 172.21.15.29:0/4237530395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 msgr2=0x7f1da80d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 --2- 172.21.15.29:0/4237530395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da80d2220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009870 tx=0x7f1d9802b980 comp rx=0 tx=0).stop 2023-10-16T14:17:10.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 -- 172.21.15.29:0/4237530395 shutdown_connections 2023-10-16T14:17:10.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 --2- 172.21.15.29:0/4237530395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 --2- 172.21.15.29:0/4237530395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da80d2220 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 -- 172.21.15.29:0/4237530395 >> 172.21.15.29:0/4237530395 conn(0x7f1da80cf570 msgr2=0x7f1da80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:10.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 -- 172.21.15.29:0/4237530395 shutdown_connections 2023-10-16T14:17:10.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.732+0000 7f1dacafc640 1 -- 172.21.15.29:0/4237530395 wait complete. 2023-10-16T14:17:10.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.734+0000 7f1dacafc640 1 Processor -- start 2023-10-16T14:17:10.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.734+0000 7f1dacafc640 1 -- start start 2023-10-16T14:17:10.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1dacafc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1dacafc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1da80d0be0 0x7f1da80c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1dacafc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da81f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1dacafc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80c7040 con 0x7f1da815cd70 2023-10-16T14:17:10.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1dacafc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da8156290 con 0x7f1da80d0be0 2023-10-16T14:17:10.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1dacafc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81f6780 con 0x7f1da80b8480 2023-10-16T14:17:10.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.735+0000 7f1da7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da81f58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da81f58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43634/0 (socket says 172.21.15.29:43634) 2023-10-16T14:17:10.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 -- 172.21.15.29:0/243874424 learned_addr learned my addr 172.21.15.29:0/243874424 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:10.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da77fe640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da6ffd640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1da80d0be0 0x7f1da80c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 -- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 msgr2=0x7f1da815d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da815d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 -- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1da80d0be0 msgr2=0x7f1da80c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1da80d0be0 0x7f1da80c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 -- 172.21.15.29:0/243874424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980094d0 con 0x7f1da815cd70 2023-10-16T14:17:10.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da77fe640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:10.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da6ffd640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1da80d0be0 0x7f1da80c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:10.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.736+0000 7f1da7fff640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da81f58c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c730 tx=0x7f1da000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:10.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.737+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0003b70 con 0x7f1da815cd70 2023-10-16T14:17:10.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.737+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0003cf0 con 0x7f1da815cd70 2023-10-16T14:17:10.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.737+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0003ea0 con 0x7f1da815cd70 2023-10-16T14:17:10.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.737+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da81f64a0 con 0x7f1da815cd70 2023-10-16T14:17:10.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.737+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da80c6c60 con 0x7f1da815cd70 2023-10-16T14:17:10.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.738+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da804e9b0 con 0x7f1da815cd70 2023-10-16T14:17:10.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.739+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1da0005310 con 0x7f1da815cd70 2023-10-16T14:17:10.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.740+0000 7f1d8ffff640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1d9406ae70 0x7f1d9406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:10.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.740+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1da0089360 con 0x7f1da815cd70 2023-10-16T14:17:10.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.741+0000 7f1da77fe640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1d9406ae70 0x7f1d9406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:10.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.742+0000 7f1da77fe640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1d9406ae70 0x7f1d9406d330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1d98036000 tx=0x7f1d9803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:10.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.744+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1da0054400 con 0x7f1da815cd70 2023-10-16T14:17:10.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.948+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1da815c000 con 0x7f1da815cd70 2023-10-16T14:17:10.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.949+0000 7f1d8ffff640 1 -- 172.21.15.29:0/243874424 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1da0057a20 con 0x7f1da815cd70 2023-10-16T14:17:10.952 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.951+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1d9406ae70 msgr2=0x7f1d9406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.951+0000 7f1dacafc640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1d9406ae70 0x7f1d9406d330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1d98036000 tx=0x7f1d9803b040 comp rx=0 tx=0).stop 2023-10-16T14:17:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 msgr2=0x7f1da81f58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da81f58c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c730 tx=0x7f1da000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 shutdown_connections 2023-10-16T14:17:10.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1d9406ae70 0x7f1d9406d330 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1da815cd70 0x7f1da81f58c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1da80d0be0 0x7f1da80c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 --2- 172.21.15.29:0/243874424 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1da80b8480 0x7f1da815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:10.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 >> 172.21.15.29:0/243874424 conn(0x7f1da80cf570 msgr2=0x7f1da81594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:10.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 shutdown_connections 2023-10-16T14:17:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:10.952+0000 7f1dacafc640 1 -- 172.21.15.29:0/243874424 wait complete. 2023-10-16T14:17:10.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T14:17:11.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.192+0000 7f287dec3640 1 Processor -- start 2023-10-16T14:17:11.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.192+0000 7f287dec3640 1 -- start start 2023-10-16T14:17:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.193+0000 7f287dec3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815cd20 0x7f28780cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.193+0000 7f287dec3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f287815d100 0x7f28780b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f287dec3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780b8430 con 0x7f28780b1810 2023-10-16T14:17:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f287dec3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780d2160 con 0x7f287815d100 2023-10-16T14:17:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f287dec3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28780ce0b0 con 0x7f287815cd20 2023-10-16T14:17:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f2877fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f287815d100 0x7f28780b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f287d6c2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f28780b1810 0x7f287806bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52018/0 (socket says 172.21.15.29:52018) 2023-10-16T14:17:11.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f287d6c2640 1 -- 172.21.15.29:0/1204401861 learned_addr learned my addr 172.21.15.29:0/1204401861 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:11.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.194+0000 7f287cec1640 1 --2- 172.21.15.29:0/1204401861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815cd20 0x7f28780cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.195+0000 7f2877fff640 1 -- 172.21.15.29:0/1204401861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815cd20 msgr2=0x7f28780cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.195+0000 7f2877fff640 1 --2- 172.21.15.29:0/1204401861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815cd20 0x7f28780cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.195+0000 7f2877fff640 1 -- 172.21.15.29:0/1204401861 >> v1:172.21.15.29:6789/0 conn(0x7f28780b1810 legacy=0x7f287806bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.195+0000 7f2877fff640 1 -- 172.21.15.29:0/1204401861 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780b8680 con 0x7f287815d100 2023-10-16T14:17:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.196+0000 7f2877fff640 1 --2- 172.21.15.29:0/1204401861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815d100 0x7f28780b4590 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f286c0097a0 tx=0x7f286c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c847dc32ae1874 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.196+0000 7f28777fe640 1 -- 172.21.15.29:0/1204401861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c030800 con 0x7f287815d100 2023-10-16T14:17:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.196+0000 7f28777fe640 1 -- 172.21.15.29:0/1204401861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c030960 con 0x7f287815d100 2023-10-16T14:17:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.196+0000 7f28777fe640 1 -- 172.21.15.29:0/1204401861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c030cf0 con 0x7f287815d100 2023-10-16T14:17:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 -- 172.21.15.29:0/1204401861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815d100 msgr2=0x7f28780b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 --2- 172.21.15.29:0/1204401861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815d100 0x7f28780b4590 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f286c0097a0 tx=0x7f286c02b940 comp rx=0 tx=0).stop 2023-10-16T14:17:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 -- 172.21.15.29:0/1204401861 shutdown_connections 2023-10-16T14:17:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 --2- 172.21.15.29:0/1204401861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815d100 0x7f28780b4590 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 --2- 172.21.15.29:0/1204401861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815cd20 0x7f28780cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 -- 172.21.15.29:0/1204401861 >> 172.21.15.29:0/1204401861 conn(0x7f28780cf590 msgr2=0x7f28780cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 -- 172.21.15.29:0/1204401861 shutdown_connections 2023-10-16T14:17:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.197+0000 7f287dec3640 1 -- 172.21.15.29:0/1204401861 wait complete. 2023-10-16T14:17:11.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.199+0000 7f287dec3640 1 Processor -- start 2023-10-16T14:17:11.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.199+0000 7f287dec3640 1 -- start start 2023-10-16T14:17:11.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.200+0000 7f287dec3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28780b1810 0x7f287814e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:11.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.200+0000 7f287dec3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 0x7f2878151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:11.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.200+0000 7f287dec3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815d100 0x7f2878152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:11.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.200+0000 7f287dec3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780c9d10 con 0x7f287815cd20 2023-10-16T14:17:11.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.200+0000 7f287dec3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287814e840 con 0x7f28780b1810 2023-10-16T14:17:11.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.200+0000 7f287dec3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287815e310 con 0x7f287815d100 2023-10-16T14:17:11.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 0x7f2878151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:11.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 0x7f2878151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43640/0 (socket says 172.21.15.29:43640) 2023-10-16T14:17:11.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 -- 172.21.15.29:0/4126484491 learned_addr learned my addr 172.21.15.29:0/4126484491 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:11.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f287cec1640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28780b1810 0x7f287814e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:11.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f287d6c2640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815d100 0x7f2878152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:11.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 -- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815d100 msgr2=0x7f2878152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:11.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815d100 0x7f2878152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 -- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28780b1810 msgr2=0x7f287814e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:11.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28780b1810 0x7f287814e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 -- 172.21.15.29:0/4126484491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c009400 con 0x7f287815cd20 2023-10-16T14:17:11.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f287cec1640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28780b1810 0x7f287814e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:11.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.201+0000 7f2877fff640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 0x7f2878151e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f286c02bf50 tx=0x7f286c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:11.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.202+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c03e070 con 0x7f287815cd20 2023-10-16T14:17:11.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.202+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c039ad0 con 0x7f287815cd20 2023-10-16T14:17:11.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.202+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c039e60 con 0x7f287815cd20 2023-10-16T14:17:11.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.202+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287815e5f0 con 0x7f287815cd20 2023-10-16T14:17:11.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.202+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287815ebb0 con 0x7f287815cd20 2023-10-16T14:17:11.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.203+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287804e9b0 con 0x7f287815cd20 2023-10-16T14:17:11.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.204+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f286c0423f0 con 0x7f287815cd20 2023-10-16T14:17:11.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.205+0000 7f28757fa640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f286406ae70 0x7f286406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:11.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.206+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f286c0ab2f0 con 0x7f287815cd20 2023-10-16T14:17:11.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.206+0000 7f287cec1640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f286406ae70 0x7f286406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:11.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.207+0000 7f287cec1640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f286406ae70 0x7f286406d330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28781505a0 tx=0x7f286800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:11.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.210+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f286c076250 con 0x7f287815cd20 2023-10-16T14:17:11.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.474+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f28780b15d0 con 0x7f287815cd20 2023-10-16T14:17:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.977+0000 7f28757fa640 1 -- 172.21.15.29:0/4126484491 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v61) v1 ==== 161+0+0 (secure 0 0 0) 0x7f286c079870 con 0x7f287815cd20 2023-10-16T14:17:11.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f286406ae70 msgr2=0x7f286406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:11.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f286406ae70 0x7f286406d330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28781505a0 tx=0x7f286800b040 comp rx=0 tx=0).stop 2023-10-16T14:17:11.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 msgr2=0x7f2878151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:11.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 0x7f2878151e20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f286c02bf50 tx=0x7f286c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 shutdown_connections 2023-10-16T14:17:11.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f286406ae70 0x7f286406d330 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f287815cd20 0x7f2878151e20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f287815d100 0x7f2878152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 --2- 172.21.15.29:0/4126484491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28780b1810 0x7f287814e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:11.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.980+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 >> 172.21.15.29:0/4126484491 conn(0x7f28780cf590 msgr2=0x7f28780b7640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:11.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.981+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 shutdown_connections 2023-10-16T14:17:11.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:11.981+0000 7f287dec3640 1 -- 172.21.15.29:0/4126484491 wait complete. 2023-10-16T14:17:12.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:17:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.225+0000 7f97dffff640 1 Processor -- start 2023-10-16T14:17:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.226+0000 7f97dffff640 1 -- start start 2023-10-16T14:17:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.227+0000 7f97dffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97e006c040 0x7f97e006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.227+0000 7f97dffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f97e00ad970 0x7f97e015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.227+0000 7f97dffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e015ebc0 0x7f97e015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.227+0000 7f97dffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e006c960 con 0x7f97e006c040 2023-10-16T14:17:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97dffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e015b9d0 con 0x7f97e00ad970 2023-10-16T14:17:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97dffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e015bcb0 con 0x7f97e015ebc0 2023-10-16T14:17:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97e006c040 0x7f97e006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97e006c040 0x7f97e006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43642/0 (socket says 172.21.15.29:43642) 2023-10-16T14:17:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 -- 172.21.15.29:0/2934531362 learned_addr learned my addr 172.21.15.29:0/2934531362 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97df7fe640 1 --2- 172.21.15.29:0/2934531362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e015ebc0 0x7f97e015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97d67fc640 1 --2- 172.21.15.29:0/2934531362 >> v2:172.21.15.170:3300/0 conn(0x7f97e00ad970 0x7f97e015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 -- 172.21.15.29:0/2934531362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e015ebc0 msgr2=0x7f97e015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:12.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 --2- 172.21.15.29:0/2934531362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e015ebc0 0x7f97e015f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 -- 172.21.15.29:0/2934531362 >> v2:172.21.15.170:3300/0 conn(0x7f97e00ad970 msgr2=0x7f97e015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:12.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 --2- 172.21.15.29:0/2934531362 >> v2:172.21.15.170:3300/0 conn(0x7f97e00ad970 0x7f97e015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.228+0000 7f97deffd640 1 -- 172.21.15.29:0/2934531362 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e015bf90 con 0x7f97e006c040 2023-10-16T14:17:12.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.229+0000 7f97deffd640 1 --2- 172.21.15.29:0/2934531362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e006c040 0x7f97e006c420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97e00cb060 tx=0x7f97d002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb91496d90e44e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:12.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.230+0000 7f97de7fc640 1 -- 172.21.15.29:0/2934531362 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d002ca80 con 0x7f97e006c040 2023-10-16T14:17:12.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.230+0000 7f97de7fc640 1 -- 172.21.15.29:0/2934531362 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d002cbe0 con 0x7f97e006c040 2023-10-16T14:17:12.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.230+0000 7f97de7fc640 1 -- 172.21.15.29:0/2934531362 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d00343f0 con 0x7f97e006c040 2023-10-16T14:17:12.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.230+0000 7f97dffff640 1 -- 172.21.15.29:0/2934531362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e006c040 msgr2=0x7f97e006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:12.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.230+0000 7f97dffff640 1 --2- 172.21.15.29:0/2934531362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e006c040 0x7f97e006c420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97e00cb060 tx=0x7f97d002bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 -- 172.21.15.29:0/2934531362 shutdown_connections 2023-10-16T14:17:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 --2- 172.21.15.29:0/2934531362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e015ebc0 0x7f97e015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 --2- 172.21.15.29:0/2934531362 >> v2:172.21.15.170:3300/0 conn(0x7f97e00ad970 0x7f97e015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 --2- 172.21.15.29:0/2934531362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e006c040 0x7f97e006c420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 -- 172.21.15.29:0/2934531362 >> 172.21.15.29:0/2934531362 conn(0x7f97e00ce6a0 msgr2=0x7f97e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 -- 172.21.15.29:0/2934531362 shutdown_connections 2023-10-16T14:17:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.231+0000 7f97dffff640 1 -- 172.21.15.29:0/2934531362 wait complete. 2023-10-16T14:17:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.233+0000 7f97dffff640 1 Processor -- start 2023-10-16T14:17:12.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.233+0000 7f97dffff640 1 -- start start 2023-10-16T14:17:12.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97dffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e006c040 0x7f97e0152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 0x7f97e0152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97dffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97e015ebc0 0x7f97e0153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97dffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0150050 con 0x7f97e00ad970 2023-10-16T14:17:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97dffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0153e60 con 0x7f97e015ebc0 2023-10-16T14:17:12.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97dffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e01538d0 con 0x7f97e006c040 2023-10-16T14:17:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.234+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 0x7f97e0152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 0x7f97e0152e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43648/0 (socket says 172.21.15.29:43648) 2023-10-16T14:17:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 -- 172.21.15.29:0/2642687172 learned_addr learned my addr 172.21.15.29:0/2642687172 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:12.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97deffd640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e006c040 0x7f97e0152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97df7fe640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97e015ebc0 0x7f97e0153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 -- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e006c040 msgr2=0x7f97e0152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e006c040 0x7f97e0152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 -- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97e015ebc0 msgr2=0x7f97e0153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:12.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97e015ebc0 0x7f97e0153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 -- 172.21.15.29:0/2642687172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0009b30 con 0x7f97e00ad970 2023-10-16T14:17:12.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.235+0000 7f97d67fc640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 0x7f97e0152e50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c570 tx=0x7f97c800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:12.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.236+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8004220 con 0x7f97e00ad970 2023-10-16T14:17:12.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.236+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c8004380 con 0x7f97e00ad970 2023-10-16T14:17:12.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.236+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8003850 con 0x7f97e00ad970 2023-10-16T14:17:12.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.236+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e0153b80 con 0x7f97e00ad970 2023-10-16T14:17:12.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.236+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e014fce0 con 0x7f97e00ad970 2023-10-16T14:17:12.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.237+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e006b330 con 0x7f97e00ad970 2023-10-16T14:17:12.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.238+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f97c80039b0 con 0x7f97e00ad970 2023-10-16T14:17:12.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.239+0000 7f97dcff9640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f97c006ae70 0x7f97c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:12.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.240+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f97c8089390 con 0x7f97e00ad970 2023-10-16T14:17:12.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.240+0000 7f97deffd640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f97c006ae70 0x7f97c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:12.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.241+0000 7f97deffd640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f97c006ae70 0x7f97c006d330 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f97d0035000 tx=0x7f97d0030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:12.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.244+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f97c8054380 con 0x7f97e00ad970 2023-10-16T14:17:12.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.438+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f97e004e9b0 con 0x7f97e00ad970 2023-10-16T14:17:12.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.445+0000 7f97dcff9640 1 -- 172.21.15.29:0/2642687172 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97c80579a0 con 0x7f97e00ad970 2023-10-16T14:17:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f97c006ae70 msgr2=0x7f97c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f97c006ae70 0x7f97c006d330 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f97d0035000 tx=0x7f97d0030000 comp rx=0 tx=0).stop 2023-10-16T14:17:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 msgr2=0x7f97e0152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 0x7f97e0152e50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c570 tx=0x7f97c800ca30 comp rx=0 tx=0).stop 2023-10-16T14:17:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 shutdown_connections 2023-10-16T14:17:12.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f97c006ae70 0x7f97c006d330 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97e00ad970 0x7f97e0152e50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f97e015ebc0 0x7f97e0153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 --2- 172.21.15.29:0/2642687172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f97e006c040 0x7f97e0152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 >> 172.21.15.29:0/2642687172 conn(0x7f97e00ce6a0 msgr2=0x7f97e015dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 shutdown_connections 2023-10-16T14:17:12.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:12.448+0000 7f97dffff640 1 -- 172.21.15.29:0/2642687172 wait complete. 2023-10-16T14:17:12.471 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:17:12.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:17:12.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:17:12.731 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:17:12.732 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:12.732 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:12.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:17:12.749 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:17:12.791 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:12.791 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:12.881 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:12.882 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:12.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:13.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.207+0000 7f4aaf861640 1 Processor -- start 2023-10-16T14:17:13.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.207+0000 7f4aaf861640 1 -- start start 2023-10-16T14:17:13.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.208+0000 7f4aaf861640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa80b0b20 0x7f4aa8164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.208+0000 7f4aaf861640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa806f7f0 con 0x7f4aa8159f00 2023-10-16T14:17:13.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4aaf861640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa806c5e0 con 0x7f4aa8159b20 2023-10-16T14:17:13.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4aaf861640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa806c8c0 con 0x7f4aa80b0b20 2023-10-16T14:17:13.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4aacdfc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4aa8159f00 0x7f4aa80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52026/0 (socket says 172.21.15.29:52026) 2023-10-16T14:17:13.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4aacdfc640 1 -- 172.21.15.29:0/848561842 learned_addr learned my addr 172.21.15.29:0/848561842 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa80b0b20 0x7f4aa8164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514108578 0 0) 0x7f4aa806f7f0 con 0x7f4aa8159f00 2023-10-16T14:17:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.209+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94005730 con 0x7f4aa8159f00 2023-10-16T14:17:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3599244879 0 0) 0x7f4a94005730 con 0x7f4aa8159f00 2023-10-16T14:17:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa80b0b20 msgr2=0x7f4aa8164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa80b0b20 0x7f4aa8164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 >> v1:172.21.15.170:6789/0 conn(0x7f4aa8159b20 legacy=0x7f4aa806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa806cba0 con 0x7f4aa8159f00 2023-10-16T14:17:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4a98001670 con 0x7f4aa8159f00 2023-10-16T14:17:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 >> v1:172.21.15.29:6789/0 conn(0x7f4aa8159f00 legacy=0x7f4aa80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a94005bb0 0x7f4a94005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a940069d0 0x7f4a94006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4a9400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa806c8c0 con 0x7f4a94007390 2023-10-16T14:17:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400e260 con 0x7f4a940069d0 2023-10-16T14:17:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400e420 con 0x7f4a94005bb0 2023-10-16T14:17:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa80b0b20 0x7f4aa8164b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.210+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4a9400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aacdfc640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a940069d0 0x7f4a94006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aad5fd640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a94005bb0 0x7f4a94005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aaddfe640 1 -- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a94005bb0 msgr2=0x7f4a94005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a94005bb0 0x7f4a94005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aaddfe640 1 -- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a940069d0 msgr2=0x7f4a94006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a940069d0 0x7f4a94006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aaddfe640 1 -- 172.21.15.29:0/848561842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94005730 con 0x7f4a94007390 2023-10-16T14:17:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.211+0000 7f4aacdfc640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a940069d0 0x7f4a94006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.212+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4a9400db90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0006fd0 tx=0x7f4a94007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b2604ba0d7d8e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.212+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0032600 con 0x7f4a94007390 2023-10-16T14:17:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.212+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa002e050 con 0x7f4a94007390 2023-10-16T14:17:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.212+0000 7f4a9ffff640 1 -- 172.21.15.29:0/848561842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00329c0 con 0x7f4a94007390 2023-10-16T14:17:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaf861640 1 -- 172.21.15.29:0/848561842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 msgr2=0x7f4a9400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaf861640 1 --2- 172.21.15.29:0/848561842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4a9400db90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0006fd0 tx=0x7f4a94007a00 comp rx=0 tx=0).stop 2023-10-16T14:17:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaddfe640 1 -- 172.21.15.29:0/848561842 reap_dead start 2023-10-16T14:17:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaf861640 1 -- 172.21.15.29:0/848561842 shutdown_connections 2023-10-16T14:17:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaf861640 1 -- 172.21.15.29:0/848561842 >> 172.21.15.29:0/848561842 conn(0x7f4aa80cb410 msgr2=0x7f4aa80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaf861640 1 -- 172.21.15.29:0/848561842 shutdown_connections 2023-10-16T14:17:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.213+0000 7f4aaf861640 1 -- 172.21.15.29:0/848561842 wait complete. 2023-10-16T14:17:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.215+0000 7f4aaf861640 1 Processor -- start 2023-10-16T14:17:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.215+0000 7f4aaf861640 1 -- start start 2023-10-16T14:17:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aaf861640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4aa81577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aaf861640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa815c500 0x7f4aa815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aaf861640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa815a310 0x7f4aa815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aaf861640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa815ad00 con 0x7f4a94007390 2023-10-16T14:17:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aaf861640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa815cec0 con 0x7f4aa815a310 2023-10-16T14:17:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aaf861640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa815d3a0 con 0x7f4aa815c500 2023-10-16T14:17:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.216+0000 7f4aad5fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4aa81577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4aa81577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43666/0 (socket says 172.21.15.29:43666) 2023-10-16T14:17:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 -- 172.21.15.29:0/2633615923 learned_addr learned my addr 172.21.15.29:0/2633615923 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aacdfc640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa815c500 0x7f4aa815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aaddfe640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa815a310 0x7f4aa815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 -- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa815c500 msgr2=0x7f4aa815c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa815c500 0x7f4aa815c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 -- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa815a310 msgr2=0x7f4aa815a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa815a310 0x7f4aa815a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 -- 172.21.15.29:0/2633615923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80c65f0 con 0x7f4a94007390 2023-10-16T14:17:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aacdfc640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa815c500 0x7f4aa815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.217+0000 7f4aad5fd640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4aa81577b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400e8c0 tx=0x7f4aa400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.218+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4003b80 con 0x7f4a94007390 2023-10-16T14:17:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.218+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4003ce0 con 0x7f4a94007390 2023-10-16T14:17:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.218+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40052f0 con 0x7f4a94007390 2023-10-16T14:17:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.218+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa815d0c0 con 0x7f4a94007390 2023-10-16T14:17:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.218+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa80c6b70 con 0x7f4a94007390 2023-10-16T14:17:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.219+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa80c6d30 con 0x7f4a94007390 2023-10-16T14:17:13.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.220+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4aa4005450 con 0x7f4a94007390 2023-10-16T14:17:13.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.221+0000 7f4a9dffb640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4a8c06aea0 0x7f4a8c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.221+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f4aa4089800 con 0x7f4a94007390 2023-10-16T14:17:13.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.222+0000 7f4aacdfc640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4a8c06aea0 0x7f4a8c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.223+0000 7f4aacdfc640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4a8c06aea0 0x7f4a8c06d360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a98001eb0 tx=0x7f4a98001510 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:13.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.225+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4aa40547f0 con 0x7f4a94007390 2023-10-16T14:17:13.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.419+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4aa80c6450 con 0x7f4a94007390 2023-10-16T14:17:13.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.422+0000 7f4a9dffb640 1 -- 172.21.15.29:0/2633615923 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f4aa80c6450 con 0x7f4a94007390 2023-10-16T14:17:13.425 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:13.426 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.424+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4a8c06aea0 msgr2=0x7f4a8c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.424+0000 7f4aaf861640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4a8c06aea0 0x7f4a8c06d360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a98001eb0 tx=0x7f4a98001510 comp rx=0 tx=0).stop 2023-10-16T14:17:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.424+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 msgr2=0x7f4aa81577b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4aa81577b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400e8c0 tx=0x7f4aa400ed80 comp rx=0 tx=0).stop 2023-10-16T14:17:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 shutdown_connections 2023-10-16T14:17:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4a8c06aea0 0x7f4a8c06d360 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4aa815a310 0x7f4aa815a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4aa815c500 0x7f4aa815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 --2- 172.21.15.29:0/2633615923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a94007390 0x7f4aa81577b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 >> 172.21.15.29:0/2633615923 conn(0x7f4aa80cb410 msgr2=0x7f4aa80b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 shutdown_connections 2023-10-16T14:17:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.425+0000 7f4aaf861640 1 -- 172.21.15.29:0/2633615923 wait complete. 2023-10-16T14:17:13.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:13.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.675+0000 7f66011f5640 1 Processor -- start 2023-10-16T14:17:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.675+0000 7f66011f5640 1 -- start start 2023-10-16T14:17:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f66011f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65fc0cd970 0x7f65fc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f66011f5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc0b5d90 con 0x7f65fc15d410 2023-10-16T14:17:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f66011f5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc160fc0 con 0x7f65fc15d030 2023-10-16T14:17:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f66011f5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65fc0d2b50 con 0x7f65fc0cd970 2023-10-16T14:17:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f65fb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65fc0cd970 0x7f65fc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f65f259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65fc15d410 0x7f65fc0cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52040/0 (socket says 172.21.15.29:52040) 2023-10-16T14:17:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.677+0000 7f65f259c640 1 -- 172.21.15.29:0/2888624990 learned_addr learned my addr 172.21.15.29:0/2888624990 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.678+0000 7f65fb59e640 1 -- 172.21.15.29:0/2888624990 >> v1:172.21.15.170:6789/0 conn(0x7f65fc15d030 legacy=0x7f65fc161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.678+0000 7f65fb59e640 1 -- 172.21.15.29:0/2888624990 >> v1:172.21.15.29:6789/0 conn(0x7f65fc15d410 legacy=0x7f65fc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.678+0000 7f65fa59c640 1 -- 172.21.15.29:0/2888624990 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495013036 0 0) 0x7f65fc0b5d90 con 0x7f65fc15d410 2023-10-16T14:17:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.678+0000 7f65fb59e640 1 -- 172.21.15.29:0/2888624990 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc0d25d0 con 0x7f65fc0cd970 2023-10-16T14:17:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.679+0000 7f65fb59e640 1 --2- 172.21.15.29:0/2888624990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc0cd970 0x7f65fc0d1630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009d90 tx=0x7f65f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607b9abda3eb5e7d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.679+0000 7f65fa59c640 1 -- 172.21.15.29:0/2888624990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f402cd00 con 0x7f65fc0cd970 2023-10-16T14:17:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.679+0000 7f65fa59c640 1 -- 172.21.15.29:0/2888624990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f402ce60 con 0x7f65fc0cd970 2023-10-16T14:17:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.679+0000 7f65fa59c640 1 -- 172.21.15.29:0/2888624990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4034660 con 0x7f65fc0cd970 2023-10-16T14:17:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 -- 172.21.15.29:0/2888624990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc0cd970 msgr2=0x7f65fc0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 --2- 172.21.15.29:0/2888624990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc0cd970 0x7f65fc0d1630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009d90 tx=0x7f65f402bf50 comp rx=0 tx=0).stop 2023-10-16T14:17:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 -- 172.21.15.29:0/2888624990 shutdown_connections 2023-10-16T14:17:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 --2- 172.21.15.29:0/2888624990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc0cd970 0x7f65fc0d1630 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 -- 172.21.15.29:0/2888624990 >> 172.21.15.29:0/2888624990 conn(0x7f65fc0cf720 msgr2=0x7f65fc0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:13.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 -- 172.21.15.29:0/2888624990 shutdown_connections 2023-10-16T14:17:13.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.680+0000 7f66011f5640 1 -- 172.21.15.29:0/2888624990 wait complete. 2023-10-16T14:17:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.682+0000 7f66011f5640 1 Processor -- start 2023-10-16T14:17:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.682+0000 7f66011f5640 1 -- start start 2023-10-16T14:17:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f66011f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65fc15d030 0x7f65fc0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f66011f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 0x7f65fc0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f66011f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65fc0c6740 0x7f65fc15f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f66011f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0c9720 con 0x7f65fc15d410 2023-10-16T14:17:13.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f66011f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0b36b0 con 0x7f65fc15d030 2023-10-16T14:17:13.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f66011f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc15ffe0 con 0x7f65fc0c6740 2023-10-16T14:17:13.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f65f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 0x7f65fc0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f65f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 0x7f65fc0c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43674/0 (socket says 172.21.15.29:43674) 2023-10-16T14:17:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f65f259c640 1 -- 172.21.15.29:0/5924199 learned_addr learned my addr 172.21.15.29:0/5924199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f65fad9d640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65fc15d030 0x7f65fc0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.683+0000 7f65fb59e640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65fc0c6740 0x7f65fc15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f259c640 1 -- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65fc0c6740 msgr2=0x7f65fc15f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f259c640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65fc0c6740 0x7f65fc15f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f259c640 1 -- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65fc15d030 msgr2=0x7f65fc0cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f259c640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65fc15d030 0x7f65fc0cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f259c640 1 -- 172.21.15.29:0/5924199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4009a70 con 0x7f65fc15d410 2023-10-16T14:17:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65fb59e640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65fc0c6740 0x7f65fc15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65fad9d640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65fc15d030 0x7f65fc0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f259c640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 0x7f65fc0c91e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f65e400c440 tx=0x7f65e400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40074e0 con 0x7f65fc15d410 2023-10-16T14:17:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4007640 con 0x7f65fc15d410 2023-10-16T14:17:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4004a20 con 0x7f65fc15d410 2023-10-16T14:17:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.684+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc160280 con 0x7f65fc15d410 2023-10-16T14:17:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.685+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc0cb510 con 0x7f65fc15d410 2023-10-16T14:17:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.686+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc1607a0 con 0x7f65fc15d410 2023-10-16T14:17:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.687+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f65e4004b80 con 0x7f65fc15d410 2023-10-16T14:17:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.688+0000 7f65f3fff640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65dc06aea0 0x7f65dc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.688+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f65e4089500 con 0x7f65fc15d410 2023-10-16T14:17:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.689+0000 7f65fad9d640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65dc06aea0 0x7f65dc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.690+0000 7f65fad9d640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65dc06aea0 0x7f65dc06d360 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65ec003eb0 tx=0x7f65ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:13.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.692+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65e4054460 con 0x7f65fc15d410 2023-10-16T14:17:13.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.891+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65fc0b8950 con 0x7f65fc15d410 2023-10-16T14:17:13.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.892+0000 7f65f3fff640 1 -- 172.21.15.29:0/5924199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65e4057a80 con 0x7f65fc15d410 2023-10-16T14:17:13.895 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.894+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65dc06aea0 msgr2=0x7f65dc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.894+0000 7f66011f5640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65dc06aea0 0x7f65dc06d360 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65ec003eb0 tx=0x7f65ec009040 comp rx=0 tx=0).stop 2023-10-16T14:17:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.894+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 msgr2=0x7f65fc0c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.894+0000 7f66011f5640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 0x7f65fc0c91e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f65e400c440 tx=0x7f65e400c900 comp rx=0 tx=0).stop 2023-10-16T14:17:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 shutdown_connections 2023-10-16T14:17:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f65dc06aea0 0x7f65dc06d360 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15d410 0x7f65fc0c91e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65fc0c6740 0x7f65fc15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 --2- 172.21.15.29:0/5924199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65fc15d030 0x7f65fc0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 >> 172.21.15.29:0/5924199 conn(0x7f65fc0cf720 msgr2=0x7f65fc15c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 shutdown_connections 2023-10-16T14:17:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:13.895+0000 7f66011f5640 1 -- 172.21.15.29:0/5924199 wait complete. 2023-10-16T14:17:13.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.143+0000 7f6a2bf39640 1 Processor -- start 2023-10-16T14:17:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.143+0000 7f6a2bf39640 1 -- start start 2023-10-16T14:17:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.145+0000 7f6a2bf39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a240cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.145+0000 7f6a2bf39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6a24071bc0 0x7f6a24069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.145+0000 7f6a2bf39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240cabe0 con 0x7f6a240b4d50 2023-10-16T14:17:14.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.145+0000 7f6a2bf39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a240b4960 con 0x7f6a24071bc0 2023-10-16T14:17:14.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.145+0000 7f6a2bf39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2406ff20 con 0x7f6a240b4580 2023-10-16T14:17:14.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.146+0000 7f6a29cd5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a240cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.146+0000 7f6a2a4d6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6a24071bc0 0x7f6a24069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.146+0000 7f6a2a4d6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6a24071bc0 0x7f6a24069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:41286/0 (socket says 172.21.15.29:41286) 2023-10-16T14:17:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.146+0000 7f6a2a4d6640 1 -- 172.21.15.29:0/2094727619 learned_addr learned my addr 172.21.15.29:0/2094727619 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.147+0000 7f6a29cd5640 1 -- 172.21.15.29:0/2094727619 >> v2:172.21.15.170:3300/0 conn(0x7f6a24071bc0 msgr2=0x7f6a24069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.147+0000 7f6a29cd5640 1 --2- 172.21.15.29:0/2094727619 >> v2:172.21.15.170:3300/0 conn(0x7f6a24071bc0 0x7f6a24069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.147+0000 7f6a29cd5640 1 -- 172.21.15.29:0/2094727619 >> v1:172.21.15.170:6789/0 conn(0x7f6a240b4d50 legacy=0x7f6a240714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.147+0000 7f6a29cd5640 1 -- 172.21.15.29:0/2094727619 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240ca2b0 con 0x7f6a240b4580 2023-10-16T14:17:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.148+0000 7f6a29cd5640 1 --2- 172.21.15.29:0/2094727619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a240cb1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a200097a0 tx=0x7f6a2002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=532453859f3ed0f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.149+0000 7f6a28cd3640 1 -- 172.21.15.29:0/2094727619 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2002d040 con 0x7f6a240b4580 2023-10-16T14:17:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.149+0000 7f6a28cd3640 1 -- 172.21.15.29:0/2094727619 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a20031810 con 0x7f6a240b4580 2023-10-16T14:17:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.149+0000 7f6a28cd3640 1 -- 172.21.15.29:0/2094727619 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20031ba0 con 0x7f6a240b4580 2023-10-16T14:17:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.149+0000 7f6a2bf39640 1 -- 172.21.15.29:0/2094727619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 msgr2=0x7f6a240cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.149+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/2094727619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a240cb1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a200097a0 tx=0x7f6a2002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.150+0000 7f6a2bf39640 1 -- 172.21.15.29:0/2094727619 shutdown_connections 2023-10-16T14:17:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.150+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/2094727619 >> v2:172.21.15.170:3300/0 conn(0x7f6a24071bc0 0x7f6a24069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.150+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/2094727619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a240cb1f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.150+0000 7f6a2bf39640 1 -- 172.21.15.29:0/2094727619 >> 172.21.15.29:0/2094727619 conn(0x7f6a2406bc10 msgr2=0x7f6a2406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.150+0000 7f6a2bf39640 1 -- 172.21.15.29:0/2094727619 shutdown_connections 2023-10-16T14:17:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.150+0000 7f6a2bf39640 1 -- 172.21.15.29:0/2094727619 wait complete. 2023-10-16T14:17:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.152+0000 7f6a2bf39640 1 Processor -- start 2023-10-16T14:17:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.152+0000 7f6a2bf39640 1 -- start start 2023-10-16T14:17:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a2bf39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 0x7f6a240c5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a2bf39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a24149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a2bf39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a240b4d50 0x7f6a240c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a2bf39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24148100 con 0x7f6a24071bc0 2023-10-16T14:17:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a2bf39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24147230 con 0x7f6a240b4d50 2023-10-16T14:17:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a2bf39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2414a1e0 con 0x7f6a240b4580 2023-10-16T14:17:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a29cd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 0x7f6a240c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a29cd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 0x7f6a240c5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42676/0 (socket says 172.21.15.29:42676) 2023-10-16T14:17:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a29cd5640 1 -- 172.21.15.29:0/3240547528 learned_addr learned my addr 172.21.15.29:0/3240547528 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.153+0000 7f6a294d4640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a24149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a2a4d6640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a240b4d50 0x7f6a240c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a29cd5640 1 -- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 msgr2=0x7f6a24149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a29cd5640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a24149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a29cd5640 1 -- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a240b4d50 msgr2=0x7f6a240c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a29cd5640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a240b4d50 0x7f6a240c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a29cd5640 1 -- 172.21.15.29:0/3240547528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20009400 con 0x7f6a24071bc0 2023-10-16T14:17:14.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a294d4640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a24149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:14.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a29cd5640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 0x7f6a240c5450 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6a200098c0 tx=0x7f6a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2003ace0 con 0x7f6a24071bc0 2023-10-16T14:17:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2003ae60 con 0x7f6a24071bc0 2023-10-16T14:17:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20044640 con 0x7f6a24071bc0 2023-10-16T14:17:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.154+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a24147480 con 0x7f6a24071bc0 2023-10-16T14:17:14.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.155+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a24147a40 con 0x7f6a24071bc0 2023-10-16T14:17:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.156+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a24070b60 con 0x7f6a24071bc0 2023-10-16T14:17:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.157+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6a2004d440 con 0x7f6a24071bc0 2023-10-16T14:17:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.158+0000 7f6a1a7fc640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a0806ae70 0x7f6a0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.158+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f6a200ab9c0 con 0x7f6a24071bc0 2023-10-16T14:17:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.158+0000 7f6a294d4640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a0806ae70 0x7f6a0806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.160+0000 7f6a294d4640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a0806ae70 0x7f6a0806d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a240b31e0 tx=0x7f6a14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.162+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a20076920 con 0x7f6a24071bc0 2023-10-16T14:17:14.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.354+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a2404e160 con 0x7f6a24071bc0 2023-10-16T14:17:14.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.355+0000 7f6a1a7fc640 1 -- 172.21.15.29:0/3240547528 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a20035020 con 0x7f6a24071bc0 2023-10-16T14:17:14.358 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a0806ae70 msgr2=0x7f6a0806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a0806ae70 0x7f6a0806d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a240b31e0 tx=0x7f6a14009040 comp rx=0 tx=0).stop 2023-10-16T14:17:14.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 msgr2=0x7f6a240c5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 0x7f6a240c5450 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6a200098c0 tx=0x7f6a200046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 shutdown_connections 2023-10-16T14:17:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6a0806ae70 0x7f6a0806d330 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a240b4d50 0x7f6a240c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a240b4580 0x7f6a24149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 --2- 172.21.15.29:0/3240547528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a24071bc0 0x7f6a240c5450 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.357+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 >> 172.21.15.29:0/3240547528 conn(0x7f6a2406bc10 msgr2=0x7f6a240b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.358+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 shutdown_connections 2023-10-16T14:17:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.358+0000 7f6a2bf39640 1 -- 172.21.15.29:0/3240547528 wait complete. 2023-10-16T14:17:14.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.607+0000 7fbc8cbf4640 1 Processor -- start 2023-10-16T14:17:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.607+0000 7fbc8cbf4640 1 -- start start 2023-10-16T14:17:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc8cbf4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc8815cd70 0x7fbc880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc8cbf4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc880b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc8cbf4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8815d150 con 0x7fbc880b8480 2023-10-16T14:17:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc8cbf4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc880d1180 con 0x7fbc8815cd70 2023-10-16T14:17:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc8cbf4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc880d1460 con 0x7fbc880cded0 2023-10-16T14:17:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc86ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbc880b8480 0x7fbc880cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33954/0 (socket says 172.21.15.29:33954) 2023-10-16T14:17:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc86ffd640 1 -- 172.21.15.29:0/1062786698 learned_addr learned my addr 172.21.15.29:0/1062786698 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.609+0000 7fbc877fe640 1 --2- 172.21.15.29:0/1062786698 >> v2:172.21.15.29:3300/0 conn(0x7fbc8815cd70 0x7fbc880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc87fff640 1 --2- 172.21.15.29:0/1062786698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc880b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc867fc640 1 -- 172.21.15.29:0/1062786698 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583182442 0 0) 0x7fbc8815d150 con 0x7fbc880b8480 2023-10-16T14:17:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc867fc640 1 -- 172.21.15.29:0/1062786698 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc74003520 con 0x7fbc880b8480 2023-10-16T14:17:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc877fe640 1 -- 172.21.15.29:0/1062786698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 msgr2=0x7fbc880b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc877fe640 1 --2- 172.21.15.29:0/1062786698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc880b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc877fe640 1 -- 172.21.15.29:0/1062786698 >> v1:172.21.15.29:6789/0 conn(0x7fbc880b8480 legacy=0x7fbc880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.610+0000 7fbc877fe640 1 -- 172.21.15.29:0/1062786698 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc880d1740 con 0x7fbc8815cd70 2023-10-16T14:17:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.611+0000 7fbc877fe640 1 --2- 172.21.15.29:0/1062786698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880d2220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc780097a0 tx=0x7fbc7802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b773b5e53ddf34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.611+0000 7fbc867fc640 1 -- 172.21.15.29:0/1062786698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7802d040 con 0x7fbc8815cd70 2023-10-16T14:17:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.611+0000 7fbc867fc640 1 -- 172.21.15.29:0/1062786698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc78031810 con 0x7fbc8815cd70 2023-10-16T14:17:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.611+0000 7fbc867fc640 1 -- 172.21.15.29:0/1062786698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78031ba0 con 0x7fbc8815cd70 2023-10-16T14:17:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/1062786698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 msgr2=0x7fbc880d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/1062786698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880d2220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc780097a0 tx=0x7fbc7802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/1062786698 shutdown_connections 2023-10-16T14:17:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/1062786698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc880b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/1062786698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880d2220 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/1062786698 >> 172.21.15.29:0/1062786698 conn(0x7fbc880cf570 msgr2=0x7fbc880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/1062786698 shutdown_connections 2023-10-16T14:17:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.612+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/1062786698 wait complete. 2023-10-16T14:17:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.614+0000 7fbc8cbf4640 1 Processor -- start 2023-10-16T14:17:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.614+0000 7fbc8cbf4640 1 -- start start 2023-10-16T14:17:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc8cbf4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc880b8480 0x7fbc881f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc8cbf4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc8815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc8cbf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc8cbf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc880aaa50 con 0x7fbc8815cd70 2023-10-16T14:17:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc8cbf4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc881f1f90 con 0x7fbc880b8480 2023-10-16T14:17:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc8cbf4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc881f2270 con 0x7fbc880cded0 2023-10-16T14:17:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42690/0 (socket says 172.21.15.29:42690) 2023-10-16T14:17:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.615+0000 7fbc87fff640 1 -- 172.21.15.29:0/3888732660 learned_addr learned my addr 172.21.15.29:0/3888732660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc87fff640 1 -- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 msgr2=0x7fbc8815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc87fff640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc8815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc87fff640 1 -- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc880b8480 msgr2=0x7fbc881f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc877fe640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc880b8480 0x7fbc881f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc87fff640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc880b8480 0x7fbc881f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc87fff640 1 -- 172.21.15.29:0/3888732660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78009400 con 0x7fbc8815cd70 2023-10-16T14:17:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc87fff640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880aa0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000d520 tx=0x7fbc8000d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.616+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc800125f0 con 0x7fbc8815cd70 2023-10-16T14:17:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.617+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc80012750 con 0x7fbc8815cd70 2023-10-16T14:17:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.617+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80012b00 con 0x7fbc8815cd70 2023-10-16T14:17:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.617+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc881f2550 con 0x7fbc8815cd70 2023-10-16T14:17:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.617+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8815a8d0 con 0x7fbc8815cd70 2023-10-16T14:17:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.618+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc881f2740 con 0x7fbc8815cd70 2023-10-16T14:17:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.619+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbc80012c60 con 0x7fbc8815cd70 2023-10-16T14:17:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.620+0000 7fbc7ffff640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbc6806aea0 0x7fbc6806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.620+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fbc80089b40 con 0x7fbc8815cd70 2023-10-16T14:17:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.621+0000 7fbc877fe640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbc6806aea0 0x7fbc6806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.622+0000 7fbc877fe640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbc6806aea0 0x7fbc6806d360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc78036000 tx=0x7fbc7803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.625+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc80054aa0 con 0x7fbc8815cd70 2023-10-16T14:17:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.817+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc8804e9b0 con 0x7fbc8815cd70 2023-10-16T14:17:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.817+0000 7fbc7ffff640 1 -- 172.21.15.29:0/3888732660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc800580c0 con 0x7fbc8815cd70 2023-10-16T14:17:14.821 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:14.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbc6806aea0 msgr2=0x7fbc6806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbc6806aea0 0x7fbc6806d360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc78036000 tx=0x7fbc7803c040 comp rx=0 tx=0).stop 2023-10-16T14:17:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 msgr2=0x7fbc880aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880aa0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000d520 tx=0x7fbc8000d9e0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 shutdown_connections 2023-10-16T14:17:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fbc6806aea0 0x7fbc6806d360 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc8815cd70 0x7fbc880aa0e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc880cded0 0x7fbc8815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 --2- 172.21.15.29:0/3888732660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc880b8480 0x7fbc881f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:14.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 >> 172.21.15.29:0/3888732660 conn(0x7fbc880cf570 msgr2=0x7fbc880b7690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:14.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 shutdown_connections 2023-10-16T14:17:14.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:14.820+0000 7fbc8cbf4640 1 -- 172.21.15.29:0/3888732660 wait complete. 2023-10-16T14:17:14.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:17:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.069+0000 7fadd759e640 1 Processor -- start 2023-10-16T14:17:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.070+0000 7fadd759e640 1 -- start start 2023-10-16T14:17:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.071+0000 7fadd759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fadd80b8480 0x7fadd80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.071+0000 7fadd759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd80d0be0 0x7fadd80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.071+0000 7fadd759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd815d150 con 0x7fadd815cd70 2023-10-16T14:17:15.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.071+0000 7fadd759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd80d1180 con 0x7fadd80b8480 2023-10-16T14:17:15.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.071+0000 7fadd759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80d1460 con 0x7fadd80d0be0 2023-10-16T14:17:15.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fadd80b8480 0x7fadd80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fadd80b8480 0x7fadd80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42698/0 (socket says 172.21.15.29:42698) 2023-10-16T14:17:15.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2582247165 learned_addr learned my addr 172.21.15.29:0/2582247165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd6d9d640 1 --2- 172.21.15.29:0/2582247165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd80d0be0 0x7fadd80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2582247165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd80d0be0 msgr2=0x7fadd80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:15.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2582247165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd80d0be0 0x7fadd80cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2582247165 >> v1:172.21.15.29:6789/0 conn(0x7fadd815cd70 legacy=0x7fadd80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.072+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2582247165 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80d1740 con 0x7fadd80b8480 2023-10-16T14:17:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.074+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2582247165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd80cd7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390cee201b97f311 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.074+0000 7fadd559a640 1 -- 172.21.15.29:0/2582247165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0030840 con 0x7fadd80b8480 2023-10-16T14:17:15.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.074+0000 7fadd559a640 1 -- 172.21.15.29:0/2582247165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc00309a0 con 0x7fadd80b8480 2023-10-16T14:17:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.074+0000 7fadd559a640 1 -- 172.21.15.29:0/2582247165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0030d30 con 0x7fadd80b8480 2023-10-16T14:17:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 -- 172.21.15.29:0/2582247165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 msgr2=0x7fadd80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 --2- 172.21.15.29:0/2582247165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd80cd7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009870 tx=0x7fadc002b980 comp rx=0 tx=0).stop 2023-10-16T14:17:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 -- 172.21.15.29:0/2582247165 shutdown_connections 2023-10-16T14:17:15.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 --2- 172.21.15.29:0/2582247165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd80d0be0 0x7fadd80cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 --2- 172.21.15.29:0/2582247165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd80cd7c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 -- 172.21.15.29:0/2582247165 >> 172.21.15.29:0/2582247165 conn(0x7fadd80cf570 msgr2=0x7fadd80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 -- 172.21.15.29:0/2582247165 shutdown_connections 2023-10-16T14:17:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.075+0000 7fadd759e640 1 -- 172.21.15.29:0/2582247165 wait complete. 2023-10-16T14:17:15.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.077+0000 7fadd759e640 1 Processor -- start 2023-10-16T14:17:15.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.077+0000 7fadd759e640 1 -- start start 2023-10-16T14:17:15.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd8152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80d0be0 0x7fadd80cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd815cd70 0x7fadd80cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80cada0 con 0x7fadd80b8480 2023-10-16T14:17:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd81525c0 con 0x7fadd80d0be0 2023-10-16T14:17:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ca190 con 0x7fadd815cd70 2023-10-16T14:17:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd8152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd8152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42704/0 (socket says 172.21.15.29:42704) 2023-10-16T14:17:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.078+0000 7fadd659c640 1 -- 172.21.15.29:0/3882218532 learned_addr learned my addr 172.21.15.29:0/3882218532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd659c640 1 -- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd815cd70 msgr2=0x7fadd80cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd659c640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd815cd70 0x7fadd80cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd659c640 1 -- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80d0be0 msgr2=0x7fadd80cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd659c640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80d0be0 0x7fadd80cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd659c640 1 -- 172.21.15.29:0/3882218532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00094d0 con 0x7fadd80b8480 2023-10-16T14:17:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd659c640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd8152080 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c730 tx=0x7fadc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:15.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8006d50 con 0x7fadd80b8480 2023-10-16T14:17:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc800ce40 con 0x7fadd80b8480 2023-10-16T14:17:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8003bb0 con 0x7fadd80b8480 2023-10-16T14:17:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd80ca470 con 0x7fadd80b8480 2023-10-16T14:17:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.079+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd80caa30 con 0x7fadd80b8480 2023-10-16T14:17:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.081+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd804e9b0 con 0x7fadd80b8480 2023-10-16T14:17:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.081+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fadc8003d10 con 0x7fadd80b8480 2023-10-16T14:17:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.083+0000 7fadceffd640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fadb806ae70 0x7fadb806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.083+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fadc80892b0 con 0x7fadd80b8480 2023-10-16T14:17:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.084+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fadb806ae70 0x7fadb806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.085+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fadb806ae70 0x7fadb806d330 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadc0036000 tx=0x7fadc003b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:15.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.088+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadc8057d30 con 0x7fadd80b8480 2023-10-16T14:17:15.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.291+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fadd80b2d60 con 0x7fadd80b8480 2023-10-16T14:17:15.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.292+0000 7fadceffd640 1 -- 172.21.15.29:0/3882218532 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fadc80578c0 con 0x7fadd80b8480 2023-10-16T14:17:15.295 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-16T14:17:15.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.294+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fadb806ae70 msgr2=0x7fadb806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.294+0000 7fadd759e640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fadb806ae70 0x7fadb806d330 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadc0036000 tx=0x7fadc003b040 comp rx=0 tx=0).stop 2023-10-16T14:17:15.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.294+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 msgr2=0x7fadd8152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.294+0000 7fadd759e640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd8152080 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c730 tx=0x7fadc800cbf0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 shutdown_connections 2023-10-16T14:17:15.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fadb806ae70 0x7fadb806d330 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd815cd70 0x7fadd80cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80d0be0 0x7fadd80cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 --2- 172.21.15.29:0/3882218532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80b8480 0x7fadd8152080 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 >> 172.21.15.29:0/3882218532 conn(0x7fadd80cf570 msgr2=0x7fadd80d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:15.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 shutdown_connections 2023-10-16T14:17:15.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.295+0000 7fadd759e640 1 -- 172.21.15.29:0/3882218532 wait complete. 2023-10-16T14:17:15.316 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:17:15.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:15.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-16T14:17:15.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:17:15.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:15.424 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:17:15.486 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:15.487 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:15.549 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:15.550 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:15.603 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:15.603 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:15.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.891+0000 7f82eef1a640 1 Processor -- start 2023-10-16T14:17:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.892+0000 7f82eef1a640 1 -- start start 2023-10-16T14:17:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.893+0000 7f82eef1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82e80ad970 0x7f82e80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.893+0000 7f82eef1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e80cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.893+0000 7f82eef1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e815b9d0 con 0x7f82e806c040 2023-10-16T14:17:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.893+0000 7f82eef1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e815bcb0 con 0x7f82e80ad970 2023-10-16T14:17:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.893+0000 7f82eef1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e815bf90 con 0x7f82e80ccfb0 2023-10-16T14:17:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82e80ad970 0x7f82e80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82e80ad970 0x7f82e80cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42708/0 (socket says 172.21.15.29:42708) 2023-10-16T14:17:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 -- 172.21.15.29:0/1313473045 learned_addr learned my addr 172.21.15.29:0/1313473045 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ee719640 1 --2- 172.21.15.29:0/1313473045 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e80cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 -- 172.21.15.29:0/1313473045 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 msgr2=0x7f82e80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 --2- 172.21.15.29:0/1313473045 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e80cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 -- 172.21.15.29:0/1313473045 >> v1:172.21.15.29:6789/0 conn(0x7f82e806c040 legacy=0x7f82e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ecf16640 1 -- 172.21.15.29:0/1313473045 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468907075 0 0) 0x7f82e815b9d0 con 0x7f82e806c040 2023-10-16T14:17:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.894+0000 7f82ed717640 1 -- 172.21.15.29:0/1313473045 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e80cc270 con 0x7f82e80ad970 2023-10-16T14:17:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.896+0000 7f82ed717640 1 --2- 172.21.15.29:0/1313473045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e80ad970 0x7f82e80cca70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009870 tx=0x7f82d402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82143399f040c7d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.896+0000 7f82ecf16640 1 -- 172.21.15.29:0/1313473045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4030840 con 0x7f82e80ad970 2023-10-16T14:17:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.896+0000 7f82ecf16640 1 -- 172.21.15.29:0/1313473045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d40309a0 con 0x7f82e80ad970 2023-10-16T14:17:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.896+0000 7f82ecf16640 1 -- 172.21.15.29:0/1313473045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4030d30 con 0x7f82e80ad970 2023-10-16T14:17:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 -- 172.21.15.29:0/1313473045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e80ad970 msgr2=0x7f82e80cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 --2- 172.21.15.29:0/1313473045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e80ad970 0x7f82e80cca70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009870 tx=0x7f82d402b980 comp rx=0 tx=0).stop 2023-10-16T14:17:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 -- 172.21.15.29:0/1313473045 shutdown_connections 2023-10-16T14:17:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 --2- 172.21.15.29:0/1313473045 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e80cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 --2- 172.21.15.29:0/1313473045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e80ad970 0x7f82e80cca70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 -- 172.21.15.29:0/1313473045 >> 172.21.15.29:0/1313473045 conn(0x7f82e80ce6a0 msgr2=0x7f82e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 -- 172.21.15.29:0/1313473045 shutdown_connections 2023-10-16T14:17:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.897+0000 7f82eef1a640 1 -- 172.21.15.29:0/1313473045 wait complete. 2023-10-16T14:17:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.899+0000 7f82eef1a640 1 Processor -- start 2023-10-16T14:17:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.899+0000 7f82eef1a640 1 -- start start 2023-10-16T14:17:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82eef1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 0x7f82e80c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82eef1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82e80ad970 0x7f82e8153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82eef1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e81f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82eef1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80c67d0 con 0x7f82e806c040 2023-10-16T14:17:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82eef1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e8154140 con 0x7f82e80ad970 2023-10-16T14:17:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82eef1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e81f6680 con 0x7f82e80ccfb0 2023-10-16T14:17:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 0x7f82e80c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 0x7f82e80c2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42716/0 (socket says 172.21.15.29:42716) 2023-10-16T14:17:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 -- 172.21.15.29:0/362846612 learned_addr learned my addr 172.21.15.29:0/362846612 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 -- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 msgr2=0x7f82e81f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e81f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 -- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82e80ad970 msgr2=0x7f82e8153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82e80ad970 0x7f82e8153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.900+0000 7f82edf18640 1 -- 172.21.15.29:0/362846612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d40094d0 con 0x7f82e806c040 2023-10-16T14:17:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.901+0000 7f82edf18640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 0x7f82e80c2a30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00c8c0 tx=0x7f82dc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.901+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc003da0 con 0x7f82e806c040 2023-10-16T14:17:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.901+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc005070 con 0x7f82e806c040 2023-10-16T14:17:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.901+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc005420 con 0x7f82e806c040 2023-10-16T14:17:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.901+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e80c5bc0 con 0x7f82e806c040 2023-10-16T14:17:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.901+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e80c6180 con 0x7f82e806c040 2023-10-16T14:17:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.902+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e806b330 con 0x7f82e806c040 2023-10-16T14:17:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.903+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f82dc005580 con 0x7f82e806c040 2023-10-16T14:17:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.904+0000 7f82e67fc640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f82cc06aea0 0x7f82cc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.904+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f82dc089050 con 0x7f82e806c040 2023-10-16T14:17:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.905+0000 7f82ed717640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f82cc06aea0 0x7f82cc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.906+0000 7f82ed717640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f82cc06aea0 0x7f82cc06d360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f82d4036000 tx=0x7f82d403b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:15.908+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f82dc0bbca0 con 0x7f82e806c040 2023-10-16T14:17:16.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.103+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82e804e9b0 con 0x7f82e806c040 2023-10-16T14:17:16.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.106+0000 7f82e67fc640 1 -- 172.21.15.29:0/362846612 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f82dc017070 con 0x7f82e806c040 2023-10-16T14:17:16.110 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:16.110 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f82cc06aea0 msgr2=0x7f82cc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f82cc06aea0 0x7f82cc06d360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f82d4036000 tx=0x7f82d403b040 comp rx=0 tx=0).stop 2023-10-16T14:17:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 msgr2=0x7f82e80c2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 0x7f82e80c2a30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00c8c0 tx=0x7f82dc00cd80 comp rx=0 tx=0).stop 2023-10-16T14:17:16.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 shutdown_connections 2023-10-16T14:17:16.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f82cc06aea0 0x7f82cc06d360 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82e80ccfb0 0x7f82e81f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82e80ad970 0x7f82e8153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 --2- 172.21.15.29:0/362846612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82e806c040 0x7f82e80c2a30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.109+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 >> 172.21.15.29:0/362846612 conn(0x7f82e80ce6a0 msgr2=0x7f82e806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:16.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.110+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 shutdown_connections 2023-10-16T14:17:16.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.110+0000 7f82eef1a640 1 -- 172.21.15.29:0/362846612 wait complete. 2023-10-16T14:17:16.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:16.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:16.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.366+0000 7fa8dd010640 1 Processor -- start 2023-10-16T14:17:16.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.366+0000 7fa8dd010640 1 -- start start 2023-10-16T14:17:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8dd010640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8d8159e90 0x7fa8d80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8dd010640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa8d806c140 0x7fa8d806c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8dd010640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d806cad0 con 0x7fa8d8159ab0 2023-10-16T14:17:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8dd010640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d815d890 con 0x7fa8d8159e90 2023-10-16T14:17:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8dd010640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d8070360 con 0x7fa8d806c140 2023-10-16T14:17:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8cffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8d8159e90 0x7fa8d80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8d6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8d8159ab0 0x7fa8d806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33988/0 (socket says 172.21.15.29:33988) 2023-10-16T14:17:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8d6d9d640 1 -- 172.21.15.29:0/3409140554 learned_addr learned my addr 172.21.15.29:0/3409140554 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8cffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8d8159e90 0x7fa8d80ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42732/0 (socket says 172.21.15.29:42732) 2023-10-16T14:17:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.368+0000 7fa8d759e640 1 --2- 172.21.15.29:0/3409140554 >> v2:172.21.15.170:3300/0 conn(0x7fa8d806c140 0x7fa8d806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.369+0000 7fa8cffff640 1 -- 172.21.15.29:0/3409140554 >> v2:172.21.15.170:3300/0 conn(0x7fa8d806c140 msgr2=0x7fa8d806c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.369+0000 7fa8cffff640 1 --2- 172.21.15.29:0/3409140554 >> v2:172.21.15.170:3300/0 conn(0x7fa8d806c140 0x7fa8d806c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.369+0000 7fa8cffff640 1 -- 172.21.15.29:0/3409140554 >> v1:172.21.15.29:6789/0 conn(0x7fa8d8159ab0 legacy=0x7fa8d806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.369+0000 7fa8cffff640 1 -- 172.21.15.29:0/3409140554 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80cd1f0 con 0x7fa8d8159e90 2023-10-16T14:17:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.370+0000 7fa8cffff640 1 --2- 172.21.15.29:0/3409140554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d8159e90 0x7fa8d80ce370 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80adcd0 tx=0x7fa8c002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b320e93b1edf60fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.371+0000 7fa8d659c640 1 -- 172.21.15.29:0/3409140554 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0031520 con 0x7fa8d8159e90 2023-10-16T14:17:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.371+0000 7fa8d659c640 1 -- 172.21.15.29:0/3409140554 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0031680 con 0x7fa8d8159e90 2023-10-16T14:17:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.371+0000 7fa8d659c640 1 -- 172.21.15.29:0/3409140554 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0031a10 con 0x7fa8d8159e90 2023-10-16T14:17:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.371+0000 7fa8dd010640 1 -- 172.21.15.29:0/3409140554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d8159e90 msgr2=0x7fa8d80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.371+0000 7fa8dd010640 1 --2- 172.21.15.29:0/3409140554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d8159e90 0x7fa8d80ce370 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80adcd0 tx=0x7fa8c002b7c0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.372+0000 7fa8dd010640 1 -- 172.21.15.29:0/3409140554 shutdown_connections 2023-10-16T14:17:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.372+0000 7fa8dd010640 1 --2- 172.21.15.29:0/3409140554 >> v2:172.21.15.170:3300/0 conn(0x7fa8d806c140 0x7fa8d806c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.372+0000 7fa8dd010640 1 --2- 172.21.15.29:0/3409140554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d8159e90 0x7fa8d80ce370 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.372+0000 7fa8dd010640 1 -- 172.21.15.29:0/3409140554 >> 172.21.15.29:0/3409140554 conn(0x7fa8d80ce9b0 msgr2=0x7fa8d80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.372+0000 7fa8dd010640 1 -- 172.21.15.29:0/3409140554 shutdown_connections 2023-10-16T14:17:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.372+0000 7fa8dd010640 1 -- 172.21.15.29:0/3409140554 wait complete. 2023-10-16T14:17:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.373+0000 7fa8dd010640 1 Processor -- start 2023-10-16T14:17:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.374+0000 7fa8dd010640 1 -- start start 2023-10-16T14:17:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8dd010640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 0x7fa8d814c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8dd010640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8d8159ab0 0x7fa8d806b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8dd010640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8d8159e90 0x7fa8d806bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8dd010640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d814c7d0 con 0x7fa8d806c140 2023-10-16T14:17:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8dd010640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d814c5f0 con 0x7fa8d8159e90 2023-10-16T14:17:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8dd010640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80c4be0 con 0x7fa8d8159ab0 2023-10-16T14:17:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 0x7fa8d814c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 0x7fa8d814c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42736/0 (socket says 172.21.15.29:42736) 2023-10-16T14:17:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 -- 172.21.15.29:0/2961904460 learned_addr learned my addr 172.21.15.29:0/2961904460 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 -- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8d8159ab0 msgr2=0x7fa8d806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8d8159ab0 0x7fa8d806b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 -- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8d8159e90 msgr2=0x7fa8d806bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8d8159e90 0x7fa8d806bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.375+0000 7fa8d6d9d640 1 -- 172.21.15.29:0/2961904460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009400 con 0x7fa8d806c140 2023-10-16T14:17:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.376+0000 7fa8d6d9d640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 0x7fa8d814c0b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800c4a0 tx=0x7fa8c800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.376+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8005ce0 con 0x7fa8d806c140 2023-10-16T14:17:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.376+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8005e40 con 0x7fa8d806c140 2023-10-16T14:17:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.376+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d80c4ec0 con 0x7fa8d806c140 2023-10-16T14:17:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.376+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8007270 con 0x7fa8d806c140 2023-10-16T14:17:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.376+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d80c5480 con 0x7fa8d806c140 2023-10-16T14:17:16.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.378+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d804e160 con 0x7fa8d806c140 2023-10-16T14:17:16.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.378+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa8c80073d0 con 0x7fa8d806c140 2023-10-16T14:17:16.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.380+0000 7fa8cf7fe640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa8b806aea0 0x7fa8b806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.380+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fa8c80889e0 con 0x7fa8d806c140 2023-10-16T14:17:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.380+0000 7fa8cffff640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa8b806aea0 0x7fa8b806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.382+0000 7fa8cffff640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa8b806aea0 0x7fa8b806d360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0035000 tx=0x7fa8c002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:16.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.384+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8c8053a90 con 0x7fa8d806c140 2023-10-16T14:17:16.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.576+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8d806cad0 con 0x7fa8d806c140 2023-10-16T14:17:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.577+0000 7fa8cf7fe640 1 -- 172.21.15.29:0/2961904460 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8d806cad0 con 0x7fa8d806c140 2023-10-16T14:17:16.581 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa8b806aea0 msgr2=0x7fa8b806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa8b806aea0 0x7fa8b806d360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0035000 tx=0x7fa8c002e000 comp rx=0 tx=0).stop 2023-10-16T14:17:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 msgr2=0x7fa8d814c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 0x7fa8d814c0b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800c4a0 tx=0x7fa8c800c960 comp rx=0 tx=0).stop 2023-10-16T14:17:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 shutdown_connections 2023-10-16T14:17:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa8b806aea0 0x7fa8b806d360 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8d8159e90 0x7fa8d806bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8d8159ab0 0x7fa8d806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 --2- 172.21.15.29:0/2961904460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d806c140 0x7fa8d814c0b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 >> 172.21.15.29:0/2961904460 conn(0x7fa8d80ce9b0 msgr2=0x7fa8d81550d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 shutdown_connections 2023-10-16T14:17:16.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.580+0000 7fa8dd010640 1 -- 172.21.15.29:0/2961904460 wait complete. 2023-10-16T14:17:16.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.823+0000 7f637efe9640 1 Processor -- start 2023-10-16T14:17:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.823+0000 7f637efe9640 1 -- start start 2023-10-16T14:17:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.824+0000 7f637efe9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6378159950 0x7f637815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.824+0000 7f637efe9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637815aef0 con 0x7f6378070450 2023-10-16T14:17:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.824+0000 7f637efe9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637815d910 con 0x7f637806c810 2023-10-16T14:17:16.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.824+0000 7f637efe9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637815e490 con 0x7f6378159950 2023-10-16T14:17:16.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.824+0000 7f637d7e6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6378070450 0x7f637806c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34004/0 (socket says 172.21.15.29:34004) 2023-10-16T14:17:16.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.824+0000 7f637d7e6640 1 -- 172.21.15.29:0/2232311180 learned_addr learned my addr 172.21.15.29:0/2232311180 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:16.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637dfe7640 1 --2- 172.21.15.29:0/2232311180 >> v2:172.21.15.170:3300/0 conn(0x7f6378159950 0x7f637815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480746943 0 0) 0x7f6368001110 con 0x7f6378070450 2023-10-16T14:17:16.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364003520 con 0x7f6378070450 2023-10-16T14:17:16.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023286288 0 0) 0x7f6364003520 con 0x7f6378070450 2023-10-16T14:17:16.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 >> v2:172.21.15.170:3300/0 conn(0x7f6378159950 msgr2=0x7f637815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:16.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 --2- 172.21.15.29:0/2232311180 >> v2:172.21.15.170:3300/0 conn(0x7f6378159950 0x7f637815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 >> v1:172.21.15.170:6789/0 conn(0x7f637806c810 legacy=0x7f63780c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637815dbf0 con 0x7f6378070450 2023-10-16T14:17:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6368001800 con 0x7f6378070450 2023-10-16T14:17:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 >> v1:172.21.15.29:6789/0 conn(0x7f6378070450 legacy=0x7f637806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640039c0 0x7f6364003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f6364004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640051d0 0x7f636400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637815e490 con 0x7f63640047c0 2023-10-16T14:17:16.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636400c100 con 0x7f63640039c0 2023-10-16T14:17:16.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.825+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636400c2c0 con 0x7f63640051d0 2023-10-16T14:17:16.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637d7e6640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f6364004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637d7e6640 1 -- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640051d0 msgr2=0x7f636400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637dfe7640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640039c0 0x7f6364003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637e7e8640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640051d0 0x7f636400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637d7e6640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63640051d0 0x7f636400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637d7e6640 1 -- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640039c0 msgr2=0x7f6364003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637d7e6640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640039c0 0x7f6364003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637d7e6640 1 -- 172.21.15.29:0/2232311180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364003520 con 0x7f63640047c0 2023-10-16T14:17:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637dfe7640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63640039c0 0x7f6364003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.826+0000 7f637e7e8640 1 -- 172.21.15.29:0/2232311180 reap_dead start 2023-10-16T14:17:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.827+0000 7f637d7e6640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f6364004c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f636800c110 tx=0x7f63640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fce7965d9eb22c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.828+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368005ad0 con 0x7f63640047c0 2023-10-16T14:17:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.828+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6368005c30 con 0x7f63640047c0 2023-10-16T14:17:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.828+0000 7f637cfe5640 1 -- 172.21.15.29:0/2232311180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636802c7e0 con 0x7f63640047c0 2023-10-16T14:17:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.828+0000 7f637efe9640 1 -- 172.21.15.29:0/2232311180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 msgr2=0x7f6364004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.828+0000 7f637efe9640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f6364004c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f636800c110 tx=0x7f63640057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.829+0000 7f637efe9640 1 -- 172.21.15.29:0/2232311180 shutdown_connections 2023-10-16T14:17:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.829+0000 7f637efe9640 1 --2- 172.21.15.29:0/2232311180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f6364004c40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.829+0000 7f637efe9640 1 -- 172.21.15.29:0/2232311180 >> 172.21.15.29:0/2232311180 conn(0x7f63780ce6a0 msgr2=0x7f63780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.829+0000 7f637efe9640 1 -- 172.21.15.29:0/2232311180 shutdown_connections 2023-10-16T14:17:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.829+0000 7f637efe9640 1 -- 172.21.15.29:0/2232311180 wait complete. 2023-10-16T14:17:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.830+0000 7f637efe9640 1 Processor -- start 2023-10-16T14:17:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.831+0000 7f637efe9640 1 -- start start 2023-10-16T14:17:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.831+0000 7f637efe9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f63780ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.831+0000 7f637efe9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63780af730 0x7f63780b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.831+0000 7f637efe9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6378143640 0x7f6378146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637efe9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63780c4e00 con 0x7f63640047c0 2023-10-16T14:17:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637efe9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63780afbb0 con 0x7f63780af730 2023-10-16T14:17:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637efe9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780ae690 con 0x7f6378143640 2023-10-16T14:17:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f63780ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f63780ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42744/0 (socket says 172.21.15.29:42744) 2023-10-16T14:17:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 -- 172.21.15.29:0/3919158309 learned_addr learned my addr 172.21.15.29:0/3919158309 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637e7e8640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6378143640 0x7f6378146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637d7e6640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63780af730 0x7f63780b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 -- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6378143640 msgr2=0x7f6378146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6378143640 0x7f6378146040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 -- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63780af730 msgr2=0x7f63780b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63780af730 0x7f63780b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637dfe7640 1 -- 172.21.15.29:0/3919158309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780ae970 con 0x7f63640047c0 2023-10-16T14:17:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.832+0000 7f637d7e6640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63780af730 0x7f63780b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.833+0000 7f637dfe7640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f63780ae070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63740107f0 tx=0x7f6374010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:16.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.833+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400eb00 con 0x7f63640047c0 2023-10-16T14:17:16.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.833+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637400ec60 con 0x7f63640047c0 2023-10-16T14:17:16.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.833+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637401c400 con 0x7f63640047c0 2023-10-16T14:17:16.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.833+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63780aec50 con 0x7f63640047c0 2023-10-16T14:17:16.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.833+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63780af210 con 0x7f63640047c0 2023-10-16T14:17:16.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.835+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637804e9b0 con 0x7f63640047c0 2023-10-16T14:17:16.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.836+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f637401c560 con 0x7f63640047c0 2023-10-16T14:17:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.837+0000 7f636e7fc640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f635c06aea0 0x7f635c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.837+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f6374089b80 con 0x7f63640047c0 2023-10-16T14:17:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.838+0000 7f637d7e6640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f635c06aea0 0x7f635c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.839+0000 7f637d7e6640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f635c06aea0 0x7f635c06d360 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6368034000 tx=0x7f6368009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:16.842+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6374054b90 con 0x7f63640047c0 2023-10-16T14:17:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.037+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63780b4620 con 0x7f63640047c0 2023-10-16T14:17:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.038+0000 7f636e7fc640 1 -- 172.21.15.29:0/3919158309 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63740581b0 con 0x7f63640047c0 2023-10-16T14:17:17.042 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f635c06aea0 msgr2=0x7f635c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f635c06aea0 0x7f635c06d360 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6368034000 tx=0x7f6368009030 comp rx=0 tx=0).stop 2023-10-16T14:17:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 msgr2=0x7f63780ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f63780ae070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63740107f0 tx=0x7f6374010cb0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 shutdown_connections 2023-10-16T14:17:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f635c06aea0 0x7f635c06d360 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6378143640 0x7f6378146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63780af730 0x7f63780b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 --2- 172.21.15.29:0/3919158309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63640047c0 0x7f63780ae070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.041+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 >> 172.21.15.29:0/3919158309 conn(0x7f63780ce6a0 msgr2=0x7f6378070860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.042+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 shutdown_connections 2023-10-16T14:17:17.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.042+0000 7f637efe9640 1 -- 172.21.15.29:0/3919158309 wait complete. 2023-10-16T14:17:17.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:17.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.284+0000 7f7ae24ca640 1 Processor -- start 2023-10-16T14:17:17.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.284+0000 7f7ae24ca640 1 -- start start 2023-10-16T14:17:17.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae24ca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 0x7f7adc0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae24ca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7adc0ca560 con 0x7f7adc06c9e0 2023-10-16T14:17:17.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae24ca640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7adc0c9950 con 0x7f7adc0c9060 2023-10-16T14:17:17.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae24ca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0c9c30 con 0x7f7adc06fb50 2023-10-16T14:17:17.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae1cc9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7adc06c9e0 0x7f7adc0ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34020/0 (socket says 172.21.15.29:34020) 2023-10-16T14:17:17.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/2830829353 learned_addr learned my addr 172.21.15.29:0/2830829353 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:17.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.286+0000 7f7ae0cc7640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 0x7f7adc0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850373800 0 0) 0x7f7adc0ca560 con 0x7f7adc06c9e0 2023-10-16T14:17:17.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac8003520 con 0x7f7adc06c9e0 2023-10-16T14:17:17.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4055524806 0 0) 0x7f7ac8003520 con 0x7f7adc06c9e0 2023-10-16T14:17:17.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 msgr2=0x7f7adc0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:17.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 0x7f7adc0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 >> v1:172.21.15.170:6789/0 conn(0x7f7adc0c9060 legacy=0x7f7adc06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc0c9f10 con 0x7f7adc06c9e0 2023-10-16T14:17:17.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7ad4001670 con 0x7f7adc06c9e0 2023-10-16T14:17:17.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 >> v1:172.21.15.29:6789/0 conn(0x7f7adc06c9e0 legacy=0x7f7adc0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ac80039a0 0x7f7ac8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 0x7f7ac80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8004d00 0x7f7ac8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0c9c30 con 0x7f7ac8004d00 2023-10-16T14:17:17.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac800ba60 con 0x7f7ac80039a0 2023-10-16T14:17:17.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac800bc20 con 0x7f7adc06fb50 2023-10-16T14:17:17.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.287+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8004d00 0x7f7ac8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae14c8640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ac80039a0 0x7f7ac8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae0cc7640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 0x7f7ac80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 msgr2=0x7f7ac80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:17.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc06fb50 0x7f7ac80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ac80039a0 msgr2=0x7f7ac8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ac80039a0 0x7f7ac8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.288+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/2830829353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac8003520 con 0x7f7ac8004d00 2023-10-16T14:17:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.289+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8004d00 0x7f7ac8005180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4001eb0 tx=0x7f7ac80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4c62af0296721e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:17.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.289+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4005ad0 con 0x7f7ac8004d00 2023-10-16T14:17:17.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.289+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad4005c30 con 0x7f7ac8004d00 2023-10-16T14:17:17.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.289+0000 7f7ad3fff640 1 -- 172.21.15.29:0/2830829353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40072b0 con 0x7f7ac8004d00 2023-10-16T14:17:17.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae24ca640 1 -- 172.21.15.29:0/2830829353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8004d00 msgr2=0x7f7ac8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/2830829353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ac8004d00 0x7f7ac8005180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4001eb0 tx=0x7f7ac80057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/2830829353 reap_dead start 2023-10-16T14:17:17.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae24ca640 1 -- 172.21.15.29:0/2830829353 shutdown_connections 2023-10-16T14:17:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae24ca640 1 -- 172.21.15.29:0/2830829353 >> 172.21.15.29:0/2830829353 conn(0x7f7adc0ce6a0 msgr2=0x7f7adc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae24ca640 1 -- 172.21.15.29:0/2830829353 shutdown_connections 2023-10-16T14:17:17.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.290+0000 7f7ae24ca640 1 -- 172.21.15.29:0/2830829353 wait complete. 2023-10-16T14:17:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.292+0000 7f7ae24ca640 1 Processor -- start 2023-10-16T14:17:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.292+0000 7f7ae24ca640 1 -- start start 2023-10-16T14:17:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae24ca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7adc156cc0 0x7f7adc15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae24ca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc1570a0 0x7f7adc15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae24ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 0x7f7adc15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae24ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc15c460 con 0x7f7adc15b2f0 2023-10-16T14:17:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae24ca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc15b850 con 0x7f7adc156cc0 2023-10-16T14:17:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae24ca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc15bb30 con 0x7f7adc1570a0 2023-10-16T14:17:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae1cc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 0x7f7adc15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae1cc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 0x7f7adc15ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42766/0 (socket says 172.21.15.29:42766) 2023-10-16T14:17:17.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.293+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/1710834150 learned_addr learned my addr 172.21.15.29:0/1710834150 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:17.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae14c8640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7adc156cc0 0x7f7adc15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae0cc7640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc1570a0 0x7f7adc15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc1570a0 msgr2=0x7f7adc15c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc1570a0 0x7f7adc15c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7adc156cc0 msgr2=0x7f7adc15adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7adc156cc0 0x7f7adc15adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae1cc9640 1 -- 172.21.15.29:0/1710834150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc15be10 con 0x7f7adc15b2f0 2023-10-16T14:17:17.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ae1cc9640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 0x7f7adc15ce10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4001eb0 tx=0x7f7ad4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.294+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4039070 con 0x7f7adc15b2f0 2023-10-16T14:17:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.295+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad403a040 con 0x7f7adc15b2f0 2023-10-16T14:17:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.295+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc15c0f0 con 0x7f7adc15b2f0 2023-10-16T14:17:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.295+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4004ce0 con 0x7f7adc15b2f0 2023-10-16T14:17:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.295+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc1fa020 con 0x7f7adc15b2f0 2023-10-16T14:17:17.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.296+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc15c610 con 0x7f7adc15b2f0 2023-10-16T14:17:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.297+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ad4032070 con 0x7f7adc15b2f0 2023-10-16T14:17:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.298+0000 7f7ad1ffb640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ac006aea0 0x7f7ac006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.298+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f7ad40aa8f0 con 0x7f7adc15b2f0 2023-10-16T14:17:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.299+0000 7f7ae14c8640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ac006aea0 0x7f7ac006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.300+0000 7f7ae14c8640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ac006aea0 0x7f7ac006d360 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8005290 tx=0x7f7ad8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:17.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.302+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ad4075850 con 0x7f7adc15b2f0 2023-10-16T14:17:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.495+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7adc15c280 con 0x7f7adc15b2f0 2023-10-16T14:17:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.496+0000 7f7ad1ffb640 1 -- 172.21.15.29:0/1710834150 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ad4044020 con 0x7f7adc15b2f0 2023-10-16T14:17:17.499 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.498+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ac006aea0 msgr2=0x7f7ac006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.498+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ac006aea0 0x7f7ac006d360 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8005290 tx=0x7f7ad8009040 comp rx=0 tx=0).stop 2023-10-16T14:17:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.498+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 msgr2=0x7f7adc15ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.498+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 0x7f7adc15ce10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4001eb0 tx=0x7f7ad4001930 comp rx=0 tx=0).stop 2023-10-16T14:17:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 shutdown_connections 2023-10-16T14:17:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ac006aea0 0x7f7ac006d360 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7adc15b2f0 0x7f7adc15ce10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7adc1570a0 0x7f7adc15c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 --2- 172.21.15.29:0/1710834150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7adc156cc0 0x7f7adc15adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 >> 172.21.15.29:0/1710834150 conn(0x7f7adc0ce6a0 msgr2=0x7f7adc0cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 shutdown_connections 2023-10-16T14:17:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.499+0000 7f7ae24ca640 1 -- 172.21.15.29:0/1710834150 wait complete. 2023-10-16T14:17:17.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:17:17.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.744+0000 7fad0caca640 1 Processor -- start 2023-10-16T14:17:17.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.744+0000 7fad0caca640 1 -- start start 2023-10-16T14:17:17.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.745+0000 7fad0caca640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fad080ad970 0x7fad080cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.745+0000 7fad0caca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0815b9d0 con 0x7fad080ccfb0 2023-10-16T14:17:17.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad0caca640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0815bcb0 con 0x7fad0806c040 2023-10-16T14:17:17.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad0caca640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0815bf90 con 0x7fad080ad970 2023-10-16T14:17:17.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad07fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fad080ccfb0 0x7fad080cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34028/0 (socket says 172.21.15.29:34028) 2023-10-16T14:17:17.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad07fff640 1 -- 172.21.15.29:0/4029017965 learned_addr learned my addr 172.21.15.29:0/4029017965 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:17.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad06ffd640 1 --2- 172.21.15.29:0/4029017965 >> v2:172.21.15.170:3300/0 conn(0x7fad080ad970 0x7fad080cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454679132 0 0) 0x7fad0815b9d0 con 0x7fad080ccfb0 2023-10-16T14:17:17.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.746+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf4003520 con 0x7fad080ccfb0 2023-10-16T14:17:17.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046332885 0 0) 0x7facf4003520 con 0x7fad080ccfb0 2023-10-16T14:17:17.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 >> v2:172.21.15.170:3300/0 conn(0x7fad080ad970 msgr2=0x7fad080cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:17.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 --2- 172.21.15.29:0/4029017965 >> v2:172.21.15.170:3300/0 conn(0x7fad080ad970 0x7fad080cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 >> v1:172.21.15.170:6789/0 conn(0x7fad0806c040 legacy=0x7fad0806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080cc270 con 0x7fad080ccfb0 2023-10-16T14:17:17.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fad000019f0 con 0x7fad080ccfb0 2023-10-16T14:17:17.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 >> v1:172.21.15.29:6789/0 conn(0x7fad080ccfb0 legacy=0x7fad080cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facf4003970 0x7facf4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7facf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facf4005180 0x7facf400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0815bf90 con 0x7facf40047c0 2023-10-16T14:17:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf400c020 con 0x7facf4005180 2023-10-16T14:17:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf400c1e0 con 0x7facf4003970 2023-10-16T14:17:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.747+0000 7fad06ffd640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7facf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad07fff640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facf4005180 0x7facf400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad077fe640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facf4003970 0x7facf4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad06ffd640 1 -- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facf4003970 msgr2=0x7facf4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad06ffd640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7facf4003970 0x7facf4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad06ffd640 1 -- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facf4005180 msgr2=0x7facf400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad06ffd640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7facf4005180 0x7facf400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad06ffd640 1 -- 172.21.15.29:0/4029017965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4003520 con 0x7facf40047c0 2023-10-16T14:17:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.748+0000 7fad07fff640 1 -- 172.21.15.29:0/4029017965 reap_dead start 2023-10-16T14:17:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.749+0000 7fad06ffd640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7facf4004c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7facf8009a90 tx=0x7facf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ea7b9495076aba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.749+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf802c680 con 0x7facf40047c0 2023-10-16T14:17:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.749+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf802c7e0 con 0x7facf40047c0 2023-10-16T14:17:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.749+0000 7fad067fc640 1 -- 172.21.15.29:0/4029017965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf802cb90 con 0x7facf40047c0 2023-10-16T14:17:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 -- 172.21.15.29:0/4029017965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 msgr2=0x7facf4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7facf4004c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7facf8009a90 tx=0x7facf40057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 -- 172.21.15.29:0/4029017965 shutdown_connections 2023-10-16T14:17:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 --2- 172.21.15.29:0/4029017965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7facf4004c40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 -- 172.21.15.29:0/4029017965 >> 172.21.15.29:0/4029017965 conn(0x7fad080ce6a0 msgr2=0x7fad080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:17.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 -- 172.21.15.29:0/4029017965 shutdown_connections 2023-10-16T14:17:17.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.750+0000 7fad0caca640 1 -- 172.21.15.29:0/4029017965 wait complete. 2023-10-16T14:17:17.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.752+0000 7fad0caca640 1 Processor -- start 2023-10-16T14:17:17.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.752+0000 7fad0caca640 1 -- start start 2023-10-16T14:17:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.753+0000 7fad0caca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7fad081f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.753+0000 7fad0caca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad080b12c0 0x7fad080b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.753+0000 7fad0caca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad080c71b0 0x7fad080c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.753+0000 7fad0caca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad08158440 con 0x7facf40047c0 2023-10-16T14:17:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.753+0000 7fad0caca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c7b70 con 0x7fad080b12c0 2023-10-16T14:17:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.753+0000 7fad0caca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad080b25e0 con 0x7fad080c71b0 2023-10-16T14:17:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7fad081f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7fad081f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42776/0 (socket says 172.21.15.29:42776) 2023-10-16T14:17:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 -- 172.21.15.29:0/2805872375 learned_addr learned my addr 172.21.15.29:0/2805872375 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad06ffd640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad080b12c0 0x7fad080b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad07fff640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad080c71b0 0x7fad080c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 -- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad080c71b0 msgr2=0x7fad080c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad080c71b0 0x7fad080c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 -- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad080b12c0 msgr2=0x7fad080b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad080b12c0 0x7fad080b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad077fe640 1 -- 172.21.15.29:0/2805872375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080b1d40 con 0x7facf40047c0 2023-10-16T14:17:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.754+0000 7fad07fff640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad080c71b0 0x7fad080c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.755+0000 7fad077fe640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7fad081f1e50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7facf000e480 tx=0x7facf000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.755+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf00052e0 con 0x7facf40047c0 2023-10-16T14:17:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.755+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf0005440 con 0x7facf40047c0 2023-10-16T14:17:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.755+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0003810 con 0x7facf40047c0 2023-10-16T14:17:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.755+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad080b2020 con 0x7facf40047c0 2023-10-16T14:17:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.755+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad08158650 con 0x7facf40047c0 2023-10-16T14:17:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.756+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0806b330 con 0x7facf40047c0 2023-10-16T14:17:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.757+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7facf0003970 con 0x7facf40047c0 2023-10-16T14:17:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.758+0000 7facfffff640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7face806ae70 0x7face806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.758+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7facf0089350 con 0x7facf40047c0 2023-10-16T14:17:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.759+0000 7fad06ffd640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7face806ae70 0x7face806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.760+0000 7fad06ffd640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7face806ae70 0x7face806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facf8036000 tx=0x7facf8030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.763+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7facf00543e0 con 0x7facf40047c0 2023-10-16T14:17:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.976+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fad081587e0 con 0x7facf40047c0 2023-10-16T14:17:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.977+0000 7facfffff640 1 -- 172.21.15.29:0/2805872375 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7facf0057a00 con 0x7facf40047c0 2023-10-16T14:17:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-16T14:17:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7face806ae70 msgr2=0x7face806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7face806ae70 0x7face806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facf8036000 tx=0x7facf8030000 comp rx=0 tx=0).stop 2023-10-16T14:17:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 msgr2=0x7fad081f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7fad081f1e50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7facf000e480 tx=0x7facf000e940 comp rx=0 tx=0).stop 2023-10-16T14:17:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 shutdown_connections 2023-10-16T14:17:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7face806ae70 0x7face806d330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad080c71b0 0x7fad080c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad080b12c0 0x7fad080b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 --2- 172.21.15.29:0/2805872375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facf40047c0 0x7fad081f1e50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 >> 172.21.15.29:0/2805872375 conn(0x7fad080ce6a0 msgr2=0x7fad080ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 shutdown_connections 2023-10-16T14:17:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:17.980+0000 7fad0caca640 1 -- 172.21.15.29:0/2805872375 wait complete. 2023-10-16T14:17:18.001 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-16T14:17:18.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:18.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-16T14:17:18.113 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:17:18.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:18.114 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-16T14:17:18.193 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:18.193 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:18.259 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:18.260 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:18.318 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:18.319 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:18.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.600+0000 7f166bec9640 1 Processor -- start 2023-10-16T14:17:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.601+0000 7f166bec9640 1 -- start start 2023-10-16T14:17:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.602+0000 7f166bec9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 0x7f16640c9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.602+0000 7f166bec9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640ae770 con 0x7f16640b0930 2023-10-16T14:17:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.602+0000 7f166bec9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166406f9d0 con 0x7f1664158fa0 2023-10-16T14:17:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.602+0000 7f166bec9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c9ae0 con 0x7f16640cca80 2023-10-16T14:17:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.603+0000 7f166a466640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16640b0930 0x7f16640cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34042/0 (socket says 172.21.15.29:34042) 2023-10-16T14:17:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.603+0000 7f166a466640 1 -- 172.21.15.29:0/1386277503 learned_addr learned my addr 172.21.15.29:0/1386277503 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:18.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.603+0000 7f1669464640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 0x7f16640c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.603+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783927157 0 0) 0x7f165c001110 con 0x7f16640b0930 2023-10-16T14:17:18.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.603+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1650003520 con 0x7f16640b0930 2023-10-16T14:17:18.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938804912 0 0) 0x7f1650003520 con 0x7f16640b0930 2023-10-16T14:17:18.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 msgr2=0x7f16640c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:18.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 0x7f16640c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 >> v1:172.21.15.170:6789/0 conn(0x7f1664158fa0 legacy=0x7f166415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166406fc90 con 0x7f16640b0930 2023-10-16T14:17:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f165c001800 con 0x7f16640b0930 2023-10-16T14:17:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 >> v1:172.21.15.29:6789/0 conn(0x7f16640b0930 legacy=0x7f16640cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16500039a0 0x7f1650003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 0x7f16500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1650004d00 0x7f1650005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c9ae0 con 0x7f1650004d00 2023-10-16T14:17:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165000cc20 con 0x7f16500039a0 2023-10-16T14:17:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16500059d0 con 0x7f16640cca80 2023-10-16T14:17:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f166a466640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1650004d00 0x7f1650005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1669464640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 0x7f16500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.604+0000 7f1669c65640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16500039a0 0x7f1650003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.605+0000 7f166a466640 1 -- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 msgr2=0x7f16500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.605+0000 7f166a466640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16640cca80 0x7f16500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.605+0000 7f166a466640 1 -- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16500039a0 msgr2=0x7f1650003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.605+0000 7f166a466640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16500039a0 0x7f1650003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.605+0000 7f166a466640 1 -- 172.21.15.29:0/1386277503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1650003520 con 0x7f1650004d00 2023-10-16T14:17:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.606+0000 7f166a466640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1650004d00 0x7f1650005180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c240 tx=0x7f16500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7016d73b96bf64eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:18.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.606+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c005ad0 con 0x7f1650004d00 2023-10-16T14:17:18.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.606+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c005c50 con 0x7f1650004d00 2023-10-16T14:17:18.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.606+0000 7f1668c63640 1 -- 172.21.15.29:0/1386277503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c02c8b0 con 0x7f1650004d00 2023-10-16T14:17:18.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166bec9640 1 -- 172.21.15.29:0/1386277503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1650004d00 msgr2=0x7f1650005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166bec9640 1 --2- 172.21.15.29:0/1386277503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1650004d00 0x7f1650005180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c240 tx=0x7f16500057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166a466640 1 -- 172.21.15.29:0/1386277503 reap_dead start 2023-10-16T14:17:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166bec9640 1 -- 172.21.15.29:0/1386277503 shutdown_connections 2023-10-16T14:17:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166bec9640 1 -- 172.21.15.29:0/1386277503 >> 172.21.15.29:0/1386277503 conn(0x7f16640cb930 msgr2=0x7f16640cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166bec9640 1 -- 172.21.15.29:0/1386277503 shutdown_connections 2023-10-16T14:17:18.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.607+0000 7f166bec9640 1 -- 172.21.15.29:0/1386277503 wait complete. 2023-10-16T14:17:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.609+0000 7f166bec9640 1 Processor -- start 2023-10-16T14:17:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.609+0000 7f166bec9640 1 -- start start 2023-10-16T14:17:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166bec9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1650004d00 0x7f16640b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166bec9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 0x7f1664155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166bec9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1664156240 0x7f16641e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166bec9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641570e0 con 0x7f1664156b60 2023-10-16T14:17:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166bec9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641573c0 con 0x7f1664156240 2023-10-16T14:17:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166bec9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16641576a0 con 0x7f1650004d00 2023-10-16T14:17:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 0x7f1664155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 0x7f1664155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42796/0 (socket says 172.21.15.29:42796) 2023-10-16T14:17:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 -- 172.21.15.29:0/84235427 learned_addr learned my addr 172.21.15.29:0/84235427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 -- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1650004d00 msgr2=0x7f16640b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1650004d00 0x7f16640b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f166a466640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1664156240 0x7f16641e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 -- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1664156240 msgr2=0x7f16641e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1664156240 0x7f16641e2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.610+0000 7f1669464640 1 -- 172.21.15.29:0/84235427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664157980 con 0x7f1664156b60 2023-10-16T14:17:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.611+0000 7f1669464640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 0x7f1664155d00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f165400f550 tx=0x7f165400fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.611+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654014870 con 0x7f1664156b60 2023-10-16T14:17:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.611+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16540149d0 con 0x7f1664156b60 2023-10-16T14:17:18.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.611+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654014d80 con 0x7f1664156b60 2023-10-16T14:17:18.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.611+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16640b24b0 con 0x7f1664156b60 2023-10-16T14:17:18.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.611+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16640b4a50 con 0x7f1664156b60 2023-10-16T14:17:18.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.612+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166404e160 con 0x7f1664156b60 2023-10-16T14:17:18.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.613+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1654015410 con 0x7f1664156b60 2023-10-16T14:17:18.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.614+0000 7f165a7fc640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f164806ae70 0x7f164806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:18.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.614+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f1654089ef0 con 0x7f1664156b60 2023-10-16T14:17:18.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.615+0000 7f1669c65640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f164806ae70 0x7f164806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:18.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.616+0000 7f1669c65640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f164806ae70 0x7f164806d330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1660007fd0 tx=0x7f1660009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:18.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.619+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1654054ef0 con 0x7f1664156b60 2023-10-16T14:17:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.813+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16640b4a50 con 0x7f1664156b60 2023-10-16T14:17:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.814+0000 7f165a7fc640 1 -- 172.21.15.29:0/84235427 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f16640b4a50 con 0x7f1664156b60 2023-10-16T14:17:18.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:18.818 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f164806ae70 msgr2=0x7f164806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f164806ae70 0x7f164806d330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1660007fd0 tx=0x7f1660009040 comp rx=0 tx=0).stop 2023-10-16T14:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 msgr2=0x7f1664155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 0x7f1664155d00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f165400f550 tx=0x7f165400fa10 comp rx=0 tx=0).stop 2023-10-16T14:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 shutdown_connections 2023-10-16T14:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f164806ae70 0x7f164806d330 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1664156b60 0x7f1664155d00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1664156240 0x7f16641e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 --2- 172.21.15.29:0/84235427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1650004d00 0x7f16640b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 >> 172.21.15.29:0/84235427 conn(0x7f16640cb930 msgr2=0x7f16640ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 shutdown_connections 2023-10-16T14:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:18.817+0000 7f166bec9640 1 -- 172.21.15.29:0/84235427 wait complete. 2023-10-16T14:17:18.841 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:18.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.067+0000 7f876599e640 1 Processor -- start 2023-10-16T14:17:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.067+0000 7f876599e640 1 -- start start 2023-10-16T14:17:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.068+0000 7f876599e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f876006a4b0 0x7f8760160980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.068+0000 7f876599e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f876006ea40 0x7f87600ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.068+0000 7f876599e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87601618b0 con 0x7f876015ce00 2023-10-16T14:17:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f876599e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87600cdc60 con 0x7f876006a4b0 2023-10-16T14:17:19.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f876599e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8760161010 con 0x7f876006ea40 2023-10-16T14:17:19.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f875effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f876015ce00 0x7f8760069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34052/0 (socket says 172.21.15.29:34052) 2023-10-16T14:17:19.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f875effd640 1 -- 172.21.15.29:0/3446156872 learned_addr learned my addr 172.21.15.29:0/3446156872 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:19.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f875e7fc640 1 --2- 172.21.15.29:0/3446156872 >> v2:172.21.15.29:3300/0 conn(0x7f876006a4b0 0x7f8760160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f875f7fe640 1 --2- 172.21.15.29:0/3446156872 >> v2:172.21.15.170:3300/0 conn(0x7f876006ea40 0x7f87600ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f875dffb640 1 -- 172.21.15.29:0/3446156872 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140799298 0 0) 0x7f87601618b0 con 0x7f876015ce00 2023-10-16T14:17:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.069+0000 7f875dffb640 1 -- 172.21.15.29:0/3446156872 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748003520 con 0x7f876015ce00 2023-10-16T14:17:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.070+0000 7f875e7fc640 1 -- 172.21.15.29:0/3446156872 >> v2:172.21.15.170:3300/0 conn(0x7f876006ea40 msgr2=0x7f87600ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.070+0000 7f875e7fc640 1 --2- 172.21.15.29:0/3446156872 >> v2:172.21.15.170:3300/0 conn(0x7f876006ea40 0x7f87600ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.070+0000 7f875e7fc640 1 -- 172.21.15.29:0/3446156872 >> v1:172.21.15.29:6789/0 conn(0x7f876015ce00 legacy=0x7f8760069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.070+0000 7f875e7fc640 1 -- 172.21.15.29:0/3446156872 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87601612f0 con 0x7f876006a4b0 2023-10-16T14:17:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.071+0000 7f875e7fc640 1 --2- 172.21.15.29:0/3446156872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876006a4b0 0x7f8760160980 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87600c34a0 tx=0x7f875402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a895559b260795c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.071+0000 7f875dffb640 1 -- 172.21.15.29:0/3446156872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875402e070 con 0x7f876006a4b0 2023-10-16T14:17:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.071+0000 7f875dffb640 1 -- 172.21.15.29:0/3446156872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87540315a0 con 0x7f876006a4b0 2023-10-16T14:17:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.071+0000 7f875dffb640 1 -- 172.21.15.29:0/3446156872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754031910 con 0x7f876006a4b0 2023-10-16T14:17:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 -- 172.21.15.29:0/3446156872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876006a4b0 msgr2=0x7f8760160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 --2- 172.21.15.29:0/3446156872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876006a4b0 0x7f8760160980 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87600c34a0 tx=0x7f875402b840 comp rx=0 tx=0).stop 2023-10-16T14:17:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 -- 172.21.15.29:0/3446156872 shutdown_connections 2023-10-16T14:17:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 --2- 172.21.15.29:0/3446156872 >> v2:172.21.15.170:3300/0 conn(0x7f876006ea40 0x7f87600ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 --2- 172.21.15.29:0/3446156872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876006a4b0 0x7f8760160980 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 -- 172.21.15.29:0/3446156872 >> 172.21.15.29:0/3446156872 conn(0x7f87600cc460 msgr2=0x7f87600cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 -- 172.21.15.29:0/3446156872 shutdown_connections 2023-10-16T14:17:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.072+0000 7f876599e640 1 -- 172.21.15.29:0/3446156872 wait complete. 2023-10-16T14:17:19.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.074+0000 7f876599e640 1 Processor -- start 2023-10-16T14:17:19.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.074+0000 7f876599e640 1 -- start start 2023-10-16T14:17:19.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f876599e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876006a4b0 0x7f87600b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f876599e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f876006ea40 0x7f87600ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f876599e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 0x7f87601f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f876599e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876015e230 con 0x7f876015ce00 2023-10-16T14:17:19.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f876599e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8760159c60 con 0x7f876006a4b0 2023-10-16T14:17:19.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f876599e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876015d360 con 0x7f876006ea40 2023-10-16T14:17:19.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f875f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 0x7f87601f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f875f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 0x7f87601f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42814/0 (socket says 172.21.15.29:42814) 2023-10-16T14:17:19.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f875f7fe640 1 -- 172.21.15.29:0/147222785 learned_addr learned my addr 172.21.15.29:0/147222785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:19.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f875f7fe640 1 -- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f876006ea40 msgr2=0x7f87600ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f875effd640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876006a4b0 0x7f87600b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.075+0000 7f875e7fc640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f876006ea40 0x7f87600ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f875f7fe640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f876006ea40 0x7f87600ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f875f7fe640 1 -- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876006a4b0 msgr2=0x7f87600b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f875f7fe640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876006a4b0 0x7f87600b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f875f7fe640 1 -- 172.21.15.29:0/147222785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8754009400 con 0x7f876015ce00 2023-10-16T14:17:19.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f875effd640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876006a4b0 0x7f87600b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:19.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f875f7fe640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 0x7f87601f2330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f875800d600 tx=0x7f875800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87580125f0 con 0x7f876015ce00 2023-10-16T14:17:19.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758012750 con 0x7f876015ce00 2023-10-16T14:17:19.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758012b00 con 0x7f876015ce00 2023-10-16T14:17:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87600b3570 con 0x7f876015ce00 2023-10-16T14:17:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.076+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876015d890 con 0x7f876015ce00 2023-10-16T14:17:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.078+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876015e370 con 0x7f876015ce00 2023-10-16T14:17:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.079+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8758012c60 con 0x7f876015ce00 2023-10-16T14:17:19.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.079+0000 7f8747fff640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f874c06ae70 0x7f874c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.080+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f8758089c00 con 0x7f876015ce00 2023-10-16T14:17:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.080+0000 7f875effd640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f874c06ae70 0x7f874c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.081+0000 7f875effd640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f874c06ae70 0x7f874c06d330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8750005c00 tx=0x7f8750009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.084+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8758054bf0 con 0x7f876015ce00 2023-10-16T14:17:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.273+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f876004e160 con 0x7f876015ce00 2023-10-16T14:17:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.274+0000 7f8747fff640 1 -- 172.21.15.29:0/147222785 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8758058210 con 0x7f876015ce00 2023-10-16T14:17:19.277 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f874c06ae70 msgr2=0x7f874c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f874c06ae70 0x7f874c06d330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8750005c00 tx=0x7f8750009040 comp rx=0 tx=0).stop 2023-10-16T14:17:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 msgr2=0x7f87601f2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 0x7f87601f2330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f875800d600 tx=0x7f875800dac0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 shutdown_connections 2023-10-16T14:17:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f874c06ae70 0x7f874c06d330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015ce00 0x7f87601f2330 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f876006ea40 0x7f87600ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 --2- 172.21.15.29:0/147222785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876006a4b0 0x7f87600b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 >> 172.21.15.29:0/147222785 conn(0x7f87600cc460 msgr2=0x7f876006cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 shutdown_connections 2023-10-16T14:17:19.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.277+0000 7f876599e640 1 -- 172.21.15.29:0/147222785 wait complete. 2023-10-16T14:17:19.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.520+0000 7f5207b16640 1 Processor -- start 2023-10-16T14:17:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.520+0000 7f5207b16640 1 -- start start 2023-10-16T14:17:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.521+0000 7f5207b16640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f520015ad70 0x7f5200069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.521+0000 7f5207b16640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52000cb030 con 0x7f52000caac0 2023-10-16T14:17:19.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.521+0000 7f5207b16640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52000c91c0 con 0x7f52000c9ca0 2023-10-16T14:17:19.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.521+0000 7f5207b16640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52000c94a0 con 0x7f520015ad70 2023-10-16T14:17:19.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.522+0000 7f52060b3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f52000caac0 0x7f52000ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34064/0 (socket says 172.21.15.29:34064) 2023-10-16T14:17:19.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.522+0000 7f52060b3640 1 -- 172.21.15.29:0/3412971002 learned_addr learned my addr 172.21.15.29:0/3412971002 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:19.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.522+0000 7f52058b2640 1 --2- 172.21.15.29:0/3412971002 >> v2:172.21.15.170:3300/0 conn(0x7f520015ad70 0x7f5200069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.522+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080903955 0 0) 0x7f52000cb030 con 0x7f52000caac0 2023-10-16T14:17:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.522+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ec003520 con 0x7f52000caac0 2023-10-16T14:17:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1209985094 0 0) 0x7f51ec003520 con 0x7f52000caac0 2023-10-16T14:17:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 >> v2:172.21.15.170:3300/0 conn(0x7f520015ad70 msgr2=0x7f5200069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 --2- 172.21.15.29:0/3412971002 >> v2:172.21.15.170:3300/0 conn(0x7f520015ad70 0x7f5200069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 >> v1:172.21.15.170:6789/0 conn(0x7f52000c9ca0 legacy=0x7f52000ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52000c9780 con 0x7f52000caac0 2023-10-16T14:17:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f51f8001670 con 0x7f52000caac0 2023-10-16T14:17:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 >> v1:172.21.15.29:6789/0 conn(0x7f52000caac0 legacy=0x7f52000ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51ec0039c0 0x7f51ec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ec0047c0 0x7f51ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec005180 0x7f51ec00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52000c94a0 con 0x7f51ec0047c0 2023-10-16T14:17:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec010490 con 0x7f51ec005180 2023-10-16T14:17:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec010650 con 0x7f51ec0039c0 2023-10-16T14:17:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52050b1640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ec0047c0 0x7f51ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52058b2640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51ec0039c0 0x7f51ec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.523+0000 7f52060b3640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec005180 0x7f51ec00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52050b1640 1 -- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51ec0039c0 msgr2=0x7f51ec003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52050b1640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51ec0039c0 0x7f51ec003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52050b1640 1 -- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec005180 msgr2=0x7f51ec00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52050b1640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec005180 0x7f51ec00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52050b1640 1 -- 172.21.15.29:0/3412971002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec003520 con 0x7f51ec0047c0 2023-10-16T14:17:19.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52058b2640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51ec0039c0 0x7f51ec003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:19.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52060b3640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec005180 0x7f51ec00fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:19.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.524+0000 7f52060b3640 1 -- 172.21.15.29:0/3412971002 reap_dead start 2023-10-16T14:17:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.525+0000 7f52050b1640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ec0047c0 0x7f51ec004c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f51f0001750 tx=0x7f51ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0f62c73c017243 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.525+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f00052d0 con 0x7f51ec0047c0 2023-10-16T14:17:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.525+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f002e050 con 0x7f51ec0047c0 2023-10-16T14:17:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.525+0000 7f52048b0640 1 -- 172.21.15.29:0/3412971002 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0003830 con 0x7f51ec0047c0 2023-10-16T14:17:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.525+0000 7f5207b16640 1 -- 172.21.15.29:0/3412971002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ec0047c0 msgr2=0x7f51ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.526+0000 7f5207b16640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ec0047c0 0x7f51ec004c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f51f0001750 tx=0x7f51ec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.526+0000 7f5207b16640 1 -- 172.21.15.29:0/3412971002 shutdown_connections 2023-10-16T14:17:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.526+0000 7f5207b16640 1 --2- 172.21.15.29:0/3412971002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51ec0047c0 0x7f51ec004c40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.526+0000 7f5207b16640 1 -- 172.21.15.29:0/3412971002 >> 172.21.15.29:0/3412971002 conn(0x7f52000ce9b0 msgr2=0x7f52000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.526+0000 7f5207b16640 1 -- 172.21.15.29:0/3412971002 shutdown_connections 2023-10-16T14:17:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.526+0000 7f5207b16640 1 -- 172.21.15.29:0/3412971002 wait complete. 2023-10-16T14:17:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.528+0000 7f5207b16640 1 Processor -- start 2023-10-16T14:17:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.528+0000 7f5207b16640 1 -- start start 2023-10-16T14:17:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f5207b16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec0047c0 0x7f520006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f5207b16640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520014b2c0 0x7f520015d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f5207b16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 0x7f52000cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f5207b16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520006be90 con 0x7f520015c340 2023-10-16T14:17:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f5207b16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520006bbf0 con 0x7f51ec0047c0 2023-10-16T14:17:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f5207b16640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520015db90 con 0x7f520014b2c0 2023-10-16T14:17:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 0x7f52000cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 0x7f52000cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42824/0 (socket says 172.21.15.29:42824) 2023-10-16T14:17:19.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 -- 172.21.15.29:0/733793931 learned_addr learned my addr 172.21.15.29:0/733793931 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:19.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52058b2640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec0047c0 0x7f520006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52050b1640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520014b2c0 0x7f520015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 -- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520014b2c0 msgr2=0x7f520015d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520014b2c0 0x7f520015d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 -- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec0047c0 msgr2=0x7f520006b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec0047c0 0x7f520006b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52060b3640 1 -- 172.21.15.29:0/733793931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520015b580 con 0x7f520015c340 2023-10-16T14:17:19.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.529+0000 7f52058b2640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec0047c0 0x7f520006b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:19.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.530+0000 7f52060b3640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 0x7f52000cc540 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f52000c5a30 tx=0x7f51f800d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.530+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8007500 con 0x7f520015c340 2023-10-16T14:17:19.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.530+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8007680 con 0x7f520015c340 2023-10-16T14:17:19.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.530+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520015b860 con 0x7f520015c340 2023-10-16T14:17:19.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.530+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004a00 con 0x7f520015c340 2023-10-16T14:17:19.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.530+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520015be20 con 0x7f520015c340 2023-10-16T14:17:19.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.531+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520004e160 con 0x7f520015c340 2023-10-16T14:17:19.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.532+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f51f8004b60 con 0x7f520015c340 2023-10-16T14:17:19.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.534+0000 7f51f67fc640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f51e406ae70 0x7f51e406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.534+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f51f8089250 con 0x7f520015c340 2023-10-16T14:17:19.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.534+0000 7f52058b2640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f51e406ae70 0x7f51e406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.535+0000 7f52058b2640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f51e406ae70 0x7f51e406d330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5200069740 tx=0x7f51fc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.538+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51f80541b0 con 0x7f520015c340 2023-10-16T14:17:19.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.728+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f52000736e0 con 0x7f520015c340 2023-10-16T14:17:19.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.729+0000 7f51f67fc640 1 -- 172.21.15.29:0/733793931 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f51f80577d0 con 0x7f520015c340 2023-10-16T14:17:19.732 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:19.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f51e406ae70 msgr2=0x7f51e406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f51e406ae70 0x7f51e406d330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5200069740 tx=0x7f51fc00b040 comp rx=0 tx=0).stop 2023-10-16T14:17:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 msgr2=0x7f52000cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 0x7f52000cc540 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f52000c5a30 tx=0x7f51f800d900 comp rx=0 tx=0).stop 2023-10-16T14:17:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 shutdown_connections 2023-10-16T14:17:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f51e406ae70 0x7f51e406d330 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520015c340 0x7f52000cc540 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520014b2c0 0x7f520015d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 --2- 172.21.15.29:0/733793931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51ec0047c0 0x7f520006b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.731+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 >> 172.21.15.29:0/733793931 conn(0x7f52000ce9b0 msgr2=0x7f52000b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.732+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 shutdown_connections 2023-10-16T14:17:19.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.732+0000 7f5207b16640 1 -- 172.21.15.29:0/733793931 wait complete. 2023-10-16T14:17:19.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:19.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.977+0000 7f52b2bd0640 1 Processor -- start 2023-10-16T14:17:19.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.978+0000 7f52b2bd0640 1 -- start start 2023-10-16T14:17:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.979+0000 7f52b2bd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f52ac0661c0 0x7f52ac0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.979+0000 7f52b2bd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac06cee0 con 0x7f52ac1618b0 2023-10-16T14:17:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.979+0000 7f52b2bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac0d0890 con 0x7f52ac15d330 2023-10-16T14:17:19.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.979+0000 7f52b2bd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ac06b1e0 con 0x7f52ac0661c0 2023-10-16T14:17:19.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.979+0000 7f52abfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f52ac1618b0 0x7f52ac065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34068/0 (socket says 172.21.15.29:34068) 2023-10-16T14:17:19.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.979+0000 7f52abfff640 1 -- 172.21.15.29:0/2283595739 learned_addr learned my addr 172.21.15.29:0/2283595739 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52b116d640 1 --2- 172.21.15.29:0/2283595739 >> v2:172.21.15.170:3300/0 conn(0x7f52ac0661c0 0x7f52ac0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772476410 0 0) 0x7f52ac06cee0 con 0x7f52ac1618b0 2023-10-16T14:17:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5294003520 con 0x7f52ac1618b0 2023-10-16T14:17:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440012437 0 0) 0x7f5294003520 con 0x7f52ac1618b0 2023-10-16T14:17:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 >> v2:172.21.15.170:3300/0 conn(0x7f52ac0661c0 msgr2=0x7f52ac0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 --2- 172.21.15.29:0/2283595739 >> v2:172.21.15.170:3300/0 conn(0x7f52ac0661c0 0x7f52ac0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 >> v1:172.21.15.170:6789/0 conn(0x7f52ac15d330 legacy=0x7f52ac06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac0d05b0 con 0x7f52ac1618b0 2023-10-16T14:17:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f52a0001670 con 0x7f52ac1618b0 2023-10-16T14:17:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 >> v1:172.21.15.29:6789/0 conn(0x7f52ac1618b0 legacy=0x7f52ac065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5294003970 0x7f5294003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52940047c0 0x7f5294004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5294005180 0x7f529400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac06b1e0 con 0x7f52940047c0 2023-10-16T14:17:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529400c120 con 0x7f5294005180 2023-10-16T14:17:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.980+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529400c2e0 con 0x7f5294003970 2023-10-16T14:17:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52abfff640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52940047c0 0x7f5294004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52b116d640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5294005180 0x7f529400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52b096c640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5294003970 0x7f5294003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52abfff640 1 -- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5294003970 msgr2=0x7f5294003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52abfff640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5294003970 0x7f5294003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52abfff640 1 -- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5294005180 msgr2=0x7f529400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52abfff640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5294005180 0x7f529400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52abfff640 1 -- 172.21.15.29:0/2283595739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5294003520 con 0x7f52940047c0 2023-10-16T14:17:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.981+0000 7f52b116d640 1 -- 172.21.15.29:0/2283595739 reap_dead start 2023-10-16T14:17:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.982+0000 7f52abfff640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52940047c0 0x7f5294004c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c240 tx=0x7f52940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7849aa5a7779514a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.982+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0005ad0 con 0x7f52940047c0 2023-10-16T14:17:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.982+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a0005c30 con 0x7f52940047c0 2023-10-16T14:17:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.982+0000 7f52ab7fe640 1 -- 172.21.15.29:0/2283595739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a002cab0 con 0x7f52940047c0 2023-10-16T14:17:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 -- 172.21.15.29:0/2283595739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52940047c0 msgr2=0x7f5294004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52940047c0 0x7f5294004c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c240 tx=0x7f52940057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 -- 172.21.15.29:0/2283595739 shutdown_connections 2023-10-16T14:17:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/2283595739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52940047c0 0x7f5294004c40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 -- 172.21.15.29:0/2283595739 >> 172.21.15.29:0/2283595739 conn(0x7f52ac0d0ab0 msgr2=0x7f52ac0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 -- 172.21.15.29:0/2283595739 shutdown_connections 2023-10-16T14:17:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.983+0000 7f52b2bd0640 1 -- 172.21.15.29:0/2283595739 wait complete. 2023-10-16T14:17:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.985+0000 7f52b2bd0640 1 Processor -- start 2023-10-16T14:17:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.985+0000 7f52b2bd0640 1 -- start start 2023-10-16T14:17:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b2bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52940047c0 0x7f52ac15c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b2bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52ac0c7af0 0x7f52ac153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b2bd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 0x7f52ac0ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b2bd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac153bd0 con 0x7f52ac0ad000 2023-10-16T14:17:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b2bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac0ac890 con 0x7f52940047c0 2023-10-16T14:17:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b2bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac0ca5f0 con 0x7f52ac0c7af0 2023-10-16T14:17:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b116d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 0x7f52ac0ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b116d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 0x7f52ac0ad480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42834/0 (socket says 172.21.15.29:42834) 2023-10-16T14:17:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.986+0000 7f52b116d640 1 -- 172.21.15.29:0/1329484287 learned_addr learned my addr 172.21.15.29:0/1329484287 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b096c640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52940047c0 0x7f52ac15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52abfff640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52ac0c7af0 0x7f52ac153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b116d640 1 -- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52ac0c7af0 msgr2=0x7f52ac153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b116d640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52ac0c7af0 0x7f52ac153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b116d640 1 -- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52940047c0 msgr2=0x7f52ac15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b116d640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52940047c0 0x7f52ac15c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b116d640 1 -- 172.21.15.29:0/1329484287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac0c7f50 con 0x7f52ac0ad000 2023-10-16T14:17:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b116d640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 0x7f52ac0ad480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0afa70 tx=0x7f52a4011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4004500 con 0x7f52ac0ad000 2023-10-16T14:17:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a4003610 con 0x7f52ac0ad000 2023-10-16T14:17:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.987+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac0ca890 con 0x7f52ac0ad000 2023-10-16T14:17:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.988+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40039c0 con 0x7f52ac0ad000 2023-10-16T14:17:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.988+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ac0acae0 con 0x7f52ac0ad000 2023-10-16T14:17:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.989+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ac0accd0 con 0x7f52ac0ad000 2023-10-16T14:17:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.990+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f52a4003b20 con 0x7f52ac0ad000 2023-10-16T14:17:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.990+0000 7f52a97fa640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f529806aea0 0x7f529806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.991+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f52a4089f30 con 0x7f52ac0ad000 2023-10-16T14:17:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.991+0000 7f52b096c640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f529806aea0 0x7f529806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.992+0000 7f52b096c640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f529806aea0 0x7f529806d360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f529c007fd0 tx=0x7f529c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:19.995+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52a4054f20 con 0x7f52ac0ad000 2023-10-16T14:17:20.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.182+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f52ac1610b0 con 0x7f52ac0ad000 2023-10-16T14:17:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.183+0000 7f52a97fa640 1 -- 172.21.15.29:0/1329484287 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f52a40190a0 con 0x7f52ac0ad000 2023-10-16T14:17:20.186 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.185+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f529806aea0 msgr2=0x7f529806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.185+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f529806aea0 0x7f529806d360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f529c007fd0 tx=0x7f529c009040 comp rx=0 tx=0).stop 2023-10-16T14:17:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.185+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 msgr2=0x7f52ac0ad480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.185+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 0x7f52ac0ad480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0afa70 tx=0x7f52a4011920 comp rx=0 tx=0).stop 2023-10-16T14:17:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 shutdown_connections 2023-10-16T14:17:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f529806aea0 0x7f529806d360 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ac0ad000 0x7f52ac0ad480 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f52ac0c7af0 0x7f52ac153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 --2- 172.21.15.29:0/1329484287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52940047c0 0x7f52ac15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 >> 172.21.15.29:0/1329484287 conn(0x7f52ac0d0ab0 msgr2=0x7f52ac157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 shutdown_connections 2023-10-16T14:17:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.186+0000 7f52b2bd0640 1 -- 172.21.15.29:0/1329484287 wait complete. 2023-10-16T14:17:20.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:17:20.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.418+0000 7fc84ad0c640 1 Processor -- start 2023-10-16T14:17:20.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.419+0000 7fc84ad0c640 1 -- start start 2023-10-16T14:17:20.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84ad0c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc8440b8480 0x7fc8440cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84ad0c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8440d0be0 0x7fc8440cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84ad0c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84415d150 con 0x7fc84415cd70 2023-10-16T14:17:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84ad0c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440d1180 con 0x7fc8440d0be0 2023-10-16T14:17:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84ad0c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440d1460 con 0x7fc8440b8480 2023-10-16T14:17:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84a50b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8440d0be0 0x7fc8440cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84a50b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8440d0be0 0x7fc8440cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42838/0 (socket says 172.21.15.29:42838) 2023-10-16T14:17:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc84a50b640 1 -- 172.21.15.29:0/116428367 learned_addr learned my addr 172.21.15.29:0/116428367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.420+0000 7fc849509640 1 --2- 172.21.15.29:0/116428367 >> v2:172.21.15.170:3300/0 conn(0x7fc8440b8480 0x7fc8440cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.421+0000 7fc84a50b640 1 -- 172.21.15.29:0/116428367 >> v2:172.21.15.170:3300/0 conn(0x7fc8440b8480 msgr2=0x7fc8440cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.421+0000 7fc84a50b640 1 --2- 172.21.15.29:0/116428367 >> v2:172.21.15.170:3300/0 conn(0x7fc8440b8480 0x7fc8440cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.421+0000 7fc84a50b640 1 -- 172.21.15.29:0/116428367 >> v1:172.21.15.170:6789/0 conn(0x7fc84415cd70 legacy=0x7fc8440d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.421+0000 7fc84a50b640 1 -- 172.21.15.29:0/116428367 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440d1740 con 0x7fc8440d0be0 2023-10-16T14:17:20.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.422+0000 7fc84a50b640 1 --2- 172.21.15.29:0/116428367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440d0be0 0x7fc8440cdd00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc844066a00 tx=0x7fc83c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbcd7bc50fd274d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:20.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.422+0000 7fc848d08640 1 -- 172.21.15.29:0/116428367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0317e0 con 0x7fc8440d0be0 2023-10-16T14:17:20.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.422+0000 7fc848d08640 1 -- 172.21.15.29:0/116428367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c031940 con 0x7fc8440d0be0 2023-10-16T14:17:20.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.422+0000 7fc848d08640 1 -- 172.21.15.29:0/116428367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c031cd0 con 0x7fc8440d0be0 2023-10-16T14:17:20.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 -- 172.21.15.29:0/116428367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440d0be0 msgr2=0x7fc8440cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/116428367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440d0be0 0x7fc8440cdd00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc844066a00 tx=0x7fc83c02b880 comp rx=0 tx=0).stop 2023-10-16T14:17:20.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 -- 172.21.15.29:0/116428367 shutdown_connections 2023-10-16T14:17:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/116428367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440d0be0 0x7fc8440cdd00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/116428367 >> v2:172.21.15.170:3300/0 conn(0x7fc8440b8480 0x7fc8440cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 -- 172.21.15.29:0/116428367 >> 172.21.15.29:0/116428367 conn(0x7fc8440cf570 msgr2=0x7fc8440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 -- 172.21.15.29:0/116428367 shutdown_connections 2023-10-16T14:17:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.423+0000 7fc84ad0c640 1 -- 172.21.15.29:0/116428367 wait complete. 2023-10-16T14:17:20.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.425+0000 7fc84ad0c640 1 Processor -- start 2023-10-16T14:17:20.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.425+0000 7fc84ad0c640 1 -- start start 2023-10-16T14:17:20.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84ad0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 0x7fc844153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84ad0c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8440d0be0 0x7fc8440cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84ad0c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc84415cd70 0x7fc84415dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84ad0c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440c9d40 con 0x7fc8440b8480 2023-10-16T14:17:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84ad0c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440ca000 con 0x7fc84415cd70 2023-10-16T14:17:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84ad0c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440ca2e0 con 0x7fc8440d0be0 2023-10-16T14:17:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 0x7fc844153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 0x7fc844153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42842/0 (socket says 172.21.15.29:42842) 2023-10-16T14:17:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 -- 172.21.15.29:0/4159262199 learned_addr learned my addr 172.21.15.29:0/4159262199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849509640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8440d0be0 0x7fc8440cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 -- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8440d0be0 msgr2=0x7fc8440cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8440d0be0 0x7fc8440cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc84a50b640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc84415cd70 0x7fc84415dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 -- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc84415cd70 msgr2=0x7fc84415dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc84415cd70 0x7fc84415dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.426+0000 7fc849d0a640 1 -- 172.21.15.29:0/4159262199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0094d0 con 0x7fc8440b8480 2023-10-16T14:17:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.427+0000 7fc849d0a640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 0x7fc844153560 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c7c0 tx=0x7fc83400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:20.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.427+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400aa50 con 0x7fc8440b8480 2023-10-16T14:17:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.427+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83400abb0 con 0x7fc8440b8480 2023-10-16T14:17:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.427+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8440ca5c0 con 0x7fc8440b8480 2023-10-16T14:17:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.427+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834012420 con 0x7fc8440b8480 2023-10-16T14:17:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.427+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84415f300 con 0x7fc8440b8480 2023-10-16T14:17:20.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.428+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8440ca800 con 0x7fc8440b8480 2023-10-16T14:17:20.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.429+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc834012580 con 0x7fc8440b8480 2023-10-16T14:17:20.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.430+0000 7fc8427fc640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc83006ae70 0x7fc83006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.430+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fc8340896d0 con 0x7fc8440b8480 2023-10-16T14:17:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.431+0000 7fc849509640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc83006ae70 0x7fc83006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.432+0000 7fc849509640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc83006ae70 0x7fc83006d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc838006fd0 tx=0x7fc838008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.435+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc834054630 con 0x7fc8440b8480 2023-10-16T14:17:20.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.640+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc84404e9b0 con 0x7fc8440b8480 2023-10-16T14:17:20.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.641+0000 7fc8427fc640 1 -- 172.21.15.29:0/4159262199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc834057c50 con 0x7fc8440b8480 2023-10-16T14:17:20.645 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-16T14:17:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.643+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc83006ae70 msgr2=0x7fc83006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.643+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc83006ae70 0x7fc83006d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc838006fd0 tx=0x7fc838008040 comp rx=0 tx=0).stop 2023-10-16T14:17:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.643+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 msgr2=0x7fc844153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:20.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.643+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 0x7fc844153560 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c7c0 tx=0x7fc83400cc80 comp rx=0 tx=0).stop 2023-10-16T14:17:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 shutdown_connections 2023-10-16T14:17:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc83006ae70 0x7fc83006d330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc84415cd70 0x7fc84415dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8440d0be0 0x7fc8440cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 --2- 172.21.15.29:0/4159262199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8440b8480 0x7fc844153560 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 >> 172.21.15.29:0/4159262199 conn(0x7fc8440cf570 msgr2=0x7fc8440ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 shutdown_connections 2023-10-16T14:17:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:20.644+0000 7fc84ad0c640 1 -- 172.21.15.29:0/4159262199 wait complete. 2023-10-16T14:17:20.667 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-16T14:17:20.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:20.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-16T14:17:20.770 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:17:20.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:20.771 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-16T14:17:20.827 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:20.827 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:20.888 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:20.889 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:20.942 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:17:20.943 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:20.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:21.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.228+0000 7f538359e640 1 Processor -- start 2023-10-16T14:17:21.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.228+0000 7f538359e640 1 -- start start 2023-10-16T14:17:21.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f538359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f538406c040 0x7f538406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f538359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f53840cda80 0x7f538415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f538359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538406c960 con 0x7f53840ad970 2023-10-16T14:17:21.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f538359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538415b9d0 con 0x7f538406c040 2023-10-16T14:17:21.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f538359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538415bcb0 con 0x7f53840cda80 2023-10-16T14:17:21.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f537bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f53840ad970 0x7f538415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34072/0 (socket says 172.21.15.29:34072) 2023-10-16T14:17:21.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f537bfff640 1 -- 172.21.15.29:0/3307621404 learned_addr learned my addr 172.21.15.29:0/3307621404 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:21.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.230+0000 7f538259c640 1 --2- 172.21.15.29:0/3307621404 >> v2:172.21.15.29:3300/0 conn(0x7f538406c040 0x7f538406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.231+0000 7f5382d9d640 1 --2- 172.21.15.29:0/3307621404 >> v2:172.21.15.170:3300/0 conn(0x7f53840cda80 0x7f538415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.231+0000 7f538259c640 1 -- 172.21.15.29:0/3307621404 >> v2:172.21.15.170:3300/0 conn(0x7f53840cda80 msgr2=0x7f538415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:21.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.231+0000 7f538259c640 1 --2- 172.21.15.29:0/3307621404 >> v2:172.21.15.170:3300/0 conn(0x7f53840cda80 0x7f538415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.231+0000 7f538259c640 1 -- 172.21.15.29:0/3307621404 >> v1:172.21.15.29:6789/0 conn(0x7f53840ad970 legacy=0x7f538415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.231+0000 7f538259c640 1 -- 172.21.15.29:0/3307621404 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538415bf90 con 0x7f538406c040 2023-10-16T14:17:21.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.231+0000 7f5381d9b640 1 -- 172.21.15.29:0/3307621404 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430972493 0 0) 0x7f538406c960 con 0x7f53840ad970 2023-10-16T14:17:21.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.232+0000 7f538259c640 1 --2- 172.21.15.29:0/3307621404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f538406c420 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f53840cf300 tx=0x7f537402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c69643e44c46b5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:21.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.232+0000 7f5381d9b640 1 -- 172.21.15.29:0/3307621404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374031830 con 0x7f538406c040 2023-10-16T14:17:21.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f5381d9b640 1 -- 172.21.15.29:0/3307621404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374031990 con 0x7f538406c040 2023-10-16T14:17:21.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f5381d9b640 1 -- 172.21.15.29:0/3307621404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374031d20 con 0x7f538406c040 2023-10-16T14:17:21.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 -- 172.21.15.29:0/3307621404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 msgr2=0x7f538406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 --2- 172.21.15.29:0/3307621404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f538406c420 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f53840cf300 tx=0x7f537402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 -- 172.21.15.29:0/3307621404 shutdown_connections 2023-10-16T14:17:21.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 --2- 172.21.15.29:0/3307621404 >> v2:172.21.15.170:3300/0 conn(0x7f53840cda80 0x7f538415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 --2- 172.21.15.29:0/3307621404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f538406c420 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 -- 172.21.15.29:0/3307621404 >> 172.21.15.29:0/3307621404 conn(0x7f53840ce6a0 msgr2=0x7f53840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:21.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 -- 172.21.15.29:0/3307621404 shutdown_connections 2023-10-16T14:17:21.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.233+0000 7f538359e640 1 -- 172.21.15.29:0/3307621404 wait complete. 2023-10-16T14:17:21.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.235+0000 7f538359e640 1 Processor -- start 2023-10-16T14:17:21.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.235+0000 7f538359e640 1 -- start start 2023-10-16T14:17:21.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.236+0000 7f538359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f53841f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.236+0000 7f538359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53840ad970 0x7f53840b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.236+0000 7f538359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53840cda80 0x7f53841587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.236+0000 7f538359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53841f2bd0 con 0x7f538406c040 2023-10-16T14:17:21.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.236+0000 7f538359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53841f2550 con 0x7f53840ad970 2023-10-16T14:17:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.236+0000 7f538359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840ae210 con 0x7f53840cda80 2023-10-16T14:17:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f53841f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f53841f2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42860/0 (socket says 172.21.15.29:42860) 2023-10-16T14:17:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 -- 172.21.15.29:0/3487010049 learned_addr learned my addr 172.21.15.29:0/3487010049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:21.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f537bfff640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53840ad970 0x7f53840b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f5382d9d640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53840cda80 0x7f53841587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 -- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53840cda80 msgr2=0x7f53841587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53840cda80 0x7f53841587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 -- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53840ad970 msgr2=0x7f53840b2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53840ad970 0x7f53840b2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.237+0000 7f538259c640 1 -- 172.21.15.29:0/3487010049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374009400 con 0x7f538406c040 2023-10-16T14:17:21.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.238+0000 7f538259c640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f53841f2ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5374009890 tx=0x7f53740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:21.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.238+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537403ba70 con 0x7f538406c040 2023-10-16T14:17:21.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.238+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537403bbf0 con 0x7f538406c040 2023-10-16T14:17:21.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.238+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53841f27f0 con 0x7f538406c040 2023-10-16T14:17:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.238+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53740453f0 con 0x7f538406c040 2023-10-16T14:17:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.238+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53840ade50 con 0x7f538406c040 2023-10-16T14:17:21.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.239+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538406b330 con 0x7f538406c040 2023-10-16T14:17:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.240+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5374045550 con 0x7f538406c040 2023-10-16T14:17:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.241+0000 7f537affd640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f536406ae70 0x7f536406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.241+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f53740ab890 con 0x7f538406c040 2023-10-16T14:17:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.242+0000 7f537bfff640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f536406ae70 0x7f536406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.243+0000 7f537bfff640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f536406ae70 0x7f536406d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f53840b0f70 tx=0x7f536c007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:21.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.245+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5374076a40 con 0x7f538406c040 2023-10-16T14:17:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.443+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f538404e9b0 con 0x7f538406c040 2023-10-16T14:17:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.446+0000 7f537affd640 1 -- 172.21.15.29:0/3487010049 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f537407a060 con 0x7f538406c040 2023-10-16T14:17:21.450 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:21.450 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f536406ae70 msgr2=0x7f536406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f536406ae70 0x7f536406d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f53840b0f70 tx=0x7f536c007b50 comp rx=0 tx=0).stop 2023-10-16T14:17:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 msgr2=0x7f53841f2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f53841f2ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5374009890 tx=0x7f53740046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 shutdown_connections 2023-10-16T14:17:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f536406ae70 0x7f536406d330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53840cda80 0x7f53841587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53840ad970 0x7f53840b2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 --2- 172.21.15.29:0/3487010049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f538406c040 0x7f53841f2ff0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 >> 172.21.15.29:0/3487010049 conn(0x7f53840ce6a0 msgr2=0x7f53840ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 shutdown_connections 2023-10-16T14:17:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.449+0000 7f538359e640 1 -- 172.21.15.29:0/3487010049 wait complete. 2023-10-16T14:17:21.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:21.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:21.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.699+0000 7f6661561640 1 Processor -- start 2023-10-16T14:17:21.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.699+0000 7f6661561640 1 -- start start 2023-10-16T14:17:21.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f6661561640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 0x7f665c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f6661561640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f6661561640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c0cb140 con 0x7f665c06a2e0 2023-10-16T14:17:21.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f6661561640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c161a50 con 0x7f665c0addb0 2023-10-16T14:17:21.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f6661561640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c06a750 con 0x7f665c0c9000 2023-10-16T14:17:21.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f665affd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 0x7f665c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f665affd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 0x7f665c0caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:41530/0 (socket says 172.21.15.29:41530) 2023-10-16T14:17:21.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f665affd640 1 -- 172.21.15.29:0/2815640096 learned_addr learned my addr 172.21.15.29:0/2815640096 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:21.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.701+0000 7f665a7fc640 1 --2- 172.21.15.29:0/2815640096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.702+0000 7f665a7fc640 1 -- 172.21.15.29:0/2815640096 >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 msgr2=0x7f665c0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.702+0000 7f665a7fc640 1 --2- 172.21.15.29:0/2815640096 >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 0x7f665c0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.702+0000 7f665a7fc640 1 -- 172.21.15.29:0/2815640096 >> v1:172.21.15.170:6789/0 conn(0x7f665c06a2e0 legacy=0x7f665c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.702+0000 7f665a7fc640 1 -- 172.21.15.29:0/2815640096 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c06b030 con 0x7f665c0c9000 2023-10-16T14:17:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.702+0000 7f665affd640 1 --2- 172.21.15.29:0/2815640096 >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 0x7f665c0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:17:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.704+0000 7f665a7fc640 1 --2- 172.21.15.29:0/2815640096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c069da0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f66440097a0 tx=0x7f664402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982f5f4e3b791fd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.704+0000 7f6659ffb640 1 -- 172.21.15.29:0/2815640096 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664402d040 con 0x7f665c0c9000 2023-10-16T14:17:21.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.704+0000 7f6659ffb640 1 -- 172.21.15.29:0/2815640096 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6644031810 con 0x7f665c0c9000 2023-10-16T14:17:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.704+0000 7f6659ffb640 1 -- 172.21.15.29:0/2815640096 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6644031ba0 con 0x7f665c0c9000 2023-10-16T14:17:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 -- 172.21.15.29:0/2815640096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 msgr2=0x7f665c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 --2- 172.21.15.29:0/2815640096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c069da0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f66440097a0 tx=0x7f664402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 -- 172.21.15.29:0/2815640096 shutdown_connections 2023-10-16T14:17:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 --2- 172.21.15.29:0/2815640096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c069da0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 --2- 172.21.15.29:0/2815640096 >> v2:172.21.15.170:3300/0 conn(0x7f665c0addb0 0x7f665c0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 -- 172.21.15.29:0/2815640096 >> 172.21.15.29:0/2815640096 conn(0x7f665c0ce9b0 msgr2=0x7f665c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 -- 172.21.15.29:0/2815640096 shutdown_connections 2023-10-16T14:17:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.705+0000 7f6661561640 1 -- 172.21.15.29:0/2815640096 wait complete. 2023-10-16T14:17:21.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.707+0000 7f6661561640 1 Processor -- start 2023-10-16T14:17:21.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.707+0000 7f6661561640 1 -- start start 2023-10-16T14:17:21.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f6661561640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 0x7f665c15dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f6661561640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c0addb0 0x7f665c163d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f6661561640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c1642a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f6661561640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0c8e50 con 0x7f665c06a2e0 2023-10-16T14:17:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f6661561640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c15e4f0 con 0x7f665c0addb0 2023-10-16T14:17:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f6661561640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0700e0 con 0x7f665c0c9000 2023-10-16T14:17:21.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f665affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 0x7f665c15dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f665affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 0x7f665c15dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42872/0 (socket says 172.21.15.29:42872) 2023-10-16T14:17:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.708+0000 7f665affd640 1 -- 172.21.15.29:0/4068522081 learned_addr learned my addr 172.21.15.29:0/4068522081 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665a7fc640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c0addb0 0x7f665c163d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665b7fe640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c1642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665affd640 1 -- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 msgr2=0x7f665c1642a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665affd640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c1642a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665affd640 1 -- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c0addb0 msgr2=0x7f665c163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665affd640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c0addb0 0x7f665c163d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665affd640 1 -- 172.21.15.29:0/4068522081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c009b30 con 0x7f665c06a2e0 2023-10-16T14:17:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665a7fc640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c0addb0 0x7f665c163d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.709+0000 7f665affd640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 0x7f665c15dfb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f664c00efc0 tx=0x7f664c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.710+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c00ce60 con 0x7f665c06a2e0 2023-10-16T14:17:21.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.710+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c00f040 con 0x7f665c06a2e0 2023-10-16T14:17:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.710+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c014680 con 0x7f665c06a2e0 2023-10-16T14:17:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.710+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6644009400 con 0x7f665c06a2e0 2023-10-16T14:17:21.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.710+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c06fe00 con 0x7f665c06a2e0 2023-10-16T14:17:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.711+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c04e160 con 0x7f665c06a2e0 2023-10-16T14:17:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.712+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f664c0147e0 con 0x7f665c06a2e0 2023-10-16T14:17:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.713+0000 7f66537fe640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f663c06ae70 0x7f663c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.713+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f664c08a130 con 0x7f665c06a2e0 2023-10-16T14:17:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.714+0000 7f665a7fc640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f663c06ae70 0x7f663c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:21.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.715+0000 7f665a7fc640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f663c06ae70 0x7f663c06d330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66440097a0 tx=0x7f664403c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:21.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.717+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f664c055120 con 0x7f665c06a2e0 2023-10-16T14:17:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.911+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f665c06c5b0 con 0x7f665c06a2e0 2023-10-16T14:17:21.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.912+0000 7f66537fe640 1 -- 172.21.15.29:0/4068522081 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f664c058740 con 0x7f665c06a2e0 2023-10-16T14:17:21.915 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.914+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f663c06ae70 msgr2=0x7f663c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.914+0000 7f6661561640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f663c06ae70 0x7f663c06d330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66440097a0 tx=0x7f664403c040 comp rx=0 tx=0).stop 2023-10-16T14:17:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.914+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 msgr2=0x7f665c15dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:21.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.914+0000 7f6661561640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 0x7f665c15dfb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f664c00efc0 tx=0x7f664c00c490 comp rx=0 tx=0).stop 2023-10-16T14:17:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 shutdown_connections 2023-10-16T14:17:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f663c06ae70 0x7f663c06d330 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f665c0c9000 0x7f665c1642a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f665c0addb0 0x7f665c163d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 --2- 172.21.15.29:0/4068522081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f665c06a2e0 0x7f665c15dfb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 >> 172.21.15.29:0/4068522081 conn(0x7f665c0ce9b0 msgr2=0x7f665c0ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 shutdown_connections 2023-10-16T14:17:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:21.915+0000 7f6661561640 1 -- 172.21.15.29:0/4068522081 wait complete. 2023-10-16T14:17:21.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:22.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.171+0000 7fd58cff0640 1 Processor -- start 2023-10-16T14:17:22.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.171+0000 7fd58cff0640 1 -- start start 2023-10-16T14:17:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd58cff0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd58815cd70 0x7fd5880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd58cff0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880b8480 0x7fd5880cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd58cff0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd5880d0be0 0x7fd5880cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd58cff0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58815d150 con 0x7fd58815cd70 2023-10-16T14:17:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd58cff0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5880d1180 con 0x7fd5880d0be0 2023-10-16T14:17:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd58cff0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880d1460 con 0x7fd5880b8480 2023-10-16T14:17:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd5877fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd58815cd70 0x7fd5880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd5877fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd58815cd70 0x7fd5880d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42876/0 (socket says 172.21.15.29:42876) 2023-10-16T14:17:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.172+0000 7fd5877fe640 1 -- 172.21.15.29:0/4029290982 learned_addr learned my addr 172.21.15.29:0/4029290982 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd587fff640 1 --2- 172.21.15.29:0/4029290982 >> v2:172.21.15.170:3300/0 conn(0x7fd5880d0be0 0x7fd5880cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd586ffd640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880b8480 0x7fd5880cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd5877fe640 1 -- 172.21.15.29:0/4029290982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880b8480 msgr2=0x7fd5880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd5877fe640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880b8480 0x7fd5880cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd5877fe640 1 -- 172.21.15.29:0/4029290982 >> v2:172.21.15.170:3300/0 conn(0x7fd5880d0be0 msgr2=0x7fd5880cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd5877fe640 1 --2- 172.21.15.29:0/4029290982 >> v2:172.21.15.170:3300/0 conn(0x7fd5880d0be0 0x7fd5880cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd5877fe640 1 -- 172.21.15.29:0/4029290982 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880d1740 con 0x7fd58815cd70 2023-10-16T14:17:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd587fff640 1 --2- 172.21.15.29:0/4029290982 >> v2:172.21.15.170:3300/0 conn(0x7fd5880d0be0 0x7fd5880cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.173+0000 7fd586ffd640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880b8480 0x7fd5880cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.174+0000 7fd5877fe640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880d2220 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5880b2520 tx=0x7fd57802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af791d2db2f6ce5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.174+0000 7fd5867fc640 1 -- 172.21.15.29:0/4029290982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57802ca80 con 0x7fd58815cd70 2023-10-16T14:17:22.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.174+0000 7fd5867fc640 1 -- 172.21.15.29:0/4029290982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57802cbe0 con 0x7fd58815cd70 2023-10-16T14:17:22.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.174+0000 7fd5867fc640 1 -- 172.21.15.29:0/4029290982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5780343f0 con 0x7fd58815cd70 2023-10-16T14:17:22.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 -- 172.21.15.29:0/4029290982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 msgr2=0x7fd5880d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880d2220 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5880b2520 tx=0x7fd57802bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 -- 172.21.15.29:0/4029290982 shutdown_connections 2023-10-16T14:17:22.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 --2- 172.21.15.29:0/4029290982 >> v2:172.21.15.170:3300/0 conn(0x7fd5880d0be0 0x7fd5880cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880b8480 0x7fd5880cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 --2- 172.21.15.29:0/4029290982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880d2220 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 -- 172.21.15.29:0/4029290982 >> 172.21.15.29:0/4029290982 conn(0x7fd5880cf570 msgr2=0x7fd5880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:22.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 -- 172.21.15.29:0/4029290982 shutdown_connections 2023-10-16T14:17:22.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.175+0000 7fd58cff0640 1 -- 172.21.15.29:0/4029290982 wait complete. 2023-10-16T14:17:22.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.177+0000 7fd58cff0640 1 Processor -- start 2023-10-16T14:17:22.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.177+0000 7fd58cff0640 1 -- start start 2023-10-16T14:17:22.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd58cff0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5880b8480 0x7fd5880c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd58cff0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880d0be0 0x7fd588150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd58cff0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd58cff0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880ccca0 con 0x7fd58815cd70 2023-10-16T14:17:22.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd58cff0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880c88a0 con 0x7fd5880b8480 2023-10-16T14:17:22.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd58cff0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880c8b80 con 0x7fd5880d0be0 2023-10-16T14:17:22.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd587fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd587fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42886/0 (socket says 172.21.15.29:42886) 2023-10-16T14:17:22.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.178+0000 7fd587fff640 1 -- 172.21.15.29:0/3397247562 learned_addr learned my addr 172.21.15.29:0/3397247562 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:22.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd5877fe640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5880b8480 0x7fd5880c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd586ffd640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880d0be0 0x7fd588150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd587fff640 1 -- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880d0be0 msgr2=0x7fd588150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd587fff640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880d0be0 0x7fd588150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd587fff640 1 -- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5880b8480 msgr2=0x7fd5880c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd587fff640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5880b8480 0x7fd5880c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd587fff640 1 -- 172.21.15.29:0/3397247562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578009b30 con 0x7fd58815cd70 2023-10-16T14:17:22.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd587fff640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880c6d80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd58000c590 tx=0x7fd58000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:22.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.179+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580004440 con 0x7fd58815cd70 2023-10-16T14:17:22.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.180+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd580003610 con 0x7fd58815cd70 2023-10-16T14:17:22.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.180+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5800039c0 con 0x7fd58815cd70 2023-10-16T14:17:22.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.180+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5880c6580 con 0x7fd58815cd70 2023-10-16T14:17:22.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.180+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5881514b0 con 0x7fd58815cd70 2023-10-16T14:17:22.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.181+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58804e9b0 con 0x7fd58815cd70 2023-10-16T14:17:22.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.182+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd580003b20 con 0x7fd58815cd70 2023-10-16T14:17:22.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.183+0000 7fd57ffff640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd56806ae70 0x7fd56806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.183+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fd580089820 con 0x7fd58815cd70 2023-10-16T14:17:22.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.183+0000 7fd5877fe640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd56806ae70 0x7fd56806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.185+0000 7fd5877fe640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd56806ae70 0x7fd56806d330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd578035000 tx=0x7fd578030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:22.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.187+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd580054810 con 0x7fd58815cd70 2023-10-16T14:17:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.380+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5880b7180 con 0x7fd58815cd70 2023-10-16T14:17:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.381+0000 7fd57ffff640 1 -- 172.21.15.29:0/3397247562 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd580057e30 con 0x7fd58815cd70 2023-10-16T14:17:22.384 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd56806ae70 msgr2=0x7fd56806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd56806ae70 0x7fd56806d330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd578035000 tx=0x7fd578030000 comp rx=0 tx=0).stop 2023-10-16T14:17:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 msgr2=0x7fd5880c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880c6d80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd58000c590 tx=0x7fd58000ca50 comp rx=0 tx=0).stop 2023-10-16T14:17:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 shutdown_connections 2023-10-16T14:17:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd56806ae70 0x7fd56806d330 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd58815cd70 0x7fd5880c6d80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5880d0be0 0x7fd588150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 --2- 172.21.15.29:0/3397247562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5880b8480 0x7fd5880c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 >> 172.21.15.29:0/3397247562 conn(0x7fd5880cf570 msgr2=0x7fd5880b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 shutdown_connections 2023-10-16T14:17:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.383+0000 7fd58cff0640 1 -- 172.21.15.29:0/3397247562 wait complete. 2023-10-16T14:17:22.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:22.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.639+0000 7f3c2f083640 1 Processor -- start 2023-10-16T14:17:22.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.640+0000 7f3c2f083640 1 -- start start 2023-10-16T14:17:22.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.641+0000 7f3c2f083640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c280b8480 0x7f3c280cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.641+0000 7f3c2f083640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c280cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.641+0000 7f3c2f083640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2815d150 con 0x7f3c2815cd70 2023-10-16T14:17:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.641+0000 7f3c2f083640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c280d1180 con 0x7f3c280b8480 2023-10-16T14:17:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.641+0000 7f3c2f083640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280d1460 con 0x7f3c280d0be0 2023-10-16T14:17:22.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c280b8480 0x7f3c280cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c280b8480 0x7f3c280cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42890/0 (socket says 172.21.15.29:42890) 2023-10-16T14:17:22.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 -- 172.21.15.29:0/3913739627 learned_addr learned my addr 172.21.15.29:0/3913739627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:22.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2e882640 1 --2- 172.21.15.29:0/3913739627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c280cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 -- 172.21.15.29:0/3913739627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 msgr2=0x7f3c280cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:22.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 --2- 172.21.15.29:0/3913739627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c280cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 -- 172.21.15.29:0/3913739627 >> v1:172.21.15.170:6789/0 conn(0x7f3c2815cd70 legacy=0x7f3c280d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.642+0000 7f3c2d880640 1 -- 172.21.15.29:0/3913739627 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c280d1740 con 0x7f3c280b8480 2023-10-16T14:17:22.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.643+0000 7f3c2d880640 1 --2- 172.21.15.29:0/3913739627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c280cd7c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009870 tx=0x7f3c1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ffc3ee6b341f21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:22.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.644+0000 7f3c2d07f640 1 -- 172.21.15.29:0/3913739627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14030840 con 0x7f3c280b8480 2023-10-16T14:17:22.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.644+0000 7f3c2d07f640 1 -- 172.21.15.29:0/3913739627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c140309a0 con 0x7f3c280b8480 2023-10-16T14:17:22.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.644+0000 7f3c2d07f640 1 -- 172.21.15.29:0/3913739627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14030d30 con 0x7f3c280b8480 2023-10-16T14:17:22.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.644+0000 7f3c2f083640 1 -- 172.21.15.29:0/3913739627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 msgr2=0x7f3c280cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.644+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3913739627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c280cd7c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009870 tx=0x7f3c1402b980 comp rx=0 tx=0).stop 2023-10-16T14:17:22.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.645+0000 7f3c2f083640 1 -- 172.21.15.29:0/3913739627 shutdown_connections 2023-10-16T14:17:22.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.645+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3913739627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c280cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.645+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3913739627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c280cd7c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.645+0000 7f3c2f083640 1 -- 172.21.15.29:0/3913739627 >> 172.21.15.29:0/3913739627 conn(0x7f3c280cf570 msgr2=0x7f3c280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:22.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.645+0000 7f3c2f083640 1 -- 172.21.15.29:0/3913739627 shutdown_connections 2023-10-16T14:17:22.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.645+0000 7f3c2f083640 1 -- 172.21.15.29:0/3913739627 wait complete. 2023-10-16T14:17:22.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.646+0000 7f3c2f083640 1 Processor -- start 2023-10-16T14:17:22.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.647+0000 7f3c2f083640 1 -- start start 2023-10-16T14:17:22.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2f083640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c281494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2f083640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c281f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2f083640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c2815cd70 0x7f3c28158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2f083640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c281f59e0 con 0x7f3c280b8480 2023-10-16T14:17:22.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2f083640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28149a10 con 0x7f3c2815cd70 2023-10-16T14:17:22.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2f083640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c281f5bf0 con 0x7f3c280d0be0 2023-10-16T14:17:22.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c281494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c281494d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42904/0 (socket says 172.21.15.29:42904) 2023-10-16T14:17:22.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 -- 172.21.15.29:0/3344849468 learned_addr learned my addr 172.21.15.29:0/3344849468 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:22.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 -- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 msgr2=0x7f3c281f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c281f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 -- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c2815cd70 msgr2=0x7f3c28158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c2815cd70 0x7f3c28158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.648+0000 7f3c2e081640 1 -- 172.21.15.29:0/3344849468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c140094d0 con 0x7f3c280b8480 2023-10-16T14:17:22.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.649+0000 7f3c2e081640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c281494d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c730 tx=0x7f3c1c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:22.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.649+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c003b70 con 0x7f3c280b8480 2023-10-16T14:17:22.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.649+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c003cf0 con 0x7f3c280b8480 2023-10-16T14:17:22.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.649+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c003ea0 con 0x7f3c280b8480 2023-10-16T14:17:22.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.649+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c281f5ed0 con 0x7f3c280b8480 2023-10-16T14:17:22.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.649+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c281f6490 con 0x7f3c280b8480 2023-10-16T14:17:22.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.650+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2804e9b0 con 0x7f3c280b8480 2023-10-16T14:17:22.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.651+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c1c00aa90 con 0x7f3c280b8480 2023-10-16T14:17:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.652+0000 7f3c26ffd640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3c0c06ae70 0x7f3c0c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.652+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f3c1c089590 con 0x7f3c280b8480 2023-10-16T14:17:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.653+0000 7f3c2d880640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3c0c06ae70 0x7f3c0c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:22.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.654+0000 7f3c2d880640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3c0c06ae70 0x7f3c0c06d330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c14036000 tx=0x7f3c1403b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.656+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c1c0544f0 con 0x7f3c280b8480 2023-10-16T14:17:22.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.847+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c280b6700 con 0x7f3c280b8480 2023-10-16T14:17:22.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.848+0000 7f3c26ffd640 1 -- 172.21.15.29:0/3344849468 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c1c057b10 con 0x7f3c280b8480 2023-10-16T14:17:22.851 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3c0c06ae70 msgr2=0x7f3c0c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3c0c06ae70 0x7f3c0c06d330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c14036000 tx=0x7f3c1403b040 comp rx=0 tx=0).stop 2023-10-16T14:17:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 msgr2=0x7f3c281494d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c281494d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c730 tx=0x7f3c1c00cbf0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 shutdown_connections 2023-10-16T14:17:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3c0c06ae70 0x7f3c0c06d330 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c2815cd70 0x7f3c28158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c280d0be0 0x7f3c281f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 --2- 172.21.15.29:0/3344849468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c280b8480 0x7f3c281494d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:22.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.850+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 >> 172.21.15.29:0/3344849468 conn(0x7f3c280cf570 msgr2=0x7f3c280b7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.851+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 shutdown_connections 2023-10-16T14:17:22.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:22.851+0000 7f3c2f083640 1 -- 172.21.15.29:0/3344849468 wait complete. 2023-10-16T14:17:22.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:17:23.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.099+0000 7fd757826640 1 Processor -- start 2023-10-16T14:17:23.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.099+0000 7fd757826640 1 -- start start 2023-10-16T14:17:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.101+0000 7fd757826640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd75015d150 0x7fd7500ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.101+0000 7fd757826640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7500ac890 con 0x7fd75006a4b0 2023-10-16T14:17:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.101+0000 7fd757826640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd750161650 con 0x7fd750160b90 2023-10-16T14:17:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.101+0000 7fd757826640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75015d530 con 0x7fd75015d150 2023-10-16T14:17:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.101+0000 7fd7555c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd75015d150 0x7fd7500ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.101+0000 7fd7555c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd75015d150 0x7fd7500ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42916/0 (socket says 172.21.15.29:42916) 2023-10-16T14:17:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.102+0000 7fd7555c2640 1 -- 172.21.15.29:0/546696256 learned_addr learned my addr 172.21.15.29:0/546696256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.102+0000 7fd7555c2640 1 -- 172.21.15.29:0/546696256 >> v1:172.21.15.170:6789/0 conn(0x7fd750160b90 legacy=0x7fd750069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.102+0000 7fd7555c2640 1 -- 172.21.15.29:0/546696256 >> v1:172.21.15.29:6789/0 conn(0x7fd75006a4b0 legacy=0x7fd7500c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.102+0000 7fd747fff640 1 -- 172.21.15.29:0/546696256 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828802777 0 0) 0x7fd7500ac890 con 0x7fd75006a4b0 2023-10-16T14:17:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.102+0000 7fd7555c2640 1 -- 172.21.15.29:0/546696256 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750161930 con 0x7fd75015d150 2023-10-16T14:17:23.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.104+0000 7fd7555c2640 1 --2- 172.21.15.29:0/546696256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75015d150 0x7fd7500ad350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009870 tx=0x7fd74c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d6c41c1870c222 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:23.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd747fff640 1 -- 172.21.15.29:0/546696256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c02f840 con 0x7fd75015d150 2023-10-16T14:17:23.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd747fff640 1 -- 172.21.15.29:0/546696256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c02f9a0 con 0x7fd75015d150 2023-10-16T14:17:23.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd747fff640 1 -- 172.21.15.29:0/546696256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c02fd30 con 0x7fd75015d150 2023-10-16T14:17:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 -- 172.21.15.29:0/546696256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75015d150 msgr2=0x7fd7500ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 --2- 172.21.15.29:0/546696256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75015d150 0x7fd7500ad350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009870 tx=0x7fd74c02ba20 comp rx=0 tx=0).stop 2023-10-16T14:17:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 -- 172.21.15.29:0/546696256 shutdown_connections 2023-10-16T14:17:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 --2- 172.21.15.29:0/546696256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75015d150 0x7fd7500ad350 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 -- 172.21.15.29:0/546696256 >> 172.21.15.29:0/546696256 conn(0x7fd7500ce9b0 msgr2=0x7fd7500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 -- 172.21.15.29:0/546696256 shutdown_connections 2023-10-16T14:17:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.105+0000 7fd757826640 1 -- 172.21.15.29:0/546696256 wait complete. 2023-10-16T14:17:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.107+0000 7fd757826640 1 Processor -- start 2023-10-16T14:17:23.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.107+0000 7fd757826640 1 -- start start 2023-10-16T14:17:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.108+0000 7fd757826640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 0x7fd7501f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.108+0000 7fd757826640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd75015d150 0x7fd7500c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.108+0000 7fd757826640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd750160b90 0x7fd7500c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.108+0000 7fd757826640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd750156770 con 0x7fd75006a4b0 2023-10-16T14:17:23.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.108+0000 7fd757826640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7501f6ab0 con 0x7fd750160b90 2023-10-16T14:17:23.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.108+0000 7fd757826640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7501f5ea0 con 0x7fd75015d150 2023-10-16T14:17:23.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 0x7fd7501f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 0x7fd7501f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42928/0 (socket says 172.21.15.29:42928) 2023-10-16T14:17:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 -- 172.21.15.29:0/791264214 learned_addr learned my addr 172.21.15.29:0/791264214 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd754dc1640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd75015d150 0x7fd7500c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd755dc3640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd750160b90 0x7fd7500c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 -- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd75015d150 msgr2=0x7fd7500c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd75015d150 0x7fd7500c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 -- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd750160b90 msgr2=0x7fd7500c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd750160b90 0x7fd7500c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.109+0000 7fd7555c2640 1 -- 172.21.15.29:0/791264214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0094d0 con 0x7fd75006a4b0 2023-10-16T14:17:23.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.110+0000 7fd7555c2640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 0x7fd7501f3450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd74c000c00 tx=0x7fd74c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:23.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.110+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c03b040 con 0x7fd75006a4b0 2023-10-16T14:17:23.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.110+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c037d50 con 0x7fd75006a4b0 2023-10-16T14:17:23.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.110+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7501f6180 con 0x7fd75006a4b0 2023-10-16T14:17:23.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.110+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c004270 con 0x7fd75006a4b0 2023-10-16T14:17:23.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.110+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7501f6740 con 0x7fd75006a4b0 2023-10-16T14:17:23.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.111+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75004e160 con 0x7fd75006a4b0 2023-10-16T14:17:23.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.112+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd74c0043d0 con 0x7fd75006a4b0 2023-10-16T14:17:23.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.113+0000 7fd745ffb640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd73406ae70 0x7fd73406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.113+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fd74c0ac750 con 0x7fd75006a4b0 2023-10-16T14:17:23.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.114+0000 7fd754dc1640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd73406ae70 0x7fd73406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.115+0000 7fd754dc1640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd73406ae70 0x7fd73406d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd7500cd2d0 tx=0x7fd740009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.118+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd74c0776b0 con 0x7fd75006a4b0 2023-10-16T14:17:23.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.327+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd7500b1300 con 0x7fd75006a4b0 2023-10-16T14:17:23.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.328+0000 7fd745ffb640 1 -- 172.21.15.29:0/791264214 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd74c07acd0 con 0x7fd75006a4b0 2023-10-16T14:17:23.331 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-16T14:17:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd73406ae70 msgr2=0x7fd73406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd73406ae70 0x7fd73406d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd7500cd2d0 tx=0x7fd740009040 comp rx=0 tx=0).stop 2023-10-16T14:17:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 msgr2=0x7fd7501f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 0x7fd7501f3450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd74c000c00 tx=0x7fd74c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 shutdown_connections 2023-10-16T14:17:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd73406ae70 0x7fd73406d330 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd750160b90 0x7fd7500c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd75015d150 0x7fd7500c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 --2- 172.21.15.29:0/791264214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd75006a4b0 0x7fd7501f3450 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.330+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 >> 172.21.15.29:0/791264214 conn(0x7fd7500ce9b0 msgr2=0x7fd7500c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.331+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 shutdown_connections 2023-10-16T14:17:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.331+0000 7fd757826640 1 -- 172.21.15.29:0/791264214 wait complete. 2023-10-16T14:17:23.353 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-16T14:17:23.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:23.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-16T14:17:23.464 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T14:17:23.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:23.465 DEBUG:teuthology.orchestra.run.smithi170:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-16T14:17:23.518 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:17:23.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:17:23.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.752+0000 7f3d84716640 1 Processor -- start 2023-10-16T14:17:23.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.753+0000 7f3d84716640 1 -- start start 2023-10-16T14:17:23.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.754+0000 7f3d84716640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.754+0000 7f3d84716640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d7c0ce390 0x7f3d7c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.754+0000 7f3d84716640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c15ed60 con 0x7f3d7c15ae80 2023-10-16T14:17:23.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.754+0000 7f3d84716640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7c06b810 con 0x7f3d7c0ce390 2023-10-16T14:17:23.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.754+0000 7f3d84716640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0707e0 con 0x7f3d7c15aaa0 2023-10-16T14:17:23.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d7c0ce390 0x7f3d7c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d7c0ce390 0x7f3d7c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42930/0 (socket says 172.21.15.29:42930) 2023-10-16T14:17:23.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 -- 172.21.15.29:0/3323359719 learned_addr learned my addr 172.21.15.29:0/3323359719 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:23.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d824b2640 1 --2- 172.21.15.29:0/3323359719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 -- 172.21.15.29:0/3323359719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 msgr2=0x7f3d7c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:23.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 --2- 172.21.15.29:0/3323359719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 -- 172.21.15.29:0/3323359719 >> v1:172.21.15.170:6789/0 conn(0x7f3d7c15ae80 legacy=0x7f3d7c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:23.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.755+0000 7f3d82cb3640 1 -- 172.21.15.29:0/3323359719 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c15ef90 con 0x7f3d7c0ce390 2023-10-16T14:17:23.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.757+0000 7f3d82cb3640 1 --2- 172.21.15.29:0/3323359719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c070f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009e50 tx=0x7f3d7402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd86efdfd5c4ccf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:23.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.757+0000 7f3d814b0640 1 -- 172.21.15.29:0/3323359719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7402ca70 con 0x7f3d7c0ce390 2023-10-16T14:17:23.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.757+0000 7f3d814b0640 1 -- 172.21.15.29:0/3323359719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7402cbd0 con 0x7f3d7c0ce390 2023-10-16T14:17:23.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.757+0000 7f3d814b0640 1 -- 172.21.15.29:0/3323359719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d740343f0 con 0x7f3d7c0ce390 2023-10-16T14:17:23.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 -- 172.21.15.29:0/3323359719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c0ce390 msgr2=0x7f3d7c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 --2- 172.21.15.29:0/3323359719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c070f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009e50 tx=0x7f3d7402bed0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 -- 172.21.15.29:0/3323359719 shutdown_connections 2023-10-16T14:17:23.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 --2- 172.21.15.29:0/3323359719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c070f50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 --2- 172.21.15.29:0/3323359719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 -- 172.21.15.29:0/3323359719 >> 172.21.15.29:0/3323359719 conn(0x7f3d7c0cb410 msgr2=0x7f3d7c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:23.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 -- 172.21.15.29:0/3323359719 shutdown_connections 2023-10-16T14:17:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.758+0000 7f3d84716640 1 -- 172.21.15.29:0/3323359719 wait complete. 2023-10-16T14:17:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.760+0000 7f3d84716640 1 Processor -- start 2023-10-16T14:17:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.760+0000 7f3d84716640 1 -- start start 2023-10-16T14:17:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d84716640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d84716640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d84716640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 0x7f3d7c14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d84716640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0c4cb0 con 0x7f3d7c15ae80 2023-10-16T14:17:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d84716640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c15b860 con 0x7f3d7c0ce390 2023-10-16T14:17:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d84716640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0c4f00 con 0x7f3d7c15aaa0 2023-10-16T14:17:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d82cb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 0x7f3d7c14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d82cb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 0x7f3d7c14eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42934/0 (socket says 172.21.15.29:42934) 2023-10-16T14:17:23.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.761+0000 7f3d82cb3640 1 -- 172.21.15.29:0/335812955 learned_addr learned my addr 172.21.15.29:0/335812955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:23.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d824b2640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d81cb1640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d82cb3640 1 -- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 msgr2=0x7f3d7c15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d82cb3640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d82cb3640 1 -- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d7c0ce390 msgr2=0x7f3d7c15be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:23.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d82cb3640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c15be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:23.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d82cb3640 1 -- 172.21.15.29:0/335812955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74009b30 con 0x7f3d7c15ae80 2023-10-16T14:17:23.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.762+0000 7f3d82cb3640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 0x7f3d7c14eea0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d74036040 tx=0x7f3d74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:23.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.763+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74009510 con 0x7f3d7c15ae80 2023-10-16T14:17:23.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.763+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7403e450 con 0x7f3d7c15ae80 2023-10-16T14:17:23.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.763+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7403e7e0 con 0x7f3d7c15ae80 2023-10-16T14:17:23.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.763+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c0c51e0 con 0x7f3d7c15ae80 2023-10-16T14:17:23.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.763+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c1f0870 con 0x7f3d7c15ae80 2023-10-16T14:17:23.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.764+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c1f0e00 con 0x7f3d7c15ae80 2023-10-16T14:17:23.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.765+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3d7403e940 con 0x7f3d7c15ae80 2023-10-16T14:17:23.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.766+0000 7f3d72ffd640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d6006ae70 0x7f3d6006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:23.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.767+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f3d740ac480 con 0x7f3d7c15ae80 2023-10-16T14:17:23.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.767+0000 7f3d824b2640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d6006ae70 0x7f3d6006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:23.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.768+0000 7f3d824b2640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d6006ae70 0x7f3d6006d330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d78006fd0 tx=0x7f3d78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:23.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:23.771+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d7c1f0e00 con 0x7f3d7c15ae80 2023-10-16T14:17:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.003+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d7c1f0e00 con 0x7f3d7c15ae80 2023-10-16T14:17:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.004+0000 7f3d72ffd640 1 -- 172.21.15.29:0/335812955 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f3d7c1f0e00 con 0x7f3d7c15ae80 2023-10-16T14:17:24.007 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:24.007 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:17:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:17:24.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d6006ae70 msgr2=0x7f3d6006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d6006ae70 0x7f3d6006d330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d78006fd0 tx=0x7f3d78008040 comp rx=0 tx=0).stop 2023-10-16T14:17:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 msgr2=0x7f3d7c14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 0x7f3d7c14eea0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d74036040 tx=0x7f3d74004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 shutdown_connections 2023-10-16T14:17:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d6006ae70 0x7f3d6006d330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c15ae80 0x7f3d7c14eea0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d7c15aaa0 0x7f3d7c15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 --2- 172.21.15.29:0/335812955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d7c0ce390 0x7f3d7c15be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 >> 172.21.15.29:0/335812955 conn(0x7f3d7c0cb410 msgr2=0x7f3d7c06f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 shutdown_connections 2023-10-16T14:17:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.007+0000 7f3d84716640 1 -- 172.21.15.29:0/335812955 wait complete. 2023-10-16T14:17:24.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.254+0000 7f1c64e44640 1 Processor -- start 2023-10-16T14:17:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.255+0000 7f1c64e44640 1 -- start start 2023-10-16T14:17:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.256+0000 7f1c64e44640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 0x7f1c6015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.256+0000 7f1c64e44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6015ed60 con 0x7f1c600ce390 2023-10-16T14:17:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.256+0000 7f1c64e44640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600707e0 con 0x7f1c6015ae80 2023-10-16T14:17:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.256+0000 7f1c64e44640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6006b810 con 0x7f1c6015aaa0 2023-10-16T14:17:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.257+0000 7f1c5ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1c600ce390 0x7f1c60070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47274/0 (socket says 172.21.15.29:47274) 2023-10-16T14:17:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.257+0000 7f1c5ed9d640 1 -- 172.21.15.29:0/3974502578 learned_addr learned my addr 172.21.15.29:0/3974502578 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.257+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 0x7f1c6015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.257+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269411731 0 0) 0x7f1c6015ed60 con 0x7f1c600ce390 2023-10-16T14:17:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.257+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c003520 con 0x7f1c600ce390 2023-10-16T14:17:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315515938 0 0) 0x7f1c4c003520 con 0x7f1c600ce390 2023-10-16T14:17:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 msgr2=0x7f1c6015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 0x7f1c6015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 >> v1:172.21.15.170:6789/0 conn(0x7f1c6015ae80 legacy=0x7f1c600cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6015ef90 con 0x7f1c600ce390 2023-10-16T14:17:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1c58001670 con 0x7f1c600ce390 2023-10-16T14:17:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 >> v1:172.21.15.29:6789/0 conn(0x7f1c600ce390 legacy=0x7f1c60070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 0x7f1c4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6006b810 con 0x7f1c4c0039a0 2023-10-16T14:17:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c00ba60 con 0x7f1c4c0047c0 2023-10-16T14:17:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c00bc20 con 0x7f1c6015aaa0 2023-10-16T14:17:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5dd9b640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.258+0000 7f1c5ed9d640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 0x7f1c4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.259+0000 7f1c5e59c640 1 -- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 msgr2=0x7f1c4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.259+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015aaa0 0x7f1c4c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.259+0000 7f1c5e59c640 1 -- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0047c0 msgr2=0x7f1c4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.259+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.259+0000 7f1c5e59c640 1 -- 172.21.15.29:0/3974502578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c003520 con 0x7f1c4c0039a0 2023-10-16T14:17:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.260+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009f90 tx=0x7f1c4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11faafd57a71684f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.260+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5002cbd0 con 0x7f1c4c0039a0 2023-10-16T14:17:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.260+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5000f040 con 0x7f1c4c0039a0 2023-10-16T14:17:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.260+0000 7f1c5d59a640 1 -- 172.21.15.29:0/3974502578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50034440 con 0x7f1c4c0039a0 2023-10-16T14:17:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c64e44640 1 -- 172.21.15.29:0/3974502578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0039a0 msgr2=0x7f1c4c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c64e44640 1 --2- 172.21.15.29:0/3974502578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009f90 tx=0x7f1c4c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c5ed9d640 1 -- 172.21.15.29:0/3974502578 reap_dead start 2023-10-16T14:17:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c64e44640 1 -- 172.21.15.29:0/3974502578 shutdown_connections 2023-10-16T14:17:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c64e44640 1 -- 172.21.15.29:0/3974502578 >> 172.21.15.29:0/3974502578 conn(0x7f1c600cb410 msgr2=0x7f1c600cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c64e44640 1 -- 172.21.15.29:0/3974502578 shutdown_connections 2023-10-16T14:17:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.261+0000 7f1c64e44640 1 -- 172.21.15.29:0/3974502578 wait complete. 2023-10-16T14:17:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.263+0000 7f1c64e44640 1 Processor -- start 2023-10-16T14:17:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.263+0000 7f1c64e44640 1 -- start start 2023-10-16T14:17:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.264+0000 7f1c64e44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0039a0 0x7f1c6006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.264+0000 7f1c64e44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 0x7f1c601f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.264+0000 7f1c64e44640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015dc60 0x7f1c601f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.264+0000 7f1c64e44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6015c700 con 0x7f1c600c4c70 2023-10-16T14:17:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.264+0000 7f1c64e44640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c600c3bd0 con 0x7f1c4c0039a0 2023-10-16T14:17:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.264+0000 7f1c64e44640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c600c3eb0 con 0x7f1c6015dc60 2023-10-16T14:17:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 0x7f1c601f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 0x7f1c601f1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47256/0 (socket says 172.21.15.29:47256) 2023-10-16T14:17:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 -- 172.21.15.29:0/4182044699 learned_addr learned my addr 172.21.15.29:0/4182044699 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0039a0 0x7f1c6006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5ed9d640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015dc60 0x7f1c601f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 -- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015dc60 msgr2=0x7f1c601f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015dc60 0x7f1c601f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 -- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0039a0 msgr2=0x7f1c6006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0039a0 0x7f1c6006cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 -- 172.21.15.29:0/4182044699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c600c4190 con 0x7f1c600c4c70 2023-10-16T14:17:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0039a0 0x7f1c6006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.265+0000 7f1c5dd9b640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 0x7f1c601f1dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800d4c0 tx=0x7f1c4800d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.266+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48005480 con 0x7f1c600c4c70 2023-10-16T14:17:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.266+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c48003610 con 0x7f1c600c4c70 2023-10-16T14:17:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.266+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c480039e0 con 0x7f1c600c4c70 2023-10-16T14:17:24.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.266+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c600c4470 con 0x7f1c600c4c70 2023-10-16T14:17:24.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.266+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6006d430 con 0x7f1c600c4c70 2023-10-16T14:17:24.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.267+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6004e160 con 0x7f1c600c4c70 2023-10-16T14:17:24.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.268+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c48003b40 con 0x7f1c600c4c70 2023-10-16T14:17:24.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.269+0000 7f1c56ffd640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1c4006ae70 0x7f1c4006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.269+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f1c48089600 con 0x7f1c600c4c70 2023-10-16T14:17:24.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.270+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1c4006ae70 0x7f1c4006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.271+0000 7f1c5e59c640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1c4006ae70 0x7f1c4006d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c50036000 tx=0x7f1c5002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:24.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.274+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c48054560 con 0x7f1c600c4c70 2023-10-16T14:17:24.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.473+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c600cdc60 con 0x7f1c600c4c70 2023-10-16T14:17:24.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.476+0000 7f1c56ffd640 1 -- 172.21.15.29:0/4182044699 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f1c48057b80 con 0x7f1c600c4c70 2023-10-16T14:17:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:24.481 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:24.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.479+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1c4006ae70 msgr2=0x7f1c4006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.479+0000 7f1c64e44640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1c4006ae70 0x7f1c4006d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c50036000 tx=0x7f1c5002e000 comp rx=0 tx=0).stop 2023-10-16T14:17:24.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.479+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 msgr2=0x7f1c601f1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.479+0000 7f1c64e44640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 0x7f1c601f1dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800d4c0 tx=0x7f1c4800d980 comp rx=0 tx=0).stop 2023-10-16T14:17:24.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 shutdown_connections 2023-10-16T14:17:24.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1c4006ae70 0x7f1c4006d330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c600c4c70 0x7f1c601f1dd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c6015dc60 0x7f1c601f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 --2- 172.21.15.29:0/4182044699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c4c0039a0 0x7f1c6006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 >> 172.21.15.29:0/4182044699 conn(0x7f1c600cb410 msgr2=0x7f1c601568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 shutdown_connections 2023-10-16T14:17:24.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.480+0000 7f1c64e44640 1 -- 172.21.15.29:0/4182044699 wait complete. 2023-10-16T14:17:24.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:16:27.863320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:16:28.506011+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:24.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:24.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.734+0000 7f69ffdd0640 1 Processor -- start 2023-10-16T14:17:24.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.734+0000 7f69ffdd0640 1 -- start start 2023-10-16T14:17:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.735+0000 7f69ffdd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f69f815d330 0x7f69f81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.735+0000 7f69ffdd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69f806c7d0 0x7f69f8065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69ffdd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f80d0890 con 0x7f69f806cc80 2023-10-16T14:17:24.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69ffdd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f806b1e0 con 0x7f69f806c7d0 2023-10-16T14:17:24.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69ffdd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f80d05b0 con 0x7f69f815d330 2023-10-16T14:17:24.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69fe36d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f69f806cc80 0x7f69f80ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47280/0 (socket says 172.21.15.29:47280) 2023-10-16T14:17:24.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69fe36d640 1 -- 172.21.15.29:0/2342341270 learned_addr learned my addr 172.21.15.29:0/2342341270 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69fd36b640 1 --2- 172.21.15.29:0/2342341270 >> v2:172.21.15.29:3300/0 conn(0x7f69f806c7d0 0x7f69f8065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.736+0000 7f69fdb6c640 1 --2- 172.21.15.29:0/2342341270 >> v2:172.21.15.170:3300/0 conn(0x7f69f815d330 0x7f69f81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.737+0000 7f69fd36b640 1 -- 172.21.15.29:0/2342341270 >> v2:172.21.15.170:3300/0 conn(0x7f69f815d330 msgr2=0x7f69f81618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:24.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.737+0000 7f69fd36b640 1 --2- 172.21.15.29:0/2342341270 >> v2:172.21.15.170:3300/0 conn(0x7f69f815d330 0x7f69f81618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.737+0000 7f69fd36b640 1 -- 172.21.15.29:0/2342341270 >> v1:172.21.15.29:6789/0 conn(0x7f69f806cc80 legacy=0x7f69f80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.737+0000 7f69fd36b640 1 -- 172.21.15.29:0/2342341270 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8160dd0 con 0x7f69f806c7d0 2023-10-16T14:17:24.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.737+0000 7f69fcb6a640 1 -- 172.21.15.29:0/2342341270 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799751418 0 0) 0x7f69f80d0890 con 0x7f69f806cc80 2023-10-16T14:17:24.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.738+0000 7f69fd36b640 1 --2- 172.21.15.29:0/2342341270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806c7d0 0x7f69f8065ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f69f80744f0 tx=0x7f69e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2530586cfd39fff4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:24.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.738+0000 7f69fcb6a640 1 -- 172.21.15.29:0/2342341270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e802e070 con 0x7f69f806c7d0 2023-10-16T14:17:24.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.738+0000 7f69fcb6a640 1 -- 172.21.15.29:0/2342341270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e80315a0 con 0x7f69f806c7d0 2023-10-16T14:17:24.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.738+0000 7f69fcb6a640 1 -- 172.21.15.29:0/2342341270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8031930 con 0x7f69f806c7d0 2023-10-16T14:17:24.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 -- 172.21.15.29:0/2342341270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806c7d0 msgr2=0x7f69f8065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/2342341270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806c7d0 0x7f69f8065ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f69f80744f0 tx=0x7f69e802b840 comp rx=0 tx=0).stop 2023-10-16T14:17:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 -- 172.21.15.29:0/2342341270 shutdown_connections 2023-10-16T14:17:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/2342341270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806c7d0 0x7f69f8065ab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/2342341270 >> v2:172.21.15.170:3300/0 conn(0x7f69f815d330 0x7f69f81618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 -- 172.21.15.29:0/2342341270 >> 172.21.15.29:0/2342341270 conn(0x7f69f80d0ab0 msgr2=0x7f69f80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 -- 172.21.15.29:0/2342341270 shutdown_connections 2023-10-16T14:17:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.739+0000 7f69ffdd0640 1 -- 172.21.15.29:0/2342341270 wait complete. 2023-10-16T14:17:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.741+0000 7f69ffdd0640 1 Processor -- start 2023-10-16T14:17:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.741+0000 7f69ffdd0640 1 -- start start 2023-10-16T14:17:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69ffdd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f806c7d0 0x7f69f81f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69ffdd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 0x7f69f80aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69ffdd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69f815d330 0x7f69f80af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69ffdd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f81f3700 con 0x7f69f806cc80 2023-10-16T14:17:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69ffdd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80b11e0 con 0x7f69f806c7d0 2023-10-16T14:17:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69ffdd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f81f2c30 con 0x7f69f815d330 2023-10-16T14:17:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.742+0000 7f69fd36b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 0x7f69f80aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 0x7f69f80aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47264/0 (socket says 172.21.15.29:47264) 2023-10-16T14:17:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 -- 172.21.15.29:0/3168486251 learned_addr learned my addr 172.21.15.29:0/3168486251 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fdb6c640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f806c7d0 0x7f69f81f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fe36d640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69f815d330 0x7f69f80af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 -- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69f815d330 msgr2=0x7f69f80af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69f815d330 0x7f69f80af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 -- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f806c7d0 msgr2=0x7f69f81f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f806c7d0 0x7f69f81f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 -- 172.21.15.29:0/3168486251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e8009400 con 0x7f69f806cc80 2023-10-16T14:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.743+0000 7f69fd36b640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 0x7f69f80aeb20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009890 tx=0x7f69e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.744+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8004230 con 0x7f69f806cc80 2023-10-16T14:17:24.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.744+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8031f00 con 0x7f69f806cc80 2023-10-16T14:17:24.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.744+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8032640 con 0x7f69f806cc80 2023-10-16T14:17:24.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.744+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f80af5a0 con 0x7f69f806cc80 2023-10-16T14:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.744+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f80c71d0 con 0x7f69f806cc80 2023-10-16T14:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.745+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f804e160 con 0x7f69f806cc80 2023-10-16T14:17:24.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.746+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f69e80327a0 con 0x7f69f806cc80 2023-10-16T14:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.747+0000 7f69ee7fc640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f69dc06ae70 0x7f69dc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.747+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f69e80ab730 con 0x7f69f806cc80 2023-10-16T14:17:24.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.748+0000 7f69fdb6c640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f69dc06ae70 0x7f69dc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.749+0000 7f69fdb6c640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f69dc06ae70 0x7f69dc06d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f69f4006fd0 tx=0x7f69f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:24.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.751+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69e8076720 con 0x7f69f806cc80 2023-10-16T14:17:24.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.945+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69f806f3a0 con 0x7f69f806cc80 2023-10-16T14:17:24.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.946+0000 7f69ee7fc640 1 -- 172.21.15.29:0/3168486251 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69e8079d40 con 0x7f69f806cc80 2023-10-16T14:17:24.949 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f69dc06ae70 msgr2=0x7f69dc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f69dc06ae70 0x7f69dc06d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f69f4006fd0 tx=0x7f69f4008040 comp rx=0 tx=0).stop 2023-10-16T14:17:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 msgr2=0x7f69f80aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 0x7f69f80aeb20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009890 tx=0x7f69e80046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 shutdown_connections 2023-10-16T14:17:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f69dc06ae70 0x7f69dc06d330 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f806cc80 0x7f69f80aeb20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69f815d330 0x7f69f80af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 --2- 172.21.15.29:0/3168486251 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69f806c7d0 0x7f69f81f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:24.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 >> 172.21.15.29:0/3168486251 conn(0x7f69f80d0ab0 msgr2=0x7f69f815f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:24.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 shutdown_connections 2023-10-16T14:17:24.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:24.948+0000 7f69ffdd0640 1 -- 172.21.15.29:0/3168486251 wait complete. 2023-10-16T14:17:24.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:17:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.187+0000 7fae4a50c640 1 Processor -- start 2023-10-16T14:17:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.188+0000 7fae4a50c640 1 -- start start 2023-10-16T14:17:25.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.189+0000 7fae4a50c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae4415aee0 0x7fae440aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.189+0000 7fae4a50c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae440cb810 0x7fae440cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.189+0000 7fae4a50c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4406c140 con 0x7fae44069da0 2023-10-16T14:17:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.189+0000 7fae4a50c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4415b2c0 con 0x7fae440cb810 2023-10-16T14:17:25.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.189+0000 7fae4a50c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4406b270 con 0x7fae4415aee0 2023-10-16T14:17:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.190+0000 7fae48aa9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fae44069da0 0x7fae4406a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47294/0 (socket says 172.21.15.29:47294) 2023-10-16T14:17:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.190+0000 7fae437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae440cb810 0x7fae440cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.190+0000 7fae48aa9640 1 -- 172.21.15.29:0/3729818986 learned_addr learned my addr 172.21.15.29:0/3729818986 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.190+0000 7fae43fff640 1 --2- 172.21.15.29:0/3729818986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae4415aee0 0x7fae440aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.190+0000 7fae42ffd640 1 -- 172.21.15.29:0/3729818986 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860646398 0 0) 0x7fae4406c140 con 0x7fae44069da0 2023-10-16T14:17:25.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.190+0000 7fae42ffd640 1 -- 172.21.15.29:0/3729818986 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c003520 con 0x7fae44069da0 2023-10-16T14:17:25.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.191+0000 7fae437fe640 1 -- 172.21.15.29:0/3729818986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae4415aee0 msgr2=0x7fae440aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:25.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.191+0000 7fae437fe640 1 --2- 172.21.15.29:0/3729818986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae4415aee0 0x7fae440aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.191+0000 7fae437fe640 1 -- 172.21.15.29:0/3729818986 >> v1:172.21.15.29:6789/0 conn(0x7fae44069da0 legacy=0x7fae4406a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.191+0000 7fae437fe640 1 -- 172.21.15.29:0/3729818986 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4406daf0 con 0x7fae440cb810 2023-10-16T14:17:25.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.192+0000 7fae437fe640 1 --2- 172.21.15.29:0/3729818986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440cbc70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fae380097a0 tx=0x7fae3802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf250dc956c77db7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:25.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.192+0000 7fae42ffd640 1 -- 172.21.15.29:0/3729818986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae380316f0 con 0x7fae440cb810 2023-10-16T14:17:25.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.192+0000 7fae42ffd640 1 -- 172.21.15.29:0/3729818986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38031850 con 0x7fae440cb810 2023-10-16T14:17:25.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.192+0000 7fae42ffd640 1 -- 172.21.15.29:0/3729818986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38031be0 con 0x7fae440cb810 2023-10-16T14:17:25.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 -- 172.21.15.29:0/3729818986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 msgr2=0x7fae440cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3729818986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440cbc70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fae380097a0 tx=0x7fae3802b790 comp rx=0 tx=0).stop 2023-10-16T14:17:25.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 -- 172.21.15.29:0/3729818986 shutdown_connections 2023-10-16T14:17:25.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3729818986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440cbc70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3729818986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae4415aee0 0x7fae440aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 -- 172.21.15.29:0/3729818986 >> 172.21.15.29:0/3729818986 conn(0x7fae440cfac0 msgr2=0x7fae440cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:25.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 -- 172.21.15.29:0/3729818986 shutdown_connections 2023-10-16T14:17:25.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.193+0000 7fae4a50c640 1 -- 172.21.15.29:0/3729818986 wait complete. 2023-10-16T14:17:25.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.195+0000 7fae4a50c640 1 Processor -- start 2023-10-16T14:17:25.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.195+0000 7fae4a50c640 1 -- start start 2023-10-16T14:17:25.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae4a50c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae44069da0 0x7fae4406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae4a50c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae4a50c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4415aee0 0x7fae441f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae4a50c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4415b850 con 0x7fae440cb810 2023-10-16T14:17:25.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae4a50c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae441e5a40 con 0x7fae4415aee0 2023-10-16T14:17:25.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae4a50c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae441e5c90 con 0x7fae44069da0 2023-10-16T14:17:25.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47292/0 (socket says 172.21.15.29:47292) 2023-10-16T14:17:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae437fe640 1 -- 172.21.15.29:0/3437776758 learned_addr learned my addr 172.21.15.29:0/3437776758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae48aa9640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4415aee0 0x7fae441f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.196+0000 7fae43fff640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae44069da0 0x7fae4406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae437fe640 1 -- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae44069da0 msgr2=0x7fae4406a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae437fe640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae44069da0 0x7fae4406a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae437fe640 1 -- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4415aee0 msgr2=0x7fae441f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae437fe640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4415aee0 0x7fae441f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae437fe640 1 -- 172.21.15.29:0/3437776758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009400 con 0x7fae440cb810 2023-10-16T14:17:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae48aa9640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4415aee0 0x7fae441f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae437fe640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440b3ed0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fae38000c00 tx=0x7fae380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38034680 con 0x7fae440cb810 2023-10-16T14:17:25.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38034820 con 0x7fae440cb810 2023-10-16T14:17:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38034bd0 con 0x7fae440cb810 2023-10-16T14:17:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.197+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae441e5f70 con 0x7fae440cb810 2023-10-16T14:17:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.198+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae441e6530 con 0x7fae440cb810 2023-10-16T14:17:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.199+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4404e160 con 0x7fae440cb810 2023-10-16T14:17:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.200+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fae3803b040 con 0x7fae440cb810 2023-10-16T14:17:25.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.200+0000 7fae40ff9640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fae3006ae70 0x7fae3006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.201+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fae380aaf60 con 0x7fae440cb810 2023-10-16T14:17:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.201+0000 7fae43fff640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fae3006ae70 0x7fae3006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.202+0000 7fae43fff640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fae3006ae70 0x7fae3006d330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fae34009a30 tx=0x7fae34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.206+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae38076120 con 0x7fae440cb810 2023-10-16T14:17:25.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.402+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae44073380 con 0x7fae440cb810 2023-10-16T14:17:25.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.403+0000 7fae40ff9640 1 -- 172.21.15.29:0/3437776758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae38079740 con 0x7fae440cb810 2023-10-16T14:17:25.407 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:17:25.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fae3006ae70 msgr2=0x7fae3006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fae3006ae70 0x7fae3006d330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fae34009a30 tx=0x7fae34008040 comp rx=0 tx=0).stop 2023-10-16T14:17:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 msgr2=0x7fae440b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440b3ed0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fae38000c00 tx=0x7fae380046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 shutdown_connections 2023-10-16T14:17:25.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fae3006ae70 0x7fae3006d330 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae440cb810 0x7fae440b3ed0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4415aee0 0x7fae441f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 --2- 172.21.15.29:0/3437776758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae44069da0 0x7fae4406a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 >> 172.21.15.29:0/3437776758 conn(0x7fae440cfac0 msgr2=0x7fae44159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:25.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 shutdown_connections 2023-10-16T14:17:25.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.406+0000 7fae4a50c640 1 -- 172.21.15.29:0/3437776758 wait complete. 2023-10-16T14:17:25.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:17:25.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.651+0000 7fd636933640 1 Processor -- start 2023-10-16T14:17:25.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.651+0000 7fd636933640 1 -- start start 2023-10-16T14:17:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd636933640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6300cb1c0 0x7fd63006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd636933640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63015eff0 con 0x7fd630065ab0 2023-10-16T14:17:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd636933640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63015ecc0 con 0x7fd63015af80 2023-10-16T14:17:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd636933640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6300d0930 con 0x7fd6300cb1c0 2023-10-16T14:17:25.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd62f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6300cb1c0 0x7fd63006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd634ed0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd630065ab0 0x7fd6300ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47300/0 (socket says 172.21.15.29:47300) 2023-10-16T14:17:25.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.653+0000 7fd634ed0640 1 -- 172.21.15.29:0/666921073 learned_addr learned my addr 172.21.15.29:0/666921073 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.654+0000 7fd62f7fe640 1 -- 172.21.15.29:0/666921073 >> v1:172.21.15.170:6789/0 conn(0x7fd63015af80 legacy=0x7fd63015f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.654+0000 7fd62f7fe640 1 -- 172.21.15.29:0/666921073 >> v1:172.21.15.29:6789/0 conn(0x7fd630065ab0 legacy=0x7fd6300ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.654+0000 7fd62f7fe640 1 -- 172.21.15.29:0/666921073 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63015b360 con 0x7fd6300cb1c0 2023-10-16T14:17:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.655+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/666921073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300cb1c0 0x7fd63006faf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd6240097a0 tx=0x7fd62402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423727e95d5f0ede server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:25.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.655+0000 7fd62effd640 1 -- 172.21.15.29:0/666921073 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62402e070 con 0x7fd6300cb1c0 2023-10-16T14:17:25.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.655+0000 7fd62effd640 1 -- 172.21.15.29:0/666921073 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6240315a0 con 0x7fd6300cb1c0 2023-10-16T14:17:25.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.655+0000 7fd62effd640 1 -- 172.21.15.29:0/666921073 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624031930 con 0x7fd6300cb1c0 2023-10-16T14:17:25.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 -- 172.21.15.29:0/666921073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300cb1c0 msgr2=0x7fd63006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 --2- 172.21.15.29:0/666921073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300cb1c0 0x7fd63006faf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd6240097a0 tx=0x7fd62402b840 comp rx=0 tx=0).stop 2023-10-16T14:17:25.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 -- 172.21.15.29:0/666921073 shutdown_connections 2023-10-16T14:17:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 --2- 172.21.15.29:0/666921073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300cb1c0 0x7fd63006faf0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 -- 172.21.15.29:0/666921073 >> 172.21.15.29:0/666921073 conn(0x7fd6300d0ab0 msgr2=0x7fd6300d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 -- 172.21.15.29:0/666921073 shutdown_connections 2023-10-16T14:17:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.656+0000 7fd636933640 1 -- 172.21.15.29:0/666921073 wait complete. 2023-10-16T14:17:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.658+0000 7fd636933640 1 Processor -- start 2023-10-16T14:17:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.658+0000 7fd636933640 1 -- start start 2023-10-16T14:17:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd636933640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd630065ab0 0x7fd6300c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd636933640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6300cb1c0 0x7fd6300c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd636933640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 0x7fd63015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd636933640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300c6b20 con 0x7fd63015af80 2023-10-16T14:17:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd636933640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300c5ea0 con 0x7fd630065ab0 2023-10-16T14:17:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd636933640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300c6180 con 0x7fd6300cb1c0 2023-10-16T14:17:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd634ed0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 0x7fd63015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd634ed0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 0x7fd63015d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47302/0 (socket says 172.21.15.29:47302) 2023-10-16T14:17:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd634ed0640 1 -- 172.21.15.29:0/4231258884 learned_addr learned my addr 172.21.15.29:0/4231258884 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd62ffff640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd630065ab0 0x7fd6300c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.659+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6300cb1c0 0x7fd6300c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd634ed0640 1 -- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6300cb1c0 msgr2=0x7fd6300c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd634ed0640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6300cb1c0 0x7fd6300c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd634ed0640 1 -- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd630065ab0 msgr2=0x7fd6300c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd634ed0640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd630065ab0 0x7fd6300c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd634ed0640 1 -- 172.21.15.29:0/4231258884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624009400 con 0x7fd63015af80 2023-10-16T14:17:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6300cb1c0 0x7fd6300c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd634ed0640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 0x7fd63015d9b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c370 tx=0x7fd62800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628005ce0 con 0x7fd63015af80 2023-10-16T14:17:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628005e40 con 0x7fd63015af80 2023-10-16T14:17:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280072d0 con 0x7fd63015af80 2023-10-16T14:17:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6300c6460 con 0x7fd63015af80 2023-10-16T14:17:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.660+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6300c7480 con 0x7fd63015af80 2023-10-16T14:17:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.661+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6300c6c60 con 0x7fd63015af80 2023-10-16T14:17:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.662+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd6280074f0 con 0x7fd63015af80 2023-10-16T14:17:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.664+0000 7fd62cff9640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61c06ae70 0x7fd61c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.664+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fd628088a00 con 0x7fd63015af80 2023-10-16T14:17:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.665+0000 7fd62ffff640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61c06ae70 0x7fd61c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:25.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.666+0000 7fd62ffff640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61c06ae70 0x7fd61c06d330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd620007fd0 tx=0x7fd620009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:25.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.668+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6300c6c60 con 0x7fd63015af80 2023-10-16T14:17:25.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.869+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6300c6c60 con 0x7fd63015af80 2023-10-16T14:17:25.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.870+0000 7fd62cff9640 1 -- 172.21.15.29:0/4231258884 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6300c6c60 con 0x7fd63015af80 2023-10-16T14:17:25.873 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:17:25.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61c06ae70 msgr2=0x7fd61c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61c06ae70 0x7fd61c06d330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd620007fd0 tx=0x7fd620009040 comp rx=0 tx=0).stop 2023-10-16T14:17:25.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 msgr2=0x7fd63015d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:25.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 0x7fd63015d9b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c370 tx=0x7fd62800c830 comp rx=0 tx=0).stop 2023-10-16T14:17:25.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 shutdown_connections 2023-10-16T14:17:25.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd61c06ae70 0x7fd61c06d330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015af80 0x7fd63015d9b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6300cb1c0 0x7fd6300c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 --2- 172.21.15.29:0/4231258884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd630065ab0 0x7fd6300c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:25.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 >> 172.21.15.29:0/4231258884 conn(0x7fd6300d0ab0 msgr2=0x7fd63006aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:25.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.873+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 shutdown_connections 2023-10-16T14:17:25.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:25.874+0000 7fd636933640 1 -- 172.21.15.29:0/4231258884 wait complete. 2023-10-16T14:17:25.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:17:26.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.112+0000 7f36801e9640 1 Processor -- start 2023-10-16T14:17:26.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.112+0000 7f36801e9640 1 -- start start 2023-10-16T14:17:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f36801e9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f367815aaa0 0x7f367815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f36801e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36780ce390 0x7f3678070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f36801e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367815ed60 con 0x7f367815ae80 2023-10-16T14:17:26.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f36801e9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367806b810 con 0x7f367815aaa0 2023-10-16T14:17:26.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f36801e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780707e0 con 0x7f36780ce390 2023-10-16T14:17:26.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f367df85640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f367815aaa0 0x7f367815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f367df85640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f367815aaa0 0x7f367815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47314/0 (socket says 172.21.15.29:47314) 2023-10-16T14:17:26.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.114+0000 7f367df85640 1 -- 172.21.15.29:0/1498906347 learned_addr learned my addr 172.21.15.29:0/1498906347 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.115+0000 7f367df85640 1 -- 172.21.15.29:0/1498906347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36780ce390 msgr2=0x7f3678070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.115+0000 7f367df85640 1 --2- 172.21.15.29:0/1498906347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36780ce390 0x7f3678070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.115+0000 7f367df85640 1 -- 172.21.15.29:0/1498906347 >> v1:172.21.15.170:6789/0 conn(0x7f367815ae80 legacy=0x7f36780cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.115+0000 7f367df85640 1 -- 172.21.15.29:0/1498906347 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367815ef90 con 0x7f367815aaa0 2023-10-16T14:17:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.116+0000 7f367df85640 1 --2- 172.21.15.29:0/1498906347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367815aaa0 0x7f367815e820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f36780cd990 tx=0x7f367402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f717851ff5efaad7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.116+0000 7f367cf83640 1 -- 172.21.15.29:0/1498906347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674031640 con 0x7f367815aaa0 2023-10-16T14:17:26.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.116+0000 7f367cf83640 1 -- 172.21.15.29:0/1498906347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36740317a0 con 0x7f367815aaa0 2023-10-16T14:17:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.116+0000 7f367cf83640 1 -- 172.21.15.29:0/1498906347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674031b30 con 0x7f367815aaa0 2023-10-16T14:17:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 -- 172.21.15.29:0/1498906347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367815aaa0 msgr2=0x7f367815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 --2- 172.21.15.29:0/1498906347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367815aaa0 0x7f367815e820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f36780cd990 tx=0x7f367402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 -- 172.21.15.29:0/1498906347 shutdown_connections 2023-10-16T14:17:26.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 --2- 172.21.15.29:0/1498906347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f36780ce390 0x7f3678070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 --2- 172.21.15.29:0/1498906347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f367815aaa0 0x7f367815e820 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 -- 172.21.15.29:0/1498906347 >> 172.21.15.29:0/1498906347 conn(0x7f36780cb410 msgr2=0x7f36780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:26.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 -- 172.21.15.29:0/1498906347 shutdown_connections 2023-10-16T14:17:26.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.117+0000 7f36801e9640 1 -- 172.21.15.29:0/1498906347 wait complete. 2023-10-16T14:17:26.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.119+0000 7f36801e9640 1 Processor -- start 2023-10-16T14:17:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.119+0000 7f36801e9640 1 -- start start 2023-10-16T14:17:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f36801e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 0x7f36781f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f36801e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f367815aaa0 0x7f367815d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f36801e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f367815ae80 0x7f367815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f36801e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780c3c80 con 0x7f36780ce390 2023-10-16T14:17:26.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f36801e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780c4e40 con 0x7f367815ae80 2023-10-16T14:17:26.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f36801e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780c3070 con 0x7f367815aaa0 2023-10-16T14:17:26.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f367df85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 0x7f36781f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f367df85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 0x7f36781f5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47320/0 (socket says 172.21.15.29:47320) 2023-10-16T14:17:26.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.120+0000 7f367df85640 1 -- 172.21.15.29:0/2932659959 learned_addr learned my addr 172.21.15.29:0/2932659959 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:26.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367d784640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f367815aaa0 0x7f367815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367df85640 1 -- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f367815aaa0 msgr2=0x7f367815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367e786640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f367815ae80 0x7f367815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367df85640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f367815aaa0 0x7f367815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367df85640 1 -- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f367815ae80 msgr2=0x7f367815d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367df85640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f367815ae80 0x7f367815d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367df85640 1 -- 172.21.15.29:0/2932659959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36740094d0 con 0x7f36780ce390 2023-10-16T14:17:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367d784640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f367815aaa0 0x7f367815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.121+0000 7f367df85640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 0x7f36781f5d00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3674000c00 tx=0x7f36740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.122+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367403ac40 con 0x7f36780ce390 2023-10-16T14:17:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.122+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367403adc0 con 0x7f36780ce390 2023-10-16T14:17:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.122+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36780c3350 con 0x7f36780ce390 2023-10-16T14:17:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.122+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674044630 con 0x7f36780ce390 2023-10-16T14:17:26.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.122+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36780c3910 con 0x7f36780ce390 2023-10-16T14:17:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.123+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36780705d0 con 0x7f36780ce390 2023-10-16T14:17:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.124+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f367402f050 con 0x7f36780ce390 2023-10-16T14:17:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.124+0000 7f366e7fc640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f365c06ae70 0x7f365c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.125+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f36740ab790 con 0x7f36780ce390 2023-10-16T14:17:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.125+0000 7f367d784640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f365c06ae70 0x7f365c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.126+0000 7f367d784640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f365c06ae70 0x7f365c06d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f36780b2220 tx=0x7f3668008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:26.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.129+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f36740766f0 con 0x7f36780ce390 2023-10-16T14:17:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.349+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36780cbd20 con 0x7f36780ce390 2023-10-16T14:17:26.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.350+0000 7f366e7fc640 1 -- 172.21.15.29:0/2932659959 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f3674031e60 con 0x7f36780ce390 2023-10-16T14:17:26.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:26.355 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:17:26.355 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:17:26.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f365c06ae70 msgr2=0x7f365c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f365c06ae70 0x7f365c06d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f36780b2220 tx=0x7f3668008040 comp rx=0 tx=0).stop 2023-10-16T14:17:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 msgr2=0x7f36781f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 0x7f36781f5d00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3674000c00 tx=0x7f36740046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 shutdown_connections 2023-10-16T14:17:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f365c06ae70 0x7f365c06d330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f367815ae80 0x7f367815d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f367815aaa0 0x7f367815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 --2- 172.21.15.29:0/2932659959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780ce390 0x7f36781f5d00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 >> 172.21.15.29:0/2932659959 conn(0x7f36780cb410 msgr2=0x7f36780b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:26.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.353+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 shutdown_connections 2023-10-16T14:17:26.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.354+0000 7f36801e9640 1 -- 172.21.15.29:0/2932659959 wait complete. 2023-10-16T14:17:26.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.592+0000 7ff204d2e640 1 Processor -- start 2023-10-16T14:17:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.593+0000 7ff204d2e640 1 -- start start 2023-10-16T14:17:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.594+0000 7ff204d2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cbe0 0x7ff2000d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.594+0000 7ff204d2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2000b4c80 0x7ff2000b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.594+0000 7ff204d2e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2000d15a0 con 0x7ff20015cfc0 2023-10-16T14:17:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.594+0000 7ff204d2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2000d1880 con 0x7ff2000b4c80 2023-10-16T14:17:26.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.594+0000 7ff204d2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000d1b60 con 0x7ff20015cbe0 2023-10-16T14:17:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2000b4c80 0x7ff2000b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2000b4c80 0x7ff2000b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47324/0 (socket says 172.21.15.29:47324) 2023-10-16T14:17:26.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 -- 172.21.15.29:0/3141783350 learned_addr learned my addr 172.21.15.29:0/3141783350 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:26.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cbe0 0x7ff2000d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 -- 172.21.15.29:0/3141783350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cbe0 msgr2=0x7ff2000d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:26.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cbe0 0x7ff2000d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 -- 172.21.15.29:0/3141783350 >> v1:172.21.15.170:6789/0 conn(0x7ff20015cfc0 legacy=0x7ff2000b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1fffff640 1 -- 172.21.15.29:0/3141783350 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2000d1e40 con 0x7ff2000b4c80 2023-10-16T14:17:26.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.595+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cbe0 0x7ff2000d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:26.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.596+0000 7ff1fffff640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000b4c80 0x7ff2000b5c20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92c0b1e906e9d72 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:26.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff1fe7fc640 1 -- 172.21.15.29:0/3141783350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f802f840 con 0x7ff2000b4c80 2023-10-16T14:17:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff1fe7fc640 1 -- 172.21.15.29:0/3141783350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f802f9a0 con 0x7ff2000b4c80 2023-10-16T14:17:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff1fe7fc640 1 -- 172.21.15.29:0/3141783350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f802fd30 con 0x7ff2000b4c80 2023-10-16T14:17:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff204d2e640 1 -- 172.21.15.29:0/3141783350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000b4c80 msgr2=0x7ff2000b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff204d2e640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000b4c80 0x7ff2000b5c20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f802ba20 comp rx=0 tx=0).stop 2023-10-16T14:17:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff204d2e640 1 -- 172.21.15.29:0/3141783350 shutdown_connections 2023-10-16T14:17:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff204d2e640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000b4c80 0x7ff2000b5c20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff204d2e640 1 --2- 172.21.15.29:0/3141783350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cbe0 0x7ff2000d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.597+0000 7ff204d2e640 1 -- 172.21.15.29:0/3141783350 >> 172.21.15.29:0/3141783350 conn(0x7ff2000cf570 msgr2=0x7ff2000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.598+0000 7ff204d2e640 1 -- 172.21.15.29:0/3141783350 shutdown_connections 2023-10-16T14:17:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.598+0000 7ff204d2e640 1 -- 172.21.15.29:0/3141783350 wait complete. 2023-10-16T14:17:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.599+0000 7ff204d2e640 1 Processor -- start 2023-10-16T14:17:26.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.599+0000 7ff204d2e640 1 -- start start 2023-10-16T14:17:26.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.600+0000 7ff204d2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2000b4c80 0x7ff200154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.600+0000 7ff204d2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 0x7ff2000c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.600+0000 7ff204d2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cfc0 0x7ff2000c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.600+0000 7ff204d2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200152830 con 0x7ff20015cbe0 2023-10-16T14:17:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.600+0000 7ff204d2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200152c80 con 0x7ff2000b4c80 2023-10-16T14:17:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.600+0000 7ff204d2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000cb3e0 con 0x7ff20015cfc0 2023-10-16T14:17:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 0x7ff2000c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 0x7ff2000c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47328/0 (socket says 172.21.15.29:47328) 2023-10-16T14:17:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 -- 172.21.15.29:0/379943295 learned_addr learned my addr 172.21.15.29:0/379943295 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2000b4c80 0x7ff200154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 -- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cfc0 msgr2=0x7ff2000c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cfc0 0x7ff2000c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 -- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2000b4c80 msgr2=0x7ff200154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2000b4c80 0x7ff200154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 -- 172.21.15.29:0/379943295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80094d0 con 0x7ff20015cbe0 2023-10-16T14:17:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.601+0000 7ff1feffd640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 0x7ff2000c8e30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c3c0 tx=0x7ff1e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.602+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80052d0 con 0x7ff20015cbe0 2023-10-16T14:17:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.602+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e8005450 con 0x7ff20015cbe0 2023-10-16T14:17:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.602+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8003830 con 0x7ff20015cbe0 2023-10-16T14:17:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.602+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2000cb6c0 con 0x7ff20015cbe0 2023-10-16T14:17:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.602+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2000cbc80 con 0x7ff20015cbe0 2023-10-16T14:17:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.603+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20004e9b0 con 0x7ff20015cbe0 2023-10-16T14:17:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.604+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff1e8003990 con 0x7ff20015cbe0 2023-10-16T14:17:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.605+0000 7ff1f7fff640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff1e006ae70 0x7ff1e006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.605+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7ff1e8088bd0 con 0x7ff20015cbe0 2023-10-16T14:17:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.606+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff1e006ae70 0x7ff1e006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.607+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff1e006ae70 0x7ff1e006d330 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0003fd0 tx=0x7ff1f0003ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.609+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1e8053bc0 con 0x7ff20015cbe0 2023-10-16T14:17:26.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.801+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff200152dc0 con 0x7ff20015cbe0 2023-10-16T14:17:26.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.802+0000 7ff1f7fff640 1 -- 172.21.15.29:0/379943295 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7ff1e8017090 con 0x7ff20015cbe0 2023-10-16T14:17:26.806 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:26.806 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff1e006ae70 msgr2=0x7ff1e006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff1e006ae70 0x7ff1e006d330 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0003fd0 tx=0x7ff1f0003ce0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 msgr2=0x7ff2000c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 0x7ff2000c8e30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c3c0 tx=0x7ff1e800c880 comp rx=0 tx=0).stop 2023-10-16T14:17:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 shutdown_connections 2023-10-16T14:17:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff1e006ae70 0x7ff1e006d330 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20015cbe0 0x7ff2000c8e30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff20015cfc0 0x7ff2000c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 --2- 172.21.15.29:0/379943295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2000b4c80 0x7ff200154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 >> 172.21.15.29:0/379943295 conn(0x7ff2000cf570 msgr2=0x7ff20015c240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 shutdown_connections 2023-10-16T14:17:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:26.805+0000 7ff204d2e640 1 -- 172.21.15.29:0/379943295 wait complete. 2023-10-16T14:17:26.827 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:17:26.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:17:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.054+0000 7fa30494b640 1 Processor -- start 2023-10-16T14:17:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.054+0000 7fa30494b640 1 -- start start 2023-10-16T14:17:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.055+0000 7fa30494b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 0x7fa3000b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.055+0000 7fa30494b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3000d15a0 con 0x7fa30015cbe0 2023-10-16T14:17:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.055+0000 7fa30494b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3000d1880 con 0x7fa3000d2640 2023-10-16T14:17:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.055+0000 7fa30494b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3000d1b60 con 0x7fa30015cfc0 2023-10-16T14:17:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.056+0000 7fa2fed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa30015cbe0 0x7fa3000d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47308/0 (socket says 172.21.15.29:47308) 2023-10-16T14:17:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.056+0000 7fa2fed9d640 1 -- 172.21.15.29:0/2305379961 learned_addr learned my addr 172.21.15.29:0/2305379961 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.056+0000 7fa2f659c640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 0x7fa3000b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.056+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082514168 0 0) 0x7fa3000d15a0 con 0x7fa30015cbe0 2023-10-16T14:17:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.056+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec003520 con 0x7fa30015cbe0 2023-10-16T14:17:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846949569 0 0) 0x7fa3000d1880 con 0x7fa3000d2640 2023-10-16T14:17:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3000d15a0 con 0x7fa3000d2640 2023-10-16T14:17:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 995879805 0 0) 0x7fa2ec003520 con 0x7fa30015cbe0 2023-10-16T14:17:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 msgr2=0x7fa3000b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 0x7fa3000b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 >> v1:172.21.15.170:6789/0 conn(0x7fa3000d2640 legacy=0x7fa3000b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3000d1e40 con 0x7fa30015cbe0 2023-10-16T14:17:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa2f00019f0 con 0x7fa30015cbe0 2023-10-16T14:17:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 >> v1:172.21.15.29:6789/0 conn(0x7fa30015cbe0 legacy=0x7fa3000d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec003b30 0x7fa2ec003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 0x7fa2ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec004d00 0x7fa2ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3000d1b60 con 0x7fa2ec003b30 2023-10-16T14:17:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec00ba30 con 0x7fa2ec004d00 2023-10-16T14:17:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec00bbf0 con 0x7fa30015cfc0 2023-10-16T14:17:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.057+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec003b30 0x7fa2ec003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.058+0000 7fa2fed9d640 1 -- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 msgr2=0x7fa2ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.058+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa30015cfc0 0x7fa2ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.058+0000 7fa2fed9d640 1 -- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec004d00 msgr2=0x7fa2ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.058+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec004d00 0x7fa2ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.058+0000 7fa2fed9d640 1 -- 172.21.15.29:0/2305379961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec003520 con 0x7fa2ec003b30 2023-10-16T14:17:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.059+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec003b30 0x7fa2ec003f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000a700 tx=0x7fa2ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdb741d81dc5399 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.061+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0034630 con 0x7fa2ec003b30 2023-10-16T14:17:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.061+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f002f070 con 0x7fa2ec003b30 2023-10-16T14:17:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.061+0000 7fa2fe59c640 1 -- 172.21.15.29:0/2305379961 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00349f0 con 0x7fa2ec003b30 2023-10-16T14:17:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.061+0000 7fa30494b640 1 -- 172.21.15.29:0/2305379961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec003b30 msgr2=0x7fa2ec003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.061+0000 7fa30494b640 1 --2- 172.21.15.29:0/2305379961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec003b30 0x7fa2ec003f30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000a700 tx=0x7fa2ec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.061+0000 7fa2ff59e640 1 -- 172.21.15.29:0/2305379961 reap_dead start 2023-10-16T14:17:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.062+0000 7fa30494b640 1 -- 172.21.15.29:0/2305379961 shutdown_connections 2023-10-16T14:17:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.062+0000 7fa30494b640 1 -- 172.21.15.29:0/2305379961 >> 172.21.15.29:0/2305379961 conn(0x7fa3000cf570 msgr2=0x7fa3000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.062+0000 7fa30494b640 1 -- 172.21.15.29:0/2305379961 shutdown_connections 2023-10-16T14:17:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.062+0000 7fa30494b640 1 -- 172.21.15.29:0/2305379961 wait complete. 2023-10-16T14:17:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.063+0000 7fa30494b640 1 Processor -- start 2023-10-16T14:17:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.064+0000 7fa30494b640 1 -- start start 2023-10-16T14:17:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa30494b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec003b30 0x7fa3001f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa30494b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3000cbbf0 0x7fa3000c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa30494b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 0x7fa3000ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa30494b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3000ca580 con 0x7fa3000c9bc0 2023-10-16T14:17:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa30494b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3001f6d50 con 0x7fa2ec003b30 2023-10-16T14:17:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa30494b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30015dde0 con 0x7fa3000cbbf0 2023-10-16T14:17:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2ff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 0x7fa3000ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2ff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 0x7fa3000ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47348/0 (socket says 172.21.15.29:47348) 2023-10-16T14:17:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2ff59e640 1 -- 172.21.15.29:0/2461261666 learned_addr learned my addr 172.21.15.29:0/2461261666 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2f659c640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3000cbbf0 0x7fa3000c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2ff59e640 1 -- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3000cbbf0 msgr2=0x7fa3000c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec003b30 0x7fa3001f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.065+0000 7fa2ff59e640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3000cbbf0 0x7fa3000c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2ff59e640 1 -- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec003b30 msgr2=0x7fa3001f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2ff59e640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec003b30 0x7fa3001f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2ff59e640 1 -- 172.21.15.29:0/2461261666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3000cc420 con 0x7fa3000c9bc0 2023-10-16T14:17:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2ff59e640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 0x7fa3000ca040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa3000b24e0 tx=0x7fa2f800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80074e0 con 0x7fa3000c9bc0 2023-10-16T14:17:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8007660 con 0x7fa3000c9bc0 2023-10-16T14:17:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3000cc700 con 0x7fa3000c9bc0 2023-10-16T14:17:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8004260 con 0x7fa3000c9bc0 2023-10-16T14:17:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.066+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3000cccc0 con 0x7fa3000c9bc0 2023-10-16T14:17:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.067+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30015dfe0 con 0x7fa3000c9bc0 2023-10-16T14:17:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.068+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa2f80057c0 con 0x7fa3000c9bc0 2023-10-16T14:17:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.069+0000 7fa2f7fff640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa2e006ae70 0x7fa2e006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.069+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fa2f8088820 con 0x7fa3000c9bc0 2023-10-16T14:17:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.070+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa2e006ae70 0x7fa2e006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.071+0000 7fa2fed9d640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa2e006ae70 0x7fa2e006d330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0034000 tx=0x7fa2f002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:27.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.073+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa30015dfe0 con 0x7fa3000c9bc0 2023-10-16T14:17:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.308+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa30015dfa0 con 0x7fa3000c9bc0 2023-10-16T14:17:27.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.309+0000 7fa2f7fff640 1 -- 172.21.15.29:0/2461261666 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7fa2f8053810 con 0x7fa3000c9bc0 2023-10-16T14:17:27.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:27.313 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:17:27.313 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:17:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.312+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa2e006ae70 msgr2=0x7fa2e006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.312+0000 7fa30494b640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa2e006ae70 0x7fa2e006d330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0034000 tx=0x7fa2f002f000 comp rx=0 tx=0).stop 2023-10-16T14:17:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.312+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 msgr2=0x7fa3000ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.312+0000 7fa30494b640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 0x7fa3000ca040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa3000b24e0 tx=0x7fa2f800c820 comp rx=0 tx=0).stop 2023-10-16T14:17:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.312+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 shutdown_connections 2023-10-16T14:17:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa2e006ae70 0x7fa2e006d330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3000c9bc0 0x7fa3000ca040 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa3000cbbf0 0x7fa3000c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 --2- 172.21.15.29:0/2461261666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2ec003b30 0x7fa3001f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 >> 172.21.15.29:0/2461261666 conn(0x7fa3000cf570 msgr2=0x7fa3000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 shutdown_connections 2023-10-16T14:17:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.313+0000 7fa30494b640 1 -- 172.21.15.29:0/2461261666 wait complete. 2023-10-16T14:17:27.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:17:27.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.555+0000 7f8c829f0640 1 Processor -- start 2023-10-16T14:17:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.556+0000 7f8c829f0640 1 -- start start 2023-10-16T14:17:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.557+0000 7f8c829f0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8c7c06c040 0x7f8c7c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.557+0000 7f8c829f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 0x7f8c7c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.557+0000 7f8c829f0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c06c960 con 0x7f8c7c0ad970 2023-10-16T14:17:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.557+0000 7f8c829f0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c15b9d0 con 0x7f8c7c06c040 2023-10-16T14:17:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.557+0000 7f8c829f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c15bcb0 con 0x7f8c7c0cda80 2023-10-16T14:17:27.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.558+0000 7f8c819ee640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8c7c06c040 0x7f8c7c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.558+0000 7f8c821ef640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 0x7f8c7c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.558+0000 7f8c821ef640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 0x7f8c7c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:46618/0 (socket says 172.21.15.29:46618) 2023-10-16T14:17:27.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.558+0000 7f8c819ee640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8c7c06c040 0x7f8c7c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58298/0 (socket says 172.21.15.29:58298) 2023-10-16T14:17:27.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.558+0000 7f8c821ef640 1 -- 172.21.15.29:0/768031284 learned_addr learned my addr 172.21.15.29:0/768031284 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:27.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.559+0000 7f8c819ee640 1 -- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 msgr2=0x7f8c7c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:27.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.559+0000 7f8c819ee640 1 --2- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 0x7f8c7c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.559+0000 7f8c819ee640 1 -- 172.21.15.29:0/768031284 >> v1:172.21.15.170:6789/0 conn(0x7f8c7c0ad970 legacy=0x7f8c7c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.559+0000 7f8c819ee640 1 -- 172.21.15.29:0/768031284 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c15bf90 con 0x7f8c7c06c040 2023-10-16T14:17:27.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.559+0000 7f8c821ef640 1 --2- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 0x7f8c7c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:17:27.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.560+0000 7f8c819ee640 1 --2- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c06c040 0x7f8c7c06c420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c780097a0 tx=0x7f8c7802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9226ed4e2ab1e401 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:27.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.561+0000 7f8c809ec640 1 -- 172.21.15.29:0/768031284 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7802d040 con 0x7f8c7c06c040 2023-10-16T14:17:27.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.561+0000 7f8c809ec640 1 -- 172.21.15.29:0/768031284 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c78031810 con 0x7f8c7c06c040 2023-10-16T14:17:27.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.561+0000 7f8c809ec640 1 -- 172.21.15.29:0/768031284 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78031ba0 con 0x7f8c7c06c040 2023-10-16T14:17:27.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.561+0000 7f8c829f0640 1 -- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c06c040 msgr2=0x7f8c7c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 --2- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c06c040 0x7f8c7c06c420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c780097a0 tx=0x7f8c7802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 -- 172.21.15.29:0/768031284 shutdown_connections 2023-10-16T14:17:27.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 --2- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0cda80 0x7f8c7c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 --2- 172.21.15.29:0/768031284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c06c040 0x7f8c7c06c420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 -- 172.21.15.29:0/768031284 >> 172.21.15.29:0/768031284 conn(0x7f8c7c0ce6a0 msgr2=0x7f8c7c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:27.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 -- 172.21.15.29:0/768031284 shutdown_connections 2023-10-16T14:17:27.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.562+0000 7f8c829f0640 1 -- 172.21.15.29:0/768031284 wait complete. 2023-10-16T14:17:27.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.563+0000 7f8c829f0640 1 Processor -- start 2023-10-16T14:17:27.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.564+0000 7f8c829f0640 1 -- start start 2023-10-16T14:17:27.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c829f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 0x7f8c7c0b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c829f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0ad970 0x7f8c7c0c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c829f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c0cda80 0x7f8c7c0c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c829f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1f1490 con 0x7f8c7c06c040 2023-10-16T14:17:27.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c829f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1f0bf0 con 0x7f8c7c0cda80 2023-10-16T14:17:27.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c829f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1f0ed0 con 0x7f8c7c0ad970 2023-10-16T14:17:27.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 0x7f8c7c0b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 0x7f8c7c0b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47358/0 (socket says 172.21.15.29:47358) 2023-10-16T14:17:27.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 -- 172.21.15.29:0/481488510 learned_addr learned my addr 172.21.15.29:0/481488510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:27.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c811ed640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0ad970 0x7f8c7c0c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 -- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0ad970 msgr2=0x7f8c7c0c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0ad970 0x7f8c7c0c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 -- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c0cda80 msgr2=0x7f8c7c0c2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c821ef640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c0cda80 0x7f8c7c0c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c0cda80 0x7f8c7c0c2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.565+0000 7f8c819ee640 1 -- 172.21.15.29:0/481488510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74009b30 con 0x7f8c7c06c040 2023-10-16T14:17:27.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.566+0000 7f8c819ee640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 0x7f8c7c0b3ba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8c780098c0 tx=0x7f8c780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:27.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.566+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7803ace0 con 0x7f8c7c06c040 2023-10-16T14:17:27.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.566+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7803ae60 con 0x7f8c7c06c040 2023-10-16T14:17:27.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.566+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78044640 con 0x7f8c7c06c040 2023-10-16T14:17:27.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.566+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c78009400 con 0x7f8c7c06c040 2023-10-16T14:17:27.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.566+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c0b1bf0 con 0x7f8c7c06c040 2023-10-16T14:17:27.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.567+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c06b330 con 0x7f8c7c06c040 2023-10-16T14:17:27.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.568+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c7804d440 con 0x7f8c7c06c040 2023-10-16T14:17:27.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.569+0000 7f8c6e7fc640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c6006ae70 0x7f8c6006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:27.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.569+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f8c780ab9b0 con 0x7f8c7c06c040 2023-10-16T14:17:27.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.570+0000 7f8c811ed640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c6006ae70 0x7f8c6006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:27.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.571+0000 7f8c811ed640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c6006ae70 0x7f8c6006d330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c1f4b50 tx=0x7f8c70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:27.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.577+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c780769a0 con 0x7f8c7c06c040 2023-10-16T14:17:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.818+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c04e9b0 con 0x7f8c7c06c040 2023-10-16T14:17:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.820+0000 7f8c6e7fc640 1 -- 172.21.15.29:0/481488510 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f8c78035090 con 0x7f8c7c06c040 2023-10-16T14:17:27.823 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:27.823 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1275364094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1275364094},{"type":"v1","addr":"172.21.15.29:6833","nonce":1275364094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/709331809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":709331809},{"type":"v1","addr":"172.21.15.29:6835","nonce":709331809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1491057332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1491057332},{"type":"v1","addr":"172.21.15.170:6836","nonce":1491057332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:16:29.503784+0000","modified":"2023-10-16T14:16:32.553956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3275719351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3275719351},{"type":"v1","addr":"172.21.15.170:6837","nonce":3275719351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T14:17:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-16T14:17:27.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.822+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c6006ae70 msgr2=0x7f8c6006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c6006ae70 0x7f8c6006d330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c1f4b50 tx=0x7f8c70009040 comp rx=0 tx=0).stop 2023-10-16T14:17:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 msgr2=0x7f8c7c0b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 0x7f8c7c0b3ba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8c780098c0 tx=0x7f8c780046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 shutdown_connections 2023-10-16T14:17:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f8c6006ae70 0x7f8c6006d330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c7c0cda80 0x7f8c7c0c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c7c0ad970 0x7f8c7c0c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 --2- 172.21.15.29:0/481488510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c7c06c040 0x7f8c7c0b3ba0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:27.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 >> 172.21.15.29:0/481488510 conn(0x7f8c7c0ce6a0 msgr2=0x7f8c7c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:27.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 shutdown_connections 2023-10-16T14:17:27.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:27.823+0000 7f8c829f0640 1 -- 172.21.15.29:0/481488510 wait complete. 2023-10-16T14:17:27.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:28.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.070+0000 7f3d4bfff640 1 Processor -- start 2023-10-16T14:17:28.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.070+0000 7f3d4bfff640 1 -- start start 2023-10-16T14:17:28.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3d4c06c040 0x7f3d4c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 0x7f3d4c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c06c960 con 0x7f3d4c15ebc0 2023-10-16T14:17:28.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c15b9d0 con 0x7f3d4c06c040 2023-10-16T14:17:28.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c15bcb0 con 0x7f3d4c0ad970 2023-10-16T14:17:28.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3d4c15ebc0 0x7f3d4c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47322/0 (socket says 172.21.15.29:47322) 2023-10-16T14:17:28.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.072+0000 7f3d4b7fe640 1 -- 172.21.15.29:0/707949085 learned_addr learned my addr 172.21.15.29:0/707949085 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:28.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d4a7fc640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 0x7f3d4c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d4affd640 1 --2- 172.21.15.29:0/707949085 >> v2:172.21.15.170:3300/0 conn(0x7f3d4c06c040 0x7f3d4c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045125325 0 0) 0x7f3d4c06c960 con 0x7f3d4c15ebc0 2023-10-16T14:17:28.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34003520 con 0x7f3d4c15ebc0 2023-10-16T14:17:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 318530396 0 0) 0x7f3d34003520 con 0x7f3d4c15ebc0 2023-10-16T14:17:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 msgr2=0x7f3d4c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 0x7f3d4c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 >> v2:172.21.15.170:3300/0 conn(0x7f3d4c06c040 msgr2=0x7f3d4c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 --2- 172.21.15.29:0/707949085 >> v2:172.21.15.170:3300/0 conn(0x7f3d4c06c040 0x7f3d4c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c15bf90 con 0x7f3d4c15ebc0 2023-10-16T14:17:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3d440019f0 con 0x7f3d4c15ebc0 2023-10-16T14:17:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.073+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 >> v1:172.21.15.29:6789/0 conn(0x7f3d4c15ebc0 legacy=0x7f3d4c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d49ffb640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d34003970 0x7f3d34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d49ffb640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 0x7f3d340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d49ffb640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d34004d00 0x7f3d34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c15b9d0 con 0x7f3d34004d00 2023-10-16T14:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c15bcb0 con 0x7f3d34003970 2023-10-16T14:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3400b980 con 0x7f3d4c0ad970 2023-10-16T14:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d4b7fe640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d34004d00 0x7f3d34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d4b7fe640 1 -- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 msgr2=0x7f3d340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d4b7fe640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0ad970 0x7f3d340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d4b7fe640 1 -- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d34003970 msgr2=0x7f3d34003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d4b7fe640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d34003970 0x7f3d34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.074+0000 7f3d4b7fe640 1 -- 172.21.15.29:0/707949085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34003520 con 0x7f3d34004d00 2023-10-16T14:17:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.075+0000 7f3d4b7fe640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d34004d00 0x7f3d34005180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400a790 tx=0x7f3d340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa33d118d6fe0f02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44034630 con 0x7f3d34004d00 2023-10-16T14:17:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4402f070 con 0x7f3d34004d00 2023-10-16T14:17:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d49ffb640 1 -- 172.21.15.29:0/707949085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44034a10 con 0x7f3d34004d00 2023-10-16T14:17:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4bfff640 1 -- 172.21.15.29:0/707949085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d34004d00 msgr2=0x7f3d34005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/707949085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d34004d00 0x7f3d34005180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400a790 tx=0x7f3d340057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4b7fe640 1 -- 172.21.15.29:0/707949085 reap_dead start 2023-10-16T14:17:28.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4bfff640 1 -- 172.21.15.29:0/707949085 shutdown_connections 2023-10-16T14:17:28.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4bfff640 1 -- 172.21.15.29:0/707949085 >> 172.21.15.29:0/707949085 conn(0x7f3d4c0ce6a0 msgr2=0x7f3d4c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:28.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4bfff640 1 -- 172.21.15.29:0/707949085 shutdown_connections 2023-10-16T14:17:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.076+0000 7f3d4bfff640 1 -- 172.21.15.29:0/707949085 wait complete. 2023-10-16T14:17:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.078+0000 7f3d4bfff640 1 Processor -- start 2023-10-16T14:17:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.079+0000 7f3d4bfff640 1 -- start start 2023-10-16T14:17:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.079+0000 7f3d4bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d4c145e40 0x7f3d4c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.079+0000 7f3d4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 0x7f3d4c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0c4810 0x7f3d4c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c146220 con 0x7f3d4c0c4390 2023-10-16T14:17:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c153620 con 0x7f3d4c145e40 2023-10-16T14:17:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0c3d70 con 0x7f3d4c0c4810 2023-10-16T14:17:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 0x7f3d4c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 0x7f3d4c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47376/0 (socket says 172.21.15.29:47376) 2023-10-16T14:17:28.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 -- 172.21.15.29:0/1532226681 learned_addr learned my addr 172.21.15.29:0/1532226681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:28.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4affd640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d4c145e40 0x7f3d4c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4b7fe640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0c4810 0x7f3d4c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 -- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0c4810 msgr2=0x7f3d4c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0c4810 0x7f3d4c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 -- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d4c145e40 msgr2=0x7f3d4c0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d4c145e40 0x7f3d4c0c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.080+0000 7f3d4a7fc640 1 -- 172.21.15.29:0/1532226681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c0c4050 con 0x7f3d4c0c4390 2023-10-16T14:17:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.081+0000 7f3d4a7fc640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 0x7f3d4c069da0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e7f0 tx=0x7f3d4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.081+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000eea0 con 0x7f3d4c0c4390 2023-10-16T14:17:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.081+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4000c940 con 0x7f3d4c0c4390 2023-10-16T14:17:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.081+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000ccd0 con 0x7f3d4c0c4390 2023-10-16T14:17:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.081+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c0c21d0 con 0x7f3d4c0c4390 2023-10-16T14:17:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.081+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c0c2790 con 0x7f3d4c0c4390 2023-10-16T14:17:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.082+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c06b330 con 0x7f3d4c0c4390 2023-10-16T14:17:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.083+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3d40013470 con 0x7f3d4c0c4390 2023-10-16T14:17:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.084+0000 7f3d33fff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d3806aea0 0x7f3d3806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.084+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f3d40089770 con 0x7f3d4c0c4390 2023-10-16T14:17:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.085+0000 7f3d4affd640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d3806aea0 0x7f3d3806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.086+0000 7f3d4affd640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d3806aea0 0x7f3d3806d360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c004770 tx=0x7f3d3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.089+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d400546d0 con 0x7f3d4c0c4390 2023-10-16T14:17:28.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.289+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d4c04e9b0 con 0x7f3d4c0c4390 2023-10-16T14:17:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.292+0000 7f3d33fff640 1 -- 172.21.15.29:0/1532226681 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f3d40057cf0 con 0x7f3d4c0c4390 2023-10-16T14:17:28.296 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:28.296 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:11.801852+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:16:27.863320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T14:16:28.506011+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.295+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d3806aea0 msgr2=0x7f3d3806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.295+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d3806aea0 0x7f3d3806d360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c004770 tx=0x7f3d3c008040 comp rx=0 tx=0).stop 2023-10-16T14:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.295+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 msgr2=0x7f3d4c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.295+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 0x7f3d4c069da0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e7f0 tx=0x7f3d4000ecb0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 shutdown_connections 2023-10-16T14:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f3d3806aea0 0x7f3d3806d360 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0c4390 0x7f3d4c069da0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d4c0c4810 0x7f3d4c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 --2- 172.21.15.29:0/1532226681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d4c145e40 0x7f3d4c0c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 >> 172.21.15.29:0/1532226681 conn(0x7f3d4c0ce6a0 msgr2=0x7f3d4c0c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 shutdown_connections 2023-10-16T14:17:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.296+0000 7f3d4bfff640 1 -- 172.21.15.29:0/1532226681 wait complete. 2023-10-16T14:17:28.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.543+0000 7f179449b640 1 Processor -- start 2023-10-16T14:17:28.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.544+0000 7f179449b640 1 -- start start 2023-10-16T14:17:28.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.545+0000 7f179449b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f178c15ad70 0x7f178c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.545+0000 7f179449b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f178c0c9ca0 0x7f178c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.545+0000 7f179449b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c0cb030 con 0x7f178c0ca3b0 2023-10-16T14:17:28.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.545+0000 7f179449b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178c0c91c0 con 0x7f178c0c9ca0 2023-10-16T14:17:28.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.545+0000 7f179449b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178c0c94a0 con 0x7f178c15ad70 2023-10-16T14:17:28.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f178c0c9ca0 0x7f178c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1792237640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f178c15ad70 0x7f178c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f178c0c9ca0 0x7f178c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47386/0 (socket says 172.21.15.29:47386) 2023-10-16T14:17:28.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 -- 172.21.15.29:0/3785315114 learned_addr learned my addr 172.21.15.29:0/3785315114 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:28.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 -- 172.21.15.29:0/3785315114 >> v2:172.21.15.170:3300/0 conn(0x7f178c15ad70 msgr2=0x7f178c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:28.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 --2- 172.21.15.29:0/3785315114 >> v2:172.21.15.170:3300/0 conn(0x7f178c15ad70 0x7f178c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 -- 172.21.15.29:0/3785315114 >> v1:172.21.15.170:6789/0 conn(0x7f178c0ca3b0 legacy=0x7f178c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:28.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1791a36640 1 -- 172.21.15.29:0/3785315114 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0c9780 con 0x7f178c0c9ca0 2023-10-16T14:17:28.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.546+0000 7f1792237640 1 --2- 172.21.15.29:0/3785315114 >> v2:172.21.15.170:3300/0 conn(0x7f178c15ad70 0x7f178c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:28.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.548+0000 7f1791a36640 1 --2- 172.21.15.29:0/3785315114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c06a2e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f178c0cf780 tx=0x7f178002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90476b3c40c3d99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:28.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.548+0000 7f1791235640 1 -- 172.21.15.29:0/3785315114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780031870 con 0x7f178c0c9ca0 2023-10-16T14:17:28.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.548+0000 7f1791235640 1 -- 172.21.15.29:0/3785315114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17800319d0 con 0x7f178c0c9ca0 2023-10-16T14:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.548+0000 7f1791235640 1 -- 172.21.15.29:0/3785315114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780031d60 con 0x7f178c0c9ca0 2023-10-16T14:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 -- 172.21.15.29:0/3785315114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 msgr2=0x7f178c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 --2- 172.21.15.29:0/3785315114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c06a2e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f178c0cf780 tx=0x7f178002b910 comp rx=0 tx=0).stop 2023-10-16T14:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 -- 172.21.15.29:0/3785315114 shutdown_connections 2023-10-16T14:17:28.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 --2- 172.21.15.29:0/3785315114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c06a2e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 --2- 172.21.15.29:0/3785315114 >> v2:172.21.15.170:3300/0 conn(0x7f178c15ad70 0x7f178c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 -- 172.21.15.29:0/3785315114 >> 172.21.15.29:0/3785315114 conn(0x7f178c0ce9b0 msgr2=0x7f178c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 -- 172.21.15.29:0/3785315114 shutdown_connections 2023-10-16T14:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.549+0000 7f179449b640 1 -- 172.21.15.29:0/3785315114 wait complete. 2023-10-16T14:17:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.551+0000 7f179449b640 1 Processor -- start 2023-10-16T14:17:28.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.551+0000 7f179449b640 1 -- start start 2023-10-16T14:17:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f179449b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c1f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f179449b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178c0ca3b0 0x7f178c14de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f179449b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178c15ad70 0x7f178c0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f179449b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0c5a80 con 0x7f178c0c9ca0 2023-10-16T14:17:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f179449b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0c5c90 con 0x7f178c15ad70 2023-10-16T14:17:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f179449b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0c5f70 con 0x7f178c0ca3b0 2023-10-16T14:17:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f1792237640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c1f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f1792237640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c1f6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47396/0 (socket says 172.21.15.29:47396) 2023-10-16T14:17:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.552+0000 7f1792237640 1 -- 172.21.15.29:0/2794472463 learned_addr learned my addr 172.21.15.29:0/2794472463 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792237640 1 -- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178c0ca3b0 msgr2=0x7f178c14de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1791a36640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178c0ca3b0 0x7f178c14de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792a38640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178c15ad70 0x7f178c0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792237640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178c0ca3b0 0x7f178c14de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792237640 1 -- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178c15ad70 msgr2=0x7f178c0cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792237640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178c15ad70 0x7f178c0cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792237640 1 -- 172.21.15.29:0/2794472463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17800094d0 con 0x7f178c0c9ca0 2023-10-16T14:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792a38640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178c15ad70 0x7f178c0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f1792237640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c1f6200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f178800d7d0 tx=0x7f178800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788004030 con 0x7f178c0c9ca0 2023-10-16T14:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1788004190 con 0x7f178c0c9ca0 2023-10-16T14:17:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788004520 con 0x7f178c0c9ca0 2023-10-16T14:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c0c6250 con 0x7f178c0c9ca0 2023-10-16T14:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.553+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c0c6810 con 0x7f178c0c9ca0 2023-10-16T14:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.555+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c04e160 con 0x7f178c0c9ca0 2023-10-16T14:17:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.555+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f17880096a0 con 0x7f178c0c9ca0 2023-10-16T14:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.556+0000 7f177effd640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f177006ae70 0x7f177006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.557+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f1788089960 con 0x7f178c0c9ca0 2023-10-16T14:17:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.557+0000 7f1791a36640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f177006ae70 0x7f177006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.558+0000 7f1791a36640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f177006ae70 0x7f177006d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1780036000 tx=0x7f178003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.561+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f17880548c0 con 0x7f178c0c9ca0 2023-10-16T14:17:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.794+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f178c0c8f50 con 0x7f178c0c9ca0 2023-10-16T14:17:28.799 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:17:28.795+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:17:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.799+0000 7f177effd640 1 -- 172.21.15.29:0/2794472463 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1788057ee0 con 0x7f178c0c9ca0 2023-10-16T14:17:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:17:28.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f177006ae70 msgr2=0x7f177006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f177006ae70 0x7f177006d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1780036000 tx=0x7f178003c040 comp rx=0 tx=0).stop 2023-10-16T14:17:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 msgr2=0x7f178c1f6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c1f6200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f178800d7d0 tx=0x7f178800dc90 comp rx=0 tx=0).stop 2023-10-16T14:17:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 shutdown_connections 2023-10-16T14:17:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f177006ae70 0x7f177006d330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178c15ad70 0x7f178c0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178c0ca3b0 0x7f178c14de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 --2- 172.21.15.29:0/2794472463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f178c0c9ca0 0x7f178c1f6200 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 >> 172.21.15.29:0/2794472463 conn(0x7f178c0ce9b0 msgr2=0x7f178c159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 shutdown_connections 2023-10-16T14:17:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:28.802+0000 7f179449b640 1 -- 172.21.15.29:0/2794472463 wait complete. 2023-10-16T14:17:28.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:17:28.846 INFO:tasks.ceph.mds.d.smithi170.stderr: -7424> 2023-10-16T14:16:35.819+0000 7f8dc7243640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:17:28.925 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:17:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.056+0000 7f99fffff640 1 Processor -- start 2023-10-16T14:17:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.056+0000 7f99fffff640 1 -- start start 2023-10-16T14:17:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.057+0000 7f99fffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9a000ad970 0x7f9a000cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.057+0000 7f99fffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 0x7f9a000cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.057+0000 7f99fffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0015b9d0 con 0x7f9a0006c040 2023-10-16T14:17:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.057+0000 7f99fffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a0015bcb0 con 0x7f9a000ad970 2023-10-16T14:17:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.057+0000 7f99fffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0015bf90 con 0x7f9a000ccfb0 2023-10-16T14:17:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.058+0000 7f99ff7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 0x7f9a000cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.058+0000 7f99f7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9a000ad970 0x7f9a000cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.058+0000 7f99ff7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 0x7f9a000cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:46680/0 (socket says 172.21.15.29:46680) 2023-10-16T14:17:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.058+0000 7f99ff7fe640 1 -- 172.21.15.29:0/1099638788 learned_addr learned my addr 172.21.15.29:0/1099638788 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.058+0000 7f99f7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9a000ad970 0x7f9a000cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58346/0 (socket says 172.21.15.29:58346) 2023-10-16T14:17:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.059+0000 7f99ff7fe640 1 -- 172.21.15.29:0/1099638788 >> v2:172.21.15.170:3300/0 conn(0x7f9a000ad970 msgr2=0x7f9a000cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.059+0000 7f99ff7fe640 1 --2- 172.21.15.29:0/1099638788 >> v2:172.21.15.170:3300/0 conn(0x7f9a000ad970 0x7f9a000cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.059+0000 7f99ff7fe640 1 -- 172.21.15.29:0/1099638788 >> v1:172.21.15.170:6789/0 conn(0x7f9a0006c040 legacy=0x7f9a0006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.059+0000 7f99ff7fe640 1 -- 172.21.15.29:0/1099638788 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a000cc270 con 0x7f9a000ccfb0 2023-10-16T14:17:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.060+0000 7f99ff7fe640 1 --2- 172.21.15.29:0/1099638788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 0x7f9a000cd460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009e50 tx=0x7f99f802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecd473472c6a6c96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.061+0000 7f99fe7fc640 1 -- 172.21.15.29:0/1099638788 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f802cb00 con 0x7f9a000ccfb0 2023-10-16T14:17:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.061+0000 7f99fe7fc640 1 -- 172.21.15.29:0/1099638788 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f802cc60 con 0x7f9a000ccfb0 2023-10-16T14:17:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.061+0000 7f99fe7fc640 1 -- 172.21.15.29:0/1099638788 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f80343f0 con 0x7f9a000ccfb0 2023-10-16T14:17:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 -- 172.21.15.29:0/1099638788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 msgr2=0x7f9a000cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 --2- 172.21.15.29:0/1099638788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 0x7f9a000cd460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009e50 tx=0x7f99f802bf60 comp rx=0 tx=0).stop 2023-10-16T14:17:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 -- 172.21.15.29:0/1099638788 shutdown_connections 2023-10-16T14:17:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 --2- 172.21.15.29:0/1099638788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ccfb0 0x7f9a000cd460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 --2- 172.21.15.29:0/1099638788 >> v2:172.21.15.170:3300/0 conn(0x7f9a000ad970 0x7f9a000cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 -- 172.21.15.29:0/1099638788 >> 172.21.15.29:0/1099638788 conn(0x7f9a000ce6a0 msgr2=0x7f9a000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 -- 172.21.15.29:0/1099638788 shutdown_connections 2023-10-16T14:17:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.062+0000 7f99fffff640 1 -- 172.21.15.29:0/1099638788 wait complete. 2023-10-16T14:17:29.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.064+0000 7f99fffff640 1 Processor -- start 2023-10-16T14:17:29.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.064+0000 7f99fffff640 1 -- start start 2023-10-16T14:17:29.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99fffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a0006c040 0x7f9a000c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:29.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99fffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ad970 0x7f9a000b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:29.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99fffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 0x7f9a0014fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:29.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99fffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000b1120 con 0x7f9a000ccfb0 2023-10-16T14:17:29.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99fffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00150340 con 0x7f9a0006c040 2023-10-16T14:17:29.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99fffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000c6710 con 0x7f9a000ad970 2023-10-16T14:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 0x7f9a0014fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 0x7f9a0014fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47400/0 (socket says 172.21.15.29:47400) 2023-10-16T14:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99ff7fe640 1 -- 172.21.15.29:0/892620017 learned_addr learned my addr 172.21.15.29:0/892620017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99f7fff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ad970 0x7f9a000b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.065+0000 7f99feffd640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a0006c040 0x7f9a000c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99ff7fe640 1 -- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ad970 msgr2=0x7f9a000b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99ff7fe640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ad970 0x7f9a000b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99ff7fe640 1 -- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a0006c040 msgr2=0x7f9a000c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99ff7fe640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a0006c040 0x7f9a000c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99ff7fe640 1 -- 172.21.15.29:0/892620017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f8009b30 con 0x7f9a000ccfb0 2023-10-16T14:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99f7fff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ad970 0x7f9a000b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99feffd640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a0006c040 0x7f9a000c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99ff7fe640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 0x7f9a0014fe00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009e50 tx=0x7f99f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8040670 con 0x7f9a000ccfb0 2023-10-16T14:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f80407f0 con 0x7f9a000ccfb0 2023-10-16T14:17:29.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8040ba0 con 0x7f9a000ccfb0 2023-10-16T14:17:29.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.066+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a000c5e70 con 0x7f9a000ccfb0 2023-10-16T14:17:29.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.067+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a000c6430 con 0x7f9a000ccfb0 2023-10-16T14:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.067+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0006b330 con 0x7f9a000ccfb0 2023-10-16T14:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.069+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f99f8040d00 con 0x7f9a000ccfb0 2023-10-16T14:17:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.070+0000 7f99fcff9640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f99e006ae70 0x7f99e006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.070+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f99f80ad0e0 con 0x7f9a000ccfb0 2023-10-16T14:17:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.071+0000 7f99feffd640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f99e006ae70 0x7f99e006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.072+0000 7f99feffd640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f99e006ae70 0x7f99e006d330 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f99f0000f60 tx=0x7f99f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.073+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f99f8078310 con 0x7f9a000ccfb0 2023-10-16T14:17:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.305+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9a0004e9b0 con 0x7f9a000ccfb0 2023-10-16T14:17:29.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.805+0000 7f99fcff9640 1 -- 172.21.15.29:0/892620017 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f99f8080e60 con 0x7f9a000ccfb0 2023-10-16T14:17:29.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f99e006ae70 msgr2=0x7f99e006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:29.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f99e006ae70 0x7f99e006d330 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f99f0000f60 tx=0x7f99f0009040 comp rx=0 tx=0).stop 2023-10-16T14:17:29.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 msgr2=0x7f9a0014fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:29.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 0x7f9a0014fe00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009e50 tx=0x7f99f8004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:29.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 shutdown_connections 2023-10-16T14:17:29.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f99e006ae70 0x7f99e006d330 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a000ccfb0 0x7f9a0014fe00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9a000ad970 0x7f9a000b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 --2- 172.21.15.29:0/892620017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9a0006c040 0x7f9a000c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:29.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 >> 172.21.15.29:0/892620017 conn(0x7f9a000ce6a0 msgr2=0x7f9a0006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:29.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 shutdown_connections 2023-10-16T14:17:29.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:29.810+0000 7f99fffff640 1 -- 172.21.15.29:0/892620017 wait complete. 2023-10-16T14:17:29.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:17:30.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.068+0000 7fb386875640 1 Processor -- start 2023-10-16T14:17:30.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.068+0000 7fb386875640 1 -- start start 2023-10-16T14:17:30.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.069+0000 7fb386875640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:30.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.069+0000 7fb386875640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:30.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.069+0000 7fb386875640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38006c960 con 0x7fb38015ebc0 2023-10-16T14:17:30.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.069+0000 7fb386875640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38015b9d0 con 0x7fb3800ad970 2023-10-16T14:17:30.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.069+0000 7fb386875640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015bcb0 con 0x7fb38006c040 2023-10-16T14:17:30.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb386074640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb38015ebc0 0x7fb3800cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47338/0 (socket says 172.21.15.29:47338) 2023-10-16T14:17:30.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb386074640 1 -- 172.21.15.29:0/3373322330 learned_addr learned my addr 172.21.15.29:0/3373322330 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:30.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb385072640 1 --2- 172.21.15.29:0/3373322330 >> v2:172.21.15.29:3300/0 conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb385873640 1 --2- 172.21.15.29:0/3373322330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb385072640 1 -- 172.21.15.29:0/3373322330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 msgr2=0x7fb38006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb384871640 1 -- 172.21.15.29:0/3373322330 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926358878 0 0) 0x7fb38006c960 con 0x7fb38015ebc0 2023-10-16T14:17:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb385072640 1 --2- 172.21.15.29:0/3373322330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb385072640 1 -- 172.21.15.29:0/3373322330 >> v1:172.21.15.29:6789/0 conn(0x7fb38015ebc0 legacy=0x7fb3800cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.070+0000 7fb385072640 1 -- 172.21.15.29:0/3373322330 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38015bf90 con 0x7fb3800ad970 2023-10-16T14:17:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.071+0000 7fb385072640 1 --2- 172.21.15.29:0/3373322330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 0x7fb38015e680 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb3700097d0 tx=0x7fb37002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969b90b12404cc31 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb384871640 1 -- 172.21.15.29:0/3373322330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370031800 con 0x7fb3800ad970 2023-10-16T14:17:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb384871640 1 -- 172.21.15.29:0/3373322330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb370031960 con 0x7fb3800ad970 2023-10-16T14:17:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb384871640 1 -- 172.21.15.29:0/3373322330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370031cf0 con 0x7fb3800ad970 2023-10-16T14:17:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 -- 172.21.15.29:0/3373322330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 msgr2=0x7fb38015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 --2- 172.21.15.29:0/3373322330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 0x7fb38015e680 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb3700097d0 tx=0x7fb37002b8a0 comp rx=0 tx=0).stop 2023-10-16T14:17:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 -- 172.21.15.29:0/3373322330 shutdown_connections 2023-10-16T14:17:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 --2- 172.21.15.29:0/3373322330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3800ad970 0x7fb38015e680 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 --2- 172.21.15.29:0/3373322330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb38006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 -- 172.21.15.29:0/3373322330 >> 172.21.15.29:0/3373322330 conn(0x7fb3800ce6a0 msgr2=0x7fb3800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 -- 172.21.15.29:0/3373322330 shutdown_connections 2023-10-16T14:17:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.072+0000 7fb386875640 1 -- 172.21.15.29:0/3373322330 wait complete. 2023-10-16T14:17:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.074+0000 7fb386875640 1 Processor -- start 2023-10-16T14:17:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.074+0000 7fb386875640 1 -- start start 2023-10-16T14:17:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.075+0000 7fb386875640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.075+0000 7fb386875640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.075+0000 7fb386875640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb3800af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.075+0000 7fb386875640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015d350 con 0x7fb38015ebc0 2023-10-16T14:17:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.075+0000 7fb386875640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3800ae590 con 0x7fb3800ad970 2023-10-16T14:17:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.075+0000 7fb386875640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3800b0db0 con 0x7fb38006c040 2023-10-16T14:17:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb3800af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb3800af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47416/0 (socket says 172.21.15.29:47416) 2023-10-16T14:17:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 -- 172.21.15.29:0/1060251915 learned_addr learned my addr 172.21.15.29:0/1060251915 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:30.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb385873640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:30.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb385072640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:30.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 -- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 msgr2=0x7fb380156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:30.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:30.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 -- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 msgr2=0x7fb3800b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 -- 172.21.15.29:0/1060251915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb370009400 con 0x7fb38015ebc0 2023-10-16T14:17:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.076+0000 7fb386074640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb3800af030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb37800c7e0 tx=0x7fb37800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.078+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37800a640 con 0x7fb38015ebc0 2023-10-16T14:17:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.078+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37800a7c0 con 0x7fb38015ebc0 2023-10-16T14:17:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.078+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37800ab70 con 0x7fb38015ebc0 2023-10-16T14:17:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.078+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38015d560 con 0x7fb38015ebc0 2023-10-16T14:17:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.078+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3800ae830 con 0x7fb38015ebc0 2023-10-16T14:17:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.079+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38006b330 con 0x7fb38015ebc0 2023-10-16T14:17:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.080+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb37800acd0 con 0x7fb38015ebc0 2023-10-16T14:17:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.081+0000 7fb3767fc640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36406ae70 0x7fb36406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:30.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.081+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5969+0+0 (secure 0 0 0) 0x7fb378088de0 con 0x7fb38015ebc0 2023-10-16T14:17:30.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.082+0000 7fb385873640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36406ae70 0x7fb36406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:30.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.083+0000 7fb385873640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36406ae70 0x7fb36406d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb37c006fd0 tx=0x7fb37c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.086+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb378053ce0 con 0x7fb38015ebc0 2023-10-16T14:17:30.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:30.275+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb38004e9b0 con 0x7fb38015ebc0 2023-10-16T14:17:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.087+0000 7fb3767fc640 1 -- 172.21.15.29:0/1060251915 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb378018070 con 0x7fb38015ebc0 2023-10-16T14:17:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:17:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.090+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36406ae70 msgr2=0x7fb36406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.090+0000 7fb386875640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36406ae70 0x7fb36406d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb37c006fd0 tx=0x7fb37c008040 comp rx=0 tx=0).stop 2023-10-16T14:17:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.090+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 msgr2=0x7fb3800af030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:31.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.090+0000 7fb386875640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb3800af030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb37800c7e0 tx=0x7fb37800cca0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 shutdown_connections 2023-10-16T14:17:31.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb36406ae70 0x7fb36406d330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb38015ebc0 0x7fb3800af030 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3800ad970 0x7fb3800b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 --2- 172.21.15.29:0/1060251915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb38006c040 0x7fb380156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 >> 172.21.15.29:0/1060251915 conn(0x7fb3800ce6a0 msgr2=0x7fb3800ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 shutdown_connections 2023-10-16T14:17:31.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.091+0000 7fb386875640 1 -- 172.21.15.29:0/1060251915 wait complete. 2023-10-16T14:17:31.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:17:31.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.344+0000 7ff6d576c640 1 Processor -- start 2023-10-16T14:17:31.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.345+0000 7ff6d576c640 1 -- start start 2023-10-16T14:17:31.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.346+0000 7ff6d576c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6d015cf60 0x7ff6d006f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:31.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.346+0000 7ff6d576c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:31.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.346+0000 7ff6d576c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d00c9620 con 0x7ff6d006f9d0 2023-10-16T14:17:31.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6d576c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d015d340 con 0x7ff6d015cf60 2023-10-16T14:17:31.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6d576c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00cc860 con 0x7ff6d00b0360 2023-10-16T14:17:31.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6d015cf60 0x7ff6d006f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:31.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6d015cf60 0x7ff6d006f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47428/0 (socket says 172.21.15.29:47428) 2023-10-16T14:17:31.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 -- 172.21.15.29:0/2879826775 learned_addr learned my addr 172.21.15.29:0/2879826775 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:31.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 -- 172.21.15.29:0/2879826775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 msgr2=0x7ff6d00b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:31.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 --2- 172.21.15.29:0/2879826775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 -- 172.21.15.29:0/2879826775 >> v1:172.21.15.170:6789/0 conn(0x7ff6d006f9d0 legacy=0x7ff6d00c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:31.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.347+0000 7ff6ceffd640 1 -- 172.21.15.29:0/2879826775 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d00ad4a0 con 0x7ff6d015cf60 2023-10-16T14:17:31.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.349+0000 7ff6ceffd640 1 --2- 172.21.15.29:0/2879826775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d006f490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00098a0 tx=0x7ff6c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1406756718c06a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:31.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.349+0000 7ff6cdffb640 1 -- 172.21.15.29:0/2879826775 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0031640 con 0x7ff6d015cf60 2023-10-16T14:17:31.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.349+0000 7ff6cdffb640 1 -- 172.21.15.29:0/2879826775 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c00317a0 con 0x7ff6d015cf60 2023-10-16T14:17:31.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.349+0000 7ff6cdffb640 1 -- 172.21.15.29:0/2879826775 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0031b30 con 0x7ff6d015cf60 2023-10-16T14:17:31.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 -- 172.21.15.29:0/2879826775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 msgr2=0x7ff6d006f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:31.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 --2- 172.21.15.29:0/2879826775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d006f490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00098a0 tx=0x7ff6c002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 -- 172.21.15.29:0/2879826775 shutdown_connections 2023-10-16T14:17:31.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 --2- 172.21.15.29:0/2879826775 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 --2- 172.21.15.29:0/2879826775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d006f490 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 -- 172.21.15.29:0/2879826775 >> 172.21.15.29:0/2879826775 conn(0x7ff6d00cb360 msgr2=0x7ff6d00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:31.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 -- 172.21.15.29:0/2879826775 shutdown_connections 2023-10-16T14:17:31.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.350+0000 7ff6d576c640 1 -- 172.21.15.29:0/2879826775 wait complete. 2023-10-16T14:17:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.353+0000 7ff6d576c640 1 Processor -- start 2023-10-16T14:17:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.353+0000 7ff6d576c640 1 -- start start 2023-10-16T14:17:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.354+0000 7ff6d576c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d006f9d0 0x7ff6d0154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:31.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.354+0000 7ff6d576c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.354+0000 7ff6d576c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d00c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.354+0000 7ff6d576c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00b2e00 con 0x7ff6d015cf60 2023-10-16T14:17:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.354+0000 7ff6d576c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00c4f70 con 0x7ff6d006f9d0 2023-10-16T14:17:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.354+0000 7ff6d576c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00b2840 con 0x7ff6d00b0360 2023-10-16T14:17:31.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d00c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:31.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d00c68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47434/0 (socket says 172.21.15.29:47434) 2023-10-16T14:17:31.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 -- 172.21.15.29:0/3556504637 learned_addr learned my addr 172.21.15.29:0/3556504637 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:31.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6ce7fc640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:31.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6ceffd640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d006f9d0 0x7ff6d0154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:31.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 -- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 msgr2=0x7ff6d00b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:31.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 -- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d006f9d0 msgr2=0x7ff6d0154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:31.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d006f9d0 0x7ff6d0154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:31.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.355+0000 7ff6cf7fe640 1 -- 172.21.15.29:0/3556504637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00094d0 con 0x7ff6d015cf60 2023-10-16T14:17:31.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.356+0000 7ff6cf7fe640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d00c68c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800b970 tx=0x7ff6c800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:31.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.356+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800cb40 con 0x7ff6d015cf60 2023-10-16T14:17:31.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.356+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c800ccc0 con 0x7ff6d015cf60 2023-10-16T14:17:31.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.356+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8014680 con 0x7ff6d015cf60 2023-10-16T14:17:31.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.356+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d00b2b20 con 0x7ff6d015cf60 2023-10-16T14:17:31.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.356+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d00c2820 con 0x7ff6d015cf60 2023-10-16T14:17:31.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.357+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d00c2a10 con 0x7ff6d015cf60 2023-10-16T14:17:31.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.358+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff6c801d3b0 con 0x7ff6d015cf60 2023-10-16T14:17:31.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.359+0000 7ff6b77fe640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff6bc06aef0 0x7ff6bc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:31.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.359+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5531+0+0 (secure 0 0 0) 0x7ff6c8089460 con 0x7ff6d015cf60 2023-10-16T14:17:31.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.360+0000 7ff6ceffd640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff6bc06aef0 0x7ff6bc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:31.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.361+0000 7ff6ceffd640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff6bc06aef0 0x7ff6bc06d3b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0036000 tx=0x7ff6c003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:31.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.364+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6c8054630 con 0x7ff6d015cf60 2023-10-16T14:17:31.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:31.557+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff6d004e160 con 0x7ff6d015cf60 2023-10-16T14:17:32.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.098+0000 7ff6b77fe640 1 -- 172.21.15.29:0/3556504637 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff6c8057c50 con 0x7ff6d015cf60 2023-10-16T14:17:32.103 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:17:32.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.101+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff6bc06aef0 msgr2=0x7ff6bc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.101+0000 7ff6d576c640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff6bc06aef0 0x7ff6bc06d3b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0036000 tx=0x7ff6c003c040 comp rx=0 tx=0).stop 2023-10-16T14:17:32.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.101+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 msgr2=0x7ff6d00c68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.101+0000 7ff6d576c640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d00c68c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800b970 tx=0x7ff6c800be30 comp rx=0 tx=0).stop 2023-10-16T14:17:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 shutdown_connections 2023-10-16T14:17:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff6bc06aef0 0x7ff6bc06d3b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d015cf60 0x7ff6d00c68c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d00b0360 0x7ff6d00b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 --2- 172.21.15.29:0/3556504637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d006f9d0 0x7ff6d0154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 >> 172.21.15.29:0/3556504637 conn(0x7ff6d00cb360 msgr2=0x7ff6d006feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 shutdown_connections 2023-10-16T14:17:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.102+0000 7ff6d576c640 1 -- 172.21.15.29:0/3556504637 wait complete. 2023-10-16T14:17:32.133 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:17:32.134 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:17:32.134 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:17:32.134 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:17:32.135 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:17:32.135 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:17:32.135 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:17:32.136 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:17:32.136 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:17:32.136 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:17:32.137 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:17:32.137 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:17:32.137 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:17:32.138 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:17:32.138 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:17:32.138 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:17:32.152 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:17:32.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:17:32.154 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:17:32.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:17:32.156 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:17:32.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:17:32.158 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:17:32.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:17:32.201 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:17:32.202 DEBUG:teuthology.parallel:result is None 2023-10-16T14:17:32.205 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:17:32.206 DEBUG:teuthology.parallel:result is None 2023-10-16T14:17:32.208 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:17:32.208 DEBUG:teuthology.parallel:result is None 2023-10-16T14:17:32.213 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:17:32.213 DEBUG:teuthology.parallel:result is None 2023-10-16T14:17:32.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:17:32.365 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:17:32.366 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:17:32.368 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:17:32.369 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:17:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.399+0000 7faf29d01640 1 Processor -- start 2023-10-16T14:17:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.399+0000 7faf29d01640 1 -- start start 2023-10-16T14:17:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf29d01640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faf240ad350 0x7faf24069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf29d01640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2406a730 con 0x7faf2406a2e0 2023-10-16T14:17:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf29d01640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf240ac890 con 0x7faf2415d150 2023-10-16T14:17:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf29d01640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2415d530 con 0x7faf240ad350 2023-10-16T14:17:32.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf1bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faf240ad350 0x7faf24069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:32.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf23fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faf2406a2e0 0x7faf240c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47366/0 (socket says 172.21.15.29:47366) 2023-10-16T14:17:32.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf23fff640 1 -- 172.21.15.29:0/1694878545 learned_addr learned my addr 172.21.15.29:0/1694878545 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:32.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf1bfff640 1 -- 172.21.15.29:0/1694878545 >> v1:172.21.15.170:6789/0 conn(0x7faf2415d150 legacy=0x7faf24160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf1bfff640 1 -- 172.21.15.29:0/1694878545 >> v1:172.21.15.29:6789/0 conn(0x7faf2406a2e0 legacy=0x7faf240c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf1bfff640 1 -- 172.21.15.29:0/1694878545 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240ad7a0 con 0x7faf240ad350 2023-10-16T14:17:32.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.401+0000 7faf22ffd640 1 -- 172.21.15.29:0/1694878545 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470710391 0 0) 0x7faf1c001490 con 0x7faf2406a2e0 2023-10-16T14:17:32.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.403+0000 7faf1bfff640 1 --2- 172.21.15.29:0/1694878545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf240ad350 0x7faf24069da0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0097a0 tx=0x7faf0c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a54dca20988f00 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.403+0000 7faf22ffd640 1 -- 172.21.15.29:0/1694878545 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c030610 con 0x7faf240ad350 2023-10-16T14:17:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.403+0000 7faf22ffd640 1 -- 172.21.15.29:0/1694878545 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c030770 con 0x7faf240ad350 2023-10-16T14:17:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.403+0000 7faf22ffd640 1 -- 172.21.15.29:0/1694878545 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c030b00 con 0x7faf240ad350 2023-10-16T14:17:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.403+0000 7faf29d01640 1 -- 172.21.15.29:0/1694878545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf240ad350 msgr2=0x7faf24069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.403+0000 7faf29d01640 1 --2- 172.21.15.29:0/1694878545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf240ad350 0x7faf24069da0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0097a0 tx=0x7faf0c02b960 comp rx=0 tx=0).stop 2023-10-16T14:17:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.404+0000 7faf29d01640 1 -- 172.21.15.29:0/1694878545 shutdown_connections 2023-10-16T14:17:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.404+0000 7faf29d01640 1 --2- 172.21.15.29:0/1694878545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf240ad350 0x7faf24069da0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.404+0000 7faf29d01640 1 -- 172.21.15.29:0/1694878545 >> 172.21.15.29:0/1694878545 conn(0x7faf240ce9b0 msgr2=0x7faf240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.404+0000 7faf29d01640 1 -- 172.21.15.29:0/1694878545 shutdown_connections 2023-10-16T14:17:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.404+0000 7faf29d01640 1 -- 172.21.15.29:0/1694878545 wait complete. 2023-10-16T14:17:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.405+0000 7faf29d01640 1 Processor -- start 2023-10-16T14:17:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 -- start start 2023-10-16T14:17:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf2406a2e0 0x7faf2415e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 0x7faf240c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2415d150 0x7faf2415b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2415ec60 con 0x7faf240ad350 2023-10-16T14:17:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2415c5a0 con 0x7faf2415d150 2023-10-16T14:17:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.406+0000 7faf29d01640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2415ee70 con 0x7faf2406a2e0 2023-10-16T14:17:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 0x7faf240c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 0x7faf240c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47472/0 (socket says 172.21.15.29:47472) 2023-10-16T14:17:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 -- 172.21.15.29:0/3741119001 learned_addr learned my addr 172.21.15.29:0/3741119001 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf237fe640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf2406a2e0 0x7faf2415e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf23fff640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2415d150 0x7faf2415b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 -- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf2406a2e0 msgr2=0x7faf2415e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf2406a2e0 0x7faf2415e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 -- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2415d150 msgr2=0x7faf2415b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:32.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2415d150 0x7faf2415b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:32.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.407+0000 7faf1bfff640 1 -- 172.21.15.29:0/3741119001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c009400 con 0x7faf240ad350 2023-10-16T14:17:32.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.408+0000 7faf1bfff640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 0x7faf240c24e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0098c0 tx=0x7faf0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:32.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.408+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c03e040 con 0x7faf240ad350 2023-10-16T14:17:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.408+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c039cb0 con 0x7faf240ad350 2023-10-16T14:17:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.408+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c044630 con 0x7faf240ad350 2023-10-16T14:17:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.408+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf240c2a20 con 0x7faf240ad350 2023-10-16T14:17:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.408+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf240b2d40 con 0x7faf240ad350 2023-10-16T14:17:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.409+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2404e160 con 0x7faf240ad350 2023-10-16T14:17:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.410+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faf0c043390 con 0x7faf240ad350 2023-10-16T14:17:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.411+0000 7faf20ff9640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7faf0406ae70 0x7faf0406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.411+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7faf0c0ab650 con 0x7faf240ad350 2023-10-16T14:17:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.411+0000 7faf237fe640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7faf0406ae70 0x7faf0406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.412+0000 7faf237fe640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7faf0406ae70 0x7faf0406d330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faf240cb370 tx=0x7faf14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:32.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.415+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faf0c076a10 con 0x7faf240ad350 2023-10-16T14:17:32.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:32.631+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faf240ace20 con 0x7faf240ad350 2023-10-16T14:17:33.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.119+0000 7faf20ff9640 1 -- 172.21.15.29:0/3741119001 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf0c07a030 con 0x7faf240ad350 2023-10-16T14:17:33.124 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:17:33.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.123+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7faf0406ae70 msgr2=0x7faf0406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.123+0000 7faf29d01640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7faf0406ae70 0x7faf0406d330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faf240cb370 tx=0x7faf14009040 comp rx=0 tx=0).stop 2023-10-16T14:17:33.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.123+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 msgr2=0x7faf240c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.123+0000 7faf29d01640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 0x7faf240c24e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0098c0 tx=0x7faf0c004730 comp rx=0 tx=0).stop 2023-10-16T14:17:33.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 shutdown_connections 2023-10-16T14:17:33.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7faf0406ae70 0x7faf0406d330 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf240ad350 0x7faf240c24e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf2415d150 0x7faf2415b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 --2- 172.21.15.29:0/3741119001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf2406a2e0 0x7faf2415e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 >> 172.21.15.29:0/3741119001 conn(0x7faf240ce9b0 msgr2=0x7faf2406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 shutdown_connections 2023-10-16T14:17:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.124+0000 7faf29d01640 1 -- 172.21.15.29:0/3741119001 wait complete. 2023-10-16T14:17:33.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:17:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.367+0000 7f39ac41a640 1 Processor -- start 2023-10-16T14:17:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.367+0000 7f39ac41a640 1 -- start start 2023-10-16T14:17:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.368+0000 7f39ac41a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f39a415aaa0 0x7f39a415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39ac41a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39a40ce390 0x7f39a4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39ac41a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a415ed60 con 0x7f39a415ae80 2023-10-16T14:17:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39ac41a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a406b810 con 0x7f39a40ce390 2023-10-16T14:17:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39ac41a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40707e0 con 0x7f39a415aaa0 2023-10-16T14:17:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39a40ce390 0x7f39a4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39a40ce390 0x7f39a4070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47488/0 (socket says 172.21.15.29:47488) 2023-10-16T14:17:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1335168305 learned_addr learned my addr 172.21.15.29:0/1335168305 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa1b6640 1 --2- 172.21.15.29:0/1335168305 >> v2:172.21.15.170:3300/0 conn(0x7f39a415aaa0 0x7f39a415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1335168305 >> v2:172.21.15.170:3300/0 conn(0x7f39a415aaa0 msgr2=0x7f39a415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 --2- 172.21.15.29:0/1335168305 >> v2:172.21.15.170:3300/0 conn(0x7f39a415aaa0 0x7f39a415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1335168305 >> v1:172.21.15.170:6789/0 conn(0x7f39a415ae80 legacy=0x7f39a40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.369+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1335168305 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a415ef90 con 0x7f39a40ce390 2023-10-16T14:17:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.371+0000 7f39aa9b7640 1 --2- 172.21.15.29:0/1335168305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a40ce390 0x7f39a4070f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f39a40cd990 tx=0x7f399c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d7521e9f450906 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.371+0000 7f39a91b4640 1 -- 172.21.15.29:0/1335168305 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c02ca80 con 0x7f39a40ce390 2023-10-16T14:17:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.371+0000 7f39a91b4640 1 -- 172.21.15.29:0/1335168305 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c02cbe0 con 0x7f39a40ce390 2023-10-16T14:17:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.371+0000 7f39a91b4640 1 -- 172.21.15.29:0/1335168305 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c0343f0 con 0x7f39a40ce390 2023-10-16T14:17:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 -- 172.21.15.29:0/1335168305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a40ce390 msgr2=0x7f39a4070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1335168305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a40ce390 0x7f39a4070f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f39a40cd990 tx=0x7f399c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 -- 172.21.15.29:0/1335168305 shutdown_connections 2023-10-16T14:17:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1335168305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a40ce390 0x7f39a4070f50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1335168305 >> v2:172.21.15.170:3300/0 conn(0x7f39a415aaa0 0x7f39a415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 -- 172.21.15.29:0/1335168305 >> 172.21.15.29:0/1335168305 conn(0x7f39a40cb410 msgr2=0x7f39a40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 -- 172.21.15.29:0/1335168305 shutdown_connections 2023-10-16T14:17:33.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.372+0000 7f39ac41a640 1 -- 172.21.15.29:0/1335168305 wait complete. 2023-10-16T14:17:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.373+0000 7f39ac41a640 1 Processor -- start 2023-10-16T14:17:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.374+0000 7f39ac41a640 1 -- start start 2023-10-16T14:17:33.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.374+0000 7f39ac41a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39a40ce390 0x7f39a415b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.374+0000 7f39ac41a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a415aaa0 0x7f39a415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39ac41a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 0x7f39a40c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39ac41a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a415bc00 con 0x7f39a415ae80 2023-10-16T14:17:33.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39ac41a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40af820 con 0x7f39a40ce390 2023-10-16T14:17:33.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39ac41a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40afb00 con 0x7f39a415aaa0 2023-10-16T14:17:33.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 0x7f39a40c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 0x7f39a40c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47498/0 (socket says 172.21.15.29:47498) 2023-10-16T14:17:33.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1659813196 learned_addr learned my addr 172.21.15.29:0/1659813196 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:33.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa1b6640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39a40ce390 0x7f39a415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39a99b5640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a415aaa0 0x7f39a415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a415aaa0 msgr2=0x7f39a415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a415aaa0 0x7f39a415c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39a40ce390 msgr2=0x7f39a415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39a40ce390 0x7f39a415b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39aa9b7640 1 -- 172.21.15.29:0/1659813196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c009b30 con 0x7f39a415ae80 2023-10-16T14:17:33.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.375+0000 7f39a99b5640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a415aaa0 0x7f39a415c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:33.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.376+0000 7f39aa9b7640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 0x7f39a40c1c50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f399c036040 tx=0x7f399c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:33.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.376+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c009510 con 0x7f39a415ae80 2023-10-16T14:17:33.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.376+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c03e450 con 0x7f39a415ae80 2023-10-16T14:17:33.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.376+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c03e7e0 con 0x7f39a415ae80 2023-10-16T14:17:33.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.376+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a40afde0 con 0x7f39a415ae80 2023-10-16T14:17:33.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.376+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a40b03a0 con 0x7f39a415ae80 2023-10-16T14:17:33.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.377+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a404e160 con 0x7f39a415ae80 2023-10-16T14:17:33.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.378+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f399c02cd50 con 0x7f39a415ae80 2023-10-16T14:17:33.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.379+0000 7f3996ffd640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f398806ae70 0x7f398806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.379+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f399c0abe30 con 0x7f39a415ae80 2023-10-16T14:17:33.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.380+0000 7f39aa1b6640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f398806ae70 0x7f398806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.381+0000 7f39aa1b6640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f398806ae70 0x7f398806d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f39a0006fd0 tx=0x7f39a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:33.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.384+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f399c077220 con 0x7f39a415ae80 2023-10-16T14:17:33.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.647+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f39a415e350 con 0x7f39a415ae80 2023-10-16T14:17:33.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.648+0000 7f3996ffd640 1 -- 172.21.15.29:0/1659813196 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f399c03e940 con 0x7f39a415ae80 2023-10-16T14:17:33.652 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:33.653 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:17:33.653 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:17:33.653 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.653 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:17:33.654 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:17:33.654 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.654 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.655 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:17:33.655 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:17:33.655 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.656 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.656 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:17:33.656 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:17:33.656 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.656 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.657 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:17:33.657 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:17:33.657 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.657 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.658 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.658 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.658 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:17:33.658 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:17:33.658 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.659 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.659 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:17:33.659 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:17:33.659 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.659 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.660 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:17:33.660 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:17:33.660 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.661 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.661 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:17:33.661 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:17:33.661 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.662 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.662 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:17:33.663 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:17:33.663 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.663 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.664 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.664 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.665 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:17:33.665 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:17:33.665 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.666 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.666 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.666 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.667 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:17:33.667 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:17:33.667 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.668 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.668 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.668 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.668 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.669 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:17:33.669 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:17:33.669 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.670 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.670 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.670 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.670 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.671 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:17:33.671 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:17:33.672 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.672 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.672 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.672 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.672 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.673 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:17:33.673 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:17:33.674 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.674 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.674 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.674 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.674 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.675 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.675 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:17:33.675 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:17:33.675 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.676 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.676 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.676 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.677 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:17:33.677 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:17:33.677 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.678 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:17:33.678 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:17:33.678 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.678 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.679 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:17:33.679 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:17:33.679 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:17:33.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:17:33.680 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:17:33.681 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.681 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:17:33.682 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:17:33.682 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:17:33.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:17:33.683 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:17:33.683 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.683 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:17:33.684 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:17:33.684 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:17:33.685 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:17:33.685 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:17:33.685 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.686 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:17:33.686 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:17:33.686 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.686 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:17:33.687 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:17:33.687 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:17:33.687 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:17:33.687 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.688 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:17:33.688 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:17:33.688 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.688 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:17:33.689 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:17:33.689 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:17:33.689 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.689 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.690 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-mds", 2023-10-16T14:17:33.690 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCkRS1lBG/QMBAAfRlht1OsWJV/F59tWjjPhA==", 2023-10-16T14:17:33.690 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.691 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-mds" 2023-10-16T14:17:33.691 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.691 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-mgr", 2023-10-16T14:17:33.692 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCkRS1lP43QMBAA48UquJMs6GOVfcVE18k2ow==", 2023-10-16T14:17:33.692 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.692 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-16T14:17:33.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.693 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-osd", 2023-10-16T14:17:33.693 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCkRS1llqbQMBAAASUMVhTN2GIayOXxX2rXNA==", 2023-10-16T14:17:33.693 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.693 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-osd" 2023-10-16T14:17:33.694 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.694 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rbd", 2023-10-16T14:17:33.694 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCkRS1lK7/QMBAAEiTMlMVkKLYoHGCU9G/2Ng==", 2023-10-16T14:17:33.695 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.695 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-16T14:17:33.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.696 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-16T14:17:33.696 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCkRS1l9dfQMBAAM3m0zU6HekAn5zeeiZ+3Iw==", 2023-10-16T14:17:33.696 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.696 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-16T14:17:33.697 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.697 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rgw", 2023-10-16T14:17:33.698 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCkRS1lcfHQMBAAsqFkMTERI5stvLmktEXGhw==", 2023-10-16T14:17:33.698 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.698 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-16T14:17:33.698 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.699 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:17:33.699 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:17:33.699 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:17:33.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:17:33.700 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.700 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:17:33.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:33.701 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:17:33.701 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:17:33.701 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:17:33.701 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:17:33.702 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:17:33.702 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:17:33.702 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.702 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:33.703 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:17:33.703 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:17:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.650+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f398806ae70 msgr2=0x7f398806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f398806ae70 0x7f398806d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f39a0006fd0 tx=0x7f39a0008040 comp rx=0 tx=0).stop 2023-10-16T14:17:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 msgr2=0x7f39a40c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 0x7f39a40c1c50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f399c036040 tx=0x7f399c004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 shutdown_connections 2023-10-16T14:17:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f398806ae70 0x7f398806d330 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a415ae80 0x7f39a40c1c50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a415aaa0 0x7f39a415c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 --2- 172.21.15.29:0/1659813196 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39a40ce390 0x7f39a415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 >> 172.21.15.29:0/1659813196 conn(0x7f39a40cb410 msgr2=0x7f39a40b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 shutdown_connections 2023-10-16T14:17:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.651+0000 7f39ac41a640 1 -- 172.21.15.29:0/1659813196 wait complete. 2023-10-16T14:17:33.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-16T14:17:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.936+0000 7f1a52cf2640 1 Processor -- start 2023-10-16T14:17:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.936+0000 7f1a52cf2640 1 -- start start 2023-10-16T14:17:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.937+0000 7f1a52cf2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159e90 0x7f1a4c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.937+0000 7f1a52cf2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a4c06c140 0x7f1a4c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.937+0000 7f1a52cf2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c070360 con 0x7f1a4c159ab0 2023-10-16T14:17:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.937+0000 7f1a52cf2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c06cb10 con 0x7f1a4c06c140 2023-10-16T14:17:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.937+0000 7f1a52cf2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c15d890 con 0x7f1a4c159e90 2023-10-16T14:17:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.937+0000 7f1a5128f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a4c06c140 0x7f1a4c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a5128f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a4c06c140 0x7f1a4c06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56278/0 (socket says 172.21.15.29:56278) 2023-10-16T14:17:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a5128f640 1 -- 172.21.15.29:0/2524886681 learned_addr learned my addr 172.21.15.29:0/2524886681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a4bfff640 1 --2- 172.21.15.29:0/2524886681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159e90 0x7f1a4c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a5128f640 1 -- 172.21.15.29:0/2524886681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159e90 msgr2=0x7f1a4c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a5128f640 1 --2- 172.21.15.29:0/2524886681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159e90 0x7f1a4c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a5128f640 1 -- 172.21.15.29:0/2524886681 >> v1:172.21.15.170:6789/0 conn(0x7f1a4c159ab0 legacy=0x7f1a4c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.938+0000 7f1a5128f640 1 -- 172.21.15.29:0/2524886681 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0cd1f0 con 0x7f1a4c06c140 2023-10-16T14:17:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.939+0000 7f1a5128f640 1 --2- 172.21.15.29:0/2524886681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c06c140 0x7f1a4c06c5d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1a440098a0 tx=0x7f1a4402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c4736211d84e8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.939+0000 7f1a4b7fe640 1 -- 172.21.15.29:0/2524886681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44031520 con 0x7f1a4c06c140 2023-10-16T14:17:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.939+0000 7f1a4b7fe640 1 -- 172.21.15.29:0/2524886681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44031680 con 0x7f1a4c06c140 2023-10-16T14:17:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.939+0000 7f1a4b7fe640 1 -- 172.21.15.29:0/2524886681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44031a10 con 0x7f1a4c06c140 2023-10-16T14:17:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 -- 172.21.15.29:0/2524886681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c06c140 msgr2=0x7f1a4c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/2524886681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c06c140 0x7f1a4c06c5d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1a440098a0 tx=0x7f1a4402b7c0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 -- 172.21.15.29:0/2524886681 shutdown_connections 2023-10-16T14:17:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/2524886681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c06c140 0x7f1a4c06c5d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/2524886681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159e90 0x7f1a4c0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 -- 172.21.15.29:0/2524886681 >> 172.21.15.29:0/2524886681 conn(0x7f1a4c0ce9b0 msgr2=0x7f1a4c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 -- 172.21.15.29:0/2524886681 shutdown_connections 2023-10-16T14:17:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.940+0000 7f1a52cf2640 1 -- 172.21.15.29:0/2524886681 wait complete. 2023-10-16T14:17:33.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.942+0000 7f1a52cf2640 1 Processor -- start 2023-10-16T14:17:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.942+0000 7f1a52cf2640 1 -- start start 2023-10-16T14:17:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a52cf2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a4c06c140 0x7f1a4c14d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a52cf2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159ab0 0x7f1a4c0c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a52cf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 0x7f1a4c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a52cf2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c14de80 con 0x7f1a4c159e90 2023-10-16T14:17:33.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a52cf2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0c3090 con 0x7f1a4c06c140 2023-10-16T14:17:33.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a52cf2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0b15d0 con 0x7f1a4c159ab0 2023-10-16T14:17:33.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a5128f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 0x7f1a4c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a5128f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 0x7f1a4c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56292/0 (socket says 172.21.15.29:56292) 2023-10-16T14:17:33.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.943+0000 7f1a5128f640 1 -- 172.21.15.29:0/1795590791 learned_addr learned my addr 172.21.15.29:0/1795590791 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:33.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a50a8e640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a4c06c140 0x7f1a4c14d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a4bfff640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159ab0 0x7f1a4c0c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a5128f640 1 -- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159ab0 msgr2=0x7f1a4c0c2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a5128f640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159ab0 0x7f1a4c0c2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a5128f640 1 -- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a4c06c140 msgr2=0x7f1a4c14d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:33.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a5128f640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a4c06c140 0x7f1a4c14d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:33.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a5128f640 1 -- 172.21.15.29:0/1795590791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a440094d0 con 0x7f1a4c159e90 2023-10-16T14:17:33.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a4bfff640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159ab0 0x7f1a4c0c2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:33.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a5128f640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 0x7f1a4c0ac890 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1a440090b0 tx=0x7f1a440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:33.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.944+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44031520 con 0x7f1a4c159e90 2023-10-16T14:17:33.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.945+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44033bd0 con 0x7f1a4c159e90 2023-10-16T14:17:33.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.945+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c167b40 con 0x7f1a4c159e90 2023-10-16T14:17:33.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.945+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4403c400 con 0x7f1a4c159e90 2023-10-16T14:17:33.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.945+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c0b1b70 con 0x7f1a4c159e90 2023-10-16T14:17:33.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.946+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4c04e160 con 0x7f1a4c159e90 2023-10-16T14:17:33.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.947+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1a4403a040 con 0x7f1a4c159e90 2023-10-16T14:17:33.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.948+0000 7f1a497fa640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a3806aea0 0x7f1a3806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:33.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.948+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1a440aaaf0 con 0x7f1a4c159e90 2023-10-16T14:17:33.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.948+0000 7f1a50a8e640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a3806aea0 0x7f1a3806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:33.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.949+0000 7f1a50a8e640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a3806aea0 0x7f1a3806d360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009f20 tx=0x7f1a3c0094d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:33.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:33.952+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a44075e50 con 0x7f1a4c159e90 2023-10-16T14:17:34.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.220+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1a4c0b01e0 con 0x7f1a4c159e90 2023-10-16T14:17:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.225+0000 7f1a497fa640 1 -- 172.21.15.29:0/1795590791 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1a4404d0a0 con 0x7f1a4c159e90 2023-10-16T14:17:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a3806aea0 msgr2=0x7f1a3806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a3806aea0 0x7f1a3806d360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009f20 tx=0x7f1a3c0094d0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 msgr2=0x7f1a4c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 0x7f1a4c0ac890 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1a440090b0 tx=0x7f1a440046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 shutdown_connections 2023-10-16T14:17:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1a3806aea0 0x7f1a3806d360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a4c159e90 0x7f1a4c0ac890 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a4c159ab0 0x7f1a4c0c2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 --2- 172.21.15.29:0/1795590791 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a4c06c140 0x7f1a4c14d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 >> 172.21.15.29:0/1795590791 conn(0x7f1a4c0ce9b0 msgr2=0x7f1a4c0b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 shutdown_connections 2023-10-16T14:17:34.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.228+0000 7f1a52cf2640 1 -- 172.21.15.29:0/1795590791 wait complete. 2023-10-16T14:17:34.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-16T14:17:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.474+0000 7f7b05568640 1 Processor -- start 2023-10-16T14:17:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.474+0000 7f7b05568640 1 -- start start 2023-10-16T14:17:34.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7b05568640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b0015d150 0x7f7b000ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7b05568640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7b00160b90 0x7f7b00160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7b05568640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b00069da0 0x7f7b0006a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7b05568640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0015d530 con 0x7f7b0015d150 2023-10-16T14:17:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7b05568640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0006a730 con 0x7f7b00160b90 2023-10-16T14:17:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7b05568640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b000ac890 con 0x7f7b00069da0 2023-10-16T14:17:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7afeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b0015d150 0x7f7b000ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7afeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b0015d150 0x7f7b000ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56298/0 (socket says 172.21.15.29:56298) 2023-10-16T14:17:34.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.476+0000 7f7afeffd640 1 -- 172.21.15.29:0/4135489291 learned_addr learned my addr 172.21.15.29:0/4135489291 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7af7fff640 1 --2- 172.21.15.29:0/4135489291 >> v2:172.21.15.170:3300/0 conn(0x7f7b00160b90 0x7f7b00160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7aff7fe640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b00069da0 0x7f7b0006a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7afeffd640 1 -- 172.21.15.29:0/4135489291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b00069da0 msgr2=0x7f7b0006a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7afeffd640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b00069da0 0x7f7b0006a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7afeffd640 1 -- 172.21.15.29:0/4135489291 >> v2:172.21.15.170:3300/0 conn(0x7f7b00160b90 msgr2=0x7f7b00160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7afeffd640 1 --2- 172.21.15.29:0/4135489291 >> v2:172.21.15.170:3300/0 conn(0x7f7b00160b90 0x7f7b00160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7afeffd640 1 -- 172.21.15.29:0/4135489291 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00161650 con 0x7f7b0015d150 2023-10-16T14:17:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.477+0000 7f7aff7fe640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b00069da0 0x7f7b0006a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.478+0000 7f7afeffd640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0015d150 0x7f7b000ad350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7b000cf990 tx=0x7f7af002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74aa1110c00fab4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.478+0000 7f7afe7fc640 1 -- 172.21.15.29:0/4135489291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af00317f0 con 0x7f7b0015d150 2023-10-16T14:17:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.478+0000 7f7afe7fc640 1 -- 172.21.15.29:0/4135489291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0031950 con 0x7f7b0015d150 2023-10-16T14:17:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.478+0000 7f7afe7fc640 1 -- 172.21.15.29:0/4135489291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0031ce0 con 0x7f7b0015d150 2023-10-16T14:17:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 -- 172.21.15.29:0/4135489291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0015d150 msgr2=0x7f7b000ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0015d150 0x7f7b000ad350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7b000cf990 tx=0x7f7af002b890 comp rx=0 tx=0).stop 2023-10-16T14:17:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 -- 172.21.15.29:0/4135489291 shutdown_connections 2023-10-16T14:17:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b00069da0 0x7f7b0006a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 --2- 172.21.15.29:0/4135489291 >> v2:172.21.15.170:3300/0 conn(0x7f7b00160b90 0x7f7b00160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 --2- 172.21.15.29:0/4135489291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b0015d150 0x7f7b000ad350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 -- 172.21.15.29:0/4135489291 >> 172.21.15.29:0/4135489291 conn(0x7f7b000ce9b0 msgr2=0x7f7b000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 -- 172.21.15.29:0/4135489291 shutdown_connections 2023-10-16T14:17:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.479+0000 7f7b05568640 1 -- 172.21.15.29:0/4135489291 wait complete. 2023-10-16T14:17:34.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.481+0000 7f7b05568640 1 Processor -- start 2023-10-16T14:17:34.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.481+0000 7f7b05568640 1 -- start start 2023-10-16T14:17:34.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.482+0000 7f7b05568640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b00069da0 0x7f7b00158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.482+0000 7f7b05568640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b0015d150 0x7f7b000b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.482+0000 7f7b05568640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 0x7f7b000b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.482+0000 7f7b05568640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0015e390 con 0x7f7b00160b90 2023-10-16T14:17:34.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.482+0000 7f7b05568640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0015d7c0 con 0x7f7b00069da0 2023-10-16T14:17:34.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.482+0000 7f7b05568640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00159270 con 0x7f7b0015d150 2023-10-16T14:17:34.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 0x7f7b000b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 0x7f7b000b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56306/0 (socket says 172.21.15.29:56306) 2023-10-16T14:17:34.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 -- 172.21.15.29:0/199502658 learned_addr learned my addr 172.21.15.29:0/199502658 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:34.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7afeffd640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b00069da0 0x7f7b00158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7af7fff640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b0015d150 0x7f7b000b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 -- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b0015d150 msgr2=0x7f7b000b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b0015d150 0x7f7b000b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 -- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b00069da0 msgr2=0x7f7b00158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b00069da0 0x7f7b00158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 -- 172.21.15.29:0/199502658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00094d0 con 0x7f7b00160b90 2023-10-16T14:17:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7afeffd640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b00069da0 0x7f7b00158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.483+0000 7f7aff7fe640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 0x7f7b000b2d10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7af800cd10 tx=0x7f7af800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.484+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af800f680 con 0x7f7b00160b90 2023-10-16T14:17:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.484+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af800f800 con 0x7f7b00160b90 2023-10-16T14:17:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.484+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af800fbb0 con 0x7f7b00160b90 2023-10-16T14:17:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.484+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0015da60 con 0x7f7b00160b90 2023-10-16T14:17:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.484+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0015e020 con 0x7f7b00160b90 2023-10-16T14:17:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.485+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0004e160 con 0x7f7b00160b90 2023-10-16T14:17:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.486+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7af800b040 con 0x7f7b00160b90 2023-10-16T14:17:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.488+0000 7f7af77fe640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ae006aea0 0x7f7ae006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.488+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7af8089c90 con 0x7f7b00160b90 2023-10-16T14:17:34.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.488+0000 7f7afeffd640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ae006aea0 0x7f7ae006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:34.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.489+0000 7f7afeffd640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ae006aea0 0x7f7ae006d360 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7af0036000 tx=0x7f7af003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:34.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.492+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7af8055080 con 0x7f7b00160b90 2023-10-16T14:17:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.756+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f7b000ade10 con 0x7f7b00160b90 2023-10-16T14:17:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.761+0000 7f7af77fe640 1 -- 172.21.15.29:0/199502658 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7af801d0a0 con 0x7f7b00160b90 2023-10-16T14:17:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.764+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ae006aea0 msgr2=0x7f7ae006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.764+0000 7f7b05568640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ae006aea0 0x7f7ae006d360 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7af0036000 tx=0x7f7af003d040 comp rx=0 tx=0).stop 2023-10-16T14:17:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.764+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 msgr2=0x7f7b000b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.764+0000 7f7b05568640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 0x7f7b000b2d10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7af800cd10 tx=0x7f7af800a650 comp rx=0 tx=0).stop 2023-10-16T14:17:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 shutdown_connections 2023-10-16T14:17:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f7ae006aea0 0x7f7ae006d360 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b00160b90 0x7f7b000b2d10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b0015d150 0x7f7b000b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 --2- 172.21.15.29:0/199502658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b00069da0 0x7f7b00158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 >> 172.21.15.29:0/199502658 conn(0x7f7b000ce9b0 msgr2=0x7f7b000c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 shutdown_connections 2023-10-16T14:17:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:34.765+0000 7f7b05568640 1 -- 172.21.15.29:0/199502658 wait complete. 2023-10-16T14:17:34.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-16T14:17:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.014+0000 7fcf6759e640 1 Processor -- start 2023-10-16T14:17:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.014+0000 7fcf6759e640 1 -- start start 2023-10-16T14:17:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.015+0000 7fcf6759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815dde0 0x7fcf680709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.015+0000 7fcf6759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcf68070f20 0x7fcf680ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.015+0000 7fcf6759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf68160a90 con 0x7fcf6815c9f0 2023-10-16T14:17:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.015+0000 7fcf6759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf680cdec0 con 0x7fcf68070f20 2023-10-16T14:17:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.015+0000 7fcf6759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680cd900 con 0x7fcf6815dde0 2023-10-16T14:17:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.016+0000 7fcf6659c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fcf6815c9f0 0x7fcf680c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:41568/0 (socket says 172.21.15.29:41568) 2023-10-16T14:17:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.016+0000 7fcf6659c640 1 -- 172.21.15.29:0/3914693139 learned_addr learned my addr 172.21.15.29:0/3914693139 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.016+0000 7fcf65d9b640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815dde0 0x7fcf680709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.016+0000 7fcf66d9d640 1 --2- 172.21.15.29:0/3914693139 >> v2:172.21.15.170:3300/0 conn(0x7fcf68070f20 0x7fcf680ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf6559a640 1 -- 172.21.15.29:0/3914693139 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772023890 0 0) 0x7fcf68160a90 con 0x7fcf6815c9f0 2023-10-16T14:17:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf6559a640 1 -- 172.21.15.29:0/3914693139 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf50003520 con 0x7fcf6815c9f0 2023-10-16T14:17:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf66d9d640 1 -- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815dde0 msgr2=0x7fcf680709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf66d9d640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815dde0 0x7fcf680709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf66d9d640 1 -- 172.21.15.29:0/3914693139 >> v1:172.21.15.170:6789/0 conn(0x7fcf6815c9f0 legacy=0x7fcf680c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf66d9d640 1 -- 172.21.15.29:0/3914693139 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf680cdbe0 con 0x7fcf68070f20 2023-10-16T14:17:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.017+0000 7fcf65d9b640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815dde0 0x7fcf680709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:17:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.018+0000 7fcf66d9d640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf680ac890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009d90 tx=0x7fcf6002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e8527b955d3c5db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.019+0000 7fcf6559a640 1 -- 172.21.15.29:0/3914693139 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6002cc50 con 0x7fcf68070f20 2023-10-16T14:17:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.019+0000 7fcf6559a640 1 -- 172.21.15.29:0/3914693139 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6002cdb0 con 0x7fcf68070f20 2023-10-16T14:17:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.019+0000 7fcf6559a640 1 -- 172.21.15.29:0/3914693139 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf60034660 con 0x7fcf68070f20 2023-10-16T14:17:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.019+0000 7fcf6759e640 1 -- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 msgr2=0x7fcf680ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.019+0000 7fcf6759e640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf680ac890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009d90 tx=0x7fcf6002bea0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.020+0000 7fcf6759e640 1 -- 172.21.15.29:0/3914693139 shutdown_connections 2023-10-16T14:17:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.020+0000 7fcf6759e640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf680ac890 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.020+0000 7fcf6759e640 1 --2- 172.21.15.29:0/3914693139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815dde0 0x7fcf680709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.020+0000 7fcf6759e640 1 -- 172.21.15.29:0/3914693139 >> 172.21.15.29:0/3914693139 conn(0x7fcf680ce6a0 msgr2=0x7fcf680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.020+0000 7fcf6759e640 1 -- 172.21.15.29:0/3914693139 shutdown_connections 2023-10-16T14:17:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.020+0000 7fcf6759e640 1 -- 172.21.15.29:0/3914693139 wait complete. 2023-10-16T14:17:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.021+0000 7fcf6759e640 1 Processor -- start 2023-10-16T14:17:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.022+0000 7fcf6759e640 1 -- start start 2023-10-16T14:17:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf68154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815c9f0 0x7fcf68154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 0x7fcf68155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6806c980 con 0x7fcf6815dde0 2023-10-16T14:17:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6806bd70 con 0x7fcf68070f20 2023-10-16T14:17:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6806c050 con 0x7fcf6815c9f0 2023-10-16T14:17:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 0x7fcf68155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 0x7fcf68155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56316/0 (socket says 172.21.15.29:56316) 2023-10-16T14:17:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 -- 172.21.15.29:0/2265915365 learned_addr learned my addr 172.21.15.29:0/2265915365 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf65d9b640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815c9f0 0x7fcf68154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf6659c640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf68154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 -- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815c9f0 msgr2=0x7fcf68154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815c9f0 0x7fcf68154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 -- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 msgr2=0x7fcf68154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf68154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.023+0000 7fcf66d9d640 1 -- 172.21.15.29:0/2265915365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c009400 con 0x7fcf6815dde0 2023-10-16T14:17:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.024+0000 7fcf66d9d640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 0x7fcf68155ee0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009d90 tx=0x7fcf60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.024+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf60040650 con 0x7fcf6815dde0 2023-10-16T14:17:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.024+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf600407b0 con 0x7fcf6815dde0 2023-10-16T14:17:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.024+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf60009a70 con 0x7fcf6815dde0 2023-10-16T14:17:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.024+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf60040b60 con 0x7fcf6815dde0 2023-10-16T14:17:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.024+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6806c610 con 0x7fcf6815dde0 2023-10-16T14:17:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.026+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf680701c0 con 0x7fcf6815dde0 2023-10-16T14:17:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.027+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcf60040cc0 con 0x7fcf6815dde0 2023-10-16T14:17:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.027+0000 7fcf56ffd640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcf4806ae70 0x7fcf4806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.028+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcf600abfc0 con 0x7fcf6815dde0 2023-10-16T14:17:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.028+0000 7fcf6659c640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcf4806ae70 0x7fcf4806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.029+0000 7fcf6659c640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcf4806ae70 0x7fcf4806d330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcf580011e0 tx=0x7fcf58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.033+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf600deca0 con 0x7fcf6815dde0 2023-10-16T14:17:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.293+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fcf68070490 con 0x7fcf6815dde0 2023-10-16T14:17:35.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.298+0000 7fcf56ffd640 1 -- 172.21.15.29:0/2265915365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fcf680701c0 con 0x7fcf6815dde0 2023-10-16T14:17:35.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcf4806ae70 msgr2=0x7fcf4806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcf4806ae70 0x7fcf4806d330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcf580011e0 tx=0x7fcf58009040 comp rx=0 tx=0).stop 2023-10-16T14:17:35.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 msgr2=0x7fcf68155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 0x7fcf68155ee0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009d90 tx=0x7fcf60004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 shutdown_connections 2023-10-16T14:17:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fcf4806ae70 0x7fcf4806d330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6815dde0 0x7fcf68155ee0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf6815c9f0 0x7fcf68154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 --2- 172.21.15.29:0/2265915365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf68070f20 0x7fcf68154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 >> 172.21.15.29:0/2265915365 conn(0x7fcf680ce6a0 msgr2=0x7fcf6815b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:35.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 shutdown_connections 2023-10-16T14:17:35.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.301+0000 7fcf6759e640 1 -- 172.21.15.29:0/2265915365 wait complete. 2023-10-16T14:17:35.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-16T14:17:35.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.533+0000 7ff5a7fff640 1 Processor -- start 2023-10-16T14:17:35.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.533+0000 7ff5a7fff640 1 -- start start 2023-10-16T14:17:35.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a815c9f0 0x7ff5a80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff5a80c9230 0x7ff5a80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5a8070f20 0x7ff5a80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a7fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a80cdec0 con 0x7ff5a8070f20 2023-10-16T14:17:35.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a7fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a8160a90 con 0x7ff5a80c9230 2023-10-16T14:17:35.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80cd900 con 0x7ff5a815c9f0 2023-10-16T14:17:35.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5a8070f20 0x7ff5a80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5a8070f20 0x7ff5a80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56324/0 (socket says 172.21.15.29:56324) 2023-10-16T14:17:35.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.535+0000 7ff5a77fe640 1 -- 172.21.15.29:0/641272640 learned_addr learned my addr 172.21.15.29:0/641272640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:35.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a77fe640 1 -- 172.21.15.29:0/641272640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a815c9f0 msgr2=0x7ff5a80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:35.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a67fc640 1 --2- 172.21.15.29:0/641272640 >> v2:172.21.15.170:3300/0 conn(0x7ff5a80c9230 0x7ff5a80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a6ffd640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a815c9f0 0x7ff5a80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a815c9f0 0x7ff5a80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a77fe640 1 -- 172.21.15.29:0/641272640 >> v2:172.21.15.170:3300/0 conn(0x7ff5a80c9230 msgr2=0x7ff5a80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/641272640 >> v2:172.21.15.170:3300/0 conn(0x7ff5a80c9230 0x7ff5a80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a77fe640 1 -- 172.21.15.29:0/641272640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a80cdbe0 con 0x7ff5a8070f20 2023-10-16T14:17:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a67fc640 1 --2- 172.21.15.29:0/641272640 >> v2:172.21.15.170:3300/0 conn(0x7ff5a80c9230 0x7ff5a80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.536+0000 7ff5a6ffd640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a815c9f0 0x7ff5a80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:35.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.537+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a8070f20 0x7ff5a80ac890 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5a80719e0 tx=0x7ff5a002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecfd5bca274956d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:35.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.537+0000 7ff5a5ffb640 1 -- 172.21.15.29:0/641272640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0031880 con 0x7ff5a8070f20 2023-10-16T14:17:35.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.537+0000 7ff5a5ffb640 1 -- 172.21.15.29:0/641272640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a00319e0 con 0x7ff5a8070f20 2023-10-16T14:17:35.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.537+0000 7ff5a5ffb640 1 -- 172.21.15.29:0/641272640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0031d70 con 0x7ff5a8070f20 2023-10-16T14:17:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 -- 172.21.15.29:0/641272640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a8070f20 msgr2=0x7ff5a80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a8070f20 0x7ff5a80ac890 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff5a80719e0 tx=0x7ff5a002b920 comp rx=0 tx=0).stop 2023-10-16T14:17:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 -- 172.21.15.29:0/641272640 shutdown_connections 2023-10-16T14:17:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a8070f20 0x7ff5a80ac890 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/641272640 >> v2:172.21.15.170:3300/0 conn(0x7ff5a80c9230 0x7ff5a80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/641272640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a815c9f0 0x7ff5a80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 -- 172.21.15.29:0/641272640 >> 172.21.15.29:0/641272640 conn(0x7ff5a80ce6a0 msgr2=0x7ff5a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 -- 172.21.15.29:0/641272640 shutdown_connections 2023-10-16T14:17:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.538+0000 7ff5a7fff640 1 -- 172.21.15.29:0/641272640 wait complete. 2023-10-16T14:17:35.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.540+0000 7ff5a7fff640 1 Processor -- start 2023-10-16T14:17:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.540+0000 7ff5a7fff640 1 -- start start 2023-10-16T14:17:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a8070f20 0x7ff5a8154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 0x7ff5a815f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5a815c9f0 0x7ff5a806bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a8155150 con 0x7ff5a80c9230 2023-10-16T14:17:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a7fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a81553c0 con 0x7ff5a815c9f0 2023-10-16T14:17:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a81556a0 con 0x7ff5a8070f20 2023-10-16T14:17:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 0x7ff5a815f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 0x7ff5a815f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56328/0 (socket says 172.21.15.29:56328) 2023-10-16T14:17:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 -- 172.21.15.29:0/820483070 learned_addr learned my addr 172.21.15.29:0/820483070 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5a815c9f0 0x7ff5a806bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 -- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a8070f20 msgr2=0x7ff5a8154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a6ffd640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a8070f20 0x7ff5a8154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a8070f20 0x7ff5a8154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 -- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5a815c9f0 msgr2=0x7ff5a806bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5a815c9f0 0x7ff5a806bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a67fc640 1 -- 172.21.15.29:0/820483070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a00094d0 con 0x7ff5a80c9230 2023-10-16T14:17:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.541+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5a815c9f0 0x7ff5a806bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.542+0000 7ff5a67fc640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 0x7ff5a815f6b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff59000c7f0 tx=0x7ff59000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:35.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.542+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590007a30 con 0x7ff5a80c9230 2023-10-16T14:17:35.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.542+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590007b90 con 0x7ff5a80c9230 2023-10-16T14:17:35.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.542+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59000ce40 con 0x7ff5a80c9230 2023-10-16T14:17:35.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.542+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a8155980 con 0x7ff5a80c9230 2023-10-16T14:17:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.542+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a8155f40 con 0x7ff5a80c9230 2023-10-16T14:17:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.543+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a804e9b0 con 0x7ff5a80c9230 2023-10-16T14:17:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.544+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff5900113f0 con 0x7ff5a80c9230 2023-10-16T14:17:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.545+0000 7ff59ffff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff58806ae70 0x7ff58806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.545+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff590089550 con 0x7ff5a80c9230 2023-10-16T14:17:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.546+0000 7ff5a6ffd640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff58806ae70 0x7ff58806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.547+0000 7ff5a6ffd640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff58806ae70 0x7ff58806d330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff598008e60 tx=0x7ff5980083b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.550+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff5900548b0 con 0x7ff5a80c9230 2023-10-16T14:17:35.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.803+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ff5a80adac0 con 0x7ff5a80c9230 2023-10-16T14:17:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.809+0000 7ff59ffff640 1 -- 172.21.15.29:0/820483070 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff590057ed0 con 0x7ff5a80c9230 2023-10-16T14:17:35.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff58806ae70 msgr2=0x7ff58806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff58806ae70 0x7ff58806d330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff598008e60 tx=0x7ff5980083b0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 msgr2=0x7ff5a815f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 0x7ff5a815f6b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff59000c7f0 tx=0x7ff59000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 shutdown_connections 2023-10-16T14:17:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff58806ae70 0x7ff58806d330 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5a80c9230 0x7ff5a815f6b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5a815c9f0 0x7ff5a806bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/820483070 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5a8070f20 0x7ff5a8154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.811+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 >> 172.21.15.29:0/820483070 conn(0x7ff5a80ce6a0 msgr2=0x7ff5a80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.812+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 shutdown_connections 2023-10-16T14:17:35.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:35.812+0000 7ff5a7fff640 1 -- 172.21.15.29:0/820483070 wait complete. 2023-10-16T14:17:35.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-16T14:17:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.042+0000 7fa0462fc640 1 Processor -- start 2023-10-16T14:17:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.043+0000 7fa0462fc640 1 -- start start 2023-10-16T14:17:36.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.044+0000 7fa0462fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0400ce6b0 0x7fa0400cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.044+0000 7fa0462fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0400cc570 0x7fa0400cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.044+0000 7fa0462fc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0400cf4a0 con 0x7fa04015ac10 2023-10-16T14:17:36.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.044+0000 7fa0462fc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0400ceb40 con 0x7fa0400ce6b0 2023-10-16T14:17:36.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.044+0000 7fa0462fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400cb550 con 0x7fa0400cc570 2023-10-16T14:17:36.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.044+0000 7fa044af9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0400ce6b0 0x7fa0400cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa044af9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0400ce6b0 0x7fa0400cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56340/0 (socket says 172.21.15.29:56340) 2023-10-16T14:17:36.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa044af9640 1 -- 172.21.15.29:0/1131822757 learned_addr learned my addr 172.21.15.29:0/1131822757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:36.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa045afb640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0400cc570 0x7fa0400cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa044af9640 1 -- 172.21.15.29:0/1131822757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0400cc570 msgr2=0x7fa0400cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:36.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa044af9640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0400cc570 0x7fa0400cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa044af9640 1 -- 172.21.15.29:0/1131822757 >> v1:172.21.15.170:6789/0 conn(0x7fa04015ac10 legacy=0x7fa0400cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:36.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa044af9640 1 -- 172.21.15.29:0/1131822757 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0400cb830 con 0x7fa0400ce6b0 2023-10-16T14:17:36.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.045+0000 7fa045afb640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0400cc570 0x7fa0400cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:36.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.046+0000 7fa044af9640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400ce6b0 0x7fa0400cc030 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa0300098a0 tx=0x7fa03002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8dead47631f306 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:36.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.046+0000 7fa037fff640 1 -- 172.21.15.29:0/1131822757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030030840 con 0x7fa0400ce6b0 2023-10-16T14:17:36.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.046+0000 7fa037fff640 1 -- 172.21.15.29:0/1131822757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0300309a0 con 0x7fa0400ce6b0 2023-10-16T14:17:36.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.046+0000 7fa037fff640 1 -- 172.21.15.29:0/1131822757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030030d30 con 0x7fa0400ce6b0 2023-10-16T14:17:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 -- 172.21.15.29:0/1131822757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400ce6b0 msgr2=0x7fa0400cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400ce6b0 0x7fa0400cc030 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa0300098a0 tx=0x7fa03002b980 comp rx=0 tx=0).stop 2023-10-16T14:17:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 -- 172.21.15.29:0/1131822757 shutdown_connections 2023-10-16T14:17:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0400cc570 0x7fa0400cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 --2- 172.21.15.29:0/1131822757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400ce6b0 0x7fa0400cc030 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 -- 172.21.15.29:0/1131822757 >> 172.21.15.29:0/1131822757 conn(0x7fa040099a10 msgr2=0x7fa0400d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:36.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 -- 172.21.15.29:0/1131822757 shutdown_connections 2023-10-16T14:17:36.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.047+0000 7fa0462fc640 1 -- 172.21.15.29:0/1131822757 wait complete. 2023-10-16T14:17:36.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.049+0000 7fa0462fc640 1 Processor -- start 2023-10-16T14:17:36.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.049+0000 7fa0462fc640 1 -- start start 2023-10-16T14:17:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0462fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 0x7fa0400c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0462fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0400ce6b0 0x7fa04015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0462fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa04015ac10 0x7fa04014b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0462fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015dd30 con 0x7fa0400cc570 2023-10-16T14:17:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0462fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015e010 con 0x7fa0400ce6b0 2023-10-16T14:17:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0462fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400c79e0 con 0x7fa04015ac10 2023-10-16T14:17:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0452fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 0x7fa0400c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0452fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 0x7fa0400c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56350/0 (socket says 172.21.15.29:56350) 2023-10-16T14:17:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.050+0000 7fa0452fa640 1 -- 172.21.15.29:0/3610818833 learned_addr learned my addr 172.21.15.29:0/3610818833 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa045afb640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa04015ac10 0x7fa04014b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa0452fa640 1 -- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa04015ac10 msgr2=0x7fa04014b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa0452fa640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa04015ac10 0x7fa04014b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa0452fa640 1 -- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0400ce6b0 msgr2=0x7fa04015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa044af9640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0400ce6b0 0x7fa04015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa0452fa640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0400ce6b0 0x7fa04015d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa0452fa640 1 -- 172.21.15.29:0/3610818833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0300094d0 con 0x7fa0400cc570 2023-10-16T14:17:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa0452fa640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 0x7fa0400c84c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b3f0 tx=0x7fa03c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.051+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c011830 con 0x7fa0400cc570 2023-10-16T14:17:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.052+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c011990 con 0x7fa0400cc570 2023-10-16T14:17:36.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.052+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c011d40 con 0x7fa0400cc570 2023-10-16T14:17:36.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.052+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0400c7cc0 con 0x7fa0400cc570 2023-10-16T14:17:36.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.052+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04015d970 con 0x7fa0400cc570 2023-10-16T14:17:36.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.053+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0400c81e0 con 0x7fa0400cc570 2023-10-16T14:17:36.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.054+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa03c01a400 con 0x7fa0400cc570 2023-10-16T14:17:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.055+0000 7fa035ffb640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa02406ae70 0x7fa02406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.055+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa03c0891a0 con 0x7fa0400cc570 2023-10-16T14:17:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.055+0000 7fa044af9640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa02406ae70 0x7fa02406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.056+0000 7fa044af9640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa02406ae70 0x7fa02406d330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa030036000 tx=0x7fa03003b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.059+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0400c81e0 con 0x7fa0400cc570 2023-10-16T14:17:36.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.313+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fa04004e9b0 con 0x7fa0400cc570 2023-10-16T14:17:36.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.318+0000 7fa035ffb640 1 -- 172.21.15.29:0/3610818833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa03c054500 con 0x7fa0400cc570 2023-10-16T14:17:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa02406ae70 msgr2=0x7fa02406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa02406ae70 0x7fa02406d330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa030036000 tx=0x7fa03003b040 comp rx=0 tx=0).stop 2023-10-16T14:17:36.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 msgr2=0x7fa0400c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 0x7fa0400c84c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b3f0 tx=0x7fa03c00b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 shutdown_connections 2023-10-16T14:17:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa02406ae70 0x7fa02406d330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa04015ac10 0x7fa04014b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0400ce6b0 0x7fa04015d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 --2- 172.21.15.29:0/3610818833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0400cc570 0x7fa0400c84c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.321+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 >> 172.21.15.29:0/3610818833 conn(0x7fa040099a10 msgr2=0x7fa0400c4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:36.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.322+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 shutdown_connections 2023-10-16T14:17:36.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.322+0000 7fa0462fc640 1 -- 172.21.15.29:0/3610818833 wait complete. 2023-10-16T14:17:36.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-16T14:17:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.546+0000 7f3047e75640 1 Processor -- start 2023-10-16T14:17:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.547+0000 7f3047e75640 1 -- start start 2023-10-16T14:17:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.548+0000 7f3047e75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.548+0000 7f3047e75640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f304006c140 0x7f304006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.548+0000 7f3047e75640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3040070360 con 0x7f3040159e90 2023-10-16T14:17:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.548+0000 7f3047e75640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304006cb10 con 0x7f304006c140 2023-10-16T14:17:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.548+0000 7f3047e75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304015d890 con 0x7f3040159ab0 2023-10-16T14:17:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.549+0000 7f3045c11640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.549+0000 7f3045410640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f3040159e90 0x7f304006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:41590/0 (socket says 172.21.15.29:41590) 2023-10-16T14:17:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.549+0000 7f3045410640 1 -- 172.21.15.29:0/801319844 learned_addr learned my addr 172.21.15.29:0/801319844 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.549+0000 7f3046412640 1 --2- 172.21.15.29:0/801319844 >> v2:172.21.15.170:3300/0 conn(0x7f304006c140 0x7f304006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.550+0000 7f3046412640 1 -- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 msgr2=0x7f30400ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.550+0000 7f3046412640 1 --2- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.550+0000 7f3046412640 1 -- 172.21.15.29:0/801319844 >> v1:172.21.15.170:6789/0 conn(0x7f3040159e90 legacy=0x7f304006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.550+0000 7f3046412640 1 -- 172.21.15.29:0/801319844 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400cd1f0 con 0x7f304006c140 2023-10-16T14:17:36.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.551+0000 7f3046412640 1 --2- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f304006c5d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3038009dc0 tx=0x7f303802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98fe815e75ca3d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:36.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.552+0000 7f3044c0f640 1 -- 172.21.15.29:0/801319844 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303802cb20 con 0x7f304006c140 2023-10-16T14:17:36.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.552+0000 7f3044c0f640 1 -- 172.21.15.29:0/801319844 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303802cc80 con 0x7f304006c140 2023-10-16T14:17:36.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.552+0000 7f3044c0f640 1 -- 172.21.15.29:0/801319844 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038034440 con 0x7f304006c140 2023-10-16T14:17:36.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.552+0000 7f3047e75640 1 -- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 msgr2=0x7f304006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.552+0000 7f3047e75640 1 --2- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f304006c5d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3038009dc0 tx=0x7f303802bd70 comp rx=0 tx=0).stop 2023-10-16T14:17:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.553+0000 7f3047e75640 1 -- 172.21.15.29:0/801319844 shutdown_connections 2023-10-16T14:17:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.553+0000 7f3047e75640 1 --2- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f304006c5d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.553+0000 7f3047e75640 1 --2- 172.21.15.29:0/801319844 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.553+0000 7f3047e75640 1 -- 172.21.15.29:0/801319844 >> 172.21.15.29:0/801319844 conn(0x7f30400ce9b0 msgr2=0x7f30400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.553+0000 7f3047e75640 1 -- 172.21.15.29:0/801319844 shutdown_connections 2023-10-16T14:17:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.553+0000 7f3047e75640 1 -- 172.21.15.29:0/801319844 wait complete. 2023-10-16T14:17:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.555+0000 7f3047e75640 1 Processor -- start 2023-10-16T14:17:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.555+0000 7f3047e75640 1 -- start start 2023-10-16T14:17:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3047e75640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f3040148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3047e75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3047e75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 0x7f30400b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3047e75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400b2310 con 0x7f3040159e90 2023-10-16T14:17:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3047e75640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400b1700 con 0x7f304006c140 2023-10-16T14:17:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3047e75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400b19e0 con 0x7f3040159ab0 2023-10-16T14:17:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3046412640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 0x7f30400b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3046412640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 0x7f30400b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56360/0 (socket says 172.21.15.29:56360) 2023-10-16T14:17:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3046412640 1 -- 172.21.15.29:0/1278257909 learned_addr learned my addr 172.21.15.29:0/1278257909 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3045c11640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f3040148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.557+0000 7f3045410640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f3046412640 1 -- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 msgr2=0x7f30400b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f3046412640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f3046412640 1 -- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 msgr2=0x7f3040148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f3046412640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f3040148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f3046412640 1 -- 172.21.15.29:0/1278257909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3038009a70 con 0x7f3040159e90 2023-10-16T14:17:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f3046412640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 0x7f30400b09c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3038037040 tx=0x7f3038004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303802cb20 con 0x7f3040159e90 2023-10-16T14:17:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.558+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038040450 con 0x7f3040159e90 2023-10-16T14:17:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.559+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038040800 con 0x7f3040159e90 2023-10-16T14:17:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.559+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30400b1cc0 con 0x7f3040159e90 2023-10-16T14:17:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.559+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304015d100 con 0x7f3040159e90 2023-10-16T14:17:36.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.560+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30400b2480 con 0x7f3040159e90 2023-10-16T14:17:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.561+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3038040960 con 0x7f3040159e90 2023-10-16T14:17:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.562+0000 7f30367fc640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f302406ae70 0x7f302406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.562+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f30380abe60 con 0x7f3040159e90 2023-10-16T14:17:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.562+0000 7f3045c11640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f302406ae70 0x7f302406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.564+0000 7f3045c11640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f302406ae70 0x7f302406d330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30400c8f90 tx=0x7f303c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.566+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30380771d0 con 0x7f3040159e90 2023-10-16T14:17:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.832+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f304004e160 con 0x7f3040159e90 2023-10-16T14:17:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.836+0000 7f30367fc640 1 -- 172.21.15.29:0/1278257909 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f303807a7f0 con 0x7f3040159e90 2023-10-16T14:17:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f302406ae70 msgr2=0x7f302406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f302406ae70 0x7f302406d330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30400c8f90 tx=0x7f303c008040 comp rx=0 tx=0).stop 2023-10-16T14:17:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 msgr2=0x7f30400b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 0x7f30400b09c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3038037040 tx=0x7f3038004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 shutdown_connections 2023-10-16T14:17:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f302406ae70 0x7f302406d330 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3040159e90 0x7f30400b09c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3040159ab0 0x7f30400b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 --2- 172.21.15.29:0/1278257909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304006c140 0x7f3040148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:36.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 >> 172.21.15.29:0/1278257909 conn(0x7f30400ce9b0 msgr2=0x7f30400ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 shutdown_connections 2023-10-16T14:17:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:36.838+0000 7f3047e75640 1 -- 172.21.15.29:0/1278257909 wait complete. 2023-10-16T14:17:36.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.063+0000 7fb9e951d640 1 Processor -- start 2023-10-16T14:17:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.064+0000 7fb9e951d640 1 -- start start 2023-10-16T14:17:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.065+0000 7fb9e951d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 0x7fb9e406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9e951d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406f7f0 con 0x7fb9e4159f00 2023-10-16T14:17:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9e951d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406c5e0 con 0x7fb9e40b0b20 2023-10-16T14:17:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9e951d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e406c8c0 con 0x7fb9e4159b20 2023-10-16T14:17:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9dbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb9e4159f00 0x7fb9e40b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42428/0 (socket says 172.21.15.29:42428) 2023-10-16T14:17:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9dbfff640 1 -- 172.21.15.29:0/3606113483 learned_addr learned my addr 172.21.15.29:0/3606113483 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151779325 0 0) 0x7fb9e406f7f0 con 0x7fb9e4159f00 2023-10-16T14:17:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d0005730 con 0x7fb9e4159f00 2023-10-16T14:17:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.066+0000 7fb9e2ffd640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 0x7fb9e406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741035204 0 0) 0x7fb9d0005730 con 0x7fb9e4159f00 2023-10-16T14:17:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 msgr2=0x7fb9e406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 0x7fb9e406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 >> v1:172.21.15.170:6789/0 conn(0x7fb9e40b0b20 legacy=0x7fb9e406fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e406cba0 con 0x7fb9e4159f00 2023-10-16T14:17:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fb9cc001670 con 0x7fb9e4159f00 2023-10-16T14:17:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 >> v1:172.21.15.29:6789/0 conn(0x7fb9e4159f00 legacy=0x7fb9e40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 0x7fb9d0005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9d0006230 0x7fb9d00069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9d0006f10 0x7fb9d0007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e406c8c0 con 0x7fb9d0006f10 2023-10-16T14:17:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d000ddd0 con 0x7fb9d0006230 2023-10-16T14:17:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d000df90 con 0x7fb9e4159b20 2023-10-16T14:17:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.067+0000 7fb9e37fe640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9d0006f10 0x7fb9d0007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9dbfff640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9d0006230 0x7fb9d00069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9e2ffd640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 0x7fb9d0005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9e37fe640 1 -- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 msgr2=0x7fb9d0005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9e37fe640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4159b20 0x7fb9d0005bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9e37fe640 1 -- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9d0006230 msgr2=0x7fb9d00069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9e37fe640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9d0006230 0x7fb9d00069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.068+0000 7fb9e37fe640 1 -- 172.21.15.29:0/3606113483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d0005730 con 0x7fb9d0006f10 2023-10-16T14:17:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e37fe640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9d0006f10 0x7fb9d0007390 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40c5650 tx=0x7fb9d0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70895e24d320a81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc003b70 con 0x7fb9d0006f10 2023-10-16T14:17:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc003cd0 con 0x7fb9d0006f10 2023-10-16T14:17:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e27fc640 1 -- 172.21.15.29:0/3606113483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc005310 con 0x7fb9d0006f10 2023-10-16T14:17:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e951d640 1 -- 172.21.15.29:0/3606113483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9d0006f10 msgr2=0x7fb9d0007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e951d640 1 --2- 172.21.15.29:0/3606113483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9d0006f10 0x7fb9d0007390 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40c5650 tx=0x7fb9d0007a00 comp rx=0 tx=0).stop 2023-10-16T14:17:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.069+0000 7fb9e37fe640 1 -- 172.21.15.29:0/3606113483 reap_dead start 2023-10-16T14:17:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.070+0000 7fb9e951d640 1 -- 172.21.15.29:0/3606113483 shutdown_connections 2023-10-16T14:17:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.070+0000 7fb9e951d640 1 -- 172.21.15.29:0/3606113483 >> 172.21.15.29:0/3606113483 conn(0x7fb9e40cb410 msgr2=0x7fb9e40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.070+0000 7fb9e951d640 1 -- 172.21.15.29:0/3606113483 shutdown_connections 2023-10-16T14:17:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.070+0000 7fb9e951d640 1 -- 172.21.15.29:0/3606113483 wait complete. 2023-10-16T14:17:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.071+0000 7fb9e951d640 1 Processor -- start 2023-10-16T14:17:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.071+0000 7fb9e951d640 1 -- start start 2023-10-16T14:17:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.072+0000 7fb9e951d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 0x7fb9e40afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.072+0000 7fb9e951d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9e4158760 0x7fb9e40c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.072+0000 7fb9e951d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4158b80 0x7fb9e40c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.072+0000 7fb9e951d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e40b0360 con 0x7fb9e4158380 2023-10-16T14:17:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.072+0000 7fb9e951d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e415b780 con 0x7fb9e4158760 2023-10-16T14:17:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.072+0000 7fb9e951d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e415ba60 con 0x7fb9e4158b80 2023-10-16T14:17:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 0x7fb9e40afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 0x7fb9e40afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56388/0 (socket says 172.21.15.29:56388) 2023-10-16T14:17:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 -- 172.21.15.29:0/4042444561 learned_addr learned my addr 172.21.15.29:0/4042444561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 -- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4158b80 msgr2=0x7fb9e40c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4158b80 0x7fb9e40c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 -- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9e4158760 msgr2=0x7fb9e40c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9e4158760 0x7fb9e40c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 -- 172.21.15.29:0/4042444561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e415bd40 con 0x7fb9e4158380 2023-10-16T14:17:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.073+0000 7fb9e2ffd640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 0x7fb9e40afb10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400e7e0 tx=0x7fb9d400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.074+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d400c650 con 0x7fb9e4158380 2023-10-16T14:17:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.074+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d400c7b0 con 0x7fb9e4158380 2023-10-16T14:17:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.074+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d400cb20 con 0x7fb9e4158380 2023-10-16T14:17:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.074+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e415c020 con 0x7fb9e4158380 2023-10-16T14:17:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.074+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e415c540 con 0x7fb9e4158380 2023-10-16T14:17:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.075+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e404e160 con 0x7fb9e4158380 2023-10-16T14:17:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.076+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb9d400cc80 con 0x7fb9e4158380 2023-10-16T14:17:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.077+0000 7fb9db7fe640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9c406aec0 0x7fb9c406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.077+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb9d40890e0 con 0x7fb9e4158380 2023-10-16T14:17:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.077+0000 7fb9dbfff640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9c406aec0 0x7fb9c406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.078+0000 7fb9dbfff640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9c406aec0 0x7fb9c406d380 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc001050 tx=0x7fb9cc001d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.081+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb9d40544c0 con 0x7fb9e4158380 2023-10-16T14:17:37.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.266+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e40b10b0 con 0x7fb9e4158380 2023-10-16T14:17:37.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.268+0000 7fb9db7fe640 1 -- 172.21.15.29:0/4042444561 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fb9d4009380 con 0x7fb9e4158380 2023-10-16T14:17:37.272 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:37.273 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:33.132549+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:37.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9c406aec0 msgr2=0x7fb9c406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9c406aec0 0x7fb9c406d380 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc001050 tx=0x7fb9cc001d90 comp rx=0 tx=0).stop 2023-10-16T14:17:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 msgr2=0x7fb9e40afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 0x7fb9e40afb10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400e7e0 tx=0x7fb9d400eca0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 shutdown_connections 2023-10-16T14:17:37.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9c406aec0 0x7fb9c406d380 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9e4158b80 0x7fb9e40c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9e4158760 0x7fb9e40c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 --2- 172.21.15.29:0/4042444561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9e4158380 0x7fb9e40afb10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 >> 172.21.15.29:0/4042444561 conn(0x7fb9e40cb410 msgr2=0x7fb9e40ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:37.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.270+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 shutdown_connections 2023-10-16T14:17:37.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.271+0000 7fb9e951d640 1 -- 172.21.15.29:0/4042444561 wait complete. 2023-10-16T14:17:37.292 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:17:37.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:17:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.513+0000 7fba1631a640 1 Processor -- start 2023-10-16T14:17:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.513+0000 7fba1631a640 1 -- start start 2023-10-16T14:17:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.514+0000 7fba1631a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fba100cf230 0x7fba100ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.514+0000 7fba1631a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba10159f80 con 0x7fba10159ba0 2023-10-16T14:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.514+0000 7fba1631a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100aead0 con 0x7fba10070a90 2023-10-16T14:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.514+0000 7fba1631a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba100cbe60 con 0x7fba100cf230 2023-10-16T14:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.515+0000 7fba15318640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fba10159ba0 0x7fba10070380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42432/0 (socket says 172.21.15.29:42432) 2023-10-16T14:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.515+0000 7fba15318640 1 -- 172.21.15.29:0/359459546 learned_addr learned my addr 172.21.15.29:0/359459546 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.515+0000 7fba14b17640 1 --2- 172.21.15.29:0/359459546 >> v2:172.21.15.170:3300/0 conn(0x7fba100cf230 0x7fba100ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.515+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269305081 0 0) 0x7fba10159f80 con 0x7fba10159ba0 2023-10-16T14:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.515+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f8003520 con 0x7fba10159ba0 2023-10-16T14:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2783153422 0 0) 0x7fb9f8003520 con 0x7fba10159ba0 2023-10-16T14:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 >> v2:172.21.15.170:3300/0 conn(0x7fba100cf230 msgr2=0x7fba100ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 --2- 172.21.15.29:0/359459546 >> v2:172.21.15.170:3300/0 conn(0x7fba100cf230 0x7fba100ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 >> v1:172.21.15.170:6789/0 conn(0x7fba10070a90 legacy=0x7fba100c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100cd400 con 0x7fba10159ba0 2023-10-16T14:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fba000019f0 con 0x7fba10159ba0 2023-10-16T14:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba14b17640 1 --2- 172.21.15.29:0/359459546 >> v2:172.21.15.170:3300/0 conn(0x7fba100cf230 0x7fba100ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 >> v1:172.21.15.29:6789/0 conn(0x7fba10159ba0 legacy=0x7fba10070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fb9f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f800c0a0 con 0x7fb9f8003970 2023-10-16T14:17:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f800c210 con 0x7fb9f8005180 2023-10-16T14:17:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f800c3d0 con 0x7fb9f80047c0 2023-10-16T14:17:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba15318640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fb9f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.516+0000 7fba15b19640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15318640 1 -- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9f80047c0 msgr2=0x7fb9f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15318640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15318640 1 -- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9f8005180 msgr2=0x7fb9f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15318640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15318640 1 -- 172.21.15.29:0/359459546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f8003520 con 0x7fb9f8003970 2023-10-16T14:17:37.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15b19640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.517+0000 7fba15b19640 1 -- 172.21.15.29:0/359459546 reap_dead start 2023-10-16T14:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.518+0000 7fba15318640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fb9f8003d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fba0000a830 tx=0x7fb9f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf480c3629163ca7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.518+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000072d0 con 0x7fb9f8003970 2023-10-16T14:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.518+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0002f070 con 0x7fb9f8003970 2023-10-16T14:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.518+0000 7fba0ffff640 1 -- 172.21.15.29:0/359459546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000076f0 con 0x7fb9f8003970 2023-10-16T14:17:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 -- 172.21.15.29:0/359459546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 msgr2=0x7fb9f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fb9f8003d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fba0000a830 tx=0x7fb9f80057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 -- 172.21.15.29:0/359459546 shutdown_connections 2023-10-16T14:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 --2- 172.21.15.29:0/359459546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fb9f8003d90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 -- 172.21.15.29:0/359459546 >> 172.21.15.29:0/359459546 conn(0x7fba100cf790 msgr2=0x7fba100cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 -- 172.21.15.29:0/359459546 shutdown_connections 2023-10-16T14:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.519+0000 7fba1631a640 1 -- 172.21.15.29:0/359459546 wait complete. 2023-10-16T14:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.521+0000 7fba1631a640 1 Processor -- start 2023-10-16T14:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.521+0000 7fba1631a640 1 -- start start 2023-10-16T14:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba1631a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fba101573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba1631a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba100c5ed0 0x7fba100c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba1631a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba100c6330 0x7fba100ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba1631a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100c6a90 con 0x7fb9f8003970 2023-10-16T14:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba1631a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100c4dc0 con 0x7fba100c6330 2023-10-16T14:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba1631a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba100ca6d0 con 0x7fba100c5ed0 2023-10-16T14:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15318640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fba101573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15318640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fba101573c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56412/0 (socket says 172.21.15.29:56412) 2023-10-16T14:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15318640 1 -- 172.21.15.29:0/533738131 learned_addr learned my addr 172.21.15.29:0/533738131 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15318640 1 -- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba100c5ed0 msgr2=0x7fba100c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15318640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba100c5ed0 0x7fba100c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15318640 1 -- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba100c6330 msgr2=0x7fba100ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.522+0000 7fba15b19640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba100c6330 0x7fba100ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba15318640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba100c6330 0x7fba100ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba15318640 1 -- 172.21.15.29:0/533738131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100c4520 con 0x7fb9f8003970 2023-10-16T14:17:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba15b19640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba100c6330 0x7fba100ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba15318640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fba101573c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fba00001cb0 tx=0x7fba00001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000074e0 con 0x7fb9f8003970 2023-10-16T14:17:37.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba00007640 con 0x7fb9f8003970 2023-10-16T14:17:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba100c4800 con 0x7fb9f8003970 2023-10-16T14:17:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000079f0 con 0x7fb9f8003970 2023-10-16T14:17:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.523+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba100cab10 con 0x7fb9f8003970 2023-10-16T14:17:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.525+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1004e9b0 con 0x7fb9f8003970 2023-10-16T14:17:37.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.526+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fba00031030 con 0x7fb9f8003970 2023-10-16T14:17:37.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.526+0000 7fba0dffb640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9fc06ae70 0x7fb9fc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.527+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba000aaf00 con 0x7fb9f8003970 2023-10-16T14:17:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.527+0000 7fba14b17640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9fc06ae70 0x7fb9fc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.528+0000 7fba14b17640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9fc06ae70 0x7fb9fc06d330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fba04006fd0 tx=0x7fba04008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.531+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fba000762f0 con 0x7fb9f8003970 2023-10-16T14:17:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.725+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba100b5710 con 0x7fb9f8003970 2023-10-16T14:17:37.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.725+0000 7fba0dffb640 1 -- 172.21.15.29:0/533738131 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba00079910 con 0x7fb9f8003970 2023-10-16T14:17:37.730 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:17:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9fc06ae70 msgr2=0x7fb9fc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9fc06ae70 0x7fb9fc06d330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fba04006fd0 tx=0x7fba04008040 comp rx=0 tx=0).stop 2023-10-16T14:17:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 msgr2=0x7fba101573c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fba101573c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fba00001cb0 tx=0x7fba00001d80 comp rx=0 tx=0).stop 2023-10-16T14:17:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 shutdown_connections 2023-10-16T14:17:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb9fc06ae70 0x7fb9fc06d330 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba100c6330 0x7fba100ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba100c5ed0 0x7fba100c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 --2- 172.21.15.29:0/533738131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9f8003970 0x7fba101573c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.728+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 >> 172.21.15.29:0/533738131 conn(0x7fba100cf790 msgr2=0x7fba1006bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.729+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 shutdown_connections 2023-10-16T14:17:37.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.729+0000 7fba1631a640 1 -- 172.21.15.29:0/533738131 wait complete. 2023-10-16T14:17:37.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:17:37.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.976+0000 7f9c6ebed640 1 Processor -- start 2023-10-16T14:17:37.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.976+0000 7f9c6ebed640 1 -- start start 2023-10-16T14:17:37.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.978+0000 7f9c6ebed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 0x7f9c6815af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.978+0000 7f9c6ebed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680b35b0 con 0x7f9c6815abb0 2023-10-16T14:17:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.978+0000 7f9c6ebed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6815c040 con 0x7f9c6815b4d0 2023-10-16T14:17:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.978+0000 7f9c6ebed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6815ba80 con 0x7f9c680cbad0 2023-10-16T14:17:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.978+0000 7f9c6dbeb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9c6815abb0 0x7f9c680b2ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42440/0 (socket says 172.21.15.29:42440) 2023-10-16T14:17:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.978+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/1561894616 learned_addr learned my addr 172.21.15.29:0/1561894616 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6d3ea640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 0x7f9c6815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806619725 0 0) 0x7f9c680b35b0 con 0x7f9c6815abb0 2023-10-16T14:17:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54003520 con 0x7f9c6815abb0 2023-10-16T14:17:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567185877 0 0) 0x7f9c54003520 con 0x7f9c6815abb0 2023-10-16T14:17:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 msgr2=0x7f9c6815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 0x7f9c6815af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 >> v1:172.21.15.170:6789/0 conn(0x7f9c6815b4d0 legacy=0x7f9c68070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.979+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6815bd60 con 0x7f9c6815abb0 2023-10-16T14:17:37.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9c64001670 con 0x7f9c6815abb0 2023-10-16T14:17:37.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 >> v1:172.21.15.29:6789/0 conn(0x7f9c6815abb0 legacy=0x7f9c680b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c540047c0 0x7f9c54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 0x7f9c540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6815ba80 con 0x7f9c540039a0 2023-10-16T14:17:37.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400bab0 con 0x7f9c540047c0 2023-10-16T14:17:37.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400bc70 con 0x7f9c680cbad0 2023-10-16T14:17:37.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6e3ec640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 0x7f9c540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.980+0000 7f9c6d3ea640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c540047c0 0x7f9c54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.981+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 msgr2=0x7f9c540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.981+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680cbad0 0x7f9c540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.981+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c540047c0 msgr2=0x7f9c54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.981+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c540047c0 0x7f9c54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.981+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/1561894616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54003520 con 0x7f9c540039a0 2023-10-16T14:17:37.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.982+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c54003dc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c240 tx=0x7f9c540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a3dd6f9db2f2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64005ad0 con 0x7f9c540039a0 2023-10-16T14:17:37.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c64005c50 con 0x7f9c540039a0 2023-10-16T14:17:37.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6cbe9640 1 -- 172.21.15.29:0/1561894616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6402c8b0 con 0x7f9c540039a0 2023-10-16T14:17:37.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6ebed640 1 -- 172.21.15.29:0/1561894616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 msgr2=0x7f9c54003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/1561894616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c54003dc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c240 tx=0x7f9c540057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6e3ec640 1 -- 172.21.15.29:0/1561894616 reap_dead start 2023-10-16T14:17:37.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6ebed640 1 -- 172.21.15.29:0/1561894616 shutdown_connections 2023-10-16T14:17:37.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6ebed640 1 -- 172.21.15.29:0/1561894616 >> 172.21.15.29:0/1561894616 conn(0x7f9c680cc230 msgr2=0x7f9c680cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:37.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6ebed640 1 -- 172.21.15.29:0/1561894616 shutdown_connections 2023-10-16T14:17:37.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.983+0000 7f9c6ebed640 1 -- 172.21.15.29:0/1561894616 wait complete. 2023-10-16T14:17:37.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.985+0000 7f9c6ebed640 1 Processor -- start 2023-10-16T14:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.985+0000 7f9c6ebed640 1 -- start start 2023-10-16T14:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6ebed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c68154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6ebed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c68152420 0x7f9c680cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6ebed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680ce4b0 0x7f9c681ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6ebed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c680ce960 con 0x7f9c540039a0 2023-10-16T14:17:37.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6ebed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c680c6920 con 0x7f9c68152420 2023-10-16T14:17:37.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6ebed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c680c6c00 con 0x7f9c680ce4b0 2023-10-16T14:17:37.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.986+0000 7f9c6dbeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c68154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c68154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56436/0 (socket says 172.21.15.29:56436) 2023-10-16T14:17:37.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/3218342389 learned_addr learned my addr 172.21.15.29:0/3218342389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6e3ec640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680ce4b0 0x7f9c681ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680ce4b0 msgr2=0x7f9c681ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6d3ea640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c68152420 0x7f9c680cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680ce4b0 0x7f9c681ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c68152420 msgr2=0x7f9c680cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:37.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c68152420 0x7f9c680cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:37.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 -- 172.21.15.29:0/3218342389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c680c6ee0 con 0x7f9c540039a0 2023-10-16T14:17:37.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.987+0000 7f9c6dbeb640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c68154160 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9c6402c450 tx=0x7f9c64001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:37.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.988+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6403c040 con 0x7f9c540039a0 2023-10-16T14:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.988+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6402ccc0 con 0x7f9c540039a0 2023-10-16T14:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.988+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640072b0 con 0x7f9c540039a0 2023-10-16T14:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.988+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c680c71c0 con 0x7f9c540039a0 2023-10-16T14:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.988+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c681eeaf0 con 0x7f9c540039a0 2023-10-16T14:17:38.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.989+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6804e6d0 con 0x7f9c540039a0 2023-10-16T14:17:38.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.990+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9c64032070 con 0x7f9c540039a0 2023-10-16T14:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.991+0000 7f9c5e7fc640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9c4c06aea0 0x7f9c4c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.991+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9c640aa3d0 con 0x7f9c540039a0 2023-10-16T14:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.992+0000 7f9c6d3ea640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9c4c06aea0 0x7f9c4c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.993+0000 7f9c6d3ea640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9c4c06aea0 0x7f9c4c06d360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9c680ccc60 tx=0x7f9c58008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:37.995+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c640757d0 con 0x7f9c540039a0 2023-10-16T14:17:38.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.194+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9c680b49f0 con 0x7f9c540039a0 2023-10-16T14:17:38.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.195+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/3218342389 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9c64046020 con 0x7f9c540039a0 2023-10-16T14:17:38.199 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:38.199 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:17:38.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:38.200 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:17:38.200 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:17:38.200 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:38.200 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:17:38.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.197+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9c4c06aea0 msgr2=0x7f9c4c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:38.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.197+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9c4c06aea0 0x7f9c4c06d360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9c680ccc60 tx=0x7f9c58008040 comp rx=0 tx=0).stop 2023-10-16T14:17:38.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.197+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 msgr2=0x7f9c68154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:38.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.197+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c68154160 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9c6402c450 tx=0x7f9c64001a00 comp rx=0 tx=0).stop 2023-10-16T14:17:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 shutdown_connections 2023-10-16T14:17:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f9c4c06aea0 0x7f9c4c06d360 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c680ce4b0 0x7f9c681ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c68152420 0x7f9c680cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 --2- 172.21.15.29:0/3218342389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c540039a0 0x7f9c68154160 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 >> 172.21.15.29:0/3218342389 conn(0x7f9c680cc230 msgr2=0x7f9c680cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:38.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 shutdown_connections 2023-10-16T14:17:38.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.198+0000 7f9c6ebed640 1 -- 172.21.15.29:0/3218342389 wait complete. 2023-10-16T14:17:38.222 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:17:38.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:17:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.453+0000 7f1ba5861640 1 Processor -- start 2023-10-16T14:17:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.453+0000 7f1ba5861640 1 -- start start 2023-10-16T14:17:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1ba5861640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1ba015ac10 0x7f1ba006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1ba5861640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ba00b0360 0x7f1ba00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1ba5861640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00ca510 0x7f1ba00ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1ba5861640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba00b0d30 con 0x7f1ba00b0360 2023-10-16T14:17:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1ba5861640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba015b310 con 0x7f1ba015ac10 2023-10-16T14:17:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1ba5861640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba006f640 con 0x7f1ba00ca510 2023-10-16T14:17:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1b97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ba00b0360 0x7f1ba00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1b97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ba00b0360 0x7f1ba00b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56438/0 (socket says 172.21.15.29:56438) 2023-10-16T14:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.455+0000 7f1b97fff640 1 -- 172.21.15.29:0/686905233 learned_addr learned my addr 172.21.15.29:0/686905233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b9f7fe640 1 --2- 172.21.15.29:0/686905233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00ca510 0x7f1ba00ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b9effd640 1 --2- 172.21.15.29:0/686905233 >> v2:172.21.15.170:3300/0 conn(0x7f1ba015ac10 0x7f1ba006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b97fff640 1 -- 172.21.15.29:0/686905233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00ca510 msgr2=0x7f1ba00ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b97fff640 1 --2- 172.21.15.29:0/686905233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00ca510 0x7f1ba00ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b97fff640 1 -- 172.21.15.29:0/686905233 >> v2:172.21.15.170:3300/0 conn(0x7f1ba015ac10 msgr2=0x7f1ba006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b97fff640 1 --2- 172.21.15.29:0/686905233 >> v2:172.21.15.170:3300/0 conn(0x7f1ba015ac10 0x7f1ba006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.456+0000 7f1b97fff640 1 -- 172.21.15.29:0/686905233 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba015b5b0 con 0x7f1ba00b0360 2023-10-16T14:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.457+0000 7f1b97fff640 1 --2- 172.21.15.29:0/686905233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00b0360 0x7f1ba00b07f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ba006d540 tx=0x7f1b8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71a4e83e3cb072a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1b9e7fc640 1 -- 172.21.15.29:0/686905233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b880317e0 con 0x7f1ba00b0360 2023-10-16T14:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1b9e7fc640 1 -- 172.21.15.29:0/686905233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88031940 con 0x7f1ba00b0360 2023-10-16T14:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1b9e7fc640 1 -- 172.21.15.29:0/686905233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88031cd0 con 0x7f1ba00b0360 2023-10-16T14:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 -- 172.21.15.29:0/686905233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00b0360 msgr2=0x7f1ba00b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 --2- 172.21.15.29:0/686905233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00b0360 0x7f1ba00b07f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ba006d540 tx=0x7f1b8802b880 comp rx=0 tx=0).stop 2023-10-16T14:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 -- 172.21.15.29:0/686905233 shutdown_connections 2023-10-16T14:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 --2- 172.21.15.29:0/686905233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00ca510 0x7f1ba00ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 --2- 172.21.15.29:0/686905233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00b0360 0x7f1ba00b07f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 --2- 172.21.15.29:0/686905233 >> v2:172.21.15.170:3300/0 conn(0x7f1ba015ac10 0x7f1ba006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 -- 172.21.15.29:0/686905233 >> 172.21.15.29:0/686905233 conn(0x7f1ba00cb360 msgr2=0x7f1ba00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 -- 172.21.15.29:0/686905233 shutdown_connections 2023-10-16T14:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.458+0000 7f1ba5861640 1 -- 172.21.15.29:0/686905233 wait complete. 2023-10-16T14:17:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.460+0000 7f1ba5861640 1 Processor -- start 2023-10-16T14:17:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.460+0000 7f1ba5861640 1 -- start start 2023-10-16T14:17:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.461+0000 7f1ba5861640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00b0360 0x7f1ba00acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.461+0000 7f1ba5861640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 0x7f1ba0159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.461+0000 7f1ba5861640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ba015ac10 0x7f1ba01595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.461+0000 7f1ba5861640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba0159fe0 con 0x7f1ba00ca510 2023-10-16T14:17:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.461+0000 7f1ba5861640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00b1500 con 0x7f1ba015ac10 2023-10-16T14:17:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.461+0000 7f1ba5861640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00b1750 con 0x7f1ba00b0360 2023-10-16T14:17:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 0x7f1ba0159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 0x7f1ba0159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56454/0 (socket says 172.21.15.29:56454) 2023-10-16T14:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 -- 172.21.15.29:0/210392144 learned_addr learned my addr 172.21.15.29:0/210392144 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b9effd640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00b0360 0x7f1ba00acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 -- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00b0360 msgr2=0x7f1ba00acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b9f7fe640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ba015ac10 0x7f1ba01595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00b0360 0x7f1ba00acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 -- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ba015ac10 msgr2=0x7f1ba01595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ba015ac10 0x7f1ba01595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 -- 172.21.15.29:0/210392144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880094d0 con 0x7f1ba00ca510 2023-10-16T14:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b9effd640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00b0360 0x7f1ba00acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.462+0000 7f1b97fff640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 0x7f1ba0159090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1b8802b8c0 tx=0x7f1b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.463+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88032880 con 0x7f1ba00ca510 2023-10-16T14:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.463+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88032a20 con 0x7f1ba00ca510 2023-10-16T14:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.463+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88032dd0 con 0x7f1ba00ca510 2023-10-16T14:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.463+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba00b1a30 con 0x7f1ba00ca510 2023-10-16T14:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.463+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba00b1ff0 con 0x7f1ba00ca510 2023-10-16T14:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.464+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba0066de0 con 0x7f1ba00ca510 2023-10-16T14:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.465+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1b880453f0 con 0x7f1ba00ca510 2023-10-16T14:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.466+0000 7f1b9cff9640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1b8006aec0 0x7f1b8006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.466+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1b880ab490 con 0x7f1ba00ca510 2023-10-16T14:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.467+0000 7f1b9effd640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1b8006aec0 0x7f1b8006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.468+0000 7f1b9effd640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1b8006aec0 0x7f1b8006d380 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006fd0 tx=0x7f1b90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.470+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1b88076b50 con 0x7f1ba00ca510 2023-10-16T14:17:38.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:38.669+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1ba015be20 con 0x7f1ba00ca510 2023-10-16T14:17:39.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.139+0000 7f1b9cff9640 1 -- 172.21.15.29:0/210392144 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b8807a170 con 0x7f1ba00ca510 2023-10-16T14:17:39.144 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:17:39.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1b8006aec0 msgr2=0x7f1b8006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1b8006aec0 0x7f1b8006d380 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006fd0 tx=0x7f1b90008040 comp rx=0 tx=0).stop 2023-10-16T14:17:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 msgr2=0x7f1ba0159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 0x7f1ba0159090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1b8802b8c0 tx=0x7f1b880046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 shutdown_connections 2023-10-16T14:17:39.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f1b8006aec0 0x7f1b8006d380 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ba00ca510 0x7f1ba0159090 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ba015ac10 0x7f1ba01595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 --2- 172.21.15.29:0/210392144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ba00b0360 0x7f1ba00acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 >> 172.21.15.29:0/210392144 conn(0x7f1ba00cb360 msgr2=0x7f1ba00ae630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.143+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 shutdown_connections 2023-10-16T14:17:39.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.144+0000 7f1ba5861640 1 -- 172.21.15.29:0/210392144 wait complete. 2023-10-16T14:17:39.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.385+0000 7fa1c9446640 1 Processor -- start 2023-10-16T14:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.385+0000 7fa1c9446640 1 -- start start 2023-10-16T14:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1c9446640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa1c4159b20 0x7fa1c406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1c9446640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 0x7fa1c406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1c9446640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c406f890 con 0x7fa1c4159f00 2023-10-16T14:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1c9446640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c406c650 con 0x7fa1c4159b20 2023-10-16T14:17:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1c9446640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c406c930 con 0x7fa1c40b0b20 2023-10-16T14:17:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1bbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa1c4159f00 0x7fa1c40b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42442/0 (socket says 172.21.15.29:42442) 2023-10-16T14:17:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.387+0000 7fa1bbfff640 1 -- 172.21.15.29:0/1883959452 learned_addr learned my addr 172.21.15.29:0/1883959452 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c37fe640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 0x7fa1c406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/1883959452 >> v2:172.21.15.170:3300/0 conn(0x7fa1c4159b20 0x7fa1c406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859006235 0 0) 0x7fa1c406f890 con 0x7fa1c4159f00 2023-10-16T14:17:39.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b0005730 con 0x7fa1c4159f00 2023-10-16T14:17:39.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689985387 0 0) 0x7fa1b0005730 con 0x7fa1c4159f00 2023-10-16T14:17:39.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 msgr2=0x7fa1c406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:39.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 0x7fa1c406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 >> v2:172.21.15.170:3300/0 conn(0x7fa1c4159b20 msgr2=0x7fa1c406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:39.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 --2- 172.21.15.29:0/1883959452 >> v2:172.21.15.170:3300/0 conn(0x7fa1c4159b20 0x7fa1c406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c406cc10 con 0x7fa1c4159f00 2023-10-16T14:17:39.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa1ac001670 con 0x7fa1c4159f00 2023-10-16T14:17:39.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 >> v1:172.21.15.29:6789/0 conn(0x7fa1c4159f00 legacy=0x7fa1c40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:39.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.388+0000 7fa1c27fc640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1b0005b80 0x7fa1b0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c27fc640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1b00069d0 0x7fa1b0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c27fc640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 0x7fa1b0007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c406c650 con 0x7fa1b0005b80 2023-10-16T14:17:39.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c406c930 con 0x7fa1b00069d0 2023-10-16T14:17:39.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b000db90 con 0x7fa1c40b0b20 2023-10-16T14:17:39.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1b0005b80 0x7fa1b0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1bbfff640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1b00069d0 0x7fa1b0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c2ffd640 1 -- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 msgr2=0x7fa1b0007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:39.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1c40b0b20 0x7fa1b0007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c2ffd640 1 -- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1b00069d0 msgr2=0x7fa1b0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:39.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1b00069d0 0x7fa1b0006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.389+0000 7fa1c2ffd640 1 -- 172.21.15.29:0/1883959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0005730 con 0x7fa1b0005b80 2023-10-16T14:17:39.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.390+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1b0005b80 0x7fa1b0005fa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4009ac0 tx=0x7fa1b0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9feb69dbb20c20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:39.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.391+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b402c710 con 0x7fa1b0005b80 2023-10-16T14:17:39.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.391+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b402c870 con 0x7fa1b0005b80 2023-10-16T14:17:39.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.391+0000 7fa1c27fc640 1 -- 172.21.15.29:0/1883959452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b402cc00 con 0x7fa1b0005b80 2023-10-16T14:17:39.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c9446640 1 -- 172.21.15.29:0/1883959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1b0005b80 msgr2=0x7fa1b0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:39.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c9446640 1 --2- 172.21.15.29:0/1883959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1b0005b80 0x7fa1b0005fa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4009ac0 tx=0x7fa1b0007a00 comp rx=0 tx=0).stop 2023-10-16T14:17:39.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c37fe640 1 -- 172.21.15.29:0/1883959452 reap_dead start 2023-10-16T14:17:39.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c9446640 1 -- 172.21.15.29:0/1883959452 shutdown_connections 2023-10-16T14:17:39.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c9446640 1 -- 172.21.15.29:0/1883959452 >> 172.21.15.29:0/1883959452 conn(0x7fa1c40cb410 msgr2=0x7fa1c40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:39.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c9446640 1 -- 172.21.15.29:0/1883959452 shutdown_connections 2023-10-16T14:17:39.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.392+0000 7fa1c9446640 1 -- 172.21.15.29:0/1883959452 wait complete. 2023-10-16T14:17:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.394+0000 7fa1c9446640 1 Processor -- start 2023-10-16T14:17:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.394+0000 7fa1c9446640 1 -- start start 2023-10-16T14:17:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c9446640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1b0005b80 0x7fa1c414ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c9446640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c4150370 0x7fa1c40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c9446640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 0x7fa1c40aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c9446640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c414ff50 con 0x7fa1c40acdd0 2023-10-16T14:17:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c9446640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40ad250 con 0x7fa1c4150370 2023-10-16T14:17:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c9446640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40afd70 con 0x7fa1b0005b80 2023-10-16T14:17:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 0x7fa1c40aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 0x7fa1c40aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56470/0 (socket says 172.21.15.29:56470) 2023-10-16T14:17:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1c37fe640 1 -- 172.21.15.29:0/423497312 learned_addr learned my addr 172.21.15.29:0/423497312 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.395+0000 7fa1bbfff640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c4150370 0x7fa1c40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c37fe640 1 -- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1b0005b80 msgr2=0x7fa1c414ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c37fe640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1b0005b80 0x7fa1c414ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c37fe640 1 -- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c4150370 msgr2=0x7fa1c40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c37fe640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c4150370 0x7fa1c40ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c37fe640 1 -- 172.21.15.29:0/423497312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40afa90 con 0x7fa1c40acdd0 2023-10-16T14:17:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c37fe640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 0x7fa1c40aeda0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40c8bd0 tx=0x7fa1bc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc010040 con 0x7fa1c40acdd0 2023-10-16T14:17:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc00bc80 con 0x7fa1c40acdd0 2023-10-16T14:17:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc014400 con 0x7fa1c40acdd0 2023-10-16T14:17:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c414f890 con 0x7fa1c40acdd0 2023-10-16T14:17:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.396+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c41507f0 con 0x7fa1c40acdd0 2023-10-16T14:17:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.397+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c4150090 con 0x7fa1c40acdd0 2023-10-16T14:17:39.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.400+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa1bc014560 con 0x7fa1c40acdd0 2023-10-16T14:17:39.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.401+0000 7fa1bb7fe640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa1a406aea0 0x7fa1a406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:39.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.401+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa1bc089fe0 con 0x7fa1c40acdd0 2023-10-16T14:17:39.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.401+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa1a406aea0 0x7fa1a406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:39.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.403+0000 7fa1c2ffd640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa1a406aea0 0x7fa1a406d360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4036000 tx=0x7fa1b402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:39.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.405+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa1bc055320 con 0x7fa1c40acdd0 2023-10-16T14:17:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:39.598+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa1c4150090 con 0x7fa1c40acdd0 2023-10-16T14:17:40.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.144+0000 7fa1bb7fe640 1 -- 172.21.15.29:0/423497312 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa1c4150090 con 0x7fa1c40acdd0 2023-10-16T14:17:40.149 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:17:40.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.148+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa1a406aea0 msgr2=0x7fa1a406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:40.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.148+0000 7fa1c9446640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa1a406aea0 0x7fa1a406d360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4036000 tx=0x7fa1b402e000 comp rx=0 tx=0).stop 2023-10-16T14:17:40.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.148+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 msgr2=0x7fa1c40aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:40.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.148+0000 7fa1c9446640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 0x7fa1c40aeda0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40c8bd0 tx=0x7fa1bc00b480 comp rx=0 tx=0).stop 2023-10-16T14:17:40.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.149+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 shutdown_connections 2023-10-16T14:17:40.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.149+0000 7fa1c9446640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa1a406aea0 0x7fa1a406d360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.149+0000 7fa1c9446640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1c40acdd0 0x7fa1c40aeda0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.149+0000 7fa1c9446640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1c4150370 0x7fa1c40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.149+0000 7fa1c9446640 1 --2- 172.21.15.29:0/423497312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1b0005b80 0x7fa1c414ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.149+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 >> 172.21.15.29:0/423497312 conn(0x7fa1c40cb410 msgr2=0x7fa1c4156ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:40.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.150+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 shutdown_connections 2023-10-16T14:17:40.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.150+0000 7fa1c9446640 1 -- 172.21.15.29:0/423497312 wait complete. 2023-10-16T14:17:40.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:17:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.379+0000 7f4b133dc640 1 Processor -- start 2023-10-16T14:17:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.379+0000 7f4b133dc640 1 -- start start 2023-10-16T14:17:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.380+0000 7f4b133dc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4b0c06a4b0 0x7f4b0c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.380+0000 7f4b133dc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b0c0c9ca0 0x7f4b0c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.380+0000 7f4b133dc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0cb030 con 0x7f4b0c15ad70 2023-10-16T14:17:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.380+0000 7f4b133dc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0c0cae30 con 0x7f4b0c0c9ca0 2023-10-16T14:17:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.380+0000 7f4b133dc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0c0c91c0 con 0x7f4b0c06a4b0 2023-10-16T14:17:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b0c0c9ca0 0x7f4b0c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b0c0c9ca0 0x7f4b0c0ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56476/0 (socket says 172.21.15.29:56476) 2023-10-16T14:17:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 -- 172.21.15.29:0/1841331725 learned_addr learned my addr 172.21.15.29:0/1841331725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11178640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4b0c15ad70 0x7f4b0c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42452/0 (socket says 172.21.15.29:42452) 2023-10-16T14:17:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b10977640 1 --2- 172.21.15.29:0/1841331725 >> v2:172.21.15.170:3300/0 conn(0x7f4b0c06a4b0 0x7f4b0c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 -- 172.21.15.29:0/1841331725 >> v2:172.21.15.170:3300/0 conn(0x7f4b0c06a4b0 msgr2=0x7f4b0c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 --2- 172.21.15.29:0/1841331725 >> v2:172.21.15.170:3300/0 conn(0x7f4b0c06a4b0 0x7f4b0c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 -- 172.21.15.29:0/1841331725 >> v1:172.21.15.29:6789/0 conn(0x7f4b0c15ad70 legacy=0x7f4b0c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b11979640 1 -- 172.21.15.29:0/1841331725 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c0c94a0 con 0x7f4b0c0c9ca0 2023-10-16T14:17:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4afffff640 1 -- 172.21.15.29:0/1841331725 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478686664 0 0) 0x7f4b0c0cb030 con 0x7f4b0c15ad70 2023-10-16T14:17:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.381+0000 7f4b10977640 1 --2- 172.21.15.29:0/1841331725 >> v2:172.21.15.170:3300/0 conn(0x7f4b0c06a4b0 0x7f4b0c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.382+0000 7f4b11979640 1 --2- 172.21.15.29:0/1841331725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c0ca8f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0ce870 tx=0x7f4b0402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764cc11f7904b332 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4afffff640 1 -- 172.21.15.29:0/1841331725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0402cb10 con 0x7f4b0c0c9ca0 2023-10-16T14:17:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4afffff640 1 -- 172.21.15.29:0/1841331725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0402cc70 con 0x7f4b0c0c9ca0 2023-10-16T14:17:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4afffff640 1 -- 172.21.15.29:0/1841331725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b040343f0 con 0x7f4b0c0c9ca0 2023-10-16T14:17:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4b133dc640 1 -- 172.21.15.29:0/1841331725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c0c9ca0 msgr2=0x7f4b0c0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4b133dc640 1 --2- 172.21.15.29:0/1841331725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c0ca8f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0ce870 tx=0x7f4b0402bf70 comp rx=0 tx=0).stop 2023-10-16T14:17:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4b133dc640 1 -- 172.21.15.29:0/1841331725 shutdown_connections 2023-10-16T14:17:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4b133dc640 1 --2- 172.21.15.29:0/1841331725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c0ca8f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4b133dc640 1 --2- 172.21.15.29:0/1841331725 >> v2:172.21.15.170:3300/0 conn(0x7f4b0c06a4b0 0x7f4b0c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.383+0000 7f4b133dc640 1 -- 172.21.15.29:0/1841331725 >> 172.21.15.29:0/1841331725 conn(0x7f4b0c0ce9b0 msgr2=0x7f4b0c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.384+0000 7f4b133dc640 1 -- 172.21.15.29:0/1841331725 shutdown_connections 2023-10-16T14:17:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.384+0000 7f4b133dc640 1 -- 172.21.15.29:0/1841331725 wait complete. 2023-10-16T14:17:40.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.385+0000 7f4b133dc640 1 Processor -- start 2023-10-16T14:17:40.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.386+0000 7f4b133dc640 1 -- start start 2023-10-16T14:17:40.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b133dc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b0c06a4b0 0x7f4b0c0afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:40.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b133dc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b133dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 0x7f4b0c0c2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b133dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c1595c0 con 0x7f4b0c15ad70 2023-10-16T14:17:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b133dc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0c33f0 con 0x7f4b0c0c9ca0 2023-10-16T14:17:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b133dc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0c5cb0 con 0x7f4b0c06a4b0 2023-10-16T14:17:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 0x7f4b0c0c2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:40.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 0x7f4b0c0c2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56486/0 (socket says 172.21.15.29:56486) 2023-10-16T14:17:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 -- 172.21.15.29:0/149777009 learned_addr learned my addr 172.21.15.29:0/149777009 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11178640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b0c06a4b0 0x7f4b0c0afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 -- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b0c06a4b0 msgr2=0x7f4b0c0afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b0c06a4b0 0x7f4b0c0afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 -- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0c0c9ca0 msgr2=0x7f4b0c159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b10977640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.387+0000 7f4b11979640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4b11979640 1 -- 172.21.15.29:0/149777009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04009b30 con 0x7f4b0c15ad70 2023-10-16T14:17:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4b11178640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b0c06a4b0 0x7f4b0c0afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4b11979640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 0x7f4b0c0c2eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b04036040 tx=0x7f4b04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04009510 con 0x7f4b0c15ad70 2023-10-16T14:17:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0403e450 con 0x7f4b0c15ad70 2023-10-16T14:17:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c0c3ec0 con 0x7f4b0c15ad70 2023-10-16T14:17:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0403e800 con 0x7f4b0c15ad70 2023-10-16T14:17:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.388+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c0c3900 con 0x7f4b0c15ad70 2023-10-16T14:17:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.390+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c0c5a80 con 0x7f4b0c15ad70 2023-10-16T14:17:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.391+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4b0403e960 con 0x7f4b0c15ad70 2023-10-16T14:17:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.392+0000 7f4afdffb640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4af006ae70 0x7f4af006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.392+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4b040ac4c0 con 0x7f4b0c15ad70 2023-10-16T14:17:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.392+0000 7f4b11178640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4af006ae70 0x7f4af006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.393+0000 7f4b11178640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4af006ae70 0x7f4af006d330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4b08000f60 tx=0x7f4b08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.397+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4b0c0c5a80 con 0x7f4b0c15ad70 2023-10-16T14:17:40.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:40.628+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4b0c0c5a80 con 0x7f4b0c15ad70 2023-10-16T14:17:41.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.172+0000 7f4afdffb640 1 -- 172.21.15.29:0/149777009 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4b0c0c5a80 con 0x7f4b0c15ad70 2023-10-16T14:17:41.177 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 4 and data pool 5 2023-10-16T14:17:41.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4af006ae70 msgr2=0x7f4af006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4af006ae70 0x7f4af006d330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4b08000f60 tx=0x7f4b08009040 comp rx=0 tx=0).stop 2023-10-16T14:17:41.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 msgr2=0x7f4b0c0c2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 0x7f4b0c0c2eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b04036040 tx=0x7f4b04004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:41.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 shutdown_connections 2023-10-16T14:17:41.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4af006ae70 0x7f4af006d330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b0c15ad70 0x7f4b0c0c2eb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b0c0c9ca0 0x7f4b0c159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 --2- 172.21.15.29:0/149777009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b0c06a4b0 0x7f4b0c0afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.176+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 >> 172.21.15.29:0/149777009 conn(0x7f4b0c0ce9b0 msgr2=0x7f4b0c0cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:41.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.177+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 shutdown_connections 2023-10-16T14:17:41.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.177+0000 7f4b133dc640 1 -- 172.21.15.29:0/149777009 wait complete. 2023-10-16T14:17:41.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:41.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.394+0000 7f5d42f67640 1 Processor -- start 2023-10-16T14:17:41.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.394+0000 7f5d42f67640 1 -- start start 2023-10-16T14:17:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.395+0000 7f5d42f67640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.395+0000 7f5d42f67640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d3c0adc00 0x7f5d3c070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.395+0000 7f5d42f67640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0ae1a0 con 0x7f5d3c160c80 2023-10-16T14:17:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.395+0000 7f5d42f67640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c0ae480 con 0x7f5d3c0adc00 2023-10-16T14:17:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.395+0000 7f5d42f67640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c0ae760 con 0x7f5d3c0ad7a0 2023-10-16T14:17:41.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d3c0adc00 0x7f5d3c070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d3c0adc00 0x7f5d3c070780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56502/0 (socket says 172.21.15.29:56502) 2023-10-16T14:17:41.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 -- 172.21.15.29:0/960143163 learned_addr learned my addr 172.21.15.29:0/960143163 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:41.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d41764640 1 --2- 172.21.15.29:0/960143163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 -- 172.21.15.29:0/960143163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 msgr2=0x7f5d3c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:41.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 --2- 172.21.15.29:0/960143163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 -- 172.21.15.29:0/960143163 >> v1:172.21.15.170:6789/0 conn(0x7f5d3c160c80 legacy=0x7f5d3c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:41.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.396+0000 7f5d42766640 1 -- 172.21.15.29:0/960143163 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0aea40 con 0x7f5d3c0adc00 2023-10-16T14:17:41.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.397+0000 7f5d42766640 1 --2- 172.21.15.29:0/960143163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c070780 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009e50 tx=0x7f5d3402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cecd29a2c3a5fe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:41.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.398+0000 7f5d40f63640 1 -- 172.21.15.29:0/960143163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3402ca70 con 0x7f5d3c0adc00 2023-10-16T14:17:41.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.398+0000 7f5d40f63640 1 -- 172.21.15.29:0/960143163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3402cbd0 con 0x7f5d3c0adc00 2023-10-16T14:17:41.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.398+0000 7f5d40f63640 1 -- 172.21.15.29:0/960143163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d340343f0 con 0x7f5d3c0adc00 2023-10-16T14:17:41.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 -- 172.21.15.29:0/960143163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c0adc00 msgr2=0x7f5d3c070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 --2- 172.21.15.29:0/960143163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c070780 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009e50 tx=0x7f5d3402bed0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 -- 172.21.15.29:0/960143163 shutdown_connections 2023-10-16T14:17:41.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 --2- 172.21.15.29:0/960143163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c070780 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 --2- 172.21.15.29:0/960143163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 -- 172.21.15.29:0/960143163 >> 172.21.15.29:0/960143163 conn(0x7f5d3c0cc230 msgr2=0x7f5d3c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:41.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 -- 172.21.15.29:0/960143163 shutdown_connections 2023-10-16T14:17:41.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.399+0000 7f5d42f67640 1 -- 172.21.15.29:0/960143163 wait complete. 2023-10-16T14:17:41.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.401+0000 7f5d42f67640 1 Processor -- start 2023-10-16T14:17:41.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.401+0000 7f5d42f67640 1 -- start start 2023-10-16T14:17:41.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42f67640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c144a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42f67640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c1422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42f67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 0x7f5d3c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42f67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c142cc0 con 0x7f5d3c160c80 2023-10-16T14:17:41.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42f67640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c142fa0 con 0x7f5d3c0adc00 2023-10-16T14:17:41.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42f67640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c143280 con 0x7f5d3c0ad7a0 2023-10-16T14:17:41.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 0x7f5d3c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 0x7f5d3c0cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56512/0 (socket says 172.21.15.29:56512) 2023-10-16T14:17:41.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 -- 172.21.15.29:0/3694347826 learned_addr learned my addr 172.21.15.29:0/3694347826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:41.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d41764640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c1422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d41f65640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c144a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 -- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 msgr2=0x7f5d3c144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c144a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 -- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d3c0adc00 msgr2=0x7f5d3c1422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.402+0000 7f5d42766640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c1422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d42766640 1 -- 172.21.15.29:0/3694347826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d34009b30 con 0x7f5d3c160c80 2023-10-16T14:17:41.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d41f65640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:41.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d41764640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c1422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:41.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d42766640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 0x7f5d3c0cde30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009e50 tx=0x7f5d34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:41.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34009510 con 0x7f5d3c160c80 2023-10-16T14:17:41.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3403e450 con 0x7f5d3c160c80 2023-10-16T14:17:41.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3403e7e0 con 0x7f5d3c160c80 2023-10-16T14:17:41.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c143560 con 0x7f5d3c160c80 2023-10-16T14:17:41.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.403+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c15c4c0 con 0x7f5d3c160c80 2023-10-16T14:17:41.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.405+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c1f2830 con 0x7f5d3c160c80 2023-10-16T14:17:41.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.405+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5d3403e940 con 0x7f5d3c160c80 2023-10-16T14:17:41.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.406+0000 7f5d327fc640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5d2006ae70 0x7f5d2006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.406+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d340ac460 con 0x7f5d3c160c80 2023-10-16T14:17:41.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.407+0000 7f5d41f65640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5d2006ae70 0x7f5d2006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.408+0000 7f5d41f65640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5d2006ae70 0x7f5d2006d330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800a830 tx=0x7f5d38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:41.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.411+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5d34077470 con 0x7f5d3c160c80 2023-10-16T14:17:41.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.606+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d3c04e9b0 con 0x7f5d3c160c80 2023-10-16T14:17:41.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.608+0000 7f5d327fc640 1 -- 172.21.15.29:0/3694347826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f5d3407aa90 con 0x7f5d3c160c80 2023-10-16T14:17:41.613 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:41.613 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:40.631071+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5d2006ae70 msgr2=0x7f5d2006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5d2006ae70 0x7f5d2006d330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800a830 tx=0x7f5d38008040 comp rx=0 tx=0).stop 2023-10-16T14:17:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 msgr2=0x7f5d3c0cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 0x7f5d3c0cde30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009e50 tx=0x7f5d34004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 shutdown_connections 2023-10-16T14:17:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5d2006ae70 0x7f5d2006d330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d3c160c80 0x7f5d3c0cde30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d3c0adc00 0x7f5d3c1422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 --2- 172.21.15.29:0/3694347826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d3c0ad7a0 0x7f5d3c144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 >> 172.21.15.29:0/3694347826 conn(0x7f5d3c0cc230 msgr2=0x7f5d3c15ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 shutdown_connections 2023-10-16T14:17:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.611+0000 7f5d42f67640 1 -- 172.21.15.29:0/3694347826 wait complete. 2023-10-16T14:17:41.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:41.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.851+0000 7fa554139640 1 Processor -- start 2023-10-16T14:17:41.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.852+0000 7fa554139640 1 -- start start 2023-10-16T14:17:41.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.853+0000 7fa554139640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa54c0b3070 0x7fa54c0cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.853+0000 7fa554139640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0b3940 con 0x7fa54c0b3e70 2023-10-16T14:17:41.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.853+0000 7fa554139640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0b45e0 con 0x7fa54c0d1b10 2023-10-16T14:17:41.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.853+0000 7fa554139640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c0b3b90 con 0x7fa54c0b3070 2023-10-16T14:17:41.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.853+0000 7fa551ed5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa54c0b3070 0x7fa54c0cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.853+0000 7fa5516d4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa54c0b3e70 0x7fa54c0c6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42454/0 (socket says 172.21.15.29:42454) 2023-10-16T14:17:41.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.854+0000 7fa5516d4640 1 -- 172.21.15.29:0/315414505 learned_addr learned my addr 172.21.15.29:0/315414505 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:41.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.854+0000 7fa551ed5640 1 -- 172.21.15.29:0/315414505 >> v1:172.21.15.170:6789/0 conn(0x7fa54c0d1b10 legacy=0x7fa54c0c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:41.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.854+0000 7fa551ed5640 1 -- 172.21.15.29:0/315414505 >> v1:172.21.15.29:6789/0 conn(0x7fa54c0b3e70 legacy=0x7fa54c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.854+0000 7fa551ed5640 1 -- 172.21.15.29:0/315414505 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0d3840 con 0x7fa54c0b3070 2023-10-16T14:17:41.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.855+0000 7fa551ed5640 1 --2- 172.21.15.29:0/315414505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c0cf500 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa548009790 tx=0x7fa54802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759a252adf95804e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:41.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.855+0000 7fa550ed3640 1 -- 172.21.15.29:0/315414505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54802d040 con 0x7fa54c0b3070 2023-10-16T14:17:41.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.855+0000 7fa550ed3640 1 -- 172.21.15.29:0/315414505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548031810 con 0x7fa54c0b3070 2023-10-16T14:17:41.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.855+0000 7fa550ed3640 1 -- 172.21.15.29:0/315414505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548031ba0 con 0x7fa54c0b3070 2023-10-16T14:17:41.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 -- 172.21.15.29:0/315414505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 msgr2=0x7fa54c0cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 --2- 172.21.15.29:0/315414505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c0cf500 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa548009790 tx=0x7fa54802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 -- 172.21.15.29:0/315414505 shutdown_connections 2023-10-16T14:17:41.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 --2- 172.21.15.29:0/315414505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c0cf500 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 -- 172.21.15.29:0/315414505 >> 172.21.15.29:0/315414505 conn(0x7fa54c0d2d20 msgr2=0x7fa54c0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:41.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 -- 172.21.15.29:0/315414505 shutdown_connections 2023-10-16T14:17:41.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.856+0000 7fa554139640 1 -- 172.21.15.29:0/315414505 wait complete. 2023-10-16T14:17:41.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.858+0000 7fa554139640 1 Processor -- start 2023-10-16T14:17:41.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.858+0000 7fa554139640 1 -- start start 2023-10-16T14:17:41.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa554139640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c15dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa554139640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa54c0b3e70 0x7fa54c0cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa554139640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa54c0d1b10 0x7fa54c0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa554139640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c15dd40 con 0x7fa54c0b3070 2023-10-16T14:17:41.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa554139640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0c9e90 con 0x7fa54c0b3e70 2023-10-16T14:17:41.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa554139640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0ca170 con 0x7fa54c0d1b10 2023-10-16T14:17:41.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa551ed5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c15dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa551ed5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c15dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56524/0 (socket says 172.21.15.29:56524) 2023-10-16T14:17:41.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.859+0000 7fa551ed5640 1 -- 172.21.15.29:0/304684958 learned_addr learned my addr 172.21.15.29:0/304684958 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:41.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa5516d4640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa54c0b3e70 0x7fa54c0cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa551ed5640 1 -- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa54c0d1b10 msgr2=0x7fa54c0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa551ed5640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa54c0d1b10 0x7fa54c0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa551ed5640 1 -- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa54c0b3e70 msgr2=0x7fa54c0cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:41.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa551ed5640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa54c0b3e70 0x7fa54c0cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:41.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa551ed5640 1 -- 172.21.15.29:0/304684958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5480093a0 con 0x7fa54c0b3070 2023-10-16T14:17:41.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa551ed5640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c15dec0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa548000c00 tx=0x7fa5480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:41.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54803ad60 con 0x7fa54c0b3070 2023-10-16T14:17:41.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548005990 con 0x7fa54c0b3070 2023-10-16T14:17:41.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548005d40 con 0x7fa54c0b3070 2023-10-16T14:17:41.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.860+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c0ca450 con 0x7fa54c0b3070 2023-10-16T14:17:41.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.861+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c15e400 con 0x7fa54c0b3070 2023-10-16T14:17:41.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.862+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c0ca970 con 0x7fa54c0b3070 2023-10-16T14:17:41.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.863+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa548005ea0 con 0x7fa54c0b3070 2023-10-16T14:17:41.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.864+0000 7fa5427fc640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa53006aec0 0x7fa53006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:41.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.865+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5480aba30 con 0x7fa54c0b3070 2023-10-16T14:17:41.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.865+0000 7fa5516d4640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa53006aec0 0x7fa53006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:41.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.866+0000 7fa5516d4640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa53006aec0 0x7fa53006d380 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0d3750 tx=0x7fa53c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:41.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:41.869+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa54c0ca970 con 0x7fa54c0b3070 2023-10-16T14:17:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.064+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa54c0ca970 con 0x7fa54c0b3070 2023-10-16T14:17:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.066+0000 7fa5427fc640 1 -- 172.21.15.29:0/304684958 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fa54807a650 con 0x7fa54c0b3070 2023-10-16T14:17:42.071 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:42.071 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:41.809525+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa53006aec0 msgr2=0x7fa53006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa53006aec0 0x7fa53006d380 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0d3750 tx=0x7fa53c009040 comp rx=0 tx=0).stop 2023-10-16T14:17:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 msgr2=0x7fa54c15dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c15dec0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa548000c00 tx=0x7fa5480046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 shutdown_connections 2023-10-16T14:17:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fa53006aec0 0x7fa53006d380 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa54c0d1b10 0x7fa54c0cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa54c0b3e70 0x7fa54c0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 --2- 172.21.15.29:0/304684958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa54c0b3070 0x7fa54c15dec0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 >> 172.21.15.29:0/304684958 conn(0x7fa54c0d2d20 msgr2=0x7fa54c15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 shutdown_connections 2023-10-16T14:17:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.069+0000 7fa554139640 1 -- 172.21.15.29:0/304684958 wait complete. 2023-10-16T14:17:42.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:17:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.309+0000 7fb59b31a640 1 Processor -- start 2023-10-16T14:17:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.309+0000 7fb59b31a640 1 -- start start 2023-10-16T14:17:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59b31a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb59406c040 0x7fb59406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59b31a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5940cda80 0x7fb59415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59b31a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59406c960 con 0x7fb5940ad970 2023-10-16T14:17:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59b31a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59415b9d0 con 0x7fb5940cda80 2023-10-16T14:17:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59b31a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59415bcb0 con 0x7fb59406c040 2023-10-16T14:17:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59ab19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5940cda80 0x7fb59415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59ab19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5940cda80 0x7fb59415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56530/0 (socket says 172.21.15.29:56530) 2023-10-16T14:17:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb59ab19640 1 -- 172.21.15.29:0/2307730774 learned_addr learned my addr 172.21.15.29:0/2307730774 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.311+0000 7fb599b17640 1 --1- 172.21.15.29:0/2307730774 >> v1:172.21.15.29:6789/0 conn(0x7fb5940ad970 0x7fb59415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42466/0 (socket says 172.21.15.29:42466) 2023-10-16T14:17:42.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb59a318640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb59406c040 0x7fb59406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb59ab19640 1 -- 172.21.15.29:0/2307730774 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb59406c040 msgr2=0x7fb59406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb59ab19640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb59406c040 0x7fb59406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb59ab19640 1 -- 172.21.15.29:0/2307730774 >> v1:172.21.15.29:6789/0 conn(0x7fb5940ad970 legacy=0x7fb59415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:42.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb59ab19640 1 -- 172.21.15.29:0/2307730774 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59415bf90 con 0x7fb5940cda80 2023-10-16T14:17:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb599316640 1 -- 172.21.15.29:0/2307730774 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401280255 0 0) 0x7fb59406c960 con 0x7fb5940ad970 2023-10-16T14:17:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.312+0000 7fb59a318640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb59406c040 0x7fb59406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:42.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.313+0000 7fb59ab19640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5940cda80 0x7fb59415ed90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009e50 tx=0x7fb58c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e36bb721dbedbc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.313+0000 7fb599316640 1 -- 172.21.15.29:0/2307730774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c02cb00 con 0x7fb5940cda80 2023-10-16T14:17:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.313+0000 7fb599316640 1 -- 172.21.15.29:0/2307730774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c02cc60 con 0x7fb5940cda80 2023-10-16T14:17:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.313+0000 7fb599316640 1 -- 172.21.15.29:0/2307730774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c0343f0 con 0x7fb5940cda80 2023-10-16T14:17:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 -- 172.21.15.29:0/2307730774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5940cda80 msgr2=0x7fb59415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5940cda80 0x7fb59415ed90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009e50 tx=0x7fb58c02bf60 comp rx=0 tx=0).stop 2023-10-16T14:17:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 -- 172.21.15.29:0/2307730774 shutdown_connections 2023-10-16T14:17:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5940cda80 0x7fb59415ed90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 --2- 172.21.15.29:0/2307730774 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb59406c040 0x7fb59406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 -- 172.21.15.29:0/2307730774 >> 172.21.15.29:0/2307730774 conn(0x7fb5940ce6a0 msgr2=0x7fb5940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 -- 172.21.15.29:0/2307730774 shutdown_connections 2023-10-16T14:17:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.314+0000 7fb59b31a640 1 -- 172.21.15.29:0/2307730774 wait complete. 2023-10-16T14:17:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.316+0000 7fb59b31a640 1 Processor -- start 2023-10-16T14:17:42.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.316+0000 7fb59b31a640 1 -- start start 2023-10-16T14:17:42.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.317+0000 7fb59b31a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 0x7fb594154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:42.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.317+0000 7fb59b31a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5940ad970 0x7fb5941571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:42.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.317+0000 7fb59b31a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5940cda80 0x7fb5940c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:42.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.317+0000 7fb59b31a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594155380 con 0x7fb59406c040 2023-10-16T14:17:42.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.317+0000 7fb59b31a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594157720 con 0x7fb5940cda80 2023-10-16T14:17:42.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.317+0000 7fb59b31a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5941f1700 con 0x7fb5940ad970 2023-10-16T14:17:42.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 0x7fb594154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:42.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 0x7fb594154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56544/0 (socket says 172.21.15.29:56544) 2023-10-16T14:17:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 -- 172.21.15.29:0/3781391094 learned_addr learned my addr 172.21.15.29:0/3781391094 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59ab19640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5940cda80 0x7fb5940c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb599b17640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5940ad970 0x7fb5941571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:42.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 -- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5940ad970 msgr2=0x7fb5941571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5940ad970 0x7fb5941571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 -- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5940cda80 msgr2=0x7fb5940c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5940cda80 0x7fb5940c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:42.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb59a318640 1 -- 172.21.15.29:0/3781391094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c009b30 con 0x7fb59406c040 2023-10-16T14:17:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.318+0000 7fb599b17640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5940ad970 0x7fb5941571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.319+0000 7fb59a318640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 0x7fb594154320 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb59000d7d0 tx=0x7fb59000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.319+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590004030 con 0x7fb59406c040 2023-10-16T14:17:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.319+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb590004190 con 0x7fb59406c040 2023-10-16T14:17:42.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.319+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590004520 con 0x7fb59406c040 2023-10-16T14:17:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.319+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb594155050 con 0x7fb59406c040 2023-10-16T14:17:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.319+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5941f0540 con 0x7fb59406c040 2023-10-16T14:17:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.320+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59406b330 con 0x7fb59406c040 2023-10-16T14:17:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.321+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb5900096a0 con 0x7fb59406c040 2023-10-16T14:17:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.322+0000 7fb58affd640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb57806ae70 0x7fb57806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:42.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.322+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5900897d0 con 0x7fb59406c040 2023-10-16T14:17:42.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.323+0000 7fb599b17640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb57806ae70 0x7fb57806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:42.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.324+0000 7fb599b17640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb57806ae70 0x7fb57806d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb5940accf0 tx=0x7fb584009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:42.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.326+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb590054870 con 0x7fb59406c040 2023-10-16T14:17:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:42.547+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb59404e9b0 con 0x7fb59406c040 2023-10-16T14:17:42.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:17:42.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:17:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.196+0000 7fb58affd640 1 -- 172.21.15.29:0/3781391094 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb590057e90 con 0x7fb59406c040 2023-10-16T14:17:43.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.200+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb57806ae70 msgr2=0x7fb57806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.200+0000 7fb59b31a640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb57806ae70 0x7fb57806d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb5940accf0 tx=0x7fb584009040 comp rx=0 tx=0).stop 2023-10-16T14:17:43.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.200+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 msgr2=0x7fb594154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.200+0000 7fb59b31a640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 0x7fb594154320 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb59000d7d0 tx=0x7fb59000dc90 comp rx=0 tx=0).stop 2023-10-16T14:17:43.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 shutdown_connections 2023-10-16T14:17:43.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fb57806ae70 0x7fb57806d330 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5940cda80 0x7fb5940c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5940ad970 0x7fb5941571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 --2- 172.21.15.29:0/3781391094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb59406c040 0x7fb594154320 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 >> 172.21.15.29:0/3781391094 conn(0x7fb5940ce6a0 msgr2=0x7fb59415dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 shutdown_connections 2023-10-16T14:17:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.201+0000 7fb59b31a640 1 -- 172.21.15.29:0/3781391094 wait complete. 2023-10-16T14:17:43.228 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:17:43.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:17:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.459+0000 7f6c1276a640 1 Processor -- start 2023-10-16T14:17:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.459+0000 7f6c1276a640 1 -- start start 2023-10-16T14:17:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c1276a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6c0c15b320 0x7f6c0c0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c1276a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c15b9d0 con 0x7f6c0c06c040 2023-10-16T14:17:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c1276a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c15bcb0 con 0x7f6c0c0ad970 2023-10-16T14:17:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c1276a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c15b7d0 con 0x7f6c0c15b320 2023-10-16T14:17:43.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c11768640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c0c06c040 0x7f6c0c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42468/0 (socket says 172.21.15.29:42468) 2023-10-16T14:17:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c11768640 1 -- 172.21.15.29:0/1126333461 learned_addr learned my addr 172.21.15.29:0/1126333461 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.461+0000 7f6c11f69640 1 --2- 172.21.15.29:0/1126333461 >> v2:172.21.15.170:3300/0 conn(0x7f6c0c15b320 0x7f6c0c0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.462+0000 7f6c03fff640 1 -- 172.21.15.29:0/1126333461 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405586675 0 0) 0x7f6c0c15b9d0 con 0x7f6c0c06c040 2023-10-16T14:17:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.462+0000 7f6c03fff640 1 -- 172.21.15.29:0/1126333461 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf8003520 con 0x7f6c0c06c040 2023-10-16T14:17:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.462+0000 7f6c11f69640 1 -- 172.21.15.29:0/1126333461 >> v1:172.21.15.170:6789/0 conn(0x7f6c0c0ad970 legacy=0x7f6c0c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.462+0000 7f6c11f69640 1 -- 172.21.15.29:0/1126333461 >> v1:172.21.15.29:6789/0 conn(0x7f6c0c06c040 legacy=0x7f6c0c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.462+0000 7f6c11f69640 1 -- 172.21.15.29:0/1126333461 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c15bf90 con 0x7f6c0c15b320 2023-10-16T14:17:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.464+0000 7f6c11f69640 1 --2- 172.21.15.29:0/1126333461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c15b320 0x7f6c0c0cd180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009dc0 tx=0x7f6c0402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7c4e9eba2a2d009 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.464+0000 7f6c03fff640 1 -- 172.21.15.29:0/1126333461 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0402cce0 con 0x7f6c0c15b320 2023-10-16T14:17:43.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c03fff640 1 -- 172.21.15.29:0/1126333461 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0402ce40 con 0x7f6c0c15b320 2023-10-16T14:17:43.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c03fff640 1 -- 172.21.15.29:0/1126333461 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04034660 con 0x7f6c0c15b320 2023-10-16T14:17:43.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 -- 172.21.15.29:0/1126333461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c15b320 msgr2=0x7f6c0c0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 --2- 172.21.15.29:0/1126333461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c15b320 0x7f6c0c0cd180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009dc0 tx=0x7f6c0402bf30 comp rx=0 tx=0).stop 2023-10-16T14:17:43.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 -- 172.21.15.29:0/1126333461 shutdown_connections 2023-10-16T14:17:43.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 --2- 172.21.15.29:0/1126333461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c15b320 0x7f6c0c0cd180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 -- 172.21.15.29:0/1126333461 >> 172.21.15.29:0/1126333461 conn(0x7f6c0c0ce6a0 msgr2=0x7f6c0c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:43.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 -- 172.21.15.29:0/1126333461 shutdown_connections 2023-10-16T14:17:43.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.465+0000 7f6c1276a640 1 -- 172.21.15.29:0/1126333461 wait complete. 2023-10-16T14:17:43.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.467+0000 7f6c1276a640 1 Processor -- start 2023-10-16T14:17:43.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.467+0000 7f6c1276a640 1 -- start start 2023-10-16T14:17:43.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c1276a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c06c040 0x7f6c0c1f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:43.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c1276a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 0x7f6c0c0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:43.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c1276a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c0c15b320 0x7f6c0c0cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:43.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c1276a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c158510 con 0x7f6c0c0ad970 2023-10-16T14:17:43.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c1276a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0b3cd0 con 0x7f6c0c06c040 2023-10-16T14:17:43.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c1276a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0b1850 con 0x7f6c0c15b320 2023-10-16T14:17:43.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c10f67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 0x7f6c0c0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:43.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c10f67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 0x7f6c0c0cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56546/0 (socket says 172.21.15.29:56546) 2023-10-16T14:17:43.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.468+0000 7f6c10f67640 1 -- 172.21.15.29:0/2349744112 learned_addr learned my addr 172.21.15.29:0/2349744112 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:43.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c11768640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c06c040 0x7f6c0c1f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:43.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c11f69640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c0c15b320 0x7f6c0c0cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:43.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c10f67640 1 -- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c0c15b320 msgr2=0x7f6c0c0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c10f67640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c0c15b320 0x7f6c0c0cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c10f67640 1 -- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c06c040 msgr2=0x7f6c0c1f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:43.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c10f67640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c06c040 0x7f6c0c1f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:43.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c10f67640 1 -- 172.21.15.29:0/2349744112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c04009a70 con 0x7f6c0c0ad970 2023-10-16T14:17:43.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c11f69640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c0c15b320 0x7f6c0c0cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:43.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c10f67640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 0x7f6c0c0cd130 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c430 tx=0x7f6bfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:43.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.469+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc0052d0 con 0x7f6c0c0ad970 2023-10-16T14:17:43.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.470+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc005430 con 0x7f6c0c0ad970 2023-10-16T14:17:43.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.470+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c0b3f00 con 0x7f6c0c0ad970 2023-10-16T14:17:43.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.470+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc003830 con 0x7f6c0c0ad970 2023-10-16T14:17:43.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.470+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c0b44c0 con 0x7f6c0c0ad970 2023-10-16T14:17:43.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.471+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c06b330 con 0x7f6c0c0ad970 2023-10-16T14:17:43.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.472+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6bfc003990 con 0x7f6c0c0ad970 2023-10-16T14:17:43.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.473+0000 7f6c01ffb640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6bf006ae70 0x7f6bf006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:43.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.473+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bfc088ca0 con 0x7f6c0c0ad970 2023-10-16T14:17:43.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.474+0000 7f6c11768640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6bf006ae70 0x7f6bf006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:43.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.475+0000 7f6c11768640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6bf006ae70 0x7f6bf006d330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c080011e0 tx=0x7f6c08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:43.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.477+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6bfc053cb0 con 0x7f6c0c0ad970 2023-10-16T14:17:43.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:43.713+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6c0c0cc270 con 0x7f6c0c0ad970 2023-10-16T14:17:44.132 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T14:17:44.128+0000 7f25e5e77640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:17:44.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.200+0000 7f6c01ffb640 1 -- 172.21.15.29:0/2349744112 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6bfc0170c0 con 0x7f6c0c0ad970 2023-10-16T14:17:44.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6bf006ae70 msgr2=0x7f6bf006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6bf006ae70 0x7f6bf006d330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c080011e0 tx=0x7f6c08009040 comp rx=0 tx=0).stop 2023-10-16T14:17:44.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 msgr2=0x7f6c0c0cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 0x7f6c0c0cd130 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c430 tx=0x7f6bfc00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 shutdown_connections 2023-10-16T14:17:44.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6bf006ae70 0x7f6bf006d330 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0c0ad970 0x7f6c0c0cd130 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c0c15b320 0x7f6c0c0cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 --2- 172.21.15.29:0/2349744112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c0c06c040 0x7f6c0c1f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.203+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 >> 172.21.15.29:0/2349744112 conn(0x7f6c0c0ce6a0 msgr2=0x7f6c0c15cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:44.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.204+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 shutdown_connections 2023-10-16T14:17:44.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.204+0000 7f6c1276a640 1 -- 172.21.15.29:0/2349744112 wait complete. 2023-10-16T14:17:44.229 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:17:44.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:17:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.450+0000 7fc5e4e7c640 1 Processor -- start 2023-10-16T14:17:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.450+0000 7fc5e4e7c640 1 -- start start 2023-10-16T14:17:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.452+0000 7fc5e4e7c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc5e00b0b20 0x7fc5e01615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.452+0000 7fc5e4e7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e015e3e0 con 0x7fc5e015d3f0 2023-10-16T14:17:44.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.452+0000 7fc5e4e7c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e00cd4a0 con 0x7fc5e00cd0c0 2023-10-16T14:17:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.452+0000 7fc5e4e7c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e015e100 con 0x7fc5e00b0b20 2023-10-16T14:17:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.452+0000 7fc5ded9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5e015d3f0 0x7fc5e015d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54618/0 (socket says 172.21.15.29:54618) 2023-10-16T14:17:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.452+0000 7fc5ded9d640 1 -- 172.21.15.29:0/1180802349 learned_addr learned my addr 172.21.15.29:0/1180802349 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:44.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5d7fff640 1 --2- 172.21.15.29:0/1180802349 >> v2:172.21.15.170:3300/0 conn(0x7fc5e00b0b20 0x7fc5e01615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598994822 0 0) 0x7fc5e015e3e0 con 0x7fc5e015d3f0 2023-10-16T14:17:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5cc003520 con 0x7fc5e015d3f0 2023-10-16T14:17:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811899086 0 0) 0x7fc5cc003520 con 0x7fc5e015d3f0 2023-10-16T14:17:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 >> v2:172.21.15.170:3300/0 conn(0x7fc5e00b0b20 msgr2=0x7fc5e01615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 --2- 172.21.15.29:0/1180802349 >> v2:172.21.15.170:3300/0 conn(0x7fc5e00b0b20 0x7fc5e01615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 >> v1:172.21.15.170:6789/0 conn(0x7fc5e00cd0c0 legacy=0x7fc5e00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0160b10 con 0x7fc5e015d3f0 2023-10-16T14:17:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc5d80019f0 con 0x7fc5e015d3f0 2023-10-16T14:17:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 >> v1:172.21.15.29:6789/0 conn(0x7fc5e015d3f0 legacy=0x7fc5e015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5cc003970 0x7fc5cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.453+0000 7fc5ddd9b640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5cc0047c0 0x7fc5cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ddd9b640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e015e100 con 0x7fc5cc005180 2023-10-16T14:17:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc00c120 con 0x7fc5cc003970 2023-10-16T14:17:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc00c2e0 con 0x7fc5cc0047c0 2023-10-16T14:17:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ded9d640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5d7fff640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5cc0047c0 0x7fc5cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5de59c640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5cc003970 0x7fc5cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ded9d640 1 -- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5cc0047c0 msgr2=0x7fc5cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:44.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ded9d640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5cc0047c0 0x7fc5cc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ded9d640 1 -- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5cc003970 msgr2=0x7fc5cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:44.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ded9d640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5cc003970 0x7fc5cc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.454+0000 7fc5ded9d640 1 -- 172.21.15.29:0/1180802349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc003520 con 0x7fc5cc005180 2023-10-16T14:17:44.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.455+0000 7fc5ded9d640 1 -- 172.21.15.29:0/1180802349 reap_dead start 2023-10-16T14:17:44.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.455+0000 7fc5ded9d640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5cc00b980 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800a830 tx=0x7fc5cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c019e22fc63d05c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.456+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80072d0 con 0x7fc5cc005180 2023-10-16T14:17:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.456+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d802f070 con 0x7fc5cc005180 2023-10-16T14:17:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.456+0000 7fc5ddd9b640 1 -- 172.21.15.29:0/1180802349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80076b0 con 0x7fc5cc005180 2023-10-16T14:17:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.456+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/1180802349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 msgr2=0x7fc5cc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.456+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5cc00b980 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800a830 tx=0x7fc5cc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.457+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/1180802349 shutdown_connections 2023-10-16T14:17:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.457+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/1180802349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5cc00b980 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.457+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/1180802349 >> 172.21.15.29:0/1180802349 conn(0x7fc5e00cb410 msgr2=0x7fc5e00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.457+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/1180802349 shutdown_connections 2023-10-16T14:17:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.457+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/1180802349 wait complete. 2023-10-16T14:17:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.458+0000 7fc5e4e7c640 1 Processor -- start 2023-10-16T14:17:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 -- start start 2023-10-16T14:17:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5e015fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e015e590 0x7fc5e00c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e01f5c50 0x7fc5e01f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e014cbe0 con 0x7fc5cc005180 2023-10-16T14:17:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e01f6610 con 0x7fc5e015e590 2023-10-16T14:17:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.459+0000 7fc5e4e7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e0160110 con 0x7fc5e01f5c50 2023-10-16T14:17:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5e015fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5e015fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37256/0 (socket says 172.21.15.29:37256) 2023-10-16T14:17:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 -- 172.21.15.29:0/2357896480 learned_addr learned my addr 172.21.15.29:0/2357896480 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5ded9d640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e01f5c50 0x7fc5e01f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 -- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e01f5c50 msgr2=0x7fc5e01f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e01f5c50 0x7fc5e01f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5d7fff640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e015e590 0x7fc5e00c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 -- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e015e590 msgr2=0x7fc5e00c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e015e590 0x7fc5e00c5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5de59c640 1 -- 172.21.15.29:0/2357896480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e015f740 con 0x7fc5cc005180 2023-10-16T14:17:44.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.460+0000 7fc5ded9d640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e01f5c50 0x7fc5e01f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:44.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.461+0000 7fc5de59c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5e015fbd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000e480 tx=0x7fc5d000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:44.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.461+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d00052e0 con 0x7fc5cc005180 2023-10-16T14:17:44.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.461+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d0005440 con 0x7fc5cc005180 2023-10-16T14:17:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.461+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0003830 con 0x7fc5cc005180 2023-10-16T14:17:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.461+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e015eb30 con 0x7fc5cc005180 2023-10-16T14:17:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.461+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e015f0f0 con 0x7fc5cc005180 2023-10-16T14:17:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.462+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e004e160 con 0x7fc5cc005180 2023-10-16T14:17:44.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.463+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc5d0003990 con 0x7fc5cc005180 2023-10-16T14:17:44.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.465+0000 7fc5d6ffd640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc5c006ae70 0x7fc5c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.465+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5d00892f0 con 0x7fc5cc005180 2023-10-16T14:17:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.466+0000 7fc5d7fff640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc5c006ae70 0x7fc5c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.467+0000 7fc5d7fff640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc5c006ae70 0x7fc5c006d330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc5e014ce90 tx=0x7fc5c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:44.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.469+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5d0054390 con 0x7fc5cc005180 2023-10-16T14:17:44.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.702+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e00b4910 con 0x7fc5cc005180 2023-10-16T14:17:44.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.703+0000 7fc5d6ffd640 1 -- 172.21.15.29:0/2357896480 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fc5d00579b0 con 0x7fc5cc005180 2023-10-16T14:17:44.708 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:17:40.632013+0000","modified":"2023-10-16T14:17:43.716037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4720},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4720":{"gid":4720,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6835/2287219370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2287219370},{"type":"v1","addr":"172.21.15.29:6835","nonce":2287219370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T14:17:44.708 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-16T14:17:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc5c006ae70 msgr2=0x7fc5c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc5c006ae70 0x7fc5c006d330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc5e014ce90 tx=0x7fc5c8008040 comp rx=0 tx=0).stop 2023-10-16T14:17:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 msgr2=0x7fc5e015fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5e015fbd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000e480 tx=0x7fc5d000e940 comp rx=0 tx=0).stop 2023-10-16T14:17:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 shutdown_connections 2023-10-16T14:17:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fc5c006ae70 0x7fc5c006d330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e01f5c50 0x7fc5e01f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e015e590 0x7fc5e00c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 --2- 172.21.15.29:0/2357896480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5cc005180 0x7fc5e015fbd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 >> 172.21.15.29:0/2357896480 conn(0x7fc5e00cb410 msgr2=0x7fc5e00c91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:44.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.706+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 shutdown_connections 2023-10-16T14:17:44.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.707+0000 7fc5e4e7c640 1 -- 172.21.15.29:0/2357896480 wait complete. 2023-10-16T14:17:44.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:17:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.953+0000 7f988b59e640 1 Processor -- start 2023-10-16T14:17:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.953+0000 7f988b59e640 1 -- start start 2023-10-16T14:17:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.955+0000 7f988b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f988c15dde0 0x7f988c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.955+0000 7f988b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c070f20 0x7f988c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.955+0000 7f988b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c160a90 con 0x7f988c15c9f0 2023-10-16T14:17:44.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.955+0000 7f988b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0cdec0 con 0x7f988c15dde0 2023-10-16T14:17:44.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.955+0000 7f988b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0cd900 con 0x7f988c070f20 2023-10-16T14:17:44.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.956+0000 7f9889d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f988c15dde0 0x7f988c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.956+0000 7f988a59c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f988c15c9f0 0x7f988c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:36700/0 (socket says 172.21.15.29:36700) 2023-10-16T14:17:44.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.956+0000 7f988a59c640 1 -- 172.21.15.29:0/3562976409 learned_addr learned my addr 172.21.15.29:0/3562976409 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.956+0000 7f988ad9d640 1 --2- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c070f20 0x7f988c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.957+0000 7f9889d9b640 1 -- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c070f20 msgr2=0x7f988c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.957+0000 7f9889d9b640 1 --2- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c070f20 0x7f988c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.957+0000 7f9889d9b640 1 -- 172.21.15.29:0/3562976409 >> v1:172.21.15.170:6789/0 conn(0x7f988c15c9f0 legacy=0x7f988c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.957+0000 7f9889d9b640 1 -- 172.21.15.29:0/3562976409 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0cdbe0 con 0x7f988c15dde0 2023-10-16T14:17:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.958+0000 7f9889d9b640 1 --2- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c15dde0 0x7f988c0709e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f98800097a0 tx=0x7f988002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2daaa5fa8643dc0d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.958+0000 7f988959a640 1 -- 172.21.15.29:0/3562976409 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988002d040 con 0x7f988c15dde0 2023-10-16T14:17:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988959a640 1 -- 172.21.15.29:0/3562976409 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9880031810 con 0x7f988c15dde0 2023-10-16T14:17:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988959a640 1 -- 172.21.15.29:0/3562976409 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880031ba0 con 0x7f988c15dde0 2023-10-16T14:17:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988b59e640 1 -- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c15dde0 msgr2=0x7f988c0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988b59e640 1 --2- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c15dde0 0x7f988c0709e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f98800097a0 tx=0x7f988002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988b59e640 1 -- 172.21.15.29:0/3562976409 shutdown_connections 2023-10-16T14:17:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988b59e640 1 --2- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c070f20 0x7f988c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988b59e640 1 --2- 172.21.15.29:0/3562976409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c15dde0 0x7f988c0709e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.959+0000 7f988b59e640 1 -- 172.21.15.29:0/3562976409 >> 172.21.15.29:0/3562976409 conn(0x7f988c0ce6a0 msgr2=0x7f988c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.960+0000 7f988b59e640 1 -- 172.21.15.29:0/3562976409 shutdown_connections 2023-10-16T14:17:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.960+0000 7f988b59e640 1 -- 172.21.15.29:0/3562976409 wait complete. 2023-10-16T14:17:44.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.961+0000 7f988b59e640 1 Processor -- start 2023-10-16T14:17:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.962+0000 7f988b59e640 1 -- start start 2023-10-16T14:17:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c070f20 0x7f988c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 0x7f988c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c15dde0 0x7f988c158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c069e30 con 0x7f988c15c9f0 2023-10-16T14:17:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c06c260 con 0x7f988c070f20 2023-10-16T14:17:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c06c540 con 0x7f988c15dde0 2023-10-16T14:17:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f9889d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 0x7f988c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f9889d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 0x7f988c0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37272/0 (socket says 172.21.15.29:37272) 2023-10-16T14:17:44.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f9889d9b640 1 -- 172.21.15.29:0/2494662289 learned_addr learned my addr 172.21.15.29:0/2494662289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988a59c640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c070f20 0x7f988c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f988ad9d640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c15dde0 0x7f988c158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f9889d9b640 1 -- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c15dde0 msgr2=0x7f988c158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.963+0000 7f9889d9b640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c15dde0 0x7f988c158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f9889d9b640 1 -- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c070f20 msgr2=0x7f988c06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f9889d9b640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c070f20 0x7f988c06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f9889d9b640 1 -- 172.21.15.29:0/2494662289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9880009400 con 0x7f988c15c9f0 2023-10-16T14:17:44.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f9889d9b640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 0x7f988c0c5770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9880000c00 tx=0x7f98800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:44.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880006e10 con 0x7f988c15c9f0 2023-10-16T14:17:44.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988003ad60 con 0x7f988c15c9f0 2023-10-16T14:17:44.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880044630 con 0x7f988c15c9f0 2023-10-16T14:17:44.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c06c820 con 0x7f988c15c9f0 2023-10-16T14:17:44.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.964+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c06b930 con 0x7f988c15c9f0 2023-10-16T14:17:44.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.965+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c160d70 con 0x7f988c15c9f0 2023-10-16T14:17:44.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.966+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f988004d450 con 0x7f988c15c9f0 2023-10-16T14:17:44.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.967+0000 7f987affd640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f986c06ae70 0x7f986c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:44.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.967+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98800ab7b0 con 0x7f988c15c9f0 2023-10-16T14:17:44.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.968+0000 7f988a59c640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f986c06ae70 0x7f986c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:44.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.969+0000 7f988a59c640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f986c06ae70 0x7f986c06d330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f987c005880 tx=0x7f987c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:44.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:44.972+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f98800768f0 con 0x7f988c15c9f0 2023-10-16T14:17:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.169+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f988c04e9b0 con 0x7f988c15c9f0 2023-10-16T14:17:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.173+0000 7f987affd640 1 -- 172.21.15.29:0/2494662289 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f9880079f10 con 0x7f988c15c9f0 2023-10-16T14:17:45.177 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:45.177 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:17:41.809525+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:17:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.175+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f986c06ae70 msgr2=0x7f986c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.175+0000 7f988b59e640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f986c06ae70 0x7f986c06d330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f987c005880 tx=0x7f987c009040 comp rx=0 tx=0).stop 2023-10-16T14:17:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.175+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 msgr2=0x7f988c0c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.175+0000 7f988b59e640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 0x7f988c0c5770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9880000c00 tx=0x7f98800046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 shutdown_connections 2023-10-16T14:17:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f986c06ae70 0x7f986c06d330 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c15c9f0 0x7f988c0c5770 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f988c15dde0 0x7f988c158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 --2- 172.21.15.29:0/2494662289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f988c070f20 0x7f988c06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 >> 172.21.15.29:0/2494662289 conn(0x7f988c0ce6a0 msgr2=0x7f988c0c14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 shutdown_connections 2023-10-16T14:17:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.176+0000 7f988b59e640 1 -- 172.21.15.29:0/2494662289 wait complete. 2023-10-16T14:17:45.200 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:17:45.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:17:45.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.429+0000 7f6c80ead640 1 Processor -- start 2023-10-16T14:17:45.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.430+0000 7f6c80ead640 1 -- start start 2023-10-16T14:17:45.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.431+0000 7f6c80ead640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6c7c0b4c80 0x7f6c7c167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.431+0000 7f6c80ead640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0d15a0 con 0x7f6c7c15cfc0 2023-10-16T14:17:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.431+0000 7f6c80ead640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0d1880 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.431+0000 7f6c80ead640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c0d1b60 con 0x7f6c7c0b4c80 2023-10-16T14:17:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.432+0000 7f6c73fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c7c15cfc0 0x7f6c7c0b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54620/0 (socket says 172.21.15.29:54620) 2023-10-16T14:17:45.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.432+0000 7f6c73fff640 1 -- 172.21.15.29:0/2716297561 learned_addr learned my addr 172.21.15.29:0/2716297561 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.432+0000 7f6c7bfff640 1 --2- 172.21.15.29:0/2716297561 >> v2:172.21.15.170:3300/0 conn(0x7f6c7c0b4c80 0x7f6c7c167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.432+0000 7f6c7affd640 1 -- 172.21.15.29:0/2716297561 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258613925 0 0) 0x7f6c7c0d15a0 con 0x7f6c7c15cfc0 2023-10-16T14:17:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.432+0000 7f6c7affd640 1 -- 172.21.15.29:0/2716297561 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c68003520 con 0x7f6c7c15cfc0 2023-10-16T14:17:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.433+0000 7f6c7bfff640 1 -- 172.21.15.29:0/2716297561 >> v1:172.21.15.170:6789/0 conn(0x7f6c7c15cbe0 legacy=0x7f6c7c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.433+0000 7f6c7affd640 1 -- 172.21.15.29:0/2716297561 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 999703229 0 0) 0x7f6c68003520 con 0x7f6c7c15cfc0 2023-10-16T14:17:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.433+0000 7f6c7bfff640 1 -- 172.21.15.29:0/2716297561 >> v1:172.21.15.29:6789/0 conn(0x7f6c7c15cfc0 legacy=0x7f6c7c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.433+0000 7f6c7bfff640 1 -- 172.21.15.29:0/2716297561 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c0d1e40 con 0x7f6c7c0b4c80 2023-10-16T14:17:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.434+0000 7f6c7bfff640 1 --2- 172.21.15.29:0/2716297561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c167910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009d90 tx=0x7f6c7402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf3055ff9ffdff5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:45.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.435+0000 7f6c7affd640 1 -- 172.21.15.29:0/2716297561 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64001670 con 0x7f6c7c0b4c80 2023-10-16T14:17:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.435+0000 7f6c7affd640 1 -- 172.21.15.29:0/2716297561 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7402c940 con 0x7f6c7c0b4c80 2023-10-16T14:17:45.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.435+0000 7f6c7affd640 1 -- 172.21.15.29:0/2716297561 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7402ccd0 con 0x7f6c7c0b4c80 2023-10-16T14:17:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 -- 172.21.15.29:0/2716297561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 msgr2=0x7f6c7c167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 --2- 172.21.15.29:0/2716297561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c167910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009d90 tx=0x7f6c7402be10 comp rx=0 tx=0).stop 2023-10-16T14:17:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 -- 172.21.15.29:0/2716297561 shutdown_connections 2023-10-16T14:17:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 --2- 172.21.15.29:0/2716297561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c167910 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 -- 172.21.15.29:0/2716297561 >> 172.21.15.29:0/2716297561 conn(0x7f6c7c0cf570 msgr2=0x7f6c7c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 -- 172.21.15.29:0/2716297561 shutdown_connections 2023-10-16T14:17:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.436+0000 7f6c80ead640 1 -- 172.21.15.29:0/2716297561 wait complete. 2023-10-16T14:17:45.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.438+0000 7f6c80ead640 1 Processor -- start 2023-10-16T14:17:45.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.438+0000 7f6c80ead640 1 -- start start 2023-10-16T14:17:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c80ead640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c0b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c80ead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 0x7f6c7c0b7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c80ead640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c7c15cfc0 0x7f6c7c156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c80ead640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0ca4d0 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c80ead640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0ca770 con 0x7f6c7c0b4c80 2023-10-16T14:17:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c80ead640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0caa50 con 0x7f6c7c15cfc0 2023-10-16T14:17:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.439+0000 7f6c73fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 0x7f6c7c0b7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 0x7f6c7c0b7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37282/0 (socket says 172.21.15.29:37282) 2023-10-16T14:17:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 -- 172.21.15.29:0/506185165 learned_addr learned my addr 172.21.15.29:0/506185165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:45.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c7b7fe640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c0b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 -- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c7c15cfc0 msgr2=0x7f6c7c156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c7bfff640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c7c15cfc0 0x7f6c7c156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c7c15cfc0 0x7f6c7c156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 -- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 msgr2=0x7f6c7c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c0b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 -- 172.21.15.29:0/506185165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74009a70 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.440+0000 7f6c73fff640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 0x7f6c7c0b7290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400c750 tx=0x7f6c6400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.441+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64005ad0 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.441+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64005c30 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.441+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64007070 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.441+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c0cad30 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.441+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c0cb010 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.442+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c1f2400 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.443+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6c640071d0 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.444+0000 7f6c78ff9640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6c5c06ae70 0x7f6c5c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.444+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c640892d0 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.445+0000 7f6c7b7fe640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6c5c06ae70 0x7f6c5c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.446+0000 7f6c7b7fe640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6c5c06ae70 0x7f6c5c06d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c157970 tx=0x7f6c6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.449+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c7c1f2400 con 0x7f6c7c15cbe0 2023-10-16T14:17:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.638+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c7c0d1260 con 0x7f6c5c06ae70 2023-10-16T14:17:45.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.666+0000 7f6c78ff9640 1 -- 172.21.15.29:0/506185165 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286291 (secure 0 0 0) 0x7f6c6c001c70 con 0x7f6c5c06ae70 2023-10-16T14:17:45.673 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:45.676 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:17:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6c5c06ae70 msgr2=0x7f6c5c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6c5c06ae70 0x7f6c5c06d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c157970 tx=0x7f6c6c009040 comp rx=0 tx=0).stop 2023-10-16T14:17:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 msgr2=0x7f6c7c0b7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 0x7f6c7c0b7290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400c750 tx=0x7f6c6400cc10 comp rx=0 tx=0).stop 2023-10-16T14:17:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 shutdown_connections 2023-10-16T14:17:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f6c5c06ae70 0x7f6c5c06d330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c15cbe0 0x7f6c7c0b7290 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c7c15cfc0 0x7f6c7c156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 --2- 172.21.15.29:0/506185165 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c7c0b4c80 0x7f6c7c0b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 >> 172.21.15.29:0/506185165 conn(0x7f6c7c0cf570 msgr2=0x7f6c7c0b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 shutdown_connections 2023-10-16T14:17:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.673+0000 7f6c80ead640 1 -- 172.21.15.29:0/506185165 wait complete. 2023-10-16T14:17:45.697 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2023-10-16T14:17:43.840500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253616,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721104,"statfs":{"total":773094113280,"available":772834410496,"internally_reserved":0,"allocated":4235264,"data_stored":2365028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":255315513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.697903"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846525+0000","last_change":"2023-10-16T14:17:40.848116+0000","last_active":"2023-10-16T14:17:41.846525+0000","last_peered":"2023-10-16T14:17:41.846525+0000","last_clean":"2023-10-16T14:17:41.846525+0000","last_became_active":"2023-10-16T14:17:40.847833+0000","last_became_peered":"2023-10-16T14:17:40.847833+0000","last_unstale":"2023-10-16T14:17:41.846525+0000","last_undegraded":"2023-10-16T14:17:41.846525+0000","last_fullsized":"2023-10-16T14:17:41.846525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:49.051078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848251+0000","last_change":"2023-10-16T14:17:40.845617+0000","last_active":"2023-10-16T14:17:41.848251+0000","last_peered":"2023-10-16T14:17:41.848251+0000","last_clean":"2023-10-16T14:17:41.848251+0000","last_became_active":"2023-10-16T14:17:40.845339+0000","last_became_peered":"2023-10-16T14:17:40.845339+0000","last_unstale":"2023-10-16T14:17:41.848251+0000","last_undegraded":"2023-10-16T14:17:41.848251+0000","last_fullsized":"2023-10-16T14:17:41.848251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:18.938616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825063+0000","last_change":"2023-10-16T14:17:40.854631+0000","last_active":"2023-10-16T14:17:41.825063+0000","last_peered":"2023-10-16T14:17:41.825063+0000","last_clean":"2023-10-16T14:17:41.825063+0000","last_became_active":"2023-10-16T14:17:40.854206+0000","last_became_peered":"2023-10-16T14:17:40.854206+0000","last_unstale":"2023-10-16T14:17:41.825063+0000","last_undegraded":"2023-10-16T14:17:41.825063+0000","last_fullsized":"2023-10-16T14:17:41.825063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:52.774190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846197+0000","last_change":"2023-10-16T14:17:40.853101+0000","last_active":"2023-10-16T14:17:41.846197+0000","last_peered":"2023-10-16T14:17:41.846197+0000","last_clean":"2023-10-16T14:17:41.846197+0000","last_became_active":"2023-10-16T14:17:40.852148+0000","last_became_peered":"2023-10-16T14:17:40.852148+0000","last_unstale":"2023-10-16T14:17:41.846197+0000","last_undegraded":"2023-10-16T14:17:41.846197+0000","last_fullsized":"2023-10-16T14:17:41.846197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:25.342080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844564+0000","last_change":"2023-10-16T14:17:40.849264+0000","last_active":"2023-10-16T14:17:41.844564+0000","last_peered":"2023-10-16T14:17:41.844564+0000","last_clean":"2023-10-16T14:17:41.844564+0000","last_became_active":"2023-10-16T14:17:40.848934+0000","last_became_peered":"2023-10-16T14:17:40.848934+0000","last_unstale":"2023-10-16T14:17:41.844564+0000","last_undegraded":"2023-10-16T14:17:41.844564+0000","last_fullsized":"2023-10-16T14:17:41.844564+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:09.847786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847850+0000","last_change":"2023-10-16T14:17:40.840888+0000","last_active":"2023-10-16T14:17:41.847850+0000","last_peered":"2023-10-16T14:17:41.847850+0000","last_clean":"2023-10-16T14:17:41.847850+0000","last_became_active":"2023-10-16T14:17:40.840609+0000","last_became_peered":"2023-10-16T14:17:40.840609+0000","last_unstale":"2023-10-16T14:17:41.847850+0000","last_undegraded":"2023-10-16T14:17:41.847850+0000","last_fullsized":"2023-10-16T14:17:41.847850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:00.040733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822209+0000","last_change":"2023-10-16T14:17:40.840540+0000","last_active":"2023-10-16T14:17:41.822209+0000","last_peered":"2023-10-16T14:17:41.822209+0000","last_clean":"2023-10-16T14:17:41.822209+0000","last_became_active":"2023-10-16T14:17:40.840235+0000","last_became_peered":"2023-10-16T14:17:40.840235+0000","last_unstale":"2023-10-16T14:17:41.822209+0000","last_undegraded":"2023-10-16T14:17:41.822209+0000","last_fullsized":"2023-10-16T14:17:41.822209+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:02.808237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822896+0000","last_change":"2023-10-16T14:17:40.846680+0000","last_active":"2023-10-16T14:17:41.822896+0000","last_peered":"2023-10-16T14:17:41.822896+0000","last_clean":"2023-10-16T14:17:41.822896+0000","last_became_active":"2023-10-16T14:17:40.846344+0000","last_became_peered":"2023-10-16T14:17:40.846344+0000","last_unstale":"2023-10-16T14:17:41.822896+0000","last_undegraded":"2023-10-16T14:17:41.822896+0000","last_fullsized":"2023-10-16T14:17:41.822896+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:00.040733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822930+0000","last_change":"2023-10-16T14:17:40.852056+0000","last_active":"2023-10-16T14:17:41.822930+0000","last_peered":"2023-10-16T14:17:41.822930+0000","last_clean":"2023-10-16T14:17:41.822930+0000","last_became_active":"2023-10-16T14:17:40.851475+0000","last_became_peered":"2023-10-16T14:17:40.851475+0000","last_unstale":"2023-10-16T14:17:41.822930+0000","last_undegraded":"2023-10-16T14:17:41.822930+0000","last_fullsized":"2023-10-16T14:17:41.822930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:30.110067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846377+0000","last_change":"2023-10-16T14:17:40.840670+0000","last_active":"2023-10-16T14:17:41.846377+0000","last_peered":"2023-10-16T14:17:41.846377+0000","last_clean":"2023-10-16T14:17:41.846377+0000","last_became_active":"2023-10-16T14:17:40.840122+0000","last_became_peered":"2023-10-16T14:17:40.840122+0000","last_unstale":"2023-10-16T14:17:41.846377+0000","last_undegraded":"2023-10-16T14:17:41.846377+0000","last_fullsized":"2023-10-16T14:17:41.846377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:30.886976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821789+0000","last_change":"2023-10-16T14:17:40.847394+0000","last_active":"2023-10-16T14:17:41.821789+0000","last_peered":"2023-10-16T14:17:41.821789+0000","last_clean":"2023-10-16T14:17:41.821789+0000","last_became_active":"2023-10-16T14:17:40.846918+0000","last_became_peered":"2023-10-16T14:17:40.846918+0000","last_unstale":"2023-10-16T14:17:41.821789+0000","last_undegraded":"2023-10-16T14:17:41.821789+0000","last_fullsized":"2023-10-16T14:17:41.821789+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:30.110067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824405+0000","last_change":"2023-10-16T14:17:40.843723+0000","last_active":"2023-10-16T14:17:41.824405+0000","last_peered":"2023-10-16T14:17:41.824405+0000","last_clean":"2023-10-16T14:17:41.824405+0000","last_became_active":"2023-10-16T14:17:40.843451+0000","last_became_peered":"2023-10-16T14:17:40.843451+0000","last_unstale":"2023-10-16T14:17:41.824405+0000","last_undegraded":"2023-10-16T14:17:41.824405+0000","last_fullsized":"2023-10-16T14:17:41.824405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:46.139143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844690+0000","last_change":"2023-10-16T14:17:40.855177+0000","last_active":"2023-10-16T14:17:41.844690+0000","last_peered":"2023-10-16T14:17:41.844690+0000","last_clean":"2023-10-16T14:17:41.844690+0000","last_became_active":"2023-10-16T14:17:40.854836+0000","last_became_peered":"2023-10-16T14:17:40.854836+0000","last_unstale":"2023-10-16T14:17:41.844690+0000","last_undegraded":"2023-10-16T14:17:41.844690+0000","last_fullsized":"2023-10-16T14:17:41.844690+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:06.304287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848123+0000","last_change":"2023-10-16T14:17:40.847370+0000","last_active":"2023-10-16T14:17:41.848123+0000","last_peered":"2023-10-16T14:17:41.848123+0000","last_clean":"2023-10-16T14:17:41.848123+0000","last_became_active":"2023-10-16T14:17:40.847040+0000","last_became_peered":"2023-10-16T14:17:40.847040+0000","last_unstale":"2023-10-16T14:17:41.848123+0000","last_undegraded":"2023-10-16T14:17:41.848123+0000","last_fullsized":"2023-10-16T14:17:41.848123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:52.774190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823887+0000","last_change":"2023-10-16T14:17:40.841785+0000","last_active":"2023-10-16T14:17:41.823887+0000","last_peered":"2023-10-16T14:17:41.823887+0000","last_clean":"2023-10-16T14:17:41.823887+0000","last_became_active":"2023-10-16T14:17:40.841421+0000","last_became_peered":"2023-10-16T14:17:40.841421+0000","last_unstale":"2023-10-16T14:17:41.823887+0000","last_undegraded":"2023-10-16T14:17:41.823887+0000","last_fullsized":"2023-10-16T14:17:41.823887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:38.253474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822810+0000","last_change":"2023-10-16T14:17:40.849272+0000","last_active":"2023-10-16T14:17:41.822810+0000","last_peered":"2023-10-16T14:17:41.822810+0000","last_clean":"2023-10-16T14:17:41.822810+0000","last_became_active":"2023-10-16T14:17:40.848910+0000","last_became_peered":"2023-10-16T14:17:40.848910+0000","last_unstale":"2023-10-16T14:17:41.822810+0000","last_undegraded":"2023-10-16T14:17:41.822810+0000","last_fullsized":"2023-10-16T14:17:41.822810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:46.139143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822433+0000","last_change":"2023-10-16T14:17:40.844851+0000","last_active":"2023-10-16T14:17:41.822433+0000","last_peered":"2023-10-16T14:17:41.822433+0000","last_clean":"2023-10-16T14:17:41.822433+0000","last_became_active":"2023-10-16T14:17:40.844562+0000","last_became_peered":"2023-10-16T14:17:40.844562+0000","last_unstale":"2023-10-16T14:17:41.822433+0000","last_undegraded":"2023-10-16T14:17:41.822433+0000","last_fullsized":"2023-10-16T14:17:41.822433+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:38.253474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848386+0000","last_change":"2023-10-16T14:17:40.843494+0000","last_active":"2023-10-16T14:17:41.848386+0000","last_peered":"2023-10-16T14:17:41.848386+0000","last_clean":"2023-10-16T14:17:41.848386+0000","last_became_active":"2023-10-16T14:17:40.843203+0000","last_became_peered":"2023-10-16T14:17:40.843203+0000","last_unstale":"2023-10-16T14:17:41.848386+0000","last_undegraded":"2023-10-16T14:17:41.848386+0000","last_fullsized":"2023-10-16T14:17:41.848386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:56.558827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848237+0000","last_change":"2023-10-16T14:17:40.847361+0000","last_active":"2023-10-16T14:17:41.848237+0000","last_peered":"2023-10-16T14:17:41.848237+0000","last_clean":"2023-10-16T14:17:41.848237+0000","last_became_active":"2023-10-16T14:17:40.846980+0000","last_became_peered":"2023-10-16T14:17:40.846980+0000","last_unstale":"2023-10-16T14:17:41.848237+0000","last_undegraded":"2023-10-16T14:17:41.848237+0000","last_fullsized":"2023-10-16T14:17:41.848237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:47.932423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.845423+0000","last_change":"2023-10-16T14:17:40.852470+0000","last_active":"2023-10-16T14:17:41.845423+0000","last_peered":"2023-10-16T14:17:41.845423+0000","last_clean":"2023-10-16T14:17:41.845423+0000","last_became_active":"2023-10-16T14:17:40.848746+0000","last_became_peered":"2023-10-16T14:17:40.848746+0000","last_unstale":"2023-10-16T14:17:41.845423+0000","last_undegraded":"2023-10-16T14:17:41.845423+0000","last_fullsized":"2023-10-16T14:17:41.845423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:51.466345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821529+0000","last_change":"2023-10-16T14:17:40.841447+0000","last_active":"2023-10-16T14:17:41.821529+0000","last_peered":"2023-10-16T14:17:41.821529+0000","last_clean":"2023-10-16T14:17:41.821529+0000","last_became_active":"2023-10-16T14:17:40.841027+0000","last_became_peered":"2023-10-16T14:17:40.841027+0000","last_unstale":"2023-10-16T14:17:41.821529+0000","last_undegraded":"2023-10-16T14:17:41.821529+0000","last_fullsized":"2023-10-16T14:17:41.821529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:55.629805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821507+0000","last_change":"2023-10-16T14:17:40.842810+0000","last_active":"2023-10-16T14:17:41.821507+0000","last_peered":"2023-10-16T14:17:41.821507+0000","last_clean":"2023-10-16T14:17:41.821507+0000","last_became_active":"2023-10-16T14:17:40.842477+0000","last_became_peered":"2023-10-16T14:17:40.842477+0000","last_unstale":"2023-10-16T14:17:41.821507+0000","last_undegraded":"2023-10-16T14:17:41.821507+0000","last_fullsized":"2023-10-16T14:17:41.821507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.991974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.817081+0000","last_change":"2023-10-16T14:17:40.839037+0000","last_active":"2023-10-16T14:17:41.817081+0000","last_peered":"2023-10-16T14:17:41.817081+0000","last_clean":"2023-10-16T14:17:41.817081+0000","last_became_active":"2023-10-16T14:17:40.838566+0000","last_became_peered":"2023-10-16T14:17:40.838566+0000","last_unstale":"2023-10-16T14:17:41.817081+0000","last_undegraded":"2023-10-16T14:17:41.817081+0000","last_fullsized":"2023-10-16T14:17:41.817081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:49.051078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821720+0000","last_change":"2023-10-16T14:17:40.844318+0000","last_active":"2023-10-16T14:17:41.821720+0000","last_peered":"2023-10-16T14:17:41.821720+0000","last_clean":"2023-10-16T14:17:41.821720+0000","last_became_active":"2023-10-16T14:17:40.844015+0000","last_became_peered":"2023-10-16T14:17:40.844015+0000","last_unstale":"2023-10-16T14:17:41.821720+0000","last_undegraded":"2023-10-16T14:17:41.821720+0000","last_fullsized":"2023-10-16T14:17:41.821720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:02.808237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846998+0000","last_change":"2023-10-16T14:17:40.854768+0000","last_active":"2023-10-16T14:17:41.846998+0000","last_peered":"2023-10-16T14:17:41.846998+0000","last_clean":"2023-10-16T14:17:41.846998+0000","last_became_active":"2023-10-16T14:17:40.854194+0000","last_became_peered":"2023-10-16T14:17:40.854194+0000","last_unstale":"2023-10-16T14:17:41.846998+0000","last_undegraded":"2023-10-16T14:17:41.846998+0000","last_fullsized":"2023-10-16T14:17:41.846998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:55.629805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848469+0000","last_change":"2023-10-16T14:17:40.838964+0000","last_active":"2023-10-16T14:17:41.848469+0000","last_peered":"2023-10-16T14:17:41.848469+0000","last_clean":"2023-10-16T14:17:41.848469+0000","last_became_active":"2023-10-16T14:17:40.838602+0000","last_became_peered":"2023-10-16T14:17:40.838602+0000","last_unstale":"2023-10-16T14:17:41.848469+0000","last_undegraded":"2023-10-16T14:17:41.848469+0000","last_fullsized":"2023-10-16T14:17:41.848469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:08.364442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847109+0000","last_change":"2023-10-16T14:17:40.855205+0000","last_active":"2023-10-16T14:17:41.847109+0000","last_peered":"2023-10-16T14:17:41.847109+0000","last_clean":"2023-10-16T14:17:41.847109+0000","last_became_active":"2023-10-16T14:17:40.852783+0000","last_became_peered":"2023-10-16T14:17:40.852783+0000","last_unstale":"2023-10-16T14:17:41.847109+0000","last_undegraded":"2023-10-16T14:17:41.847109+0000","last_fullsized":"2023-10-16T14:17:41.847109+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:24.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824611+0000","last_change":"2023-10-16T14:17:40.847434+0000","last_active":"2023-10-16T14:17:41.824611+0000","last_peered":"2023-10-16T14:17:41.824611+0000","last_clean":"2023-10-16T14:17:41.824611+0000","last_became_active":"2023-10-16T14:17:40.846738+0000","last_became_peered":"2023-10-16T14:17:40.846738+0000","last_unstale":"2023-10-16T14:17:41.824611+0000","last_undegraded":"2023-10-16T14:17:41.824611+0000","last_fullsized":"2023-10-16T14:17:41.824611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:00.040733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821876+0000","last_change":"2023-10-16T14:17:40.849150+0000","last_active":"2023-10-16T14:17:41.821876+0000","last_peered":"2023-10-16T14:17:41.821876+0000","last_clean":"2023-10-16T14:17:41.821876+0000","last_became_active":"2023-10-16T14:17:40.848879+0000","last_became_peered":"2023-10-16T14:17:40.848879+0000","last_unstale":"2023-10-16T14:17:41.821876+0000","last_undegraded":"2023-10-16T14:17:41.821876+0000","last_fullsized":"2023-10-16T14:17:41.821876+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:24.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822969+0000","last_change":"2023-10-16T14:17:40.840727+0000","last_active":"2023-10-16T14:17:41.822969+0000","last_peered":"2023-10-16T14:17:41.822969+0000","last_clean":"2023-10-16T14:17:41.822969+0000","last_became_active":"2023-10-16T14:17:40.840414+0000","last_became_peered":"2023-10-16T14:17:40.840414+0000","last_unstale":"2023-10-16T14:17:41.822969+0000","last_undegraded":"2023-10-16T14:17:41.822969+0000","last_fullsized":"2023-10-16T14:17:41.822969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:24.359426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846100+0000","last_change":"2023-10-16T14:17:40.844158+0000","last_active":"2023-10-16T14:17:41.846100+0000","last_peered":"2023-10-16T14:17:41.846100+0000","last_clean":"2023-10-16T14:17:41.846100+0000","last_became_active":"2023-10-16T14:17:40.843829+0000","last_became_peered":"2023-10-16T14:17:40.843829+0000","last_unstale":"2023-10-16T14:17:41.846100+0000","last_undegraded":"2023-10-16T14:17:41.846100+0000","last_fullsized":"2023-10-16T14:17:41.846100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:51.466345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821233+0000","last_change":"2023-10-16T14:17:40.836246+0000","last_active":"2023-10-16T14:17:41.821233+0000","last_peered":"2023-10-16T14:17:41.821233+0000","last_clean":"2023-10-16T14:17:41.821233+0000","last_became_active":"2023-10-16T14:17:40.835953+0000","last_became_peered":"2023-10-16T14:17:40.835953+0000","last_unstale":"2023-10-16T14:17:41.821233+0000","last_undegraded":"2023-10-16T14:17:41.821233+0000","last_fullsized":"2023-10-16T14:17:41.821233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.991974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818287+0000","last_change":"2023-10-16T14:17:40.847268+0000","last_active":"2023-10-16T14:17:41.818287+0000","last_peered":"2023-10-16T14:17:41.818287+0000","last_clean":"2023-10-16T14:17:41.818287+0000","last_became_active":"2023-10-16T14:17:40.846956+0000","last_became_peered":"2023-10-16T14:17:40.846956+0000","last_unstale":"2023-10-16T14:17:41.818287+0000","last_undegraded":"2023-10-16T14:17:41.818287+0000","last_fullsized":"2023-10-16T14:17:41.818287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:28.134316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846662+0000","last_change":"2023-10-16T14:17:40.184353+0000","last_active":"2023-10-16T14:17:41.846662+0000","last_peered":"2023-10-16T14:17:41.846662+0000","last_clean":"2023-10-16T14:17:41.846662+0000","last_became_active":"2023-10-16T14:17:40.184046+0000","last_became_peered":"2023-10-16T14:17:40.184046+0000","last_unstale":"2023-10-16T14:17:41.846662+0000","last_undegraded":"2023-10-16T14:17:41.846662+0000","last_fullsized":"2023-10-16T14:17:41.846662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:05.505579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846527+0000","last_change":"2023-10-16T14:17:40.853333+0000","last_active":"2023-10-16T14:17:41.846527+0000","last_peered":"2023-10-16T14:17:41.846527+0000","last_clean":"2023-10-16T14:17:41.846527+0000","last_became_active":"2023-10-16T14:17:40.852060+0000","last_became_peered":"2023-10-16T14:17:40.852060+0000","last_unstale":"2023-10-16T14:17:41.846527+0000","last_undegraded":"2023-10-16T14:17:41.846527+0000","last_fullsized":"2023-10-16T14:17:41.846527+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:55.403159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818337+0000","last_change":"2023-10-16T14:17:40.170561+0000","last_active":"2023-10-16T14:17:41.818337+0000","last_peered":"2023-10-16T14:17:41.818337+0000","last_clean":"2023-10-16T14:17:41.818337+0000","last_became_active":"2023-10-16T14:17:40.170277+0000","last_became_peered":"2023-10-16T14:17:40.170277+0000","last_unstale":"2023-10-16T14:17:41.818337+0000","last_undegraded":"2023-10-16T14:17:41.818337+0000","last_fullsized":"2023-10-16T14:17:41.818337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:07.707835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820934+0000","last_change":"2023-10-16T14:17:40.842954+0000","last_active":"2023-10-16T14:17:41.820934+0000","last_peered":"2023-10-16T14:17:41.820934+0000","last_clean":"2023-10-16T14:17:41.820934+0000","last_became_active":"2023-10-16T14:17:40.842521+0000","last_became_peered":"2023-10-16T14:17:40.842521+0000","last_unstale":"2023-10-16T14:17:41.820934+0000","last_undegraded":"2023-10-16T14:17:41.820934+0000","last_fullsized":"2023-10-16T14:17:41.820934+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:30.886976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848485+0000","last_change":"2023-10-16T14:17:40.180609+0000","last_active":"2023-10-16T14:17:41.848485+0000","last_peered":"2023-10-16T14:17:41.848485+0000","last_clean":"2023-10-16T14:17:41.848485+0000","last_became_active":"2023-10-16T14:17:40.180248+0000","last_became_peered":"2023-10-16T14:17:40.180248+0000","last_unstale":"2023-10-16T14:17:41.848485+0000","last_undegraded":"2023-10-16T14:17:41.848485+0000","last_fullsized":"2023-10-16T14:17:41.848485+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:24.336846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847123+0000","last_change":"2023-10-16T14:17:40.855526+0000","last_active":"2023-10-16T14:17:41.847123+0000","last_peered":"2023-10-16T14:17:41.847123+0000","last_clean":"2023-10-16T14:17:41.847123+0000","last_became_active":"2023-10-16T14:17:40.853150+0000","last_became_peered":"2023-10-16T14:17:40.853150+0000","last_unstale":"2023-10-16T14:17:41.847123+0000","last_undegraded":"2023-10-16T14:17:41.847123+0000","last_fullsized":"2023-10-16T14:17:41.847123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:30.110067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.845402+0000","last_change":"2023-10-16T14:17:40.179066+0000","last_active":"2023-10-16T14:17:41.845402+0000","last_peered":"2023-10-16T14:17:41.845402+0000","last_clean":"2023-10-16T14:17:41.845402+0000","last_became_active":"2023-10-16T14:17:40.177489+0000","last_became_peered":"2023-10-16T14:17:40.177489+0000","last_unstale":"2023-10-16T14:17:41.845402+0000","last_undegraded":"2023-10-16T14:17:41.845402+0000","last_fullsized":"2023-10-16T14:17:41.845402+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:02:31.868085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823322+0000","last_change":"2023-10-16T14:17:40.842227+0000","last_active":"2023-10-16T14:17:41.823322+0000","last_peered":"2023-10-16T14:17:41.823322+0000","last_clean":"2023-10-16T14:17:41.823322+0000","last_became_active":"2023-10-16T14:17:40.841922+0000","last_became_peered":"2023-10-16T14:17:40.841922+0000","last_unstale":"2023-10-16T14:17:41.823322+0000","last_undegraded":"2023-10-16T14:17:41.823322+0000","last_fullsized":"2023-10-16T14:17:41.823322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:18.224519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822009+0000","last_change":"2023-10-16T14:17:40.182015+0000","last_active":"2023-10-16T14:17:41.822009+0000","last_peered":"2023-10-16T14:17:41.822009+0000","last_clean":"2023-10-16T14:17:41.822009+0000","last_became_active":"2023-10-16T14:17:40.181734+0000","last_became_peered":"2023-10-16T14:17:40.181734+0000","last_unstale":"2023-10-16T14:17:41.822009+0000","last_undegraded":"2023-10-16T14:17:41.822009+0000","last_fullsized":"2023-10-16T14:17:41.822009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.397925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824190+0000","last_change":"2023-10-16T14:17:40.849311+0000","last_active":"2023-10-16T14:17:41.824190+0000","last_peered":"2023-10-16T14:17:41.824190+0000","last_clean":"2023-10-16T14:17:41.824190+0000","last_became_active":"2023-10-16T14:17:40.847998+0000","last_became_peered":"2023-10-16T14:17:40.847998+0000","last_unstale":"2023-10-16T14:17:41.824190+0000","last_undegraded":"2023-10-16T14:17:41.824190+0000","last_fullsized":"2023-10-16T14:17:41.824190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:23.202538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821501+0000","last_change":"2023-10-16T14:17:40.185804+0000","last_active":"2023-10-16T14:17:41.821501+0000","last_peered":"2023-10-16T14:17:41.821501+0000","last_clean":"2023-10-16T14:17:41.821501+0000","last_became_active":"2023-10-16T14:17:40.185533+0000","last_became_peered":"2023-10-16T14:17:40.185533+0000","last_unstale":"2023-10-16T14:17:41.821501+0000","last_undegraded":"2023-10-16T14:17:41.821501+0000","last_fullsized":"2023-10-16T14:17:41.821501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:27.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820563+0000","last_change":"2023-10-16T14:17:40.839911+0000","last_active":"2023-10-16T14:17:41.820563+0000","last_peered":"2023-10-16T14:17:41.820563+0000","last_clean":"2023-10-16T14:17:41.820563+0000","last_became_active":"2023-10-16T14:17:40.838848+0000","last_became_peered":"2023-10-16T14:17:40.838848+0000","last_unstale":"2023-10-16T14:17:41.820563+0000","last_undegraded":"2023-10-16T14:17:41.820563+0000","last_fullsized":"2023-10-16T14:17:41.820563+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:56.091494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847725+0000","last_change":"2023-10-16T14:17:40.173988+0000","last_active":"2023-10-16T14:17:41.847725+0000","last_peered":"2023-10-16T14:17:41.847725+0000","last_clean":"2023-10-16T14:17:41.847725+0000","last_became_active":"2023-10-16T14:17:40.173677+0000","last_became_peered":"2023-10-16T14:17:40.173677+0000","last_unstale":"2023-10-16T14:17:41.847725+0000","last_undegraded":"2023-10-16T14:17:41.847725+0000","last_fullsized":"2023-10-16T14:17:41.847725+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:48.045845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846012+0000","last_change":"2023-10-16T14:17:40.849561+0000","last_active":"2023-10-16T14:17:41.846012+0000","last_peered":"2023-10-16T14:17:41.846012+0000","last_clean":"2023-10-16T14:17:41.846012+0000","last_became_active":"2023-10-16T14:17:40.849234+0000","last_became_peered":"2023-10-16T14:17:40.849234+0000","last_unstale":"2023-10-16T14:17:41.846012+0000","last_undegraded":"2023-10-16T14:17:41.846012+0000","last_fullsized":"2023-10-16T14:17:41.846012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.900473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820810+0000","last_change":"2023-10-16T14:17:40.178142+0000","last_active":"2023-10-16T14:17:41.820810+0000","last_peered":"2023-10-16T14:17:41.820810+0000","last_clean":"2023-10-16T14:17:41.820810+0000","last_became_active":"2023-10-16T14:17:40.177075+0000","last_became_peered":"2023-10-16T14:17:40.177075+0000","last_unstale":"2023-10-16T14:17:41.820810+0000","last_undegraded":"2023-10-16T14:17:41.820810+0000","last_fullsized":"2023-10-16T14:17:41.820810+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:59.566646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825061+0000","last_change":"2023-10-16T14:17:40.844998+0000","last_active":"2023-10-16T14:17:41.825061+0000","last_peered":"2023-10-16T14:17:41.825061+0000","last_clean":"2023-10-16T14:17:41.825061+0000","last_became_active":"2023-10-16T14:17:40.844492+0000","last_became_peered":"2023-10-16T14:17:40.844492+0000","last_unstale":"2023-10-16T14:17:41.825061+0000","last_undegraded":"2023-10-16T14:17:41.825061+0000","last_fullsized":"2023-10-16T14:17:41.825061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:18.224519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848762+0000","last_change":"2023-10-16T14:17:40.176614+0000","last_active":"2023-10-16T14:17:41.848762+0000","last_peered":"2023-10-16T14:17:41.848762+0000","last_clean":"2023-10-16T14:17:41.848762+0000","last_became_active":"2023-10-16T14:17:40.176225+0000","last_became_peered":"2023-10-16T14:17:40.176225+0000","last_unstale":"2023-10-16T14:17:41.848762+0000","last_undegraded":"2023-10-16T14:17:41.848762+0000","last_fullsized":"2023-10-16T14:17:41.848762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:45.133909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848728+0000","last_change":"2023-10-16T14:17:40.852565+0000","last_active":"2023-10-16T14:17:41.848728+0000","last_peered":"2023-10-16T14:17:41.848728+0000","last_clean":"2023-10-16T14:17:41.848728+0000","last_became_active":"2023-10-16T14:17:40.852024+0000","last_became_peered":"2023-10-16T14:17:40.852024+0000","last_unstale":"2023-10-16T14:17:41.848728+0000","last_undegraded":"2023-10-16T14:17:41.848728+0000","last_fullsized":"2023-10-16T14:17:41.848728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:57.803633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818629+0000","last_change":"2023-10-16T14:17:40.181478+0000","last_active":"2023-10-16T14:17:41.818629+0000","last_peered":"2023-10-16T14:17:41.818629+0000","last_clean":"2023-10-16T14:17:41.818629+0000","last_became_active":"2023-10-16T14:17:40.181181+0000","last_became_peered":"2023-10-16T14:17:40.181181+0000","last_unstale":"2023-10-16T14:17:41.818629+0000","last_undegraded":"2023-10-16T14:17:41.818629+0000","last_fullsized":"2023-10-16T14:17:41.818629+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:57.099074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848212+0000","last_change":"2023-10-16T14:17:40.841747+0000","last_active":"2023-10-16T14:17:41.848212+0000","last_peered":"2023-10-16T14:17:41.848212+0000","last_clean":"2023-10-16T14:17:41.848212+0000","last_became_active":"2023-10-16T14:17:40.839067+0000","last_became_peered":"2023-10-16T14:17:40.839067+0000","last_unstale":"2023-10-16T14:17:41.848212+0000","last_undegraded":"2023-10-16T14:17:41.848212+0000","last_fullsized":"2023-10-16T14:17:41.848212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:45.303224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825228+0000","last_change":"2023-10-16T14:17:40.184621+0000","last_active":"2023-10-16T14:17:41.825228+0000","last_peered":"2023-10-16T14:17:41.825228+0000","last_clean":"2023-10-16T14:17:41.825228+0000","last_became_active":"2023-10-16T14:17:40.184347+0000","last_became_peered":"2023-10-16T14:17:40.184347+0000","last_unstale":"2023-10-16T14:17:41.825228+0000","last_undegraded":"2023-10-16T14:17:41.825228+0000","last_fullsized":"2023-10-16T14:17:41.825228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:29.104833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824715+0000","last_change":"2023-10-16T14:17:40.853283+0000","last_active":"2023-10-16T14:17:41.824715+0000","last_peered":"2023-10-16T14:17:41.824715+0000","last_clean":"2023-10-16T14:17:41.824715+0000","last_became_active":"2023-10-16T14:17:40.852659+0000","last_became_peered":"2023-10-16T14:17:40.852659+0000","last_unstale":"2023-10-16T14:17:41.824715+0000","last_undegraded":"2023-10-16T14:17:41.824715+0000","last_fullsized":"2023-10-16T14:17:41.824715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:24.359426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848396+0000","last_change":"2023-10-16T14:17:40.180701+0000","last_active":"2023-10-16T14:17:41.848396+0000","last_peered":"2023-10-16T14:17:41.848396+0000","last_clean":"2023-10-16T14:17:41.848396+0000","last_became_active":"2023-10-16T14:17:40.180404+0000","last_became_peered":"2023-10-16T14:17:40.180404+0000","last_unstale":"2023-10-16T14:17:41.848396+0000","last_undegraded":"2023-10-16T14:17:41.848396+0000","last_fullsized":"2023-10-16T14:17:41.848396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:37.248241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847165+0000","last_change":"2023-10-16T14:17:40.849603+0000","last_active":"2023-10-16T14:17:41.847165+0000","last_peered":"2023-10-16T14:17:41.847165+0000","last_clean":"2023-10-16T14:17:41.847165+0000","last_became_active":"2023-10-16T14:17:40.849301+0000","last_became_peered":"2023-10-16T14:17:40.849301+0000","last_unstale":"2023-10-16T14:17:41.847165+0000","last_undegraded":"2023-10-16T14:17:41.847165+0000","last_fullsized":"2023-10-16T14:17:41.847165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:02.808237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822114+0000","last_change":"2023-10-16T14:17:40.177211+0000","last_active":"2023-10-16T14:17:41.822114+0000","last_peered":"2023-10-16T14:17:41.822114+0000","last_clean":"2023-10-16T14:17:41.822114+0000","last_became_active":"2023-10-16T14:17:40.176900+0000","last_became_peered":"2023-10-16T14:17:40.176900+0000","last_unstale":"2023-10-16T14:17:41.822114+0000","last_undegraded":"2023-10-16T14:17:41.822114+0000","last_fullsized":"2023-10-16T14:17:41.822114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:48.045845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848087+0000","last_change":"2023-10-16T14:17:40.844990+0000","last_active":"2023-10-16T14:17:41.848087+0000","last_peered":"2023-10-16T14:17:41.848087+0000","last_clean":"2023-10-16T14:17:41.848087+0000","last_became_active":"2023-10-16T14:17:40.844664+0000","last_became_peered":"2023-10-16T14:17:40.844664+0000","last_unstale":"2023-10-16T14:17:41.848087+0000","last_undegraded":"2023-10-16T14:17:41.848087+0000","last_fullsized":"2023-10-16T14:17:41.848087+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:18.224519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846122+0000","last_change":"2023-10-16T14:17:40.181839+0000","last_active":"2023-10-16T14:17:41.846122+0000","last_peered":"2023-10-16T14:17:41.846122+0000","last_clean":"2023-10-16T14:17:41.846122+0000","last_became_active":"2023-10-16T14:17:40.181558+0000","last_became_peered":"2023-10-16T14:17:40.181558+0000","last_unstale":"2023-10-16T14:17:41.846122+0000","last_undegraded":"2023-10-16T14:17:41.846122+0000","last_fullsized":"2023-10-16T14:17:41.846122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:27.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821617+0000","last_change":"2023-10-16T14:17:40.836881+0000","last_active":"2023-10-16T14:17:41.821617+0000","last_peered":"2023-10-16T14:17:41.821617+0000","last_clean":"2023-10-16T14:17:41.821617+0000","last_became_active":"2023-10-16T14:17:40.836084+0000","last_became_peered":"2023-10-16T14:17:40.836084+0000","last_unstale":"2023-10-16T14:17:41.821617+0000","last_undegraded":"2023-10-16T14:17:41.821617+0000","last_fullsized":"2023-10-16T14:17:41.821617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:55.629805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847694+0000","last_change":"2023-10-16T14:17:40.175606+0000","last_active":"2023-10-16T14:17:41.847694+0000","last_peered":"2023-10-16T14:17:41.847694+0000","last_clean":"2023-10-16T14:17:41.847694+0000","last_became_active":"2023-10-16T14:17:40.175275+0000","last_became_peered":"2023-10-16T14:17:40.175275+0000","last_unstale":"2023-10-16T14:17:41.847694+0000","last_undegraded":"2023-10-16T14:17:41.847694+0000","last_fullsized":"2023-10-16T14:17:41.847694+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:23.016735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846144+0000","last_change":"2023-10-16T14:17:40.849781+0000","last_active":"2023-10-16T14:17:41.846144+0000","last_peered":"2023-10-16T14:17:41.846144+0000","last_clean":"2023-10-16T14:17:41.846144+0000","last_became_active":"2023-10-16T14:17:40.849453+0000","last_became_peered":"2023-10-16T14:17:40.849453+0000","last_unstale":"2023-10-16T14:17:41.846144+0000","last_undegraded":"2023-10-16T14:17:41.846144+0000","last_fullsized":"2023-10-16T14:17:41.846144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:08.713068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848284+0000","last_change":"2023-10-16T14:17:40.176322+0000","last_active":"2023-10-16T14:17:41.848284+0000","last_peered":"2023-10-16T14:17:41.848284+0000","last_clean":"2023-10-16T14:17:41.848284+0000","last_became_active":"2023-10-16T14:17:40.175613+0000","last_became_peered":"2023-10-16T14:17:40.175613+0000","last_unstale":"2023-10-16T14:17:41.848284+0000","last_undegraded":"2023-10-16T14:17:41.848284+0000","last_fullsized":"2023-10-16T14:17:41.848284+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:12.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820681+0000","last_change":"2023-10-16T14:17:40.844761+0000","last_active":"2023-10-16T14:17:41.820681+0000","last_peered":"2023-10-16T14:17:41.820681+0000","last_clean":"2023-10-16T14:17:41.820681+0000","last_became_active":"2023-10-16T14:17:40.844495+0000","last_became_peered":"2023-10-16T14:17:40.844495+0000","last_unstale":"2023-10-16T14:17:41.820681+0000","last_undegraded":"2023-10-16T14:17:41.820681+0000","last_fullsized":"2023-10-16T14:17:41.820681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.900473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847014+0000","last_change":"2023-10-16T14:17:40.185147+0000","last_active":"2023-10-16T14:17:41.847014+0000","last_peered":"2023-10-16T14:17:41.847014+0000","last_clean":"2023-10-16T14:17:41.847014+0000","last_became_active":"2023-10-16T14:17:40.184823+0000","last_became_peered":"2023-10-16T14:17:40.184823+0000","last_unstale":"2023-10-16T14:17:41.847014+0000","last_undegraded":"2023-10-16T14:17:41.847014+0000","last_fullsized":"2023-10-16T14:17:41.847014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:23.016735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825106+0000","last_change":"2023-10-16T14:17:40.845511+0000","last_active":"2023-10-16T14:17:41.825106+0000","last_peered":"2023-10-16T14:17:41.825106+0000","last_clean":"2023-10-16T14:17:41.825106+0000","last_became_active":"2023-10-16T14:17:40.844473+0000","last_became_peered":"2023-10-16T14:17:40.844473+0000","last_unstale":"2023-10-16T14:17:41.825106+0000","last_undegraded":"2023-10-16T14:17:41.825106+0000","last_fullsized":"2023-10-16T14:17:41.825106+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:45.303224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823572+0000","last_change":"2023-10-16T14:17:40.181484+0000","last_active":"2023-10-16T14:17:41.823572+0000","last_peered":"2023-10-16T14:17:41.823572+0000","last_clean":"2023-10-16T14:17:41.823572+0000","last_became_active":"2023-10-16T14:17:40.181196+0000","last_became_peered":"2023-10-16T14:17:40.181196+0000","last_unstale":"2023-10-16T14:17:41.823572+0000","last_undegraded":"2023-10-16T14:17:41.823572+0000","last_fullsized":"2023-10-16T14:17:41.823572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:23.016735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846421+0000","last_change":"2023-10-16T14:17:40.853402+0000","last_active":"2023-10-16T14:17:41.846421+0000","last_peered":"2023-10-16T14:17:41.846421+0000","last_clean":"2023-10-16T14:17:41.846421+0000","last_became_active":"2023-10-16T14:17:40.852595+0000","last_became_peered":"2023-10-16T14:17:40.852595+0000","last_unstale":"2023-10-16T14:17:41.846421+0000","last_undegraded":"2023-10-16T14:17:41.846421+0000","last_fullsized":"2023-10-16T14:17:41.846421+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.991974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822519+0000","last_change":"2023-10-16T14:17:40.173956+0000","last_active":"2023-10-16T14:17:41.822519+0000","last_peered":"2023-10-16T14:17:41.822519+0000","last_clean":"2023-10-16T14:17:41.822519+0000","last_became_active":"2023-10-16T14:17:40.173562+0000","last_became_peered":"2023-10-16T14:17:40.173562+0000","last_unstale":"2023-10-16T14:17:41.822519+0000","last_undegraded":"2023-10-16T14:17:41.822519+0000","last_fullsized":"2023-10-16T14:17:41.822519+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:12.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844819+0000","last_change":"2023-10-16T14:17:40.847347+0000","last_active":"2023-10-16T14:17:41.844819+0000","last_peered":"2023-10-16T14:17:41.844819+0000","last_clean":"2023-10-16T14:17:41.844819+0000","last_became_active":"2023-10-16T14:17:40.847084+0000","last_became_peered":"2023-10-16T14:17:40.847084+0000","last_unstale":"2023-10-16T14:17:41.844819+0000","last_undegraded":"2023-10-16T14:17:41.844819+0000","last_fullsized":"2023-10-16T14:17:41.844819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:00.571880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821185+0000","last_change":"2023-10-16T14:17:40.182230+0000","last_active":"2023-10-16T14:17:41.821185+0000","last_peered":"2023-10-16T14:17:41.821185+0000","last_clean":"2023-10-16T14:17:41.821185+0000","last_became_active":"2023-10-16T14:17:40.181964+0000","last_became_peered":"2023-10-16T14:17:40.181964+0000","last_unstale":"2023-10-16T14:17:41.821185+0000","last_undegraded":"2023-10-16T14:17:41.821185+0000","last_fullsized":"2023-10-16T14:17:41.821185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:07.707835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.845153+0000","last_change":"2023-10-16T14:17:40.848588+0000","last_active":"2023-10-16T14:17:41.845153+0000","last_peered":"2023-10-16T14:17:41.845153+0000","last_clean":"2023-10-16T14:17:41.845153+0000","last_became_active":"2023-10-16T14:17:40.848272+0000","last_became_peered":"2023-10-16T14:17:40.848272+0000","last_unstale":"2023-10-16T14:17:41.845153+0000","last_undegraded":"2023-10-16T14:17:41.845153+0000","last_fullsized":"2023-10-16T14:17:41.845153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:06.510813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.817810+0000","last_change":"2023-10-16T14:17:40.170369+0000","last_active":"2023-10-16T14:17:41.817810+0000","last_peered":"2023-10-16T14:17:41.817810+0000","last_clean":"2023-10-16T14:17:41.817810+0000","last_became_active":"2023-10-16T14:17:40.170066+0000","last_became_peered":"2023-10-16T14:17:40.170066+0000","last_unstale":"2023-10-16T14:17:41.817810+0000","last_undegraded":"2023-10-16T14:17:41.817810+0000","last_fullsized":"2023-10-16T14:17:41.817810+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:17.999625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844820+0000","last_change":"2023-10-16T14:17:40.846195+0000","last_active":"2023-10-16T14:17:41.844820+0000","last_peered":"2023-10-16T14:17:41.844820+0000","last_clean":"2023-10-16T14:17:41.844820+0000","last_became_active":"2023-10-16T14:17:40.845901+0000","last_became_peered":"2023-10-16T14:17:40.845901+0000","last_unstale":"2023-10-16T14:17:41.844820+0000","last_undegraded":"2023-10-16T14:17:41.844820+0000","last_fullsized":"2023-10-16T14:17:41.844820+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:51.395464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846377+0000","last_change":"2023-10-16T14:17:40.185602+0000","last_active":"2023-10-16T14:17:41.846377+0000","last_peered":"2023-10-16T14:17:41.846377+0000","last_clean":"2023-10-16T14:17:41.846377+0000","last_became_active":"2023-10-16T14:17:40.185322+0000","last_became_peered":"2023-10-16T14:17:40.185322+0000","last_unstale":"2023-10-16T14:17:41.846377+0000","last_undegraded":"2023-10-16T14:17:41.846377+0000","last_fullsized":"2023-10-16T14:17:41.846377+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:17.999625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844547+0000","last_change":"2023-10-16T14:17:40.842568+0000","last_active":"2023-10-16T14:17:41.844547+0000","last_peered":"2023-10-16T14:17:41.844547+0000","last_clean":"2023-10-16T14:17:41.844547+0000","last_became_active":"2023-10-16T14:17:40.839552+0000","last_became_peered":"2023-10-16T14:17:40.839552+0000","last_unstale":"2023-10-16T14:17:41.844547+0000","last_undegraded":"2023-10-16T14:17:41.844547+0000","last_fullsized":"2023-10-16T14:17:41.844547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:01.083679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820793+0000","last_change":"2023-10-16T14:17:40.178755+0000","last_active":"2023-10-16T14:17:41.820793+0000","last_peered":"2023-10-16T14:17:41.820793+0000","last_clean":"2023-10-16T14:17:41.820793+0000","last_became_active":"2023-10-16T14:17:40.178480+0000","last_became_peered":"2023-10-16T14:17:40.178480+0000","last_unstale":"2023-10-16T14:17:41.820793+0000","last_undegraded":"2023-10-16T14:17:41.820793+0000","last_fullsized":"2023-10-16T14:17:41.820793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:05.505579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821705+0000","last_change":"2023-10-16T14:17:40.844412+0000","last_active":"2023-10-16T14:17:41.821705+0000","last_peered":"2023-10-16T14:17:41.821705+0000","last_clean":"2023-10-16T14:17:41.821705+0000","last_became_active":"2023-10-16T14:17:40.844096+0000","last_became_peered":"2023-10-16T14:17:40.844096+0000","last_unstale":"2023-10-16T14:17:41.821705+0000","last_undegraded":"2023-10-16T14:17:41.821705+0000","last_fullsized":"2023-10-16T14:17:41.821705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:24.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823415+0000","last_change":"2023-10-16T14:17:40.183863+0000","last_active":"2023-10-16T14:17:41.823415+0000","last_peered":"2023-10-16T14:17:41.823415+0000","last_clean":"2023-10-16T14:17:41.823415+0000","last_became_active":"2023-10-16T14:17:40.183562+0000","last_became_peered":"2023-10-16T14:17:40.183562+0000","last_unstale":"2023-10-16T14:17:41.823415+0000","last_undegraded":"2023-10-16T14:17:41.823415+0000","last_fullsized":"2023-10-16T14:17:41.823415+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:23.239599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844919+0000","last_change":"2023-10-16T14:17:40.854181+0000","last_active":"2023-10-16T14:17:41.844919+0000","last_peered":"2023-10-16T14:17:41.844919+0000","last_clean":"2023-10-16T14:17:41.844919+0000","last_became_active":"2023-10-16T14:17:40.853150+0000","last_became_peered":"2023-10-16T14:17:40.853150+0000","last_unstale":"2023-10-16T14:17:41.844919+0000","last_undegraded":"2023-10-16T14:17:41.844919+0000","last_fullsized":"2023-10-16T14:17:41.844919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:19.004858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847125+0000","last_change":"2023-10-16T14:17:40.185045+0000","last_active":"2023-10-16T14:17:41.847125+0000","last_peered":"2023-10-16T14:17:41.847125+0000","last_clean":"2023-10-16T14:17:41.847125+0000","last_became_active":"2023-10-16T14:17:40.184309+0000","last_became_peered":"2023-10-16T14:17:40.184309+0000","last_unstale":"2023-10-16T14:17:41.847125+0000","last_undegraded":"2023-10-16T14:17:41.847125+0000","last_fullsized":"2023-10-16T14:17:41.847125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:12.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825221+0000","last_change":"2023-10-16T14:17:40.857321+0000","last_active":"2023-10-16T14:17:41.825221+0000","last_peered":"2023-10-16T14:17:41.825221+0000","last_clean":"2023-10-16T14:17:41.825221+0000","last_became_active":"2023-10-16T14:17:40.857024+0000","last_became_peered":"2023-10-16T14:17:40.857024+0000","last_unstale":"2023-10-16T14:17:41.825221+0000","last_undegraded":"2023-10-16T14:17:41.825221+0000","last_fullsized":"2023-10-16T14:17:41.825221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:47.932423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:42.376898+0000","last_change":"2023-10-16T14:17:40.178661+0000","last_active":"2023-10-16T14:17:42.376898+0000","last_peered":"2023-10-16T14:17:42.376898+0000","last_clean":"2023-10-16T14:17:42.376898+0000","last_became_active":"2023-10-16T14:17:40.178347+0000","last_became_peered":"2023-10-16T14:17:40.178347+0000","last_unstale":"2023-10-16T14:17:42.376898+0000","last_undegraded":"2023-10-16T14:17:42.376898+0000","last_fullsized":"2023-10-16T14:17:42.376898+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.397925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818229+0000","last_change":"2023-10-16T14:17:40.847648+0000","last_active":"2023-10-16T14:17:41.818229+0000","last_peered":"2023-10-16T14:17:41.818229+0000","last_clean":"2023-10-16T14:17:41.818229+0000","last_became_active":"2023-10-16T14:17:40.845959+0000","last_became_peered":"2023-10-16T14:17:40.845959+0000","last_unstale":"2023-10-16T14:17:41.818229+0000","last_undegraded":"2023-10-16T14:17:41.818229+0000","last_fullsized":"2023-10-16T14:17:41.818229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:13.566571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846412+0000","last_change":"2023-10-16T14:17:40.175762+0000","last_active":"2023-10-16T14:17:41.846412+0000","last_peered":"2023-10-16T14:17:41.846412+0000","last_clean":"2023-10-16T14:17:41.846412+0000","last_became_active":"2023-10-16T14:17:40.174027+0000","last_became_peered":"2023-10-16T14:17:40.174027+0000","last_unstale":"2023-10-16T14:17:41.846412+0000","last_undegraded":"2023-10-16T14:17:41.846412+0000","last_fullsized":"2023-10-16T14:17:41.846412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:48.045845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846205+0000","last_change":"2023-10-16T14:17:40.850057+0000","last_active":"2023-10-16T14:17:41.846205+0000","last_peered":"2023-10-16T14:17:41.846205+0000","last_clean":"2023-10-16T14:17:41.846205+0000","last_became_active":"2023-10-16T14:17:40.849121+0000","last_became_peered":"2023-10-16T14:17:40.849121+0000","last_unstale":"2023-10-16T14:17:41.846205+0000","last_undegraded":"2023-10-16T14:17:41.846205+0000","last_fullsized":"2023-10-16T14:17:41.846205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:58.104307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818023+0000","last_change":"2023-10-16T14:17:40.178916+0000","last_active":"2023-10-16T14:17:41.818023+0000","last_peered":"2023-10-16T14:17:41.818023+0000","last_clean":"2023-10-16T14:17:41.818023+0000","last_became_active":"2023-10-16T14:17:40.178615+0000","last_became_peered":"2023-10-16T14:17:40.178615+0000","last_unstale":"2023-10-16T14:17:41.818023+0000","last_undegraded":"2023-10-16T14:17:41.818023+0000","last_fullsized":"2023-10-16T14:17:41.818023+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:05.505579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.817452+0000","last_change":"2023-10-16T14:17:40.844066+0000","last_active":"2023-10-16T14:17:41.817452+0000","last_peered":"2023-10-16T14:17:41.817452+0000","last_clean":"2023-10-16T14:17:41.817452+0000","last_became_active":"2023-10-16T14:17:40.842493+0000","last_became_peered":"2023-10-16T14:17:40.842493+0000","last_unstale":"2023-10-16T14:17:41.817452+0000","last_undegraded":"2023-10-16T14:17:41.817452+0000","last_fullsized":"2023-10-16T14:17:41.817452+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:24.021968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846360+0000","last_change":"2023-10-16T14:17:40.185039+0000","last_active":"2023-10-16T14:17:41.846360+0000","last_peered":"2023-10-16T14:17:41.846360+0000","last_clean":"2023-10-16T14:17:41.846360+0000","last_became_active":"2023-10-16T14:17:40.184716+0000","last_became_peered":"2023-10-16T14:17:40.184716+0000","last_unstale":"2023-10-16T14:17:41.846360+0000","last_undegraded":"2023-10-16T14:17:41.846360+0000","last_fullsized":"2023-10-16T14:17:41.846360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:55.086260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848089+0000","last_change":"2023-10-16T14:17:40.844811+0000","last_active":"2023-10-16T14:17:41.848089+0000","last_peered":"2023-10-16T14:17:41.848089+0000","last_clean":"2023-10-16T14:17:41.848089+0000","last_became_active":"2023-10-16T14:17:40.844360+0000","last_became_peered":"2023-10-16T14:17:40.844360+0000","last_unstale":"2023-10-16T14:17:41.848089+0000","last_undegraded":"2023-10-16T14:17:41.848089+0000","last_fullsized":"2023-10-16T14:17:41.848089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:24.359426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821014+0000","last_change":"2023-10-16T14:17:40.181263+0000","last_active":"2023-10-16T14:17:41.821014+0000","last_peered":"2023-10-16T14:17:41.821014+0000","last_clean":"2023-10-16T14:17:41.821014+0000","last_became_active":"2023-10-16T14:17:40.180873+0000","last_became_peered":"2023-10-16T14:17:40.180873+0000","last_unstale":"2023-10-16T14:17:41.821014+0000","last_undegraded":"2023-10-16T14:17:41.821014+0000","last_fullsized":"2023-10-16T14:17:41.821014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:50.461112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820927+0000","last_change":"2023-10-16T14:17:40.842963+0000","last_active":"2023-10-16T14:17:41.820927+0000","last_peered":"2023-10-16T14:17:41.820927+0000","last_clean":"2023-10-16T14:17:41.820927+0000","last_became_active":"2023-10-16T14:17:40.842665+0000","last_became_peered":"2023-10-16T14:17:40.842665+0000","last_unstale":"2023-10-16T14:17:41.820927+0000","last_undegraded":"2023-10-16T14:17:41.820927+0000","last_fullsized":"2023-10-16T14:17:41.820927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:25.342080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848022+0000","last_change":"2023-10-16T14:17:40.175340+0000","last_active":"2023-10-16T14:17:41.848022+0000","last_peered":"2023-10-16T14:17:41.848022+0000","last_clean":"2023-10-16T14:17:41.848022+0000","last_became_active":"2023-10-16T14:17:40.175079+0000","last_became_peered":"2023-10-16T14:17:40.175079+0000","last_unstale":"2023-10-16T14:17:41.848022+0000","last_undegraded":"2023-10-16T14:17:41.848022+0000","last_fullsized":"2023-10-16T14:17:41.848022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.397925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822793+0000","last_change":"2023-10-16T14:17:40.847603+0000","last_active":"2023-10-16T14:17:41.822793+0000","last_peered":"2023-10-16T14:17:41.822793+0000","last_clean":"2023-10-16T14:17:41.822793+0000","last_became_active":"2023-10-16T14:17:40.847183+0000","last_became_peered":"2023-10-16T14:17:40.847183+0000","last_unstale":"2023-10-16T14:17:41.822793+0000","last_undegraded":"2023-10-16T14:17:41.822793+0000","last_fullsized":"2023-10-16T14:17:41.822793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:23.202538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846204+0000","last_change":"2023-10-16T14:17:40.184287+0000","last_active":"2023-10-16T14:17:41.846204+0000","last_peered":"2023-10-16T14:17:41.846204+0000","last_clean":"2023-10-16T14:17:41.846204+0000","last_became_active":"2023-10-16T14:17:40.183992+0000","last_became_peered":"2023-10-16T14:17:40.183992+0000","last_unstale":"2023-10-16T14:17:41.846204+0000","last_undegraded":"2023-10-16T14:17:41.846204+0000","last_fullsized":"2023-10-16T14:17:41.846204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:00.078445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-16T14:17:40.820960+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:17:40.820960+0000","last_peered":"2023-10-16T14:17:40.820960+0000","last_clean":"2023-10-16T14:17:40.820960+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:17:40.820960+0000","last_undegraded":"2023-10-16T14:17:40.820960+0000","last_fullsized":"2023-10-16T14:17:40.820960+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":970752,"data_stored":737892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.81499999999999995}]},{"osd":1,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.018}]},{"osd":3,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.153},{"interface":"front","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.304},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.343}]},{"osd":5,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.256}]},{"osd":6,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.423},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.4970000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":148050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":2.2160000000000002},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":3.863}]},{"osd":1,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":3.7879999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":3.5800000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":2.0179999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":4.9790000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":5.4160000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":5},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":5.4640000000000004}]},{"osd":5,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":3.9279999999999999},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.8330000000000002,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":3.6749999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.663,"15min":1.663},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":4.6859999999999999},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":4.883}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":380928,"data_stored":148106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.3540000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.876},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.087}]},{"osd":2,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.2},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.8680000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.25,"15min":1.25},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.419},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":1.8169999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.863,"5min":2.863,"15min":2.863},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.9059999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.762},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":3.794,"5min":3.794,"15min":3.794},"last":1.1379999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.615,"5min":1.615,"15min":1.615},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.9990000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":147538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"last":1.2749999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":1.675},{"interface":"front","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":5.4640000000000004,"5min":5.4640000000000004,"15min":5.4640000000000004},"last":1.6479999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":5.617,"5min":5.617,"15min":5.617},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.702}]},{"osd":3,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":7.4089999999999998,"5min":7.4089999999999998,"15min":7.4089999999999998},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.536}]},{"osd":5,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":5.4109999999999996,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":9.8239999999999998,"5min":9.8239999999999998,"15min":9.8239999999999998},"last":1.587}]},{"osd":6,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":5.782,"5min":5.782,"15min":5.782},"last":1.5680000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":3.7799999999999998,"5min":3.7799999999999998,"15min":3.7799999999999998},"last":1.6180000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":389120,"data_stored":149068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.4420000000000002,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":1.002},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.288}]},{"osd":2,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.23},{"interface":"front","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.3440000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":1.073},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.319}]},{"osd":5,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.431},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.4910000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":1.268},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.3799999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.667,"15min":1.667},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.161}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":376832,"data_stored":147504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.88100000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":3.25,"5min":3.25,"15min":3.25},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.9350000000000001,"5min":2.9350000000000001,"15min":2.9350000000000001},"last":1.484}]},{"osd":6,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.536,"5min":3.536,"15min":3.536},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.085}]},{"osd":7,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":5.2199999999999998,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.181}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":389120,"data_stored":149068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.61099999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.9820000000000002,"5min":2.9820000000000002,"15min":2.9820000000000002},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.879},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":1.107,"5min":1.107,"15min":1.107},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.73099999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":1.204,"5min":1.204,"15min":1.204},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.492,"5min":1.492,"15min":1.492},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.3410000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.077}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":966656,"data_stored":737802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.3320000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.165,"15min":1.165},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.653,"5min":3.653,"15min":3.653},"last":1.9379999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":3.1400000000000001,"5min":3.1400000000000001,"15min":3.1400000000000001},"last":1.718}]},{"osd":3,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.669},{"interface":"front","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.7649999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":1.534},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":3.6190000000000002,"5min":3.6190000000000002,"15min":3.6190000000000002},"last":2.1699999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":3.2370000000000001,"5min":3.2370000000000001,"15min":3.2370000000000001},"last":1.595},{"interface":"front","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.819,"5min":2.819,"15min":2.819},"last":1.1839999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":2.0539999999999998},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":3.0979999999999999,"5min":3.0979999999999999,"15min":3.0979999999999999},"last":1.26}]},{"osd":7,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.829},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"max":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"last":1.8740000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:17:45.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:17:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.966+0000 7f55e2060640 1 Processor -- start 2023-10-16T14:17:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.967+0000 7f55e2060640 1 -- start start 2023-10-16T14:17:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.968+0000 7f55e2060640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f55dc0b4c80 0x7f55dc167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.968+0000 7f55e2060640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0d15a0 con 0x7f55dc15cbe0 2023-10-16T14:17:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55e2060640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0d1880 con 0x7f55dc15cfc0 2023-10-16T14:17:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55e2060640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0d1b60 con 0x7f55dc0b4c80 2023-10-16T14:17:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55e105e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f55dc15cbe0 0x7f55dc0d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54628/0 (socket says 172.21.15.29:54628) 2023-10-16T14:17:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55e105e640 1 -- 172.21.15.29:0/3141852655 learned_addr learned my addr 172.21.15.29:0/3141852655 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55e185f640 1 --2- 172.21.15.29:0/3141852655 >> v2:172.21.15.170:3300/0 conn(0x7f55dc0b4c80 0x7f55dc167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103024919 0 0) 0x7f55dc0d15a0 con 0x7f55dc15cbe0 2023-10-16T14:17:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.969+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c8003520 con 0x7f55dc15cbe0 2023-10-16T14:17:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903786213 0 0) 0x7f55c8003520 con 0x7f55dc15cbe0 2023-10-16T14:17:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 >> v2:172.21.15.170:3300/0 conn(0x7f55dc0b4c80 msgr2=0x7f55dc167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 --2- 172.21.15.29:0/3141852655 >> v2:172.21.15.170:3300/0 conn(0x7f55dc0b4c80 0x7f55dc167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 >> v1:172.21.15.170:6789/0 conn(0x7f55dc15cfc0 legacy=0x7f55dc0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0d1e40 con 0x7f55dc15cbe0 2023-10-16T14:17:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f55d8001670 con 0x7f55dc15cbe0 2023-10-16T14:17:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 >> v1:172.21.15.29:6789/0 conn(0x7f55dc15cbe0 legacy=0x7f55dc0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55c8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c8005180 0x7f55c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0d1b60 con 0x7f55c8003970 2023-10-16T14:17:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c800c000 con 0x7f55c80047c0 2023-10-16T14:17:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c800c1c0 con 0x7f55c8005180 2023-10-16T14:17:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55e105e640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55c8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.970+0000 7f55e085d640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.971+0000 7f55e105e640 1 -- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c8005180 msgr2=0x7f55c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.971+0000 7f55e105e640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c8005180 0x7f55c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.971+0000 7f55e105e640 1 -- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c80047c0 msgr2=0x7f55c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.971+0000 7f55e105e640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.971+0000 7f55e105e640 1 -- 172.21.15.29:0/3141852655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c8003520 con 0x7f55c8003970 2023-10-16T14:17:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.971+0000 7f55e185f640 1 -- 172.21.15.29:0/3141852655 reap_dead start 2023-10-16T14:17:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.972+0000 7f55e105e640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55c8003d70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f55d800c240 tx=0x7f55c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5800a889c672c40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.972+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d8005ad0 con 0x7f55c8003970 2023-10-16T14:17:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.972+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d8005c30 con 0x7f55c8003970 2023-10-16T14:17:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.972+0000 7f55d3fff640 1 -- 172.21.15.29:0/3141852655 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d802c890 con 0x7f55c8003970 2023-10-16T14:17:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 -- 172.21.15.29:0/3141852655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 msgr2=0x7f55c8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55c8003d70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f55d800c240 tx=0x7f55c80057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 -- 172.21.15.29:0/3141852655 shutdown_connections 2023-10-16T14:17:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 --2- 172.21.15.29:0/3141852655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55c8003d70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 -- 172.21.15.29:0/3141852655 >> 172.21.15.29:0/3141852655 conn(0x7f55dc0cf570 msgr2=0x7f55dc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 -- 172.21.15.29:0/3141852655 shutdown_connections 2023-10-16T14:17:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.973+0000 7f55e2060640 1 -- 172.21.15.29:0/3141852655 wait complete. 2023-10-16T14:17:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.975+0000 7f55e2060640 1 Processor -- start 2023-10-16T14:17:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.975+0000 7f55e2060640 1 -- start start 2023-10-16T14:17:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e2060640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55dc1f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e2060640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55dc15dfa0 0x7f55dc0b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e2060640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55dc0b70c0 0x7f55dc156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e2060640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc158830 con 0x7f55c8003970 2023-10-16T14:17:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e2060640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0b7540 con 0x7f55dc0b70c0 2023-10-16T14:17:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e2060640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc15e9d0 con 0x7f55dc15dfa0 2023-10-16T14:17:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e105e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55dc1f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e105e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55dc1f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37288/0 (socket says 172.21.15.29:37288) 2023-10-16T14:17:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.976+0000 7f55e105e640 1 -- 172.21.15.29:0/3539427328 learned_addr learned my addr 172.21.15.29:0/3539427328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e185f640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55dc0b70c0 0x7f55dc156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e085d640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55dc15dfa0 0x7f55dc0b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e105e640 1 -- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55dc15dfa0 msgr2=0x7f55dc0b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e105e640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55dc15dfa0 0x7f55dc0b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e105e640 1 -- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55dc0b70c0 msgr2=0x7f55dc156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:45.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e105e640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55dc0b70c0 0x7f55dc156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:45.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e105e640 1 -- 172.21.15.29:0/3539427328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc1f28c0 con 0x7f55c8003970 2023-10-16T14:17:45.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e185f640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55dc0b70c0 0x7f55dc156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:45.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e085d640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55dc15dfa0 0x7f55dc0b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:45.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e105e640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55dc1f2380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f55d802c430 tx=0x7f55d802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:45.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d803c040 con 0x7f55c8003970 2023-10-16T14:17:45.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d802cca0 con 0x7f55c8003970 2023-10-16T14:17:45.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d8007270 con 0x7f55c8003970 2023-10-16T14:17:45.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.977+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc1573d0 con 0x7f55c8003970 2023-10-16T14:17:45.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.978+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc15ec20 con 0x7f55c8003970 2023-10-16T14:17:45.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.979+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc0cfea0 con 0x7f55c8003970 2023-10-16T14:17:45.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.980+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f55d8032070 con 0x7f55c8003970 2023-10-16T14:17:45.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.980+0000 7f55d1ffb640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f55c006ae70 0x7f55c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:45.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.981+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55d80aa710 con 0x7f55c8003970 2023-10-16T14:17:45.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.981+0000 7f55e085d640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f55c006ae70 0x7f55c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:45.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.982+0000 7f55e085d640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f55c006ae70 0x7f55c006d330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0d0f10 tx=0x7f55cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:45.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:45.985+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f55d8075900 con 0x7f55c8003970 2023-10-16T14:17:46.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.173+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55dc0d2120 con 0x7f55c006ae70 2023-10-16T14:17:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.196+0000 7f55d1ffb640 1 -- 172.21.15.29:0/3539427328 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286291 (secure 0 0 0) 0x7f55cc0143f0 con 0x7f55c006ae70 2023-10-16T14:17:46.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:17:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f55c006ae70 msgr2=0x7f55c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f55c006ae70 0x7f55c006d330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0d0f10 tx=0x7f55cc009040 comp rx=0 tx=0).stop 2023-10-16T14:17:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 msgr2=0x7f55dc1f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55dc1f2380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f55d802c430 tx=0x7f55d802e070 comp rx=0 tx=0).stop 2023-10-16T14:17:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 shutdown_connections 2023-10-16T14:17:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f55c006ae70 0x7f55c006d330 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55dc0b70c0 0x7f55dc156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55dc15dfa0 0x7f55dc0b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 --2- 172.21.15.29:0/3539427328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003970 0x7f55dc1f2380 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.203+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 >> 172.21.15.29:0/3539427328 conn(0x7f55dc0cf570 msgr2=0x7f55dc0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.204+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 shutdown_connections 2023-10-16T14:17:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.204+0000 7f55e2060640 1 -- 172.21.15.29:0/3539427328 wait complete. 2023-10-16T14:17:46.230 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-10-16T14:17:45.841558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253616,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721104,"statfs":{"total":773094113280,"available":772834410496,"internally_reserved":0,"allocated":4235264,"data_stored":2365028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":255315513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002439"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846525+0000","last_change":"2023-10-16T14:17:40.848116+0000","last_active":"2023-10-16T14:17:41.846525+0000","last_peered":"2023-10-16T14:17:41.846525+0000","last_clean":"2023-10-16T14:17:41.846525+0000","last_became_active":"2023-10-16T14:17:40.847833+0000","last_became_peered":"2023-10-16T14:17:40.847833+0000","last_unstale":"2023-10-16T14:17:41.846525+0000","last_undegraded":"2023-10-16T14:17:41.846525+0000","last_fullsized":"2023-10-16T14:17:41.846525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:49.051078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848251+0000","last_change":"2023-10-16T14:17:40.845617+0000","last_active":"2023-10-16T14:17:41.848251+0000","last_peered":"2023-10-16T14:17:41.848251+0000","last_clean":"2023-10-16T14:17:41.848251+0000","last_became_active":"2023-10-16T14:17:40.845339+0000","last_became_peered":"2023-10-16T14:17:40.845339+0000","last_unstale":"2023-10-16T14:17:41.848251+0000","last_undegraded":"2023-10-16T14:17:41.848251+0000","last_fullsized":"2023-10-16T14:17:41.848251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:18.938616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825063+0000","last_change":"2023-10-16T14:17:40.854631+0000","last_active":"2023-10-16T14:17:41.825063+0000","last_peered":"2023-10-16T14:17:41.825063+0000","last_clean":"2023-10-16T14:17:41.825063+0000","last_became_active":"2023-10-16T14:17:40.854206+0000","last_became_peered":"2023-10-16T14:17:40.854206+0000","last_unstale":"2023-10-16T14:17:41.825063+0000","last_undegraded":"2023-10-16T14:17:41.825063+0000","last_fullsized":"2023-10-16T14:17:41.825063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:52.774190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846197+0000","last_change":"2023-10-16T14:17:40.853101+0000","last_active":"2023-10-16T14:17:41.846197+0000","last_peered":"2023-10-16T14:17:41.846197+0000","last_clean":"2023-10-16T14:17:41.846197+0000","last_became_active":"2023-10-16T14:17:40.852148+0000","last_became_peered":"2023-10-16T14:17:40.852148+0000","last_unstale":"2023-10-16T14:17:41.846197+0000","last_undegraded":"2023-10-16T14:17:41.846197+0000","last_fullsized":"2023-10-16T14:17:41.846197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:25.342080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844564+0000","last_change":"2023-10-16T14:17:40.849264+0000","last_active":"2023-10-16T14:17:41.844564+0000","last_peered":"2023-10-16T14:17:41.844564+0000","last_clean":"2023-10-16T14:17:41.844564+0000","last_became_active":"2023-10-16T14:17:40.848934+0000","last_became_peered":"2023-10-16T14:17:40.848934+0000","last_unstale":"2023-10-16T14:17:41.844564+0000","last_undegraded":"2023-10-16T14:17:41.844564+0000","last_fullsized":"2023-10-16T14:17:41.844564+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:09.847786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847850+0000","last_change":"2023-10-16T14:17:40.840888+0000","last_active":"2023-10-16T14:17:41.847850+0000","last_peered":"2023-10-16T14:17:41.847850+0000","last_clean":"2023-10-16T14:17:41.847850+0000","last_became_active":"2023-10-16T14:17:40.840609+0000","last_became_peered":"2023-10-16T14:17:40.840609+0000","last_unstale":"2023-10-16T14:17:41.847850+0000","last_undegraded":"2023-10-16T14:17:41.847850+0000","last_fullsized":"2023-10-16T14:17:41.847850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:00.040733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822209+0000","last_change":"2023-10-16T14:17:40.840540+0000","last_active":"2023-10-16T14:17:41.822209+0000","last_peered":"2023-10-16T14:17:41.822209+0000","last_clean":"2023-10-16T14:17:41.822209+0000","last_became_active":"2023-10-16T14:17:40.840235+0000","last_became_peered":"2023-10-16T14:17:40.840235+0000","last_unstale":"2023-10-16T14:17:41.822209+0000","last_undegraded":"2023-10-16T14:17:41.822209+0000","last_fullsized":"2023-10-16T14:17:41.822209+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:02.808237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822896+0000","last_change":"2023-10-16T14:17:40.846680+0000","last_active":"2023-10-16T14:17:41.822896+0000","last_peered":"2023-10-16T14:17:41.822896+0000","last_clean":"2023-10-16T14:17:41.822896+0000","last_became_active":"2023-10-16T14:17:40.846344+0000","last_became_peered":"2023-10-16T14:17:40.846344+0000","last_unstale":"2023-10-16T14:17:41.822896+0000","last_undegraded":"2023-10-16T14:17:41.822896+0000","last_fullsized":"2023-10-16T14:17:41.822896+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:00.040733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822930+0000","last_change":"2023-10-16T14:17:40.852056+0000","last_active":"2023-10-16T14:17:41.822930+0000","last_peered":"2023-10-16T14:17:41.822930+0000","last_clean":"2023-10-16T14:17:41.822930+0000","last_became_active":"2023-10-16T14:17:40.851475+0000","last_became_peered":"2023-10-16T14:17:40.851475+0000","last_unstale":"2023-10-16T14:17:41.822930+0000","last_undegraded":"2023-10-16T14:17:41.822930+0000","last_fullsized":"2023-10-16T14:17:41.822930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:30.110067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846377+0000","last_change":"2023-10-16T14:17:40.840670+0000","last_active":"2023-10-16T14:17:41.846377+0000","last_peered":"2023-10-16T14:17:41.846377+0000","last_clean":"2023-10-16T14:17:41.846377+0000","last_became_active":"2023-10-16T14:17:40.840122+0000","last_became_peered":"2023-10-16T14:17:40.840122+0000","last_unstale":"2023-10-16T14:17:41.846377+0000","last_undegraded":"2023-10-16T14:17:41.846377+0000","last_fullsized":"2023-10-16T14:17:41.846377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:30.886976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821789+0000","last_change":"2023-10-16T14:17:40.847394+0000","last_active":"2023-10-16T14:17:41.821789+0000","last_peered":"2023-10-16T14:17:41.821789+0000","last_clean":"2023-10-16T14:17:41.821789+0000","last_became_active":"2023-10-16T14:17:40.846918+0000","last_became_peered":"2023-10-16T14:17:40.846918+0000","last_unstale":"2023-10-16T14:17:41.821789+0000","last_undegraded":"2023-10-16T14:17:41.821789+0000","last_fullsized":"2023-10-16T14:17:41.821789+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:30.110067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824405+0000","last_change":"2023-10-16T14:17:40.843723+0000","last_active":"2023-10-16T14:17:41.824405+0000","last_peered":"2023-10-16T14:17:41.824405+0000","last_clean":"2023-10-16T14:17:41.824405+0000","last_became_active":"2023-10-16T14:17:40.843451+0000","last_became_peered":"2023-10-16T14:17:40.843451+0000","last_unstale":"2023-10-16T14:17:41.824405+0000","last_undegraded":"2023-10-16T14:17:41.824405+0000","last_fullsized":"2023-10-16T14:17:41.824405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:46.139143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844690+0000","last_change":"2023-10-16T14:17:40.855177+0000","last_active":"2023-10-16T14:17:41.844690+0000","last_peered":"2023-10-16T14:17:41.844690+0000","last_clean":"2023-10-16T14:17:41.844690+0000","last_became_active":"2023-10-16T14:17:40.854836+0000","last_became_peered":"2023-10-16T14:17:40.854836+0000","last_unstale":"2023-10-16T14:17:41.844690+0000","last_undegraded":"2023-10-16T14:17:41.844690+0000","last_fullsized":"2023-10-16T14:17:41.844690+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:06.304287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848123+0000","last_change":"2023-10-16T14:17:40.847370+0000","last_active":"2023-10-16T14:17:41.848123+0000","last_peered":"2023-10-16T14:17:41.848123+0000","last_clean":"2023-10-16T14:17:41.848123+0000","last_became_active":"2023-10-16T14:17:40.847040+0000","last_became_peered":"2023-10-16T14:17:40.847040+0000","last_unstale":"2023-10-16T14:17:41.848123+0000","last_undegraded":"2023-10-16T14:17:41.848123+0000","last_fullsized":"2023-10-16T14:17:41.848123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:52.774190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823887+0000","last_change":"2023-10-16T14:17:40.841785+0000","last_active":"2023-10-16T14:17:41.823887+0000","last_peered":"2023-10-16T14:17:41.823887+0000","last_clean":"2023-10-16T14:17:41.823887+0000","last_became_active":"2023-10-16T14:17:40.841421+0000","last_became_peered":"2023-10-16T14:17:40.841421+0000","last_unstale":"2023-10-16T14:17:41.823887+0000","last_undegraded":"2023-10-16T14:17:41.823887+0000","last_fullsized":"2023-10-16T14:17:41.823887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:38.253474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822810+0000","last_change":"2023-10-16T14:17:40.849272+0000","last_active":"2023-10-16T14:17:41.822810+0000","last_peered":"2023-10-16T14:17:41.822810+0000","last_clean":"2023-10-16T14:17:41.822810+0000","last_became_active":"2023-10-16T14:17:40.848910+0000","last_became_peered":"2023-10-16T14:17:40.848910+0000","last_unstale":"2023-10-16T14:17:41.822810+0000","last_undegraded":"2023-10-16T14:17:41.822810+0000","last_fullsized":"2023-10-16T14:17:41.822810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:46.139143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822433+0000","last_change":"2023-10-16T14:17:40.844851+0000","last_active":"2023-10-16T14:17:41.822433+0000","last_peered":"2023-10-16T14:17:41.822433+0000","last_clean":"2023-10-16T14:17:41.822433+0000","last_became_active":"2023-10-16T14:17:40.844562+0000","last_became_peered":"2023-10-16T14:17:40.844562+0000","last_unstale":"2023-10-16T14:17:41.822433+0000","last_undegraded":"2023-10-16T14:17:41.822433+0000","last_fullsized":"2023-10-16T14:17:41.822433+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:38.253474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848386+0000","last_change":"2023-10-16T14:17:40.843494+0000","last_active":"2023-10-16T14:17:41.848386+0000","last_peered":"2023-10-16T14:17:41.848386+0000","last_clean":"2023-10-16T14:17:41.848386+0000","last_became_active":"2023-10-16T14:17:40.843203+0000","last_became_peered":"2023-10-16T14:17:40.843203+0000","last_unstale":"2023-10-16T14:17:41.848386+0000","last_undegraded":"2023-10-16T14:17:41.848386+0000","last_fullsized":"2023-10-16T14:17:41.848386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:56.558827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848237+0000","last_change":"2023-10-16T14:17:40.847361+0000","last_active":"2023-10-16T14:17:41.848237+0000","last_peered":"2023-10-16T14:17:41.848237+0000","last_clean":"2023-10-16T14:17:41.848237+0000","last_became_active":"2023-10-16T14:17:40.846980+0000","last_became_peered":"2023-10-16T14:17:40.846980+0000","last_unstale":"2023-10-16T14:17:41.848237+0000","last_undegraded":"2023-10-16T14:17:41.848237+0000","last_fullsized":"2023-10-16T14:17:41.848237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:47.932423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.845423+0000","last_change":"2023-10-16T14:17:40.852470+0000","last_active":"2023-10-16T14:17:41.845423+0000","last_peered":"2023-10-16T14:17:41.845423+0000","last_clean":"2023-10-16T14:17:41.845423+0000","last_became_active":"2023-10-16T14:17:40.848746+0000","last_became_peered":"2023-10-16T14:17:40.848746+0000","last_unstale":"2023-10-16T14:17:41.845423+0000","last_undegraded":"2023-10-16T14:17:41.845423+0000","last_fullsized":"2023-10-16T14:17:41.845423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:51.466345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821529+0000","last_change":"2023-10-16T14:17:40.841447+0000","last_active":"2023-10-16T14:17:41.821529+0000","last_peered":"2023-10-16T14:17:41.821529+0000","last_clean":"2023-10-16T14:17:41.821529+0000","last_became_active":"2023-10-16T14:17:40.841027+0000","last_became_peered":"2023-10-16T14:17:40.841027+0000","last_unstale":"2023-10-16T14:17:41.821529+0000","last_undegraded":"2023-10-16T14:17:41.821529+0000","last_fullsized":"2023-10-16T14:17:41.821529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:55.629805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821507+0000","last_change":"2023-10-16T14:17:40.842810+0000","last_active":"2023-10-16T14:17:41.821507+0000","last_peered":"2023-10-16T14:17:41.821507+0000","last_clean":"2023-10-16T14:17:41.821507+0000","last_became_active":"2023-10-16T14:17:40.842477+0000","last_became_peered":"2023-10-16T14:17:40.842477+0000","last_unstale":"2023-10-16T14:17:41.821507+0000","last_undegraded":"2023-10-16T14:17:41.821507+0000","last_fullsized":"2023-10-16T14:17:41.821507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.991974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.817081+0000","last_change":"2023-10-16T14:17:40.839037+0000","last_active":"2023-10-16T14:17:41.817081+0000","last_peered":"2023-10-16T14:17:41.817081+0000","last_clean":"2023-10-16T14:17:41.817081+0000","last_became_active":"2023-10-16T14:17:40.838566+0000","last_became_peered":"2023-10-16T14:17:40.838566+0000","last_unstale":"2023-10-16T14:17:41.817081+0000","last_undegraded":"2023-10-16T14:17:41.817081+0000","last_fullsized":"2023-10-16T14:17:41.817081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:49.051078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821720+0000","last_change":"2023-10-16T14:17:40.844318+0000","last_active":"2023-10-16T14:17:41.821720+0000","last_peered":"2023-10-16T14:17:41.821720+0000","last_clean":"2023-10-16T14:17:41.821720+0000","last_became_active":"2023-10-16T14:17:40.844015+0000","last_became_peered":"2023-10-16T14:17:40.844015+0000","last_unstale":"2023-10-16T14:17:41.821720+0000","last_undegraded":"2023-10-16T14:17:41.821720+0000","last_fullsized":"2023-10-16T14:17:41.821720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:02.808237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846998+0000","last_change":"2023-10-16T14:17:40.854768+0000","last_active":"2023-10-16T14:17:41.846998+0000","last_peered":"2023-10-16T14:17:41.846998+0000","last_clean":"2023-10-16T14:17:41.846998+0000","last_became_active":"2023-10-16T14:17:40.854194+0000","last_became_peered":"2023-10-16T14:17:40.854194+0000","last_unstale":"2023-10-16T14:17:41.846998+0000","last_undegraded":"2023-10-16T14:17:41.846998+0000","last_fullsized":"2023-10-16T14:17:41.846998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:55.629805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848469+0000","last_change":"2023-10-16T14:17:40.838964+0000","last_active":"2023-10-16T14:17:41.848469+0000","last_peered":"2023-10-16T14:17:41.848469+0000","last_clean":"2023-10-16T14:17:41.848469+0000","last_became_active":"2023-10-16T14:17:40.838602+0000","last_became_peered":"2023-10-16T14:17:40.838602+0000","last_unstale":"2023-10-16T14:17:41.848469+0000","last_undegraded":"2023-10-16T14:17:41.848469+0000","last_fullsized":"2023-10-16T14:17:41.848469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:08.364442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847109+0000","last_change":"2023-10-16T14:17:40.855205+0000","last_active":"2023-10-16T14:17:41.847109+0000","last_peered":"2023-10-16T14:17:41.847109+0000","last_clean":"2023-10-16T14:17:41.847109+0000","last_became_active":"2023-10-16T14:17:40.852783+0000","last_became_peered":"2023-10-16T14:17:40.852783+0000","last_unstale":"2023-10-16T14:17:41.847109+0000","last_undegraded":"2023-10-16T14:17:41.847109+0000","last_fullsized":"2023-10-16T14:17:41.847109+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:24.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824611+0000","last_change":"2023-10-16T14:17:40.847434+0000","last_active":"2023-10-16T14:17:41.824611+0000","last_peered":"2023-10-16T14:17:41.824611+0000","last_clean":"2023-10-16T14:17:41.824611+0000","last_became_active":"2023-10-16T14:17:40.846738+0000","last_became_peered":"2023-10-16T14:17:40.846738+0000","last_unstale":"2023-10-16T14:17:41.824611+0000","last_undegraded":"2023-10-16T14:17:41.824611+0000","last_fullsized":"2023-10-16T14:17:41.824611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:00.040733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821876+0000","last_change":"2023-10-16T14:17:40.849150+0000","last_active":"2023-10-16T14:17:41.821876+0000","last_peered":"2023-10-16T14:17:41.821876+0000","last_clean":"2023-10-16T14:17:41.821876+0000","last_became_active":"2023-10-16T14:17:40.848879+0000","last_became_peered":"2023-10-16T14:17:40.848879+0000","last_unstale":"2023-10-16T14:17:41.821876+0000","last_undegraded":"2023-10-16T14:17:41.821876+0000","last_fullsized":"2023-10-16T14:17:41.821876+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:24.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822969+0000","last_change":"2023-10-16T14:17:40.840727+0000","last_active":"2023-10-16T14:17:41.822969+0000","last_peered":"2023-10-16T14:17:41.822969+0000","last_clean":"2023-10-16T14:17:41.822969+0000","last_became_active":"2023-10-16T14:17:40.840414+0000","last_became_peered":"2023-10-16T14:17:40.840414+0000","last_unstale":"2023-10-16T14:17:41.822969+0000","last_undegraded":"2023-10-16T14:17:41.822969+0000","last_fullsized":"2023-10-16T14:17:41.822969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:24.359426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846100+0000","last_change":"2023-10-16T14:17:40.844158+0000","last_active":"2023-10-16T14:17:41.846100+0000","last_peered":"2023-10-16T14:17:41.846100+0000","last_clean":"2023-10-16T14:17:41.846100+0000","last_became_active":"2023-10-16T14:17:40.843829+0000","last_became_peered":"2023-10-16T14:17:40.843829+0000","last_unstale":"2023-10-16T14:17:41.846100+0000","last_undegraded":"2023-10-16T14:17:41.846100+0000","last_fullsized":"2023-10-16T14:17:41.846100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:51.466345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821233+0000","last_change":"2023-10-16T14:17:40.836246+0000","last_active":"2023-10-16T14:17:41.821233+0000","last_peered":"2023-10-16T14:17:41.821233+0000","last_clean":"2023-10-16T14:17:41.821233+0000","last_became_active":"2023-10-16T14:17:40.835953+0000","last_became_peered":"2023-10-16T14:17:40.835953+0000","last_unstale":"2023-10-16T14:17:41.821233+0000","last_undegraded":"2023-10-16T14:17:41.821233+0000","last_fullsized":"2023-10-16T14:17:41.821233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.991974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818287+0000","last_change":"2023-10-16T14:17:40.847268+0000","last_active":"2023-10-16T14:17:41.818287+0000","last_peered":"2023-10-16T14:17:41.818287+0000","last_clean":"2023-10-16T14:17:41.818287+0000","last_became_active":"2023-10-16T14:17:40.846956+0000","last_became_peered":"2023-10-16T14:17:40.846956+0000","last_unstale":"2023-10-16T14:17:41.818287+0000","last_undegraded":"2023-10-16T14:17:41.818287+0000","last_fullsized":"2023-10-16T14:17:41.818287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:28.134316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846662+0000","last_change":"2023-10-16T14:17:40.184353+0000","last_active":"2023-10-16T14:17:41.846662+0000","last_peered":"2023-10-16T14:17:41.846662+0000","last_clean":"2023-10-16T14:17:41.846662+0000","last_became_active":"2023-10-16T14:17:40.184046+0000","last_became_peered":"2023-10-16T14:17:40.184046+0000","last_unstale":"2023-10-16T14:17:41.846662+0000","last_undegraded":"2023-10-16T14:17:41.846662+0000","last_fullsized":"2023-10-16T14:17:41.846662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:05.505579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846527+0000","last_change":"2023-10-16T14:17:40.853333+0000","last_active":"2023-10-16T14:17:41.846527+0000","last_peered":"2023-10-16T14:17:41.846527+0000","last_clean":"2023-10-16T14:17:41.846527+0000","last_became_active":"2023-10-16T14:17:40.852060+0000","last_became_peered":"2023-10-16T14:17:40.852060+0000","last_unstale":"2023-10-16T14:17:41.846527+0000","last_undegraded":"2023-10-16T14:17:41.846527+0000","last_fullsized":"2023-10-16T14:17:41.846527+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:55.403159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818337+0000","last_change":"2023-10-16T14:17:40.170561+0000","last_active":"2023-10-16T14:17:41.818337+0000","last_peered":"2023-10-16T14:17:41.818337+0000","last_clean":"2023-10-16T14:17:41.818337+0000","last_became_active":"2023-10-16T14:17:40.170277+0000","last_became_peered":"2023-10-16T14:17:40.170277+0000","last_unstale":"2023-10-16T14:17:41.818337+0000","last_undegraded":"2023-10-16T14:17:41.818337+0000","last_fullsized":"2023-10-16T14:17:41.818337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:07.707835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820934+0000","last_change":"2023-10-16T14:17:40.842954+0000","last_active":"2023-10-16T14:17:41.820934+0000","last_peered":"2023-10-16T14:17:41.820934+0000","last_clean":"2023-10-16T14:17:41.820934+0000","last_became_active":"2023-10-16T14:17:40.842521+0000","last_became_peered":"2023-10-16T14:17:40.842521+0000","last_unstale":"2023-10-16T14:17:41.820934+0000","last_undegraded":"2023-10-16T14:17:41.820934+0000","last_fullsized":"2023-10-16T14:17:41.820934+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:30.886976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848485+0000","last_change":"2023-10-16T14:17:40.180609+0000","last_active":"2023-10-16T14:17:41.848485+0000","last_peered":"2023-10-16T14:17:41.848485+0000","last_clean":"2023-10-16T14:17:41.848485+0000","last_became_active":"2023-10-16T14:17:40.180248+0000","last_became_peered":"2023-10-16T14:17:40.180248+0000","last_unstale":"2023-10-16T14:17:41.848485+0000","last_undegraded":"2023-10-16T14:17:41.848485+0000","last_fullsized":"2023-10-16T14:17:41.848485+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:24.336846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847123+0000","last_change":"2023-10-16T14:17:40.855526+0000","last_active":"2023-10-16T14:17:41.847123+0000","last_peered":"2023-10-16T14:17:41.847123+0000","last_clean":"2023-10-16T14:17:41.847123+0000","last_became_active":"2023-10-16T14:17:40.853150+0000","last_became_peered":"2023-10-16T14:17:40.853150+0000","last_unstale":"2023-10-16T14:17:41.847123+0000","last_undegraded":"2023-10-16T14:17:41.847123+0000","last_fullsized":"2023-10-16T14:17:41.847123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:30.110067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.845402+0000","last_change":"2023-10-16T14:17:40.179066+0000","last_active":"2023-10-16T14:17:41.845402+0000","last_peered":"2023-10-16T14:17:41.845402+0000","last_clean":"2023-10-16T14:17:41.845402+0000","last_became_active":"2023-10-16T14:17:40.177489+0000","last_became_peered":"2023-10-16T14:17:40.177489+0000","last_unstale":"2023-10-16T14:17:41.845402+0000","last_undegraded":"2023-10-16T14:17:41.845402+0000","last_fullsized":"2023-10-16T14:17:41.845402+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:02:31.868085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823322+0000","last_change":"2023-10-16T14:17:40.842227+0000","last_active":"2023-10-16T14:17:41.823322+0000","last_peered":"2023-10-16T14:17:41.823322+0000","last_clean":"2023-10-16T14:17:41.823322+0000","last_became_active":"2023-10-16T14:17:40.841922+0000","last_became_peered":"2023-10-16T14:17:40.841922+0000","last_unstale":"2023-10-16T14:17:41.823322+0000","last_undegraded":"2023-10-16T14:17:41.823322+0000","last_fullsized":"2023-10-16T14:17:41.823322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:18.224519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822009+0000","last_change":"2023-10-16T14:17:40.182015+0000","last_active":"2023-10-16T14:17:41.822009+0000","last_peered":"2023-10-16T14:17:41.822009+0000","last_clean":"2023-10-16T14:17:41.822009+0000","last_became_active":"2023-10-16T14:17:40.181734+0000","last_became_peered":"2023-10-16T14:17:40.181734+0000","last_unstale":"2023-10-16T14:17:41.822009+0000","last_undegraded":"2023-10-16T14:17:41.822009+0000","last_fullsized":"2023-10-16T14:17:41.822009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.397925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824190+0000","last_change":"2023-10-16T14:17:40.849311+0000","last_active":"2023-10-16T14:17:41.824190+0000","last_peered":"2023-10-16T14:17:41.824190+0000","last_clean":"2023-10-16T14:17:41.824190+0000","last_became_active":"2023-10-16T14:17:40.847998+0000","last_became_peered":"2023-10-16T14:17:40.847998+0000","last_unstale":"2023-10-16T14:17:41.824190+0000","last_undegraded":"2023-10-16T14:17:41.824190+0000","last_fullsized":"2023-10-16T14:17:41.824190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:23.202538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821501+0000","last_change":"2023-10-16T14:17:40.185804+0000","last_active":"2023-10-16T14:17:41.821501+0000","last_peered":"2023-10-16T14:17:41.821501+0000","last_clean":"2023-10-16T14:17:41.821501+0000","last_became_active":"2023-10-16T14:17:40.185533+0000","last_became_peered":"2023-10-16T14:17:40.185533+0000","last_unstale":"2023-10-16T14:17:41.821501+0000","last_undegraded":"2023-10-16T14:17:41.821501+0000","last_fullsized":"2023-10-16T14:17:41.821501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:27.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820563+0000","last_change":"2023-10-16T14:17:40.839911+0000","last_active":"2023-10-16T14:17:41.820563+0000","last_peered":"2023-10-16T14:17:41.820563+0000","last_clean":"2023-10-16T14:17:41.820563+0000","last_became_active":"2023-10-16T14:17:40.838848+0000","last_became_peered":"2023-10-16T14:17:40.838848+0000","last_unstale":"2023-10-16T14:17:41.820563+0000","last_undegraded":"2023-10-16T14:17:41.820563+0000","last_fullsized":"2023-10-16T14:17:41.820563+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:56.091494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847725+0000","last_change":"2023-10-16T14:17:40.173988+0000","last_active":"2023-10-16T14:17:41.847725+0000","last_peered":"2023-10-16T14:17:41.847725+0000","last_clean":"2023-10-16T14:17:41.847725+0000","last_became_active":"2023-10-16T14:17:40.173677+0000","last_became_peered":"2023-10-16T14:17:40.173677+0000","last_unstale":"2023-10-16T14:17:41.847725+0000","last_undegraded":"2023-10-16T14:17:41.847725+0000","last_fullsized":"2023-10-16T14:17:41.847725+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:48.045845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846012+0000","last_change":"2023-10-16T14:17:40.849561+0000","last_active":"2023-10-16T14:17:41.846012+0000","last_peered":"2023-10-16T14:17:41.846012+0000","last_clean":"2023-10-16T14:17:41.846012+0000","last_became_active":"2023-10-16T14:17:40.849234+0000","last_became_peered":"2023-10-16T14:17:40.849234+0000","last_unstale":"2023-10-16T14:17:41.846012+0000","last_undegraded":"2023-10-16T14:17:41.846012+0000","last_fullsized":"2023-10-16T14:17:41.846012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.900473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820810+0000","last_change":"2023-10-16T14:17:40.178142+0000","last_active":"2023-10-16T14:17:41.820810+0000","last_peered":"2023-10-16T14:17:41.820810+0000","last_clean":"2023-10-16T14:17:41.820810+0000","last_became_active":"2023-10-16T14:17:40.177075+0000","last_became_peered":"2023-10-16T14:17:40.177075+0000","last_unstale":"2023-10-16T14:17:41.820810+0000","last_undegraded":"2023-10-16T14:17:41.820810+0000","last_fullsized":"2023-10-16T14:17:41.820810+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:59.566646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825061+0000","last_change":"2023-10-16T14:17:40.844998+0000","last_active":"2023-10-16T14:17:41.825061+0000","last_peered":"2023-10-16T14:17:41.825061+0000","last_clean":"2023-10-16T14:17:41.825061+0000","last_became_active":"2023-10-16T14:17:40.844492+0000","last_became_peered":"2023-10-16T14:17:40.844492+0000","last_unstale":"2023-10-16T14:17:41.825061+0000","last_undegraded":"2023-10-16T14:17:41.825061+0000","last_fullsized":"2023-10-16T14:17:41.825061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:18.224519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848762+0000","last_change":"2023-10-16T14:17:40.176614+0000","last_active":"2023-10-16T14:17:41.848762+0000","last_peered":"2023-10-16T14:17:41.848762+0000","last_clean":"2023-10-16T14:17:41.848762+0000","last_became_active":"2023-10-16T14:17:40.176225+0000","last_became_peered":"2023-10-16T14:17:40.176225+0000","last_unstale":"2023-10-16T14:17:41.848762+0000","last_undegraded":"2023-10-16T14:17:41.848762+0000","last_fullsized":"2023-10-16T14:17:41.848762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:45.133909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848728+0000","last_change":"2023-10-16T14:17:40.852565+0000","last_active":"2023-10-16T14:17:41.848728+0000","last_peered":"2023-10-16T14:17:41.848728+0000","last_clean":"2023-10-16T14:17:41.848728+0000","last_became_active":"2023-10-16T14:17:40.852024+0000","last_became_peered":"2023-10-16T14:17:40.852024+0000","last_unstale":"2023-10-16T14:17:41.848728+0000","last_undegraded":"2023-10-16T14:17:41.848728+0000","last_fullsized":"2023-10-16T14:17:41.848728+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:57.803633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818629+0000","last_change":"2023-10-16T14:17:40.181478+0000","last_active":"2023-10-16T14:17:41.818629+0000","last_peered":"2023-10-16T14:17:41.818629+0000","last_clean":"2023-10-16T14:17:41.818629+0000","last_became_active":"2023-10-16T14:17:40.181181+0000","last_became_peered":"2023-10-16T14:17:40.181181+0000","last_unstale":"2023-10-16T14:17:41.818629+0000","last_undegraded":"2023-10-16T14:17:41.818629+0000","last_fullsized":"2023-10-16T14:17:41.818629+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:57.099074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848212+0000","last_change":"2023-10-16T14:17:40.841747+0000","last_active":"2023-10-16T14:17:41.848212+0000","last_peered":"2023-10-16T14:17:41.848212+0000","last_clean":"2023-10-16T14:17:41.848212+0000","last_became_active":"2023-10-16T14:17:40.839067+0000","last_became_peered":"2023-10-16T14:17:40.839067+0000","last_unstale":"2023-10-16T14:17:41.848212+0000","last_undegraded":"2023-10-16T14:17:41.848212+0000","last_fullsized":"2023-10-16T14:17:41.848212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:45.303224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825228+0000","last_change":"2023-10-16T14:17:40.184621+0000","last_active":"2023-10-16T14:17:41.825228+0000","last_peered":"2023-10-16T14:17:41.825228+0000","last_clean":"2023-10-16T14:17:41.825228+0000","last_became_active":"2023-10-16T14:17:40.184347+0000","last_became_peered":"2023-10-16T14:17:40.184347+0000","last_unstale":"2023-10-16T14:17:41.825228+0000","last_undegraded":"2023-10-16T14:17:41.825228+0000","last_fullsized":"2023-10-16T14:17:41.825228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:29.104833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.824715+0000","last_change":"2023-10-16T14:17:40.853283+0000","last_active":"2023-10-16T14:17:41.824715+0000","last_peered":"2023-10-16T14:17:41.824715+0000","last_clean":"2023-10-16T14:17:41.824715+0000","last_became_active":"2023-10-16T14:17:40.852659+0000","last_became_peered":"2023-10-16T14:17:40.852659+0000","last_unstale":"2023-10-16T14:17:41.824715+0000","last_undegraded":"2023-10-16T14:17:41.824715+0000","last_fullsized":"2023-10-16T14:17:41.824715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:24.359426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848396+0000","last_change":"2023-10-16T14:17:40.180701+0000","last_active":"2023-10-16T14:17:41.848396+0000","last_peered":"2023-10-16T14:17:41.848396+0000","last_clean":"2023-10-16T14:17:41.848396+0000","last_became_active":"2023-10-16T14:17:40.180404+0000","last_became_peered":"2023-10-16T14:17:40.180404+0000","last_unstale":"2023-10-16T14:17:41.848396+0000","last_undegraded":"2023-10-16T14:17:41.848396+0000","last_fullsized":"2023-10-16T14:17:41.848396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:37.248241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847165+0000","last_change":"2023-10-16T14:17:40.849603+0000","last_active":"2023-10-16T14:17:41.847165+0000","last_peered":"2023-10-16T14:17:41.847165+0000","last_clean":"2023-10-16T14:17:41.847165+0000","last_became_active":"2023-10-16T14:17:40.849301+0000","last_became_peered":"2023-10-16T14:17:40.849301+0000","last_unstale":"2023-10-16T14:17:41.847165+0000","last_undegraded":"2023-10-16T14:17:41.847165+0000","last_fullsized":"2023-10-16T14:17:41.847165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:02.808237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822114+0000","last_change":"2023-10-16T14:17:40.177211+0000","last_active":"2023-10-16T14:17:41.822114+0000","last_peered":"2023-10-16T14:17:41.822114+0000","last_clean":"2023-10-16T14:17:41.822114+0000","last_became_active":"2023-10-16T14:17:40.176900+0000","last_became_peered":"2023-10-16T14:17:40.176900+0000","last_unstale":"2023-10-16T14:17:41.822114+0000","last_undegraded":"2023-10-16T14:17:41.822114+0000","last_fullsized":"2023-10-16T14:17:41.822114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:48.045845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848087+0000","last_change":"2023-10-16T14:17:40.844990+0000","last_active":"2023-10-16T14:17:41.848087+0000","last_peered":"2023-10-16T14:17:41.848087+0000","last_clean":"2023-10-16T14:17:41.848087+0000","last_became_active":"2023-10-16T14:17:40.844664+0000","last_became_peered":"2023-10-16T14:17:40.844664+0000","last_unstale":"2023-10-16T14:17:41.848087+0000","last_undegraded":"2023-10-16T14:17:41.848087+0000","last_fullsized":"2023-10-16T14:17:41.848087+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:18.224519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846122+0000","last_change":"2023-10-16T14:17:40.181839+0000","last_active":"2023-10-16T14:17:41.846122+0000","last_peered":"2023-10-16T14:17:41.846122+0000","last_clean":"2023-10-16T14:17:41.846122+0000","last_became_active":"2023-10-16T14:17:40.181558+0000","last_became_peered":"2023-10-16T14:17:40.181558+0000","last_unstale":"2023-10-16T14:17:41.846122+0000","last_undegraded":"2023-10-16T14:17:41.846122+0000","last_fullsized":"2023-10-16T14:17:41.846122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:27.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821617+0000","last_change":"2023-10-16T14:17:40.836881+0000","last_active":"2023-10-16T14:17:41.821617+0000","last_peered":"2023-10-16T14:17:41.821617+0000","last_clean":"2023-10-16T14:17:41.821617+0000","last_became_active":"2023-10-16T14:17:40.836084+0000","last_became_peered":"2023-10-16T14:17:40.836084+0000","last_unstale":"2023-10-16T14:17:41.821617+0000","last_undegraded":"2023-10-16T14:17:41.821617+0000","last_fullsized":"2023-10-16T14:17:41.821617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:55.629805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847694+0000","last_change":"2023-10-16T14:17:40.175606+0000","last_active":"2023-10-16T14:17:41.847694+0000","last_peered":"2023-10-16T14:17:41.847694+0000","last_clean":"2023-10-16T14:17:41.847694+0000","last_became_active":"2023-10-16T14:17:40.175275+0000","last_became_peered":"2023-10-16T14:17:40.175275+0000","last_unstale":"2023-10-16T14:17:41.847694+0000","last_undegraded":"2023-10-16T14:17:41.847694+0000","last_fullsized":"2023-10-16T14:17:41.847694+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:23.016735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846144+0000","last_change":"2023-10-16T14:17:40.849781+0000","last_active":"2023-10-16T14:17:41.846144+0000","last_peered":"2023-10-16T14:17:41.846144+0000","last_clean":"2023-10-16T14:17:41.846144+0000","last_became_active":"2023-10-16T14:17:40.849453+0000","last_became_peered":"2023-10-16T14:17:40.849453+0000","last_unstale":"2023-10-16T14:17:41.846144+0000","last_undegraded":"2023-10-16T14:17:41.846144+0000","last_fullsized":"2023-10-16T14:17:41.846144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:08.713068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848284+0000","last_change":"2023-10-16T14:17:40.176322+0000","last_active":"2023-10-16T14:17:41.848284+0000","last_peered":"2023-10-16T14:17:41.848284+0000","last_clean":"2023-10-16T14:17:41.848284+0000","last_became_active":"2023-10-16T14:17:40.175613+0000","last_became_peered":"2023-10-16T14:17:40.175613+0000","last_unstale":"2023-10-16T14:17:41.848284+0000","last_undegraded":"2023-10-16T14:17:41.848284+0000","last_fullsized":"2023-10-16T14:17:41.848284+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:12.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820681+0000","last_change":"2023-10-16T14:17:40.844761+0000","last_active":"2023-10-16T14:17:41.820681+0000","last_peered":"2023-10-16T14:17:41.820681+0000","last_clean":"2023-10-16T14:17:41.820681+0000","last_became_active":"2023-10-16T14:17:40.844495+0000","last_became_peered":"2023-10-16T14:17:40.844495+0000","last_unstale":"2023-10-16T14:17:41.820681+0000","last_undegraded":"2023-10-16T14:17:41.820681+0000","last_fullsized":"2023-10-16T14:17:41.820681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.900473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847014+0000","last_change":"2023-10-16T14:17:40.185147+0000","last_active":"2023-10-16T14:17:41.847014+0000","last_peered":"2023-10-16T14:17:41.847014+0000","last_clean":"2023-10-16T14:17:41.847014+0000","last_became_active":"2023-10-16T14:17:40.184823+0000","last_became_peered":"2023-10-16T14:17:40.184823+0000","last_unstale":"2023-10-16T14:17:41.847014+0000","last_undegraded":"2023-10-16T14:17:41.847014+0000","last_fullsized":"2023-10-16T14:17:41.847014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:23.016735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825106+0000","last_change":"2023-10-16T14:17:40.845511+0000","last_active":"2023-10-16T14:17:41.825106+0000","last_peered":"2023-10-16T14:17:41.825106+0000","last_clean":"2023-10-16T14:17:41.825106+0000","last_became_active":"2023-10-16T14:17:40.844473+0000","last_became_peered":"2023-10-16T14:17:40.844473+0000","last_unstale":"2023-10-16T14:17:41.825106+0000","last_undegraded":"2023-10-16T14:17:41.825106+0000","last_fullsized":"2023-10-16T14:17:41.825106+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:45.303224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823572+0000","last_change":"2023-10-16T14:17:40.181484+0000","last_active":"2023-10-16T14:17:41.823572+0000","last_peered":"2023-10-16T14:17:41.823572+0000","last_clean":"2023-10-16T14:17:41.823572+0000","last_became_active":"2023-10-16T14:17:40.181196+0000","last_became_peered":"2023-10-16T14:17:40.181196+0000","last_unstale":"2023-10-16T14:17:41.823572+0000","last_undegraded":"2023-10-16T14:17:41.823572+0000","last_fullsized":"2023-10-16T14:17:41.823572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:23.016735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846421+0000","last_change":"2023-10-16T14:17:40.853402+0000","last_active":"2023-10-16T14:17:41.846421+0000","last_peered":"2023-10-16T14:17:41.846421+0000","last_clean":"2023-10-16T14:17:41.846421+0000","last_became_active":"2023-10-16T14:17:40.852595+0000","last_became_peered":"2023-10-16T14:17:40.852595+0000","last_unstale":"2023-10-16T14:17:41.846421+0000","last_undegraded":"2023-10-16T14:17:41.846421+0000","last_fullsized":"2023-10-16T14:17:41.846421+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.991974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822519+0000","last_change":"2023-10-16T14:17:40.173956+0000","last_active":"2023-10-16T14:17:41.822519+0000","last_peered":"2023-10-16T14:17:41.822519+0000","last_clean":"2023-10-16T14:17:41.822519+0000","last_became_active":"2023-10-16T14:17:40.173562+0000","last_became_peered":"2023-10-16T14:17:40.173562+0000","last_unstale":"2023-10-16T14:17:41.822519+0000","last_undegraded":"2023-10-16T14:17:41.822519+0000","last_fullsized":"2023-10-16T14:17:41.822519+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:12.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844819+0000","last_change":"2023-10-16T14:17:40.847347+0000","last_active":"2023-10-16T14:17:41.844819+0000","last_peered":"2023-10-16T14:17:41.844819+0000","last_clean":"2023-10-16T14:17:41.844819+0000","last_became_active":"2023-10-16T14:17:40.847084+0000","last_became_peered":"2023-10-16T14:17:40.847084+0000","last_unstale":"2023-10-16T14:17:41.844819+0000","last_undegraded":"2023-10-16T14:17:41.844819+0000","last_fullsized":"2023-10-16T14:17:41.844819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:00.571880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821185+0000","last_change":"2023-10-16T14:17:40.182230+0000","last_active":"2023-10-16T14:17:41.821185+0000","last_peered":"2023-10-16T14:17:41.821185+0000","last_clean":"2023-10-16T14:17:41.821185+0000","last_became_active":"2023-10-16T14:17:40.181964+0000","last_became_peered":"2023-10-16T14:17:40.181964+0000","last_unstale":"2023-10-16T14:17:41.821185+0000","last_undegraded":"2023-10-16T14:17:41.821185+0000","last_fullsized":"2023-10-16T14:17:41.821185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:07.707835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.845153+0000","last_change":"2023-10-16T14:17:40.848588+0000","last_active":"2023-10-16T14:17:41.845153+0000","last_peered":"2023-10-16T14:17:41.845153+0000","last_clean":"2023-10-16T14:17:41.845153+0000","last_became_active":"2023-10-16T14:17:40.848272+0000","last_became_peered":"2023-10-16T14:17:40.848272+0000","last_unstale":"2023-10-16T14:17:41.845153+0000","last_undegraded":"2023-10-16T14:17:41.845153+0000","last_fullsized":"2023-10-16T14:17:41.845153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:06.510813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.817810+0000","last_change":"2023-10-16T14:17:40.170369+0000","last_active":"2023-10-16T14:17:41.817810+0000","last_peered":"2023-10-16T14:17:41.817810+0000","last_clean":"2023-10-16T14:17:41.817810+0000","last_became_active":"2023-10-16T14:17:40.170066+0000","last_became_peered":"2023-10-16T14:17:40.170066+0000","last_unstale":"2023-10-16T14:17:41.817810+0000","last_undegraded":"2023-10-16T14:17:41.817810+0000","last_fullsized":"2023-10-16T14:17:41.817810+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:17.999625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844820+0000","last_change":"2023-10-16T14:17:40.846195+0000","last_active":"2023-10-16T14:17:41.844820+0000","last_peered":"2023-10-16T14:17:41.844820+0000","last_clean":"2023-10-16T14:17:41.844820+0000","last_became_active":"2023-10-16T14:17:40.845901+0000","last_became_peered":"2023-10-16T14:17:40.845901+0000","last_unstale":"2023-10-16T14:17:41.844820+0000","last_undegraded":"2023-10-16T14:17:41.844820+0000","last_fullsized":"2023-10-16T14:17:41.844820+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:51.395464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846377+0000","last_change":"2023-10-16T14:17:40.185602+0000","last_active":"2023-10-16T14:17:41.846377+0000","last_peered":"2023-10-16T14:17:41.846377+0000","last_clean":"2023-10-16T14:17:41.846377+0000","last_became_active":"2023-10-16T14:17:40.185322+0000","last_became_peered":"2023-10-16T14:17:40.185322+0000","last_unstale":"2023-10-16T14:17:41.846377+0000","last_undegraded":"2023-10-16T14:17:41.846377+0000","last_fullsized":"2023-10-16T14:17:41.846377+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:17.999625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844547+0000","last_change":"2023-10-16T14:17:40.842568+0000","last_active":"2023-10-16T14:17:41.844547+0000","last_peered":"2023-10-16T14:17:41.844547+0000","last_clean":"2023-10-16T14:17:41.844547+0000","last_became_active":"2023-10-16T14:17:40.839552+0000","last_became_peered":"2023-10-16T14:17:40.839552+0000","last_unstale":"2023-10-16T14:17:41.844547+0000","last_undegraded":"2023-10-16T14:17:41.844547+0000","last_fullsized":"2023-10-16T14:17:41.844547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:01.083679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820793+0000","last_change":"2023-10-16T14:17:40.178755+0000","last_active":"2023-10-16T14:17:41.820793+0000","last_peered":"2023-10-16T14:17:41.820793+0000","last_clean":"2023-10-16T14:17:41.820793+0000","last_became_active":"2023-10-16T14:17:40.178480+0000","last_became_peered":"2023-10-16T14:17:40.178480+0000","last_unstale":"2023-10-16T14:17:41.820793+0000","last_undegraded":"2023-10-16T14:17:41.820793+0000","last_fullsized":"2023-10-16T14:17:41.820793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:05.505579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821705+0000","last_change":"2023-10-16T14:17:40.844412+0000","last_active":"2023-10-16T14:17:41.821705+0000","last_peered":"2023-10-16T14:17:41.821705+0000","last_clean":"2023-10-16T14:17:41.821705+0000","last_became_active":"2023-10-16T14:17:40.844096+0000","last_became_peered":"2023-10-16T14:17:40.844096+0000","last_unstale":"2023-10-16T14:17:41.821705+0000","last_undegraded":"2023-10-16T14:17:41.821705+0000","last_fullsized":"2023-10-16T14:17:41.821705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:24.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.823415+0000","last_change":"2023-10-16T14:17:40.183863+0000","last_active":"2023-10-16T14:17:41.823415+0000","last_peered":"2023-10-16T14:17:41.823415+0000","last_clean":"2023-10-16T14:17:41.823415+0000","last_became_active":"2023-10-16T14:17:40.183562+0000","last_became_peered":"2023-10-16T14:17:40.183562+0000","last_unstale":"2023-10-16T14:17:41.823415+0000","last_undegraded":"2023-10-16T14:17:41.823415+0000","last_fullsized":"2023-10-16T14:17:41.823415+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:23.239599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.844919+0000","last_change":"2023-10-16T14:17:40.854181+0000","last_active":"2023-10-16T14:17:41.844919+0000","last_peered":"2023-10-16T14:17:41.844919+0000","last_clean":"2023-10-16T14:17:41.844919+0000","last_became_active":"2023-10-16T14:17:40.853150+0000","last_became_peered":"2023-10-16T14:17:40.853150+0000","last_unstale":"2023-10-16T14:17:41.844919+0000","last_undegraded":"2023-10-16T14:17:41.844919+0000","last_fullsized":"2023-10-16T14:17:41.844919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:19.004858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.847125+0000","last_change":"2023-10-16T14:17:40.185045+0000","last_active":"2023-10-16T14:17:41.847125+0000","last_peered":"2023-10-16T14:17:41.847125+0000","last_clean":"2023-10-16T14:17:41.847125+0000","last_became_active":"2023-10-16T14:17:40.184309+0000","last_became_peered":"2023-10-16T14:17:40.184309+0000","last_unstale":"2023-10-16T14:17:41.847125+0000","last_undegraded":"2023-10-16T14:17:41.847125+0000","last_fullsized":"2023-10-16T14:17:41.847125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:12.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.825221+0000","last_change":"2023-10-16T14:17:40.857321+0000","last_active":"2023-10-16T14:17:41.825221+0000","last_peered":"2023-10-16T14:17:41.825221+0000","last_clean":"2023-10-16T14:17:41.825221+0000","last_became_active":"2023-10-16T14:17:40.857024+0000","last_became_peered":"2023-10-16T14:17:40.857024+0000","last_unstale":"2023-10-16T14:17:41.825221+0000","last_undegraded":"2023-10-16T14:17:41.825221+0000","last_fullsized":"2023-10-16T14:17:41.825221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:47.932423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:42.376898+0000","last_change":"2023-10-16T14:17:40.178661+0000","last_active":"2023-10-16T14:17:42.376898+0000","last_peered":"2023-10-16T14:17:42.376898+0000","last_clean":"2023-10-16T14:17:42.376898+0000","last_became_active":"2023-10-16T14:17:40.178347+0000","last_became_peered":"2023-10-16T14:17:40.178347+0000","last_unstale":"2023-10-16T14:17:42.376898+0000","last_undegraded":"2023-10-16T14:17:42.376898+0000","last_fullsized":"2023-10-16T14:17:42.376898+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.397925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818229+0000","last_change":"2023-10-16T14:17:40.847648+0000","last_active":"2023-10-16T14:17:41.818229+0000","last_peered":"2023-10-16T14:17:41.818229+0000","last_clean":"2023-10-16T14:17:41.818229+0000","last_became_active":"2023-10-16T14:17:40.845959+0000","last_became_peered":"2023-10-16T14:17:40.845959+0000","last_unstale":"2023-10-16T14:17:41.818229+0000","last_undegraded":"2023-10-16T14:17:41.818229+0000","last_fullsized":"2023-10-16T14:17:41.818229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:42:13.566571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846412+0000","last_change":"2023-10-16T14:17:40.175762+0000","last_active":"2023-10-16T14:17:41.846412+0000","last_peered":"2023-10-16T14:17:41.846412+0000","last_clean":"2023-10-16T14:17:41.846412+0000","last_became_active":"2023-10-16T14:17:40.174027+0000","last_became_peered":"2023-10-16T14:17:40.174027+0000","last_unstale":"2023-10-16T14:17:41.846412+0000","last_undegraded":"2023-10-16T14:17:41.846412+0000","last_fullsized":"2023-10-16T14:17:41.846412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:48.045845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846205+0000","last_change":"2023-10-16T14:17:40.850057+0000","last_active":"2023-10-16T14:17:41.846205+0000","last_peered":"2023-10-16T14:17:41.846205+0000","last_clean":"2023-10-16T14:17:41.846205+0000","last_became_active":"2023-10-16T14:17:40.849121+0000","last_became_peered":"2023-10-16T14:17:40.849121+0000","last_unstale":"2023-10-16T14:17:41.846205+0000","last_undegraded":"2023-10-16T14:17:41.846205+0000","last_fullsized":"2023-10-16T14:17:41.846205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:58.104307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.818023+0000","last_change":"2023-10-16T14:17:40.178916+0000","last_active":"2023-10-16T14:17:41.818023+0000","last_peered":"2023-10-16T14:17:41.818023+0000","last_clean":"2023-10-16T14:17:41.818023+0000","last_became_active":"2023-10-16T14:17:40.178615+0000","last_became_peered":"2023-10-16T14:17:40.178615+0000","last_unstale":"2023-10-16T14:17:41.818023+0000","last_undegraded":"2023-10-16T14:17:41.818023+0000","last_fullsized":"2023-10-16T14:17:41.818023+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:05.505579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.817452+0000","last_change":"2023-10-16T14:17:40.844066+0000","last_active":"2023-10-16T14:17:41.817452+0000","last_peered":"2023-10-16T14:17:41.817452+0000","last_clean":"2023-10-16T14:17:41.817452+0000","last_became_active":"2023-10-16T14:17:40.842493+0000","last_became_peered":"2023-10-16T14:17:40.842493+0000","last_unstale":"2023-10-16T14:17:41.817452+0000","last_undegraded":"2023-10-16T14:17:41.817452+0000","last_fullsized":"2023-10-16T14:17:41.817452+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:24.021968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846360+0000","last_change":"2023-10-16T14:17:40.185039+0000","last_active":"2023-10-16T14:17:41.846360+0000","last_peered":"2023-10-16T14:17:41.846360+0000","last_clean":"2023-10-16T14:17:41.846360+0000","last_became_active":"2023-10-16T14:17:40.184716+0000","last_became_peered":"2023-10-16T14:17:40.184716+0000","last_unstale":"2023-10-16T14:17:41.846360+0000","last_undegraded":"2023-10-16T14:17:41.846360+0000","last_fullsized":"2023-10-16T14:17:41.846360+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:55.086260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848089+0000","last_change":"2023-10-16T14:17:40.844811+0000","last_active":"2023-10-16T14:17:41.848089+0000","last_peered":"2023-10-16T14:17:41.848089+0000","last_clean":"2023-10-16T14:17:41.848089+0000","last_became_active":"2023-10-16T14:17:40.844360+0000","last_became_peered":"2023-10-16T14:17:40.844360+0000","last_unstale":"2023-10-16T14:17:41.848089+0000","last_undegraded":"2023-10-16T14:17:41.848089+0000","last_fullsized":"2023-10-16T14:17:41.848089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:24.359426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.821014+0000","last_change":"2023-10-16T14:17:40.181263+0000","last_active":"2023-10-16T14:17:41.821014+0000","last_peered":"2023-10-16T14:17:41.821014+0000","last_clean":"2023-10-16T14:17:41.821014+0000","last_became_active":"2023-10-16T14:17:40.180873+0000","last_became_peered":"2023-10-16T14:17:40.180873+0000","last_unstale":"2023-10-16T14:17:41.821014+0000","last_undegraded":"2023-10-16T14:17:41.821014+0000","last_fullsized":"2023-10-16T14:17:41.821014+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:50.461112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.820927+0000","last_change":"2023-10-16T14:17:40.842963+0000","last_active":"2023-10-16T14:17:41.820927+0000","last_peered":"2023-10-16T14:17:41.820927+0000","last_clean":"2023-10-16T14:17:41.820927+0000","last_became_active":"2023-10-16T14:17:40.842665+0000","last_became_peered":"2023-10-16T14:17:40.842665+0000","last_unstale":"2023-10-16T14:17:41.820927+0000","last_undegraded":"2023-10-16T14:17:41.820927+0000","last_fullsized":"2023-10-16T14:17:41.820927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:25.342080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.848022+0000","last_change":"2023-10-16T14:17:40.175340+0000","last_active":"2023-10-16T14:17:41.848022+0000","last_peered":"2023-10-16T14:17:41.848022+0000","last_clean":"2023-10-16T14:17:41.848022+0000","last_became_active":"2023-10-16T14:17:40.175079+0000","last_became_peered":"2023-10-16T14:17:40.175079+0000","last_unstale":"2023-10-16T14:17:41.848022+0000","last_undegraded":"2023-10-16T14:17:41.848022+0000","last_fullsized":"2023-10-16T14:17:41.848022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.397925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.822793+0000","last_change":"2023-10-16T14:17:40.847603+0000","last_active":"2023-10-16T14:17:41.822793+0000","last_peered":"2023-10-16T14:17:41.822793+0000","last_clean":"2023-10-16T14:17:41.822793+0000","last_became_active":"2023-10-16T14:17:40.847183+0000","last_became_peered":"2023-10-16T14:17:40.847183+0000","last_unstale":"2023-10-16T14:17:41.822793+0000","last_undegraded":"2023-10-16T14:17:41.822793+0000","last_fullsized":"2023-10-16T14:17:41.822793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:40.139861+0000","last_clean_scrub_stamp":"2023-10-16T14:17:40.139861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:23.202538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-16T14:17:41.846204+0000","last_change":"2023-10-16T14:17:40.184287+0000","last_active":"2023-10-16T14:17:41.846204+0000","last_peered":"2023-10-16T14:17:41.846204+0000","last_clean":"2023-10-16T14:17:41.846204+0000","last_became_active":"2023-10-16T14:17:40.183992+0000","last_became_peered":"2023-10-16T14:17:40.183992+0000","last_unstale":"2023-10-16T14:17:41.846204+0000","last_undegraded":"2023-10-16T14:17:41.846204+0000","last_fullsized":"2023-10-16T14:17:41.846204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:17:39.134628+0000","last_clean_scrub_stamp":"2023-10-16T14:17:39.134628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:00.078445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-16T14:17:40.820960+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:17:40.820960+0000","last_peered":"2023-10-16T14:17:40.820960+0000","last_clean":"2023-10-16T14:17:40.820960+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:17:40.820960+0000","last_undegraded":"2023-10-16T14:17:40.820960+0000","last_fullsized":"2023-10-16T14:17:40.820960+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":970752,"data_stored":737892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.81499999999999995}]},{"osd":1,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.018}]},{"osd":3,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.153},{"interface":"front","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.304},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.343}]},{"osd":5,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.256}]},{"osd":6,"last update":"Mon Oct 16 14:17:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.423},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.4970000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":148050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":2.2160000000000002},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":3.863}]},{"osd":1,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":3.7879999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":3.5800000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7250000000000001,"5min":2.7250000000000001,"15min":2.7250000000000001},"last":2.0179999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":4.9790000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":5.4160000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":5},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":5.4640000000000004}]},{"osd":5,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":3.9279999999999999},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.8330000000000002,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":3.6749999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.663,"15min":1.663},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":4.6859999999999999},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":4.883}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":380928,"data_stored":148106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.3540000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.876},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.087}]},{"osd":2,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.2},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.8680000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.25,"15min":1.25},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.419},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":1.8169999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.863,"5min":2.863,"15min":2.863},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.9059999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.762},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":3.794,"5min":3.794,"15min":3.794},"last":1.1379999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.615,"5min":1.615,"15min":1.615},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.9990000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":147538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"last":1.2749999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":1.675},{"interface":"front","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":5.4640000000000004,"5min":5.4640000000000004,"15min":5.4640000000000004},"last":1.6479999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":5.617,"5min":5.617,"15min":5.617},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.702}]},{"osd":3,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":7.4089999999999998,"5min":7.4089999999999998,"15min":7.4089999999999998},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.536}]},{"osd":5,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":5.4109999999999996,"5min":5.4109999999999996,"15min":5.4109999999999996},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":9.8239999999999998,"5min":9.8239999999999998,"15min":9.8239999999999998},"last":1.587}]},{"osd":6,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":5.782,"5min":5.782,"15min":5.782},"last":1.5680000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.647,"5min":1.647,"15min":1.647},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":3.7799999999999998,"5min":3.7799999999999998,"15min":3.7799999999999998},"last":1.6180000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":389120,"data_stored":149068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.4420000000000002,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":1.002},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.288}]},{"osd":2,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.23},{"interface":"front","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.3440000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":1.073},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.319}]},{"osd":5,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.431},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.4910000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":1.268},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.3799999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:17:20 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.667,"15min":1.667},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.161}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":376832,"data_stored":147504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.085,"5min":1.085,"15min":1.085},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.88100000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":3.25,"5min":3.25,"15min":3.25},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.9350000000000001,"5min":2.9350000000000001,"15min":2.9350000000000001},"last":1.484}]},{"osd":6,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":3.536,"5min":3.536,"15min":3.536},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.085}]},{"osd":7,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":5.2199999999999998,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.181}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":389120,"data_stored":149068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.61099999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.9820000000000002,"5min":2.9820000000000002,"15min":2.9820000000000002},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.242,"5min":3.242,"15min":3.242},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.879},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":1.107,"5min":1.107,"15min":1.107},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.73099999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:17:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":1.204,"5min":1.204,"15min":1.204},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.492,"5min":1.492,"15min":1.492},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.3410000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.077}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":966656,"data_stored":737802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.3320000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.165,"15min":1.165},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.653,"5min":3.653,"15min":3.653},"last":1.9379999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":3.1400000000000001,"5min":3.1400000000000001,"15min":3.1400000000000001},"last":1.718}]},{"osd":3,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.669},{"interface":"front","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.7649999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":1.534},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":3.6190000000000002,"5min":3.6190000000000002,"15min":3.6190000000000002},"last":2.1699999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":3.2370000000000001,"5min":3.2370000000000001,"15min":3.2370000000000001},"last":1.595},{"interface":"front","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.819,"5min":2.819,"15min":2.819},"last":1.1839999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":2.0539999999999998},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":3.0979999999999999,"5min":3.0979999999999999,"15min":3.0979999999999999},"last":1.26}]},{"osd":7,"last update":"Mon Oct 16 14:17:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.829},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"max":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"last":1.8740000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:17:46.236 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:17:46.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:17:46.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.512+0000 7ff969a8a640 1 Processor -- start 2023-10-16T14:17:46.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.513+0000 7ff969a8a640 1 -- start start 2023-10-16T14:17:46.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.514+0000 7ff969a8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff96415dc90 0x7ff9640c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:46.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.514+0000 7ff969a8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff964071300 0x7ff9640ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:46.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.514+0000 7ff969a8a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff964070760 con 0x7ff9640c9230 2023-10-16T14:17:46.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.514+0000 7ff969a8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9640cdac0 con 0x7ff96415dc90 2023-10-16T14:17:46.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.514+0000 7ff969a8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff964070110 con 0x7ff964071300 2023-10-16T14:17:46.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.515+0000 7ff963fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff9640c9230 0x7ff964070bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54630/0 (socket says 172.21.15.29:54630) 2023-10-16T14:17:46.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.515+0000 7ff963fff640 1 -- 172.21.15.29:0/248500882 learned_addr learned my addr 172.21.15.29:0/248500882 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:46.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.515+0000 7ff968a88640 1 --2- 172.21.15.29:0/248500882 >> v2:172.21.15.170:3300/0 conn(0x7ff96415dc90 0x7ff9640c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:46.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.515+0000 7ff969289640 1 --2- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff964071300 0x7ff9640ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:46.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.515+0000 7ff9637fe640 1 -- 172.21.15.29:0/248500882 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50741392 0 0) 0x7ff964070760 con 0x7ff9640c9230 2023-10-16T14:17:46.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.516+0000 7ff9637fe640 1 -- 172.21.15.29:0/248500882 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff954003520 con 0x7ff9640c9230 2023-10-16T14:17:46.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.516+0000 7ff968a88640 1 -- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff964071300 msgr2=0x7ff9640ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:46.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.516+0000 7ff968a88640 1 --2- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff964071300 0x7ff9640ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.516+0000 7ff968a88640 1 -- 172.21.15.29:0/248500882 >> v1:172.21.15.29:6789/0 conn(0x7ff9640c9230 legacy=0x7ff964070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.516+0000 7ff9637fe640 1 -- 172.21.15.29:0/248500882 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285966124 0 0) 0x7ff954003520 con 0x7ff9640c9230 2023-10-16T14:17:46.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.516+0000 7ff968a88640 1 -- 172.21.15.29:0/248500882 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640703f0 con 0x7ff96415dc90 2023-10-16T14:17:46.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.517+0000 7ff968a88640 1 --2- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff96415dc90 0x7ff9640c8cf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff9580097a0 tx=0x7ff95802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4338fa9096b1aad2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:46.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.518+0000 7ff9637fe640 1 -- 172.21.15.29:0/248500882 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950001670 con 0x7ff96415dc90 2023-10-16T14:17:46.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.518+0000 7ff9637fe640 1 -- 172.21.15.29:0/248500882 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95802e050 con 0x7ff96415dc90 2023-10-16T14:17:46.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.518+0000 7ff9637fe640 1 -- 172.21.15.29:0/248500882 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958030730 con 0x7ff96415dc90 2023-10-16T14:17:46.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.518+0000 7ff969a8a640 1 -- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff96415dc90 msgr2=0x7ff9640c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.518+0000 7ff969a8a640 1 --2- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff96415dc90 0x7ff9640c8cf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff9580097a0 tx=0x7ff95802b790 comp rx=0 tx=0).stop 2023-10-16T14:17:46.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.519+0000 7ff969a8a640 1 -- 172.21.15.29:0/248500882 shutdown_connections 2023-10-16T14:17:46.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.519+0000 7ff969a8a640 1 --2- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff964071300 0x7ff9640ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.519+0000 7ff969a8a640 1 --2- 172.21.15.29:0/248500882 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff96415dc90 0x7ff9640c8cf0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.519+0000 7ff969a8a640 1 -- 172.21.15.29:0/248500882 >> 172.21.15.29:0/248500882 conn(0x7ff9640ce6a0 msgr2=0x7ff9640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:46.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.519+0000 7ff969a8a640 1 -- 172.21.15.29:0/248500882 shutdown_connections 2023-10-16T14:17:46.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.519+0000 7ff969a8a640 1 -- 172.21.15.29:0/248500882 wait complete. 2023-10-16T14:17:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.521+0000 7ff969a8a640 1 Processor -- start 2023-10-16T14:17:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.522+0000 7ff969a8a640 1 -- start start 2023-10-16T14:17:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff969a8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 0x7ff9641557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff969a8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9640c9230 0x7ff9640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff969a8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff96415dc90 0x7ff9640c13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff969a8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9640c5880 con 0x7ff964071300 2023-10-16T14:17:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff969a8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9640b2090 con 0x7ff9640c9230 2023-10-16T14:17:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff969a8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9640b2370 con 0x7ff96415dc90 2023-10-16T14:17:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.523+0000 7ff968a88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 0x7ff9641557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 0x7ff9641557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37294/0 (socket says 172.21.15.29:37294) 2023-10-16T14:17:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 -- 172.21.15.29:0/1012867898 learned_addr learned my addr 172.21.15.29:0/1012867898 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff963fff640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9640c9230 0x7ff9640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff969289640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff96415dc90 0x7ff9640c13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 -- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff96415dc90 msgr2=0x7ff9640c13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff96415dc90 0x7ff9640c13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 -- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9640c9230 msgr2=0x7ff9640b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9640c9230 0x7ff9640b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.524+0000 7ff968a88640 1 -- 172.21.15.29:0/1012867898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff958009400 con 0x7ff964071300 2023-10-16T14:17:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.525+0000 7ff968a88640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 0x7ff9641557d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff958000c00 tx=0x7ff9580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.525+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95803c880 con 0x7ff964071300 2023-10-16T14:17:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.525+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95803c9e0 con 0x7ff964071300 2023-10-16T14:17:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.525+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95803cd90 con 0x7ff964071300 2023-10-16T14:17:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.525+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9640b2650 con 0x7ff964071300 2023-10-16T14:17:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.525+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9640b2c10 con 0x7ff964071300 2023-10-16T14:17:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.526+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96404e9b0 con 0x7ff964071300 2023-10-16T14:17:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.527+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff958039040 con 0x7ff964071300 2023-10-16T14:17:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.528+0000 7ff9617fa640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff94806aea0 0x7ff94806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.529+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9580ab120 con 0x7ff964071300 2023-10-16T14:17:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.529+0000 7ff963fff640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff94806aea0 0x7ff94806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:46.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.530+0000 7ff963fff640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff94806aea0 0x7ff94806d360 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff9640c8650 tx=0x7ff950001a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:46.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.532+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff958076300 con 0x7ff964071300 2023-10-16T14:17:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.794+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff9640b4620 con 0x7ff964071300 2023-10-16T14:17:46.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.798+0000 7ff9617fa640 1 -- 172.21.15.29:0/1012867898 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff9580370b0 con 0x7ff964071300 2023-10-16T14:17:46.803 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:17:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:17:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:17:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:17:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:17:46.804 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:17:46.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.800+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff94806aea0 msgr2=0x7ff94806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.800+0000 7ff969a8a640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff94806aea0 0x7ff94806d360 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff9640c8650 tx=0x7ff950001a50 comp rx=0 tx=0).stop 2023-10-16T14:17:46.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.800+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 msgr2=0x7ff9641557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:46.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 0x7ff9641557d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff958000c00 tx=0x7ff9580046c0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 shutdown_connections 2023-10-16T14:17:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff94806aea0 0x7ff94806d360 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff96415dc90 0x7ff9640c13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9640c9230 0x7ff9640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 --2- 172.21.15.29:0/1012867898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff964071300 0x7ff9641557d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:46.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 >> 172.21.15.29:0/1012867898 conn(0x7ff9640ce6a0 msgr2=0x7ff964070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:46.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 shutdown_connections 2023-10-16T14:17:46.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:46.801+0000 7ff969a8a640 1 -- 172.21.15.29:0/1012867898 wait complete. 2023-10-16T14:17:46.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:17:47.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.041+0000 7fd066780640 1 Processor -- start 2023-10-16T14:17:47.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.041+0000 7fd066780640 1 -- start start 2023-10-16T14:17:47.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.043+0000 7fd066780640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0600ad970 0x7fd0600cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.043+0000 7fd066780640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06015b9d0 con 0x7fd0600ccfb0 2023-10-16T14:17:47.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.043+0000 7fd066780640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06015bcb0 con 0x7fd06006c040 2023-10-16T14:17:47.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.043+0000 7fd066780640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06015bf90 con 0x7fd0600ad970 2023-10-16T14:17:47.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.043+0000 7fd065f7f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd0600ccfb0 0x7fd0600cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54636/0 (socket says 172.21.15.29:54636) 2023-10-16T14:17:47.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.043+0000 7fd065f7f640 1 -- 172.21.15.29:0/3875268595 learned_addr learned my addr 172.21.15.29:0/3875268595 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:47.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd064f7d640 1 --2- 172.21.15.29:0/3875268595 >> v2:172.21.15.170:3300/0 conn(0x7fd0600ad970 0x7fd0600cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295112260 0 0) 0x7fd06015b9d0 con 0x7fd0600ccfb0 2023-10-16T14:17:47.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd04c003520 con 0x7fd0600ccfb0 2023-10-16T14:17:47.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1842627159 0 0) 0x7fd04c003520 con 0x7fd0600ccfb0 2023-10-16T14:17:47.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 >> v2:172.21.15.170:3300/0 conn(0x7fd0600ad970 msgr2=0x7fd0600cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 --2- 172.21.15.29:0/3875268595 >> v2:172.21.15.170:3300/0 conn(0x7fd0600ad970 0x7fd0600cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 >> v1:172.21.15.170:6789/0 conn(0x7fd06006c040 legacy=0x7fd06006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0600cc270 con 0x7fd0600ccfb0 2023-10-16T14:17:47.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fd0580019f0 con 0x7fd0600ccfb0 2023-10-16T14:17:47.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 >> v1:172.21.15.29:6789/0 conn(0x7fd0600ccfb0 legacy=0x7fd0600cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd04c0039c0 0x7fd04c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd04c0047c0 0x7fd04c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c005180 0x7fd04c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06015bf90 con 0x7fd04c0039c0 2023-10-16T14:17:47.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c00c080 con 0x7fd04c005180 2023-10-16T14:17:47.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.044+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c00c240 con 0x7fd04c0047c0 2023-10-16T14:17:47.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd06577e640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd04c0039c0 0x7fd04c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd064f7d640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd04c0047c0 0x7fd04c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd06577e640 1 -- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd04c0047c0 msgr2=0x7fd04c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd06577e640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd04c0047c0 0x7fd04c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd065f7f640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c005180 0x7fd04c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd06577e640 1 -- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c005180 msgr2=0x7fd04c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd06577e640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c005180 0x7fd04c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd06577e640 1 -- 172.21.15.29:0/3875268595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c003520 con 0x7fd04c0039c0 2023-10-16T14:17:47.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd064f7d640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd04c0047c0 0x7fd04c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:47.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd065f7f640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c005180 0x7fd04c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:47.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.045+0000 7fd065f7f640 1 -- 172.21.15.29:0/3875268595 reap_dead start 2023-10-16T14:17:47.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.046+0000 7fd06577e640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd04c0039c0 0x7fd04c003de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd05c001750 tx=0x7fd04c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b7c480beadd7a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c0052d0 con 0x7fd04c0039c0 2023-10-16T14:17:47.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c02e050 con 0x7fd04c0039c0 2023-10-16T14:17:47.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd057fff640 1 -- 172.21.15.29:0/3875268595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c003830 con 0x7fd04c0039c0 2023-10-16T14:17:47.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd066780640 1 -- 172.21.15.29:0/3875268595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd04c0039c0 msgr2=0x7fd04c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd066780640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd04c0039c0 0x7fd04c003de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd05c001750 tx=0x7fd04c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd066780640 1 -- 172.21.15.29:0/3875268595 shutdown_connections 2023-10-16T14:17:47.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd066780640 1 --2- 172.21.15.29:0/3875268595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd04c0039c0 0x7fd04c003de0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.047+0000 7fd066780640 1 -- 172.21.15.29:0/3875268595 >> 172.21.15.29:0/3875268595 conn(0x7fd0600ce6a0 msgr2=0x7fd0600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:47.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.048+0000 7fd066780640 1 -- 172.21.15.29:0/3875268595 shutdown_connections 2023-10-16T14:17:47.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.048+0000 7fd066780640 1 -- 172.21.15.29:0/3875268595 wait complete. 2023-10-16T14:17:47.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.049+0000 7fd066780640 1 Processor -- start 2023-10-16T14:17:47.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.049+0000 7fd066780640 1 -- start start 2023-10-16T14:17:47.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.050+0000 7fd066780640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c0039c0 0x7fd060155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.050+0000 7fd066780640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 0x7fd0601e6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.050+0000 7fd066780640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0601e4480 0x7fd0601e4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.050+0000 7fd066780640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0601e7230 con 0x7fd0601e6870 2023-10-16T14:17:47.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.050+0000 7fd066780640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0601e5d50 con 0x7fd04c0039c0 2023-10-16T14:17:47.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.050+0000 7fd066780640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd060155ca0 con 0x7fd0601e4480 2023-10-16T14:17:47.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 0x7fd0601e6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 0x7fd0601e6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37306/0 (socket says 172.21.15.29:37306) 2023-10-16T14:17:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 -- 172.21.15.29:0/60197139 learned_addr learned my addr 172.21.15.29:0/60197139 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd06577e640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c0039c0 0x7fd060155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 -- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0601e4480 msgr2=0x7fd0601e4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0601e4480 0x7fd0601e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 -- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c0039c0 msgr2=0x7fd060155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c0039c0 0x7fd060155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.051+0000 7fd064f7d640 1 -- 172.21.15.29:0/60197139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0601e5140 con 0x7fd0601e6870 2023-10-16T14:17:47.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.052+0000 7fd064f7d640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 0x7fd0601e6cf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd0500107c0 tx=0x7fd050010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.052+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05000eb00 con 0x7fd0601e6870 2023-10-16T14:17:47.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.052+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050010ed0 con 0x7fd0601e6870 2023-10-16T14:17:47.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.052+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0601e5420 con 0x7fd0601e6870 2023-10-16T14:17:47.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.052+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05000ee60 con 0x7fd0601e6870 2023-10-16T14:17:47.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.052+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0601e59e0 con 0x7fd0601e6870 2023-10-16T14:17:47.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.053+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06006b330 con 0x7fd0601e6870 2023-10-16T14:17:47.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.054+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd05001c400 con 0x7fd0601e6870 2023-10-16T14:17:47.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.055+0000 7fd055ffb640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd04406aea0 0x7fd04406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.055+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd050089a10 con 0x7fd0601e6870 2023-10-16T14:17:47.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.056+0000 7fd06577e640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd04406aea0 0x7fd04406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.057+0000 7fd06577e640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd04406aea0 0x7fd04406d360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd05c033000 tx=0x7fd05c009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.060+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd050054a20 con 0x7fd0601e6870 2023-10-16T14:17:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.291+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0601e5f00 con 0x7fd0601e6870 2023-10-16T14:17:47.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.292+0000 7fd055ffb640 1 -- 172.21.15.29:0/60197139 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fd0601e5f00 con 0x7fd0601e6870 2023-10-16T14:17:47.296 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:17:47.296 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:17:40.632013+0000","modified":"2023-10-16T14:17:43.716037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4720},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4720":{"gid":4720,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6835/2287219370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2287219370},{"type":"v1","addr":"172.21.15.29:6835","nonce":2287219370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T14:17:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-16T14:17:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd04406aea0 msgr2=0x7fd04406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd04406aea0 0x7fd04406d360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd05c033000 tx=0x7fd05c009030 comp rx=0 tx=0).stop 2023-10-16T14:17:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 msgr2=0x7fd0601e6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 0x7fd0601e6cf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd0500107c0 tx=0x7fd050010c80 comp rx=0 tx=0).stop 2023-10-16T14:17:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 shutdown_connections 2023-10-16T14:17:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fd04406aea0 0x7fd04406d360 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0601e6870 0x7fd0601e6cf0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0601e4480 0x7fd0601e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 --2- 172.21.15.29:0/60197139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd04c0039c0 0x7fd060155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 >> 172.21.15.29:0/60197139 conn(0x7fd0600ce6a0 msgr2=0x7fd060069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 shutdown_connections 2023-10-16T14:17:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.295+0000 7fd066780640 1 -- 172.21.15.29:0/60197139 wait complete. 2023-10-16T14:17:47.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:17:40.632013+0000', 'modified': '2023-10-16T14:17:43.716037+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4720}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4720': {'gid': 4720, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.29:6835/2287219370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 2287219370}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2287219370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:17:47.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:17:47.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:17:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.958+0000 7ff3ded54640 1 Processor -- start 2023-10-16T14:17:47.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.958+0000 7ff3ded54640 1 -- start start 2023-10-16T14:17:47.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.959+0000 7ff3ded54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.959+0000 7ff3ded54640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 0x7ff3d815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.960+0000 7ff3ded54640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d806c960 con 0x7ff3d80ad970 2023-10-16T14:17:47.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.960+0000 7ff3ded54640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d815b9d0 con 0x7ff3d80cda80 2023-10-16T14:17:47.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.960+0000 7ff3ded54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d815bcb0 con 0x7ff3d806c040 2023-10-16T14:17:47.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.960+0000 7ff3de553640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 0x7ff3d815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.961+0000 7ff3ddd52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.961+0000 7ff3dd551640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7ff3d80ad970 0x7ff3d815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:36740/0 (socket says 172.21.15.29:36740) 2023-10-16T14:17:47.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.961+0000 7ff3de553640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 0x7ff3d815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:58420/0 (socket says 172.21.15.29:58420) 2023-10-16T14:17:47.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.961+0000 7ff3dd551640 1 -- 172.21.15.29:0/3521735997 learned_addr learned my addr 172.21.15.29:0/3521735997 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:47.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.961+0000 7ff3dcd50640 1 -- 172.21.15.29:0/3521735997 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43694393 0 0) 0x7ff3d806c960 con 0x7ff3d80ad970 2023-10-16T14:17:47.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.962+0000 7ff3dcd50640 1 -- 172.21.15.29:0/3521735997 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4003520 con 0x7ff3d80ad970 2023-10-16T14:17:47.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.962+0000 7ff3ddd52640 1 -- 172.21.15.29:0/3521735997 >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 msgr2=0x7ff3d815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.962+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/3521735997 >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 0x7ff3d815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.962+0000 7ff3ddd52640 1 -- 172.21.15.29:0/3521735997 >> v1:172.21.15.170:6789/0 conn(0x7ff3d80ad970 legacy=0x7ff3d815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.962+0000 7ff3ddd52640 1 -- 172.21.15.29:0/3521735997 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d815bf90 con 0x7ff3d806c040 2023-10-16T14:17:47.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.962+0000 7ff3de553640 1 --2- 172.21.15.29:0/3521735997 >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 0x7ff3d815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:17:47.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.963+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/3521735997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d806c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097a0 tx=0x7ff3d402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0cbc51580c44ef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.964+0000 7ff3dcd50640 1 -- 172.21.15.29:0/3521735997 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d402d040 con 0x7ff3d806c040 2023-10-16T14:17:47.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.964+0000 7ff3dcd50640 1 -- 172.21.15.29:0/3521735997 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d4031810 con 0x7ff3d806c040 2023-10-16T14:17:47.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.964+0000 7ff3dcd50640 1 -- 172.21.15.29:0/3521735997 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4031b80 con 0x7ff3d806c040 2023-10-16T14:17:47.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 -- 172.21.15.29:0/3521735997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 msgr2=0x7ff3d806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3521735997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d806c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097a0 tx=0x7ff3d402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 -- 172.21.15.29:0/3521735997 shutdown_connections 2023-10-16T14:17:47.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3521735997 >> v2:172.21.15.170:3300/0 conn(0x7ff3d80cda80 0x7ff3d815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3521735997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d806c420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 -- 172.21.15.29:0/3521735997 >> 172.21.15.29:0/3521735997 conn(0x7ff3d80ce6a0 msgr2=0x7ff3d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:47.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 -- 172.21.15.29:0/3521735997 shutdown_connections 2023-10-16T14:17:47.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.965+0000 7ff3ded54640 1 -- 172.21.15.29:0/3521735997 wait complete. 2023-10-16T14:17:47.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.967+0000 7ff3ded54640 1 Processor -- start 2023-10-16T14:17:47.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.967+0000 7ff3ded54640 1 -- start start 2023-10-16T14:17:47.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ded54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d814c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ded54640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3d80ad970 0x7ff3d80c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ded54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 0x7ff3d815ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ded54640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d81f07d0 con 0x7ff3d80cda80 2023-10-16T14:17:47.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ded54640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80ae470 con 0x7ff3d80ad970 2023-10-16T14:17:47.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ded54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80ae750 con 0x7ff3d806c040 2023-10-16T14:17:47.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3de553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 0x7ff3d815ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3de553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 0x7ff3d815ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37312/0 (socket says 172.21.15.29:37312) 2023-10-16T14:17:47.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3de553640 1 -- 172.21.15.29:0/3504194468 learned_addr learned my addr 172.21.15.29:0/3504194468 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:47.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d814c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.968+0000 7ff3dd551640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3d80ad970 0x7ff3d80c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3de553640 1 -- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 msgr2=0x7ff3d814c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3de553640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d814c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3de553640 1 -- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3d80ad970 msgr2=0x7ff3d80c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3de553640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3d80ad970 0x7ff3d80c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3de553640 1 -- 172.21.15.29:0/3504194468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0009a70 con 0x7ff3d80cda80 2023-10-16T14:17:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d814c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3dd551640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3d80ad970 0x7ff3d80c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3de553640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 0x7ff3d815ca50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000efc0 tx=0x7ff3d0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3ce7fc640 1 -- 172.21.15.29:0/3504194468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0009d70 con 0x7ff3d80cda80 2023-10-16T14:17:47.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3ce7fc640 1 -- 172.21.15.29:0/3504194468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d000f040 con 0x7ff3d80cda80 2023-10-16T14:17:47.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3ce7fc640 1 -- 172.21.15.29:0/3504194468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0003990 con 0x7ff3d80cda80 2023-10-16T14:17:47.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d4009400 con 0x7ff3d80cda80 2023-10-16T14:17:47.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.969+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d80aed10 con 0x7ff3d80cda80 2023-10-16T14:17:47.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.972+0000 7ff3ce7fc640 1 -- 172.21.15.29:0/3504194468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff3d00144a0 con 0x7ff3d80cda80 2023-10-16T14:17:47.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.972+0000 7ff3ce7fc640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff3bc06aea0 0x7ff3bc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.973+0000 7ff3ce7fc640 1 -- 172.21.15.29:0/3504194468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3d008a250 con 0x7ff3d80cda80 2023-10-16T14:17:47.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.973+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff3bc06aea0 0x7ff3bc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.973+0000 7ff3e2c27740 1 Processor -- start 2023-10-16T14:17:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.973+0000 7ff3e2c27740 1 -- start start 2023-10-16T14:17:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff3bc06aea0 0x7ff3bc06d360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4036000 tx=0x7ff3d403c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 0x55edec277070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2775b0 0x55edec2a7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec1977a0 con 0x55edec2a4170 2023-10-16T14:17:47.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec1e9740 con 0x55edec2775b0 2023-10-16T14:17:47.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.974+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edec1e6e30 con 0x55edec2a4550 2023-10-16T14:17:47.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 0x55edec277070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 0x55edec277070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37328/0 (socket says 172.21.15.29:37328) 2023-10-16T14:17:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 -- 172.21.15.29:0/2522562461 learned_addr learned my addr 172.21.15.29:0/2522562461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3de553640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2775b0 0x55edec2a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3dd551640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 -- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 msgr2=0x55edec2a6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2a6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 -- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2775b0 msgr2=0x55edec2a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2775b0 0x55edec2a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3ddd52640 1 -- 172.21.15.29:0/2522562461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edec1a6910 con 0x55edec2a4170 2023-10-16T14:17:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3dd551640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2a6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.975+0000 7ff3de553640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2775b0 0x55edec2a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.976+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 0x55edec277070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3d402bec0 tx=0x7ff3d4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70d2fc1a942dd09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.976+0000 7ff3bb7fe640 1 -- 172.21.15.29:0/2522562461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4031cc0 con 0x55edec2a4170 2023-10-16T14:17:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.976+0000 7ff3bb7fe640 1 -- 172.21.15.29:0/2522562461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d4031e20 con 0x55edec2a4170 2023-10-16T14:17:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.976+0000 7ff3bb7fe640 1 -- 172.21.15.29:0/2522562461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d404d400 con 0x55edec2a4170 2023-10-16T14:17:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.976+0000 7ff3e2c27740 1 -- 172.21.15.29:0/2522562461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 msgr2=0x55edec277070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.976+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 0x55edec277070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3d402bec0 tx=0x7ff3d4030000 comp rx=0 tx=0).stop 2023-10-16T14:17:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 -- 172.21.15.29:0/2522562461 shutdown_connections 2023-10-16T14:17:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2775b0 0x55edec2a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2a6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/2522562461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4170 0x55edec277070 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 -- 172.21.15.29:0/2522562461 >> 172.21.15.29:0/2522562461 conn(0x55edec279380 msgr2=0x55edec2c1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 -- 172.21.15.29:0/2522562461 shutdown_connections 2023-10-16T14:17:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 -- 172.21.15.29:0/2522562461 wait complete. 2023-10-16T14:17:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.977+0000 7ff3e2c27740 1 -- start start 2023-10-16T14:17:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.978+0000 7ff3e2c27740 1 Processor -- start 2023-10-16T14:17:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2baa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2baf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec1e2720 con 0x55edec2a1cd0 2023-10-16T14:17:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec180970 con 0x55edec2a4170 2023-10-16T14:17:47.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edec180610 con 0x55edec2a4550 2023-10-16T14:17:47.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3e2c27740 10 client.0 _ll_register_callbacks cb 0x55edec1a29d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:17:47.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2baf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2baf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37338/0 (socket says 172.21.15.29:37338) 2023-10-16T14:17:47.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 -- 172.21.15.29:0/1033107981 learned_addr learned my addr 172.21.15.29:0/1033107981 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3dd551640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2baa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 -- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 msgr2=0x55edec2baa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2baa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 -- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 msgr2=0x55edec2c2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2c2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.979+0000 7ff3de553640 1 -- 172.21.15.29:0/1033107981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d815bf90 con 0x55edec2a1cd0 2023-10-16T14:17:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3de553640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2baf40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0058f50 tx=0x7ff3d00612c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59708daa73315b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:17:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1033107981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d00586b0 con 0x55edec2a1cd0 2023-10-16T14:17:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1033107981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3d008a070 con 0x55edec2a1cd0 2023-10-16T14:17:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55edec209160 con 0x55edec2a1cd0 2023-10-16T14:17:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1033107981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3d00882a0 con 0x55edec2a1cd0 2023-10-16T14:17:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3e2c27740 10 client.5202 fetch_fsmap learned FSMap version 17 2023-10-16T14:17:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55edec1e0da0 con 0x55edec2a1cd0 2023-10-16T14:17:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.980+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1033107981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d0074020 con 0x55edec2a1cd0 2023-10-16T14:17:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1033107981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff3d0060e30 con 0x55edec2a1cd0 2023-10-16T14:17:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3e2c27740 10 client.5202 fetch_fsmap finished waiting for FSMap version 17 2023-10-16T14:17:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3e2c27740 10 client.5202 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:17:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3e2c27740 20 client.5202 populate_metadata read hostname 'smithi029' 2023-10-16T14:17:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec2a8a80 0x55edec2b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3e2c27740 4 client.5202 mds_command: new command op to 4720 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:17:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 --> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55edec1b4cc0 con 0x55edec2a8a80 2023-10-16T14:17:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.981+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec2a8a80 0x55edec2b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.982+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec2a8a80 0x55edec2b9750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8044e764d6559faa server_cookie=ca126c6456cde0ab in_seq=0 out_seq=0 2023-10-16T14:17:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.982+0000 7ff3b8ff9640 10 client.5202 ms_handle_connect on v2:172.21.15.29:6833/2287219370 2023-10-16T14:17:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.982+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1033107981 <== mds.0 v2:172.21.15.29:6833/2287219370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55edec1b4cc0 con 0x55edec2a8a80 2023-10-16T14:17:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.982+0000 7ff3b8ff9640 10 client.5202 handle_command_reply: tid=0 2023-10-16T14:17:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.982+0000 7ff3e2c27740 1 client.5202 shutdown 2023-10-16T14:17:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 msgr2=0x55edec2baf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2baf40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0058f50 tx=0x7ff3d00612c0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 shutdown_connections 2023-10-16T14:17:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2baf40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec2a8a80 0x55edec2b9750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4550 0x55edec2baa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1033107981 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2c2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 >> 172.21.15.29:0/1033107981 conn(0x55edec2775b0 msgr2=0x55edec2a6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3b8ff9640 0 client.5202 ms_handle_reset on v2:172.21.15.29:6833/2287219370 2023-10-16T14:17:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 shutdown_connections 2023-10-16T14:17:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1033107981 wait complete. 2023-10-16T14:17:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:47.983+0000 7ff3e2c27740 20 client.5202 trim_cache size 0 max 16384 2023-10-16T14:17:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.000+0000 7ff3e2c27740 1 Processor -- start 2023-10-16T14:17:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.000+0000 7ff3e2c27740 1 -- start start 2023-10-16T14:17:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a1cd0 0x55edec2aa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2bb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 0x55edec2c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edebf46f20 con 0x55edec2a4550 2023-10-16T14:17:48.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec2c5a40 con 0x55edec2a4170 2023-10-16T14:17:48.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edec2bc240 con 0x55edec2a1cd0 2023-10-16T14:17:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3de553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 0x55edec2c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3de553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 0x55edec2c4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37352/0 (socket says 172.21.15.29:37352) 2023-10-16T14:17:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.001+0000 7ff3de553640 1 -- 172.21.15.29:0/3246319760 learned_addr learned my addr 172.21.15.29:0/3246319760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3dd551640 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2bb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3de553640 1 -- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a1cd0 msgr2=0x55edec2aa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:48.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3de553640 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a1cd0 0x55edec2aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3de553640 1 -- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 msgr2=0x55edec2bb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:48.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3de553640 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2bb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3de553640 1 -- 172.21.15.29:0/3246319760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edec17fb80 con 0x55edec2a4550 2023-10-16T14:17:48.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3de553640 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 0x55edec2c4c30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00554c0 tx=0x7ff3d008a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb054eda26e61c39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3bbfff640 1 -- 172.21.15.29:0/3246319760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0055300 con 0x55edec2a4550 2023-10-16T14:17:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3bbfff640 1 -- 172.21.15.29:0/3246319760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d0074070 con 0x55edec2a4550 2023-10-16T14:17:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.002+0000 7ff3bbfff640 1 -- 172.21.15.29:0/3246319760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d008a070 con 0x55edec2a4550 2023-10-16T14:17:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 -- 172.21.15.29:0/3246319760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 msgr2=0x55edec2c4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 0x55edec2c4c30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00554c0 tx=0x7ff3d008a000 comp rx=0 tx=0).stop 2023-10-16T14:17:48.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 -- 172.21.15.29:0/3246319760 shutdown_connections 2023-10-16T14:17:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a4550 0x55edec2c4c30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4170 0x55edec2bb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/3246319760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a1cd0 0x55edec2aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 -- 172.21.15.29:0/3246319760 >> 172.21.15.29:0/3246319760 conn(0x55edec2775b0 msgr2=0x55edec2a6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 -- 172.21.15.29:0/3246319760 shutdown_connections 2023-10-16T14:17:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.003+0000 7ff3e2c27740 1 -- 172.21.15.29:0/3246319760 wait complete. 2023-10-16T14:17:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.004+0000 7ff3e2c27740 1 -- start start 2023-10-16T14:17:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.004+0000 7ff3e2c27740 1 Processor -- start 2023-10-16T14:17:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2941e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4170 0x55edec2b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4550 0x55edec2b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec2d12a0 con 0x55edec2a1cd0 2023-10-16T14:17:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edec294a70 con 0x55edec2a4550 2023-10-16T14:17:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edec2c4070 con 0x55edec2a4170 2023-10-16T14:17:48.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3e2c27740 10 client.0 _ll_register_callbacks cb 0x55edec2aae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:17:48.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3ddd52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2941e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3ddd52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2941e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37366/0 (socket says 172.21.15.29:37366) 2023-10-16T14:17:48.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.005+0000 7ff3ddd52640 1 -- 172.21.15.29:0/1340949850 learned_addr learned my addr 172.21.15.29:0/1340949850 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:48.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3de553640 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4550 0x55edec2b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3ddd52640 1 -- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4170 msgr2=0x55edec2b7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4170 0x55edec2b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3ddd52640 1 -- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4550 msgr2=0x55edec2b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4550 0x55edec2b82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3ddd52640 1 -- 172.21.15.29:0/1340949850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55edec1e0da0 con 0x55edec2a1cd0 2023-10-16T14:17:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3ddd52640 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2941e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40098c0 tx=0x7ff3d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b694a552d7a4f85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:48.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:17:48.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1340949850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4031c70 con 0x55edec2a1cd0 2023-10-16T14:17:48.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1340949850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3d4059bb0 con 0x55edec2a1cd0 2023-10-16T14:17:48.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3d4031e20 con 0x55edec2a1cd0 2023-10-16T14:17:48.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.006+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1340949850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3d4059d70 con 0x55edec2a1cd0 2023-10-16T14:17:48.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 10 client.5208 fetch_fsmap learned FSMap version 17 2023-10-16T14:17:48.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55edec1a5170 con 0x55edec2a1cd0 2023-10-16T14:17:48.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1340949850 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d4031e20 con 0x55edec2a1cd0 2023-10-16T14:17:48.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1340949850 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff3d4056040 con 0x55edec2a1cd0 2023-10-16T14:17:48.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 10 client.5208 fetch_fsmap finished waiting for FSMap version 17 2023-10-16T14:17:48.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 10 client.5208 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:17:48.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 20 client.5208 populate_metadata read hostname 'smithi029' 2023-10-16T14:17:48.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec297d80 0x55edec29b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 4 client.5208 mds_command: new command op to 4720 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:17:48.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 --> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55edec1db700 con 0x55edec297d80 2023-10-16T14:17:48.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.007+0000 7ff3dd551640 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec297d80 0x55edec29b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3dd551640 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec297d80 0x55edec29b030 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c317503552cba6ec server_cookie=e2333980193ed929 in_seq=0 out_seq=0 2023-10-16T14:17:48.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3b8ff9640 10 client.5208 ms_handle_connect on v2:172.21.15.29:6833/2287219370 2023-10-16T14:17:48.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3b8ff9640 1 -- 172.21.15.29:0/1340949850 <== mds.0 v2:172.21.15.29:6833/2287219370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55edec1db700 con 0x55edec297d80 2023-10-16T14:17:48.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3b8ff9640 10 client.5208 handle_command_reply: tid=0 2023-10-16T14:17:48.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 client.5208 shutdown 2023-10-16T14:17:48.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 msgr2=0x55edec2941e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2941e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40098c0 tx=0x7ff3d4004730 comp rx=0 tx=0).stop 2023-10-16T14:17:48.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 shutdown_connections 2023-10-16T14:17:48.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:6833/2287219370,v1:172.21.15.29:6835/2287219370] conn(0x55edec297d80 0x55edec29b030 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55edec2a4550 0x55edec2b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55edec2a4170 0x55edec2b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 --2- 172.21.15.29:0/1340949850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55edec2a1cd0 0x55edec2941e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3b8ff9640 0 client.5208 ms_handle_reset on v2:172.21.15.29:6833/2287219370 2023-10-16T14:17:48.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 >> 172.21.15.29:0/1340949850 conn(0x55edec2775b0 msgr2=0x55edec2baac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:48.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 shutdown_connections 2023-10-16T14:17:48.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.008+0000 7ff3e2c27740 1 -- 172.21.15.29:0/1340949850 wait complete. 2023-10-16T14:17:48.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3e2c27740 20 client.5208 trim_cache size 0 max 16384 2023-10-16T14:17:48.026 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:17:48.027 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:17:48.027 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:17:48.027 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4720, 2023-10-16T14:17:48.027 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:17:48.027 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:17:48.028 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:17:48.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8334969340000002, 2023-10-16T14:17:48.028 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 17, 2023-10-16T14:17:48.028 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 33, 2023-10-16T14:17:48.028 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 33, 2023-10-16T14:17:48.029 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 15.640246346 2023-10-16T14:17:48.029 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:17:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff3bc06aea0 msgr2=0x7ff3bc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff3bc06aea0 0x7ff3bc06d360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4036000 tx=0x7ff3d403c040 comp rx=0 tx=0).stop 2023-10-16T14:17:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 msgr2=0x7ff3d815ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 0x7ff3d815ca50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000efc0 tx=0x7ff3d0009f90 comp rx=0 tx=0).stop 2023-10-16T14:17:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 shutdown_connections 2023-10-16T14:17:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7ff3bc06aea0 0x7ff3bc06d360 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3d80cda80 0x7ff3d815ca50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3d80ad970 0x7ff3d80c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 --2- 172.21.15.29:0/3504194468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3d806c040 0x7ff3d814c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 >> 172.21.15.29:0/3504194468 conn(0x7ff3d80ce6a0 msgr2=0x7ff3d815e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 shutdown_connections 2023-10-16T14:17:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.009+0000 7ff3ded54640 1 -- 172.21.15.29:0/3504194468 wait complete. 2023-10-16T14:17:48.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:17:48.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.257+0000 7f81408b0640 1 Processor -- start 2023-10-16T14:17:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.257+0000 7f81408b0640 1 -- start start 2023-10-16T14:17:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f81408b0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f813c15aaa0 0x7f813c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f81408b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f813c0ce390 0x7f813c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f81408b0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c15ed60 con 0x7f813c15ae80 2023-10-16T14:17:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f81408b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c06b810 con 0x7f813c0ce390 2023-10-16T14:17:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f81408b0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c0707e0 con 0x7f813c15aaa0 2023-10-16T14:17:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f813ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f813c0ce390 0x7f813c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f813ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f813c0ce390 0x7f813c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37374/0 (socket says 172.21.15.29:37374) 2023-10-16T14:17:48.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.259+0000 7f813ad9d640 1 -- 172.21.15.29:0/1903762049 learned_addr learned my addr 172.21.15.29:0/1903762049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:48.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.260+0000 7f813a59c640 1 --2- 172.21.15.29:0/1903762049 >> v2:172.21.15.170:3300/0 conn(0x7f813c15aaa0 0x7f813c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.260+0000 7f813ad9d640 1 -- 172.21.15.29:0/1903762049 >> v2:172.21.15.170:3300/0 conn(0x7f813c15aaa0 msgr2=0x7f813c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:48.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.260+0000 7f813ad9d640 1 --2- 172.21.15.29:0/1903762049 >> v2:172.21.15.170:3300/0 conn(0x7f813c15aaa0 0x7f813c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.260+0000 7f813ad9d640 1 -- 172.21.15.29:0/1903762049 >> v1:172.21.15.170:6789/0 conn(0x7f813c15ae80 legacy=0x7f813c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:48.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.260+0000 7f813ad9d640 1 -- 172.21.15.29:0/1903762049 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c15ef90 con 0x7f813c0ce390 2023-10-16T14:17:48.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.261+0000 7f813ad9d640 1 --2- 172.21.15.29:0/1903762049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c070f50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f813c0cd990 tx=0x7f813402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641cba362399248b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f8139d9b640 1 -- 172.21.15.29:0/1903762049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813402ca80 con 0x7f813c0ce390 2023-10-16T14:17:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f8139d9b640 1 -- 172.21.15.29:0/1903762049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813402cbe0 con 0x7f813c0ce390 2023-10-16T14:17:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f8139d9b640 1 -- 172.21.15.29:0/1903762049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81340343f0 con 0x7f813c0ce390 2023-10-16T14:17:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 -- 172.21.15.29:0/1903762049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 msgr2=0x7f813c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 --2- 172.21.15.29:0/1903762049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c070f50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f813c0cd990 tx=0x7f813402bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 -- 172.21.15.29:0/1903762049 shutdown_connections 2023-10-16T14:17:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 --2- 172.21.15.29:0/1903762049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c070f50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 --2- 172.21.15.29:0/1903762049 >> v2:172.21.15.170:3300/0 conn(0x7f813c15aaa0 0x7f813c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 -- 172.21.15.29:0/1903762049 >> 172.21.15.29:0/1903762049 conn(0x7f813c0cb410 msgr2=0x7f813c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:48.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 -- 172.21.15.29:0/1903762049 shutdown_connections 2023-10-16T14:17:48.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.262+0000 7f81408b0640 1 -- 172.21.15.29:0/1903762049 wait complete. 2023-10-16T14:17:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.264+0000 7f81408b0640 1 Processor -- start 2023-10-16T14:17:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.264+0000 7f81408b0640 1 -- start start 2023-10-16T14:17:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.265+0000 7f81408b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.265+0000 7f81408b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f813c15aaa0 0x7f813c15d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.265+0000 7f81408b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f813c15ae80 0x7f813c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.265+0000 7f81408b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0c4ae0 con 0x7f813c0ce390 2023-10-16T14:17:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.265+0000 7f81408b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1561d0 con 0x7f813c15aaa0 2023-10-16T14:17:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.265+0000 7f81408b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0c3ef0 con 0x7f813c15ae80 2023-10-16T14:17:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37376/0 (socket says 172.21.15.29:37376) 2023-10-16T14:17:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 -- 172.21.15.29:0/3997043482 learned_addr learned my addr 172.21.15.29:0/3997043482 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f8133fff640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f813c15aaa0 0x7f813c15d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 -- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f813c15ae80 msgr2=0x7f813c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f813c15ae80 0x7f813c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 -- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f813c15aaa0 msgr2=0x7f813c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f813c15aaa0 0x7f813c15d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.266+0000 7f813a59c640 1 -- 172.21.15.29:0/3997043482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8134009b30 con 0x7f813c0ce390 2023-10-16T14:17:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.267+0000 7f813a59c640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c156870 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f812c00d830 tx=0x7f812c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.267+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00b6c0 con 0x7f813c0ce390 2023-10-16T14:17:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.267+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c00b820 con 0x7f813c0ce390 2023-10-16T14:17:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.267+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00bbb0 con 0x7f813c0ce390 2023-10-16T14:17:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.267+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c0c4140 con 0x7f813c0ce390 2023-10-16T14:17:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.267+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c0c4700 con 0x7f813c0ce390 2023-10-16T14:17:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.268+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c04e160 con 0x7f813c0ce390 2023-10-16T14:17:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.269+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f812c01e460 con 0x7f813c0ce390 2023-10-16T14:17:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.270+0000 7f8132ffd640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f811c06aea0 0x7f811c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.270+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f812c089cd0 con 0x7f813c0ce390 2023-10-16T14:17:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.271+0000 7f8133fff640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f811c06aea0 0x7f811c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.272+0000 7f8133fff640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f811c06aea0 0x7f811c06d360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8124006fd0 tx=0x7f8124008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.274+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f812c054e10 con 0x7f813c0ce390 2023-10-16T14:17:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.510+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f813c0cdc60 con 0x7f813c0ce390 2023-10-16T14:17:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.511+0000 7f8132ffd640 1 -- 172.21.15.29:0/3997043482 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f812c058430 con 0x7f813c0ce390 2023-10-16T14:17:48.515 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:17:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.513+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f811c06aea0 msgr2=0x7f811c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.513+0000 7f81408b0640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f811c06aea0 0x7f811c06d360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8124006fd0 tx=0x7f8124008040 comp rx=0 tx=0).stop 2023-10-16T14:17:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.513+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 msgr2=0x7f813c156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.513+0000 7f81408b0640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c156870 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f812c00d830 tx=0x7f812c00dcf0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 shutdown_connections 2023-10-16T14:17:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f811c06aea0 0x7f811c06d360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f813c15ae80 0x7f813c15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f813c15aaa0 0x7f813c15d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 --2- 172.21.15.29:0/3997043482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f813c0ce390 0x7f813c156870 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 >> 172.21.15.29:0/3997043482 conn(0x7f813c0cb410 msgr2=0x7f813c06fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 shutdown_connections 2023-10-16T14:17:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:48.514+0000 7f81408b0640 1 -- 172.21.15.29:0/3997043482 wait complete. 2023-10-16T14:17:48.538 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:17:48.539 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:17:48.539 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:17:48.540 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:17:48.540 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:17:48.540 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:17:48.541 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:17:48.541 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:17:48.541 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:17:48.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:48.542 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:17:48.600 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:17:48.601 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:17:48.601 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:17:48.601 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.601 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:17:48.601 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.602 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:17:48.602 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:17:48.602 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:17:48.602 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:17:48.602 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:17:48.603 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:17:48.603 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:17:48.603 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:17:48.603 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:17:48.603 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:17:48.604 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42256sec preferred_lft 42256sec 2023-10-16T14:17:48.604 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:17:48.604 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.604 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:17:48.604 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:17:48.604 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:17:48.605 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:17:48.605 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:17:48.605 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:17:48.605 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.605 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:17:48.606 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.606 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:17:48.606 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:17:48.606 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:17:48.606 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.607 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:17:48.607 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:17:48.607 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:17:48.607 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:17:48.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:48.609 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:17:48.706 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:17:48.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:48.711 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:17:48.772 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:17:48.773 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:17:48.773 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:17:48.773 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:17:48.773 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:17:48.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:48.774 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:17:48.775 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:17:48.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:17:48.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:17:48.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:17:48.776 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:17:48.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:48.978 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:17:49.034 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:17:49.035 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:17:49.035 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:17:49.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:49.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:17:49.139 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:17:49.139 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:17:49.203 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:17:49.203 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:17:49.204 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:17:49.261 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:17:49.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:49.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:17:49.413 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:17:49.414 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:17:49.414 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:17:49.438 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:17:49.438 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:17:49.438 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:17:49.439 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:17:49.439 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:17:49.464 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:17:49.464 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:17:49.465 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:17:49.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:17:49.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:17:49.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:17:49.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:17:49.466 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:17:49.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:17:49.469 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:17:49.469 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:17:49.469 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:17:49.469 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:17:49.472 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:17:49.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:49.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:17:49.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:49.563 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:17:49.631 INFO:teuthology.orchestra.run.smithi029.stdout:client5217 2023-10-16T14:17:49.631 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:17:49.632 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:17:49.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:17:49.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.912+0000 7f0cd3575640 1 Processor -- start 2023-10-16T14:17:49.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.912+0000 7f0cd3575640 1 -- start start 2023-10-16T14:17:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.913+0000 7f0cd3575640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 0x7f0ccc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd3575640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0b8480 0x7f0ccc0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd3575640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc15d150 con 0x7f0ccc0d0be0 2023-10-16T14:17:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd3575640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ccc0d1180 con 0x7f0ccc0b8480 2023-10-16T14:17:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd3575640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0d1460 con 0x7f0ccc15cd70 2023-10-16T14:17:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd2d74640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0ccc0d0be0 0x7f0ccc0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54642/0 (socket says 172.21.15.29:54642) 2023-10-16T14:17:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd2d74640 1 -- 172.21.15.29:0/3612517812 learned_addr learned my addr 172.21.15.29:0/3612517812 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd1d72640 1 --2- 172.21.15.29:0/3612517812 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0b8480 0x7f0ccc0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.914+0000 7f0cd2573640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 0x7f0ccc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485222728 0 0) 0x7f0ccc15d150 con 0x7f0ccc0d0be0 2023-10-16T14:17:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb4003520 con 0x7f0ccc0d0be0 2023-10-16T14:17:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216396063 0 0) 0x7f0cb4003520 con 0x7f0ccc0d0be0 2023-10-16T14:17:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 msgr2=0x7f0ccc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 0x7f0ccc0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0b8480 msgr2=0x7f0ccc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 --2- 172.21.15.29:0/3612517812 >> v2:172.21.15.170:3300/0 conn(0x7f0ccc0b8480 0x7f0ccc0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0d1740 con 0x7f0ccc0d0be0 2023-10-16T14:17:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0cc4001670 con 0x7f0ccc0d0be0 2023-10-16T14:17:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 >> v1:172.21.15.29:6789/0 conn(0x7f0ccc0d0be0 legacy=0x7f0ccc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 0x7f0cb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0d1180 con 0x7f0cb4004d00 2023-10-16T14:17:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0d1460 con 0x7f0cb4003970 2023-10-16T14:17:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.915+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400b980 con 0x7f0ccc15cd70 2023-10-16T14:17:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2d74640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2573640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd1d72640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 0x7f0cb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2d74640 1 -- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 msgr2=0x7f0cb40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2d74640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ccc15cd70 0x7f0cb40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2d74640 1 -- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cb4003970 msgr2=0x7f0cb4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2d74640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2d74640 1 -- 172.21.15.29:0/3612517812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4003520 con 0x7f0cb4004d00 2023-10-16T14:17:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.916+0000 7f0cd2573640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.918+0000 7f0cd2d74640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c110 tx=0x7f0cb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e1975ee3abc6f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.918+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4005ad0 con 0x7f0cb4004d00 2023-10-16T14:17:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.918+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4005c50 con 0x7f0cb4004d00 2023-10-16T14:17:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.918+0000 7f0cd1571640 1 -- 172.21.15.29:0/3612517812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc402c830 con 0x7f0cb4004d00 2023-10-16T14:17:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd3575640 1 -- 172.21.15.29:0/3612517812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb4004d00 msgr2=0x7f0cb4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd3575640 1 --2- 172.21.15.29:0/3612517812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c110 tx=0x7f0cb40057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd2d74640 1 -- 172.21.15.29:0/3612517812 reap_dead start 2023-10-16T14:17:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd3575640 1 -- 172.21.15.29:0/3612517812 shutdown_connections 2023-10-16T14:17:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd3575640 1 -- 172.21.15.29:0/3612517812 >> 172.21.15.29:0/3612517812 conn(0x7f0ccc0cf570 msgr2=0x7f0ccc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd3575640 1 -- 172.21.15.29:0/3612517812 shutdown_connections 2023-10-16T14:17:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.919+0000 7f0cd3575640 1 -- 172.21.15.29:0/3612517812 wait complete. 2023-10-16T14:17:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.921+0000 7f0cd3575640 1 Processor -- start 2023-10-16T14:17:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.921+0000 7f0cd3575640 1 -- start start 2023-10-16T14:17:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.922+0000 7f0cd3575640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cb4004d00 0x7f0ccc0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.922+0000 7f0cd3575640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 0x7f0ccc0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.922+0000 7f0cd3575640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0b24b0 0x7f0ccc15f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.922+0000 7f0cd3575640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0b4260 con 0x7f0ccc0b3180 2023-10-16T14:17:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.922+0000 7f0cd3575640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0b1780 con 0x7f0ccc0b24b0 2023-10-16T14:17:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.922+0000 7f0cd3575640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0cc6c0 con 0x7f0cb4004d00 2023-10-16T14:17:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 0x7f0ccc0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 0x7f0ccc0b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37400/0 (socket says 172.21.15.29:37400) 2023-10-16T14:17:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 -- 172.21.15.29:0/2020899116 learned_addr learned my addr 172.21.15.29:0/2020899116 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 -- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cb4004d00 msgr2=0x7f0ccc0cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cb4004d00 0x7f0ccc0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 -- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0b24b0 msgr2=0x7f0ccc15f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0b24b0 0x7f0ccc15f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.923+0000 7f0cd1d72640 1 -- 172.21.15.29:0/2020899116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0b3f80 con 0x7f0ccc0b3180 2023-10-16T14:17:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.924+0000 7f0cd1d72640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 0x7f0ccc0b3680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00116d0 tx=0x7f0cc0011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.924+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000f940 con 0x7f0ccc0b3180 2023-10-16T14:17:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.924+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc000faa0 con 0x7f0ccc0b3180 2023-10-16T14:17:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.924+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc0b19d0 con 0x7f0ccc0b3180 2023-10-16T14:17:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.924+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000fe10 con 0x7f0ccc0b3180 2023-10-16T14:17:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.924+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc0b1f90 con 0x7f0ccc0b3180 2023-10-16T14:17:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.925+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc04e9b0 con 0x7f0ccc0b3180 2023-10-16T14:17:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.927+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0cc0015460 con 0x7f0ccc0b3180 2023-10-16T14:17:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.927+0000 7f0ccaffd640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0cb806ae70 0x7f0cb806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.928+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cc0089f50 con 0x7f0ccc0b3180 2023-10-16T14:17:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.928+0000 7f0cd2573640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0cb806ae70 0x7f0cb806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.929+0000 7f0cd2573640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0cb806ae70 0x7f0cb806d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009f80 tx=0x7f0cbc009750 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:49.932+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0cc0054f60 con 0x7f0ccc0b3180 2023-10-16T14:17:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.156+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ccc0d1d00 con 0x7f0cb806ae70 2023-10-16T14:17:50.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.161+0000 7f0ccaffd640 1 -- 172.21.15.29:0/2020899116 <== mgr.4100 v2:172.21.15.170:6832/1268688927 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0cbc00f040 con 0x7f0cb806ae70 2023-10-16T14:17:50.164 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:17:50.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:17:50.165 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:17:50.165 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:17:50.165 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:17:50.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.162+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0cb806ae70 msgr2=0x7f0cb806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.162+0000 7f0cd3575640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0cb806ae70 0x7f0cb806d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009f80 tx=0x7f0cbc009750 comp rx=0 tx=0).stop 2023-10-16T14:17:50.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.162+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 msgr2=0x7f0ccc0b3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.162+0000 7f0cd3575640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 0x7f0ccc0b3680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00116d0 tx=0x7f0cc0011b90 comp rx=0 tx=0).stop 2023-10-16T14:17:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 shutdown_connections 2023-10-16T14:17:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0cb806ae70 0x7f0cb806d330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ccc0b3180 0x7f0ccc0b3680 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ccc0b24b0 0x7f0ccc15f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 --2- 172.21.15.29:0/2020899116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cb4004d00 0x7f0ccc0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 >> 172.21.15.29:0/2020899116 conn(0x7f0ccc0cf570 msgr2=0x7f0ccc0b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:50.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 shutdown_connections 2023-10-16T14:17:50.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.163+0000 7f0cd3575640 1 -- 172.21.15.29:0/2020899116 wait complete. 2023-10-16T14:17:50.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:17:50.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.411+0000 7f0a61aa9640 1 Processor -- start 2023-10-16T14:17:50.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.412+0000 7f0a61aa9640 1 -- start start 2023-10-16T14:17:50.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a61aa9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a5c15ad70 0x7f0a5c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a61aa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a61aa9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0a5c0ca3b0 0x7f0a5c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a61aa9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5c0cad80 con 0x7f0a5c15ad70 2023-10-16T14:17:50.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a61aa9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5c0cb030 con 0x7f0a5c0ca3b0 2023-10-16T14:17:50.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a61aa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0c91c0 con 0x7f0a5c0c9ca0 2023-10-16T14:17:50.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.413+0000 7f0a5b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a5c15ad70 0x7f0a5c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:50.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a5c15ad70 0x7f0a5c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37404/0 (socket says 172.21.15.29:37404) 2023-10-16T14:17:50.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 -- 172.21.15.29:0/89822485 learned_addr learned my addr 172.21.15.29:0/89822485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:50.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5bfff640 1 --2- 172.21.15.29:0/89822485 >> v2:172.21.15.170:3300/0 conn(0x7f0a5c0ca3b0 0x7f0a5c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:50.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 -- 172.21.15.29:0/89822485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 msgr2=0x7f0a5c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:50.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 --2- 172.21.15.29:0/89822485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 -- 172.21.15.29:0/89822485 >> v2:172.21.15.170:3300/0 conn(0x7f0a5c0ca3b0 msgr2=0x7f0a5c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:50.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 --2- 172.21.15.29:0/89822485 >> v2:172.21.15.170:3300/0 conn(0x7f0a5c0ca3b0 0x7f0a5c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5b7fe640 1 -- 172.21.15.29:0/89822485 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c0c94a0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.414+0000 7f0a5bfff640 1 --2- 172.21.15.29:0/89822485 >> v2:172.21.15.170:3300/0 conn(0x7f0a5c0ca3b0 0x7f0a5c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:50.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.415+0000 7f0a5b7fe640 1 --2- 172.21.15.29:0/89822485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c069da0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0cf820 tx=0x7f0a4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71cf1e57626748c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:50.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.415+0000 7f0a5a7fc640 1 -- 172.21.15.29:0/89822485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c02ca80 con 0x7f0a5c15ad70 2023-10-16T14:17:50.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.415+0000 7f0a5a7fc640 1 -- 172.21.15.29:0/89822485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c02cbe0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a5a7fc640 1 -- 172.21.15.29:0/89822485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0343f0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 -- 172.21.15.29:0/89822485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 msgr2=0x7f0a5c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/89822485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c069da0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c0cf820 tx=0x7f0a4c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 -- 172.21.15.29:0/89822485 shutdown_connections 2023-10-16T14:17:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/89822485 >> v2:172.21.15.170:3300/0 conn(0x7f0a5c0ca3b0 0x7f0a5c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/89822485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/89822485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c069da0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 -- 172.21.15.29:0/89822485 >> 172.21.15.29:0/89822485 conn(0x7f0a5c0ce9b0 msgr2=0x7f0a5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:50.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 -- 172.21.15.29:0/89822485 shutdown_connections 2023-10-16T14:17:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.416+0000 7f0a61aa9640 1 -- 172.21.15.29:0/89822485 wait complete. 2023-10-16T14:17:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.418+0000 7f0a61aa9640 1 Processor -- start 2023-10-16T14:17:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.418+0000 7f0a61aa9640 1 -- start start 2023-10-16T14:17:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a61aa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c0adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a61aa9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a5c0ca3b0 0x7f0a5c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a61aa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a61aa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c159750 con 0x7f0a5c15ad70 2023-10-16T14:17:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a61aa9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0b3d60 con 0x7f0a5c0ca3b0 2023-10-16T14:17:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a61aa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0aee70 con 0x7f0a5c0c9ca0 2023-10-16T14:17:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.419+0000 7f0a5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37410/0 (socket says 172.21.15.29:37410) 2023-10-16T14:17:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 -- 172.21.15.29:0/349460697 learned_addr learned my addr 172.21.15.29:0/349460697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5affd640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a5c0ca3b0 0x7f0a5c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5b7fe640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c0adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 -- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 msgr2=0x7f0a5c0adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c0adb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 -- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a5c0ca3b0 msgr2=0x7f0a5c158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a5c0ca3b0 0x7f0a5c158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 -- 172.21.15.29:0/349460697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c009b30 con 0x7f0a5c15ad70 2023-10-16T14:17:50.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.420+0000 7f0a5bfff640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c159210 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c7f0 tx=0x7f0a5400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.421+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54007820 con 0x7f0a5c15ad70 2023-10-16T14:17:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.421+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a54007980 con 0x7f0a5c15ad70 2023-10-16T14:17:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.421+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54007d70 con 0x7f0a5c15ad70 2023-10-16T14:17:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.421+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c0b3430 con 0x7f0a5c15ad70 2023-10-16T14:17:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.421+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c0b39f0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.422+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c04e160 con 0x7f0a5c15ad70 2023-10-16T14:17:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.423+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0a540113f0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.424+0000 7f0a53fff640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0a3c06ae70 0x7f0a3c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.424+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a540898d0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.425+0000 7f0a5b7fe640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0a3c06ae70 0x7f0a3c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.426+0000 7f0a5b7fe640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0a3c06ae70 0x7f0a3c06d330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c035000 tx=0x7f0a4c030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.429+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0a540549f0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.674+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f0a5c0b3ea0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.822+0000 7f0a53fff640 1 -- 172.21.15.29:0/349460697 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0a5401f0a0 con 0x7f0a5c15ad70 2023-10-16T14:17:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0a3c06ae70 msgr2=0x7f0a3c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0a3c06ae70 0x7f0a3c06d330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c035000 tx=0x7f0a4c030000 comp rx=0 tx=0).stop 2023-10-16T14:17:50.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 msgr2=0x7f0a5c159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:50.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c159210 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c7f0 tx=0x7f0a5400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 shutdown_connections 2023-10-16T14:17:50.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f0a3c06ae70 0x7f0a3c06d330 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a5c15ad70 0x7f0a5c159210 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a5c0ca3b0 0x7f0a5c158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 --2- 172.21.15.29:0/349460697 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0a5c0c9ca0 0x7f0a5c0adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.825+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 >> 172.21.15.29:0/349460697 conn(0x7f0a5c0ce9b0 msgr2=0x7f0a5c0c2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.826+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 shutdown_connections 2023-10-16T14:17:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:50.826+0000 7f0a61aa9640 1 -- 172.21.15.29:0/349460697 wait complete. 2023-10-16T14:17:50.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:17:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.056+0000 7fdae438a640 1 Processor -- start 2023-10-16T14:17:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.056+0000 7fdae438a640 1 -- start start 2023-10-16T14:17:51.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.058+0000 7fdae438a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdadc070c10 0x7fdadc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.058+0000 7fdae438a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc161740 con 0x7fdadc0cf5f0 2023-10-16T14:17:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.058+0000 7fdae438a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc06b0f0 con 0x7fdadc070830 2023-10-16T14:17:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.058+0000 7fdae438a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdadc1619b0 con 0x7fdadc070c10 2023-10-16T14:17:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae1925640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdadc0cf5f0 0x7fdadc0cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54644/0 (socket says 172.21.15.29:54644) 2023-10-16T14:17:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae1925640 1 -- 172.21.15.29:0/2028776100 learned_addr learned my addr 172.21.15.29:0/2028776100 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae2927640 1 --2- 172.21.15.29:0/2028776100 >> v2:172.21.15.29:3300/0 conn(0x7fdadc070c10 0x7fdadc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae1124640 1 -- 172.21.15.29:0/2028776100 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628818747 0 0) 0x7fdadc161740 con 0x7fdadc0cf5f0 2023-10-16T14:17:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae1124640 1 -- 172.21.15.29:0/2028776100 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac8003520 con 0x7fdadc0cf5f0 2023-10-16T14:17:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae2927640 1 -- 172.21.15.29:0/2028776100 >> v1:172.21.15.170:6789/0 conn(0x7fdadc070830 legacy=0x7fdadc06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae2927640 1 -- 172.21.15.29:0/2028776100 >> v1:172.21.15.29:6789/0 conn(0x7fdadc0cf5f0 legacy=0x7fdadc0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.059+0000 7fdae2927640 1 -- 172.21.15.29:0/2028776100 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc06b3d0 con 0x7fdadc070c10 2023-10-16T14:17:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.060+0000 7fdae2927640 1 --2- 172.21.15.29:0/2028776100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc06d3c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009d90 tx=0x7fdad402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d37f907969e7a88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.061+0000 7fdae1124640 1 -- 172.21.15.29:0/2028776100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad402cbd0 con 0x7fdadc070c10 2023-10-16T14:17:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.061+0000 7fdae1124640 1 -- 172.21.15.29:0/2028776100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad402cd30 con 0x7fdadc070c10 2023-10-16T14:17:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.061+0000 7fdae1124640 1 -- 172.21.15.29:0/2028776100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad402cea0 con 0x7fdadc070c10 2023-10-16T14:17:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.061+0000 7fdae438a640 1 -- 172.21.15.29:0/2028776100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 msgr2=0x7fdadc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.061+0000 7fdae438a640 1 --2- 172.21.15.29:0/2028776100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc06d3c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009d90 tx=0x7fdad402be20 comp rx=0 tx=0).stop 2023-10-16T14:17:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.062+0000 7fdae438a640 1 -- 172.21.15.29:0/2028776100 shutdown_connections 2023-10-16T14:17:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.062+0000 7fdae438a640 1 --2- 172.21.15.29:0/2028776100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc06d3c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.062+0000 7fdae438a640 1 -- 172.21.15.29:0/2028776100 >> 172.21.15.29:0/2028776100 conn(0x7fdadc0cfac0 msgr2=0x7fdadc0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.062+0000 7fdae438a640 1 -- 172.21.15.29:0/2028776100 shutdown_connections 2023-10-16T14:17:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.062+0000 7fdae438a640 1 -- 172.21.15.29:0/2028776100 wait complete. 2023-10-16T14:17:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.063+0000 7fdae438a640 1 Processor -- start 2023-10-16T14:17:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.064+0000 7fdae438a640 1 -- start start 2023-10-16T14:17:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.064+0000 7fdae438a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdadc070830 0x7fdadc1f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae438a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae438a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdadc0cf5f0 0x7fdadc0b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae438a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc0afa60 con 0x7fdadc070c10 2023-10-16T14:17:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae438a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc0b08a0 con 0x7fdadc0cf5f0 2023-10-16T14:17:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae438a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc0afc90 con 0x7fdadc070830 2023-10-16T14:17:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc0af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37440/0 (socket says 172.21.15.29:37440) 2023-10-16T14:17:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 -- 172.21.15.29:0/361745810 learned_addr learned my addr 172.21.15.29:0/361745810 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:51.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae2126640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdadc070830 0x7fdadc1f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae2927640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdadc0cf5f0 0x7fdadc0b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 -- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdadc070830 msgr2=0x7fdadc1f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdadc070830 0x7fdadc1f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 -- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdadc0cf5f0 msgr2=0x7fdadc0b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdadc0cf5f0 0x7fdadc0b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae1925640 1 -- 172.21.15.29:0/361745810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad4009a70 con 0x7fdadc070c10 2023-10-16T14:17:51.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae2927640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdadc0cf5f0 0x7fdadc0b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:51.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.065+0000 7fdae2126640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdadc070830 0x7fdadc1f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:51.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.066+0000 7fdae1925640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc0af520 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00c410 tx=0x7fdacc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.066+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc005ce0 con 0x7fdadc070c10 2023-10-16T14:17:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.066+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdacc005e40 con 0x7fdadc070c10 2023-10-16T14:17:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.066+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc0072f0 con 0x7fdadc070c10 2023-10-16T14:17:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.066+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc0aff70 con 0x7fdadc070c10 2023-10-16T14:17:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.066+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdadc0b0530 con 0x7fdadc070c10 2023-10-16T14:17:51.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.068+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc04e160 con 0x7fdadc070c10 2023-10-16T14:17:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.068+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fdacc007510 con 0x7fdadc070c10 2023-10-16T14:17:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.069+0000 7fdad2ffd640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 0x7fdac006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.069+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdacc088c60 con 0x7fdadc070c10 2023-10-16T14:17:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.069+0000 7fdae2126640 1 -- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 msgr2=0x7fdac006d3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.069+0000 7fdae2126640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 0x7fdac006d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:17:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.074+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdacc053eb0 con 0x7fdadc070c10 2023-10-16T14:17:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.269+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fdadc06d080 con 0x7fdadc070c10 2023-10-16T14:17:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.271+0000 7fdae2126640 1 -- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 msgr2=0x7fdac006d3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.271+0000 7fdae2126640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 0x7fdac006d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:17:51.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.275+0000 7fdad2ffd640 1 -- 172.21.15.29:0/361745810 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdacc0170d0 con 0x7fdadc070c10 2023-10-16T14:17:51.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.278+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 msgr2=0x7fdac006d3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:17:51.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.278+0000 7fdae438a640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 0x7fdac006d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.278+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 msgr2=0x7fdadc0af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.278+0000 7fdae438a640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc0af520 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00c410 tx=0x7fdacc00c8d0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 shutdown_connections 2023-10-16T14:17:51.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7fdac006af10 0x7fdac006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc070c10 0x7fdadc0af520 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdadc0cf5f0 0x7fdadc0b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 --2- 172.21.15.29:0/361745810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdadc070830 0x7fdadc1f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 >> 172.21.15.29:0/361745810 conn(0x7fdadc0cfac0 msgr2=0x7fdadc0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:51.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 shutdown_connections 2023-10-16T14:17:51.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.279+0000 7fdae438a640 1 -- 172.21.15.29:0/361745810 wait complete. 2023-10-16T14:17:51.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:17:51.358 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:51.356+0000 7f6017a4a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:17:51.390 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:51.386+0000 7f3842841040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:17:51.410 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:51.408+0000 7f6017a4a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:17:51.446 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:51.442+0000 7f3842841040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:17:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.518+0000 7f5cbe7fc640 1 Processor -- start 2023-10-16T14:17:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.518+0000 7f5cbe7fc640 1 -- start start 2023-10-16T14:17:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.520+0000 7f5cbe7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cb80cd970 0x7f5cb80d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.520+0000 7f5cbe7fc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80b5d90 con 0x7f5cb815d410 2023-10-16T14:17:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.520+0000 7f5cbe7fc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb8160fc0 con 0x7f5cb815d030 2023-10-16T14:17:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.520+0000 7f5cbe7fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80d2b50 con 0x7f5cb80cd970 2023-10-16T14:17:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.520+0000 7f5caffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5cb815d410 0x7f5cb80cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54658/0 (socket says 172.21.15.29:54658) 2023-10-16T14:17:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.520+0000 7f5caffff640 1 -- 172.21.15.29:0/1437179519 learned_addr learned my addr 172.21.15.29:0/1437179519 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cbcd99640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cb80cd970 0x7f5cb80d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644635416 0 0) 0x7f5cb80b5d90 con 0x7f5cb815d410 2023-10-16T14:17:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca4003520 con 0x7f5cb815d410 2023-10-16T14:17:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436798257 0 0) 0x7f5ca4003520 con 0x7f5cb815d410 2023-10-16T14:17:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cb80cd970 msgr2=0x7f5cb80d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cb80cd970 0x7f5cb80d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 >> v1:172.21.15.170:6789/0 conn(0x7f5cb815d030 legacy=0x7f5cb8161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb80d25d0 con 0x7f5cb815d410 2023-10-16T14:17:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5ca0001670 con 0x7f5cb815d410 2023-10-16T14:17:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 >> v1:172.21.15.29:6789/0 conn(0x7f5cb815d410 legacy=0x7f5cb80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ca40039a0 0x7f5ca4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca40047c0 0x7f5ca4004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ca4005140 0x7f5ca400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca400bf50 con 0x7f5ca4005140 2023-10-16T14:17:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca400c090 con 0x7f5ca40039a0 2023-10-16T14:17:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.521+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca400c1d0 con 0x7f5ca40047c0 2023-10-16T14:17:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cbcd99640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ca4005140 0x7f5ca400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cb7fff640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ca40039a0 0x7f5ca4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cbcd99640 1 -- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca40047c0 msgr2=0x7f5ca4004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5caffff640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca40047c0 0x7f5ca4004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cbcd99640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca40047c0 0x7f5ca4004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cbcd99640 1 -- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ca40039a0 msgr2=0x7f5ca4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cbcd99640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ca40039a0 0x7f5ca4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cbcd99640 1 -- 172.21.15.29:0/1437179519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4003520 con 0x7f5ca4005140 2023-10-16T14:17:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5cb7fff640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5ca40039a0 0x7f5ca4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:17:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.522+0000 7f5caffff640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca40047c0 0x7f5ca4004c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.523+0000 7f5cbcd99640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ca4005140 0x7f5ca400b980 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009f90 tx=0x7f5ca40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5a8acf47371ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.523+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb002cbc0 con 0x7f5ca4005140 2023-10-16T14:17:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.523+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb000f040 con 0x7f5ca4005140 2023-10-16T14:17:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.523+0000 7f5cb77fe640 1 -- 172.21.15.29:0/1437179519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0034440 con 0x7f5ca4005140 2023-10-16T14:17:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/1437179519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ca4005140 msgr2=0x7f5ca400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/1437179519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ca4005140 0x7f5ca400b980 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009f90 tx=0x7f5ca40057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbcd99640 1 -- 172.21.15.29:0/1437179519 reap_dead start 2023-10-16T14:17:51.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/1437179519 shutdown_connections 2023-10-16T14:17:51.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/1437179519 >> 172.21.15.29:0/1437179519 conn(0x7f5cb80cf720 msgr2=0x7f5cb80cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:51.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/1437179519 shutdown_connections 2023-10-16T14:17:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.524+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/1437179519 wait complete. 2023-10-16T14:17:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.526+0000 7f5cbe7fc640 1 Processor -- start 2023-10-16T14:17:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.526+0000 7f5cbe7fc640 1 -- start start 2023-10-16T14:17:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5cbe7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca4005140 0x7f5cb80cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5cbe7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 0x7f5cb81f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5cbe7fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cb81f6b50 0x7f5cb80c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5cbe7fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80c6870 con 0x7f5cb81f61d0 2023-10-16T14:17:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5cbe7fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb815c5f0 con 0x7f5cb81f6b50 2023-10-16T14:17:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5cbe7fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb81f6fb0 con 0x7f5ca4005140 2023-10-16T14:17:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5caffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 0x7f5cb81f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5caffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 0x7f5cb81f6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37458/0 (socket says 172.21.15.29:37458) 2023-10-16T14:17:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.527+0000 7f5caffff640 1 -- 172.21.15.29:0/4235838376 learned_addr learned my addr 172.21.15.29:0/4235838376 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5caffff640 1 -- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca4005140 msgr2=0x7f5cb80cd4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:17:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5caffff640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca4005140 0x7f5cb80cd4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5caffff640 1 -- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cb81f6b50 msgr2=0x7f5cb80c6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5caffff640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cb81f6b50 0x7f5cb80c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5caffff640 1 -- 172.21.15.29:0/4235838376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb815fe10 con 0x7f5cb81f61d0 2023-10-16T14:17:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5caffff640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 0x7f5cb81f6610 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000d840 tx=0x7f5ca000dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca00074e0 con 0x7f5cb81f61d0 2023-10-16T14:17:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca0007640 con 0x7f5cb81f61d0 2023-10-16T14:17:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca00049e0 con 0x7f5cb81f61d0 2023-10-16T14:17:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb81600f0 con 0x7f5cb81f61d0 2023-10-16T14:17:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.528+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb81606b0 con 0x7f5cb81f61d0 2023-10-16T14:17:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.530+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb80c6630 con 0x7f5cb81f61d0 2023-10-16T14:17:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.531+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f5ca0004b40 con 0x7f5cb81f61d0 2023-10-16T14:17:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.531+0000 7f5cb57fa640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 0x7f5c9806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.531+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ca0089050 con 0x7f5cb81f61d0 2023-10-16T14:17:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.532+0000 7f5cb7fff640 1 -- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 msgr2=0x7f5c9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.532+0000 7f5cb7fff640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 0x7f5c9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:17:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.535+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5ca0054070 con 0x7f5cb81f61d0 2023-10-16T14:17:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.730+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5cb804e160 con 0x7f5cb81f61d0 2023-10-16T14:17:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.733+0000 7f5cb7fff640 1 -- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 msgr2=0x7f5c9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.733+0000 7f5cb7fff640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 0x7f5c9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:17:51.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.736+0000 7f5cb57fa640 1 -- 172.21.15.29:0/4235838376 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5ca0019020 con 0x7f5cb81f61d0 2023-10-16T14:17:51.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 msgr2=0x7f5c9806d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:17:51.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 0x7f5c9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 msgr2=0x7f5cb81f6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 0x7f5cb81f6610 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000d840 tx=0x7f5ca000dd00 comp rx=0 tx=0).stop 2023-10-16T14:17:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 shutdown_connections 2023-10-16T14:17:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f5c9806aec0 0x7f5c9806d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cb81f61d0 0x7f5cb81f6610 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cb81f6b50 0x7f5cb80c6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 --2- 172.21.15.29:0/4235838376 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5ca4005140 0x7f5cb80cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:51.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 >> 172.21.15.29:0/4235838376 conn(0x7f5cb80cf720 msgr2=0x7f5cb80cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:51.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 shutdown_connections 2023-10-16T14:17:51.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:51.739+0000 7f5cbe7fc640 1 -- 172.21.15.29:0/4235838376 wait complete. 2023-10-16T14:17:51.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:51.762 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:17:51.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:51.820 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-16T14:17:51.861 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:51.858+0000 7f6017a4a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:17:51.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-10-16T14:17:51.913 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:51.910+0000 7f6017a4a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:17:51.930 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:51.926+0000 7f3842841040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:17:51.963 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:51.960+0000 7f6017a4a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:17:51.986 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:51.982+0000 7f3842841040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:17:52.023 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.020+0000 7f6017a4a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:17:52.039 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.036+0000 7f3842841040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:17:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.066+0000 7f4d11b39640 1 Processor -- start 2023-10-16T14:17:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.067+0000 7f4d11b39640 1 -- start start 2023-10-16T14:17:52.071 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.069+0000 7f6017a4a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:17:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.068+0000 7f4d11b39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 0x7f4d0c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.068+0000 7f4d11b39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4d0c0ae260 0x7f4d0c06a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.068+0000 7f4d11b39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0c0ce800 con 0x7f4d0c0ce350 2023-10-16T14:17:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.068+0000 7f4d11b39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0c0cdec0 con 0x7f4d0c0ae260 2023-10-16T14:17:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.068+0000 7f4d11b39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c0acdd0 con 0x7f4d0c160ad0 2023-10-16T14:17:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.069+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 0x7f4d0c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.069+0000 7f4d0bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4d0c0ae260 0x7f4d0c06a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.069+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 0x7f4d0c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:53856/0 (socket says 172.21.15.29:53856) 2023-10-16T14:17:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.069+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/630140075 learned_addr learned my addr 172.21.15.29:0/630140075 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.070+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/630140075 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c0ae260 msgr2=0x7f4d0c06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.070+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/630140075 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c0ae260 0x7f4d0c06a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.070+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/630140075 >> v1:172.21.15.170:6789/0 conn(0x7f4d0c0ce350 legacy=0x7f4d0c160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.070+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/630140075 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c0ae740 con 0x7f4d0c160ad0 2023-10-16T14:17:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.070+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/630140075 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c0ae260 0x7f4d0c06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:17:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.071+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/630140075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 0x7f4d0c0ac890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc009870 tx=0x7f4cfc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88706054f1c9af11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.071+0000 7f4d0affd640 1 -- 172.21.15.29:0/630140075 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc030840 con 0x7f4d0c160ad0 2023-10-16T14:17:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.071+0000 7f4d0affd640 1 -- 172.21.15.29:0/630140075 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cfc0309a0 con 0x7f4d0c160ad0 2023-10-16T14:17:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.071+0000 7f4d0affd640 1 -- 172.21.15.29:0/630140075 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc030d30 con 0x7f4d0c160ad0 2023-10-16T14:17:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 -- 172.21.15.29:0/630140075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 msgr2=0x7f4d0c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 --2- 172.21.15.29:0/630140075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 0x7f4d0c0ac890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc009870 tx=0x7f4cfc02b980 comp rx=0 tx=0).stop 2023-10-16T14:17:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 -- 172.21.15.29:0/630140075 shutdown_connections 2023-10-16T14:17:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 --2- 172.21.15.29:0/630140075 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c0ae260 0x7f4d0c06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 --2- 172.21.15.29:0/630140075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c160ad0 0x7f4d0c0ac890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 -- 172.21.15.29:0/630140075 >> 172.21.15.29:0/630140075 conn(0x7f4d0c06bc10 msgr2=0x7f4d0c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.072+0000 7f4d11b39640 1 -- 172.21.15.29:0/630140075 shutdown_connections 2023-10-16T14:17:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.073+0000 7f4d11b39640 1 -- 172.21.15.29:0/630140075 wait complete. 2023-10-16T14:17:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.075+0000 7f4d11b39640 1 Processor -- start 2023-10-16T14:17:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.076+0000 7f4d11b39640 1 -- start start 2023-10-16T14:17:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d11b39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ae260 0x7f4d0c158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d11b39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ce350 0x7f4d0c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d11b39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c160ad0 0x7f4d0c070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d11b39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c0c7170 con 0x7f4d0c160ad0 2023-10-16T14:17:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d11b39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c158fb0 con 0x7f4d0c0ce350 2023-10-16T14:17:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d11b39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c0b2f00 con 0x7f4d0c0ae260 2023-10-16T14:17:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c160ad0 0x7f4d0c070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c160ad0 0x7f4d0c070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37460/0 (socket says 172.21.15.29:37460) 2023-10-16T14:17:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.077+0000 7f4d0bfff640 1 -- 172.21.15.29:0/3908211703 learned_addr learned my addr 172.21.15.29:0/3908211703 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d03fff640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ce350 0x7f4d0c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ae260 0x7f4d0c158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0bfff640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ae260 msgr2=0x7f4d0c158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ae260 0x7f4d0c158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0bfff640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ce350 msgr2=0x7f4d0c0b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ce350 0x7f4d0c0b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0bfff640 1 -- 172.21.15.29:0/3908211703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d04009a70 con 0x7f4d0c160ad0 2023-10-16T14:17:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.078+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c160ad0 0x7f4d0c070c90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400efc0 tx=0x7f4d0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:52.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.079+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0400cd30 con 0x7f4d0c160ad0 2023-10-16T14:17:52.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.079+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0400f040 con 0x7f4d0c160ad0 2023-10-16T14:17:52.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.079+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04003bd0 con 0x7f4d0c160ad0 2023-10-16T14:17:52.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.079+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc0094d0 con 0x7f4d0c160ad0 2023-10-16T14:17:52.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.079+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c0c6560 con 0x7f4d0c160ad0 2023-10-16T14:17:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.080+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0c0ced10 con 0x7f4d0c160ad0 2023-10-16T14:17:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.081+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f4d04014480 con 0x7f4d0c160ad0 2023-10-16T14:17:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.082+0000 7f4d08ff9640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 0x7f4cec06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.082+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d0408a190 con 0x7f4d0c160ad0 2023-10-16T14:17:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.082+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 msgr2=0x7f4cec06d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.082+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 0x7f4cec06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:17:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.086+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4d0c0ced10 con 0x7f4d0c160ad0 2023-10-16T14:17:52.109 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.105+0000 7f3842841040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:17:52.161 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.157+0000 7f3842841040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:17:52.182 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.178+0000 7f6017a4a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:17:52.244 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.240+0000 7f6017a4a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:17:52.279 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.275+0000 7f3842841040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:17:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.283+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 msgr2=0x7f4cec06d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.283+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 0x7f4cec06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:17:52.293 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.289+0000 7f6017a4a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:17:52.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.300+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 --> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d0c0cea50 con 0x7f4cec06aec0 2023-10-16T14:17:52.346 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.341+0000 7f3842841040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:17:52.350 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.346+0000 7f6017a4a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:17:52.399 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.395+0000 7f3842841040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:17:52.400 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.396+0000 7f6017a4a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:17:52.458 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.454+0000 7f3842841040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:17:52.511 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.507+0000 7f3842841040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:17:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.684+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 msgr2=0x7f4cec06d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:52.684+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 0x7f4cec06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:17:52.779 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.776+0000 7f6017a4a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:17:52.913 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:52.909+0000 7f3842841040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:17:52.938 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:52.935+0000 7f6017a4a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:17:53.047 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.043+0000 7f6017a4a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:17:53.083 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.079+0000 7f3842841040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:17:53.097 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.094+0000 7f6017a4a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:17:53.198 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.194+0000 7f3842841040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:17:53.204 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.201+0000 7f6017a4a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:17:53.251 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.247+0000 7f3842841040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:17:53.311 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.307+0000 7f6017a4a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:17:53.364 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.360+0000 7f3842841040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:17:53.466 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.462+0000 7f6017a4a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:17:53.475 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.471+0000 7f3842841040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:17:53.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:53.485+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 msgr2=0x7f4cec06d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1268688927 2023-10-16T14:17:53.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:53.485+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 0x7f4cec06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:17:53.515 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.511+0000 7f6017a4a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:17:53.633 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:53.629+0000 7f6017a4a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:17:53.640 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.636+0000 7f3842841040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:17:53.692 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.688+0000 7f3842841040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:17:53.817 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:53.813+0000 7f3842841040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:17:54.459 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:54.455+0000 7f6017a4a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:17:54.724 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:54.720+0000 7f3842841040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:17:54.753 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:54.750+0000 7f6017a4a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:17:54.811 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:54.807+0000 7f6017a4a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:17:54.924 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:54.921+0000 7f6017a4a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:17:54.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:54.942+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f4d04055620 con 0x7f4d0c160ad0 2023-10-16T14:17:54.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:54.943+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 msgr2=0x7f4cec06d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:17:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:54.943+0000 7f4d08ff9640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1268688927,v1:172.21.15.170:6833/1268688927] conn(0x7f4cec06aec0 0x7f4cec06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:55.074 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:55.070+0000 7f3842841040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:17:55.144 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:55.140+0000 7f3842841040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:17:55.151 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f9751d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.151 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f9751d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.151 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f9751d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.151 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f9751d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.152 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f9751d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.152 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f944d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.152 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f944d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.152 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f944d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.152 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f944d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.153 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:17:55.147+0000 7f5f944d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:17:55.267 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:17:55.263+0000 7f3842841040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:17:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:55.955+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4d04060cd0 con 0x7f4d0c160ad0 2023-10-16T14:17:55.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:55.956+0000 7f4d08ff9640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec035aa0 0x7f4cec074d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:55.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:55.956+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cec035e80 con 0x7f4cec035aa0 2023-10-16T14:17:55.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:55.958+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec035aa0 0x7f4cec074d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:55.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:55.963+0000 7f4d0b7fe640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec035aa0 0x7f4cec074d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc036000 tx=0x7f4cfc008030 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:56.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.145+0000 7f4d08ff9640 1 -- 172.21.15.29:0/3908211703 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4cec035e80 con 0x7f4cec035aa0 2023-10-16T14:17:56.150 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-16T14:17:56.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec035aa0 msgr2=0x7f4cec074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec035aa0 0x7f4cec074d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc036000 tx=0x7f4cfc008030 comp rx=0 tx=0).stop 2023-10-16T14:17:56.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c160ad0 msgr2=0x7f4d0c070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 --2- 172.21.15.29:0/3908211703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c160ad0 0x7f4d0c070c90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400efc0 tx=0x7f4d0400c490 comp rx=0 tx=0).stop 2023-10-16T14:17:56.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d0bfff640 1 -- 172.21.15.29:0/3908211703 reap_dead start 2023-10-16T14:17:56.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 shutdown_connections 2023-10-16T14:17:56.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 >> 172.21.15.29:0/3908211703 conn(0x7f4d0c06bc10 msgr2=0x7f4d0c0af290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:56.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 shutdown_connections 2023-10-16T14:17:56.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.149+0000 7f4d11b39640 1 -- 172.21.15.29:0/3908211703 wait complete. 2023-10-16T14:17:56.174 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-16T14:17:56.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:56.175 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-16T14:17:56.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-10-16T14:17:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.456+0000 7f403b3f0640 1 Processor -- start 2023-10-16T14:17:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.456+0000 7f403b3f0640 1 -- start start 2023-10-16T14:17:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.457+0000 7f403b3f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f403406c040 0x7f403406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.457+0000 7f403b3f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.457+0000 7f403b3f0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403406c960 con 0x7f40340ad970 2023-10-16T14:17:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.457+0000 7f403b3f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403415b9d0 con 0x7f403406c040 2023-10-16T14:17:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.457+0000 7f403b3f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403415bcb0 con 0x7f40340cda80 2023-10-16T14:17:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f403a3ee640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f403406c040 0x7f403406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f4039bed640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f40340ad970 0x7f403415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41798/0 (socket says 172.21.15.29:41798) 2023-10-16T14:17:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f4039bed640 1 -- 172.21.15.29:0/3928159997 learned_addr learned my addr 172.21.15.29:0/3928159997 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f403a3ee640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f403406c040 0x7f403406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57738/0 (socket says 172.21.15.29:57738) 2023-10-16T14:17:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f403abef640 1 --2- 172.21.15.29:0/3928159997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f403a3ee640 1 -- 172.21.15.29:0/3928159997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 msgr2=0x7f403415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f403a3ee640 1 --2- 172.21.15.29:0/3928159997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.458+0000 7f40393ec640 1 -- 172.21.15.29:0/3928159997 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770542480 0 0) 0x7f403406c960 con 0x7f40340ad970 2023-10-16T14:17:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.459+0000 7f403a3ee640 1 -- 172.21.15.29:0/3928159997 >> v1:172.21.15.29:6789/0 conn(0x7f40340ad970 legacy=0x7f403415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.459+0000 7f403a3ee640 1 -- 172.21.15.29:0/3928159997 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403415bf90 con 0x7f403406c040 2023-10-16T14:17:56.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.460+0000 7f403a3ee640 1 --2- 172.21.15.29:0/3928159997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f403406c420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4030009d90 tx=0x7f403002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c22266c3b93d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.460+0000 7f40393ec640 1 -- 172.21.15.29:0/3928159997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403002cc50 con 0x7f403406c040 2023-10-16T14:17:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.460+0000 7f40393ec640 1 -- 172.21.15.29:0/3928159997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403002cdb0 con 0x7f403406c040 2023-10-16T14:17:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.460+0000 7f40393ec640 1 -- 172.21.15.29:0/3928159997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030034660 con 0x7f403406c040 2023-10-16T14:17:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 -- 172.21.15.29:0/3928159997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 msgr2=0x7f403406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 --2- 172.21.15.29:0/3928159997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f403406c420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4030009d90 tx=0x7f403002bea0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 -- 172.21.15.29:0/3928159997 shutdown_connections 2023-10-16T14:17:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 --2- 172.21.15.29:0/3928159997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 --2- 172.21.15.29:0/3928159997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f403406c420 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 -- 172.21.15.29:0/3928159997 >> 172.21.15.29:0/3928159997 conn(0x7f40340ce6a0 msgr2=0x7f40340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 -- 172.21.15.29:0/3928159997 shutdown_connections 2023-10-16T14:17:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.461+0000 7f403b3f0640 1 -- 172.21.15.29:0/3928159997 wait complete. 2023-10-16T14:17:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.463+0000 7f403b3f0640 1 Processor -- start 2023-10-16T14:17:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.463+0000 7f403b3f0640 1 -- start start 2023-10-16T14:17:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403b3f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f40341509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403b3f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40340ad970 0x7f40340b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403b3f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403b3f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340c64e0 con 0x7f403406c040 2023-10-16T14:17:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403b3f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340c5910 con 0x7f40340ad970 2023-10-16T14:17:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403b3f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340c5bb0 con 0x7f40340cda80 2023-10-16T14:17:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403a3ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f40341509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403a3ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f40341509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57740/0 (socket says 172.21.15.29:57740) 2023-10-16T14:17:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.464+0000 7f403a3ee640 1 -- 172.21.15.29:0/1788308263 learned_addr learned my addr 172.21.15.29:0/1788308263 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f4039bed640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40340ad970 0x7f40340b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403a3ee640 1 -- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 msgr2=0x7f403415a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403a3ee640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403a3ee640 1 -- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40340ad970 msgr2=0x7f40340b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403a3ee640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40340ad970 0x7f40340b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403a3ee640 1 -- 172.21.15.29:0/1788308263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030009a70 con 0x7f403406c040 2023-10-16T14:17:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403a3ee640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f40341509c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4030037040 tx=0x7f4030004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030040650 con 0x7f403406c040 2023-10-16T14:17:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40300407b0 con 0x7f403406c040 2023-10-16T14:17:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030040ba0 con 0x7f403406c040 2023-10-16T14:17:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.465+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40340c5e90 con 0x7f403406c040 2023-10-16T14:17:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.466+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40340b34a0 con 0x7f403406c040 2023-10-16T14:17:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.467+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403406b330 con 0x7f403406c040 2023-10-16T14:17:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.467+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4030040d00 con 0x7f403406c040 2023-10-16T14:17:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.468+0000 7f402affd640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f401806ad90 0x7f401806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.469+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f40300abff0 con 0x7f403406c040 2023-10-16T14:17:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.469+0000 7f4039bed640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f401806ad90 0x7f401806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.470+0000 7f4039bed640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f401806ad90 0x7f401806d250 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40340c51f0 tx=0x7f4024009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.473+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f40300deca0 con 0x7f403406c040 2023-10-16T14:17:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.688+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f40340cc550 con 0x7f401806ad90 2023-10-16T14:17:56.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.854+0000 7f402affd640 1 -- 172.21.15.29:0/1788308263 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4024001350 con 0x7f401806ad90 2023-10-16T14:17:56.859 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-16T14:17:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.858+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f401806ad90 msgr2=0x7f401806d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.858+0000 7f403b3f0640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f401806ad90 0x7f401806d250 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40340c51f0 tx=0x7f4024009040 comp rx=0 tx=0).stop 2023-10-16T14:17:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.858+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 msgr2=0x7f40341509c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.858+0000 7f403b3f0640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f40341509c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4030037040 tx=0x7f4030004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 shutdown_connections 2023-10-16T14:17:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f401806ad90 0x7f401806d250 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40340cda80 0x7f403415a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40340ad970 0x7f40340b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 --2- 172.21.15.29:0/1788308263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f403406c040 0x7f40341509c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 >> 172.21.15.29:0/1788308263 conn(0x7f40340ce6a0 msgr2=0x7f403406a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 shutdown_connections 2023-10-16T14:17:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:56.859+0000 7f403b3f0640 1 -- 172.21.15.29:0/1788308263 wait complete. 2023-10-16T14:17:56.886 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-16T14:17:56.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:56.887 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-16T14:17:56.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-10-16T14:17:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.170+0000 7fc5fd70f640 1 Processor -- start 2023-10-16T14:17:57.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.170+0000 7fc5fd70f640 1 -- start start 2023-10-16T14:17:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.172+0000 7fc5fd70f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc5f80caac0 0x7fc5f80ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.172+0000 7fc5fd70f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f80cb030 con 0x7fc5f815ad70 2023-10-16T14:17:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.172+0000 7fc5fd70f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f80c91c0 con 0x7fc5f806a4b0 2023-10-16T14:17:57.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.172+0000 7fc5fd70f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f80c94a0 con 0x7fc5f80caac0 2023-10-16T14:17:57.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.172+0000 7fc5f6ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5f815ad70 0x7fc5f8069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41806/0 (socket says 172.21.15.29:41806) 2023-10-16T14:17:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.172+0000 7fc5f6ffd640 1 -- 172.21.15.29:0/2366706584 learned_addr learned my addr 172.21.15.29:0/2366706584 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f77fe640 1 --2- 172.21.15.29:0/2366706584 >> v2:172.21.15.170:3300/0 conn(0x7fc5f80caac0 0x7fc5f80ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153906987 0 0) 0x7fc5f80cb030 con 0x7fc5f815ad70 2023-10-16T14:17:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e0003520 con 0x7fc5f815ad70 2023-10-16T14:17:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848803797 0 0) 0x7fc5f80c91c0 con 0x7fc5f806a4b0 2023-10-16T14:17:57.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f80cb030 con 0x7fc5f806a4b0 2023-10-16T14:17:57.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659807627 0 0) 0x7fc5e0003520 con 0x7fc5f815ad70 2023-10-16T14:17:57.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 >> v2:172.21.15.170:3300/0 conn(0x7fc5f80caac0 msgr2=0x7fc5f80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:57.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 --2- 172.21.15.29:0/2366706584 >> v2:172.21.15.170:3300/0 conn(0x7fc5f80caac0 0x7fc5f80ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 >> v1:172.21.15.170:6789/0 conn(0x7fc5f806a4b0 legacy=0x7fc5f80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f80c9780 con 0x7fc5f815ad70 2023-10-16T14:17:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.173+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc5e80019f0 con 0x7fc5f815ad70 2023-10-16T14:17:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 >> v1:172.21.15.29:6789/0 conn(0x7fc5f815ad70 legacy=0x7fc5f8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e0003b50 0x7fc5e0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e00047c0 0x7fc5e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0005180 0x7fc5e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f80c94a0 con 0x7fc5e0003b50 2023-10-16T14:17:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e000c050 con 0x7fc5e00047c0 2023-10-16T14:17:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e000c210 con 0x7fc5e0005180 2023-10-16T14:17:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e0003b50 0x7fc5e0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f6ffd640 1 -- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0005180 msgr2=0x7fc5e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0005180 0x7fc5e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f6ffd640 1 -- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e00047c0 msgr2=0x7fc5e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.174+0000 7fc5f67fc640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e00047c0 0x7fc5e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.175+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5e00047c0 0x7fc5e0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.175+0000 7fc5f6ffd640 1 -- 172.21.15.29:0/2366706584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0003520 con 0x7fc5e0003b50 2023-10-16T14:17:57.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.175+0000 7fc5f77fe640 1 -- 172.21.15.29:0/2366706584 reap_dead start 2023-10-16T14:17:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.176+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e0003b50 0x7fc5e0003f50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800a700 tx=0x7fc5e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa72879be082c31f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.176+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8034630 con 0x7fc5e0003b50 2023-10-16T14:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.176+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e802f070 con 0x7fc5e0003b50 2023-10-16T14:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.176+0000 7fc5f5ffb640 1 -- 172.21.15.29:0/2366706584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8034a30 con 0x7fc5e0003b50 2023-10-16T14:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.176+0000 7fc5fd70f640 1 -- 172.21.15.29:0/2366706584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e0003b50 msgr2=0x7fc5e0003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.176+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e0003b50 0x7fc5e0003f50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800a700 tx=0x7fc5e00057f0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.177+0000 7fc5fd70f640 1 -- 172.21.15.29:0/2366706584 shutdown_connections 2023-10-16T14:17:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.177+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/2366706584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e0003b50 0x7fc5e0003f50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.177+0000 7fc5fd70f640 1 -- 172.21.15.29:0/2366706584 >> 172.21.15.29:0/2366706584 conn(0x7fc5f80ce9b0 msgr2=0x7fc5f80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.177+0000 7fc5fd70f640 1 -- 172.21.15.29:0/2366706584 shutdown_connections 2023-10-16T14:17:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.177+0000 7fc5fd70f640 1 -- 172.21.15.29:0/2366706584 wait complete. 2023-10-16T14:17:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 Processor -- start 2023-10-16T14:17:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 -- start start 2023-10-16T14:17:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0003b50 0x7fc5f80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 0x7fc5f80c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f80c48c0 0x7fc5f81f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8159730 con 0x7fc5f81f3070 2023-10-16T14:17:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f80c4d40 con 0x7fc5f80c48c0 2023-10-16T14:17:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5fd70f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f80c2820 con 0x7fc5e0003b50 2023-10-16T14:17:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5f67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 0x7fc5f80c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5f67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 0x7fc5f80c4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57760/0 (socket says 172.21.15.29:57760) 2023-10-16T14:17:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5f67fc640 1 -- 172.21.15.29:0/3129608540 learned_addr learned my addr 172.21.15.29:0/3129608540 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0003b50 0x7fc5f80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.180+0000 7fc5f77fe640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f80c48c0 0x7fc5f81f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5f67fc640 1 -- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0003b50 msgr2=0x7fc5f80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5f67fc640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0003b50 0x7fc5f80b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5f67fc640 1 -- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f80c48c0 msgr2=0x7fc5f81f5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5f67fc640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f80c48c0 0x7fc5f81f5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5f67fc640 1 -- 172.21.15.29:0/3129608540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f80c2b00 con 0x7fc5f81f3070 2023-10-16T14:17:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5f67fc640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 0x7fc5f80c4380 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00d540 tx=0x7fc5ec00da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec012800 con 0x7fc5f81f3070 2023-10-16T14:17:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec012960 con 0x7fc5f81f3070 2023-10-16T14:17:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec012d10 con 0x7fc5f81f3070 2023-10-16T14:17:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.181+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f80c2de0 con 0x7fc5f81f3070 2023-10-16T14:17:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.182+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f80c33a0 con 0x7fc5f81f3070 2023-10-16T14:17:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.183+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f804e160 con 0x7fc5f81f3070 2023-10-16T14:17:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.184+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fc5ec019360 con 0x7fc5f81f3070 2023-10-16T14:17:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.185+0000 7fc5df7fe640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc5e406afe0 0x7fc5e406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.185+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc5ec089970 con 0x7fc5f81f3070 2023-10-16T14:17:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.185+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc5e406afe0 0x7fc5e406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.187+0000 7fc5f6ffd640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc5e406afe0 0x7fc5e406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8034000 tx=0x7fc5e802f000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.189+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5ec054790 con 0x7fc5f81f3070 2023-10-16T14:17:57.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.408+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5f80c9ca0 con 0x7fc5e406afe0 2023-10-16T14:17:57.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.572+0000 7fc5df7fe640 1 -- 172.21.15.29:0/3129608540 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc5e8004b60 con 0x7fc5e406afe0 2023-10-16T14:17:57.577 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-16T14:17:57.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc5e406afe0 msgr2=0x7fc5e406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc5e406afe0 0x7fc5e406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8034000 tx=0x7fc5e802f000 comp rx=0 tx=0).stop 2023-10-16T14:17:57.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 msgr2=0x7fc5f80c4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 0x7fc5f80c4380 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00d540 tx=0x7fc5ec00da00 comp rx=0 tx=0).stop 2023-10-16T14:17:57.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 shutdown_connections 2023-10-16T14:17:57.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc5e406afe0 0x7fc5e406d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5f81f3070 0x7fc5f80c4380 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5f80c48c0 0x7fc5f81f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 --2- 172.21.15.29:0/3129608540 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5e0003b50 0x7fc5f80b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 >> 172.21.15.29:0/3129608540 conn(0x7fc5f80ce9b0 msgr2=0x7fc5f80af950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:57.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 shutdown_connections 2023-10-16T14:17:57.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.577+0000 7fc5fd70f640 1 -- 172.21.15.29:0/3129608540 wait complete. 2023-10-16T14:17:57.602 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-16T14:17:57.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:57.603 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-16T14:17:57.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-10-16T14:17:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.877+0000 7f5d58c90640 1 Processor -- start 2023-10-16T14:17:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.877+0000 7f5d58c90640 1 -- start start 2023-10-16T14:17:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.878+0000 7f5d58c90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d5415ad70 0x7f5d54069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.878+0000 7f5d58c90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d540caac0 0x7f5d5406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.878+0000 7f5d58c90640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540cb030 con 0x7f5d540c9ca0 2023-10-16T14:17:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.878+0000 7f5d58c90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540c91c0 con 0x7f5d5415ad70 2023-10-16T14:17:57.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.878+0000 7f5d58c90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540c94a0 con 0x7f5d540caac0 2023-10-16T14:17:57.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d5415ad70 0x7f5d54069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d5415ad70 0x7f5d54069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57776/0 (socket says 172.21.15.29:57776) 2023-10-16T14:17:57.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 -- 172.21.15.29:0/1324187338 learned_addr learned my addr 172.21.15.29:0/1324187338 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d52d9d640 1 --2- 172.21.15.29:0/1324187338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d540caac0 0x7f5d5406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 -- 172.21.15.29:0/1324187338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d540caac0 msgr2=0x7f5d5406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 --2- 172.21.15.29:0/1324187338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d540caac0 0x7f5d5406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 -- 172.21.15.29:0/1324187338 >> v1:172.21.15.170:6789/0 conn(0x7f5d540c9ca0 legacy=0x7f5d540ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.879+0000 7f5d5259c640 1 -- 172.21.15.29:0/1324187338 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540c9780 con 0x7f5d5415ad70 2023-10-16T14:17:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.880+0000 7f5d5259c640 1 --2- 172.21.15.29:0/1324187338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d5415ad70 0x7f5d54069da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009870 tx=0x7f5d4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf45b822176404ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.881+0000 7f5d51d9b640 1 -- 172.21.15.29:0/1324187338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44031640 con 0x7f5d5415ad70 2023-10-16T14:17:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.881+0000 7f5d51d9b640 1 -- 172.21.15.29:0/1324187338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d440317a0 con 0x7f5d5415ad70 2023-10-16T14:17:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.881+0000 7f5d51d9b640 1 -- 172.21.15.29:0/1324187338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44031b30 con 0x7f5d5415ad70 2023-10-16T14:17:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.881+0000 7f5d58c90640 1 -- 172.21.15.29:0/1324187338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d5415ad70 msgr2=0x7f5d54069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.881+0000 7f5d58c90640 1 --2- 172.21.15.29:0/1324187338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d5415ad70 0x7f5d54069da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009870 tx=0x7f5d4402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.882+0000 7f5d58c90640 1 -- 172.21.15.29:0/1324187338 shutdown_connections 2023-10-16T14:17:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.882+0000 7f5d58c90640 1 --2- 172.21.15.29:0/1324187338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d540caac0 0x7f5d5406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.882+0000 7f5d58c90640 1 --2- 172.21.15.29:0/1324187338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d5415ad70 0x7f5d54069da0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.882+0000 7f5d58c90640 1 -- 172.21.15.29:0/1324187338 >> 172.21.15.29:0/1324187338 conn(0x7f5d540ce9b0 msgr2=0x7f5d540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.882+0000 7f5d58c90640 1 -- 172.21.15.29:0/1324187338 shutdown_connections 2023-10-16T14:17:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.882+0000 7f5d58c90640 1 -- 172.21.15.29:0/1324187338 wait complete. 2023-10-16T14:17:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.884+0000 7f5d58c90640 1 Processor -- start 2023-10-16T14:17:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.884+0000 7f5d58c90640 1 -- start start 2023-10-16T14:17:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d58c90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d540c9ca0 0x7f5d540c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d58c90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 0x7f5d540aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d58c90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d5415ad70 0x7f5d541528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d58c90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5415a4f0 con 0x7f5d540caac0 2023-10-16T14:17:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d58c90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540c7970 con 0x7f5d540c9ca0 2023-10-16T14:17:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d58c90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540c7c50 con 0x7f5d5415ad70 2023-10-16T14:17:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 0x7f5d540aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 0x7f5d540aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57778/0 (socket says 172.21.15.29:57778) 2023-10-16T14:17:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 -- 172.21.15.29:0/2053982823 learned_addr learned my addr 172.21.15.29:0/2053982823 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d52d9d640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d5415ad70 0x7f5d541528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d5259c640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d540c9ca0 0x7f5d540c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 -- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d5415ad70 msgr2=0x7f5d541528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d5415ad70 0x7f5d541528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 -- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d540c9ca0 msgr2=0x7f5d540c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d540c9ca0 0x7f5d540c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.885+0000 7f5d4bfff640 1 -- 172.21.15.29:0/2053982823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d440094d0 con 0x7f5d540caac0 2023-10-16T14:17:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.886+0000 7f5d4bfff640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 0x7f5d540aeb10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00c430 tx=0x7f5d3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.886+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c011790 con 0x7f5d540caac0 2023-10-16T14:17:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.886+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c0118f0 con 0x7f5d540caac0 2023-10-16T14:17:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.886+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c011ce0 con 0x7f5d540caac0 2023-10-16T14:17:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.886+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d540c7f30 con 0x7f5d540caac0 2023-10-16T14:17:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.886+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d540c84f0 con 0x7f5d540caac0 2023-10-16T14:17:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.887+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d540ad480 con 0x7f5d540caac0 2023-10-16T14:17:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.888+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f5d3c01c460 con 0x7f5d540caac0 2023-10-16T14:17:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.889+0000 7f5d4affd640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f5d3406afe0 0x7f5d3406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.889+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5d3c0899e0 con 0x7f5d540caac0 2023-10-16T14:17:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.890+0000 7f5d5259c640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f5d3406afe0 0x7f5d3406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.893+0000 7f5d5259c640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f5d3406afe0 0x7f5d3406d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d44036000 tx=0x7f5d4403c040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:57.894+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5d3c054750 con 0x7f5d540caac0 2023-10-16T14:17:58.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.108+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d5404e160 con 0x7f5d3406afe0 2023-10-16T14:17:58.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.270+0000 7f5d4affd640 1 -- 172.21.15.29:0/2053982823 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5d5404e160 con 0x7f5d3406afe0 2023-10-16T14:17:58.274 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-16T14:17:58.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f5d3406afe0 msgr2=0x7f5d3406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f5d3406afe0 0x7f5d3406d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d44036000 tx=0x7f5d4403c040 comp rx=0 tx=0).stop 2023-10-16T14:17:58.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 msgr2=0x7f5d540aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 0x7f5d540aeb10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00c430 tx=0x7f5d3c00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 shutdown_connections 2023-10-16T14:17:58.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f5d3406afe0 0x7f5d3406d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540caac0 0x7f5d540aeb10 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d5415ad70 0x7f5d541528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 --2- 172.21.15.29:0/2053982823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d540c9ca0 0x7f5d540c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 >> 172.21.15.29:0/2053982823 conn(0x7f5d540ce9b0 msgr2=0x7f5d5406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:58.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 shutdown_connections 2023-10-16T14:17:58.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.273+0000 7f5d58c90640 1 -- 172.21.15.29:0/2053982823 wait complete. 2023-10-16T14:17:58.296 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-16T14:17:58.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:58.297 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-16T14:17:58.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-10-16T14:17:58.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.570+0000 7f6f80210640 1 Processor -- start 2023-10-16T14:17:58.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.571+0000 7f6f80210640 1 -- start start 2023-10-16T14:17:58.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.572+0000 7f6f80210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:58.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.572+0000 7f6f80210640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f780c9ca0 0x7f6f780ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:58.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.572+0000 7f6f80210640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f780cb030 con 0x7f6f7815ad70 2023-10-16T14:17:58.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.572+0000 7f6f80210640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f780cae30 con 0x7f6f780c9ca0 2023-10-16T14:17:58.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.572+0000 7f6f80210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780c91c0 con 0x7f6f7806a4b0 2023-10-16T14:17:58.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.573+0000 7f6f7e7ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f780c9ca0 0x7f6f780ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:58.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.573+0000 7f6f7dfac640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f6f7815ad70 0x7f6f78069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:34370/0 (socket says 172.21.15.29:34370) 2023-10-16T14:17:58.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.573+0000 7f6f7e7ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f780c9ca0 0x7f6f780ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57788/0 (socket says 172.21.15.29:57788) 2023-10-16T14:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.573+0000 7f6f7dfac640 1 -- 172.21.15.29:0/553478150 learned_addr learned my addr 172.21.15.29:0/553478150 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.573+0000 7f6f7d7ab640 1 --2- 172.21.15.29:0/553478150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.574+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/553478150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 msgr2=0x7f6f780ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.574+0000 7f6f7e7ad640 1 --2- 172.21.15.29:0/553478150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.574+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/553478150 >> v1:172.21.15.170:6789/0 conn(0x7f6f7815ad70 legacy=0x7f6f78069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.574+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/553478150 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f780c94a0 con 0x7f6f780c9ca0 2023-10-16T14:17:58.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.575+0000 7f6f7e7ad640 1 --2- 172.21.15.29:0/553478150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f780c9ca0 0x7f6f780ca8f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009d90 tx=0x7f6f7002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2886ef1ac35c5a07 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:58.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.575+0000 7f6f7cfaa640 1 -- 172.21.15.29:0/553478150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7002cc60 con 0x7f6f780c9ca0 2023-10-16T14:17:58.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.575+0000 7f6f7cfaa640 1 -- 172.21.15.29:0/553478150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7002cdc0 con 0x7f6f780c9ca0 2023-10-16T14:17:58.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.575+0000 7f6f7cfaa640 1 -- 172.21.15.29:0/553478150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70034660 con 0x7f6f780c9ca0 2023-10-16T14:17:58.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 -- 172.21.15.29:0/553478150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f780c9ca0 msgr2=0x7f6f780ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 --2- 172.21.15.29:0/553478150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f780c9ca0 0x7f6f780ca8f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009d90 tx=0x7f6f7002beb0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 -- 172.21.15.29:0/553478150 shutdown_connections 2023-10-16T14:17:58.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 --2- 172.21.15.29:0/553478150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f780c9ca0 0x7f6f780ca8f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 --2- 172.21.15.29:0/553478150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 -- 172.21.15.29:0/553478150 >> 172.21.15.29:0/553478150 conn(0x7f6f780ce9b0 msgr2=0x7f6f780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:58.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 -- 172.21.15.29:0/553478150 shutdown_connections 2023-10-16T14:17:58.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.576+0000 7f6f80210640 1 -- 172.21.15.29:0/553478150 wait complete. 2023-10-16T14:17:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.578+0000 7f6f80210640 1 Processor -- start 2023-10-16T14:17:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.579+0000 7f6f80210640 1 -- start start 2023-10-16T14:17:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f80210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f80210640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f780c9ca0 0x7f6f780c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f80210640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 0x7f6f780c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f80210640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780a8b20 con 0x7f6f7815ad70 2023-10-16T14:17:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f80210640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780a8e00 con 0x7f6f780c9ca0 2023-10-16T14:17:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f80210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780a90e0 con 0x7f6f7806a4b0 2023-10-16T14:17:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f7e7ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 0x7f6f780c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f7e7ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 0x7f6f780c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57802/0 (socket says 172.21.15.29:57802) 2023-10-16T14:17:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.580+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/2272495180 learned_addr learned my addr 172.21.15.29:0/2272495180 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7dfac640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7d7ab640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f780c9ca0 0x7f6f780c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 msgr2=0x7f6f780a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7e7ad640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f780c9ca0 msgr2=0x7f6f780c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7e7ad640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f780c9ca0 0x7f6f780c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7e7ad640 1 -- 172.21.15.29:0/2272495180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70009a70 con 0x7f6f7815ad70 2023-10-16T14:17:58.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.581+0000 7f6f7e7ad640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 0x7f6f780c8190 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009d90 tx=0x7f6f70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.582+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7003e610 con 0x7f6f7815ad70 2023-10-16T14:17:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.582+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7003e770 con 0x7f6f7815ad70 2023-10-16T14:17:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.582+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7003eb20 con 0x7f6f7815ad70 2023-10-16T14:17:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.582+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f780ae3a0 con 0x7f6f7815ad70 2023-10-16T14:17:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.582+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f780a9860 con 0x7f6f7815ad70 2023-10-16T14:17:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.583+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f780ae200 con 0x7f6f7815ad70 2023-10-16T14:17:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.584+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6f7003b050 con 0x7f6f7815ad70 2023-10-16T14:17:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.585+0000 7f6f6a7fc640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6f5c06b010 0x7f6f5c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.585+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f6f700abe70 con 0x7f6f7815ad70 2023-10-16T14:17:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.586+0000 7f6f7dfac640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6f5c06b010 0x7f6f5c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:58.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.587+0000 7f6f7dfac640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6f5c06b010 0x7f6f5c06d4d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f74005880 tx=0x7f6f74009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:58.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.589+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6f780ae200 con 0x7f6f7815ad70 2023-10-16T14:17:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.810+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f780c9780 con 0x7f6f5c06b010 2023-10-16T14:17:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.974+0000 7f6f6a7fc640 1 -- 172.21.15.29:0/2272495180 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6f740012a0 con 0x7f6f5c06b010 2023-10-16T14:17:58.977 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-16T14:17:58.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6f5c06b010 msgr2=0x7f6f5c06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6f5c06b010 0x7f6f5c06d4d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f74005880 tx=0x7f6f74009040 comp rx=0 tx=0).stop 2023-10-16T14:17:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 msgr2=0x7f6f780c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 0x7f6f780c8190 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009d90 tx=0x7f6f70004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 shutdown_connections 2023-10-16T14:17:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6f5c06b010 0x7f6f5c06d4d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f7815ad70 0x7f6f780c8190 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f780c9ca0 0x7f6f780c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 --2- 172.21.15.29:0/2272495180 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f7806a4b0 0x7f6f780a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 >> 172.21.15.29:0/2272495180 conn(0x7f6f780ce9b0 msgr2=0x7f6f7806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 shutdown_connections 2023-10-16T14:17:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:58.979+0000 7f6f80210640 1 -- 172.21.15.29:0/2272495180 wait complete. 2023-10-16T14:17:59.008 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-16T14:17:59.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:59.009 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-16T14:17:59.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-10-16T14:17:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.282+0000 7fb9ceef7640 1 Processor -- start 2023-10-16T14:17:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.282+0000 7fb9ceef7640 1 -- start start 2023-10-16T14:17:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.283+0000 7fb9ceef7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9c815ad70 0x7fb9c8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.283+0000 7fb9ceef7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb9c80c9ca0 0x7fb9c806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.283+0000 7fb9ceef7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80cb030 con 0x7fb9c80ca3b0 2023-10-16T14:17:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ceef7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80c91c0 con 0x7fb9c815ad70 2023-10-16T14:17:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ceef7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80c94a0 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9c815ad70 0x7fb9c8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9c815ad70 0x7fb9c8069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57812/0 (socket says 172.21.15.29:57812) 2023-10-16T14:17:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1078632760 learned_addr learned my addr 172.21.15.29:0/1078632760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9c7fff640 1 --2- 172.21.15.29:0/1078632760 >> v2:172.21.15.170:3300/0 conn(0x7fb9c80c9ca0 0x7fb9c806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1078632760 >> v2:172.21.15.170:3300/0 conn(0x7fb9c80c9ca0 msgr2=0x7fb9c806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 --2- 172.21.15.29:0/1078632760 >> v2:172.21.15.170:3300/0 conn(0x7fb9c80c9ca0 0x7fb9c806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1078632760 >> v1:172.21.15.170:6789/0 conn(0x7fb9c80ca3b0 legacy=0x7fb9c80ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:17:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1078632760 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80c9780 con 0x7fb9c815ad70 2023-10-16T14:17:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.284+0000 7fb9c7fff640 1 --2- 172.21.15.29:0/1078632760 >> v2:172.21.15.170:3300/0 conn(0x7fb9c80c9ca0 0x7fb9c806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:17:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.285+0000 7fb9ccc93640 1 --2- 172.21.15.29:0/1078632760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c815ad70 0x7fb9c8069da0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80ce870 tx=0x7fb9b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe0d27f62b6d8f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.286+0000 7fb9c77fe640 1 -- 172.21.15.29:0/1078632760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b802ca80 con 0x7fb9c815ad70 2023-10-16T14:17:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.286+0000 7fb9c77fe640 1 -- 172.21.15.29:0/1078632760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b802cbe0 con 0x7fb9c815ad70 2023-10-16T14:17:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.286+0000 7fb9c77fe640 1 -- 172.21.15.29:0/1078632760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80343f0 con 0x7fb9c815ad70 2023-10-16T14:17:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.286+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1078632760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c815ad70 msgr2=0x7fb9c8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.286+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1078632760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c815ad70 0x7fb9c8069da0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80ce870 tx=0x7fb9b802bee0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.287+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1078632760 shutdown_connections 2023-10-16T14:17:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.287+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1078632760 >> v2:172.21.15.170:3300/0 conn(0x7fb9c80c9ca0 0x7fb9c806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.287+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1078632760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c815ad70 0x7fb9c8069da0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.287+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1078632760 >> 172.21.15.29:0/1078632760 conn(0x7fb9c80ce9b0 msgr2=0x7fb9c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.287+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1078632760 shutdown_connections 2023-10-16T14:17:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.287+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1078632760 wait complete. 2023-10-16T14:17:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.288+0000 7fb9ceef7640 1 Processor -- start 2023-10-16T14:17:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.289+0000 7fb9ceef7640 1 -- start start 2023-10-16T14:17:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ceef7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 0x7fb9c815ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ceef7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9c80ca3b0 0x7fb9c8158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ceef7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9c815ad70 0x7fb9c80c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ceef7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c815d960 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ceef7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c815e330 con 0x7fb9c80ca3b0 2023-10-16T14:17:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ceef7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c815cd50 con 0x7fb9c815ad70 2023-10-16T14:17:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 0x7fb9c815ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 0x7fb9c815ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57818/0 (socket says 172.21.15.29:57818) 2023-10-16T14:17:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1721185077 learned_addr learned my addr 172.21.15.29:0/1721185077 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:17:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9c7fff640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9c80ca3b0 0x7fb9c8158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9c815ad70 msgr2=0x7fb9c80c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9c815ad70 0x7fb9c80c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9c80ca3b0 msgr2=0x7fb9c8158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9c80ca3b0 0x7fb9c8158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.290+0000 7fb9ccc93640 1 -- 172.21.15.29:0/1721185077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8009b30 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.291+0000 7fb9ccc93640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 0x7fb9c815ddf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000f80 tx=0x7fb9b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.291+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b803e450 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.291+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b803e5f0 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.291+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b803e9a0 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.291+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c815d030 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.291+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c815d5f0 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.292+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c804e160 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.293+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fb9b802cd50 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.294+0000 7fb9c57fa640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb9b406b0a0 0x7fb9b406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:17:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.294+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb9b80ac640 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.295+0000 7fb9c7fff640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb9b406b0a0 0x7fb9b406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:17:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.296+0000 7fb9c7fff640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb9b406b0a0 0x7fb9b406d560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80cf170 tx=0x7fb9bc008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:17:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.299+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb9b8077360 con 0x7fb9c80c9ca0 2023-10-16T14:17:59.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.515+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9c80ad4d0 con 0x7fb9b406b0a0 2023-10-16T14:17:59.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.679+0000 7fb9c57fa640 1 -- 172.21.15.29:0/1721185077 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb9bc00c040 con 0x7fb9b406b0a0 2023-10-16T14:17:59.683 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-16T14:17:59.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb9b406b0a0 msgr2=0x7fb9b406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:59.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb9b406b0a0 0x7fb9b406d560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80cf170 tx=0x7fb9bc008040 comp rx=0 tx=0).stop 2023-10-16T14:17:59.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 msgr2=0x7fb9c815ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:17:59.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 0x7fb9c815ddf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000f80 tx=0x7fb9b8004a40 comp rx=0 tx=0).stop 2023-10-16T14:17:59.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 shutdown_connections 2023-10-16T14:17:59.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb9b406b0a0 0x7fb9b406d560 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9c815ad70 0x7fb9c80c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9c80ca3b0 0x7fb9c8158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 --2- 172.21.15.29:0/1721185077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9c80c9ca0 0x7fb9c815ddf0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:17:59.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 >> 172.21.15.29:0/1721185077 conn(0x7fb9c80ce9b0 msgr2=0x7fb9c8070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:17:59.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 shutdown_connections 2023-10-16T14:17:59.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.683+0000 7fb9ceef7640 1 -- 172.21.15.29:0/1721185077 wait complete. 2023-10-16T14:17:59.710 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-16T14:17:59.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:17:59.711 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-16T14:17:59.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-10-16T14:17:59.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.996+0000 7f69e5547640 1 Processor -- start 2023-10-16T14:17:59.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.996+0000 7f69e5547640 1 -- start start 2023-10-16T14:18:00.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.997+0000 7f69e5547640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e006c040 0x7f69e006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.997+0000 7f69e5547640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69e00ad970 0x7f69e015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.997+0000 7f69e5547640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e006c960 con 0x7f69e015ebc0 2023-10-16T14:18:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.997+0000 7f69e5547640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e015b9d0 con 0x7f69e00ad970 2023-10-16T14:18:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.997+0000 7f69e5547640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e015bcb0 con 0x7f69e006c040 2023-10-16T14:18:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69d7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69e00ad970 0x7f69e015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69e4d46640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f69e015ebc0 0x7f69e00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41812/0 (socket says 172.21.15.29:41812) 2023-10-16T14:18:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69d7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69e00ad970 0x7f69e015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57820/0 (socket says 172.21.15.29:57820) 2023-10-16T14:18:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69e4d46640 1 -- 172.21.15.29:0/2901254393 learned_addr learned my addr 172.21.15.29:0/2901254393 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69dffff640 1 --2- 172.21.15.29:0/2901254393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e006c040 0x7f69e006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69d7fff640 1 -- 172.21.15.29:0/2901254393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e006c040 msgr2=0x7f69e006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69d7fff640 1 --2- 172.21.15.29:0/2901254393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e006c040 0x7f69e006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69d7fff640 1 -- 172.21.15.29:0/2901254393 >> v1:172.21.15.29:6789/0 conn(0x7f69e015ebc0 legacy=0x7f69e00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69d7fff640 1 -- 172.21.15.29:0/2901254393 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e015bf90 con 0x7f69e00ad970 2023-10-16T14:18:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:17:59.998+0000 7f69df7fe640 1 -- 172.21.15.29:0/2901254393 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899133618 0 0) 0x7f69e006c960 con 0x7f69e015ebc0 2023-10-16T14:18:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.000+0000 7f69d7fff640 1 --2- 172.21.15.29:0/2901254393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e00ad970 0x7f69e015e680 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69c80097d0 tx=0x7f69c802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa8c06bcf3fa2b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.000+0000 7f69df7fe640 1 -- 172.21.15.29:0/2901254393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8031800 con 0x7f69e00ad970 2023-10-16T14:18:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.000+0000 7f69df7fe640 1 -- 172.21.15.29:0/2901254393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c8031960 con 0x7f69e00ad970 2023-10-16T14:18:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.000+0000 7f69df7fe640 1 -- 172.21.15.29:0/2901254393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8031cf0 con 0x7f69e00ad970 2023-10-16T14:18:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 -- 172.21.15.29:0/2901254393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e00ad970 msgr2=0x7f69e015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 --2- 172.21.15.29:0/2901254393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e00ad970 0x7f69e015e680 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69c80097d0 tx=0x7f69c802b8a0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 -- 172.21.15.29:0/2901254393 shutdown_connections 2023-10-16T14:18:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 --2- 172.21.15.29:0/2901254393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e00ad970 0x7f69e015e680 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 --2- 172.21.15.29:0/2901254393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e006c040 0x7f69e006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 -- 172.21.15.29:0/2901254393 >> 172.21.15.29:0/2901254393 conn(0x7f69e00ce6a0 msgr2=0x7f69e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 -- 172.21.15.29:0/2901254393 shutdown_connections 2023-10-16T14:18:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.001+0000 7f69e5547640 1 -- 172.21.15.29:0/2901254393 wait complete. 2023-10-16T14:18:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.003+0000 7f69e5547640 1 Processor -- start 2023-10-16T14:18:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.003+0000 7f69e5547640 1 -- start start 2023-10-16T14:18:00.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69e5547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 0x7f69e00cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69e5547640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69e00ad970 0x7f69e00adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69e5547640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e015ebc0 0x7f69e0158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69e5547640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00ae5b0 con 0x7f69e006c040 2023-10-16T14:18:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69e5547640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00acc70 con 0x7f69e00ad970 2023-10-16T14:18:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69e5547640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00ae9f0 con 0x7f69e015ebc0 2023-10-16T14:18:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 0x7f69e00cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 0x7f69e00cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57834/0 (socket says 172.21.15.29:57834) 2023-10-16T14:18:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.004+0000 7f69dffff640 1 -- 172.21.15.29:0/3895634215 learned_addr learned my addr 172.21.15.29:0/3895634215 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dffff640 1 -- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e015ebc0 msgr2=0x7f69e0158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dffff640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e015ebc0 0x7f69e0158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69d7fff640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69e00ad970 0x7f69e00adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dffff640 1 -- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69e00ad970 msgr2=0x7f69e00adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dffff640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69e00ad970 0x7f69e00adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dffff640 1 -- 172.21.15.29:0/3895634215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8009400 con 0x7f69e006c040 2023-10-16T14:18:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dffff640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 0x7f69e00cd920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f69d000d5f0 tx=0x7f69d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00125f0 con 0x7f69e006c040 2023-10-16T14:18:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d0012750 con 0x7f69e006c040 2023-10-16T14:18:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e00aecd0 con 0x7f69e006c040 2023-10-16T14:18:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.005+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0012b00 con 0x7f69e006c040 2023-10-16T14:18:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.006+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e01f5b80 con 0x7f69e006c040 2023-10-16T14:18:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.007+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e006b330 con 0x7f69e006c040 2023-10-16T14:18:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.008+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f69d0012c60 con 0x7f69e006c040 2023-10-16T14:18:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.009+0000 7f69dd7fa640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f69c006b070 0x7f69c006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.009+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f69d0089f80 con 0x7f69e006c040 2023-10-16T14:18:00.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.009+0000 7f69d7fff640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f69c006b070 0x7f69c006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.011+0000 7f69d7fff640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f69c006b070 0x7f69c006d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803c040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:00.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.013+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69d0054ca0 con 0x7f69e006c040 2023-10-16T14:18:00.175 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:18:00.171+0000 7f5f86cbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:18:00.175 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:18:00.171+0000 7f5f86cbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:18:00.175 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:18:00.171+0000 7f5f86cbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:18:00.175 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:18:00.171+0000 7f5f86cbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:18:00.176 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:18:00.171+0000 7f5f86cbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:18:00.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.241+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f69e00cc550 con 0x7f69c006b070 2023-10-16T14:18:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.585+0000 7f69dd7fa640 1 -- 172.21.15.29:0/3895634215 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f69c8031ad0 con 0x7f69c006b070 2023-10-16T14:18:00.589 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-16T14:18:00.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f69c006b070 msgr2=0x7f69c006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f69c006b070 0x7f69c006d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803c040 comp rx=0 tx=0).stop 2023-10-16T14:18:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 msgr2=0x7f69e00cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 0x7f69e00cd920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f69d000d5f0 tx=0x7f69d000dab0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 shutdown_connections 2023-10-16T14:18:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f69c006b070 0x7f69c006d530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69e015ebc0 0x7f69e0158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69e00ad970 0x7f69e00adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 --2- 172.21.15.29:0/3895634215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69e006c040 0x7f69e00cd920 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.589+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 >> 172.21.15.29:0/3895634215 conn(0x7f69e00ce6a0 msgr2=0x7f69e015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.590+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 shutdown_connections 2023-10-16T14:18:00.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.590+0000 7f69e5547640 1 -- 172.21.15.29:0/3895634215 wait complete. 2023-10-16T14:18:00.614 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-16T14:18:00.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:18:00.615 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-16T14:18:00.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-10-16T14:18:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.897+0000 7f757f259640 1 Processor -- start 2023-10-16T14:18:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.897+0000 7f757f259640 1 -- start start 2023-10-16T14:18:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.899+0000 7f757f259640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f757806c040 0x7f757806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.899+0000 7f757f259640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75780cda80 0x7f757815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.899+0000 7f757f259640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757806c960 con 0x7f75780ad970 2023-10-16T14:18:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757f259640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757815b9d0 con 0x7f757806c040 2023-10-16T14:18:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757f259640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757815bcb0 con 0x7f75780cda80 2023-10-16T14:18:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f757806c040 0x7f757806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f757806c040 0x7f757806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57838/0 (socket says 172.21.15.29:57838) 2023-10-16T14:18:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 -- 172.21.15.29:0/2528442062 learned_addr learned my addr 172.21.15.29:0/2528442062 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 -- 172.21.15.29:0/2528442062 >> v2:172.21.15.170:3300/0 conn(0x7f75780cda80 msgr2=0x7f757815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 --2- 172.21.15.29:0/2528442062 >> v2:172.21.15.170:3300/0 conn(0x7f75780cda80 0x7f757815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 -- 172.21.15.29:0/2528442062 >> v1:172.21.15.29:6789/0 conn(0x7f75780ad970 legacy=0x7f757815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.900+0000 7f757e257640 1 -- 172.21.15.29:0/2528442062 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757815bf90 con 0x7f757806c040 2023-10-16T14:18:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.902+0000 7f757e257640 1 --2- 172.21.15.29:0/2528442062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f757806c040 0x7f757806c420 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f75780cf300 tx=0x7f757402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9c69a47b467678 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.902+0000 7f757d255640 1 -- 172.21.15.29:0/2528442062 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574030860 con 0x7f757806c040 2023-10-16T14:18:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.902+0000 7f757d255640 1 -- 172.21.15.29:0/2528442062 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75740309c0 con 0x7f757806c040 2023-10-16T14:18:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.902+0000 7f757d255640 1 -- 172.21.15.29:0/2528442062 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574030d50 con 0x7f757806c040 2023-10-16T14:18:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 -- 172.21.15.29:0/2528442062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f757806c040 msgr2=0x7f757806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 --2- 172.21.15.29:0/2528442062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f757806c040 0x7f757806c420 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f75780cf300 tx=0x7f757402b9a0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 -- 172.21.15.29:0/2528442062 shutdown_connections 2023-10-16T14:18:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 --2- 172.21.15.29:0/2528442062 >> v2:172.21.15.170:3300/0 conn(0x7f75780cda80 0x7f757815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 --2- 172.21.15.29:0/2528442062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f757806c040 0x7f757806c420 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 -- 172.21.15.29:0/2528442062 >> 172.21.15.29:0/2528442062 conn(0x7f75780ce6a0 msgr2=0x7f75780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 -- 172.21.15.29:0/2528442062 shutdown_connections 2023-10-16T14:18:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.903+0000 7f757f259640 1 -- 172.21.15.29:0/2528442062 wait complete. 2023-10-16T14:18:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.905+0000 7f757f259640 1 Processor -- start 2023-10-16T14:18:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.905+0000 7f757f259640 1 -- start start 2023-10-16T14:18:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757f259640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f757806c040 0x7f75780b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757f259640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 0x7f75780b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757f259640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75780cda80 0x7f75780ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757f259640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75780b0d70 con 0x7f75780ad970 2023-10-16T14:18:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757f259640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75780ad220 con 0x7f757806c040 2023-10-16T14:18:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757f259640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757814b570 con 0x7f75780cda80 2023-10-16T14:18:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757da56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 0x7f75780b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757da56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 0x7f75780b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57850/0 (socket says 172.21.15.29:57850) 2023-10-16T14:18:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757da56640 1 -- 172.21.15.29:0/1234597696 learned_addr learned my addr 172.21.15.29:0/1234597696 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757e257640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f757806c040 0x7f75780b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.906+0000 7f757ea58640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75780cda80 0x7f75780ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757da56640 1 -- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75780cda80 msgr2=0x7f75780ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757da56640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75780cda80 0x7f75780ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757da56640 1 -- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f757806c040 msgr2=0x7f75780b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757da56640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f757806c040 0x7f75780b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757da56640 1 -- 172.21.15.29:0/1234597696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75740094d0 con 0x7f75780ad970 2023-10-16T14:18:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757ea58640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75780cda80 0x7f75780ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f757da56640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 0x7f75780b0830 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f756800c430 tx=0x7f756800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:00.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75680052d0 con 0x7f75780ad970 2023-10-16T14:18:00.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7568005430 con 0x7f75780ad970 2023-10-16T14:18:00.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.907+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568003830 con 0x7f75780ad970 2023-10-16T14:18:00.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.908+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757814a960 con 0x7f75780ad970 2023-10-16T14:18:00.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.908+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757814af20 con 0x7f75780ad970 2023-10-16T14:18:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.909+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757814b700 con 0x7f75780ad970 2023-10-16T14:18:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.910+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f7568003990 con 0x7f75780ad970 2023-10-16T14:18:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.911+0000 7f756effd640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f755c06b070 0x7f755c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.911+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7568089120 con 0x7f75780ad970 2023-10-16T14:18:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.912+0000 7f757e257640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f755c06b070 0x7f755c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.913+0000 7f757e257640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f755c06b070 0x7f755c06d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7574036000 tx=0x7f7574008030 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:00.916+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7568053db0 con 0x7f75780ad970 2023-10-16T14:18:01.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.003+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7568054180 con 0x7f75780ad970 2023-10-16T14:18:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.165+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f75780cc550 con 0x7f755c06b070 2023-10-16T14:18:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.389+0000 7f756effd640 1 -- 172.21.15.29:0/1234597696 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7574044470 con 0x7f755c06b070 2023-10-16T14:18:01.393 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-16T14:18:01.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f755c06b070 msgr2=0x7f755c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f755c06b070 0x7f755c06d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7574036000 tx=0x7f7574008030 comp rx=0 tx=0).stop 2023-10-16T14:18:01.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 msgr2=0x7f75780b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 0x7f75780b0830 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f756800c430 tx=0x7f756800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 shutdown_connections 2023-10-16T14:18:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f755c06b070 0x7f755c06d530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75780ad970 0x7f75780b0830 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75780cda80 0x7f75780ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 --2- 172.21.15.29:0/1234597696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f757806c040 0x7f75780b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 >> 172.21.15.29:0/1234597696 conn(0x7f75780ce6a0 msgr2=0x7f757806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 shutdown_connections 2023-10-16T14:18:01.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.393+0000 7f757f259640 1 -- 172.21.15.29:0/1234597696 wait complete. 2023-10-16T14:18:01.416 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-16T14:18:01.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:18:01.417 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-16T14:18:01.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-10-16T14:18:01.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.699+0000 7f660994e640 1 Processor -- start 2023-10-16T14:18:01.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.699+0000 7f660994e640 1 -- start start 2023-10-16T14:18:01.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.700+0000 7f660994e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f660406c9e0 0x7f66040ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:01.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.700+0000 7f660994e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66040ca560 con 0x7f660406fb50 2023-10-16T14:18:01.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.700+0000 7f660994e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66040c9950 con 0x7f66040c9060 2023-10-16T14:18:01.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.700+0000 7f660994e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66040c9c30 con 0x7f660406c9e0 2023-10-16T14:18:01.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f660914d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f660406c9e0 0x7f66040ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:01.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f65fbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f660406fb50 0x7f66040cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41840/0 (socket says 172.21.15.29:41840) 2023-10-16T14:18:01.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f65fbfff640 1 -- 172.21.15.29:0/2459307697 learned_addr learned my addr 172.21.15.29:0/2459307697 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:01.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f660914d640 1 -- 172.21.15.29:0/2459307697 >> v1:172.21.15.170:6789/0 conn(0x7f66040c9060 legacy=0x7f660406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f660914d640 1 -- 172.21.15.29:0/2459307697 >> v1:172.21.15.29:6789/0 conn(0x7f660406fb50 legacy=0x7f66040cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f660914d640 1 -- 172.21.15.29:0/2459307697 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66040c9f10 con 0x7f660406c9e0 2023-10-16T14:18:01.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.701+0000 7f65fb7fe640 1 -- 172.21.15.29:0/2459307697 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396704385 0 0) 0x7f66040ca560 con 0x7f660406fb50 2023-10-16T14:18:01.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.703+0000 7f660914d640 1 --2- 172.21.15.29:0/2459307697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f660406c9e0 0x7f66040ccfb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009d90 tx=0x7f65fc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1df72cae2582563 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:01.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.703+0000 7f65fb7fe640 1 -- 172.21.15.29:0/2459307697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc02cd00 con 0x7f660406c9e0 2023-10-16T14:18:01.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.703+0000 7f65fb7fe640 1 -- 172.21.15.29:0/2459307697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65fc02ce60 con 0x7f660406c9e0 2023-10-16T14:18:01.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.703+0000 7f65fb7fe640 1 -- 172.21.15.29:0/2459307697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc034660 con 0x7f660406c9e0 2023-10-16T14:18:01.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 -- 172.21.15.29:0/2459307697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f660406c9e0 msgr2=0x7f66040ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 --2- 172.21.15.29:0/2459307697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f660406c9e0 0x7f66040ccfb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009d90 tx=0x7f65fc02bf50 comp rx=0 tx=0).stop 2023-10-16T14:18:01.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 -- 172.21.15.29:0/2459307697 shutdown_connections 2023-10-16T14:18:01.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 --2- 172.21.15.29:0/2459307697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f660406c9e0 0x7f66040ccfb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 -- 172.21.15.29:0/2459307697 >> 172.21.15.29:0/2459307697 conn(0x7f66040ce6a0 msgr2=0x7f66040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:01.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 -- 172.21.15.29:0/2459307697 shutdown_connections 2023-10-16T14:18:01.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.704+0000 7f660994e640 1 -- 172.21.15.29:0/2459307697 wait complete. 2023-10-16T14:18:01.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.706+0000 7f660994e640 1 Processor -- start 2023-10-16T14:18:01.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.706+0000 7f660994e640 1 -- start start 2023-10-16T14:18:01.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660994e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f660406c9e0 0x7f6604163c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:01.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660994e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f660406fb50 0x7f66041f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:01.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660994e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 0x7f66040c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:01.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660994e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66040b12c0 con 0x7f66040c9060 2023-10-16T14:18:01.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660994e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66040b15a0 con 0x7f660406c9e0 2023-10-16T14:18:01.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660994e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66040b1880 con 0x7f660406fb50 2023-10-16T14:18:01.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660914d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 0x7f66040c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:01.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660914d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 0x7f66040c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57876/0 (socket says 172.21.15.29:57876) 2023-10-16T14:18:01.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.707+0000 7f660914d640 1 -- 172.21.15.29:0/932717256 learned_addr learned my addr 172.21.15.29:0/932717256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:01.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f65fbfff640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f660406fb50 0x7f66041f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:01.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660894c640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f660406c9e0 0x7f6604163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:01.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660914d640 1 -- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f660406fb50 msgr2=0x7f66041f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660914d640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f660406fb50 0x7f66041f6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660914d640 1 -- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f660406c9e0 msgr2=0x7f6604163c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:01.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660914d640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f660406c9e0 0x7f6604163c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:01.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660914d640 1 -- 172.21.15.29:0/932717256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc009a70 con 0x7f66040c9060 2023-10-16T14:18:01.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660914d640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 0x7f66040c7780 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009d90 tx=0x7f65fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:01.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc003b90 con 0x7f66040c9060 2023-10-16T14:18:01.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65fc003cf0 con 0x7f66040c9060 2023-10-16T14:18:01.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66040b1b60 con 0x7f66040c9060 2023-10-16T14:18:01.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc004230 con 0x7f66040c9060 2023-10-16T14:18:01.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.708+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66040b2360 con 0x7f66040c9060 2023-10-16T14:18:01.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.710+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66040b2080 con 0x7f66040c9060 2023-10-16T14:18:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.710+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f65fc03b050 con 0x7f66040c9060 2023-10-16T14:18:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.711+0000 7f65f97fa640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65f006b100 0x7f65f006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.712+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f65fc0ac520 con 0x7f66040c9060 2023-10-16T14:18:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.712+0000 7f660894c640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65f006b100 0x7f65f006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.713+0000 7f660894c640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65f006b100 0x7f65f006d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6600000f60 tx=0x7f6600009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.716+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66040b2080 con 0x7f66040c9060 2023-10-16T14:18:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:01.932+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f66040c1f10 con 0x7f65f006b100 2023-10-16T14:18:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.106+0000 7f65f97fa640 1 -- 172.21.15.29:0/932717256 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6600001240 con 0x7f65f006b100 2023-10-16T14:18:02.109 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-16T14:18:02.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65f006b100 msgr2=0x7f65f006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65f006b100 0x7f65f006d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6600000f60 tx=0x7f6600009040 comp rx=0 tx=0).stop 2023-10-16T14:18:02.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 msgr2=0x7f66040c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 0x7f66040c7780 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009d90 tx=0x7f65fc004a40 comp rx=0 tx=0).stop 2023-10-16T14:18:02.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 shutdown_connections 2023-10-16T14:18:02.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65f006b100 0x7f65f006d5c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66040c9060 0x7f66040c7780 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f660406fb50 0x7f66041f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 --2- 172.21.15.29:0/932717256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f660406c9e0 0x7f6604163c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 >> 172.21.15.29:0/932717256 conn(0x7f66040ce6a0 msgr2=0x7f66040ad560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 shutdown_connections 2023-10-16T14:18:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.110+0000 7f660994e640 1 -- 172.21.15.29:0/932717256 wait complete. 2023-10-16T14:18:02.135 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-16T14:18:02.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:18:02.135 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-16T14:18:02.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-10-16T14:18:02.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.417+0000 7f60510d7640 1 Processor -- start 2023-10-16T14:18:02.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.417+0000 7f60510d7640 1 -- start start 2023-10-16T14:18:02.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.419+0000 7f60510d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c159950 0x7f604c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.419+0000 7f60510d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f604c070450 0x7f604c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.419+0000 7f60510d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f604c06c100 0x7f604c06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.419+0000 7f60510d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c15e490 con 0x7f604c06c100 2023-10-16T14:18:02.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.419+0000 7f60510d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c070df0 con 0x7f604c070450 2023-10-16T14:18:02.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.419+0000 7f60510d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c15aef0 con 0x7f604c159950 2023-10-16T14:18:02.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f604c06c100 0x7f604c06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:02.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f604c06c100 0x7f604c06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57878/0 (socket says 172.21.15.29:57878) 2023-10-16T14:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 -- 172.21.15.29:0/4150654617 learned_addr learned my addr 172.21.15.29:0/4150654617 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f6043fff640 1 --2- 172.21.15.29:0/4150654617 >> v2:172.21.15.170:3300/0 conn(0x7f604c070450 0x7f604c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 -- 172.21.15.29:0/4150654617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c159950 msgr2=0x7f604c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f604bfff640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c159950 0x7f604c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c159950 0x7f604c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 -- 172.21.15.29:0/4150654617 >> v2:172.21.15.170:3300/0 conn(0x7f604c070450 msgr2=0x7f604c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 --2- 172.21.15.29:0/4150654617 >> v2:172.21.15.170:3300/0 conn(0x7f604c070450 0x7f604c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f60508d6640 1 -- 172.21.15.29:0/4150654617 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c15d910 con 0x7f604c06c100 2023-10-16T14:18:02.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f6043fff640 1 --2- 172.21.15.29:0/4150654617 >> v2:172.21.15.170:3300/0 conn(0x7f604c070450 0x7f604c0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:02.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.420+0000 7f604bfff640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c159950 0x7f604c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:02.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.421+0000 7f60508d6640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c06c570 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f604c0719e0 tx=0x7f604402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0ce6187f7c7fd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.422+0000 7f604b7fe640 1 -- 172.21.15.29:0/4150654617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604402c9f0 con 0x7f604c06c100 2023-10-16T14:18:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.422+0000 7f604b7fe640 1 -- 172.21.15.29:0/4150654617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604402cb50 con 0x7f604c06c100 2023-10-16T14:18:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.422+0000 7f604b7fe640 1 -- 172.21.15.29:0/4150654617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60440343f0 con 0x7f604c06c100 2023-10-16T14:18:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 -- 172.21.15.29:0/4150654617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 msgr2=0x7f604c06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c06c570 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f604c0719e0 tx=0x7f604402be50 comp rx=0 tx=0).stop 2023-10-16T14:18:02.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 -- 172.21.15.29:0/4150654617 shutdown_connections 2023-10-16T14:18:02.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c06c570 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 --2- 172.21.15.29:0/4150654617 >> v2:172.21.15.170:3300/0 conn(0x7f604c070450 0x7f604c0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 --2- 172.21.15.29:0/4150654617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c159950 0x7f604c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 -- 172.21.15.29:0/4150654617 >> 172.21.15.29:0/4150654617 conn(0x7f604c0ce6a0 msgr2=0x7f604c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 -- 172.21.15.29:0/4150654617 shutdown_connections 2023-10-16T14:18:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.423+0000 7f60510d7640 1 -- 172.21.15.29:0/4150654617 wait complete. 2023-10-16T14:18:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.425+0000 7f60510d7640 1 Processor -- start 2023-10-16T14:18:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.425+0000 7f60510d7640 1 -- start start 2023-10-16T14:18:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f60510d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c1f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f60510d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c070450 0x7f604c14fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f60510d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604c159950 0x7f604c0c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f60510d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c1f6ae0 con 0x7f604c06c100 2023-10-16T14:18:02.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f60510d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c5e90 con 0x7f604c159950 2023-10-16T14:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f60510d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c150120 con 0x7f604c070450 2023-10-16T14:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f604bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c1f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f604bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c1f6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57880/0 (socket says 172.21.15.29:57880) 2023-10-16T14:18:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f604bfff640 1 -- 172.21.15.29:0/1379831304 learned_addr learned my addr 172.21.15.29:0/1379831304 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:02.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f604bfff640 1 -- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c070450 msgr2=0x7f604c14fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.427+0000 7f604bfff640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c070450 0x7f604c14fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f604bfff640 1 -- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604c159950 msgr2=0x7f604c0c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f60508d6640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604c159950 0x7f604c0c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:02.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f604bfff640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604c159950 0x7f604c0c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f604bfff640 1 -- 172.21.15.29:0/1379831304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044009b30 con 0x7f604c06c100 2023-10-16T14:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f604bfff640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c1f6500 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c7f0 tx=0x7f603c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c007a30 con 0x7f604c06c100 2023-10-16T14:18:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c007b90 con 0x7f604c06c100 2023-10-16T14:18:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c15b2c0 con 0x7f604c06c100 2023-10-16T14:18:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c00ce40 con 0x7f604c06c100 2023-10-16T14:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.428+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c15ce30 con 0x7f604c06c100 2023-10-16T14:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.429+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c15b090 con 0x7f604c06c100 2023-10-16T14:18:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.431+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f603c0113f0 con 0x7f604c06c100 2023-10-16T14:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.432+0000 7f60497fa640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f602c06b100 0x7f602c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.432+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f603c089e10 con 0x7f604c06c100 2023-10-16T14:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.433+0000 7f6043fff640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f602c06b100 0x7f602c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.434+0000 7f6043fff640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f602c06b100 0x7f602c06d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6034006fd0 tx=0x7f6034008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.436+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f604c15b090 con 0x7f604c06c100 2023-10-16T14:18:02.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.652+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f604c15ded0 con 0x7f602c06b100 2023-10-16T14:18:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.820+0000 7f60497fa640 1 -- 172.21.15.29:0/1379831304 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f603400ae00 con 0x7f602c06b100 2023-10-16T14:18:02.824 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-16T14:18:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f602c06b100 msgr2=0x7f602c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f602c06b100 0x7f602c06d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6034006fd0 tx=0x7f6034008040 comp rx=0 tx=0).stop 2023-10-16T14:18:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 msgr2=0x7f604c1f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c1f6500 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c7f0 tx=0x7f603c00cbf0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 shutdown_connections 2023-10-16T14:18:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f602c06b100 0x7f602c06d5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f604c159950 0x7f604c0c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f604c070450 0x7f604c14fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 --2- 172.21.15.29:0/1379831304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f604c06c100 0x7f604c1f6500 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 >> 172.21.15.29:0/1379831304 conn(0x7f604c0ce6a0 msgr2=0x7f604c0af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.824+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 shutdown_connections 2023-10-16T14:18:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:02.825+0000 7f60510d7640 1 -- 172.21.15.29:0/1379831304 wait complete. 2023-10-16T14:18:02.850 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-16T14:18:02.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-10-16T14:18:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.076+0000 7f0552075640 1 Processor -- start 2023-10-16T14:18:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.077+0000 7f0552075640 1 -- start start 2023-10-16T14:18:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.078+0000 7f0552075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 0x7f054c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.078+0000 7f0552075640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c0cb030 con 0x7f054c0c9ca0 2023-10-16T14:18:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.078+0000 7f0552075640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c0c91c0 con 0x7f054c15ad70 2023-10-16T14:18:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.078+0000 7f0552075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0c94a0 con 0x7f054c06a4b0 2023-10-16T14:18:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.078+0000 7f054bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f054c0c9ca0 0x7f054c0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41856/0 (socket says 172.21.15.29:41856) 2023-10-16T14:18:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.078+0000 7f054bfff640 1 -- 172.21.15.29:0/2813921883 learned_addr learned my addr 172.21.15.29:0/2813921883 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f0543fff640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 0x7f054c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322000020 0 0) 0x7f054c0cb030 con 0x7f054c0c9ca0 2023-10-16T14:18:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0538003520 con 0x7f054c0c9ca0 2023-10-16T14:18:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47062176 0 0) 0x7f0538003520 con 0x7f054c0c9ca0 2023-10-16T14:18:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 msgr2=0x7f054c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 0x7f054c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 >> v1:172.21.15.170:6789/0 conn(0x7f054c15ad70 legacy=0x7f054c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0c9780 con 0x7f054c0c9ca0 2023-10-16T14:18:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f05440019f0 con 0x7f054c0c9ca0 2023-10-16T14:18:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 >> v1:172.21.15.29:6789/0 conn(0x7f054c0c9ca0 legacy=0x7f054c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05380039a0 0x7f0538003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05380047c0 0x7f0538004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 0x7f0538005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0c94a0 con 0x7f05380047c0 2023-10-16T14:18:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053800ba30 con 0x7f05380039a0 2023-10-16T14:18:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.079+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053800bbf0 con 0x7f054c06a4b0 2023-10-16T14:18:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f0543fff640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05380047c0 0x7f0538004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f054b7fe640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05380039a0 0x7f0538003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f0543fff640 1 -- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 msgr2=0x7f0538005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f0543fff640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f054c06a4b0 0x7f0538005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f0543fff640 1 -- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05380039a0 msgr2=0x7f0538003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f0543fff640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05380039a0 0x7f0538003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f0543fff640 1 -- 172.21.15.29:0/2813921883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0538003520 con 0x7f05380047c0 2023-10-16T14:18:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.080+0000 7f054b7fe640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05380039a0 0x7f0538003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.081+0000 7f0543fff640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05380047c0 0x7f0538004c40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0534009a90 tx=0x7f05380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a2463a00db655d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053402c6b0 con 0x7f05380047c0 2023-10-16T14:18:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053402c810 con 0x7f05380047c0 2023-10-16T14:18:03.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f054affd640 1 -- 172.21.15.29:0/2813921883 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053402cba0 con 0x7f05380047c0 2023-10-16T14:18:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f0552075640 1 -- 172.21.15.29:0/2813921883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05380047c0 msgr2=0x7f0538004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f0552075640 1 --2- 172.21.15.29:0/2813921883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05380047c0 0x7f0538004c40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0534009a90 tx=0x7f05380057f0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f054bfff640 1 -- 172.21.15.29:0/2813921883 reap_dead start 2023-10-16T14:18:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f0552075640 1 -- 172.21.15.29:0/2813921883 shutdown_connections 2023-10-16T14:18:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f0552075640 1 -- 172.21.15.29:0/2813921883 >> 172.21.15.29:0/2813921883 conn(0x7f054c0ce9b0 msgr2=0x7f054c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.082+0000 7f0552075640 1 -- 172.21.15.29:0/2813921883 shutdown_connections 2023-10-16T14:18:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.083+0000 7f0552075640 1 -- 172.21.15.29:0/2813921883 wait complete. 2023-10-16T14:18:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.085+0000 7f0552075640 1 Processor -- start 2023-10-16T14:18:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.085+0000 7f0552075640 1 -- start start 2023-10-16T14:18:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0552075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05380047c0 0x7f054c0b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0552075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 0x7f054c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0552075640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054c0ac890 0x7f054c0acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0552075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0c6e80 con 0x7f054c0cc750 2023-10-16T14:18:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0552075640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0cd110 con 0x7f054c0ac890 2023-10-16T14:18:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0552075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0cd360 con 0x7f05380047c0 2023-10-16T14:18:03.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 0x7f054c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 0x7f054c0ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57900/0 (socket says 172.21.15.29:57900) 2023-10-16T14:18:03.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 -- 172.21.15.29:0/1507082033 learned_addr learned my addr 172.21.15.29:0/1507082033 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:03.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 -- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05380047c0 msgr2=0x7f054c0b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05380047c0 0x7f054c0b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 -- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054c0ac890 msgr2=0x7f054c0acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054c0ac890 0x7f054c0acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.086+0000 7f0543fff640 1 -- 172.21.15.29:0/1507082033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0cd640 con 0x7f054c0cc750 2023-10-16T14:18:03.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.087+0000 7f0543fff640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 0x7f054c0ccbd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0534037040 tx=0x7f0534004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:03.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.087+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534003810 con 0x7f054c0cc750 2023-10-16T14:18:03.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.087+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534003990 con 0x7f054c0cc750 2023-10-16T14:18:03.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.087+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534003d40 con 0x7f054c0cc750 2023-10-16T14:18:03.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.087+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c0cd920 con 0x7f054c0cc750 2023-10-16T14:18:03.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.087+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c0cdee0 con 0x7f054c0cc750 2023-10-16T14:18:03.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.089+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c0ce070 con 0x7f054c0cc750 2023-10-16T14:18:03.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.090+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0534034440 con 0x7f054c0cc750 2023-10-16T14:18:03.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.091+0000 7f0548ff9640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f052c06b130 0x7f052c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.091+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f05340abe10 con 0x7f054c0cc750 2023-10-16T14:18:03.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.092+0000 7f054b7fe640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f052c06b130 0x7f052c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.093+0000 7f054b7fe640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f052c06b130 0x7f052c06d5f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f053c0016d0 tx=0x7f053c009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:03.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.096+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05340af030 con 0x7f054c0cc750 2023-10-16T14:18:03.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.317+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f054c0adbe0 con 0x7f052c06b130 2023-10-16T14:18:03.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.453+0000 7f0548ff9640 1 -- 172.21.15.29:0/1507082033 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f053c0013b0 con 0x7f052c06b130 2023-10-16T14:18:03.457 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-16T14:18:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f052c06b130 msgr2=0x7f052c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f052c06b130 0x7f052c06d5f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f053c0016d0 tx=0x7f053c009040 comp rx=0 tx=0).stop 2023-10-16T14:18:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 msgr2=0x7f054c0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 0x7f054c0ccbd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0534037040 tx=0x7f0534004730 comp rx=0 tx=0).stop 2023-10-16T14:18:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 shutdown_connections 2023-10-16T14:18:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f052c06b130 0x7f052c06d5f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f054c0cc750 0x7f054c0ccbd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054c0ac890 0x7f054c0acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 --2- 172.21.15.29:0/1507082033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05380047c0 0x7f054c0b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 >> 172.21.15.29:0/1507082033 conn(0x7f054c0ce9b0 msgr2=0x7f054c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 shutdown_connections 2023-10-16T14:18:03.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.457+0000 7f0552075640 1 -- 172.21.15.29:0/1507082033 wait complete. 2023-10-16T14:18:03.482 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-16T14:18:03.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-10-16T14:18:03.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.705+0000 7fb6f811d640 1 Processor -- start 2023-10-16T14:18:03.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.705+0000 7fb6f811d640 1 -- start start 2023-10-16T14:18:03.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f811d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6f00caac0 0x7fb6f00ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f811d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f00cb030 con 0x7fb6f006a4b0 2023-10-16T14:18:03.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f811d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f00c91c0 con 0x7fb6f015ad70 2023-10-16T14:18:03.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f811d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f00c94a0 con 0x7fb6f00caac0 2023-10-16T14:18:03.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f56b8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6f006a4b0 0x7fb6f00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41872/0 (socket says 172.21.15.29:41872) 2023-10-16T14:18:03.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f56b8640 1 -- 172.21.15.29:0/2263499266 learned_addr learned my addr 172.21.15.29:0/2263499266 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:03.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.707+0000 7fb6f66ba640 1 --2- 172.21.15.29:0/2263499266 >> v2:172.21.15.29:3300/0 conn(0x7fb6f00caac0 0x7fb6f00ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.708+0000 7fb6f4eb7640 1 -- 172.21.15.29:0/2263499266 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082858052 0 0) 0x7fb6f00cb030 con 0x7fb6f006a4b0 2023-10-16T14:18:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.708+0000 7fb6f4eb7640 1 -- 172.21.15.29:0/2263499266 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc003520 con 0x7fb6f006a4b0 2023-10-16T14:18:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.708+0000 7fb6f66ba640 1 -- 172.21.15.29:0/2263499266 >> v1:172.21.15.170:6789/0 conn(0x7fb6f015ad70 legacy=0x7fb6f0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.708+0000 7fb6f66ba640 1 -- 172.21.15.29:0/2263499266 >> v1:172.21.15.29:6789/0 conn(0x7fb6f006a4b0 legacy=0x7fb6f00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.708+0000 7fb6f66ba640 1 -- 172.21.15.29:0/2263499266 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f00c9780 con 0x7fb6f00caac0 2023-10-16T14:18:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.709+0000 7fb6f66ba640 1 --2- 172.21.15.29:0/2263499266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f00caac0 0x7fb6f00ad4d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009d90 tx=0x7fb6e802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df864d2f56551400 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.709+0000 7fb6f4eb7640 1 -- 172.21.15.29:0/2263499266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e802cd00 con 0x7fb6f00caac0 2023-10-16T14:18:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f4eb7640 1 -- 172.21.15.29:0/2263499266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e802ce60 con 0x7fb6f00caac0 2023-10-16T14:18:03.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f4eb7640 1 -- 172.21.15.29:0/2263499266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8034660 con 0x7fb6f00caac0 2023-10-16T14:18:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 -- 172.21.15.29:0/2263499266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f00caac0 msgr2=0x7fb6f00ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 --2- 172.21.15.29:0/2263499266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f00caac0 0x7fb6f00ad4d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009d90 tx=0x7fb6e802bf50 comp rx=0 tx=0).stop 2023-10-16T14:18:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 -- 172.21.15.29:0/2263499266 shutdown_connections 2023-10-16T14:18:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 --2- 172.21.15.29:0/2263499266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f00caac0 0x7fb6f00ad4d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 -- 172.21.15.29:0/2263499266 >> 172.21.15.29:0/2263499266 conn(0x7fb6f00ce9b0 msgr2=0x7fb6f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 -- 172.21.15.29:0/2263499266 shutdown_connections 2023-10-16T14:18:03.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.710+0000 7fb6f811d640 1 -- 172.21.15.29:0/2263499266 wait complete. 2023-10-16T14:18:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.712+0000 7fb6f811d640 1 Processor -- start 2023-10-16T14:18:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.712+0000 7fb6f811d640 1 -- start start 2023-10-16T14:18:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.713+0000 7fb6f811d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6f006a4b0 0x7fb6f0159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.713+0000 7fb6f811d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6f00caac0 0x7fb6f00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.713+0000 7fb6f811d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 0x7fb6f00c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.713+0000 7fb6f811d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0159b50 con 0x7fb6f015ad70 2023-10-16T14:18:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.713+0000 7fb6f811d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00c8120 con 0x7fb6f006a4b0 2023-10-16T14:18:03.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.713+0000 7fb6f811d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00ad190 con 0x7fb6f00caac0 2023-10-16T14:18:03.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 0x7fb6f00c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 0x7fb6f00c4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57922/0 (socket says 172.21.15.29:57922) 2023-10-16T14:18:03.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 -- 172.21.15.29:0/3510468646 learned_addr learned my addr 172.21.15.29:0/3510468646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:03.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f56b8640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6f00caac0 0x7fb6f00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f5eb9640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6f006a4b0 0x7fb6f0159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 -- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6f00caac0 msgr2=0x7fb6f00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6f00caac0 0x7fb6f00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 -- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6f006a4b0 msgr2=0x7fb6f0159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6f006a4b0 0x7fb6f0159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f66ba640 1 -- 172.21.15.29:0/3510468646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8009a70 con 0x7fb6f015ad70 2023-10-16T14:18:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.714+0000 7fb6f5eb9640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6f006a4b0 0x7fb6f0159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.715+0000 7fb6f66ba640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 0x7fb6f00c4670 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009d90 tx=0x7fb6e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.715+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e803d600 con 0x7fb6f015ad70 2023-10-16T14:18:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.715+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e803d780 con 0x7fb6f015ad70 2023-10-16T14:18:03.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.715+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e803db30 con 0x7fb6f015ad70 2023-10-16T14:18:03.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.715+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f00ad470 con 0x7fb6f015ad70 2023-10-16T14:18:03.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.715+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f00c6950 con 0x7fb6f015ad70 2023-10-16T14:18:03.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.716+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f004e160 con 0x7fb6f015ad70 2023-10-16T14:18:03.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.717+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb6e803b040 con 0x7fb6f015ad70 2023-10-16T14:18:03.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.718+0000 7fb6e27fc640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb6d406b100 0x7fb6d406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:03.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.718+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb6e80ac180 con 0x7fb6f015ad70 2023-10-16T14:18:03.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.719+0000 7fb6f5eb9640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb6d406b100 0x7fb6d406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:03.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.720+0000 7fb6f5eb9640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb6d406b100 0x7fb6d406d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb6f01f57d0 tx=0x7fb6ec009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:03.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.724+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6e8076d30 con 0x7fb6f015ad70 2023-10-16T14:18:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:03.948+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6f00c9ca0 con 0x7fb6d406b100 2023-10-16T14:18:04.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.078+0000 7fb6e27fc640 1 -- 172.21.15.29:0/3510468646 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb6ec001340 con 0x7fb6d406b100 2023-10-16T14:18:04.082 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-16T14:18:04.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb6d406b100 msgr2=0x7fb6d406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb6d406b100 0x7fb6d406d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb6f01f57d0 tx=0x7fb6ec009040 comp rx=0 tx=0).stop 2023-10-16T14:18:04.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 msgr2=0x7fb6f00c4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 0x7fb6f00c4670 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009d90 tx=0x7fb6e8004a40 comp rx=0 tx=0).stop 2023-10-16T14:18:04.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 shutdown_connections 2023-10-16T14:18:04.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb6d406b100 0x7fb6d406d5c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6f015ad70 0x7fb6f00c4670 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6f00caac0 0x7fb6f00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 --2- 172.21.15.29:0/3510468646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6f006a4b0 0x7fb6f0159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 >> 172.21.15.29:0/3510468646 conn(0x7fb6f00ce9b0 msgr2=0x7fb6f015d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:04.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 shutdown_connections 2023-10-16T14:18:04.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.082+0000 7fb6f811d640 1 -- 172.21.15.29:0/3510468646 wait complete. 2023-10-16T14:18:04.106 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-16T14:18:04.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-10-16T14:18:04.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.332+0000 7fb8a64c3640 1 Processor -- start 2023-10-16T14:18:04.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.333+0000 7fb8a64c3640 1 -- start start 2023-10-16T14:18:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.334+0000 7fb8a64c3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 0x7fb8a00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.334+0000 7fb8a64c3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8a00d0be0 0x7fb8a00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.334+0000 7fb8a64c3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a015d150 con 0x7fb8a015cd70 2023-10-16T14:18:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.334+0000 7fb8a64c3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a00d1180 con 0x7fb8a00d0be0 2023-10-16T14:18:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.334+0000 7fb8a64c3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00d1460 con 0x7fb8a00b8480 2023-10-16T14:18:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.335+0000 7fb8a4cc0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 0x7fb8a00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.335+0000 7fb8a5cc2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8a00d0be0 0x7fb8a00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.335+0000 7fb8a4cc0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 0x7fb8a00cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:53988/0 (socket says 172.21.15.29:53988) 2023-10-16T14:18:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.335+0000 7fb8a4cc0640 1 -- 172.21.15.29:0/3373768634 learned_addr learned my addr 172.21.15.29:0/3373768634 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.336+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 msgr2=0x7fb8a00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.336+0000 7fb8a5cc2640 1 --2- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 0x7fb8a00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.336+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/3373768634 >> v1:172.21.15.170:6789/0 conn(0x7fb8a015cd70 legacy=0x7fb8a00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.336+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/3373768634 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00d1740 con 0x7fb8a00d0be0 2023-10-16T14:18:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.336+0000 7fb8a4cc0640 1 --2- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 0x7fb8a00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:18:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.337+0000 7fb8a5cc2640 1 --2- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00d0be0 0x7fb8a00cdd00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097a0 tx=0x7fb89802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11223a9777e0c985 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.338+0000 7fb897fff640 1 -- 172.21.15.29:0/3373768634 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89802d040 con 0x7fb8a00d0be0 2023-10-16T14:18:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb897fff640 1 -- 172.21.15.29:0/3373768634 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb898031810 con 0x7fb8a00d0be0 2023-10-16T14:18:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb897fff640 1 -- 172.21.15.29:0/3373768634 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898031ba0 con 0x7fb8a00d0be0 2023-10-16T14:18:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 -- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00d0be0 msgr2=0x7fb8a00cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00d0be0 0x7fb8a00cdd00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097a0 tx=0x7fb89802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 -- 172.21.15.29:0/3373768634 shutdown_connections 2023-10-16T14:18:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00d0be0 0x7fb8a00cdd00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/3373768634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00b8480 0x7fb8a00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 -- 172.21.15.29:0/3373768634 >> 172.21.15.29:0/3373768634 conn(0x7fb8a00cf570 msgr2=0x7fb8a00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 -- 172.21.15.29:0/3373768634 shutdown_connections 2023-10-16T14:18:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.339+0000 7fb8a64c3640 1 -- 172.21.15.29:0/3373768634 wait complete. 2023-10-16T14:18:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.341+0000 7fb8a64c3640 1 Processor -- start 2023-10-16T14:18:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.341+0000 7fb8a64c3640 1 -- start start 2023-10-16T14:18:04.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.342+0000 7fb8a64c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00b8480 0x7fb8a015ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.342+0000 7fb8a64c3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00d0be0 0x7fb8a00cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.342+0000 7fb8a64c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 0x7fb8a00cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.342+0000 7fb8a64c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0151820 con 0x7fb8a015cd70 2023-10-16T14:18:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.342+0000 7fb8a64c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00cb880 con 0x7fb8a00b8480 2023-10-16T14:18:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.342+0000 7fb8a64c3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00cbd10 con 0x7fb8a00d0be0 2023-10-16T14:18:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 0x7fb8a00cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 0x7fb8a00cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37702/0 (socket says 172.21.15.29:37702) 2023-10-16T14:18:04.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/796689173 learned_addr learned my addr 172.21.15.29:0/796689173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a54c1640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00b8480 0x7fb8a015ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00d0be0 msgr2=0x7fb8a00cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00d0be0 0x7fb8a00cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00b8480 msgr2=0x7fb8a015ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00b8480 0x7fb8a015ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.343+0000 7fb8a5cc2640 1 -- 172.21.15.29:0/796689173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8900094d0 con 0x7fb8a015cd70 2023-10-16T14:18:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.344+0000 7fb8a5cc2640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 0x7fb8a00cc640 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb8980098c0 tx=0x7fb8980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.344+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898031610 con 0x7fb8a015cd70 2023-10-16T14:18:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.344+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89803ace0 con 0x7fb8a015cd70 2023-10-16T14:18:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.344+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898042630 con 0x7fb8a015cd70 2023-10-16T14:18:04.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.344+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb898009400 con 0x7fb8a015cd70 2023-10-16T14:18:04.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.344+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a00caf50 con 0x7fb8a015cd70 2023-10-16T14:18:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.345+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a00cba30 con 0x7fb8a015cd70 2023-10-16T14:18:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.346+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb89804b3b0 con 0x7fb8a015cd70 2023-10-16T14:18:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.348+0000 7fb895ffb640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb88406b100 0x7fb88406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.348+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb8980abbc0 con 0x7fb8a015cd70 2023-10-16T14:18:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.348+0000 7fb8a54c1640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb88406b100 0x7fb88406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.350+0000 7fb8a54c1640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb88406b100 0x7fb88406d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0012e0 tx=0x7fb89c009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:04.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.352+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb8a00cba30 con 0x7fb8a015cd70 2023-10-16T14:18:04.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.573+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8a00d1d00 con 0x7fb88406b100 2023-10-16T14:18:04.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.702+0000 7fb895ffb640 1 -- 172.21.15.29:0/796689173 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb89c001a00 con 0x7fb88406b100 2023-10-16T14:18:04.706 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-16T14:18:04.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.706+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb88406b100 msgr2=0x7fb88406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.706+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb88406b100 0x7fb88406d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0012e0 tx=0x7fb89c009040 comp rx=0 tx=0).stop 2023-10-16T14:18:04.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.706+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 msgr2=0x7fb8a00cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.706+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 0x7fb8a00cc640 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb8980098c0 tx=0x7fb8980046c0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 shutdown_connections 2023-10-16T14:18:04.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb88406b100 0x7fb88406d5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a015cd70 0x7fb8a00cc640 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8a00d0be0 0x7fb8a00cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 --2- 172.21.15.29:0/796689173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8a00b8480 0x7fb8a015ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 >> 172.21.15.29:0/796689173 conn(0x7fb8a00cf570 msgr2=0x7fb8a00b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:04.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 shutdown_connections 2023-10-16T14:18:04.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.707+0000 7fb8a64c3640 1 -- 172.21.15.29:0/796689173 wait complete. 2023-10-16T14:18:04.730 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-16T14:18:04.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-10-16T14:18:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.958+0000 7fbea4c1a640 1 Processor -- start 2023-10-16T14:18:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.958+0000 7fbea4c1a640 1 -- start start 2023-10-16T14:18:04.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbea4c1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbea0159950 0x7fbea015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbea4c1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbea4c1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea015aef0 con 0x7fbea0070450 2023-10-16T14:18:04.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbea4c1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea0070e20 con 0x7fbea0159950 2023-10-16T14:18:04.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbea4c1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea015e490 con 0x7fbea006c810 2023-10-16T14:18:04.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbe9f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbea0159950 0x7fbea015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbe9f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbea0159950 0x7fbea015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37716/0 (socket says 172.21.15.29:37716) 2023-10-16T14:18:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbe9f7fe640 1 -- 172.21.15.29:0/315946681 learned_addr learned my addr 172.21.15.29:0/315946681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbe9f7fe640 1 -- 172.21.15.29:0/315946681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 msgr2=0x7fbea00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.960+0000 7fbe9ffff640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.961+0000 7fbe9f7fe640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea00708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.961+0000 7fbe9f7fe640 1 -- 172.21.15.29:0/315946681 >> v1:172.21.15.170:6789/0 conn(0x7fbea0070450 legacy=0x7fbea006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.961+0000 7fbe9f7fe640 1 -- 172.21.15.29:0/315946681 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea006cc60 con 0x7fbea0159950 2023-10-16T14:18:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.961+0000 7fbe9ffff640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea00708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.962+0000 7fbe9f7fe640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0159950 0x7fbea015a9b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009870 tx=0x7fbe9002da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c8aac7036684ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.962+0000 7fbe9e7fc640 1 -- 172.21.15.29:0/315946681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9002f830 con 0x7fbea0159950 2023-10-16T14:18:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.962+0000 7fbe9e7fc640 1 -- 172.21.15.29:0/315946681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9002f990 con 0x7fbea0159950 2023-10-16T14:18:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.962+0000 7fbe9e7fc640 1 -- 172.21.15.29:0/315946681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9002fd20 con 0x7fbea0159950 2023-10-16T14:18:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 -- 172.21.15.29:0/315946681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0159950 msgr2=0x7fbea015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0159950 0x7fbea015a9b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009870 tx=0x7fbe9002da70 comp rx=0 tx=0).stop 2023-10-16T14:18:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 -- 172.21.15.29:0/315946681 shutdown_connections 2023-10-16T14:18:04.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea00708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/315946681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0159950 0x7fbea015a9b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 -- 172.21.15.29:0/315946681 >> 172.21.15.29:0/315946681 conn(0x7fbea00ce6a0 msgr2=0x7fbea00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 -- 172.21.15.29:0/315946681 shutdown_connections 2023-10-16T14:18:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.963+0000 7fbea4c1a640 1 -- 172.21.15.29:0/315946681 wait complete. 2023-10-16T14:18:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.965+0000 7fbea4c1a640 1 Processor -- start 2023-10-16T14:18:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.965+0000 7fbea4c1a640 1 -- start start 2023-10-16T14:18:04.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbea4c1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbea4c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 0x7fbea00c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbea4c1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbea0159950 0x7fbea0154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbea4c1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea014c6d0 con 0x7fbea0070450 2023-10-16T14:18:04.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbea4c1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea014c9b0 con 0x7fbea0159950 2023-10-16T14:18:04.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbea4c1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea014cc90 con 0x7fbea006c810 2023-10-16T14:18:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbe9effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 0x7fbea00c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbe9effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 0x7fbea00c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37724/0 (socket says 172.21.15.29:37724) 2023-10-16T14:18:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.966+0000 7fbe9effd640 1 -- 172.21.15.29:0/2014019819 learned_addr learned my addr 172.21.15.29:0/2014019819 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9ffff640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbea0159950 0x7fbea0154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9f7fe640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9effd640 1 -- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 msgr2=0x7fbea014d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9effd640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea014d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9effd640 1 -- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbea0159950 msgr2=0x7fbea0154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:04.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9effd640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbea0159950 0x7fbea0154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:04.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9effd640 1 -- 172.21.15.29:0/2014019819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900094d0 con 0x7fbea0070450 2023-10-16T14:18:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe9effd640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 0x7fbea00c21b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c430 tx=0x7fbe9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94011790 con 0x7fbea0070450 2023-10-16T14:18:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe940118f0 con 0x7fbea0070450 2023-10-16T14:18:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94011ce0 con 0x7fbea0070450 2023-10-16T14:18:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.967+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea014cf70 con 0x7fbea0070450 2023-10-16T14:18:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.968+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea0148bc0 con 0x7fbea0070450 2023-10-16T14:18:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.969+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea014d160 con 0x7fbea0070450 2023-10-16T14:18:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.970+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbe9401c460 con 0x7fbea0070450 2023-10-16T14:18:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.971+0000 7fbe87fff640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbe8c06b130 0x7fbe8c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.971+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbe94089a80 con 0x7fbea0070450 2023-10-16T14:18:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.971+0000 7fbe9f7fe640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbe8c06b130 0x7fbe8c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.973+0000 7fbe9f7fe640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbe8c06b130 0x7fbe8c06d5f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbe90038000 tx=0x7fbe9003b040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:04.975+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe94054790 con 0x7fbea0070450 2023-10-16T14:18:05.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.190+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbea014cb20 con 0x7fbe8c06b130 2023-10-16T14:18:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.324+0000 7fbe87fff640 1 -- 172.21.15.29:0/2014019819 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fbea014cb20 con 0x7fbe8c06b130 2023-10-16T14:18:05.327 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-16T14:18:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.327+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbe8c06b130 msgr2=0x7fbe8c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.327+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbe8c06b130 0x7fbe8c06d5f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbe90038000 tx=0x7fbe9003b040 comp rx=0 tx=0).stop 2023-10-16T14:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 msgr2=0x7fbea00c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 0x7fbea00c21b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c430 tx=0x7fbe9400c8f0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 shutdown_connections 2023-10-16T14:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbe8c06b130 0x7fbe8c06d5f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbea0070450 0x7fbea00c21b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbea0159950 0x7fbea0154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 --2- 172.21.15.29:0/2014019819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbea006c810 0x7fbea014d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 >> 172.21.15.29:0/2014019819 conn(0x7fbea00ce6a0 msgr2=0x7fbea006bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 shutdown_connections 2023-10-16T14:18:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.328+0000 7fbea4c1a640 1 -- 172.21.15.29:0/2014019819 wait complete. 2023-10-16T14:18:05.352 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-16T14:18:05.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-10-16T14:18:05.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.580+0000 7efd86be2640 1 Processor -- start 2023-10-16T14:18:05.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.580+0000 7efd86be2640 1 -- start start 2023-10-16T14:18:05.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.581+0000 7efd86be2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd8006c040 0x7efd8006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:05.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.581+0000 7efd86be2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd800ad970 0x7efd8015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.581+0000 7efd86be2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8006c960 con 0x7efd8015ebc0 2023-10-16T14:18:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.581+0000 7efd86be2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8015b9d0 con 0x7efd8006c040 2023-10-16T14:18:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.581+0000 7efd86be2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8015bcb0 con 0x7efd800ad970 2023-10-16T14:18:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd8006c040 0x7efd8006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd8006c040 0x7efd8006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37740/0 (socket says 172.21.15.29:37740) 2023-10-16T14:18:05.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 -- 172.21.15.29:0/3706183355 learned_addr learned my addr 172.21.15.29:0/3706183355 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:05.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd853df640 1 --2- 172.21.15.29:0/3706183355 >> v2:172.21.15.170:3300/0 conn(0x7efd800ad970 0x7efd8015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:05.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 -- 172.21.15.29:0/3706183355 >> v2:172.21.15.170:3300/0 conn(0x7efd800ad970 msgr2=0x7efd8015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:05.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 --2- 172.21.15.29:0/3706183355 >> v2:172.21.15.170:3300/0 conn(0x7efd800ad970 0x7efd8015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 -- 172.21.15.29:0/3706183355 >> v1:172.21.15.170:6789/0 conn(0x7efd8015ebc0 legacy=0x7efd800cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:05.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd85be0640 1 -- 172.21.15.29:0/3706183355 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8015bf90 con 0x7efd8006c040 2023-10-16T14:18:05.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.582+0000 7efd853df640 1 --2- 172.21.15.29:0/3706183355 >> v2:172.21.15.170:3300/0 conn(0x7efd800ad970 0x7efd8015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:05.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.583+0000 7efd85be0640 1 --2- 172.21.15.29:0/3706183355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8006c040 0x7efd8006c420 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7efd800cb060 tx=0x7efd7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd23f2f691d4d2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:05.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.584+0000 7efd84bde640 1 -- 172.21.15.29:0/3706183355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c031870 con 0x7efd8006c040 2023-10-16T14:18:05.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.584+0000 7efd84bde640 1 -- 172.21.15.29:0/3706183355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c0319d0 con 0x7efd8006c040 2023-10-16T14:18:05.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.584+0000 7efd84bde640 1 -- 172.21.15.29:0/3706183355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c031d60 con 0x7efd8006c040 2023-10-16T14:18:05.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.584+0000 7efd86be2640 1 -- 172.21.15.29:0/3706183355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8006c040 msgr2=0x7efd8006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 --2- 172.21.15.29:0/3706183355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8006c040 0x7efd8006c420 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7efd800cb060 tx=0x7efd7c02b910 comp rx=0 tx=0).stop 2023-10-16T14:18:05.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 -- 172.21.15.29:0/3706183355 shutdown_connections 2023-10-16T14:18:05.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 --2- 172.21.15.29:0/3706183355 >> v2:172.21.15.170:3300/0 conn(0x7efd800ad970 0x7efd8015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 --2- 172.21.15.29:0/3706183355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8006c040 0x7efd8006c420 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 -- 172.21.15.29:0/3706183355 >> 172.21.15.29:0/3706183355 conn(0x7efd800ce6a0 msgr2=0x7efd800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:05.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 -- 172.21.15.29:0/3706183355 shutdown_connections 2023-10-16T14:18:05.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.585+0000 7efd86be2640 1 -- 172.21.15.29:0/3706183355 wait complete. 2023-10-16T14:18:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.587+0000 7efd86be2640 1 Processor -- start 2023-10-16T14:18:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.587+0000 7efd86be2640 1 -- start start 2023-10-16T14:18:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd86be2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8006c040 0x7efd801566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd86be2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd800ad970 0x7efd800b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd86be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 0x7efd800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd86be2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd800afe60 con 0x7efd8015ebc0 2023-10-16T14:18:05.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd86be2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd800af270 con 0x7efd800ad970 2023-10-16T14:18:05.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd86be2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd800b26b0 con 0x7efd8006c040 2023-10-16T14:18:05.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd863e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 0x7efd800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:05.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd863e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 0x7efd800b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37742/0 (socket says 172.21.15.29:37742) 2023-10-16T14:18:05.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd863e1640 1 -- 172.21.15.29:0/1721913046 learned_addr learned my addr 172.21.15.29:0/1721913046 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:05.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd85be0640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8006c040 0x7efd801566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:05.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.589+0000 7efd853df640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd800ad970 0x7efd800b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:05.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd863e1640 1 -- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8006c040 msgr2=0x7efd801566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd863e1640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8006c040 0x7efd801566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd863e1640 1 -- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd800ad970 msgr2=0x7efd800b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd863e1640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd800ad970 0x7efd800b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd863e1640 1 -- 172.21.15.29:0/1721913046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0094d0 con 0x7efd8015ebc0 2023-10-16T14:18:05.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd85be0640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8006c040 0x7efd801566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:18:05.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd863e1640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 0x7efd800b2170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efd7800c780 tx=0x7efd7800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:05.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78003d80 con 0x7efd8015ebc0 2023-10-16T14:18:05.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7800ce90 con 0x7efd8015ebc0 2023-10-16T14:18:05.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd780052b0 con 0x7efd8015ebc0 2023-10-16T14:18:05.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.590+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd800af4c0 con 0x7efd8015ebc0 2023-10-16T14:18:05.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.591+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd800afa80 con 0x7efd8015ebc0 2023-10-16T14:18:05.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.592+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8006b330 con 0x7efd8015ebc0 2023-10-16T14:18:05.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.593+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7efd78005410 con 0x7efd8015ebc0 2023-10-16T14:18:05.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.594+0000 7efd767fc640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd6406b100 0x7efd6406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:05.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.594+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7efd78088f20 con 0x7efd8015ebc0 2023-10-16T14:18:05.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.594+0000 7efd85be0640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd6406b100 0x7efd6406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:05.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.596+0000 7efd85be0640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd6406b100 0x7efd6406d5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd7c036000 tx=0x7efd7c03c040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:05.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.598+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efd78053c00 con 0x7efd8015ebc0 2023-10-16T14:18:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.815+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efd800cc550 con 0x7efd6406b100 2023-10-16T14:18:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.942+0000 7efd767fc640 1 -- 172.21.15.29:0/1721913046 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7efd7c031e20 con 0x7efd6406b100 2023-10-16T14:18:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-16T14:18:05.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd6406b100 msgr2=0x7efd6406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd6406b100 0x7efd6406d5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd7c036000 tx=0x7efd7c03c040 comp rx=0 tx=0).stop 2023-10-16T14:18:05.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 msgr2=0x7efd800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:05.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 0x7efd800b2170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efd7800c780 tx=0x7efd7800cc40 comp rx=0 tx=0).stop 2023-10-16T14:18:05.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 shutdown_connections 2023-10-16T14:18:05.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd6406b100 0x7efd6406d5c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd8015ebc0 0x7efd800b2170 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd800ad970 0x7efd800b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 --2- 172.21.15.29:0/1721913046 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd8006c040 0x7efd801566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:05.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 >> 172.21.15.29:0/1721913046 conn(0x7efd800ce6a0 msgr2=0x7efd80159c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:05.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 shutdown_connections 2023-10-16T14:18:05.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:05.946+0000 7efd86be2640 1 -- 172.21.15.29:0/1721913046 wait complete. 2023-10-16T14:18:05.972 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-16T14:18:05.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-10-16T14:18:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.196+0000 7f43b1ba5640 1 Processor -- start 2023-10-16T14:18:06.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.196+0000 7f43b1ba5640 1 -- start start 2023-10-16T14:18:06.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.197+0000 7f43b1ba5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f43ac0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.197+0000 7f43b1ba5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f43ac0d0be0 0x7f43ac0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.197+0000 7f43b1ba5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac15d150 con 0x7f43ac15cd70 2023-10-16T14:18:06.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.197+0000 7f43b1ba5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ac0d1180 con 0x7f43ac0d0be0 2023-10-16T14:18:06.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.197+0000 7f43b1ba5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d1460 con 0x7f43ac0b8480 2023-10-16T14:18:06.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.198+0000 7f43b0ba3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f43ac15cd70 0x7f43ac0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40888/0 (socket says 172.21.15.29:40888) 2023-10-16T14:18:06.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.198+0000 7f43b0ba3640 1 -- 172.21.15.29:0/2902226523 learned_addr learned my addr 172.21.15.29:0/2902226523 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:06.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.198+0000 7f43abfff640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f43ac0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.198+0000 7f43b13a4640 1 --2- 172.21.15.29:0/2902226523 >> v2:172.21.15.170:3300/0 conn(0x7f43ac0d0be0 0x7f43ac0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.198+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920484378 0 0) 0x7f43ac15d150 con 0x7f43ac15cd70 2023-10-16T14:18:06.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.198+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4398003520 con 0x7f43ac15cd70 2023-10-16T14:18:06.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264174375 0 0) 0x7f4398003520 con 0x7f43ac15cd70 2023-10-16T14:18:06.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 msgr2=0x7f43ac0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:06.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f43ac0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 >> v2:172.21.15.170:3300/0 conn(0x7f43ac0d0be0 msgr2=0x7f43ac0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:06.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 --2- 172.21.15.29:0/2902226523 >> v2:172.21.15.170:3300/0 conn(0x7f43ac0d0be0 0x7f43ac0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac0d1740 con 0x7f43ac15cd70 2023-10-16T14:18:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f439c0019f0 con 0x7f43ac15cd70 2023-10-16T14:18:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 >> v1:172.21.15.29:6789/0 conn(0x7f43ac15cd70 legacy=0x7f43ac0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f4398003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4398004d00 0x7f4398005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d1180 con 0x7f4398003ef0 2023-10-16T14:18:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d1460 con 0x7f4398004d00 2023-10-16T14:18:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439800b980 con 0x7f43ac0b8480 2023-10-16T14:18:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.199+0000 7f43abfff640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43b0ba3640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f4398003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43abfff640 1 -- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 msgr2=0x7f4398003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43abfff640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f4398003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43abfff640 1 -- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4398004d00 msgr2=0x7f4398005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43abfff640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4398004d00 0x7f4398005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43abfff640 1 -- 172.21.15.29:0/2902226523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4398003520 con 0x7f4398003ef0 2023-10-16T14:18:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.200+0000 7f43b0ba3640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b8480 0x7f4398003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.201+0000 7f43abfff640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43980047c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4394009ac0 tx=0x7f43980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267b312929c4db9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.201+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439402c6b0 con 0x7f4398003ef0 2023-10-16T14:18:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.201+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439402c810 con 0x7f4398003ef0 2023-10-16T14:18:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.201+0000 7f43ab7fe640 1 -- 172.21.15.29:0/2902226523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439402cba0 con 0x7f4398003ef0 2023-10-16T14:18:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b1ba5640 1 -- 172.21.15.29:0/2902226523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 msgr2=0x7f43980047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/2902226523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43980047c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4394009ac0 tx=0x7f43980057f0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b13a4640 1 -- 172.21.15.29:0/2902226523 reap_dead start 2023-10-16T14:18:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b1ba5640 1 -- 172.21.15.29:0/2902226523 shutdown_connections 2023-10-16T14:18:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b1ba5640 1 -- 172.21.15.29:0/2902226523 >> 172.21.15.29:0/2902226523 conn(0x7f43ac0cf570 msgr2=0x7f43ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b1ba5640 1 -- 172.21.15.29:0/2902226523 shutdown_connections 2023-10-16T14:18:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.202+0000 7f43b1ba5640 1 -- 172.21.15.29:0/2902226523 wait complete. 2023-10-16T14:18:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.204+0000 7f43b1ba5640 1 Processor -- start 2023-10-16T14:18:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.204+0000 7f43b1ba5640 1 -- start start 2023-10-16T14:18:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b1ba5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43ac0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b1ba5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b7660 0x7f43ac0b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b1ba5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac1f27e0 0x7f43ac1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b1ba5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0b8020 con 0x7f4398003ef0 2023-10-16T14:18:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b1ba5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac1f6610 con 0x7f43ac1f27e0 2023-10-16T14:18:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b1ba5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac1f5a00 con 0x7f43ac0b7660 2023-10-16T14:18:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b0ba3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43ac0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b0ba3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43ac0c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37768/0 (socket says 172.21.15.29:37768) 2023-10-16T14:18:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43b0ba3640 1 -- 172.21.15.29:0/941312503 learned_addr learned my addr 172.21.15.29:0/941312503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.205+0000 7f43abfff640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b7660 0x7f43ac0b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b0ba3640 1 -- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b7660 msgr2=0x7f43ac0b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b13a4640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac1f27e0 0x7f43ac1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b0ba3640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b7660 0x7f43ac0b7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b0ba3640 1 -- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac1f27e0 msgr2=0x7f43ac1f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b0ba3640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac1f27e0 0x7f43ac1f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b0ba3640 1 -- 172.21.15.29:0/941312503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac1f5ce0 con 0x7f4398003ef0 2023-10-16T14:18:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b0ba3640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43ac0c8c90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f439c00dfc0 tx=0x7f439c00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c004d60 con 0x7f4398003ef0 2023-10-16T14:18:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c0070d0 con 0x7f4398003ef0 2023-10-16T14:18:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c007460 con 0x7f4398003ef0 2023-10-16T14:18:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac1f5fc0 con 0x7f4398003ef0 2023-10-16T14:18:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.206+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac0c91d0 con 0x7f4398003ef0 2023-10-16T14:18:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.207+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ac04e9b0 con 0x7f4398003ef0 2023-10-16T14:18:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.209+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f439c0075c0 con 0x7f4398003ef0 2023-10-16T14:18:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.209+0000 7f43a97fa640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f438c06b130 0x7f438c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.210+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f439c089e70 con 0x7f4398003ef0 2023-10-16T14:18:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.210+0000 7f43abfff640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f438c06b130 0x7f438c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.212+0000 7f43abfff640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f438c06b130 0x7f438c06d5f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4394036000 tx=0x7f4394030000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.214+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f439c0549a0 con 0x7f4398003ef0 2023-10-16T14:18:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.435+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f43ac0d1d00 con 0x7f438c06b130 2023-10-16T14:18:06.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.567+0000 7f43a97fa640 1 -- 172.21.15.29:0/941312503 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f439402c980 con 0x7f438c06b130 2023-10-16T14:18:06.571 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-16T14:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.570+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f438c06b130 msgr2=0x7f438c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.570+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f438c06b130 0x7f438c06d5f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4394036000 tx=0x7f4394030000 comp rx=0 tx=0).stop 2023-10-16T14:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.570+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 msgr2=0x7f43ac0c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.570+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43ac0c8c90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f439c00dfc0 tx=0x7f439c00b560 comp rx=0 tx=0).stop 2023-10-16T14:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 shutdown_connections 2023-10-16T14:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f438c06b130 0x7f438c06d5f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43ac1f27e0 0x7f43ac1f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43ac0b7660 0x7f43ac0b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 --2- 172.21.15.29:0/941312503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4398003ef0 0x7f43ac0c8c90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 >> 172.21.15.29:0/941312503 conn(0x7f43ac0cf570 msgr2=0x7f43ac15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 shutdown_connections 2023-10-16T14:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.571+0000 7f43b1ba5640 1 -- 172.21.15.29:0/941312503 wait complete. 2023-10-16T14:18:06.594 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-16T14:18:06.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-10-16T14:18:06.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.816+0000 7fb2498f3640 1 Processor -- start 2023-10-16T14:18:06.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.817+0000 7fb2498f3640 1 -- start start 2023-10-16T14:18:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.818+0000 7fb2498f3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2440b8480 0x7fb2440cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.818+0000 7fb2498f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.818+0000 7fb2498f3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24415d150 con 0x7fb24415cd70 2023-10-16T14:18:06.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.818+0000 7fb2498f3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2440d1180 con 0x7fb2440b8480 2023-10-16T14:18:06.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.818+0000 7fb2498f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440d1460 con 0x7fb2440d0be0 2023-10-16T14:18:06.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.818+0000 7fb2488f1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb24415cd70 0x7fb2440d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40900/0 (socket says 172.21.15.29:40900) 2023-10-16T14:18:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb2488f1640 1 -- 172.21.15.29:0/3426535391 learned_addr learned my addr 172.21.15.29:0/3426535391 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb23bfff640 1 --2- 172.21.15.29:0/3426535391 >> v2:172.21.15.29:3300/0 conn(0x7fb2440b8480 0x7fb2440cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb2490f2640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb23bfff640 1 -- 172.21.15.29:0/3426535391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 msgr2=0x7fb2440cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb23bfff640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb23bfff640 1 -- 172.21.15.29:0/3426535391 >> v1:172.21.15.29:6789/0 conn(0x7fb24415cd70 legacy=0x7fb2440d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb23bfff640 1 -- 172.21.15.29:0/3426535391 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440d1740 con 0x7fb2440b8480 2023-10-16T14:18:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb23b7fe640 1 -- 172.21.15.29:0/3426535391 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552729851 0 0) 0x7fb24415d150 con 0x7fb24415cd70 2023-10-16T14:18:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.819+0000 7fb2490f2640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.820+0000 7fb23bfff640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2440b8480 0x7fb2440cd7c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb2340097a0 tx=0x7fb23402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66c4c6ee66e5185 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb23b7fe640 1 -- 172.21.15.29:0/3426535391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23402d040 con 0x7fb2440b8480 2023-10-16T14:18:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb23b7fe640 1 -- 172.21.15.29:0/3426535391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb234031810 con 0x7fb2440b8480 2023-10-16T14:18:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb23b7fe640 1 -- 172.21.15.29:0/3426535391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234031ba0 con 0x7fb2440b8480 2023-10-16T14:18:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 -- 172.21.15.29:0/3426535391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2440b8480 msgr2=0x7fb2440cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2440b8480 0x7fb2440cd7c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb2340097a0 tx=0x7fb23402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 -- 172.21.15.29:0/3426535391 shutdown_connections 2023-10-16T14:18:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 --2- 172.21.15.29:0/3426535391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2440b8480 0x7fb2440cd7c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 -- 172.21.15.29:0/3426535391 >> 172.21.15.29:0/3426535391 conn(0x7fb2440cf570 msgr2=0x7fb2440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 -- 172.21.15.29:0/3426535391 shutdown_connections 2023-10-16T14:18:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.821+0000 7fb2498f3640 1 -- 172.21.15.29:0/3426535391 wait complete. 2023-10-16T14:18:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.823+0000 7fb2498f3640 1 Processor -- start 2023-10-16T14:18:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.823+0000 7fb2498f3640 1 -- start start 2023-10-16T14:18:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.824+0000 7fb2498f3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2440b8480 0x7fb244150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.824+0000 7fb2498f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.824+0000 7fb2498f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 0x7fb2441f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.824+0000 7fb2498f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440cbc60 con 0x7fb24415cd70 2023-10-16T14:18:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.824+0000 7fb2498f3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440cb050 con 0x7fb2440b8480 2023-10-16T14:18:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.824+0000 7fb2498f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440cb330 con 0x7fb2440d0be0 2023-10-16T14:18:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 0x7fb2441f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 0x7fb2441f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37790/0 (socket says 172.21.15.29:37790) 2023-10-16T14:18:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 -- 172.21.15.29:0/2376873284 learned_addr learned my addr 172.21.15.29:0/2376873284 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb23bfff640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2488f1640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2440b8480 0x7fb244150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 -- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 msgr2=0x7fb2440ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 -- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2440b8480 msgr2=0x7fb244150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2440b8480 0x7fb244150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2490f2640 1 -- 172.21.15.29:0/2376873284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb234009400 con 0x7fb24415cd70 2023-10-16T14:18:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.825+0000 7fb2488f1640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2440b8480 0x7fb244150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.826+0000 7fb2490f2640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 0x7fb2441f2170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00ce40 tx=0x7fb23c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.826+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c00d840 con 0x7fb24415cd70 2023-10-16T14:18:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.826+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2440cb610 con 0x7fb24415cd70 2023-10-16T14:18:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.826+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c00d9a0 con 0x7fb24415cd70 2023-10-16T14:18:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.826+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c00dd50 con 0x7fb24415cd70 2023-10-16T14:18:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.826+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb244151420 con 0x7fb24415cd70 2023-10-16T14:18:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.828+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24404e9b0 con 0x7fb24415cd70 2023-10-16T14:18:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.831+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb23c01e3b0 con 0x7fb24415cd70 2023-10-16T14:18:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.832+0000 7fb2397fa640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb23006b100 0x7fb23006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.832+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb23c08ab50 con 0x7fb24415cd70 2023-10-16T14:18:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.833+0000 7fb2488f1640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb23006b100 0x7fb23006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.834+0000 7fb2488f1640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb23006b100 0x7fb23006d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb240000f60 tx=0x7fb240009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:06.834+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb23c055700 con 0x7fb24415cd70 2023-10-16T14:18:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.052+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2440d1d00 con 0x7fb23006b100 2023-10-16T14:18:07.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.177+0000 7fb2397fa640 1 -- 172.21.15.29:0/2376873284 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb2400012f0 con 0x7fb23006b100 2023-10-16T14:18:07.181 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-16T14:18:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb23006b100 msgr2=0x7fb23006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb23006b100 0x7fb23006d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb240000f60 tx=0x7fb240009040 comp rx=0 tx=0).stop 2023-10-16T14:18:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 msgr2=0x7fb2441f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 0x7fb2441f2170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00ce40 tx=0x7fb23c00a710 comp rx=0 tx=0).stop 2023-10-16T14:18:07.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 shutdown_connections 2023-10-16T14:18:07.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb23006b100 0x7fb23006d5c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24415cd70 0x7fb2441f2170 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2440d0be0 0x7fb2440ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.180+0000 7fb2498f3640 1 --2- 172.21.15.29:0/2376873284 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2440b8480 0x7fb244150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.181+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 >> 172.21.15.29:0/2376873284 conn(0x7fb2440cf570 msgr2=0x7fb2440b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:07.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.181+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 shutdown_connections 2023-10-16T14:18:07.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.181+0000 7fb2498f3640 1 -- 172.21.15.29:0/2376873284 wait complete. 2023-10-16T14:18:07.204 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-16T14:18:07.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-10-16T14:18:07.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.432+0000 7f38bc8e5640 1 Processor -- start 2023-10-16T14:18:07.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.432+0000 7f38bc8e5640 1 -- start start 2023-10-16T14:18:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.433+0000 7f38bc8e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f38b8160ad0 0x7f38b80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.433+0000 7f38bc8e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:07.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.433+0000 7f38bc8e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b80ce800 con 0x7f38b80ae260 2023-10-16T14:18:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.433+0000 7f38bc8e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b80cdec0 con 0x7f38b8160ad0 2023-10-16T14:18:07.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.433+0000 7f38bc8e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80acdd0 con 0x7f38b80ce350 2023-10-16T14:18:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.434+0000 7f38b5d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.434+0000 7f38b659c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f38b8160ad0 0x7f38b80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.434+0000 7f38b659c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f38b8160ad0 0x7f38b80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:53612/0 (socket says 172.21.15.29:53612) 2023-10-16T14:18:07.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.434+0000 7f38b659c640 1 -- 172.21.15.29:0/157871179 learned_addr learned my addr 172.21.15.29:0/157871179 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.434+0000 7f38b5d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:54086/0 (socket says 172.21.15.29:54086) 2023-10-16T14:18:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.434+0000 7f38b6d9d640 1 --1- 172.21.15.29:0/157871179 >> v1:172.21.15.170:6789/0 conn(0x7f38b80ae260 0x7f38b80ccd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:50924/0 (socket says 172.21.15.29:50924) 2023-10-16T14:18:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.435+0000 7f38b659c640 1 -- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 msgr2=0x7f38b8160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.435+0000 7f38b659c640 1 --2- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.435+0000 7f38b659c640 1 -- 172.21.15.29:0/157871179 >> v1:172.21.15.170:6789/0 conn(0x7f38b80ae260 legacy=0x7f38b80ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.435+0000 7f38b659c640 1 -- 172.21.15.29:0/157871179 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b81613f0 con 0x7f38b8160ad0 2023-10-16T14:18:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.436+0000 7f38b659c640 1 --2- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b8160ad0 0x7f38b80ac890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009e50 tx=0x7f38a802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27963a41d13d0756 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.437+0000 7f38b559a640 1 -- 172.21.15.29:0/157871179 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a802cb00 con 0x7f38b8160ad0 2023-10-16T14:18:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.437+0000 7f38b559a640 1 -- 172.21.15.29:0/157871179 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a802cc60 con 0x7f38b8160ad0 2023-10-16T14:18:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.437+0000 7f38b559a640 1 -- 172.21.15.29:0/157871179 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a80343f0 con 0x7f38b8160ad0 2023-10-16T14:18:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.437+0000 7f38bc8e5640 1 -- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b8160ad0 msgr2=0x7f38b80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.437+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b8160ad0 0x7f38b80ac890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009e50 tx=0x7f38a802bf60 comp rx=0 tx=0).stop 2023-10-16T14:18:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.438+0000 7f38bc8e5640 1 -- 172.21.15.29:0/157871179 shutdown_connections 2023-10-16T14:18:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.438+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.438+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/157871179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b8160ad0 0x7f38b80ac890 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.438+0000 7f38bc8e5640 1 -- 172.21.15.29:0/157871179 >> 172.21.15.29:0/157871179 conn(0x7f38b806bc10 msgr2=0x7f38b806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.438+0000 7f38bc8e5640 1 -- 172.21.15.29:0/157871179 shutdown_connections 2023-10-16T14:18:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.438+0000 7f38bc8e5640 1 -- 172.21.15.29:0/157871179 wait complete. 2023-10-16T14:18:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.439+0000 7f38bc8e5640 1 Processor -- start 2023-10-16T14:18:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.440+0000 7f38bc8e5640 1 -- start start 2023-10-16T14:18:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38bc8e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b80ae260 0x7f38b815c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38bc8e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38bc8e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 0x7f38b8070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38bc8e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c4770 con 0x7f38b8160ad0 2023-10-16T14:18:07.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38bc8e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80cb2c0 con 0x7f38b80ae260 2023-10-16T14:18:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38bc8e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b8071280 con 0x7f38b80ce350 2023-10-16T14:18:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38b6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 0x7f38b8070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38b6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 0x7f38b8070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37792/0 (socket says 172.21.15.29:37792) 2023-10-16T14:18:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38b6d9d640 1 -- 172.21.15.29:0/3307066485 learned_addr learned my addr 172.21.15.29:0/3307066485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38b659c640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b80ae260 0x7f38b815c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.441+0000 7f38b5d9b640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b6d9d640 1 -- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 msgr2=0x7f38b8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b6d9d640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b6d9d640 1 -- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b80ae260 msgr2=0x7f38b815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b6d9d640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b80ae260 0x7f38b815c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b6d9d640 1 -- 172.21.15.29:0/3307066485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8009b30 con 0x7f38b8160ad0 2023-10-16T14:18:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b659c640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b80ae260 0x7f38b815c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:18:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38b6d9d640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 0x7f38b8070d40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f38b000c430 tx=0x7f38b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00074e0 con 0x7f38b8160ad0 2023-10-16T14:18:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b0007640 con 0x7f38b8160ad0 2023-10-16T14:18:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00049e0 con 0x7f38b8160ad0 2023-10-16T14:18:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.442+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b8071480 con 0x7f38b8160ad0 2023-10-16T14:18:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.443+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b80c40b0 con 0x7f38b8160ad0 2023-10-16T14:18:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.444+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b80c48b0 con 0x7f38b8160ad0 2023-10-16T14:18:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.445+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f38b0004b40 con 0x7f38b8160ad0 2023-10-16T14:18:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.446+0000 7f38aeffd640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f389806b100 0x7f389806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.446+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f38b00894e0 con 0x7f38b8160ad0 2023-10-16T14:18:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.447+0000 7f38b659c640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f389806b100 0x7f389806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.448+0000 7f38b659c640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f389806b100 0x7f389806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f38a8035000 tx=0x7f38a8030000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.451+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f38b80c48b0 con 0x7f38b8160ad0 2023-10-16T14:18:07.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.673+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f38b80cea50 con 0x7f389806b100 2023-10-16T14:18:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.808+0000 7f38aeffd640 1 -- 172.21.15.29:0/3307066485 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f38a80440c0 con 0x7f389806b100 2023-10-16T14:18:07.811 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-16T14:18:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.811+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f389806b100 msgr2=0x7f389806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.811+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f389806b100 0x7f389806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f38a8035000 tx=0x7f38a8030000 comp rx=0 tx=0).stop 2023-10-16T14:18:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.811+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 msgr2=0x7f38b8070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:07.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.811+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 0x7f38b8070d40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f38b000c430 tx=0x7f38b000c8f0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 shutdown_connections 2023-10-16T14:18:07.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f389806b100 0x7f389806d5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b8160ad0 0x7f38b8070d40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f38b80ce350 0x7f38b8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 --2- 172.21.15.29:0/3307066485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38b80ae260 0x7f38b815c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:07.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 >> 172.21.15.29:0/3307066485 conn(0x7f38b806bc10 msgr2=0x7f38b80aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:07.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 shutdown_connections 2023-10-16T14:18:07.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:07.812+0000 7f38bc8e5640 1 -- 172.21.15.29:0/3307066485 wait complete. 2023-10-16T14:18:07.836 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-16T14:18:07.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-10-16T14:18:08.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.063+0000 7f20d179e640 1 Processor -- start 2023-10-16T14:18:08.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.063+0000 7f20d179e640 1 -- start start 2023-10-16T14:18:08.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d179e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20cc15b320 0x7f20cc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d179e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc15b9d0 con 0x7f20cc06c040 2023-10-16T14:18:08.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d179e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc15bcb0 con 0x7f20cc0ad970 2023-10-16T14:18:08.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d179e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc15b7d0 con 0x7f20cc15b320 2023-10-16T14:18:08.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d0f9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20cc15b320 0x7f20cc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d0f9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20cc15b320 0x7f20cc0cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37798/0 (socket says 172.21.15.29:37798) 2023-10-16T14:18:08.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.065+0000 7f20d0f9d640 1 -- 172.21.15.29:0/1687303606 learned_addr learned my addr 172.21.15.29:0/1687303606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.066+0000 7f20d0f9d640 1 -- 172.21.15.29:0/1687303606 >> v1:172.21.15.170:6789/0 conn(0x7f20cc0ad970 legacy=0x7f20cc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.066+0000 7f20d0f9d640 1 -- 172.21.15.29:0/1687303606 >> v1:172.21.15.29:6789/0 conn(0x7f20cc06c040 legacy=0x7f20cc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.066+0000 7f20d0f9d640 1 -- 172.21.15.29:0/1687303606 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc15bf90 con 0x7f20cc15b320 2023-10-16T14:18:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.067+0000 7f20d0f9d640 1 --2- 172.21.15.29:0/1687303606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc15b320 0x7f20cc0cd180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009e80 tx=0x7f20c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f0ca3e1d1a9023 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.067+0000 7f20caffd640 1 -- 172.21.15.29:0/1687303606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c402cb00 con 0x7f20cc15b320 2023-10-16T14:18:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.067+0000 7f20caffd640 1 -- 172.21.15.29:0/1687303606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c402cc60 con 0x7f20cc15b320 2023-10-16T14:18:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.067+0000 7f20caffd640 1 -- 172.21.15.29:0/1687303606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c40343f0 con 0x7f20cc15b320 2023-10-16T14:18:08.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 -- 172.21.15.29:0/1687303606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc15b320 msgr2=0x7f20cc0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 --2- 172.21.15.29:0/1687303606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc15b320 0x7f20cc0cd180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009e80 tx=0x7f20c402bf60 comp rx=0 tx=0).stop 2023-10-16T14:18:08.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 -- 172.21.15.29:0/1687303606 shutdown_connections 2023-10-16T14:18:08.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 --2- 172.21.15.29:0/1687303606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc15b320 0x7f20cc0cd180 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 -- 172.21.15.29:0/1687303606 >> 172.21.15.29:0/1687303606 conn(0x7f20cc0ce6a0 msgr2=0x7f20cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:08.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 -- 172.21.15.29:0/1687303606 shutdown_connections 2023-10-16T14:18:08.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.068+0000 7f20d179e640 1 -- 172.21.15.29:0/1687303606 wait complete. 2023-10-16T14:18:08.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.070+0000 7f20d179e640 1 Processor -- start 2023-10-16T14:18:08.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.070+0000 7f20d179e640 1 -- start start 2023-10-16T14:18:08.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20d179e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20cc06c040 0x7f20cc0ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20d179e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 0x7f20cc151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20d179e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20cc15b320 0x7f20cc1f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20d179e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0b1050 con 0x7f20cc0ad970 2023-10-16T14:18:08.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20d179e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc1f5f60 con 0x7f20cc06c040 2023-10-16T14:18:08.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20d179e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc1f5c30 con 0x7f20cc15b320 2023-10-16T14:18:08.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 0x7f20cc151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.071+0000 7f20cb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 0x7f20cc151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37808/0 (socket says 172.21.15.29:37808) 2023-10-16T14:18:08.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 -- 172.21.15.29:0/3308870422 learned_addr learned my addr 172.21.15.29:0/3308870422 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:08.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cbfff640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20cc06c040 0x7f20cc0ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 -- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20cc15b320 msgr2=0x7f20cc1f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20cc15b320 0x7f20cc1f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 -- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20cc06c040 msgr2=0x7f20cc0ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20cc06c040 0x7f20cc0ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 -- 172.21.15.29:0/3308870422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4009b30 con 0x7f20cc0ad970 2023-10-16T14:18:08.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.072+0000 7f20cb7fe640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 0x7f20cc151b70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f20c000b3f0 tx=0x7f20c000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:08.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.073+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0011830 con 0x7f20cc0ad970 2023-10-16T14:18:08.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.073+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0011990 con 0x7f20cc0ad970 2023-10-16T14:18:08.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.073+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0011d40 con 0x7f20cc0ad970 2023-10-16T14:18:08.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.073+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc0ae7e0 con 0x7f20cc0ad970 2023-10-16T14:18:08.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.073+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc0aeda0 con 0x7f20cc0ad970 2023-10-16T14:18:08.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.074+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20cc1f60f0 con 0x7f20cc0ad970 2023-10-16T14:18:08.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.075+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f20c001a400 con 0x7f20cc0ad970 2023-10-16T14:18:08.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.076+0000 7f20c8ff9640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f20b806b100 0x7f20b806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.076+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f20c0089970 con 0x7f20cc0ad970 2023-10-16T14:18:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.077+0000 7f20cbfff640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f20b806b100 0x7f20b806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.078+0000 7f20cbfff640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f20b806b100 0x7f20b806d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f20bc000f60 tx=0x7f20bc009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:08.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.080+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f20c00545b0 con 0x7f20cc0ad970 2023-10-16T14:18:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.297+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f20cc0cc550 con 0x7f20b806b100 2023-10-16T14:18:08.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.432+0000 7f20c8ff9640 1 -- 172.21.15.29:0/3308870422 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f20bc0012c0 con 0x7f20b806b100 2023-10-16T14:18:08.436 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-16T14:18:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f20b806b100 msgr2=0x7f20b806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f20b806b100 0x7f20b806d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f20bc000f60 tx=0x7f20bc009040 comp rx=0 tx=0).stop 2023-10-16T14:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 msgr2=0x7f20cc151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 0x7f20cc151b70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f20c000b3f0 tx=0x7f20c000b8b0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 shutdown_connections 2023-10-16T14:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f20b806b100 0x7f20b806d5c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20cc0ad970 0x7f20cc151b70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20cc15b320 0x7f20cc1f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 --2- 172.21.15.29:0/3308870422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20cc06c040 0x7f20cc0ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 >> 172.21.15.29:0/3308870422 conn(0x7f20cc0ce6a0 msgr2=0x7f20cc0ca7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 shutdown_connections 2023-10-16T14:18:08.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.436+0000 7f20d179e640 1 -- 172.21.15.29:0/3308870422 wait complete. 2023-10-16T14:18:08.460 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-16T14:18:08.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-10-16T14:18:08.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.692+0000 7f08a73ed640 1 Processor -- start 2023-10-16T14:18:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.692+0000 7f08a73ed640 1 -- start start 2023-10-16T14:18:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.694+0000 7f08a73ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f08a006c040 0x7f08a006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.694+0000 7f08a73ed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a015f4a0 con 0x7f08a00cda80 2023-10-16T14:18:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.694+0000 7f08a73ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a006c960 con 0x7f08a00ad970 2023-10-16T14:18:08.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.694+0000 7f08a73ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a015b9d0 con 0x7f08a006c040 2023-10-16T14:18:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.695+0000 7f08a6bec640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f08a00cda80 0x7f08a015ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40920/0 (socket says 172.21.15.29:40920) 2023-10-16T14:18:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.695+0000 7f08a6bec640 1 -- 172.21.15.29:0/693136763 learned_addr learned my addr 172.21.15.29:0/693136763 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:18:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.695+0000 7f08a63eb640 1 --2- 172.21.15.29:0/693136763 >> v2:172.21.15.170:3300/0 conn(0x7f08a006c040 0x7f08a006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.695+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911472846 0 0) 0x7f08a015f4a0 con 0x7f08a00cda80 2023-10-16T14:18:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.695+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f088c003520 con 0x7f08a00cda80 2023-10-16T14:18:08.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694098410 0 0) 0x7f088c003520 con 0x7f08a00cda80 2023-10-16T14:18:08.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 >> v2:172.21.15.170:3300/0 conn(0x7f08a006c040 msgr2=0x7f08a006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:08.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 --2- 172.21.15.29:0/693136763 >> v2:172.21.15.170:3300/0 conn(0x7f08a006c040 0x7f08a006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 >> v1:172.21.15.170:6789/0 conn(0x7f08a00ad970 legacy=0x7f08a015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a015bcb0 con 0x7f08a00cda80 2023-10-16T14:18:08.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f08980019f0 con 0x7f08a00cda80 2023-10-16T14:18:08.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 >> v1:172.21.15.29:6789/0 conn(0x7f08a00cda80 legacy=0x7f08a015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f088c0039c0 0x7f088c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f088c0047c0 0x7f088c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c005180 0x7f088c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a015b9d0 con 0x7f088c0047c0 2023-10-16T14:18:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c00c080 con 0x7f088c005180 2023-10-16T14:18:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c00c240 con 0x7f088c0039c0 2023-10-16T14:18:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a5bea640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f088c0047c0 0x7f088c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a63eb640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f088c0039c0 0x7f088c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.696+0000 7f08a6bec640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c005180 0x7f088c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.697+0000 7f08a5bea640 1 -- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f088c0039c0 msgr2=0x7f088c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.697+0000 7f08a5bea640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f088c0039c0 0x7f088c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.697+0000 7f08a5bea640 1 -- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c005180 msgr2=0x7f088c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:18:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.697+0000 7f08a5bea640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c005180 0x7f088c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.697+0000 7f08a5bea640 1 -- 172.21.15.29:0/693136763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c003520 con 0x7f088c0047c0 2023-10-16T14:18:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.697+0000 7f08a6bec640 1 -- 172.21.15.29:0/693136763 reap_dead start 2023-10-16T14:18:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.698+0000 7f08a5bea640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f088c0047c0 0x7f088c004c40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0890001720 tx=0x7f088c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c551c506e178a244 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.698+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08900052d0 con 0x7f088c0047c0 2023-10-16T14:18:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.698+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089002e050 con 0x7f088c0047c0 2023-10-16T14:18:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.698+0000 7f08a53e9640 1 -- 172.21.15.29:0/693136763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890003830 con 0x7f088c0047c0 2023-10-16T14:18:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 -- 172.21.15.29:0/693136763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f088c0047c0 msgr2=0x7f088c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f088c0047c0 0x7f088c004c40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0890001720 tx=0x7f088c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 -- 172.21.15.29:0/693136763 shutdown_connections 2023-10-16T14:18:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 --2- 172.21.15.29:0/693136763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f088c0047c0 0x7f088c004c40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 -- 172.21.15.29:0/693136763 >> 172.21.15.29:0/693136763 conn(0x7f08a00ce6a0 msgr2=0x7f08a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 -- 172.21.15.29:0/693136763 shutdown_connections 2023-10-16T14:18:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.699+0000 7f08a73ed640 1 -- 172.21.15.29:0/693136763 wait complete. 2023-10-16T14:18:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.701+0000 7f08a73ed640 1 Processor -- start 2023-10-16T14:18:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.701+0000 7f08a73ed640 1 -- start start 2023-10-16T14:18:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a73ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c0047c0 0x7f08a014fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a73ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 0x7f08a00c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a73ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08a00ca280 0x7f08a01f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a73ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00ca6e0 con 0x7f08a015dd30 2023-10-16T14:18:08.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a73ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a0150000 con 0x7f088c0047c0 2023-10-16T14:18:08.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a73ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00c8560 con 0x7f08a00ca280 2023-10-16T14:18:08.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a5bea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 0x7f08a00c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a5bea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 0x7f08a00c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37826/0 (socket says 172.21.15.29:37826) 2023-10-16T14:18:08.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.702+0000 7f08a5bea640 1 -- 172.21.15.29:0/2778450830 learned_addr learned my addr 172.21.15.29:0/2778450830 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:18:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a6bec640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08a00ca280 0x7f08a01f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a5bea640 1 -- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08a00ca280 msgr2=0x7f08a01f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a63eb640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c0047c0 0x7f08a014fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a5bea640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08a00ca280 0x7f08a01f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a5bea640 1 -- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c0047c0 msgr2=0x7f08a014fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:08.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a5bea640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c0047c0 0x7f08a014fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:08.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a5bea640 1 -- 172.21.15.29:0/2778450830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00b1d60 con 0x7f08a015dd30 2023-10-16T14:18:08.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a63eb640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c0047c0 0x7f08a014fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:18:08.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a5bea640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 0x7f08a00c9d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f089002bbc0 tx=0x7f0890001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:08.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08900076c0 con 0x7f08a015dd30 2023-10-16T14:18:08.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0890007a30 con 0x7f08a015dd30 2023-10-16T14:18:08.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890006c20 con 0x7f08a015dd30 2023-10-16T14:18:08.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.703+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a00b2040 con 0x7f08a015dd30 2023-10-16T14:18:08.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.704+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a00b2600 con 0x7f08a015dd30 2023-10-16T14:18:08.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.705+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a006b330 con 0x7f08a015dd30 2023-10-16T14:18:08.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.706+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f08900035f0 con 0x7f08a015dd30 2023-10-16T14:18:08.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.707+0000 7f0896ffd640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f088406b100 0x7f088406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:18:08.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.707+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f08900ab8a0 con 0x7f08a015dd30 2023-10-16T14:18:08.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.708+0000 7f08a63eb640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f088406b100 0x7f088406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:18:08.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.709+0000 7f08a63eb640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f088406b100 0x7f088406d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f08a0063ab0 tx=0x7f089c008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:18:08.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.712+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0890079f30 con 0x7f08a015dd30 2023-10-16T14:18:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:08.930+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f08a00cc270 con 0x7f088406b100 2023-10-16T14:18:09.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.057+0000 7f0896ffd640 1 -- 172.21.15.29:0/2778450830 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f089c0128a0 con 0x7f088406b100 2023-10-16T14:18:09.061 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-16T14:18:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f088406b100 msgr2=0x7f088406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f088406b100 0x7f088406d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f08a0063ab0 tx=0x7f089c008040 comp rx=0 tx=0).stop 2023-10-16T14:18:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 msgr2=0x7f08a00c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:18:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 0x7f08a00c9d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f089002bbc0 tx=0x7f0890001120 comp rx=0 tx=0).stop 2023-10-16T14:18:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 shutdown_connections 2023-10-16T14:18:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f088406b100 0x7f088406d5c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08a015dd30 0x7f08a00c9d40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:09.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08a00ca280 0x7f08a01f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:09.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 --2- 172.21.15.29:0/2778450830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f088c0047c0 0x7f08a014fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:18:09.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 >> 172.21.15.29:0/2778450830 conn(0x7f08a00ce6a0 msgr2=0x7f08a00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:18:09.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 shutdown_connections 2023-10-16T14:18:09.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:18:09.061+0000 7f08a73ed640 1 -- 172.21.15.29:0/2778450830 wait complete. 2023-10-16T14:18:09.084 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-16T14:18:13.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:18:13.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:18:43.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:18:43.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:19:13.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:19:13.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:19:43.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:19:43.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:20:13.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:20:13.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:20:43.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:20:43.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:21:13.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:21:13.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:21:44.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:21:44.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:22:01.109 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:22:01.105+0000 7f5f7b425640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:22:01.109 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:22:01.105+0000 7f5f7b425640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:22:01.109 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:22:01.105+0000 7f5f7b425640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:22:01.109 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:22:01.105+0000 7f5f7b425640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:22:01.110 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:22:01.105+0000 7f5f7b425640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:22:14.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:22:14.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:22:44.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:22:44.476 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:23:14.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:23:14.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:23:44.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:23:44.800 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:24:14.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:24:14.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:24:45.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:24:45.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:25:15.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:25:15.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:25:45.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:25:45.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:26:15.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:26:15.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:26:45.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:26:45.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:27:15.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:27:15.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:27:46.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:27:46.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:28:16.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:28:16.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:28:46.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:28:46.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:29:16.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:29:16.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:29:22.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-10-16T14:29:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.322+0000 7fc7a91a0640 1 Processor -- start 2023-10-16T14:29:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.322+0000 7fc7a91a0640 1 -- start start 2023-10-16T14:29:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.323+0000 7fc7a91a0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc7a415cd70 0x7fc7a40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.323+0000 7fc7a91a0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a40b8480 0x7fc7a40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.323+0000 7fc7a91a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc7a91a0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a415d150 con 0x7fc7a40b8480 2023-10-16T14:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc7a91a0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40d1180 con 0x7fc7a415cd70 2023-10-16T14:29:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc7a91a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40d1460 con 0x7fc7a40d0be0 2023-10-16T14:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a40b8480 0x7fc7a40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a40b8480 0x7fc7a40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35252/0 (socket says 172.21.15.29:35252) 2023-10-16T14:29:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 -- 172.21.15.29:0/656743586 learned_addr learned my addr 172.21.15.29:0/656743586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc7a3fff640 1 --2- 172.21.15.29:0/656743586 >> v2:172.21.15.170:3300/0 conn(0x7fc7a415cd70 0x7fc7a40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc7a899f640 1 --2- 172.21.15.29:0/656743586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 -- 172.21.15.29:0/656743586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 msgr2=0x7fc7a40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/656743586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a40cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 -- 172.21.15.29:0/656743586 >> v2:172.21.15.170:3300/0 conn(0x7fc7a415cd70 msgr2=0x7fc7a40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/656743586 >> v2:172.21.15.170:3300/0 conn(0x7fc7a415cd70 0x7fc7a40d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.324+0000 7fc79b7fe640 1 -- 172.21.15.29:0/656743586 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40d1740 con 0x7fc7a40b8480 2023-10-16T14:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.326+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/656743586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a40cd7c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40b56a0 tx=0x7fc78c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda06b2be73a5503 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.326+0000 7fc7a37fe640 1 -- 172.21.15.29:0/656743586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c031870 con 0x7fc7a40b8480 2023-10-16T14:29:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.326+0000 7fc7a37fe640 1 -- 172.21.15.29:0/656743586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c0319d0 con 0x7fc7a40b8480 2023-10-16T14:29:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.326+0000 7fc7a37fe640 1 -- 172.21.15.29:0/656743586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c031d60 con 0x7fc7a40b8480 2023-10-16T14:29:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.326+0000 7fc7a91a0640 1 -- 172.21.15.29:0/656743586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 msgr2=0x7fc7a40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.326+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/656743586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a40cd7c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40b56a0 tx=0x7fc78c02b910 comp rx=0 tx=0).stop 2023-10-16T14:29:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 -- 172.21.15.29:0/656743586 shutdown_connections 2023-10-16T14:29:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/656743586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/656743586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a40cd7c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/656743586 >> v2:172.21.15.170:3300/0 conn(0x7fc7a415cd70 0x7fc7a40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 -- 172.21.15.29:0/656743586 >> 172.21.15.29:0/656743586 conn(0x7fc7a40cf570 msgr2=0x7fc7a40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:22.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 -- 172.21.15.29:0/656743586 shutdown_connections 2023-10-16T14:29:22.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.327+0000 7fc7a91a0640 1 -- 172.21.15.29:0/656743586 wait complete. 2023-10-16T14:29:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.329+0000 7fc7a91a0640 1 Processor -- start 2023-10-16T14:29:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.329+0000 7fc7a91a0640 1 -- start start 2023-10-16T14:29:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a91a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a4151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a91a0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a415dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a91a0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7a415cd70 0x7fc7a40cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a91a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4155e90 con 0x7fc7a40b8480 2023-10-16T14:29:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a91a0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4151e40 con 0x7fc7a415cd70 2023-10-16T14:29:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a91a0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c9d40 con 0x7fc7a40d0be0 2023-10-16T14:29:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a4151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a4151900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35256/0 (socket says 172.21.15.29:35256) 2023-10-16T14:29:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.330+0000 7fc7a3fff640 1 -- 172.21.15.29:0/3875731145 learned_addr learned my addr 172.21.15.29:0/3875731145 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a3fff640 1 -- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 msgr2=0x7fc7a415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a3fff640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a415dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a3fff640 1 -- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7a415cd70 msgr2=0x7fc7a40cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a899f640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7a415cd70 0x7fc7a40cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a3fff640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7a415cd70 0x7fc7a40cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a3fff640 1 -- 172.21.15.29:0/3875731145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0094d0 con 0x7fc7a40b8480 2023-10-16T14:29:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a899f640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7a415cd70 0x7fc7a40cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a3fff640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a4151900 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc79400ce10 tx=0x7fc79400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794003b90 con 0x7fc7a40b8480 2023-10-16T14:29:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc794003cf0 con 0x7fc7a40b8480 2023-10-16T14:29:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a40ca020 con 0x7fc7a40b8480 2023-10-16T14:29:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.331+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794003ea0 con 0x7fc7a40b8480 2023-10-16T14:29:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.332+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a40ca5e0 con 0x7fc7a40b8480 2023-10-16T14:29:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.333+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a404e9b0 con 0x7fc7a40b8480 2023-10-16T14:29:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.334+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc794004240 con 0x7fc7a40b8480 2023-10-16T14:29:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.335+0000 7fc7a17fa640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc78406b070 0x7fc78406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.335+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc794012070 con 0x7fc7a40b8480 2023-10-16T14:29:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.335+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc78406b070 0x7fc78406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.336+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc78406b070 0x7fc78406d530 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc78c036000 tx=0x7fc78c030000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.340+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7940551d0 con 0x7fc7a40b8480 2023-10-16T14:29:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.555+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7a40d1d00 con 0x7fc78406b070 2023-10-16T14:29:22.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.718+0000 7fc7a17fa640 1 -- 172.21.15.29:0/3875731145 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc78c03a420 con 0x7fc78406b070 2023-10-16T14:29:22.723 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T14:29:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.721+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc78406b070 msgr2=0x7fc78406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.721+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc78406b070 0x7fc78406d530 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc78c036000 tx=0x7fc78c030000 comp rx=0 tx=0).stop 2023-10-16T14:29:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.721+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 msgr2=0x7fc7a4151900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.721+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a4151900 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc79400ce10 tx=0x7fc79400a660 comp rx=0 tx=0).stop 2023-10-16T14:29:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 shutdown_connections 2023-10-16T14:29:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc78406b070 0x7fc78406d530 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7a415cd70 0x7fc7a40cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7a40d0be0 0x7fc7a415dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 --2- 172.21.15.29:0/3875731145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a40b8480 0x7fc7a4151900 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 >> 172.21.15.29:0/3875731145 conn(0x7fc7a40cf570 msgr2=0x7fc7a40b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 shutdown_connections 2023-10-16T14:29:22.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.722+0000 7fc7a91a0640 1 -- 172.21.15.29:0/3875731145 wait complete. 2023-10-16T14:29:22.747 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T14:29:22.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-10-16T14:29:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.973+0000 7fb34d306640 1 Processor -- start 2023-10-16T14:29:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.973+0000 7fb34d306640 1 -- start start 2023-10-16T14:29:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.974+0000 7fb34d306640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.974+0000 7fb34d306640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb3480d0be0 0x7fb3480cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.974+0000 7fb34d306640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34815d150 con 0x7fb34815cd70 2023-10-16T14:29:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.974+0000 7fb34d306640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480d1180 con 0x7fb3480d0be0 2023-10-16T14:29:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.974+0000 7fb34d306640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480d1460 con 0x7fb3480b8480 2023-10-16T14:29:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.975+0000 7fb34cb05640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb3480d0be0 0x7fb3480cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.975+0000 7fb33f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.975+0000 7fb34cb05640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb3480d0be0 0x7fb3480cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:40686/0 (socket says 172.21.15.29:40686) 2023-10-16T14:29:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.975+0000 7fb34cb05640 1 -- 172.21.15.29:0/993068196 learned_addr learned my addr 172.21.15.29:0/993068196 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.976+0000 7fb33f7fe640 1 -- 172.21.15.29:0/993068196 >> v2:172.21.15.170:3300/0 conn(0x7fb3480d0be0 msgr2=0x7fb3480cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.976+0000 7fb33f7fe640 1 --2- 172.21.15.29:0/993068196 >> v2:172.21.15.170:3300/0 conn(0x7fb3480d0be0 0x7fb3480cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.976+0000 7fb33f7fe640 1 -- 172.21.15.29:0/993068196 >> v1:172.21.15.170:6789/0 conn(0x7fb34815cd70 legacy=0x7fb3480d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.976+0000 7fb33f7fe640 1 -- 172.21.15.29:0/993068196 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480d1740 con 0x7fb3480b8480 2023-10-16T14:29:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.977+0000 7fb33f7fe640 1 --2- 172.21.15.29:0/993068196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480cd7c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb3300097a0 tx=0x7fb33002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623d497ec254a714 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.977+0000 7fb3477fe640 1 -- 172.21.15.29:0/993068196 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33002d040 con 0x7fb3480b8480 2023-10-16T14:29:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb3477fe640 1 -- 172.21.15.29:0/993068196 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb330031810 con 0x7fb3480b8480 2023-10-16T14:29:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb3477fe640 1 -- 172.21.15.29:0/993068196 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330031ba0 con 0x7fb3480b8480 2023-10-16T14:29:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 -- 172.21.15.29:0/993068196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 msgr2=0x7fb3480cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 --2- 172.21.15.29:0/993068196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480cd7c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb3300097a0 tx=0x7fb33002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 -- 172.21.15.29:0/993068196 shutdown_connections 2023-10-16T14:29:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 --2- 172.21.15.29:0/993068196 >> v2:172.21.15.170:3300/0 conn(0x7fb3480d0be0 0x7fb3480cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 --2- 172.21.15.29:0/993068196 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480cd7c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 -- 172.21.15.29:0/993068196 >> 172.21.15.29:0/993068196 conn(0x7fb3480cf570 msgr2=0x7fb3480cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.978+0000 7fb34d306640 1 -- 172.21.15.29:0/993068196 shutdown_connections 2023-10-16T14:29:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.979+0000 7fb34d306640 1 -- 172.21.15.29:0/993068196 wait complete. 2023-10-16T14:29:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.980+0000 7fb34d306640 1 Processor -- start 2023-10-16T14:29:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.980+0000 7fb34d306640 1 -- start start 2023-10-16T14:29:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.981+0000 7fb34d306640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.981+0000 7fb34d306640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3480d0be0 0x7fb34815d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.981+0000 7fb34d306640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 0x7fb348151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.981+0000 7fb34d306640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34815e1e0 con 0x7fb34815cd70 2023-10-16T14:29:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.981+0000 7fb34d306640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34815e4c0 con 0x7fb3480d0be0 2023-10-16T14:29:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.981+0000 7fb34d306640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34815e7a0 con 0x7fb3480b8480 2023-10-16T14:29:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 0x7fb348151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 0x7fb348151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35266/0 (socket says 172.21.15.29:35266) 2023-10-16T14:29:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 -- 172.21.15.29:0/2957839221 learned_addr learned my addr 172.21.15.29:0/2957839221 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb33f7fe640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3480d0be0 0x7fb34815d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb347fff640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 -- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 msgr2=0x7fb3480b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 -- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3480d0be0 msgr2=0x7fb34815d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3480d0be0 0x7fb34815d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.982+0000 7fb34cb05640 1 -- 172.21.15.29:0/2957839221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb330009400 con 0x7fb34815cd70 2023-10-16T14:29:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.983+0000 7fb34cb05640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 0x7fb348151c60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb34000e7c0 tx=0x7fb34000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.983+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340009570 con 0x7fb34815cd70 2023-10-16T14:29:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.983+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34000c940 con 0x7fb34815cd70 2023-10-16T14:29:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.983+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3480cb400 con 0x7fb34815cd70 2023-10-16T14:29:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.983+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000ccf0 con 0x7fb34815cd70 2023-10-16T14:29:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.983+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3480b6660 con 0x7fb34815cd70 2023-10-16T14:29:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.984+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3480cb640 con 0x7fb34815cd70 2023-10-16T14:29:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.986+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fb340013470 con 0x7fb34815cd70 2023-10-16T14:29:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.987+0000 7fb3457fa640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb32806b070 0x7fb32806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.987+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb340089cc0 con 0x7fb34815cd70 2023-10-16T14:29:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.987+0000 7fb347fff640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb32806b070 0x7fb32806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.988+0000 7fb347fff640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb32806b070 0x7fb32806d530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3380012e0 tx=0x7fb338009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:22.991+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb340090080 con 0x7fb34815cd70 2023-10-16T14:29:23.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.210+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3480d1d00 con 0x7fb32806b070 2023-10-16T14:29:23.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.380+0000 7fb3457fa640 1 -- 172.21.15.29:0/2957839221 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb338001a00 con 0x7fb32806b070 2023-10-16T14:29:23.384 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T14:29:23.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb32806b070 msgr2=0x7fb32806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb32806b070 0x7fb32806d530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3380012e0 tx=0x7fb338009040 comp rx=0 tx=0).stop 2023-10-16T14:29:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 msgr2=0x7fb348151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 0x7fb348151c60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb34000e7c0 tx=0x7fb34000ec80 comp rx=0 tx=0).stop 2023-10-16T14:29:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 shutdown_connections 2023-10-16T14:29:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb32806b070 0x7fb32806d530 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815cd70 0x7fb348151c60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3480d0be0 0x7fb34815d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 --2- 172.21.15.29:0/2957839221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3480b8480 0x7fb3480b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.383+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 >> 172.21.15.29:0/2957839221 conn(0x7fb3480cf570 msgr2=0x7fb3480d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.384+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 shutdown_connections 2023-10-16T14:29:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.384+0000 7fb34d306640 1 -- 172.21.15.29:0/2957839221 wait complete. 2023-10-16T14:29:23.408 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T14:29:23.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-10-16T14:29:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.636+0000 7f9f0267c640 1 Processor -- start 2023-10-16T14:29:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.636+0000 7f9f0267c640 1 -- start start 2023-10-16T14:29:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.637+0000 7f9f0267c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9efc0cded0 0x7f9efc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.637+0000 7f9f0267c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc15d150 con 0x7f9efc0b8480 2023-10-16T14:29:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.637+0000 7f9f0267c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0d1180 con 0x7f9efc15cd70 2023-10-16T14:29:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.637+0000 7f9f0267c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc0d1460 con 0x7f9efc0cded0 2023-10-16T14:29:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9f00e79640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9efc0b8480 0x7f9efc0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41596/0 (socket says 172.21.15.29:41596) 2023-10-16T14:29:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9f00e79640 1 -- 172.21.15.29:0/3771358033 learned_addr learned my addr 172.21.15.29:0/3771358033 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9f01e7b640 1 --2- 172.21.15.29:0/3771358033 >> v2:172.21.15.29:3300/0 conn(0x7f9efc0cded0 0x7f9efc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9ef3fff640 1 -- 172.21.15.29:0/3771358033 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724414703 0 0) 0x7f9efc15d150 con 0x7f9efc0b8480 2023-10-16T14:29:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9ef3fff640 1 -- 172.21.15.29:0/3771358033 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4003520 con 0x7f9efc0b8480 2023-10-16T14:29:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9f01e7b640 1 -- 172.21.15.29:0/3771358033 >> v1:172.21.15.170:6789/0 conn(0x7f9efc15cd70 legacy=0x7f9efc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9f01e7b640 1 -- 172.21.15.29:0/3771358033 >> v1:172.21.15.29:6789/0 conn(0x7f9efc0b8480 legacy=0x7f9efc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.638+0000 7f9f01e7b640 1 -- 172.21.15.29:0/3771358033 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0d1740 con 0x7f9efc0cded0 2023-10-16T14:29:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.640+0000 7f9f01e7b640 1 --2- 172.21.15.29:0/3771358033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc0b5c90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009d90 tx=0x7f9ef402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2857b29509bc8ac0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.640+0000 7f9ef3fff640 1 -- 172.21.15.29:0/3771358033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef402ccf0 con 0x7f9efc0cded0 2023-10-16T14:29:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.640+0000 7f9ef3fff640 1 -- 172.21.15.29:0/3771358033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef402ce50 con 0x7f9efc0cded0 2023-10-16T14:29:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.640+0000 7f9ef3fff640 1 -- 172.21.15.29:0/3771358033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4034640 con 0x7f9efc0cded0 2023-10-16T14:29:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.640+0000 7f9f0267c640 1 -- 172.21.15.29:0/3771358033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 msgr2=0x7f9efc0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.640+0000 7f9f0267c640 1 --2- 172.21.15.29:0/3771358033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc0b5c90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009d90 tx=0x7f9ef402bf40 comp rx=0 tx=0).stop 2023-10-16T14:29:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.641+0000 7f9f0267c640 1 -- 172.21.15.29:0/3771358033 shutdown_connections 2023-10-16T14:29:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.641+0000 7f9f0267c640 1 --2- 172.21.15.29:0/3771358033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc0b5c90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.641+0000 7f9f0267c640 1 -- 172.21.15.29:0/3771358033 >> 172.21.15.29:0/3771358033 conn(0x7f9efc0cf570 msgr2=0x7f9efc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.641+0000 7f9f0267c640 1 -- 172.21.15.29:0/3771358033 shutdown_connections 2023-10-16T14:29:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.641+0000 7f9f0267c640 1 -- 172.21.15.29:0/3771358033 wait complete. 2023-10-16T14:29:23.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.643+0000 7f9f0267c640 1 Processor -- start 2023-10-16T14:29:23.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.643+0000 7f9f0267c640 1 -- start start 2023-10-16T14:29:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f0267c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9efc0b8480 0x7f9efc0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f0267c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f0267c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9efc15cd70 0x7f9efc1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f0267c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc152c60 con 0x7f9efc0cded0 2023-10-16T14:29:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f0267c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0cc5b0 con 0x7f9efc0b8480 2023-10-16T14:29:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f0267c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0cbbe0 con 0x7f9efc15cd70 2023-10-16T14:29:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f00e79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f00e79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35280/0 (socket says 172.21.15.29:35280) 2023-10-16T14:29:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.644+0000 7f9f00e79640 1 -- 172.21.15.29:0/2895998740 learned_addr learned my addr 172.21.15.29:0/2895998740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:23.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f0167a640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9efc0b8480 0x7f9efc0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:23.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f00e79640 1 -- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9efc15cd70 msgr2=0x7f9efc1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f00e79640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9efc15cd70 0x7f9efc1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f00e79640 1 -- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9efc0b8480 msgr2=0x7f9efc0cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:23.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f00e79640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9efc0b8480 0x7f9efc0cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:23.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f00e79640 1 -- 172.21.15.29:0/2895998740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4009a70 con 0x7f9efc0cded0 2023-10-16T14:29:23.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f0167a640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9efc0b8480 0x7f9efc0cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:23.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f00e79640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc142a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c500 tx=0x7f9ef800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:23.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8005ce0 con 0x7f9efc0cded0 2023-10-16T14:29:23.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8005e40 con 0x7f9efc0cded0 2023-10-16T14:29:23.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0cafd0 con 0x7f9efc0cded0 2023-10-16T14:29:23.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.645+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8007330 con 0x7f9efc0cded0 2023-10-16T14:29:23.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.646+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc0cb590 con 0x7f9efc0cded0 2023-10-16T14:29:23.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.647+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc0b1be0 con 0x7f9efc0cded0 2023-10-16T14:29:23.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.648+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f9ef8007490 con 0x7f9efc0cded0 2023-10-16T14:29:23.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.649+0000 7f9ef1ffb640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f9ee806b070 0x7f9ee806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:23.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.649+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9ef80890b0 con 0x7f9efc0cded0 2023-10-16T14:29:23.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.649+0000 7f9f0167a640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f9ee806b070 0x7f9ee806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:23.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.650+0000 7f9f0167a640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f9ee806b070 0x7f9ee806d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0015e0 tx=0x7f9eec009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:23.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.653+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ef8053ea0 con 0x7f9efc0cded0 2023-10-16T14:29:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:23.869+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9efc0d1d00 con 0x7f9ee806b070 2023-10-16T14:29:24.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.039+0000 7f9ef1ffb640 1 -- 172.21.15.29:0/2895998740 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9eec0012a0 con 0x7f9ee806b070 2023-10-16T14:29:24.044 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T14:29:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.042+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f9ee806b070 msgr2=0x7f9ee806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.042+0000 7f9f0267c640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f9ee806b070 0x7f9ee806d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0015e0 tx=0x7f9eec009040 comp rx=0 tx=0).stop 2023-10-16T14:29:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.042+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 msgr2=0x7f9efc142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.042+0000 7f9f0267c640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc142a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c500 tx=0x7f9ef800c9c0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 shutdown_connections 2023-10-16T14:29:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f9ee806b070 0x7f9ee806d530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9efc0cded0 0x7f9efc142a40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9efc15cd70 0x7f9efc1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 --2- 172.21.15.29:0/2895998740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9efc0b8480 0x7f9efc0cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 >> 172.21.15.29:0/2895998740 conn(0x7f9efc0cf570 msgr2=0x7f9efc15d340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:24.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 shutdown_connections 2023-10-16T14:29:24.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.043+0000 7f9f0267c640 1 -- 172.21.15.29:0/2895998740 wait complete. 2023-10-16T14:29:24.068 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T14:29:24.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-10-16T14:29:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.297+0000 7f8100924640 1 Processor -- start 2023-10-16T14:29:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.298+0000 7f8100924640 1 -- start start 2023-10-16T14:29:24.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.299+0000 7f8100924640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.299+0000 7f8100924640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80fc0ad970 0x7f80fc15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.299+0000 7f8100924640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc06c960 con 0x7f80fc15ebc0 2023-10-16T14:29:24.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.299+0000 7f8100924640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80fc15b9d0 con 0x7f80fc0ad970 2023-10-16T14:29:24.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.299+0000 7f8100924640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc15bcb0 con 0x7f80fc06c040 2023-10-16T14:29:24.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.299+0000 7f80f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80fc0ad970 0x7f80fc15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80fc0ad970 0x7f80fc15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53244/0 (socket says 172.21.15.29:53244) 2023-10-16T14:29:24.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80f3fff640 1 -- 172.21.15.29:0/3681469669 learned_addr learned my addr 172.21.15.29:0/3681469669 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:24.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80fad9d640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80f3fff640 1 -- 172.21.15.29:0/3681469669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 msgr2=0x7f80fc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80f3fff640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80f3fff640 1 -- 172.21.15.29:0/3681469669 >> v1:172.21.15.170:6789/0 conn(0x7f80fc15ebc0 legacy=0x7f80fc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80f3fff640 1 -- 172.21.15.29:0/3681469669 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc15bf90 con 0x7f80fc0ad970 2023-10-16T14:29:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.300+0000 7f80fad9d640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.301+0000 7f80f3fff640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc15e680 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f80e40098a0 tx=0x7f80e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314b7f17e1e051f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.302+0000 7f80fa59c640 1 -- 172.21.15.29:0/3681469669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4031640 con 0x7f80fc0ad970 2023-10-16T14:29:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.302+0000 7f80fa59c640 1 -- 172.21.15.29:0/3681469669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e40317a0 con 0x7f80fc0ad970 2023-10-16T14:29:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.302+0000 7f80fa59c640 1 -- 172.21.15.29:0/3681469669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4031b30 con 0x7f80fc0ad970 2023-10-16T14:29:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.302+0000 7f8100924640 1 -- 172.21.15.29:0/3681469669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 msgr2=0x7f80fc15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.302+0000 7f8100924640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc15e680 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f80e40098a0 tx=0x7f80e402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.303+0000 7f8100924640 1 -- 172.21.15.29:0/3681469669 shutdown_connections 2023-10-16T14:29:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.303+0000 7f8100924640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc15e680 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.303+0000 7f8100924640 1 --2- 172.21.15.29:0/3681469669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.303+0000 7f8100924640 1 -- 172.21.15.29:0/3681469669 >> 172.21.15.29:0/3681469669 conn(0x7f80fc0ce6a0 msgr2=0x7f80fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.303+0000 7f8100924640 1 -- 172.21.15.29:0/3681469669 shutdown_connections 2023-10-16T14:29:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.303+0000 7f8100924640 1 -- 172.21.15.29:0/3681469669 wait complete. 2023-10-16T14:29:24.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.304+0000 7f8100924640 1 Processor -- start 2023-10-16T14:29:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.305+0000 7f8100924640 1 -- start start 2023-10-16T14:29:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.305+0000 7f8100924640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.305+0000 7f8100924640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.305+0000 7f8100924640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc15ebc0 0x7f80fc151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f8100924640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc1550d0 con 0x7f80fc0ad970 2023-10-16T14:29:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f8100924640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc153ff0 con 0x7f80fc15ebc0 2023-10-16T14:29:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f8100924640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc152f50 con 0x7f80fc06c040 2023-10-16T14:29:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53252/0 (socket says 172.21.15.29:53252) 2023-10-16T14:29:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 -- 172.21.15.29:0/2424877041 learned_addr learned my addr 172.21.15.29:0/2424877041 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80fad9d640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80fb59e640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc15ebc0 0x7f80fc151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 -- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 msgr2=0x7f80fc154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 -- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc15ebc0 msgr2=0x7f80fc151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc15ebc0 0x7f80fc151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.306+0000 7f80f3fff640 1 -- 172.21.15.29:0/2424877041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40094d0 con 0x7f80fc0ad970 2023-10-16T14:29:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.307+0000 7f80f3fff640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc148d70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f80e4000c00 tx=0x7f80e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.307+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e403ac60 con 0x7f80fc0ad970 2023-10-16T14:29:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.307+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e403ade0 con 0x7f80fc0ad970 2023-10-16T14:29:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.307+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4044630 con 0x7f80fc0ad970 2023-10-16T14:29:24.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.307+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc153230 con 0x7f80fc0ad970 2023-10-16T14:29:24.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.307+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc1537f0 con 0x7f80fc0ad970 2023-10-16T14:29:24.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.308+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc06b330 con 0x7f80fc0ad970 2023-10-16T14:29:24.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.310+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f80e402f050 con 0x7f80fc0ad970 2023-10-16T14:29:24.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.310+0000 7f80f37fe640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f80dc06b070 0x7f80dc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.311+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f80e40abdc0 con 0x7f80fc0ad970 2023-10-16T14:29:24.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.311+0000 7f80fad9d640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f80dc06b070 0x7f80dc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.312+0000 7f80fad9d640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f80dc06b070 0x7f80dc06d530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00a950 tx=0x7f80ec008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:24.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.315+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f80e4076940 con 0x7f80fc0ad970 2023-10-16T14:29:24.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.535+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f80fc0cc550 con 0x7f80dc06b070 2023-10-16T14:29:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.710+0000 7f80f37fe640 1 -- 172.21.15.29:0/2424877041 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f80ec00c050 con 0x7f80dc06b070 2023-10-16T14:29:24.714 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T14:29:24.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f80dc06b070 msgr2=0x7f80dc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f80dc06b070 0x7f80dc06d530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00a950 tx=0x7f80ec008040 comp rx=0 tx=0).stop 2023-10-16T14:29:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 msgr2=0x7f80fc148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc148d70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f80e4000c00 tx=0x7f80e40046c0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 shutdown_connections 2023-10-16T14:29:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f80dc06b070 0x7f80dc06d530 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0ad970 0x7f80fc148d70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc15ebc0 0x7f80fc151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 --2- 172.21.15.29:0/2424877041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc06c040 0x7f80fc154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.713+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 >> 172.21.15.29:0/2424877041 conn(0x7f80fc0ce6a0 msgr2=0x7f80fc0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:24.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.714+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 shutdown_connections 2023-10-16T14:29:24.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.714+0000 7f8100924640 1 -- 172.21.15.29:0/2424877041 wait complete. 2023-10-16T14:29:24.738 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T14:29:24.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-10-16T14:29:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.970+0000 7f4d1141a640 1 Processor -- start 2023-10-16T14:29:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.970+0000 7f4d1141a640 1 -- start start 2023-10-16T14:29:24.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.971+0000 7f4d1141a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4d0c06c040 0x7f4d0c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.971+0000 7f4d1141a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ad970 0x7f4d0c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.971+0000 7f4d1141a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d0c15ebc0 0x7f4d0c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.971+0000 7f4d1141a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0c06c960 con 0x7f4d0c15ebc0 2023-10-16T14:29:24.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.971+0000 7f4d1141a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0c15b9d0 con 0x7f4d0c06c040 2023-10-16T14:29:24.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.971+0000 7f4d1141a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c15bcb0 con 0x7f4d0c0ad970 2023-10-16T14:29:24.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d0c15ebc0 0x7f4d0c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d0c15ebc0 0x7f4d0c15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53256/0 (socket says 172.21.15.29:53256) 2023-10-16T14:29:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 -- 172.21.15.29:0/3332742233 learned_addr learned my addr 172.21.15.29:0/3332742233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d03fff640 1 --2- 172.21.15.29:0/3332742233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ad970 0x7f4d0c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/3332742233 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c06c040 0x7f4d0c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 -- 172.21.15.29:0/3332742233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ad970 msgr2=0x7f4d0c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 --2- 172.21.15.29:0/3332742233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ad970 0x7f4d0c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 -- 172.21.15.29:0/3332742233 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c06c040 msgr2=0x7f4d0c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 --2- 172.21.15.29:0/3332742233 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c06c040 0x7f4d0c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.972+0000 7f4d10c19640 1 -- 172.21.15.29:0/3332742233 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c15bf90 con 0x7f4d0c15ebc0 2023-10-16T14:29:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.973+0000 7f4d10c19640 1 --2- 172.21.15.29:0/3332742233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c15ebc0 0x7f4d0c15f050 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009e50 tx=0x7f4d0402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc61ddf7c8e614c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.974+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3332742233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0402ca70 con 0x7f4d0c15ebc0 2023-10-16T14:29:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.974+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3332742233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0402cbd0 con 0x7f4d0c15ebc0 2023-10-16T14:29:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.974+0000 7f4d0b7fe640 1 -- 172.21.15.29:0/3332742233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d040343f0 con 0x7f4d0c15ebc0 2023-10-16T14:29:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.974+0000 7f4d1141a640 1 -- 172.21.15.29:0/3332742233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c15ebc0 msgr2=0x7f4d0c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.974+0000 7f4d1141a640 1 --2- 172.21.15.29:0/3332742233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c15ebc0 0x7f4d0c15f050 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009e50 tx=0x7f4d0402bed0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 -- 172.21.15.29:0/3332742233 shutdown_connections 2023-10-16T14:29:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 --2- 172.21.15.29:0/3332742233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c15ebc0 0x7f4d0c15f050 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 --2- 172.21.15.29:0/3332742233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c0ad970 0x7f4d0c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 --2- 172.21.15.29:0/3332742233 >> v2:172.21.15.170:3300/0 conn(0x7f4d0c06c040 0x7f4d0c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 -- 172.21.15.29:0/3332742233 >> 172.21.15.29:0/3332742233 conn(0x7f4d0c0ce6a0 msgr2=0x7f4d0c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 -- 172.21.15.29:0/3332742233 shutdown_connections 2023-10-16T14:29:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.975+0000 7f4d1141a640 1 -- 172.21.15.29:0/3332742233 wait complete. 2023-10-16T14:29:24.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.976+0000 7f4d1141a640 1 Processor -- start 2023-10-16T14:29:24.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.977+0000 7f4d1141a640 1 -- start start 2023-10-16T14:29:24.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d1141a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 0x7f4d0c0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d1141a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ad970 0x7f4d0c145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d1141a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c15ebc0 0x7f4d0c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d1141a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c156f80 con 0x7f4d0c06c040 2023-10-16T14:29:24.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d1141a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c0c5090 con 0x7f4d0c0ad970 2023-10-16T14:29:24.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d1141a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c145aa0 con 0x7f4d0c15ebc0 2023-10-16T14:29:24.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 0x7f4d0c0c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 0x7f4d0c0c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53260/0 (socket says 172.21.15.29:53260) 2023-10-16T14:29:24.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 -- 172.21.15.29:0/387588562 learned_addr learned my addr 172.21.15.29:0/387588562 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:24.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d03fff640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ad970 0x7f4d0c145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d10c19640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c15ebc0 0x7f4d0c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 -- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c15ebc0 msgr2=0x7f4d0c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c15ebc0 0x7f4d0c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 -- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ad970 msgr2=0x7f4d0c145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:24.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ad970 0x7f4d0c145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.978+0000 7f4d0bfff640 1 -- 172.21.15.29:0/387588562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d04009b30 con 0x7f4d0c06c040 2023-10-16T14:29:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.979+0000 7f4d0bfff640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 0x7f4d0c0c4b50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00d830 tx=0x7f4cfc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.979+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00be10 con 0x7f4d0c06c040 2023-10-16T14:29:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.979+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cfc0098b0 con 0x7f4d0c06c040 2023-10-16T14:29:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.979+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0c0c5aa0 con 0x7f4d0c06c040 2023-10-16T14:29:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.979+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc009a40 con 0x7f4d0c06c040 2023-10-16T14:29:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.979+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c0c6060 con 0x7f4d0c06c040 2023-10-16T14:29:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.980+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0c06b330 con 0x7f4d0c06c040 2023-10-16T14:29:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.982+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f4cfc009ba0 con 0x7f4d0c06c040 2023-10-16T14:29:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.982+0000 7f4d097fa640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec06b070 0x7f4cec06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.983+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4cfc089e90 con 0x7f4d0c06c040 2023-10-16T14:29:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.983+0000 7f4d03fff640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec06b070 0x7f4cec06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.984+0000 7f4d03fff640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec06b070 0x7f4cec06d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4006fd0 tx=0x7f4cf4008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:24.988+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4cfc054b20 con 0x7f4d0c06c040 2023-10-16T14:29:25.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.206+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d0c0cc550 con 0x7f4cec06b070 2023-10-16T14:29:25.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.375+0000 7f4d097fa640 1 -- 172.21.15.29:0/387588562 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4cf4009480 con 0x7f4cec06b070 2023-10-16T14:29:25.379 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T14:29:25.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.379+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec06b070 msgr2=0x7f4cec06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.379+0000 7f4d1141a640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec06b070 0x7f4cec06d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4006fd0 tx=0x7f4cf4008040 comp rx=0 tx=0).stop 2023-10-16T14:29:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.379+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 msgr2=0x7f4d0c0c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.379+0000 7f4d1141a640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 0x7f4d0c0c4b50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00d830 tx=0x7f4cfc00dcf0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 shutdown_connections 2023-10-16T14:29:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f4cec06b070 0x7f4cec06d530 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d0c15ebc0 0x7f4d0c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d0c0ad970 0x7f4d0c145560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 --2- 172.21.15.29:0/387588562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d0c06c040 0x7f4d0c0c4b50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 >> 172.21.15.29:0/387588562 conn(0x7f4d0c0ce6a0 msgr2=0x7f4d0c0c9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 shutdown_connections 2023-10-16T14:29:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.380+0000 7f4d1141a640 1 -- 172.21.15.29:0/387588562 wait complete. 2023-10-16T14:29:25.406 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T14:29:25.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-10-16T14:29:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.629+0000 7f2f1ec7f640 1 Processor -- start 2023-10-16T14:29:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.629+0000 7f2f1ec7f640 1 -- start start 2023-10-16T14:29:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.631+0000 7f2f1ec7f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2f18159950 0x7f2f1815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.631+0000 7f2f1ec7f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f18070450 0x7f2f180708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.631+0000 7f2f1ec7f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1806c100 0x7f2f1806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.631+0000 7f2f1ec7f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f18070e20 con 0x7f2f18070450 2023-10-16T14:29:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.631+0000 7f2f1ec7f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1815e490 con 0x7f2f18159950 2023-10-16T14:29:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.631+0000 7f2f1ec7f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1815aef0 con 0x7f2f1806c100 2023-10-16T14:29:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f18070450 0x7f2f180708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f18070450 0x7f2f180708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53268/0 (socket says 172.21.15.29:53268) 2023-10-16T14:29:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 -- 172.21.15.29:0/2815823983 learned_addr learned my addr 172.21.15.29:0/2815823983 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1e47e640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1806c100 0x7f2f1806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1dc7d640 1 --2- 172.21.15.29:0/2815823983 >> v2:172.21.15.170:3300/0 conn(0x7f2f18159950 0x7f2f1815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 -- 172.21.15.29:0/2815823983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1806c100 msgr2=0x7f2f1806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1806c100 0x7f2f1806c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 -- 172.21.15.29:0/2815823983 >> v2:172.21.15.170:3300/0 conn(0x7f2f18159950 msgr2=0x7f2f1815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/2815823983 >> v2:172.21.15.170:3300/0 conn(0x7f2f18159950 0x7f2f1815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1d47c640 1 -- 172.21.15.29:0/2815823983 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1815d910 con 0x7f2f18070450 2023-10-16T14:29:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1e47e640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1806c100 0x7f2f1806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.632+0000 7f2f1dc7d640 1 --2- 172.21.15.29:0/2815823983 >> v2:172.21.15.170:3300/0 conn(0x7f2f18159950 0x7f2f1815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:25.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.633+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f18070450 0x7f2f180708e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2f180ca950 tx=0x7f2f0802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5736ce6041afa64 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.634+0000 7f2f1cc7b640 1 -- 172.21.15.29:0/2815823983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f080317c0 con 0x7f2f18070450 2023-10-16T14:29:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.634+0000 7f2f1cc7b640 1 -- 172.21.15.29:0/2815823983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08031920 con 0x7f2f18070450 2023-10-16T14:29:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.634+0000 7f2f1cc7b640 1 -- 172.21.15.29:0/2815823983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08031cb0 con 0x7f2f18070450 2023-10-16T14:29:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.634+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/2815823983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f18070450 msgr2=0x7f2f180708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.634+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f18070450 0x7f2f180708e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2f180ca950 tx=0x7f2f0802b860 comp rx=0 tx=0).stop 2023-10-16T14:29:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/2815823983 shutdown_connections 2023-10-16T14:29:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f1806c100 0x7f2f1806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/2815823983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f18070450 0x7f2f180708e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/2815823983 >> v2:172.21.15.170:3300/0 conn(0x7f2f18159950 0x7f2f1815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/2815823983 >> 172.21.15.29:0/2815823983 conn(0x7f2f180ce6a0 msgr2=0x7f2f180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/2815823983 shutdown_connections 2023-10-16T14:29:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.635+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/2815823983 wait complete. 2023-10-16T14:29:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.636+0000 7f2f1ec7f640 1 Processor -- start 2023-10-16T14:29:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.637+0000 7f2f1ec7f640 1 -- start start 2023-10-16T14:29:25.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1ec7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 0x7f2f180c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1ec7f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f18070450 0x7f2f180c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1ec7f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f18159950 0x7f2f18150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1ec7f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1815ba30 con 0x7f2f1806c100 2023-10-16T14:29:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1ec7f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1814b880 con 0x7f2f18070450 2023-10-16T14:29:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1ec7f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180c2c50 con 0x7f2f18159950 2023-10-16T14:29:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 0x7f2f180c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 0x7f2f180c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53280/0 (socket says 172.21.15.29:53280) 2023-10-16T14:29:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 -- 172.21.15.29:0/4025187919 learned_addr learned my addr 172.21.15.29:0/4025187919 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 -- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f18159950 msgr2=0x7f2f18150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1e47e640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f18159950 0x7f2f18150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f18159950 0x7f2f18150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 -- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f18070450 msgr2=0x7f2f180c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f18070450 0x7f2f180c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f18070450 0x7f2f180c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1dc7d640 1 -- 172.21.15.29:0/4025187919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f080094d0 con 0x7f2f1806c100 2023-10-16T14:29:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.638+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f18070450 0x7f2f180c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.639+0000 7f2f1dc7d640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 0x7f2f180c4b80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400c7f0 tx=0x7f2f1400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.639+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14007a30 con 0x7f2f1806c100 2023-10-16T14:29:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.639+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f14007b90 con 0x7f2f1806c100 2023-10-16T14:29:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.639+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1400ce40 con 0x7f2f1806c100 2023-10-16T14:29:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.639+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1815b090 con 0x7f2f1806c100 2023-10-16T14:29:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.639+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1815b650 con 0x7f2f1806c100 2023-10-16T14:29:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.640+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1804e9b0 con 0x7f2f1806c100 2023-10-16T14:29:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.641+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2f140113f0 con 0x7f2f1806c100 2023-10-16T14:29:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.642+0000 7f2f0e7fc640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2efc06b0a0 0x7f2efc06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.642+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2f14089da0 con 0x7f2f1806c100 2023-10-16T14:29:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.643+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2efc06b0a0 0x7f2efc06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.644+0000 7f2f1d47c640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2efc06b0a0 0x7f2efc06d560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f08036000 tx=0x7f2f0803d040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.647+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2f14054920 con 0x7f2f1806c100 2023-10-16T14:29:25.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:25.865+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f1815ded0 con 0x7f2efc06b0a0 2023-10-16T14:29:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.039+0000 7f2f0e7fc640 1 -- 172.21.15.29:0/4025187919 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2f08031c10 con 0x7f2efc06b0a0 2023-10-16T14:29:26.044 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T14:29:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2efc06b0a0 msgr2=0x7f2efc06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2efc06b0a0 0x7f2efc06d560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f08036000 tx=0x7f2f0803d040 comp rx=0 tx=0).stop 2023-10-16T14:29:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 msgr2=0x7f2f180c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 0x7f2f180c4b80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400c7f0 tx=0x7f2f1400cbf0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 shutdown_connections 2023-10-16T14:29:26.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2efc06b0a0 0x7f2efc06d560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f18159950 0x7f2f18150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f18070450 0x7f2f180c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 --2- 172.21.15.29:0/4025187919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f1806c100 0x7f2f180c4b80 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 >> 172.21.15.29:0/4025187919 conn(0x7f2f180ce6a0 msgr2=0x7f2f180cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:26.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 shutdown_connections 2023-10-16T14:29:26.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.043+0000 7f2f1ec7f640 1 -- 172.21.15.29:0/4025187919 wait complete. 2023-10-16T14:29:26.069 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T14:29:26.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-10-16T14:29:26.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.300+0000 7f213351d640 1 Processor -- start 2023-10-16T14:29:26.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.300+0000 7f213351d640 1 -- start start 2023-10-16T14:29:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.301+0000 7f213351d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f212c160530 0x7f212c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.301+0000 7f213351d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f212c065ab0 0x7f212c160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.301+0000 7f213351d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212c0ac890 con 0x7f212c161470 2023-10-16T14:29:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.301+0000 7f213351d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212c0ae800 con 0x7f212c160530 2023-10-16T14:29:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.301+0000 7f213351d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212c0ae1f0 con 0x7f212c065ab0 2023-10-16T14:29:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f212c160530 0x7f212c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f212c160530 0x7f212c1609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53292/0 (socket says 172.21.15.29:53292) 2023-10-16T14:29:26.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 -- 172.21.15.29:0/3820858605 learned_addr learned my addr 172.21.15.29:0/3820858605 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:26.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f2130ab8640 1 --2- 172.21.15.29:0/3820858605 >> v2:172.21.15.170:3300/0 conn(0x7f212c065ab0 0x7f212c160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 -- 172.21.15.29:0/3820858605 >> v2:172.21.15.170:3300/0 conn(0x7f212c065ab0 msgr2=0x7f212c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:26.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 --2- 172.21.15.29:0/3820858605 >> v2:172.21.15.170:3300/0 conn(0x7f212c065ab0 0x7f212c160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 -- 172.21.15.29:0/3820858605 >> v1:172.21.15.170:6789/0 conn(0x7f212c161470 legacy=0x7f212c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:26.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.302+0000 7f21312b9640 1 -- 172.21.15.29:0/3820858605 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c0acae0 con 0x7f212c160530 2023-10-16T14:29:26.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.303+0000 7f21312b9640 1 --2- 172.21.15.29:0/3820858605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c1609f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f212c06f640 tx=0x7f212802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1950f533bee0d8a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:26.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.303+0000 7f2123fff640 1 -- 172.21.15.29:0/3820858605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21280317e0 con 0x7f212c160530 2023-10-16T14:29:26.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f2123fff640 1 -- 172.21.15.29:0/3820858605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2128031940 con 0x7f212c160530 2023-10-16T14:29:26.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f2123fff640 1 -- 172.21.15.29:0/3820858605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128031cd0 con 0x7f212c160530 2023-10-16T14:29:26.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 -- 172.21.15.29:0/3820858605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 msgr2=0x7f212c1609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 --2- 172.21.15.29:0/3820858605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c1609f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f212c06f640 tx=0x7f212802b880 comp rx=0 tx=0).stop 2023-10-16T14:29:26.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 -- 172.21.15.29:0/3820858605 shutdown_connections 2023-10-16T14:29:26.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 --2- 172.21.15.29:0/3820858605 >> v2:172.21.15.170:3300/0 conn(0x7f212c065ab0 0x7f212c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 --2- 172.21.15.29:0/3820858605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c1609f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 -- 172.21.15.29:0/3820858605 >> 172.21.15.29:0/3820858605 conn(0x7f212c06bc10 msgr2=0x7f212c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:26.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 -- 172.21.15.29:0/3820858605 shutdown_connections 2023-10-16T14:29:26.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.304+0000 7f213351d640 1 -- 172.21.15.29:0/3820858605 wait complete. 2023-10-16T14:29:26.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.306+0000 7f213351d640 1 Processor -- start 2023-10-16T14:29:26.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.306+0000 7f213351d640 1 -- start start 2023-10-16T14:29:26.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f213351d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c065ab0 0x7f212c150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f213351d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c0b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f213351d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c161470 0x7f212c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f213351d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c0c36f0 con 0x7f212c160530 2023-10-16T14:29:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f213351d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c0c39d0 con 0x7f212c161470 2023-10-16T14:29:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f213351d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c0c3cb0 con 0x7f212c065ab0 2023-10-16T14:29:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f2130ab8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c0b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f2130ab8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c0b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53296/0 (socket says 172.21.15.29:53296) 2023-10-16T14:29:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.307+0000 7f2130ab8640 1 -- 172.21.15.29:0/2058600077 learned_addr learned my addr 172.21.15.29:0/2058600077 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2131aba640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c161470 0x7f212c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2130ab8640 1 -- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c065ab0 msgr2=0x7f212c150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f21312b9640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c065ab0 0x7f212c150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2130ab8640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c065ab0 0x7f212c150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2130ab8640 1 -- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c161470 msgr2=0x7f212c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2130ab8640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c161470 0x7f212c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2130ab8640 1 -- 172.21.15.29:0/2058600077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21280094d0 con 0x7f212c160530 2023-10-16T14:29:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2131aba640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c161470 0x7f212c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.308+0000 7f2130ab8640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c0b3a80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f211c00d800 tx=0x7f211c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.309+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c00b6c0 con 0x7f212c160530 2023-10-16T14:29:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.309+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c00b820 con 0x7f212c160530 2023-10-16T14:29:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.309+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c00bbb0 con 0x7f212c160530 2023-10-16T14:29:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.309+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c0c3f90 con 0x7f212c160530 2023-10-16T14:29:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.309+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c0c4550 con 0x7f212c160530 2023-10-16T14:29:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.310+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212c0c4710 con 0x7f212c160530 2023-10-16T14:29:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.311+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f211c00bd10 con 0x7f212c160530 2023-10-16T14:29:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.312+0000 7f2121ffb640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f211006b070 0x7f211006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.312+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f211c08a040 con 0x7f212c160530 2023-10-16T14:29:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.313+0000 7f21312b9640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f211006b070 0x7f211006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.314+0000 7f21312b9640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f211006b070 0x7f211006d530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2128036000 tx=0x7f212803d040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.316+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f211c054d70 con 0x7f212c160530 2023-10-16T14:29:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.533+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f212c158a50 con 0x7f211006b070 2023-10-16T14:29:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.704+0000 7f2121ffb640 1 -- 172.21.15.29:0/2058600077 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f212c158a50 con 0x7f211006b070 2023-10-16T14:29:26.709 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T14:29:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f211006b070 msgr2=0x7f211006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f211006b070 0x7f211006d530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2128036000 tx=0x7f212803d040 comp rx=0 tx=0).stop 2023-10-16T14:29:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 msgr2=0x7f212c0b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c0b3a80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f211c00d800 tx=0x7f211c00dcc0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 shutdown_connections 2023-10-16T14:29:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f211006b070 0x7f211006d530 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f212c160530 0x7f212c0b3a80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f212c161470 0x7f212c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 --2- 172.21.15.29:0/2058600077 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f212c065ab0 0x7f212c150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 >> 172.21.15.29:0/2058600077 conn(0x7f212c06bc10 msgr2=0x7f212c0b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 shutdown_connections 2023-10-16T14:29:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.708+0000 7f213351d640 1 -- 172.21.15.29:0/2058600077 wait complete. 2023-10-16T14:29:26.734 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T14:29:26.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-10-16T14:29:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.963+0000 7f962afa5640 1 Processor -- start 2023-10-16T14:29:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.963+0000 7f962afa5640 1 -- start start 2023-10-16T14:29:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f962afa5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f962415d150 0x7f96240ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f962afa5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f9624160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f962afa5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9624069da0 0x7f962406a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f962afa5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962415d530 con 0x7f9624069da0 2023-10-16T14:29:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f962afa5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962406a6f0 con 0x7f962415d150 2023-10-16T14:29:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f962afa5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96240ac890 con 0x7f9624160b90 2023-10-16T14:29:26.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f9629542640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9624069da0 0x7f962406a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f9628d41640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f962415d150 0x7f96240ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f9629542640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9624069da0 0x7f962406a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53306/0 (socket says 172.21.15.29:53306) 2023-10-16T14:29:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f9629542640 1 -- 172.21.15.29:0/762384144 learned_addr learned my addr 172.21.15.29:0/762384144 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.965+0000 7f9623fff640 1 --2- 172.21.15.29:0/762384144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f9624160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.966+0000 7f9629542640 1 -- 172.21.15.29:0/762384144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 msgr2=0x7f9624160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.966+0000 7f9629542640 1 --2- 172.21.15.29:0/762384144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f9624160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.966+0000 7f9629542640 1 -- 172.21.15.29:0/762384144 >> v2:172.21.15.170:3300/0 conn(0x7f962415d150 msgr2=0x7f96240ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.966+0000 7f9629542640 1 --2- 172.21.15.29:0/762384144 >> v2:172.21.15.170:3300/0 conn(0x7f962415d150 0x7f96240ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.966+0000 7f9629542640 1 -- 172.21.15.29:0/762384144 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624161650 con 0x7f9624069da0 2023-10-16T14:29:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.967+0000 7f9629542640 1 --2- 172.21.15.29:0/762384144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f962406a1b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f96240b15a0 tx=0x7f961c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df30c28477a57d3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f96237fe640 1 -- 172.21.15.29:0/762384144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c02ca70 con 0x7f9624069da0 2023-10-16T14:29:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f96237fe640 1 -- 172.21.15.29:0/762384144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c02cbd0 con 0x7f9624069da0 2023-10-16T14:29:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f96237fe640 1 -- 172.21.15.29:0/762384144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c0343f0 con 0x7f9624069da0 2023-10-16T14:29:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 -- 172.21.15.29:0/762384144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 msgr2=0x7f962406a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 --2- 172.21.15.29:0/762384144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f962406a1b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f96240b15a0 tx=0x7f961c02bed0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 -- 172.21.15.29:0/762384144 shutdown_connections 2023-10-16T14:29:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 --2- 172.21.15.29:0/762384144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f962406a1b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 --2- 172.21.15.29:0/762384144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f9624160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 --2- 172.21.15.29:0/762384144 >> v2:172.21.15.170:3300/0 conn(0x7f962415d150 0x7f96240ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.968+0000 7f962afa5640 1 -- 172.21.15.29:0/762384144 >> 172.21.15.29:0/762384144 conn(0x7f96240ce9b0 msgr2=0x7f96240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.969+0000 7f962afa5640 1 -- 172.21.15.29:0/762384144 shutdown_connections 2023-10-16T14:29:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.969+0000 7f962afa5640 1 -- 172.21.15.29:0/762384144 wait complete. 2023-10-16T14:29:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.970+0000 7f962afa5640 1 Processor -- start 2023-10-16T14:29:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.971+0000 7f962afa5640 1 -- start start 2023-10-16T14:29:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f962afa5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f96240c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f962afa5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f962415d150 0x7f96240b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f962afa5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f96240ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f962afa5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96240b0940 con 0x7f9624069da0 2023-10-16T14:29:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f962afa5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96240b0b90 con 0x7f962415d150 2023-10-16T14:29:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f962afa5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96240b0e70 con 0x7f9624160b90 2023-10-16T14:29:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f96240c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f96240c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53312/0 (socket says 172.21.15.29:53312) 2023-10-16T14:29:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 -- 172.21.15.29:0/4263781537 learned_addr learned my addr 172.21.15.29:0/4263781537 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 -- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 msgr2=0x7f96240ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f96240ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 -- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f962415d150 msgr2=0x7f96240b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f962415d150 0x7f96240b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.972+0000 7f9628d41640 1 -- 172.21.15.29:0/4263781537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c009b30 con 0x7f9624069da0 2023-10-16T14:29:26.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.973+0000 7f9628d41640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f96240c8340 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f961800e890 tx=0x7f961800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:26.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.973+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800c940 con 0x7f9624069da0 2023-10-16T14:29:26.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.973+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961800caa0 con 0x7f9624069da0 2023-10-16T14:29:26.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.973+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800ce30 con 0x7f9624069da0 2023-10-16T14:29:26.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.973+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96240b3910 con 0x7f9624069da0 2023-10-16T14:29:26.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.973+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96241f69e0 con 0x7f9624069da0 2023-10-16T14:29:26.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.975+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962404e160 con 0x7f9624069da0 2023-10-16T14:29:26.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.975+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f9618013470 con 0x7f9624069da0 2023-10-16T14:29:26.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.976+0000 7f96217fa640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f960806b070 0x7f960806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:26.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.976+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9618089c70 con 0x7f9624069da0 2023-10-16T14:29:26.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.977+0000 7f9623fff640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f960806b070 0x7f960806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:26.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.978+0000 7f9623fff640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f960806b070 0x7f960806d530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9610006a90 tx=0x7f9610006a20 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:26.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:26.980+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9618054990 con 0x7f9624069da0 2023-10-16T14:29:27.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.201+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f96240b3790 con 0x7f960806b070 2023-10-16T14:29:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.375+0000 7f96217fa640 1 -- 172.21.15.29:0/4263781537 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f96100096c0 con 0x7f960806b070 2023-10-16T14:29:27.380 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T14:29:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f960806b070 msgr2=0x7f960806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f960806b070 0x7f960806d530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9610006a90 tx=0x7f9610006a20 comp rx=0 tx=0).stop 2023-10-16T14:29:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 msgr2=0x7f96240c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f96240c8340 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f961800e890 tx=0x7f961800ed50 comp rx=0 tx=0).stop 2023-10-16T14:29:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 shutdown_connections 2023-10-16T14:29:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f960806b070 0x7f960806d530 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9624160b90 0x7f96240ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f962415d150 0x7f96240b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 --2- 172.21.15.29:0/4263781537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9624069da0 0x7f96240c8340 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 >> 172.21.15.29:0/4263781537 conn(0x7f96240ce9b0 msgr2=0x7f96240c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 shutdown_connections 2023-10-16T14:29:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.379+0000 7f962afa5640 1 -- 172.21.15.29:0/4263781537 wait complete. 2023-10-16T14:29:27.403 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T14:29:27.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-10-16T14:29:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.626+0000 7f65cff8a640 1 Processor -- start 2023-10-16T14:29:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.626+0000 7f65cff8a640 1 -- start start 2023-10-16T14:29:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cff8a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65c815d150 0x7f65c80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cff8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f65c8160b90 0x7f65c8160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cff8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c8069da0 0x7f65c806a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cff8a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c815d530 con 0x7f65c815d150 2023-10-16T14:29:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cff8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c806a730 con 0x7f65c8160b90 2023-10-16T14:29:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cff8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80ac890 con 0x7f65c8069da0 2023-10-16T14:29:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cdd26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65c815d150 0x7f65c80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.628+0000 7f65cdd26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65c815d150 0x7f65c80ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53328/0 (socket says 172.21.15.29:53328) 2023-10-16T14:29:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cdd26640 1 -- 172.21.15.29:0/1804333345 learned_addr learned my addr 172.21.15.29:0/1804333345 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:27.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cd525640 1 --2- 172.21.15.29:0/1804333345 >> v2:172.21.15.170:3300/0 conn(0x7f65c8160b90 0x7f65c8160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65ce527640 1 --2- 172.21.15.29:0/1804333345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c8069da0 0x7f65c806a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cdd26640 1 -- 172.21.15.29:0/1804333345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c8069da0 msgr2=0x7f65c806a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:27.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cdd26640 1 --2- 172.21.15.29:0/1804333345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c8069da0 0x7f65c806a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cdd26640 1 -- 172.21.15.29:0/1804333345 >> v2:172.21.15.170:3300/0 conn(0x7f65c8160b90 msgr2=0x7f65c8160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:27.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cdd26640 1 --2- 172.21.15.29:0/1804333345 >> v2:172.21.15.170:3300/0 conn(0x7f65c8160b90 0x7f65c8160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cdd26640 1 -- 172.21.15.29:0/1804333345 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c8161650 con 0x7f65c815d150 2023-10-16T14:29:27.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.629+0000 7f65cd525640 1 --2- 172.21.15.29:0/1804333345 >> v2:172.21.15.170:3300/0 conn(0x7f65c8160b90 0x7f65c8160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:27.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.631+0000 7f65cdd26640 1 --2- 172.21.15.29:0/1804333345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c815d150 0x7f65c80ad350 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f65c80cf990 tx=0x7f65c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7912054f39bd9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.631+0000 7f65ccd24640 1 -- 172.21.15.29:0/1804333345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c40317e0 con 0x7f65c815d150 2023-10-16T14:29:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.631+0000 7f65ccd24640 1 -- 172.21.15.29:0/1804333345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4031940 con 0x7f65c815d150 2023-10-16T14:29:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.631+0000 7f65ccd24640 1 -- 172.21.15.29:0/1804333345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4031cd0 con 0x7f65c815d150 2023-10-16T14:29:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 -- 172.21.15.29:0/1804333345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c815d150 msgr2=0x7f65c80ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 --2- 172.21.15.29:0/1804333345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c815d150 0x7f65c80ad350 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f65c80cf990 tx=0x7f65c402b880 comp rx=0 tx=0).stop 2023-10-16T14:29:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 -- 172.21.15.29:0/1804333345 shutdown_connections 2023-10-16T14:29:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 --2- 172.21.15.29:0/1804333345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c8069da0 0x7f65c806a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 --2- 172.21.15.29:0/1804333345 >> v2:172.21.15.170:3300/0 conn(0x7f65c8160b90 0x7f65c8160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 --2- 172.21.15.29:0/1804333345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c815d150 0x7f65c80ad350 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 -- 172.21.15.29:0/1804333345 >> 172.21.15.29:0/1804333345 conn(0x7f65c80ce9b0 msgr2=0x7f65c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 -- 172.21.15.29:0/1804333345 shutdown_connections 2023-10-16T14:29:27.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.632+0000 7f65cff8a640 1 -- 172.21.15.29:0/1804333345 wait complete. 2023-10-16T14:29:27.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.634+0000 7f65cff8a640 1 Processor -- start 2023-10-16T14:29:27.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.634+0000 7f65cff8a640 1 -- start start 2023-10-16T14:29:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.635+0000 7f65cff8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c8069da0 0x7f65c8151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.635+0000 7f65cff8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815d150 0x7f65c80ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.635+0000 7f65cff8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 0x7f65c81f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.635+0000 7f65cff8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8151ad0 con 0x7f65c8160b90 2023-10-16T14:29:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.635+0000 7f65cff8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80aec90 con 0x7f65c8069da0 2023-10-16T14:29:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.635+0000 7f65cff8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80aef70 con 0x7f65c815d150 2023-10-16T14:29:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 0x7f65c81f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 0x7f65c81f61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53342/0 (socket says 172.21.15.29:53342) 2023-10-16T14:29:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 -- 172.21.15.29:0/4117783869 learned_addr learned my addr 172.21.15.29:0/4117783869 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:27.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65cd525640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815d150 0x7f65c80ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65cdd26640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c8069da0 0x7f65c8151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 -- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815d150 msgr2=0x7f65c80ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:27.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815d150 0x7f65c80ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 -- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c8069da0 msgr2=0x7f65c8151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:27.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c8069da0 0x7f65c8151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:27.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 -- 172.21.15.29:0/4117783869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c40094d0 con 0x7f65c8160b90 2023-10-16T14:29:27.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65cdd26640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c8069da0 0x7f65c8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:27.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.636+0000 7f65ce527640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 0x7f65c81f61d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f65c0007d20 tx=0x7f65c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:27.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.637+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0014420 con 0x7f65c8160b90 2023-10-16T14:29:27.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.637+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c00145a0 con 0x7f65c8160b90 2023-10-16T14:29:27.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.637+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0014950 con 0x7f65c8160b90 2023-10-16T14:29:27.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.637+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c80af250 con 0x7f65c8160b90 2023-10-16T14:29:27.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.637+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c815e880 con 0x7f65c8160b90 2023-10-16T14:29:27.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.638+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c815eaa0 con 0x7f65c8160b90 2023-10-16T14:29:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.639+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f65c0014ab0 con 0x7f65c8160b90 2023-10-16T14:29:27.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.640+0000 7f65be7fc640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65ac06b070 0x7f65ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.640+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f65c008a3f0 con 0x7f65c8160b90 2023-10-16T14:29:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.641+0000 7f65cdd26640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65ac06b070 0x7f65ac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.642+0000 7f65cdd26640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65ac06b070 0x7f65ac06d530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65c4036000 tx=0x7f65c403d040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:27.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.644+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65c0055080 con 0x7f65c8160b90 2023-10-16T14:29:27.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:27.861+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f65c80af0e0 con 0x7f65ac06b070 2023-10-16T14:29:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.037+0000 7f65be7fc640 1 -- 172.21.15.29:0/4117783869 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f65c80af0e0 con 0x7f65ac06b070 2023-10-16T14:29:28.041 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T14:29:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.040+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65ac06b070 msgr2=0x7f65ac06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.040+0000 7f65cff8a640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65ac06b070 0x7f65ac06d530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65c4036000 tx=0x7f65c403d040 comp rx=0 tx=0).stop 2023-10-16T14:29:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.040+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 msgr2=0x7f65c81f61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.040+0000 7f65cff8a640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 0x7f65c81f61d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f65c0007d20 tx=0x7f65c000e4b0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 shutdown_connections 2023-10-16T14:29:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f65ac06b070 0x7f65ac06d530 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8160b90 0x7f65c81f61d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65c815d150 0x7f65c80ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 --2- 172.21.15.29:0/4117783869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65c8069da0 0x7f65c8151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 >> 172.21.15.29:0/4117783869 conn(0x7f65c80ce9b0 msgr2=0x7f65c80c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 shutdown_connections 2023-10-16T14:29:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.041+0000 7f65cff8a640 1 -- 172.21.15.29:0/4117783869 wait complete. 2023-10-16T14:29:28.066 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T14:29:28.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-10-16T14:29:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.278+0000 7f93f2e46640 1 Processor -- start 2023-10-16T14:29:28.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.278+0000 7f93f2e46640 1 -- start start 2023-10-16T14:29:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.280+0000 7f93f2e46640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f93ec0caac0 0x7f93ec0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.280+0000 7f93f2e46640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0cb030 con 0x7f93ec15ad70 2023-10-16T14:29:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.280+0000 7f93f2e46640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0c91c0 con 0x7f93ec06a4b0 2023-10-16T14:29:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.280+0000 7f93f2e46640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec0c94a0 con 0x7f93ec0caac0 2023-10-16T14:29:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.280+0000 7f93f0be2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f93ec15ad70 0x7f93ec069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54324/0 (socket says 172.21.15.29:54324) 2023-10-16T14:29:28.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.280+0000 7f93f0be2640 1 -- 172.21.15.29:0/3171261427 learned_addr learned my addr 172.21.15.29:0/3171261427 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:28.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93f13e3640 1 --2- 172.21.15.29:0/3171261427 >> v2:172.21.15.170:3300/0 conn(0x7f93ec0caac0 0x7f93ec0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938827756 0 0) 0x7f93ec0cb030 con 0x7f93ec15ad70 2023-10-16T14:29:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d4003520 con 0x7f93ec15ad70 2023-10-16T14:29:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016550399 0 0) 0x7f93d4003520 con 0x7f93ec15ad70 2023-10-16T14:29:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 >> v2:172.21.15.170:3300/0 conn(0x7f93ec0caac0 msgr2=0x7f93ec0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 --2- 172.21.15.29:0/3171261427 >> v2:172.21.15.170:3300/0 conn(0x7f93ec0caac0 0x7f93ec0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 >> v1:172.21.15.170:6789/0 conn(0x7f93ec06a4b0 legacy=0x7f93ec0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0c9780 con 0x7f93ec15ad70 2023-10-16T14:29:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f93e80019f0 con 0x7f93ec15ad70 2023-10-16T14:29:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 >> v1:172.21.15.29:6789/0 conn(0x7f93ec15ad70 legacy=0x7f93ec069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93d40047c0 0x7f93d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93d4005180 0x7f93d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0c94a0 con 0x7f93d40039c0 2023-10-16T14:29:28.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d400c080 con 0x7f93d4005180 2023-10-16T14:29:28.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.281+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d400c240 con 0x7f93d40047c0 2023-10-16T14:29:28.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f0be2640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f13e3640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93d4005180 0x7f93d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93e3fff640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93d40047c0 0x7f93d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f0be2640 1 -- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93d40047c0 msgr2=0x7f93d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:28.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f0be2640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93d40047c0 0x7f93d4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f0be2640 1 -- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93d4005180 msgr2=0x7f93d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:28.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f0be2640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93d4005180 0x7f93d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f13e3640 1 -- 172.21.15.29:0/3171261427 reap_dead start 2023-10-16T14:29:28.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.282+0000 7f93f0be2640 1 -- 172.21.15.29:0/3171261427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4003520 con 0x7f93d40039c0 2023-10-16T14:29:28.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.283+0000 7f93f0be2640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93d4003de0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f93e800a760 tx=0x7f93d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1990011e14d4d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.283+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8034630 con 0x7f93d40039c0 2023-10-16T14:29:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e802f070 con 0x7f93d40039c0 2023-10-16T14:29:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93e37fe640 1 -- 172.21.15.29:0/3171261427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8034a50 con 0x7f93d40039c0 2023-10-16T14:29:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 -- 172.21.15.29:0/3171261427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 msgr2=0x7f93d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93d4003de0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f93e800a760 tx=0x7f93d40057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 -- 172.21.15.29:0/3171261427 shutdown_connections 2023-10-16T14:29:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 --2- 172.21.15.29:0/3171261427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93d4003de0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 -- 172.21.15.29:0/3171261427 >> 172.21.15.29:0/3171261427 conn(0x7f93ec0ce9b0 msgr2=0x7f93ec0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 -- 172.21.15.29:0/3171261427 shutdown_connections 2023-10-16T14:29:28.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.284+0000 7f93f2e46640 1 -- 172.21.15.29:0/3171261427 wait complete. 2023-10-16T14:29:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.286+0000 7f93f2e46640 1 Processor -- start 2023-10-16T14:29:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.286+0000 7f93f2e46640 1 -- start start 2023-10-16T14:29:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.287+0000 7f93f2e46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93ec0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.287+0000 7f93f2e46640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93ec0acdd0 0x7f93ec1f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.287+0000 7f93f2e46640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93ec1f5cc0 0x7f93ec159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.287+0000 7f93f2e46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec1f6140 con 0x7f93d40039c0 2023-10-16T14:29:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.287+0000 7f93f2e46640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0adf60 con 0x7f93ec0acdd0 2023-10-16T14:29:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.287+0000 7f93f2e46640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0ad350 con 0x7f93ec1f5cc0 2023-10-16T14:29:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93ec0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93ec0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53366/0 (socket says 172.21.15.29:53366) 2023-10-16T14:29:28.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 -- 172.21.15.29:0/51064132 learned_addr learned my addr 172.21.15.29:0/51064132 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93e3fff640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93ec0acdd0 0x7f93ec1f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f13e3640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93ec1f5cc0 0x7f93ec159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 -- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93ec1f5cc0 msgr2=0x7f93ec159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93ec1f5cc0 0x7f93ec159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 -- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93ec0acdd0 msgr2=0x7f93ec1f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93ec0acdd0 0x7f93ec1f5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f0be2640 1 -- 172.21.15.29:0/51064132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec0ad630 con 0x7f93d40039c0 2023-10-16T14:29:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.288+0000 7f93f13e3640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93ec1f5cc0 0x7f93ec159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.289+0000 7f93f0be2640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93ec0ac890 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f93e800a760 tx=0x7f93e8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:28.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.289+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8004ea0 con 0x7f93d40039c0 2023-10-16T14:29:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.289+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e803f470 con 0x7f93d40039c0 2023-10-16T14:29:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.289+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e803f7e0 con 0x7f93d40039c0 2023-10-16T14:29:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.289+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec0ad910 con 0x7f93d40039c0 2023-10-16T14:29:28.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.289+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec0c8db0 con 0x7f93d40039c0 2023-10-16T14:29:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.290+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec04e160 con 0x7f93d40039c0 2023-10-16T14:29:28.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.291+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f93e8031030 con 0x7f93d40039c0 2023-10-16T14:29:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.292+0000 7f93e17fa640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f93d806b070 0x7f93d806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.292+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f93e80ab5e0 con 0x7f93d40039c0 2023-10-16T14:29:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.293+0000 7f93e3fff640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f93d806b070 0x7f93d806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.294+0000 7f93e3fff640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f93d806b070 0x7f93d806d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0b3990 tx=0x7f93dc009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:28.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.297+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f93e8076210 con 0x7f93d40039c0 2023-10-16T14:29:28.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.517+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f93ec0c9ca0 con 0x7f93d806b070 2023-10-16T14:29:28.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.692+0000 7f93e17fa640 1 -- 172.21.15.29:0/51064132 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f93dc00d050 con 0x7f93d806b070 2023-10-16T14:29:28.697 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T14:29:28.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f93d806b070 msgr2=0x7f93d806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f93d806b070 0x7f93d806d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0b3990 tx=0x7f93dc009040 comp rx=0 tx=0).stop 2023-10-16T14:29:28.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 msgr2=0x7f93ec0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:28.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93ec0ac890 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f93e800a760 tx=0x7f93e8001cb0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 shutdown_connections 2023-10-16T14:29:28.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f93d806b070 0x7f93d806d530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f93ec1f5cc0 0x7f93ec159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93ec0acdd0 0x7f93ec1f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 --2- 172.21.15.29:0/51064132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93d40039c0 0x7f93ec0ac890 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:28.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 >> 172.21.15.29:0/51064132 conn(0x7f93ec0ce9b0 msgr2=0x7f93ec0cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:28.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.696+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 shutdown_connections 2023-10-16T14:29:28.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:28.697+0000 7f93f2e46640 1 -- 172.21.15.29:0/51064132 wait complete. 2023-10-16T14:29:28.719 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T14:29:28.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:28.720 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T14:29:28.778 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:28.779 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:28.779 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:28.779 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_01_UTC 2023-10-16T14:29:28.779 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:28.780 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:28.780 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:28.780 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:28.780 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:28.780 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:28.781 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:28.781 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:28.782 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_18_00_UTC', 'scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_01_UTC', 'scheduled-2023-10-16-14_22_01_UTC', 'scheduled-2023-10-16-14_23_01_UTC', 'scheduled-2023-10-16-14_24_00_UTC', 'scheduled-2023-10-16-14_25_00_UTC', 'scheduled-2023-10-16-14_26_00_UTC', 'scheduled-2023-10-16-14_27_00_UTC', 'scheduled-2023-10-16-14_28_00_UTC', 'scheduled-2023-10-16-14_29_00_UTC'] 2023-10-16T14:29:28.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-10-16T14:29:29.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.001+0000 7f021a89e640 1 Processor -- start 2023-10-16T14:29:29.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.001+0000 7f021a89e640 1 -- start start 2023-10-16T14:29:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.003+0000 7f021a89e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f021415dde0 0x7f02140709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.003+0000 7f021a89e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0214070f20 0x7f02140ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.003+0000 7f021a89e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214160a90 con 0x7f021415c9f0 2023-10-16T14:29:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.003+0000 7f021a89e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02140cdec0 con 0x7f0214070f20 2023-10-16T14:29:29.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.003+0000 7f021a89e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02140cd900 con 0x7f021415dde0 2023-10-16T14:29:29.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.003+0000 7f021a09d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0214070f20 0x7f02140ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021a09d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0214070f20 0x7f02140ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53380/0 (socket says 172.21.15.29:53380) 2023-10-16T14:29:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021a09d640 1 -- 172.21.15.29:0/1865870940 learned_addr learned my addr 172.21.15.29:0/1865870940 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021909b640 1 --2- 172.21.15.29:0/1865870940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f021415dde0 0x7f02140709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021a09d640 1 -- 172.21.15.29:0/1865870940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f021415dde0 msgr2=0x7f02140709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:29.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021a09d640 1 --2- 172.21.15.29:0/1865870940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f021415dde0 0x7f02140709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021a09d640 1 -- 172.21.15.29:0/1865870940 >> v1:172.21.15.170:6789/0 conn(0x7f021415c9f0 legacy=0x7f02140c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.004+0000 7f021a09d640 1 -- 172.21.15.29:0/1865870940 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140cdbe0 con 0x7f0214070f20 2023-10-16T14:29:29.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.006+0000 7f021a09d640 1 --2- 172.21.15.29:0/1865870940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214070f20 0x7f02140ac890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f020c009e50 tx=0x7f020c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ffd24e1a936afc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:29.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.006+0000 7f021889a640 1 -- 172.21.15.29:0/1865870940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c02cb00 con 0x7f0214070f20 2023-10-16T14:29:29.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.006+0000 7f021889a640 1 -- 172.21.15.29:0/1865870940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c02cc60 con 0x7f0214070f20 2023-10-16T14:29:29.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.006+0000 7f021889a640 1 -- 172.21.15.29:0/1865870940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c0343f0 con 0x7f0214070f20 2023-10-16T14:29:29.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 -- 172.21.15.29:0/1865870940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214070f20 msgr2=0x7f02140ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 --2- 172.21.15.29:0/1865870940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214070f20 0x7f02140ac890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f020c009e50 tx=0x7f020c02bf60 comp rx=0 tx=0).stop 2023-10-16T14:29:29.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 -- 172.21.15.29:0/1865870940 shutdown_connections 2023-10-16T14:29:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 --2- 172.21.15.29:0/1865870940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214070f20 0x7f02140ac890 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 --2- 172.21.15.29:0/1865870940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f021415dde0 0x7f02140709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 -- 172.21.15.29:0/1865870940 >> 172.21.15.29:0/1865870940 conn(0x7f02140ce6a0 msgr2=0x7f02140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 -- 172.21.15.29:0/1865870940 shutdown_connections 2023-10-16T14:29:29.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.007+0000 7f021a89e640 1 -- 172.21.15.29:0/1865870940 wait complete. 2023-10-16T14:29:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.009+0000 7f021a89e640 1 Processor -- start 2023-10-16T14:29:29.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.009+0000 7f021a89e640 1 -- start start 2023-10-16T14:29:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a89e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214070f20 0x7f021406c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a89e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f021415c9f0 0x7f02140c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a89e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 0x7f021415ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a89e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021406c1e0 con 0x7f021415dde0 2023-10-16T14:29:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a89e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021406b5d0 con 0x7f021415c9f0 2023-10-16T14:29:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a89e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021406b8b0 con 0x7f0214070f20 2023-10-16T14:29:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a09d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 0x7f021415ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a09d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 0x7f021415ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53382/0 (socket says 172.21.15.29:53382) 2023-10-16T14:29:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.010+0000 7f021a09d640 1 -- 172.21.15.29:0/1081928582 learned_addr learned my addr 172.21.15.29:0/1081928582 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:29.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021909b640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f021415c9f0 0x7f02140c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021989c640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214070f20 0x7f021406c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a09d640 1 -- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214070f20 msgr2=0x7f021406c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a09d640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214070f20 0x7f021406c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a09d640 1 -- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f021415c9f0 msgr2=0x7f02140c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a09d640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f021415c9f0 0x7f02140c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a09d640 1 -- 172.21.15.29:0/1081928582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c009b30 con 0x7f021415dde0 2023-10-16T14:29:29.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a09d640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 0x7f021415ad00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f020c009e50 tx=0x7f020c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:29.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c009510 con 0x7f021415dde0 2023-10-16T14:29:29.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c03e450 con 0x7f021415dde0 2023-10-16T14:29:29.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021406bb90 con 0x7f021415dde0 2023-10-16T14:29:29.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.011+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c03e800 con 0x7f021415dde0 2023-10-16T14:29:29.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.012+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021406cbb0 con 0x7f021415dde0 2023-10-16T14:29:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.013+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021406c390 con 0x7f021415dde0 2023-10-16T14:29:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.014+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f020c03e960 con 0x7f021415dde0 2023-10-16T14:29:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.015+0000 7f020a7fc640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f01f806b0a0 0x7f01f806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.015+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f020c0ac870 con 0x7f021415dde0 2023-10-16T14:29:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.015+0000 7f021989c640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f01f806b0a0 0x7f01f806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.016+0000 7f021989c640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f01f806b0a0 0x7f01f806d560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02141f5690 tx=0x7f0210009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.019+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f021406c390 con 0x7f021415dde0 2023-10-16T14:29:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.238+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f02140704c0 con 0x7f01f806b0a0 2023-10-16T14:29:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.256+0000 7f020a7fc640 1 -- 172.21.15.29:0/1081928582 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f02100012e0 con 0x7f01f806b0a0 2023-10-16T14:29:29.260 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:29.261 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:55", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f01f806b0a0 msgr2=0x7f01f806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f01f806b0a0 0x7f01f806d560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02141f5690 tx=0x7f0210009040 comp rx=0 tx=0).stop 2023-10-16T14:29:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 msgr2=0x7f021415ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 0x7f021415ad00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f020c009e50 tx=0x7f020c004a40 comp rx=0 tx=0).stop 2023-10-16T14:29:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 shutdown_connections 2023-10-16T14:29:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f01f806b0a0 0x7f01f806d560 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021415dde0 0x7f021415ad00 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.259+0000 7f021a89e640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f021415c9f0 0x7f02140c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.260+0000 7f021a89e640 1 --2- 172.21.15.29:0/1081928582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214070f20 0x7f021406c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.260+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 >> 172.21.15.29:0/1081928582 conn(0x7f02140ce6a0 msgr2=0x7f02140cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.260+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 shutdown_connections 2023-10-16T14:29:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.260+0000 7f021a89e640 1 -- 172.21.15.29:0/1081928582 wait complete. 2023-10-16T14:29:29.283 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:55", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:29.283 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:17:55', 'first': '2023-10-16T14:18:00', 'last': '2023-10-16T14:29:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:29.284 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T14:29:29.284 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T14:29:29.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:29.285 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T14:29:29.341 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:29.341 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:29.342 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:29.342 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:29.342 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:29.342 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:29.342 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:29.343 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:29.345 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:29.345 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:29.345 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:29.346 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:29.347 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_18_00_UTC', 'scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_00_UTC', 'scheduled-2023-10-16-14_24_00_UTC', 'scheduled-2023-10-16-14_25_00_UTC', 'scheduled-2023-10-16-14_26_01_UTC', 'scheduled-2023-10-16-14_27_00_UTC', 'scheduled-2023-10-16-14_28_00_UTC', 'scheduled-2023-10-16-14_29_01_UTC'] 2023-10-16T14:29:29.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-10-16T14:29:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.573+0000 7fc5bfac2640 1 Processor -- start 2023-10-16T14:29:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.573+0000 7fc5bfac2640 1 -- start start 2023-10-16T14:29:29.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5bfac2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5b8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5bfac2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cb030 con 0x7fc5b80caac0 2023-10-16T14:29:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5bfac2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80c91c0 con 0x7fc5b80c9ca0 2023-10-16T14:29:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5bfac2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c94a0 con 0x7fc5b815ad70 2023-10-16T14:29:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5be05f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5b80caac0 0x7fc5b80ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54332/0 (socket says 172.21.15.29:54332) 2023-10-16T14:29:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5be05f640 1 -- 172.21.15.29:0/3978667626 learned_addr learned my addr 172.21.15.29:0/3978667626 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5bd85e640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5b8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.575+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851172221 0 0) 0x7fc5b80cb030 con 0x7fc5b80caac0 2023-10-16T14:29:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4003520 con 0x7fc5b80caac0 2023-10-16T14:29:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600254931 0 0) 0x7fc5a4003520 con 0x7fc5b80caac0 2023-10-16T14:29:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 msgr2=0x7fc5b8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5b8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 >> v1:172.21.15.170:6789/0 conn(0x7fc5b80c9ca0 legacy=0x7fc5b80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80c9780 con 0x7fc5b80caac0 2023-10-16T14:29:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc5b00019f0 con 0x7fc5b80caac0 2023-10-16T14:29:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 >> v1:172.21.15.29:6789/0 conn(0x7fc5b80caac0 legacy=0x7fc5b80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5a40039a0 0x7fc5a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c94a0 con 0x7fc5a40047c0 2023-10-16T14:29:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a400ba60 con 0x7fc5a40039a0 2023-10-16T14:29:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a400bc20 con 0x7fc5b815ad70 2023-10-16T14:29:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.576+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd85e640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5a40039a0 0x7fc5a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd05d640 1 -- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 msgr2=0x7fc5a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5be05f640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd05d640 1 -- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5a40039a0 msgr2=0x7fc5a4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:29.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5a40039a0 0x7fc5a4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd05d640 1 -- 172.21.15.29:0/3978667626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4003520 con 0x7fc5a40047c0 2023-10-16T14:29:29.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5bd85e640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5a40039a0 0x7fc5a4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:29.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.577+0000 7fc5be05f640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815ad70 0x7fc5a4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.578+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5a4004c40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8001720 tx=0x7fc5a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ae9364a3e4ca88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80052d0 con 0x7fc5a40047c0 2023-10-16T14:29:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a802e050 con 0x7fc5a40047c0 2023-10-16T14:29:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bc85c640 1 -- 172.21.15.29:0/3978667626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80035f0 con 0x7fc5a40047c0 2023-10-16T14:29:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bfac2640 1 -- 172.21.15.29:0/3978667626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 msgr2=0x7fc5a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/3978667626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5a4004c40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8001720 tx=0x7fc5a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5be05f640 1 -- 172.21.15.29:0/3978667626 reap_dead start 2023-10-16T14:29:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bfac2640 1 -- 172.21.15.29:0/3978667626 shutdown_connections 2023-10-16T14:29:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.579+0000 7fc5bfac2640 1 -- 172.21.15.29:0/3978667626 >> 172.21.15.29:0/3978667626 conn(0x7fc5b80ce9b0 msgr2=0x7fc5b80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.580+0000 7fc5bfac2640 1 -- 172.21.15.29:0/3978667626 shutdown_connections 2023-10-16T14:29:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.580+0000 7fc5bfac2640 1 -- 172.21.15.29:0/3978667626 wait complete. 2023-10-16T14:29:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.581+0000 7fc5bfac2640 1 Processor -- start 2023-10-16T14:29:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.581+0000 7fc5bfac2640 1 -- start start 2023-10-16T14:29:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.582+0000 7fc5bfac2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5b8156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.582+0000 7fc5bfac2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5b815d4b0 0x7fc5b815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.582+0000 7fc5bfac2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815d930 0x7fc5b815cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.582+0000 7fc5bfac2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8156590 con 0x7fc5a40047c0 2023-10-16T14:29:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.582+0000 7fc5bfac2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c6a50 con 0x7fc5b815d4b0 2023-10-16T14:29:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.582+0000 7fc5bfac2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c5e40 con 0x7fc5b815d930 2023-10-16T14:29:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5b8156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5b8156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53400/0 (socket says 172.21.15.29:53400) 2023-10-16T14:29:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 -- 172.21.15.29:0/4116825002 learned_addr learned my addr 172.21.15.29:0/4116825002 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5b815d4b0 0x7fc5b815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5be05f640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815d930 0x7fc5b815cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 -- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815d930 msgr2=0x7fc5b815cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815d930 0x7fc5b815cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 -- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5b815d4b0 msgr2=0x7fc5b815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5b815d4b0 0x7fc5b815c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5bd85e640 1 -- 172.21.15.29:0/4116825002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80c6120 con 0x7fc5a40047c0 2023-10-16T14:29:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.583+0000 7fc5be05f640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815d930 0x7fc5b815cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:29.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.584+0000 7fc5bd85e640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5b8156050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40116d0 tx=0x7fc5b4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:29.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.584+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400f9c0 con 0x7fc5a40047c0 2023-10-16T14:29:29.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.584+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400fb20 con 0x7fc5a40047c0 2023-10-16T14:29:29.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.584+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400fed0 con 0x7fc5a40047c0 2023-10-16T14:29:29.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.584+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80c6400 con 0x7fc5a40047c0 2023-10-16T14:29:29.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.584+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b81ee230 con 0x7fc5a40047c0 2023-10-16T14:29:29.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.585+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b80adba0 con 0x7fc5a40047c0 2023-10-16T14:29:29.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.586+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc5b4015460 con 0x7fc5a40047c0 2023-10-16T14:29:29.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.587+0000 7fc5ae7fc640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc59c06b070 0x7fc59c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:29.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.587+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc5b408a3a0 con 0x7fc5a40047c0 2023-10-16T14:29:29.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.588+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc59c06b070 0x7fc59c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:29.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.589+0000 7fc5bd05d640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc59c06b070 0x7fc59c06d530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8033000 tx=0x7fc5a8009030 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:29.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.592+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5b40550c0 con 0x7fc5a40047c0 2023-10-16T14:29:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.807+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5b80c8710 con 0x7fc59c06b070 2023-10-16T14:29:29.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.826+0000 7fc5ae7fc640 1 -- 172.21.15.29:0/4116825002 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fc5a8005390 con 0x7fc59c06b070 2023-10-16T14:29:29.830 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:29.831 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:56", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc59c06b070 msgr2=0x7fc59c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc59c06b070 0x7fc59c06d530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8033000 tx=0x7fc5a8009030 comp rx=0 tx=0).stop 2023-10-16T14:29:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 msgr2=0x7fc5b8156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:29.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5b8156050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40116d0 tx=0x7fc5b4011b90 comp rx=0 tx=0).stop 2023-10-16T14:29:29.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 shutdown_connections 2023-10-16T14:29:29.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fc59c06b070 0x7fc59c06d530 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5b815d930 0x7fc5b815cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5b815d4b0 0x7fc5b815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 --2- 172.21.15.29:0/4116825002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5a40047c0 0x7fc5b8156050 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:29.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 >> 172.21.15.29:0/4116825002 conn(0x7fc5b80ce9b0 msgr2=0x7fc5b80cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:29.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 shutdown_connections 2023-10-16T14:29:29.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:29.829+0000 7fc5bfac2640 1 -- 172.21.15.29:0/4116825002 wait complete. 2023-10-16T14:29:29.852 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:56", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:29.853 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:17:56', 'first': '2023-10-16T14:18:00', 'last': '2023-10-16T14:29:01', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:29.853 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T14:29:29.853 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T14:29:29.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:29.854 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T14:29:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:29.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:29.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:29.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:29.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:29.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:29.919 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:29.919 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:29.919 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:29.919 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:29.920 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:29.921 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_18_00_UTC', 'scheduled-2023-10-16-14_19_01_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_00_UTC', 'scheduled-2023-10-16-14_24_01_UTC', 'scheduled-2023-10-16-14_25_00_UTC', 'scheduled-2023-10-16-14_26_00_UTC', 'scheduled-2023-10-16-14_27_00_UTC', 'scheduled-2023-10-16-14_28_01_UTC', 'scheduled-2023-10-16-14_29_01_UTC'] 2023-10-16T14:29:29.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-10-16T14:29:30.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.149+0000 7fdbbe260640 1 Processor -- start 2023-10-16T14:29:30.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.149+0000 7fdbbe260640 1 -- start start 2023-10-16T14:29:30.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbe260640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdbb80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbe260640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb815b9d0 con 0x7fdbb806c040 2023-10-16T14:29:30.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbe260640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb815bcb0 con 0x7fdbb80ccfb0 2023-10-16T14:29:30.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbe260640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb815bf90 con 0x7fdbb80ad970 2023-10-16T14:29:30.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbd25e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdbb806c040 0x7fdbb806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54346/0 (socket says 172.21.15.29:54346) 2023-10-16T14:29:30.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbd25e640 1 -- 172.21.15.29:0/227818163 learned_addr learned my addr 172.21.15.29:0/227818163 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:30.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.151+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdbb80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710070624 0 0) 0x7fdbb815b9d0 con 0x7fdbb806c040 2023-10-16T14:29:30.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba4003520 con 0x7fdbb806c040 2023-10-16T14:29:30.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917034486 0 0) 0x7fdba4003520 con 0x7fdbb806c040 2023-10-16T14:29:30.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 msgr2=0x7fdbb80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:30.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdbb80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 >> v1:172.21.15.170:6789/0 conn(0x7fdbb80ccfb0 legacy=0x7fdbb80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80cc270 con 0x7fdbb806c040 2023-10-16T14:29:30.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fdbb4001670 con 0x7fdbb806c040 2023-10-16T14:29:30.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 >> v1:172.21.15.29:6789/0 conn(0x7fdbb806c040 legacy=0x7fdbb806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba40039a0 0x7fdba4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba40047c0 0x7fdba4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdba40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb815bf90 con 0x7fdba40047c0 2023-10-16T14:29:30.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba400bab0 con 0x7fdba40039a0 2023-10-16T14:29:30.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba400bc70 con 0x7fdbb80ad970 2023-10-16T14:29:30.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.152+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba40047c0 0x7fdba4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbd25e640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba40039a0 0x7fdba4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbca5d640 1 -- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 msgr2=0x7fdba40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:30.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbda5f640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdba40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdba40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbca5d640 1 -- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba40039a0 msgr2=0x7fdba4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:30.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdba40039a0 0x7fdba4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbca5d640 1 -- 172.21.15.29:0/227818163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4003520 con 0x7fdba40047c0 2023-10-16T14:29:30.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.153+0000 7fdbbda5f640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdbb80ad970 0x7fdba40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:30.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.154+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba40047c0 0x7fdba4004c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009ac0 tx=0x7fdba40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d32f8b1e61307a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:30.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.154+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02c6b0 con 0x7fdba40047c0 2023-10-16T14:29:30.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.154+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac02c810 con 0x7fdba40047c0 2023-10-16T14:29:30.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.154+0000 7fdbabfff640 1 -- 172.21.15.29:0/227818163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02cba0 con 0x7fdba40047c0 2023-10-16T14:29:30.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbe260640 1 -- 172.21.15.29:0/227818163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba40047c0 msgr2=0x7fdba4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbe260640 1 --2- 172.21.15.29:0/227818163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba40047c0 0x7fdba4004c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009ac0 tx=0x7fdba40057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbda5f640 1 -- 172.21.15.29:0/227818163 reap_dead start 2023-10-16T14:29:30.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbe260640 1 -- 172.21.15.29:0/227818163 shutdown_connections 2023-10-16T14:29:30.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbe260640 1 -- 172.21.15.29:0/227818163 >> 172.21.15.29:0/227818163 conn(0x7fdbb80ce6a0 msgr2=0x7fdbb80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbe260640 1 -- 172.21.15.29:0/227818163 shutdown_connections 2023-10-16T14:29:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.155+0000 7fdbbe260640 1 -- 172.21.15.29:0/227818163 wait complete. 2023-10-16T14:29:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.157+0000 7fdbbe260640 1 Processor -- start 2023-10-16T14:29:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.157+0000 7fdbbe260640 1 -- start start 2023-10-16T14:29:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbe260640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba40047c0 0x7fdbb81560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbe260640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 0x7fdbb8154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbe260640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbb8154aa0 0x7fdbb80c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbe260640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8155020 con 0x7fdbb80c4930 2023-10-16T14:29:30.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbe260640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8155300 con 0x7fdbb8154aa0 2023-10-16T14:29:30.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbe260640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb81555e0 con 0x7fdba40047c0 2023-10-16T14:29:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbca5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 0x7fdbb8154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbca5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 0x7fdbb8154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53422/0 (socket says 172.21.15.29:53422) 2023-10-16T14:29:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.158+0000 7fdbbca5d640 1 -- 172.21.15.29:0/3867742671 learned_addr learned my addr 172.21.15.29:0/3867742671 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbca5d640 1 -- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba40047c0 msgr2=0x7fdbb81560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbd25e640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba40047c0 0x7fdbb81560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba40047c0 0x7fdbb81560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbda5f640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbb8154aa0 0x7fdbb80c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbca5d640 1 -- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbb8154aa0 msgr2=0x7fdbb80c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbb8154aa0 0x7fdbb80c75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbca5d640 1 -- 172.21.15.29:0/3867742671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80cdcd0 con 0x7fdbb80c4930 2023-10-16T14:29:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.159+0000 7fdbbca5d640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 0x7fdbb8154560 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdbac02c250 tx=0x7fdbac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:30.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.160+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac003810 con 0x7fdbb80c4930 2023-10-16T14:29:30.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.160+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac003970 con 0x7fdbb80c4930 2023-10-16T14:29:30.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.160+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac003ce0 con 0x7fdbb80c4930 2023-10-16T14:29:30.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.160+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb81558c0 con 0x7fdbb80c4930 2023-10-16T14:29:30.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.160+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb80c4db0 con 0x7fdbb80c4930 2023-10-16T14:29:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.161+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb806b330 con 0x7fdbb80c4930 2023-10-16T14:29:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.162+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fdbac034440 con 0x7fdbb80c4930 2023-10-16T14:29:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.163+0000 7fdba9ffb640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdb9c06b070 0x7fdb9c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.163+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdbac0abdf0 con 0x7fdbb80c4930 2023-10-16T14:29:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.164+0000 7fdbbd25e640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdb9c06b070 0x7fdb9c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.165+0000 7fdbbd25e640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdb9c06b070 0x7fdb9c06d530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4001eb0 tx=0x7fdbb4000f10 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.168+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdbac076b30 con 0x7fdbb80c4930 2023-10-16T14:29:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.383+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbb80cdb50 con 0x7fdb9c06b070 2023-10-16T14:29:30.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.401+0000 7fdba9ffb640 1 -- 172.21.15.29:0/3867742671 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fdbb40014b0 con 0x7fdb9c06b070 2023-10-16T14:29:30.405 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:30.405 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:57", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:30.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.403+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdb9c06b070 msgr2=0x7fdb9c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdb9c06b070 0x7fdb9c06d530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4001eb0 tx=0x7fdbb4000f10 comp rx=0 tx=0).stop 2023-10-16T14:29:30.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 msgr2=0x7fdbb8154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 0x7fdbb8154560 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdbac02c250 tx=0x7fdbac004730 comp rx=0 tx=0).stop 2023-10-16T14:29:30.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 shutdown_connections 2023-10-16T14:29:30.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdb9c06b070 0x7fdb9c06d530 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbb80c4930 0x7fdbb8154560 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdbb8154aa0 0x7fdbb80c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 --2- 172.21.15.29:0/3867742671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdba40047c0 0x7fdbb81560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 >> 172.21.15.29:0/3867742671 conn(0x7fdbb80ce6a0 msgr2=0x7fdbb815cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:30.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 shutdown_connections 2023-10-16T14:29:30.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.404+0000 7fdbbe260640 1 -- 172.21.15.29:0/3867742671 wait complete. 2023-10-16T14:29:30.427 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:57", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:30.427 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:17:57', 'first': '2023-10-16T14:18:00', 'last': '2023-10-16T14:29:01', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:30.428 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T14:29:30.428 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T14:29:30.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:30.429 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T14:29:30.487 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:30.487 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:30.487 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:30.489 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:30.489 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:30.489 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:30.489 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:30.490 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_18_00_UTC', 'scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_01_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_00_UTC', 'scheduled-2023-10-16-14_24_01_UTC', 'scheduled-2023-10-16-14_25_01_UTC', 'scheduled-2023-10-16-14_26_01_UTC', 'scheduled-2023-10-16-14_27_01_UTC', 'scheduled-2023-10-16-14_28_01_UTC', 'scheduled-2023-10-16-14_29_01_UTC'] 2023-10-16T14:29:30.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-10-16T14:29:30.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.716+0000 7f90071f5640 1 Processor -- start 2023-10-16T14:29:30.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.716+0000 7f90071f5640 1 -- start start 2023-10-16T14:29:30.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.717+0000 7f90071f5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9000159e90 0x7f90000ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.717+0000 7f90071f5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9000070360 con 0x7f900006c140 2023-10-16T14:29:30.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.717+0000 7f90071f5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90000cd1f0 con 0x7f9000159ab0 2023-10-16T14:29:30.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.717+0000 7f90071f5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900015d890 con 0x7f9000159e90 2023-10-16T14:29:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.717+0000 7f9005792640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f900006c140 0x7f900015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54352/0 (socket says 172.21.15.29:54352) 2023-10-16T14:29:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.717+0000 7f9005792640 1 -- 172.21.15.29:0/2882290053 learned_addr learned my addr 172.21.15.29:0/2882290053 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8ffffff640 1 --2- 172.21.15.29:0/2882290053 >> v2:172.21.15.170:3300/0 conn(0x7f9000159e90 0x7f90000ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090360373 0 0) 0x7f9000070360 con 0x7f900006c140 2023-10-16T14:29:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0005730 con 0x7f900006c140 2023-10-16T14:29:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179810652 0 0) 0x7f8ff0005730 con 0x7f900006c140 2023-10-16T14:29:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 >> v2:172.21.15.170:3300/0 conn(0x7f9000159e90 msgr2=0x7f90000ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 --2- 172.21.15.29:0/2882290053 >> v2:172.21.15.170:3300/0 conn(0x7f9000159e90 0x7f90000ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 >> v1:172.21.15.170:6789/0 conn(0x7f9000159ab0 legacy=0x7f900006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90000cd4d0 con 0x7f900006c140 2023-10-16T14:29:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.718+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8ff80019f0 con 0x7f900006c140 2023-10-16T14:29:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 >> v1:172.21.15.29:6789/0 conn(0x7f900006c140 legacy=0x7f900015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ff0005b80 0x7f8ff0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f8ff0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ff00074b0 0x7f8ff000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900015d890 con 0x7f8ff00069d0 2023-10-16T14:29:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff000e380 con 0x7f8ff0005b80 2023-10-16T14:29:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff000e540 con 0x7f8ff00074b0 2023-10-16T14:29:30.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8ffffff640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f8ff0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f9004f91640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ff0005b80 0x7f8ff0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8ffffff640 1 -- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ff00074b0 msgr2=0x7f8ff000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:30.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8ffffff640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ff00074b0 0x7f8ff000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8ffffff640 1 -- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ff0005b80 msgr2=0x7f8ff0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:30.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8ffffff640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ff0005b80 0x7f8ff0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f8ffffff640 1 -- 172.21.15.29:0/2882290053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0005730 con 0x7f8ff00069d0 2023-10-16T14:29:30.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.719+0000 7f9005792640 1 -- 172.21.15.29:0/2882290053 reap_dead start 2023-10-16T14:29:30.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.721+0000 7f8ffffff640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f8ff0006e50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009a90 tx=0x7f8ff0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d529b6d3f6a55e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:30.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.721+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec02c680 con 0x7f8ff00069d0 2023-10-16T14:29:30.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.721+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec02c7e0 con 0x7f8ff00069d0 2023-10-16T14:29:30.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.721+0000 7f8fff7fe640 1 -- 172.21.15.29:0/2882290053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec02cb70 con 0x7f8ff00069d0 2023-10-16T14:29:30.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.721+0000 7f90071f5640 1 -- 172.21.15.29:0/2882290053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 msgr2=0x7f8ff0006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.721+0000 7f90071f5640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f8ff0006e50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009a90 tx=0x7f8ff0007a00 comp rx=0 tx=0).stop 2023-10-16T14:29:30.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.722+0000 7f90071f5640 1 -- 172.21.15.29:0/2882290053 shutdown_connections 2023-10-16T14:29:30.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.722+0000 7f90071f5640 1 --2- 172.21.15.29:0/2882290053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f8ff0006e50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.722+0000 7f90071f5640 1 -- 172.21.15.29:0/2882290053 >> 172.21.15.29:0/2882290053 conn(0x7f90000ce9b0 msgr2=0x7f90000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:30.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.722+0000 7f90071f5640 1 -- 172.21.15.29:0/2882290053 shutdown_connections 2023-10-16T14:29:30.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.722+0000 7f90071f5640 1 -- 172.21.15.29:0/2882290053 wait complete. 2023-10-16T14:29:30.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.723+0000 7f90071f5640 1 Processor -- start 2023-10-16T14:29:30.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.724+0000 7f90071f5640 1 -- start start 2023-10-16T14:29:30.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f90071f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f90000c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f90071f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90000c4440 0x7f900014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f90071f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f900006c550 0x7f900006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f90071f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900006cf10 con 0x7f8ff00069d0 2023-10-16T14:29:30.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f90071f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90000c76a0 con 0x7f90000c4440 2023-10-16T14:29:30.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f90071f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90000c6a90 con 0x7f900006c550 2023-10-16T14:29:30.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f90000c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f90000c6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53432/0 (socket says 172.21.15.29:53432) 2023-10-16T14:29:30.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 -- 172.21.15.29:0/143707978 learned_addr learned my addr 172.21.15.29:0/143707978 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:30.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 -- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f900006c550 msgr2=0x7f900006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9005792640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f900006c550 0x7f900006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f8ffffff640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90000c4440 0x7f900014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f900006c550 0x7f900006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 -- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90000c4440 msgr2=0x7f900014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90000c4440 0x7f900014ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.725+0000 7f9004f91640 1 -- 172.21.15.29:0/143707978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90000c6d70 con 0x7f8ff00069d0 2023-10-16T14:29:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.726+0000 7f9004f91640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f90000c6490 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400e580 tx=0x7f8ff400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.726+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff40052e0 con 0x7f8ff00069d0 2023-10-16T14:29:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.726+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4005440 con 0x7f8ff00069d0 2023-10-16T14:29:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.726+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4003810 con 0x7f8ff00069d0 2023-10-16T14:29:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.726+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90000c7050 con 0x7f8ff00069d0 2023-10-16T14:29:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.726+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900014e310 con 0x7f8ff00069d0 2023-10-16T14:29:30.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.727+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900004e160 con 0x7f8ff00069d0 2023-10-16T14:29:30.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.728+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8ff4003970 con 0x7f8ff00069d0 2023-10-16T14:29:30.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.729+0000 7f8ffd7fa640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fe406b0a0 0x7f8fe406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:30.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.729+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8ff40898b0 con 0x7f8ff00069d0 2023-10-16T14:29:30.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.730+0000 7f8ffffff640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fe406b0a0 0x7f8fe406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:30.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.731+0000 7f8ffffff640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fe406b0a0 0x7f8fe406d560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec030000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:30.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.733+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ff4054550 con 0x7f8ff00069d0 2023-10-16T14:29:30.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.949+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90000cda90 con 0x7f8fe406b0a0 2023-10-16T14:29:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.966+0000 7f8ffd7fa640 1 -- 172.21.15.29:0/143707978 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f8fec0041c0 con 0x7f8fe406b0a0 2023-10-16T14:29:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:30.970 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:58", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:30.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fe406b0a0 msgr2=0x7f8fe406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fe406b0a0 0x7f8fe406d560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec030000 comp rx=0 tx=0).stop 2023-10-16T14:29:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 msgr2=0x7f90000c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f90000c6490 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400e580 tx=0x7f8ff400ea40 comp rx=0 tx=0).stop 2023-10-16T14:29:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 shutdown_connections 2023-10-16T14:29:30.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fe406b0a0 0x7f8fe406d560 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f900006c550 0x7f900006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90000c4440 0x7f900014ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 --2- 172.21.15.29:0/143707978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ff00069d0 0x7f90000c6490 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 >> 172.21.15.29:0/143707978 conn(0x7f90000ce9b0 msgr2=0x7f90000ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:30.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 shutdown_connections 2023-10-16T14:29:30.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:30.969+0000 7f90071f5640 1 -- 172.21.15.29:0/143707978 wait complete. 2023-10-16T14:29:30.991 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:58", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:30.992 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:17:58', 'first': '2023-10-16T14:18:00', 'last': '2023-10-16T14:29:01', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:30.992 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T14:29:30.992 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T14:29:30.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:30.993 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T14:29:31.049 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:31.049 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:31.049 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:31.050 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_01_UTC 2023-10-16T14:29:31.050 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:31.050 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:31.050 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:31.050 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:31.051 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:31.051 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:31.051 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:31.051 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:31.052 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_18_00_UTC', 'scheduled-2023-10-16-14_19_01_UTC', 'scheduled-2023-10-16-14_20_01_UTC', 'scheduled-2023-10-16-14_21_01_UTC', 'scheduled-2023-10-16-14_22_01_UTC', 'scheduled-2023-10-16-14_23_01_UTC', 'scheduled-2023-10-16-14_24_01_UTC', 'scheduled-2023-10-16-14_25_00_UTC', 'scheduled-2023-10-16-14_26_00_UTC', 'scheduled-2023-10-16-14_27_00_UTC', 'scheduled-2023-10-16-14_28_00_UTC', 'scheduled-2023-10-16-14_29_01_UTC'] 2023-10-16T14:29:31.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-10-16T14:29:31.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.279+0000 7fa87c8ff640 1 Processor -- start 2023-10-16T14:29:31.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.279+0000 7fa87c8ff640 1 -- start start 2023-10-16T14:29:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.280+0000 7fa87c8ff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa87806c040 0x7fa87806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.280+0000 7fa87c8ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87815f4a0 con 0x7fa8780cda80 2023-10-16T14:29:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.280+0000 7fa87c8ff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87806c960 con 0x7fa8780ad970 2023-10-16T14:29:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.280+0000 7fa87c8ff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87815b9d0 con 0x7fa87806c040 2023-10-16T14:29:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.280+0000 7fa87759e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8780cda80 0x7fa87815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54362/0 (socket says 172.21.15.29:54362) 2023-10-16T14:29:31.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.280+0000 7fa87759e640 1 -- 172.21.15.29:0/4145290479 learned_addr learned my addr 172.21.15.29:0/4145290479 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:31.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa876d9d640 1 --2- 172.21.15.29:0/4145290479 >> v2:172.21.15.170:3300/0 conn(0x7fa87806c040 0x7fa87806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776544689 0 0) 0x7fa87815f4a0 con 0x7fa8780cda80 2023-10-16T14:29:31.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa860003520 con 0x7fa8780cda80 2023-10-16T14:29:31.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410081418 0 0) 0x7fa860003520 con 0x7fa8780cda80 2023-10-16T14:29:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 >> v2:172.21.15.170:3300/0 conn(0x7fa87806c040 msgr2=0x7fa87806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 --2- 172.21.15.29:0/4145290479 >> v2:172.21.15.170:3300/0 conn(0x7fa87806c040 0x7fa87806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 >> v1:172.21.15.170:6789/0 conn(0x7fa8780ad970 legacy=0x7fa87815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87815bcb0 con 0x7fa8780cda80 2023-10-16T14:29:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa8700019f0 con 0x7fa8780cda80 2023-10-16T14:29:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 >> v1:172.21.15.29:6789/0 conn(0x7fa8780cda80 legacy=0x7fa87815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8600039c0 0x7fa860003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.281+0000 7fa875d9b640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa875d9b640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa860005180 0x7fa86000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87815b9d0 con 0x7fa8600047c0 2023-10-16T14:29:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86000c080 con 0x7fa860005180 2023-10-16T14:29:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86000c240 con 0x7fa8600039c0 2023-10-16T14:29:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87659c640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87759e640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa860005180 0x7fa86000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa876d9d640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8600039c0 0x7fa860003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87659c640 1 -- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8600039c0 msgr2=0x7fa860003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87659c640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8600039c0 0x7fa860003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87659c640 1 -- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa860005180 msgr2=0x7fa86000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87659c640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa860005180 0x7fa86000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87659c640 1 -- 172.21.15.29:0/4145290479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860003520 con 0x7fa8600047c0 2023-10-16T14:29:31.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.282+0000 7fa87759e640 1 -- 172.21.15.29:0/4145290479 reap_dead start 2023-10-16T14:29:31.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.284+0000 7fa87659c640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8600047c0 0x7fa860004c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa86c001720 tx=0x7fa8600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8138653b3829c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:31.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.284+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0052d0 con 0x7fa8600047c0 2023-10-16T14:29:31.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.284+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c02e050 con 0x7fa8600047c0 2023-10-16T14:29:31.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.284+0000 7fa875d9b640 1 -- 172.21.15.29:0/4145290479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c003830 con 0x7fa8600047c0 2023-10-16T14:29:31.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 -- 172.21.15.29:0/4145290479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8600047c0 msgr2=0x7fa860004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8600047c0 0x7fa860004c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa86c001720 tx=0x7fa8600057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 -- 172.21.15.29:0/4145290479 shutdown_connections 2023-10-16T14:29:31.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/4145290479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 -- 172.21.15.29:0/4145290479 >> 172.21.15.29:0/4145290479 conn(0x7fa8780ce6a0 msgr2=0x7fa8780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:31.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 -- 172.21.15.29:0/4145290479 shutdown_connections 2023-10-16T14:29:31.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.285+0000 7fa87c8ff640 1 -- 172.21.15.29:0/4145290479 wait complete. 2023-10-16T14:29:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.287+0000 7fa87c8ff640 1 Processor -- start 2023-10-16T14:29:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.287+0000 7fa87c8ff640 1 -- start start 2023-10-16T14:29:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8600047c0 0x7fa87814da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 0x7fa8781f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa878155a00 0x7fa878155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878154ff0 con 0x7fa8781f5e30 2023-10-16T14:29:31.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781f68f0 con 0x7fa8600047c0 2023-10-16T14:29:31.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781f6bd0 con 0x7fa878155a00 2023-10-16T14:29:31.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 0x7fa8781f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 0x7fa8781f62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53448/0 (socket says 172.21.15.29:53448) 2023-10-16T14:29:31.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 -- 172.21.15.29:0/1476526785 learned_addr learned my addr 172.21.15.29:0/1476526785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:31.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa876d9d640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8600047c0 0x7fa87814da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 -- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa878155a00 msgr2=0x7fa878155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa878155a00 0x7fa878155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 -- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8600047c0 msgr2=0x7fa87814da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8600047c0 0x7fa87814da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 -- 172.21.15.29:0/1476526785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa878155200 con 0x7fa8781f5e30 2023-10-16T14:29:31.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa876d9d640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8600047c0 0x7fa87814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:31.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87659c640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 0x7fa8781f62b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa86c005040 tx=0x7fa86c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:31.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c007640 con 0x7fa8781f5e30 2023-10-16T14:29:31.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c0069e0 con 0x7fa8781f5e30 2023-10-16T14:29:31.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8781554e0 con 0x7fa8781f5e30 2023-10-16T14:29:31.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c006d90 con 0x7fa8781f5e30 2023-10-16T14:29:31.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.289+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87814df40 con 0x7fa8781f5e30 2023-10-16T14:29:31.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.290+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87806b330 con 0x7fa8781f5e30 2023-10-16T14:29:31.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.291+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa86c0035f0 con 0x7fa8781f5e30 2023-10-16T14:29:31.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.293+0000 7fa85f7fe640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa86406b070 0x7fa86406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.293+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa86c0ab910 con 0x7fa8781f5e30 2023-10-16T14:29:31.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.293+0000 7fa876d9d640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa86406b070 0x7fa86406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.295+0000 7fa876d9d640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa86406b070 0x7fa86406d530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa868009a80 tx=0x7fa868008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:31.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.297+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa86c076520 con 0x7fa8781f5e30 2023-10-16T14:29:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.510+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8780cc270 con 0x7fa86406b070 2023-10-16T14:29:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.527+0000 7fa85f7fe640 1 -- 172.21.15.29:0/1476526785 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fa868012a10 con 0x7fa86406b070 2023-10-16T14:29:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:31.532 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:58", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.530+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa86406b070 msgr2=0x7fa86406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.530+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa86406b070 0x7fa86406d530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa868009a80 tx=0x7fa868008040 comp rx=0 tx=0).stop 2023-10-16T14:29:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.530+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 msgr2=0x7fa8781f62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.530+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 0x7fa8781f62b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa86c005040 tx=0x7fa86c001120 comp rx=0 tx=0).stop 2023-10-16T14:29:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 shutdown_connections 2023-10-16T14:29:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa86406b070 0x7fa86406d530 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8781f5e30 0x7fa8781f62b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa878155a00 0x7fa878155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 --2- 172.21.15.29:0/1476526785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8600047c0 0x7fa87814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 >> 172.21.15.29:0/1476526785 conn(0x7fa8780ce6a0 msgr2=0x7fa87806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 shutdown_connections 2023-10-16T14:29:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.531+0000 7fa87c8ff640 1 -- 172.21.15.29:0/1476526785 wait complete. 2023-10-16T14:29:31.554 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:58", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:01", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:31.554 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:17:58', 'first': '2023-10-16T14:18:00', 'last': '2023-10-16T14:29:01', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:31.554 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T14:29:31.555 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T14:29:31.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:31.555 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T14:29:31.612 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:31.613 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:31.613 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:31.613 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:31.614 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:31.614 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:31.615 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:31.615 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:31.615 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:31.615 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:31.616 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:31.616 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:31.618 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_18_00_UTC', 'scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_01_UTC', 'scheduled-2023-10-16-14_23_01_UTC', 'scheduled-2023-10-16-14_24_01_UTC', 'scheduled-2023-10-16-14_25_01_UTC', 'scheduled-2023-10-16-14_26_00_UTC', 'scheduled-2023-10-16-14_27_00_UTC', 'scheduled-2023-10-16-14_28_00_UTC', 'scheduled-2023-10-16-14_29_00_UTC'] 2023-10-16T14:29:31.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-10-16T14:29:31.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.842+0000 7f5629de9640 1 Processor -- start 2023-10-16T14:29:31.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.842+0000 7f5629de9640 1 -- start start 2023-10-16T14:29:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.842+0000 7f5629de9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f562406c040 0x7f562406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5629de9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f56240ad970 0x7f562415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5629de9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562406c960 con 0x7f562415ebc0 2023-10-16T14:29:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5629de9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562415b9d0 con 0x7f562406c040 2023-10-16T14:29:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5629de9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562415bcb0 con 0x7f56240ad970 2023-10-16T14:29:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5628de7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f562406c040 0x7f562406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5628de7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f562406c040 0x7f562406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53450/0 (socket says 172.21.15.29:53450) 2023-10-16T14:29:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5628de7640 1 -- 172.21.15.29:0/193640191 learned_addr learned my addr 172.21.15.29:0/193640191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.843+0000 7f5623fff640 1 --2- 172.21.15.29:0/193640191 >> v2:172.21.15.170:3300/0 conn(0x7f56240ad970 0x7f562415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.844+0000 7f5628de7640 1 -- 172.21.15.29:0/193640191 >> v2:172.21.15.170:3300/0 conn(0x7f56240ad970 msgr2=0x7f562415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.844+0000 7f5628de7640 1 --2- 172.21.15.29:0/193640191 >> v2:172.21.15.170:3300/0 conn(0x7f56240ad970 0x7f562415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.844+0000 7f5628de7640 1 -- 172.21.15.29:0/193640191 >> v1:172.21.15.170:6789/0 conn(0x7f562415ebc0 legacy=0x7f56240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.844+0000 7f5628de7640 1 -- 172.21.15.29:0/193640191 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562415bf90 con 0x7f562406c040 2023-10-16T14:29:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.846+0000 7f5628de7640 1 --2- 172.21.15.29:0/193640191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562406c040 0x7f562406c420 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f56240cb060 tx=0x7f561802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d36a1e186a2e7f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f56237fe640 1 -- 172.21.15.29:0/193640191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618031870 con 0x7f562406c040 2023-10-16T14:29:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f56237fe640 1 -- 172.21.15.29:0/193640191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56180319d0 con 0x7f562406c040 2023-10-16T14:29:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f56237fe640 1 -- 172.21.15.29:0/193640191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618031d60 con 0x7f562406c040 2023-10-16T14:29:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 -- 172.21.15.29:0/193640191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562406c040 msgr2=0x7f562406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 --2- 172.21.15.29:0/193640191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562406c040 0x7f562406c420 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f56240cb060 tx=0x7f561802b910 comp rx=0 tx=0).stop 2023-10-16T14:29:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 -- 172.21.15.29:0/193640191 shutdown_connections 2023-10-16T14:29:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 --2- 172.21.15.29:0/193640191 >> v2:172.21.15.170:3300/0 conn(0x7f56240ad970 0x7f562415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 --2- 172.21.15.29:0/193640191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562406c040 0x7f562406c420 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 -- 172.21.15.29:0/193640191 >> 172.21.15.29:0/193640191 conn(0x7f56240ce6a0 msgr2=0x7f56240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 -- 172.21.15.29:0/193640191 shutdown_connections 2023-10-16T14:29:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.847+0000 7f5629de9640 1 -- 172.21.15.29:0/193640191 wait complete. 2023-10-16T14:29:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.849+0000 7f5629de9640 1 Processor -- start 2023-10-16T14:29:31.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.849+0000 7f5629de9640 1 -- start start 2023-10-16T14:29:31.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.850+0000 7f5629de9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f562406c040 0x7f5624149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.850+0000 7f5629de9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56240ad970 0x7f56240ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.850+0000 7f5629de9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 0x7f56240ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.850+0000 7f5629de9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56240aefb0 con 0x7f562415ebc0 2023-10-16T14:29:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.850+0000 7f5629de9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56240b2b70 con 0x7f56240ad970 2023-10-16T14:29:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.850+0000 7f5629de9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562414a0c0 con 0x7f562406c040 2023-10-16T14:29:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 0x7f56240ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 0x7f56240ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53462/0 (socket says 172.21.15.29:53462) 2023-10-16T14:29:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 -- 172.21.15.29:0/2034093529 learned_addr learned my addr 172.21.15.29:0/2034093529 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:31.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f5628de7640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f562406c040 0x7f5624149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f5623fff640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56240ad970 0x7f56240ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 -- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f562406c040 msgr2=0x7f5624149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f562406c040 0x7f5624149b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 -- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56240ad970 msgr2=0x7f56240ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56240ad970 0x7f56240ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f56295e8640 1 -- 172.21.15.29:0/2034093529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56180094d0 con 0x7f562415ebc0 2023-10-16T14:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.851+0000 7f5628de7640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f562406c040 0x7f5624149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.852+0000 7f56295e8640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 0x7f56240ac890 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f561c00c780 tx=0x7f561c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.852+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c004b60 con 0x7f562415ebc0 2023-10-16T14:29:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.852+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c00ce90 con 0x7f562415ebc0 2023-10-16T14:29:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.852+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c00a850 con 0x7f562415ebc0 2023-10-16T14:29:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.852+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56240aecd0 con 0x7f562415ebc0 2023-10-16T14:29:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.852+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56240b2790 con 0x7f562415ebc0 2023-10-16T14:29:31.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.853+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562406b330 con 0x7f562415ebc0 2023-10-16T14:29:31.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.854+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f561c00a9b0 con 0x7f562415ebc0 2023-10-16T14:29:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.855+0000 7f56217fa640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f560806b070 0x7f560806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.855+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f561c089c40 con 0x7f562415ebc0 2023-10-16T14:29:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.856+0000 7f5628de7640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f560806b070 0x7f560806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.857+0000 7f5628de7640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f560806b070 0x7f560806d530 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5618036000 tx=0x7f561803c040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:31.859+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f561c054970 con 0x7f562415ebc0 2023-10-16T14:29:32.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.074+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56240cc550 con 0x7f560806b070 2023-10-16T14:29:32.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.093+0000 7f56217fa640 1 -- 172.21.15.29:0/2034093529 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f561803a420 con 0x7f560806b070 2023-10-16T14:29:32.096 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:32.097 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:59", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:32.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f560806b070 msgr2=0x7f560806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f560806b070 0x7f560806d530 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5618036000 tx=0x7f561803c040 comp rx=0 tx=0).stop 2023-10-16T14:29:32.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 msgr2=0x7f56240ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 0x7f56240ac890 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f561c00c780 tx=0x7f561c00cc40 comp rx=0 tx=0).stop 2023-10-16T14:29:32.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 shutdown_connections 2023-10-16T14:29:32.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f560806b070 0x7f560806d530 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f562415ebc0 0x7f56240ac890 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56240ad970 0x7f56240ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 --2- 172.21.15.29:0/2034093529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f562406c040 0x7f5624149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 >> 172.21.15.29:0/2034093529 conn(0x7f56240ce6a0 msgr2=0x7f56240c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:32.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.096+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 shutdown_connections 2023-10-16T14:29:32.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.097+0000 7f5629de9640 1 -- 172.21.15.29:0/2034093529 wait complete. 2023-10-16T14:29:32.120 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:17:59", "first": "2023-10-16T14:18:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T14:29:32.120 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:17:59', 'first': '2023-10-16T14:18:00', 'last': '2023-10-16T14:29:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:32.121 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T14:29:32.121 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T14:29:32.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:32.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T14:29:32.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:32.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:32.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:32.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:32.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:32.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:32.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:32.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:32.182 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:32.182 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:32.182 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:32.183 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_00_UTC', 'scheduled-2023-10-16-14_24_00_UTC', 'scheduled-2023-10-16-14_25_01_UTC', 'scheduled-2023-10-16-14_26_01_UTC', 'scheduled-2023-10-16-14_27_01_UTC', 'scheduled-2023-10-16-14_28_00_UTC', 'scheduled-2023-10-16-14_29_00_UTC'] 2023-10-16T14:29:32.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-10-16T14:29:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.412+0000 7f8ff43a9640 1 Processor -- start 2023-10-16T14:29:32.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.412+0000 7f8ff43a9640 1 -- start start 2023-10-16T14:29:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.413+0000 7f8ff43a9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8fec15ad70 0x7f8fec069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.413+0000 7f8ff43a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fec0c9ca0 0x7f8fec06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.413+0000 7f8ff43a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec0ca3b0 0x7f8fec0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.413+0000 7f8ff43a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0cad80 con 0x7f8fec0c9ca0 2023-10-16T14:29:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.413+0000 7f8ff43a9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0cb030 con 0x7f8fec15ad70 2023-10-16T14:29:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.413+0000 7f8ff43a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c91c0 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fec0c9ca0 0x7f8fec06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fec0c9ca0 0x7f8fec06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53474/0 (socket says 172.21.15.29:53474) 2023-10-16T14:29:32.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 -- 172.21.15.29:0/682143718 learned_addr learned my addr 172.21.15.29:0/682143718 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff2145640 1 --2- 172.21.15.29:0/682143718 >> v2:172.21.15.170:3300/0 conn(0x7f8fec15ad70 0x7f8fec069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff2946640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec0ca3b0 0x7f8fec0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 -- 172.21.15.29:0/682143718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec0ca3b0 msgr2=0x7f8fec0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec0ca3b0 0x7f8fec0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 -- 172.21.15.29:0/682143718 >> v2:172.21.15.170:3300/0 conn(0x7f8fec15ad70 msgr2=0x7f8fec069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:32.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 --2- 172.21.15.29:0/682143718 >> v2:172.21.15.170:3300/0 conn(0x7f8fec15ad70 0x7f8fec069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff1944640 1 -- 172.21.15.29:0/682143718 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec0c94a0 con 0x7f8fec0c9ca0 2023-10-16T14:29:32.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.414+0000 7f8ff2946640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec0ca3b0 0x7f8fec0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:32.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.415+0000 7f8ff1944640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec06a2e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0cf780 tx=0x7f8fdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a142d7f5d1ec85a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:32.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff1143640 1 -- 172.21.15.29:0/682143718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc031870 con 0x7f8fec0c9ca0 2023-10-16T14:29:32.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff1143640 1 -- 172.21.15.29:0/682143718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc0319d0 con 0x7f8fec0c9ca0 2023-10-16T14:29:32.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff1143640 1 -- 172.21.15.29:0/682143718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc031d60 con 0x7f8fec0c9ca0 2023-10-16T14:29:32.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 -- 172.21.15.29:0/682143718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0c9ca0 msgr2=0x7f8fec06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec06a2e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0cf780 tx=0x7f8fdc02b910 comp rx=0 tx=0).stop 2023-10-16T14:29:32.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 -- 172.21.15.29:0/682143718 shutdown_connections 2023-10-16T14:29:32.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec0ca3b0 0x7f8fec0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/682143718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec06a2e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/682143718 >> v2:172.21.15.170:3300/0 conn(0x7f8fec15ad70 0x7f8fec069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.416+0000 7f8ff43a9640 1 -- 172.21.15.29:0/682143718 >> 172.21.15.29:0/682143718 conn(0x7f8fec0ce9b0 msgr2=0x7f8fec0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:32.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.417+0000 7f8ff43a9640 1 -- 172.21.15.29:0/682143718 shutdown_connections 2023-10-16T14:29:32.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.417+0000 7f8ff43a9640 1 -- 172.21.15.29:0/682143718 wait complete. 2023-10-16T14:29:32.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.418+0000 7f8ff43a9640 1 Processor -- start 2023-10-16T14:29:32.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.418+0000 7f8ff43a9640 1 -- start start 2023-10-16T14:29:32.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.419+0000 7f8ff43a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec1f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.419+0000 7f8ff43a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 0x7f8fec0ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.419+0000 7f8ff43a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec15ad70 0x7f8fec0c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.419+0000 7f8ff43a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1f21c0 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.419+0000 7f8ff43a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c6dd0 con 0x7f8fec0c9ca0 2023-10-16T14:29:32.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.419+0000 7f8ff43a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0ad2a0 con 0x7f8fec15ad70 2023-10-16T14:29:32.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 0x7f8fec0ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 0x7f8fec0ae560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53490/0 (socket says 172.21.15.29:53490) 2023-10-16T14:29:32.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 -- 172.21.15.29:0/620785884 learned_addr learned my addr 172.21.15.29:0/620785884 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:32.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff2145640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec1f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff2946640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec15ad70 0x7f8fec0c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 -- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec15ad70 msgr2=0x7f8fec0c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec15ad70 0x7f8fec0c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 -- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fec0c9ca0 msgr2=0x7f8fec1f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec1f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.420+0000 7f8ff1944640 1 -- 172.21.15.29:0/620785884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0094d0 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.421+0000 7f8ff1944640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 0x7f8fec0ae560 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc02b950 tx=0x7f8fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:32.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.421+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc03ad30 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.421+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc0443f0 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.421+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc0447a0 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.421+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec0ad580 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.421+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec0adb40 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.422+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec04e160 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.423+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8fdc04d450 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.424+0000 7f8fe2ffd640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fd006b070 0x7f8fd006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.424+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8fdc02d040 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.425+0000 7f8ff2145640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fd006b070 0x7f8fd006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.426+0000 7f8ff2145640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fd006b070 0x7f8fd006d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8006fd0 tx=0x7f8fe8008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:32.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.429+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8fdc0deca0 con 0x7f8fec0ca3b0 2023-10-16T14:29:32.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.646+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8fec0c9780 con 0x7f8fd006b070 2023-10-16T14:29:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.662+0000 7f8fe2ffd640 1 -- 172.21.15.29:0/620785884 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f8fe800f860 con 0x7f8fd006b070 2023-10-16T14:29:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:32.666 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:00", "first": "2023-10-16T14:19:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fd006b070 msgr2=0x7f8fd006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fd006b070 0x7f8fd006d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8006fd0 tx=0x7f8fe8008040 comp rx=0 tx=0).stop 2023-10-16T14:29:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 msgr2=0x7f8fec0ae560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 0x7f8fec0ae560 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc02b950 tx=0x7f8fdc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 shutdown_connections 2023-10-16T14:29:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f8fd006b070 0x7f8fd006d530 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fec0ca3b0 0x7f8fec0ae560 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8fec15ad70 0x7f8fec0c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 --2- 172.21.15.29:0/620785884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8fec0c9ca0 0x7f8fec1f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.665+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 >> 172.21.15.29:0/620785884 conn(0x7f8fec0ce9b0 msgr2=0x7f8fec0afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.666+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 shutdown_connections 2023-10-16T14:29:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.666+0000 7f8ff43a9640 1 -- 172.21.15.29:0/620785884 wait complete. 2023-10-16T14:29:32.688 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:00", "first": "2023-10-16T14:19:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:32.689 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:18:00', 'first': '2023-10-16T14:19:00', 'last': '2023-10-16T14:29:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:32.689 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T14:29:32.690 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T14:29:32.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:32.691 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T14:29:32.750 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:32.751 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:32.751 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:32.751 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:32.751 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:32.751 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:32.752 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:32.752 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:32.752 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:32.752 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:32.752 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:32.753 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_19_01_UTC', 'scheduled-2023-10-16-14_20_01_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_00_UTC', 'scheduled-2023-10-16-14_24_00_UTC', 'scheduled-2023-10-16-14_25_00_UTC', 'scheduled-2023-10-16-14_26_00_UTC', 'scheduled-2023-10-16-14_27_00_UTC', 'scheduled-2023-10-16-14_28_01_UTC', 'scheduled-2023-10-16-14_29_00_UTC'] 2023-10-16T14:29:32.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-10-16T14:29:32.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.989+0000 7fb682c7c640 1 Processor -- start 2023-10-16T14:29:32.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.989+0000 7fb682c7c640 1 -- start start 2023-10-16T14:29:32.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.991+0000 7fb682c7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:32.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.991+0000 7fb682c7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c15b9d0 con 0x7fb67c0ccfb0 2023-10-16T14:29:32.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.991+0000 7fb682c7c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c15bcb0 con 0x7fb67c06c040 2023-10-16T14:29:32.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.991+0000 7fb682c7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c15bf90 con 0x7fb67c0ad970 2023-10-16T14:29:32.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.991+0000 7fb68247b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb67c0ccfb0 0x7fb67c0cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54376/0 (socket says 172.21.15.29:54376) 2023-10-16T14:29:32.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.991+0000 7fb68247b640 1 -- 172.21.15.29:0/2876563875 learned_addr learned my addr 172.21.15.29:0/2876563875 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb681479640 1 --2- 172.21.15.29:0/2876563875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb680c78640 1 -- 172.21.15.29:0/2876563875 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702711520 0 0) 0x7fb67c15b9d0 con 0x7fb67c0ccfb0 2023-10-16T14:29:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb680c78640 1 -- 172.21.15.29:0/2876563875 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb668003520 con 0x7fb67c0ccfb0 2023-10-16T14:29:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb681479640 1 -- 172.21.15.29:0/2876563875 >> v1:172.21.15.170:6789/0 conn(0x7fb67c06c040 legacy=0x7fb67c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb680c78640 1 -- 172.21.15.29:0/2876563875 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411256760 0 0) 0x7fb668003520 con 0x7fb67c0ccfb0 2023-10-16T14:29:32.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb681479640 1 -- 172.21.15.29:0/2876563875 >> v1:172.21.15.29:6789/0 conn(0x7fb67c0ccfb0 legacy=0x7fb67c0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.992+0000 7fb681479640 1 -- 172.21.15.29:0/2876563875 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0cc270 con 0x7fb67c0ad970 2023-10-16T14:29:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.994+0000 7fb681479640 1 --2- 172.21.15.29:0/2876563875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0cca70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009800 tx=0x7fb66c02b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24f47b2cc9b40991 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.994+0000 7fb680c78640 1 -- 172.21.15.29:0/2876563875 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740019f0 con 0x7fb67c0ad970 2023-10-16T14:29:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.994+0000 7fb680c78640 1 -- 172.21.15.29:0/2876563875 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c02e050 con 0x7fb67c0ad970 2023-10-16T14:29:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.994+0000 7fb680c78640 1 -- 172.21.15.29:0/2876563875 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c030710 con 0x7fb67c0ad970 2023-10-16T14:29:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 -- 172.21.15.29:0/2876563875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 msgr2=0x7fb67c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2876563875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0cca70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009800 tx=0x7fb66c02b690 comp rx=0 tx=0).stop 2023-10-16T14:29:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 -- 172.21.15.29:0/2876563875 shutdown_connections 2023-10-16T14:29:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2876563875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0cca70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 -- 172.21.15.29:0/2876563875 >> 172.21.15.29:0/2876563875 conn(0x7fb67c0ce6a0 msgr2=0x7fb67c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 -- 172.21.15.29:0/2876563875 shutdown_connections 2023-10-16T14:29:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.995+0000 7fb682c7c640 1 -- 172.21.15.29:0/2876563875 wait complete. 2023-10-16T14:29:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.998+0000 7fb682c7c640 1 Processor -- start 2023-10-16T14:29:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.998+0000 7fb682c7c640 1 -- start start 2023-10-16T14:29:33.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.999+0000 7fb682c7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67c06c040 0x7fb67c0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.999+0000 7fb682c7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.999+0000 7fb682c7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 0x7fb67c156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.999+0000 7fb682c7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0c80a0 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.999+0000 7fb682c7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1570e0 con 0x7fb67c06c040 2023-10-16T14:29:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:32.999+0000 7fb682c7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0af1a0 con 0x7fb67c0ad970 2023-10-16T14:29:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 0x7fb67c156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 0x7fb67c156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53494/0 (socket says 172.21.15.29:53494) 2023-10-16T14:29:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 -- 172.21.15.29:0/2684115029 learned_addr learned my addr 172.21.15.29:0/2684115029 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb681479640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb681c7a640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67c06c040 0x7fb67c0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 -- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 msgr2=0x7fb67c0c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 -- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67c06c040 msgr2=0x7fb67c0aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67c06c040 0x7fb67c0aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb68247b640 1 -- 172.21.15.29:0/2684115029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c009400 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.000+0000 7fb681479640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.001+0000 7fb68247b640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 0x7fb67c156ba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb67400caf0 tx=0x7fb674002060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.001+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400a700 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.001+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67400a880 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.001+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400ac30 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.001+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c0cdcd0 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.001+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c0af760 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.002+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c06b330 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.003+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fb6740070d0 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.004+0000 7fb6727fc640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb66006b070 0x7fb66006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.004+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb674089120 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.005+0000 7fb681c7a640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb66006b070 0x7fb66006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.006+0000 7fb681c7a640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb66006b070 0x7fb66006d530 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb678000f60 tx=0x7fb678009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.009+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb674053db0 con 0x7fb67c0ccfb0 2023-10-16T14:29:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.229+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb67c0cdb50 con 0x7fb66006b070 2023-10-16T14:29:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.246+0000 7fb6727fc640 1 -- 172.21.15.29:0/2684115029 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fb678001450 con 0x7fb66006b070 2023-10-16T14:29:33.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:33.249 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:01", "first": "2023-10-16T14:19:01", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.248+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb66006b070 msgr2=0x7fb66006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.248+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb66006b070 0x7fb66006d530 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb678000f60 tx=0x7fb678009040 comp rx=0 tx=0).stop 2023-10-16T14:29:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.248+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 msgr2=0x7fb67c156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.248+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 0x7fb67c156ba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb67400caf0 tx=0x7fb674002060 comp rx=0 tx=0).stop 2023-10-16T14:29:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 shutdown_connections 2023-10-16T14:29:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb66006b070 0x7fb66006d530 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67c0ccfb0 0x7fb67c156ba0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67c0ad970 0x7fb67c0c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 --2- 172.21.15.29:0/2684115029 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67c06c040 0x7fb67c0aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 >> 172.21.15.29:0/2684115029 conn(0x7fb67c0ce6a0 msgr2=0x7fb67c15dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 shutdown_connections 2023-10-16T14:29:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.249+0000 7fb682c7c640 1 -- 172.21.15.29:0/2684115029 wait complete. 2023-10-16T14:29:33.270 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:01", "first": "2023-10-16T14:19:01", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:33.271 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:18:01', 'first': '2023-10-16T14:19:01', 'last': '2023-10-16T14:29:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:33.271 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T14:29:33.271 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T14:29:33.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:33.272 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T14:29:33.329 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:33.329 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:33.329 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:33.329 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:33.330 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:33.330 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:33.330 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:33.330 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:33.331 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:33.331 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:33.331 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:33.332 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_00_UTC', 'scheduled-2023-10-16-14_24_00_UTC', 'scheduled-2023-10-16-14_25_01_UTC', 'scheduled-2023-10-16-14_26_01_UTC', 'scheduled-2023-10-16-14_27_01_UTC', 'scheduled-2023-10-16-14_28_01_UTC', 'scheduled-2023-10-16-14_29_00_UTC'] 2023-10-16T14:29:33.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-10-16T14:29:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.555+0000 7f40cbfff640 1 Processor -- start 2023-10-16T14:29:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.555+0000 7f40cbfff640 1 -- start start 2023-10-16T14:29:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40cbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40cc15dde0 0x7f40cc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc070f20 0x7f40cc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40cbfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc160a90 con 0x7f40cc15c9f0 2023-10-16T14:29:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40cbfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc0cdec0 con 0x7f40cc15dde0 2023-10-16T14:29:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40cbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0cd900 con 0x7f40cc070f20 2023-10-16T14:29:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40c3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40cc15dde0 0x7f40cc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40c3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40cc15dde0 0x7f40cc0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53506/0 (socket says 172.21.15.29:53506) 2023-10-16T14:29:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.557+0000 7f40c3fff640 1 -- 172.21.15.29:0/1817802826 learned_addr learned my addr 172.21.15.29:0/1817802826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.558+0000 7f40cb7fe640 1 --2- 172.21.15.29:0/1817802826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc070f20 0x7f40cc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.558+0000 7f40c3fff640 1 -- 172.21.15.29:0/1817802826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc070f20 msgr2=0x7f40cc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.558+0000 7f40c3fff640 1 --2- 172.21.15.29:0/1817802826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc070f20 0x7f40cc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.558+0000 7f40c3fff640 1 -- 172.21.15.29:0/1817802826 >> v1:172.21.15.170:6789/0 conn(0x7f40cc15c9f0 legacy=0x7f40cc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.558+0000 7f40c3fff640 1 -- 172.21.15.29:0/1817802826 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0cdbe0 con 0x7f40cc15dde0 2023-10-16T14:29:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.559+0000 7f40c3fff640 1 --2- 172.21.15.29:0/1817802826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15dde0 0x7f40cc0709e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009870 tx=0x7f40b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dcbfe4c8040c76 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40ca7fc640 1 -- 172.21.15.29:0/1817802826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4030840 con 0x7f40cc15dde0 2023-10-16T14:29:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40ca7fc640 1 -- 172.21.15.29:0/1817802826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b40309a0 con 0x7f40cc15dde0 2023-10-16T14:29:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40ca7fc640 1 -- 172.21.15.29:0/1817802826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4030d30 con 0x7f40cc15dde0 2023-10-16T14:29:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 -- 172.21.15.29:0/1817802826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15dde0 msgr2=0x7f40cc0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1817802826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15dde0 0x7f40cc0709e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009870 tx=0x7f40b402b980 comp rx=0 tx=0).stop 2023-10-16T14:29:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 -- 172.21.15.29:0/1817802826 shutdown_connections 2023-10-16T14:29:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1817802826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc070f20 0x7f40cc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1817802826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15dde0 0x7f40cc0709e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 -- 172.21.15.29:0/1817802826 >> 172.21.15.29:0/1817802826 conn(0x7f40cc0ce6a0 msgr2=0x7f40cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 -- 172.21.15.29:0/1817802826 shutdown_connections 2023-10-16T14:29:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.560+0000 7f40cbfff640 1 -- 172.21.15.29:0/1817802826 wait complete. 2023-10-16T14:29:33.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.562+0000 7f40cbfff640 1 Processor -- start 2023-10-16T14:29:33.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.562+0000 7f40cbfff640 1 -- start start 2023-10-16T14:29:33.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40cc070f20 0x7f40cc06c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 0x7f40cc14ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc15dde0 0x7f40cc0c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40cbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0c4a10 con 0x7f40cc15c9f0 2023-10-16T14:29:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40cbfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0c7d40 con 0x7f40cc070f20 2023-10-16T14:29:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40cbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0c8020 con 0x7f40cc15dde0 2023-10-16T14:29:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.563+0000 7f40c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 0x7f40cc14ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 0x7f40cc14ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53512/0 (socket says 172.21.15.29:53512) 2023-10-16T14:29:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 -- 172.21.15.29:0/1487544939 learned_addr learned my addr 172.21.15.29:0/1487544939 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40caffd640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40cc070f20 0x7f40cc06c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40cb7fe640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc15dde0 0x7f40cc0c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 -- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc15dde0 msgr2=0x7f40cc0c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc15dde0 0x7f40cc0c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 -- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40cc070f20 msgr2=0x7f40cc06c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40cc070f20 0x7f40cc06c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 -- 172.21.15.29:0/1487544939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40094d0 con 0x7f40cc15c9f0 2023-10-16T14:29:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40cb7fe640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc15dde0 0x7f40cc0c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.564+0000 7f40c3fff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 0x7f40cc14ba60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f40b402b9c0 tx=0x7f40b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:33.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.565+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b403e070 con 0x7f40cc15c9f0 2023-10-16T14:29:33.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.565+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b4039a10 con 0x7f40cc15c9f0 2023-10-16T14:29:33.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.565+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4039de0 con 0x7f40cc15c9f0 2023-10-16T14:29:33.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.565+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40cc0c8300 con 0x7f40cc15c9f0 2023-10-16T14:29:33.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.565+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40cc0c7400 con 0x7f40cc15c9f0 2023-10-16T14:29:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.567+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40cc04e9b0 con 0x7f40cc15c9f0 2023-10-16T14:29:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.567+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f40b404b400 con 0x7f40cc15c9f0 2023-10-16T14:29:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.568+0000 7f40c37fe640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f40ac06b070 0x7f40ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.568+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f40b40ac160 con 0x7f40cc15c9f0 2023-10-16T14:29:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.569+0000 7f40caffd640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f40ac06b070 0x7f40ac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:33.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.570+0000 7f40caffd640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f40ac06b070 0x7f40ac06d530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40bc000f60 tx=0x7f40bc009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.573+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f40b4076e10 con 0x7f40cc15c9f0 2023-10-16T14:29:33.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.785+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40cc0704c0 con 0x7f40ac06b070 2023-10-16T14:29:33.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.804+0000 7f40c37fe640 1 -- 172.21.15.29:0/1487544939 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f40bc001360 con 0x7f40ac06b070 2023-10-16T14:29:33.807 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:33.807 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:01", "first": "2023-10-16T14:19:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:33.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f40ac06b070 msgr2=0x7f40ac06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f40ac06b070 0x7f40ac06d530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40bc000f60 tx=0x7f40bc009040 comp rx=0 tx=0).stop 2023-10-16T14:29:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 msgr2=0x7f40cc14ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 0x7f40cc14ba60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f40b402b9c0 tx=0x7f40b40046c0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 shutdown_connections 2023-10-16T14:29:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f40ac06b070 0x7f40ac06d530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40cc15c9f0 0x7f40cc14ba60 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f40cc15dde0 0x7f40cc0c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 --2- 172.21.15.29:0/1487544939 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f40cc070f20 0x7f40cc06c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 >> 172.21.15.29:0/1487544939 conn(0x7f40cc0ce6a0 msgr2=0x7f40cc15a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.807+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 shutdown_connections 2023-10-16T14:29:33.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:33.808+0000 7f40cbfff640 1 -- 172.21.15.29:0/1487544939 wait complete. 2023-10-16T14:29:33.828 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:01", "first": "2023-10-16T14:19:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:33.829 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:18:01', 'first': '2023-10-16T14:19:00', 'last': '2023-10-16T14:29:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:33.829 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T14:29:33.829 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T14:29:33.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:33.830 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T14:29:33.891 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:33.891 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:33.891 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:33.891 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:33.893 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:33.893 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:33.894 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_19_00_UTC', 'scheduled-2023-10-16-14_20_00_UTC', 'scheduled-2023-10-16-14_21_00_UTC', 'scheduled-2023-10-16-14_22_00_UTC', 'scheduled-2023-10-16-14_23_01_UTC', 'scheduled-2023-10-16-14_24_01_UTC', 'scheduled-2023-10-16-14_25_00_UTC', 'scheduled-2023-10-16-14_26_00_UTC', 'scheduled-2023-10-16-14_27_01_UTC', 'scheduled-2023-10-16-14_28_00_UTC', 'scheduled-2023-10-16-14_29_00_UTC'] 2023-10-16T14:29:33.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-10-16T14:29:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.121+0000 7fbefdd47640 1 Processor -- start 2023-10-16T14:29:34.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.122+0000 7fbefdd47640 1 -- start start 2023-10-16T14:29:34.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.123+0000 7fbefdd47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbef80ac890 0x7fbef80accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.123+0000 7fbefdd47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef8160a90 con 0x7fbef815dde0 2023-10-16T14:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.123+0000 7fbefdd47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef80ad200 con 0x7fbef815c9f0 2023-10-16T14:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.123+0000 7fbefdd47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef80cdec0 con 0x7fbef80ac890 2023-10-16T14:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.124+0000 7fbefd546640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbef80ac890 0x7fbef80accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.124+0000 7fbefd546640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbef80ac890 0x7fbef80accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59984/0 (socket says 172.21.15.29:59984) 2023-10-16T14:29:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.124+0000 7fbefd546640 1 -- 172.21.15.29:0/2177547045 learned_addr learned my addr 172.21.15.29:0/2177547045 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.124+0000 7fbefd546640 1 -- 172.21.15.29:0/2177547045 >> v1:172.21.15.170:6789/0 conn(0x7fbef815c9f0 legacy=0x7fbef80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.124+0000 7fbefd546640 1 -- 172.21.15.29:0/2177547045 >> v1:172.21.15.29:6789/0 conn(0x7fbef815dde0 legacy=0x7fbef80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.124+0000 7fbefd546640 1 -- 172.21.15.29:0/2177547045 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80cd900 con 0x7fbef80ac890 2023-10-16T14:29:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.125+0000 7fbefd546640 1 --2- 172.21.15.29:0/2177547045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef80accc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbef80ad390 tx=0x7fbef002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5f97e3325cb9aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.125+0000 7fbeef7fe640 1 -- 172.21.15.29:0/2177547045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef002cb00 con 0x7fbef80ac890 2023-10-16T14:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbeef7fe640 1 -- 172.21.15.29:0/2177547045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef002cc60 con 0x7fbef80ac890 2023-10-16T14:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbeef7fe640 1 -- 172.21.15.29:0/2177547045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef00343f0 con 0x7fbef80ac890 2023-10-16T14:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 -- 172.21.15.29:0/2177547045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 msgr2=0x7fbef80accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2177547045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef80accc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbef80ad390 tx=0x7fbef002bf60 comp rx=0 tx=0).stop 2023-10-16T14:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 -- 172.21.15.29:0/2177547045 shutdown_connections 2023-10-16T14:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2177547045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef80accc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 -- 172.21.15.29:0/2177547045 >> 172.21.15.29:0/2177547045 conn(0x7fbef80ce6a0 msgr2=0x7fbef80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 -- 172.21.15.29:0/2177547045 shutdown_connections 2023-10-16T14:29:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.126+0000 7fbefdd47640 1 -- 172.21.15.29:0/2177547045 wait complete. 2023-10-16T14:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.128+0000 7fbefdd47640 1 Processor -- start 2023-10-16T14:29:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.128+0000 7fbefdd47640 1 -- start start 2023-10-16T14:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefdd47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef8150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefdd47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef815c9f0 0x7fbef806c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefdd47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef815dde0 0x7fbef80c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefdd47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c3df0 con 0x7fbef80ac890 2023-10-16T14:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefdd47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef8151110 con 0x7fbef815dde0 2023-10-16T14:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefdd47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c31e0 con 0x7fbef815c9f0 2023-10-16T14:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefcd45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef8150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefcd45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef8150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59998/0 (socket says 172.21.15.29:59998) 2023-10-16T14:29:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.129+0000 7fbefcd45640 1 -- 172.21.15.29:0/2047825402 learned_addr learned my addr 172.21.15.29:0/2047825402 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefcd45640 1 -- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef815c9f0 msgr2=0x7fbef806c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefcd45640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef815c9f0 0x7fbef806c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefcd45640 1 -- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef815dde0 msgr2=0x7fbef80c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefcd45640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef815dde0 0x7fbef80c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefcd45640 1 -- 172.21.15.29:0/2047825402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef0009b30 con 0x7fbef80ac890 2023-10-16T14:29:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefcd45640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef8150bd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c430 tx=0x7fbef400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4011790 con 0x7fbef80ac890 2023-10-16T14:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef40118f0 con 0x7fbef80ac890 2023-10-16T14:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4011ce0 con 0x7fbef80ac890 2023-10-16T14:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.130+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef80c34c0 con 0x7fbef80ac890 2023-10-16T14:29:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.131+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef80c3a80 con 0x7fbef80ac890 2023-10-16T14:29:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.132+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef80c3c40 con 0x7fbef80ac890 2023-10-16T14:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.133+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fbef401c460 con 0x7fbef80ac890 2023-10-16T14:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.134+0000 7fbeed7fa640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbee406b0c0 0x7fbee406d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.134+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbef4089a50 con 0x7fbef80ac890 2023-10-16T14:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.134+0000 7fbeeffff640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbee406b0c0 0x7fbee406d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.135+0000 7fbeeffff640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbee406b0c0 0x7fbee406d580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000f60 tx=0x7fbee80079b0 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.138+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbef4054760 con 0x7fbef80ac890 2023-10-16T14:29:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.353+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbef80701e0 con 0x7fbee406b0c0 2023-10-16T14:29:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.371+0000 7fbeed7fa640 1 -- 172.21.15.29:0/2047825402 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fbee80012b0 con 0x7fbee406b0c0 2023-10-16T14:29:34.375 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:34.375 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:02", "first": "2023-10-16T14:19:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:34.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbee406b0c0 msgr2=0x7fbee406d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbee406b0c0 0x7fbee406d580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000f60 tx=0x7fbee80079b0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 msgr2=0x7fbef8150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef8150bd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c430 tx=0x7fbef400c8f0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 shutdown_connections 2023-10-16T14:29:34.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fbee406b0c0 0x7fbee406d580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbef815dde0 0x7fbef80c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbef815c9f0 0x7fbef806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 --2- 172.21.15.29:0/2047825402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef80ac890 0x7fbef8150bd0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.374+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 >> 172.21.15.29:0/2047825402 conn(0x7fbef80ce6a0 msgr2=0x7fbef80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:34.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.375+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 shutdown_connections 2023-10-16T14:29:34.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.375+0000 7fbefdd47640 1 -- 172.21.15.29:0/2047825402 wait complete. 2023-10-16T14:29:34.397 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:18:02", "first": "2023-10-16T14:19:00", "last": "2023-10-16T14:29:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T14:29:34.397 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:18:02', 'first': '2023-10-16T14:19:00', 'last': '2023-10-16T14:29:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T14:29:34.398 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T14:29:34.398 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T14:29:34.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-10-16T14:29:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.610+0000 7f290954e640 1 Processor -- start 2023-10-16T14:29:34.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.610+0000 7f290954e640 1 -- start start 2023-10-16T14:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.611+0000 7f290954e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f29040b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.611+0000 7f290954e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290415d150 con 0x7f29040b8480 2023-10-16T14:29:34.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.611+0000 7f290954e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29040d1180 con 0x7f290415cd70 2023-10-16T14:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.611+0000 7f290954e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29040d1460 con 0x7f29040cded0 2023-10-16T14:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.611+0000 7f29037fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f29040b8480 0x7f29040cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59892/0 (socket says 172.21.15.29:59892) 2023-10-16T14:29:34.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.611+0000 7f29037fe640 1 -- 172.21.15.29:0/1937683930 learned_addr learned my addr 172.21.15.29:0/1937683930 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2908d4d640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f29040b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457349613 0 0) 0x7f290415d150 con 0x7f29040b8480 2023-10-16T14:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f0003520 con 0x7f29040b8480 2023-10-16T14:29:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189344888 0 0) 0x7f28f0003520 con 0x7f29040b8480 2023-10-16T14:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 msgr2=0x7f29040b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f29040b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 >> v1:172.21.15.170:6789/0 conn(0x7f290415cd70 legacy=0x7f29040d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29040d1740 con 0x7f29040b8480 2023-10-16T14:29:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f28ec001690 con 0x7f29040b8480 2023-10-16T14:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.612+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 >> v1:172.21.15.29:6789/0 conn(0x7f29040b8480 legacy=0x7f29040cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2902ffd640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00039a0 0x7f28f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2902ffd640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f28f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2902ffd640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f0004d00 0x7f28f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040d1460 con 0x7f28f00039a0 2023-10-16T14:29:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f000ba60 con 0x7f28f0004d00 2023-10-16T14:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f000bc20 con 0x7f29040cded0 2023-10-16T14:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2903fff640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00039a0 0x7f28f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f29037fe640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f28f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2908d4d640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f0004d00 0x7f28f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2903fff640 1 -- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 msgr2=0x7f28f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2903fff640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f28f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2903fff640 1 -- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f0004d00 msgr2=0x7f28f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2903fff640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28f0004d00 0x7f28f0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f2903fff640 1 -- 172.21.15.29:0/1937683930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f0003520 con 0x7f28f00039a0 2023-10-16T14:29:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.613+0000 7f29037fe640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29040cded0 0x7f28f00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.614+0000 7f2903fff640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00039a0 0x7f28f0003dc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f28f4001e50 tx=0x7f28f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb79551747b42bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:34.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4003b70 con 0x7f28f00039a0 2023-10-16T14:29:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4003cd0 con 0x7f28f00039a0 2023-10-16T14:29:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f2902ffd640 1 -- 172.21.15.29:0/1937683930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4005310 con 0x7f28f00039a0 2023-10-16T14:29:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f290954e640 1 -- 172.21.15.29:0/1937683930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00039a0 msgr2=0x7f28f0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f290954e640 1 --2- 172.21.15.29:0/1937683930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28f00039a0 0x7f28f0003dc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f28f4001e50 tx=0x7f28f00057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f2908d4d640 1 -- 172.21.15.29:0/1937683930 reap_dead start 2023-10-16T14:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f290954e640 1 -- 172.21.15.29:0/1937683930 shutdown_connections 2023-10-16T14:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f290954e640 1 -- 172.21.15.29:0/1937683930 >> 172.21.15.29:0/1937683930 conn(0x7f29040cf570 msgr2=0x7f29040cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.615+0000 7f290954e640 1 -- 172.21.15.29:0/1937683930 shutdown_connections 2023-10-16T14:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.616+0000 7f290954e640 1 -- 172.21.15.29:0/1937683930 wait complete. 2023-10-16T14:29:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.617+0000 7f290954e640 1 Processor -- start 2023-10-16T14:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.617+0000 7f290954e640 1 -- start start 2023-10-16T14:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f290954e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00039a0 0x7f290415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f290954e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904151080 0x7f2904150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f290954e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 0x7f29040b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f290954e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040b4b70 con 0x7f29040b41b0 2023-10-16T14:29:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f290954e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290415f3e0 con 0x7f2904151080 2023-10-16T14:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f290954e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290415f6c0 con 0x7f28f00039a0 2023-10-16T14:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.618+0000 7f2908d4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 0x7f29040b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 0x7f29040b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60016/0 (socket says 172.21.15.29:60016) 2023-10-16T14:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 -- 172.21.15.29:0/4188776285 learned_addr learned my addr 172.21.15.29:0/4188776285 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 -- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00039a0 msgr2=0x7f290415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00039a0 0x7f290415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 -- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904151080 msgr2=0x7f2904150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904151080 0x7f2904150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 -- 172.21.15.29:0/4188776285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290415f9a0 con 0x7f29040b41b0 2023-10-16T14:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2908d4d640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 0x7f29040b4630 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f29040cd0f0 tx=0x7f28fc0109f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc014800 con 0x7f29040b41b0 2023-10-16T14:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc014980 con 0x7f29040b41b0 2023-10-16T14:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc014d30 con 0x7f29040b41b0 2023-10-16T14:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.619+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290415fc80 con 0x7f29040b41b0 2023-10-16T14:29:34.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.620+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290415eff0 con 0x7f29040b41b0 2023-10-16T14:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.621+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29041601a0 con 0x7f29040b41b0 2023-10-16T14:29:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.622+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f28fc012390 con 0x7f29040b41b0 2023-10-16T14:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.623+0000 7f2900ff9640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f28e406b070 0x7f28e406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.623+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f28fc0894e0 con 0x7f29040b41b0 2023-10-16T14:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.624+0000 7f2903fff640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f28e406b070 0x7f28e406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.625+0000 7f2903fff640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f28e406b070 0x7f28e406d530 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28f4034000 tx=0x7f28f4030000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:34.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.628+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f29041601a0 con 0x7f29040b41b0 2023-10-16T14:29:34.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:34.833+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f29040d1d00 con 0x7f28e406b070 2023-10-16T14:29:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.024+0000 7f2900ff9640 1 -- 172.21.15.29:0/4188776285 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f28f40039f0 con 0x7f28e406b070 2023-10-16T14:29:35.028 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-16T14:29:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.027+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f28e406b070 msgr2=0x7f28e406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.027+0000 7f290954e640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f28e406b070 0x7f28e406d530 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28f4034000 tx=0x7f28f4030000 comp rx=0 tx=0).stop 2023-10-16T14:29:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.027+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 msgr2=0x7f29040b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.027+0000 7f290954e640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 0x7f29040b4630 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f29040cd0f0 tx=0x7f28fc0109f0 comp rx=0 tx=0).stop 2023-10-16T14:29:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 shutdown_connections 2023-10-16T14:29:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f28e406b070 0x7f28e406d530 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29040b41b0 0x7f29040b4630 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2904151080 0x7f2904150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 --2- 172.21.15.29:0/4188776285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28f00039a0 0x7f290415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 >> 172.21.15.29:0/4188776285 conn(0x7f29040cf570 msgr2=0x7f29040cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 shutdown_connections 2023-10-16T14:29:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:35.028+0000 7f290954e640 1 -- 172.21.15.29:0/4188776285 wait complete. 2023-10-16T14:29:35.051 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-16T14:29:35.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.052 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T14:29:35.112 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:35.113 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:35.113 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:35.113 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_01_UTC 2023-10-16T14:29:35.113 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:35.114 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:35.114 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:35.114 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:35.114 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:35.114 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:35.115 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:35.115 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:35.116 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:35.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.117 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_18_00_UTC) 2023-10-16T14:29:35.185 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:35.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.186 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:35.263 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:35.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.264 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:35.340 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_21_01_UTC 2023-10-16T14:29:35.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.341 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_21_01_UTC) 2023-10-16T14:29:35.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:35.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.415 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_22_01_UTC) 2023-10-16T14:29:35.495 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:35.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.496 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_23_01_UTC) 2023-10-16T14:29:35.572 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:35.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.574 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_24_00_UTC) 2023-10-16T14:29:35.649 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:35.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.650 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_25_00_UTC) 2023-10-16T14:29:35.735 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:35.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.736 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_26_00_UTC) 2023-10-16T14:29:35.821 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:35.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.822 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_27_00_UTC) 2023-10-16T14:29:35.907 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:35.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.908 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_28_00_UTC) 2023-10-16T14:29:35.989 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:35.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:35.990 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-14_29_00_UTC) 2023-10-16T14:29:36.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:36.067 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-16T14:29:36.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-10-16T14:29:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.366+0000 7f75b3b01640 1 Processor -- start 2023-10-16T14:29:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.366+0000 7f75b3b01640 1 -- start start 2023-10-16T14:29:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.367+0000 7f75b3b01640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 0x7f75ac0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.367+0000 7f75b3b01640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75ac06c140 0x7f75ac06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b3b01640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac070360 con 0x7f75ac159ab0 2023-10-16T14:29:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b3b01640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac06cb10 con 0x7f75ac06c140 2023-10-16T14:29:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b3b01640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac15d890 con 0x7f75ac159e90 2023-10-16T14:29:36.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b189d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f75ac159ab0 0x7f75ac06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59894/0 (socket says 172.21.15.29:59894) 2023-10-16T14:29:36.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b189d640 1 -- 172.21.15.29:0/3964174992 learned_addr learned my addr 172.21.15.29:0/3964174992 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:36.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b209e640 1 --2- 172.21.15.29:0/3964174992 >> v2:172.21.15.170:3300/0 conn(0x7f75ac06c140 0x7f75ac06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.368+0000 7f75b109c640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 0x7f75ac0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481249052 0 0) 0x7f75ac070360 con 0x7f75ac159ab0 2023-10-16T14:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7598003520 con 0x7f75ac159ab0 2023-10-16T14:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2100188030 0 0) 0x7f7598003520 con 0x7f75ac159ab0 2023-10-16T14:29:36.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 msgr2=0x7f75ac0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 0x7f75ac0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 >> v2:172.21.15.170:3300/0 conn(0x7f75ac06c140 msgr2=0x7f75ac06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 --2- 172.21.15.29:0/3964174992 >> v2:172.21.15.170:3300/0 conn(0x7f75ac06c140 0x7f75ac06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0cd1f0 con 0x7f75ac159ab0 2023-10-16T14:29:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f75a8001670 con 0x7f75ac159ab0 2023-10-16T14:29:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 >> v1:172.21.15.29:6789/0 conn(0x7f75ac159ab0 legacy=0x7f75ac06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 0x7f7598003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7598003f40 0x7f75980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f7598005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac06cb10 con 0x7f7598004d00 2023-10-16T14:29:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac15d890 con 0x7f7598003f40 2023-10-16T14:29:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.369+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759800bab0 con 0x7f75ac159e90 2023-10-16T14:29:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b209e640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f7598005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b189d640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 0x7f7598003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b109c640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7598003f40 0x7f75980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b209e640 1 -- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 msgr2=0x7f7598003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b209e640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac159e90 0x7f7598003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b209e640 1 -- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7598003f40 msgr2=0x7f75980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b209e640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7598003f40 0x7f75980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b209e640 1 -- 172.21.15.29:0/3964174992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598003520 con 0x7f7598004d00 2023-10-16T14:29:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.370+0000 7f75b109c640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7598003f40 0x7f75980047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.371+0000 7f75b209e640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f7598005180 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009fc0 tx=0x7f75980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5c1926a980b597 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a402cc60 con 0x7f7598004d00 2023-10-16T14:29:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400f040 con 0x7f7598004d00 2023-10-16T14:29:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b089b640 1 -- 172.21.15.29:0/3964174992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4034440 con 0x7f7598004d00 2023-10-16T14:29:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b3b01640 1 -- 172.21.15.29:0/3964174992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 msgr2=0x7f7598005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b3b01640 1 --2- 172.21.15.29:0/3964174992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f7598005180 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009fc0 tx=0x7f75980057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b209e640 1 -- 172.21.15.29:0/3964174992 reap_dead start 2023-10-16T14:29:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b3b01640 1 -- 172.21.15.29:0/3964174992 shutdown_connections 2023-10-16T14:29:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b3b01640 1 -- 172.21.15.29:0/3964174992 >> 172.21.15.29:0/3964174992 conn(0x7f75ac0ce9b0 msgr2=0x7f75ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.372+0000 7f75b3b01640 1 -- 172.21.15.29:0/3964174992 shutdown_connections 2023-10-16T14:29:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.373+0000 7f75b3b01640 1 -- 172.21.15.29:0/3964174992 wait complete. 2023-10-16T14:29:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.374+0000 7f75b3b01640 1 Processor -- start 2023-10-16T14:29:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.374+0000 7f75b3b01640 1 -- start start 2023-10-16T14:29:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.375+0000 7f75b3b01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f75ac1e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.375+0000 7f75b3b01640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75ac0c8de0 0x7f75ac1f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.375+0000 7f75b3b01640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac0b3a20 0x7f75ac14bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.375+0000 7f75b3b01640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c6670 con 0x7f7598004d00 2023-10-16T14:29:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.375+0000 7f75b3b01640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c5a80 con 0x7f75ac0c8de0 2023-10-16T14:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.375+0000 7f75b3b01640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c5cd0 con 0x7f75ac0b3a20 2023-10-16T14:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f75ac1e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f75ac1e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60030/0 (socket says 172.21.15.29:60030) 2023-10-16T14:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 -- 172.21.15.29:0/1623486305 learned_addr learned my addr 172.21.15.29:0/1623486305 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b109c640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75ac0c8de0 0x7f75ac1f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b209e640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac0b3a20 0x7f75ac14bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 -- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac0b3a20 msgr2=0x7f75ac14bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac0b3a20 0x7f75ac14bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 -- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75ac0c8de0 msgr2=0x7f75ac1f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75ac0c8de0 0x7f75ac1f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 -- 172.21.15.29:0/1623486305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0c5fb0 con 0x7f7598004d00 2023-10-16T14:29:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b109c640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75ac0c8de0 0x7f75ac1f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.376+0000 7f75b189d640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f75ac1e4bc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f75a800e7b0 tx=0x7f75a800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.377+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a800ee60 con 0x7f7598004d00 2023-10-16T14:29:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.377+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a800c870 con 0x7f7598004d00 2023-10-16T14:29:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.377+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac0c6290 con 0x7f7598004d00 2023-10-16T14:29:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.377+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a800cc20 con 0x7f7598004d00 2023-10-16T14:29:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.377+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac1f0630 con 0x7f7598004d00 2023-10-16T14:29:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.378+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac04e160 con 0x7f7598004d00 2023-10-16T14:29:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.379+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f75a801d460 con 0x7f7598004d00 2023-10-16T14:29:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.380+0000 7f75a27fc640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f759006b070 0x7f759006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.380+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f75a8089980 con 0x7f7598004d00 2023-10-16T14:29:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.381+0000 7f75b109c640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f759006b070 0x7f759006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.382+0000 7f75b109c640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f759006b070 0x7f759006d530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0af8a0 tx=0x7f759c008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.384+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75a80546b0 con 0x7f7598004d00 2023-10-16T14:29:36.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.600+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f75ac0cd7b0 con 0x7f759006b070 2023-10-16T14:29:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.803+0000 7f75a27fc640 1 -- 172.21.15.29:0/1623486305 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f759c009730 con 0x7f759006b070 2023-10-16T14:29:36.806 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-16T14:29:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f759006b070 msgr2=0x7f759006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f759006b070 0x7f759006d530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0af8a0 tx=0x7f759c008040 comp rx=0 tx=0).stop 2023-10-16T14:29:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 msgr2=0x7f75ac1e4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f75ac1e4bc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f75a800e7b0 tx=0x7f75a800ec70 comp rx=0 tx=0).stop 2023-10-16T14:29:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 shutdown_connections 2023-10-16T14:29:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f759006b070 0x7f759006d530 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75ac0b3a20 0x7f75ac14bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75ac0c8de0 0x7f75ac1f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 --2- 172.21.15.29:0/1623486305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598004d00 0x7f75ac1e4bc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.806+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 >> 172.21.15.29:0/1623486305 conn(0x7f75ac0ce9b0 msgr2=0x7f75ac0c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.807+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 shutdown_connections 2023-10-16T14:29:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:36.807+0000 7f75b3b01640 1 -- 172.21.15.29:0/1623486305 wait complete. 2023-10-16T14:29:36.829 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-16T14:29:36.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:36.830 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T14:29:36.886 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:36.886 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:36.886 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:36.886 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:36.886 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:36.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:36.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:36.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:36.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:36.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:36.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:36.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:36.889 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:36.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:36.889 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_18_00_UTC) 2023-10-16T14:29:36.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:36.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:36.963 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:37.037 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:37.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.038 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:37.110 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:37.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.111 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:37.184 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:37.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.185 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:37.259 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:37.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.259 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_23_00_UTC) 2023-10-16T14:29:37.335 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:37.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.336 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_24_00_UTC) 2023-10-16T14:29:37.416 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:37.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.416 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_25_00_UTC) 2023-10-16T14:29:37.491 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:37.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.492 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_26_01_UTC) 2023-10-16T14:29:37.568 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:37.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.569 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_27_00_UTC) 2023-10-16T14:29:37.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:37.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.644 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_28_00_UTC) 2023-10-16T14:29:37.717 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:37.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.719 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-14_29_01_UTC) 2023-10-16T14:29:37.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:37.796 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-16T14:29:37.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-10-16T14:29:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.089+0000 7f1346256640 1 Processor -- start 2023-10-16T14:29:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.089+0000 7f1346256640 1 -- start start 2023-10-16T14:29:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.090+0000 7f1346256640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13400ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.090+0000 7f1346256640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13400c9ca0 0x7f13400ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:38.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.090+0000 7f1346256640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13400cb030 con 0x7f134015ad70 2023-10-16T14:29:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.090+0000 7f1346256640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13400cae30 con 0x7f13400c9ca0 2023-10-16T14:29:38.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.090+0000 7f1346256640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13400c91c0 con 0x7f134006a4b0 2023-10-16T14:29:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13400c9ca0 0x7f13400ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13400c9ca0 0x7f13400ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60036/0 (socket says 172.21.15.29:60036) 2023-10-16T14:29:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133ffff640 1 -- 172.21.15.29:0/316613555 learned_addr learned my addr 172.21.15.29:0/316613555 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:38.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133effd640 1 --2- 172.21.15.29:0/316613555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13400ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133ffff640 1 -- 172.21.15.29:0/316613555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 msgr2=0x7f13400ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133ffff640 1 --2- 172.21.15.29:0/316613555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13400ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.091+0000 7f133ffff640 1 -- 172.21.15.29:0/316613555 >> v1:172.21.15.170:6789/0 conn(0x7f134015ad70 legacy=0x7f1340069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:38.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.092+0000 7f133ffff640 1 -- 172.21.15.29:0/316613555 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13400c94a0 con 0x7f13400c9ca0 2023-10-16T14:29:38.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.093+0000 7f133ffff640 1 --2- 172.21.15.29:0/316613555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f13400ca8f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1338009e50 tx=0x7f133802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0d0461b103d3e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.093+0000 7f133e7fc640 1 -- 172.21.15.29:0/316613555 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133802cb00 con 0x7f13400c9ca0 2023-10-16T14:29:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.093+0000 7f133e7fc640 1 -- 172.21.15.29:0/316613555 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133802cc60 con 0x7f13400c9ca0 2023-10-16T14:29:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.093+0000 7f133e7fc640 1 -- 172.21.15.29:0/316613555 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13380343f0 con 0x7f13400c9ca0 2023-10-16T14:29:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 -- 172.21.15.29:0/316613555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 msgr2=0x7f13400ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 --2- 172.21.15.29:0/316613555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f13400ca8f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1338009e50 tx=0x7f133802bf60 comp rx=0 tx=0).stop 2023-10-16T14:29:38.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 -- 172.21.15.29:0/316613555 shutdown_connections 2023-10-16T14:29:38.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 --2- 172.21.15.29:0/316613555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f13400ca8f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 --2- 172.21.15.29:0/316613555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13400ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 -- 172.21.15.29:0/316613555 >> 172.21.15.29:0/316613555 conn(0x7f13400ce9b0 msgr2=0x7f13400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:38.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 -- 172.21.15.29:0/316613555 shutdown_connections 2023-10-16T14:29:38.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.094+0000 7f1346256640 1 -- 172.21.15.29:0/316613555 wait complete. 2023-10-16T14:29:38.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.096+0000 7f1346256640 1 Processor -- start 2023-10-16T14:29:38.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.096+0000 7f1346256640 1 -- start start 2023-10-16T14:29:38.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f1346256640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13401f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:38.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f1346256640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f134015de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:38.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f1346256640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f134015ad70 0x7f1340156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:38.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f1346256640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134015e3c0 con 0x7f13400c9ca0 2023-10-16T14:29:38.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f1346256640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13401f32a0 con 0x7f134015ad70 2023-10-16T14:29:38.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f1346256640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13401f2a00 con 0x7f134006a4b0 2023-10-16T14:29:38.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f133effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f134015de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:38.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f133effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f134015de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60048/0 (socket says 172.21.15.29:60048) 2023-10-16T14:29:38.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.097+0000 7f133effd640 1 -- 172.21.15.29:0/3706297427 learned_addr learned my addr 172.21.15.29:0/3706297427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:38.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133f7fe640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13401f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:38.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133effd640 1 -- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 msgr2=0x7f13401f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:38.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133effd640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13401f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133effd640 1 -- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f134015ad70 msgr2=0x7f1340156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:38.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133effd640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f134015ad70 0x7f1340156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133effd640 1 -- 172.21.15.29:0/3706297427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338009b30 con 0x7f13400c9ca0 2023-10-16T14:29:38.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133f7fe640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13401f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:38.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.098+0000 7f133effd640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f134015de80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f133400d800 tx=0x7f133400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:38.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.099+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133400b6c0 con 0x7f13400c9ca0 2023-10-16T14:29:38.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.099+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133400b820 con 0x7f13400c9ca0 2023-10-16T14:29:38.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.099+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133400bbb0 con 0x7f13400c9ca0 2023-10-16T14:29:38.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.099+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13401f2ce0 con 0x7f13400c9ca0 2023-10-16T14:29:38.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.099+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134015c5d0 con 0x7f13400c9ca0 2023-10-16T14:29:38.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.100+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134004e160 con 0x7f13400c9ca0 2023-10-16T14:29:38.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.101+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f133400bd10 con 0x7f13400c9ca0 2023-10-16T14:29:38.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.102+0000 7f1327fff640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f132c06b070 0x7f132c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:38.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.102+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1334089ff0 con 0x7f13400c9ca0 2023-10-16T14:29:38.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.103+0000 7f133f7fe640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f132c06b070 0x7f132c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:38.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.104+0000 7f133f7fe640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f132c06b070 0x7f132c06d530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1330000f60 tx=0x7f1330009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:38.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.107+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1334054d70 con 0x7f13400c9ca0 2023-10-16T14:29:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.452+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f13400c9780 con 0x7f132c06b070 2023-10-16T14:29:38.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.628+0000 7f1327fff640 1 -- 172.21.15.29:0/3706297427 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f1330001340 con 0x7f132c06b070 2023-10-16T14:29:38.632 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-16T14:29:38.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.631+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f132c06b070 msgr2=0x7f132c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:38.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f132c06b070 0x7f132c06d530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1330000f60 tx=0x7f1330009040 comp rx=0 tx=0).stop 2023-10-16T14:29:38.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 msgr2=0x7f134015de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:38.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f134015de80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f133400d800 tx=0x7f133400dcc0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 shutdown_connections 2023-10-16T14:29:38.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f132c06b070 0x7f132c06d530 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400c9ca0 0x7f134015de80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f134015ad70 0x7f1340156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 --2- 172.21.15.29:0/3706297427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f134006a4b0 0x7f13401f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 >> 172.21.15.29:0/3706297427 conn(0x7f13400ce9b0 msgr2=0x7f13400b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 shutdown_connections 2023-10-16T14:29:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:38.632+0000 7f1346256640 1 -- 172.21.15.29:0/3706297427 wait complete. 2023-10-16T14:29:38.659 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-16T14:29:38.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:38.659 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T14:29:38.716 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:38.716 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:38.716 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:38.716 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:38.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:38.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:38.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:38.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:38.717 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:38.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:38.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:38.718 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:38.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:38.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:38.720 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_18_00_UTC) 2023-10-16T14:29:38.796 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:38.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:38.797 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_19_01_UTC) 2023-10-16T14:29:38.846 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:38.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:38.847 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:38.902 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:38.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:38.903 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:38.972 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:38.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:38.973 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:39.049 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:39.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.050 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_23_00_UTC) 2023-10-16T14:29:39.123 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:39.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.124 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_24_01_UTC) 2023-10-16T14:29:39.197 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:39.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.199 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_25_00_UTC) 2023-10-16T14:29:39.253 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:39.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.255 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_26_00_UTC) 2023-10-16T14:29:39.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:39.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.308 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_27_00_UTC) 2023-10-16T14:29:39.364 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:39.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.365 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_28_01_UTC) 2023-10-16T14:29:39.426 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:39.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.427 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-14_29_01_UTC) 2023-10-16T14:29:39.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:39.485 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-16T14:29:39.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-10-16T14:29:39.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.755+0000 7f1850cbb640 1 Processor -- start 2023-10-16T14:29:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.756+0000 7f1850cbb640 1 -- start start 2023-10-16T14:29:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f1850cbb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c070b60 0x7f184c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f1850cbb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f184c06c100 0x7f184c06c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f1850cbb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c15aef0 con 0x7f184c159950 2023-10-16T14:29:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f1850cbb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c15e490 con 0x7f184c06c100 2023-10-16T14:29:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f1850cbb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c15d910 con 0x7f184c070b60 2023-10-16T14:29:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f184b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f184c159950 0x7f184c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59908/0 (socket says 172.21.15.29:59908) 2023-10-16T14:29:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f184b7fe640 1 -- 172.21.15.29:0/672443124 learned_addr learned my addr 172.21.15.29:0/672443124 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.757+0000 7f184bfff640 1 --2- 172.21.15.29:0/672443124 >> v2:172.21.15.29:3300/0 conn(0x7f184c06c100 0x7f184c06c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184a7fc640 1 -- 172.21.15.29:0/672443124 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833773610 0 0) 0x7f183c001110 con 0x7f184c159950 2023-10-16T14:29:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184affd640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c070b60 0x7f184c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184a7fc640 1 -- 172.21.15.29:0/672443124 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834003520 con 0x7f184c159950 2023-10-16T14:29:39.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184bfff640 1 -- 172.21.15.29:0/672443124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c070b60 msgr2=0x7f184c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:39.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184bfff640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c070b60 0x7f184c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:39.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184bfff640 1 -- 172.21.15.29:0/672443124 >> v1:172.21.15.29:6789/0 conn(0x7f184c159950 legacy=0x7f184c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:39.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184bfff640 1 -- 172.21.15.29:0/672443124 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c15dbf0 con 0x7f184c06c100 2023-10-16T14:29:39.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.758+0000 7f184affd640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c070b60 0x7f184c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:39.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.759+0000 7f184bfff640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c06c100 0x7f184c06c550 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1844009dc0 tx=0x7f184402bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0abf2716ec699a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:39.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.759+0000 7f184a7fc640 1 -- 172.21.15.29:0/672443124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184402cba0 con 0x7f184c06c100 2023-10-16T14:29:39.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.759+0000 7f184a7fc640 1 -- 172.21.15.29:0/672443124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184402cd00 con 0x7f184c06c100 2023-10-16T14:29:39.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.759+0000 7f184a7fc640 1 -- 172.21.15.29:0/672443124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844034640 con 0x7f184c06c100 2023-10-16T14:29:39.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 -- 172.21.15.29:0/672443124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c06c100 msgr2=0x7f184c06c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:39.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c06c100 0x7f184c06c550 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1844009dc0 tx=0x7f184402bdf0 comp rx=0 tx=0).stop 2023-10-16T14:29:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 -- 172.21.15.29:0/672443124 shutdown_connections 2023-10-16T14:29:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c06c100 0x7f184c06c550 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 --2- 172.21.15.29:0/672443124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c070b60 0x7f184c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 -- 172.21.15.29:0/672443124 >> 172.21.15.29:0/672443124 conn(0x7f184c0ce6a0 msgr2=0x7f184c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 -- 172.21.15.29:0/672443124 shutdown_connections 2023-10-16T14:29:39.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.760+0000 7f1850cbb640 1 -- 172.21.15.29:0/672443124 wait complete. 2023-10-16T14:29:39.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.762+0000 7f1850cbb640 1 Processor -- start 2023-10-16T14:29:39.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.762+0000 7f1850cbb640 1 -- start start 2023-10-16T14:29:39.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f1850cbb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f184c06c100 0x7f184c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:39.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f1850cbb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 0x7f184c154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:39.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f1850cbb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c159950 0x7f184c14e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:39.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f1850cbb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c15b230 con 0x7f184c070b60 2023-10-16T14:29:39.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f1850cbb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0c7d90 con 0x7f184c06c100 2023-10-16T14:29:39.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f1850cbb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0c8070 con 0x7f184c159950 2023-10-16T14:29:39.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f184affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 0x7f184c154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:39.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f184affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 0x7f184c154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60070/0 (socket says 172.21.15.29:60070) 2023-10-16T14:29:39.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.763+0000 7f184affd640 1 -- 172.21.15.29:0/3987112709 learned_addr learned my addr 172.21.15.29:0/3987112709 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:39.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184b7fe640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f184c06c100 0x7f184c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:39.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184bfff640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c159950 0x7f184c14e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:39.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184affd640 1 -- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c159950 msgr2=0x7f184c14e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:39.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184affd640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c159950 0x7f184c14e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:39.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184affd640 1 -- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f184c06c100 msgr2=0x7f184c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:39.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184affd640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f184c06c100 0x7f184c0c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:39.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184affd640 1 -- 172.21.15.29:0/3987112709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844009a70 con 0x7f184c070b60 2023-10-16T14:29:39.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184bfff640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c159950 0x7f184c14e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:39.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f184affd640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 0x7f184c154860 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f184000c7f0 tx=0x7f184000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.764+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840007a30 con 0x7f184c070b60 2023-10-16T14:29:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.765+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1840007b90 con 0x7f184c070b60 2023-10-16T14:29:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.765+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c0c8350 con 0x7f184c070b60 2023-10-16T14:29:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.765+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184000ce40 con 0x7f184c070b60 2023-10-16T14:29:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.765+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c15d180 con 0x7f184c070b60 2023-10-16T14:29:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.766+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c15b370 con 0x7f184c070b60 2023-10-16T14:29:39.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.767+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f18400113f0 con 0x7f184c070b60 2023-10-16T14:29:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.768+0000 7f1833fff640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f183806b070 0x7f183806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.768+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f1840089e40 con 0x7f184c070b60 2023-10-16T14:29:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.768+0000 7f184b7fe640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f183806b070 0x7f183806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.770+0000 7f184b7fe640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f183806b070 0x7f183806d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f183c007fd0 tx=0x7f183c009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.772+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f184c15b370 con 0x7f184c070b60 2023-10-16T14:29:39.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:39.987+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f184c15e1b0 con 0x7f183806b070 2023-10-16T14:29:40.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.201+0000 7f1833fff640 1 -- 172.21.15.29:0/3987112709 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f183c0058b0 con 0x7f183806b070 2023-10-16T14:29:40.205 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-16T14:29:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f183806b070 msgr2=0x7f183806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f183806b070 0x7f183806d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f183c007fd0 tx=0x7f183c009040 comp rx=0 tx=0).stop 2023-10-16T14:29:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 msgr2=0x7f184c154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 0x7f184c154860 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f184000c7f0 tx=0x7f184000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:29:40.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 shutdown_connections 2023-10-16T14:29:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f183806b070 0x7f183806d530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184c070b60 0x7f184c154860 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f184c159950 0x7f184c14e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 --2- 172.21.15.29:0/3987112709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f184c06c100 0x7f184c0c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:40.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 >> 172.21.15.29:0/3987112709 conn(0x7f184c0ce6a0 msgr2=0x7f184c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 shutdown_connections 2023-10-16T14:29:40.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:40.204+0000 7f1850cbb640 1 -- 172.21.15.29:0/3987112709 wait complete. 2023-10-16T14:29:40.228 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-16T14:29:40.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.229 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T14:29:40.280 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:40.281 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:40.281 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:40.281 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:40.281 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:40.281 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:40.282 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:40.282 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:40.282 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:40.282 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:40.282 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:40.283 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:40.284 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:40.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.284 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_18_00_UTC) 2023-10-16T14:29:40.365 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:40.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.366 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:40.432 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:40.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.433 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_20_01_UTC) 2023-10-16T14:29:40.492 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:40.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.493 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:40.558 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:40.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.559 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:40.627 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:40.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.628 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_23_00_UTC) 2023-10-16T14:29:40.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:40.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.701 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_24_01_UTC) 2023-10-16T14:29:40.758 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:40.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.759 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_25_01_UTC) 2023-10-16T14:29:40.827 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:40.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.828 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_26_01_UTC) 2023-10-16T14:29:40.902 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:40.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.903 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_27_01_UTC) 2023-10-16T14:29:40.967 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:40.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:40.968 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_28_01_UTC) 2023-10-16T14:29:41.034 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:41.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:41.035 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-14_29_01_UTC) 2023-10-16T14:29:41.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:41.100 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-16T14:29:41.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-10-16T14:29:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.379+0000 7fa7e5555640 1 Processor -- start 2023-10-16T14:29:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.379+0000 7fa7e5555640 1 -- start start 2023-10-16T14:29:41.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7e5555640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7e00cf710 0x7fa7e00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:41.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7e5555640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa7e00b6cd0 0x7fa7e00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:41.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7e5555640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00d3980 con 0x7fa7e00cfaf0 2023-10-16T14:29:41.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7e5555640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e00d3c60 con 0x7fa7e00cf710 2023-10-16T14:29:41.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7e5555640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e00ce630 con 0x7fa7e00b6cd0 2023-10-16T14:29:41.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7deffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7e00cf710 0x7fa7e00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:41.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7de7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa7e00cfaf0 0x7fa7e00d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59920/0 (socket says 172.21.15.29:59920) 2023-10-16T14:29:41.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.381+0000 7fa7de7fc640 1 -- 172.21.15.29:0/1284088981 learned_addr learned my addr 172.21.15.29:0/1284088981 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:41.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.382+0000 7fa7deffd640 1 -- 172.21.15.29:0/1284088981 >> v2:172.21.15.170:3300/0 conn(0x7fa7e00b6cd0 msgr2=0x7fa7e00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:41.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.382+0000 7fa7df7fe640 1 --2- 172.21.15.29:0/1284088981 >> v2:172.21.15.170:3300/0 conn(0x7fa7e00b6cd0 0x7fa7e00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:41.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.382+0000 7fa7deffd640 1 --2- 172.21.15.29:0/1284088981 >> v2:172.21.15.170:3300/0 conn(0x7fa7e00b6cd0 0x7fa7e00b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.382+0000 7fa7deffd640 1 -- 172.21.15.29:0/1284088981 >> v1:172.21.15.29:6789/0 conn(0x7fa7e00cfaf0 legacy=0x7fa7e00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:41.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.382+0000 7fa7ddffb640 1 -- 172.21.15.29:0/1284088981 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565898019 0 0) 0x7fa7e00d3980 con 0x7fa7e00cfaf0 2023-10-16T14:29:41.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.382+0000 7fa7deffd640 1 -- 172.21.15.29:0/1284088981 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00ce910 con 0x7fa7e00cf710 2023-10-16T14:29:41.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.383+0000 7fa7deffd640 1 --2- 172.21.15.29:0/1284088981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00b2d10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00b5890 tx=0x7fa7d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6729a4d4244931 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:41.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.383+0000 7fa7ddffb640 1 -- 172.21.15.29:0/1284088981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d002e070 con 0x7fa7e00cf710 2023-10-16T14:29:41.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.383+0000 7fa7ddffb640 1 -- 172.21.15.29:0/1284088981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d00315a0 con 0x7fa7e00cf710 2023-10-16T14:29:41.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.383+0000 7fa7ddffb640 1 -- 172.21.15.29:0/1284088981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0031930 con 0x7fa7e00cf710 2023-10-16T14:29:41.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 -- 172.21.15.29:0/1284088981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 msgr2=0x7fa7e00b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:41.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 --2- 172.21.15.29:0/1284088981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00b2d10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00b5890 tx=0x7fa7d002b840 comp rx=0 tx=0).stop 2023-10-16T14:29:41.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 -- 172.21.15.29:0/1284088981 shutdown_connections 2023-10-16T14:29:41.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 --2- 172.21.15.29:0/1284088981 >> v2:172.21.15.170:3300/0 conn(0x7fa7e00b6cd0 0x7fa7e00b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 --2- 172.21.15.29:0/1284088981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00b2d10 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 -- 172.21.15.29:0/1284088981 >> 172.21.15.29:0/1284088981 conn(0x7fa7e00d2cc0 msgr2=0x7fa7e00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:41.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 -- 172.21.15.29:0/1284088981 shutdown_connections 2023-10-16T14:29:41.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.384+0000 7fa7e5555640 1 -- 172.21.15.29:0/1284088981 wait complete. 2023-10-16T14:29:41.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.386+0000 7fa7e5555640 1 Processor -- start 2023-10-16T14:29:41.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.386+0000 7fa7e5555640 1 -- start start 2023-10-16T14:29:41.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.387+0000 7fa7e5555640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e00b6cd0 0x7fa7e00b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:41.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.387+0000 7fa7e5555640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:41.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.387+0000 7fa7e5555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7e00cfaf0 0x7fa7e00d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:41.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.387+0000 7fa7e5555640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00b87a0 con 0x7fa7e00cf710 2023-10-16T14:29:41.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7e5555640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00b0780 con 0x7fa7e00b6cd0 2023-10-16T14:29:41.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7e5555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00caa80 con 0x7fa7e00cfaf0 2023-10-16T14:29:41.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:41.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60080/0 (socket says 172.21.15.29:60080) 2023-10-16T14:29:41.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 -- 172.21.15.29:0/4184975201 learned_addr learned my addr 172.21.15.29:0/4184975201 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:41.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 -- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7e00cfaf0 msgr2=0x7fa7e00d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:41.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7e00cfaf0 0x7fa7e00d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 -- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e00b6cd0 msgr2=0x7fa7e00b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:41.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e00b6cd0 0x7fa7e00b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.388+0000 7fa7de7fc640 1 -- 172.21.15.29:0/4184975201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d00093a0 con 0x7fa7e00cf710 2023-10-16T14:29:41.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.389+0000 7fa7de7fc640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00ca540 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800c500 tx=0x7fa7c800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:41.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.389+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8007560 con 0x7fa7e00cf710 2023-10-16T14:29:41.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.389+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c80047e0 con 0x7fa7e00cf710 2023-10-16T14:29:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.389+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e00af890 con 0x7fa7e00cf710 2023-10-16T14:29:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.389+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8004b90 con 0x7fa7e00cf710 2023-10-16T14:29:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.389+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e00afe50 con 0x7fa7e00cf710 2023-10-16T14:29:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.390+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e004e160 con 0x7fa7e00cf710 2023-10-16T14:29:41.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.391+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa7c8004cf0 con 0x7fa7e00cf710 2023-10-16T14:29:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.392+0000 7fa7d77fe640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa7c006b070 0x7fa7c006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.392+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fa7c8089230 con 0x7fa7e00cf710 2023-10-16T14:29:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.393+0000 7fa7deffd640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa7c006b070 0x7fa7c006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:41.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.394+0000 7fa7deffd640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa7c006b070 0x7fa7c006d530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0036000 tx=0x7fa7d0030000 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:41.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.397+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7c8053ce0 con 0x7fa7e00cf710 2023-10-16T14:29:41.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.620+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7e00b08f0 con 0x7fa7c006b070 2023-10-16T14:29:41.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.816+0000 7fa7d77fe640 1 -- 172.21.15.29:0/4184975201 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fa7e00b08f0 con 0x7fa7c006b070 2023-10-16T14:29:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-16T14:29:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa7c006b070 msgr2=0x7fa7c006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa7c006b070 0x7fa7c006d530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0036000 tx=0x7fa7d0030000 comp rx=0 tx=0).stop 2023-10-16T14:29:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 msgr2=0x7fa7e00ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00ca540 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800c500 tx=0x7fa7c800c9c0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 shutdown_connections 2023-10-16T14:29:41.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa7c006b070 0x7fa7c006d530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7e00cf710 0x7fa7e00ca540 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7e00cfaf0 0x7fa7e00d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 --2- 172.21.15.29:0/4184975201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7e00b6cd0 0x7fa7e00b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 >> 172.21.15.29:0/4184975201 conn(0x7fa7e00d2cc0 msgr2=0x7fa7e01601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.820+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 shutdown_connections 2023-10-16T14:29:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:41.821+0000 7fa7e5555640 1 -- 172.21.15.29:0/4184975201 wait complete. 2023-10-16T14:29:41.845 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-16T14:29:41.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:41.846 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T14:29:41.905 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:41.906 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:41.906 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:41.906 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_01_UTC 2023-10-16T14:29:41.906 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:41.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:41.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:41.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:41.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:41.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:41.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:41.908 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:41.909 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:41.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:41.909 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_18_00_UTC) 2023-10-16T14:29:41.977 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:41.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:41.977 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_19_01_UTC) 2023-10-16T14:29:42.042 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:42.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.043 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_20_01_UTC) 2023-10-16T14:29:42.112 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_21_01_UTC 2023-10-16T14:29:42.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.113 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_21_01_UTC) 2023-10-16T14:29:42.212 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:42.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.213 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_22_01_UTC) 2023-10-16T14:29:42.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:42.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.297 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_23_01_UTC) 2023-10-16T14:29:42.349 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:42.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.350 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_24_01_UTC) 2023-10-16T14:29:42.400 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:42.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.401 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_25_00_UTC) 2023-10-16T14:29:42.453 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:42.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.454 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_26_00_UTC) 2023-10-16T14:29:42.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:42.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.515 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_27_00_UTC) 2023-10-16T14:29:42.583 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:42.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.584 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_28_00_UTC) 2023-10-16T14:29:42.650 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_29_01_UTC 2023-10-16T14:29:42.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.651 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-14_29_01_UTC) 2023-10-16T14:29:42.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:42.736 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-16T14:29:42.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-10-16T14:29:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.983+0000 7efdb4ba9640 1 Processor -- start 2023-10-16T14:29:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.984+0000 7efdb4ba9640 1 -- start start 2023-10-16T14:29:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.985+0000 7efdb4ba9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 0x7efdb0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.985+0000 7efdb4ba9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb00cb030 con 0x7efdb00c9ca0 2023-10-16T14:29:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.985+0000 7efdb4ba9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb00c91c0 con 0x7efdb00caac0 2023-10-16T14:29:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.985+0000 7efdb4ba9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c94a0 con 0x7efdb015ad70 2023-10-16T14:29:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdadd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efdb00c9ca0 0x7efdb00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59932/0 (socket says 172.21.15.29:59932) 2023-10-16T14:29:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdadd9b640 1 -- 172.21.15.29:0/890455867 learned_addr learned my addr 172.21.15.29:0/890455867 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdae59c640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 0x7efdb0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427598886 0 0) 0x7efdb00cb030 con 0x7efdb00c9ca0 2023-10-16T14:29:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c003520 con 0x7efdb00c9ca0 2023-10-16T14:29:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1489446824 0 0) 0x7efd9c003520 con 0x7efdb00c9ca0 2023-10-16T14:29:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 msgr2=0x7efdb0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.986+0000 7efdad59a640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 0x7efdb0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 >> v1:172.21.15.170:6789/0 conn(0x7efdb00caac0 legacy=0x7efdb00ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00c9780 con 0x7efdb00c9ca0 2023-10-16T14:29:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7efd98001670 con 0x7efdb00c9ca0 2023-10-16T14:29:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 >> v1:172.21.15.29:6789/0 conn(0x7efdb00c9ca0 legacy=0x7efdb00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efd9c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 0x7efd9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd9c004d00 0x7efd9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00c94a0 con 0x7efd9c0039a0 2023-10-16T14:29:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c00ba60 con 0x7efd9c004d00 2023-10-16T14:29:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c00bc20 con 0x7efdb015ad70 2023-10-16T14:29:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdae59c640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efd9c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdadd9b640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 0x7efd9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdae59c640 1 -- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 msgr2=0x7efd9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdae59c640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb015ad70 0x7efd9c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdae59c640 1 -- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd9c004d00 msgr2=0x7efd9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.987+0000 7efdaed9d640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd9c004d00 0x7efd9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.988+0000 7efdae59c640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd9c004d00 0x7efd9c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.988+0000 7efdae59c640 1 -- 172.21.15.29:0/890455867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c003520 con 0x7efd9c0039a0 2023-10-16T14:29:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.988+0000 7efdaed9d640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd9c004d00 0x7efd9c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.989+0000 7efdae59c640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efd9c003dc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efda0009f90 tx=0x7efd9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2bc5391696efd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.989+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda002cbd0 con 0x7efd9c0039a0 2023-10-16T14:29:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.989+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda000f040 con 0x7efd9c0039a0 2023-10-16T14:29:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.989+0000 7efdad59a640 1 -- 172.21.15.29:0/890455867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0034440 con 0x7efd9c0039a0 2023-10-16T14:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.989+0000 7efdb4ba9640 1 -- 172.21.15.29:0/890455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 msgr2=0x7efd9c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.989+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/890455867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efd9c003dc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efda0009f90 tx=0x7efd9c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.990+0000 7efdaed9d640 1 -- 172.21.15.29:0/890455867 reap_dead start 2023-10-16T14:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.990+0000 7efdb4ba9640 1 -- 172.21.15.29:0/890455867 shutdown_connections 2023-10-16T14:29:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.990+0000 7efdb4ba9640 1 -- 172.21.15.29:0/890455867 >> 172.21.15.29:0/890455867 conn(0x7efdb00ce9b0 msgr2=0x7efdb00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.990+0000 7efdb4ba9640 1 -- 172.21.15.29:0/890455867 shutdown_connections 2023-10-16T14:29:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.990+0000 7efdb4ba9640 1 -- 172.21.15.29:0/890455867 wait complete. 2023-10-16T14:29:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.991+0000 7efdb4ba9640 1 Processor -- start 2023-10-16T14:29:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.992+0000 7efdb4ba9640 1 -- start start 2023-10-16T14:29:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdb4ba9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efdb00acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdb4ba9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb0158920 0x7efdb0158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdb4ba9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb00c6610 0x7efdb00c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdb4ba9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0158790 con 0x7efd9c0039a0 2023-10-16T14:29:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdb4ba9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01592e0 con 0x7efdb00c6610 2023-10-16T14:29:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdb4ba9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01f0610 con 0x7efdb0158920 2023-10-16T14:29:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efdb00acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efdb00acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60090/0 (socket says 172.21.15.29:60090) 2023-10-16T14:29:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 -- 172.21.15.29:0/407249204 learned_addr learned my addr 172.21.15.29:0/407249204 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 -- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb0158920 msgr2=0x7efdb0158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb0158920 0x7efdb0158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 -- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb00c6610 msgr2=0x7efdb00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb00c6610 0x7efdb00c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.993+0000 7efdae59c640 1 -- 172.21.15.29:0/407249204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb01f08f0 con 0x7efd9c0039a0 2023-10-16T14:29:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.994+0000 7efdae59c640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efdb00acc20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efda002c570 tx=0x7efda0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.994+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0034d60 con 0x7efd9c0039a0 2023-10-16T14:29:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.994+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda0040450 con 0x7efd9c0039a0 2023-10-16T14:29:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.994+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0040800 con 0x7efd9c0039a0 2023-10-16T14:29:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.994+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb01f0bd0 con 0x7efd9c0039a0 2023-10-16T14:29:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.994+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb01f1190 con 0x7efd9c0039a0 2023-10-16T14:29:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.995+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb0066a40 con 0x7efd9c0039a0 2023-10-16T14:29:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.996+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7efda0040960 con 0x7efd9c0039a0 2023-10-16T14:29:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.997+0000 7efda6ffd640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd9006b070 0x7efd9006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.997+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6743+0+0 (secure 0 0 0) 0x7efda00ac620 con 0x7efd9c0039a0 2023-10-16T14:29:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.998+0000 7efdadd9b640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd9006b070 0x7efd9006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:42.999+0000 7efdadd9b640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd9006b070 0x7efd9006d530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efdb0070af0 tx=0x7efd98009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.002+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efda00770c0 con 0x7efd9c0039a0 2023-10-16T14:29:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.247+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efdb01f1320 con 0x7efd9006b070 2023-10-16T14:29:43.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.388+0000 7efda6ffd640 1 -- 172.21.15.29:0/407249204 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7efdb01f1320 con 0x7efd9006b070 2023-10-16T14:29:43.391 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-16T14:29:43.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd9006b070 msgr2=0x7efd9006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:43.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd9006b070 0x7efd9006d530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efdb0070af0 tx=0x7efd98009040 comp rx=0 tx=0).stop 2023-10-16T14:29:43.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 msgr2=0x7efdb00acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:43.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efdb00acc20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efda002c570 tx=0x7efda0009750 comp rx=0 tx=0).stop 2023-10-16T14:29:43.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 shutdown_connections 2023-10-16T14:29:43.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7efd9006b070 0x7efd9006d530 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdb00c6610 0x7efdb00c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdb0158920 0x7efdb0158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 --2- 172.21.15.29:0/407249204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd9c0039a0 0x7efdb00acc20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:43.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 >> 172.21.15.29:0/407249204 conn(0x7efdb00ce9b0 msgr2=0x7efdb015d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:43.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 shutdown_connections 2023-10-16T14:29:43.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:43.391+0000 7efdb4ba9640 1 -- 172.21.15.29:0/407249204 wait complete. 2023-10-16T14:29:43.413 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-16T14:29:43.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.414 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T14:29:43.462 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:43.463 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:43.463 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:43.463 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:43.463 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:43.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:43.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:43.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:43.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:43.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:43.465 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:43.465 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:43.467 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_18_00_UTC 2023-10-16T14:29:43.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.468 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_18_00_UTC) 2023-10-16T14:29:43.583 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:43.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.584 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:43.637 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:43.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.638 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:43.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:43.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.693 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:43.753 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_22_01_UTC 2023-10-16T14:29:43.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.754 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_22_01_UTC) 2023-10-16T14:29:43.817 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:43.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.818 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_23_01_UTC) 2023-10-16T14:29:43.888 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:43.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.889 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_24_01_UTC) 2023-10-16T14:29:43.956 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:43.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:43.956 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_25_01_UTC) 2023-10-16T14:29:44.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:44.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:44.030 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_26_00_UTC) 2023-10-16T14:29:44.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:44.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:44.102 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_27_00_UTC) 2023-10-16T14:29:44.208 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:44.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:44.208 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_28_00_UTC) 2023-10-16T14:29:44.260 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:44.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:44.261 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-14_29_00_UTC) 2023-10-16T14:29:44.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:44.319 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-16T14:29:44.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-10-16T14:29:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.630+0000 7f86da0df640 1 Processor -- start 2023-10-16T14:29:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.630+0000 7f86da0df640 1 -- start start 2023-10-16T14:29:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.632+0000 7f86da0df640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86d406fa20 0x7f86d40ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.632+0000 7f86da0df640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d40acc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.632+0000 7f86da0df640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d406d900 con 0x7f86d406d520 2023-10-16T14:29:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.632+0000 7f86da0df640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d415b1d0 con 0x7f86d406fa20 2023-10-16T14:29:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.632+0000 7f86da0df640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d40b24f0 con 0x7f86d4065cc0 2023-10-16T14:29:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d90dd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f86d406d520 0x7f86d40b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60522/0 (socket says 172.21.15.29:60522) 2023-10-16T14:29:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d90dd640 1 -- 172.21.15.29:0/2563026544 learned_addr learned my addr 172.21.15.29:0/2563026544 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d88dc640 1 --2- 172.21.15.29:0/2563026544 >> v2:172.21.15.29:3300/0 conn(0x7f86d406fa20 0x7f86d40ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d98de640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d40acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86cbfff640 1 -- 172.21.15.29:0/2563026544 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048250760 0 0) 0x7f86d406d900 con 0x7f86d406d520 2023-10-16T14:29:44.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86cbfff640 1 -- 172.21.15.29:0/2563026544 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0007930 con 0x7f86d406d520 2023-10-16T14:29:44.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d88dc640 1 -- 172.21.15.29:0/2563026544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 msgr2=0x7f86d40acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:44.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d88dc640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d40acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:44.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d88dc640 1 -- 172.21.15.29:0/2563026544 >> v1:172.21.15.29:6789/0 conn(0x7f86d406d520 legacy=0x7f86d40b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:44.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d88dc640 1 -- 172.21.15.29:0/2563026544 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d406feb0 con 0x7f86d406fa20 2023-10-16T14:29:44.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.633+0000 7f86d98de640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d40acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:44.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.635+0000 7f86d88dc640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d406fa20 0x7f86d40ac750 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f86c40097a0 tx=0x7f86c402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49545f7ddf42663d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:44.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86cbfff640 1 -- 172.21.15.29:0/2563026544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c402f050 con 0x7f86d406fa20 2023-10-16T14:29:44.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86cbfff640 1 -- 172.21.15.29:0/2563026544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c4031700 con 0x7f86d406fa20 2023-10-16T14:29:44.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86cbfff640 1 -- 172.21.15.29:0/2563026544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c4031a70 con 0x7f86d406fa20 2023-10-16T14:29:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 -- 172.21.15.29:0/2563026544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d406fa20 msgr2=0x7f86d40ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d406fa20 0x7f86d40ac750 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f86c40097a0 tx=0x7f86c402b7a0 comp rx=0 tx=0).stop 2023-10-16T14:29:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 -- 172.21.15.29:0/2563026544 shutdown_connections 2023-10-16T14:29:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d40acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 --2- 172.21.15.29:0/2563026544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d406fa20 0x7f86d40ac750 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 -- 172.21.15.29:0/2563026544 >> 172.21.15.29:0/2563026544 conn(0x7f86d40b0a80 msgr2=0x7f86d40b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 -- 172.21.15.29:0/2563026544 shutdown_connections 2023-10-16T14:29:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.636+0000 7f86da0df640 1 -- 172.21.15.29:0/2563026544 wait complete. 2023-10-16T14:29:44.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.638+0000 7f86da0df640 1 Processor -- start 2023-10-16T14:29:44.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.638+0000 7f86da0df640 1 -- start start 2023-10-16T14:29:44.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.639+0000 7f86da0df640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d41470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.639+0000 7f86da0df640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d406d520 0x7f86d41edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.639+0000 7f86da0df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 0x7f86d40ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.639+0000 7f86da0df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4147620 con 0x7f86d415bbe0 2023-10-16T14:29:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.639+0000 7f86da0df640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4144970 con 0x7f86d406d520 2023-10-16T14:29:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.639+0000 7f86da0df640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4144c50 con 0x7f86d4065cc0 2023-10-16T14:29:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 0x7f86d40ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:44.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 0x7f86d40ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53194/0 (socket says 172.21.15.29:53194) 2023-10-16T14:29:44.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 -- 172.21.15.29:0/2806173885 learned_addr learned my addr 172.21.15.29:0/2806173885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:44.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d90dd640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d41470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:44.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d88dc640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d406d520 0x7f86d41edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:44.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 -- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 msgr2=0x7f86d41470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:44.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d41470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:44.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 -- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d406d520 msgr2=0x7f86d41edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:44.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d406d520 0x7f86d41edb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:44.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.640+0000 7f86d98de640 1 -- 172.21.15.29:0/2806173885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4009400 con 0x7f86d415bbe0 2023-10-16T14:29:44.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.641+0000 7f86d98de640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 0x7f86d40ae8f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f86cc007df0 tx=0x7f86cc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:44.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.641+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc003990 con 0x7f86d415bbe0 2023-10-16T14:29:44.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.641+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc00f040 con 0x7f86d415bbe0 2023-10-16T14:29:44.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.641+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc003d30 con 0x7f86d415bbe0 2023-10-16T14:29:44.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.641+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d4144f30 con 0x7f86d415bbe0 2023-10-16T14:29:44.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.641+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d41454f0 con 0x7f86d415bbe0 2023-10-16T14:29:44.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.643+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d41456e0 con 0x7f86d415bbe0 2023-10-16T14:29:44.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.643+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f86cc004030 con 0x7f86d415bbe0 2023-10-16T14:29:44.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.645+0000 7f86c9ffb640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f86b806b070 0x7f86b806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:44.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.645+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f86cc08a200 con 0x7f86d415bbe0 2023-10-16T14:29:44.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.646+0000 7f86d90dd640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f86b806b070 0x7f86b806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:44.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.647+0000 7f86d90dd640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f86b806b070 0x7f86b806d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f86d0005880 tx=0x7f86d0009040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:44.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.649+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86cc054ae0 con 0x7f86d415bbe0 2023-10-16T14:29:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:44.873+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f86d4070470 con 0x7f86b806b070 2023-10-16T14:29:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.080+0000 7f86c9ffb640 1 -- 172.21.15.29:0/2806173885 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f86d00013f0 con 0x7f86b806b070 2023-10-16T14:29:45.083 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-16T14:29:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f86b806b070 msgr2=0x7f86b806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f86b806b070 0x7f86b806d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f86d0005880 tx=0x7f86d0009040 comp rx=0 tx=0).stop 2023-10-16T14:29:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 msgr2=0x7f86d40ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 0x7f86d40ae8f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f86cc007df0 tx=0x7f86cc00d4a0 comp rx=0 tx=0).stop 2023-10-16T14:29:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 shutdown_connections 2023-10-16T14:29:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f86b806b070 0x7f86b806d530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86d415bbe0 0x7f86d40ae8f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86d406d520 0x7f86d41edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 --2- 172.21.15.29:0/2806173885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86d4065cc0 0x7f86d41470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 >> 172.21.15.29:0/2806173885 conn(0x7f86d40b0a80 msgr2=0x7f86d40c9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.084+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 shutdown_connections 2023-10-16T14:29:45.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:45.085+0000 7f86da0df640 1 -- 172.21.15.29:0/2806173885 wait complete. 2023-10-16T14:29:45.115 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-16T14:29:45.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.116 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T14:29:45.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:45.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:45.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:45.173 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:45.173 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:45.173 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:45.173 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:45.173 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:45.174 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:45.174 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:45.174 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:45.175 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:45.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.176 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:45.237 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:45.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.238 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:45.300 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:45.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.301 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:45.380 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:45.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.380 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:45.428 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:45.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.428 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_23_00_UTC) 2023-10-16T14:29:45.476 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:45.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.477 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_24_00_UTC) 2023-10-16T14:29:45.526 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:45.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.527 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_25_01_UTC) 2023-10-16T14:29:45.592 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:45.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.593 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_26_01_UTC) 2023-10-16T14:29:45.660 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:45.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.660 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_27_01_UTC) 2023-10-16T14:29:45.726 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:45.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.727 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_28_00_UTC) 2023-10-16T14:29:45.793 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:45.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.794 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-14_29_00_UTC) 2023-10-16T14:29:45.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:45.872 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-16T14:29:45.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-10-16T14:29:46.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.176+0000 7f2216135640 1 Processor -- start 2023-10-16T14:29:46.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.176+0000 7f2216135640 1 -- start start 2023-10-16T14:29:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2216135640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f221015cd70 0x7f22100d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2216135640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22100b8480 0x7f22100cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2216135640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f22100d0be0 0x7f22100cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2216135640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f221015d150 con 0x7f22100b8480 2023-10-16T14:29:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2216135640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22100d1180 con 0x7f22100d0be0 2023-10-16T14:29:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2216135640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22100d1460 con 0x7f221015cd70 2023-10-16T14:29:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2214932640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22100b8480 0x7f22100cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2214932640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22100b8480 0x7f22100cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53198/0 (socket says 172.21.15.29:53198) 2023-10-16T14:29:46.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.178+0000 7f2214932640 1 -- 172.21.15.29:0/3716415702 learned_addr learned my addr 172.21.15.29:0/3716415702 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:46.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2215934640 1 --2- 172.21.15.29:0/3716415702 >> v2:172.21.15.170:3300/0 conn(0x7f22100d0be0 0x7f22100cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2215133640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f221015cd70 0x7f22100d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2214932640 1 -- 172.21.15.29:0/3716415702 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f221015cd70 msgr2=0x7f22100d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2214932640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f221015cd70 0x7f22100d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2214932640 1 -- 172.21.15.29:0/3716415702 >> v2:172.21.15.170:3300/0 conn(0x7f22100d0be0 msgr2=0x7f22100cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2214932640 1 --2- 172.21.15.29:0/3716415702 >> v2:172.21.15.170:3300/0 conn(0x7f22100d0be0 0x7f22100cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2214932640 1 -- 172.21.15.29:0/3716415702 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22100d1740 con 0x7f22100b8480 2023-10-16T14:29:46.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2215934640 1 --2- 172.21.15.29:0/3716415702 >> v2:172.21.15.170:3300/0 conn(0x7f22100d0be0 0x7f22100cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:46.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.179+0000 7f2215133640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f221015cd70 0x7f22100d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:46.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.180+0000 7f2214932640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22100b8480 0x7f22100cd7c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f22100b2520 tx=0x7f220002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c459adde25e83611 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:46.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.180+0000 7f2207fff640 1 -- 172.21.15.29:0/3716415702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200031870 con 0x7f22100b8480 2023-10-16T14:29:46.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.180+0000 7f2207fff640 1 -- 172.21.15.29:0/3716415702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22000319d0 con 0x7f22100b8480 2023-10-16T14:29:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.180+0000 7f2207fff640 1 -- 172.21.15.29:0/3716415702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200031d60 con 0x7f22100b8480 2023-10-16T14:29:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 -- 172.21.15.29:0/3716415702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22100b8480 msgr2=0x7f22100cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22100b8480 0x7f22100cd7c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f22100b2520 tx=0x7f220002b910 comp rx=0 tx=0).stop 2023-10-16T14:29:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 -- 172.21.15.29:0/3716415702 shutdown_connections 2023-10-16T14:29:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 --2- 172.21.15.29:0/3716415702 >> v2:172.21.15.170:3300/0 conn(0x7f22100d0be0 0x7f22100cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22100b8480 0x7f22100cd7c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 --2- 172.21.15.29:0/3716415702 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f221015cd70 0x7f22100d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 -- 172.21.15.29:0/3716415702 >> 172.21.15.29:0/3716415702 conn(0x7f22100cf570 msgr2=0x7f22100cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 -- 172.21.15.29:0/3716415702 shutdown_connections 2023-10-16T14:29:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.181+0000 7f2216135640 1 -- 172.21.15.29:0/3716415702 wait complete. 2023-10-16T14:29:46.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.183+0000 7f2216135640 1 Processor -- start 2023-10-16T14:29:46.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.184+0000 7f2216135640 1 -- start start 2023-10-16T14:29:46.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2216135640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22100b8480 0x7f2210152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2216135640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22100d0be0 0x7f22100c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2216135640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 0x7f2210152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2216135640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22101536b0 con 0x7f221015cd70 2023-10-16T14:29:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2216135640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2210152550 con 0x7f22100b8480 2023-10-16T14:29:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2216135640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22100c7cf0 con 0x7f22100d0be0 2023-10-16T14:29:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2215934640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 0x7f2210152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2215934640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 0x7f2210152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53212/0 (socket says 172.21.15.29:53212) 2023-10-16T14:29:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.185+0000 7f2215934640 1 -- 172.21.15.29:0/1832075698 learned_addr learned my addr 172.21.15.29:0/1832075698 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2214932640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22100d0be0 0x7f22100c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215133640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22100b8480 0x7f2210152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215934640 1 -- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22100d0be0 msgr2=0x7f22100c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215934640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22100d0be0 0x7f22100c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215934640 1 -- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22100b8480 msgr2=0x7f2210152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:46.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215934640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22100b8480 0x7f2210152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215934640 1 -- 172.21.15.29:0/1832075698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000094d0 con 0x7f221015cd70 2023-10-16T14:29:46.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2214932640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22100d0be0 0x7f22100c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:46.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215133640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22100b8480 0x7f2210152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:46.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2215934640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 0x7f2210152a80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f220800ce40 tx=0x7f220800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800d840 con 0x7f221015cd70 2023-10-16T14:29:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220800d9a0 con 0x7f221015cd70 2023-10-16T14:29:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800dd50 con 0x7f221015cd70 2023-10-16T14:29:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.186+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22100c7fd0 con 0x7f221015cd70 2023-10-16T14:29:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.187+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22100c8590 con 0x7f221015cd70 2023-10-16T14:29:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.188+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22100b2ce0 con 0x7f221015cd70 2023-10-16T14:29:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.189+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f220801e3b0 con 0x7f221015cd70 2023-10-16T14:29:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.190+0000 7f2205ffb640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f21f406b070 0x7f21f406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.190+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f220808a750 con 0x7f221015cd70 2023-10-16T14:29:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.190+0000 7f2215133640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f21f406b070 0x7f21f406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.192+0000 7f2215133640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f21f406b070 0x7f21f406d530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f220c006fd0 tx=0x7f220c008040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:46.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.194+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f22080551b0 con 0x7f221015cd70 2023-10-16T14:29:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.440+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f22100d1d00 con 0x7f21f406b070 2023-10-16T14:29:46.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.628+0000 7f2205ffb640 1 -- 172.21.15.29:0/1832075698 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f220c00c040 con 0x7f21f406b070 2023-10-16T14:29:46.631 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-16T14:29:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.631+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f21f406b070 msgr2=0x7f21f406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:46.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.631+0000 7f2216135640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f21f406b070 0x7f21f406d530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f220c006fd0 tx=0x7f220c008040 comp rx=0 tx=0).stop 2023-10-16T14:29:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.631+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 msgr2=0x7f2210152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.631+0000 7f2216135640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 0x7f2210152a80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f220800ce40 tx=0x7f220800a710 comp rx=0 tx=0).stop 2023-10-16T14:29:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 shutdown_connections 2023-10-16T14:29:46.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f21f406b070 0x7f21f406d530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f221015cd70 0x7f2210152a80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22100d0be0 0x7f22100c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 --2- 172.21.15.29:0/1832075698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22100b8480 0x7f2210152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 >> 172.21.15.29:0/1832075698 conn(0x7f22100cf570 msgr2=0x7f22100b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 shutdown_connections 2023-10-16T14:29:46.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:46.632+0000 7f2216135640 1 -- 172.21.15.29:0/1832075698 wait complete. 2023-10-16T14:29:46.657 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-16T14:29:46.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:46.658 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T14:29:46.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:29:46.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:29:46.710 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:46.710 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:46.711 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:46.711 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:46.711 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:46.711 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:46.711 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:46.712 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:46.712 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:46.712 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:46.712 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:46.713 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_19_01_UTC 2023-10-16T14:29:46.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:46.714 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_19_01_UTC) 2023-10-16T14:29:46.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_20_01_UTC 2023-10-16T14:29:46.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:46.833 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_20_01_UTC) 2023-10-16T14:29:46.914 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:46.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:46.915 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:46.994 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:46.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:46.995 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:47.068 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:47.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.069 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_23_00_UTC) 2023-10-16T14:29:47.174 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:47.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.175 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_24_00_UTC) 2023-10-16T14:29:47.245 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:47.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.246 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_25_00_UTC) 2023-10-16T14:29:47.301 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:47.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.302 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_26_00_UTC) 2023-10-16T14:29:47.360 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_27_00_UTC 2023-10-16T14:29:47.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.361 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_27_00_UTC) 2023-10-16T14:29:47.450 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:47.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.451 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_28_01_UTC) 2023-10-16T14:29:47.503 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:47.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.504 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-14_29_00_UTC) 2023-10-16T14:29:47.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:47.558 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-16T14:29:47.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-10-16T14:29:47.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.799+0000 7fd200940640 1 Processor -- start 2023-10-16T14:29:47.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.800+0000 7fd200940640 1 -- start start 2023-10-16T14:29:47.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.801+0000 7fd200940640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc0ad7a0 0x7fd1fc070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:47.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.801+0000 7fd200940640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0ae1a0 con 0x7fd1fc0adc00 2023-10-16T14:29:47.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.801+0000 7fd200940640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0ae480 con 0x7fd1fc160c80 2023-10-16T14:29:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.801+0000 7fd200940640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc0ae760 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.802+0000 7fd1fb59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1fc0adc00 0x7fd1fc140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60534/0 (socket says 172.21.15.29:60534) 2023-10-16T14:29:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.802+0000 7fd1fb59e640 1 -- 172.21.15.29:0/1160091170 learned_addr learned my addr 172.21.15.29:0/1160091170 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.802+0000 7fd1fa59c640 1 --2- 172.21.15.29:0/1160091170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc0ad7a0 0x7fd1fc070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.802+0000 7fd1f9d9b640 1 -- 172.21.15.29:0/1160091170 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145626601 0 0) 0x7fd1fc0ae1a0 con 0x7fd1fc0adc00 2023-10-16T14:29:47.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.802+0000 7fd1f9d9b640 1 -- 172.21.15.29:0/1160091170 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e4003520 con 0x7fd1fc0adc00 2023-10-16T14:29:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.803+0000 7fd1f9d9b640 1 -- 172.21.15.29:0/1160091170 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132037003 0 0) 0x7fd1e4003520 con 0x7fd1fc0adc00 2023-10-16T14:29:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.803+0000 7fd1fa59c640 1 -- 172.21.15.29:0/1160091170 >> v1:172.21.15.170:6789/0 conn(0x7fd1fc160c80 legacy=0x7fd1fc161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.803+0000 7fd1fa59c640 1 -- 172.21.15.29:0/1160091170 >> v1:172.21.15.29:6789/0 conn(0x7fd1fc0adc00 legacy=0x7fd1fc140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.803+0000 7fd1fa59c640 1 -- 172.21.15.29:0/1160091170 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc0aea40 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.804+0000 7fd1fa59c640 1 --2- 172.21.15.29:0/1160091170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc0ad7a0 0x7fd1fc070240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1f00097d0 tx=0x7fd1f002b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7bb7bbbec86b82e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd1f9d9b640 1 -- 172.21.15.29:0/1160091170 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40019f0 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd1f9d9b640 1 -- 172.21.15.29:0/1160091170 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f002e050 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd1f9d9b640 1 -- 172.21.15.29:0/1160091170 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0030730 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd200940640 1 -- 172.21.15.29:0/1160091170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc0ad7a0 msgr2=0x7fd1fc070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd200940640 1 --2- 172.21.15.29:0/1160091170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc0ad7a0 0x7fd1fc070240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1f00097d0 tx=0x7fd1f002b660 comp rx=0 tx=0).stop 2023-10-16T14:29:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd200940640 1 -- 172.21.15.29:0/1160091170 shutdown_connections 2023-10-16T14:29:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.805+0000 7fd200940640 1 --2- 172.21.15.29:0/1160091170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc0ad7a0 0x7fd1fc070240 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.806+0000 7fd200940640 1 -- 172.21.15.29:0/1160091170 >> 172.21.15.29:0/1160091170 conn(0x7fd1fc0cc230 msgr2=0x7fd1fc0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.806+0000 7fd200940640 1 -- 172.21.15.29:0/1160091170 shutdown_connections 2023-10-16T14:29:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.806+0000 7fd200940640 1 -- 172.21.15.29:0/1160091170 wait complete. 2023-10-16T14:29:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.808+0000 7fd200940640 1 Processor -- start 2023-10-16T14:29:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.808+0000 7fd200940640 1 -- start start 2023-10-16T14:29:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd200940640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 0x7fd1fc1f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd200940640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1fc0adc00 0x7fd1fc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd200940640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc160c80 0x7fd1fc0b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd200940640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc1e5d70 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd200940640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc1e5fc0 con 0x7fd1fc0adc00 2023-10-16T14:29:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd200940640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc1e62a0 con 0x7fd1fc160c80 2023-10-16T14:29:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.809+0000 7fd1fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 0x7fd1fc1f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 0x7fd1fc1f1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53224/0 (socket says 172.21.15.29:53224) 2023-10-16T14:29:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 -- 172.21.15.29:0/3757733755 learned_addr learned my addr 172.21.15.29:0/3757733755 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fb59e640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc160c80 0x7fd1fc0b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fa59c640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1fc0adc00 0x7fd1fc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 -- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc160c80 msgr2=0x7fd1fc0b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc160c80 0x7fd1fc0b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 -- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1fc0adc00 msgr2=0x7fd1fc0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1fc0adc00 0x7fd1fc0b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:47.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 -- 172.21.15.29:0/3757733755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f0009400 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.810+0000 7fd1fad9d640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 0x7fd1fc1f1cb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00c550 tx=0x7fd1ec00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.811+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec005480 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.811+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ec0035f0 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.811+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec0039a0 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.811+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1fc1e6580 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.811+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1fc1f21f0 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.813+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1fc1e6aa0 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.813+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd1ec003b00 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.815+0000 7fd1e37fe640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fd1e806b070 0x7fd1e806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.815+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6695+0+0 (secure 0 0 0) 0x7fd1ec089480 con 0x7fd1fc0ad7a0 2023-10-16T14:29:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.816+0000 7fd1fa59c640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fd1e806b070 0x7fd1e806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.817+0000 7fd1fa59c640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fd1e806b070 0x7fd1e806d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0034000 tx=0x7fd1f0008030 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:47.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:47.819+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd1fc1e6aa0 con 0x7fd1fc0ad7a0 2023-10-16T14:29:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.040+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1fc1f23e0 con 0x7fd1e806b070 2023-10-16T14:29:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.219+0000 7fd1e37fe640 1 -- 172.21.15.29:0/3757733755 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fd1fc1f23e0 con 0x7fd1e806b070 2023-10-16T14:29:48.222 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-16T14:29:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fd1e806b070 msgr2=0x7fd1e806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fd1e806b070 0x7fd1e806d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0034000 tx=0x7fd1f0008030 comp rx=0 tx=0).stop 2023-10-16T14:29:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 msgr2=0x7fd1fc1f1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 0x7fd1fc1f1cb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00c550 tx=0x7fd1ec00ca10 comp rx=0 tx=0).stop 2023-10-16T14:29:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 shutdown_connections 2023-10-16T14:29:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fd1e806b070 0x7fd1e806d530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1fc160c80 0x7fd1fc0b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1fc0adc00 0x7fd1fc0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 --2- 172.21.15.29:0/3757733755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1fc0ad7a0 0x7fd1fc1f1cb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 >> 172.21.15.29:0/3757733755 conn(0x7fd1fc0cc230 msgr2=0x7fd1fc15d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 shutdown_connections 2023-10-16T14:29:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:48.222+0000 7fd200940640 1 -- 172.21.15.29:0/3757733755 wait complete. 2023-10-16T14:29:48.245 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-16T14:29:48.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.246 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T14:29:48.293 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:48.295 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:48.295 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:48.295 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:48.295 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:48.295 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:48.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:48.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.297 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:48.394 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:48.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.398 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:48.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:48.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.464 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:48.517 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:48.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.518 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:48.576 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_23_00_UTC 2023-10-16T14:29:48.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.577 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_23_00_UTC) 2023-10-16T14:29:48.638 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_24_00_UTC 2023-10-16T14:29:48.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.639 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_24_00_UTC) 2023-10-16T14:29:48.708 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_25_01_UTC 2023-10-16T14:29:48.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.709 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_25_01_UTC) 2023-10-16T14:29:48.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_26_01_UTC 2023-10-16T14:29:48.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.784 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_26_01_UTC) 2023-10-16T14:29:48.864 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:48.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.865 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_27_01_UTC) 2023-10-16T14:29:48.951 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_28_01_UTC 2023-10-16T14:29:48.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:48.952 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_28_01_UTC) 2023-10-16T14:29:49.026 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:49.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:49.027 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-14_29_00_UTC) 2023-10-16T14:29:49.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:49.106 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-16T14:29:49.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-10-16T14:29:49.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.403+0000 7f8528eb4640 1 Processor -- start 2023-10-16T14:29:49.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.403+0000 7f8528eb4640 1 -- start start 2023-10-16T14:29:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f8528eb4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85240ad970 0x7f85240cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f8528eb4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f85240ccfb0 0x7f85240cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f8528eb4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852415b9d0 con 0x7f852406c040 2023-10-16T14:29:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f8528eb4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852415bcb0 con 0x7f85240ad970 2023-10-16T14:29:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f8528eb4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852415bf90 con 0x7f85240ccfb0 2023-10-16T14:29:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f851bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85240ad970 0x7f85240cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f85237fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f852406c040 0x7f852406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60546/0 (socket says 172.21.15.29:60546) 2023-10-16T14:29:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.404+0000 7f85237fe640 1 -- 172.21.15.29:0/3268796178 learned_addr learned my addr 172.21.15.29:0/3268796178 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:49.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.405+0000 7f8523fff640 1 --2- 172.21.15.29:0/3268796178 >> v2:172.21.15.170:3300/0 conn(0x7f85240ccfb0 0x7f85240cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:49.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.405+0000 7f851bfff640 1 -- 172.21.15.29:0/3268796178 >> v2:172.21.15.170:3300/0 conn(0x7f85240ccfb0 msgr2=0x7f85240cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.405+0000 7f851bfff640 1 --2- 172.21.15.29:0/3268796178 >> v2:172.21.15.170:3300/0 conn(0x7f85240ccfb0 0x7f85240cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.405+0000 7f851bfff640 1 -- 172.21.15.29:0/3268796178 >> v1:172.21.15.29:6789/0 conn(0x7f852406c040 legacy=0x7f852406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.405+0000 7f851bfff640 1 -- 172.21.15.29:0/3268796178 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240cc270 con 0x7f85240ad970 2023-10-16T14:29:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.405+0000 7f8522ffd640 1 -- 172.21.15.29:0/3268796178 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090031913 0 0) 0x7f852415b9d0 con 0x7f852406c040 2023-10-16T14:29:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.406+0000 7f851bfff640 1 --2- 172.21.15.29:0/3268796178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ad970 0x7f85240cca70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f850c0097a0 tx=0x7f850c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0abb942f194ef2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:49.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.406+0000 7f8522ffd640 1 -- 172.21.15.29:0/3268796178 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c031830 con 0x7f85240ad970 2023-10-16T14:29:49.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.406+0000 7f8522ffd640 1 -- 172.21.15.29:0/3268796178 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c031990 con 0x7f85240ad970 2023-10-16T14:29:49.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.406+0000 7f8522ffd640 1 -- 172.21.15.29:0/3268796178 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c031d20 con 0x7f85240ad970 2023-10-16T14:29:49.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 -- 172.21.15.29:0/3268796178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ad970 msgr2=0x7f85240cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:49.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 --2- 172.21.15.29:0/3268796178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ad970 0x7f85240cca70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f850c0097a0 tx=0x7f850c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 -- 172.21.15.29:0/3268796178 shutdown_connections 2023-10-16T14:29:49.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 --2- 172.21.15.29:0/3268796178 >> v2:172.21.15.170:3300/0 conn(0x7f85240ccfb0 0x7f85240cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 --2- 172.21.15.29:0/3268796178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ad970 0x7f85240cca70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 -- 172.21.15.29:0/3268796178 >> 172.21.15.29:0/3268796178 conn(0x7f85240ce6a0 msgr2=0x7f85240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:49.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 -- 172.21.15.29:0/3268796178 shutdown_connections 2023-10-16T14:29:49.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.407+0000 7f8528eb4640 1 -- 172.21.15.29:0/3268796178 wait complete. 2023-10-16T14:29:49.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.409+0000 7f8528eb4640 1 Processor -- start 2023-10-16T14:29:49.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.409+0000 7f8528eb4640 1 -- start start 2023-10-16T14:29:49.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.410+0000 7f8528eb4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f852406c040 0x7f852414ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:49.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.410+0000 7f8528eb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85240ad970 0x7f852415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:49.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.410+0000 7f8528eb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 0x7f85240c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:49.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.410+0000 7f8528eb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85240aee80 con 0x7f85240ccfb0 2023-10-16T14:29:49.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.410+0000 7f8528eb4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852415d0d0 con 0x7f852406c040 2023-10-16T14:29:49.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.410+0000 7f8528eb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852415d980 con 0x7f85240ad970 2023-10-16T14:29:49.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 0x7f85240c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:49.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 0x7f85240c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53246/0 (socket says 172.21.15.29:53246) 2023-10-16T14:29:49.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 -- 172.21.15.29:0/2625815458 learned_addr learned my addr 172.21.15.29:0/2625815458 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:49.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f85237fe640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f852406c040 0x7f852414ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:49.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f851bfff640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85240ad970 0x7f852415cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:49.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 -- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85240ad970 msgr2=0x7f852415cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:49.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85240ad970 0x7f852415cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 -- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f852406c040 msgr2=0x7f852414ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:49.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f852406c040 0x7f852414ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 -- 172.21.15.29:0/2625815458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009400 con 0x7f85240ccfb0 2023-10-16T14:29:49.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f851bfff640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85240ad970 0x7f852415cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:49.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.411+0000 7f8523fff640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 0x7f85240c9d40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f851c007e60 tx=0x7f851c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:49.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.412+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c014420 con 0x7f85240ccfb0 2023-10-16T14:29:49.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.412+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c0145a0 con 0x7f85240ccfb0 2023-10-16T14:29:49.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.412+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c014950 con 0x7f85240ccfb0 2023-10-16T14:29:49.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.412+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85240ae270 con 0x7f85240ccfb0 2023-10-16T14:29:49.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.412+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85240ae830 con 0x7f85240ccfb0 2023-10-16T14:29:49.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.413+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852406b330 con 0x7f85240ccfb0 2023-10-16T14:29:49.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.414+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f851c014ab0 con 0x7f85240ccfb0 2023-10-16T14:29:49.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.415+0000 7f8520ff9640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f850406b070 0x7f850406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:49.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.415+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f851c08a6d0 con 0x7f85240ccfb0 2023-10-16T14:29:49.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.416+0000 7f85237fe640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f850406b070 0x7f850406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:49.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.417+0000 7f85237fe640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f850406b070 0x7f850406d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8514000f60 tx=0x7f8514007b50 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:49.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.419+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f851c055100 con 0x7f85240ccfb0 2023-10-16T14:29:49.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.645+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 --> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f85240b1cd0 con 0x7f850406b070 2023-10-16T14:29:49.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.855+0000 7f8520ff9640 1 -- 172.21.15.29:0/2625815458 <== mgr.4861 v2:172.21.15.170:6832/1418556441 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f85240b1cd0 con 0x7f850406b070 2023-10-16T14:29:49.858 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-16T14:29:49.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.859+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f850406b070 msgr2=0x7f850406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:49.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.859+0000 7f8528eb4640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f850406b070 0x7f850406d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8514000f60 tx=0x7f8514007b50 comp rx=0 tx=0).stop 2023-10-16T14:29:49.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.859+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 msgr2=0x7f85240c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:49.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.859+0000 7f8528eb4640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 0x7f85240c9d40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f851c007e60 tx=0x7f851c00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 shutdown_connections 2023-10-16T14:29:49.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f850406b070 0x7f850406d530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85240ccfb0 0x7f85240c9d40 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85240ad970 0x7f852415cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 --2- 172.21.15.29:0/2625815458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f852406c040 0x7f852414ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:49.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 >> 172.21.15.29:0/2625815458 conn(0x7f85240ce6a0 msgr2=0x7f852406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:49.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 shutdown_connections 2023-10-16T14:29:49.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:49.860+0000 7f8528eb4640 1 -- 172.21.15.29:0/2625815458 wait complete. 2023-10-16T14:29:49.884 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-16T14:29:49.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:49.885 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T14:29:49.943 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:49.944 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:49.944 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:49.944 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:49.944 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:49.945 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:49.945 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:49.945 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:49.945 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:49.945 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:49.946 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:49.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_19_00_UTC 2023-10-16T14:29:49.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:49.948 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_19_00_UTC) 2023-10-16T14:29:50.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_20_00_UTC 2023-10-16T14:29:50.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.030 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_20_00_UTC) 2023-10-16T14:29:50.142 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_21_00_UTC 2023-10-16T14:29:50.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.143 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_21_00_UTC) 2023-10-16T14:29:50.227 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_22_00_UTC 2023-10-16T14:29:50.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.228 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_22_00_UTC) 2023-10-16T14:29:50.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_23_01_UTC 2023-10-16T14:29:50.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.284 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_23_01_UTC) 2023-10-16T14:29:50.335 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_24_01_UTC 2023-10-16T14:29:50.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.336 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_24_01_UTC) 2023-10-16T14:29:50.400 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_25_00_UTC 2023-10-16T14:29:50.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.401 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_25_00_UTC) 2023-10-16T14:29:50.462 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_26_00_UTC 2023-10-16T14:29:50.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.463 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_26_00_UTC) 2023-10-16T14:29:50.520 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_27_01_UTC 2023-10-16T14:29:50.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.521 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_27_01_UTC) 2023-10-16T14:29:50.576 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_28_00_UTC 2023-10-16T14:29:50.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.577 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_28_00_UTC) 2023-10-16T14:29:50.640 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_29_00_UTC 2023-10-16T14:29:50.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.642 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-14_29_00_UTC) 2023-10-16T14:29:50.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:50.712 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-16T14:29:50.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:29:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.003+0000 7fdee6f54640 1 Processor -- start 2023-10-16T14:29:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.004+0000 7fdee6f54640 1 -- start start 2023-10-16T14:29:51.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.005+0000 7fdee6f54640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:51.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.005+0000 7fdee6f54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 0x7fdee015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:51.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.005+0000 7fdee6f54640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee006c960 con 0x7fdee00ad970 2023-10-16T14:29:51.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.005+0000 7fdee6f54640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee015b9d0 con 0x7fdee006c040 2023-10-16T14:29:51.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.005+0000 7fdee6f54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee015bcb0 con 0x7fdee00cda80 2023-10-16T14:29:51.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:51.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53248/0 (socket says 172.21.15.29:53248) 2023-10-16T14:29:51.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 -- 172.21.15.29:0/2391570797 learned_addr learned my addr 172.21.15.29:0/2391570797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 -- 172.21.15.29:0/2391570797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 msgr2=0x7fdee015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 --2- 172.21.15.29:0/2391570797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 0x7fdee015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 -- 172.21.15.29:0/2391570797 >> v1:172.21.15.170:6789/0 conn(0x7fdee00ad970 legacy=0x7fdee015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.006+0000 7fdee5f52640 1 -- 172.21.15.29:0/2391570797 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee015bf90 con 0x7fdee006c040 2023-10-16T14:29:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.007+0000 7fdee5f52640 1 --2- 172.21.15.29:0/2391570797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee006c040 0x7fdee006c420 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009870 tx=0x7fdedc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e260d14afb1e7a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee4f50640 1 -- 172.21.15.29:0/2391570797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc031640 con 0x7fdee006c040 2023-10-16T14:29:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee4f50640 1 -- 172.21.15.29:0/2391570797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc0317a0 con 0x7fdee006c040 2023-10-16T14:29:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee4f50640 1 -- 172.21.15.29:0/2391570797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc031b30 con 0x7fdee006c040 2023-10-16T14:29:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 -- 172.21.15.29:0/2391570797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee006c040 msgr2=0x7fdee006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2391570797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee006c040 0x7fdee006c420 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009870 tx=0x7fdedc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:29:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 -- 172.21.15.29:0/2391570797 shutdown_connections 2023-10-16T14:29:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2391570797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 0x7fdee015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2391570797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 -- 172.21.15.29:0/2391570797 >> 172.21.15.29:0/2391570797 conn(0x7fdee00ce6a0 msgr2=0x7fdee00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.008+0000 7fdee6f54640 1 -- 172.21.15.29:0/2391570797 shutdown_connections 2023-10-16T14:29:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.009+0000 7fdee6f54640 1 -- 172.21.15.29:0/2391570797 wait complete. 2023-10-16T14:29:51.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.010+0000 7fdee6f54640 1 Processor -- start 2023-10-16T14:29:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.011+0000 7fdee6f54640 1 -- start start 2023-10-16T14:29:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.011+0000 7fdee6f54640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee006c040 0x7fdee00ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.011+0000 7fdee6f54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 0x7fdee00c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee6f54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 0x7fdee00af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee6f54640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00af290 con 0x7fdee00ad970 2023-10-16T14:29:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee6f54640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00ae390 con 0x7fdee006c040 2023-10-16T14:29:51.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee6f54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00ae610 con 0x7fdee00cda80 2023-10-16T14:29:51.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 0x7fdee00c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:51.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 0x7fdee00c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53262/0 (socket says 172.21.15.29:53262) 2023-10-16T14:29:51.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 -- 172.21.15.29:0/2808390783 learned_addr learned my addr 172.21.15.29:0/2808390783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5f52640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee006c040 0x7fdee00ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 -- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 msgr2=0x7fdee00af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 0x7fdee00af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 -- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee006c040 msgr2=0x7fdee00ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee006c040 0x7fdee00ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.012+0000 7fdee5751640 1 -- 172.21.15.29:0/2808390783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc0094d0 con 0x7fdee00ad970 2023-10-16T14:29:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.013+0000 7fdee5751640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 0x7fdee00c76d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fded000b3c0 tx=0x7fded000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.013+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0011830 con 0x7fdee00ad970 2023-10-16T14:29:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.013+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded0011990 con 0x7fdee00ad970 2023-10-16T14:29:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.013+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0011d60 con 0x7fdee00ad970 2023-10-16T14:29:51.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.013+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee00ae8f0 con 0x7fdee00ad970 2023-10-16T14:29:51.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.013+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee00aeeb0 con 0x7fdee00ad970 2023-10-16T14:29:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.014+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee006b330 con 0x7fdee00ad970 2023-10-16T14:29:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.015+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fded001b460 con 0x7fdee00ad970 2023-10-16T14:29:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.016+0000 7fded67fc640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec406b070 0x7fdec406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.016+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7075+0+0 (secure 0 0 0) 0x7fded0089cb0 con 0x7fdee00ad970 2023-10-16T14:29:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.017+0000 7fdee5f52640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec406b070 0x7fdec406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.018+0000 7fdee5f52640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec406b070 0x7fdec406d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdedc036000 tx=0x7fdedc03c040 comp rx=0 tx=0).ready entity=mgr.4861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.021+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fded0054570 con 0x7fdee00ad970 2023-10-16T14:29:51.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:51.294+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fdee004e9b0 con 0x7fdee00ad970 2023-10-16T14:29:52.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.116+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7fded0057b90 con 0x7fdee00ad970 2023-10-16T14:29:52.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.121+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec406b070 msgr2=0x7fdec406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.121+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec406b070 0x7fdec406d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdedc036000 tx=0x7fdedc03c040 comp rx=0 tx=0).stop 2023-10-16T14:29:52.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.121+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 msgr2=0x7fdee00c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 0x7fdee00c76d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fded000b3c0 tx=0x7fded000b880 comp rx=0 tx=0).stop 2023-10-16T14:29:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fded67fc640 1 -- 172.21.15.29:0/2808390783 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fded0054940 con 0x7fdee00ad970 2023-10-16T14:29:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 shutdown_connections 2023-10-16T14:29:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec406b070 0x7fdec406d530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdee00ad970 0x7fdee00c76d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00cda80 0x7fdee00af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee006c040 0x7fdee00ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.122+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 >> 172.21.15.29:0/2808390783 conn(0x7fdee00ce6a0 msgr2=0x7fdee006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.123+0000 7fded67fc640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec4034080 0x7fdec4036470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.124+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 shutdown_connections 2023-10-16T14:29:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.124+0000 7fdee6f54640 1 --2- 172.21.15.29:0/2808390783 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fdec4034080 0x7fdec4036470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.124+0000 7fdee6f54640 1 -- 172.21.15.29:0/2808390783 wait complete. 2023-10-16T14:29:52.157 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:29:52.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:29:52.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:29:52.355 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:29:52.356 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:29:52.356 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:29:52.421 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:29:52.422 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:29:52.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:29:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.662+0000 7fb241173640 1 Processor -- start 2023-10-16T14:29:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.662+0000 7fb241173640 1 -- start start 2023-10-16T14:29:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.664+0000 7fb241173640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb23c15aaa0 0x7fb23c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.664+0000 7fb241173640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23c15ed60 con 0x7fb23c0ce390 2023-10-16T14:29:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.664+0000 7fb241173640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23c0707e0 con 0x7fb23c15ae80 2023-10-16T14:29:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.664+0000 7fb241173640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c06b810 con 0x7fb23c15aaa0 2023-10-16T14:29:52.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.664+0000 7fb23b59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb23c0ce390 0x7fb23c070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60548/0 (socket says 172.21.15.29:60548) 2023-10-16T14:29:52.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.664+0000 7fb23b59e640 1 -- 172.21.15.29:0/244927287 learned_addr learned my addr 172.21.15.29:0/244927287 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:52.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23ad9d640 1 --2- 172.21.15.29:0/244927287 >> v2:172.21.15.170:3300/0 conn(0x7fb23c15aaa0 0x7fb23c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642168880 0 0) 0x7fb23c15ed60 con 0x7fb23c0ce390 2023-10-16T14:29:52.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb21c003520 con 0x7fb23c0ce390 2023-10-16T14:29:52.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920124844 0 0) 0x7fb21c003520 con 0x7fb23c0ce390 2023-10-16T14:29:52.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 >> v2:172.21.15.170:3300/0 conn(0x7fb23c15aaa0 msgr2=0x7fb23c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:52.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 --2- 172.21.15.29:0/244927287 >> v2:172.21.15.170:3300/0 conn(0x7fb23c15aaa0 0x7fb23c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 >> v1:172.21.15.170:6789/0 conn(0x7fb23c15ae80 legacy=0x7fb23c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c15ef90 con 0x7fb23c0ce390 2023-10-16T14:29:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.665+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fb2340019f0 con 0x7fb23c0ce390 2023-10-16T14:29:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 >> v1:172.21.15.29:6789/0 conn(0x7fb23c0ce390 legacy=0x7fb23c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0039c0 0x7fb21c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0047c0 0x7fb21c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c005180 0x7fb21c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c06b810 con 0x7fb21c005180 2023-10-16T14:29:52.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c00c080 con 0x7fb21c0039c0 2023-10-16T14:29:52.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c010650 con 0x7fb21c0047c0 2023-10-16T14:29:52.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23b59e640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c005180 0x7fb21c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23ad9d640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0039c0 0x7fb21c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23b59e640 1 -- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0047c0 msgr2=0x7fb21c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23b59e640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0047c0 0x7fb21c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23b59e640 1 -- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0039c0 msgr2=0x7fb21c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23b59e640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0039c0 0x7fb21c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.666+0000 7fb23b59e640 1 -- 172.21.15.29:0/244927287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c003520 con 0x7fb21c005180 2023-10-16T14:29:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.667+0000 7fb23ad9d640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0039c0 0x7fb21c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:52.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.667+0000 7fb23b59e640 1 -- 172.21.15.29:0/244927287 reap_dead start 2023-10-16T14:29:52.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.668+0000 7fb23b59e640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c005180 0x7fb21c00b980 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb23400a750 tx=0x7fb21c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62346d173731933 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:52.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.668+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234034630 con 0x7fb21c005180 2023-10-16T14:29:52.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.668+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23402f070 con 0x7fb21c005180 2023-10-16T14:29:52.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.668+0000 7fb23a59c640 1 -- 172.21.15.29:0/244927287 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2340349f0 con 0x7fb21c005180 2023-10-16T14:29:52.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.668+0000 7fb241173640 1 -- 172.21.15.29:0/244927287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c005180 msgr2=0x7fb21c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.668+0000 7fb241173640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c005180 0x7fb21c00b980 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb23400a750 tx=0x7fb21c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.669+0000 7fb241173640 1 -- 172.21.15.29:0/244927287 shutdown_connections 2023-10-16T14:29:52.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.669+0000 7fb241173640 1 --2- 172.21.15.29:0/244927287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c005180 0x7fb21c00b980 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.669+0000 7fb241173640 1 -- 172.21.15.29:0/244927287 >> 172.21.15.29:0/244927287 conn(0x7fb23c0cb410 msgr2=0x7fb23c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:52.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.669+0000 7fb241173640 1 -- 172.21.15.29:0/244927287 shutdown_connections 2023-10-16T14:29:52.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.669+0000 7fb241173640 1 -- 172.21.15.29:0/244927287 wait complete. 2023-10-16T14:29:52.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.670+0000 7fb241173640 1 Processor -- start 2023-10-16T14:29:52.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.671+0000 7fb241173640 1 -- start start 2023-10-16T14:29:52.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb241173640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c005180 0x7fb23c152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb241173640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 0x7fb23c0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb241173640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23c1ef0b0 0x7fb23c0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb241173640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c157f70 con 0x7fb23c1eec30 2023-10-16T14:29:52.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb241173640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c1581c0 con 0x7fb21c005180 2023-10-16T14:29:52.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb241173640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c1584a0 con 0x7fb23c1ef0b0 2023-10-16T14:29:52.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 0x7fb23c0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:52.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 0x7fb23c0ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53298/0 (socket says 172.21.15.29:53298) 2023-10-16T14:29:52.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 -- 172.21.15.29:0/63384834 learned_addr learned my addr 172.21.15.29:0/63384834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:52.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb23ad9d640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c005180 0x7fb23c152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:52.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb23b59e640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23c1ef0b0 0x7fb23c0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:52.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 -- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23c1ef0b0 msgr2=0x7fb23c0ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23c1ef0b0 0x7fb23c0ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 -- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c005180 msgr2=0x7fb23c152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c005180 0x7fb23c152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.672+0000 7fb233fff640 1 -- 172.21.15.29:0/63384834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c158780 con 0x7fb23c1eec30 2023-10-16T14:29:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.673+0000 7fb233fff640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 0x7fb23c0ca180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c430 tx=0x7fb22400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.673+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2240052d0 con 0x7fb23c1eec30 2023-10-16T14:29:52.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.673+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224005450 con 0x7fb23c1eec30 2023-10-16T14:29:52.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.673+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224007260 con 0x7fb23c1eec30 2023-10-16T14:29:52.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.673+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c158a60 con 0x7fb23c1eec30 2023-10-16T14:29:52.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.673+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c158f80 con 0x7fb23c1eec30 2023-10-16T14:29:52.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.674+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c158bf0 con 0x7fb23c1eec30 2023-10-16T14:29:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.677+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fb2240073c0 con 0x7fb23c1eec30 2023-10-16T14:29:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.678+0000 7fb2337fe640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 0x7fb22006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.678+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb224089030 con 0x7fb23c1eec30 2023-10-16T14:29:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.678+0000 7fb23ad9d640 1 -- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 msgr2=0x7fb22006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.678+0000 7fb23ad9d640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 0x7fb22006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:52.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.683+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb224053aa0 con 0x7fb23c1eec30 2023-10-16T14:29:52.733 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:52.729+0000 7fb39f413040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:29:52.785 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:52.782+0000 7efcbc6b6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:29:52.786 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:52.782+0000 7fb39f413040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:29:52.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:52.839+0000 7efcbc6b6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:29:52.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.876+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb23c04e160 con 0x7fb23c1eec30 2023-10-16T14:29:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.878+0000 7fb2337fe640 1 -- 172.21.15.29:0/63384834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+18028 (secure 0 0 0) 0x7fb2240570c0 con 0x7fb23c1eec30 2023-10-16T14:29:52.882 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:52.882 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:29:52.110875+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":232,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/176252655":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/1855634044":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3139659877":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3736763821":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6833/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6832/1268688927":"2023-10-17T14:17:54.926664+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":4},{"begin":8,"length":1},{"begin":10,"length":1},{"begin":12,"length":1},{"begin":16,"length":1},{"begin":18,"length":2},{"begin":24,"length":2},{"begin":30,"length":1},{"begin":33,"length":5},{"begin":40,"length":1},{"begin":42,"length":3},{"begin":49,"length":1},{"begin":52,"length":2},{"begin":55,"length":1},{"begin":58,"length":1},{"begin":60,"length":2},{"begin":64,"length":1},{"begin":70,"length":1},{"begin":73,"length":1},{"begin":75,"length":1},{"begin":77,"length":1},{"begin":80,"length":1},{"begin":82,"length":3},{"begin":86,"length":1},{"begin":90,"length":1},{"begin":92,"length":1},{"begin":94,"length":2},{"begin":97,"length":2},{"begin":102,"length":3},{"begin":107,"length":4},{"begin":113,"length":5},{"begin":186,"length":47}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":19,"length":1},{"begin":33,"length":1},{"begin":40,"length":1},{"begin":55,"length":1},{"begin":64,"length":1},{"begin":73,"length":1},{"begin":80,"length":1},{"begin":95,"length":1},{"begin":103,"length":1},{"begin":224,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:29:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.879+0000 7fb23ad9d640 1 -- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 msgr2=0x7fb22006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:52.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.879+0000 7fb23ad9d640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 0x7fb22006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:29:52.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 msgr2=0x7fb22006d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 0x7fb22006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 msgr2=0x7fb23c0ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 0x7fb23c0ca180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c430 tx=0x7fb22400c8f0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 shutdown_connections 2023-10-16T14:29:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fb22006b050 0x7fb22006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb23c1eec30 0x7fb23c0ca180 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb23c1ef0b0 0x7fb23c0ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 --2- 172.21.15.29:0/63384834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c005180 0x7fb23c152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 >> 172.21.15.29:0/63384834 conn(0x7fb23c0cb410 msgr2=0x7fb23c1566c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 shutdown_connections 2023-10-16T14:29:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:52.881+0000 7fb241173640 1 -- 172.21.15.29:0/63384834 wait complete. 2023-10-16T14:29:52.902 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:17:38.728376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:17:39.652475+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 232, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:29:52.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:29:53.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.106+0000 7f577a6d7640 1 Processor -- start 2023-10-16T14:29:53.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.106+0000 7f577a6d7640 1 -- start start 2023-10-16T14:29:53.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f577a6d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57740cf710 0x7f57740b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f577a6d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f577a6d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f57740b38e0 0x7f57740d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f577a6d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57740d3c60 con 0x7f57740cf710 2023-10-16T14:29:53.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f577a6d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57740d3980 con 0x7f57740b38e0 2023-10-16T14:29:53.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f577a6d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57740ce630 con 0x7f57740cfaf0 2023-10-16T14:29:53.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f5773fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57740cf710 0x7f57740b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f5773fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57740cf710 0x7f57740b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53300/0 (socket says 172.21.15.29:53300) 2023-10-16T14:29:53.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f5773fff640 1 -- 172.21.15.29:0/2163365653 learned_addr learned my addr 172.21.15.29:0/2163365653 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f5773fff640 1 -- 172.21.15.29:0/2163365653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 msgr2=0x7f57740b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f5773fff640 1 --2- 172.21.15.29:0/2163365653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.108+0000 7f5773fff640 1 -- 172.21.15.29:0/2163365653 >> v2:172.21.15.170:3300/0 conn(0x7f57740b38e0 msgr2=0x7f57740d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:53.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.109+0000 7f5773fff640 1 --2- 172.21.15.29:0/2163365653 >> v2:172.21.15.170:3300/0 conn(0x7f57740b38e0 0x7f57740d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.109+0000 7f5773fff640 1 -- 172.21.15.29:0/2163365653 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740ce910 con 0x7f57740cf710 2023-10-16T14:29:53.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.110+0000 7f5773fff640 1 --2- 172.21.15.29:0/2163365653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f57740b2d10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f577406cd60 tx=0x7f576402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed680a6edd86f9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:53.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.110+0000 7f5772ffd640 1 -- 172.21.15.29:0/2163365653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764004230 con 0x7f57740cf710 2023-10-16T14:29:53.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.110+0000 7f5772ffd640 1 -- 172.21.15.29:0/2163365653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5764004390 con 0x7f57740cf710 2023-10-16T14:29:53.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.110+0000 7f5772ffd640 1 -- 172.21.15.29:0/2163365653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764003810 con 0x7f57740cf710 2023-10-16T14:29:53.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 -- 172.21.15.29:0/2163365653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 msgr2=0x7f57740b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 --2- 172.21.15.29:0/2163365653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f57740b2d10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f577406cd60 tx=0x7f576402b860 comp rx=0 tx=0).stop 2023-10-16T14:29:53.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 -- 172.21.15.29:0/2163365653 shutdown_connections 2023-10-16T14:29:53.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 --2- 172.21.15.29:0/2163365653 >> v2:172.21.15.170:3300/0 conn(0x7f57740b38e0 0x7f57740d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 --2- 172.21.15.29:0/2163365653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 --2- 172.21.15.29:0/2163365653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f57740b2d10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 -- 172.21.15.29:0/2163365653 >> 172.21.15.29:0/2163365653 conn(0x7f57740d2cc0 msgr2=0x7f57740d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:53.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 -- 172.21.15.29:0/2163365653 shutdown_connections 2023-10-16T14:29:53.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.111+0000 7f577a6d7640 1 -- 172.21.15.29:0/2163365653 wait complete. 2023-10-16T14:29:53.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.113+0000 7f577a6d7640 1 Processor -- start 2023-10-16T14:29:53.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.113+0000 7f577a6d7640 1 -- start start 2023-10-16T14:29:53.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f577a6d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57740b38e0 0x7f577415e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f577a6d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f577415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f577a6d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f577a6d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740b1430 con 0x7f57740cf710 2023-10-16T14:29:53.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f577a6d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57741566d0 con 0x7f57740b38e0 2023-10-16T14:29:53.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f577a6d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5774156110 con 0x7f57740cfaf0 2023-10-16T14:29:53.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f57737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f577415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f57737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f577415e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53308/0 (socket says 172.21.15.29:53308) 2023-10-16T14:29:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.114+0000 7f57737fe640 1 -- 172.21.15.29:0/1176120652 learned_addr learned my addr 172.21.15.29:0/1176120652 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f5773fff640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57740b38e0 0x7f577415e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f57737fe640 1 -- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 msgr2=0x7f57740af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f5778c74640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f57737fe640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f57737fe640 1 -- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57740b38e0 msgr2=0x7f577415e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f57737fe640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57740b38e0 0x7f577415e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f57737fe640 1 -- 172.21.15.29:0/1176120652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764009460 con 0x7f57740cf710 2023-10-16T14:29:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f5773fff640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57740b38e0 0x7f577415e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f5778c74640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f57737fe640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f577415e6a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f576800b410 tx=0x7f576800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.115+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768011830 con 0x7f57740cf710 2023-10-16T14:29:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.116+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57680119b0 con 0x7f57740cf710 2023-10-16T14:29:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.116+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768011d60 con 0x7f57740cf710 2023-10-16T14:29:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.116+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57741563f0 con 0x7f57740cf710 2023-10-16T14:29:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.116+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57741569d0 con 0x7f57740cf710 2023-10-16T14:29:53.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.117+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57740b1570 con 0x7f57740cf710 2023-10-16T14:29:53.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.118+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f576801a400 con 0x7f57740cf710 2023-10-16T14:29:53.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.119+0000 7f5770ff9640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 0x7f576006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.119+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5768089df0 con 0x7f57740cf710 2023-10-16T14:29:53.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.119+0000 7f5773fff640 1 -- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 msgr2=0x7f576006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:53.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.119+0000 7f5773fff640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 0x7f576006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:53.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.123+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f57740b1570 con 0x7f57740cf710 2023-10-16T14:29:53.229 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.226+0000 7fb39f413040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:29:53.281 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.278+0000 7fb39f413040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:29:53.320 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.317+0000 7efcbc6b6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:29:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.317+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57740b1570 con 0x7f57740cf710 2023-10-16T14:29:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.318+0000 7f5770ff9640 1 -- 172.21.15.29:0/1176120652 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57740b1570 con 0x7f57740cf710 2023-10-16T14:29:53.321 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:29:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.320+0000 7f5773fff640 1 -- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 msgr2=0x7f576006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.320+0000 7f5773fff640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 0x7f576006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:29:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 msgr2=0x7f576006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 0x7f576006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 msgr2=0x7f577415e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f577415e6a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f576800b410 tx=0x7f576800b8d0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 shutdown_connections 2023-10-16T14:29:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f576006b020 0x7f576006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57740cf710 0x7f577415e6a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57740cfaf0 0x7f57740af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 --2- 172.21.15.29:0/1176120652 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57740b38e0 0x7f577415e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 >> 172.21.15.29:0/1176120652 conn(0x7f57740d2cc0 msgr2=0x7f5774160150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 shutdown_connections 2023-10-16T14:29:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.321+0000 7f577a6d7640 1 -- 172.21.15.29:0/1176120652 wait complete. 2023-10-16T14:29:53.331 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.327+0000 7fb39f413040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:29:53.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:29:53.376 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.374+0000 7efcbc6b6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:29:53.391 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.387+0000 7fb39f413040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:29:53.431 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.428+0000 7efcbc6b6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:29:53.439 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.436+0000 7fb39f413040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:29:53.496 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.493+0000 7efcbc6b6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:29:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.538+0000 7fa2bef2e640 1 Processor -- start 2023-10-16T14:29:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.539+0000 7fa2bef2e640 1 -- start start 2023-10-16T14:29:53.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.540+0000 7fa2bef2e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa2b815cd70 0x7fa2b80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.540+0000 7fa2bef2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2b80b8480 0x7fa2b80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.540+0000 7fa2bef2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80d0be0 0x7fa2b80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.540+0000 7fa2bef2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b815d150 con 0x7fa2b80b8480 2023-10-16T14:29:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.540+0000 7fa2bef2e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b80d1180 con 0x7fa2b815cd70 2023-10-16T14:29:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.540+0000 7fa2bef2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80d1460 con 0x7fa2b80d0be0 2023-10-16T14:29:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2b80b8480 0x7fa2b80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2b80b8480 0x7fa2b80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53310/0 (socket says 172.21.15.29:53310) 2023-10-16T14:29:53.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 -- 172.21.15.29:0/876854371 learned_addr learned my addr 172.21.15.29:0/876854371 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:53.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bdf2c640 1 --2- 172.21.15.29:0/876854371 >> v2:172.21.15.170:3300/0 conn(0x7fa2b815cd70 0x7fa2b80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2be72d640 1 --2- 172.21.15.29:0/876854371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80d0be0 0x7fa2b80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 -- 172.21.15.29:0/876854371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80d0be0 msgr2=0x7fa2b80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:53.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 --2- 172.21.15.29:0/876854371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80d0be0 0x7fa2b80cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 -- 172.21.15.29:0/876854371 >> v2:172.21.15.170:3300/0 conn(0x7fa2b815cd70 msgr2=0x7fa2b80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:53.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 --2- 172.21.15.29:0/876854371 >> v2:172.21.15.170:3300/0 conn(0x7fa2b815cd70 0x7fa2b80d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.541+0000 7fa2bd72b640 1 -- 172.21.15.29:0/876854371 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80d1740 con 0x7fa2b80b8480 2023-10-16T14:29:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.542+0000 7fa2bd72b640 1 --2- 172.21.15.29:0/876854371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80b8480 0x7fa2b80cd7c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80b56a0 tx=0x7fa2a802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a6fb399cd40b34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bcf2a640 1 -- 172.21.15.29:0/876854371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8031860 con 0x7fa2b80b8480 2023-10-16T14:29:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bcf2a640 1 -- 172.21.15.29:0/876854371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a80319c0 con 0x7fa2b80b8480 2023-10-16T14:29:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bcf2a640 1 -- 172.21.15.29:0/876854371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8031d50 con 0x7fa2b80b8480 2023-10-16T14:29:53.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 -- 172.21.15.29:0/876854371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80b8480 msgr2=0x7fa2b80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/876854371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80b8480 0x7fa2b80cd7c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80b56a0 tx=0x7fa2a802b900 comp rx=0 tx=0).stop 2023-10-16T14:29:53.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 -- 172.21.15.29:0/876854371 shutdown_connections 2023-10-16T14:29:53.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/876854371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80d0be0 0x7fa2b80cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/876854371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80b8480 0x7fa2b80cd7c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/876854371 >> v2:172.21.15.170:3300/0 conn(0x7fa2b815cd70 0x7fa2b80d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 -- 172.21.15.29:0/876854371 >> 172.21.15.29:0/876854371 conn(0x7fa2b80cf570 msgr2=0x7fa2b80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:53.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.543+0000 7fa2bef2e640 1 -- 172.21.15.29:0/876854371 shutdown_connections 2023-10-16T14:29:53.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.544+0000 7fa2bef2e640 1 -- 172.21.15.29:0/876854371 wait complete. 2023-10-16T14:29:53.551 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.545+0000 7fb39f413040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:29:53.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.545+0000 7fa2bef2e640 1 Processor -- start 2023-10-16T14:29:53.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.545+0000 7fa2bef2e640 1 -- start start 2023-10-16T14:29:53.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.546+0000 7fa2bef2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80b8480 0x7fa2b814d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.546+0000 7fa2bef2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 0x7fa2b815fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.546+0000 7fa2bef2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2b815cd70 0x7fa2b80cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.546+0000 7fa2bef2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b81f2190 con 0x7fa2b80d0be0 2023-10-16T14:29:53.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.546+0000 7fa2bef2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b81f5dc0 con 0x7fa2b815cd70 2023-10-16T14:29:53.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.546+0000 7fa2bef2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b81f60a0 con 0x7fa2b80b8480 2023-10-16T14:29:53.553 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.546+0000 7efcbc6b6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:29:53.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 0x7fa2b815fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 0x7fa2b815fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53320/0 (socket says 172.21.15.29:53320) 2023-10-16T14:29:53.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 -- 172.21.15.29:0/2016709821 learned_addr learned my addr 172.21.15.29:0/2016709821 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:53.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bdf2c640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80b8480 0x7fa2b814d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2be72d640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2b815cd70 0x7fa2b80cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:53.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 -- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80b8480 msgr2=0x7fa2b814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80b8480 0x7fa2b814d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 -- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2b815cd70 msgr2=0x7fa2b80cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2b815cd70 0x7fa2b80cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2bd72b640 1 -- 172.21.15.29:0/2016709821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a80094d0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.547+0000 7fa2be72d640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2b815cd70 0x7fa2b80cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:53.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.548+0000 7fa2bd72b640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 0x7fa2b815fc00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa2a802b940 tx=0x7fa2a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:53.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.548+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a803ad30 con 0x7fa2b80d0be0 2023-10-16T14:29:53.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.548+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a80443f0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.548+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b81f6380 con 0x7fa2b80d0be0 2023-10-16T14:29:53.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.548+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80447a0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.548+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b80caac0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.549+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b80cacb0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.551+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fa2a804d450 con 0x7fa2b80d0be0 2023-10-16T14:29:53.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.552+0000 7fa2ae7fc640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 0x7fa29c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:53.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.552+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa2a80ac330 con 0x7fa2b80d0be0 2023-10-16T14:29:53.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.552+0000 7fa2bdf2c640 1 -- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 msgr2=0x7fa29c06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:53.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.552+0000 7fa2bdf2c640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 0x7fa29c06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:53.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.556+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2a8076c80 con 0x7fa2b80d0be0 2023-10-16T14:29:53.613 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.610+0000 7fb39f413040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:29:53.663 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.659+0000 7fb39f413040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:29:53.669 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.667+0000 7efcbc6b6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:29:53.718 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.715+0000 7fb39f413040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:29:53.736 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.733+0000 7efcbc6b6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:29:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.752+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa2b804e9b0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.753+0000 7fa2ae7fc640 1 -- 172.21.15.29:0/2016709821 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v49) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa2a807a2a0 con 0x7fa2b80d0be0 2023-10-16T14:29:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.753+0000 7fa2bdf2c640 1 -- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 msgr2=0x7fa29c06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.753+0000 7fa2bdf2c640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 0x7fa29c06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:29:53.756 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:29:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 msgr2=0x7fa29c06d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:53.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 0x7fa29c06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 msgr2=0x7fa2b815fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 0x7fa2b815fc00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa2a802b940 tx=0x7fa2a80046c0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 shutdown_connections 2023-10-16T14:29:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7fa29c06b050 0x7fa29c06d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2b80d0be0 0x7fa2b815fc00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2b815cd70 0x7fa2b80cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 --2- 172.21.15.29:0/2016709821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2b80b8480 0x7fa2b814d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.755+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 >> 172.21.15.29:0/2016709821 conn(0x7fa2b80cf570 msgr2=0x7fa2b80b68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.756+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 shutdown_connections 2023-10-16T14:29:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:53.756+0000 7fa2bef2e640 1 -- 172.21.15.29:0/2016709821 wait complete. 2023-10-16T14:29:53.768 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:53.764+0000 7fb39f413040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:29:53.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:29:53.789 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.786+0000 7efcbc6b6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:29:53.850 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.847+0000 7efcbc6b6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:29:53.904 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:53.901+0000 7efcbc6b6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:29:54.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.002+0000 7f05b5d00640 1 Processor -- start 2023-10-16T14:29:54.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.002+0000 7f05b5d00640 1 -- start start 2023-10-16T14:29:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05b5d00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05b00cf710 0x7f05b00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05b5d00640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05b00cfaf0 0x7f05b00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05b5d00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00d3980 con 0x7f05b00b38e0 2023-10-16T14:29:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05b5d00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b00d3c60 con 0x7f05b00cf710 2023-10-16T14:29:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05b5d00640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b00ce630 con 0x7f05b00cfaf0 2023-10-16T14:29:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05af7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05b00cf710 0x7f05b00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05affff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f05b00b38e0 0x7f05b00d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43606/0 (socket says 172.21.15.29:43606) 2023-10-16T14:29:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.004+0000 7f05affff640 1 -- 172.21.15.29:0/4112600673 learned_addr learned my addr 172.21.15.29:0/4112600673 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:54.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.005+0000 7f05aeffd640 1 --2- 172.21.15.29:0/4112600673 >> v2:172.21.15.170:3300/0 conn(0x7f05b00cfaf0 0x7f05b00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.005+0000 7f05af7fe640 1 -- 172.21.15.29:0/4112600673 >> v2:172.21.15.170:3300/0 conn(0x7f05b00cfaf0 msgr2=0x7f05b00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.005+0000 7f05af7fe640 1 --2- 172.21.15.29:0/4112600673 >> v2:172.21.15.170:3300/0 conn(0x7f05b00cfaf0 0x7f05b00b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.005+0000 7f05af7fe640 1 -- 172.21.15.29:0/4112600673 >> v1:172.21.15.29:6789/0 conn(0x7f05b00b38e0 legacy=0x7f05b00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.005+0000 7f05af7fe640 1 -- 172.21.15.29:0/4112600673 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00ce910 con 0x7f05b00cf710 2023-10-16T14:29:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.005+0000 7f05ae7fc640 1 -- 172.21.15.29:0/4112600673 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272782298 0 0) 0x7f05b00d3980 con 0x7f05b00b38e0 2023-10-16T14:29:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.006+0000 7f05af7fe640 1 --2- 172.21.15.29:0/4112600673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cf710 0x7f05b00b2d10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f05b008d210 tx=0x7f05a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b70d0efa9acc24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.006+0000 7f05ae7fc640 1 -- 172.21.15.29:0/4112600673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a002e070 con 0x7f05b00cf710 2023-10-16T14:29:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.006+0000 7f05ae7fc640 1 -- 172.21.15.29:0/4112600673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a00315a0 con 0x7f05b00cf710 2023-10-16T14:29:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.006+0000 7f05ae7fc640 1 -- 172.21.15.29:0/4112600673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0031930 con 0x7f05b00cf710 2023-10-16T14:29:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 -- 172.21.15.29:0/4112600673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cf710 msgr2=0x7f05b00b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 --2- 172.21.15.29:0/4112600673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cf710 0x7f05b00b2d10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f05b008d210 tx=0x7f05a002b840 comp rx=0 tx=0).stop 2023-10-16T14:29:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 -- 172.21.15.29:0/4112600673 shutdown_connections 2023-10-16T14:29:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 --2- 172.21.15.29:0/4112600673 >> v2:172.21.15.170:3300/0 conn(0x7f05b00cfaf0 0x7f05b00b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 --2- 172.21.15.29:0/4112600673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cf710 0x7f05b00b2d10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 -- 172.21.15.29:0/4112600673 >> 172.21.15.29:0/4112600673 conn(0x7f05b00d2cc0 msgr2=0x7f05b00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 -- 172.21.15.29:0/4112600673 shutdown_connections 2023-10-16T14:29:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.007+0000 7f05b5d00640 1 -- 172.21.15.29:0/4112600673 wait complete. 2023-10-16T14:29:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.009+0000 7f05b5d00640 1 Processor -- start 2023-10-16T14:29:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.009+0000 7f05b5d00640 1 -- start start 2023-10-16T14:29:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05b5d00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05b00b38e0 0x7f05b00ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05b5d00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05b00cf710 0x7f05b01f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05b5d00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 0x7f05b00b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05b5d00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b015d190 con 0x7f05b00cfaf0 2023-10-16T14:29:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05b5d00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b015cbd0 con 0x7f05b00b38e0 2023-10-16T14:29:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05b5d00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b015ceb0 con 0x7f05b00cf710 2023-10-16T14:29:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 0x7f05b00b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 0x7f05b00b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57578/0 (socket says 172.21.15.29:57578) 2023-10-16T14:29:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.010+0000 7f05affff640 1 -- 172.21.15.29:0/3300323557 learned_addr learned my addr 172.21.15.29:0/3300323557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05af7fe640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05b00b38e0 0x7f05b00ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05aeffd640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05b00cf710 0x7f05b01f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05affff640 1 -- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05b00cf710 msgr2=0x7f05b01f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05affff640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05b00cf710 0x7f05b01f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05affff640 1 -- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05b00b38e0 msgr2=0x7f05b00ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05affff640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05b00b38e0 0x7f05b00ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05affff640 1 -- 172.21.15.29:0/3300323557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a00093a0 con 0x7f05b00cfaf0 2023-10-16T14:29:54.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05af7fe640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05b00b38e0 0x7f05b00ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:54.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f05affff640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 0x7f05b00b7c90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f05a800c780 tx=0x7f05a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:54.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.011+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a800ce90 con 0x7f05b00cfaf0 2023-10-16T14:29:54.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.012+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a800a640 con 0x7f05b00cfaf0 2023-10-16T14:29:54.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.012+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a800a9d0 con 0x7f05b00cfaf0 2023-10-16T14:29:54.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.012+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b01f5e40 con 0x7f05b00cfaf0 2023-10-16T14:29:54.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.012+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b00cac40 con 0x7f05b00cfaf0 2023-10-16T14:29:54.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.013+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b01f6360 con 0x7f05b00cfaf0 2023-10-16T14:29:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.014+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f05a800ab30 con 0x7f05b00cfaf0 2023-10-16T14:29:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.015+0000 7f0597fff640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 0x7f059c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.015+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f05a8089220 con 0x7f05b00cfaf0 2023-10-16T14:29:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.015+0000 7f05af7fe640 1 -- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 msgr2=0x7f059c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.015+0000 7f05af7fe640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 0x7f059c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.019+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05b01f6360 con 0x7f05b00cfaf0 2023-10-16T14:29:54.145 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.141+0000 7fb39f413040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:29:54.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.216+0000 7f05af7fe640 1 -- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 msgr2=0x7f059c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:54.220 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:29:54.302 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.299+0000 7fb39f413040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:29:54.409 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.406+0000 7fb39f413040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:29:54.459 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.455+0000 7fb39f413040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:29:54.565 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.562+0000 7fb39f413040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:29:54.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.668+0000 7fb39f413040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:29:54.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:54.310+0000 7efcbc6b6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:29:54.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:54.478+0000 7efcbc6b6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:29:54.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.216+0000 7f05af7fe640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 0x7f059c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:29:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.217+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05b01f6360 con 0x7f05b00cfaf0 2023-10-16T14:29:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.218+0000 7f0597fff640 1 -- 172.21.15.29:0/3300323557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v49) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05b01f6360 con 0x7f05b00cfaf0 2023-10-16T14:29:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 msgr2=0x7f059c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 0x7f059c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 msgr2=0x7f05b00b7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 0x7f05b00b7c90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f05a800c780 tx=0x7f05a800cc40 comp rx=0 tx=0).stop 2023-10-16T14:29:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 shutdown_connections 2023-10-16T14:29:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f059c06b020 0x7f059c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05b00cfaf0 0x7f05b00b7c90 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05b00cf710 0x7f05b01f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 --2- 172.21.15.29:0/3300323557 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05b00b38e0 0x7f05b00ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 >> 172.21.15.29:0/3300323557 conn(0x7f05b00d2cc0 msgr2=0x7f05b0160110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:54.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 shutdown_connections 2023-10-16T14:29:54.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.220+0000 7f05b5d00640 1 -- 172.21.15.29:0/3300323557 wait complete. 2023-10-16T14:29:54.676 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:54.591+0000 7efcbc6b6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:29:54.676 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:54.643+0000 7efcbc6b6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:29:54.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:29:54.774 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:54.772+0000 7efcbc6b6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:29:54.827 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.823+0000 7fb39f413040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:29:54.875 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.871+0000 7fb39f413040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:29:54.891 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:54.888+0000 7efcbc6b6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:29:54.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.890+0000 7f6c3359e640 1 Processor -- start 2023-10-16T14:29:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.890+0000 7f6c3359e640 1 -- start start 2023-10-16T14:29:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c340ac890 0x7f6c340accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34160a90 con 0x7f6c3415c9f0 2023-10-16T14:29:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340ad200 con 0x7f6c3415dde0 2023-10-16T14:29:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c340cdec0 con 0x7f6c340ac890 2023-10-16T14:29:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c3415c9f0 0x7f6c340c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43614/0 (socket says 172.21.15.29:43614) 2023-10-16T14:29:54.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3259c640 1 -- 172.21.15.29:0/1541135013 learned_addr learned my addr 172.21.15.29:0/1541135013 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:54.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c32d9d640 1 --2- 172.21.15.29:0/1541135013 >> v2:172.21.15.29:3300/0 conn(0x7f6c340ac890 0x7f6c340accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3159a640 1 -- 172.21.15.29:0/1541135013 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177299963 0 0) 0x7f6c34160a90 con 0x7f6c3415c9f0 2023-10-16T14:29:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.892+0000 7f6c3159a640 1 -- 172.21.15.29:0/1541135013 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1c003520 con 0x7f6c3415c9f0 2023-10-16T14:29:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.893+0000 7f6c32d9d640 1 -- 172.21.15.29:0/1541135013 >> v1:172.21.15.170:6789/0 conn(0x7f6c3415dde0 legacy=0x7f6c340709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.893+0000 7f6c32d9d640 1 -- 172.21.15.29:0/1541135013 >> v1:172.21.15.29:6789/0 conn(0x7f6c3415c9f0 legacy=0x7f6c340c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.893+0000 7f6c32d9d640 1 -- 172.21.15.29:0/1541135013 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340cd900 con 0x7f6c340ac890 2023-10-16T14:29:54.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.894+0000 7f6c32d9d640 1 --2- 172.21.15.29:0/1541135013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c340ac890 0x7f6c340accc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6c340ad390 tx=0x7f6c2c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b69c3938baed0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.894+0000 7f6c3159a640 1 -- 172.21.15.29:0/1541135013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c02cce0 con 0x7f6c340ac890 2023-10-16T14:29:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.894+0000 7f6c3159a640 1 -- 172.21.15.29:0/1541135013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c02ce40 con 0x7f6c340ac890 2023-10-16T14:29:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.894+0000 7f6c3159a640 1 -- 172.21.15.29:0/1541135013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c034640 con 0x7f6c340ac890 2023-10-16T14:29:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 -- 172.21.15.29:0/1541135013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c340ac890 msgr2=0x7f6c340accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 --2- 172.21.15.29:0/1541135013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c340ac890 0x7f6c340accc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6c340ad390 tx=0x7f6c2c02bf30 comp rx=0 tx=0).stop 2023-10-16T14:29:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 -- 172.21.15.29:0/1541135013 shutdown_connections 2023-10-16T14:29:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 --2- 172.21.15.29:0/1541135013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c340ac890 0x7f6c340accc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 -- 172.21.15.29:0/1541135013 >> 172.21.15.29:0/1541135013 conn(0x7f6c340ce6a0 msgr2=0x7f6c340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 -- 172.21.15.29:0/1541135013 shutdown_connections 2023-10-16T14:29:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.895+0000 7f6c3359e640 1 -- 172.21.15.29:0/1541135013 wait complete. 2023-10-16T14:29:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.897+0000 7f6c3359e640 1 Processor -- start 2023-10-16T14:29:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.897+0000 7f6c3359e640 1 -- start start 2023-10-16T14:29:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c340ac890 0x7f6c340c1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c3415c9f0 0x7f6c34153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 0x7f6c340cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c341f35f0 con 0x7f6c3415dde0 2023-10-16T14:29:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340c19d0 con 0x7f6c340ac890 2023-10-16T14:29:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34153290 con 0x7f6c3415c9f0 2023-10-16T14:29:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c32d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 0x7f6c340cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c32d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 0x7f6c340cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57588/0 (socket says 172.21.15.29:57588) 2023-10-16T14:29:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c32d9d640 1 -- 172.21.15.29:0/3768134736 learned_addr learned my addr 172.21.15.29:0/3768134736 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c3259c640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c340ac890 0x7f6c340c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.898+0000 7f6c31d9b640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c3415c9f0 0x7f6c34153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c32d9d640 1 -- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c3415c9f0 msgr2=0x7f6c34153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c32d9d640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c3415c9f0 0x7f6c34153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c32d9d640 1 -- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c340ac890 msgr2=0x7f6c340c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c32d9d640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c340ac890 0x7f6c340c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c32d9d640 1 -- 172.21.15.29:0/3768134736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c009a70 con 0x7f6c3415dde0 2023-10-16T14:29:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c3259c640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c340ac890 0x7f6c340c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:54.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c31d9b640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c3415c9f0 0x7f6c34153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c32d9d640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 0x7f6c340cc860 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e80 tx=0x7f6c2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c03e610 con 0x7f6c3415dde0 2023-10-16T14:29:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c03e770 con 0x7f6c3415dde0 2023-10-16T14:29:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c03eae0 con 0x7f6c3415dde0 2023-10-16T14:29:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.899+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c341f3310 con 0x7f6c3415dde0 2023-10-16T14:29:54.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.900+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c341537f0 con 0x7f6c3415dde0 2023-10-16T14:29:54.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.901+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3404e9b0 con 0x7f6c3415dde0 2023-10-16T14:29:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.903+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f6c2c03b050 con 0x7f6c3415dde0 2023-10-16T14:29:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.903+0000 7f6c22ffd640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 0x7f6c1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.904+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6c2c0ac270 con 0x7f6c3415dde0 2023-10-16T14:29:54.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.904+0000 7f6c3259c640 1 -- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 msgr2=0x7f6c1406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.904+0000 7f6c3259c640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 0x7f6c1406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:54.908+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c2c076cd0 con 0x7f6c3415dde0 2023-10-16T14:29:54.994 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:54.990+0000 7fb39f413040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:29:55.060 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:55.057+0000 7efcbc6b6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:29:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.105+0000 7f6c3259c640 1 -- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 msgr2=0x7f6c1406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.105+0000 7f6c3259c640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 0x7f6c1406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:29:55.120 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:55.117+0000 7efcbc6b6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:29:55.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.126+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6c340adac0 con 0x7f6c3415dde0 2023-10-16T14:29:55.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.127+0000 7f6c22ffd640 1 -- 172.21.15.29:0/3768134736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v50) v1 ==== 80+0+359 (secure 0 0 0) 0x7f6c2c039180 con 0x7f6c3415dde0 2023-10-16T14:29:55.130 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/176252655 2023-10-17T14:17:54.926664+0000 2023-10-16T14:29:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1855634044 2023-10-17T14:17:54.926664+0000 2023-10-16T14:29:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3139659877 2023-10-17T14:17:54.926664+0000 2023-10-16T14:29:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3736763821 2023-10-17T14:17:54.926664+0000 2023-10-16T14:29:55.131 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/1268688927 2023-10-17T14:17:54.926664+0000 2023-10-16T14:29:55.132 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/1268688927 2023-10-17T14:17:54.926664+0000 2023-10-16T14:29:55.133 INFO:teuthology.orchestra.run.smithi029.stderr:listed 6 entries 2023-10-16T14:29:55.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 msgr2=0x7f6c1406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:55.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 0x7f6c1406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 msgr2=0x7f6c340cc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:55.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 0x7f6c340cc860 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e80 tx=0x7f6c2c004a40 comp rx=0 tx=0).stop 2023-10-16T14:29:55.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 shutdown_connections 2023-10-16T14:29:55.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f6c1406b020 0x7f6c1406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c3415dde0 0x7f6c340cc860 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c3415c9f0 0x7f6c34153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 --2- 172.21.15.29:0/3768134736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c340ac890 0x7f6c340c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 >> 172.21.15.29:0/3768134736 conn(0x7f6c340ce6a0 msgr2=0x7f6c3415b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:55.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 shutdown_connections 2023-10-16T14:29:55.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.130+0000 7f6c3359e640 1 -- 172.21.15.29:0/3768134736 wait complete. 2023-10-16T14:29:55.153 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:29:55.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:55.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:29:55.241 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:29:55.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:29:55.242 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:29:55.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:55.254+0000 7efcbc6b6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:29:55.288 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:29:55.288 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:29:55.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:29:55.343 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:29:55.344 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:29:55.397 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:29:55.398 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:29:55.398 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:29:55.447 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:29:55.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:29:55.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.661+0000 7f2b9a4e5640 1 Processor -- start 2023-10-16T14:29:55.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.661+0000 7f2b9a4e5640 1 -- start start 2023-10-16T14:29:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.662+0000 7f2b9a4e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b9415aaa0 0x7f2b9415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.662+0000 7f2b9a4e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b940ce390 0x7f2b94070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.662+0000 7f2b9a4e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9415ed60 con 0x7f2b9415ae80 2023-10-16T14:29:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.662+0000 7f2b9a4e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9406b810 con 0x7f2b940ce390 2023-10-16T14:29:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.662+0000 7f2b9a4e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b940707e0 con 0x7f2b9415aaa0 2023-10-16T14:29:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.662+0000 7f2b98a82640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b940ce390 0x7f2b94070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b98a82640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b940ce390 0x7f2b94070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57604/0 (socket says 172.21.15.29:57604) 2023-10-16T14:29:55.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b98a82640 1 -- 172.21.15.29:0/2501001794 learned_addr learned my addr 172.21.15.29:0/2501001794 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:55.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b93fff640 1 --2- 172.21.15.29:0/2501001794 >> v2:172.21.15.170:3300/0 conn(0x7f2b9415aaa0 0x7f2b9415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:55.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b98a82640 1 -- 172.21.15.29:0/2501001794 >> v2:172.21.15.170:3300/0 conn(0x7f2b9415aaa0 msgr2=0x7f2b9415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:55.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b98a82640 1 --2- 172.21.15.29:0/2501001794 >> v2:172.21.15.170:3300/0 conn(0x7f2b9415aaa0 0x7f2b9415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b98a82640 1 -- 172.21.15.29:0/2501001794 >> v1:172.21.15.29:6789/0 conn(0x7f2b9415ae80 legacy=0x7f2b940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:55.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.663+0000 7f2b98a82640 1 -- 172.21.15.29:0/2501001794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9415ef90 con 0x7f2b940ce390 2023-10-16T14:29:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.664+0000 7f2b98a82640 1 --2- 172.21.15.29:0/2501001794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b940ce390 0x7f2b94070f50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2b940cd990 tx=0x7f2b8c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a446daa33e5f51d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b937fe640 1 -- 172.21.15.29:0/2501001794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c030860 con 0x7f2b940ce390 2023-10-16T14:29:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b937fe640 1 -- 172.21.15.29:0/2501001794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c0309c0 con 0x7f2b940ce390 2023-10-16T14:29:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b937fe640 1 -- 172.21.15.29:0/2501001794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c030d50 con 0x7f2b940ce390 2023-10-16T14:29:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/2501001794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b940ce390 msgr2=0x7f2b94070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/2501001794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b940ce390 0x7f2b94070f50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2b940cd990 tx=0x7f2b8c02b9a0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/2501001794 shutdown_connections 2023-10-16T14:29:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/2501001794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b940ce390 0x7f2b94070f50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/2501001794 >> v2:172.21.15.170:3300/0 conn(0x7f2b9415aaa0 0x7f2b9415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.665+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/2501001794 >> 172.21.15.29:0/2501001794 conn(0x7f2b940cb410 msgr2=0x7f2b940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.666+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/2501001794 shutdown_connections 2023-10-16T14:29:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.666+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/2501001794 wait complete. 2023-10-16T14:29:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.668+0000 7f2b9a4e5640 1 Processor -- start 2023-10-16T14:29:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.668+0000 7f2b9a4e5640 1 -- start start 2023-10-16T14:29:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b9a4e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b940ce390 0x7f2b940c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b9a4e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9415aaa0 0x7f2b941f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b9a4e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 0x7f2b940af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b9a4e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b940c3460 con 0x7f2b9415ae80 2023-10-16T14:29:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b9a4e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b940c2490 con 0x7f2b940ce390 2023-10-16T14:29:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b9a4e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b941f1810 con 0x7f2b9415aaa0 2023-10-16T14:29:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b98a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 0x7f2b940af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b98a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 0x7f2b940af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57612/0 (socket says 172.21.15.29:57612) 2023-10-16T14:29:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b98a82640 1 -- 172.21.15.29:0/1786908664 learned_addr learned my addr 172.21.15.29:0/1786908664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.669+0000 7f2b93fff640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b940ce390 0x7f2b940c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b8b7fe640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9415aaa0 0x7f2b941f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b98a82640 1 -- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9415aaa0 msgr2=0x7f2b941f12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b98a82640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9415aaa0 0x7f2b941f12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b98a82640 1 -- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b940ce390 msgr2=0x7f2b940c2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b98a82640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b940ce390 0x7f2b940c2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b98a82640 1 -- 172.21.15.29:0/1786908664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c0094d0 con 0x7f2b9415ae80 2023-10-16T14:29:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b98a82640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 0x7f2b940af3b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0090b0 tx=0x7f2b8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.670+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c039c90 con 0x7f2b9415ae80 2023-10-16T14:29:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.671+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c039df0 con 0x7f2b9415ae80 2023-10-16T14:29:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.671+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9415be40 con 0x7f2b9415ae80 2023-10-16T14:29:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.671+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c043640 con 0x7f2b9415ae80 2023-10-16T14:29:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.671+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9415c0b0 con 0x7f2b9415ae80 2023-10-16T14:29:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.672+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b940c2a40 con 0x7f2b9415ae80 2023-10-16T14:29:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.673+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f2b8c0437a0 con 0x7f2b9415ae80 2023-10-16T14:29:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.674+0000 7f2b917fa640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 0x7f2b7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.674+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2b8c03e070 con 0x7f2b9415ae80 2023-10-16T14:29:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.674+0000 7f2b93fff640 1 -- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 msgr2=0x7f2b7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.674+0000 7f2b93fff640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 0x7f2b7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.682+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b8c0778d0 con 0x7f2b9415ae80 2023-10-16T14:29:55.799 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:55.796+0000 7fb39f413040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:29:55.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.875+0000 7f2b93fff640 1 -- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 msgr2=0x7f2b7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:55.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.875+0000 7f2b93fff640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 0x7f2b7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:29:55.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.913+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b9415c2a0 con 0x7f2b9415ae80 2023-10-16T14:29:55.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.914+0000 7f2b917fa640 1 -- 172.21.15.29:0/1786908664 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f2b8c034090 con 0x7f2b9415ae80 2023-10-16T14:29:55.917 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:55.917 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:17:40.632013+0000","modified":"2023-10-16T14:17:43.716037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4720},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4720":{"gid":4720,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6835/2287219370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2287219370},{"type":"v1","addr":"172.21.15.29:6835","nonce":2287219370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T14:29:55.917 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-16T14:29:55.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 msgr2=0x7f2b7406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:55.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 0x7f2b7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 msgr2=0x7f2b940af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:55.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 0x7f2b940af3b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0090b0 tx=0x7f2b8c004730 comp rx=0 tx=0).stop 2023-10-16T14:29:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 shutdown_connections 2023-10-16T14:29:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f2b7406b020 0x7f2b7406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9415ae80 0x7f2b940af3b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9415aaa0 0x7f2b941f12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 --2- 172.21.15.29:0/1786908664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b940ce390 0x7f2b940c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 >> 172.21.15.29:0/1786908664 conn(0x7f2b940cb410 msgr2=0x7f2b94070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 shutdown_connections 2023-10-16T14:29:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:55.916+0000 7f2b9a4e5640 1 -- 172.21.15.29:0/1786908664 wait complete. 2023-10-16T14:29:55.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:29:56.090 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.087+0000 7fb39f413040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:29:56.123 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:56.120+0000 7efcbc6b6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:29:56.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.153+0000 7f415a154640 1 Processor -- start 2023-10-16T14:29:56.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.153+0000 7f415a154640 1 -- start start 2023-10-16T14:29:56.156 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.153+0000 7fb39f413040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:29:56.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.154+0000 7f415a154640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41541573e0 0x7f41540ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.154+0000 7f415a154640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4154070950 0x7f415415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.154+0000 7f415a154640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540ca3a0 con 0x7f4154159e70 2023-10-16T14:29:56.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.154+0000 7f415a154640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41540cad70 con 0x7f4154070950 2023-10-16T14:29:56.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.154+0000 7f415a154640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c9d70 con 0x7f41541573e0 2023-10-16T14:29:56.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4158951640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4154159e70 0x7f4154070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43642/0 (socket says 172.21.15.29:43642) 2023-10-16T14:29:56.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4158951640 1 -- 172.21.15.29:0/3128672712 learned_addr learned my addr 172.21.15.29:0/3128672712 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:56.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4159152640 1 --2- 172.21.15.29:0/3128672712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41541573e0 0x7f41540ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4159953640 1 --2- 172.21.15.29:0/3128672712 >> v2:172.21.15.170:3300/0 conn(0x7f4154070950 0x7f415415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f414bfff640 1 -- 172.21.15.29:0/3128672712 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791582376 0 0) 0x7f41540ca3a0 con 0x7f4154159e70 2023-10-16T14:29:56.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f414bfff640 1 -- 172.21.15.29:0/3128672712 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140003520 con 0x7f4154159e70 2023-10-16T14:29:56.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4159152640 1 -- 172.21.15.29:0/3128672712 >> v2:172.21.15.170:3300/0 conn(0x7f4154070950 msgr2=0x7f415415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:56.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4159152640 1 --2- 172.21.15.29:0/3128672712 >> v2:172.21.15.170:3300/0 conn(0x7f4154070950 0x7f415415a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f4159152640 1 -- 172.21.15.29:0/3128672712 >> v1:172.21.15.29:6789/0 conn(0x7f4154159e70 legacy=0x7f4154070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.155+0000 7f414bfff640 1 -- 172.21.15.29:0/3128672712 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452740324 0 0) 0x7f4140003520 con 0x7f4154159e70 2023-10-16T14:29:56.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.156+0000 7f4159152640 1 -- 172.21.15.29:0/3128672712 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41541577c0 con 0x7f41541573e0 2023-10-16T14:29:56.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.157+0000 7f4159152640 1 --2- 172.21.15.29:0/3128672712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41541573e0 0x7f41540ca830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41440097a0 tx=0x7f414402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=275f0f5c06573047 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:56.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.157+0000 7f414bfff640 1 -- 172.21.15.29:0/3128672712 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150001670 con 0x7f41541573e0 2023-10-16T14:29:56.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.157+0000 7f414bfff640 1 -- 172.21.15.29:0/3128672712 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414402e050 con 0x7f41541573e0 2023-10-16T14:29:56.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.157+0000 7f414bfff640 1 -- 172.21.15.29:0/3128672712 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144030710 con 0x7f41541573e0 2023-10-16T14:29:56.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 -- 172.21.15.29:0/3128672712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41541573e0 msgr2=0x7f41540ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 --2- 172.21.15.29:0/3128672712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41541573e0 0x7f41540ca830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41440097a0 tx=0x7f414402b790 comp rx=0 tx=0).stop 2023-10-16T14:29:56.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 -- 172.21.15.29:0/3128672712 shutdown_connections 2023-10-16T14:29:56.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 --2- 172.21.15.29:0/3128672712 >> v2:172.21.15.170:3300/0 conn(0x7f4154070950 0x7f415415a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 --2- 172.21.15.29:0/3128672712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41541573e0 0x7f41540ca830 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 -- 172.21.15.29:0/3128672712 >> 172.21.15.29:0/3128672712 conn(0x7f41540cc230 msgr2=0x7f41540cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:56.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 -- 172.21.15.29:0/3128672712 shutdown_connections 2023-10-16T14:29:56.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.158+0000 7f415a154640 1 -- 172.21.15.29:0/3128672712 wait complete. 2023-10-16T14:29:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.160+0000 7f415a154640 1 Processor -- start 2023-10-16T14:29:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.160+0000 7f415a154640 1 -- start start 2023-10-16T14:29:56.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f415a154640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154070950 0x7f41541f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f415a154640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 0x7f41541544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f415a154640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154159e70 0x7f4154153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f415a154640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154154a10 con 0x7f41541573e0 2023-10-16T14:29:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f415a154640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541542b0 con 0x7f4154159e70 2023-10-16T14:29:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f415a154640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4154154c50 con 0x7f4154070950 2023-10-16T14:29:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f4158951640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 0x7f41541544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f4158951640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 0x7f41541544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57624/0 (socket says 172.21.15.29:57624) 2023-10-16T14:29:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f4158951640 1 -- 172.21.15.29:0/128712592 learned_addr learned my addr 172.21.15.29:0/128712592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f4159152640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154070950 0x7f41541f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.161+0000 7f4159953640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154159e70 0x7f4154153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4158951640 1 -- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154070950 msgr2=0x7f41541f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4158951640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154070950 0x7f41541f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4158951640 1 -- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154159e70 msgr2=0x7f4154153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4158951640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154159e70 0x7f4154153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4158951640 1 -- 172.21.15.29:0/128712592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4144009400 con 0x7f41541573e0 2023-10-16T14:29:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4159953640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154159e70 0x7f4154153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.162+0000 7f4158951640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 0x7f41541544d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f415000c750 tx=0x7f415000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.163+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150005ad0 con 0x7f41541573e0 2023-10-16T14:29:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.163+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4150005c30 con 0x7f41541573e0 2023-10-16T14:29:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.163+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150007070 con 0x7f41541573e0 2023-10-16T14:29:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.163+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4154154f30 con 0x7f41541573e0 2023-10-16T14:29:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.163+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41541f3890 con 0x7f41541573e0 2023-10-16T14:29:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.165+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41541f3ab0 con 0x7f41541573e0 2023-10-16T14:29:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.169+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f41500071d0 con 0x7f41541573e0 2023-10-16T14:29:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.171+0000 7f4149ffb640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f413806b070 0x7f413806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.171+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6279+0+0 (secure 0 0 0) 0x7f4150089730 con 0x7f41541573e0 2023-10-16T14:29:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.171+0000 7f4159152640 1 -- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f413806b070 msgr2=0x7f413806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1418556441 2023-10-16T14:29:56.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.171+0000 7f4159152640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f413806b070 0x7f413806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:29:56.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.173+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41500543b0 con 0x7f41541573e0 2023-10-16T14:29:56.275 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.272+0000 7fb39f413040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:29:56.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.290+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4150054780 con 0x7f41541573e0 2023-10-16T14:29:56.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.290+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f413806b070 msgr2=0x7f413806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:29:56.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.290+0000 7f4149ffb640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f413806b070 0x7f413806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.367 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.363+0000 7fb320c41640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.367 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.363+0000 7fb320c41640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.367 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.363+0000 7fb320c41640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.367 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.363+0000 7fb320c41640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.367 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.363+0000 7fb320c41640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.368 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.364+0000 7fb322c45640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.368 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.364+0000 7fb322c45640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.368 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.364+0000 7fb322c45640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.369 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.364+0000 7fb322c45640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.369 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:29:56.364+0000 7fb322c45640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:29:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.386+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f415404e9b0 con 0x7f41541573e0 2023-10-16T14:29:56.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.388+0000 7f4149ffb640 1 -- 172.21.15.29:0/128712592 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17955 (secure 0 0 0) 0x7f4150077170 con 0x7f41541573e0 2023-10-16T14:29:56.392 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:56.392 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":52,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:29:56.276597+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/728555731":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6832/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6833/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3736763821":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3139659877":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/2403070055":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2826870214":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/1855634044":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6832/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6833/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2300081340":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3069016888":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/176252655":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/1408092989":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3530276997":"2023-10-17T14:29:56.276500+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":4},{"begin":8,"length":1},{"begin":19,"length":1},{"begin":33,"length":1},{"begin":36,"length":2},{"begin":40,"length":1},{"begin":55,"length":1},{"begin":64,"length":1},{"begin":73,"length":1},{"begin":80,"length":1},{"begin":95,"length":1},{"begin":103,"length":1},{"begin":108,"length":1},{"begin":112,"length":1},{"begin":114,"length":4},{"begin":222,"length":12}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:29:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.390+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 msgr2=0x7f41541544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.390+0000 7f415a154640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 0x7f41541544d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f415000c750 tx=0x7f415000cc10 comp rx=0 tx=0).stop 2023-10-16T14:29:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 shutdown_connections 2023-10-16T14:29:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:6832/1418556441,v1:172.21.15.170:6833/1418556441] conn(0x7f413806b070 0x7f413806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41541573e0 0x7f41541544d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4154159e70 0x7f4154153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 --2- 172.21.15.29:0/128712592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154070950 0x7f41541f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 >> 172.21.15.29:0/128712592 conn(0x7f41540cc230 msgr2=0x7f4154069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 shutdown_connections 2023-10-16T14:29:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.391+0000 7f415a154640 1 -- 172.21.15.29:0/128712592 wait complete. 2023-10-16T14:29:56.410 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:17:38.728376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:17:39.652475+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 233, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:29:56.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:29:56.472 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:56.469+0000 7efcbc6b6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:29:56.534 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:56.531+0000 7efcbc6b6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:29:56.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.604+0000 7fcf8d32e640 1 Processor -- start 2023-10-16T14:29:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.604+0000 7fcf8d32e640 1 -- start start 2023-10-16T14:29:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.605+0000 7fcf8d32e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf880b1ca0 0x7fcf880d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.605+0000 7fcf8d32e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.605+0000 7fcf8d32e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880d0970 con 0x7fcf880b2080 2023-10-16T14:29:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.605+0000 7fcf8d32e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf880d0ea0 con 0x7fcf880b1ca0 2023-10-16T14:29:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.605+0000 7fcf8d32e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880d0bc0 con 0x7fcf880cfc40 2023-10-16T14:29:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf880b1ca0 0x7fcf880d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf880b1ca0 0x7fcf880d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57640/0 (socket says 172.21.15.29:57640) 2023-10-16T14:29:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 -- 172.21.15.29:0/635457778 learned_addr learned my addr 172.21.15.29:0/635457778 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf8cb2d640 1 --2- 172.21.15.29:0/635457778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 -- 172.21.15.29:0/635457778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 msgr2=0x7fcf880b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 --2- 172.21.15.29:0/635457778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 -- 172.21.15.29:0/635457778 >> v1:172.21.15.170:6789/0 conn(0x7fcf880b2080 legacy=0x7fcf880ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.606+0000 7fcf87fff640 1 -- 172.21.15.29:0/635457778 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880cd940 con 0x7fcf880b1ca0 2023-10-16T14:29:56.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.607+0000 7fcf87fff640 1 --2- 172.21.15.29:0/635457778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b1ca0 0x7fcf880d36b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009870 tx=0x7fcf7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5bc0d3d03495d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:56.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.608+0000 7fcf877fe640 1 -- 172.21.15.29:0/635457778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78030840 con 0x7fcf880b1ca0 2023-10-16T14:29:56.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.608+0000 7fcf877fe640 1 -- 172.21.15.29:0/635457778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf780309a0 con 0x7fcf880b1ca0 2023-10-16T14:29:56.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.608+0000 7fcf877fe640 1 -- 172.21.15.29:0/635457778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78030d30 con 0x7fcf880b1ca0 2023-10-16T14:29:56.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.608+0000 7fcf8d32e640 1 -- 172.21.15.29:0/635457778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b1ca0 msgr2=0x7fcf880d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.608+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/635457778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b1ca0 0x7fcf880d36b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009870 tx=0x7fcf7802b980 comp rx=0 tx=0).stop 2023-10-16T14:29:56.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.609+0000 7fcf8d32e640 1 -- 172.21.15.29:0/635457778 shutdown_connections 2023-10-16T14:29:56.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.609+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/635457778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.609+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/635457778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b1ca0 0x7fcf880d36b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.609+0000 7fcf8d32e640 1 -- 172.21.15.29:0/635457778 >> 172.21.15.29:0/635457778 conn(0x7fcf880d2b90 msgr2=0x7fcf880d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:56.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.609+0000 7fcf8d32e640 1 -- 172.21.15.29:0/635457778 shutdown_connections 2023-10-16T14:29:56.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.609+0000 7fcf8d32e640 1 -- 172.21.15.29:0/635457778 wait complete. 2023-10-16T14:29:56.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.610+0000 7fcf8d32e640 1 Processor -- start 2023-10-16T14:29:56.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.611+0000 7fcf8d32e640 1 -- start start 2023-10-16T14:29:56.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf8d32e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf880b1ca0 0x7fcf8809fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf8d32e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 0x7fcf881f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf8d32e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:56.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf8d32e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881f3440 con 0x7fcf880b2080 2023-10-16T14:29:56.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf8d32e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880c66c0 con 0x7fcf880b1ca0 2023-10-16T14:29:56.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf8d32e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880c6910 con 0x7fcf880cfc40 2023-10-16T14:29:56.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 0x7fcf881f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 0x7fcf881f2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57646/0 (socket says 172.21.15.29:57646) 2023-10-16T14:29:56.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 -- 172.21.15.29:0/1140263734 learned_addr learned my addr 172.21.15.29:0/1140263734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf87fff640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf880b1ca0 0x7fcf8809fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:56.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 -- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 msgr2=0x7fcf880b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 -- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf880b1ca0 msgr2=0x7fcf8809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf880b1ca0 0x7fcf8809fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf7ffff640 1 -- 172.21.15.29:0/1140263734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf780094d0 con 0x7fcf880b2080 2023-10-16T14:29:56.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.612+0000 7fcf87fff640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf880b1ca0 0x7fcf8809fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.613+0000 7fcf7ffff640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 0x7fcf881f2f00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000b3f0 tx=0x7fcf7000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:56.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.613+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70011830 con 0x7fcf880b2080 2023-10-16T14:29:56.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.613+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70011990 con 0x7fcf880b2080 2023-10-16T14:29:56.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.613+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70011d40 con 0x7fcf880b2080 2023-10-16T14:29:56.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.613+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf880c6bf0 con 0x7fcf880b2080 2023-10-16T14:29:56.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.613+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf880c71b0 con 0x7fcf880b2080 2023-10-16T14:29:56.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.614+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8804e9b0 con 0x7fcf880b2080 2023-10-16T14:29:56.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.615+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fcf7001a400 con 0x7fcf880b2080 2023-10-16T14:29:56.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.617+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7fcf70089370 con 0x7fcf880b2080 2023-10-16T14:29:56.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.621+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf700543f0 con 0x7fcf880b2080 2023-10-16T14:29:56.665 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:29:56.653+0000 7efcbc6b6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:29:56.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.812+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf8815f760 con 0x7fcf880b2080 2023-10-16T14:29:56.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.813+0000 7fcf857fa640 1 -- 172.21.15.29:0/1140263734 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf70057a10 con 0x7fcf880b2080 2023-10-16T14:29:56.816 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:29:56.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.815+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 msgr2=0x7fcf881f2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:56.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.815+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 0x7fcf881f2f00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000b3f0 tx=0x7fcf7000b8b0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 shutdown_connections 2023-10-16T14:29:56.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880b2080 0x7fcf881f2f00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880cfc40 0x7fcf880b7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 --2- 172.21.15.29:0/1140263734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf880b1ca0 0x7fcf8809fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:56.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 >> 172.21.15.29:0/1140263734 conn(0x7fcf880d2b90 msgr2=0x7fcf88160060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:56.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 shutdown_connections 2023-10-16T14:29:56.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:56.816+0000 7fcf8d32e640 1 -- 172.21.15.29:0/1140263734 wait complete. 2023-10-16T14:29:56.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:29:57.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.061+0000 7fec47fff640 1 Processor -- start 2023-10-16T14:29:57.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.061+0000 7fec47fff640 1 -- start start 2023-10-16T14:29:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec47fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec4815dde0 0x7fec480709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec47fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48160a90 con 0x7fec4815c9f0 2023-10-16T14:29:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec47fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec480cd900 con 0x7fec48070f20 2023-10-16T14:29:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec47fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec48071350 con 0x7fec4815dde0 2023-10-16T14:29:57.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec46ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fec4815c9f0 0x7fec480c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43654/0 (socket says 172.21.15.29:43654) 2023-10-16T14:29:57.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec46ffd640 1 -- 172.21.15.29:0/1800451762 learned_addr learned my addr 172.21.15.29:0/1800451762 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:57.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.063+0000 7fec467fc640 1 --2- 172.21.15.29:0/1800451762 >> v2:172.21.15.29:3300/0 conn(0x7fec4815dde0 0x7fec480709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.064+0000 7fec467fc640 1 -- 172.21.15.29:0/1800451762 >> v1:172.21.15.170:6789/0 conn(0x7fec48070f20 legacy=0x7fec480ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:57.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.064+0000 7fec467fc640 1 -- 172.21.15.29:0/1800451762 >> v1:172.21.15.29:6789/0 conn(0x7fec4815c9f0 legacy=0x7fec480c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.064+0000 7fec467fc640 1 -- 172.21.15.29:0/1800451762 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec480cdbe0 con 0x7fec4815dde0 2023-10-16T14:29:57.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.065+0000 7fec467fc640 1 --2- 172.21.15.29:0/1800451762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815dde0 0x7fec480709e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0097a0 tx=0x7fec3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6710970b4817fc24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.065+0000 7fec45ffb640 1 -- 172.21.15.29:0/1800451762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c031830 con 0x7fec4815dde0 2023-10-16T14:29:57.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.065+0000 7fec45ffb640 1 -- 172.21.15.29:0/1800451762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c031990 con 0x7fec4815dde0 2023-10-16T14:29:57.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.065+0000 7fec45ffb640 1 -- 172.21.15.29:0/1800451762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c031d20 con 0x7fec4815dde0 2023-10-16T14:29:57.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 -- 172.21.15.29:0/1800451762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815dde0 msgr2=0x7fec480709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 --2- 172.21.15.29:0/1800451762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815dde0 0x7fec480709e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0097a0 tx=0x7fec3c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 -- 172.21.15.29:0/1800451762 shutdown_connections 2023-10-16T14:29:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 --2- 172.21.15.29:0/1800451762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815dde0 0x7fec480709e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 -- 172.21.15.29:0/1800451762 >> 172.21.15.29:0/1800451762 conn(0x7fec480ce6a0 msgr2=0x7fec480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 -- 172.21.15.29:0/1800451762 shutdown_connections 2023-10-16T14:29:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.066+0000 7fec47fff640 1 -- 172.21.15.29:0/1800451762 wait complete. 2023-10-16T14:29:57.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.068+0000 7fec47fff640 1 Processor -- start 2023-10-16T14:29:57.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.068+0000 7fec47fff640 1 -- start start 2023-10-16T14:29:57.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec47fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec48070f20 0x7fec4806c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 0x7fec4806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec47fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4815dde0 0x7fec48150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec47fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec480c74e0 con 0x7fec4815c9f0 2023-10-16T14:29:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec47fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec480c77c0 con 0x7fec4815dde0 2023-10-16T14:29:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec47fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec480c7aa0 con 0x7fec48070f20 2023-10-16T14:29:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec467fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 0x7fec4806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec467fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 0x7fec4806a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57670/0 (socket says 172.21.15.29:57670) 2023-10-16T14:29:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.069+0000 7fec467fc640 1 -- 172.21.15.29:0/1980360862 learned_addr learned my addr 172.21.15.29:0/1980360862 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec46ffd640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec48070f20 0x7fec4806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec477fe640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4815dde0 0x7fec48150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec467fc640 1 -- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec48070f20 msgr2=0x7fec4806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec467fc640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec48070f20 0x7fec4806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec467fc640 1 -- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4815dde0 msgr2=0x7fec48150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec467fc640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4815dde0 0x7fec48150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec467fc640 1 -- 172.21.15.29:0/1980360862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c009400 con 0x7fec4815c9f0 2023-10-16T14:29:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec477fe640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4815dde0 0x7fec48150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec46ffd640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec48070f20 0x7fec4806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec467fc640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 0x7fec4806a900 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0098c0 tx=0x7fec3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c03ba70 con 0x7fec4815c9f0 2023-10-16T14:29:57.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c03bbf0 con 0x7fec4815c9f0 2023-10-16T14:29:57.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.070+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0453f0 con 0x7fec4815c9f0 2023-10-16T14:29:57.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.071+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec480c7d80 con 0x7fec4815c9f0 2023-10-16T14:29:57.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.071+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec480c7190 con 0x7fec4815c9f0 2023-10-16T14:29:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.072+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec480c82a0 con 0x7fec4815c9f0 2023-10-16T14:29:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.073+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fec3c04e450 con 0x7fec4815c9f0 2023-10-16T14:29:57.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.074+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7fec3c0ab7e0 con 0x7fec4815c9f0 2023-10-16T14:29:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.078+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec480c82a0 con 0x7fec4815c9f0 2023-10-16T14:29:57.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.272+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec480c82a0 con 0x7fec4815c9f0 2023-10-16T14:29:57.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.274+0000 7fec2f7fe640 1 -- 172.21.15.29:0/1980360862 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec3c035020 con 0x7fec4815c9f0 2023-10-16T14:29:57.277 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:29:57.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 msgr2=0x7fec4806a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 0x7fec4806a900 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0098c0 tx=0x7fec3c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 shutdown_connections 2023-10-16T14:29:57.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec4815c9f0 0x7fec4806a900 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4815dde0 0x7fec48150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 --2- 172.21.15.29:0/1980360862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec48070f20 0x7fec4806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.276+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 >> 172.21.15.29:0/1980360862 conn(0x7fec480ce6a0 msgr2=0x7fec4815bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.277+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 shutdown_connections 2023-10-16T14:29:57.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.277+0000 7fec47fff640 1 -- 172.21.15.29:0/1980360862 wait complete. 2023-10-16T14:29:57.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:29:57.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.527+0000 7f4ea1840640 1 Processor -- start 2023-10-16T14:29:57.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.528+0000 7f4ea1840640 1 -- start start 2023-10-16T14:29:57.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.529+0000 7f4ea1840640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e9c0ad970 0x7f4e9c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.529+0000 7f4ea1840640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c15b9d0 con 0x7f4e9c06c040 2023-10-16T14:29:57.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.529+0000 7f4ea1840640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c15bcb0 con 0x7f4e9c0ccfb0 2023-10-16T14:29:57.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.529+0000 7f4ea1840640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9c15bf90 con 0x7f4e9c0ad970 2023-10-16T14:29:57.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4ea083e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4e9c06c040 0x7f4e9c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43656/0 (socket says 172.21.15.29:43656) 2023-10-16T14:29:57.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4ea083e640 1 -- 172.21.15.29:0/2004683823 learned_addr learned my addr 172.21.15.29:0/2004683823 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:57.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4e9bfff640 1 --2- 172.21.15.29:0/2004683823 >> v2:172.21.15.29:3300/0 conn(0x7f4e9c0ad970 0x7f4e9c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4e9b7fe640 1 -- 172.21.15.29:0/2004683823 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89233161 0 0) 0x7f4e9c15b9d0 con 0x7f4e9c06c040 2023-10-16T14:29:57.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4e9b7fe640 1 -- 172.21.15.29:0/2004683823 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c003520 con 0x7f4e9c06c040 2023-10-16T14:29:57.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4e9bfff640 1 -- 172.21.15.29:0/2004683823 >> v1:172.21.15.170:6789/0 conn(0x7f4e9c0ccfb0 legacy=0x7f4e9c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4e9bfff640 1 -- 172.21.15.29:0/2004683823 >> v1:172.21.15.29:6789/0 conn(0x7f4e9c06c040 legacy=0x7f4e9c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.530+0000 7f4e9bfff640 1 -- 172.21.15.29:0/2004683823 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c0cc270 con 0x7f4e9c0ad970 2023-10-16T14:29:57.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.532+0000 7f4e9bfff640 1 --2- 172.21.15.29:0/2004683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0cca70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097d0 tx=0x7f4e8802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1589b257b3269 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.532+0000 7f4e9b7fe640 1 -- 172.21.15.29:0/2004683823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88030800 con 0x7f4e9c0ad970 2023-10-16T14:29:57.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.532+0000 7f4e9b7fe640 1 -- 172.21.15.29:0/2004683823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e88030960 con 0x7f4e9c0ad970 2023-10-16T14:29:57.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.532+0000 7f4e9b7fe640 1 -- 172.21.15.29:0/2004683823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88030cf0 con 0x7f4e9c0ad970 2023-10-16T14:29:57.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 -- 172.21.15.29:0/2004683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 msgr2=0x7f4e9c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 --2- 172.21.15.29:0/2004683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0cca70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097d0 tx=0x7f4e8802b940 comp rx=0 tx=0).stop 2023-10-16T14:29:57.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 -- 172.21.15.29:0/2004683823 shutdown_connections 2023-10-16T14:29:57.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 --2- 172.21.15.29:0/2004683823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0cca70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 -- 172.21.15.29:0/2004683823 >> 172.21.15.29:0/2004683823 conn(0x7f4e9c0ce6a0 msgr2=0x7f4e9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:57.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 -- 172.21.15.29:0/2004683823 shutdown_connections 2023-10-16T14:29:57.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.533+0000 7f4ea1840640 1 -- 172.21.15.29:0/2004683823 wait complete. 2023-10-16T14:29:57.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.535+0000 7f4ea1840640 1 Processor -- start 2023-10-16T14:29:57.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.535+0000 7f4ea1840640 1 -- start start 2023-10-16T14:29:57.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4ea1840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e9c06c040 0x7f4e9c154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4ea1840640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4ea1840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e9c0ccfb0 0x7f4e9c1f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4ea1840640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0b2e80 con 0x7f4e9c0ad970 2023-10-16T14:29:57.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4ea1840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0b3160 con 0x7f4e9c0ccfb0 2023-10-16T14:29:57.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4ea1840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0b3440 con 0x7f4e9c06c040 2023-10-16T14:29:57.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4e9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4e9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57676/0 (socket says 172.21.15.29:57676) 2023-10-16T14:29:57.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.536+0000 7f4e9bfff640 1 -- 172.21.15.29:0/3051644586 learned_addr learned my addr 172.21.15.29:0/3051644586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:57.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e9bfff640 1 -- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e9c06c040 msgr2=0x7f4e9c154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e9bfff640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e9c06c040 0x7f4e9c154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e9bfff640 1 -- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e9c0ccfb0 msgr2=0x7f4e9c1f00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e9bfff640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e9c0ccfb0 0x7f4e9c1f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e9bfff640 1 -- 172.21.15.29:0/3051644586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88009400 con 0x7f4e9c0ad970 2023-10-16T14:29:57.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e9bfff640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0b2870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4e880098f0 tx=0x7f4e88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8803e040 con 0x7f4e9c0ad970 2023-10-16T14:29:57.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e88039cb0 con 0x7f4e9c0ad970 2023-10-16T14:29:57.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88044630 con 0x7f4e9c0ad970 2023-10-16T14:29:57.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c0b3720 con 0x7f4e9c0ad970 2023-10-16T14:29:57.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.537+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c1f05f0 con 0x7f4e9c0ad970 2023-10-16T14:29:57.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.539+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c06b330 con 0x7f4e9c0ad970 2023-10-16T14:29:57.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.539+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4e8804d3b0 con 0x7f4e9c0ad970 2023-10-16T14:29:57.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.540+0000 7f4e997fa640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4e8006acf0 0x7f4e8006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.541+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f4e880abc20 con 0x7f4e9c0ad970 2023-10-16T14:29:57.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.541+0000 7f4ea083e640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4e8006acf0 0x7f4e8006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.542+0000 7f4ea083e640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4e8006acf0 0x7f4e8006d1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c1f3d00 tx=0x7f4e90007930 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.545+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e88076c80 con 0x7f4e9c0ad970 2023-10-16T14:29:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.739+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4e9c04e9b0 con 0x7f4e9c0ad970 2023-10-16T14:29:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.740+0000 7f4e997fa640 1 -- 172.21.15.29:0/3051644586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4e88034020 con 0x7f4e9c0ad970 2023-10-16T14:29:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:29:57.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.742+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4e8006acf0 msgr2=0x7f4e8006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.742+0000 7f4ea1840640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4e8006acf0 0x7f4e8006d1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c1f3d00 tx=0x7f4e90007930 comp rx=0 tx=0).stop 2023-10-16T14:29:57.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.742+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 msgr2=0x7f4e9c0b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.742+0000 7f4ea1840640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0b2870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4e880098f0 tx=0x7f4e88004730 comp rx=0 tx=0).stop 2023-10-16T14:29:57.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 shutdown_connections 2023-10-16T14:29:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4e8006acf0 0x7f4e8006d1b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c0ad970 0x7f4e9c0b2870 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e9c0ccfb0 0x7f4e9c1f00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 --2- 172.21.15.29:0/3051644586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e9c06c040 0x7f4e9c154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 >> 172.21.15.29:0/3051644586 conn(0x7f4e9c0ce6a0 msgr2=0x7f4e9c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:57.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 shutdown_connections 2023-10-16T14:29:57.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.743+0000 7f4ea1840640 1 -- 172.21.15.29:0/3051644586 wait complete. 2023-10-16T14:29:57.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:29:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.982+0000 7ffbefbd0640 1 Processor -- start 2023-10-16T14:29:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.983+0000 7ffbefbd0640 1 -- start start 2023-10-16T14:29:57.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.984+0000 7ffbefbd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffbe80ae220 0x7ffbe80cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.984+0000 7ffbefbd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffbe8160f30 0x7ffbe8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.984+0000 7ffbefbd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe806a730 con 0x7ffbe806a2e0 2023-10-16T14:29:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.984+0000 7ffbefbd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe80cf760 con 0x7ffbe8160f30 2023-10-16T14:29:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.984+0000 7ffbefbd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe81614f0 con 0x7ffbe80ae220 2023-10-16T14:29:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.985+0000 7ffbee16d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffbe806a2e0 0x7ffbe80705d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43660/0 (socket says 172.21.15.29:43660) 2023-10-16T14:29:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.985+0000 7ffbee16d640 1 -- 172.21.15.29:0/2704099691 learned_addr learned my addr 172.21.15.29:0/2704099691 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.985+0000 7ffbed16b640 1 --2- 172.21.15.29:0/2704099691 >> v2:172.21.15.29:3300/0 conn(0x7ffbe8160f30 0x7ffbe8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.985+0000 7ffbed96c640 1 --2- 172.21.15.29:0/2704099691 >> v2:172.21.15.170:3300/0 conn(0x7ffbe80ae220 0x7ffbe80cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.985+0000 7ffbec96a640 1 -- 172.21.15.29:0/2704099691 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078557993 0 0) 0x7ffbe0001490 con 0x7ffbe806a2e0 2023-10-16T14:29:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.985+0000 7ffbec96a640 1 -- 172.21.15.29:0/2704099691 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd4003520 con 0x7ffbe806a2e0 2023-10-16T14:29:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.986+0000 7ffbed16b640 1 -- 172.21.15.29:0/2704099691 >> v2:172.21.15.170:3300/0 conn(0x7ffbe80ae220 msgr2=0x7ffbe80cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.986+0000 7ffbed16b640 1 --2- 172.21.15.29:0/2704099691 >> v2:172.21.15.170:3300/0 conn(0x7ffbe80ae220 0x7ffbe80cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.986+0000 7ffbed16b640 1 -- 172.21.15.29:0/2704099691 >> v1:172.21.15.29:6789/0 conn(0x7ffbe806a2e0 legacy=0x7ffbe80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.986+0000 7ffbed16b640 1 -- 172.21.15.29:0/2704099691 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe81617d0 con 0x7ffbe8160f30 2023-10-16T14:29:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.987+0000 7ffbed16b640 1 --2- 172.21.15.29:0/2704099691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe8160f30 0x7ffbe8069da0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffbe806ae60 tx=0x7ffbd802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101ff79784f06a7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.987+0000 7ffbec96a640 1 -- 172.21.15.29:0/2704099691 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd802e070 con 0x7ffbe8160f30 2023-10-16T14:29:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.987+0000 7ffbec96a640 1 -- 172.21.15.29:0/2704099691 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd80315a0 con 0x7ffbe8160f30 2023-10-16T14:29:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.987+0000 7ffbec96a640 1 -- 172.21.15.29:0/2704099691 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8031930 con 0x7ffbe8160f30 2023-10-16T14:29:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2704099691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe8160f30 msgr2=0x7ffbe8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2704099691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe8160f30 0x7ffbe8069da0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffbe806ae60 tx=0x7ffbd802b840 comp rx=0 tx=0).stop 2023-10-16T14:29:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2704099691 shutdown_connections 2023-10-16T14:29:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2704099691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe8160f30 0x7ffbe8069da0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2704099691 >> v2:172.21.15.170:3300/0 conn(0x7ffbe80ae220 0x7ffbe80cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2704099691 >> 172.21.15.29:0/2704099691 conn(0x7ffbe80cfb70 msgr2=0x7ffbe80cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2704099691 shutdown_connections 2023-10-16T14:29:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.988+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2704099691 wait complete. 2023-10-16T14:29:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.990+0000 7ffbefbd0640 1 Processor -- start 2023-10-16T14:29:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.990+0000 7ffbefbd0640 1 -- start start 2023-10-16T14:29:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbefbd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 0x7ffbe814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbefbd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbe80ae220 0x7ffbe80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbefbd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbe8160f30 0x7ffbe806e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbefbd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe814d5b0 con 0x7ffbe806a2e0 2023-10-16T14:29:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbefbd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c7fc0 con 0x7ffbe80ae220 2023-10-16T14:29:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbefbd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80acdd0 con 0x7ffbe8160f30 2023-10-16T14:29:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 0x7ffbe814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 0x7ffbe814d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57698/0 (socket says 172.21.15.29:57698) 2023-10-16T14:29:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 -- 172.21.15.29:0/2048455128 learned_addr learned my addr 172.21.15.29:0/2048455128 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 -- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbe8160f30 msgr2=0x7ffbe806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbe8160f30 0x7ffbe806e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 -- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbe80ae220 msgr2=0x7ffbe80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbe80ae220 0x7ffbe80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.991+0000 7ffbed96c640 1 -- 172.21.15.29:0/2048455128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd8009400 con 0x7ffbe806a2e0 2023-10-16T14:29:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.992+0000 7ffbed96c640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 0x7ffbe814d070 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400e870 tx=0x7ffbe400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.992+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe40094f0 con 0x7ffbe806a2e0 2023-10-16T14:29:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.992+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe400c940 con 0x7ffbe806a2e0 2023-10-16T14:29:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.992+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400ccf0 con 0x7ffbe806a2e0 2023-10-16T14:29:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.992+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe80c6c40 con 0x7ffbe806a2e0 2023-10-16T14:29:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.992+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe80c7200 con 0x7ffbe806a2e0 2023-10-16T14:29:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.993+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe804e160 con 0x7ffbe806a2e0 2023-10-16T14:29:58.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.994+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ffbe4013470 con 0x7ffbe806a2e0 2023-10-16T14:29:58.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.995+0000 7ffbde7fc640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ffbcc06acc0 0x7ffbcc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.995+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7ffbe40897b0 con 0x7ffbe806a2e0 2023-10-16T14:29:58.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.996+0000 7ffbed16b640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ffbcc06acc0 0x7ffbcc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:57.997+0000 7ffbed16b640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ffbcc06acc0 0x7ffbcc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8036000 tx=0x7ffbd8030000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.000+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffbe4054740 con 0x7ffbe806a2e0 2023-10-16T14:29:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.229+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe80b5af0 con 0x7ffbe806a2e0 2023-10-16T14:29:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.230+0000 7ffbde7fc640 1 -- 172.21.15.29:0/2048455128 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7ffbe4057d60 con 0x7ffbe806a2e0 2023-10-16T14:29:58.235 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:58.235 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:17:40.632013+0000","modified":"2023-10-16T14:17:43.716037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4720},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4720":{"gid":4720,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6835/2287219370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2287219370},{"type":"v1","addr":"172.21.15.29:6835","nonce":2287219370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T14:29:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-16T14:29:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ffbcc06acc0 msgr2=0x7ffbcc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ffbcc06acc0 0x7ffbcc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8036000 tx=0x7ffbd8030000 comp rx=0 tx=0).stop 2023-10-16T14:29:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 msgr2=0x7ffbe814d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 0x7ffbe814d070 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400e870 tx=0x7ffbe400ed30 comp rx=0 tx=0).stop 2023-10-16T14:29:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 shutdown_connections 2023-10-16T14:29:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ffbcc06acc0 0x7ffbcc06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbe8160f30 0x7ffbe806e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbe80ae220 0x7ffbe80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 --2- 172.21.15.29:0/2048455128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe806a2e0 0x7ffbe814d070 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 >> 172.21.15.29:0/2048455128 conn(0x7ffbe80cfb70 msgr2=0x7ffbe80ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 shutdown_connections 2023-10-16T14:29:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.233+0000 7ffbefbd0640 1 -- 172.21.15.29:0/2048455128 wait complete. 2023-10-16T14:29:58.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:29:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.462+0000 7ff86131d640 1 Processor -- start 2023-10-16T14:29:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.463+0000 7ff86131d640 1 -- start start 2023-10-16T14:29:58.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.465+0000 7ff86131d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 0x7ff85c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.465+0000 7ff86131d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff85c159f00 0x7ff85c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.465+0000 7ff86131d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c06f7f0 con 0x7ff85c06c120 2023-10-16T14:29:58.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.465+0000 7ff86131d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85c06c5e0 con 0x7ff85c159f00 2023-10-16T14:29:58.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.465+0000 7ff86131d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c06c8c0 con 0x7ff85c159b20 2023-10-16T14:29:58.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.466+0000 7ff85affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 0x7ff85c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.466+0000 7ff85affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 0x7ff85c06bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:36216/0 (socket says 172.21.15.29:36216) 2023-10-16T14:29:58.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.466+0000 7ff85affd640 1 -- 172.21.15.29:0/1975838172 learned_addr learned my addr 172.21.15.29:0/1975838172 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:58.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.466+0000 7ff853fff640 1 --2- 172.21.15.29:0/1975838172 >> v2:172.21.15.170:3300/0 conn(0x7ff85c159f00 0x7ff85c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.467+0000 7ff85affd640 1 -- 172.21.15.29:0/1975838172 >> v2:172.21.15.170:3300/0 conn(0x7ff85c159f00 msgr2=0x7ff85c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:58.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.467+0000 7ff85affd640 1 --2- 172.21.15.29:0/1975838172 >> v2:172.21.15.170:3300/0 conn(0x7ff85c159f00 0x7ff85c0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.467+0000 7ff85affd640 1 -- 172.21.15.29:0/1975838172 >> v1:172.21.15.170:6789/0 conn(0x7ff85c06c120 legacy=0x7ff85c0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.467+0000 7ff85affd640 1 -- 172.21.15.29:0/1975838172 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c06cba0 con 0x7ff85c159b20 2023-10-16T14:29:58.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.467+0000 7ff85a7fc640 1 -- 172.21.15.29:0/1975838172 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209645795 0 0) 0x7ff85c06f7f0 con 0x7ff85c06c120 2023-10-16T14:29:58.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.468+0000 7ff85affd640 1 --2- 172.21.15.29:0/1975838172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 0x7ff85c06bbe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009870 tx=0x7ff84c02db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d04d17482935f8f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.468+0000 7ff85a7fc640 1 -- 172.21.15.29:0/1975838172 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c02e940 con 0x7ff85c159b20 2023-10-16T14:29:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.468+0000 7ff85a7fc640 1 -- 172.21.15.29:0/1975838172 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c02eaa0 con 0x7ff85c159b20 2023-10-16T14:29:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.468+0000 7ff85a7fc640 1 -- 172.21.15.29:0/1975838172 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c02ee30 con 0x7ff85c159b20 2023-10-16T14:29:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 -- 172.21.15.29:0/1975838172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 msgr2=0x7ff85c06bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 --2- 172.21.15.29:0/1975838172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 0x7ff85c06bbe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009870 tx=0x7ff84c02db90 comp rx=0 tx=0).stop 2023-10-16T14:29:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 -- 172.21.15.29:0/1975838172 shutdown_connections 2023-10-16T14:29:58.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 --2- 172.21.15.29:0/1975838172 >> v2:172.21.15.170:3300/0 conn(0x7ff85c159f00 0x7ff85c0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 --2- 172.21.15.29:0/1975838172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159b20 0x7ff85c06bbe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 -- 172.21.15.29:0/1975838172 >> 172.21.15.29:0/1975838172 conn(0x7ff85c0cb410 msgr2=0x7ff85c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:58.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 -- 172.21.15.29:0/1975838172 shutdown_connections 2023-10-16T14:29:58.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.469+0000 7ff86131d640 1 -- 172.21.15.29:0/1975838172 wait complete. 2023-10-16T14:29:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.471+0000 7ff86131d640 1 Processor -- start 2023-10-16T14:29:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.471+0000 7ff86131d640 1 -- start start 2023-10-16T14:29:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff86131d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 0x7ff85c0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff86131d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff85c159b20 0x7ff85c151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff86131d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159f00 0x7ff85c070220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff86131d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c151a70 con 0x7ff85c06c120 2023-10-16T14:29:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff86131d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0b4160 con 0x7ff85c159b20 2023-10-16T14:29:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff86131d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c15a3b0 con 0x7ff85c159f00 2023-10-16T14:29:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff85affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 0x7ff85c0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff85affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 0x7ff85c0c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57704/0 (socket says 172.21.15.29:57704) 2023-10-16T14:29:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.472+0000 7ff85affd640 1 -- 172.21.15.29:0/24071414 learned_addr learned my addr 172.21.15.29:0/24071414 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff853fff640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff85c159b20 0x7ff85c151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85b7fe640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159f00 0x7ff85c070220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85affd640 1 -- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159f00 msgr2=0x7ff85c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85affd640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159f00 0x7ff85c070220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85affd640 1 -- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff85c159b20 msgr2=0x7ff85c151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85affd640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff85c159b20 0x7ff85c151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85affd640 1 -- 172.21.15.29:0/24071414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c0094d0 con 0x7ff85c06c120 2023-10-16T14:29:58.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff853fff640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff85c159b20 0x7ff85c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:58.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff85affd640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 0x7ff85c0c5a60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009870 tx=0x7ff84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.473+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c041680 con 0x7ff85c06c120 2023-10-16T14:29:58.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.474+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c0417e0 con 0x7ff85c06c120 2023-10-16T14:29:58.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.474+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c041b90 con 0x7ff85c06c120 2023-10-16T14:29:58.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.474+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c15a690 con 0x7ff85c06c120 2023-10-16T14:29:58.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.474+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c15ac50 con 0x7ff85c06c120 2023-10-16T14:29:58.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.475+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c04e160 con 0x7ff85c06c120 2023-10-16T14:29:58.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.476+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff84c041cf0 con 0x7ff85c06c120 2023-10-16T14:29:58.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.477+0000 7ff8537fe640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff83c06acc0 0x7ff83c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.477+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7ff84c0ac320 con 0x7ff85c06c120 2023-10-16T14:29:58.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.477+0000 7ff853fff640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff83c06acc0 0x7ff83c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.478+0000 7ff853fff640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff83c06acc0 0x7ff83c06d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0b05e0 tx=0x7ff844008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.481+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff84c0773d0 con 0x7ff85c06c120 2023-10-16T14:29:58.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.674+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85c0c9610 con 0x7ff85c06c120 2023-10-16T14:29:58.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.677+0000 7ff8537fe640 1 -- 172.21.15.29:0/24071414 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17955 (secure 0 0 0) 0x7ff84c07a9f0 con 0x7ff85c06c120 2023-10-16T14:29:58.680 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:58.681 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":52,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:29:56.276597+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/728555731":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6832/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6833/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3736763821":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3139659877":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/2403070055":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2826870214":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/1855634044":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6832/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6833/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2300081340":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3069016888":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/176252655":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/1408092989":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3530276997":"2023-10-17T14:29:56.276500+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":4},{"begin":8,"length":1},{"begin":19,"length":1},{"begin":33,"length":1},{"begin":36,"length":2},{"begin":40,"length":1},{"begin":55,"length":1},{"begin":64,"length":1},{"begin":73,"length":1},{"begin":80,"length":1},{"begin":95,"length":1},{"begin":103,"length":1},{"begin":108,"length":1},{"begin":112,"length":1},{"begin":114,"length":4},{"begin":222,"length":12}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:29:58.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff83c06acc0 msgr2=0x7ff83c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff83c06acc0 0x7ff83c06d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0b05e0 tx=0x7ff844008040 comp rx=0 tx=0).stop 2023-10-16T14:29:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 msgr2=0x7ff85c0c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 0x7ff85c0c5a60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009870 tx=0x7ff84c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 shutdown_connections 2023-10-16T14:29:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff83c06acc0 0x7ff83c06d180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff85c159f00 0x7ff85c070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff85c159b20 0x7ff85c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 --2- 172.21.15.29:0/24071414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff85c06c120 0x7ff85c0c5a60 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 >> 172.21.15.29:0/24071414 conn(0x7ff85c0cb410 msgr2=0x7ff85c0cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 shutdown_connections 2023-10-16T14:29:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.680+0000 7ff86131d640 1 -- 172.21.15.29:0/24071414 wait complete. 2023-10-16T14:29:58.701 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:29:58.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:29:58.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.927+0000 7f093ab0a640 1 Processor -- start 2023-10-16T14:29:58.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.927+0000 7f093ab0a640 1 -- start start 2023-10-16T14:29:58.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f093ab0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09340ad970 0x7f09340cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f093ab0a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f09340ccfb0 0x7f09340cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f093ab0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093415b9d0 con 0x7f093406c040 2023-10-16T14:29:58.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f093ab0a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093415bcb0 con 0x7f09340ad970 2023-10-16T14:29:58.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f093ab0a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093415bf90 con 0x7f09340ccfb0 2023-10-16T14:29:58.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f0939b08640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f093406c040 0x7f093406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43676/0 (socket says 172.21.15.29:43676) 2023-10-16T14:29:58.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f0939b08640 1 -- 172.21.15.29:0/3172480956 learned_addr learned my addr 172.21.15.29:0/3172480956 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:58.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.929+0000 7f0939307640 1 --2- 172.21.15.29:0/3172480956 >> v2:172.21.15.29:3300/0 conn(0x7f09340ad970 0x7f09340cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f093a309640 1 --2- 172.21.15.29:0/3172480956 >> v2:172.21.15.170:3300/0 conn(0x7f09340ccfb0 0x7f09340cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f0938b06640 1 -- 172.21.15.29:0/3172480956 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584067031 0 0) 0x7f093415b9d0 con 0x7f093406c040 2023-10-16T14:29:58.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f0939307640 1 -- 172.21.15.29:0/3172480956 >> v2:172.21.15.170:3300/0 conn(0x7f09340ccfb0 msgr2=0x7f09340cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:58.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f0939307640 1 --2- 172.21.15.29:0/3172480956 >> v2:172.21.15.170:3300/0 conn(0x7f09340ccfb0 0x7f09340cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f0939307640 1 -- 172.21.15.29:0/3172480956 >> v1:172.21.15.29:6789/0 conn(0x7f093406c040 legacy=0x7f093406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f0939307640 1 -- 172.21.15.29:0/3172480956 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340cc270 con 0x7f09340ad970 2023-10-16T14:29:58.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.930+0000 7f093a309640 1 --2- 172.21.15.29:0/3172480956 >> v2:172.21.15.170:3300/0 conn(0x7f09340ccfb0 0x7f09340cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:58.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.931+0000 7f0939307640 1 --2- 172.21.15.29:0/3172480956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09340ad970 0x7f09340cca70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f09240097a0 tx=0x7f092402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db67020721f7836 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.931+0000 7f0938b06640 1 -- 172.21.15.29:0/3172480956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924031830 con 0x7f09340ad970 2023-10-16T14:29:58.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.931+0000 7f0938b06640 1 -- 172.21.15.29:0/3172480956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924031990 con 0x7f09340ad970 2023-10-16T14:29:58.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.931+0000 7f0938b06640 1 -- 172.21.15.29:0/3172480956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924031d20 con 0x7f09340ad970 2023-10-16T14:29:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 -- 172.21.15.29:0/3172480956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09340ad970 msgr2=0x7f09340cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3172480956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09340ad970 0x7f09340cca70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f09240097a0 tx=0x7f092402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 -- 172.21.15.29:0/3172480956 shutdown_connections 2023-10-16T14:29:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3172480956 >> v2:172.21.15.170:3300/0 conn(0x7f09340ccfb0 0x7f09340cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3172480956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09340ad970 0x7f09340cca70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 -- 172.21.15.29:0/3172480956 >> 172.21.15.29:0/3172480956 conn(0x7f09340ce6a0 msgr2=0x7f09340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 -- 172.21.15.29:0/3172480956 shutdown_connections 2023-10-16T14:29:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.932+0000 7f093ab0a640 1 -- 172.21.15.29:0/3172480956 wait complete. 2023-10-16T14:29:58.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.934+0000 7f093ab0a640 1 Processor -- start 2023-10-16T14:29:58.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.934+0000 7f093ab0a640 1 -- start start 2023-10-16T14:29:58.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f093ab0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 0x7f09340aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f093ab0a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09340ad970 0x7f093414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f093ab0a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09340ccfb0 0x7f09340afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f093ab0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09340c7350 con 0x7f093406c040 2023-10-16T14:29:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f093ab0a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09340adf10 con 0x7f09340ad970 2023-10-16T14:29:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f093ab0a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09340ae1f0 con 0x7f09340ccfb0 2023-10-16T14:29:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 0x7f09340aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 0x7f09340aecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57730/0 (socket says 172.21.15.29:57730) 2023-10-16T14:29:58.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 -- 172.21.15.29:0/3306627946 learned_addr learned my addr 172.21.15.29:0/3306627946 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:58.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 -- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09340ccfb0 msgr2=0x7f09340afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09340ccfb0 0x7f09340afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 -- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09340ad970 msgr2=0x7f093414eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09340ad970 0x7f093414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.935+0000 7f0939b08640 1 -- 172.21.15.29:0/3306627946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924009400 con 0x7f093406c040 2023-10-16T14:29:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.936+0000 7f0939b08640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 0x7f09340aecd0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f093000c4d0 tx=0x7f093000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.936+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09300074e0 con 0x7f093406c040 2023-10-16T14:29:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.936+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930007660 con 0x7f093406c040 2023-10-16T14:29:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.936+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930004a20 con 0x7f093406c040 2023-10-16T14:29:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.936+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09340ae4d0 con 0x7f093406c040 2023-10-16T14:29:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.936+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09340affe0 con 0x7f093406c040 2023-10-16T14:29:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.937+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093406b330 con 0x7f093406c040 2023-10-16T14:29:58.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.938+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0930004b80 con 0x7f093406c040 2023-10-16T14:29:58.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.939+0000 7f092a7fc640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f091806acc0 0x7f091806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:58.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.939+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f0930088760 con 0x7f093406c040 2023-10-16T14:29:58.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.940+0000 7f0939307640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f091806acc0 0x7f091806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:58.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.941+0000 7f0939307640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f091806acc0 0x7f091806d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0924036000 tx=0x7f0924030000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:58.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:58.944+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f09300537c0 con 0x7f093406c040 2023-10-16T14:29:59.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.170+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f0930053b90 con 0x7f093406c040 2023-10-16T14:29:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.178+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f093404e9b0 con 0x7f093406c040 2023-10-16T14:29:59.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.179+0000 7f092a7fc640 1 -- 172.21.15.29:0/3306627946 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f09300733e0 con 0x7f093406c040 2023-10-16T14:29:59.182 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:59.182 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:17:40.632013+0000","modified":"2023-10-16T14:17:43.716037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4720},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4720":{"gid":4720,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6835/2287219370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2287219370},{"type":"v1","addr":"172.21.15.29:6835","nonce":2287219370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T14:29:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-16T14:29:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f091806acc0 msgr2=0x7f091806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f091806acc0 0x7f091806d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0924036000 tx=0x7f0924030000 comp rx=0 tx=0).stop 2023-10-16T14:29:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 msgr2=0x7f09340aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 0x7f09340aecd0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f093000c4d0 tx=0x7f093000c990 comp rx=0 tx=0).stop 2023-10-16T14:29:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 shutdown_connections 2023-10-16T14:29:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f091806acc0 0x7f091806d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09340ccfb0 0x7f09340afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09340ad970 0x7f093414eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 --2- 172.21.15.29:0/3306627946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f093406c040 0x7f09340aecd0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 >> 172.21.15.29:0/3306627946 conn(0x7f09340ce6a0 msgr2=0x7f093415a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:59.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 shutdown_connections 2023-10-16T14:29:59.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.182+0000 7f093ab0a640 1 -- 172.21.15.29:0/3306627946 wait complete. 2023-10-16T14:29:59.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:29:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.420+0000 7f6d29d4c640 1 Processor -- start 2023-10-16T14:29:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.420+0000 7f6d29d4c640 1 -- start start 2023-10-16T14:29:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.421+0000 7f6d29d4c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d241593b0 0x7f6d2406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.421+0000 7f6d29d4c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6d240cc920 0x7f6d240ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.421+0000 7f6d29d4c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d24159790 con 0x7f6d240caa10 2023-10-16T14:29:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.421+0000 7f6d29d4c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d240c9b20 con 0x7f6d240cc920 2023-10-16T14:29:59.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.421+0000 7f6d29d4c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d240c9e00 con 0x7f6d241593b0 2023-10-16T14:29:59.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.422+0000 7f6d237fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d241593b0 0x7f6d2406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.422+0000 7f6d1bfff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f6d240caa10 0x7f6d2406fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:39896/0 (socket says 172.21.15.29:39896) 2023-10-16T14:29:59.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.422+0000 7f6d1bfff640 1 -- 172.21.15.29:0/1498833806 learned_addr learned my addr 172.21.15.29:0/1498833806 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:29:59.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.422+0000 7f6d23fff640 1 --2- 172.21.15.29:0/1498833806 >> v2:172.21.15.170:3300/0 conn(0x7f6d240cc920 0x7f6d240ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.423+0000 7f6d22ffd640 1 -- 172.21.15.29:0/1498833806 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673469530 0 0) 0x7f6d24159790 con 0x7f6d240caa10 2023-10-16T14:29:59.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.423+0000 7f6d22ffd640 1 -- 172.21.15.29:0/1498833806 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d10003520 con 0x7f6d240caa10 2023-10-16T14:29:59.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.423+0000 7f6d237fe640 1 -- 172.21.15.29:0/1498833806 >> v2:172.21.15.170:3300/0 conn(0x7f6d240cc920 msgr2=0x7f6d240ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:59.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.423+0000 7f6d237fe640 1 --2- 172.21.15.29:0/1498833806 >> v2:172.21.15.170:3300/0 conn(0x7f6d240cc920 0x7f6d240ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.423+0000 7f6d237fe640 1 -- 172.21.15.29:0/1498833806 >> v1:172.21.15.170:6789/0 conn(0x7f6d240caa10 legacy=0x7f6d2406fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.423+0000 7f6d237fe640 1 -- 172.21.15.29:0/1498833806 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d240ca0e0 con 0x7f6d241593b0 2023-10-16T14:29:59.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.425+0000 7f6d237fe640 1 --2- 172.21.15.29:0/1498833806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d241593b0 0x7f6d2406f550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d140097a0 tx=0x7f6d1402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c31b8588fb4d0d39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.426+0000 7f6d22ffd640 1 -- 172.21.15.29:0/1498833806 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1402f050 con 0x7f6d241593b0 2023-10-16T14:29:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.426+0000 7f6d22ffd640 1 -- 172.21.15.29:0/1498833806 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14031700 con 0x7f6d241593b0 2023-10-16T14:29:59.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.426+0000 7f6d22ffd640 1 -- 172.21.15.29:0/1498833806 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14031a90 con 0x7f6d241593b0 2023-10-16T14:29:59.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.426+0000 7f6d29d4c640 1 -- 172.21.15.29:0/1498833806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d241593b0 msgr2=0x7f6d2406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.426+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/1498833806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d241593b0 0x7f6d2406f550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d140097a0 tx=0x7f6d1402b7a0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.427+0000 7f6d29d4c640 1 -- 172.21.15.29:0/1498833806 shutdown_connections 2023-10-16T14:29:59.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.427+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/1498833806 >> v2:172.21.15.170:3300/0 conn(0x7f6d240cc920 0x7f6d240ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.427+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/1498833806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d241593b0 0x7f6d2406f550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.427+0000 7f6d29d4c640 1 -- 172.21.15.29:0/1498833806 >> 172.21.15.29:0/1498833806 conn(0x7f6d240cb410 msgr2=0x7f6d240cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:59.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.427+0000 7f6d29d4c640 1 -- 172.21.15.29:0/1498833806 shutdown_connections 2023-10-16T14:29:59.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.427+0000 7f6d29d4c640 1 -- 172.21.15.29:0/1498833806 wait complete. 2023-10-16T14:29:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.428+0000 7f6d29d4c640 1 Processor -- start 2023-10-16T14:29:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.429+0000 7f6d29d4c640 1 -- start start 2023-10-16T14:29:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d29d4c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d240caa10 0x7f6d24147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d29d4c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d240cc920 0x7f6d2415d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d29d4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 0x7f6d2415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d29d4c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d24148fc0 con 0x7f6d241593b0 2023-10-16T14:29:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d29d4c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2414d1b0 con 0x7f6d240cc920 2023-10-16T14:29:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d29d4c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2414d490 con 0x7f6d240caa10 2023-10-16T14:29:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 0x7f6d2415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 0x7f6d2415d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57738/0 (socket says 172.21.15.29:57738) 2023-10-16T14:29:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d23fff640 1 -- 172.21.15.29:0/2981195402 learned_addr learned my addr 172.21.15.29:0/2981195402 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d1bfff640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d240cc920 0x7f6d2415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.430+0000 7f6d237fe640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d240caa10 0x7f6d24147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d23fff640 1 -- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d240caa10 msgr2=0x7f6d24147970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d23fff640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d240caa10 0x7f6d24147970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d23fff640 1 -- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d240cc920 msgr2=0x7f6d2415d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d23fff640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d240cc920 0x7f6d2415d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d23fff640 1 -- 172.21.15.29:0/2981195402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d14009400 con 0x7f6d241593b0 2023-10-16T14:29:59.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d1bfff640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d240cc920 0x7f6d2415d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:29:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d23fff640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 0x7f6d2415d870 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00efc0 tx=0x7f6d1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c014480 con 0x7f6d241593b0 2023-10-16T14:29:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c00f040 con 0x7f6d241593b0 2023-10-16T14:29:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c014820 con 0x7f6d241593b0 2023-10-16T14:29:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.431+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2414d770 con 0x7f6d241593b0 2023-10-16T14:29:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.432+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d24147eb0 con 0x7f6d241593b0 2023-10-16T14:29:59.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.433+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2414d990 con 0x7f6d241593b0 2023-10-16T14:29:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.434+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6d1c014980 con 0x7f6d241593b0 2023-10-16T14:29:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.435+0000 7f6d20ff9640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6d0406af00 0x7f6d0406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.435+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f6d1c08a800 con 0x7f6d241593b0 2023-10-16T14:29:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.436+0000 7f6d237fe640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6d0406af00 0x7f6d0406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.437+0000 7f6d237fe640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6d0406af00 0x7f6d0406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d14035000 tx=0x7f6d14008030 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:59.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.442+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d1c055320 con 0x7f6d241593b0 2023-10-16T14:29:59.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.675+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d2404e160 con 0x7f6d241593b0 2023-10-16T14:29:59.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.676+0000 7f6d20ff9640 1 -- 172.21.15.29:0/2981195402 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f6d1c058940 con 0x7f6d241593b0 2023-10-16T14:29:59.679 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:29:59.679 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:17:40.632013+0000","modified":"2023-10-16T14:17:43.716037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4720},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4720":{"gid":4720,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6835/2287219370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2287219370},{"type":"v1","addr":"172.21.15.29:6835","nonce":2287219370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T14:29:59.679 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-16T14:29:59.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6d0406af00 msgr2=0x7f6d0406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6d0406af00 0x7f6d0406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d14035000 tx=0x7f6d14008030 comp rx=0 tx=0).stop 2023-10-16T14:29:59.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 msgr2=0x7f6d2415d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 0x7f6d2415d870 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00efc0 tx=0x7f6d1c00c490 comp rx=0 tx=0).stop 2023-10-16T14:29:59.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 shutdown_connections 2023-10-16T14:29:59.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6d0406af00 0x7f6d0406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d241593b0 0x7f6d2415d870 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d240cc920 0x7f6d2415d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 --2- 172.21.15.29:0/2981195402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d240caa10 0x7f6d24147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 >> 172.21.15.29:0/2981195402 conn(0x7f6d240cb410 msgr2=0x7f6d2406cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:59.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.678+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 shutdown_connections 2023-10-16T14:29:59.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.679+0000 7f6d29d4c640 1 -- 172.21.15.29:0/2981195402 wait complete. 2023-10-16T14:29:59.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:29:59.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.929+0000 7fbe3a7ae640 1 Processor -- start 2023-10-16T14:29:59.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.929+0000 7fbe3a7ae640 1 -- start start 2023-10-16T14:29:59.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe3a7ae640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe3415cd70 0x7fbe340d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe3a7ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbe340b8480 0x7fbe340cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe3a7ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe340d0be0 0x7fbe340cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe3a7ae640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe3415d150 con 0x7fbe3415cd70 2023-10-16T14:29:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe3a7ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe340d1180 con 0x7fbe340b8480 2023-10-16T14:29:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe3a7ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe340d1460 con 0x7fbe340d0be0 2023-10-16T14:29:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe397ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe3415cd70 0x7fbe340d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe397ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe3415cd70 0x7fbe340d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57740/0 (socket says 172.21.15.29:57740) 2023-10-16T14:29:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.931+0000 7fbe397ac640 1 -- 172.21.15.29:0/2943367652 learned_addr learned my addr 172.21.15.29:0/2943367652 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe397ac640 1 -- 172.21.15.29:0/2943367652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe340d0be0 msgr2=0x7fbe340cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe397ac640 1 --2- 172.21.15.29:0/2943367652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe340d0be0 0x7fbe340cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe397ac640 1 -- 172.21.15.29:0/2943367652 >> v2:172.21.15.170:3300/0 conn(0x7fbe340b8480 msgr2=0x7fbe340cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:29:59.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe38fab640 1 --2- 172.21.15.29:0/2943367652 >> v2:172.21.15.170:3300/0 conn(0x7fbe340b8480 0x7fbe340cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe397ac640 1 --2- 172.21.15.29:0/2943367652 >> v2:172.21.15.170:3300/0 conn(0x7fbe340b8480 0x7fbe340cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe397ac640 1 -- 172.21.15.29:0/2943367652 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe340d1740 con 0x7fbe3415cd70 2023-10-16T14:29:59.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.932+0000 7fbe38fab640 1 --2- 172.21.15.29:0/2943367652 >> v2:172.21.15.170:3300/0 conn(0x7fbe340b8480 0x7fbe340cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.933+0000 7fbe397ac640 1 --2- 172.21.15.29:0/2943367652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe3415cd70 0x7fbe340d2220 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbe34066a00 tx=0x7fbe3002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2b6d169365cc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.933+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2943367652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30031860 con 0x7fbe3415cd70 2023-10-16T14:29:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.933+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2943367652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe300319c0 con 0x7fbe3415cd70 2023-10-16T14:29:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.933+0000 7fbe2bfff640 1 -- 172.21.15.29:0/2943367652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30031d50 con 0x7fbe3415cd70 2023-10-16T14:29:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/2943367652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe3415cd70 msgr2=0x7fbe340d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/2943367652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe3415cd70 0x7fbe340d2220 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbe34066a00 tx=0x7fbe3002b900 comp rx=0 tx=0).stop 2023-10-16T14:29:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/2943367652 shutdown_connections 2023-10-16T14:29:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/2943367652 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe340d0be0 0x7fbe340cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/2943367652 >> v2:172.21.15.170:3300/0 conn(0x7fbe340b8480 0x7fbe340cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/2943367652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe3415cd70 0x7fbe340d2220 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/2943367652 >> 172.21.15.29:0/2943367652 conn(0x7fbe340cf570 msgr2=0x7fbe340cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:29:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/2943367652 shutdown_connections 2023-10-16T14:29:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.934+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/2943367652 wait complete. 2023-10-16T14:29:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.936+0000 7fbe3a7ae640 1 Processor -- start 2023-10-16T14:29:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.936+0000 7fbe3a7ae640 1 -- start start 2023-10-16T14:29:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe3a7ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe340b8480 0x7fbe340c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe3a7ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 0x7fbe3415f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe3a7ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe3415cd70 0x7fbe3415fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe3a7ae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3415ede0 con 0x7fbe340d0be0 2023-10-16T14:29:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe3a7ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3415e1d0 con 0x7fbe340b8480 2023-10-16T14:29:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe3a7ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3415e4b0 con 0x7fbe3415cd70 2023-10-16T14:29:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe38fab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 0x7fbe3415f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe38fab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 0x7fbe3415f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57746/0 (socket says 172.21.15.29:57746) 2023-10-16T14:29:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.937+0000 7fbe38fab640 1 -- 172.21.15.29:0/3112614184 learned_addr learned my addr 172.21.15.29:0/3112614184 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:29:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe397ac640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe340b8480 0x7fbe340c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe38fab640 1 -- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe3415cd70 msgr2=0x7fbe3415fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe38fab640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe3415cd70 0x7fbe3415fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe38fab640 1 -- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe340b8480 msgr2=0x7fbe340c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:29:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe38fab640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe340b8480 0x7fbe340c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:29:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe38fab640 1 -- 172.21.15.29:0/3112614184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300094d0 con 0x7fbe340d0be0 2023-10-16T14:29:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe397ac640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe340b8480 0x7fbe340c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:29:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe38fab640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 0x7fbe3415f760 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c7f0 tx=0x7fbe2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24007820 con 0x7fbe340d0be0 2023-10-16T14:29:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe24007980 con 0x7fbe340d0be0 2023-10-16T14:29:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24007cf0 con 0x7fbe340d0be0 2023-10-16T14:29:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3415e790 con 0x7fbe340d0be0 2023-10-16T14:29:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.938+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3415d4a0 con 0x7fbe340d0be0 2023-10-16T14:29:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.940+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe34153540 con 0x7fbe340d0be0 2023-10-16T14:29:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.941+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbe240113f0 con 0x7fbe340d0be0 2023-10-16T14:29:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.942+0000 7fbe29ffb640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fbe1806af30 0x7fbe1806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:29:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.942+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6622+0+0 (secure 0 0 0) 0x7fbe24089e80 con 0x7fbe340d0be0 2023-10-16T14:29:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.942+0000 7fbe397ac640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fbe1806af30 0x7fbe1806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:29:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.943+0000 7fbe397ac640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fbe1806af30 0x7fbe1806d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe30036000 tx=0x7fbe3003c040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:29:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:29:59.946+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe34153540 con 0x7fbe340d0be0 2023-10-16T14:30:00.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.147+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe34153520 con 0x7fbe340d0be0 2023-10-16T14:30:00.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.149+0000 7fbe29ffb640 1 -- 172.21.15.29:0/3112614184 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17955 (secure 0 0 0) 0x7fbe2401f090 con 0x7fbe340d0be0 2023-10-16T14:30:00.153 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:00.153 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":53,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:30:00.099977+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:17:38.728376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T14:17:39.652475+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/728555731":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6832/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6833/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3736763821":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3139659877":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/2403070055":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2826870214":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/1855634044":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6832/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6833/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2300081340":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3069016888":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/176252655":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/1408092989":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3530276997":"2023-10-17T14:29:56.276500+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":4},{"begin":8,"length":1},{"begin":19,"length":1},{"begin":33,"length":1},{"begin":36,"length":2},{"begin":40,"length":1},{"begin":55,"length":1},{"begin":64,"length":1},{"begin":73,"length":1},{"begin":80,"length":1},{"begin":95,"length":1},{"begin":103,"length":1},{"begin":108,"length":1},{"begin":112,"length":1},{"begin":114,"length":4},{"begin":222,"length":12}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:30:00.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fbe1806af30 msgr2=0x7fbe1806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fbe1806af30 0x7fbe1806d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe30036000 tx=0x7fbe3003c040 comp rx=0 tx=0).stop 2023-10-16T14:30:00.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 msgr2=0x7fbe3415f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 0x7fbe3415f760 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c7f0 tx=0x7fbe2400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 shutdown_connections 2023-10-16T14:30:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fbe1806af30 0x7fbe1806d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe340d0be0 0x7fbe3415f760 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe3415cd70 0x7fbe3415fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 --2- 172.21.15.29:0/3112614184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe340b8480 0x7fbe340c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 >> 172.21.15.29:0/3112614184 conn(0x7fbe340cf570 msgr2=0x7fbe340b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:00.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.152+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 shutdown_connections 2023-10-16T14:30:00.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.153+0000 7fbe3a7ae640 1 -- 172.21.15.29:0/3112614184 wait complete. 2023-10-16T14:30:00.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:30:00.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.399+0000 7f8b844b4640 1 Processor -- start 2023-10-16T14:30:00.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.399+0000 7f8b844b4640 1 -- start start 2023-10-16T14:30:00.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.400+0000 7f8b844b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b7c160530 0x7f8b7c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.400+0000 7f8b844b4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b7c065ab0 0x7f8b7c160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.400+0000 7f8b844b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c161470 0x7f8b7c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.400+0000 7f8b844b4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7c0ac890 con 0x7f8b7c160530 2023-10-16T14:30:00.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.400+0000 7f8b844b4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7c0ae800 con 0x7f8b7c065ab0 2023-10-16T14:30:00.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.400+0000 7f8b844b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c0ae1f0 con 0x7f8b7c161470 2023-10-16T14:30:00.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b7c160530 0x7f8b7c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b7c160530 0x7f8b7c1609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57758/0 (socket says 172.21.15.29:57758) 2023-10-16T14:30:00.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 -- 172.21.15.29:0/1461874248 learned_addr learned my addr 172.21.15.29:0/1461874248 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 -- 172.21.15.29:0/1461874248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c161470 msgr2=0x7f8b7c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 --2- 172.21.15.29:0/1461874248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c161470 0x7f8b7c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 -- 172.21.15.29:0/1461874248 >> v2:172.21.15.170:3300/0 conn(0x7f8b7c065ab0 msgr2=0x7f8b7c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 --2- 172.21.15.29:0/1461874248 >> v2:172.21.15.170:3300/0 conn(0x7f8b7c065ab0 0x7f8b7c160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.401+0000 7f8b82250640 1 -- 172.21.15.29:0/1461874248 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c0acae0 con 0x7f8b7c160530 2023-10-16T14:30:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.403+0000 7f8b82250640 1 --2- 172.21.15.29:0/1461874248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c160530 0x7f8b7c1609f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c06f640 tx=0x7f8b7802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772fe807f799bf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:00.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.403+0000 7f8b8124e640 1 -- 172.21.15.29:0/1461874248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b780317e0 con 0x7f8b7c160530 2023-10-16T14:30:00.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.403+0000 7f8b8124e640 1 -- 172.21.15.29:0/1461874248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78031940 con 0x7f8b7c160530 2023-10-16T14:30:00.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.403+0000 7f8b8124e640 1 -- 172.21.15.29:0/1461874248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78031cd0 con 0x7f8b7c160530 2023-10-16T14:30:00.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.403+0000 7f8b844b4640 1 -- 172.21.15.29:0/1461874248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c160530 msgr2=0x7f8b7c1609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.403+0000 7f8b844b4640 1 --2- 172.21.15.29:0/1461874248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c160530 0x7f8b7c1609f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c06f640 tx=0x7f8b7802b880 comp rx=0 tx=0).stop 2023-10-16T14:30:00.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 -- 172.21.15.29:0/1461874248 shutdown_connections 2023-10-16T14:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 --2- 172.21.15.29:0/1461874248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c161470 0x7f8b7c0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 --2- 172.21.15.29:0/1461874248 >> v2:172.21.15.170:3300/0 conn(0x7f8b7c065ab0 0x7f8b7c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 --2- 172.21.15.29:0/1461874248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c160530 0x7f8b7c1609f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 -- 172.21.15.29:0/1461874248 >> 172.21.15.29:0/1461874248 conn(0x7f8b7c06bc10 msgr2=0x7f8b7c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 -- 172.21.15.29:0/1461874248 shutdown_connections 2023-10-16T14:30:00.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.404+0000 7f8b844b4640 1 -- 172.21.15.29:0/1461874248 wait complete. 2023-10-16T14:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.406+0000 7f8b844b4640 1 Processor -- start 2023-10-16T14:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.406+0000 7f8b844b4640 1 -- start start 2023-10-16T14:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.407+0000 7f8b844b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 0x7f8b7c157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.407+0000 7f8b844b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c160530 0x7f8b7c0c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.407+0000 7f8b844b4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7c161470 0x7f8b7c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.407+0000 7f8b844b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c0b2000 con 0x7f8b7c065ab0 2023-10-16T14:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.407+0000 7f8b844b4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c0b2250 con 0x7f8b7c161470 2023-10-16T14:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.407+0000 7f8b844b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c0b2530 con 0x7f8b7c160530 2023-10-16T14:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 0x7f8b7c157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 0x7f8b7c157960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57768/0 (socket says 172.21.15.29:57768) 2023-10-16T14:30:00.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 -- 172.21.15.29:0/3463050798 learned_addr learned my addr 172.21.15.29:0/3463050798 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:00.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82a51640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7c161470 0x7f8b7c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b81a4f640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c160530 0x7f8b7c0c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 -- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c160530 msgr2=0x7f8b7c0c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c160530 0x7f8b7c0c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 -- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7c161470 msgr2=0x7f8b7c0b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7c161470 0x7f8b7c0b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82250640 1 -- 172.21.15.29:0/3463050798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780094d0 con 0x7f8b7c065ab0 2023-10-16T14:30:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b82a51640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7c161470 0x7f8b7c0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.408+0000 7f8b81a4f640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c160530 0x7f8b7c0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.409+0000 7f8b82250640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 0x7f8b7c157960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8b780097c0 tx=0x7f8b780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.409+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b780329c0 con 0x7f8b7c065ab0 2023-10-16T14:30:00.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.409+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78032b20 con 0x7f8b7c065ab0 2023-10-16T14:30:00.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.409+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b780453f0 con 0x7f8b7c065ab0 2023-10-16T14:30:00.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.409+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c0b2810 con 0x7f8b7c065ab0 2023-10-16T14:30:00.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.409+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c0b2dd0 con 0x7f8b7c065ab0 2023-10-16T14:30:00.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.410+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7c04e160 con 0x7f8b7c065ab0 2023-10-16T14:30:00.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.411+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8b7804d3f0 con 0x7f8b7c065ab0 2023-10-16T14:30:00.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.412+0000 7f8b72ffd640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f8b6006af00 0x7f8b6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.412+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f8b780abac0 con 0x7f8b7c065ab0 2023-10-16T14:30:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.413+0000 7f8b81a4f640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f8b6006af00 0x7f8b6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.414+0000 7f8b81a4f640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f8b6006af00 0x7f8b6006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c006fd0 tx=0x7f8b6c008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:00.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.417+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b78076690 con 0x7f8b7c065ab0 2023-10-16T14:30:00.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.648+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8b7c0b2670 con 0x7f8b7c065ab0 2023-10-16T14:30:00.651 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:30:00.649+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:30:00.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.652+0000 7f8b72ffd640 1 -- 172.21.15.29:0/3463050798 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8b7c0b2670 con 0x7f8b7c065ab0 2023-10-16T14:30:00.655 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:30:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.657+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f8b6006af00 msgr2=0x7f8b6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.657+0000 7f8b844b4640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f8b6006af00 0x7f8b6006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c006fd0 tx=0x7f8b6c008040 comp rx=0 tx=0).stop 2023-10-16T14:30:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.657+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 msgr2=0x7f8b7c157960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.657+0000 7f8b844b4640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 0x7f8b7c157960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8b780097c0 tx=0x7f8b780046c0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 shutdown_connections 2023-10-16T14:30:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f8b6006af00 0x7f8b6006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7c161470 0x7f8b7c0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7c160530 0x7f8b7c0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 --2- 172.21.15.29:0/3463050798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7c065ab0 0x7f8b7c157960 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 >> 172.21.15.29:0/3463050798 conn(0x7f8b7c06bc10 msgr2=0x7f8b7c15eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:00.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.658+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 shutdown_connections 2023-10-16T14:30:00.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.659+0000 7f8b844b4640 1 -- 172.21.15.29:0/3463050798 wait complete. 2023-10-16T14:30:00.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:30:00.755 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:30:00.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.880+0000 7f3e5c809640 1 Processor -- start 2023-10-16T14:30:00.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.881+0000 7f3e5c809640 1 -- start start 2023-10-16T14:30:00.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.882+0000 7f3e5c809640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3e5815ad70 0x7f3e58069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.882+0000 7f3e5c809640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580cb030 con 0x7f3e580caac0 2023-10-16T14:30:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.882+0000 7f3e5c809640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580c91c0 con 0x7f3e580c9ca0 2023-10-16T14:30:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.882+0000 7f3e5c809640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e580c94a0 con 0x7f3e5815ad70 2023-10-16T14:30:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.882+0000 7f3e56d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3e580caac0 0x7f3e580ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43698/0 (socket says 172.21.15.29:43698) 2023-10-16T14:30:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.882+0000 7f3e56d9d640 1 -- 172.21.15.29:0/2094474194 learned_addr learned my addr 172.21.15.29:0/2094474194 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422067903 0 0) 0x7f3e580cb030 con 0x7f3e580caac0 2023-10-16T14:30:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e44003520 con 0x7f3e580caac0 2023-10-16T14:30:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e5659c640 1 --2- 172.21.15.29:0/2094474194 >> v2:172.21.15.170:3300/0 conn(0x7f3e5815ad70 0x7f3e58069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815209604 0 0) 0x7f3e44003520 con 0x7f3e580caac0 2023-10-16T14:30:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 >> v2:172.21.15.170:3300/0 conn(0x7f3e5815ad70 msgr2=0x7f3e58069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 --2- 172.21.15.29:0/2094474194 >> v2:172.21.15.170:3300/0 conn(0x7f3e5815ad70 0x7f3e58069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 >> v1:172.21.15.170:6789/0 conn(0x7f3e580c9ca0 legacy=0x7f3e580ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580c9780 con 0x7f3e580caac0 2023-10-16T14:30:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3e500019f0 con 0x7f3e580caac0 2023-10-16T14:30:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 >> v1:172.21.15.29:6789/0 conn(0x7f3e580caac0 legacy=0x7f3e580ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e440039c0 0x7f3e44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.883+0000 7f3e55d9b640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e440047c0 0x7f3e44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e55d9b640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440051d0 0x7f3e4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580c94a0 con 0x7f3e440039c0 2023-10-16T14:30:00.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4400c120 con 0x7f3e440047c0 2023-10-16T14:30:00.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4400c2e0 con 0x7f3e440051d0 2023-10-16T14:30:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e5659c640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e440039c0 0x7f3e44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e4ffff640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e440047c0 0x7f3e44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e56d9d640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440051d0 0x7f3e4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e5659c640 1 -- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440051d0 msgr2=0x7f3e4400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:00.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e5659c640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440051d0 0x7f3e4400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e5659c640 1 -- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e440047c0 msgr2=0x7f3e44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:00.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e5659c640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e440047c0 0x7f3e44004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e5659c640 1 -- 172.21.15.29:0/2094474194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e44003520 con 0x7f3e440039c0 2023-10-16T14:30:00.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e56d9d640 1 -- 172.21.15.29:0/2094474194 reap_dead start 2023-10-16T14:30:00.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.884+0000 7f3e4ffff640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e440047c0 0x7f3e44004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:00.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.885+0000 7f3e5659c640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e440039c0 0x7f3e44003de0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800ba30 tx=0x7f3e440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264635d27dd6f0e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:00.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48032660 con 0x7f3e440039c0 2023-10-16T14:30:00.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e480327c0 con 0x7f3e440039c0 2023-10-16T14:30:00.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e55d9b640 1 -- 172.21.15.29:0/2094474194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48032b50 con 0x7f3e440039c0 2023-10-16T14:30:00.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e5c809640 1 -- 172.21.15.29:0/2094474194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e440039c0 msgr2=0x7f3e44003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e5c809640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e440039c0 0x7f3e44003de0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800ba30 tx=0x7f3e440057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e5c809640 1 -- 172.21.15.29:0/2094474194 shutdown_connections 2023-10-16T14:30:00.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e5c809640 1 --2- 172.21.15.29:0/2094474194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e440039c0 0x7f3e44003de0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.886+0000 7f3e5c809640 1 -- 172.21.15.29:0/2094474194 >> 172.21.15.29:0/2094474194 conn(0x7f3e580ce9b0 msgr2=0x7f3e580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.887+0000 7f3e5c809640 1 -- 172.21.15.29:0/2094474194 shutdown_connections 2023-10-16T14:30:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.887+0000 7f3e5c809640 1 -- 172.21.15.29:0/2094474194 wait complete. 2023-10-16T14:30:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.888+0000 7f3e5c809640 1 Processor -- start 2023-10-16T14:30:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.889+0000 7f3e5c809640 1 -- start start 2023-10-16T14:30:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5c809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440039c0 0x7f3e5815ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5c809640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 0x7f3e581f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5c809640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e58158330 0x7f3e580ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5c809640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581f2350 con 0x7f3e580add10 2023-10-16T14:30:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5c809640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580cc930 con 0x7f3e58158330 2023-10-16T14:30:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5c809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5815d490 con 0x7f3e440039c0 2023-10-16T14:30:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 0x7f3e581f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 0x7f3e581f1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57810/0 (socket says 172.21.15.29:57810) 2023-10-16T14:30:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e4ffff640 1 -- 172.21.15.29:0/1975119238 learned_addr learned my addr 172.21.15.29:0/1975119238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e5659c640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440039c0 0x7f3e5815ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.890+0000 7f3e56d9d640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e58158330 0x7f3e580ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4ffff640 1 -- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440039c0 msgr2=0x7f3e5815ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4ffff640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440039c0 0x7f3e5815ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4ffff640 1 -- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e58158330 msgr2=0x7f3e580ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4ffff640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e58158330 0x7f3e580ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4ffff640 1 -- 172.21.15.29:0/1975119238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5815d770 con 0x7f3e580add10 2023-10-16T14:30:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4ffff640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 0x7f3e581f1e10 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000d4a0 tx=0x7f3e4000d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40005480 con 0x7f3e580add10 2023-10-16T14:30:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e40007030 con 0x7f3e580add10 2023-10-16T14:30:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5815da50 con 0x7f3e580add10 2023-10-16T14:30:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40007400 con 0x7f3e580add10 2023-10-16T14:30:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.891+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e580ad550 con 0x7f3e580add10 2023-10-16T14:30:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.892+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580ccac0 con 0x7f3e580add10 2023-10-16T14:30:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.894+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3e40007560 con 0x7f3e580add10 2023-10-16T14:30:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.894+0000 7f3e4effd640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f3e3806af00 0x7f3e3806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.895+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3e40089290 con 0x7f3e580add10 2023-10-16T14:30:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.895+0000 7f3e5659c640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f3e3806af00 0x7f3e3806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.896+0000 7f3e5659c640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f3e3806af00 0x7f3e3806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800ba00 tx=0x7f3e4803d040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:00.899+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e40053ff0 con 0x7f3e580add10 2023-10-16T14:30:01.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.132+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3e5815df10 con 0x7f3e580add10 2023-10-16T14:30:01.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.191+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e400543c0 con 0x7f3e580add10 2023-10-16T14:30:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.656+0000 7f3e4effd640 1 -- 172.21.15.29:0/1975119238 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3e400715f0 con 0x7f3e580add10 2023-10-16T14:30:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f3e3806af00 msgr2=0x7f3e3806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f3e3806af00 0x7f3e3806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800ba00 tx=0x7f3e4803d040 comp rx=0 tx=0).stop 2023-10-16T14:30:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 msgr2=0x7f3e581f1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 0x7f3e581f1e10 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000d4a0 tx=0x7f3e4000d960 comp rx=0 tx=0).stop 2023-10-16T14:30:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 shutdown_connections 2023-10-16T14:30:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f3e3806af00 0x7f3e3806d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e580add10 0x7f3e581f1e10 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e58158330 0x7f3e580ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 --2- 172.21.15.29:0/1975119238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e440039c0 0x7f3e5815ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 >> 172.21.15.29:0/1975119238 conn(0x7f3e580ce9b0 msgr2=0x7f3e58159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 shutdown_connections 2023-10-16T14:30:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.660+0000 7f3e5c809640 1 -- 172.21.15.29:0/1975119238 wait complete. 2023-10-16T14:30:01.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:30:01.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.911+0000 7fc2c7fff640 1 Processor -- start 2023-10-16T14:30:01.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.911+0000 7fc2c7fff640 1 -- start start 2023-10-16T14:30:01.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.913+0000 7fc2c7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc2c815c9f0 0x7fc2c80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:01.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.913+0000 7fc2c7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:01.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.913+0000 7fc2c7fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c8160a90 con 0x7fc2c80c9230 2023-10-16T14:30:01.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.913+0000 7fc2c7fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c80cdec0 con 0x7fc2c815c9f0 2023-10-16T14:30:01.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.913+0000 7fc2c7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80cd900 con 0x7fc2c815dde0 2023-10-16T14:30:01.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.914+0000 7fc2c77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:01.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.914+0000 7fc2c77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:36318/0 (socket says 172.21.15.29:36318) 2023-10-16T14:30:01.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.914+0000 7fc2c77fe640 1 -- 172.21.15.29:0/2706597305 learned_addr learned my addr 172.21.15.29:0/2706597305 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:01.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.914+0000 7fc2c6ffd640 1 --2- 172.21.15.29:0/2706597305 >> v2:172.21.15.170:3300/0 conn(0x7fc2c815c9f0 0x7fc2c80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:01.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.915+0000 7fc2c77fe640 1 -- 172.21.15.29:0/2706597305 >> v2:172.21.15.170:3300/0 conn(0x7fc2c815c9f0 msgr2=0x7fc2c80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:01.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.915+0000 7fc2c77fe640 1 --2- 172.21.15.29:0/2706597305 >> v2:172.21.15.170:3300/0 conn(0x7fc2c815c9f0 0x7fc2c80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.915+0000 7fc2c77fe640 1 -- 172.21.15.29:0/2706597305 >> v1:172.21.15.170:6789/0 conn(0x7fc2c80c9230 legacy=0x7fc2c80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:01.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.915+0000 7fc2c77fe640 1 -- 172.21.15.29:0/2706597305 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c80cdbe0 con 0x7fc2c815dde0 2023-10-16T14:30:01.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.916+0000 7fc2c77fe640 1 --2- 172.21.15.29:0/2706597305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c80ac890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0009e50 tx=0x7fc2c002bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d14aaa5bb6e9ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:01.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.916+0000 7fc2c5ffb640 1 -- 172.21.15.29:0/2706597305 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c002caf0 con 0x7fc2c815dde0 2023-10-16T14:30:01.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c5ffb640 1 -- 172.21.15.29:0/2706597305 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c002cc50 con 0x7fc2c815dde0 2023-10-16T14:30:01.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c5ffb640 1 -- 172.21.15.29:0/2706597305 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00343f0 con 0x7fc2c815dde0 2023-10-16T14:30:01.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 -- 172.21.15.29:0/2706597305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 msgr2=0x7fc2c80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:01.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/2706597305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c80ac890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0009e50 tx=0x7fc2c002bf50 comp rx=0 tx=0).stop 2023-10-16T14:30:01.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 -- 172.21.15.29:0/2706597305 shutdown_connections 2023-10-16T14:30:01.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/2706597305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c80ac890 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/2706597305 >> v2:172.21.15.170:3300/0 conn(0x7fc2c815c9f0 0x7fc2c80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 -- 172.21.15.29:0/2706597305 >> 172.21.15.29:0/2706597305 conn(0x7fc2c80ce6a0 msgr2=0x7fc2c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:01.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.917+0000 7fc2c7fff640 1 -- 172.21.15.29:0/2706597305 shutdown_connections 2023-10-16T14:30:01.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.918+0000 7fc2c7fff640 1 -- 172.21.15.29:0/2706597305 wait complete. 2023-10-16T14:30:01.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.919+0000 7fc2c7fff640 1 Processor -- start 2023-10-16T14:30:01.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.919+0000 7fc2c7fff640 1 -- start start 2023-10-16T14:30:01.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.920+0000 7fc2c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 0x7fc2c806c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:01.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.920+0000 7fc2c7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2c815c9f0 0x7fc2c81f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:01.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.920+0000 7fc2c7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c81f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.920+0000 7fc2c7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80c2520 con 0x7fc2c80c9230 2023-10-16T14:30:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.920+0000 7fc2c7fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806b0c0 con 0x7fc2c815c9f0 2023-10-16T14:30:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.920+0000 7fc2c7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806b3a0 con 0x7fc2c815dde0 2023-10-16T14:30:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 0x7fc2c806c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 0x7fc2c806c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57822/0 (socket says 172.21.15.29:57822) 2023-10-16T14:30:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 -- 172.21.15.29:0/3672056352 learned_addr learned my addr 172.21.15.29:0/3672056352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c67fc640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2c815c9f0 0x7fc2c81f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c77fe640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c81f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 -- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 msgr2=0x7fc2c81f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c81f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 -- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2c815c9f0 msgr2=0x7fc2c81f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2c815c9f0 0x7fc2c81f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.921+0000 7fc2c6ffd640 1 -- 172.21.15.29:0/3672056352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c0009b30 con 0x7fc2c80c9230 2023-10-16T14:30:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.922+0000 7fc2c6ffd640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 0x7fc2c806c160 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e9b0 tx=0x7fc2b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.922+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8009670 con 0x7fc2c80c9230 2023-10-16T14:30:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.922+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b800c9a0 con 0x7fc2c80c9230 2023-10-16T14:30:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.922+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b800cd50 con 0x7fc2c80c9230 2023-10-16T14:30:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.922+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c806b680 con 0x7fc2c80c9230 2023-10-16T14:30:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.922+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c806bc40 con 0x7fc2c80c9230 2023-10-16T14:30:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.923+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c804e9b0 con 0x7fc2c80c9230 2023-10-16T14:30:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.924+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc2b8013470 con 0x7fc2c80c9230 2023-10-16T14:30:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.925+0000 7fc2bf7fe640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2a806af90 0x7fc2a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.925+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc2b8089d20 con 0x7fc2c80c9230 2023-10-16T14:30:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.926+0000 7fc2c67fc640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2a806af90 0x7fc2a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.927+0000 7fc2c67fc640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2a806af90 0x7fc2a806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2c81515f0 tx=0x7fc2b0009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:01.930+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2b80549e0 con 0x7fc2c80c9230 2023-10-16T14:30:02.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.124+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc2c80adac0 con 0x7fc2c80c9230 2023-10-16T14:30:02.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.660+0000 7fc2bf7fe640 1 -- 172.21.15.29:0/3672056352 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v55) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc2b80170d0 con 0x7fc2c80c9230 2023-10-16T14:30:02.664 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:30:02.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.664+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2a806af90 msgr2=0x7fc2a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:02.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.664+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2a806af90 0x7fc2a806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2c81515f0 tx=0x7fc2b0009040 comp rx=0 tx=0).stop 2023-10-16T14:30:02.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.664+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 msgr2=0x7fc2c806c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:02.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.664+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 0x7fc2c806c160 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e9b0 tx=0x7fc2b800ee70 comp rx=0 tx=0).stop 2023-10-16T14:30:02.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.666+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 shutdown_connections 2023-10-16T14:30:02.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.666+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2a806af90 0x7fc2a806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.666+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2c815dde0 0x7fc2c81f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.666+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2c815c9f0 0x7fc2c81f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.666+0000 7fc2c7fff640 1 --2- 172.21.15.29:0/3672056352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2c80c9230 0x7fc2c806c160 secure :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e9b0 tx=0x7fc2b800ee70 comp rx=0 tx=0).stop 2023-10-16T14:30:02.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.666+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 >> 172.21.15.29:0/3672056352 conn(0x7fc2c80ce6a0 msgr2=0x7fc2c815bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:02.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.667+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 shutdown_connections 2023-10-16T14:30:02.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.667+0000 7fc2c7fff640 1 -- 172.21.15.29:0/3672056352 wait complete. 2023-10-16T14:30:02.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:30:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.899+0000 7f7737154640 1 Processor -- start 2023-10-16T14:30:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.899+0000 7f7737154640 1 -- start start 2023-10-16T14:30:02.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.900+0000 7f7737154640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f773015ce60 0x7f7730161460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.900+0000 7f7737154640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7737154640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77300cdb30 con 0x7f773015e140 2023-10-16T14:30:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7737154640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77300ce340 con 0x7f773015ce60 2023-10-16T14:30:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7737154640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773015d770 con 0x7f773015d240 2023-10-16T14:30:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7736152640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f773015ce60 0x7f7730161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:02.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7736953640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f773015e140 0x7f77300d0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43702/0 (socket says 172.21.15.29:43702) 2023-10-16T14:30:02.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7736953640 1 -- 172.21.15.29:0/1009989820 learned_addr learned my addr 172.21.15.29:0/1009989820 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7735951640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.901+0000 7f7736152640 1 -- 172.21.15.29:0/1009989820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 msgr2=0x7f773015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.902+0000 7f7736152640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773015dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.902+0000 7f7736152640 1 -- 172.21.15.29:0/1009989820 >> v1:172.21.15.29:6789/0 conn(0x7f773015e140 legacy=0x7f77300d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.902+0000 7f7736152640 1 -- 172.21.15.29:0/1009989820 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77300c99b0 con 0x7f773015ce60 2023-10-16T14:30:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.902+0000 7f7735951640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773015dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.903+0000 7f7736152640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f7730161460 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f772c0097a0 tx=0x7f772c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdd0ffff5196195 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.903+0000 7f7735150640 1 -- 172.21.15.29:0/1009989820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c031780 con 0x7f773015ce60 2023-10-16T14:30:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.903+0000 7f7735150640 1 -- 172.21.15.29:0/1009989820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c0318e0 con 0x7f773015ce60 2023-10-16T14:30:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.903+0000 7f7735150640 1 -- 172.21.15.29:0/1009989820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c031c70 con 0x7f773015ce60 2023-10-16T14:30:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 -- 172.21.15.29:0/1009989820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 msgr2=0x7f7730161460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f7730161460 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f772c0097a0 tx=0x7f772c02b820 comp rx=0 tx=0).stop 2023-10-16T14:30:02.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 -- 172.21.15.29:0/1009989820 shutdown_connections 2023-10-16T14:30:02.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773015dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 --2- 172.21.15.29:0/1009989820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f7730161460 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 -- 172.21.15.29:0/1009989820 >> 172.21.15.29:0/1009989820 conn(0x7f77300cf550 msgr2=0x7f77300cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 -- 172.21.15.29:0/1009989820 shutdown_connections 2023-10-16T14:30:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.904+0000 7f7737154640 1 -- 172.21.15.29:0/1009989820 wait complete. 2023-10-16T14:30:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.906+0000 7f7737154640 1 Processor -- start 2023-10-16T14:30:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.906+0000 7f7737154640 1 -- start start 2023-10-16T14:30:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7737154640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f77300c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7737154640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773014d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7737154640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f773015e140 0x7f77300d1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7737154640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300c7dc0 con 0x7f773015ce60 2023-10-16T14:30:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7737154640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300cabf0 con 0x7f773015e140 2023-10-16T14:30:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7737154640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77300cae90 con 0x7f773015d240 2023-10-16T14:30:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7736152640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f77300c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7736152640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f77300c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57842/0 (socket says 172.21.15.29:57842) 2023-10-16T14:30:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7736152640 1 -- 172.21.15.29:0/1625801320 learned_addr learned my addr 172.21.15.29:0/1625801320 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.907+0000 7f7735951640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773014d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7736152640 1 -- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 msgr2=0x7f773014d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7736152640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773014d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7736152640 1 -- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f773015e140 msgr2=0x7f77300d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7736152640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f773015e140 0x7f77300d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7736152640 1 -- 172.21.15.29:0/1625801320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c009400 con 0x7f773015ce60 2023-10-16T14:30:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7735951640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773014d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7736152640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f77300c9190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f772c034040 tx=0x7f772c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c0328a0 con 0x7f773015ce60 2023-10-16T14:30:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c032a20 con 0x7f773015ce60 2023-10-16T14:30:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c032dd0 con 0x7f773015ce60 2023-10-16T14:30:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773014dd80 con 0x7f773015ce60 2023-10-16T14:30:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.908+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77300c7800 con 0x7f773015ce60 2023-10-16T14:30:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.909+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773004e9b0 con 0x7f773015ce60 2023-10-16T14:30:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.911+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f772c04e440 con 0x7f773015ce60 2023-10-16T14:30:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.911+0000 7f7726ffd640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f771406afe0 0x7f771406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.912+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6002+0+0 (secure 0 0 0) 0x7f772c0abc80 con 0x7f773015ce60 2023-10-16T14:30:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.912+0000 7f7735951640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f771406afe0 0x7f771406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:02.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.913+0000 7f7735951640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f771406afe0 0x7f771406d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f773015e690 tx=0x7f7720008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:02.916+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f772c0769c0 con 0x7f773015ce60 2023-10-16T14:30:03.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.111+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f77300cd910 con 0x7f773015ce60 2023-10-16T14:30:03.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.673+0000 7f7726ffd640 1 -- 172.21.15.29:0/1625801320 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v56) v1 ==== 169+0+0 (secure 0 0 0) 0x7f772c079fe0 con 0x7f773015ce60 2023-10-16T14:30:03.677 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:30:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.678+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f771406afe0 msgr2=0x7f771406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.678+0000 7f7737154640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f771406afe0 0x7f771406d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f773015e690 tx=0x7f7720008040 comp rx=0 tx=0).stop 2023-10-16T14:30:03.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.678+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 msgr2=0x7f77300c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.678+0000 7f7737154640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f77300c9190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f772c034040 tx=0x7f772c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.679+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 shutdown_connections 2023-10-16T14:30:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.679+0000 7f7737154640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f771406afe0 0x7f771406d4a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.679+0000 7f7737154640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f773015e140 0x7f77300d1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.679+0000 7f7737154640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f773015d240 0x7f773014d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.679+0000 7f7737154640 1 --2- 172.21.15.29:0/1625801320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f773015ce60 0x7f77300c9190 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.680+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 >> 172.21.15.29:0/1625801320 conn(0x7f77300cf550 msgr2=0x7f77300b1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.680+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 shutdown_connections 2023-10-16T14:30:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.680+0000 7f7737154640 1 -- 172.21.15.29:0/1625801320 wait complete. 2023-10-16T14:30:03.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:30:03.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.916+0000 7f15b759e640 1 Processor -- start 2023-10-16T14:30:03.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.916+0000 7f15b759e640 1 -- start start 2023-10-16T14:30:03.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.917+0000 7f15b759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15b815b320 0x7f15b80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.917+0000 7f15b759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b815b9d0 con 0x7f15b80ad970 2023-10-16T14:30:03.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b815bcb0 con 0x7f15b806c040 2023-10-16T14:30:03.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b815b7d0 con 0x7f15b815b320 2023-10-16T14:30:03.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b5d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f15b80ad970 0x7f15b80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53280/0 (socket says 172.21.15.29:53280) 2023-10-16T14:30:03.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b5d9b640 1 -- 172.21.15.29:0/1714701960 learned_addr learned my addr 172.21.15.29:0/1714701960 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:03.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794129993 0 0) 0x7f15b815b9d0 con 0x7f15b80ad970 2023-10-16T14:30:03.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a0003520 con 0x7f15b80ad970 2023-10-16T14:30:03.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.918+0000 7f15b6d9d640 1 --2- 172.21.15.29:0/1714701960 >> v2:172.21.15.170:3300/0 conn(0x7f15b815b320 0x7f15b80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012409917 0 0) 0x7f15a0003520 con 0x7f15b80ad970 2023-10-16T14:30:03.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 >> v2:172.21.15.170:3300/0 conn(0x7f15b815b320 msgr2=0x7f15b80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:03.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 --2- 172.21.15.29:0/1714701960 >> v2:172.21.15.170:3300/0 conn(0x7f15b815b320 0x7f15b80cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 >> v1:172.21.15.170:6789/0 conn(0x7f15b806c040 legacy=0x7f15b806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b815bf90 con 0x7f15b80ad970 2023-10-16T14:30:03.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f15ac001670 con 0x7f15b80ad970 2023-10-16T14:30:03.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 >> v1:172.21.15.29:6789/0 conn(0x7f15b80ad970 legacy=0x7f15b80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15a00039a0 0x7f15a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a0005180 0x7f15a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b815b7d0 con 0x7f15a00047c0 2023-10-16T14:30:03.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000c050 con 0x7f15a00039a0 2023-10-16T14:30:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000c210 con 0x7f15a0005180 2023-10-16T14:30:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.919+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b6d9d640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a0005180 0x7f15a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b659c640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15a00039a0 0x7f15a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b5d9b640 1 -- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a0005180 msgr2=0x7f15a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a0005180 0x7f15a000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b5d9b640 1 -- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15a00039a0 msgr2=0x7f15a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15a00039a0 0x7f15a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b5d9b640 1 -- 172.21.15.29:0/1714701960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a0003520 con 0x7f15a00047c0 2023-10-16T14:30:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b6d9d640 1 -- 172.21.15.29:0/1714701960 reap_dead start 2023-10-16T14:30:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.920+0000 7f15b659c640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15a00039a0 0x7f15a0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.921+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00c210 tx=0x7f15a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcde31490b760d25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.921+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac005ad0 con 0x7f15a00047c0 2023-10-16T14:30:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.921+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac005c30 con 0x7f15a00047c0 2023-10-16T14:30:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.921+0000 7f15b559a640 1 -- 172.21.15.29:0/1714701960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac02ca50 con 0x7f15a00047c0 2023-10-16T14:30:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 -- 172.21.15.29:0/1714701960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a00047c0 msgr2=0x7f15a0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00c210 tx=0x7f15a00057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 -- 172.21.15.29:0/1714701960 shutdown_connections 2023-10-16T14:30:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 --2- 172.21.15.29:0/1714701960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a00047c0 0x7f15a0004c40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 -- 172.21.15.29:0/1714701960 >> 172.21.15.29:0/1714701960 conn(0x7f15b80ce6a0 msgr2=0x7f15b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 -- 172.21.15.29:0/1714701960 shutdown_connections 2023-10-16T14:30:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.922+0000 7f15b759e640 1 -- 172.21.15.29:0/1714701960 wait complete. 2023-10-16T14:30:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.924+0000 7f15b759e640 1 Processor -- start 2023-10-16T14:30:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.924+0000 7f15b759e640 1 -- start start 2023-10-16T14:30:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a00047c0 0x7f15b814b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 0x7f15b81f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b80afa20 0x7f15b80afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80b03e0 con 0x7f15b81f0400 2023-10-16T14:30:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81f0ef0 con 0x7f15b80afa20 2023-10-16T14:30:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81f11d0 con 0x7f15a00047c0 2023-10-16T14:30:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 0x7f15b81f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 0x7f15b81f0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51474/0 (socket says 172.21.15.29:51474) 2023-10-16T14:30:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.925+0000 7f15b5d9b640 1 -- 172.21.15.29:0/2954466056 learned_addr learned my addr 172.21.15.29:0/2954466056 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b659c640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a00047c0 0x7f15b814b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b6d9d640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b80afa20 0x7f15b80afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b5d9b640 1 -- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a00047c0 msgr2=0x7f15b814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a00047c0 0x7f15b814b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b5d9b640 1 -- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b80afa20 msgr2=0x7f15b80afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b80afa20 0x7f15b80afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b5d9b640 1 -- 172.21.15.29:0/2954466056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80b26d0 con 0x7f15b81f0400 2023-10-16T14:30:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.926+0000 7f15b5d9b640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 0x7f15b81f0880 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02c3b0 tx=0x7f15ac001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.927+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac039030 con 0x7f15b81f0400 2023-10-16T14:30:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.927+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac02cc20 con 0x7f15b81f0400 2023-10-16T14:30:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.927+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b80b29b0 con 0x7f15b81f0400 2023-10-16T14:30:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.927+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac03c040 con 0x7f15b81f0400 2023-10-16T14:30:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.927+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b815cd30 con 0x7f15b81f0400 2023-10-16T14:30:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.928+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b806b330 con 0x7f15b81f0400 2023-10-16T14:30:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.929+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f15ac032070 con 0x7f15b81f0400 2023-10-16T14:30:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.930+0000 7f15a6ffd640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f159806af90 0x7f159806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:03.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.930+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f15ac0aad20 con 0x7f15b81f0400 2023-10-16T14:30:03.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.931+0000 7f15b659c640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f159806af90 0x7f159806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:03.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.932+0000 7f15b659c640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f159806af90 0x7f159806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f15a8007fd0 tx=0x7f15a8009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:03.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:03.934+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f15ac075d20 con 0x7f15b81f0400 2023-10-16T14:30:04.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.138+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f15b80cc550 con 0x7f15b81f0400 2023-10-16T14:30:04.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.143+0000 7f15a6ffd640 1 -- 172.21.15.29:0/2954466056 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f15ac079340 con 0x7f15b81f0400 2023-10-16T14:30:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f159806af90 msgr2=0x7f159806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f159806af90 0x7f159806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f15a8007fd0 tx=0x7f15a8009040 comp rx=0 tx=0).stop 2023-10-16T14:30:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 msgr2=0x7f15b81f0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 0x7f15b81f0880 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02c3b0 tx=0x7f15ac001a00 comp rx=0 tx=0).stop 2023-10-16T14:30:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 shutdown_connections 2023-10-16T14:30:04.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f159806af90 0x7f159806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15b81f0400 0x7f15b81f0880 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15b80afa20 0x7f15b80afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 --2- 172.21.15.29:0/2954466056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f15a00047c0 0x7f15b814b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 >> 172.21.15.29:0/2954466056 conn(0x7f15b80ce6a0 msgr2=0x7f15b80ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 shutdown_connections 2023-10-16T14:30:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.146+0000 7f15b759e640 1 -- 172.21.15.29:0/2954466056 wait complete. 2023-10-16T14:30:04.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:30:04.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.399+0000 7efd4e84f640 1 Processor -- start 2023-10-16T14:30:04.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.399+0000 7efd4e84f640 1 -- start start 2023-10-16T14:30:04.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.401+0000 7efd4e84f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd4815ae80 0x7efd4815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.401+0000 7efd4e84f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd480ce390 0x7efd48070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.401+0000 7efd4e84f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd480707e0 con 0x7efd4815aaa0 2023-10-16T14:30:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.401+0000 7efd4e84f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd4806b7a0 con 0x7efd480ce390 2023-10-16T14:30:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.401+0000 7efd4e84f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd4815ef20 con 0x7efd4815ae80 2023-10-16T14:30:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.401+0000 7efd4cdec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd480ce390 0x7efd48070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd4cdec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd480ce390 0x7efd48070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51488/0 (socket says 172.21.15.29:51488) 2023-10-16T14:30:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd4cdec640 1 -- 172.21.15.29:0/3578582159 learned_addr learned my addr 172.21.15.29:0/3578582159 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd3ffff640 1 --2- 172.21.15.29:0/3578582159 >> v2:172.21.15.170:3300/0 conn(0x7efd4815ae80 0x7efd4815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd4cdec640 1 -- 172.21.15.29:0/3578582159 >> v2:172.21.15.170:3300/0 conn(0x7efd4815ae80 msgr2=0x7efd4815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:04.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd4cdec640 1 --2- 172.21.15.29:0/3578582159 >> v2:172.21.15.170:3300/0 conn(0x7efd4815ae80 0x7efd4815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd4cdec640 1 -- 172.21.15.29:0/3578582159 >> v1:172.21.15.29:6789/0 conn(0x7efd4815aaa0 legacy=0x7efd480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd4cdec640 1 -- 172.21.15.29:0/3578582159 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4815f200 con 0x7efd480ce390 2023-10-16T14:30:04.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.402+0000 7efd3ffff640 1 --2- 172.21.15.29:0/3578582159 >> v2:172.21.15.170:3300/0 conn(0x7efd4815ae80 0x7efd4815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:04.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.403+0000 7efd4cdec640 1 --2- 172.21.15.29:0/3578582159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd48070f50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efd480b45e0 tx=0x7efd4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f14e2fe8c642bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:04.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.403+0000 7efd477fe640 1 -- 172.21.15.29:0/3578582159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4002ca80 con 0x7efd480ce390 2023-10-16T14:30:04.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd477fe640 1 -- 172.21.15.29:0/3578582159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4002cbe0 con 0x7efd480ce390 2023-10-16T14:30:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd477fe640 1 -- 172.21.15.29:0/3578582159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400343f0 con 0x7efd480ce390 2023-10-16T14:30:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 -- 172.21.15.29:0/3578582159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 msgr2=0x7efd48070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3578582159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd48070f50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efd480b45e0 tx=0x7efd4002bee0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 -- 172.21.15.29:0/3578582159 shutdown_connections 2023-10-16T14:30:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3578582159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd48070f50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3578582159 >> v2:172.21.15.170:3300/0 conn(0x7efd4815ae80 0x7efd4815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 -- 172.21.15.29:0/3578582159 >> 172.21.15.29:0/3578582159 conn(0x7efd480cb410 msgr2=0x7efd480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:04.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 -- 172.21.15.29:0/3578582159 shutdown_connections 2023-10-16T14:30:04.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.404+0000 7efd4e84f640 1 -- 172.21.15.29:0/3578582159 wait complete. 2023-10-16T14:30:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.406+0000 7efd4e84f640 1 Processor -- start 2023-10-16T14:30:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.406+0000 7efd4e84f640 1 -- start start 2023-10-16T14:30:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.407+0000 7efd4e84f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd480c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.407+0000 7efd4e84f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4815aaa0 0x7efd480c3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.407+0000 7efd4e84f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4815ae80 0x7efd480af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.407+0000 7efd4e84f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd480c4ac0 con 0x7efd480ce390 2023-10-16T14:30:04.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.407+0000 7efd4e84f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd480c3eb0 con 0x7efd4815ae80 2023-10-16T14:30:04.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.407+0000 7efd4e84f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd480c4190 con 0x7efd4815aaa0 2023-10-16T14:30:04.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd480c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd480c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51498/0 (socket says 172.21.15.29:51498) 2023-10-16T14:30:04.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 -- 172.21.15.29:0/3595141277 learned_addr learned my addr 172.21.15.29:0/3595141277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:04.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd3ffff640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4815aaa0 0x7efd480c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd4cdec640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4815ae80 0x7efd480af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 -- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4815aaa0 msgr2=0x7efd480c3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4815aaa0 0x7efd480c3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 -- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4815ae80 msgr2=0x7efd480af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4815ae80 0x7efd480af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 -- 172.21.15.29:0/3595141277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40009b30 con 0x7efd480ce390 2023-10-16T14:30:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.408+0000 7efd47fff640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd480c4f50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efd3800c450 tx=0x7efd3800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.409+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd380052d0 con 0x7efd480ce390 2023-10-16T14:30:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.409+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd38005430 con 0x7efd480ce390 2023-10-16T14:30:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.409+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd38012870 con 0x7efd480ce390 2023-10-16T14:30:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.409+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd480c4470 con 0x7efd480ce390 2023-10-16T14:30:04.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.409+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd480c3a80 con 0x7efd480ce390 2023-10-16T14:30:04.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.410+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd480c3c40 con 0x7efd480ce390 2023-10-16T14:30:04.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.411+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd380129d0 con 0x7efd480ce390 2023-10-16T14:30:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.412+0000 7efd457fa640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efd2806af90 0x7efd2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.412+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7efd380892f0 con 0x7efd480ce390 2023-10-16T14:30:04.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.413+0000 7efd3ffff640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efd2806af90 0x7efd2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.414+0000 7efd3ffff640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efd2806af90 0x7efd2806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efd481f3b60 tx=0x7efd30008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:04.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.416+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efd38054260 con 0x7efd480ce390 2023-10-16T14:30:04.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.624+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7efd4804e160 con 0x7efd480ce390 2023-10-16T14:30:04.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.629+0000 7efd457fa640 1 -- 172.21.15.29:0/3595141277 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v14) v1 ==== 115+0+0 (secure 0 0 0) 0x7efd38057880 con 0x7efd480ce390 2023-10-16T14:30:04.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efd2806af90 msgr2=0x7efd2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efd2806af90 0x7efd2806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efd481f3b60 tx=0x7efd30008040 comp rx=0 tx=0).stop 2023-10-16T14:30:04.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 msgr2=0x7efd480c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd480c4f50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efd3800c450 tx=0x7efd3800c910 comp rx=0 tx=0).stop 2023-10-16T14:30:04.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 shutdown_connections 2023-10-16T14:30:04.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efd2806af90 0x7efd2806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4815ae80 0x7efd480af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd4815aaa0 0x7efd480c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 --2- 172.21.15.29:0/3595141277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd480ce390 0x7efd480c4f50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 >> 172.21.15.29:0/3595141277 conn(0x7efd480cb410 msgr2=0x7efd480b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:04.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 shutdown_connections 2023-10-16T14:30:04.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.632+0000 7efd4e84f640 1 -- 172.21.15.29:0/3595141277 wait complete. 2023-10-16T14:30:04.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:30:04.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.872+0000 7f1d8ce10640 1 Processor -- start 2023-10-16T14:30:04.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.872+0000 7f1d8ce10640 1 -- start start 2023-10-16T14:30:04.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.874+0000 7f1d8ce10640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1d8806a4b0 0x7f1d880ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.874+0000 7f1d8ce10640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d880cb030 con 0x7f1d880c9ca0 2023-10-16T14:30:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.874+0000 7f1d8ce10640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d880c91c0 con 0x7f1d8815ad70 2023-10-16T14:30:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.874+0000 7f1d8ce10640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d880c94a0 con 0x7f1d8806a4b0 2023-10-16T14:30:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.874+0000 7f1d86d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d880c9ca0 0x7f1d880ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53296/0 (socket says 172.21.15.29:53296) 2023-10-16T14:30:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.874+0000 7f1d86d9d640 1 -- 172.21.15.29:0/504145348 learned_addr learned my addr 172.21.15.29:0/504145348 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737420529 0 0) 0x7f1d880cb030 con 0x7f1d880c9ca0 2023-10-16T14:30:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d70003520 con 0x7f1d880c9ca0 2023-10-16T14:30:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/504145348 >> v2:172.21.15.170:3300/0 conn(0x7f1d8806a4b0 0x7f1d880ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747819198 0 0) 0x7f1d70003520 con 0x7f1d880c9ca0 2023-10-16T14:30:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 >> v2:172.21.15.170:3300/0 conn(0x7f1d8806a4b0 msgr2=0x7f1d880ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 --2- 172.21.15.29:0/504145348 >> v2:172.21.15.170:3300/0 conn(0x7f1d8806a4b0 0x7f1d880ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 >> v1:172.21.15.170:6789/0 conn(0x7f1d8815ad70 legacy=0x7f1d88069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d880c9780 con 0x7f1d880c9ca0 2023-10-16T14:30:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1d800019f0 con 0x7f1d880c9ca0 2023-10-16T14:30:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 >> v1:172.21.15.29:6789/0 conn(0x7f1d880c9ca0 legacy=0x7f1d880ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d70003970 0x7f1d70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d700047c0 0x7f1d70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d70005180 0x7f1d7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880c94a0 con 0x7f1d700047c0 2023-10-16T14:30:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7000c0d0 con 0x7f1d70003970 2023-10-16T14:30:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.875+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7000c290 con 0x7f1d70005180 2023-10-16T14:30:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d700047c0 0x7f1d70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d86d9d640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d70005180 0x7f1d7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d8659c640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d70003970 0x7f1d70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d85d9b640 1 -- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d70005180 msgr2=0x7f1d7000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d70005180 0x7f1d7000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d85d9b640 1 -- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d70003970 msgr2=0x7f1d70003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d70003970 0x7f1d70003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d85d9b640 1 -- 172.21.15.29:0/504145348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d70003520 con 0x7f1d700047c0 2023-10-16T14:30:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.876+0000 7f1d86d9d640 1 -- 172.21.15.29:0/504145348 reap_dead start 2023-10-16T14:30:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.877+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d700047c0 0x7f1d70004c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00ba30 tx=0x7f1d700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f2eb03347bd17f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c032640 con 0x7f1d700047c0 2023-10-16T14:30:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0327a0 con 0x7f1d700047c0 2023-10-16T14:30:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8559a640 1 -- 172.21.15.29:0/504145348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c032b30 con 0x7f1d700047c0 2023-10-16T14:30:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8ce10640 1 -- 172.21.15.29:0/504145348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d700047c0 msgr2=0x7f1d70004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d700047c0 0x7f1d70004c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00ba30 tx=0x7f1d700057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8ce10640 1 -- 172.21.15.29:0/504145348 shutdown_connections 2023-10-16T14:30:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/504145348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d700047c0 0x7f1d70004c40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8ce10640 1 -- 172.21.15.29:0/504145348 >> 172.21.15.29:0/504145348 conn(0x7f1d880ce9b0 msgr2=0x7f1d880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.878+0000 7f1d8ce10640 1 -- 172.21.15.29:0/504145348 shutdown_connections 2023-10-16T14:30:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.879+0000 7f1d8ce10640 1 -- 172.21.15.29:0/504145348 wait complete. 2023-10-16T14:30:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.880+0000 7f1d8ce10640 1 Processor -- start 2023-10-16T14:30:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.880+0000 7f1d8ce10640 1 -- start start 2023-10-16T14:30:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.881+0000 7f1d8ce10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d700047c0 0x7f1d88157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.881+0000 7f1d8ce10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 0x7f1d880b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.881+0000 7f1d8ce10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d880cc540 0x7f1d880cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.881+0000 7f1d8ce10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d88158350 con 0x7f1d880b0be0 2023-10-16T14:30:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.881+0000 7f1d8ce10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880b1580 con 0x7f1d880cc540 2023-10-16T14:30:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.881+0000 7f1d8ce10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d881f1d00 con 0x7f1d700047c0 2023-10-16T14:30:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 0x7f1d880b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 0x7f1d880b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51520/0 (socket says 172.21.15.29:51520) 2023-10-16T14:30:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 -- 172.21.15.29:0/1663413598 learned_addr learned my addr 172.21.15.29:0/1663413598 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d8659c640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d700047c0 0x7f1d88157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d86d9d640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d880cc540 0x7f1d880cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 -- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d700047c0 msgr2=0x7f1d88157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d700047c0 0x7f1d88157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 -- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d880cc540 msgr2=0x7f1d880cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d880cc540 0x7f1d880cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.882+0000 7f1d85d9b640 1 -- 172.21.15.29:0/1663413598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d880cd1e0 con 0x7f1d880b0be0 2023-10-16T14:30:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.883+0000 7f1d85d9b640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 0x7f1d880b1040 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c02bf90 tx=0x7f1d7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.883+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c03b9f0 con 0x7f1d880b0be0 2023-10-16T14:30:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.883+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c03bb50 con 0x7f1d880b0be0 2023-10-16T14:30:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.883+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c030410 con 0x7f1d880b0be0 2023-10-16T14:30:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.883+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d881f2190 con 0x7f1d880b0be0 2023-10-16T14:30:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.883+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d881f13d0 con 0x7f1d880b0be0 2023-10-16T14:30:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.885+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8804e160 con 0x7f1d880b0be0 2023-10-16T14:30:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.886+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1d7c030570 con 0x7f1d880b0be0 2023-10-16T14:30:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.887+0000 7f1d76ffd640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f1d6806af90 0x7f1d6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.887+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f1d7c0ac060 con 0x7f1d880b0be0 2023-10-16T14:30:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.887+0000 7f1d8659c640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f1d6806af90 0x7f1d6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.889+0000 7f1d8659c640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f1d6806af90 0x7f1d6806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1d88069740 tx=0x7f1d78009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:04.891+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1d7c076f70 con 0x7f1d880b0be0 2023-10-16T14:30:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.088+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1d881f1e40 con 0x7f1d880b0be0 2023-10-16T14:30:05.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.095+0000 7f1d76ffd640 1 -- 172.21.15.29:0/1663413598 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1d7c07a590 con 0x7f1d880b0be0 2023-10-16T14:30:05.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.098+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f1d6806af90 msgr2=0x7f1d6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.098+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f1d6806af90 0x7f1d6806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1d88069740 tx=0x7f1d78009040 comp rx=0 tx=0).stop 2023-10-16T14:30:05.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 msgr2=0x7f1d880b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 0x7f1d880b1040 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c02bf90 tx=0x7f1d7c004730 comp rx=0 tx=0).stop 2023-10-16T14:30:05.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 shutdown_connections 2023-10-16T14:30:05.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f1d6806af90 0x7f1d6806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d880b0be0 0x7f1d880b1040 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d880cc540 0x7f1d880cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 --2- 172.21.15.29:0/1663413598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d700047c0 0x7f1d88157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 >> 172.21.15.29:0/1663413598 conn(0x7f1d880ce9b0 msgr2=0x7f1d880ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:05.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 shutdown_connections 2023-10-16T14:30:05.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.099+0000 7f1d8ce10640 1 -- 172.21.15.29:0/1663413598 wait complete. 2023-10-16T14:30:05.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T14:30:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.349+0000 7fca3d111640 1 Processor -- start 2023-10-16T14:30:05.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.349+0000 7fca3d111640 1 -- start start 2023-10-16T14:30:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.350+0000 7fca3d111640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fca3815cd20 0x7fca380cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.350+0000 7fca3d111640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca3815d100 0x7fca380b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.350+0000 7fca3d111640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca380b8430 con 0x7fca380b1810 2023-10-16T14:30:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.350+0000 7fca3d111640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca380d2160 con 0x7fca3815d100 2023-10-16T14:30:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.350+0000 7fca3d111640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca380ce0b0 con 0x7fca3815cd20 2023-10-16T14:30:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca377fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca3815d100 0x7fca380b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca3c910640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fca380b1810 0x7fca3806bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53298/0 (socket says 172.21.15.29:53298) 2023-10-16T14:30:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca3c910640 1 -- 172.21.15.29:0/3117046579 learned_addr learned my addr 172.21.15.29:0/3117046579 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:05.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca37fff640 1 --2- 172.21.15.29:0/3117046579 >> v2:172.21.15.170:3300/0 conn(0x7fca3815cd20 0x7fca380cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca377fe640 1 -- 172.21.15.29:0/3117046579 >> v2:172.21.15.170:3300/0 conn(0x7fca3815cd20 msgr2=0x7fca380cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca377fe640 1 --2- 172.21.15.29:0/3117046579 >> v2:172.21.15.170:3300/0 conn(0x7fca3815cd20 0x7fca380cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca377fe640 1 -- 172.21.15.29:0/3117046579 >> v1:172.21.15.29:6789/0 conn(0x7fca380b1810 legacy=0x7fca3806bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.352+0000 7fca377fe640 1 -- 172.21.15.29:0/3117046579 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca380b8680 con 0x7fca3815d100 2023-10-16T14:30:05.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.351+0000 7fca36ffd640 1 -- 172.21.15.29:0/3117046579 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850174326 0 0) 0x7fca380b8430 con 0x7fca380b1810 2023-10-16T14:30:05.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.353+0000 7fca377fe640 1 --2- 172.21.15.29:0/3117046579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815d100 0x7fca380b4590 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fca380cf4f0 tx=0x7fca2c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805524392a27f2a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:05.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.353+0000 7fca36ffd640 1 -- 172.21.15.29:0/3117046579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c030610 con 0x7fca3815d100 2023-10-16T14:30:05.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.353+0000 7fca36ffd640 1 -- 172.21.15.29:0/3117046579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c030770 con 0x7fca3815d100 2023-10-16T14:30:05.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.353+0000 7fca36ffd640 1 -- 172.21.15.29:0/3117046579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c030b00 con 0x7fca3815d100 2023-10-16T14:30:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 -- 172.21.15.29:0/3117046579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815d100 msgr2=0x7fca380b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 --2- 172.21.15.29:0/3117046579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815d100 0x7fca380b4590 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fca380cf4f0 tx=0x7fca2c02b960 comp rx=0 tx=0).stop 2023-10-16T14:30:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 -- 172.21.15.29:0/3117046579 shutdown_connections 2023-10-16T14:30:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 --2- 172.21.15.29:0/3117046579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815d100 0x7fca380b4590 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 --2- 172.21.15.29:0/3117046579 >> v2:172.21.15.170:3300/0 conn(0x7fca3815cd20 0x7fca380cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 -- 172.21.15.29:0/3117046579 >> 172.21.15.29:0/3117046579 conn(0x7fca380cf590 msgr2=0x7fca380cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 -- 172.21.15.29:0/3117046579 shutdown_connections 2023-10-16T14:30:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.354+0000 7fca3d111640 1 -- 172.21.15.29:0/3117046579 wait complete. 2023-10-16T14:30:05.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.356+0000 7fca3d111640 1 Processor -- start 2023-10-16T14:30:05.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.356+0000 7fca3d111640 1 -- start start 2023-10-16T14:30:05.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca3d111640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca380b1810 0x7fca381530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca3d111640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 0x7fca381f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca3d111640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca3815d100 0x7fca380b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca3d111640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca38152300 con 0x7fca3815cd20 2023-10-16T14:30:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca3d111640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca381525e0 con 0x7fca380b1810 2023-10-16T14:30:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca3d111640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca381528c0 con 0x7fca3815d100 2023-10-16T14:30:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 0x7fca381f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 0x7fca381f67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51534/0 (socket says 172.21.15.29:51534) 2023-10-16T14:30:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 -- 172.21.15.29:0/2862560993 learned_addr learned my addr 172.21.15.29:0/2862560993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca37fff640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca380b1810 0x7fca381530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 -- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca3815d100 msgr2=0x7fca380b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca3815d100 0x7fca380b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 -- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca380b1810 msgr2=0x7fca381530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca380b1810 0x7fca381530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.357+0000 7fca377fe640 1 -- 172.21.15.29:0/2862560993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c009400 con 0x7fca3815cd20 2023-10-16T14:30:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.358+0000 7fca377fe640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 0x7fca381f67c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009890 tx=0x7fca2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.358+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c03e040 con 0x7fca3815cd20 2023-10-16T14:30:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.358+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c039cb0 con 0x7fca3815cd20 2023-10-16T14:30:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.358+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c044630 con 0x7fca3815cd20 2023-10-16T14:30:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.358+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca38152ba0 con 0x7fca3815cd20 2023-10-16T14:30:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.358+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca381602b0 con 0x7fca3815cd20 2023-10-16T14:30:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.359+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca38152dc0 con 0x7fca3815cd20 2023-10-16T14:30:05.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.360+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fca2c044790 con 0x7fca3815cd20 2023-10-16T14:30:05.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.361+0000 7fca34ff9640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fca2406af90 0x7fca2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.361+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fca2c0abd50 con 0x7fca3815cd20 2023-10-16T14:30:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.362+0000 7fca37fff640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fca2406af90 0x7fca2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.363+0000 7fca37fff640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fca2406af90 0x7fca2406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fca28006fd0 tx=0x7fca28008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.366+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fca2c076bb0 con 0x7fca3815cd20 2023-10-16T14:30:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.629+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fca38152440 con 0x7fca3815cd20 2023-10-16T14:30:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.678+0000 7fca34ff9640 1 -- 172.21.15.29:0/2862560993 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v489) v1 ==== 158+0+0 (secure 0 0 0) 0x7fca38152440 con 0x7fca3815cd20 2023-10-16T14:30:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fca2406af90 msgr2=0x7fca2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fca2406af90 0x7fca2406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fca28006fd0 tx=0x7fca28008040 comp rx=0 tx=0).stop 2023-10-16T14:30:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 msgr2=0x7fca381f67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 0x7fca381f67c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009890 tx=0x7fca2c004730 comp rx=0 tx=0).stop 2023-10-16T14:30:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 shutdown_connections 2023-10-16T14:30:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fca2406af90 0x7fca2406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815cd20 0x7fca381f67c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca3815d100 0x7fca380b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 --2- 172.21.15.29:0/2862560993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca380b1810 0x7fca381530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 >> 172.21.15.29:0/2862560993 conn(0x7fca380cf590 msgr2=0x7fca380cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 shutdown_connections 2023-10-16T14:30:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.681+0000 7fca3d111640 1 -- 172.21.15.29:0/2862560993 wait complete. 2023-10-16T14:30:05.704 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-16T14:30:05.704 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-16T14:30:05.705 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:30:05.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:30:05.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.921+0000 7f0f7e13c640 1 Processor -- start 2023-10-16T14:30:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.922+0000 7f0f7e13c640 1 -- start start 2023-10-16T14:30:05.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.923+0000 7f0f7e13c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0f780ad970 0x7f0f780cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.923+0000 7f0f7e13c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7815b9d0 con 0x7f0f7806c040 2023-10-16T14:30:05.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.924+0000 7f0f7e13c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7815bcb0 con 0x7f0f780ccfb0 2023-10-16T14:30:05.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.924+0000 7f0f7e13c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7815bf90 con 0x7f0f780ad970 2023-10-16T14:30:05.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.924+0000 7f0f7d13a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0f7806c040 0x7f0f7806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53314/0 (socket says 172.21.15.29:53314) 2023-10-16T14:30:05.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.924+0000 7f0f7d13a640 1 -- 172.21.15.29:0/1569462724 learned_addr learned my addr 172.21.15.29:0/1569462724 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:05.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.924+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233309036 0 0) 0x7f0f7815b9d0 con 0x7f0f7806c040 2023-10-16T14:30:05.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.924+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f64003520 con 0x7f0f7806c040 2023-10-16T14:30:05.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1569462724 >> v2:172.21.15.170:3300/0 conn(0x7f0f780ad970 0x7f0f780cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3098092606 0 0) 0x7f0f64003520 con 0x7f0f7806c040 2023-10-16T14:30:05.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 >> v2:172.21.15.170:3300/0 conn(0x7f0f780ad970 msgr2=0x7f0f780cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:05.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 --2- 172.21.15.29:0/1569462724 >> v2:172.21.15.170:3300/0 conn(0x7f0f780ad970 0x7f0f780cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 >> v1:172.21.15.170:6789/0 conn(0x7f0f780ccfb0 legacy=0x7f0f780cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:05.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780cc270 con 0x7f0f7806c040 2023-10-16T14:30:05.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0f74001670 con 0x7f0f7806c040 2023-10-16T14:30:05.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 >> v1:172.21.15.29:6789/0 conn(0x7f0f7806c040 legacy=0x7f0f7806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f640039c0 0x7f0f64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f640051d0 0x7f0f6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7815bf90 con 0x7f0f640047c0 2023-10-16T14:30:05.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6400c120 con 0x7f0f640039c0 2023-10-16T14:30:05.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6400c2e0 con 0x7f0f640051d0 2023-10-16T14:30:05.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.925+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7d13a640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f640039c0 0x7f0f64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7d93b640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f640051d0 0x7f0f6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7c939640 1 -- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f640051d0 msgr2=0x7f0f6400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:05.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f640051d0 0x7f0f6400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7c939640 1 -- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f640039c0 msgr2=0x7f0f64003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:05.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f640039c0 0x7f0f64003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7c939640 1 -- 172.21.15.29:0/1569462724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64003520 con 0x7f0f640047c0 2023-10-16T14:30:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7d93b640 1 -- 172.21.15.29:0/1569462724 reap_dead start 2023-10-16T14:30:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.926+0000 7f0f7d13a640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f640039c0 0x7f0f64003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.927+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f64004c40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00ba10 tx=0x7f0f640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c479622ce89034 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.927+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c032630 con 0x7f0f640047c0 2023-10-16T14:30:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.927+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c032790 con 0x7f0f640047c0 2023-10-16T14:30:05.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.927+0000 7f0f6bfff640 1 -- 172.21.15.29:0/1569462724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c032b20 con 0x7f0f640047c0 2023-10-16T14:30:05.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1569462724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 msgr2=0x7f0f64004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f64004c40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00ba10 tx=0x7f0f640057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1569462724 shutdown_connections 2023-10-16T14:30:05.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1569462724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f64004c40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1569462724 >> 172.21.15.29:0/1569462724 conn(0x7f0f780ce6a0 msgr2=0x7f0f780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:05.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1569462724 shutdown_connections 2023-10-16T14:30:05.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.928+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1569462724 wait complete. 2023-10-16T14:30:05.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.930+0000 7f0f7e13c640 1 Processor -- start 2023-10-16T14:30:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.930+0000 7f0f7e13c640 1 -- start start 2023-10-16T14:30:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7e13c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f780aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7e13c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f78156b20 0x7f0f78156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7e13c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f7815e8e0 0x7f0f780c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7e13c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780adef0 con 0x7f0f640047c0 2023-10-16T14:30:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7e13c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780ae1d0 con 0x7f0f78156b20 2023-10-16T14:30:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7e13c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780ae4b0 con 0x7f0f7815e8e0 2023-10-16T14:30:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7d13a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f780aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7d13a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f780aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51540/0 (socket says 172.21.15.29:51540) 2023-10-16T14:30:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7d13a640 1 -- 172.21.15.29:0/1519307275 learned_addr learned my addr 172.21.15.29:0/1519307275 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:05.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.931+0000 7f0f7d93b640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f7815e8e0 0x7f0f780c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7d13a640 1 -- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f7815e8e0 msgr2=0x7f0f780c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f78156b20 0x7f0f78156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7d13a640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f7815e8e0 0x7f0f780c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7d13a640 1 -- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f78156b20 msgr2=0x7f0f78156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7d13a640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f78156b20 0x7f0f78156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:05.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7d13a640 1 -- 172.21.15.29:0/1519307275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780ae790 con 0x7f0f640047c0 2023-10-16T14:30:05.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7d13a640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f780aef90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0f74000fb0 tx=0x7f0f7400db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:05.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f740072d0 con 0x7f0f640047c0 2023-10-16T14:30:05.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f74007430 con 0x7f0f640047c0 2023-10-16T14:30:05.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f780aea70 con 0x7f0f640047c0 2023-10-16T14:30:05.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f740047e0 con 0x7f0f640047c0 2023-10-16T14:30:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.932+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7815ed70 con 0x7f0f640047c0 2023-10-16T14:30:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.934+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7806b330 con 0x7f0f640047c0 2023-10-16T14:30:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.934+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0f74004940 con 0x7f0f640047c0 2023-10-16T14:30:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.935+0000 7f0f69ffb640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f0f5c06afc0 0x7f0f5c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.935+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f0f74089460 con 0x7f0f640047c0 2023-10-16T14:30:05.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.936+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f0f5c06afc0 0x7f0f5c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:05.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.937+0000 7f0f7c939640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f0f5c06afc0 0x7f0f5c06d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f780ade20 tx=0x7f0f6c03d040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:05.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:05.940+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0f74054340 con 0x7f0f640047c0 2023-10-16T14:30:06.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.139+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f780743c0 con 0x7f0f640047c0 2023-10-16T14:30:06.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.141+0000 7f0f69ffb640 1 -- 172.21.15.29:0/1519307275 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+13941 (secure 0 0 0) 0x7f0f74057960 con 0x7f0f640047c0 2023-10-16T14:30:06.145 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:06.145 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:30:03.661615+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/2287219370":"2023-10-17T14:30:00.649249+0000","172.21.15.170:0/728555731":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6832/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6833/1268688927":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/3736763821":"2023-10-17T14:17:54.926664+0000","172.21.15.29:6833/2287219370":"2023-10-17T14:30:00.649249+0000","172.21.15.170:0/3139659877":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/2403070055":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2826870214":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/1855634044":"2023-10-17T14:17:54.926664+0000","172.21.15.170:6832/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:6833/1418556441":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/2300081340":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3069016888":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/176252655":"2023-10-17T14:17:54.926664+0000","172.21.15.170:0/1408092989":"2023-10-17T14:29:56.276500+0000","172.21.15.170:0/3530276997":"2023-10-17T14:29:56.276500+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:30:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f0f5c06afc0 msgr2=0x7f0f5c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f0f5c06afc0 0x7f0f5c06d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f780ade20 tx=0x7f0f6c03d040 comp rx=0 tx=0).stop 2023-10-16T14:30:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 msgr2=0x7f0f780aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f780aef90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0f74000fb0 tx=0x7f0f7400db50 comp rx=0 tx=0).stop 2023-10-16T14:30:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 shutdown_connections 2023-10-16T14:30:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f0f5c06afc0 0x7f0f5c06d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f7815e8e0 0x7f0f780c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f78156b20 0x7f0f78156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 --2- 172.21.15.29:0/1519307275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f640047c0 0x7f0f780aef90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 >> 172.21.15.29:0/1519307275 conn(0x7f0f780ce6a0 msgr2=0x7f0f780c41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 shutdown_connections 2023-10-16T14:30:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.144+0000 7f0f7e13c640 1 -- 172.21.15.29:0/1519307275 wait complete. 2023-10-16T14:30:06.168 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:30:06.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:30:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.394+0000 7fd93759e640 1 Processor -- start 2023-10-16T14:30:06.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.394+0000 7fd93759e640 1 -- start start 2023-10-16T14:30:06.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9380cfc40 0x7fd9380d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380d0970 con 0x7fd9380b1ca0 2023-10-16T14:30:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380d0ea0 con 0x7fd9380b2080 2023-10-16T14:30:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9380d0bc0 con 0x7fd9380cfc40 2023-10-16T14:30:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9380b1ca0 0x7fd9380ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53320/0 (socket says 172.21.15.29:53320) 2023-10-16T14:30:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93659c640 1 -- 172.21.15.29:0/1479988594 learned_addr learned my addr 172.21.15.29:0/1479988594 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd936d9d640 1 --2- 172.21.15.29:0/1479988594 >> v2:172.21.15.29:3300/0 conn(0x7fd9380cfc40 0x7fd9380d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93559a640 1 -- 172.21.15.29:0/1479988594 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088718893 0 0) 0x7fd9380d0970 con 0x7fd9380b1ca0 2023-10-16T14:30:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.396+0000 7fd93559a640 1 -- 172.21.15.29:0/1479988594 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924003520 con 0x7fd9380b1ca0 2023-10-16T14:30:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.397+0000 7fd936d9d640 1 -- 172.21.15.29:0/1479988594 >> v1:172.21.15.170:6789/0 conn(0x7fd9380b2080 legacy=0x7fd9380b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.397+0000 7fd936d9d640 1 -- 172.21.15.29:0/1479988594 >> v1:172.21.15.29:6789/0 conn(0x7fd9380b1ca0 legacy=0x7fd9380ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.397+0000 7fd936d9d640 1 -- 172.21.15.29:0/1479988594 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380cd940 con 0x7fd9380cfc40 2023-10-16T14:30:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.398+0000 7fd936d9d640 1 --2- 172.21.15.29:0/1479988594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380cfc40 0x7fd9380d36b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd9300097a0 tx=0x7fd93002b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2e65ef63967812 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.398+0000 7fd93559a640 1 -- 172.21.15.29:0/1479988594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93002f800 con 0x7fd9380cfc40 2023-10-16T14:30:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.398+0000 7fd93559a640 1 -- 172.21.15.29:0/1479988594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93002f960 con 0x7fd9380cfc40 2023-10-16T14:30:06.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.398+0000 7fd93559a640 1 -- 172.21.15.29:0/1479988594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93002fcf0 con 0x7fd9380cfc40 2023-10-16T14:30:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 -- 172.21.15.29:0/1479988594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380cfc40 msgr2=0x7fd9380d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 --2- 172.21.15.29:0/1479988594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380cfc40 0x7fd9380d36b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd9300097a0 tx=0x7fd93002b9e0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 -- 172.21.15.29:0/1479988594 shutdown_connections 2023-10-16T14:30:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 --2- 172.21.15.29:0/1479988594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380cfc40 0x7fd9380d36b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 -- 172.21.15.29:0/1479988594 >> 172.21.15.29:0/1479988594 conn(0x7fd9380d2b90 msgr2=0x7fd9380d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:06.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 -- 172.21.15.29:0/1479988594 shutdown_connections 2023-10-16T14:30:06.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.399+0000 7fd93759e640 1 -- 172.21.15.29:0/1479988594 wait complete. 2023-10-16T14:30:06.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.401+0000 7fd93759e640 1 Processor -- start 2023-10-16T14:30:06.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.401+0000 7fd93759e640 1 -- start start 2023-10-16T14:30:06.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 0x7fd93814cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9380b2080 0x7fd9380cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9380cfc40 0x7fd93815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380cb7b0 con 0x7fd9380b1ca0 2023-10-16T14:30:06.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380cb320 con 0x7fd9380cfc40 2023-10-16T14:30:06.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380ca710 con 0x7fd9380b2080 2023-10-16T14:30:06.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 0x7fd93814cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 0x7fd93814cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51556/0 (socket says 172.21.15.29:51556) 2023-10-16T14:30:06.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.402+0000 7fd93659c640 1 -- 172.21.15.29:0/2367999458 learned_addr learned my addr 172.21.15.29:0/2367999458 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:06.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd93659c640 1 -- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9380b2080 msgr2=0x7fd9380cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd936d9d640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9380cfc40 0x7fd93815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd935d9b640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9380b2080 0x7fd9380cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd93659c640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9380b2080 0x7fd9380cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd93659c640 1 -- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9380cfc40 msgr2=0x7fd93815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd93659c640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9380cfc40 0x7fd93815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd93659c640 1 -- 172.21.15.29:0/2367999458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930009400 con 0x7fd9380b1ca0 2023-10-16T14:30:06.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd936d9d640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9380cfc40 0x7fd93815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:06.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd93659c640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 0x7fd93814cf40 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd92800cbc0 tx=0x7fd92800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:06.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.403+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92800ad40 con 0x7fd9380b1ca0 2023-10-16T14:30:06.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.404+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd928012400 con 0x7fd9380b1ca0 2023-10-16T14:30:06.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.404+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9380ca9f0 con 0x7fd9380b1ca0 2023-10-16T14:30:06.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.404+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92800aea0 con 0x7fd9380b1ca0 2023-10-16T14:30:06.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.404+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9380cafb0 con 0x7fd9380b1ca0 2023-10-16T14:30:06.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.405+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93804e9b0 con 0x7fd9380b1ca0 2023-10-16T14:30:06.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.406+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd928007070 con 0x7fd9380b1ca0 2023-10-16T14:30:06.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.407+0000 7fd92effd640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd91806af90 0x7fd91806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.407+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fd928089750 con 0x7fd9380b1ca0 2023-10-16T14:30:06.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.407+0000 7fd935d9b640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd91806af90 0x7fd91806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.409+0000 7fd935d9b640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd91806af90 0x7fd91806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9380c6000 tx=0x7fd920009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:06.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.411+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd928054750 con 0x7fd9380b1ca0 2023-10-16T14:30:06.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.607+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd938073a40 con 0x7fd9380b1ca0 2023-10-16T14:30:06.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.607+0000 7fd92effd640 1 -- 172.21.15.29:0/2367999458 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9280170a0 con 0x7fd9380b1ca0 2023-10-16T14:30:06.611 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:30:06.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd91806af90 msgr2=0x7fd91806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd91806af90 0x7fd91806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9380c6000 tx=0x7fd920009040 comp rx=0 tx=0).stop 2023-10-16T14:30:06.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 msgr2=0x7fd93814cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 0x7fd93814cf40 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd92800cbc0 tx=0x7fd92800a430 comp rx=0 tx=0).stop 2023-10-16T14:30:06.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 shutdown_connections 2023-10-16T14:30:06.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd91806af90 0x7fd91806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9380cfc40 0x7fd93815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9380b2080 0x7fd9380cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 --2- 172.21.15.29:0/2367999458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9380b1ca0 0x7fd93814cf40 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.610+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 >> 172.21.15.29:0/2367999458 conn(0x7fd9380d2b90 msgr2=0x7fd9380b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.611+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 shutdown_connections 2023-10-16T14:30:06.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.611+0000 7fd93759e640 1 -- 172.21.15.29:0/2367999458 wait complete. 2023-10-16T14:30:06.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T14:30:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.854+0000 7fc440a4a640 1 Processor -- start 2023-10-16T14:30:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.855+0000 7fc440a4a640 1 -- start start 2023-10-16T14:30:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.856+0000 7fc440a4a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc43c15a9c0 0x7fc43c160940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.856+0000 7fc440a4a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c0b2370 0x7fc43c0b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.856+0000 7fc440a4a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c06cef0 con 0x7fc43c0cf850 2023-10-16T14:30:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.856+0000 7fc440a4a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c0b2d20 con 0x7fc43c15a9c0 2023-10-16T14:30:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.856+0000 7fc440a4a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c15ada0 con 0x7fc43c0b2370 2023-10-16T14:30:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc43c15a9c0 0x7fc43c160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc43c15a9c0 0x7fc43c160940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51566/0 (socket says 172.21.15.29:51566) 2023-10-16T14:30:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 -- 172.21.15.29:0/2255640644 learned_addr learned my addr 172.21.15.29:0/2255640644 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43a59c640 1 --1- 172.21.15.29:0/2255640644 >> v1:172.21.15.29:6789/0 conn(0x7fc43c0cf850 0x7fc43c071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53336/0 (socket says 172.21.15.29:53336) 2023-10-16T14:30:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43b59e640 1 --2- 172.21.15.29:0/2255640644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c0b2370 0x7fc43c0b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 -- 172.21.15.29:0/2255640644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c0b2370 msgr2=0x7fc43c0b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 --2- 172.21.15.29:0/2255640644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c0b2370 0x7fc43c0b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 -- 172.21.15.29:0/2255640644 >> v1:172.21.15.29:6789/0 conn(0x7fc43c0cf850 legacy=0x7fc43c071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.857+0000 7fc43ad9d640 1 -- 172.21.15.29:0/2255640644 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c15d350 con 0x7fc43c15a9c0 2023-10-16T14:30:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.859+0000 7fc43ad9d640 1 --2- 172.21.15.29:0/2255640644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c15a9c0 0x7fc43c160940 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009e50 tx=0x7fc42c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c0e413ec793127 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.859+0000 7fc439d9b640 1 -- 172.21.15.29:0/2255640644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c02ca80 con 0x7fc43c15a9c0 2023-10-16T14:30:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.859+0000 7fc439d9b640 1 -- 172.21.15.29:0/2255640644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c02cbe0 con 0x7fc43c15a9c0 2023-10-16T14:30:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.859+0000 7fc439d9b640 1 -- 172.21.15.29:0/2255640644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0343f0 con 0x7fc43c15a9c0 2023-10-16T14:30:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.859+0000 7fc440a4a640 1 -- 172.21.15.29:0/2255640644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c15a9c0 msgr2=0x7fc43c160940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.859+0000 7fc440a4a640 1 --2- 172.21.15.29:0/2255640644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c15a9c0 0x7fc43c160940 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009e50 tx=0x7fc42c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.860+0000 7fc440a4a640 1 -- 172.21.15.29:0/2255640644 shutdown_connections 2023-10-16T14:30:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.860+0000 7fc440a4a640 1 --2- 172.21.15.29:0/2255640644 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c0b2370 0x7fc43c0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.860+0000 7fc440a4a640 1 --2- 172.21.15.29:0/2255640644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c15a9c0 0x7fc43c160940 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.860+0000 7fc440a4a640 1 -- 172.21.15.29:0/2255640644 >> 172.21.15.29:0/2255640644 conn(0x7fc43c0cd370 msgr2=0x7fc43c0cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.860+0000 7fc440a4a640 1 -- 172.21.15.29:0/2255640644 shutdown_connections 2023-10-16T14:30:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.860+0000 7fc440a4a640 1 -- 172.21.15.29:0/2255640644 wait complete. 2023-10-16T14:30:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.861+0000 7fc440a4a640 1 Processor -- start 2023-10-16T14:30:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.862+0000 7fc440a4a640 1 -- start start 2023-10-16T14:30:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.862+0000 7fc440a4a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc43c0b2370 0x7fc43c1506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.862+0000 7fc440a4a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 0x7fc43c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc440a4a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c15a9c0 0x7fc43c0cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc440a4a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0c8ba0 con 0x7fc43c0cf850 2023-10-16T14:30:06.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc440a4a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0c8e40 con 0x7fc43c0b2370 2023-10-16T14:30:06.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc440a4a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0c9120 con 0x7fc43c15a9c0 2023-10-16T14:30:06.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 0x7fc43c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 0x7fc43c0cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51568/0 (socket says 172.21.15.29:51568) 2023-10-16T14:30:06.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 -- 172.21.15.29:0/4080615288 learned_addr learned my addr 172.21.15.29:0/4080615288 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:06.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43b59e640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c15a9c0 0x7fc43c0cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 -- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c15a9c0 msgr2=0x7fc43c0cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43ad9d640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc43c0b2370 0x7fc43c1506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c15a9c0 0x7fc43c0cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 -- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc43c0b2370 msgr2=0x7fc43c1506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:06.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc43c0b2370 0x7fc43c1506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:06.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43a59c640 1 -- 172.21.15.29:0/4080615288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c009b30 con 0x7fc43c0cf850 2023-10-16T14:30:06.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.863+0000 7fc43b59e640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c15a9c0 0x7fc43c0cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:06.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.864+0000 7fc43a59c640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 0x7fc43c0cb7a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c3c0 tx=0x7fc42400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:06.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.864+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4240074e0 con 0x7fc43c0cf850 2023-10-16T14:30:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.864+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc424007640 con 0x7fc43c0cf850 2023-10-16T14:30:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.864+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424004a00 con 0x7fc43c0cf850 2023-10-16T14:30:06.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.864+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c0c9400 con 0x7fc43c0cf850 2023-10-16T14:30:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.864+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c150bf0 con 0x7fc43c0cf850 2023-10-16T14:30:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.865+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c0c9920 con 0x7fc43c0cf850 2023-10-16T14:30:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.866+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc424004b60 con 0x7fc43c0cf850 2023-10-16T14:30:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.867+0000 7fc4337fe640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc41c06af90 0x7fc41c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:06.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.867+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fc424089120 con 0x7fc43c0cf850 2023-10-16T14:30:06.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.868+0000 7fc43ad9d640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc41c06af90 0x7fc41c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:06.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.869+0000 7fc43ad9d640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc41c06af90 0x7fc41c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc42c035000 tx=0x7fc42c030000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:06.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:06.871+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc43c0c9920 con 0x7fc43c0cf850 2023-10-16T14:30:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.139+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fc43c0c9920 con 0x7fc43c0cf850 2023-10-16T14:30:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.693+0000 7fc4337fe640 1 -- 172.21.15.29:0/4080615288 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v491) v1 ==== 157+0+0 (secure 0 0 0) 0x7fc43c0c9920 con 0x7fc43c0cf850 2023-10-16T14:30:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc41c06af90 msgr2=0x7fc41c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc41c06af90 0x7fc41c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc42c035000 tx=0x7fc42c030000 comp rx=0 tx=0).stop 2023-10-16T14:30:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 msgr2=0x7fc43c0cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 0x7fc43c0cb7a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c3c0 tx=0x7fc42400c7c0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 shutdown_connections 2023-10-16T14:30:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc41c06af90 0x7fc41c06d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc43c0cf850 0x7fc43c0cb7a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc43c15a9c0 0x7fc43c0cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 --2- 172.21.15.29:0/4080615288 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc43c0b2370 0x7fc43c1506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.696+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 >> 172.21.15.29:0/4080615288 conn(0x7fc43c0cd370 msgr2=0x7fc43c152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.697+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 shutdown_connections 2023-10-16T14:30:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.697+0000 7fc440a4a640 1 -- 172.21.15.29:0/4080615288 wait complete. 2023-10-16T14:30:07.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:30:07.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.956+0000 7fc547fff640 1 Processor -- start 2023-10-16T14:30:07.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.956+0000 7fc547fff640 1 -- start start 2023-10-16T14:30:07.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.958+0000 7fc547fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5480d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.958+0000 7fc547fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54815d150 con 0x7fc5480b8480 2023-10-16T14:30:07.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.958+0000 7fc547fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5480d1180 con 0x7fc5480cded0 2023-10-16T14:30:07.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.958+0000 7fc547fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5480d1460 con 0x7fc54815cd70 2023-10-16T14:30:07.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.958+0000 7fc5467fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5480b8480 0x7fc5480cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53342/0 (socket says 172.21.15.29:53342) 2023-10-16T14:30:07.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.958+0000 7fc5467fc640 1 -- 172.21.15.29:0/1182075352 learned_addr learned my addr 172.21.15.29:0/1182075352 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:07.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc546ffd640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5480d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842779452 0 0) 0x7fc54815d150 con 0x7fc5480b8480 2023-10-16T14:30:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534003520 con 0x7fc5480b8480 2023-10-16T14:30:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990327293 0 0) 0x7fc534003520 con 0x7fc5480b8480 2023-10-16T14:30:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 msgr2=0x7fc5480d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5480d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 >> v1:172.21.15.170:6789/0 conn(0x7fc5480cded0 legacy=0x7fc5480b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5480d1740 con 0x7fc5480b8480 2023-10-16T14:30:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc530001670 con 0x7fc5480b8480 2023-10-16T14:30:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.959+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 >> v1:172.21.15.29:6789/0 conn(0x7fc5480b8480 legacy=0x7fc5480cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc545ffb640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5340039a0 0x7fc534003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc545ffb640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc545ffb640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc534004d00 0x7fc534005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5480d1460 con 0x7fc5340039a0 2023-10-16T14:30:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53400ba30 con 0x7fc534004d00 2023-10-16T14:30:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53400bbf0 con 0x7fc54815cd70 2023-10-16T14:30:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc546ffd640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5340039a0 0x7fc534003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc5467fc640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc546ffd640 1 -- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 msgr2=0x7fc5340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc546ffd640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc546ffd640 1 -- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc534004d00 msgr2=0x7fc534005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc546ffd640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc534004d00 0x7fc534005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc546ffd640 1 -- 172.21.15.29:0/1182075352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc534003520 con 0x7fc5340039a0 2023-10-16T14:30:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.960+0000 7fc5467fc640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815cd70 0x7fc5340047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.961+0000 7fc546ffd640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5340039a0 0x7fc534003dc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc538009f90 tx=0x7fc5340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b7edff79f5eaee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53802cbd0 con 0x7fc5340039a0 2023-10-16T14:30:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53800f040 con 0x7fc5340039a0 2023-10-16T14:30:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc545ffb640 1 -- 172.21.15.29:0/1182075352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc538034440 con 0x7fc5340039a0 2023-10-16T14:30:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc547fff640 1 -- 172.21.15.29:0/1182075352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5340039a0 msgr2=0x7fc534003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc547fff640 1 --2- 172.21.15.29:0/1182075352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5340039a0 0x7fc534003dc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc538009f90 tx=0x7fc5340057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc5477fe640 1 -- 172.21.15.29:0/1182075352 reap_dead start 2023-10-16T14:30:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc547fff640 1 -- 172.21.15.29:0/1182075352 shutdown_connections 2023-10-16T14:30:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc547fff640 1 -- 172.21.15.29:0/1182075352 >> 172.21.15.29:0/1182075352 conn(0x7fc5480cf570 msgr2=0x7fc5480cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.962+0000 7fc547fff640 1 -- 172.21.15.29:0/1182075352 shutdown_connections 2023-10-16T14:30:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.963+0000 7fc547fff640 1 -- 172.21.15.29:0/1182075352 wait complete. 2023-10-16T14:30:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.964+0000 7fc547fff640 1 Processor -- start 2023-10-16T14:30:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.964+0000 7fc547fff640 1 -- start start 2023-10-16T14:30:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.965+0000 7fc547fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5340039a0 0x7fc5481f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.965+0000 7fc547fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815dd50 0x7fc54815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.965+0000 7fc547fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 0x7fc54815f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.965+0000 7fc547fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc548150a40 con 0x7fc54815e1d0 2023-10-16T14:30:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.965+0000 7fc547fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5480a9590 con 0x7fc5340039a0 2023-10-16T14:30:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.965+0000 7fc547fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54815e730 con 0x7fc54815dd50 2023-10-16T14:30:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 0x7fc54815f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 0x7fc54815f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51578/0 (socket says 172.21.15.29:51578) 2023-10-16T14:30:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 -- 172.21.15.29:0/824414963 learned_addr learned my addr 172.21.15.29:0/824414963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc546ffd640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5340039a0 0x7fc5481f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5467fc640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815dd50 0x7fc54815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 -- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815dd50 msgr2=0x7fc54815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815dd50 0x7fc54815ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 -- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5340039a0 msgr2=0x7fc5481f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5340039a0 0x7fc5481f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5477fe640 1 -- 172.21.15.29:0/824414963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54815ea10 con 0x7fc54815e1d0 2023-10-16T14:30:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.966+0000 7fc5467fc640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815dd50 0x7fc54815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:07.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.967+0000 7fc5477fe640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 0x7fc54815f230 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc5480b56a0 tx=0x7fc54000c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:07.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.967+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540004220 con 0x7fc54815e1d0 2023-10-16T14:30:07.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.967+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5400043a0 con 0x7fc54815e1d0 2023-10-16T14:30:07.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.967+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540007310 con 0x7fc54815e1d0 2023-10-16T14:30:07.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.967+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5480a9800 con 0x7fc54815e1d0 2023-10-16T14:30:07.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.967+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54815f950 con 0x7fc54815e1d0 2023-10-16T14:30:07.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.968+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5480a9a20 con 0x7fc54815e1d0 2023-10-16T14:30:07.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.969+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc540007470 con 0x7fc54815e1d0 2023-10-16T14:30:07.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.970+0000 7fc53ffff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc52806afc0 0x7fc52806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:07.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.970+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fc540088760 con 0x7fc54815e1d0 2023-10-16T14:30:07.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.971+0000 7fc546ffd640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc52806afc0 0x7fc52806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:07.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.972+0000 7fc546ffd640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc52806afc0 0x7fc52806d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc538036000 tx=0x7fc53802e000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:07.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:07.974+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5400536d0 con 0x7fc54815e1d0 2023-10-16T14:30:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.170+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc54804e9b0 con 0x7fc54815e1d0 2023-10-16T14:30:08.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.177+0000 7fc53ffff640 1 -- 172.21.15.29:0/824414963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc540056cf0 con 0x7fc54815e1d0 2023-10-16T14:30:08.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc52806afc0 msgr2=0x7fc52806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc52806afc0 0x7fc52806d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc538036000 tx=0x7fc53802e000 comp rx=0 tx=0).stop 2023-10-16T14:30:08.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 msgr2=0x7fc54815f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 0x7fc54815f230 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc5480b56a0 tx=0x7fc54000c570 comp rx=0 tx=0).stop 2023-10-16T14:30:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 shutdown_connections 2023-10-16T14:30:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc52806afc0 0x7fc52806d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc54815e1d0 0x7fc54815f230 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc54815dd50 0x7fc54815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 --2- 172.21.15.29:0/824414963 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5340039a0 0x7fc5481f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 >> 172.21.15.29:0/824414963 conn(0x7fc5480cf570 msgr2=0x7fc5480b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:08.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 shutdown_connections 2023-10-16T14:30:08.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.180+0000 7fc547fff640 1 -- 172.21.15.29:0/824414963 wait complete. 2023-10-16T14:30:08.204 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:30:08.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:30:08.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:30:08.475 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:30:08.476 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:30:08.476 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:30:08.540 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:30:08.541 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:30:08.541 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:30:08.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:30:08.598 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:30:08.598 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:30:08.663 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:30:08.664 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:30:08.664 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:30:08.723 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:30:08.724 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:30:08.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:30:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.963+0000 7eff1eda7640 1 Processor -- start 2023-10-16T14:30:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.963+0000 7eff1eda7640 1 -- start start 2023-10-16T14:30:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1eda7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7eff180ad970 0x7eff180cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1eda7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1815b9d0 con 0x7eff1806c040 2023-10-16T14:30:08.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1eda7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1815bcb0 con 0x7eff180ccfb0 2023-10-16T14:30:08.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1eda7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff1815bf90 con 0x7eff180ad970 2023-10-16T14:30:08.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1dda5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7eff1806c040 0x7eff1806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53346/0 (socket says 172.21.15.29:53346) 2023-10-16T14:30:08.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1dda5640 1 -- 172.21.15.29:0/1293867363 learned_addr learned my addr 172.21.15.29:0/1293867363 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:08.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305131151 0 0) 0x7eff1815b9d0 con 0x7eff1806c040 2023-10-16T14:30:08.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.965+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff04003520 con 0x7eff1806c040 2023-10-16T14:30:08.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1d5a4640 1 --2- 172.21.15.29:0/1293867363 >> v2:172.21.15.170:3300/0 conn(0x7eff180ad970 0x7eff180cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455033300 0 0) 0x7eff04003520 con 0x7eff1806c040 2023-10-16T14:30:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 >> v2:172.21.15.170:3300/0 conn(0x7eff180ad970 msgr2=0x7eff180cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 --2- 172.21.15.29:0/1293867363 >> v2:172.21.15.170:3300/0 conn(0x7eff180ad970 0x7eff180cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 >> v1:172.21.15.170:6789/0 conn(0x7eff180ccfb0 legacy=0x7eff180cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180cc270 con 0x7eff1806c040 2023-10-16T14:30:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7eff14001670 con 0x7eff1806c040 2023-10-16T14:30:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 >> v1:172.21.15.29:6789/0 conn(0x7eff1806c040 legacy=0x7eff1806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff04003970 0x7eff04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff040047c0 0x7eff04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04005180 0x7eff0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1815bf90 con 0x7eff04003970 2023-10-16T14:30:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0400c0d0 con 0x7eff040047c0 2023-10-16T14:30:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0400c290 con 0x7eff04005180 2023-10-16T14:30:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.966+0000 7eff1dda5640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff04003970 0x7eff04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1dda5640 1 -- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04005180 msgr2=0x7eff0400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1dda5640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04005180 0x7eff0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1dda5640 1 -- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff040047c0 msgr2=0x7eff04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1d5a4640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff040047c0 0x7eff04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1dda5640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff040047c0 0x7eff04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1dda5640 1 -- 172.21.15.29:0/1293867363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff04003520 con 0x7eff04003970 2023-10-16T14:30:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1e5a6640 1 -- 172.21.15.29:0/1293867363 reap_dead start 2023-10-16T14:30:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.967+0000 7eff1d5a4640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff040047c0 0x7eff04004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.968+0000 7eff1dda5640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff04003970 0x7eff04003d90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7eff1400c160 tx=0x7eff040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7f194fc4ecf05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.968+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14005ad0 con 0x7eff04003970 2023-10-16T14:30:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.968+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14005c50 con 0x7eff04003970 2023-10-16T14:30:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.968+0000 7eff1cda3640 1 -- 172.21.15.29:0/1293867363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1402c870 con 0x7eff04003970 2023-10-16T14:30:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 -- 172.21.15.29:0/1293867363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff04003970 msgr2=0x7eff04003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff04003970 0x7eff04003d90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7eff1400c160 tx=0x7eff040057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 -- 172.21.15.29:0/1293867363 shutdown_connections 2023-10-16T14:30:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1293867363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff04003970 0x7eff04003d90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 -- 172.21.15.29:0/1293867363 >> 172.21.15.29:0/1293867363 conn(0x7eff180ce6a0 msgr2=0x7eff180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:08.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 -- 172.21.15.29:0/1293867363 shutdown_connections 2023-10-16T14:30:08.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.969+0000 7eff1eda7640 1 -- 172.21.15.29:0/1293867363 wait complete. 2023-10-16T14:30:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.971+0000 7eff1eda7640 1 Processor -- start 2023-10-16T14:30:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.971+0000 7eff1eda7640 1 -- start start 2023-10-16T14:30:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1eda7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04003970 0x7eff18149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1eda7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff181e63c0 0x7eff181e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1eda7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 0x7eff180ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1eda7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff181e6d80 con 0x7eff180ae0f0 2023-10-16T14:30:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1eda7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180aee60 con 0x7eff181e63c0 2023-10-16T14:30:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1eda7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff180af140 con 0x7eff04003970 2023-10-16T14:30:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1e5a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 0x7eff180ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1e5a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 0x7eff180ae5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51594/0 (socket says 172.21.15.29:51594) 2023-10-16T14:30:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.972+0000 7eff1e5a6640 1 -- 172.21.15.29:0/1974023682 learned_addr learned my addr 172.21.15.29:0/1974023682 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:08.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1d5a4640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff181e63c0 0x7eff181e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:08.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1e5a6640 1 -- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04003970 msgr2=0x7eff18149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1e5a6640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04003970 0x7eff18149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1e5a6640 1 -- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff181e63c0 msgr2=0x7eff181e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1e5a6640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff181e63c0 0x7eff181e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1e5a6640 1 -- 172.21.15.29:0/1974023682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180af420 con 0x7eff180ae0f0 2023-10-16T14:30:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1e5a6640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 0x7eff180ae5a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7eff180cb060 tx=0x7eff1000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10005d40 con 0x7eff180ae0f0 2023-10-16T14:30:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1000ced0 con 0x7eff180ae0f0 2023-10-16T14:30:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10005ea0 con 0x7eff180ae0f0 2023-10-16T14:30:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff180af700 con 0x7eff180ae0f0 2023-10-16T14:30:08.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.973+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff180aeae0 con 0x7eff180ae0f0 2023-10-16T14:30:08.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.975+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1806b330 con 0x7eff180ae0f0 2023-10-16T14:30:08.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.976+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7eff1000a850 con 0x7eff180ae0f0 2023-10-16T14:30:08.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.977+0000 7eff0e7fc640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efefc06afc0 0x7efefc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.977+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7eff100885c0 con 0x7eff180ae0f0 2023-10-16T14:30:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.977+0000 7eff1dda5640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efefc06afc0 0x7efefc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.979+0000 7eff1dda5640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efefc06afc0 0x7efefc06d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff14034000 tx=0x7eff14007000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:08.981+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff10053660 con 0x7eff180ae0f0 2023-10-16T14:30:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.212+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff1804e9b0 con 0x7eff180ae0f0 2023-10-16T14:30:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.213+0000 7eff0e7fc640 1 -- 172.21.15.29:0/1974023682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3702 (secure 0 0 0) 0x7eff10017e90 con 0x7eff180ae0f0 2023-10-16T14:30:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:09.217 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1485160087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1485160087},{"type":"v1","addr":"172.21.15.29:6834","nonce":1485160087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1254713888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1254713888},{"type":"v1","addr":"172.21.15.170:6836","nonce":1254713888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3935625902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3935625902},{"type":"v1","addr":"172.21.15.170:6837","nonce":3935625902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/788456623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":788456623},{"type":"v1","addr":"172.21.15.29:6835","nonce":788456623}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2023-10-16T14:30:09.218 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2023-10-16T14:30:09.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.215+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efefc06afc0 msgr2=0x7efefc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:09.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.215+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efefc06afc0 0x7efefc06d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff14034000 tx=0x7eff14007000 comp rx=0 tx=0).stop 2023-10-16T14:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.215+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 msgr2=0x7eff180ae5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.215+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 0x7eff180ae5a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7eff180cb060 tx=0x7eff1000cb90 comp rx=0 tx=0).stop 2023-10-16T14:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 shutdown_connections 2023-10-16T14:30:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7efefc06afc0 0x7efefc06d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff180ae0f0 0x7eff180ae5a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff181e63c0 0x7eff181e6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 --2- 172.21.15.29:0/1974023682 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff04003970 0x7eff18149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 >> 172.21.15.29:0/1974023682 conn(0x7eff180ce6a0 msgr2=0x7eff180c1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:09.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 shutdown_connections 2023-10-16T14:30:09.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.216+0000 7eff1eda7640 1 -- 172.21.15.29:0/1974023682 wait complete. 2023-10-16T14:30:09.241 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:30:09.241 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:30:09.241 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:30:09.242 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:30:09.242 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:30:09.242 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:30:09.243 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:30:09.243 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:30:09.243 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:30:09.243 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:30:09.244 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:30:09.244 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:30:09.244 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:30:09.245 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:30:09.245 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:30:09.245 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:30:09.260 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:30:09.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:30:09.266 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:30:09.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:30:09.274 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:30:09.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:30:09.279 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:30:09.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:30:09.313 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:30:09.314 DEBUG:teuthology.parallel:result is None 2023-10-16T14:30:09.321 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:30:09.322 DEBUG:teuthology.parallel:result is None 2023-10-16T14:30:09.326 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:30:09.327 DEBUG:teuthology.parallel:result is None 2023-10-16T14:30:09.334 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:30:09.334 DEBUG:teuthology.parallel:result is None 2023-10-16T14:30:09.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:30:09.479 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:30:09.484 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:30:09.495 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:30:09.496 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:30:09.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.529+0000 7f800a7c5640 1 Processor -- start 2023-10-16T14:30:09.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.529+0000 7f800a7c5640 1 -- start start 2023-10-16T14:30:09.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f800a7c5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f800406a4b0 0x7f80040ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:09.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f800a7c5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f80040c9ca0 0x7f80040ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:09.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f800a7c5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80040cb030 con 0x7f800415ad70 2023-10-16T14:30:09.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f800a7c5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040cae30 con 0x7f800406a4b0 2023-10-16T14:30:09.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f800a7c5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040c91c0 con 0x7f80040c9ca0 2023-10-16T14:30:09.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f7ffbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f800406a4b0 0x7f80040ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:09.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f8003fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f800415ad70 0x7f8004069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53362/0 (socket says 172.21.15.29:53362) 2023-10-16T14:30:09.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f7ffbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f800406a4b0 0x7f80040ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51642/0 (socket says 172.21.15.29:51642) 2023-10-16T14:30:09.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f8003fff640 1 -- 172.21.15.29:0/3025721524 learned_addr learned my addr 172.21.15.29:0/3025721524 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:09.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.530+0000 7f8008d62640 1 --2- 172.21.15.29:0/3025721524 >> v2:172.21.15.170:3300/0 conn(0x7f80040c9ca0 0x7f80040ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:09.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.531+0000 7f7ffbfff640 1 -- 172.21.15.29:0/3025721524 >> v2:172.21.15.170:3300/0 conn(0x7f80040c9ca0 msgr2=0x7f80040ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:09.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.531+0000 7f7ffbfff640 1 --2- 172.21.15.29:0/3025721524 >> v2:172.21.15.170:3300/0 conn(0x7f80040c9ca0 0x7f80040ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.531+0000 7f7ffbfff640 1 -- 172.21.15.29:0/3025721524 >> v1:172.21.15.29:6789/0 conn(0x7f800415ad70 legacy=0x7f8004069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:09.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.531+0000 7f7ffbfff640 1 -- 172.21.15.29:0/3025721524 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80040c94a0 con 0x7f800406a4b0 2023-10-16T14:30:09.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.532+0000 7f7ffbfff640 1 --2- 172.21.15.29:0/3025721524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800406a4b0 0x7f80040ca3b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f800406f270 tx=0x7f7fec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e621940434769359 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:09.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.532+0000 7f80037fe640 1 -- 172.21.15.29:0/3025721524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec031830 con 0x7f800406a4b0 2023-10-16T14:30:09.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.532+0000 7f80037fe640 1 -- 172.21.15.29:0/3025721524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec031990 con 0x7f800406a4b0 2023-10-16T14:30:09.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.532+0000 7f80037fe640 1 -- 172.21.15.29:0/3025721524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec031d20 con 0x7f800406a4b0 2023-10-16T14:30:09.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 -- 172.21.15.29:0/3025721524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800406a4b0 msgr2=0x7f80040ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:09.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 --2- 172.21.15.29:0/3025721524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800406a4b0 0x7f80040ca3b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f800406f270 tx=0x7f7fec02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 -- 172.21.15.29:0/3025721524 shutdown_connections 2023-10-16T14:30:09.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 --2- 172.21.15.29:0/3025721524 >> v2:172.21.15.170:3300/0 conn(0x7f80040c9ca0 0x7f80040ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 --2- 172.21.15.29:0/3025721524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800406a4b0 0x7f80040ca3b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 -- 172.21.15.29:0/3025721524 >> 172.21.15.29:0/3025721524 conn(0x7f80040ce9b0 msgr2=0x7f80040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:09.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 -- 172.21.15.29:0/3025721524 shutdown_connections 2023-10-16T14:30:09.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.533+0000 7f800a7c5640 1 -- 172.21.15.29:0/3025721524 wait complete. 2023-10-16T14:30:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.535+0000 7f800a7c5640 1 Processor -- start 2023-10-16T14:30:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.535+0000 7f800a7c5640 1 -- start start 2023-10-16T14:30:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f800a7c5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f800406a4b0 0x7f80041f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f800a7c5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80040c9ca0 0x7f80040cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f800a7c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 0x7f8004158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f800a7c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040b1700 con 0x7f800415ad70 2023-10-16T14:30:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f800a7c5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80041586b0 con 0x7f80040c9ca0 2023-10-16T14:30:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f800a7c5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80041f2bd0 con 0x7f800406a4b0 2023-10-16T14:30:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f8008d62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 0x7f8004158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f8008d62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 0x7f8004158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51648/0 (socket says 172.21.15.29:51648) 2023-10-16T14:30:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f8008d62640 1 -- 172.21.15.29:0/1080734716 learned_addr learned my addr 172.21.15.29:0/1080734716 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.536+0000 7f8003fff640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f800406a4b0 0x7f80041f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f7ffbfff640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80040c9ca0 0x7f80040cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f8008d62640 1 -- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f800406a4b0 msgr2=0x7f80041f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f8008d62640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f800406a4b0 0x7f80041f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f8008d62640 1 -- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80040c9ca0 msgr2=0x7f80040cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f8008d62640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80040c9ca0 0x7f80040cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f8008d62640 1 -- 172.21.15.29:0/1080734716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec009400 con 0x7f800415ad70 2023-10-16T14:30:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f8008d62640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 0x7f8004158170 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00efc0 tx=0x7f7ffc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc014480 con 0x7f800415ad70 2023-10-16T14:30:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.537+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc00f040 con 0x7f800415ad70 2023-10-16T14:30:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.538+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc014820 con 0x7f800415ad70 2023-10-16T14:30:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.538+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80041f2eb0 con 0x7f800415ad70 2023-10-16T14:30:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.538+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80041f3470 con 0x7f800415ad70 2023-10-16T14:30:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.539+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80040b3ca0 con 0x7f800415ad70 2023-10-16T14:30:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.540+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7ffc014980 con 0x7f800415ad70 2023-10-16T14:30:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.541+0000 7f80017fa640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7fe406af90 0x7f7fe406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.541+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f7ffc08a5a0 con 0x7f800415ad70 2023-10-16T14:30:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.541+0000 7f8003fff640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7fe406af90 0x7f7fe406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.542+0000 7f8003fff640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7fe406af90 0x7f7fe406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4005880 tx=0x7f7ff4009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.545+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ffc055550 con 0x7f800415ad70 2023-10-16T14:30:09.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:09.753+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f800404e160 con 0x7f800415ad70 2023-10-16T14:30:11.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.137+0000 7f80017fa640 1 -- 172.21.15.29:0/1080734716 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7ffc058b70 con 0x7f800415ad70 2023-10-16T14:30:11.142 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:30:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7fe406af90 msgr2=0x7f7fe406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7fe406af90 0x7f7fe406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4005880 tx=0x7f7ff4009040 comp rx=0 tx=0).stop 2023-10-16T14:30:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 msgr2=0x7f8004158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 0x7f8004158170 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00efc0 tx=0x7f7ffc00c490 comp rx=0 tx=0).stop 2023-10-16T14:30:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 shutdown_connections 2023-10-16T14:30:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7fe406af90 0x7f7fe406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f800415ad70 0x7f8004158170 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80040c9ca0 0x7f80040cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 --2- 172.21.15.29:0/1080734716 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f800406a4b0 0x7f80041f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 >> 172.21.15.29:0/1080734716 conn(0x7f80040ce9b0 msgr2=0x7f80040afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 shutdown_connections 2023-10-16T14:30:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.142+0000 7f800a7c5640 1 -- 172.21.15.29:0/1080734716 wait complete. 2023-10-16T14:30:11.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:30:11.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.382+0000 7f701cdd7640 1 Processor -- start 2023-10-16T14:30:11.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.382+0000 7f701cdd7640 1 -- start start 2023-10-16T14:30:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.383+0000 7f701cdd7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f701815ae80 0x7f701815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.383+0000 7f701cdd7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701815ed60 con 0x7f70180ce390 2023-10-16T14:30:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.383+0000 7f701cdd7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180707e0 con 0x7f701815aaa0 2023-10-16T14:30:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.383+0000 7f701cdd7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701806b810 con 0x7f701815ae80 2023-10-16T14:30:11.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f700dd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f701815ae80 0x7f701815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f7016d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70180ce390 0x7f7018070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53364/0 (socket says 172.21.15.29:53364) 2023-10-16T14:30:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f7016d9d640 1 -- 172.21.15.29:0/378740016 learned_addr learned my addr 172.21.15.29:0/378740016 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f7015d9b640 1 -- 172.21.15.29:0/378740016 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650725737 0 0) 0x7f701815ed60 con 0x7f70180ce390 2023-10-16T14:30:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f7015d9b640 1 -- 172.21.15.29:0/378740016 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004003520 con 0x7f70180ce390 2023-10-16T14:30:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f700dd9b640 1 -- 172.21.15.29:0/378740016 >> v1:172.21.15.170:6789/0 conn(0x7f701815aaa0 legacy=0x7f70180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f700dd9b640 1 -- 172.21.15.29:0/378740016 >> v1:172.21.15.29:6789/0 conn(0x7f70180ce390 legacy=0x7f7018070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.384+0000 7f700dd9b640 1 -- 172.21.15.29:0/378740016 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701815ef90 con 0x7f701815ae80 2023-10-16T14:30:11.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.386+0000 7f700dd9b640 1 --2- 172.21.15.29:0/378740016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f701815ae80 0x7f701815e820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7000009d90 tx=0x7f700002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcc80fba3205831 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:11.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.386+0000 7f7015d9b640 1 -- 172.21.15.29:0/378740016 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700002ccf0 con 0x7f701815ae80 2023-10-16T14:30:11.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.386+0000 7f7015d9b640 1 -- 172.21.15.29:0/378740016 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700002ce50 con 0x7f701815ae80 2023-10-16T14:30:11.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.386+0000 7f7015d9b640 1 -- 172.21.15.29:0/378740016 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000034640 con 0x7f701815ae80 2023-10-16T14:30:11.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.386+0000 7f701cdd7640 1 -- 172.21.15.29:0/378740016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f701815ae80 msgr2=0x7f701815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.386+0000 7f701cdd7640 1 --2- 172.21.15.29:0/378740016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f701815ae80 0x7f701815e820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7000009d90 tx=0x7f700002bf40 comp rx=0 tx=0).stop 2023-10-16T14:30:11.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.387+0000 7f701cdd7640 1 -- 172.21.15.29:0/378740016 shutdown_connections 2023-10-16T14:30:11.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.387+0000 7f701cdd7640 1 --2- 172.21.15.29:0/378740016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f701815ae80 0x7f701815e820 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.387+0000 7f701cdd7640 1 -- 172.21.15.29:0/378740016 >> 172.21.15.29:0/378740016 conn(0x7f70180cb410 msgr2=0x7f70180cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:11.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.387+0000 7f701cdd7640 1 -- 172.21.15.29:0/378740016 shutdown_connections 2023-10-16T14:30:11.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.387+0000 7f701cdd7640 1 -- 172.21.15.29:0/378740016 wait complete. 2023-10-16T14:30:11.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.389+0000 7f701cdd7640 1 Processor -- start 2023-10-16T14:30:11.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.389+0000 7f701cdd7640 1 -- start start 2023-10-16T14:30:11.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701cdd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 0x7f7018157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701cdd7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f701815aaa0 0x7f701815b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701cdd7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f701815ae80 0x7f70180c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701cdd7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701815db80 con 0x7f70180ce390 2023-10-16T14:30:11.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701cdd7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7018158490 con 0x7f701815aaa0 2023-10-16T14:30:11.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701cdd7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701815cc60 con 0x7f701815ae80 2023-10-16T14:30:11.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 0x7f7018157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 0x7f7018157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51666/0 (socket says 172.21.15.29:51666) 2023-10-16T14:30:11.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f701659c640 1 -- 172.21.15.29:0/2916120836 learned_addr learned my addr 172.21.15.29:0/2916120836 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f700dd9b640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f701815aaa0 0x7f701815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.390+0000 7f7016d9d640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f701815ae80 0x7f70180c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701659c640 1 -- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f701815ae80 msgr2=0x7f70180c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701659c640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f701815ae80 0x7f70180c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701659c640 1 -- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f701815aaa0 msgr2=0x7f701815b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701659c640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f701815aaa0 0x7f701815b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701659c640 1 -- 172.21.15.29:0/2916120836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000009a70 con 0x7f70180ce390 2023-10-16T14:30:11.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701659c640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 0x7f7018157f50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f700800c400 tx=0x7f700800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:11.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70080052d0 con 0x7f70180ce390 2023-10-16T14:30:11.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008005450 con 0x7f70180ce390 2023-10-16T14:30:11.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008007260 con 0x7f70180ce390 2023-10-16T14:30:11.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.391+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701815b7d0 con 0x7f70180ce390 2023-10-16T14:30:11.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.392+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701815d1e0 con 0x7f70180ce390 2023-10-16T14:30:11.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.393+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701804e160 con 0x7f70180ce390 2023-10-16T14:30:11.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.394+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70080073c0 con 0x7f70180ce390 2023-10-16T14:30:11.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.395+0000 7f700f7fe640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6ff806af90 0x7f6ff806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.395+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f70080888e0 con 0x7f70180ce390 2023-10-16T14:30:11.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.395+0000 7f700dd9b640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6ff806af90 0x7f6ff806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.397+0000 7f700dd9b640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6ff806af90 0x7f6ff806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7000036000 tx=0x7f7000030000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.399+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7008053b30 con 0x7f70180ce390 2023-10-16T14:30:11.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.666+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f70180cdc60 con 0x7f70180ce390 2023-10-16T14:30:11.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.667+0000 7f700f7fe640 1 -- 172.21.15.29:0/2916120836 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7008057150 con 0x7f70180ce390 2023-10-16T14:30:11.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:11.672 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:30:11.672 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:30:11.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.672 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:30:11.672 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:30:11.673 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.673 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.673 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:30:11.673 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:30:11.673 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.674 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.674 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.674 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.674 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:30:11.674 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:30:11.675 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.675 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.675 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:30:11.675 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:30:11.675 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.676 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:30:11.676 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:30:11.677 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.677 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.677 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:30:11.677 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:30:11.677 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.678 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:30:11.678 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:30:11.678 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.679 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.679 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:30:11.679 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:30:11.679 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.679 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.680 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.680 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:30:11.680 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:30:11.680 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.681 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.681 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.681 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.681 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.682 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:30:11.682 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:30:11.682 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.683 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.683 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.683 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:30:11.683 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:30:11.684 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.684 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.684 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.685 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:30:11.685 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:30:11.685 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.686 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.686 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.686 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.686 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.687 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:30:11.687 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:30:11.687 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.687 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.687 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.688 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.688 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.688 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:30:11.688 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:30:11.689 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.689 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.689 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.689 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.689 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.690 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:30:11.690 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:30:11.690 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.691 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.691 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.691 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.691 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.692 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:30:11.692 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:30:11.692 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.692 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:30:11.692 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:30:11.693 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.693 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.693 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:30:11.693 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:30:11.694 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.694 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.694 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:30:11.694 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:30:11.694 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.695 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:30:11.695 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:30:11.695 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.696 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.696 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:30:11.696 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:30:11.696 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:30:11.696 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.697 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:30:11.697 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:30:11.697 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.698 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.698 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:30:11.698 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:30:11.698 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:30:11.698 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.699 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:30:11.699 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:30:11.699 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.699 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:30:11.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:30:11.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:30:11.700 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:30:11.700 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.701 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:30:11.701 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:30:11.701 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.701 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:30:11.702 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:30:11.702 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:30:11.702 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.702 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.703 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:30:11.703 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:30:11.703 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:30:11.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:30:11.704 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:30:11.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:11.704 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:30:11.704 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:30:11.705 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:30:11.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:30:11.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:30:11.705 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:30:11.705 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.706 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:11.706 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:30:11.706 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:30:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6ff806af90 msgr2=0x7f6ff806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6ff806af90 0x7f6ff806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7000036000 tx=0x7f7000030000 comp rx=0 tx=0).stop 2023-10-16T14:30:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 msgr2=0x7f7018157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 0x7f7018157f50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f700800c400 tx=0x7f700800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 shutdown_connections 2023-10-16T14:30:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f6ff806af90 0x7f6ff806d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f701815ae80 0x7f70180c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f701815aaa0 0x7f701815b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 --2- 172.21.15.29:0/2916120836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70180ce390 0x7f7018157f50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 >> 172.21.15.29:0/2916120836 conn(0x7f70180cb410 msgr2=0x7f701806ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 shutdown_connections 2023-10-16T14:30:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.670+0000 7f701cdd7640 1 -- 172.21.15.29:0/2916120836 wait complete. 2023-10-16T14:30:11.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:30:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.932+0000 7f4dca9b8640 1 Processor -- start 2023-10-16T14:30:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.933+0000 7f4dca9b8640 1 -- start start 2023-10-16T14:30:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.934+0000 7f4dca9b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415dde0 0x7f4dc40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.934+0000 7f4dca9b8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc4160a90 con 0x7f4dc4070f20 2023-10-16T14:30:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dca9b8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40cd900 con 0x7f4dc415c9f0 2023-10-16T14:30:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dca9b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40cdec0 con 0x7f4dc415dde0 2023-10-16T14:30:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dca1b7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4dc4070f20 0x7f4dc40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53366/0 (socket says 172.21.15.29:53366) 2023-10-16T14:30:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dca1b7640 1 -- 172.21.15.29:0/4002755442 learned_addr learned my addr 172.21.15.29:0/4002755442 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dc91b5640 1 --2- 172.21.15.29:0/4002755442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415dde0 0x7f4dc40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dc89b4640 1 -- 172.21.15.29:0/4002755442 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360447304 0 0) 0x7f4dc4160a90 con 0x7f4dc4070f20 2023-10-16T14:30:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.935+0000 7f4dc89b4640 1 -- 172.21.15.29:0/4002755442 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db0003520 con 0x7f4dc4070f20 2023-10-16T14:30:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.936+0000 7f4dc91b5640 1 -- 172.21.15.29:0/4002755442 >> v1:172.21.15.170:6789/0 conn(0x7f4dc415c9f0 legacy=0x7f4dc40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.936+0000 7f4dc91b5640 1 -- 172.21.15.29:0/4002755442 >> v1:172.21.15.29:6789/0 conn(0x7f4dc4070f20 legacy=0x7f4dc40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.936+0000 7f4dc91b5640 1 -- 172.21.15.29:0/4002755442 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40cdbe0 con 0x7f4dc415dde0 2023-10-16T14:30:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.937+0000 7f4dc91b5640 1 --2- 172.21.15.29:0/4002755442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415dde0 0x7f4dc40709e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4db40097d0 tx=0x7f4db402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a74e29db5c22187 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dc89b4640 1 -- 172.21.15.29:0/4002755442 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4030800 con 0x7f4dc415dde0 2023-10-16T14:30:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dc89b4640 1 -- 172.21.15.29:0/4002755442 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4030960 con 0x7f4dc415dde0 2023-10-16T14:30:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dc89b4640 1 -- 172.21.15.29:0/4002755442 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4030cf0 con 0x7f4dc415dde0 2023-10-16T14:30:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 -- 172.21.15.29:0/4002755442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415dde0 msgr2=0x7f4dc40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/4002755442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415dde0 0x7f4dc40709e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4db40097d0 tx=0x7f4db402b940 comp rx=0 tx=0).stop 2023-10-16T14:30:11.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 -- 172.21.15.29:0/4002755442 shutdown_connections 2023-10-16T14:30:11.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/4002755442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415dde0 0x7f4dc40709e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 -- 172.21.15.29:0/4002755442 >> 172.21.15.29:0/4002755442 conn(0x7f4dc40ce6a0 msgr2=0x7f4dc40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:11.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 -- 172.21.15.29:0/4002755442 shutdown_connections 2023-10-16T14:30:11.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.938+0000 7f4dca9b8640 1 -- 172.21.15.29:0/4002755442 wait complete. 2023-10-16T14:30:11.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.940+0000 7f4dca9b8640 1 Processor -- start 2023-10-16T14:30:11.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.940+0000 7f4dca9b8640 1 -- start start 2023-10-16T14:30:11.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.941+0000 7f4dca9b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 0x7f4dc40c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.941+0000 7f4dca9b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415c9f0 0x7f4dc4069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.941+0000 7f4dca9b8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc415dde0 0x7f4dc41f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.941+0000 7f4dca9b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40c1aa0 con 0x7f4dc4070f20 2023-10-16T14:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.941+0000 7f4dca9b8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc41f6870 con 0x7f4dc415dde0 2023-10-16T14:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.941+0000 7f4dca9b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc406a8e0 con 0x7f4dc415c9f0 2023-10-16T14:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 0x7f4dc40c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 0x7f4dc40c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51674/0 (socket says 172.21.15.29:51674) 2023-10-16T14:30:11.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 -- 172.21.15.29:0/2477158185 learned_addr learned my addr 172.21.15.29:0/2477158185 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc91b5640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415c9f0 0x7f4dc4069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dca1b7640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc415dde0 0x7f4dc41f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 -- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415c9f0 msgr2=0x7f4dc4069fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415c9f0 0x7f4dc4069fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 -- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc415dde0 msgr2=0x7f4dc41f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:11.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc415dde0 0x7f4dc41f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 -- 172.21.15.29:0/2477158185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4009400 con 0x7f4dc4070f20 2023-10-16T14:30:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.942+0000 7f4dc99b6640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 0x7f4dc40c8270 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000c450 tx=0x7f4dc000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.943+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00052d0 con 0x7f4dc4070f20 2023-10-16T14:30:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.943+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc0005430 con 0x7f4dc4070f20 2023-10-16T14:30:11.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.943+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0012870 con 0x7f4dc4070f20 2023-10-16T14:30:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.943+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc41f6590 con 0x7f4dc4070f20 2023-10-16T14:30:11.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.943+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc41e8980 con 0x7f4dc4070f20 2023-10-16T14:30:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.944+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc404e9b0 con 0x7f4dc4070f20 2023-10-16T14:30:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.945+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4dc00129d0 con 0x7f4dc4070f20 2023-10-16T14:30:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.946+0000 7f4dba7fc640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4da806af90 0x7f4da806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:11.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.946+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4dc0088f90 con 0x7f4dc4070f20 2023-10-16T14:30:11.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.947+0000 7f4dc91b5640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4da806af90 0x7f4da806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:11.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.948+0000 7f4dc91b5640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4da806af90 0x7f4da806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4db4036000 tx=0x7f4db4008030 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:11.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:11.950+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4dc0054270 con 0x7f4dc4070f20 2023-10-16T14:30:12.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.141+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dc406a590 con 0x7f4dc4070f20 2023-10-16T14:30:12.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.143+0000 7f4dba7fc640 1 -- 172.21.15.29:0/2477158185 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+12857 (secure 0 0 0) 0x7f4dc0057890 con 0x7f4dc4070f20 2023-10-16T14:30:12.147 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:12.147 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":61,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:30:11.135045+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:30:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4da806af90 msgr2=0x7f4da806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4da806af90 0x7f4da806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4db4036000 tx=0x7f4db4008030 comp rx=0 tx=0).stop 2023-10-16T14:30:12.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 msgr2=0x7f4dc40c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 0x7f4dc40c8270 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000c450 tx=0x7f4dc000c910 comp rx=0 tx=0).stop 2023-10-16T14:30:12.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 shutdown_connections 2023-10-16T14:30:12.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f4da806af90 0x7f4da806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc415dde0 0x7f4dc41f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dc415c9f0 0x7f4dc4069fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 --2- 172.21.15.29:0/2477158185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dc4070f20 0x7f4dc40c8270 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.145+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 >> 172.21.15.29:0/2477158185 conn(0x7f4dc40ce6a0 msgr2=0x7f4dc415e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:12.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.146+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 shutdown_connections 2023-10-16T14:30:12.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.146+0000 7f4dca9b8640 1 -- 172.21.15.29:0/2477158185 wait complete. 2023-10-16T14:30:12.169 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:30:12.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:30:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.392+0000 7fefbbfff640 1 Processor -- start 2023-10-16T14:30:12.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.393+0000 7fefbbfff640 1 -- start start 2023-10-16T14:30:12.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefbc0b3ec0 0x7fefbc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbbfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fefbc0b43c0 0x7fefbc0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbbfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefbc06be30 con 0x7fefbc0b0ea0 2023-10-16T14:30:12.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbbfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefbc06ca00 con 0x7fefbc0b3ec0 2023-10-16T14:30:12.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbbfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefbc0ada10 con 0x7fefbc0b43c0 2023-10-16T14:30:12.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefbc0b3ec0 0x7fefbc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefbc0b3ec0 0x7fefbc0cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51678/0 (socket says 172.21.15.29:51678) 2023-10-16T14:30:12.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefbaffd640 1 -- 172.21.15.29:0/1558521451 learned_addr learned my addr 172.21.15.29:0/1558521451 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:12.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.394+0000 7fefba7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fefbc0b0ea0 0x7fefbc15a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53380/0 (socket says 172.21.15.29:53380) 2023-10-16T14:30:12.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.395+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/1558521451 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0b43c0 0x7fefbc0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.395+0000 7fefbaffd640 1 -- 172.21.15.29:0/1558521451 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0b43c0 msgr2=0x7fefbc0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.395+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1558521451 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0b43c0 0x7fefbc0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.395+0000 7fefbaffd640 1 -- 172.21.15.29:0/1558521451 >> v1:172.21.15.29:6789/0 conn(0x7fefbc0b0ea0 legacy=0x7fefbc15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.395+0000 7fefb9ffb640 1 -- 172.21.15.29:0/1558521451 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065804837 0 0) 0x7fefbc06be30 con 0x7fefbc0b0ea0 2023-10-16T14:30:12.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.395+0000 7fefbaffd640 1 -- 172.21.15.29:0/1558521451 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc06c060 con 0x7fefbc0b3ec0 2023-10-16T14:30:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.396+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1558521451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b3ec0 0x7fefbc0cd260 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fefbc071b60 tx=0x7fefac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9294a6da70bacc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.396+0000 7fefb9ffb640 1 -- 172.21.15.29:0/1558521451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac02ca80 con 0x7fefbc0b3ec0 2023-10-16T14:30:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefb9ffb640 1 -- 172.21.15.29:0/1558521451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac02cbe0 con 0x7fefbc0b3ec0 2023-10-16T14:30:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefb9ffb640 1 -- 172.21.15.29:0/1558521451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac0343f0 con 0x7fefbc0b3ec0 2023-10-16T14:30:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 -- 172.21.15.29:0/1558521451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b3ec0 msgr2=0x7fefbc0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1558521451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b3ec0 0x7fefbc0cd260 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fefbc071b60 tx=0x7fefac02bee0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 -- 172.21.15.29:0/1558521451 shutdown_connections 2023-10-16T14:30:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1558521451 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0b43c0 0x7fefbc0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1558521451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b3ec0 0x7fefbc0cd260 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 -- 172.21.15.29:0/1558521451 >> 172.21.15.29:0/1558521451 conn(0x7fefbc0b0170 msgr2=0x7fefbc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:12.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 -- 172.21.15.29:0/1558521451 shutdown_connections 2023-10-16T14:30:12.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.397+0000 7fefbbfff640 1 -- 172.21.15.29:0/1558521451 wait complete. 2023-10-16T14:30:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.399+0000 7fefbbfff640 1 Processor -- start 2023-10-16T14:30:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.399+0000 7fefbbfff640 1 -- start start 2023-10-16T14:30:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.400+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc0b0ea0 0x7fefbc153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.400+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b3ec0 0x7fefbc070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.400+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 0x7fefbc06f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.400+0000 7fefbbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc154100 con 0x7fefbc0b43c0 2023-10-16T14:30:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.400+0000 7fefbbfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0712a0 con 0x7fefbc0b0ea0 2023-10-16T14:30:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.400+0000 7fefbbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc06fcc0 con 0x7fefbc0b3ec0 2023-10-16T14:30:12.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 0x7fefbc06f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 0x7fefbc06f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51692/0 (socket says 172.21.15.29:51692) 2023-10-16T14:30:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 -- 172.21.15.29:0/3477882080 learned_addr learned my addr 172.21.15.29:0/3477882080 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbaffd640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc0b0ea0 0x7fefbc153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefba7fc640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b3ec0 0x7fefbc070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 -- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b3ec0 msgr2=0x7fefbc070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b3ec0 0x7fefbc070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 -- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc0b0ea0 msgr2=0x7fefbc153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc0b0ea0 0x7fefbc153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 -- 172.21.15.29:0/3477882080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac009b30 con 0x7fefbc0b43c0 2023-10-16T14:30:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbaffd640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc0b0ea0 0x7fefbc153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefba7fc640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b3ec0 0x7fefbc070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.401+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 0x7fefbc06f270 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fefb400d5d0 tx=0x7fefb400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.402+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40125f0 con 0x7fefbc0b43c0 2023-10-16T14:30:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.402+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb4012770 con 0x7fefbc0b43c0 2023-10-16T14:30:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.402+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc06ffa0 con 0x7fefbc0b43c0 2023-10-16T14:30:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.402+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4012b20 con 0x7fefbc0b43c0 2023-10-16T14:30:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.402+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc070560 con 0x7fefbc0b43c0 2023-10-16T14:30:12.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.403+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc06c600 con 0x7fefbc0b43c0 2023-10-16T14:30:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.404+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fefb4012c80 con 0x7fefbc0b43c0 2023-10-16T14:30:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.405+0000 7fefa37fe640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fefa806af90 0x7fefa806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.405+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fefb40898c0 con 0x7fefbc0b43c0 2023-10-16T14:30:12.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.406+0000 7fefbaffd640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fefa806af90 0x7fefa806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.407+0000 7fefbaffd640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fefa806af90 0x7fefa806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fefac035000 tx=0x7fefac030000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.409+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefbc06c600 con 0x7fefbc0b43c0 2023-10-16T14:30:12.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.601+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fefbc06c600 con 0x7fefbc0b43c0 2023-10-16T14:30:12.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.602+0000 7fefa37fe640 1 -- 172.21.15.29:0/3477882080 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefbc06c600 con 0x7fefbc0b43c0 2023-10-16T14:30:12.606 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:30:12.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fefa806af90 msgr2=0x7fefa806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fefa806af90 0x7fefa806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fefac035000 tx=0x7fefac030000 comp rx=0 tx=0).stop 2023-10-16T14:30:12.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 msgr2=0x7fefbc06f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 0x7fefbc06f270 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fefb400d5d0 tx=0x7fefb400da90 comp rx=0 tx=0).stop 2023-10-16T14:30:12.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 shutdown_connections 2023-10-16T14:30:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fefa806af90 0x7fefa806d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0b43c0 0x7fefbc06f270 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b3ec0 0x7fefbc070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 --2- 172.21.15.29:0/3477882080 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc0b0ea0 0x7fefbc153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 >> 172.21.15.29:0/3477882080 conn(0x7fefbc0b0170 msgr2=0x7fefbc0cd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:12.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 shutdown_connections 2023-10-16T14:30:12.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.605+0000 7fefbbfff640 1 -- 172.21.15.29:0/3477882080 wait complete. 2023-10-16T14:30:12.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:30:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.846+0000 7f2d6616e640 1 Processor -- start 2023-10-16T14:30:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.847+0000 7f2d6616e640 1 -- start start 2023-10-16T14:30:12.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.848+0000 7f2d6616e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2d6006c040 0x7f2d6006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.848+0000 7f2d6616e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d600ad970 0x7f2d6015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.848+0000 7f2d6616e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6006c960 con 0x7f2d6015ebc0 2023-10-16T14:30:12.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.848+0000 7f2d6616e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6015b9d0 con 0x7f2d6006c040 2023-10-16T14:30:12.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.848+0000 7f2d6616e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6015bcb0 con 0x7f2d600ad970 2023-10-16T14:30:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.849+0000 7f2d6496b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d600ad970 0x7f2d6015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.849+0000 7f2d6516c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2d6006c040 0x7f2d6006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.849+0000 7f2d6496b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d600ad970 0x7f2d6015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:35602/0 (socket says 172.21.15.29:35602) 2023-10-16T14:30:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.849+0000 7f2d6496b640 1 -- 172.21.15.29:0/1888003256 learned_addr learned my addr 172.21.15.29:0/1888003256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.850+0000 7f2d6516c640 1 -- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d600ad970 msgr2=0x7f2d6015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.850+0000 7f2d6516c640 1 --2- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d600ad970 0x7f2d6015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.850+0000 7f2d6516c640 1 -- 172.21.15.29:0/1888003256 >> v1:172.21.15.170:6789/0 conn(0x7f2d6015ebc0 legacy=0x7f2d600cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.850+0000 7f2d6516c640 1 -- 172.21.15.29:0/1888003256 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6015bf90 con 0x7f2d6006c040 2023-10-16T14:30:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.851+0000 7f2d6516c640 1 --2- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6006c040 0x7f2d6006c420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0097d0 tx=0x7f2d5c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98054eeaa925a17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.852+0000 7f2d57fff640 1 -- 172.21.15.29:0/1888003256 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c02d040 con 0x7f2d6006c040 2023-10-16T14:30:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.852+0000 7f2d57fff640 1 -- 172.21.15.29:0/1888003256 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5c031810 con 0x7f2d6006c040 2023-10-16T14:30:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.852+0000 7f2d57fff640 1 -- 172.21.15.29:0/1888003256 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c031ba0 con 0x7f2d6006c040 2023-10-16T14:30:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 -- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6006c040 msgr2=0x7f2d6006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 --2- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6006c040 0x7f2d6006c420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0097d0 tx=0x7f2d5c02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 -- 172.21.15.29:0/1888003256 shutdown_connections 2023-10-16T14:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 --2- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d600ad970 0x7f2d6015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 --2- 172.21.15.29:0/1888003256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6006c040 0x7f2d6006c420 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 -- 172.21.15.29:0/1888003256 >> 172.21.15.29:0/1888003256 conn(0x7f2d600ce6a0 msgr2=0x7f2d600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.853+0000 7f2d6616e640 1 -- 172.21.15.29:0/1888003256 shutdown_connections 2023-10-16T14:30:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.854+0000 7f2d6616e640 1 -- 172.21.15.29:0/1888003256 wait complete. 2023-10-16T14:30:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.856+0000 7f2d6616e640 1 Processor -- start 2023-10-16T14:30:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.856+0000 7f2d6616e640 1 -- start start 2023-10-16T14:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.857+0000 7f2d6616e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d6006c040 0x7f2d600aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.857+0000 7f2d6616e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 0x7f2d600c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6616e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6015ebc0 0x7f2d600c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6616e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600b0cd0 con 0x7f2d600ad970 2023-10-16T14:30:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6616e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600b0fb0 con 0x7f2d6015ebc0 2023-10-16T14:30:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6616e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600ae140 con 0x7f2d6006c040 2023-10-16T14:30:12.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6496b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 0x7f2d600c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6496b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 0x7f2d600c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51694/0 (socket says 172.21.15.29:51694) 2023-10-16T14:30:12.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6496b640 1 -- 172.21.15.29:0/474748441 learned_addr learned my addr 172.21.15.29:0/474748441 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:12.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6516c640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d6006c040 0x7f2d600aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6596d640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6015ebc0 0x7f2d600c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6496b640 1 -- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d6006c040 msgr2=0x7f2d600aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.858+0000 7f2d6496b640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d6006c040 0x7f2d600aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6496b640 1 -- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6015ebc0 msgr2=0x7f2d600c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6496b640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6015ebc0 0x7f2d600c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6496b640 1 -- 172.21.15.29:0/474748441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c009400 con 0x7f2d600ad970 2023-10-16T14:30:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6596d640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6015ebc0 0x7f2d600c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6496b640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 0x7f2d600c96c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000e840 tx=0x7f2d5000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5000c8e0 con 0x7f2d600ad970 2023-10-16T14:30:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5000ca40 con 0x7f2d600ad970 2023-10-16T14:30:12.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d600ae420 con 0x7f2d600ad970 2023-10-16T14:30:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5000cdf0 con 0x7f2d600ad970 2023-10-16T14:30:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.859+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d600af160 con 0x7f2d600ad970 2023-10-16T14:30:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.860+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6006b330 con 0x7f2d600ad970 2023-10-16T14:30:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.862+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d50013470 con 0x7f2d600ad970 2023-10-16T14:30:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.862+0000 7f2d55ffb640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f2d4406afc0 0x7f2d4406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.863+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2d50089580 con 0x7f2d600ad970 2023-10-16T14:30:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.863+0000 7f2d6516c640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f2d4406afc0 0x7f2d4406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.865+0000 7f2d6516c640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f2d4406afc0 0x7f2d4406d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c036000 tx=0x7f2d5c03c040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:12.867+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2d500547d0 con 0x7f2d600ad970 2023-10-16T14:30:13.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.065+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2d6004e9b0 con 0x7f2d600ad970 2023-10-16T14:30:13.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.065+0000 7f2d55ffb640 1 -- 172.21.15.29:0/474748441 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v61) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2d50057df0 con 0x7f2d600ad970 2023-10-16T14:30:13.070 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:13.070 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:30:13.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:13.072 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:30:13.072 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:30:13.072 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:13.073 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:30:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f2d4406afc0 msgr2=0x7f2d4406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f2d4406afc0 0x7f2d4406d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c036000 tx=0x7f2d5c03c040 comp rx=0 tx=0).stop 2023-10-16T14:30:13.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 msgr2=0x7f2d600c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 0x7f2d600c96c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000e840 tx=0x7f2d5000ed00 comp rx=0 tx=0).stop 2023-10-16T14:30:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 shutdown_connections 2023-10-16T14:30:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f2d4406afc0 0x7f2d4406d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d600ad970 0x7f2d600c96c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.068+0000 7f2d6616e640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d6015ebc0 0x7f2d600c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.069+0000 7f2d6616e640 1 --2- 172.21.15.29:0/474748441 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d6006c040 0x7f2d600aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.069+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 >> 172.21.15.29:0/474748441 conn(0x7f2d600ce6a0 msgr2=0x7f2d6006a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.069+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 shutdown_connections 2023-10-16T14:30:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.069+0000 7f2d6616e640 1 -- 172.21.15.29:0/474748441 wait complete. 2023-10-16T14:30:13.092 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:30:13.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:30:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.315+0000 7fcafd7f4640 1 Processor -- start 2023-10-16T14:30:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.315+0000 7fcafd7f4640 1 -- start start 2023-10-16T14:30:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.317+0000 7fcafd7f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 0x7fcaf80c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.317+0000 7fcafd7f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80c8e70 con 0x7fcaf815ad80 2023-10-16T14:30:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.317+0000 7fcafd7f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80b4170 con 0x7fcaf806d280 2023-10-16T14:30:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.317+0000 7fcafd7f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80adde0 con 0x7fcaf80cae40 2023-10-16T14:30:13.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.317+0000 7fcaf7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcaf815ad80 0x7fcaf806cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53394/0 (socket says 172.21.15.29:53394) 2023-10-16T14:30:13.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.317+0000 7fcaf7fff640 1 -- 172.21.15.29:0/2301954790 learned_addr learned my addr 172.21.15.29:0/2301954790 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:13.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883319117 0 0) 0x7fcaf80c8e70 con 0x7fcaf815ad80 2023-10-16T14:30:13.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae4003520 con 0x7fcaf815ad80 2023-10-16T14:30:13.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcafcff3640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 0x7fcaf80c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153460317 0 0) 0x7fcae4003520 con 0x7fcaf815ad80 2023-10-16T14:30:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 msgr2=0x7fcaf80c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 0x7fcaf80c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 >> v1:172.21.15.170:6789/0 conn(0x7fcaf806d280 legacy=0x7fcaf80c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80b4410 con 0x7fcaf815ad80 2023-10-16T14:30:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fcae8001670 con 0x7fcaf815ad80 2023-10-16T14:30:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcafcff3640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 0x7fcaf80c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 >> v1:172.21.15.29:6789/0 conn(0x7fcaf815ad80 legacy=0x7fcaf806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcae4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcae40047c0 0x7fcae4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 0x7fcae40051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80adde0 con 0x7fcae4003a10 2023-10-16T14:30:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae400bb10 con 0x7fcae40047c0 2023-10-16T14:30:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae400bcd0 con 0x7fcaf80cae40 2023-10-16T14:30:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.318+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcae4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf77fe640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcae40047c0 0x7fcae4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf7fff640 1 -- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 msgr2=0x7fcae40051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf80cae40 0x7fcae40051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf7fff640 1 -- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcae40047c0 msgr2=0x7fcae4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcae40047c0 0x7fcae4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf7fff640 1 -- 172.21.15.29:0/2301954790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4003520 con 0x7fcae4003a10 2023-10-16T14:30:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.319+0000 7fcaf77fe640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcae40047c0 0x7fcae4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.320+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcae4003e30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c240 tx=0x7fcae40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc01ffc89e7b3d44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.320+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae8005ad0 con 0x7fcae4003a10 2023-10-16T14:30:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.320+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae8005c30 con 0x7fcae4003a10 2023-10-16T14:30:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.320+0000 7fcaf6ffd640 1 -- 172.21.15.29:0/2301954790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae802c8b0 con 0x7fcae4003a10 2023-10-16T14:30:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafd7f4640 1 -- 172.21.15.29:0/2301954790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 msgr2=0x7fcae4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/2301954790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcae4003e30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c240 tx=0x7fcae40057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafcff3640 1 -- 172.21.15.29:0/2301954790 reap_dead start 2023-10-16T14:30:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafd7f4640 1 -- 172.21.15.29:0/2301954790 shutdown_connections 2023-10-16T14:30:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafd7f4640 1 -- 172.21.15.29:0/2301954790 >> 172.21.15.29:0/2301954790 conn(0x7fcaf80ce820 msgr2=0x7fcaf80cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafd7f4640 1 -- 172.21.15.29:0/2301954790 shutdown_connections 2023-10-16T14:30:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.321+0000 7fcafd7f4640 1 -- 172.21.15.29:0/2301954790 wait complete. 2023-10-16T14:30:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.323+0000 7fcafd7f4640 1 Processor -- start 2023-10-16T14:30:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.323+0000 7fcafd7f4640 1 -- start start 2023-10-16T14:30:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcafd7f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcaf80af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcafd7f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf814bd00 0x7fcaf80c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcafd7f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcaf80c4820 0x7fcaf80a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcafd7f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80c4ca0 con 0x7fcae4003a10 2023-10-16T14:30:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcafd7f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80ad2e0 con 0x7fcaf80c4820 2023-10-16T14:30:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcafd7f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80c80c0 con 0x7fcaf814bd00 2023-10-16T14:30:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcaf80af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcaf80af670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51722/0 (socket says 172.21.15.29:51722) 2023-10-16T14:30:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 -- 172.21.15.29:0/1411474048 learned_addr learned my addr 172.21.15.29:0/1411474048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 -- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf814bd00 msgr2=0x7fcaf80c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf814bd00 0x7fcaf80c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 -- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcaf80c4820 msgr2=0x7fcaf80a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:13.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcaf80c4820 0x7fcaf80a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:13.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.324+0000 7fcaf7fff640 1 -- 172.21.15.29:0/1411474048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80ac520 con 0x7fcae4003a10 2023-10-16T14:30:13.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.325+0000 7fcaf7fff640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcaf80af670 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcae8001930 tx=0x7fcae8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:13.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.325+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae8039070 con 0x7fcae4003a10 2023-10-16T14:30:13.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.325+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae802dd50 con 0x7fcae4003a10 2023-10-16T14:30:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.325+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae803c040 con 0x7fcae4003a10 2023-10-16T14:30:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.325+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf80ac800 con 0x7fcae4003a10 2023-10-16T14:30:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.325+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf80acdc0 con 0x7fcae4003a10 2023-10-16T14:30:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.326+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf804e9b0 con 0x7fcae4003a10 2023-10-16T14:30:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.327+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcae8032070 con 0x7fcae4003a10 2023-10-16T14:30:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.328+0000 7fcaf4ff9640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcad806afc0 0x7fcad806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.328+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcae80aaaa0 con 0x7fcae4003a10 2023-10-16T14:30:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.329+0000 7fcaf77fe640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcad806afc0 0x7fcad806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.330+0000 7fcaf77fe640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcad806afc0 0x7fcad806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcae0007fd0 tx=0x7fcae0009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.332+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcae8075d90 con 0x7fcae4003a10 2023-10-16T14:30:13.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:13.529+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcaf80aeb60 con 0x7fcae4003a10 2023-10-16T14:30:14.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.229+0000 7fcaf4ff9640 1 -- 172.21.15.29:0/1411474048 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v62) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcae8046070 con 0x7fcae4003a10 2023-10-16T14:30:14.233 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:30:14.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.232+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcad806afc0 msgr2=0x7fcad806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.232+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcad806afc0 0x7fcad806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcae0007fd0 tx=0x7fcae0009040 comp rx=0 tx=0).stop 2023-10-16T14:30:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.232+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 msgr2=0x7fcaf80af670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcaf80af670 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcae8001930 tx=0x7fcae8001a00 comp rx=0 tx=0).stop 2023-10-16T14:30:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 shutdown_connections 2023-10-16T14:30:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcad806afc0 0x7fcad806d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcaf80c4820 0x7fcaf80a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcaf814bd00 0x7fcaf80c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 --2- 172.21.15.29:0/1411474048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae4003a10 0x7fcaf80af670 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 >> 172.21.15.29:0/1411474048 conn(0x7fcaf80ce820 msgr2=0x7fcaf80c9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 shutdown_connections 2023-10-16T14:30:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.233+0000 7fcafd7f4640 1 -- 172.21.15.29:0/1411474048 wait complete. 2023-10-16T14:30:14.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:30:14.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.475+0000 7f68f9d4e640 1 Processor -- start 2023-10-16T14:30:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.475+0000 7f68f9d4e640 1 -- start start 2023-10-16T14:30:14.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.476+0000 7f68f9d4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f68f4159ac0 0x7f68f415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.476+0000 7f68f9d4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68f40704e0 0x7f68f4070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.476+0000 7f68f9d4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f415e5c0 con 0x7f68f40cc270 2023-10-16T14:30:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.476+0000 7f68f9d4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f4070eb0 con 0x7f68f40704e0 2023-10-16T14:30:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.476+0000 7f68f9d4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f406b810 con 0x7f68f4159ac0 2023-10-16T14:30:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68f40704e0 0x7f68f4070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:14.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68f40704e0 0x7f68f4070970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60724/0 (socket says 172.21.15.29:60724) 2023-10-16T14:30:14.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 -- 172.21.15.29:0/1303711518 learned_addr learned my addr 172.21.15.29:0/1303711518 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f8d4c640 1 --2- 172.21.15.29:0/1303711518 >> v2:172.21.15.170:3300/0 conn(0x7f68f4159ac0 0x7f68f415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 -- 172.21.15.29:0/1303711518 >> v2:172.21.15.170:3300/0 conn(0x7f68f4159ac0 msgr2=0x7f68f415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 --2- 172.21.15.29:0/1303711518 >> v2:172.21.15.170:3300/0 conn(0x7f68f4159ac0 0x7f68f415e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 -- 172.21.15.29:0/1303711518 >> v1:172.21.15.170:6789/0 conn(0x7f68f40cc270 legacy=0x7f68f40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.477+0000 7f68f954d640 1 -- 172.21.15.29:0/1303711518 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40cc810 con 0x7f68f40704e0 2023-10-16T14:30:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.479+0000 7f68f954d640 1 --2- 172.21.15.29:0/1303711518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f40704e0 0x7f68f4070970 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f68f4071b00 tx=0x7f68ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21abcddea60303a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.479+0000 7f68f37fe640 1 -- 172.21.15.29:0/1303711518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec02ca80 con 0x7f68f40704e0 2023-10-16T14:30:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.479+0000 7f68f37fe640 1 -- 172.21.15.29:0/1303711518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec02cbe0 con 0x7f68f40704e0 2023-10-16T14:30:14.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.479+0000 7f68f37fe640 1 -- 172.21.15.29:0/1303711518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0343f0 con 0x7f68f40704e0 2023-10-16T14:30:14.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 -- 172.21.15.29:0/1303711518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f40704e0 msgr2=0x7f68f4070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:14.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/1303711518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f40704e0 0x7f68f4070970 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f68f4071b00 tx=0x7f68ec02bee0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 -- 172.21.15.29:0/1303711518 shutdown_connections 2023-10-16T14:30:14.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/1303711518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f40704e0 0x7f68f4070970 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/1303711518 >> v2:172.21.15.170:3300/0 conn(0x7f68f4159ac0 0x7f68f415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 -- 172.21.15.29:0/1303711518 >> 172.21.15.29:0/1303711518 conn(0x7f68f40ce6a0 msgr2=0x7f68f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:14.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 -- 172.21.15.29:0/1303711518 shutdown_connections 2023-10-16T14:30:14.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.480+0000 7f68f9d4e640 1 -- 172.21.15.29:0/1303711518 wait complete. 2023-10-16T14:30:14.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.482+0000 7f68f9d4e640 1 Processor -- start 2023-10-16T14:30:14.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.482+0000 7f68f9d4e640 1 -- start start 2023-10-16T14:30:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f9d4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68f40704e0 0x7f68f4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f9d4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68f40cc270 0x7f68f40ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f9d4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 0x7f68f40cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f9d4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f40b27a0 con 0x7f68f4159ac0 2023-10-16T14:30:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f9d4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f40b3600 con 0x7f68f40704e0 2023-10-16T14:30:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f9d4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f4157100 con 0x7f68f40cc270 2023-10-16T14:30:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f954d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 0x7f68f40cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f954d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 0x7f68f40cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60734/0 (socket says 172.21.15.29:60734) 2023-10-16T14:30:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.483+0000 7f68f954d640 1 -- 172.21.15.29:0/3833036419 learned_addr learned my addr 172.21.15.29:0/3833036419 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f8d4c640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68f40704e0 0x7f68f4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f3fff640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68f40cc270 0x7f68f40ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f954d640 1 -- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68f40cc270 msgr2=0x7f68f40ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f954d640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68f40cc270 0x7f68f40ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f954d640 1 -- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68f40704e0 msgr2=0x7f68f4156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f954d640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68f40704e0 0x7f68f4156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f954d640 1 -- 172.21.15.29:0/3833036419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec009b30 con 0x7f68f4159ac0 2023-10-16T14:30:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f3fff640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68f40cc270 0x7f68f40ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.484+0000 7f68f954d640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 0x7f68f40cd040 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009f40 tx=0x7f68ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.485+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec009510 con 0x7f68f4159ac0 2023-10-16T14:30:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.485+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec03e450 con 0x7f68f4159ac0 2023-10-16T14:30:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.485+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f40b29f0 con 0x7f68f4159ac0 2023-10-16T14:30:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.485+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec03e7e0 con 0x7f68f4159ac0 2023-10-16T14:30:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.485+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f40b2fb0 con 0x7f68f4159ac0 2023-10-16T14:30:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.486+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f40b3790 con 0x7f68f4159ac0 2023-10-16T14:30:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.487+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f68ec02cd50 con 0x7f68f4159ac0 2023-10-16T14:30:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.488+0000 7f68f17fa640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f68d806af90 0x7f68d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.488+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f68ec0ac340 con 0x7f68f4159ac0 2023-10-16T14:30:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.489+0000 7f68f8d4c640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f68d806af90 0x7f68d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.490+0000 7f68f8d4c640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f68d806af90 0x7f68d806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f68e8006fd0 tx=0x7f68e8008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.492+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f68ec077530 con 0x7f68f4159ac0 2023-10-16T14:30:14.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:14.685+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f68f415e5c0 con 0x7f68f4159ac0 2023-10-16T14:30:15.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.241+0000 7f68f17fa640 1 -- 172.21.15.29:0/3833036419 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v63) v1 ==== 170+0+0 (secure 0 0 0) 0x7f68f415e5c0 con 0x7f68f4159ac0 2023-10-16T14:30:15.245 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:30:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.245+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f68d806af90 msgr2=0x7f68d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.245+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f68d806af90 0x7f68d806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f68e8006fd0 tx=0x7f68e8008040 comp rx=0 tx=0).stop 2023-10-16T14:30:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.245+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 msgr2=0x7f68f40cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.245+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 0x7f68f40cd040 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009f40 tx=0x7f68ec004a40 comp rx=0 tx=0).stop 2023-10-16T14:30:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 shutdown_connections 2023-10-16T14:30:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f68d806af90 0x7f68d806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68f4159ac0 0x7f68f40cd040 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68f40cc270 0x7f68f40ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 --2- 172.21.15.29:0/3833036419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f68f40704e0 0x7f68f4156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 >> 172.21.15.29:0/3833036419 conn(0x7f68f40ce6a0 msgr2=0x7f68f406bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 shutdown_connections 2023-10-16T14:30:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.246+0000 7f68f9d4e640 1 -- 172.21.15.29:0/3833036419 wait complete. 2023-10-16T14:30:15.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:30:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.476+0000 7f215674c640 1 Processor -- start 2023-10-16T14:30:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.476+0000 7f215674c640 1 -- start start 2023-10-16T14:30:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f215674c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21500caac0 0x7f21500ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f215674c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500cb030 con 0x7f215015ad70 2023-10-16T14:30:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f215674c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500c91c0 con 0x7f215006a4b0 2023-10-16T14:30:15.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f215674c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21500c94a0 con 0x7f21500caac0 2023-10-16T14:30:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f2154ce9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21500caac0 0x7f21500ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f2154ce9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21500caac0 0x7f21500ad4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60742/0 (socket says 172.21.15.29:60742) 2023-10-16T14:30:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.478+0000 7f2154ce9640 1 -- 172.21.15.29:0/355248715 learned_addr learned my addr 172.21.15.29:0/355248715 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:15.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.479+0000 7f2154ce9640 1 -- 172.21.15.29:0/355248715 >> v1:172.21.15.170:6789/0 conn(0x7f215006a4b0 legacy=0x7f21500ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:15.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.479+0000 7f2154ce9640 1 -- 172.21.15.29:0/355248715 >> v1:172.21.15.29:6789/0 conn(0x7f215015ad70 legacy=0x7f2150069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:15.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.479+0000 7f2154ce9640 1 -- 172.21.15.29:0/355248715 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21500c9780 con 0x7f21500caac0 2023-10-16T14:30:15.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.480+0000 7f2154ce9640 1 --2- 172.21.15.29:0/355248715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21500caac0 0x7f21500ad4d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2148009e50 tx=0x7f214802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2f155ebd8f3cce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:15.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.480+0000 7f214f7fe640 1 -- 172.21.15.29:0/355248715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214802cb10 con 0x7f21500caac0 2023-10-16T14:30:15.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.480+0000 7f214f7fe640 1 -- 172.21.15.29:0/355248715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214802cc70 con 0x7f21500caac0 2023-10-16T14:30:15.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.480+0000 7f214f7fe640 1 -- 172.21.15.29:0/355248715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480343f0 con 0x7f21500caac0 2023-10-16T14:30:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 -- 172.21.15.29:0/355248715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21500caac0 msgr2=0x7f21500ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:15.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 --2- 172.21.15.29:0/355248715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21500caac0 0x7f21500ad4d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2148009e50 tx=0x7f214802bf70 comp rx=0 tx=0).stop 2023-10-16T14:30:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 -- 172.21.15.29:0/355248715 shutdown_connections 2023-10-16T14:30:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 --2- 172.21.15.29:0/355248715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21500caac0 0x7f21500ad4d0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 -- 172.21.15.29:0/355248715 >> 172.21.15.29:0/355248715 conn(0x7f21500ce9b0 msgr2=0x7f21500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 -- 172.21.15.29:0/355248715 shutdown_connections 2023-10-16T14:30:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.481+0000 7f215674c640 1 -- 172.21.15.29:0/355248715 wait complete. 2023-10-16T14:30:15.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.483+0000 7f215674c640 1 Processor -- start 2023-10-16T14:30:15.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.483+0000 7f215674c640 1 -- start start 2023-10-16T14:30:15.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f215674c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 0x7f21501f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:15.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f215674c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21500caac0 0x7f21501f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:15.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f215674c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215015ad70 0x7f215015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:15.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f215674c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21501f2b00 con 0x7f215006a4b0 2023-10-16T14:30:15.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f215674c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215015cd10 con 0x7f215015ad70 2023-10-16T14:30:15.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f215674c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215015cf60 con 0x7f21500caac0 2023-10-16T14:30:15.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f214ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 0x7f21501f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:15.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f214ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 0x7f21501f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60752/0 (socket says 172.21.15.29:60752) 2023-10-16T14:30:15.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.484+0000 7f214ffff640 1 -- 172.21.15.29:0/527713804 learned_addr learned my addr 172.21.15.29:0/527713804 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f2154ce9640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215015ad70 0x7f215015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f2147fff640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21500caac0 0x7f21501f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f214ffff640 1 -- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21500caac0 msgr2=0x7f21501f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f214ffff640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21500caac0 0x7f21501f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f214ffff640 1 -- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215015ad70 msgr2=0x7f215015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:15.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f214ffff640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215015ad70 0x7f215015bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:15.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f214ffff640 1 -- 172.21.15.29:0/527713804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148009b30 con 0x7f215006a4b0 2023-10-16T14:30:15.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.485+0000 7f214ffff640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 0x7f21501f2080 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f214000c400 tx=0x7f214000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:15.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.486+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400052d0 con 0x7f215006a4b0 2023-10-16T14:30:15.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.486+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2140005430 con 0x7f215006a4b0 2023-10-16T14:30:15.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.486+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140003830 con 0x7f215006a4b0 2023-10-16T14:30:15.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.486+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215015d240 con 0x7f215006a4b0 2023-10-16T14:30:15.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.486+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215015d800 con 0x7f215006a4b0 2023-10-16T14:30:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.487+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215004e160 con 0x7f215006a4b0 2023-10-16T14:30:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.488+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2140003990 con 0x7f215006a4b0 2023-10-16T14:30:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.489+0000 7f214d7fa640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f213006af90 0x7f213006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.489+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2140088d30 con 0x7f215006a4b0 2023-10-16T14:30:15.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.490+0000 7f2147fff640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f213006af90 0x7f213006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:15.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.491+0000 7f2147fff640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f213006af90 0x7f213006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21500c21d0 tx=0x7f2138009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:15.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.496+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2140053c90 con 0x7f215006a4b0 2023-10-16T14:30:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:15.725+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f215015d990 con 0x7f215006a4b0 2023-10-16T14:30:16.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.245+0000 7f214d7fa640 1 -- 172.21.15.29:0/527713804 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f21400170c0 con 0x7f215006a4b0 2023-10-16T14:30:16.250 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 6 and data pool 7 2023-10-16T14:30:16.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.252+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f213006af90 msgr2=0x7f213006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.252+0000 7f215674c640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f213006af90 0x7f213006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21500c21d0 tx=0x7f2138009040 comp rx=0 tx=0).stop 2023-10-16T14:30:16.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.252+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 msgr2=0x7f21501f2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.252+0000 7f215674c640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 0x7f21501f2080 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f214000c400 tx=0x7f214000c8c0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.255+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 shutdown_connections 2023-10-16T14:30:16.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.255+0000 7f215674c640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f213006af90 0x7f213006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.255+0000 7f215674c640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215015ad70 0x7f215015bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.255+0000 7f215674c640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21500caac0 0x7f21501f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.255+0000 7f215674c640 1 --2- 172.21.15.29:0/527713804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f215006a4b0 0x7f21501f2080 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.255+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 >> 172.21.15.29:0/527713804 conn(0x7f21500ce9b0 msgr2=0x7f2150069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.258+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 shutdown_connections 2023-10-16T14:30:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.259+0000 7f215674c640 1 -- 172.21.15.29:0/527713804 wait complete. 2023-10-16T14:30:16.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:30:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.482+0000 7f9100f8a640 1 Processor -- start 2023-10-16T14:30:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.482+0000 7f9100f8a640 1 -- start start 2023-10-16T14:30:16.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.483+0000 7f9100f8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.483+0000 7f9100f8a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f90fc15ae80 0x7f90fc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.483+0000 7f9100f8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f90fc0ccaa0 0x7f90fc070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.483+0000 7f9100f8a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc06b810 con 0x7f90fc15ae80 2023-10-16T14:30:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.483+0000 7f9100f8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc15ed60 con 0x7f90fc0ccaa0 2023-10-16T14:30:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.483+0000 7f9100f8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0707e0 con 0x7f90fc15aaa0 2023-10-16T14:30:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f90fc15ae80 0x7f90fc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f90fc15ae80 0x7f90fc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60768/0 (socket says 172.21.15.29:60768) 2023-10-16T14:30:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 -- 172.21.15.29:0/2588020673 learned_addr learned my addr 172.21.15.29:0/2588020673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 -- 172.21.15.29:0/2588020673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 msgr2=0x7f90fc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 --2- 172.21.15.29:0/2588020673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 -- 172.21.15.29:0/2588020673 >> v2:172.21.15.170:3300/0 conn(0x7f90fc0ccaa0 msgr2=0x7f90fc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 --2- 172.21.15.29:0/2588020673 >> v2:172.21.15.170:3300/0 conn(0x7f90fc0ccaa0 0x7f90fc070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.484+0000 7f90f9d9b640 1 -- 172.21.15.29:0/2588020673 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc15ef90 con 0x7f90fc15ae80 2023-10-16T14:30:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.485+0000 7f90f9d9b640 1 --2- 172.21.15.29:0/2588020673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc0cc560 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0ae3c0 tx=0x7f90e402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417e10e2f11612bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f90f959a640 1 -- 172.21.15.29:0/2588020673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031880 con 0x7f90fc15ae80 2023-10-16T14:30:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f90f959a640 1 -- 172.21.15.29:0/2588020673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e40319e0 con 0x7f90fc15ae80 2023-10-16T14:30:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f90f959a640 1 -- 172.21.15.29:0/2588020673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031d70 con 0x7f90fc15ae80 2023-10-16T14:30:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 -- 172.21.15.29:0/2588020673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 msgr2=0x7f90fc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 --2- 172.21.15.29:0/2588020673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc0cc560 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0ae3c0 tx=0x7f90e402b920 comp rx=0 tx=0).stop 2023-10-16T14:30:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 -- 172.21.15.29:0/2588020673 shutdown_connections 2023-10-16T14:30:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 --2- 172.21.15.29:0/2588020673 >> v2:172.21.15.170:3300/0 conn(0x7f90fc0ccaa0 0x7f90fc070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 --2- 172.21.15.29:0/2588020673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc0cc560 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 --2- 172.21.15.29:0/2588020673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.486+0000 7f9100f8a640 1 -- 172.21.15.29:0/2588020673 >> 172.21.15.29:0/2588020673 conn(0x7f90fc0cb410 msgr2=0x7f90fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:16.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.487+0000 7f9100f8a640 1 -- 172.21.15.29:0/2588020673 shutdown_connections 2023-10-16T14:30:16.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.487+0000 7f9100f8a640 1 -- 172.21.15.29:0/2588020673 wait complete. 2023-10-16T14:30:16.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.488+0000 7f9100f8a640 1 Processor -- start 2023-10-16T14:30:16.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.488+0000 7f9100f8a640 1 -- start start 2023-10-16T14:30:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.489+0000 7f9100f8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.489+0000 7f9100f8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc1f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.489+0000 7f9100f8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc1f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.489+0000 7f9100f8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc15bfa0 con 0x7f90fc15ae80 2023-10-16T14:30:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.489+0000 7f9100f8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc15c280 con 0x7f90fc0ccaa0 2023-10-16T14:30:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.489+0000 7f9100f8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc15c560 con 0x7f90fc15aaa0 2023-10-16T14:30:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc1f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc1f0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60774/0 (socket says 172.21.15.29:60774) 2023-10-16T14:30:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 -- 172.21.15.29:0/3011433213 learned_addr learned my addr 172.21.15.29:0/3011433213 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 -- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 msgr2=0x7f90fc1f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90f9d9b640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc1f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fa59c640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc1f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 -- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90fc0ccaa0 msgr2=0x7f90fc0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fad9d640 1 -- 172.21.15.29:0/3011433213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40094d0 con 0x7f90fc15ae80 2023-10-16T14:30:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.490+0000 7f90fa59c640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.491+0000 7f90fad9d640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc1f0c20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f90f400b970 tx=0x7f90f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.491+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400cb40 con 0x7f90fc15ae80 2023-10-16T14:30:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.491+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f400ccc0 con 0x7f90fc15ae80 2023-10-16T14:30:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.491+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc15c840 con 0x7f90fc15ae80 2023-10-16T14:30:16.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.491+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4014680 con 0x7f90fc15ae80 2023-10-16T14:30:16.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.491+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc0c8490 con 0x7f90fc15ae80 2023-10-16T14:30:16.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.492+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc0c8680 con 0x7f90fc15ae80 2023-10-16T14:30:16.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.493+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f90f401d3b0 con 0x7f90fc15ae80 2023-10-16T14:30:16.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.494+0000 7f90f2ffd640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f90dc06afc0 0x7f90dc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.494+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90f4089710 con 0x7f90fc15ae80 2023-10-16T14:30:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.495+0000 7f90fa59c640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f90dc06afc0 0x7f90dc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.496+0000 7f90fa59c640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f90dc06afc0 0x7f90dc06d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00a870 tx=0x7f90ec008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:16.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.498+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f90f40546a0 con 0x7f90fc15ae80 2023-10-16T14:30:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.695+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90fc04e160 con 0x7f90fc15ae80 2023-10-16T14:30:16.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.697+0000 7f90f2ffd640 1 -- 172.21.15.29:0/3011433213 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16526 (secure 0 0 0) 0x7f90f4057cc0 con 0x7f90fc15ae80 2023-10-16T14:30:16.701 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:16.701 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":64,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:30:15.727933+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:30:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f90dc06afc0 msgr2=0x7f90dc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f90dc06afc0 0x7f90dc06d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00a870 tx=0x7f90ec008040 comp rx=0 tx=0).stop 2023-10-16T14:30:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 msgr2=0x7f90fc1f0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc1f0c20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f90f400b970 tx=0x7f90f400be30 comp rx=0 tx=0).stop 2023-10-16T14:30:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 shutdown_connections 2023-10-16T14:30:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f90dc06afc0 0x7f90dc06d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90fc15ae80 0x7f90fc1f0c20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90fc15aaa0 0x7f90fc1f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 --2- 172.21.15.29:0/3011433213 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 >> 172.21.15.29:0/3011433213 conn(0x7f90fc0cb410 msgr2=0x7f90fc0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 shutdown_connections 2023-10-16T14:30:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.700+0000 7f9100f8a640 1 -- 172.21.15.29:0/3011433213 wait complete. 2023-10-16T14:30:16.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:30:16.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:30:16.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:30:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.936+0000 7fb16ae7c640 1 Processor -- start 2023-10-16T14:30:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.936+0000 7fb16ae7c640 1 -- start start 2023-10-16T14:30:16.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb16ae7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb16415aaa0 0x7fb16415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb16ae7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1640ce390 0x7fb164070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb16ae7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16415ed60 con 0x7fb16415ae80 2023-10-16T14:30:16.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb16ae7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16406b810 con 0x7fb1640ce390 2023-10-16T14:30:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb16ae7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1640707e0 con 0x7fb16415aaa0 2023-10-16T14:30:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb163fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb16415ae80 0x7fb1640cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39458/0 (socket says 172.21.15.29:39458) 2023-10-16T14:30:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb163fff640 1 -- 172.21.15.29:0/4266658749 learned_addr learned my addr 172.21.15.29:0/4266658749 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb169419640 1 --2- 172.21.15.29:0/4266658749 >> v2:172.21.15.29:3300/0 conn(0x7fb1640ce390 0x7fb164070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.938+0000 7fb168c18640 1 --2- 172.21.15.29:0/4266658749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb16415aaa0 0x7fb16415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.939+0000 7fb1637fe640 1 -- 172.21.15.29:0/4266658749 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709115247 0 0) 0x7fb16415ed60 con 0x7fb16415ae80 2023-10-16T14:30:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.939+0000 7fb169419640 1 -- 172.21.15.29:0/4266658749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb16415aaa0 msgr2=0x7fb16415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.939+0000 7fb169419640 1 --2- 172.21.15.29:0/4266658749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb16415aaa0 0x7fb16415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.939+0000 7fb169419640 1 -- 172.21.15.29:0/4266658749 >> v1:172.21.15.29:6789/0 conn(0x7fb16415ae80 legacy=0x7fb1640cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.939+0000 7fb169419640 1 -- 172.21.15.29:0/4266658749 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16415ef90 con 0x7fb1640ce390 2023-10-16T14:30:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.940+0000 7fb169419640 1 --2- 172.21.15.29:0/4266658749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640ce390 0x7fb164070f50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009d90 tx=0x7fb15c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b01ec402a8f4b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.940+0000 7fb1637fe640 1 -- 172.21.15.29:0/4266658749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c02cc60 con 0x7fb1640ce390 2023-10-16T14:30:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.940+0000 7fb1637fe640 1 -- 172.21.15.29:0/4266658749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c02cdc0 con 0x7fb1640ce390 2023-10-16T14:30:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.940+0000 7fb1637fe640 1 -- 172.21.15.29:0/4266658749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c034660 con 0x7fb1640ce390 2023-10-16T14:30:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 -- 172.21.15.29:0/4266658749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640ce390 msgr2=0x7fb164070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/4266658749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640ce390 0x7fb164070f50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009d90 tx=0x7fb15c02beb0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 -- 172.21.15.29:0/4266658749 shutdown_connections 2023-10-16T14:30:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/4266658749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1640ce390 0x7fb164070f50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/4266658749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb16415aaa0 0x7fb16415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 -- 172.21.15.29:0/4266658749 >> 172.21.15.29:0/4266658749 conn(0x7fb1640cb410 msgr2=0x7fb1640cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 -- 172.21.15.29:0/4266658749 shutdown_connections 2023-10-16T14:30:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.941+0000 7fb16ae7c640 1 -- 172.21.15.29:0/4266658749 wait complete. 2023-10-16T14:30:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.943+0000 7fb16ae7c640 1 Processor -- start 2023-10-16T14:30:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.943+0000 7fb16ae7c640 1 -- start start 2023-10-16T14:30:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.944+0000 7fb16ae7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1640ce390 0x7fb16406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.944+0000 7fb16ae7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb16415aaa0 0x7fb1640c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.944+0000 7fb16ae7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 0x7fb16414d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.944+0000 7fb16ae7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415d980 con 0x7fb16415ae80 2023-10-16T14:30:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.944+0000 7fb16ae7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415dc60 con 0x7fb16415aaa0 2023-10-16T14:30:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.944+0000 7fb16ae7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415df40 con 0x7fb1640ce390 2023-10-16T14:30:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 0x7fb16414d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 0x7fb16414d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60798/0 (socket says 172.21.15.29:60798) 2023-10-16T14:30:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 -- 172.21.15.29:0/907835531 learned_addr learned my addr 172.21.15.29:0/907835531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb168c18640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1640ce390 0x7fb16406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb163fff640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb16415aaa0 0x7fb1640c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 -- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1640ce390 msgr2=0x7fb16406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1640ce390 0x7fb16406cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 -- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb16415aaa0 msgr2=0x7fb1640c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb16415aaa0 0x7fb1640c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 -- 172.21.15.29:0/907835531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c009a70 con 0x7fb16415ae80 2023-10-16T14:30:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.945+0000 7fb169419640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 0x7fb16414d980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009d90 tx=0x7fb15c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.946+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c003b90 con 0x7fb16415ae80 2023-10-16T14:30:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.946+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c003cf0 con 0x7fb16415ae80 2023-10-16T14:30:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.946+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c004270 con 0x7fb16415ae80 2023-10-16T14:30:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.946+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1640c2540 con 0x7fb16415ae80 2023-10-16T14:30:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.946+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16406d430 con 0x7fb16415ae80 2023-10-16T14:30:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.947+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16404e160 con 0x7fb16415ae80 2023-10-16T14:30:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.948+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb15c03b050 con 0x7fb16415ae80 2023-10-16T14:30:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.949+0000 7fb1617fa640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fb14806af90 0x7fb14806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.950+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb15c0abdd0 con 0x7fb16415ae80 2023-10-16T14:30:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.950+0000 7fb168c18640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fb14806af90 0x7fb14806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.951+0000 7fb168c18640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fb14806af90 0x7fb14806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb158006fd0 tx=0x7fb158008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:16.954+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb15c076cd0 con 0x7fb16415ae80 2023-10-16T14:30:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.147+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb16415f520 con 0x7fb16415ae80 2023-10-16T14:30:17.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.150+0000 7fb1617fa640 1 -- 172.21.15.29:0/907835531 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16526 (secure 0 0 0) 0x7fb15c039090 con 0x7fb16415ae80 2023-10-16T14:30:17.155 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:17.155 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":64,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:30:15.727933+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:30:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fb14806af90 msgr2=0x7fb14806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fb14806af90 0x7fb14806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb158006fd0 tx=0x7fb158008040 comp rx=0 tx=0).stop 2023-10-16T14:30:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 msgr2=0x7fb16414d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 0x7fb16414d980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009d90 tx=0x7fb15c004a40 comp rx=0 tx=0).stop 2023-10-16T14:30:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 shutdown_connections 2023-10-16T14:30:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fb14806af90 0x7fb14806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16415ae80 0x7fb16414d980 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb16415aaa0 0x7fb1640c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 --2- 172.21.15.29:0/907835531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1640ce390 0x7fb16406cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 >> 172.21.15.29:0/907835531 conn(0x7fb1640cb410 msgr2=0x7fb16406f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 shutdown_connections 2023-10-16T14:30:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.153+0000 7fb16ae7c640 1 -- 172.21.15.29:0/907835531 wait complete. 2023-10-16T14:30:17.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:30:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.456+0000 7f7a6f98a640 1 Processor -- start 2023-10-16T14:30:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.456+0000 7f7a6f98a640 1 -- start start 2023-10-16T14:30:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.457+0000 7f7a6f98a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a680ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.457+0000 7f7a6f98a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680ce800 con 0x7f7a680ae260 2023-10-16T14:30:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.457+0000 7f7a6f98a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680cdec0 con 0x7f7a680ce350 2023-10-16T14:30:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.457+0000 7f7a6f98a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680acdd0 con 0x7f7a68160ad0 2023-10-16T14:30:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.458+0000 7f7a6df27640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7a680ae260 0x7f7a6806a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39462/0 (socket says 172.21.15.29:39462) 2023-10-16T14:30:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.458+0000 7f7a6df27640 1 -- 172.21.15.29:0/2603188285 learned_addr learned my addr 172.21.15.29:0/2603188285 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.458+0000 7f7a6d726640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a680ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.458+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082286222 0 0) 0x7f7a680ce800 con 0x7f7a680ae260 2023-10-16T14:30:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.458+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a54003520 con 0x7f7a680ae260 2023-10-16T14:30:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1573480477 0 0) 0x7f7a54003520 con 0x7f7a680ae260 2023-10-16T14:30:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 msgr2=0x7f7a680ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a680ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 >> v1:172.21.15.170:6789/0 conn(0x7f7a680ce350 legacy=0x7f7a68160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680ae740 con 0x7f7a680ae260 2023-10-16T14:30:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7a600019f0 con 0x7f7a680ae260 2023-10-16T14:30:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 >> v1:172.21.15.29:6789/0 conn(0x7f7a680ae260 legacy=0x7f7a6806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a54004d00 0x7f7a54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680acdd0 con 0x7f7a54003f20 2023-10-16T14:30:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5400ba60 con 0x7f7a54004d00 2023-10-16T14:30:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5400bc20 con 0x7f7a68160ad0 2023-10-16T14:30:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a6cf25640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.459+0000 7f7a6d726640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6df27640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a54004d00 0x7f7a54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6cf25640 1 -- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 msgr2=0x7f7a540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6cf25640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6cf25640 1 -- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a54004d00 msgr2=0x7f7a54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6cf25640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a54004d00 0x7f7a54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6cf25640 1 -- 172.21.15.29:0/2603188285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54003520 con 0x7f7a54003f20 2023-10-16T14:30:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.460+0000 7f7a6d726640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a68160ad0 0x7f7a540039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:17.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.461+0000 7f7a6cf25640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a540047c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7a58001720 tx=0x7f7a540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30a360aba9ea925 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:17.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.461+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a580052d0 con 0x7f7a54003f20 2023-10-16T14:30:17.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.461+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5802e050 con 0x7f7a54003f20 2023-10-16T14:30:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.461+0000 7f7a5ffff640 1 -- 172.21.15.29:0/2603188285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a58003830 con 0x7f7a54003f20 2023-10-16T14:30:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2603188285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 msgr2=0x7f7a540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2603188285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a540047c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7a58001720 tx=0x7f7a540057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6df27640 1 -- 172.21.15.29:0/2603188285 reap_dead start 2023-10-16T14:30:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2603188285 shutdown_connections 2023-10-16T14:30:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2603188285 >> 172.21.15.29:0/2603188285 conn(0x7f7a6806bc10 msgr2=0x7f7a6806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2603188285 shutdown_connections 2023-10-16T14:30:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.462+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2603188285 wait complete. 2023-10-16T14:30:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.464+0000 7f7a6f98a640 1 Processor -- start 2023-10-16T14:30:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.464+0000 7f7a6f98a640 1 -- start start 2023-10-16T14:30:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6f98a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a6815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6f98a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a6815bdf0 0x7f7a6815b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6f98a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6815d810 0x7f7a680b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6f98a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a68160650 con 0x7f7a54003f20 2023-10-16T14:30:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6f98a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a68160090 con 0x7f7a6815d810 2023-10-16T14:30:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6f98a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a68160370 con 0x7f7a6815bdf0 2023-10-16T14:30:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6d726640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a6815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6d726640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a6815c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60808/0 (socket says 172.21.15.29:60808) 2023-10-16T14:30:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.465+0000 7f7a6d726640 1 -- 172.21.15.29:0/2669937377 learned_addr learned my addr 172.21.15.29:0/2669937377 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6d726640 1 -- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a6815bdf0 msgr2=0x7f7a6815b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6d726640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a6815bdf0 0x7f7a6815b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6d726640 1 -- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6815d810 msgr2=0x7f7a680b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6d726640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6815d810 0x7f7a680b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6d726640 1 -- 172.21.15.29:0/2669937377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680c3820 con 0x7f7a54003f20 2023-10-16T14:30:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6d726640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a6815c2b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7a640116d0 tx=0x7f7a64011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6400f9c0 con 0x7f7a54003f20 2023-10-16T14:30:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6400fb20 con 0x7f7a54003f20 2023-10-16T14:30:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6400fed0 con 0x7f7a54003f20 2023-10-16T14:30:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.466+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a680c3b00 con 0x7f7a54003f20 2023-10-16T14:30:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.467+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6815ba90 con 0x7f7a54003f20 2023-10-16T14:30:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.468+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6804e160 con 0x7f7a54003f20 2023-10-16T14:30:17.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.469+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a64015460 con 0x7f7a54003f20 2023-10-16T14:30:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.470+0000 7f7a5dffb640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7a4c06afc0 0x7f7a4c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.470+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a64089fe0 con 0x7f7a54003f20 2023-10-16T14:30:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.470+0000 7f7a6cf25640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7a4c06afc0 0x7f7a4c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.472+0000 7f7a6cf25640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7a4c06afc0 0x7f7a4c06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a58033000 tx=0x7f7a5802f000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.474+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a64054fb0 con 0x7f7a54003f20 2023-10-16T14:30:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:17.707+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7a680caa00 con 0x7f7a54003f20 2023-10-16T14:30:18.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.290+0000 7f7a5dffb640 1 -- 172.21.15.29:0/2669937377 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7a640220b0 con 0x7f7a54003f20 2023-10-16T14:30:18.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7a4c06afc0 msgr2=0x7f7a4c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7a4c06afc0 0x7f7a4c06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a58033000 tx=0x7f7a5802f000 comp rx=0 tx=0).stop 2023-10-16T14:30:18.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 msgr2=0x7f7a6815c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a6815c2b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7a640116d0 tx=0x7f7a64011b90 comp rx=0 tx=0).stop 2023-10-16T14:30:18.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 shutdown_connections 2023-10-16T14:30:18.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f7a4c06afc0 0x7f7a4c06d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a6815d810 0x7f7a680b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a6815bdf0 0x7f7a6815b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 --2- 172.21.15.29:0/2669937377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a54003f20 0x7f7a6815c2b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 >> 172.21.15.29:0/2669937377 conn(0x7f7a6806bc10 msgr2=0x7f7a6806a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:18.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 shutdown_connections 2023-10-16T14:30:18.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.295+0000 7f7a6f98a640 1 -- 172.21.15.29:0/2669937377 wait complete. 2023-10-16T14:30:18.320 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:30:18.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:30:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.551+0000 7fcfa42e0640 1 Processor -- start 2023-10-16T14:30:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.551+0000 7fcfa42e0640 1 -- start start 2023-10-16T14:30:18.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.553+0000 7fcfa42e0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf9c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.553+0000 7fcfa42e0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0cb030 con 0x7fcf9c06a4b0 2023-10-16T14:30:18.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.553+0000 7fcfa42e0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c0c91c0 con 0x7fcf9c15ad70 2023-10-16T14:30:18.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.553+0000 7fcfa42e0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c94a0 con 0x7fcf9c0caac0 2023-10-16T14:30:18.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.553+0000 7fcfa187b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcf9c06a4b0 0x7fcf9c0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39472/0 (socket says 172.21.15.29:39472) 2023-10-16T14:30:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa187b640 1 -- 172.21.15.29:0/883476118 learned_addr learned my addr 172.21.15.29:0/883476118 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa287d640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf9c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081401352 0 0) 0x7fcf9c0cb030 con 0x7fcf9c06a4b0 2023-10-16T14:30:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf88003520 con 0x7fcf9c06a4b0 2023-10-16T14:30:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200831024 0 0) 0x7fcf88003520 con 0x7fcf9c06a4b0 2023-10-16T14:30:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 msgr2=0x7fcf9c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf9c0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 >> v1:172.21.15.170:6789/0 conn(0x7fcf9c15ad70 legacy=0x7fcf9c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0c9780 con 0x7fcf9c06a4b0 2023-10-16T14:30:18.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.554+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fcf8c001670 con 0x7fcf9c06a4b0 2023-10-16T14:30:18.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 >> v1:172.21.15.29:6789/0 conn(0x7fcf9c06a4b0 legacy=0x7fcf9c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880039a0 0x7fcf88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c94a0 con 0x7fcf880039a0 2023-10-16T14:30:18.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8800ba60 con 0x7fcf88004d00 2023-10-16T14:30:18.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8800bc20 con 0x7fcf9c0caac0 2023-10-16T14:30:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa207c640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880039a0 0x7fcf88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa187b640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.555+0000 7fcfa287d640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa207c640 1 -- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 msgr2=0x7fcf880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa207c640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa207c640 1 -- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf88004d00 msgr2=0x7fcf88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa207c640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa207c640 1 -- 172.21.15.29:0/883476118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88003520 con 0x7fcf880039a0 2023-10-16T14:30:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa287d640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.556+0000 7fcfa187b640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf9c0caac0 0x7fcf880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.557+0000 7fcfa207c640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880039a0 0x7fcf88003dc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcf98001720 tx=0x7fcf880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe436b9fff96221 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.557+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980052d0 con 0x7fcf880039a0 2023-10-16T14:30:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.558+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9802e050 con 0x7fcf880039a0 2023-10-16T14:30:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.558+0000 7fcfa107a640 1 -- 172.21.15.29:0/883476118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980035f0 con 0x7fcf880039a0 2023-10-16T14:30:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.558+0000 7fcfa42e0640 1 -- 172.21.15.29:0/883476118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880039a0 msgr2=0x7fcf88003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.558+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/883476118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf880039a0 0x7fcf88003dc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcf98001720 tx=0x7fcf880057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.558+0000 7fcfa287d640 1 -- 172.21.15.29:0/883476118 reap_dead start 2023-10-16T14:30:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.559+0000 7fcfa42e0640 1 -- 172.21.15.29:0/883476118 shutdown_connections 2023-10-16T14:30:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.559+0000 7fcfa42e0640 1 -- 172.21.15.29:0/883476118 >> 172.21.15.29:0/883476118 conn(0x7fcf9c0ce9b0 msgr2=0x7fcf9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.559+0000 7fcfa42e0640 1 -- 172.21.15.29:0/883476118 shutdown_connections 2023-10-16T14:30:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.559+0000 7fcfa42e0640 1 -- 172.21.15.29:0/883476118 wait complete. 2023-10-16T14:30:18.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.561+0000 7fcfa42e0640 1 Processor -- start 2023-10-16T14:30:18.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.561+0000 7fcfa42e0640 1 -- start start 2023-10-16T14:30:18.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa42e0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880039a0 0x7fcf9c0c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa42e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 0x7fcf9c0c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa42e0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf9c0c39e0 0x7fcf9c0a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa42e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c4ba0 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa42e0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c3f20 con 0x7fcf9c0c39e0 2023-10-16T14:30:18.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa42e0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c4200 con 0x7fcf880039a0 2023-10-16T14:30:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa187b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 0x7fcf9c0c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa187b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 0x7fcf9c0c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60818/0 (socket says 172.21.15.29:60818) 2023-10-16T14:30:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa187b640 1 -- 172.21.15.29:0/813499578 learned_addr learned my addr 172.21.15.29:0/813499578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.562+0000 7fcfa207c640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880039a0 0x7fcf9c0c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa287d640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf9c0c39e0 0x7fcf9c0a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa187b640 1 -- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880039a0 msgr2=0x7fcf9c0c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa187b640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880039a0 0x7fcf9c0c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa187b640 1 -- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf9c0c39e0 msgr2=0x7fcf9c0a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa187b640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf9c0c39e0 0x7fcf9c0a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa187b640 1 -- 172.21.15.29:0/813499578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0c44e0 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa287d640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf9c0c39e0 0x7fcf9c0a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa187b640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 0x7fcf9c0c34a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e730 tx=0x7fcf8c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00ee40 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c00c8d0 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00cc40 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.563+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c0c47c0 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.564+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c0b0e90 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.565+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c0c4ce0 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.566+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf8c01d460 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.566+0000 7fcf92ffd640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcf8006af90 0x7fcf8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:18.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.567+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf8c089760 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.567+0000 7fcfa207c640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcf8006af90 0x7fcf8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:18.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.568+0000 7fcfa207c640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcf8006af90 0x7fcf8006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf98033000 tx=0x7fcf98009030 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:18.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.571+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf8c0546f0 con 0x7fcf9c0b0a10 2023-10-16T14:30:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:18.804+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcf9c04e160 con 0x7fcf9c0b0a10 2023-10-16T14:30:19.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.299+0000 7fcf92ffd640 1 -- 172.21.15.29:0/813499578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcf8c057d10 con 0x7fcf9c0b0a10 2023-10-16T14:30:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcf8006af90 msgr2=0x7fcf8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcf8006af90 0x7fcf8006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf98033000 tx=0x7fcf98009030 comp rx=0 tx=0).stop 2023-10-16T14:30:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 msgr2=0x7fcf9c0c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 0x7fcf9c0c34a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e730 tx=0x7fcf8c00ebf0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 shutdown_connections 2023-10-16T14:30:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fcf8006af90 0x7fcf8006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9c0b0a10 0x7fcf9c0c34a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf9c0c39e0 0x7fcf9c0a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 --2- 172.21.15.29:0/813499578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcf880039a0 0x7fcf9c0c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 >> 172.21.15.29:0/813499578 conn(0x7fcf9c0ce9b0 msgr2=0x7fcf9c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:19.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 shutdown_connections 2023-10-16T14:30:19.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.303+0000 7fcfa42e0640 1 -- 172.21.15.29:0/813499578 wait complete. 2023-10-16T14:30:19.330 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:30:19.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:30:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.551+0000 7ffa1cd43640 1 Processor -- start 2023-10-16T14:30:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.551+0000 7ffa1cd43640 1 -- start start 2023-10-16T14:30:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa1cd43640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffa18153740 0x7ffa1815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa1cd43640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa180adbe0 0x7ffa180ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa1cd43640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa180cd4a0 con 0x7ffa180ccb00 2023-10-16T14:30:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa1cd43640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1815ac60 con 0x7ffa180adbe0 2023-10-16T14:30:19.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa1cd43640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa18070030 con 0x7ffa18153740 2023-10-16T14:30:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa17fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa180adbe0 0x7ffa180ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa17fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa180adbe0 0x7ffa180ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60820/0 (socket says 172.21.15.29:60820) 2023-10-16T14:30:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.553+0000 7ffa17fff640 1 -- 172.21.15.29:0/1819320990 learned_addr learned my addr 172.21.15.29:0/1819320990 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.554+0000 7ffa177fe640 1 --2- 172.21.15.29:0/1819320990 >> v2:172.21.15.170:3300/0 conn(0x7ffa18153740 0x7ffa1815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.554+0000 7ffa17fff640 1 -- 172.21.15.29:0/1819320990 >> v2:172.21.15.170:3300/0 conn(0x7ffa18153740 msgr2=0x7ffa1815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.554+0000 7ffa17fff640 1 --2- 172.21.15.29:0/1819320990 >> v2:172.21.15.170:3300/0 conn(0x7ffa18153740 0x7ffa1815a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.554+0000 7ffa17fff640 1 -- 172.21.15.29:0/1819320990 >> v1:172.21.15.170:6789/0 conn(0x7ffa180ccb00 legacy=0x7ffa180ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.554+0000 7ffa17fff640 1 -- 172.21.15.29:0/1819320990 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18070310 con 0x7ffa180adbe0 2023-10-16T14:30:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.555+0000 7ffa17fff640 1 --2- 172.21.15.29:0/1819320990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa180adbe0 0x7ffa180ccf60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffa18073ca0 tx=0x7ffa1002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edddd013c39be643 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.555+0000 7ffa16ffd640 1 -- 172.21.15.29:0/1819320990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1002c9f0 con 0x7ffa180adbe0 2023-10-16T14:30:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa16ffd640 1 -- 172.21.15.29:0/1819320990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1002cb50 con 0x7ffa180adbe0 2023-10-16T14:30:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa16ffd640 1 -- 172.21.15.29:0/1819320990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa100343f0 con 0x7ffa180adbe0 2023-10-16T14:30:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 -- 172.21.15.29:0/1819320990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa180adbe0 msgr2=0x7ffa180ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/1819320990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa180adbe0 0x7ffa180ccf60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffa18073ca0 tx=0x7ffa1002be50 comp rx=0 tx=0).stop 2023-10-16T14:30:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 -- 172.21.15.29:0/1819320990 shutdown_connections 2023-10-16T14:30:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/1819320990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa180adbe0 0x7ffa180ccf60 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/1819320990 >> v2:172.21.15.170:3300/0 conn(0x7ffa18153740 0x7ffa1815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 -- 172.21.15.29:0/1819320990 >> 172.21.15.29:0/1819320990 conn(0x7ffa180ce6a0 msgr2=0x7ffa180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 -- 172.21.15.29:0/1819320990 shutdown_connections 2023-10-16T14:30:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.556+0000 7ffa1cd43640 1 -- 172.21.15.29:0/1819320990 wait complete. 2023-10-16T14:30:19.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.558+0000 7ffa1cd43640 1 Processor -- start 2023-10-16T14:30:19.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.558+0000 7ffa1cd43640 1 -- start start 2023-10-16T14:30:19.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa1cd43640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa180adbe0 0x7ffa18152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:19.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa1cd43640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa180ccb00 0x7ffa180b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:19.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa1cd43640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 0x7ffa180b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:19.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa1cd43640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa180ad3d0 con 0x7ffa18153740 2023-10-16T14:30:19.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa1cd43640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa180aca30 con 0x7ffa180adbe0 2023-10-16T14:30:19.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa1cd43640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa180acd10 con 0x7ffa180ccb00 2023-10-16T14:30:19.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 0x7ffa180b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:19.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 0x7ffa180b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60830/0 (socket says 172.21.15.29:60830) 2023-10-16T14:30:19.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.559+0000 7ffa17fff640 1 -- 172.21.15.29:0/626001995 learned_addr learned my addr 172.21.15.29:0/626001995 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:19.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa17fff640 1 -- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa180ccb00 msgr2=0x7ffa180b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa177fe640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa180adbe0 0x7ffa18152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:19.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa0effd640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa180ccb00 0x7ffa180b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:19.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa17fff640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa180ccb00 0x7ffa180b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa17fff640 1 -- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa180adbe0 msgr2=0x7ffa18152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa17fff640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa180adbe0 0x7ffa18152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa17fff640 1 -- 172.21.15.29:0/626001995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10009b30 con 0x7ffa18153740 2023-10-16T14:30:19.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa177fe640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa180adbe0 0x7ffa18152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:19.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.560+0000 7ffa17fff640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 0x7ffa180b39d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009f40 tx=0x7ffa10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:19.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.561+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1003e450 con 0x7ffa18153740 2023-10-16T14:30:19.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.561+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1003e5d0 con 0x7ffa18153740 2023-10-16T14:30:19.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.561+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1003e980 con 0x7ffa18153740 2023-10-16T14:30:19.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.561+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa180acff0 con 0x7ffa18153740 2023-10-16T14:30:19.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.561+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa180c2490 con 0x7ffa18153740 2023-10-16T14:30:19.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.562+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1806fc30 con 0x7ffa18153740 2023-10-16T14:30:19.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.563+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa1003eae0 con 0x7ffa18153740 2023-10-16T14:30:19.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.564+0000 7ffa14ff9640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff9f806afc0 0x7ff9f806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:19.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.564+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa100ac4d0 con 0x7ffa18153740 2023-10-16T14:30:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.565+0000 7ffa177fe640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff9f806afc0 0x7ff9f806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.566+0000 7ffa177fe640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff9f806afc0 0x7ff9f806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffa1806f4f0 tx=0x7ffa08008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.568+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa10077460 con 0x7ffa18153740 2023-10-16T14:30:19.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.806+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa1804e9b0 con 0x7ffa18153740 2023-10-16T14:30:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.808+0000 7ffa14ff9640 1 -- 172.21.15.29:0/626001995 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7ffa1007aa80 con 0x7ffa18153740 2023-10-16T14:30:19.812 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:19.812 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:30:15.728624+0000","modified":"2023-10-16T14:30:18.806702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1869070988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1869070988},{"type":"v1","addr":"172.21.15.29:6835","nonce":1869070988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T14:30:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-16T14:30:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff9f806afc0 msgr2=0x7ff9f806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff9f806afc0 0x7ff9f806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffa1806f4f0 tx=0x7ffa08008040 comp rx=0 tx=0).stop 2023-10-16T14:30:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 msgr2=0x7ffa180b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 0x7ffa180b39d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009f40 tx=0x7ffa10004a40 comp rx=0 tx=0).stop 2023-10-16T14:30:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 shutdown_connections 2023-10-16T14:30:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7ff9f806afc0 0x7ff9f806d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa18153740 0x7ffa180b39d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffa180ccb00 0x7ffa180b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 --2- 172.21.15.29:0/626001995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa180adbe0 0x7ffa18152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 >> 172.21.15.29:0/626001995 conn(0x7ffa180ce6a0 msgr2=0x7ffa18155b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 shutdown_connections 2023-10-16T14:30:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:19.811+0000 7ffa1cd43640 1 -- 172.21.15.29:0/626001995 wait complete. 2023-10-16T14:30:19.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:30:20.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.042+0000 7f2533239640 1 Processor -- start 2023-10-16T14:30:20.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.042+0000 7f2533239640 1 -- start start 2023-10-16T14:30:20.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.043+0000 7f2533239640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c06c040 0x7f252c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.043+0000 7f2533239640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f252c0ad970 0x7f252c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.043+0000 7f2533239640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c06c960 con 0x7f252c15ebc0 2023-10-16T14:30:20.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.043+0000 7f2533239640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c15b9d0 con 0x7f252c0ad970 2023-10-16T14:30:20.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.043+0000 7f2533239640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c15bcb0 con 0x7f252c06c040 2023-10-16T14:30:20.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f252c0ad970 0x7f252c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f252c0ad970 0x7f252c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60844/0 (socket says 172.21.15.29:60844) 2023-10-16T14:30:20.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 -- 172.21.15.29:0/1187696451 learned_addr learned my addr 172.21.15.29:0/1187696451 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:20.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2532237640 1 --2- 172.21.15.29:0/1187696451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c06c040 0x7f252c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 -- 172.21.15.29:0/1187696451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c06c040 msgr2=0x7f252c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:20.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 --2- 172.21.15.29:0/1187696451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c06c040 0x7f252c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 -- 172.21.15.29:0/1187696451 >> v1:172.21.15.170:6789/0 conn(0x7f252c15ebc0 legacy=0x7f252c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:20.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.044+0000 7f2531a36640 1 -- 172.21.15.29:0/1187696451 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c15bf90 con 0x7f252c0ad970 2023-10-16T14:30:20.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.045+0000 7f2531a36640 1 --2- 172.21.15.29:0/1187696451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c0ad970 0x7f252c15e680 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f251c0098a0 tx=0x7f251c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c547774c9921e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:20.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2531235640 1 -- 172.21.15.29:0/1187696451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c031640 con 0x7f252c0ad970 2023-10-16T14:30:20.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2531235640 1 -- 172.21.15.29:0/1187696451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c0317a0 con 0x7f252c0ad970 2023-10-16T14:30:20.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2531235640 1 -- 172.21.15.29:0/1187696451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c031b30 con 0x7f252c0ad970 2023-10-16T14:30:20.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 -- 172.21.15.29:0/1187696451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c0ad970 msgr2=0x7f252c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 --2- 172.21.15.29:0/1187696451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c0ad970 0x7f252c15e680 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f251c0098a0 tx=0x7f251c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 -- 172.21.15.29:0/1187696451 shutdown_connections 2023-10-16T14:30:20.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 --2- 172.21.15.29:0/1187696451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c0ad970 0x7f252c15e680 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 --2- 172.21.15.29:0/1187696451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c06c040 0x7f252c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 -- 172.21.15.29:0/1187696451 >> 172.21.15.29:0/1187696451 conn(0x7f252c0ce6a0 msgr2=0x7f252c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:20.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 -- 172.21.15.29:0/1187696451 shutdown_connections 2023-10-16T14:30:20.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.046+0000 7f2533239640 1 -- 172.21.15.29:0/1187696451 wait complete. 2023-10-16T14:30:20.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.048+0000 7f2533239640 1 Processor -- start 2023-10-16T14:30:20.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.048+0000 7f2533239640 1 -- start start 2023-10-16T14:30:20.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2533239640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f252c06c040 0x7f252c0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2533239640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c0ad970 0x7f252c1efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2533239640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 0x7f252c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2533239640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0b2ff0 con 0x7f252c15ebc0 2023-10-16T14:30:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2533239640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0b3a60 con 0x7f252c06c040 2023-10-16T14:30:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2533239640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0b23e0 con 0x7f252c0ad970 2023-10-16T14:30:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2532a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 0x7f252c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2532a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 0x7f252c1f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60846/0 (socket says 172.21.15.29:60846) 2023-10-16T14:30:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.049+0000 7f2532a38640 1 -- 172.21.15.29:0/2038558948 learned_addr learned my addr 172.21.15.29:0/2038558948 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2532a38640 1 -- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c0ad970 msgr2=0x7f252c1efab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2532a38640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c0ad970 0x7f252c1efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2532a38640 1 -- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f252c06c040 msgr2=0x7f252c0b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2532a38640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f252c06c040 0x7f252c0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2532a38640 1 -- 172.21.15.29:0/2038558948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c0094d0 con 0x7f252c15ebc0 2023-10-16T14:30:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2532a38640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 0x7f252c1f3860 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f252400c410 tx=0x7f252400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524011790 con 0x7f252c15ebc0 2023-10-16T14:30:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25240118f0 con 0x7f252c15ebc0 2023-10-16T14:30:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524011ce0 con 0x7f252c15ebc0 2023-10-16T14:30:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c0b26c0 con 0x7f252c15ebc0 2023-10-16T14:30:20.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.050+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c0b2c80 con 0x7f252c15ebc0 2023-10-16T14:30:20.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.051+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c06b330 con 0x7f252c15ebc0 2023-10-16T14:30:20.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.052+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f252401c460 con 0x7f252c15ebc0 2023-10-16T14:30:20.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.053+0000 7f2522ffd640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f251006afc0 0x7f251006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.053+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25240894b0 con 0x7f252c15ebc0 2023-10-16T14:30:20.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.054+0000 7f2532237640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f251006afc0 0x7f251006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.055+0000 7f2532237640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f251006afc0 0x7f251006d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2528007dc0 tx=0x7f25280072f0 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.058+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25240543b0 con 0x7f252c15ebc0 2023-10-16T14:30:20.204 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T14:30:20.200+0000 7f128d15a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:30:20.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.253+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f252c04e9b0 con 0x7f252c15ebc0 2023-10-16T14:30:20.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.255+0000 7f2522ffd640 1 -- 172.21.15.29:0/2038558948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16513 (secure 0 0 0) 0x7f25240579d0 con 0x7f252c15ebc0 2023-10-16T14:30:20.259 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:20.259 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":65,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:30:17.263022+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:30:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f251006afc0 msgr2=0x7f251006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f251006afc0 0x7f251006d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2528007dc0 tx=0x7f25280072f0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 msgr2=0x7f252c1f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 0x7f252c1f3860 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f252400c410 tx=0x7f252400c8d0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 shutdown_connections 2023-10-16T14:30:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f251006afc0 0x7f251006d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f252c15ebc0 0x7f252c1f3860 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f252c0ad970 0x7f252c1efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.257+0000 7f2533239640 1 --2- 172.21.15.29:0/2038558948 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f252c06c040 0x7f252c0b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.258+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 >> 172.21.15.29:0/2038558948 conn(0x7f252c0ce6a0 msgr2=0x7f252c15f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:20.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.258+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 shutdown_connections 2023-10-16T14:30:20.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.258+0000 7f2533239640 1 -- 172.21.15.29:0/2038558948 wait complete. 2023-10-16T14:30:20.279 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:30:20.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:30:20.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.479+0000 7f9209f8c640 1 Processor -- start 2023-10-16T14:30:20.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.479+0000 7f9209f8c640 1 -- start start 2023-10-16T14:30:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.480+0000 7f9209f8c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f920406c040 0x7f920406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.480+0000 7f9209f8c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92040ad970 0x7f920415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.480+0000 7f9209f8c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920406c960 con 0x7f920415ebc0 2023-10-16T14:30:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.480+0000 7f9209f8c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f920415b9d0 con 0x7f92040ad970 2023-10-16T14:30:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.480+0000 7f9209f8c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f920415bcb0 con 0x7f920406c040 2023-10-16T14:30:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f9203fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92040ad970 0x7f920415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f920978b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f920415ebc0 0x7f92040cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39482/0 (socket says 172.21.15.29:39482) 2023-10-16T14:30:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f920978b640 1 -- 172.21.15.29:0/548577304 learned_addr learned my addr 172.21.15.29:0/548577304 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f9208f8a640 1 --2- 172.21.15.29:0/548577304 >> v2:172.21.15.170:3300/0 conn(0x7f920406c040 0x7f920406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f9203fff640 1 -- 172.21.15.29:0/548577304 >> v2:172.21.15.170:3300/0 conn(0x7f920406c040 msgr2=0x7f920406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f9203fff640 1 --2- 172.21.15.29:0/548577304 >> v2:172.21.15.170:3300/0 conn(0x7f920406c040 0x7f920406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f9203fff640 1 -- 172.21.15.29:0/548577304 >> v1:172.21.15.29:6789/0 conn(0x7f920415ebc0 legacy=0x7f92040cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f9203fff640 1 -- 172.21.15.29:0/548577304 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920415bf90 con 0x7f92040ad970 2023-10-16T14:30:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.481+0000 7f92037fe640 1 -- 172.21.15.29:0/548577304 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466650594 0 0) 0x7f920406c960 con 0x7f920415ebc0 2023-10-16T14:30:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.483+0000 7f9203fff640 1 --2- 172.21.15.29:0/548577304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92040ad970 0x7f920415e680 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f92040cf350 tx=0x7f91f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67eeab9d2d02237 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.483+0000 7f92037fe640 1 -- 172.21.15.29:0/548577304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0031830 con 0x7f92040ad970 2023-10-16T14:30:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.483+0000 7f92037fe640 1 -- 172.21.15.29:0/548577304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f0031990 con 0x7f92040ad970 2023-10-16T14:30:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.483+0000 7f92037fe640 1 -- 172.21.15.29:0/548577304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0031d20 con 0x7f92040ad970 2023-10-16T14:30:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.483+0000 7f9209f8c640 1 -- 172.21.15.29:0/548577304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92040ad970 msgr2=0x7f920415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.483+0000 7f9209f8c640 1 --2- 172.21.15.29:0/548577304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92040ad970 0x7f920415e680 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f92040cf350 tx=0x7f91f002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.484+0000 7f9209f8c640 1 -- 172.21.15.29:0/548577304 shutdown_connections 2023-10-16T14:30:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.484+0000 7f9209f8c640 1 --2- 172.21.15.29:0/548577304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92040ad970 0x7f920415e680 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.484+0000 7f9209f8c640 1 --2- 172.21.15.29:0/548577304 >> v2:172.21.15.170:3300/0 conn(0x7f920406c040 0x7f920406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.484+0000 7f9209f8c640 1 -- 172.21.15.29:0/548577304 >> 172.21.15.29:0/548577304 conn(0x7f92040ce6a0 msgr2=0x7f92040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.484+0000 7f9209f8c640 1 -- 172.21.15.29:0/548577304 shutdown_connections 2023-10-16T14:30:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.484+0000 7f9209f8c640 1 -- 172.21.15.29:0/548577304 wait complete. 2023-10-16T14:30:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.485+0000 7f9209f8c640 1 Processor -- start 2023-10-16T14:30:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.486+0000 7f9209f8c640 1 -- start start 2023-10-16T14:30:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f9209f8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f920406c040 0x7f92041511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f9209f8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040ad970 0x7f92040c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f9209f8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 0x7f92040c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f9209f8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040b4510 con 0x7f920415ebc0 2023-10-16T14:30:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f9209f8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040aef40 con 0x7f92040ad970 2023-10-16T14:30:20.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f9209f8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92040af220 con 0x7f920406c040 2023-10-16T14:30:20.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 0x7f92040c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 0x7f92040c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60864/0 (socket says 172.21.15.29:60864) 2023-10-16T14:30:20.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 -- 172.21.15.29:0/926474603 learned_addr learned my addr 172.21.15.29:0/926474603 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:20.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 -- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f920406c040 msgr2=0x7f92041511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f920406c040 0x7f92041511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 -- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040ad970 msgr2=0x7f92040c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040ad970 0x7f92040c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.487+0000 7f920978b640 1 -- 172.21.15.29:0/926474603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0009400 con 0x7f920415ebc0 2023-10-16T14:30:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.488+0000 7f920978b640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 0x7f92040c77b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00c7e0 tx=0x7f91fc00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.488+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00a640 con 0x7f920415ebc0 2023-10-16T14:30:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.488+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc00a7c0 con 0x7f920415ebc0 2023-10-16T14:30:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.488+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92040af500 con 0x7f920415ebc0 2023-10-16T14:30:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.488+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00ab70 con 0x7f920415ebc0 2023-10-16T14:30:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.488+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92041516f0 con 0x7f920415ebc0 2023-10-16T14:30:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.489+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920406b330 con 0x7f920415ebc0 2023-10-16T14:30:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.490+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f91fc00acd0 con 0x7f920415ebc0 2023-10-16T14:30:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.491+0000 7f92017fa640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f91e806af90 0x7f91e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.491+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91fc089040 con 0x7f920415ebc0 2023-10-16T14:30:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.492+0000 7f9208f8a640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f91e806af90 0x7f91e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.493+0000 7f9208f8a640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f91e806af90 0x7f91e806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91f800a810 tx=0x7f91f8008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:20.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.496+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f91fc0bbca0 con 0x7f920415ebc0 2023-10-16T14:30:20.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.684+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 --> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92040cc270 con 0x7f91e806af90 2023-10-16T14:30:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.724+0000 7f92017fa640 1 -- 172.21.15.29:0/926474603 <== mgr.5113 v2:172.21.15.170:6832/3423886595 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286305 (secure 0 0 0) 0x7f91f800a400 con 0x7f91e806af90 2023-10-16T14:30:20.729 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:20.733 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:30:20.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f91e806af90 msgr2=0x7f91e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f91e806af90 0x7f91e806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91f800a810 tx=0x7f91f8008040 comp rx=0 tx=0).stop 2023-10-16T14:30:20.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 msgr2=0x7f92040c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:20.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 0x7f92040c77b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00c7e0 tx=0x7f91fc00cca0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 shutdown_connections 2023-10-16T14:30:20.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f91e806af90 0x7f91e806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f920415ebc0 0x7f92040c77b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92040ad970 0x7f92040c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 --2- 172.21.15.29:0/926474603 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f920406c040 0x7f92041511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:20.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 >> 172.21.15.29:0/926474603 conn(0x7f92040ce6a0 msgr2=0x7f92040c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:20.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 shutdown_connections 2023-10-16T14:30:20.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:20.730+0000 7f9209f8c640 1 -- 172.21.15.29:0/926474603 wait complete. 2023-10-16T14:30:20.754 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T14:30:20.321155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326032,"kb_used_data":7176,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754648688,"statfs":{"total":773094113280,"available":772760256512,"internally_reserved":0,"allocated":7348224,"data_stored":4077396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":326356537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002191"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284146+0000","last_change":"2023-10-16T14:30:16.271259+0000","last_active":"2023-10-16T14:30:17.284146+0000","last_peered":"2023-10-16T14:30:17.284146+0000","last_clean":"2023-10-16T14:30:17.284146+0000","last_became_active":"2023-10-16T14:30:16.270883+0000","last_became_peered":"2023-10-16T14:30:16.270883+0000","last_unstale":"2023-10-16T14:30:17.284146+0000","last_undegraded":"2023-10-16T14:30:17.284146+0000","last_fullsized":"2023-10-16T14:30:17.284146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:49.392964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.294733+0000","last_change":"2023-10-16T14:30:16.293633+0000","last_active":"2023-10-16T14:30:16.294733+0000","last_peered":"2023-10-16T14:30:16.294733+0000","last_clean":"2023-10-16T14:30:16.294733+0000","last_became_active":"2023-10-16T14:30:16.292147+0000","last_became_peered":"2023-10-16T14:30:16.292147+0000","last_unstale":"2023-10-16T14:30:16.294733+0000","last_undegraded":"2023-10-16T14:30:16.294733+0000","last_fullsized":"2023-10-16T14:30:16.294733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:23.799327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283388+0000","last_change":"2023-10-16T14:30:16.274092+0000","last_active":"2023-10-16T14:30:17.283388+0000","last_peered":"2023-10-16T14:30:17.283388+0000","last_clean":"2023-10-16T14:30:17.283388+0000","last_became_active":"2023-10-16T14:30:16.273785+0000","last_became_peered":"2023-10-16T14:30:16.273785+0000","last_unstale":"2023-10-16T14:30:17.283388+0000","last_undegraded":"2023-10-16T14:30:17.283388+0000","last_fullsized":"2023-10-16T14:30:17.283388+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:48.494974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278390+0000","last_change":"2023-10-16T14:30:16.291888+0000","last_active":"2023-10-16T14:30:17.278390+0000","last_peered":"2023-10-16T14:30:17.278390+0000","last_clean":"2023-10-16T14:30:17.278390+0000","last_became_active":"2023-10-16T14:30:16.291596+0000","last_became_peered":"2023-10-16T14:30:16.291596+0000","last_unstale":"2023-10-16T14:30:17.278390+0000","last_undegraded":"2023-10-16T14:30:17.278390+0000","last_fullsized":"2023-10-16T14:30:17.278390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:25.908520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.291448+0000","last_change":"2023-10-16T14:30:16.283296+0000","last_active":"2023-10-16T14:30:16.291448+0000","last_peered":"2023-10-16T14:30:16.291448+0000","last_clean":"2023-10-16T14:30:16.291448+0000","last_became_active":"2023-10-16T14:30:16.282996+0000","last_became_peered":"2023-10-16T14:30:16.282996+0000","last_unstale":"2023-10-16T14:30:16.291448+0000","last_undegraded":"2023-10-16T14:30:16.291448+0000","last_fullsized":"2023-10-16T14:30:16.291448+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:19.654589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.281086+0000","last_change":"2023-10-16T14:30:16.274153+0000","last_active":"2023-10-16T14:30:16.281086+0000","last_peered":"2023-10-16T14:30:16.281086+0000","last_clean":"2023-10-16T14:30:16.281086+0000","last_became_active":"2023-10-16T14:30:16.270144+0000","last_became_peered":"2023-10-16T14:30:16.270144+0000","last_unstale":"2023-10-16T14:30:16.281086+0000","last_undegraded":"2023-10-16T14:30:16.281086+0000","last_fullsized":"2023-10-16T14:30:16.281086+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:55.567177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284159+0000","last_change":"2023-10-16T14:30:16.272721+0000","last_active":"2023-10-16T14:30:17.284159+0000","last_peered":"2023-10-16T14:30:17.284159+0000","last_clean":"2023-10-16T14:30:17.284159+0000","last_became_active":"2023-10-16T14:30:16.272369+0000","last_became_peered":"2023-10-16T14:30:16.272369+0000","last_unstale":"2023-10-16T14:30:17.284159+0000","last_undegraded":"2023-10-16T14:30:17.284159+0000","last_fullsized":"2023-10-16T14:30:17.284159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:24.868942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277721+0000","last_change":"2023-10-16T14:30:16.280070+0000","last_active":"2023-10-16T14:30:17.277721+0000","last_peered":"2023-10-16T14:30:17.277721+0000","last_clean":"2023-10-16T14:30:17.277721+0000","last_became_active":"2023-10-16T14:30:16.279563+0000","last_became_peered":"2023-10-16T14:30:16.279563+0000","last_unstale":"2023-10-16T14:30:17.277721+0000","last_undegraded":"2023-10-16T14:30:17.277721+0000","last_fullsized":"2023-10-16T14:30:17.277721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:40:36.309295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278505+0000","last_change":"2023-10-16T14:30:16.264159+0000","last_active":"2023-10-16T14:30:17.278505+0000","last_peered":"2023-10-16T14:30:17.278505+0000","last_clean":"2023-10-16T14:30:17.278505+0000","last_became_active":"2023-10-16T14:30:16.263858+0000","last_became_peered":"2023-10-16T14:30:16.263858+0000","last_unstale":"2023-10-16T14:30:17.278505+0000","last_undegraded":"2023-10-16T14:30:17.278505+0000","last_fullsized":"2023-10-16T14:30:17.278505+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:26.202732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278739+0000","last_change":"2023-10-16T14:30:16.290583+0000","last_active":"2023-10-16T14:30:17.278739+0000","last_peered":"2023-10-16T14:30:17.278739+0000","last_clean":"2023-10-16T14:30:17.278739+0000","last_became_active":"2023-10-16T14:30:16.290289+0000","last_became_peered":"2023-10-16T14:30:16.290289+0000","last_unstale":"2023-10-16T14:30:17.278739+0000","last_undegraded":"2023-10-16T14:30:17.278739+0000","last_fullsized":"2023-10-16T14:30:17.278739+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:13.437481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278198+0000","last_change":"2023-10-16T14:30:16.274076+0000","last_active":"2023-10-16T14:30:17.278198+0000","last_peered":"2023-10-16T14:30:17.278198+0000","last_clean":"2023-10-16T14:30:17.278198+0000","last_became_active":"2023-10-16T14:30:16.273742+0000","last_became_peered":"2023-10-16T14:30:16.273742+0000","last_unstale":"2023-10-16T14:30:17.278198+0000","last_undegraded":"2023-10-16T14:30:17.278198+0000","last_fullsized":"2023-10-16T14:30:17.278198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:32.934311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290192+0000","last_change":"2023-10-16T14:30:16.285712+0000","last_active":"2023-10-16T14:30:16.290192+0000","last_peered":"2023-10-16T14:30:16.290192+0000","last_clean":"2023-10-16T14:30:16.290192+0000","last_became_active":"2023-10-16T14:30:16.285217+0000","last_became_peered":"2023-10-16T14:30:16.285217+0000","last_unstale":"2023-10-16T14:30:16.290192+0000","last_undegraded":"2023-10-16T14:30:16.290192+0000","last_fullsized":"2023-10-16T14:30:16.290192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:57.063012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.271272+0000","last_change":"2023-10-16T14:30:16.269813+0000","last_active":"2023-10-16T14:30:16.271272+0000","last_peered":"2023-10-16T14:30:16.271272+0000","last_clean":"2023-10-16T14:30:16.271272+0000","last_became_active":"2023-10-16T14:30:16.269490+0000","last_became_peered":"2023-10-16T14:30:16.269490+0000","last_unstale":"2023-10-16T14:30:16.271272+0000","last_undegraded":"2023-10-16T14:30:16.271272+0000","last_fullsized":"2023-10-16T14:30:16.271272+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:36.906191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278143+0000","last_change":"2023-10-16T14:30:16.274836+0000","last_active":"2023-10-16T14:30:17.278143+0000","last_peered":"2023-10-16T14:30:17.278143+0000","last_clean":"2023-10-16T14:30:17.278143+0000","last_became_active":"2023-10-16T14:30:16.274528+0000","last_became_peered":"2023-10-16T14:30:16.274528+0000","last_unstale":"2023-10-16T14:30:17.278143+0000","last_undegraded":"2023-10-16T14:30:17.278143+0000","last_fullsized":"2023-10-16T14:30:17.278143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:35.313814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.272391+0000","last_change":"2023-10-16T14:30:16.270552+0000","last_active":"2023-10-16T14:30:16.272391+0000","last_peered":"2023-10-16T14:30:16.272391+0000","last_clean":"2023-10-16T14:30:16.272391+0000","last_became_active":"2023-10-16T14:30:16.270267+0000","last_became_peered":"2023-10-16T14:30:16.270267+0000","last_unstale":"2023-10-16T14:30:16.272391+0000","last_undegraded":"2023-10-16T14:30:16.272391+0000","last_fullsized":"2023-10-16T14:30:16.272391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:21.384315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290120+0000","last_change":"2023-10-16T14:30:16.283627+0000","last_active":"2023-10-16T14:30:16.290120+0000","last_peered":"2023-10-16T14:30:16.290120+0000","last_clean":"2023-10-16T14:30:16.290120+0000","last_became_active":"2023-10-16T14:30:16.283351+0000","last_became_peered":"2023-10-16T14:30:16.283351+0000","last_unstale":"2023-10-16T14:30:16.290120+0000","last_undegraded":"2023-10-16T14:30:16.290120+0000","last_fullsized":"2023-10-16T14:30:16.290120+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:17.848156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284017+0000","last_change":"2023-10-16T14:30:16.286115+0000","last_active":"2023-10-16T14:30:17.284017+0000","last_peered":"2023-10-16T14:30:17.284017+0000","last_clean":"2023-10-16T14:30:17.284017+0000","last_became_active":"2023-10-16T14:30:16.283841+0000","last_became_peered":"2023-10-16T14:30:16.283841+0000","last_unstale":"2023-10-16T14:30:17.284017+0000","last_undegraded":"2023-10-16T14:30:17.284017+0000","last_fullsized":"2023-10-16T14:30:17.284017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:25:10.169897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283854+0000","last_change":"2023-10-16T14:30:16.285932+0000","last_active":"2023-10-16T14:30:17.283854+0000","last_peered":"2023-10-16T14:30:17.283854+0000","last_clean":"2023-10-16T14:30:17.283854+0000","last_became_active":"2023-10-16T14:30:16.285586+0000","last_became_peered":"2023-10-16T14:30:16.285586+0000","last_unstale":"2023-10-16T14:30:17.283854+0000","last_undegraded":"2023-10-16T14:30:17.283854+0000","last_fullsized":"2023-10-16T14:30:17.283854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:39.785736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.295452+0000","last_change":"2023-10-16T14:30:16.293490+0000","last_active":"2023-10-16T14:30:16.295452+0000","last_peered":"2023-10-16T14:30:16.295452+0000","last_clean":"2023-10-16T14:30:16.295452+0000","last_became_active":"2023-10-16T14:30:16.293143+0000","last_became_peered":"2023-10-16T14:30:16.293143+0000","last_unstale":"2023-10-16T14:30:16.295452+0000","last_undegraded":"2023-10-16T14:30:16.295452+0000","last_fullsized":"2023-10-16T14:30:16.295452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:40:36.309295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.291449+0000","last_change":"2023-10-16T14:30:16.284045+0000","last_active":"2023-10-16T14:30:16.291449+0000","last_peered":"2023-10-16T14:30:16.291449+0000","last_clean":"2023-10-16T14:30:16.291449+0000","last_became_active":"2023-10-16T14:30:16.283390+0000","last_became_peered":"2023-10-16T14:30:16.283390+0000","last_unstale":"2023-10-16T14:30:16.291449+0000","last_undegraded":"2023-10-16T14:30:16.291449+0000","last_fullsized":"2023-10-16T14:30:16.291449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.533166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296635+0000","last_change":"2023-10-16T14:30:16.282760+0000","last_active":"2023-10-16T14:30:16.296635+0000","last_peered":"2023-10-16T14:30:16.296635+0000","last_clean":"2023-10-16T14:30:16.296635+0000","last_became_active":"2023-10-16T14:30:16.282436+0000","last_became_peered":"2023-10-16T14:30:16.282436+0000","last_unstale":"2023-10-16T14:30:16.296635+0000","last_undegraded":"2023-10-16T14:30:16.296635+0000","last_fullsized":"2023-10-16T14:30:16.296635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:23.447810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278236+0000","last_change":"2023-10-16T14:30:16.283998+0000","last_active":"2023-10-16T14:30:17.278236+0000","last_peered":"2023-10-16T14:30:17.278236+0000","last_clean":"2023-10-16T14:30:17.278236+0000","last_became_active":"2023-10-16T14:30:16.283710+0000","last_became_peered":"2023-10-16T14:30:16.283710+0000","last_unstale":"2023-10-16T14:30:17.278236+0000","last_undegraded":"2023-10-16T14:30:17.278236+0000","last_fullsized":"2023-10-16T14:30:17.278236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:30.369966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.292824+0000","last_change":"2023-10-16T14:30:16.288810+0000","last_active":"2023-10-16T14:30:16.292824+0000","last_peered":"2023-10-16T14:30:16.292824+0000","last_clean":"2023-10-16T14:30:16.292824+0000","last_became_active":"2023-10-16T14:30:16.287063+0000","last_became_peered":"2023-10-16T14:30:16.287063+0000","last_unstale":"2023-10-16T14:30:16.292824+0000","last_undegraded":"2023-10-16T14:30:16.292824+0000","last_fullsized":"2023-10-16T14:30:16.292824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:23.531990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290140+0000","last_change":"2023-10-16T14:30:16.287720+0000","last_active":"2023-10-16T14:30:16.290140+0000","last_peered":"2023-10-16T14:30:16.290140+0000","last_clean":"2023-10-16T14:30:16.290140+0000","last_became_active":"2023-10-16T14:30:16.287090+0000","last_became_peered":"2023-10-16T14:30:16.287090+0000","last_unstale":"2023-10-16T14:30:16.290140+0000","last_undegraded":"2023-10-16T14:30:16.290140+0000","last_fullsized":"2023-10-16T14:30:16.290140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:25.258309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293491+0000","last_change":"2023-10-16T14:30:16.290669+0000","last_active":"2023-10-16T14:30:16.293491+0000","last_peered":"2023-10-16T14:30:16.293491+0000","last_clean":"2023-10-16T14:30:16.293491+0000","last_became_active":"2023-10-16T14:30:16.289468+0000","last_became_peered":"2023-10-16T14:30:16.289468+0000","last_unstale":"2023-10-16T14:30:16.293491+0000","last_undegraded":"2023-10-16T14:30:16.293491+0000","last_fullsized":"2023-10-16T14:30:16.293491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:13.661918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.272738+0000","last_change":"2023-10-16T14:30:16.270850+0000","last_active":"2023-10-16T14:30:16.272738+0000","last_peered":"2023-10-16T14:30:16.272738+0000","last_clean":"2023-10-16T14:30:16.272738+0000","last_became_active":"2023-10-16T14:30:16.270520+0000","last_became_peered":"2023-10-16T14:30:16.270520+0000","last_unstale":"2023-10-16T14:30:16.272738+0000","last_undegraded":"2023-10-16T14:30:16.272738+0000","last_fullsized":"2023-10-16T14:30:16.272738+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:59.451693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283479+0000","last_change":"2023-10-16T14:30:16.282285+0000","last_active":"2023-10-16T14:30:17.283479+0000","last_peered":"2023-10-16T14:30:17.283479+0000","last_clean":"2023-10-16T14:30:17.283479+0000","last_became_active":"2023-10-16T14:30:16.282002+0000","last_became_peered":"2023-10-16T14:30:16.282002+0000","last_unstale":"2023-10-16T14:30:17.283479+0000","last_undegraded":"2023-10-16T14:30:17.283479+0000","last_fullsized":"2023-10-16T14:30:17.283479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:13.167772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290174+0000","last_change":"2023-10-16T14:30:16.285532+0000","last_active":"2023-10-16T14:30:16.290174+0000","last_peered":"2023-10-16T14:30:16.290174+0000","last_clean":"2023-10-16T14:30:16.290174+0000","last_became_active":"2023-10-16T14:30:16.284922+0000","last_became_peered":"2023-10-16T14:30:16.284922+0000","last_unstale":"2023-10-16T14:30:16.290174+0000","last_undegraded":"2023-10-16T14:30:16.290174+0000","last_fullsized":"2023-10-16T14:30:16.290174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.873972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.283724+0000","last_change":"2023-10-16T14:30:16.280593+0000","last_active":"2023-10-16T14:30:16.283724+0000","last_peered":"2023-10-16T14:30:16.283724+0000","last_clean":"2023-10-16T14:30:16.283724+0000","last_became_active":"2023-10-16T14:30:16.280256+0000","last_became_peered":"2023-10-16T14:30:16.280256+0000","last_unstale":"2023-10-16T14:30:16.283724+0000","last_undegraded":"2023-10-16T14:30:16.283724+0000","last_fullsized":"2023-10-16T14:30:16.283724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:34.169815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282218+0000","last_change":"2023-10-16T14:30:16.277071+0000","last_active":"2023-10-16T14:30:17.282218+0000","last_peered":"2023-10-16T14:30:17.282218+0000","last_clean":"2023-10-16T14:30:17.282218+0000","last_became_active":"2023-10-16T14:30:16.275980+0000","last_became_peered":"2023-10-16T14:30:16.275980+0000","last_unstale":"2023-10-16T14:30:17.282218+0000","last_undegraded":"2023-10-16T14:30:17.282218+0000","last_fullsized":"2023-10-16T14:30:17.282218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:19.024489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296604+0000","last_change":"2023-10-16T14:30:16.285768+0000","last_active":"2023-10-16T14:30:16.296604+0000","last_peered":"2023-10-16T14:30:16.296604+0000","last_clean":"2023-10-16T14:30:16.296604+0000","last_became_active":"2023-10-16T14:30:16.285473+0000","last_became_peered":"2023-10-16T14:30:16.285473+0000","last_unstale":"2023-10-16T14:30:16.296604+0000","last_undegraded":"2023-10-16T14:30:16.296604+0000","last_fullsized":"2023-10-16T14:30:16.296604+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:39.425799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.275025+0000","last_change":"2023-10-16T14:30:16.273191+0000","last_active":"2023-10-16T14:30:16.275025+0000","last_peered":"2023-10-16T14:30:16.275025+0000","last_clean":"2023-10-16T14:30:16.275025+0000","last_became_active":"2023-10-16T14:30:16.272897+0000","last_became_peered":"2023-10-16T14:30:16.272897+0000","last_unstale":"2023-10-16T14:30:16.275025+0000","last_undegraded":"2023-10-16T14:30:16.275025+0000","last_fullsized":"2023-10-16T14:30:16.275025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:18.674198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.274268+0000","last_change":"2023-10-16T14:30:16.271326+0000","last_active":"2023-10-16T14:30:16.274268+0000","last_peered":"2023-10-16T14:30:16.274268+0000","last_clean":"2023-10-16T14:30:16.274268+0000","last_became_active":"2023-10-16T14:30:16.270612+0000","last_became_peered":"2023-10-16T14:30:16.270612+0000","last_unstale":"2023-10-16T14:30:16.274268+0000","last_undegraded":"2023-10-16T14:30:16.274268+0000","last_fullsized":"2023-10-16T14:30:16.274268+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:30:36.816276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"64'4","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298445+0000","last_change":"2023-10-16T14:30:15.279144+0000","last_active":"2023-10-16T14:30:16.298445+0000","last_peered":"2023-10-16T14:30:16.298445+0000","last_clean":"2023-10-16T14:30:16.298445+0000","last_became_active":"2023-10-16T14:30:15.278906+0000","last_became_peered":"2023-10-16T14:30:15.278906+0000","last_unstale":"2023-10-16T14:30:16.298445+0000","last_undegraded":"2023-10-16T14:30:16.298445+0000","last_fullsized":"2023-10-16T14:30:16.298445+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:04.400754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.294458+0000","last_change":"2023-10-16T14:30:16.291623+0000","last_active":"2023-10-16T14:30:16.294458+0000","last_peered":"2023-10-16T14:30:16.294458+0000","last_clean":"2023-10-16T14:30:16.294458+0000","last_became_active":"2023-10-16T14:30:16.291355+0000","last_became_peered":"2023-10-16T14:30:16.291355+0000","last_unstale":"2023-10-16T14:30:16.294458+0000","last_undegraded":"2023-10-16T14:30:16.294458+0000","last_fullsized":"2023-10-16T14:30:16.294458+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:27.097251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"64'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284454+0000","last_change":"2023-10-16T14:30:15.279658+0000","last_active":"2023-10-16T14:30:17.284454+0000","last_peered":"2023-10-16T14:30:17.284454+0000","last_clean":"2023-10-16T14:30:17.284454+0000","last_became_active":"2023-10-16T14:30:15.279354+0000","last_became_peered":"2023-10-16T14:30:15.279354+0000","last_unstale":"2023-10-16T14:30:17.284454+0000","last_undegraded":"2023-10-16T14:30:17.284454+0000","last_fullsized":"2023-10-16T14:30:17.284454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:14.206153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290152+0000","last_change":"2023-10-16T14:30:16.280828+0000","last_active":"2023-10-16T14:30:16.290152+0000","last_peered":"2023-10-16T14:30:16.290152+0000","last_clean":"2023-10-16T14:30:16.290152+0000","last_became_active":"2023-10-16T14:30:16.280559+0000","last_became_peered":"2023-10-16T14:30:16.280559+0000","last_unstale":"2023-10-16T14:30:16.290152+0000","last_undegraded":"2023-10-16T14:30:16.290152+0000","last_fullsized":"2023-10-16T14:30:16.290152+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:27.556242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.286160+0000","last_change":"2023-10-16T14:30:15.264675+0000","last_active":"2023-10-16T14:30:16.286160+0000","last_peered":"2023-10-16T14:30:16.286160+0000","last_clean":"2023-10-16T14:30:16.286160+0000","last_became_active":"2023-10-16T14:30:15.264372+0000","last_became_peered":"2023-10-16T14:30:15.264372+0000","last_unstale":"2023-10-16T14:30:16.286160+0000","last_undegraded":"2023-10-16T14:30:16.286160+0000","last_fullsized":"2023-10-16T14:30:16.286160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:31.068794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293429+0000","last_change":"2023-10-16T14:30:16.291668+0000","last_active":"2023-10-16T14:30:16.293429+0000","last_peered":"2023-10-16T14:30:16.293429+0000","last_clean":"2023-10-16T14:30:16.293429+0000","last_became_active":"2023-10-16T14:30:16.290199+0000","last_became_peered":"2023-10-16T14:30:16.290199+0000","last_unstale":"2023-10-16T14:30:16.293429+0000","last_undegraded":"2023-10-16T14:30:16.293429+0000","last_fullsized":"2023-10-16T14:30:16.293429+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:03:19.060659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"64'2","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283756+0000","last_change":"2023-10-16T14:30:15.279865+0000","last_active":"2023-10-16T14:30:17.283756+0000","last_peered":"2023-10-16T14:30:17.283756+0000","last_clean":"2023-10-16T14:30:17.283756+0000","last_became_active":"2023-10-16T14:30:15.277434+0000","last_became_peered":"2023-10-16T14:30:15.277434+0000","last_unstale":"2023-10-16T14:30:17.283756+0000","last_undegraded":"2023-10-16T14:30:17.283756+0000","last_fullsized":"2023-10-16T14:30:17.283756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:52.708275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282705+0000","last_change":"2023-10-16T14:30:16.284193+0000","last_active":"2023-10-16T14:30:17.282705+0000","last_peered":"2023-10-16T14:30:17.282705+0000","last_clean":"2023-10-16T14:30:17.282705+0000","last_became_active":"2023-10-16T14:30:16.283906+0000","last_became_peered":"2023-10-16T14:30:16.283906+0000","last_unstale":"2023-10-16T14:30:17.282705+0000","last_undegraded":"2023-10-16T14:30:17.282705+0000","last_fullsized":"2023-10-16T14:30:17.282705+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:34.548998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"64'1","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298679+0000","last_change":"2023-10-16T14:30:15.274153+0000","last_active":"2023-10-16T14:30:16.298679+0000","last_peered":"2023-10-16T14:30:16.298679+0000","last_clean":"2023-10-16T14:30:16.298679+0000","last_became_active":"2023-10-16T14:30:15.271972+0000","last_became_peered":"2023-10-16T14:30:15.271972+0000","last_unstale":"2023-10-16T14:30:16.298679+0000","last_undegraded":"2023-10-16T14:30:16.298679+0000","last_fullsized":"2023-10-16T14:30:16.298679+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:50.984821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278014+0000","last_change":"2023-10-16T14:30:16.278022+0000","last_active":"2023-10-16T14:30:17.278014+0000","last_peered":"2023-10-16T14:30:17.278014+0000","last_clean":"2023-10-16T14:30:17.278014+0000","last_became_active":"2023-10-16T14:30:16.277786+0000","last_became_peered":"2023-10-16T14:30:16.277786+0000","last_unstale":"2023-10-16T14:30:17.278014+0000","last_undegraded":"2023-10-16T14:30:17.278014+0000","last_fullsized":"2023-10-16T14:30:17.278014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:43.792287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"64'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282752+0000","last_change":"2023-10-16T14:30:15.277899+0000","last_active":"2023-10-16T14:30:17.282752+0000","last_peered":"2023-10-16T14:30:17.282752+0000","last_clean":"2023-10-16T14:30:17.282752+0000","last_became_active":"2023-10-16T14:30:15.277145+0000","last_became_peered":"2023-10-16T14:30:15.277145+0000","last_unstale":"2023-10-16T14:30:17.282752+0000","last_undegraded":"2023-10-16T14:30:17.282752+0000","last_fullsized":"2023-10-16T14:30:17.282752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:41.209934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277836+0000","last_change":"2023-10-16T14:30:16.279889+0000","last_active":"2023-10-16T14:30:17.277836+0000","last_peered":"2023-10-16T14:30:17.277836+0000","last_clean":"2023-10-16T14:30:17.277836+0000","last_became_active":"2023-10-16T14:30:16.279178+0000","last_became_peered":"2023-10-16T14:30:16.279178+0000","last_unstale":"2023-10-16T14:30:17.277836+0000","last_undegraded":"2023-10-16T14:30:17.277836+0000","last_fullsized":"2023-10-16T14:30:17.277836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:04.145884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.276913+0000","last_change":"2023-10-16T14:30:15.264794+0000","last_active":"2023-10-16T14:30:16.276913+0000","last_peered":"2023-10-16T14:30:16.276913+0000","last_clean":"2023-10-16T14:30:16.276913+0000","last_became_active":"2023-10-16T14:30:15.263964+0000","last_became_peered":"2023-10-16T14:30:15.263964+0000","last_unstale":"2023-10-16T14:30:16.276913+0000","last_undegraded":"2023-10-16T14:30:16.276913+0000","last_fullsized":"2023-10-16T14:30:16.276913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:39.586641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.292845+0000","last_change":"2023-10-16T14:30:16.288860+0000","last_active":"2023-10-16T14:30:16.292845+0000","last_peered":"2023-10-16T14:30:16.292845+0000","last_clean":"2023-10-16T14:30:16.292845+0000","last_became_active":"2023-10-16T14:30:16.288413+0000","last_became_peered":"2023-10-16T14:30:16.288413+0000","last_unstale":"2023-10-16T14:30:16.292845+0000","last_undegraded":"2023-10-16T14:30:16.292845+0000","last_fullsized":"2023-10-16T14:30:16.292845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:23:48.421046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277855+0000","last_change":"2023-10-16T14:30:15.268275+0000","last_active":"2023-10-16T14:30:17.277855+0000","last_peered":"2023-10-16T14:30:17.277855+0000","last_clean":"2023-10-16T14:30:17.277855+0000","last_became_active":"2023-10-16T14:30:15.268007+0000","last_became_peered":"2023-10-16T14:30:15.268007+0000","last_unstale":"2023-10-16T14:30:17.277855+0000","last_undegraded":"2023-10-16T14:30:17.277855+0000","last_fullsized":"2023-10-16T14:30:17.277855+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:08.427082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284185+0000","last_change":"2023-10-16T14:30:16.285494+0000","last_active":"2023-10-16T14:30:17.284185+0000","last_peered":"2023-10-16T14:30:17.284185+0000","last_clean":"2023-10-16T14:30:17.284185+0000","last_became_active":"2023-10-16T14:30:16.285101+0000","last_became_peered":"2023-10-16T14:30:16.285101+0000","last_unstale":"2023-10-16T14:30:17.284185+0000","last_undegraded":"2023-10-16T14:30:17.284185+0000","last_fullsized":"2023-10-16T14:30:17.284185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:34.450415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284885+0000","last_change":"2023-10-16T14:30:15.270246+0000","last_active":"2023-10-16T14:30:17.284885+0000","last_peered":"2023-10-16T14:30:17.284885+0000","last_clean":"2023-10-16T14:30:17.284885+0000","last_became_active":"2023-10-16T14:30:15.269830+0000","last_became_peered":"2023-10-16T14:30:15.269830+0000","last_unstale":"2023-10-16T14:30:17.284885+0000","last_undegraded":"2023-10-16T14:30:17.284885+0000","last_fullsized":"2023-10-16T14:30:17.284885+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:02.116331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284502+0000","last_change":"2023-10-16T14:30:16.286018+0000","last_active":"2023-10-16T14:30:17.284502+0000","last_peered":"2023-10-16T14:30:17.284502+0000","last_clean":"2023-10-16T14:30:17.284502+0000","last_became_active":"2023-10-16T14:30:16.285738+0000","last_became_peered":"2023-10-16T14:30:16.285738+0000","last_unstale":"2023-10-16T14:30:17.284502+0000","last_undegraded":"2023-10-16T14:30:17.284502+0000","last_fullsized":"2023-10-16T14:30:17.284502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:08.405329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278895+0000","last_change":"2023-10-16T14:30:15.277563+0000","last_active":"2023-10-16T14:30:17.278895+0000","last_peered":"2023-10-16T14:30:17.278895+0000","last_clean":"2023-10-16T14:30:17.278895+0000","last_became_active":"2023-10-16T14:30:15.277231+0000","last_became_peered":"2023-10-16T14:30:15.277231+0000","last_unstale":"2023-10-16T14:30:17.278895+0000","last_undegraded":"2023-10-16T14:30:17.278895+0000","last_fullsized":"2023-10-16T14:30:17.278895+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:21.987501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293453+0000","last_change":"2023-10-16T14:30:16.279690+0000","last_active":"2023-10-16T14:30:16.293453+0000","last_peered":"2023-10-16T14:30:16.293453+0000","last_clean":"2023-10-16T14:30:16.293453+0000","last_became_active":"2023-10-16T14:30:16.279394+0000","last_became_peered":"2023-10-16T14:30:16.279394+0000","last_unstale":"2023-10-16T14:30:16.293453+0000","last_undegraded":"2023-10-16T14:30:16.293453+0000","last_fullsized":"2023-10-16T14:30:16.293453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:57.063012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"64'3","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285326+0000","last_change":"2023-10-16T14:30:15.278383+0000","last_active":"2023-10-16T14:30:17.285326+0000","last_peered":"2023-10-16T14:30:17.285326+0000","last_clean":"2023-10-16T14:30:17.285326+0000","last_became_active":"2023-10-16T14:30:15.278081+0000","last_became_peered":"2023-10-16T14:30:15.278081+0000","last_unstale":"2023-10-16T14:30:17.285326+0000","last_undegraded":"2023-10-16T14:30:17.285326+0000","last_fullsized":"2023-10-16T14:30:17.285326+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:40.004801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284713+0000","last_change":"2023-10-16T14:30:16.274192+0000","last_active":"2023-10-16T14:30:17.284713+0000","last_peered":"2023-10-16T14:30:17.284713+0000","last_clean":"2023-10-16T14:30:17.284713+0000","last_became_active":"2023-10-16T14:30:16.273902+0000","last_became_peered":"2023-10-16T14:30:16.273902+0000","last_unstale":"2023-10-16T14:30:17.284713+0000","last_undegraded":"2023-10-16T14:30:17.284713+0000","last_fullsized":"2023-10-16T14:30:17.284713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:09.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"64'6","reported_seq":17,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.300542+0000","last_change":"2023-10-16T14:30:15.269321+0000","last_active":"2023-10-16T14:30:16.300542+0000","last_peered":"2023-10-16T14:30:16.300542+0000","last_clean":"2023-10-16T14:30:16.300542+0000","last_became_active":"2023-10-16T14:30:15.268847+0000","last_became_peered":"2023-10-16T14:30:15.268847+0000","last_unstale":"2023-10-16T14:30:16.300542+0000","last_undegraded":"2023-10-16T14:30:16.300542+0000","last_fullsized":"2023-10-16T14:30:16.300542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:49.547519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296211+0000","last_change":"2023-10-16T14:30:16.295293+0000","last_active":"2023-10-16T14:30:16.296211+0000","last_peered":"2023-10-16T14:30:16.296211+0000","last_clean":"2023-10-16T14:30:16.296211+0000","last_became_active":"2023-10-16T14:30:16.293992+0000","last_became_peered":"2023-10-16T14:30:16.293992+0000","last_unstale":"2023-10-16T14:30:16.296211+0000","last_undegraded":"2023-10-16T14:30:16.296211+0000","last_fullsized":"2023-10-16T14:30:16.296211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:50.521569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.299034+0000","last_change":"2023-10-16T14:30:15.274587+0000","last_active":"2023-10-16T14:30:16.299034+0000","last_peered":"2023-10-16T14:30:16.299034+0000","last_clean":"2023-10-16T14:30:16.299034+0000","last_became_active":"2023-10-16T14:30:15.274275+0000","last_became_peered":"2023-10-16T14:30:15.274275+0000","last_unstale":"2023-10-16T14:30:16.299034+0000","last_undegraded":"2023-10-16T14:30:16.299034+0000","last_fullsized":"2023-10-16T14:30:16.299034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:16.420170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278740+0000","last_change":"2023-10-16T14:30:16.281020+0000","last_active":"2023-10-16T14:30:17.278740+0000","last_peered":"2023-10-16T14:30:17.278740+0000","last_clean":"2023-10-16T14:30:17.278740+0000","last_became_active":"2023-10-16T14:30:16.280854+0000","last_became_peered":"2023-10-16T14:30:16.280854+0000","last_unstale":"2023-10-16T14:30:17.278740+0000","last_undegraded":"2023-10-16T14:30:17.278740+0000","last_fullsized":"2023-10-16T14:30:17.278740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:24.053085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284689+0000","last_change":"2023-10-16T14:30:15.271411+0000","last_active":"2023-10-16T14:30:17.284689+0000","last_peered":"2023-10-16T14:30:17.284689+0000","last_clean":"2023-10-16T14:30:17.284689+0000","last_became_active":"2023-10-16T14:30:15.271128+0000","last_became_peered":"2023-10-16T14:30:15.271128+0000","last_unstale":"2023-10-16T14:30:17.284689+0000","last_undegraded":"2023-10-16T14:30:17.284689+0000","last_fullsized":"2023-10-16T14:30:17.284689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:54.265571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.269326+0000","last_change":"2023-10-16T14:30:16.268088+0000","last_active":"2023-10-16T14:30:16.269326+0000","last_peered":"2023-10-16T14:30:16.269326+0000","last_clean":"2023-10-16T14:30:16.269326+0000","last_became_active":"2023-10-16T14:30:16.267814+0000","last_became_peered":"2023-10-16T14:30:16.267814+0000","last_unstale":"2023-10-16T14:30:16.269326+0000","last_undegraded":"2023-10-16T14:30:16.269326+0000","last_fullsized":"2023-10-16T14:30:16.269326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.873972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.255383+0000","last_change":"2023-10-16T14:30:15.274467+0000","last_active":"2023-10-16T14:30:16.255383+0000","last_peered":"2023-10-16T14:30:16.255383+0000","last_clean":"2023-10-16T14:30:16.255383+0000","last_became_active":"2023-10-16T14:30:15.273326+0000","last_became_peered":"2023-10-16T14:30:15.273326+0000","last_unstale":"2023-10-16T14:30:16.255383+0000","last_undegraded":"2023-10-16T14:30:16.255383+0000","last_fullsized":"2023-10-16T14:30:16.255383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:52.708275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284215+0000","last_change":"2023-10-16T14:30:16.282230+0000","last_active":"2023-10-16T14:30:17.284215+0000","last_peered":"2023-10-16T14:30:17.284215+0000","last_clean":"2023-10-16T14:30:17.284215+0000","last_became_active":"2023-10-16T14:30:16.281969+0000","last_became_peered":"2023-10-16T14:30:16.281969+0000","last_unstale":"2023-10-16T14:30:17.284215+0000","last_undegraded":"2023-10-16T14:30:17.284215+0000","last_fullsized":"2023-10-16T14:30:17.284215+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:40.686426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"64'8","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277983+0000","last_change":"2023-10-16T14:30:15.266815+0000","last_active":"2023-10-16T14:30:17.277983+0000","last_peered":"2023-10-16T14:30:17.277983+0000","last_clean":"2023-10-16T14:30:17.277983+0000","last_became_active":"2023-10-16T14:30:15.265618+0000","last_became_peered":"2023-10-16T14:30:15.265618+0000","last_unstale":"2023-10-16T14:30:17.277983+0000","last_undegraded":"2023-10-16T14:30:17.277983+0000","last_fullsized":"2023-10-16T14:30:17.277983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:26.080451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.285210+0000","last_change":"2023-10-16T14:30:16.284141+0000","last_active":"2023-10-16T14:30:16.285210+0000","last_peered":"2023-10-16T14:30:16.285210+0000","last_clean":"2023-10-16T14:30:16.285210+0000","last_became_active":"2023-10-16T14:30:16.282984+0000","last_became_peered":"2023-10-16T14:30:16.282984+0000","last_unstale":"2023-10-16T14:30:16.285210+0000","last_undegraded":"2023-10-16T14:30:16.285210+0000","last_fullsized":"2023-10-16T14:30:16.285210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:45.727988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"64'1","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.297560+0000","last_change":"2023-10-16T14:30:15.267941+0000","last_active":"2023-10-16T14:30:16.297560+0000","last_peered":"2023-10-16T14:30:16.297560+0000","last_clean":"2023-10-16T14:30:16.297560+0000","last_became_active":"2023-10-16T14:30:15.263703+0000","last_became_peered":"2023-10-16T14:30:15.263703+0000","last_unstale":"2023-10-16T14:30:16.297560+0000","last_undegraded":"2023-10-16T14:30:16.297560+0000","last_fullsized":"2023-10-16T14:30:16.297560+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:26.909324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278740+0000","last_change":"2023-10-16T14:30:16.279668+0000","last_active":"2023-10-16T14:30:17.278740+0000","last_peered":"2023-10-16T14:30:17.278740+0000","last_clean":"2023-10-16T14:30:17.278740+0000","last_became_active":"2023-10-16T14:30:16.279347+0000","last_became_peered":"2023-10-16T14:30:16.279347+0000","last_unstale":"2023-10-16T14:30:17.278740+0000","last_undegraded":"2023-10-16T14:30:17.278740+0000","last_fullsized":"2023-10-16T14:30:17.278740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:42.370647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'2","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284505+0000","last_change":"2023-10-16T14:30:15.272893+0000","last_active":"2023-10-16T14:30:17.284505+0000","last_peered":"2023-10-16T14:30:17.284505+0000","last_clean":"2023-10-16T14:30:17.284505+0000","last_became_active":"2023-10-16T14:30:15.272311+0000","last_became_peered":"2023-10-16T14:30:15.272311+0000","last_unstale":"2023-10-16T14:30:17.284505+0000","last_undegraded":"2023-10-16T14:30:17.284505+0000","last_fullsized":"2023-10-16T14:30:17.284505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:10.133346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298719+0000","last_change":"2023-10-16T14:30:16.297811+0000","last_active":"2023-10-16T14:30:16.298719+0000","last_peered":"2023-10-16T14:30:16.298719+0000","last_clean":"2023-10-16T14:30:16.298719+0000","last_became_active":"2023-10-16T14:30:16.297480+0000","last_became_peered":"2023-10-16T14:30:16.297480+0000","last_unstale":"2023-10-16T14:30:16.298719+0000","last_undegraded":"2023-10-16T14:30:16.298719+0000","last_fullsized":"2023-10-16T14:30:16.298719+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:04.145884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"64'1","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.284402+0000","last_change":"2023-10-16T14:30:15.270596+0000","last_active":"2023-10-16T14:30:16.284402+0000","last_peered":"2023-10-16T14:30:16.284402+0000","last_clean":"2023-10-16T14:30:16.284402+0000","last_became_active":"2023-10-16T14:30:15.270432+0000","last_became_peered":"2023-10-16T14:30:15.270432+0000","last_unstale":"2023-10-16T14:30:16.284402+0000","last_undegraded":"2023-10-16T14:30:16.284402+0000","last_fullsized":"2023-10-16T14:30:16.284402+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:10.411620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.289600+0000","last_change":"2023-10-16T14:30:16.284093+0000","last_active":"2023-10-16T14:30:16.289600+0000","last_peered":"2023-10-16T14:30:16.289600+0000","last_clean":"2023-10-16T14:30:16.289600+0000","last_became_active":"2023-10-16T14:30:16.283778+0000","last_became_peered":"2023-10-16T14:30:16.283778+0000","last_unstale":"2023-10-16T14:30:16.289600+0000","last_undegraded":"2023-10-16T14:30:16.289600+0000","last_fullsized":"2023-10-16T14:30:16.289600+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:09.443883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.255099+0000","last_change":"2023-10-16T14:30:15.275930+0000","last_active":"2023-10-16T14:30:16.255099+0000","last_peered":"2023-10-16T14:30:16.255099+0000","last_clean":"2023-10-16T14:30:16.255099+0000","last_became_active":"2023-10-16T14:30:15.275653+0000","last_became_peered":"2023-10-16T14:30:15.275653+0000","last_unstale":"2023-10-16T14:30:16.255099+0000","last_undegraded":"2023-10-16T14:30:16.255099+0000","last_fullsized":"2023-10-16T14:30:16.255099+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:39.586641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293417+0000","last_change":"2023-10-16T14:30:16.279050+0000","last_active":"2023-10-16T14:30:16.293417+0000","last_peered":"2023-10-16T14:30:16.293417+0000","last_clean":"2023-10-16T14:30:16.293417+0000","last_became_active":"2023-10-16T14:30:16.278713+0000","last_became_peered":"2023-10-16T14:30:16.278713+0000","last_unstale":"2023-10-16T14:30:16.293417+0000","last_undegraded":"2023-10-16T14:30:16.293417+0000","last_fullsized":"2023-10-16T14:30:16.293417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:16.561457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298757+0000","last_change":"2023-10-16T14:30:15.272414+0000","last_active":"2023-10-16T14:30:16.298757+0000","last_peered":"2023-10-16T14:30:16.298757+0000","last_clean":"2023-10-16T14:30:16.298757+0000","last_became_active":"2023-10-16T14:30:15.271597+0000","last_became_peered":"2023-10-16T14:30:15.271597+0000","last_unstale":"2023-10-16T14:30:16.298757+0000","last_undegraded":"2023-10-16T14:30:16.298757+0000","last_fullsized":"2023-10-16T14:30:16.298757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:26:43.026194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.288364+0000","last_change":"2023-10-16T14:30:16.286327+0000","last_active":"2023-10-16T14:30:16.288364+0000","last_peered":"2023-10-16T14:30:16.288364+0000","last_clean":"2023-10-16T14:30:16.288364+0000","last_became_active":"2023-10-16T14:30:16.286068+0000","last_became_peered":"2023-10-16T14:30:16.286068+0000","last_unstale":"2023-10-16T14:30:16.288364+0000","last_undegraded":"2023-10-16T14:30:16.288364+0000","last_fullsized":"2023-10-16T14:30:16.288364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:53.776633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.273244+0000","last_change":"2023-10-16T14:30:15.275710+0000","last_active":"2023-10-16T14:30:16.273244+0000","last_peered":"2023-10-16T14:30:16.273244+0000","last_clean":"2023-10-16T14:30:16.273244+0000","last_became_active":"2023-10-16T14:30:15.275421+0000","last_became_peered":"2023-10-16T14:30:15.275421+0000","last_unstale":"2023-10-16T14:30:16.273244+0000","last_undegraded":"2023-10-16T14:30:16.273244+0000","last_fullsized":"2023-10-16T14:30:16.273244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:20.367515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.282754+0000","last_change":"2023-10-16T14:30:16.276370+0000","last_active":"2023-10-16T14:30:16.282754+0000","last_peered":"2023-10-16T14:30:16.282754+0000","last_clean":"2023-10-16T14:30:16.282754+0000","last_became_active":"2023-10-16T14:30:16.275612+0000","last_became_peered":"2023-10-16T14:30:16.275612+0000","last_unstale":"2023-10-16T14:30:16.282754+0000","last_undegraded":"2023-10-16T14:30:16.282754+0000","last_fullsized":"2023-10-16T14:30:16.282754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:36.513930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277804+0000","last_change":"2023-10-16T14:30:15.276675+0000","last_active":"2023-10-16T14:30:17.277804+0000","last_peered":"2023-10-16T14:30:17.277804+0000","last_clean":"2023-10-16T14:30:17.277804+0000","last_became_active":"2023-10-16T14:30:15.276361+0000","last_became_peered":"2023-10-16T14:30:15.276361+0000","last_unstale":"2023-10-16T14:30:17.277804+0000","last_undegraded":"2023-10-16T14:30:17.277804+0000","last_fullsized":"2023-10-16T14:30:17.277804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:47.590806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277803+0000","last_change":"2023-10-16T14:30:16.263916+0000","last_active":"2023-10-16T14:30:17.277803+0000","last_peered":"2023-10-16T14:30:17.277803+0000","last_clean":"2023-10-16T14:30:17.277803+0000","last_became_active":"2023-10-16T14:30:16.262297+0000","last_became_peered":"2023-10-16T14:30:16.262297+0000","last_unstale":"2023-10-16T14:30:17.277803+0000","last_undegraded":"2023-10-16T14:30:17.277803+0000","last_fullsized":"2023-10-16T14:30:17.277803+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:13.661918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282867+0000","last_change":"2023-10-16T14:30:15.274645+0000","last_active":"2023-10-16T14:30:17.282867+0000","last_peered":"2023-10-16T14:30:17.282867+0000","last_clean":"2023-10-16T14:30:17.282867+0000","last_became_active":"2023-10-16T14:30:15.274331+0000","last_became_peered":"2023-10-16T14:30:15.274331+0000","last_unstale":"2023-10-16T14:30:17.282867+0000","last_undegraded":"2023-10-16T14:30:17.282867+0000","last_fullsized":"2023-10-16T14:30:17.282867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:54.550376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296218+0000","last_change":"2023-10-16T14:30:16.295329+0000","last_active":"2023-10-16T14:30:16.296218+0000","last_peered":"2023-10-16T14:30:16.296218+0000","last_clean":"2023-10-16T14:30:16.296218+0000","last_became_active":"2023-10-16T14:30:16.294969+0000","last_became_peered":"2023-10-16T14:30:16.294969+0000","last_unstale":"2023-10-16T14:30:16.296218+0000","last_undegraded":"2023-10-16T14:30:16.296218+0000","last_fullsized":"2023-10-16T14:30:16.296218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:44:16.883415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.256236+0000","last_change":"2023-10-16T14:30:15.275135+0000","last_active":"2023-10-16T14:30:16.256236+0000","last_peered":"2023-10-16T14:30:16.256236+0000","last_clean":"2023-10-16T14:30:16.256236+0000","last_became_active":"2023-10-16T14:30:15.274869+0000","last_became_peered":"2023-10-16T14:30:15.274869+0000","last_unstale":"2023-10-16T14:30:16.256236+0000","last_undegraded":"2023-10-16T14:30:16.256236+0000","last_fullsized":"2023-10-16T14:30:16.256236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:21.798778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278390+0000","last_change":"2023-10-16T14:30:16.275774+0000","last_active":"2023-10-16T14:30:17.278390+0000","last_peered":"2023-10-16T14:30:17.278390+0000","last_clean":"2023-10-16T14:30:17.278390+0000","last_became_active":"2023-10-16T14:30:16.275503+0000","last_became_peered":"2023-10-16T14:30:16.275503+0000","last_unstale":"2023-10-16T14:30:17.278390+0000","last_undegraded":"2023-10-16T14:30:17.278390+0000","last_fullsized":"2023-10-16T14:30:17.278390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:25.396087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285617+0000","last_change":"2023-10-16T14:30:15.278789+0000","last_active":"2023-10-16T14:30:17.285617+0000","last_peered":"2023-10-16T14:30:17.285617+0000","last_clean":"2023-10-16T14:30:17.285617+0000","last_became_active":"2023-10-16T14:30:15.278360+0000","last_became_peered":"2023-10-16T14:30:15.278360+0000","last_unstale":"2023-10-16T14:30:17.285617+0000","last_undegraded":"2023-10-16T14:30:17.285617+0000","last_fullsized":"2023-10-16T14:30:17.285617+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:26:35.162457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285061+0000","last_change":"2023-10-16T14:30:16.274257+0000","last_active":"2023-10-16T14:30:17.285061+0000","last_peered":"2023-10-16T14:30:17.285061+0000","last_clean":"2023-10-16T14:30:17.285061+0000","last_became_active":"2023-10-16T14:30:16.273967+0000","last_became_peered":"2023-10-16T14:30:16.273967+0000","last_unstale":"2023-10-16T14:30:17.285061+0000","last_undegraded":"2023-10-16T14:30:17.285061+0000","last_fullsized":"2023-10-16T14:30:17.285061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:28.253135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.261797+0000","last_change":"2023-10-16T14:30:15.264857+0000","last_active":"2023-10-16T14:30:16.261797+0000","last_peered":"2023-10-16T14:30:16.261797+0000","last_clean":"2023-10-16T14:30:16.261797+0000","last_became_active":"2023-10-16T14:30:15.264568+0000","last_became_peered":"2023-10-16T14:30:15.264568+0000","last_unstale":"2023-10-16T14:30:16.261797+0000","last_undegraded":"2023-10-16T14:30:16.261797+0000","last_fullsized":"2023-10-16T14:30:16.261797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:56.567240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.281065+0000","last_change":"2023-10-16T14:30:16.273426+0000","last_active":"2023-10-16T14:30:16.281065+0000","last_peered":"2023-10-16T14:30:16.281065+0000","last_clean":"2023-10-16T14:30:16.281065+0000","last_became_active":"2023-10-16T14:30:16.270683+0000","last_became_peered":"2023-10-16T14:30:16.270683+0000","last_unstale":"2023-10-16T14:30:16.281065+0000","last_undegraded":"2023-10-16T14:30:16.281065+0000","last_fullsized":"2023-10-16T14:30:16.281065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:55.282371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285461+0000","last_change":"2023-10-16T14:30:15.276481+0000","last_active":"2023-10-16T14:30:17.285461+0000","last_peered":"2023-10-16T14:30:17.285461+0000","last_clean":"2023-10-16T14:30:17.285461+0000","last_became_active":"2023-10-16T14:30:15.276134+0000","last_became_peered":"2023-10-16T14:30:15.276134+0000","last_unstale":"2023-10-16T14:30:17.285461+0000","last_undegraded":"2023-10-16T14:30:17.285461+0000","last_fullsized":"2023-10-16T14:30:17.285461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:26:43.026194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296613+0000","last_change":"2023-10-16T14:30:16.279024+0000","last_active":"2023-10-16T14:30:16.296613+0000","last_peered":"2023-10-16T14:30:16.296613+0000","last_clean":"2023-10-16T14:30:16.296613+0000","last_became_active":"2023-10-16T14:30:16.278721+0000","last_became_peered":"2023-10-16T14:30:16.278721+0000","last_unstale":"2023-10-16T14:30:16.296613+0000","last_undegraded":"2023-10-16T14:30:16.296613+0000","last_fullsized":"2023-10-16T14:30:16.296613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:11.428420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282925+0000","last_change":"2023-10-16T14:30:15.275836+0000","last_active":"2023-10-16T14:30:17.282925+0000","last_peered":"2023-10-16T14:30:17.282925+0000","last_clean":"2023-10-16T14:30:17.282925+0000","last_became_active":"2023-10-16T14:30:15.275012+0000","last_became_peered":"2023-10-16T14:30:15.275012+0000","last_unstale":"2023-10-16T14:30:17.282925+0000","last_undegraded":"2023-10-16T14:30:17.282925+0000","last_fullsized":"2023-10-16T14:30:17.282925+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:34.029244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293161+0000","last_change":"2023-10-16T14:30:16.292065+0000","last_active":"2023-10-16T14:30:16.293161+0000","last_peered":"2023-10-16T14:30:16.293161+0000","last_clean":"2023-10-16T14:30:16.293161+0000","last_became_active":"2023-10-16T14:30:16.291604+0000","last_became_peered":"2023-10-16T14:30:16.291604+0000","last_unstale":"2023-10-16T14:30:16.293161+0000","last_undegraded":"2023-10-16T14:30:16.293161+0000","last_fullsized":"2023-10-16T14:30:16.293161+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:23.447810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.260835+0000","last_change":"2023-10-16T14:30:15.279688+0000","last_active":"2023-10-16T14:30:16.260835+0000","last_peered":"2023-10-16T14:30:16.260835+0000","last_clean":"2023-10-16T14:30:16.260835+0000","last_became_active":"2023-10-16T14:30:15.279394+0000","last_became_peered":"2023-10-16T14:30:15.279394+0000","last_unstale":"2023-10-16T14:30:16.260835+0000","last_undegraded":"2023-10-16T14:30:16.260835+0000","last_fullsized":"2023-10-16T14:30:16.260835+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:00.533698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282208+0000","last_change":"2023-10-16T14:30:16.268774+0000","last_active":"2023-10-16T14:30:17.282208+0000","last_peered":"2023-10-16T14:30:17.282208+0000","last_clean":"2023-10-16T14:30:17.282208+0000","last_became_active":"2023-10-16T14:30:16.268460+0000","last_became_peered":"2023-10-16T14:30:16.268460+0000","last_unstale":"2023-10-16T14:30:17.282208+0000","last_undegraded":"2023-10-16T14:30:17.282208+0000","last_fullsized":"2023-10-16T14:30:17.282208+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:37.791494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277976+0000","last_change":"2023-10-16T14:30:15.266274+0000","last_active":"2023-10-16T14:30:17.277976+0000","last_peered":"2023-10-16T14:30:17.277976+0000","last_clean":"2023-10-16T14:30:17.277976+0000","last_became_active":"2023-10-16T14:30:15.265975+0000","last_became_peered":"2023-10-16T14:30:15.265975+0000","last_unstale":"2023-10-16T14:30:17.277976+0000","last_undegraded":"2023-10-16T14:30:17.277976+0000","last_fullsized":"2023-10-16T14:30:17.277976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:44:15.866615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278926+0000","last_change":"2023-10-16T14:30:16.282250+0000","last_active":"2023-10-16T14:30:17.278926+0000","last_peered":"2023-10-16T14:30:17.278926+0000","last_clean":"2023-10-16T14:30:17.278926+0000","last_became_active":"2023-10-16T14:30:16.281994+0000","last_became_peered":"2023-10-16T14:30:16.281994+0000","last_unstale":"2023-10-16T14:30:17.278926+0000","last_undegraded":"2023-10-16T14:30:17.278926+0000","last_fullsized":"2023-10-16T14:30:17.278926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:38.169051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.297339+0000","last_change":"2023-10-16T14:30:15.269396+0000","last_active":"2023-10-16T14:30:16.297339+0000","last_peered":"2023-10-16T14:30:16.297339+0000","last_clean":"2023-10-16T14:30:16.297339+0000","last_became_active":"2023-10-16T14:30:15.267714+0000","last_became_peered":"2023-10-16T14:30:15.267714+0000","last_unstale":"2023-10-16T14:30:16.297339+0000","last_undegraded":"2023-10-16T14:30:16.297339+0000","last_fullsized":"2023-10-16T14:30:16.297339+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:50.984821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"52'139","reported_seq":294,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.257464+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:30:16.257464+0000","last_peered":"2023-10-16T14:30:16.257464+0000","last_clean":"2023-10-16T14:30:16.257464+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:30:16.257464+0000","last_undegraded":"2023-10-16T14:30:16.257464+0000","last_fullsized":"2023-10-16T14:30:16.257464+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":1359872,"data_stored":951938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.121,"15min":1.1639999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.42799999999999999},"max":{"1min":1.385,"5min":1.752,"15min":1.9159999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.179,"15min":1.1359999999999999},"min":{"1min":0.85399999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"max":{"1min":1.476,"5min":1.673,"15min":1.673},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2170000000000001,"15min":1.1779999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.69599999999999995},"max":{"1min":1.611,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.429},{"interface":"front","average":{"1min":1.165,"5min":1.2290000000000001,"15min":1.22},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.67600000000000005},"max":{"1min":1.5309999999999999,"5min":1.716,"15min":2.0219999999999998},"last":1.105}]},{"osd":2,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.321,"15min":1.2889999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.63500000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.137,"5min":1.232,"15min":1.2230000000000001},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.63400000000000001},"max":{"1min":1.4490000000000001,"5min":1.7250000000000001,"15min":1.885},"last":1.5289999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3049999999999999,"15min":1.2889999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.75900000000000001},"max":{"1min":1.583,"5min":1.746,"15min":2.048},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.375,"15min":1.371},"min":{"1min":0.99199999999999999,"5min":0.93000000000000005,"15min":0.71899999999999997},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.2509999999999999},"last":1.008}]},{"osd":4,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.365,"15min":1.3680000000000001},"min":{"1min":0.96199999999999997,"5min":0.82899999999999996,"15min":0.67600000000000005},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2410000000000001,"15min":1.278},"min":{"1min":0.82099999999999995,"5min":0.73099999999999998,"15min":0.496},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.153},"last":0.94599999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.407,"15min":1.4239999999999999},"min":{"1min":0.94399999999999995,"5min":0.79800000000000004,"15min":0.61199999999999999},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2530000000000001},"last":1.149},{"interface":"front","average":{"1min":1.492,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":0.998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2970000000000002},"last":1.385}]},{"osd":6,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4299999999999999},"min":{"1min":1.056,"5min":0.92200000000000004,"15min":0.63100000000000001},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.048},{"interface":"front","average":{"1min":1.573,"5min":1.494,"15min":1.522},"min":{"1min":1.1459999999999999,"5min":0.92500000000000004,"15min":0.59299999999999997},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.427},"last":1.2869999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575017,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36096,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335744,"statfs":{"total":96636764160,"available":96599801856,"internally_reserved":0,"allocated":770048,"data_stored":362152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1619999999999999,"15min":1.1599999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.66500000000000004},"max":{"1min":1.667,"5min":1.77,"15min":2.2160000000000002},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2529999999999999,"15min":1.228},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.6459999999999999,"5min":2.0529999999999999,"15min":3.863},"last":1.2}]},{"osd":1,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.284,"15min":1.2649999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.415,"5min":1.9330000000000001,"15min":2.2719999999999998},"last":1.425},{"interface":"front","average":{"1min":1.401,"5min":1.3120000000000001,"15min":1.321},"min":{"1min":0.94699999999999995,"5min":0.84099999999999997,"15min":0.80500000000000005},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":1.0900000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.2030000000000001,"15min":1.218},"min":{"1min":0.747,"5min":0.747,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.653,"15min":3.5800000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3600000000000001,"15min":1.3660000000000001},"min":{"1min":0.95599999999999996,"5min":0.88,"15min":0.70599999999999996},"max":{"1min":1.8799999999999999,"5min":2.1429999999999998,"15min":2.7250000000000001},"last":1.607}]},{"osd":3,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.347,"15min":1.345},"min":{"1min":0.97099999999999997,"5min":0.89400000000000002,"15min":0.77500000000000002},"max":{"1min":1.6819999999999999,"5min":1.875,"15min":4.9790000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.3240000000000001,"15min":1.3680000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.71399999999999997},"max":{"1min":1.546,"5min":1.8420000000000001,"15min":5.4160000000000004},"last":1.3440000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.302,"15min":1.331},"min":{"1min":0.91100000000000003,"5min":0.78700000000000003,"15min":0.59499999999999997},"max":{"1min":2.1070000000000002,"5min":2.2360000000000002,"15min":5},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.3580000000000001,"15min":1.401},"min":{"1min":0.878,"5min":0.85699999999999998,"15min":0.70799999999999996},"max":{"1min":1.9279999999999999,"5min":1.9810000000000001,"15min":5.4640000000000004},"last":1.29}]},{"osd":5,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3080000000000001,"15min":1.3400000000000001},"min":{"1min":0.92300000000000004,"5min":0.85299999999999998,"15min":0.754},"max":{"1min":1.794,"5min":1.794,"15min":3.9279999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.494,"15min":1.5249999999999999},"min":{"1min":1.194,"5min":0.97399999999999998,"15min":0.66100000000000003},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":3.6749999999999998},"last":1.6679999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5720000000000001,"15min":1.5840000000000001},"min":{"1min":1.2589999999999999,"5min":1.1439999999999999,"15min":0.69999999999999996},"max":{"1min":1.986,"5min":2.1139999999999999,"15min":4.6859999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.534,"15min":1.587},"min":{"1min":1.2589999999999999,"5min":1.01,"15min":0.68400000000000005},"max":{"1min":1.986,"5min":2.0139999999999998,"15min":4.883},"last":1.5589999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575017,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":765952,"data_stored":361550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1200000000000001,"15min":1.1180000000000001},"min":{"1min":0.68400000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.196,"15min":1.161},"min":{"1min":0.85799999999999998,"5min":0.78100000000000003,"15min":0.66900000000000004},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":2.2629999999999999},"last":0.70799999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.1919999999999999,"15min":1.171},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.64600000000000002},"max":{"1min":1.6779999999999999,"5min":1.726,"15min":1.726},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2270000000000001,"15min":1.2450000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.748},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.022}]},{"osd":2,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2769999999999999,"15min":1.254},"min":{"1min":0.91600000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.3759999999999999,"15min":1.369},"min":{"1min":1.0549999999999999,"5min":0.95199999999999996,"15min":0.84099999999999997},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.3500000000000001},"last":0.82899999999999996}]},{"osd":3,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2909999999999999,"15min":1.274},"min":{"1min":0.95899999999999996,"5min":0.95599999999999996,"15min":0.76200000000000001},"max":{"1min":1.742,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4379999999999999,"15min":1.4079999999999999},"min":{"1min":1.0609999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.0780000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3660000000000001,"15min":1.341},"min":{"1min":0.85899999999999999,"5min":0.83499999999999996,"15min":0.54700000000000004},"max":{"1min":1.752,"5min":1.9610000000000001,"15min":2.863},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.714,"5min":1.55,"15min":1.528},"min":{"1min":0.93300000000000005,"5min":0.82199999999999995,"15min":0.61699999999999999},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.175}]},{"osd":6,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4059999999999999,"15min":1.4039999999999999},"min":{"1min":0.88800000000000001,"5min":0.72699999999999998,"15min":0.61799999999999999},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":1.349},{"interface":"front","average":{"1min":1.448,"5min":1.4630000000000001,"15min":1.4830000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.70199999999999996},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.794},"last":1.2789999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.593,"15min":1.571},"min":{"1min":1.397,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.641,"15min":1.6180000000000001},"min":{"1min":1.1180000000000001,"5min":1.038,"15min":0.67700000000000005},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.415},"last":1.2230000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575018,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45832,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326008,"statfs":{"total":96636764160,"available":96589832192,"internally_reserved":0,"allocated":774144,"data_stored":363092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1830000000000001,"15min":1.202},"min":{"1min":0.72899999999999998,"5min":0.71899999999999997,"15min":0.57599999999999996},"max":{"1min":1.7789999999999999,"5min":1.782,"15min":5.367},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1919999999999999,"15min":1.236},"min":{"1min":0.82099999999999995,"5min":0.75,"15min":0.65000000000000002},"max":{"1min":1.641,"5min":1.8500000000000001,"15min":6.3159999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2050000000000001,"15min":1.1679999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.65900000000000003},"max":{"1min":1.5600000000000001,"5min":1.609,"15min":1.78},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.222,"15min":1.2909999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.782,"5min":1.782,"15min":5.4640000000000004},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3919999999999999,"15min":1.4259999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.69499999999999995},"max":{"1min":1.7410000000000001,"5min":1.788,"15min":5.617},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3660000000000001,"15min":1.411},"min":{"1min":0.95899999999999996,"5min":0.80800000000000005,"15min":0.745},"max":{"1min":1.7390000000000001,"5min":1.788,"15min":6.391},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.341,"15min":1.381},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.71399999999999997},"max":{"1min":1.5900000000000001,"5min":1.7370000000000001,"15min":7.4089999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.327,"15min":1.2889999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":2.0339999999999998},"last":0.437}]},{"osd":5,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.2829999999999999,"15min":1.347},"min":{"1min":0.95499999999999996,"5min":0.85099999999999998,"15min":0.48699999999999999},"max":{"1min":1.696,"5min":1.7490000000000001,"15min":5.4109999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.349,"15min":1.4419999999999999},"min":{"1min":0.98999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.5880000000000001,"5min":1.78,"15min":9.8239999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3300000000000001,"15min":1.2969999999999999},"min":{"1min":0.96199999999999997,"5min":0.76500000000000001,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":1.8089999999999999,"15min":2.4729999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.54,"5min":1.4470000000000001,"15min":1.51},"min":{"1min":1.298,"5min":0.92100000000000004,"15min":0.44700000000000001},"max":{"1min":1.891,"5min":1.891,"15min":5.782},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.474,"15min":1.53},"min":{"1min":1.1539999999999999,"5min":0.90800000000000003,"15min":0.78100000000000003},"max":{"1min":1.7549999999999999,"5min":1.9099999999999999,"15min":6.4379999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4810000000000001,"15min":1.4970000000000001},"min":{"1min":1.2110000000000001,"5min":1.012,"15min":0.67400000000000004},"max":{"1min":1.8080000000000001,"5min":1.8380000000000001,"15min":3.7799999999999998},"last":0.49399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575017,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40260,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331580,"statfs":{"total":96636764160,"available":96595537920,"internally_reserved":0,"allocated":774144,"data_stored":362118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.89900000000000002,"15min":0.93100000000000005},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.35699999999999998},"max":{"1min":1.4419999999999999,"5min":1.518,"15min":2.4420000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0780000000000001,"15min":1.04},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.42099999999999999},"max":{"1min":1.4850000000000001,"5min":1.746,"15min":1.982},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.96099999999999997,"15min":0.96199999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.32500000000000001},"max":{"1min":1.0069999999999999,"5min":1.8799999999999999,"15min":2.149},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.159,"15min":1.165},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.4259999999999999,"5min":1.6759999999999999,"15min":2.1600000000000001},"last":0.314}]},{"osd":2,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.3759999999999999,"15min":1.339},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.44800000000000001},"max":{"1min":1.639,"5min":1.9550000000000001,"15min":2.1259999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4670000000000001,"15min":1.4390000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.501},"max":{"1min":1.77,"5min":1.9470000000000001,"15min":2.6299999999999999},"last":0.5}]},{"osd":4,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.2569999999999999,"15min":1.2929999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.5049999999999999,"5min":1.7170000000000001,"15min":2.4529999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.363,"15min":1.389},"min":{"1min":1.038,"5min":1.002,"15min":0.75},"max":{"1min":1.845,"5min":1.909,"15min":2.3700000000000001},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5449999999999999,"15min":1.573},"min":{"1min":1.103,"5min":1.103,"15min":0.83799999999999997},"max":{"1min":1.907,"5min":2.0569999999999999,"15min":2.552},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.383,"5min":1.4790000000000001,"15min":1.522},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.609,"5min":2.3119999999999998,"15min":2.7589999999999999},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4259999999999999,"15min":1.4530000000000001},"min":{"1min":0.90600000000000003,"5min":0.90400000000000003,"15min":0.73799999999999999},"max":{"1min":1.976,"5min":2.008,"15min":2.3399999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.522,"15min":1.552},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9510000000000001,"5min":2.1230000000000002,"15min":2.347},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5980000000000001,"15min":1.6279999999999999},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.8440000000000001,"5min":2.395,"15min":2.8820000000000001},"last":0.751},{"interface":"front","average":{"1min":1.522,"5min":1.587,"15min":1.5840000000000001},"min":{"1min":1.038,"5min":1.038,"15min":0.86399999999999999},"max":{"1min":1.8819999999999999,"5min":2.0710000000000002,"15min":2.6179999999999999},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575018,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40268,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331572,"statfs":{"total":96636764160,"available":96595529728,"internally_reserved":0,"allocated":778240,"data_stored":363114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.88100000000000001,"15min":0.873},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.92500000000000004,"5min":1.7150000000000001,"15min":2.472},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.0249999999999999,"15min":1.036},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.488,"5min":1.7070000000000001,"15min":2.8879999999999999},"last":1.0409999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.98399999999999999,"15min":1},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3340000000000001,"5min":1.571,"15min":2.8090000000000002},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1659999999999999,"15min":1.1479999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":2.2829999999999999},"last":1.3640000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.286,"15min":1.216},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.728,"5min":2.085,"15min":2.085},"last":0.996},{"interface":"front","average":{"1min":1.05,"5min":1.2909999999999999,"15min":1.2789999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.42499999999999999},"max":{"1min":1.3819999999999999,"5min":1.917,"15min":2.6720000000000002},"last":1.524}]},{"osd":4,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.429,"15min":1.423},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.016,"5min":2.1859999999999999,"15min":2.5499999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.3879999999999999,"15min":1.3700000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":3.016,"5min":3.016,"15min":3.016},"last":1.3320000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.4399999999999999,"15min":1.4379999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.968,"5min":1.9730000000000001,"15min":3.25},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.5569999999999999,"15min":1.5349999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.78300000000000003},"max":{"1min":1.901,"5min":2.27,"15min":2.9350000000000001},"last":1.6339999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.508,"15min":1.508},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":3.536},"last":1.204},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5289999999999999,"15min":1.5429999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":3.3929999999999998},"last":1.4750000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.5800000000000001,"15min":1.617},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":5.2199999999999998},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.579,"5min":1.665,"15min":1.6559999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":3.496},"last":1.5720000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54144,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317696,"statfs":{"total":96636764160,"available":96581320704,"internally_reserved":0,"allocated":765952,"data_stored":361550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.91600000000000004,"15min":0.92300000000000004},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.44900000000000001},"max":{"1min":1.169,"5min":1.7669999999999999,"15min":1.821},"last":1.161},{"interface":"front","average":{"1min":1.044,"5min":1.1060000000000001,"15min":1.1000000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.40000000000000002},"max":{"1min":1.4279999999999999,"5min":1.712,"15min":1.7370000000000001},"last":1.004}]},{"osd":2,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.999,"15min":0.997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.47599999999999998},"max":{"1min":1.615,"5min":1.774,"15min":1.774},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.1910000000000001,"15min":1.159},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.48299999999999998},"max":{"1min":1.5960000000000001,"5min":1.667,"15min":1.796},"last":1.3420000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.347,"15min":1.333},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.52400000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.601},{"interface":"front","average":{"1min":1.163,"5min":1.242,"15min":1.194},"min":{"1min":0.621,"5min":0.621,"15min":0.45800000000000002},"max":{"1min":1.75,"5min":1.855,"15min":2.2050000000000001},"last":1.2789999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.325,"15min":1.3080000000000001},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.69599999999999995},"max":{"1min":1.6200000000000001,"5min":1.8260000000000001,"15min":2.9820000000000002},"last":1.097},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4650000000000001,"15min":1.4430000000000001},"min":{"1min":1.028,"5min":1.028,"15min":0.59499999999999997},"max":{"1min":1.843,"5min":2.258,"15min":3.242},"last":1.8120000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.472,"15min":1.4339999999999999},"min":{"1min":1.0960000000000001,"5min":0.97399999999999998,"15min":0.75700000000000001},"max":{"1min":1.8360000000000001,"5min":2.032,"15min":2.3690000000000002},"last":1.393},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4810000000000001,"15min":1.4159999999999999},"min":{"1min":1.052,"5min":1.0389999999999999,"15min":0.63700000000000001},"max":{"1min":1.7869999999999999,"5min":2.1339999999999999,"15min":2.181},"last":1.4199999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.569,"15min":1.554},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.69299999999999995},"max":{"1min":2.1019999999999999,"5min":2.3359999999999999,"15min":2.476},"last":1.927},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5469999999999999},"min":{"1min":1.153,"5min":1.0580000000000001,"15min":0.73099999999999998},"max":{"1min":2.0169999999999999,"5min":2.238,"15min":2.6160000000000001},"last":1.46}]},{"osd":7,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.6779999999999999,"15min":1.625},"min":{"1min":1.224,"5min":1.1990000000000001,"15min":0.79300000000000004},"max":{"1min":2.1000000000000001,"5min":2.2639999999999998,"15min":2.5059999999999998},"last":1.524},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.6040000000000001,"15min":1.5620000000000001},"min":{"1min":1.0569999999999999,"5min":1.034,"15min":0.88800000000000001},"max":{"1min":1.9119999999999999,"5min":2.2170000000000001,"15min":2.3410000000000002},"last":1.552}]}]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":951882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.93799999999999994,"15min":0.95899999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.42199999999999999},"max":{"1min":1.093,"5min":1.498,"15min":2.2650000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1930000000000001,"15min":1.169},"min":{"1min":0.497,"5min":0.497,"15min":0.36399999999999999},"max":{"1min":1.605,"5min":1.859,"15min":2.6030000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2609999999999999,"15min":1.2450000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.8360000000000001,"5min":2.2970000000000002,"15min":3.653},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1899999999999999,"15min":1.2210000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.47399999999999998},"max":{"1min":2.012,"5min":2.012,"15min":3.1400000000000001},"last":1.175}]},{"osd":3,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.004,"15min":1.014},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.40100000000000002},"max":{"1min":1.4119999999999999,"5min":1.9410000000000001,"15min":2.391},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.2649999999999999,"15min":1.25},"min":{"1min":0.437,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.577,"5min":2.1600000000000001,"15min":2.79},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3129999999999999,"15min":1.304},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.61599999999999999},"max":{"1min":1.6539999999999999,"5min":1.806,"15min":2.7370000000000001},"last":1.054},{"interface":"front","average":{"1min":1.401,"5min":1.484,"15min":1.478},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.65900000000000003},"max":{"1min":1.6990000000000001,"5min":2.6259999999999999,"15min":3.6190000000000002},"last":1.085}]},{"osd":5,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.4330000000000001,"15min":1.45},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58099999999999996},"max":{"1min":1.8720000000000001,"5min":2.2400000000000002,"15min":3.2370000000000001},"last":1.375},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3600000000000001,"15min":1.389},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.67900000000000005},"max":{"1min":1.645,"5min":2.4889999999999999,"15min":2.819},"last":1.1319999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.5960000000000001,"15min":1.6040000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.75},"max":{"1min":1.966,"5min":2.4209999999999998,"15min":3.5779999999999998},"last":1.347},{"interface":"front","average":{"1min":1.429,"5min":1.504,"15min":1.528},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":0.82199999999999995},"max":{"1min":2.1440000000000001,"5min":2.1459999999999999,"15min":3.0979999999999999},"last":1.29}]},{"osd":7,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.603,"15min":1.607},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.67100000000000004},"max":{"1min":1.911,"5min":2.2650000000000001,"15min":3.3340000000000001},"last":1.472},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5469999999999999,"15min":1.615},"min":{"1min":0.877,"5min":0.877,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":2.976,"15min":6.5839999999999996},"last":1.202}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:30:20.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:30:21.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.024+0000 7fd9bceca640 1 Processor -- start 2023-10-16T14:30:21.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.024+0000 7fd9bceca640 1 -- start start 2023-10-16T14:30:21.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.025+0000 7fd9bceca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.025+0000 7fd9bceca640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9b8070f20 0x7fd9b80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9bceca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8160a90 con 0x7fd9b815c9f0 2023-10-16T14:30:21.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9bceca640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b80cdec0 con 0x7fd9b8070f20 2023-10-16T14:30:21.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9bceca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80cd900 con 0x7fd9b815dde0 2023-10-16T14:30:21.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9b7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9b8070f20 0x7fd9b80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9b77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9b815c9f0 0x7fd9b80c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39488/0 (socket says 172.21.15.29:39488) 2023-10-16T14:30:21.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9b77fe640 1 -- 172.21.15.29:0/2552210763 learned_addr learned my addr 172.21.15.29:0/2552210763 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:21.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.026+0000 7fd9b6ffd640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.027+0000 7fd9b7fff640 1 -- 172.21.15.29:0/2552210763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 msgr2=0x7fd9b80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:21.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.027+0000 7fd9b7fff640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.027+0000 7fd9b7fff640 1 -- 172.21.15.29:0/2552210763 >> v1:172.21.15.29:6789/0 conn(0x7fd9b815c9f0 legacy=0x7fd9b80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.027+0000 7fd9b7fff640 1 -- 172.21.15.29:0/2552210763 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b80cdbe0 con 0x7fd9b8070f20 2023-10-16T14:30:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.027+0000 7fd9b6ffd640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.027+0000 7fd9b67fc640 1 -- 172.21.15.29:0/2552210763 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523922127 0 0) 0x7fd9b8160a90 con 0x7fd9b815c9f0 2023-10-16T14:30:21.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.028+0000 7fd9b7fff640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8070f20 0x7fd9b80ac890 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3dea64b8d989f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.028+0000 7fd9b67fc640 1 -- 172.21.15.29:0/2552210763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b002cc60 con 0x7fd9b8070f20 2023-10-16T14:30:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.028+0000 7fd9b67fc640 1 -- 172.21.15.29:0/2552210763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b002cdc0 con 0x7fd9b8070f20 2023-10-16T14:30:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.028+0000 7fd9b67fc640 1 -- 172.21.15.29:0/2552210763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0034660 con 0x7fd9b8070f20 2023-10-16T14:30:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 -- 172.21.15.29:0/2552210763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8070f20 msgr2=0x7fd9b80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8070f20 0x7fd9b80ac890 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b002beb0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 -- 172.21.15.29:0/2552210763 shutdown_connections 2023-10-16T14:30:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b8070f20 0x7fd9b80ac890 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 --2- 172.21.15.29:0/2552210763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 -- 172.21.15.29:0/2552210763 >> 172.21.15.29:0/2552210763 conn(0x7fd9b80ce6a0 msgr2=0x7fd9b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 -- 172.21.15.29:0/2552210763 shutdown_connections 2023-10-16T14:30:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.029+0000 7fd9bceca640 1 -- 172.21.15.29:0/2552210763 wait complete. 2023-10-16T14:30:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.031+0000 7fd9bceca640 1 Processor -- start 2023-10-16T14:30:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.031+0000 7fd9bceca640 1 -- start start 2023-10-16T14:30:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9bceca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9b8070f20 0x7fd9b8153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9bceca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 0x7fd9b81f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9bceca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b8150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9bceca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b8155b00 con 0x7fd9b815c9f0 2023-10-16T14:30:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9bceca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b81f5ee0 con 0x7fd9b8070f20 2023-10-16T14:30:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9bceca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b81f6180 con 0x7fd9b815dde0 2023-10-16T14:30:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 0x7fd9b81f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 0x7fd9b81f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60874/0 (socket says 172.21.15.29:60874) 2023-10-16T14:30:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.032+0000 7fd9b6ffd640 1 -- 172.21.15.29:0/1806901367 learned_addr learned my addr 172.21.15.29:0/1806901367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b7fff640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b8150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b77fe640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9b8070f20 0x7fd9b8153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b6ffd640 1 -- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 msgr2=0x7fd9b8150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b6ffd640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b8150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b6ffd640 1 -- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9b8070f20 msgr2=0x7fd9b8153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b6ffd640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9b8070f20 0x7fd9b8153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b6ffd640 1 -- 172.21.15.29:0/1806901367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b0009a70 con 0x7fd9b815c9f0 2023-10-16T14:30:21.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b77fe640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9b8070f20 0x7fd9b8153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:21.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd9b6ffd640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 0x7fd9b81f59a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c7f0 tx=0x7fd9ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:21.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.033+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac00cea0 con 0x7fd9b815c9f0 2023-10-16T14:30:21.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.034+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ac007a30 con 0x7fd9b815c9f0 2023-10-16T14:30:21.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.034+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b81f6460 con 0x7fd9b815c9f0 2023-10-16T14:30:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.034+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac007dc0 con 0x7fd9b815c9f0 2023-10-16T14:30:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.034+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b8155d10 con 0x7fd9b815c9f0 2023-10-16T14:30:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.035+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b804e9b0 con 0x7fd9b815c9f0 2023-10-16T14:30:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.036+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd9ac0113f0 con 0x7fd9b815c9f0 2023-10-16T14:30:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.037+0000 7fd99ffff640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd9a406af90 0x7fd9a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.037+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9ac0899e0 con 0x7fd9b815c9f0 2023-10-16T14:30:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.038+0000 7fd9b77fe640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd9a406af90 0x7fd9a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.039+0000 7fd9b77fe640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd9a406af90 0x7fd9a406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8000f60 tx=0x7fd9a8009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.041+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd9ac054860 con 0x7fd9b815c9f0 2023-10-16T14:30:21.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.230+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 --> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9b80701e0 con 0x7fd9a406af90 2023-10-16T14:30:21.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.253+0000 7fd99ffff640 1 -- 172.21.15.29:0/1806901367 <== mgr.5113 v2:172.21.15.170:6832/3423886595 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286305 (secure 0 0 0) 0x7fd9a80013d0 con 0x7fd9a406af90 2023-10-16T14:30:21.260 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:30:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd9a406af90 msgr2=0x7fd9a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd9a406af90 0x7fd9a406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8000f60 tx=0x7fd9a8009040 comp rx=0 tx=0).stop 2023-10-16T14:30:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 msgr2=0x7fd9b81f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 0x7fd9b81f59a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c7f0 tx=0x7fd9ac00ccb0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 shutdown_connections 2023-10-16T14:30:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd9a406af90 0x7fd9a406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b815c9f0 0x7fd9b81f59a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9b815dde0 0x7fd9b8150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 --2- 172.21.15.29:0/1806901367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9b8070f20 0x7fd9b8153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 >> 172.21.15.29:0/1806901367 conn(0x7fd9b80ce6a0 msgr2=0x7fd9b80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 shutdown_connections 2023-10-16T14:30:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.258+0000 7fd9bceca640 1 -- 172.21.15.29:0/1806901367 wait complete. 2023-10-16T14:30:21.282 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T14:30:20.321155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326032,"kb_used_data":7176,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754648688,"statfs":{"total":773094113280,"available":772760256512,"internally_reserved":0,"allocated":7348224,"data_stored":4077396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":326356537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002191"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284146+0000","last_change":"2023-10-16T14:30:16.271259+0000","last_active":"2023-10-16T14:30:17.284146+0000","last_peered":"2023-10-16T14:30:17.284146+0000","last_clean":"2023-10-16T14:30:17.284146+0000","last_became_active":"2023-10-16T14:30:16.270883+0000","last_became_peered":"2023-10-16T14:30:16.270883+0000","last_unstale":"2023-10-16T14:30:17.284146+0000","last_undegraded":"2023-10-16T14:30:17.284146+0000","last_fullsized":"2023-10-16T14:30:17.284146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:49.392964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.294733+0000","last_change":"2023-10-16T14:30:16.293633+0000","last_active":"2023-10-16T14:30:16.294733+0000","last_peered":"2023-10-16T14:30:16.294733+0000","last_clean":"2023-10-16T14:30:16.294733+0000","last_became_active":"2023-10-16T14:30:16.292147+0000","last_became_peered":"2023-10-16T14:30:16.292147+0000","last_unstale":"2023-10-16T14:30:16.294733+0000","last_undegraded":"2023-10-16T14:30:16.294733+0000","last_fullsized":"2023-10-16T14:30:16.294733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:23.799327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283388+0000","last_change":"2023-10-16T14:30:16.274092+0000","last_active":"2023-10-16T14:30:17.283388+0000","last_peered":"2023-10-16T14:30:17.283388+0000","last_clean":"2023-10-16T14:30:17.283388+0000","last_became_active":"2023-10-16T14:30:16.273785+0000","last_became_peered":"2023-10-16T14:30:16.273785+0000","last_unstale":"2023-10-16T14:30:17.283388+0000","last_undegraded":"2023-10-16T14:30:17.283388+0000","last_fullsized":"2023-10-16T14:30:17.283388+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:48.494974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278390+0000","last_change":"2023-10-16T14:30:16.291888+0000","last_active":"2023-10-16T14:30:17.278390+0000","last_peered":"2023-10-16T14:30:17.278390+0000","last_clean":"2023-10-16T14:30:17.278390+0000","last_became_active":"2023-10-16T14:30:16.291596+0000","last_became_peered":"2023-10-16T14:30:16.291596+0000","last_unstale":"2023-10-16T14:30:17.278390+0000","last_undegraded":"2023-10-16T14:30:17.278390+0000","last_fullsized":"2023-10-16T14:30:17.278390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:25.908520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.291448+0000","last_change":"2023-10-16T14:30:16.283296+0000","last_active":"2023-10-16T14:30:16.291448+0000","last_peered":"2023-10-16T14:30:16.291448+0000","last_clean":"2023-10-16T14:30:16.291448+0000","last_became_active":"2023-10-16T14:30:16.282996+0000","last_became_peered":"2023-10-16T14:30:16.282996+0000","last_unstale":"2023-10-16T14:30:16.291448+0000","last_undegraded":"2023-10-16T14:30:16.291448+0000","last_fullsized":"2023-10-16T14:30:16.291448+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:19.654589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.281086+0000","last_change":"2023-10-16T14:30:16.274153+0000","last_active":"2023-10-16T14:30:16.281086+0000","last_peered":"2023-10-16T14:30:16.281086+0000","last_clean":"2023-10-16T14:30:16.281086+0000","last_became_active":"2023-10-16T14:30:16.270144+0000","last_became_peered":"2023-10-16T14:30:16.270144+0000","last_unstale":"2023-10-16T14:30:16.281086+0000","last_undegraded":"2023-10-16T14:30:16.281086+0000","last_fullsized":"2023-10-16T14:30:16.281086+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:55.567177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284159+0000","last_change":"2023-10-16T14:30:16.272721+0000","last_active":"2023-10-16T14:30:17.284159+0000","last_peered":"2023-10-16T14:30:17.284159+0000","last_clean":"2023-10-16T14:30:17.284159+0000","last_became_active":"2023-10-16T14:30:16.272369+0000","last_became_peered":"2023-10-16T14:30:16.272369+0000","last_unstale":"2023-10-16T14:30:17.284159+0000","last_undegraded":"2023-10-16T14:30:17.284159+0000","last_fullsized":"2023-10-16T14:30:17.284159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:24.868942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277721+0000","last_change":"2023-10-16T14:30:16.280070+0000","last_active":"2023-10-16T14:30:17.277721+0000","last_peered":"2023-10-16T14:30:17.277721+0000","last_clean":"2023-10-16T14:30:17.277721+0000","last_became_active":"2023-10-16T14:30:16.279563+0000","last_became_peered":"2023-10-16T14:30:16.279563+0000","last_unstale":"2023-10-16T14:30:17.277721+0000","last_undegraded":"2023-10-16T14:30:17.277721+0000","last_fullsized":"2023-10-16T14:30:17.277721+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:40:36.309295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278505+0000","last_change":"2023-10-16T14:30:16.264159+0000","last_active":"2023-10-16T14:30:17.278505+0000","last_peered":"2023-10-16T14:30:17.278505+0000","last_clean":"2023-10-16T14:30:17.278505+0000","last_became_active":"2023-10-16T14:30:16.263858+0000","last_became_peered":"2023-10-16T14:30:16.263858+0000","last_unstale":"2023-10-16T14:30:17.278505+0000","last_undegraded":"2023-10-16T14:30:17.278505+0000","last_fullsized":"2023-10-16T14:30:17.278505+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:26.202732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278739+0000","last_change":"2023-10-16T14:30:16.290583+0000","last_active":"2023-10-16T14:30:17.278739+0000","last_peered":"2023-10-16T14:30:17.278739+0000","last_clean":"2023-10-16T14:30:17.278739+0000","last_became_active":"2023-10-16T14:30:16.290289+0000","last_became_peered":"2023-10-16T14:30:16.290289+0000","last_unstale":"2023-10-16T14:30:17.278739+0000","last_undegraded":"2023-10-16T14:30:17.278739+0000","last_fullsized":"2023-10-16T14:30:17.278739+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:13.437481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278198+0000","last_change":"2023-10-16T14:30:16.274076+0000","last_active":"2023-10-16T14:30:17.278198+0000","last_peered":"2023-10-16T14:30:17.278198+0000","last_clean":"2023-10-16T14:30:17.278198+0000","last_became_active":"2023-10-16T14:30:16.273742+0000","last_became_peered":"2023-10-16T14:30:16.273742+0000","last_unstale":"2023-10-16T14:30:17.278198+0000","last_undegraded":"2023-10-16T14:30:17.278198+0000","last_fullsized":"2023-10-16T14:30:17.278198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:32.934311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290192+0000","last_change":"2023-10-16T14:30:16.285712+0000","last_active":"2023-10-16T14:30:16.290192+0000","last_peered":"2023-10-16T14:30:16.290192+0000","last_clean":"2023-10-16T14:30:16.290192+0000","last_became_active":"2023-10-16T14:30:16.285217+0000","last_became_peered":"2023-10-16T14:30:16.285217+0000","last_unstale":"2023-10-16T14:30:16.290192+0000","last_undegraded":"2023-10-16T14:30:16.290192+0000","last_fullsized":"2023-10-16T14:30:16.290192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:57.063012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.271272+0000","last_change":"2023-10-16T14:30:16.269813+0000","last_active":"2023-10-16T14:30:16.271272+0000","last_peered":"2023-10-16T14:30:16.271272+0000","last_clean":"2023-10-16T14:30:16.271272+0000","last_became_active":"2023-10-16T14:30:16.269490+0000","last_became_peered":"2023-10-16T14:30:16.269490+0000","last_unstale":"2023-10-16T14:30:16.271272+0000","last_undegraded":"2023-10-16T14:30:16.271272+0000","last_fullsized":"2023-10-16T14:30:16.271272+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:36.906191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278143+0000","last_change":"2023-10-16T14:30:16.274836+0000","last_active":"2023-10-16T14:30:17.278143+0000","last_peered":"2023-10-16T14:30:17.278143+0000","last_clean":"2023-10-16T14:30:17.278143+0000","last_became_active":"2023-10-16T14:30:16.274528+0000","last_became_peered":"2023-10-16T14:30:16.274528+0000","last_unstale":"2023-10-16T14:30:17.278143+0000","last_undegraded":"2023-10-16T14:30:17.278143+0000","last_fullsized":"2023-10-16T14:30:17.278143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:35.313814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.272391+0000","last_change":"2023-10-16T14:30:16.270552+0000","last_active":"2023-10-16T14:30:16.272391+0000","last_peered":"2023-10-16T14:30:16.272391+0000","last_clean":"2023-10-16T14:30:16.272391+0000","last_became_active":"2023-10-16T14:30:16.270267+0000","last_became_peered":"2023-10-16T14:30:16.270267+0000","last_unstale":"2023-10-16T14:30:16.272391+0000","last_undegraded":"2023-10-16T14:30:16.272391+0000","last_fullsized":"2023-10-16T14:30:16.272391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:21.384315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290120+0000","last_change":"2023-10-16T14:30:16.283627+0000","last_active":"2023-10-16T14:30:16.290120+0000","last_peered":"2023-10-16T14:30:16.290120+0000","last_clean":"2023-10-16T14:30:16.290120+0000","last_became_active":"2023-10-16T14:30:16.283351+0000","last_became_peered":"2023-10-16T14:30:16.283351+0000","last_unstale":"2023-10-16T14:30:16.290120+0000","last_undegraded":"2023-10-16T14:30:16.290120+0000","last_fullsized":"2023-10-16T14:30:16.290120+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:17.848156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284017+0000","last_change":"2023-10-16T14:30:16.286115+0000","last_active":"2023-10-16T14:30:17.284017+0000","last_peered":"2023-10-16T14:30:17.284017+0000","last_clean":"2023-10-16T14:30:17.284017+0000","last_became_active":"2023-10-16T14:30:16.283841+0000","last_became_peered":"2023-10-16T14:30:16.283841+0000","last_unstale":"2023-10-16T14:30:17.284017+0000","last_undegraded":"2023-10-16T14:30:17.284017+0000","last_fullsized":"2023-10-16T14:30:17.284017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:25:10.169897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283854+0000","last_change":"2023-10-16T14:30:16.285932+0000","last_active":"2023-10-16T14:30:17.283854+0000","last_peered":"2023-10-16T14:30:17.283854+0000","last_clean":"2023-10-16T14:30:17.283854+0000","last_became_active":"2023-10-16T14:30:16.285586+0000","last_became_peered":"2023-10-16T14:30:16.285586+0000","last_unstale":"2023-10-16T14:30:17.283854+0000","last_undegraded":"2023-10-16T14:30:17.283854+0000","last_fullsized":"2023-10-16T14:30:17.283854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:39.785736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.295452+0000","last_change":"2023-10-16T14:30:16.293490+0000","last_active":"2023-10-16T14:30:16.295452+0000","last_peered":"2023-10-16T14:30:16.295452+0000","last_clean":"2023-10-16T14:30:16.295452+0000","last_became_active":"2023-10-16T14:30:16.293143+0000","last_became_peered":"2023-10-16T14:30:16.293143+0000","last_unstale":"2023-10-16T14:30:16.295452+0000","last_undegraded":"2023-10-16T14:30:16.295452+0000","last_fullsized":"2023-10-16T14:30:16.295452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:40:36.309295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.291449+0000","last_change":"2023-10-16T14:30:16.284045+0000","last_active":"2023-10-16T14:30:16.291449+0000","last_peered":"2023-10-16T14:30:16.291449+0000","last_clean":"2023-10-16T14:30:16.291449+0000","last_became_active":"2023-10-16T14:30:16.283390+0000","last_became_peered":"2023-10-16T14:30:16.283390+0000","last_unstale":"2023-10-16T14:30:16.291449+0000","last_undegraded":"2023-10-16T14:30:16.291449+0000","last_fullsized":"2023-10-16T14:30:16.291449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.533166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296635+0000","last_change":"2023-10-16T14:30:16.282760+0000","last_active":"2023-10-16T14:30:16.296635+0000","last_peered":"2023-10-16T14:30:16.296635+0000","last_clean":"2023-10-16T14:30:16.296635+0000","last_became_active":"2023-10-16T14:30:16.282436+0000","last_became_peered":"2023-10-16T14:30:16.282436+0000","last_unstale":"2023-10-16T14:30:16.296635+0000","last_undegraded":"2023-10-16T14:30:16.296635+0000","last_fullsized":"2023-10-16T14:30:16.296635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:23.447810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278236+0000","last_change":"2023-10-16T14:30:16.283998+0000","last_active":"2023-10-16T14:30:17.278236+0000","last_peered":"2023-10-16T14:30:17.278236+0000","last_clean":"2023-10-16T14:30:17.278236+0000","last_became_active":"2023-10-16T14:30:16.283710+0000","last_became_peered":"2023-10-16T14:30:16.283710+0000","last_unstale":"2023-10-16T14:30:17.278236+0000","last_undegraded":"2023-10-16T14:30:17.278236+0000","last_fullsized":"2023-10-16T14:30:17.278236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:30.369966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.292824+0000","last_change":"2023-10-16T14:30:16.288810+0000","last_active":"2023-10-16T14:30:16.292824+0000","last_peered":"2023-10-16T14:30:16.292824+0000","last_clean":"2023-10-16T14:30:16.292824+0000","last_became_active":"2023-10-16T14:30:16.287063+0000","last_became_peered":"2023-10-16T14:30:16.287063+0000","last_unstale":"2023-10-16T14:30:16.292824+0000","last_undegraded":"2023-10-16T14:30:16.292824+0000","last_fullsized":"2023-10-16T14:30:16.292824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:23.531990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290140+0000","last_change":"2023-10-16T14:30:16.287720+0000","last_active":"2023-10-16T14:30:16.290140+0000","last_peered":"2023-10-16T14:30:16.290140+0000","last_clean":"2023-10-16T14:30:16.290140+0000","last_became_active":"2023-10-16T14:30:16.287090+0000","last_became_peered":"2023-10-16T14:30:16.287090+0000","last_unstale":"2023-10-16T14:30:16.290140+0000","last_undegraded":"2023-10-16T14:30:16.290140+0000","last_fullsized":"2023-10-16T14:30:16.290140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:25.258309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293491+0000","last_change":"2023-10-16T14:30:16.290669+0000","last_active":"2023-10-16T14:30:16.293491+0000","last_peered":"2023-10-16T14:30:16.293491+0000","last_clean":"2023-10-16T14:30:16.293491+0000","last_became_active":"2023-10-16T14:30:16.289468+0000","last_became_peered":"2023-10-16T14:30:16.289468+0000","last_unstale":"2023-10-16T14:30:16.293491+0000","last_undegraded":"2023-10-16T14:30:16.293491+0000","last_fullsized":"2023-10-16T14:30:16.293491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:13.661918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.272738+0000","last_change":"2023-10-16T14:30:16.270850+0000","last_active":"2023-10-16T14:30:16.272738+0000","last_peered":"2023-10-16T14:30:16.272738+0000","last_clean":"2023-10-16T14:30:16.272738+0000","last_became_active":"2023-10-16T14:30:16.270520+0000","last_became_peered":"2023-10-16T14:30:16.270520+0000","last_unstale":"2023-10-16T14:30:16.272738+0000","last_undegraded":"2023-10-16T14:30:16.272738+0000","last_fullsized":"2023-10-16T14:30:16.272738+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:59.451693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283479+0000","last_change":"2023-10-16T14:30:16.282285+0000","last_active":"2023-10-16T14:30:17.283479+0000","last_peered":"2023-10-16T14:30:17.283479+0000","last_clean":"2023-10-16T14:30:17.283479+0000","last_became_active":"2023-10-16T14:30:16.282002+0000","last_became_peered":"2023-10-16T14:30:16.282002+0000","last_unstale":"2023-10-16T14:30:17.283479+0000","last_undegraded":"2023-10-16T14:30:17.283479+0000","last_fullsized":"2023-10-16T14:30:17.283479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:13.167772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290174+0000","last_change":"2023-10-16T14:30:16.285532+0000","last_active":"2023-10-16T14:30:16.290174+0000","last_peered":"2023-10-16T14:30:16.290174+0000","last_clean":"2023-10-16T14:30:16.290174+0000","last_became_active":"2023-10-16T14:30:16.284922+0000","last_became_peered":"2023-10-16T14:30:16.284922+0000","last_unstale":"2023-10-16T14:30:16.290174+0000","last_undegraded":"2023-10-16T14:30:16.290174+0000","last_fullsized":"2023-10-16T14:30:16.290174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.873972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.283724+0000","last_change":"2023-10-16T14:30:16.280593+0000","last_active":"2023-10-16T14:30:16.283724+0000","last_peered":"2023-10-16T14:30:16.283724+0000","last_clean":"2023-10-16T14:30:16.283724+0000","last_became_active":"2023-10-16T14:30:16.280256+0000","last_became_peered":"2023-10-16T14:30:16.280256+0000","last_unstale":"2023-10-16T14:30:16.283724+0000","last_undegraded":"2023-10-16T14:30:16.283724+0000","last_fullsized":"2023-10-16T14:30:16.283724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:34.169815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282218+0000","last_change":"2023-10-16T14:30:16.277071+0000","last_active":"2023-10-16T14:30:17.282218+0000","last_peered":"2023-10-16T14:30:17.282218+0000","last_clean":"2023-10-16T14:30:17.282218+0000","last_became_active":"2023-10-16T14:30:16.275980+0000","last_became_peered":"2023-10-16T14:30:16.275980+0000","last_unstale":"2023-10-16T14:30:17.282218+0000","last_undegraded":"2023-10-16T14:30:17.282218+0000","last_fullsized":"2023-10-16T14:30:17.282218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:19.024489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296604+0000","last_change":"2023-10-16T14:30:16.285768+0000","last_active":"2023-10-16T14:30:16.296604+0000","last_peered":"2023-10-16T14:30:16.296604+0000","last_clean":"2023-10-16T14:30:16.296604+0000","last_became_active":"2023-10-16T14:30:16.285473+0000","last_became_peered":"2023-10-16T14:30:16.285473+0000","last_unstale":"2023-10-16T14:30:16.296604+0000","last_undegraded":"2023-10-16T14:30:16.296604+0000","last_fullsized":"2023-10-16T14:30:16.296604+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:39.425799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.275025+0000","last_change":"2023-10-16T14:30:16.273191+0000","last_active":"2023-10-16T14:30:16.275025+0000","last_peered":"2023-10-16T14:30:16.275025+0000","last_clean":"2023-10-16T14:30:16.275025+0000","last_became_active":"2023-10-16T14:30:16.272897+0000","last_became_peered":"2023-10-16T14:30:16.272897+0000","last_unstale":"2023-10-16T14:30:16.275025+0000","last_undegraded":"2023-10-16T14:30:16.275025+0000","last_fullsized":"2023-10-16T14:30:16.275025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:18.674198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.274268+0000","last_change":"2023-10-16T14:30:16.271326+0000","last_active":"2023-10-16T14:30:16.274268+0000","last_peered":"2023-10-16T14:30:16.274268+0000","last_clean":"2023-10-16T14:30:16.274268+0000","last_became_active":"2023-10-16T14:30:16.270612+0000","last_became_peered":"2023-10-16T14:30:16.270612+0000","last_unstale":"2023-10-16T14:30:16.274268+0000","last_undegraded":"2023-10-16T14:30:16.274268+0000","last_fullsized":"2023-10-16T14:30:16.274268+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:30:36.816276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"64'4","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298445+0000","last_change":"2023-10-16T14:30:15.279144+0000","last_active":"2023-10-16T14:30:16.298445+0000","last_peered":"2023-10-16T14:30:16.298445+0000","last_clean":"2023-10-16T14:30:16.298445+0000","last_became_active":"2023-10-16T14:30:15.278906+0000","last_became_peered":"2023-10-16T14:30:15.278906+0000","last_unstale":"2023-10-16T14:30:16.298445+0000","last_undegraded":"2023-10-16T14:30:16.298445+0000","last_fullsized":"2023-10-16T14:30:16.298445+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:04.400754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.294458+0000","last_change":"2023-10-16T14:30:16.291623+0000","last_active":"2023-10-16T14:30:16.294458+0000","last_peered":"2023-10-16T14:30:16.294458+0000","last_clean":"2023-10-16T14:30:16.294458+0000","last_became_active":"2023-10-16T14:30:16.291355+0000","last_became_peered":"2023-10-16T14:30:16.291355+0000","last_unstale":"2023-10-16T14:30:16.294458+0000","last_undegraded":"2023-10-16T14:30:16.294458+0000","last_fullsized":"2023-10-16T14:30:16.294458+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:27.097251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"64'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284454+0000","last_change":"2023-10-16T14:30:15.279658+0000","last_active":"2023-10-16T14:30:17.284454+0000","last_peered":"2023-10-16T14:30:17.284454+0000","last_clean":"2023-10-16T14:30:17.284454+0000","last_became_active":"2023-10-16T14:30:15.279354+0000","last_became_peered":"2023-10-16T14:30:15.279354+0000","last_unstale":"2023-10-16T14:30:17.284454+0000","last_undegraded":"2023-10-16T14:30:17.284454+0000","last_fullsized":"2023-10-16T14:30:17.284454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:14.206153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.290152+0000","last_change":"2023-10-16T14:30:16.280828+0000","last_active":"2023-10-16T14:30:16.290152+0000","last_peered":"2023-10-16T14:30:16.290152+0000","last_clean":"2023-10-16T14:30:16.290152+0000","last_became_active":"2023-10-16T14:30:16.280559+0000","last_became_peered":"2023-10-16T14:30:16.280559+0000","last_unstale":"2023-10-16T14:30:16.290152+0000","last_undegraded":"2023-10-16T14:30:16.290152+0000","last_fullsized":"2023-10-16T14:30:16.290152+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:27.556242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.286160+0000","last_change":"2023-10-16T14:30:15.264675+0000","last_active":"2023-10-16T14:30:16.286160+0000","last_peered":"2023-10-16T14:30:16.286160+0000","last_clean":"2023-10-16T14:30:16.286160+0000","last_became_active":"2023-10-16T14:30:15.264372+0000","last_became_peered":"2023-10-16T14:30:15.264372+0000","last_unstale":"2023-10-16T14:30:16.286160+0000","last_undegraded":"2023-10-16T14:30:16.286160+0000","last_fullsized":"2023-10-16T14:30:16.286160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:31.068794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293429+0000","last_change":"2023-10-16T14:30:16.291668+0000","last_active":"2023-10-16T14:30:16.293429+0000","last_peered":"2023-10-16T14:30:16.293429+0000","last_clean":"2023-10-16T14:30:16.293429+0000","last_became_active":"2023-10-16T14:30:16.290199+0000","last_became_peered":"2023-10-16T14:30:16.290199+0000","last_unstale":"2023-10-16T14:30:16.293429+0000","last_undegraded":"2023-10-16T14:30:16.293429+0000","last_fullsized":"2023-10-16T14:30:16.293429+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:03:19.060659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"64'2","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.283756+0000","last_change":"2023-10-16T14:30:15.279865+0000","last_active":"2023-10-16T14:30:17.283756+0000","last_peered":"2023-10-16T14:30:17.283756+0000","last_clean":"2023-10-16T14:30:17.283756+0000","last_became_active":"2023-10-16T14:30:15.277434+0000","last_became_peered":"2023-10-16T14:30:15.277434+0000","last_unstale":"2023-10-16T14:30:17.283756+0000","last_undegraded":"2023-10-16T14:30:17.283756+0000","last_fullsized":"2023-10-16T14:30:17.283756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:52.708275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282705+0000","last_change":"2023-10-16T14:30:16.284193+0000","last_active":"2023-10-16T14:30:17.282705+0000","last_peered":"2023-10-16T14:30:17.282705+0000","last_clean":"2023-10-16T14:30:17.282705+0000","last_became_active":"2023-10-16T14:30:16.283906+0000","last_became_peered":"2023-10-16T14:30:16.283906+0000","last_unstale":"2023-10-16T14:30:17.282705+0000","last_undegraded":"2023-10-16T14:30:17.282705+0000","last_fullsized":"2023-10-16T14:30:17.282705+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:34.548998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"64'1","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298679+0000","last_change":"2023-10-16T14:30:15.274153+0000","last_active":"2023-10-16T14:30:16.298679+0000","last_peered":"2023-10-16T14:30:16.298679+0000","last_clean":"2023-10-16T14:30:16.298679+0000","last_became_active":"2023-10-16T14:30:15.271972+0000","last_became_peered":"2023-10-16T14:30:15.271972+0000","last_unstale":"2023-10-16T14:30:16.298679+0000","last_undegraded":"2023-10-16T14:30:16.298679+0000","last_fullsized":"2023-10-16T14:30:16.298679+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:50.984821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278014+0000","last_change":"2023-10-16T14:30:16.278022+0000","last_active":"2023-10-16T14:30:17.278014+0000","last_peered":"2023-10-16T14:30:17.278014+0000","last_clean":"2023-10-16T14:30:17.278014+0000","last_became_active":"2023-10-16T14:30:16.277786+0000","last_became_peered":"2023-10-16T14:30:16.277786+0000","last_unstale":"2023-10-16T14:30:17.278014+0000","last_undegraded":"2023-10-16T14:30:17.278014+0000","last_fullsized":"2023-10-16T14:30:17.278014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:43.792287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"64'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282752+0000","last_change":"2023-10-16T14:30:15.277899+0000","last_active":"2023-10-16T14:30:17.282752+0000","last_peered":"2023-10-16T14:30:17.282752+0000","last_clean":"2023-10-16T14:30:17.282752+0000","last_became_active":"2023-10-16T14:30:15.277145+0000","last_became_peered":"2023-10-16T14:30:15.277145+0000","last_unstale":"2023-10-16T14:30:17.282752+0000","last_undegraded":"2023-10-16T14:30:17.282752+0000","last_fullsized":"2023-10-16T14:30:17.282752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:41.209934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277836+0000","last_change":"2023-10-16T14:30:16.279889+0000","last_active":"2023-10-16T14:30:17.277836+0000","last_peered":"2023-10-16T14:30:17.277836+0000","last_clean":"2023-10-16T14:30:17.277836+0000","last_became_active":"2023-10-16T14:30:16.279178+0000","last_became_peered":"2023-10-16T14:30:16.279178+0000","last_unstale":"2023-10-16T14:30:17.277836+0000","last_undegraded":"2023-10-16T14:30:17.277836+0000","last_fullsized":"2023-10-16T14:30:17.277836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:04.145884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.276913+0000","last_change":"2023-10-16T14:30:15.264794+0000","last_active":"2023-10-16T14:30:16.276913+0000","last_peered":"2023-10-16T14:30:16.276913+0000","last_clean":"2023-10-16T14:30:16.276913+0000","last_became_active":"2023-10-16T14:30:15.263964+0000","last_became_peered":"2023-10-16T14:30:15.263964+0000","last_unstale":"2023-10-16T14:30:16.276913+0000","last_undegraded":"2023-10-16T14:30:16.276913+0000","last_fullsized":"2023-10-16T14:30:16.276913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:39.586641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.292845+0000","last_change":"2023-10-16T14:30:16.288860+0000","last_active":"2023-10-16T14:30:16.292845+0000","last_peered":"2023-10-16T14:30:16.292845+0000","last_clean":"2023-10-16T14:30:16.292845+0000","last_became_active":"2023-10-16T14:30:16.288413+0000","last_became_peered":"2023-10-16T14:30:16.288413+0000","last_unstale":"2023-10-16T14:30:16.292845+0000","last_undegraded":"2023-10-16T14:30:16.292845+0000","last_fullsized":"2023-10-16T14:30:16.292845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:23:48.421046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277855+0000","last_change":"2023-10-16T14:30:15.268275+0000","last_active":"2023-10-16T14:30:17.277855+0000","last_peered":"2023-10-16T14:30:17.277855+0000","last_clean":"2023-10-16T14:30:17.277855+0000","last_became_active":"2023-10-16T14:30:15.268007+0000","last_became_peered":"2023-10-16T14:30:15.268007+0000","last_unstale":"2023-10-16T14:30:17.277855+0000","last_undegraded":"2023-10-16T14:30:17.277855+0000","last_fullsized":"2023-10-16T14:30:17.277855+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:08.427082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284185+0000","last_change":"2023-10-16T14:30:16.285494+0000","last_active":"2023-10-16T14:30:17.284185+0000","last_peered":"2023-10-16T14:30:17.284185+0000","last_clean":"2023-10-16T14:30:17.284185+0000","last_became_active":"2023-10-16T14:30:16.285101+0000","last_became_peered":"2023-10-16T14:30:16.285101+0000","last_unstale":"2023-10-16T14:30:17.284185+0000","last_undegraded":"2023-10-16T14:30:17.284185+0000","last_fullsized":"2023-10-16T14:30:17.284185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:34.450415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284885+0000","last_change":"2023-10-16T14:30:15.270246+0000","last_active":"2023-10-16T14:30:17.284885+0000","last_peered":"2023-10-16T14:30:17.284885+0000","last_clean":"2023-10-16T14:30:17.284885+0000","last_became_active":"2023-10-16T14:30:15.269830+0000","last_became_peered":"2023-10-16T14:30:15.269830+0000","last_unstale":"2023-10-16T14:30:17.284885+0000","last_undegraded":"2023-10-16T14:30:17.284885+0000","last_fullsized":"2023-10-16T14:30:17.284885+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:02.116331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284502+0000","last_change":"2023-10-16T14:30:16.286018+0000","last_active":"2023-10-16T14:30:17.284502+0000","last_peered":"2023-10-16T14:30:17.284502+0000","last_clean":"2023-10-16T14:30:17.284502+0000","last_became_active":"2023-10-16T14:30:16.285738+0000","last_became_peered":"2023-10-16T14:30:16.285738+0000","last_unstale":"2023-10-16T14:30:17.284502+0000","last_undegraded":"2023-10-16T14:30:17.284502+0000","last_fullsized":"2023-10-16T14:30:17.284502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:08.405329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278895+0000","last_change":"2023-10-16T14:30:15.277563+0000","last_active":"2023-10-16T14:30:17.278895+0000","last_peered":"2023-10-16T14:30:17.278895+0000","last_clean":"2023-10-16T14:30:17.278895+0000","last_became_active":"2023-10-16T14:30:15.277231+0000","last_became_peered":"2023-10-16T14:30:15.277231+0000","last_unstale":"2023-10-16T14:30:17.278895+0000","last_undegraded":"2023-10-16T14:30:17.278895+0000","last_fullsized":"2023-10-16T14:30:17.278895+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:21.987501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293453+0000","last_change":"2023-10-16T14:30:16.279690+0000","last_active":"2023-10-16T14:30:16.293453+0000","last_peered":"2023-10-16T14:30:16.293453+0000","last_clean":"2023-10-16T14:30:16.293453+0000","last_became_active":"2023-10-16T14:30:16.279394+0000","last_became_peered":"2023-10-16T14:30:16.279394+0000","last_unstale":"2023-10-16T14:30:16.293453+0000","last_undegraded":"2023-10-16T14:30:16.293453+0000","last_fullsized":"2023-10-16T14:30:16.293453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:57.063012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"64'3","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285326+0000","last_change":"2023-10-16T14:30:15.278383+0000","last_active":"2023-10-16T14:30:17.285326+0000","last_peered":"2023-10-16T14:30:17.285326+0000","last_clean":"2023-10-16T14:30:17.285326+0000","last_became_active":"2023-10-16T14:30:15.278081+0000","last_became_peered":"2023-10-16T14:30:15.278081+0000","last_unstale":"2023-10-16T14:30:17.285326+0000","last_undegraded":"2023-10-16T14:30:17.285326+0000","last_fullsized":"2023-10-16T14:30:17.285326+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:40.004801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284713+0000","last_change":"2023-10-16T14:30:16.274192+0000","last_active":"2023-10-16T14:30:17.284713+0000","last_peered":"2023-10-16T14:30:17.284713+0000","last_clean":"2023-10-16T14:30:17.284713+0000","last_became_active":"2023-10-16T14:30:16.273902+0000","last_became_peered":"2023-10-16T14:30:16.273902+0000","last_unstale":"2023-10-16T14:30:17.284713+0000","last_undegraded":"2023-10-16T14:30:17.284713+0000","last_fullsized":"2023-10-16T14:30:17.284713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:09.477466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"64'6","reported_seq":17,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.300542+0000","last_change":"2023-10-16T14:30:15.269321+0000","last_active":"2023-10-16T14:30:16.300542+0000","last_peered":"2023-10-16T14:30:16.300542+0000","last_clean":"2023-10-16T14:30:16.300542+0000","last_became_active":"2023-10-16T14:30:15.268847+0000","last_became_peered":"2023-10-16T14:30:15.268847+0000","last_unstale":"2023-10-16T14:30:16.300542+0000","last_undegraded":"2023-10-16T14:30:16.300542+0000","last_fullsized":"2023-10-16T14:30:16.300542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:49.547519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296211+0000","last_change":"2023-10-16T14:30:16.295293+0000","last_active":"2023-10-16T14:30:16.296211+0000","last_peered":"2023-10-16T14:30:16.296211+0000","last_clean":"2023-10-16T14:30:16.296211+0000","last_became_active":"2023-10-16T14:30:16.293992+0000","last_became_peered":"2023-10-16T14:30:16.293992+0000","last_unstale":"2023-10-16T14:30:16.296211+0000","last_undegraded":"2023-10-16T14:30:16.296211+0000","last_fullsized":"2023-10-16T14:30:16.296211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:50.521569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.299034+0000","last_change":"2023-10-16T14:30:15.274587+0000","last_active":"2023-10-16T14:30:16.299034+0000","last_peered":"2023-10-16T14:30:16.299034+0000","last_clean":"2023-10-16T14:30:16.299034+0000","last_became_active":"2023-10-16T14:30:15.274275+0000","last_became_peered":"2023-10-16T14:30:15.274275+0000","last_unstale":"2023-10-16T14:30:16.299034+0000","last_undegraded":"2023-10-16T14:30:16.299034+0000","last_fullsized":"2023-10-16T14:30:16.299034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:16.420170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278740+0000","last_change":"2023-10-16T14:30:16.281020+0000","last_active":"2023-10-16T14:30:17.278740+0000","last_peered":"2023-10-16T14:30:17.278740+0000","last_clean":"2023-10-16T14:30:17.278740+0000","last_became_active":"2023-10-16T14:30:16.280854+0000","last_became_peered":"2023-10-16T14:30:16.280854+0000","last_unstale":"2023-10-16T14:30:17.278740+0000","last_undegraded":"2023-10-16T14:30:17.278740+0000","last_fullsized":"2023-10-16T14:30:17.278740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:24.053085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284689+0000","last_change":"2023-10-16T14:30:15.271411+0000","last_active":"2023-10-16T14:30:17.284689+0000","last_peered":"2023-10-16T14:30:17.284689+0000","last_clean":"2023-10-16T14:30:17.284689+0000","last_became_active":"2023-10-16T14:30:15.271128+0000","last_became_peered":"2023-10-16T14:30:15.271128+0000","last_unstale":"2023-10-16T14:30:17.284689+0000","last_undegraded":"2023-10-16T14:30:17.284689+0000","last_fullsized":"2023-10-16T14:30:17.284689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:54.265571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.269326+0000","last_change":"2023-10-16T14:30:16.268088+0000","last_active":"2023-10-16T14:30:16.269326+0000","last_peered":"2023-10-16T14:30:16.269326+0000","last_clean":"2023-10-16T14:30:16.269326+0000","last_became_active":"2023-10-16T14:30:16.267814+0000","last_became_peered":"2023-10-16T14:30:16.267814+0000","last_unstale":"2023-10-16T14:30:16.269326+0000","last_undegraded":"2023-10-16T14:30:16.269326+0000","last_fullsized":"2023-10-16T14:30:16.269326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:46.873972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.255383+0000","last_change":"2023-10-16T14:30:15.274467+0000","last_active":"2023-10-16T14:30:16.255383+0000","last_peered":"2023-10-16T14:30:16.255383+0000","last_clean":"2023-10-16T14:30:16.255383+0000","last_became_active":"2023-10-16T14:30:15.273326+0000","last_became_peered":"2023-10-16T14:30:15.273326+0000","last_unstale":"2023-10-16T14:30:16.255383+0000","last_undegraded":"2023-10-16T14:30:16.255383+0000","last_fullsized":"2023-10-16T14:30:16.255383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:52.708275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284215+0000","last_change":"2023-10-16T14:30:16.282230+0000","last_active":"2023-10-16T14:30:17.284215+0000","last_peered":"2023-10-16T14:30:17.284215+0000","last_clean":"2023-10-16T14:30:17.284215+0000","last_became_active":"2023-10-16T14:30:16.281969+0000","last_became_peered":"2023-10-16T14:30:16.281969+0000","last_unstale":"2023-10-16T14:30:17.284215+0000","last_undegraded":"2023-10-16T14:30:17.284215+0000","last_fullsized":"2023-10-16T14:30:17.284215+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:40.686426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"64'8","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277983+0000","last_change":"2023-10-16T14:30:15.266815+0000","last_active":"2023-10-16T14:30:17.277983+0000","last_peered":"2023-10-16T14:30:17.277983+0000","last_clean":"2023-10-16T14:30:17.277983+0000","last_became_active":"2023-10-16T14:30:15.265618+0000","last_became_peered":"2023-10-16T14:30:15.265618+0000","last_unstale":"2023-10-16T14:30:17.277983+0000","last_undegraded":"2023-10-16T14:30:17.277983+0000","last_fullsized":"2023-10-16T14:30:17.277983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:26.080451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.285210+0000","last_change":"2023-10-16T14:30:16.284141+0000","last_active":"2023-10-16T14:30:16.285210+0000","last_peered":"2023-10-16T14:30:16.285210+0000","last_clean":"2023-10-16T14:30:16.285210+0000","last_became_active":"2023-10-16T14:30:16.282984+0000","last_became_peered":"2023-10-16T14:30:16.282984+0000","last_unstale":"2023-10-16T14:30:16.285210+0000","last_undegraded":"2023-10-16T14:30:16.285210+0000","last_fullsized":"2023-10-16T14:30:16.285210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:45.727988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"64'1","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.297560+0000","last_change":"2023-10-16T14:30:15.267941+0000","last_active":"2023-10-16T14:30:16.297560+0000","last_peered":"2023-10-16T14:30:16.297560+0000","last_clean":"2023-10-16T14:30:16.297560+0000","last_became_active":"2023-10-16T14:30:15.263703+0000","last_became_peered":"2023-10-16T14:30:15.263703+0000","last_unstale":"2023-10-16T14:30:16.297560+0000","last_undegraded":"2023-10-16T14:30:16.297560+0000","last_fullsized":"2023-10-16T14:30:16.297560+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:26.909324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278740+0000","last_change":"2023-10-16T14:30:16.279668+0000","last_active":"2023-10-16T14:30:17.278740+0000","last_peered":"2023-10-16T14:30:17.278740+0000","last_clean":"2023-10-16T14:30:17.278740+0000","last_became_active":"2023-10-16T14:30:16.279347+0000","last_became_peered":"2023-10-16T14:30:16.279347+0000","last_unstale":"2023-10-16T14:30:17.278740+0000","last_undegraded":"2023-10-16T14:30:17.278740+0000","last_fullsized":"2023-10-16T14:30:17.278740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:42.370647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"64'2","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.284505+0000","last_change":"2023-10-16T14:30:15.272893+0000","last_active":"2023-10-16T14:30:17.284505+0000","last_peered":"2023-10-16T14:30:17.284505+0000","last_clean":"2023-10-16T14:30:17.284505+0000","last_became_active":"2023-10-16T14:30:15.272311+0000","last_became_peered":"2023-10-16T14:30:15.272311+0000","last_unstale":"2023-10-16T14:30:17.284505+0000","last_undegraded":"2023-10-16T14:30:17.284505+0000","last_fullsized":"2023-10-16T14:30:17.284505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:10.133346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298719+0000","last_change":"2023-10-16T14:30:16.297811+0000","last_active":"2023-10-16T14:30:16.298719+0000","last_peered":"2023-10-16T14:30:16.298719+0000","last_clean":"2023-10-16T14:30:16.298719+0000","last_became_active":"2023-10-16T14:30:16.297480+0000","last_became_peered":"2023-10-16T14:30:16.297480+0000","last_unstale":"2023-10-16T14:30:16.298719+0000","last_undegraded":"2023-10-16T14:30:16.298719+0000","last_fullsized":"2023-10-16T14:30:16.298719+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:04.145884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"64'1","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.284402+0000","last_change":"2023-10-16T14:30:15.270596+0000","last_active":"2023-10-16T14:30:16.284402+0000","last_peered":"2023-10-16T14:30:16.284402+0000","last_clean":"2023-10-16T14:30:16.284402+0000","last_became_active":"2023-10-16T14:30:15.270432+0000","last_became_peered":"2023-10-16T14:30:15.270432+0000","last_unstale":"2023-10-16T14:30:16.284402+0000","last_undegraded":"2023-10-16T14:30:16.284402+0000","last_fullsized":"2023-10-16T14:30:16.284402+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:10.411620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.289600+0000","last_change":"2023-10-16T14:30:16.284093+0000","last_active":"2023-10-16T14:30:16.289600+0000","last_peered":"2023-10-16T14:30:16.289600+0000","last_clean":"2023-10-16T14:30:16.289600+0000","last_became_active":"2023-10-16T14:30:16.283778+0000","last_became_peered":"2023-10-16T14:30:16.283778+0000","last_unstale":"2023-10-16T14:30:16.289600+0000","last_undegraded":"2023-10-16T14:30:16.289600+0000","last_fullsized":"2023-10-16T14:30:16.289600+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:09.443883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.255099+0000","last_change":"2023-10-16T14:30:15.275930+0000","last_active":"2023-10-16T14:30:16.255099+0000","last_peered":"2023-10-16T14:30:16.255099+0000","last_clean":"2023-10-16T14:30:16.255099+0000","last_became_active":"2023-10-16T14:30:15.275653+0000","last_became_peered":"2023-10-16T14:30:15.275653+0000","last_unstale":"2023-10-16T14:30:16.255099+0000","last_undegraded":"2023-10-16T14:30:16.255099+0000","last_fullsized":"2023-10-16T14:30:16.255099+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:39.586641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293417+0000","last_change":"2023-10-16T14:30:16.279050+0000","last_active":"2023-10-16T14:30:16.293417+0000","last_peered":"2023-10-16T14:30:16.293417+0000","last_clean":"2023-10-16T14:30:16.293417+0000","last_became_active":"2023-10-16T14:30:16.278713+0000","last_became_peered":"2023-10-16T14:30:16.278713+0000","last_unstale":"2023-10-16T14:30:16.293417+0000","last_undegraded":"2023-10-16T14:30:16.293417+0000","last_fullsized":"2023-10-16T14:30:16.293417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:16.561457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.298757+0000","last_change":"2023-10-16T14:30:15.272414+0000","last_active":"2023-10-16T14:30:16.298757+0000","last_peered":"2023-10-16T14:30:16.298757+0000","last_clean":"2023-10-16T14:30:16.298757+0000","last_became_active":"2023-10-16T14:30:15.271597+0000","last_became_peered":"2023-10-16T14:30:15.271597+0000","last_unstale":"2023-10-16T14:30:16.298757+0000","last_undegraded":"2023-10-16T14:30:16.298757+0000","last_fullsized":"2023-10-16T14:30:16.298757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:26:43.026194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.288364+0000","last_change":"2023-10-16T14:30:16.286327+0000","last_active":"2023-10-16T14:30:16.288364+0000","last_peered":"2023-10-16T14:30:16.288364+0000","last_clean":"2023-10-16T14:30:16.288364+0000","last_became_active":"2023-10-16T14:30:16.286068+0000","last_became_peered":"2023-10-16T14:30:16.286068+0000","last_unstale":"2023-10-16T14:30:16.288364+0000","last_undegraded":"2023-10-16T14:30:16.288364+0000","last_fullsized":"2023-10-16T14:30:16.288364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:53.776633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.273244+0000","last_change":"2023-10-16T14:30:15.275710+0000","last_active":"2023-10-16T14:30:16.273244+0000","last_peered":"2023-10-16T14:30:16.273244+0000","last_clean":"2023-10-16T14:30:16.273244+0000","last_became_active":"2023-10-16T14:30:15.275421+0000","last_became_peered":"2023-10-16T14:30:15.275421+0000","last_unstale":"2023-10-16T14:30:16.273244+0000","last_undegraded":"2023-10-16T14:30:16.273244+0000","last_fullsized":"2023-10-16T14:30:16.273244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:20.367515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.282754+0000","last_change":"2023-10-16T14:30:16.276370+0000","last_active":"2023-10-16T14:30:16.282754+0000","last_peered":"2023-10-16T14:30:16.282754+0000","last_clean":"2023-10-16T14:30:16.282754+0000","last_became_active":"2023-10-16T14:30:16.275612+0000","last_became_peered":"2023-10-16T14:30:16.275612+0000","last_unstale":"2023-10-16T14:30:16.282754+0000","last_undegraded":"2023-10-16T14:30:16.282754+0000","last_fullsized":"2023-10-16T14:30:16.282754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:36.513930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277804+0000","last_change":"2023-10-16T14:30:15.276675+0000","last_active":"2023-10-16T14:30:17.277804+0000","last_peered":"2023-10-16T14:30:17.277804+0000","last_clean":"2023-10-16T14:30:17.277804+0000","last_became_active":"2023-10-16T14:30:15.276361+0000","last_became_peered":"2023-10-16T14:30:15.276361+0000","last_unstale":"2023-10-16T14:30:17.277804+0000","last_undegraded":"2023-10-16T14:30:17.277804+0000","last_fullsized":"2023-10-16T14:30:17.277804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:47.590806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277803+0000","last_change":"2023-10-16T14:30:16.263916+0000","last_active":"2023-10-16T14:30:17.277803+0000","last_peered":"2023-10-16T14:30:17.277803+0000","last_clean":"2023-10-16T14:30:17.277803+0000","last_became_active":"2023-10-16T14:30:16.262297+0000","last_became_peered":"2023-10-16T14:30:16.262297+0000","last_unstale":"2023-10-16T14:30:17.277803+0000","last_undegraded":"2023-10-16T14:30:17.277803+0000","last_fullsized":"2023-10-16T14:30:17.277803+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:13.661918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282867+0000","last_change":"2023-10-16T14:30:15.274645+0000","last_active":"2023-10-16T14:30:17.282867+0000","last_peered":"2023-10-16T14:30:17.282867+0000","last_clean":"2023-10-16T14:30:17.282867+0000","last_became_active":"2023-10-16T14:30:15.274331+0000","last_became_peered":"2023-10-16T14:30:15.274331+0000","last_unstale":"2023-10-16T14:30:17.282867+0000","last_undegraded":"2023-10-16T14:30:17.282867+0000","last_fullsized":"2023-10-16T14:30:17.282867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:54.550376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296218+0000","last_change":"2023-10-16T14:30:16.295329+0000","last_active":"2023-10-16T14:30:16.296218+0000","last_peered":"2023-10-16T14:30:16.296218+0000","last_clean":"2023-10-16T14:30:16.296218+0000","last_became_active":"2023-10-16T14:30:16.294969+0000","last_became_peered":"2023-10-16T14:30:16.294969+0000","last_unstale":"2023-10-16T14:30:16.296218+0000","last_undegraded":"2023-10-16T14:30:16.296218+0000","last_fullsized":"2023-10-16T14:30:16.296218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:44:16.883415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.256236+0000","last_change":"2023-10-16T14:30:15.275135+0000","last_active":"2023-10-16T14:30:16.256236+0000","last_peered":"2023-10-16T14:30:16.256236+0000","last_clean":"2023-10-16T14:30:16.256236+0000","last_became_active":"2023-10-16T14:30:15.274869+0000","last_became_peered":"2023-10-16T14:30:15.274869+0000","last_unstale":"2023-10-16T14:30:16.256236+0000","last_undegraded":"2023-10-16T14:30:16.256236+0000","last_fullsized":"2023-10-16T14:30:16.256236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:21.798778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278390+0000","last_change":"2023-10-16T14:30:16.275774+0000","last_active":"2023-10-16T14:30:17.278390+0000","last_peered":"2023-10-16T14:30:17.278390+0000","last_clean":"2023-10-16T14:30:17.278390+0000","last_became_active":"2023-10-16T14:30:16.275503+0000","last_became_peered":"2023-10-16T14:30:16.275503+0000","last_unstale":"2023-10-16T14:30:17.278390+0000","last_undegraded":"2023-10-16T14:30:17.278390+0000","last_fullsized":"2023-10-16T14:30:17.278390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:25.396087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285617+0000","last_change":"2023-10-16T14:30:15.278789+0000","last_active":"2023-10-16T14:30:17.285617+0000","last_peered":"2023-10-16T14:30:17.285617+0000","last_clean":"2023-10-16T14:30:17.285617+0000","last_became_active":"2023-10-16T14:30:15.278360+0000","last_became_peered":"2023-10-16T14:30:15.278360+0000","last_unstale":"2023-10-16T14:30:17.285617+0000","last_undegraded":"2023-10-16T14:30:17.285617+0000","last_fullsized":"2023-10-16T14:30:17.285617+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:26:35.162457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285061+0000","last_change":"2023-10-16T14:30:16.274257+0000","last_active":"2023-10-16T14:30:17.285061+0000","last_peered":"2023-10-16T14:30:17.285061+0000","last_clean":"2023-10-16T14:30:17.285061+0000","last_became_active":"2023-10-16T14:30:16.273967+0000","last_became_peered":"2023-10-16T14:30:16.273967+0000","last_unstale":"2023-10-16T14:30:17.285061+0000","last_undegraded":"2023-10-16T14:30:17.285061+0000","last_fullsized":"2023-10-16T14:30:17.285061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:28.253135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.261797+0000","last_change":"2023-10-16T14:30:15.264857+0000","last_active":"2023-10-16T14:30:16.261797+0000","last_peered":"2023-10-16T14:30:16.261797+0000","last_clean":"2023-10-16T14:30:16.261797+0000","last_became_active":"2023-10-16T14:30:15.264568+0000","last_became_peered":"2023-10-16T14:30:15.264568+0000","last_unstale":"2023-10-16T14:30:16.261797+0000","last_undegraded":"2023-10-16T14:30:16.261797+0000","last_fullsized":"2023-10-16T14:30:16.261797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:56.567240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.281065+0000","last_change":"2023-10-16T14:30:16.273426+0000","last_active":"2023-10-16T14:30:16.281065+0000","last_peered":"2023-10-16T14:30:16.281065+0000","last_clean":"2023-10-16T14:30:16.281065+0000","last_became_active":"2023-10-16T14:30:16.270683+0000","last_became_peered":"2023-10-16T14:30:16.270683+0000","last_unstale":"2023-10-16T14:30:16.281065+0000","last_undegraded":"2023-10-16T14:30:16.281065+0000","last_fullsized":"2023-10-16T14:30:16.281065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:55.282371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.285461+0000","last_change":"2023-10-16T14:30:15.276481+0000","last_active":"2023-10-16T14:30:17.285461+0000","last_peered":"2023-10-16T14:30:17.285461+0000","last_clean":"2023-10-16T14:30:17.285461+0000","last_became_active":"2023-10-16T14:30:15.276134+0000","last_became_peered":"2023-10-16T14:30:15.276134+0000","last_unstale":"2023-10-16T14:30:17.285461+0000","last_undegraded":"2023-10-16T14:30:17.285461+0000","last_fullsized":"2023-10-16T14:30:17.285461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:26:43.026194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.296613+0000","last_change":"2023-10-16T14:30:16.279024+0000","last_active":"2023-10-16T14:30:16.296613+0000","last_peered":"2023-10-16T14:30:16.296613+0000","last_clean":"2023-10-16T14:30:16.296613+0000","last_became_active":"2023-10-16T14:30:16.278721+0000","last_became_peered":"2023-10-16T14:30:16.278721+0000","last_unstale":"2023-10-16T14:30:16.296613+0000","last_undegraded":"2023-10-16T14:30:16.296613+0000","last_fullsized":"2023-10-16T14:30:16.296613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:11.428420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282925+0000","last_change":"2023-10-16T14:30:15.275836+0000","last_active":"2023-10-16T14:30:17.282925+0000","last_peered":"2023-10-16T14:30:17.282925+0000","last_clean":"2023-10-16T14:30:17.282925+0000","last_became_active":"2023-10-16T14:30:15.275012+0000","last_became_peered":"2023-10-16T14:30:15.275012+0000","last_unstale":"2023-10-16T14:30:17.282925+0000","last_undegraded":"2023-10-16T14:30:17.282925+0000","last_fullsized":"2023-10-16T14:30:17.282925+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:34.029244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.293161+0000","last_change":"2023-10-16T14:30:16.292065+0000","last_active":"2023-10-16T14:30:16.293161+0000","last_peered":"2023-10-16T14:30:16.293161+0000","last_clean":"2023-10-16T14:30:16.293161+0000","last_became_active":"2023-10-16T14:30:16.291604+0000","last_became_peered":"2023-10-16T14:30:16.291604+0000","last_unstale":"2023-10-16T14:30:16.293161+0000","last_undegraded":"2023-10-16T14:30:16.293161+0000","last_fullsized":"2023-10-16T14:30:16.293161+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:23.447810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.260835+0000","last_change":"2023-10-16T14:30:15.279688+0000","last_active":"2023-10-16T14:30:16.260835+0000","last_peered":"2023-10-16T14:30:16.260835+0000","last_clean":"2023-10-16T14:30:16.260835+0000","last_became_active":"2023-10-16T14:30:15.279394+0000","last_became_peered":"2023-10-16T14:30:15.279394+0000","last_unstale":"2023-10-16T14:30:16.260835+0000","last_undegraded":"2023-10-16T14:30:16.260835+0000","last_fullsized":"2023-10-16T14:30:16.260835+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:00.533698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.282208+0000","last_change":"2023-10-16T14:30:16.268774+0000","last_active":"2023-10-16T14:30:17.282208+0000","last_peered":"2023-10-16T14:30:17.282208+0000","last_clean":"2023-10-16T14:30:17.282208+0000","last_became_active":"2023-10-16T14:30:16.268460+0000","last_became_peered":"2023-10-16T14:30:16.268460+0000","last_unstale":"2023-10-16T14:30:17.282208+0000","last_undegraded":"2023-10-16T14:30:17.282208+0000","last_fullsized":"2023-10-16T14:30:17.282208+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:37.791494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.277976+0000","last_change":"2023-10-16T14:30:15.266274+0000","last_active":"2023-10-16T14:30:17.277976+0000","last_peered":"2023-10-16T14:30:17.277976+0000","last_clean":"2023-10-16T14:30:17.277976+0000","last_became_active":"2023-10-16T14:30:15.265975+0000","last_became_peered":"2023-10-16T14:30:15.265975+0000","last_unstale":"2023-10-16T14:30:17.277976+0000","last_undegraded":"2023-10-16T14:30:17.277976+0000","last_fullsized":"2023-10-16T14:30:17.277976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:44:15.866615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-16T14:30:17.278926+0000","last_change":"2023-10-16T14:30:16.282250+0000","last_active":"2023-10-16T14:30:17.278926+0000","last_peered":"2023-10-16T14:30:17.278926+0000","last_clean":"2023-10-16T14:30:17.278926+0000","last_became_active":"2023-10-16T14:30:16.281994+0000","last_became_peered":"2023-10-16T14:30:16.281994+0000","last_unstale":"2023-10-16T14:30:17.278926+0000","last_undegraded":"2023-10-16T14:30:17.278926+0000","last_fullsized":"2023-10-16T14:30:17.278926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:15.231268+0000","last_clean_scrub_stamp":"2023-10-16T14:30:15.231268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:38.169051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"64'2","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.297339+0000","last_change":"2023-10-16T14:30:15.269396+0000","last_active":"2023-10-16T14:30:16.297339+0000","last_peered":"2023-10-16T14:30:16.297339+0000","last_clean":"2023-10-16T14:30:16.297339+0000","last_became_active":"2023-10-16T14:30:15.267714+0000","last_became_peered":"2023-10-16T14:30:15.267714+0000","last_unstale":"2023-10-16T14:30:16.297339+0000","last_undegraded":"2023-10-16T14:30:16.297339+0000","last_fullsized":"2023-10-16T14:30:16.297339+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:30:14.214467+0000","last_clean_scrub_stamp":"2023-10-16T14:30:14.214467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:50.984821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"52'139","reported_seq":294,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-10-16T14:30:16.257464+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:30:16.257464+0000","last_peered":"2023-10-16T14:30:16.257464+0000","last_clean":"2023-10-16T14:30:16.257464+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:30:16.257464+0000","last_undegraded":"2023-10-16T14:30:16.257464+0000","last_fullsized":"2023-10-16T14:30:16.257464+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":1359872,"data_stored":951938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.121,"15min":1.1639999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.42799999999999999},"max":{"1min":1.385,"5min":1.752,"15min":1.9159999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.179,"15min":1.1359999999999999},"min":{"1min":0.85399999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"max":{"1min":1.476,"5min":1.673,"15min":1.673},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2170000000000001,"15min":1.1779999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.69599999999999995},"max":{"1min":1.611,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.429},{"interface":"front","average":{"1min":1.165,"5min":1.2290000000000001,"15min":1.22},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.67600000000000005},"max":{"1min":1.5309999999999999,"5min":1.716,"15min":2.0219999999999998},"last":1.105}]},{"osd":2,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.321,"15min":1.2889999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.63500000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.137,"5min":1.232,"15min":1.2230000000000001},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.63400000000000001},"max":{"1min":1.4490000000000001,"5min":1.7250000000000001,"15min":1.885},"last":1.5289999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3049999999999999,"15min":1.2889999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.75900000000000001},"max":{"1min":1.583,"5min":1.746,"15min":2.048},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.375,"15min":1.371},"min":{"1min":0.99199999999999999,"5min":0.93000000000000005,"15min":0.71899999999999997},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.2509999999999999},"last":1.008}]},{"osd":4,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.365,"15min":1.3680000000000001},"min":{"1min":0.96199999999999997,"5min":0.82899999999999996,"15min":0.67600000000000005},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2410000000000001,"15min":1.278},"min":{"1min":0.82099999999999995,"5min":0.73099999999999998,"15min":0.496},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.153},"last":0.94599999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.407,"15min":1.4239999999999999},"min":{"1min":0.94399999999999995,"5min":0.79800000000000004,"15min":0.61199999999999999},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2530000000000001},"last":1.149},{"interface":"front","average":{"1min":1.492,"5min":1.4650000000000001,"15min":1.508},"min":{"1min":0.998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2970000000000002},"last":1.385}]},{"osd":6,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4299999999999999},"min":{"1min":1.056,"5min":0.92200000000000004,"15min":0.63100000000000001},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.048},{"interface":"front","average":{"1min":1.573,"5min":1.494,"15min":1.522},"min":{"1min":1.1459999999999999,"5min":0.92500000000000004,"15min":0.59299999999999997},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.427},"last":1.2869999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575017,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36096,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335744,"statfs":{"total":96636764160,"available":96599801856,"internally_reserved":0,"allocated":770048,"data_stored":362152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1619999999999999,"15min":1.1599999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.66500000000000004},"max":{"1min":1.667,"5min":1.77,"15min":2.2160000000000002},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2529999999999999,"15min":1.228},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.6459999999999999,"5min":2.0529999999999999,"15min":3.863},"last":1.2}]},{"osd":1,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.284,"15min":1.2649999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.415,"5min":1.9330000000000001,"15min":2.2719999999999998},"last":1.425},{"interface":"front","average":{"1min":1.401,"5min":1.3120000000000001,"15min":1.321},"min":{"1min":0.94699999999999995,"5min":0.84099999999999997,"15min":0.80500000000000005},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":1.0900000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.2030000000000001,"15min":1.218},"min":{"1min":0.747,"5min":0.747,"15min":0.59899999999999998},"max":{"1min":1.3779999999999999,"5min":1.653,"15min":3.5800000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3600000000000001,"15min":1.3660000000000001},"min":{"1min":0.95599999999999996,"5min":0.88,"15min":0.70599999999999996},"max":{"1min":1.8799999999999999,"5min":2.1429999999999998,"15min":2.7250000000000001},"last":1.607}]},{"osd":3,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.347,"15min":1.345},"min":{"1min":0.97099999999999997,"5min":0.89400000000000002,"15min":0.77500000000000002},"max":{"1min":1.6819999999999999,"5min":1.875,"15min":4.9790000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.3240000000000001,"15min":1.3680000000000001},"min":{"1min":0.872,"5min":0.872,"15min":0.71399999999999997},"max":{"1min":1.546,"5min":1.8420000000000001,"15min":5.4160000000000004},"last":1.3440000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.302,"15min":1.331},"min":{"1min":0.91100000000000003,"5min":0.78700000000000003,"15min":0.59499999999999997},"max":{"1min":2.1070000000000002,"5min":2.2360000000000002,"15min":5},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.3580000000000001,"15min":1.401},"min":{"1min":0.878,"5min":0.85699999999999998,"15min":0.70799999999999996},"max":{"1min":1.9279999999999999,"5min":1.9810000000000001,"15min":5.4640000000000004},"last":1.29}]},{"osd":5,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3080000000000001,"15min":1.3400000000000001},"min":{"1min":0.92300000000000004,"5min":0.85299999999999998,"15min":0.754},"max":{"1min":1.794,"5min":1.794,"15min":3.9279999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.494,"15min":1.5249999999999999},"min":{"1min":1.194,"5min":0.97399999999999998,"15min":0.66100000000000003},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":3.6749999999999998},"last":1.6679999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5720000000000001,"15min":1.5840000000000001},"min":{"1min":1.2589999999999999,"5min":1.1439999999999999,"15min":0.69999999999999996},"max":{"1min":1.986,"5min":2.1139999999999999,"15min":4.6859999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.534,"15min":1.587},"min":{"1min":1.2589999999999999,"5min":1.01,"15min":0.68400000000000005},"max":{"1min":1.986,"5min":2.0139999999999998,"15min":4.883},"last":1.5589999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575017,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":765952,"data_stored":361550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1200000000000001,"15min":1.1180000000000001},"min":{"1min":0.68400000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.196,"15min":1.161},"min":{"1min":0.85799999999999998,"5min":0.78100000000000003,"15min":0.66900000000000004},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":2.2629999999999999},"last":0.70799999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.1919999999999999,"15min":1.171},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.64600000000000002},"max":{"1min":1.6779999999999999,"5min":1.726,"15min":1.726},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2270000000000001,"15min":1.2450000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.748},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.022}]},{"osd":2,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2769999999999999,"15min":1.254},"min":{"1min":0.91600000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.3759999999999999,"15min":1.369},"min":{"1min":1.0549999999999999,"5min":0.95199999999999996,"15min":0.84099999999999997},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.3500000000000001},"last":0.82899999999999996}]},{"osd":3,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2909999999999999,"15min":1.274},"min":{"1min":0.95899999999999996,"5min":0.95599999999999996,"15min":0.76200000000000001},"max":{"1min":1.742,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4379999999999999,"15min":1.4079999999999999},"min":{"1min":1.0609999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.0780000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3660000000000001,"15min":1.341},"min":{"1min":0.85899999999999999,"5min":0.83499999999999996,"15min":0.54700000000000004},"max":{"1min":1.752,"5min":1.9610000000000001,"15min":2.863},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.714,"5min":1.55,"15min":1.528},"min":{"1min":0.93300000000000005,"5min":0.82199999999999995,"15min":0.61699999999999999},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.175}]},{"osd":6,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4059999999999999,"15min":1.4039999999999999},"min":{"1min":0.88800000000000001,"5min":0.72699999999999998,"15min":0.61799999999999999},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":1.349},{"interface":"front","average":{"1min":1.448,"5min":1.4630000000000001,"15min":1.4830000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.70199999999999996},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.794},"last":1.2789999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:29:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.593,"15min":1.571},"min":{"1min":1.397,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.641,"15min":1.6180000000000001},"min":{"1min":1.1180000000000001,"5min":1.038,"15min":0.67700000000000005},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.415},"last":1.2230000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575018,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45832,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326008,"statfs":{"total":96636764160,"available":96589832192,"internally_reserved":0,"allocated":774144,"data_stored":363092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1830000000000001,"15min":1.202},"min":{"1min":0.72899999999999998,"5min":0.71899999999999997,"15min":0.57599999999999996},"max":{"1min":1.7789999999999999,"5min":1.782,"15min":5.367},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1919999999999999,"15min":1.236},"min":{"1min":0.82099999999999995,"5min":0.75,"15min":0.65000000000000002},"max":{"1min":1.641,"5min":1.8500000000000001,"15min":6.3159999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2050000000000001,"15min":1.1679999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.65900000000000003},"max":{"1min":1.5600000000000001,"5min":1.609,"15min":1.78},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.222,"15min":1.2909999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.782,"5min":1.782,"15min":5.4640000000000004},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3919999999999999,"15min":1.4259999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.69499999999999995},"max":{"1min":1.7410000000000001,"5min":1.788,"15min":5.617},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3660000000000001,"15min":1.411},"min":{"1min":0.95899999999999996,"5min":0.80800000000000005,"15min":0.745},"max":{"1min":1.7390000000000001,"5min":1.788,"15min":6.391},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.341,"15min":1.381},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.71399999999999997},"max":{"1min":1.5900000000000001,"5min":1.7370000000000001,"15min":7.4089999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.327,"15min":1.2889999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":2.0339999999999998},"last":0.437}]},{"osd":5,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.2829999999999999,"15min":1.347},"min":{"1min":0.95499999999999996,"5min":0.85099999999999998,"15min":0.48699999999999999},"max":{"1min":1.696,"5min":1.7490000000000001,"15min":5.4109999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.349,"15min":1.4419999999999999},"min":{"1min":0.98999999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.5880000000000001,"5min":1.78,"15min":9.8239999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3300000000000001,"15min":1.2969999999999999},"min":{"1min":0.96199999999999997,"5min":0.76500000000000001,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":1.8089999999999999,"15min":2.4729999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.54,"5min":1.4470000000000001,"15min":1.51},"min":{"1min":1.298,"5min":0.92100000000000004,"15min":0.44700000000000001},"max":{"1min":1.891,"5min":1.891,"15min":5.782},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Oct 16 14:29:47 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.474,"15min":1.53},"min":{"1min":1.1539999999999999,"5min":0.90800000000000003,"15min":0.78100000000000003},"max":{"1min":1.7549999999999999,"5min":1.9099999999999999,"15min":6.4379999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4810000000000001,"15min":1.4970000000000001},"min":{"1min":1.2110000000000001,"5min":1.012,"15min":0.67400000000000004},"max":{"1min":1.8080000000000001,"5min":1.8380000000000001,"15min":3.7799999999999998},"last":0.49399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575017,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40260,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331580,"statfs":{"total":96636764160,"available":96595537920,"internally_reserved":0,"allocated":774144,"data_stored":362118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.89900000000000002,"15min":0.93100000000000005},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.35699999999999998},"max":{"1min":1.4419999999999999,"5min":1.518,"15min":2.4420000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0780000000000001,"15min":1.04},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.42099999999999999},"max":{"1min":1.4850000000000001,"5min":1.746,"15min":1.982},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.96099999999999997,"15min":0.96199999999999997},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.32500000000000001},"max":{"1min":1.0069999999999999,"5min":1.8799999999999999,"15min":2.149},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.159,"15min":1.165},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.4259999999999999,"5min":1.6759999999999999,"15min":2.1600000000000001},"last":0.314}]},{"osd":2,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.3759999999999999,"15min":1.339},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.44800000000000001},"max":{"1min":1.639,"5min":1.9550000000000001,"15min":2.1259999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4670000000000001,"15min":1.4390000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.501},"max":{"1min":1.77,"5min":1.9470000000000001,"15min":2.6299999999999999},"last":0.5}]},{"osd":4,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.2569999999999999,"15min":1.2929999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.5049999999999999,"5min":1.7170000000000001,"15min":2.4529999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.363,"15min":1.389},"min":{"1min":1.038,"5min":1.002,"15min":0.75},"max":{"1min":1.845,"5min":1.909,"15min":2.3700000000000001},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5449999999999999,"15min":1.573},"min":{"1min":1.103,"5min":1.103,"15min":0.83799999999999997},"max":{"1min":1.907,"5min":2.0569999999999999,"15min":2.552},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.383,"5min":1.4790000000000001,"15min":1.522},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.609,"5min":2.3119999999999998,"15min":2.7589999999999999},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4259999999999999,"15min":1.4530000000000001},"min":{"1min":0.90600000000000003,"5min":0.90400000000000003,"15min":0.73799999999999999},"max":{"1min":1.976,"5min":2.008,"15min":2.3399999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.522,"15min":1.552},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9510000000000001,"5min":2.1230000000000002,"15min":2.347},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5980000000000001,"15min":1.6279999999999999},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.8440000000000001,"5min":2.395,"15min":2.8820000000000001},"last":0.751},{"interface":"front","average":{"1min":1.522,"5min":1.587,"15min":1.5840000000000001},"min":{"1min":1.038,"5min":1.038,"15min":0.86399999999999999},"max":{"1min":1.8819999999999999,"5min":2.0710000000000002,"15min":2.6179999999999999},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575018,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40268,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331572,"statfs":{"total":96636764160,"available":96595529728,"internally_reserved":0,"allocated":778240,"data_stored":363114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.88100000000000001,"15min":0.873},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.92500000000000004,"5min":1.7150000000000001,"15min":2.472},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.0249999999999999,"15min":1.036},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.488,"5min":1.7070000000000001,"15min":2.8879999999999999},"last":1.0409999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.98399999999999999,"15min":1},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3340000000000001,"5min":1.571,"15min":2.8090000000000002},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1659999999999999,"15min":1.1479999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":2.2829999999999999},"last":1.3640000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.286,"15min":1.216},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.728,"5min":2.085,"15min":2.085},"last":0.996},{"interface":"front","average":{"1min":1.05,"5min":1.2909999999999999,"15min":1.2789999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.42499999999999999},"max":{"1min":1.3819999999999999,"5min":1.917,"15min":2.6720000000000002},"last":1.524}]},{"osd":4,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.429,"15min":1.423},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.016,"5min":2.1859999999999999,"15min":2.5499999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.3879999999999999,"15min":1.3700000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":3.016,"5min":3.016,"15min":3.016},"last":1.3320000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.4399999999999999,"15min":1.4379999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.968,"5min":1.9730000000000001,"15min":3.25},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.5569999999999999,"15min":1.5349999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.78300000000000003},"max":{"1min":1.901,"5min":2.27,"15min":2.9350000000000001},"last":1.6339999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.508,"15min":1.508},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":3.536},"last":1.204},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5289999999999999,"15min":1.5429999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":3.3929999999999998},"last":1.4750000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:29:45 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.5800000000000001,"15min":1.617},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":5.2199999999999998},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.579,"5min":1.665,"15min":1.6559999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":3.496},"last":1.5720000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54144,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317696,"statfs":{"total":96636764160,"available":96581320704,"internally_reserved":0,"allocated":765952,"data_stored":361550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.91600000000000004,"15min":0.92300000000000004},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.44900000000000001},"max":{"1min":1.169,"5min":1.7669999999999999,"15min":1.821},"last":1.161},{"interface":"front","average":{"1min":1.044,"5min":1.1060000000000001,"15min":1.1000000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.40000000000000002},"max":{"1min":1.4279999999999999,"5min":1.712,"15min":1.7370000000000001},"last":1.004}]},{"osd":2,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.999,"15min":0.997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.47599999999999998},"max":{"1min":1.615,"5min":1.774,"15min":1.774},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.1910000000000001,"15min":1.159},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.48299999999999998},"max":{"1min":1.5960000000000001,"5min":1.667,"15min":1.796},"last":1.3420000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.347,"15min":1.333},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.52400000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.601},{"interface":"front","average":{"1min":1.163,"5min":1.242,"15min":1.194},"min":{"1min":0.621,"5min":0.621,"15min":0.45800000000000002},"max":{"1min":1.75,"5min":1.855,"15min":2.2050000000000001},"last":1.2789999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.325,"15min":1.3080000000000001},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.69599999999999995},"max":{"1min":1.6200000000000001,"5min":1.8260000000000001,"15min":2.9820000000000002},"last":1.097},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4650000000000001,"15min":1.4430000000000001},"min":{"1min":1.028,"5min":1.028,"15min":0.59499999999999997},"max":{"1min":1.843,"5min":2.258,"15min":3.242},"last":1.8120000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.472,"15min":1.4339999999999999},"min":{"1min":1.0960000000000001,"5min":0.97399999999999998,"15min":0.75700000000000001},"max":{"1min":1.8360000000000001,"5min":2.032,"15min":2.3690000000000002},"last":1.393},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4810000000000001,"15min":1.4159999999999999},"min":{"1min":1.052,"5min":1.0389999999999999,"15min":0.63700000000000001},"max":{"1min":1.7869999999999999,"5min":2.1339999999999999,"15min":2.181},"last":1.4199999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.569,"15min":1.554},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.69299999999999995},"max":{"1min":2.1019999999999999,"5min":2.3359999999999999,"15min":2.476},"last":1.927},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5469999999999999},"min":{"1min":1.153,"5min":1.0580000000000001,"15min":0.73099999999999998},"max":{"1min":2.0169999999999999,"5min":2.238,"15min":2.6160000000000001},"last":1.46}]},{"osd":7,"last update":"Mon Oct 16 14:29:42 2023","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.6779999999999999,"15min":1.625},"min":{"1min":1.224,"5min":1.1990000000000001,"15min":0.79300000000000004},"max":{"1min":2.1000000000000001,"5min":2.2639999999999998,"15min":2.5059999999999998},"last":1.524},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.6040000000000001,"15min":1.5620000000000001},"min":{"1min":1.0569999999999999,"5min":1.034,"15min":0.88800000000000001},"max":{"1min":1.9119999999999999,"5min":2.2170000000000001,"15min":2.3410000000000002},"last":1.552}]}]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":951882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.93799999999999994,"15min":0.95899999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.42199999999999999},"max":{"1min":1.093,"5min":1.498,"15min":2.2650000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1930000000000001,"15min":1.169},"min":{"1min":0.497,"5min":0.497,"15min":0.36399999999999999},"max":{"1min":1.605,"5min":1.859,"15min":2.6030000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2609999999999999,"15min":1.2450000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.8360000000000001,"5min":2.2970000000000002,"15min":3.653},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1899999999999999,"15min":1.2210000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.47399999999999998},"max":{"1min":2.012,"5min":2.012,"15min":3.1400000000000001},"last":1.175}]},{"osd":3,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.004,"15min":1.014},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.40100000000000002},"max":{"1min":1.4119999999999999,"5min":1.9410000000000001,"15min":2.391},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.2649999999999999,"15min":1.25},"min":{"1min":0.437,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.577,"5min":2.1600000000000001,"15min":2.79},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3129999999999999,"15min":1.304},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.61599999999999999},"max":{"1min":1.6539999999999999,"5min":1.806,"15min":2.7370000000000001},"last":1.054},{"interface":"front","average":{"1min":1.401,"5min":1.484,"15min":1.478},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.65900000000000003},"max":{"1min":1.6990000000000001,"5min":2.6259999999999999,"15min":3.6190000000000002},"last":1.085}]},{"osd":5,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.4330000000000001,"15min":1.45},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58099999999999996},"max":{"1min":1.8720000000000001,"5min":2.2400000000000002,"15min":3.2370000000000001},"last":1.375},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3600000000000001,"15min":1.389},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.67900000000000005},"max":{"1min":1.645,"5min":2.4889999999999999,"15min":2.819},"last":1.1319999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.5960000000000001,"15min":1.6040000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.75},"max":{"1min":1.966,"5min":2.4209999999999998,"15min":3.5779999999999998},"last":1.347},{"interface":"front","average":{"1min":1.429,"5min":1.504,"15min":1.528},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":0.82199999999999995},"max":{"1min":2.1440000000000001,"5min":2.1459999999999999,"15min":3.0979999999999999},"last":1.29}]},{"osd":7,"last update":"Mon Oct 16 14:29:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.603,"15min":1.607},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.67100000000000004},"max":{"1min":1.911,"5min":2.2650000000000001,"15min":3.3340000000000001},"last":1.472},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5469999999999999,"15min":1.615},"min":{"1min":0.877,"5min":0.877,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":2.976,"15min":6.5839999999999996},"last":1.202}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:30:21.289 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:30:21.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:30:21.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.562+0000 7f53ca6d7640 1 Processor -- start 2023-10-16T14:30:21.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.562+0000 7f53ca6d7640 1 -- start start 2023-10-16T14:30:21.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53ca6d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c4159ac0 0x7f53c415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53ca6d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53c40704e0 0x7f53c4070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53ca6d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c415e5c0 con 0x7f53c40cc270 2023-10-16T14:30:21.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53ca6d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c4070eb0 con 0x7f53c40704e0 2023-10-16T14:30:21.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53ca6d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c406b810 con 0x7f53c4159ac0 2023-10-16T14:30:21.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53c8ed4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f53c40cc270 0x7f53c40c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39504/0 (socket says 172.21.15.29:39504) 2023-10-16T14:30:21.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53c8ed4640 1 -- 172.21.15.29:0/922754606 learned_addr learned my addr 172.21.15.29:0/922754606 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:21.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53c9ed6640 1 --2- 172.21.15.29:0/922754606 >> v2:172.21.15.29:3300/0 conn(0x7f53c40704e0 0x7f53c4070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.564+0000 7f53c96d5640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c4159ac0 0x7f53c415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53bbfff640 1 -- 172.21.15.29:0/922754606 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316943137 0 0) 0x7f53b4001110 con 0x7f53c40cc270 2023-10-16T14:30:21.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53bbfff640 1 -- 172.21.15.29:0/922754606 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b0003520 con 0x7f53c40cc270 2023-10-16T14:30:21.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53c9ed6640 1 -- 172.21.15.29:0/922754606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c4159ac0 msgr2=0x7f53c415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:21.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53c9ed6640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c4159ac0 0x7f53c415e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53c9ed6640 1 -- 172.21.15.29:0/922754606 >> v1:172.21.15.29:6789/0 conn(0x7f53c40cc270 legacy=0x7f53c40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53c9ed6640 1 -- 172.21.15.29:0/922754606 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c40cc810 con 0x7f53c40704e0 2023-10-16T14:30:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.565+0000 7f53c96d5640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c4159ac0 0x7f53c415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.566+0000 7f53c9ed6640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c4070970 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009d90 tx=0x7f53bc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96de9b7905ea4246 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.566+0000 7f53bbfff640 1 -- 172.21.15.29:0/922754606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc02cc50 con 0x7f53c40704e0 2023-10-16T14:30:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.566+0000 7f53bbfff640 1 -- 172.21.15.29:0/922754606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53bc02cdb0 con 0x7f53c40704e0 2023-10-16T14:30:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.566+0000 7f53bbfff640 1 -- 172.21.15.29:0/922754606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc034640 con 0x7f53c40704e0 2023-10-16T14:30:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 -- 172.21.15.29:0/922754606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 msgr2=0x7f53c4070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c4070970 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009d90 tx=0x7f53bc02bea0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 -- 172.21.15.29:0/922754606 shutdown_connections 2023-10-16T14:30:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c4070970 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/922754606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c4159ac0 0x7f53c415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 -- 172.21.15.29:0/922754606 >> 172.21.15.29:0/922754606 conn(0x7f53c40ce6a0 msgr2=0x7f53c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 -- 172.21.15.29:0/922754606 shutdown_connections 2023-10-16T14:30:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.567+0000 7f53ca6d7640 1 -- 172.21.15.29:0/922754606 wait complete. 2023-10-16T14:30:21.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.569+0000 7f53ca6d7640 1 Processor -- start 2023-10-16T14:30:21.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.569+0000 7f53ca6d7640 1 -- start start 2023-10-16T14:30:21.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53ca6d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c40acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53ca6d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c40cc270 0x7f53c41489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53ca6d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53c4159ac0 0x7f53c40cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53ca6d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c40a9960 con 0x7f53c40704e0 2023-10-16T14:30:21.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53ca6d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c414ff00 con 0x7f53c4159ac0 2023-10-16T14:30:21.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53ca6d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c40ad0b0 con 0x7f53c40cc270 2023-10-16T14:30:21.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53c96d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c40acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53c96d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c40acb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60892/0 (socket says 172.21.15.29:60892) 2023-10-16T14:30:21.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.570+0000 7f53c96d5640 1 -- 172.21.15.29:0/3951260408 learned_addr learned my addr 172.21.15.29:0/3951260408 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:21.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53c96d5640 1 -- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c40cc270 msgr2=0x7f53c41489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53c96d5640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c40cc270 0x7f53c41489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53c96d5640 1 -- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53c4159ac0 msgr2=0x7f53c40cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53c96d5640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53c4159ac0 0x7f53c40cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53c96d5640 1 -- 172.21.15.29:0/3951260408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc009a70 con 0x7f53c40704e0 2023-10-16T14:30:21.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53c96d5640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c40acb70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f53c000e7e0 tx=0x7f53c000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0004030 con 0x7f53c40704e0 2023-10-16T14:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c0004190 con 0x7f53c40704e0 2023-10-16T14:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0004520 con 0x7f53c40704e0 2023-10-16T14:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c40a9030 con 0x7f53c40704e0 2023-10-16T14:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.571+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c40a95f0 con 0x7f53c40704e0 2023-10-16T14:30:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.573+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c406b980 con 0x7f53c40704e0 2023-10-16T14:30:21.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.574+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f53c000c6d0 con 0x7f53c40704e0 2023-10-16T14:30:21.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.574+0000 7f53b9ffb640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f53a806afc0 0x7f53a806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:21.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.575+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53c00895d0 con 0x7f53c40704e0 2023-10-16T14:30:21.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.575+0000 7f53c8ed4640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f53a806afc0 0x7f53a806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:21.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.576+0000 7f53c8ed4640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f53a806afc0 0x7f53a806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000f60 tx=0x7f53b4001110 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:21.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.579+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f53c00543c0 con 0x7f53c40704e0 2023-10-16T14:30:21.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.850+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f53c404e9b0 con 0x7f53c40704e0 2023-10-16T14:30:21.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.854+0000 7f53b9ffb640 1 -- 172.21.15.29:0/3951260408 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f53c00579e0 con 0x7f53c40704e0 2023-10-16T14:30:21.859 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:30:21.859 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:30:21.859 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:30:21.859 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:30:21.860 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:30:21.860 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:30:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f53a806afc0 msgr2=0x7f53a806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f53a806afc0 0x7f53a806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000f60 tx=0x7f53b4001110 comp rx=0 tx=0).stop 2023-10-16T14:30:21.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 msgr2=0x7f53c40acb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:21.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c40acb70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f53c000e7e0 tx=0x7f53c000eca0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 shutdown_connections 2023-10-16T14:30:21.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f53a806afc0 0x7f53a806d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53c4159ac0 0x7f53c40cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53c40cc270 0x7f53c41489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 --2- 172.21.15.29:0/3951260408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c40704e0 0x7f53c40acb70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:21.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 >> 172.21.15.29:0/3951260408 conn(0x7f53c40ce6a0 msgr2=0x7f53c4150fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:21.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 shutdown_connections 2023-10-16T14:30:21.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:21.857+0000 7f53ca6d7640 1 -- 172.21.15.29:0/3951260408 wait complete. 2023-10-16T14:30:21.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:30:22.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.108+0000 7f36418ca640 1 Processor -- start 2023-10-16T14:30:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.108+0000 7f36418ca640 1 -- start start 2023-10-16T14:30:22.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.109+0000 7f36418ca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 0x7f363c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.109+0000 7f36418ca640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f363c070f20 0x7f363c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f36418ca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c160a90 con 0x7f363c15c9f0 2023-10-16T14:30:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f36418ca640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363c0cdec0 con 0x7f363c070f20 2023-10-16T14:30:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f36418ca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0cd900 con 0x7f363c15dde0 2023-10-16T14:30:22.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f36408c8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f363c15c9f0 0x7f363c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39520/0 (socket says 172.21.15.29:39520) 2023-10-16T14:30:22.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f36408c8640 1 -- 172.21.15.29:0/3146652019 learned_addr learned my addr 172.21.15.29:0/3146652019 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:22.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f363bfff640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 0x7f363c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.110+0000 7f36410c9640 1 --2- 172.21.15.29:0/3146652019 >> v2:172.21.15.170:3300/0 conn(0x7f363c070f20 0x7f363c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658568761 0 0) 0x7f363c160a90 con 0x7f363c15c9f0 2023-10-16T14:30:22.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f362c003520 con 0x7f363c15c9f0 2023-10-16T14:30:22.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 574251864 0 0) 0x7f362c003520 con 0x7f363c15c9f0 2023-10-16T14:30:22.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 msgr2=0x7f363c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:22.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 0x7f363c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 >> v2:172.21.15.170:3300/0 conn(0x7f363c070f20 msgr2=0x7f363c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:22.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 --2- 172.21.15.29:0/3146652019 >> v2:172.21.15.170:3300/0 conn(0x7f363c070f20 0x7f363c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0cdbe0 con 0x7f363c15c9f0 2023-10-16T14:30:22.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3630001670 con 0x7f363c15c9f0 2023-10-16T14:30:22.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 >> v1:172.21.15.29:6789/0 conn(0x7f363c15c9f0 legacy=0x7f363c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:22.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 0x7f362c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f362c003f40 0x7f362c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f362c004d00 0x7f362c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0cd900 con 0x7f362c004d00 2023-10-16T14:30:22.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362c00b980 con 0x7f362c003f40 2023-10-16T14:30:22.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.111+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362c00bac0 con 0x7f363c15dde0 2023-10-16T14:30:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36410c9640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f362c004d00 0x7f362c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36408c8640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 0x7f362c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f363bfff640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f362c003f40 0x7f362c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36410c9640 1 -- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 msgr2=0x7f362c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36410c9640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f363c15dde0 0x7f362c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36410c9640 1 -- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f362c003f40 msgr2=0x7f362c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:22.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36410c9640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f362c003f40 0x7f362c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.112+0000 7f36410c9640 1 -- 172.21.15.29:0/3146652019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c003520 con 0x7f362c004d00 2023-10-16T14:30:22.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.113+0000 7f36410c9640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f362c004d00 0x7f362c005180 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f363400e170 tx=0x7f362c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5277ecc1b4d2cf9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:22.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.113+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363402cc20 con 0x7f362c004d00 2023-10-16T14:30:22.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.113+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363400f040 con 0x7f362c004d00 2023-10-16T14:30:22.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.113+0000 7f363b7fe640 1 -- 172.21.15.29:0/3146652019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634034440 con 0x7f362c004d00 2023-10-16T14:30:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36418ca640 1 -- 172.21.15.29:0/3146652019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f362c004d00 msgr2=0x7f362c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36418ca640 1 --2- 172.21.15.29:0/3146652019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f362c004d00 0x7f362c005180 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f363400e170 tx=0x7f362c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36410c9640 1 -- 172.21.15.29:0/3146652019 reap_dead start 2023-10-16T14:30:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36418ca640 1 -- 172.21.15.29:0/3146652019 shutdown_connections 2023-10-16T14:30:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36418ca640 1 -- 172.21.15.29:0/3146652019 >> 172.21.15.29:0/3146652019 conn(0x7f363c0ce6a0 msgr2=0x7f363c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36418ca640 1 -- 172.21.15.29:0/3146652019 shutdown_connections 2023-10-16T14:30:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.114+0000 7f36418ca640 1 -- 172.21.15.29:0/3146652019 wait complete. 2023-10-16T14:30:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.116+0000 7f36418ca640 1 Processor -- start 2023-10-16T14:30:22.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.116+0000 7f36418ca640 1 -- start start 2023-10-16T14:30:22.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f36418ca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f362c004d00 0x7f363c0c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f36418ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 0x7f363c06a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f36418ca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f363c157d10 0x7f363c158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f36418ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c1432a0 con 0x7f363c069da0 2023-10-16T14:30:22.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f36418ca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c06a760 con 0x7f363c157d10 2023-10-16T14:30:22.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f36418ca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c158a40 con 0x7f362c004d00 2023-10-16T14:30:22.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.117+0000 7f363bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 0x7f363c06a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 0x7f363c06a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60910/0 (socket says 172.21.15.29:60910) 2023-10-16T14:30:22.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 -- 172.21.15.29:0/3287118758 learned_addr learned my addr 172.21.15.29:0/3287118758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f36408c8640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f362c004d00 0x7f363c0c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 -- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f362c004d00 msgr2=0x7f363c0c18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f362c004d00 0x7f363c0c18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 -- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f363c157d10 msgr2=0x7f363c158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f363c157d10 0x7f363c158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 -- 172.21.15.29:0/3287118758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c158d20 con 0x7f363c069da0 2023-10-16T14:30:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f363bfff640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 0x7f363c06a220 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3628011590 tx=0x7f3628011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628004220 con 0x7f363c069da0 2023-10-16T14:30:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.118+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3628004380 con 0x7f363c069da0 2023-10-16T14:30:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.119+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c159000 con 0x7f363c069da0 2023-10-16T14:30:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.119+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628015930 con 0x7f363c069da0 2023-10-16T14:30:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.119+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c1586d0 con 0x7f363c069da0 2023-10-16T14:30:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.120+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c04e9b0 con 0x7f363c069da0 2023-10-16T14:30:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.121+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3628015a90 con 0x7f363c069da0 2023-10-16T14:30:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.122+0000 7f36397fa640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f362006afc0 0x7f362006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.122+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36280898d0 con 0x7f363c069da0 2023-10-16T14:30:22.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.122+0000 7f36408c8640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f362006afc0 0x7f362006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:22.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.124+0000 7f36408c8640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f362006afc0 0x7f362006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f36300058b0 tx=0x7f3630000f50 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:22.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.126+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f36280547d0 con 0x7f363c069da0 2023-10-16T14:30:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.360+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f363c0ad880 con 0x7f363c069da0 2023-10-16T14:30:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.361+0000 7f36397fa640 1 -- 172.21.15.29:0/3287118758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f3628022090 con 0x7f363c069da0 2023-10-16T14:30:22.365 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:22.365 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:30:15.728624+0000","modified":"2023-10-16T14:30:18.806702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1869070988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1869070988},{"type":"v1","addr":"172.21.15.29:6835","nonce":1869070988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T14:30:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-16T14:30:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f362006afc0 msgr2=0x7f362006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f362006afc0 0x7f362006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f36300058b0 tx=0x7f3630000f50 comp rx=0 tx=0).stop 2023-10-16T14:30:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 msgr2=0x7f363c06a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 0x7f363c06a220 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3628011590 tx=0x7f3628011a50 comp rx=0 tx=0).stop 2023-10-16T14:30:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 shutdown_connections 2023-10-16T14:30:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f362006afc0 0x7f362006d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c069da0 0x7f363c06a220 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f363c157d10 0x7f363c158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 --2- 172.21.15.29:0/3287118758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f362c004d00 0x7f363c0c18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 >> 172.21.15.29:0/3287118758 conn(0x7f363c0ce6a0 msgr2=0x7f363c0b0140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 shutdown_connections 2023-10-16T14:30:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:22.364+0000 7f36418ca640 1 -- 172.21.15.29:0/3287118758 wait complete. 2023-10-16T14:30:22.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:30:15.728624+0000', 'modified': '2023-10-16T14:30:18.806702+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5874': {'gid': 5874, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/1869070988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 1869070988}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 1869070988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:30:22.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:30:22.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:30:23.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.024+0000 7f131b49a640 1 Processor -- start 2023-10-16T14:30:23.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.025+0000 7f131b49a640 1 -- start start 2023-10-16T14:30:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.026+0000 7f131b49a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 0x7f131406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.026+0000 7f131b49a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f13140cda80 0x7f131415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.026+0000 7f131b49a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131406c960 con 0x7f13140ad970 2023-10-16T14:30:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.026+0000 7f131b49a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131415b9d0 con 0x7f13140cda80 2023-10-16T14:30:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.026+0000 7f131b49a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131415bcb0 con 0x7f131406c040 2023-10-16T14:30:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.027+0000 7f1319c97640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f13140ad970 0x7f131415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39532/0 (socket says 172.21.15.29:39532) 2023-10-16T14:30:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.027+0000 7f1319c97640 1 -- 172.21.15.29:0/4276906687 learned_addr learned my addr 172.21.15.29:0/4276906687 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.027+0000 7f131a498640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 0x7f131406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.027+0000 7f131ac99640 1 --2- 172.21.15.29:0/4276906687 >> v2:172.21.15.170:3300/0 conn(0x7f13140cda80 0x7f131415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.027+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635482774 0 0) 0x7f131406c960 con 0x7f13140ad970 2023-10-16T14:30:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.027+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1300003520 con 0x7f13140ad970 2023-10-16T14:30:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811893880 0 0) 0x7f1300003520 con 0x7f13140ad970 2023-10-16T14:30:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 msgr2=0x7f131406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 0x7f131406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 >> v2:172.21.15.170:3300/0 conn(0x7f13140cda80 msgr2=0x7f131415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 --2- 172.21.15.29:0/4276906687 >> v2:172.21.15.170:3300/0 conn(0x7f13140cda80 0x7f131415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131415bf90 con 0x7f13140ad970 2023-10-16T14:30:23.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1304001670 con 0x7f13140ad970 2023-10-16T14:30:23.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 >> v1:172.21.15.29:6789/0 conn(0x7f13140ad970 legacy=0x7f131415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1300003970 0x7f1300003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13000047c0 0x7f1300004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 0x7f1300005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131415b9d0 con 0x7f13000047c0 2023-10-16T14:30:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131415bcb0 con 0x7f1300003970 2023-10-16T14:30:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130000b980 con 0x7f131406c040 2023-10-16T14:30:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.028+0000 7f1319c97640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13000047c0 0x7f1300004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f131a498640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1300003970 0x7f1300003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f1319c97640 1 -- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 msgr2=0x7f1300005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f1319c97640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f131406c040 0x7f1300005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f1319c97640 1 -- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1300003970 msgr2=0x7f1300003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f1319c97640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1300003970 0x7f1300003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f1319c97640 1 -- 172.21.15.29:0/4276906687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300003520 con 0x7f13000047c0 2023-10-16T14:30:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.029+0000 7f131a498640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1300003970 0x7f1300003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.030+0000 7f1319c97640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13000047c0 0x7f1300004c40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f130400c290 tx=0x7f13000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13389da58740fbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.030+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304005ad0 con 0x7f13000047c0 2023-10-16T14:30:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.030+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304005c30 con 0x7f13000047c0 2023-10-16T14:30:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.030+0000 7f1319496640 1 -- 172.21.15.29:0/4276906687 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130402c8d0 con 0x7f13000047c0 2023-10-16T14:30:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131b49a640 1 -- 172.21.15.29:0/4276906687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13000047c0 msgr2=0x7f1300004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131b49a640 1 --2- 172.21.15.29:0/4276906687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13000047c0 0x7f1300004c40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f130400c290 tx=0x7f13000057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131ac99640 1 -- 172.21.15.29:0/4276906687 reap_dead start 2023-10-16T14:30:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131b49a640 1 -- 172.21.15.29:0/4276906687 shutdown_connections 2023-10-16T14:30:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131b49a640 1 -- 172.21.15.29:0/4276906687 >> 172.21.15.29:0/4276906687 conn(0x7f13140ce6a0 msgr2=0x7f13140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131b49a640 1 -- 172.21.15.29:0/4276906687 shutdown_connections 2023-10-16T14:30:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.031+0000 7f131b49a640 1 -- 172.21.15.29:0/4276906687 wait complete. 2023-10-16T14:30:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.033+0000 7f131b49a640 1 Processor -- start 2023-10-16T14:30:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.033+0000 7f131b49a640 1 -- start start 2023-10-16T14:30:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131b49a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13000047c0 0x7f131415db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131b49a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13141595c0 0x7f1314159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131b49a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 0x7f1314145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131b49a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13140c6660 con 0x7f13141458b0 2023-10-16T14:30:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131b49a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13140c8280 con 0x7f13141595c0 2023-10-16T14:30:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131b49a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13140c8560 con 0x7f13000047c0 2023-10-16T14:30:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131ac99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 0x7f1314145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131ac99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 0x7f1314145d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60922/0 (socket says 172.21.15.29:60922) 2023-10-16T14:30:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.034+0000 7f131ac99640 1 -- 172.21.15.29:0/346739724 learned_addr learned my addr 172.21.15.29:0/346739724 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f1319c97640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13141595c0 0x7f1314159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131a498640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13000047c0 0x7f131415db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131ac99640 1 -- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13000047c0 msgr2=0x7f131415db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131ac99640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13000047c0 0x7f131415db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131ac99640 1 -- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13141595c0 msgr2=0x7f1314159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131ac99640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13141595c0 0x7f1314159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131ac99640 1 -- 172.21.15.29:0/346739724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1314158b00 con 0x7f13141458b0 2023-10-16T14:30:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f1319c97640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13141595c0 0x7f1314159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f131ac99640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 0x7f1314145d60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f131406d2e0 tx=0x7f130c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f130affd640 1 -- 172.21.15.29:0/346739724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c010040 con 0x7f13141458b0 2023-10-16T14:30:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f130affd640 1 -- 172.21.15.29:0/346739724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c00bae0 con 0x7f13141458b0 2023-10-16T14:30:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.035+0000 7f130affd640 1 -- 172.21.15.29:0/346739724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c00be70 con 0x7f13141458b0 2023-10-16T14:30:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.036+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1314158de0 con 0x7f13141458b0 2023-10-16T14:30:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.036+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131415e070 con 0x7f13141458b0 2023-10-16T14:30:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.038+0000 7f130affd640 1 -- 172.21.15.29:0/346739724 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f130c01d400 con 0x7f13141458b0 2023-10-16T14:30:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.039+0000 7f130affd640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f12f806af90 0x7f12f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.039+0000 7f130affd640 1 -- 172.21.15.29:0/346739724 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f130c089fe0 con 0x7f13141458b0 2023-10-16T14:30:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.039+0000 7f131a498640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f12f806af90 0x7f12f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.040+0000 7f131f367740 1 Processor -- start 2023-10-16T14:30:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.040+0000 7f131f367740 1 -- start start 2023-10-16T14:30:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131a498640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f12f806af90 0x7f12f806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1310010910 tx=0x7f1310008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131f367740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3dca070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3df9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3dca5b0 0x559cd3dfa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131f367740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3cea7a0 con 0x559cd3df7170 2023-10-16T14:30:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3d3c740 con 0x559cd3df7550 2023-10-16T14:30:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cd3d39e30 con 0x559cd3dca5b0 2023-10-16T14:30:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131a498640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3dca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131a498640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3dca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60924/0 (socket says 172.21.15.29:60924) 2023-10-16T14:30:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131a498640 1 -- 172.21.15.29:0/3409453279 learned_addr learned my addr 172.21.15.29:0/3409453279 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f1319c97640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3df9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131a498640 1 -- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3dca5b0 msgr2=0x559cd3dfa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.041+0000 7f131ac99640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3dca5b0 0x559cd3dfa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f131a498640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3dca5b0 0x559cd3dfa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f131a498640 1 -- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 msgr2=0x559cd3df9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f131a498640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3df9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f131a498640 1 -- 172.21.15.29:0/3409453279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cd3cf9910 con 0x559cd3df7170 2023-10-16T14:30:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f1319c97640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3df9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f131ac99640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3dca5b0 0x559cd3dfa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f131a498640 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3dca070 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1310009e40 tx=0x7f1310018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f263d40e266240d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f12f7fff640 1 -- 172.21.15.29:0/3409453279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310024070 con 0x559cd3df7170 2023-10-16T14:30:23.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f12f7fff640 1 -- 172.21.15.29:0/3409453279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131001f0d0 con 0x559cd3df7170 2023-10-16T14:30:23.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.042+0000 7f12f7fff640 1 -- 172.21.15.29:0/3409453279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131001d420 con 0x559cd3df7170 2023-10-16T14:30:23.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 -- 172.21.15.29:0/3409453279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 msgr2=0x559cd3dca070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3dca070 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1310009e40 tx=0x7f1310018000 comp rx=0 tx=0).stop 2023-10-16T14:30:23.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 -- 172.21.15.29:0/3409453279 shutdown_connections 2023-10-16T14:30:23.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3dca5b0 0x559cd3dfa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3df9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 --2- 172.21.15.29:0/3409453279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3dca070 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 -- 172.21.15.29:0/3409453279 >> 172.21.15.29:0/3409453279 conn(0x559cd3dcc380 msgr2=0x559cd3e14f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 -- 172.21.15.29:0/3409453279 shutdown_connections 2023-10-16T14:30:23.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.043+0000 7f131f367740 1 -- 172.21.15.29:0/3409453279 wait complete. 2023-10-16T14:30:23.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.044+0000 7f131f367740 1 -- start start 2023-10-16T14:30:23.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.044+0000 7f131f367740 1 Processor -- start 2023-10-16T14:30:23.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7170 0x559cd3df5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 0x559cd3e11e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df6100 0x559cd3e12340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3d35720 con 0x559cd3df7550 2023-10-16T14:30:23.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3cd3970 con 0x559cd3df7170 2023-10-16T14:30:23.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cd3cd3610 con 0x559cd3df6100 2023-10-16T14:30:23.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.045+0000 7f131f367740 10 client.0 _ll_register_callbacks cb 0x559cd3cf59d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:30:23.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 0x559cd3e11e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 0x559cd3e11e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60926/0 (socket says 172.21.15.29:60926) 2023-10-16T14:30:23.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 -- 172.21.15.29:0/784446205 learned_addr learned my addr 172.21.15.29:0/784446205 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f131a498640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7170 0x559cd3df5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f131ac99640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df6100 0x559cd3e12340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 -- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df6100 msgr2=0x559cd3e12340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df6100 0x559cd3e12340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 -- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7170 msgr2=0x559cd3df5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7170 0x559cd3df5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f1319c97640 1 -- 172.21.15.29:0/784446205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1310009b20 con 0x559cd3df7550 2023-10-16T14:30:23.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f131ac99640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df6100 0x559cd3e12340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.046+0000 7f131a498640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7170 0x559cd3df5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f1319c97640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 0x559cd3e11e00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f130400c3b0 tx=0x7f1304001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb783c0e9788c02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f12f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:30:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f12f57fa640 1 -- 172.21.15.29:0/784446205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304001c80 con 0x559cd3df7550 2023-10-16T14:30:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f12f57fa640 1 -- 172.21.15.29:0/784446205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f130403c050 con 0x559cd3df7550 2023-10-16T14:30:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559cd3d5c160 con 0x559cd3df7550 2023-10-16T14:30:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f12f57fa640 1 -- 172.21.15.29:0/784446205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f130402cae0 con 0x559cd3df7550 2023-10-16T14:30:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f131f367740 10 client.6024 fetch_fsmap learned FSMap version 25 2023-10-16T14:30:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x559cd3d33da0 con 0x559cd3df7550 2023-10-16T14:30:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.047+0000 7f12f57fa640 1 -- 172.21.15.29:0/784446205 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559cd3d5c160 con 0x559cd3df7550 2023-10-16T14:30:23.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f12f57fa640 1 -- 172.21.15.29:0/784446205 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f130402cc40 con 0x559cd3df7550 2023-10-16T14:30:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131f367740 10 client.6024 fetch_fsmap finished waiting for FSMap version 25 2023-10-16T14:30:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131f367740 10 client.6024 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:30:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131f367740 20 client.6024 populate_metadata read hostname 'smithi029' 2023-10-16T14:30:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131f367740 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3e14c70 0x559cd3e0f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131f367740 4 client.6024 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:30:23.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 --> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559cd3d07cc0 con 0x559cd3e14c70 2023-10-16T14:30:23.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131a498640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3e14c70 0x559cd3e0f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f131a498640 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3e14c70 0x559cd3e0f900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea330f9ad3c25486 server_cookie=d87e367a2a5fdd78 in_seq=0 out_seq=0 2023-10-16T14:30:23.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.048+0000 7f12f57fa640 10 client.6024 ms_handle_connect on v2:172.21.15.29:6834/1869070988 2023-10-16T14:30:23.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.049+0000 7f12f57fa640 1 -- 172.21.15.29:0/784446205 <== mds.0 v2:172.21.15.29:6834/1869070988 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559cd3d07cc0 con 0x559cd3e14c70 2023-10-16T14:30:23.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.049+0000 7f12f57fa640 10 client.6024 handle_command_reply: tid=0 2023-10-16T14:30:23.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.049+0000 7f131f367740 1 client.6024 shutdown 2023-10-16T14:30:23.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 msgr2=0x559cd3e11e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 0x559cd3e11e00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f130400c3b0 tx=0x7f1304001a00 comp rx=0 tx=0).stop 2023-10-16T14:30:23.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 shutdown_connections 2023-10-16T14:30:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3e14c70 0x559cd3e0f900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7550 0x559cd3e11e00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df6100 0x559cd3e12340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 --2- 172.21.15.29:0/784446205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7170 0x559cd3df5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f12f57fa640 0 client.6024 ms_handle_reset on v2:172.21.15.29:6834/1869070988 2023-10-16T14:30:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 >> 172.21.15.29:0/784446205 conn(0x559cd3dca5b0 msgr2=0x559cd3dcdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 shutdown_connections 2023-10-16T14:30:23.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 1 -- 172.21.15.29:0/784446205 wait complete. 2023-10-16T14:30:23.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.050+0000 7f131f367740 20 client.6024 trim_cache size 0 max 16384 2023-10-16T14:30:23.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.067+0000 7f131f367740 1 Processor -- start 2023-10-16T14:30:23.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.067+0000 7f131f367740 1 -- start start 2023-10-16T14:30:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df6100 0x559cd3df9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f131f367740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3e189c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f131f367740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3a99f20 con 0x559cd3df7170 2023-10-16T14:30:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3e15bc0 con 0x559cd3df6100 2023-10-16T14:30:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cd3dc5a70 con 0x559cd3df7550 2023-10-16T14:30:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f1319c97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3e189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f1319c97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3e189c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60928/0 (socket says 172.21.15.29:60928) 2023-10-16T14:30:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.068+0000 7f1319c97640 1 -- 172.21.15.29:0/1970623057 learned_addr learned my addr 172.21.15.29:0/1970623057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f131a498640 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df6100 0x559cd3df9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f1319c97640 1 -- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7550 msgr2=0x559cd3e0e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f131ac99640 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f1319c97640 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f1319c97640 1 -- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df6100 msgr2=0x559cd3df9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f1319c97640 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df6100 0x559cd3df9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f1319c97640 1 -- 172.21.15.29:0/1970623057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cd3cd2b80 con 0x559cd3df7170 2023-10-16T14:30:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f131a498640 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df6100 0x559cd3df9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.069+0000 7f1319c97640 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3e189c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1304001a40 tx=0x7f130403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e58c8d5346faeff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f1308ff9640 1 -- 172.21.15.29:0/1970623057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304001c80 con 0x559cd3df7170 2023-10-16T14:30:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f1308ff9640 1 -- 172.21.15.29:0/1970623057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130402cae0 con 0x559cd3df7170 2023-10-16T14:30:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f1308ff9640 1 -- 172.21.15.29:0/1970623057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304007280 con 0x559cd3df7170 2023-10-16T14:30:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 -- 172.21.15.29:0/1970623057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 msgr2=0x559cd3e189c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3e189c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1304001a40 tx=0x7f130403f000 comp rx=0 tx=0).stop 2023-10-16T14:30:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 -- 172.21.15.29:0/1970623057 shutdown_connections 2023-10-16T14:30:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df7170 0x559cd3e189c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 --2- 172.21.15.29:0/1970623057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df6100 0x559cd3df9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 -- 172.21.15.29:0/1970623057 >> 172.21.15.29:0/1970623057 conn(0x559cd3dca5b0 msgr2=0x559cd3e12e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 -- 172.21.15.29:0/1970623057 shutdown_connections 2023-10-16T14:30:23.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.070+0000 7f131f367740 1 -- 172.21.15.29:0/1970623057 wait complete. 2023-10-16T14:30:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.071+0000 7f131f367740 1 -- start start 2023-10-16T14:30:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.071+0000 7f131f367740 1 Processor -- start 2023-10-16T14:30:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 0x559cd3e65ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7170 0x559cd3dfc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3dcf4a0 con 0x559cd3df6100 2023-10-16T14:30:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cd3e7cc10 con 0x559cd3df7550 2023-10-16T14:30:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559cd3e661e0 con 0x559cd3df7170 2023-10-16T14:30:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.072+0000 7f131f367740 10 client.0 _ll_register_callbacks cb 0x7f13000036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:30:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 0x559cd3e65ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 0x559cd3e65ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60934/0 (socket says 172.21.15.29:60934) 2023-10-16T14:30:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 -- 172.21.15.29:0/144107148 learned_addr learned my addr 172.21.15.29:0/144107148 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 -- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7170 msgr2=0x559cd3dfc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7170 0x559cd3dfc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131ac99640 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 -- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 msgr2=0x559cd3e0e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 -- 172.21.15.29:0/144107148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559cd3d33da0 con 0x559cd3df6100 2023-10-16T14:30:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f131a498640 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 0x559cd3e65ca0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1310008010 tx=0x7f1310004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35380e2b1176e09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.073+0000 7f12f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:30:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f12f57fa640 1 -- 172.21.15.29:0/144107148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310037070 con 0x559cd3df6100 2023-10-16T14:30:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f12f57fa640 1 -- 172.21.15.29:0/144107148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13100339e0 con 0x559cd3df6100 2023-10-16T14:30:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f131001f0d0 con 0x559cd3df6100 2023-10-16T14:30:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f12f57fa640 1 -- 172.21.15.29:0/144107148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1310024040 con 0x559cd3df6100 2023-10-16T14:30:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 10 client.6030 fetch_fsmap learned FSMap version 25 2023-10-16T14:30:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x559cd3cf8170 con 0x559cd3df6100 2023-10-16T14:30:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f12f57fa640 1 -- 172.21.15.29:0/144107148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131001f0d0 con 0x559cd3df6100 2023-10-16T14:30:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f12f57fa640 1 -- 172.21.15.29:0/144107148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f131003a020 con 0x559cd3df6100 2023-10-16T14:30:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 10 client.6030 fetch_fsmap finished waiting for FSMap version 25 2023-10-16T14:30:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 10 client.6030 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:30:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 20 client.6030 populate_metadata read hostname 'smithi029' 2023-10-16T14:30:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3dfb4b0 0x559cd3dfb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 4 client.6030 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:30:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.074+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 --> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559cd3d2e700 con 0x559cd3dfb4b0 2023-10-16T14:30:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.075+0000 7f1319c97640 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3dfb4b0 0x559cd3dfb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.075+0000 7f1319c97640 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3dfb4b0 0x559cd3dfb890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9c7ead500f8b2eb server_cookie=3853adbd9a18ee5a in_seq=0 out_seq=0 2023-10-16T14:30:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.075+0000 7f12f57fa640 10 client.6030 ms_handle_connect on v2:172.21.15.29:6834/1869070988 2023-10-16T14:30:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.075+0000 7f12f57fa640 1 -- 172.21.15.29:0/144107148 <== mds.0 v2:172.21.15.29:6834/1869070988 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559cd3d2e700 con 0x559cd3dfb4b0 2023-10-16T14:30:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.075+0000 7f12f57fa640 10 client.6030 handle_command_reply: tid=0 2023-10-16T14:30:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 client.6030 shutdown 2023-10-16T14:30:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 msgr2=0x559cd3e65ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 0x559cd3e65ca0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1310008010 tx=0x7f1310004730 comp rx=0 tx=0).stop 2023-10-16T14:30:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 shutdown_connections 2023-10-16T14:30:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:6834/1869070988,v1:172.21.15.29:6835/1869070988] conn(0x559cd3dfb4b0 0x559cd3dfb890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559cd3df7550 0x559cd3e0e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559cd3df7170 0x559cd3dfc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 --2- 172.21.15.29:0/144107148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559cd3df6100 0x559cd3e65ca0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f12f57fa640 0 client.6030 ms_handle_reset on v2:172.21.15.29:6834/1869070988 2023-10-16T14:30:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 >> 172.21.15.29:0/144107148 conn(0x559cd3dca5b0 msgr2=0x559cd3df9940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 shutdown_connections 2023-10-16T14:30:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 1 -- 172.21.15.29:0/144107148 wait complete. 2023-10-16T14:30:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.076+0000 7f131f367740 20 client.6030 trim_cache size 0 max 16384 2023-10-16T14:30:23.095 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:30:23.095 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:30:23.095 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:30:23.095 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 5874, 2023-10-16T14:30:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:30:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:30:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:30:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8281219169999998, 2023-10-16T14:30:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 25, 2023-10-16T14:30:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 65, 2023-10-16T14:30:23.097 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 65, 2023-10-16T14:30:23.097 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.589554543 2023-10-16T14:30:23.097 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:30:23.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f12f806af90 msgr2=0x7f12f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f12f806af90 0x7f12f806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1310010910 tx=0x7f1310008040 comp rx=0 tx=0).stop 2023-10-16T14:30:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 msgr2=0x7f1314145d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 0x7f1314145d60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f131406d2e0 tx=0x7f130c00b480 comp rx=0 tx=0).stop 2023-10-16T14:30:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 shutdown_connections 2023-10-16T14:30:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f12f806af90 0x7f12f806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141458b0 0x7f1314145d60 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13141595c0 0x7f1314159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 --2- 172.21.15.29:0/346739724 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13000047c0 0x7f131415db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 >> 172.21.15.29:0/346739724 conn(0x7f13140ce6a0 msgr2=0x7f131415d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 shutdown_connections 2023-10-16T14:30:23.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.077+0000 7f131b49a640 1 -- 172.21.15.29:0/346739724 wait complete. 2023-10-16T14:30:23.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:30:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.315+0000 7fca0b9b6640 1 Processor -- start 2023-10-16T14:30:23.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.316+0000 7fca0b9b6640 1 -- start start 2023-10-16T14:30:23.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.317+0000 7fca0b9b6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fca040cfdd0 0x7fca040b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.317+0000 7fca0b9b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca040b09c0 0x7fca0415d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.317+0000 7fca0b9b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca040cf940 con 0x7fca0406b8c0 2023-10-16T14:30:23.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.317+0000 7fca0b9b6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca040ce990 con 0x7fca040b09c0 2023-10-16T14:30:23.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.317+0000 7fca0b9b6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0415e290 con 0x7fca040cfdd0 2023-10-16T14:30:23.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca040b09c0 0x7fca0415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca040b09c0 0x7fca0415d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60942/0 (socket says 172.21.15.29:60942) 2023-10-16T14:30:23.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 -- 172.21.15.29:0/2948790078 learned_addr learned my addr 172.21.15.29:0/2948790078 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca08f51640 1 --2- 172.21.15.29:0/2948790078 >> v2:172.21.15.170:3300/0 conn(0x7fca040cfdd0 0x7fca040b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 -- 172.21.15.29:0/2948790078 >> v2:172.21.15.170:3300/0 conn(0x7fca040cfdd0 msgr2=0x7fca040b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 --2- 172.21.15.29:0/2948790078 >> v2:172.21.15.170:3300/0 conn(0x7fca040cfdd0 0x7fca040b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 -- 172.21.15.29:0/2948790078 >> v1:172.21.15.170:6789/0 conn(0x7fca0406b8c0 legacy=0x7fca0406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.318+0000 7fca09f53640 1 -- 172.21.15.29:0/2948790078 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040cf010 con 0x7fca040b09c0 2023-10-16T14:30:23.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.320+0000 7fca09f53640 1 --2- 172.21.15.29:0/2948790078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415d180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fca0406cfb0 tx=0x7fc9fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fb967cacbdde93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.320+0000 7fc9fbfff640 1 -- 172.21.15.29:0/2948790078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc02ca80 con 0x7fca040b09c0 2023-10-16T14:30:23.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.320+0000 7fc9fbfff640 1 -- 172.21.15.29:0/2948790078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc02cbe0 con 0x7fca040b09c0 2023-10-16T14:30:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.320+0000 7fc9fbfff640 1 -- 172.21.15.29:0/2948790078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0343f0 con 0x7fca040b09c0 2023-10-16T14:30:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 -- 172.21.15.29:0/2948790078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 msgr2=0x7fca0415d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/2948790078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415d180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fca0406cfb0 tx=0x7fc9fc02bee0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 -- 172.21.15.29:0/2948790078 shutdown_connections 2023-10-16T14:30:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/2948790078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415d180 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/2948790078 >> v2:172.21.15.170:3300/0 conn(0x7fca040cfdd0 0x7fca040b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 -- 172.21.15.29:0/2948790078 >> 172.21.15.29:0/2948790078 conn(0x7fca04099ea0 msgr2=0x7fca040d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 -- 172.21.15.29:0/2948790078 shutdown_connections 2023-10-16T14:30:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.321+0000 7fca0b9b6640 1 -- 172.21.15.29:0/2948790078 wait complete. 2023-10-16T14:30:23.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.323+0000 7fca0b9b6640 1 Processor -- start 2023-10-16T14:30:23.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.323+0000 7fca0b9b6640 1 -- start start 2023-10-16T14:30:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca0b9b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca0406b8c0 0x7fca0414a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca0b9b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca0b9b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca040cfdd0 0x7fca040cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca0b9b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0415e570 con 0x7fca040b09c0 2023-10-16T14:30:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca0b9b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0415f7e0 con 0x7fca040cfdd0 2023-10-16T14:30:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca0b9b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca040cde80 con 0x7fca0406b8c0 2023-10-16T14:30:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca08f51640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca08f51640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60952/0 (socket says 172.21.15.29:60952) 2023-10-16T14:30:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca08f51640 1 -- 172.21.15.29:0/4043758972 learned_addr learned my addr 172.21.15.29:0/4043758972 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.324+0000 7fca09752640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca0406b8c0 0x7fca0414a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca09f53640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca040cfdd0 0x7fca040cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca08f51640 1 -- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca0406b8c0 msgr2=0x7fca0414a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca08f51640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca0406b8c0 0x7fca0414a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca08f51640 1 -- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca040cfdd0 msgr2=0x7fca040cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca08f51640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca040cfdd0 0x7fca040cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca08f51640 1 -- 172.21.15.29:0/4043758972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc009b30 con 0x7fca040b09c0 2023-10-16T14:30:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca09f53640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca040cfdd0 0x7fca040cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.325+0000 7fca08f51640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415f2a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400c820 tx=0x7fc9f400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.326+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4004020 con 0x7fca040b09c0 2023-10-16T14:30:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.326+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f4004180 con 0x7fca040b09c0 2023-10-16T14:30:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.326+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0415e7c0 con 0x7fca040b09c0 2023-10-16T14:30:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.326+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4007830 con 0x7fca040b09c0 2023-10-16T14:30:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.326+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0415ed80 con 0x7fca040b09c0 2023-10-16T14:30:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.328+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0404e160 con 0x7fca040b09c0 2023-10-16T14:30:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.328+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9f4007990 con 0x7fca040b09c0 2023-10-16T14:30:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.330+0000 7fc9f9ffb640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc9e806af90 0x7fc9e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.330+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9f4089540 con 0x7fca040b09c0 2023-10-16T14:30:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.330+0000 7fca09752640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc9e806af90 0x7fc9e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.332+0000 7fca09752640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc9e806af90 0x7fc9e806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca040b08a0 tx=0x7fca00009040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.334+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9f40543d0 con 0x7fca040b09c0 2023-10-16T14:30:23.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.576+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fca040719c0 con 0x7fca040b09c0 2023-10-16T14:30:23.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.577+0000 7fc9f9ffb640 1 -- 172.21.15.29:0/4043758972 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc9f40579f0 con 0x7fca040b09c0 2023-10-16T14:30:23.581 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:30:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc9e806af90 msgr2=0x7fc9e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc9e806af90 0x7fc9e806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca040b08a0 tx=0x7fca00009040 comp rx=0 tx=0).stop 2023-10-16T14:30:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 msgr2=0x7fca0415f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:23.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415f2a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400c820 tx=0x7fc9f400cce0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 shutdown_connections 2023-10-16T14:30:23.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc9e806af90 0x7fc9e806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca040b09c0 0x7fca0415f2a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca040cfdd0 0x7fca040cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 --2- 172.21.15.29:0/4043758972 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca0406b8c0 0x7fca0414a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 >> 172.21.15.29:0/4043758972 conn(0x7fca04099ea0 msgr2=0x7fca040d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 shutdown_connections 2023-10-16T14:30:23.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:23.580+0000 7fca0b9b6640 1 -- 172.21.15.29:0/4043758972 wait complete. 2023-10-16T14:30:23.604 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:30:23.605 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:30:23.605 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:30:23.606 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:30:23.606 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:30:23.606 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:30:23.607 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:30:23.607 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:30:23.608 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:30:23.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:23.609 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:30:23.665 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:30:23.666 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:30:23.666 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:30:23.666 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.666 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:30:23.666 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.667 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:30:23.667 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:30:23.667 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:30:23.667 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:30:23.667 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:30:23.668 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:30:23.668 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:30:23.668 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:30:23.668 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:30:23.668 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:30:23.669 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41501sec preferred_lft 41501sec 2023-10-16T14:30:23.669 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:30:23.669 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.669 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:30:23.669 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:30:23.669 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:30:23.670 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:30:23.670 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:30:23.670 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:30:23.670 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.671 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:30:23.671 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.671 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:30:23.671 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:30:23.671 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:30:23.671 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.672 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:30:23.672 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:30:23.672 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:30:23.672 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:30:23.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:23.674 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:30:23.771 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:30:23.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:23.777 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:30:23.837 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:30:23.837 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:30:23.837 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:30:23.837 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:30:23.838 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:30:23.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:23.838 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:30:23.839 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:30:23.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:30:23.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:30:23.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:30:23.840 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:30:24.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:24.041 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:30:24.097 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:30:24.097 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:30:24.098 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:30:24.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:24.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:30:24.199 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:30:24.199 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:30:24.256 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:30:24.257 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:30:24.257 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:30:24.318 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:30:24.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:24.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:30:24.470 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:30:24.471 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:30:24.471 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:30:24.496 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:30:24.496 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:30:24.497 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:30:24.497 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:30:24.497 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:30:24.523 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:30:24.524 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:30:24.524 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:30:24.524 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:30:24.524 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:30:24.525 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:30:24.525 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:30:24.525 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:30:24.526 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:30:24.526 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:30:24.526 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:30:24.526 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:30:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:30:24.529 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:30:24.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:24.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:30:24.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:24.621 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:30:24.691 INFO:teuthology.orchestra.run.smithi029.stdout:client6039 2023-10-16T14:30:24.692 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:30:24.692 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:30:24.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:30:24.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.986+0000 7f5211e78640 1 Processor -- start 2023-10-16T14:30:24.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.986+0000 7f5211e78640 1 -- start start 2023-10-16T14:30:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f5211e78640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c06c040 0x7f520c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f5211e78640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f520c0ad970 0x7f520c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f5211e78640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f520c15ebc0 0x7f520c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:24.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f5211e78640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c06c960 con 0x7f520c0ad970 2023-10-16T14:30:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f5211e78640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c15b9d0 con 0x7f520c15ebc0 2023-10-16T14:30:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f5211e78640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c15bcb0 con 0x7f520c06c040 2023-10-16T14:30:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.988+0000 7f520bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f520c0ad970 0x7f520c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f520c0ad970 0x7f520c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59298/0 (socket says 172.21.15.29:59298) 2023-10-16T14:30:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 -- 172.21.15.29:0/1734180938 learned_addr learned my addr 172.21.15.29:0/1734180938 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f5210e76640 1 --2- 172.21.15.29:0/1734180938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c06c040 0x7f520c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f5211677640 1 --2- 172.21.15.29:0/1734180938 >> v2:172.21.15.170:3300/0 conn(0x7f520c15ebc0 0x7f520c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 -- 172.21.15.29:0/1734180938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c06c040 msgr2=0x7f520c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:24.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 --2- 172.21.15.29:0/1734180938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c06c040 0x7f520c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:24.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 -- 172.21.15.29:0/1734180938 >> v2:172.21.15.170:3300/0 conn(0x7f520c15ebc0 msgr2=0x7f520c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:24.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 --2- 172.21.15.29:0/1734180938 >> v2:172.21.15.170:3300/0 conn(0x7f520c15ebc0 0x7f520c15f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:24.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.989+0000 7f520bfff640 1 -- 172.21.15.29:0/1734180938 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c15bf90 con 0x7f520c0ad970 2023-10-16T14:30:24.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.991+0000 7f520bfff640 1 --2- 172.21.15.29:0/1734180938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c0ad970 0x7f520c15e680 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f51f8009870 tx=0x7f51f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5e16afa2d2d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.991+0000 7f520b7fe640 1 -- 172.21.15.29:0/1734180938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8031640 con 0x7f520c0ad970 2023-10-16T14:30:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.991+0000 7f520b7fe640 1 -- 172.21.15.29:0/1734180938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f80317a0 con 0x7f520c0ad970 2023-10-16T14:30:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.991+0000 7f520b7fe640 1 -- 172.21.15.29:0/1734180938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8031b30 con 0x7f520c0ad970 2023-10-16T14:30:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 -- 172.21.15.29:0/1734180938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c0ad970 msgr2=0x7f520c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 --2- 172.21.15.29:0/1734180938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c0ad970 0x7f520c15e680 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f51f8009870 tx=0x7f51f802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:30:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 -- 172.21.15.29:0/1734180938 shutdown_connections 2023-10-16T14:30:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 --2- 172.21.15.29:0/1734180938 >> v2:172.21.15.170:3300/0 conn(0x7f520c15ebc0 0x7f520c15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 --2- 172.21.15.29:0/1734180938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c0ad970 0x7f520c15e680 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 --2- 172.21.15.29:0/1734180938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c06c040 0x7f520c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 -- 172.21.15.29:0/1734180938 >> 172.21.15.29:0/1734180938 conn(0x7f520c0ce6a0 msgr2=0x7f520c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 -- 172.21.15.29:0/1734180938 shutdown_connections 2023-10-16T14:30:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.992+0000 7f5211e78640 1 -- 172.21.15.29:0/1734180938 wait complete. 2023-10-16T14:30:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.994+0000 7f5211e78640 1 Processor -- start 2023-10-16T14:30:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.994+0000 7f5211e78640 1 -- start start 2023-10-16T14:30:25.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5211e78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 0x7f520c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5211e78640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f520c0ad970 0x7f520c1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5211e78640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c15ebc0 0x7f520c0c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5211e78640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c0c8550 con 0x7f520c06c040 2023-10-16T14:30:25.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5211e78640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c0c7cb0 con 0x7f520c0ad970 2023-10-16T14:30:25.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5211e78640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c0c7f90 con 0x7f520c15ebc0 2023-10-16T14:30:25.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 0x7f520c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 0x7f520c0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59302/0 (socket says 172.21.15.29:59302) 2023-10-16T14:30:25.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 -- 172.21.15.29:0/1864752411 learned_addr learned my addr 172.21.15.29:0/1864752411 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:25.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 -- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c15ebc0 msgr2=0x7f520c0c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c15ebc0 0x7f520c0c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 -- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f520c0ad970 msgr2=0x7f520c1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f520c0ad970 0x7f520c1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.995+0000 7f5210e76640 1 -- 172.21.15.29:0/1864752411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f80094d0 con 0x7f520c06c040 2023-10-16T14:30:25.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.996+0000 7f5210e76640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 0x7f520c0c4b80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f520000d800 tx=0x7f520000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.996+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000096a0 con 0x7f520c06c040 2023-10-16T14:30:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.996+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5200009800 con 0x7f520c06c040 2023-10-16T14:30:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.996+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200009990 con 0x7f520c06c040 2023-10-16T14:30:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.996+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c0c8270 con 0x7f520c06c040 2023-10-16T14:30:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.996+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c155010 con 0x7f520c06c040 2023-10-16T14:30:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.997+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c06b330 con 0x7f520c06c040 2023-10-16T14:30:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.998+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5200009af0 con 0x7f520c06c040 2023-10-16T14:30:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.999+0000 7f52097fa640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f51f006af90 0x7f51f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:24.999+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5200089ca0 con 0x7f520c06c040 2023-10-16T14:30:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.000+0000 7f520bfff640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f51f006af90 0x7f51f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.001+0000 7f520bfff640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f51f006af90 0x7f51f006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f51f8036000 tx=0x7f51f8030000 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.004+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5200054ba0 con 0x7f520c06c040 2023-10-16T14:30:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.221+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 --> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f520c0cc550 con 0x7f51f006af90 2023-10-16T14:30:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.225+0000 7f52097fa640 1 -- 172.21.15.29:0/1864752411 <== mgr.5113 v2:172.21.15.170:6832/3423886595 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f51f8031910 con 0x7f51f006af90 2023-10-16T14:30:25.229 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:30:25.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:30:25.230 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:30:25.230 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:30:25.230 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:30:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f51f006af90 msgr2=0x7f51f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f51f006af90 0x7f51f006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f51f8036000 tx=0x7f51f8030000 comp rx=0 tx=0).stop 2023-10-16T14:30:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 msgr2=0x7f520c0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 0x7f520c0c4b80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f520000d800 tx=0x7f520000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 shutdown_connections 2023-10-16T14:30:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f51f006af90 0x7f51f006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f520c15ebc0 0x7f520c0c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f520c0ad970 0x7f520c1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 --2- 172.21.15.29:0/1864752411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f520c06c040 0x7f520c0c4b80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 >> 172.21.15.29:0/1864752411 conn(0x7f520c0ce6a0 msgr2=0x7f520c0c9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 shutdown_connections 2023-10-16T14:30:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.228+0000 7f5211e78640 1 -- 172.21.15.29:0/1864752411 wait complete. 2023-10-16T14:30:25.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:30:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.479+0000 7f29f88d9640 1 Processor -- start 2023-10-16T14:30:25.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.479+0000 7f29f88d9640 1 -- start start 2023-10-16T14:30:25.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.480+0000 7f29f88d9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29f40ca0a0 0x7f29f406c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.480+0000 7f29f88d9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f415d910 con 0x7f29f4070b60 2023-10-16T14:30:25.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f88d9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f415dbf0 con 0x7f29f4159950 2023-10-16T14:30:25.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f88d9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f415e490 con 0x7f29f40ca0a0 2023-10-16T14:30:25.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f29f4070b60 0x7f29f406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48022/0 (socket says 172.21.15.29:48022) 2023-10-16T14:30:25.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f259c640 1 -- 172.21.15.29:0/996695640 learned_addr learned my addr 172.21.15.29:0/996695640 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:25.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f359e640 1 --2- 172.21.15.29:0/996695640 >> v2:172.21.15.29:3300/0 conn(0x7f29f40ca0a0 0x7f29f406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f359e640 1 -- 172.21.15.29:0/996695640 >> v1:172.21.15.170:6789/0 conn(0x7f29f4159950 legacy=0x7f29f4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.481+0000 7f29f359e640 1 -- 172.21.15.29:0/996695640 >> v1:172.21.15.29:6789/0 conn(0x7f29f4070b60 legacy=0x7f29f406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.482+0000 7f29f359e640 1 -- 172.21.15.29:0/996695640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f415ded0 con 0x7f29f40ca0a0 2023-10-16T14:30:25.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.482+0000 7f29f1d9b640 1 -- 172.21.15.29:0/996695640 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469217184 0 0) 0x7f29f415d910 con 0x7f29f4070b60 2023-10-16T14:30:25.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.483+0000 7f29f359e640 1 --2- 172.21.15.29:0/996695640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f40ca0a0 0x7f29f406c810 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009d90 tx=0x7f29ec02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faac8092bbac36c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.483+0000 7f29f1d9b640 1 -- 172.21.15.29:0/996695640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec02cbd0 con 0x7f29f40ca0a0 2023-10-16T14:30:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.483+0000 7f29f1d9b640 1 -- 172.21.15.29:0/996695640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec02cd30 con 0x7f29f40ca0a0 2023-10-16T14:30:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.483+0000 7f29f1d9b640 1 -- 172.21.15.29:0/996695640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec02cea0 con 0x7f29f40ca0a0 2023-10-16T14:30:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 -- 172.21.15.29:0/996695640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f40ca0a0 msgr2=0x7f29f406c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 --2- 172.21.15.29:0/996695640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f40ca0a0 0x7f29f406c810 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009d90 tx=0x7f29ec02be20 comp rx=0 tx=0).stop 2023-10-16T14:30:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 -- 172.21.15.29:0/996695640 shutdown_connections 2023-10-16T14:30:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 --2- 172.21.15.29:0/996695640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f40ca0a0 0x7f29f406c810 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 -- 172.21.15.29:0/996695640 >> 172.21.15.29:0/996695640 conn(0x7f29f40ce6a0 msgr2=0x7f29f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 -- 172.21.15.29:0/996695640 shutdown_connections 2023-10-16T14:30:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.484+0000 7f29f88d9640 1 -- 172.21.15.29:0/996695640 wait complete. 2023-10-16T14:30:25.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.486+0000 7f29f88d9640 1 Processor -- start 2023-10-16T14:30:25.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.486+0000 7f29f88d9640 1 -- start start 2023-10-16T14:30:25.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f88d9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f4070b60 0x7f29f40c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f88d9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f40ca0a0 0x7f29f41f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f88d9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 0x7f29f40ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f88d9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40c8dc0 con 0x7f29f4159950 2023-10-16T14:30:25.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f88d9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40c7b90 con 0x7f29f40ca0a0 2023-10-16T14:30:25.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f88d9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f41f13a0 con 0x7f29f4070b60 2023-10-16T14:30:25.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 0x7f29f40ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 0x7f29f40ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59318/0 (socket says 172.21.15.29:59318) 2023-10-16T14:30:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f359e640 1 -- 172.21.15.29:0/3300981545 learned_addr learned my addr 172.21.15.29:0/3300981545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f259c640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f40ca0a0 0x7f29f41f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.487+0000 7f29f2d9d640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f4070b60 0x7f29f40c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f359e640 1 -- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f4070b60 msgr2=0x7f29f40c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f359e640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f4070b60 0x7f29f40c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f359e640 1 -- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f40ca0a0 msgr2=0x7f29f41f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:25.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f359e640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f40ca0a0 0x7f29f41f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:25.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f359e640 1 -- 172.21.15.29:0/3300981545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec009a70 con 0x7f29f4159950 2023-10-16T14:30:25.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f259c640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f40ca0a0 0x7f29f41f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:25.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f359e640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 0x7f29f40ae070 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009d90 tx=0x7f29ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:25.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec02cea0 con 0x7f29f4159950 2023-10-16T14:30:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec03e630 con 0x7f29f4159950 2023-10-16T14:30:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f41f10c0 con 0x7f29f4159950 2023-10-16T14:30:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.488+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec03e9e0 con 0x7f29f4159950 2023-10-16T14:30:25.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.489+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f40c80a0 con 0x7f29f4159950 2023-10-16T14:30:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.490+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f404e9b0 con 0x7f29f4159950 2023-10-16T14:30:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.491+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f29ec03b050 con 0x7f29f4159950 2023-10-16T14:30:25.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.492+0000 7f29db7fe640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f29e006af90 0x7f29e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:25.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.492+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29ec0ac090 con 0x7f29f4159950 2023-10-16T14:30:25.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.493+0000 7f29f2d9d640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f29e006af90 0x7f29e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:25.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.494+0000 7f29f2d9d640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f29e006af90 0x7f29e006d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29e4006fd0 tx=0x7f29e4008040 comp rx=0 tx=0).ready entity=mgr.5113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:25.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.497+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f29ec077020 con 0x7f29f4159950 2023-10-16T14:30:25.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:25.742+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f29f40b4620 con 0x7f29f4159950 2023-10-16T14:30:26.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.129+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f29ec07a640 con 0x7f29f4159950 2023-10-16T14:30:26.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.136+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f29e006af90 msgr2=0x7f29e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.136+0000 7f29f88d9640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f29e006af90 0x7f29e006d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29e4006fd0 tx=0x7f29e4008040 comp rx=0 tx=0).stop 2023-10-16T14:30:26.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.136+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 msgr2=0x7f29f40ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29db7fe640 1 -- 172.21.15.29:0/3300981545 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f29ec0773f0 con 0x7f29f4159950 2023-10-16T14:30:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 0x7f29f40ae070 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009d90 tx=0x7f29ec004a40 comp rx=0 tx=0).stop 2023-10-16T14:30:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 shutdown_connections 2023-10-16T14:30:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f29e006af90 0x7f29e006d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f4159950 0x7f29f40ae070 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29f40ca0a0 0x7f29f41f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 --2- 172.21.15.29:0/3300981545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29f4070b60 0x7f29f40c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 >> 172.21.15.29:0/3300981545 conn(0x7f29f40ce6a0 msgr2=0x7f29f406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:26.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.137+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 shutdown_connections 2023-10-16T14:30:26.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.138+0000 7f29f88d9640 1 -- 172.21.15.29:0/3300981545 wait complete. 2023-10-16T14:30:26.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:30:26.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.368+0000 7f77aa809640 1 Processor -- start 2023-10-16T14:30:26.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.368+0000 7f77aa809640 1 -- start start 2023-10-16T14:30:26.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77aa809640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77a406b8c0 0x7f77a406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77aa809640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f77a415d130 0x7f77a40cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77aa809640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a40cd390 con 0x7f77a406b1a0 2023-10-16T14:30:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77aa809640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a406c1e0 con 0x7f77a406b8c0 2023-10-16T14:30:26.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77aa809640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a40cc2f0 con 0x7f77a415d130 2023-10-16T14:30:26.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77a3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77a406b8c0 0x7f77a406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77a8da6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f77a406b1a0 0x7f77a40cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48036/0 (socket says 172.21.15.29:48036) 2023-10-16T14:30:26.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f77a8da6640 1 -- 172.21.15.29:0/2165080076 learned_addr learned my addr 172.21.15.29:0/2165080076 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.370+0000 7f779bfff640 1 --2- 172.21.15.29:0/2165080076 >> v2:172.21.15.170:3300/0 conn(0x7f77a415d130 0x7f77a40cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.371+0000 7f77a3fff640 1 -- 172.21.15.29:0/2165080076 >> v2:172.21.15.170:3300/0 conn(0x7f77a415d130 msgr2=0x7f77a40cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.371+0000 7f77a3fff640 1 --2- 172.21.15.29:0/2165080076 >> v2:172.21.15.170:3300/0 conn(0x7f77a415d130 0x7f77a40cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.371+0000 7f77a3fff640 1 -- 172.21.15.29:0/2165080076 >> v1:172.21.15.29:6789/0 conn(0x7f77a406b1a0 legacy=0x7f77a40cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.371+0000 7f77a3fff640 1 -- 172.21.15.29:0/2165080076 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a40cc5d0 con 0x7f77a406b8c0 2023-10-16T14:30:26.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.371+0000 7f77a37fe640 1 -- 172.21.15.29:0/2165080076 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46680450 0 0) 0x7f77a40cd390 con 0x7f77a406b1a0 2023-10-16T14:30:26.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.372+0000 7f77a3fff640 1 --2- 172.21.15.29:0/2165080076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a406b8c0 0x7f77a406bca0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f77a4073d50 tx=0x7f779402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7de353c7ba2ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.372+0000 7f77a37fe640 1 -- 172.21.15.29:0/2165080076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794031830 con 0x7f77a406b8c0 2023-10-16T14:30:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.372+0000 7f77a37fe640 1 -- 172.21.15.29:0/2165080076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794031990 con 0x7f77a406b8c0 2023-10-16T14:30:26.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.372+0000 7f77a37fe640 1 -- 172.21.15.29:0/2165080076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794031d20 con 0x7f77a406b8c0 2023-10-16T14:30:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 -- 172.21.15.29:0/2165080076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a406b8c0 msgr2=0x7f77a406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 --2- 172.21.15.29:0/2165080076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a406b8c0 0x7f77a406bca0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f77a4073d50 tx=0x7f779402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 -- 172.21.15.29:0/2165080076 shutdown_connections 2023-10-16T14:30:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 --2- 172.21.15.29:0/2165080076 >> v2:172.21.15.170:3300/0 conn(0x7f77a415d130 0x7f77a40cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 --2- 172.21.15.29:0/2165080076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a406b8c0 0x7f77a406bca0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 -- 172.21.15.29:0/2165080076 >> 172.21.15.29:0/2165080076 conn(0x7f77a40d0ad0 msgr2=0x7f77a40d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 -- 172.21.15.29:0/2165080076 shutdown_connections 2023-10-16T14:30:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.373+0000 7f77aa809640 1 -- 172.21.15.29:0/2165080076 wait complete. 2023-10-16T14:30:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.375+0000 7f77aa809640 1 Processor -- start 2023-10-16T14:30:26.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.375+0000 7f77aa809640 1 -- start start 2023-10-16T14:30:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77aa809640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77a406b1a0 0x7f77a409b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77aa809640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77a406b8c0 0x7f77a414eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77aa809640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 0x7f77a414f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77aa809640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40c96f0 con 0x7f77a415d130 2023-10-16T14:30:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77aa809640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a414f5a0 con 0x7f77a406b8c0 2023-10-16T14:30:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77aa809640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40c8a70 con 0x7f77a406b1a0 2023-10-16T14:30:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77a8da6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 0x7f77a414f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77a8da6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 0x7f77a414f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59346/0 (socket says 172.21.15.29:59346) 2023-10-16T14:30:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.376+0000 7f77a8da6640 1 -- 172.21.15.29:0/2480260346 learned_addr learned my addr 172.21.15.29:0/2480260346 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f779bfff640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77a406b8c0 0x7f77a414eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a3fff640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77a406b1a0 0x7f77a409b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a8da6640 1 -- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77a406b1a0 msgr2=0x7f77a409b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a8da6640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77a406b1a0 0x7f77a409b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a8da6640 1 -- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77a406b8c0 msgr2=0x7f77a414eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a8da6640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77a406b8c0 0x7f77a414eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a8da6640 1 -- 172.21.15.29:0/2480260346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77940093a0 con 0x7f77a415d130 2023-10-16T14:30:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a8da6640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 0x7f77a414f060 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f779c00c7e0 tx=0x7f779c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.377+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c00a640 con 0x7f77a415d130 2023-10-16T14:30:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.378+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c00a7c0 con 0x7f77a415d130 2023-10-16T14:30:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.378+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c00ab70 con 0x7f77a415d130 2023-10-16T14:30:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.378+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a40c8d50 con 0x7f77a415d130 2023-10-16T14:30:26.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.378+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a40c9310 con 0x7f77a415d130 2023-10-16T14:30:26.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.379+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a40c9830 con 0x7f77a415d130 2023-10-16T14:30:26.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.380+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f779c00acd0 con 0x7f77a415d130 2023-10-16T14:30:26.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.381+0000 7f77a17fa640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 0x7f778406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.381+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f779c088e70 con 0x7f77a415d130 2023-10-16T14:30:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.381+0000 7f77a3fff640 1 -- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 msgr2=0x7f778406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.381+0000 7f77a3fff640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 0x7f778406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:30:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.386+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f77a40c9830 con 0x7f77a415d130 2023-10-16T14:30:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.581+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f77a40c9830 con 0x7f77a415d130 2023-10-16T14:30:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.582+0000 7f77a3fff640 1 -- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 msgr2=0x7f778406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:26.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.582+0000 7f77a3fff640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 0x7f778406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:30:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.586+0000 7f77a17fa640 1 -- 172.21.15.29:0/2480260346 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f77a40c9830 con 0x7f77a415d130 2023-10-16T14:30:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 msgr2=0x7f778406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:30:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 0x7f778406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 msgr2=0x7f77a414f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 0x7f77a414f060 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f779c00c7e0 tx=0x7f779c00cca0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 shutdown_connections 2023-10-16T14:30:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7f778406afe0 0x7f778406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77a415d130 0x7f77a414f060 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77a406b8c0 0x7f77a414eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 --2- 172.21.15.29:0/2480260346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77a406b1a0 0x7f77a409b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 >> 172.21.15.29:0/2480260346 conn(0x7f77a40d0ad0 msgr2=0x7f77a415c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:26.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 shutdown_connections 2023-10-16T14:30:26.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.589+0000 7f77aa809640 1 -- 172.21.15.29:0/2480260346 wait complete. 2023-10-16T14:30:26.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:30:26.655 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:26.651+0000 7f7b01632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:30:26.707 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:26.704+0000 7f7b01632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:30:26.712 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:26.709+0000 7f989e245040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:30:26.773 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:26.769+0000 7f989e245040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:30:26.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.828+0000 7fc2ee461640 1 Processor -- start 2023-10-16T14:30:26.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.828+0000 7fc2ee461640 1 -- start start 2023-10-16T14:30:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2ee461640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc2e80b3420 0x7fc2e80b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2ee461640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 0x7fc2e80b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2ee461640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e80d3980 con 0x7fc2e80cf710 2023-10-16T14:30:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2ee461640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80d3c60 con 0x7fc2e80b3420 2023-10-16T14:30:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2ee461640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80ce630 con 0x7fc2e80cfaf0 2023-10-16T14:30:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2e7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc2e80cf710 0x7fc2e80b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48044/0 (socket says 172.21.15.29:48044) 2023-10-16T14:30:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.830+0000 7fc2e7fff640 1 -- 172.21.15.29:0/3701194497 learned_addr learned my addr 172.21.15.29:0/3701194497 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:30:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2ec9fe640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 0x7fc2e80b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e77fe640 1 --2- 172.21.15.29:0/3701194497 >> v2:172.21.15.170:3300/0 conn(0x7fc2e80b3420 0x7fc2e80b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410708622 0 0) 0x7fc2e80d3980 con 0x7fc2e80cf710 2023-10-16T14:30:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d0003520 con 0x7fc2e80cf710 2023-10-16T14:30:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4043983050 0 0) 0x7fc2d0003520 con 0x7fc2e80cf710 2023-10-16T14:30:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 msgr2=0x7fc2e80b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 0x7fc2e80b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 >> v2:172.21.15.170:3300/0 conn(0x7fc2e80b3420 msgr2=0x7fc2e80b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 --2- 172.21.15.29:0/3701194497 >> v2:172.21.15.170:3300/0 conn(0x7fc2e80b3420 0x7fc2e80b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80ce910 con 0x7fc2e80cf710 2023-10-16T14:30:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc2d80017f0 con 0x7fc2e80cf710 2023-10-16T14:30:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 >> v1:172.21.15.29:6789/0 conn(0x7fc2e80cf710 legacy=0x7fc2e80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2d0003970 0x7fc2d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.831+0000 7fc2e6ffd640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 0x7fc2d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2e6ffd640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80d3c60 con 0x7fc2d0004d00 2023-10-16T14:30:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80ce630 con 0x7fc2d0003970 2023-10-16T14:30:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d000ba00 con 0x7fc2e80cfaf0 2023-10-16T14:30:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2ec9fe640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2e7fff640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2d0003970 0x7fc2d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2e77fe640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 0x7fc2d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2ec9fe640 1 -- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 msgr2=0x7fc2d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2ec9fe640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2e80cfaf0 0x7fc2d00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2ec9fe640 1 -- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2d0003970 msgr2=0x7fc2d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2ec9fe640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2d0003970 0x7fc2d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.832+0000 7fc2ec9fe640 1 -- 172.21.15.29:0/3701194497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0003520 con 0x7fc2d0004d00 2023-10-16T14:30:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.833+0000 7fc2ec9fe640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009fc0 tx=0x7fc2d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c43b2d7d052593 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.833+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e002cc20 con 0x7fc2d0004d00 2023-10-16T14:30:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000f040 con 0x7fc2d0004d00 2023-10-16T14:30:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2e6ffd640 1 -- 172.21.15.29:0/3701194497 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0034440 con 0x7fc2d0004d00 2023-10-16T14:30:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ee461640 1 -- 172.21.15.29:0/3701194497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d0004d00 msgr2=0x7fc2d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ee461640 1 --2- 172.21.15.29:0/3701194497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009fc0 tx=0x7fc2d00057f0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ec9fe640 1 -- 172.21.15.29:0/3701194497 reap_dead start 2023-10-16T14:30:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ee461640 1 -- 172.21.15.29:0/3701194497 shutdown_connections 2023-10-16T14:30:26.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ee461640 1 -- 172.21.15.29:0/3701194497 >> 172.21.15.29:0/3701194497 conn(0x7fc2e80d2cc0 msgr2=0x7fc2e80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:26.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ee461640 1 -- 172.21.15.29:0/3701194497 shutdown_connections 2023-10-16T14:30:26.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.834+0000 7fc2ee461640 1 -- 172.21.15.29:0/3701194497 wait complete. 2023-10-16T14:30:26.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.836+0000 7fc2ee461640 1 Processor -- start 2023-10-16T14:30:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.836+0000 7fc2ee461640 1 -- start start 2023-10-16T14:30:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2ee461640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2d0004d00 0x7fc2e8156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2ee461640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 0x7fc2e80b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2ee461640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2e80b2570 0x7fc2e806b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2ee461640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81f3300 con 0x7fc2e8156870 2023-10-16T14:30:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2ee461640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81565a0 con 0x7fc2e80b2570 2023-10-16T14:30:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2ee461640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81f3720 con 0x7fc2d0004d00 2023-10-16T14:30:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.837+0000 7fc2e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 0x7fc2e80b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 0x7fc2e80b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59370/0 (socket says 172.21.15.29:59370) 2023-10-16T14:30:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 -- 172.21.15.29:0/4034127517 learned_addr learned my addr 172.21.15.29:0/4034127517 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2ec9fe640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2e80b2570 0x7fc2e806b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e7fff640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2d0004d00 0x7fc2e8156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 -- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2d0004d00 msgr2=0x7fc2e8156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2d0004d00 0x7fc2e8156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 -- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2e80b2570 msgr2=0x7fc2e806b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2e80b2570 0x7fc2e806b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 -- 172.21.15.29:0/4034127517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e81f2680 con 0x7fc2e8156870 2023-10-16T14:30:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.838+0000 7fc2e77fe640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 0x7fc2e80b2030 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0108e0 tx=0x7fc2dc010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.839+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc00eb60 con 0x7fc2e8156870 2023-10-16T14:30:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.839+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc00ecc0 con 0x7fc2e8156870 2023-10-16T14:30:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.839+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e81f2960 con 0x7fc2e8156870 2023-10-16T14:30:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.839+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc01c400 con 0x7fc2e8156870 2023-10-16T14:30:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.839+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e81f2f20 con 0x7fc2e8156870 2023-10-16T14:30:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.840+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e81f3440 con 0x7fc2e8156870 2023-10-16T14:30:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.841+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc2dc01c560 con 0x7fc2e8156870 2023-10-16T14:30:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.842+0000 7fc2e4ff9640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 0x7fc2d406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.842+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2dc089f30 con 0x7fc2e8156870 2023-10-16T14:30:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.842+0000 7fc2e7fff640 1 -- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 msgr2=0x7fc2d406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.842+0000 7fc2e7fff640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 0x7fc2d406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:30:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:26.846+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2e81f3440 con 0x7fc2e8156870 2023-10-16T14:30:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.038+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc2e81f3440 con 0x7fc2e8156870 2023-10-16T14:30:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.043+0000 7fc2e7fff640 1 -- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 msgr2=0x7fc2d406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.043+0000 7fc2e7fff640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 0x7fc2d406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:30:27.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.043+0000 7fc2e4ff9640 1 -- 172.21.15.29:0/4034127517 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc2e81f3440 con 0x7fc2e8156870 2023-10-16T14:30:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 msgr2=0x7fc2d406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:30:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 0x7fc2d406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 msgr2=0x7fc2e80b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 0x7fc2e80b2030 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0108e0 tx=0x7fc2dc010da0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 shutdown_connections 2023-10-16T14:30:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fc2d406afe0 0x7fc2d406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2e8156870 0x7fc2e80b2030 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2e80b2570 0x7fc2e806b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 --2- 172.21.15.29:0/4034127517 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2d0004d00 0x7fc2e8156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 >> 172.21.15.29:0/4034127517 conn(0x7fc2e80d2cc0 msgr2=0x7fc2e80b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 shutdown_connections 2023-10-16T14:30:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.046+0000 7fc2ee461640 1 -- 172.21.15.29:0/4034127517 wait complete. 2023-10-16T14:30:27.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:27.069 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:30:27.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T14:30:27.157 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.153+0000 7f7b01632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:30:27.210 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.206+0000 7f7b01632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:30:27.258 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.254+0000 7f989e245040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:30:27.260 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.256+0000 7f7b01632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:30:27.314 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.310+0000 7f989e245040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:30:27.320 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.317+0000 7f7b01632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:30:27.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.343+0000 7fd1aef97640 1 Processor -- start 2023-10-16T14:30:27.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.344+0000 7fd1aef97640 1 -- start start 2023-10-16T14:30:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1aef97640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1aef97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1aef97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1aef97640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a806b1e0 con 0x7fd1a806cc80 2023-10-16T14:30:27.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1aef97640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a80d0890 con 0x7fd1a815d330 2023-10-16T14:30:27.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1aef97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80d05b0 con 0x7fd1a806c7d0 2023-10-16T14:30:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1ad534640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1ad534640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59382/0 (socket says 172.21.15.29:59382) 2023-10-16T14:30:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1ad534640 1 -- 172.21.15.29:0/4120090927 learned_addr learned my addr 172.21.15.29:0/4120090927 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1a7fff640 1 --2- 172.21.15.29:0/4120090927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.346+0000 7fd1acd33640 1 --2- 172.21.15.29:0/4120090927 >> v2:172.21.15.170:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.347+0000 7fd1ad534640 1 -- 172.21.15.29:0/4120090927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806c7d0 msgr2=0x7fd1a8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.347+0000 7fd1ad534640 1 --2- 172.21.15.29:0/4120090927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.347+0000 7fd1ad534640 1 -- 172.21.15.29:0/4120090927 >> v2:172.21.15.170:3300/0 conn(0x7fd1a815d330 msgr2=0x7fd1a81618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.347+0000 7fd1ad534640 1 --2- 172.21.15.29:0/4120090927 >> v2:172.21.15.170:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.347+0000 7fd1ad534640 1 -- 172.21.15.29:0/4120090927 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8160dd0 con 0x7fd1a806cc80 2023-10-16T14:30:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.348+0000 7fd1ad534640 1 --2- 172.21.15.29:0/4120090927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80744f0 tx=0x7fd1a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693d945a6eb8e5db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.348+0000 7fd1a77fe640 1 -- 172.21.15.29:0/4120090927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a002c9e0 con 0x7fd1a806cc80 2023-10-16T14:30:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.348+0000 7fd1a77fe640 1 -- 172.21.15.29:0/4120090927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a002cb40 con 0x7fd1a806cc80 2023-10-16T14:30:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.348+0000 7fd1a77fe640 1 -- 172.21.15.29:0/4120090927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00343f0 con 0x7fd1a806cc80 2023-10-16T14:30:27.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 -- 172.21.15.29:0/4120090927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a806cc80 msgr2=0x7fd1a8065ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:27.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 --2- 172.21.15.29:0/4120090927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80744f0 tx=0x7fd1a002be40 comp rx=0 tx=0).stop 2023-10-16T14:30:27.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 -- 172.21.15.29:0/4120090927 shutdown_connections 2023-10-16T14:30:27.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 --2- 172.21.15.29:0/4120090927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 --2- 172.21.15.29:0/4120090927 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 --2- 172.21.15.29:0/4120090927 >> v2:172.21.15.170:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 -- 172.21.15.29:0/4120090927 >> 172.21.15.29:0/4120090927 conn(0x7fd1a80d0ab0 msgr2=0x7fd1a80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:27.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 -- 172.21.15.29:0/4120090927 shutdown_connections 2023-10-16T14:30:27.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.349+0000 7fd1aef97640 1 -- 172.21.15.29:0/4120090927 wait complete. 2023-10-16T14:30:27.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.352+0000 7fd1aef97640 1 Processor -- start 2023-10-16T14:30:27.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.352+0000 7fd1aef97640 1 -- start start 2023-10-16T14:30:27.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1aef97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1a806c7d0 0x7fd1a8160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1aef97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806cc80 0x7fd1a80c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1aef97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a815d330 0x7fd1a815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1aef97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80ca240 con 0x7fd1a815d330 2023-10-16T14:30:27.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1aef97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80c5a10 con 0x7fd1a806c7d0 2023-10-16T14:30:27.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1aef97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8160640 con 0x7fd1a806cc80 2023-10-16T14:30:27.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1ad534640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a815d330 0x7fd1a815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:27.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1ad534640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a815d330 0x7fd1a815deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59398/0 (socket says 172.21.15.29:59398) 2023-10-16T14:30:27.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1ad534640 1 -- 172.21.15.29:0/3844411516 learned_addr learned my addr 172.21.15.29:0/3844411516 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:27.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.354+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1a806c7d0 0x7fd1a8160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:27.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1a7fff640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806cc80 0x7fd1a80c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:27.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1ad534640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806cc80 msgr2=0x7fd1a80c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:27.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1ad534640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd1a806cc80 0x7fd1a80c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1ad534640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1a806c7d0 msgr2=0x7fd1a8160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:27.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1ad534640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1a806c7d0 0x7fd1a8160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1ad534640 1 -- 172.21.15.29:0/3844411516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a0009b30 con 0x7fd1a815d330 2023-10-16T14:30:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1ad534640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a815d330 0x7fd1a815deb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0036040 tx=0x7fd1a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0009510 con 0x7fd1a815d330 2023-10-16T14:30:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a003e450 con 0x7fd1a815d330 2023-10-16T14:30:27.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a80c50e0 con 0x7fd1a815d330 2023-10-16T14:30:27.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.355+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a003e800 con 0x7fd1a815d330 2023-10-16T14:30:27.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.356+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a80c56a0 con 0x7fd1a815d330 2023-10-16T14:30:27.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.357+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a804e160 con 0x7fd1a815d330 2023-10-16T14:30:27.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.358+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fd1a003e960 con 0x7fd1a815d330 2023-10-16T14:30:27.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.358+0000 7fd1a57fa640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 0x7fd18806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:27.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.359+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1a00ac640 con 0x7fd1a815d330 2023-10-16T14:30:27.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.359+0000 7fd1acd33640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 msgr2=0x7fd18806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:27.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.359+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 0x7fd18806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:30:27.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.363+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd1a00774c0 con 0x7fd1a815d330 2023-10-16T14:30:27.371 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.368+0000 7f7b01632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:30:27.372 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.368+0000 7f989e245040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:30:27.436 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.433+0000 7f989e245040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:30:27.481 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.478+0000 7f7b01632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:30:27.488 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.485+0000 7f989e245040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:30:27.543 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.540+0000 7f7b01632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:30:27.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.560+0000 7fd1acd33640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 msgr2=0x7fd18806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:27.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.560+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 0x7fd18806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:30:27.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.577+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 --> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1a8161390 con 0x7fd18806b010 2023-10-16T14:30:27.592 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.589+0000 7f7b01632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:30:27.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.601+0000 7f989e245040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:30:27.650 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.646+0000 7f7b01632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:30:27.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.667+0000 7f989e245040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:30:27.699 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:27.696+0000 7f7b01632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:30:27.722 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.719+0000 7f989e245040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:30:27.782 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.778+0000 7f989e245040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:30:27.834 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:27.831+0000 7f989e245040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:30:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.961+0000 7fd1acd33640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 msgr2=0x7fd18806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:27.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:27.961+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 0x7fd18806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:30:28.078 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.075+0000 7f7b01632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:30:28.236 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.232+0000 7f989e245040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:30:28.240 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.236+0000 7f7b01632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:30:28.347 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.343+0000 7f7b01632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:30:28.396 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.393+0000 7f7b01632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:30:28.404 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.400+0000 7f989e245040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:30:28.502 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.499+0000 7f7b01632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:30:28.516 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.513+0000 7f989e245040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:30:28.569 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.565+0000 7f989e245040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:30:28.608 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.604+0000 7f7b01632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:30:28.681 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.678+0000 7f989e245040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:30:28.764 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.760+0000 7f7b01632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:30:28.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:28.762+0000 7fd1acd33640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 msgr2=0x7fd18806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3423886595 2023-10-16T14:30:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:28.762+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 0x7fd18806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:30:28.794 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.790+0000 7f989e245040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:30:28.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.808+0000 7f7b01632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:30:28.931 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:28.927+0000 7f7b01632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:30:28.959 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:28.955+0000 7f989e245040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:30:29.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:29.008+0000 7f989e245040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:30:29.137 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:29.133+0000 7f989e245040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:30:29.749 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:29.745+0000 7f7b01632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:30:30.012 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:30.008+0000 7f989e245040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:30:30.044 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.040+0000 7f7b01632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:30:30.100 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.097+0000 7f7b01632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:30:30.216 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.212+0000 7f7b01632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:30:30.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:30.237+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 20) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fd1a00aaac0 con 0x7fd1a815d330 2023-10-16T14:30:30.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:30.237+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 msgr2=0x7fd18806d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:30:30.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:30.237+0000 7fd1a57fa640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/3423886595,v1:172.21.15.170:6833/3423886595] conn(0x7fd18806b010 0x7fd18806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:30.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:30.333+0000 7f989e245040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:30:30.399 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:30.395+0000 7f989e245040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:30:30.416 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.413+0000 7f7a7d188640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.416 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.413+0000 7f7a7d188640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.417 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.413+0000 7f7a7d188640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.417 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.413+0000 7f7a7d188640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.417 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.413+0000 7f7a7d188640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.417 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.414+0000 7f7a7f9cd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.414+0000 7f7a7f9cd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.414+0000 7f7a7f9cd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.414+0000 7f7a7f9cd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:30:30.414+0000 7f7a7f9cd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:30:30.521 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:30:30.517+0000 7f989e245040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:30:31.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.248+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 21) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd1a00810b0 con 0x7fd1a815d330 2023-10-16T14:30:31.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.249+0000 7fd1a57fa640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd188035910 0x7fd188074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.249+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 --> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd188035cf0 con 0x7fd188035910 2023-10-16T14:30:31.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.250+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd188035910 0x7fd188074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.255+0000 7fd1acd33640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd188035910 0x7fd188074ed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd198006f30 tx=0x7fd198008040 comp rx=0 tx=0).ready entity=mgr.5054 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:31.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.421+0000 7fd1a57fa640 1 -- 172.21.15.29:0/3844411516 <== mgr.5054 v2:172.21.15.170:6832/636880041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd188035cf0 con 0x7fd188035910 2023-10-16T14:30:31.425 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-16T14:30:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.424+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd188035910 msgr2=0x7fd188074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.424+0000 7fd1aef97640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd188035910 0x7fd188074ed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd198006f30 tx=0x7fd198008040 comp rx=0 tx=0).stop 2023-10-16T14:30:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.424+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a815d330 msgr2=0x7fd1a815deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.424+0000 7fd1aef97640 1 --2- 172.21.15.29:0/3844411516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a815d330 0x7fd1a815deb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0036040 tx=0x7fd1a0004a40 comp rx=0 tx=0).stop 2023-10-16T14:30:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.424+0000 7fd1ad534640 1 -- 172.21.15.29:0/3844411516 reap_dead start 2023-10-16T14:30:31.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.425+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 shutdown_connections 2023-10-16T14:30:31.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.425+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 >> 172.21.15.29:0/3844411516 conn(0x7fd1a80d0ab0 msgr2=0x7fd1a8065ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:31.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.425+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 shutdown_connections 2023-10-16T14:30:31.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.425+0000 7fd1aef97640 1 -- 172.21.15.29:0/3844411516 wait complete. 2023-10-16T14:30:31.450 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T14:30:31.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-10-16T14:30:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.678+0000 7fa6eda09640 1 Processor -- start 2023-10-16T14:30:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.679+0000 7fa6eda09640 1 -- start start 2023-10-16T14:30:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.680+0000 7fa6eda09640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e806a4b0 0x7fa6e80ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.680+0000 7fa6eda09640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6e80c9ca0 0x7fa6e80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.680+0000 7fa6eda09640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e80cb030 con 0x7fa6e815ad70 2023-10-16T14:30:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.680+0000 7fa6eda09640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e80cae30 con 0x7fa6e80c9ca0 2023-10-16T14:30:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.680+0000 7fa6eda09640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80c91c0 con 0x7fa6e806a4b0 2023-10-16T14:30:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6e80c9ca0 0x7fa6e80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6e80c9ca0 0x7fa6e80ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59408/0 (socket says 172.21.15.29:59408) 2023-10-16T14:30:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 -- 172.21.15.29:0/2994552209 learned_addr learned my addr 172.21.15.29:0/2994552209 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:31.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e67fc640 1 --2- 172.21.15.29:0/2994552209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e806a4b0 0x7fa6e80ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 -- 172.21.15.29:0/2994552209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e806a4b0 msgr2=0x7fa6e80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 --2- 172.21.15.29:0/2994552209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e806a4b0 0x7fa6e80ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 -- 172.21.15.29:0/2994552209 >> v1:172.21.15.170:6789/0 conn(0x7fa6e815ad70 legacy=0x7fa6e8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.681+0000 7fa6e77fe640 1 -- 172.21.15.29:0/2994552209 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80c94a0 con 0x7fa6e80c9ca0 2023-10-16T14:30:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.682+0000 7fa6e77fe640 1 --2- 172.21.15.29:0/2994552209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e80ca8f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009870 tx=0x7fa6e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fedb476de362ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.683+0000 7fa6e5ffb640 1 -- 172.21.15.29:0/2994552209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0030840 con 0x7fa6e80c9ca0 2023-10-16T14:30:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.683+0000 7fa6e5ffb640 1 -- 172.21.15.29:0/2994552209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e00309a0 con 0x7fa6e80c9ca0 2023-10-16T14:30:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.683+0000 7fa6e5ffb640 1 -- 172.21.15.29:0/2994552209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0030d30 con 0x7fa6e80c9ca0 2023-10-16T14:30:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.683+0000 7fa6eda09640 1 -- 172.21.15.29:0/2994552209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e80c9ca0 msgr2=0x7fa6e80ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.683+0000 7fa6eda09640 1 --2- 172.21.15.29:0/2994552209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e80ca8f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009870 tx=0x7fa6e002b980 comp rx=0 tx=0).stop 2023-10-16T14:30:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.684+0000 7fa6eda09640 1 -- 172.21.15.29:0/2994552209 shutdown_connections 2023-10-16T14:30:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.684+0000 7fa6eda09640 1 --2- 172.21.15.29:0/2994552209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e80ca8f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.684+0000 7fa6eda09640 1 --2- 172.21.15.29:0/2994552209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e806a4b0 0x7fa6e80ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.684+0000 7fa6eda09640 1 -- 172.21.15.29:0/2994552209 >> 172.21.15.29:0/2994552209 conn(0x7fa6e80ce9b0 msgr2=0x7fa6e80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.684+0000 7fa6eda09640 1 -- 172.21.15.29:0/2994552209 shutdown_connections 2023-10-16T14:30:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.684+0000 7fa6eda09640 1 -- 172.21.15.29:0/2994552209 wait complete. 2023-10-16T14:30:31.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.685+0000 7fa6eda09640 1 Processor -- start 2023-10-16T14:30:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 -- start start 2023-10-16T14:30:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 0x7fa6e81efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e81f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e815ad70 0x7fa6e80c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80c7400 con 0x7fa6e806a4b0 2023-10-16T14:30:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80c76e0 con 0x7fa6e80c9ca0 2023-10-16T14:30:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.686+0000 7fa6eda09640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80c79c0 con 0x7fa6e815ad70 2023-10-16T14:30:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 0x7fa6e81efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 0x7fa6e81efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59410/0 (socket says 172.21.15.29:59410) 2023-10-16T14:30:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 -- 172.21.15.29:0/4056931340 learned_addr learned my addr 172.21.15.29:0/4056931340 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e67fc640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e81f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e77fe640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e815ad70 0x7fa6e80c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 -- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e815ad70 msgr2=0x7fa6e80c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e815ad70 0x7fa6e80c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 -- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6e80c9ca0 msgr2=0x7fa6e81f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e81f14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e6ffd640 1 -- 172.21.15.29:0/4056931340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00094d0 con 0x7fa6e806a4b0 2023-10-16T14:30:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.687+0000 7fa6e77fe640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e815ad70 0x7fa6e80c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:30:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.688+0000 7fa6e6ffd640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 0x7fa6e81efe00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c730 tx=0x7fa6d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.688+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8003b70 con 0x7fa6e806a4b0 2023-10-16T14:30:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.688+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d8003cf0 con 0x7fa6e806a4b0 2023-10-16T14:30:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.688+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8003ea0 con 0x7fa6e806a4b0 2023-10-16T14:30:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.688+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e80a8520 con 0x7fa6e806a4b0 2023-10-16T14:30:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.688+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e81f1a10 con 0x7fa6e806a4b0 2023-10-16T14:30:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.689+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e804e160 con 0x7fa6e806a4b0 2023-10-16T14:30:31.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.690+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa6d800aa90 con 0x7fa6e806a4b0 2023-10-16T14:30:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.692+0000 7fa6dffff640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fa6c806ad10 0x7fa6c806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.692+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa6d80892f0 con 0x7fa6e806a4b0 2023-10-16T14:30:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.692+0000 7fa6e67fc640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fa6c806ad10 0x7fa6c806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:31.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.694+0000 7fa6e67fc640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fa6c806ad10 0x7fa6c806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa6e80a9f80 tx=0x7fa6d0008040 comp rx=0 tx=0).ready entity=mgr.5054 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:31.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.696+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa6d80544b0 con 0x7fa6e806a4b0 2023-10-16T14:30:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:31.902+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 --> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6e80c9780 con 0x7fa6c806ad10 2023-10-16T14:30:32.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.045+0000 7fa6dffff640 1 -- 172.21.15.29:0/4056931340 <== mgr.5054 v2:172.21.15.170:6832/636880041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa6d000c050 con 0x7fa6c806ad10 2023-10-16T14:30:32.049 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-16T14:30:32.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.047+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fa6c806ad10 msgr2=0x7fa6c806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.047+0000 7fa6eda09640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fa6c806ad10 0x7fa6c806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa6e80a9f80 tx=0x7fa6d0008040 comp rx=0 tx=0).stop 2023-10-16T14:30:32.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.047+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 msgr2=0x7fa6e81efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.047+0000 7fa6eda09640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 0x7fa6e81efe00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c730 tx=0x7fa6d800cbf0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 shutdown_connections 2023-10-16T14:30:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fa6c806ad10 0x7fa6c806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6e815ad70 0x7fa6e80c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6e80c9ca0 0x7fa6e81f14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 --2- 172.21.15.29:0/4056931340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6e806a4b0 0x7fa6e81efe00 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 >> 172.21.15.29:0/4056931340 conn(0x7fa6e80ce9b0 msgr2=0x7fa6e80ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 shutdown_connections 2023-10-16T14:30:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.048+0000 7fa6eda09640 1 -- 172.21.15.29:0/4056931340 wait complete. 2023-10-16T14:30:32.072 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T14:30:32.072 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-14_31_00 in ~28s... 2023-10-16T14:30:32.073 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-14_32_00 in ~88s... 2023-10-16T14:30:32.073 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-10-16T14:30:32.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T14:30:32.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.295+0000 7f643b075640 1 Processor -- start 2023-10-16T14:30:32.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.295+0000 7f643b075640 1 -- start start 2023-10-16T14:30:32.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f643b075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64340cf710 0x7f64340b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f643b075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340cfaf0 0x7f64340b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f643b075640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f64340b38e0 0x7f64340d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f643b075640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64340d3c60 con 0x7f64340cf710 2023-10-16T14:30:32.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f643b075640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64340d3980 con 0x7f64340b38e0 2023-10-16T14:30:32.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f643b075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64340ce630 con 0x7f64340cfaf0 2023-10-16T14:30:32.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f6438e11640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64340cf710 0x7f64340b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f6438e11640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64340cf710 0x7f64340b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59426/0 (socket says 172.21.15.29:59426) 2023-10-16T14:30:32.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.297+0000 7f6438e11640 1 -- 172.21.15.29:0/1638415638 learned_addr learned my addr 172.21.15.29:0/1638415638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:32.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6433fff640 1 --2- 172.21.15.29:0/1638415638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340cfaf0 0x7f64340b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6439612640 1 --2- 172.21.15.29:0/1638415638 >> v2:172.21.15.170:3300/0 conn(0x7f64340b38e0 0x7f64340d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6438e11640 1 -- 172.21.15.29:0/1638415638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340cfaf0 msgr2=0x7f64340b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6438e11640 1 --2- 172.21.15.29:0/1638415638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340cfaf0 0x7f64340b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6438e11640 1 -- 172.21.15.29:0/1638415638 >> v2:172.21.15.170:3300/0 conn(0x7f64340b38e0 msgr2=0x7f64340d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:30:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6438e11640 1 --2- 172.21.15.29:0/1638415638 >> v2:172.21.15.170:3300/0 conn(0x7f64340b38e0 0x7f64340d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.298+0000 7f6438e11640 1 -- 172.21.15.29:0/1638415638 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64340ce910 con 0x7f64340cf710 2023-10-16T14:30:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.299+0000 7f6438e11640 1 --2- 172.21.15.29:0/1638415638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340b2d10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f643406cd60 tx=0x7f642802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b4e01e9ef8bd42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.299+0000 7f64337fe640 1 -- 172.21.15.29:0/1638415638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428004230 con 0x7f64340cf710 2023-10-16T14:30:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.299+0000 7f64337fe640 1 -- 172.21.15.29:0/1638415638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6428004390 con 0x7f64340cf710 2023-10-16T14:30:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.299+0000 7f64337fe640 1 -- 172.21.15.29:0/1638415638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428003810 con 0x7f64340cf710 2023-10-16T14:30:32.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 -- 172.21.15.29:0/1638415638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 msgr2=0x7f64340b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 --2- 172.21.15.29:0/1638415638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340b2d10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f643406cd60 tx=0x7f642802b860 comp rx=0 tx=0).stop 2023-10-16T14:30:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 -- 172.21.15.29:0/1638415638 shutdown_connections 2023-10-16T14:30:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 --2- 172.21.15.29:0/1638415638 >> v2:172.21.15.170:3300/0 conn(0x7f64340b38e0 0x7f64340d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 --2- 172.21.15.29:0/1638415638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340cfaf0 0x7f64340b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 --2- 172.21.15.29:0/1638415638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340b2d10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.300+0000 7f643b075640 1 -- 172.21.15.29:0/1638415638 >> 172.21.15.29:0/1638415638 conn(0x7f64340d2cc0 msgr2=0x7f64340d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:32.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.301+0000 7f643b075640 1 -- 172.21.15.29:0/1638415638 shutdown_connections 2023-10-16T14:30:32.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.301+0000 7f643b075640 1 -- 172.21.15.29:0/1638415638 wait complete. 2023-10-16T14:30:32.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.303+0000 7f643b075640 1 Processor -- start 2023-10-16T14:30:32.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.303+0000 7f643b075640 1 -- start start 2023-10-16T14:30:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f643b075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340b38e0 0x7f64340c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f643b075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f643b075640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64340cfaf0 0x7f64340c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f643b075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64340c8ca0 con 0x7f64340cf710 2023-10-16T14:30:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f643b075640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64340cc670 con 0x7f64340cfaf0 2023-10-16T14:30:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f643b075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64340cba60 con 0x7f64340b38e0 2023-10-16T14:30:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f6433fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f6433fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59428/0 (socket says 172.21.15.29:59428) 2023-10-16T14:30:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f6433fff640 1 -- 172.21.15.29:0/3047443085 learned_addr learned my addr 172.21.15.29:0/3047443085 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:30:32.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f6439612640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64340cfaf0 0x7f64340c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.304+0000 7f6438e11640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340b38e0 0x7f64340c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6433fff640 1 -- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340b38e0 msgr2=0x7f64340c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6433fff640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340b38e0 0x7f64340c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6433fff640 1 -- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64340cfaf0 msgr2=0x7f64340c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6433fff640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64340cfaf0 0x7f64340c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6433fff640 1 -- 172.21.15.29:0/3047443085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6428009460 con 0x7f64340cf710 2023-10-16T14:30:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6439612640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64340cfaf0 0x7f64340c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:30:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f6433fff640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340c67f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f642000d620 tx=0x7f642000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642000b620 con 0x7f64340cf710 2023-10-16T14:30:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642000b780 con 0x7f64340cf710 2023-10-16T14:30:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642000bb30 con 0x7f64340cf710 2023-10-16T14:30:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.305+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64340cbd40 con 0x7f64340cf710 2023-10-16T14:30:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.306+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64340cc300 con 0x7f64340cf710 2023-10-16T14:30:32.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.307+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64340afff0 con 0x7f64340cf710 2023-10-16T14:30:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.308+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f642000bc90 con 0x7f64340cf710 2023-10-16T14:30:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.309+0000 7f64317fa640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f641806b010 0x7f641806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:30:32.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.309+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6420089e50 con 0x7f64340cf710 2023-10-16T14:30:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.309+0000 7f6438e11640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f641806b010 0x7f641806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:30:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.310+0000 7f6438e11640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f641806b010 0x7f641806d4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6428034000 tx=0x7f6428008030 comp rx=0 tx=0).ready entity=mgr.5054 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:30:32.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.313+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6420054bf0 con 0x7f64340cf710 2023-10-16T14:30:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.529+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 --> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64340c87e0 con 0x7f641806b010 2023-10-16T14:30:32.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.547+0000 7f64317fa640 1 -- 172.21.15.29:0/3047443085 <== mgr.5054 v2:172.21.15.170:6832/636880041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f64340c87e0 con 0x7f641806b010 2023-10-16T14:30:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:30:32.550 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T14:30:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f641806b010 msgr2=0x7f641806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f641806b010 0x7f641806d4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6428034000 tx=0x7f6428008030 comp rx=0 tx=0).stop 2023-10-16T14:30:32.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 msgr2=0x7f64340c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:30:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340c67f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f642000d620 tx=0x7f642000dae0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 shutdown_connections 2023-10-16T14:30:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f641806b010 0x7f641806d4d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64340cf710 0x7f64340c67f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64340cfaf0 0x7f64340c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 --2- 172.21.15.29:0/3047443085 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f64340b38e0 0x7f64340c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:30:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 >> 172.21.15.29:0/3047443085 conn(0x7f64340d2cc0 msgr2=0x7f6434160070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:30:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 shutdown_connections 2023-10-16T14:30:32.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:30:32.549+0000 7f643b075640 1 -- 172.21.15.29:0/3047443085 wait complete. 2023-10-16T14:30:32.572 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T14:30:32.572 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-10-16T14:30:32.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:32.573 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:32.633 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:32.633 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:32.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:33.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:33.635 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:33.704 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:33.704 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:33.705 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:34.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:34.707 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:34.774 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:34.774 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:34.774 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:35.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:35.776 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:35.841 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:35.841 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:35.842 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:36.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:36.844 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:36.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:36.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:36.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:37.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:37.913 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:37.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:37.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:37.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:38.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:38.981 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:39.047 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:39.048 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:39.048 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:40.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:40.049 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:40.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:40.116 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:40.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:41.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:41.118 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:41.183 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:41.183 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:41.183 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:42.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:42.186 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:42.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:42.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:42.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:43.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:43.255 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:43.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:43.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:43.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:44.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:44.323 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:44.391 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:44.391 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:44.391 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:45.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:45.393 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:45.460 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:45.461 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:45.461 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:46.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:46.462 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:46.529 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:46.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:46.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:46.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:30:46.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:30:47.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:47.532 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:47.599 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:47.600 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:47.600 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:48.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:48.601 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:48.668 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:48.669 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:48.669 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:49.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:49.671 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:49.736 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:49.736 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:49.737 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:50.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:50.738 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:50.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:50.807 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:50.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:51.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:51.809 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:51.879 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:51.880 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:51.880 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:52.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:52.882 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:52.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:52.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:52.951 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:53.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:53.953 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:54.022 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:54.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:54.023 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:55.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:55.024 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:55.090 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:55.090 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:55.091 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:56.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:56.093 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:56.161 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:56.162 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:56.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:57.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:57.164 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:57.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:57.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:57.232 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:58.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:58.233 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:58.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:58.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:58.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:30:59.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:30:59.303 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:30:59.370 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:30:59.370 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:30:59.370 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:00.072 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:31:00.069+0000 7f7a6f96d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:31:00.073 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:31:00.069+0000 7f7a6f96d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:31:00.073 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:31:00.069+0000 7f7a6f96d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:31:00.073 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:31:00.069+0000 7f7a6f96d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:31:00.073 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:31:00.069+0000 7f7a6f96d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:31:00.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:00.372 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:00.439 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:00.440 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:00.440 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:00.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:00.441 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-14_31_00_UTC'] 2023-10-16T14:31:00.441 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697466632.0727818, now = 1697466660.4418325, timo = 28 2023-10-16T14:31:01.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:01.443 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:01.509 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:01.510 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:01.511 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:01.511 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:02.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:02.513 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:02.577 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:02.578 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:02.579 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:02.579 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:03.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:03.581 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:03.648 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:03.649 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:03.649 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:03.650 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:04.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:04.652 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:04.716 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:04.717 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:04.718 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:04.718 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:05.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:05.720 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:05.787 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:05.787 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:05.787 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:06.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:06.789 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:06.855 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:06.856 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:06.856 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:06.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:07.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:07.858 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:07.924 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:07.925 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:07.926 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:07.926 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:08.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:08.928 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:08.995 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:08.996 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:08.996 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:08.997 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:09.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:09.998 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:10.065 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:10.066 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:10.066 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:10.067 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:11.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:11.068 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:11.134 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:11.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:11.136 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:11.136 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:12.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:12.137 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:12.203 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:12.204 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:12.204 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:12.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:13.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:13.206 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:13.269 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:13.271 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:13.271 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:13.271 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:14.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:14.273 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:14.337 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:14.339 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:14.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:14.341 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:15.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:15.342 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:15.406 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:15.408 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:15.408 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:15.408 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:16.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:16.410 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:16.476 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:16.478 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:16.479 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:16.479 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:17.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:31:17.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:31:17.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:17.480 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:17.545 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:17.546 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:17.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:17.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:18.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:18.549 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:18.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:18.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:18.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:19.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:19.618 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:19.683 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:19.684 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:19.685 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:19.685 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:20.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:20.687 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:20.753 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:20.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:20.758 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:20.758 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:21.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:21.760 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:21.826 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:21.827 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:21.827 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:21.828 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:22.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:22.829 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:22.895 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:22.896 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:22.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:22.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:23.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:23.899 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:23.966 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:23.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:23.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:23.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:24.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:24.970 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:25.035 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:25.036 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:25.036 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:25.037 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:26.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:26.038 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:26.104 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:26.105 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:26.106 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:26.106 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:27.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:27.108 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:27.175 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:27.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:27.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:27.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:28.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:28.179 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:28.245 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:28.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:28.246 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:28.247 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:29.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:29.249 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:29.313 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:29.315 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:29.315 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:29.316 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:30.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:30.318 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:30.385 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:30.386 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:30.387 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:30.387 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:31.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:31.389 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:31.451 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:31.452 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:31.452 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:31.453 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:32.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:32.454 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:32.520 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:32.521 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:32.522 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:32.522 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:33.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:33.524 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:33.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:33.592 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:33.592 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:33.592 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:34.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:34.594 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:34.660 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:34.661 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:34.662 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:34.662 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:35.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:35.664 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:35.729 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:35.730 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:35.731 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:35.731 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:36.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:36.733 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:36.797 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:36.798 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:36.799 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:36.799 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:37.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:37.801 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:37.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:37.868 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:37.868 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:37.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:38.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:38.871 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:38.937 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:38.939 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:38.939 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:38.939 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:39.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:39.941 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:40.005 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:40.007 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:40.007 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:40.007 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:41.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:41.009 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:41.074 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:41.076 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:41.076 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:41.076 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:42.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:42.078 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:42.145 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:42.146 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:42.146 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:42.147 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:43.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:43.148 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:43.213 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:43.214 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:43.215 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:43.215 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:44.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:44.217 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:44.283 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:44.285 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:44.285 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:44.285 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:45.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:45.288 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:45.354 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:45.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:45.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:45.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:46.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:46.360 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:46.423 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:46.424 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:46.424 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:46.425 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:47.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:31:47.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:31:47.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:47.426 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:47.486 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:47.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:47.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:47.489 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:48.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:48.490 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:48.556 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:48.557 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:48.558 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:48.558 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:49.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:49.560 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:49.626 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:49.628 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:49.628 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:49.629 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:50.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:50.631 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:50.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:50.698 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:50.699 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:50.699 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:51.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:51.701 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:51.769 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:51.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:51.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:51.771 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:52.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:52.773 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:52.836 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:52.837 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:52.837 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:52.838 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:53.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:53.840 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:53.906 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:53.907 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:53.908 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:53.908 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:54.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:54.910 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:54.977 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:54.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:54.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:54.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:55.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:55.981 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:56.047 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:56.049 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:56.049 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:56.050 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:57.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:57.051 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:57.116 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:57.118 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:57.118 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:57.118 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:58.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:58.120 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:58.186 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:58.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:58.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:58.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:31:59.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:31:59.190 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:31:59.260 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:31:59.261 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:31:59.261 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:31:59.262 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:32:00.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:00.264 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:32:00.328 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:32:00.329 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_32_00_UTC 2023-10-16T14:32:00.330 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_32_00_UTC', 'scheduled-2023-10-16-14_31_00_UTC'} 2023-10-16T14:32:00.330 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-14_32_00_UTC'} 2023-10-16T14:32:00.330 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:32:00.331 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-14_32_00_UTC'] 2023-10-16T14:32:00.331 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697466632.0727818, now = 1697466720.3314028, timo = 88 2023-10-16T14:32:01.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T14:32:01.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.569+0000 7f711bc10640 1 Processor -- start 2023-10-16T14:32:01.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.569+0000 7f711bc10640 1 -- start start 2023-10-16T14:32:01.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711bc10640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f711415ad70 0x7f7114069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711bc10640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71140c9ca0 0x7f711406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711bc10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71140ca3b0 0x7f71140ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711bc10640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71140cad80 con 0x7f71140c9ca0 2023-10-16T14:32:01.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711bc10640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71140cb030 con 0x7f711415ad70 2023-10-16T14:32:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711bc10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71140c91c0 con 0x7f71140ca3b0 2023-10-16T14:32:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f71191ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71140c9ca0 0x7f711406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f71191ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71140c9ca0 0x7f711406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47884/0 (socket says 172.21.15.29:47884) 2023-10-16T14:32:01.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f71191ab640 1 -- 172.21.15.29:0/3889728632 learned_addr learned my addr 172.21.15.29:0/3889728632 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:01.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71191ab640 1 -- 172.21.15.29:0/3889728632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71140ca3b0 msgr2=0x7f71140ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:01.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.571+0000 7f711a1ad640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71140ca3b0 0x7f71140ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71199ac640 1 --2- 172.21.15.29:0/3889728632 >> v2:172.21.15.170:3300/0 conn(0x7f711415ad70 0x7f7114069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71191ab640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71140ca3b0 0x7f71140ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71191ab640 1 -- 172.21.15.29:0/3889728632 >> v2:172.21.15.170:3300/0 conn(0x7f711415ad70 msgr2=0x7f7114069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:01.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71191ab640 1 --2- 172.21.15.29:0/3889728632 >> v2:172.21.15.170:3300/0 conn(0x7f711415ad70 0x7f7114069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71191ab640 1 -- 172.21.15.29:0/3889728632 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71140c94a0 con 0x7f71140c9ca0 2023-10-16T14:32:01.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f711a1ad640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71140ca3b0 0x7f71140ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:01.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.572+0000 7f71199ac640 1 --2- 172.21.15.29:0/3889728632 >> v2:172.21.15.170:3300/0 conn(0x7f711415ad70 0x7f7114069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:01.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.573+0000 7f71191ab640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140c9ca0 0x7f711406a2e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f71140cf780 tx=0x7f710402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08cb17e4782707a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:01.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.573+0000 7f71189aa640 1 -- 172.21.15.29:0/3889728632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104031880 con 0x7f71140c9ca0 2023-10-16T14:32:01.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.573+0000 7f71189aa640 1 -- 172.21.15.29:0/3889728632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71040319e0 con 0x7f71140c9ca0 2023-10-16T14:32:01.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.573+0000 7f71189aa640 1 -- 172.21.15.29:0/3889728632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104031d70 con 0x7f71140c9ca0 2023-10-16T14:32:01.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 -- 172.21.15.29:0/3889728632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140c9ca0 msgr2=0x7f711406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:01.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140c9ca0 0x7f711406a2e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f71140cf780 tx=0x7f710402b920 comp rx=0 tx=0).stop 2023-10-16T14:32:01.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 -- 172.21.15.29:0/3889728632 shutdown_connections 2023-10-16T14:32:01.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f71140ca3b0 0x7f71140ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 --2- 172.21.15.29:0/3889728632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140c9ca0 0x7f711406a2e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 --2- 172.21.15.29:0/3889728632 >> v2:172.21.15.170:3300/0 conn(0x7f711415ad70 0x7f7114069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 -- 172.21.15.29:0/3889728632 >> 172.21.15.29:0/3889728632 conn(0x7f71140ce9b0 msgr2=0x7f71140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:01.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 -- 172.21.15.29:0/3889728632 shutdown_connections 2023-10-16T14:32:01.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.574+0000 7f711bc10640 1 -- 172.21.15.29:0/3889728632 wait complete. 2023-10-16T14:32:01.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.576+0000 7f711bc10640 1 Processor -- start 2023-10-16T14:32:01.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.576+0000 7f711bc10640 1 -- start start 2023-10-16T14:32:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f711bc10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71140c9ca0 0x7f71140aa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f711bc10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 0x7f711414be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f711bc10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f711415ad70 0x7f71140b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f711bc10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71140aa140 con 0x7f71140ca3b0 2023-10-16T14:32:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f711bc10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71140a9530 con 0x7f71140c9ca0 2023-10-16T14:32:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f711bc10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71140a9810 con 0x7f711415ad70 2023-10-16T14:32:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.577+0000 7f71191ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 0x7f711414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 0x7f711414be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47894/0 (socket says 172.21.15.29:47894) 2023-10-16T14:32:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 -- 172.21.15.29:0/1455033160 learned_addr learned my addr 172.21.15.29:0/1455033160 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:01.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f711a1ad640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f711415ad70 0x7f71140b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 -- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f711415ad70 msgr2=0x7f71140b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71199ac640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71140c9ca0 0x7f71140aa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f711415ad70 0x7f71140b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 -- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71140c9ca0 msgr2=0x7f71140aa5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71140c9ca0 0x7f71140aa5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 -- 172.21.15.29:0/1455033160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040094d0 con 0x7f71140ca3b0 2023-10-16T14:32:01.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f711a1ad640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f711415ad70 0x7f71140b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:01.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.578+0000 7f71191ab640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 0x7f711414be20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f710402b960 tx=0x7f71040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:01.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.579+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104031870 con 0x7f71140ca3b0 2023-10-16T14:32:01.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.579+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71040443f0 con 0x7f71140ca3b0 2023-10-16T14:32:01.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.579+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71140a9af0 con 0x7f71140ca3b0 2023-10-16T14:32:01.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.579+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71040447a0 con 0x7f71140ca3b0 2023-10-16T14:32:01.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.579+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711414c360 con 0x7f71140ca3b0 2023-10-16T14:32:01.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.580+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71140aa2f0 con 0x7f71140ca3b0 2023-10-16T14:32:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.581+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f710404d450 con 0x7f71140ca3b0 2023-10-16T14:32:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.582+0000 7f710a7fc640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f70f806b100 0x7f70f806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:01.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.583+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f71040ac550 con 0x7f71140ca3b0 2023-10-16T14:32:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.583+0000 7f71199ac640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f70f806b100 0x7f70f806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.585+0000 7f71199ac640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f70f806b100 0x7f70f806d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f711000b840 tx=0x7f7110008040 comp rx=0 tx=0).ready entity=mgr.5054 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.588+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f71140aa2f0 con 0x7f71140ca3b0 2023-10-16T14:32:01.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.806+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 --> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f71140c9780 con 0x7f70f806b100 2023-10-16T14:32:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.971+0000 7f710a7fc640 1 -- 172.21.15.29:0/1455033160 <== mgr.5054 v2:172.21.15.170:6832/636880041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f711000c040 con 0x7f70f806b100 2023-10-16T14:32:01.974 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-16T14:32:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f70f806b100 msgr2=0x7f70f806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f70f806b100 0x7f70f806d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f711000b840 tx=0x7f7110008040 comp rx=0 tx=0).stop 2023-10-16T14:32:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 msgr2=0x7f711414be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 0x7f711414be20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f710402b960 tx=0x7f71040046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 shutdown_connections 2023-10-16T14:32:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f70f806b100 0x7f70f806d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71140ca3b0 0x7f711414be20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f711415ad70 0x7f71140b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 --2- 172.21.15.29:0/1455033160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f71140c9ca0 0x7f71140aa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 >> 172.21.15.29:0/1455033160 conn(0x7f71140ce9b0 msgr2=0x7f711406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 shutdown_connections 2023-10-16T14:32:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:01.975+0000 7f711bc10640 1 -- 172.21.15.29:0/1455033160 wait complete. 2023-10-16T14:32:01.997 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T14:32:01.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:01.998 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:32:02.055 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:32:02.055 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_32_00_UTC 2023-10-16T14:32:02.056 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_31_00_UTC 2023-10-16T14:32:02.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:02.057 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_31_00_UTC) 2023-10-16T14:32:02.121 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_32_00_UTC 2023-10-16T14:32:02.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:02.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_32_00_UTC) 2023-10-16T14:32:02.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:02.187 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T14:32:02.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:32:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.472+0000 7fd0f4b93640 1 Processor -- start 2023-10-16T14:32:02.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.472+0000 7fd0f4b93640 1 -- start start 2023-10-16T14:32:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.474+0000 7fd0f4b93640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0f006c040 0x7fd0f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.474+0000 7fd0f4b93640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0f00ad970 0x7fd0f015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.474+0000 7fd0f4b93640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f015ebc0 0x7fd0f015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.474+0000 7fd0f4b93640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f006c960 con 0x7fd0f00ad970 2023-10-16T14:32:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.474+0000 7fd0f4b93640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f015b9d0 con 0x7fd0f006c040 2023-10-16T14:32:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.474+0000 7fd0f4b93640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f015bcb0 con 0x7fd0f015ebc0 2023-10-16T14:32:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0f00ad970 0x7fd0f015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0f00ad970 0x7fd0f015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47902/0 (socket says 172.21.15.29:47902) 2023-10-16T14:32:02.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/3773922666 learned_addr learned my addr 172.21.15.29:0/3773922666 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0effff640 1 --2- 172.21.15.29:0/3773922666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f015ebc0 0x7fd0f015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0ef7fe640 1 --2- 172.21.15.29:0/3773922666 >> v2:172.21.15.170:3300/0 conn(0x7fd0f006c040 0x7fd0f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/3773922666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f015ebc0 msgr2=0x7fd0f015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 --2- 172.21.15.29:0/3773922666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f015ebc0 0x7fd0f015f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/3773922666 >> v2:172.21.15.170:3300/0 conn(0x7fd0f006c040 msgr2=0x7fd0f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 --2- 172.21.15.29:0/3773922666 >> v2:172.21.15.170:3300/0 conn(0x7fd0f006c040 0x7fd0f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/3773922666 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f015bf90 con 0x7fd0f00ad970 2023-10-16T14:32:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.475+0000 7fd0ef7fe640 1 --2- 172.21.15.29:0/3773922666 >> v2:172.21.15.170:3300/0 conn(0x7fd0f006c040 0x7fd0f006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.476+0000 7fd0e6ffd640 1 --2- 172.21.15.29:0/3773922666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f015e680 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00cf350 tx=0x7fd0d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c145b6830a8f219b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0eeffd640 1 -- 172.21.15.29:0/3773922666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8031870 con 0x7fd0f00ad970 2023-10-16T14:32:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0eeffd640 1 -- 172.21.15.29:0/3773922666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d80319d0 con 0x7fd0f00ad970 2023-10-16T14:32:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0eeffd640 1 -- 172.21.15.29:0/3773922666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8031d60 con 0x7fd0f00ad970 2023-10-16T14:32:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 -- 172.21.15.29:0/3773922666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 msgr2=0x7fd0f015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/3773922666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f015e680 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00cf350 tx=0x7fd0d802b910 comp rx=0 tx=0).stop 2023-10-16T14:32:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 -- 172.21.15.29:0/3773922666 shutdown_connections 2023-10-16T14:32:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/3773922666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f015ebc0 0x7fd0f015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/3773922666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f015e680 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/3773922666 >> v2:172.21.15.170:3300/0 conn(0x7fd0f006c040 0x7fd0f006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 -- 172.21.15.29:0/3773922666 >> 172.21.15.29:0/3773922666 conn(0x7fd0f00ce6a0 msgr2=0x7fd0f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 -- 172.21.15.29:0/3773922666 shutdown_connections 2023-10-16T14:32:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.477+0000 7fd0f4b93640 1 -- 172.21.15.29:0/3773922666 wait complete. 2023-10-16T14:32:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.479+0000 7fd0f4b93640 1 Processor -- start 2023-10-16T14:32:02.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.479+0000 7fd0f4b93640 1 -- start start 2023-10-16T14:32:02.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.480+0000 7fd0f4b93640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f006c040 0x7fd0f014d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.480+0000 7fd0f4b93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f00c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.480+0000 7fd0f4b93640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0f015ebc0 0x7fd0f00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.480+0000 7fd0f4b93640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f014dcb0 con 0x7fd0f00ad970 2023-10-16T14:32:02.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.480+0000 7fd0f4b93640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00b2850 con 0x7fd0f015ebc0 2023-10-16T14:32:02.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.480+0000 7fd0f4b93640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00ca410 con 0x7fd0f006c040 2023-10-16T14:32:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f00c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f00c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47904/0 (socket says 172.21.15.29:47904) 2023-10-16T14:32:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/1391545657 learned_addr learned my addr 172.21.15.29:0/1391545657 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:02.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0ef7fe640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f006c040 0x7fd0f014d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0effff640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0f015ebc0 0x7fd0f00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f006c040 msgr2=0x7fd0f014d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f006c040 0x7fd0f014d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0f015ebc0 msgr2=0x7fd0f00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:02.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0f015ebc0 0x7fd0f00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 -- 172.21.15.29:0/1391545657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80094d0 con 0x7fd0f00ad970 2023-10-16T14:32:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0effff640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0f015ebc0 0x7fd0f00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.481+0000 7fd0e6ffd640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f00c9ed0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80097c0 tx=0x7fd0d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.482+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d803ac60 con 0x7fd0f00ad970 2023-10-16T14:32:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.482+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d803adc0 con 0x7fd0f00ad970 2023-10-16T14:32:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.482+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8044610 con 0x7fd0f00ad970 2023-10-16T14:32:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.482+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f00b2d40 con 0x7fd0f00ad970 2023-10-16T14:32:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.482+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f00b3300 con 0x7fd0f00ad970 2023-10-16T14:32:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.483+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f006b330 con 0x7fd0f00ad970 2023-10-16T14:32:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.484+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd0d804d450 con 0x7fd0f00ad970 2023-10-16T14:32:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.485+0000 7fd0ecff9640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd0d006b100 0x7fd0d006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.485+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fd0d802d040 con 0x7fd0f00ad970 2023-10-16T14:32:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.486+0000 7fd0ef7fe640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd0d006b100 0x7fd0d006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:02.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.487+0000 7fd0ef7fe640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd0d006b100 0x7fd0d006d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00c4b70 tx=0x7fd0e0008470 comp rx=0 tx=0).ready entity=mgr.5054 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:02.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.490+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0d8076db0 con 0x7fd0f00ad970 2023-10-16T14:32:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:02.737+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd0f004e9b0 con 0x7fd0f00ad970 2023-10-16T14:32:03.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.216+0000 7fd0ecff9640 1 -- 172.21.15.29:0/1391545657 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v25) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd0d80350a0 con 0x7fd0f00ad970 2023-10-16T14:32:03.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.219+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd0d006b100 msgr2=0x7fd0d006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.219+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd0d006b100 0x7fd0d006d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00c4b70 tx=0x7fd0e0008470 comp rx=0 tx=0).stop 2023-10-16T14:32:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.219+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 msgr2=0x7fd0f00c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:03.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.219+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f00c9ed0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80097c0 tx=0x7fd0d80046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.220+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 shutdown_connections 2023-10-16T14:32:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.220+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fd0d006b100 0x7fd0d006d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.220+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0f00ad970 0x7fd0f00c9ed0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.220+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0f015ebc0 0x7fd0f00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.220+0000 7fd0f4b93640 1 --2- 172.21.15.29:0/1391545657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0f006c040 0x7fd0f014d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.220+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 >> 172.21.15.29:0/1391545657 conn(0x7fd0f00ce6a0 msgr2=0x7fd0f00cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.221+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 shutdown_connections 2023-10-16T14:32:03.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.221+0000 7fd0f4b93640 1 -- 172.21.15.29:0/1391545657 wait complete. 2023-10-16T14:32:03.248 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:32:03.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:03.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:03.471 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:32:03.471 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:03.472 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:03.529 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:03.530 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:03.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:03.745 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:03.742+0000 7f266b411040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:32:03.772 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:03.769+0000 7efd13645040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:32:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.780+0000 7ff84a1ed640 1 Processor -- start 2023-10-16T14:32:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.781+0000 7ff84a1ed640 1 -- start start 2023-10-16T14:32:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff84a1ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff84406c040 0x7ff84406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff84a1ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8440ad970 0x7ff84415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff84a1ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff84415ebc0 0x7ff84415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff84a1ed640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84406c960 con 0x7ff8440ad970 2023-10-16T14:32:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff84a1ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84415b9d0 con 0x7ff84415ebc0 2023-10-16T14:32:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff84a1ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84415bcb0 con 0x7ff84406c040 2023-10-16T14:32:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff8489ea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8440ad970 0x7ff84415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff8489ea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8440ad970 0x7ff84415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47926/0 (socket says 172.21.15.29:47926) 2023-10-16T14:32:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.782+0000 7ff8489ea640 1 -- 172.21.15.29:0/2136486752 learned_addr learned my addr 172.21.15.29:0/2136486752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8491eb640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff84406c040 0x7ff84406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8489ea640 1 -- 172.21.15.29:0/2136486752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff84406c040 msgr2=0x7ff84406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8489ea640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff84406c040 0x7ff84406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8489ea640 1 -- 172.21.15.29:0/2136486752 >> v2:172.21.15.170:3300/0 conn(0x7ff84415ebc0 msgr2=0x7ff84415f050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T14:32:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8489ea640 1 --2- 172.21.15.29:0/2136486752 >> v2:172.21.15.170:3300/0 conn(0x7ff84415ebc0 0x7ff84415f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8489ea640 1 -- 172.21.15.29:0/2136486752 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84415bf90 con 0x7ff8440ad970 2023-10-16T14:32:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.783+0000 7ff8491eb640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff84406c040 0x7ff84406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.784+0000 7ff8489ea640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8440ad970 0x7ff84415e680 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff834009870 tx=0x7ff83402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7247a04daff419e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.784+0000 7ff83bfff640 1 -- 172.21.15.29:0/2136486752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834031870 con 0x7ff8440ad970 2023-10-16T14:32:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.784+0000 7ff83bfff640 1 -- 172.21.15.29:0/2136486752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8340319d0 con 0x7ff8440ad970 2023-10-16T14:32:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.784+0000 7ff83bfff640 1 -- 172.21.15.29:0/2136486752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834031d60 con 0x7ff8440ad970 2023-10-16T14:32:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2136486752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8440ad970 msgr2=0x7ff84415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8440ad970 0x7ff84415e680 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff834009870 tx=0x7ff83402b910 comp rx=0 tx=0).stop 2023-10-16T14:32:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2136486752 shutdown_connections 2023-10-16T14:32:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2136486752 >> v2:172.21.15.170:3300/0 conn(0x7ff84415ebc0 0x7ff84415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8440ad970 0x7ff84415e680 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2136486752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff84406c040 0x7ff84406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2136486752 >> 172.21.15.29:0/2136486752 conn(0x7ff8440ce6a0 msgr2=0x7ff8440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2136486752 shutdown_connections 2023-10-16T14:32:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.785+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2136486752 wait complete. 2023-10-16T14:32:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.787+0000 7ff84a1ed640 1 Processor -- start 2023-10-16T14:32:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.787+0000 7ff84a1ed640 1 -- start start 2023-10-16T14:32:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff84a1ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff84406c040 0x7ff8440c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff84a1ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8440ad970 0x7ff8440ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff84a1ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 0x7ff8440acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff84a1ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440c7700 con 0x7ff84415ebc0 2023-10-16T14:32:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff84a1ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84414da70 con 0x7ff84406c040 2023-10-16T14:32:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff84a1ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440c6d60 con 0x7ff8440ad970 2023-10-16T14:32:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff8499ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 0x7ff8440acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff8499ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 0x7ff8440acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47928/0 (socket says 172.21.15.29:47928) 2023-10-16T14:32:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.788+0000 7ff8499ec640 1 -- 172.21.15.29:0/2788983206 learned_addr learned my addr 172.21.15.29:0/2788983206 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8491eb640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff84406c040 0x7ff8440c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8499ec640 1 -- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8440ad970 msgr2=0x7ff8440ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8489ea640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8440ad970 0x7ff8440ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8499ec640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8440ad970 0x7ff8440ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8499ec640 1 -- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff84406c040 msgr2=0x7ff8440c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8499ec640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff84406c040 0x7ff8440c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:03.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8499ec640 1 -- 172.21.15.29:0/2788983206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340094d0 con 0x7ff84415ebc0 2023-10-16T14:32:03.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8489ea640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8440ad970 0x7ff8440ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:03.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8491eb640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff84406c040 0x7ff8440c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:03.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff8499ec640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 0x7ff8440acdd0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00b990 tx=0x7ff83c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:03.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.789+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c00cba0 con 0x7ff84415ebc0 2023-10-16T14:32:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.790+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c00cd00 con 0x7ff84415ebc0 2023-10-16T14:32:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.790+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c014680 con 0x7ff84415ebc0 2023-10-16T14:32:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.790+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8440c7040 con 0x7ff84415ebc0 2023-10-16T14:32:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.790+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84414a600 con 0x7ff84415ebc0 2023-10-16T14:32:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.791+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84406b330 con 0x7ff84415ebc0 2023-10-16T14:32:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.792+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff83c01d3b0 con 0x7ff84415ebc0 2023-10-16T14:32:03.800 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:03.795+0000 7f266b411040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:32:03.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.793+0000 7ff839ffb640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 0x7ff82806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:03.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.793+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6111+0+0 (secure 0 0 0) 0x7ff83c0898c0 con 0x7ff84415ebc0 2023-10-16T14:32:03.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.793+0000 7ff8491eb640 1 -- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 msgr2=0x7ff82806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:03.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.794+0000 7ff8491eb640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 0x7ff82806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:03.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.798+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff83c0546c0 con 0x7ff84415ebc0 2023-10-16T14:32:03.830 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:03.828+0000 7efd13645040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:32:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.991+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8440c7840 con 0x7ff84415ebc0 2023-10-16T14:32:03.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.994+0000 7ff839ffb640 1 -- 172.21.15.29:0/2788983206 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+17113 (secure 0 0 0) 0x7ff8440c7840 con 0x7ff84415ebc0 2023-10-16T14:32:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:03.998 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:31:32.170332+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3247649762":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2698007731":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4263280508":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2380256585":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3575752004":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3937243709":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6833/3423886595":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6832/3423886595":"2023-10-17T14:30:30.217790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.995+0000 7ff8491eb640 1 -- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 msgr2=0x7ff82806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:03.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.995+0000 7ff8491eb640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 0x7ff82806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 msgr2=0x7ff82806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:04.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 0x7ff82806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 msgr2=0x7ff8440acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 0x7ff8440acdd0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00b990 tx=0x7ff83c00be50 comp rx=0 tx=0).stop 2023-10-16T14:32:04.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 shutdown_connections 2023-10-16T14:32:04.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff82806afc0 0x7ff82806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84415ebc0 0x7ff8440acdd0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8440ad970 0x7ff8440ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 --2- 172.21.15.29:0/2788983206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff84406c040 0x7ff8440c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.997+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 >> 172.21.15.29:0/2788983206 conn(0x7ff8440ce6a0 msgr2=0x7ff8440c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.998+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 shutdown_connections 2023-10-16T14:32:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:03.998+0000 7ff84a1ed640 1 -- 172.21.15.29:0/2788983206 wait complete. 2023-10-16T14:32:04.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:30:13.584764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:30:14.740756+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:04.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.226+0000 7fc42eeca640 1 Processor -- start 2023-10-16T14:32:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.227+0000 7fc42eeca640 1 -- start start 2023-10-16T14:32:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.228+0000 7fc42eeca640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc42815e350 0x7fc4280cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.228+0000 7fc42eeca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42806fbc0 con 0x7fc42815f090 2023-10-16T14:32:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.228+0000 7fc42eeca640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280c9920 con 0x7fc428070970 2023-10-16T14:32:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.228+0000 7fc42eeca640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42815e730 con 0x7fc42815e350 2023-10-16T14:32:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42e6c9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc42815f090 0x7fc4280ca9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58604/0 (socket says 172.21.15.29:58604) 2023-10-16T14:32:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42e6c9640 1 -- 172.21.15.29:0/67954486 learned_addr learned my addr 172.21.15.29:0/67954486 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42dec8640 1 --2- 172.21.15.29:0/67954486 >> v2:172.21.15.170:3300/0 conn(0x7fc42815e350 0x7fc4280cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031935103 0 0) 0x7fc42806fbc0 con 0x7fc42815f090 2023-10-16T14:32:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc414003520 con 0x7fc42815f090 2023-10-16T14:32:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483173969 0 0) 0x7fc414003520 con 0x7fc42815f090 2023-10-16T14:32:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 >> v2:172.21.15.170:3300/0 conn(0x7fc42815e350 msgr2=0x7fc4280cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42cec6640 1 --2- 172.21.15.29:0/67954486 >> v2:172.21.15.170:3300/0 conn(0x7fc42815e350 0x7fc4280cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.229+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 >> v1:172.21.15.170:6789/0 conn(0x7fc428070970 legacy=0x7fc42815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280c9c00 con 0x7fc42815f090 2023-10-16T14:32:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc4200019f0 con 0x7fc42815f090 2023-10-16T14:32:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 >> v1:172.21.15.29:6789/0 conn(0x7fc42815f090 legacy=0x7fc4280ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc414003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4140047c0 0x7fc414004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4140051d0 0x7fc41400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41400c0f0 con 0x7fc4140039c0 2023-10-16T14:32:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41400c260 con 0x7fc4140051d0 2023-10-16T14:32:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41400c420 con 0x7fc4140047c0 2023-10-16T14:32:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42dec8640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc414003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42d6c7640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4140047c0 0x7fc414004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42e6c9640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4140051d0 0x7fc41400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42dec8640 1 -- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4140047c0 msgr2=0x7fc414004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42dec8640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4140047c0 0x7fc414004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.230+0000 7fc42dec8640 1 -- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4140051d0 msgr2=0x7fc41400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.231+0000 7fc42dec8640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4140051d0 0x7fc41400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.231+0000 7fc42dec8640 1 -- 172.21.15.29:0/67954486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414003520 con 0x7fc4140039c0 2023-10-16T14:32:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.231+0000 7fc42d6c7640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4140047c0 0x7fc414004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.231+0000 7fc42e6c9640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4140051d0 0x7fc41400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.231+0000 7fc42e6c9640 1 -- 172.21.15.29:0/67954486 reap_dead start 2023-10-16T14:32:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.232+0000 7fc42dec8640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc414003de0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc424009a90 tx=0x7fc4140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8af2612a31fd730 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.232+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42402c680 con 0x7fc4140039c0 2023-10-16T14:32:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.232+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42402c7e0 con 0x7fc4140039c0 2023-10-16T14:32:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.232+0000 7fc42cec6640 1 -- 172.21.15.29:0/67954486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42402cb70 con 0x7fc4140039c0 2023-10-16T14:32:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 -- 172.21.15.29:0/67954486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 msgr2=0x7fc414003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc414003de0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc424009a90 tx=0x7fc4140057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 -- 172.21.15.29:0/67954486 shutdown_connections 2023-10-16T14:32:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 --2- 172.21.15.29:0/67954486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc414003de0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 -- 172.21.15.29:0/67954486 >> 172.21.15.29:0/67954486 conn(0x7fc4280b0a50 msgr2=0x7fc4280b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 -- 172.21.15.29:0/67954486 shutdown_connections 2023-10-16T14:32:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.233+0000 7fc42eeca640 1 -- 172.21.15.29:0/67954486 wait complete. 2023-10-16T14:32:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.235+0000 7fc42eeca640 1 Processor -- start 2023-10-16T14:32:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.235+0000 7fc42eeca640 1 -- start start 2023-10-16T14:32:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42eeca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc428148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42eeca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc428155b60 0x7fc4280c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42eeca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4280c4150 0x7fc4280c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42eeca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280bfb70 con 0x7fc4140039c0 2023-10-16T14:32:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42eeca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428156340 con 0x7fc4280c4150 2023-10-16T14:32:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42eeca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42809c910 con 0x7fc428155b60 2023-10-16T14:32:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42dec8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc428148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42dec8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc428148a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50774/0 (socket says 172.21.15.29:50774) 2023-10-16T14:32:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.236+0000 7fc42dec8640 1 -- 172.21.15.29:0/3720675740 learned_addr learned my addr 172.21.15.29:0/3720675740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42d6c7640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc428155b60 0x7fc4280c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42e6c9640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4280c4150 0x7fc4280c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42dec8640 1 -- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc428155b60 msgr2=0x7fc4280c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42dec8640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc428155b60 0x7fc4280c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42dec8640 1 -- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4280c4150 msgr2=0x7fc4280c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42dec8640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4280c4150 0x7fc4280c45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42dec8640 1 -- 172.21.15.29:0/3720675740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42809c350 con 0x7fc4140039c0 2023-10-16T14:32:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42dec8640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc428148a60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc424037040 tx=0x7fc424004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424003810 con 0x7fc4140039c0 2023-10-16T14:32:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc424003970 con 0x7fc4140039c0 2023-10-16T14:32:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424003d00 con 0x7fc4140039c0 2023-10-16T14:32:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.237+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42809c630 con 0x7fc4140039c0 2023-10-16T14:32:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.238+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428148fa0 con 0x7fc4140039c0 2023-10-16T14:32:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.239+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42806e500 con 0x7fc4140039c0 2023-10-16T14:32:04.250 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.241+0000 7f266b411040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:32:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.240+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fc424034440 con 0x7fc4140039c0 2023-10-16T14:32:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.241+0000 7fc41e7fc640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 0x7fc40c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.241+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fc4240ab900 con 0x7fc4140039c0 2023-10-16T14:32:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.241+0000 7fc42d6c7640 1 -- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 msgr2=0x7fc40c06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.241+0000 7fc42d6c7640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 0x7fc40c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.245+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc424076840 con 0x7fc4140039c0 2023-10-16T14:32:04.297 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.293+0000 7f266b411040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:32:04.313 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.310+0000 7efd13645040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:32:04.347 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.343+0000 7f266b411040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:32:04.368 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.365+0000 7efd13645040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:32:04.407 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.403+0000 7f266b411040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:32:04.421 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.418+0000 7efd13645040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:32:04.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.439+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc428149130 con 0x7fc4140039c0 2023-10-16T14:32:04.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.440+0000 7fc41e7fc640 1 -- 172.21.15.29:0/3720675740 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc428149130 con 0x7fc4140039c0 2023-10-16T14:32:04.443 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:04.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.442+0000 7fc42d6c7640 1 -- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 msgr2=0x7fc40c06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.442+0000 7fc42d6c7640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 0x7fc40c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:04.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 msgr2=0x7fc40c06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:04.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 0x7fc40c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 msgr2=0x7fc428148a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc428148a60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc424037040 tx=0x7fc424004730 comp rx=0 tx=0).stop 2023-10-16T14:32:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 shutdown_connections 2023-10-16T14:32:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc40c06afc0 0x7fc40c06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4280c4150 0x7fc4280c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc428155b60 0x7fc4280c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 --2- 172.21.15.29:0/3720675740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4140039c0 0x7fc428148a60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 >> 172.21.15.29:0/3720675740 conn(0x7fc4280b0a50 msgr2=0x7fc4280700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 shutdown_connections 2023-10-16T14:32:04.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.443+0000 7fc42eeca640 1 -- 172.21.15.29:0/3720675740 wait complete. 2023-10-16T14:32:04.455 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.452+0000 7f266b411040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:32:04.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:32:04.488 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.485+0000 7efd13645040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:32:04.542 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.539+0000 7efd13645040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:32:04.565 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.562+0000 7f266b411040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:32:04.627 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.623+0000 7f266b411040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:04.660 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.657+0000 7efd13645040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:32:04.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.665+0000 7fb09653c640 1 Processor -- start 2023-10-16T14:32:04.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.665+0000 7fb09653c640 1 -- start start 2023-10-16T14:32:04.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.666+0000 7fb09653c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb09015cd70 0x7fb0900d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.666+0000 7fb09653c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0900cded0 0x7fb0900b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.666+0000 7fb09653c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09015d150 con 0x7fb0900b8480 2023-10-16T14:32:04.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.666+0000 7fb09653c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0900d1180 con 0x7fb0900cded0 2023-10-16T14:32:04.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.666+0000 7fb09653c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0900d1460 con 0x7fb09015cd70 2023-10-16T14:32:04.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb095d3b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0900cded0 0x7fb0900b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb094d39640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb0900b8480 0x7fb0900cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58618/0 (socket says 172.21.15.29:58618) 2023-10-16T14:32:04.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb094d39640 1 -- 172.21.15.29:0/1005205413 learned_addr learned my addr 172.21.15.29:0/1005205413 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:04.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb09553a640 1 --2- 172.21.15.29:0/1005205413 >> v2:172.21.15.170:3300/0 conn(0x7fb09015cd70 0x7fb0900d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb095d3b640 1 -- 172.21.15.29:0/1005205413 >> v2:172.21.15.170:3300/0 conn(0x7fb09015cd70 msgr2=0x7fb0900d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:04.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb095d3b640 1 --2- 172.21.15.29:0/1005205413 >> v2:172.21.15.170:3300/0 conn(0x7fb09015cd70 0x7fb0900d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb095d3b640 1 -- 172.21.15.29:0/1005205413 >> v1:172.21.15.29:6789/0 conn(0x7fb0900b8480 legacy=0x7fb0900cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.667+0000 7fb087fff640 1 -- 172.21.15.29:0/1005205413 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316520534 0 0) 0x7fb09015d150 con 0x7fb0900b8480 2023-10-16T14:32:04.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.668+0000 7fb095d3b640 1 -- 172.21.15.29:0/1005205413 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0900d1740 con 0x7fb0900cded0 2023-10-16T14:32:04.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.668+0000 7fb09553a640 1 --2- 172.21.15.29:0/1005205413 >> v2:172.21.15.170:3300/0 conn(0x7fb09015cd70 0x7fb0900d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:04.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.669+0000 7fb095d3b640 1 --2- 172.21.15.29:0/1005205413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0900cded0 0x7fb0900b5c90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb0900b56a0 tx=0x7fb08802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12976e78dd4e2eb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:04.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.669+0000 7fb087fff640 1 -- 172.21.15.29:0/1005205413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08802cc70 con 0x7fb0900cded0 2023-10-16T14:32:04.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb087fff640 1 -- 172.21.15.29:0/1005205413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08802cdd0 con 0x7fb0900cded0 2023-10-16T14:32:04.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb087fff640 1 -- 172.21.15.29:0/1005205413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088034660 con 0x7fb0900cded0 2023-10-16T14:32:04.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 -- 172.21.15.29:0/1005205413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0900cded0 msgr2=0x7fb0900b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 --2- 172.21.15.29:0/1005205413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0900cded0 0x7fb0900b5c90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb0900b56a0 tx=0x7fb08802bec0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 -- 172.21.15.29:0/1005205413 shutdown_connections 2023-10-16T14:32:04.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 --2- 172.21.15.29:0/1005205413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0900cded0 0x7fb0900b5c90 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 --2- 172.21.15.29:0/1005205413 >> v2:172.21.15.170:3300/0 conn(0x7fb09015cd70 0x7fb0900d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 -- 172.21.15.29:0/1005205413 >> 172.21.15.29:0/1005205413 conn(0x7fb0900cf570 msgr2=0x7fb0900cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:04.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.670+0000 7fb09653c640 1 -- 172.21.15.29:0/1005205413 shutdown_connections 2023-10-16T14:32:04.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.671+0000 7fb09653c640 1 -- 172.21.15.29:0/1005205413 wait complete. 2023-10-16T14:32:04.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.672+0000 7fb09653c640 1 Processor -- start 2023-10-16T14:32:04.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.672+0000 7fb09653c640 1 -- start start 2023-10-16T14:32:04.678 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.672+0000 7f266b411040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:32:04.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.673+0000 7fb09653c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0900b8480 0x7fb09014de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.673+0000 7fb09653c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0900cded0 0x7fb0900c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.673+0000 7fb09653c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 0x7fb0900b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.673+0000 7fb09653c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900cb290 con 0x7fb09015cd70 2023-10-16T14:32:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.673+0000 7fb09653c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900ca680 con 0x7fb0900cded0 2023-10-16T14:32:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.673+0000 7fb09653c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0900ca960 con 0x7fb0900b8480 2023-10-16T14:32:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 0x7fb0900b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 0x7fb0900b62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50792/0 (socket says 172.21.15.29:50792) 2023-10-16T14:32:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 -- 172.21.15.29:0/3544202512 learned_addr learned my addr 172.21.15.29:0/3544202512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb094d39640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0900cded0 0x7fb0900c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb09553a640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0900b8480 0x7fb09014de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 -- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0900b8480 msgr2=0x7fb09014de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0900b8480 0x7fb09014de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 -- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0900cded0 msgr2=0x7fb0900c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0900cded0 0x7fb0900c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.674+0000 7fb095d3b640 1 -- 172.21.15.29:0/3544202512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088009a70 con 0x7fb09015cd70 2023-10-16T14:32:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.675+0000 7fb095d3b640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 0x7fb0900b62b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb088009e80 tx=0x7fb088004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.675+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088040650 con 0x7fb09015cd70 2023-10-16T14:32:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.675+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0880407b0 con 0x7fb09015cd70 2023-10-16T14:32:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.675+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0900cac40 con 0x7fb09015cd70 2023-10-16T14:32:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.675+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088040b60 con 0x7fb09015cd70 2023-10-16T14:32:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.675+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09015a000 con 0x7fb09015cd70 2023-10-16T14:32:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.676+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0900cb440 con 0x7fb09015cd70 2023-10-16T14:32:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.677+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb088040cc0 con 0x7fb09015cd70 2023-10-16T14:32:04.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.678+0000 7fb085ffb640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fb07406af90 0x7fb07406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:04.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.678+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fb0880ac660 con 0x7fb09015cd70 2023-10-16T14:32:04.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.678+0000 7fb09553a640 1 -- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fb07406af90 msgr2=0x7fb07406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:04.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.679+0000 7fb09553a640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fb07406af90 0x7fb07406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:04.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.683+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0900cb440 con 0x7fb09015cd70 2023-10-16T14:32:04.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.727+0000 7efd13645040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:04.733 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.730+0000 7f266b411040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:32:04.783 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:04.779+0000 7f266b411040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:32:04.783 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.780+0000 7efd13645040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:32:04.843 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.840+0000 7efd13645040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:32:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.876+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb09004e9b0 con 0x7fb09015cd70 2023-10-16T14:32:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.877+0000 7fb085ffb640 1 -- 172.21.15.29:0/3544202512 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v68) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0880773d0 con 0x7fb09015cd70 2023-10-16T14:32:04.880 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:32:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fb07406af90 msgr2=0x7fb07406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fb07406af90 0x7fb07406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 msgr2=0x7fb0900b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 0x7fb0900b62b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb088009e80 tx=0x7fb088004a40 comp rx=0 tx=0).stop 2023-10-16T14:32:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 shutdown_connections 2023-10-16T14:32:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fb07406af90 0x7fb07406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb09015cd70 0x7fb0900b62b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0900cded0 0x7fb0900c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 --2- 172.21.15.29:0/3544202512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0900b8480 0x7fb09014de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.879+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 >> 172.21.15.29:0/3544202512 conn(0x7fb0900cf570 msgr2=0x7fb0900b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.880+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 shutdown_connections 2023-10-16T14:32:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:04.880+0000 7fb09653c640 1 -- 172.21.15.29:0/3544202512 wait complete. 2023-10-16T14:32:04.897 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:04.894+0000 7efd13645040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:32:04.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:32:05.158 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.155+0000 7f266b411040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:32:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.159+0000 7f8d8a68a640 1 Processor -- start 2023-10-16T14:32:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.159+0000 7f8d8a68a640 1 -- start start 2023-10-16T14:32:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.161+0000 7f8d8a68a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 0x7f8d840c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.161+0000 7f8d8a68a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8d840ce280 0x7f8d840ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.161+0000 7f8d8a68a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d840acd70 con 0x7f8d84065ab0 2023-10-16T14:32:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.161+0000 7f8d8a68a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d840ac4a0 con 0x7f8d840ce280 2023-10-16T14:32:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.161+0000 7f8d8a68a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d84065f40 con 0x7f8d8406d220 2023-10-16T14:32:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.162+0000 7f8d88e87640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8d84065ab0 0x7f8d840cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58632/0 (socket says 172.21.15.29:58632) 2023-10-16T14:32:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.162+0000 7f8d88e87640 1 -- 172.21.15.29:0/1438646907 learned_addr learned my addr 172.21.15.29:0/1438646907 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.162+0000 7f8d89688640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 0x7f8d840c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.162+0000 7f8d89e89640 1 --2- 172.21.15.29:0/1438646907 >> v2:172.21.15.170:3300/0 conn(0x7f8d840ce280 0x7f8d840ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.162+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033592445 0 0) 0x7f8d840acd70 con 0x7f8d84065ab0 2023-10-16T14:32:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.162+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70003520 con 0x7f8d84065ab0 2023-10-16T14:32:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615935553 0 0) 0x7f8d70003520 con 0x7f8d84065ab0 2023-10-16T14:32:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 msgr2=0x7f8d840c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 0x7f8d840c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 >> v2:172.21.15.170:3300/0 conn(0x7f8d840ce280 msgr2=0x7f8d840ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 --2- 172.21.15.29:0/1438646907 >> v2:172.21.15.170:3300/0 conn(0x7f8d840ce280 0x7f8d840ad200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d840ac720 con 0x7f8d84065ab0 2023-10-16T14:32:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8d74001670 con 0x7f8d84065ab0 2023-10-16T14:32:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 >> v1:172.21.15.29:6789/0 conn(0x7f8d84065ab0 legacy=0x7f8d840cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d700047c0 0x7f8d70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 0x7f8d70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840ac4a0 con 0x7f8d70003970 2023-10-16T14:32:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d84065f40 con 0x7f8d700047c0 2023-10-16T14:32:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7000b980 con 0x7f8d8406d220 2023-10-16T14:32:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.163+0000 7f8d89688640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.164+0000 7f8d89688640 1 -- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 msgr2=0x7f8d70005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.164+0000 7f8d89688640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8406d220 0x7f8d70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.164+0000 7f8d89688640 1 -- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d700047c0 msgr2=0x7f8d70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.164+0000 7f8d89688640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d700047c0 0x7f8d70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.164+0000 7f8d89688640 1 -- 172.21.15.29:0/1438646907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70003520 con 0x7f8d70003970 2023-10-16T14:32:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.165+0000 7f8d89688640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d70003d90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009ac0 tx=0x7f8d700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d62808b86413cad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.165+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8002c6d0 con 0x7f8d70003970 2023-10-16T14:32:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.165+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8002c830 con 0x7f8d70003970 2023-10-16T14:32:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.165+0000 7f8d7bfff640 1 -- 172.21.15.29:0/1438646907 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8002cbc0 con 0x7f8d70003970 2023-10-16T14:32:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d8a68a640 1 -- 172.21.15.29:0/1438646907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 msgr2=0x7f8d70003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/1438646907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d70003d90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009ac0 tx=0x7f8d700057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d89e89640 1 -- 172.21.15.29:0/1438646907 reap_dead start 2023-10-16T14:32:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d8a68a640 1 -- 172.21.15.29:0/1438646907 shutdown_connections 2023-10-16T14:32:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d8a68a640 1 -- 172.21.15.29:0/1438646907 >> 172.21.15.29:0/1438646907 conn(0x7f8d840b0a80 msgr2=0x7f8d840b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d8a68a640 1 -- 172.21.15.29:0/1438646907 shutdown_connections 2023-10-16T14:32:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.166+0000 7f8d8a68a640 1 -- 172.21.15.29:0/1438646907 wait complete. 2023-10-16T14:32:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.168+0000 7f8d8a68a640 1 Processor -- start 2023-10-16T14:32:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.168+0000 7f8d8a68a640 1 -- start start 2023-10-16T14:32:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d8a68a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d84148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d8a68a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8415f460 0x7f8d8415f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d8a68a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8415d470 0x7f8d8415d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d8a68a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8415fe20 con 0x7f8d70003970 2023-10-16T14:32:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d8a68a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8415de30 con 0x7f8d8415f460 2023-10-16T14:32:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d8a68a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8406b200 con 0x7f8d8415d470 2023-10-16T14:32:05.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d89688640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d84148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d89688640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d84148800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50808/0 (socket says 172.21.15.29:50808) 2023-10-16T14:32:05.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.169+0000 7f8d89688640 1 -- 172.21.15.29:0/332840797 learned_addr learned my addr 172.21.15.29:0/332840797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:05.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89e89640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8415d470 0x7f8d8415d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d88e87640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8415f460 0x7f8d8415f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89688640 1 -- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8415d470 msgr2=0x7f8d8415d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89688640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8415d470 0x7f8d8415d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89688640 1 -- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8415f460 msgr2=0x7f8d8415f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89688640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8415f460 0x7f8d8415f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89688640 1 -- 172.21.15.29:0/332840797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d84160020 con 0x7f8d70003970 2023-10-16T14:32:05.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.170+0000 7f8d89688640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d84148800 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8d8002c2d0 tx=0x7f8d80004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:05.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.171+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80003810 con 0x7f8d70003970 2023-10-16T14:32:05.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.171+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d80003970 con 0x7f8d70003970 2023-10-16T14:32:05.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.171+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80003ce0 con 0x7f8d70003970 2023-10-16T14:32:05.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.171+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d84160300 con 0x7f8d70003970 2023-10-16T14:32:05.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.171+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8406ae90 con 0x7f8d70003970 2023-10-16T14:32:05.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.172+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8406b050 con 0x7f8d70003970 2023-10-16T14:32:05.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.173+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f8d80040660 con 0x7f8d70003970 2023-10-16T14:32:05.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.174+0000 7f8d79ffb640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 0x7f8d6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.174+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f8d800ab950 con 0x7f8d70003970 2023-10-16T14:32:05.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.174+0000 7f8d88e87640 1 -- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 msgr2=0x7f8d6806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:05.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.174+0000 7f8d88e87640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 0x7f8d6806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:05.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.179+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d80076880 con 0x7f8d70003970 2023-10-16T14:32:05.320 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:05.317+0000 7efd13645040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:32:05.330 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.326+0000 7f266b411040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:32:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.375+0000 7f8d88e87640 1 -- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 msgr2=0x7f8d6806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:05.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.375+0000 7f8d88e87640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 0x7f8d6806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:05.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.383+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d8404e9b0 con 0x7f8d70003970 2023-10-16T14:32:05.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.384+0000 7f8d79ffb640 1 -- 172.21.15.29:0/332840797 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d800044e0 con 0x7f8d70003970 2023-10-16T14:32:05.387 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:32:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.386+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 msgr2=0x7f8d6806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.386+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 0x7f8d6806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 msgr2=0x7f8d84148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d84148800 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8d8002c2d0 tx=0x7f8d80004730 comp rx=0 tx=0).stop 2023-10-16T14:32:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 shutdown_connections 2023-10-16T14:32:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f8d6806af90 0x7f8d6806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8415d470 0x7f8d8415d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8415f460 0x7f8d8415f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 --2- 172.21.15.29:0/332840797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d70003970 0x7f8d84148800 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 >> 172.21.15.29:0/332840797 conn(0x7f8d840b0a80 msgr2=0x7f8d840cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 shutdown_connections 2023-10-16T14:32:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.387+0000 7f8d8a68a640 1 -- 172.21.15.29:0/332840797 wait complete. 2023-10-16T14:32:05.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:32:05.447 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.443+0000 7f266b411040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:32:05.498 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.494+0000 7f266b411040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:32:05.499 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:05.497+0000 7efd13645040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:32:05.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.600+0000 7fc880947640 1 Processor -- start 2023-10-16T14:32:05.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.600+0000 7fc880947640 1 -- start start 2023-10-16T14:32:05.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc880947640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c159950 0x7fc87c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc880947640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc87c070450 0x7fc87c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc880947640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc87c06c100 0x7fc87c06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc880947640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c15e490 con 0x7fc87c06c100 2023-10-16T14:32:05.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc880947640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c070df0 con 0x7fc87c070450 2023-10-16T14:32:05.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc880947640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c15aef0 con 0x7fc87c159950 2023-10-16T14:32:05.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc87bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc87c06c100 0x7fc87c06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc87bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc87c06c100 0x7fc87c06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50810/0 (socket says 172.21.15.29:50810) 2023-10-16T14:32:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.602+0000 7fc87bfff640 1 -- 172.21.15.29:0/2265881238 learned_addr learned my addr 172.21.15.29:0/2265881238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc87b7fe640 1 --2- 172.21.15.29:0/2265881238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c159950 0x7fc87c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc873fff640 1 --2- 172.21.15.29:0/2265881238 >> v2:172.21.15.170:3300/0 conn(0x7fc87c070450 0x7fc87c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.608 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.602+0000 7f266b411040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:05.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc87bfff640 1 -- 172.21.15.29:0/2265881238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c159950 msgr2=0x7fc87c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:05.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc87bfff640 1 --2- 172.21.15.29:0/2265881238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c159950 0x7fc87c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc87bfff640 1 -- 172.21.15.29:0/2265881238 >> v2:172.21.15.170:3300/0 conn(0x7fc87c070450 msgr2=0x7fc87c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:05.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc87bfff640 1 --2- 172.21.15.29:0/2265881238 >> v2:172.21.15.170:3300/0 conn(0x7fc87c070450 0x7fc87c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.603+0000 7fc87bfff640 1 -- 172.21.15.29:0/2265881238 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c15d910 con 0x7fc87c06c100 2023-10-16T14:32:05.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.605+0000 7fc87bfff640 1 --2- 172.21.15.29:0/2265881238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c06c570 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0719e0 tx=0x7fc87402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355cde44f29648fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:05.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.605+0000 7fc87affd640 1 -- 172.21.15.29:0/2265881238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87402c9e0 con 0x7fc87c06c100 2023-10-16T14:32:05.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.605+0000 7fc87affd640 1 -- 172.21.15.29:0/2265881238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87402cb40 con 0x7fc87c06c100 2023-10-16T14:32:05.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.605+0000 7fc87affd640 1 -- 172.21.15.29:0/2265881238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8740343f0 con 0x7fc87c06c100 2023-10-16T14:32:05.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 -- 172.21.15.29:0/2265881238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 msgr2=0x7fc87c06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 --2- 172.21.15.29:0/2265881238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c06c570 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0719e0 tx=0x7fc87402be40 comp rx=0 tx=0).stop 2023-10-16T14:32:05.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 -- 172.21.15.29:0/2265881238 shutdown_connections 2023-10-16T14:32:05.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 --2- 172.21.15.29:0/2265881238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c06c570 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 --2- 172.21.15.29:0/2265881238 >> v2:172.21.15.170:3300/0 conn(0x7fc87c070450 0x7fc87c0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 --2- 172.21.15.29:0/2265881238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c159950 0x7fc87c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 -- 172.21.15.29:0/2265881238 >> 172.21.15.29:0/2265881238 conn(0x7fc87c0ce6a0 msgr2=0x7fc87c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 -- 172.21.15.29:0/2265881238 shutdown_connections 2023-10-16T14:32:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.606+0000 7fc880947640 1 -- 172.21.15.29:0/2265881238 wait complete. 2023-10-16T14:32:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.608+0000 7fc880947640 1 Processor -- start 2023-10-16T14:32:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.608+0000 7fc880947640 1 -- start start 2023-10-16T14:32:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc880947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc880947640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c070450 0x7fc87c0ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc880947640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c159950 0x7fc87c0b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc880947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0c3bb0 con 0x7fc87c06c100 2023-10-16T14:32:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc880947640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c149f80 con 0x7fc87c159950 2023-10-16T14:32:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc880947640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c14a260 con 0x7fc87c070450 2023-10-16T14:32:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc87b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc87b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c1f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50826/0 (socket says 172.21.15.29:50826) 2023-10-16T14:32:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc87b7fe640 1 -- 172.21.15.29:0/2826509950 learned_addr learned my addr 172.21.15.29:0/2826509950 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc873fff640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c070450 0x7fc87c0ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.609+0000 7fc87bfff640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c159950 0x7fc87c0b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87b7fe640 1 -- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c070450 msgr2=0x7fc87c0ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87b7fe640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c070450 0x7fc87c0ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87b7fe640 1 -- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c159950 msgr2=0x7fc87c0b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87b7fe640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c159950 0x7fc87c0b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87b7fe640 1 -- 172.21.15.29:0/2826509950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc874009b30 con 0x7fc87c06c100 2023-10-16T14:32:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc873fff640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c070450 0x7fc87c0ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87bfff640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c159950 0x7fc87c0b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc87b7fe640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c1f40f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00d800 tx=0x7fc86c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c00b6c0 con 0x7fc87c06c100 2023-10-16T14:32:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c00b820 con 0x7fc87c06c100 2023-10-16T14:32:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c00bbb0 con 0x7fc87c06c100 2023-10-16T14:32:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c0c3e20 con 0x7fc87c06c100 2023-10-16T14:32:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.610+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c147630 con 0x7fc87c06c100 2023-10-16T14:32:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.612+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c0c3a10 con 0x7fc87c06c100 2023-10-16T14:32:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.612+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fc86c00bd10 con 0x7fc87c06c100 2023-10-16T14:32:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.613+0000 7fc878ff9640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 0x7fc85c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.613+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fc86c089f20 con 0x7fc87c06c100 2023-10-16T14:32:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.613+0000 7fc873fff640 1 -- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 msgr2=0x7fc85c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:05.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.613+0000 7fc873fff640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 0x7fc85c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:05.622 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:05.618+0000 7efd13645040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:32:05.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.622+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc87c0c3a10 con 0x7fc87c06c100 2023-10-16T14:32:05.676 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:05.673+0000 7efd13645040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:32:05.713 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.709+0000 7f266b411040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:32:05.790 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:05.787+0000 7efd13645040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.814+0000 7fc873fff640 1 -- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 msgr2=0x7fc85c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.814+0000 7fc873fff640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 0x7fc85c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.829+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc87c0c3a10 con 0x7fc87c06c100 2023-10-16T14:32:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.830+0000 7fc878ff9640 1 -- 172.21.15.29:0/2826509950 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v69) v1 ==== 80+0+478 (secure 0 0 0) 0x7fc87c0c3a10 con 0x7fc87c06c100 2023-10-16T14:32:05.834 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3247649762 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.834 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2698007731 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.834 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/4263280508 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.835 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2380256585 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.835 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3575752004 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.836 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3937243709 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.836 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/3423886595 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.836 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/3423886595 2023-10-17T14:30:30.217790+0000 2023-10-16T14:32:05.838 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T14:32:05.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 msgr2=0x7fc85c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 0x7fc85c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 msgr2=0x7fc87c1f40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c1f40f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00d800 tx=0x7fc86c00dcc0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 shutdown_connections 2023-10-16T14:32:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc85c06af90 0x7fc85c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c159950 0x7fc87c0b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c070450 0x7fc87c0ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 --2- 172.21.15.29:0/2826509950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c06c100 0x7fc87c1f40f0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 >> 172.21.15.29:0/2826509950 conn(0x7fc87c0ce6a0 msgr2=0x7fc87c0cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 shutdown_connections 2023-10-16T14:32:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:05.833+0000 7fc880947640 1 -- 172.21.15.29:0/2826509950 wait complete. 2023-10-16T14:32:05.853 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:32:05.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:05.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:32:05.869 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.865+0000 7f266b411040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:05.906 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:05.904+0000 7efd13645040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:32:05.917 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:05.914+0000 7f266b411040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:05.936 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:32:05.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:05.937 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:32:05.983 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:05.983 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:06.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:06.039 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:06.039 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:06.040 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:06.036+0000 7f266b411040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:32:06.079 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:06.076+0000 7efd13645040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:06.095 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:32:06.096 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:06.096 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:06.131 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:06.128+0000 7efd13645040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:06.152 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:06.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:06.270 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:06.267+0000 7efd13645040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:32:06.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.358+0000 7f88394da640 1 Processor -- start 2023-10-16T14:32:06.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.359+0000 7f88394da640 1 -- start start 2023-10-16T14:32:06.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.360+0000 7f88394da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340b0b20 0x7f88341615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.360+0000 7f88394da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883415e3e0 con 0x7f883415d3f0 2023-10-16T14:32:06.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.360+0000 7f88394da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88340cd4a0 con 0x7f88340cd0c0 2023-10-16T14:32:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.360+0000 7f88394da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883415e100 con 0x7f88340b0b20 2023-10-16T14:32:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.360+0000 7f88337fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f883415d3f0 0x7f883415d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58634/0 (socket says 172.21.15.29:58634) 2023-10-16T14:32:06.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.360+0000 7f88337fe640 1 -- 172.21.15.29:0/1745661478 learned_addr learned my addr 172.21.15.29:0/1745661478 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f882a7fc640 1 --2- 172.21.15.29:0/1745661478 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340b0b20 0x7f88341615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f88327fc640 1 -- 172.21.15.29:0/1745661478 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476202081 0 0) 0x7f883415e3e0 con 0x7f883415d3f0 2023-10-16T14:32:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f88327fc640 1 -- 172.21.15.29:0/1745661478 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8820003520 con 0x7f883415d3f0 2023-10-16T14:32:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f882a7fc640 1 -- 172.21.15.29:0/1745661478 >> v1:172.21.15.170:6789/0 conn(0x7f88340cd0c0 legacy=0x7f88340b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f882a7fc640 1 -- 172.21.15.29:0/1745661478 >> v1:172.21.15.29:6789/0 conn(0x7f883415d3f0 legacy=0x7f883415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f88327fc640 1 -- 172.21.15.29:0/1745661478 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997046735 0 0) 0x7f8820003520 con 0x7f883415d3f0 2023-10-16T14:32:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.361+0000 7f882a7fc640 1 -- 172.21.15.29:0/1745661478 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8834160b10 con 0x7f88340b0b20 2023-10-16T14:32:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.363+0000 7f882a7fc640 1 --2- 172.21.15.29:0/1745661478 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340b0b20 0x7f88341615f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f881c0097d0 tx=0x7f881c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3198725358927ccf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.363+0000 7f88327fc640 1 -- 172.21.15.29:0/1745661478 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c0019f0 con 0x7f88340b0b20 2023-10-16T14:32:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88327fc640 1 -- 172.21.15.29:0/1745661478 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c02e050 con 0x7f88340b0b20 2023-10-16T14:32:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88327fc640 1 -- 172.21.15.29:0/1745661478 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c030710 con 0x7f88340b0b20 2023-10-16T14:32:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 -- 172.21.15.29:0/1745661478 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340b0b20 msgr2=0x7f88341615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 --2- 172.21.15.29:0/1745661478 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340b0b20 0x7f88341615f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f881c0097d0 tx=0x7f881c02b770 comp rx=0 tx=0).stop 2023-10-16T14:32:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 -- 172.21.15.29:0/1745661478 shutdown_connections 2023-10-16T14:32:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 --2- 172.21.15.29:0/1745661478 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340b0b20 0x7f88341615f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 -- 172.21.15.29:0/1745661478 >> 172.21.15.29:0/1745661478 conn(0x7f88340cb410 msgr2=0x7f88340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:06.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 -- 172.21.15.29:0/1745661478 shutdown_connections 2023-10-16T14:32:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.364+0000 7f88394da640 1 -- 172.21.15.29:0/1745661478 wait complete. 2023-10-16T14:32:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.366+0000 7f88394da640 1 Processor -- start 2023-10-16T14:32:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.366+0000 7f88394da640 1 -- start start 2023-10-16T14:32:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88394da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88340b0b20 0x7f88340adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88394da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340cd0c0 0x7f88340b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88394da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 0x7f8834153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88394da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88340c4150 con 0x7f883415d3f0 2023-10-16T14:32:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88394da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88340c3860 con 0x7f88340b0b20 2023-10-16T14:32:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88394da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88340b2ce0 con 0x7f88340cd0c0 2023-10-16T14:32:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 0x7f8834153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 0x7f8834153130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50838/0 (socket says 172.21.15.29:50838) 2023-10-16T14:32:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.367+0000 7f88337fe640 1 -- 172.21.15.29:0/2182782407 learned_addr learned my addr 172.21.15.29:0/2182782407 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f882a7fc640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340cd0c0 0x7f88340b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f8832ffd640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88340b0b20 0x7f88340adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f88337fe640 1 -- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340cd0c0 msgr2=0x7f88340b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f88337fe640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340cd0c0 0x7f88340b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f88337fe640 1 -- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88340b0b20 msgr2=0x7f88340adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f88337fe640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88340b0b20 0x7f88340adab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f88337fe640 1 -- 172.21.15.29:0/2182782407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c009400 con 0x7f883415d3f0 2023-10-16T14:32:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.368+0000 7f88337fe640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 0x7f8834153130 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f882c001830 tx=0x7f882c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.369+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c012400 con 0x7f883415d3f0 2023-10-16T14:32:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.369+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c012560 con 0x7f883415d3f0 2023-10-16T14:32:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.369+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88340c3b00 con 0x7f883415d3f0 2023-10-16T14:32:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.369+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c012910 con 0x7f883415d3f0 2023-10-16T14:32:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.369+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88340b29d0 con 0x7f883415d3f0 2023-10-16T14:32:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.370+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883404e160 con 0x7f883415d3f0 2023-10-16T14:32:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.372+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f882c012a70 con 0x7f883415d3f0 2023-10-16T14:32:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.373+0000 7f882bfff640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 0x7f881406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.373+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f882c08a140 con 0x7f883415d3f0 2023-10-16T14:32:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.373+0000 7f8832ffd640 1 -- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 msgr2=0x7f881406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.374+0000 7f8832ffd640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 0x7f881406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.378+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f882c054e60 con 0x7f883415d3f0 2023-10-16T14:32:06.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.575+0000 7f8832ffd640 1 -- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 msgr2=0x7f881406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:06.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.575+0000 7f8832ffd640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 0x7f881406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.605+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88340b4910 con 0x7f883415d3f0 2023-10-16T14:32:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.606+0000 7f882bfff640 1 -- 172.21.15.29:0/2182782407 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f882c058480 con 0x7f883415d3f0 2023-10-16T14:32:06.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:06.609 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:30:15.728624+0000","modified":"2023-10-16T14:30:18.806702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1869070988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1869070988},{"type":"v1","addr":"172.21.15.29:6835","nonce":1869070988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T14:32:06.610 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-16T14:32:06.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 msgr2=0x7f881406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:06.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 0x7f881406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 msgr2=0x7f8834153130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 0x7f8834153130 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f882c001830 tx=0x7f882c00a430 comp rx=0 tx=0).stop 2023-10-16T14:32:06.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 shutdown_connections 2023-10-16T14:32:06.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7f881406af90 0x7f881406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f883415d3f0 0x7f8834153130 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88340cd0c0 0x7f88340b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 --2- 172.21.15.29:0/2182782407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88340b0b20 0x7f88340adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.608+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 >> 172.21.15.29:0/2182782407 conn(0x7f88340cb410 msgr2=0x7f88340ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:06.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.609+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 shutdown_connections 2023-10-16T14:32:06.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.609+0000 7f88394da640 1 -- 172.21.15.29:0/2182782407 wait complete. 2023-10-16T14:32:06.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.834+0000 7ff4b759e640 1 Processor -- start 2023-10-16T14:32:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.835+0000 7ff4b759e640 1 -- start start 2023-10-16T14:32:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cbe0 0x7ff4b80d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4b80b4c80 0x7ff4b80b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b80d15a0 con 0x7ff4b815cfc0 2023-10-16T14:32:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b80d1880 con 0x7ff4b80b4c80 2023-10-16T14:32:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80d1b60 con 0x7ff4b815cbe0 2023-10-16T14:32:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4b80b4c80 0x7ff4b80b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b5d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff4b815cfc0 0x7ff4b80b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58644/0 (socket says 172.21.15.29:58644) 2023-10-16T14:32:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.836+0000 7ff4b5d9b640 1 -- 172.21.15.29:0/2957226783 learned_addr learned my addr 172.21.15.29:0/2957226783 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b659c640 1 --2- 172.21.15.29:0/2957226783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cbe0 0x7ff4b80d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b559a640 1 -- 172.21.15.29:0/2957226783 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275863866 0 0) 0x7ff4b80d15a0 con 0x7ff4b815cfc0 2023-10-16T14:32:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b559a640 1 -- 172.21.15.29:0/2957226783 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a0003520 con 0x7ff4b815cfc0 2023-10-16T14:32:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b6d9d640 1 -- 172.21.15.29:0/2957226783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cbe0 msgr2=0x7ff4b80d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b6d9d640 1 --2- 172.21.15.29:0/2957226783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cbe0 0x7ff4b80d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b6d9d640 1 -- 172.21.15.29:0/2957226783 >> v1:172.21.15.29:6789/0 conn(0x7ff4b815cfc0 legacy=0x7ff4b80b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.837+0000 7ff4b6d9d640 1 -- 172.21.15.29:0/2957226783 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b80d1e40 con 0x7ff4b80b4c80 2023-10-16T14:32:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.838+0000 7ff4b6d9d640 1 --2- 172.21.15.29:0/2957226783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80b5c20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009d90 tx=0x7ff4b002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60f39662ac97731 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.838+0000 7ff4b559a640 1 -- 172.21.15.29:0/2957226783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b002cce0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.838+0000 7ff4b559a640 1 -- 172.21.15.29:0/2957226783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b002ce40 con 0x7ff4b80b4c80 2023-10-16T14:32:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.838+0000 7ff4b559a640 1 -- 172.21.15.29:0/2957226783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0034660 con 0x7ff4b80b4c80 2023-10-16T14:32:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 -- 172.21.15.29:0/2957226783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 msgr2=0x7ff4b80b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 --2- 172.21.15.29:0/2957226783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80b5c20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009d90 tx=0x7ff4b002bf30 comp rx=0 tx=0).stop 2023-10-16T14:32:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 -- 172.21.15.29:0/2957226783 shutdown_connections 2023-10-16T14:32:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 --2- 172.21.15.29:0/2957226783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80b5c20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 --2- 172.21.15.29:0/2957226783 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cbe0 0x7ff4b80d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 -- 172.21.15.29:0/2957226783 >> 172.21.15.29:0/2957226783 conn(0x7ff4b80cf570 msgr2=0x7ff4b80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:06.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 -- 172.21.15.29:0/2957226783 shutdown_connections 2023-10-16T14:32:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.839+0000 7ff4b759e640 1 -- 172.21.15.29:0/2957226783 wait complete. 2023-10-16T14:32:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.841+0000 7ff4b759e640 1 Processor -- start 2023-10-16T14:32:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.841+0000 7ff4b759e640 1 -- start start 2023-10-16T14:32:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4b815cbe0 0x7ff4b80c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cfc0 0x7ff4b80c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80c6310 con 0x7ff4b80b4c80 2023-10-16T14:32:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b815f150 con 0x7ff4b815cbe0 2023-10-16T14:32:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b815f430 con 0x7ff4b815cfc0 2023-10-16T14:32:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50866/0 (socket says 172.21.15.29:50866) 2023-10-16T14:32:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.842+0000 7ff4b659c640 1 -- 172.21.15.29:0/4044268681 learned_addr learned my addr 172.21.15.29:0/4044268681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b659c640 1 -- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cfc0 msgr2=0x7ff4b80c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b659c640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cfc0 0x7ff4b80c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b659c640 1 -- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4b815cbe0 msgr2=0x7ff4b80c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b659c640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4b815cbe0 0x7ff4b80c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b659c640 1 -- 172.21.15.29:0/4044268681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0009a70 con 0x7ff4b80b4c80 2023-10-16T14:32:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b659c640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80c9e30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800d800 tx=0x7ff4a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a800b6c0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a800b820 con 0x7ff4b80b4c80 2023-10-16T14:32:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a800bbb0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b815f710 con 0x7ff4b80b4c80 2023-10-16T14:32:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.843+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b815fcd0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.845+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b804e9b0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.846+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff4a800bd10 con 0x7ff4b80b4c80 2023-10-16T14:32:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.846+0000 7ff4a6ffd640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 0x7ff49806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.847+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6183+0+0 (secure 0 0 0) 0x7ff4a8089fb0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.847+0000 7ff4b5d9b640 1 -- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 msgr2=0x7ff49806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.847+0000 7ff4b5d9b640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 0x7ff49806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:06.851+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff4a8054cd0 con 0x7ff4b80b4c80 2023-10-16T14:32:06.872 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:06.868+0000 7f266b411040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:32:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.045+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4b80b3280 con 0x7ff4b80b4c80 2023-10-16T14:32:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.046+0000 7ff4a6ffd640 1 -- 172.21.15.29:0/4044268681 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17244 (secure 0 0 0) 0x7ff4a80582f0 con 0x7ff4b80b4c80 2023-10-16T14:32:07.049 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:07.049 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":70,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:06.219930+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3247649762":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2698007731":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4263280508":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2380256585":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3575752004":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3937243709":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6833/3423886595":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6832/3423886595":"2023-10-17T14:30:30.217790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.048+0000 7ff4b5d9b640 1 -- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 msgr2=0x7ff49806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.048+0000 7ff4b5d9b640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 0x7ff49806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 msgr2=0x7ff49806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 0x7ff49806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 msgr2=0x7ff4b80c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80c9e30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800d800 tx=0x7ff4a800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 shutdown_connections 2023-10-16T14:32:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7ff49806af90 0x7ff49806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4b815cfc0 0x7ff4b80c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4b815cbe0 0x7ff4b80c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 --2- 172.21.15.29:0/4044268681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4b80b4c80 0x7ff4b80c9e30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 >> 172.21.15.29:0/4044268681 conn(0x7ff4b80cf570 msgr2=0x7ff4b80b5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 shutdown_connections 2023-10-16T14:32:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.049+0000 7ff4b759e640 1 -- 172.21.15.29:0/4044268681 wait complete. 2023-10-16T14:32:07.068 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:30:13.584764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:30:14.740756+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:07.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:07.113 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:07.110+0000 7efd13645040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:32:07.164 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.160+0000 7f266b411040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:32:07.221 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.217+0000 7f266b411040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:32:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.276+0000 7fc91364b640 1 Processor -- start 2023-10-16T14:32:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.276+0000 7fc91364b640 1 -- start start 2023-10-16T14:32:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc91364b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0cf710 0x7fc90c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc91364b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc90c0cfaf0 0x7fc90c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc91364b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc90c0b38e0 0x7fc90c0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc91364b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90c0d3c60 con 0x7fc90c0b38e0 2023-10-16T14:32:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc91364b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90c0d3980 con 0x7fc90c0cfaf0 2023-10-16T14:32:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc91364b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0ce630 con 0x7fc90c0cf710 2023-10-16T14:32:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc911be8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc90c0b38e0 0x7fc90c0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc911be8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc90c0b38e0 0x7fc90c0d0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50880/0 (socket says 172.21.15.29:50880) 2023-10-16T14:32:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc911be8640 1 -- 172.21.15.29:0/1524940306 learned_addr learned my addr 172.21.15.29:0/1524940306 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:07.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc910be6640 1 --2- 172.21.15.29:0/1524940306 >> v2:172.21.15.170:3300/0 conn(0x7fc90c0cfaf0 0x7fc90c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.278+0000 7fc9113e7640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0cf710 0x7fc90c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.279+0000 7fc911be8640 1 -- 172.21.15.29:0/1524940306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0cf710 msgr2=0x7fc90c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:07.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.279+0000 7fc911be8640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0cf710 0x7fc90c0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.279+0000 7fc911be8640 1 -- 172.21.15.29:0/1524940306 >> v2:172.21.15.170:3300/0 conn(0x7fc90c0cfaf0 msgr2=0x7fc90c0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:07.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.279+0000 7fc911be8640 1 --2- 172.21.15.29:0/1524940306 >> v2:172.21.15.170:3300/0 conn(0x7fc90c0cfaf0 0x7fc90c0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.279+0000 7fc911be8640 1 -- 172.21.15.29:0/1524940306 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c0ce910 con 0x7fc90c0b38e0 2023-10-16T14:32:07.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.279+0000 7fc9113e7640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0cf710 0x7fc90c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:07.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.280+0000 7fc911be8640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0b38e0 0x7fc90c0d0aa0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc90c066a90 tx=0x7fc90402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c855134f71a046a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:07.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.280+0000 7fc903fff640 1 -- 172.21.15.29:0/1524940306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90402c9e0 con 0x7fc90c0b38e0 2023-10-16T14:32:07.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.280+0000 7fc903fff640 1 -- 172.21.15.29:0/1524940306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90402cb40 con 0x7fc90c0b38e0 2023-10-16T14:32:07.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.280+0000 7fc903fff640 1 -- 172.21.15.29:0/1524940306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9040343f0 con 0x7fc90c0b38e0 2023-10-16T14:32:07.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 -- 172.21.15.29:0/1524940306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0b38e0 msgr2=0x7fc90c0d0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0b38e0 0x7fc90c0d0aa0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc90c066a90 tx=0x7fc90402be40 comp rx=0 tx=0).stop 2023-10-16T14:32:07.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 -- 172.21.15.29:0/1524940306 shutdown_connections 2023-10-16T14:32:07.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0b38e0 0x7fc90c0d0aa0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 --2- 172.21.15.29:0/1524940306 >> v2:172.21.15.170:3300/0 conn(0x7fc90c0cfaf0 0x7fc90c0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 --2- 172.21.15.29:0/1524940306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0cf710 0x7fc90c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 -- 172.21.15.29:0/1524940306 >> 172.21.15.29:0/1524940306 conn(0x7fc90c0d2cc0 msgr2=0x7fc90c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:07.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 -- 172.21.15.29:0/1524940306 shutdown_connections 2023-10-16T14:32:07.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.281+0000 7fc91364b640 1 -- 172.21.15.29:0/1524940306 wait complete. 2023-10-16T14:32:07.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.283+0000 7fc91364b640 1 Processor -- start 2023-10-16T14:32:07.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.283+0000 7fc91364b640 1 -- start start 2023-10-16T14:32:07.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.284+0000 7fc91364b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0b38e0 0x7fc90c15e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.284+0000 7fc91364b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 0x7fc90c150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.284+0000 7fc91364b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc90c0cfaf0 0x7fc90c0b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.284+0000 7fc91364b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c06b4f0 con 0x7fc90c0cf710 2023-10-16T14:32:07.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.284+0000 7fc91364b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0c9f10 con 0x7fc90c0cfaf0 2023-10-16T14:32:07.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.284+0000 7fc91364b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0ca1f0 con 0x7fc90c0b38e0 2023-10-16T14:32:07.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 0x7fc90c150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 0x7fc90c150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50890/0 (socket says 172.21.15.29:50890) 2023-10-16T14:32:07.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 -- 172.21.15.29:0/3671532680 learned_addr learned my addr 172.21.15.29:0/3671532680 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:07.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc9113e7640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0b38e0 0x7fc90c15e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc911be8640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc90c0cfaf0 0x7fc90c0b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 -- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0b38e0 msgr2=0x7fc90c15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0b38e0 0x7fc90c15e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 -- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc90c0cfaf0 msgr2=0x7fc90c0b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc90c0cfaf0 0x7fc90c0b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.285+0000 7fc910be6640 1 -- 172.21.15.29:0/3671532680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904009b30 con 0x7fc90c0cf710 2023-10-16T14:32:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.286+0000 7fc910be6640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 0x7fc90c150d80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00d5f0 tx=0x7fc8fc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:07.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.286+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0125f0 con 0x7fc90c0cf710 2023-10-16T14:32:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.286+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc012750 con 0x7fc90c0cf710 2023-10-16T14:32:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.286+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc012b00 con 0x7fc90c0cf710 2023-10-16T14:32:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.286+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c0ca4d0 con 0x7fc90c0cf710 2023-10-16T14:32:07.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.286+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c06afe0 con 0x7fc90c0cf710 2023-10-16T14:32:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.287+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c0c9d70 con 0x7fc90c0cf710 2023-10-16T14:32:07.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.288+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fc8fc012c60 con 0x7fc90c0cf710 2023-10-16T14:32:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.290+0000 7fc901ffb640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc8f006af90 0x7fc8f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.290+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6183+0+0 (secure 0 0 0) 0x7fc8fc089d50 con 0x7fc90c0cf710 2023-10-16T14:32:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.290+0000 7fc9113e7640 1 -- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc8f006af90 msgr2=0x7fc8f006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/636880041 2023-10-16T14:32:07.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.290+0000 7fc9113e7640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc8f006af90 0x7fc8f006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.293+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc90c0c9d70 con 0x7fc90c0cf710 2023-10-16T14:32:07.340 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.336+0000 7f266b411040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:32:07.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.358+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 26) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc8fc054ed0 con 0x7fc90c0cf710 2023-10-16T14:32:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.358+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc8f006af90 msgr2=0x7fc8f006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:07.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.358+0000 7fc901ffb640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc8f006af90 0x7fc8f006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.430 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.426+0000 7f25f0e9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.430 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.426+0000 7f25f0e9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.430 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.426+0000 7f25f0e9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.430 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.426+0000 7f25f0e9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.430 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.426+0000 7f25f0e9f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.431 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.427+0000 7f25ede99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.431 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.427+0000 7f25ede99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.432 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.427+0000 7f25ede99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.432 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.427+0000 7f25ede99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.432 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:07.427+0000 7f25ede99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:07.445 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:07.443+0000 7efd13645040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:32:07.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.497+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc90c0c9d70 con 0x7fc90c0cf710 2023-10-16T14:32:07.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.498+0000 7fc901ffb640 1 -- 172.21.15.29:0/3671532680 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc90c0c9d70 con 0x7fc90c0cf710 2023-10-16T14:32:07.501 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 msgr2=0x7fc90c150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 0x7fc90c150d80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00d5f0 tx=0x7fc8fc00dab0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 shutdown_connections 2023-10-16T14:32:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:6832/636880041,v1:172.21.15.170:6833/636880041] conn(0x7fc8f006af90 0x7fc8f006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc90c0cf710 0x7fc90c150d80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc90c0cfaf0 0x7fc90c0b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 --2- 172.21.15.29:0/3671532680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc90c0b38e0 0x7fc90c15e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 >> 172.21.15.29:0/3671532680 conn(0x7fc90c0d2cc0 msgr2=0x7fc90c0d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 shutdown_connections 2023-10-16T14:32:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.500+0000 7fc91364b640 1 -- 172.21.15.29:0/3671532680 wait complete. 2023-10-16T14:32:07.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:07.504+0000 7efd13645040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:32:07.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:32:07.630 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:07.628+0000 7efd13645040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:32:07.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.766+0000 7f005aa1a640 1 Processor -- start 2023-10-16T14:32:07.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.766+0000 7f005aa1a640 1 -- start start 2023-10-16T14:32:07.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f005aa1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f00540caac0 0x7f00540ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f005aa1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00540cb030 con 0x7f005406a4b0 2023-10-16T14:32:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f005aa1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00540c91c0 con 0x7f005415ad70 2023-10-16T14:32:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f005aa1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00540c94a0 con 0x7f00540caac0 2023-10-16T14:32:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f004bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f005406a4b0 0x7f00540ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58652/0 (socket says 172.21.15.29:58652) 2023-10-16T14:32:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f004bfff640 1 -- 172.21.15.29:0/2547851543 learned_addr learned my addr 172.21.15.29:0/2547851543 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f0058fb7640 1 --2- 172.21.15.29:0/2547851543 >> v2:172.21.15.170:3300/0 conn(0x7f00540caac0 0x7f00540ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.768+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478462429 0 0) 0x7f00540cb030 con 0x7f005406a4b0 2023-10-16T14:32:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0040003520 con 0x7f005406a4b0 2023-10-16T14:32:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933730673 0 0) 0x7f0040003520 con 0x7f005406a4b0 2023-10-16T14:32:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 >> v2:172.21.15.170:3300/0 conn(0x7f00540caac0 msgr2=0x7f00540ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 --2- 172.21.15.29:0/2547851543 >> v2:172.21.15.170:3300/0 conn(0x7f00540caac0 0x7f00540ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 >> v1:172.21.15.170:6789/0 conn(0x7f005415ad70 legacy=0x7f0054069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540c9780 con 0x7f005406a4b0 2023-10-16T14:32:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f003c001670 con 0x7f005406a4b0 2023-10-16T14:32:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 >> v1:172.21.15.29:6789/0 conn(0x7f005406a4b0 legacy=0x7f00540ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400039c0 0x7f0040003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00400047c0 0x7f0040004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0040005180 0x7f004000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00540c94a0 con 0x7f00400047c0 2023-10-16T14:32:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004000c050 con 0x7f00400039c0 2023-10-16T14:32:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004000c210 con 0x7f0040005180 2023-10-16T14:32:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.769+0000 7f004bfff640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00400047c0 0x7f0040004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f0053fff640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400039c0 0x7f0040003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f004bfff640 1 -- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0040005180 msgr2=0x7f004000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f004bfff640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0040005180 0x7f004000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f004bfff640 1 -- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400039c0 msgr2=0x7f0040003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f004bfff640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400039c0 0x7f0040003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f004bfff640 1 -- 172.21.15.29:0/2547851543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040003520 con 0x7f00400047c0 2023-10-16T14:32:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.770+0000 7f0058fb7640 1 -- 172.21.15.29:0/2547851543 reap_dead start 2023-10-16T14:32:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.771+0000 7f004bfff640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00400047c0 0x7f0040004c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f003c00c190 tx=0x7f00400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7f8c268b984140 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.771+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c005ad0 con 0x7f00400047c0 2023-10-16T14:32:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.771+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c005c30 con 0x7f00400047c0 2023-10-16T14:32:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f00537fe640 1 -- 172.21.15.29:0/2547851543 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c02c7c0 con 0x7f00400047c0 2023-10-16T14:32:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 -- 172.21.15.29:0/2547851543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00400047c0 msgr2=0x7f0040004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00400047c0 0x7f0040004c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f003c00c190 tx=0x7f00400057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 -- 172.21.15.29:0/2547851543 shutdown_connections 2023-10-16T14:32:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 --2- 172.21.15.29:0/2547851543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00400047c0 0x7f0040004c40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 -- 172.21.15.29:0/2547851543 >> 172.21.15.29:0/2547851543 conn(0x7f00540ce9b0 msgr2=0x7f00540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 -- 172.21.15.29:0/2547851543 shutdown_connections 2023-10-16T14:32:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.772+0000 7f005aa1a640 1 -- 172.21.15.29:0/2547851543 wait complete. 2023-10-16T14:32:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.774+0000 7f005aa1a640 1 Processor -- start 2023-10-16T14:32:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.774+0000 7f005aa1a640 1 -- start start 2023-10-16T14:32:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.775+0000 7f005aa1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400047c0 0x7f00540c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.775+0000 7f005aa1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 0x7f0054151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.775+0000 7f005aa1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0054151e00 0x7f0054152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.775+0000 7f005aa1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00541527c0 con 0x7f00540c4c90 2023-10-16T14:32:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.775+0000 7f005aa1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00541516d0 con 0x7f00400047c0 2023-10-16T14:32:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.775+0000 7f005aa1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00540c8780 con 0x7f0054151e00 2023-10-16T14:32:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 0x7f0054151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 0x7f0054151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50908/0 (socket says 172.21.15.29:50908) 2023-10-16T14:32:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 -- 172.21.15.29:0/1020383395 learned_addr learned my addr 172.21.15.29:0/1020383395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f0053fff640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400047c0 0x7f00540c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f0058fb7640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0054151e00 0x7f0054152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 -- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0054151e00 msgr2=0x7f0054152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0054151e00 0x7f0054152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 -- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400047c0 msgr2=0x7f00540c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400047c0 0x7f00540c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.776+0000 7f004bfff640 1 -- 172.21.15.29:0/1020383395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540c5110 con 0x7f00540c4c90 2023-10-16T14:32:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.777+0000 7f004bfff640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 0x7f0054151190 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f003c02c360 tx=0x7f003c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.777+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c03c040 con 0x7f00540c4c90 2023-10-16T14:32:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.777+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c02ccb0 con 0x7f00540c4c90 2023-10-16T14:32:07.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.777+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00540c7b70 con 0x7f00540c4c90 2023-10-16T14:32:07.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.777+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c0072b0 con 0x7f00540c4c90 2023-10-16T14:32:07.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.777+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00540c8130 con 0x7f00540c4c90 2023-10-16T14:32:07.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.778+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005404e160 con 0x7f00540c4c90 2023-10-16T14:32:07.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.779+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f003c032070 con 0x7f00540c4c90 2023-10-16T14:32:07.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.780+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f003c0aa570 con 0x7f00540c4c90 2023-10-16T14:32:07.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.785+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f003c075720 con 0x7f00540c4c90 2023-10-16T14:32:07.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.977+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0054070380 con 0x7f00540c4c90 2023-10-16T14:32:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.978+0000 7f00517fa640 1 -- 172.21.15.29:0/1020383395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f003c078d40 con 0x7f00540c4c90 2023-10-16T14:32:07.982 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:32:07.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 msgr2=0x7f0054151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:07.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 0x7f0054151190 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f003c02c360 tx=0x7f003c02e070 comp rx=0 tx=0).stop 2023-10-16T14:32:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 shutdown_connections 2023-10-16T14:32:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00540c4c90 0x7f0054151190 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0054151e00 0x7f0054152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 --2- 172.21.15.29:0/1020383395 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00400047c0 0x7f00540c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 >> 172.21.15.29:0/1020383395 conn(0x7f00540ce9b0 msgr2=0x7f00540ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:07.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 shutdown_connections 2023-10-16T14:32:07.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:07.981+0000 7f005aa1a640 1 -- 172.21.15.29:0/1020383395 wait complete. 2023-10-16T14:32:08.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:32:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.216+0000 7f2809955640 1 Processor -- start 2023-10-16T14:32:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.216+0000 7f2809955640 1 -- start start 2023-10-16T14:32:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809955640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2804159950 0x7f280415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809955640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2804070450 0x7f28040708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809955640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f280406c100 0x7f280406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809955640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280415e490 con 0x7f280406c100 2023-10-16T14:32:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809955640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2804070df0 con 0x7f2804070450 2023-10-16T14:32:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809955640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280415aef0 con 0x7f2804159950 2023-10-16T14:32:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809154640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f280406c100 0x7f280406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809154640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f280406c100 0x7f280406c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50920/0 (socket says 172.21.15.29:50920) 2023-10-16T14:32:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.218+0000 7f2809154640 1 -- 172.21.15.29:0/151136013 learned_addr learned my addr 172.21.15.29:0/151136013 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f27fbfff640 1 --2- 172.21.15.29:0/151136013 >> v2:172.21.15.170:3300/0 conn(0x7f2804070450 0x7f28040708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f2808953640 1 --2- 172.21.15.29:0/151136013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2804159950 0x7f280415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f2809154640 1 -- 172.21.15.29:0/151136013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2804159950 msgr2=0x7f280415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f2809154640 1 --2- 172.21.15.29:0/151136013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2804159950 0x7f280415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f2809154640 1 -- 172.21.15.29:0/151136013 >> v2:172.21.15.170:3300/0 conn(0x7f2804070450 msgr2=0x7f28040708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f2809154640 1 --2- 172.21.15.29:0/151136013 >> v2:172.21.15.170:3300/0 conn(0x7f2804070450 0x7f28040708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.219+0000 7f2809154640 1 -- 172.21.15.29:0/151136013 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280415d910 con 0x7f280406c100 2023-10-16T14:32:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.220+0000 7f2809154640 1 --2- 172.21.15.29:0/151136013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280406c100 0x7f280406c570 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f28040719e0 tx=0x7f27fc02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3665a8e13ecef0c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.220+0000 7f27fb7fe640 1 -- 172.21.15.29:0/151136013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc02ede0 con 0x7f280406c100 2023-10-16T14:32:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.220+0000 7f27fb7fe640 1 -- 172.21.15.29:0/151136013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc036440 con 0x7f280406c100 2023-10-16T14:32:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.220+0000 7f27fb7fe640 1 -- 172.21.15.29:0/151136013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0367d0 con 0x7f280406c100 2023-10-16T14:32:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 -- 172.21.15.29:0/151136013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280406c100 msgr2=0x7f280406c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 --2- 172.21.15.29:0/151136013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280406c100 0x7f280406c570 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f28040719e0 tx=0x7f27fc02e030 comp rx=0 tx=0).stop 2023-10-16T14:32:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 -- 172.21.15.29:0/151136013 shutdown_connections 2023-10-16T14:32:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 --2- 172.21.15.29:0/151136013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f280406c100 0x7f280406c570 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 --2- 172.21.15.29:0/151136013 >> v2:172.21.15.170:3300/0 conn(0x7f2804070450 0x7f28040708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 --2- 172.21.15.29:0/151136013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2804159950 0x7f280415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 -- 172.21.15.29:0/151136013 >> 172.21.15.29:0/151136013 conn(0x7f28040ce6a0 msgr2=0x7f28040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 -- 172.21.15.29:0/151136013 shutdown_connections 2023-10-16T14:32:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.221+0000 7f2809955640 1 -- 172.21.15.29:0/151136013 wait complete. 2023-10-16T14:32:08.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.223+0000 7f2809955640 1 Processor -- start 2023-10-16T14:32:08.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.223+0000 7f2809955640 1 -- start start 2023-10-16T14:32:08.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809955640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f280406c100 0x7f28041f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809955640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2804070450 0x7f280414da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809955640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 0x7f28040c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809955640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28040c5430 con 0x7f2804159950 2023-10-16T14:32:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809955640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28040afc00 con 0x7f2804070450 2023-10-16T14:32:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809955640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28040afee0 con 0x7f280406c100 2023-10-16T14:32:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809154640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 0x7f28040c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809154640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 0x7f28040c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50932/0 (socket says 172.21.15.29:50932) 2023-10-16T14:32:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.224+0000 7f2809154640 1 -- 172.21.15.29:0/4290006259 learned_addr learned my addr 172.21.15.29:0/4290006259 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f27fbfff640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2804070450 0x7f280414da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f2809154640 1 -- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f280406c100 msgr2=0x7f28041f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f2809154640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f280406c100 0x7f28041f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f2809154640 1 -- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2804070450 msgr2=0x7f280414da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f2809154640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2804070450 0x7f280414da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f2809154640 1 -- 172.21.15.29:0/4290006259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc00b770 con 0x7f2804159950 2023-10-16T14:32:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f27fbfff640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2804070450 0x7f280414da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f2809154640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 0x7f28040c21b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00b2c0 tx=0x7f27fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.225+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc034750 con 0x7f2804159950 2023-10-16T14:32:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.226+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc0348f0 con 0x7f2804159950 2023-10-16T14:32:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.226+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28041f5a10 con 0x7f2804159950 2023-10-16T14:32:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.226+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc034ca0 con 0x7f2804159950 2023-10-16T14:32:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.226+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28041f5fd0 con 0x7f2804159950 2023-10-16T14:32:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.227+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280404e9b0 con 0x7f2804159950 2023-10-16T14:32:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.228+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f27fc041460 con 0x7f2804159950 2023-10-16T14:32:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.229+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f27fc0ab890 con 0x7f2804159950 2023-10-16T14:32:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.233+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f27fc076a40 con 0x7f2804159950 2023-10-16T14:32:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.365+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f27fc0764f0 con 0x7f2804159950 2023-10-16T14:32:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.366+0000 7f27f97fa640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f27f00706b0 0x7f27f0072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.368+0000 7f2808953640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f27f00706b0 0x7f27f0072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.370+0000 7f2808953640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f27f00706b0 0x7f27f0072aa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2800009540 tx=0x7f2800009450 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.440+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28040b4620 con 0x7f2804159950 2023-10-16T14:32:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.441+0000 7f27f97fa640 1 -- 172.21.15.29:0/4290006259 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f27fc0906e0 con 0x7f2804159950 2023-10-16T14:32:08.445 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:32:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.443+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f27f00706b0 msgr2=0x7f27f0072aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.443+0000 7f2809955640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f27f00706b0 0x7f27f0072aa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2800009540 tx=0x7f2800009450 comp rx=0 tx=0).stop 2023-10-16T14:32:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.443+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 msgr2=0x7f28040c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.443+0000 7f2809955640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 0x7f28040c21b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00b2c0 tx=0x7f27fc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 shutdown_connections 2023-10-16T14:32:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f27f00706b0 0x7f27f0072aa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2804159950 0x7f28040c21b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2804070450 0x7f280414da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 --2- 172.21.15.29:0/4290006259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f280406c100 0x7f28041f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 >> 172.21.15.29:0/4290006259 conn(0x7f28040ce6a0 msgr2=0x7f280406b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 shutdown_connections 2023-10-16T14:32:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.444+0000 7f2809955640 1 -- 172.21.15.29:0/4290006259 wait complete. 2023-10-16T14:32:08.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.686+0000 7fd47b59e640 1 Processor -- start 2023-10-16T14:32:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.687+0000 7fd47b59e640 1 -- start start 2023-10-16T14:32:08.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.688+0000 7fd47b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd47c0b8480 0x7fd47c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.688+0000 7fd47b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd47c0d0be0 0x7fd47c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.688+0000 7fd47b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c15d150 con 0x7fd47c15cd70 2023-10-16T14:32:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.688+0000 7fd47b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c0d1180 con 0x7fd47c0b8480 2023-10-16T14:32:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.688+0000 7fd47b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c0d1460 con 0x7fd47c0d0be0 2023-10-16T14:32:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd47c0b8480 0x7fd47c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd47c0b8480 0x7fd47c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50942/0 (socket says 172.21.15.29:50942) 2023-10-16T14:32:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 -- 172.21.15.29:0/2565481208 learned_addr learned my addr 172.21.15.29:0/2565481208 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd47ad9d640 1 --2- 172.21.15.29:0/2565481208 >> v2:172.21.15.170:3300/0 conn(0x7fd47c0d0be0 0x7fd47c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 -- 172.21.15.29:0/2565481208 >> v2:172.21.15.170:3300/0 conn(0x7fd47c0d0be0 msgr2=0x7fd47c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 --2- 172.21.15.29:0/2565481208 >> v2:172.21.15.170:3300/0 conn(0x7fd47c0d0be0 0x7fd47c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 -- 172.21.15.29:0/2565481208 >> v1:172.21.15.170:6789/0 conn(0x7fd47c15cd70 legacy=0x7fd47c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.689+0000 7fd479d9b640 1 -- 172.21.15.29:0/2565481208 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0d1740 con 0x7fd47c0b8480 2023-10-16T14:32:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.690+0000 7fd479d9b640 1 --2- 172.21.15.29:0/2565481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c0cd7c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0b8910 tx=0x7fd46402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b048b7822b3722 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:08.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.691+0000 7fd47959a640 1 -- 172.21.15.29:0/2565481208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464031870 con 0x7fd47c0b8480 2023-10-16T14:32:08.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.691+0000 7fd47959a640 1 -- 172.21.15.29:0/2565481208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4640319d0 con 0x7fd47c0b8480 2023-10-16T14:32:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.691+0000 7fd47959a640 1 -- 172.21.15.29:0/2565481208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464031d60 con 0x7fd47c0b8480 2023-10-16T14:32:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.691+0000 7fd47b59e640 1 -- 172.21.15.29:0/2565481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 msgr2=0x7fd47c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.691+0000 7fd47b59e640 1 --2- 172.21.15.29:0/2565481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c0cd7c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0b8910 tx=0x7fd46402b910 comp rx=0 tx=0).stop 2023-10-16T14:32:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.692+0000 7fd47b59e640 1 -- 172.21.15.29:0/2565481208 shutdown_connections 2023-10-16T14:32:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.692+0000 7fd47b59e640 1 --2- 172.21.15.29:0/2565481208 >> v2:172.21.15.170:3300/0 conn(0x7fd47c0d0be0 0x7fd47c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.692+0000 7fd47b59e640 1 --2- 172.21.15.29:0/2565481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c0cd7c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.692+0000 7fd47b59e640 1 -- 172.21.15.29:0/2565481208 >> 172.21.15.29:0/2565481208 conn(0x7fd47c0cf570 msgr2=0x7fd47c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.692+0000 7fd47b59e640 1 -- 172.21.15.29:0/2565481208 shutdown_connections 2023-10-16T14:32:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.692+0000 7fd47b59e640 1 -- 172.21.15.29:0/2565481208 wait complete. 2023-10-16T14:32:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.693+0000 7fd47b59e640 1 Processor -- start 2023-10-16T14:32:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.694+0000 7fd47b59e640 1 -- start start 2023-10-16T14:32:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.694+0000 7fd47b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.694+0000 7fd47b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd47c0d0be0 0x7fd47c15e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.694+0000 7fd47b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd47c15cd70 0x7fd47c15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c15f500 con 0x7fd47c0b8480 2023-10-16T14:32:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c15ec60 con 0x7fd47c15cd70 2023-10-16T14:32:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c15ef40 con 0x7fd47c0d0be0 2023-10-16T14:32:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50956/0 (socket says 172.21.15.29:50956) 2023-10-16T14:32:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47a59c640 1 -- 172.21.15.29:0/525988357 learned_addr learned my addr 172.21.15.29:0/525988357 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd479d9b640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd47c0d0be0 0x7fd47c15e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47ad9d640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd47c15cd70 0x7fd47c15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47a59c640 1 -- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd47c0d0be0 msgr2=0x7fd47c15e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47a59c640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd47c0d0be0 0x7fd47c15e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.695+0000 7fd47a59c640 1 -- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd47c15cd70 msgr2=0x7fd47c15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd47a59c640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd47c15cd70 0x7fd47c15e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd47a59c640 1 -- 172.21.15.29:0/525988357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4640094d0 con 0x7fd47c0b8480 2023-10-16T14:32:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd47a59c640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c154c00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00c870 tx=0x7fd46c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c003b70 con 0x7fd47c0b8480 2023-10-16T14:32:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c003cf0 con 0x7fd47c0b8480 2023-10-16T14:32:08.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00aa90 con 0x7fd47c0b8480 2023-10-16T14:32:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c15f220 con 0x7fd47c0b8480 2023-10-16T14:32:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.696+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c15ff80 con 0x7fd47c0b8480 2023-10-16T14:32:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.697+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c1529a0 con 0x7fd47c0b8480 2023-10-16T14:32:08.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.698+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd46c00abf0 con 0x7fd47c0b8480 2023-10-16T14:32:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.699+0000 7fd472ffd640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd45c06acc0 0x7fd45c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.699+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fd46c089400 con 0x7fd47c0b8480 2023-10-16T14:32:08.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.700+0000 7fd479d9b640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd45c06acc0 0x7fd45c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.701+0000 7fd479d9b640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd45c06acc0 0x7fd45c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd464036000 tx=0x7fd46403c040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:08.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.704+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd47c1529a0 con 0x7fd47c0b8480 2023-10-16T14:32:08.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.934+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd47c152d80 con 0x7fd47c0b8480 2023-10-16T14:32:08.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.935+0000 7fd472ffd640 1 -- 172.21.15.29:0/525988357 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fd47c152d80 con 0x7fd47c0b8480 2023-10-16T14:32:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:08.940 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:30:15.728624+0000","modified":"2023-10-16T14:30:18.806702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1869070988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1869070988},{"type":"v1","addr":"172.21.15.29:6835","nonce":1869070988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T14:32:08.941 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-16T14:32:08.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd45c06acc0 msgr2=0x7fd45c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd45c06acc0 0x7fd45c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd464036000 tx=0x7fd46403c040 comp rx=0 tx=0).stop 2023-10-16T14:32:08.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 msgr2=0x7fd47c154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c154c00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00c870 tx=0x7fd46c00cd30 comp rx=0 tx=0).stop 2023-10-16T14:32:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 shutdown_connections 2023-10-16T14:32:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd45c06acc0 0x7fd45c06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd47c15cd70 0x7fd47c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd47c0d0be0 0x7fd47c15e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 --2- 172.21.15.29:0/525988357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd47c0b8480 0x7fd47c154c00 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 >> 172.21.15.29:0/525988357 conn(0x7fd47c0cf570 msgr2=0x7fd47c0d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 shutdown_connections 2023-10-16T14:32:08.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:08.938+0000 7fd47b59e640 1 -- 172.21.15.29:0/525988357 wait complete. 2023-10-16T14:32:08.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.183+0000 7fe005911640 1 Processor -- start 2023-10-16T14:32:09.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.184+0000 7fe005911640 1 -- start start 2023-10-16T14:32:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.185+0000 7fe005911640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 0x7fe0000c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.185+0000 7fe005911640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe0000c9230 0x7fe0000709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.185+0000 7fe005911640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe000160a90 con 0x7fe000070f20 2023-10-16T14:32:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.185+0000 7fe005911640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0000cdec0 con 0x7fe0000c9230 2023-10-16T14:32:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.185+0000 7fe005911640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000cd900 con 0x7fe00015c9f0 2023-10-16T14:32:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.186+0000 7fe00490f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 0x7fe0000c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.186+0000 7fe00490f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 0x7fe0000c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:46780/0 (socket says 172.21.15.29:46780) 2023-10-16T14:32:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.186+0000 7fe00490f640 1 -- 172.21.15.29:0/1983198620 learned_addr learned my addr 172.21.15.29:0/1983198620 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.186+0000 7fdffffff640 1 --2- 172.21.15.29:0/1983198620 >> v2:172.21.15.170:3300/0 conn(0x7fe0000c9230 0x7fe0000709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.187+0000 7fe00490f640 1 -- 172.21.15.29:0/1983198620 >> v2:172.21.15.170:3300/0 conn(0x7fe0000c9230 msgr2=0x7fe0000709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.187+0000 7fe00490f640 1 --2- 172.21.15.29:0/1983198620 >> v2:172.21.15.170:3300/0 conn(0x7fe0000c9230 0x7fe0000709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.187+0000 7fe00490f640 1 -- 172.21.15.29:0/1983198620 >> v1:172.21.15.170:6789/0 conn(0x7fe000070f20 legacy=0x7fe0000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.187+0000 7fe00490f640 1 -- 172.21.15.29:0/1983198620 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0000cdbe0 con 0x7fe00015c9f0 2023-10-16T14:32:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.188+0000 7fe00490f640 1 --2- 172.21.15.29:0/1983198620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 0x7fe0000c8cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009870 tx=0x7fdff402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b552fa59601af18e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fdfff7fe640 1 -- 172.21.15.29:0/1983198620 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4030840 con 0x7fe00015c9f0 2023-10-16T14:32:09.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fdfff7fe640 1 -- 172.21.15.29:0/1983198620 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff40309a0 con 0x7fe00015c9f0 2023-10-16T14:32:09.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fdfff7fe640 1 -- 172.21.15.29:0/1983198620 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4030d30 con 0x7fe00015c9f0 2023-10-16T14:32:09.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fe005911640 1 -- 172.21.15.29:0/1983198620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 msgr2=0x7fe0000c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fe005911640 1 --2- 172.21.15.29:0/1983198620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 0x7fe0000c8cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009870 tx=0x7fdff402b980 comp rx=0 tx=0).stop 2023-10-16T14:32:09.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fe005911640 1 -- 172.21.15.29:0/1983198620 shutdown_connections 2023-10-16T14:32:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fe005911640 1 --2- 172.21.15.29:0/1983198620 >> v2:172.21.15.170:3300/0 conn(0x7fe0000c9230 0x7fe0000709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fe005911640 1 --2- 172.21.15.29:0/1983198620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00015c9f0 0x7fe0000c8cf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.189+0000 7fe005911640 1 -- 172.21.15.29:0/1983198620 >> 172.21.15.29:0/1983198620 conn(0x7fe0000ce6a0 msgr2=0x7fe0000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.190+0000 7fe005911640 1 -- 172.21.15.29:0/1983198620 shutdown_connections 2023-10-16T14:32:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.190+0000 7fe005911640 1 -- 172.21.15.29:0/1983198620 wait complete. 2023-10-16T14:32:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.191+0000 7fe005911640 1 Processor -- start 2023-10-16T14:32:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.191+0000 7fe005911640 1 -- start start 2023-10-16T14:32:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.192+0000 7fe005911640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe000070f20 0x7fe0001efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.192+0000 7fe005911640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0000c9230 0x7fe0000c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.192+0000 7fe005911640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 0x7fe0001f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.192+0000 7fe005911640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000c5330 con 0x7fe00015c9f0 2023-10-16T14:32:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.192+0000 7fe005911640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000c47f0 con 0x7fe0000c9230 2023-10-16T14:32:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.192+0000 7fe005911640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000c4a90 con 0x7fe000070f20 2023-10-16T14:32:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe005110640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 0x7fe0001f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe005110640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 0x7fe0001f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50958/0 (socket says 172.21.15.29:50958) 2023-10-16T14:32:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe005110640 1 -- 172.21.15.29:0/3346391664 learned_addr learned my addr 172.21.15.29:0/3346391664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe00490f640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe000070f20 0x7fe0001efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fdffffff640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0000c9230 0x7fe0000c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe005110640 1 -- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe000070f20 msgr2=0x7fe0001efb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe005110640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe000070f20 0x7fe0001efb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.193+0000 7fe005110640 1 -- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0000c9230 msgr2=0x7fe0000c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fe005110640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0000c9230 0x7fe0000c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fe005110640 1 -- 172.21.15.29:0/3346391664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff40094d0 con 0x7fe00015c9f0 2023-10-16T14:32:09.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fdffffff640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0000c9230 0x7fe0000c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:09.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fe005110640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 0x7fe0001f5de0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdff800c7a0 tx=0x7fdff800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:09.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8003b70 con 0x7fe00015c9f0 2023-10-16T14:32:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8003cf0 con 0x7fe00015c9f0 2023-10-16T14:32:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8003ea0 con 0x7fe00015c9f0 2023-10-16T14:32:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0000c4d70 con 0x7fe00015c9f0 2023-10-16T14:32:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.194+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00015dfc0 con 0x7fe00015c9f0 2023-10-16T14:32:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.195+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00015de20 con 0x7fe00015c9f0 2023-10-16T14:32:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.197+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fdff800aa90 con 0x7fe00015c9f0 2023-10-16T14:32:09.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.197+0000 7fdffd7fa640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fdfe406acc0 0x7fdfe406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.198+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fdff8089440 con 0x7fe00015c9f0 2023-10-16T14:32:09.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.198+0000 7fe00490f640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fdfe406acc0 0x7fdfe406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.199+0000 7fe00490f640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fdfe406acc0 0x7fdfe406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdff4036000 tx=0x7fdff403b040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:09.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.202+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe00015de20 con 0x7fe00015c9f0 2023-10-16T14:32:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.378+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdff800ad40 con 0x7fe00015c9f0 2023-10-16T14:32:09.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.399+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe00015de20 con 0x7fe00015c9f0 2023-10-16T14:32:09.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.402+0000 7fdffd7fa640 1 -- 172.21.15.29:0/3346391664 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+17684 (secure 0 0 0) 0x7fe00015de20 con 0x7fe00015c9f0 2023-10-16T14:32:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:09.406 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":71,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:07.341192+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/76368088":"2023-10-17T14:32:07.341132+0000","172.21.15.170:6832/636880041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2930685041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2214932809":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2595110151":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3247649762":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2698007731":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4263280508":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2380256585":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4004225158":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3575752004":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3621036751":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3937243709":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6833/636880041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:6833/3423886595":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6832/3423886595":"2023-10-17T14:30:30.217790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:09.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.404+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fdfe406acc0 msgr2=0x7fdfe406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.404+0000 7fe005911640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fdfe406acc0 0x7fdfe406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdff4036000 tx=0x7fdff403b040 comp rx=0 tx=0).stop 2023-10-16T14:32:09.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.404+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 msgr2=0x7fe0001f5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.404+0000 7fe005911640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 0x7fe0001f5de0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdff800c7a0 tx=0x7fdff800cc60 comp rx=0 tx=0).stop 2023-10-16T14:32:09.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 shutdown_connections 2023-10-16T14:32:09.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fdfe406acc0 0x7fdfe406d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00015c9f0 0x7fe0001f5de0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0000c9230 0x7fe0000c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 --2- 172.21.15.29:0/3346391664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe000070f20 0x7fe0001efb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 >> 172.21.15.29:0/3346391664 conn(0x7fe0000ce6a0 msgr2=0x7fe0000cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:09.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 shutdown_connections 2023-10-16T14:32:09.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.405+0000 7fe005911640 1 -- 172.21.15.29:0/3346391664 wait complete. 2023-10-16T14:32:09.430 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:32:09.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:09.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.648+0000 7f31b6393640 1 Processor -- start 2023-10-16T14:32:09.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.649+0000 7f31b6393640 1 -- start start 2023-10-16T14:32:09.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.650+0000 7f31b6393640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f31b006c040 0x7f31b006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.650+0000 7f31b6393640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31b00ad970 0x7f31b015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.650+0000 7f31b6393640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b006c960 con 0x7f31b015ebc0 2023-10-16T14:32:09.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.650+0000 7f31b6393640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b015b9d0 con 0x7f31b006c040 2023-10-16T14:32:09.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.650+0000 7f31b6393640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b015bcb0 con 0x7f31b00ad970 2023-10-16T14:32:09.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.650+0000 7f31b5391640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f31b006c040 0x7f31b006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.651+0000 7f31b5391640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f31b006c040 0x7f31b006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50972/0 (socket says 172.21.15.29:50972) 2023-10-16T14:32:09.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.651+0000 7f31b5391640 1 -- 172.21.15.29:0/2613425494 learned_addr learned my addr 172.21.15.29:0/2613425494 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:09.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.651+0000 7f31b5391640 1 -- 172.21.15.29:0/2613425494 >> v2:172.21.15.170:3300/0 conn(0x7f31b00ad970 msgr2=0x7f31b015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:09.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.651+0000 7f31b5391640 1 --2- 172.21.15.29:0/2613425494 >> v2:172.21.15.170:3300/0 conn(0x7f31b00ad970 0x7f31b015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.651+0000 7f31b5391640 1 -- 172.21.15.29:0/2613425494 >> v1:172.21.15.170:6789/0 conn(0x7f31b015ebc0 legacy=0x7f31b00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:09.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.651+0000 7f31b5391640 1 -- 172.21.15.29:0/2613425494 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b015bf90 con 0x7f31b006c040 2023-10-16T14:32:09.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.652+0000 7f31b5391640 1 --2- 172.21.15.29:0/2613425494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b006c040 0x7f31b006c420 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f31b00cb060 tx=0x7f31ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a7d3350d97d745 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:09.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.652+0000 7f31a7fff640 1 -- 172.21.15.29:0/2613425494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac031870 con 0x7f31b006c040 2023-10-16T14:32:09.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31a7fff640 1 -- 172.21.15.29:0/2613425494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac0319d0 con 0x7f31b006c040 2023-10-16T14:32:09.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31a7fff640 1 -- 172.21.15.29:0/2613425494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac031d60 con 0x7f31b006c040 2023-10-16T14:32:09.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 -- 172.21.15.29:0/2613425494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b006c040 msgr2=0x7f31b006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 --2- 172.21.15.29:0/2613425494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b006c040 0x7f31b006c420 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f31b00cb060 tx=0x7f31ac02b910 comp rx=0 tx=0).stop 2023-10-16T14:32:09.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 -- 172.21.15.29:0/2613425494 shutdown_connections 2023-10-16T14:32:09.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 --2- 172.21.15.29:0/2613425494 >> v2:172.21.15.170:3300/0 conn(0x7f31b00ad970 0x7f31b015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 --2- 172.21.15.29:0/2613425494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b006c040 0x7f31b006c420 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 -- 172.21.15.29:0/2613425494 >> 172.21.15.29:0/2613425494 conn(0x7f31b00ce6a0 msgr2=0x7f31b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:09.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 -- 172.21.15.29:0/2613425494 shutdown_connections 2023-10-16T14:32:09.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.653+0000 7f31b6393640 1 -- 172.21.15.29:0/2613425494 wait complete. 2023-10-16T14:32:09.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.655+0000 7f31b6393640 1 Processor -- start 2023-10-16T14:32:09.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.655+0000 7f31b6393640 1 -- start start 2023-10-16T14:32:09.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b6393640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31b006c040 0x7f31b00b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b6393640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31b00ad970 0x7f31b00b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b6393640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 0x7f31b0148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b6393640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b00b1910 con 0x7f31b015ebc0 2023-10-16T14:32:09.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b6393640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b00b31a0 con 0x7f31b00ad970 2023-10-16T14:32:09.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b6393640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b00b3440 con 0x7f31b006c040 2023-10-16T14:32:09.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.656+0000 7f31b5b92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 0x7f31b0148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 0x7f31b0148a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50980/0 (socket says 172.21.15.29:50980) 2023-10-16T14:32:09.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 -- 172.21.15.29:0/1869450354 learned_addr learned my addr 172.21.15.29:0/1869450354 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:09.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b4b90640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31b00ad970 0x7f31b00b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5391640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31b006c040 0x7f31b00b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 -- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31b006c040 msgr2=0x7f31b00b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31b006c040 0x7f31b00b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 -- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31b00ad970 msgr2=0x7f31b00b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31b00ad970 0x7f31b00b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 -- 172.21.15.29:0/1869450354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0094d0 con 0x7f31b015ebc0 2023-10-16T14:32:09.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5391640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31b006c040 0x7f31b00b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:09.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.657+0000 7f31b5b92640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 0x7f31b0148a70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f31a800c780 tx=0x7f31a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:09.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.658+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8003d80 con 0x7f31b015ebc0 2023-10-16T14:32:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.658+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a800ce90 con 0x7f31b015ebc0 2023-10-16T14:32:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.658+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a80052b0 con 0x7f31b015ebc0 2023-10-16T14:32:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.658+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b00b3720 con 0x7f31b015ebc0 2023-10-16T14:32:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.658+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b00b15a0 con 0x7f31b015ebc0 2023-10-16T14:32:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.659+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b006b330 con 0x7f31b015ebc0 2023-10-16T14:32:09.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.660+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f31a8005410 con 0x7f31b015ebc0 2023-10-16T14:32:09.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.661+0000 7f31a5ffb640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f319406af00 0x7f319406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:09.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.661+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f31a8088e60 con 0x7f31b015ebc0 2023-10-16T14:32:09.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.662+0000 7f31b5391640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f319406af00 0x7f319406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:09.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.663+0000 7f31b5391640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f319406af00 0x7f319406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f31ac036000 tx=0x7f31ac03c040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:09.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.666+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f31a8053ba0 con 0x7f31b015ebc0 2023-10-16T14:32:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.900+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31b004e9b0 con 0x7f31b015ebc0 2023-10-16T14:32:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.901+0000 7f31a5ffb640 1 -- 172.21.15.29:0/1869450354 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f31a8017020 con 0x7f31b015ebc0 2023-10-16T14:32:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:09.905 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:30:15.728624+0000","modified":"2023-10-16T14:30:18.806702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1869070988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1869070988},{"type":"v1","addr":"172.21.15.29:6835","nonce":1869070988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T14:32:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-16T14:32:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.903+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f319406af00 msgr2=0x7f319406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.903+0000 7f31b6393640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f319406af00 0x7f319406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f31ac036000 tx=0x7f31ac03c040 comp rx=0 tx=0).stop 2023-10-16T14:32:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.903+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 msgr2=0x7f31b0148a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.903+0000 7f31b6393640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 0x7f31b0148a70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f31a800c780 tx=0x7f31a800cc40 comp rx=0 tx=0).stop 2023-10-16T14:32:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 shutdown_connections 2023-10-16T14:32:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f319406af00 0x7f319406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31b015ebc0 0x7f31b0148a70 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31b00ad970 0x7f31b00b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 --2- 172.21.15.29:0/1869450354 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31b006c040 0x7f31b00b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 >> 172.21.15.29:0/1869450354 conn(0x7f31b00ce6a0 msgr2=0x7f31b00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 shutdown_connections 2023-10-16T14:32:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:09.904+0000 7f31b6393640 1 -- 172.21.15.29:0/1869450354 wait complete. 2023-10-16T14:32:09.928 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.148+0000 7f6f02ed7640 1 Processor -- start 2023-10-16T14:32:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.148+0000 7f6f02ed7640 1 -- start start 2023-10-16T14:32:10.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f02ed7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6efc160c80 0x7f6efc161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f02ed7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0ad7a0 0x7f6efc070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f02ed7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0ae1a0 con 0x7f6efc0adc00 2023-10-16T14:32:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f02ed7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6efc0ae480 con 0x7f6efc160c80 2023-10-16T14:32:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f02ed7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc0ae760 con 0x7f6efc0ad7a0 2023-10-16T14:32:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f01ed5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6efc160c80 0x7f6efc161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f01ed5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6efc160c80 0x7f6efc161060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50992/0 (socket says 172.21.15.29:50992) 2023-10-16T14:32:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f01ed5640 1 -- 172.21.15.29:0/296945209 learned_addr learned my addr 172.21.15.29:0/296945209 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.150+0000 7f6f016d4640 1 --2- 172.21.15.29:0/296945209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0ad7a0 0x7f6efc070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.151+0000 7f6f01ed5640 1 -- 172.21.15.29:0/296945209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0ad7a0 msgr2=0x7f6efc070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.151+0000 7f6f01ed5640 1 --2- 172.21.15.29:0/296945209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0ad7a0 0x7f6efc070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.151+0000 7f6f01ed5640 1 -- 172.21.15.29:0/296945209 >> v1:172.21.15.29:6789/0 conn(0x7f6efc0adc00 legacy=0x7f6efc142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.151+0000 7f6f01ed5640 1 -- 172.21.15.29:0/296945209 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc0aea40 con 0x7f6efc160c80 2023-10-16T14:32:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.152+0000 7f6f01ed5640 1 --2- 172.21.15.29:0/296945209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc161060 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009e50 tx=0x7f6ef002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5188dc799a6341da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:10.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.152+0000 7f6f00ed3640 1 -- 172.21.15.29:0/296945209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef002cb00 con 0x7f6efc160c80 2023-10-16T14:32:10.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.152+0000 7f6f00ed3640 1 -- 172.21.15.29:0/296945209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef002cc60 con 0x7f6efc160c80 2023-10-16T14:32:10.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.152+0000 7f6f00ed3640 1 -- 172.21.15.29:0/296945209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00343f0 con 0x7f6efc160c80 2023-10-16T14:32:10.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 -- 172.21.15.29:0/296945209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 msgr2=0x7f6efc161060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/296945209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc161060 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009e50 tx=0x7f6ef002bf60 comp rx=0 tx=0).stop 2023-10-16T14:32:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 -- 172.21.15.29:0/296945209 shutdown_connections 2023-10-16T14:32:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/296945209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0ad7a0 0x7f6efc070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/296945209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc161060 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 -- 172.21.15.29:0/296945209 >> 172.21.15.29:0/296945209 conn(0x7f6efc0cc230 msgr2=0x7f6efc0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 -- 172.21.15.29:0/296945209 shutdown_connections 2023-10-16T14:32:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.153+0000 7f6f02ed7640 1 -- 172.21.15.29:0/296945209 wait complete. 2023-10-16T14:32:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.155+0000 7f6f02ed7640 1 Processor -- start 2023-10-16T14:32:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.155+0000 7f6f02ed7640 1 -- start start 2023-10-16T14:32:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f02ed7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6efc0ad7a0 0x7f6efc15f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f02ed7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0adc00 0x7f6efc1f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f02ed7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f02ed7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06c7c0 con 0x7f6efc160c80 2023-10-16T14:32:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f02ed7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06d340 con 0x7f6efc0ad7a0 2023-10-16T14:32:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f02ed7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06ca10 con 0x7f6efc0adc00 2023-10-16T14:32:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f026d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f026d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51006/0 (socket says 172.21.15.29:51006) 2023-10-16T14:32:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.156+0000 7f6f026d6640 1 -- 172.21.15.29:0/1165417630 learned_addr learned my addr 172.21.15.29:0/1165417630 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f016d4640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0adc00 0x7f6efc1f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f01ed5640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6efc0ad7a0 0x7f6efc15f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f026d6640 1 -- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0adc00 msgr2=0x7f6efc1f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f026d6640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0adc00 0x7f6efc1f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f026d6640 1 -- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6efc0ad7a0 msgr2=0x7f6efc15f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f026d6640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6efc0ad7a0 0x7f6efc15f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f026d6640 1 -- 172.21.15.29:0/1165417630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0009b30 con 0x7f6efc160c80 2023-10-16T14:32:10.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f026d6640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc145b80 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400c400 tx=0x7f6ef400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40052d0 con 0x7f6efc160c80 2023-10-16T14:32:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef4005430 con 0x7f6efc160c80 2023-10-16T14:32:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc06ccf0 con 0x7f6efc160c80 2023-10-16T14:32:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.157+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4003810 con 0x7f6efc160c80 2023-10-16T14:32:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.158+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc1f2040 con 0x7f6efc160c80 2023-10-16T14:32:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.159+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6efc06d4d0 con 0x7f6efc160c80 2023-10-16T14:32:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.163+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6ef4003970 con 0x7f6efc160c80 2023-10-16T14:32:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.164+0000 7f6efa7fc640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ee006af30 0x7f6ee006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.164+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f6ef40893f0 con 0x7f6efc160c80 2023-10-16T14:32:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.166+0000 7f6f01ed5640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ee006af30 0x7f6ee006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.167+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ef4054130 con 0x7f6efc160c80 2023-10-16T14:32:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.168+0000 7f6f01ed5640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ee006af30 0x7f6ee006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0035000 tx=0x7f6ef0030000 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:10.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.405+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6efc04e9b0 con 0x7f6efc160c80 2023-10-16T14:32:10.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.406+0000 7f6efa7fc640 1 -- 172.21.15.29:0/1165417630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f6ef4057750 con 0x7f6efc160c80 2023-10-16T14:32:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:10.410 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:30:15.728624+0000","modified":"2023-10-16T14:30:18.806702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1869070988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1869070988},{"type":"v1","addr":"172.21.15.29:6835","nonce":1869070988}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T14:32:10.411 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2023-10-16T14:32:10.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ee006af30 msgr2=0x7f6ee006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ee006af30 0x7f6ee006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0035000 tx=0x7f6ef0030000 comp rx=0 tx=0).stop 2023-10-16T14:32:10.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 msgr2=0x7f6efc145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc145b80 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400c400 tx=0x7f6ef400c8c0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 shutdown_connections 2023-10-16T14:32:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ee006af30 0x7f6ee006d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6efc160c80 0x7f6efc145b80 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6efc0adc00 0x7f6efc1f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 --2- 172.21.15.29:0/1165417630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6efc0ad7a0 0x7f6efc15f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.409+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 >> 172.21.15.29:0/1165417630 conn(0x7f6efc0cc230 msgr2=0x7f6efc15d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.410+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 shutdown_connections 2023-10-16T14:32:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.410+0000 7f6f02ed7640 1 -- 172.21.15.29:0/1165417630 wait complete. 2023-10-16T14:32:10.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:10.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.655+0000 7fd296ea3640 1 Processor -- start 2023-10-16T14:32:10.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.655+0000 7fd296ea3640 1 -- start start 2023-10-16T14:32:10.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.656+0000 7fd296ea3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2900ca830 0x7fd290070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.656+0000 7fd296ea3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd290070780 0x7fd29006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.656+0000 7fd296ea3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900ca3a0 con 0x7fd2901573e0 2023-10-16T14:32:10.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.656+0000 7fd296ea3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd290070c10 con 0x7fd2900ca830 2023-10-16T14:32:10.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.656+0000 7fd296ea3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c9d70 con 0x7fd290070780 2023-10-16T14:32:10.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2900ca830 0x7fd290070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2900ca830 0x7fd290070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51020/0 (socket says 172.21.15.29:51020) 2023-10-16T14:32:10.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 -- 172.21.15.29:0/686923048 learned_addr learned my addr 172.21.15.29:0/686923048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:10.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd295ea1640 1 --1- 172.21.15.29:0/686923048 >> v1:172.21.15.29:6789/0 conn(0x7fd2901573e0 0x7fd290159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58666/0 (socket says 172.21.15.29:58666) 2023-10-16T14:32:10.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 -- 172.21.15.29:0/686923048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd290070780 msgr2=0x7fd29006bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:10.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 --2- 172.21.15.29:0/686923048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd290070780 0x7fd29006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 -- 172.21.15.29:0/686923048 >> v1:172.21.15.29:6789/0 conn(0x7fd2901573e0 legacy=0x7fd290159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.657+0000 7fd2956a0640 1 -- 172.21.15.29:0/686923048 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2901577c0 con 0x7fd2900ca830 2023-10-16T14:32:10.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.659+0000 7fd2956a0640 1 --2- 172.21.15.29:0/686923048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca830 0x7fd290070240 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd284009870 tx=0x7fd28402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7e457cee0f6e10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:10.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.659+0000 7fd294e9f640 1 -- 172.21.15.29:0/686923048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284030840 con 0x7fd2900ca830 2023-10-16T14:32:10.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.659+0000 7fd294e9f640 1 -- 172.21.15.29:0/686923048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2840309a0 con 0x7fd2900ca830 2023-10-16T14:32:10.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.659+0000 7fd294e9f640 1 -- 172.21.15.29:0/686923048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284030d30 con 0x7fd2900ca830 2023-10-16T14:32:10.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 -- 172.21.15.29:0/686923048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca830 msgr2=0x7fd290070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 --2- 172.21.15.29:0/686923048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca830 0x7fd290070240 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd284009870 tx=0x7fd28402b980 comp rx=0 tx=0).stop 2023-10-16T14:32:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 -- 172.21.15.29:0/686923048 shutdown_connections 2023-10-16T14:32:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 --2- 172.21.15.29:0/686923048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd290070780 0x7fd29006bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 --2- 172.21.15.29:0/686923048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca830 0x7fd290070240 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 -- 172.21.15.29:0/686923048 >> 172.21.15.29:0/686923048 conn(0x7fd2900cc230 msgr2=0x7fd2900cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:10.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 -- 172.21.15.29:0/686923048 shutdown_connections 2023-10-16T14:32:10.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.660+0000 7fd296ea3640 1 -- 172.21.15.29:0/686923048 wait complete. 2023-10-16T14:32:10.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.662+0000 7fd296ea3640 1 Processor -- start 2023-10-16T14:32:10.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.662+0000 7fd296ea3640 1 -- start start 2023-10-16T14:32:10.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd296ea3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 0x7fd2901eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd296ea3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2900ca830 0x7fd290153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd296ea3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2901573e0 0x7fd29006c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd296ea3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29014b900 con 0x7fd290070780 2023-10-16T14:32:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd296ea3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29014bbe0 con 0x7fd2901573e0 2023-10-16T14:32:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd296ea3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29014bec0 con 0x7fd2900ca830 2023-10-16T14:32:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd295ea1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 0x7fd2901eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd295ea1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 0x7fd2901eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51034/0 (socket says 172.21.15.29:51034) 2023-10-16T14:32:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.663+0000 7fd295ea1640 1 -- 172.21.15.29:0/3212686297 learned_addr learned my addr 172.21.15.29:0/3212686297 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd2956a0640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2900ca830 0x7fd290153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd295ea1640 1 -- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2900ca830 msgr2=0x7fd290153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd2966a2640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2901573e0 0x7fd29006c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd295ea1640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2900ca830 0x7fd290153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd295ea1640 1 -- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2901573e0 msgr2=0x7fd29006c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd295ea1640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2901573e0 0x7fd29006c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd295ea1640 1 -- 172.21.15.29:0/3212686297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2840094d0 con 0x7fd290070780 2023-10-16T14:32:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd2956a0640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2900ca830 0x7fd290153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd295ea1640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 0x7fd2901eef30 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00c4d0 tx=0x7fd28c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c007480 con 0x7fd290070780 2023-10-16T14:32:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c0075e0 con 0x7fd290070780 2023-10-16T14:32:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0049e0 con 0x7fd290070780 2023-10-16T14:32:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.664+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29014c1a0 con 0x7fd290070780 2023-10-16T14:32:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.665+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29014c760 con 0x7fd290070780 2023-10-16T14:32:10.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.666+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29004e9b0 con 0x7fd290070780 2023-10-16T14:32:10.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.667+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd28c004b40 con 0x7fd290070780 2023-10-16T14:32:10.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.668+0000 7fd2827fc640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd27406af50 0x7fd27406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:10.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.668+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fd28c088dd0 con 0x7fd290070780 2023-10-16T14:32:10.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.668+0000 7fd2956a0640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd27406af50 0x7fd27406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:10.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.670+0000 7fd2956a0640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd27406af50 0x7fd27406d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd284036000 tx=0x7fd28403b040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:10.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.672+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd28c053c40 con 0x7fd290070780 2023-10-16T14:32:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.869+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2900b5730 con 0x7fd290070780 2023-10-16T14:32:10.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.873+0000 7fd2827fc640 1 -- 172.21.15.29:0/3212686297 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+17684 (secure 0 0 0) 0x7fd28c057260 con 0x7fd290070780 2023-10-16T14:32:10.877 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:10.877 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":72,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:10.157182+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:30:13.584764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T14:30:14.740756+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.35","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/76368088":"2023-10-17T14:32:07.341132+0000","172.21.15.170:6832/636880041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2930685041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2214932809":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2595110151":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3247649762":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2698007731":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4263280508":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2380256585":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4004225158":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3575752004":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3621036751":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3937243709":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6833/636880041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:6833/3423886595":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6832/3423886595":"2023-10-17T14:30:30.217790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:10.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd27406af50 msgr2=0x7fd27406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd27406af50 0x7fd27406d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd284036000 tx=0x7fd28403b040 comp rx=0 tx=0).stop 2023-10-16T14:32:10.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 msgr2=0x7fd2901eef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:10.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 0x7fd2901eef30 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00c4d0 tx=0x7fd28c00c990 comp rx=0 tx=0).stop 2023-10-16T14:32:10.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 shutdown_connections 2023-10-16T14:32:10.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fd27406af50 0x7fd27406d410 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2901573e0 0x7fd29006c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2900ca830 0x7fd290153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 --2- 172.21.15.29:0/3212686297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd290070780 0x7fd2901eef30 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:10.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 >> 172.21.15.29:0/3212686297 conn(0x7fd2900cc230 msgr2=0x7fd290158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:10.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 shutdown_connections 2023-10-16T14:32:10.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:10.875+0000 7fd296ea3640 1 -- 172.21.15.29:0/3212686297 wait complete. 2023-10-16T14:32:10.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:32:11.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.125+0000 7fde1789b640 1 Processor -- start 2023-10-16T14:32:11.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.125+0000 7fde1789b640 1 -- start start 2023-10-16T14:32:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde1789b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fde100ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde1789b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde10070360 con 0x7fde1006c140 2023-10-16T14:32:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde1789b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde100cd1f0 con 0x7fde10159e90 2023-10-16T14:32:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde1789b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1015d890 con 0x7fde10159ab0 2023-10-16T14:32:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde15e38640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fde1006c140 0x7fde1015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58676/0 (socket says 172.21.15.29:58676) 2023-10-16T14:32:11.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde15e38640 1 -- 172.21.15.29:0/413198681 learned_addr learned my addr 172.21.15.29:0/413198681 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.127+0000 7fde15637640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fde100ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437019260 0 0) 0x7fde10070360 con 0x7fde1006c140 2023-10-16T14:32:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddfc005730 con 0x7fde1006c140 2023-10-16T14:32:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533143840 0 0) 0x7fddfc005730 con 0x7fde1006c140 2023-10-16T14:32:11.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 msgr2=0x7fde100ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:11.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fde100ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 >> v1:172.21.15.170:6789/0 conn(0x7fde10159e90 legacy=0x7fde1006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100cd4d0 con 0x7fde1006c140 2023-10-16T14:32:11.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fde080019f0 con 0x7fde1006c140 2023-10-16T14:32:11.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 >> v1:172.21.15.29:6789/0 conn(0x7fde1006c140 legacy=0x7fde1015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc005bb0 0x7fddfc005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fddfc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fddfc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1015d890 con 0x7fddfc006f10 2023-10-16T14:32:11.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc00dd50 con 0x7fddfc005bb0 2023-10-16T14:32:11.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.128+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc00df10 con 0x7fde10159ab0 2023-10-16T14:32:11.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15e38640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fddfc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15637640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc005bb0 0x7fddfc005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde14e36640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fddfc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15e38640 1 -- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 msgr2=0x7fddfc0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:11.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15e38640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fddfc0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15e38640 1 -- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc005bb0 msgr2=0x7fddfc005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:11.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15e38640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc005bb0 0x7fddfc005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15e38640 1 -- 172.21.15.29:0/413198681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc005730 con 0x7fddfc006f10 2023-10-16T14:32:11.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde14e36640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde10159ab0 0x7fddfc0069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:11.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.129+0000 7fde15637640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fddfc005bb0 0x7fddfc005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:11.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.130+0000 7fde15e38640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fddfc007390 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fde0800a820 tx=0x7fddfc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3067d0bb1e0605c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:11.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080072d0 con 0x7fddfc006f10 2023-10-16T14:32:11.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0802f070 con 0x7fddfc006f10 2023-10-16T14:32:11.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde07fff640 1 -- 172.21.15.29:0/413198681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080076b0 con 0x7fddfc006f10 2023-10-16T14:32:11.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde1789b640 1 -- 172.21.15.29:0/413198681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 msgr2=0x7fddfc007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde1789b640 1 --2- 172.21.15.29:0/413198681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fddfc007390 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fde0800a820 tx=0x7fddfc007a00 comp rx=0 tx=0).stop 2023-10-16T14:32:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde15e38640 1 -- 172.21.15.29:0/413198681 reap_dead start 2023-10-16T14:32:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde1789b640 1 -- 172.21.15.29:0/413198681 shutdown_connections 2023-10-16T14:32:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.131+0000 7fde1789b640 1 -- 172.21.15.29:0/413198681 >> 172.21.15.29:0/413198681 conn(0x7fde100ce9b0 msgr2=0x7fde100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.132+0000 7fde1789b640 1 -- 172.21.15.29:0/413198681 shutdown_connections 2023-10-16T14:32:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.132+0000 7fde1789b640 1 -- 172.21.15.29:0/413198681 wait complete. 2023-10-16T14:32:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.133+0000 7fde1789b640 1 Processor -- start 2023-10-16T14:32:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.133+0000 7fde1789b640 1 -- start start 2023-10-16T14:32:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.134+0000 7fde1789b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fde1014cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.134+0000 7fde1789b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1006a340 0x7fde1006a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.134+0000 7fde1789b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde100c7c70 0x7fde100c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.134+0000 7fde1789b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1015ae70 con 0x7fddfc006f10 2023-10-16T14:32:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.134+0000 7fde1789b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1006ad00 con 0x7fde100c7c70 2023-10-16T14:32:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.134+0000 7fde1789b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1014d400 con 0x7fde1006a340 2023-10-16T14:32:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fde1014cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fde1014cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51038/0 (socket says 172.21.15.29:51038) 2023-10-16T14:32:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 -- 172.21.15.29:0/1952262884 learned_addr learned my addr 172.21.15.29:0/1952262884 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde14e36640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1006a340 0x7fde1006a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 -- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1006a340 msgr2=0x7fde1006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15e38640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde100c7c70 0x7fde100c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1006a340 0x7fde1006a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 -- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde100c7c70 msgr2=0x7fde100c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde100c7c70 0x7fde100c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15637640 1 -- 172.21.15.29:0/1952262884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1015a4d0 con 0x7fddfc006f10 2023-10-16T14:32:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde14e36640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1006a340 0x7fde1006a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.135+0000 7fde15e38640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde100c7c70 0x7fde100c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.136+0000 7fde15637640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fde1014cec0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fde0c011570 tx=0x7fde0c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.136+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c015870 con 0x7fddfc006f10 2023-10-16T14:32:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.136+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c0159d0 con 0x7fddfc006f10 2023-10-16T14:32:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.136+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c015d40 con 0x7fddfc006f10 2023-10-16T14:32:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.136+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1015a7b0 con 0x7fddfc006f10 2023-10-16T14:32:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.136+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde100c8630 con 0x7fddfc006f10 2023-10-16T14:32:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.137+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1015b0e0 con 0x7fddfc006f10 2023-10-16T14:32:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.138+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fde0c00f460 con 0x7fddfc006f10 2023-10-16T14:32:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.139+0000 7fde05ffb640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fddf406af00 0x7fddf406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.139+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fde0c08a270 con 0x7fddfc006f10 2023-10-16T14:32:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.140+0000 7fde14e36640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fddf406af00 0x7fddf406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.141+0000 7fde14e36640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fddf406af00 0x7fddf406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde10158f40 tx=0x7fde00009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:11.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.143+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde0c054e90 con 0x7fddfc006f10 2023-10-16T14:32:11.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.374+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fde1004e160 con 0x7fddfc006f10 2023-10-16T14:32:11.379 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:32:11.375+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:32:11.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.378+0000 7fde05ffb640 1 -- 172.21.15.29:0/1952262884 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7fde0c0584b0 con 0x7fddfc006f10 2023-10-16T14:32:11.382 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:32:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.382+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fddf406af00 msgr2=0x7fddf406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.382+0000 7fde1789b640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fddf406af00 0x7fddf406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde10158f40 tx=0x7fde00009040 comp rx=0 tx=0).stop 2023-10-16T14:32:11.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.382+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 msgr2=0x7fde1014cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.382+0000 7fde1789b640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fde1014cec0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fde0c011570 tx=0x7fde0c011a30 comp rx=0 tx=0).stop 2023-10-16T14:32:11.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 shutdown_connections 2023-10-16T14:32:11.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fddf406af00 0x7fddf406d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde100c7c70 0x7fde100c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1006a340 0x7fde1006a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 --2- 172.21.15.29:0/1952262884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddfc006f10 0x7fde1014cec0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 >> 172.21.15.29:0/1952262884 conn(0x7fde100ce9b0 msgr2=0x7fde1006b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 shutdown_connections 2023-10-16T14:32:11.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.383+0000 7fde1789b640 1 -- 172.21.15.29:0/1952262884 wait complete. 2023-10-16T14:32:11.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:32:11.457 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:32:11.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.600+0000 7f41ef4bb640 1 Processor -- start 2023-10-16T14:32:11.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.601+0000 7f41ef4bb640 1 -- start start 2023-10-16T14:32:11.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.602+0000 7f41ef4bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41e8159ab0 0x7f41e80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.602+0000 7f41ef4bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e8070360 con 0x7f41e8159e90 2023-10-16T14:32:11.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.602+0000 7f41ef4bb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e80cd1f0 con 0x7f41e806c140 2023-10-16T14:32:11.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.602+0000 7f41ef4bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e815d890 con 0x7f41e8159ab0 2023-10-16T14:32:11.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.602+0000 7f41ed257640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41e8159ab0 0x7f41e80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.602+0000 7f41eca56640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41e8159e90 0x7f41e806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58688/0 (socket says 172.21.15.29:58688) 2023-10-16T14:32:11.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.603+0000 7f41eca56640 1 -- 172.21.15.29:0/2654184698 learned_addr learned my addr 172.21.15.29:0/2654184698 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:11.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.603+0000 7f41ed257640 1 -- 172.21.15.29:0/2654184698 >> v1:172.21.15.170:6789/0 conn(0x7f41e806c140 legacy=0x7f41e815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:11.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.603+0000 7f41ed257640 1 -- 172.21.15.29:0/2654184698 >> v1:172.21.15.29:6789/0 conn(0x7f41e8159e90 legacy=0x7f41e806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.603+0000 7f41ed257640 1 -- 172.21.15.29:0/2654184698 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80cd4d0 con 0x7f41e8159ab0 2023-10-16T14:32:11.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.604+0000 7f41ed257640 1 --2- 172.21.15.29:0/2654184698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159ab0 0x7f41e80ce370 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f41d80097a0 tx=0x7f41d802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c4c9f74c3eca8e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41e7fff640 1 -- 172.21.15.29:0/2654184698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d802f050 con 0x7f41e8159ab0 2023-10-16T14:32:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41e7fff640 1 -- 172.21.15.29:0/2654184698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d8031700 con 0x7f41e8159ab0 2023-10-16T14:32:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41e7fff640 1 -- 172.21.15.29:0/2654184698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8031a90 con 0x7f41e8159ab0 2023-10-16T14:32:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41ef4bb640 1 -- 172.21.15.29:0/2654184698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159ab0 msgr2=0x7f41e80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/2654184698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159ab0 0x7f41e80ce370 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f41d80097a0 tx=0x7f41d802b7a0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41ef4bb640 1 -- 172.21.15.29:0/2654184698 shutdown_connections 2023-10-16T14:32:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/2654184698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159ab0 0x7f41e80ce370 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41ef4bb640 1 -- 172.21.15.29:0/2654184698 >> 172.21.15.29:0/2654184698 conn(0x7f41e80ce9b0 msgr2=0x7f41e80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.605+0000 7f41ef4bb640 1 -- 172.21.15.29:0/2654184698 shutdown_connections 2023-10-16T14:32:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.606+0000 7f41ef4bb640 1 -- 172.21.15.29:0/2654184698 wait complete. 2023-10-16T14:32:11.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.607+0000 7f41ef4bb640 1 Processor -- start 2023-10-16T14:32:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.607+0000 7f41ef4bb640 1 -- start start 2023-10-16T14:32:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.608+0000 7f41ef4bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41e806c140 0x7f41e8150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.608+0000 7f41ef4bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41e8159ab0 0x7f41e80c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.608+0000 7f41ef4bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 0x7f41e806b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.608+0000 7f41ef4bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80c37a0 con 0x7f41e8159e90 2023-10-16T14:32:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.608+0000 7f41ef4bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80c75e0 con 0x7f41e806c140 2023-10-16T14:32:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.608+0000 7f41ef4bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80c78c0 con 0x7f41e8159ab0 2023-10-16T14:32:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 0x7f41e806b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 0x7f41e806b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51076/0 (socket says 172.21.15.29:51076) 2023-10-16T14:32:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 -- 172.21.15.29:0/733073066 learned_addr learned my addr 172.21.15.29:0/733073066 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41ed257640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41e806c140 0x7f41e8150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eca56640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41e8159ab0 0x7f41e80c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 -- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41e8159ab0 msgr2=0x7f41e80c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41e8159ab0 0x7f41e80c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 -- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41e806c140 msgr2=0x7f41e8150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41e806c140 0x7f41e8150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eda58640 1 -- 172.21.15.29:0/733073066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d8009400 con 0x7f41e8159e90 2023-10-16T14:32:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41ed257640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41e806c140 0x7f41e8150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.609+0000 7f41eca56640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41e8159ab0 0x7f41e80c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.610+0000 7f41eda58640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 0x7f41e806b540 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f41e000b970 tx=0x7f41e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.610+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e000cb40 con 0x7f41e8159e90 2023-10-16T14:32:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.610+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e000cca0 con 0x7f41e8159e90 2023-10-16T14:32:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.610+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0014680 con 0x7f41e8159e90 2023-10-16T14:32:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.610+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e80c7ba0 con 0x7f41e8159e90 2023-10-16T14:32:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.610+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e80c8160 con 0x7f41e8159e90 2023-10-16T14:32:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.611+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e804e160 con 0x7f41e8159e90 2023-10-16T14:32:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.612+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f41e001d3b0 con 0x7f41e8159e90 2023-10-16T14:32:11.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.613+0000 7f41e5ffb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f41d406af90 0x7f41d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:11.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.614+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6541+0+0 (secure 0 0 0) 0x7f41e00898d0 con 0x7f41e8159e90 2023-10-16T14:32:11.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.614+0000 7f41ed257640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f41d406af90 0x7f41d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:11.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.615+0000 7f41ed257640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f41d406af90 0x7f41d406d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41d8035000 tx=0x7f41d8008030 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.618+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41e0054520 con 0x7f41e8159e90 2023-10-16T14:32:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:11.850+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f41e80cf0a0 con 0x7f41e8159e90 2023-10-16T14:32:12.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.434+0000 7f41e5ffb640 1 -- 172.21.15.29:0/733073066 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f41e0057b40 con 0x7f41e8159e90 2023-10-16T14:32:12.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f41d406af90 msgr2=0x7f41d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:12.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f41d406af90 0x7f41d406d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41d8035000 tx=0x7f41d8008030 comp rx=0 tx=0).stop 2023-10-16T14:32:12.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 msgr2=0x7f41e806b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:12.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 0x7f41e806b540 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f41e000b970 tx=0x7f41e000be30 comp rx=0 tx=0).stop 2023-10-16T14:32:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 shutdown_connections 2023-10-16T14:32:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f41d406af90 0x7f41d406d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41e8159e90 0x7f41e806b540 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41e8159ab0 0x7f41e80c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 --2- 172.21.15.29:0/733073066 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41e806c140 0x7f41e8150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 >> 172.21.15.29:0/733073066 conn(0x7f41e80ce9b0 msgr2=0x7f41e80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 shutdown_connections 2023-10-16T14:32:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.438+0000 7f41ef4bb640 1 -- 172.21.15.29:0/733073066 wait complete. 2023-10-16T14:32:12.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:32:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.691+0000 7faba7846640 1 Processor -- start 2023-10-16T14:32:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.691+0000 7faba7846640 1 -- start start 2023-10-16T14:32:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba7846640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba00c9ab0 0x7faba00c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba7846640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faba00b0360 0x7faba00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba7846640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba00c9620 con 0x7faba015cf60 2023-10-16T14:32:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba7846640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba015d340 con 0x7faba00b0360 2023-10-16T14:32:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba7846640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba00cc860 con 0x7faba00c9ab0 2023-10-16T14:32:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba55e2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faba015cf60 0x7faba006f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58694/0 (socket says 172.21.15.29:58694) 2023-10-16T14:32:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba55e2640 1 -- 172.21.15.29:0/4004072116 learned_addr learned my addr 172.21.15.29:0/4004072116 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba5de3640 1 --2- 172.21.15.29:0/4004072116 >> v2:172.21.15.29:3300/0 conn(0x7faba00b0360 0x7faba00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.693+0000 7faba4de1640 1 --2- 172.21.15.29:0/4004072116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba00c9ab0 0x7faba00c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.694+0000 7fab97fff640 1 -- 172.21.15.29:0/4004072116 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992388624 0 0) 0x7faba00c9620 con 0x7faba015cf60 2023-10-16T14:32:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.694+0000 7fab97fff640 1 -- 172.21.15.29:0/4004072116 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab8c003520 con 0x7faba015cf60 2023-10-16T14:32:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.694+0000 7faba5de3640 1 -- 172.21.15.29:0/4004072116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba00c9ab0 msgr2=0x7faba00c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.694+0000 7faba5de3640 1 --2- 172.21.15.29:0/4004072116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba00c9ab0 0x7faba00c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.694+0000 7faba5de3640 1 -- 172.21.15.29:0/4004072116 >> v1:172.21.15.29:6789/0 conn(0x7faba015cf60 legacy=0x7faba006f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.694+0000 7faba5de3640 1 -- 172.21.15.29:0/4004072116 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba00ad4a0 con 0x7faba00b0360 2023-10-16T14:32:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.695+0000 7faba5de3640 1 --2- 172.21.15.29:0/4004072116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00b07f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fab98009d90 tx=0x7fab9802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd06629ae866aeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.695+0000 7fab97fff640 1 -- 172.21.15.29:0/4004072116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9802cc50 con 0x7faba00b0360 2023-10-16T14:32:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.695+0000 7fab97fff640 1 -- 172.21.15.29:0/4004072116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9802cdb0 con 0x7faba00b0360 2023-10-16T14:32:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.695+0000 7fab97fff640 1 -- 172.21.15.29:0/4004072116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98034660 con 0x7faba00b0360 2023-10-16T14:32:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 -- 172.21.15.29:0/4004072116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 msgr2=0x7faba00b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 --2- 172.21.15.29:0/4004072116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00b07f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fab98009d90 tx=0x7fab9802bea0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 -- 172.21.15.29:0/4004072116 shutdown_connections 2023-10-16T14:32:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 --2- 172.21.15.29:0/4004072116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00b07f0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 --2- 172.21.15.29:0/4004072116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba00c9ab0 0x7faba00c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 -- 172.21.15.29:0/4004072116 >> 172.21.15.29:0/4004072116 conn(0x7faba00cb360 msgr2=0x7faba00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 -- 172.21.15.29:0/4004072116 shutdown_connections 2023-10-16T14:32:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.696+0000 7faba7846640 1 -- 172.21.15.29:0/4004072116 wait complete. 2023-10-16T14:32:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.698+0000 7faba7846640 1 Processor -- start 2023-10-16T14:32:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.698+0000 7faba7846640 1 -- start start 2023-10-16T14:32:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba7846640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba7846640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faba00c9ab0 0x7faba00c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba7846640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba015cf60 0x7faba00c25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba7846640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba00c4eb0 con 0x7faba00b0360 2023-10-16T14:32:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba7846640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba00c5ef0 con 0x7faba00c9ab0 2023-10-16T14:32:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba7846640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba00b3040 con 0x7faba015cf60 2023-10-16T14:32:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba55e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba55e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51104/0 (socket says 172.21.15.29:51104) 2023-10-16T14:32:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.699+0000 7faba55e2640 1 -- 172.21.15.29:0/2127447448 learned_addr learned my addr 172.21.15.29:0/2127447448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba5de3640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba015cf60 0x7faba00c25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba4de1640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faba00c9ab0 0x7faba00c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba55e2640 1 -- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba015cf60 msgr2=0x7faba00c25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba55e2640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba015cf60 0x7faba00c25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba55e2640 1 -- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faba00c9ab0 msgr2=0x7faba00c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba55e2640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faba00c9ab0 0x7faba00c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba55e2640 1 -- 172.21.15.29:0/2127447448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98009a70 con 0x7faba00b0360 2023-10-16T14:32:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba5de3640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba015cf60 0x7faba00c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba55e2640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00c59b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00c500 tx=0x7fab9c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c005ce0 con 0x7faba00b0360 2023-10-16T14:32:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c005e40 con 0x7faba00b0360 2023-10-16T14:32:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.700+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba00c6100 con 0x7faba00b0360 2023-10-16T14:32:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.701+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c007330 con 0x7faba00b0360 2023-10-16T14:32:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.701+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba00c66c0 con 0x7faba00b0360 2023-10-16T14:32:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.702+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba004e160 con 0x7faba00b0360 2023-10-16T14:32:12.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.703+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab9c007490 con 0x7faba00b0360 2023-10-16T14:32:12.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.704+0000 7fab95ffb640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fab8406af90 0x7fab8406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:12.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.704+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fab9c089180 con 0x7faba00b0360 2023-10-16T14:32:12.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.704+0000 7faba4de1640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fab8406af90 0x7fab8406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.706+0000 7faba4de1640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fab8406af90 0x7fab8406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fab90006fd0 tx=0x7fab90008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.708+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab9c053e90 con 0x7faba00b0360 2023-10-16T14:32:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:12.906+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faba00cb0d0 con 0x7faba00b0360 2023-10-16T14:32:13.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.438+0000 7fab95ffb640 1 -- 172.21.15.29:0/2127447448 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 181+0+0 (secure 0 0 0) 0x7fab9c0574b0 con 0x7faba00b0360 2023-10-16T14:32:13.442 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:32:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.441+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fab8406af90 msgr2=0x7fab8406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.441+0000 7faba7846640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fab8406af90 0x7fab8406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fab90006fd0 tx=0x7fab90008040 comp rx=0 tx=0).stop 2023-10-16T14:32:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.441+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 msgr2=0x7faba00c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:13.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.441+0000 7faba7846640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00c59b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00c500 tx=0x7fab9c00c9c0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 shutdown_connections 2023-10-16T14:32:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fab8406af90 0x7fab8406d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faba015cf60 0x7faba00c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faba00c9ab0 0x7faba00c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 --2- 172.21.15.29:0/2127447448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faba00b0360 0x7faba00c59b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 >> 172.21.15.29:0/2127447448 conn(0x7faba00cb360 msgr2=0x7faba00c9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 shutdown_connections 2023-10-16T14:32:13.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.442+0000 7faba7846640 1 -- 172.21.15.29:0/2127447448 wait complete. 2023-10-16T14:32:13.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:32:13.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.691+0000 7f6ad8df4640 1 Processor -- start 2023-10-16T14:32:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.692+0000 7f6ad8df4640 1 -- start start 2023-10-16T14:32:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.693+0000 7f6ad8df4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad416cea0 0x7f6ad40cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.693+0000 7f6ad8df4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ad40cd930 0x7f6ad40c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.693+0000 7f6ad8df4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad40ccc80 con 0x7f6ad408a450 2023-10-16T14:32:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.693+0000 7f6ad8df4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad406c530 con 0x7f6ad40cd930 2023-10-16T14:32:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.693+0000 7f6ad8df4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad406c810 con 0x7f6ad416cea0 2023-10-16T14:32:13.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.694+0000 7f6ad3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6ad408a450 0x7f6ad416d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58696/0 (socket says 172.21.15.29:58696) 2023-10-16T14:32:13.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.694+0000 7f6ad3fff640 1 -- 172.21.15.29:0/4244629777 learned_addr learned my addr 172.21.15.29:0/4244629777 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:13.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.694+0000 7f6acbfff640 1 --2- 172.21.15.29:0/4244629777 >> v2:172.21.15.29:3300/0 conn(0x7f6ad40cd930 0x7f6ad40c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:13.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.694+0000 7f6ad37fe640 1 --2- 172.21.15.29:0/4244629777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad416cea0 0x7f6ad40cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:13.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.694+0000 7f6ad2ffd640 1 -- 172.21.15.29:0/4244629777 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209094747 0 0) 0x7f6ad40ccc80 con 0x7f6ad408a450 2023-10-16T14:32:13.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.817+0000 7f6ad2ffd640 1 -- 172.21.15.29:0/4244629777 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac0003500 con 0x7f6ad408a450 2023-10-16T14:32:13.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.818+0000 7f6ad2ffd640 1 -- 172.21.15.29:0/4244629777 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3317448738 0 0) 0x7f6ac0003500 con 0x7f6ad408a450 2023-10-16T14:32:13.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.818+0000 7f6acbfff640 1 -- 172.21.15.29:0/4244629777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad416cea0 msgr2=0x7f6ad40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:13.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.818+0000 7f6acbfff640 1 --2- 172.21.15.29:0/4244629777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad416cea0 0x7f6ad40cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.818+0000 7f6acbfff640 1 -- 172.21.15.29:0/4244629777 >> v1:172.21.15.29:6789/0 conn(0x7f6ad408a450 legacy=0x7f6ad416d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:13.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.818+0000 7f6acbfff640 1 -- 172.21.15.29:0/4244629777 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad406caf0 con 0x7f6ad40cd930 2023-10-16T14:32:13.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.822+0000 7f6acbfff640 1 --2- 172.21.15.29:0/4244629777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad40cd930 0x7f6ad40c92b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c860 tx=0x7f6abc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9f742db48b8db2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:13.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.822+0000 7f6ad2ffd640 1 -- 172.21.15.29:0/4244629777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc0019f0 con 0x7f6ad40cd930 2023-10-16T14:32:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.823+0000 7f6ad2ffd640 1 -- 172.21.15.29:0/4244629777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc030070 con 0x7f6ad40cd930 2023-10-16T14:32:13.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.823+0000 7f6ad2ffd640 1 -- 172.21.15.29:0/4244629777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc02cbe0 con 0x7f6ad40cd930 2023-10-16T14:32:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 -- 172.21.15.29:0/4244629777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad40cd930 msgr2=0x7f6ad40c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/4244629777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad40cd930 0x7f6ad40c92b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c860 tx=0x7f6abc02beb0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 -- 172.21.15.29:0/4244629777 shutdown_connections 2023-10-16T14:32:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/4244629777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad40cd930 0x7f6ad40c92b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/4244629777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad416cea0 0x7f6ad40cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 -- 172.21.15.29:0/4244629777 >> 172.21.15.29:0/4244629777 conn(0x7f6ad40ce6a0 msgr2=0x7f6ad40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:13.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 -- 172.21.15.29:0/4244629777 shutdown_connections 2023-10-16T14:32:13.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.824+0000 7f6ad8df4640 1 -- 172.21.15.29:0/4244629777 wait complete. 2023-10-16T14:32:13.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.827+0000 7f6ad8df4640 1 Processor -- start 2023-10-16T14:32:13.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.827+0000 7f6ad8df4640 1 -- start start 2023-10-16T14:32:13.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6ad8df4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ad408a450 0x7f6ad4207390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:13.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6ad8df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 0x7f6ad40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:13.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6ad8df4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad40ca2c0 0x7f6ad40ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:13.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6ad8df4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40cac90 con 0x7f6ad416cea0 2023-10-16T14:32:13.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6ad8df4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40c62f0 con 0x7f6ad408a450 2023-10-16T14:32:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6ad8df4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad406ad70 con 0x7f6ad40ca2c0 2023-10-16T14:32:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6acbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 0x7f6ad40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6acbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 0x7f6ad40c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51112/0 (socket says 172.21.15.29:51112) 2023-10-16T14:32:13.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.829+0000 7f6acbfff640 1 -- 172.21.15.29:0/2585300731 learned_addr learned my addr 172.21.15.29:0/2585300731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6ad37fe640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ad408a450 0x7f6ad4207390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6ad3fff640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad40ca2c0 0x7f6ad40ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6acbfff640 1 -- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad40ca2c0 msgr2=0x7f6ad40ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6acbfff640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad40ca2c0 0x7f6ad40ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6acbfff640 1 -- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ad408a450 msgr2=0x7f6ad4207390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6acbfff640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ad408a450 0x7f6ad4207390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6acbfff640 1 -- 172.21.15.29:0/2585300731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc00c4c0 con 0x7f6ad416cea0 2023-10-16T14:32:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6ad37fe640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ad408a450 0x7f6ad4207390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6ad3fff640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad40ca2c0 0x7f6ad40ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:13.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6acbfff640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 0x7f6ad40c7000 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c860 tx=0x7f6abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.830+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc003a20 con 0x7f6ad416cea0 2023-10-16T14:32:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.831+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc003b80 con 0x7f6ad416cea0 2023-10-16T14:32:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.831+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc041470 con 0x7f6ad416cea0 2023-10-16T14:32:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.831+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad406aa90 con 0x7f6ad416cea0 2023-10-16T14:32:13.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.831+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad40c6800 con 0x7f6ad416cea0 2023-10-16T14:32:13.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.832+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad404e9b0 con 0x7f6ad416cea0 2023-10-16T14:32:13.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.833+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6abc0415d0 con 0x7f6ad416cea0 2023-10-16T14:32:13.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.834+0000 7f6ad0ff9640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ab406af90 0x7f6ab406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:13.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.835+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6103+0+0 (secure 0 0 0) 0x7f6abc0ac8d0 con 0x7f6ad416cea0 2023-10-16T14:32:13.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.835+0000 7f6ad37fe640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ab406af90 0x7f6ab406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:13.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.837+0000 7f6ad37fe640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ab406af90 0x7f6ab406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4005db0 tx=0x7f6ac4008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:13.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:13.840+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6abc0776f0 con 0x7f6ad416cea0 2023-10-16T14:32:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.049+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6ad406c270 con 0x7f6ad416cea0 2023-10-16T14:32:14.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.453+0000 7f6ad0ff9640 1 -- 172.21.15.29:0/2585300731 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6abc07ad10 con 0x7f6ad416cea0 2023-10-16T14:32:14.459 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:32:14.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.456+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ab406af90 msgr2=0x7f6ab406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.456+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ab406af90 0x7f6ab406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4005db0 tx=0x7f6ac4008040 comp rx=0 tx=0).stop 2023-10-16T14:32:14.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.456+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 msgr2=0x7f6ad40c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.456+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 0x7f6ad40c7000 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c860 tx=0x7f6abc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 shutdown_connections 2023-10-16T14:32:14.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ab406af90 0x7f6ab406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad416cea0 0x7f6ad40c7000 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ad40ca2c0 0x7f6ad40ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 --2- 172.21.15.29:0/2585300731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ad408a450 0x7f6ad4207390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 >> 172.21.15.29:0/2585300731 conn(0x7f6ad40ce6a0 msgr2=0x7f6ad4169720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:14.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 shutdown_connections 2023-10-16T14:32:14.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.457+0000 7f6ad8df4640 1 -- 172.21.15.29:0/2585300731 wait complete. 2023-10-16T14:32:14.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:32:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.702+0000 7f78afda9640 1 Processor -- start 2023-10-16T14:32:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.703+0000 7f78afda9640 1 -- start start 2023-10-16T14:32:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78afda9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f78a815af80 0x7f78a80cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78afda9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78a8065ab0 0x7f78a8065f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78afda9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a815eff0 con 0x7f78a806faf0 2023-10-16T14:32:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78afda9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a8066480 con 0x7f78a8065ab0 2023-10-16T14:32:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78afda9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80d0930 con 0x7f78a815af80 2023-10-16T14:32:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78ae346640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78a8065ab0 0x7f78a8065f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78ae346640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78a8065ab0 0x7f78a8065f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54862/0 (socket says 172.21.15.29:54862) 2023-10-16T14:32:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.704+0000 7f78ae346640 1 -- 172.21.15.29:0/82379765 learned_addr learned my addr 172.21.15.29:0/82379765 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.705+0000 7f78ae346640 1 -- 172.21.15.29:0/82379765 >> v2:172.21.15.170:3300/0 conn(0x7f78a815af80 msgr2=0x7f78a80cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.705+0000 7f78ae346640 1 --2- 172.21.15.29:0/82379765 >> v2:172.21.15.170:3300/0 conn(0x7f78a815af80 0x7f78a80cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.705+0000 7f78ae346640 1 -- 172.21.15.29:0/82379765 >> v1:172.21.15.170:6789/0 conn(0x7f78a806faf0 legacy=0x7f78a815f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.705+0000 7f78ae346640 1 -- 172.21.15.29:0/82379765 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a815ecc0 con 0x7f78a8065ab0 2023-10-16T14:32:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.706+0000 7f78ae346640 1 --2- 172.21.15.29:0/82379765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8065ab0 0x7f78a8065f40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f78a80c7aa0 tx=0x7f78a002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86735568e20d7f29 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.706+0000 7f78acb43640 1 -- 172.21.15.29:0/82379765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00317d0 con 0x7f78a8065ab0 2023-10-16T14:32:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.706+0000 7f78acb43640 1 -- 172.21.15.29:0/82379765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0031930 con 0x7f78a8065ab0 2023-10-16T14:32:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.706+0000 7f78acb43640 1 -- 172.21.15.29:0/82379765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0031cc0 con 0x7f78a8065ab0 2023-10-16T14:32:14.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 -- 172.21.15.29:0/82379765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8065ab0 msgr2=0x7f78a8065f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 --2- 172.21.15.29:0/82379765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8065ab0 0x7f78a8065f40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f78a80c7aa0 tx=0x7f78a002b870 comp rx=0 tx=0).stop 2023-10-16T14:32:14.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 -- 172.21.15.29:0/82379765 shutdown_connections 2023-10-16T14:32:14.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 --2- 172.21.15.29:0/82379765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a8065ab0 0x7f78a8065f40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 --2- 172.21.15.29:0/82379765 >> v2:172.21.15.170:3300/0 conn(0x7f78a815af80 0x7f78a80cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 -- 172.21.15.29:0/82379765 >> 172.21.15.29:0/82379765 conn(0x7f78a80d0ab0 msgr2=0x7f78a80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:14.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 -- 172.21.15.29:0/82379765 shutdown_connections 2023-10-16T14:32:14.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.707+0000 7f78afda9640 1 -- 172.21.15.29:0/82379765 wait complete. 2023-10-16T14:32:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.709+0000 7f78afda9640 1 Processor -- start 2023-10-16T14:32:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.709+0000 7f78afda9640 1 -- start start 2023-10-16T14:32:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78afda9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a8065ab0 0x7f78a80c3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78afda9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 0x7f78a80c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78afda9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a815af80 0x7f78a80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78afda9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80c9cb0 con 0x7f78a806faf0 2023-10-16T14:32:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78afda9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80c9f90 con 0x7f78a8065ab0 2023-10-16T14:32:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78afda9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80ca270 con 0x7f78a815af80 2023-10-16T14:32:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78ad344640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 0x7f78a80c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78ad344640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 0x7f78a80c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54876/0 (socket says 172.21.15.29:54876) 2023-10-16T14:32:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.710+0000 7f78ad344640 1 -- 172.21.15.29:0/3309479093 learned_addr learned my addr 172.21.15.29:0/3309479093 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78ad344640 1 -- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a815af80 msgr2=0x7f78a80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78ad344640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a815af80 0x7f78a80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78ad344640 1 -- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a8065ab0 msgr2=0x7f78a80c3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78ad344640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a8065ab0 0x7f78a80c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78ad344640 1 -- 172.21.15.29:0/3309479093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00094d0 con 0x7f78a806faf0 2023-10-16T14:32:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78ad344640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 0x7f78a80c9260 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f789800b3e0 tx=0x7f789800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898011830 con 0x7f78a806faf0 2023-10-16T14:32:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78980119b0 con 0x7f78a806faf0 2023-10-16T14:32:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898011d60 con 0x7f78a806faf0 2023-10-16T14:32:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a80ca550 con 0x7f78a806faf0 2023-10-16T14:32:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.711+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a80c3660 con 0x7f78a806faf0 2023-10-16T14:32:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.713+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a80caa70 con 0x7f78a806faf0 2023-10-16T14:32:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.714+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f789801a400 con 0x7f78a806faf0 2023-10-16T14:32:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.714+0000 7f789e7fc640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f788c06af90 0x7f788c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.715+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7898089a30 con 0x7f78a806faf0 2023-10-16T14:32:14.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.715+0000 7f78adb45640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f788c06af90 0x7f788c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:14.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.717+0000 7f78adb45640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f788c06af90 0x7f788c06d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f78a400ab90 tx=0x7f78a4009250 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.719+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78a80caa70 con 0x7f78a806faf0 2023-10-16T14:32:14.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.912+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f78a80caa70 con 0x7f78a806faf0 2023-10-16T14:32:14.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.920+0000 7f789e7fc640 1 -- 172.21.15.29:0/3309479093 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f78a80caa70 con 0x7f78a806faf0 2023-10-16T14:32:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f788c06af90 msgr2=0x7f788c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f788c06af90 0x7f788c06d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f78a400ab90 tx=0x7f78a4009250 comp rx=0 tx=0).stop 2023-10-16T14:32:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 msgr2=0x7f78a80c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 0x7f78a80c9260 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f789800b3e0 tx=0x7f789800b8a0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 shutdown_connections 2023-10-16T14:32:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f788c06af90 0x7f788c06d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78a806faf0 0x7f78a80c9260 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78a815af80 0x7f78a80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 --2- 172.21.15.29:0/3309479093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78a8065ab0 0x7f78a80c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 >> 172.21.15.29:0/3309479093 conn(0x7f78a80d0ab0 msgr2=0x7f78a80c3960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 shutdown_connections 2023-10-16T14:32:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:14.923+0000 7f78afda9640 1 -- 172.21.15.29:0/3309479093 wait complete. 2023-10-16T14:32:14.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:32:15.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.180+0000 7f404b168640 1 Processor -- start 2023-10-16T14:32:15.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.180+0000 7f404b168640 1 -- start start 2023-10-16T14:32:15.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.181+0000 7f404b168640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f404406b8c0 0x7f404406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.181+0000 7f404b168640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40440b0970 con 0x7f4044160440 2023-10-16T14:32:15.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.181+0000 7f404b168640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40440cfca0 con 0x7f40440b0e00 2023-10-16T14:32:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.181+0000 7f404b168640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404406b440 con 0x7f404406b8c0 2023-10-16T14:32:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.181+0000 7f4048f04640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4044160440 0x7f40440d1410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59266/0 (socket says 172.21.15.29:59266) 2023-10-16T14:32:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.181+0000 7f4048f04640 1 -- 172.21.15.29:0/3737251381 learned_addr learned my addr 172.21.15.29:0/3737251381 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.182+0000 7f4043fff640 1 --2- 172.21.15.29:0/3737251381 >> v2:172.21.15.29:3300/0 conn(0x7f404406b8c0 0x7f404406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.182+0000 7f40437fe640 1 -- 172.21.15.29:0/3737251381 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053983056 0 0) 0x7f40440b0970 con 0x7f4044160440 2023-10-16T14:32:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.182+0000 7f40437fe640 1 -- 172.21.15.29:0/3737251381 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4030003520 con 0x7f4044160440 2023-10-16T14:32:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.182+0000 7f4043fff640 1 -- 172.21.15.29:0/3737251381 >> v1:172.21.15.170:6789/0 conn(0x7f40440b0e00 legacy=0x7f40440b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:15.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.182+0000 7f4043fff640 1 -- 172.21.15.29:0/3737251381 >> v1:172.21.15.29:6789/0 conn(0x7f4044160440 legacy=0x7f40440d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.182+0000 7f4043fff640 1 -- 172.21.15.29:0/3737251381 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40440cff80 con 0x7f404406b8c0 2023-10-16T14:32:15.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.183+0000 7f4043fff640 1 --2- 172.21.15.29:0/3737251381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404406b8c0 0x7f404406bd50 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f402c0097a0 tx=0x7f402c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd94ac2fe11a2c1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:15.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f40437fe640 1 -- 172.21.15.29:0/3737251381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c031830 con 0x7f404406b8c0 2023-10-16T14:32:15.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f40437fe640 1 -- 172.21.15.29:0/3737251381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c031990 con 0x7f404406b8c0 2023-10-16T14:32:15.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f40437fe640 1 -- 172.21.15.29:0/3737251381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c031d20 con 0x7f404406b8c0 2023-10-16T14:32:15.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 -- 172.21.15.29:0/3737251381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404406b8c0 msgr2=0x7f404406bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 --2- 172.21.15.29:0/3737251381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404406b8c0 0x7f404406bd50 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f402c0097a0 tx=0x7f402c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 -- 172.21.15.29:0/3737251381 shutdown_connections 2023-10-16T14:32:15.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 --2- 172.21.15.29:0/3737251381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f404406b8c0 0x7f404406bd50 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 -- 172.21.15.29:0/3737251381 >> 172.21.15.29:0/3737251381 conn(0x7f4044099ea0 msgr2=0x7f40440d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:15.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 -- 172.21.15.29:0/3737251381 shutdown_connections 2023-10-16T14:32:15.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.184+0000 7f404b168640 1 -- 172.21.15.29:0/3737251381 wait complete. 2023-10-16T14:32:15.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.186+0000 7f404b168640 1 Processor -- start 2023-10-16T14:32:15.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.186+0000 7f404b168640 1 -- start start 2023-10-16T14:32:15.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.187+0000 7f404b168640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f404406b8c0 0x7f404414f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.187+0000 7f404b168640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 0x7f40440c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.187+0000 7f404b168640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4044160440 0x7f40441530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.187+0000 7f404b168640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404414fc00 con 0x7f40440b0e00 2023-10-16T14:32:15.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.187+0000 7f404b168640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440c5e20 con 0x7f404406b8c0 2023-10-16T14:32:15.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.187+0000 7f404b168640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40440c6100 con 0x7f4044160440 2023-10-16T14:32:15.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 0x7f40440c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 0x7f40440c55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54882/0 (socket says 172.21.15.29:54882) 2023-10-16T14:32:15.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 -- 172.21.15.29:0/3829663632 learned_addr learned my addr 172.21.15.29:0/3829663632 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:15.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4048f04640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f404406b8c0 0x7f404414f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4049705640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4044160440 0x7f40441530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 -- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4044160440 msgr2=0x7f40441530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4044160440 0x7f40441530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 -- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f404406b8c0 msgr2=0x7f404414f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f404406b8c0 0x7f404414f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.188+0000 7f4043fff640 1 -- 172.21.15.29:0/3829663632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009400 con 0x7f40440b0e00 2023-10-16T14:32:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.189+0000 7f4043fff640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 0x7f40440c55d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f402c000c00 tx=0x7f402c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.189+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c03a8f0 con 0x7f40440b0e00 2023-10-16T14:32:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.189+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c03aa50 con 0x7f40440b0e00 2023-10-16T14:32:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.189+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c03ae00 con 0x7f40440b0e00 2023-10-16T14:32:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.189+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40440c63e0 con 0x7f40440b0e00 2023-10-16T14:32:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.189+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40440c69a0 con 0x7f40440b0e00 2023-10-16T14:32:15.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.190+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40440c6b90 con 0x7f40440b0e00 2023-10-16T14:32:15.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.191+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f402c04d450 con 0x7f40440b0e00 2023-10-16T14:32:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.192+0000 7f40417fa640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f402406af90 0x7f402406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.192+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f402c0ab8c0 con 0x7f40440b0e00 2023-10-16T14:32:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.193+0000 7f4048f04640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f402406af90 0x7f402406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.194+0000 7f4048f04640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f402406af90 0x7f402406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4034001460 tx=0x7f4034009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:15.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.197+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f402c076790 con 0x7f40440b0e00 2023-10-16T14:32:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.392+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f404404e160 con 0x7f40440b0e00 2023-10-16T14:32:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.399+0000 7f40417fa640 1 -- 172.21.15.29:0/3829663632 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7f402c035070 con 0x7f40440b0e00 2023-10-16T14:32:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.401+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f402406af90 msgr2=0x7f402406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.401+0000 7f404b168640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f402406af90 0x7f402406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4034001460 tx=0x7f4034009040 comp rx=0 tx=0).stop 2023-10-16T14:32:15.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.401+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 msgr2=0x7f40440c55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.401+0000 7f404b168640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 0x7f40440c55d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f402c000c00 tx=0x7f402c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 shutdown_connections 2023-10-16T14:32:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f402406af90 0x7f402406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40440b0e00 0x7f40440c55d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4044160440 0x7f40441530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 --2- 172.21.15.29:0/3829663632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f404406b8c0 0x7f404414f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 >> 172.21.15.29:0/3829663632 conn(0x7f4044099ea0 msgr2=0x7f40440b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 shutdown_connections 2023-10-16T14:32:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.402+0000 7f404b168640 1 -- 172.21.15.29:0/3829663632 wait complete. 2023-10-16T14:32:15.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:32:15.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.651+0000 7fe8a2490640 1 Processor -- start 2023-10-16T14:32:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.652+0000 7fe8a2490640 1 -- start start 2023-10-16T14:32:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.653+0000 7fe8a2490640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe89c159950 0x7fe89c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.653+0000 7fe8a2490640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe89c06c810 0x7fe89c0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.653+0000 7fe8a2490640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c15aef0 con 0x7fe89c070450 2023-10-16T14:32:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.653+0000 7fe8a2490640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c070e20 con 0x7fe89c159950 2023-10-16T14:32:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.653+0000 7fe8a2490640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c15e490 con 0x7fe89c06c810 2023-10-16T14:32:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a0c8d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe89c070450 0x7fe89c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59270/0 (socket says 172.21.15.29:59270) 2023-10-16T14:32:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a0c8d640 1 -- 172.21.15.29:0/1160632040 learned_addr learned my addr 172.21.15.29:0/1160632040 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a148e640 1 --2- 172.21.15.29:0/1160632040 >> v2:172.21.15.29:3300/0 conn(0x7fe89c159950 0x7fe89c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a1c8f640 1 --2- 172.21.15.29:0/1160632040 >> v2:172.21.15.170:3300/0 conn(0x7fe89c06c810 0x7fe89c0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe893fff640 1 -- 172.21.15.29:0/1160632040 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988573831 0 0) 0x7fe88c001110 con 0x7fe89c070450 2023-10-16T14:32:15.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe893fff640 1 -- 172.21.15.29:0/1160632040 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe888003520 con 0x7fe89c070450 2023-10-16T14:32:15.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a148e640 1 -- 172.21.15.29:0/1160632040 >> v2:172.21.15.170:3300/0 conn(0x7fe89c06c810 msgr2=0x7fe89c0708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a148e640 1 --2- 172.21.15.29:0/1160632040 >> v2:172.21.15.170:3300/0 conn(0x7fe89c06c810 0x7fe89c0708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a148e640 1 -- 172.21.15.29:0/1160632040 >> v1:172.21.15.29:6789/0 conn(0x7fe89c070450 legacy=0x7fe89c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.654+0000 7fe8a148e640 1 -- 172.21.15.29:0/1160632040 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c15d910 con 0x7fe89c159950 2023-10-16T14:32:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.656+0000 7fe8a148e640 1 --2- 172.21.15.29:0/1160632040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c159950 0x7fe89c15a9b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0716a0 tx=0x7fe89802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a876f97d9425cfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.656+0000 7fe893fff640 1 -- 172.21.15.29:0/1160632040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89802e070 con 0x7fe89c159950 2023-10-16T14:32:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.656+0000 7fe893fff640 1 -- 172.21.15.29:0/1160632040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8980315a0 con 0x7fe89c159950 2023-10-16T14:32:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.656+0000 7fe893fff640 1 -- 172.21.15.29:0/1160632040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898031930 con 0x7fe89c159950 2023-10-16T14:32:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 -- 172.21.15.29:0/1160632040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c159950 msgr2=0x7fe89c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 --2- 172.21.15.29:0/1160632040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c159950 0x7fe89c15a9b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0716a0 tx=0x7fe89802b840 comp rx=0 tx=0).stop 2023-10-16T14:32:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 -- 172.21.15.29:0/1160632040 shutdown_connections 2023-10-16T14:32:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 --2- 172.21.15.29:0/1160632040 >> v2:172.21.15.170:3300/0 conn(0x7fe89c06c810 0x7fe89c0708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 --2- 172.21.15.29:0/1160632040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c159950 0x7fe89c15a9b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 -- 172.21.15.29:0/1160632040 >> 172.21.15.29:0/1160632040 conn(0x7fe89c0ce6a0 msgr2=0x7fe89c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 -- 172.21.15.29:0/1160632040 shutdown_connections 2023-10-16T14:32:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.657+0000 7fe8a2490640 1 -- 172.21.15.29:0/1160632040 wait complete. 2023-10-16T14:32:15.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.659+0000 7fe8a2490640 1 Processor -- start 2023-10-16T14:32:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.659+0000 7fe8a2490640 1 -- start start 2023-10-16T14:32:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a2490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 0x7fe89c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a2490640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe89c070450 0x7fe89c167120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a2490640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe89c159950 0x7fe89c1658f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a2490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0ad8d0 con 0x7fe89c06c810 2023-10-16T14:32:15.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a2490640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0acc50 con 0x7fe89c070450 2023-10-16T14:32:15.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a2490640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0acf30 con 0x7fe89c159950 2023-10-16T14:32:15.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a148e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 0x7fe89c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a148e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 0x7fe89c0b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54900/0 (socket says 172.21.15.29:54900) 2023-10-16T14:32:15.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.660+0000 7fe8a148e640 1 -- 172.21.15.29:0/2052144363 learned_addr learned my addr 172.21.15.29:0/2052144363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:15.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a148e640 1 -- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe89c159950 msgr2=0x7fe89c1658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a148e640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe89c159950 0x7fe89c1658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a148e640 1 -- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe89c070450 msgr2=0x7fe89c167120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a148e640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe89c070450 0x7fe89c167120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a148e640 1 -- 172.21.15.29:0/2052144363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898009400 con 0x7fe89c06c810 2023-10-16T14:32:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a148e640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 0x7fe89c0b03e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fe898009890 tx=0x7fe8980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898032840 con 0x7fe89c06c810 2023-10-16T14:32:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898031f00 con 0x7fe89c06c810 2023-10-16T14:32:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898032be0 con 0x7fe89c06c810 2023-10-16T14:32:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.661+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c0ad210 con 0x7fe89c06c810 2023-10-16T14:32:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.662+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c0ac890 con 0x7fe89c06c810 2023-10-16T14:32:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.663+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c04e9b0 con 0x7fe89c06c810 2023-10-16T14:32:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.664+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe898032d40 con 0x7fe89c06c810 2023-10-16T14:32:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.665+0000 7fe891ffb640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe88006afe0 0x7fe88006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.665+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe8980ab680 con 0x7fe89c06c810 2023-10-16T14:32:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.666+0000 7fe8a0c8d640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe88006afe0 0x7fe88006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.667+0000 7fe8a0c8d640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe88006afe0 0x7fe88006d4a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe88c005880 tx=0x7fe88c001150 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.670+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe898076640 con 0x7fe89c06c810 2023-10-16T14:32:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.862+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe89c0b0f50 con 0x7fe89c06c810 2023-10-16T14:32:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.868+0000 7fe891ffb640 1 -- 172.21.15.29:0/2052144363 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe898079c60 con 0x7fe89c06c810 2023-10-16T14:32:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe88006afe0 msgr2=0x7fe88006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe88006afe0 0x7fe88006d4a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe88c005880 tx=0x7fe88c001150 comp rx=0 tx=0).stop 2023-10-16T14:32:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 msgr2=0x7fe89c0b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 0x7fe89c0b03e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fe898009890 tx=0x7fe8980046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 shutdown_connections 2023-10-16T14:32:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe88006afe0 0x7fe88006d4a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe89c159950 0x7fe89c1658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe89c070450 0x7fe89c167120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 --2- 172.21.15.29:0/2052144363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89c06c810 0x7fe89c0b03e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.870+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 >> 172.21.15.29:0/2052144363 conn(0x7fe89c0ce6a0 msgr2=0x7fe89c15d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.871+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 shutdown_connections 2023-10-16T14:32:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:15.871+0000 7fe8a2490640 1 -- 172.21.15.29:0/2052144363 wait complete. 2023-10-16T14:32:15.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T14:32:16.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.121+0000 7fc715335640 1 Processor -- start 2023-10-16T14:32:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.122+0000 7fc715335640 1 -- start start 2023-10-16T14:32:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.123+0000 7fc715335640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc710070d70 0x7fc71006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.123+0000 7fc715335640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7100712c0 0x7fc71006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.123+0000 7fc715335640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71006c9c0 con 0x7fc71015aad0 2023-10-16T14:32:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.123+0000 7fc715335640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100b4620 con 0x7fc7100712c0 2023-10-16T14:32:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.123+0000 7fc715335640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100b4340 con 0x7fc710070d70 2023-10-16T14:32:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc70ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc71015aad0 0x7fc7100cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59286/0 (socket says 172.21.15.29:59286) 2023-10-16T14:32:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc70ffff640 1 -- 172.21.15.29:0/47618572 learned_addr learned my addr 172.21.15.29:0/47618572 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc714b34640 1 --2- 172.21.15.29:0/47618572 >> v2:172.21.15.29:3300/0 conn(0x7fc7100712c0 0x7fc71006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc707fff640 1 --2- 172.21.15.29:0/47618572 >> v2:172.21.15.170:3300/0 conn(0x7fc710070d70 0x7fc71006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc70f7fe640 1 -- 172.21.15.29:0/47618572 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846907351 0 0) 0x7fc71006c9c0 con 0x7fc71015aad0 2023-10-16T14:32:16.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc714b34640 1 -- 172.21.15.29:0/47618572 >> v2:172.21.15.170:3300/0 conn(0x7fc710070d70 msgr2=0x7fc71006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:16.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc714b34640 1 --2- 172.21.15.29:0/47618572 >> v2:172.21.15.170:3300/0 conn(0x7fc710070d70 0x7fc71006bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc714b34640 1 -- 172.21.15.29:0/47618572 >> v1:172.21.15.29:6789/0 conn(0x7fc71015aad0 legacy=0x7fc7100cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.124+0000 7fc714b34640 1 -- 172.21.15.29:0/47618572 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71006f520 con 0x7fc7100712c0 2023-10-16T14:32:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.125+0000 7fc714b34640 1 --2- 172.21.15.29:0/47618572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7100712c0 0x7fc71006c480 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc71006d4a0 tx=0x7fc70802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b9c8ce1b77230f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.126+0000 7fc70f7fe640 1 -- 172.21.15.29:0/47618572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70802cd10 con 0x7fc7100712c0 2023-10-16T14:32:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.126+0000 7fc70f7fe640 1 -- 172.21.15.29:0/47618572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70802ce70 con 0x7fc7100712c0 2023-10-16T14:32:16.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.126+0000 7fc70f7fe640 1 -- 172.21.15.29:0/47618572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708034640 con 0x7fc7100712c0 2023-10-16T14:32:16.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.126+0000 7fc715335640 1 -- 172.21.15.29:0/47618572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7100712c0 msgr2=0x7fc71006c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.126+0000 7fc715335640 1 --2- 172.21.15.29:0/47618572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7100712c0 0x7fc71006c480 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc71006d4a0 tx=0x7fc70802bf60 comp rx=0 tx=0).stop 2023-10-16T14:32:16.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.127+0000 7fc715335640 1 -- 172.21.15.29:0/47618572 shutdown_connections 2023-10-16T14:32:16.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.127+0000 7fc715335640 1 --2- 172.21.15.29:0/47618572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7100712c0 0x7fc71006c480 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.127+0000 7fc715335640 1 --2- 172.21.15.29:0/47618572 >> v2:172.21.15.170:3300/0 conn(0x7fc710070d70 0x7fc71006bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.127+0000 7fc715335640 1 -- 172.21.15.29:0/47618572 >> 172.21.15.29:0/47618572 conn(0x7fc7100b0a90 msgr2=0x7fc7100ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:16.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.127+0000 7fc715335640 1 -- 172.21.15.29:0/47618572 shutdown_connections 2023-10-16T14:32:16.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.127+0000 7fc715335640 1 -- 172.21.15.29:0/47618572 wait complete. 2023-10-16T14:32:16.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.128+0000 7fc715335640 1 Processor -- start 2023-10-16T14:32:16.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.129+0000 7fc715335640 1 -- start start 2023-10-16T14:32:16.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc715335640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc710070d70 0x7fc710070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc715335640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7100712c0 0x7fc7100c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc715335640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 0x7fc7101ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc715335640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100c7480 con 0x7fc71015aad0 2023-10-16T14:32:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc715335640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015d920 con 0x7fc710070d70 2023-10-16T14:32:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc715335640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015dc00 con 0x7fc7100712c0 2023-10-16T14:32:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc714b34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 0x7fc7101ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc714b34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 0x7fc7101ef190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54912/0 (socket says 172.21.15.29:54912) 2023-10-16T14:32:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc714b34640 1 -- 172.21.15.29:0/4042582498 learned_addr learned my addr 172.21.15.29:0/4042582498 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc707fff640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7100712c0 0x7fc7100c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.130+0000 7fc714b34640 1 -- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7100712c0 msgr2=0x7fc7100c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc714b34640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7100712c0 0x7fc7100c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc714b34640 1 -- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc710070d70 msgr2=0x7fc710070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc714b34640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc710070d70 0x7fc710070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc714b34640 1 -- 172.21.15.29:0/4042582498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708009a70 con 0x7fc71015aad0 2023-10-16T14:32:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc714b34640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 0x7fc7101ef190 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc70802c570 tx=0x7fc708004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70803e610 con 0x7fc71015aad0 2023-10-16T14:32:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70803e770 con 0x7fc71015aad0 2023-10-16T14:32:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70803eae0 con 0x7fc71015aad0 2023-10-16T14:32:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.131+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7100c7110 con 0x7fc71015aad0 2023-10-16T14:32:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.132+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7101475a0 con 0x7fc71015aad0 2023-10-16T14:32:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.133+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71004e9b0 con 0x7fc71015aad0 2023-10-16T14:32:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.134+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc70803b050 con 0x7fc71015aad0 2023-10-16T14:32:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.135+0000 7fc70d7fa640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc6f006afc0 0x7fc6f006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.135+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc7080abd50 con 0x7fc71015aad0 2023-10-16T14:32:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.136+0000 7fc70ffff640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc6f006afc0 0x7fc6f006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.137+0000 7fc70ffff640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc6f006afc0 0x7fc6f006d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc700007fd0 tx=0x7fc700007960 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:16.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.139+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7080af030 con 0x7fc71015aad0 2023-10-16T14:32:16.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.406+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fc7100c75c0 con 0x7fc71015aad0 2023-10-16T14:32:16.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.459+0000 7fc70d7fa640 1 -- 172.21.15.29:0/4042582498 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v573) v1 ==== 154+0+0 (secure 0 0 0) 0x7fc708076e60 con 0x7fc71015aad0 2023-10-16T14:32:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc6f006afc0 msgr2=0x7fc6f006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc6f006afc0 0x7fc6f006d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc700007fd0 tx=0x7fc700007960 comp rx=0 tx=0).stop 2023-10-16T14:32:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 msgr2=0x7fc7101ef190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 0x7fc7101ef190 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc70802c570 tx=0x7fc708004a40 comp rx=0 tx=0).stop 2023-10-16T14:32:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 shutdown_connections 2023-10-16T14:32:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc6f006afc0 0x7fc6f006d480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc71015aad0 0x7fc7101ef190 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7100712c0 0x7fc7100c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 --2- 172.21.15.29:0/4042582498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc710070d70 0x7fc710070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.461+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 >> 172.21.15.29:0/4042582498 conn(0x7fc7100b0a90 msgr2=0x7fc7100af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.462+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 shutdown_connections 2023-10-16T14:32:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.462+0000 7fc715335640 1 -- 172.21.15.29:0/4042582498 wait complete. 2023-10-16T14:32:16.484 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-16T14:32:16.485 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-16T14:32:16.485 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:32:16.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.707+0000 7f66aa285640 1 Processor -- start 2023-10-16T14:32:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.707+0000 7f66aa285640 1 -- start start 2023-10-16T14:32:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.708+0000 7f66aa285640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66a4159890 0x7f66a4070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.708+0000 7f66aa285640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40b03b0 0x7f66a40cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.708+0000 7f66aa285640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a4071510 con 0x7f66a40710b0 2023-10-16T14:32:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.708+0000 7f66aa285640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a40b1720 con 0x7f66a4159890 2023-10-16T14:32:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.708+0000 7f66aa285640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40b1a00 con 0x7f66a40b03b0 2023-10-16T14:32:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a8a82640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f66a40710b0 0x7f66a40afca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59288/0 (socket says 172.21.15.29:59288) 2023-10-16T14:32:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a8a82640 1 -- 172.21.15.29:0/1267345892 learned_addr learned my addr 172.21.15.29:0/1267345892 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a9283640 1 --2- 172.21.15.29:0/1267345892 >> v2:172.21.15.29:3300/0 conn(0x7f66a4159890 0x7f66a4070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a9a84640 1 --2- 172.21.15.29:0/1267345892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40b03b0 0x7f66a40cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a9283640 1 -- 172.21.15.29:0/1267345892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40b03b0 msgr2=0x7f66a40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a9283640 1 --2- 172.21.15.29:0/1267345892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40b03b0 0x7f66a40cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a9283640 1 -- 172.21.15.29:0/1267345892 >> v1:172.21.15.29:6789/0 conn(0x7f66a40710b0 legacy=0x7f66a40afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f66a9283640 1 -- 172.21.15.29:0/1267345892 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40b1ce0 con 0x7f66a4159890 2023-10-16T14:32:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.709+0000 7f669bfff640 1 -- 172.21.15.29:0/1267345892 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573122646 0 0) 0x7f6694001110 con 0x7f66a40710b0 2023-10-16T14:32:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.710+0000 7f66a9283640 1 --2- 172.21.15.29:0/1267345892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a4159890 0x7f66a4070b70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f66a00097a0 tx=0x7f66a002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d699264a68063e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f669bfff640 1 -- 172.21.15.29:0/1267345892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a002d040 con 0x7f66a4159890 2023-10-16T14:32:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f669bfff640 1 -- 172.21.15.29:0/1267345892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a0031810 con 0x7f66a4159890 2023-10-16T14:32:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f669bfff640 1 -- 172.21.15.29:0/1267345892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0031ba0 con 0x7f66a4159890 2023-10-16T14:32:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f66aa285640 1 -- 172.21.15.29:0/1267345892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a4159890 msgr2=0x7f66a4070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f66aa285640 1 --2- 172.21.15.29:0/1267345892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a4159890 0x7f66a4070b70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f66a00097a0 tx=0x7f66a002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f66aa285640 1 -- 172.21.15.29:0/1267345892 shutdown_connections 2023-10-16T14:32:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f66aa285640 1 --2- 172.21.15.29:0/1267345892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40b03b0 0x7f66a40cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f66aa285640 1 --2- 172.21.15.29:0/1267345892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a4159890 0x7f66a4070b70 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.711+0000 7f66aa285640 1 -- 172.21.15.29:0/1267345892 >> 172.21.15.29:0/1267345892 conn(0x7f66a4069da0 msgr2=0x7f66a406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:16.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.712+0000 7f66aa285640 1 -- 172.21.15.29:0/1267345892 shutdown_connections 2023-10-16T14:32:16.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.712+0000 7f66aa285640 1 -- 172.21.15.29:0/1267345892 wait complete. 2023-10-16T14:32:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.713+0000 7f66aa285640 1 Processor -- start 2023-10-16T14:32:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.713+0000 7f66aa285640 1 -- start start 2023-10-16T14:32:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.714+0000 7f66aa285640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40710b0 0x7f66a41525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.714+0000 7f66aa285640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 0x7f66a406c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.714+0000 7f66aa285640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66a4159890 0x7f66a414fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.714+0000 7f66aa285640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4152b10 con 0x7f66a40b03b0 2023-10-16T14:32:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66aa285640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a414f530 con 0x7f66a4159890 2023-10-16T14:32:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66aa285640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a41f2db0 con 0x7f66a40710b0 2023-10-16T14:32:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 0x7f66a406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 0x7f66a406c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54922/0 (socket says 172.21.15.29:54922) 2023-10-16T14:32:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 -- 172.21.15.29:0/3320012171 learned_addr learned my addr 172.21.15.29:0/3320012171 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a9283640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40710b0 0x7f66a41525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a9a84640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66a4159890 0x7f66a414fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 -- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40710b0 msgr2=0x7f66a41525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40710b0 0x7f66a41525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 -- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66a4159890 msgr2=0x7f66a414fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66a4159890 0x7f66a414fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.715+0000 7f66a8a82640 1 -- 172.21.15.29:0/3320012171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a0009400 con 0x7f66a40b03b0 2023-10-16T14:32:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.716+0000 7f66a8a82640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 0x7f66a406c250 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f669400c750 tx=0x7f669400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.716+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694005ab0 con 0x7f66a40b03b0 2023-10-16T14:32:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.716+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6694005c10 con 0x7f66a40b03b0 2023-10-16T14:32:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.716+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694007070 con 0x7f66a40b03b0 2023-10-16T14:32:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.716+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a41f3090 con 0x7f66a40b03b0 2023-10-16T14:32:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.716+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a4151dd0 con 0x7f66a40b03b0 2023-10-16T14:32:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.717+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a4151950 con 0x7f66a40b03b0 2023-10-16T14:32:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.718+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f66940071d0 con 0x7f66a40b03b0 2023-10-16T14:32:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.719+0000 7f6699ffb640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f668806af90 0x7f668806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.719+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f6694088c00 con 0x7f66a40b03b0 2023-10-16T14:32:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.720+0000 7f66a9283640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f668806af90 0x7f668806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.721+0000 7f66a9283640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f668806af90 0x7f668806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f66a0036000 tx=0x7f66a003c040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:16.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.723+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66a4151950 con 0x7f66a40b03b0 2023-10-16T14:32:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.918+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66a4071510 con 0x7f66a40b03b0 2023-10-16T14:32:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.920+0000 7f6699ffb640 1 -- 172.21.15.29:0/3320012171 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+14002 (secure 0 0 0) 0x7f66a4071510 con 0x7f66a40b03b0 2023-10-16T14:32:16.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:16.925 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:14.451033+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6834/1869070988":"2023-10-17T14:32:11.376982+0000","172.21.15.170:0/76368088":"2023-10-17T14:32:07.341132+0000","172.21.15.170:6832/636880041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2930685041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2214932809":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/2595110151":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3247649762":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2698007731":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/4263280508":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/2380256585":"2023-10-17T14:30:30.217790+0000","172.21.15.29:6835/1869070988":"2023-10-17T14:32:11.376982+0000","172.21.15.170:0/4004225158":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3575752004":"2023-10-17T14:30:30.217790+0000","172.21.15.170:0/3621036751":"2023-10-17T14:32:07.341132+0000","172.21.15.170:0/3937243709":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6833/636880041":"2023-10-17T14:32:07.341132+0000","172.21.15.170:6833/3423886595":"2023-10-17T14:30:30.217790+0000","172.21.15.170:6832/3423886595":"2023-10-17T14:30:30.217790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f668806af90 msgr2=0x7f668806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f668806af90 0x7f668806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f66a0036000 tx=0x7f66a003c040 comp rx=0 tx=0).stop 2023-10-16T14:32:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 msgr2=0x7f66a406c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 0x7f66a406c250 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f669400c750 tx=0x7f669400cc10 comp rx=0 tx=0).stop 2023-10-16T14:32:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 shutdown_connections 2023-10-16T14:32:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f668806af90 0x7f668806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66a40b03b0 0x7f66a406c250 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66a4159890 0x7f66a414fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 --2- 172.21.15.29:0/3320012171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66a40710b0 0x7f66a41525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 >> 172.21.15.29:0/3320012171 conn(0x7f66a4069da0 msgr2=0x7f66a4158760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 shutdown_connections 2023-10-16T14:32:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:16.923+0000 7f66aa285640 1 -- 172.21.15.29:0/3320012171 wait complete. 2023-10-16T14:32:16.947 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:16.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:17.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.168+0000 7ff70317a640 1 Processor -- start 2023-10-16T14:32:17.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.168+0000 7ff70317a640 1 -- start start 2023-10-16T14:32:17.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff70317a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6fc15ceb0 0x7ff6fc0d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff70317a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc0d3650 0x7ff6fc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff70317a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc0d0f40 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff70317a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6fc0d1cb0 con 0x7ff6fc15ceb0 2023-10-16T14:32:17.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff70317a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0d0c60 con 0x7ff6fc0d3650 2023-10-16T14:32:17.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff702178640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6fc15ceb0 0x7ff6fc0d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff702979640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6fc0b2cc0 0x7ff6fc0b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59300/0 (socket says 172.21.15.29:59300) 2023-10-16T14:32:17.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff702979640 1 -- 172.21.15.29:0/3408807794 learned_addr learned my addr 172.21.15.29:0/3408807794 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:17.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.170+0000 7ff701977640 1 --2- 172.21.15.29:0/3408807794 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc0d3650 0x7ff6fc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.171+0000 7ff702178640 1 -- 172.21.15.29:0/3408807794 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc0d3650 msgr2=0x7ff6fc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:17.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.171+0000 7ff702178640 1 --2- 172.21.15.29:0/3408807794 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc0d3650 0x7ff6fc0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.171+0000 7ff702178640 1 -- 172.21.15.29:0/3408807794 >> v1:172.21.15.29:6789/0 conn(0x7ff6fc0b2cc0 legacy=0x7ff6fc0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.171+0000 7ff702178640 1 -- 172.21.15.29:0/3408807794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc0b21e0 con 0x7ff6fc15ceb0 2023-10-16T14:32:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.172+0000 7ff702178640 1 --2- 172.21.15.29:0/3408807794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc15ceb0 0x7ff6fc0d10f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009d90 tx=0x7ff6f002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e445ffedce363f70 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.172+0000 7ff701176640 1 -- 172.21.15.29:0/3408807794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002cbd0 con 0x7ff6fc15ceb0 2023-10-16T14:32:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.172+0000 7ff701176640 1 -- 172.21.15.29:0/3408807794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f002cd30 con 0x7ff6fc15ceb0 2023-10-16T14:32:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.172+0000 7ff701176640 1 -- 172.21.15.29:0/3408807794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002cea0 con 0x7ff6fc15ceb0 2023-10-16T14:32:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 -- 172.21.15.29:0/3408807794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc15ceb0 msgr2=0x7ff6fc0d10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 --2- 172.21.15.29:0/3408807794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc15ceb0 0x7ff6fc0d10f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009d90 tx=0x7ff6f002be20 comp rx=0 tx=0).stop 2023-10-16T14:32:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 -- 172.21.15.29:0/3408807794 shutdown_connections 2023-10-16T14:32:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 --2- 172.21.15.29:0/3408807794 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc0d3650 0x7ff6fc0d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 --2- 172.21.15.29:0/3408807794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc15ceb0 0x7ff6fc0d10f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 -- 172.21.15.29:0/3408807794 >> 172.21.15.29:0/3408807794 conn(0x7ff6fc0d2b30 msgr2=0x7ff6fc0d2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 -- 172.21.15.29:0/3408807794 shutdown_connections 2023-10-16T14:32:17.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.173+0000 7ff70317a640 1 -- 172.21.15.29:0/3408807794 wait complete. 2023-10-16T14:32:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.175+0000 7ff70317a640 1 Processor -- start 2023-10-16T14:32:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.176+0000 7ff70317a640 1 -- start start 2023-10-16T14:32:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff70317a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 0x7ff6fc0afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff70317a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6fc0d3650 0x7ff6fc0b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff70317a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc15ceb0 0x7ff6fc14dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff70317a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc09d9c0 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff70317a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc09dca0 con 0x7ff6fc0d3650 2023-10-16T14:32:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff70317a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc09df80 con 0x7ff6fc15ceb0 2023-10-16T14:32:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff702178640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 0x7ff6fc0afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff702178640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 0x7ff6fc0afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54944/0 (socket says 172.21.15.29:54944) 2023-10-16T14:32:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.177+0000 7ff702178640 1 -- 172.21.15.29:0/423211476 learned_addr learned my addr 172.21.15.29:0/423211476 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff702178640 1 -- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc15ceb0 msgr2=0x7ff6fc14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff702178640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc15ceb0 0x7ff6fc14dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff702178640 1 -- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6fc0d3650 msgr2=0x7ff6fc0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff702178640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6fc0d3650 0x7ff6fc0b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff702178640 1 -- 172.21.15.29:0/423211476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0009a70 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff702178640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 0x7ff6fc0afce0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009d90 tx=0x7ff6f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002cea0 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f0040650 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0040a00 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.178+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6fc09e260 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.179+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc14ed90 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.180+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6fc09e780 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.181+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6f0040b60 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.182+0000 7ff6faffd640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff6e006af90 0x7ff6e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.182+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ff6f00ac480 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.183+0000 7ff701977640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff6e006af90 0x7ff6e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.184+0000 7ff701977640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff6e006af90 0x7ff6e006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc1f55e0 tx=0x7ff6e8008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.187+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6fc09e780 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.382+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6fc09e780 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.383+0000 7ff6faffd640 1 -- 172.21.15.29:0/423211476 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6fc09e780 con 0x7ff6fc0b2cc0 2023-10-16T14:32:17.387 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:17.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff6e006af90 msgr2=0x7ff6e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff6e006af90 0x7ff6e006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc1f55e0 tx=0x7ff6e8008040 comp rx=0 tx=0).stop 2023-10-16T14:32:17.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 msgr2=0x7ff6fc0afce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 0x7ff6fc0afce0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009d90 tx=0x7ff6f0004a40 comp rx=0 tx=0).stop 2023-10-16T14:32:17.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 shutdown_connections 2023-10-16T14:32:17.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff6e006af90 0x7ff6e006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6fc15ceb0 0x7ff6fc14dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6fc0d3650 0x7ff6fc0b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 --2- 172.21.15.29:0/423211476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0b2cc0 0x7ff6fc0afce0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 >> 172.21.15.29:0/423211476 conn(0x7ff6fc0d2b30 msgr2=0x7ff6fc0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:17.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 shutdown_connections 2023-10-16T14:32:17.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.386+0000 7ff70317a640 1 -- 172.21.15.29:0/423211476 wait complete. 2023-10-16T14:32:17.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T14:32:17.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:32:17.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:32:17.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.614+0000 7f5654fed640 1 Processor -- start 2023-10-16T14:32:17.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.614+0000 7f5654fed640 1 -- start start 2023-10-16T14:32:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.615+0000 7f5654fed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56501599c0 0x7f56500ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f5654fed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565006bab0 con 0x7f56500ca6f0 2023-10-16T14:32:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f5654fed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565006bd90 con 0x7f56500cda90 2023-10-16T14:32:17.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f5654fed640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565006f300 con 0x7f56501599c0 2023-10-16T14:32:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f564f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56501599c0 0x7f56500ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f564f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56501599c0 0x7f56500ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54948/0 (socket says 172.21.15.29:54948) 2023-10-16T14:32:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f564f7fe640 1 -- 172.21.15.29:0/1913827241 learned_addr learned my addr 172.21.15.29:0/1913827241 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f564f7fe640 1 -- 172.21.15.29:0/1913827241 >> v1:172.21.15.170:6789/0 conn(0x7f56500cda90 legacy=0x7f565006f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.616+0000 7f564f7fe640 1 -- 172.21.15.29:0/1913827241 >> v1:172.21.15.29:6789/0 conn(0x7f56500ca6f0 legacy=0x7f56500cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:17.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.617+0000 7f564f7fe640 1 -- 172.21.15.29:0/1913827241 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565006c070 con 0x7f56501599c0 2023-10-16T14:32:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.618+0000 7f564f7fe640 1 --2- 172.21.15.29:0/1913827241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56500ca1b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5640009870 tx=0x7f564002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a102df6d832c5080 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.618+0000 7f564e7fc640 1 -- 172.21.15.29:0/1913827241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640031860 con 0x7f56501599c0 2023-10-16T14:32:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.618+0000 7f564e7fc640 1 -- 172.21.15.29:0/1913827241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56400319c0 con 0x7f56501599c0 2023-10-16T14:32:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.618+0000 7f564e7fc640 1 -- 172.21.15.29:0/1913827241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640031d50 con 0x7f56501599c0 2023-10-16T14:32:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 -- 172.21.15.29:0/1913827241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 msgr2=0x7f56500ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 --2- 172.21.15.29:0/1913827241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56500ca1b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5640009870 tx=0x7f564002b900 comp rx=0 tx=0).stop 2023-10-16T14:32:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 -- 172.21.15.29:0/1913827241 shutdown_connections 2023-10-16T14:32:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 --2- 172.21.15.29:0/1913827241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56500ca1b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 -- 172.21.15.29:0/1913827241 >> 172.21.15.29:0/1913827241 conn(0x7f56500b0a90 msgr2=0x7f56500ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 -- 172.21.15.29:0/1913827241 shutdown_connections 2023-10-16T14:32:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.619+0000 7f5654fed640 1 -- 172.21.15.29:0/1913827241 wait complete. 2023-10-16T14:32:17.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.621+0000 7f5654fed640 1 Processor -- start 2023-10-16T14:32:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.621+0000 7f5654fed640 1 -- start start 2023-10-16T14:32:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f5654fed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56500ca6f0 0x7f56500b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f5654fed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56500cda90 0x7f5650149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f5654fed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56501f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f5654fed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b1300 con 0x7f56501599c0 2023-10-16T14:32:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f5654fed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b1b10 con 0x7f56500ca6f0 2023-10-16T14:32:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f5654fed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565014a1c0 con 0x7f56500cda90 2023-10-16T14:32:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f564ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56501f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f564ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56501f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54952/0 (socket says 172.21.15.29:54952) 2023-10-16T14:32:17.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f564ffff640 1 -- 172.21.15.29:0/1561986561 learned_addr learned my addr 172.21.15.29:0/1561986561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:17.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.622+0000 7f564f7fe640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56500ca6f0 0x7f56500b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564ffff640 1 -- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56500cda90 msgr2=0x7f5650149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564effd640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56500cda90 0x7f5650149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564ffff640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56500cda90 0x7f5650149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564ffff640 1 -- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56500ca6f0 msgr2=0x7f56500b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564ffff640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56500ca6f0 0x7f56500b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564ffff640 1 -- 172.21.15.29:0/1561986561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400094d0 con 0x7f56501599c0 2023-10-16T14:32:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f564ffff640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56501f29e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f564800c410 tx=0x7f564800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648007230 con 0x7f56501599c0 2023-10-16T14:32:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5648007390 con 0x7f56501599c0 2023-10-16T14:32:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56480067d0 con 0x7f56501599c0 2023-10-16T14:32:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56500b1550 con 0x7f56501599c0 2023-10-16T14:32:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.623+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565015bc00 con 0x7f56501599c0 2023-10-16T14:32:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.625+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565015be20 con 0x7f56501599c0 2023-10-16T14:32:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.626+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5648006930 con 0x7f56501599c0 2023-10-16T14:32:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.627+0000 7f5647fff640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f563406af90 0x7f563406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.627+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f5648088c60 con 0x7f56501599c0 2023-10-16T14:32:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.627+0000 7f564f7fe640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f563406af90 0x7f563406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.629+0000 7f564f7fe640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f563406af90 0x7f563406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5640036000 tx=0x7f564003c040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.631+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5648053c40 con 0x7f56501599c0 2023-10-16T14:32:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:17.896+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f565004e9b0 con 0x7f56501599c0 2023-10-16T14:32:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.472+0000 7f5647fff640 1 -- 172.21.15.29:0/1561986561 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v575) v1 ==== 159+0+0 (secure 0 0 0) 0x7f5648017020 con 0x7f56501599c0 2023-10-16T14:32:18.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.475+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f563406af90 msgr2=0x7f563406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.475+0000 7f5654fed640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f563406af90 0x7f563406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5640036000 tx=0x7f564003c040 comp rx=0 tx=0).stop 2023-10-16T14:32:18.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 msgr2=0x7f56501f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56501f29e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f564800c410 tx=0x7f564800c8d0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 shutdown_connections 2023-10-16T14:32:18.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f563406af90 0x7f563406d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56501599c0 0x7f56501f29e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56500cda90 0x7f5650149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 --2- 172.21.15.29:0/1561986561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56500ca6f0 0x7f56500b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 >> 172.21.15.29:0/1561986561 conn(0x7f56500b0a90 msgr2=0x7f56500af180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:18.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 shutdown_connections 2023-10-16T14:32:18.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.476+0000 7f5654fed640 1 -- 172.21.15.29:0/1561986561 wait complete. 2023-10-16T14:32:18.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:32:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.731+0000 7f76b2db6640 1 Processor -- start 2023-10-16T14:32:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.731+0000 7f76b2db6640 1 -- start start 2023-10-16T14:32:18.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.732+0000 7f76b2db6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76ac070830 0x7f76ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.733+0000 7f76b2db6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.733+0000 7f76b2db6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ac161740 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.733+0000 7f76b2db6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac0cdbd0 con 0x7f76ac070830 2023-10-16T14:32:18.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.733+0000 7f76b2db6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac06b0f0 con 0x7f76ac070c10 2023-10-16T14:32:18.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.733+0000 7f76b1353640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:18.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b0b52640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76ac070830 0x7f76ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:18.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76a3fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f76ac0cf5f0 0x7f76ac06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:49214/0 (socket says 172.21.15.29:49214) 2023-10-16T14:32:18.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b1353640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:49124/0 (socket says 172.21.15.29:49124) 2023-10-16T14:32:18.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b1353640 1 -- 172.21.15.29:0/1821389158 learned_addr learned my addr 172.21.15.29:0/1821389158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b0b52640 1 -- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 msgr2=0x7f76ac0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b0b52640 1 --2- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b0b52640 1 -- 172.21.15.29:0/1821389158 >> v1:172.21.15.170:6789/0 conn(0x7f76ac0cf5f0 legacy=0x7f76ac06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b0b52640 1 -- 172.21.15.29:0/1821389158 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac1619b0 con 0x7f76ac070830 2023-10-16T14:32:18.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.734+0000 7f76b1353640 1 --2- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:32:18.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.736+0000 7f76b0b52640 1 --2- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac06d3c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009d90 tx=0x7f76a802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=195f2488053282fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:18.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.736+0000 7f76a37fe640 1 -- 172.21.15.29:0/1821389158 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a802cb30 con 0x7f76ac070830 2023-10-16T14:32:18.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.736+0000 7f76a37fe640 1 -- 172.21.15.29:0/1821389158 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a802cc90 con 0x7f76ac070830 2023-10-16T14:32:18.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.736+0000 7f76a37fe640 1 -- 172.21.15.29:0/1821389158 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8034440 con 0x7f76ac070830 2023-10-16T14:32:18.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 -- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 msgr2=0x7f76ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 --2- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac06d3c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009d90 tx=0x7f76a802bd80 comp rx=0 tx=0).stop 2023-10-16T14:32:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 -- 172.21.15.29:0/1821389158 shutdown_connections 2023-10-16T14:32:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 --2- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 --2- 172.21.15.29:0/1821389158 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac06d3c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 -- 172.21.15.29:0/1821389158 >> 172.21.15.29:0/1821389158 conn(0x7f76ac0cfac0 msgr2=0x7f76ac0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:18.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 -- 172.21.15.29:0/1821389158 shutdown_connections 2023-10-16T14:32:18.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.737+0000 7f76b2db6640 1 -- 172.21.15.29:0/1821389158 wait complete. 2023-10-16T14:32:18.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.740+0000 7f76b2db6640 1 Processor -- start 2023-10-16T14:32:18.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.740+0000 7f76b2db6640 1 -- start start 2023-10-16T14:32:18.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b2db6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac15f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:18.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b2db6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:18.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b2db6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 0x7f76ac1f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:18.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b2db6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac0b3240 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b2db6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac1f3aa0 con 0x7f76ac070830 2023-10-16T14:32:18.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b2db6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac0b1c60 con 0x7f76ac070c10 2023-10-16T14:32:18.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b1353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 0x7f76ac1f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:18.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b1353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 0x7f76ac1f3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54954/0 (socket says 172.21.15.29:54954) 2023-10-16T14:32:18.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.741+0000 7f76b1353640 1 -- 172.21.15.29:0/3787159519 learned_addr learned my addr 172.21.15.29:0/3787159519 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:18.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b0b52640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:18.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76a3fff640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:18.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b1353640 1 -- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 msgr2=0x7f76ac0b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b1353640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b1353640 1 -- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 msgr2=0x7f76ac15f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b1353640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac15f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b1353640 1 -- 172.21.15.29:0/3787159519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40094d0 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b0b52640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.742+0000 7f76b1353640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 0x7f76ac1f3560 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f76a400e960 tx=0x7f76a400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.743+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4004020 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.743+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4004180 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.743+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a400ca00 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.743+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a8009a70 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.743+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac0b2220 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.745+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac04e160 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.745+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f76a400cb60 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.746+0000 7f76a17fa640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f769806af90 0x7f769806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:18.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.746+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f76a4089650 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.747+0000 7f76b0b52640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f769806af90 0x7f769806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:18.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.748+0000 7f76b0b52640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f769806af90 0x7f769806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f76a8036000 tx=0x7f76a8030000 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:18.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.751+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f76a4054640 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.946+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f76ac0c6950 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.954+0000 7f76a17fa640 1 -- 172.21.15.29:0/3787159519 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f76a4057c60 con 0x7f76ac0cf5f0 2023-10-16T14:32:18.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f769806af90 msgr2=0x7f769806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f769806af90 0x7f769806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f76a8036000 tx=0x7f76a8030000 comp rx=0 tx=0).stop 2023-10-16T14:32:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 msgr2=0x7f76ac1f3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 0x7f76ac1f3560 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f76a400e960 tx=0x7f76a400ee20 comp rx=0 tx=0).stop 2023-10-16T14:32:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 shutdown_connections 2023-10-16T14:32:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f769806af90 0x7f769806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac0cf5f0 0x7f76ac1f3560 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac070c10 0x7f76ac0b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 --2- 172.21.15.29:0/3787159519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070830 0x7f76ac15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.956+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 >> 172.21.15.29:0/3787159519 conn(0x7f76ac0cfac0 msgr2=0x7f76ac0ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.957+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 shutdown_connections 2023-10-16T14:32:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:18.957+0000 7f76b2db6640 1 -- 172.21.15.29:0/3787159519 wait complete. 2023-10-16T14:32:18.981 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:32:18.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:19.127 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:19.255 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:32:19.256 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:19.256 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:19.317 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:32:19.318 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:19.318 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:19.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:19.379 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:19.379 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:19.444 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:32:19.445 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:19.445 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:19.505 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:19.506 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:32:19.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:19.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.742+0000 7fc0a3460640 1 Processor -- start 2023-10-16T14:32:19.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.743+0000 7fc0a3460640 1 -- start start 2023-10-16T14:32:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.744+0000 7fc0a3460640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc09c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.744+0000 7fc0a3460640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c160a90 con 0x7fc09c15c9f0 2023-10-16T14:32:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.744+0000 7fc0a3460640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0cd900 con 0x7fc09c070f20 2023-10-16T14:32:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.744+0000 7fc0a3460640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0cdec0 con 0x7fc09c15dde0 2023-10-16T14:32:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.744+0000 7fc0a245e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc09c15c9f0 0x7fc09c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59320/0 (socket says 172.21.15.29:59320) 2023-10-16T14:32:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.744+0000 7fc0a245e640 1 -- 172.21.15.29:0/2290335577 learned_addr learned my addr 172.21.15.29:0/2290335577 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a1c5d640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc09c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560719291 0 0) 0x7fc09c160a90 con 0x7fc09c15c9f0 2023-10-16T14:32:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088003520 con 0x7fc09c15c9f0 2023-10-16T14:32:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 430048485 0 0) 0x7fc088003520 con 0x7fc09c15c9f0 2023-10-16T14:32:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 msgr2=0x7fc09c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc09c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 >> v1:172.21.15.170:6789/0 conn(0x7fc09c070f20 legacy=0x7fc09c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0cdbe0 con 0x7fc09c15c9f0 2023-10-16T14:32:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc098001670 con 0x7fc09c15c9f0 2023-10-16T14:32:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a1c5d640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc09c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 >> v1:172.21.15.29:6789/0 conn(0x7fc09c15c9f0 legacy=0x7fc09c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880039a0 0x7fc088003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc0880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc088004d00 0x7fc088005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800ba60 con 0x7fc0880039a0 2023-10-16T14:32:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800bbd0 con 0x7fc088004d00 2023-10-16T14:32:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.745+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800bd90 con 0x7fc09c15dde0 2023-10-16T14:32:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880039a0 0x7fc088003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a1c5d640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc0880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a245e640 1 -- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 msgr2=0x7fc0880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c15dde0 0x7fc0880047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a245e640 1 -- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc088004d00 msgr2=0x7fc088005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc088004d00 0x7fc088005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.746+0000 7fc0a245e640 1 -- 172.21.15.29:0/2290335577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088003520 con 0x7fc0880039a0 2023-10-16T14:32:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.747+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880039a0 0x7fc088003dc0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc09800c210 tx=0x7fc0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce17c25abb55e18c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098005ad0 con 0x7fc0880039a0 2023-10-16T14:32:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc098005c30 con 0x7fc0880039a0 2023-10-16T14:32:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a145c640 1 -- 172.21.15.29:0/2290335577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09802c7c0 con 0x7fc0880039a0 2023-10-16T14:32:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a3460640 1 -- 172.21.15.29:0/2290335577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880039a0 msgr2=0x7fc088003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2290335577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0880039a0 0x7fc088003dc0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc09800c210 tx=0x7fc0880057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a2c5f640 1 -- 172.21.15.29:0/2290335577 reap_dead start 2023-10-16T14:32:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a3460640 1 -- 172.21.15.29:0/2290335577 shutdown_connections 2023-10-16T14:32:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a3460640 1 -- 172.21.15.29:0/2290335577 >> 172.21.15.29:0/2290335577 conn(0x7fc09c0ce6a0 msgr2=0x7fc09c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a3460640 1 -- 172.21.15.29:0/2290335577 shutdown_connections 2023-10-16T14:32:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.748+0000 7fc0a3460640 1 -- 172.21.15.29:0/2290335577 wait complete. 2023-10-16T14:32:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.750+0000 7fc0a3460640 1 Processor -- start 2023-10-16T14:32:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.750+0000 7fc0a3460640 1 -- start start 2023-10-16T14:32:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.751+0000 7fc0a3460640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc0880039a0 0x7fc09c0adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.751+0000 7fc0a3460640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c0ae450 0x7fc09c1425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.751+0000 7fc0a3460640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 0x7fc09c1459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.751+0000 7fc0a3460640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0c3d60 con 0x7fc09c142b30 2023-10-16T14:32:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.751+0000 7fc0a3460640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0c4040 con 0x7fc0880039a0 2023-10-16T14:32:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.751+0000 7fc0a3460640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0c4320 con 0x7fc09c0ae450 2023-10-16T14:32:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 0x7fc09c1459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 0x7fc09c1459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54980/0 (socket says 172.21.15.29:54980) 2023-10-16T14:32:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 -- 172.21.15.29:0/2705297895 learned_addr learned my addr 172.21.15.29:0/2705297895 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a1c5d640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c0ae450 0x7fc09c1425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc0880039a0 0x7fc09c0adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 -- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c0ae450 msgr2=0x7fc09c1425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c0ae450 0x7fc09c1425f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 -- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc0880039a0 msgr2=0x7fc09c0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc0880039a0 0x7fc09c0adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 -- 172.21.15.29:0/2705297895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0c4600 con 0x7fc09c142b30 2023-10-16T14:32:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.752+0000 7fc0a2c5f640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 0x7fc09c1459b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0b1260 tx=0x7fc09400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.753+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094005d40 con 0x7fc09c142b30 2023-10-16T14:32:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.753+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09400ced0 con 0x7fc09c142b30 2023-10-16T14:32:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.753+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09400a850 con 0x7fc09c142b30 2023-10-16T14:32:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.753+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c0c48e0 con 0x7fc09c142b30 2023-10-16T14:32:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.753+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c145ef0 con 0x7fc09c142b30 2023-10-16T14:32:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.754+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c161670 con 0x7fc09c142b30 2023-10-16T14:32:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.755+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc09400a9b0 con 0x7fc09c142b30 2023-10-16T14:32:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.756+0000 7fc092ffd640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc08006afc0 0x7fc08006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.756+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc094088950 con 0x7fc09c142b30 2023-10-16T14:32:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.757+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc08006afc0 0x7fc08006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.758+0000 7fc0a245e640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc08006afc0 0x7fc08006d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc098034000 tx=0x7fc098009030 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.760+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc0940538a0 con 0x7fc09c142b30 2023-10-16T14:32:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.992+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc09c04e9b0 con 0x7fc09c142b30 2023-10-16T14:32:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.993+0000 7fc092ffd640 1 -- 172.21.15.29:0/2705297895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fc094017090 con 0x7fc09c142b30 2023-10-16T14:32:19.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:19.999 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1399970661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1399970661},{"type":"v1","addr":"172.21.15.170:6836","nonce":1399970661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3219176838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3219176838},{"type":"v1","addr":"172.21.15.170:6837","nonce":3219176838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2719860432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2719860432},{"type":"v1","addr":"172.21.15.29:6833","nonce":2719860432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1204771179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1204771179},{"type":"v1","addr":"172.21.15.29:6835","nonce":1204771179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[]} 2023-10-16T14:32:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2023-10-16T14:32:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc08006afc0 msgr2=0x7fc08006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc08006afc0 0x7fc08006d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc098034000 tx=0x7fc098009030 comp rx=0 tx=0).stop 2023-10-16T14:32:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 msgr2=0x7fc09c1459b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 0x7fc09c1459b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0b1260 tx=0x7fc09400cb90 comp rx=0 tx=0).stop 2023-10-16T14:32:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 shutdown_connections 2023-10-16T14:32:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fc08006afc0 0x7fc08006d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc09c142b30 0x7fc09c1459b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc09c0ae450 0x7fc09c1425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 --2- 172.21.15.29:0/2705297895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc0880039a0 0x7fc09c0adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 >> 172.21.15.29:0/2705297895 conn(0x7fc09c0ce6a0 msgr2=0x7fc09c0c1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 shutdown_connections 2023-10-16T14:32:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:19.996+0000 7fc0a3460640 1 -- 172.21.15.29:0/2705297895 wait complete. 2023-10-16T14:32:20.020 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:32:20.020 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:32:20.021 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:32:20.021 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:32:20.022 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:32:20.022 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:32:20.022 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:32:20.022 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:32:20.023 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:32:20.023 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:32:20.023 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:32:20.024 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:32:20.024 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:32:20.024 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:32:20.025 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:32:20.025 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:32:20.038 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:32:20.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:32:20.044 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:32:20.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:32:20.046 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:32:20.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:32:20.055 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:32:20.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:32:20.095 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:32:20.096 DEBUG:teuthology.parallel:result is None 2023-10-16T14:32:20.103 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:32:20.103 DEBUG:teuthology.parallel:result is None 2023-10-16T14:32:20.105 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:32:20.105 DEBUG:teuthology.parallel:result is None 2023-10-16T14:32:20.113 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:32:20.114 DEBUG:teuthology.parallel:result is None 2023-10-16T14:32:20.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:32:20.258 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:32:20.259 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:32:20.270 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:32:20.271 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:32:20.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.309+0000 7f9c65997640 1 Processor -- start 2023-10-16T14:32:20.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.309+0000 7f9c65997640 1 -- start start 2023-10-16T14:32:20.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.310+0000 7f9c65997640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9c600caac0 0x7f9c600ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:20.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.310+0000 7f9c65997640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c600cb030 con 0x7f9c6015ad70 2023-10-16T14:32:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.310+0000 7f9c65997640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c600c91c0 con 0x7f9c6006a4b0 2023-10-16T14:32:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.310+0000 7f9c65997640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c600c94a0 con 0x7f9c600caac0 2023-10-16T14:32:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9c6015ad70 0x7f9c60069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59324/0 (socket says 172.21.15.29:59324) 2023-10-16T14:32:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5effd640 1 -- 172.21.15.29:0/465669595 learned_addr learned my addr 172.21.15.29:0/465669595 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:20.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5f7fe640 1 --2- 172.21.15.29:0/465669595 >> v2:172.21.15.170:3300/0 conn(0x7f9c600caac0 0x7f9c600ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:20.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418261702 0 0) 0x7f9c600cb030 con 0x7f9c6015ad70 2023-10-16T14:32:20.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c48003520 con 0x7f9c6015ad70 2023-10-16T14:32:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714176199 0 0) 0x7f9c600c91c0 con 0x7f9c6006a4b0 2023-10-16T14:32:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c600cb030 con 0x7f9c6006a4b0 2023-10-16T14:32:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5f7fe640 1 -- 172.21.15.29:0/465669595 >> v1:172.21.15.170:6789/0 conn(0x7f9c6006a4b0 legacy=0x7f9c600ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034964497 0 0) 0x7f9c48003520 con 0x7f9c6015ad70 2023-10-16T14:32:20.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5f7fe640 1 -- 172.21.15.29:0/465669595 >> v1:172.21.15.29:6789/0 conn(0x7f9c6015ad70 legacy=0x7f9c60069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.311+0000 7f9c5f7fe640 1 -- 172.21.15.29:0/465669595 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c600c9780 con 0x7f9c600caac0 2023-10-16T14:32:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c5f7fe640 1 --2- 172.21.15.29:0/465669595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c600caac0 0x7f9c600ad4d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009d90 tx=0x7f9c5802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87e32afc06abb7c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c50001670 con 0x7f9c600caac0 2023-10-16T14:32:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5802c940 con 0x7f9c600caac0 2023-10-16T14:32:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c5dffb640 1 -- 172.21.15.29:0/465669595 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5802ccd0 con 0x7f9c600caac0 2023-10-16T14:32:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c65997640 1 -- 172.21.15.29:0/465669595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c600caac0 msgr2=0x7f9c600ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c65997640 1 --2- 172.21.15.29:0/465669595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c600caac0 0x7f9c600ad4d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009d90 tx=0x7f9c5802be10 comp rx=0 tx=0).stop 2023-10-16T14:32:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c65997640 1 -- 172.21.15.29:0/465669595 shutdown_connections 2023-10-16T14:32:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c65997640 1 --2- 172.21.15.29:0/465669595 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c600caac0 0x7f9c600ad4d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.313+0000 7f9c65997640 1 -- 172.21.15.29:0/465669595 >> 172.21.15.29:0/465669595 conn(0x7f9c600ce9b0 msgr2=0x7f9c600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.314+0000 7f9c65997640 1 -- 172.21.15.29:0/465669595 shutdown_connections 2023-10-16T14:32:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.314+0000 7f9c65997640 1 -- 172.21.15.29:0/465669595 wait complete. 2023-10-16T14:32:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.315+0000 7f9c65997640 1 Processor -- start 2023-10-16T14:32:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.315+0000 7f9c65997640 1 -- start start 2023-10-16T14:32:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c65997640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6006a4b0 0x7f9c600adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c65997640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 0x7f9c60149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c65997640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6015ad70 0x7f9c600ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c65997640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6015bce0 con 0x7f9c600caac0 2023-10-16T14:32:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c65997640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600ae190 con 0x7f9c6015ad70 2023-10-16T14:32:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c65997640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c600ae420 con 0x7f9c6006a4b0 2023-10-16T14:32:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.316+0000 7f9c5e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 0x7f9c60149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 0x7f9c60149b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55024/0 (socket says 172.21.15.29:55024) 2023-10-16T14:32:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/1263534336 learned_addr learned my addr 172.21.15.29:0/1263534336 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5effd640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6006a4b0 0x7f9c600adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5f7fe640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6015ad70 0x7f9c600ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6006a4b0 msgr2=0x7f9c600adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6006a4b0 0x7f9c600adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6015ad70 msgr2=0x7f9c600ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6015ad70 0x7f9c600ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 -- 172.21.15.29:0/1263534336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58009a70 con 0x7f9c600caac0 2023-10-16T14:32:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c5e7fc640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 0x7f9c60149b80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400c490 tx=0x7f9c5400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c540052d0 con 0x7f9c600caac0 2023-10-16T14:32:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c54005430 con 0x7f9c600caac0 2023-10-16T14:32:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54012870 con 0x7f9c600caac0 2023-10-16T14:32:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.317+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c600c5c20 con 0x7f9c600caac0 2023-10-16T14:32:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.318+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c60156e50 con 0x7f9c600caac0 2023-10-16T14:32:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.319+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6004e160 con 0x7f9c600caac0 2023-10-16T14:32:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.320+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9c540129d0 con 0x7f9c600caac0 2023-10-16T14:32:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.320+0000 7f9c47fff640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9c4c06af90 0x7f9c4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.321+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f9c54089380 con 0x7f9c600caac0 2023-10-16T14:32:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.321+0000 7f9c5effd640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9c4c06af90 0x7f9c4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.322+0000 7f9c5effd640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9c4c06af90 0x7f9c4c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9c500011e0 tx=0x7f9c50009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.325+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c540542d0 con 0x7f9c600caac0 2023-10-16T14:32:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:20.531+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9c6006fe00 con 0x7f9c600caac0 2023-10-16T14:32:21.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.990+0000 7f9c47fff640 1 -- 172.21.15.29:0/1263534336 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v80) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9c540578f0 con 0x7f9c600caac0 2023-10-16T14:32:21.994 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:32:21.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.993+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9c4c06af90 msgr2=0x7f9c4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:21.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.993+0000 7f9c65997640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9c4c06af90 0x7f9c4c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9c500011e0 tx=0x7f9c50009040 comp rx=0 tx=0).stop 2023-10-16T14:32:21.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.993+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 msgr2=0x7f9c60149b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:21.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.993+0000 7f9c65997640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 0x7f9c60149b80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400c490 tx=0x7f9c5400c950 comp rx=0 tx=0).stop 2023-10-16T14:32:21.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 shutdown_connections 2023-10-16T14:32:21.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9c4c06af90 0x7f9c4c06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:21.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c600caac0 0x7f9c60149b80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:21.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c6015ad70 0x7f9c600ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:21.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 --2- 172.21.15.29:0/1263534336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c6006a4b0 0x7f9c600adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:21.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 >> 172.21.15.29:0/1263534336 conn(0x7f9c600ce9b0 msgr2=0x7f9c6006b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:22.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 shutdown_connections 2023-10-16T14:32:22.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:21.994+0000 7f9c65997640 1 -- 172.21.15.29:0/1263534336 wait complete. 2023-10-16T14:32:22.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:32:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.235+0000 7f020e2da640 1 Processor -- start 2023-10-16T14:32:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.235+0000 7f020e2da640 1 -- start start 2023-10-16T14:32:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020e2da640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f02080ad970 0x7f02080cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020e2da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020815b9d0 con 0x7f020806c040 2023-10-16T14:32:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020e2da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020815bcb0 con 0x7f02080ccfb0 2023-10-16T14:32:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020e2da640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020815bf90 con 0x7f02080ad970 2023-10-16T14:32:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020d2d8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f020806c040 0x7f020806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59332/0 (socket says 172.21.15.29:59332) 2023-10-16T14:32:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020d2d8640 1 -- 172.21.15.29:0/4155989565 learned_addr learned my addr 172.21.15.29:0/4155989565 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.237+0000 7f020cad7640 1 --2- 172.21.15.29:0/4155989565 >> v2:172.21.15.29:3300/0 conn(0x7f02080ad970 0x7f02080cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.238+0000 7f020cad7640 1 -- 172.21.15.29:0/4155989565 >> v1:172.21.15.170:6789/0 conn(0x7f02080ccfb0 legacy=0x7f02080cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.238+0000 7f020cad7640 1 -- 172.21.15.29:0/4155989565 >> v1:172.21.15.29:6789/0 conn(0x7f020806c040 legacy=0x7f020806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.238+0000 7f020cad7640 1 -- 172.21.15.29:0/4155989565 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02080cc270 con 0x7f02080ad970 2023-10-16T14:32:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.239+0000 7f020cad7640 1 --2- 172.21.15.29:0/4155989565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080ad970 0x7f02080cca70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f01f80097d0 tx=0x7f01f802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc17dffc96a3330 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.239+0000 7f01fffff640 1 -- 172.21.15.29:0/4155989565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f802d040 con 0x7f02080ad970 2023-10-16T14:32:22.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.239+0000 7f01fffff640 1 -- 172.21.15.29:0/4155989565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f8031810 con 0x7f02080ad970 2023-10-16T14:32:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.239+0000 7f01fffff640 1 -- 172.21.15.29:0/4155989565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8031ba0 con 0x7f02080ad970 2023-10-16T14:32:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 -- 172.21.15.29:0/4155989565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080ad970 msgr2=0x7f02080cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 --2- 172.21.15.29:0/4155989565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080ad970 0x7f02080cca70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f01f80097d0 tx=0x7f01f802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 -- 172.21.15.29:0/4155989565 shutdown_connections 2023-10-16T14:32:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 --2- 172.21.15.29:0/4155989565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080ad970 0x7f02080cca70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 -- 172.21.15.29:0/4155989565 >> 172.21.15.29:0/4155989565 conn(0x7f02080ce6a0 msgr2=0x7f02080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:22.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 -- 172.21.15.29:0/4155989565 shutdown_connections 2023-10-16T14:32:22.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.240+0000 7f020e2da640 1 -- 172.21.15.29:0/4155989565 wait complete. 2023-10-16T14:32:22.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.242+0000 7f020e2da640 1 Processor -- start 2023-10-16T14:32:22.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.242+0000 7f020e2da640 1 -- start start 2023-10-16T14:32:22.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020e2da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 0x7f02081f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020e2da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02080ad970 0x7f02080b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020e2da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02080ccfb0 0x7f02080c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020e2da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080ae470 con 0x7f020806c040 2023-10-16T14:32:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020e2da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02081f28a0 con 0x7f02080ccfb0 2023-10-16T14:32:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020e2da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02080b0960 con 0x7f02080ad970 2023-10-16T14:32:22.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020d2d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 0x7f02081f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020d2d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 0x7f02081f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55040/0 (socket says 172.21.15.29:55040) 2023-10-16T14:32:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.243+0000 7f020d2d8640 1 -- 172.21.15.29:0/2172789020 learned_addr learned my addr 172.21.15.29:0/2172789020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020cad7640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02080ad970 0x7f02080b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020d2d8640 1 -- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02080ad970 msgr2=0x7f02080b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020d2d8640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02080ad970 0x7f02080b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020d2d8640 1 -- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02080ccfb0 msgr2=0x7f02080c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020dad9640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02080ccfb0 0x7f02080c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020d2d8640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02080ccfb0 0x7f02080c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020d2d8640 1 -- 172.21.15.29:0/2172789020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f8009400 con 0x7f020806c040 2023-10-16T14:32:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020d2d8640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 0x7f02081f2360 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f020400c480 tx=0x7f020400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:22.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204005ce0 con 0x7f020806c040 2023-10-16T14:32:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204005e40 con 0x7f020806c040 2023-10-16T14:32:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040072d0 con 0x7f020806c040 2023-10-16T14:32:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.244+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02080b0c40 con 0x7f020806c040 2023-10-16T14:32:22.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.245+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02080ae100 con 0x7f020806c040 2023-10-16T14:32:22.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.246+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020806b330 con 0x7f020806c040 2023-10-16T14:32:22.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.247+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f02040074f0 con 0x7f020806c040 2023-10-16T14:32:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.247+0000 7f01fdffb640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f01ec06af90 0x7f01ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.248+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02040884a0 con 0x7f020806c040 2023-10-16T14:32:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.248+0000 7f020cad7640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f01ec06af90 0x7f01ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.249+0000 7f020cad7640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f01ec06af90 0x7f01ec06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01f8036000 tx=0x7f01f803c040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.252+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0204053850 con 0x7f020806c040 2023-10-16T14:32:22.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.518+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f020804e9b0 con 0x7f020806c040 2023-10-16T14:32:22.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.519+0000 7f01fdffb640 1 -- 172.21.15.29:0/2172789020 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0204056e70 con 0x7f020806c040 2023-10-16T14:32:22.523 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:22.523 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:32:22.523 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:32:22.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.524 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:32:22.524 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:32:22.524 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.524 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.524 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:32:22.525 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:32:22.525 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.526 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:32:22.526 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:32:22.526 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.526 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.527 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:32:22.527 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:32:22.527 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.527 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.528 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:32:22.528 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:32:22.528 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.528 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.528 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:32:22.529 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:32:22.529 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.530 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:32:22.530 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:32:22.530 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.530 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.530 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:32:22.531 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:32:22.531 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.531 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.532 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:32:22.532 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:32:22.532 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.532 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.533 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.533 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.533 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.534 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:32:22.534 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:32:22.534 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.534 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.534 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.535 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.535 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.536 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:32:22.536 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:32:22.537 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.537 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.537 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.537 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.537 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.538 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:32:22.538 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:32:22.538 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.539 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.539 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.539 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.539 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:32:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:32:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.540 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.541 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:32:22.542 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:32:22.542 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.542 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.542 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.542 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.543 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.543 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:32:22.543 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:32:22.543 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.544 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.544 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.544 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.545 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:32:22.545 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:32:22.545 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.545 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:32:22.546 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:32:22.546 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.546 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.547 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:32:22.547 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:32:22.547 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.547 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.547 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:32:22.548 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:32:22.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:32:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:32:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:32:22.549 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:32:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:32:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.550 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:32:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:32:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:32:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:32:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:32:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:32:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:32:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:32:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:32:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:32:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:32:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:32:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:32:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:32:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:32:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:32:22.555 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:32:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:32:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:32:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:32:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:32:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:32:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:32:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:32:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:32:22.558 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:32:22.559 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:32:22.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:22.559 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:32:22.559 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:32:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f01ec06af90 msgr2=0x7f01ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f01ec06af90 0x7f01ec06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01f8036000 tx=0x7f01f803c040 comp rx=0 tx=0).stop 2023-10-16T14:32:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 msgr2=0x7f02081f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 0x7f02081f2360 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f020400c480 tx=0x7f020400c940 comp rx=0 tx=0).stop 2023-10-16T14:32:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 shutdown_connections 2023-10-16T14:32:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f01ec06af90 0x7f01ec06d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f02080ccfb0 0x7f02080c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02080ad970 0x7f02080b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 --2- 172.21.15.29:0/2172789020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f020806c040 0x7f02081f2360 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 >> 172.21.15.29:0/2172789020 conn(0x7f02080ce6a0 msgr2=0x7f02080c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 shutdown_connections 2023-10-16T14:32:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.522+0000 7f020e2da640 1 -- 172.21.15.29:0/2172789020 wait complete. 2023-10-16T14:32:22.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.788+0000 7fb74758c640 1 Processor -- start 2023-10-16T14:32:22.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.788+0000 7fb74758c640 1 -- start start 2023-10-16T14:32:22.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb74758c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015d1b0 0x7fb7400b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb74758c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7400d2420 0x7fb7400cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb74758c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400d2870 con 0x7fb74015cdd0 2023-10-16T14:32:22.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb74758c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7400d1380 con 0x7fb7400d2420 2023-10-16T14:32:22.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb74758c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400d1660 con 0x7fb74015d1b0 2023-10-16T14:32:22.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb746d8b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7400d2420 0x7fb7400cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb746d8b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7400d2420 0x7fb7400cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55046/0 (socket says 172.21.15.29:55046) 2023-10-16T14:32:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb74658a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb74015cdd0 0x7fb7400d0ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59344/0 (socket says 172.21.15.29:59344) 2023-10-16T14:32:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.789+0000 7fb746d8b640 1 -- 172.21.15.29:0/3899426413 learned_addr learned my addr 172.21.15.29:0/3899426413 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:22.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.790+0000 7fb745d89640 1 --2- 172.21.15.29:0/3899426413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015d1b0 0x7fb7400b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.790+0000 7fb746d8b640 1 -- 172.21.15.29:0/3899426413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015d1b0 msgr2=0x7fb7400b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:22.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.790+0000 7fb746d8b640 1 --2- 172.21.15.29:0/3899426413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015d1b0 0x7fb7400b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.790+0000 7fb746d8b640 1 -- 172.21.15.29:0/3899426413 >> v1:172.21.15.29:6789/0 conn(0x7fb74015cdd0 legacy=0x7fb7400d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.790+0000 7fb746d8b640 1 -- 172.21.15.29:0/3899426413 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400d1940 con 0x7fb7400d2420 2023-10-16T14:32:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.791+0000 7fb746d8b640 1 --2- 172.21.15.29:0/3899426413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7400d2420 0x7fb7400cd810 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb738009870 tx=0x7fb73802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faadc7f963944de4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.791+0000 7fb745588640 1 -- 172.21.15.29:0/3899426413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73802f830 con 0x7fb7400d2420 2023-10-16T14:32:22.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.791+0000 7fb745588640 1 -- 172.21.15.29:0/3899426413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73802f990 con 0x7fb7400d2420 2023-10-16T14:32:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.791+0000 7fb745588640 1 -- 172.21.15.29:0/3899426413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73802fd20 con 0x7fb7400d2420 2023-10-16T14:32:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 -- 172.21.15.29:0/3899426413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7400d2420 msgr2=0x7fb7400cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 --2- 172.21.15.29:0/3899426413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7400d2420 0x7fb7400cd810 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb738009870 tx=0x7fb73802ba10 comp rx=0 tx=0).stop 2023-10-16T14:32:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 -- 172.21.15.29:0/3899426413 shutdown_connections 2023-10-16T14:32:22.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 --2- 172.21.15.29:0/3899426413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7400d2420 0x7fb7400cd810 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 --2- 172.21.15.29:0/3899426413 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015d1b0 0x7fb7400b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 -- 172.21.15.29:0/3899426413 >> 172.21.15.29:0/3899426413 conn(0x7fb7400cf550 msgr2=0x7fb7400cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:22.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 -- 172.21.15.29:0/3899426413 shutdown_connections 2023-10-16T14:32:22.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.792+0000 7fb74758c640 1 -- 172.21.15.29:0/3899426413 wait complete. 2023-10-16T14:32:22.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.794+0000 7fb74758c640 1 Processor -- start 2023-10-16T14:32:22.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.794+0000 7fb74758c640 1 -- start start 2023-10-16T14:32:22.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.795+0000 7fb74758c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7400d2420 0x7fb74014e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.795+0000 7fb74758c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015cdd0 0x7fb7400c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.795+0000 7fb74758c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 0x7fb74015f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.795+0000 7fb74758c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400ca510 con 0x7fb74015d1b0 2023-10-16T14:32:22.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.795+0000 7fb74758c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400ca7f0 con 0x7fb7400d2420 2023-10-16T14:32:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.795+0000 7fb74758c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400caad0 con 0x7fb74015cdd0 2023-10-16T14:32:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 0x7fb74015f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 0x7fb74015f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55048/0 (socket says 172.21.15.29:55048) 2023-10-16T14:32:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 -- 172.21.15.29:0/2475721323 learned_addr learned my addr 172.21.15.29:0/2475721323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:22.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb745d89640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015cdd0 0x7fb7400c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb74658a640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7400d2420 0x7fb74014e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 -- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015cdd0 msgr2=0x7fb7400c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015cdd0 0x7fb7400c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 -- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7400d2420 msgr2=0x7fb74014e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:22.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7400d2420 0x7fb74014e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.796+0000 7fb746d8b640 1 -- 172.21.15.29:0/2475721323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380094d0 con 0x7fb74015d1b0 2023-10-16T14:32:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.797+0000 7fb746d8b640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 0x7fb74015f5d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb738009870 tx=0x7fb738004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.797+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738037c40 con 0x7fb74015d1b0 2023-10-16T14:32:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.797+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb738037dc0 con 0x7fb74015d1b0 2023-10-16T14:32:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.797+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738041640 con 0x7fb74015d1b0 2023-10-16T14:32:22.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.797+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7400cadb0 con 0x7fb74015d1b0 2023-10-16T14:32:22.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.797+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7400ca0e0 con 0x7fb74015d1b0 2023-10-16T14:32:22.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.798+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7400caff0 con 0x7fb74015d1b0 2023-10-16T14:32:22.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.799+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb7380417a0 con 0x7fb74015d1b0 2023-10-16T14:32:22.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.800+0000 7fb736ffd640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb72406af90 0x7fb72406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:22.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.800+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb73803c070 con 0x7fb74015d1b0 2023-10-16T14:32:22.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.801+0000 7fb74658a640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb72406af90 0x7fb72406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:22.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.802+0000 7fb74658a640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb72406af90 0x7fb72406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0012e0 tx=0x7fb73c001720 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:22.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:22.804+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb7380deca0 con 0x7fb74015d1b0 2023-10-16T14:32:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.003+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7400674f0 con 0x7fb74015d1b0 2023-10-16T14:32:23.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.004+0000 7fb736ffd640 1 -- 172.21.15.29:0/2475721323 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fb7400caff0 con 0x7fb74015d1b0 2023-10-16T14:32:23.008 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:23.008 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":80,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:21.987869+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:23.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb72406af90 msgr2=0x7fb72406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb72406af90 0x7fb72406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0012e0 tx=0x7fb73c001720 comp rx=0 tx=0).stop 2023-10-16T14:32:23.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 msgr2=0x7fb74015f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 0x7fb74015f5d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb738009870 tx=0x7fb738004730 comp rx=0 tx=0).stop 2023-10-16T14:32:23.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 shutdown_connections 2023-10-16T14:32:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb72406af90 0x7fb72406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb74015d1b0 0x7fb74015f5d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb74015cdd0 0x7fb7400c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 --2- 172.21.15.29:0/2475721323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7400d2420 0x7fb74014e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 >> 172.21.15.29:0/2475721323 conn(0x7fb7400cf550 msgr2=0x7fb7400b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 shutdown_connections 2023-10-16T14:32:23.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.007+0000 7fb74758c640 1 -- 172.21.15.29:0/2475721323 wait complete. 2023-10-16T14:32:23.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:23.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.265+0000 7f21db59e640 1 Processor -- start 2023-10-16T14:32:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.266+0000 7f21db59e640 1 -- start start 2023-10-16T14:32:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.267+0000 7f21db59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f21dc0b3ec0 0x7f21dc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.267+0000 7f21db59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 0x7f21dc0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.267+0000 7f21db59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc06be30 con 0x7f21dc0b43c0 2023-10-16T14:32:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.267+0000 7f21db59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc06ca00 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.267+0000 7f21db59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0ada10 con 0x7f21dc0b0ea0 2023-10-16T14:32:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.268+0000 7f21d9d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 0x7f21dc0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.268+0000 7f21d9d9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 0x7f21dc0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:49238/0 (socket says 172.21.15.29:49238) 2023-10-16T14:32:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.268+0000 7f21d9d9b640 1 -- 172.21.15.29:0/2455308824 learned_addr learned my addr 172.21.15.29:0/2455308824 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.268+0000 7f21dad9d640 1 --1- 172.21.15.29:0/2455308824 >> v1:172.21.15.170:6789/0 conn(0x7f21dc0b43c0 0x7f21dc15a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:49282/0 (socket says 172.21.15.29:49282) 2023-10-16T14:32:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.268+0000 7f21da59c640 1 --2- 172.21.15.29:0/2455308824 >> v2:172.21.15.170:3300/0 conn(0x7f21dc0b3ec0 0x7f21dc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.269+0000 7f21d9d9b640 1 -- 172.21.15.29:0/2455308824 >> v2:172.21.15.170:3300/0 conn(0x7f21dc0b3ec0 msgr2=0x7f21dc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.269+0000 7f21d9d9b640 1 --2- 172.21.15.29:0/2455308824 >> v2:172.21.15.170:3300/0 conn(0x7f21dc0b3ec0 0x7f21dc0cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.269+0000 7f21d9d9b640 1 -- 172.21.15.29:0/2455308824 >> v1:172.21.15.170:6789/0 conn(0x7f21dc0b43c0 legacy=0x7f21dc15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.269+0000 7f21d9d9b640 1 -- 172.21.15.29:0/2455308824 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc06c060 con 0x7f21dc0b0ea0 2023-10-16T14:32:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.270+0000 7f21d9d9b640 1 --2- 172.21.15.29:0/2455308824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 0x7f21dc0cd7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009870 tx=0x7f21d002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af8ff7456b42b26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21d959a640 1 -- 172.21.15.29:0/2455308824 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0031640 con 0x7f21dc0b0ea0 2023-10-16T14:32:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21d959a640 1 -- 172.21.15.29:0/2455308824 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d00317a0 con 0x7f21dc0b0ea0 2023-10-16T14:32:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21d959a640 1 -- 172.21.15.29:0/2455308824 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0031b30 con 0x7f21dc0b0ea0 2023-10-16T14:32:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 -- 172.21.15.29:0/2455308824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 msgr2=0x7f21dc0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 --2- 172.21.15.29:0/2455308824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 0x7f21dc0cd7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009870 tx=0x7f21d002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 -- 172.21.15.29:0/2455308824 shutdown_connections 2023-10-16T14:32:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 --2- 172.21.15.29:0/2455308824 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b0ea0 0x7f21dc0cd7a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 --2- 172.21.15.29:0/2455308824 >> v2:172.21.15.170:3300/0 conn(0x7f21dc0b3ec0 0x7f21dc0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 -- 172.21.15.29:0/2455308824 >> 172.21.15.29:0/2455308824 conn(0x7f21dc0b0170 msgr2=0x7f21dc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.271+0000 7f21db59e640 1 -- 172.21.15.29:0/2455308824 shutdown_connections 2023-10-16T14:32:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.272+0000 7f21db59e640 1 -- 172.21.15.29:0/2455308824 wait complete. 2023-10-16T14:32:23.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.274+0000 7f21db59e640 1 Processor -- start 2023-10-16T14:32:23.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.274+0000 7f21db59e640 1 -- start start 2023-10-16T14:32:23.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21db59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21dc0b0ea0 0x7f21dc0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21db59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 0x7f21dc15c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21db59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b43c0 0x7f21dc1e3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21db59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0c77b0 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21db59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc15d170 con 0x7f21dc0b0ea0 2023-10-16T14:32:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21db59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1e3a60 con 0x7f21dc0b43c0 2023-10-16T14:32:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.275+0000 7f21d9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 0x7f21dc15c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 0x7f21dc15c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55054/0 (socket says 172.21.15.29:55054) 2023-10-16T14:32:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 -- 172.21.15.29:0/1530615330 learned_addr learned my addr 172.21.15.29:0/1530615330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21da59c640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21dc0b0ea0 0x7f21dc0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21dad9d640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b43c0 0x7f21dc1e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 -- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b43c0 msgr2=0x7f21dc1e3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b43c0 0x7f21dc1e3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 -- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21dc0b0ea0 msgr2=0x7f21dc0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21dc0b0ea0 0x7f21dc0c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.276+0000 7f21d9d9b640 1 -- 172.21.15.29:0/1530615330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00094d0 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.277+0000 7f21d9d9b640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 0x7f21dc15c8b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f21d002b930 tx=0x7f21d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.277+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d003a930 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.277+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d003aa90 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.277+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d003ae40 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.277+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc070c40 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.278+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc071200 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.279+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc04e9b0 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.280+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f21d002f050 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.282+0000 7f21caffd640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f21bc06af90 0x7f21bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.282+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f21d00ab230 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.283+0000 7f21da59c640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f21bc06af90 0x7f21bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.284+0000 7f21da59c640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f21bc06af90 0x7f21bc06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009aa0 tx=0x7f21cc009190 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.287+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f21d0076420 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.479+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21dc06abd0 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.480+0000 7f21caffd640 1 -- 172.21.15.29:0/1530615330 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21d0079a40 con 0x7f21dc0b3ec0 2023-10-16T14:32:23.484 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:23.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.482+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f21bc06af90 msgr2=0x7f21bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.482+0000 7f21db59e640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f21bc06af90 0x7f21bc06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009aa0 tx=0x7f21cc009190 comp rx=0 tx=0).stop 2023-10-16T14:32:23.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 msgr2=0x7f21dc15c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 0x7f21dc15c8b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f21d002b930 tx=0x7f21d00046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 shutdown_connections 2023-10-16T14:32:23.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f21bc06af90 0x7f21bc06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21dc0b3ec0 0x7f21dc15c8b0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21dc0b43c0 0x7f21dc1e3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 --2- 172.21.15.29:0/1530615330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f21dc0b0ea0 0x7f21dc0c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 >> 172.21.15.29:0/1530615330 conn(0x7f21dc0b0170 msgr2=0x7f21dc0af610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:23.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 shutdown_connections 2023-10-16T14:32:23.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.483+0000 7f21db59e640 1 -- 172.21.15.29:0/1530615330 wait complete. 2023-10-16T14:32:23.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:32:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.711+0000 7f64694ec640 1 Processor -- start 2023-10-16T14:32:23.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.711+0000 7f64694ec640 1 -- start start 2023-10-16T14:32:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.712+0000 7f64694ec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6464159950 0x7f646415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.712+0000 7f64694ec640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6464070450 0x7f64640708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.712+0000 7f64694ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f646406c100 0x7f646406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.712+0000 7f64694ec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6464070e20 con 0x7f6464159950 2023-10-16T14:32:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.712+0000 7f64694ec640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646415e490 con 0x7f6464070450 2023-10-16T14:32:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.712+0000 7f64694ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646415aef0 con 0x7f646406c100 2023-10-16T14:32:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6464159950 0x7f646415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6464159950 0x7f646415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55062/0 (socket says 172.21.15.29:55062) 2023-10-16T14:32:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 -- 172.21.15.29:0/2352319115 learned_addr learned my addr 172.21.15.29:0/2352319115 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6468ceb640 1 --2- 172.21.15.29:0/2352319115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f646406c100 0x7f646406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f645bfff640 1 --2- 172.21.15.29:0/2352319115 >> v2:172.21.15.170:3300/0 conn(0x7f6464070450 0x7f64640708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 -- 172.21.15.29:0/2352319115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f646406c100 msgr2=0x7f646406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 --2- 172.21.15.29:0/2352319115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f646406c100 0x7f646406c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 -- 172.21.15.29:0/2352319115 >> v2:172.21.15.170:3300/0 conn(0x7f6464070450 msgr2=0x7f64640708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 --2- 172.21.15.29:0/2352319115 >> v2:172.21.15.170:3300/0 conn(0x7f6464070450 0x7f64640708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.713+0000 7f6463fff640 1 -- 172.21.15.29:0/2352319115 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646415d910 con 0x7f6464159950 2023-10-16T14:32:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.714+0000 7f6463fff640 1 --2- 172.21.15.29:0/2352319115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415a9b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f64640cab70 tx=0x7f645402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c1bfcd1129ba56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64637fe640 1 -- 172.21.15.29:0/2352319115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64540317c0 con 0x7f6464159950 2023-10-16T14:32:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64637fe640 1 -- 172.21.15.29:0/2352319115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454031920 con 0x7f6464159950 2023-10-16T14:32:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64637fe640 1 -- 172.21.15.29:0/2352319115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454031cb0 con 0x7f6464159950 2023-10-16T14:32:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 -- 172.21.15.29:0/2352319115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 msgr2=0x7f646415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 --2- 172.21.15.29:0/2352319115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415a9b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f64640cab70 tx=0x7f645402b860 comp rx=0 tx=0).stop 2023-10-16T14:32:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 -- 172.21.15.29:0/2352319115 shutdown_connections 2023-10-16T14:32:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 --2- 172.21.15.29:0/2352319115 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f646406c100 0x7f646406c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 --2- 172.21.15.29:0/2352319115 >> v2:172.21.15.170:3300/0 conn(0x7f6464070450 0x7f64640708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 --2- 172.21.15.29:0/2352319115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415a9b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.715+0000 7f64694ec640 1 -- 172.21.15.29:0/2352319115 >> 172.21.15.29:0/2352319115 conn(0x7f64640ce6a0 msgr2=0x7f64640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.716+0000 7f64694ec640 1 -- 172.21.15.29:0/2352319115 shutdown_connections 2023-10-16T14:32:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.716+0000 7f64694ec640 1 -- 172.21.15.29:0/2352319115 wait complete. 2023-10-16T14:32:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.718+0000 7f64694ec640 1 Processor -- start 2023-10-16T14:32:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.718+0000 7f64694ec640 1 -- start start 2023-10-16T14:32:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f64694ec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646406c100 0x7f6464151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f64694ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6464070450 0x7f646414f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f64694ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f64694ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640c5ee0 con 0x7f6464159950 2023-10-16T14:32:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f64694ec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646414f5d0 con 0x7f646406c100 2023-10-16T14:32:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f64694ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64640c5c00 con 0x7f6464070450 2023-10-16T14:32:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f6468ceb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f6468ceb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55072/0 (socket says 172.21.15.29:55072) 2023-10-16T14:32:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.719+0000 7f6468ceb640 1 -- 172.21.15.29:0/1659337276 learned_addr learned my addr 172.21.15.29:0/1659337276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f645bfff640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6464070450 0x7f646414f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6463fff640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646406c100 0x7f6464151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6468ceb640 1 -- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6464070450 msgr2=0x7f646414f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6468ceb640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6464070450 0x7f646414f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6468ceb640 1 -- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646406c100 msgr2=0x7f6464151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6468ceb640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646406c100 0x7f6464151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6468ceb640 1 -- 172.21.15.29:0/1659337276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64540094d0 con 0x7f6464159950 2023-10-16T14:32:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6463fff640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646406c100 0x7f6464151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f6468ceb640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415c4e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f645c007e60 tx=0x7f645c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c014420 con 0x7f6464159950 2023-10-16T14:32:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c0145a0 con 0x7f6464159950 2023-10-16T14:32:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.720+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c014950 con 0x7f6464159950 2023-10-16T14:32:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.721+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646415ba00 con 0x7f6464159950 2023-10-16T14:32:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.721+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646415bfc0 con 0x7f6464159950 2023-10-16T14:32:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.722+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646404e9b0 con 0x7f6464159950 2023-10-16T14:32:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.723+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f645c014ab0 con 0x7f6464159950 2023-10-16T14:32:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.723+0000 7f64617fa640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f644406af90 0x7f644406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.724+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f645c089da0 con 0x7f6464159950 2023-10-16T14:32:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.724+0000 7f6463fff640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f644406af90 0x7f644406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.725+0000 7f6463fff640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f644406af90 0x7f644406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6454036000 tx=0x7f645403d040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:23.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.728+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f645c055080 con 0x7f6464159950 2023-10-16T14:32:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.928+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f64640c8a60 con 0x7f6464159950 2023-10-16T14:32:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.929+0000 7f64617fa640 1 -- 172.21.15.29:0/1659337276 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v80) v1 ==== 84+0+65 (secure 0 0 0) 0x7f645c0586a0 con 0x7f6464159950 2023-10-16T14:32:23.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:23.933 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:32:23.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:32:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:32:23.934 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:23.934 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:32:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f644406af90 msgr2=0x7f644406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f644406af90 0x7f644406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6454036000 tx=0x7f645403d040 comp rx=0 tx=0).stop 2023-10-16T14:32:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 msgr2=0x7f646415c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415c4e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f645c007e60 tx=0x7f645c00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 shutdown_connections 2023-10-16T14:32:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f644406af90 0x7f644406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6464159950 0x7f646415c4e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6464070450 0x7f646414f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.931+0000 7f64694ec640 1 --2- 172.21.15.29:0/1659337276 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f646406c100 0x7f6464151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.932+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 >> 172.21.15.29:0/1659337276 conn(0x7f64640ce6a0 msgr2=0x7f64640c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.932+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 shutdown_connections 2023-10-16T14:32:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:23.932+0000 7f64694ec640 1 -- 172.21.15.29:0/1659337276 wait complete. 2023-10-16T14:32:23.953 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:32:23.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:32:24.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.171+0000 7f88038e1640 1 Processor -- start 2023-10-16T14:32:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.171+0000 7f88038e1640 1 -- start start 2023-10-16T14:32:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.172+0000 7f88038e1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87fc0b0360 0x7f87fc0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.172+0000 7f88038e1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f87fc0ca510 0x7f87fc0ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.172+0000 7f88038e1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc15b310 con 0x7f87fc15ac10 2023-10-16T14:32:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.172+0000 7f88038e1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc15f800 con 0x7f87fc0b0360 2023-10-16T14:32:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.172+0000 7f88038e1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc0b0d30 con 0x7f87fc0ca510 2023-10-16T14:32:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87fc0b0360 0x7f87fc0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:24.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87fc0b0360 0x7f87fc0b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56610/0 (socket says 172.21.15.29:56610) 2023-10-16T14:32:24.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 -- 172.21.15.29:0/1513442506 learned_addr learned my addr 172.21.15.29:0/1513442506 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8801e7e640 1 --2- 172.21.15.29:0/1513442506 >> v2:172.21.15.170:3300/0 conn(0x7f87fc0ca510 0x7f87fc0ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 -- 172.21.15.29:0/1513442506 >> v2:172.21.15.170:3300/0 conn(0x7f87fc0ca510 msgr2=0x7f87fc0ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 --2- 172.21.15.29:0/1513442506 >> v2:172.21.15.170:3300/0 conn(0x7f87fc0ca510 0x7f87fc0ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 -- 172.21.15.29:0/1513442506 >> v1:172.21.15.29:6789/0 conn(0x7f87fc15ac10 legacy=0x7f87fc06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8800e7c640 1 -- 172.21.15.29:0/1513442506 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc06f640 con 0x7f87fc0b0360 2023-10-16T14:32:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.173+0000 7f8801e7e640 1 --2- 172.21.15.29:0/1513442506 >> v2:172.21.15.170:3300/0 conn(0x7f87fc0ca510 0x7f87fc0ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.174+0000 7f8800e7c640 1 --2- 172.21.15.29:0/1513442506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0b0360 0x7f87fc0b07f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f87fc06d730 tx=0x7f87f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bff5ed7855a71a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.175+0000 7f87effff640 1 -- 172.21.15.29:0/1513442506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0031870 con 0x7f87fc0b0360 2023-10-16T14:32:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.175+0000 7f87effff640 1 -- 172.21.15.29:0/1513442506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f00319d0 con 0x7f87fc0b0360 2023-10-16T14:32:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.175+0000 7f87effff640 1 -- 172.21.15.29:0/1513442506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0031d60 con 0x7f87fc0b0360 2023-10-16T14:32:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 -- 172.21.15.29:0/1513442506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0b0360 msgr2=0x7f87fc0b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 --2- 172.21.15.29:0/1513442506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0b0360 0x7f87fc0b07f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f87fc06d730 tx=0x7f87f002b910 comp rx=0 tx=0).stop 2023-10-16T14:32:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 -- 172.21.15.29:0/1513442506 shutdown_connections 2023-10-16T14:32:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 --2- 172.21.15.29:0/1513442506 >> v2:172.21.15.170:3300/0 conn(0x7f87fc0ca510 0x7f87fc0ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 --2- 172.21.15.29:0/1513442506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0b0360 0x7f87fc0b07f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 -- 172.21.15.29:0/1513442506 >> 172.21.15.29:0/1513442506 conn(0x7f87fc0cb360 msgr2=0x7f87fc0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 -- 172.21.15.29:0/1513442506 shutdown_connections 2023-10-16T14:32:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.176+0000 7f88038e1640 1 -- 172.21.15.29:0/1513442506 wait complete. 2023-10-16T14:32:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.177+0000 7f88038e1640 1 Processor -- start 2023-10-16T14:32:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.178+0000 7f88038e1640 1 -- start start 2023-10-16T14:32:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f88038e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87fc0b0360 0x7f87fc0c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f88038e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 0x7f87fc0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f88038e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87fc15ac10 0x7f87fc0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f88038e1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0cd6f0 con 0x7f87fc0ca510 2023-10-16T14:32:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f88038e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b2200 con 0x7f87fc0b0360 2023-10-16T14:32:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f88038e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b2cf0 con 0x7f87fc15ac10 2023-10-16T14:32:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f8800e7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 0x7f87fc0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f8800e7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 0x7f87fc0c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56626/0 (socket says 172.21.15.29:56626) 2023-10-16T14:32:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f8800e7c640 1 -- 172.21.15.29:0/3894966875 learned_addr learned my addr 172.21.15.29:0/3894966875 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f8800e7c640 1 -- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87fc15ac10 msgr2=0x7f87fc0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f8800e7c640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87fc15ac10 0x7f87fc0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f8800e7c640 1 -- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87fc0b0360 msgr2=0x7f87fc0c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:24.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.179+0000 7f880167d640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87fc0b0360 0x7f87fc0c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:24.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f8800e7c640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87fc0b0360 0x7f87fc0c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:24.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f8800e7c640 1 -- 172.21.15.29:0/3894966875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00094d0 con 0x7f87fc0ca510 2023-10-16T14:32:24.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f8800e7c640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 0x7f87fc0c9440 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f87f002b950 tx=0x7f87f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:24.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0031860 con 0x7f87fc0ca510 2023-10-16T14:32:24.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f00443f0 con 0x7f87fc0ca510 2023-10-16T14:32:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc0c87d0 con 0x7f87fc0ca510 2023-10-16T14:32:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f00447a0 con 0x7f87fc0ca510 2023-10-16T14:32:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.180+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc0b2730 con 0x7f87fc0ca510 2023-10-16T14:32:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.181+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc0cd830 con 0x7f87fc0ca510 2023-10-16T14:32:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.182+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87f004d450 con 0x7f87fc0ca510 2023-10-16T14:32:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.183+0000 7f87edffb640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f87e006af90 0x7f87e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.183+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f87f002d040 con 0x7f87fc0ca510 2023-10-16T14:32:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.184+0000 7f880167d640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f87e006af90 0x7f87e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.185+0000 7f880167d640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f87e006af90 0x7f87e006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f87fc14f390 tx=0x7f87f8009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.187+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87fc0cd830 con 0x7f87fc0ca510 2023-10-16T14:32:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:24.383+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f87fc0cd830 con 0x7f87fc0ca510 2023-10-16T14:32:25.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.020+0000 7f87edffb640 1 -- 172.21.15.29:0/3894966875 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v81) v1 ==== 138+0+0 (secure 0 0 0) 0x7f87fc0cd830 con 0x7f87fc0ca510 2023-10-16T14:32:25.024 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:32:25.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.024+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f87e006af90 msgr2=0x7f87e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:25.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.024+0000 7f88038e1640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f87e006af90 0x7f87e006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f87fc14f390 tx=0x7f87f8009040 comp rx=0 tx=0).stop 2023-10-16T14:32:25.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 msgr2=0x7f87fc0c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:25.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 0x7f87fc0c9440 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f87f002b950 tx=0x7f87f00046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 shutdown_connections 2023-10-16T14:32:25.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f87e006af90 0x7f87e006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87fc0ca510 0x7f87fc0c9440 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87fc15ac10 0x7f87fc0c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 --2- 172.21.15.29:0/3894966875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87fc0b0360 0x7f87fc0c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 >> 172.21.15.29:0/3894966875 conn(0x7f87fc0cb360 msgr2=0x7f87fc06c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:25.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 shutdown_connections 2023-10-16T14:32:25.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.025+0000 7f88038e1640 1 -- 172.21.15.29:0/3894966875 wait complete. 2023-10-16T14:32:25.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:32:25.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.273+0000 7f9d0694e640 1 Processor -- start 2023-10-16T14:32:25.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.273+0000 7f9d0694e640 1 -- start start 2023-10-16T14:32:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.274+0000 7f9d0694e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9d00159ac0 0x7f9d0015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.274+0000 7f9d0694e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d000cc270 0x7f9d000c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0694e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0015e5c0 con 0x7f9d000704e0 2023-10-16T14:32:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0694e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0006b810 con 0x7f9d000cc270 2023-10-16T14:32:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0694e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d000cc810 con 0x7f9d00159ac0 2023-10-16T14:32:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0514b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d000cc270 0x7f9d000c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0514b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d000cc270 0x7f9d000c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56642/0 (socket says 172.21.15.29:56642) 2023-10-16T14:32:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0514b640 1 -- 172.21.15.29:0/311565521 learned_addr learned my addr 172.21.15.29:0/311565521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0594c640 1 --2- 172.21.15.29:0/311565521 >> v2:172.21.15.170:3300/0 conn(0x7f9d00159ac0 0x7f9d0015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.275+0000 7f9d0514b640 1 -- 172.21.15.29:0/311565521 >> v2:172.21.15.170:3300/0 conn(0x7f9d00159ac0 msgr2=0x7f9d0015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.276+0000 7f9d0514b640 1 --2- 172.21.15.29:0/311565521 >> v2:172.21.15.170:3300/0 conn(0x7f9d00159ac0 0x7f9d0015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.276+0000 7f9d0514b640 1 -- 172.21.15.29:0/311565521 >> v1:172.21.15.170:6789/0 conn(0x7f9d000704e0 legacy=0x7f9d0006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:25.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.276+0000 7f9d0514b640 1 -- 172.21.15.29:0/311565521 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0006b1c0 con 0x7f9d000cc270 2023-10-16T14:32:25.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.277+0000 7f9d0514b640 1 --2- 172.21.15.29:0/311565521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000c8cf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9d00071b00 tx=0x7f9cf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f33f6ee7cb493 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.277+0000 7f9d0494a640 1 -- 172.21.15.29:0/311565521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0031870 con 0x7f9d000cc270 2023-10-16T14:32:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.277+0000 7f9d0494a640 1 -- 172.21.15.29:0/311565521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf00319d0 con 0x7f9d000cc270 2023-10-16T14:32:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.277+0000 7f9d0494a640 1 -- 172.21.15.29:0/311565521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0031d60 con 0x7f9d000cc270 2023-10-16T14:32:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 -- 172.21.15.29:0/311565521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 msgr2=0x7f9d000c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 --2- 172.21.15.29:0/311565521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000c8cf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9d00071b00 tx=0x7f9cf002b910 comp rx=0 tx=0).stop 2023-10-16T14:32:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 -- 172.21.15.29:0/311565521 shutdown_connections 2023-10-16T14:32:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 --2- 172.21.15.29:0/311565521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000c8cf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 --2- 172.21.15.29:0/311565521 >> v2:172.21.15.170:3300/0 conn(0x7f9d00159ac0 0x7f9d0015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 -- 172.21.15.29:0/311565521 >> 172.21.15.29:0/311565521 conn(0x7f9d000ce6a0 msgr2=0x7f9d000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 -- 172.21.15.29:0/311565521 shutdown_connections 2023-10-16T14:32:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.278+0000 7f9d0694e640 1 -- 172.21.15.29:0/311565521 wait complete. 2023-10-16T14:32:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.280+0000 7f9d0694e640 1 Processor -- start 2023-10-16T14:32:25.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.280+0000 7f9d0694e640 1 -- start start 2023-10-16T14:32:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0694e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d000704e0 0x7f9d000b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0694e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0694e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d00159ac0 0x7f9d0014a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0694e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0014b910 con 0x7f9d000cc270 2023-10-16T14:32:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0694e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d000b1860 con 0x7f9d000704e0 2023-10-16T14:32:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0694e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d000b1b40 con 0x7f9d00159ac0 2023-10-16T14:32:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0514b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:25.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0514b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56650/0 (socket says 172.21.15.29:56650) 2023-10-16T14:32:25.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0514b640 1 -- 172.21.15.29:0/122834365 learned_addr learned my addr 172.21.15.29:0/122834365 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:25.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.281+0000 7f9d0594c640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d000704e0 0x7f9d000b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:25.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0514b640 1 -- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d00159ac0 msgr2=0x7f9d0014a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0514b640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d00159ac0 0x7f9d0014a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0514b640 1 -- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d000704e0 msgr2=0x7f9d000b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0514b640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d000704e0 0x7f9d000b26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0514b640 1 -- 172.21.15.29:0/122834365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00094d0 con 0x7f9d000cc270 2023-10-16T14:32:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0514b640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000b2c00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0000c00 tx=0x7f9cf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf003ad30 con 0x7f9d000cc270 2023-10-16T14:32:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf003ae90 con 0x7f9d000cc270 2023-10-16T14:32:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0044630 con 0x7f9d000cc270 2023-10-16T14:32:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d000b1e20 con 0x7f9d000cc270 2023-10-16T14:32:25.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.282+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d000b12c0 con 0x7f9d000cc270 2023-10-16T14:32:25.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.283+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d000668c0 con 0x7f9d000cc270 2023-10-16T14:32:25.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.285+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9cf004d450 con 0x7f9d000cc270 2023-10-16T14:32:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.286+0000 7f9cf67fc640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9ce406afc0 0x7f9ce406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.287+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9cf002d040 con 0x7f9d000cc270 2023-10-16T14:32:25.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.287+0000 7f9d0594c640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9ce406afc0 0x7f9ce406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:25.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.288+0000 7f9d0594c640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9ce406afc0 0x7f9ce406d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc006fd0 tx=0x7f9cfc008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:25.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.291+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9cf0077130 con 0x7f9d000cc270 2023-10-16T14:32:25.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:25.488+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9d0015e5c0 con 0x7f9d000cc270 2023-10-16T14:32:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.027+0000 7f9cf67fc640 1 -- 172.21.15.29:0/122834365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v82) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9d0015e5c0 con 0x7f9d000cc270 2023-10-16T14:32:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:32:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.032+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9ce406afc0 msgr2=0x7f9ce406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.032+0000 7f9d0694e640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9ce406afc0 0x7f9ce406d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc006fd0 tx=0x7f9cfc008040 comp rx=0 tx=0).stop 2023-10-16T14:32:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.032+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 msgr2=0x7f9d000b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.032+0000 7f9d0694e640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000b2c00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0000c00 tx=0x7f9cf00046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.033+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 shutdown_connections 2023-10-16T14:32:26.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.033+0000 7f9d0694e640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f9ce406afc0 0x7f9ce406d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.033+0000 7f9d0694e640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d000cc270 0x7f9d000b2c00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.033+0000 7f9d0694e640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d00159ac0 0x7f9d0014a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.033+0000 7f9d0694e640 1 --2- 172.21.15.29:0/122834365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d000704e0 0x7f9d000b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.033+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 >> 172.21.15.29:0/122834365 conn(0x7f9d000ce6a0 msgr2=0x7f9d000c9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.034+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 shutdown_connections 2023-10-16T14:32:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.034+0000 7f9d0694e640 1 -- 172.21.15.29:0/122834365 wait complete. 2023-10-16T14:32:26.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:32:26.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.264+0000 7f1825abb640 1 Processor -- start 2023-10-16T14:32:26.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.265+0000 7f1825abb640 1 -- start start 2023-10-16T14:32:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.266+0000 7f1825abb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f182006c850 0x7f18200ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.266+0000 7f1825abb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1820070360 con 0x7f1820159ab0 2023-10-16T14:32:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.266+0000 7f1825abb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18200cd1f0 con 0x7f1820159e90 2023-10-16T14:32:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.266+0000 7f1825abb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182015d890 con 0x7f182006c850 2023-10-16T14:32:26.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.266+0000 7f181f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1820159ab0 0x7f182006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44756/0 (socket says 172.21.15.29:44756) 2023-10-16T14:32:26.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.266+0000 7f181f7fe640 1 -- 172.21.15.29:0/1953947501 learned_addr learned my addr 172.21.15.29:0/1953947501 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303934641 0 0) 0x7f1820070360 con 0x7f1820159ab0 2023-10-16T14:32:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f180c003520 con 0x7f1820159ab0 2023-10-16T14:32:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181ffff640 1 --2- 172.21.15.29:0/1953947501 >> v2:172.21.15.170:3300/0 conn(0x7f182006c850 0x7f18200ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600146711 0 0) 0x7f180c003520 con 0x7f1820159ab0 2023-10-16T14:32:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 >> v2:172.21.15.170:3300/0 conn(0x7f182006c850 msgr2=0x7f18200ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 --2- 172.21.15.29:0/1953947501 >> v2:172.21.15.170:3300/0 conn(0x7f182006c850 0x7f18200ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 >> v1:172.21.15.170:6789/0 conn(0x7f1820159e90 legacy=0x7f182006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18200cd4d0 con 0x7f1820159ab0 2023-10-16T14:32:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1810001690 con 0x7f1820159ab0 2023-10-16T14:32:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 >> v1:172.21.15.29:6789/0 conn(0x7f1820159ab0 legacy=0x7f182006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f180c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f180c0047c0 0x7f180c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f180c005380 0x7f180c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182015d890 con 0x7f180c003970 2023-10-16T14:32:26.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c00c210 con 0x7f180c0047c0 2023-10-16T14:32:26.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.267+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c00c3d0 con 0x7f180c005380 2023-10-16T14:32:26.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f180c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f1817fff640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f180c0047c0 0x7f180c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181ffff640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f180c005380 0x7f180c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181f7fe640 1 -- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f180c005380 msgr2=0x7f180c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:26.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f180c005380 0x7f180c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181f7fe640 1 -- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f180c0047c0 msgr2=0x7f180c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:26.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f180c0047c0 0x7f180c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181f7fe640 1 -- 172.21.15.29:0/1953947501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c003520 con 0x7f180c003970 2023-10-16T14:32:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f1817fff640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f180c0047c0 0x7f180c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.268+0000 7f181ffff640 1 -- 172.21.15.29:0/1953947501 reap_dead start 2023-10-16T14:32:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.269+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f180c003d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f18200704f0 tx=0x7f180c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea90be68df4a8818 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810005ad0 con 0x7f180c003970 2023-10-16T14:32:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810005c30 con 0x7f180c003970 2023-10-16T14:32:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f181effd640 1 -- 172.21.15.29:0/1953947501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181002c850 con 0x7f180c003970 2023-10-16T14:32:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 -- 172.21.15.29:0/1953947501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 msgr2=0x7f180c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f180c003d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f18200704f0 tx=0x7f180c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 -- 172.21.15.29:0/1953947501 shutdown_connections 2023-10-16T14:32:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 --2- 172.21.15.29:0/1953947501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f180c003d90 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 -- 172.21.15.29:0/1953947501 >> 172.21.15.29:0/1953947501 conn(0x7f18200ce9b0 msgr2=0x7f18200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 -- 172.21.15.29:0/1953947501 shutdown_connections 2023-10-16T14:32:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.270+0000 7f1825abb640 1 -- 172.21.15.29:0/1953947501 wait complete. 2023-10-16T14:32:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.272+0000 7f1825abb640 1 Processor -- start 2023-10-16T14:32:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.272+0000 7f1825abb640 1 -- start start 2023-10-16T14:32:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.273+0000 7f1825abb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f18200c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.273+0000 7f1825abb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1820158fb0 0x7f18200af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.273+0000 7f1825abb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18200af980 0x7f182015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.273+0000 7f1825abb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18200afe00 con 0x7f180c003970 2023-10-16T14:32:26.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.273+0000 7f1825abb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201e32d0 con 0x7f18200af980 2023-10-16T14:32:26.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.273+0000 7f1825abb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18200c5350 con 0x7f1820158fb0 2023-10-16T14:32:26.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f18200c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f18200c4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56670/0 (socket says 172.21.15.29:56670) 2023-10-16T14:32:26.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 -- 172.21.15.29:0/1198038280 learned_addr learned my addr 172.21.15.29:0/1198038280 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:26.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181ffff640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18200af980 0x7f182015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f1817fff640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1820158fb0 0x7f18200af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 -- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1820158fb0 msgr2=0x7f18200af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1820158fb0 0x7f18200af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 -- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18200af980 msgr2=0x7f182015a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:26.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18200af980 0x7f182015a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:26.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 -- 172.21.15.29:0/1198038280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182015c900 con 0x7f180c003970 2023-10-16T14:32:26.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.274+0000 7f181f7fe640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f18200c4e10 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1810000f60 tx=0x7f181002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:26.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.275+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181003c040 con 0x7f180c003970 2023-10-16T14:32:26.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.275+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181002cc60 con 0x7f180c003970 2023-10-16T14:32:26.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.275+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182015bd80 con 0x7f180c003970 2023-10-16T14:32:26.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.275+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100072b0 con 0x7f180c003970 2023-10-16T14:32:26.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.275+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182015c340 con 0x7f180c003970 2023-10-16T14:32:26.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.276+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18201e3440 con 0x7f180c003970 2023-10-16T14:32:26.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.277+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1810032070 con 0x7f180c003970 2023-10-16T14:32:26.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.279+0000 7f181cff9640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f180006af90 0x7f180006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:26.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.279+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f18100aa700 con 0x7f180c003970 2023-10-16T14:32:26.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.279+0000 7f1817fff640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f180006af90 0x7f180006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:26.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.280+0000 7f1817fff640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f180006af90 0x7f180006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1808000c00 tx=0x7f1808009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:26.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.284+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f18201e3440 con 0x7f180c003970 2023-10-16T14:32:26.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:26.517+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f18201e3440 con 0x7f180c003970 2023-10-16T14:32:27.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.170+0000 7f181cff9640 1 -- 172.21.15.29:0/1198038280 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v29) v1 ==== 172+0+0 (secure 0 0 0) 0x7f18201e3440 con 0x7f180c003970 2023-10-16T14:32:27.174 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 8 and data pool 9 2023-10-16T14:32:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f180006af90 msgr2=0x7f180006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f180006af90 0x7f180006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1808000c00 tx=0x7f1808009040 comp rx=0 tx=0).stop 2023-10-16T14:32:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 msgr2=0x7f18200c4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f18200c4e10 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1810000f60 tx=0x7f181002e070 comp rx=0 tx=0).stop 2023-10-16T14:32:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 shutdown_connections 2023-10-16T14:32:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f180006af90 0x7f180006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f18200af980 0x7f182015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1820158fb0 0x7f18200af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 --2- 172.21.15.29:0/1198038280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f180c003970 0x7f18200c4e10 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.173+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 >> 172.21.15.29:0/1198038280 conn(0x7f18200ce9b0 msgr2=0x7f1820069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.174+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 shutdown_connections 2023-10-16T14:32:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.174+0000 7f1825abb640 1 -- 172.21.15.29:0/1198038280 wait complete. 2023-10-16T14:32:27.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:27.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.399+0000 7f9090ada640 1 Processor -- start 2023-10-16T14:32:27.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.400+0000 7f9090ada640 1 -- start start 2023-10-16T14:32:27.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.401+0000 7f9090ada640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f908c06c3f0 0x7f908c06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.401+0000 7f9090ada640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f908c15e0c0 0x7f908c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.401+0000 7f9090ada640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0cd630 con 0x7f908c15e540 2023-10-16T14:32:27.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.401+0000 7f9090ada640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c0ce1b0 con 0x7f908c15e0c0 2023-10-16T14:32:27.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.401+0000 7f9090ada640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c06cd10 con 0x7f908c06c3f0 2023-10-16T14:32:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f908c15e0c0 0x7f908c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f908c15e0c0 0x7f908c0b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56676/0 (socket says 172.21.15.29:56676) 2023-10-16T14:32:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 -- 172.21.15.29:0/798729799 learned_addr learned my addr 172.21.15.29:0/798729799 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:27.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f908a59c640 1 --2- 172.21.15.29:0/798729799 >> v2:172.21.15.170:3300/0 conn(0x7f908c06c3f0 0x7f908c06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 -- 172.21.15.29:0/798729799 >> v2:172.21.15.170:3300/0 conn(0x7f908c06c3f0 msgr2=0x7f908c06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 --2- 172.21.15.29:0/798729799 >> v2:172.21.15.170:3300/0 conn(0x7f908c06c3f0 0x7f908c06c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 -- 172.21.15.29:0/798729799 >> v1:172.21.15.170:6789/0 conn(0x7f908c15e540 legacy=0x7f908c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f9083fff640 1 -- 172.21.15.29:0/798729799 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0d2170 con 0x7f908c15e0c0 2023-10-16T14:32:27.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.402+0000 7f908a59c640 1 --2- 172.21.15.29:0/798729799 >> v2:172.21.15.170:3300/0 conn(0x7f908c06c3f0 0x7f908c06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:27.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.403+0000 7f9083fff640 1 --2- 172.21.15.29:0/798729799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e0c0 0x7f908c0b4730 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f908c0d3d00 tx=0x7f907402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249f940abee2d366 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:27.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.404+0000 7f9089d9b640 1 -- 172.21.15.29:0/798729799 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074031870 con 0x7f908c15e0c0 2023-10-16T14:32:27.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.404+0000 7f9089d9b640 1 -- 172.21.15.29:0/798729799 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90740319d0 con 0x7f908c15e0c0 2023-10-16T14:32:27.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.404+0000 7f9089d9b640 1 -- 172.21.15.29:0/798729799 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074031d60 con 0x7f908c15e0c0 2023-10-16T14:32:27.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.404+0000 7f9090ada640 1 -- 172.21.15.29:0/798729799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e0c0 msgr2=0x7f908c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.404+0000 7f9090ada640 1 --2- 172.21.15.29:0/798729799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e0c0 0x7f908c0b4730 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f908c0d3d00 tx=0x7f907402b910 comp rx=0 tx=0).stop 2023-10-16T14:32:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.405+0000 7f9090ada640 1 -- 172.21.15.29:0/798729799 shutdown_connections 2023-10-16T14:32:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.405+0000 7f9090ada640 1 --2- 172.21.15.29:0/798729799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e0c0 0x7f908c0b4730 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.405+0000 7f9090ada640 1 --2- 172.21.15.29:0/798729799 >> v2:172.21.15.170:3300/0 conn(0x7f908c06c3f0 0x7f908c06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.405+0000 7f9090ada640 1 -- 172.21.15.29:0/798729799 >> 172.21.15.29:0/798729799 conn(0x7f908c0d2cb0 msgr2=0x7f908c0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.405+0000 7f9090ada640 1 -- 172.21.15.29:0/798729799 shutdown_connections 2023-10-16T14:32:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.405+0000 7f9090ada640 1 -- 172.21.15.29:0/798729799 wait complete. 2023-10-16T14:32:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.406+0000 7f9090ada640 1 Processor -- start 2023-10-16T14:32:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.407+0000 7f9090ada640 1 -- start start 2023-10-16T14:32:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f9090ada640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f908c06c3f0 0x7f908c1f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f9090ada640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f908c15e0c0 0x7f908c159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f9090ada640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 0x7f908c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f9090ada640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1f2cc0 con 0x7f908c15e540 2023-10-16T14:32:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f9090ada640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1f5b00 con 0x7f908c15e0c0 2023-10-16T14:32:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f9090ada640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0cace0 con 0x7f908c06c3f0 2023-10-16T14:32:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f908ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 0x7f908c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f908ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 0x7f908c0cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56686/0 (socket says 172.21.15.29:56686) 2023-10-16T14:32:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f908ad9d640 1 -- 172.21.15.29:0/1211429695 learned_addr learned my addr 172.21.15.29:0/1211429695 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.408+0000 7f908a59c640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f908c06c3f0 0x7f908c1f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f908ad9d640 1 -- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f908c06c3f0 msgr2=0x7f908c1f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f908ad9d640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f908c06c3f0 0x7f908c1f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f908ad9d640 1 -- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f908c15e0c0 msgr2=0x7f908c159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f908ad9d640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f908c15e0c0 0x7f908c159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f908ad9d640 1 -- 172.21.15.29:0/1211429695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740094d0 con 0x7f908c15e540 2023-10-16T14:32:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f908ad9d640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 0x7f908c0cb170 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f908400c7b0 tx=0x7f908400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084004b60 con 0x7f908c15e540 2023-10-16T14:32:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084004cc0 con 0x7f908c15e540 2023-10-16T14:32:27.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c0ca0d0 con 0x7f908c15e540 2023-10-16T14:32:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908400aa90 con 0x7f908c15e540 2023-10-16T14:32:27.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.409+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c0ca690 con 0x7f908c15e540 2023-10-16T14:32:27.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.411+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c0cae70 con 0x7f908c15e540 2023-10-16T14:32:27.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.412+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f908400abf0 con 0x7f908c15e540 2023-10-16T14:32:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.413+0000 7f9082ffd640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f906c06af60 0x7f906c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.413+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9084089940 con 0x7f908c15e540 2023-10-16T14:32:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.414+0000 7f908a59c640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f906c06af60 0x7f906c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.415+0000 7f908a59c640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f906c06af60 0x7f906c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f907c00a950 tx=0x7f907c008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:27.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.417+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f90840548d0 con 0x7f908c15e540 2023-10-16T14:32:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.611+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f908c04e160 con 0x7f908c15e540 2023-10-16T14:32:27.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.614+0000 7f9082ffd640 1 -- 172.21.15.29:0/1211429695 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f9084057ef0 con 0x7f908c15e540 2023-10-16T14:32:27.620 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:27.620 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:26.520228+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f906c06af60 msgr2=0x7f906c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f906c06af60 0x7f906c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f907c00a950 tx=0x7f907c008040 comp rx=0 tx=0).stop 2023-10-16T14:32:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 msgr2=0x7f908c0cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 0x7f908c0cb170 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f908400c7b0 tx=0x7f908400cc70 comp rx=0 tx=0).stop 2023-10-16T14:32:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 shutdown_connections 2023-10-16T14:32:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f906c06af60 0x7f906c06d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f908c15e540 0x7f908c0cb170 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f908c15e0c0 0x7f908c159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 --2- 172.21.15.29:0/1211429695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f908c06c3f0 0x7f908c1f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 >> 172.21.15.29:0/1211429695 conn(0x7f908c0d2cb0 msgr2=0x7f908c15e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:27.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 shutdown_connections 2023-10-16T14:32:27.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.617+0000 7f9090ada640 1 -- 172.21.15.29:0/1211429695 wait complete. 2023-10-16T14:32:27.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:27.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.855+0000 7f03398a3640 1 Processor -- start 2023-10-16T14:32:27.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.856+0000 7f03398a3640 1 -- start start 2023-10-16T14:32:27.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.857+0000 7f03398a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03340ad970 0x7f03340cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.857+0000 7f03398a3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f03340ccfb0 0x7f03340cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.857+0000 7f03398a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033415b9d0 con 0x7f033406c040 2023-10-16T14:32:27.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.857+0000 7f03398a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033415bcb0 con 0x7f03340ad970 2023-10-16T14:32:27.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.857+0000 7f03398a3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033415bf90 con 0x7f03340ccfb0 2023-10-16T14:32:27.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.857+0000 7f03388a1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f033406c040 0x7f033406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44772/0 (socket says 172.21.15.29:44772) 2023-10-16T14:32:27.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.858+0000 7f03388a1640 1 -- 172.21.15.29:0/415765201 learned_addr learned my addr 172.21.15.29:0/415765201 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:27.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.858+0000 7f03390a2640 1 --2- 172.21.15.29:0/415765201 >> v2:172.21.15.170:3300/0 conn(0x7f03340ccfb0 0x7f03340cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.858+0000 7f0333fff640 1 --2- 172.21.15.29:0/415765201 >> v2:172.21.15.29:3300/0 conn(0x7f03340ad970 0x7f03340cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.858+0000 7f03337fe640 1 -- 172.21.15.29:0/415765201 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885506759 0 0) 0x7f033415b9d0 con 0x7f033406c040 2023-10-16T14:32:27.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.858+0000 7f03337fe640 1 -- 172.21.15.29:0/415765201 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c003520 con 0x7f033406c040 2023-10-16T14:32:27.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.859+0000 7f03390a2640 1 -- 172.21.15.29:0/415765201 >> v2:172.21.15.29:3300/0 conn(0x7f03340ad970 msgr2=0x7f03340cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:27.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.859+0000 7f03390a2640 1 --2- 172.21.15.29:0/415765201 >> v2:172.21.15.29:3300/0 conn(0x7f03340ad970 0x7f03340cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.859+0000 7f03390a2640 1 -- 172.21.15.29:0/415765201 >> v1:172.21.15.29:6789/0 conn(0x7f033406c040 legacy=0x7f033406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.859+0000 7f03390a2640 1 -- 172.21.15.29:0/415765201 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03340cc270 con 0x7f03340ccfb0 2023-10-16T14:32:27.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.860+0000 7f03390a2640 1 --2- 172.21.15.29:0/415765201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03340ccfb0 0x7f03340cd460 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f032c0097a0 tx=0x7f032c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42ec32a936346b18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:27.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.860+0000 7f03337fe640 1 -- 172.21.15.29:0/415765201 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c031830 con 0x7f03340ccfb0 2023-10-16T14:32:27.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.860+0000 7f03337fe640 1 -- 172.21.15.29:0/415765201 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c031990 con 0x7f03340ccfb0 2023-10-16T14:32:27.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.860+0000 7f03337fe640 1 -- 172.21.15.29:0/415765201 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c031d00 con 0x7f03340ccfb0 2023-10-16T14:32:27.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 -- 172.21.15.29:0/415765201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03340ccfb0 msgr2=0x7f03340cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 --2- 172.21.15.29:0/415765201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03340ccfb0 0x7f03340cd460 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f032c0097a0 tx=0x7f032c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 -- 172.21.15.29:0/415765201 shutdown_connections 2023-10-16T14:32:27.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 --2- 172.21.15.29:0/415765201 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03340ccfb0 0x7f03340cd460 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 --2- 172.21.15.29:0/415765201 >> v2:172.21.15.29:3300/0 conn(0x7f03340ad970 0x7f03340cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 -- 172.21.15.29:0/415765201 >> 172.21.15.29:0/415765201 conn(0x7f03340ce6a0 msgr2=0x7f03340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:27.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 -- 172.21.15.29:0/415765201 shutdown_connections 2023-10-16T14:32:27.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.861+0000 7f03398a3640 1 -- 172.21.15.29:0/415765201 wait complete. 2023-10-16T14:32:27.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.864+0000 7f03398a3640 1 Processor -- start 2023-10-16T14:32:27.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.864+0000 7f03398a3640 1 -- start start 2023-10-16T14:32:27.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.865+0000 7f03398a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f033406c040 0x7f03341558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.865+0000 7f03398a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 0x7f03340b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.865+0000 7f03398a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03340ccfb0 0x7f03340ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.865+0000 7f03398a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c9190 con 0x7f03340ad970 2023-10-16T14:32:27.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.865+0000 7f03398a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0334157430 con 0x7f033406c040 2023-10-16T14:32:27.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.865+0000 7f03398a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0334156e70 con 0x7f03340ccfb0 2023-10-16T14:32:27.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 0x7f03340b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 0x7f03340b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56700/0 (socket says 172.21.15.29:56700) 2023-10-16T14:32:27.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 -- 172.21.15.29:0/317563681 learned_addr learned my addr 172.21.15.29:0/317563681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:27.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f03388a1640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f033406c040 0x7f03341558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f03390a2640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03340ccfb0 0x7f03340ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 -- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03340ccfb0 msgr2=0x7f03340ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03340ccfb0 0x7f03340ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 -- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f033406c040 msgr2=0x7f03341558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:27.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f033406c040 0x7f03341558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:27.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.866+0000 7f0333fff640 1 -- 172.21.15.29:0/317563681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c009400 con 0x7f03340ad970 2023-10-16T14:32:27.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.867+0000 7f0333fff640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 0x7f03340b3d70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f032800e9b0 tx=0x7f032800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:27.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.867+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328009670 con 0x7f03340ad970 2023-10-16T14:32:27.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.867+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032800c9a0 con 0x7f03340ad970 2023-10-16T14:32:27.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.867+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0334157150 con 0x7f03340ad970 2023-10-16T14:32:27.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.867+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032800cd30 con 0x7f03340ad970 2023-10-16T14:32:27.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.867+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0334155e30 con 0x7f03340ad970 2023-10-16T14:32:27.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.868+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033406b330 con 0x7f03340ad970 2023-10-16T14:32:27.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.869+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0328013470 con 0x7f03340ad970 2023-10-16T14:32:27.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.870+0000 7f03317fa640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f032006af90 0x7f032006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:27.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.870+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03280899b0 con 0x7f03340ad970 2023-10-16T14:32:27.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.871+0000 7f03388a1640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f032006af90 0x7f032006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:27.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.872+0000 7f03388a1640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f032006af90 0x7f032006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0324005c00 tx=0x7f0324009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:27.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:27.875+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0328054940 con 0x7f03340ad970 2023-10-16T14:32:28.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.069+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f033404e9b0 con 0x7f03340ad970 2023-10-16T14:32:28.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.072+0000 7f03317fa640 1 -- 172.21.15.29:0/317563681 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f0328057f60 con 0x7f03340ad970 2023-10-16T14:32:28.077 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:28.077 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:28.052105+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.075+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f032006af90 msgr2=0x7f032006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.075+0000 7f03398a3640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f032006af90 0x7f032006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0324005c00 tx=0x7f0324009040 comp rx=0 tx=0).stop 2023-10-16T14:32:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.075+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 msgr2=0x7f03340b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.075+0000 7f03398a3640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 0x7f03340b3d70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f032800e9b0 tx=0x7f032800ee70 comp rx=0 tx=0).stop 2023-10-16T14:32:28.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 shutdown_connections 2023-10-16T14:32:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f032006af90 0x7f032006d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03340ad970 0x7f03340b3d70 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03340ccfb0 0x7f03340ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 --2- 172.21.15.29:0/317563681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f033406c040 0x7f03341558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 >> 172.21.15.29:0/317563681 conn(0x7f03340ce6a0 msgr2=0x7f03340c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 shutdown_connections 2023-10-16T14:32:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.076+0000 7f03398a3640 1 -- 172.21.15.29:0/317563681 wait complete. 2023-10-16T14:32:28.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:32:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.321+0000 7f04b821a640 1 Processor -- start 2023-10-16T14:32:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.321+0000 7f04b821a640 1 -- start start 2023-10-16T14:32:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b821a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b015ad70 0x7f04b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b821a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04b00c9ca0 0x7f04b006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b821a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b00cb030 con 0x7f04b00ca3b0 2023-10-16T14:32:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b821a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b00c91c0 con 0x7f04b00c9ca0 2023-10-16T14:32:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b821a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00c94a0 con 0x7f04b015ad70 2023-10-16T14:32:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b67b7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f04b00ca3b0 0x7f04b00ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44788/0 (socket says 172.21.15.29:44788) 2023-10-16T14:32:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.323+0000 7f04b67b7640 1 -- 172.21.15.29:0/3130091592 learned_addr learned my addr 172.21.15.29:0/3130091592 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b57b5640 1 --2- 172.21.15.29:0/3130091592 >> v2:172.21.15.29:3300/0 conn(0x7f04b00c9ca0 0x7f04b006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b5fb6640 1 --2- 172.21.15.29:0/3130091592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b015ad70 0x7f04b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b4fb4640 1 -- 172.21.15.29:0/3130091592 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172712088 0 0) 0x7f04b00cb030 con 0x7f04b00ca3b0 2023-10-16T14:32:28.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b4fb4640 1 -- 172.21.15.29:0/3130091592 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c003520 con 0x7f04b00ca3b0 2023-10-16T14:32:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b57b5640 1 -- 172.21.15.29:0/3130091592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b015ad70 msgr2=0x7f04b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b57b5640 1 --2- 172.21.15.29:0/3130091592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b015ad70 0x7f04b0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b57b5640 1 -- 172.21.15.29:0/3130091592 >> v1:172.21.15.29:6789/0 conn(0x7f04b00ca3b0 legacy=0x7f04b00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:28.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.324+0000 7f04b57b5640 1 -- 172.21.15.29:0/3130091592 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b00c9780 con 0x7f04b00c9ca0 2023-10-16T14:32:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.325+0000 7f04b57b5640 1 --2- 172.21.15.29:0/3130091592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b00c9ca0 0x7f04b006a2e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f04a00097a0 tx=0x7f04a002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f1ff2b25887024 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b4fb4640 1 -- 172.21.15.29:0/3130091592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a002d040 con 0x7f04b00c9ca0 2023-10-16T14:32:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b4fb4640 1 -- 172.21.15.29:0/3130091592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0031810 con 0x7f04b00c9ca0 2023-10-16T14:32:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b4fb4640 1 -- 172.21.15.29:0/3130091592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0031ba0 con 0x7f04b00c9ca0 2023-10-16T14:32:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b821a640 1 -- 172.21.15.29:0/3130091592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b00c9ca0 msgr2=0x7f04b006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b821a640 1 --2- 172.21.15.29:0/3130091592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b00c9ca0 0x7f04b006a2e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f04a00097a0 tx=0x7f04a002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b821a640 1 -- 172.21.15.29:0/3130091592 shutdown_connections 2023-10-16T14:32:28.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b821a640 1 --2- 172.21.15.29:0/3130091592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b00c9ca0 0x7f04b006a2e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b821a640 1 --2- 172.21.15.29:0/3130091592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b015ad70 0x7f04b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.326+0000 7f04b821a640 1 -- 172.21.15.29:0/3130091592 >> 172.21.15.29:0/3130091592 conn(0x7f04b00ce9b0 msgr2=0x7f04b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.327+0000 7f04b821a640 1 -- 172.21.15.29:0/3130091592 shutdown_connections 2023-10-16T14:32:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.327+0000 7f04b821a640 1 -- 172.21.15.29:0/3130091592 wait complete. 2023-10-16T14:32:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.328+0000 7f04b821a640 1 Processor -- start 2023-10-16T14:32:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.329+0000 7f04b821a640 1 -- start start 2023-10-16T14:32:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.329+0000 7f04b821a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04b00c9ca0 0x7f04b00c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b821a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b00ca3b0 0x7f04b0146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b821a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 0x7f04b0144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b821a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0146be0 con 0x7f04b015ad70 2023-10-16T14:32:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b821a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0146ec0 con 0x7f04b00c9ca0 2023-10-16T14:32:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b821a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00b1650 con 0x7f04b00ca3b0 2023-10-16T14:32:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 0x7f04b0144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 0x7f04b0144100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56712/0 (socket says 172.21.15.29:56712) 2023-10-16T14:32:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 -- 172.21.15.29:0/2846253694 learned_addr learned my addr 172.21.15.29:0/2846253694 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b57b5640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b00ca3b0 0x7f04b0146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b5fb6640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04b00c9ca0 0x7f04b00c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 -- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b00ca3b0 msgr2=0x7f04b0146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b00ca3b0 0x7f04b0146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 -- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04b00c9ca0 msgr2=0x7f04b00c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04b00c9ca0 0x7f04b00c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b67b7640 1 -- 172.21.15.29:0/2846253694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a0009400 con 0x7f04b015ad70 2023-10-16T14:32:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b57b5640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b00ca3b0 0x7f04b0146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.330+0000 7f04b5fb6640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04b00c9ca0 0x7f04b00c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.331+0000 7f04b67b7640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 0x7f04b0144100 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f04a800cb40 tx=0x7f04a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.331+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a800aca0 con 0x7f04b015ad70 2023-10-16T14:32:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.331+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a800ae00 con 0x7f04b015ad70 2023-10-16T14:32:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.331+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8012620 con 0x7f04b015ad70 2023-10-16T14:32:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.331+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b00b1930 con 0x7f04b015ad70 2023-10-16T14:32:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.331+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0151880 con 0x7f04b015ad70 2023-10-16T14:32:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.333+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b00b1b70 con 0x7f04b015ad70 2023-10-16T14:32:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.334+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f04a8007070 con 0x7f04b015ad70 2023-10-16T14:32:28.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.335+0000 7f04a67fc640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f049406af90 0x7f049406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:28.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.335+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04a8089880 con 0x7f04b015ad70 2023-10-16T14:32:28.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.336+0000 7f04b5fb6640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f049406af90 0x7f049406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:28.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.337+0000 7f04b5fb6640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f049406af90 0x7f049406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009a20 tx=0x7f04ac008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:28.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.339+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f04b00b1b70 con 0x7f04b015ad70 2023-10-16T14:32:28.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:28.574+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f04b00b1b70 con 0x7f04b015ad70 2023-10-16T14:32:29.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.183+0000 7f04a67fc640 1 -- 172.21.15.29:0/2846253694 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f04b00b1b70 con 0x7f04b015ad70 2023-10-16T14:32:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.186+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f049406af90 msgr2=0x7f049406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.186+0000 7f04b821a640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f049406af90 0x7f049406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009a20 tx=0x7f04ac008040 comp rx=0 tx=0).stop 2023-10-16T14:32:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.186+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 msgr2=0x7f04b0144100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.186+0000 7f04b821a640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 0x7f04b0144100 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f04a800cb40 tx=0x7f04a800a060 comp rx=0 tx=0).stop 2023-10-16T14:32:29.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 shutdown_connections 2023-10-16T14:32:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f049406af90 0x7f049406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b015ad70 0x7f04b0144100 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04b00ca3b0 0x7f04b0146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 --2- 172.21.15.29:0/2846253694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04b00c9ca0 0x7f04b00c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 >> 172.21.15.29:0/2846253694 conn(0x7f04b00ce9b0 msgr2=0x7f04b015b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 shutdown_connections 2023-10-16T14:32:29.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.187+0000 7f04b821a640 1 -- 172.21.15.29:0/2846253694 wait complete. 2023-10-16T14:32:29.211 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:32:29.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:32:29.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.443+0000 7f9266867640 1 Processor -- start 2023-10-16T14:32:29.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.444+0000 7f9266867640 1 -- start start 2023-10-16T14:32:29.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.445+0000 7f9266867640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f926015ac10 0x7f92600ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:29.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.445+0000 7f9266867640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cecc0 0x7f92600cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:29.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.445+0000 7f9266867640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92600cf4a0 con 0x7f92600cc570 2023-10-16T14:32:29.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.445+0000 7f9266867640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92600cb550 con 0x7f926015ac10 2023-10-16T14:32:29.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.445+0000 7f9266867640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92600cb830 con 0x7f92600cecc0 2023-10-16T14:32:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f926015ac10 0x7f92600ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f926015ac10 0x7f92600ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56714/0 (socket says 172.21.15.29:56714) 2023-10-16T14:32:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 -- 172.21.15.29:0/4148480227 learned_addr learned my addr 172.21.15.29:0/4148480227 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265064640 1 --2- 172.21.15.29:0/4148480227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cecc0 0x7f92600cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 -- 172.21.15.29:0/4148480227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cecc0 msgr2=0x7f92600cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:29.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 --2- 172.21.15.29:0/4148480227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cecc0 0x7f92600cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 -- 172.21.15.29:0/4148480227 >> v1:172.21.15.170:6789/0 conn(0x7f92600cc570 legacy=0x7f92600cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:29.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.446+0000 7f9265865640 1 -- 172.21.15.29:0/4148480227 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92600cbb10 con 0x7f926015ac10 2023-10-16T14:32:29.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.448+0000 7f9265865640 1 --2- 172.21.15.29:0/4148480227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f926015ac10 0x7f92600ce6b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f925c009870 tx=0x7f925c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b14f5528dd1f01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:29.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.448+0000 7f9264863640 1 -- 172.21.15.29:0/4148480227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c02f840 con 0x7f926015ac10 2023-10-16T14:32:29.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.448+0000 7f9264863640 1 -- 172.21.15.29:0/4148480227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c02f9a0 con 0x7f926015ac10 2023-10-16T14:32:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.448+0000 7f9264863640 1 -- 172.21.15.29:0/4148480227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c02fd30 con 0x7f926015ac10 2023-10-16T14:32:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 -- 172.21.15.29:0/4148480227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f926015ac10 msgr2=0x7f92600ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 --2- 172.21.15.29:0/4148480227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f926015ac10 0x7f92600ce6b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f925c009870 tx=0x7f925c02ba20 comp rx=0 tx=0).stop 2023-10-16T14:32:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 -- 172.21.15.29:0/4148480227 shutdown_connections 2023-10-16T14:32:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 --2- 172.21.15.29:0/4148480227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cecc0 0x7f92600cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 --2- 172.21.15.29:0/4148480227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f926015ac10 0x7f92600ce6b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 -- 172.21.15.29:0/4148480227 >> 172.21.15.29:0/4148480227 conn(0x7f9260099a10 msgr2=0x7f92600d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 -- 172.21.15.29:0/4148480227 shutdown_connections 2023-10-16T14:32:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.449+0000 7f9266867640 1 -- 172.21.15.29:0/4148480227 wait complete. 2023-10-16T14:32:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.452+0000 7f9266867640 1 Processor -- start 2023-10-16T14:32:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.452+0000 7f9266867640 1 -- start start 2023-10-16T14:32:29.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9266867640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cc570 0x7f9260156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:29.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9266867640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 0x7f92600c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:29.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9266867640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f926015ac10 0x7f92600c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:29.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9266867640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92600b3720 con 0x7f92600cecc0 2023-10-16T14:32:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9266867640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9260156680 con 0x7f926015ac10 2023-10-16T14:32:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9266867640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92601f4700 con 0x7f92600cc570 2023-10-16T14:32:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9265064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 0x7f92600c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9265064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 0x7f92600c8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56716/0 (socket says 172.21.15.29:56716) 2023-10-16T14:32:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.453+0000 7f9265064640 1 -- 172.21.15.29:0/542931811 learned_addr learned my addr 172.21.15.29:0/542931811 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265865640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cc570 0x7f9260156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9266066640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f926015ac10 0x7f92600c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265064640 1 -- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cc570 msgr2=0x7f9260156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265064640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cc570 0x7f9260156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265064640 1 -- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f926015ac10 msgr2=0x7f92600c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265064640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f926015ac10 0x7f92600c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:29.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265064640 1 -- 172.21.15.29:0/542931811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c0094d0 con 0x7f92600cecc0 2023-10-16T14:32:29.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.454+0000 7f9265064640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 0x7f92600c8780 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f925400d800 tx=0x7f925400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:29.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.455+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925400b6c0 con 0x7f92600cecc0 2023-10-16T14:32:29.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.455+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925400b820 con 0x7f92600cecc0 2023-10-16T14:32:29.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.455+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925400bbb0 con 0x7f92600cecc0 2023-10-16T14:32:29.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.455+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92600c7cc0 con 0x7f92600cecc0 2023-10-16T14:32:29.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.455+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92600c8280 con 0x7f92600cecc0 2023-10-16T14:32:29.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.456+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92600c8440 con 0x7f92600cecc0 2023-10-16T14:32:29.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.457+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f925400bd10 con 0x7f92600cecc0 2023-10-16T14:32:29.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.458+0000 7f92527fc640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f924406af90 0x7f924406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:29.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.458+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9254089dd0 con 0x7f92600cecc0 2023-10-16T14:32:29.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.459+0000 7f9265865640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f924406af90 0x7f924406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:29.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.460+0000 7f9265865640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f924406af90 0x7f924406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f925c036000 tx=0x7f925c008030 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:29.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.463+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9254054cd0 con 0x7f92600cecc0 2023-10-16T14:32:29.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:29.695+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f926004e9b0 con 0x7f92600cecc0 2023-10-16T14:32:30.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.192+0000 7f92527fc640 1 -- 172.21.15.29:0/542931811 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7f92540582f0 con 0x7f92600cecc0 2023-10-16T14:32:30.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f924406af90 msgr2=0x7f924406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f924406af90 0x7f924406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f925c036000 tx=0x7f925c008030 comp rx=0 tx=0).stop 2023-10-16T14:32:30.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 msgr2=0x7f92600c8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 0x7f92600c8780 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f925400d800 tx=0x7f925400dcc0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 shutdown_connections 2023-10-16T14:32:30.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f924406af90 0x7f924406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92600cecc0 0x7f92600c8780 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f926015ac10 0x7f92600c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 --2- 172.21.15.29:0/542931811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f92600cc570 0x7f9260156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.196+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 >> 172.21.15.29:0/542931811 conn(0x7f9260099a10 msgr2=0x7f92600c4440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.197+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 shutdown_connections 2023-10-16T14:32:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.197+0000 7f9266867640 1 -- 172.21.15.29:0/542931811 wait complete. 2023-10-16T14:32:30.220 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:32:30.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:30.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.447+0000 7ff457fff640 1 Processor -- start 2023-10-16T14:32:30.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.447+0000 7ff457fff640 1 -- start start 2023-10-16T14:32:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff457fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff45806c040 0x7ff45806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff457fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff45815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff457fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45806c960 con 0x7ff4580ad970 2023-10-16T14:32:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff457fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45815b9d0 con 0x7ff45806c040 2023-10-16T14:32:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff457fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45815bcb0 con 0x7ff4580cda80 2023-10-16T14:32:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff44ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff4580ad970 0x7ff45815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44800/0 (socket says 172.21.15.29:44800) 2023-10-16T14:32:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.449+0000 7ff44ffff640 1 -- 172.21.15.29:0/2114188805 learned_addr learned my addr 172.21.15.29:0/2114188805 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff456ffd640 1 --2- 172.21.15.29:0/2114188805 >> v2:172.21.15.170:3300/0 conn(0x7ff45806c040 0x7ff45806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068638383 0 0) 0x7ff45806c960 con 0x7ff4580ad970 2023-10-16T14:32:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff444003520 con 0x7ff4580ad970 2023-10-16T14:32:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4577fe640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff45815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260385166 0 0) 0x7ff444003520 con 0x7ff4580ad970 2023-10-16T14:32:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 msgr2=0x7ff45815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff45815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 >> v2:172.21.15.170:3300/0 conn(0x7ff45806c040 msgr2=0x7ff45806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 --2- 172.21.15.29:0/2114188805 >> v2:172.21.15.170:3300/0 conn(0x7ff45806c040 0x7ff45806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45815bf90 con 0x7ff4580ad970 2023-10-16T14:32:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7ff440001670 con 0x7ff4580ad970 2023-10-16T14:32:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 >> v1:172.21.15.29:6789/0 conn(0x7ff4580ad970 legacy=0x7ff45815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff444003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff4440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff444004d00 0x7ff444005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45815b9d0 con 0x7ff444003970 2023-10-16T14:32:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.450+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45815bcb0 con 0x7ff444004d00 2023-10-16T14:32:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44400b9d0 con 0x7ff4580cda80 2023-10-16T14:32:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff456ffd640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff444003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff44ffff640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff4440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff4577fe640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff444004d00 0x7ff444005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff456ffd640 1 -- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 msgr2=0x7ff4440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff456ffd640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff4440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff456ffd640 1 -- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff444004d00 msgr2=0x7ff444005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff456ffd640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff444004d00 0x7ff444005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff456ffd640 1 -- 172.21.15.29:0/2114188805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff444003520 con 0x7ff444003970 2023-10-16T14:32:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff4577fe640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff444004d00 0x7ff444005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.451+0000 7ff44ffff640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4580cda80 0x7ff4440047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.452+0000 7ff456ffd640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff444003d90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff448009ac0 tx=0x7ff4440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c190b19cfeba056c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44802c710 con 0x7ff444003970 2023-10-16T14:32:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44802c870 con 0x7ff444003970 2023-10-16T14:32:30.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff4567fc640 1 -- 172.21.15.29:0/2114188805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44802cc00 con 0x7ff444003970 2023-10-16T14:32:30.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff457fff640 1 -- 172.21.15.29:0/2114188805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 msgr2=0x7ff444003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff457fff640 1 --2- 172.21.15.29:0/2114188805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff444003d90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff448009ac0 tx=0x7ff4440057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff4577fe640 1 -- 172.21.15.29:0/2114188805 reap_dead start 2023-10-16T14:32:30.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff457fff640 1 -- 172.21.15.29:0/2114188805 shutdown_connections 2023-10-16T14:32:30.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.453+0000 7ff457fff640 1 -- 172.21.15.29:0/2114188805 >> 172.21.15.29:0/2114188805 conn(0x7ff4580ce6a0 msgr2=0x7ff4580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.454+0000 7ff457fff640 1 -- 172.21.15.29:0/2114188805 shutdown_connections 2023-10-16T14:32:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.454+0000 7ff457fff640 1 -- 172.21.15.29:0/2114188805 wait complete. 2023-10-16T14:32:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.456+0000 7ff457fff640 1 Processor -- start 2023-10-16T14:32:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.456+0000 7ff457fff640 1 -- start start 2023-10-16T14:32:30.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff457fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff4581f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff457fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff45814d760 0x7ff45814bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff457fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff45814c100 0x7ff45814af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff457fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45814c6c0 con 0x7ff444003970 2023-10-16T14:32:30.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff457fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45814c9a0 con 0x7ff45814c100 2023-10-16T14:32:30.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff457fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45814cc80 con 0x7ff45814d760 2023-10-16T14:32:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff4581f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff4581f5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56720/0 (socket says 172.21.15.29:56720) 2023-10-16T14:32:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 -- 172.21.15.29:0/34524627 learned_addr learned my addr 172.21.15.29:0/34524627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 -- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff45814d760 msgr2=0x7ff45814bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff45814d760 0x7ff45814bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 -- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff45814c100 msgr2=0x7ff45814af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff4577fe640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff45814c100 0x7ff45814af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.457+0000 7ff456ffd640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff45814c100 0x7ff45814af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff456ffd640 1 -- 172.21.15.29:0/34524627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45814cf60 con 0x7ff444003970 2023-10-16T14:32:30.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff456ffd640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff4581f5dd0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff44802c2b0 tx=0x7ff448004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448003810 con 0x7ff444003970 2023-10-16T14:32:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44802de90 con 0x7ff444003970 2023-10-16T14:32:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448003b70 con 0x7ff444003970 2023-10-16T14:32:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45814d240 con 0x7ff444003970 2023-10-16T14:32:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.458+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4581f6310 con 0x7ff444003970 2023-10-16T14:32:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.459+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4581f6500 con 0x7ff444003970 2023-10-16T14:32:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.461+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff448003cd0 con 0x7ff444003970 2023-10-16T14:32:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.462+0000 7ff44f7fe640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff43806af90 0x7ff43806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.462+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4480ab660 con 0x7ff444003970 2023-10-16T14:32:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.462+0000 7ff44ffff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff43806af90 0x7ff43806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.464+0000 7ff44ffff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff43806af90 0x7ff43806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4400017a0 tx=0x7ff440009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:30.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.466+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff448076600 con 0x7ff444003970 2023-10-16T14:32:30.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.697+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4580cded0 con 0x7ff444003970 2023-10-16T14:32:30.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.699+0000 7ff44f7fe640 1 -- 172.21.15.29:0/34524627 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4916 (secure 0 0 0) 0x7ff448079c20 con 0x7ff444003970 2023-10-16T14:32:30.703 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:30.703 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:32:26.521447+0000","modified":"2023-10-16T14:32:29.697489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3822716445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3822716445},{"type":"v1","addr":"172.21.15.170:6837","nonce":3822716445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T14:32:30.703 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2023-10-16T14:32:30.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff43806af90 msgr2=0x7ff43806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff43806af90 0x7ff43806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4400017a0 tx=0x7ff440009040 comp rx=0 tx=0).stop 2023-10-16T14:32:30.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 msgr2=0x7ff4581f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff4581f5dd0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff44802c2b0 tx=0x7ff448004730 comp rx=0 tx=0).stop 2023-10-16T14:32:30.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 shutdown_connections 2023-10-16T14:32:30.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7ff43806af90 0x7ff43806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff45814c100 0x7ff45814af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff45814d760 0x7ff45814bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 --2- 172.21.15.29:0/34524627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff444003970 0x7ff4581f5dd0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 >> 172.21.15.29:0/34524627 conn(0x7ff4580ce6a0 msgr2=0x7ff4580b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 shutdown_connections 2023-10-16T14:32:30.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.701+0000 7ff457fff640 1 -- 172.21.15.29:0/34524627 wait complete. 2023-10-16T14:32:30.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:30.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.945+0000 7fb51359e640 1 Processor -- start 2023-10-16T14:32:30.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.946+0000 7fb51359e640 1 -- start start 2023-10-16T14:32:30.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.947+0000 7fb51359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb51406c040 0x7fb51406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.947+0000 7fb51359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51415f4a0 con 0x7fb5140ad970 2023-10-16T14:32:30.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.947+0000 7fb51359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51406c960 con 0x7fb5140cda80 2023-10-16T14:32:30.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.947+0000 7fb51359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51415b9d0 con 0x7fb51406c040 2023-10-16T14:32:30.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.948+0000 7fb50bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb5140ad970 0x7fb51415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44814/0 (socket says 172.21.15.29:44814) 2023-10-16T14:32:30.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.948+0000 7fb50bfff640 1 -- 172.21.15.29:0/2439708998 learned_addr learned my addr 172.21.15.29:0/2439708998 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:30.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.948+0000 7fb51259c640 1 --2- 172.21.15.29:0/2439708998 >> v2:172.21.15.170:3300/0 conn(0x7fb51406c040 0x7fb51406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.948+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693464307 0 0) 0x7fb51415f4a0 con 0x7fb5140ad970 2023-10-16T14:32:30.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.948+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb500003520 con 0x7fb5140ad970 2023-10-16T14:32:30.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3439552010 0 0) 0x7fb500003520 con 0x7fb5140ad970 2023-10-16T14:32:30.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 >> v2:172.21.15.170:3300/0 conn(0x7fb51406c040 msgr2=0x7fb51406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 --2- 172.21.15.29:0/2439708998 >> v2:172.21.15.170:3300/0 conn(0x7fb51406c040 0x7fb51406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 >> v1:172.21.15.170:6789/0 conn(0x7fb5140cda80 legacy=0x7fb51415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51415bcb0 con 0x7fb5140ad970 2023-10-16T14:32:30.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fb4fc001670 con 0x7fb5140ad970 2023-10-16T14:32:30.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 >> v1:172.21.15.29:6789/0 conn(0x7fb5140ad970 legacy=0x7fb51415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5000039c0 0x7fb500003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5000047c0 0x7fb500004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb500005180 0x7fb50000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51415b9d0 con 0x7fb500005180 2023-10-16T14:32:30.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50000c080 con 0x7fb5000047c0 2023-10-16T14:32:30.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50000c240 con 0x7fb5000039c0 2023-10-16T14:32:30.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.949+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb500005180 0x7fb50000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb50bfff640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5000047c0 0x7fb500004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb51259c640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5000039c0 0x7fb500003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb512d9d640 1 -- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5000039c0 msgr2=0x7fb500003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5000039c0 0x7fb500003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb512d9d640 1 -- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5000047c0 msgr2=0x7fb500004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:30.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5000047c0 0x7fb500004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb512d9d640 1 -- 172.21.15.29:0/2439708998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb500003520 con 0x7fb500005180 2023-10-16T14:32:30.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.950+0000 7fb51259c640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5000039c0 0x7fb500003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:30.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.951+0000 7fb512d9d640 1 -- 172.21.15.29:0/2439708998 reap_dead start 2023-10-16T14:32:30.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.951+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb500005180 0x7fb50000b980 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb50c001e50 tx=0x7fb5000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd32f84045f5bc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:30.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.951+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c003b70 con 0x7fb500005180 2023-10-16T14:32:30.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.951+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c003cf0 con 0x7fb500005180 2023-10-16T14:32:30.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.951+0000 7fb511d9b640 1 -- 172.21.15.29:0/2439708998 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c005310 con 0x7fb500005180 2023-10-16T14:32:30.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 -- 172.21.15.29:0/2439708998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb500005180 msgr2=0x7fb50000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb500005180 0x7fb50000b980 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb50c001e50 tx=0x7fb5000057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 -- 172.21.15.29:0/2439708998 shutdown_connections 2023-10-16T14:32:30.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 --2- 172.21.15.29:0/2439708998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb500005180 0x7fb50000b980 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 -- 172.21.15.29:0/2439708998 >> 172.21.15.29:0/2439708998 conn(0x7fb5140ce6a0 msgr2=0x7fb5140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:30.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 -- 172.21.15.29:0/2439708998 shutdown_connections 2023-10-16T14:32:30.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.952+0000 7fb51359e640 1 -- 172.21.15.29:0/2439708998 wait complete. 2023-10-16T14:32:30.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.954+0000 7fb51359e640 1 Processor -- start 2023-10-16T14:32:30.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.954+0000 7fb51359e640 1 -- start start 2023-10-16T14:32:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb51359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb500005180 0x7fb514151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb51359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb514159220 0x7fb5140c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb51359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 0x7fb5140c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb51359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141528e0 con 0x7fb514159680 2023-10-16T14:32:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb51359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb514153860 con 0x7fb500005180 2023-10-16T14:32:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb51359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb514153b40 con 0x7fb514159220 2023-10-16T14:32:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb512d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 0x7fb5140c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb512d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 0x7fb5140c7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56750/0 (socket says 172.21.15.29:56750) 2023-10-16T14:32:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.955+0000 7fb512d9d640 1 -- 172.21.15.29:0/2114629375 learned_addr learned my addr 172.21.15.29:0/2114629375 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb51259c640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb500005180 0x7fb514151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb50bfff640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb514159220 0x7fb5140c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb512d9d640 1 -- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb514159220 msgr2=0x7fb5140c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb514159220 0x7fb5140c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb512d9d640 1 -- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb500005180 msgr2=0x7fb514151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb500005180 0x7fb514151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb512d9d640 1 -- 172.21.15.29:0/2114629375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb514153e20 con 0x7fb514159680 2023-10-16T14:32:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.956+0000 7fb512d9d640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 0x7fb5140c7210 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb50c001f70 tx=0x7fb50c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.957+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c006d50 con 0x7fb514159680 2023-10-16T14:32:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.957+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c003b80 con 0x7fb514159680 2023-10-16T14:32:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.957+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c0396c0 con 0x7fb514159680 2023-10-16T14:32:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.957+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb514154100 con 0x7fb514159680 2023-10-16T14:32:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.957+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb514151dd0 con 0x7fb514159680 2023-10-16T14:32:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.958+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51406b330 con 0x7fb514159680 2023-10-16T14:32:30.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.959+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb50c039820 con 0x7fb514159680 2023-10-16T14:32:30.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.960+0000 7fb50affd640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb4f406afc0 0x7fb4f406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:30.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.960+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb50c0ab5a0 con 0x7fb514159680 2023-10-16T14:32:30.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.961+0000 7fb51259c640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb4f406afc0 0x7fb4f406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:30.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.962+0000 7fb51259c640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb4f406afc0 0x7fb4f406d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb504009aa0 tx=0x7fb504008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:30.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:30.964+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb50c076420 con 0x7fb514159680 2023-10-16T14:32:31.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.157+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5141578c0 con 0x7fb514159680 2023-10-16T14:32:31.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.160+0000 7fb50affd640 1 -- 172.21.15.29:0/2114629375 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16565 (secure 0 0 0) 0x7fb5141578c0 con 0x7fb514159680 2023-10-16T14:32:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:31.164 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:28.052105+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.162+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb4f406afc0 msgr2=0x7fb4f406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.162+0000 7fb51359e640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb4f406afc0 0x7fb4f406d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb504009aa0 tx=0x7fb504008040 comp rx=0 tx=0).stop 2023-10-16T14:32:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 msgr2=0x7fb5140c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 0x7fb5140c7210 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb50c001f70 tx=0x7fb50c0014a0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 shutdown_connections 2023-10-16T14:32:31.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fb4f406afc0 0x7fb4f406d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514159680 0x7fb5140c7210 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb514159220 0x7fb5140c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 --2- 172.21.15.29:0/2114629375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb500005180 0x7fb514151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 >> 172.21.15.29:0/2114629375 conn(0x7fb5140ce6a0 msgr2=0x7fb514069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:31.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 shutdown_connections 2023-10-16T14:32:31.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.163+0000 7fb51359e640 1 -- 172.21.15.29:0/2114629375 wait complete. 2023-10-16T14:32:31.185 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:32:31.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:32:31.190 INFO:tasks.ceph.mds.b.smithi170.stderr:2023-10-16T14:32:31.186+0000 7ff759a64640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:32:31.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.398+0000 7fe490c60640 1 Processor -- start 2023-10-16T14:32:31.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.398+0000 7fe490c60640 1 -- start start 2023-10-16T14:32:31.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.399+0000 7fe490c60640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c15cd70 0x7fe48c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.399+0000 7fe490c60640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe48c0b8480 0x7fe48c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.399+0000 7fe490c60640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe48c0d0be0 0x7fe48c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.399+0000 7fe490c60640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c15d150 con 0x7fe48c0d0be0 2023-10-16T14:32:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.399+0000 7fe490c60640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0d1180 con 0x7fe48c0b8480 2023-10-16T14:32:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.399+0000 7fe490c60640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0d1460 con 0x7fe48c15cd70 2023-10-16T14:32:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe48c0d0be0 0x7fe48c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe48c0d0be0 0x7fe48c0cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56760/0 (socket says 172.21.15.29:56760) 2023-10-16T14:32:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 -- 172.21.15.29:0/1919728910 learned_addr learned my addr 172.21.15.29:0/1919728910 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48affd640 1 --2- 172.21.15.29:0/1919728910 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0b8480 0x7fe48c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48b7fe640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c15cd70 0x7fe48c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 -- 172.21.15.29:0/1919728910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c15cd70 msgr2=0x7fe48c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c15cd70 0x7fe48c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 -- 172.21.15.29:0/1919728910 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0b8480 msgr2=0x7fe48c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 --2- 172.21.15.29:0/1919728910 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0b8480 0x7fe48c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48bfff640 1 -- 172.21.15.29:0/1919728910 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0d1740 con 0x7fe48c0d0be0 2023-10-16T14:32:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.400+0000 7fe48b7fe640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c15cd70 0x7fe48c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.401+0000 7fe48bfff640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0d0be0 0x7fe48c0cdd00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe48c066ab0 tx=0x7fe48402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cd0050f66c3169 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.402+0000 7fe48a7fc640 1 -- 172.21.15.29:0/1919728910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484031870 con 0x7fe48c0d0be0 2023-10-16T14:32:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.402+0000 7fe48a7fc640 1 -- 172.21.15.29:0/1919728910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4840319d0 con 0x7fe48c0d0be0 2023-10-16T14:32:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.402+0000 7fe48a7fc640 1 -- 172.21.15.29:0/1919728910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484031d60 con 0x7fe48c0d0be0 2023-10-16T14:32:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.402+0000 7fe490c60640 1 -- 172.21.15.29:0/1919728910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0d0be0 msgr2=0x7fe48c0cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.402+0000 7fe490c60640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0d0be0 0x7fe48c0cdd00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe48c066ab0 tx=0x7fe48402b910 comp rx=0 tx=0).stop 2023-10-16T14:32:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 -- 172.21.15.29:0/1919728910 shutdown_connections 2023-10-16T14:32:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0d0be0 0x7fe48c0cdd00 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 --2- 172.21.15.29:0/1919728910 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0b8480 0x7fe48c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 --2- 172.21.15.29:0/1919728910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c15cd70 0x7fe48c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 -- 172.21.15.29:0/1919728910 >> 172.21.15.29:0/1919728910 conn(0x7fe48c0cf570 msgr2=0x7fe48c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 -- 172.21.15.29:0/1919728910 shutdown_connections 2023-10-16T14:32:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.403+0000 7fe490c60640 1 -- 172.21.15.29:0/1919728910 wait complete. 2023-10-16T14:32:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.405+0000 7fe490c60640 1 Processor -- start 2023-10-16T14:32:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.405+0000 7fe490c60640 1 -- start start 2023-10-16T14:32:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe490c60640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe48c0b8480 0x7fe48c1f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe490c60640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c0d0be0 0x7fe48c1f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe490c60640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 0x7fe48c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe490c60640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c15eb20 con 0x7fe48c15cd70 2023-10-16T14:32:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe490c60640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c15e840 con 0x7fe48c0b8480 2023-10-16T14:32:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe490c60640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c7b60 con 0x7fe48c0d0be0 2023-10-16T14:32:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 0x7fe48c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 0x7fe48c15aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56774/0 (socket says 172.21.15.29:56774) 2023-10-16T14:32:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 -- 172.21.15.29:0/1789528957 learned_addr learned my addr 172.21.15.29:0/1789528957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48affd640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c0d0be0 0x7fe48c1f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 -- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c0d0be0 msgr2=0x7fe48c1f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c0d0be0 0x7fe48c1f5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 -- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe48c0b8480 msgr2=0x7fe48c1f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe48c0b8480 0x7fe48c1f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.406+0000 7fe48bfff640 1 -- 172.21.15.29:0/1789528957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840094d0 con 0x7fe48c15cd70 2023-10-16T14:32:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.407+0000 7fe48bfff640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 0x7fe48c15aa20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe48402b950 tx=0x7fe4840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.407+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48403ac60 con 0x7fe48c15cd70 2023-10-16T14:32:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.407+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48403ade0 con 0x7fe48c15cd70 2023-10-16T14:32:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.407+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484044630 con 0x7fe48c15cd70 2023-10-16T14:32:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.407+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0c7e40 con 0x7fe48c15cd70 2023-10-16T14:32:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.407+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c1f55b0 con 0x7fe48c15cd70 2023-10-16T14:32:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.408+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0c79c0 con 0x7fe48c15cd70 2023-10-16T14:32:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.409+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe484044790 con 0x7fe48c15cd70 2023-10-16T14:32:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.410+0000 7fe483fff640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.410+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe48402d040 con 0x7fe48c15cd70 2023-10-16T14:32:31.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.411+0000 7fe48b7fe640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.412+0000 7fe48b7fe640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe46c06af90 0x7fe46c06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007130 tx=0x7fe47c007300 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:31.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.415+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe48c0c79c0 con 0x7fe48c15cd70 2023-10-16T14:32:31.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.604+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 --> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe48c15ffe0 con 0x7fe46c06af90 2023-10-16T14:32:31.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.633+0000 7fe483fff640 1 -- 172.21.15.29:0/1789528957 <== mgr.5135 v2:172.21.15.170:6832/885795747 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286353 (secure 0 0 0) 0x7fe48c15ffe0 con 0x7fe46c06af90 2023-10-16T14:32:31.639 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:32:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe46c06af90 msgr2=0x7fe46c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe46c06af90 0x7fe46c06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007130 tx=0x7fe47c007300 comp rx=0 tx=0).stop 2023-10-16T14:32:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 msgr2=0x7fe48c15aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 0x7fe48c15aa20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe48402b950 tx=0x7fe4840046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 shutdown_connections 2023-10-16T14:32:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c15cd70 0x7fe48c15aa20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c0d0be0 0x7fe48c1f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 --2- 172.21.15.29:0/1789528957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe48c0b8480 0x7fe48c1f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 >> 172.21.15.29:0/1789528957 conn(0x7fe48c0cf570 msgr2=0x7fe48c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 shutdown_connections 2023-10-16T14:32:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.640+0000 7fe490c60640 1 -- 172.21.15.29:0/1789528957 wait complete. 2023-10-16T14:32:31.663 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T14:32:31.384062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336228,"kb_used_data":9064,"kb_used_omap":12,"kb_used_meta":327027,"kb_avail":754638492,"statfs":{"total":773094113280,"available":772749815808,"internally_reserved":0,"allocated":9281536,"data_stored":5046372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":334876217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002193"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060662+0000","last_change":"2023-10-16T14:32:27.096141+0000","last_active":"2023-10-16T14:32:28.060662+0000","last_peered":"2023-10-16T14:32:28.060662+0000","last_clean":"2023-10-16T14:32:28.060662+0000","last_became_active":"2023-10-16T14:32:27.095834+0000","last_became_peered":"2023-10-16T14:32:27.095834+0000","last_unstale":"2023-10-16T14:32:28.060662+0000","last_undegraded":"2023-10-16T14:32:28.060662+0000","last_fullsized":"2023-10-16T14:32:28.060662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:35.658898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.089893+0000","last_change":"2023-10-16T14:32:27.081367+0000","last_active":"2023-10-16T14:32:27.089893+0000","last_peered":"2023-10-16T14:32:27.089893+0000","last_clean":"2023-10-16T14:32:27.089893+0000","last_became_active":"2023-10-16T14:32:27.081081+0000","last_became_peered":"2023-10-16T14:32:27.081081+0000","last_unstale":"2023-10-16T14:32:27.089893+0000","last_undegraded":"2023-10-16T14:32:27.089893+0000","last_fullsized":"2023-10-16T14:32:27.089893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:45.554454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065378+0000","last_change":"2023-10-16T14:32:27.074983+0000","last_active":"2023-10-16T14:32:28.065378+0000","last_peered":"2023-10-16T14:32:28.065378+0000","last_clean":"2023-10-16T14:32:28.065378+0000","last_became_active":"2023-10-16T14:32:27.074448+0000","last_became_peered":"2023-10-16T14:32:27.074448+0000","last_unstale":"2023-10-16T14:32:28.065378+0000","last_undegraded":"2023-10-16T14:32:28.065378+0000","last_fullsized":"2023-10-16T14:32:28.065378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:07.700052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.097732+0000","last_change":"2023-10-16T14:32:27.097251+0000","last_active":"2023-10-16T14:32:27.097732+0000","last_peered":"2023-10-16T14:32:27.097732+0000","last_clean":"2023-10-16T14:32:27.097732+0000","last_became_active":"2023-10-16T14:32:27.096940+0000","last_became_peered":"2023-10-16T14:32:27.096940+0000","last_unstale":"2023-10-16T14:32:27.097732+0000","last_undegraded":"2023-10-16T14:32:27.097732+0000","last_fullsized":"2023-10-16T14:32:27.097732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:14.799841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060456+0000","last_change":"2023-10-16T14:32:27.096782+0000","last_active":"2023-10-16T14:32:28.060456+0000","last_peered":"2023-10-16T14:32:28.060456+0000","last_clean":"2023-10-16T14:32:28.060456+0000","last_became_active":"2023-10-16T14:32:27.092020+0000","last_became_peered":"2023-10-16T14:32:27.092020+0000","last_unstale":"2023-10-16T14:32:28.060456+0000","last_undegraded":"2023-10-16T14:32:28.060456+0000","last_fullsized":"2023-10-16T14:32:28.060456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:51.811557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096608+0000","last_change":"2023-10-16T14:32:27.093796+0000","last_active":"2023-10-16T14:32:27.096608+0000","last_peered":"2023-10-16T14:32:27.096608+0000","last_clean":"2023-10-16T14:32:27.096608+0000","last_became_active":"2023-10-16T14:32:27.093480+0000","last_became_peered":"2023-10-16T14:32:27.093480+0000","last_unstale":"2023-10-16T14:32:27.096608+0000","last_undegraded":"2023-10-16T14:32:27.096608+0000","last_fullsized":"2023-10-16T14:32:27.096608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:12.029035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062407+0000","last_change":"2023-10-16T14:32:27.091530+0000","last_active":"2023-10-16T14:32:28.062407+0000","last_peered":"2023-10-16T14:32:28.062407+0000","last_clean":"2023-10-16T14:32:28.062407+0000","last_became_active":"2023-10-16T14:32:27.090082+0000","last_became_peered":"2023-10-16T14:32:27.090082+0000","last_unstale":"2023-10-16T14:32:28.062407+0000","last_undegraded":"2023-10-16T14:32:28.062407+0000","last_fullsized":"2023-10-16T14:32:28.062407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:20.317503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.077897+0000","last_change":"2023-10-16T14:32:27.075711+0000","last_active":"2023-10-16T14:32:27.077897+0000","last_peered":"2023-10-16T14:32:27.077897+0000","last_clean":"2023-10-16T14:32:27.077897+0000","last_became_active":"2023-10-16T14:32:27.075295+0000","last_became_peered":"2023-10-16T14:32:27.075295+0000","last_unstale":"2023-10-16T14:32:27.077897+0000","last_undegraded":"2023-10-16T14:32:27.077897+0000","last_fullsized":"2023-10-16T14:32:27.077897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:47.249405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064444+0000","last_change":"2023-10-16T14:32:27.086675+0000","last_active":"2023-10-16T14:32:28.064444+0000","last_peered":"2023-10-16T14:32:28.064444+0000","last_clean":"2023-10-16T14:32:28.064444+0000","last_became_active":"2023-10-16T14:32:27.086405+0000","last_became_peered":"2023-10-16T14:32:27.086405+0000","last_unstale":"2023-10-16T14:32:28.064444+0000","last_undegraded":"2023-10-16T14:32:28.064444+0000","last_fullsized":"2023-10-16T14:32:28.064444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:13.491565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.098033+0000","last_change":"2023-10-16T14:32:27.095614+0000","last_active":"2023-10-16T14:32:27.098033+0000","last_peered":"2023-10-16T14:32:27.098033+0000","last_clean":"2023-10-16T14:32:27.098033+0000","last_became_active":"2023-10-16T14:32:27.093064+0000","last_became_peered":"2023-10-16T14:32:27.093064+0000","last_unstale":"2023-10-16T14:32:27.098033+0000","last_undegraded":"2023-10-16T14:32:27.098033+0000","last_fullsized":"2023-10-16T14:32:27.098033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:35.658898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096891+0000","last_change":"2023-10-16T14:32:27.086798+0000","last_active":"2023-10-16T14:32:27.096891+0000","last_peered":"2023-10-16T14:32:27.096891+0000","last_clean":"2023-10-16T14:32:27.096891+0000","last_became_active":"2023-10-16T14:32:27.086464+0000","last_became_peered":"2023-10-16T14:32:27.086464+0000","last_unstale":"2023-10-16T14:32:27.096891+0000","last_undegraded":"2023-10-16T14:32:27.096891+0000","last_fullsized":"2023-10-16T14:32:27.096891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:00.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092328+0000","last_change":"2023-10-16T14:32:27.087692+0000","last_active":"2023-10-16T14:32:27.092328+0000","last_peered":"2023-10-16T14:32:27.092328+0000","last_clean":"2023-10-16T14:32:27.092328+0000","last_became_active":"2023-10-16T14:32:27.086123+0000","last_became_peered":"2023-10-16T14:32:27.086123+0000","last_unstale":"2023-10-16T14:32:27.092328+0000","last_undegraded":"2023-10-16T14:32:27.092328+0000","last_fullsized":"2023-10-16T14:32:27.092328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:51.303159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091297+0000","last_change":"2023-10-16T14:32:27.089453+0000","last_active":"2023-10-16T14:32:27.091297+0000","last_peered":"2023-10-16T14:32:27.091297+0000","last_clean":"2023-10-16T14:32:27.091297+0000","last_became_active":"2023-10-16T14:32:27.088596+0000","last_became_peered":"2023-10-16T14:32:27.088596+0000","last_unstale":"2023-10-16T14:32:27.091297+0000","last_undegraded":"2023-10-16T14:32:27.091297+0000","last_fullsized":"2023-10-16T14:32:27.091297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:10.629767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062551+0000","last_change":"2023-10-16T14:32:27.089558+0000","last_active":"2023-10-16T14:32:28.062551+0000","last_peered":"2023-10-16T14:32:28.062551+0000","last_clean":"2023-10-16T14:32:28.062551+0000","last_became_active":"2023-10-16T14:32:27.089219+0000","last_became_peered":"2023-10-16T14:32:27.089219+0000","last_unstale":"2023-10-16T14:32:28.062551+0000","last_undegraded":"2023-10-16T14:32:28.062551+0000","last_fullsized":"2023-10-16T14:32:28.062551+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:12.296929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062472+0000","last_change":"2023-10-16T14:32:27.077318+0000","last_active":"2023-10-16T14:32:28.062472+0000","last_peered":"2023-10-16T14:32:28.062472+0000","last_clean":"2023-10-16T14:32:28.062472+0000","last_became_active":"2023-10-16T14:32:27.076872+0000","last_became_peered":"2023-10-16T14:32:27.076872+0000","last_unstale":"2023-10-16T14:32:28.062472+0000","last_undegraded":"2023-10-16T14:32:28.062472+0000","last_fullsized":"2023-10-16T14:32:28.062472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:56.523991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093714+0000","last_change":"2023-10-16T14:32:27.089017+0000","last_active":"2023-10-16T14:32:27.093714+0000","last_peered":"2023-10-16T14:32:27.093714+0000","last_clean":"2023-10-16T14:32:27.093714+0000","last_became_active":"2023-10-16T14:32:27.088729+0000","last_became_peered":"2023-10-16T14:32:27.088729+0000","last_unstale":"2023-10-16T14:32:27.093714+0000","last_undegraded":"2023-10-16T14:32:27.093714+0000","last_fullsized":"2023-10-16T14:32:27.093714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:22.218376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095134+0000","last_change":"2023-10-16T14:32:27.090448+0000","last_active":"2023-10-16T14:32:27.095134+0000","last_peered":"2023-10-16T14:32:27.095134+0000","last_clean":"2023-10-16T14:32:27.095134+0000","last_became_active":"2023-10-16T14:32:27.090108+0000","last_became_peered":"2023-10-16T14:32:27.090108+0000","last_unstale":"2023-10-16T14:32:27.095134+0000","last_undegraded":"2023-10-16T14:32:27.095134+0000","last_fullsized":"2023-10-16T14:32:27.095134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:32.074873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065957+0000","last_change":"2023-10-16T14:32:27.088139+0000","last_active":"2023-10-16T14:32:28.065957+0000","last_peered":"2023-10-16T14:32:28.065957+0000","last_clean":"2023-10-16T14:32:28.065957+0000","last_became_active":"2023-10-16T14:32:27.086923+0000","last_became_peered":"2023-10-16T14:32:27.086923+0000","last_unstale":"2023-10-16T14:32:28.065957+0000","last_undegraded":"2023-10-16T14:32:28.065957+0000","last_fullsized":"2023-10-16T14:32:28.065957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:16.613233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060300+0000","last_change":"2023-10-16T14:32:27.093558+0000","last_active":"2023-10-16T14:32:28.060300+0000","last_peered":"2023-10-16T14:32:28.060300+0000","last_clean":"2023-10-16T14:32:28.060300+0000","last_became_active":"2023-10-16T14:32:27.093251+0000","last_became_peered":"2023-10-16T14:32:27.093251+0000","last_unstale":"2023-10-16T14:32:28.060300+0000","last_undegraded":"2023-10-16T14:32:28.060300+0000","last_fullsized":"2023-10-16T14:32:28.060300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:46.969213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.089637+0000","last_change":"2023-10-16T14:32:27.083171+0000","last_active":"2023-10-16T14:32:27.089637+0000","last_peered":"2023-10-16T14:32:27.089637+0000","last_clean":"2023-10-16T14:32:27.089637+0000","last_became_active":"2023-10-16T14:32:27.082464+0000","last_became_peered":"2023-10-16T14:32:27.082464+0000","last_unstale":"2023-10-16T14:32:27.089637+0000","last_undegraded":"2023-10-16T14:32:27.089637+0000","last_fullsized":"2023-10-16T14:32:27.089637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.158536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064991+0000","last_change":"2023-10-16T14:32:27.085600+0000","last_active":"2023-10-16T14:32:28.064991+0000","last_peered":"2023-10-16T14:32:28.064991+0000","last_clean":"2023-10-16T14:32:28.064991+0000","last_became_active":"2023-10-16T14:32:27.085296+0000","last_became_peered":"2023-10-16T14:32:27.085296+0000","last_unstale":"2023-10-16T14:32:28.064991+0000","last_undegraded":"2023-10-16T14:32:28.064991+0000","last_fullsized":"2023-10-16T14:32:28.064991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:10.836410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095408+0000","last_change":"2023-10-16T14:32:27.094503+0000","last_active":"2023-10-16T14:32:27.095408+0000","last_peered":"2023-10-16T14:32:27.095408+0000","last_clean":"2023-10-16T14:32:27.095408+0000","last_became_active":"2023-10-16T14:32:27.093470+0000","last_became_peered":"2023-10-16T14:32:27.093470+0000","last_unstale":"2023-10-16T14:32:27.095408+0000","last_undegraded":"2023-10-16T14:32:27.095408+0000","last_fullsized":"2023-10-16T14:32:27.095408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:41.739097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063959+0000","last_change":"2023-10-16T14:32:27.074556+0000","last_active":"2023-10-16T14:32:28.063959+0000","last_peered":"2023-10-16T14:32:28.063959+0000","last_clean":"2023-10-16T14:32:28.063959+0000","last_became_active":"2023-10-16T14:32:27.073186+0000","last_became_peered":"2023-10-16T14:32:27.073186+0000","last_unstale":"2023-10-16T14:32:28.063959+0000","last_undegraded":"2023-10-16T14:32:28.063959+0000","last_fullsized":"2023-10-16T14:32:28.063959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:45.554454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.090364+0000","last_change":"2023-10-16T14:32:27.086553+0000","last_active":"2023-10-16T14:32:27.090364+0000","last_peered":"2023-10-16T14:32:27.090364+0000","last_clean":"2023-10-16T14:32:27.090364+0000","last_became_active":"2023-10-16T14:32:27.084756+0000","last_became_peered":"2023-10-16T14:32:27.084756+0000","last_unstale":"2023-10-16T14:32:27.090364+0000","last_undegraded":"2023-10-16T14:32:27.090364+0000","last_fullsized":"2023-10-16T14:32:27.090364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:14.710888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060779+0000","last_change":"2023-10-16T14:32:27.098329+0000","last_active":"2023-10-16T14:32:28.060779+0000","last_peered":"2023-10-16T14:32:28.060779+0000","last_clean":"2023-10-16T14:32:28.060779+0000","last_became_active":"2023-10-16T14:32:27.098014+0000","last_became_peered":"2023-10-16T14:32:27.098014+0000","last_unstale":"2023-10-16T14:32:28.060779+0000","last_undegraded":"2023-10-16T14:32:28.060779+0000","last_fullsized":"2023-10-16T14:32:28.060779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:51.476382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093033+0000","last_change":"2023-10-16T14:32:27.090198+0000","last_active":"2023-10-16T14:32:27.093033+0000","last_peered":"2023-10-16T14:32:27.093033+0000","last_clean":"2023-10-16T14:32:27.093033+0000","last_became_active":"2023-10-16T14:32:27.089922+0000","last_became_peered":"2023-10-16T14:32:27.089922+0000","last_unstale":"2023-10-16T14:32:27.093033+0000","last_undegraded":"2023-10-16T14:32:27.093033+0000","last_fullsized":"2023-10-16T14:32:27.093033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:29.230685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091719+0000","last_change":"2023-10-16T14:32:27.088490+0000","last_active":"2023-10-16T14:32:27.091719+0000","last_peered":"2023-10-16T14:32:27.091719+0000","last_clean":"2023-10-16T14:32:27.091719+0000","last_became_active":"2023-10-16T14:32:27.088165+0000","last_became_peered":"2023-10-16T14:32:27.088165+0000","last_unstale":"2023-10-16T14:32:27.091719+0000","last_undegraded":"2023-10-16T14:32:27.091719+0000","last_fullsized":"2023-10-16T14:32:27.091719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:13.587912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060555+0000","last_change":"2023-10-16T14:32:27.095953+0000","last_active":"2023-10-16T14:32:28.060555+0000","last_peered":"2023-10-16T14:32:28.060555+0000","last_clean":"2023-10-16T14:32:28.060555+0000","last_became_active":"2023-10-16T14:32:27.095640+0000","last_became_peered":"2023-10-16T14:32:27.095640+0000","last_unstale":"2023-10-16T14:32:28.060555+0000","last_undegraded":"2023-10-16T14:32:28.060555+0000","last_fullsized":"2023-10-16T14:32:28.060555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:00.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091755+0000","last_change":"2023-10-16T14:32:27.086578+0000","last_active":"2023-10-16T14:32:27.091755+0000","last_peered":"2023-10-16T14:32:27.091755+0000","last_clean":"2023-10-16T14:32:27.091755+0000","last_became_active":"2023-10-16T14:32:27.086253+0000","last_became_peered":"2023-10-16T14:32:27.086253+0000","last_unstale":"2023-10-16T14:32:27.091755+0000","last_undegraded":"2023-10-16T14:32:27.091755+0000","last_fullsized":"2023-10-16T14:32:27.091755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:18.755495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064388+0000","last_change":"2023-10-16T14:32:27.072284+0000","last_active":"2023-10-16T14:32:28.064388+0000","last_peered":"2023-10-16T14:32:28.064388+0000","last_clean":"2023-10-16T14:32:28.064388+0000","last_became_active":"2023-10-16T14:32:27.070929+0000","last_became_peered":"2023-10-16T14:32:27.070929+0000","last_unstale":"2023-10-16T14:32:28.064388+0000","last_undegraded":"2023-10-16T14:32:28.064388+0000","last_fullsized":"2023-10-16T14:32:28.064388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:18.921454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060106+0000","last_change":"2023-10-16T14:32:27.089824+0000","last_active":"2023-10-16T14:32:28.060106+0000","last_peered":"2023-10-16T14:32:28.060106+0000","last_clean":"2023-10-16T14:32:28.060106+0000","last_became_active":"2023-10-16T14:32:27.089472+0000","last_became_peered":"2023-10-16T14:32:27.089472+0000","last_unstale":"2023-10-16T14:32:28.060106+0000","last_undegraded":"2023-10-16T14:32:28.060106+0000","last_fullsized":"2023-10-16T14:32:28.060106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:41.162259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099336+0000","last_change":"2023-10-16T14:32:27.097790+0000","last_active":"2023-10-16T14:32:27.099336+0000","last_peered":"2023-10-16T14:32:27.099336+0000","last_clean":"2023-10-16T14:32:27.099336+0000","last_became_active":"2023-10-16T14:32:27.096989+0000","last_became_peered":"2023-10-16T14:32:27.096989+0000","last_unstale":"2023-10-16T14:32:27.099336+0000","last_undegraded":"2023-10-16T14:32:27.099336+0000","last_fullsized":"2023-10-16T14:32:27.099336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:00:32.174270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095548+0000","last_change":"2023-10-16T14:32:27.092125+0000","last_active":"2023-10-16T14:32:27.095548+0000","last_peered":"2023-10-16T14:32:27.095548+0000","last_clean":"2023-10-16T14:32:27.095548+0000","last_became_active":"2023-10-16T14:32:27.090135+0000","last_became_peered":"2023-10-16T14:32:27.090135+0000","last_unstale":"2023-10-16T14:32:27.095548+0000","last_undegraded":"2023-10-16T14:32:27.095548+0000","last_fullsized":"2023-10-16T14:32:27.095548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:34.589282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.194271+0000","last_change":"2023-10-16T14:32:26.066926+0000","last_active":"2023-10-16T14:32:27.194271+0000","last_peered":"2023-10-16T14:32:27.194271+0000","last_clean":"2023-10-16T14:32:27.194271+0000","last_became_active":"2023-10-16T14:32:26.061602+0000","last_became_peered":"2023-10-16T14:32:26.061602+0000","last_unstale":"2023-10-16T14:32:27.194271+0000","last_undegraded":"2023-10-16T14:32:27.194271+0000","last_fullsized":"2023-10-16T14:32:27.194271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:49.199887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096705+0000","last_change":"2023-10-16T14:32:27.088114+0000","last_active":"2023-10-16T14:32:27.096705+0000","last_peered":"2023-10-16T14:32:27.096705+0000","last_clean":"2023-10-16T14:32:27.096705+0000","last_became_active":"2023-10-16T14:32:27.087807+0000","last_became_peered":"2023-10-16T14:32:27.087807+0000","last_unstale":"2023-10-16T14:32:27.096705+0000","last_undegraded":"2023-10-16T14:32:27.096705+0000","last_fullsized":"2023-10-16T14:32:27.096705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:52.106751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.190857+0000","last_change":"2023-10-16T14:32:26.069056+0000","last_active":"2023-10-16T14:32:27.190857+0000","last_peered":"2023-10-16T14:32:27.190857+0000","last_clean":"2023-10-16T14:32:27.190857+0000","last_became_active":"2023-10-16T14:32:26.068602+0000","last_became_peered":"2023-10-16T14:32:26.068602+0000","last_unstale":"2023-10-16T14:32:27.190857+0000","last_undegraded":"2023-10-16T14:32:27.190857+0000","last_fullsized":"2023-10-16T14:32:27.190857+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:17.905586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093702+0000","last_change":"2023-10-16T14:32:27.091506+0000","last_active":"2023-10-16T14:32:27.093702+0000","last_peered":"2023-10-16T14:32:27.093702+0000","last_clean":"2023-10-16T14:32:27.093702+0000","last_became_active":"2023-10-16T14:32:27.091162+0000","last_became_peered":"2023-10-16T14:32:27.091162+0000","last_unstale":"2023-10-16T14:32:27.093702+0000","last_undegraded":"2023-10-16T14:32:27.093702+0000","last_fullsized":"2023-10-16T14:32:27.093702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:25.686068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.193963+0000","last_change":"2023-10-16T14:32:26.044577+0000","last_active":"2023-10-16T14:32:27.193963+0000","last_peered":"2023-10-16T14:32:27.193963+0000","last_clean":"2023-10-16T14:32:27.193963+0000","last_became_active":"2023-10-16T14:32:26.044320+0000","last_became_peered":"2023-10-16T14:32:26.044320+0000","last_unstale":"2023-10-16T14:32:27.193963+0000","last_undegraded":"2023-10-16T14:32:27.193963+0000","last_fullsized":"2023-10-16T14:32:27.193963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:33.221898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.087769+0000","last_change":"2023-10-16T14:32:27.085630+0000","last_active":"2023-10-16T14:32:27.087769+0000","last_peered":"2023-10-16T14:32:27.087769+0000","last_clean":"2023-10-16T14:32:27.087769+0000","last_became_active":"2023-10-16T14:32:27.085299+0000","last_became_peered":"2023-10-16T14:32:27.085299+0000","last_unstale":"2023-10-16T14:32:27.087769+0000","last_undegraded":"2023-10-16T14:32:27.087769+0000","last_fullsized":"2023-10-16T14:32:27.087769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:38.346197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.191829+0000","last_change":"2023-10-16T14:32:26.058393+0000","last_active":"2023-10-16T14:32:27.191829+0000","last_peered":"2023-10-16T14:32:27.191829+0000","last_clean":"2023-10-16T14:32:27.191829+0000","last_became_active":"2023-10-16T14:32:26.058104+0000","last_became_peered":"2023-10-16T14:32:26.058104+0000","last_unstale":"2023-10-16T14:32:27.191829+0000","last_undegraded":"2023-10-16T14:32:27.191829+0000","last_fullsized":"2023-10-16T14:32:27.191829+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:06.321202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095263+0000","last_change":"2023-10-16T14:32:27.093577+0000","last_active":"2023-10-16T14:32:27.095263+0000","last_peered":"2023-10-16T14:32:27.095263+0000","last_clean":"2023-10-16T14:32:27.095263+0000","last_became_active":"2023-10-16T14:32:27.092940+0000","last_became_peered":"2023-10-16T14:32:27.092940+0000","last_unstale":"2023-10-16T14:32:27.095263+0000","last_undegraded":"2023-10-16T14:32:27.095263+0000","last_fullsized":"2023-10-16T14:32:27.095263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:30.968363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"83'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063085+0000","last_change":"2023-10-16T14:32:26.063887+0000","last_active":"2023-10-16T14:32:28.063085+0000","last_peered":"2023-10-16T14:32:28.063085+0000","last_clean":"2023-10-16T14:32:28.063085+0000","last_became_active":"2023-10-16T14:32:26.063169+0000","last_became_peered":"2023-10-16T14:32:26.063169+0000","last_unstale":"2023-10-16T14:32:28.063085+0000","last_undegraded":"2023-10-16T14:32:28.063085+0000","last_fullsized":"2023-10-16T14:32:28.063085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:45.139919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.089867+0000","last_change":"2023-10-16T14:32:27.084249+0000","last_active":"2023-10-16T14:32:27.089867+0000","last_peered":"2023-10-16T14:32:27.089867+0000","last_clean":"2023-10-16T14:32:27.089867+0000","last_became_active":"2023-10-16T14:32:27.083857+0000","last_became_peered":"2023-10-16T14:32:27.083857+0000","last_unstale":"2023-10-16T14:32:27.089867+0000","last_undegraded":"2023-10-16T14:32:27.089867+0000","last_fullsized":"2023-10-16T14:32:27.089867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:45.275937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.193183+0000","last_change":"2023-10-16T14:32:26.068093+0000","last_active":"2023-10-16T14:32:27.193183+0000","last_peered":"2023-10-16T14:32:27.193183+0000","last_clean":"2023-10-16T14:32:27.193183+0000","last_became_active":"2023-10-16T14:32:26.067766+0000","last_became_peered":"2023-10-16T14:32:26.067766+0000","last_unstale":"2023-10-16T14:32:27.193183+0000","last_undegraded":"2023-10-16T14:32:27.193183+0000","last_fullsized":"2023-10-16T14:32:27.193183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:56.648060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.088923+0000","last_change":"2023-10-16T14:32:27.084862+0000","last_active":"2023-10-16T14:32:27.088923+0000","last_peered":"2023-10-16T14:32:27.088923+0000","last_clean":"2023-10-16T14:32:27.088923+0000","last_became_active":"2023-10-16T14:32:27.084323+0000","last_became_peered":"2023-10-16T14:32:27.084323+0000","last_unstale":"2023-10-16T14:32:27.088923+0000","last_undegraded":"2023-10-16T14:32:27.088923+0000","last_fullsized":"2023-10-16T14:32:27.088923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:45.041903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.082792+0000","last_change":"2023-10-16T14:32:26.064084+0000","last_active":"2023-10-16T14:32:27.082792+0000","last_peered":"2023-10-16T14:32:27.082792+0000","last_clean":"2023-10-16T14:32:27.082792+0000","last_became_active":"2023-10-16T14:32:26.063323+0000","last_became_peered":"2023-10-16T14:32:26.063323+0000","last_unstale":"2023-10-16T14:32:27.082792+0000","last_undegraded":"2023-10-16T14:32:27.082792+0000","last_fullsized":"2023-10-16T14:32:27.082792+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:41.572256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092980+0000","last_change":"2023-10-16T14:32:27.090801+0000","last_active":"2023-10-16T14:32:27.092980+0000","last_peered":"2023-10-16T14:32:27.092980+0000","last_clean":"2023-10-16T14:32:27.092980+0000","last_became_active":"2023-10-16T14:32:27.090208+0000","last_became_peered":"2023-10-16T14:32:27.090208+0000","last_unstale":"2023-10-16T14:32:27.092980+0000","last_undegraded":"2023-10-16T14:32:27.092980+0000","last_fullsized":"2023-10-16T14:32:27.092980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:30.455378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062427+0000","last_change":"2023-10-16T14:32:26.071347+0000","last_active":"2023-10-16T14:32:28.062427+0000","last_peered":"2023-10-16T14:32:28.062427+0000","last_clean":"2023-10-16T14:32:28.062427+0000","last_became_active":"2023-10-16T14:32:26.071016+0000","last_became_peered":"2023-10-16T14:32:26.071016+0000","last_unstale":"2023-10-16T14:32:28.062427+0000","last_undegraded":"2023-10-16T14:32:28.062427+0000","last_fullsized":"2023-10-16T14:32:28.062427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:53.950061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065120+0000","last_change":"2023-10-16T14:32:27.085538+0000","last_active":"2023-10-16T14:32:28.065120+0000","last_peered":"2023-10-16T14:32:28.065120+0000","last_clean":"2023-10-16T14:32:28.065120+0000","last_became_active":"2023-10-16T14:32:27.085251+0000","last_became_peered":"2023-10-16T14:32:27.085251+0000","last_unstale":"2023-10-16T14:32:28.065120+0000","last_undegraded":"2023-10-16T14:32:28.065120+0000","last_fullsized":"2023-10-16T14:32:28.065120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:50.770169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065736+0000","last_change":"2023-10-16T14:32:26.058663+0000","last_active":"2023-10-16T14:32:28.065736+0000","last_peered":"2023-10-16T14:32:28.065736+0000","last_clean":"2023-10-16T14:32:28.065736+0000","last_became_active":"2023-10-16T14:32:26.058398+0000","last_became_peered":"2023-10-16T14:32:26.058398+0000","last_unstale":"2023-10-16T14:32:28.065736+0000","last_undegraded":"2023-10-16T14:32:28.065736+0000","last_fullsized":"2023-10-16T14:32:28.065736+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:31.059005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.066080+0000","last_change":"2023-10-16T14:32:27.078718+0000","last_active":"2023-10-16T14:32:28.066080+0000","last_peered":"2023-10-16T14:32:28.066080+0000","last_clean":"2023-10-16T14:32:28.066080+0000","last_became_active":"2023-10-16T14:32:27.078237+0000","last_became_peered":"2023-10-16T14:32:27.078237+0000","last_unstale":"2023-10-16T14:32:28.066080+0000","last_undegraded":"2023-10-16T14:32:28.066080+0000","last_fullsized":"2023-10-16T14:32:28.066080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:22.099583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.060278+0000","last_change":"2023-10-16T14:32:26.065276+0000","last_active":"2023-10-16T14:32:27.060278+0000","last_peered":"2023-10-16T14:32:27.060278+0000","last_clean":"2023-10-16T14:32:27.060278+0000","last_became_active":"2023-10-16T14:32:26.064996+0000","last_became_peered":"2023-10-16T14:32:26.064996+0000","last_unstale":"2023-10-16T14:32:27.060278+0000","last_undegraded":"2023-10-16T14:32:27.060278+0000","last_fullsized":"2023-10-16T14:32:27.060278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:06.684184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.094952+0000","last_change":"2023-10-16T14:32:27.084668+0000","last_active":"2023-10-16T14:32:27.094952+0000","last_peered":"2023-10-16T14:32:27.094952+0000","last_clean":"2023-10-16T14:32:27.094952+0000","last_became_active":"2023-10-16T14:32:27.084290+0000","last_became_peered":"2023-10-16T14:32:27.084290+0000","last_unstale":"2023-10-16T14:32:27.094952+0000","last_undegraded":"2023-10-16T14:32:27.094952+0000","last_fullsized":"2023-10-16T14:32:27.094952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:51.811557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"83'3","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.066134+0000","last_change":"2023-10-16T14:32:26.059570+0000","last_active":"2023-10-16T14:32:28.066134+0000","last_peered":"2023-10-16T14:32:28.066134+0000","last_clean":"2023-10-16T14:32:28.066134+0000","last_became_active":"2023-10-16T14:32:26.059227+0000","last_became_peered":"2023-10-16T14:32:26.059227+0000","last_unstale":"2023-10-16T14:32:28.066134+0000","last_undegraded":"2023-10-16T14:32:28.066134+0000","last_fullsized":"2023-10-16T14:32:28.066134+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:43.387829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065737+0000","last_change":"2023-10-16T14:32:27.080844+0000","last_active":"2023-10-16T14:32:28.065737+0000","last_peered":"2023-10-16T14:32:28.065737+0000","last_clean":"2023-10-16T14:32:28.065737+0000","last_became_active":"2023-10-16T14:32:27.080535+0000","last_became_peered":"2023-10-16T14:32:27.080535+0000","last_unstale":"2023-10-16T14:32:28.065737+0000","last_undegraded":"2023-10-16T14:32:28.065737+0000","last_fullsized":"2023-10-16T14:32:28.065737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:36.035450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"83'6","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063727+0000","last_change":"2023-10-16T14:32:26.070854+0000","last_active":"2023-10-16T14:32:28.063727+0000","last_peered":"2023-10-16T14:32:28.063727+0000","last_clean":"2023-10-16T14:32:28.063727+0000","last_became_active":"2023-10-16T14:32:26.070511+0000","last_became_peered":"2023-10-16T14:32:26.070511+0000","last_unstale":"2023-10-16T14:32:28.063727+0000","last_undegraded":"2023-10-16T14:32:28.063727+0000","last_fullsized":"2023-10-16T14:32:28.063727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:13.898978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099773+0000","last_change":"2023-10-16T14:32:27.099060+0000","last_active":"2023-10-16T14:32:27.099773+0000","last_peered":"2023-10-16T14:32:27.099773+0000","last_clean":"2023-10-16T14:32:27.099773+0000","last_became_active":"2023-10-16T14:32:27.098747+0000","last_became_peered":"2023-10-16T14:32:27.098747+0000","last_unstale":"2023-10-16T14:32:27.099773+0000","last_undegraded":"2023-10-16T14:32:27.099773+0000","last_fullsized":"2023-10-16T14:32:27.099773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:34.934405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.194627+0000","last_change":"2023-10-16T14:32:26.066908+0000","last_active":"2023-10-16T14:32:27.194627+0000","last_peered":"2023-10-16T14:32:27.194627+0000","last_clean":"2023-10-16T14:32:27.194627+0000","last_became_active":"2023-10-16T14:32:26.066342+0000","last_became_peered":"2023-10-16T14:32:26.066342+0000","last_unstale":"2023-10-16T14:32:27.194627+0000","last_undegraded":"2023-10-16T14:32:27.194627+0000","last_fullsized":"2023-10-16T14:32:27.194627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:16.461838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063933+0000","last_change":"2023-10-16T14:32:27.089165+0000","last_active":"2023-10-16T14:32:28.063933+0000","last_peered":"2023-10-16T14:32:28.063933+0000","last_clean":"2023-10-16T14:32:28.063933+0000","last_became_active":"2023-10-16T14:32:27.088890+0000","last_became_peered":"2023-10-16T14:32:27.088890+0000","last_unstale":"2023-10-16T14:32:28.063933+0000","last_undegraded":"2023-10-16T14:32:28.063933+0000","last_fullsized":"2023-10-16T14:32:28.063933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:35.136365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.072560+0000","last_change":"2023-10-16T14:32:26.063177+0000","last_active":"2023-10-16T14:32:27.072560+0000","last_peered":"2023-10-16T14:32:27.072560+0000","last_clean":"2023-10-16T14:32:27.072560+0000","last_became_active":"2023-10-16T14:32:26.062887+0000","last_became_peered":"2023-10-16T14:32:26.062887+0000","last_unstale":"2023-10-16T14:32:27.072560+0000","last_undegraded":"2023-10-16T14:32:27.072560+0000","last_fullsized":"2023-10-16T14:32:27.072560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:24.710141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062211+0000","last_change":"2023-10-16T14:32:27.075554+0000","last_active":"2023-10-16T14:32:28.062211+0000","last_peered":"2023-10-16T14:32:28.062211+0000","last_clean":"2023-10-16T14:32:28.062211+0000","last_became_active":"2023-10-16T14:32:27.075120+0000","last_became_peered":"2023-10-16T14:32:27.075120+0000","last_unstale":"2023-10-16T14:32:28.062211+0000","last_undegraded":"2023-10-16T14:32:28.062211+0000","last_fullsized":"2023-10-16T14:32:28.062211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.841791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.082920+0000","last_change":"2023-10-16T14:32:26.052648+0000","last_active":"2023-10-16T14:32:27.082920+0000","last_peered":"2023-10-16T14:32:27.082920+0000","last_clean":"2023-10-16T14:32:27.082920+0000","last_became_active":"2023-10-16T14:32:26.052247+0000","last_became_peered":"2023-10-16T14:32:26.052247+0000","last_unstale":"2023-10-16T14:32:27.082920+0000","last_undegraded":"2023-10-16T14:32:27.082920+0000","last_fullsized":"2023-10-16T14:32:27.082920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:43.863681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064335+0000","last_change":"2023-10-16T14:32:27.082795+0000","last_active":"2023-10-16T14:32:28.064335+0000","last_peered":"2023-10-16T14:32:28.064335+0000","last_clean":"2023-10-16T14:32:28.064335+0000","last_became_active":"2023-10-16T14:32:27.082488+0000","last_became_peered":"2023-10-16T14:32:27.082488+0000","last_unstale":"2023-10-16T14:32:28.064335+0000","last_undegraded":"2023-10-16T14:32:28.064335+0000","last_fullsized":"2023-10-16T14:32:28.064335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:18.755495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"83'8","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063754+0000","last_change":"2023-10-16T14:32:26.062317+0000","last_active":"2023-10-16T14:32:28.063754+0000","last_peered":"2023-10-16T14:32:28.063754+0000","last_clean":"2023-10-16T14:32:28.063754+0000","last_became_active":"2023-10-16T14:32:26.061519+0000","last_became_peered":"2023-10-16T14:32:26.061519+0000","last_unstale":"2023-10-16T14:32:28.063754+0000","last_undegraded":"2023-10-16T14:32:28.063754+0000","last_fullsized":"2023-10-16T14:32:28.063754+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:06.321202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065800+0000","last_change":"2023-10-16T14:32:27.078888+0000","last_active":"2023-10-16T14:32:28.065800+0000","last_peered":"2023-10-16T14:32:28.065800+0000","last_clean":"2023-10-16T14:32:28.065800+0000","last_became_active":"2023-10-16T14:32:27.078598+0000","last_became_peered":"2023-10-16T14:32:27.078598+0000","last_unstale":"2023-10-16T14:32:28.065800+0000","last_undegraded":"2023-10-16T14:32:28.065800+0000","last_fullsized":"2023-10-16T14:32:28.065800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:47.249405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.191515+0000","last_change":"2023-10-16T14:32:26.065271+0000","last_active":"2023-10-16T14:32:27.191515+0000","last_peered":"2023-10-16T14:32:27.191515+0000","last_clean":"2023-10-16T14:32:27.191515+0000","last_became_active":"2023-10-16T14:32:26.064844+0000","last_became_peered":"2023-10-16T14:32:26.064844+0000","last_unstale":"2023-10-16T14:32:27.191515+0000","last_undegraded":"2023-10-16T14:32:27.191515+0000","last_fullsized":"2023-10-16T14:32:27.191515+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:34.471465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099757+0000","last_change":"2023-10-16T14:32:27.099055+0000","last_active":"2023-10-16T14:32:27.099757+0000","last_peered":"2023-10-16T14:32:27.099757+0000","last_clean":"2023-10-16T14:32:27.099757+0000","last_became_active":"2023-10-16T14:32:27.098588+0000","last_became_peered":"2023-10-16T14:32:27.098588+0000","last_unstale":"2023-10-16T14:32:27.099757+0000","last_undegraded":"2023-10-16T14:32:27.099757+0000","last_fullsized":"2023-10-16T14:32:27.099757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:12.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.195054+0000","last_change":"2023-10-16T14:32:26.048364+0000","last_active":"2023-10-16T14:32:27.195054+0000","last_peered":"2023-10-16T14:32:27.195054+0000","last_clean":"2023-10-16T14:32:27.195054+0000","last_became_active":"2023-10-16T14:32:26.048010+0000","last_became_peered":"2023-10-16T14:32:26.048010+0000","last_unstale":"2023-10-16T14:32:27.195054+0000","last_undegraded":"2023-10-16T14:32:27.195054+0000","last_fullsized":"2023-10-16T14:32:27.195054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:01.775709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.076325+0000","last_change":"2023-10-16T14:32:27.070297+0000","last_active":"2023-10-16T14:32:27.076325+0000","last_peered":"2023-10-16T14:32:27.076325+0000","last_clean":"2023-10-16T14:32:27.076325+0000","last_became_active":"2023-10-16T14:32:27.069610+0000","last_became_peered":"2023-10-16T14:32:27.069610+0000","last_unstale":"2023-10-16T14:32:27.076325+0000","last_undegraded":"2023-10-16T14:32:27.076325+0000","last_fullsized":"2023-10-16T14:32:27.076325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:35.136365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.191926+0000","last_change":"2023-10-16T14:32:26.053019+0000","last_active":"2023-10-16T14:32:27.191926+0000","last_peered":"2023-10-16T14:32:27.191926+0000","last_clean":"2023-10-16T14:32:27.191926+0000","last_became_active":"2023-10-16T14:32:26.052181+0000","last_became_peered":"2023-10-16T14:32:26.052181+0000","last_unstale":"2023-10-16T14:32:27.191926+0000","last_undegraded":"2023-10-16T14:32:27.191926+0000","last_fullsized":"2023-10-16T14:32:27.191926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:15.685139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.066090+0000","last_change":"2023-10-16T14:32:27.084063+0000","last_active":"2023-10-16T14:32:28.066090+0000","last_peered":"2023-10-16T14:32:28.066090+0000","last_clean":"2023-10-16T14:32:28.066090+0000","last_became_active":"2023-10-16T14:32:27.083721+0000","last_became_peered":"2023-10-16T14:32:27.083721+0000","last_unstale":"2023-10-16T14:32:28.066090+0000","last_undegraded":"2023-10-16T14:32:28.066090+0000","last_fullsized":"2023-10-16T14:32:28.066090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.832025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.078623+0000","last_change":"2023-10-16T14:32:26.049723+0000","last_active":"2023-10-16T14:32:27.078623+0000","last_peered":"2023-10-16T14:32:27.078623+0000","last_clean":"2023-10-16T14:32:27.078623+0000","last_became_active":"2023-10-16T14:32:26.049452+0000","last_became_peered":"2023-10-16T14:32:26.049452+0000","last_unstale":"2023-10-16T14:32:27.078623+0000","last_undegraded":"2023-10-16T14:32:27.078623+0000","last_fullsized":"2023-10-16T14:32:27.078623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:06.837100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099620+0000","last_change":"2023-10-16T14:32:27.098948+0000","last_active":"2023-10-16T14:32:27.099620+0000","last_peered":"2023-10-16T14:32:27.099620+0000","last_clean":"2023-10-16T14:32:27.099620+0000","last_became_active":"2023-10-16T14:32:27.098611+0000","last_became_peered":"2023-10-16T14:32:27.098611+0000","last_unstale":"2023-10-16T14:32:27.099620+0000","last_undegraded":"2023-10-16T14:32:27.099620+0000","last_fullsized":"2023-10-16T14:32:27.099620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:42.753061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.192152+0000","last_change":"2023-10-16T14:32:26.067358+0000","last_active":"2023-10-16T14:32:27.192152+0000","last_peered":"2023-10-16T14:32:27.192152+0000","last_clean":"2023-10-16T14:32:27.192152+0000","last_became_active":"2023-10-16T14:32:26.067105+0000","last_became_peered":"2023-10-16T14:32:26.067105+0000","last_unstale":"2023-10-16T14:32:27.192152+0000","last_undegraded":"2023-10-16T14:32:27.192152+0000","last_fullsized":"2023-10-16T14:32:27.192152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:35.019582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.097032+0000","last_change":"2023-10-16T14:32:27.094277+0000","last_active":"2023-10-16T14:32:27.097032+0000","last_peered":"2023-10-16T14:32:27.097032+0000","last_clean":"2023-10-16T14:32:27.097032+0000","last_became_active":"2023-10-16T14:32:27.093999+0000","last_became_peered":"2023-10-16T14:32:27.093999+0000","last_unstale":"2023-10-16T14:32:27.097032+0000","last_undegraded":"2023-10-16T14:32:27.097032+0000","last_fullsized":"2023-10-16T14:32:27.097032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:45.033569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.070756+0000","last_change":"2023-10-16T14:32:26.058045+0000","last_active":"2023-10-16T14:32:27.070756+0000","last_peered":"2023-10-16T14:32:27.070756+0000","last_clean":"2023-10-16T14:32:27.070756+0000","last_became_active":"2023-10-16T14:32:26.056976+0000","last_became_peered":"2023-10-16T14:32:26.056976+0000","last_unstale":"2023-10-16T14:32:27.070756+0000","last_undegraded":"2023-10-16T14:32:27.070756+0000","last_fullsized":"2023-10-16T14:32:27.070756+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:45.953345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091004+0000","last_change":"2023-10-16T14:32:27.087116+0000","last_active":"2023-10-16T14:32:27.091004+0000","last_peered":"2023-10-16T14:32:27.091004+0000","last_clean":"2023-10-16T14:32:27.091004+0000","last_became_active":"2023-10-16T14:32:27.086772+0000","last_became_peered":"2023-10-16T14:32:27.086772+0000","last_unstale":"2023-10-16T14:32:27.091004+0000","last_undegraded":"2023-10-16T14:32:27.091004+0000","last_fullsized":"2023-10-16T14:32:27.091004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:30.444544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.082501+0000","last_change":"2023-10-16T14:32:26.065155+0000","last_active":"2023-10-16T14:32:27.082501+0000","last_peered":"2023-10-16T14:32:27.082501+0000","last_clean":"2023-10-16T14:32:27.082501+0000","last_became_active":"2023-10-16T14:32:26.064901+0000","last_became_peered":"2023-10-16T14:32:26.064901+0000","last_unstale":"2023-10-16T14:32:27.082501+0000","last_undegraded":"2023-10-16T14:32:27.082501+0000","last_fullsized":"2023-10-16T14:32:27.082501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:57.587995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093740+0000","last_change":"2023-10-16T14:32:27.092544+0000","last_active":"2023-10-16T14:32:27.093740+0000","last_peered":"2023-10-16T14:32:27.093740+0000","last_clean":"2023-10-16T14:32:27.093740+0000","last_became_active":"2023-10-16T14:32:27.092229+0000","last_became_peered":"2023-10-16T14:32:27.092229+0000","last_unstale":"2023-10-16T14:32:27.093740+0000","last_undegraded":"2023-10-16T14:32:27.093740+0000","last_fullsized":"2023-10-16T14:32:27.093740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:37.783295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.193739+0000","last_change":"2023-10-16T14:32:26.066887+0000","last_active":"2023-10-16T14:32:27.193739+0000","last_peered":"2023-10-16T14:32:27.193739+0000","last_clean":"2023-10-16T14:32:27.193739+0000","last_became_active":"2023-10-16T14:32:26.065597+0000","last_became_peered":"2023-10-16T14:32:26.065597+0000","last_unstale":"2023-10-16T14:32:27.193739+0000","last_undegraded":"2023-10-16T14:32:27.193739+0000","last_fullsized":"2023-10-16T14:32:27.193739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:40.146391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092165+0000","last_change":"2023-10-16T14:32:27.088681+0000","last_active":"2023-10-16T14:32:27.092165+0000","last_peered":"2023-10-16T14:32:27.092165+0000","last_clean":"2023-10-16T14:32:27.092165+0000","last_became_active":"2023-10-16T14:32:27.088389+0000","last_became_peered":"2023-10-16T14:32:27.088389+0000","last_unstale":"2023-10-16T14:32:27.092165+0000","last_undegraded":"2023-10-16T14:32:27.092165+0000","last_fullsized":"2023-10-16T14:32:27.092165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:51.220283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.081592+0000","last_change":"2023-10-16T14:32:26.067218+0000","last_active":"2023-10-16T14:32:27.081592+0000","last_peered":"2023-10-16T14:32:27.081592+0000","last_clean":"2023-10-16T14:32:27.081592+0000","last_became_active":"2023-10-16T14:32:26.060284+0000","last_became_peered":"2023-10-16T14:32:26.060284+0000","last_unstale":"2023-10-16T14:32:27.081592+0000","last_undegraded":"2023-10-16T14:32:27.081592+0000","last_fullsized":"2023-10-16T14:32:27.081592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:50.377529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.087384+0000","last_change":"2023-10-16T14:32:27.081250+0000","last_active":"2023-10-16T14:32:27.087384+0000","last_peered":"2023-10-16T14:32:27.087384+0000","last_clean":"2023-10-16T14:32:27.087384+0000","last_became_active":"2023-10-16T14:32:27.080688+0000","last_became_peered":"2023-10-16T14:32:27.080688+0000","last_unstale":"2023-10-16T14:32:27.087384+0000","last_undegraded":"2023-10-16T14:32:27.087384+0000","last_fullsized":"2023-10-16T14:32:27.087384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:06.072326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.076313+0000","last_change":"2023-10-16T14:32:26.065863+0000","last_active":"2023-10-16T14:32:27.076313+0000","last_peered":"2023-10-16T14:32:27.076313+0000","last_clean":"2023-10-16T14:32:27.076313+0000","last_became_active":"2023-10-16T14:32:26.065563+0000","last_became_peered":"2023-10-16T14:32:26.065563+0000","last_unstale":"2023-10-16T14:32:27.076313+0000","last_undegraded":"2023-10-16T14:32:27.076313+0000","last_fullsized":"2023-10-16T14:32:27.076313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:24.670200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091667+0000","last_change":"2023-10-16T14:32:27.087885+0000","last_active":"2023-10-16T14:32:27.091667+0000","last_peered":"2023-10-16T14:32:27.091667+0000","last_clean":"2023-10-16T14:32:27.091667+0000","last_became_active":"2023-10-16T14:32:27.087519+0000","last_became_peered":"2023-10-16T14:32:27.087519+0000","last_unstale":"2023-10-16T14:32:27.091667+0000","last_undegraded":"2023-10-16T14:32:27.091667+0000","last_fullsized":"2023-10-16T14:32:27.091667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:10.629767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.084102+0000","last_change":"2023-10-16T14:32:26.062610+0000","last_active":"2023-10-16T14:32:27.084102+0000","last_peered":"2023-10-16T14:32:27.084102+0000","last_clean":"2023-10-16T14:32:27.084102+0000","last_became_active":"2023-10-16T14:32:26.062178+0000","last_became_peered":"2023-10-16T14:32:26.062178+0000","last_unstale":"2023-10-16T14:32:27.084102+0000","last_undegraded":"2023-10-16T14:32:27.084102+0000","last_fullsized":"2023-10-16T14:32:27.084102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:43.161255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.086234+0000","last_change":"2023-10-16T14:32:27.080147+0000","last_active":"2023-10-16T14:32:27.086234+0000","last_peered":"2023-10-16T14:32:27.086234+0000","last_clean":"2023-10-16T14:32:27.086234+0000","last_became_active":"2023-10-16T14:32:27.079877+0000","last_became_peered":"2023-10-16T14:32:27.079877+0000","last_unstale":"2023-10-16T14:32:27.086234+0000","last_undegraded":"2023-10-16T14:32:27.086234+0000","last_fullsized":"2023-10-16T14:32:27.086234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:10.836410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.068599+0000","last_change":"2023-10-16T14:32:26.063314+0000","last_active":"2023-10-16T14:32:27.068599+0000","last_peered":"2023-10-16T14:32:27.068599+0000","last_clean":"2023-10-16T14:32:27.068599+0000","last_became_active":"2023-10-16T14:32:26.063034+0000","last_became_peered":"2023-10-16T14:32:26.063034+0000","last_unstale":"2023-10-16T14:32:27.068599+0000","last_undegraded":"2023-10-16T14:32:27.068599+0000","last_fullsized":"2023-10-16T14:32:27.068599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:42:27.717271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065366+0000","last_change":"2023-10-16T14:32:27.082613+0000","last_active":"2023-10-16T14:32:28.065366+0000","last_peered":"2023-10-16T14:32:28.065366+0000","last_clean":"2023-10-16T14:32:28.065366+0000","last_became_active":"2023-10-16T14:32:27.082220+0000","last_became_peered":"2023-10-16T14:32:27.082220+0000","last_unstale":"2023-10-16T14:32:28.065366+0000","last_undegraded":"2023-10-16T14:32:28.065366+0000","last_fullsized":"2023-10-16T14:32:28.065366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:35.487333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063901+0000","last_change":"2023-10-16T14:32:26.070689+0000","last_active":"2023-10-16T14:32:28.063901+0000","last_peered":"2023-10-16T14:32:28.063901+0000","last_clean":"2023-10-16T14:32:28.063901+0000","last_became_active":"2023-10-16T14:32:26.070372+0000","last_became_peered":"2023-10-16T14:32:26.070372+0000","last_unstale":"2023-10-16T14:32:28.063901+0000","last_undegraded":"2023-10-16T14:32:28.063901+0000","last_fullsized":"2023-10-16T14:32:28.063901+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:45.870448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096942+0000","last_change":"2023-10-16T14:32:27.090107+0000","last_active":"2023-10-16T14:32:27.096942+0000","last_peered":"2023-10-16T14:32:27.096942+0000","last_clean":"2023-10-16T14:32:27.096942+0000","last_became_active":"2023-10-16T14:32:27.089782+0000","last_became_peered":"2023-10-16T14:32:27.089782+0000","last_unstale":"2023-10-16T14:32:27.096942+0000","last_undegraded":"2023-10-16T14:32:27.096942+0000","last_fullsized":"2023-10-16T14:32:27.096942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:27.351413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.068555+0000","last_change":"2023-10-16T14:32:26.049197+0000","last_active":"2023-10-16T14:32:27.068555+0000","last_peered":"2023-10-16T14:32:27.068555+0000","last_clean":"2023-10-16T14:32:27.068555+0000","last_became_active":"2023-10-16T14:32:26.048884+0000","last_became_peered":"2023-10-16T14:32:26.048884+0000","last_unstale":"2023-10-16T14:32:27.068555+0000","last_undegraded":"2023-10-16T14:32:27.068555+0000","last_fullsized":"2023-10-16T14:32:27.068555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:15.191642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092959+0000","last_change":"2023-10-16T14:32:27.087915+0000","last_active":"2023-10-16T14:32:27.092959+0000","last_peered":"2023-10-16T14:32:27.092959+0000","last_clean":"2023-10-16T14:32:27.092959+0000","last_became_active":"2023-10-16T14:32:27.087589+0000","last_became_peered":"2023-10-16T14:32:27.087589+0000","last_unstale":"2023-10-16T14:32:27.092959+0000","last_undegraded":"2023-10-16T14:32:27.092959+0000","last_fullsized":"2023-10-16T14:32:27.092959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:44.403697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.062152+0000","last_change":"2023-10-16T14:32:26.060497+0000","last_active":"2023-10-16T14:32:27.062152+0000","last_peered":"2023-10-16T14:32:27.062152+0000","last_clean":"2023-10-16T14:32:27.062152+0000","last_became_active":"2023-10-16T14:32:26.060035+0000","last_became_peered":"2023-10-16T14:32:26.060035+0000","last_unstale":"2023-10-16T14:32:27.062152+0000","last_undegraded":"2023-10-16T14:32:27.062152+0000","last_fullsized":"2023-10-16T14:32:27.062152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:15.597365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091667+0000","last_change":"2023-10-16T14:32:27.085134+0000","last_active":"2023-10-16T14:32:27.091667+0000","last_peered":"2023-10-16T14:32:27.091667+0000","last_clean":"2023-10-16T14:32:27.091667+0000","last_became_active":"2023-10-16T14:32:27.084849+0000","last_became_peered":"2023-10-16T14:32:27.084849+0000","last_unstale":"2023-10-16T14:32:27.091667+0000","last_undegraded":"2023-10-16T14:32:27.091667+0000","last_fullsized":"2023-10-16T14:32:27.091667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:22.099583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.076702+0000","last_change":"2023-10-16T14:32:26.059821+0000","last_active":"2023-10-16T14:32:27.076702+0000","last_peered":"2023-10-16T14:32:27.076702+0000","last_clean":"2023-10-16T14:32:27.076702+0000","last_became_active":"2023-10-16T14:32:26.059512+0000","last_became_peered":"2023-10-16T14:32:26.059512+0000","last_unstale":"2023-10-16T14:32:27.076702+0000","last_undegraded":"2023-10-16T14:32:27.076702+0000","last_fullsized":"2023-10-16T14:32:27.076702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:44.017701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"71'161","reported_seq":392,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.060098+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:32:27.060098+0000","last_peered":"2023-10-16T14:32:27.060098+0000","last_clean":"2023-10-16T14:32:27.060098+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:32:27.060098+0000","last_undegraded":"2023-10-16T14:32:27.060098+0000","last_fullsized":"2023-10-16T14:32:27.060098+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41076,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330764,"statfs":{"total":96636764160,"available":96594702336,"internally_reserved":0,"allocated":1605632,"data_stored":1073130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0900000000000001,"15min":1.1559999999999999},"min":{"1min":0.755,"5min":0.64900000000000002,"15min":0.42799999999999999},"max":{"1min":1.833,"5min":1.833,"15min":1.9159999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.169,"15min":1.1519999999999999},"min":{"1min":0.69599999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4490000000000001,"5min":1.673,"15min":1.673},"last":1.2609999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.1879999999999999,"15min":1.1859999999999999},"min":{"1min":0.83599999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.5209999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.264,"5min":1.212,"15min":1.2310000000000001},"min":{"1min":0.78400000000000003,"5min":0.73799999999999999,"15min":0.67600000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":2.0219999999999998},"last":1.3300000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.319,"15min":1.3},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.63500000000000001},"max":{"1min":1.5740000000000001,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.512},{"interface":"front","average":{"1min":1.319,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.63400000000000001},"max":{"1min":1.669,"5min":1.7250000000000001,"15min":1.885},"last":1.216}]},{"osd":3,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3169999999999999,"15min":1.3},"min":{"1min":1.008,"5min":0.81799999999999995,"15min":0.75900000000000001},"max":{"1min":1.6000000000000001,"5min":1.746,"15min":2.048},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.3680000000000001,"15min":1.379},"min":{"1min":1.0880000000000001,"5min":0.80900000000000005,"15min":0.71899999999999997},"max":{"1min":1.768,"5min":2.1269999999999998,"15min":2.2509999999999999},"last":1.425}]},{"osd":4,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.361,"15min":1.365},"min":{"1min":1.0720000000000001,"5min":0.82299999999999995,"15min":0.67600000000000005},"max":{"1min":1.9239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.26,"15min":1.2649999999999999},"min":{"1min":0.85399999999999998,"5min":0.80800000000000005,"15min":0.496},"max":{"1min":1.643,"5min":2.0539999999999998,"15min":2.153},"last":1.379}]},{"osd":5,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.415,"15min":1.4139999999999999},"min":{"1min":0.94799999999999995,"5min":0.94399999999999995,"15min":0.61199999999999999},"max":{"1min":2.0049999999999999,"5min":2.2000000000000002,"15min":2.2530000000000001},"last":1.635},{"interface":"front","average":{"1min":1.508,"5min":1.464,"15min":1.494},"min":{"1min":1.048,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.1349999999999998,"5min":2.2450000000000001,"15min":2.2970000000000002},"last":1.1879999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.4490000000000001,"15min":1.4370000000000001},"min":{"1min":1.0589999999999999,"5min":1.048,"15min":0.63100000000000001},"max":{"1min":2.23,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.476},{"interface":"front","average":{"1min":1.488,"5min":1.4950000000000001,"15min":1.514},"min":{"1min":1.139,"5min":0.88700000000000001,"15min":0.59299999999999997},"max":{"1min":1.893,"5min":2.3260000000000001,"15min":2.427},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36328,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335512,"statfs":{"total":96636764160,"available":96599564288,"internally_reserved":0,"allocated":1003520,"data_stored":482602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.161,"15min":1.169},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.2160000000000002},"last":1.258},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2370000000000001,"15min":1.2310000000000001},"min":{"1min":0.81299999999999994,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.607,"5min":2.2709999999999999,"15min":3.863},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.252,"15min":1.2569999999999999},"min":{"1min":0.97299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.726,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.581},{"interface":"front","average":{"1min":1.302,"5min":1.321,"15min":1.3029999999999999},"min":{"1min":0.94599999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.667,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":0.94899999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.1830000000000001,"15min":1.214},"min":{"1min":0.98399999999999999,"5min":0.747,"15min":0.59899999999999998},"max":{"1min":1.5409999999999999,"5min":1.708,"15min":3.5800000000000001},"last":1.194},{"interface":"front","average":{"1min":1.389,"5min":1.3440000000000001,"15min":1.3600000000000001},"min":{"1min":0.98999999999999999,"5min":0.73599999999999999,"15min":0.70599999999999996},"max":{"1min":1.704,"5min":1.8799999999999999,"15min":2.7250000000000001},"last":1.52}]},{"osd":3,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.347,"15min":1.3520000000000001},"min":{"1min":0.97299999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.802,"5min":1.96,"15min":4.9790000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.319,"15min":1.3580000000000001},"min":{"1min":1.119,"5min":0.77200000000000002,"15min":0.71399999999999997},"max":{"1min":1.6439999999999999,"5min":1.913,"15min":5.4160000000000004},"last":1.5609999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2949999999999999,"15min":1.3129999999999999},"min":{"1min":0.79600000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8049999999999999,"5min":2.1070000000000002,"15min":5},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.367,"15min":1.385},"min":{"1min":1.0029999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.73,"5min":2.1789999999999998,"15min":5.4640000000000004},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.278,"15min":1.321},"min":{"1min":0.89600000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.859,"5min":1.859,"15min":3.9279999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4810000000000001,"15min":1.5029999999999999},"min":{"1min":1.111,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.8919999999999999,"5min":2.3330000000000002,"15min":3.6749999999999998},"last":1.387}]},{"osd":7,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5569999999999999,"15min":1.5669999999999999},"min":{"1min":1.1539999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.8939999999999999,"5min":2.0790000000000002,"15min":4.6859999999999999},"last":1.353},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.2130000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":4.883},"last":1.6319999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36328,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335512,"statfs":{"total":96636764160,"available":96599564288,"internally_reserved":0,"allocated":1011712,"data_stored":483254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.113,"15min":1.117},"min":{"1min":0.77600000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5649999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.2130000000000001,"15min":1.1779999999999999},"min":{"1min":0.91500000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.651,"5min":1.8340000000000001,"15min":2.2629999999999999},"last":1.127}]},{"osd":1,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1679999999999999,"15min":1.177},"min":{"1min":0.93999999999999995,"5min":0.65200000000000002,"15min":0.64600000000000002},"max":{"1min":1.5129999999999999,"5min":1.726,"15min":1.726},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.218,"15min":1.246},"min":{"1min":0.90400000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5600000000000001,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.2410000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2330000000000001,"15min":1.24},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.587,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.214},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.333,"15min":1.3580000000000001},"min":{"1min":1.1160000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.6359999999999999,"5min":2.1200000000000001,"15min":2.3500000000000001},"last":1.403}]},{"osd":3,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2629999999999999,"15min":1.2709999999999999},"min":{"1min":0.94399999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6220000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3819999999999999,"15min":1.3879999999999999},"min":{"1min":0.92300000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.6919999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.347}]},{"osd":4,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.74399999999999999,"5min":0.63900000000000001,"15min":0.54700000000000004},"max":{"1min":1.661,"5min":1.8069999999999999,"15min":2.863},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.514,"15min":1.51},"min":{"1min":0.85399999999999998,"5min":0.79600000000000004,"15min":0.61699999999999999},"max":{"1min":1.7609999999999999,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.4399999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3680000000000001,"15min":1.3879999999999999},"min":{"1min":0.84499999999999997,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.712,"5min":2.012,"15min":2.012},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.407,"15min":1.448},"min":{"1min":0.92800000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5900000000000001,"5min":2.0099999999999998,"15min":3.794},"last":1.3180000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5369999999999999,"15min":1.5569999999999999},"min":{"1min":1.0129999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.835,"5min":2.641,"15min":2.641},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5920000000000001,"15min":1.609},"min":{"1min":1.198,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8180000000000001,"5min":2.2010000000000001,"15min":2.415},"last":1.6259999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46060,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325780,"statfs":{"total":96636764160,"available":96589598720,"internally_reserved":0,"allocated":1015808,"data_stored":484214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1619999999999999,"15min":1.165},"min":{"1min":0.84899999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.748,"5min":1.881,"15min":5.367},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.135,"15min":1.1919999999999999},"min":{"1min":0.874,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3799999999999999,"5min":1.7589999999999999,"15min":6.3159999999999998},"last":0.93100000000000005}]},{"osd":1,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1679999999999999,"15min":1.173},"min":{"1min":0.85999999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5569999999999999,"5min":1.5880000000000001,"15min":1.78},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.238,"15min":1.2609999999999999},"min":{"1min":0.90600000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6100000000000001,"5min":1.782,"15min":5.4640000000000004},"last":1.179}]},{"osd":2,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.405,"15min":1.411},"min":{"1min":0.89600000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.821,"5min":1.821,"15min":5.617},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.355,"15min":1.3859999999999999},"min":{"1min":0.84799999999999998,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":6.391},"last":1.29}]},{"osd":3,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.304,"15min":1.341},"min":{"1min":1.0289999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.603,"5min":1.9330000000000001,"15min":7.4089999999999998},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.294,"15min":1.3},"min":{"1min":0.90300000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.6919999999999999,"5min":1.8899999999999999,"15min":2.0339999999999998},"last":1.1020000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3029999999999999,"15min":1.3069999999999999},"min":{"1min":1.0129999999999999,"5min":0.61499999999999999,"15min":0.48699999999999999},"max":{"1min":1.641,"5min":1.696,"15min":5.4109999999999996},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.325,"15min":1.3759999999999999},"min":{"1min":0.96099999999999997,"5min":0.76600000000000001,"15min":0.66100000000000003},"max":{"1min":1.5649999999999999,"5min":1.712,"15min":9.8239999999999998},"last":1.6140000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3169999999999999,"15min":1.292},"min":{"1min":0.83799999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.869,"5min":1.869,"15min":2.4729999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.468,"15min":1.4750000000000001},"min":{"1min":1.1160000000000001,"5min":0.71999999999999997,"15min":0.44700000000000001},"max":{"1min":1.6859999999999999,"5min":1.891,"15min":5.782},"last":1.7230000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4610000000000001,"15min":1.488},"min":{"1min":1.2050000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7769999999999999,"5min":1.9099999999999999,"15min":6.4379999999999997},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.4770000000000001,"15min":1.4690000000000001},"min":{"1min":1.014,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.966,"5min":2.0579999999999998,"15min":3.7799999999999998},"last":1.5409999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40496,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331344,"statfs":{"total":96636764160,"available":96595296256,"internally_reserved":0,"allocated":1011712,"data_stored":482658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.84799999999999998,"15min":0.90700000000000003},"min":{"1min":0.377,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.3919999999999999,"5min":1.4419999999999999,"15min":2.4420000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.04,"15min":1.03},"min":{"1min":0.746,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4810000000000001,"5min":1.913,"15min":1.982},"last":0.91600000000000004}]},{"osd":1,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.92000000000000004,"15min":0.94799999999999995},"min":{"1min":0.63900000000000001,"5min":0.437,"15min":0.32500000000000001},"max":{"1min":1.484,"5min":1.7390000000000001,"15min":2.149},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1319999999999999,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.314,"15min":0.314},"max":{"1min":1.474,"5min":1.6499999999999999,"15min":2.1600000000000001},"last":1.2210000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.276,"15min":1.329},"min":{"1min":0.86299999999999999,"5min":0.53800000000000003,"15min":0.44800000000000001},"max":{"1min":1.7050000000000001,"5min":1.895,"15min":2.1259999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4039999999999999,"15min":1.4410000000000001},"min":{"1min":0.78000000000000003,"5min":0.5,"15min":0.5},"max":{"1min":1.76,"5min":1.8660000000000001,"15min":2.6299999999999999},"last":0.88300000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.23,"15min":1.2729999999999999},"min":{"1min":0.94699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.552,"5min":1.6819999999999999,"15min":2.4529999999999998},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.3129999999999999,"15min":1.3500000000000001},"min":{"1min":1.0740000000000001,"5min":0.629,"15min":0.629},"max":{"1min":1.5920000000000001,"5min":1.909,"15min":2.3700000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5069999999999999,"15min":1.542},"min":{"1min":1.2370000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.7150000000000001,"5min":2.0070000000000001,"15min":2.552},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.421,"15min":1.4890000000000001},"min":{"1min":0.92500000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.841,"5min":2.3119999999999998,"15min":2.7589999999999999},"last":1.319}]},{"osd":6,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.3899999999999999,"15min":1.4319999999999999},"min":{"1min":0.89600000000000002,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.6639999999999999,"5min":2.181,"15min":2.3399999999999999},"last":1.26},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.476,"15min":1.5129999999999999},"min":{"1min":1.0229999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7529999999999999,"5min":1.9510000000000001,"15min":2.347},"last":1.5960000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.5289999999999999,"15min":1.601},"min":{"1min":1.036,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.952,"5min":2.395,"15min":2.8820000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5229999999999999,"15min":1.5649999999999999},"min":{"1min":0.83799999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.6910000000000001,"5min":2.1000000000000001,"15min":2.6179999999999999},"last":1.357}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44648,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327192,"statfs":{"total":96636764160,"available":96591044608,"internally_reserved":0,"allocated":1011712,"data_stored":482658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.879,"15min":0.876},"min":{"1min":0.63900000000000001,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":1.2549999999999999,"5min":1.556,"15min":2.472},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.141,"5min":1.0089999999999999,"15min":1.032},"min":{"1min":0.71899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.585,"5min":1.585,"15min":2.8879999999999999},"last":0.70299999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.98699999999999999,"15min":1.008},"min":{"1min":0.77700000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":3.234,"15min":3.234},"last":1.143},{"interface":"front","average":{"1min":1.173,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.91300000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4850000000000001,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.419}]},{"osd":3,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.236,"15min":1.2370000000000001},"min":{"1min":0.94299999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.29},{"interface":"front","average":{"1min":1.335,"5min":1.294,"15min":1.3},"min":{"1min":0.97399999999999998,"5min":0.57599999999999996,"15min":0.42499999999999999},"max":{"1min":1.6899999999999999,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.3080000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.452,"15min":1.431},"min":{"1min":1.0649999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.873,"5min":2.2869999999999999,"15min":2.5499999999999998},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.3879999999999999,"15min":1.373},"min":{"1min":1.1220000000000001,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.869,"5min":3.016,"15min":3.016},"last":1.2549999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.419,"15min":1.4239999999999999},"min":{"1min":1.169,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.3650000000000002,"5min":2.4100000000000001,"15min":3.25},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5629999999999999,"15min":1.55},"min":{"1min":1.0329999999999999,"5min":0.79200000000000004,"15min":0.78300000000000003},"max":{"1min":1.972,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.647}]},{"osd":6,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.4930000000000001,"15min":1.5009999999999999},"min":{"1min":1.0760000000000001,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.9950000000000001,"5min":3.0760000000000001,"15min":3.536},"last":1.371},{"interface":"front","average":{"1min":1.575,"5min":1.522,"15min":1.5369999999999999},"min":{"1min":1.1779999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.5230000000000001,"5min":2.5920000000000001,"15min":3.3929999999999998},"last":1.4610000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.5700000000000001,"15min":1.5900000000000001},"min":{"1min":1.075,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.0840000000000001,"5min":3.1899999999999999,"15min":5.2199999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.647,"15min":1.651},"min":{"1min":1.353,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4529999999999998,"5min":2.7519999999999998,"15min":3.496},"last":1.7010000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54372,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317468,"statfs":{"total":96636764160,"available":96581087232,"internally_reserved":0,"allocated":1007616,"data_stored":482692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.878,"15min":0.90800000000000003},"min":{"1min":0.73099999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.421,"5min":1.569,"15min":1.821},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.125,"15min":1.107},"min":{"1min":0.629,"5min":0.378,"15min":0.378},"max":{"1min":1.776,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0649999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.98999999999999999,"15min":0.99099999999999999},"min":{"1min":0.71599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.627,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.107,"15min":1.1579999999999999},"min":{"1min":0.69899999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5960000000000001,"5min":1.756,"15min":1.796},"last":1.288}]},{"osd":3,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.56299999999999994,"15min":0.52400000000000002},"max":{"1min":1.776,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.1870000000000001,"15min":1.2010000000000001},"min":{"1min":0.64700000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.756,"5min":1.8100000000000001,"15min":2.2050000000000001},"last":1.425}]},{"osd":4,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.95599999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.544,"5min":2.069,"15min":2.9820000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.387,"15min":1.423},"min":{"1min":1.0089999999999999,"5min":0.61399999999999999,"15min":0.59499999999999997},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":3.242},"last":1.607}]},{"osd":5,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4370000000000001,"15min":1.4379999999999999},"min":{"1min":1.073,"5min":0.623,"15min":0.623},"max":{"1min":1.9319999999999999,"5min":2.032,"15min":2.3690000000000002},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.415,"15min":1.4199999999999999},"min":{"1min":0.91100000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.869,"5min":2.1469999999999998,"15min":2.181},"last":1.202}]},{"osd":6,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.492,"15min":1.5409999999999999},"min":{"1min":0.98099999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0230000000000001,"5min":2.125,"15min":2.476},"last":1.714},{"interface":"front","average":{"1min":1.569,"5min":1.542,"15min":1.546},"min":{"1min":1.01,"5min":0.86299999999999999,"15min":0.73099999999999998},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.6160000000000001},"last":1.258}]},{"osd":7,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.625,"15min":1.6259999999999999},"min":{"1min":1.2150000000000001,"5min":0.90600000000000003,"15min":0.79300000000000004},"max":{"1min":2.1920000000000002,"5min":2.2639999999999998,"15min":2.5059999999999998},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.532,"15min":1.5580000000000001},"min":{"1min":1.054,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.069,"5min":2.1080000000000001,"15min":2.3410000000000002},"last":1.506}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36920,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334920,"statfs":{"total":96636764160,"available":96598958080,"internally_reserved":0,"allocated":1613824,"data_stored":1075164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.89500000000000002,"15min":0.94599999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42199999999999999},"max":{"1min":1.1759999999999999,"5min":1.464,"15min":2.2650000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1200000000000001,"15min":1.167},"min":{"1min":0.65600000000000003,"5min":0.497,"15min":0.36399999999999999},"max":{"1min":1.833,"5min":1.833,"15min":2.6030000000000002},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.242,"15min":1.2529999999999999},"min":{"1min":0.68600000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.7290000000000001,"5min":1.887,"15min":3.653},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.194,"5min":1.1910000000000001,"15min":1.2150000000000001},"min":{"1min":0.73399999999999999,"5min":0.626,"15min":0.47399999999999998},"max":{"1min":1.6819999999999999,"5min":2.012,"15min":3.1400000000000001},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.0129999999999999},"min":{"1min":0.68500000000000005,"5min":0.46600000000000003,"15min":0.40100000000000002},"max":{"1min":1.889,"5min":1.889,"15min":2.391},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2070000000000001,"15min":1.24},"min":{"1min":0.68600000000000005,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.6020000000000001,"5min":1.8660000000000001,"15min":2.79},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.29,"15min":1.306},"min":{"1min":0.89400000000000002,"5min":0.80100000000000005,"15min":0.61599999999999999},"max":{"1min":1.704,"5min":1.9259999999999999,"15min":2.7370000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.484,"5min":1.4570000000000001,"15min":1.468},"min":{"1min":0.99199999999999999,"5min":0.75800000000000001,"15min":0.65900000000000003},"max":{"1min":1.966,"5min":2.032,"15min":3.6190000000000002},"last":0.76500000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4099999999999999,"15min":1.4319999999999999},"min":{"1min":0.79000000000000004,"5min":0.78400000000000003,"15min":0.58099999999999996},"max":{"1min":2.0150000000000001,"5min":2.097,"15min":3.2370000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3280000000000001,"15min":1.373},"min":{"1min":0.96399999999999997,"5min":0.84899999999999998,"15min":0.67900000000000005},"max":{"1min":1.6040000000000001,"5min":1.7150000000000001,"15min":2.819},"last":0.89600000000000002}]},{"osd":6,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.5580000000000001,"15min":1.5840000000000001},"min":{"1min":1.042,"5min":0.83499999999999996,"15min":0.75},"max":{"1min":2.097,"5min":2.097,"15min":3.5779999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5049999999999999,"15min":1.5089999999999999},"min":{"1min":1.0609999999999999,"5min":0.98899999999999999,"15min":0.82199999999999995},"max":{"1min":1.877,"5min":2.1459999999999999,"15min":3.0979999999999999},"last":1.02}]},{"osd":7,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.575,"15min":1.593},"min":{"1min":1.024,"5min":0.73699999999999999,"15min":0.67100000000000004},"max":{"1min":1.974,"5min":2.2650000000000001,"15min":3.3340000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.4930000000000001,"15min":1.5700000000000001},"min":{"1min":0.98899999999999999,"5min":0.877,"15min":0.78400000000000003},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":6.5839999999999996},"last":1.22}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:32:31.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:32:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.932+0000 7faa010d7640 1 Processor -- start 2023-10-16T14:32:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.932+0000 7faa010d7640 1 -- start start 2023-10-16T14:32:31.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7faa010d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9fc0cdf30 0x7fa9fc071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7faa010d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc071fa0 0x7fa9fc06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7faa010d7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0d12f0 con 0x7fa9fc15cc80 2023-10-16T14:32:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7faa010d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9fc0d0500 con 0x7fa9fc0cdf30 2023-10-16T14:32:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7faa010d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc0723d0 con 0x7fa9fc071fa0 2023-10-16T14:32:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7faa008d6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc071fa0 0x7fa9fc06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7fa9fbfff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fa9fc15cc80 0x7fa9fc0cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:36618/0 (socket says 172.21.15.29:36618) 2023-10-16T14:32:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.934+0000 7fa9fb7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa9fc0cdf30 0x7fa9fc071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.935+0000 7fa9fbfff640 1 -- 172.21.15.29:0/526332858 learned_addr learned my addr 172.21.15.29:0/526332858 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.935+0000 7fa9fb7fe640 1 -- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc071fa0 msgr2=0x7fa9fc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.935+0000 7fa9fb7fe640 1 --2- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc071fa0 0x7fa9fc06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.935+0000 7fa9fb7fe640 1 -- 172.21.15.29:0/526332858 >> v1:172.21.15.170:6789/0 conn(0x7fa9fc15cc80 legacy=0x7fa9fc0cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.935+0000 7fa9fb7fe640 1 -- 172.21.15.29:0/526332858 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0d06e0 con 0x7fa9fc0cdf30 2023-10-16T14:32:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.937+0000 7fa9fb7fe640 1 --2- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc0cdf30 0x7fa9fc071a60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00097a0 tx=0x7fa9f002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f2cebeeffb8e84f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.937+0000 7fa9faffd640 1 -- 172.21.15.29:0/526332858 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f002d040 con 0x7fa9fc0cdf30 2023-10-16T14:32:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.937+0000 7fa9faffd640 1 -- 172.21.15.29:0/526332858 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f0031810 con 0x7fa9fc0cdf30 2023-10-16T14:32:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.937+0000 7fa9faffd640 1 -- 172.21.15.29:0/526332858 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0031ba0 con 0x7fa9fc0cdf30 2023-10-16T14:32:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 -- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc0cdf30 msgr2=0x7fa9fc071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 --2- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc0cdf30 0x7fa9fc071a60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00097a0 tx=0x7fa9f002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 -- 172.21.15.29:0/526332858 shutdown_connections 2023-10-16T14:32:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 --2- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc071fa0 0x7fa9fc06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 --2- 172.21.15.29:0/526332858 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc0cdf30 0x7fa9fc071a60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 -- 172.21.15.29:0/526332858 >> 172.21.15.29:0/526332858 conn(0x7fa9fc065ed0 msgr2=0x7fa9fc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 -- 172.21.15.29:0/526332858 shutdown_connections 2023-10-16T14:32:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.938+0000 7faa010d7640 1 -- 172.21.15.29:0/526332858 wait complete. 2023-10-16T14:32:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.940+0000 7faa010d7640 1 Processor -- start 2023-10-16T14:32:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.941+0000 7faa010d7640 1 -- start start 2023-10-16T14:32:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.942+0000 7faa010d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc071fa0 0x7fa9fc1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.942+0000 7faa010d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc0cdf30 0x7fa9fc153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.942+0000 7faa010d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 0x7fa9fc0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.942+0000 7faa010d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc0c5c50 con 0x7fa9fc15cc80 2023-10-16T14:32:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.942+0000 7faa010d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1526f0 con 0x7fa9fc071fa0 2023-10-16T14:32:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.942+0000 7faa010d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc154600 con 0x7fa9fc0cdf30 2023-10-16T14:32:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 0x7fa9fc0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 0x7fa9fc0b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56780/0 (socket says 172.21.15.29:56780) 2023-10-16T14:32:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 -- 172.21.15.29:0/370329612 learned_addr learned my addr 172.21.15.29:0/370329612 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7fa9fbfff640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc071fa0 0x7fa9fc1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7fa9fb7fe640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc0cdf30 0x7fa9fc153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 -- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc0cdf30 msgr2=0x7fa9fc153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc0cdf30 0x7fa9fc153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 -- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc071fa0 msgr2=0x7fa9fc1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc071fa0 0x7fa9fc1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7faa008d6640 1 -- 172.21.15.29:0/370329612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0009400 con 0x7fa9fc15cc80 2023-10-16T14:32:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.943+0000 7fa9fb7fe640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc0cdf30 0x7fa9fc153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.944+0000 7faa008d6640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 0x7fa9fc0b01d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e930 tx=0x7fa9f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.944+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4009690 con 0x7fa9fc15cc80 2023-10-16T14:32:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.944+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f400c980 con 0x7fa9fc15cc80 2023-10-16T14:32:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.944+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400cd30 con 0x7fa9fc15cc80 2023-10-16T14:32:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.944+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc0c5600 con 0x7fa9fc15cc80 2023-10-16T14:32:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.944+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc1541f0 con 0x7fa9fc15cc80 2023-10-16T14:32:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.945+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc0c5460 con 0x7fa9fc15cc80 2023-10-16T14:32:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.946+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa9f4013470 con 0x7fa9fc15cc80 2023-10-16T14:32:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.947+0000 7fa9f8ff9640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fa9e806af90 0x7fa9e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.947+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9f4089990 con 0x7fa9fc15cc80 2023-10-16T14:32:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.948+0000 7fa9fbfff640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fa9e806af90 0x7fa9e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.949+0000 7fa9fbfff640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fa9e806af90 0x7fa9e806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec005c00 tx=0x7fa9ec009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:31.951+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa9fc0c5460 con 0x7fa9fc15cc80 2023-10-16T14:32:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.142+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 --> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9fc1543e0 con 0x7fa9e806af90 2023-10-16T14:32:32.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.165+0000 7fa9f8ff9640 1 -- 172.21.15.29:0/370329612 <== mgr.5135 v2:172.21.15.170:6832/885795747 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286353 (secure 0 0 0) 0x7fa9ec00fdd0 con 0x7fa9e806af90 2023-10-16T14:32:32.173 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:32:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.170+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fa9e806af90 msgr2=0x7fa9e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.170+0000 7faa010d7640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fa9e806af90 0x7fa9e806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec005c00 tx=0x7fa9ec009040 comp rx=0 tx=0).stop 2023-10-16T14:32:32.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.170+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 msgr2=0x7fa9fc0b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.170+0000 7faa010d7640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 0x7fa9fc0b01d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e930 tx=0x7fa9f400edf0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 shutdown_connections 2023-10-16T14:32:32.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7fa9e806af90 0x7fa9e806d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9fc15cc80 0x7fa9fc0b01d0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa9fc0cdf30 0x7fa9fc153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 --2- 172.21.15.29:0/370329612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa9fc071fa0 0x7fa9fc1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 >> 172.21.15.29:0/370329612 conn(0x7fa9fc065ed0 msgr2=0x7fa9fc0b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:32.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 shutdown_connections 2023-10-16T14:32:32.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.171+0000 7faa010d7640 1 -- 172.21.15.29:0/370329612 wait complete. 2023-10-16T14:32:32.195 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T14:32:31.384062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336228,"kb_used_data":9064,"kb_used_omap":12,"kb_used_meta":327027,"kb_avail":754638492,"statfs":{"total":773094113280,"available":772749815808,"internally_reserved":0,"allocated":9281536,"data_stored":5046372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":334876217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002193"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060662+0000","last_change":"2023-10-16T14:32:27.096141+0000","last_active":"2023-10-16T14:32:28.060662+0000","last_peered":"2023-10-16T14:32:28.060662+0000","last_clean":"2023-10-16T14:32:28.060662+0000","last_became_active":"2023-10-16T14:32:27.095834+0000","last_became_peered":"2023-10-16T14:32:27.095834+0000","last_unstale":"2023-10-16T14:32:28.060662+0000","last_undegraded":"2023-10-16T14:32:28.060662+0000","last_fullsized":"2023-10-16T14:32:28.060662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:35.658898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.089893+0000","last_change":"2023-10-16T14:32:27.081367+0000","last_active":"2023-10-16T14:32:27.089893+0000","last_peered":"2023-10-16T14:32:27.089893+0000","last_clean":"2023-10-16T14:32:27.089893+0000","last_became_active":"2023-10-16T14:32:27.081081+0000","last_became_peered":"2023-10-16T14:32:27.081081+0000","last_unstale":"2023-10-16T14:32:27.089893+0000","last_undegraded":"2023-10-16T14:32:27.089893+0000","last_fullsized":"2023-10-16T14:32:27.089893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:45.554454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065378+0000","last_change":"2023-10-16T14:32:27.074983+0000","last_active":"2023-10-16T14:32:28.065378+0000","last_peered":"2023-10-16T14:32:28.065378+0000","last_clean":"2023-10-16T14:32:28.065378+0000","last_became_active":"2023-10-16T14:32:27.074448+0000","last_became_peered":"2023-10-16T14:32:27.074448+0000","last_unstale":"2023-10-16T14:32:28.065378+0000","last_undegraded":"2023-10-16T14:32:28.065378+0000","last_fullsized":"2023-10-16T14:32:28.065378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:07.700052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.097732+0000","last_change":"2023-10-16T14:32:27.097251+0000","last_active":"2023-10-16T14:32:27.097732+0000","last_peered":"2023-10-16T14:32:27.097732+0000","last_clean":"2023-10-16T14:32:27.097732+0000","last_became_active":"2023-10-16T14:32:27.096940+0000","last_became_peered":"2023-10-16T14:32:27.096940+0000","last_unstale":"2023-10-16T14:32:27.097732+0000","last_undegraded":"2023-10-16T14:32:27.097732+0000","last_fullsized":"2023-10-16T14:32:27.097732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:14.799841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060456+0000","last_change":"2023-10-16T14:32:27.096782+0000","last_active":"2023-10-16T14:32:28.060456+0000","last_peered":"2023-10-16T14:32:28.060456+0000","last_clean":"2023-10-16T14:32:28.060456+0000","last_became_active":"2023-10-16T14:32:27.092020+0000","last_became_peered":"2023-10-16T14:32:27.092020+0000","last_unstale":"2023-10-16T14:32:28.060456+0000","last_undegraded":"2023-10-16T14:32:28.060456+0000","last_fullsized":"2023-10-16T14:32:28.060456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:51.811557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096608+0000","last_change":"2023-10-16T14:32:27.093796+0000","last_active":"2023-10-16T14:32:27.096608+0000","last_peered":"2023-10-16T14:32:27.096608+0000","last_clean":"2023-10-16T14:32:27.096608+0000","last_became_active":"2023-10-16T14:32:27.093480+0000","last_became_peered":"2023-10-16T14:32:27.093480+0000","last_unstale":"2023-10-16T14:32:27.096608+0000","last_undegraded":"2023-10-16T14:32:27.096608+0000","last_fullsized":"2023-10-16T14:32:27.096608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:12.029035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062407+0000","last_change":"2023-10-16T14:32:27.091530+0000","last_active":"2023-10-16T14:32:28.062407+0000","last_peered":"2023-10-16T14:32:28.062407+0000","last_clean":"2023-10-16T14:32:28.062407+0000","last_became_active":"2023-10-16T14:32:27.090082+0000","last_became_peered":"2023-10-16T14:32:27.090082+0000","last_unstale":"2023-10-16T14:32:28.062407+0000","last_undegraded":"2023-10-16T14:32:28.062407+0000","last_fullsized":"2023-10-16T14:32:28.062407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:20.317503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.077897+0000","last_change":"2023-10-16T14:32:27.075711+0000","last_active":"2023-10-16T14:32:27.077897+0000","last_peered":"2023-10-16T14:32:27.077897+0000","last_clean":"2023-10-16T14:32:27.077897+0000","last_became_active":"2023-10-16T14:32:27.075295+0000","last_became_peered":"2023-10-16T14:32:27.075295+0000","last_unstale":"2023-10-16T14:32:27.077897+0000","last_undegraded":"2023-10-16T14:32:27.077897+0000","last_fullsized":"2023-10-16T14:32:27.077897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:47.249405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064444+0000","last_change":"2023-10-16T14:32:27.086675+0000","last_active":"2023-10-16T14:32:28.064444+0000","last_peered":"2023-10-16T14:32:28.064444+0000","last_clean":"2023-10-16T14:32:28.064444+0000","last_became_active":"2023-10-16T14:32:27.086405+0000","last_became_peered":"2023-10-16T14:32:27.086405+0000","last_unstale":"2023-10-16T14:32:28.064444+0000","last_undegraded":"2023-10-16T14:32:28.064444+0000","last_fullsized":"2023-10-16T14:32:28.064444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:13.491565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.098033+0000","last_change":"2023-10-16T14:32:27.095614+0000","last_active":"2023-10-16T14:32:27.098033+0000","last_peered":"2023-10-16T14:32:27.098033+0000","last_clean":"2023-10-16T14:32:27.098033+0000","last_became_active":"2023-10-16T14:32:27.093064+0000","last_became_peered":"2023-10-16T14:32:27.093064+0000","last_unstale":"2023-10-16T14:32:27.098033+0000","last_undegraded":"2023-10-16T14:32:27.098033+0000","last_fullsized":"2023-10-16T14:32:27.098033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:35.658898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096891+0000","last_change":"2023-10-16T14:32:27.086798+0000","last_active":"2023-10-16T14:32:27.096891+0000","last_peered":"2023-10-16T14:32:27.096891+0000","last_clean":"2023-10-16T14:32:27.096891+0000","last_became_active":"2023-10-16T14:32:27.086464+0000","last_became_peered":"2023-10-16T14:32:27.086464+0000","last_unstale":"2023-10-16T14:32:27.096891+0000","last_undegraded":"2023-10-16T14:32:27.096891+0000","last_fullsized":"2023-10-16T14:32:27.096891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:00.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092328+0000","last_change":"2023-10-16T14:32:27.087692+0000","last_active":"2023-10-16T14:32:27.092328+0000","last_peered":"2023-10-16T14:32:27.092328+0000","last_clean":"2023-10-16T14:32:27.092328+0000","last_became_active":"2023-10-16T14:32:27.086123+0000","last_became_peered":"2023-10-16T14:32:27.086123+0000","last_unstale":"2023-10-16T14:32:27.092328+0000","last_undegraded":"2023-10-16T14:32:27.092328+0000","last_fullsized":"2023-10-16T14:32:27.092328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:51.303159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091297+0000","last_change":"2023-10-16T14:32:27.089453+0000","last_active":"2023-10-16T14:32:27.091297+0000","last_peered":"2023-10-16T14:32:27.091297+0000","last_clean":"2023-10-16T14:32:27.091297+0000","last_became_active":"2023-10-16T14:32:27.088596+0000","last_became_peered":"2023-10-16T14:32:27.088596+0000","last_unstale":"2023-10-16T14:32:27.091297+0000","last_undegraded":"2023-10-16T14:32:27.091297+0000","last_fullsized":"2023-10-16T14:32:27.091297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:10.629767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062551+0000","last_change":"2023-10-16T14:32:27.089558+0000","last_active":"2023-10-16T14:32:28.062551+0000","last_peered":"2023-10-16T14:32:28.062551+0000","last_clean":"2023-10-16T14:32:28.062551+0000","last_became_active":"2023-10-16T14:32:27.089219+0000","last_became_peered":"2023-10-16T14:32:27.089219+0000","last_unstale":"2023-10-16T14:32:28.062551+0000","last_undegraded":"2023-10-16T14:32:28.062551+0000","last_fullsized":"2023-10-16T14:32:28.062551+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:12.296929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062472+0000","last_change":"2023-10-16T14:32:27.077318+0000","last_active":"2023-10-16T14:32:28.062472+0000","last_peered":"2023-10-16T14:32:28.062472+0000","last_clean":"2023-10-16T14:32:28.062472+0000","last_became_active":"2023-10-16T14:32:27.076872+0000","last_became_peered":"2023-10-16T14:32:27.076872+0000","last_unstale":"2023-10-16T14:32:28.062472+0000","last_undegraded":"2023-10-16T14:32:28.062472+0000","last_fullsized":"2023-10-16T14:32:28.062472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:56.523991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093714+0000","last_change":"2023-10-16T14:32:27.089017+0000","last_active":"2023-10-16T14:32:27.093714+0000","last_peered":"2023-10-16T14:32:27.093714+0000","last_clean":"2023-10-16T14:32:27.093714+0000","last_became_active":"2023-10-16T14:32:27.088729+0000","last_became_peered":"2023-10-16T14:32:27.088729+0000","last_unstale":"2023-10-16T14:32:27.093714+0000","last_undegraded":"2023-10-16T14:32:27.093714+0000","last_fullsized":"2023-10-16T14:32:27.093714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:22.218376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095134+0000","last_change":"2023-10-16T14:32:27.090448+0000","last_active":"2023-10-16T14:32:27.095134+0000","last_peered":"2023-10-16T14:32:27.095134+0000","last_clean":"2023-10-16T14:32:27.095134+0000","last_became_active":"2023-10-16T14:32:27.090108+0000","last_became_peered":"2023-10-16T14:32:27.090108+0000","last_unstale":"2023-10-16T14:32:27.095134+0000","last_undegraded":"2023-10-16T14:32:27.095134+0000","last_fullsized":"2023-10-16T14:32:27.095134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:32.074873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065957+0000","last_change":"2023-10-16T14:32:27.088139+0000","last_active":"2023-10-16T14:32:28.065957+0000","last_peered":"2023-10-16T14:32:28.065957+0000","last_clean":"2023-10-16T14:32:28.065957+0000","last_became_active":"2023-10-16T14:32:27.086923+0000","last_became_peered":"2023-10-16T14:32:27.086923+0000","last_unstale":"2023-10-16T14:32:28.065957+0000","last_undegraded":"2023-10-16T14:32:28.065957+0000","last_fullsized":"2023-10-16T14:32:28.065957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:16.613233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060300+0000","last_change":"2023-10-16T14:32:27.093558+0000","last_active":"2023-10-16T14:32:28.060300+0000","last_peered":"2023-10-16T14:32:28.060300+0000","last_clean":"2023-10-16T14:32:28.060300+0000","last_became_active":"2023-10-16T14:32:27.093251+0000","last_became_peered":"2023-10-16T14:32:27.093251+0000","last_unstale":"2023-10-16T14:32:28.060300+0000","last_undegraded":"2023-10-16T14:32:28.060300+0000","last_fullsized":"2023-10-16T14:32:28.060300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:46.969213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.089637+0000","last_change":"2023-10-16T14:32:27.083171+0000","last_active":"2023-10-16T14:32:27.089637+0000","last_peered":"2023-10-16T14:32:27.089637+0000","last_clean":"2023-10-16T14:32:27.089637+0000","last_became_active":"2023-10-16T14:32:27.082464+0000","last_became_peered":"2023-10-16T14:32:27.082464+0000","last_unstale":"2023-10-16T14:32:27.089637+0000","last_undegraded":"2023-10-16T14:32:27.089637+0000","last_fullsized":"2023-10-16T14:32:27.089637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:06.158536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064991+0000","last_change":"2023-10-16T14:32:27.085600+0000","last_active":"2023-10-16T14:32:28.064991+0000","last_peered":"2023-10-16T14:32:28.064991+0000","last_clean":"2023-10-16T14:32:28.064991+0000","last_became_active":"2023-10-16T14:32:27.085296+0000","last_became_peered":"2023-10-16T14:32:27.085296+0000","last_unstale":"2023-10-16T14:32:28.064991+0000","last_undegraded":"2023-10-16T14:32:28.064991+0000","last_fullsized":"2023-10-16T14:32:28.064991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:10.836410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095408+0000","last_change":"2023-10-16T14:32:27.094503+0000","last_active":"2023-10-16T14:32:27.095408+0000","last_peered":"2023-10-16T14:32:27.095408+0000","last_clean":"2023-10-16T14:32:27.095408+0000","last_became_active":"2023-10-16T14:32:27.093470+0000","last_became_peered":"2023-10-16T14:32:27.093470+0000","last_unstale":"2023-10-16T14:32:27.095408+0000","last_undegraded":"2023-10-16T14:32:27.095408+0000","last_fullsized":"2023-10-16T14:32:27.095408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:41.739097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063959+0000","last_change":"2023-10-16T14:32:27.074556+0000","last_active":"2023-10-16T14:32:28.063959+0000","last_peered":"2023-10-16T14:32:28.063959+0000","last_clean":"2023-10-16T14:32:28.063959+0000","last_became_active":"2023-10-16T14:32:27.073186+0000","last_became_peered":"2023-10-16T14:32:27.073186+0000","last_unstale":"2023-10-16T14:32:28.063959+0000","last_undegraded":"2023-10-16T14:32:28.063959+0000","last_fullsized":"2023-10-16T14:32:28.063959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:45.554454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.090364+0000","last_change":"2023-10-16T14:32:27.086553+0000","last_active":"2023-10-16T14:32:27.090364+0000","last_peered":"2023-10-16T14:32:27.090364+0000","last_clean":"2023-10-16T14:32:27.090364+0000","last_became_active":"2023-10-16T14:32:27.084756+0000","last_became_peered":"2023-10-16T14:32:27.084756+0000","last_unstale":"2023-10-16T14:32:27.090364+0000","last_undegraded":"2023-10-16T14:32:27.090364+0000","last_fullsized":"2023-10-16T14:32:27.090364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:14.710888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060779+0000","last_change":"2023-10-16T14:32:27.098329+0000","last_active":"2023-10-16T14:32:28.060779+0000","last_peered":"2023-10-16T14:32:28.060779+0000","last_clean":"2023-10-16T14:32:28.060779+0000","last_became_active":"2023-10-16T14:32:27.098014+0000","last_became_peered":"2023-10-16T14:32:27.098014+0000","last_unstale":"2023-10-16T14:32:28.060779+0000","last_undegraded":"2023-10-16T14:32:28.060779+0000","last_fullsized":"2023-10-16T14:32:28.060779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:51.476382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093033+0000","last_change":"2023-10-16T14:32:27.090198+0000","last_active":"2023-10-16T14:32:27.093033+0000","last_peered":"2023-10-16T14:32:27.093033+0000","last_clean":"2023-10-16T14:32:27.093033+0000","last_became_active":"2023-10-16T14:32:27.089922+0000","last_became_peered":"2023-10-16T14:32:27.089922+0000","last_unstale":"2023-10-16T14:32:27.093033+0000","last_undegraded":"2023-10-16T14:32:27.093033+0000","last_fullsized":"2023-10-16T14:32:27.093033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:29.230685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091719+0000","last_change":"2023-10-16T14:32:27.088490+0000","last_active":"2023-10-16T14:32:27.091719+0000","last_peered":"2023-10-16T14:32:27.091719+0000","last_clean":"2023-10-16T14:32:27.091719+0000","last_became_active":"2023-10-16T14:32:27.088165+0000","last_became_peered":"2023-10-16T14:32:27.088165+0000","last_unstale":"2023-10-16T14:32:27.091719+0000","last_undegraded":"2023-10-16T14:32:27.091719+0000","last_fullsized":"2023-10-16T14:32:27.091719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:13.587912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060555+0000","last_change":"2023-10-16T14:32:27.095953+0000","last_active":"2023-10-16T14:32:28.060555+0000","last_peered":"2023-10-16T14:32:28.060555+0000","last_clean":"2023-10-16T14:32:28.060555+0000","last_became_active":"2023-10-16T14:32:27.095640+0000","last_became_peered":"2023-10-16T14:32:27.095640+0000","last_unstale":"2023-10-16T14:32:28.060555+0000","last_undegraded":"2023-10-16T14:32:28.060555+0000","last_fullsized":"2023-10-16T14:32:28.060555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:00.182919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091755+0000","last_change":"2023-10-16T14:32:27.086578+0000","last_active":"2023-10-16T14:32:27.091755+0000","last_peered":"2023-10-16T14:32:27.091755+0000","last_clean":"2023-10-16T14:32:27.091755+0000","last_became_active":"2023-10-16T14:32:27.086253+0000","last_became_peered":"2023-10-16T14:32:27.086253+0000","last_unstale":"2023-10-16T14:32:27.091755+0000","last_undegraded":"2023-10-16T14:32:27.091755+0000","last_fullsized":"2023-10-16T14:32:27.091755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:18.755495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064388+0000","last_change":"2023-10-16T14:32:27.072284+0000","last_active":"2023-10-16T14:32:28.064388+0000","last_peered":"2023-10-16T14:32:28.064388+0000","last_clean":"2023-10-16T14:32:28.064388+0000","last_became_active":"2023-10-16T14:32:27.070929+0000","last_became_peered":"2023-10-16T14:32:27.070929+0000","last_unstale":"2023-10-16T14:32:28.064388+0000","last_undegraded":"2023-10-16T14:32:28.064388+0000","last_fullsized":"2023-10-16T14:32:28.064388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:18.921454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.060106+0000","last_change":"2023-10-16T14:32:27.089824+0000","last_active":"2023-10-16T14:32:28.060106+0000","last_peered":"2023-10-16T14:32:28.060106+0000","last_clean":"2023-10-16T14:32:28.060106+0000","last_became_active":"2023-10-16T14:32:27.089472+0000","last_became_peered":"2023-10-16T14:32:27.089472+0000","last_unstale":"2023-10-16T14:32:28.060106+0000","last_undegraded":"2023-10-16T14:32:28.060106+0000","last_fullsized":"2023-10-16T14:32:28.060106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:41.162259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099336+0000","last_change":"2023-10-16T14:32:27.097790+0000","last_active":"2023-10-16T14:32:27.099336+0000","last_peered":"2023-10-16T14:32:27.099336+0000","last_clean":"2023-10-16T14:32:27.099336+0000","last_became_active":"2023-10-16T14:32:27.096989+0000","last_became_peered":"2023-10-16T14:32:27.096989+0000","last_unstale":"2023-10-16T14:32:27.099336+0000","last_undegraded":"2023-10-16T14:32:27.099336+0000","last_fullsized":"2023-10-16T14:32:27.099336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:00:32.174270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095548+0000","last_change":"2023-10-16T14:32:27.092125+0000","last_active":"2023-10-16T14:32:27.095548+0000","last_peered":"2023-10-16T14:32:27.095548+0000","last_clean":"2023-10-16T14:32:27.095548+0000","last_became_active":"2023-10-16T14:32:27.090135+0000","last_became_peered":"2023-10-16T14:32:27.090135+0000","last_unstale":"2023-10-16T14:32:27.095548+0000","last_undegraded":"2023-10-16T14:32:27.095548+0000","last_fullsized":"2023-10-16T14:32:27.095548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:34.589282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.194271+0000","last_change":"2023-10-16T14:32:26.066926+0000","last_active":"2023-10-16T14:32:27.194271+0000","last_peered":"2023-10-16T14:32:27.194271+0000","last_clean":"2023-10-16T14:32:27.194271+0000","last_became_active":"2023-10-16T14:32:26.061602+0000","last_became_peered":"2023-10-16T14:32:26.061602+0000","last_unstale":"2023-10-16T14:32:27.194271+0000","last_undegraded":"2023-10-16T14:32:27.194271+0000","last_fullsized":"2023-10-16T14:32:27.194271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:49.199887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096705+0000","last_change":"2023-10-16T14:32:27.088114+0000","last_active":"2023-10-16T14:32:27.096705+0000","last_peered":"2023-10-16T14:32:27.096705+0000","last_clean":"2023-10-16T14:32:27.096705+0000","last_became_active":"2023-10-16T14:32:27.087807+0000","last_became_peered":"2023-10-16T14:32:27.087807+0000","last_unstale":"2023-10-16T14:32:27.096705+0000","last_undegraded":"2023-10-16T14:32:27.096705+0000","last_fullsized":"2023-10-16T14:32:27.096705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:52.106751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.190857+0000","last_change":"2023-10-16T14:32:26.069056+0000","last_active":"2023-10-16T14:32:27.190857+0000","last_peered":"2023-10-16T14:32:27.190857+0000","last_clean":"2023-10-16T14:32:27.190857+0000","last_became_active":"2023-10-16T14:32:26.068602+0000","last_became_peered":"2023-10-16T14:32:26.068602+0000","last_unstale":"2023-10-16T14:32:27.190857+0000","last_undegraded":"2023-10-16T14:32:27.190857+0000","last_fullsized":"2023-10-16T14:32:27.190857+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:17.905586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093702+0000","last_change":"2023-10-16T14:32:27.091506+0000","last_active":"2023-10-16T14:32:27.093702+0000","last_peered":"2023-10-16T14:32:27.093702+0000","last_clean":"2023-10-16T14:32:27.093702+0000","last_became_active":"2023-10-16T14:32:27.091162+0000","last_became_peered":"2023-10-16T14:32:27.091162+0000","last_unstale":"2023-10-16T14:32:27.093702+0000","last_undegraded":"2023-10-16T14:32:27.093702+0000","last_fullsized":"2023-10-16T14:32:27.093702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:25.686068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.193963+0000","last_change":"2023-10-16T14:32:26.044577+0000","last_active":"2023-10-16T14:32:27.193963+0000","last_peered":"2023-10-16T14:32:27.193963+0000","last_clean":"2023-10-16T14:32:27.193963+0000","last_became_active":"2023-10-16T14:32:26.044320+0000","last_became_peered":"2023-10-16T14:32:26.044320+0000","last_unstale":"2023-10-16T14:32:27.193963+0000","last_undegraded":"2023-10-16T14:32:27.193963+0000","last_fullsized":"2023-10-16T14:32:27.193963+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:33.221898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.087769+0000","last_change":"2023-10-16T14:32:27.085630+0000","last_active":"2023-10-16T14:32:27.087769+0000","last_peered":"2023-10-16T14:32:27.087769+0000","last_clean":"2023-10-16T14:32:27.087769+0000","last_became_active":"2023-10-16T14:32:27.085299+0000","last_became_peered":"2023-10-16T14:32:27.085299+0000","last_unstale":"2023-10-16T14:32:27.087769+0000","last_undegraded":"2023-10-16T14:32:27.087769+0000","last_fullsized":"2023-10-16T14:32:27.087769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:38.346197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.191829+0000","last_change":"2023-10-16T14:32:26.058393+0000","last_active":"2023-10-16T14:32:27.191829+0000","last_peered":"2023-10-16T14:32:27.191829+0000","last_clean":"2023-10-16T14:32:27.191829+0000","last_became_active":"2023-10-16T14:32:26.058104+0000","last_became_peered":"2023-10-16T14:32:26.058104+0000","last_unstale":"2023-10-16T14:32:27.191829+0000","last_undegraded":"2023-10-16T14:32:27.191829+0000","last_fullsized":"2023-10-16T14:32:27.191829+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:06.321202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.095263+0000","last_change":"2023-10-16T14:32:27.093577+0000","last_active":"2023-10-16T14:32:27.095263+0000","last_peered":"2023-10-16T14:32:27.095263+0000","last_clean":"2023-10-16T14:32:27.095263+0000","last_became_active":"2023-10-16T14:32:27.092940+0000","last_became_peered":"2023-10-16T14:32:27.092940+0000","last_unstale":"2023-10-16T14:32:27.095263+0000","last_undegraded":"2023-10-16T14:32:27.095263+0000","last_fullsized":"2023-10-16T14:32:27.095263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:30.968363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"83'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063085+0000","last_change":"2023-10-16T14:32:26.063887+0000","last_active":"2023-10-16T14:32:28.063085+0000","last_peered":"2023-10-16T14:32:28.063085+0000","last_clean":"2023-10-16T14:32:28.063085+0000","last_became_active":"2023-10-16T14:32:26.063169+0000","last_became_peered":"2023-10-16T14:32:26.063169+0000","last_unstale":"2023-10-16T14:32:28.063085+0000","last_undegraded":"2023-10-16T14:32:28.063085+0000","last_fullsized":"2023-10-16T14:32:28.063085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:45.139919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.089867+0000","last_change":"2023-10-16T14:32:27.084249+0000","last_active":"2023-10-16T14:32:27.089867+0000","last_peered":"2023-10-16T14:32:27.089867+0000","last_clean":"2023-10-16T14:32:27.089867+0000","last_became_active":"2023-10-16T14:32:27.083857+0000","last_became_peered":"2023-10-16T14:32:27.083857+0000","last_unstale":"2023-10-16T14:32:27.089867+0000","last_undegraded":"2023-10-16T14:32:27.089867+0000","last_fullsized":"2023-10-16T14:32:27.089867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:39:45.275937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.193183+0000","last_change":"2023-10-16T14:32:26.068093+0000","last_active":"2023-10-16T14:32:27.193183+0000","last_peered":"2023-10-16T14:32:27.193183+0000","last_clean":"2023-10-16T14:32:27.193183+0000","last_became_active":"2023-10-16T14:32:26.067766+0000","last_became_peered":"2023-10-16T14:32:26.067766+0000","last_unstale":"2023-10-16T14:32:27.193183+0000","last_undegraded":"2023-10-16T14:32:27.193183+0000","last_fullsized":"2023-10-16T14:32:27.193183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:56.648060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.088923+0000","last_change":"2023-10-16T14:32:27.084862+0000","last_active":"2023-10-16T14:32:27.088923+0000","last_peered":"2023-10-16T14:32:27.088923+0000","last_clean":"2023-10-16T14:32:27.088923+0000","last_became_active":"2023-10-16T14:32:27.084323+0000","last_became_peered":"2023-10-16T14:32:27.084323+0000","last_unstale":"2023-10-16T14:32:27.088923+0000","last_undegraded":"2023-10-16T14:32:27.088923+0000","last_fullsized":"2023-10-16T14:32:27.088923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:45.041903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.082792+0000","last_change":"2023-10-16T14:32:26.064084+0000","last_active":"2023-10-16T14:32:27.082792+0000","last_peered":"2023-10-16T14:32:27.082792+0000","last_clean":"2023-10-16T14:32:27.082792+0000","last_became_active":"2023-10-16T14:32:26.063323+0000","last_became_peered":"2023-10-16T14:32:26.063323+0000","last_unstale":"2023-10-16T14:32:27.082792+0000","last_undegraded":"2023-10-16T14:32:27.082792+0000","last_fullsized":"2023-10-16T14:32:27.082792+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:41.572256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092980+0000","last_change":"2023-10-16T14:32:27.090801+0000","last_active":"2023-10-16T14:32:27.092980+0000","last_peered":"2023-10-16T14:32:27.092980+0000","last_clean":"2023-10-16T14:32:27.092980+0000","last_became_active":"2023-10-16T14:32:27.090208+0000","last_became_peered":"2023-10-16T14:32:27.090208+0000","last_unstale":"2023-10-16T14:32:27.092980+0000","last_undegraded":"2023-10-16T14:32:27.092980+0000","last_fullsized":"2023-10-16T14:32:27.092980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:30.455378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062427+0000","last_change":"2023-10-16T14:32:26.071347+0000","last_active":"2023-10-16T14:32:28.062427+0000","last_peered":"2023-10-16T14:32:28.062427+0000","last_clean":"2023-10-16T14:32:28.062427+0000","last_became_active":"2023-10-16T14:32:26.071016+0000","last_became_peered":"2023-10-16T14:32:26.071016+0000","last_unstale":"2023-10-16T14:32:28.062427+0000","last_undegraded":"2023-10-16T14:32:28.062427+0000","last_fullsized":"2023-10-16T14:32:28.062427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:53.950061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065120+0000","last_change":"2023-10-16T14:32:27.085538+0000","last_active":"2023-10-16T14:32:28.065120+0000","last_peered":"2023-10-16T14:32:28.065120+0000","last_clean":"2023-10-16T14:32:28.065120+0000","last_became_active":"2023-10-16T14:32:27.085251+0000","last_became_peered":"2023-10-16T14:32:27.085251+0000","last_unstale":"2023-10-16T14:32:28.065120+0000","last_undegraded":"2023-10-16T14:32:28.065120+0000","last_fullsized":"2023-10-16T14:32:28.065120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:50.770169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065736+0000","last_change":"2023-10-16T14:32:26.058663+0000","last_active":"2023-10-16T14:32:28.065736+0000","last_peered":"2023-10-16T14:32:28.065736+0000","last_clean":"2023-10-16T14:32:28.065736+0000","last_became_active":"2023-10-16T14:32:26.058398+0000","last_became_peered":"2023-10-16T14:32:26.058398+0000","last_unstale":"2023-10-16T14:32:28.065736+0000","last_undegraded":"2023-10-16T14:32:28.065736+0000","last_fullsized":"2023-10-16T14:32:28.065736+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:31.059005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.066080+0000","last_change":"2023-10-16T14:32:27.078718+0000","last_active":"2023-10-16T14:32:28.066080+0000","last_peered":"2023-10-16T14:32:28.066080+0000","last_clean":"2023-10-16T14:32:28.066080+0000","last_became_active":"2023-10-16T14:32:27.078237+0000","last_became_peered":"2023-10-16T14:32:27.078237+0000","last_unstale":"2023-10-16T14:32:28.066080+0000","last_undegraded":"2023-10-16T14:32:28.066080+0000","last_fullsized":"2023-10-16T14:32:28.066080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:22.099583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.060278+0000","last_change":"2023-10-16T14:32:26.065276+0000","last_active":"2023-10-16T14:32:27.060278+0000","last_peered":"2023-10-16T14:32:27.060278+0000","last_clean":"2023-10-16T14:32:27.060278+0000","last_became_active":"2023-10-16T14:32:26.064996+0000","last_became_peered":"2023-10-16T14:32:26.064996+0000","last_unstale":"2023-10-16T14:32:27.060278+0000","last_undegraded":"2023-10-16T14:32:27.060278+0000","last_fullsized":"2023-10-16T14:32:27.060278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:06.684184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.094952+0000","last_change":"2023-10-16T14:32:27.084668+0000","last_active":"2023-10-16T14:32:27.094952+0000","last_peered":"2023-10-16T14:32:27.094952+0000","last_clean":"2023-10-16T14:32:27.094952+0000","last_became_active":"2023-10-16T14:32:27.084290+0000","last_became_peered":"2023-10-16T14:32:27.084290+0000","last_unstale":"2023-10-16T14:32:27.094952+0000","last_undegraded":"2023-10-16T14:32:27.094952+0000","last_fullsized":"2023-10-16T14:32:27.094952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:51.811557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"83'3","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.066134+0000","last_change":"2023-10-16T14:32:26.059570+0000","last_active":"2023-10-16T14:32:28.066134+0000","last_peered":"2023-10-16T14:32:28.066134+0000","last_clean":"2023-10-16T14:32:28.066134+0000","last_became_active":"2023-10-16T14:32:26.059227+0000","last_became_peered":"2023-10-16T14:32:26.059227+0000","last_unstale":"2023-10-16T14:32:28.066134+0000","last_undegraded":"2023-10-16T14:32:28.066134+0000","last_fullsized":"2023-10-16T14:32:28.066134+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:43.387829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065737+0000","last_change":"2023-10-16T14:32:27.080844+0000","last_active":"2023-10-16T14:32:28.065737+0000","last_peered":"2023-10-16T14:32:28.065737+0000","last_clean":"2023-10-16T14:32:28.065737+0000","last_became_active":"2023-10-16T14:32:27.080535+0000","last_became_peered":"2023-10-16T14:32:27.080535+0000","last_unstale":"2023-10-16T14:32:28.065737+0000","last_undegraded":"2023-10-16T14:32:28.065737+0000","last_fullsized":"2023-10-16T14:32:28.065737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:36.035450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"83'6","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063727+0000","last_change":"2023-10-16T14:32:26.070854+0000","last_active":"2023-10-16T14:32:28.063727+0000","last_peered":"2023-10-16T14:32:28.063727+0000","last_clean":"2023-10-16T14:32:28.063727+0000","last_became_active":"2023-10-16T14:32:26.070511+0000","last_became_peered":"2023-10-16T14:32:26.070511+0000","last_unstale":"2023-10-16T14:32:28.063727+0000","last_undegraded":"2023-10-16T14:32:28.063727+0000","last_fullsized":"2023-10-16T14:32:28.063727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:13.898978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099773+0000","last_change":"2023-10-16T14:32:27.099060+0000","last_active":"2023-10-16T14:32:27.099773+0000","last_peered":"2023-10-16T14:32:27.099773+0000","last_clean":"2023-10-16T14:32:27.099773+0000","last_became_active":"2023-10-16T14:32:27.098747+0000","last_became_peered":"2023-10-16T14:32:27.098747+0000","last_unstale":"2023-10-16T14:32:27.099773+0000","last_undegraded":"2023-10-16T14:32:27.099773+0000","last_fullsized":"2023-10-16T14:32:27.099773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:34.934405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.194627+0000","last_change":"2023-10-16T14:32:26.066908+0000","last_active":"2023-10-16T14:32:27.194627+0000","last_peered":"2023-10-16T14:32:27.194627+0000","last_clean":"2023-10-16T14:32:27.194627+0000","last_became_active":"2023-10-16T14:32:26.066342+0000","last_became_peered":"2023-10-16T14:32:26.066342+0000","last_unstale":"2023-10-16T14:32:27.194627+0000","last_undegraded":"2023-10-16T14:32:27.194627+0000","last_fullsized":"2023-10-16T14:32:27.194627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:16.461838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063933+0000","last_change":"2023-10-16T14:32:27.089165+0000","last_active":"2023-10-16T14:32:28.063933+0000","last_peered":"2023-10-16T14:32:28.063933+0000","last_clean":"2023-10-16T14:32:28.063933+0000","last_became_active":"2023-10-16T14:32:27.088890+0000","last_became_peered":"2023-10-16T14:32:27.088890+0000","last_unstale":"2023-10-16T14:32:28.063933+0000","last_undegraded":"2023-10-16T14:32:28.063933+0000","last_fullsized":"2023-10-16T14:32:28.063933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:35.136365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.072560+0000","last_change":"2023-10-16T14:32:26.063177+0000","last_active":"2023-10-16T14:32:27.072560+0000","last_peered":"2023-10-16T14:32:27.072560+0000","last_clean":"2023-10-16T14:32:27.072560+0000","last_became_active":"2023-10-16T14:32:26.062887+0000","last_became_peered":"2023-10-16T14:32:26.062887+0000","last_unstale":"2023-10-16T14:32:27.072560+0000","last_undegraded":"2023-10-16T14:32:27.072560+0000","last_fullsized":"2023-10-16T14:32:27.072560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:24.710141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.062211+0000","last_change":"2023-10-16T14:32:27.075554+0000","last_active":"2023-10-16T14:32:28.062211+0000","last_peered":"2023-10-16T14:32:28.062211+0000","last_clean":"2023-10-16T14:32:28.062211+0000","last_became_active":"2023-10-16T14:32:27.075120+0000","last_became_peered":"2023-10-16T14:32:27.075120+0000","last_unstale":"2023-10-16T14:32:28.062211+0000","last_undegraded":"2023-10-16T14:32:28.062211+0000","last_fullsized":"2023-10-16T14:32:28.062211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.841791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.082920+0000","last_change":"2023-10-16T14:32:26.052648+0000","last_active":"2023-10-16T14:32:27.082920+0000","last_peered":"2023-10-16T14:32:27.082920+0000","last_clean":"2023-10-16T14:32:27.082920+0000","last_became_active":"2023-10-16T14:32:26.052247+0000","last_became_peered":"2023-10-16T14:32:26.052247+0000","last_unstale":"2023-10-16T14:32:27.082920+0000","last_undegraded":"2023-10-16T14:32:27.082920+0000","last_fullsized":"2023-10-16T14:32:27.082920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:43.863681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.064335+0000","last_change":"2023-10-16T14:32:27.082795+0000","last_active":"2023-10-16T14:32:28.064335+0000","last_peered":"2023-10-16T14:32:28.064335+0000","last_clean":"2023-10-16T14:32:28.064335+0000","last_became_active":"2023-10-16T14:32:27.082488+0000","last_became_peered":"2023-10-16T14:32:27.082488+0000","last_unstale":"2023-10-16T14:32:28.064335+0000","last_undegraded":"2023-10-16T14:32:28.064335+0000","last_fullsized":"2023-10-16T14:32:28.064335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:18.755495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"83'8","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063754+0000","last_change":"2023-10-16T14:32:26.062317+0000","last_active":"2023-10-16T14:32:28.063754+0000","last_peered":"2023-10-16T14:32:28.063754+0000","last_clean":"2023-10-16T14:32:28.063754+0000","last_became_active":"2023-10-16T14:32:26.061519+0000","last_became_peered":"2023-10-16T14:32:26.061519+0000","last_unstale":"2023-10-16T14:32:28.063754+0000","last_undegraded":"2023-10-16T14:32:28.063754+0000","last_fullsized":"2023-10-16T14:32:28.063754+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:06.321202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065800+0000","last_change":"2023-10-16T14:32:27.078888+0000","last_active":"2023-10-16T14:32:28.065800+0000","last_peered":"2023-10-16T14:32:28.065800+0000","last_clean":"2023-10-16T14:32:28.065800+0000","last_became_active":"2023-10-16T14:32:27.078598+0000","last_became_peered":"2023-10-16T14:32:27.078598+0000","last_unstale":"2023-10-16T14:32:28.065800+0000","last_undegraded":"2023-10-16T14:32:28.065800+0000","last_fullsized":"2023-10-16T14:32:28.065800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:47.249405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.191515+0000","last_change":"2023-10-16T14:32:26.065271+0000","last_active":"2023-10-16T14:32:27.191515+0000","last_peered":"2023-10-16T14:32:27.191515+0000","last_clean":"2023-10-16T14:32:27.191515+0000","last_became_active":"2023-10-16T14:32:26.064844+0000","last_became_peered":"2023-10-16T14:32:26.064844+0000","last_unstale":"2023-10-16T14:32:27.191515+0000","last_undegraded":"2023-10-16T14:32:27.191515+0000","last_fullsized":"2023-10-16T14:32:27.191515+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:34.471465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099757+0000","last_change":"2023-10-16T14:32:27.099055+0000","last_active":"2023-10-16T14:32:27.099757+0000","last_peered":"2023-10-16T14:32:27.099757+0000","last_clean":"2023-10-16T14:32:27.099757+0000","last_became_active":"2023-10-16T14:32:27.098588+0000","last_became_peered":"2023-10-16T14:32:27.098588+0000","last_unstale":"2023-10-16T14:32:27.099757+0000","last_undegraded":"2023-10-16T14:32:27.099757+0000","last_fullsized":"2023-10-16T14:32:27.099757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:36:12.085356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.195054+0000","last_change":"2023-10-16T14:32:26.048364+0000","last_active":"2023-10-16T14:32:27.195054+0000","last_peered":"2023-10-16T14:32:27.195054+0000","last_clean":"2023-10-16T14:32:27.195054+0000","last_became_active":"2023-10-16T14:32:26.048010+0000","last_became_peered":"2023-10-16T14:32:26.048010+0000","last_unstale":"2023-10-16T14:32:27.195054+0000","last_undegraded":"2023-10-16T14:32:27.195054+0000","last_fullsized":"2023-10-16T14:32:27.195054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:01.775709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.076325+0000","last_change":"2023-10-16T14:32:27.070297+0000","last_active":"2023-10-16T14:32:27.076325+0000","last_peered":"2023-10-16T14:32:27.076325+0000","last_clean":"2023-10-16T14:32:27.076325+0000","last_became_active":"2023-10-16T14:32:27.069610+0000","last_became_peered":"2023-10-16T14:32:27.069610+0000","last_unstale":"2023-10-16T14:32:27.076325+0000","last_undegraded":"2023-10-16T14:32:27.076325+0000","last_fullsized":"2023-10-16T14:32:27.076325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:35.136365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.191926+0000","last_change":"2023-10-16T14:32:26.053019+0000","last_active":"2023-10-16T14:32:27.191926+0000","last_peered":"2023-10-16T14:32:27.191926+0000","last_clean":"2023-10-16T14:32:27.191926+0000","last_became_active":"2023-10-16T14:32:26.052181+0000","last_became_peered":"2023-10-16T14:32:26.052181+0000","last_unstale":"2023-10-16T14:32:27.191926+0000","last_undegraded":"2023-10-16T14:32:27.191926+0000","last_fullsized":"2023-10-16T14:32:27.191926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:15.685139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.066090+0000","last_change":"2023-10-16T14:32:27.084063+0000","last_active":"2023-10-16T14:32:28.066090+0000","last_peered":"2023-10-16T14:32:28.066090+0000","last_clean":"2023-10-16T14:32:28.066090+0000","last_became_active":"2023-10-16T14:32:27.083721+0000","last_became_peered":"2023-10-16T14:32:27.083721+0000","last_unstale":"2023-10-16T14:32:28.066090+0000","last_undegraded":"2023-10-16T14:32:28.066090+0000","last_fullsized":"2023-10-16T14:32:28.066090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:19.832025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.078623+0000","last_change":"2023-10-16T14:32:26.049723+0000","last_active":"2023-10-16T14:32:27.078623+0000","last_peered":"2023-10-16T14:32:27.078623+0000","last_clean":"2023-10-16T14:32:27.078623+0000","last_became_active":"2023-10-16T14:32:26.049452+0000","last_became_peered":"2023-10-16T14:32:26.049452+0000","last_unstale":"2023-10-16T14:32:27.078623+0000","last_undegraded":"2023-10-16T14:32:27.078623+0000","last_fullsized":"2023-10-16T14:32:27.078623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:06.837100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.099620+0000","last_change":"2023-10-16T14:32:27.098948+0000","last_active":"2023-10-16T14:32:27.099620+0000","last_peered":"2023-10-16T14:32:27.099620+0000","last_clean":"2023-10-16T14:32:27.099620+0000","last_became_active":"2023-10-16T14:32:27.098611+0000","last_became_peered":"2023-10-16T14:32:27.098611+0000","last_unstale":"2023-10-16T14:32:27.099620+0000","last_undegraded":"2023-10-16T14:32:27.099620+0000","last_fullsized":"2023-10-16T14:32:27.099620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:42.753061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.192152+0000","last_change":"2023-10-16T14:32:26.067358+0000","last_active":"2023-10-16T14:32:27.192152+0000","last_peered":"2023-10-16T14:32:27.192152+0000","last_clean":"2023-10-16T14:32:27.192152+0000","last_became_active":"2023-10-16T14:32:26.067105+0000","last_became_peered":"2023-10-16T14:32:26.067105+0000","last_unstale":"2023-10-16T14:32:27.192152+0000","last_undegraded":"2023-10-16T14:32:27.192152+0000","last_fullsized":"2023-10-16T14:32:27.192152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:35.019582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.097032+0000","last_change":"2023-10-16T14:32:27.094277+0000","last_active":"2023-10-16T14:32:27.097032+0000","last_peered":"2023-10-16T14:32:27.097032+0000","last_clean":"2023-10-16T14:32:27.097032+0000","last_became_active":"2023-10-16T14:32:27.093999+0000","last_became_peered":"2023-10-16T14:32:27.093999+0000","last_unstale":"2023-10-16T14:32:27.097032+0000","last_undegraded":"2023-10-16T14:32:27.097032+0000","last_fullsized":"2023-10-16T14:32:27.097032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:45.033569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.070756+0000","last_change":"2023-10-16T14:32:26.058045+0000","last_active":"2023-10-16T14:32:27.070756+0000","last_peered":"2023-10-16T14:32:27.070756+0000","last_clean":"2023-10-16T14:32:27.070756+0000","last_became_active":"2023-10-16T14:32:26.056976+0000","last_became_peered":"2023-10-16T14:32:26.056976+0000","last_unstale":"2023-10-16T14:32:27.070756+0000","last_undegraded":"2023-10-16T14:32:27.070756+0000","last_fullsized":"2023-10-16T14:32:27.070756+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:45.953345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091004+0000","last_change":"2023-10-16T14:32:27.087116+0000","last_active":"2023-10-16T14:32:27.091004+0000","last_peered":"2023-10-16T14:32:27.091004+0000","last_clean":"2023-10-16T14:32:27.091004+0000","last_became_active":"2023-10-16T14:32:27.086772+0000","last_became_peered":"2023-10-16T14:32:27.086772+0000","last_unstale":"2023-10-16T14:32:27.091004+0000","last_undegraded":"2023-10-16T14:32:27.091004+0000","last_fullsized":"2023-10-16T14:32:27.091004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:30.444544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.082501+0000","last_change":"2023-10-16T14:32:26.065155+0000","last_active":"2023-10-16T14:32:27.082501+0000","last_peered":"2023-10-16T14:32:27.082501+0000","last_clean":"2023-10-16T14:32:27.082501+0000","last_became_active":"2023-10-16T14:32:26.064901+0000","last_became_peered":"2023-10-16T14:32:26.064901+0000","last_unstale":"2023-10-16T14:32:27.082501+0000","last_undegraded":"2023-10-16T14:32:27.082501+0000","last_fullsized":"2023-10-16T14:32:27.082501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:57.587995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.093740+0000","last_change":"2023-10-16T14:32:27.092544+0000","last_active":"2023-10-16T14:32:27.093740+0000","last_peered":"2023-10-16T14:32:27.093740+0000","last_clean":"2023-10-16T14:32:27.093740+0000","last_became_active":"2023-10-16T14:32:27.092229+0000","last_became_peered":"2023-10-16T14:32:27.092229+0000","last_unstale":"2023-10-16T14:32:27.093740+0000","last_undegraded":"2023-10-16T14:32:27.093740+0000","last_fullsized":"2023-10-16T14:32:27.093740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:37.783295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.193739+0000","last_change":"2023-10-16T14:32:26.066887+0000","last_active":"2023-10-16T14:32:27.193739+0000","last_peered":"2023-10-16T14:32:27.193739+0000","last_clean":"2023-10-16T14:32:27.193739+0000","last_became_active":"2023-10-16T14:32:26.065597+0000","last_became_peered":"2023-10-16T14:32:26.065597+0000","last_unstale":"2023-10-16T14:32:27.193739+0000","last_undegraded":"2023-10-16T14:32:27.193739+0000","last_fullsized":"2023-10-16T14:32:27.193739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:40.146391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092165+0000","last_change":"2023-10-16T14:32:27.088681+0000","last_active":"2023-10-16T14:32:27.092165+0000","last_peered":"2023-10-16T14:32:27.092165+0000","last_clean":"2023-10-16T14:32:27.092165+0000","last_became_active":"2023-10-16T14:32:27.088389+0000","last_became_peered":"2023-10-16T14:32:27.088389+0000","last_unstale":"2023-10-16T14:32:27.092165+0000","last_undegraded":"2023-10-16T14:32:27.092165+0000","last_fullsized":"2023-10-16T14:32:27.092165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:51.220283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.081592+0000","last_change":"2023-10-16T14:32:26.067218+0000","last_active":"2023-10-16T14:32:27.081592+0000","last_peered":"2023-10-16T14:32:27.081592+0000","last_clean":"2023-10-16T14:32:27.081592+0000","last_became_active":"2023-10-16T14:32:26.060284+0000","last_became_peered":"2023-10-16T14:32:26.060284+0000","last_unstale":"2023-10-16T14:32:27.081592+0000","last_undegraded":"2023-10-16T14:32:27.081592+0000","last_fullsized":"2023-10-16T14:32:27.081592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:50.377529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.087384+0000","last_change":"2023-10-16T14:32:27.081250+0000","last_active":"2023-10-16T14:32:27.087384+0000","last_peered":"2023-10-16T14:32:27.087384+0000","last_clean":"2023-10-16T14:32:27.087384+0000","last_became_active":"2023-10-16T14:32:27.080688+0000","last_became_peered":"2023-10-16T14:32:27.080688+0000","last_unstale":"2023-10-16T14:32:27.087384+0000","last_undegraded":"2023-10-16T14:32:27.087384+0000","last_fullsized":"2023-10-16T14:32:27.087384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:06.072326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.076313+0000","last_change":"2023-10-16T14:32:26.065863+0000","last_active":"2023-10-16T14:32:27.076313+0000","last_peered":"2023-10-16T14:32:27.076313+0000","last_clean":"2023-10-16T14:32:27.076313+0000","last_became_active":"2023-10-16T14:32:26.065563+0000","last_became_peered":"2023-10-16T14:32:26.065563+0000","last_unstale":"2023-10-16T14:32:27.076313+0000","last_undegraded":"2023-10-16T14:32:27.076313+0000","last_fullsized":"2023-10-16T14:32:27.076313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:24.670200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091667+0000","last_change":"2023-10-16T14:32:27.087885+0000","last_active":"2023-10-16T14:32:27.091667+0000","last_peered":"2023-10-16T14:32:27.091667+0000","last_clean":"2023-10-16T14:32:27.091667+0000","last_became_active":"2023-10-16T14:32:27.087519+0000","last_became_peered":"2023-10-16T14:32:27.087519+0000","last_unstale":"2023-10-16T14:32:27.091667+0000","last_undegraded":"2023-10-16T14:32:27.091667+0000","last_fullsized":"2023-10-16T14:32:27.091667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:10.629767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.084102+0000","last_change":"2023-10-16T14:32:26.062610+0000","last_active":"2023-10-16T14:32:27.084102+0000","last_peered":"2023-10-16T14:32:27.084102+0000","last_clean":"2023-10-16T14:32:27.084102+0000","last_became_active":"2023-10-16T14:32:26.062178+0000","last_became_peered":"2023-10-16T14:32:26.062178+0000","last_unstale":"2023-10-16T14:32:27.084102+0000","last_undegraded":"2023-10-16T14:32:27.084102+0000","last_fullsized":"2023-10-16T14:32:27.084102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:43.161255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.086234+0000","last_change":"2023-10-16T14:32:27.080147+0000","last_active":"2023-10-16T14:32:27.086234+0000","last_peered":"2023-10-16T14:32:27.086234+0000","last_clean":"2023-10-16T14:32:27.086234+0000","last_became_active":"2023-10-16T14:32:27.079877+0000","last_became_peered":"2023-10-16T14:32:27.079877+0000","last_unstale":"2023-10-16T14:32:27.086234+0000","last_undegraded":"2023-10-16T14:32:27.086234+0000","last_fullsized":"2023-10-16T14:32:27.086234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:10.836410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.068599+0000","last_change":"2023-10-16T14:32:26.063314+0000","last_active":"2023-10-16T14:32:27.068599+0000","last_peered":"2023-10-16T14:32:27.068599+0000","last_clean":"2023-10-16T14:32:27.068599+0000","last_became_active":"2023-10-16T14:32:26.063034+0000","last_became_peered":"2023-10-16T14:32:26.063034+0000","last_unstale":"2023-10-16T14:32:27.068599+0000","last_undegraded":"2023-10-16T14:32:27.068599+0000","last_fullsized":"2023-10-16T14:32:27.068599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:42:27.717271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.065366+0000","last_change":"2023-10-16T14:32:27.082613+0000","last_active":"2023-10-16T14:32:28.065366+0000","last_peered":"2023-10-16T14:32:28.065366+0000","last_clean":"2023-10-16T14:32:28.065366+0000","last_became_active":"2023-10-16T14:32:27.082220+0000","last_became_peered":"2023-10-16T14:32:27.082220+0000","last_unstale":"2023-10-16T14:32:28.065366+0000","last_undegraded":"2023-10-16T14:32:28.065366+0000","last_fullsized":"2023-10-16T14:32:28.065366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:35.487333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T14:32:28.063901+0000","last_change":"2023-10-16T14:32:26.070689+0000","last_active":"2023-10-16T14:32:28.063901+0000","last_peered":"2023-10-16T14:32:28.063901+0000","last_clean":"2023-10-16T14:32:28.063901+0000","last_became_active":"2023-10-16T14:32:26.070372+0000","last_became_peered":"2023-10-16T14:32:26.070372+0000","last_unstale":"2023-10-16T14:32:28.063901+0000","last_undegraded":"2023-10-16T14:32:28.063901+0000","last_fullsized":"2023-10-16T14:32:28.063901+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:45.870448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.096942+0000","last_change":"2023-10-16T14:32:27.090107+0000","last_active":"2023-10-16T14:32:27.096942+0000","last_peered":"2023-10-16T14:32:27.096942+0000","last_clean":"2023-10-16T14:32:27.096942+0000","last_became_active":"2023-10-16T14:32:27.089782+0000","last_became_peered":"2023-10-16T14:32:27.089782+0000","last_unstale":"2023-10-16T14:32:27.096942+0000","last_undegraded":"2023-10-16T14:32:27.096942+0000","last_fullsized":"2023-10-16T14:32:27.096942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:27.351413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.068555+0000","last_change":"2023-10-16T14:32:26.049197+0000","last_active":"2023-10-16T14:32:27.068555+0000","last_peered":"2023-10-16T14:32:27.068555+0000","last_clean":"2023-10-16T14:32:27.068555+0000","last_became_active":"2023-10-16T14:32:26.048884+0000","last_became_peered":"2023-10-16T14:32:26.048884+0000","last_unstale":"2023-10-16T14:32:27.068555+0000","last_undegraded":"2023-10-16T14:32:27.068555+0000","last_fullsized":"2023-10-16T14:32:27.068555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:15.191642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.092959+0000","last_change":"2023-10-16T14:32:27.087915+0000","last_active":"2023-10-16T14:32:27.092959+0000","last_peered":"2023-10-16T14:32:27.092959+0000","last_clean":"2023-10-16T14:32:27.092959+0000","last_became_active":"2023-10-16T14:32:27.087589+0000","last_became_peered":"2023-10-16T14:32:27.087589+0000","last_unstale":"2023-10-16T14:32:27.092959+0000","last_undegraded":"2023-10-16T14:32:27.092959+0000","last_fullsized":"2023-10-16T14:32:27.092959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:44.403697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.062152+0000","last_change":"2023-10-16T14:32:26.060497+0000","last_active":"2023-10-16T14:32:27.062152+0000","last_peered":"2023-10-16T14:32:27.062152+0000","last_clean":"2023-10-16T14:32:27.062152+0000","last_became_active":"2023-10-16T14:32:26.060035+0000","last_became_peered":"2023-10-16T14:32:26.060035+0000","last_unstale":"2023-10-16T14:32:27.062152+0000","last_undegraded":"2023-10-16T14:32:27.062152+0000","last_fullsized":"2023-10-16T14:32:27.062152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:15.597365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.091667+0000","last_change":"2023-10-16T14:32:27.085134+0000","last_active":"2023-10-16T14:32:27.091667+0000","last_peered":"2023-10-16T14:32:27.091667+0000","last_clean":"2023-10-16T14:32:27.091667+0000","last_became_active":"2023-10-16T14:32:27.084849+0000","last_became_peered":"2023-10-16T14:32:27.084849+0000","last_unstale":"2023-10-16T14:32:27.091667+0000","last_undegraded":"2023-10-16T14:32:27.091667+0000","last_fullsized":"2023-10-16T14:32:27.091667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:26.021223+0000","last_clean_scrub_stamp":"2023-10-16T14:32:26.021223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:22.099583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.076702+0000","last_change":"2023-10-16T14:32:26.059821+0000","last_active":"2023-10-16T14:32:27.076702+0000","last_peered":"2023-10-16T14:32:27.076702+0000","last_clean":"2023-10-16T14:32:27.076702+0000","last_became_active":"2023-10-16T14:32:26.059512+0000","last_became_peered":"2023-10-16T14:32:26.059512+0000","last_unstale":"2023-10-16T14:32:27.076702+0000","last_undegraded":"2023-10-16T14:32:27.076702+0000","last_fullsized":"2023-10-16T14:32:27.076702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:32:25.005355+0000","last_clean_scrub_stamp":"2023-10-16T14:32:25.005355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:44.017701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"71'161","reported_seq":392,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-16T14:32:27.060098+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:32:27.060098+0000","last_peered":"2023-10-16T14:32:27.060098+0000","last_clean":"2023-10-16T14:32:27.060098+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:32:27.060098+0000","last_undegraded":"2023-10-16T14:32:27.060098+0000","last_fullsized":"2023-10-16T14:32:27.060098+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41076,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330764,"statfs":{"total":96636764160,"available":96594702336,"internally_reserved":0,"allocated":1605632,"data_stored":1073130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0900000000000001,"15min":1.1559999999999999},"min":{"1min":0.755,"5min":0.64900000000000002,"15min":0.42799999999999999},"max":{"1min":1.833,"5min":1.833,"15min":1.9159999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.169,"15min":1.1519999999999999},"min":{"1min":0.69599999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4490000000000001,"5min":1.673,"15min":1.673},"last":1.2609999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.1879999999999999,"15min":1.1859999999999999},"min":{"1min":0.83599999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.5209999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.264,"5min":1.212,"15min":1.2310000000000001},"min":{"1min":0.78400000000000003,"5min":0.73799999999999999,"15min":0.67600000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":2.0219999999999998},"last":1.3300000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.319,"15min":1.3},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.63500000000000001},"max":{"1min":1.5740000000000001,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.512},{"interface":"front","average":{"1min":1.319,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.63400000000000001},"max":{"1min":1.669,"5min":1.7250000000000001,"15min":1.885},"last":1.216}]},{"osd":3,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3169999999999999,"15min":1.3},"min":{"1min":1.008,"5min":0.81799999999999995,"15min":0.75900000000000001},"max":{"1min":1.6000000000000001,"5min":1.746,"15min":2.048},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.3680000000000001,"15min":1.379},"min":{"1min":1.0880000000000001,"5min":0.80900000000000005,"15min":0.71899999999999997},"max":{"1min":1.768,"5min":2.1269999999999998,"15min":2.2509999999999999},"last":1.425}]},{"osd":4,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.361,"15min":1.365},"min":{"1min":1.0720000000000001,"5min":0.82299999999999995,"15min":0.67600000000000005},"max":{"1min":1.9239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.26,"15min":1.2649999999999999},"min":{"1min":0.85399999999999998,"5min":0.80800000000000005,"15min":0.496},"max":{"1min":1.643,"5min":2.0539999999999998,"15min":2.153},"last":1.379}]},{"osd":5,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.415,"15min":1.4139999999999999},"min":{"1min":0.94799999999999995,"5min":0.94399999999999995,"15min":0.61199999999999999},"max":{"1min":2.0049999999999999,"5min":2.2000000000000002,"15min":2.2530000000000001},"last":1.635},{"interface":"front","average":{"1min":1.508,"5min":1.464,"15min":1.494},"min":{"1min":1.048,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.1349999999999998,"5min":2.2450000000000001,"15min":2.2970000000000002},"last":1.1879999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:31:51 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.4490000000000001,"15min":1.4370000000000001},"min":{"1min":1.0589999999999999,"5min":1.048,"15min":0.63100000000000001},"max":{"1min":2.23,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.476},{"interface":"front","average":{"1min":1.488,"5min":1.4950000000000001,"15min":1.514},"min":{"1min":1.139,"5min":0.88700000000000001,"15min":0.59299999999999997},"max":{"1min":1.893,"5min":2.3260000000000001,"15min":2.427},"last":1.5660000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36328,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335512,"statfs":{"total":96636764160,"available":96599564288,"internally_reserved":0,"allocated":1003520,"data_stored":482602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.161,"15min":1.169},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.2160000000000002},"last":1.258},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2370000000000001,"15min":1.2310000000000001},"min":{"1min":0.81299999999999994,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.607,"5min":2.2709999999999999,"15min":3.863},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.252,"15min":1.2569999999999999},"min":{"1min":0.97299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.726,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.581},{"interface":"front","average":{"1min":1.302,"5min":1.321,"15min":1.3029999999999999},"min":{"1min":0.94599999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.667,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":0.94899999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.1830000000000001,"15min":1.214},"min":{"1min":0.98399999999999999,"5min":0.747,"15min":0.59899999999999998},"max":{"1min":1.5409999999999999,"5min":1.708,"15min":3.5800000000000001},"last":1.194},{"interface":"front","average":{"1min":1.389,"5min":1.3440000000000001,"15min":1.3600000000000001},"min":{"1min":0.98999999999999999,"5min":0.73599999999999999,"15min":0.70599999999999996},"max":{"1min":1.704,"5min":1.8799999999999999,"15min":2.7250000000000001},"last":1.52}]},{"osd":3,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.347,"15min":1.3520000000000001},"min":{"1min":0.97299999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.802,"5min":1.96,"15min":4.9790000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.319,"15min":1.3580000000000001},"min":{"1min":1.119,"5min":0.77200000000000002,"15min":0.71399999999999997},"max":{"1min":1.6439999999999999,"5min":1.913,"15min":5.4160000000000004},"last":1.5609999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2949999999999999,"15min":1.3129999999999999},"min":{"1min":0.79600000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8049999999999999,"5min":2.1070000000000002,"15min":5},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.367,"15min":1.385},"min":{"1min":1.0029999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.73,"5min":2.1789999999999998,"15min":5.4640000000000004},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.278,"15min":1.321},"min":{"1min":0.89600000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.859,"5min":1.859,"15min":3.9279999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4810000000000001,"15min":1.5029999999999999},"min":{"1min":1.111,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.8919999999999999,"5min":2.3330000000000002,"15min":3.6749999999999998},"last":1.387}]},{"osd":7,"last update":"Mon Oct 16 14:31:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5569999999999999,"15min":1.5669999999999999},"min":{"1min":1.1539999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.8939999999999999,"5min":2.0790000000000002,"15min":4.6859999999999999},"last":1.353},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5229999999999999,"15min":1.5609999999999999},"min":{"1min":1.2130000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":4.883},"last":1.6319999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36328,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335512,"statfs":{"total":96636764160,"available":96599564288,"internally_reserved":0,"allocated":1011712,"data_stored":483254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.113,"15min":1.117},"min":{"1min":0.77600000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5649999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.2130000000000001,"15min":1.1779999999999999},"min":{"1min":0.91500000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.651,"5min":1.8340000000000001,"15min":2.2629999999999999},"last":1.127}]},{"osd":1,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1679999999999999,"15min":1.177},"min":{"1min":0.93999999999999995,"5min":0.65200000000000002,"15min":0.64600000000000002},"max":{"1min":1.5129999999999999,"5min":1.726,"15min":1.726},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.218,"15min":1.246},"min":{"1min":0.90400000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5600000000000001,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.2410000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.2330000000000001,"15min":1.24},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.587,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.214},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.333,"15min":1.3580000000000001},"min":{"1min":1.1160000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.6359999999999999,"5min":2.1200000000000001,"15min":2.3500000000000001},"last":1.403}]},{"osd":3,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2629999999999999,"15min":1.2709999999999999},"min":{"1min":0.94399999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.6220000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3819999999999999,"15min":1.3879999999999999},"min":{"1min":0.92300000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.6919999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.347}]},{"osd":4,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.74399999999999999,"5min":0.63900000000000001,"15min":0.54700000000000004},"max":{"1min":1.661,"5min":1.8069999999999999,"15min":2.863},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.514,"15min":1.51},"min":{"1min":0.85399999999999998,"5min":0.79600000000000004,"15min":0.61699999999999999},"max":{"1min":1.7609999999999999,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.4399999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3680000000000001,"15min":1.3879999999999999},"min":{"1min":0.84499999999999997,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.712,"5min":2.012,"15min":2.012},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.407,"15min":1.448},"min":{"1min":0.92800000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5900000000000001,"5min":2.0099999999999998,"15min":3.794},"last":1.3180000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:50 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5369999999999999,"15min":1.5569999999999999},"min":{"1min":1.0129999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.835,"5min":2.641,"15min":2.641},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5920000000000001,"15min":1.609},"min":{"1min":1.198,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8180000000000001,"5min":2.2010000000000001,"15min":2.415},"last":1.6259999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46060,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325780,"statfs":{"total":96636764160,"available":96589598720,"internally_reserved":0,"allocated":1015808,"data_stored":484214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1619999999999999,"15min":1.165},"min":{"1min":0.84899999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.748,"5min":1.881,"15min":5.367},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.135,"15min":1.1919999999999999},"min":{"1min":0.874,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3799999999999999,"5min":1.7589999999999999,"15min":6.3159999999999998},"last":0.93100000000000005}]},{"osd":1,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1679999999999999,"15min":1.173},"min":{"1min":0.85999999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.5569999999999999,"5min":1.5880000000000001,"15min":1.78},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.238,"15min":1.2609999999999999},"min":{"1min":0.90600000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6100000000000001,"5min":1.782,"15min":5.4640000000000004},"last":1.179}]},{"osd":2,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.405,"15min":1.411},"min":{"1min":0.89600000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.821,"5min":1.821,"15min":5.617},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.355,"15min":1.3859999999999999},"min":{"1min":0.84799999999999998,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":6.391},"last":1.29}]},{"osd":3,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.304,"15min":1.341},"min":{"1min":1.0289999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.603,"5min":1.9330000000000001,"15min":7.4089999999999998},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.294,"15min":1.3},"min":{"1min":0.90300000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.6919999999999999,"5min":1.8899999999999999,"15min":2.0339999999999998},"last":1.1020000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3029999999999999,"15min":1.3069999999999999},"min":{"1min":1.0129999999999999,"5min":0.61499999999999999,"15min":0.48699999999999999},"max":{"1min":1.641,"5min":1.696,"15min":5.4109999999999996},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.325,"15min":1.3759999999999999},"min":{"1min":0.96099999999999997,"5min":0.76600000000000001,"15min":0.66100000000000003},"max":{"1min":1.5649999999999999,"5min":1.712,"15min":9.8239999999999998},"last":1.6140000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3169999999999999,"15min":1.292},"min":{"1min":0.83799999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.869,"5min":1.869,"15min":2.4729999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.468,"15min":1.4750000000000001},"min":{"1min":1.1160000000000001,"5min":0.71999999999999997,"15min":0.44700000000000001},"max":{"1min":1.6859999999999999,"5min":1.891,"15min":5.782},"last":1.7230000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:53 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4610000000000001,"15min":1.488},"min":{"1min":1.2050000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7769999999999999,"5min":1.9099999999999999,"15min":6.4379999999999997},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.4770000000000001,"15min":1.4690000000000001},"min":{"1min":1.014,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.966,"5min":2.0579999999999998,"15min":3.7799999999999998},"last":1.5409999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40496,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331344,"statfs":{"total":96636764160,"available":96595296256,"internally_reserved":0,"allocated":1011712,"data_stored":482658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.84799999999999998,"15min":0.90700000000000003},"min":{"1min":0.377,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.3919999999999999,"5min":1.4419999999999999,"15min":2.4420000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.04,"15min":1.03},"min":{"1min":0.746,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4810000000000001,"5min":1.913,"15min":1.982},"last":0.91600000000000004}]},{"osd":1,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.92000000000000004,"15min":0.94799999999999995},"min":{"1min":0.63900000000000001,"5min":0.437,"15min":0.32500000000000001},"max":{"1min":1.484,"5min":1.7390000000000001,"15min":2.149},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1319999999999999,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.314,"15min":0.314},"max":{"1min":1.474,"5min":1.6499999999999999,"15min":2.1600000000000001},"last":1.2210000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.276,"15min":1.329},"min":{"1min":0.86299999999999999,"5min":0.53800000000000003,"15min":0.44800000000000001},"max":{"1min":1.7050000000000001,"5min":1.895,"15min":2.1259999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4039999999999999,"15min":1.4410000000000001},"min":{"1min":0.78000000000000003,"5min":0.5,"15min":0.5},"max":{"1min":1.76,"5min":1.8660000000000001,"15min":2.6299999999999999},"last":0.88300000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.23,"15min":1.2729999999999999},"min":{"1min":0.94699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.552,"5min":1.6819999999999999,"15min":2.4529999999999998},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.3129999999999999,"15min":1.3500000000000001},"min":{"1min":1.0740000000000001,"5min":0.629,"15min":0.629},"max":{"1min":1.5920000000000001,"5min":1.909,"15min":2.3700000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5069999999999999,"15min":1.542},"min":{"1min":1.2370000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.7150000000000001,"5min":2.0070000000000001,"15min":2.552},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.421,"15min":1.4890000000000001},"min":{"1min":0.92500000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.841,"5min":2.3119999999999998,"15min":2.7589999999999999},"last":1.319}]},{"osd":6,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.3899999999999999,"15min":1.4319999999999999},"min":{"1min":0.89600000000000002,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.6639999999999999,"5min":2.181,"15min":2.3399999999999999},"last":1.26},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.476,"15min":1.5129999999999999},"min":{"1min":1.0229999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7529999999999999,"5min":1.9510000000000001,"15min":2.347},"last":1.5960000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.5289999999999999,"15min":1.601},"min":{"1min":1.036,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.952,"5min":2.395,"15min":2.8820000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5229999999999999,"15min":1.5649999999999999},"min":{"1min":0.83799999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.6910000000000001,"5min":2.1000000000000001,"15min":2.6179999999999999},"last":1.357}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44648,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327192,"statfs":{"total":96636764160,"available":96591044608,"internally_reserved":0,"allocated":1011712,"data_stored":482658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.879,"15min":0.876},"min":{"1min":0.63900000000000001,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":1.2549999999999999,"5min":1.556,"15min":2.472},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.141,"5min":1.0089999999999999,"15min":1.032},"min":{"1min":0.71899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.585,"5min":1.585,"15min":2.8879999999999999},"last":0.70299999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.98699999999999999,"15min":1.008},"min":{"1min":0.77700000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":3.234,"15min":3.234},"last":1.143},{"interface":"front","average":{"1min":1.173,"5min":1.1719999999999999,"15min":1.163},"min":{"1min":0.91300000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4850000000000001,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.419}]},{"osd":3,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.236,"15min":1.2370000000000001},"min":{"1min":0.94299999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.29},{"interface":"front","average":{"1min":1.335,"5min":1.294,"15min":1.3},"min":{"1min":0.97399999999999998,"5min":0.57599999999999996,"15min":0.42499999999999999},"max":{"1min":1.6899999999999999,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.3080000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.452,"15min":1.431},"min":{"1min":1.0649999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.873,"5min":2.2869999999999999,"15min":2.5499999999999998},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.3879999999999999,"15min":1.373},"min":{"1min":1.1220000000000001,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.869,"5min":3.016,"15min":3.016},"last":1.2549999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.419,"15min":1.4239999999999999},"min":{"1min":1.169,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.3650000000000002,"5min":2.4100000000000001,"15min":3.25},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5629999999999999,"15min":1.55},"min":{"1min":1.0329999999999999,"5min":0.79200000000000004,"15min":0.78300000000000003},"max":{"1min":1.972,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.647}]},{"osd":6,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.4930000000000001,"15min":1.5009999999999999},"min":{"1min":1.0760000000000001,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.9950000000000001,"5min":3.0760000000000001,"15min":3.536},"last":1.371},{"interface":"front","average":{"1min":1.575,"5min":1.522,"15min":1.5369999999999999},"min":{"1min":1.1779999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.5230000000000001,"5min":2.5920000000000001,"15min":3.3929999999999998},"last":1.4610000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.5700000000000001,"15min":1.5900000000000001},"min":{"1min":1.075,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.0840000000000001,"5min":3.1899999999999999,"15min":5.2199999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.647,"15min":1.651},"min":{"1min":1.353,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4529999999999998,"5min":2.7519999999999998,"15min":3.496},"last":1.7010000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54372,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317468,"statfs":{"total":96636764160,"available":96581087232,"internally_reserved":0,"allocated":1007616,"data_stored":482692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.878,"15min":0.90800000000000003},"min":{"1min":0.73099999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.421,"5min":1.569,"15min":1.821},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.125,"15min":1.107},"min":{"1min":0.629,"5min":0.378,"15min":0.378},"max":{"1min":1.776,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0649999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.98999999999999999,"15min":0.99099999999999999},"min":{"1min":0.71599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.627,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.107,"15min":1.1579999999999999},"min":{"1min":0.69899999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5960000000000001,"5min":1.756,"15min":1.796},"last":1.288}]},{"osd":3,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.56299999999999994,"15min":0.52400000000000002},"max":{"1min":1.776,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.1870000000000001,"15min":1.2010000000000001},"min":{"1min":0.64700000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.756,"5min":1.8100000000000001,"15min":2.2050000000000001},"last":1.425}]},{"osd":4,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2749999999999999,"15min":1.29},"min":{"1min":0.95599999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.544,"5min":2.069,"15min":2.9820000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.387,"15min":1.423},"min":{"1min":1.0089999999999999,"5min":0.61399999999999999,"15min":0.59499999999999997},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":3.242},"last":1.607}]},{"osd":5,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4370000000000001,"15min":1.4379999999999999},"min":{"1min":1.073,"5min":0.623,"15min":0.623},"max":{"1min":1.9319999999999999,"5min":2.032,"15min":2.3690000000000002},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.415,"15min":1.4199999999999999},"min":{"1min":0.91100000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.869,"5min":2.1469999999999998,"15min":2.181},"last":1.202}]},{"osd":6,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.492,"15min":1.5409999999999999},"min":{"1min":0.98099999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0230000000000001,"5min":2.125,"15min":2.476},"last":1.714},{"interface":"front","average":{"1min":1.569,"5min":1.542,"15min":1.546},"min":{"1min":1.01,"5min":0.86299999999999999,"15min":0.73099999999999998},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.6160000000000001},"last":1.258}]},{"osd":7,"last update":"Mon Oct 16 14:31:43 2023","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.625,"15min":1.6259999999999999},"min":{"1min":1.2150000000000001,"5min":0.90600000000000003,"15min":0.79300000000000004},"max":{"1min":2.1920000000000002,"5min":2.2639999999999998,"15min":2.5059999999999998},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.532,"15min":1.5580000000000001},"min":{"1min":1.054,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.069,"5min":2.1080000000000001,"15min":2.3410000000000002},"last":1.506}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36920,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334920,"statfs":{"total":96636764160,"available":96598958080,"internally_reserved":0,"allocated":1613824,"data_stored":1075164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.89500000000000002,"15min":0.94599999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42199999999999999},"max":{"1min":1.1759999999999999,"5min":1.464,"15min":2.2650000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1200000000000001,"15min":1.167},"min":{"1min":0.65600000000000003,"5min":0.497,"15min":0.36399999999999999},"max":{"1min":1.833,"5min":1.833,"15min":2.6030000000000002},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.242,"15min":1.2529999999999999},"min":{"1min":0.68600000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.7290000000000001,"5min":1.887,"15min":3.653},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.194,"5min":1.1910000000000001,"15min":1.2150000000000001},"min":{"1min":0.73399999999999999,"5min":0.626,"15min":0.47399999999999998},"max":{"1min":1.6819999999999999,"5min":2.012,"15min":3.1400000000000001},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.0129999999999999},"min":{"1min":0.68500000000000005,"5min":0.46600000000000003,"15min":0.40100000000000002},"max":{"1min":1.889,"5min":1.889,"15min":2.391},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2070000000000001,"15min":1.24},"min":{"1min":0.68600000000000005,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.6020000000000001,"5min":1.8660000000000001,"15min":2.79},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.29,"15min":1.306},"min":{"1min":0.89400000000000002,"5min":0.80100000000000005,"15min":0.61599999999999999},"max":{"1min":1.704,"5min":1.9259999999999999,"15min":2.7370000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.484,"5min":1.4570000000000001,"15min":1.468},"min":{"1min":0.99199999999999999,"5min":0.75800000000000001,"15min":0.65900000000000003},"max":{"1min":1.966,"5min":2.032,"15min":3.6190000000000002},"last":0.76500000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4099999999999999,"15min":1.4319999999999999},"min":{"1min":0.79000000000000004,"5min":0.78400000000000003,"15min":0.58099999999999996},"max":{"1min":2.0150000000000001,"5min":2.097,"15min":3.2370000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3280000000000001,"15min":1.373},"min":{"1min":0.96399999999999997,"5min":0.84899999999999998,"15min":0.67900000000000005},"max":{"1min":1.6040000000000001,"5min":1.7150000000000001,"15min":2.819},"last":0.89600000000000002}]},{"osd":6,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.5580000000000001,"15min":1.5840000000000001},"min":{"1min":1.042,"5min":0.83499999999999996,"15min":0.75},"max":{"1min":2.097,"5min":2.097,"15min":3.5779999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.5049999999999999,"15min":1.5089999999999999},"min":{"1min":1.0609999999999999,"5min":0.98899999999999999,"15min":0.82199999999999995},"max":{"1min":1.877,"5min":2.1459999999999999,"15min":3.0979999999999999},"last":1.02}]},{"osd":7,"last update":"Mon Oct 16 14:31:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.575,"15min":1.593},"min":{"1min":1.024,"5min":0.73699999999999999,"15min":0.67100000000000004},"max":{"1min":1.974,"5min":2.2650000000000001,"15min":3.3340000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.4930000000000001,"15min":1.5700000000000001},"min":{"1min":0.98899999999999999,"5min":0.877,"15min":0.78400000000000003},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":6.5839999999999996},"last":1.22}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:32:32.201 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:32:32.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:32:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.473+0000 7f867d640640 1 Processor -- start 2023-10-16T14:32:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.474+0000 7f867d640640 1 -- start start 2023-10-16T14:32:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.475+0000 7f867d640640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f867815cb50 0x7f8678161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.475+0000 7f867d640640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86780b4c80 0x7f86780d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.475+0000 7f867d640640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86781616c0 con 0x7f86780b1690 2023-10-16T14:32:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.475+0000 7f867d640640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86780cdbf0 con 0x7f86780b4c80 2023-10-16T14:32:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.475+0000 7f867d640640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86780cde90 con 0x7f867815cb50 2023-10-16T14:32:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86780b4c80 0x7f86780d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86780b4c80 0x7f86780d1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56788/0 (socket says 172.21.15.29:56788) 2023-10-16T14:32:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 -- 172.21.15.29:0/1906510656 learned_addr learned my addr 172.21.15.29:0/1906510656 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f8677fff640 1 --2- 172.21.15.29:0/1906510656 >> v2:172.21.15.170:3300/0 conn(0x7f867815cb50 0x7f8678161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 -- 172.21.15.29:0/1906510656 >> v2:172.21.15.170:3300/0 conn(0x7f867815cb50 msgr2=0x7f8678161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 --2- 172.21.15.29:0/1906510656 >> v2:172.21.15.170:3300/0 conn(0x7f867815cb50 0x7f8678161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 -- 172.21.15.29:0/1906510656 >> v1:172.21.15.170:6789/0 conn(0x7f86780b1690 legacy=0x7f86780b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.476+0000 7f867ce3f640 1 -- 172.21.15.29:0/1906510656 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780ce170 con 0x7f86780b4c80 2023-10-16T14:32:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.478+0000 7f867ce3f640 1 --2- 172.21.15.29:0/1906510656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f86780d1730 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f86780724b0 tx=0x7f867002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91ef3ad59a80a53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.478+0000 7f8676ffd640 1 -- 172.21.15.29:0/1906510656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670031740 con 0x7f86780b4c80 2023-10-16T14:32:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.478+0000 7f8676ffd640 1 -- 172.21.15.29:0/1906510656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86700318a0 con 0x7f86780b4c80 2023-10-16T14:32:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.478+0000 7f8676ffd640 1 -- 172.21.15.29:0/1906510656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670031c30 con 0x7f86780b4c80 2023-10-16T14:32:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 -- 172.21.15.29:0/1906510656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 msgr2=0x7f86780d1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 --2- 172.21.15.29:0/1906510656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f86780d1730 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f86780724b0 tx=0x7f867002b7e0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 -- 172.21.15.29:0/1906510656 shutdown_connections 2023-10-16T14:32:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 --2- 172.21.15.29:0/1906510656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f86780d1730 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 --2- 172.21.15.29:0/1906510656 >> v2:172.21.15.170:3300/0 conn(0x7f867815cb50 0x7f8678161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 -- 172.21.15.29:0/1906510656 >> 172.21.15.29:0/1906510656 conn(0x7f86780cf570 msgr2=0x7f86780cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 -- 172.21.15.29:0/1906510656 shutdown_connections 2023-10-16T14:32:32.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.479+0000 7f867d640640 1 -- 172.21.15.29:0/1906510656 wait complete. 2023-10-16T14:32:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.481+0000 7f867d640640 1 Processor -- start 2023-10-16T14:32:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.481+0000 7f867d640640 1 -- start start 2023-10-16T14:32:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f867d640640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86780b1690 0x7f867815e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f867d640640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f867815f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f867d640640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f867815cb50 0x7f867815f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f867d640640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86780c9dd0 con 0x7f86780b4c80 2023-10-16T14:32:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f867d640640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867815d2d0 con 0x7f86780b1690 2023-10-16T14:32:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f867d640640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867815d5b0 con 0x7f867815cb50 2023-10-16T14:32:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f86777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f867815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f86777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f867815f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56796/0 (socket says 172.21.15.29:56796) 2023-10-16T14:32:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.482+0000 7f86777fe640 1 -- 172.21.15.29:0/2112981144 learned_addr learned my addr 172.21.15.29:0/2112981144 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f8677fff640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86780b1690 0x7f867815e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f86777fe640 1 -- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f867815cb50 msgr2=0x7f867815f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f867ce3f640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f867815cb50 0x7f867815f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f86777fe640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f867815cb50 0x7f867815f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f86777fe640 1 -- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86780b1690 msgr2=0x7f867815e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f86777fe640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86780b1690 0x7f867815e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f86777fe640 1 -- 172.21.15.29:0/2112981144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700094b0 con 0x7f86780b4c80 2023-10-16T14:32:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f8677fff640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86780b1690 0x7f867815e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f86777fe640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f867815f060 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f866000c430 tx=0x7f866000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660011790 con 0x7f86780b4c80 2023-10-16T14:32:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86600118f0 con 0x7f86780b4c80 2023-10-16T14:32:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660011ce0 con 0x7f86780b4c80 2023-10-16T14:32:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.483+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867815d890 con 0x7f86780b4c80 2023-10-16T14:32:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.484+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867815de50 con 0x7f86780b4c80 2023-10-16T14:32:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.485+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867804e9b0 con 0x7f86780b4c80 2023-10-16T14:32:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.486+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f866001c460 con 0x7f86780b4c80 2023-10-16T14:32:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.486+0000 7f8674ff9640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f865806afe0 0x7f865806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.487+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86600897e0 con 0x7f86780b4c80 2023-10-16T14:32:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.487+0000 7f8677fff640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f865806afe0 0x7f865806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.488+0000 7f8677fff640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f865806afe0 0x7f865806d4a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8668009f50 tx=0x7f866800b040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.491+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86600546d0 con 0x7f86780b4c80 2023-10-16T14:32:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.757+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f86780d0790 con 0x7f86780b4c80 2023-10-16T14:32:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.762+0000 7f8674ff9640 1 -- 172.21.15.29:0/2112981144 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8660057cf0 con 0x7f86780b4c80 2023-10-16T14:32:32.766 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:32:32.767 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:32:32.767 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:32:32.767 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:32:32.767 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:32:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:32:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f865806afe0 msgr2=0x7f865806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f865806afe0 0x7f865806d4a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8668009f50 tx=0x7f866800b040 comp rx=0 tx=0).stop 2023-10-16T14:32:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 msgr2=0x7f867815f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f867815f060 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f866000c430 tx=0x7f866000c8f0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 shutdown_connections 2023-10-16T14:32:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f865806afe0 0x7f865806d4a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86780b4c80 0x7f867815f060 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f867815cb50 0x7f867815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 --2- 172.21.15.29:0/2112981144 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86780b1690 0x7f867815e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 >> 172.21.15.29:0/2112981144 conn(0x7f86780cf570 msgr2=0x7f86780b6100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 shutdown_connections 2023-10-16T14:32:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:32.765+0000 7f867d640640 1 -- 172.21.15.29:0/2112981144 wait complete. 2023-10-16T14:32:32.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.007+0000 7f929bfff640 1 Processor -- start 2023-10-16T14:32:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.008+0000 7f929bfff640 1 -- start start 2023-10-16T14:32:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.009+0000 7f929bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 0x7f929c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.009+0000 7f929bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929c15f4a0 con 0x7f929c0cda80 2023-10-16T14:32:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.009+0000 7f929bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929c06c960 con 0x7f929c0ad970 2023-10-16T14:32:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.009+0000 7f929bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c15b9d0 con 0x7f929c06c040 2023-10-16T14:32:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.010+0000 7f929b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f929c0cda80 0x7f929c15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44818/0 (socket says 172.21.15.29:44818) 2023-10-16T14:32:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.010+0000 7f929b7fe640 1 -- 172.21.15.29:0/264258723 learned_addr learned my addr 172.21.15.29:0/264258723 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:33.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.010+0000 7f929affd640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 0x7f929c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.010+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085707277 0 0) 0x7f929c15f4a0 con 0x7f929c0cda80 2023-10-16T14:32:33.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.010+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9288003520 con 0x7f929c0cda80 2023-10-16T14:32:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245756027 0 0) 0x7f9288003520 con 0x7f929c0cda80 2023-10-16T14:32:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 msgr2=0x7f929c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 0x7f929c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 >> v1:172.21.15.170:6789/0 conn(0x7f929c0ad970 legacy=0x7f929c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c15bcb0 con 0x7f929c0cda80 2023-10-16T14:32:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9294001670 con 0x7f929c0cda80 2023-10-16T14:32:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 >> v1:172.21.15.29:6789/0 conn(0x7f929c0cda80 legacy=0x7f929c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92880039a0 0x7f9288003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 0x7f92880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f9288005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c15b9d0 con 0x7f9288004d00 2023-10-16T14:32:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928800ba30 con 0x7f92880039a0 2023-10-16T14:32:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928800bba0 con 0x7f929c06c040 2023-10-16T14:32:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.011+0000 7f929b7fe640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f9288005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929affd640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92880039a0 0x7f9288003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929a7fc640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 0x7f92880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929b7fe640 1 -- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 msgr2=0x7f92880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929b7fe640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c06c040 0x7f92880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929b7fe640 1 -- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92880039a0 msgr2=0x7f9288003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929b7fe640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f92880039a0 0x7f9288003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.012+0000 7f929b7fe640 1 -- 172.21.15.29:0/264258723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9288003520 con 0x7f9288004d00 2023-10-16T14:32:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.013+0000 7f929b7fe640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f9288005180 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f929400c190 tx=0x7f92880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96048200abd67cb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.013+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294005ad0 con 0x7f9288004d00 2023-10-16T14:32:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.013+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9294005c30 con 0x7f9288004d00 2023-10-16T14:32:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.013+0000 7f9299ffb640 1 -- 172.21.15.29:0/264258723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929402c810 con 0x7f9288004d00 2023-10-16T14:32:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929bfff640 1 -- 172.21.15.29:0/264258723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 msgr2=0x7f9288005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929bfff640 1 --2- 172.21.15.29:0/264258723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f9288005180 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f929400c190 tx=0x7f92880057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929b7fe640 1 -- 172.21.15.29:0/264258723 reap_dead start 2023-10-16T14:32:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929bfff640 1 -- 172.21.15.29:0/264258723 shutdown_connections 2023-10-16T14:32:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929bfff640 1 -- 172.21.15.29:0/264258723 >> 172.21.15.29:0/264258723 conn(0x7f929c0ce6a0 msgr2=0x7f929c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929bfff640 1 -- 172.21.15.29:0/264258723 shutdown_connections 2023-10-16T14:32:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.014+0000 7f929bfff640 1 -- 172.21.15.29:0/264258723 wait complete. 2023-10-16T14:32:33.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.016+0000 7f929bfff640 1 Processor -- start 2023-10-16T14:32:33.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.016+0000 7f929bfff640 1 -- start start 2023-10-16T14:32:33.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f929c1f05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c1514e0 0x7f929c0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c154340 0x7f929c14f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1f14c0 con 0x7f9288004d00 2023-10-16T14:32:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1f0f00 con 0x7f929c154340 2023-10-16T14:32:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1f11e0 con 0x7f929c1514e0 2023-10-16T14:32:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f929c1f05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f929c1f05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56808/0 (socket says 172.21.15.29:56808) 2023-10-16T14:32:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.017+0000 7f929affd640 1 -- 172.21.15.29:0/3475002731 learned_addr learned my addr 172.21.15.29:0/3475002731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929b7fe640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c154340 0x7f929c14f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929affd640 1 -- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c1514e0 msgr2=0x7f929c0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929a7fc640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c1514e0 0x7f929c0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929affd640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c1514e0 0x7f929c0c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929affd640 1 -- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c154340 msgr2=0x7f929c14f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929affd640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c154340 0x7f929c14f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929affd640 1 -- 172.21.15.29:0/3475002731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c153b40 con 0x7f9288004d00 2023-10-16T14:32:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929b7fe640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c154340 0x7f929c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f929affd640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f929c1f05b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f928c011e30 tx=0x7f928c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.018+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c00e040 con 0x7f9288004d00 2023-10-16T14:32:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.019+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c00f040 con 0x7f9288004d00 2023-10-16T14:32:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.019+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c009c00 con 0x7f9288004d00 2023-10-16T14:32:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.019+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c153e20 con 0x7f9288004d00 2023-10-16T14:32:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.019+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c1f0cd0 con 0x7f9288004d00 2023-10-16T14:32:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.020+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c06b330 con 0x7f9288004d00 2023-10-16T14:32:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.021+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f928c015850 con 0x7f9288004d00 2023-10-16T14:32:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.022+0000 7f9293fff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f927c06af90 0x7f927c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.022+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f928c08a5d0 con 0x7f9288004d00 2023-10-16T14:32:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.023+0000 7f929a7fc640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f927c06af90 0x7f927c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.024+0000 7f929a7fc640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f927c06af90 0x7f927c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f929c1539c0 tx=0x7f9284007b70 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.026+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f928c0554f0 con 0x7f9288004d00 2023-10-16T14:32:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.262+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f929c04e9b0 con 0x7f9288004d00 2023-10-16T14:32:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.263+0000 7f9293fff640 1 -- 172.21.15.29:0/3475002731 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f928c058b10 con 0x7f9288004d00 2023-10-16T14:32:33.267 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:33.267 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:32:26.521447+0000","modified":"2023-10-16T14:32:29.697489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3822716445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3822716445},{"type":"v1","addr":"172.21.15.170:6837","nonce":3822716445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T14:32:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2023-10-16T14:32:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f927c06af90 msgr2=0x7f927c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f927c06af90 0x7f927c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f929c1539c0 tx=0x7f9284007b70 comp rx=0 tx=0).stop 2023-10-16T14:32:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 msgr2=0x7f929c1f05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f929c1f05b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f928c011e30 tx=0x7f928c00f580 comp rx=0 tx=0).stop 2023-10-16T14:32:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 shutdown_connections 2023-10-16T14:32:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f927c06af90 0x7f927c06d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f929c154340 0x7f929c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f929c1514e0 0x7f929c0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 --2- 172.21.15.29:0/3475002731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9288004d00 0x7f929c1f05b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 >> 172.21.15.29:0/3475002731 conn(0x7f929c0ce6a0 msgr2=0x7f929c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 shutdown_connections 2023-10-16T14:32:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.266+0000 7f929bfff640 1 -- 172.21.15.29:0/3475002731 wait complete. 2023-10-16T14:32:33.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:32:26.521447+0000', 'modified': '2023-10-16T14:32:29.697489+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5584}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5584': {'gid': 5584, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6837/3822716445', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6835', 'nonce': 3822716445}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 3822716445}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:32:33.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:32:33.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T14:32:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.922+0000 7f4f36864640 1 Processor -- start 2023-10-16T14:32:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.922+0000 7f4f36864640 1 -- start start 2023-10-16T14:32:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.923+0000 7f4f36864640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f3015ad70 0x7f4f30069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.923+0000 7f4f36864640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 0x7f4f3006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.923+0000 7f4f36864640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f300cb030 con 0x7f4f300ca3b0 2023-10-16T14:32:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.923+0000 7f4f36864640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f300c91c0 con 0x7f4f3015ad70 2023-10-16T14:32:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.923+0000 7f4f36864640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300c94a0 con 0x7f4f300c9ca0 2023-10-16T14:32:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.924+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 0x7f4f3006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.924+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 0x7f4f3006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:56276/0 (socket says 172.21.15.29:56276) 2023-10-16T14:32:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.924+0000 7f4f27fff640 1 -- 172.21.15.29:0/179225278 learned_addr learned my addr 172.21.15.29:0/179225278 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.924+0000 7f4f34e01640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f4f300ca3b0 0x7f4f300ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:46132/0 (socket says 172.21.15.29:46132) 2023-10-16T14:32:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.924+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/179225278 >> v2:172.21.15.170:3300/0 conn(0x7f4f3015ad70 0x7f4f30069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.925+0000 7f4f27fff640 1 -- 172.21.15.29:0/179225278 >> v2:172.21.15.170:3300/0 conn(0x7f4f3015ad70 msgr2=0x7f4f30069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.925+0000 7f4f27fff640 1 --2- 172.21.15.29:0/179225278 >> v2:172.21.15.170:3300/0 conn(0x7f4f3015ad70 0x7f4f30069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.925+0000 7f4f27fff640 1 -- 172.21.15.29:0/179225278 >> v1:172.21.15.170:6789/0 conn(0x7f4f300ca3b0 legacy=0x7f4f300ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.925+0000 7f4f27fff640 1 -- 172.21.15.29:0/179225278 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f300c9780 con 0x7f4f300c9ca0 2023-10-16T14:32:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.926+0000 7f4f27fff640 1 --2- 172.21.15.29:0/179225278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 0x7f4f3006a2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009870 tx=0x7f4f1802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5e019cac003036 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f2f7fe640 1 -- 172.21.15.29:0/179225278 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18030840 con 0x7f4f300c9ca0 2023-10-16T14:32:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f2f7fe640 1 -- 172.21.15.29:0/179225278 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f180309a0 con 0x7f4f300c9ca0 2023-10-16T14:32:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f2f7fe640 1 -- 172.21.15.29:0/179225278 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18030d30 con 0x7f4f300c9ca0 2023-10-16T14:32:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 -- 172.21.15.29:0/179225278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 msgr2=0x7f4f3006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 --2- 172.21.15.29:0/179225278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 0x7f4f3006a2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009870 tx=0x7f4f1802b980 comp rx=0 tx=0).stop 2023-10-16T14:32:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 -- 172.21.15.29:0/179225278 shutdown_connections 2023-10-16T14:32:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 --2- 172.21.15.29:0/179225278 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300c9ca0 0x7f4f3006a2e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 --2- 172.21.15.29:0/179225278 >> v2:172.21.15.170:3300/0 conn(0x7f4f3015ad70 0x7f4f30069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 -- 172.21.15.29:0/179225278 >> 172.21.15.29:0/179225278 conn(0x7f4f300ce9b0 msgr2=0x7f4f300cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 -- 172.21.15.29:0/179225278 shutdown_connections 2023-10-16T14:32:33.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.927+0000 7f4f36864640 1 -- 172.21.15.29:0/179225278 wait complete. 2023-10-16T14:32:33.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.929+0000 7f4f36864640 1 Processor -- start 2023-10-16T14:32:33.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.929+0000 7f4f36864640 1 -- start start 2023-10-16T14:32:33.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.930+0000 7f4f36864640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 0x7f4f301f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.930+0000 7f4f36864640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300ca3b0 0x7f4f30159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.930+0000 7f4f36864640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3015ad70 0x7f4f300c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.930+0000 7f4f36864640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300b0510 con 0x7f4f300c9ca0 2023-10-16T14:32:33.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.930+0000 7f4f36864640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f301596c0 con 0x7f4f3015ad70 2023-10-16T14:32:33.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.930+0000 7f4f36864640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300c5fc0 con 0x7f4f300ca3b0 2023-10-16T14:32:33.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 0x7f4f301f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 0x7f4f301f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42984/0 (socket says 172.21.15.29:42984) 2023-10-16T14:32:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2374720090 learned_addr learned my addr 172.21.15.29:0/2374720090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f27fff640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300ca3b0 0x7f4f30159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f34e01640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3015ad70 0x7f4f300c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300ca3b0 msgr2=0x7f4f30159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300ca3b0 0x7f4f30159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3015ad70 msgr2=0x7f4f300c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3015ad70 0x7f4f300c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.931+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2374720090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f180094d0 con 0x7f4f300c9ca0 2023-10-16T14:32:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.932+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 0x7f4f301f3030 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000e870 tx=0x7f4f2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.932+0000 7f4f2d7fa640 1 -- 172.21.15.29:0/2374720090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f200094f0 con 0x7f4f300c9ca0 2023-10-16T14:32:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.932+0000 7f4f2d7fa640 1 -- 172.21.15.29:0/2374720090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2000c940 con 0x7f4f300c9ca0 2023-10-16T14:32:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.932+0000 7f4f2d7fa640 1 -- 172.21.15.29:0/2374720090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000ccf0 con 0x7f4f300c9ca0 2023-10-16T14:32:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.932+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f300c6290 con 0x7f4f300c9ca0 2023-10-16T14:32:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.932+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f300c6850 con 0x7f4f300c9ca0 2023-10-16T14:32:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.935+0000 7f4f2d7fa640 1 -- 172.21.15.29:0/2374720090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f20013470 con 0x7f4f300c9ca0 2023-10-16T14:32:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.936+0000 7f4f2d7fa640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f4f1006af90 0x7f4f1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.936+0000 7f4f2d7fa640 1 -- 172.21.15.29:0/2374720090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f200898c0 con 0x7f4f300c9ca0 2023-10-16T14:32:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.936+0000 7f4f393f6740 1 Processor -- start 2023-10-16T14:32:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.936+0000 7f4f27fff640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f4f1006af90 0x7f4f1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.936+0000 7f4f393f6740 1 -- start start 2023-10-16T14:32:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.937+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d1bce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.937+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d1ecd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.937+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1bd3c0 0x55be4d1ed250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.937+0000 7f4f393f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d0dd7a0 con 0x55be4d1e9f80 2023-10-16T14:32:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.937+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d12f740 con 0x55be4d1ea360 2023-10-16T14:32:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.937+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be4d12ce30 con 0x55be4d1bd3c0 2023-10-16T14:32:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d1bce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f27fff640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f4f1006af90 0x7f4f1006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f18036000 tx=0x7f4f18008030 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d1bce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42986/0 (socket says 172.21.15.29:42986) 2023-10-16T14:32:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2528567960 learned_addr learned my addr 172.21.15.29:0/2528567960 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f27fff640 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d1ecd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1bd3c0 msgr2=0x55be4d1ed250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1bd3c0 0x55be4d1ed250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 msgr2=0x55be4d1ecd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d1ecd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 -- 172.21.15.29:0/2528567960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be4d0ec910 con 0x55be4d1e9f80 2023-10-16T14:32:33.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.938+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d1bce80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4f200584a0 tx=0x7f4f20058d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c921d9def6497643 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f25ffb640 1 -- 172.21.15.29:0/2528567960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20060810 con 0x55be4d1e9f80 2023-10-16T14:32:33.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f25ffb640 1 -- 172.21.15.29:0/2528567960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20087d40 con 0x55be4d1e9f80 2023-10-16T14:32:33.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f25ffb640 1 -- 172.21.15.29:0/2528567960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20087eb0 con 0x55be4d1e9f80 2023-10-16T14:32:33.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 -- 172.21.15.29:0/2528567960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 msgr2=0x55be4d1bce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d1bce80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4f200584a0 tx=0x7f4f20058d50 comp rx=0 tx=0).stop 2023-10-16T14:32:33.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 -- 172.21.15.29:0/2528567960 shutdown_connections 2023-10-16T14:32:33.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1bd3c0 0x55be4d1ed250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d1ecd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 --2- 172.21.15.29:0/2528567960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d1bce80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.939+0000 7f4f393f6740 1 -- 172.21.15.29:0/2528567960 >> 172.21.15.29:0/2528567960 conn(0x55be4d1bf190 msgr2=0x55be4d207d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.940+0000 7f4f393f6740 1 -- 172.21.15.29:0/2528567960 shutdown_connections 2023-10-16T14:32:33.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.940+0000 7f4f393f6740 1 -- 172.21.15.29:0/2528567960 wait complete. 2023-10-16T14:32:33.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.940+0000 7f4f393f6740 1 -- start start 2023-10-16T14:32:33.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.941+0000 7f4f393f6740 1 Processor -- start 2023-10-16T14:32:33.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.941+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d206160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1ea360 0x55be4d207ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f393f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d128720 con 0x55be4d1e9f80 2023-10-16T14:32:33.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d0c6970 con 0x55be4d1ddd40 2023-10-16T14:32:33.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be4d0c6610 con 0x55be4d1ea360 2023-10-16T14:32:33.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f393f6740 10 client.0 _ll_register_callbacks cb 0x55be4d0e89d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:32:33.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d206160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d206160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42992/0 (socket says 172.21.15.29:42992) 2023-10-16T14:32:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 -- 172.21.15.29:0/428171916 learned_addr learned my addr 172.21.15.29:0/428171916 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 -- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1ea360 msgr2=0x55be4d207ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1ea360 0x55be4d207ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 -- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 msgr2=0x55be4d1de1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 -- 172.21.15.29:0/428171916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f300c6850 con 0x55be4d1e9f80 2023-10-16T14:32:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d206160 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4f200586d0 tx=0x7f4f20009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d608f27072b081e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.942+0000 7f4f0effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:32:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f0effd640 1 -- 172.21.15.29:0/428171916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20057c10 con 0x55be4d1e9f80 2023-10-16T14:32:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f0effd640 1 -- 172.21.15.29:0/428171916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f20060380 con 0x55be4d1e9f80 2023-10-16T14:32:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be4d14f160 con 0x55be4d1e9f80 2023-10-16T14:32:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f0effd640 1 -- 172.21.15.29:0/428171916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f20060540 con 0x55be4d1e9f80 2023-10-16T14:32:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 10 client.6516 fetch_fsmap learned FSMap version 33 2023-10-16T14:32:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55be4d126da0 con 0x55be4d1e9f80 2023-10-16T14:32:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f0effd640 1 -- 172.21.15.29:0/428171916 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55be4d14f160 con 0x55be4d1e9f80 2023-10-16T14:32:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f0effd640 1 -- 172.21.15.29:0/428171916 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f20011070 con 0x55be4d1e9f80 2023-10-16T14:32:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 10 client.6516 fetch_fsmap finished waiting for FSMap version 33 2023-10-16T14:32:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 10 client.6516 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:32:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 20 client.6516 populate_metadata read hostname 'smithi029' 2023-10-16T14:32:33.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d21bac0 0x55be4d203440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 4 client.6516 mds_command: new command op to 5584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:32:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.943+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 --> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be4d0facc0 con 0x55be4d21bac0 2023-10-16T14:32:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.944+0000 7f4f27fff640 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d21bac0 0x55be4d203440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.945+0000 7f4f27fff640 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d21bac0 0x55be4d203440 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f3479223f6713 server_cookie=34abafc91495646d in_seq=0 out_seq=0 2023-10-16T14:32:33.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.945+0000 7f4f0effd640 10 client.6516 ms_handle_connect on v2:172.21.15.170:6835/3822716445 2023-10-16T14:32:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f0effd640 1 -- 172.21.15.29:0/428171916 <== mds.0 v2:172.21.15.170:6835/3822716445 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55be4d0facc0 con 0x55be4d21bac0 2023-10-16T14:32:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f0effd640 10 client.6516 handle_command_reply: tid=0 2023-10-16T14:32:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 client.6516 shutdown 2023-10-16T14:32:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 msgr2=0x55be4d206160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d206160 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4f200586d0 tx=0x7f4f20009f20 comp rx=0 tx=0).stop 2023-10-16T14:32:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 shutdown_connections 2023-10-16T14:32:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d21bac0 0x55be4d203440 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1ea360 0x55be4d207ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f0effd640 0 client.6516 ms_handle_reset on v2:172.21.15.170:6835/3822716445 2023-10-16T14:32:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 --2- 172.21.15.29:0/428171916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1e9f80 0x55be4d206160 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 >> 172.21.15.29:0/428171916 conn(0x55be4d1bd3c0 msgr2=0x55be4d1c0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 shutdown_connections 2023-10-16T14:32:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 1 -- 172.21.15.29:0/428171916 wait complete. 2023-10-16T14:32:33.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.947+0000 7f4f393f6740 20 client.6516 trim_cache size 0 max 16384 2023-10-16T14:32:33.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.964+0000 7f4f393f6740 1 Processor -- start 2023-10-16T14:32:33.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.964+0000 7f4f393f6740 1 -- start start 2023-10-16T14:32:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d221ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d2016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 0x55be4d1ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f393f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4ce8cf20 con 0x55be4d1ea360 2023-10-16T14:32:33.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d203f70 con 0x55be4d1ddd40 2023-10-16T14:32:33.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be4d0f5860 con 0x55be4d1e9f80 2023-10-16T14:32:33.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f34e01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 0x55be4d1ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f34e01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 0x55be4d1ef1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43000/0 (socket says 172.21.15.29:43000) 2023-10-16T14:32:33.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.965+0000 7f4f34e01640 1 -- 172.21.15.29:0/64188815 learned_addr learned my addr 172.21.15.29:0/64188815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d221ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f27fff640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d2016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f34e01640 1 -- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 msgr2=0x55be4d2016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f34e01640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d2016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f34e01640 1 -- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 msgr2=0x55be4d221ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f34e01640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d221ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f34e01640 1 -- 172.21.15.29:0/64188815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be4d0c5b80 con 0x55be4d1ea360 2023-10-16T14:32:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f27fff640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d2016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d221ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f34e01640 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 0x55be4d1ef1b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4f280071b0 tx=0x7f4f28007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c6d3d529e1f1ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f267fc640 1 -- 172.21.15.29:0/64188815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f280019b0 con 0x55be4d1ea360 2023-10-16T14:32:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f267fc640 1 -- 172.21.15.29:0/64188815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f28001b30 con 0x55be4d1ea360 2023-10-16T14:32:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.966+0000 7f4f267fc640 1 -- 172.21.15.29:0/64188815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28001ee0 con 0x55be4d1ea360 2023-10-16T14:32:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 -- 172.21.15.29:0/64188815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 msgr2=0x55be4d1ef1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 0x55be4d1ef1b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4f280071b0 tx=0x7f4f28007670 comp rx=0 tx=0).stop 2023-10-16T14:32:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 -- 172.21.15.29:0/64188815 shutdown_connections 2023-10-16T14:32:33.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ea360 0x55be4d1ef1b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d2016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 --2- 172.21.15.29:0/64188815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ddd40 0x55be4d221ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 -- 172.21.15.29:0/64188815 >> 172.21.15.29:0/64188815 conn(0x55be4d1bd3c0 msgr2=0x55be4d1de1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 -- 172.21.15.29:0/64188815 shutdown_connections 2023-10-16T14:32:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.967+0000 7f4f393f6740 1 -- 172.21.15.29:0/64188815 wait complete. 2023-10-16T14:32:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.968+0000 7f4f393f6740 1 -- start start 2023-10-16T14:32:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.968+0000 7f4f393f6740 1 Processor -- start 2023-10-16T14:32:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d1de710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d202860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d26ff00 con 0x55be4d1ddd40 2023-10-16T14:32:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4d270110 con 0x55be4d1ea360 2023-10-16T14:32:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be4d2703f0 con 0x55be4d1e9f80 2023-10-16T14:32:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f393f6740 10 client.0 _ll_register_callbacks cb 0x7f4f20057f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:32:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43004/0 (socket says 172.21.15.29:43004) 2023-10-16T14:32:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.969+0000 7f4f2ffff640 1 -- 172.21.15.29:0/3041960407 learned_addr learned my addr 172.21.15.29:0/3041960407 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f2ffff640 1 -- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 msgr2=0x55be4d1de710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d1de710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f2ffff640 1 -- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 msgr2=0x55be4d202860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d202860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f2ffff640 1 -- 172.21.15.29:0/3041960407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2800f040 con 0x55be4d1ddd40 2023-10-16T14:32:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f2ffff640 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4f2005c820 tx=0x7f4f20054980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f237749f64b91f32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f0effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:32:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f0effd640 1 -- 172.21.15.29:0/3041960407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20057c10 con 0x55be4d1ddd40 2023-10-16T14:32:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f0effd640 1 -- 172.21.15.29:0/3041960407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f2005c170 con 0x55be4d1ddd40 2023-10-16T14:32:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f20087d40 con 0x55be4d1ddd40 2023-10-16T14:32:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f0effd640 1 -- 172.21.15.29:0/3041960407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f20058050 con 0x55be4d1ddd40 2023-10-16T14:32:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f393f6740 10 client.6522 fetch_fsmap learned FSMap version 33 2023-10-16T14:32:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.970+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55be4d0eb170 con 0x55be4d1ddd40 2023-10-16T14:32:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f0effd640 1 -- 172.21.15.29:0/3041960407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20087d40 con 0x55be4d1ddd40 2023-10-16T14:32:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f0effd640 1 -- 172.21.15.29:0/3041960407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f20011050 con 0x55be4d1ddd40 2023-10-16T14:32:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f393f6740 10 client.6522 fetch_fsmap finished waiting for FSMap version 33 2023-10-16T14:32:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f393f6740 10 client.6522 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:32:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f393f6740 20 client.6522 populate_metadata read hostname 'smithi029' 2023-10-16T14:32:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f393f6740 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d202da0 0x55be4d203180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f393f6740 4 client.6522 mds_command: new command op to 5584 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:32:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 --> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55be4d121700 con 0x55be4d202da0 2023-10-16T14:32:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.971+0000 7f4f27fff640 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d202da0 0x55be4d203180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.972+0000 7f4f27fff640 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d202da0 0x55be4d203180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5960a068c8057ba1 server_cookie=50d612a4696cea91 in_seq=0 out_seq=0 2023-10-16T14:32:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.972+0000 7f4f0effd640 10 client.6522 ms_handle_connect on v2:172.21.15.170:6835/3822716445 2023-10-16T14:32:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f0effd640 1 -- 172.21.15.29:0/3041960407 <== mds.0 v2:172.21.15.170:6835/3822716445 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x55be4d121700 con 0x55be4d202da0 2023-10-16T14:32:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f0effd640 10 client.6522 handle_command_reply: tid=0 2023-10-16T14:32:33.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 client.6522 shutdown 2023-10-16T14:32:33.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 msgr2=0x55be4d1de1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4f2005c820 tx=0x7f4f20054980 comp rx=0 tx=0).stop 2023-10-16T14:32:33.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 shutdown_connections 2023-10-16T14:32:33.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:6835/3822716445,v1:172.21.15.170:6837/3822716445] conn(0x55be4d202da0 0x55be4d203180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55be4d1ea360 0x55be4d202860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55be4d1e9f80 0x55be4d1de710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 --2- 172.21.15.29:0/3041960407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55be4d1ddd40 0x55be4d1de1d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 >> 172.21.15.29:0/3041960407 conn(0x55be4d1bd3c0 msgr2=0x55be4d1c3050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f0effd640 0 client.6522 ms_handle_reset on v2:172.21.15.170:6835/3822716445 2023-10-16T14:32:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 shutdown_connections 2023-10-16T14:32:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.973+0000 7f4f393f6740 1 -- 172.21.15.29:0/3041960407 wait complete. 2023-10-16T14:32:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f393f6740 20 client.6522 trim_cache size 0 max 16384 2023-10-16T14:32:33.991 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:32:33.991 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:32:33.991 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:32:33.991 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 5584, 2023-10-16T14:32:33.991 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:32:33.992 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:32:33.992 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:32:33.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.799773064, 2023-10-16T14:32:33.992 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 33, 2023-10-16T14:32:33.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 84, 2023-10-16T14:32:33.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 84, 2023-10-16T14:32:33.993 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.700569608 2023-10-16T14:32:33.993 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:32:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f4f1006af90 msgr2=0x7f4f1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f4f1006af90 0x7f4f1006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f18036000 tx=0x7f4f18008030 comp rx=0 tx=0).stop 2023-10-16T14:32:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 msgr2=0x7f4f301f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 0x7f4f301f3030 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000e870 tx=0x7f4f2000ed30 comp rx=0 tx=0).stop 2023-10-16T14:32:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 shutdown_connections 2023-10-16T14:32:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f4f1006af90 0x7f4f1006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f3015ad70 0x7f4f300c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f300ca3b0 0x7f4f30159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 --2- 172.21.15.29:0/2374720090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f300c9ca0 0x7f4f301f3030 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.974+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 >> 172.21.15.29:0/2374720090 conn(0x7f4f300ce9b0 msgr2=0x7f4f3015ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.975+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 shutdown_connections 2023-10-16T14:32:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:33.975+0000 7f4f36864640 1 -- 172.21.15.29:0/2374720090 wait complete. 2023-10-16T14:32:33.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:32:34.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.209+0000 7f2aaae97640 1 Processor -- start 2023-10-16T14:32:34.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.209+0000 7f2aaae97640 1 -- start start 2023-10-16T14:32:34.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aaae97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2aa40cd0c0 0x7f2aa41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:34.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aaae97640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa415e3e0 con 0x7f2aa40b0410 2023-10-16T14:32:34.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aaae97640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa40cd4a0 con 0x7f2aa415db00 2023-10-16T14:32:34.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aaae97640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa415e100 con 0x7f2aa40cd0c0 2023-10-16T14:32:34.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aa8c33640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2aa40cd0c0 0x7f2aa41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:34.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aa8c33640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2aa40cd0c0 0x7f2aa41615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43008/0 (socket says 172.21.15.29:43008) 2023-10-16T14:32:34.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.211+0000 7f2aa8c33640 1 -- 172.21.15.29:0/1030956898 learned_addr learned my addr 172.21.15.29:0/1030956898 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:34.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.212+0000 7f2aa8c33640 1 -- 172.21.15.29:0/1030956898 >> v1:172.21.15.170:6789/0 conn(0x7f2aa415db00 legacy=0x7f2aa406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:34.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.212+0000 7f2aa8c33640 1 -- 172.21.15.29:0/1030956898 >> v1:172.21.15.29:6789/0 conn(0x7f2aa40b0410 legacy=0x7f2aa415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:34.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.212+0000 7f2aa8c33640 1 -- 172.21.15.29:0/1030956898 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa4160b10 con 0x7f2aa40cd0c0 2023-10-16T14:32:34.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.213+0000 7f2aa8c33640 1 --2- 172.21.15.29:0/1030956898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa41615f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009e50 tx=0x7f2aa002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df8aa9b692f43d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:34.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.213+0000 7f2a9b7fe640 1 -- 172.21.15.29:0/1030956898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa002ca80 con 0x7f2aa40cd0c0 2023-10-16T14:32:34.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.213+0000 7f2a9b7fe640 1 -- 172.21.15.29:0/1030956898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa002cbe0 con 0x7f2aa40cd0c0 2023-10-16T14:32:34.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.214+0000 7f2a9b7fe640 1 -- 172.21.15.29:0/1030956898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00343f0 con 0x7f2aa40cd0c0 2023-10-16T14:32:34.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.214+0000 7f2aaae97640 1 -- 172.21.15.29:0/1030956898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40cd0c0 msgr2=0x7f2aa41615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:34.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.214+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1030956898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa41615f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009e50 tx=0x7f2aa002bee0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.214+0000 7f2aaae97640 1 -- 172.21.15.29:0/1030956898 shutdown_connections 2023-10-16T14:32:34.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.215+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1030956898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa41615f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.215+0000 7f2aaae97640 1 -- 172.21.15.29:0/1030956898 >> 172.21.15.29:0/1030956898 conn(0x7f2aa40cb410 msgr2=0x7f2aa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:34.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.215+0000 7f2aaae97640 1 -- 172.21.15.29:0/1030956898 shutdown_connections 2023-10-16T14:32:34.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.215+0000 7f2aaae97640 1 -- 172.21.15.29:0/1030956898 wait complete. 2023-10-16T14:32:34.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.216+0000 7f2aaae97640 1 Processor -- start 2023-10-16T14:32:34.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.216+0000 7f2aaae97640 1 -- start start 2023-10-16T14:32:34.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.217+0000 7f2aaae97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2aa40b0410 0x7f2aa4154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:34.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.217+0000 7f2aaae97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa4153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:34.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.217+0000 7f2aaae97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 0x7f2aa40ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:34.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.217+0000 7f2aaae97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41549c0 con 0x7f2aa415db00 2023-10-16T14:32:34.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.217+0000 7f2aaae97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa415fbe0 con 0x7f2aa40cd0c0 2023-10-16T14:32:34.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.217+0000 7f2aaae97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa415fec0 con 0x7f2aa40b0410 2023-10-16T14:32:34.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 0x7f2aa40ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 0x7f2aa40ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43012/0 (socket says 172.21.15.29:43012) 2023-10-16T14:32:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 -- 172.21.15.29:0/1178646568 learned_addr learned my addr 172.21.15.29:0/1178646568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa8c33640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2aa40b0410 0x7f2aa4154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2a9bfff640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa4153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 -- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2aa40b0410 msgr2=0x7f2aa4154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2aa40b0410 0x7f2aa4154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 -- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40cd0c0 msgr2=0x7f2aa4153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:34.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa4153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.218+0000 7f2aa9434640 1 -- 172.21.15.29:0/1178646568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0009b30 con 0x7f2aa415db00 2023-10-16T14:32:34.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.219+0000 7f2aa9434640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 0x7f2aa40ca010 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b410 tx=0x7f2a9c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:34.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.219+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c004020 con 0x7f2aa415db00 2023-10-16T14:32:34.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.219+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c004180 con 0x7f2aa415db00 2023-10-16T14:32:34.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.219+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c011630 con 0x7f2aa415db00 2023-10-16T14:32:34.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.219+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa41601a0 con 0x7f2aa415db00 2023-10-16T14:32:34.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.219+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa4160760 con 0x7f2aa415db00 2023-10-16T14:32:34.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.221+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa404e160 con 0x7f2aa415db00 2023-10-16T14:32:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.221+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a9c011790 con 0x7f2aa415db00 2023-10-16T14:32:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.222+0000 7f2a997fa640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f2a9006af90 0x7f2a9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.222+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a9c0893b0 con 0x7f2aa415db00 2023-10-16T14:32:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.223+0000 7f2aa8c33640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f2a9006af90 0x7f2a9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.224+0000 7f2aa8c33640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f2a9006af90 0x7f2a9006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0035000 tx=0x7f2aa0030000 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:34.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.227+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2a9c0543e0 con 0x7f2aa415db00 2023-10-16T14:32:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.455+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2aa40b4910 con 0x7f2aa415db00 2023-10-16T14:32:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.456+0000 7f2a997fa640 1 -- 172.21.15.29:0/1178646568 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2a9c057a00 con 0x7f2aa415db00 2023-10-16T14:32:34.460 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:32:34.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f2a9006af90 msgr2=0x7f2a9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:34.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f2a9006af90 0x7f2a9006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0035000 tx=0x7f2aa0030000 comp rx=0 tx=0).stop 2023-10-16T14:32:34.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 msgr2=0x7f2aa40ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 0x7f2aa40ca010 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b410 tx=0x7f2a9c00b8d0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 shutdown_connections 2023-10-16T14:32:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f2a9006af90 0x7f2a9006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa415db00 0x7f2aa40ca010 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40cd0c0 0x7f2aa4153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 --2- 172.21.15.29:0/1178646568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2aa40b0410 0x7f2aa4154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:34.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 >> 172.21.15.29:0/1178646568 conn(0x7f2aa40cb410 msgr2=0x7f2aa415c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:34.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 shutdown_connections 2023-10-16T14:32:34.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:34.459+0000 7f2aaae97640 1 -- 172.21.15.29:0/1178646568 wait complete. 2023-10-16T14:32:34.481 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:32:34.481 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:32:34.481 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:32:34.482 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:32:34.482 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:32:34.483 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:32:34.483 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:32:34.483 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:32:34.484 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:32:34.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:34.484 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:32:34.536 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:32:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:32:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:32:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:32:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.537 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:32:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:32:34.538 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:32:34.538 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:32:34.538 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:32:34.538 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:32:34.538 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:32:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:32:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:32:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:32:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41370sec preferred_lft 41370sec 2023-10-16T14:32:34.539 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:32:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.540 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:32:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:32:34.540 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:32:34.540 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:32:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:32:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:32:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:32:34.541 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.541 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:32:34.542 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:32:34.542 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:32:34.542 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.542 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:32:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:32:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:32:34.543 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:32:34.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:34.544 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:32:34.640 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:32:34.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:34.645 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:32:34.703 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:32:34.704 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:32:34.704 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:32:34.704 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:32:34.705 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:32:34.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:34.706 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:32:34.707 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:32:34.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:32:34.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:32:34.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:32:34.709 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:32:34.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:34.913 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:32:34.970 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:32:34.971 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:32:34.972 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:32:34.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:34.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:32:35.080 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:32:35.081 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:32:35.142 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:32:35.143 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:32:35.143 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:32:35.193 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:32:35.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:35.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:32:35.337 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:32:35.337 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:32:35.337 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:32:35.360 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:32:35.360 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:32:35.360 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:32:35.361 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:32:35.361 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:32:35.387 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:32:35.388 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:32:35.388 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:32:35.388 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:32:35.388 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:32:35.389 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:32:35.389 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:32:35.389 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:32:35.390 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:32:35.390 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:32:35.391 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:32:35.391 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:32:35.391 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:32:35.395 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:32:35.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:35.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:32:35.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:35.496 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:32:35.587 INFO:teuthology.orchestra.run.smithi029.stdout:client6531 2023-10-16T14:32:35.590 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:35.590 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:35.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:32:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.879+0000 7f6b0759e640 1 Processor -- start 2023-10-16T14:32:35.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.879+0000 7f6b0759e640 1 -- start start 2023-10-16T14:32:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6b0815cc80 0x7f6b080cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080d12f0 con 0x7f6b08072170 2023-10-16T14:32:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080d06e0 con 0x7f6b080cdd60 2023-10-16T14:32:35.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b080d0500 con 0x7f6b0815cc80 2023-10-16T14:32:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b06d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6b08072170 0x7f6b0806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44310/0 (socket says 172.21.15.29:44310) 2023-10-16T14:32:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b06d9d640 1 -- 172.21.15.29:0/456124982 learned_addr learned my addr 172.21.15.29:0/456124982 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0659c640 1 --2- 172.21.15.29:0/456124982 >> v2:172.21.15.170:3300/0 conn(0x7f6b0815cc80 0x7f6b080cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035327815 0 0) 0x7f6b080d12f0 con 0x7f6b08072170 2023-10-16T14:32:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.881+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af0003520 con 0x7f6b08072170 2023-10-16T14:32:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3216203158 0 0) 0x7f6af0003520 con 0x7f6b08072170 2023-10-16T14:32:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 >> v2:172.21.15.170:3300/0 conn(0x7f6b0815cc80 msgr2=0x7f6b080cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 --2- 172.21.15.29:0/456124982 >> v2:172.21.15.170:3300/0 conn(0x7f6b0815cc80 0x7f6b080cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 >> v1:172.21.15.170:6789/0 conn(0x7f6b080cdd60 legacy=0x7f6b08071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080d09c0 con 0x7f6b08072170 2023-10-16T14:32:35.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6b00001670 con 0x7f6b08072170 2023-10-16T14:32:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 >> v1:172.21.15.29:6789/0 conn(0x7f6b08072170 legacy=0x7f6b0806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6af0003970 0x7f6af0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6af00047c0 0x7f6af0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6af000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080d0500 con 0x7f6af0005180 2023-10-16T14:32:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af000c020 con 0x7f6af0003970 2023-10-16T14:32:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af000c1e0 con 0x7f6af00047c0 2023-10-16T14:32:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.882+0000 7f6b06d9d640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6af000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b0659c640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6af0003970 0x7f6af0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b05d9b640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6af00047c0 0x7f6af0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b06d9d640 1 -- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6af00047c0 msgr2=0x7f6af0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b06d9d640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6af00047c0 0x7f6af0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b06d9d640 1 -- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6af0003970 msgr2=0x7f6af0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:35.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b06d9d640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6af0003970 0x7f6af0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.883+0000 7f6b06d9d640 1 -- 172.21.15.29:0/456124982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0003520 con 0x7f6af0005180 2023-10-16T14:32:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.884+0000 7f6b06d9d640 1 -- 172.21.15.29:0/456124982 reap_dead start 2023-10-16T14:32:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.884+0000 7f6b06d9d640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6af000b980 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c190 tx=0x7f6af00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126348291ea02b5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.884+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00005ad0 con 0x7f6af0005180 2023-10-16T14:32:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.884+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b00005c70 con 0x7f6af0005180 2023-10-16T14:32:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.884+0000 7f6b0559a640 1 -- 172.21.15.29:0/456124982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0002c8b0 con 0x7f6af0005180 2023-10-16T14:32:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 -- 172.21.15.29:0/456124982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 msgr2=0x7f6af000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6af000b980 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c190 tx=0x7f6af00057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 -- 172.21.15.29:0/456124982 shutdown_connections 2023-10-16T14:32:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 --2- 172.21.15.29:0/456124982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6af000b980 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 -- 172.21.15.29:0/456124982 >> 172.21.15.29:0/456124982 conn(0x7f6b08065ed0 msgr2=0x7f6b080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 -- 172.21.15.29:0/456124982 shutdown_connections 2023-10-16T14:32:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.885+0000 7f6b0759e640 1 -- 172.21.15.29:0/456124982 wait complete. 2023-10-16T14:32:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 Processor -- start 2023-10-16T14:32:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 -- start start 2023-10-16T14:32:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6b08149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b080af010 0x7f6b080af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b080b4f60 0x7f6b080b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08159040 con 0x7f6af0005180 2023-10-16T14:32:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080af9d0 con 0x7f6b080b4f60 2023-10-16T14:32:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.888+0000 7f6b0759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0814a240 con 0x7f6b080af010 2023-10-16T14:32:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6b08149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6b08149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43038/0 (socket says 172.21.15.29:43038) 2023-10-16T14:32:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 -- 172.21.15.29:0/2640778711 learned_addr learned my addr 172.21.15.29:0/2640778711 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b05d9b640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b080af010 0x7f6b080af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 -- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b080af010 msgr2=0x7f6b080af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b06d9d640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b080b4f60 0x7f6b080b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b080af010 0x7f6b080af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 -- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b080b4f60 msgr2=0x7f6b080b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b080b4f60 0x7f6b080b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b0659c640 1 -- 172.21.15.29:0/2640778711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08149c60 con 0x7f6af0005180 2023-10-16T14:32:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.889+0000 7f6b05d9b640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b080af010 0x7f6b080af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.890+0000 7f6b0659c640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6b08149250 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009fc0 tx=0x7f6af800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.890+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af800e040 con 0x7f6af0005180 2023-10-16T14:32:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.890+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af800f040 con 0x7f6af0005180 2023-10-16T14:32:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.890+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8015870 con 0x7f6af0005180 2023-10-16T14:32:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.890+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b080b5920 con 0x7f6af0005180 2023-10-16T14:32:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.890+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b08149ed0 con 0x7f6af0005180 2023-10-16T14:32:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.891+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0804e9b0 con 0x7f6af0005180 2023-10-16T14:32:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.892+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6af80159d0 con 0x7f6af0005180 2023-10-16T14:32:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.893+0000 7f6af6ffd640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ae806af90 0x7f6ae806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.893+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6af808a640 con 0x7f6af0005180 2023-10-16T14:32:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.894+0000 7f6b05d9b640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ae806af90 0x7f6ae806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.895+0000 7f6b05d9b640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ae806af90 0x7f6ae806d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b080afcf0 tx=0x7f6afc009040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:35.898+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6af80554c0 con 0x7f6af0005180 2023-10-16T14:32:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.113+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 --> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b080d0f80 con 0x7f6ae806af90 2023-10-16T14:32:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.118+0000 7f6af6ffd640 1 -- 172.21.15.29:0/2640778711 <== mgr.5135 v2:172.21.15.170:6832/885795747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6afc00d050 con 0x7f6ae806af90 2023-10-16T14:32:36.121 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:32:36.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:32:36.121 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:32:36.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:32:36.122 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:32:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.119+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ae806af90 msgr2=0x7f6ae806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.119+0000 7f6b0759e640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ae806af90 0x7f6ae806d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b080afcf0 tx=0x7f6afc009040 comp rx=0 tx=0).stop 2023-10-16T14:32:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 msgr2=0x7f6b08149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6b08149250 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009fc0 tx=0x7f6af800f580 comp rx=0 tx=0).stop 2023-10-16T14:32:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 shutdown_connections 2023-10-16T14:32:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f6ae806af90 0x7f6ae806d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b080b4f60 0x7f6b080b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b080af010 0x7f6b080af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 --2- 172.21.15.29:0/2640778711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6af0005180 0x7f6b08149250 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 >> 172.21.15.29:0/2640778711 conn(0x7f6b08065ed0 msgr2=0x7f6b0815da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:36.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 shutdown_connections 2023-10-16T14:32:36.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.121+0000 7f6b0759e640 1 -- 172.21.15.29:0/2640778711 wait complete. 2023-10-16T14:32:36.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:32:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.365+0000 7f03b0959640 1 Processor -- start 2023-10-16T14:32:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.365+0000 7f03b0959640 1 -- start start 2023-10-16T14:32:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03b0959640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac15cc80 0x7f03ac0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03b0959640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03ac0cdd60 0x7f03ac071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03b0959640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f03ac071fa0 0x7f03ac1607e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03b0959640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0d0500 con 0x7f03ac0cdd60 2023-10-16T14:32:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03b0959640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0ce1c0 con 0x7f03ac071fa0 2023-10-16T14:32:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03b0959640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0d12f0 con 0x7f03ac15cc80 2023-10-16T14:32:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.367+0000 7f03aaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03ac0cdd60 0x7f03ac071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03ac0cdd60 0x7f03ac071a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43050/0 (socket says 172.21.15.29:43050) 2023-10-16T14:32:36.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 -- 172.21.15.29:0/2898554743 learned_addr learned my addr 172.21.15.29:0/2898554743 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03abfff640 1 --2- 172.21.15.29:0/2898554743 >> v2:172.21.15.170:3300/0 conn(0x7f03ac071fa0 0x7f03ac1607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03ab7fe640 1 --2- 172.21.15.29:0/2898554743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac15cc80 0x7f03ac0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 -- 172.21.15.29:0/2898554743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac15cc80 msgr2=0x7f03ac0cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 --2- 172.21.15.29:0/2898554743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac15cc80 0x7f03ac0cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 -- 172.21.15.29:0/2898554743 >> v2:172.21.15.170:3300/0 conn(0x7f03ac071fa0 msgr2=0x7f03ac1607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 --2- 172.21.15.29:0/2898554743 >> v2:172.21.15.170:3300/0 conn(0x7f03ac071fa0 0x7f03ac1607e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.368+0000 7f03aaffd640 1 -- 172.21.15.29:0/2898554743 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac072410 con 0x7f03ac0cdd60 2023-10-16T14:32:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.369+0000 7f03aaffd640 1 --2- 172.21.15.29:0/2898554743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac0cdd60 0x7f03ac071a60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b27f0 tx=0x7f03a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164b4708f71c7c1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03aa7fc640 1 -- 172.21.15.29:0/2898554743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0031870 con 0x7f03ac0cdd60 2023-10-16T14:32:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03aa7fc640 1 -- 172.21.15.29:0/2898554743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a00319d0 con 0x7f03ac0cdd60 2023-10-16T14:32:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03aa7fc640 1 -- 172.21.15.29:0/2898554743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0031d60 con 0x7f03ac0cdd60 2023-10-16T14:32:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 -- 172.21.15.29:0/2898554743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac0cdd60 msgr2=0x7f03ac071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 --2- 172.21.15.29:0/2898554743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac0cdd60 0x7f03ac071a60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0b27f0 tx=0x7f03a002b910 comp rx=0 tx=0).stop 2023-10-16T14:32:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 -- 172.21.15.29:0/2898554743 shutdown_connections 2023-10-16T14:32:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 --2- 172.21.15.29:0/2898554743 >> v2:172.21.15.170:3300/0 conn(0x7f03ac071fa0 0x7f03ac1607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 --2- 172.21.15.29:0/2898554743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac0cdd60 0x7f03ac071a60 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 --2- 172.21.15.29:0/2898554743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac15cc80 0x7f03ac0cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 -- 172.21.15.29:0/2898554743 >> 172.21.15.29:0/2898554743 conn(0x7f03ac065ed0 msgr2=0x7f03ac0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:36.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 -- 172.21.15.29:0/2898554743 shutdown_connections 2023-10-16T14:32:36.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.371+0000 7f03b0959640 1 -- 172.21.15.29:0/2898554743 wait complete. 2023-10-16T14:32:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.372+0000 7f03b0959640 1 Processor -- start 2023-10-16T14:32:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.373+0000 7f03b0959640 1 -- start start 2023-10-16T14:32:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03b0959640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ac071fa0 0x7f03ac157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03b0959640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac0cdd60 0x7f03ac15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03b0959640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 0x7f03ac0b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03b0959640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0b4400 con 0x7f03ac15cc80 2023-10-16T14:32:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03b0959640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0b46e0 con 0x7f03ac071fa0 2023-10-16T14:32:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03b0959640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0b49c0 con 0x7f03ac0cdd60 2023-10-16T14:32:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 0x7f03ac0b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 0x7f03ac0b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43052/0 (socket says 172.21.15.29:43052) 2023-10-16T14:32:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.374+0000 7f03abfff640 1 -- 172.21.15.29:0/2041880709 learned_addr learned my addr 172.21.15.29:0/2041880709 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03ab7fe640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ac071fa0 0x7f03ac157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03aaffd640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac0cdd60 0x7f03ac15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03abfff640 1 -- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac0cdd60 msgr2=0x7f03ac15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03abfff640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac0cdd60 0x7f03ac15f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03abfff640 1 -- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ac071fa0 msgr2=0x7f03ac157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03abfff640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ac071fa0 0x7f03ac157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03abfff640 1 -- 172.21.15.29:0/2041880709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a00094d0 con 0x7f03ac15cc80 2023-10-16T14:32:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03abfff640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 0x7f03ac0b51c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f03a400cd70 tx=0x7f03a400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4003b90 con 0x7f03ac15cc80 2023-10-16T14:32:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a4003cf0 con 0x7f03ac15cc80 2023-10-16T14:32:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4004240 con 0x7f03ac15cc80 2023-10-16T14:32:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.375+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac0b4ca0 con 0x7f03ac15cc80 2023-10-16T14:32:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.376+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ac15fe90 con 0x7f03ac15cc80 2023-10-16T14:32:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.377+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ac04e9b0 con 0x7f03ac15cc80 2023-10-16T14:32:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.378+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03a40043a0 con 0x7f03ac15cc80 2023-10-16T14:32:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.379+0000 7f0393fff640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f039806af90 0x7f039806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.379+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03a4012070 con 0x7f03ac15cc80 2023-10-16T14:32:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.380+0000 7f03ab7fe640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f039806af90 0x7f039806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.382+0000 7f03ab7fe640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f039806af90 0x7f039806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f039c006fd0 tx=0x7f039c008040 comp rx=0 tx=0).ready entity=mgr.5135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.384+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03a4055120 con 0x7f03ac15cc80 2023-10-16T14:32:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:36.628+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f03ac0b4540 con 0x7f03ac15cc80 2023-10-16T14:32:37.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.224+0000 7f0393fff640 1 -- 172.21.15.29:0/2041880709 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v30) v1 ==== 92+0+0 (secure 0 0 0) 0x7f03ac0b4540 con 0x7f03ac15cc80 2023-10-16T14:32:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.227+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f039806af90 msgr2=0x7f039806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.227+0000 7f03b0959640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f039806af90 0x7f039806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f039c006fd0 tx=0x7f039c008040 comp rx=0 tx=0).stop 2023-10-16T14:32:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.227+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 msgr2=0x7f03ac0b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.227+0000 7f03b0959640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 0x7f03ac0b51c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f03a400cd70 tx=0x7f03a400a660 comp rx=0 tx=0).stop 2023-10-16T14:32:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 shutdown_connections 2023-10-16T14:32:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f039806af90 0x7f039806d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ac15cc80 0x7f03ac0b51c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03ac0cdd60 0x7f03ac15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 --2- 172.21.15.29:0/2041880709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ac071fa0 0x7f03ac157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 >> 172.21.15.29:0/2041880709 conn(0x7f03ac065ed0 msgr2=0x7f03ac0b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.228+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 shutdown_connections 2023-10-16T14:32:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.229+0000 7f03b0959640 1 -- 172.21.15.29:0/2041880709 wait complete. 2023-10-16T14:32:37.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:32:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.457+0000 7f6726064640 1 Processor -- start 2023-10-16T14:32:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.457+0000 7f6726064640 1 -- start start 2023-10-16T14:32:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.458+0000 7f6726064640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6720159ab0 0x7f67200ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.458+0000 7f6726064640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6720159e90 0x7f672006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f6726064640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67200cd1f0 con 0x7f672006c140 2023-10-16T14:32:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f6726064640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672015d890 con 0x7f6720159e90 2023-10-16T14:32:37.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f6726064640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67200cd4d0 con 0x7f6720159ab0 2023-10-16T14:32:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f6717fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6720159e90 0x7f672006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f6717fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6720159e90 0x7f672006fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43080/0 (socket says 172.21.15.29:43080) 2023-10-16T14:32:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f6717fff640 1 -- 172.21.15.29:0/2370744636 learned_addr learned my addr 172.21.15.29:0/2370744636 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.459+0000 7f671f7fe640 1 --2- 172.21.15.29:0/2370744636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6720159ab0 0x7f67200ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.460+0000 7f6717fff640 1 -- 172.21.15.29:0/2370744636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6720159ab0 msgr2=0x7f67200ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.460+0000 7f6717fff640 1 --2- 172.21.15.29:0/2370744636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6720159ab0 0x7f67200ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.460+0000 7f6717fff640 1 -- 172.21.15.29:0/2370744636 >> v1:172.21.15.29:6789/0 conn(0x7f672006c140 legacy=0x7f672015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.460+0000 7f6717fff640 1 -- 172.21.15.29:0/2370744636 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200cd7b0 con 0x7f6720159e90 2023-10-16T14:32:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.460+0000 7f671effd640 1 -- 172.21.15.29:0/2370744636 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871363200 0 0) 0x7f67200cd1f0 con 0x7f672006c140 2023-10-16T14:32:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.461+0000 7f6717fff640 1 --2- 172.21.15.29:0/2370744636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159e90 0x7f672006fc50 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f67080098a0 tx=0x7f670802b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c4639bb5befb9e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.461+0000 7f671effd640 1 -- 172.21.15.29:0/2370744636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708032640 con 0x7f6720159e90 2023-10-16T14:32:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.461+0000 7f671effd640 1 -- 172.21.15.29:0/2370744636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670802e070 con 0x7f6720159e90 2023-10-16T14:32:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f671effd640 1 -- 172.21.15.29:0/2370744636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080329c0 con 0x7f6720159e90 2023-10-16T14:32:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 -- 172.21.15.29:0/2370744636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159e90 msgr2=0x7f672006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 --2- 172.21.15.29:0/2370744636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159e90 0x7f672006fc50 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f67080098a0 tx=0x7f670802b850 comp rx=0 tx=0).stop 2023-10-16T14:32:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 -- 172.21.15.29:0/2370744636 shutdown_connections 2023-10-16T14:32:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 --2- 172.21.15.29:0/2370744636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159e90 0x7f672006fc50 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 --2- 172.21.15.29:0/2370744636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6720159ab0 0x7f67200ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 -- 172.21.15.29:0/2370744636 >> 172.21.15.29:0/2370744636 conn(0x7f67200ce9b0 msgr2=0x7f67200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.462+0000 7f6726064640 1 -- 172.21.15.29:0/2370744636 shutdown_connections 2023-10-16T14:32:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.463+0000 7f6726064640 1 -- 172.21.15.29:0/2370744636 wait complete. 2023-10-16T14:32:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.464+0000 7f6726064640 1 Processor -- start 2023-10-16T14:32:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.464+0000 7f6726064640 1 -- start start 2023-10-16T14:32:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.465+0000 7f6726064640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f672006c140 0x7f6720158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.465+0000 7f6726064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 0x7f67200b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.465+0000 7f6726064640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6720159e90 0x7f67201f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.465+0000 7f6726064640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200b15d0 con 0x7f6720159ab0 2023-10-16T14:32:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.465+0000 7f6726064640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672015a840 con 0x7f6720159e90 2023-10-16T14:32:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.465+0000 7f6726064640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672015cb50 con 0x7f672006c140 2023-10-16T14:32:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 0x7f67200b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 0x7f67200b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43082/0 (socket says 172.21.15.29:43082) 2023-10-16T14:32:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 -- 172.21.15.29:0/558583704 learned_addr learned my addr 172.21.15.29:0/558583704 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f671f7fe640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f672006c140 0x7f6720158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 -- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f672006c140 msgr2=0x7f6720158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f672006c140 0x7f6720158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 -- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6720159e90 msgr2=0x7f67201f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f671ffff640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6720159e90 0x7f67201f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6720159e90 0x7f67201f5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.466+0000 7f6717fff640 1 -- 172.21.15.29:0/558583704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080094d0 con 0x7f6720159ab0 2023-10-16T14:32:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.467+0000 7f6717fff640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 0x7f67200b36b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6708000c00 tx=0x7f67080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.467+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080455f0 con 0x7f6720159ab0 2023-10-16T14:32:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.467+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6708045750 con 0x7f6720159ab0 2023-10-16T14:32:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.467+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708045b00 con 0x7f6720159ab0 2023-10-16T14:32:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.467+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672015cda0 con 0x7f6720159ab0 2023-10-16T14:32:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.467+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672015a4d0 con 0x7f6720159ab0 2023-10-16T14:32:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.468+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672015d2c0 con 0x7f6720159ab0 2023-10-16T14:32:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.469+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f670803b3f0 con 0x7f6720159ab0 2023-10-16T14:32:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.471+0000 7f671cff9640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 0x7f670006d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.471+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67080abaa0 con 0x7f6720159ab0 2023-10-16T14:32:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.471+0000 7f671f7fe640 1 -- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 msgr2=0x7f670006d520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.471+0000 7f671f7fe640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 0x7f670006d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.475+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f672015d2c0 con 0x7f6720159ab0 2023-10-16T14:32:37.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.670+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f672015a330 con 0x7f6720159ab0 2023-10-16T14:32:37.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.672+0000 7f671f7fe640 1 -- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 msgr2=0x7f670006d520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.672+0000 7f671f7fe640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 0x7f670006d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.675+0000 7f671cff9640 1 -- 172.21.15.29:0/558583704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f672015a330 con 0x7f6720159ab0 2023-10-16T14:32:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 msgr2=0x7f670006d520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 0x7f670006d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 msgr2=0x7f67200b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 0x7f67200b36b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6708000c00 tx=0x7f67080046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 shutdown_connections 2023-10-16T14:32:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f670006b060 0x7f670006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720159ab0 0x7f67200b36b0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6720159e90 0x7f67201f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 --2- 172.21.15.29:0/558583704 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f672006c140 0x7f6720158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 >> 172.21.15.29:0/558583704 conn(0x7f67200ce9b0 msgr2=0x7f672006b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 shutdown_connections 2023-10-16T14:32:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.678+0000 7f6726064640 1 -- 172.21.15.29:0/558583704 wait complete. 2023-10-16T14:32:37.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:32:37.752 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:37.748+0000 7f53fac11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:32:37.800 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:37.798+0000 7f53d841a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:32:37.804 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:37.800+0000 7f53fac11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:32:37.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:37.859+0000 7f53d841a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:32:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.916+0000 7f3068402640 1 Processor -- start 2023-10-16T14:32:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.916+0000 7f3068402640 1 -- start start 2023-10-16T14:32:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.917+0000 7f3068402640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f30600cf710 0x7f30600b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.917+0000 7f3068402640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30600d18c0 con 0x7f30600cfaf0 2023-10-16T14:32:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.917+0000 7f3068402640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30600d3980 con 0x7f30600b6cd0 2023-10-16T14:32:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.917+0000 7f3068402640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30600d3c60 con 0x7f30600cf710 2023-10-16T14:32:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.918+0000 7f306599d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30600cfaf0 0x7f30600d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44350/0 (socket says 172.21.15.29:44350) 2023-10-16T14:32:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.918+0000 7f306599d640 1 -- 172.21.15.29:0/2679204172 learned_addr learned my addr 172.21.15.29:0/2679204172 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.918+0000 7f306619e640 1 --2- 172.21.15.29:0/2679204172 >> v2:172.21.15.170:3300/0 conn(0x7f30600cf710 0x7f30600b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.918+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926825431 0 0) 0x7f30600d18c0 con 0x7f30600cfaf0 2023-10-16T14:32:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.918+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c003520 con 0x7f30600cfaf0 2023-10-16T14:32:37.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176852940 0 0) 0x7f304c003520 con 0x7f30600cfaf0 2023-10-16T14:32:37.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 >> v2:172.21.15.170:3300/0 conn(0x7f30600cf710 msgr2=0x7f30600b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:37.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 --2- 172.21.15.29:0/2679204172 >> v2:172.21.15.170:3300/0 conn(0x7f30600cf710 0x7f30600b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 >> v1:172.21.15.170:6789/0 conn(0x7f30600b6cd0 legacy=0x7f30600d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30600ce630 con 0x7f30600cfaf0 2023-10-16T14:32:37.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3050001670 con 0x7f30600cfaf0 2023-10-16T14:32:37.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 >> v1:172.21.15.29:6789/0 conn(0x7f30600cfaf0 legacy=0x7f30600d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f304c0039c0 0x7f304c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0047c0 0x7f304c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0051d0 0x7f304c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30600d3c60 con 0x7f304c0051d0 2023-10-16T14:32:37.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c00c120 con 0x7f304c0047c0 2023-10-16T14:32:37.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c00c2e0 con 0x7f304c0039c0 2023-10-16T14:32:37.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.919+0000 7f306699f640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0051d0 0x7f304c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306599d640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0047c0 0x7f304c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306619e640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f304c0039c0 0x7f304c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306699f640 1 -- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f304c0039c0 msgr2=0x7f304c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:37.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306699f640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f304c0039c0 0x7f304c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306699f640 1 -- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0047c0 msgr2=0x7f304c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:37.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306699f640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0047c0 0x7f304c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.920+0000 7f306699f640 1 -- 172.21.15.29:0/2679204172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c003520 con 0x7f304c0051d0 2023-10-16T14:32:37.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.921+0000 7f306699f640 1 -- 172.21.15.29:0/2679204172 reap_dead start 2023-10-16T14:32:37.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.921+0000 7f306699f640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0051d0 0x7f304c00b980 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3058001e20 tx=0x7f304c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1906909c17e5fc6a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:37.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.921+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058003b70 con 0x7f304c0051d0 2023-10-16T14:32:37.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.921+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3058003cd0 con 0x7f304c0051d0 2023-10-16T14:32:37.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.921+0000 7f306519c640 1 -- 172.21.15.29:0/2679204172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058005310 con 0x7f304c0051d0 2023-10-16T14:32:37.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 -- 172.21.15.29:0/2679204172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0051d0 msgr2=0x7f304c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0051d0 0x7f304c00b980 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3058001e20 tx=0x7f304c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 -- 172.21.15.29:0/2679204172 shutdown_connections 2023-10-16T14:32:37.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 --2- 172.21.15.29:0/2679204172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0051d0 0x7f304c00b980 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 -- 172.21.15.29:0/2679204172 >> 172.21.15.29:0/2679204172 conn(0x7f30600d2cc0 msgr2=0x7f30600d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:37.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 -- 172.21.15.29:0/2679204172 shutdown_connections 2023-10-16T14:32:37.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.922+0000 7f3068402640 1 -- 172.21.15.29:0/2679204172 wait complete. 2023-10-16T14:32:37.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.924+0000 7f3068402640 1 Processor -- start 2023-10-16T14:32:37.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.924+0000 7f3068402640 1 -- start start 2023-10-16T14:32:37.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f3068402640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0051d0 0x7f30600af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f3068402640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30601e71c0 0x7f3060149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f3068402640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 0x7f306015fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f3068402640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3060149d30 con 0x7f30601498b0 2023-10-16T14:32:37.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f3068402640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306014a220 con 0x7f304c0051d0 2023-10-16T14:32:37.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f3068402640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30601e6400 con 0x7f30601e71c0 2023-10-16T14:32:37.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f306699f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 0x7f306015fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f306699f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 0x7f306015fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43108/0 (socket says 172.21.15.29:43108) 2023-10-16T14:32:37.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.925+0000 7f306699f640 1 -- 172.21.15.29:0/593258224 learned_addr learned my addr 172.21.15.29:0/593258224 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:37.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306619e640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0051d0 0x7f30600af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306599d640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30601e71c0 0x7f3060149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:37.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306699f640 1 -- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30601e71c0 msgr2=0x7f3060149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306699f640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30601e71c0 0x7f3060149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306699f640 1 -- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0051d0 msgr2=0x7f30600af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306699f640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0051d0 0x7f30600af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306699f640 1 -- 172.21.15.29:0/593258224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30601e66e0 con 0x7f30601498b0 2023-10-16T14:32:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306599d640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30601e71c0 0x7f3060149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f306699f640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 0x7f306015fca0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3058001f40 tx=0x7f30580014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058006d50 con 0x7f30601498b0 2023-10-16T14:32:37.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3058007030 con 0x7f30601498b0 2023-10-16T14:32:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30580073c0 con 0x7f30601498b0 2023-10-16T14:32:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.926+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30601e69c0 con 0x7f30601498b0 2023-10-16T14:32:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.927+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30600afe70 con 0x7f30601498b0 2023-10-16T14:32:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.928+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30600b0030 con 0x7f30601498b0 2023-10-16T14:32:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.929+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3058007520 con 0x7f30601498b0 2023-10-16T14:32:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.930+0000 7f3056ffd640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 0x7f304406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.930+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30580ab4e0 con 0x7f30601498b0 2023-10-16T14:32:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.930+0000 7f306619e640 1 -- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 msgr2=0x7f304406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.930+0000 7f306619e640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 0x7f304406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:37.935+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30600b0030 con 0x7f30601498b0 2023-10-16T14:32:38.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.129+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f30600b0030 con 0x7f30601498b0 2023-10-16T14:32:38.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.131+0000 7f306619e640 1 -- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 msgr2=0x7f304406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:38.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.131+0000 7f306619e640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 0x7f304406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:38.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.136+0000 7f3056ffd640 1 -- 172.21.15.29:0/593258224 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f30600b0030 con 0x7f30601498b0 2023-10-16T14:32:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 msgr2=0x7f304406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 0x7f304406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 msgr2=0x7f306015fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 0x7f306015fca0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3058001f40 tx=0x7f30580014a0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 shutdown_connections 2023-10-16T14:32:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f304406afe0 0x7f304406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30601498b0 0x7f306015fca0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30601e71c0 0x7f3060149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 --2- 172.21.15.29:0/593258224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f304c0051d0 0x7f30600af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 >> 172.21.15.29:0/593258224 conn(0x7f30600d2cc0 msgr2=0x7f3060156660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:38.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 shutdown_connections 2023-10-16T14:32:38.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.139+0000 7f3068402640 1 -- 172.21.15.29:0/593258224 wait complete. 2023-10-16T14:32:38.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:38.168 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:32:38.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T14:32:38.250 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.247+0000 7f53fac11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:32:38.303 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.300+0000 7f53fac11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:32:38.353 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.349+0000 7f53fac11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:32:38.353 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.351+0000 7f53d841a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:32:38.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.406+0000 7f53d841a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:32:38.413 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.409+0000 7f53fac11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:32:38.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.441+0000 7f1a5a210640 1 Processor -- start 2023-10-16T14:32:38.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.441+0000 7f1a5a210640 1 -- start start 2023-10-16T14:32:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a5a210640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a5406c850 0x7f1a540ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a5a210640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a54070360 con 0x7f1a54159ab0 2023-10-16T14:32:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a5a210640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a540cd1f0 con 0x7f1a54159e90 2023-10-16T14:32:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a5a210640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5415d890 con 0x7f1a5406c850 2023-10-16T14:32:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a537fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a54159ab0 0x7f1a5406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44362/0 (socket says 172.21.15.29:44362) 2023-10-16T14:32:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a537fe640 1 -- 172.21.15.29:0/1193568100 learned_addr learned my addr 172.21.15.29:0/1193568100 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.443+0000 7f1a53fff640 1 --2- 172.21.15.29:0/1193568100 >> v2:172.21.15.29:3300/0 conn(0x7f1a5406c850 0x7f1a540ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.444+0000 7f1a527fc640 1 -- 172.21.15.29:0/1193568100 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066912242 0 0) 0x7f1a54070360 con 0x7f1a54159ab0 2023-10-16T14:32:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.444+0000 7f1a527fc640 1 -- 172.21.15.29:0/1193568100 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a3c003520 con 0x7f1a54159ab0 2023-10-16T14:32:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.444+0000 7f1a53fff640 1 -- 172.21.15.29:0/1193568100 >> v1:172.21.15.170:6789/0 conn(0x7f1a54159e90 legacy=0x7f1a5406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.444+0000 7f1a53fff640 1 -- 172.21.15.29:0/1193568100 >> v1:172.21.15.29:6789/0 conn(0x7f1a54159ab0 legacy=0x7f1a5406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.444+0000 7f1a53fff640 1 -- 172.21.15.29:0/1193568100 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a540cd4d0 con 0x7f1a5406c850 2023-10-16T14:32:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.445+0000 7f1a53fff640 1 --2- 172.21.15.29:0/1193568100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a5406c850 0x7f1a540ce370 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0097a0 tx=0x7f1a4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd2d7e56f6e3da0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.445+0000 7f1a527fc640 1 -- 172.21.15.29:0/1193568100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c02e070 con 0x7f1a5406c850 2023-10-16T14:32:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.445+0000 7f1a527fc640 1 -- 172.21.15.29:0/1193568100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c0315a0 con 0x7f1a5406c850 2023-10-16T14:32:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.445+0000 7f1a527fc640 1 -- 172.21.15.29:0/1193568100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c031930 con 0x7f1a5406c850 2023-10-16T14:32:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.446+0000 7f1a5a210640 1 -- 172.21.15.29:0/1193568100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a5406c850 msgr2=0x7f1a540ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.446+0000 7f1a5a210640 1 --2- 172.21.15.29:0/1193568100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a5406c850 0x7f1a540ce370 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0097a0 tx=0x7f1a4c02b840 comp rx=0 tx=0).stop 2023-10-16T14:32:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.446+0000 7f1a5a210640 1 -- 172.21.15.29:0/1193568100 shutdown_connections 2023-10-16T14:32:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.446+0000 7f1a5a210640 1 --2- 172.21.15.29:0/1193568100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a5406c850 0x7f1a540ce370 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.446+0000 7f1a5a210640 1 -- 172.21.15.29:0/1193568100 >> 172.21.15.29:0/1193568100 conn(0x7f1a540ce9b0 msgr2=0x7f1a540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:38.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.446+0000 7f1a5a210640 1 -- 172.21.15.29:0/1193568100 shutdown_connections 2023-10-16T14:32:38.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.447+0000 7f1a5a210640 1 -- 172.21.15.29:0/1193568100 wait complete. 2023-10-16T14:32:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.449+0000 7f1a5a210640 1 Processor -- start 2023-10-16T14:32:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.449+0000 7f1a5a210640 1 -- start start 2023-10-16T14:32:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a5a210640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a5406c850 0x7f1a541580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a5a210640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a54159ab0 0x7f1a540c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a5a210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54159e90 0x7f1a540c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a5a210640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a541e4450 con 0x7f1a54159ab0 2023-10-16T14:32:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a5a210640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540c5fc0 con 0x7f1a5406c850 2023-10-16T14:32:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a5a210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a541e2d10 con 0x7f1a54159e90 2023-10-16T14:32:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a54159ab0 0x7f1a540c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a54159ab0 0x7f1a540c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43132/0 (socket says 172.21.15.29:43132) 2023-10-16T14:32:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.451+0000 7f1a52ffd640 1 -- 172.21.15.29:0/2065376199 learned_addr learned my addr 172.21.15.29:0/2065376199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a5406c850 0x7f1a541580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a52ffd640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54159e90 msgr2=0x7f1a540c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a52ffd640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a54159e90 0x7f1a540c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a52ffd640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a5406c850 msgr2=0x7f1a541580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a52ffd640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a5406c850 0x7f1a541580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a52ffd640 1 -- 172.21.15.29:0/2065376199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c009400 con 0x7f1a54159ab0 2023-10-16T14:32:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a5406c850 0x7f1a541580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.452+0000 7f1a52ffd640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a54159ab0 0x7f1a540c5a80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800c730 tx=0x7f1a4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.453+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48003b70 con 0x7f1a54159ab0 2023-10-16T14:32:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.453+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a48003cf0 con 0x7f1a54159ab0 2023-10-16T14:32:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.453+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4800aa90 con 0x7f1a54159ab0 2023-10-16T14:32:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.453+0000 7f1a5a210640 1 -- 172.21.15.29:0/2065376199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a541e3840 con 0x7f1a54159ab0 2023-10-16T14:32:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.453+0000 7f1a5a210640 1 -- 172.21.15.29:0/2065376199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a541e3e00 con 0x7f1a54159ab0 2023-10-16T14:32:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.454+0000 7f1a5a210640 1 -- 172.21.15.29:0/2065376199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a540cde70 con 0x7f1a54159ab0 2023-10-16T14:32:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.455+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1a4800abf0 con 0x7f1a54159ab0 2023-10-16T14:32:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.456+0000 7f1a3bfff640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 0x7f1a4006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.456+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a48089620 con 0x7f1a54159ab0 2023-10-16T14:32:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.457+0000 7f1a537fe640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 msgr2=0x7f1a4006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.457+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 0x7f1a4006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:38.462 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.459+0000 7f53fac11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:32:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.461+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a480545c0 con 0x7f1a54159ab0 2023-10-16T14:32:38.468 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.465+0000 7f53d841a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:32:38.532 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.530+0000 7f53d841a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:32:38.573 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.569+0000 7f53fac11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:32:38.584 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.581+0000 7f53d841a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:32:38.635 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.631+0000 7f53fac11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:38.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.658+0000 7f1a537fe640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 msgr2=0x7f1a4006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:38.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.658+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 0x7f1a4006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:38.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:38.675+0000 7f1a5a210640 1 -- 172.21.15.29:0/2065376199 --> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a540cd7b0 con 0x7f1a4006afe0 2023-10-16T14:32:38.684 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.680+0000 7f53fac11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:32:38.702 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.699+0000 7f53d841a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:32:38.741 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.737+0000 7f53fac11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:32:38.768 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.765+0000 7f53d841a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:38.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:38.787+0000 7f53fac11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:32:38.820 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.817+0000 7f53d841a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:32:38.879 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.876+0000 7f53d841a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:32:38.932 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:38.929+0000 7f53d841a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:32:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:39.059+0000 7f1a537fe640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 msgr2=0x7f1a4006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:39.059+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 0x7f1a4006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:32:39.171 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.167+0000 7f53fac11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:32:39.331 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.328+0000 7f53fac11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:32:39.337 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:39.334+0000 7f53d841a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:32:39.439 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.435+0000 7f53fac11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:32:39.489 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.485+0000 7f53fac11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:32:39.504 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:39.501+0000 7f53d841a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:32:39.596 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.592+0000 7f53fac11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:39.617 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:39.614+0000 7f53d841a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:32:39.670 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:39.667+0000 7f53d841a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:32:39.702 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.698+0000 7f53fac11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:32:39.783 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:39.780+0000 7f53d841a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:39.858 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.854+0000 7f53fac11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:39.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:39.860+0000 7f1a537fe640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 msgr2=0x7f1a4006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/885795747 2023-10-16T14:32:39.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:39.860+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 0x7f1a4006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:32:39.894 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:39.891+0000 7f53d841a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:32:39.907 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:39.903+0000 7f53fac11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:40.026 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:40.022+0000 7f53fac11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:32:40.060 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:40.056+0000 7f53d841a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:40.111 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:40.108+0000 7f53d841a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:40.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:40.234+0000 7f53d841a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:32:40.861 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:40.857+0000 7f53fac11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:32:41.122 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:41.119+0000 7f53d841a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:32:41.155 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.151+0000 7f53fac11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:32:41.212 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.208+0000 7f53fac11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:32:41.325 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.321+0000 7f53fac11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:32:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:41.343+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 31) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f1a48087b40 con 0x7f1a54159ab0 2023-10-16T14:32:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:41.343+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 msgr2=0x7f1a4006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:41.343+0000 7f1a3bfff640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/885795747,v1:172.21.15.170:6833/885795747] conn(0x7f1a4006afe0 0x7f1a4006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:41.445 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:41.442+0000 7f53d841a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:32:41.509 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f537a279640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.509 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f537a279640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.509 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f537a279640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.509 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f537a279640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.510 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f537a279640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.511 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:41.506+0000 7f53d841a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:32:41.512 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f5377233640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.512 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f5377233640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.512 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f5377233640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.513 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f5377233640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.513 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:41.505+0000 7f5377233640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:41.631 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:41.628+0000 7f53d841a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:32:42.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.355+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 32) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1a480560a0 con 0x7f1a54159ab0 2023-10-16T14:32:42.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.356+0000 7f1a3bfff640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a40035880 0x7f1a40074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.356+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 --> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a40035c60 con 0x7f1a40035880 2023-10-16T14:32:42.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.360+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a40035880 0x7f1a40074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.374+0000 7f1a537fe640 1 --2- 172.21.15.29:0/2065376199 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a40035880 0x7f1a40074ea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1a44007fd0 tx=0x7f1a44009040 comp rx=0 tx=0).ready entity=mgr.5692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:42.396 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:42.392+0000 7f5381287640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T14:32:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.393+0000 7f1a3bfff640 1 -- 172.21.15.29:0/2065376199 <== mgr.5692 v2:172.21.15.170:6832/3148236210 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f1a40035c60 con 0x7f1a40035880 2023-10-16T14:32:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T14:32:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:32:42.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:42.418 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T14:32:42.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:32:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.719+0000 7f1a27fff640 1 Processor -- start 2023-10-16T14:32:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.719+0000 7f1a27fff640 1 -- start start 2023-10-16T14:32:42.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.720+0000 7f1a27fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 0x7f1a280cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.720+0000 7f1a27fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1a280ccfb0 0x7f1a280cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.720+0000 7f1a27fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2815b9d0 con 0x7f1a2806c040 2023-10-16T14:32:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.720+0000 7f1a27fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2815bcb0 con 0x7f1a280ccfb0 2023-10-16T14:32:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.720+0000 7f1a27fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815bf90 con 0x7f1a280ad970 2023-10-16T14:32:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.720+0000 7f1a26ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a2806c040 0x7f1a2806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44364/0 (socket says 172.21.15.29:44364) 2023-10-16T14:32:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a26ffd640 1 -- 172.21.15.29:0/1464324662 learned_addr learned my addr 172.21.15.29:0/1464324662 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:42.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a267fc640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 0x7f1a280cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1464324662 >> v2:172.21.15.170:3300/0 conn(0x7f1a280ccfb0 0x7f1a280cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216213337 0 0) 0x7f1a2815b9d0 con 0x7f1a2806c040 2023-10-16T14:32:42.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a10003520 con 0x7f1a2806c040 2023-10-16T14:32:42.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1466411210 0 0) 0x7f1a10003520 con 0x7f1a2806c040 2023-10-16T14:32:42.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 msgr2=0x7f1a280cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:42.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 0x7f1a280cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 >> v2:172.21.15.170:3300/0 conn(0x7f1a280ccfb0 msgr2=0x7f1a280cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 --2- 172.21.15.29:0/1464324662 >> v2:172.21.15.170:3300/0 conn(0x7f1a280ccfb0 0x7f1a280cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280cc270 con 0x7f1a2806c040 2023-10-16T14:32:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.721+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1a180019f0 con 0x7f1a2806c040 2023-10-16T14:32:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 >> v1:172.21.15.29:6789/0 conn(0x7f1a2806c040 legacy=0x7f1a2806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10003970 0x7f1a10003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 0x7f1a100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815bcb0 con 0x7f1a10004d00 2023-10-16T14:32:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815bf90 con 0x7f1a10003970 2023-10-16T14:32:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1000b980 con 0x7f1a280ad970 2023-10-16T14:32:42.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a26ffd640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10003970 0x7f1a10003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a267fc640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 0x7f1a100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a277fe640 1 -- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 msgr2=0x7f1a100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:42.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a280ad970 0x7f1a100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a277fe640 1 -- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10003970 msgr2=0x7f1a10003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10003970 0x7f1a10003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.722+0000 7f1a277fe640 1 -- 172.21.15.29:0/1464324662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10003520 con 0x7f1a10004d00 2023-10-16T14:32:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.723+0000 7f1a26ffd640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10003970 0x7f1a10003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009af0 tx=0x7f1a100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944fc2aaafae2a1b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2002c710 con 0x7f1a10004d00 2023-10-16T14:32:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2002c870 con 0x7f1a10004d00 2023-10-16T14:32:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a25ffb640 1 -- 172.21.15.29:0/1464324662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2002cc00 con 0x7f1a10004d00 2023-10-16T14:32:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a27fff640 1 -- 172.21.15.29:0/1464324662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a10004d00 msgr2=0x7f1a10005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1464324662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a10004d00 0x7f1a10005180 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009af0 tx=0x7f1a100057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.724+0000 7f1a277fe640 1 -- 172.21.15.29:0/1464324662 reap_dead start 2023-10-16T14:32:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.725+0000 7f1a27fff640 1 -- 172.21.15.29:0/1464324662 shutdown_connections 2023-10-16T14:32:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.725+0000 7f1a27fff640 1 -- 172.21.15.29:0/1464324662 >> 172.21.15.29:0/1464324662 conn(0x7f1a280ce6a0 msgr2=0x7f1a280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.725+0000 7f1a27fff640 1 -- 172.21.15.29:0/1464324662 shutdown_connections 2023-10-16T14:32:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.725+0000 7f1a27fff640 1 -- 172.21.15.29:0/1464324662 wait complete. 2023-10-16T14:32:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.726+0000 7f1a27fff640 1 Processor -- start 2023-10-16T14:32:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.727+0000 7f1a27fff640 1 -- start start 2023-10-16T14:32:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.727+0000 7f1a27fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10004d00 0x7f1a28156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a27fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a281515e0 0x7f1a280c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 0x7f1a280c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a27fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28158200 con 0x7f1a28151a60 2023-10-16T14:32:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a27fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280c3bb0 con 0x7f1a10004d00 2023-10-16T14:32:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a27fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28157290 con 0x7f1a281515e0 2023-10-16T14:32:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 0x7f1a280c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 0x7f1a280c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43142/0 (socket says 172.21.15.29:43142) 2023-10-16T14:32:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 -- 172.21.15.29:0/1063028449 learned_addr learned my addr 172.21.15.29:0/1063028449 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a267fc640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a281515e0 0x7f1a280c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a26ffd640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10004d00 0x7f1a28156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 -- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a281515e0 msgr2=0x7f1a280c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a281515e0 0x7f1a280c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 -- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10004d00 msgr2=0x7f1a28156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10004d00 0x7f1a28156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:42.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a277fe640 1 -- 172.21.15.29:0/1063028449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280c2fa0 con 0x7f1a28151a60 2023-10-16T14:32:42.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.728+0000 7f1a26ffd640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10004d00 0x7f1a28156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:42.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.729+0000 7f1a277fe640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 0x7f1a280c7cc0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009af0 tx=0x7f1a20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:42.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.729+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2002c710 con 0x7f1a28151a60 2023-10-16T14:32:42.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.729+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20003830 con 0x7f1a28151a60 2023-10-16T14:32:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.729+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20003be0 con 0x7f1a28151a60 2023-10-16T14:32:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.729+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a280c3280 con 0x7f1a28151a60 2023-10-16T14:32:42.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.729+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a280c3840 con 0x7f1a28151a60 2023-10-16T14:32:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.730+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2806b330 con 0x7f1a28151a60 2023-10-16T14:32:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.731+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 32) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1a20003d40 con 0x7f1a28151a60 2023-10-16T14:32:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.732+0000 7f1a0f7fe640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a1406ad10 0x7f1a1406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.732+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f1a200aa3e0 con 0x7f1a28151a60 2023-10-16T14:32:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.733+0000 7f1a26ffd640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a1406ad10 0x7f1a1406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.734+0000 7f1a26ffd640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a1406ad10 0x7f1a1406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a280cda70 tx=0x7f1a180017b0 comp rx=0 tx=0).ready entity=mgr.5692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.737+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a20075670 con 0x7f1a28151a60 2023-10-16T14:32:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:42.977+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f1a280cc530 con 0x7f1a28151a60 2023-10-16T14:32:43.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.401+0000 7f1a0f7fe640 1 -- 172.21.15.29:0/1063028449 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v33) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1a280cc530 con 0x7f1a28151a60 2023-10-16T14:32:43.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.405+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a1406ad10 msgr2=0x7f1a1406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:43.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.405+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a1406ad10 0x7f1a1406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a280cda70 tx=0x7f1a180017b0 comp rx=0 tx=0).stop 2023-10-16T14:32:43.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.405+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 msgr2=0x7f1a280c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:43.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.405+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 0x7f1a280c7cc0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009af0 tx=0x7f1a20004730 comp rx=0 tx=0).stop 2023-10-16T14:32:43.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.405+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 shutdown_connections 2023-10-16T14:32:43.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.405+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f1a1406ad10 0x7f1a1406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:43.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.406+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a28151a60 0x7f1a280c7cc0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:43.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.406+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a281515e0 0x7f1a280c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:43.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.406+0000 7f1a27fff640 1 --2- 172.21.15.29:0/1063028449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a10004d00 0x7f1a28156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:43.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.406+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 >> 172.21.15.29:0/1063028449 conn(0x7f1a280ce6a0 msgr2=0x7f1a280c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:43.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.407+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 shutdown_connections 2023-10-16T14:32:43.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:43.407+0000 7f1a27fff640 1 -- 172.21.15.29:0/1063028449 wait complete. 2023-10-16T14:32:43.432 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:32:43.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:43.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:43.651 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:32:43.651 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:43.652 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:43.707 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:43.708 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:43.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:43.932 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:43.928+0000 7f1df6c4c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:32:43.937 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:43.935+0000 7fe2d1a42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:32:43.984 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:43.981+0000 7f1df6c4c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:32:43.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:43.991+0000 7fe2d1a42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:32:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.002+0000 7f74d3fff640 1 Processor -- start 2023-10-16T14:32:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.002+0000 7f74d3fff640 1 -- start start 2023-10-16T14:32:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.003+0000 7f74d3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f74d415cd70 0x7f74d40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74d40b8480 0x7f74d40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40d0be0 0x7f74d40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d415d150 con 0x7f74d40b8480 2023-10-16T14:32:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d3fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d40d1180 con 0x7f74d415cd70 2023-10-16T14:32:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40d1460 con 0x7f74d40d0be0 2023-10-16T14:32:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74d40b8480 0x7f74d40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74d40b8480 0x7f74d40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59264/0 (socket says 172.21.15.29:59264) 2023-10-16T14:32:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d27fc640 1 -- 172.21.15.29:0/1583553615 learned_addr learned my addr 172.21.15.29:0/1583553615 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d2ffd640 1 --2- 172.21.15.29:0/1583553615 >> v2:172.21.15.170:3300/0 conn(0x7f74d415cd70 0x7f74d40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d37fe640 1 --2- 172.21.15.29:0/1583553615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40d0be0 0x7f74d40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.004+0000 7f74d27fc640 1 -- 172.21.15.29:0/1583553615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40d0be0 msgr2=0x7f74d40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.005+0000 7f74d27fc640 1 --2- 172.21.15.29:0/1583553615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40d0be0 0x7f74d40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.005+0000 7f74d27fc640 1 -- 172.21.15.29:0/1583553615 >> v2:172.21.15.170:3300/0 conn(0x7f74d415cd70 msgr2=0x7f74d40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.005+0000 7f74d27fc640 1 --2- 172.21.15.29:0/1583553615 >> v2:172.21.15.170:3300/0 conn(0x7f74d415cd70 0x7f74d40d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.005+0000 7f74d27fc640 1 -- 172.21.15.29:0/1583553615 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d40d1740 con 0x7f74d40b8480 2023-10-16T14:32:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.006+0000 7f74d27fc640 1 --2- 172.21.15.29:0/1583553615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40b8480 0x7f74d40cd7c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f74d40b56a0 tx=0x7f74c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686860841791257f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.006+0000 7f74d1ffb640 1 -- 172.21.15.29:0/1583553615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8031870 con 0x7f74d40b8480 2023-10-16T14:32:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.006+0000 7f74d1ffb640 1 -- 172.21.15.29:0/1583553615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c80319d0 con 0x7f74d40b8480 2023-10-16T14:32:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.006+0000 7f74d1ffb640 1 -- 172.21.15.29:0/1583553615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8031d60 con 0x7f74d40b8480 2023-10-16T14:32:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 -- 172.21.15.29:0/1583553615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40b8480 msgr2=0x7f74d40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1583553615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40b8480 0x7f74d40cd7c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f74d40b56a0 tx=0x7f74c802b910 comp rx=0 tx=0).stop 2023-10-16T14:32:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 -- 172.21.15.29:0/1583553615 shutdown_connections 2023-10-16T14:32:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1583553615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40d0be0 0x7f74d40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1583553615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40b8480 0x7f74d40cd7c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1583553615 >> v2:172.21.15.170:3300/0 conn(0x7f74d415cd70 0x7f74d40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 -- 172.21.15.29:0/1583553615 >> 172.21.15.29:0/1583553615 conn(0x7f74d40cf570 msgr2=0x7f74d40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 -- 172.21.15.29:0/1583553615 shutdown_connections 2023-10-16T14:32:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.007+0000 7f74d3fff640 1 -- 172.21.15.29:0/1583553615 wait complete. 2023-10-16T14:32:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.009+0000 7f74d3fff640 1 Processor -- start 2023-10-16T14:32:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.009+0000 7f74d3fff640 1 -- start start 2023-10-16T14:32:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40b8480 0x7f74d415e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 0x7f74d40cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d415cd70 0x7f74d4150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4150850 con 0x7f74d40d0be0 2023-10-16T14:32:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d3fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d415ea80 con 0x7f74d415cd70 2023-10-16T14:32:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d415ed60 con 0x7f74d40b8480 2023-10-16T14:32:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 0x7f74d40cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 0x7f74d40cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59272/0 (socket says 172.21.15.29:59272) 2023-10-16T14:32:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d27fc640 1 -- 172.21.15.29:0/1973819614 learned_addr learned my addr 172.21.15.29:0/1973819614 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.010+0000 7f74d37fe640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d415cd70 0x7f74d4150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d2ffd640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40b8480 0x7f74d415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d27fc640 1 -- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40b8480 msgr2=0x7f74d415e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d27fc640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40b8480 0x7f74d415e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d27fc640 1 -- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d415cd70 msgr2=0x7f74d4150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d27fc640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d415cd70 0x7f74d4150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d27fc640 1 -- 172.21.15.29:0/1973819614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c80094d0 con 0x7f74d40d0be0 2023-10-16T14:32:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d2ffd640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40b8480 0x7f74d415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74d27fc640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 0x7f74d40cc4e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f74c802b950 tx=0x7f74c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.011+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c803ad30 con 0x7f74d40d0be0 2023-10-16T14:32:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.012+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c80443f0 con 0x7f74d40d0be0 2023-10-16T14:32:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.012+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c80447a0 con 0x7f74d40d0be0 2023-10-16T14:32:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.012+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d415f040 con 0x7f74d40d0be0 2023-10-16T14:32:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.012+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d415f600 con 0x7f74d40d0be0 2023-10-16T14:32:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.013+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d415eea0 con 0x7f74d40d0be0 2023-10-16T14:32:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.015+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f74c804d450 con 0x7f74d40d0be0 2023-10-16T14:32:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.015+0000 7f74bb7fe640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f74c006afc0 0x7f74c006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.016+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f74c802d040 con 0x7f74d40d0be0 2023-10-16T14:32:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.016+0000 7f74d2ffd640 1 -- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f74c006afc0 msgr2=0x7f74c006d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.016+0000 7f74d2ffd640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f74c006afc0 0x7f74c006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:44.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.020+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74d415eea0 con 0x7f74d40d0be0 2023-10-16T14:32:44.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.210+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74d415eea0 con 0x7f74d40d0be0 2023-10-16T14:32:44.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.213+0000 7f74bb7fe640 1 -- 172.21.15.29:0/1973819614 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+17072 (secure 0 0 0) 0x7f74d415eea0 con 0x7f74d40d0be0 2023-10-16T14:32:44.217 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:44.217 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:41.327723+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3718615290":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6833/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6832/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/3939932024":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1995904373":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1527444499":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1533610994":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/4190723338":"2023-10-17T14:32:41.327646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f74c006afc0 msgr2=0x7f74c006d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f74c006afc0 0x7f74c006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 msgr2=0x7f74d40cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 0x7f74d40cc4e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f74c802b950 tx=0x7f74c80046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 shutdown_connections 2023-10-16T14:32:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f74c006afc0 0x7f74c006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40d0be0 0x7f74d40cc4e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d415cd70 0x7f74d4150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 --2- 172.21.15.29:0/1973819614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40b8480 0x7f74d415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 >> 172.21.15.29:0/1973819614 conn(0x7f74d40cf570 msgr2=0x7f74d40b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.216+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 shutdown_connections 2023-10-16T14:32:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.217+0000 7f74d3fff640 1 -- 172.21.15.29:0/1973819614 wait complete. 2023-10-16T14:32:44.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:32:24.444824+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:32:25.549191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:44.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:44.433 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.429+0000 7f1df6c4c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:32:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.451+0000 7f8ea54e1640 1 Processor -- start 2023-10-16T14:32:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.452+0000 7f8ea54e1640 1 -- start start 2023-10-16T14:32:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8ea54e1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8ea006a4b0 0x7f8ea00ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8ea54e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea00c9ca0 0x7f8ea00ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8ea54e1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00cb030 con 0x7f8ea015ad70 2023-10-16T14:32:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8ea54e1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea00cae30 con 0x7f8ea006a4b0 2023-10-16T14:32:44.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8ea54e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00c91c0 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8e967fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8ea006a4b0 0x7f8ea00ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8e967fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8ea006a4b0 0x7f8ea00ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59288/0 (socket says 172.21.15.29:59288) 2023-10-16T14:32:44.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8e967fc640 1 -- 172.21.15.29:0/1955199900 learned_addr learned my addr 172.21.15.29:0/1955199900 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:44.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8e9f7fe640 1 --2- 172.21.15.29:0/1955199900 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea00c9ca0 0x7f8ea00ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8e967fc640 1 -- 172.21.15.29:0/1955199900 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea00c9ca0 msgr2=0x7f8ea00ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.454+0000 7f8e967fc640 1 --2- 172.21.15.29:0/1955199900 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea00c9ca0 0x7f8ea00ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.455+0000 7f8e967fc640 1 -- 172.21.15.29:0/1955199900 >> v1:172.21.15.170:6789/0 conn(0x7f8ea015ad70 legacy=0x7f8ea0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.455+0000 7f8e967fc640 1 -- 172.21.15.29:0/1955199900 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00c94a0 con 0x7f8ea006a4b0 2023-10-16T14:32:44.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.456+0000 7f8e967fc640 1 --2- 172.21.15.29:0/1955199900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea006a4b0 0x7f8ea00ca3b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009870 tx=0x7f8e8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b193296ba2d48b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.456+0000 7f8e9e7fc640 1 -- 172.21.15.29:0/1955199900 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88031640 con 0x7f8ea006a4b0 2023-10-16T14:32:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.456+0000 7f8e9e7fc640 1 -- 172.21.15.29:0/1955199900 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e880317a0 con 0x7f8ea006a4b0 2023-10-16T14:32:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.456+0000 7f8e9e7fc640 1 -- 172.21.15.29:0/1955199900 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88031b30 con 0x7f8ea006a4b0 2023-10-16T14:32:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 -- 172.21.15.29:0/1955199900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea006a4b0 msgr2=0x7f8ea00ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/1955199900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea006a4b0 0x7f8ea00ca3b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009870 tx=0x7f8e8802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 -- 172.21.15.29:0/1955199900 shutdown_connections 2023-10-16T14:32:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/1955199900 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea00c9ca0 0x7f8ea00ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/1955199900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea006a4b0 0x7f8ea00ca3b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 -- 172.21.15.29:0/1955199900 >> 172.21.15.29:0/1955199900 conn(0x7f8ea00ce9b0 msgr2=0x7f8ea00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 -- 172.21.15.29:0/1955199900 shutdown_connections 2023-10-16T14:32:44.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.457+0000 7f8ea54e1640 1 -- 172.21.15.29:0/1955199900 wait complete. 2023-10-16T14:32:44.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.459+0000 7f8ea54e1640 1 Processor -- start 2023-10-16T14:32:44.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.459+0000 7f8ea54e1640 1 -- start start 2023-10-16T14:32:44.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8ea54e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea006a4b0 0x7f8ea014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8ea54e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 0x7f8ea015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8ea54e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea015ad70 0x7f8ea014eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8ea54e1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea015d000 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8ea54e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea015c380 con 0x7f8ea006a4b0 2023-10-16T14:32:44.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8ea54e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea015c660 con 0x7f8ea015ad70 2023-10-16T14:32:44.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8e967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 0x7f8ea015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8e967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 0x7f8ea015baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59298/0 (socket says 172.21.15.29:59298) 2023-10-16T14:32:44.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8e967fc640 1 -- 172.21.15.29:0/131247586 learned_addr learned my addr 172.21.15.29:0/131247586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.460+0000 7f8e9effd640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea006a4b0 0x7f8ea014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e9f7fe640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea015ad70 0x7f8ea014eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e967fc640 1 -- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea015ad70 msgr2=0x7f8ea014eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e967fc640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea015ad70 0x7f8ea014eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e967fc640 1 -- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea006a4b0 msgr2=0x7f8ea014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e967fc640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea006a4b0 0x7f8ea014e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e967fc640 1 -- 172.21.15.29:0/131247586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e880094d0 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e967fc640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 0x7f8ea015baa0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009870 tx=0x7f8e880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8803ac60 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8803adc0 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88044630 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.461+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea015c940 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.462+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea014e5a0 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.463+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea015d140 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.464+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8e8802f050 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.465+0000 7f8e97fff640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.465+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8e880ab940 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.465+0000 7f8e9effd640 1 -- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f8e8006af90 msgr2=0x7f8e8006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.465+0000 7f8e9effd640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:44.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.469+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ea015d140 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.475+0000 7fe2d1a42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:32:44.485 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.482+0000 7f1df6c4c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:32:44.533 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.530+0000 7fe2d1a42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:32:44.535 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.532+0000 7f1df6c4c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:32:44.586 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.584+0000 7fe2d1a42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:32:44.596 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.592+0000 7f1df6c4c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:32:44.644 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.640+0000 7f1df6c4c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:32:44.650 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.648+0000 7fe2d1a42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:32:44.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.661+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ea015d140 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.661+0000 7f8e97fff640 1 -- 172.21.15.29:0/131247586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ea015d140 con 0x7f8ea00c9ca0 2023-10-16T14:32:44.664 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:44.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f8e8006af90 msgr2=0x7f8e8006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:44.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 msgr2=0x7f8ea015baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 0x7f8ea015baa0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009870 tx=0x7f8e880046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 shutdown_connections 2023-10-16T14:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00c9ca0 0x7f8ea015baa0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ea015ad70 0x7f8ea014eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 --2- 172.21.15.29:0/131247586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea006a4b0 0x7f8ea014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 >> 172.21.15.29:0/131247586 conn(0x7f8ea00ce9b0 msgr2=0x7f8ea00b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:44.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 shutdown_connections 2023-10-16T14:32:44.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.664+0000 7f8ea54e1640 1 -- 172.21.15.29:0/131247586 wait complete. 2023-10-16T14:32:44.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:32:44.705 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.702+0000 7fe2d1a42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:32:44.753 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.750+0000 7f1df6c4c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:32:44.817 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.811+0000 7f1df6c4c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:44.823 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.820+0000 7fe2d1a42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:32:44.865 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.861+0000 7f1df6c4c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:32:44.890 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.888+0000 7fe2d1a42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:44.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.902+0000 7ff13c256640 1 Processor -- start 2023-10-16T14:32:44.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.903+0000 7ff13c256640 1 -- start start 2023-10-16T14:32:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff13c256640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff134159ab0 0x7ff1340ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff13c256640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff13406c140 0x7ff13406c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff13c256640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13406cad0 con 0x7ff134159e90 2023-10-16T14:32:44.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff13c256640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13415d890 con 0x7ff13406c140 2023-10-16T14:32:44.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff13c256640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff134070360 con 0x7ff134159ab0 2023-10-16T14:32:44.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff1397f1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff134159e90 0x7ff13406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33544/0 (socket says 172.21.15.29:33544) 2023-10-16T14:32:44.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff1397f1640 1 -- 172.21.15.29:0/2260473581 learned_addr learned my addr 172.21.15.29:0/2260473581 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:44.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff13a7f3640 1 --2- 172.21.15.29:0/2260473581 >> v2:172.21.15.29:3300/0 conn(0x7ff13406c140 0x7ff13406c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.904+0000 7ff139ff2640 1 --2- 172.21.15.29:0/2260473581 >> v2:172.21.15.170:3300/0 conn(0x7ff134159ab0 0x7ff1340ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.905+0000 7ff138ff0640 1 -- 172.21.15.29:0/2260473581 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583996896 0 0) 0x7ff124001110 con 0x7ff134159e90 2023-10-16T14:32:44.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.905+0000 7ff138ff0640 1 -- 172.21.15.29:0/2260473581 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff120003520 con 0x7ff134159e90 2023-10-16T14:32:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.905+0000 7ff13a7f3640 1 -- 172.21.15.29:0/2260473581 >> v2:172.21.15.170:3300/0 conn(0x7ff134159ab0 msgr2=0x7ff1340ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.905+0000 7ff13a7f3640 1 --2- 172.21.15.29:0/2260473581 >> v2:172.21.15.170:3300/0 conn(0x7ff134159ab0 0x7ff1340ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.905+0000 7ff13a7f3640 1 -- 172.21.15.29:0/2260473581 >> v1:172.21.15.29:6789/0 conn(0x7ff134159e90 legacy=0x7ff13406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.905+0000 7ff13a7f3640 1 -- 172.21.15.29:0/2260473581 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1340cd1f0 con 0x7ff13406c140 2023-10-16T14:32:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.906+0000 7ff13a7f3640 1 --2- 172.21.15.29:0/2260473581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff13406c590 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff1340cfa00 tx=0x7ff12c02bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24797d1936229cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:44.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.906+0000 7ff138ff0640 1 -- 172.21.15.29:0/2260473581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c02cb50 con 0x7ff13406c140 2023-10-16T14:32:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.906+0000 7ff138ff0640 1 -- 172.21.15.29:0/2260473581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c02ccb0 con 0x7ff13406c140 2023-10-16T14:32:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.906+0000 7ff138ff0640 1 -- 172.21.15.29:0/2260473581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c034640 con 0x7ff13406c140 2023-10-16T14:32:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 -- 172.21.15.29:0/2260473581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 msgr2=0x7ff13406c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 --2- 172.21.15.29:0/2260473581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff13406c590 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff1340cfa00 tx=0x7ff12c02bda0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 -- 172.21.15.29:0/2260473581 shutdown_connections 2023-10-16T14:32:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 --2- 172.21.15.29:0/2260473581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff13406c590 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 --2- 172.21.15.29:0/2260473581 >> v2:172.21.15.170:3300/0 conn(0x7ff134159ab0 0x7ff1340ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 -- 172.21.15.29:0/2260473581 >> 172.21.15.29:0/2260473581 conn(0x7ff1340ce9b0 msgr2=0x7ff1340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 -- 172.21.15.29:0/2260473581 shutdown_connections 2023-10-16T14:32:44.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.907+0000 7ff13c256640 1 -- 172.21.15.29:0/2260473581 wait complete. 2023-10-16T14:32:44.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.909+0000 7ff13c256640 1 Processor -- start 2023-10-16T14:32:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.909+0000 7ff13c256640 1 -- start start 2023-10-16T14:32:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff13c256640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff1340b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff13c256640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff134159ab0 0x7ff134154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff13c256640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff134159e90 0x7ff1340ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff13c256640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1341e4fd0 con 0x7ff13406c140 2023-10-16T14:32:44.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff13c256640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1341e52b0 con 0x7ff134159ab0 2023-10-16T14:32:44.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff13c256640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1341e5590 con 0x7ff134159e90 2023-10-16T14:32:44.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff139ff2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff1340b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff139ff2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff1340b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59308/0 (socket says 172.21.15.29:59308) 2023-10-16T14:32:44.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff139ff2640 1 -- 172.21.15.29:0/4238391627 learned_addr learned my addr 172.21.15.29:0/4238391627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:44.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.910+0000 7ff1397f1640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff134159ab0 0x7ff134154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:44.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff139ff2640 1 -- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff134159e90 msgr2=0x7ff1340ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff139ff2640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff134159e90 0x7ff1340ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff139ff2640 1 -- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff134159ab0 msgr2=0x7ff134154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:44.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff139ff2640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff134159ab0 0x7ff134154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:44.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff139ff2640 1 -- 172.21.15.29:0/4238391627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c009a70 con 0x7ff13406c140 2023-10-16T14:32:44.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff1397f1640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff134159ab0 0x7ff134154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:44.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff139ff2640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff1340b3ed0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff13000e9b0 tx=0x7ff13000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:44.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff130009670 con 0x7ff13406c140 2023-10-16T14:32:44.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13000c9a0 con 0x7ff13406c140 2023-10-16T14:32:44.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1341e5870 con 0x7ff13406c140 2023-10-16T14:32:44.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13000cd50 con 0x7ff13406c140 2023-10-16T14:32:44.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.911+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1340b4410 con 0x7ff13406c140 2023-10-16T14:32:44.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.912+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13404e160 con 0x7ff13406c140 2023-10-16T14:32:44.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.913+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff130013470 con 0x7ff13406c140 2023-10-16T14:32:44.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.915+0000 7ff12a7fc640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff11806af90 0x7ff11806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:44.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.915+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff130089bb0 con 0x7ff13406c140 2023-10-16T14:32:44.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.915+0000 7ff1397f1640 1 -- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff11806af90 msgr2=0x7ff11806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:44.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.915+0000 7ff1397f1640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff11806af90 0x7ff11806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:44.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:44.919+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1300549e0 con 0x7ff13406c140 2023-10-16T14:32:44.923 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.920+0000 7f1df6c4c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:32:44.945 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:44.942+0000 7fe2d1a42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:32:44.974 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:44.970+0000 7f1df6c4c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:32:45.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.002+0000 7fe2d1a42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:32:45.058 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.055+0000 7fe2d1a42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:32:45.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.111+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1340cda60 con 0x7ff13406c140 2023-10-16T14:32:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.111+0000 7ff12a7fc640 1 -- 172.21.15.29:0/4238391627 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff130017020 con 0x7ff13406c140 2023-10-16T14:32:45.114 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:32:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff11806af90 msgr2=0x7ff11806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff11806af90 0x7ff11806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 msgr2=0x7ff1340b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff1340b3ed0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff13000e9b0 tx=0x7ff13000ee70 comp rx=0 tx=0).stop 2023-10-16T14:32:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 shutdown_connections 2023-10-16T14:32:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff11806af90 0x7ff11806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff134159e90 0x7ff1340ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff134159ab0 0x7ff134154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 --2- 172.21.15.29:0/4238391627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff13406c140 0x7ff1340b3ed0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 >> 172.21.15.29:0/4238391627 conn(0x7ff1340ce9b0 msgr2=0x7ff1340af040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:45.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 shutdown_connections 2023-10-16T14:32:45.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.114+0000 7ff13c256640 1 -- 172.21.15.29:0/4238391627 wait complete. 2023-10-16T14:32:45.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:32:45.360 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:45.356+0000 7f1df6c4c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:32:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.389+0000 7f4d3f5f5640 1 Processor -- start 2023-10-16T14:32:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.389+0000 7f4d3f5f5640 1 -- start start 2023-10-16T14:32:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3f5f5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4d38071bc0 0x7f4d380cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3f5f5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d380cabe0 con 0x7f4d380b4580 2023-10-16T14:32:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3f5f5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d380b4960 con 0x7f4d380b4d50 2023-10-16T14:32:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3f5f5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3806ff20 con 0x7f4d38071bc0 2023-10-16T14:32:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3d391640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4d380b4580 0x7f4d380714b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33560/0 (socket says 172.21.15.29:33560) 2023-10-16T14:32:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3d391640 1 -- 172.21.15.29:0/2832345675 learned_addr learned my addr 172.21.15.29:0/2832345675 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.391+0000 7f4d3db92640 1 --2- 172.21.15.29:0/2832345675 >> v2:172.21.15.170:3300/0 conn(0x7f4d38071bc0 0x7f4d380cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.392+0000 7f4d2ffff640 1 -- 172.21.15.29:0/2832345675 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445027720 0 0) 0x7f4d380cabe0 con 0x7f4d380b4580 2023-10-16T14:32:45.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.392+0000 7f4d2ffff640 1 -- 172.21.15.29:0/2832345675 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d24003520 con 0x7f4d380b4580 2023-10-16T14:32:45.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.392+0000 7f4d3db92640 1 -- 172.21.15.29:0/2832345675 >> v1:172.21.15.170:6789/0 conn(0x7f4d380b4d50 legacy=0x7f4d38069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.392+0000 7f4d3db92640 1 -- 172.21.15.29:0/2832345675 >> v1:172.21.15.29:6789/0 conn(0x7f4d380b4580 legacy=0x7f4d380714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.392+0000 7f4d2ffff640 1 -- 172.21.15.29:0/2832345675 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438582795 0 0) 0x7f4d24003520 con 0x7f4d380b4580 2023-10-16T14:32:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.392+0000 7f4d3db92640 1 -- 172.21.15.29:0/2832345675 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380ca2b0 con 0x7f4d38071bc0 2023-10-16T14:32:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.393+0000 7f4d3db92640 1 --2- 172.21.15.29:0/2832345675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d38071bc0 0x7f4d380cb1f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009d90 tx=0x7f4d3002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c658ec44d6a417e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:45.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.394+0000 7f4d2ffff640 1 -- 172.21.15.29:0/2832345675 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34001670 con 0x7f4d38071bc0 2023-10-16T14:32:45.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.394+0000 7f4d2ffff640 1 -- 172.21.15.29:0/2832345675 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3002c940 con 0x7f4d38071bc0 2023-10-16T14:32:45.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.394+0000 7f4d2ffff640 1 -- 172.21.15.29:0/2832345675 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3002ccb0 con 0x7f4d38071bc0 2023-10-16T14:32:45.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/2832345675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d38071bc0 msgr2=0x7f4d380cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/2832345675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d38071bc0 0x7f4d380cb1f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009d90 tx=0x7f4d3002be10 comp rx=0 tx=0).stop 2023-10-16T14:32:45.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/2832345675 shutdown_connections 2023-10-16T14:32:45.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/2832345675 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d38071bc0 0x7f4d380cb1f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/2832345675 >> 172.21.15.29:0/2832345675 conn(0x7f4d3806bc10 msgr2=0x7f4d3806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:45.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/2832345675 shutdown_connections 2023-10-16T14:32:45.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.395+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/2832345675 wait complete. 2023-10-16T14:32:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.397+0000 7f4d3f5f5640 1 Processor -- start 2023-10-16T14:32:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.397+0000 7f4d3f5f5640 1 -- start start 2023-10-16T14:32:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3f5f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 0x7f4d380ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3f5f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d380b4580 0x7f4d380b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3f5f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d380b4d50 0x7f4d380b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3f5f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381f65f0 con 0x7f4d38071bc0 2023-10-16T14:32:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3f5f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380ac890 con 0x7f4d380b4580 2023-10-16T14:32:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3f5f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381f5970 con 0x7f4d380b4d50 2023-10-16T14:32:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3d391640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 0x7f4d380ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3d391640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 0x7f4d380ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59312/0 (socket says 172.21.15.29:59312) 2023-10-16T14:32:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3d391640 1 -- 172.21.15.29:0/30978152 learned_addr learned my addr 172.21.15.29:0/30978152 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3cb90640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d380b4580 0x7f4d380b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.399+0000 7f4d3db92640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d380b4d50 0x7f4d380b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3d391640 1 -- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d380b4d50 msgr2=0x7f4d380b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3d391640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d380b4d50 0x7f4d380b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3d391640 1 -- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d380b4580 msgr2=0x7f4d380b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3d391640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d380b4580 0x7f4d380b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3d391640 1 -- 172.21.15.29:0/30978152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30009a70 con 0x7f4d38071bc0 2023-10-16T14:32:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3cb90640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d380b4580 0x7f4d380b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3d391640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 0x7f4d380ccd80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c760 tx=0x7f4d3400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34005ad0 con 0x7f4d38071bc0 2023-10-16T14:32:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d34005c30 con 0x7f4d38071bc0 2023-10-16T14:32:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34007070 con 0x7f4d38071bc0 2023-10-16T14:32:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.400+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d381f5c50 con 0x7f4d38071bc0 2023-10-16T14:32:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.401+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d381f6210 con 0x7f4d38071bc0 2023-10-16T14:32:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.402+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d380c2b50 con 0x7f4d38071bc0 2023-10-16T14:32:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.403+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f4d340071d0 con 0x7f4d38071bc0 2023-10-16T14:32:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.404+0000 7f4d2dffb640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 0x7f4d1c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.404+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f4d340896f0 con 0x7f4d38071bc0 2023-10-16T14:32:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.404+0000 7f4d3cb90640 1 -- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 msgr2=0x7f4d1c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.404+0000 7f4d3cb90640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 0x7f4d1c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:45.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.408+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4d34054490 con 0x7f4d38071bc0 2023-10-16T14:32:45.467 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.464+0000 7fe2d1a42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:32:45.521 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:45.518+0000 7f1df6c4c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:32:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.602+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4d381f6730 con 0x7f4d38071bc0 2023-10-16T14:32:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.603+0000 7f4d2dffb640 1 -- 172.21.15.29:0/30978152 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d381f6730 con 0x7f4d38071bc0 2023-10-16T14:32:45.606 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:32:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.605+0000 7f4d3cb90640 1 -- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 msgr2=0x7f4d1c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.605+0000 7f4d3cb90640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 0x7f4d1c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.605+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 msgr2=0x7f4d1c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.605+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 0x7f4d1c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 msgr2=0x7f4d380ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 0x7f4d380ccd80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c760 tx=0x7f4d3400cc20 comp rx=0 tx=0).stop 2023-10-16T14:32:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 shutdown_connections 2023-10-16T14:32:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f4d1c06af90 0x7f4d1c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d380b4d50 0x7f4d380b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d380b4580 0x7f4d380b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 --2- 172.21.15.29:0/30978152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d38071bc0 0x7f4d380ccd80 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 >> 172.21.15.29:0/30978152 conn(0x7f4d3806bc10 msgr2=0x7f4d380b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 shutdown_connections 2023-10-16T14:32:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.606+0000 7f4d3f5f5640 1 -- 172.21.15.29:0/30978152 wait complete. 2023-10-16T14:32:45.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:32:45.628 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:45.624+0000 7f1df6c4c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:32:45.639 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.637+0000 7fe2d1a42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:32:45.678 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:45.674+0000 7f1df6c4c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:32:45.754 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.751+0000 7fe2d1a42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:32:45.784 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:45.781+0000 7f1df6c4c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:45.808 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.805+0000 7fe2d1a42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:32:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.837+0000 7f2093d68640 1 Processor -- start 2023-10-16T14:32:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.838+0000 7f2093d68640 1 -- start start 2023-10-16T14:32:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.839+0000 7f2093d68640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.839+0000 7f2093d68640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f208c06c140 0x7f208c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.839+0000 7f2093d68640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c070360 con 0x7f208c159ab0 2023-10-16T14:32:45.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.839+0000 7f2093d68640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c06cb10 con 0x7f208c06c140 2023-10-16T14:32:45.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.839+0000 7f2093d68640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c15d890 con 0x7f208c159e90 2023-10-16T14:32:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.840+0000 7f2092305640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f208c06c140 0x7f208c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.840+0000 7f2092305640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f208c06c140 0x7f208c06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:39232/0 (socket says 172.21.15.29:39232) 2023-10-16T14:32:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.840+0000 7f2092305640 1 -- 172.21.15.29:0/2682085190 learned_addr learned my addr 172.21.15.29:0/2682085190 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.840+0000 7f2091303640 1 --2- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.841+0000 7f2092305640 1 -- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 msgr2=0x7f208c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.841+0000 7f2092305640 1 --2- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.841+0000 7f2092305640 1 -- 172.21.15.29:0/2682085190 >> v1:172.21.15.170:6789/0 conn(0x7f208c159ab0 legacy=0x7f208c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.841+0000 7f2092305640 1 -- 172.21.15.29:0/2682085190 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0cd1f0 con 0x7f208c06c140 2023-10-16T14:32:45.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.842+0000 7f2092305640 1 --2- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c06c140 0x7f208c06c5d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2084009e80 tx=0x7f208402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb3076348881a4b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.842+0000 7f2090b02640 1 -- 172.21.15.29:0/2682085190 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208402c9c0 con 0x7f208c06c140 2023-10-16T14:32:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.842+0000 7f2090b02640 1 -- 172.21.15.29:0/2682085190 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208402cb20 con 0x7f208c06c140 2023-10-16T14:32:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.842+0000 7f2090b02640 1 -- 172.21.15.29:0/2682085190 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20840343f0 con 0x7f208c06c140 2023-10-16T14:32:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 -- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c06c140 msgr2=0x7f208c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 --2- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c06c140 0x7f208c06c5d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2084009e80 tx=0x7f208402be20 comp rx=0 tx=0).stop 2023-10-16T14:32:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 -- 172.21.15.29:0/2682085190 shutdown_connections 2023-10-16T14:32:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 --2- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c06c140 0x7f208c06c5d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 --2- 172.21.15.29:0/2682085190 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 -- 172.21.15.29:0/2682085190 >> 172.21.15.29:0/2682085190 conn(0x7f208c0ce9b0 msgr2=0x7f208c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:45.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 -- 172.21.15.29:0/2682085190 shutdown_connections 2023-10-16T14:32:45.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.843+0000 7f2093d68640 1 -- 172.21.15.29:0/2682085190 wait complete. 2023-10-16T14:32:45.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.845+0000 7f2093d68640 1 Processor -- start 2023-10-16T14:32:45.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.845+0000 7f2093d68640 1 -- start start 2023-10-16T14:32:45.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2093d68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 0x7f208c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2093d68640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c159ab0 0x7f208c148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2093d68640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2093d68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c1495e0 con 0x7f208c06c140 2023-10-16T14:32:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2093d68640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c148640 con 0x7f208c159ab0 2023-10-16T14:32:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2093d68640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c148960 con 0x7f208c159e90 2023-10-16T14:32:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2091b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 0x7f208c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2091b04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 0x7f208c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59326/0 (socket says 172.21.15.29:59326) 2023-10-16T14:32:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2091b04640 1 -- 172.21.15.29:0/1836613176 learned_addr learned my addr 172.21.15.29:0/1836613176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2091303640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c159ab0 0x7f208c148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2092305640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2091b04640 1 -- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 msgr2=0x7f208c0c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.846+0000 7f2091b04640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2091b04640 1 -- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c159ab0 msgr2=0x7f208c148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2091b04640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c159ab0 0x7f208c148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2091b04640 1 -- 172.21.15.29:0/1836613176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084009b30 con 0x7f208c06c140 2023-10-16T14:32:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2091303640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c159ab0 0x7f208c148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2091b04640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 0x7f208c0c5a80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f208800c430 tx=0x7f208800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880052d0 con 0x7f208c06c140 2023-10-16T14:32:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2088005430 con 0x7f208c06c140 2023-10-16T14:32:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088003830 con 0x7f208c06c140 2023-10-16T14:32:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c148c40 con 0x7f208c06c140 2023-10-16T14:32:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.847+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c149200 con 0x7f208c06c140 2023-10-16T14:32:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.848+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c04e160 con 0x7f208c06c140 2023-10-16T14:32:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.850+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2088003990 con 0x7f208c06c140 2023-10-16T14:32:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.851+0000 7f20827fc640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 0x7f207006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.851+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2088088e60 con 0x7f208c06c140 2023-10-16T14:32:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.852+0000 7f2091303640 1 -- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 msgr2=0x7f207006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.852+0000 7f2091303640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 0x7f207006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:45.856+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2088053c90 con 0x7f208c06c140 2023-10-16T14:32:45.892 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:45.888+0000 7f1df6c4c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:32:45.926 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:45.923+0000 7fe2d1a42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:46.040 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:46.037+0000 7fe2d1a42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:32:46.050 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:46.044+0000 7f1df6c4c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.053+0000 7f2091303640 1 -- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 msgr2=0x7f207006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.053+0000 7f2091303640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 0x7f207006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.059+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f208c0ccb60 con 0x7f208c06c140 2023-10-16T14:32:46.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.060+0000 7f20827fc640 1 -- 172.21.15.29:0/1836613176 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v85) v1 ==== 80+0+476 (secure 0 0 0) 0x7f2088017080 con 0x7f208c06c140 2023-10-16T14:32:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3718615290 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/885795747 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/885795747 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3939932024 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1995904373 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1527444499 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1533610994 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.065 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/4190723338 2023-10-17T14:32:41.327646+0000 2023-10-16T14:32:46.065 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T14:32:46.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 msgr2=0x7f207006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:46.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 0x7f207006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 msgr2=0x7f208c0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 0x7f208c0c5a80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f208800c430 tx=0x7f208800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 shutdown_connections 2023-10-16T14:32:46.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f207006af90 0x7f207006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f208c159e90 0x7f208c0c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f208c159ab0 0x7f208c148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 --2- 172.21.15.29:0/1836613176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f208c06c140 0x7f208c0c5a80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 >> 172.21.15.29:0/1836613176 conn(0x7f208c0ce9b0 msgr2=0x7f208c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:46.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 shutdown_connections 2023-10-16T14:32:46.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.063+0000 7f2093d68640 1 -- 172.21.15.29:0/1836613176 wait complete. 2023-10-16T14:32:46.085 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:32:46.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:46.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:32:46.096 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:46.092+0000 7f1df6c4c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:46.209 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:46.206+0000 7fe2d1a42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:32:46.218 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:46.214+0000 7f1df6c4c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:32:46.224 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:32:46.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:32:46.225 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:32:46.265 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:46.262+0000 7fe2d1a42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:32:46.269 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:46.269 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:46.325 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:46.326 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:46.326 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:46.382 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:32:46.383 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:46.383 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:46.392 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:46.389+0000 7fe2d1a42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:32:46.439 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:46.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:46.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.654+0000 7f061f413640 1 Processor -- start 2023-10-16T14:32:46.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.655+0000 7f061f413640 1 -- start start 2023-10-16T14:32:46.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.656+0000 7f061f413640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:46.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.656+0000 7f061f413640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180ca580 con 0x7f061806fa20 2023-10-16T14:32:46.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.656+0000 7f061f413640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180ca2a0 con 0x7f0618159290 2023-10-16T14:32:46.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.656+0000 7f061f413640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06180ad5e0 con 0x7f06180ce280 2023-10-16T14:32:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061dc10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061ec12640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f061806fa20 0x7f06180ada00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33574/0 (socket says 172.21.15.29:33574) 2023-10-16T14:32:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061ec12640 1 -- 172.21.15.29:0/2189353849 learned_addr learned my addr 172.21.15.29:0/2189353849 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061dc10640 1 -- 172.21.15.29:0/2189353849 >> v1:172.21.15.170:6789/0 conn(0x7f0618159290 legacy=0x7f06180cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061dc10640 1 -- 172.21.15.29:0/2189353849 >> v1:172.21.15.29:6789/0 conn(0x7f061806fa20 legacy=0x7f06180ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061dc10640 1 -- 172.21.15.29:0/2189353849 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180acf20 con 0x7f06180ce280 2023-10-16T14:32:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.657+0000 7f061d40f640 1 -- 172.21.15.29:0/2189353849 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029102416 0 0) 0x7f06180ca580 con 0x7f061806fa20 2023-10-16T14:32:46.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.658+0000 7f061dc10640 1 --2- 172.21.15.29:0/2189353849 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180c95e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06080097d0 tx=0x7f060802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf5978c418b1272 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061d40f640 1 -- 172.21.15.29:0/2189353849 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0608031890 con 0x7f06180ce280 2023-10-16T14:32:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061d40f640 1 -- 172.21.15.29:0/2189353849 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06080319f0 con 0x7f06180ce280 2023-10-16T14:32:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061d40f640 1 -- 172.21.15.29:0/2189353849 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0608031d80 con 0x7f06180ce280 2023-10-16T14:32:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061f413640 1 -- 172.21.15.29:0/2189353849 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 msgr2=0x7f06180c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061f413640 1 --2- 172.21.15.29:0/2189353849 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180c95e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06080097d0 tx=0x7f060802b930 comp rx=0 tx=0).stop 2023-10-16T14:32:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061f413640 1 -- 172.21.15.29:0/2189353849 shutdown_connections 2023-10-16T14:32:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.659+0000 7f061f413640 1 --2- 172.21.15.29:0/2189353849 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180c95e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.660+0000 7f061f413640 1 -- 172.21.15.29:0/2189353849 >> 172.21.15.29:0/2189353849 conn(0x7f06180b0a80 msgr2=0x7f06180b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.660+0000 7f061f413640 1 -- 172.21.15.29:0/2189353849 shutdown_connections 2023-10-16T14:32:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.660+0000 7f061f413640 1 -- 172.21.15.29:0/2189353849 wait complete. 2023-10-16T14:32:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.661+0000 7f061f413640 1 Processor -- start 2023-10-16T14:32:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.662+0000 7f061f413640 1 -- start start 2023-10-16T14:32:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.662+0000 7f061f413640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f061806fa20 0x7f06180b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061f413640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061f413640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 0x7f06180b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061f413640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061809b0e0 con 0x7f0618159290 2023-10-16T14:32:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061f413640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06180b49e0 con 0x7f061806fa20 2023-10-16T14:32:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061f413640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061809a7b0 con 0x7f06180ce280 2023-10-16T14:32:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 0x7f06180b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 0x7f06180b1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59334/0 (socket says 172.21.15.29:59334) 2023-10-16T14:32:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 -- 172.21.15.29:0/1920079334 learned_addr learned my addr 172.21.15.29:0/1920079334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061dc10640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061e411640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f061806fa20 0x7f06180b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 -- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 msgr2=0x7f06180b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 -- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f061806fa20 msgr2=0x7f06180b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.663+0000 7f061ec12640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f061806fa20 0x7f06180b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f061ec12640 1 -- 172.21.15.29:0/1920079334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608009400 con 0x7f0618159290 2023-10-16T14:32:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f061dc10640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:32:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f061ec12640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 0x7f06180b1bb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f061000c7a0 tx=0x7f061000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000a640 con 0x7f0618159290 2023-10-16T14:32:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061000a7a0 con 0x7f0618159290 2023-10-16T14:32:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000abd0 con 0x7f0618159290 2023-10-16T14:32:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061809aa90 con 0x7f0618159290 2023-10-16T14:32:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.664+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06180b20f0 con 0x7f0618159290 2023-10-16T14:32:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.665+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061809b290 con 0x7f0618159290 2023-10-16T14:32:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.667+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f061000ad30 con 0x7f0618159290 2023-10-16T14:32:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.667+0000 7f060effd640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 0x7f05fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.668+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0610088e90 con 0x7f0618159290 2023-10-16T14:32:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.668+0000 7f061e411640 1 -- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 msgr2=0x7f05fc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.668+0000 7f061e411640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 0x7f05fc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.672+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f061809b290 con 0x7f0618159290 2023-10-16T14:32:46.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.869+0000 7f061e411640 1 -- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 msgr2=0x7f05fc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:46.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.869+0000 7f061e411640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 0x7f05fc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:46.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.901+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f061809b290 con 0x7f0618159290 2023-10-16T14:32:46.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.902+0000 7f060effd640 1 -- 172.21.15.29:0/1920079334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f061809b290 con 0x7f0618159290 2023-10-16T14:32:46.905 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:46.905 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:32:26.521447+0000","modified":"2023-10-16T14:32:29.697489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3822716445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3822716445},{"type":"v1","addr":"172.21.15.170:6837","nonce":3822716445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T14:32:46.906 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2023-10-16T14:32:46.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 msgr2=0x7f05fc06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:46.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 0x7f05fc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 msgr2=0x7f06180b1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:46.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 0x7f06180b1bb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f061000c7a0 tx=0x7f061000cc60 comp rx=0 tx=0).stop 2023-10-16T14:32:46.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 shutdown_connections 2023-10-16T14:32:46.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7f05fc06af90 0x7f05fc06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0618159290 0x7f06180b1bb0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06180ce280 0x7f06180b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 --2- 172.21.15.29:0/1920079334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f061806fa20 0x7f06180b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.904+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 >> 172.21.15.29:0/1920079334 conn(0x7f06180b0a80 msgr2=0x7f06180ad900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.905+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 shutdown_connections 2023-10-16T14:32:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:46.905+0000 7f061f413640 1 -- 172.21.15.29:0/1920079334 wait complete. 2023-10-16T14:32:46.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:47.052 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.048+0000 7f1df6c4c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:32:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.117+0000 7ff3b1128640 1 Processor -- start 2023-10-16T14:32:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.117+0000 7ff3b1128640 1 -- start start 2023-10-16T14:32:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.118+0000 7ff3b1128640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff3ac15cbe0 0x7ff3ac0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.118+0000 7ff3b1128640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac15cfc0 0x7ff3ac0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.118+0000 7ff3b1128640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac0d15a0 con 0x7ff3ac0d1090 2023-10-16T14:32:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.118+0000 7ff3b1128640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ac0d1880 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.118+0000 7ff3b1128640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0d1b60 con 0x7ff3ac15cfc0 2023-10-16T14:32:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3abfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff3ac15cbe0 0x7ff3ac0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3b0927640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3ac0d1090 0x7ff3ac06bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33586/0 (socket says 172.21.15.29:33586) 2023-10-16T14:32:47.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3b0927640 1 -- 172.21.15.29:0/1096486996 learned_addr learned my addr 172.21.15.29:0/1096486996 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3ab7fe640 1 --2- 172.21.15.29:0/1096486996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac15cfc0 0x7ff3ac0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3abfff640 1 -- 172.21.15.29:0/1096486996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac15cfc0 msgr2=0x7ff3ac0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3abfff640 1 --2- 172.21.15.29:0/1096486996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac15cfc0 0x7ff3ac0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3abfff640 1 -- 172.21.15.29:0/1096486996 >> v1:172.21.15.29:6789/0 conn(0x7ff3ac0d1090 legacy=0x7ff3ac06bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3abfff640 1 -- 172.21.15.29:0/1096486996 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac0d1e40 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.119+0000 7ff3aaffd640 1 -- 172.21.15.29:0/1096486996 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022681469 0 0) 0x7ff3ac0d15a0 con 0x7ff3ac0d1090 2023-10-16T14:32:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.120+0000 7ff3abfff640 1 --2- 172.21.15.29:0/1096486996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac0d0b50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0097a0 tx=0x7ff39c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae807e43b1c3ca32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3aaffd640 1 -- 172.21.15.29:0/1096486996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c030800 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3aaffd640 1 -- 172.21.15.29:0/1096486996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c030960 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3aaffd640 1 -- 172.21.15.29:0/1096486996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c030cf0 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3b1128640 1 -- 172.21.15.29:0/1096486996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 msgr2=0x7ff3ac0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3b1128640 1 --2- 172.21.15.29:0/1096486996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac0d0b50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0097a0 tx=0x7ff39c02b940 comp rx=0 tx=0).stop 2023-10-16T14:32:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3b1128640 1 -- 172.21.15.29:0/1096486996 shutdown_connections 2023-10-16T14:32:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3b1128640 1 --2- 172.21.15.29:0/1096486996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac15cfc0 0x7ff3ac0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.121+0000 7ff3b1128640 1 --2- 172.21.15.29:0/1096486996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac0d0b50 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.122+0000 7ff3b1128640 1 -- 172.21.15.29:0/1096486996 >> 172.21.15.29:0/1096486996 conn(0x7ff3ac0cf570 msgr2=0x7ff3ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.122+0000 7ff3b1128640 1 -- 172.21.15.29:0/1096486996 shutdown_connections 2023-10-16T14:32:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.122+0000 7ff3b1128640 1 -- 172.21.15.29:0/1096486996 wait complete. 2023-10-16T14:32:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.123+0000 7ff3b1128640 1 Processor -- start 2023-10-16T14:32:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.123+0000 7ff3b1128640 1 -- start start 2023-10-16T14:32:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.124+0000 7ff3b1128640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac0d1090 0x7ff3ac15f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.124+0000 7ff3b1128640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac1f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.124+0000 7ff3b1128640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3ac15cfc0 0x7ff3ac0c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.124+0000 7ff3b1128640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac1f67c0 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.124+0000 7ff3b1128640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac15e540 con 0x7ff3ac15cfc0 2023-10-16T14:32:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.124+0000 7ff3b1128640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac15e820 con 0x7ff3ac0d1090 2023-10-16T14:32:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac1f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac1f5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59352/0 (socket says 172.21.15.29:59352) 2023-10-16T14:32:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 -- 172.21.15.29:0/3400724117 learned_addr learned my addr 172.21.15.29:0/3400724117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3b0927640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3ac15cfc0 0x7ff3ac0c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3abfff640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac0d1090 0x7ff3ac15f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 -- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac0d1090 msgr2=0x7ff3ac15f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac0d1090 0x7ff3ac15f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 -- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3ac15cfc0 msgr2=0x7ff3ac0c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3ac15cfc0 0x7ff3ac0c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 -- 172.21.15.29:0/3400724117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c009400 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3b0927640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3ac15cfc0 0x7ff3ac0c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.125+0000 7ff3ab7fe640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac1f5b20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000d5f0 tx=0x7ff3a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.126+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00125f0 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.126+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0012750 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.126+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0012b00 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.126+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac15eb00 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.126+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac15f0c0 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.127+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac15f280 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.128+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff3a0012c60 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.129+0000 7ff3a8ff9640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 0x7ff39806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.129+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff3a0089cf0 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.129+0000 7ff3abfff640 1 -- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 msgr2=0x7ff39806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.129+0000 7ff3abfff640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 0x7ff39806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:32:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.133+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff3a008e070 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.247 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:47.244+0000 7fe2d1a42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:32:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.330+0000 7ff3abfff640 1 -- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 msgr2=0x7ff39806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3148236210 2023-10-16T14:32:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.330+0000 7ff3abfff640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 0x7ff39806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:32:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.330+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3ac0cff80 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.332+0000 7ff3a8ff9640 1 -- 172.21.15.29:0/3400724117 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+17072 (secure 0 0 0) 0x7ff3a0054bc0 con 0x7ff3ac15cbe0 2023-10-16T14:32:47.336 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:47.336 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:41.327723+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3718615290":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6833/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6832/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/3939932024":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1995904373":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1527444499":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1533610994":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/4190723338":"2023-10-17T14:32:41.327646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:47.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 msgr2=0x7ff39806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:32:47.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 0x7ff39806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 msgr2=0x7ff3ac1f5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac1f5b20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000d5f0 tx=0x7ff3a000dab0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 shutdown_connections 2023-10-16T14:32:47.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:6832/3148236210,v1:172.21.15.170:6833/3148236210] conn(0x7ff39806af90 0x7ff39806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3ac15cbe0 0x7ff3ac1f5b20 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3ac15cfc0 0x7ff3ac0c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 --2- 172.21.15.29:0/3400724117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3ac0d1090 0x7ff3ac15f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 >> 172.21.15.29:0/3400724117 conn(0x7ff3ac0cf570 msgr2=0x7ff3ac0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 shutdown_connections 2023-10-16T14:32:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.335+0000 7ff3b1128640 1 -- 172.21.15.29:0/3400724117 wait complete. 2023-10-16T14:32:47.351 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.347+0000 7f1df6c4c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:32:47.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:32:24.444824+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:32:25.549191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:47.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:47.408 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.404+0000 7f1df6c4c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:32:47.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.519+0000 7f1df6c4c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:32:47.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.570+0000 7efd7ebb6640 1 Processor -- start 2023-10-16T14:32:47.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.570+0000 7efd7ebb6640 1 -- start start 2023-10-16T14:32:47.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.571+0000 7efd7ebb6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd78159ab0 0x7efd780ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.571+0000 7efd7ebb6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd7806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.571+0000 7efd7ebb6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd78070360 con 0x7efd78159e90 2023-10-16T14:32:47.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.571+0000 7efd7ebb6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7806cb10 con 0x7efd78159ab0 2023-10-16T14:32:47.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.571+0000 7efd7ebb6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7815d890 con 0x7efd7806c140 2023-10-16T14:32:47.575 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:47.571+0000 7fe2d1a42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:32:47.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd6ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7efd78159e90 0x7efd7806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:40386/0 (socket says 172.21.15.29:40386) 2023-10-16T14:32:47.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd6ffff640 1 -- 172.21.15.29:0/1411894157 learned_addr learned my addr 172.21.15.29:0/1411894157 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:47.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd7c952640 1 --2- 172.21.15.29:0/1411894157 >> v2:172.21.15.29:3300/0 conn(0x7efd78159ab0 0x7efd780ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd7d153640 1 --2- 172.21.15.29:0/1411894157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd7806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd6f7fe640 1 -- 172.21.15.29:0/1411894157 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166067760 0 0) 0x7efd78070360 con 0x7efd78159e90 2023-10-16T14:32:47.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd7c952640 1 -- 172.21.15.29:0/1411894157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 msgr2=0x7efd7806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:47.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd7c952640 1 --2- 172.21.15.29:0/1411894157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd7806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd7c952640 1 -- 172.21.15.29:0/1411894157 >> v1:172.21.15.170:6789/0 conn(0x7efd78159e90 legacy=0x7efd7806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.572+0000 7efd7c952640 1 -- 172.21.15.29:0/1411894157 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780cd1f0 con 0x7efd78159ab0 2023-10-16T14:32:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.574+0000 7efd7c952640 1 --2- 172.21.15.29:0/1411894157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159ab0 0x7efd780ce370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efd74009d90 tx=0x7efd7402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e297028745f58a95 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.574+0000 7efd6f7fe640 1 -- 172.21.15.29:0/1411894157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7402cb40 con 0x7efd78159ab0 2023-10-16T14:32:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.574+0000 7efd6f7fe640 1 -- 172.21.15.29:0/1411894157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7402cca0 con 0x7efd78159ab0 2023-10-16T14:32:47.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.574+0000 7efd6f7fe640 1 -- 172.21.15.29:0/1411894157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74034660 con 0x7efd78159ab0 2023-10-16T14:32:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.574+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1411894157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159ab0 msgr2=0x7efd780ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.574+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1411894157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159ab0 0x7efd780ce370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efd74009d90 tx=0x7efd7402bd90 comp rx=0 tx=0).stop 2023-10-16T14:32:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.575+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1411894157 shutdown_connections 2023-10-16T14:32:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.575+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1411894157 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd7806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.575+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1411894157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159ab0 0x7efd780ce370 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.575+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1411894157 >> 172.21.15.29:0/1411894157 conn(0x7efd780ce9b0 msgr2=0x7efd780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:47.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.575+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1411894157 shutdown_connections 2023-10-16T14:32:47.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.575+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1411894157 wait complete. 2023-10-16T14:32:47.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.576+0000 7efd7ebb6640 1 Processor -- start 2023-10-16T14:32:47.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.576+0000 7efd7ebb6640 1 -- start start 2023-10-16T14:32:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.577+0000 7efd7ebb6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd78157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.577+0000 7efd7ebb6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd78159ab0 0x7efd781f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.577+0000 7efd7ebb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 0x7efd781f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.577+0000 7efd7ebb6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7815cf00 con 0x7efd78159e90 2023-10-16T14:32:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.577+0000 7efd7ebb6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7806b840 con 0x7efd78159ab0 2023-10-16T14:32:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.577+0000 7efd7ebb6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd781f2f10 con 0x7efd7806c140 2023-10-16T14:32:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 0x7efd781f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7c952640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd78157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 0x7efd781f3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59382/0 (socket says 172.21.15.29:59382) 2023-10-16T14:32:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd6ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd78159ab0 0x7efd781f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7c952640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd78157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:45394/0 (socket says 172.21.15.29:45394) 2023-10-16T14:32:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 -- 172.21.15.29:0/1823678789 learned_addr learned my addr 172.21.15.29:0/1823678789 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 -- 172.21.15.29:0/1823678789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 msgr2=0x7efd78157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd78157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 -- 172.21.15.29:0/1823678789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd78159ab0 msgr2=0x7efd781f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd78159ab0 0x7efd781f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 -- 172.21.15.29:0/1823678789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74009a70 con 0x7efd78159e90 2023-10-16T14:32:47.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.578+0000 7efd7d153640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 0x7efd781f3110 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efd7000e820 tx=0x7efd7000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.579+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70009510 con 0x7efd78159e90 2023-10-16T14:32:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.579+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7000ee70 con 0x7efd78159e90 2023-10-16T14:32:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.579+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7000cb80 con 0x7efd78159e90 2023-10-16T14:32:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.579+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd781f3650 con 0x7efd78159e90 2023-10-16T14:32:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.579+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd780c2940 con 0x7efd78159e90 2023-10-16T14:32:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.580+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7804e160 con 0x7efd78159e90 2023-10-16T14:32:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.581+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 88940+0+0 (secure 0 0 0) 0x7efd7000cce0 con 0x7efd78159e90 2023-10-16T14:32:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.582+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efd70089430 con 0x7efd78159e90 2023-10-16T14:32:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.590+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efd70054640 con 0x7efd78159e90 2023-10-16T14:32:47.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:32:47.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:32:47.616 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.612+0000 7f1d7bf28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.616 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.612+0000 7f1d7bf28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.612+0000 7f1d7bf28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.612+0000 7f1d7bf28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.618 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.612+0000 7f1d7bf28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.619 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.613+0000 7f1d78f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.619 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.613+0000 7f1d78f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.619 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.613+0000 7f1d78f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.619 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.613+0000 7f1d78f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.619 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:32:47.613+0000 7f1d78f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:32:47.641 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:47.639+0000 7fe2d1a42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:32:47.763 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:32:47.760+0000 7fe2d1a42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:32:47.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.783+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd780ccb60 con 0x7efd78159e90 2023-10-16T14:32:47.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.784+0000 7efd6d7fa640 1 -- 172.21.15.29:0/1823678789 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd70057c60 con 0x7efd78159e90 2023-10-16T14:32:47.787 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:47.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 msgr2=0x7efd781f3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:47.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 0x7efd781f3110 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efd7000e820 tx=0x7efd7000ec20 comp rx=0 tx=0).stop 2023-10-16T14:32:47.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 shutdown_connections 2023-10-16T14:32:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd78159e90 0x7efd781f3110 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd78159ab0 0x7efd781f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 --2- 172.21.15.29:0/1823678789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd7806c140 0x7efd78157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 >> 172.21.15.29:0/1823678789 conn(0x7efd780ce9b0 msgr2=0x7efd780b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 shutdown_connections 2023-10-16T14:32:47.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:47.787+0000 7efd7ebb6640 1 -- 172.21.15.29:0/1823678789 wait complete. 2023-10-16T14:32:47.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:32:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.028+0000 7f4c6e852640 1 Processor -- start 2023-10-16T14:32:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.028+0000 7f4c6e852640 1 -- start start 2023-10-16T14:32:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6e852640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c680ad970 0x7f4c680cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6e852640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c680ccfb0 0x7f4c680cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6e852640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6815b9d0 con 0x7f4c6806c040 2023-10-16T14:32:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6e852640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6815bcb0 con 0x7f4c680ad970 2023-10-16T14:32:48.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6e852640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6815bf90 con 0x7f4c680ccfb0 2023-10-16T14:32:48.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6d04f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c680ad970 0x7f4c680cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6d04f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c680ad970 0x7f4c680cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59384/0 (socket says 172.21.15.29:59384) 2023-10-16T14:32:48.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.030+0000 7f4c6d04f640 1 -- 172.21.15.29:0/3729269296 learned_addr learned my addr 172.21.15.29:0/3729269296 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.031+0000 7f4c6e051640 1 --2- 172.21.15.29:0/3729269296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c680ccfb0 0x7f4c680cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.031+0000 7f4c6d04f640 1 -- 172.21.15.29:0/3729269296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c680ccfb0 msgr2=0x7f4c680cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.031+0000 7f4c6d04f640 1 --2- 172.21.15.29:0/3729269296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c680ccfb0 0x7f4c680cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.031+0000 7f4c6d04f640 1 -- 172.21.15.29:0/3729269296 >> v1:172.21.15.29:6789/0 conn(0x7f4c6806c040 legacy=0x7f4c6806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.031+0000 7f4c6d04f640 1 -- 172.21.15.29:0/3729269296 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680cc270 con 0x7f4c680ad970 2023-10-16T14:32:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.032+0000 7f4c6d04f640 1 --2- 172.21.15.29:0/3729269296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ad970 0x7f4c680cca70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009870 tx=0x7f4c5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c699902dfd2e9380 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.032+0000 7f4c6c84e640 1 -- 172.21.15.29:0/3729269296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58030840 con 0x7f4c680ad970 2023-10-16T14:32:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.032+0000 7f4c6c84e640 1 -- 172.21.15.29:0/3729269296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c580309a0 con 0x7f4c680ad970 2023-10-16T14:32:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.032+0000 7f4c6c84e640 1 -- 172.21.15.29:0/3729269296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58030d30 con 0x7f4c680ad970 2023-10-16T14:32:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 -- 172.21.15.29:0/3729269296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ad970 msgr2=0x7f4c680cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 --2- 172.21.15.29:0/3729269296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ad970 0x7f4c680cca70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009870 tx=0x7f4c5802b980 comp rx=0 tx=0).stop 2023-10-16T14:32:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 -- 172.21.15.29:0/3729269296 shutdown_connections 2023-10-16T14:32:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 --2- 172.21.15.29:0/3729269296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c680ccfb0 0x7f4c680cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 --2- 172.21.15.29:0/3729269296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ad970 0x7f4c680cca70 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 -- 172.21.15.29:0/3729269296 >> 172.21.15.29:0/3729269296 conn(0x7f4c680ce6a0 msgr2=0x7f4c680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 -- 172.21.15.29:0/3729269296 shutdown_connections 2023-10-16T14:32:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.033+0000 7f4c6e852640 1 -- 172.21.15.29:0/3729269296 wait complete. 2023-10-16T14:32:48.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.035+0000 7f4c6e852640 1 Processor -- start 2023-10-16T14:32:48.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.035+0000 7f4c6e852640 1 -- start start 2023-10-16T14:32:48.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e852640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c6806c040 0x7f4c680c1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e852640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c680ad970 0x7f4c68145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e852640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 0x7f4c68151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e852640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68150b80 con 0x7f4c680ccfb0 2023-10-16T14:32:48.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e852640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680c2000 con 0x7f4c680ad970 2023-10-16T14:32:48.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e852640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6814fae0 con 0x7f4c6806c040 2023-10-16T14:32:48.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e051640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 0x7f4c68151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e051640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 0x7f4c68151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59396/0 (socket says 172.21.15.29:59396) 2023-10-16T14:32:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.036+0000 7f4c6e051640 1 -- 172.21.15.29:0/2716301008 learned_addr learned my addr 172.21.15.29:0/2716301008 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e051640 1 -- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c6806c040 msgr2=0x7f4c680c1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6d850640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c6806c040 0x7f4c680c1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6d04f640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c680ad970 0x7f4c68145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e051640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c6806c040 0x7f4c680c1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e051640 1 -- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c680ad970 msgr2=0x7f4c68145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e051640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c680ad970 0x7f4c68145880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e051640 1 -- 172.21.15.29:0/2716301008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c580094d0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6d850640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c6806c040 0x7f4c680c1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6d04f640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c680ad970 0x7f4c68145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e051640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 0x7f4c68151890 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4c60007e60 tx=0x7f4c6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60014420 con 0x7f4c680ccfb0 2023-10-16T14:32:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c600145a0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60014950 con 0x7f4c680ccfb0 2023-10-16T14:32:48.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.037+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6814fdc0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.038+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c68150380 con 0x7f4c680ccfb0 2023-10-16T14:32:48.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.039+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6806b330 con 0x7f4c680ccfb0 2023-10-16T14:32:48.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.040+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4c60014ab0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.041+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4c60089ca0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.045+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4c6008e070 con 0x7f4c680ccfb0 2023-10-16T14:32:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.228+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c6804e9b0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.229+0000 7f4c5e7fc640 1 -- 172.21.15.29:0/2716301008 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4c60054ef0 con 0x7f4c680ccfb0 2023-10-16T14:32:48.232 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:32:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 msgr2=0x7f4c68151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 0x7f4c68151890 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4c60007e60 tx=0x7f4c6000e4b0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 shutdown_connections 2023-10-16T14:32:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680ccfb0 0x7f4c68151890 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c680ad970 0x7f4c68145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 --2- 172.21.15.29:0/2716301008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c6806c040 0x7f4c680c1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 >> 172.21.15.29:0/2716301008 conn(0x7f4c680ce6a0 msgr2=0x7f4c6815d580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 shutdown_connections 2023-10-16T14:32:48.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.231+0000 7f4c6e852640 1 -- 172.21.15.29:0/2716301008 wait complete. 2023-10-16T14:32:48.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:32:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.472+0000 7febaecad640 1 Processor -- start 2023-10-16T14:32:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.473+0000 7febaecad640 1 -- start start 2023-10-16T14:32:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febaecad640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7feba815cd70 0x7feba80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febaecad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 0x7feba80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febaecad640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba815d150 con 0x7feba80d0be0 2023-10-16T14:32:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febaecad640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba80d1180 con 0x7feba815cd70 2023-10-16T14:32:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febaecad640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba80d1460 con 0x7feba80b8480 2023-10-16T14:32:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febae4ac640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feba80d0be0 0x7feba80b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33602/0 (socket says 172.21.15.29:33602) 2023-10-16T14:32:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.474+0000 7febae4ac640 1 -- 172.21.15.29:0/2692783727 learned_addr learned my addr 172.21.15.29:0/2692783727 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:48.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febadcab640 1 --2- 172.21.15.29:0/2692783727 >> v2:172.21.15.170:3300/0 conn(0x7feba815cd70 0x7feba80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febad4aa640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 0x7feba80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357181698 0 0) 0x7feba815d150 con 0x7feba80d0be0 2023-10-16T14:32:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94003520 con 0x7feba80d0be0 2023-10-16T14:32:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012811648 0 0) 0x7feb94003520 con 0x7feba80d0be0 2023-10-16T14:32:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 msgr2=0x7feba80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:48.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 0x7feba80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 >> v2:172.21.15.170:3300/0 conn(0x7feba815cd70 msgr2=0x7feba80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 --2- 172.21.15.29:0/2692783727 >> v2:172.21.15.170:3300/0 conn(0x7feba815cd70 0x7feba80d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80d1740 con 0x7feba80d0be0 2023-10-16T14:32:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.475+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7feba00019f0 con 0x7feba80d0be0 2023-10-16T14:32:48.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 >> v1:172.21.15.29:6789/0 conn(0x7feba80d0be0 legacy=0x7feba80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb94003970 0x7feb94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 0x7feb940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb94004d00 0x7feb94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80d1180 con 0x7feb94003970 2023-10-16T14:32:48.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80d1460 con 0x7feb94004d00 2023-10-16T14:32:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9400b980 con 0x7feba80b8480 2023-10-16T14:32:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febadcab640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb94003970 0x7feb94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febad4aa640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 0x7feb940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febae4ac640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb94004d00 0x7feb94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febadcab640 1 -- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 msgr2=0x7feb940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febadcab640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba80b8480 0x7feb940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febadcab640 1 -- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb94004d00 msgr2=0x7feb94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febadcab640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb94004d00 0x7feb94005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.476+0000 7febadcab640 1 -- 172.21.15.29:0/2692783727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94003520 con 0x7feb94003970 2023-10-16T14:32:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.477+0000 7febadcab640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb94003970 0x7feb94003d90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7feba4009ac0 tx=0x7feb940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9379c5877149d27c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba402c710 con 0x7feb94003970 2023-10-16T14:32:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba402c870 con 0x7feb94003970 2023-10-16T14:32:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febacca9640 1 -- 172.21.15.29:0/2692783727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba402cc00 con 0x7feb94003970 2023-10-16T14:32:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febaecad640 1 -- 172.21.15.29:0/2692783727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb94003970 msgr2=0x7feb94003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febaecad640 1 --2- 172.21.15.29:0/2692783727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb94003970 0x7feb94003d90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7feba4009ac0 tx=0x7feb940057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febae4ac640 1 -- 172.21.15.29:0/2692783727 reap_dead start 2023-10-16T14:32:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febaecad640 1 -- 172.21.15.29:0/2692783727 shutdown_connections 2023-10-16T14:32:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febaecad640 1 -- 172.21.15.29:0/2692783727 >> 172.21.15.29:0/2692783727 conn(0x7feba80cf570 msgr2=0x7feba80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febaecad640 1 -- 172.21.15.29:0/2692783727 shutdown_connections 2023-10-16T14:32:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.478+0000 7febaecad640 1 -- 172.21.15.29:0/2692783727 wait complete. 2023-10-16T14:32:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.480+0000 7febaecad640 1 Processor -- start 2023-10-16T14:32:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.480+0000 7febaecad640 1 -- start start 2023-10-16T14:32:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.481+0000 7febaecad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb94003970 0x7feba81f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.481+0000 7febaecad640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba80cc6d0 0x7feba815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.481+0000 7febaecad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 0x7feba81e6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febaecad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba81f61c0 con 0x7feba815dbe0 2023-10-16T14:32:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febaecad640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba81f64a0 con 0x7feba80cc6d0 2023-10-16T14:32:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febaecad640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba81f6780 con 0x7feb94003970 2023-10-16T14:32:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 0x7feba81e6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 0x7feba81e6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59416/0 (socket says 172.21.15.29:59416) 2023-10-16T14:32:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 -- 172.21.15.29:0/2903698845 learned_addr learned my addr 172.21.15.29:0/2903698845 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febad4aa640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba80cc6d0 0x7feba815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febadcab640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb94003970 0x7feba81f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 -- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb94003970 msgr2=0x7feba81f58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb94003970 0x7feba81f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 -- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba80cc6d0 msgr2=0x7feba815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba80cc6d0 0x7feba815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febae4ac640 1 -- 172.21.15.29:0/2903698845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba815d3e0 con 0x7feba815dbe0 2023-10-16T14:32:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.482+0000 7febad4aa640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba80cc6d0 0x7feba815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.483+0000 7febae4ac640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 0x7feba81e6550 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7feba00012e0 tx=0x7feba000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.483+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0004d60 con 0x7feba815dbe0 2023-10-16T14:32:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.483+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba000bb80 con 0x7feba815dbe0 2023-10-16T14:32:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.483+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba815d6c0 con 0x7feba815dbe0 2023-10-16T14:32:48.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.483+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0007070 con 0x7feba815dbe0 2023-10-16T14:32:48.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.483+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba815ef40 con 0x7feba815dbe0 2023-10-16T14:32:48.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.484+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba804e9b0 con 0x7feba815dbe0 2023-10-16T14:32:48.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.485+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 88940+0+0 (secure 0 0 0) 0x7feba00071d0 con 0x7feba815dbe0 2023-10-16T14:32:48.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.486+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7feba0089870 con 0x7feba815dbe0 2023-10-16T14:32:48.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.490+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feba0054940 con 0x7feba815dbe0 2023-10-16T14:32:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.546+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7feba008bcf0 con 0x7feba815dbe0 2023-10-16T14:32:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.546+0000 7feb9e7fc640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7feb8c0705c0 0x7feb8c0729b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.550+0000 7febadcab640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7feb8c0705c0 0x7feb8c0729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.556+0000 7febadcab640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7feb8c0705c0 0x7feb8c0729b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feba4036000 tx=0x7feba4030000 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.686+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feba815bc50 con 0x7feba815dbe0 2023-10-16T14:32:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.687+0000 7feb9e7fc640 1 -- 172.21.15.29:0/2903698845 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7feba0074010 con 0x7feba815dbe0 2023-10-16T14:32:48.690 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:32:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.689+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7feb8c0705c0 msgr2=0x7feb8c0729b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.689+0000 7febaecad640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7feb8c0705c0 0x7feb8c0729b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feba4036000 tx=0x7feba4030000 comp rx=0 tx=0).stop 2023-10-16T14:32:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 msgr2=0x7feba81e6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 0x7feba81e6550 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7feba00012e0 tx=0x7feba000b4a0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 shutdown_connections 2023-10-16T14:32:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7feb8c0705c0 0x7feb8c0729b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba815dbe0 0x7feba81e6550 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba80cc6d0 0x7feba815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 --2- 172.21.15.29:0/2903698845 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb94003970 0x7feba81f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 >> 172.21.15.29:0/2903698845 conn(0x7feba80cf570 msgr2=0x7feba80b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 shutdown_connections 2023-10-16T14:32:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.690+0000 7febaecad640 1 -- 172.21.15.29:0/2903698845 wait complete. 2023-10-16T14:32:48.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:48.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.936+0000 7f054d13c640 1 Processor -- start 2023-10-16T14:32:48.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.936+0000 7f054d13c640 1 -- start start 2023-10-16T14:32:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.937+0000 7f054d13c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f054806c040 0x7f054806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.937+0000 7f054d13c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05480cda80 0x7f054815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.937+0000 7f054d13c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054806c960 con 0x7f05480ad970 2023-10-16T14:32:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.937+0000 7f054d13c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054815b9d0 con 0x7f05480cda80 2023-10-16T14:32:48.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.937+0000 7f054d13c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054815bcb0 con 0x7f054806c040 2023-10-16T14:32:48.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f05477fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f05480ad970 0x7f054815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33604/0 (socket says 172.21.15.29:33604) 2023-10-16T14:32:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f05477fe640 1 -- 172.21.15.29:0/3675491311 learned_addr learned my addr 172.21.15.29:0/3675491311 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f054c93b640 1 --2- 172.21.15.29:0/3675491311 >> v2:172.21.15.29:3300/0 conn(0x7f05480cda80 0x7f054815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f0547fff640 1 --2- 172.21.15.29:0/3675491311 >> v2:172.21.15.170:3300/0 conn(0x7f054806c040 0x7f054806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f0546ffd640 1 -- 172.21.15.29:0/3675491311 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162831710 0 0) 0x7f054806c960 con 0x7f05480ad970 2023-10-16T14:32:48.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f0546ffd640 1 -- 172.21.15.29:0/3675491311 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0530003520 con 0x7f05480ad970 2023-10-16T14:32:48.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.938+0000 7f054c93b640 1 -- 172.21.15.29:0/3675491311 >> v2:172.21.15.170:3300/0 conn(0x7f054806c040 msgr2=0x7f054806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:48.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.939+0000 7f054c93b640 1 --2- 172.21.15.29:0/3675491311 >> v2:172.21.15.170:3300/0 conn(0x7f054806c040 0x7f054806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.939+0000 7f054c93b640 1 -- 172.21.15.29:0/3675491311 >> v1:172.21.15.29:6789/0 conn(0x7f05480ad970 legacy=0x7f054815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.939+0000 7f054c93b640 1 -- 172.21.15.29:0/3675491311 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054815bf90 con 0x7f05480cda80 2023-10-16T14:32:48.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.939+0000 7f0547fff640 1 --2- 172.21.15.29:0/3675491311 >> v2:172.21.15.170:3300/0 conn(0x7f054806c040 0x7f054806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:48.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.940+0000 7f054c93b640 1 --2- 172.21.15.29:0/3675491311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f054815ed90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f05480cf350 tx=0x7f054002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f2ee0f3e0bcaa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.940+0000 7f0546ffd640 1 -- 172.21.15.29:0/3675491311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054002cc70 con 0x7f05480cda80 2023-10-16T14:32:48.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.940+0000 7f0546ffd640 1 -- 172.21.15.29:0/3675491311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054002cdd0 con 0x7f05480cda80 2023-10-16T14:32:48.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.940+0000 7f0546ffd640 1 -- 172.21.15.29:0/3675491311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540034660 con 0x7f05480cda80 2023-10-16T14:32:48.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 -- 172.21.15.29:0/3675491311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 msgr2=0x7f054815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 --2- 172.21.15.29:0/3675491311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f054815ed90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f05480cf350 tx=0x7f054002bec0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 -- 172.21.15.29:0/3675491311 shutdown_connections 2023-10-16T14:32:48.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 --2- 172.21.15.29:0/3675491311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f054815ed90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 --2- 172.21.15.29:0/3675491311 >> v2:172.21.15.170:3300/0 conn(0x7f054806c040 0x7f054806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 -- 172.21.15.29:0/3675491311 >> 172.21.15.29:0/3675491311 conn(0x7f05480ce6a0 msgr2=0x7f05480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:48.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 -- 172.21.15.29:0/3675491311 shutdown_connections 2023-10-16T14:32:48.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.941+0000 7f054d13c640 1 -- 172.21.15.29:0/3675491311 wait complete. 2023-10-16T14:32:48.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.943+0000 7f054d13c640 1 Processor -- start 2023-10-16T14:32:48.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.943+0000 7f054d13c640 1 -- start start 2023-10-16T14:32:48.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054d13c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054806c040 0x7f05480b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054d13c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05480ad970 0x7f05481f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054d13c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f05480c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054d13c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480c5bf0 con 0x7f05480cda80 2023-10-16T14:32:48.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054d13c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481f0bf0 con 0x7f054806c040 2023-10-16T14:32:48.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054d13c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05481f0ed0 con 0x7f05480ad970 2023-10-16T14:32:48.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054c93b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f05480c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054c93b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f05480c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59436/0 (socket says 172.21.15.29:59436) 2023-10-16T14:32:48.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f054c93b640 1 -- 172.21.15.29:0/2064894713 learned_addr learned my addr 172.21.15.29:0/2064894713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:48.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.944+0000 7f0547fff640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054806c040 0x7f05480b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f05477fe640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05480ad970 0x7f05481f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054c93b640 1 -- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05480ad970 msgr2=0x7f05481f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054c93b640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05480ad970 0x7f05481f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054c93b640 1 -- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054806c040 msgr2=0x7f05480b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:48.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054c93b640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054806c040 0x7f05480b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:48.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054c93b640 1 -- 172.21.15.29:0/2064894713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540009a70 con 0x7f05480cda80 2023-10-16T14:32:48.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f05477fe640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05480ad970 0x7f05481f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:48.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054c93b640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f05480c7af0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0540009e80 tx=0x7f0540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054003e610 con 0x7f05480cda80 2023-10-16T14:32:48.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054003e770 con 0x7f05480cda80 2023-10-16T14:32:48.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054003eb60 con 0x7f05480cda80 2023-10-16T14:32:48.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.945+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05481f11b0 con 0x7f05480cda80 2023-10-16T14:32:48.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.946+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05481f0840 con 0x7f05480cda80 2023-10-16T14:32:48.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.947+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054806b310 con 0x7f05480cda80 2023-10-16T14:32:48.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.948+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f054003b050 con 0x7f05480cda80 2023-10-16T14:32:48.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.949+0000 7f0544ff9640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f053406acc0 0x7f053406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:48.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.949+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f05400ac0a0 con 0x7f05480cda80 2023-10-16T14:32:48.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.950+0000 7f0547fff640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f053406acc0 0x7f053406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:48.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.951+0000 7f0547fff640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f053406acc0 0x7f053406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0538006fd0 tx=0x7f0538008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:48.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:48.954+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0540077300 con 0x7f05480cda80 2023-10-16T14:32:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.192+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f054804e9b0 con 0x7f05480cda80 2023-10-16T14:32:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.193+0000 7f0544ff9640 1 -- 172.21.15.29:0/2064894713 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f0540039090 con 0x7f05480cda80 2023-10-16T14:32:49.196 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:49.196 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:32:26.521447+0000","modified":"2023-10-16T14:32:29.697489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3822716445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3822716445},{"type":"v1","addr":"172.21.15.170:6837","nonce":3822716445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T14:32:49.197 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2023-10-16T14:32:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.195+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f053406acc0 msgr2=0x7f053406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.195+0000 7f054d13c640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f053406acc0 0x7f053406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0538006fd0 tx=0x7f0538008040 comp rx=0 tx=0).stop 2023-10-16T14:32:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.195+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 msgr2=0x7f05480c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.195+0000 7f054d13c640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f05480c7af0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0540009e80 tx=0x7f0540004a40 comp rx=0 tx=0).stop 2023-10-16T14:32:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 shutdown_connections 2023-10-16T14:32:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f053406acc0 0x7f053406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05480cda80 0x7f05480c7af0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05480ad970 0x7f05481f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 --2- 172.21.15.29:0/2064894713 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f054806c040 0x7f05480b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 >> 172.21.15.29:0/2064894713 conn(0x7f05480ce6a0 msgr2=0x7f054815e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:49.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 shutdown_connections 2023-10-16T14:32:49.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.196+0000 7f054d13c640 1 -- 172.21.15.29:0/2064894713 wait complete. 2023-10-16T14:32:49.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.441+0000 7f9ec61ed640 1 Processor -- start 2023-10-16T14:32:49.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.442+0000 7f9ec61ed640 1 -- start start 2023-10-16T14:32:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.443+0000 7f9ec61ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9ec006c040 0x7f9ec006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.443+0000 7f9ec61ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9ec015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.443+0000 7f9ec61ed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec006c960 con 0x7f9ec00ad970 2023-10-16T14:32:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.443+0000 7f9ec61ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec015b9d0 con 0x7f9ec006c040 2023-10-16T14:32:49.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.443+0000 7f9ec61ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec015bcb0 con 0x7f9ec00cda80 2023-10-16T14:32:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9ec49ea640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9ec00ad970 0x7f9ec015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33614/0 (socket says 172.21.15.29:33614) 2023-10-16T14:32:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9ec49ea640 1 -- 172.21.15.29:0/218325551 learned_addr learned my addr 172.21.15.29:0/218325551 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9ec51eb640 1 --2- 172.21.15.29:0/218325551 >> v2:172.21.15.170:3300/0 conn(0x7f9ec006c040 0x7f9ec006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9ec59ec640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9ec015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666896135 0 0) 0x7f9ec006c960 con 0x7f9ec00ad970 2023-10-16T14:32:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eac003520 con 0x7f9ec00ad970 2023-10-16T14:32:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.444+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2406487809 0 0) 0x7f9eac003520 con 0x7f9ec00ad970 2023-10-16T14:32:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 msgr2=0x7f9ec015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9ec015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 >> v2:172.21.15.170:3300/0 conn(0x7f9ec006c040 msgr2=0x7f9ec006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 --2- 172.21.15.29:0/218325551 >> v2:172.21.15.170:3300/0 conn(0x7f9ec006c040 0x7f9ec006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec015bf90 con 0x7f9ec00ad970 2023-10-16T14:32:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9eb0001670 con 0x7f9ec00ad970 2023-10-16T14:32:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 >> v1:172.21.15.29:6789/0 conn(0x7f9ec00ad970 legacy=0x7f9ec015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac003970 0x7f9eac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9eac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec015b9d0 con 0x7f9eac0047c0 2023-10-16T14:32:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec015bcb0 con 0x7f9eac003970 2023-10-16T14:32:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac00b980 con 0x7f9ec00cda80 2023-10-16T14:32:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9ec49ea640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.445+0000 7f9ec51eb640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac003970 0x7f9eac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec59ec640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9eac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec49ea640 1 -- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 msgr2=0x7f9eac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec49ea640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9eac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec49ea640 1 -- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac003970 msgr2=0x7f9eac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec49ea640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac003970 0x7f9eac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec49ea640 1 -- 172.21.15.29:0/218325551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac003520 con 0x7f9eac0047c0 2023-10-16T14:32:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.446+0000 7f9ec59ec640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec00cda80 0x7f9eac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.447+0000 7f9ec49ea640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000c290 tx=0x7f9eac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0bd223bdf43c56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.447+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0005ad0 con 0x7f9eac0047c0 2023-10-16T14:32:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.447+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0005c30 con 0x7f9eac0047c0 2023-10-16T14:32:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.447+0000 7f9eb7fff640 1 -- 172.21.15.29:0/218325551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb002c890 con 0x7f9eac0047c0 2023-10-16T14:32:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec61ed640 1 -- 172.21.15.29:0/218325551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0047c0 msgr2=0x7f9eac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/218325551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9eac0047c0 0x7f9eac004c40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000c290 tx=0x7f9eac0057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec59ec640 1 -- 172.21.15.29:0/218325551 reap_dead start 2023-10-16T14:32:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec61ed640 1 -- 172.21.15.29:0/218325551 shutdown_connections 2023-10-16T14:32:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec61ed640 1 -- 172.21.15.29:0/218325551 >> 172.21.15.29:0/218325551 conn(0x7f9ec00ce6a0 msgr2=0x7f9ec00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec61ed640 1 -- 172.21.15.29:0/218325551 shutdown_connections 2023-10-16T14:32:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.448+0000 7f9ec61ed640 1 -- 172.21.15.29:0/218325551 wait complete. 2023-10-16T14:32:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.450+0000 7f9ec61ed640 1 Processor -- start 2023-10-16T14:32:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.450+0000 7f9ec61ed640 1 -- start start 2023-10-16T14:32:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec61ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac0047c0 0x7f9ec0153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec61ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec015a070 0x7f9ec0152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec61ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 0x7f9ec00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec61ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00c70f0 con 0x7f9ec00c5770 2023-10-16T14:32:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec61ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00c6860 con 0x7f9eac0047c0 2023-10-16T14:32:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec61ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0152a80 con 0x7f9ec015a070 2023-10-16T14:32:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec59ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 0x7f9ec00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec59ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 0x7f9ec00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59462/0 (socket says 172.21.15.29:59462) 2023-10-16T14:32:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.451+0000 7f9ec59ec640 1 -- 172.21.15.29:0/719594471 learned_addr learned my addr 172.21.15.29:0/719594471 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec59ec640 1 -- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec015a070 msgr2=0x7f9ec0152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec49ea640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec015a070 0x7f9ec0152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec51eb640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac0047c0 0x7f9ec0153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec59ec640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec015a070 0x7f9ec0152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec59ec640 1 -- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac0047c0 msgr2=0x7f9ec0153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec59ec640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac0047c0 0x7f9ec0153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec59ec640 1 -- 172.21.15.29:0/719594471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec00c5fc0 con 0x7f9ec00c5770 2023-10-16T14:32:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec59ec640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 0x7f9ec00ac890 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00b1260 tx=0x7f9eb800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8017400 con 0x7f9ec00c5770 2023-10-16T14:32:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb8011040 con 0x7f9ec00c5770 2023-10-16T14:32:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8017780 con 0x7f9ec00c5770 2023-10-16T14:32:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.452+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec00c62a0 con 0x7f9ec00c5770 2023-10-16T14:32:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.453+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec00c5bf0 con 0x7f9ec00c5770 2023-10-16T14:32:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.454+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec006b330 con 0x7f9ec00c5770 2023-10-16T14:32:49.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.455+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9eb80203b0 con 0x7f9ec00c5770 2023-10-16T14:32:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.456+0000 7f9eb5ffb640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f9ea406acc0 0x7f9ea406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.456+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9eb8089870 con 0x7f9ec00c5770 2023-10-16T14:32:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.456+0000 7f9ec51eb640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f9ea406acc0 0x7f9ea406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.457+0000 7f9ec51eb640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f9ea406acc0 0x7f9ea406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc0098d0 tx=0x7f9ebc008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.460+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9eb80548e0 con 0x7f9ec00c5770 2023-10-16T14:32:49.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.560+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9eb808bcf0 con 0x7f9ec00c5770 2023-10-16T14:32:49.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.667+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ec004e9b0 con 0x7f9ec00c5770 2023-10-16T14:32:49.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.668+0000 7f9eb5ffb640 1 -- 172.21.15.29:0/719594471 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17581 (secure 0 0 0) 0x7f9eb8071210 con 0x7f9ec00c5770 2023-10-16T14:32:49.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:49.672 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:47.524771+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/492174776":"2023-10-17T14:32:47.524726+0000","172.21.15.170:6833/3148236210":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/2958194535":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/4241822527":"2023-10-17T14:32:47.524726+0000","172.21.15.170:6832/3148236210":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/3280001564":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/2013150990":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/1077293322":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/3718615290":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6833/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6832/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/3939932024":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1995904373":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1527444499":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1533610994":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/4190723338":"2023-10-17T14:32:41.327646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:49.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f9ea406acc0 msgr2=0x7f9ea406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f9ea406acc0 0x7f9ea406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc0098d0 tx=0x7f9ebc008040 comp rx=0 tx=0).stop 2023-10-16T14:32:49.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 msgr2=0x7f9ec00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 0x7f9ec00ac890 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00b1260 tx=0x7f9eb800f4a0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 shutdown_connections 2023-10-16T14:32:49.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f9ea406acc0 0x7f9ea406d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ec00c5770 0x7f9ec00ac890 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ec015a070 0x7f9ec0152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 --2- 172.21.15.29:0/719594471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9eac0047c0 0x7f9ec0153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 >> 172.21.15.29:0/719594471 conn(0x7f9ec00ce6a0 msgr2=0x7f9ec00ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:49.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 shutdown_connections 2023-10-16T14:32:49.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.671+0000 7f9ec61ed640 1 -- 172.21.15.29:0/719594471 wait complete. 2023-10-16T14:32:49.692 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:32:49.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.916+0000 7f6a63900640 1 Processor -- start 2023-10-16T14:32:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.916+0000 7f6a63900640 1 -- start start 2023-10-16T14:32:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.917+0000 7f6a63900640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a5c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.917+0000 7f6a63900640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c0cd630 con 0x7f6a5c06c3f0 2023-10-16T14:32:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.917+0000 7f6a63900640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c0ce1b0 con 0x7f6a5c0b4c70 2023-10-16T14:32:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.917+0000 7f6a63900640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c15e7d0 con 0x7f6a5c06c7d0 2023-10-16T14:32:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.918+0000 7f6a6169c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6a5c06c3f0 0x7f6a5c15e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33620/0 (socket says 172.21.15.29:33620) 2023-10-16T14:32:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.918+0000 7f6a6169c640 1 -- 172.21.15.29:0/3391558385 learned_addr learned my addr 172.21.15.29:0/3391558385 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.918+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a5c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.918+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667062024 0 0) 0x7f6a5c0cd630 con 0x7f6a5c06c3f0 2023-10-16T14:32:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a48003520 con 0x7f6a5c06c3f0 2023-10-16T14:32:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291353297 0 0) 0x7f6a48003520 con 0x7f6a5c06c3f0 2023-10-16T14:32:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 msgr2=0x7f6a5c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a5c0b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 >> v1:172.21.15.170:6789/0 conn(0x7f6a5c0b4c70 legacy=0x7f6a5c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c0d2170 con 0x7f6a5c06c3f0 2023-10-16T14:32:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6a580019f0 con 0x7f6a5c06c3f0 2023-10-16T14:32:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 >> v1:172.21.15.29:6789/0 conn(0x7f6a5c06c3f0 legacy=0x7f6a5c15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a480039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a48004d00 0x7f6a48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4800ba30 con 0x7f6a48003f20 2023-10-16T14:32:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4800bba0 con 0x7f6a48004d00 2023-10-16T14:32:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4800bd60 con 0x7f6a5c06c7d0 2023-10-16T14:32:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.919+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a6169c640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a480039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a60e9b640 1 -- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 msgr2=0x7f6a480039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a61e9d640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a48004d00 0x7f6a48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a480039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a60e9b640 1 -- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a48004d00 msgr2=0x7f6a48005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a48004d00 0x7f6a48005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a60e9b640 1 -- 172.21.15.29:0/3391558385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48003520 con 0x7f6a48003f20 2023-10-16T14:32:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a61e9d640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a48004d00 0x7f6a48005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.920+0000 7f6a6169c640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06c7d0 0x7f6a480039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.922+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a480047c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009ac0 tx=0x7f6a480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b769387d5c55561a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.922+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c02c6b0 con 0x7f6a48003f20 2023-10-16T14:32:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.922+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c02c810 con 0x7f6a48003f20 2023-10-16T14:32:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.922+0000 7f6a53fff640 1 -- 172.21.15.29:0/3391558385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c02cba0 con 0x7f6a48003f20 2023-10-16T14:32:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a63900640 1 -- 172.21.15.29:0/3391558385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 msgr2=0x7f6a480047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a63900640 1 --2- 172.21.15.29:0/3391558385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a480047c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009ac0 tx=0x7f6a480057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a61e9d640 1 -- 172.21.15.29:0/3391558385 reap_dead start 2023-10-16T14:32:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a63900640 1 -- 172.21.15.29:0/3391558385 shutdown_connections 2023-10-16T14:32:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a63900640 1 -- 172.21.15.29:0/3391558385 >> 172.21.15.29:0/3391558385 conn(0x7f6a5c0d2cb0 msgr2=0x7f6a5c0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a63900640 1 -- 172.21.15.29:0/3391558385 shutdown_connections 2023-10-16T14:32:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.923+0000 7f6a63900640 1 -- 172.21.15.29:0/3391558385 wait complete. 2023-10-16T14:32:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.925+0000 7f6a63900640 1 Processor -- start 2023-10-16T14:32:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.925+0000 7f6a63900640 1 -- start start 2023-10-16T14:32:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a63900640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a5c0d1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a63900640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c0c91c0 0x7f6a5c0c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a63900640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15a0b0 0x7f6a5c15a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a63900640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c15aa70 con 0x7f6a48003f20 2023-10-16T14:32:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a63900640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0c9620 con 0x7f6a5c0c91c0 2023-10-16T14:32:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a63900640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0ca520 con 0x7f6a5c15a0b0 2023-10-16T14:32:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a6169c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a5c0d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a6169c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a5c0d1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59466/0 (socket says 172.21.15.29:59466) 2023-10-16T14:32:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.926+0000 7f6a6169c640 1 -- 172.21.15.29:0/4095491230 learned_addr learned my addr 172.21.15.29:0/4095491230 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a61e9d640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15a0b0 0x7f6a5c15a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a6169c640 1 -- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15a0b0 msgr2=0x7f6a5c15a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a6169c640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15a0b0 0x7f6a5c15a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a6169c640 1 -- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c0c91c0 msgr2=0x7f6a5c0c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a6169c640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c0c91c0 0x7f6a5c0c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a6169c640 1 -- 172.21.15.29:0/4095491230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c0ca800 con 0x7f6a48003f20 2023-10-16T14:32:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a6169c640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a5c0d1820 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6a58001370 tx=0x7f6a5800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58004d60 con 0x7f6a48003f20 2023-10-16T14:32:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a580070d0 con 0x7f6a48003f20 2023-10-16T14:32:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58007440 con 0x7f6a48003f20 2023-10-16T14:32:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.927+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5c0caae0 con 0x7f6a48003f20 2023-10-16T14:32:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.928+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5c0cb0a0 con 0x7f6a48003f20 2023-10-16T14:32:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.929+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5c04e160 con 0x7f6a48003f20 2023-10-16T14:32:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.930+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6a580075a0 con 0x7f6a48003f20 2023-10-16T14:32:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.930+0000 7f6a51ffb640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6a4006af00 0x7f6a4006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.931+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6a58089db0 con 0x7f6a48003f20 2023-10-16T14:32:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.931+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6a4006af00 0x7f6a4006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.932+0000 7f6a60e9b640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6a4006af00 0x7f6a4006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c036000 tx=0x7f6a4c02e000 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:49.935+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a5808e070 con 0x7f6a48003f20 2023-10-16T14:32:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.169+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a5c0d0c80 con 0x7f6a48003f20 2023-10-16T14:32:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.170+0000 7f6a51ffb640 1 -- 172.21.15.29:0/4095491230 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f6a58019020 con 0x7f6a48003f20 2023-10-16T14:32:50.173 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:50.174 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:32:26.521447+0000","modified":"2023-10-16T14:32:29.697489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3822716445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3822716445},{"type":"v1","addr":"172.21.15.170:6837","nonce":3822716445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T14:32:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2023-10-16T14:32:50.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6a4006af00 msgr2=0x7f6a4006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6a4006af00 0x7f6a4006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c036000 tx=0x7f6a4c02e000 comp rx=0 tx=0).stop 2023-10-16T14:32:50.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 msgr2=0x7f6a5c0d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a5c0d1820 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6a58001370 tx=0x7f6a5800b560 comp rx=0 tx=0).stop 2023-10-16T14:32:50.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 shutdown_connections 2023-10-16T14:32:50.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15a0b0 0x7f6a5c15a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c0c91c0 0x7f6a5c0c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6a4006af00 0x7f6a4006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 --2- 172.21.15.29:0/4095491230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a48003f20 0x7f6a5c0d1820 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 >> 172.21.15.29:0/4095491230 conn(0x7f6a5c0d2cb0 msgr2=0x7f6a5c15f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:50.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 shutdown_connections 2023-10-16T14:32:50.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.173+0000 7f6a63900640 1 -- 172.21.15.29:0/4095491230 wait complete. 2023-10-16T14:32:50.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.423+0000 7fa220c02640 1 Processor -- start 2023-10-16T14:32:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.423+0000 7fa220c02640 1 -- start start 2023-10-16T14:32:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.424+0000 7fa220c02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa21c0ad970 0x7fa21c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.424+0000 7fa220c02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 0x7fa21c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.424+0000 7fa220c02640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c15b9d0 con 0x7fa21c06c040 2023-10-16T14:32:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.424+0000 7fa220c02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c15bcb0 con 0x7fa21c0ad970 2023-10-16T14:32:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.424+0000 7fa220c02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c15bf90 con 0x7fa21c0ccfb0 2023-10-16T14:32:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.424+0000 7fa21b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa21c06c040 0x7fa21c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33622/0 (socket says 172.21.15.29:33622) 2023-10-16T14:32:50.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21b7fe640 1 -- 172.21.15.29:0/404516700 learned_addr learned my addr 172.21.15.29:0/404516700 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21bfff640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 0x7fa21c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21affd640 1 --2- 172.21.15.29:0/404516700 >> v2:172.21.15.170:3300/0 conn(0x7fa21c0ad970 0x7fa21c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188600580 0 0) 0x7fa21c15b9d0 con 0x7fa21c06c040 2023-10-16T14:32:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa208003520 con 0x7fa21c06c040 2023-10-16T14:32:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1553351647 0 0) 0x7fa208003520 con 0x7fa21c06c040 2023-10-16T14:32:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 msgr2=0x7fa21c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 0x7fa21c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 >> v2:172.21.15.170:3300/0 conn(0x7fa21c0ad970 msgr2=0x7fa21c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.425+0000 7fa21a7fc640 1 --2- 172.21.15.29:0/404516700 >> v2:172.21.15.170:3300/0 conn(0x7fa21c0ad970 0x7fa21c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0cc270 con 0x7fa21c06c040 2023-10-16T14:32:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa20c0019f0 con 0x7fa21c06c040 2023-10-16T14:32:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 >> v1:172.21.15.29:6789/0 conn(0x7fa21c06c040 legacy=0x7fa21c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa208003970 0x7fa208003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2080047c0 0x7fa208004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 0x7fa208005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c15bcb0 con 0x7fa208003970 2023-10-16T14:32:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c15bf90 con 0x7fa2080047c0 2023-10-16T14:32:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20800b980 con 0x7fa21c0ccfb0 2023-10-16T14:32:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa208003970 0x7fa208003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21affd640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2080047c0 0x7fa208004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21b7fe640 1 -- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 msgr2=0x7fa208005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.426+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c0ccfb0 0x7fa208005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.427+0000 7fa21b7fe640 1 -- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2080047c0 msgr2=0x7fa208004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.427+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2080047c0 0x7fa208004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.427+0000 7fa21b7fe640 1 -- 172.21.15.29:0/404516700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208003520 con 0x7fa208003970 2023-10-16T14:32:50.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa208003970 0x7fa208003d90 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00a780 tx=0x7fa2080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b51d01cc3255d0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:50.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c034630 con 0x7fa208003970 2023-10-16T14:32:50.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c02f070 con 0x7fa208003970 2023-10-16T14:32:50.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa21a7fc640 1 -- 172.21.15.29:0/404516700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0349d0 con 0x7fa208003970 2023-10-16T14:32:50.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa220c02640 1 -- 172.21.15.29:0/404516700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa208003970 msgr2=0x7fa208003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa220c02640 1 --2- 172.21.15.29:0/404516700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa208003970 0x7fa208003d90 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00a780 tx=0x7fa2080057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.428+0000 7fa21bfff640 1 -- 172.21.15.29:0/404516700 reap_dead start 2023-10-16T14:32:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.429+0000 7fa220c02640 1 -- 172.21.15.29:0/404516700 shutdown_connections 2023-10-16T14:32:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.429+0000 7fa220c02640 1 -- 172.21.15.29:0/404516700 >> 172.21.15.29:0/404516700 conn(0x7fa21c0ce6a0 msgr2=0x7fa21c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.429+0000 7fa220c02640 1 -- 172.21.15.29:0/404516700 shutdown_connections 2023-10-16T14:32:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.429+0000 7fa220c02640 1 -- 172.21.15.29:0/404516700 wait complete. 2023-10-16T14:32:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.430+0000 7fa220c02640 1 Processor -- start 2023-10-16T14:32:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.431+0000 7fa220c02640 1 -- start start 2023-10-16T14:32:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa220c02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa208003970 0x7fa21c0ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa220c02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c14c980 0x7fa21c0c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa220c02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 0x7fa21c15d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa220c02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c15dbe0 con 0x7fa21c15d220 2023-10-16T14:32:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa220c02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0c2f60 con 0x7fa208003970 2023-10-16T14:32:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa220c02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0c2350 con 0x7fa21c14c980 2023-10-16T14:32:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 0x7fa21c15d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 0x7fa21c15d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59496/0 (socket says 172.21.15.29:59496) 2023-10-16T14:32:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 -- 172.21.15.29:0/1987627017 learned_addr learned my addr 172.21.15.29:0/1987627017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 -- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c14c980 msgr2=0x7fa21c0c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c14c980 0x7fa21c0c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa208003970 0x7fa21c0ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 -- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa208003970 msgr2=0x7fa21c0ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa208003970 0x7fa21c0ae680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.432+0000 7fa21bfff640 1 -- 172.21.15.29:0/1987627017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0c2630 con 0x7fa21c15d220 2023-10-16T14:32:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.433+0000 7fa21bfff640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 0x7fa21c15d6a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa21c15c0d0 tx=0x7fa21400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.433+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21400c680 con 0x7fa21c15d220 2023-10-16T14:32:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.433+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21400c7e0 con 0x7fa21c15d220 2023-10-16T14:32:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.433+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21400cb90 con 0x7fa21c15d220 2023-10-16T14:32:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.433+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c0c2910 con 0x7fa21c15d220 2023-10-16T14:32:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.433+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c0c6690 con 0x7fa21c15d220 2023-10-16T14:32:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.434+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c06b330 con 0x7fa21c15d220 2023-10-16T14:32:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.436+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa21400ccf0 con 0x7fa21c15d220 2023-10-16T14:32:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.436+0000 7fa213fff640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa1fc06af50 0x7fa1fc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.437+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa214089540 con 0x7fa21c15d220 2023-10-16T14:32:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.437+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa1fc06af50 0x7fa1fc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.438+0000 7fa21b7fe640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa1fc06af50 0x7fa1fc06d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0013d0 tx=0x7fa20c009030 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:50.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.441+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2140543d0 con 0x7fa21c15d220 2023-10-16T14:32:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.670+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa21c04e9b0 con 0x7fa21c15d220 2023-10-16T14:32:50.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.671+0000 7fa213fff640 1 -- 172.21.15.29:0/1987627017 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fa2140579f0 con 0x7fa21c15d220 2023-10-16T14:32:50.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:50.675 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:32:26.521447+0000","modified":"2023-10-16T14:32:29.697489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5584},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5584":{"gid":5584,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3822716445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3822716445},{"type":"v1","addr":"172.21.15.170:6837","nonce":3822716445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T14:32:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2023-10-16T14:32:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.674+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa1fc06af50 msgr2=0x7fa1fc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.674+0000 7fa220c02640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa1fc06af50 0x7fa1fc06d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0013d0 tx=0x7fa20c009030 comp rx=0 tx=0).stop 2023-10-16T14:32:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.674+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 msgr2=0x7fa21c15d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.674+0000 7fa220c02640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 0x7fa21c15d6a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa21c15c0d0 tx=0x7fa21400ec00 comp rx=0 tx=0).stop 2023-10-16T14:32:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 shutdown_connections 2023-10-16T14:32:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa1fc06af50 0x7fa1fc06d410 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa21c15d220 0x7fa21c15d6a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa21c14c980 0x7fa21c0c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 --2- 172.21.15.29:0/1987627017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa208003970 0x7fa21c0ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 >> 172.21.15.29:0/1987627017 conn(0x7fa21c0ce6a0 msgr2=0x7fa21c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 shutdown_connections 2023-10-16T14:32:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.675+0000 7fa220c02640 1 -- 172.21.15.29:0/1987627017 wait complete. 2023-10-16T14:32:50.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.921+0000 7f39f522e640 1 Processor -- start 2023-10-16T14:32:50.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.921+0000 7f39f522e640 1 -- start start 2023-10-16T14:32:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.922+0000 7f39f522e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.922+0000 7f39f522e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f39f015cfc0 0x7f39f00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39f522e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f00d15a0 con 0x7f39f00d1090 2023-10-16T14:32:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39f522e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f00d1880 con 0x7f39f015cfc0 2023-10-16T14:32:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39f522e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00d1b60 con 0x7f39f015cbe0 2023-10-16T14:32:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39effff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39effff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f00d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:45524/0 (socket says 172.21.15.29:45524) 2023-10-16T14:32:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39effff640 1 -- 172.21.15.29:0/547805858 learned_addr learned my addr 172.21.15.29:0/547805858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.923+0000 7f39ef7fe640 1 --2- 172.21.15.29:0/547805858 >> v2:172.21.15.170:3300/0 conn(0x7f39f015cfc0 0x7f39f00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.924+0000 7f39effff640 1 -- 172.21.15.29:0/547805858 >> v2:172.21.15.170:3300/0 conn(0x7f39f015cfc0 msgr2=0x7f39f00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.924+0000 7f39effff640 1 --2- 172.21.15.29:0/547805858 >> v2:172.21.15.170:3300/0 conn(0x7f39f015cfc0 0x7f39f00b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.924+0000 7f39effff640 1 -- 172.21.15.29:0/547805858 >> v1:172.21.15.170:6789/0 conn(0x7f39f00d1090 legacy=0x7f39f006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.924+0000 7f39effff640 1 -- 172.21.15.29:0/547805858 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00d1e40 con 0x7f39f015cbe0 2023-10-16T14:32:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.925+0000 7f39effff640 1 --2- 172.21.15.29:0/547805858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f00d0b50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009870 tx=0x7f39e002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298d1e38b94253c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39eeffd640 1 -- 172.21.15.29:0/547805858 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e002f830 con 0x7f39f015cbe0 2023-10-16T14:32:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39eeffd640 1 -- 172.21.15.29:0/547805858 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e002f990 con 0x7f39f015cbe0 2023-10-16T14:32:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39eeffd640 1 -- 172.21.15.29:0/547805858 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e002fd20 con 0x7f39f015cbe0 2023-10-16T14:32:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39f522e640 1 -- 172.21.15.29:0/547805858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 msgr2=0x7f39f00d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39f522e640 1 --2- 172.21.15.29:0/547805858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f00d0b50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009870 tx=0x7f39e002ba10 comp rx=0 tx=0).stop 2023-10-16T14:32:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39f522e640 1 -- 172.21.15.29:0/547805858 shutdown_connections 2023-10-16T14:32:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39f522e640 1 --2- 172.21.15.29:0/547805858 >> v2:172.21.15.170:3300/0 conn(0x7f39f015cfc0 0x7f39f00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39f522e640 1 --2- 172.21.15.29:0/547805858 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f00d0b50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.926+0000 7f39f522e640 1 -- 172.21.15.29:0/547805858 >> 172.21.15.29:0/547805858 conn(0x7f39f00cf570 msgr2=0x7f39f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.927+0000 7f39f522e640 1 -- 172.21.15.29:0/547805858 shutdown_connections 2023-10-16T14:32:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.927+0000 7f39f522e640 1 -- 172.21.15.29:0/547805858 wait complete. 2023-10-16T14:32:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.928+0000 7f39f522e640 1 Processor -- start 2023-10-16T14:32:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.929+0000 7f39f522e640 1 -- start start 2023-10-16T14:32:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f522e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00d1090 0x7f39f01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f522e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f01f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f522e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 0x7f39f00c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f522e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00c6870 con 0x7f39f015cfc0 2023-10-16T14:32:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f522e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f01f27d0 con 0x7f39f00d1090 2023-10-16T14:32:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f522e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f01550b0 con 0x7f39f015cbe0 2023-10-16T14:32:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f4a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 0x7f39f00c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f4a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 0x7f39f00c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59510/0 (socket says 172.21.15.29:59510) 2023-10-16T14:32:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f4a2d640 1 -- 172.21.15.29:0/549631496 learned_addr learned my addr 172.21.15.29:0/549631496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39ef7fe640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f01f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39effff640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00d1090 0x7f39f01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.930+0000 7f39f4a2d640 1 -- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 msgr2=0x7f39f01f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f4a2d640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f01f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f4a2d640 1 -- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00d1090 msgr2=0x7f39f01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f4a2d640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00d1090 0x7f39f01529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f4a2d640 1 -- 172.21.15.29:0/549631496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e00094d0 con 0x7f39f015cfc0 2023-10-16T14:32:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39ef7fe640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f01f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f4a2d640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 0x7f39f00c8de0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c3f0 tx=0x7f39e800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e80052d0 con 0x7f39f015cfc0 2023-10-16T14:32:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e8005430 con 0x7f39f015cfc0 2023-10-16T14:32:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8003830 con 0x7f39f015cfc0 2023-10-16T14:32:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f0155390 con 0x7f39f015cfc0 2023-10-16T14:32:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.931+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f00c88c0 con 0x7f39f015cfc0 2023-10-16T14:32:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.932+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f00c8ae0 con 0x7f39f015cfc0 2023-10-16T14:32:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.934+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f39e8003990 con 0x7f39f015cfc0 2023-10-16T14:32:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.934+0000 7f39ecff9640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f39dc06af00 0x7f39dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.935+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f39e8088e80 con 0x7f39f015cfc0 2023-10-16T14:32:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.935+0000 7f39effff640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f39dc06af00 0x7f39dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.936+0000 7f39effff640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f39dc06af00 0x7f39dc06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39e0036000 tx=0x7f39e0039040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:50.939+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39e8053b90 con 0x7f39f015cfc0 2023-10-16T14:32:51.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.140+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39f00c86f0 con 0x7f39f015cfc0 2023-10-16T14:32:51.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.143+0000 7f39ecff9640 1 -- 172.21.15.29:0/549631496 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17581 (secure 0 0 0) 0x7f39e80571b0 con 0x7f39f015cfc0 2023-10-16T14:32:51.147 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:51.147 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:47.524771+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:32:24.444824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T14:32:25.549191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/492174776":"2023-10-17T14:32:47.524726+0000","172.21.15.170:6833/3148236210":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/2958194535":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/4241822527":"2023-10-17T14:32:47.524726+0000","172.21.15.170:6832/3148236210":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/3280001564":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/2013150990":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/1077293322":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/3718615290":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6833/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6832/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/3939932024":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1995904373":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1527444499":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1533610994":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/4190723338":"2023-10-17T14:32:41.327646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f39dc06af00 msgr2=0x7f39dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f39dc06af00 0x7f39dc06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f39e0036000 tx=0x7f39e0039040 comp rx=0 tx=0).stop 2023-10-16T14:32:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 msgr2=0x7f39f00c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 0x7f39f00c8de0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c3f0 tx=0x7f39e800c8b0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 shutdown_connections 2023-10-16T14:32:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f39dc06af00 0x7f39dc06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39f015cfc0 0x7f39f00c8de0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39f015cbe0 0x7f39f01f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 --2- 172.21.15.29:0/549631496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39f00d1090 0x7f39f01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.146+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 >> 172.21.15.29:0/549631496 conn(0x7f39f00cf570 msgr2=0x7f39f00b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:51.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.147+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 shutdown_connections 2023-10-16T14:32:51.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.147+0000 7f39f522e640 1 -- 172.21.15.29:0/549631496 wait complete. 2023-10-16T14:32:51.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:32:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.399+0000 7f3c68eda640 1 Processor -- start 2023-10-16T14:32:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.399+0000 7f3c68eda640 1 -- start start 2023-10-16T14:32:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.400+0000 7f3c68eda640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3c6406c040 0x7f3c6406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c68eda640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c640ad970 0x7f3c6415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c68eda640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6406c960 con 0x7f3c6415ebc0 2023-10-16T14:32:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c68eda640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6415b9d0 con 0x7f3c640ad970 2023-10-16T14:32:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c68eda640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6415bcb0 con 0x7f3c6406c040 2023-10-16T14:32:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c62ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c640ad970 0x7f3c6415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c62ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c640ad970 0x7f3c6415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59522/0 (socket says 172.21.15.29:59522) 2023-10-16T14:32:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c62ffd640 1 -- 172.21.15.29:0/3177222117 learned_addr learned my addr 172.21.15.29:0/3177222117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.401+0000 7f3c637fe640 1 --2- 172.21.15.29:0/3177222117 >> v2:172.21.15.170:3300/0 conn(0x7f3c6406c040 0x7f3c6406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.402+0000 7f3c62ffd640 1 -- 172.21.15.29:0/3177222117 >> v2:172.21.15.170:3300/0 conn(0x7f3c6406c040 msgr2=0x7f3c6406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.402+0000 7f3c62ffd640 1 --2- 172.21.15.29:0/3177222117 >> v2:172.21.15.170:3300/0 conn(0x7f3c6406c040 0x7f3c6406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.402+0000 7f3c62ffd640 1 -- 172.21.15.29:0/3177222117 >> v1:172.21.15.29:6789/0 conn(0x7f3c6415ebc0 legacy=0x7f3c640cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.402+0000 7f3c627fc640 1 -- 172.21.15.29:0/3177222117 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189079752 0 0) 0x7f3c6406c960 con 0x7f3c6415ebc0 2023-10-16T14:32:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.402+0000 7f3c62ffd640 1 -- 172.21.15.29:0/3177222117 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6415bf90 con 0x7f3c640ad970 2023-10-16T14:32:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.403+0000 7f3c62ffd640 1 --2- 172.21.15.29:0/3177222117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c6415e680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3c640cf350 tx=0x7f3c5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162e35d48ee7a554 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c627fc640 1 -- 172.21.15.29:0/3177222117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58030860 con 0x7f3c640ad970 2023-10-16T14:32:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c627fc640 1 -- 172.21.15.29:0/3177222117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c580309c0 con 0x7f3c640ad970 2023-10-16T14:32:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c627fc640 1 -- 172.21.15.29:0/3177222117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58030d50 con 0x7f3c640ad970 2023-10-16T14:32:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 -- 172.21.15.29:0/3177222117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 msgr2=0x7f3c6415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 --2- 172.21.15.29:0/3177222117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c6415e680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3c640cf350 tx=0x7f3c5802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 -- 172.21.15.29:0/3177222117 shutdown_connections 2023-10-16T14:32:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 --2- 172.21.15.29:0/3177222117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c6415e680 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 --2- 172.21.15.29:0/3177222117 >> v2:172.21.15.170:3300/0 conn(0x7f3c6406c040 0x7f3c6406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 -- 172.21.15.29:0/3177222117 >> 172.21.15.29:0/3177222117 conn(0x7f3c640ce6a0 msgr2=0x7f3c640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 -- 172.21.15.29:0/3177222117 shutdown_connections 2023-10-16T14:32:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.404+0000 7f3c68eda640 1 -- 172.21.15.29:0/3177222117 wait complete. 2023-10-16T14:32:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.406+0000 7f3c68eda640 1 Processor -- start 2023-10-16T14:32:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.406+0000 7f3c68eda640 1 -- start start 2023-10-16T14:32:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.407+0000 7f3c68eda640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c6406c040 0x7f3c6414ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.407+0000 7f3c68eda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c640ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.407+0000 7f3c68eda640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c6415ebc0 0x7f3c640acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.407+0000 7f3c68eda640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640c98a0 con 0x7f3c640ad970 2023-10-16T14:32:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.407+0000 7f3c68eda640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640b0e10 con 0x7f3c6415ebc0 2023-10-16T14:32:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.407+0000 7f3c68eda640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6414f110 con 0x7f3c6406c040 2023-10-16T14:32:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c640ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c640ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59530/0 (socket says 172.21.15.29:59530) 2023-10-16T14:32:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 -- 172.21.15.29:0/1766296810 learned_addr learned my addr 172.21.15.29:0/1766296810 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c637fe640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c6406c040 0x7f3c6414ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c63fff640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c6415ebc0 0x7f3c640acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 -- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c6406c040 msgr2=0x7f3c6414ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c6406c040 0x7f3c6414ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 -- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c6415ebc0 msgr2=0x7f3c640acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c6415ebc0 0x7f3c640acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.408+0000 7f3c62ffd640 1 -- 172.21.15.29:0/1766296810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c580094d0 con 0x7f3c640ad970 2023-10-16T14:32:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.409+0000 7f3c62ffd640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c640ac890 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3c58000c00 tx=0x7f3c580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.409+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5803e070 con 0x7f3c640ad970 2023-10-16T14:32:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.409+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58004030 con 0x7f3c640ad970 2023-10-16T14:32:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.409+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c640c8f00 con 0x7f3c640ad970 2023-10-16T14:32:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.409+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58004400 con 0x7f3c640ad970 2023-10-16T14:32:51.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.409+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c640c94c0 con 0x7f3c640ad970 2023-10-16T14:32:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.410+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6406b330 con 0x7f3c640ad970 2023-10-16T14:32:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.411+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3c58042070 con 0x7f3c640ad970 2023-10-16T14:32:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.412+0000 7f3c4bfff640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f3c5006af00 0x7f3c5006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.412+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3c580ab8c0 con 0x7f3c640ad970 2023-10-16T14:32:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.413+0000 7f3c637fe640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f3c5006af00 0x7f3c5006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.414+0000 7f3c637fe640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f3c5006af00 0x7f3c5006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c641f0360 tx=0x7f3c54008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.416+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c58076670 con 0x7f3c640ad970 2023-10-16T14:32:51.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.582+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3c58076a40 con 0x7f3c640ad970 2023-10-16T14:32:51.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.661+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c6406c650 con 0x7f3c640ad970 2023-10-16T14:32:51.665 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:32:51.662+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:32:51.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.666+0000 7f3c4bfff640 1 -- 172.21.15.29:0/1766296810 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3c5808fea0 con 0x7f3c640ad970 2023-10-16T14:32:51.668 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:32:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.668+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f3c5006af00 msgr2=0x7f3c5006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.668+0000 7f3c68eda640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f3c5006af00 0x7f3c5006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c641f0360 tx=0x7f3c54008040 comp rx=0 tx=0).stop 2023-10-16T14:32:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.668+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 msgr2=0x7f3c640ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.668+0000 7f3c68eda640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c640ac890 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3c58000c00 tx=0x7f3c580046c0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 shutdown_connections 2023-10-16T14:32:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f3c5006af00 0x7f3c5006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c640ad970 0x7f3c640ac890 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c6415ebc0 0x7f3c640acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 --2- 172.21.15.29:0/1766296810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c6406c040 0x7f3c6414ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 >> 172.21.15.29:0/1766296810 conn(0x7f3c640ce6a0 msgr2=0x7f3c640c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 shutdown_connections 2023-10-16T14:32:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.669+0000 7f3c68eda640 1 -- 172.21.15.29:0/1766296810 wait complete. 2023-10-16T14:32:51.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:32:51.701 INFO:tasks.ceph.mds.b.smithi170.stderr: -3997> 2023-10-16T14:32:31.186+0000 7ff759a64640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:32:51.791 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:32:51.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.912+0000 7f62dd98e640 1 Processor -- start 2023-10-16T14:32:51.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.912+0000 7f62dd98e640 1 -- start start 2023-10-16T14:32:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dd98e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62d806c040 0x7f62d806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dd98e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d80ad970 0x7f62d815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dd98e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d806c960 con 0x7f62d815ebc0 2023-10-16T14:32:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dd98e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d815b9d0 con 0x7f62d806c040 2023-10-16T14:32:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dd98e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d815bcb0 con 0x7f62d80ad970 2023-10-16T14:32:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dc98c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62d806c040 0x7f62d806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dc98c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62d806c040 0x7f62d806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59542/0 (socket says 172.21.15.29:59542) 2023-10-16T14:32:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.914+0000 7f62dc98c640 1 -- 172.21.15.29:0/2319518448 learned_addr learned my addr 172.21.15.29:0/2319518448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.915+0000 7f62d7fff640 1 --2- 172.21.15.29:0/2319518448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d80ad970 0x7f62d815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.915+0000 7f62dc98c640 1 -- 172.21.15.29:0/2319518448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d80ad970 msgr2=0x7f62d815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.915+0000 7f62dc98c640 1 --2- 172.21.15.29:0/2319518448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d80ad970 0x7f62d815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.915+0000 7f62dc98c640 1 -- 172.21.15.29:0/2319518448 >> v1:172.21.15.170:6789/0 conn(0x7f62d815ebc0 legacy=0x7f62d80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.915+0000 7f62dc98c640 1 -- 172.21.15.29:0/2319518448 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d815bf90 con 0x7f62d806c040 2023-10-16T14:32:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.916+0000 7f62dc98c640 1 --2- 172.21.15.29:0/2319518448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d806c040 0x7f62d806c420 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0098a0 tx=0x7f62cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a09a36ad59d589 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.916+0000 7f62d77fe640 1 -- 172.21.15.29:0/2319518448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc031640 con 0x7f62d806c040 2023-10-16T14:32:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.916+0000 7f62d77fe640 1 -- 172.21.15.29:0/2319518448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc0317a0 con 0x7f62d806c040 2023-10-16T14:32:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.916+0000 7f62d77fe640 1 -- 172.21.15.29:0/2319518448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc031b30 con 0x7f62d806c040 2023-10-16T14:32:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 -- 172.21.15.29:0/2319518448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d806c040 msgr2=0x7f62d806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 --2- 172.21.15.29:0/2319518448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d806c040 0x7f62d806c420 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0098a0 tx=0x7f62cc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 -- 172.21.15.29:0/2319518448 shutdown_connections 2023-10-16T14:32:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 --2- 172.21.15.29:0/2319518448 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d80ad970 0x7f62d815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 --2- 172.21.15.29:0/2319518448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d806c040 0x7f62d806c420 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 -- 172.21.15.29:0/2319518448 >> 172.21.15.29:0/2319518448 conn(0x7f62d80ce6a0 msgr2=0x7f62d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 -- 172.21.15.29:0/2319518448 shutdown_connections 2023-10-16T14:32:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.917+0000 7f62dd98e640 1 -- 172.21.15.29:0/2319518448 wait complete. 2023-10-16T14:32:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.919+0000 7f62dd98e640 1 Processor -- start 2023-10-16T14:32:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.919+0000 7f62dd98e640 1 -- start start 2023-10-16T14:32:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd98e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d806c040 0x7f62d8155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd98e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d80ad970 0x7f62d81543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd98e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 0x7f62d8153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd98e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d8154e10 con 0x7f62d815ebc0 2023-10-16T14:32:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd98e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d81550f0 con 0x7f62d80ad970 2023-10-16T14:32:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd98e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d81553d0 con 0x7f62d806c040 2023-10-16T14:32:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.920+0000 7f62dd18d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 0x7f62d8153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 0x7f62d8153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59552/0 (socket says 172.21.15.29:59552) 2023-10-16T14:32:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 -- 172.21.15.29:0/4137567235 learned_addr learned my addr 172.21.15.29:0/4137567235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dc98c640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d806c040 0x7f62d8155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62d7fff640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d80ad970 0x7f62d81543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 -- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d806c040 msgr2=0x7f62d8155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d806c040 0x7f62d8155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 -- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d80ad970 msgr2=0x7f62d81543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d80ad970 0x7f62d81543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 -- 172.21.15.29:0/4137567235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0094d0 con 0x7f62d815ebc0 2023-10-16T14:32:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62d7fff640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d80ad970 0x7f62d81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.921+0000 7f62dd18d640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 0x7f62d8153760 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c780 tx=0x7f62d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.922+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0004b60 con 0x7f62d815ebc0 2023-10-16T14:32:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.922+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d000ce90 con 0x7f62d815ebc0 2023-10-16T14:32:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.922+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d000a850 con 0x7f62d815ebc0 2023-10-16T14:32:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.922+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d80c5770 con 0x7f62d815ebc0 2023-10-16T14:32:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.922+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d80c7850 con 0x7f62d815ebc0 2023-10-16T14:32:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.923+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d806b330 con 0x7f62d815ebc0 2023-10-16T14:32:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.924+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f62d000a9b0 con 0x7f62d815ebc0 2023-10-16T14:32:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.925+0000 7f62d57fa640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f62bc06af90 0x7f62bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.925+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f62d0089c70 con 0x7f62d815ebc0 2023-10-16T14:32:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.926+0000 7f62dc98c640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f62bc06af90 0x7f62bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.927+0000 7f62dc98c640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f62bc06af90 0x7f62bc06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f62cc036000 tx=0x7f62cc03c040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:51.930+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f62d00548f0 con 0x7f62d815ebc0 2023-10-16T14:32:52.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.170+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f62d804e9b0 con 0x7f62d815ebc0 2023-10-16T14:32:52.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.669+0000 7f62d57fa640 1 -- 172.21.15.29:0/4137567235 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f62d0057f10 con 0x7f62d815ebc0 2023-10-16T14:32:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f62bc06af90 msgr2=0x7f62bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f62bc06af90 0x7f62bc06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f62cc036000 tx=0x7f62cc03c040 comp rx=0 tx=0).stop 2023-10-16T14:32:52.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 msgr2=0x7f62d8153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 0x7f62d8153760 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c780 tx=0x7f62d000cc40 comp rx=0 tx=0).stop 2023-10-16T14:32:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 shutdown_connections 2023-10-16T14:32:52.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f62bc06af90 0x7f62bc06d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d815ebc0 0x7f62d8153760 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d80ad970 0x7f62d81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 --2- 172.21.15.29:0/4137567235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d806c040 0x7f62d8155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.673+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 >> 172.21.15.29:0/4137567235 conn(0x7f62d80ce6a0 msgr2=0x7f62d8159350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:52.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.674+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 shutdown_connections 2023-10-16T14:32:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.674+0000 7f62dd98e640 1 -- 172.21.15.29:0/4137567235 wait complete. 2023-10-16T14:32:52.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:32:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.924+0000 7fec17f68640 1 Processor -- start 2023-10-16T14:32:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.924+0000 7fec17f68640 1 -- start start 2023-10-16T14:32:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.925+0000 7fec17f68640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec1015cf50 0x7fec100d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.925+0000 7fec17f68640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b8a10 0x7fec100d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.925+0000 7fec17f68640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1015d330 con 0x7fec100b4750 2023-10-16T14:32:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec17f68640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec100d1680 con 0x7fec1015cf50 2023-10-16T14:32:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec17f68640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec100d1960 con 0x7fec100b8a10 2023-10-16T14:32:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec15d04640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec1015cf50 0x7fec100d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec16505640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fec100b4750 0x7fec1006bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33636/0 (socket says 172.21.15.29:33636) 2023-10-16T14:32:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec15d04640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec1015cf50 0x7fec100d2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59566/0 (socket says 172.21.15.29:59566) 2023-10-16T14:32:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec16505640 1 -- 172.21.15.29:0/1689445565 learned_addr learned my addr 172.21.15.29:0/1689445565 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.926+0000 7fec15503640 1 --2- 172.21.15.29:0/1689445565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b8a10 0x7fec100d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.927+0000 7fec15d04640 1 -- 172.21.15.29:0/1689445565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b8a10 msgr2=0x7fec100d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.927+0000 7fec15d04640 1 --2- 172.21.15.29:0/1689445565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b8a10 0x7fec100d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.927+0000 7fec15d04640 1 -- 172.21.15.29:0/1689445565 >> v1:172.21.15.29:6789/0 conn(0x7fec100b4750 legacy=0x7fec1006bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.927+0000 7fec15d04640 1 -- 172.21.15.29:0/1689445565 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec100d1c40 con 0x7fec1015cf50 2023-10-16T14:32:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.927+0000 7fec14d02640 1 -- 172.21.15.29:0/1689445565 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160126943 0 0) 0x7fec1015d330 con 0x7fec100b4750 2023-10-16T14:32:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.928+0000 7fec15d04640 1 --2- 172.21.15.29:0/1689445565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec100d2720 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009d90 tx=0x7fec0c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518956955902e1f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.928+0000 7fec14d02640 1 -- 172.21.15.29:0/1689445565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c02cc60 con 0x7fec1015cf50 2023-10-16T14:32:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.928+0000 7fec14d02640 1 -- 172.21.15.29:0/1689445565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c02cdc0 con 0x7fec1015cf50 2023-10-16T14:32:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.928+0000 7fec14d02640 1 -- 172.21.15.29:0/1689445565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c034660 con 0x7fec1015cf50 2023-10-16T14:32:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 -- 172.21.15.29:0/1689445565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 msgr2=0x7fec100d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 --2- 172.21.15.29:0/1689445565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec100d2720 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009d90 tx=0x7fec0c02beb0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 -- 172.21.15.29:0/1689445565 shutdown_connections 2023-10-16T14:32:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 --2- 172.21.15.29:0/1689445565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b8a10 0x7fec100d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 --2- 172.21.15.29:0/1689445565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec100d2720 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 -- 172.21.15.29:0/1689445565 >> 172.21.15.29:0/1689445565 conn(0x7fec100cf750 msgr2=0x7fec100cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:52.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 -- 172.21.15.29:0/1689445565 shutdown_connections 2023-10-16T14:32:52.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.929+0000 7fec17f68640 1 -- 172.21.15.29:0/1689445565 wait complete. 2023-10-16T14:32:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.931+0000 7fec17f68640 1 Processor -- start 2023-10-16T14:32:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.931+0000 7fec17f68640 1 -- start start 2023-10-16T14:32:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec17f68640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b4750 0x7fec100cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec17f68640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec100b8a10 0x7fec101f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec17f68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec101f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec17f68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec101f2730 con 0x7fec1015cf50 2023-10-16T14:32:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec17f68640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec10151a90 con 0x7fec100b8a10 2023-10-16T14:32:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec17f68640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec10154bf0 con 0x7fec100b4750 2023-10-16T14:32:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec16505640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec101f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec16505640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec101f33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59568/0 (socket says 172.21.15.29:59568) 2023-10-16T14:32:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec16505640 1 -- 172.21.15.29:0/394187162 learned_addr learned my addr 172.21.15.29:0/394187162 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.932+0000 7fec15d04640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b4750 0x7fec100cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec15503640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec100b8a10 0x7fec101f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec16505640 1 -- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b4750 msgr2=0x7fec100cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec16505640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b4750 0x7fec100cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec16505640 1 -- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec100b8a10 msgr2=0x7fec101f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec16505640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec100b8a10 0x7fec101f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:52.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec16505640 1 -- 172.21.15.29:0/394187162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c009a70 con 0x7fec1015cf50 2023-10-16T14:32:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec15503640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec100b8a10 0x7fec101f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec16505640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec101f33b0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fec0800c440 tx=0x7fec0800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec080074e0 con 0x7fec1015cf50 2023-10-16T14:32:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec08007640 con 0x7fec1015cf50 2023-10-16T14:32:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec101f31c0 con 0x7fec1015cf50 2023-10-16T14:32:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08004a20 con 0x7fec1015cf50 2023-10-16T14:32:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.933+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec101f2c00 con 0x7fec1015cf50 2023-10-16T14:32:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.934+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec10072350 con 0x7fec1015cf50 2023-10-16T14:32:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.936+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec08004b80 con 0x7fec1015cf50 2023-10-16T14:32:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.936+0000 7fec067fc640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7febf406af90 0x7febf406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.937+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fec08089900 con 0x7fec1015cf50 2023-10-16T14:32:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.937+0000 7fec15d04640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7febf406af90 0x7febf406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.938+0000 7fec15d04640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7febf406af90 0x7febf406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fec0c036000 tx=0x7fec0c030000 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:52.941+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec080544f0 con 0x7fec1015cf50 2023-10-16T14:32:53.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.134+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fec1004e160 con 0x7fec1015cf50 2023-10-16T14:32:53.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.673+0000 7fec067fc640 1 -- 172.21.15.29:0/394187162 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v88) v1 ==== 181+0+0 (secure 0 0 0) 0x7fec08001060 con 0x7fec1015cf50 2023-10-16T14:32:53.676 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:32:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7febf406af90 msgr2=0x7febf406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7febf406af90 0x7febf406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fec0c036000 tx=0x7fec0c030000 comp rx=0 tx=0).stop 2023-10-16T14:32:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 msgr2=0x7fec101f33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec101f33b0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fec0800c440 tx=0x7fec0800c900 comp rx=0 tx=0).stop 2023-10-16T14:32:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 shutdown_connections 2023-10-16T14:32:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7febf406af90 0x7febf406d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec1015cf50 0x7fec101f33b0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec100b8a10 0x7fec101f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 --2- 172.21.15.29:0/394187162 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec100b4750 0x7fec100cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.676+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 >> 172.21.15.29:0/394187162 conn(0x7fec100cf750 msgr2=0x7fec100b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.677+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 shutdown_connections 2023-10-16T14:32:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.677+0000 7fec17f68640 1 -- 172.21.15.29:0/394187162 wait complete. 2023-10-16T14:32:53.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:32:53.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.913+0000 7f314034c640 1 Processor -- start 2023-10-16T14:32:53.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.913+0000 7f314034c640 1 -- start start 2023-10-16T14:32:53.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.914+0000 7f314034c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380b0360 0x7f31380b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:53.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.914+0000 7f314034c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31380ca510 0x7f31380ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:53.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.914+0000 7f314034c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31380b0d00 con 0x7f313815ac10 2023-10-16T14:32:53.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.914+0000 7f314034c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313815f800 con 0x7f31380ca510 2023-10-16T14:32:53.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.914+0000 7f314034c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313806ce40 con 0x7f31380b0360 2023-10-16T14:32:53.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.915+0000 7f313d8e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380b0360 0x7f31380b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:53.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.915+0000 7f313e8e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31380ca510 0x7f31380ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:53.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.915+0000 7f313d8e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380b0360 0x7f31380b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:34328/0 (socket says 172.21.15.29:34328) 2023-10-16T14:32:53.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.915+0000 7f313d8e7640 1 -- 172.21.15.29:0/3653269906 learned_addr learned my addr 172.21.15.29:0/3653269906 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:53.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.916+0000 7f313e8e9640 1 -- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380b0360 msgr2=0x7f31380b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:53.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.916+0000 7f313e8e9640 1 --2- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380b0360 0x7f31380b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.916+0000 7f313e8e9640 1 -- 172.21.15.29:0/3653269906 >> v1:172.21.15.170:6789/0 conn(0x7f313815ac10 legacy=0x7f313806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:53.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.916+0000 7f313e8e9640 1 -- 172.21.15.29:0/3653269906 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313815b310 con 0x7f31380ca510 2023-10-16T14:32:53.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.917+0000 7f313e8e9640 1 --2- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380ca510 0x7f31380ca940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f313002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0c25ef46eca0736 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:53.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.918+0000 7f313d0e6640 1 -- 172.21.15.29:0/3653269906 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313002cbd0 con 0x7f31380ca510 2023-10-16T14:32:53.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.918+0000 7f313d0e6640 1 -- 172.21.15.29:0/3653269906 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313002cd30 con 0x7f31380ca510 2023-10-16T14:32:53.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.918+0000 7f313d0e6640 1 -- 172.21.15.29:0/3653269906 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313002cea0 con 0x7f31380ca510 2023-10-16T14:32:53.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.918+0000 7f314034c640 1 -- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380ca510 msgr2=0x7f31380ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:53.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.918+0000 7f314034c640 1 --2- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380ca510 0x7f31380ca940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f313002be20 comp rx=0 tx=0).stop 2023-10-16T14:32:53.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.919+0000 7f314034c640 1 -- 172.21.15.29:0/3653269906 shutdown_connections 2023-10-16T14:32:53.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.919+0000 7f314034c640 1 --2- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380ca510 0x7f31380ca940 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.919+0000 7f314034c640 1 --2- 172.21.15.29:0/3653269906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380b0360 0x7f31380b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.919+0000 7f314034c640 1 -- 172.21.15.29:0/3653269906 >> 172.21.15.29:0/3653269906 conn(0x7f31380cb360 msgr2=0x7f31380cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:53.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.919+0000 7f314034c640 1 -- 172.21.15.29:0/3653269906 shutdown_connections 2023-10-16T14:32:53.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.919+0000 7f314034c640 1 -- 172.21.15.29:0/3653269906 wait complete. 2023-10-16T14:32:53.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.921+0000 7f314034c640 1 Processor -- start 2023-10-16T14:32:53.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.921+0000 7f314034c640 1 -- start start 2023-10-16T14:32:53.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f314034c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380b0360 0x7f31380c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:53.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f314034c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380ca510 0x7f31380c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:53.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f314034c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 0x7f31380c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:53.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f314034c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380c6070 con 0x7f313815ac10 2023-10-16T14:32:53.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f314034c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380ccd80 con 0x7f31380b0360 2023-10-16T14:32:53.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f314034c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380cd060 con 0x7f31380ca510 2023-10-16T14:32:53.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f313e8e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 0x7f31380c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:53.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f313e8e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 0x7f31380c2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33406/0 (socket says 172.21.15.29:33406) 2023-10-16T14:32:53.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f313e8e9640 1 -- 172.21.15.29:0/1884069249 learned_addr learned my addr 172.21.15.29:0/1884069249 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:53.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f313e0e8640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380b0360 0x7f31380c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:53.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f313e8e9640 1 -- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380ca510 msgr2=0x7f31380c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:53.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.923+0000 7f313d8e7640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380ca510 0x7f31380c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:53.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f313e8e9640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380ca510 0x7f31380c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f313e8e9640 1 -- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380b0360 msgr2=0x7f31380c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:53.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f313e8e9640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380b0360 0x7f31380c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:53.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f313e8e9640 1 -- 172.21.15.29:0/1884069249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130009a70 con 0x7f313815ac10 2023-10-16T14:32:53.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f313e0e8640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380b0360 0x7f31380c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:53.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f313e8e9640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 0x7f31380c2410 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f3130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:53.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313002cea0 con 0x7f313815ac10 2023-10-16T14:32:53.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3130040650 con 0x7f313815ac10 2023-10-16T14:32:53.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130040a00 con 0x7f313815ac10 2023-10-16T14:32:53.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.924+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31380cd340 con 0x7f313815ac10 2023-10-16T14:32:53.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.925+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31380c2950 con 0x7f313815ac10 2023-10-16T14:32:53.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.925+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31380cd860 con 0x7f313815ac10 2023-10-16T14:32:53.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.927+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3130040b60 con 0x7f313815ac10 2023-10-16T14:32:53.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.928+0000 7f312effd640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f311c06af90 0x7f311c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:53.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.928+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6045+0+0 (secure 0 0 0) 0x7f31300ac520 con 0x7f313815ac10 2023-10-16T14:32:53.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.928+0000 7f313e0e8640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f311c06af90 0x7f311c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:53.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.929+0000 7f313e0e8640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f311c06af90 0x7f311c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3134000f60 tx=0x7f3134009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:53.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:53.933+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f31380cd860 con 0x7f313815ac10 2023-10-16T14:32:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.125+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f31380cd860 con 0x7f313815ac10 2023-10-16T14:32:54.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.686+0000 7f312effd640 1 -- 172.21.15.29:0/1884069249 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v89) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31380cd860 con 0x7f313815ac10 2023-10-16T14:32:54.690 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:32:54.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.690+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f311c06af90 msgr2=0x7f311c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:54.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.690+0000 7f314034c640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f311c06af90 0x7f311c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3134000f60 tx=0x7f3134009040 comp rx=0 tx=0).stop 2023-10-16T14:32:54.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.690+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 msgr2=0x7f31380c2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.690+0000 7f314034c640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 0x7f31380c2410 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f3130004a40 comp rx=0 tx=0).stop 2023-10-16T14:32:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.691+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 shutdown_connections 2023-10-16T14:32:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.691+0000 7f314034c640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f311c06af90 0x7f311c06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.691+0000 7f314034c640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f313815ac10 0x7f31380c2410 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.691+0000 7f314034c640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31380ca510 0x7f31380c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.691+0000 7f314034c640 1 --2- 172.21.15.29:0/1884069249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31380b0360 0x7f31380c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.691+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 >> 172.21.15.29:0/1884069249 conn(0x7f31380cb360 msgr2=0x7f313806b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.692+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 shutdown_connections 2023-10-16T14:32:54.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.693+0000 7f314034c640 1 -- 172.21.15.29:0/1884069249 wait complete. 2023-10-16T14:32:54.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:32:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.946+0000 7f7f330d7640 1 Processor -- start 2023-10-16T14:32:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.947+0000 7f7f330d7640 1 -- start start 2023-10-16T14:32:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.948+0000 7f7f330d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7f2c160240 0x7f7f2c0caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.948+0000 7f7f330d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f2c160620 0x7f7f2c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.948+0000 7f7f330d7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c06afe0 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.948+0000 7f7f330d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cf540 con 0x7f7f2c160620 2023-10-16T14:32:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.948+0000 7f7f330d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cf820 con 0x7f7f2c160240 2023-10-16T14:32:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f2c160620 0x7f7f2c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f2c160620 0x7f7f2c0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33420/0 (socket says 172.21.15.29:33420) 2023-10-16T14:32:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 -- 172.21.15.29:0/578647790 learned_addr learned my addr 172.21.15.29:0/578647790 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f320d5640 1 --2- 172.21.15.29:0/578647790 >> v2:172.21.15.170:3300/0 conn(0x7f7f2c160240 0x7f7f2c0caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 -- 172.21.15.29:0/578647790 >> v2:172.21.15.170:3300/0 conn(0x7f7f2c160240 msgr2=0x7f7f2c0caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 --2- 172.21.15.29:0/578647790 >> v2:172.21.15.170:3300/0 conn(0x7f7f2c160240 0x7f7f2c0caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 -- 172.21.15.29:0/578647790 >> v1:172.21.15.170:6789/0 conn(0x7f7f2c0b1ca0 legacy=0x7f7f2c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.949+0000 7f7f318d4640 1 -- 172.21.15.29:0/578647790 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c0cfb00 con 0x7f7f2c160620 2023-10-16T14:32:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.951+0000 7f7f318d4640 1 --2- 172.21.15.29:0/578647790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c160620 0x7f7f2c0ceaa0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c06ff20 tx=0x7f7f1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f9f27139519468 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:54.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.951+0000 7f7f310d3640 1 -- 172.21.15.29:0/578647790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0317e0 con 0x7f7f2c160620 2023-10-16T14:32:54.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.951+0000 7f7f310d3640 1 -- 172.21.15.29:0/578647790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c031940 con 0x7f7f2c160620 2023-10-16T14:32:54.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.951+0000 7f7f310d3640 1 -- 172.21.15.29:0/578647790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c031cd0 con 0x7f7f2c160620 2023-10-16T14:32:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 -- 172.21.15.29:0/578647790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c160620 msgr2=0x7f7f2c0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 --2- 172.21.15.29:0/578647790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c160620 0x7f7f2c0ceaa0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c06ff20 tx=0x7f7f1c02b880 comp rx=0 tx=0).stop 2023-10-16T14:32:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 -- 172.21.15.29:0/578647790 shutdown_connections 2023-10-16T14:32:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 --2- 172.21.15.29:0/578647790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c160620 0x7f7f2c0ceaa0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 --2- 172.21.15.29:0/578647790 >> v2:172.21.15.170:3300/0 conn(0x7f7f2c160240 0x7f7f2c0caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 -- 172.21.15.29:0/578647790 >> 172.21.15.29:0/578647790 conn(0x7f7f2c099a10 msgr2=0x7f7f2c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 -- 172.21.15.29:0/578647790 shutdown_connections 2023-10-16T14:32:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.952+0000 7f7f330d7640 1 -- 172.21.15.29:0/578647790 wait complete. 2023-10-16T14:32:54.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.954+0000 7f7f330d7640 1 Processor -- start 2023-10-16T14:32:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.954+0000 7f7f330d7640 1 -- start start 2023-10-16T14:32:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f330d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 0x7f7f2c0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f330d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c160240 0x7f7f2c0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f330d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f2c160620 0x7f7f2c154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f330d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0c8890 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f330d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0c9e50 con 0x7f7f2c160240 2023-10-16T14:32:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f330d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0c79e0 con 0x7f7f2c160620 2023-10-16T14:32:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 0x7f7f2c0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 0x7f7f2c0c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33436/0 (socket says 172.21.15.29:33436) 2023-10-16T14:32:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 -- 172.21.15.29:0/2672201323 learned_addr learned my addr 172.21.15.29:0/2672201323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 -- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f2c160620 msgr2=0x7f7f2c154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f2c160620 0x7f7f2c154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 -- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c160240 msgr2=0x7f7f2c0c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c160240 0x7f7f2c0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.956+0000 7f7f320d5640 1 -- 172.21.15.29:0/2672201323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c0094d0 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.957+0000 7f7f320d5640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 0x7f7f2c0c9910 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800c7f0 tx=0x7f7f2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.957+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28004220 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.957+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f280043a0 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.957+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28007860 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.957+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c0c7c80 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.957+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c0c8240 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.958+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c04e9b0 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.959+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f28007bd0 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.960+0000 7f7f22ffd640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7f1006af90 0x7f7f1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.960+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7f28089800 con 0x7f7f2c0b1ca0 2023-10-16T14:32:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.961+0000 7f7f318d4640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7f1006af90 0x7f7f1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:54.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.962+0000 7f7f318d4640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7f1006af90 0x7f7f1006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c036000 tx=0x7f7f1c030000 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:54.966+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f28054770 con 0x7f7f2c0b1ca0 2023-10-16T14:32:55.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.166+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7f2c072140 con 0x7f7f2c0b1ca0 2023-10-16T14:32:55.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.172+0000 7f7f22ffd640 1 -- 172.21.15.29:0/2672201323 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7f2801e0a0 con 0x7f7f2c0b1ca0 2023-10-16T14:32:55.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7f1006af90 msgr2=0x7f7f1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7f1006af90 0x7f7f1006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c036000 tx=0x7f7f1c030000 comp rx=0 tx=0).stop 2023-10-16T14:32:55.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 msgr2=0x7f7f2c0c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 0x7f7f2c0c9910 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800c7f0 tx=0x7f7f2800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 shutdown_connections 2023-10-16T14:32:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f2c160620 0x7f7f2c154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c160240 0x7f7f2c0c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7f1006af90 0x7f7f1006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 --2- 172.21.15.29:0/2672201323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c0b1ca0 0x7f7f2c0c9910 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 >> 172.21.15.29:0/2672201323 conn(0x7f7f2c099a10 msgr2=0x7f7f2c06dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 shutdown_connections 2023-10-16T14:32:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.175+0000 7f7f330d7640 1 -- 172.21.15.29:0/2672201323 wait complete. 2023-10-16T14:32:55.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:32:55.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.410+0000 7fe3c7555640 1 Processor -- start 2023-10-16T14:32:55.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.411+0000 7fe3c7555640 1 -- start start 2023-10-16T14:32:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.412+0000 7fe3c7555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c0159ad0 0x7fe3c0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.412+0000 7fe3c7555640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c015d650 0x7fe3c006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.412+0000 7fe3c7555640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00cd260 con 0x7fe3c006b3e0 2023-10-16T14:32:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.412+0000 7fe3c7555640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c0071470 con 0x7fe3c015d650 2023-10-16T14:32:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.412+0000 7fe3c7555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c015de30 con 0x7fe3c0159ad0 2023-10-16T14:32:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c015d650 0x7fe3c006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c015d650 0x7fe3c006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33438/0 (socket says 172.21.15.29:33438) 2023-10-16T14:32:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 -- 172.21.15.29:0/2119600378 learned_addr learned my addr 172.21.15.29:0/2119600378 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c6d54640 1 --1- 172.21.15.29:0/2119600378 >> v1:172.21.15.29:6789/0 conn(0x7fe3c006b3e0 0x7fe3c00c9de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43056/0 (socket says 172.21.15.29:43056) 2023-10-16T14:32:55.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c6553640 1 --2- 172.21.15.29:0/2119600378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c0159ad0 0x7fe3c0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 -- 172.21.15.29:0/2119600378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c0159ad0 msgr2=0x7fe3c0070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 --2- 172.21.15.29:0/2119600378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c0159ad0 0x7fe3c0070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 -- 172.21.15.29:0/2119600378 >> v1:172.21.15.29:6789/0 conn(0x7fe3c006b3e0 legacy=0x7fe3c00c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.413+0000 7fe3c5d52640 1 -- 172.21.15.29:0/2119600378 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c015e110 con 0x7fe3c015d650 2023-10-16T14:32:55.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.414+0000 7fe3c5d52640 1 --2- 172.21.15.29:0/2119600378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c006aea0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000bb10 tx=0x7fe3b002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2c4a47d8d112dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c5551640 1 -- 172.21.15.29:0/2119600378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0036440 con 0x7fe3c015d650 2023-10-16T14:32:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c5551640 1 -- 172.21.15.29:0/2119600378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b00365a0 con 0x7fe3c015d650 2023-10-16T14:32:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c5551640 1 -- 172.21.15.29:0/2119600378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0036930 con 0x7fe3c015d650 2023-10-16T14:32:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c7555640 1 -- 172.21.15.29:0/2119600378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 msgr2=0x7fe3c006aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c7555640 1 --2- 172.21.15.29:0/2119600378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c006aea0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000bb10 tx=0x7fe3b002e150 comp rx=0 tx=0).stop 2023-10-16T14:32:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c7555640 1 -- 172.21.15.29:0/2119600378 shutdown_connections 2023-10-16T14:32:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c7555640 1 --2- 172.21.15.29:0/2119600378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c006aea0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c7555640 1 --2- 172.21.15.29:0/2119600378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c0159ad0 0x7fe3c0070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.415+0000 7fe3c7555640 1 -- 172.21.15.29:0/2119600378 >> 172.21.15.29:0/2119600378 conn(0x7fe3c00b0170 msgr2=0x7fe3c00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.416+0000 7fe3c7555640 1 -- 172.21.15.29:0/2119600378 shutdown_connections 2023-10-16T14:32:55.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.416+0000 7fe3c7555640 1 -- 172.21.15.29:0/2119600378 wait complete. 2023-10-16T14:32:55.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.417+0000 7fe3c7555640 1 Processor -- start 2023-10-16T14:32:55.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.417+0000 7fe3c7555640 1 -- start start 2023-10-16T14:32:55.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.418+0000 7fe3c7555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c006b3e0 0x7fe3c00c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.418+0000 7fe3c7555640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3c0159ad0 0x7fe3c00b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.418+0000 7fe3c7555640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c00b16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.418+0000 7fe3c7555640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c015d150 con 0x7fe3c015d650 2023-10-16T14:32:55.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.418+0000 7fe3c7555640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00c7840 con 0x7fe3c0159ad0 2023-10-16T14:32:55.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.418+0000 7fe3c7555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00c6c30 con 0x7fe3c006b3e0 2023-10-16T14:32:55.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c00b16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c00b16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33452/0 (socket says 172.21.15.29:33452) 2023-10-16T14:32:55.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 -- 172.21.15.29:0/497670059 learned_addr learned my addr 172.21.15.29:0/497670059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:55.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6553640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c006b3e0 0x7fe3c00c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c5d52640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3c0159ad0 0x7fe3c00b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 -- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c006b3e0 msgr2=0x7fe3c00c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c006b3e0 0x7fe3c00c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 -- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3c0159ad0 msgr2=0x7fe3c00b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3c0159ad0 0x7fe3c00b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.419+0000 7fe3c6d54640 1 -- 172.21.15.29:0/497670059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b000b770 con 0x7fe3c015d650 2023-10-16T14:32:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.420+0000 7fe3c6d54640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c00b16e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8001eb0 tx=0x7fe3b800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.420+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80074e0 con 0x7fe3c015d650 2023-10-16T14:32:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.420+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8007640 con 0x7fe3c015d650 2023-10-16T14:32:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.420+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80049e0 con 0x7fe3c015d650 2023-10-16T14:32:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.420+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c00c6f10 con 0x7fe3c015d650 2023-10-16T14:32:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.420+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c00c74d0 con 0x7fe3c015d650 2023-10-16T14:32:55.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.422+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c00c79d0 con 0x7fe3c015d650 2023-10-16T14:32:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.423+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe3b8004b40 con 0x7fe3c015d650 2023-10-16T14:32:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.423+0000 7fe3b6ffd640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fe3a406af90 0x7fe3a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.424+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe3b80890a0 con 0x7fe3c015d650 2023-10-16T14:32:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.424+0000 7fe3c6553640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fe3a406af90 0x7fe3a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.426+0000 7fe3c6553640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fe3a406af90 0x7fe3a406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009a60 tx=0x7fe3bc003990 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.428+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3b8054090 con 0x7fe3c015d650 2023-10-16T14:32:55.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.628+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe3c004e9b0 con 0x7fe3c015d650 2023-10-16T14:32:55.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.637+0000 7fe3b6ffd640 1 -- 172.21.15.29:0/497670059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v26) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe3b80576b0 con 0x7fe3c015d650 2023-10-16T14:32:55.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fe3a406af90 msgr2=0x7fe3a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fe3a406af90 0x7fe3a406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009a60 tx=0x7fe3bc003990 comp rx=0 tx=0).stop 2023-10-16T14:32:55.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 msgr2=0x7fe3c00b16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c00b16e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8001eb0 tx=0x7fe3b800c820 comp rx=0 tx=0).stop 2023-10-16T14:32:55.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 shutdown_connections 2023-10-16T14:32:55.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fe3a406af90 0x7fe3a406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c015d650 0x7fe3c00b16e0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3c0159ad0 0x7fe3c00b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 --2- 172.21.15.29:0/497670059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3c006b3e0 0x7fe3c00c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.642+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 >> 172.21.15.29:0/497670059 conn(0x7fe3c00b0170 msgr2=0x7fe3c00cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:55.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.643+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 shutdown_connections 2023-10-16T14:32:55.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.643+0000 7fe3c7555640 1 -- 172.21.15.29:0/497670059 wait complete. 2023-10-16T14:32:55.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:32:55.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.874+0000 7f9565659640 1 Processor -- start 2023-10-16T14:32:55.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.875+0000 7f9565659640 1 -- start start 2023-10-16T14:32:55.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.876+0000 7f9565659640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f95600ad970 0x7f95600cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.876+0000 7f9565659640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956015b9d0 con 0x7f956006c040 2023-10-16T14:32:55.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.876+0000 7f9565659640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956015bcb0 con 0x7f95600ccfb0 2023-10-16T14:32:55.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.876+0000 7f9565659640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956015bf90 con 0x7f95600ad970 2023-10-16T14:32:55.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f956006c040 0x7f956006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43066/0 (socket says 172.21.15.29:43066) 2023-10-16T14:32:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955ffff640 1 -- 172.21.15.29:0/3158816079 learned_addr learned my addr 172.21.15.29:0/3158816079 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253900622 0 0) 0x7f956015b9d0 con 0x7f956006c040 2023-10-16T14:32:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c003520 con 0x7f956006c040 2023-10-16T14:32:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f9557fff640 1 --2- 172.21.15.29:0/3158816079 >> v2:172.21.15.170:3300/0 conn(0x7f95600ad970 0x7f95600cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136020001 0 0) 0x7f954c003520 con 0x7f956006c040 2023-10-16T14:32:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 >> v2:172.21.15.170:3300/0 conn(0x7f95600ad970 msgr2=0x7f95600cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 --2- 172.21.15.29:0/3158816079 >> v2:172.21.15.170:3300/0 conn(0x7f95600ad970 0x7f95600cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 >> v1:172.21.15.170:6789/0 conn(0x7f95600ccfb0 legacy=0x7f95600cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600cc270 con 0x7f956006c040 2023-10-16T14:32:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9550001670 con 0x7f956006c040 2023-10-16T14:32:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.877+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 >> v1:172.21.15.29:6789/0 conn(0x7f956006c040 legacy=0x7f956006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955f7fe640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0039c0 0x7f954c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955f7fe640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c0047c0 0x7f954c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955f7fe640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f954c0052a0 0x7f954c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956015bf90 con 0x7f954c0047c0 2023-10-16T14:32:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c00c1f0 con 0x7f954c0052a0 2023-10-16T14:32:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c00c3b0 con 0x7f954c0039c0 2023-10-16T14:32:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f9557fff640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c0047c0 0x7f954c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f9564e58640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f954c0052a0 0x7f954c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f955ffff640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0039c0 0x7f954c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f9557fff640 1 -- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0039c0 msgr2=0x7f954c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f9557fff640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0039c0 0x7f954c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f9557fff640 1 -- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f954c0052a0 msgr2=0x7f954c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.878+0000 7f9557fff640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f954c0052a0 0x7f954c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.879+0000 7f9564e58640 1 -- 172.21.15.29:0/3158816079 reap_dead start 2023-10-16T14:32:55.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.879+0000 7f9557fff640 1 -- 172.21.15.29:0/3158816079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c003520 con 0x7f954c0047c0 2023-10-16T14:32:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.880+0000 7f9557fff640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c0047c0 0x7f954c004c40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f95600ade20 tx=0x7f954c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d867b0f31d88edb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.880+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548031660 con 0x7f954c0047c0 2023-10-16T14:32:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.880+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95480317c0 con 0x7f954c0047c0 2023-10-16T14:32:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.880+0000 7f955f7fe640 1 -- 172.21.15.29:0/3158816079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548031b50 con 0x7f954c0047c0 2023-10-16T14:32:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 -- 172.21.15.29:0/3158816079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c0047c0 msgr2=0x7f954c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c0047c0 0x7f954c004c40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f95600ade20 tx=0x7f954c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 -- 172.21.15.29:0/3158816079 shutdown_connections 2023-10-16T14:32:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 --2- 172.21.15.29:0/3158816079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c0047c0 0x7f954c004c40 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 -- 172.21.15.29:0/3158816079 >> 172.21.15.29:0/3158816079 conn(0x7f95600ce6a0 msgr2=0x7f95600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 -- 172.21.15.29:0/3158816079 shutdown_connections 2023-10-16T14:32:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.881+0000 7f9565659640 1 -- 172.21.15.29:0/3158816079 wait complete. 2023-10-16T14:32:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.883+0000 7f9565659640 1 Processor -- start 2023-10-16T14:32:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.883+0000 7f9565659640 1 -- start start 2023-10-16T14:32:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9565659640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0047c0 0x7f95600add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9565659640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9560156e70 0x7f95600b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9565659640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 0x7f956015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9565659640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956015c5e0 con 0x7f95600aff30 2023-10-16T14:32:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9565659640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956015c270 con 0x7f9560156e70 2023-10-16T14:32:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9565659640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95601f2c90 con 0x7f954c0047c0 2023-10-16T14:32:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9564e58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 0x7f956015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9564e58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 0x7f956015d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33470/0 (socket says 172.21.15.29:33470) 2023-10-16T14:32:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9564e58640 1 -- 172.21.15.29:0/3471269642 learned_addr learned my addr 172.21.15.29:0/3471269642 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f955ffff640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0047c0 0x7f95600add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.884+0000 7f9557fff640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9560156e70 0x7f95600b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9564e58640 1 -- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0047c0 msgr2=0x7f95600add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9564e58640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0047c0 0x7f95600add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9564e58640 1 -- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9560156e70 msgr2=0x7f95600b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9564e58640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9560156e70 0x7f95600b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9564e58640 1 -- 172.21.15.29:0/3471269642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95601f2f70 con 0x7f95600aff30 2023-10-16T14:32:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9564e58640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 0x7f956015d9f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f95600b1260 tx=0x7f955800e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558006d60 con 0x7f95600aff30 2023-10-16T14:32:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558007030 con 0x7f95600aff30 2023-10-16T14:32:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580073e0 con 0x7f95600aff30 2023-10-16T14:32:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95600b0bd0 con 0x7f95600aff30 2023-10-16T14:32:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.885+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956015df30 con 0x7f95600aff30 2023-10-16T14:32:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.887+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956006b330 con 0x7f95600aff30 2023-10-16T14:32:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.887+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9558007540 con 0x7f95600aff30 2023-10-16T14:32:55.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.888+0000 7f955d7fa640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f954006afc0 0x7f954006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.888+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f95580893c0 con 0x7f95600aff30 2023-10-16T14:32:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.889+0000 7f955ffff640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f954006afc0 0x7f954006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.890+0000 7f955ffff640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f954006afc0 0x7f954006d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9550001050 tx=0x7f9550009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:55.892+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9558054310 con 0x7f95600aff30 2023-10-16T14:32:56.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.089+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f956015c720 con 0x7f95600aff30 2023-10-16T14:32:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.094+0000 7f955d7fa640 1 -- 172.21.15.29:0/3471269642 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7f956015c720 con 0x7f95600aff30 2023-10-16T14:32:56.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f954006afc0 msgr2=0x7f954006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f954006afc0 0x7f954006d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9550001050 tx=0x7f9550009040 comp rx=0 tx=0).stop 2023-10-16T14:32:56.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 msgr2=0x7f956015d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 0x7f956015d9f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f95600b1260 tx=0x7f955800e9e0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 shutdown_connections 2023-10-16T14:32:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f954006afc0 0x7f954006d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95600aff30 0x7f956015d9f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9560156e70 0x7f95600b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 --2- 172.21.15.29:0/3471269642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f954c0047c0 0x7f95600add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 >> 172.21.15.29:0/3471269642 conn(0x7f95600ce6a0 msgr2=0x7f95600c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:56.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 shutdown_connections 2023-10-16T14:32:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.098+0000 7f9565659640 1 -- 172.21.15.29:0/3471269642 wait complete. 2023-10-16T14:32:56.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T14:32:56.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.352+0000 7f7cfa20d640 1 Processor -- start 2023-10-16T14:32:56.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.352+0000 7f7cfa20d640 1 -- start start 2023-10-16T14:32:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cfa20d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7cf40712c0 0x7f7cf4070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cfa20d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf40b4620 con 0x7f7cf415aad0 2023-10-16T14:32:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cfa20d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf40b4340 con 0x7f7cf4070d70 2023-10-16T14:32:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cfa20d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf406f520 con 0x7f7cf40712c0 2023-10-16T14:32:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cf920b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7cf415aad0 0x7f7cf40cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43082/0 (socket says 172.21.15.29:43082) 2023-10-16T14:32:56.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cf920b640 1 -- 172.21.15.29:0/1802422238 learned_addr learned my addr 172.21.15.29:0/1802422238 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:56.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.354+0000 7f7cf9a0c640 1 --2- 172.21.15.29:0/1802422238 >> v2:172.21.15.29:3300/0 conn(0x7f7cf40712c0 0x7f7cf4070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.355+0000 7f7cebfff640 1 -- 172.21.15.29:0/1802422238 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231639423 0 0) 0x7f7cf40b4620 con 0x7f7cf415aad0 2023-10-16T14:32:56.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.355+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/1802422238 >> v1:172.21.15.170:6789/0 conn(0x7f7cf4070d70 legacy=0x7f7cf406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:56.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.355+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/1802422238 >> v1:172.21.15.29:6789/0 conn(0x7f7cf415aad0 legacy=0x7f7cf40cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.355+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/1802422238 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf406f800 con 0x7f7cf40712c0 2023-10-16T14:32:56.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.356+0000 7f7cf9a0c640 1 --2- 172.21.15.29:0/1802422238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf40712c0 0x7f7cf4070000 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9864934b880e43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:56.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.356+0000 7f7cebfff640 1 -- 172.21.15.29:0/1802422238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec02cd00 con 0x7f7cf40712c0 2023-10-16T14:32:56.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.356+0000 7f7cebfff640 1 -- 172.21.15.29:0/1802422238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec02ce60 con 0x7f7cf40712c0 2023-10-16T14:32:56.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.356+0000 7f7cebfff640 1 -- 172.21.15.29:0/1802422238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec034660 con 0x7f7cf40712c0 2023-10-16T14:32:56.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 -- 172.21.15.29:0/1802422238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf40712c0 msgr2=0x7f7cf4070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/1802422238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf40712c0 0x7f7cf4070000 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec02bf50 comp rx=0 tx=0).stop 2023-10-16T14:32:56.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 -- 172.21.15.29:0/1802422238 shutdown_connections 2023-10-16T14:32:56.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/1802422238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf40712c0 0x7f7cf4070000 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 -- 172.21.15.29:0/1802422238 >> 172.21.15.29:0/1802422238 conn(0x7f7cf40b0a90 msgr2=0x7f7cf40ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:56.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 -- 172.21.15.29:0/1802422238 shutdown_connections 2023-10-16T14:32:56.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.357+0000 7f7cfa20d640 1 -- 172.21.15.29:0/1802422238 wait complete. 2023-10-16T14:32:56.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.359+0000 7f7cfa20d640 1 Processor -- start 2023-10-16T14:32:56.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.359+0000 7f7cfa20d640 1 -- start start 2023-10-16T14:32:56.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cfa20d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cf4070d70 0x7f7cf41eff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cfa20d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cf40712c0 0x7f7cf415d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cfa20d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 0x7f7cf40cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cfa20d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40c5c70 con 0x7f7cf415aad0 2023-10-16T14:32:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cfa20d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40c5f50 con 0x7f7cf4070d70 2023-10-16T14:32:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cfa20d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40c6230 con 0x7f7cf40712c0 2023-10-16T14:32:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cf9a0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 0x7f7cf40cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cf9a0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 0x7f7cf40cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33488/0 (socket says 172.21.15.29:33488) 2023-10-16T14:32:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.360+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/685969935 learned_addr learned my addr 172.21.15.29:0/685969935 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:56.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf920b640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cf4070d70 0x7f7cf41eff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cf40712c0 msgr2=0x7f7cf415d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf8a0a640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cf40712c0 0x7f7cf415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf9a0c640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cf40712c0 0x7f7cf415d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cf4070d70 msgr2=0x7f7cf41eff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf9a0c640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cf4070d70 0x7f7cf41eff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf9a0c640 1 -- 172.21.15.29:0/685969935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec009a70 con 0x7f7cf415aad0 2023-10-16T14:32:56.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf920b640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cf4070d70 0x7f7cf41eff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:56.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf8a0a640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cf40712c0 0x7f7cf415d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:56.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cf9a0c640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 0x7f7cf40cc630 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:56.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec03e670 con 0x7f7cf415aad0 2023-10-16T14:32:56.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec03e7f0 con 0x7f7cf415aad0 2023-10-16T14:32:56.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec03eba0 con 0x7f7cf415aad0 2023-10-16T14:32:56.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.361+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf40c6510 con 0x7f7cf415aad0 2023-10-16T14:32:56.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.362+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf415de70 con 0x7f7cf415aad0 2023-10-16T14:32:56.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.363+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf40c6a30 con 0x7f7cf415aad0 2023-10-16T14:32:56.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.364+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7cec03b040 con 0x7f7cf415aad0 2023-10-16T14:32:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.365+0000 7f7ce9ffb640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7cd806af90 0x7f7cd806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.365+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7cec0ac380 con 0x7f7cf415aad0 2023-10-16T14:32:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.366+0000 7f7cf920b640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7cd806af90 0x7f7cd806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.367+0000 7f7cf920b640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7cd806af90 0x7f7cd806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0000f60 tx=0x7f7cf0009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.370+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7cf40c6a30 con 0x7f7cf415aad0 2023-10-16T14:32:56.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.638+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f7cf40c6a30 con 0x7f7cf415aad0 2023-10-16T14:32:56.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.691+0000 7f7ce9ffb640 1 -- 172.21.15.29:0/685969935 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v608) v1 ==== 156+0+0 (secure 0 0 0) 0x7f7cf40c6a30 con 0x7f7cf415aad0 2023-10-16T14:32:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7cd806af90 msgr2=0x7f7cd806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7cd806af90 0x7f7cd806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0000f60 tx=0x7f7cf0009040 comp rx=0 tx=0).stop 2023-10-16T14:32:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 msgr2=0x7f7cf40cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 0x7f7cf40cc630 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec004a40 comp rx=0 tx=0).stop 2023-10-16T14:32:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 shutdown_connections 2023-10-16T14:32:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7cd806af90 0x7f7cd806d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7cf415aad0 0x7f7cf40cc630 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7cf40712c0 0x7f7cf415d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.695+0000 7f7cfa20d640 1 --2- 172.21.15.29:0/685969935 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cf4070d70 0x7f7cf41eff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.696+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 >> 172.21.15.29:0/685969935 conn(0x7f7cf40b0a90 msgr2=0x7f7cf4159da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.696+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 shutdown_connections 2023-10-16T14:32:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.696+0000 7f7cfa20d640 1 -- 172.21.15.29:0/685969935 wait complete. 2023-10-16T14:32:56.719 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-16T14:32:56.719 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-16T14:32:56.720 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:32:56.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:32:56.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.945+0000 7f747759e640 1 Processor -- start 2023-10-16T14:32:56.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.945+0000 7f747759e640 1 -- start start 2023-10-16T14:32:56.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.946+0000 7f747759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.946+0000 7f747759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7478070950 0x7f747815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.946+0000 7f747759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780ca3a0 con 0x7f7478159e70 2023-10-16T14:32:56.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.946+0000 7f747759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74780cad70 con 0x7f7478070950 2023-10-16T14:32:56.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.946+0000 7f747759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74780c9d70 con 0x7f74781573e0 2023-10-16T14:32:56.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.947+0000 7f747659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.947+0000 7f747659c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:34410/0 (socket says 172.21.15.29:34410) 2023-10-16T14:32:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.947+0000 7f746dd9b640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f7478159e70 0x7f7478070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:42448/0 (socket says 172.21.15.29:42448) 2023-10-16T14:32:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.947+0000 7f7476d9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7478070950 0x7f747815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.947+0000 7f747659c640 1 -- 172.21.15.29:0/1342656825 learned_addr learned my addr 172.21.15.29:0/1342656825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.948+0000 7f747659c640 1 -- 172.21.15.29:0/1342656825 >> v2:172.21.15.170:3300/0 conn(0x7f7478070950 msgr2=0x7f747815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.948+0000 7f747659c640 1 --2- 172.21.15.29:0/1342656825 >> v2:172.21.15.170:3300/0 conn(0x7f7478070950 0x7f747815a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.948+0000 7f747659c640 1 -- 172.21.15.29:0/1342656825 >> v1:172.21.15.170:6789/0 conn(0x7f7478159e70 legacy=0x7f7478070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.948+0000 7f747659c640 1 -- 172.21.15.29:0/1342656825 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74781577c0 con 0x7f74781573e0 2023-10-16T14:32:56.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.950+0000 7f747659c640 1 --2- 172.21.15.29:0/1342656825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780ca830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7468009870 tx=0x7f746802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=870ce197a12c125d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:56.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.950+0000 7f7475d9b640 1 -- 172.21.15.29:0/1342656825 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468030840 con 0x7f74781573e0 2023-10-16T14:32:56.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.950+0000 7f7475d9b640 1 -- 172.21.15.29:0/1342656825 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74680309a0 con 0x7f74781573e0 2023-10-16T14:32:56.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.950+0000 7f7475d9b640 1 -- 172.21.15.29:0/1342656825 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468030d30 con 0x7f74781573e0 2023-10-16T14:32:56.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 -- 172.21.15.29:0/1342656825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 msgr2=0x7f74780ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 --2- 172.21.15.29:0/1342656825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780ca830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7468009870 tx=0x7f746802b980 comp rx=0 tx=0).stop 2023-10-16T14:32:56.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 -- 172.21.15.29:0/1342656825 shutdown_connections 2023-10-16T14:32:56.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 --2- 172.21.15.29:0/1342656825 >> v2:172.21.15.170:3300/0 conn(0x7f7478070950 0x7f747815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 --2- 172.21.15.29:0/1342656825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780ca830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 -- 172.21.15.29:0/1342656825 >> 172.21.15.29:0/1342656825 conn(0x7f74780cc230 msgr2=0x7f74780cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 -- 172.21.15.29:0/1342656825 shutdown_connections 2023-10-16T14:32:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.951+0000 7f747759e640 1 -- 172.21.15.29:0/1342656825 wait complete. 2023-10-16T14:32:56.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.954+0000 7f747759e640 1 Processor -- start 2023-10-16T14:32:56.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.954+0000 7f747759e640 1 -- start start 2023-10-16T14:32:56.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 0x7f7478151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7478159e70 0x7f747815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74780c9670 con 0x7f7478070950 2023-10-16T14:32:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74780c90b0 con 0x7f7478159e70 2023-10-16T14:32:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74780c9390 con 0x7f74781573e0 2023-10-16T14:32:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 0x7f7478151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 0x7f7478151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33504/0 (socket says 172.21.15.29:33504) 2023-10-16T14:32:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.955+0000 7f747659c640 1 -- 172.21.15.29:0/2353384436 learned_addr learned my addr 172.21.15.29:0/2353384436 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f7476d9d640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7478159e70 0x7f747815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f747659c640 1 -- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 msgr2=0x7f74780c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f747659c640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f747659c640 1 -- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7478159e70 msgr2=0x7f747815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f747659c640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7478159e70 0x7f747815aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f747659c640 1 -- 172.21.15.29:0/2353384436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74680094d0 con 0x7f7478070950 2023-10-16T14:32:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f747659c640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 0x7f7478151cc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7468009870 tx=0x7f7468004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.956+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468039c90 con 0x7f7478070950 2023-10-16T14:32:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.957+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7468039df0 con 0x7f7478070950 2023-10-16T14:32:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.957+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74780ac9c0 con 0x7f7478070950 2023-10-16T14:32:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.957+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468043640 con 0x7f7478070950 2023-10-16T14:32:56.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.957+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7478152200 con 0x7f7478070950 2023-10-16T14:32:56.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.958+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747804e9b0 con 0x7f7478070950 2023-10-16T14:32:56.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.959+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74680437a0 con 0x7f7478070950 2023-10-16T14:32:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.960+0000 7f746f7fe640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f745806af90 0x7f745806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.961+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f746803e070 con 0x7f7478070950 2023-10-16T14:32:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.961+0000 7f746dd9b640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f745806af90 0x7f745806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.962+0000 7f746dd9b640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f745806af90 0x7f745806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7460000f60 tx=0x7f7460007a00 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:56.965+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7468077780 con 0x7f7478070950 2023-10-16T14:32:57.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.158+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74780b5730 con 0x7f7478070950 2023-10-16T14:32:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.161+0000 7f746f7fe640 1 -- 172.21.15.29:0/2353384436 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+14005 (secure 0 0 0) 0x7f746807ada0 con 0x7f7478070950 2023-10-16T14:32:57.164 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:32:57.164 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":89,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:32:54.683508+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/492174776":"2023-10-17T14:32:47.524726+0000","172.21.15.170:6837/3822716445":"2023-10-17T14:32:51.662610+0000","172.21.15.170:6833/3148236210":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/2958194535":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/4241822527":"2023-10-17T14:32:47.524726+0000","172.21.15.170:6832/3148236210":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/3280001564":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/2013150990":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/1077293322":"2023-10-17T14:32:47.524726+0000","172.21.15.170:0/3718615290":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6833/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:6835/3822716445":"2023-10-17T14:32:51.662610+0000","172.21.15.170:6832/885795747":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/3939932024":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1995904373":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1527444499":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/1533610994":"2023-10-17T14:32:41.327646+0000","172.21.15.170:0/4190723338":"2023-10-17T14:32:41.327646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:32:57.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f745806af90 msgr2=0x7f745806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f745806af90 0x7f745806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7460000f60 tx=0x7f7460007a00 comp rx=0 tx=0).stop 2023-10-16T14:32:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 msgr2=0x7f7478151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 0x7f7478151cc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7468009870 tx=0x7f7468004730 comp rx=0 tx=0).stop 2023-10-16T14:32:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 shutdown_connections 2023-10-16T14:32:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7478159e70 0x7f747815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74781573e0 0x7f74780c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f745806af90 0x7f745806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 --2- 172.21.15.29:0/2353384436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7478070950 0x7f7478151cc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 >> 172.21.15.29:0/2353384436 conn(0x7f74780cc230 msgr2=0x7f74780c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.163+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 shutdown_connections 2023-10-16T14:32:57.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.164+0000 7f747759e640 1 -- 172.21.15.29:0/2353384436 wait complete. 2023-10-16T14:32:57.185 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:32:57.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:32:57.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.415+0000 7f25e3fff640 1 Processor -- start 2023-10-16T14:32:57.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.416+0000 7f25e3fff640 1 -- start start 2023-10-16T14:32:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.417+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 0x7f25e4070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.417+0000 7f25e3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f25e415d650 0x7f25e406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.417+0000 7f25e3fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40cd260 con 0x7f25e406b3e0 2023-10-16T14:32:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.417+0000 7f25e3fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e4071470 con 0x7f25e415d650 2023-10-16T14:32:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.417+0000 7f25e3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e415de30 con 0x7f25e4159ad0 2023-10-16T14:32:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.418+0000 7f25e37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25e406b3e0 0x7f25e40c9de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43096/0 (socket says 172.21.15.29:43096) 2023-10-16T14:32:57.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.418+0000 7f25e37fe640 1 -- 172.21.15.29:0/141012811 learned_addr learned my addr 172.21.15.29:0/141012811 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:57.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.418+0000 7f25e27fc640 1 --2- 172.21.15.29:0/141012811 >> v2:172.21.15.170:3300/0 conn(0x7f25e415d650 0x7f25e406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.418+0000 7f25e2ffd640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 0x7f25e4070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.418+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290389407 0 0) 0x7f25e40cd260 con 0x7f25e406b3e0 2023-10-16T14:32:57.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.418+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc005730 con 0x7f25e406b3e0 2023-10-16T14:32:57.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1868426535 0 0) 0x7f25cc005730 con 0x7f25e406b3e0 2023-10-16T14:32:57.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 msgr2=0x7f25e4070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 0x7f25e4070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 >> v2:172.21.15.170:3300/0 conn(0x7f25e415d650 msgr2=0x7f25e406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 --2- 172.21.15.29:0/141012811 >> v2:172.21.15.170:3300/0 conn(0x7f25e415d650 0x7f25e406aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e415e110 con 0x7f25e406b3e0 2023-10-16T14:32:57.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f25dc001670 con 0x7f25e406b3e0 2023-10-16T14:32:57.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 >> v1:172.21.15.29:6789/0 conn(0x7f25e406b3e0 legacy=0x7f25e40c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 0x7f25cc005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25cc006150 0x7f25cc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25cc006f10 0x7f25cc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e4071470 con 0x7f25cc006150 2023-10-16T14:32:57.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e415de30 con 0x7f25cc006f10 2023-10-16T14:32:57.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc00db90 con 0x7f25e4159ad0 2023-10-16T14:32:57.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.419+0000 7f25e27fc640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25cc006150 0x7f25cc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e2ffd640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 0x7f25cc005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e37fe640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25cc006f10 0x7f25cc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e27fc640 1 -- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 msgr2=0x7f25cc005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e27fc640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25e4159ad0 0x7f25cc005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e27fc640 1 -- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25cc006f10 msgr2=0x7f25cc007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e27fc640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25cc006f10 0x7f25cc007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e27fc640 1 -- 172.21.15.29:0/141012811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc005730 con 0x7f25cc006150 2023-10-16T14:32:57.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.420+0000 7f25e37fe640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25cc006f10 0x7f25cc007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:57.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.421+0000 7f25e27fc640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25cc006150 0x7f25cc0069d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009af0 tx=0x7f25cc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194e12afc421d27c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:57.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.421+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d802c710 con 0x7f25cc006150 2023-10-16T14:32:57.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.421+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d802c870 con 0x7f25cc006150 2023-10-16T14:32:57.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.421+0000 7f25e1ffb640 1 -- 172.21.15.29:0/141012811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d802cc00 con 0x7f25cc006150 2023-10-16T14:32:57.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e3fff640 1 -- 172.21.15.29:0/141012811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25cc006150 msgr2=0x7f25cc0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e3fff640 1 --2- 172.21.15.29:0/141012811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25cc006150 0x7f25cc0069d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009af0 tx=0x7f25cc007a00 comp rx=0 tx=0).stop 2023-10-16T14:32:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e37fe640 1 -- 172.21.15.29:0/141012811 reap_dead start 2023-10-16T14:32:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e3fff640 1 -- 172.21.15.29:0/141012811 shutdown_connections 2023-10-16T14:32:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e3fff640 1 -- 172.21.15.29:0/141012811 >> 172.21.15.29:0/141012811 conn(0x7f25e40b0170 msgr2=0x7f25e40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e3fff640 1 -- 172.21.15.29:0/141012811 shutdown_connections 2023-10-16T14:32:57.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.422+0000 7f25e3fff640 1 -- 172.21.15.29:0/141012811 wait complete. 2023-10-16T14:32:57.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.424+0000 7f25e3fff640 1 Processor -- start 2023-10-16T14:32:57.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.424+0000 7f25e3fff640 1 -- start start 2023-10-16T14:32:57.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25cc006150 0x7f25e414cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 0x7f25e40c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25e40ae050 0x7f25e40ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e40aea40 con 0x7f25e40c64e0 2023-10-16T14:32:57.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e3fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e40c8520 con 0x7f25e40ae050 2023-10-16T14:32:57.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e40c7480 con 0x7f25cc006150 2023-10-16T14:32:57.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 0x7f25e40c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 0x7f25e40c6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33524/0 (socket says 172.21.15.29:33524) 2023-10-16T14:32:57.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.425+0000 7f25e27fc640 1 -- 172.21.15.29:0/2004094253 learned_addr learned my addr 172.21.15.29:0/2004094253 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e2ffd640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25cc006150 0x7f25e414cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e37fe640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25e40ae050 0x7f25e40ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e27fc640 1 -- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25cc006150 msgr2=0x7f25e414cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e27fc640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25cc006150 0x7f25e414cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e27fc640 1 -- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25e40ae050 msgr2=0x7f25e40ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e27fc640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25e40ae050 0x7f25e40ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e27fc640 1 -- 172.21.15.29:0/2004094253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e40c7760 con 0x7f25e40c64e0 2023-10-16T14:32:57.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e37fe640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25e40ae050 0x7f25e40ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:57.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.426+0000 7f25e27fc640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 0x7f25e40c6b50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f25d8000c00 tx=0x7f25d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.427+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8003810 con 0x7f25e40c64e0 2023-10-16T14:32:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.427+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d802de90 con 0x7f25e40c64e0 2023-10-16T14:32:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.427+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8003b90 con 0x7f25e40c64e0 2023-10-16T14:32:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.427+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e40c7a40 con 0x7f25e40c64e0 2023-10-16T14:32:57.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.427+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e40c8000 con 0x7f25e40c64e0 2023-10-16T14:32:57.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.428+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e40c8190 con 0x7f25e40c64e0 2023-10-16T14:32:57.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.429+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f25d8003cf0 con 0x7f25e40c64e0 2023-10-16T14:32:57.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.430+0000 7f25cbfff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f25d006afc0 0x7f25d006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.430+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f25d80ab8b0 con 0x7f25e40c64e0 2023-10-16T14:32:57.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.431+0000 7f25e2ffd640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f25d006afc0 0x7f25d006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.432+0000 7f25e2ffd640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f25d006afc0 0x7f25d006d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25d4007440 tx=0x7f25d400b040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:57.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.434+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25d8076770 con 0x7f25e40c64e0 2023-10-16T14:32:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.629+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25e404e9b0 con 0x7f25e40c64e0 2023-10-16T14:32:57.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.630+0000 7f25cbfff640 1 -- 172.21.15.29:0/2004094253 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25d80390a0 con 0x7f25e40c64e0 2023-10-16T14:32:57.633 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:32:57.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.632+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f25d006afc0 msgr2=0x7f25d006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.632+0000 7f25e3fff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f25d006afc0 0x7f25d006d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25d4007440 tx=0x7f25d400b040 comp rx=0 tx=0).stop 2023-10-16T14:32:57.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.632+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 msgr2=0x7f25e40c6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.632+0000 7f25e3fff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 0x7f25e40c6b50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f25d8000c00 tx=0x7f25d8004730 comp rx=0 tx=0).stop 2023-10-16T14:32:57.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 shutdown_connections 2023-10-16T14:32:57.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f25d006afc0 0x7f25d006d480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25e40c64e0 0x7f25e40c6b50 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25e40ae050 0x7f25e40ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 --2- 172.21.15.29:0/2004094253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25cc006150 0x7f25e414cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 >> 172.21.15.29:0/2004094253 conn(0x7f25e40b0170 msgr2=0x7f25e4157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:57.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 shutdown_connections 2023-10-16T14:32:57.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.633+0000 7f25e3fff640 1 -- 172.21.15.29:0/2004094253 wait complete. 2023-10-16T14:32:57.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T14:32:57.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.881+0000 7f99925d0640 1 Processor -- start 2023-10-16T14:32:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.882+0000 7f99925d0640 1 -- start start 2023-10-16T14:32:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.883+0000 7f99925d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c15ae80 0x7f998c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.883+0000 7f99925d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c15ed60 con 0x7f998c15aaa0 2023-10-16T14:32:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.883+0000 7f99925d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c0707e0 con 0x7f998c0ce390 2023-10-16T14:32:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.883+0000 7f99925d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c06b810 con 0x7f998c15ae80 2023-10-16T14:32:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f998c15aaa0 0x7f998c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43102/0 (socket says 172.21.15.29:43102) 2023-10-16T14:32:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998bfff640 1 -- 172.21.15.29:0/3380918112 learned_addr learned my addr 172.21.15.29:0/3380918112 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731941410 0 0) 0x7f998c15ed60 con 0x7f998c15aaa0 2023-10-16T14:32:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f9983fff640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c15ae80 0x7f998c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978003520 con 0x7f998c15aaa0 2023-10-16T14:32:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1786253574 0 0) 0x7f9978003520 con 0x7f998c15aaa0 2023-10-16T14:32:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c15ae80 msgr2=0x7f998c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c15ae80 0x7f998c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 >> v1:172.21.15.170:6789/0 conn(0x7f998c0ce390 legacy=0x7f998c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c15ef90 con 0x7f998c15aaa0 2023-10-16T14:32:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.884+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f997c001670 con 0x7f998c15aaa0 2023-10-16T14:32:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 >> v1:172.21.15.29:6789/0 conn(0x7f998c15aaa0 legacy=0x7f998c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f9978003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99780047c0 0x7f9978004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9978005180 0x7f997800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c06b810 con 0x7f99780039a0 2023-10-16T14:32:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997800c020 con 0x7f99780047c0 2023-10-16T14:32:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997800c1e0 con 0x7f9978005180 2023-10-16T14:32:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f9983fff640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c15ae80 0x7f998c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998bfff640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f9978003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f9990b6d640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9978005180 0x7f997800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998bfff640 1 -- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9978005180 msgr2=0x7f997800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998bfff640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9978005180 0x7f997800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998bfff640 1 -- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99780047c0 msgr2=0x7f9978004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998bfff640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99780047c0 0x7f9978004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.885+0000 7f998bfff640 1 -- 172.21.15.29:0/3380918112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978003520 con 0x7f99780039a0 2023-10-16T14:32:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f998bfff640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f9978003dc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c240 tx=0x7f99780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3d73ca3a04dc34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c005ad0 con 0x7f99780039a0 2023-10-16T14:32:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c005c30 con 0x7f99780039a0 2023-10-16T14:32:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f998b7fe640 1 -- 172.21.15.29:0/3380918112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c02c8b0 con 0x7f99780039a0 2023-10-16T14:32:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f99925d0640 1 -- 172.21.15.29:0/3380918112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 msgr2=0x7f9978003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f99925d0640 1 --2- 172.21.15.29:0/3380918112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f9978003dc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c240 tx=0x7f99780057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.887+0000 7f9990b6d640 1 -- 172.21.15.29:0/3380918112 reap_dead start 2023-10-16T14:32:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.888+0000 7f99925d0640 1 -- 172.21.15.29:0/3380918112 shutdown_connections 2023-10-16T14:32:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.888+0000 7f99925d0640 1 -- 172.21.15.29:0/3380918112 >> 172.21.15.29:0/3380918112 conn(0x7f998c0cb410 msgr2=0x7f998c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.888+0000 7f99925d0640 1 -- 172.21.15.29:0/3380918112 shutdown_connections 2023-10-16T14:32:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.888+0000 7f99925d0640 1 -- 172.21.15.29:0/3380918112 wait complete. 2023-10-16T14:32:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.889+0000 7f99925d0640 1 Processor -- start 2023-10-16T14:32:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.890+0000 7f99925d0640 1 -- start start 2023-10-16T14:32:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f99925d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f998c15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f99925d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f998c0c3180 0x7f998c15d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f99925d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c0af680 0x7f998c1ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f99925d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0c24c0 con 0x7f99780039a0 2023-10-16T14:32:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f99925d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0c3600 con 0x7f998c0c3180 2023-10-16T14:32:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f99925d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c15def0 con 0x7f998c0af680 2023-10-16T14:32:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f998c15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f998c15b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33548/0 (socket says 172.21.15.29:33548) 2023-10-16T14:32:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 -- 172.21.15.29:0/1259643719 learned_addr learned my addr 172.21.15.29:0/1259643719 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 -- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c0af680 msgr2=0x7f998c1ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c0af680 0x7f998c1ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 -- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f998c0c3180 msgr2=0x7f998c15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:57.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f998c0c3180 0x7f998c15d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:57.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.891+0000 7f998bfff640 1 -- 172.21.15.29:0/1259643719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c15e190 con 0x7f99780039a0 2023-10-16T14:32:57.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.892+0000 7f998bfff640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f998c15b2e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f997c02c450 tx=0x7f997c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:57.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.892+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c039070 con 0x7f99780039a0 2023-10-16T14:32:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.892+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c02dd50 con 0x7f99780039a0 2023-10-16T14:32:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.892+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c03c040 con 0x7f99780039a0 2023-10-16T14:32:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.892+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c0c26a0 con 0x7f99780039a0 2023-10-16T14:32:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.892+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c0c2c60 con 0x7f99780039a0 2023-10-16T14:32:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.893+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c15f530 con 0x7f99780039a0 2023-10-16T14:32:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.894+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f997c032070 con 0x7f99780039a0 2023-10-16T14:32:57.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.895+0000 7f99897fa640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f996c06afc0 0x7f996c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:57.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.895+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f997c0aae40 con 0x7f99780039a0 2023-10-16T14:32:57.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.896+0000 7f9983fff640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f996c06afc0 0x7f996c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:57.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.897+0000 7f9983fff640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f996c06afc0 0x7f996c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9974006fd0 tx=0x7f9974008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:57.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:57.900+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f998c15f530 con 0x7f99780039a0 2023-10-16T14:32:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.162+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f998c15f530 con 0x7f99780039a0 2023-10-16T14:32:58.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.703+0000 7f99897fa640 1 -- 172.21.15.29:0/1259643719 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v610) v1 ==== 169+0+0 (secure 0 0 0) 0x7f998c15f530 con 0x7f99780039a0 2023-10-16T14:32:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f996c06afc0 msgr2=0x7f996c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f996c06afc0 0x7f996c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9974006fd0 tx=0x7f9974008040 comp rx=0 tx=0).stop 2023-10-16T14:32:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 msgr2=0x7f998c15b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f998c15b2e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f997c02c450 tx=0x7f997c001a00 comp rx=0 tx=0).stop 2023-10-16T14:32:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 shutdown_connections 2023-10-16T14:32:58.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f998c0af680 0x7f998c1ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f998c0c3180 0x7f998c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f996c06afc0 0x7f996c06d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.706+0000 7f99925d0640 1 --2- 172.21.15.29:0/1259643719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99780039a0 0x7f998c15b2e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.707+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 >> 172.21.15.29:0/1259643719 conn(0x7f998c0cb410 msgr2=0x7f998c159560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:58.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.707+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 shutdown_connections 2023-10-16T14:32:58.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.707+0000 7f99925d0640 1 -- 172.21.15.29:0/1259643719 wait complete. 2023-10-16T14:32:58.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:32:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.968+0000 7f688e3e9640 1 Processor -- start 2023-10-16T14:32:58.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.968+0000 7f688e3e9640 1 -- start start 2023-10-16T14:32:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.969+0000 7f688e3e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f688815cd70 0x7f68880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.969+0000 7f688e3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f68880b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.969+0000 7f688e3e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688815d150 con 0x7f68880b8480 2023-10-16T14:32:58.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.969+0000 7f688e3e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68880d1180 con 0x7f688815cd70 2023-10-16T14:32:58.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.969+0000 7f688e3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68880d1460 con 0x7f68880cded0 2023-10-16T14:32:58.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.970+0000 7f688cbe6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f68880b8480 0x7f68880cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43112/0 (socket says 172.21.15.29:43112) 2023-10-16T14:32:58.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.970+0000 7f688cbe6640 1 -- 172.21.15.29:0/1408144016 learned_addr learned my addr 172.21.15.29:0/1408144016 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:58.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.970+0000 7f688d3e7640 1 --2- 172.21.15.29:0/1408144016 >> v2:172.21.15.170:3300/0 conn(0x7f688815cd70 0x7f68880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.970+0000 7f688dbe8640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f68880b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.970+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929537061 0 0) 0x7f688815d150 con 0x7f68880b8480 2023-10-16T14:32:58.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.970+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6878003520 con 0x7f68880b8480 2023-10-16T14:32:58.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233338331 0 0) 0x7f6878003520 con 0x7f68880b8480 2023-10-16T14:32:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 msgr2=0x7f68880b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f68880b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 >> v2:172.21.15.170:3300/0 conn(0x7f688815cd70 msgr2=0x7f68880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 --2- 172.21.15.29:0/1408144016 >> v2:172.21.15.170:3300/0 conn(0x7f688815cd70 0x7f68880d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68880d1740 con 0x7f68880b8480 2023-10-16T14:32:58.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6874001690 con 0x7f68880b8480 2023-10-16T14:32:58.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f688dbe8640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f68880b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:32:58.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 >> v1:172.21.15.29:6789/0 conn(0x7f68880b8480 legacy=0x7f68880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:58.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f6878003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878003eb0 0x7f68780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6878004d00 0x7f6878005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68880d1180 con 0x7f6878004d00 2023-10-16T14:32:58.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68880d1460 con 0x7f6878003eb0 2023-10-16T14:32:58.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687800b9b0 con 0x7f68880cded0 2023-10-16T14:32:58.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.971+0000 7f688dbe8640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6878004d00 0x7f6878005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688d3e7640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f6878003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688cbe6640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878003eb0 0x7f68780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688dbe8640 1 -- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 msgr2=0x7f6878003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:58.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688dbe8640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880cded0 0x7f6878003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688dbe8640 1 -- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878003eb0 msgr2=0x7f68780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:58.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688dbe8640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878003eb0 0x7f68780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.972+0000 7f688dbe8640 1 -- 172.21.15.29:0/1408144016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878003520 con 0x7f6878004d00 2023-10-16T14:32:58.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.973+0000 7f688dbe8640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6878004d00 0x7f6878005180 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f68880b8930 tx=0x7f68780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ad0f6730ebd92a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6880004030 con 0x7f6878004d00 2023-10-16T14:32:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6880004190 con 0x7f6878004d00 2023-10-16T14:32:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f6887fff640 1 -- 172.21.15.29:0/1408144016 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6880003610 con 0x7f6878004d00 2023-10-16T14:32:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688e3e9640 1 -- 172.21.15.29:0/1408144016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6878004d00 msgr2=0x7f6878005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688e3e9640 1 --2- 172.21.15.29:0/1408144016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6878004d00 0x7f6878005180 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f68880b8930 tx=0x7f68780057f0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688dbe8640 1 -- 172.21.15.29:0/1408144016 reap_dead start 2023-10-16T14:32:58.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688e3e9640 1 -- 172.21.15.29:0/1408144016 shutdown_connections 2023-10-16T14:32:58.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688e3e9640 1 -- 172.21.15.29:0/1408144016 >> 172.21.15.29:0/1408144016 conn(0x7f68880cf570 msgr2=0x7f68880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:58.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688e3e9640 1 -- 172.21.15.29:0/1408144016 shutdown_connections 2023-10-16T14:32:58.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.974+0000 7f688e3e9640 1 -- 172.21.15.29:0/1408144016 wait complete. 2023-10-16T14:32:58.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.976+0000 7f688e3e9640 1 Processor -- start 2023-10-16T14:32:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.976+0000 7f688e3e9640 1 -- start start 2023-10-16T14:32:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.977+0000 7f688e3e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878004d00 0x7f68880b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.977+0000 7f688e3e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 0x7f68880d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.977+0000 7f688e3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880d2fc0 0x7f68880c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688e3e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68881f2400 con 0x7f688815aee0 2023-10-16T14:32:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688e3e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68880ca5a0 con 0x7f6878004d00 2023-10-16T14:32:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688e3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68880d3440 con 0x7f68880d2fc0 2023-10-16T14:32:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 0x7f68880d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 0x7f68880d2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33574/0 (socket says 172.21.15.29:33574) 2023-10-16T14:32:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 -- 172.21.15.29:0/2852316802 learned_addr learned my addr 172.21.15.29:0/2852316802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688d3e7640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878004d00 0x7f68880b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 -- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880d2fc0 msgr2=0x7f68880c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:58.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880d2fc0 0x7f68880c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 -- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878004d00 msgr2=0x7f68880b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:58.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878004d00 0x7f68880b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:58.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.978+0000 7f688cbe6640 1 -- 172.21.15.29:0/2852316802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68880c77d0 con 0x7f688815aee0 2023-10-16T14:32:58.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.979+0000 7f688cbe6640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 0x7f68880d2a80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6874001ce0 tx=0x7f687400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:58.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.979+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400c7f0 con 0x7f688815aee0 2023-10-16T14:32:58.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.979+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400c970 con 0x7f688815aee0 2023-10-16T14:32:58.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.979+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400cd20 con 0x7f688815aee0 2023-10-16T14:32:58.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.979+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68880ca7b0 con 0x7f688815aee0 2023-10-16T14:32:58.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.979+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68880c9dd0 con 0x7f688815aee0 2023-10-16T14:32:58.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.980+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68880b18c0 con 0x7f688815aee0 2023-10-16T14:32:58.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.982+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6874013420 con 0x7f688815aee0 2023-10-16T14:32:58.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.983+0000 7f6885ffb640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f686c06afc0 0x7f686c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:58.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.983+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f68740893f0 con 0x7f688815aee0 2023-10-16T14:32:58.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.983+0000 7f688d3e7640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f686c06afc0 0x7f686c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:58.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.985+0000 7f688d3e7640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f686c06afc0 0x7f686c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68880d0cf0 tx=0x7f687c00b040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:58.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:58.987+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f68740543e0 con 0x7f688815aee0 2023-10-16T14:32:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.182+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f68880c7960 con 0x7f688815aee0 2023-10-16T14:32:59.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.189+0000 7f6885ffb640 1 -- 172.21.15.29:0/2852316802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6874001650 con 0x7f688815aee0 2023-10-16T14:32:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.192+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f686c06afc0 msgr2=0x7f686c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.192+0000 7f688e3e9640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f686c06afc0 0x7f686c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68880d0cf0 tx=0x7f687c00b040 comp rx=0 tx=0).stop 2023-10-16T14:32:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.192+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 msgr2=0x7f68880d2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.192+0000 7f688e3e9640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 0x7f68880d2a80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6874001ce0 tx=0x7f687400ecd0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 shutdown_connections 2023-10-16T14:32:59.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f686c06afc0 0x7f686c06d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f688815aee0 0x7f68880d2a80 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f68880d2fc0 0x7f68880c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 --2- 172.21.15.29:0/2852316802 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6878004d00 0x7f68880b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 >> 172.21.15.29:0/2852316802 conn(0x7f68880cf570 msgr2=0x7f68880b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:59.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 shutdown_connections 2023-10-16T14:32:59.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.193+0000 7f688e3e9640 1 -- 172.21.15.29:0/2852316802 wait complete. 2023-10-16T14:32:59.216 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:32:59.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:59.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:32:59.458 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:32:59.459 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:59.459 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:59.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:32:59.524 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:59.524 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:59.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:59.584 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:32:59.585 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:59.647 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:32:59.647 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:32:59.648 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:32:59.707 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:32:59.708 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:32:59.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:32:59.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.935+0000 7ffba3e9b640 1 Processor -- start 2023-10-16T14:32:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.935+0000 7ffba3e9b640 1 -- start start 2023-10-16T14:32:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba3e9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffb9c15aaa0 0x7ffb9c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba3e9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffb9c0ce390 0x7ffb9c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba3e9b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c15ed60 con 0x7ffb9c15ae80 2023-10-16T14:32:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba3e9b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c06b810 con 0x7ffb9c0ce390 2023-10-16T14:32:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba3e9b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c0707e0 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba2438640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffb9c0ce390 0x7ffb9c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba1436640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffb9c15ae80 0x7ffb9c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43114/0 (socket says 172.21.15.29:43114) 2023-10-16T14:32:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.937+0000 7ffba1436640 1 -- 172.21.15.29:0/598656974 learned_addr learned my addr 172.21.15.29:0/598656974 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:32:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.938+0000 7ffba1c37640 1 --2- 172.21.15.29:0/598656974 >> v2:172.21.15.170:3300/0 conn(0x7ffb9c15aaa0 0x7ffb9c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.938+0000 7ffba2438640 1 -- 172.21.15.29:0/598656974 >> v2:172.21.15.170:3300/0 conn(0x7ffb9c15aaa0 msgr2=0x7ffb9c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:32:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.938+0000 7ffba2438640 1 --2- 172.21.15.29:0/598656974 >> v2:172.21.15.170:3300/0 conn(0x7ffb9c15aaa0 0x7ffb9c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.938+0000 7ffba2438640 1 -- 172.21.15.29:0/598656974 >> v1:172.21.15.29:6789/0 conn(0x7ffb9c15ae80 legacy=0x7ffb9c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.938+0000 7ffba2438640 1 -- 172.21.15.29:0/598656974 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c15ef90 con 0x7ffb9c0ce390 2023-10-16T14:32:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.938+0000 7ffba0c35640 1 -- 172.21.15.29:0/598656974 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377026460 0 0) 0x7ffb9c15ed60 con 0x7ffb9c15ae80 2023-10-16T14:32:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.939+0000 7ffba2438640 1 --2- 172.21.15.29:0/598656974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c070f50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0cd990 tx=0x7ffb9402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cab6305bd3879dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.939+0000 7ffba0c35640 1 -- 172.21.15.29:0/598656974 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9402cc70 con 0x7ffb9c0ce390 2023-10-16T14:32:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba0c35640 1 -- 172.21.15.29:0/598656974 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9402cdd0 con 0x7ffb9c0ce390 2023-10-16T14:32:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba0c35640 1 -- 172.21.15.29:0/598656974 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94034660 con 0x7ffb9c0ce390 2023-10-16T14:32:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 -- 172.21.15.29:0/598656974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c0ce390 msgr2=0x7ffb9c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/598656974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c070f50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c0cd990 tx=0x7ffb9402bec0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 -- 172.21.15.29:0/598656974 shutdown_connections 2023-10-16T14:32:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/598656974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c070f50 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/598656974 >> v2:172.21.15.170:3300/0 conn(0x7ffb9c15aaa0 0x7ffb9c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 -- 172.21.15.29:0/598656974 >> 172.21.15.29:0/598656974 conn(0x7ffb9c0cb410 msgr2=0x7ffb9c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:32:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 -- 172.21.15.29:0/598656974 shutdown_connections 2023-10-16T14:32:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.940+0000 7ffba3e9b640 1 -- 172.21.15.29:0/598656974 wait complete. 2023-10-16T14:32:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.942+0000 7ffba3e9b640 1 Processor -- start 2023-10-16T14:32:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.942+0000 7ffba3e9b640 1 -- start start 2023-10-16T14:32:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba3e9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba3e9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 0x7ffb9c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba3e9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb9c15ae80 0x7ffb9c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba3e9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c1f5c80 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba3e9b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0c86f0 con 0x7ffb9c0ce390 2023-10-16T14:32:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba3e9b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c1f5ed0 con 0x7ffb9c15ae80 2023-10-16T14:32:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba1436640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 0x7ffb9c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba1436640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 0x7ffb9c0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33592/0 (socket says 172.21.15.29:33592) 2023-10-16T14:32:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.943+0000 7ffba1436640 1 -- 172.21.15.29:0/3007773510 learned_addr learned my addr 172.21.15.29:0/3007773510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:32:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba2438640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb9c15ae80 0x7ffb9c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1c37640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1436640 1 -- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb9c15ae80 msgr2=0x7ffb9c0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1436640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb9c15ae80 0x7ffb9c0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1436640 1 -- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb9c0ce390 msgr2=0x7ffb9c14d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:32:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1436640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c14d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:32:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1436640 1 -- 172.21.15.29:0/3007773510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94009a70 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1c37640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c14d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:32:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffba1436640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 0x7ffb9c0cc4b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00c410 tx=0x7ffb8c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.944+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c0074e0 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.945+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c007660 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.945+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c004a20 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.945+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c1f61b0 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.945+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c1f69b0 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.946+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c1f66d0 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.947+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffb8c004b80 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.948+0000 7ffb927fc640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ffb8006af90 0x7ffb8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:32:59.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.948+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ffb8c089550 con 0x7ffb9c15aaa0 2023-10-16T14:32:59.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.949+0000 7ffba1c37640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ffb8006af90 0x7ffb8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:32:59.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.950+0000 7ffba1c37640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ffb8006af90 0x7ffb8006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:32:59.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:32:59.953+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffb9c1f66d0 con 0x7ffb9c15aaa0 2023-10-16T14:33:00.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.188+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb9c1f66d0 con 0x7ffb9c15aaa0 2023-10-16T14:33:00.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.191+0000 7ffb927fc640 1 -- 172.21.15.29:0/3007773510 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3702 (secure 0 0 0) 0x7ffb9c1f66d0 con 0x7ffb9c15aaa0 2023-10-16T14:33:00.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:00.194 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/977368069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":977368069},{"type":"v1","addr":"172.21.15.170:6836","nonce":977368069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5572,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1771777540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1771777540},{"type":"v1","addr":"172.21.15.29:6834","nonce":1771777540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2278231410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2278231410},{"type":"v1","addr":"172.21.15.29:6835","nonce":2278231410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5863,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1904101262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1904101262},{"type":"v1","addr":"172.21.15.170:6837","nonce":1904101262}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2023-10-16T14:33:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 35 2023-10-16T14:33:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ffb8006af90 msgr2=0x7ffb8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ffb8006af90 0x7ffb8006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).stop 2023-10-16T14:33:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 msgr2=0x7ffb9c0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 0x7ffb9c0cc4b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00c410 tx=0x7ffb8c00c8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 shutdown_connections 2023-10-16T14:33:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ffb8006af90 0x7ffb8006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb9c15aaa0 0x7ffb9c0cc4b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb9c15ae80 0x7ffb9c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 --2- 172.21.15.29:0/3007773510 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb9c0ce390 0x7ffb9c14d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 >> 172.21.15.29:0/3007773510 conn(0x7ffb9c0cb410 msgr2=0x7ffb9c06cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:00.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 shutdown_connections 2023-10-16T14:33:00.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.194+0000 7ffba3e9b640 1 -- 172.21.15.29:0/3007773510 wait complete. 2023-10-16T14:33:00.217 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:33:00.218 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:33:00.218 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:33:00.219 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:00.219 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:33:00.219 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:33:00.219 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:33:00.220 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:00.220 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:33:00.220 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:33:00.221 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:33:00.221 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:00.221 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:33:00.221 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:33:00.222 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:33:00.222 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:00.238 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:33:00.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:33:00.241 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:33:00.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:33:00.244 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:33:00.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:33:00.245 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:33:00.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:33:00.291 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:33:00.292 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:00.293 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:33:00.293 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:00.299 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:33:00.300 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:00.301 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:33:00.301 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:00.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:33:00.453 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:33:00.457 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:33:00.458 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:33:00.459 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:33:00.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.495+0000 7fc2f267c640 1 Processor -- start 2023-10-16T14:33:00.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.495+0000 7fc2f267c640 1 -- start start 2023-10-16T14:33:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2f267c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec06a4b0 0x7fc2ec0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2f267c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec0cb030 con 0x7fc2ec0c9ca0 2023-10-16T14:33:00.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2f267c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec0c91c0 con 0x7fc2ec15ad70 2023-10-16T14:33:00.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2f267c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0c94a0 con 0x7fc2ec06a4b0 2023-10-16T14:33:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2f0c19640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc2ec0c9ca0 0x7fc2ec0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43140/0 (socket says 172.21.15.29:43140) 2023-10-16T14:33:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2f0c19640 1 -- 172.21.15.29:0/1376184593 learned_addr learned my addr 172.21.15.29:0/1376184593 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2e3fff640 1 --2- 172.21.15.29:0/1376184593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec06a4b0 0x7fc2ec0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.497+0000 7fc2eb7fe640 1 -- 172.21.15.29:0/1376184593 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661505346 0 0) 0x7fc2ec0cb030 con 0x7fc2ec0c9ca0 2023-10-16T14:33:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.498+0000 7fc2eb7fe640 1 -- 172.21.15.29:0/1376184593 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d8003520 con 0x7fc2ec0c9ca0 2023-10-16T14:33:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.498+0000 7fc2e3fff640 1 -- 172.21.15.29:0/1376184593 >> v1:172.21.15.170:6789/0 conn(0x7fc2ec15ad70 legacy=0x7fc2ec069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.498+0000 7fc2eb7fe640 1 -- 172.21.15.29:0/1376184593 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926343244 0 0) 0x7fc2ec0c91c0 con 0x7fc2ec15ad70 2023-10-16T14:33:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.498+0000 7fc2e3fff640 1 -- 172.21.15.29:0/1376184593 >> v1:172.21.15.29:6789/0 conn(0x7fc2ec0c9ca0 legacy=0x7fc2ec0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.498+0000 7fc2e3fff640 1 -- 172.21.15.29:0/1376184593 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0c9780 con 0x7fc2ec06a4b0 2023-10-16T14:33:00.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.499+0000 7fc2e3fff640 1 --2- 172.21.15.29:0/1376184593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec06a4b0 0x7fc2ec0ca3b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40097d0 tx=0x7fc2d402b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca18f56530f0e4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:00.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2eb7fe640 1 -- 172.21.15.29:0/1376184593 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4001970 con 0x7fc2ec06a4b0 2023-10-16T14:33:00.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2eb7fe640 1 -- 172.21.15.29:0/1376184593 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d402e050 con 0x7fc2ec06a4b0 2023-10-16T14:33:00.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2eb7fe640 1 -- 172.21.15.29:0/1376184593 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4030730 con 0x7fc2ec06a4b0 2023-10-16T14:33:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2f267c640 1 -- 172.21.15.29:0/1376184593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec06a4b0 msgr2=0x7fc2ec0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2f267c640 1 --2- 172.21.15.29:0/1376184593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec06a4b0 0x7fc2ec0ca3b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40097d0 tx=0x7fc2d402b730 comp rx=0 tx=0).stop 2023-10-16T14:33:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2f267c640 1 -- 172.21.15.29:0/1376184593 shutdown_connections 2023-10-16T14:33:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2f267c640 1 --2- 172.21.15.29:0/1376184593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec06a4b0 0x7fc2ec0ca3b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2f267c640 1 -- 172.21.15.29:0/1376184593 >> 172.21.15.29:0/1376184593 conn(0x7fc2ec0ce9b0 msgr2=0x7fc2ec0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.500+0000 7fc2f267c640 1 -- 172.21.15.29:0/1376184593 shutdown_connections 2023-10-16T14:33:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.501+0000 7fc2f267c640 1 -- 172.21.15.29:0/1376184593 wait complete. 2023-10-16T14:33:00.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.502+0000 7fc2f267c640 1 Processor -- start 2023-10-16T14:33:00.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.502+0000 7fc2f267c640 1 -- start start 2023-10-16T14:33:00.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.503+0000 7fc2f267c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2ec06a4b0 0x7fc2ec0aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:00.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.503+0000 7fc2f267c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec0c9ca0 0x7fc2ec0c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.503+0000 7fc2f267c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 0x7fc2ec0aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.503+0000 7fc2f267c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0c81a0 con 0x7fc2ec15ad70 2023-10-16T14:33:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.503+0000 7fc2f267c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0aac60 con 0x7fc2ec06a4b0 2023-10-16T14:33:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.503+0000 7fc2f267c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0c7590 con 0x7fc2ec0c9ca0 2023-10-16T14:33:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 0x7fc2ec0aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:00.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 0x7fc2ec0aaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33630/0 (socket says 172.21.15.29:33630) 2023-10-16T14:33:00.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 -- 172.21.15.29:0/2084640718 learned_addr learned my addr 172.21.15.29:0/2084640718 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:00.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2e3fff640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec0c9ca0 0x7fc2ec0c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:00.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2ebfff640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2ec06a4b0 0x7fc2ec0aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:00.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 -- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec0c9ca0 msgr2=0x7fc2ec0c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec0c9ca0 0x7fc2ec0c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 -- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2ec06a4b0 msgr2=0x7fc2ec0aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2ec06a4b0 0x7fc2ec0aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.504+0000 7fc2f0c19640 1 -- 172.21.15.29:0/2084640718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4009400 con 0x7fc2ec15ad70 2023-10-16T14:33:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.505+0000 7fc2f0c19640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 0x7fc2ec0aaf00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4002230 tx=0x7fc2e400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:00.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.505+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e400aa20 con 0x7fc2ec15ad70 2023-10-16T14:33:00.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.505+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e400ab80 con 0x7fc2ec15ad70 2023-10-16T14:33:00.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.505+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4004e30 con 0x7fc2ec15ad70 2023-10-16T14:33:00.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.505+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec0c7870 con 0x7fc2ec15ad70 2023-10-16T14:33:00.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.505+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec0c7e30 con 0x7fc2ec15ad70 2023-10-16T14:33:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.506+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec0c8350 con 0x7fc2ec15ad70 2023-10-16T14:33:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.507+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc2e40070d0 con 0x7fc2ec15ad70 2023-10-16T14:33:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.509+0000 7fc2e97fa640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fc2cc06af90 0x7fc2cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.509+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc2e4088ee0 con 0x7fc2ec15ad70 2023-10-16T14:33:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.509+0000 7fc2ebfff640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fc2cc06af90 0x7fc2cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.510+0000 7fc2ebfff640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fc2cc06af90 0x7fc2cc06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc000f60 tx=0x7fc2dc009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.513+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2ec0c8350 con 0x7fc2ec15ad70 2023-10-16T14:33:00.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:00.714+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc2ec0c8350 con 0x7fc2ec15ad70 2023-10-16T14:33:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.210+0000 7fc2e97fa640 1 -- 172.21.15.29:0/2084640718 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v91) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc2ec0c8350 con 0x7fc2ec15ad70 2023-10-16T14:33:01.214 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:33:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.213+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fc2cc06af90 msgr2=0x7fc2cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.213+0000 7fc2f267c640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fc2cc06af90 0x7fc2cc06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc000f60 tx=0x7fc2dc009040 comp rx=0 tx=0).stop 2023-10-16T14:33:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.213+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 msgr2=0x7fc2ec0aaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.213+0000 7fc2f267c640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 0x7fc2ec0aaf00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4002230 tx=0x7fc2e400cf50 comp rx=0 tx=0).stop 2023-10-16T14:33:01.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 shutdown_connections 2023-10-16T14:33:01.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fc2cc06af90 0x7fc2cc06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2ec15ad70 0x7fc2ec0aaf00 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2ec0c9ca0 0x7fc2ec0c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 --2- 172.21.15.29:0/2084640718 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2ec06a4b0 0x7fc2ec0aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 >> 172.21.15.29:0/2084640718 conn(0x7fc2ec0ce9b0 msgr2=0x7fc2ec0af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 shutdown_connections 2023-10-16T14:33:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.214+0000 7fc2f267c640 1 -- 172.21.15.29:0/2084640718 wait complete. 2023-10-16T14:33:01.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:33:01.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.458+0000 7f0c47fff640 1 Processor -- start 2023-10-16T14:33:01.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.458+0000 7f0c47fff640 1 -- start start 2023-10-16T14:33:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.460+0000 7f0c47fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0c4806c040 0x7f0c4806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.460+0000 7f0c47fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 0x7f0c4815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.460+0000 7f0c47fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4806c960 con 0x7f0c480ad970 2023-10-16T14:33:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.460+0000 7f0c47fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4815b9d0 con 0x7f0c4806c040 2023-10-16T14:33:01.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.460+0000 7f0c47fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4815bcb0 con 0x7f0c480cda80 2023-10-16T14:33:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.460+0000 7f0c467fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0c480ad970 0x7f0c4815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43156/0 (socket says 172.21.15.29:43156) 2023-10-16T14:33:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.461+0000 7f0c467fc640 1 -- 172.21.15.29:0/2937504302 learned_addr learned my addr 172.21.15.29:0/2937504302 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.461+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 0x7f0c4815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.461+0000 7f0c46ffd640 1 --2- 172.21.15.29:0/2937504302 >> v2:172.21.15.170:3300/0 conn(0x7f0c4806c040 0x7f0c4806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.461+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689699262 0 0) 0x7f0c4806c960 con 0x7f0c480ad970 2023-10-16T14:33:01.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.461+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c30003520 con 0x7f0c480ad970 2023-10-16T14:33:01.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002100411 0 0) 0x7f0c30003520 con 0x7f0c480ad970 2023-10-16T14:33:01.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 msgr2=0x7f0c4815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:01.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 0x7f0c4815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 >> v2:172.21.15.170:3300/0 conn(0x7f0c4806c040 msgr2=0x7f0c4806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:01.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 --2- 172.21.15.29:0/2937504302 >> v2:172.21.15.170:3300/0 conn(0x7f0c4806c040 0x7f0c4806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4815bf90 con 0x7f0c480ad970 2023-10-16T14:33:01.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0c3c001670 con 0x7f0c480ad970 2023-10-16T14:33:01.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 >> v1:172.21.15.29:6789/0 conn(0x7f0c480ad970 legacy=0x7f0c4815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 0x7f0c30003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c30003f40 0x7f0c300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4815b9d0 con 0x7f0c30004d00 2023-10-16T14:33:01.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4815bcb0 con 0x7f0c30003f40 2023-10-16T14:33:01.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3000b980 con 0x7f0c480cda80 2023-10-16T14:33:01.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c467fc640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c30003f40 0x7f0c300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.462+0000 7f0c46ffd640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 0x7f0c30003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.463+0000 7f0c477fe640 1 -- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 msgr2=0x7f0c30003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:01.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.463+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480cda80 0x7f0c30003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.463+0000 7f0c477fe640 1 -- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c30003f40 msgr2=0x7f0c300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:01.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.463+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c30003f40 0x7f0c300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.463+0000 7f0c477fe640 1 -- 172.21.15.29:0/2937504302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30003520 con 0x7f0c30004d00 2023-10-16T14:33:01.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.464+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c30005180 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009af0 tx=0x7f0c300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b9752dcc942d60 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:01.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.464+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4002c6b0 con 0x7f0c30004d00 2023-10-16T14:33:01.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.464+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4002c810 con 0x7f0c30004d00 2023-10-16T14:33:01.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.464+0000 7f0c45ffb640 1 -- 172.21.15.29:0/2937504302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4002cba0 con 0x7f0c30004d00 2023-10-16T14:33:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c47fff640 1 -- 172.21.15.29:0/2937504302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 msgr2=0x7f0c30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2937504302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c30005180 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009af0 tx=0x7f0c300057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c477fe640 1 -- 172.21.15.29:0/2937504302 reap_dead start 2023-10-16T14:33:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c47fff640 1 -- 172.21.15.29:0/2937504302 shutdown_connections 2023-10-16T14:33:01.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c47fff640 1 -- 172.21.15.29:0/2937504302 >> 172.21.15.29:0/2937504302 conn(0x7f0c480ce6a0 msgr2=0x7f0c480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:01.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c47fff640 1 -- 172.21.15.29:0/2937504302 shutdown_connections 2023-10-16T14:33:01.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.465+0000 7f0c47fff640 1 -- 172.21.15.29:0/2937504302 wait complete. 2023-10-16T14:33:01.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.467+0000 7f0c47fff640 1 Processor -- start 2023-10-16T14:33:01.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.467+0000 7f0c47fff640 1 -- start start 2023-10-16T14:33:01.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c480c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c47fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480c2eb0 0x7f0c48147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c47fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c480ca680 0x7f0c480ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c47fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c480c3330 con 0x7f0c30004d00 2023-10-16T14:33:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c47fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4814ff40 con 0x7f0c480ca680 2023-10-16T14:33:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c47fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48147f80 con 0x7f0c480c2eb0 2023-10-16T14:33:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c46ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c480c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c46ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c480c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33650/0 (socket says 172.21.15.29:33650) 2023-10-16T14:33:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.468+0000 7f0c46ffd640 1 -- 172.21.15.29:0/2685101120 learned_addr learned my addr 172.21.15.29:0/2685101120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c480ca680 0x7f0c480ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c46ffd640 1 -- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480c2eb0 msgr2=0x7f0c48147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c467fc640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480c2eb0 0x7f0c48147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c46ffd640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480c2eb0 0x7f0c48147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c46ffd640 1 -- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c480ca680 msgr2=0x7f0c480ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c46ffd640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c480ca680 0x7f0c480ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c46ffd640 1 -- 172.21.15.29:0/2685101120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c480c3ae0 con 0x7f0c30004d00 2023-10-16T14:33:01.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c477fe640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c480ca680 0x7f0c480ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:01.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c46ffd640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c480c21d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0c38007f60 tx=0x7f0c38007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:01.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3800e030 con 0x7f0c30004d00 2023-10-16T14:33:01.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c38016470 con 0x7f0c30004d00 2023-10-16T14:33:01.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38016820 con 0x7f0c30004d00 2023-10-16T14:33:01.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.469+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c480c3520 con 0x7f0c30004d00 2023-10-16T14:33:01.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.470+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c480c9e80 con 0x7f0c30004d00 2023-10-16T14:33:01.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.471+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4806b330 con 0x7f0c30004d00 2023-10-16T14:33:01.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.472+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c38016b90 con 0x7f0c30004d00 2023-10-16T14:33:01.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.472+0000 7f0c2f7fe640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0c3406af90 0x7f0c3406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:01.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.473+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0c3800f070 con 0x7f0c30004d00 2023-10-16T14:33:01.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.473+0000 7f0c467fc640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0c3406af90 0x7f0c3406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:01.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.474+0000 7f0c467fc640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0c3406af90 0x7f0c3406d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c001eb0 tx=0x7f0c3c001b80 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:01.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.477+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c38055980 con 0x7f0c30004d00 2023-10-16T14:33:01.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.741+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0c4804e9b0 con 0x7f0c30004d00 2023-10-16T14:33:01.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.742+0000 7f0c2f7fe640 1 -- 172.21.15.29:0/2685101120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0c38058fa0 con 0x7f0c30004d00 2023-10-16T14:33:01.745 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:01.745 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:33:01.745 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:33:01.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.746 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:33:01.746 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:33:01.746 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:01.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:01.747 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.747 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.748 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:33:01.748 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:33:01.748 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.748 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.749 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:01.749 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:01.749 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.750 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:33:01.750 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:33:01.750 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.750 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:01.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:01.751 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.752 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:33:01.752 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:33:01.752 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.752 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.752 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:01.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:01.753 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.753 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.754 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:33:01.754 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:33:01.754 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.754 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.754 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.755 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.755 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:33:01.755 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:33:01.756 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.756 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.757 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:33:01.757 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:33:01.757 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.758 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.758 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.759 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:33:01.759 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:33:01.759 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.759 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.759 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.760 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.760 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.760 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:33:01.760 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:33:01.761 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.761 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.761 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.761 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.761 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.762 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:33:01.762 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:33:01.762 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.763 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.763 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.763 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.764 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:33:01.764 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:33:01.764 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.764 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.764 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.765 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.765 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.765 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:33:01.766 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:33:01.766 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.766 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:01.766 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:01.766 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.767 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.767 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:33:01.767 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:33:01.767 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.768 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.768 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:33:01.768 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:33:01.768 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.769 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:33:01.769 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:33:01.769 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.769 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:33:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:33:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:33:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.771 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:33:01.771 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:33:01.771 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:33:01.772 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:33:01.772 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:33:01.772 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.773 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:33:01.773 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:33:01.773 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:01.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:33:01.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:33:01.774 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:33:01.774 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.774 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:33:01.775 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:33:01.775 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:33:01.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:01.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:01.776 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.776 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:33:01.777 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:33:01.777 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:33:01.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:33:01.777 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:01.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:01.778 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:33:01.778 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:33:01.779 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:01.779 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:33:01.779 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:33:01.779 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:01.779 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:01.780 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:33:01.780 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:33:01.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.744+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0c3406af90 msgr2=0x7f0c3406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.744+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0c3406af90 0x7f0c3406d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c001eb0 tx=0x7f0c3c001b80 comp rx=0 tx=0).stop 2023-10-16T14:33:01.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.744+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 msgr2=0x7f0c480c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:01.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.744+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c480c21d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0c38007f60 tx=0x7f0c38007f90 comp rx=0 tx=0).stop 2023-10-16T14:33:01.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 shutdown_connections 2023-10-16T14:33:01.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c480ca680 0x7f0c480ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c480c2eb0 0x7f0c48147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0c3406af90 0x7f0c3406d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 --2- 172.21.15.29:0/2685101120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c30004d00 0x7f0c480c21d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 >> 172.21.15.29:0/2685101120 conn(0x7f0c480ce6a0 msgr2=0x7f0c4806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 shutdown_connections 2023-10-16T14:33:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:01.745+0000 7f0c47fff640 1 -- 172.21.15.29:0/2685101120 wait complete. 2023-10-16T14:33:01.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:02.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.018+0000 7fa16e840640 1 Processor -- start 2023-10-16T14:33:02.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.018+0000 7fa16e840640 1 -- start start 2023-10-16T14:33:02.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.019+0000 7fa16e840640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa16815b320 0x7fa1680cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.019+0000 7fa16e840640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16815b9d0 con 0x7fa16806c040 2023-10-16T14:33:02.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.019+0000 7fa16e840640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16815bcb0 con 0x7fa1680ad970 2023-10-16T14:33:02.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.019+0000 7fa16e840640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16815b7d0 con 0x7fa16815b320 2023-10-16T14:33:02.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.020+0000 7fa16d83e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa16806c040 0x7fa16806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43170/0 (socket says 172.21.15.29:43170) 2023-10-16T14:33:02.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.020+0000 7fa16d83e640 1 -- 172.21.15.29:0/3008269147 learned_addr learned my addr 172.21.15.29:0/3008269147 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:02.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.020+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652813869 0 0) 0x7fa16815b9d0 con 0x7fa16806c040 2023-10-16T14:33:02.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.020+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa150003520 con 0x7fa16806c040 2023-10-16T14:33:02.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.020+0000 7fa16e03f640 1 --2- 172.21.15.29:0/3008269147 >> v2:172.21.15.170:3300/0 conn(0x7fa16815b320 0x7fa1680cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539332970 0 0) 0x7fa150003520 con 0x7fa16806c040 2023-10-16T14:33:02.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 >> v2:172.21.15.170:3300/0 conn(0x7fa16815b320 msgr2=0x7fa1680cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:02.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 --2- 172.21.15.29:0/3008269147 >> v2:172.21.15.170:3300/0 conn(0x7fa16815b320 0x7fa1680cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 >> v1:172.21.15.170:6789/0 conn(0x7fa1680ad970 legacy=0x7fa1680cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:02.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16815bf90 con 0x7fa16806c040 2023-10-16T14:33:02.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa158001670 con 0x7fa16806c040 2023-10-16T14:33:02.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 >> v1:172.21.15.29:6789/0 conn(0x7fa16806c040 legacy=0x7fa16806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500039f0 0x7fa150003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1500047c0 0x7fa150004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1500051f0 0x7fa15000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16815b7d0 con 0x7fa1500047c0 2023-10-16T14:33:02.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15000c150 con 0x7fa1500051f0 2023-10-16T14:33:02.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15000c310 con 0x7fa1500039f0 2023-10-16T14:33:02.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16d03d640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1500047c0 0x7fa150004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16d03d640 1 -- 172.21.15.29:0/3008269147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500039f0 msgr2=0x7fa150003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:02.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16d03d640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500039f0 0x7fa150003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16d03d640 1 -- 172.21.15.29:0/3008269147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1500051f0 msgr2=0x7fa15000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:02.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16d03d640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1500051f0 0x7fa15000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16d03d640 1 -- 172.21.15.29:0/3008269147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150003520 con 0x7fa1500047c0 2023-10-16T14:33:02.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.021+0000 7fa16e03f640 1 -- 172.21.15.29:0/3008269147 reap_dead start 2023-10-16T14:33:02.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.022+0000 7fa16d03d640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1500047c0 0x7fa150004c40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa15c004480 tx=0x7fa1500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93049975d9e175f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15c0305f0 con 0x7fa1500047c0 2023-10-16T14:33:02.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15c030750 con 0x7fa1500047c0 2023-10-16T14:33:02.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16c83c640 1 -- 172.21.15.29:0/3008269147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15c030b40 con 0x7fa1500047c0 2023-10-16T14:33:02.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 -- 172.21.15.29:0/3008269147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1500047c0 msgr2=0x7fa150004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1500047c0 0x7fa150004c40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa15c004480 tx=0x7fa1500057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 -- 172.21.15.29:0/3008269147 shutdown_connections 2023-10-16T14:33:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 --2- 172.21.15.29:0/3008269147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1500047c0 0x7fa150004c40 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 -- 172.21.15.29:0/3008269147 >> 172.21.15.29:0/3008269147 conn(0x7fa1680ce6a0 msgr2=0x7fa1680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 -- 172.21.15.29:0/3008269147 shutdown_connections 2023-10-16T14:33:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.023+0000 7fa16e840640 1 -- 172.21.15.29:0/3008269147 wait complete. 2023-10-16T14:33:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.025+0000 7fa16e840640 1 Processor -- start 2023-10-16T14:33:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.025+0000 7fa16e840640 1 -- start start 2023-10-16T14:33:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.026+0000 7fa16e840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500047c0 0x7fa1680ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.026+0000 7fa16e840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1680c8cf0 0x7fa168154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.026+0000 7fa16e840640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 0x7fa1680acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.026+0000 7fa16e840640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680ad280 con 0x7fa1680ac890 2023-10-16T14:33:02.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.026+0000 7fa16e840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1680c9170 con 0x7fa1680c8cf0 2023-10-16T14:33:02.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.026+0000 7fa16e840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1681551c0 con 0x7fa1500047c0 2023-10-16T14:33:02.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 0x7fa1680acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 0x7fa1680acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33662/0 (socket says 172.21.15.29:33662) 2023-10-16T14:33:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 -- 172.21.15.29:0/3724236896 learned_addr learned my addr 172.21.15.29:0/3724236896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16d03d640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1680c8cf0 0x7fa168154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16d83e640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500047c0 0x7fa1680ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 -- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500047c0 msgr2=0x7fa1680ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500047c0 0x7fa1680ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 -- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1680c8cf0 msgr2=0x7fa168154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1680c8cf0 0x7fa168154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 -- 172.21.15.29:0/3724236896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1680c93f0 con 0x7fa1680ac890 2023-10-16T14:33:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16d03d640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1680c8cf0 0x7fa168154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16d83e640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500047c0 0x7fa1680ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:02.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.027+0000 7fa16e03f640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 0x7fa1680acd40 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa16815b700 tx=0x7fa16000d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.028+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160004240 con 0x7fa1680ac890 2023-10-16T14:33:02.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.028+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1600043a0 con 0x7fa1680ac890 2023-10-16T14:33:02.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.028+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160003850 con 0x7fa1680ac890 2023-10-16T14:33:02.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.028+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1680c96d0 con 0x7fa1680ac890 2023-10-16T14:33:02.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.028+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1680c9c90 con 0x7fa1680ac890 2023-10-16T14:33:02.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.029+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16806b330 con 0x7fa1680ac890 2023-10-16T14:33:02.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.030+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa1600039b0 con 0x7fa1680ac890 2023-10-16T14:33:02.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.031+0000 7fa1667fc640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa15406af90 0x7fa15406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.031+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa160089080 con 0x7fa1680ac890 2023-10-16T14:33:02.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.032+0000 7fa16d83e640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa15406af90 0x7fa15406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.033+0000 7fa16d83e640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa15406af90 0x7fa15406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa168063ab0 tx=0x7fa158009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.036+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa160054370 con 0x7fa1680ac890 2023-10-16T14:33:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.234+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa16804e9b0 con 0x7fa1680ac890 2023-10-16T14:33:02.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.236+0000 7fa1667fc640 1 -- 172.21.15.29:0/3724236896 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fa160057990 con 0x7fa1680ac890 2023-10-16T14:33:02.239 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:02.239 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":93,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:01.222378+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:02.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa15406af90 msgr2=0x7fa15406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa15406af90 0x7fa15406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa168063ab0 tx=0x7fa158009040 comp rx=0 tx=0).stop 2023-10-16T14:33:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 msgr2=0x7fa1680acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 0x7fa1680acd40 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa16815b700 tx=0x7fa16000d8a0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 shutdown_connections 2023-10-16T14:33:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fa15406af90 0x7fa15406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1680ac890 0x7fa1680acd40 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1680c8cf0 0x7fa168154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 --2- 172.21.15.29:0/3724236896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1500047c0 0x7fa1680ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 >> 172.21.15.29:0/3724236896 conn(0x7fa1680ce6a0 msgr2=0x7fa1680c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.239+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 shutdown_connections 2023-10-16T14:33:02.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.240+0000 7fa16e840640 1 -- 172.21.15.29:0/3724236896 wait complete. 2023-10-16T14:33:02.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:33:02.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:33:02.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.492+0000 7fb84759e640 1 Processor -- start 2023-10-16T14:33:02.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.492+0000 7fb84759e640 1 -- start start 2023-10-16T14:33:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.493+0000 7fb84759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb848070f30 0x7fb84806aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.493+0000 7fb84759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb84806b3e0 0x7fb8480c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.493+0000 7fb84759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480cd260 con 0x7fb848159ad0 2023-10-16T14:33:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.493+0000 7fb84759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84815de30 con 0x7fb848070f30 2023-10-16T14:33:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.493+0000 7fb84759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84815e110 con 0x7fb84806b3e0 2023-10-16T14:33:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb84659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb848159ad0 0x7fb84815d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43180/0 (socket says 172.21.15.29:43180) 2023-10-16T14:33:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb84659c640 1 -- 172.21.15.29:0/2564528307 learned_addr learned my addr 172.21.15.29:0/2564528307 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb845d9b640 1 --2- 172.21.15.29:0/2564528307 >> v2:172.21.15.29:3300/0 conn(0x7fb848070f30 0x7fb84806aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb846d9d640 1 --2- 172.21.15.29:0/2564528307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb84806b3e0 0x7fb8480c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb84559a640 1 -- 172.21.15.29:0/2564528307 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20690261 0 0) 0x7fb8480cd260 con 0x7fb848159ad0 2023-10-16T14:33:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb84559a640 1 -- 172.21.15.29:0/2564528307 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb830003520 con 0x7fb848159ad0 2023-10-16T14:33:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb845d9b640 1 -- 172.21.15.29:0/2564528307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb84806b3e0 msgr2=0x7fb8480c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:02.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb845d9b640 1 --2- 172.21.15.29:0/2564528307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb84806b3e0 0x7fb8480c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb845d9b640 1 -- 172.21.15.29:0/2564528307 >> v1:172.21.15.29:6789/0 conn(0x7fb848159ad0 legacy=0x7fb84815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.494+0000 7fb845d9b640 1 -- 172.21.15.29:0/2564528307 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84815e3f0 con 0x7fb848070f30 2023-10-16T14:33:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.496+0000 7fb845d9b640 1 --2- 172.21.15.29:0/2564528307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb848070f30 0x7fb84806aea0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0097a0 tx=0x7fb83c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174b97866a63f9ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.496+0000 7fb84559a640 1 -- 172.21.15.29:0/2564528307 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c02d040 con 0x7fb848070f30 2023-10-16T14:33:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.496+0000 7fb84559a640 1 -- 172.21.15.29:0/2564528307 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83c031810 con 0x7fb848070f30 2023-10-16T14:33:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.496+0000 7fb84559a640 1 -- 172.21.15.29:0/2564528307 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c031b80 con 0x7fb848070f30 2023-10-16T14:33:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.496+0000 7fb84759e640 1 -- 172.21.15.29:0/2564528307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb848070f30 msgr2=0x7fb84806aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.496+0000 7fb84759e640 1 --2- 172.21.15.29:0/2564528307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb848070f30 0x7fb84806aea0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0097a0 tx=0x7fb83c02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.497+0000 7fb84759e640 1 -- 172.21.15.29:0/2564528307 shutdown_connections 2023-10-16T14:33:02.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.497+0000 7fb84759e640 1 --2- 172.21.15.29:0/2564528307 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb84806b3e0 0x7fb8480c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.497+0000 7fb84759e640 1 --2- 172.21.15.29:0/2564528307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb848070f30 0x7fb84806aea0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.497+0000 7fb84759e640 1 -- 172.21.15.29:0/2564528307 >> 172.21.15.29:0/2564528307 conn(0x7fb8480b0170 msgr2=0x7fb8480b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:02.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.497+0000 7fb84759e640 1 -- 172.21.15.29:0/2564528307 shutdown_connections 2023-10-16T14:33:02.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.497+0000 7fb84759e640 1 -- 172.21.15.29:0/2564528307 wait complete. 2023-10-16T14:33:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.498+0000 7fb84759e640 1 Processor -- start 2023-10-16T14:33:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.499+0000 7fb84759e640 1 -- start start 2023-10-16T14:33:02.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.499+0000 7fb84759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 0x7fb848153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.499+0000 7fb84759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb848070f30 0x7fb8480c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.499+0000 7fb84759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb848159ad0 0x7fb8480ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb848154360 con 0x7fb84806b3e0 2023-10-16T14:33:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb848158460 con 0x7fb848070f30 2023-10-16T14:33:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480c6aa0 con 0x7fb848159ad0 2023-10-16T14:33:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 0x7fb848153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 0x7fb848153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33672/0 (socket says 172.21.15.29:33672) 2023-10-16T14:33:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 -- 172.21.15.29:0/2909707748 learned_addr learned my addr 172.21.15.29:0/2909707748 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb846d9d640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb848159ad0 0x7fb8480ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb845d9b640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb848070f30 0x7fb8480c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 -- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb848159ad0 msgr2=0x7fb8480ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb848159ad0 0x7fb8480ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 -- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb848070f30 msgr2=0x7fb8480c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb848070f30 0x7fb8480c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb84659c640 1 -- 172.21.15.29:0/2909707748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c009400 con 0x7fb84806b3e0 2023-10-16T14:33:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.500+0000 7fb846d9d640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb848159ad0 0x7fb8480ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.501+0000 7fb84659c640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 0x7fb848153e20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb83800c390 tx=0x7fb83800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.501+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838005ce0 con 0x7fb84806b3e0 2023-10-16T14:33:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.501+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838005e40 con 0x7fb84806b3e0 2023-10-16T14:33:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.501+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380072f0 con 0x7fb84806b3e0 2023-10-16T14:33:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.501+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8480c6d80 con 0x7fb84806b3e0 2023-10-16T14:33:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.501+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8480c7ac0 con 0x7fb84806b3e0 2023-10-16T14:33:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.502+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84804e9b0 con 0x7fb84806b3e0 2023-10-16T14:33:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.503+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb838007450 con 0x7fb84806b3e0 2023-10-16T14:33:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.504+0000 7fb836ffd640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb82806af90 0x7fb82806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.504+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb838088a60 con 0x7fb84806b3e0 2023-10-16T14:33:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.505+0000 7fb845d9b640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb82806af90 0x7fb82806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.506+0000 7fb845d9b640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb82806af90 0x7fb82806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb83c036000 tx=0x7fb83c03c040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.508+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb838053d50 con 0x7fb84806b3e0 2023-10-16T14:33:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.701+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8480b4600 con 0x7fb84806b3e0 2023-10-16T14:33:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.702+0000 7fb836ffd640 1 -- 172.21.15.29:0/2909707748 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb838017020 con 0x7fb84806b3e0 2023-10-16T14:33:02.705 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:33:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.704+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb82806af90 msgr2=0x7fb82806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.704+0000 7fb84759e640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb82806af90 0x7fb82806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb83c036000 tx=0x7fb83c03c040 comp rx=0 tx=0).stop 2023-10-16T14:33:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.704+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 msgr2=0x7fb848153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.704+0000 7fb84759e640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 0x7fb848153e20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb83800c390 tx=0x7fb83800c850 comp rx=0 tx=0).stop 2023-10-16T14:33:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 shutdown_connections 2023-10-16T14:33:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb848159ad0 0x7fb8480ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb848070f30 0x7fb8480c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb82806af90 0x7fb82806d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 --2- 172.21.15.29:0/2909707748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb84806b3e0 0x7fb848153e20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 >> 172.21.15.29:0/2909707748 conn(0x7fb8480b0170 msgr2=0x7fb848158e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:02.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 shutdown_connections 2023-10-16T14:33:02.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.705+0000 7fb84759e640 1 -- 172.21.15.29:0/2909707748 wait complete. 2023-10-16T14:33:02.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:33:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.953+0000 7fd0a3fff640 1 Processor -- start 2023-10-16T14:33:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.954+0000 7fd0a3fff640 1 -- start start 2023-10-16T14:33:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.955+0000 7fd0a3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0a406c040 0x7fd0a406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.955+0000 7fd0a3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0a40ad970 0x7fd0a415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd0a3fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a406c960 con 0x7fd0a415ebc0 2023-10-16T14:33:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd0a3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a415b9d0 con 0x7fd0a406c040 2023-10-16T14:33:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd0a3fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a415bcb0 con 0x7fd0a40ad970 2023-10-16T14:33:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd0a37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd0a415ebc0 0x7fd0a40cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43184/0 (socket says 172.21.15.29:43184) 2023-10-16T14:33:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd0a37fe640 1 -- 172.21.15.29:0/945493695 learned_addr learned my addr 172.21.15.29:0/945493695 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd0a2ffd640 1 --2- 172.21.15.29:0/945493695 >> v2:172.21.15.29:3300/0 conn(0x7fd0a406c040 0x7fd0a406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.956+0000 7fd09bfff640 1 --2- 172.21.15.29:0/945493695 >> v2:172.21.15.170:3300/0 conn(0x7fd0a40ad970 0x7fd0a415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.957+0000 7fd0a27fc640 1 -- 172.21.15.29:0/945493695 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724491089 0 0) 0x7fd0a406c960 con 0x7fd0a415ebc0 2023-10-16T14:33:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.957+0000 7fd0a2ffd640 1 -- 172.21.15.29:0/945493695 >> v2:172.21.15.170:3300/0 conn(0x7fd0a40ad970 msgr2=0x7fd0a415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.957+0000 7fd0a2ffd640 1 --2- 172.21.15.29:0/945493695 >> v2:172.21.15.170:3300/0 conn(0x7fd0a40ad970 0x7fd0a415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.957+0000 7fd0a2ffd640 1 -- 172.21.15.29:0/945493695 >> v1:172.21.15.29:6789/0 conn(0x7fd0a415ebc0 legacy=0x7fd0a40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.957+0000 7fd0a2ffd640 1 -- 172.21.15.29:0/945493695 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a415bf90 con 0x7fd0a406c040 2023-10-16T14:33:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.958+0000 7fd0a2ffd640 1 --2- 172.21.15.29:0/945493695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a406c040 0x7fd0a406c420 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40cb060 tx=0x7fd09402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d4a9b28c50103a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.958+0000 7fd0a27fc640 1 -- 172.21.15.29:0/945493695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094031830 con 0x7fd0a406c040 2023-10-16T14:33:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.958+0000 7fd0a27fc640 1 -- 172.21.15.29:0/945493695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd094031990 con 0x7fd0a406c040 2023-10-16T14:33:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.958+0000 7fd0a27fc640 1 -- 172.21.15.29:0/945493695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094031d20 con 0x7fd0a406c040 2023-10-16T14:33:02.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 -- 172.21.15.29:0/945493695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a406c040 msgr2=0x7fd0a406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/945493695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a406c040 0x7fd0a406c420 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40cb060 tx=0x7fd09402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 -- 172.21.15.29:0/945493695 shutdown_connections 2023-10-16T14:33:02.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/945493695 >> v2:172.21.15.170:3300/0 conn(0x7fd0a40ad970 0x7fd0a415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/945493695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a406c040 0x7fd0a406c420 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 -- 172.21.15.29:0/945493695 >> 172.21.15.29:0/945493695 conn(0x7fd0a40ce6a0 msgr2=0x7fd0a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:02.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 -- 172.21.15.29:0/945493695 shutdown_connections 2023-10-16T14:33:02.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.959+0000 7fd0a3fff640 1 -- 172.21.15.29:0/945493695 wait complete. 2023-10-16T14:33:02.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.961+0000 7fd0a3fff640 1 Processor -- start 2023-10-16T14:33:02.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.961+0000 7fd0a3fff640 1 -- start start 2023-10-16T14:33:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd0a3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0a406c040 0x7fd0a40cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd0a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 0x7fd0a40b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd0a3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0a415ebc0 0x7fd0a41568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd0a3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40b2540 con 0x7fd0a40ad970 2023-10-16T14:33:02.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd0a3fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a41583f0 con 0x7fd0a415ebc0 2023-10-16T14:33:02.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd0a3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a4158050 con 0x7fd0a406c040 2023-10-16T14:33:02.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 0x7fd0a40b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 0x7fd0a40b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33686/0 (socket says 172.21.15.29:33686) 2023-10-16T14:33:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.962+0000 7fd09bfff640 1 -- 172.21.15.29:0/2180414248 learned_addr learned my addr 172.21.15.29:0/2180414248 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a2ffd640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0a406c040 0x7fd0a40cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a37fe640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0a415ebc0 0x7fd0a41568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd09bfff640 1 -- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0a406c040 msgr2=0x7fd0a40cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd09bfff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0a406c040 0x7fd0a40cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd09bfff640 1 -- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0a415ebc0 msgr2=0x7fd0a41568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd09bfff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0a415ebc0 0x7fd0a41568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd09bfff640 1 -- 172.21.15.29:0/2180414248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094009400 con 0x7fd0a40ad970 2023-10-16T14:33:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a37fe640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0a415ebc0 0x7fd0a41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd09bfff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 0x7fd0a40b12c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d600 tx=0x7fd08c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c004240 con 0x7fd0a40ad970 2023-10-16T14:33:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c0043a0 con 0x7fd0a40ad970 2023-10-16T14:33:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c003830 con 0x7fd0a40ad970 2023-10-16T14:33:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.963+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a40b1ca0 con 0x7fd0a40ad970 2023-10-16T14:33:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.964+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a40b2260 con 0x7fd0a40ad970 2023-10-16T14:33:02.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.965+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a406b330 con 0x7fd0a40ad970 2023-10-16T14:33:02.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.966+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd08c003990 con 0x7fd0a40ad970 2023-10-16T14:33:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.967+0000 7fd0a0ff9640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd08406af90 0x7fd08406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.967+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd08c089630 con 0x7fd0a40ad970 2023-10-16T14:33:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.967+0000 7fd0a2ffd640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd08406af90 0x7fd08406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.969+0000 7fd0a2ffd640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd08406af90 0x7fd08406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd094036000 tx=0x7fd094008030 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:02.971+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd08c054920 con 0x7fd0a40ad970 2023-10-16T14:33:03.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.164+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd0a4158530 con 0x7fd0a40ad970 2023-10-16T14:33:03.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.165+0000 7fd0a0ff9640 1 -- 172.21.15.29:0/2180414248 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v93) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd0a4158530 con 0x7fd0a40ad970 2023-10-16T14:33:03.168 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:03.168 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:33:03.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:03.169 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:33:03.169 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:33:03.169 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:03.169 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:33:03.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd08406af90 msgr2=0x7fd08406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:03.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd08406af90 0x7fd08406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd094036000 tx=0x7fd094008030 comp rx=0 tx=0).stop 2023-10-16T14:33:03.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 msgr2=0x7fd0a40b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:03.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 0x7fd0a40b12c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d600 tx=0x7fd08c00dac0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 shutdown_connections 2023-10-16T14:33:03.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd08406af90 0x7fd08406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a40ad970 0x7fd0a40b12c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0a415ebc0 0x7fd0a41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 --2- 172.21.15.29:0/2180414248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0a406c040 0x7fd0a40cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 >> 172.21.15.29:0/2180414248 conn(0x7fd0a40ce6a0 msgr2=0x7fd0a40acf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:03.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 shutdown_connections 2023-10-16T14:33:03.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.168+0000 7fd0a3fff640 1 -- 172.21.15.29:0/2180414248 wait complete. 2023-10-16T14:33:03.191 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:33:03.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:33:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.425+0000 7f76d7fff640 1 Processor -- start 2023-10-16T14:33:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.425+0000 7f76d7fff640 1 -- start start 2023-10-16T14:33:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.427+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 0x7f76d80c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.427+0000 7f76d7fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80c8e70 con 0x7f76d815ad80 2023-10-16T14:33:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.427+0000 7f76d7fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80b4170 con 0x7f76d80cae40 2023-10-16T14:33:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.427+0000 7f76d7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80adde0 con 0x7f76d806d280 2023-10-16T14:33:03.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.427+0000 7f76d6ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76d815ad80 0x7f76d806cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43196/0 (socket says 172.21.15.29:43196) 2023-10-16T14:33:03.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.427+0000 7f76d6ffd640 1 -- 172.21.15.29:0/2310314296 learned_addr learned my addr 172.21.15.29:0/2310314296 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710954516 0 0) 0x7f76d80c8e70 con 0x7f76d815ad80 2023-10-16T14:33:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4003520 con 0x7f76d815ad80 2023-10-16T14:33:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d67fc640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 0x7f76d80c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2804034769 0 0) 0x7f76c4003520 con 0x7f76d815ad80 2023-10-16T14:33:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 msgr2=0x7f76d80c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 0x7f76d80c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 >> v1:172.21.15.170:6789/0 conn(0x7f76d80cae40 legacy=0x7f76d80c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80b4410 con 0x7f76d815ad80 2023-10-16T14:33:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f76c0001670 con 0x7f76d815ad80 2023-10-16T14:33:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 >> v1:172.21.15.29:6789/0 conn(0x7f76d815ad80 legacy=0x7f76d806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.428+0000 7f76d5ffb640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d5ffb640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c40047c0 0x7f76c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d5ffb640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 0x7f76c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80adde0 con 0x7f76c4003a10 2023-10-16T14:33:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c400bb10 con 0x7f76c40047c0 2023-10-16T14:33:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c400bcd0 con 0x7f76d806d280 2023-10-16T14:33:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d67fc640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c40047c0 0x7f76c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d77fe640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 0x7f76c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d6ffd640 1 -- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 msgr2=0x7f76c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d806d280 0x7f76c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d6ffd640 1 -- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c40047c0 msgr2=0x7f76c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c40047c0 0x7f76c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d6ffd640 1 -- 172.21.15.29:0/2310314296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c4003520 con 0x7f76c4003a10 2023-10-16T14:33:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.429+0000 7f76d67fc640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76c40047c0 0x7f76c4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.430+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76c4003e10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c210 tx=0x7f76c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66213b568725b3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c002cae0 con 0x7f76c4003a10 2023-10-16T14:33:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c002cc60 con 0x7f76c4003a10 2023-10-16T14:33:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d5ffb640 1 -- 172.21.15.29:0/2310314296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c00058d0 con 0x7f76c4003a10 2023-10-16T14:33:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d7fff640 1 -- 172.21.15.29:0/2310314296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 msgr2=0x7f76c4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d7fff640 1 --2- 172.21.15.29:0/2310314296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76c4003e10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c210 tx=0x7f76c40057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d77fe640 1 -- 172.21.15.29:0/2310314296 reap_dead start 2023-10-16T14:33:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d7fff640 1 -- 172.21.15.29:0/2310314296 shutdown_connections 2023-10-16T14:33:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d7fff640 1 -- 172.21.15.29:0/2310314296 >> 172.21.15.29:0/2310314296 conn(0x7f76d80ce820 msgr2=0x7f76d80cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d7fff640 1 -- 172.21.15.29:0/2310314296 shutdown_connections 2023-10-16T14:33:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.431+0000 7f76d7fff640 1 -- 172.21.15.29:0/2310314296 wait complete. 2023-10-16T14:33:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.433+0000 7f76d7fff640 1 Processor -- start 2023-10-16T14:33:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.433+0000 7f76d7fff640 1 -- start start 2023-10-16T14:33:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.434+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76d80c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.434+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76d80c39b0 0x7f76d80c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.434+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d8158ed0 0x7f76d80c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.434+0000 7f76d7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80c8720 con 0x7f76c4003a10 2023-10-16T14:33:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.434+0000 7f76d7fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80c2910 con 0x7f76d80c39b0 2023-10-16T14:33:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.434+0000 7f76d7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80c2bf0 con 0x7f76d8158ed0 2023-10-16T14:33:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76d80c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76d80c2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33700/0 (socket says 172.21.15.29:33700) 2023-10-16T14:33:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 -- 172.21.15.29:0/955239689 learned_addr learned my addr 172.21.15.29:0/955239689 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d67fc640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76d80c39b0 0x7f76d80c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d77fe640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d8158ed0 0x7f76d80c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 -- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d8158ed0 msgr2=0x7f76d80c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d8158ed0 0x7f76d80c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 -- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76d80c39b0 msgr2=0x7f76d80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:03.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76d80c39b0 0x7f76d80c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:03.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 -- 172.21.15.29:0/955239689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80c2ed0 con 0x7f76c4003a10 2023-10-16T14:33:03.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.435+0000 7f76d6ffd640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76d80c2310 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f76c002c480 tx=0x7f76c0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:03.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.436+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c003c050 con 0x7f76c4003a10 2023-10-16T14:33:03.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.436+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c003a030 con 0x7f76c4003a10 2023-10-16T14:33:03.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.436+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d80c31b0 con 0x7f76c4003a10 2023-10-16T14:33:03.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.436+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0007070 con 0x7f76c4003a10 2023-10-16T14:33:03.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.436+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d8159360 con 0x7f76c4003a10 2023-10-16T14:33:03.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.437+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d80c36d0 con 0x7f76c4003a10 2023-10-16T14:33:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.438+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f76c0032070 con 0x7f76c4003a10 2023-10-16T14:33:03.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.439+0000 7f76cf7fe640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f76b806afc0 0x7f76b806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.439+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f76c00aaa60 con 0x7f76c4003a10 2023-10-16T14:33:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.440+0000 7f76d67fc640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f76b806afc0 0x7f76b806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.441+0000 7f76d67fc640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f76b806afc0 0x7f76b806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f76d80b2bb0 tx=0x7f76c800b040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:03.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.443+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f76d80c36d0 con 0x7f76c4003a10 2023-10-16T14:33:03.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:03.640+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f76d80c36d0 con 0x7f76c4003a10 2023-10-16T14:33:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.225+0000 7f76cf7fe640 1 -- 172.21.15.29:0/955239689 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v94) v1 ==== 138+0+0 (secure 0 0 0) 0x7f76d80c36d0 con 0x7f76c4003a10 2023-10-16T14:33:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:33:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f76b806afc0 msgr2=0x7f76b806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f76b806afc0 0x7f76b806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f76d80b2bb0 tx=0x7f76c800b040 comp rx=0 tx=0).stop 2023-10-16T14:33:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 msgr2=0x7f76d80c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76d80c2310 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f76c002c480 tx=0x7f76c0001a00 comp rx=0 tx=0).stop 2023-10-16T14:33:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 shutdown_connections 2023-10-16T14:33:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76d8158ed0 0x7f76d80c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76d80c39b0 0x7f76d80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f76b806afc0 0x7f76b806d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 --2- 172.21.15.29:0/955239689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76c4003a10 0x7f76d80c2310 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.229+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 >> 172.21.15.29:0/955239689 conn(0x7f76d80ce820 msgr2=0x7f76d80ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.230+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 shutdown_connections 2023-10-16T14:33:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.230+0000 7f76d7fff640 1 -- 172.21.15.29:0/955239689 wait complete. 2023-10-16T14:33:04.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:33:04.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.482+0000 7eff74d33640 1 Processor -- start 2023-10-16T14:33:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.482+0000 7eff74d33640 1 -- start start 2023-10-16T14:33:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff74d33640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 0x7eff7015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff74d33640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff700707e0 con 0x7eff700ce390 2023-10-16T14:33:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff74d33640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7006b7a0 con 0x7eff7015aaa0 2023-10-16T14:33:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff74d33640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015ef20 con 0x7eff7015ae80 2023-10-16T14:33:04.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff6ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7eff700ce390 0x7eff70070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36750/0 (socket says 172.21.15.29:36750) 2023-10-16T14:33:04.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff6ed9d640 1 -- 172.21.15.29:0/2890860018 learned_addr learned my addr 172.21.15.29:0/2890860018 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.484+0000 7eff67fff640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 0x7eff7015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247624796 0 0) 0x7eff700707e0 con 0x7eff700ce390 2023-10-16T14:33:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c003520 con 0x7eff700ce390 2023-10-16T14:33:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 313930401 0 0) 0x7eff5c003520 con 0x7eff700ce390 2023-10-16T14:33:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 msgr2=0x7eff7015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 0x7eff7015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 >> v1:172.21.15.170:6789/0 conn(0x7eff7015aaa0 legacy=0x7eff700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7015f200 con 0x7eff700ce390 2023-10-16T14:33:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7eff680019f0 con 0x7eff700ce390 2023-10-16T14:33:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 >> v1:172.21.15.29:6789/0 conn(0x7eff700ce390 legacy=0x7eff70070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 0x7eff5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff5c004d00 0x7eff5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015ef20 con 0x7eff5c0039a0 2023-10-16T14:33:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00bab0 con 0x7eff5c004d00 2023-10-16T14:33:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.485+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00bc70 con 0x7eff7015ae80 2023-10-16T14:33:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6ed9d640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff5c004d00 0x7eff5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff67fff640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 0x7eff5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6e59c640 1 -- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 msgr2=0x7eff5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff7015ae80 0x7eff5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6e59c640 1 -- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff5c004d00 msgr2=0x7eff5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff5c004d00 0x7eff5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.486+0000 7eff6e59c640 1 -- 172.21.15.29:0/2890860018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c003520 con 0x7eff5c0039a0 2023-10-16T14:33:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.487+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff5c003dc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7eff60001750 tx=0x7eff5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1757f81ac398746 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff600052d0 con 0x7eff5c0039a0 2023-10-16T14:33:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6002e050 con 0x7eff5c0039a0 2023-10-16T14:33:04.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff6dd9b640 1 -- 172.21.15.29:0/2890860018 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff600035f0 con 0x7eff5c0039a0 2023-10-16T14:33:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff74d33640 1 -- 172.21.15.29:0/2890860018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 msgr2=0x7eff5c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff74d33640 1 --2- 172.21.15.29:0/2890860018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff5c003dc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7eff60001750 tx=0x7eff5c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff6ed9d640 1 -- 172.21.15.29:0/2890860018 reap_dead start 2023-10-16T14:33:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff74d33640 1 -- 172.21.15.29:0/2890860018 shutdown_connections 2023-10-16T14:33:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff74d33640 1 -- 172.21.15.29:0/2890860018 >> 172.21.15.29:0/2890860018 conn(0x7eff700cb410 msgr2=0x7eff700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:04.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff74d33640 1 -- 172.21.15.29:0/2890860018 shutdown_connections 2023-10-16T14:33:04.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.488+0000 7eff74d33640 1 -- 172.21.15.29:0/2890860018 wait complete. 2023-10-16T14:33:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.490+0000 7eff74d33640 1 Processor -- start 2023-10-16T14:33:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.490+0000 7eff74d33640 1 -- start start 2023-10-16T14:33:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.491+0000 7eff74d33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff70153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.491+0000 7eff74d33640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff700c7240 0x7eff700c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.491+0000 7eff74d33640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff700b04c0 0x7eff700b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.491+0000 7eff74d33640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff700b0e80 con 0x7eff5c0039a0 2023-10-16T14:33:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.491+0000 7eff74d33640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70153660 con 0x7eff700b04c0 2023-10-16T14:33:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.491+0000 7eff74d33640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7015d440 con 0x7eff700c7240 2023-10-16T14:33:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff70153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff70153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53922/0 (socket says 172.21.15.29:53922) 2023-10-16T14:33:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 -- 172.21.15.29:0/2217045594 learned_addr learned my addr 172.21.15.29:0/2217045594 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff67fff640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff700c7240 0x7eff700c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6ed9d640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff700b04c0 0x7eff700b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 -- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff700c7240 msgr2=0x7eff700c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff700c7240 0x7eff700c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 -- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff700b04c0 msgr2=0x7eff700b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff700b04c0 0x7eff700b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff6e59c640 1 -- 172.21.15.29:0/2217045594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7015d6e0 con 0x7eff5c0039a0 2023-10-16T14:33:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.492+0000 7eff67fff640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff700c7240 0x7eff700c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:04.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.493+0000 7eff6e59c640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff70153120 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7eff60008fd0 tx=0x7eff60001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:04.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.493+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff60007a30 con 0x7eff5c0039a0 2023-10-16T14:33:04.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.493+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff600069e0 con 0x7eff5c0039a0 2023-10-16T14:33:04.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.493+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff60006d90 con 0x7eff5c0039a0 2023-10-16T14:33:04.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.493+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7015d9c0 con 0x7eff5c0039a0 2023-10-16T14:33:04.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.493+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff701f6b00 con 0x7eff5c0039a0 2023-10-16T14:33:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.494+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7004e160 con 0x7eff5c0039a0 2023-10-16T14:33:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.495+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7eff600035f0 con 0x7eff5c0039a0 2023-10-16T14:33:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.496+0000 7eff66ffd640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7eff5006afe0 0x7eff5006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.496+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5227+0+0 (secure 0 0 0) 0x7eff600ab610 con 0x7eff5c0039a0 2023-10-16T14:33:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.497+0000 7eff67fff640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7eff5006afe0 0x7eff5006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:04.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.498+0000 7eff67fff640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7eff5006afe0 0x7eff5006d4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff7015d330 tx=0x7eff58008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:04.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.501+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff60076660 con 0x7eff5c0039a0 2023-10-16T14:33:04.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:04.698+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7eff700cdc60 con 0x7eff5c0039a0 2023-10-16T14:33:05.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.237+0000 7eff66ffd640 1 -- 172.21.15.29:0/2217045594 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v95) v1 ==== 170+0+0 (secure 0 0 0) 0x7eff60079c80 con 0x7eff5c0039a0 2023-10-16T14:33:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:33:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7eff5006afe0 msgr2=0x7eff5006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7eff5006afe0 0x7eff5006d4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff7015d330 tx=0x7eff58008040 comp rx=0 tx=0).stop 2023-10-16T14:33:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 msgr2=0x7eff70153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff70153120 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7eff60008fd0 tx=0x7eff60001120 comp rx=0 tx=0).stop 2023-10-16T14:33:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 shutdown_connections 2023-10-16T14:33:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7eff700b04c0 0x7eff700b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7eff700c7240 0x7eff700c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7eff5006afe0 0x7eff5006d4a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 --2- 172.21.15.29:0/2217045594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5c0039a0 0x7eff70153120 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 >> 172.21.15.29:0/2217045594 conn(0x7eff700cb410 msgr2=0x7eff7006c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 shutdown_connections 2023-10-16T14:33:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.241+0000 7eff74d33640 1 -- 172.21.15.29:0/2217045594 wait complete. 2023-10-16T14:33:05.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:33:05.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.470+0000 7f5f69400640 1 Processor -- start 2023-10-16T14:33:05.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.471+0000 7f5f69400640 1 -- start start 2023-10-16T14:33:05.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.472+0000 7f5f69400640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6415ad70 0x7f5f64069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:05.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.472+0000 7f5f69400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f640c9ca0 0x7f5f6406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:05.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.472+0000 7f5f69400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640cb030 con 0x7f5f640ca3b0 2023-10-16T14:33:05.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.472+0000 7f5f69400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640c91c0 con 0x7f5f640c9ca0 2023-10-16T14:33:05.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.472+0000 7f5f69400640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640c94a0 con 0x7f5f6415ad70 2023-10-16T14:33:05.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.472+0000 7f5f5bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f640c9ca0 0x7f5f6406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:05.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f637fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f640ca3b0 0x7f5f640ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36764/0 (socket says 172.21.15.29:36764) 2023-10-16T14:33:05.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f637fe640 1 -- 172.21.15.29:0/2750928328 learned_addr learned my addr 172.21.15.29:0/2750928328 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:05.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f62ffd640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6415ad70 0x7f5f64069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:05.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f5bfff640 1 -- 172.21.15.29:0/2750928328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6415ad70 msgr2=0x7f5f64069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:05.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f5bfff640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6415ad70 0x7f5f64069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f5bfff640 1 -- 172.21.15.29:0/2750928328 >> v1:172.21.15.29:6789/0 conn(0x7f5f640ca3b0 legacy=0x7f5f640ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:05.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f5bfff640 1 -- 172.21.15.29:0/2750928328 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640c9780 con 0x7f5f640c9ca0 2023-10-16T14:33:05.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f62ffd640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6415ad70 0x7f5f64069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:05.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.473+0000 7f5f627fc640 1 -- 172.21.15.29:0/2750928328 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872806886 0 0) 0x7f5f640cb030 con 0x7f5f640ca3b0 2023-10-16T14:33:05.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.474+0000 7f5f5bfff640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640c9ca0 0x7f5f6406a2e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0097a0 tx=0x7f5f4c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179eff1b30812a55 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f627fc640 1 -- 172.21.15.29:0/2750928328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c02d040 con 0x7f5f640c9ca0 2023-10-16T14:33:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f627fc640 1 -- 172.21.15.29:0/2750928328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c031810 con 0x7f5f640c9ca0 2023-10-16T14:33:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f627fc640 1 -- 172.21.15.29:0/2750928328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c031ba0 con 0x7f5f640c9ca0 2023-10-16T14:33:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 -- 172.21.15.29:0/2750928328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640c9ca0 msgr2=0x7f5f6406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640c9ca0 0x7f5f6406a2e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0097a0 tx=0x7f5f4c02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 -- 172.21.15.29:0/2750928328 shutdown_connections 2023-10-16T14:33:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640c9ca0 0x7f5f6406a2e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 --2- 172.21.15.29:0/2750928328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6415ad70 0x7f5f64069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 -- 172.21.15.29:0/2750928328 >> 172.21.15.29:0/2750928328 conn(0x7f5f640ce9b0 msgr2=0x7f5f640cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.475+0000 7f5f69400640 1 -- 172.21.15.29:0/2750928328 shutdown_connections 2023-10-16T14:33:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.476+0000 7f5f69400640 1 -- 172.21.15.29:0/2750928328 wait complete. 2023-10-16T14:33:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.477+0000 7f5f69400640 1 Processor -- start 2023-10-16T14:33:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.478+0000 7f5f69400640 1 -- start start 2023-10-16T14:33:05.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.478+0000 7f5f69400640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f640c9ca0 0x7f5f64151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:05.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f69400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 0x7f5f640c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:05.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f69400640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f6415ad70 0x7f5f640c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:05.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f69400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640c61e0 con 0x7f5f640ca3b0 2023-10-16T14:33:05.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f69400640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640af4f0 con 0x7f5f6415ad70 2023-10-16T14:33:05.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f69400640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6415dac0 con 0x7f5f640c9ca0 2023-10-16T14:33:05.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 0x7f5f640c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:05.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 0x7f5f640c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53946/0 (socket says 172.21.15.29:53946) 2023-10-16T14:33:05.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 -- 172.21.15.29:0/1928584330 learned_addr learned my addr 172.21.15.29:0/1928584330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:05.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f637fe640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f6415ad70 0x7f5f640c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:05.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 -- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f640c9ca0 msgr2=0x7f5f64151d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:33:05.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f640c9ca0 0x7f5f64151d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 -- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f6415ad70 msgr2=0x7f5f640c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:05.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f6415ad70 0x7f5f640c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:05.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.479+0000 7f5f5bfff640 1 -- 172.21.15.29:0/1928584330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c009400 con 0x7f5f640ca3b0 2023-10-16T14:33:05.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.480+0000 7f5f5bfff640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 0x7f5f640c7c70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0098c0 tx=0x7f5f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:05.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.480+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c03ad60 con 0x7f5f640ca3b0 2023-10-16T14:33:05.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.480+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c044400 con 0x7f5f640ca3b0 2023-10-16T14:33:05.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.480+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0447b0 con 0x7f5f640ca3b0 2023-10-16T14:33:05.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.480+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6415dda0 con 0x7f5f640ca3b0 2023-10-16T14:33:05.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.480+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f640c5e00 con 0x7f5f640ca3b0 2023-10-16T14:33:05.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.481+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640c5f90 con 0x7f5f640ca3b0 2023-10-16T14:33:05.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.483+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f4c04d440 con 0x7f5f640ca3b0 2023-10-16T14:33:05.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.484+0000 7f5f5b7fe640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f5f4406af90 0x7f5f4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:05.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.484+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5f4c0ac0c0 con 0x7f5f640ca3b0 2023-10-16T14:33:05.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.485+0000 7f5f62ffd640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f5f4406af90 0x7f5f4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:05.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.486+0000 7f5f62ffd640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f5f4406af90 0x7f5f4406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f54006fd0 tx=0x7f5f54008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:05.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.488+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f4c077240 con 0x7f5f640ca3b0 2023-10-16T14:33:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:05.721+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5f6404e160 con 0x7f5f640ca3b0 2023-10-16T14:33:06.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.266+0000 7f5f5b7fe640 1 -- 172.21.15.29:0/1928584330 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5f4c0350c0 con 0x7f5f640ca3b0 2023-10-16T14:33:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 10 and data pool 11 2023-10-16T14:33:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.274+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f5f4406af90 msgr2=0x7f5f4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.274+0000 7f5f69400640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f5f4406af90 0x7f5f4406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f54006fd0 tx=0x7f5f54008040 comp rx=0 tx=0).stop 2023-10-16T14:33:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.274+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 msgr2=0x7f5f640c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.274+0000 7f5f69400640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 0x7f5f640c7c70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0098c0 tx=0x7f5f4c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.275+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 shutdown_connections 2023-10-16T14:33:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.275+0000 7f5f69400640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f5f4406af90 0x7f5f4406d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.275+0000 7f5f69400640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f640ca3b0 0x7f5f640c7c70 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.275+0000 7f5f69400640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f6415ad70 0x7f5f640c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.275+0000 7f5f69400640 1 --2- 172.21.15.29:0/1928584330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f640c9ca0 0x7f5f64151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.275+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 >> 172.21.15.29:0/1928584330 conn(0x7f5f640ce9b0 msgr2=0x7f5f640acce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.277+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 shutdown_connections 2023-10-16T14:33:06.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.278+0000 7f5f69400640 1 -- 172.21.15.29:0/1928584330 wait complete. 2023-10-16T14:33:06.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.488+0000 7fd02652c640 1 Processor -- start 2023-10-16T14:33:06.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.489+0000 7fd02652c640 1 -- start start 2023-10-16T14:33:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd02652c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02015c850 0x7fd0200cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd02652c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd02006b4c0 0x7fd0200c9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd02652c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02006ffe0 con 0x7fd0200cdc40 2023-10-16T14:33:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd02652c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0200cb9e0 con 0x7fd02006b4c0 2023-10-16T14:33:06.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd02652c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200ccf40 con 0x7fd02015c850 2023-10-16T14:33:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd024ac9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd02006b4c0 0x7fd0200c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd024ac9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd02006b4c0 0x7fd0200c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53952/0 (socket says 172.21.15.29:53952) 2023-10-16T14:33:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd024ac9640 1 -- 172.21.15.29:0/1117972090 learned_addr learned my addr 172.21.15.29:0/1117972090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:06.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.490+0000 7fd017fff640 1 --1- 172.21.15.29:0/1117972090 >> v1:172.21.15.29:6789/0 conn(0x7fd0200cdc40 0x7fd02006adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36772/0 (socket says 172.21.15.29:36772) 2023-10-16T14:33:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.491+0000 7fd024ac9640 1 -- 172.21.15.29:0/1117972090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02015c850 msgr2=0x7fd0200cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.491+0000 7fd024ac9640 1 --2- 172.21.15.29:0/1117972090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02015c850 0x7fd0200cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.491+0000 7fd024ac9640 1 -- 172.21.15.29:0/1117972090 >> v1:172.21.15.29:6789/0 conn(0x7fd0200cdc40 legacy=0x7fd02006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.491+0000 7fd024ac9640 1 -- 172.21.15.29:0/1117972090 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200cb420 con 0x7fd02006b4c0 2023-10-16T14:33:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.491+0000 7fd01f7fe640 1 -- 172.21.15.29:0/1117972090 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810239156 0 0) 0x7fd02006ffe0 con 0x7fd0200cdc40 2023-10-16T14:33:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.492+0000 7fd024ac9640 1 --2- 172.21.15.29:0/1117972090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02006b4c0 0x7fd0200c9890 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd01800d150 tx=0x7fd01802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60471ac6b742fbe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.492+0000 7fd01f7fe640 1 -- 172.21.15.29:0/1117972090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01802edf0 con 0x7fd02006b4c0 2023-10-16T14:33:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.492+0000 7fd01f7fe640 1 -- 172.21.15.29:0/1117972090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd018032070 con 0x7fd02006b4c0 2023-10-16T14:33:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.492+0000 7fd01f7fe640 1 -- 172.21.15.29:0/1117972090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018038620 con 0x7fd02006b4c0 2023-10-16T14:33:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 -- 172.21.15.29:0/1117972090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02006b4c0 msgr2=0x7fd0200c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 --2- 172.21.15.29:0/1117972090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02006b4c0 0x7fd0200c9890 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd01800d150 tx=0x7fd01802e040 comp rx=0 tx=0).stop 2023-10-16T14:33:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 -- 172.21.15.29:0/1117972090 shutdown_connections 2023-10-16T14:33:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 --2- 172.21.15.29:0/1117972090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02006b4c0 0x7fd0200c9890 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 --2- 172.21.15.29:0/1117972090 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02015c850 0x7fd0200cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 -- 172.21.15.29:0/1117972090 >> 172.21.15.29:0/1117972090 conn(0x7fd0200b0d30 msgr2=0x7fd0200b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 -- 172.21.15.29:0/1117972090 shutdown_connections 2023-10-16T14:33:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.493+0000 7fd02652c640 1 -- 172.21.15.29:0/1117972090 wait complete. 2023-10-16T14:33:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.495+0000 7fd02652c640 1 Processor -- start 2023-10-16T14:33:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.495+0000 7fd02652c640 1 -- start start 2023-10-16T14:33:06.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd02652c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02006b4c0 0x7fd02015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd02652c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 0x7fd0200b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd02652c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd02015c850 0x7fd0200b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd02652c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200b41d0 con 0x7fd0200cdc40 2023-10-16T14:33:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd02652c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200c7c30 con 0x7fd02015c850 2023-10-16T14:33:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd02652c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200c7f10 con 0x7fd02006b4c0 2023-10-16T14:33:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd017fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 0x7fd0200b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd017fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 0x7fd0200b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53956/0 (socket says 172.21.15.29:53956) 2023-10-16T14:33:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.496+0000 7fd017fff640 1 -- 172.21.15.29:0/3361964350 learned_addr learned my addr 172.21.15.29:0/3361964350 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd017fff640 1 -- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02006b4c0 msgr2=0x7fd02015ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd017fff640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02006b4c0 0x7fd02015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd024ac9640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd02015c850 0x7fd0200b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd017fff640 1 -- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd02015c850 msgr2=0x7fd0200b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd017fff640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd02015c850 0x7fd0200b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd017fff640 1 -- 172.21.15.29:0/3361964350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01800bd40 con 0x7fd0200cdc40 2023-10-16T14:33:06.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd017fff640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 0x7fd0200b3410 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd00800c430 tx=0x7fd00800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0080074e0 con 0x7fd0200cdc40 2023-10-16T14:33:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd008007660 con 0x7fd0200cdc40 2023-10-16T14:33:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008004a20 con 0x7fd0200cdc40 2023-10-16T14:33:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.497+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0200c81f0 con 0x7fd0200cdc40 2023-10-16T14:33:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.498+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0200c7850 con 0x7fd0200cdc40 2023-10-16T14:33:06.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.499+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0200c8430 con 0x7fd0200cdc40 2023-10-16T14:33:06.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.500+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd008004b80 con 0x7fd0200cdc40 2023-10-16T14:33:06.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.501+0000 7fd01d7fa640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd00006af90 0x7fd00006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.501+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0080895a0 con 0x7fd0200cdc40 2023-10-16T14:33:06.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.502+0000 7fd01ffff640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd00006af90 0x7fd00006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.503+0000 7fd01ffff640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd00006af90 0x7fd00006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd010009710 tx=0x7fd010006c90 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:06.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.506+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0200c8430 con 0x7fd0200cdc40 2023-10-16T14:33:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.700+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0200c8430 con 0x7fd0200cdc40 2023-10-16T14:33:06.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.702+0000 7fd01d7fa640 1 -- 172.21.15.29:0/3361964350 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16581 (secure 0 0 0) 0x7fd0200c8430 con 0x7fd0200cdc40 2023-10-16T14:33:06.705 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:06.706 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":96,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:05.722622+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.704+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd00006af90 msgr2=0x7fd00006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.704+0000 7fd02652c640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd00006af90 0x7fd00006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd010009710 tx=0x7fd010006c90 comp rx=0 tx=0).stop 2023-10-16T14:33:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 msgr2=0x7fd0200b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 0x7fd0200b3410 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd00800c430 tx=0x7fd00800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 shutdown_connections 2023-10-16T14:33:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fd00006af90 0x7fd00006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0200cdc40 0x7fd0200b3410 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd02015c850 0x7fd0200b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 --2- 172.21.15.29:0/3361964350 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd02006b4c0 0x7fd02015ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 >> 172.21.15.29:0/3361964350 conn(0x7fd0200b0d30 msgr2=0x7fd02009e290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 shutdown_connections 2023-10-16T14:33:06.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.705+0000 7fd02652c640 1 -- 172.21.15.29:0/3361964350 wait complete. 2023-10-16T14:33:06.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.938+0000 7f6b16e85640 1 Processor -- start 2023-10-16T14:33:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.938+0000 7f6b16e85640 1 -- start start 2023-10-16T14:33:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.940+0000 7f6b16e85640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6b1015cbe0 0x7f6b100d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.940+0000 7f6b16e85640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b100b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.940+0000 7f6b16e85640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b100d15a0 con 0x7f6b100d1090 2023-10-16T14:33:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.940+0000 7f6b16e85640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b100d1880 con 0x7f6b1015cbe0 2023-10-16T14:33:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.940+0000 7f6b16e85640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100d1b60 con 0x7f6b1015cfc0 2023-10-16T14:33:06.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.941+0000 7f6b15682640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b100b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.941+0000 7f6b15682640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b100b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:50288/0 (socket says 172.21.15.29:50288) 2023-10-16T14:33:06.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.941+0000 7f6b15e83640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6b1015cbe0 0x7f6b100d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.941+0000 7f6b15682640 1 -- 172.21.15.29:0/3382683673 learned_addr learned my addr 172.21.15.29:0/3382683673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:06.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.942+0000 7f6b15682640 1 -- 172.21.15.29:0/3382683673 >> v2:172.21.15.170:3300/0 conn(0x7f6b1015cbe0 msgr2=0x7f6b100d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:06.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.942+0000 7f6b15682640 1 --2- 172.21.15.29:0/3382683673 >> v2:172.21.15.170:3300/0 conn(0x7f6b1015cbe0 0x7f6b100d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.942+0000 7f6b15682640 1 -- 172.21.15.29:0/3382683673 >> v1:172.21.15.170:6789/0 conn(0x7f6b100d1090 legacy=0x7f6b1006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.942+0000 7f6b15682640 1 -- 172.21.15.29:0/3382683673 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b100d1e40 con 0x7f6b1015cfc0 2023-10-16T14:33:06.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.943+0000 7f6b15682640 1 --2- 172.21.15.29:0/3382683673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b100b4570 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009870 tx=0x7f6b0002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=addcd041c06876f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:06.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b14e81640 1 -- 172.21.15.29:0/3382683673 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0002f830 con 0x7f6b1015cfc0 2023-10-16T14:33:06.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b14e81640 1 -- 172.21.15.29:0/3382683673 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0002f990 con 0x7f6b1015cfc0 2023-10-16T14:33:06.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b14e81640 1 -- 172.21.15.29:0/3382683673 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0002fd20 con 0x7f6b1015cfc0 2023-10-16T14:33:06.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 -- 172.21.15.29:0/3382683673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 msgr2=0x7f6b100b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3382683673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b100b4570 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009870 tx=0x7f6b0002ba10 comp rx=0 tx=0).stop 2023-10-16T14:33:06.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 -- 172.21.15.29:0/3382683673 shutdown_connections 2023-10-16T14:33:06.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3382683673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b100b4570 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3382683673 >> v2:172.21.15.170:3300/0 conn(0x7f6b1015cbe0 0x7f6b100d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 -- 172.21.15.29:0/3382683673 >> 172.21.15.29:0/3382683673 conn(0x7f6b100cf570 msgr2=0x7f6b100cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:06.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 -- 172.21.15.29:0/3382683673 shutdown_connections 2023-10-16T14:33:06.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.944+0000 7f6b16e85640 1 -- 172.21.15.29:0/3382683673 wait complete. 2023-10-16T14:33:06.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.946+0000 7f6b16e85640 1 Processor -- start 2023-10-16T14:33:06.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.946+0000 7f6b16e85640 1 -- start start 2023-10-16T14:33:06.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b16e85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b100d1090 0x7f6b100aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b16e85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 0x7f6b100b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b16e85640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b101f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b16e85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100b52e0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b16e85640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1015aae0 con 0x7f6b100d1090 2023-10-16T14:33:06.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b16e85640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100c9bc0 con 0x7f6b1015cfc0 2023-10-16T14:33:06.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b15682640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 0x7f6b100b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b15682640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 0x7f6b100b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53966/0 (socket says 172.21.15.29:53966) 2023-10-16T14:33:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.947+0000 7f6b15682640 1 -- 172.21.15.29:0/3288566833 learned_addr learned my addr 172.21.15.29:0/3288566833 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15e83640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b100d1090 0x7f6b100aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b16684640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b101f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15682640 1 -- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 msgr2=0x7f6b101f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15682640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b101f1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15682640 1 -- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b100d1090 msgr2=0x7f6b100aaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:06.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15682640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b100d1090 0x7f6b100aaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:06.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15682640 1 -- 172.21.15.29:0/3288566833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000094d0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.948+0000 7f6b15682640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 0x7f6b100b1030 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6b0002ba50 tx=0x7f6b00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.949+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00037c40 con 0x7f6b1015cbe0 2023-10-16T14:33:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.949+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b00037da0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.949+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b000426c0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.949+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b100b15d0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.949+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b100ca110 con 0x7f6b1015cbe0 2023-10-16T14:33:06.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.950+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1004e9b0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.951+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b0002d3c0 con 0x7f6b1015cbe0 2023-10-16T14:33:06.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.952+0000 7f6b067fc640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6af406af90 0x7f6af406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:06.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.952+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b0003c070 con 0x7f6b1015cbe0 2023-10-16T14:33:06.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.953+0000 7f6b15e83640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6af406af90 0x7f6af406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:06.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.954+0000 7f6b15e83640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6af406af90 0x7f6af406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009ad0 tx=0x7f6b0c008040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:06.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:06.956+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6b00076b60 con 0x7f6b1015cbe0 2023-10-16T14:33:07.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.155+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b100b1570 con 0x7f6b1015cbe0 2023-10-16T14:33:07.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.158+0000 7f6b067fc640 1 -- 172.21.15.29:0/3288566833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16581 (secure 0 0 0) 0x7f6b00047020 con 0x7f6b1015cbe0 2023-10-16T14:33:07.161 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:07.161 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":96,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:05.722622+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:07.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.160+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6af406af90 msgr2=0x7f6af406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.160+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6af406af90 0x7f6af406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009ad0 tx=0x7f6b0c008040 comp rx=0 tx=0).stop 2023-10-16T14:33:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.160+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 msgr2=0x7f6b100b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.160+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 0x7f6b100b1030 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6b0002ba50 tx=0x7f6b00004730 comp rx=0 tx=0).stop 2023-10-16T14:33:07.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 shutdown_connections 2023-10-16T14:33:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6af406af90 0x7f6af406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015cbe0 0x7f6b100b1030 secure :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6b0002ba50 tx=0x7f6b00004730 comp rx=0 tx=0).stop 2023-10-16T14:33:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b1015cfc0 0x7f6b101f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 --2- 172.21.15.29:0/3288566833 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b100d1090 0x7f6b100aaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 >> 172.21.15.29:0/3288566833 conn(0x7f6b100cf570 msgr2=0x7f6b1015c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:07.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 shutdown_connections 2023-10-16T14:33:07.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.161+0000 7f6b16e85640 1 -- 172.21.15.29:0/3288566833 wait complete. 2023-10-16T14:33:07.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:33:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.413+0000 7f0f2ed87640 1 Processor -- start 2023-10-16T14:33:07.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.413+0000 7f0f2ed87640 1 -- start start 2023-10-16T14:33:07.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.414+0000 7f0f2ed87640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f280ad7a0 0x7f0f28070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.414+0000 7f0f2ed87640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f280ae1a0 con 0x7f0f280adc30 2023-10-16T14:33:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.414+0000 7f0f2ed87640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f280ae480 con 0x7f0f28160c80 2023-10-16T14:33:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.414+0000 7f0f2ed87640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f280ae760 con 0x7f0f280ad7a0 2023-10-16T14:33:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2d584640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f280ad7a0 0x7f0f28070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:07.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2d584640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f280ad7a0 0x7f0f28070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53980/0 (socket says 172.21.15.29:53980) 2023-10-16T14:33:07.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2e586640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0f280adc30 0x7f0f28140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36776/0 (socket says 172.21.15.29:36776) 2023-10-16T14:33:07.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2d584640 1 -- 172.21.15.29:0/2934805308 learned_addr learned my addr 172.21.15.29:0/2934805308 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2d584640 1 -- 172.21.15.29:0/2934805308 >> v1:172.21.15.170:6789/0 conn(0x7f0f28160c80 legacy=0x7f0f28161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2d584640 1 -- 172.21.15.29:0/2934805308 >> v1:172.21.15.29:6789/0 conn(0x7f0f280adc30 legacy=0x7f0f28140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.415+0000 7f0f2d584640 1 -- 172.21.15.29:0/2934805308 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280aea40 con 0x7f0f280ad7a0 2023-10-16T14:33:07.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.416+0000 7f0f2d584640 1 --2- 172.21.15.29:0/2934805308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f28070240 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009870 tx=0x7f0f1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fd12094f6428a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:07.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.417+0000 7f0f2cd83640 1 -- 172.21.15.29:0/2934805308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18030860 con 0x7f0f280ad7a0 2023-10-16T14:33:07.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.417+0000 7f0f2cd83640 1 -- 172.21.15.29:0/2934805308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f180309c0 con 0x7f0f280ad7a0 2023-10-16T14:33:07.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.417+0000 7f0f2cd83640 1 -- 172.21.15.29:0/2934805308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18030d50 con 0x7f0f280ad7a0 2023-10-16T14:33:07.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.417+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2934805308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 msgr2=0x7f0f28070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:07.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.417+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2934805308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f28070240 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009870 tx=0x7f0f1802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.418+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2934805308 shutdown_connections 2023-10-16T14:33:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.418+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2934805308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f28070240 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.418+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2934805308 >> 172.21.15.29:0/2934805308 conn(0x7f0f280cc230 msgr2=0x7f0f280cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.418+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2934805308 shutdown_connections 2023-10-16T14:33:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.418+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2934805308 wait complete. 2023-10-16T14:33:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.419+0000 7f0f2ed87640 1 Processor -- start 2023-10-16T14:33:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.420+0000 7f0f2ed87640 1 -- start start 2023-10-16T14:33:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.420+0000 7f0f2ed87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f280c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.420+0000 7f0f2ed87640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f280adc30 0x7f0f280c2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.420+0000 7f0f2ed87640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f28160c80 0x7f0f281f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.420+0000 7f0f2ed87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280c4180 con 0x7f0f280ad7a0 2023-10-16T14:33:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.420+0000 7f0f2ed87640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f28142480 con 0x7f0f280adc30 2023-10-16T14:33:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2ed87640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f28142760 con 0x7f0f28160c80 2023-10-16T14:33:07.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f280c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f280c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53996/0 (socket says 172.21.15.29:53996) 2023-10-16T14:33:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 -- 172.21.15.29:0/2285390239 learned_addr learned my addr 172.21.15.29:0/2285390239 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2d584640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f280adc30 0x7f0f280c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 -- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f28160c80 msgr2=0x7f0f281f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f28160c80 0x7f0f281f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 -- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f280adc30 msgr2=0x7f0f280c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f280adc30 0x7f0f280c2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.421+0000 7f0f2dd85640 1 -- 172.21.15.29:0/2285390239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180094d0 con 0x7f0f280ad7a0 2023-10-16T14:33:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.422+0000 7f0f2dd85640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f280c3c40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400ba50 tx=0x7f0f2400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.422+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2400cbe0 con 0x7f0f280ad7a0 2023-10-16T14:33:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.422+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2400cd80 con 0x7f0f280ad7a0 2023-10-16T14:33:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.422+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24014680 con 0x7f0f280ad7a0 2023-10-16T14:33:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.422+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f28142a40 con 0x7f0f280ad7a0 2023-10-16T14:33:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.422+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f28143000 con 0x7f0f280ad7a0 2023-10-16T14:33:07.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.423+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2804e9b0 con 0x7f0f280ad7a0 2023-10-16T14:33:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.424+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0f240147e0 con 0x7f0f280ad7a0 2023-10-16T14:33:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.425+0000 7f0f1e7fc640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0f0c06af90 0x7f0f0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.425+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f24089660 con 0x7f0f280ad7a0 2023-10-16T14:33:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.426+0000 7f0f2d584640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0f0c06af90 0x7f0f0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:07.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.427+0000 7f0f2d584640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0f0c06af90 0x7f0f0c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0f18036000 tx=0x7f0f18008030 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:07.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.430+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0f24054610 con 0x7f0f280ad7a0 2023-10-16T14:33:07.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:07.662+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0f280b5730 con 0x7f0f280ad7a0 2023-10-16T14:33:08.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.277+0000 7f0f1e7fc640 1 -- 172.21.15.29:0/2285390239 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0f24057c30 con 0x7f0f280ad7a0 2023-10-16T14:33:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.280+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0f0c06af90 msgr2=0x7f0f0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.280+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0f0c06af90 0x7f0f0c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0f18036000 tx=0x7f0f18008030 comp rx=0 tx=0).stop 2023-10-16T14:33:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.280+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 msgr2=0x7f0f280c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:08.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.280+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f280c3c40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400ba50 tx=0x7f0f2400bf10 comp rx=0 tx=0).stop 2023-10-16T14:33:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 shutdown_connections 2023-10-16T14:33:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f28160c80 0x7f0f281f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f280adc30 0x7f0f280c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0f0c06af90 0x7f0f0c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 --2- 172.21.15.29:0/2285390239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f280ad7a0 0x7f0f280c3c40 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 >> 172.21.15.29:0/2285390239 conn(0x7f0f280cc230 msgr2=0x7f0f2815d660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 shutdown_connections 2023-10-16T14:33:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.281+0000 7f0f2ed87640 1 -- 172.21.15.29:0/2285390239 wait complete. 2023-10-16T14:33:08.306 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:33:08.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:33:08.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.537+0000 7f6ad3675640 1 Processor -- start 2023-10-16T14:33:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.538+0000 7f6ad3675640 1 -- start start 2023-10-16T14:33:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.539+0000 7f6ad3675640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.539+0000 7f6ad3675640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.539+0000 7f6ad3675640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc0cb030 con 0x7f6acc15ad70 2023-10-16T14:33:08.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.539+0000 7f6ad3675640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc0cae30 con 0x7f6acc06a4b0 2023-10-16T14:33:08.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.539+0000 7f6ad3675640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c91c0 con 0x7f6acc0c9ca0 2023-10-16T14:33:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.540+0000 7f6ad1411640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f6acc15ad70 0x7f6acc069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:33598/0 (socket says 172.21.15.29:33598) 2023-10-16T14:33:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.540+0000 7f6ad1411640 1 -- 172.21.15.29:0/839434233 learned_addr learned my addr 172.21.15.29:0/839434233 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.540+0000 7f6ad1c12640 1 --2- 172.21.15.29:0/839434233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.540+0000 7f6ad0c10640 1 --2- 172.21.15.29:0/839434233 >> v2:172.21.15.170:3300/0 conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ac3fff640 1 -- 172.21.15.29:0/839434233 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973193019 0 0) 0x7f6acc0cb030 con 0x7f6acc15ad70 2023-10-16T14:33:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ac3fff640 1 -- 172.21.15.29:0/839434233 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab8003520 con 0x7f6acc15ad70 2023-10-16T14:33:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ad1c12640 1 -- 172.21.15.29:0/839434233 >> v2:172.21.15.170:3300/0 conn(0x7f6acc06a4b0 msgr2=0x7f6acc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ad1c12640 1 --2- 172.21.15.29:0/839434233 >> v2:172.21.15.170:3300/0 conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ad1c12640 1 -- 172.21.15.29:0/839434233 >> v1:172.21.15.170:6789/0 conn(0x7f6acc15ad70 legacy=0x7f6acc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ad1c12640 1 -- 172.21.15.29:0/839434233 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0c94a0 con 0x7f6acc0c9ca0 2023-10-16T14:33:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.541+0000 7f6ad0c10640 1 --2- 172.21.15.29:0/839434233 >> v2:172.21.15.170:3300/0 conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:33:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.542+0000 7f6ad1c12640 1 --2- 172.21.15.29:0/839434233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40097a0 tx=0x7f6ac402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e791534e4daf34a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.543+0000 7f6ac3fff640 1 -- 172.21.15.29:0/839434233 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac402d040 con 0x7f6acc0c9ca0 2023-10-16T14:33:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.543+0000 7f6ac3fff640 1 -- 172.21.15.29:0/839434233 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac4031810 con 0x7f6acc0c9ca0 2023-10-16T14:33:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.543+0000 7f6ac3fff640 1 -- 172.21.15.29:0/839434233 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4031ba0 con 0x7f6acc0c9ca0 2023-10-16T14:33:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.543+0000 7f6ad3675640 1 -- 172.21.15.29:0/839434233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 msgr2=0x7f6acc0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.543+0000 7f6ad3675640 1 --2- 172.21.15.29:0/839434233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40097a0 tx=0x7f6ac402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.544+0000 7f6ad3675640 1 -- 172.21.15.29:0/839434233 shutdown_connections 2023-10-16T14:33:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.544+0000 7f6ad3675640 1 --2- 172.21.15.29:0/839434233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.544+0000 7f6ad3675640 1 --2- 172.21.15.29:0/839434233 >> v2:172.21.15.170:3300/0 conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.544+0000 7f6ad3675640 1 -- 172.21.15.29:0/839434233 >> 172.21.15.29:0/839434233 conn(0x7f6acc0ce9b0 msgr2=0x7f6acc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.544+0000 7f6ad3675640 1 -- 172.21.15.29:0/839434233 shutdown_connections 2023-10-16T14:33:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.544+0000 7f6ad3675640 1 -- 172.21.15.29:0/839434233 wait complete. 2023-10-16T14:33:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.545+0000 7f6ad3675640 1 Processor -- start 2023-10-16T14:33:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.546+0000 7f6ad3675640 1 -- start start 2023-10-16T14:33:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.546+0000 7f6ad3675640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc06a4b0 0x7f6acc0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad3675640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad3675640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 0x7f6acc0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad3675640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc14b6e0 con 0x7f6acc15ad70 2023-10-16T14:33:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad3675640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c68f0 con 0x7f6acc06a4b0 2023-10-16T14:33:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad3675640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c8800 con 0x7f6acc0c9ca0 2023-10-16T14:33:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 0x7f6acc0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 0x7f6acc0b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54000/0 (socket says 172.21.15.29:54000) 2023-10-16T14:33:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 -- 172.21.15.29:0/3288768932 learned_addr learned my addr 172.21.15.29:0/3288768932 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1411640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc06a4b0 0x7f6acc0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad0c10640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 -- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 msgr2=0x7f6acc0b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 -- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc06a4b0 msgr2=0x7f6acc0c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc06a4b0 0x7f6acc0c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.547+0000 7f6ad1c12640 1 -- 172.21.15.29:0/3288768932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc009400 con 0x7f6acc15ad70 2023-10-16T14:33:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.548+0000 7f6ad1c12640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 0x7f6acc0b2620 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6ac402b8f0 tx=0x7f6ac40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.548+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac403ace0 con 0x7f6acc15ad70 2023-10-16T14:33:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.548+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac403ae60 con 0x7f6acc15ad70 2023-10-16T14:33:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.548+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac40443f0 con 0x7f6acc15ad70 2023-10-16T14:33:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.548+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac4009400 con 0x7f6acc15ad70 2023-10-16T14:33:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.548+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc0c8d30 con 0x7f6acc15ad70 2023-10-16T14:33:08.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.549+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc04e160 con 0x7f6acc15ad70 2023-10-16T14:33:08.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.550+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ac4044550 con 0x7f6acc15ad70 2023-10-16T14:33:08.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.551+0000 7f6ac1ffb640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6ab006af90 0x7f6ab006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:08.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.551+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ac40aba60 con 0x7f6acc15ad70 2023-10-16T14:33:08.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.552+0000 7f6ad1411640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6ab006af90 0x7f6ab006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:08.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.553+0000 7f6ad1411640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6ab006af90 0x7f6ab006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8001560 tx=0x7f6ac8009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:08.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.556+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ac40768e0 con 0x7f6acc15ad70 2023-10-16T14:33:08.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:08.789+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6acc069ed0 con 0x7f6acc15ad70 2023-10-16T14:33:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.287+0000 7f6ac1ffb640 1 -- 172.21.15.29:0/3288768932 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ac4035020 con 0x7f6acc15ad70 2023-10-16T14:33:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6ab006af90 msgr2=0x7f6ab006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6ab006af90 0x7f6ab006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8001560 tx=0x7f6ac8009040 comp rx=0 tx=0).stop 2023-10-16T14:33:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 msgr2=0x7f6acc0b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 0x7f6acc0b2620 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6ac402b8f0 tx=0x7f6ac40046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 shutdown_connections 2023-10-16T14:33:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f6ab006af90 0x7f6ab006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc15ad70 0x7f6acc0b2620 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6acc0c9ca0 0x7f6acc0b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 --2- 172.21.15.29:0/3288768932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc06a4b0 0x7f6acc0c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.291+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 >> 172.21.15.29:0/3288768932 conn(0x7f6acc0ce9b0 msgr2=0x7f6acc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.292+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 shutdown_connections 2023-10-16T14:33:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.292+0000 7f6ad3675640 1 -- 172.21.15.29:0/3288768932 wait complete. 2023-10-16T14:33:09.315 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:33:09.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.538+0000 7fb5f28e5640 1 Processor -- start 2023-10-16T14:33:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.539+0000 7fb5f28e5640 1 -- start start 2023-10-16T14:33:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.540+0000 7fb5f28e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5ec0cd0c0 0x7fb5ec1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.540+0000 7fb5f28e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0b0410 0x7fb5ec0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.540+0000 7fb5f28e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec0b0db0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.540+0000 7fb5f28e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ec15e3e0 con 0x7fb5ec0cd0c0 2023-10-16T14:33:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.540+0000 7fb5f28e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0cd4a0 con 0x7fb5ec0b0410 2023-10-16T14:33:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5ebfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5ec0cd0c0 0x7fb5ec1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5ebfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5ec0cd0c0 0x7fb5ec1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54006/0 (socket says 172.21.15.29:54006) 2023-10-16T14:33:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5ebfff640 1 -- 172.21.15.29:0/293136679 learned_addr learned my addr 172.21.15.29:0/293136679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5e3fff640 1 --2- 172.21.15.29:0/293136679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0b0410 0x7fb5ec0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5ebfff640 1 -- 172.21.15.29:0/293136679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0b0410 msgr2=0x7fb5ec0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5ebfff640 1 --2- 172.21.15.29:0/293136679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0b0410 0x7fb5ec0b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.541+0000 7fb5ebfff640 1 -- 172.21.15.29:0/293136679 >> v1:172.21.15.170:6789/0 conn(0x7fb5ec15d3f0 legacy=0x7fb5ec15d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.542+0000 7fb5ebfff640 1 -- 172.21.15.29:0/293136679 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec15e100 con 0x7fb5ec0cd0c0 2023-10-16T14:33:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.543+0000 7fb5ebfff640 1 --2- 172.21.15.29:0/293136679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0cd0c0 0x7fb5ec1615f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009870 tx=0x7fb5dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f7e8de2dd6a418 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5eb7fe640 1 -- 172.21.15.29:0/293136679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc031640 con 0x7fb5ec0cd0c0 2023-10-16T14:33:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5eb7fe640 1 -- 172.21.15.29:0/293136679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc0317a0 con 0x7fb5ec0cd0c0 2023-10-16T14:33:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5eb7fe640 1 -- 172.21.15.29:0/293136679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc031b30 con 0x7fb5ec0cd0c0 2023-10-16T14:33:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 -- 172.21.15.29:0/293136679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0cd0c0 msgr2=0x7fb5ec1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/293136679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0cd0c0 0x7fb5ec1615f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009870 tx=0x7fb5dc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 -- 172.21.15.29:0/293136679 shutdown_connections 2023-10-16T14:33:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/293136679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0b0410 0x7fb5ec0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/293136679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec0cd0c0 0x7fb5ec1615f0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 -- 172.21.15.29:0/293136679 >> 172.21.15.29:0/293136679 conn(0x7fb5ec0cb410 msgr2=0x7fb5ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 -- 172.21.15.29:0/293136679 shutdown_connections 2023-10-16T14:33:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.544+0000 7fb5f28e5640 1 -- 172.21.15.29:0/293136679 wait complete. 2023-10-16T14:33:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.546+0000 7fb5f28e5640 1 Processor -- start 2023-10-16T14:33:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.546+0000 7fb5f28e5640 1 -- start start 2023-10-16T14:33:09.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.547+0000 7fb5f28e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ec0b0410 0x7fb5ec146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.547+0000 7fb5f28e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0cd0c0 0x7fb5ec0aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.547+0000 7fb5f28e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 0x7fb5ec0af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f28e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec147260 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f28e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0b40a0 con 0x7fb5ec0b0410 2023-10-16T14:33:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f28e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0b4380 con 0x7fb5ec0cd0c0 2023-10-16T14:33:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 0x7fb5ec0af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 0x7fb5ec0af310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54008/0 (socket says 172.21.15.29:54008) 2023-10-16T14:33:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 -- 172.21.15.29:0/3253221696 learned_addr learned my addr 172.21.15.29:0/3253221696 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5ebfff640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ec0b0410 0x7fb5ec146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5e3fff640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0cd0c0 0x7fb5ec0aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 -- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0cd0c0 msgr2=0x7fb5ec0aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0cd0c0 0x7fb5ec0aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 -- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ec0b0410 msgr2=0x7fb5ec146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ec0b0410 0x7fb5ec146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5f0e82640 1 -- 172.21.15.29:0/3253221696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc0094d0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.548+0000 7fb5e3fff640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0cd0c0 0x7fb5ec0aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:09.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.549+0000 7fb5f0e82640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 0x7fb5ec0af310 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400c7a0 tx=0x7fb5e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:09.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.549+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4003b70 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.549+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4003cf0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.549+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400aa90 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.549+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ec146ba0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.549+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ec1467a0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.550+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ec1473a0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.552+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5e400abf0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.552+0000 7fb5e97fa640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb5cc06af90 0x7fb5cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:09.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.553+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5e4089680 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.553+0000 7fb5ebfff640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb5cc06af90 0x7fb5cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.554+0000 7fb5ebfff640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb5cc06af90 0x7fb5cc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc036000 tx=0x7fb5dc03c040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.558+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb5ec1473a0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.790+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5ec1473a0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.792+0000 7fb5e97fa640 1 -- 172.21.15.29:0/3253221696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fb5ec1473a0 con 0x7fb5ec15d3f0 2023-10-16T14:33:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:09.796 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:05.723125+0000","modified":"2023-10-16T14:33:08.790919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5926":{"gid":5926,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3812375996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3812375996},{"type":"v1","addr":"172.21.15.170:6837","nonce":3812375996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T14:33:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-16T14:33:09.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb5cc06af90 msgr2=0x7fb5cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb5cc06af90 0x7fb5cc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc036000 tx=0x7fb5dc03c040 comp rx=0 tx=0).stop 2023-10-16T14:33:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 msgr2=0x7fb5ec0af310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 0x7fb5ec0af310 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400c7a0 tx=0x7fb5e400cc60 comp rx=0 tx=0).stop 2023-10-16T14:33:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 shutdown_connections 2023-10-16T14:33:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fb5cc06af90 0x7fb5cc06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5ec15d3f0 0x7fb5ec0af310 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5ec0cd0c0 0x7fb5ec0aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 --2- 172.21.15.29:0/3253221696 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5ec0b0410 0x7fb5ec146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.794+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 >> 172.21.15.29:0/3253221696 conn(0x7fb5ec0cb410 msgr2=0x7fb5ec15c820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.795+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 shutdown_connections 2023-10-16T14:33:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:09.795+0000 7fb5f28e5640 1 -- 172.21.15.29:0/3253221696 wait complete. 2023-10-16T14:33:09.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:10.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.037+0000 7fade4e56640 1 Processor -- start 2023-10-16T14:33:10.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.037+0000 7fade4e56640 1 -- start start 2023-10-16T14:33:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.038+0000 7fade4e56640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fade015ae80 0x7fade015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.038+0000 7fade4e56640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fade00ce390 0x7fade0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.038+0000 7fade4e56640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade015ed60 con 0x7fade015aaa0 2023-10-16T14:33:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.038+0000 7fade4e56640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade006b810 con 0x7fade015ae80 2023-10-16T14:33:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.038+0000 7fade4e56640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00707e0 con 0x7fade00ce390 2023-10-16T14:33:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fade015ae80 0x7fade015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fade015ae80 0x7fade015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54020/0 (socket says 172.21.15.29:54020) 2023-10-16T14:33:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2861816942 learned_addr learned my addr 172.21.15.29:0/2861816942 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:10.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadded9d640 1 --2- 172.21.15.29:0/2861816942 >> v2:172.21.15.170:3300/0 conn(0x7fade00ce390 0x7fade0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2861816942 >> v2:172.21.15.170:3300/0 conn(0x7fade00ce390 msgr2=0x7fade0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2861816942 >> v2:172.21.15.170:3300/0 conn(0x7fade00ce390 0x7fade0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2861816942 >> v1:172.21.15.170:6789/0 conn(0x7fade015aaa0 legacy=0x7fade00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadd5d9b640 1 -- 172.21.15.29:0/2861816942 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade015ef90 con 0x7fade015ae80 2023-10-16T14:33:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.039+0000 7fadded9d640 1 --2- 172.21.15.29:0/2861816942 >> v2:172.21.15.170:3300/0 conn(0x7fade00ce390 0x7fade0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.041+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2861816942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade015ae80 0x7fade015e820 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fade00c2490 tx=0x7fadc802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46257e7d76b9539 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.041+0000 7fadddd9b640 1 -- 172.21.15.29:0/2861816942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8030860 con 0x7fade015ae80 2023-10-16T14:33:10.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.041+0000 7fadddd9b640 1 -- 172.21.15.29:0/2861816942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc80309c0 con 0x7fade015ae80 2023-10-16T14:33:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.041+0000 7fadddd9b640 1 -- 172.21.15.29:0/2861816942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8030d50 con 0x7fade015ae80 2023-10-16T14:33:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 -- 172.21.15.29:0/2861816942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade015ae80 msgr2=0x7fade015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 --2- 172.21.15.29:0/2861816942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade015ae80 0x7fade015e820 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fade00c2490 tx=0x7fadc802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 -- 172.21.15.29:0/2861816942 shutdown_connections 2023-10-16T14:33:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 --2- 172.21.15.29:0/2861816942 >> v2:172.21.15.170:3300/0 conn(0x7fade00ce390 0x7fade0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 --2- 172.21.15.29:0/2861816942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade015ae80 0x7fade015e820 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 -- 172.21.15.29:0/2861816942 >> 172.21.15.29:0/2861816942 conn(0x7fade00cb410 msgr2=0x7fade00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 -- 172.21.15.29:0/2861816942 shutdown_connections 2023-10-16T14:33:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.042+0000 7fade4e56640 1 -- 172.21.15.29:0/2861816942 wait complete. 2023-10-16T14:33:10.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.044+0000 7fade4e56640 1 Processor -- start 2023-10-16T14:33:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.044+0000 7fade4e56640 1 -- start start 2023-10-16T14:33:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fade4e56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 0x7fade00c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fade4e56640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fade015aaa0 0x7fade00c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fade4e56640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fade015ae80 0x7fade01f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fade4e56640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c9160 con 0x7fade00ce390 2023-10-16T14:33:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fade4e56640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade015d210 con 0x7fade015aaa0 2023-10-16T14:33:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fade4e56640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade015d4f0 con 0x7fade015ae80 2023-10-16T14:33:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 0x7fade00c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 0x7fade00c5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54022/0 (socket says 172.21.15.29:54022) 2023-10-16T14:33:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 -- 172.21.15.29:0/2076748758 learned_addr learned my addr 172.21.15.29:0/2076748758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:10.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 -- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fade015ae80 msgr2=0x7fade01f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fade015ae80 0x7fade01f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 -- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fade015aaa0 msgr2=0x7fade00c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fade015aaa0 0x7fade00c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fade015aaa0 0x7fade00c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadde59c640 1 -- 172.21.15.29:0/2076748758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc80094d0 con 0x7fade00ce390 2023-10-16T14:33:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.045+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fade015aaa0 0x7fade00c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.046+0000 7fadde59c640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 0x7fade00c5d70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fadd000c450 tx=0x7fadd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.046+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00052d0 con 0x7fade00ce390 2023-10-16T14:33:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.046+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0005430 con 0x7fade00ce390 2023-10-16T14:33:10.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.046+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0012870 con 0x7fade00ce390 2023-10-16T14:33:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.046+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade015d7d0 con 0x7fade00ce390 2023-10-16T14:33:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.046+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade00c8b20 con 0x7fade00ce390 2023-10-16T14:33:10.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.047+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade00c8d10 con 0x7fade00ce390 2023-10-16T14:33:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.048+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fadd00129d0 con 0x7fade00ce390 2023-10-16T14:33:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.049+0000 7fadd77fe640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fadc006af90 0x7fadc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.049+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadd0089370 con 0x7fade00ce390 2023-10-16T14:33:10.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.050+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fadc006af90 0x7fadc006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.051+0000 7fadd5d9b640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fadc006af90 0x7fadc006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fadc8036000 tx=0x7fadc803b040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.053+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadd0054270 con 0x7fade00ce390 2023-10-16T14:33:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.252+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fade015dcf0 con 0x7fade00ce390 2023-10-16T14:33:10.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.255+0000 7fadd77fe640 1 -- 172.21.15.29:0/2076748758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16568 (secure 0 0 0) 0x7fadd0057890 con 0x7fade00ce390 2023-10-16T14:33:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:10.259 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":97,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:07.241914+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fadc006af90 msgr2=0x7fadc006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fadc006af90 0x7fadc006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fadc8036000 tx=0x7fadc803b040 comp rx=0 tx=0).stop 2023-10-16T14:33:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 msgr2=0x7fade00c5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 0x7fade00c5d70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fadd000c450 tx=0x7fadd000c910 comp rx=0 tx=0).stop 2023-10-16T14:33:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 shutdown_connections 2023-10-16T14:33:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fade015ae80 0x7fade01f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fade015aaa0 0x7fade00c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7fadc006af90 0x7fadc006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 --2- 172.21.15.29:0/2076748758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00ce390 0x7fade00c5d70 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 >> 172.21.15.29:0/2076748758 conn(0x7fade00cb410 msgr2=0x7fade00b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 shutdown_connections 2023-10-16T14:33:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.258+0000 7fade4e56640 1 -- 172.21.15.29:0/2076748758 wait complete. 2023-10-16T14:33:10.282 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:33:10.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:33:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.503+0000 7ff442613640 1 Processor -- start 2023-10-16T14:33:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.503+0000 7ff442613640 1 -- start start 2023-10-16T14:33:10.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.504+0000 7ff442613640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0b8480 0x7ff43c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.504+0000 7ff442613640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff43c0d0be0 0x7ff43c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff442613640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c15d150 con 0x7ff43c15cd70 2023-10-16T14:33:10.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff442613640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0d1180 con 0x7ff43c0d0be0 2023-10-16T14:33:10.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff442613640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0d1460 con 0x7ff43c0b8480 2023-10-16T14:33:10.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff441611640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff43c15cd70 0x7ff43c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36780/0 (socket says 172.21.15.29:36780) 2023-10-16T14:33:10.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff441611640 1 -- 172.21.15.29:0/1842325856 learned_addr learned my addr 172.21.15.29:0/1842325856 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:10.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff441e12640 1 --2- 172.21.15.29:0/1842325856 >> v2:172.21.15.29:3300/0 conn(0x7ff43c0d0be0 0x7ff43c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff440e10640 1 --2- 172.21.15.29:0/1842325856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0b8480 0x7ff43c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.505+0000 7ff433fff640 1 -- 172.21.15.29:0/1842325856 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185839651 0 0) 0x7ff43c15d150 con 0x7ff43c15cd70 2023-10-16T14:33:10.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.506+0000 7ff433fff640 1 -- 172.21.15.29:0/1842325856 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff428003520 con 0x7ff43c15cd70 2023-10-16T14:33:10.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.506+0000 7ff441e12640 1 -- 172.21.15.29:0/1842325856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0b8480 msgr2=0x7ff43c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:10.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.506+0000 7ff441e12640 1 --2- 172.21.15.29:0/1842325856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0b8480 0x7ff43c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.506+0000 7ff441e12640 1 -- 172.21.15.29:0/1842325856 >> v1:172.21.15.29:6789/0 conn(0x7ff43c15cd70 legacy=0x7ff43c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.506+0000 7ff441e12640 1 -- 172.21.15.29:0/1842325856 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c0d1740 con 0x7ff43c0d0be0 2023-10-16T14:33:10.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.507+0000 7ff441e12640 1 --2- 172.21.15.29:0/1842325856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0d0be0 0x7ff43c0cdd00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7ff434009d90 tx=0x7ff43402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17baa94f60d05d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.507+0000 7ff433fff640 1 -- 172.21.15.29:0/1842325856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43402cc60 con 0x7ff43c0d0be0 2023-10-16T14:33:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.507+0000 7ff433fff640 1 -- 172.21.15.29:0/1842325856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43402cdc0 con 0x7ff43c0d0be0 2023-10-16T14:33:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.507+0000 7ff433fff640 1 -- 172.21.15.29:0/1842325856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434034660 con 0x7ff43c0d0be0 2023-10-16T14:33:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 -- 172.21.15.29:0/1842325856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0d0be0 msgr2=0x7ff43c0cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 --2- 172.21.15.29:0/1842325856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0d0be0 0x7ff43c0cdd00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7ff434009d90 tx=0x7ff43402beb0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 -- 172.21.15.29:0/1842325856 shutdown_connections 2023-10-16T14:33:10.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 --2- 172.21.15.29:0/1842325856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0d0be0 0x7ff43c0cdd00 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 --2- 172.21.15.29:0/1842325856 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0b8480 0x7ff43c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 -- 172.21.15.29:0/1842325856 >> 172.21.15.29:0/1842325856 conn(0x7ff43c0cf570 msgr2=0x7ff43c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:10.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 -- 172.21.15.29:0/1842325856 shutdown_connections 2023-10-16T14:33:10.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.508+0000 7ff442613640 1 -- 172.21.15.29:0/1842325856 wait complete. 2023-10-16T14:33:10.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.510+0000 7ff442613640 1 Processor -- start 2023-10-16T14:33:10.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.510+0000 7ff442613640 1 -- start start 2023-10-16T14:33:10.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff442613640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff43c0b8480 0x7ff43c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff442613640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0d0be0 0x7ff43c150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff442613640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 0x7ff43c1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff442613640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c1f64c0 con 0x7ff43c15cd70 2023-10-16T14:33:10.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff442613640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0cb110 con 0x7ff43c0b8480 2023-10-16T14:33:10.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff442613640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0cb3f0 con 0x7ff43c0d0be0 2023-10-16T14:33:10.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 0x7ff43c1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 0x7ff43c1511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54044/0 (socket says 172.21.15.29:54044) 2023-10-16T14:33:10.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 -- 172.21.15.29:0/248739118 learned_addr learned my addr 172.21.15.29:0/248739118 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:10.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 -- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0d0be0 msgr2=0x7ff43c150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0d0be0 0x7ff43c150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 -- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff43c0b8480 msgr2=0x7ff43c15a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff43c0b8480 0x7ff43c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.511+0000 7ff441e12640 1 -- 172.21.15.29:0/248739118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff434009a70 con 0x7ff43c15cd70 2023-10-16T14:33:10.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.512+0000 7ff441e12640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 0x7ff43c1511e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff434009d90 tx=0x7ff434004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.512+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434040650 con 0x7ff43c15cd70 2023-10-16T14:33:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.512+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4340407b0 con 0x7ff43c15cd70 2023-10-16T14:33:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.512+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c0cb6d0 con 0x7ff43c15cd70 2023-10-16T14:33:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.512+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434040b60 con 0x7ff43c15cd70 2023-10-16T14:33:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.512+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c15af20 con 0x7ff43c15cd70 2023-10-16T14:33:10.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.513+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c15b0b0 con 0x7ff43c15cd70 2023-10-16T14:33:10.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.514+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff434040cc0 con 0x7ff43c15cd70 2023-10-16T14:33:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.515+0000 7ff431ffb640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff42006af90 0x7ff42006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.515+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4340ac470 con 0x7ff43c15cd70 2023-10-16T14:33:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.516+0000 7ff441611640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff42006af90 0x7ff42006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.517+0000 7ff441611640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff42006af90 0x7ff42006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff438001610 tx=0x7ff438007930 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.520+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff434077410 con 0x7ff43c15cd70 2023-10-16T14:33:10.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.711+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 --> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff43c0d1d00 con 0x7ff42006af90 2023-10-16T14:33:10.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.740+0000 7ff431ffb640 1 -- 172.21.15.29:0/248739118 <== mgr.5390 v2:172.21.15.170:6832/610139797 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286667 (secure 0 0 0) 0x7ff4380058b0 con 0x7ff42006af90 2023-10-16T14:33:10.748 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:10.749 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:33:10.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff42006af90 msgr2=0x7ff42006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff42006af90 0x7ff42006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff438001610 tx=0x7ff438007930 comp rx=0 tx=0).stop 2023-10-16T14:33:10.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 msgr2=0x7ff43c1511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:10.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 0x7ff43c1511e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff434009d90 tx=0x7ff434004a40 comp rx=0 tx=0).stop 2023-10-16T14:33:10.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 shutdown_connections 2023-10-16T14:33:10.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff42006af90 0x7ff42006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c15cd70 0x7ff43c1511e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff43c0d0be0 0x7ff43c150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 --2- 172.21.15.29:0/248739118 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff43c0b8480 0x7ff43c15a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 >> 172.21.15.29:0/248739118 conn(0x7ff43c0cf570 msgr2=0x7ff43c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 shutdown_connections 2023-10-16T14:33:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:10.746+0000 7ff442613640 1 -- 172.21.15.29:0/248739118 wait complete. 2023-10-16T14:33:10.768 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T14:33:09.570350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":307,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337600,"kb_used_data":10448,"kb_used_omap":12,"kb_used_meta":327027,"kb_avail":754637120,"statfs":{"total":773094113280,"available":772748410880,"internally_reserved":0,"allocated":10698752,"data_stored":5724572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":334876217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.342752"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254163+0000","last_change":"2023-10-16T14:33:06.280994+0000","last_active":"2023-10-16T14:33:07.254163+0000","last_peered":"2023-10-16T14:33:07.254163+0000","last_clean":"2023-10-16T14:33:07.254163+0000","last_became_active":"2023-10-16T14:33:06.280696+0000","last_became_peered":"2023-10-16T14:33:06.280696+0000","last_unstale":"2023-10-16T14:33:07.254163+0000","last_undegraded":"2023-10-16T14:33:07.254163+0000","last_fullsized":"2023-10-16T14:33:07.254163+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:50.049165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987178+0000","last_change":"2023-10-16T14:33:06.286651+0000","last_active":"2023-10-16T14:33:07.987178+0000","last_peered":"2023-10-16T14:33:07.987178+0000","last_clean":"2023-10-16T14:33:07.987178+0000","last_became_active":"2023-10-16T14:33:06.286052+0000","last_became_peered":"2023-10-16T14:33:06.286052+0000","last_unstale":"2023-10-16T14:33:07.987178+0000","last_undegraded":"2023-10-16T14:33:07.987178+0000","last_fullsized":"2023-10-16T14:33:07.987178+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:10.219830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988340+0000","last_change":"2023-10-16T14:33:06.277653+0000","last_active":"2023-10-16T14:33:07.988340+0000","last_peered":"2023-10-16T14:33:07.988340+0000","last_clean":"2023-10-16T14:33:07.988340+0000","last_became_active":"2023-10-16T14:33:06.274841+0000","last_became_peered":"2023-10-16T14:33:06.274841+0000","last_unstale":"2023-10-16T14:33:07.988340+0000","last_undegraded":"2023-10-16T14:33:07.988340+0000","last_fullsized":"2023-10-16T14:33:07.988340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:51.862361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985423+0000","last_change":"2023-10-16T14:33:06.289645+0000","last_active":"2023-10-16T14:33:07.985423+0000","last_peered":"2023-10-16T14:33:07.985423+0000","last_clean":"2023-10-16T14:33:07.985423+0000","last_became_active":"2023-10-16T14:33:06.288437+0000","last_became_peered":"2023-10-16T14:33:06.288437+0000","last_unstale":"2023-10-16T14:33:07.985423+0000","last_undegraded":"2023-10-16T14:33:07.985423+0000","last_fullsized":"2023-10-16T14:33:07.985423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:29.975227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989189+0000","last_change":"2023-10-16T14:33:06.285412+0000","last_active":"2023-10-16T14:33:07.989189+0000","last_peered":"2023-10-16T14:33:07.989189+0000","last_clean":"2023-10-16T14:33:07.989189+0000","last_became_active":"2023-10-16T14:33:06.284238+0000","last_became_peered":"2023-10-16T14:33:06.284238+0000","last_unstale":"2023-10-16T14:33:07.989189+0000","last_undegraded":"2023-10-16T14:33:07.989189+0000","last_fullsized":"2023-10-16T14:33:07.989189+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:29.975227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986095+0000","last_change":"2023-10-16T14:33:06.285477+0000","last_active":"2023-10-16T14:33:07.986095+0000","last_peered":"2023-10-16T14:33:07.986095+0000","last_clean":"2023-10-16T14:33:07.986095+0000","last_became_active":"2023-10-16T14:33:06.285115+0000","last_became_peered":"2023-10-16T14:33:06.285115+0000","last_unstale":"2023-10-16T14:33:07.986095+0000","last_undegraded":"2023-10-16T14:33:07.986095+0000","last_fullsized":"2023-10-16T14:33:07.986095+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:01.101300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985342+0000","last_change":"2023-10-16T14:33:06.275068+0000","last_active":"2023-10-16T14:33:07.985342+0000","last_peered":"2023-10-16T14:33:07.985342+0000","last_clean":"2023-10-16T14:33:07.985342+0000","last_became_active":"2023-10-16T14:33:06.274725+0000","last_became_peered":"2023-10-16T14:33:06.274725+0000","last_unstale":"2023-10-16T14:33:07.985342+0000","last_undegraded":"2023-10-16T14:33:07.985342+0000","last_fullsized":"2023-10-16T14:33:07.985342+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:42.540472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986525+0000","last_change":"2023-10-16T14:33:06.278379+0000","last_active":"2023-10-16T14:33:07.986525+0000","last_peered":"2023-10-16T14:33:07.986525+0000","last_clean":"2023-10-16T14:33:07.986525+0000","last_became_active":"2023-10-16T14:33:06.277921+0000","last_became_peered":"2023-10-16T14:33:06.277921+0000","last_unstale":"2023-10-16T14:33:07.986525+0000","last_undegraded":"2023-10-16T14:33:07.986525+0000","last_fullsized":"2023-10-16T14:33:07.986525+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:20.033542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989830+0000","last_change":"2023-10-16T14:33:06.286928+0000","last_active":"2023-10-16T14:33:07.989830+0000","last_peered":"2023-10-16T14:33:07.989830+0000","last_clean":"2023-10-16T14:33:07.989830+0000","last_became_active":"2023-10-16T14:33:06.285868+0000","last_became_peered":"2023-10-16T14:33:06.285868+0000","last_unstale":"2023-10-16T14:33:07.989830+0000","last_undegraded":"2023-10-16T14:33:07.989830+0000","last_fullsized":"2023-10-16T14:33:07.989830+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:04.523835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253922+0000","last_change":"2023-10-16T14:33:06.279865+0000","last_active":"2023-10-16T14:33:07.253922+0000","last_peered":"2023-10-16T14:33:07.253922+0000","last_clean":"2023-10-16T14:33:07.253922+0000","last_became_active":"2023-10-16T14:33:06.279562+0000","last_became_peered":"2023-10-16T14:33:06.279562+0000","last_unstale":"2023-10-16T14:33:07.253922+0000","last_undegraded":"2023-10-16T14:33:07.253922+0000","last_fullsized":"2023-10-16T14:33:07.253922+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:40.265140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253487+0000","last_change":"2023-10-16T14:33:06.285269+0000","last_active":"2023-10-16T14:33:07.253487+0000","last_peered":"2023-10-16T14:33:07.253487+0000","last_clean":"2023-10-16T14:33:07.253487+0000","last_became_active":"2023-10-16T14:33:06.283208+0000","last_became_peered":"2023-10-16T14:33:06.283208+0000","last_unstale":"2023-10-16T14:33:07.253487+0000","last_undegraded":"2023-10-16T14:33:07.253487+0000","last_fullsized":"2023-10-16T14:33:07.253487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:22.731595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985830+0000","last_change":"2023-10-16T14:33:06.284062+0000","last_active":"2023-10-16T14:33:07.985830+0000","last_peered":"2023-10-16T14:33:07.985830+0000","last_clean":"2023-10-16T14:33:07.985830+0000","last_became_active":"2023-10-16T14:33:06.282483+0000","last_became_peered":"2023-10-16T14:33:06.282483+0000","last_unstale":"2023-10-16T14:33:07.985830+0000","last_undegraded":"2023-10-16T14:33:07.985830+0000","last_fullsized":"2023-10-16T14:33:07.985830+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:31.878477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248564+0000","last_change":"2023-10-16T14:33:06.290607+0000","last_active":"2023-10-16T14:33:07.248564+0000","last_peered":"2023-10-16T14:33:07.248564+0000","last_clean":"2023-10-16T14:33:07.248564+0000","last_became_active":"2023-10-16T14:33:06.290263+0000","last_became_peered":"2023-10-16T14:33:06.290263+0000","last_unstale":"2023-10-16T14:33:07.248564+0000","last_undegraded":"2023-10-16T14:33:07.248564+0000","last_fullsized":"2023-10-16T14:33:07.248564+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:20.033542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987913+0000","last_change":"2023-10-16T14:33:06.276849+0000","last_active":"2023-10-16T14:33:07.987913+0000","last_peered":"2023-10-16T14:33:07.987913+0000","last_clean":"2023-10-16T14:33:07.987913+0000","last_became_active":"2023-10-16T14:33:06.276498+0000","last_became_peered":"2023-10-16T14:33:06.276498+0000","last_unstale":"2023-10-16T14:33:07.987913+0000","last_undegraded":"2023-10-16T14:33:07.987913+0000","last_fullsized":"2023-10-16T14:33:07.987913+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:51.501988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248015+0000","last_change":"2023-10-16T14:33:06.280052+0000","last_active":"2023-10-16T14:33:07.248015+0000","last_peered":"2023-10-16T14:33:07.248015+0000","last_clean":"2023-10-16T14:33:07.248015+0000","last_became_active":"2023-10-16T14:33:06.279520+0000","last_became_peered":"2023-10-16T14:33:06.279520+0000","last_unstale":"2023-10-16T14:33:07.248015+0000","last_undegraded":"2023-10-16T14:33:07.248015+0000","last_fullsized":"2023-10-16T14:33:07.248015+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:12.449231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.250843+0000","last_change":"2023-10-16T14:33:06.283574+0000","last_active":"2023-10-16T14:33:07.250843+0000","last_peered":"2023-10-16T14:33:07.250843+0000","last_clean":"2023-10-16T14:33:07.250843+0000","last_became_active":"2023-10-16T14:33:06.282123+0000","last_became_peered":"2023-10-16T14:33:06.282123+0000","last_unstale":"2023-10-16T14:33:07.250843+0000","last_undegraded":"2023-10-16T14:33:07.250843+0000","last_fullsized":"2023-10-16T14:33:07.250843+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:23.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989834+0000","last_change":"2023-10-16T14:33:06.286917+0000","last_active":"2023-10-16T14:33:07.989834+0000","last_peered":"2023-10-16T14:33:07.989834+0000","last_clean":"2023-10-16T14:33:07.989834+0000","last_became_active":"2023-10-16T14:33:06.286215+0000","last_became_peered":"2023-10-16T14:33:06.286215+0000","last_unstale":"2023-10-16T14:33:07.989834+0000","last_undegraded":"2023-10-16T14:33:07.989834+0000","last_fullsized":"2023-10-16T14:33:07.989834+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:31.878477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988983+0000","last_change":"2023-10-16T14:33:06.282367+0000","last_active":"2023-10-16T14:33:07.988983+0000","last_peered":"2023-10-16T14:33:07.988983+0000","last_clean":"2023-10-16T14:33:07.988983+0000","last_became_active":"2023-10-16T14:33:06.279214+0000","last_became_peered":"2023-10-16T14:33:06.279214+0000","last_unstale":"2023-10-16T14:33:07.988983+0000","last_undegraded":"2023-10-16T14:33:07.988983+0000","last_fullsized":"2023-10-16T14:33:07.988983+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:29.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985745+0000","last_change":"2023-10-16T14:33:06.285861+0000","last_active":"2023-10-16T14:33:07.985745+0000","last_peered":"2023-10-16T14:33:07.985745+0000","last_clean":"2023-10-16T14:33:07.985745+0000","last_became_active":"2023-10-16T14:33:06.285567+0000","last_became_peered":"2023-10-16T14:33:06.285567+0000","last_unstale":"2023-10-16T14:33:07.985745+0000","last_undegraded":"2023-10-16T14:33:07.985745+0000","last_fullsized":"2023-10-16T14:33:07.985745+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:12.449231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248314+0000","last_change":"2023-10-16T14:33:06.269884+0000","last_active":"2023-10-16T14:33:07.248314+0000","last_peered":"2023-10-16T14:33:07.248314+0000","last_clean":"2023-10-16T14:33:07.248314+0000","last_became_active":"2023-10-16T14:33:06.269555+0000","last_became_peered":"2023-10-16T14:33:06.269555+0000","last_unstale":"2023-10-16T14:33:07.248314+0000","last_undegraded":"2023-10-16T14:33:07.248314+0000","last_fullsized":"2023-10-16T14:33:07.248314+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:54.119904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252608+0000","last_change":"2023-10-16T14:33:06.259400+0000","last_active":"2023-10-16T14:33:07.252608+0000","last_peered":"2023-10-16T14:33:07.252608+0000","last_clean":"2023-10-16T14:33:07.252608+0000","last_became_active":"2023-10-16T14:33:06.258423+0000","last_became_peered":"2023-10-16T14:33:06.258423+0000","last_unstale":"2023-10-16T14:33:07.252608+0000","last_undegraded":"2023-10-16T14:33:07.252608+0000","last_fullsized":"2023-10-16T14:33:07.252608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:09.960593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252622+0000","last_change":"2023-10-16T14:33:06.282134+0000","last_active":"2023-10-16T14:33:07.252622+0000","last_peered":"2023-10-16T14:33:07.252622+0000","last_clean":"2023-10-16T14:33:07.252622+0000","last_became_active":"2023-10-16T14:33:06.281392+0000","last_became_peered":"2023-10-16T14:33:06.281392+0000","last_unstale":"2023-10-16T14:33:07.252622+0000","last_undegraded":"2023-10-16T14:33:07.252622+0000","last_fullsized":"2023-10-16T14:33:07.252622+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:48.026556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254107+0000","last_change":"2023-10-16T14:33:06.271190+0000","last_active":"2023-10-16T14:33:07.254107+0000","last_peered":"2023-10-16T14:33:07.254107+0000","last_clean":"2023-10-16T14:33:07.254107+0000","last_became_active":"2023-10-16T14:33:06.270927+0000","last_became_peered":"2023-10-16T14:33:06.270927+0000","last_unstale":"2023-10-16T14:33:07.254107+0000","last_undegraded":"2023-10-16T14:33:07.254107+0000","last_fullsized":"2023-10-16T14:33:07.254107+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:35.400724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.251969+0000","last_change":"2023-10-16T14:33:06.281005+0000","last_active":"2023-10-16T14:33:07.251969+0000","last_peered":"2023-10-16T14:33:07.251969+0000","last_clean":"2023-10-16T14:33:07.251969+0000","last_became_active":"2023-10-16T14:33:06.280720+0000","last_became_peered":"2023-10-16T14:33:06.280720+0000","last_unstale":"2023-10-16T14:33:07.251969+0000","last_undegraded":"2023-10-16T14:33:07.251969+0000","last_fullsized":"2023-10-16T14:33:07.251969+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:11.766477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254612+0000","last_change":"2023-10-16T14:33:06.265626+0000","last_active":"2023-10-16T14:33:07.254612+0000","last_peered":"2023-10-16T14:33:07.254612+0000","last_clean":"2023-10-16T14:33:07.254612+0000","last_became_active":"2023-10-16T14:33:06.265212+0000","last_became_peered":"2023-10-16T14:33:06.265212+0000","last_unstale":"2023-10-16T14:33:07.254612+0000","last_undegraded":"2023-10-16T14:33:07.254612+0000","last_fullsized":"2023-10-16T14:33:07.254612+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:20.944155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985872+0000","last_change":"2023-10-16T14:33:06.280574+0000","last_active":"2023-10-16T14:33:07.985872+0000","last_peered":"2023-10-16T14:33:07.985872+0000","last_clean":"2023-10-16T14:33:07.985872+0000","last_became_active":"2023-10-16T14:33:06.280309+0000","last_became_peered":"2023-10-16T14:33:06.280309+0000","last_unstale":"2023-10-16T14:33:07.985872+0000","last_undegraded":"2023-10-16T14:33:07.985872+0000","last_fullsized":"2023-10-16T14:33:07.985872+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:00.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988911+0000","last_change":"2023-10-16T14:33:06.292587+0000","last_active":"2023-10-16T14:33:07.988911+0000","last_peered":"2023-10-16T14:33:07.988911+0000","last_clean":"2023-10-16T14:33:07.988911+0000","last_became_active":"2023-10-16T14:33:06.290219+0000","last_became_peered":"2023-10-16T14:33:06.290219+0000","last_unstale":"2023-10-16T14:33:07.988911+0000","last_undegraded":"2023-10-16T14:33:07.988911+0000","last_fullsized":"2023-10-16T14:33:07.988911+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:25.409664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252380+0000","last_change":"2023-10-16T14:33:06.280855+0000","last_active":"2023-10-16T14:33:07.252380+0000","last_peered":"2023-10-16T14:33:07.252380+0000","last_clean":"2023-10-16T14:33:07.252380+0000","last_became_active":"2023-10-16T14:33:06.279199+0000","last_became_peered":"2023-10-16T14:33:06.279199+0000","last_unstale":"2023-10-16T14:33:07.252380+0000","last_undegraded":"2023-10-16T14:33:07.252380+0000","last_fullsized":"2023-10-16T14:33:07.252380+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:22.558261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988097+0000","last_change":"2023-10-16T14:33:06.281075+0000","last_active":"2023-10-16T14:33:07.988097+0000","last_peered":"2023-10-16T14:33:07.988097+0000","last_clean":"2023-10-16T14:33:07.988097+0000","last_became_active":"2023-10-16T14:33:06.280782+0000","last_became_peered":"2023-10-16T14:33:06.280782+0000","last_unstale":"2023-10-16T14:33:07.988097+0000","last_undegraded":"2023-10-16T14:33:07.988097+0000","last_fullsized":"2023-10-16T14:33:07.988097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:06.225751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984506+0000","last_change":"2023-10-16T14:33:06.285547+0000","last_active":"2023-10-16T14:33:07.984506+0000","last_peered":"2023-10-16T14:33:07.984506+0000","last_clean":"2023-10-16T14:33:07.984506+0000","last_became_active":"2023-10-16T14:33:06.284195+0000","last_became_peered":"2023-10-16T14:33:06.284195+0000","last_unstale":"2023-10-16T14:33:07.984506+0000","last_undegraded":"2023-10-16T14:33:07.984506+0000","last_fullsized":"2023-10-16T14:33:07.984506+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:23.285738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984810+0000","last_change":"2023-10-16T14:33:06.279779+0000","last_active":"2023-10-16T14:33:07.984810+0000","last_peered":"2023-10-16T14:33:07.984810+0000","last_clean":"2023-10-16T14:33:07.984810+0000","last_became_active":"2023-10-16T14:33:06.279483+0000","last_became_peered":"2023-10-16T14:33:06.279483+0000","last_unstale":"2023-10-16T14:33:07.984810+0000","last_undegraded":"2023-10-16T14:33:07.984810+0000","last_fullsized":"2023-10-16T14:33:07.984810+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:54.491439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987030+0000","last_change":"2023-10-16T14:33:06.287674+0000","last_active":"2023-10-16T14:33:07.987030+0000","last_peered":"2023-10-16T14:33:07.987030+0000","last_clean":"2023-10-16T14:33:07.987030+0000","last_became_active":"2023-10-16T14:33:06.287244+0000","last_became_peered":"2023-10-16T14:33:06.287244+0000","last_unstale":"2023-10-16T14:33:07.987030+0000","last_undegraded":"2023-10-16T14:33:07.987030+0000","last_fullsized":"2023-10-16T14:33:07.987030+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:34.170987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254103+0000","last_change":"2023-10-16T14:33:06.280569+0000","last_active":"2023-10-16T14:33:07.254103+0000","last_peered":"2023-10-16T14:33:07.254103+0000","last_clean":"2023-10-16T14:33:07.254103+0000","last_became_active":"2023-10-16T14:33:06.280252+0000","last_became_peered":"2023-10-16T14:33:06.280252+0000","last_unstale":"2023-10-16T14:33:07.254103+0000","last_undegraded":"2023-10-16T14:33:07.254103+0000","last_fullsized":"2023-10-16T14:33:07.254103+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:57.083160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.255067+0000","last_change":"2023-10-16T14:33:05.273054+0000","last_active":"2023-10-16T14:33:07.255067+0000","last_peered":"2023-10-16T14:33:07.255067+0000","last_clean":"2023-10-16T14:33:07.255067+0000","last_became_active":"2023-10-16T14:33:05.272368+0000","last_became_peered":"2023-10-16T14:33:05.272368+0000","last_unstale":"2023-10-16T14:33:07.255067+0000","last_undegraded":"2023-10-16T14:33:07.255067+0000","last_fullsized":"2023-10-16T14:33:07.255067+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.906363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.990022+0000","last_change":"2023-10-16T14:33:06.287094+0000","last_active":"2023-10-16T14:33:07.990022+0000","last_peered":"2023-10-16T14:33:07.990022+0000","last_clean":"2023-10-16T14:33:07.990022+0000","last_became_active":"2023-10-16T14:33:06.286622+0000","last_became_peered":"2023-10-16T14:33:06.286622+0000","last_unstale":"2023-10-16T14:33:07.990022+0000","last_undegraded":"2023-10-16T14:33:07.990022+0000","last_fullsized":"2023-10-16T14:33:07.990022+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:28.644767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248761+0000","last_change":"2023-10-16T14:33:05.263975+0000","last_active":"2023-10-16T14:33:07.248761+0000","last_peered":"2023-10-16T14:33:07.248761+0000","last_clean":"2023-10-16T14:33:07.248761+0000","last_became_active":"2023-10-16T14:33:05.263695+0000","last_became_peered":"2023-10-16T14:33:05.263695+0000","last_unstale":"2023-10-16T14:33:07.248761+0000","last_undegraded":"2023-10-16T14:33:07.248761+0000","last_fullsized":"2023-10-16T14:33:07.248761+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:21.593041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986364+0000","last_change":"2023-10-16T14:33:06.290319+0000","last_active":"2023-10-16T14:33:07.986364+0000","last_peered":"2023-10-16T14:33:07.986364+0000","last_clean":"2023-10-16T14:33:07.986364+0000","last_became_active":"2023-10-16T14:33:06.289380+0000","last_became_peered":"2023-10-16T14:33:06.289380+0000","last_unstale":"2023-10-16T14:33:07.986364+0000","last_undegraded":"2023-10-16T14:33:07.986364+0000","last_fullsized":"2023-10-16T14:33:07.986364+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:54.119904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.255291+0000","last_change":"2023-10-16T14:33:05.270881+0000","last_active":"2023-10-16T14:33:07.255291+0000","last_peered":"2023-10-16T14:33:07.255291+0000","last_clean":"2023-10-16T14:33:07.255291+0000","last_became_active":"2023-10-16T14:33:05.270584+0000","last_became_peered":"2023-10-16T14:33:05.270584+0000","last_unstale":"2023-10-16T14:33:07.255291+0000","last_undegraded":"2023-10-16T14:33:07.255291+0000","last_fullsized":"2023-10-16T14:33:07.255291+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:13.125033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253558+0000","last_change":"2023-10-16T14:33:06.266392+0000","last_active":"2023-10-16T14:33:07.253558+0000","last_peered":"2023-10-16T14:33:07.253558+0000","last_clean":"2023-10-16T14:33:07.253558+0000","last_became_active":"2023-10-16T14:33:06.266025+0000","last_became_peered":"2023-10-16T14:33:06.266025+0000","last_unstale":"2023-10-16T14:33:07.253558+0000","last_undegraded":"2023-10-16T14:33:07.253558+0000","last_fullsized":"2023-10-16T14:33:07.253558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:05.567929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986880+0000","last_change":"2023-10-16T14:33:05.269053+0000","last_active":"2023-10-16T14:33:07.986880+0000","last_peered":"2023-10-16T14:33:07.986880+0000","last_clean":"2023-10-16T14:33:07.986880+0000","last_became_active":"2023-10-16T14:33:05.268500+0000","last_became_peered":"2023-10-16T14:33:05.268500+0000","last_unstale":"2023-10-16T14:33:07.986880+0000","last_undegraded":"2023-10-16T14:33:07.986880+0000","last_fullsized":"2023-10-16T14:33:07.986880+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:11.097913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984859+0000","last_change":"2023-10-16T14:33:06.270746+0000","last_active":"2023-10-16T14:33:07.984859+0000","last_peered":"2023-10-16T14:33:07.984859+0000","last_clean":"2023-10-16T14:33:07.984859+0000","last_became_active":"2023-10-16T14:33:06.270428+0000","last_became_peered":"2023-10-16T14:33:06.270428+0000","last_unstale":"2023-10-16T14:33:07.984859+0000","last_undegraded":"2023-10-16T14:33:07.984859+0000","last_fullsized":"2023-10-16T14:33:07.984859+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:57.342973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988733+0000","last_change":"2023-10-16T14:33:05.272771+0000","last_active":"2023-10-16T14:33:07.988733+0000","last_peered":"2023-10-16T14:33:07.988733+0000","last_clean":"2023-10-16T14:33:07.988733+0000","last_became_active":"2023-10-16T14:33:05.272448+0000","last_became_peered":"2023-10-16T14:33:05.272448+0000","last_unstale":"2023-10-16T14:33:07.988733+0000","last_undegraded":"2023-10-16T14:33:07.988733+0000","last_fullsized":"2023-10-16T14:33:07.988733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:56.068730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252708+0000","last_change":"2023-10-16T14:33:06.271512+0000","last_active":"2023-10-16T14:33:07.252708+0000","last_peered":"2023-10-16T14:33:07.252708+0000","last_clean":"2023-10-16T14:33:07.252708+0000","last_became_active":"2023-10-16T14:33:06.271071+0000","last_became_peered":"2023-10-16T14:33:06.271071+0000","last_unstale":"2023-10-16T14:33:07.252708+0000","last_undegraded":"2023-10-16T14:33:07.252708+0000","last_fullsized":"2023-10-16T14:33:07.252708+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:36.960370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985481+0000","last_change":"2023-10-16T14:33:05.267089+0000","last_active":"2023-10-16T14:33:07.985481+0000","last_peered":"2023-10-16T14:33:07.985481+0000","last_clean":"2023-10-16T14:33:07.985481+0000","last_became_active":"2023-10-16T14:33:05.266565+0000","last_became_peered":"2023-10-16T14:33:05.266565+0000","last_unstale":"2023-10-16T14:33:07.985481+0000","last_undegraded":"2023-10-16T14:33:07.985481+0000","last_fullsized":"2023-10-16T14:33:07.985481+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:29.558673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984598+0000","last_change":"2023-10-16T14:33:06.279029+0000","last_active":"2023-10-16T14:33:07.984598+0000","last_peered":"2023-10-16T14:33:07.984598+0000","last_clean":"2023-10-16T14:33:07.984598+0000","last_became_active":"2023-10-16T14:33:06.278629+0000","last_became_peered":"2023-10-16T14:33:06.278629+0000","last_unstale":"2023-10-16T14:33:07.984598+0000","last_undegraded":"2023-10-16T14:33:07.984598+0000","last_fullsized":"2023-10-16T14:33:07.984598+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:26.091375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984823+0000","last_change":"2023-10-16T14:33:05.273729+0000","last_active":"2023-10-16T14:33:07.984823+0000","last_peered":"2023-10-16T14:33:07.984823+0000","last_clean":"2023-10-16T14:33:07.984823+0000","last_became_active":"2023-10-16T14:33:05.273320+0000","last_became_peered":"2023-10-16T14:33:05.273320+0000","last_unstale":"2023-10-16T14:33:07.984823+0000","last_undegraded":"2023-10-16T14:33:07.984823+0000","last_fullsized":"2023-10-16T14:33:07.984823+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:04.128955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248173+0000","last_change":"2023-10-16T14:33:06.260963+0000","last_active":"2023-10-16T14:33:07.248173+0000","last_peered":"2023-10-16T14:33:07.248173+0000","last_clean":"2023-10-16T14:33:07.248173+0000","last_became_active":"2023-10-16T14:33:06.260201+0000","last_became_peered":"2023-10-16T14:33:06.260201+0000","last_unstale":"2023-10-16T14:33:07.248173+0000","last_undegraded":"2023-10-16T14:33:07.248173+0000","last_fullsized":"2023-10-16T14:33:07.248173+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:01.101300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984506+0000","last_change":"2023-10-16T14:33:05.259428+0000","last_active":"2023-10-16T14:33:07.984506+0000","last_peered":"2023-10-16T14:33:07.984506+0000","last_clean":"2023-10-16T14:33:07.984506+0000","last_became_active":"2023-10-16T14:33:05.259064+0000","last_became_peered":"2023-10-16T14:33:05.259064+0000","last_unstale":"2023-10-16T14:33:07.984506+0000","last_undegraded":"2023-10-16T14:33:07.984506+0000","last_fullsized":"2023-10-16T14:33:07.984506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:19.019112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986198+0000","last_change":"2023-10-16T14:33:06.273488+0000","last_active":"2023-10-16T14:33:07.986198+0000","last_peered":"2023-10-16T14:33:07.986198+0000","last_clean":"2023-10-16T14:33:07.986198+0000","last_became_active":"2023-10-16T14:33:06.270425+0000","last_became_peered":"2023-10-16T14:33:06.270425+0000","last_unstale":"2023-10-16T14:33:07.986198+0000","last_undegraded":"2023-10-16T14:33:07.986198+0000","last_fullsized":"2023-10-16T14:33:07.986198+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:37.480286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254709+0000","last_change":"2023-10-16T14:33:05.268560+0000","last_active":"2023-10-16T14:33:07.254709+0000","last_peered":"2023-10-16T14:33:07.254709+0000","last_clean":"2023-10-16T14:33:07.254709+0000","last_became_active":"2023-10-16T14:33:05.267645+0000","last_became_peered":"2023-10-16T14:33:05.267645+0000","last_unstale":"2023-10-16T14:33:07.254709+0000","last_undegraded":"2023-10-16T14:33:07.254709+0000","last_fullsized":"2023-10-16T14:33:07.254709+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:22:20.778752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254517+0000","last_change":"2023-10-16T14:33:06.294561+0000","last_active":"2023-10-16T14:33:07.254517+0000","last_peered":"2023-10-16T14:33:07.254517+0000","last_clean":"2023-10-16T14:33:07.254517+0000","last_became_active":"2023-10-16T14:33:06.292779+0000","last_became_peered":"2023-10-16T14:33:06.292779+0000","last_unstale":"2023-10-16T14:33:07.254517+0000","last_undegraded":"2023-10-16T14:33:07.254517+0000","last_fullsized":"2023-10-16T14:33:07.254517+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:06.225751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987096+0000","last_change":"2023-10-16T14:33:05.271493+0000","last_active":"2023-10-16T14:33:07.987096+0000","last_peered":"2023-10-16T14:33:07.987096+0000","last_clean":"2023-10-16T14:33:07.987096+0000","last_became_active":"2023-10-16T14:33:05.270940+0000","last_became_peered":"2023-10-16T14:33:05.270940+0000","last_unstale":"2023-10-16T14:33:07.987096+0000","last_undegraded":"2023-10-16T14:33:07.987096+0000","last_fullsized":"2023-10-16T14:33:07.987096+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:33.793509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986754+0000","last_change":"2023-10-16T14:33:06.290978+0000","last_active":"2023-10-16T14:33:07.986754+0000","last_peered":"2023-10-16T14:33:07.986754+0000","last_clean":"2023-10-16T14:33:07.986754+0000","last_became_active":"2023-10-16T14:33:06.289599+0000","last_became_peered":"2023-10-16T14:33:06.289599+0000","last_unstale":"2023-10-16T14:33:07.986754+0000","last_undegraded":"2023-10-16T14:33:07.986754+0000","last_fullsized":"2023-10-16T14:33:07.986754+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:02.431469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"96'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989087+0000","last_change":"2023-10-16T14:33:05.267126+0000","last_active":"2023-10-16T14:33:07.989087+0000","last_peered":"2023-10-16T14:33:07.989087+0000","last_clean":"2023-10-16T14:33:07.989087+0000","last_became_active":"2023-10-16T14:33:05.266702+0000","last_became_peered":"2023-10-16T14:33:05.266702+0000","last_unstale":"2023-10-16T14:33:07.989087+0000","last_undegraded":"2023-10-16T14:33:07.989087+0000","last_fullsized":"2023-10-16T14:33:07.989087+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:34.386293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987285+0000","last_change":"2023-10-16T14:33:06.276963+0000","last_active":"2023-10-16T14:33:07.987285+0000","last_peered":"2023-10-16T14:33:07.987285+0000","last_clean":"2023-10-16T14:33:07.987285+0000","last_became_active":"2023-10-16T14:33:06.276673+0000","last_became_peered":"2023-10-16T14:33:06.276673+0000","last_unstale":"2023-10-16T14:33:07.987285+0000","last_undegraded":"2023-10-16T14:33:07.987285+0000","last_fullsized":"2023-10-16T14:33:07.987285+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:59.522562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"96'6","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987165+0000","last_change":"2023-10-16T14:33:05.262619+0000","last_active":"2023-10-16T14:33:07.987165+0000","last_peered":"2023-10-16T14:33:07.987165+0000","last_clean":"2023-10-16T14:33:07.987165+0000","last_became_active":"2023-10-16T14:33:05.262101+0000","last_became_peered":"2023-10-16T14:33:05.262101+0000","last_unstale":"2023-10-16T14:33:07.987165+0000","last_undegraded":"2023-10-16T14:33:07.987165+0000","last_fullsized":"2023-10-16T14:33:07.987165+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:51.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253217+0000","last_change":"2023-10-16T14:33:06.281272+0000","last_active":"2023-10-16T14:33:07.253217+0000","last_peered":"2023-10-16T14:33:07.253217+0000","last_clean":"2023-10-16T14:33:07.253217+0000","last_became_active":"2023-10-16T14:33:06.281005+0000","last_became_peered":"2023-10-16T14:33:06.281005+0000","last_unstale":"2023-10-16T14:33:07.253217+0000","last_undegraded":"2023-10-16T14:33:07.253217+0000","last_fullsized":"2023-10-16T14:33:07.253217+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:44.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985669+0000","last_change":"2023-10-16T14:33:05.269636+0000","last_active":"2023-10-16T14:33:07.985669+0000","last_peered":"2023-10-16T14:33:07.985669+0000","last_clean":"2023-10-16T14:33:07.985669+0000","last_became_active":"2023-10-16T14:33:05.269271+0000","last_became_peered":"2023-10-16T14:33:05.269271+0000","last_unstale":"2023-10-16T14:33:07.985669+0000","last_undegraded":"2023-10-16T14:33:07.985669+0000","last_fullsized":"2023-10-16T14:33:07.985669+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:24.346415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986239+0000","last_change":"2023-10-16T14:33:06.271783+0000","last_active":"2023-10-16T14:33:07.986239+0000","last_peered":"2023-10-16T14:33:07.986239+0000","last_clean":"2023-10-16T14:33:07.986239+0000","last_became_active":"2023-10-16T14:33:06.270496+0000","last_became_peered":"2023-10-16T14:33:06.270496+0000","last_unstale":"2023-10-16T14:33:07.986239+0000","last_undegraded":"2023-10-16T14:33:07.986239+0000","last_fullsized":"2023-10-16T14:33:07.986239+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:05.492228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985721+0000","last_change":"2023-10-16T14:33:05.270100+0000","last_active":"2023-10-16T14:33:07.985721+0000","last_peered":"2023-10-16T14:33:07.985721+0000","last_clean":"2023-10-16T14:33:07.985721+0000","last_became_active":"2023-10-16T14:33:05.267814+0000","last_became_peered":"2023-10-16T14:33:05.267814+0000","last_unstale":"2023-10-16T14:33:07.985721+0000","last_undegraded":"2023-10-16T14:33:07.985721+0000","last_fullsized":"2023-10-16T14:33:07.985721+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:02.881371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.247923+0000","last_change":"2023-10-16T14:33:06.279613+0000","last_active":"2023-10-16T14:33:07.247923+0000","last_peered":"2023-10-16T14:33:07.247923+0000","last_clean":"2023-10-16T14:33:07.247923+0000","last_became_active":"2023-10-16T14:33:06.279257+0000","last_became_peered":"2023-10-16T14:33:06.279257+0000","last_unstale":"2023-10-16T14:33:07.247923+0000","last_undegraded":"2023-10-16T14:33:07.247923+0000","last_fullsized":"2023-10-16T14:33:07.247923+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:13.783995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984429+0000","last_change":"2023-10-16T14:33:05.265988+0000","last_active":"2023-10-16T14:33:07.984429+0000","last_peered":"2023-10-16T14:33:07.984429+0000","last_clean":"2023-10-16T14:33:07.984429+0000","last_became_active":"2023-10-16T14:33:05.265714+0000","last_became_peered":"2023-10-16T14:33:05.265714+0000","last_unstale":"2023-10-16T14:33:07.984429+0000","last_undegraded":"2023-10-16T14:33:07.984429+0000","last_fullsized":"2023-10-16T14:33:07.984429+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:17.837153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987218+0000","last_change":"2023-10-16T14:33:06.286088+0000","last_active":"2023-10-16T14:33:07.987218+0000","last_peered":"2023-10-16T14:33:07.987218+0000","last_clean":"2023-10-16T14:33:07.987218+0000","last_became_active":"2023-10-16T14:33:06.284565+0000","last_became_peered":"2023-10-16T14:33:06.284565+0000","last_unstale":"2023-10-16T14:33:07.987218+0000","last_undegraded":"2023-10-16T14:33:07.987218+0000","last_fullsized":"2023-10-16T14:33:07.987218+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:26.046849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"96'8","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.250938+0000","last_change":"2023-10-16T14:33:05.275803+0000","last_active":"2023-10-16T14:33:07.250938+0000","last_peered":"2023-10-16T14:33:07.250938+0000","last_clean":"2023-10-16T14:33:07.250938+0000","last_became_active":"2023-10-16T14:33:05.275198+0000","last_became_peered":"2023-10-16T14:33:05.275198+0000","last_unstale":"2023-10-16T14:33:07.250938+0000","last_undegraded":"2023-10-16T14:33:07.250938+0000","last_fullsized":"2023-10-16T14:33:07.250938+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:29.410911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254237+0000","last_change":"2023-10-16T14:33:06.274038+0000","last_active":"2023-10-16T14:33:07.254237+0000","last_peered":"2023-10-16T14:33:07.254237+0000","last_clean":"2023-10-16T14:33:07.254237+0000","last_became_active":"2023-10-16T14:33:06.273570+0000","last_became_peered":"2023-10-16T14:33:06.273570+0000","last_unstale":"2023-10-16T14:33:07.254237+0000","last_undegraded":"2023-10-16T14:33:07.254237+0000","last_fullsized":"2023-10-16T14:33:07.254237+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:09.660436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987154+0000","last_change":"2023-10-16T14:33:05.270898+0000","last_active":"2023-10-16T14:33:07.987154+0000","last_peered":"2023-10-16T14:33:07.987154+0000","last_clean":"2023-10-16T14:33:07.987154+0000","last_became_active":"2023-10-16T14:33:05.269881+0000","last_became_peered":"2023-10-16T14:33:05.269881+0000","last_unstale":"2023-10-16T14:33:07.987154+0000","last_undegraded":"2023-10-16T14:33:07.987154+0000","last_fullsized":"2023-10-16T14:33:07.987154+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:10.563882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254835+0000","last_change":"2023-10-16T14:33:06.273336+0000","last_active":"2023-10-16T14:33:07.254835+0000","last_peered":"2023-10-16T14:33:07.254835+0000","last_clean":"2023-10-16T14:33:07.254835+0000","last_became_active":"2023-10-16T14:33:06.273054+0000","last_became_peered":"2023-10-16T14:33:06.273054+0000","last_unstale":"2023-10-16T14:33:07.254835+0000","last_undegraded":"2023-10-16T14:33:07.254835+0000","last_fullsized":"2023-10-16T14:33:07.254835+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:49.739625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254526+0000","last_change":"2023-10-16T14:33:05.271016+0000","last_active":"2023-10-16T14:33:07.254526+0000","last_peered":"2023-10-16T14:33:07.254526+0000","last_clean":"2023-10-16T14:33:07.254526+0000","last_became_active":"2023-10-16T14:33:05.269379+0000","last_became_peered":"2023-10-16T14:33:05.269379+0000","last_unstale":"2023-10-16T14:33:07.254526+0000","last_undegraded":"2023-10-16T14:33:07.254526+0000","last_fullsized":"2023-10-16T14:33:07.254526+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:54.891635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986221+0000","last_change":"2023-10-16T14:33:06.281783+0000","last_active":"2023-10-16T14:33:07.986221+0000","last_peered":"2023-10-16T14:33:07.986221+0000","last_clean":"2023-10-16T14:33:07.986221+0000","last_became_active":"2023-10-16T14:33:06.281511+0000","last_became_peered":"2023-10-16T14:33:06.281511+0000","last_unstale":"2023-10-16T14:33:07.986221+0000","last_undegraded":"2023-10-16T14:33:07.986221+0000","last_fullsized":"2023-10-16T14:33:07.986221+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:28.644767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985342+0000","last_change":"2023-10-16T14:33:05.273682+0000","last_active":"2023-10-16T14:33:07.985342+0000","last_peered":"2023-10-16T14:33:07.985342+0000","last_clean":"2023-10-16T14:33:07.985342+0000","last_became_active":"2023-10-16T14:33:05.272671+0000","last_became_peered":"2023-10-16T14:33:05.272671+0000","last_unstale":"2023-10-16T14:33:07.985342+0000","last_undegraded":"2023-10-16T14:33:07.985342+0000","last_fullsized":"2023-10-16T14:33:07.985342+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:57.398530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987611+0000","last_change":"2023-10-16T14:33:06.276899+0000","last_active":"2023-10-16T14:33:07.987611+0000","last_peered":"2023-10-16T14:33:07.987611+0000","last_clean":"2023-10-16T14:33:07.987611+0000","last_became_active":"2023-10-16T14:33:06.276575+0000","last_became_peered":"2023-10-16T14:33:06.276575+0000","last_unstale":"2023-10-16T14:33:07.987611+0000","last_undegraded":"2023-10-16T14:33:07.987611+0000","last_fullsized":"2023-10-16T14:33:07.987611+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:35.729049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253537+0000","last_change":"2023-10-16T14:33:05.276515+0000","last_active":"2023-10-16T14:33:07.253537+0000","last_peered":"2023-10-16T14:33:07.253537+0000","last_clean":"2023-10-16T14:33:07.253537+0000","last_became_active":"2023-10-16T14:33:05.276112+0000","last_became_peered":"2023-10-16T14:33:05.276112+0000","last_unstale":"2023-10-16T14:33:07.253537+0000","last_undegraded":"2023-10-16T14:33:07.253537+0000","last_fullsized":"2023-10-16T14:33:07.253537+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:45.581017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988453+0000","last_change":"2023-10-16T14:33:06.280867+0000","last_active":"2023-10-16T14:33:07.988453+0000","last_peered":"2023-10-16T14:33:07.988453+0000","last_clean":"2023-10-16T14:33:07.988453+0000","last_became_active":"2023-10-16T14:33:06.280604+0000","last_became_peered":"2023-10-16T14:33:06.280604+0000","last_unstale":"2023-10-16T14:33:07.988453+0000","last_undegraded":"2023-10-16T14:33:07.988453+0000","last_fullsized":"2023-10-16T14:33:07.988453+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:31:19.851053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985947+0000","last_change":"2023-10-16T14:33:05.266919+0000","last_active":"2023-10-16T14:33:07.985947+0000","last_peered":"2023-10-16T14:33:07.985947+0000","last_clean":"2023-10-16T14:33:07.985947+0000","last_became_active":"2023-10-16T14:33:05.266518+0000","last_became_peered":"2023-10-16T14:33:05.266518+0000","last_unstale":"2023-10-16T14:33:07.985947+0000","last_undegraded":"2023-10-16T14:33:07.985947+0000","last_fullsized":"2023-10-16T14:33:07.985947+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:01.201377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986655+0000","last_change":"2023-10-16T14:33:06.283373+0000","last_active":"2023-10-16T14:33:07.986655+0000","last_peered":"2023-10-16T14:33:07.986655+0000","last_clean":"2023-10-16T14:33:07.986655+0000","last_became_active":"2023-10-16T14:33:06.280559+0000","last_became_peered":"2023-10-16T14:33:06.280559+0000","last_unstale":"2023-10-16T14:33:07.986655+0000","last_undegraded":"2023-10-16T14:33:07.986655+0000","last_fullsized":"2023-10-16T14:33:07.986655+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:50.049165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985020+0000","last_change":"2023-10-16T14:33:05.264128+0000","last_active":"2023-10-16T14:33:07.985020+0000","last_peered":"2023-10-16T14:33:07.985020+0000","last_clean":"2023-10-16T14:33:07.985020+0000","last_became_active":"2023-10-16T14:33:05.263842+0000","last_became_peered":"2023-10-16T14:33:05.263842+0000","last_unstale":"2023-10-16T14:33:07.985020+0000","last_undegraded":"2023-10-16T14:33:07.985020+0000","last_fullsized":"2023-10-16T14:33:07.985020+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:37.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987423+0000","last_change":"2023-10-16T14:33:06.283404+0000","last_active":"2023-10-16T14:33:07.987423+0000","last_peered":"2023-10-16T14:33:07.987423+0000","last_clean":"2023-10-16T14:33:07.987423+0000","last_became_active":"2023-10-16T14:33:06.283119+0000","last_became_peered":"2023-10-16T14:33:06.283119+0000","last_unstale":"2023-10-16T14:33:07.987423+0000","last_undegraded":"2023-10-16T14:33:07.987423+0000","last_fullsized":"2023-10-16T14:33:07.987423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:51.044247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.247857+0000","last_change":"2023-10-16T14:33:05.262319+0000","last_active":"2023-10-16T14:33:07.247857+0000","last_peered":"2023-10-16T14:33:07.247857+0000","last_clean":"2023-10-16T14:33:07.247857+0000","last_became_active":"2023-10-16T14:33:05.262028+0000","last_became_peered":"2023-10-16T14:33:05.262028+0000","last_unstale":"2023-10-16T14:33:07.247857+0000","last_undegraded":"2023-10-16T14:33:07.247857+0000","last_fullsized":"2023-10-16T14:33:07.247857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:51.961764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248835+0000","last_change":"2023-10-16T14:33:06.277414+0000","last_active":"2023-10-16T14:33:07.248835+0000","last_peered":"2023-10-16T14:33:07.248835+0000","last_clean":"2023-10-16T14:33:07.248835+0000","last_became_active":"2023-10-16T14:33:06.277104+0000","last_became_peered":"2023-10-16T14:33:06.277104+0000","last_unstale":"2023-10-16T14:33:07.248835+0000","last_undegraded":"2023-10-16T14:33:07.248835+0000","last_fullsized":"2023-10-16T14:33:07.248835+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:02.431469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987128+0000","last_change":"2023-10-16T14:33:05.269622+0000","last_active":"2023-10-16T14:33:07.987128+0000","last_peered":"2023-10-16T14:33:07.987128+0000","last_clean":"2023-10-16T14:33:07.987128+0000","last_became_active":"2023-10-16T14:33:05.269266+0000","last_became_peered":"2023-10-16T14:33:05.269266+0000","last_unstale":"2023-10-16T14:33:07.987128+0000","last_undegraded":"2023-10-16T14:33:07.987128+0000","last_fullsized":"2023-10-16T14:33:07.987128+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:42.623094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986208+0000","last_change":"2023-10-16T14:33:06.275097+0000","last_active":"2023-10-16T14:33:07.986208+0000","last_peered":"2023-10-16T14:33:07.986208+0000","last_clean":"2023-10-16T14:33:07.986208+0000","last_became_active":"2023-10-16T14:33:06.274800+0000","last_became_peered":"2023-10-16T14:33:06.274800+0000","last_unstale":"2023-10-16T14:33:07.986208+0000","last_undegraded":"2023-10-16T14:33:07.986208+0000","last_fullsized":"2023-10-16T14:33:07.986208+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:33.996935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986452+0000","last_change":"2023-10-16T14:33:05.267780+0000","last_active":"2023-10-16T14:33:07.986452+0000","last_peered":"2023-10-16T14:33:07.986452+0000","last_clean":"2023-10-16T14:33:07.986452+0000","last_became_active":"2023-10-16T14:33:05.267523+0000","last_became_peered":"2023-10-16T14:33:05.267523+0000","last_unstale":"2023-10-16T14:33:07.986452+0000","last_undegraded":"2023-10-16T14:33:07.986452+0000","last_fullsized":"2023-10-16T14:33:07.986452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:02.907250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254488+0000","last_change":"2023-10-16T14:33:06.295780+0000","last_active":"2023-10-16T14:33:07.254488+0000","last_peered":"2023-10-16T14:33:07.254488+0000","last_clean":"2023-10-16T14:33:07.254488+0000","last_became_active":"2023-10-16T14:33:06.291980+0000","last_became_peered":"2023-10-16T14:33:06.291980+0000","last_unstale":"2023-10-16T14:33:07.254488+0000","last_undegraded":"2023-10-16T14:33:07.254488+0000","last_fullsized":"2023-10-16T14:33:07.254488+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:51.044247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987515+0000","last_change":"2023-10-16T14:33:05.260854+0000","last_active":"2023-10-16T14:33:07.987515+0000","last_peered":"2023-10-16T14:33:07.987515+0000","last_clean":"2023-10-16T14:33:07.987515+0000","last_became_active":"2023-10-16T14:33:05.260559+0000","last_became_peered":"2023-10-16T14:33:05.260559+0000","last_unstale":"2023-10-16T14:33:07.987515+0000","last_undegraded":"2023-10-16T14:33:07.987515+0000","last_fullsized":"2023-10-16T14:33:07.987515+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:53.105474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254832+0000","last_change":"2023-10-16T14:33:06.276344+0000","last_active":"2023-10-16T14:33:07.254832+0000","last_peered":"2023-10-16T14:33:07.254832+0000","last_clean":"2023-10-16T14:33:07.254832+0000","last_became_active":"2023-10-16T14:33:06.269991+0000","last_became_peered":"2023-10-16T14:33:06.269991+0000","last_unstale":"2023-10-16T14:33:07.254832+0000","last_undegraded":"2023-10-16T14:33:07.254832+0000","last_fullsized":"2023-10-16T14:33:07.254832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:51.862361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248164+0000","last_change":"2023-10-16T14:33:05.271677+0000","last_active":"2023-10-16T14:33:07.248164+0000","last_peered":"2023-10-16T14:33:07.248164+0000","last_clean":"2023-10-16T14:33:07.248164+0000","last_became_active":"2023-10-16T14:33:05.271283+0000","last_became_peered":"2023-10-16T14:33:05.271283+0000","last_unstale":"2023-10-16T14:33:07.248164+0000","last_undegraded":"2023-10-16T14:33:07.248164+0000","last_fullsized":"2023-10-16T14:33:07.248164+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:54:17.250940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988041+0000","last_change":"2023-10-16T14:33:06.281177+0000","last_active":"2023-10-16T14:33:07.988041+0000","last_peered":"2023-10-16T14:33:07.988041+0000","last_clean":"2023-10-16T14:33:07.988041+0000","last_became_active":"2023-10-16T14:33:06.280946+0000","last_became_peered":"2023-10-16T14:33:06.280946+0000","last_unstale":"2023-10-16T14:33:07.988041+0000","last_undegraded":"2023-10-16T14:33:07.988041+0000","last_fullsized":"2023-10-16T14:33:07.988041+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:52.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989428+0000","last_change":"2023-10-16T14:33:05.266588+0000","last_active":"2023-10-16T14:33:07.989428+0000","last_peered":"2023-10-16T14:33:07.989428+0000","last_clean":"2023-10-16T14:33:07.989428+0000","last_became_active":"2023-10-16T14:33:05.266303+0000","last_became_peered":"2023-10-16T14:33:05.266303+0000","last_unstale":"2023-10-16T14:33:07.989428+0000","last_undegraded":"2023-10-16T14:33:07.989428+0000","last_fullsized":"2023-10-16T14:33:07.989428+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:09.205400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985606+0000","last_change":"2023-10-16T14:33:06.290344+0000","last_active":"2023-10-16T14:33:07.985606+0000","last_peered":"2023-10-16T14:33:07.985606+0000","last_clean":"2023-10-16T14:33:07.985606+0000","last_became_active":"2023-10-16T14:33:06.289622+0000","last_became_peered":"2023-10-16T14:33:06.289622+0000","last_unstale":"2023-10-16T14:33:07.985606+0000","last_undegraded":"2023-10-16T14:33:07.985606+0000","last_fullsized":"2023-10-16T14:33:07.985606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:13.783995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984826+0000","last_change":"2023-10-16T14:33:05.272774+0000","last_active":"2023-10-16T14:33:07.984826+0000","last_peered":"2023-10-16T14:33:07.984826+0000","last_clean":"2023-10-16T14:33:07.984826+0000","last_became_active":"2023-10-16T14:33:05.272478+0000","last_became_peered":"2023-10-16T14:33:05.272478+0000","last_unstale":"2023-10-16T14:33:07.984826+0000","last_undegraded":"2023-10-16T14:33:07.984826+0000","last_fullsized":"2023-10-16T14:33:07.984826+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:59.242105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985776+0000","last_change":"2023-10-16T14:33:06.272448+0000","last_active":"2023-10-16T14:33:07.985776+0000","last_peered":"2023-10-16T14:33:07.985776+0000","last_clean":"2023-10-16T14:33:07.985776+0000","last_became_active":"2023-10-16T14:33:06.272171+0000","last_became_peered":"2023-10-16T14:33:06.272171+0000","last_unstale":"2023-10-16T14:33:07.985776+0000","last_undegraded":"2023-10-16T14:33:07.985776+0000","last_fullsized":"2023-10-16T14:33:07.985776+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:04.523835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.250873+0000","last_change":"2023-10-16T14:33:05.274159+0000","last_active":"2023-10-16T14:33:07.250873+0000","last_peered":"2023-10-16T14:33:07.250873+0000","last_clean":"2023-10-16T14:33:07.250873+0000","last_became_active":"2023-10-16T14:33:05.273529+0000","last_became_peered":"2023-10-16T14:33:05.273529+0000","last_unstale":"2023-10-16T14:33:07.250873+0000","last_undegraded":"2023-10-16T14:33:07.250873+0000","last_fullsized":"2023-10-16T14:33:07.250873+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:52.990469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985124+0000","last_change":"2023-10-16T14:33:06.284496+0000","last_active":"2023-10-16T14:33:07.985124+0000","last_peered":"2023-10-16T14:33:07.985124+0000","last_clean":"2023-10-16T14:33:07.985124+0000","last_became_active":"2023-10-16T14:33:06.284213+0000","last_became_peered":"2023-10-16T14:33:06.284213+0000","last_unstale":"2023-10-16T14:33:07.985124+0000","last_undegraded":"2023-10-16T14:33:07.985124+0000","last_fullsized":"2023-10-16T14:33:07.985124+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.607472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986745+0000","last_change":"2023-10-16T14:33:05.263747+0000","last_active":"2023-10-16T14:33:07.986745+0000","last_peered":"2023-10-16T14:33:07.986745+0000","last_clean":"2023-10-16T14:33:07.986745+0000","last_became_active":"2023-10-16T14:33:05.263464+0000","last_became_peered":"2023-10-16T14:33:05.263464+0000","last_unstale":"2023-10-16T14:33:07.986745+0000","last_undegraded":"2023-10-16T14:33:07.986745+0000","last_fullsized":"2023-10-16T14:33:07.986745+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:54:17.250940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988380+0000","last_change":"2023-10-16T14:33:06.282571+0000","last_active":"2023-10-16T14:33:07.988380+0000","last_peered":"2023-10-16T14:33:07.988380+0000","last_clean":"2023-10-16T14:33:07.988380+0000","last_became_active":"2023-10-16T14:33:06.282266+0000","last_became_peered":"2023-10-16T14:33:06.282266+0000","last_unstale":"2023-10-16T14:33:07.988380+0000","last_undegraded":"2023-10-16T14:33:07.988380+0000","last_fullsized":"2023-10-16T14:33:07.988380+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:49.739625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985631+0000","last_change":"2023-10-16T14:33:05.269864+0000","last_active":"2023-10-16T14:33:07.985631+0000","last_peered":"2023-10-16T14:33:07.985631+0000","last_clean":"2023-10-16T14:33:07.985631+0000","last_became_active":"2023-10-16T14:33:05.269506+0000","last_became_peered":"2023-10-16T14:33:05.269506+0000","last_unstale":"2023-10-16T14:33:07.985631+0000","last_undegraded":"2023-10-16T14:33:07.985631+0000","last_fullsized":"2023-10-16T14:33:07.985631+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:51.961764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"86'175","reported_seq":455,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-16T14:33:06.253357+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:33:06.253357+0000","last_peered":"2023-10-16T14:33:06.253357+0000","last_clean":"2023-10-16T14:33:06.253357+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:33:06.253357+0000","last_undegraded":"2023-10-16T14:33:06.253357+0000","last_fullsized":"2023-10-16T14:33:06.253357+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575050,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1785856,"data_stored":1159936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0640000000000001,"15min":1.147},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.42799999999999999},"max":{"1min":1.784,"5min":1.833,"15min":1.9159999999999999},"last":1.601},{"interface":"front","average":{"1min":1.081,"5min":1.133,"15min":1.157},"min":{"1min":0.77800000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.522,"5min":1.6220000000000001,"15min":1.673},"last":1.4219999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1599999999999999,"15min":1.1799999999999999},"min":{"1min":0.71599999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3859999999999999,"5min":1.7030000000000001,"15min":1.8320000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.22,"15min":1.2390000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.091}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.298,"15min":1.296},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63500000000000001},"max":{"1min":1.512,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.353},{"interface":"front","average":{"1min":1.137,"5min":1.23,"15min":1.234},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.996}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.298,"15min":1.2989999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3500000000000001,"15min":1.379},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2509999999999999},"last":1.7090000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3580000000000001,"15min":1.3600000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.835,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.456},{"interface":"front","average":{"1min":1.236,"5min":1.25,"15min":1.258},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.9530000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.86699999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3939999999999999,"15min":1.407},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.232},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4750000000000001,"15min":1.4870000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.4910000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.2850000000000001,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.504,"15min":1.512},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":2.359,"5min":2.359,"15min":2.427},"last":1.766}]}]},{"osd":6,"up_from":13,"seq":55834575049,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335336,"statfs":{"total":96636764160,"available":96599384064,"internally_reserved":0,"allocated":1191936,"data_stored":569044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.169},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.6990000000000001,"5min":1.7569999999999999,"15min":2.2160000000000002},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.82599999999999996,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.577,"5min":2.2709999999999999,"15min":3.863},"last":1.026}]},{"osd":1,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.2509999999999999},"min":{"1min":0.92400000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.581,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.3,"15min":1.29},"min":{"1min":0.81799999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7849999999999999,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":1.1160000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.147,"15min":1.1990000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.59899999999999998},"max":{"1min":1.3400000000000001,"5min":1.708,"15min":3.5800000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.232,"5min":1.3180000000000001,"15min":1.3500000000000001},"min":{"1min":0.80200000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.52,"5min":1.8799999999999999,"15min":2.1429999999999998},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3300000000000001,"15min":1.353},"min":{"1min":0.85599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.8480000000000001,"5min":1.96,"15min":4.9790000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.286,"15min":1.3400000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":1.6080000000000001,"5min":1.913,"15min":5.4160000000000004},"last":0.89400000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.296,"15min":1.3},"min":{"1min":0.748,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8009999999999999,"5min":2.1070000000000002,"15min":5},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3640000000000001,"15min":1.377},"min":{"1min":0.94999999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.897,"5min":2.1789999999999998,"15min":5.4640000000000004},"last":1.232}]},{"osd":5,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2949999999999999,"15min":1.3160000000000001},"min":{"1min":0.89600000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.76,"5min":1.859,"15min":3.9279999999999999},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.482,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":3.6749999999999998},"last":1.3100000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.5600000000000001},"min":{"1min":1.105,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":4.6859999999999999},"last":1.389},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.508,"15min":1.542},"min":{"1min":1.0840000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9139999999999999,"5min":2.0459999999999998,"15min":4.883},"last":1.6870000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575049,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36488,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335352,"statfs":{"total":96636764160,"available":96599400448,"internally_reserved":0,"allocated":1175552,"data_stored":562742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.099,"15min":1.1160000000000001},"min":{"1min":0.73599999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5760000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1990000000000001,"15min":1.1819999999999999},"min":{"1min":0.63300000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.7729999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.2909999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.444,"5min":1.726,"15min":1.726},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.2430000000000001},"min":{"1min":0.81499999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5249999999999999,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.1899999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.244,"15min":1.2390000000000001},"min":{"1min":0.71599999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5720000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.141},{"interface":"front","average":{"1min":1.304,"5min":1.3200000000000001,"15min":1.353},"min":{"1min":0.85799999999999998,"5min":0.625,"15min":0.625},"max":{"1min":2.0049999999999999,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":0.94599999999999995}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.76900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":1.083},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.351,"15min":1.3779999999999999},"min":{"1min":0.85699999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.6819999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.6890000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3049999999999999,"15min":1.3169999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.54700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.349},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5049999999999999,"15min":1.5029999999999999},"min":{"1min":1.073,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.853,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.601}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3520000000000001,"15min":1.3819999999999999},"min":{"1min":0.97299999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.504},{"interface":"front","average":{"1min":1.385,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":0.77700000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.556}]},{"osd":7,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.5489999999999999},"min":{"1min":0.93400000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.956,"5min":2.641,"15min":2.641},"last":1.397},{"interface":"front","average":{"1min":1.536,"5min":1.579,"15min":1.603},"min":{"1min":1.246,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.1909999999999998,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.724}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46232,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325608,"statfs":{"total":96636764160,"available":96589422592,"internally_reserved":0,"allocated":1187840,"data_stored":569022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1559999999999999,"15min":1.1479999999999999},"min":{"1min":0.58999999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7689999999999999,"5min":1.881,"15min":2.0339999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.999,"5min":1.1020000000000001,"15min":1.1639999999999999},"min":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4830000000000001,"5min":1.7589999999999999,"15min":1.9199999999999999},"last":0.99299999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1479999999999999,"15min":1.177},"min":{"1min":0.58399999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.78},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.1910000000000001,"15min":1.2330000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.716,"5min":1.782,"15min":2.3690000000000002},"last":1.0620000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3700000000000001,"15min":1.399},"min":{"1min":0.68100000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.3400000000000001,"15min":1.365},"min":{"1min":0.69399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7769999999999999,"5min":1.8799999999999999,"15min":1.8839999999999999},"last":1.421}]},{"osd":3,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.5920000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.238,"5min":1.268,"15min":1.304},"min":{"1min":0.53900000000000003,"5min":0.437,"15min":0.437},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.3879999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.306,"15min":1.288},"min":{"1min":0.78800000000000003,"5min":0.61499999999999999,"15min":0.48699999999999999},"max":{"1min":1.639,"5min":1.696,"15min":1.8220000000000001},"last":1.179},{"interface":"front","average":{"1min":1.395,"5min":1.3460000000000001,"15min":1.349},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.806,"5min":1.806,"15min":1.964},"last":1.115}]},{"osd":6,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.323,"15min":1.2909999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4690000000000001,"15min":1.4610000000000001},"min":{"1min":0.74199999999999999,"5min":0.71999999999999997,"15min":0.44700000000000001},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.2890000000000001},"last":1.351}]},{"osd":7,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.4730000000000001,"15min":1.4690000000000001},"min":{"1min":0.76100000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9259999999999999},"last":1.244},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.466,"15min":1.458},"min":{"1min":0.66700000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8979999999999999,"5min":2.0579999999999998,"15min":2.29},"last":1.512}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40664,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331176,"statfs":{"total":96636764160,"available":96595124224,"internally_reserved":0,"allocated":1191936,"data_stored":569064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.50800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0609999999999999,"5min":1.4419999999999999,"15min":1.6639999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":1.0169999999999999},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.3320000000000001,"5min":1.913,"15min":1.982},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90000000000000002,"15min":0.94099999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.32500000000000001},"max":{"1min":1.337,"5min":1.7390000000000001,"15min":2.149},"last":1.095},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.089,"15min":1.139},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.415,"5min":1.6499999999999999,"15min":2.1600000000000001},"last":0.88}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2050000000000001,"15min":1.3180000000000001},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.44800000000000001},"max":{"1min":1.6830000000000001,"5min":1.895,"15min":2.1259999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.103,"5min":1.331,"15min":1.4259999999999999},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.5},"max":{"1min":1.8029999999999999,"5min":1.8660000000000001,"15min":2.6299999999999999},"last":1.0629999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1910000000000001,"15min":1.2569999999999999},"min":{"1min":0.77100000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.768,"5min":1.768,"15min":1.9239999999999999},"last":1.349},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.298,"15min":1.3300000000000001},"min":{"1min":0.93999999999999995,"5min":0.629,"15min":0.629},"max":{"1min":1.7689999999999999,"5min":1.909,"15min":1.9259999999999999},"last":1.7270000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.4550000000000001,"15min":1.516},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7170000000000001,"5min":2.0070000000000001,"15min":2.23},"last":1.786},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3899999999999999,"15min":1.468},"min":{"1min":0.90700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6220000000000001,"5min":1.95,"15min":2.3570000000000002},"last":1.2889999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.361,"15min":1.421},"min":{"1min":0.95999999999999996,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.837,"5min":2.181,"15min":2.181},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4419999999999999,"15min":1.4910000000000001},"min":{"1min":0.82599999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.6799999999999999,"5min":1.9510000000000001,"15min":2.347},"last":1.8280000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.4870000000000001,"15min":1.581},"min":{"1min":1.0069999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7310000000000001,"5min":2.1589999999999998,"15min":2.395},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.472,"15min":1.5509999999999999},"min":{"1min":0.749,"5min":0.629,"15min":0.629},"max":{"1min":1.8160000000000001,"5min":2.1000000000000001,"15min":2.1739999999999999},"last":1.5629999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575050,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44816,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327024,"statfs":{"total":96636764160,"available":96590872576,"internally_reserved":0,"allocated":1183744,"data_stored":564264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.4319999999999999,"5min":1.556,"15min":2.472},"last":1.151},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96299999999999997,"15min":1.024},"min":{"1min":0.48299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3560000000000001,"5min":1.585,"15min":2.3260000000000001},"last":1.044}]},{"osd":1,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.0029999999999999},"min":{"1min":0.50800000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.454,"5min":3.234,"15min":3.234},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.1459999999999999,"15min":1.1619999999999999},"min":{"1min":0.52200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6299999999999999,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.331}]},{"osd":3,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.1830000000000001,"15min":1.234},"min":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.2629999999999999,"15min":1.2969999999999999},"min":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6879999999999999,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.2250000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.415,"15min":1.425},"min":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.766,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.3400000000000001,"15min":1.3620000000000001},"min":{"1min":0.80400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6759999999999999,"5min":3.016,"15min":3.016},"last":1.7190000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.393,"15min":1.415},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8169999999999999,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.762},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.552},"min":{"1min":0.96599999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.3039999999999998,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.4319999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.448,"15min":1.486},"min":{"1min":0.85899999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.0129999999999999,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.887},{"interface":"front","average":{"1min":1.46,"5min":1.5,"15min":1.5289999999999999},"min":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.3610000000000002,"5min":2.5920000000000001,"15min":3.3929999999999998},"last":1.9199999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.52,"15min":1.5640000000000001},"min":{"1min":0.80700000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7789999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.617,"15min":1.6439999999999999},"min":{"1min":0.88,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.206,"5min":2.7519999999999998,"15min":2.944},"last":1.996}]}]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54568,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317272,"statfs":{"total":96636764160,"available":96580886528,"internally_reserved":0,"allocated":1204224,"data_stored":571180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.874,"15min":0.90100000000000002},"min":{"1min":0.54700000000000004,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3819999999999999,"5min":1.569,"15min":1.821},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.075,"15min":1.0980000000000001},"min":{"1min":0.40799999999999997,"5min":0.378,"15min":0.378},"max":{"1min":1.601,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.452}]},{"osd":2,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95399999999999996,"15min":0.98599999999999999},"min":{"1min":0.40300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.524,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0880000000000001,"15min":1.1599999999999999},"min":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6919999999999999,"5min":1.756,"15min":1.796},"last":1.22}]},{"osd":3,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.2749999999999999,"15min":1.3200000000000001},"min":{"1min":0.58899999999999997,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.167,"15min":1.1990000000000001},"min":{"1min":0.55700000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7509999999999999,"5min":1.8100000000000001,"15min":2.2050000000000001},"last":0.999}]},{"osd":4,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.238,"15min":1.2829999999999999},"min":{"1min":0.85399999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8700000000000001,"5min":2.069,"15min":2.069},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3700000000000001,"15min":1.419},"min":{"1min":0.82499999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.899,"5min":2.899,"15min":2.899},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.403,"15min":1.4370000000000001},"min":{"1min":0.97999999999999998,"5min":0.623,"15min":0.623},"max":{"1min":1.6499999999999999,"5min":2.032,"15min":2.3690000000000002},"last":1.353},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.371,"15min":1.421},"min":{"1min":0.84399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8779999999999999,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":1.405}]},{"osd":6,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.484,"15min":1.5389999999999999},"min":{"1min":0.96399999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.694,"5min":2.694,"15min":2.694},"last":1.498},{"interface":"front","average":{"1min":1.353,"5min":1.4910000000000001,"15min":1.538},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.73099999999999998},"max":{"1min":1.925,"5min":2.2610000000000001,"15min":2.6160000000000001},"last":1.2529999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5800000000000001,"15min":1.6160000000000001},"min":{"1min":1.052,"5min":0.90600000000000003,"15min":0.79300000000000004},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5059999999999998},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.488,"15min":1.552},"min":{"1min":0.998,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.262},"last":1.5329999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1777664,"data_stored":1159320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.875,"15min":0.94299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.7929999999999999},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.097,"15min":1.163},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.36399999999999999},"max":{"1min":1.508,"5min":1.833,"15min":1.867},"last":1.71}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.204,"15min":1.2470000000000001},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1.6359999999999999,"5min":1.887,"15min":2.2970000000000002},"last":1.994},{"interface":"front","average":{"1min":1.093,"5min":1.159,"15min":1.2050000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6000000000000001,"5min":2.012,"15min":2.2330000000000001},"last":1.847}]},{"osd":3,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.96099999999999997,"15min":1.0029999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40100000000000002},"max":{"1min":1.3149999999999999,"5min":1.889,"15min":1.9410000000000001},"last":1.044},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.181,"15min":1.238},"min":{"1min":0.45000000000000001,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.823,"5min":1.823,"15min":2.5339999999999998},"last":2.1190000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.266,"15min":1.302},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.61599999999999999},"max":{"1min":1.786,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.796},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4419999999999999,"15min":1.462},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.7809999999999999,"5min":2.032,"15min":2.6259999999999999},"last":2.4550000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.383,"15min":1.419},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.02,"5min":2.097,"15min":2.5880000000000001},"last":2.1930000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.302,"15min":1.3600000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.67900000000000005},"max":{"1min":1.778,"5min":1.778,"15min":2.4889999999999999},"last":1.492}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5389999999999999,"15min":1.573},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8819999999999999,"5min":2.097,"15min":2.7599999999999998},"last":2.8279999999999998},{"interface":"front","average":{"1min":1.46,"5min":1.4770000000000001,"15min":1.5},"min":{"1min":1.02,"5min":0.98899999999999999,"15min":0.89000000000000001},"max":{"1min":1.9299999999999999,"5min":2.1459999999999999,"15min":2.5089999999999999},"last":2.5350000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5409999999999999,"15min":1.585},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1040000000000001,"5min":2.2650000000000001,"15min":2.6509999999999998},"last":2.335},{"interface":"front","average":{"1min":1.423,"5min":1.472,"15min":1.544},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.976},"last":1.9410000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:33:10.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:33:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.037+0000 7f74d8f0a640 1 Processor -- start 2023-10-16T14:33:11.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.037+0000 7f74d8f0a640 1 -- start start 2023-10-16T14:33:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d8f0a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d406c040 0x7f74d406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d8f0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74d40cda80 0x7f74d415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d8f0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d406c960 con 0x7f74d40ad970 2023-10-16T14:33:11.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d8f0a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d415b9d0 con 0x7f74d40cda80 2023-10-16T14:33:11.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d8f0a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d415bcb0 con 0x7f74d406c040 2023-10-16T14:33:11.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74d40cda80 0x7f74d415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74d40cda80 0x7f74d415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54056/0 (socket says 172.21.15.29:54056) 2023-10-16T14:33:11.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.039+0000 7f74d3fff640 1 -- 172.21.15.29:0/2827225610 learned_addr learned my addr 172.21.15.29:0/2827225610 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:11.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d3fff640 1 -- 172.21.15.29:0/2827225610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d406c040 msgr2=0x7f74d406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:11.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d37fe640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d406c040 0x7f74d406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d3fff640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d406c040 0x7f74d406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d3fff640 1 -- 172.21.15.29:0/2827225610 >> v1:172.21.15.29:6789/0 conn(0x7f74d40ad970 legacy=0x7f74d415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d3fff640 1 -- 172.21.15.29:0/2827225610 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d415bf90 con 0x7f74d40cda80 2023-10-16T14:33:11.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d2ffd640 1 -- 172.21.15.29:0/2827225610 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248946475 0 0) 0x7f74d406c960 con 0x7f74d40ad970 2023-10-16T14:33:11.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.040+0000 7f74d37fe640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d406c040 0x7f74d406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:11.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.041+0000 7f74d3fff640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40cda80 0x7f74d415ed90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009e50 tx=0x7f74cc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5e8d7c2bbe37b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:11.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.041+0000 7f74d2ffd640 1 -- 172.21.15.29:0/2827225610 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc02cb00 con 0x7f74d40cda80 2023-10-16T14:33:11.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.041+0000 7f74d2ffd640 1 -- 172.21.15.29:0/2827225610 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc02cc60 con 0x7f74d40cda80 2023-10-16T14:33:11.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.041+0000 7f74d2ffd640 1 -- 172.21.15.29:0/2827225610 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0343f0 con 0x7f74d40cda80 2023-10-16T14:33:11.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 -- 172.21.15.29:0/2827225610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40cda80 msgr2=0x7f74d415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40cda80 0x7f74d415ed90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009e50 tx=0x7f74cc02bf60 comp rx=0 tx=0).stop 2023-10-16T14:33:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 -- 172.21.15.29:0/2827225610 shutdown_connections 2023-10-16T14:33:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d40cda80 0x7f74d415ed90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/2827225610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d406c040 0x7f74d406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 -- 172.21.15.29:0/2827225610 >> 172.21.15.29:0/2827225610 conn(0x7f74d40ce6a0 msgr2=0x7f74d40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 -- 172.21.15.29:0/2827225610 shutdown_connections 2023-10-16T14:33:11.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.042+0000 7f74d8f0a640 1 -- 172.21.15.29:0/2827225610 wait complete. 2023-10-16T14:33:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.044+0000 7f74d8f0a640 1 Processor -- start 2023-10-16T14:33:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.044+0000 7f74d8f0a640 1 -- start start 2023-10-16T14:33:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d8f0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 0x7f74d41f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d8f0a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d40ad970 0x7f74d41542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d8f0a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40cda80 0x7f74d4156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d8f0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4155450 con 0x7f74d406c040 2023-10-16T14:33:11.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d8f0a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d41531c0 con 0x7f74d40ad970 2023-10-16T14:33:11.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d8f0a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4155170 con 0x7f74d40cda80 2023-10-16T14:33:11.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 0x7f74d41f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 0x7f74d41f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54070/0 (socket says 172.21.15.29:54070) 2023-10-16T14:33:11.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 -- 172.21.15.29:0/4069128237 learned_addr learned my addr 172.21.15.29:0/4069128237 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:11.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 -- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40cda80 msgr2=0x7f74d4156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40cda80 0x7f74d4156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 -- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d40ad970 msgr2=0x7f74d41542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.045+0000 7f74d37fe640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d40ad970 0x7f74d41542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d37fe640 1 -- 172.21.15.29:0/4069128237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc009b30 con 0x7f74d406c040 2023-10-16T14:33:11.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d37fe640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 0x7f74d41f1bf0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c7f0 tx=0x7f74c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:11.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4007820 con 0x7f74d406c040 2023-10-16T14:33:11.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4007980 con 0x7f74d406c040 2023-10-16T14:33:11.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4007cf0 con 0x7f74d406c040 2023-10-16T14:33:11.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d4152b70 con 0x7f74d406c040 2023-10-16T14:33:11.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.046+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d41f2130 con 0x7f74d406c040 2023-10-16T14:33:11.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.047+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d406b330 con 0x7f74d406c040 2023-10-16T14:33:11.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.048+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74c40113f0 con 0x7f74d406c040 2023-10-16T14:33:11.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.049+0000 7f74d0ff9640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f74b406af90 0x7f74b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.049+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74c4089bb0 con 0x7f74d406c040 2023-10-16T14:33:11.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.050+0000 7f74cbfff640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f74b406af90 0x7f74b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.051+0000 7f74cbfff640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f74b406af90 0x7f74b406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74d40c5c70 tx=0x7f74bc007930 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:11.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.054+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74c4054a40 con 0x7f74d406c040 2023-10-16T14:33:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.240+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 --> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74d40cc550 con 0x7f74b406af90 2023-10-16T14:33:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.263+0000 7f74d0ff9640 1 -- 172.21.15.29:0/4069128237 <== mgr.5390 v2:172.21.15.170:6832/610139797 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286667 (secure 0 0 0) 0x7f74bc005070 con 0x7f74b406af90 2023-10-16T14:33:11.269 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:11.274 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:33:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f74b406af90 msgr2=0x7f74b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f74b406af90 0x7f74b406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74d40c5c70 tx=0x7f74bc007930 comp rx=0 tx=0).stop 2023-10-16T14:33:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 msgr2=0x7f74d41f1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 0x7f74d41f1bf0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c7f0 tx=0x7f74c400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 shutdown_connections 2023-10-16T14:33:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74d40cda80 0x7f74d4156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74d40ad970 0x7f74d41542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f74b406af90 0x7f74b406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 --2- 172.21.15.29:0/4069128237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d406c040 0x7f74d41f1bf0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 >> 172.21.15.29:0/4069128237 conn(0x7f74d40ce6a0 msgr2=0x7f74d4159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:11.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.269+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 shutdown_connections 2023-10-16T14:33:11.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.270+0000 7f74d8f0a640 1 -- 172.21.15.29:0/4069128237 wait complete. 2023-10-16T14:33:11.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T14:33:09.570350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":307,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337600,"kb_used_data":10448,"kb_used_omap":12,"kb_used_meta":327027,"kb_avail":754637120,"statfs":{"total":773094113280,"available":772748410880,"internally_reserved":0,"allocated":10698752,"data_stored":5724572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":334876217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.342752"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254163+0000","last_change":"2023-10-16T14:33:06.280994+0000","last_active":"2023-10-16T14:33:07.254163+0000","last_peered":"2023-10-16T14:33:07.254163+0000","last_clean":"2023-10-16T14:33:07.254163+0000","last_became_active":"2023-10-16T14:33:06.280696+0000","last_became_peered":"2023-10-16T14:33:06.280696+0000","last_unstale":"2023-10-16T14:33:07.254163+0000","last_undegraded":"2023-10-16T14:33:07.254163+0000","last_fullsized":"2023-10-16T14:33:07.254163+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:50.049165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987178+0000","last_change":"2023-10-16T14:33:06.286651+0000","last_active":"2023-10-16T14:33:07.987178+0000","last_peered":"2023-10-16T14:33:07.987178+0000","last_clean":"2023-10-16T14:33:07.987178+0000","last_became_active":"2023-10-16T14:33:06.286052+0000","last_became_peered":"2023-10-16T14:33:06.286052+0000","last_unstale":"2023-10-16T14:33:07.987178+0000","last_undegraded":"2023-10-16T14:33:07.987178+0000","last_fullsized":"2023-10-16T14:33:07.987178+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:10.219830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988340+0000","last_change":"2023-10-16T14:33:06.277653+0000","last_active":"2023-10-16T14:33:07.988340+0000","last_peered":"2023-10-16T14:33:07.988340+0000","last_clean":"2023-10-16T14:33:07.988340+0000","last_became_active":"2023-10-16T14:33:06.274841+0000","last_became_peered":"2023-10-16T14:33:06.274841+0000","last_unstale":"2023-10-16T14:33:07.988340+0000","last_undegraded":"2023-10-16T14:33:07.988340+0000","last_fullsized":"2023-10-16T14:33:07.988340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:51.862361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985423+0000","last_change":"2023-10-16T14:33:06.289645+0000","last_active":"2023-10-16T14:33:07.985423+0000","last_peered":"2023-10-16T14:33:07.985423+0000","last_clean":"2023-10-16T14:33:07.985423+0000","last_became_active":"2023-10-16T14:33:06.288437+0000","last_became_peered":"2023-10-16T14:33:06.288437+0000","last_unstale":"2023-10-16T14:33:07.985423+0000","last_undegraded":"2023-10-16T14:33:07.985423+0000","last_fullsized":"2023-10-16T14:33:07.985423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:29.975227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989189+0000","last_change":"2023-10-16T14:33:06.285412+0000","last_active":"2023-10-16T14:33:07.989189+0000","last_peered":"2023-10-16T14:33:07.989189+0000","last_clean":"2023-10-16T14:33:07.989189+0000","last_became_active":"2023-10-16T14:33:06.284238+0000","last_became_peered":"2023-10-16T14:33:06.284238+0000","last_unstale":"2023-10-16T14:33:07.989189+0000","last_undegraded":"2023-10-16T14:33:07.989189+0000","last_fullsized":"2023-10-16T14:33:07.989189+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:29.975227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986095+0000","last_change":"2023-10-16T14:33:06.285477+0000","last_active":"2023-10-16T14:33:07.986095+0000","last_peered":"2023-10-16T14:33:07.986095+0000","last_clean":"2023-10-16T14:33:07.986095+0000","last_became_active":"2023-10-16T14:33:06.285115+0000","last_became_peered":"2023-10-16T14:33:06.285115+0000","last_unstale":"2023-10-16T14:33:07.986095+0000","last_undegraded":"2023-10-16T14:33:07.986095+0000","last_fullsized":"2023-10-16T14:33:07.986095+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:01.101300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985342+0000","last_change":"2023-10-16T14:33:06.275068+0000","last_active":"2023-10-16T14:33:07.985342+0000","last_peered":"2023-10-16T14:33:07.985342+0000","last_clean":"2023-10-16T14:33:07.985342+0000","last_became_active":"2023-10-16T14:33:06.274725+0000","last_became_peered":"2023-10-16T14:33:06.274725+0000","last_unstale":"2023-10-16T14:33:07.985342+0000","last_undegraded":"2023-10-16T14:33:07.985342+0000","last_fullsized":"2023-10-16T14:33:07.985342+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:42.540472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986525+0000","last_change":"2023-10-16T14:33:06.278379+0000","last_active":"2023-10-16T14:33:07.986525+0000","last_peered":"2023-10-16T14:33:07.986525+0000","last_clean":"2023-10-16T14:33:07.986525+0000","last_became_active":"2023-10-16T14:33:06.277921+0000","last_became_peered":"2023-10-16T14:33:06.277921+0000","last_unstale":"2023-10-16T14:33:07.986525+0000","last_undegraded":"2023-10-16T14:33:07.986525+0000","last_fullsized":"2023-10-16T14:33:07.986525+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:20.033542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989830+0000","last_change":"2023-10-16T14:33:06.286928+0000","last_active":"2023-10-16T14:33:07.989830+0000","last_peered":"2023-10-16T14:33:07.989830+0000","last_clean":"2023-10-16T14:33:07.989830+0000","last_became_active":"2023-10-16T14:33:06.285868+0000","last_became_peered":"2023-10-16T14:33:06.285868+0000","last_unstale":"2023-10-16T14:33:07.989830+0000","last_undegraded":"2023-10-16T14:33:07.989830+0000","last_fullsized":"2023-10-16T14:33:07.989830+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:04.523835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253922+0000","last_change":"2023-10-16T14:33:06.279865+0000","last_active":"2023-10-16T14:33:07.253922+0000","last_peered":"2023-10-16T14:33:07.253922+0000","last_clean":"2023-10-16T14:33:07.253922+0000","last_became_active":"2023-10-16T14:33:06.279562+0000","last_became_peered":"2023-10-16T14:33:06.279562+0000","last_unstale":"2023-10-16T14:33:07.253922+0000","last_undegraded":"2023-10-16T14:33:07.253922+0000","last_fullsized":"2023-10-16T14:33:07.253922+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:40.265140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253487+0000","last_change":"2023-10-16T14:33:06.285269+0000","last_active":"2023-10-16T14:33:07.253487+0000","last_peered":"2023-10-16T14:33:07.253487+0000","last_clean":"2023-10-16T14:33:07.253487+0000","last_became_active":"2023-10-16T14:33:06.283208+0000","last_became_peered":"2023-10-16T14:33:06.283208+0000","last_unstale":"2023-10-16T14:33:07.253487+0000","last_undegraded":"2023-10-16T14:33:07.253487+0000","last_fullsized":"2023-10-16T14:33:07.253487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:22.731595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985830+0000","last_change":"2023-10-16T14:33:06.284062+0000","last_active":"2023-10-16T14:33:07.985830+0000","last_peered":"2023-10-16T14:33:07.985830+0000","last_clean":"2023-10-16T14:33:07.985830+0000","last_became_active":"2023-10-16T14:33:06.282483+0000","last_became_peered":"2023-10-16T14:33:06.282483+0000","last_unstale":"2023-10-16T14:33:07.985830+0000","last_undegraded":"2023-10-16T14:33:07.985830+0000","last_fullsized":"2023-10-16T14:33:07.985830+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:31.878477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248564+0000","last_change":"2023-10-16T14:33:06.290607+0000","last_active":"2023-10-16T14:33:07.248564+0000","last_peered":"2023-10-16T14:33:07.248564+0000","last_clean":"2023-10-16T14:33:07.248564+0000","last_became_active":"2023-10-16T14:33:06.290263+0000","last_became_peered":"2023-10-16T14:33:06.290263+0000","last_unstale":"2023-10-16T14:33:07.248564+0000","last_undegraded":"2023-10-16T14:33:07.248564+0000","last_fullsized":"2023-10-16T14:33:07.248564+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:20.033542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987913+0000","last_change":"2023-10-16T14:33:06.276849+0000","last_active":"2023-10-16T14:33:07.987913+0000","last_peered":"2023-10-16T14:33:07.987913+0000","last_clean":"2023-10-16T14:33:07.987913+0000","last_became_active":"2023-10-16T14:33:06.276498+0000","last_became_peered":"2023-10-16T14:33:06.276498+0000","last_unstale":"2023-10-16T14:33:07.987913+0000","last_undegraded":"2023-10-16T14:33:07.987913+0000","last_fullsized":"2023-10-16T14:33:07.987913+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:51.501988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248015+0000","last_change":"2023-10-16T14:33:06.280052+0000","last_active":"2023-10-16T14:33:07.248015+0000","last_peered":"2023-10-16T14:33:07.248015+0000","last_clean":"2023-10-16T14:33:07.248015+0000","last_became_active":"2023-10-16T14:33:06.279520+0000","last_became_peered":"2023-10-16T14:33:06.279520+0000","last_unstale":"2023-10-16T14:33:07.248015+0000","last_undegraded":"2023-10-16T14:33:07.248015+0000","last_fullsized":"2023-10-16T14:33:07.248015+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:12.449231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.250843+0000","last_change":"2023-10-16T14:33:06.283574+0000","last_active":"2023-10-16T14:33:07.250843+0000","last_peered":"2023-10-16T14:33:07.250843+0000","last_clean":"2023-10-16T14:33:07.250843+0000","last_became_active":"2023-10-16T14:33:06.282123+0000","last_became_peered":"2023-10-16T14:33:06.282123+0000","last_unstale":"2023-10-16T14:33:07.250843+0000","last_undegraded":"2023-10-16T14:33:07.250843+0000","last_fullsized":"2023-10-16T14:33:07.250843+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:23.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989834+0000","last_change":"2023-10-16T14:33:06.286917+0000","last_active":"2023-10-16T14:33:07.989834+0000","last_peered":"2023-10-16T14:33:07.989834+0000","last_clean":"2023-10-16T14:33:07.989834+0000","last_became_active":"2023-10-16T14:33:06.286215+0000","last_became_peered":"2023-10-16T14:33:06.286215+0000","last_unstale":"2023-10-16T14:33:07.989834+0000","last_undegraded":"2023-10-16T14:33:07.989834+0000","last_fullsized":"2023-10-16T14:33:07.989834+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:31.878477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988983+0000","last_change":"2023-10-16T14:33:06.282367+0000","last_active":"2023-10-16T14:33:07.988983+0000","last_peered":"2023-10-16T14:33:07.988983+0000","last_clean":"2023-10-16T14:33:07.988983+0000","last_became_active":"2023-10-16T14:33:06.279214+0000","last_became_peered":"2023-10-16T14:33:06.279214+0000","last_unstale":"2023-10-16T14:33:07.988983+0000","last_undegraded":"2023-10-16T14:33:07.988983+0000","last_fullsized":"2023-10-16T14:33:07.988983+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:29.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985745+0000","last_change":"2023-10-16T14:33:06.285861+0000","last_active":"2023-10-16T14:33:07.985745+0000","last_peered":"2023-10-16T14:33:07.985745+0000","last_clean":"2023-10-16T14:33:07.985745+0000","last_became_active":"2023-10-16T14:33:06.285567+0000","last_became_peered":"2023-10-16T14:33:06.285567+0000","last_unstale":"2023-10-16T14:33:07.985745+0000","last_undegraded":"2023-10-16T14:33:07.985745+0000","last_fullsized":"2023-10-16T14:33:07.985745+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:12.449231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248314+0000","last_change":"2023-10-16T14:33:06.269884+0000","last_active":"2023-10-16T14:33:07.248314+0000","last_peered":"2023-10-16T14:33:07.248314+0000","last_clean":"2023-10-16T14:33:07.248314+0000","last_became_active":"2023-10-16T14:33:06.269555+0000","last_became_peered":"2023-10-16T14:33:06.269555+0000","last_unstale":"2023-10-16T14:33:07.248314+0000","last_undegraded":"2023-10-16T14:33:07.248314+0000","last_fullsized":"2023-10-16T14:33:07.248314+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:54.119904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252608+0000","last_change":"2023-10-16T14:33:06.259400+0000","last_active":"2023-10-16T14:33:07.252608+0000","last_peered":"2023-10-16T14:33:07.252608+0000","last_clean":"2023-10-16T14:33:07.252608+0000","last_became_active":"2023-10-16T14:33:06.258423+0000","last_became_peered":"2023-10-16T14:33:06.258423+0000","last_unstale":"2023-10-16T14:33:07.252608+0000","last_undegraded":"2023-10-16T14:33:07.252608+0000","last_fullsized":"2023-10-16T14:33:07.252608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:09.960593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252622+0000","last_change":"2023-10-16T14:33:06.282134+0000","last_active":"2023-10-16T14:33:07.252622+0000","last_peered":"2023-10-16T14:33:07.252622+0000","last_clean":"2023-10-16T14:33:07.252622+0000","last_became_active":"2023-10-16T14:33:06.281392+0000","last_became_peered":"2023-10-16T14:33:06.281392+0000","last_unstale":"2023-10-16T14:33:07.252622+0000","last_undegraded":"2023-10-16T14:33:07.252622+0000","last_fullsized":"2023-10-16T14:33:07.252622+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:48.026556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254107+0000","last_change":"2023-10-16T14:33:06.271190+0000","last_active":"2023-10-16T14:33:07.254107+0000","last_peered":"2023-10-16T14:33:07.254107+0000","last_clean":"2023-10-16T14:33:07.254107+0000","last_became_active":"2023-10-16T14:33:06.270927+0000","last_became_peered":"2023-10-16T14:33:06.270927+0000","last_unstale":"2023-10-16T14:33:07.254107+0000","last_undegraded":"2023-10-16T14:33:07.254107+0000","last_fullsized":"2023-10-16T14:33:07.254107+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:35.400724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.251969+0000","last_change":"2023-10-16T14:33:06.281005+0000","last_active":"2023-10-16T14:33:07.251969+0000","last_peered":"2023-10-16T14:33:07.251969+0000","last_clean":"2023-10-16T14:33:07.251969+0000","last_became_active":"2023-10-16T14:33:06.280720+0000","last_became_peered":"2023-10-16T14:33:06.280720+0000","last_unstale":"2023-10-16T14:33:07.251969+0000","last_undegraded":"2023-10-16T14:33:07.251969+0000","last_fullsized":"2023-10-16T14:33:07.251969+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:11.766477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254612+0000","last_change":"2023-10-16T14:33:06.265626+0000","last_active":"2023-10-16T14:33:07.254612+0000","last_peered":"2023-10-16T14:33:07.254612+0000","last_clean":"2023-10-16T14:33:07.254612+0000","last_became_active":"2023-10-16T14:33:06.265212+0000","last_became_peered":"2023-10-16T14:33:06.265212+0000","last_unstale":"2023-10-16T14:33:07.254612+0000","last_undegraded":"2023-10-16T14:33:07.254612+0000","last_fullsized":"2023-10-16T14:33:07.254612+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:20.944155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985872+0000","last_change":"2023-10-16T14:33:06.280574+0000","last_active":"2023-10-16T14:33:07.985872+0000","last_peered":"2023-10-16T14:33:07.985872+0000","last_clean":"2023-10-16T14:33:07.985872+0000","last_became_active":"2023-10-16T14:33:06.280309+0000","last_became_peered":"2023-10-16T14:33:06.280309+0000","last_unstale":"2023-10-16T14:33:07.985872+0000","last_undegraded":"2023-10-16T14:33:07.985872+0000","last_fullsized":"2023-10-16T14:33:07.985872+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:00.545089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988911+0000","last_change":"2023-10-16T14:33:06.292587+0000","last_active":"2023-10-16T14:33:07.988911+0000","last_peered":"2023-10-16T14:33:07.988911+0000","last_clean":"2023-10-16T14:33:07.988911+0000","last_became_active":"2023-10-16T14:33:06.290219+0000","last_became_peered":"2023-10-16T14:33:06.290219+0000","last_unstale":"2023-10-16T14:33:07.988911+0000","last_undegraded":"2023-10-16T14:33:07.988911+0000","last_fullsized":"2023-10-16T14:33:07.988911+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:25.409664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252380+0000","last_change":"2023-10-16T14:33:06.280855+0000","last_active":"2023-10-16T14:33:07.252380+0000","last_peered":"2023-10-16T14:33:07.252380+0000","last_clean":"2023-10-16T14:33:07.252380+0000","last_became_active":"2023-10-16T14:33:06.279199+0000","last_became_peered":"2023-10-16T14:33:06.279199+0000","last_unstale":"2023-10-16T14:33:07.252380+0000","last_undegraded":"2023-10-16T14:33:07.252380+0000","last_fullsized":"2023-10-16T14:33:07.252380+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:22.558261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988097+0000","last_change":"2023-10-16T14:33:06.281075+0000","last_active":"2023-10-16T14:33:07.988097+0000","last_peered":"2023-10-16T14:33:07.988097+0000","last_clean":"2023-10-16T14:33:07.988097+0000","last_became_active":"2023-10-16T14:33:06.280782+0000","last_became_peered":"2023-10-16T14:33:06.280782+0000","last_unstale":"2023-10-16T14:33:07.988097+0000","last_undegraded":"2023-10-16T14:33:07.988097+0000","last_fullsized":"2023-10-16T14:33:07.988097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:06.225751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984506+0000","last_change":"2023-10-16T14:33:06.285547+0000","last_active":"2023-10-16T14:33:07.984506+0000","last_peered":"2023-10-16T14:33:07.984506+0000","last_clean":"2023-10-16T14:33:07.984506+0000","last_became_active":"2023-10-16T14:33:06.284195+0000","last_became_peered":"2023-10-16T14:33:06.284195+0000","last_unstale":"2023-10-16T14:33:07.984506+0000","last_undegraded":"2023-10-16T14:33:07.984506+0000","last_fullsized":"2023-10-16T14:33:07.984506+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:23.285738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984810+0000","last_change":"2023-10-16T14:33:06.279779+0000","last_active":"2023-10-16T14:33:07.984810+0000","last_peered":"2023-10-16T14:33:07.984810+0000","last_clean":"2023-10-16T14:33:07.984810+0000","last_became_active":"2023-10-16T14:33:06.279483+0000","last_became_peered":"2023-10-16T14:33:06.279483+0000","last_unstale":"2023-10-16T14:33:07.984810+0000","last_undegraded":"2023-10-16T14:33:07.984810+0000","last_fullsized":"2023-10-16T14:33:07.984810+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:54.491439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987030+0000","last_change":"2023-10-16T14:33:06.287674+0000","last_active":"2023-10-16T14:33:07.987030+0000","last_peered":"2023-10-16T14:33:07.987030+0000","last_clean":"2023-10-16T14:33:07.987030+0000","last_became_active":"2023-10-16T14:33:06.287244+0000","last_became_peered":"2023-10-16T14:33:06.287244+0000","last_unstale":"2023-10-16T14:33:07.987030+0000","last_undegraded":"2023-10-16T14:33:07.987030+0000","last_fullsized":"2023-10-16T14:33:07.987030+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:34.170987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254103+0000","last_change":"2023-10-16T14:33:06.280569+0000","last_active":"2023-10-16T14:33:07.254103+0000","last_peered":"2023-10-16T14:33:07.254103+0000","last_clean":"2023-10-16T14:33:07.254103+0000","last_became_active":"2023-10-16T14:33:06.280252+0000","last_became_peered":"2023-10-16T14:33:06.280252+0000","last_unstale":"2023-10-16T14:33:07.254103+0000","last_undegraded":"2023-10-16T14:33:07.254103+0000","last_fullsized":"2023-10-16T14:33:07.254103+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:57.083160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.255067+0000","last_change":"2023-10-16T14:33:05.273054+0000","last_active":"2023-10-16T14:33:07.255067+0000","last_peered":"2023-10-16T14:33:07.255067+0000","last_clean":"2023-10-16T14:33:07.255067+0000","last_became_active":"2023-10-16T14:33:05.272368+0000","last_became_peered":"2023-10-16T14:33:05.272368+0000","last_unstale":"2023-10-16T14:33:07.255067+0000","last_undegraded":"2023-10-16T14:33:07.255067+0000","last_fullsized":"2023-10-16T14:33:07.255067+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.906363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.990022+0000","last_change":"2023-10-16T14:33:06.287094+0000","last_active":"2023-10-16T14:33:07.990022+0000","last_peered":"2023-10-16T14:33:07.990022+0000","last_clean":"2023-10-16T14:33:07.990022+0000","last_became_active":"2023-10-16T14:33:06.286622+0000","last_became_peered":"2023-10-16T14:33:06.286622+0000","last_unstale":"2023-10-16T14:33:07.990022+0000","last_undegraded":"2023-10-16T14:33:07.990022+0000","last_fullsized":"2023-10-16T14:33:07.990022+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:28.644767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248761+0000","last_change":"2023-10-16T14:33:05.263975+0000","last_active":"2023-10-16T14:33:07.248761+0000","last_peered":"2023-10-16T14:33:07.248761+0000","last_clean":"2023-10-16T14:33:07.248761+0000","last_became_active":"2023-10-16T14:33:05.263695+0000","last_became_peered":"2023-10-16T14:33:05.263695+0000","last_unstale":"2023-10-16T14:33:07.248761+0000","last_undegraded":"2023-10-16T14:33:07.248761+0000","last_fullsized":"2023-10-16T14:33:07.248761+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:21.593041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986364+0000","last_change":"2023-10-16T14:33:06.290319+0000","last_active":"2023-10-16T14:33:07.986364+0000","last_peered":"2023-10-16T14:33:07.986364+0000","last_clean":"2023-10-16T14:33:07.986364+0000","last_became_active":"2023-10-16T14:33:06.289380+0000","last_became_peered":"2023-10-16T14:33:06.289380+0000","last_unstale":"2023-10-16T14:33:07.986364+0000","last_undegraded":"2023-10-16T14:33:07.986364+0000","last_fullsized":"2023-10-16T14:33:07.986364+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:54.119904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.255291+0000","last_change":"2023-10-16T14:33:05.270881+0000","last_active":"2023-10-16T14:33:07.255291+0000","last_peered":"2023-10-16T14:33:07.255291+0000","last_clean":"2023-10-16T14:33:07.255291+0000","last_became_active":"2023-10-16T14:33:05.270584+0000","last_became_peered":"2023-10-16T14:33:05.270584+0000","last_unstale":"2023-10-16T14:33:07.255291+0000","last_undegraded":"2023-10-16T14:33:07.255291+0000","last_fullsized":"2023-10-16T14:33:07.255291+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:13.125033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253558+0000","last_change":"2023-10-16T14:33:06.266392+0000","last_active":"2023-10-16T14:33:07.253558+0000","last_peered":"2023-10-16T14:33:07.253558+0000","last_clean":"2023-10-16T14:33:07.253558+0000","last_became_active":"2023-10-16T14:33:06.266025+0000","last_became_peered":"2023-10-16T14:33:06.266025+0000","last_unstale":"2023-10-16T14:33:07.253558+0000","last_undegraded":"2023-10-16T14:33:07.253558+0000","last_fullsized":"2023-10-16T14:33:07.253558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:05.567929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986880+0000","last_change":"2023-10-16T14:33:05.269053+0000","last_active":"2023-10-16T14:33:07.986880+0000","last_peered":"2023-10-16T14:33:07.986880+0000","last_clean":"2023-10-16T14:33:07.986880+0000","last_became_active":"2023-10-16T14:33:05.268500+0000","last_became_peered":"2023-10-16T14:33:05.268500+0000","last_unstale":"2023-10-16T14:33:07.986880+0000","last_undegraded":"2023-10-16T14:33:07.986880+0000","last_fullsized":"2023-10-16T14:33:07.986880+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:11.097913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984859+0000","last_change":"2023-10-16T14:33:06.270746+0000","last_active":"2023-10-16T14:33:07.984859+0000","last_peered":"2023-10-16T14:33:07.984859+0000","last_clean":"2023-10-16T14:33:07.984859+0000","last_became_active":"2023-10-16T14:33:06.270428+0000","last_became_peered":"2023-10-16T14:33:06.270428+0000","last_unstale":"2023-10-16T14:33:07.984859+0000","last_undegraded":"2023-10-16T14:33:07.984859+0000","last_fullsized":"2023-10-16T14:33:07.984859+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:57.342973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988733+0000","last_change":"2023-10-16T14:33:05.272771+0000","last_active":"2023-10-16T14:33:07.988733+0000","last_peered":"2023-10-16T14:33:07.988733+0000","last_clean":"2023-10-16T14:33:07.988733+0000","last_became_active":"2023-10-16T14:33:05.272448+0000","last_became_peered":"2023-10-16T14:33:05.272448+0000","last_unstale":"2023-10-16T14:33:07.988733+0000","last_undegraded":"2023-10-16T14:33:07.988733+0000","last_fullsized":"2023-10-16T14:33:07.988733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:56.068730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.252708+0000","last_change":"2023-10-16T14:33:06.271512+0000","last_active":"2023-10-16T14:33:07.252708+0000","last_peered":"2023-10-16T14:33:07.252708+0000","last_clean":"2023-10-16T14:33:07.252708+0000","last_became_active":"2023-10-16T14:33:06.271071+0000","last_became_peered":"2023-10-16T14:33:06.271071+0000","last_unstale":"2023-10-16T14:33:07.252708+0000","last_undegraded":"2023-10-16T14:33:07.252708+0000","last_fullsized":"2023-10-16T14:33:07.252708+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:36.960370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985481+0000","last_change":"2023-10-16T14:33:05.267089+0000","last_active":"2023-10-16T14:33:07.985481+0000","last_peered":"2023-10-16T14:33:07.985481+0000","last_clean":"2023-10-16T14:33:07.985481+0000","last_became_active":"2023-10-16T14:33:05.266565+0000","last_became_peered":"2023-10-16T14:33:05.266565+0000","last_unstale":"2023-10-16T14:33:07.985481+0000","last_undegraded":"2023-10-16T14:33:07.985481+0000","last_fullsized":"2023-10-16T14:33:07.985481+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:29.558673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984598+0000","last_change":"2023-10-16T14:33:06.279029+0000","last_active":"2023-10-16T14:33:07.984598+0000","last_peered":"2023-10-16T14:33:07.984598+0000","last_clean":"2023-10-16T14:33:07.984598+0000","last_became_active":"2023-10-16T14:33:06.278629+0000","last_became_peered":"2023-10-16T14:33:06.278629+0000","last_unstale":"2023-10-16T14:33:07.984598+0000","last_undegraded":"2023-10-16T14:33:07.984598+0000","last_fullsized":"2023-10-16T14:33:07.984598+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:26.091375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984823+0000","last_change":"2023-10-16T14:33:05.273729+0000","last_active":"2023-10-16T14:33:07.984823+0000","last_peered":"2023-10-16T14:33:07.984823+0000","last_clean":"2023-10-16T14:33:07.984823+0000","last_became_active":"2023-10-16T14:33:05.273320+0000","last_became_peered":"2023-10-16T14:33:05.273320+0000","last_unstale":"2023-10-16T14:33:07.984823+0000","last_undegraded":"2023-10-16T14:33:07.984823+0000","last_fullsized":"2023-10-16T14:33:07.984823+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:04.128955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248173+0000","last_change":"2023-10-16T14:33:06.260963+0000","last_active":"2023-10-16T14:33:07.248173+0000","last_peered":"2023-10-16T14:33:07.248173+0000","last_clean":"2023-10-16T14:33:07.248173+0000","last_became_active":"2023-10-16T14:33:06.260201+0000","last_became_peered":"2023-10-16T14:33:06.260201+0000","last_unstale":"2023-10-16T14:33:07.248173+0000","last_undegraded":"2023-10-16T14:33:07.248173+0000","last_fullsized":"2023-10-16T14:33:07.248173+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:01.101300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984506+0000","last_change":"2023-10-16T14:33:05.259428+0000","last_active":"2023-10-16T14:33:07.984506+0000","last_peered":"2023-10-16T14:33:07.984506+0000","last_clean":"2023-10-16T14:33:07.984506+0000","last_became_active":"2023-10-16T14:33:05.259064+0000","last_became_peered":"2023-10-16T14:33:05.259064+0000","last_unstale":"2023-10-16T14:33:07.984506+0000","last_undegraded":"2023-10-16T14:33:07.984506+0000","last_fullsized":"2023-10-16T14:33:07.984506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:19.019112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986198+0000","last_change":"2023-10-16T14:33:06.273488+0000","last_active":"2023-10-16T14:33:07.986198+0000","last_peered":"2023-10-16T14:33:07.986198+0000","last_clean":"2023-10-16T14:33:07.986198+0000","last_became_active":"2023-10-16T14:33:06.270425+0000","last_became_peered":"2023-10-16T14:33:06.270425+0000","last_unstale":"2023-10-16T14:33:07.986198+0000","last_undegraded":"2023-10-16T14:33:07.986198+0000","last_fullsized":"2023-10-16T14:33:07.986198+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:37.480286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254709+0000","last_change":"2023-10-16T14:33:05.268560+0000","last_active":"2023-10-16T14:33:07.254709+0000","last_peered":"2023-10-16T14:33:07.254709+0000","last_clean":"2023-10-16T14:33:07.254709+0000","last_became_active":"2023-10-16T14:33:05.267645+0000","last_became_peered":"2023-10-16T14:33:05.267645+0000","last_unstale":"2023-10-16T14:33:07.254709+0000","last_undegraded":"2023-10-16T14:33:07.254709+0000","last_fullsized":"2023-10-16T14:33:07.254709+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:22:20.778752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254517+0000","last_change":"2023-10-16T14:33:06.294561+0000","last_active":"2023-10-16T14:33:07.254517+0000","last_peered":"2023-10-16T14:33:07.254517+0000","last_clean":"2023-10-16T14:33:07.254517+0000","last_became_active":"2023-10-16T14:33:06.292779+0000","last_became_peered":"2023-10-16T14:33:06.292779+0000","last_unstale":"2023-10-16T14:33:07.254517+0000","last_undegraded":"2023-10-16T14:33:07.254517+0000","last_fullsized":"2023-10-16T14:33:07.254517+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:06.225751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987096+0000","last_change":"2023-10-16T14:33:05.271493+0000","last_active":"2023-10-16T14:33:07.987096+0000","last_peered":"2023-10-16T14:33:07.987096+0000","last_clean":"2023-10-16T14:33:07.987096+0000","last_became_active":"2023-10-16T14:33:05.270940+0000","last_became_peered":"2023-10-16T14:33:05.270940+0000","last_unstale":"2023-10-16T14:33:07.987096+0000","last_undegraded":"2023-10-16T14:33:07.987096+0000","last_fullsized":"2023-10-16T14:33:07.987096+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:33.793509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986754+0000","last_change":"2023-10-16T14:33:06.290978+0000","last_active":"2023-10-16T14:33:07.986754+0000","last_peered":"2023-10-16T14:33:07.986754+0000","last_clean":"2023-10-16T14:33:07.986754+0000","last_became_active":"2023-10-16T14:33:06.289599+0000","last_became_peered":"2023-10-16T14:33:06.289599+0000","last_unstale":"2023-10-16T14:33:07.986754+0000","last_undegraded":"2023-10-16T14:33:07.986754+0000","last_fullsized":"2023-10-16T14:33:07.986754+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:02.431469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"96'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989087+0000","last_change":"2023-10-16T14:33:05.267126+0000","last_active":"2023-10-16T14:33:07.989087+0000","last_peered":"2023-10-16T14:33:07.989087+0000","last_clean":"2023-10-16T14:33:07.989087+0000","last_became_active":"2023-10-16T14:33:05.266702+0000","last_became_peered":"2023-10-16T14:33:05.266702+0000","last_unstale":"2023-10-16T14:33:07.989087+0000","last_undegraded":"2023-10-16T14:33:07.989087+0000","last_fullsized":"2023-10-16T14:33:07.989087+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:34.386293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987285+0000","last_change":"2023-10-16T14:33:06.276963+0000","last_active":"2023-10-16T14:33:07.987285+0000","last_peered":"2023-10-16T14:33:07.987285+0000","last_clean":"2023-10-16T14:33:07.987285+0000","last_became_active":"2023-10-16T14:33:06.276673+0000","last_became_peered":"2023-10-16T14:33:06.276673+0000","last_unstale":"2023-10-16T14:33:07.987285+0000","last_undegraded":"2023-10-16T14:33:07.987285+0000","last_fullsized":"2023-10-16T14:33:07.987285+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:59.522562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"96'6","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987165+0000","last_change":"2023-10-16T14:33:05.262619+0000","last_active":"2023-10-16T14:33:07.987165+0000","last_peered":"2023-10-16T14:33:07.987165+0000","last_clean":"2023-10-16T14:33:07.987165+0000","last_became_active":"2023-10-16T14:33:05.262101+0000","last_became_peered":"2023-10-16T14:33:05.262101+0000","last_unstale":"2023-10-16T14:33:07.987165+0000","last_undegraded":"2023-10-16T14:33:07.987165+0000","last_fullsized":"2023-10-16T14:33:07.987165+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:51.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253217+0000","last_change":"2023-10-16T14:33:06.281272+0000","last_active":"2023-10-16T14:33:07.253217+0000","last_peered":"2023-10-16T14:33:07.253217+0000","last_clean":"2023-10-16T14:33:07.253217+0000","last_became_active":"2023-10-16T14:33:06.281005+0000","last_became_peered":"2023-10-16T14:33:06.281005+0000","last_unstale":"2023-10-16T14:33:07.253217+0000","last_undegraded":"2023-10-16T14:33:07.253217+0000","last_fullsized":"2023-10-16T14:33:07.253217+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:44.437919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985669+0000","last_change":"2023-10-16T14:33:05.269636+0000","last_active":"2023-10-16T14:33:07.985669+0000","last_peered":"2023-10-16T14:33:07.985669+0000","last_clean":"2023-10-16T14:33:07.985669+0000","last_became_active":"2023-10-16T14:33:05.269271+0000","last_became_peered":"2023-10-16T14:33:05.269271+0000","last_unstale":"2023-10-16T14:33:07.985669+0000","last_undegraded":"2023-10-16T14:33:07.985669+0000","last_fullsized":"2023-10-16T14:33:07.985669+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:24.346415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986239+0000","last_change":"2023-10-16T14:33:06.271783+0000","last_active":"2023-10-16T14:33:07.986239+0000","last_peered":"2023-10-16T14:33:07.986239+0000","last_clean":"2023-10-16T14:33:07.986239+0000","last_became_active":"2023-10-16T14:33:06.270496+0000","last_became_peered":"2023-10-16T14:33:06.270496+0000","last_unstale":"2023-10-16T14:33:07.986239+0000","last_undegraded":"2023-10-16T14:33:07.986239+0000","last_fullsized":"2023-10-16T14:33:07.986239+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:05.492228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985721+0000","last_change":"2023-10-16T14:33:05.270100+0000","last_active":"2023-10-16T14:33:07.985721+0000","last_peered":"2023-10-16T14:33:07.985721+0000","last_clean":"2023-10-16T14:33:07.985721+0000","last_became_active":"2023-10-16T14:33:05.267814+0000","last_became_peered":"2023-10-16T14:33:05.267814+0000","last_unstale":"2023-10-16T14:33:07.985721+0000","last_undegraded":"2023-10-16T14:33:07.985721+0000","last_fullsized":"2023-10-16T14:33:07.985721+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:02.881371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.247923+0000","last_change":"2023-10-16T14:33:06.279613+0000","last_active":"2023-10-16T14:33:07.247923+0000","last_peered":"2023-10-16T14:33:07.247923+0000","last_clean":"2023-10-16T14:33:07.247923+0000","last_became_active":"2023-10-16T14:33:06.279257+0000","last_became_peered":"2023-10-16T14:33:06.279257+0000","last_unstale":"2023-10-16T14:33:07.247923+0000","last_undegraded":"2023-10-16T14:33:07.247923+0000","last_fullsized":"2023-10-16T14:33:07.247923+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:13.783995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984429+0000","last_change":"2023-10-16T14:33:05.265988+0000","last_active":"2023-10-16T14:33:07.984429+0000","last_peered":"2023-10-16T14:33:07.984429+0000","last_clean":"2023-10-16T14:33:07.984429+0000","last_became_active":"2023-10-16T14:33:05.265714+0000","last_became_peered":"2023-10-16T14:33:05.265714+0000","last_unstale":"2023-10-16T14:33:07.984429+0000","last_undegraded":"2023-10-16T14:33:07.984429+0000","last_fullsized":"2023-10-16T14:33:07.984429+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:17.837153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987218+0000","last_change":"2023-10-16T14:33:06.286088+0000","last_active":"2023-10-16T14:33:07.987218+0000","last_peered":"2023-10-16T14:33:07.987218+0000","last_clean":"2023-10-16T14:33:07.987218+0000","last_became_active":"2023-10-16T14:33:06.284565+0000","last_became_peered":"2023-10-16T14:33:06.284565+0000","last_unstale":"2023-10-16T14:33:07.987218+0000","last_undegraded":"2023-10-16T14:33:07.987218+0000","last_fullsized":"2023-10-16T14:33:07.987218+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:26.046849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"96'8","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.250938+0000","last_change":"2023-10-16T14:33:05.275803+0000","last_active":"2023-10-16T14:33:07.250938+0000","last_peered":"2023-10-16T14:33:07.250938+0000","last_clean":"2023-10-16T14:33:07.250938+0000","last_became_active":"2023-10-16T14:33:05.275198+0000","last_became_peered":"2023-10-16T14:33:05.275198+0000","last_unstale":"2023-10-16T14:33:07.250938+0000","last_undegraded":"2023-10-16T14:33:07.250938+0000","last_fullsized":"2023-10-16T14:33:07.250938+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:29.410911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254237+0000","last_change":"2023-10-16T14:33:06.274038+0000","last_active":"2023-10-16T14:33:07.254237+0000","last_peered":"2023-10-16T14:33:07.254237+0000","last_clean":"2023-10-16T14:33:07.254237+0000","last_became_active":"2023-10-16T14:33:06.273570+0000","last_became_peered":"2023-10-16T14:33:06.273570+0000","last_unstale":"2023-10-16T14:33:07.254237+0000","last_undegraded":"2023-10-16T14:33:07.254237+0000","last_fullsized":"2023-10-16T14:33:07.254237+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:09.660436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987154+0000","last_change":"2023-10-16T14:33:05.270898+0000","last_active":"2023-10-16T14:33:07.987154+0000","last_peered":"2023-10-16T14:33:07.987154+0000","last_clean":"2023-10-16T14:33:07.987154+0000","last_became_active":"2023-10-16T14:33:05.269881+0000","last_became_peered":"2023-10-16T14:33:05.269881+0000","last_unstale":"2023-10-16T14:33:07.987154+0000","last_undegraded":"2023-10-16T14:33:07.987154+0000","last_fullsized":"2023-10-16T14:33:07.987154+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:10.563882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254835+0000","last_change":"2023-10-16T14:33:06.273336+0000","last_active":"2023-10-16T14:33:07.254835+0000","last_peered":"2023-10-16T14:33:07.254835+0000","last_clean":"2023-10-16T14:33:07.254835+0000","last_became_active":"2023-10-16T14:33:06.273054+0000","last_became_peered":"2023-10-16T14:33:06.273054+0000","last_unstale":"2023-10-16T14:33:07.254835+0000","last_undegraded":"2023-10-16T14:33:07.254835+0000","last_fullsized":"2023-10-16T14:33:07.254835+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:49.739625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254526+0000","last_change":"2023-10-16T14:33:05.271016+0000","last_active":"2023-10-16T14:33:07.254526+0000","last_peered":"2023-10-16T14:33:07.254526+0000","last_clean":"2023-10-16T14:33:07.254526+0000","last_became_active":"2023-10-16T14:33:05.269379+0000","last_became_peered":"2023-10-16T14:33:05.269379+0000","last_unstale":"2023-10-16T14:33:07.254526+0000","last_undegraded":"2023-10-16T14:33:07.254526+0000","last_fullsized":"2023-10-16T14:33:07.254526+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:54.891635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986221+0000","last_change":"2023-10-16T14:33:06.281783+0000","last_active":"2023-10-16T14:33:07.986221+0000","last_peered":"2023-10-16T14:33:07.986221+0000","last_clean":"2023-10-16T14:33:07.986221+0000","last_became_active":"2023-10-16T14:33:06.281511+0000","last_became_peered":"2023-10-16T14:33:06.281511+0000","last_unstale":"2023-10-16T14:33:07.986221+0000","last_undegraded":"2023-10-16T14:33:07.986221+0000","last_fullsized":"2023-10-16T14:33:07.986221+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:28.644767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985342+0000","last_change":"2023-10-16T14:33:05.273682+0000","last_active":"2023-10-16T14:33:07.985342+0000","last_peered":"2023-10-16T14:33:07.985342+0000","last_clean":"2023-10-16T14:33:07.985342+0000","last_became_active":"2023-10-16T14:33:05.272671+0000","last_became_peered":"2023-10-16T14:33:05.272671+0000","last_unstale":"2023-10-16T14:33:07.985342+0000","last_undegraded":"2023-10-16T14:33:07.985342+0000","last_fullsized":"2023-10-16T14:33:07.985342+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:57.398530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987611+0000","last_change":"2023-10-16T14:33:06.276899+0000","last_active":"2023-10-16T14:33:07.987611+0000","last_peered":"2023-10-16T14:33:07.987611+0000","last_clean":"2023-10-16T14:33:07.987611+0000","last_became_active":"2023-10-16T14:33:06.276575+0000","last_became_peered":"2023-10-16T14:33:06.276575+0000","last_unstale":"2023-10-16T14:33:07.987611+0000","last_undegraded":"2023-10-16T14:33:07.987611+0000","last_fullsized":"2023-10-16T14:33:07.987611+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:35.729049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.253537+0000","last_change":"2023-10-16T14:33:05.276515+0000","last_active":"2023-10-16T14:33:07.253537+0000","last_peered":"2023-10-16T14:33:07.253537+0000","last_clean":"2023-10-16T14:33:07.253537+0000","last_became_active":"2023-10-16T14:33:05.276112+0000","last_became_peered":"2023-10-16T14:33:05.276112+0000","last_unstale":"2023-10-16T14:33:07.253537+0000","last_undegraded":"2023-10-16T14:33:07.253537+0000","last_fullsized":"2023-10-16T14:33:07.253537+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:45.581017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988453+0000","last_change":"2023-10-16T14:33:06.280867+0000","last_active":"2023-10-16T14:33:07.988453+0000","last_peered":"2023-10-16T14:33:07.988453+0000","last_clean":"2023-10-16T14:33:07.988453+0000","last_became_active":"2023-10-16T14:33:06.280604+0000","last_became_peered":"2023-10-16T14:33:06.280604+0000","last_unstale":"2023-10-16T14:33:07.988453+0000","last_undegraded":"2023-10-16T14:33:07.988453+0000","last_fullsized":"2023-10-16T14:33:07.988453+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:31:19.851053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985947+0000","last_change":"2023-10-16T14:33:05.266919+0000","last_active":"2023-10-16T14:33:07.985947+0000","last_peered":"2023-10-16T14:33:07.985947+0000","last_clean":"2023-10-16T14:33:07.985947+0000","last_became_active":"2023-10-16T14:33:05.266518+0000","last_became_peered":"2023-10-16T14:33:05.266518+0000","last_unstale":"2023-10-16T14:33:07.985947+0000","last_undegraded":"2023-10-16T14:33:07.985947+0000","last_fullsized":"2023-10-16T14:33:07.985947+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:01.201377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986655+0000","last_change":"2023-10-16T14:33:06.283373+0000","last_active":"2023-10-16T14:33:07.986655+0000","last_peered":"2023-10-16T14:33:07.986655+0000","last_clean":"2023-10-16T14:33:07.986655+0000","last_became_active":"2023-10-16T14:33:06.280559+0000","last_became_peered":"2023-10-16T14:33:06.280559+0000","last_unstale":"2023-10-16T14:33:07.986655+0000","last_undegraded":"2023-10-16T14:33:07.986655+0000","last_fullsized":"2023-10-16T14:33:07.986655+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:50.049165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985020+0000","last_change":"2023-10-16T14:33:05.264128+0000","last_active":"2023-10-16T14:33:07.985020+0000","last_peered":"2023-10-16T14:33:07.985020+0000","last_clean":"2023-10-16T14:33:07.985020+0000","last_became_active":"2023-10-16T14:33:05.263842+0000","last_became_peered":"2023-10-16T14:33:05.263842+0000","last_unstale":"2023-10-16T14:33:07.985020+0000","last_undegraded":"2023-10-16T14:33:07.985020+0000","last_fullsized":"2023-10-16T14:33:07.985020+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:37.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987423+0000","last_change":"2023-10-16T14:33:06.283404+0000","last_active":"2023-10-16T14:33:07.987423+0000","last_peered":"2023-10-16T14:33:07.987423+0000","last_clean":"2023-10-16T14:33:07.987423+0000","last_became_active":"2023-10-16T14:33:06.283119+0000","last_became_peered":"2023-10-16T14:33:06.283119+0000","last_unstale":"2023-10-16T14:33:07.987423+0000","last_undegraded":"2023-10-16T14:33:07.987423+0000","last_fullsized":"2023-10-16T14:33:07.987423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:51.044247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.247857+0000","last_change":"2023-10-16T14:33:05.262319+0000","last_active":"2023-10-16T14:33:07.247857+0000","last_peered":"2023-10-16T14:33:07.247857+0000","last_clean":"2023-10-16T14:33:07.247857+0000","last_became_active":"2023-10-16T14:33:05.262028+0000","last_became_peered":"2023-10-16T14:33:05.262028+0000","last_unstale":"2023-10-16T14:33:07.247857+0000","last_undegraded":"2023-10-16T14:33:07.247857+0000","last_fullsized":"2023-10-16T14:33:07.247857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:51.961764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248835+0000","last_change":"2023-10-16T14:33:06.277414+0000","last_active":"2023-10-16T14:33:07.248835+0000","last_peered":"2023-10-16T14:33:07.248835+0000","last_clean":"2023-10-16T14:33:07.248835+0000","last_became_active":"2023-10-16T14:33:06.277104+0000","last_became_peered":"2023-10-16T14:33:06.277104+0000","last_unstale":"2023-10-16T14:33:07.248835+0000","last_undegraded":"2023-10-16T14:33:07.248835+0000","last_fullsized":"2023-10-16T14:33:07.248835+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:02.431469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987128+0000","last_change":"2023-10-16T14:33:05.269622+0000","last_active":"2023-10-16T14:33:07.987128+0000","last_peered":"2023-10-16T14:33:07.987128+0000","last_clean":"2023-10-16T14:33:07.987128+0000","last_became_active":"2023-10-16T14:33:05.269266+0000","last_became_peered":"2023-10-16T14:33:05.269266+0000","last_unstale":"2023-10-16T14:33:07.987128+0000","last_undegraded":"2023-10-16T14:33:07.987128+0000","last_fullsized":"2023-10-16T14:33:07.987128+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:42.623094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986208+0000","last_change":"2023-10-16T14:33:06.275097+0000","last_active":"2023-10-16T14:33:07.986208+0000","last_peered":"2023-10-16T14:33:07.986208+0000","last_clean":"2023-10-16T14:33:07.986208+0000","last_became_active":"2023-10-16T14:33:06.274800+0000","last_became_peered":"2023-10-16T14:33:06.274800+0000","last_unstale":"2023-10-16T14:33:07.986208+0000","last_undegraded":"2023-10-16T14:33:07.986208+0000","last_fullsized":"2023-10-16T14:33:07.986208+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:33.996935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986452+0000","last_change":"2023-10-16T14:33:05.267780+0000","last_active":"2023-10-16T14:33:07.986452+0000","last_peered":"2023-10-16T14:33:07.986452+0000","last_clean":"2023-10-16T14:33:07.986452+0000","last_became_active":"2023-10-16T14:33:05.267523+0000","last_became_peered":"2023-10-16T14:33:05.267523+0000","last_unstale":"2023-10-16T14:33:07.986452+0000","last_undegraded":"2023-10-16T14:33:07.986452+0000","last_fullsized":"2023-10-16T14:33:07.986452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:02.907250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254488+0000","last_change":"2023-10-16T14:33:06.295780+0000","last_active":"2023-10-16T14:33:07.254488+0000","last_peered":"2023-10-16T14:33:07.254488+0000","last_clean":"2023-10-16T14:33:07.254488+0000","last_became_active":"2023-10-16T14:33:06.291980+0000","last_became_peered":"2023-10-16T14:33:06.291980+0000","last_unstale":"2023-10-16T14:33:07.254488+0000","last_undegraded":"2023-10-16T14:33:07.254488+0000","last_fullsized":"2023-10-16T14:33:07.254488+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:51.044247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.987515+0000","last_change":"2023-10-16T14:33:05.260854+0000","last_active":"2023-10-16T14:33:07.987515+0000","last_peered":"2023-10-16T14:33:07.987515+0000","last_clean":"2023-10-16T14:33:07.987515+0000","last_became_active":"2023-10-16T14:33:05.260559+0000","last_became_peered":"2023-10-16T14:33:05.260559+0000","last_unstale":"2023-10-16T14:33:07.987515+0000","last_undegraded":"2023-10-16T14:33:07.987515+0000","last_fullsized":"2023-10-16T14:33:07.987515+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:53.105474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.254832+0000","last_change":"2023-10-16T14:33:06.276344+0000","last_active":"2023-10-16T14:33:07.254832+0000","last_peered":"2023-10-16T14:33:07.254832+0000","last_clean":"2023-10-16T14:33:07.254832+0000","last_became_active":"2023-10-16T14:33:06.269991+0000","last_became_peered":"2023-10-16T14:33:06.269991+0000","last_unstale":"2023-10-16T14:33:07.254832+0000","last_undegraded":"2023-10-16T14:33:07.254832+0000","last_fullsized":"2023-10-16T14:33:07.254832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:51.862361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.248164+0000","last_change":"2023-10-16T14:33:05.271677+0000","last_active":"2023-10-16T14:33:07.248164+0000","last_peered":"2023-10-16T14:33:07.248164+0000","last_clean":"2023-10-16T14:33:07.248164+0000","last_became_active":"2023-10-16T14:33:05.271283+0000","last_became_peered":"2023-10-16T14:33:05.271283+0000","last_unstale":"2023-10-16T14:33:07.248164+0000","last_undegraded":"2023-10-16T14:33:07.248164+0000","last_fullsized":"2023-10-16T14:33:07.248164+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:54:17.250940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988041+0000","last_change":"2023-10-16T14:33:06.281177+0000","last_active":"2023-10-16T14:33:07.988041+0000","last_peered":"2023-10-16T14:33:07.988041+0000","last_clean":"2023-10-16T14:33:07.988041+0000","last_became_active":"2023-10-16T14:33:06.280946+0000","last_became_peered":"2023-10-16T14:33:06.280946+0000","last_unstale":"2023-10-16T14:33:07.988041+0000","last_undegraded":"2023-10-16T14:33:07.988041+0000","last_fullsized":"2023-10-16T14:33:07.988041+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:52.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.989428+0000","last_change":"2023-10-16T14:33:05.266588+0000","last_active":"2023-10-16T14:33:07.989428+0000","last_peered":"2023-10-16T14:33:07.989428+0000","last_clean":"2023-10-16T14:33:07.989428+0000","last_became_active":"2023-10-16T14:33:05.266303+0000","last_became_peered":"2023-10-16T14:33:05.266303+0000","last_unstale":"2023-10-16T14:33:07.989428+0000","last_undegraded":"2023-10-16T14:33:07.989428+0000","last_fullsized":"2023-10-16T14:33:07.989428+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:09.205400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985606+0000","last_change":"2023-10-16T14:33:06.290344+0000","last_active":"2023-10-16T14:33:07.985606+0000","last_peered":"2023-10-16T14:33:07.985606+0000","last_clean":"2023-10-16T14:33:07.985606+0000","last_became_active":"2023-10-16T14:33:06.289622+0000","last_became_peered":"2023-10-16T14:33:06.289622+0000","last_unstale":"2023-10-16T14:33:07.985606+0000","last_undegraded":"2023-10-16T14:33:07.985606+0000","last_fullsized":"2023-10-16T14:33:07.985606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:13.783995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.984826+0000","last_change":"2023-10-16T14:33:05.272774+0000","last_active":"2023-10-16T14:33:07.984826+0000","last_peered":"2023-10-16T14:33:07.984826+0000","last_clean":"2023-10-16T14:33:07.984826+0000","last_became_active":"2023-10-16T14:33:05.272478+0000","last_became_peered":"2023-10-16T14:33:05.272478+0000","last_unstale":"2023-10-16T14:33:07.984826+0000","last_undegraded":"2023-10-16T14:33:07.984826+0000","last_fullsized":"2023-10-16T14:33:07.984826+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:59.242105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985776+0000","last_change":"2023-10-16T14:33:06.272448+0000","last_active":"2023-10-16T14:33:07.985776+0000","last_peered":"2023-10-16T14:33:07.985776+0000","last_clean":"2023-10-16T14:33:07.985776+0000","last_became_active":"2023-10-16T14:33:06.272171+0000","last_became_peered":"2023-10-16T14:33:06.272171+0000","last_unstale":"2023-10-16T14:33:07.985776+0000","last_undegraded":"2023-10-16T14:33:07.985776+0000","last_fullsized":"2023-10-16T14:33:07.985776+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:04.523835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.250873+0000","last_change":"2023-10-16T14:33:05.274159+0000","last_active":"2023-10-16T14:33:07.250873+0000","last_peered":"2023-10-16T14:33:07.250873+0000","last_clean":"2023-10-16T14:33:07.250873+0000","last_became_active":"2023-10-16T14:33:05.273529+0000","last_became_peered":"2023-10-16T14:33:05.273529+0000","last_unstale":"2023-10-16T14:33:07.250873+0000","last_undegraded":"2023-10-16T14:33:07.250873+0000","last_fullsized":"2023-10-16T14:33:07.250873+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:52.990469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985124+0000","last_change":"2023-10-16T14:33:06.284496+0000","last_active":"2023-10-16T14:33:07.985124+0000","last_peered":"2023-10-16T14:33:07.985124+0000","last_clean":"2023-10-16T14:33:07.985124+0000","last_became_active":"2023-10-16T14:33:06.284213+0000","last_became_peered":"2023-10-16T14:33:06.284213+0000","last_unstale":"2023-10-16T14:33:07.985124+0000","last_undegraded":"2023-10-16T14:33:07.985124+0000","last_fullsized":"2023-10-16T14:33:07.985124+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:22.607472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.986745+0000","last_change":"2023-10-16T14:33:05.263747+0000","last_active":"2023-10-16T14:33:07.986745+0000","last_peered":"2023-10-16T14:33:07.986745+0000","last_clean":"2023-10-16T14:33:07.986745+0000","last_became_active":"2023-10-16T14:33:05.263464+0000","last_became_peered":"2023-10-16T14:33:05.263464+0000","last_unstale":"2023-10-16T14:33:07.986745+0000","last_undegraded":"2023-10-16T14:33:07.986745+0000","last_fullsized":"2023-10-16T14:33:07.986745+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:54:17.250940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.988380+0000","last_change":"2023-10-16T14:33:06.282571+0000","last_active":"2023-10-16T14:33:07.988380+0000","last_peered":"2023-10-16T14:33:07.988380+0000","last_clean":"2023-10-16T14:33:07.988380+0000","last_became_active":"2023-10-16T14:33:06.282266+0000","last_became_peered":"2023-10-16T14:33:06.282266+0000","last_unstale":"2023-10-16T14:33:07.988380+0000","last_undegraded":"2023-10-16T14:33:07.988380+0000","last_fullsized":"2023-10-16T14:33:07.988380+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:05.226281+0000","last_clean_scrub_stamp":"2023-10-16T14:33:05.226281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:49.739625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T14:33:07.985631+0000","last_change":"2023-10-16T14:33:05.269864+0000","last_active":"2023-10-16T14:33:07.985631+0000","last_peered":"2023-10-16T14:33:07.985631+0000","last_clean":"2023-10-16T14:33:07.985631+0000","last_became_active":"2023-10-16T14:33:05.269506+0000","last_became_peered":"2023-10-16T14:33:05.269506+0000","last_unstale":"2023-10-16T14:33:07.985631+0000","last_undegraded":"2023-10-16T14:33:07.985631+0000","last_fullsized":"2023-10-16T14:33:07.985631+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:04.211851+0000","last_clean_scrub_stamp":"2023-10-16T14:33:04.211851+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:51.961764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"86'175","reported_seq":455,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-16T14:33:06.253357+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:33:06.253357+0000","last_peered":"2023-10-16T14:33:06.253357+0000","last_clean":"2023-10-16T14:33:06.253357+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:33:06.253357+0000","last_undegraded":"2023-10-16T14:33:06.253357+0000","last_fullsized":"2023-10-16T14:33:06.253357+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575050,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41244,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1785856,"data_stored":1159936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0640000000000001,"15min":1.147},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.42799999999999999},"max":{"1min":1.784,"5min":1.833,"15min":1.9159999999999999},"last":1.601},{"interface":"front","average":{"1min":1.081,"5min":1.133,"15min":1.157},"min":{"1min":0.77800000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.522,"5min":1.6220000000000001,"15min":1.673},"last":1.4219999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1599999999999999,"15min":1.1799999999999999},"min":{"1min":0.71599999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3859999999999999,"5min":1.7030000000000001,"15min":1.8320000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.22,"15min":1.2390000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.091}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.298,"15min":1.296},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63500000000000001},"max":{"1min":1.512,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.353},{"interface":"front","average":{"1min":1.137,"5min":1.23,"15min":1.234},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.996}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.298,"15min":1.2989999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3500000000000001,"15min":1.379},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2509999999999999},"last":1.7090000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3580000000000001,"15min":1.3600000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.835,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.456},{"interface":"front","average":{"1min":1.236,"5min":1.25,"15min":1.258},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.9530000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.86699999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3939999999999999,"15min":1.407},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.232},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4750000000000001,"15min":1.4870000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.4910000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.2850000000000001,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.504,"15min":1.512},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":2.359,"5min":2.359,"15min":2.427},"last":1.766}]}]},{"osd":6,"up_from":13,"seq":55834575049,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36504,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335336,"statfs":{"total":96636764160,"available":96599384064,"internally_reserved":0,"allocated":1191936,"data_stored":569044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.169},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.6990000000000001,"5min":1.7569999999999999,"15min":2.2160000000000002},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.82599999999999996,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.577,"5min":2.2709999999999999,"15min":3.863},"last":1.026}]},{"osd":1,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.2509999999999999},"min":{"1min":0.92400000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.581,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.3,"15min":1.29},"min":{"1min":0.81799999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7849999999999999,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":1.1160000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.147,"15min":1.1990000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.59899999999999998},"max":{"1min":1.3400000000000001,"5min":1.708,"15min":3.5800000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.232,"5min":1.3180000000000001,"15min":1.3500000000000001},"min":{"1min":0.80200000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.52,"5min":1.8799999999999999,"15min":2.1429999999999998},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3300000000000001,"15min":1.353},"min":{"1min":0.85599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.8480000000000001,"5min":1.96,"15min":4.9790000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.286,"15min":1.3400000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":1.6080000000000001,"5min":1.913,"15min":5.4160000000000004},"last":0.89400000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.296,"15min":1.3},"min":{"1min":0.748,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8009999999999999,"5min":2.1070000000000002,"15min":5},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3640000000000001,"15min":1.377},"min":{"1min":0.94999999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.897,"5min":2.1789999999999998,"15min":5.4640000000000004},"last":1.232}]},{"osd":5,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2949999999999999,"15min":1.3160000000000001},"min":{"1min":0.89600000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.76,"5min":1.859,"15min":3.9279999999999999},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.482,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":3.6749999999999998},"last":1.3100000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.5600000000000001},"min":{"1min":1.105,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":4.6859999999999999},"last":1.389},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.508,"15min":1.542},"min":{"1min":1.0840000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9139999999999999,"5min":2.0459999999999998,"15min":4.883},"last":1.6870000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575049,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36488,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335352,"statfs":{"total":96636764160,"available":96599400448,"internally_reserved":0,"allocated":1175552,"data_stored":562742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.099,"15min":1.1160000000000001},"min":{"1min":0.73599999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5760000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1990000000000001,"15min":1.1819999999999999},"min":{"1min":0.63300000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.7729999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.2909999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.444,"5min":1.726,"15min":1.726},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.2430000000000001},"min":{"1min":0.81499999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5249999999999999,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.1899999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.244,"15min":1.2390000000000001},"min":{"1min":0.71599999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5720000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.141},{"interface":"front","average":{"1min":1.304,"5min":1.3200000000000001,"15min":1.353},"min":{"1min":0.85799999999999998,"5min":0.625,"15min":0.625},"max":{"1min":2.0049999999999999,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":0.94599999999999995}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.76900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":1.083},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.351,"15min":1.3779999999999999},"min":{"1min":0.85699999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.6819999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.6890000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3049999999999999,"15min":1.3169999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.54700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.349},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5049999999999999,"15min":1.5029999999999999},"min":{"1min":1.073,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.853,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.601}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3520000000000001,"15min":1.3819999999999999},"min":{"1min":0.97299999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.504},{"interface":"front","average":{"1min":1.385,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":0.77700000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.556}]},{"osd":7,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.5489999999999999},"min":{"1min":0.93400000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.956,"5min":2.641,"15min":2.641},"last":1.397},{"interface":"front","average":{"1min":1.536,"5min":1.579,"15min":1.603},"min":{"1min":1.246,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.1909999999999998,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.724}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46232,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325608,"statfs":{"total":96636764160,"available":96589422592,"internally_reserved":0,"allocated":1187840,"data_stored":569022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1559999999999999,"15min":1.1479999999999999},"min":{"1min":0.58999999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7689999999999999,"5min":1.881,"15min":2.0339999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.999,"5min":1.1020000000000001,"15min":1.1639999999999999},"min":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4830000000000001,"5min":1.7589999999999999,"15min":1.9199999999999999},"last":0.99299999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1479999999999999,"15min":1.177},"min":{"1min":0.58399999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.78},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.1910000000000001,"15min":1.2330000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.716,"5min":1.782,"15min":2.3690000000000002},"last":1.0620000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3700000000000001,"15min":1.399},"min":{"1min":0.68100000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.3400000000000001,"15min":1.365},"min":{"1min":0.69399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7769999999999999,"5min":1.8799999999999999,"15min":1.8839999999999999},"last":1.421}]},{"osd":3,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.5920000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.238,"5min":1.268,"15min":1.304},"min":{"1min":0.53900000000000003,"5min":0.437,"15min":0.437},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.3879999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.306,"15min":1.288},"min":{"1min":0.78800000000000003,"5min":0.61499999999999999,"15min":0.48699999999999999},"max":{"1min":1.639,"5min":1.696,"15min":1.8220000000000001},"last":1.179},{"interface":"front","average":{"1min":1.395,"5min":1.3460000000000001,"15min":1.349},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.806,"5min":1.806,"15min":1.964},"last":1.115}]},{"osd":6,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.323,"15min":1.2909999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4690000000000001,"15min":1.4610000000000001},"min":{"1min":0.74199999999999999,"5min":0.71999999999999997,"15min":0.44700000000000001},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.2890000000000001},"last":1.351}]},{"osd":7,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.4730000000000001,"15min":1.4690000000000001},"min":{"1min":0.76100000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9259999999999999},"last":1.244},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.466,"15min":1.458},"min":{"1min":0.66700000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8979999999999999,"5min":2.0579999999999998,"15min":2.29},"last":1.512}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40664,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331176,"statfs":{"total":96636764160,"available":96595124224,"internally_reserved":0,"allocated":1191936,"data_stored":569064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.50800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0609999999999999,"5min":1.4419999999999999,"15min":1.6639999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":1.0169999999999999},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.3320000000000001,"5min":1.913,"15min":1.982},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90000000000000002,"15min":0.94099999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.32500000000000001},"max":{"1min":1.337,"5min":1.7390000000000001,"15min":2.149},"last":1.095},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.089,"15min":1.139},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.415,"5min":1.6499999999999999,"15min":2.1600000000000001},"last":0.88}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2050000000000001,"15min":1.3180000000000001},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.44800000000000001},"max":{"1min":1.6830000000000001,"5min":1.895,"15min":2.1259999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.103,"5min":1.331,"15min":1.4259999999999999},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.5},"max":{"1min":1.8029999999999999,"5min":1.8660000000000001,"15min":2.6299999999999999},"last":1.0629999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1910000000000001,"15min":1.2569999999999999},"min":{"1min":0.77100000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.768,"5min":1.768,"15min":1.9239999999999999},"last":1.349},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.298,"15min":1.3300000000000001},"min":{"1min":0.93999999999999995,"5min":0.629,"15min":0.629},"max":{"1min":1.7689999999999999,"5min":1.909,"15min":1.9259999999999999},"last":1.7270000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.4550000000000001,"15min":1.516},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7170000000000001,"5min":2.0070000000000001,"15min":2.23},"last":1.786},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3899999999999999,"15min":1.468},"min":{"1min":0.90700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6220000000000001,"5min":1.95,"15min":2.3570000000000002},"last":1.2889999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.361,"15min":1.421},"min":{"1min":0.95999999999999996,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.837,"5min":2.181,"15min":2.181},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4419999999999999,"15min":1.4910000000000001},"min":{"1min":0.82599999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.6799999999999999,"5min":1.9510000000000001,"15min":2.347},"last":1.8280000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.4870000000000001,"15min":1.581},"min":{"1min":1.0069999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7310000000000001,"5min":2.1589999999999998,"15min":2.395},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.472,"15min":1.5509999999999999},"min":{"1min":0.749,"5min":0.629,"15min":0.629},"max":{"1min":1.8160000000000001,"5min":2.1000000000000001,"15min":2.1739999999999999},"last":1.5629999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575050,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44816,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327024,"statfs":{"total":96636764160,"available":96590872576,"internally_reserved":0,"allocated":1183744,"data_stored":564264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.4319999999999999,"5min":1.556,"15min":2.472},"last":1.151},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96299999999999997,"15min":1.024},"min":{"1min":0.48299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3560000000000001,"5min":1.585,"15min":2.3260000000000001},"last":1.044}]},{"osd":1,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.0029999999999999},"min":{"1min":0.50800000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.454,"5min":3.234,"15min":3.234},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.1459999999999999,"15min":1.1619999999999999},"min":{"1min":0.52200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6299999999999999,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.331}]},{"osd":3,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.1830000000000001,"15min":1.234},"min":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.2629999999999999,"15min":1.2969999999999999},"min":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6879999999999999,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.2250000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.415,"15min":1.425},"min":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.766,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.3400000000000001,"15min":1.3620000000000001},"min":{"1min":0.80400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6759999999999999,"5min":3.016,"15min":3.016},"last":1.7190000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.393,"15min":1.415},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8169999999999999,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.762},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.552},"min":{"1min":0.96599999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.3039999999999998,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.4319999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.448,"15min":1.486},"min":{"1min":0.85899999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.0129999999999999,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.887},{"interface":"front","average":{"1min":1.46,"5min":1.5,"15min":1.5289999999999999},"min":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.3610000000000002,"5min":2.5920000000000001,"15min":3.3929999999999998},"last":1.9199999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.52,"15min":1.5640000000000001},"min":{"1min":0.80700000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7789999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.617,"15min":1.6439999999999999},"min":{"1min":0.88,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.206,"5min":2.7519999999999998,"15min":2.944},"last":1.996}]}]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54568,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317272,"statfs":{"total":96636764160,"available":96580886528,"internally_reserved":0,"allocated":1204224,"data_stored":571180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.874,"15min":0.90100000000000002},"min":{"1min":0.54700000000000004,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3819999999999999,"5min":1.569,"15min":1.821},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.075,"15min":1.0980000000000001},"min":{"1min":0.40799999999999997,"5min":0.378,"15min":0.378},"max":{"1min":1.601,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.452}]},{"osd":2,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95399999999999996,"15min":0.98599999999999999},"min":{"1min":0.40300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.524,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0880000000000001,"15min":1.1599999999999999},"min":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6919999999999999,"5min":1.756,"15min":1.796},"last":1.22}]},{"osd":3,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.2749999999999999,"15min":1.3200000000000001},"min":{"1min":0.58899999999999997,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.167,"15min":1.1990000000000001},"min":{"1min":0.55700000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7509999999999999,"5min":1.8100000000000001,"15min":2.2050000000000001},"last":0.999}]},{"osd":4,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.238,"15min":1.2829999999999999},"min":{"1min":0.85399999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8700000000000001,"5min":2.069,"15min":2.069},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3700000000000001,"15min":1.419},"min":{"1min":0.82499999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.899,"5min":2.899,"15min":2.899},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.403,"15min":1.4370000000000001},"min":{"1min":0.97999999999999998,"5min":0.623,"15min":0.623},"max":{"1min":1.6499999999999999,"5min":2.032,"15min":2.3690000000000002},"last":1.353},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.371,"15min":1.421},"min":{"1min":0.84399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8779999999999999,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":1.405}]},{"osd":6,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.484,"15min":1.5389999999999999},"min":{"1min":0.96399999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.694,"5min":2.694,"15min":2.694},"last":1.498},{"interface":"front","average":{"1min":1.353,"5min":1.4910000000000001,"15min":1.538},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.73099999999999998},"max":{"1min":1.925,"5min":2.2610000000000001,"15min":2.6160000000000001},"last":1.2529999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:32:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5800000000000001,"15min":1.6160000000000001},"min":{"1min":1.052,"5min":0.90600000000000003,"15min":0.79300000000000004},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5059999999999998},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.488,"15min":1.552},"min":{"1min":0.998,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.262},"last":1.5329999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1777664,"data_stored":1159320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.875,"15min":0.94299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.7929999999999999},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.097,"15min":1.163},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.36399999999999999},"max":{"1min":1.508,"5min":1.833,"15min":1.867},"last":1.71}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.204,"15min":1.2470000000000001},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1.6359999999999999,"5min":1.887,"15min":2.2970000000000002},"last":1.994},{"interface":"front","average":{"1min":1.093,"5min":1.159,"15min":1.2050000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6000000000000001,"5min":2.012,"15min":2.2330000000000001},"last":1.847}]},{"osd":3,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.96099999999999997,"15min":1.0029999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40100000000000002},"max":{"1min":1.3149999999999999,"5min":1.889,"15min":1.9410000000000001},"last":1.044},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.181,"15min":1.238},"min":{"1min":0.45000000000000001,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.823,"5min":1.823,"15min":2.5339999999999998},"last":2.1190000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.266,"15min":1.302},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.61599999999999999},"max":{"1min":1.786,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.796},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4419999999999999,"15min":1.462},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.7809999999999999,"5min":2.032,"15min":2.6259999999999999},"last":2.4550000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.383,"15min":1.419},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.02,"5min":2.097,"15min":2.5880000000000001},"last":2.1930000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.302,"15min":1.3600000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.67900000000000005},"max":{"1min":1.778,"5min":1.778,"15min":2.4889999999999999},"last":1.492}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5389999999999999,"15min":1.573},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8819999999999999,"5min":2.097,"15min":2.7599999999999998},"last":2.8279999999999998},{"interface":"front","average":{"1min":1.46,"5min":1.4770000000000001,"15min":1.5},"min":{"1min":1.02,"5min":0.98899999999999999,"15min":0.89000000000000001},"max":{"1min":1.9299999999999999,"5min":2.1459999999999999,"15min":2.5089999999999999},"last":2.5350000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5409999999999999,"15min":1.585},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1040000000000001,"5min":2.2650000000000001,"15min":2.6509999999999998},"last":2.335},{"interface":"front","average":{"1min":1.423,"5min":1.472,"15min":1.544},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.976},"last":1.9410000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:33:11.297 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:33:11.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:33:11.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.566+0000 7ff3788a8640 1 Processor -- start 2023-10-16T14:33:11.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.566+0000 7ff3788a8640 1 -- start start 2023-10-16T14:33:11.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.567+0000 7ff3788a8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff37415cb50 0x7ff374161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff3788a8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3740b4c80 0x7ff3740d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff3788a8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3741616c0 con 0x7ff3740b1690 2023-10-16T14:33:11.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff3788a8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740cdbf0 con 0x7ff37415cb50 2023-10-16T14:33:11.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff3788a8640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740cde90 con 0x7ff3740b4c80 2023-10-16T14:33:11.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff372d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff37415cb50 0x7ff374161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff36a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3740b1690 0x7ff3740b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36796/0 (socket says 172.21.15.29:36796) 2023-10-16T14:33:11.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff372d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff37415cb50 0x7ff374161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54084/0 (socket says 172.21.15.29:54084) 2023-10-16T14:33:11.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff36a59c640 1 -- 172.21.15.29:0/1100175807 learned_addr learned my addr 172.21.15.29:0/1100175807 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:11.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff372d9d640 1 -- 172.21.15.29:0/1100175807 >> v2:172.21.15.170:3300/0 conn(0x7ff3740b4c80 msgr2=0x7ff3740d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1100175807 >> v2:172.21.15.170:3300/0 conn(0x7ff3740b4c80 0x7ff3740d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff372d9d640 1 -- 172.21.15.29:0/1100175807 >> v1:172.21.15.29:6789/0 conn(0x7ff3740b1690 legacy=0x7ff3740b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff372d9d640 1 -- 172.21.15.29:0/1100175807 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740ce170 con 0x7ff37415cb50 2023-10-16T14:33:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.568+0000 7ff37259c640 1 -- 172.21.15.29:0/1100175807 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426116904 0 0) 0x7ff3741616c0 con 0x7ff3740b1690 2023-10-16T14:33:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.570+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1100175807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff374161180 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff3740b1ae0 tx=0x7ff36402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8b41c54cec8754 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.570+0000 7ff37259c640 1 -- 172.21.15.29:0/1100175807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36402cb60 con 0x7ff37415cb50 2023-10-16T14:33:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.570+0000 7ff37259c640 1 -- 172.21.15.29:0/1100175807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36402ccc0 con 0x7ff37415cb50 2023-10-16T14:33:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.570+0000 7ff37259c640 1 -- 172.21.15.29:0/1100175807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364034660 con 0x7ff37415cb50 2023-10-16T14:33:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.570+0000 7ff3788a8640 1 -- 172.21.15.29:0/1100175807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 msgr2=0x7ff374161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.570+0000 7ff3788a8640 1 --2- 172.21.15.29:0/1100175807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff374161180 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff3740b1ae0 tx=0x7ff36402bdb0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.571+0000 7ff3788a8640 1 -- 172.21.15.29:0/1100175807 shutdown_connections 2023-10-16T14:33:11.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.571+0000 7ff3788a8640 1 --2- 172.21.15.29:0/1100175807 >> v2:172.21.15.170:3300/0 conn(0x7ff3740b4c80 0x7ff3740d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.571+0000 7ff3788a8640 1 --2- 172.21.15.29:0/1100175807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff374161180 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.571+0000 7ff3788a8640 1 -- 172.21.15.29:0/1100175807 >> 172.21.15.29:0/1100175807 conn(0x7ff3740cf570 msgr2=0x7ff3740cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:11.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.571+0000 7ff3788a8640 1 -- 172.21.15.29:0/1100175807 shutdown_connections 2023-10-16T14:33:11.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.571+0000 7ff3788a8640 1 -- 172.21.15.29:0/1100175807 wait complete. 2023-10-16T14:33:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.572+0000 7ff3788a8640 1 Processor -- start 2023-10-16T14:33:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.572+0000 7ff3788a8640 1 -- start start 2023-10-16T14:33:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.573+0000 7ff3788a8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3740b1690 0x7ff3741f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.573+0000 7ff3788a8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3740b4c80 0x7ff3741f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.573+0000 7ff3788a8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff3740c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.573+0000 7ff3788a8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37415cf30 con 0x7ff37415cb50 2023-10-16T14:33:11.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.573+0000 7ff3788a8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37415d1a0 con 0x7ff3740b1690 2023-10-16T14:33:11.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.573+0000 7ff3788a8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3741f28d0 con 0x7ff3740b4c80 2023-10-16T14:33:11.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff3740c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff3740c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54098/0 (socket says 172.21.15.29:54098) 2023-10-16T14:33:11.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 -- 172.21.15.29:0/3882128917 learned_addr learned my addr 172.21.15.29:0/3882128917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:11.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff372d9d640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3740b1690 0x7ff3741f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff36a59c640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3740b4c80 0x7ff3741f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 -- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3740b4c80 msgr2=0x7ff3741f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3740b4c80 0x7ff3741f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 -- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3740b1690 msgr2=0x7ff3741f64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3740b1690 0x7ff3741f64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.574+0000 7ff37359e640 1 -- 172.21.15.29:0/3882128917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff364009a70 con 0x7ff37415cb50 2023-10-16T14:33:11.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.575+0000 7ff37359e640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff3740c9bc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00b410 tx=0x7ff36c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:11.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.575+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c005a10 con 0x7ff37415cb50 2023-10-16T14:33:11.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.575+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c005b70 con 0x7ff37415cb50 2023-10-16T14:33:11.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.575+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c011630 con 0x7ff37415cb50 2023-10-16T14:33:11.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.575+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3741f2b80 con 0x7ff37415cb50 2023-10-16T14:33:11.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.575+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3741f3140 con 0x7ff37415cb50 2023-10-16T14:33:11.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.576+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37415d330 con 0x7ff37415cb50 2023-10-16T14:33:11.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.577+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff36c011790 con 0x7ff37415cb50 2023-10-16T14:33:11.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.578+0000 7ff36bfff640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff35406afc0 0x7ff35406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:11.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.578+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff36c0894b0 con 0x7ff37415cb50 2023-10-16T14:33:11.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.579+0000 7ff372d9d640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff35406afc0 0x7ff35406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.580+0000 7ff372d9d640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff35406afc0 0x7ff35406d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff364036000 tx=0x7ff364030000 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.582+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff36c054450 con 0x7ff37415cb50 2023-10-16T14:33:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.835+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff3741f3360 con 0x7ff37415cb50 2023-10-16T14:33:11.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.840+0000 7ff36bfff640 1 -- 172.21.15.29:0/3882128917 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff36c0150b0 con 0x7ff37415cb50 2023-10-16T14:33:11.844 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:33:11.845 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:33:11.845 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:33:11.845 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:33:11.845 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:33:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:33:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff35406afc0 msgr2=0x7ff35406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff35406afc0 0x7ff35406d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff364036000 tx=0x7ff364030000 comp rx=0 tx=0).stop 2023-10-16T14:33:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 msgr2=0x7ff3740c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff3740c9bc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00b410 tx=0x7ff36c00b8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 shutdown_connections 2023-10-16T14:33:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7ff35406afc0 0x7ff35406d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff37415cb50 0x7ff3740c9bc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3740b4c80 0x7ff3741f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 --2- 172.21.15.29:0/3882128917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3740b1690 0x7ff3741f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 >> 172.21.15.29:0/3882128917 conn(0x7ff3740cf570 msgr2=0x7ff3740b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 shutdown_connections 2023-10-16T14:33:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:11.844+0000 7ff3788a8640 1 -- 172.21.15.29:0/3882128917 wait complete. 2023-10-16T14:33:11.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.071+0000 7f5697185640 1 Processor -- start 2023-10-16T14:33:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.072+0000 7f5697185640 1 -- start start 2023-10-16T14:33:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.073+0000 7f5697185640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f569006c040 0x7f569006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.073+0000 7f5697185640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569015f4a0 con 0x7f56900ad970 2023-10-16T14:33:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.073+0000 7f5697185640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569006c960 con 0x7f56900cda80 2023-10-16T14:33:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.073+0000 7f5697185640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569015b9d0 con 0x7f569006c040 2023-10-16T14:33:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.074+0000 7f5695982640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56900ad970 0x7f569015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36798/0 (socket says 172.21.15.29:36798) 2023-10-16T14:33:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.074+0000 7f5695982640 1 -- 172.21.15.29:0/2835387392 learned_addr learned my addr 172.21.15.29:0/2835387392 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.074+0000 7f5696183640 1 --2- 172.21.15.29:0/2835387392 >> v2:172.21.15.170:3300/0 conn(0x7f569006c040 0x7f569006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.074+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215196276 0 0) 0x7f569015f4a0 con 0x7f56900ad970 2023-10-16T14:33:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.074+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f567c003520 con 0x7f56900ad970 2023-10-16T14:33:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1803921390 0 0) 0x7f567c003520 con 0x7f56900ad970 2023-10-16T14:33:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 >> v2:172.21.15.170:3300/0 conn(0x7f569006c040 msgr2=0x7f569006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 --2- 172.21.15.29:0/2835387392 >> v2:172.21.15.170:3300/0 conn(0x7f569006c040 0x7f569006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 >> v1:172.21.15.170:6789/0 conn(0x7f56900cda80 legacy=0x7f569015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569015bcb0 con 0x7f56900ad970 2023-10-16T14:33:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5680001670 con 0x7f56900ad970 2023-10-16T14:33:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 >> v1:172.21.15.29:6789/0 conn(0x7f56900ad970 legacy=0x7f569015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0039c0 0x7f567c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f567c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c005180 0x7f567c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569015b9d0 con 0x7f567c0047c0 2023-10-16T14:33:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c00c050 con 0x7f567c005180 2023-10-16T14:33:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c00c210 con 0x7f567c0039c0 2023-10-16T14:33:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.075+0000 7f5695982640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f567c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5696183640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0039c0 0x7f567c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5696984640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c005180 0x7f567c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5695982640 1 -- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0039c0 msgr2=0x7f567c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5695982640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0039c0 0x7f567c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5695982640 1 -- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c005180 msgr2=0x7f567c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5695982640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c005180 0x7f567c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5695982640 1 -- 172.21.15.29:0/2835387392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c003520 con 0x7f567c0047c0 2023-10-16T14:33:12.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5696183640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f567c0039c0 0x7f567c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:12.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5696984640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f567c005180 0x7f567c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:12.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.076+0000 7f5696984640 1 -- 172.21.15.29:0/2835387392 reap_dead start 2023-10-16T14:33:12.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.077+0000 7f5695982640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f567c004c40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f568000c160 tx=0x7f567c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e594480b3dd9a23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:12.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.077+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680005ad0 con 0x7f567c0047c0 2023-10-16T14:33:12.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.077+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5680005c30 con 0x7f567c0047c0 2023-10-16T14:33:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.077+0000 7f5695181640 1 -- 172.21.15.29:0/2835387392 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568002c7c0 con 0x7f567c0047c0 2023-10-16T14:33:12.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 -- 172.21.15.29:0/2835387392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 msgr2=0x7f567c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f567c004c40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f568000c160 tx=0x7f567c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 -- 172.21.15.29:0/2835387392 shutdown_connections 2023-10-16T14:33:12.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 --2- 172.21.15.29:0/2835387392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f567c004c40 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 -- 172.21.15.29:0/2835387392 >> 172.21.15.29:0/2835387392 conn(0x7f56900ce6a0 msgr2=0x7f56900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 -- 172.21.15.29:0/2835387392 shutdown_connections 2023-10-16T14:33:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.078+0000 7f5697185640 1 -- 172.21.15.29:0/2835387392 wait complete. 2023-10-16T14:33:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.080+0000 7f5697185640 1 Processor -- start 2023-10-16T14:33:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.080+0000 7f5697185640 1 -- start start 2023-10-16T14:33:12.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5697185640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f56900b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5697185640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56900caa90 0x7f56900c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5697185640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56900c8220 0x7f56900ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5697185640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56900ca670 con 0x7f567c0047c0 2023-10-16T14:33:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5697185640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569015c270 con 0x7f56900c8220 2023-10-16T14:33:12.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5697185640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569015c550 con 0x7f56900caa90 2023-10-16T14:33:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5696183640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f56900b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5696183640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f56900b0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54120/0 (socket says 172.21.15.29:54120) 2023-10-16T14:33:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5696183640 1 -- 172.21.15.29:0/743594232 learned_addr learned my addr 172.21.15.29:0/743594232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5695982640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56900caa90 0x7f56900c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5696984640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56900c8220 0x7f56900ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.081+0000 7f5696183640 1 -- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56900caa90 msgr2=0x7f56900c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5696183640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56900caa90 0x7f56900c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5696183640 1 -- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56900c8220 msgr2=0x7f56900ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5696183640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56900c8220 0x7f56900ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5696183640 1 -- 172.21.15.29:0/743594232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569015c830 con 0x7f567c0047c0 2023-10-16T14:33:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5696984640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56900c8220 0x7f56900ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5696183640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f56900b0770 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f568c010840 tx=0x7f568c010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c00eb60 con 0x7f567c0047c0 2023-10-16T14:33:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c00ecc0 con 0x7f567c0047c0 2023-10-16T14:33:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c01c400 con 0x7f567c0047c0 2023-10-16T14:33:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.082+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56900b0cb0 con 0x7f567c0047c0 2023-10-16T14:33:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.083+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56900acdd0 con 0x7f567c0047c0 2023-10-16T14:33:12.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.084+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569006b330 con 0x7f567c0047c0 2023-10-16T14:33:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.085+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f568c01c560 con 0x7f567c0047c0 2023-10-16T14:33:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.086+0000 7f5686ffd640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f567406af90 0x7f567406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.086+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f568c089c00 con 0x7f567c0047c0 2023-10-16T14:33:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.086+0000 7f5695982640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f567406af90 0x7f567406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.087+0000 7f5695982640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f567406af90 0x7f567406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5680034000 tx=0x7f5680009030 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.090+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f568c090080 con 0x7f567c0047c0 2023-10-16T14:33:12.229 INFO:tasks.ceph.mds.b.smithi170.stderr:2023-10-16T14:33:12.225+0000 7f6167adc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:33:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.317+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56900ca3d0 con 0x7f567c0047c0 2023-10-16T14:33:12.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.318+0000 7f5686ffd640 1 -- 172.21.15.29:0/743594232 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f56900ca3d0 con 0x7f567c0047c0 2023-10-16T14:33:12.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:12.323 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:05.723125+0000","modified":"2023-10-16T14:33:08.790919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5926":{"gid":5926,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3812375996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3812375996},{"type":"v1","addr":"172.21.15.170:6837","nonce":3812375996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T14:33:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-16T14:33:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f567406af90 msgr2=0x7f567406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f567406af90 0x7f567406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5680034000 tx=0x7f5680009030 comp rx=0 tx=0).stop 2023-10-16T14:33:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 msgr2=0x7f56900b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f56900b0770 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f568c010840 tx=0x7f568c010d00 comp rx=0 tx=0).stop 2023-10-16T14:33:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 shutdown_connections 2023-10-16T14:33:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f56900c8220 0x7f56900ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56900caa90 0x7f56900c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f567406af90 0x7f567406d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 --2- 172.21.15.29:0/743594232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c0047c0 0x7f56900b0770 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 >> 172.21.15.29:0/743594232 conn(0x7f56900ce6a0 msgr2=0x7f569015d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 shutdown_connections 2023-10-16T14:33:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.321+0000 7f5697185640 1 -- 172.21.15.29:0/743594232 wait complete. 2023-10-16T14:33:12.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:33:05.723125+0000', 'modified': '2023-10-16T14:33:08.790919+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5926}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5926': {'gid': 5926, 'name': 'b', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6837/3812375996', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 3812375996}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 3812375996}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:33:12.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:33:12.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T14:33:12.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.980+0000 7f77da92e640 1 Processor -- start 2023-10-16T14:33:12.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.980+0000 7f77da92e640 1 -- start start 2023-10-16T14:33:12.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.981+0000 7f77da92e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77d415dde0 0x7f77d40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.981+0000 7f77da92e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d4070f20 0x7f77d40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:12.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77da92e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d4160a90 con 0x7f77d415c9f0 2023-10-16T14:33:12.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77da92e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d40cdec0 con 0x7f77d415dde0 2023-10-16T14:33:12.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77da92e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40cd900 con 0x7f77d4070f20 2023-10-16T14:33:12.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77d912b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77d415dde0 0x7f77d40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77d912b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f77d415dde0 0x7f77d40709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54130/0 (socket says 172.21.15.29:54130) 2023-10-16T14:33:12.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77d912b640 1 -- 172.21.15.29:0/3796363713 learned_addr learned my addr 172.21.15.29:0/3796363713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:12.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.982+0000 7f77da12d640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d4070f20 0x7f77d40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:12.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.983+0000 7f77d912b640 1 -- 172.21.15.29:0/3796363713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d4070f20 msgr2=0x7f77d40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:12.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.983+0000 7f77d912b640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d4070f20 0x7f77d40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.983+0000 7f77d912b640 1 -- 172.21.15.29:0/3796363713 >> v1:172.21.15.170:6789/0 conn(0x7f77d415c9f0 legacy=0x7f77d40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:12.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.983+0000 7f77d912b640 1 -- 172.21.15.29:0/3796363713 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40cdbe0 con 0x7f77d415dde0 2023-10-16T14:33:12.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.983+0000 7f77da12d640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d4070f20 0x7f77d40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:12.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.984+0000 7f77d912b640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d415dde0 0x7f77d40709e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009870 tx=0x7f77c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd019a83f48b0538 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:12.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.984+0000 7f77d892a640 1 -- 172.21.15.29:0/3796363713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4031640 con 0x7f77d415dde0 2023-10-16T14:33:12.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.984+0000 7f77d892a640 1 -- 172.21.15.29:0/3796363713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c40317a0 con 0x7f77d415dde0 2023-10-16T14:33:12.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.984+0000 7f77d892a640 1 -- 172.21.15.29:0/3796363713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4031b30 con 0x7f77d415dde0 2023-10-16T14:33:12.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 -- 172.21.15.29:0/3796363713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d415dde0 msgr2=0x7f77d40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:12.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d415dde0 0x7f77d40709e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009870 tx=0x7f77c402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 -- 172.21.15.29:0/3796363713 shutdown_connections 2023-10-16T14:33:12.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d4070f20 0x7f77d40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 --2- 172.21.15.29:0/3796363713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d415dde0 0x7f77d40709e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:12.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 -- 172.21.15.29:0/3796363713 >> 172.21.15.29:0/3796363713 conn(0x7f77d40ce6a0 msgr2=0x7f77d40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:12.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 -- 172.21.15.29:0/3796363713 shutdown_connections 2023-10-16T14:33:12.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.985+0000 7f77da92e640 1 -- 172.21.15.29:0/3796363713 wait complete. 2023-10-16T14:33:12.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.987+0000 7f77da92e640 1 Processor -- start 2023-10-16T14:33:12.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.987+0000 7f77da92e640 1 -- start start 2023-10-16T14:33:13.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.988+0000 7f77da92e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 0x7f77d4150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da92e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d415c9f0 0x7f77d406b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da92e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77d415dde0 0x7f77d4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da92e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d406c220 con 0x7f77d4070f20 2023-10-16T14:33:13.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da92e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40c40e0 con 0x7f77d415dde0 2023-10-16T14:33:13.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da92e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40c2390 con 0x7f77d415c9f0 2023-10-16T14:33:13.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 0x7f77d4150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 0x7f77d4150bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54132/0 (socket says 172.21.15.29:54132) 2023-10-16T14:33:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 -- 172.21.15.29:0/518961752 learned_addr learned my addr 172.21.15.29:0/518961752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da12d640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77d415dde0 0x7f77d4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d912b640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d415c9f0 0x7f77d406b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 -- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d415c9f0 msgr2=0x7f77d406b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d415c9f0 0x7f77d406b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 -- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77d415dde0 msgr2=0x7f77d4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77d415dde0 0x7f77d4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77d992c640 1 -- 172.21.15.29:0/518961752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40094d0 con 0x7f77d4070f20 2023-10-16T14:33:13.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.989+0000 7f77da12d640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77d415dde0 0x7f77d4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:13.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.990+0000 7f77d992c640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 0x7f77d4150bf0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c440 tx=0x7f77d000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.990+0000 7f77ca7fc640 1 -- 172.21.15.29:0/518961752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0005480 con 0x7f77d4070f20 2023-10-16T14:33:13.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.990+0000 7f77ca7fc640 1 -- 172.21.15.29:0/518961752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d00035f0 con 0x7f77d4070f20 2023-10-16T14:33:13.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.990+0000 7f77ca7fc640 1 -- 172.21.15.29:0/518961752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d00039c0 con 0x7f77d4070f20 2023-10-16T14:33:13.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.990+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d40c48f0 con 0x7f77d4070f20 2023-10-16T14:33:13.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.990+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d4151130 con 0x7f77d4070f20 2023-10-16T14:33:13.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.993+0000 7f77ca7fc640 1 -- 172.21.15.29:0/518961752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f77d0003b20 con 0x7f77d4070f20 2023-10-16T14:33:13.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.994+0000 7f77ca7fc640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f77b806afc0 0x7f77b806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.994+0000 7f77ca7fc640 1 -- 172.21.15.29:0/518961752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77d0088820 con 0x7f77d4070f20 2023-10-16T14:33:13.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.994+0000 7f77de7f6740 1 Processor -- start 2023-10-16T14:33:13.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.994+0000 7f77de7f6740 1 -- start start 2023-10-16T14:33:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.994+0000 7f77d912b640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f77b806afc0 0x7f77b806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.995+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99ba1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.995+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bd0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.995+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99ba15b0 0x55cd99bd1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.995+0000 7f77de7f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99ac17a0 con 0x55cd99bce550 2023-10-16T14:33:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.995+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99b13740 con 0x55cd99bce170 2023-10-16T14:33:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.995+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd99b10e30 con 0x55cd99ba15b0 2023-10-16T14:33:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bd0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bd0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54134/0 (socket says 172.21.15.29:54134) 2023-10-16T14:33:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 -- 172.21.15.29:0/3362528508 learned_addr learned my addr 172.21.15.29:0/3362528508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d992c640 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99ba1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f77b806afc0 0x7f77b806d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77c4036000 tx=0x7f77c403c040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77da12d640 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99ba15b0 0x55cd99bd1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 -- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99ba15b0 msgr2=0x55cd99bd1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99ba15b0 0x55cd99bd1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 -- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 msgr2=0x55cd99ba1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99ba1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.996+0000 7f77d912b640 1 -- 172.21.15.29:0/3362528508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd99ad0910 con 0x55cd99bce550 2023-10-16T14:33:13.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.997+0000 7f77d912b640 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bd0f00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f77c4006da0 tx=0x7f77c4008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fd625fbc979efc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.997+0000 7f77b77fe640 1 -- 172.21.15.29:0/3362528508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c403aa30 con 0x55cd99bce550 2023-10-16T14:33:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.997+0000 7f77b77fe640 1 -- 172.21.15.29:0/3362528508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c403ab90 con 0x55cd99bce550 2023-10-16T14:33:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.997+0000 7f77b77fe640 1 -- 172.21.15.29:0/3362528508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c404e410 con 0x55cd99bce550 2023-10-16T14:33:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.997+0000 7f77de7f6740 1 -- 172.21.15.29:0/3362528508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 msgr2=0x55cd99bd0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.997+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bd0f00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f77c4006da0 tx=0x7f77c4008030 comp rx=0 tx=0).stop 2023-10-16T14:33:13.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 -- 172.21.15.29:0/3362528508 shutdown_connections 2023-10-16T14:33:13.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bd0f00 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99ba15b0 0x55cd99bd1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3362528508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99ba1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 -- 172.21.15.29:0/3362528508 >> 172.21.15.29:0/3362528508 conn(0x55cd99ba3380 msgr2=0x55cd99bebf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 -- 172.21.15.29:0/3362528508 shutdown_connections 2023-10-16T14:33:13.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 -- 172.21.15.29:0/3362528508 wait complete. 2023-10-16T14:33:13.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.998+0000 7f77de7f6740 1 -- start start 2023-10-16T14:33:13.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:12.999+0000 7f77de7f6740 1 Processor -- start 2023-10-16T14:33:13.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bbba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bc1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99bc2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99b0c720 con 0x55cd99bce550 2023-10-16T14:33:13.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99aaa970 con 0x55cd99bce170 2023-10-16T14:33:13.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd99aaa610 con 0x55cd99bbbfc0 2023-10-16T14:33:13.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77de7f6740 10 client.0 _ll_register_callbacks cb 0x55cd99acc9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:33:13.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bc1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bc1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54144/0 (socket says 172.21.15.29:54144) 2023-10-16T14:33:13.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 -- 172.21.15.29:0/1005198075 learned_addr learned my addr 172.21.15.29:0/1005198075 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:13.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d992c640 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bbba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 -- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 msgr2=0x55cd99bc2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99bc2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 -- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 msgr2=0x55cd99bbba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bbba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.000+0000 7f77d912b640 1 -- 172.21.15.29:0/1005198075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d40cdbe0 con 0x55cd99bce550 2023-10-16T14:33:13.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77d912b640 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bc1ae0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f77c4006da0 tx=0x7f77c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12858eecb2beaa83 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:33:13.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77b4ff9640 1 -- 172.21.15.29:0/1005198075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c404ed10 con 0x55cd99bce550 2023-10-16T14:33:13.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77b4ff9640 1 -- 172.21.15.29:0/1005198075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77c4059af0 con 0x55cd99bce550 2023-10-16T14:33:13.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd99b33160 con 0x55cd99bce550 2023-10-16T14:33:13.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77b4ff9640 1 -- 172.21.15.29:0/1005198075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77c404ee70 con 0x55cd99bce550 2023-10-16T14:33:13.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77de7f6740 10 client.6987 fetch_fsmap learned FSMap version 42 2023-10-16T14:33:13.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55cd99b0ada0 con 0x55cd99bce550 2023-10-16T14:33:13.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77b4ff9640 1 -- 172.21.15.29:0/1005198075 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cd99b33160 con 0x55cd99bce550 2023-10-16T14:33:13.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.001+0000 7f77b4ff9640 1 -- 172.21.15.29:0/1005198075 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 42) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f77c4059cf0 con 0x55cd99bce550 2023-10-16T14:33:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77de7f6740 10 client.6987 fetch_fsmap finished waiting for FSMap version 42 2023-10-16T14:33:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77de7f6740 10 client.6987 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:33:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77de7f6740 20 client.6987 populate_metadata read hostname 'smithi029' 2023-10-16T14:33:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77de7f6740 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99bffe30 0x55cd99bbc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77de7f6740 4 client.6987 mds_command: new command op to 5926 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:33:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 --> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd99adecc0 con 0x55cd99bffe30 2023-10-16T14:33:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.002+0000 7f77d992c640 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99bffe30 0x55cd99bbc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.003+0000 7f77d992c640 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99bffe30 0x55cd99bbc6d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918afe88326f8c25 server_cookie=b3b9b5527b7a66e4 in_seq=0 out_seq=0 2023-10-16T14:33:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.003+0000 7f77b4ff9640 10 client.6987 ms_handle_connect on v2:172.21.15.170:6836/3812375996 2023-10-16T14:33:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.005+0000 7f77b4ff9640 1 -- 172.21.15.29:0/1005198075 <== mds.0 v2:172.21.15.170:6836/3812375996 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cd99adecc0 con 0x55cd99bffe30 2023-10-16T14:33:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.005+0000 7f77b4ff9640 10 client.6987 handle_command_reply: tid=0 2023-10-16T14:33:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.005+0000 7f77de7f6740 1 client.6987 shutdown 2023-10-16T14:33:13.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 msgr2=0x55cd99bc1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bc1ae0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f77c4006da0 tx=0x7f77c4004730 comp rx=0 tx=0).stop 2023-10-16T14:33:13.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 shutdown_connections 2023-10-16T14:33:13.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99bffe30 0x55cd99bbc6d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce550 0x55cd99bc1ae0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99bc2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 --2- 172.21.15.29:0/1005198075 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bbba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 >> 172.21.15.29:0/1005198075 conn(0x55cd99ba15b0 msgr2=0x55cd99bd0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77b4ff9640 0 client.6987 ms_handle_reset on v2:172.21.15.170:6836/3812375996 2023-10-16T14:33:13.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 shutdown_connections 2023-10-16T14:33:13.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 1 -- 172.21.15.29:0/1005198075 wait complete. 2023-10-16T14:33:13.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.006+0000 7f77de7f6740 20 client.6987 trim_cache size 0 max 16384 2023-10-16T14:33:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.023+0000 7f77de7f6740 1 Processor -- start 2023-10-16T14:33:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.023+0000 7f77de7f6740 1 -- start start 2023-10-16T14:33:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 0x55cd99bbb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bed330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bce550 0x55cd99bfebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77de7f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99870f20 con 0x55cd99bbbfc0 2023-10-16T14:33:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99bd3880 con 0x55cd99bce170 2023-10-16T14:33:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd99be6ca0 con 0x55cd99bce550 2023-10-16T14:33:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 0x55cd99bbb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 0x55cd99bbb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54156/0 (socket says 172.21.15.29:54156) 2023-10-16T14:33:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 -- 172.21.15.29:0/87971499 learned_addr learned my addr 172.21.15.29:0/87971499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 -- 172.21.15.29:0/87971499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bce550 msgr2=0x55cd99bfebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bce550 0x55cd99bfebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 -- 172.21.15.29:0/87971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 msgr2=0x55cd99bed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.024+0000 7f77d992c640 1 -- 172.21.15.29:0/87971499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd99aa9b80 con 0x55cd99bbbfc0 2023-10-16T14:33:13.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77d992c640 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 0x55cd99bbb2f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f77d0057210 tx=0x7f77d001d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7cdaefac218d34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77b7fff640 1 -- 172.21.15.29:0/87971499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d00558b0 con 0x55cd99bbbfc0 2023-10-16T14:33:13.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77b7fff640 1 -- 172.21.15.29:0/87971499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d0053c20 con 0x55cd99bbbfc0 2023-10-16T14:33:13.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77b7fff640 1 -- 172.21.15.29:0/87971499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0086f10 con 0x55cd99bbbfc0 2023-10-16T14:33:13.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 -- 172.21.15.29:0/87971499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 msgr2=0x55cd99bbb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 0x55cd99bbb2f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f77d0057210 tx=0x7f77d001d040 comp rx=0 tx=0).stop 2023-10-16T14:33:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 -- 172.21.15.29:0/87971499 shutdown_connections 2023-10-16T14:33:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bce550 0x55cd99bfebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce170 0x55cd99bed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 --2- 172.21.15.29:0/87971499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bbbfc0 0x55cd99bbb2f0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.025+0000 7f77de7f6740 1 -- 172.21.15.29:0/87971499 >> 172.21.15.29:0/87971499 conn(0x55cd99ba15b0 msgr2=0x55cd99be6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.026+0000 7f77de7f6740 1 -- 172.21.15.29:0/87971499 shutdown_connections 2023-10-16T14:33:13.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.026+0000 7f77de7f6740 1 -- 172.21.15.29:0/87971499 wait complete. 2023-10-16T14:33:13.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.026+0000 7f77de7f6740 1 -- start start 2023-10-16T14:33:13.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.026+0000 7f77de7f6740 1 Processor -- start 2023-10-16T14:33:13.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.027+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99c53e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.027+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 0x55cd99be1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77de7f6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce550 0x55cd99bcb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77de7f6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99c06ce0 con 0x55cd99bce170 2023-10-16T14:33:13.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd99bcc440 con 0x55cd99bce550 2023-10-16T14:33:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77de7f6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd99be19c0 con 0x55cd99bbbfc0 2023-10-16T14:33:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77de7f6740 10 client.0 _ll_register_callbacks cb 0x7f77c4031a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:33:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 0x55cd99be1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 0x55cd99be1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54160/0 (socket says 172.21.15.29:54160) 2023-10-16T14:33:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 -- 172.21.15.29:0/3928609220 learned_addr learned my addr 172.21.15.29:0/3928609220 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d992c640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99c53e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77da12d640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce550 0x55cd99bcb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 -- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 msgr2=0x55cd99c53e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99c53e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 -- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce550 msgr2=0x55cd99bcb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce550 0x55cd99bcb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.028+0000 7f77d912b640 1 -- 172.21.15.29:0/3928609220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d4151130 con 0x55cd99bce170 2023-10-16T14:33:13.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77d912b640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 0x55cd99be1e50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f77c4031b60 tx=0x7f77c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571e048e8b039ca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:33:13.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77b4ff9640 1 -- 172.21.15.29:0/3928609220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4004210 con 0x55cd99bce170 2023-10-16T14:33:13.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd99ba5d20 con 0x55cd99bce170 2023-10-16T14:33:13.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77b4ff9640 1 -- 172.21.15.29:0/3928609220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77c4003610 con 0x55cd99bce170 2023-10-16T14:33:13.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77b4ff9640 1 -- 172.21.15.29:0/3928609220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77c40037d0 con 0x55cd99bce170 2023-10-16T14:33:13.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77de7f6740 10 client.6993 fetch_fsmap learned FSMap version 42 2023-10-16T14:33:13.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55cd99acf170 con 0x55cd99bce170 2023-10-16T14:33:13.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.029+0000 7f77b4ff9640 1 -- 172.21.15.29:0/3928609220 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cd99ba5d20 con 0x55cd99bce170 2023-10-16T14:33:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77b4ff9640 1 -- 172.21.15.29:0/3928609220 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 42) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f77c4003930 con 0x55cd99bce170 2023-10-16T14:33:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77de7f6740 10 client.6993 fetch_fsmap finished waiting for FSMap version 42 2023-10-16T14:33:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77de7f6740 10 client.6993 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:33:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77de7f6740 20 client.6993 populate_metadata read hostname 'smithi029' 2023-10-16T14:33:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99ba4d10 0x55cd99ba50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77de7f6740 4 client.6993 mds_command: new command op to 5926 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:33:13.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 --> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f77c403ab90 con 0x55cd99ba4d10 2023-10-16T14:33:13.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.030+0000 7f77d992c640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99ba4d10 0x55cd99ba50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.031+0000 7f77d992c640 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99ba4d10 0x55cd99ba50f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf7dda5f614d302 server_cookie=b6e38282498f4a03 in_seq=0 out_seq=0 2023-10-16T14:33:13.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.031+0000 7f77b4ff9640 10 client.6993 ms_handle_connect on v2:172.21.15.170:6836/3812375996 2023-10-16T14:33:13.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77b4ff9640 1 -- 172.21.15.29:0/3928609220 <== mds.0 v2:172.21.15.170:6836/3812375996 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7f77c403ab90 con 0x55cd99ba4d10 2023-10-16T14:33:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77b4ff9640 10 client.6993 handle_command_reply: tid=0 2023-10-16T14:33:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 client.6993 shutdown 2023-10-16T14:33:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 msgr2=0x55cd99be1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 0x55cd99be1e50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f77c4031b60 tx=0x7f77c40046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 shutdown_connections 2023-10-16T14:33:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:6836/3812375996,v1:172.21.15.170:6837/3812375996] conn(0x55cd99ba4d10 0x55cd99ba50f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd99bce170 0x55cd99be1e50 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cd99bce550 0x55cd99bcb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 --2- 172.21.15.29:0/3928609220 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55cd99bbbfc0 0x55cd99c53e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 >> 172.21.15.29:0/3928609220 conn(0x55cd99ba15b0 msgr2=0x55cd99bed100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.032+0000 7f77b4ff9640 0 client.6993 ms_handle_reset on v2:172.21.15.170:6836/3812375996 2023-10-16T14:33:13.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 shutdown_connections 2023-10-16T14:33:13.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77de7f6740 1 -- 172.21.15.29:0/3928609220 wait complete. 2023-10-16T14:33:13.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77de7f6740 20 client.6993 trim_cache size 0 max 16384 2023-10-16T14:33:13.059 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:33:13.060 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:33:13.060 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:33:13.060 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 5926, 2023-10-16T14:33:13.060 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:33:13.060 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:33:13.061 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:33:13.061 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.7659114069999999, 2023-10-16T14:33:13.061 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 42, 2023-10-16T14:33:13.061 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 97, 2023-10-16T14:33:13.061 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 97, 2023-10-16T14:33:13.062 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.57233954 2023-10-16T14:33:13.062 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:33:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f77b806afc0 msgr2=0x7f77b806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f77b806afc0 0x7f77b806d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77c4036000 tx=0x7f77c403c040 comp rx=0 tx=0).stop 2023-10-16T14:33:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 msgr2=0x7f77d4150bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 0x7f77d4150bf0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c440 tx=0x7f77d000c900 comp rx=0 tx=0).stop 2023-10-16T14:33:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 shutdown_connections 2023-10-16T14:33:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77d415dde0 0x7f77d4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77d415c9f0 0x7f77d406b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f77b806afc0 0x7f77b806d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 --2- 172.21.15.29:0/518961752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77d4070f20 0x7f77d4150bf0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.033+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 >> 172.21.15.29:0/518961752 conn(0x7f77d40ce6a0 msgr2=0x7f77d40c9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.034+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 shutdown_connections 2023-10-16T14:33:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.034+0000 7f77da92e640 1 -- 172.21.15.29:0/518961752 wait complete. 2023-10-16T14:33:13.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:33:13.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.288+0000 7f94466f2640 1 Processor -- start 2023-10-16T14:33:13.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.288+0000 7f94466f2640 1 -- start start 2023-10-16T14:33:13.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.289+0000 7f94466f2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f944015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.289+0000 7f94466f2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94400ce390 0x7f9440070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.289+0000 7f94466f2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944015ed60 con 0x7f944015ae80 2023-10-16T14:33:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.289+0000 7f94466f2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944006b810 con 0x7f94400ce390 2023-10-16T14:33:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.289+0000 7f94466f2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400707e0 con 0x7f944015aaa0 2023-10-16T14:33:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9437fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f944015ae80 0x7f94400cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36810/0 (socket says 172.21.15.29:36810) 2023-10-16T14:33:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9437fff640 1 -- 172.21.15.29:0/3696500214 learned_addr learned my addr 172.21.15.29:0/3696500214 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9444c8f640 1 --2- 172.21.15.29:0/3696500214 >> v2:172.21.15.29:3300/0 conn(0x7f94400ce390 0x7f9440070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f943ffff640 1 --2- 172.21.15.29:0/3696500214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f944015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f943f7fe640 1 -- 172.21.15.29:0/3696500214 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774684891 0 0) 0x7f944015ed60 con 0x7f944015ae80 2023-10-16T14:33:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f943f7fe640 1 -- 172.21.15.29:0/3696500214 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f942c003520 con 0x7f944015ae80 2023-10-16T14:33:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9444c8f640 1 -- 172.21.15.29:0/3696500214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 msgr2=0x7f944015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9444c8f640 1 --2- 172.21.15.29:0/3696500214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f944015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9444c8f640 1 -- 172.21.15.29:0/3696500214 >> v1:172.21.15.29:6789/0 conn(0x7f944015ae80 legacy=0x7f94400cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.290+0000 7f9444c8f640 1 -- 172.21.15.29:0/3696500214 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944015ef90 con 0x7f94400ce390 2023-10-16T14:33:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.291+0000 7f9444c8f640 1 --2- 172.21.15.29:0/3696500214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f9440070f50 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9438009d90 tx=0x7f943802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13155dc773670c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f943f7fe640 1 -- 172.21.15.29:0/3696500214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943802cc50 con 0x7f94400ce390 2023-10-16T14:33:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f943f7fe640 1 -- 172.21.15.29:0/3696500214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943802cdb0 con 0x7f94400ce390 2023-10-16T14:33:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f943f7fe640 1 -- 172.21.15.29:0/3696500214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438034640 con 0x7f94400ce390 2023-10-16T14:33:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 -- 172.21.15.29:0/3696500214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 msgr2=0x7f9440070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 --2- 172.21.15.29:0/3696500214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f9440070f50 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9438009d90 tx=0x7f943802bea0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 -- 172.21.15.29:0/3696500214 shutdown_connections 2023-10-16T14:33:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 --2- 172.21.15.29:0/3696500214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f9440070f50 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 --2- 172.21.15.29:0/3696500214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f944015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 -- 172.21.15.29:0/3696500214 >> 172.21.15.29:0/3696500214 conn(0x7f94400cb410 msgr2=0x7f94400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.292+0000 7f94466f2640 1 -- 172.21.15.29:0/3696500214 shutdown_connections 2023-10-16T14:33:13.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.293+0000 7f94466f2640 1 -- 172.21.15.29:0/3696500214 wait complete. 2023-10-16T14:33:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.294+0000 7f94466f2640 1 Processor -- start 2023-10-16T14:33:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.294+0000 7f94466f2640 1 -- start start 2023-10-16T14:33:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.295+0000 7f94466f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f94401f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.295+0000 7f94466f2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f94401f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.295+0000 7f94466f2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015ae80 0x7f94400ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.295+0000 7f94466f2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94400af120 con 0x7f94400ce390 2023-10-16T14:33:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.295+0000 7f94466f2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94400af400 con 0x7f944015ae80 2023-10-16T14:33:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.295+0000 7f94466f2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400af6e0 con 0x7f944015aaa0 2023-10-16T14:33:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f94401f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f94401f5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54176/0 (socket says 172.21.15.29:54176) 2023-10-16T14:33:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 -- 172.21.15.29:0/4041179256 learned_addr learned my addr 172.21.15.29:0/4041179256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f9437fff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f94401f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 -- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 msgr2=0x7f94401f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f94401f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 -- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015ae80 msgr2=0x7f94400ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015ae80 0x7f94400ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 -- 172.21.15.29:0/4041179256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9438009a70 con 0x7f94400ce390 2023-10-16T14:33:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f9437fff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f94401f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.296+0000 7f943ffff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f94401f5b90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f943000d800 tx=0x7f943000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.297+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943000b6c0 con 0x7f94400ce390 2023-10-16T14:33:13.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.297+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943000b820 con 0x7f94400ce390 2023-10-16T14:33:13.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.297+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94400af9c0 con 0x7f94400ce390 2023-10-16T14:33:13.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.297+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943000bbb0 con 0x7f94400ce390 2023-10-16T14:33:13.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.297+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94400ae930 con 0x7f94400ce390 2023-10-16T14:33:13.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.298+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944004e160 con 0x7f94400ce390 2023-10-16T14:33:13.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.299+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f943000bd10 con 0x7f94400ce390 2023-10-16T14:33:13.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.300+0000 7f943d7fa640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.300+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9430089cf0 con 0x7f94400ce390 2023-10-16T14:33:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.300+0000 7f9437fff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.301+0000 7f9437fff640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f942006af90 0x7f942006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f944006d780 tx=0x7f9428009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:13.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.304+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f943008e070 con 0x7f94400ce390 2023-10-16T14:33:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.537+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f94400cdc60 con 0x7f94400ce390 2023-10-16T14:33:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.538+0000 7f943d7fa640 1 -- 172.21.15.29:0/4041179256 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9430054c90 con 0x7f94400ce390 2023-10-16T14:33:13.542 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:33:13.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f942006af90 msgr2=0x7f942006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f942006af90 0x7f942006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f944006d780 tx=0x7f9428009040 comp rx=0 tx=0).stop 2023-10-16T14:33:13.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 msgr2=0x7f94401f5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:13.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f94401f5b90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f943000d800 tx=0x7f943000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 shutdown_connections 2023-10-16T14:33:13.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015ae80 0x7f94400ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015aaa0 0x7f94401f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 --2- 172.21.15.29:0/4041179256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94400ce390 0x7f94401f5b90 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.541+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 >> 172.21.15.29:0/4041179256 conn(0x7f94400cb410 msgr2=0x7f94400b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.542+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 shutdown_connections 2023-10-16T14:33:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:13.542+0000 7f94466f2640 1 -- 172.21.15.29:0/4041179256 wait complete. 2023-10-16T14:33:13.564 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:33:13.564 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:33:13.564 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:33:13.565 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:33:13.565 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:13.565 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:33:13.566 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:33:13.566 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:33:13.566 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:33:13.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:13.567 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:33:13.624 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:33:13.625 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:33:13.625 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:33:13.625 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.625 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:33:13.626 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.626 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:33:13.626 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:13.626 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:33:13.626 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:33:13.627 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:13.627 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:33:13.627 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:33:13.627 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:13.627 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:33:13.628 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:33:13.628 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41331sec preferred_lft 41331sec 2023-10-16T14:33:13.628 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:33:13.628 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.628 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:33:13.629 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:13.629 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:33:13.629 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:33:13.629 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:13.629 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:33:13.630 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.630 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:33:13.630 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.630 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:33:13.630 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:33:13.631 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:33:13.631 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.631 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:33:13.631 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:33:13.631 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:33:13.631 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:13.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:13.633 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:33:13.737 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:33:13.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:13.741 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:33:13.817 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:33:13.818 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:33:13.818 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:33:13.818 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:33:13.818 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:33:13.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:13.819 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:33:13.819 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:33:13.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:33:13.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:33:13.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:33:13.821 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:33:14.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:14.032 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:33:14.088 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:33:14.089 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:33:14.089 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:33:14.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:14.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:33:14.192 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:33:14.193 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:14.254 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:33:14.255 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:33:14.255 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:14.313 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:33:14.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:14.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:14.460 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:14.460 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:33:14.461 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:33:14.483 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:14.483 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:33:14.484 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:33:14.484 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:33:14.484 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:33:14.511 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:14.511 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:33:14.512 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:33:14.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:33:14.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:33:14.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:33:14.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:33:14.513 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:33:14.516 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:33:14.516 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:33:14.517 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:33:14.517 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:33:14.517 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:33:14.522 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:33:14.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:14.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:14.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:14.605 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:33:14.676 INFO:teuthology.orchestra.run.smithi029.stdout:client7005 2023-10-16T14:33:14.677 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:14.677 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:14.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:33:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.956+0000 7f9827467640 1 Processor -- start 2023-10-16T14:33:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.956+0000 7f9827467640 1 -- start start 2023-10-16T14:33:14.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9827467640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f98200cded0 0x7f98200b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9827467640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982015d150 con 0x7f98200b8480 2023-10-16T14:33:14.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9827467640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200d1180 con 0x7f982015cd70 2023-10-16T14:33:14.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9827467640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200d1460 con 0x7f98200cded0 2023-10-16T14:33:14.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9825c64640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f98200b8480 0x7f98200cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44164/0 (socket says 172.21.15.29:44164) 2023-10-16T14:33:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9825c64640 1 -- 172.21.15.29:0/1258941914 learned_addr learned my addr 172.21.15.29:0/1258941914 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.958+0000 7f9826c66640 1 --2- 172.21.15.29:0/1258941914 >> v2:172.21.15.170:3300/0 conn(0x7f98200cded0 0x7f98200b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79622349 0 0) 0x7f982015d150 con 0x7f98200b8480 2023-10-16T14:33:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c003520 con 0x7f98200b8480 2023-10-16T14:33:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4079013672 0 0) 0x7f980c003520 con 0x7f98200b8480 2023-10-16T14:33:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 >> v2:172.21.15.170:3300/0 conn(0x7f98200cded0 msgr2=0x7f98200b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 --2- 172.21.15.29:0/1258941914 >> v2:172.21.15.170:3300/0 conn(0x7f98200cded0 0x7f98200b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 >> v1:172.21.15.170:6789/0 conn(0x7f982015cd70 legacy=0x7f98200d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200d1740 con 0x7f98200b8480 2023-10-16T14:33:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9814001670 con 0x7f98200b8480 2023-10-16T14:33:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 >> v1:172.21.15.29:6789/0 conn(0x7f98200b8480 legacy=0x7f98200cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980c003970 0x7f980c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200d1460 con 0x7f980c005180 2023-10-16T14:33:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00c020 con 0x7f980c0047c0 2023-10-16T14:33:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.959+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00c1e0 con 0x7f980c003970 2023-10-16T14:33:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826c66640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9825c64640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826465640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980c003970 0x7f980c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826c66640 1 -- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980c003970 msgr2=0x7f980c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826c66640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980c003970 0x7f980c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826c66640 1 -- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c0047c0 msgr2=0x7f980c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826c66640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9826c66640 1 -- 172.21.15.29:0/1258941914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c003520 con 0x7f980c005180 2023-10-16T14:33:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.960+0000 7f9825c64640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c0047c0 0x7f980c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:14.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.961+0000 7f9826c66640 1 -- 172.21.15.29:0/1258941914 reap_dead start 2023-10-16T14:33:14.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.961+0000 7f9826c66640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f980c005180 0x7f980c00b980 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9818009fc0 tx=0x7f980c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b846775554f0e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:14.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.962+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981802cc60 con 0x7f980c005180 2023-10-16T14:33:14.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.962+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981800f040 con 0x7f980c005180 2023-10-16T14:33:14.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.962+0000 7f9825463640 1 -- 172.21.15.29:0/1258941914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818003990 con 0x7f980c005180 2023-10-16T14:33:14.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.962+0000 7f9827467640 1 -- 172.21.15.29:0/1258941914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f980c005180 msgr2=0x7f980c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:14.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.962+0000 7f9827467640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f980c005180 0x7f980c00b980 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9818009fc0 tx=0x7f980c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.963+0000 7f9827467640 1 -- 172.21.15.29:0/1258941914 shutdown_connections 2023-10-16T14:33:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.963+0000 7f9827467640 1 --2- 172.21.15.29:0/1258941914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f980c005180 0x7f980c00b980 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.963+0000 7f9827467640 1 -- 172.21.15.29:0/1258941914 >> 172.21.15.29:0/1258941914 conn(0x7f98200cf570 msgr2=0x7f98200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.963+0000 7f9827467640 1 -- 172.21.15.29:0/1258941914 shutdown_connections 2023-10-16T14:33:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.963+0000 7f9827467640 1 -- 172.21.15.29:0/1258941914 wait complete. 2023-10-16T14:33:14.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.965+0000 7f9827467640 1 Processor -- start 2023-10-16T14:33:14.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.966+0000 7f9827467640 1 -- start start 2023-10-16T14:33:14.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9827467640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c005180 0x7f98201f5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9827467640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 0x7f9820155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9827467640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98201555d0 0x7f98200cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9827467640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820155a70 con 0x7f98201f3620 2023-10-16T14:33:14.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9827467640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200ca340 con 0x7f980c005180 2023-10-16T14:33:14.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9827467640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200ca620 con 0x7f98201555d0 2023-10-16T14:33:14.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.967+0000 7f9825c64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 0x7f9820155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 0x7f9820155090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54886/0 (socket says 172.21.15.29:54886) 2023-10-16T14:33:14.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 -- 172.21.15.29:0/3715002028 learned_addr learned my addr 172.21.15.29:0/3715002028 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:14.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9826465640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c005180 0x7f98201f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9826c66640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98201555d0 0x7f98200cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 -- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98201555d0 msgr2=0x7f98200cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:14.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98201555d0 0x7f98200cb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 -- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c005180 msgr2=0x7f98201f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:14.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c005180 0x7f98201f5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:14.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.968+0000 7f9825c64640 1 -- 172.21.15.29:0/3715002028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200ca900 con 0x7f98201f3620 2023-10-16T14:33:14.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.969+0000 7f9825c64640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 0x7f9820155090 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9814001200 tx=0x7f981400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:14.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.969+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400c640 con 0x7f98201f3620 2023-10-16T14:33:14.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.969+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981400c7a0 con 0x7f98201f3620 2023-10-16T14:33:14.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.969+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400cb30 con 0x7f98201f3620 2023-10-16T14:33:14.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.969+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98200cabe0 con 0x7f98201f3620 2023-10-16T14:33:14.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.969+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98201f3220 con 0x7f98201f3620 2023-10-16T14:33:14.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.970+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982004e9b0 con 0x7f98201f3620 2023-10-16T14:33:14.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.971+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f981401d460 con 0x7f98201f3620 2023-10-16T14:33:14.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.972+0000 7f9812ffd640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f980406af90 0x7f980406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:14.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.972+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9814089830 con 0x7f98201f3620 2023-10-16T14:33:14.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.973+0000 7f9826465640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f980406af90 0x7f980406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:14.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.974+0000 7f9826465640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f980406af90 0x7f980406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f981c007fd0 tx=0x7f981c009040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:14.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:14.977+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f98140547c0 con 0x7f98201f3620 2023-10-16T14:33:15.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.202+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 --> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f98201f3440 con 0x7f980406af90 2023-10-16T14:33:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.206+0000 7f9812ffd640 1 -- 172.21.15.29:0/3715002028 <== mgr.5390 v2:172.21.15.170:6832/610139797 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f98201f3440 con 0x7f980406af90 2023-10-16T14:33:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:33:15.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:15.210 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:33:15.210 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:15.211 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:33:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f980406af90 msgr2=0x7f980406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f980406af90 0x7f980406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f981c007fd0 tx=0x7f981c009040 comp rx=0 tx=0).stop 2023-10-16T14:33:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 msgr2=0x7f9820155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 0x7f9820155090 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9814001200 tx=0x7f981400eb90 comp rx=0 tx=0).stop 2023-10-16T14:33:15.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 shutdown_connections 2023-10-16T14:33:15.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f980406af90 0x7f980406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98201f3620 0x7f9820155090 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98201555d0 0x7f98200cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 --2- 172.21.15.29:0/3715002028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f980c005180 0x7f98201f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.208+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 >> 172.21.15.29:0/3715002028 conn(0x7f98200cf570 msgr2=0x7f98200b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.209+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 shutdown_connections 2023-10-16T14:33:15.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.209+0000 7f9827467640 1 -- 172.21.15.29:0/3715002028 wait complete. 2023-10-16T14:33:15.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:33:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.457+0000 7f91f32f3640 1 Processor -- start 2023-10-16T14:33:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.457+0000 7f91f32f3640 1 -- start start 2023-10-16T14:33:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.459+0000 7f91f32f3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91ec0ad970 0x7f91ec0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.459+0000 7f91f32f3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f91ec0ccfb0 0x7f91ec0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.459+0000 7f91f32f3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec15b9d0 con 0x7f91ec06c040 2023-10-16T14:33:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.459+0000 7f91f32f3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec15bcb0 con 0x7f91ec0ad970 2023-10-16T14:33:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.459+0000 7f91f32f3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec15bf90 con 0x7f91ec0ccfb0 2023-10-16T14:33:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.459+0000 7f91f1af0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91ec0ad970 0x7f91ec0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f1af0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91ec0ad970 0x7f91ec0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54894/0 (socket says 172.21.15.29:54894) 2023-10-16T14:33:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f1af0640 1 -- 172.21.15.29:0/1444285500 learned_addr learned my addr 172.21.15.29:0/1444285500 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f2af2640 1 --2- 172.21.15.29:0/1444285500 >> v2:172.21.15.170:3300/0 conn(0x7f91ec0ccfb0 0x7f91ec0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f1af0640 1 -- 172.21.15.29:0/1444285500 >> v2:172.21.15.170:3300/0 conn(0x7f91ec0ccfb0 msgr2=0x7f91ec0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f1af0640 1 --2- 172.21.15.29:0/1444285500 >> v2:172.21.15.170:3300/0 conn(0x7f91ec0ccfb0 0x7f91ec0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f1af0640 1 -- 172.21.15.29:0/1444285500 >> v1:172.21.15.170:6789/0 conn(0x7f91ec06c040 legacy=0x7f91ec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f1af0640 1 -- 172.21.15.29:0/1444285500 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec0cc270 con 0x7f91ec0ad970 2023-10-16T14:33:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.460+0000 7f91f2af2640 1 --2- 172.21.15.29:0/1444285500 >> v2:172.21.15.170:3300/0 conn(0x7f91ec0ccfb0 0x7f91ec0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.461+0000 7f91f1af0640 1 --2- 172.21.15.29:0/1444285500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec0ad970 0x7f91ec0cca70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009870 tx=0x7f91dc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff13fd42f685ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f12ef640 1 -- 172.21.15.29:0/1444285500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc030850 con 0x7f91ec0ad970 2023-10-16T14:33:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f12ef640 1 -- 172.21.15.29:0/1444285500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc0309b0 con 0x7f91ec0ad970 2023-10-16T14:33:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f12ef640 1 -- 172.21.15.29:0/1444285500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc030d40 con 0x7f91ec0ad970 2023-10-16T14:33:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f32f3640 1 -- 172.21.15.29:0/1444285500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec0ad970 msgr2=0x7f91ec0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f32f3640 1 --2- 172.21.15.29:0/1444285500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec0ad970 0x7f91ec0cca70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009870 tx=0x7f91dc02b990 comp rx=0 tx=0).stop 2023-10-16T14:33:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f32f3640 1 -- 172.21.15.29:0/1444285500 shutdown_connections 2023-10-16T14:33:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f32f3640 1 --2- 172.21.15.29:0/1444285500 >> v2:172.21.15.170:3300/0 conn(0x7f91ec0ccfb0 0x7f91ec0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f32f3640 1 --2- 172.21.15.29:0/1444285500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec0ad970 0x7f91ec0cca70 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.462+0000 7f91f32f3640 1 -- 172.21.15.29:0/1444285500 >> 172.21.15.29:0/1444285500 conn(0x7f91ec0ce6a0 msgr2=0x7f91ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.463+0000 7f91f32f3640 1 -- 172.21.15.29:0/1444285500 shutdown_connections 2023-10-16T14:33:15.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.463+0000 7f91f32f3640 1 -- 172.21.15.29:0/1444285500 wait complete. 2023-10-16T14:33:15.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.465+0000 7f91f32f3640 1 Processor -- start 2023-10-16T14:33:15.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.465+0000 7f91f32f3640 1 -- start start 2023-10-16T14:33:15.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f32f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 0x7f91ec153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:15.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f32f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91ec0ad970 0x7f91ec0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:15.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f32f3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91ec0ccfb0 0x7f91ec0af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:15.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f32f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0ae100 con 0x7f91ec06c040 2023-10-16T14:33:15.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f32f3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec154060 con 0x7f91ec0ccfb0 2023-10-16T14:33:15.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f32f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0ae3a0 con 0x7f91ec0ad970 2023-10-16T14:33:15.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f22f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 0x7f91ec153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:15.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f22f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 0x7f91ec153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54906/0 (socket says 172.21.15.29:54906) 2023-10-16T14:33:15.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.466+0000 7f91f22f1640 1 -- 172.21.15.29:0/124347937 learned_addr learned my addr 172.21.15.29:0/124347937 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:15.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f2af2640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91ec0ccfb0 0x7f91ec0af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:15.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f1af0640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91ec0ad970 0x7f91ec0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:15.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f22f1640 1 -- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91ec0ad970 msgr2=0x7f91ec0b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:15.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f22f1640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91ec0ad970 0x7f91ec0b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f22f1640 1 -- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91ec0ccfb0 msgr2=0x7f91ec0af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:15.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f22f1640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91ec0ccfb0 0x7f91ec0af160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:15.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f22f1640 1 -- 172.21.15.29:0/124347937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0094d0 con 0x7f91ec06c040 2023-10-16T14:33:15.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91f22f1640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 0x7f91ec153b20 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c450 tx=0x7f91e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:15.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.467+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e80052d0 con 0x7f91ec06c040 2023-10-16T14:33:15.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.468+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e8005430 con 0x7f91ec06c040 2023-10-16T14:33:15.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.468+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8012870 con 0x7f91ec06c040 2023-10-16T14:33:15.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.468+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec0ae680 con 0x7f91ec06c040 2023-10-16T14:33:15.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.468+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec0aec40 con 0x7f91ec06c040 2023-10-16T14:33:15.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.469+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec06b330 con 0x7f91ec06c040 2023-10-16T14:33:15.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.470+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f91e80129d0 con 0x7f91ec06c040 2023-10-16T14:33:15.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.471+0000 7f91e2ffd640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f91d006afc0 0x7f91d006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:15.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.471+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91e8089330 con 0x7f91ec06c040 2023-10-16T14:33:15.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.472+0000 7f91f1af0640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f91d006afc0 0x7f91d006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:15.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.473+0000 7f91f1af0640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f91d006afc0 0x7f91d006d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91dc036000 tx=0x7f91dc03b040 comp rx=0 tx=0).ready entity=mgr.5390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:15.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.476+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f91e8054230 con 0x7f91ec06c040 2023-10-16T14:33:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:15.720+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f91ec06da40 con 0x7f91ec06c040 2023-10-16T14:33:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.338+0000 7f91e2ffd640 1 -- 172.21.15.29:0/124347937 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v38) v1 ==== 92+0+0 (secure 0 0 0) 0x7f91e8057850 con 0x7f91ec06c040 2023-10-16T14:33:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.341+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f91d006afc0 msgr2=0x7f91d006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.341+0000 7f91f32f3640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f91d006afc0 0x7f91d006d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91dc036000 tx=0x7f91dc03b040 comp rx=0 tx=0).stop 2023-10-16T14:33:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.341+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 msgr2=0x7f91ec153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.341+0000 7f91f32f3640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 0x7f91ec153b20 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c450 tx=0x7f91e800c910 comp rx=0 tx=0).stop 2023-10-16T14:33:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 shutdown_connections 2023-10-16T14:33:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91ec0ccfb0 0x7f91ec0af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91ec0ad970 0x7f91ec0b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f91d006afc0 0x7f91d006d480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 --2- 172.21.15.29:0/124347937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec06c040 0x7f91ec153b20 secure :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c450 tx=0x7f91e800c910 comp rx=0 tx=0).stop 2023-10-16T14:33:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 >> 172.21.15.29:0/124347937 conn(0x7f91ec0ce6a0 msgr2=0x7f91ec0c5890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 shutdown_connections 2023-10-16T14:33:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.342+0000 7f91f32f3640 1 -- 172.21.15.29:0/124347937 wait complete. 2023-10-16T14:33:16.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:33:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.569+0000 7f0f0cb75640 1 Processor -- start 2023-10-16T14:33:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.569+0000 7f0f0cb75640 1 -- start start 2023-10-16T14:33:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.570+0000 7f0f0cb75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f080cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.570+0000 7f0f0cb75640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0f080cdd60 0x7f0f08071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.570+0000 7f0f0cb75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f08071fa0 0x7f0f081607e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.570+0000 7f0f0cb75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f080d0500 con 0x7f0f08071fa0 2023-10-16T14:33:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.570+0000 7f0f0cb75640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f080ce1c0 con 0x7f0f080cdd60 2023-10-16T14:33:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.570+0000 7f0f0cb75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080d12f0 con 0x7f0f0815cc80 2023-10-16T14:33:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0f07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f08071fa0 0x7f0f081607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0f07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f08071fa0 0x7f0f081607e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54934/0 (socket says 172.21.15.29:54934) 2023-10-16T14:33:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0f07fff640 1 -- 172.21.15.29:0/439920117 learned_addr learned my addr 172.21.15.29:0/439920117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:16.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0f077fe640 1 --2- 172.21.15.29:0/439920117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f080cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:16.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0efffff640 1 --2- 172.21.15.29:0/439920117 >> v2:172.21.15.170:3300/0 conn(0x7f0f080cdd60 0x7f0f08071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:16.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0f07fff640 1 -- 172.21.15.29:0/439920117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 msgr2=0x7f0f080cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:16.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.571+0000 7f0f07fff640 1 --2- 172.21.15.29:0/439920117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f080cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.572+0000 7f0f07fff640 1 -- 172.21.15.29:0/439920117 >> v2:172.21.15.170:3300/0 conn(0x7f0f080cdd60 msgr2=0x7f0f08071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.572+0000 7f0f07fff640 1 --2- 172.21.15.29:0/439920117 >> v2:172.21.15.170:3300/0 conn(0x7f0f080cdd60 0x7f0f08071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.572+0000 7f0f07fff640 1 -- 172.21.15.29:0/439920117 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08072410 con 0x7f0f08071fa0 2023-10-16T14:33:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.572+0000 7f0efffff640 1 --2- 172.21.15.29:0/439920117 >> v2:172.21.15.170:3300/0 conn(0x7f0f080cdd60 0x7f0f08071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.573+0000 7f0f07fff640 1 --2- 172.21.15.29:0/439920117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f081607e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f0f080d19e0 tx=0x7f0f0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfad92f33a7dda6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.573+0000 7f0f06ffd640 1 -- 172.21.15.29:0/439920117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0002ca70 con 0x7f0f08071fa0 2023-10-16T14:33:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.573+0000 7f0f06ffd640 1 -- 172.21.15.29:0/439920117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0002cbd0 con 0x7f0f08071fa0 2023-10-16T14:33:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.573+0000 7f0f06ffd640 1 -- 172.21.15.29:0/439920117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000343f0 con 0x7f0f08071fa0 2023-10-16T14:33:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 -- 172.21.15.29:0/439920117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 msgr2=0x7f0f081607e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/439920117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f081607e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f0f080d19e0 tx=0x7f0f0002bed0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 -- 172.21.15.29:0/439920117 shutdown_connections 2023-10-16T14:33:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/439920117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f081607e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/439920117 >> v2:172.21.15.170:3300/0 conn(0x7f0f080cdd60 0x7f0f08071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/439920117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f080cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 -- 172.21.15.29:0/439920117 >> 172.21.15.29:0/439920117 conn(0x7f0f08065ed0 msgr2=0x7f0f080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 -- 172.21.15.29:0/439920117 shutdown_connections 2023-10-16T14:33:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.574+0000 7f0f0cb75640 1 -- 172.21.15.29:0/439920117 wait complete. 2023-10-16T14:33:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.576+0000 7f0f0cb75640 1 Processor -- start 2023-10-16T14:33:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.576+0000 7f0f0cb75640 1 -- start start 2023-10-16T14:33:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f0cb75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f080c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f0cb75640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f080cdd60 0x7f0f080b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f0cb75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f0815a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f0cb75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c8110 con 0x7f0f08071fa0 2023-10-16T14:33:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f0cb75640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0815de90 con 0x7f0f080cdd60 2023-10-16T14:33:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f0cb75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c8360 con 0x7f0f0815cc80 2023-10-16T14:33:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f080c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f080c8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54938/0 (socket says 172.21.15.29:54938) 2023-10-16T14:33:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.577+0000 7f0f077fe640 1 -- 172.21.15.29:0/4225893708 learned_addr learned my addr 172.21.15.29:0/4225893708 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f07fff640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f0815a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0efffff640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f080cdd60 0x7f0f080b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f077fe640 1 -- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 msgr2=0x7f0f0815a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:16.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f077fe640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f0815a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f077fe640 1 -- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f080cdd60 msgr2=0x7f0f080b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:16.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f077fe640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f080cdd60 0x7f0f080b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f077fe640 1 -- 172.21.15.29:0/4225893708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f00009b30 con 0x7f0f08071fa0 2023-10-16T14:33:16.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f077fe640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f080c8e40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800e870 tx=0x7f0ef800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80094f0 con 0x7f0f08071fa0 2023-10-16T14:33:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef800c940 con 0x7f0f08071fa0 2023-10-16T14:33:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef800ccf0 con 0x7f0f08071fa0 2023-10-16T14:33:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.578+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f080c8640 con 0x7f0f08071fa0 2023-10-16T14:33:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.579+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f080c9380 con 0x7f0f08071fa0 2023-10-16T14:33:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.580+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0804e9b0 con 0x7f0f08071fa0 2023-10-16T14:33:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.581+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0ef8013470 con 0x7f0f08071fa0 2023-10-16T14:33:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.582+0000 7f0f04ff9640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 0x7f0ee806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.582+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ef80898c0 con 0x7f0f08071fa0 2023-10-16T14:33:16.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.582+0000 7f0efffff640 1 -- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 msgr2=0x7f0ee806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:16.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.582+0000 7f0efffff640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 0x7f0ee806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:16.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.586+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ef8054850 con 0x7f0f08071fa0 2023-10-16T14:33:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.783+0000 7f0efffff640 1 -- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 msgr2=0x7f0ee806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.783+0000 7f0efffff640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 0x7f0ee806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.783+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f0f080b6860 con 0x7f0f08071fa0 2023-10-16T14:33:16.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.789+0000 7f0f04ff9640 1 -- 172.21.15.29:0/4225893708 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0ef80210d0 con 0x7f0f08071fa0 2023-10-16T14:33:16.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 msgr2=0x7f0ee806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 0x7f0ee806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 msgr2=0x7f0f080c8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f080c8e40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800e870 tx=0x7f0ef800ed30 comp rx=0 tx=0).stop 2023-10-16T14:33:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 shutdown_connections 2023-10-16T14:33:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f0815cc80 0x7f0f0815a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f080cdd60 0x7f0f080b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f0ee806afe0 0x7f0ee806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 --2- 172.21.15.29:0/4225893708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f08071fa0 0x7f0f080c8e40 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 >> 172.21.15.29:0/4225893708 conn(0x7f0f08065ed0 msgr2=0x7f0f080b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 shutdown_connections 2023-10-16T14:33:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:16.792+0000 7f0f0cb75640 1 -- 172.21.15.29:0/4225893708 wait complete. 2023-10-16T14:33:16.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:33:16.868 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:16.864+0000 7f953e84f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:33:16.899 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:16.895+0000 7f88a4613040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:33:16.920 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:16.917+0000 7f953e84f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:33:16.959 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:16.955+0000 7f88a4613040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:33:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.031+0000 7f41ba2da640 1 Processor -- start 2023-10-16T14:33:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.031+0000 7f41ba2da640 1 -- start start 2023-10-16T14:33:17.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.033+0000 7f41ba2da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 0x7f41b40c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.033+0000 7f41ba2da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40acd70 con 0x7f41b4065ab0 2023-10-16T14:33:17.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.033+0000 7f41ba2da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40ac4a0 con 0x7f41b406d220 2023-10-16T14:33:17.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.033+0000 7f41ba2da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4065f40 con 0x7f41b40ce280 2023-10-16T14:33:17.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.033+0000 7f41b8ad7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41b4065ab0 0x7f41b40ad200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44174/0 (socket says 172.21.15.29:44174) 2023-10-16T14:33:17.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.033+0000 7f41b8ad7640 1 -- 172.21.15.29:0/1574325091 learned_addr learned my addr 172.21.15.29:0/1574325091 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41b9ad9640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 0x7f41b40c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955165726 0 0) 0x7f41b40acd70 con 0x7f41b4065ab0 2023-10-16T14:33:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0003520 con 0x7f41b4065ab0 2023-10-16T14:33:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058669676 0 0) 0x7f41a0003520 con 0x7f41b4065ab0 2023-10-16T14:33:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 msgr2=0x7f41b40c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 0x7f41b40c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 >> v1:172.21.15.170:6789/0 conn(0x7f41b406d220 legacy=0x7f41b40cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40ac720 con 0x7f41b4065ab0 2023-10-16T14:33:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f41a4001670 con 0x7f41b4065ab0 2023-10-16T14:33:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 >> v1:172.21.15.29:6789/0 conn(0x7f41b4065ab0 legacy=0x7f41b40ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a00039a0 0x7f41a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41a00047c0 0x7f41a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 0x7f41a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4065f40 con 0x7f41a00047c0 2023-10-16T14:33:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.034+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a000ba60 con 0x7f41a00039a0 2023-10-16T14:33:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a000bc20 con 0x7f41b40ce280 2023-10-16T14:33:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41a00047c0 0x7f41a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b92d8640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a00039a0 0x7f41a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b8ad7640 1 -- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 msgr2=0x7f41a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b9ad9640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 0x7f41a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41b40ce280 0x7f41a0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b8ad7640 1 -- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a00039a0 msgr2=0x7f41a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a00039a0 0x7f41a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b8ad7640 1 -- 172.21.15.29:0/1574325091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0003520 con 0x7f41a00047c0 2023-10-16T14:33:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.035+0000 7f41b92d8640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41a00039a0 0x7f41a0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.036+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41a00047c0 0x7f41a0004c40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007fd0 tx=0x7f41a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e472c6ce44ec9e03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.036+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a4005ad0 con 0x7f41a00047c0 2023-10-16T14:33:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.036+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a4005c30 con 0x7f41a00047c0 2023-10-16T14:33:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.036+0000 7f41abfff640 1 -- 172.21.15.29:0/1574325091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a402c850 con 0x7f41a00047c0 2023-10-16T14:33:17.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.037+0000 7f41ba2da640 1 -- 172.21.15.29:0/1574325091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41a00047c0 msgr2=0x7f41a0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.037+0000 7f41ba2da640 1 --2- 172.21.15.29:0/1574325091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41a00047c0 0x7f41a0004c40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007fd0 tx=0x7f41a00057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.037+0000 7f41b9ad9640 1 -- 172.21.15.29:0/1574325091 reap_dead start 2023-10-16T14:33:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.037+0000 7f41ba2da640 1 -- 172.21.15.29:0/1574325091 shutdown_connections 2023-10-16T14:33:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.037+0000 7f41ba2da640 1 -- 172.21.15.29:0/1574325091 >> 172.21.15.29:0/1574325091 conn(0x7f41b40b0a80 msgr2=0x7f41b40b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.038+0000 7f41ba2da640 1 -- 172.21.15.29:0/1574325091 shutdown_connections 2023-10-16T14:33:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.038+0000 7f41ba2da640 1 -- 172.21.15.29:0/1574325091 wait complete. 2023-10-16T14:33:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.039+0000 7f41ba2da640 1 Processor -- start 2023-10-16T14:33:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.040+0000 7f41ba2da640 1 -- start start 2023-10-16T14:33:17.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.040+0000 7f41ba2da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41a00047c0 0x7f41b4146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.040+0000 7f41ba2da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 0x7f41b4159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.040+0000 7f41ba2da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b40b1bb0 0x7f41b40b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41ba2da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4159860 con 0x7f41b4158ea0 2023-10-16T14:33:17.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41ba2da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40b2dc0 con 0x7f41b40b1bb0 2023-10-16T14:33:17.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41ba2da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b406a180 con 0x7f41a00047c0 2023-10-16T14:33:17.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 0x7f41b4159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 0x7f41b4159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54966/0 (socket says 172.21.15.29:54966) 2023-10-16T14:33:17.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 -- 172.21.15.29:0/2246923942 learned_addr learned my addr 172.21.15.29:0/2246923942 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:17.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b9ad9640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b40b1bb0 0x7f41b40b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b92d8640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41a00047c0 0x7f41b4146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 -- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41a00047c0 msgr2=0x7f41b4146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41a00047c0 0x7f41b4146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 -- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b40b1bb0 msgr2=0x7f41b40b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b40b1bb0 0x7f41b40b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.041+0000 7f41b8ad7640 1 -- 172.21.15.29:0/2246923942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40b3890 con 0x7f41b4158ea0 2023-10-16T14:33:17.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.042+0000 7f41b8ad7640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 0x7f41b4159320 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c280 tx=0x7f41a402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:17.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.042+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a403c040 con 0x7f41b4158ea0 2023-10-16T14:33:17.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.042+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a402cc60 con 0x7f41b4158ea0 2023-10-16T14:33:17.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.042+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a4007070 con 0x7f41b4158ea0 2023-10-16T14:33:17.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.042+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b40b2570 con 0x7f41b4158ea0 2023-10-16T14:33:17.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.042+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b40b2800 con 0x7f41b4158ea0 2023-10-16T14:33:17.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.043+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b40b33c0 con 0x7f41b4158ea0 2023-10-16T14:33:17.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.044+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f41a4032070 con 0x7f41b4158ea0 2023-10-16T14:33:17.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.045+0000 7f41a9ffb640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 0x7f419806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.046+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41a40aad30 con 0x7f41b4158ea0 2023-10-16T14:33:17.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.046+0000 7f41b92d8640 1 -- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 msgr2=0x7f419806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:17.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.046+0000 7f41b92d8640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 0x7f419806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:17.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.050+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41b40b33c0 con 0x7f41b4158ea0 2023-10-16T14:33:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.240+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f41b40b33c0 con 0x7f41b4158ea0 2023-10-16T14:33:17.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.245+0000 7f41a9ffb640 1 -- 172.21.15.29:0/2246923942 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f41b40b33c0 con 0x7f41b4158ea0 2023-10-16T14:33:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.247+0000 7f41b92d8640 1 -- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 msgr2=0x7f419806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.247+0000 7f41b92d8640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 0x7f419806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.247+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 msgr2=0x7f419806d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.247+0000 7f41ba2da640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 0x7f419806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.247+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 msgr2=0x7f41b4159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.247+0000 7f41ba2da640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 0x7f41b4159320 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c280 tx=0x7f41a402e070 comp rx=0 tx=0).stop 2023-10-16T14:33:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 shutdown_connections 2023-10-16T14:33:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f419806b010 0x7f419806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4158ea0 0x7f41b4159320 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b40b1bb0 0x7f41b40b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 --2- 172.21.15.29:0/2246923942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41a00047c0 0x7f41b4146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 >> 172.21.15.29:0/2246923942 conn(0x7f41b40b0a80 msgr2=0x7f41b4099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 shutdown_connections 2023-10-16T14:33:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.248+0000 7f41ba2da640 1 -- 172.21.15.29:0/2246923942 wait complete. 2023-10-16T14:33:17.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T14:33:17.365 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.361+0000 7f953e84f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:33:17.416 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.412+0000 7f953e84f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:33:17.440 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.436+0000 7f88a4613040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:33:17.465 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.462+0000 7f953e84f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:33:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.477+0000 7f7a79bae640 1 Processor -- start 2023-10-16T14:33:17.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.477+0000 7f7a79bae640 1 -- start start 2023-10-16T14:33:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.478+0000 7f7a79bae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7a740b8480 0x7f7a740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.478+0000 7f7a79bae640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7415d150 con 0x7f7a7415cd70 2023-10-16T14:33:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.478+0000 7f7a79bae640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a740d1180 con 0x7f7a740d0be0 2023-10-16T14:33:17.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.478+0000 7f7a79bae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a740d1460 con 0x7f7a740b8480 2023-10-16T14:33:17.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.479+0000 7f7a78bac640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7a7415cd70 0x7f7a740d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44184/0 (socket says 172.21.15.29:44184) 2023-10-16T14:33:17.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.479+0000 7f7a78bac640 1 -- 172.21.15.29:0/583849069 learned_addr learned my addr 172.21.15.29:0/583849069 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:17.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.479+0000 7f7a73fff640 1 --2- 172.21.15.29:0/583849069 >> v2:172.21.15.170:3300/0 conn(0x7f7a740b8480 0x7f7a740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.479+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833006602 0 0) 0x7f7a7415d150 con 0x7f7a7415cd70 2023-10-16T14:33:17.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.479+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a64003520 con 0x7f7a7415cd70 2023-10-16T14:33:17.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3642145357 0 0) 0x7f7a64003520 con 0x7f7a7415cd70 2023-10-16T14:33:17.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 >> v2:172.21.15.170:3300/0 conn(0x7f7a740b8480 msgr2=0x7f7a740cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:17.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 --2- 172.21.15.29:0/583849069 >> v2:172.21.15.170:3300/0 conn(0x7f7a740b8480 0x7f7a740cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 >> v1:172.21.15.170:6789/0 conn(0x7f7a740d0be0 legacy=0x7f7a740b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a740d1740 con 0x7f7a7415cd70 2023-10-16T14:33:17.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7a68001670 con 0x7f7a7415cd70 2023-10-16T14:33:17.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 >> v1:172.21.15.29:6789/0 conn(0x7f7a7415cd70 legacy=0x7f7a740d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640039c0 0x7f7a64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a640047c0 0x7f7a64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a740d1460 con 0x7f7a640051d0 2023-10-16T14:33:17.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6400c120 con 0x7f7a640047c0 2023-10-16T14:33:17.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6400c2e0 con 0x7f7a640039c0 2023-10-16T14:33:17.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a793ad640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.480+0000 7f7a78bac640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640039c0 0x7f7a64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a73fff640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a640047c0 0x7f7a64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a793ad640 1 -- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640039c0 msgr2=0x7f7a64003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:17.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a793ad640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a640039c0 0x7f7a64003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a793ad640 1 -- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a640047c0 msgr2=0x7f7a64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:17.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a793ad640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a640047c0 0x7f7a64004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a793ad640 1 -- 172.21.15.29:0/583849069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64003520 con 0x7f7a640051d0 2023-10-16T14:33:17.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.481+0000 7f7a793ad640 1 -- 172.21.15.29:0/583849069 reap_dead start 2023-10-16T14:33:17.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.482+0000 7f7a793ad640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a6400b980 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c001e20 tx=0x7f7a640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c0106533eee8b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:17.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.482+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c003b70 con 0x7f7a640051d0 2023-10-16T14:33:17.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.482+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c003cd0 con 0x7f7a640051d0 2023-10-16T14:33:17.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.482+0000 7f7a737fe640 1 -- 172.21.15.29:0/583849069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c005310 con 0x7f7a640051d0 2023-10-16T14:33:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 -- 172.21.15.29:0/583849069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 msgr2=0x7f7a6400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a6400b980 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c001e20 tx=0x7f7a640057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 -- 172.21.15.29:0/583849069 shutdown_connections 2023-10-16T14:33:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 --2- 172.21.15.29:0/583849069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a6400b980 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 -- 172.21.15.29:0/583849069 >> 172.21.15.29:0/583849069 conn(0x7f7a740cf570 msgr2=0x7f7a740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 -- 172.21.15.29:0/583849069 shutdown_connections 2023-10-16T14:33:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.483+0000 7f7a79bae640 1 -- 172.21.15.29:0/583849069 wait complete. 2023-10-16T14:33:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.484+0000 7f7a79bae640 1 Processor -- start 2023-10-16T14:33:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.485+0000 7f7a79bae640 1 -- start start 2023-10-16T14:33:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a79bae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a741598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a79bae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a74150ca0 0x7f7a7415f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a79bae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a7415fec0 0x7f7a7413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a79bae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a74160330 con 0x7f7a640051d0 2023-10-16T14:33:17.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a79bae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a74158550 con 0x7f7a74150ca0 2023-10-16T14:33:17.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a79bae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a74158830 con 0x7f7a7415fec0 2023-10-16T14:33:17.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a741598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a741598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54992/0 (socket says 172.21.15.29:54992) 2023-10-16T14:33:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 -- 172.21.15.29:0/2875398929 learned_addr learned my addr 172.21.15.29:0/2875398929 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a74150ca0 0x7f7a7415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a7415fec0 msgr2=0x7f7a7413fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a7415fec0 0x7f7a7413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a74150ca0 msgr2=0x7f7a7415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a74150ca0 0x7f7a7415f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.486+0000 7f7a78bac640 1 -- 172.21.15.29:0/2875398929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74158b10 con 0x7f7a640051d0 2023-10-16T14:33:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.487+0000 7f7a78bac640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a640051d0 0x7f7a741598b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7a68001200 tx=0x7f7a6800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.487+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800c6c0 con 0x7f7a640051d0 2023-10-16T14:33:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.487+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6800c840 con 0x7f7a640051d0 2023-10-16T14:33:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.487+0000 7f7a79bae640 1 -- 172.21.15.29:0/2875398929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a74158df0 con 0x7f7a640051d0 2023-10-16T14:33:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.487+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800cbf0 con 0x7f7a640051d0 2023-10-16T14:33:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.487+0000 7f7a79bae640 1 -- 172.21.15.29:0/2875398929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a741593b0 con 0x7f7a640051d0 2023-10-16T14:33:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.488+0000 7f7a79bae640 1 -- 172.21.15.29:0/2875398929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a74159570 con 0x7f7a640051d0 2023-10-16T14:33:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.489+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7a6800cd50 con 0x7f7a640051d0 2023-10-16T14:33:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.490+0000 7f7a717fa640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 0x7f7a5806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.490+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a68089740 con 0x7f7a640051d0 2023-10-16T14:33:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.490+0000 7f7a73fff640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 msgr2=0x7f7a5806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:17.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.490+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 0x7f7a5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:17.501 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.493+0000 7f88a4613040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:33:17.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.494+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a68054640 con 0x7f7a640051d0 2023-10-16T14:33:17.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.523+0000 7f953e84f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:33:17.550 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.546+0000 7f88a4613040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:33:17.575 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.572+0000 7f953e84f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:33:17.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.610+0000 7f88a4613040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:33:17.665 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.661+0000 7f88a4613040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:33:17.684 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.681+0000 7f953e84f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:33:17.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.691+0000 7f7a73fff640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 msgr2=0x7f7a5806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:17.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.691+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 0x7f7a5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:17.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:17.710+0000 7f7a79bae640 1 -- 172.21.15.29:0/2875398929 --> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a740d1d00 con 0x7f7a5806afe0 2023-10-16T14:33:17.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:33:17.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:33:17.747 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.744+0000 7f953e84f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:17.782 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.778+0000 7f88a4613040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:33:17.799 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.796+0000 7f953e84f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:33:17.849 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.845+0000 7f88a4613040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:17.859 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.856+0000 7f953e84f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:33:17.900 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.896+0000 7f88a4613040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:33:17.910 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:17.906+0000 7f953e84f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:33:17.959 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:17.955+0000 7f88a4613040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:33:18.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.008+0000 7f88a4613040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:33:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:18.092+0000 7f7a73fff640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 msgr2=0x7f7a5806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:18.092+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 0x7f7a5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:33:18.287 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.284+0000 7f953e84f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:33:18.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.405+0000 7f88a4613040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:33:18.446 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.442+0000 7f953e84f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:33:18.553 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.550+0000 7f953e84f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:33:18.575 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.572+0000 7f88a4613040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:33:18.602 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.599+0000 7f953e84f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:33:18.689 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.685+0000 7f88a4613040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:33:18.710 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.707+0000 7f953e84f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:18.741 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.737+0000 7f88a4613040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:33:18.816 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.813+0000 7f953e84f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:33:18.853 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.850+0000 7f88a4613040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:18.893+0000 7f7a73fff640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 msgr2=0x7f7a5806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/610139797 2023-10-16T14:33:18.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:18.893+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 0x7f7a5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:33:18.965 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:18.962+0000 7f88a4613040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:33:18.973 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:18.969+0000 7f953e84f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:19.022 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:19.018+0000 7f953e84f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:19.129 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:19.125+0000 7f88a4613040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:19.140 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:19.136+0000 7f953e84f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:33:19.180 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:19.177+0000 7f88a4613040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:19.305 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:19.301+0000 7f88a4613040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:33:19.972 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:19.968+0000 7f953e84f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:33:20.172 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:20.168+0000 7f88a4613040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:33:20.266 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.262+0000 7f953e84f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:33:20.324 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.320+0000 7f953e84f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:33:20.440 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.437+0000 7f953e84f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:33:20.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:20.458+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 39) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f7a68054a10 con 0x7f7a640051d0 2023-10-16T14:33:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:20.458+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 msgr2=0x7f7a5806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:20.458+0000 7f7a717fa640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/610139797,v1:172.21.15.170:6833/610139797] conn(0x7f7a5806afe0 0x7f7a5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:20.490 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:20.487+0000 7f88a4613040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:33:20.556 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:20.553+0000 7f88a4613040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:33:20.602 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.599+0000 7f94bb3e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.603 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.599+0000 7f94bb3e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.603 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.599+0000 7f94bb3e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.603 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.599+0000 7f94bb3e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.604 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.599+0000 7f94bb3e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.604 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.600+0000 7f94bdc28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.605 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.600+0000 7f94bdc28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.605 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.600+0000 7f94bdc28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.605 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.600+0000 7f94bdc28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.605 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:20.600+0000 7f94bdc28640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:20.682 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:20.678+0000 7f88a4613040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:33:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.467+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 40) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7a680577f0 con 0x7f7a640051d0 2023-10-16T14:33:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.468+0000 7f7a717fa640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f7a58035880 0x7f7a58074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.468+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 --> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a58035c60 con 0x7f7a58035880 2023-10-16T14:33:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.478+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f7a58035880 0x7f7a58074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.491+0000 7f7a73fff640 1 --2- 172.21.15.29:0/2875398929 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f7a58035880 0x7f7a58074ea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7a741510e0 tx=0x7f7a60008040 comp rx=0 tx=0).ready entity=mgr.6019 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:21.515 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:21.511+0000 7f94c5437640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T14:33:21.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.511+0000 7f7a717fa640 1 -- 172.21.15.29:0/2875398929 <== mgr.6019 v2:172.21.15.170:6832/3288786257 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f7a58035c60 con 0x7f7a58035880 2023-10-16T14:33:21.516 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T14:33:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:33:21.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:33:21.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.811+0000 7f39ae00d640 1 Processor -- start 2023-10-16T14:33:21.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.812+0000 7f39ae00d640 1 -- start start 2023-10-16T14:33:21.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ae00d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f39a815cba0 0x7f39a8071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ae00d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f39a80cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ae00d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a806bb60 con 0x7f39a80ccdc0 2023-10-16T14:33:21.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ae00d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a80716b0 con 0x7f39a815cba0 2023-10-16T14:33:21.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ae00d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80d09e0 con 0x7f39a80cdf50 2023-10-16T14:33:21.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ac80a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f39a80ccdc0 0x7f39a80cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44200/0 (socket says 172.21.15.29:44200) 2023-10-16T14:33:21.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.813+0000 7f39ac80a640 1 -- 172.21.15.29:0/3968598798 learned_addr learned my addr 172.21.15.29:0/3968598798 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f39ad00b640 1 --2- 172.21.15.29:0/3968598798 >> v2:172.21.15.170:3300/0 conn(0x7f39a815cba0 0x7f39a8071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f39ad80c640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f39a80cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729254577 0 0) 0x7f39a806bb60 con 0x7f39a80ccdc0 2023-10-16T14:33:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994003520 con 0x7f39a80ccdc0 2023-10-16T14:33:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592030104 0 0) 0x7f3994003520 con 0x7f39a80ccdc0 2023-10-16T14:33:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 msgr2=0x7f39a80cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f39a80cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 >> v2:172.21.15.170:3300/0 conn(0x7f39a815cba0 msgr2=0x7f39a8071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:21.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 --2- 172.21.15.29:0/3968598798 >> v2:172.21.15.170:3300/0 conn(0x7f39a815cba0 0x7f39a8071bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a806af50 con 0x7f39a80ccdc0 2023-10-16T14:33:21.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3998001670 con 0x7f39a80ccdc0 2023-10-16T14:33:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 >> v1:172.21.15.29:6789/0 conn(0x7f39a80ccdc0 legacy=0x7f39a80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f3994003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994003f40 0x7f39940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3994004d00 0x7f3994005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80716b0 con 0x7f3994004d00 2023-10-16T14:33:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80d09e0 con 0x7f3994003f40 2023-10-16T14:33:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.814+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399400b980 con 0x7f39a80cdf50 2023-10-16T14:33:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad80c640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3994004d00 0x7f3994005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad00b640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f3994003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ac80a640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994003f40 0x7f39940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad80c640 1 -- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 msgr2=0x7f3994003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad80c640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f3994003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad80c640 1 -- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994003f40 msgr2=0x7f39940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad80c640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994003f40 0x7f39940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad80c640 1 -- 172.21.15.29:0/3968598798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994003520 con 0x7f3994004d00 2023-10-16T14:33:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ad00b640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a80cdf50 0x7f3994003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.815+0000 7f39ac80a640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994003f40 0x7f39940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.816+0000 7f39ad80c640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3994004d00 0x7f3994005180 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009fc0 tx=0x7f39940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe09a0ed5c25e3de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a002cbd0 con 0x7f3994004d00 2023-10-16T14:33:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000f040 con 0x7f3994004d00 2023-10-16T14:33:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f399ffff640 1 -- 172.21.15.29:0/3968598798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0034440 con 0x7f3994004d00 2023-10-16T14:33:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f39ae00d640 1 -- 172.21.15.29:0/3968598798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3994004d00 msgr2=0x7f3994005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f39ae00d640 1 --2- 172.21.15.29:0/3968598798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3994004d00 0x7f3994005180 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009fc0 tx=0x7f39940057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f39ad80c640 1 -- 172.21.15.29:0/3968598798 reap_dead start 2023-10-16T14:33:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f39ae00d640 1 -- 172.21.15.29:0/3968598798 shutdown_connections 2023-10-16T14:33:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.817+0000 7f39ae00d640 1 -- 172.21.15.29:0/3968598798 >> 172.21.15.29:0/3968598798 conn(0x7f39a8065ed0 msgr2=0x7f39a80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.818+0000 7f39ae00d640 1 -- 172.21.15.29:0/3968598798 shutdown_connections 2023-10-16T14:33:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.818+0000 7f39ae00d640 1 -- 172.21.15.29:0/3968598798 wait complete. 2023-10-16T14:33:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.819+0000 7f39ae00d640 1 Processor -- start 2023-10-16T14:33:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.820+0000 7f39ae00d640 1 -- start start 2023-10-16T14:33:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ae00d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994004d00 0x7f39a815a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ae00d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 0x7f39a80b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ae00d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a815adf0 0x7f39a80b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ae00d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a81f2370 con 0x7f39a81e5fa0 2023-10-16T14:33:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ae00d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a81e6430 con 0x7f3994004d00 2023-10-16T14:33:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ae00d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a81e6710 con 0x7f39a815adf0 2023-10-16T14:33:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ac80a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 0x7f39a80b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ac80a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 0x7f39a80b5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55004/0 (socket says 172.21.15.29:55004) 2023-10-16T14:33:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ac80a640 1 -- 172.21.15.29:0/292861653 learned_addr learned my addr 172.21.15.29:0/292861653 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ad80c640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a815adf0 0x7f39a80b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.821+0000 7f39ad00b640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994004d00 0x7f39a815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ac80a640 1 -- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a815adf0 msgr2=0x7f39a80b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ac80a640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a815adf0 0x7f39a80b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ac80a640 1 -- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994004d00 msgr2=0x7f39a815a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ac80a640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994004d00 0x7f39a815a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ac80a640 1 -- 172.21.15.29:0/292861653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80add60 con 0x7f39a81e5fa0 2023-10-16T14:33:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ad00b640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994004d00 0x7f39a815a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ac80a640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 0x7f39a80b5260 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f399800d8e0 tx=0x7f399800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39980074e0 con 0x7f39a81e5fa0 2023-10-16T14:33:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998007640 con 0x7f39a81e5fa0 2023-10-16T14:33:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998004a20 con 0x7f39a81e5fa0 2023-10-16T14:33:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a80ae040 con 0x7f39a81e5fa0 2023-10-16T14:33:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.822+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a81e69f0 con 0x7f39a81e5fa0 2023-10-16T14:33:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.823+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a804e9b0 con 0x7f39a81e5fa0 2023-10-16T14:33:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.824+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3998004b80 con 0x7f39a81e5fa0 2023-10-16T14:33:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.825+0000 7f399dffb640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f398c06ad10 0x7f398c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:21.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.825+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3998088ea0 con 0x7f39a81e5fa0 2023-10-16T14:33:21.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.826+0000 7f39ad00b640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f398c06ad10 0x7f398c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:21.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.827+0000 7f39ad00b640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f398c06ad10 0x7f398c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009b20 tx=0x7f39a4008040 comp rx=0 tx=0).ready entity=mgr.6019 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:21.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:21.830+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3998054060 con 0x7f39a81e5fa0 2023-10-16T14:33:22.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.073+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f39a80b6860 con 0x7f39a81e5fa0 2023-10-16T14:33:22.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.466+0000 7f399dffb640 1 -- 172.21.15.29:0/292861653 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v41) v1 ==== 93+0+0 (secure 0 0 0) 0x7f39980190a0 con 0x7f39a81e5fa0 2023-10-16T14:33:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f398c06ad10 msgr2=0x7f398c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f398c06ad10 0x7f398c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009b20 tx=0x7f39a4008040 comp rx=0 tx=0).stop 2023-10-16T14:33:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 msgr2=0x7f39a80b5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 0x7f39a80b5260 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f399800d8e0 tx=0x7f399800dda0 comp rx=0 tx=0).stop 2023-10-16T14:33:22.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 shutdown_connections 2023-10-16T14:33:22.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f398c06ad10 0x7f398c06d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:22.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39a81e5fa0 0x7f39a80b5260 secure :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f399800d8e0 tx=0x7f399800dda0 comp rx=0 tx=0).stop 2023-10-16T14:33:22.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39a815adf0 0x7f39a80b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:22.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 --2- 172.21.15.29:0/292861653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3994004d00 0x7f39a815a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:22.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.470+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 >> 172.21.15.29:0/292861653 conn(0x7f39a8065ed0 msgr2=0x7f39a80afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:22.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.471+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 shutdown_connections 2023-10-16T14:33:22.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:22.471+0000 7f39ae00d640 1 -- 172.21.15.29:0/292861653 wait complete. 2023-10-16T14:33:22.497 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:33:22.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:33:22.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:33:22.719 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:33:22.719 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:22.720 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:22.775 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:22.776 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:22.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:22.995 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:22.991+0000 7ffa04413040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:33:23.041 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.037+0000 7f038083e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:33:23.047 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.043+0000 7ffa04413040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:33:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.087+0000 7f30fee90640 1 Processor -- start 2023-10-16T14:33:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.087+0000 7f30fee90640 1 -- start start 2023-10-16T14:33:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.089+0000 7f30fee90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30f8159b20 0x7f30f806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.089+0000 7f30fee90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f806f7f0 con 0x7f30f8159f00 2023-10-16T14:33:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.089+0000 7f30fee90640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f806c5e0 con 0x7f30f80b0b20 2023-10-16T14:33:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.089+0000 7f30fee90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f806c8c0 con 0x7f30f8159b20 2023-10-16T14:33:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30f7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30f8159f00 0x7f30f80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44226/0 (socket says 172.21.15.29:44226) 2023-10-16T14:33:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30f7fff640 1 -- 172.21.15.29:0/2562378753 learned_addr learned my addr 172.21.15.29:0/2562378753 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30fcc2c640 1 --2- 172.21.15.29:0/2562378753 >> v2:172.21.15.29:3300/0 conn(0x7f30f8159b20 0x7f30f806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2562378753 >> v1:172.21.15.170:6789/0 conn(0x7f30f80b0b20 legacy=0x7f30f806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2562378753 >> v1:172.21.15.29:6789/0 conn(0x7f30f8159f00 legacy=0x7f30f80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2562378753 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f806cba0 con 0x7f30f8159b20 2023-10-16T14:33:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.090+0000 7f30f77fe640 1 -- 172.21.15.29:0/2562378753 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071441119 0 0) 0x7f30f806f7f0 con 0x7f30f8159f00 2023-10-16T14:33:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.091+0000 7f30fcc2c640 1 --2- 172.21.15.29:0/2562378753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f8159b20 0x7f30f806bbe0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0097a0 tx=0x7f30ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495fd3ce3f16f568 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.092+0000 7f30f77fe640 1 -- 172.21.15.29:0/2562378753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec030610 con 0x7f30f8159b20 2023-10-16T14:33:23.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.092+0000 7f30f77fe640 1 -- 172.21.15.29:0/2562378753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec030770 con 0x7f30f8159b20 2023-10-16T14:33:23.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.092+0000 7f30f77fe640 1 -- 172.21.15.29:0/2562378753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec030b00 con 0x7f30f8159b20 2023-10-16T14:33:23.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 -- 172.21.15.29:0/2562378753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f8159b20 msgr2=0x7f30f806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 --2- 172.21.15.29:0/2562378753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f8159b20 0x7f30f806bbe0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0097a0 tx=0x7f30ec02b960 comp rx=0 tx=0).stop 2023-10-16T14:33:23.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 -- 172.21.15.29:0/2562378753 shutdown_connections 2023-10-16T14:33:23.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 --2- 172.21.15.29:0/2562378753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f8159b20 0x7f30f806bbe0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 -- 172.21.15.29:0/2562378753 >> 172.21.15.29:0/2562378753 conn(0x7f30f80cb410 msgr2=0x7f30f80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 -- 172.21.15.29:0/2562378753 shutdown_connections 2023-10-16T14:33:23.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.093+0000 7f30fee90640 1 -- 172.21.15.29:0/2562378753 wait complete. 2023-10-16T14:33:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.096+0000 7f30fee90640 1 Processor -- start 2023-10-16T14:33:23.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.096+0000 7f30fee90640 1 -- start start 2023-10-16T14:33:23.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fee90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 0x7f30f8156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fee90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30f8159b20 0x7f30f815ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fee90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30f8159f00 0x7f30f815d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fee90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f80c6650 con 0x7f30f80b0b20 2023-10-16T14:33:23.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fee90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81f0a20 con 0x7f30f8159b20 2023-10-16T14:33:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fee90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f80c5a60 con 0x7f30f8159f00 2023-10-16T14:33:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fcc2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 0x7f30f8156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fcc2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 0x7f30f8156fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55026/0 (socket says 172.21.15.29:55026) 2023-10-16T14:33:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.098+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2055217473 learned_addr learned my addr 172.21.15.29:0/2055217473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:23.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fd42d640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30f8159f00 0x7f30f815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30f7fff640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30f8159b20 0x7f30f815ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30f8159f00 msgr2=0x7f30f815d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fcc2c640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30f8159f00 0x7f30f815d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30f8159b20 msgr2=0x7f30f815ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fcc2c640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30f8159b20 0x7f30f815ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fcc2c640 1 -- 172.21.15.29:0/2055217473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec009400 con 0x7f30f80b0b20 2023-10-16T14:33:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30f7fff640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30f8159b20 0x7f30f815ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:23.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30fcc2c640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 0x7f30f8156fb0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f30ec02bf70 tx=0x7f30ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:23.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec03e040 con 0x7f30f80b0b20 2023-10-16T14:33:23.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.099+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec039cb0 con 0x7f30f80b0b20 2023-10-16T14:33:23.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.100+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec043680 con 0x7f30f80b0b20 2023-10-16T14:33:23.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.100+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f80c5cb0 con 0x7f30f80b0b20 2023-10-16T14:33:23.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.100+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f80c6270 con 0x7f30f80b0b20 2023-10-16T14:33:23.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.101+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f80c6430 con 0x7f30f80b0b20 2023-10-16T14:33:23.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.102+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f30ec0423e0 con 0x7f30f80b0b20 2023-10-16T14:33:23.108 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.103+0000 7f038083e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:33:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.103+0000 7f30f57fa640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 0x7f30dc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.103+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f30ec0abe80 con 0x7f30f80b0b20 2023-10-16T14:33:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.103+0000 7f30f7fff640 1 -- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 msgr2=0x7f30dc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.103+0000 7f30f7fff640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 0x7f30dc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:23.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.107+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30ec076bb0 con 0x7f30f80b0b20 2023-10-16T14:33:23.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.302+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30f804e160 con 0x7f30f80b0b20 2023-10-16T14:33:23.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.304+0000 7f30f7fff640 1 -- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 msgr2=0x7f30dc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:23.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.304+0000 7f30f7fff640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 0x7f30dc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:23.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.305+0000 7f30f57fa640 1 -- 172.21.15.29:0/2055217473 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17070 (secure 0 0 0) 0x7f30ec07a1d0 con 0x7f30f80b0b20 2023-10-16T14:33:23.309 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:23.309 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:20.443416+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/434807034":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/503917561":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/661241058":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/565742787":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1207268771":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6833/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6832/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/329350515":"2023-10-17T14:33:20.443300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:23.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 msgr2=0x7f30dc06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:23.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 0x7f30dc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 msgr2=0x7f30f8156fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 0x7f30f8156fb0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f30ec02bf70 tx=0x7f30ec004730 comp rx=0 tx=0).stop 2023-10-16T14:33:23.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 shutdown_connections 2023-10-16T14:33:23.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f30dc06afc0 0x7f30dc06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30f8159f00 0x7f30f815d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30f8159b20 0x7f30f815ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 --2- 172.21.15.29:0/2055217473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30f80b0b20 0x7f30f8156fb0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 >> 172.21.15.29:0/2055217473 conn(0x7f30f80cb410 msgr2=0x7f30f80aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:23.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 shutdown_connections 2023-10-16T14:33:23.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.308+0000 7f30fee90640 1 -- 172.21.15.29:0/2055217473 wait complete. 2023-10-16T14:33:23.329 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:33:03.693665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:33:04.756815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:33:23.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:33:23.492 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.488+0000 7ffa04413040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:33:23.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.529+0000 7f0c0ffff640 1 Processor -- start 2023-10-16T14:33:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.529+0000 7f0c0ffff640 1 -- start start 2023-10-16T14:33:23.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0c100b2080 0x7f0c100d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 0x7f0c100b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100d0970 con 0x7f0c100b1ca0 2023-10-16T14:33:23.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c100d0ea0 con 0x7f0c100b2080 2023-10-16T14:33:23.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c100d0bc0 con 0x7f0c100cfc40 2023-10-16T14:33:23.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 0x7f0c100b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0e7fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0c100b2080 0x7f0c100d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 0x7f0c100b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:58826/0 (socket says 172.21.15.29:58826) 2023-10-16T14:33:23.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.531+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/3129606044 learned_addr learned my addr 172.21.15.29:0/3129606044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:23.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.532+0000 7f0c0dffb640 1 -- 172.21.15.29:0/3129606044 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680953469 0 0) 0x7f0c100d0970 con 0x7f0c100b1ca0 2023-10-16T14:33:23.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.532+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/3129606044 >> v2:172.21.15.170:3300/0 conn(0x7f0c100b2080 msgr2=0x7f0c100d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:23.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.532+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/3129606044 >> v2:172.21.15.170:3300/0 conn(0x7f0c100b2080 0x7f0c100d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.532+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/3129606044 >> v1:172.21.15.170:6789/0 conn(0x7f0c100b1ca0 legacy=0x7f0c100ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.532+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/3129606044 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c100cd940 con 0x7f0c100cfc40 2023-10-16T14:33:23.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.533+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/3129606044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 0x7f0c100b24c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c0802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21dc3a877b9ce7b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:23.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0dffb640 1 -- 172.21.15.29:0/3129606044 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0802cb90 con 0x7f0c100cfc40 2023-10-16T14:33:23.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0dffb640 1 -- 172.21.15.29:0/3129606044 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0802ccf0 con 0x7f0c100cfc40 2023-10-16T14:33:23.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0dffb640 1 -- 172.21.15.29:0/3129606044 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08034610 con 0x7f0c100cfc40 2023-10-16T14:33:23.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0ffff640 1 -- 172.21.15.29:0/3129606044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 msgr2=0x7f0c100b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/3129606044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 0x7f0c100b24c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c0802bff0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0ffff640 1 -- 172.21.15.29:0/3129606044 shutdown_connections 2023-10-16T14:33:23.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/3129606044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100cfc40 0x7f0c100b24c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/3129606044 >> v2:172.21.15.170:3300/0 conn(0x7f0c100b2080 0x7f0c100d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.534+0000 7f0c0ffff640 1 -- 172.21.15.29:0/3129606044 >> 172.21.15.29:0/3129606044 conn(0x7f0c100d2b90 msgr2=0x7f0c100d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.535+0000 7f0c0ffff640 1 -- 172.21.15.29:0/3129606044 shutdown_connections 2023-10-16T14:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.535+0000 7f0c0ffff640 1 -- 172.21.15.29:0/3129606044 wait complete. 2023-10-16T14:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.536+0000 7f0c0ffff640 1 Processor -- start 2023-10-16T14:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.536+0000 7f0c0ffff640 1 -- start start 2023-10-16T14:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.537+0000 7f0c0ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100b1ca0 0x7f0c10152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.537+0000 7f0c0ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c100b2080 0x7f0c10150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.537+0000 7f0c0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 0x7f0c1014f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.537+0000 7f0c0ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c10150900 con 0x7f0c100cfc40 2023-10-16T14:33:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.537+0000 7f0c0ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c101512d0 con 0x7f0c100b2080 2023-10-16T14:33:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.537+0000 7f0c0ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1014fcf0 con 0x7f0c100b1ca0 2023-10-16T14:33:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 0x7f0c1014f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 0x7f0c1014f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55028/0 (socket says 172.21.15.29:55028) 2023-10-16T14:33:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2403546949 learned_addr learned my addr 172.21.15.29:0/2403546949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0effd640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100b1ca0 0x7f0c10152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0e7fc640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c100b2080 0x7f0c10150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100b1ca0 msgr2=0x7f0c10152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100b1ca0 0x7f0c10152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c100b2080 msgr2=0x7f0c10150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c100b2080 0x7f0c10150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2403546949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08009b30 con 0x7f0c100cfc40 2023-10-16T14:33:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.538+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 0x7f0c1014f4a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0c08036040 tx=0x7f0c08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.539+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08009510 con 0x7f0c100cfc40 2023-10-16T14:33:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.539+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c08040470 con 0x7f0c100cfc40 2023-10-16T14:33:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.539+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1014ffd0 con 0x7f0c100cfc40 2023-10-16T14:33:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.539+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08040800 con 0x7f0c100cfc40 2023-10-16T14:33:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.539+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c10150590 con 0x7f0c100cfc40 2023-10-16T14:33:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.540+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c10150a70 con 0x7f0c100cfc40 2023-10-16T14:33:23.550 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.543+0000 7ffa04413040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:33:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.541+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0c08040960 con 0x7f0c100cfc40 2023-10-16T14:33:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.543+0000 7f0bf77fe640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f0bfc06af90 0x7f0bfc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.543+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0c080abfb0 con 0x7f0c100cfc40 2023-10-16T14:33:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.543+0000 7f0c0effd640 1 -- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f0bfc06af90 msgr2=0x7f0bfc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.543+0000 7f0c0effd640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f0bfc06af90 0x7f0bfc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.547+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c08076de0 con 0x7f0c100cfc40 2023-10-16T14:33:23.588 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.584+0000 7f038083e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:33:23.596 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.593+0000 7ffa04413040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:33:23.643 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.639+0000 7f038083e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:33:23.657 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.653+0000 7ffa04413040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:33:23.695 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.692+0000 7f038083e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:33:23.705 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.702+0000 7ffa04413040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:33:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.737+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c1004e9b0 con 0x7f0c100cfc40 2023-10-16T14:33:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.737+0000 7f0bf77fe640 1 -- 172.21.15.29:0/2403546949 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c0802ce60 con 0x7f0c100cfc40 2023-10-16T14:33:23.741 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:33:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f0bfc06af90 msgr2=0x7f0bfc06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f0bfc06af90 0x7f0bfc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 msgr2=0x7f0c1014f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 0x7f0c1014f4a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0c08036040 tx=0x7f0c08004a40 comp rx=0 tx=0).stop 2023-10-16T14:33:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 shutdown_connections 2023-10-16T14:33:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f0bfc06af90 0x7f0bfc06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c100cfc40 0x7f0c1014f4a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c100b2080 0x7f0c10150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2403546949 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c100b1ca0 0x7f0c10152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 >> 172.21.15.29:0/2403546949 conn(0x7f0c100d2b90 msgr2=0x7f0c100d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 shutdown_connections 2023-10-16T14:33:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.740+0000 7f0c0ffff640 1 -- 172.21.15.29:0/2403546949 wait complete. 2023-10-16T14:33:23.760 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.757+0000 7f038083e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:33:23.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:33:23.813 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.809+0000 7f038083e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:33:23.815 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.811+0000 7ffa04413040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:33:23.877 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.873+0000 7ffa04413040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:23.926 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.922+0000 7ffa04413040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:33:23.932 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.928+0000 7f038083e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:33:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.964+0000 7f635e652640 1 Processor -- start 2023-10-16T14:33:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.964+0000 7f635e652640 1 -- start start 2023-10-16T14:33:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f635e652640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f635815d410 0x7f63580d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f635e652640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63580cd260 0x7f63580cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f635e652640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63580b5d90 con 0x7f635815d030 2023-10-16T14:33:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f635e652640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63580d2b50 con 0x7f635815d410 2023-10-16T14:33:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f635e652640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63580cdbf0 con 0x7f63580cd260 2023-10-16T14:33:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f634ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f635815d410 0x7f63580d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f6357fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f635815d030 0x7f6358161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58010/0 (socket says 172.21.15.29:58010) 2023-10-16T14:33:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f6357fff640 1 -- 172.21.15.29:0/2995929809 learned_addr learned my addr 172.21.15.29:0/2995929809 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.966+0000 7f635cbef640 1 --2- 172.21.15.29:0/2995929809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63580cd260 0x7f63580cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.967+0000 7f634ffff640 1 -- 172.21.15.29:0/2995929809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63580cd260 msgr2=0x7f63580cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.967+0000 7f634ffff640 1 --2- 172.21.15.29:0/2995929809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63580cd260 0x7f63580cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.967+0000 7f634ffff640 1 -- 172.21.15.29:0/2995929809 >> v1:172.21.15.29:6789/0 conn(0x7f635815d030 legacy=0x7f6358161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.967+0000 7f634ffff640 1 -- 172.21.15.29:0/2995929809 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358160fc0 con 0x7f635815d410 2023-10-16T14:33:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.967+0000 7f63577fe640 1 -- 172.21.15.29:0/2995929809 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518121881 0 0) 0x7f63580b5d90 con 0x7f635815d030 2023-10-16T14:33:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.968+0000 7f634ffff640 1 --2- 172.21.15.29:0/2995929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635815d410 0x7f63580d1630 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f63400097a0 tx=0x7f634002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e656071c9d2b69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:23.984 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:23.980+0000 7ffa04413040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:33:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.969+0000 7f63577fe640 1 -- 172.21.15.29:0/2995929809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340030800 con 0x7f635815d410 2023-10-16T14:33:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.969+0000 7f63577fe640 1 -- 172.21.15.29:0/2995929809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6340030960 con 0x7f635815d410 2023-10-16T14:33:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.969+0000 7f63577fe640 1 -- 172.21.15.29:0/2995929809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340030cf0 con 0x7f635815d410 2023-10-16T14:33:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.969+0000 7f635e652640 1 -- 172.21.15.29:0/2995929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635815d410 msgr2=0x7f63580d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.969+0000 7f635e652640 1 --2- 172.21.15.29:0/2995929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635815d410 0x7f63580d1630 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f63400097a0 tx=0x7f634002b940 comp rx=0 tx=0).stop 2023-10-16T14:33:23.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.970+0000 7f635e652640 1 -- 172.21.15.29:0/2995929809 shutdown_connections 2023-10-16T14:33:23.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.970+0000 7f635e652640 1 --2- 172.21.15.29:0/2995929809 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63580cd260 0x7f63580cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.970+0000 7f635e652640 1 --2- 172.21.15.29:0/2995929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f635815d410 0x7f63580d1630 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.970+0000 7f635e652640 1 -- 172.21.15.29:0/2995929809 >> 172.21.15.29:0/2995929809 conn(0x7f63580cf720 msgr2=0x7f63580cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:23.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.970+0000 7f635e652640 1 -- 172.21.15.29:0/2995929809 shutdown_connections 2023-10-16T14:33:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.970+0000 7f635e652640 1 -- 172.21.15.29:0/2995929809 wait complete. 2023-10-16T14:33:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.971+0000 7f635e652640 1 Processor -- start 2023-10-16T14:33:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.972+0000 7f635e652640 1 -- start start 2023-10-16T14:33:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635e652640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 0x7f6358158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635e652640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f635815d030 0x7f63580c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635e652640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635815d410 0x7f63580b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635e652640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63581e75b0 con 0x7f63580cd260 2023-10-16T14:33:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635e652640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635815dc40 con 0x7f635815d410 2023-10-16T14:33:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635e652640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635815df20 con 0x7f635815d030 2023-10-16T14:33:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 0x7f6358158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 0x7f6358158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37776/0 (socket says 172.21.15.29:37776) 2023-10-16T14:33:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 -- 172.21.15.29:0/1287424910 learned_addr learned my addr 172.21.15.29:0/1287424910 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f634ffff640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f635815d030 0x7f63580c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f635cbef640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635815d410 0x7f63580b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 -- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f635815d030 msgr2=0x7f63580c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f635815d030 0x7f63580c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 -- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635815d410 msgr2=0x7f63580b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635815d410 0x7f63580b3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f6357fff640 1 -- 172.21.15.29:0/1287424910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6340009400 con 0x7f63580cd260 2023-10-16T14:33:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.973+0000 7f634ffff640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f635815d030 0x7f63580c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.974+0000 7f6357fff640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 0x7f6358158270 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f634800c4a0 tx=0x7f634800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.974+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348005ce0 con 0x7f63580cd260 2023-10-16T14:33:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.974+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6348005e40 con 0x7f63580cd260 2023-10-16T14:33:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.974+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63480072b0 con 0x7f63580cd260 2023-10-16T14:33:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.974+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635815e200 con 0x7f63580cd260 2023-10-16T14:33:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.974+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63581587b0 con 0x7f63580cd260 2023-10-16T14:33:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.975+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63580cafd0 con 0x7f63580cd260 2023-10-16T14:33:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.976+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6348007410 con 0x7f63580cd260 2023-10-16T14:33:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.978+0000 7f63557fa640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f633806af90 0x7f633806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.978+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6348088e90 con 0x7f63580cd260 2023-10-16T14:33:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.978+0000 7f634ffff640 1 -- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f633806af90 msgr2=0x7f633806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.978+0000 7f634ffff640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f633806af90 0x7f633806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:23.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:23.982+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63580cafd0 con 0x7f63580cd260 2023-10-16T14:33:24.001 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:23.998+0000 7f038083e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:24.034 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.031+0000 7ffa04413040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:33:24.053 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.050+0000 7f038083e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:33:24.113 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.110+0000 7f038083e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:33:24.167 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.163+0000 7f038083e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:33:24.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.171+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63580cafd0 con 0x7f63580cd260 2023-10-16T14:33:24.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.172+0000 7f63557fa640 1 -- 172.21.15.29:0/1287424910 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63580cafd0 con 0x7f63580cd260 2023-10-16T14:33:24.176 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:33:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f633806af90 msgr2=0x7f633806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f633806af90 0x7f633806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 msgr2=0x7f6358158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 0x7f6358158270 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f634800c4a0 tx=0x7f634800c960 comp rx=0 tx=0).stop 2023-10-16T14:33:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 shutdown_connections 2023-10-16T14:33:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f633806af90 0x7f633806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f635815d410 0x7f63580b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f635815d030 0x7f63580c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 --2- 172.21.15.29:0/1287424910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63580cd260 0x7f6358158270 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 >> 172.21.15.29:0/1287424910 conn(0x7f63580cf720 msgr2=0x7f63580b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 shutdown_connections 2023-10-16T14:33:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.175+0000 7f635e652640 1 -- 172.21.15.29:0/1287424910 wait complete. 2023-10-16T14:33:24.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:33:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.409+0000 7f6448b4e640 1 Processor -- start 2023-10-16T14:33:24.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.409+0000 7f6448b4e640 1 -- start start 2023-10-16T14:33:24.414 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.411+0000 7ffa04413040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:33:24.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.411+0000 7f6448b4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f64440b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.411+0000 7f6448b4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f64440710e0 0x7f6444071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.411+0000 7f6448b4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6444071a80 con 0x7f64440abd40 2023-10-16T14:33:24.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.411+0000 7f6448b4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6444159fb0 con 0x7f64440710e0 2023-10-16T14:33:24.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.411+0000 7f6448b4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64440ae890 con 0x7f644406d5a0 2023-10-16T14:33:24.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.412+0000 7f64437fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f64440b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.412+0000 7f6443fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f64440710e0 0x7f6444071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.412+0000 7f64437fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f64440b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59166/0 (socket says 172.21.15.29:59166) 2023-10-16T14:33:24.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.412+0000 7f64437fe640 1 -- 172.21.15.29:0/1180532375 learned_addr learned my addr 172.21.15.29:0/1180532375 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.412+0000 7f6443fff640 1 -- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 msgr2=0x7f64440b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.412+0000 7f6443fff640 1 --2- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f64440b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.413+0000 7f6443fff640 1 -- 172.21.15.29:0/1180532375 >> v1:172.21.15.170:6789/0 conn(0x7f64440abd40 legacy=0x7f64440ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.413+0000 7f6442ffd640 1 -- 172.21.15.29:0/1180532375 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110902664 0 0) 0x7f6444071a80 con 0x7f64440abd40 2023-10-16T14:33:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.413+0000 7f6443fff640 1 -- 172.21.15.29:0/1180532375 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440ac1c0 con 0x7f64440710e0 2023-10-16T14:33:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.413+0000 7f64437fe640 1 --2- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f64440b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:33:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.414+0000 7f6443fff640 1 --2- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440710e0 0x7f6444071540 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f643c009d90 tx=0x7f643c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97829c9ca1e7c1b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:24.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.414+0000 7f6442ffd640 1 -- 172.21.15.29:0/1180532375 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c02cc60 con 0x7f64440710e0 2023-10-16T14:33:24.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6442ffd640 1 -- 172.21.15.29:0/1180532375 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c02cdc0 con 0x7f64440710e0 2023-10-16T14:33:24.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6442ffd640 1 -- 172.21.15.29:0/1180532375 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c034660 con 0x7f64440710e0 2023-10-16T14:33:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 -- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440710e0 msgr2=0x7f6444071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 --2- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440710e0 0x7f6444071540 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f643c009d90 tx=0x7f643c02beb0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 -- 172.21.15.29:0/1180532375 shutdown_connections 2023-10-16T14:33:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 --2- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440710e0 0x7f6444071540 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 --2- 172.21.15.29:0/1180532375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f64440b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 -- 172.21.15.29:0/1180532375 >> 172.21.15.29:0/1180532375 conn(0x7f64440b0a80 msgr2=0x7f64440b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 -- 172.21.15.29:0/1180532375 shutdown_connections 2023-10-16T14:33:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.415+0000 7f6448b4e640 1 -- 172.21.15.29:0/1180532375 wait complete. 2023-10-16T14:33:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.417+0000 7f6448b4e640 1 Processor -- start 2023-10-16T14:33:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.417+0000 7f6448b4e640 1 -- start start 2023-10-16T14:33:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f6448b4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f644415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f6448b4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 0x7f644406c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f6448b4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440abd40 0x7f6444154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f6448b4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644415a260 con 0x7f64440710e0 2023-10-16T14:33:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f6448b4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64440c31b0 con 0x7f64440abd40 2023-10-16T14:33:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f6448b4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644406c6a0 con 0x7f644406d5a0 2023-10-16T14:33:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f643bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 0x7f644406c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f643bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 0x7f644406c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37780/0 (socket says 172.21.15.29:37780) 2023-10-16T14:33:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.418+0000 7f643bfff640 1 -- 172.21.15.29:0/2233191560 learned_addr learned my addr 172.21.15.29:0/2233191560 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f64437fe640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f644415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f6443fff640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440abd40 0x7f6444154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f643bfff640 1 -- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 msgr2=0x7f644415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f643bfff640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f644415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f643bfff640 1 -- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440abd40 msgr2=0x7f6444154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f643bfff640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440abd40 0x7f6444154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f643bfff640 1 -- 172.21.15.29:0/2233191560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64340094d0 con 0x7f64440710e0 2023-10-16T14:33:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f643bfff640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 0x7f644406c160 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f642c00c410 tx=0x7f642c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c0035f0 con 0x7f64440710e0 2023-10-16T14:33:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c003770 con 0x7f64440710e0 2023-10-16T14:33:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.419+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c003b20 con 0x7f64440710e0 2023-10-16T14:33:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.420+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c009a70 con 0x7f64440710e0 2023-10-16T14:33:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.420+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64440c2b60 con 0x7f64440710e0 2023-10-16T14:33:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.421+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64440c3320 con 0x7f64440710e0 2023-10-16T14:33:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.422+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f642c003c80 con 0x7f64440710e0 2023-10-16T14:33:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.422+0000 7f6440ff9640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f642406af90 0x7f642406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.423+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f642c088e80 con 0x7f64440710e0 2023-10-16T14:33:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.423+0000 7f64437fe640 1 -- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f642406af90 msgr2=0x7f642406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.423+0000 7f64437fe640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f642406af90 0x7f642406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.427+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f642c053d50 con 0x7f64440710e0 2023-10-16T14:33:24.577 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.574+0000 7ffa04413040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:33:24.579 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.576+0000 7f038083e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:33:24.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.619+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f644404e9b0 con 0x7f64440710e0 2023-10-16T14:33:24.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.620+0000 7f6440ff9640 1 -- 172.21.15.29:0/2233191560 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f642c017020 con 0x7f64440710e0 2023-10-16T14:33:24.624 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:33:24.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f642406af90 msgr2=0x7f642406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:24.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f642406af90 0x7f642406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 msgr2=0x7f644406c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 0x7f644406c160 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f642c00c410 tx=0x7f642c00c8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 shutdown_connections 2023-10-16T14:33:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f642406af90 0x7f642406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64440710e0 0x7f644406c160 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64440abd40 0x7f6444154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 --2- 172.21.15.29:0/2233191560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f644406d5a0 0x7f644415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.622+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 >> 172.21.15.29:0/2233191560 conn(0x7f64440b0a80 msgr2=0x7f64440ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.623+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 shutdown_connections 2023-10-16T14:33:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.623+0000 7f6448b4e640 1 -- 172.21.15.29:0/2233191560 wait complete. 2023-10-16T14:33:24.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:33:24.684 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.680+0000 7ffa04413040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:33:24.733 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.730+0000 7ffa04413040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:33:24.754 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.750+0000 7f038083e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:33:24.840 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.836+0000 7ffa04413040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.856+0000 7fbfbbdd0640 1 Processor -- start 2023-10-16T14:33:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.857+0000 7fbfbbdd0640 1 -- start start 2023-10-16T14:33:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.858+0000 7fbfbbdd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ce350 0x7fbfb4160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.858+0000 7fbfbbdd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfb40ae260 0x7fbfb406a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.858+0000 7fbfbbdd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40ce800 con 0x7fbfb4160ad0 2023-10-16T14:33:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.858+0000 7fbfbbdd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb40cdec0 con 0x7fbfb40ae260 2023-10-16T14:33:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.858+0000 7fbfbbdd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb41613f0 con 0x7fbfb40ce350 2023-10-16T14:33:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.858+0000 7fbfb9b6c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbfb4160ad0 0x7fbfb40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58012/0 (socket says 172.21.15.29:58012) 2023-10-16T14:33:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfb9b6c640 1 -- 172.21.15.29:0/2335926555 learned_addr learned my addr 172.21.15.29:0/2335926555 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfba36d640 1 --2- 172.21.15.29:0/2335926555 >> v2:172.21.15.29:3300/0 conn(0x7fbfb40ae260 0x7fbfb406a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfb936b640 1 --2- 172.21.15.29:0/2335926555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ce350 0x7fbfb4160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfb8b6a640 1 -- 172.21.15.29:0/2335926555 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464589320 0 0) 0x7fbfb40ce800 con 0x7fbfb4160ad0 2023-10-16T14:33:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfb8b6a640 1 -- 172.21.15.29:0/2335926555 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa0003520 con 0x7fbfb4160ad0 2023-10-16T14:33:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfba36d640 1 -- 172.21.15.29:0/2335926555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ce350 msgr2=0x7fbfb4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfba36d640 1 --2- 172.21.15.29:0/2335926555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ce350 0x7fbfb4160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfba36d640 1 -- 172.21.15.29:0/2335926555 >> v1:172.21.15.29:6789/0 conn(0x7fbfb4160ad0 legacy=0x7fbfb40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.859+0000 7fbfba36d640 1 -- 172.21.15.29:0/2335926555 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40ae740 con 0x7fbfb40ae260 2023-10-16T14:33:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.861+0000 7fbfba36d640 1 --2- 172.21.15.29:0/2335926555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ae260 0x7fbfb406a780 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009d90 tx=0x7fbfac02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703b600c5351e47b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.861+0000 7fbfb8b6a640 1 -- 172.21.15.29:0/2335926555 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac02cc50 con 0x7fbfb40ae260 2023-10-16T14:33:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.861+0000 7fbfb8b6a640 1 -- 172.21.15.29:0/2335926555 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac02cdb0 con 0x7fbfb40ae260 2023-10-16T14:33:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.861+0000 7fbfb8b6a640 1 -- 172.21.15.29:0/2335926555 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac034660 con 0x7fbfb40ae260 2023-10-16T14:33:24.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/2335926555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ae260 msgr2=0x7fbfb406a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/2335926555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ae260 0x7fbfb406a780 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009d90 tx=0x7fbfac02bea0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/2335926555 shutdown_connections 2023-10-16T14:33:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/2335926555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ae260 0x7fbfb406a780 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/2335926555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ce350 0x7fbfb4160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/2335926555 >> 172.21.15.29:0/2335926555 conn(0x7fbfb406bc10 msgr2=0x7fbfb406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/2335926555 shutdown_connections 2023-10-16T14:33:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.862+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/2335926555 wait complete. 2023-10-16T14:33:24.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.864+0000 7fbfbbdd0640 1 Processor -- start 2023-10-16T14:33:24.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.864+0000 7fbfbbdd0640 1 -- start start 2023-10-16T14:33:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfbbdd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ae260 0x7fbfb41f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfbbdd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 0x7fbfb4156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfbbdd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb4160ad0 0x7fbfb415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfbbdd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40c33e0 con 0x7fbfb40ce350 2023-10-16T14:33:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfbbdd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb415f840 con 0x7fbfb4160ad0 2023-10-16T14:33:24.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfbbdd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb41f2450 con 0x7fbfb40ae260 2023-10-16T14:33:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfb936b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 0x7fbfb4156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfb936b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 0x7fbfb4156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37800/0 (socket says 172.21.15.29:37800) 2023-10-16T14:33:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfb936b640 1 -- 172.21.15.29:0/1510751231 learned_addr learned my addr 172.21.15.29:0/1510751231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.865+0000 7fbfb9b6c640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ae260 0x7fbfb41f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfba36d640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb4160ad0 0x7fbfb415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfb936b640 1 -- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ae260 msgr2=0x7fbfb41f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfb936b640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ae260 0x7fbfb41f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfb936b640 1 -- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb4160ad0 msgr2=0x7fbfb415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfb936b640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb4160ad0 0x7fbfb415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfb936b640 1 -- 172.21.15.29:0/1510751231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac009a70 con 0x7fbfb40ce350 2023-10-16T14:33:24.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfba36d640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb4160ad0 0x7fbfb415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:24.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfb936b640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 0x7fbfb4156040 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400d800 tx=0x7fbfa400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:24.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400b6c0 con 0x7fbfb40ce350 2023-10-16T14:33:24.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa400b820 con 0x7fbfb40ce350 2023-10-16T14:33:24.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb40b21a0 con 0x7fbfb40ce350 2023-10-16T14:33:24.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.866+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400bbb0 con 0x7fbfb40ce350 2023-10-16T14:33:24.876 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.866+0000 7f038083e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:33:24.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.867+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb41565e0 con 0x7fbfb40ce350 2023-10-16T14:33:24.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.868+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb404e160 con 0x7fbfb40ce350 2023-10-16T14:33:24.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.869+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fbfa400bd10 con 0x7fbfb40ce350 2023-10-16T14:33:24.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.870+0000 7fbfaa7fc640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 0x7fbf9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:24.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.870+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbfa4089eb0 con 0x7fbfb40ce350 2023-10-16T14:33:24.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.870+0000 7fbfb9b6c640 1 -- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 msgr2=0x7fbf9806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:24.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.870+0000 7fbfb9b6c640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 0x7fbf9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:24.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:24.878+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbfa4054c50 con 0x7fbfb40ce350 2023-10-16T14:33:24.927 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:24.923+0000 7f038083e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:33:24.949 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:24.945+0000 7ffa04413040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:33:25.041 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:25.037+0000 7f038083e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.071+0000 7fbfb9b6c640 1 -- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 msgr2=0x7fbf9806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.071+0000 7fbfb9b6c640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 0x7fbf9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:25.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.085+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbfb4065d60 con 0x7fbfb40ce350 2023-10-16T14:33:25.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.086+0000 7fbfaa7fc640 1 -- 172.21.15.29:0/1510751231 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v98) v1 ==== 80+0+471 (secure 0 0 0) 0x7fbfa4058270 con 0x7fbfb40ce350 2023-10-16T14:33:25.090 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/434807034 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.090 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/503917561 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.090 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/661241058 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.090 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/565742787 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.091 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1207268771 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.091 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/610139797 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.091 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/610139797 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.091 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/329350515 2023-10-17T14:33:20.443300+0000 2023-10-16T14:33:25.091 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T14:33:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.088+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 msgr2=0x7fbf9806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.088+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 0x7fbf9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.088+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 msgr2=0x7fbfb4156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.088+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 0x7fbfb4156040 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400d800 tx=0x7fbfa400dcc0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 shutdown_connections 2023-10-16T14:33:25.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fbf9806af90 0x7fbf9806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40ce350 0x7fbfb4156040 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb4160ad0 0x7fbfb415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 --2- 172.21.15.29:0/1510751231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb40ae260 0x7fbfb41f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 >> 172.21.15.29:0/1510751231 conn(0x7fbfb406bc10 msgr2=0x7fbfb40af370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:25.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 shutdown_connections 2023-10-16T14:33:25.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.089+0000 7fbfbbdd0640 1 -- 172.21.15.29:0/1510751231 wait complete. 2023-10-16T14:33:25.105 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:25.101+0000 7ffa04413040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:25.110 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:33:25.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:25.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:25.153 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:25.150+0000 7ffa04413040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:25.161 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:25.157+0000 7f038083e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:33:25.218 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:33:25.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:25.219 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:33:25.272 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:33:25.272 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:25.274 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:25.270+0000 7ffa04413040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:33:25.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:33:25.327 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:25.327 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:25.337 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:25.334+0000 7f038083e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:25.382 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:33:25.383 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:33:25.384 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:25.392 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:25.389+0000 7f038083e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:25.439 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:33:25.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:25.519 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:25.516+0000 7f038083e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:33:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.649+0000 7fe0f3232640 1 Processor -- start 2023-10-16T14:33:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.649+0000 7fe0f3232640 1 -- start start 2023-10-16T14:33:25.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f3232640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe0ec15ae80 0x7fe0ec15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f3232640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0ec0ce390 0x7fe0ec070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f3232640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec15ed60 con 0x7fe0ec15aaa0 2023-10-16T14:33:25.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f3232640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec06b810 con 0x7fe0ec15ae80 2023-10-16T14:33:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f3232640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0707e0 con 0x7fe0ec0ce390 2023-10-16T14:33:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f0fce640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe0ec15aaa0 0x7fe0ec0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58020/0 (socket says 172.21.15.29:58020) 2023-10-16T14:33:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f0fce640 1 -- 172.21.15.29:0/3470558806 learned_addr learned my addr 172.21.15.29:0/3470558806 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0f17cf640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0ec0ce390 0x7fe0ec070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0ebfff640 1 --2- 172.21.15.29:0/3470558806 >> v2:172.21.15.170:3300/0 conn(0x7fe0ec15ae80 0x7fe0ec15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140984830 0 0) 0x7fe0ec15ed60 con 0x7fe0ec15aaa0 2023-10-16T14:33:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.651+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d8003520 con 0x7fe0ec15aaa0 2023-10-16T14:33:25.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594671383 0 0) 0x7fe0d8003520 con 0x7fe0ec15aaa0 2023-10-16T14:33:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0ec0ce390 msgr2=0x7fe0ec070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0ec0ce390 0x7fe0ec070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 >> v2:172.21.15.170:3300/0 conn(0x7fe0ec15ae80 msgr2=0x7fe0ec15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 --2- 172.21.15.29:0/3470558806 >> v2:172.21.15.170:3300/0 conn(0x7fe0ec15ae80 0x7fe0ec15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec15ef90 con 0x7fe0ec15aaa0 2023-10-16T14:33:25.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe0dc001a10 con 0x7fe0ec15aaa0 2023-10-16T14:33:25.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 >> v1:172.21.15.29:6789/0 conn(0x7fe0ec15aaa0 legacy=0x7fe0ec0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:25.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d8003970 0x7fe0d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d80047c0 0x7fe0d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d8005180 0x7fe0d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec06b810 con 0x7fe0d8003970 2023-10-16T14:33:25.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0707e0 con 0x7fe0d8005180 2023-10-16T14:33:25.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d800bef0 con 0x7fe0d80047c0 2023-10-16T14:33:25.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0f17cf640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0ec0ce390 0x7fe0ec070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:25.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.652+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d8003970 0x7fe0d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:25.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.653+0000 7fe0f0fce640 1 -- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d80047c0 msgr2=0x7fe0d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:25.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.653+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d80047c0 0x7fe0d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.653+0000 7fe0f0fce640 1 -- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d8005180 msgr2=0x7fe0d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:25.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.653+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d8005180 0x7fe0d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.653+0000 7fe0f0fce640 1 -- 172.21.15.29:0/3470558806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8003520 con 0x7fe0d8003970 2023-10-16T14:33:25.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.654+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d8003970 0x7fe0d8003d90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00a850 tx=0x7fe0d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e4a5a4eadd215 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:25.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.654+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc034630 con 0x7fe0d8003970 2023-10-16T14:33:25.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.654+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc02f070 con 0x7fe0d8003970 2023-10-16T14:33:25.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.654+0000 7fe0eb7fe640 1 -- 172.21.15.29:0/3470558806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0349d0 con 0x7fe0d8003970 2023-10-16T14:33:25.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f3232640 1 -- 172.21.15.29:0/3470558806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d8003970 msgr2=0x7fe0d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:25.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f3232640 1 --2- 172.21.15.29:0/3470558806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d8003970 0x7fe0d8003d90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00a850 tx=0x7fe0d80057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f17cf640 1 -- 172.21.15.29:0/3470558806 reap_dead start 2023-10-16T14:33:25.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f3232640 1 -- 172.21.15.29:0/3470558806 shutdown_connections 2023-10-16T14:33:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f3232640 1 -- 172.21.15.29:0/3470558806 >> 172.21.15.29:0/3470558806 conn(0x7fe0ec0cb410 msgr2=0x7fe0ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f3232640 1 -- 172.21.15.29:0/3470558806 shutdown_connections 2023-10-16T14:33:25.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.655+0000 7fe0f3232640 1 -- 172.21.15.29:0/3470558806 wait complete. 2023-10-16T14:33:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.657+0000 7fe0f3232640 1 Processor -- start 2023-10-16T14:33:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.657+0000 7fe0f3232640 1 -- start start 2023-10-16T14:33:25.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f3232640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d8003970 0x7fe0ec15de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f3232640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 0x7fe0ec0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f3232640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ec14e010 0x7fe0ec0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f3232640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec15e380 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f3232640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0c32a0 con 0x7fe0ec14e010 2023-10-16T14:33:25.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f3232640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0c3580 con 0x7fe0d8003970 2023-10-16T14:33:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 0x7fe0ec0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 0x7fe0ec0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37828/0 (socket says 172.21.15.29:37828) 2023-10-16T14:33:25.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 -- 172.21.15.29:0/2632740452 learned_addr learned my addr 172.21.15.29:0/2632740452 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f17cf640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ec14e010 0x7fe0ec0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d8003970 0x7fe0ec15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 -- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d8003970 msgr2=0x7fe0ec15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d8003970 0x7fe0ec15de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 -- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ec14e010 msgr2=0x7fe0ec0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ec14e010 0x7fe0ec0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0ebfff640 1 -- 172.21.15.29:0/2632740452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec0c3860 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.658+0000 7fe0f17cf640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ec14e010 0x7fe0ec0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.659+0000 7fe0ebfff640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 0x7fe0ec0ac890 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e7c0 tx=0x7fe0d400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.659+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400eed0 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.659+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400c940 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.659+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400ccf0 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.659+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec0c3b40 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.659+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec15d990 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.660+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec04e160 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.661+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe0d4013470 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.662+0000 7fe0e97fa640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 0x7fe0cc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.662+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe0d4089970 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.662+0000 7fe0f0fce640 1 -- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 msgr2=0x7fe0cc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.662+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 0x7fe0cc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.667+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe0d40547b0 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.863+0000 7fe0f0fce640 1 -- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 msgr2=0x7fe0cc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:25.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.863+0000 7fe0f0fce640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 0x7fe0cc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:25.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.897+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0ec0cdc60 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.898+0000 7fe0e97fa640 1 -- 172.21.15.29:0/2632740452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fe0d4017090 con 0x7fe0ec14dbb0 2023-10-16T14:33:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:25.902 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:05.723125+0000","modified":"2023-10-16T14:33:08.790919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5926":{"gid":5926,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3812375996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3812375996},{"type":"v1","addr":"172.21.15.170:6837","nonce":3812375996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T14:33:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-16T14:33:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 msgr2=0x7fe0cc06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 0x7fe0cc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 msgr2=0x7fe0ec0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 0x7fe0ec0ac890 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e7c0 tx=0x7fe0d400ec80 comp rx=0 tx=0).stop 2023-10-16T14:33:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 shutdown_connections 2023-10-16T14:33:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7fe0cc06afc0 0x7fe0cc06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0ec14dbb0 0x7fe0ec0ac890 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ec14e010 0x7fe0ec0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 --2- 172.21.15.29:0/2632740452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d8003970 0x7fe0ec15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 >> 172.21.15.29:0/2632740452 conn(0x7fe0ec0cb410 msgr2=0x7fe0ec0b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:25.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 shutdown_connections 2023-10-16T14:33:25.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:25.900+0000 7fe0f3232640 1 -- 172.21.15.29:0/2632740452 wait complete. 2023-10-16T14:33:25.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:26.101 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.097+0000 7ffa04413040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:33:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.125+0000 7f9444aec640 1 Processor -- start 2023-10-16T14:33:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.125+0000 7f9444aec640 1 -- start start 2023-10-16T14:33:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.126+0000 7f9444aec640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f944006c3f0 0x7f944006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f9444aec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e0c0 0x7f94400b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f9444aec640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94400cd630 con 0x7f944015e540 2023-10-16T14:33:26.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f9444aec640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94400ce1b0 con 0x7f944006c3f0 2023-10-16T14:33:26.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f9444aec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944006cd10 con 0x7f944015e0c0 2023-10-16T14:33:26.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f943e59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f944006c3f0 0x7f944006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f9437fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e0c0 0x7f94400b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f943e59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f944006c3f0 0x7f944006c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:39640/0 (socket says 172.21.15.29:39640) 2023-10-16T14:33:26.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.127+0000 7f943e59c640 1 -- 172.21.15.29:0/1141967604 learned_addr learned my addr 172.21.15.29:0/1141967604 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:26.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.128+0000 7f943e59c640 1 -- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e0c0 msgr2=0x7f94400b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:26.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.128+0000 7f943e59c640 1 --2- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e0c0 0x7f94400b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.128+0000 7f943e59c640 1 -- 172.21.15.29:0/1141967604 >> v1:172.21.15.170:6789/0 conn(0x7f944015e540 legacy=0x7f94400b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.128+0000 7f943e59c640 1 -- 172.21.15.29:0/1141967604 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94400d2170 con 0x7f944006c3f0 2023-10-16T14:33:26.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.128+0000 7f9437fff640 1 --2- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e0c0 0x7f94400b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:33:26.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.129+0000 7f943e59c640 1 --2- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944006c3f0 0x7f944006c7d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9430009870 tx=0x7f943002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc1daaea0f39579 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:26.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.130+0000 7f943dd9b640 1 -- 172.21.15.29:0/1141967604 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430030840 con 0x7f944006c3f0 2023-10-16T14:33:26.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.130+0000 7f943dd9b640 1 -- 172.21.15.29:0/1141967604 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94300309a0 con 0x7f944006c3f0 2023-10-16T14:33:26.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.130+0000 7f943dd9b640 1 -- 172.21.15.29:0/1141967604 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430030d30 con 0x7f944006c3f0 2023-10-16T14:33:26.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.131+0000 7f9444aec640 1 -- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944006c3f0 msgr2=0x7f944006c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.131+0000 7f9444aec640 1 --2- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944006c3f0 0x7f944006c7d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9430009870 tx=0x7f943002b980 comp rx=0 tx=0).stop 2023-10-16T14:33:26.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.132+0000 7f9444aec640 1 -- 172.21.15.29:0/1141967604 shutdown_connections 2023-10-16T14:33:26.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.132+0000 7f9444aec640 1 --2- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e0c0 0x7f94400b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.132+0000 7f9444aec640 1 --2- 172.21.15.29:0/1141967604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944006c3f0 0x7f944006c7d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.132+0000 7f9444aec640 1 -- 172.21.15.29:0/1141967604 >> 172.21.15.29:0/1141967604 conn(0x7f94400d2cb0 msgr2=0x7f94400d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:26.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.132+0000 7f9444aec640 1 -- 172.21.15.29:0/1141967604 shutdown_connections 2023-10-16T14:33:26.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.132+0000 7f9444aec640 1 -- 172.21.15.29:0/1141967604 wait complete. 2023-10-16T14:33:26.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.135+0000 7f9444aec640 1 Processor -- start 2023-10-16T14:33:26.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.136+0000 7f9444aec640 1 -- start start 2023-10-16T14:33:26.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f9444aec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 0x7f9440155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f9444aec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015e0c0 0x7f9440158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f9444aec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e540 0x7f9440158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f9444aec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94400ccfb0 con 0x7f944006c3f0 2023-10-16T14:33:26.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f9444aec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94401557b0 con 0x7f944015e0c0 2023-10-16T14:33:26.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f9444aec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400cc3a0 con 0x7f944015e540 2023-10-16T14:33:26.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f943e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 0x7f9440155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f943e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 0x7f9440155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37842/0 (socket says 172.21.15.29:37842) 2023-10-16T14:33:26.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.138+0000 7f943e59c640 1 -- 172.21.15.29:0/1815696302 learned_addr learned my addr 172.21.15.29:0/1815696302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:26.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f9437fff640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015e0c0 0x7f9440158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943ed9d640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e540 0x7f9440158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943e59c640 1 -- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e540 msgr2=0x7f9440158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943e59c640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e540 0x7f9440158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943e59c640 1 -- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015e0c0 msgr2=0x7f9440158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943e59c640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015e0c0 0x7f9440158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943e59c640 1 -- 172.21.15.29:0/1815696302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428009400 con 0x7f944006c3f0 2023-10-16T14:33:26.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.139+0000 7f943e59c640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 0x7f9440155270 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f943002b9c0 tx=0x7f9430004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:26.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.140+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430039a90 con 0x7f944006c3f0 2023-10-16T14:33:26.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.140+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430039bf0 con 0x7f944006c3f0 2023-10-16T14:33:26.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.140+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430043410 con 0x7f944006c3f0 2023-10-16T14:33:26.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.140+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94300094d0 con 0x7f944006c3f0 2023-10-16T14:33:26.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.140+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94400cc960 con 0x7f944006c3f0 2023-10-16T14:33:26.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.141+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94400cd860 con 0x7f944006c3f0 2023-10-16T14:33:26.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.142+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f943002f390 con 0x7f944006c3f0 2023-10-16T14:33:26.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.143+0000 7f9436ffd640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.143+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f94300abbb0 con 0x7f944006c3f0 2023-10-16T14:33:26.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.143+0000 7f9437fff640 1 -- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 msgr2=0x7f942006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:26.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.143+0000 7f9437fff640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:26.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.147+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f94400cd860 con 0x7f944006c3f0 2023-10-16T14:33:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.342+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94400cd860 con 0x7f944006c3f0 2023-10-16T14:33:26.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.343+0000 7f9436ffd640 1 -- 172.21.15.29:0/1815696302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17070 (secure 0 0 0) 0x7f94400cd860 con 0x7f944006c3f0 2023-10-16T14:33:26.348 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:26.348 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:20.443416+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/434807034":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/503917561":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/661241058":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/565742787":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1207268771":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6833/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6832/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/329350515":"2023-10-17T14:33:20.443300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:26.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.344+0000 7f9437fff640 1 -- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 msgr2=0x7f942006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3288786257 2023-10-16T14:33:26.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.344+0000 7f9437fff640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:26.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 msgr2=0x7f942006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:26.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 msgr2=0x7f9440155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 0x7f9440155270 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f943002b9c0 tx=0x7f9430004730 comp rx=0 tx=0).stop 2023-10-16T14:33:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 shutdown_connections 2023-10-16T14:33:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:6832/3288786257,v1:172.21.15.170:6833/3288786257] conn(0x7f942006af90 0x7f942006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f944015e540 0x7f9440158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f944015e0c0 0x7f9440158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 --2- 172.21.15.29:0/1815696302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944006c3f0 0x7f9440155270 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 >> 172.21.15.29:0/1815696302 conn(0x7f94400d2cb0 msgr2=0x7f944015ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 shutdown_connections 2023-10-16T14:33:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.347+0000 7f9444aec640 1 -- 172.21.15.29:0/1815696302 wait complete. 2023-10-16T14:33:26.377 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:33:03.693665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:33:04.756815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:33:26.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:33:26.390 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:26.387+0000 7f038083e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:33:26.398 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.394+0000 7ffa04413040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:33:26.455 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.451+0000 7ffa04413040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:33:26.570 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.567+0000 7ffa04413040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:33:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.599+0000 7faf502f7640 1 Processor -- start 2023-10-16T14:33:26.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.599+0000 7faf502f7640 1 -- start start 2023-10-16T14:33:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf502f7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf502f7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf4806faf0 0x7faf4815f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf502f7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4815eff0 con 0x7faf48065ab0 2023-10-16T14:33:26.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf502f7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf480d0930 con 0x7faf4806faf0 2023-10-16T14:33:26.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf502f7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4815ecc0 con 0x7faf4815af80 2023-10-16T14:33:26.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf4d892640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf4806faf0 0x7faf4815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf4d892640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf4806faf0 0x7faf4815f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37858/0 (socket says 172.21.15.29:37858) 2023-10-16T14:33:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf4d892640 1 -- 172.21.15.29:0/3146096850 learned_addr learned my addr 172.21.15.29:0/3146096850 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.601+0000 7faf4e093640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.602+0000 7faf4d892640 1 -- 172.21.15.29:0/3146096850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 msgr2=0x7faf480cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:26.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.602+0000 7faf4d892640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.602+0000 7faf4d892640 1 -- 172.21.15.29:0/3146096850 >> v1:172.21.15.170:6789/0 conn(0x7faf48065ab0 legacy=0x7faf480ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.602+0000 7faf4d892640 1 -- 172.21.15.29:0/3146096850 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4815b360 con 0x7faf4806faf0 2023-10-16T14:33:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.602+0000 7faf4e093640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.603+0000 7faf4d892640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf4806faf0 0x7faf4815f480 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7faf38009870 tx=0x7faf3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2720353ccd1dbc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.603+0000 7faf4d091640 1 -- 172.21.15.29:0/3146096850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf38030840 con 0x7faf4806faf0 2023-10-16T14:33:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.603+0000 7faf4d091640 1 -- 172.21.15.29:0/3146096850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf380309a0 con 0x7faf4806faf0 2023-10-16T14:33:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.603+0000 7faf4d091640 1 -- 172.21.15.29:0/3146096850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf38030d30 con 0x7faf4806faf0 2023-10-16T14:33:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 -- 172.21.15.29:0/3146096850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf4806faf0 msgr2=0x7faf4815f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf4806faf0 0x7faf4815f480 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7faf38009870 tx=0x7faf3802b980 comp rx=0 tx=0).stop 2023-10-16T14:33:26.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 -- 172.21.15.29:0/3146096850 shutdown_connections 2023-10-16T14:33:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf4806faf0 0x7faf4815f480 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 --2- 172.21.15.29:0/3146096850 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 -- 172.21.15.29:0/3146096850 >> 172.21.15.29:0/3146096850 conn(0x7faf480d0ab0 msgr2=0x7faf480d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 -- 172.21.15.29:0/3146096850 shutdown_connections 2023-10-16T14:33:26.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.604+0000 7faf502f7640 1 -- 172.21.15.29:0/3146096850 wait complete. 2023-10-16T14:33:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.606+0000 7faf502f7640 1 Processor -- start 2023-10-16T14:33:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.606+0000 7faf502f7640 1 -- start start 2023-10-16T14:33:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf502f7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 0x7faf4814f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf502f7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf4806faf0 0x7faf480b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf502f7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:26.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf502f7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b4040 con 0x7faf48065ab0 2023-10-16T14:33:26.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf502f7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b6180 con 0x7faf4806faf0 2023-10-16T14:33:26.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf502f7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b5bc0 con 0x7faf4815af80 2023-10-16T14:33:26.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.607+0000 7faf4e093640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 0x7faf4814f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 0x7faf4814f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37870/0 (socket says 172.21.15.29:37870) 2023-10-16T14:33:26.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 -- 172.21.15.29:0/26673666 learned_addr learned my addr 172.21.15.29:0/26673666 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:26.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4d892640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf4806faf0 0x7faf480b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e894640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 -- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 msgr2=0x7faf480b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 -- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf4806faf0 msgr2=0x7faf480b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf4806faf0 0x7faf480b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 -- 172.21.15.29:0/26673666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf380094d0 con 0x7faf48065ab0 2023-10-16T14:33:26.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf4e093640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 0x7faf4814f2c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7faf4400c820 tx=0x7faf4400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:26.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.608+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44007a30 con 0x7faf48065ab0 2023-10-16T14:33:26.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.609+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf44007b90 con 0x7faf48065ab0 2023-10-16T14:33:26.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.609+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4400ce70 con 0x7faf48065ab0 2023-10-16T14:33:26.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.609+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf480b5ea0 con 0x7faf48065ab0 2023-10-16T14:33:26.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.609+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4814f800 con 0x7faf48065ab0 2023-10-16T14:33:26.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.610+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf480b3b30 con 0x7faf48065ab0 2023-10-16T14:33:26.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.611+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 42) v1 ==== 88940+0+0 (secure 0 0 0) 0x7faf440113f0 con 0x7faf48065ab0 2023-10-16T14:33:26.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.612+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faf44089600 con 0x7faf48065ab0 2023-10-16T14:33:26.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.617+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faf480b3b30 con 0x7faf48065ab0 2023-10-16T14:33:26.663 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.659+0000 7ff985c52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.663 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.659+0000 7ff985c52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.659+0000 7ff985c52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.659+0000 7ff985c52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.659+0000 7ff985c52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.660+0000 7ff989459640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.660+0000 7ff989459640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.665 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.660+0000 7ff989459640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.665 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.660+0000 7ff989459640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.665 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:26.660+0000 7ff989459640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:33:26.726 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:26.723+0000 7f038083e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:33:26.786 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:26.782+0000 7f038083e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:33:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.811+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf480b3b30 con 0x7faf48065ab0 2023-10-16T14:33:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.812+0000 7faf3effd640 1 -- 172.21.15.29:0/26673666 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf480b3b30 con 0x7faf48065ab0 2023-10-16T14:33:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:33:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.814+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 msgr2=0x7faf4814f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.814+0000 7faf502f7640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 0x7faf4814f2c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7faf4400c820 tx=0x7faf4400cc20 comp rx=0 tx=0).stop 2023-10-16T14:33:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 shutdown_connections 2023-10-16T14:33:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf4815af80 0x7faf480b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faf4806faf0 0x7faf480b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 --2- 172.21.15.29:0/26673666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf48065ab0 0x7faf4814f2c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 >> 172.21.15.29:0/26673666 conn(0x7faf480d0ab0 msgr2=0x7faf4815cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 shutdown_connections 2023-10-16T14:33:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:26.815+0000 7faf502f7640 1 -- 172.21.15.29:0/26673666 wait complete. 2023-10-16T14:33:26.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:33:26.907 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:26.903+0000 7f038083e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.101+0000 7f6ee694e640 1 Processor -- start 2023-10-16T14:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.101+0000 7f6ee694e640 1 -- start start 2023-10-16T14:33:27.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.102+0000 7f6ee694e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ee015ad70 0x7f6ee0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6ee694e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6ee00c9ca0 0x7f6ee006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6ee694e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00cb030 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6ee694e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00c91c0 con 0x7f6ee015ad70 2023-10-16T14:33:27.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6ee694e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00c94a0 con 0x7f6ee00c9ca0 2023-10-16T14:33:27.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ee015ad70 0x7f6ee0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ee015ad70 0x7f6ee0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37872/0 (socket says 172.21.15.29:37872) 2023-10-16T14:33:27.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 -- 172.21.15.29:0/482742055 learned_addr learned my addr 172.21.15.29:0/482742055 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:27.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edf7fe640 1 --2- 172.21.15.29:0/482742055 >> v2:172.21.15.170:3300/0 conn(0x7f6ee00c9ca0 0x7f6ee006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 -- 172.21.15.29:0/482742055 >> v2:172.21.15.170:3300/0 conn(0x7f6ee00c9ca0 msgr2=0x7f6ee006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:27.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 --2- 172.21.15.29:0/482742055 >> v2:172.21.15.170:3300/0 conn(0x7f6ee00c9ca0 0x7f6ee006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 -- 172.21.15.29:0/482742055 >> v1:172.21.15.29:6789/0 conn(0x7f6ee00ca3b0 legacy=0x7f6ee00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.103+0000 7f6edffff640 1 -- 172.21.15.29:0/482742055 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00c9780 con 0x7f6ee015ad70 2023-10-16T14:33:27.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.105+0000 7f6edffff640 1 --2- 172.21.15.29:0/482742055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee015ad70 0x7f6ee0069da0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00ce870 tx=0x7f6ed002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d522da2272480b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:27.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.105+0000 7f6edeffd640 1 -- 172.21.15.29:0/482742055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed002cb10 con 0x7f6ee015ad70 2023-10-16T14:33:27.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.105+0000 7f6edeffd640 1 -- 172.21.15.29:0/482742055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed002cc70 con 0x7f6ee015ad70 2023-10-16T14:33:27.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.105+0000 7f6edeffd640 1 -- 172.21.15.29:0/482742055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00343f0 con 0x7f6ee015ad70 2023-10-16T14:33:27.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 -- 172.21.15.29:0/482742055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee015ad70 msgr2=0x7f6ee0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 --2- 172.21.15.29:0/482742055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee015ad70 0x7f6ee0069da0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00ce870 tx=0x7f6ed002bf70 comp rx=0 tx=0).stop 2023-10-16T14:33:27.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 -- 172.21.15.29:0/482742055 shutdown_connections 2023-10-16T14:33:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 --2- 172.21.15.29:0/482742055 >> v2:172.21.15.170:3300/0 conn(0x7f6ee00c9ca0 0x7f6ee006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 --2- 172.21.15.29:0/482742055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee015ad70 0x7f6ee0069da0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 -- 172.21.15.29:0/482742055 >> 172.21.15.29:0/482742055 conn(0x7f6ee00ce9b0 msgr2=0x7f6ee00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 -- 172.21.15.29:0/482742055 shutdown_connections 2023-10-16T14:33:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.106+0000 7f6ee694e640 1 -- 172.21.15.29:0/482742055 wait complete. 2023-10-16T14:33:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.108+0000 7f6ee694e640 1 Processor -- start 2023-10-16T14:33:27.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.108+0000 7f6ee694e640 1 -- start start 2023-10-16T14:33:27.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6ee694e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ee00c9ca0 0x7f6ee0149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6ee694e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 0x7f6ee00b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6ee694e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee015ad70 0x7f6ee00b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6ee694e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee015d0c0 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6ee694e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00c88a0 con 0x7f6ee015ad70 2023-10-16T14:33:27.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6ee694e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee015dc20 con 0x7f6ee00c9ca0 2023-10-16T14:33:27.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6edf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 0x7f6ee00b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6edf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 0x7f6ee00b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37880/0 (socket says 172.21.15.29:37880) 2023-10-16T14:33:27.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.109+0000 7f6edf7fe640 1 -- 172.21.15.29:0/536360096 learned_addr learned my addr 172.21.15.29:0/536360096 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6edf7fe640 1 -- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ee00c9ca0 msgr2=0x7f6ee0149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6edf7fe640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ee00c9ca0 0x7f6ee0149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6edf7fe640 1 -- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee015ad70 msgr2=0x7f6ee00b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6ee4eeb640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee015ad70 0x7f6ee00b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6edf7fe640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee015ad70 0x7f6ee00b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6edf7fe640 1 -- 172.21.15.29:0/536360096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0009b30 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6ee4eeb640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee015ad70 0x7f6ee00b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.110+0000 7f6edf7fe640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 0x7f6ee00b15d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800c820 tx=0x7f6ec800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.111+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8004220 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.111+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec80043a0 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.111+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8007a70 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.111+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee015d2f0 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.111+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee015d8b0 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.112+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee015b810 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.113+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 42) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6ec8007bd0 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.114+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6ec80891d0 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.118+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ec8054330 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.312+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6ee004e160 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.313+0000 7f6edcff9640 1 -- 172.21.15.29:0/536360096 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ec8057950 con 0x7f6ee00ca3b0 2023-10-16T14:33:27.317 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:33:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.315+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 msgr2=0x7f6ee00b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.315+0000 7f6ee694e640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 0x7f6ee00b15d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800c820 tx=0x7f6ec800cce0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 shutdown_connections 2023-10-16T14:33:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ee00ca3b0 0x7f6ee00b15d0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ee015ad70 0x7f6ee00b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 --2- 172.21.15.29:0/536360096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ee00c9ca0 0x7f6ee0149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 >> 172.21.15.29:0/536360096 conn(0x7f6ee00ce9b0 msgr2=0x7f6ee00afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 shutdown_connections 2023-10-16T14:33:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.316+0000 7f6ee694e640 1 -- 172.21.15.29:0/536360096 wait complete. 2023-10-16T14:33:27.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:33:27.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.622+0000 7f949754e640 1 Processor -- start 2023-10-16T14:33:27.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.623+0000 7f949754e640 1 -- start start 2023-10-16T14:33:27.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.624+0000 7f949754e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f949015ad70 0x7f9490069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.624+0000 7f949754e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94900caac0 0x7f949006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.624+0000 7f949754e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900cb030 con 0x7f94900c9ca0 2023-10-16T14:33:27.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.624+0000 7f949754e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94900c91c0 con 0x7f94900caac0 2023-10-16T14:33:27.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.624+0000 7f949754e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94900c94a0 con 0x7f949015ad70 2023-10-16T14:33:27.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.624+0000 7f9495aeb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94900caac0 0x7f949006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f9495aeb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94900caac0 0x7f949006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37892/0 (socket says 172.21.15.29:37892) 2023-10-16T14:33:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f9495aeb640 1 -- 172.21.15.29:0/2628732021 learned_addr learned my addr 172.21.15.29:0/2628732021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f94952ea640 1 --2- 172.21.15.29:0/2628732021 >> v2:172.21.15.170:3300/0 conn(0x7f949015ad70 0x7f9490069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f9495aeb640 1 -- 172.21.15.29:0/2628732021 >> v2:172.21.15.170:3300/0 conn(0x7f949015ad70 msgr2=0x7f9490069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:27.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f9495aeb640 1 --2- 172.21.15.29:0/2628732021 >> v2:172.21.15.170:3300/0 conn(0x7f949015ad70 0x7f9490069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f9495aeb640 1 -- 172.21.15.29:0/2628732021 >> v1:172.21.15.29:6789/0 conn(0x7f94900c9ca0 legacy=0x7f94900ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.625+0000 7f9495aeb640 1 -- 172.21.15.29:0/2628732021 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900c9780 con 0x7f94900caac0 2023-10-16T14:33:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.626+0000 7f9495aeb640 1 --2- 172.21.15.29:0/2628732021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94900caac0 0x7f949006a2e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f94900cf780 tx=0x7f948802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127f95fda2a26867 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.627+0000 7f948ffff640 1 -- 172.21.15.29:0/2628732021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488030860 con 0x7f94900caac0 2023-10-16T14:33:27.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.627+0000 7f948ffff640 1 -- 172.21.15.29:0/2628732021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94880309c0 con 0x7f94900caac0 2023-10-16T14:33:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.627+0000 7f948ffff640 1 -- 172.21.15.29:0/2628732021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488030d50 con 0x7f94900caac0 2023-10-16T14:33:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.627+0000 7f949754e640 1 -- 172.21.15.29:0/2628732021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94900caac0 msgr2=0x7f949006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.627+0000 7f949754e640 1 --2- 172.21.15.29:0/2628732021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94900caac0 0x7f949006a2e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f94900cf780 tx=0x7f948802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.628+0000 7f949754e640 1 -- 172.21.15.29:0/2628732021 shutdown_connections 2023-10-16T14:33:27.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.628+0000 7f949754e640 1 --2- 172.21.15.29:0/2628732021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94900caac0 0x7f949006a2e0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.628+0000 7f949754e640 1 --2- 172.21.15.29:0/2628732021 >> v2:172.21.15.170:3300/0 conn(0x7f949015ad70 0x7f9490069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.628+0000 7f949754e640 1 -- 172.21.15.29:0/2628732021 >> 172.21.15.29:0/2628732021 conn(0x7f94900ce9b0 msgr2=0x7f94900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:27.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.628+0000 7f949754e640 1 -- 172.21.15.29:0/2628732021 shutdown_connections 2023-10-16T14:33:27.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.628+0000 7f949754e640 1 -- 172.21.15.29:0/2628732021 wait complete. 2023-10-16T14:33:27.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.630+0000 7f949754e640 1 Processor -- start 2023-10-16T14:33:27.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.630+0000 7f949754e640 1 -- start start 2023-10-16T14:33:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f949754e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94900c9ca0 0x7f94901535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f949754e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f94900caac0 0x7f949015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f949754e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 0x7f94900af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f949754e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94901f26f0 con 0x7f949015ad70 2023-10-16T14:33:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f949754e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900c6ce0 con 0x7f94900c9ca0 2023-10-16T14:33:27.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f949754e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949015cc20 con 0x7f94900caac0 2023-10-16T14:33:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 0x7f94900af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 0x7f94900af8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37900/0 (socket says 172.21.15.29:37900) 2023-10-16T14:33:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 -- 172.21.15.29:0/768855030 learned_addr learned my addr 172.21.15.29:0/768855030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f94952ea640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94900c9ca0 0x7f94901535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9494ae9640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f94900caac0 0x7f949015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 -- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f94900caac0 msgr2=0x7f949015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f94900caac0 0x7f949015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 -- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94900c9ca0 msgr2=0x7f94901535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94900c9ca0 0x7f94901535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.632+0000 7f9495aeb640 1 -- 172.21.15.29:0/768855030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94880094d0 con 0x7f949015ad70 2023-10-16T14:33:27.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.633+0000 7f9495aeb640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 0x7f94900af8a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f948802b9e0 tx=0x7f9488004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:27.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.633+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488039c90 con 0x7f949015ad70 2023-10-16T14:33:27.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.633+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9488039df0 con 0x7f949015ad70 2023-10-16T14:33:27.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.633+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488043640 con 0x7f949015ad70 2023-10-16T14:33:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.633+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94901f1820 con 0x7f949015ad70 2023-10-16T14:33:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.633+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94901f1d50 con 0x7f949015ad70 2023-10-16T14:33:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.634+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949004e160 con 0x7f949015ad70 2023-10-16T14:33:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.636+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f94880437a0 con 0x7f949015ad70 2023-10-16T14:33:27.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.636+0000 7f948dffb640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f947c06ad10 0x7f947c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.637+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f948803e070 con 0x7f949015ad70 2023-10-16T14:33:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.637+0000 7f94952ea640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f947c06ad10 0x7f947c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:27.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.638+0000 7f94952ea640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f947c06ad10 0x7f947c06d1d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94800073f0 tx=0x7f948000b040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:27.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.641+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9488077730 con 0x7f949015ad70 2023-10-16T14:33:27.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.835+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f949006cdd0 con 0x7f949015ad70 2023-10-16T14:33:27.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.836+0000 7f948dffb640 1 -- 172.21.15.29:0/768855030 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f948807ad50 con 0x7f949015ad70 2023-10-16T14:33:27.839 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:33:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f947c06ad10 msgr2=0x7f947c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f947c06ad10 0x7f947c06d1d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94800073f0 tx=0x7f948000b040 comp rx=0 tx=0).stop 2023-10-16T14:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 msgr2=0x7f94900af8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 0x7f94900af8a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f948802b9e0 tx=0x7f9488004730 comp rx=0 tx=0).stop 2023-10-16T14:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 shutdown_connections 2023-10-16T14:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f947c06ad10 0x7f947c06d1d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f949015ad70 0x7f94900af8a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f94900caac0 0x7f949015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.839+0000 7f949754e640 1 --2- 172.21.15.29:0/768855030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94900c9ca0 0x7f94901535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.840+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 >> 172.21.15.29:0/768855030 conn(0x7f94900ce9b0 msgr2=0x7f94900cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.840+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 shutdown_connections 2023-10-16T14:33:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:27.840+0000 7f949754e640 1 -- 172.21.15.29:0/768855030 wait complete. 2023-10-16T14:33:27.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.081+0000 7f6f61575640 1 Processor -- start 2023-10-16T14:33:28.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.082+0000 7f6f61575640 1 -- start start 2023-10-16T14:33:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.083+0000 7f6f61575640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f5c0ca0a0 0x7f6f5c06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.083+0000 7f6f61575640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5c15d910 con 0x7f6f5c159950 2023-10-16T14:33:28.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.083+0000 7f6f61575640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5c15dbf0 con 0x7f6f5c070b60 2023-10-16T14:33:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.083+0000 7f6f61575640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5c15e490 con 0x7f6f5c0ca0a0 2023-10-16T14:33:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.083+0000 7f6f60d74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f5c0ca0a0 0x7f6f5c06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.084+0000 7f6f5bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6f5c159950 0x7f6f5c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58048/0 (socket says 172.21.15.29:58048) 2023-10-16T14:33:28.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.084+0000 7f6f5bfff640 1 -- 172.21.15.29:0/2734025661 learned_addr learned my addr 172.21.15.29:0/2734025661 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.084+0000 7f6f60d74640 1 -- 172.21.15.29:0/2734025661 >> v1:172.21.15.170:6789/0 conn(0x7f6f5c070b60 legacy=0x7f6f5c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.084+0000 7f6f60d74640 1 -- 172.21.15.29:0/2734025661 >> v1:172.21.15.29:6789/0 conn(0x7f6f5c159950 legacy=0x7f6f5c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.084+0000 7f6f5b7fe640 1 -- 172.21.15.29:0/2734025661 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792673723 0 0) 0x7f6f5c15d910 con 0x7f6f5c159950 2023-10-16T14:33:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.084+0000 7f6f60d74640 1 -- 172.21.15.29:0/2734025661 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c15ded0 con 0x7f6f5c0ca0a0 2023-10-16T14:33:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.085+0000 7f6f60d74640 1 --2- 172.21.15.29:0/2734025661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c06c810 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009d90 tx=0x7f6f5402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c755e01f90864490 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f5b7fe640 1 -- 172.21.15.29:0/2734025661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5402cbd0 con 0x7f6f5c0ca0a0 2023-10-16T14:33:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f5b7fe640 1 -- 172.21.15.29:0/2734025661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5402cd30 con 0x7f6f5c0ca0a0 2023-10-16T14:33:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f5b7fe640 1 -- 172.21.15.29:0/2734025661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5402cea0 con 0x7f6f5c0ca0a0 2023-10-16T14:33:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 -- 172.21.15.29:0/2734025661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c0ca0a0 msgr2=0x7f6f5c06c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 --2- 172.21.15.29:0/2734025661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c06c810 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009d90 tx=0x7f6f5402be20 comp rx=0 tx=0).stop 2023-10-16T14:33:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 -- 172.21.15.29:0/2734025661 shutdown_connections 2023-10-16T14:33:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 --2- 172.21.15.29:0/2734025661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c06c810 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 -- 172.21.15.29:0/2734025661 >> 172.21.15.29:0/2734025661 conn(0x7f6f5c0ce6a0 msgr2=0x7f6f5c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 -- 172.21.15.29:0/2734025661 shutdown_connections 2023-10-16T14:33:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.086+0000 7f6f61575640 1 -- 172.21.15.29:0/2734025661 wait complete. 2023-10-16T14:33:28.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.088+0000 7f6f61575640 1 Processor -- start 2023-10-16T14:33:28.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.088+0000 7f6f61575640 1 -- start start 2023-10-16T14:33:28.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.089+0000 7f6f61575640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f5c070b60 0x7f6f5c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.089+0000 7f6f61575640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f61575640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 0x7f6f5c0c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f61575640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0c26e0 con 0x7f6f5c159950 2023-10-16T14:33:28.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f61575640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0c29c0 con 0x7f6f5c0ca0a0 2023-10-16T14:33:28.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f61575640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0c2ca0 con 0x7f6f5c070b60 2023-10-16T14:33:28.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 0x7f6f5c0c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 0x7f6f5c0c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37918/0 (socket says 172.21.15.29:37918) 2023-10-16T14:33:28.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 -- 172.21.15.29:0/1961383721 learned_addr learned my addr 172.21.15.29:0/1961383721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:28.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f5bfff640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f5c070b60 0x7f6f5c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f53fff640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 -- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f5c070b60 msgr2=0x7f6f5c157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f5c070b60 0x7f6f5c157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 -- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5c0ca0a0 msgr2=0x7f6f5c154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.090+0000 7f6f60d74640 1 -- 172.21.15.29:0/1961383721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54009a70 con 0x7f6f5c159950 2023-10-16T14:33:28.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.091+0000 7f6f60d74640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 0x7f6f5c0c6050 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009d90 tx=0x7f6f54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:28.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.091+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5402cea0 con 0x7f6f5c159950 2023-10-16T14:33:28.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.091+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5403e630 con 0x7f6f5c159950 2023-10-16T14:33:28.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.091+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c0c2f80 con 0x7f6f5c159950 2023-10-16T14:33:28.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.091+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5403e9e0 con 0x7f6f5c159950 2023-10-16T14:33:28.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.091+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c157960 con 0x7f6f5c159950 2023-10-16T14:33:28.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.092+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c1f5a90 con 0x7f6f5c159950 2023-10-16T14:33:28.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.093+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6f5403b050 con 0x7f6f5c159950 2023-10-16T14:33:28.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.094+0000 7f6f597fa640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6f3c06acf0 0x7f6f3c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.094+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f540abbd0 con 0x7f6f5c159950 2023-10-16T14:33:28.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.095+0000 7f6f5bfff640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6f3c06acf0 0x7f6f3c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.096+0000 7f6f5bfff640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6f3c06acf0 0x7f6f3c06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c000f60 tx=0x7f6f4c007c00 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:28.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.099+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6f54076c40 con 0x7f6f5c159950 2023-10-16T14:33:28.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.330+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f5c04e9b0 con 0x7f6f5c159950 2023-10-16T14:33:28.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.331+0000 7f6f597fa640 1 -- 172.21.15.29:0/1961383721 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f6f54039020 con 0x7f6f5c159950 2023-10-16T14:33:28.335 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:28.335 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:05.723125+0000","modified":"2023-10-16T14:33:08.790919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5926":{"gid":5926,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3812375996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3812375996},{"type":"v1","addr":"172.21.15.170:6837","nonce":3812375996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T14:33:28.336 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-16T14:33:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.333+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6f3c06acf0 msgr2=0x7f6f3c06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.333+0000 7f6f61575640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6f3c06acf0 0x7f6f3c06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c000f60 tx=0x7f6f4c007c00 comp rx=0 tx=0).stop 2023-10-16T14:33:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 msgr2=0x7f6f5c0c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 0x7f6f5c0c6050 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009d90 tx=0x7f6f54004a40 comp rx=0 tx=0).stop 2023-10-16T14:33:28.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 shutdown_connections 2023-10-16T14:33:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6f3c06acf0 0x7f6f3c06d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f5c159950 0x7f6f5c0c6050 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f5c0ca0a0 0x7f6f5c154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 --2- 172.21.15.29:0/1961383721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f5c070b60 0x7f6f5c157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 >> 172.21.15.29:0/1961383721 conn(0x7f6f5c0ce6a0 msgr2=0x7f6f5c070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 shutdown_connections 2023-10-16T14:33:28.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.334+0000 7f6f61575640 1 -- 172.21.15.29:0/1961383721 wait complete. 2023-10-16T14:33:28.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:28.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.570+0000 7f9b0624f640 1 Processor -- start 2023-10-16T14:33:28.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.570+0000 7f9b0624f640 1 -- start start 2023-10-16T14:33:28.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9b0624f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9b0015aaa0 0x7f9b0015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9b0624f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 0x7f9b00070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9b0624f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0015ed60 con 0x7f9b0015ae80 2023-10-16T14:33:28.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9b0624f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0006b810 con 0x7f9b0015aaa0 2023-10-16T14:33:28.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9b0624f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000707e0 con 0x7f9b000ce390 2023-10-16T14:33:28.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9af7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b0015ae80 0x7f9b000cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58052/0 (socket says 172.21.15.29:58052) 2023-10-16T14:33:28.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9af7fff640 1 -- 172.21.15.29:0/445211526 learned_addr learned my addr 172.21.15.29:0/445211526 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:28.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.572+0000 7f9aff7fe640 1 --2- 172.21.15.29:0/445211526 >> v2:172.21.15.170:3300/0 conn(0x7f9b0015aaa0 0x7f9b0015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afffff640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 0x7f9b00070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079020198 0 0) 0x7f9b0015ed60 con 0x7f9b0015ae80 2023-10-16T14:33:28.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aec003520 con 0x7f9b0015ae80 2023-10-16T14:33:28.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2913392196 0 0) 0x7f9aec003520 con 0x7f9b0015ae80 2023-10-16T14:33:28.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 msgr2=0x7f9b00070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:28.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 0x7f9b00070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 >> v2:172.21.15.170:3300/0 conn(0x7f9b0015aaa0 msgr2=0x7f9b0015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:28.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 --2- 172.21.15.29:0/445211526 >> v2:172.21.15.170:3300/0 conn(0x7f9b0015aaa0 0x7f9b0015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0015ef90 con 0x7f9b0015ae80 2023-10-16T14:33:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9ae8001690 con 0x7f9b0015ae80 2023-10-16T14:33:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 >> v1:172.21.15.29:6789/0 conn(0x7f9b0015ae80 legacy=0x7f9b000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec003970 0x7f9aec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 0x7f9aec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec004d00 0x7f9aec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0006b810 con 0x7f9aec004d00 2023-10-16T14:33:28.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000707e0 con 0x7f9aec003970 2023-10-16T14:33:28.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.573+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00b980 con 0x7f9b000ce390 2023-10-16T14:33:28.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9afffff640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec004d00 0x7f9aec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9aff7fe640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec003970 0x7f9aec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9af7fff640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 0x7f9aec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9afffff640 1 -- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 msgr2=0x7f9aec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:28.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9afffff640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000ce390 0x7f9aec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9afffff640 1 -- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec003970 msgr2=0x7f9aec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:28.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9afffff640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec003970 0x7f9aec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.574+0000 7f9afffff640 1 -- 172.21.15.29:0/445211526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec003520 con 0x7f9aec004d00 2023-10-16T14:33:28.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.575+0000 7f9afffff640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec004d00 0x7f9aec005180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009fc0 tx=0x7f9aec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abac97fdf1ada6b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:28.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af802cc20 con 0x7f9aec004d00 2023-10-16T14:33:28.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af800f040 con 0x7f9aec004d00 2023-10-16T14:33:28.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9afeffd640 1 -- 172.21.15.29:0/445211526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8034440 con 0x7f9aec004d00 2023-10-16T14:33:28.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9b0624f640 1 -- 172.21.15.29:0/445211526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec004d00 msgr2=0x7f9aec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9b0624f640 1 --2- 172.21.15.29:0/445211526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aec004d00 0x7f9aec005180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009fc0 tx=0x7f9aec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9afffff640 1 -- 172.21.15.29:0/445211526 reap_dead start 2023-10-16T14:33:28.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9b0624f640 1 -- 172.21.15.29:0/445211526 shutdown_connections 2023-10-16T14:33:28.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.576+0000 7f9b0624f640 1 -- 172.21.15.29:0/445211526 >> 172.21.15.29:0/445211526 conn(0x7f9b000cb410 msgr2=0x7f9b000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:28.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.577+0000 7f9b0624f640 1 -- 172.21.15.29:0/445211526 shutdown_connections 2023-10-16T14:33:28.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.577+0000 7f9b0624f640 1 -- 172.21.15.29:0/445211526 wait complete. 2023-10-16T14:33:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.578+0000 7f9b0624f640 1 Processor -- start 2023-10-16T14:33:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.579+0000 7f9b0624f640 1 -- start start 2023-10-16T14:33:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9b0624f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec004d00 0x7f9b0014ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9b0624f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 0x7f9b000b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9b0624f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000c50d0 0x7f9b000b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9b0624f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000c4030 con 0x7f9b0015b440 2023-10-16T14:33:28.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9b0624f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000c4310 con 0x7f9aec004d00 2023-10-16T14:33:28.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9b0624f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000c45f0 con 0x7f9b000c50d0 2023-10-16T14:33:28.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 0x7f9b000b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 0x7f9b000b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37930/0 (socket says 172.21.15.29:37930) 2023-10-16T14:33:28.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 -- 172.21.15.29:0/1653875521 learned_addr learned my addr 172.21.15.29:0/1653875521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9aff7fe640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec004d00 0x7f9b0014ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9afffff640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000c50d0 0x7f9b000b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 -- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000c50d0 msgr2=0x7f9b000b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000c50d0 0x7f9b000b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 -- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec004d00 msgr2=0x7f9b0014ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec004d00 0x7f9b0014ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.580+0000 7f9af7fff640 1 -- 172.21.15.29:0/1653875521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000c48d0 con 0x7f9b0015b440 2023-10-16T14:33:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.581+0000 7f9af7fff640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 0x7f9b000b2c90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8001eb0 tx=0x7f9ae800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.581+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae800c870 con 0x7f9b0015b440 2023-10-16T14:33:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.581+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae800c9f0 con 0x7f9b0015b440 2023-10-16T14:33:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.581+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b000c4bb0 con 0x7f9b0015b440 2023-10-16T14:33:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.581+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae800cda0 con 0x7f9b0015b440 2023-10-16T14:33:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.581+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0014b230 con 0x7f9b0015b440 2023-10-16T14:33:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.582+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b000c4df0 con 0x7f9b0015b440 2023-10-16T14:33:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.583+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9ae801d460 con 0x7f9b0015b440 2023-10-16T14:33:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.584+0000 7f9afcff9640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f9ae006acc0 0x7f9ae006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.584+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9ae8089690 con 0x7f9b0015b440 2023-10-16T14:33:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.585+0000 7f9aff7fe640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f9ae006acc0 0x7f9ae006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.586+0000 7f9aff7fe640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f9ae006acc0 0x7f9ae006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b000b39b0 tx=0x7f9af0008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:28.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.589+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b000c4df0 con 0x7f9b0015b440 2023-10-16T14:33:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.604+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9ae8054ad0 con 0x7f9b0015b440 2023-10-16T14:33:28.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.788+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b000c4df0 con 0x7f9b0015b440 2023-10-16T14:33:28.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.789+0000 7f9afcff9640 1 -- 172.21.15.29:0/1653875521 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17579 (secure 0 0 0) 0x7f9b000c4df0 con 0x7f9b0015b440 2023-10-16T14:33:28.794 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:28.794 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":99,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:26.572859+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/136212456":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/3176118466":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/2083683232":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/1621614164":"2023-10-17T14:33:26.572781+0000","172.21.15.170:6833/3288786257":"2023-10-17T14:33:26.572781+0000","172.21.15.170:6832/3288786257":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/2674280726":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/434807034":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/503917561":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/661241058":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/565742787":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1207268771":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6833/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6832/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1288542658":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/329350515":"2023-10-17T14:33:20.443300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f9ae006acc0 msgr2=0x7f9ae006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f9ae006acc0 0x7f9ae006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b000b39b0 tx=0x7f9af0008040 comp rx=0 tx=0).stop 2023-10-16T14:33:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 msgr2=0x7f9b000b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 0x7f9b000b2c90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8001eb0 tx=0x7f9ae800ed50 comp rx=0 tx=0).stop 2023-10-16T14:33:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 shutdown_connections 2023-10-16T14:33:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f9ae006acc0 0x7f9ae006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b0015b440 0x7f9b000b2c90 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b000c50d0 0x7f9b000b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 --2- 172.21.15.29:0/1653875521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9aec004d00 0x7f9b0014ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.792+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 >> 172.21.15.29:0/1653875521 conn(0x7f9b000cb410 msgr2=0x7f9b000c17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.793+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 shutdown_connections 2023-10-16T14:33:28.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:28.793+0000 7f9b0624f640 1 -- 172.21.15.29:0/1653875521 wait complete. 2023-10-16T14:33:28.816 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:33:28.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.044+0000 7f05d2e0e640 1 Processor -- start 2023-10-16T14:33:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.044+0000 7f05d2e0e640 1 -- start start 2023-10-16T14:33:29.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.046+0000 7f05d2e0e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05cc15aaa0 0x7f05cc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.046+0000 7f05d2e0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 0x7f05cc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.046+0000 7f05d2e0e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc15ed60 con 0x7f05cc0ccaa0 2023-10-16T14:33:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.046+0000 7f05d2e0e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05cc06b810 con 0x7f05cc15aaa0 2023-10-16T14:33:29.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.046+0000 7f05d2e0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0707e0 con 0x7f05cc15ae80 2023-10-16T14:33:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.047+0000 7f05d0baa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05cc15aaa0 0x7f05cc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.047+0000 7f05cbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 0x7f05cc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.047+0000 7f05d0baa640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05cc15aaa0 0x7f05cc15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:39782/0 (socket says 172.21.15.29:39782) 2023-10-16T14:33:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.047+0000 7f05cbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 0x7f05cc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59318/0 (socket says 172.21.15.29:59318) 2023-10-16T14:33:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.047+0000 7f05d0baa640 1 -- 172.21.15.29:0/1958448504 learned_addr learned my addr 172.21.15.29:0/1958448504 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.047+0000 7f05cbfff640 1 -- 172.21.15.29:0/1958448504 >> v2:172.21.15.170:3300/0 conn(0x7f05cc15aaa0 msgr2=0x7f05cc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.048+0000 7f05cbfff640 1 --2- 172.21.15.29:0/1958448504 >> v2:172.21.15.170:3300/0 conn(0x7f05cc15aaa0 0x7f05cc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.048+0000 7f05cbfff640 1 -- 172.21.15.29:0/1958448504 >> v1:172.21.15.170:6789/0 conn(0x7f05cc0ccaa0 legacy=0x7f05cc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.048+0000 7f05cbfff640 1 -- 172.21.15.29:0/1958448504 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc15ef90 con 0x7f05cc15ae80 2023-10-16T14:33:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.049+0000 7f05cbfff640 1 --2- 172.21.15.29:0/1958448504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 0x7f05cc0cc560 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097a0 tx=0x7f05b802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49c929a144a9917 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.049+0000 7f05cb7fe640 1 -- 172.21.15.29:0/1958448504 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b802d040 con 0x7f05cc15ae80 2023-10-16T14:33:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.049+0000 7f05cb7fe640 1 -- 172.21.15.29:0/1958448504 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8031810 con 0x7f05cc15ae80 2023-10-16T14:33:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.049+0000 7f05cb7fe640 1 -- 172.21.15.29:0/1958448504 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8031ba0 con 0x7f05cc15ae80 2023-10-16T14:33:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 -- 172.21.15.29:0/1958448504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 msgr2=0x7f05cc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/1958448504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 0x7f05cc0cc560 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097a0 tx=0x7f05b802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 -- 172.21.15.29:0/1958448504 shutdown_connections 2023-10-16T14:33:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/1958448504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc15ae80 0x7f05cc0cc560 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/1958448504 >> v2:172.21.15.170:3300/0 conn(0x7f05cc15aaa0 0x7f05cc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 -- 172.21.15.29:0/1958448504 >> 172.21.15.29:0/1958448504 conn(0x7f05cc0cb410 msgr2=0x7f05cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 -- 172.21.15.29:0/1958448504 shutdown_connections 2023-10-16T14:33:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.050+0000 7f05d2e0e640 1 -- 172.21.15.29:0/1958448504 wait complete. 2023-10-16T14:33:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.053+0000 7f05d2e0e640 1 Processor -- start 2023-10-16T14:33:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.053+0000 7f05d2e0e640 1 -- start start 2023-10-16T14:33:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.054+0000 7f05d2e0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc0ccaa0 0x7f05cc149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.054+0000 7f05d2e0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05cc15aaa0 0x7f05cc0c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.054+0000 7f05d2e0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 0x7f05cc15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.054+0000 7f05d2e0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc154270 con 0x7f05cc15ae80 2023-10-16T14:33:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.054+0000 7f05d2e0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc15c7c0 con 0x7f05cc15aaa0 2023-10-16T14:33:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.054+0000 7f05d2e0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc0c29b0 con 0x7f05cc0ccaa0 2023-10-16T14:33:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 0x7f05cc15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 0x7f05cc15cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37942/0 (socket says 172.21.15.29:37942) 2023-10-16T14:33:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 -- 172.21.15.29:0/3292567013 learned_addr learned my addr 172.21.15.29:0/3292567013 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05cbfff640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05cc15aaa0 0x7f05cc0c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d0baa640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc0ccaa0 0x7f05cc149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 -- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc0ccaa0 msgr2=0x7f05cc149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc0ccaa0 0x7f05cc149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 -- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05cc15aaa0 msgr2=0x7f05cc0c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05cc15aaa0 0x7f05cc0c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.055+0000 7f05d13ab640 1 -- 172.21.15.29:0/3292567013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8009400 con 0x7f05cc15ae80 2023-10-16T14:33:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.056+0000 7f05d13ab640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 0x7f05cc15cef0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f05c400c6b0 tx=0x7f05c400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.056+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4003b70 con 0x7f05cc15ae80 2023-10-16T14:33:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.056+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c4003cf0 con 0x7f05cc15ae80 2023-10-16T14:33:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.056+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4003ea0 con 0x7f05cc15ae80 2023-10-16T14:33:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.056+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05cc155720 con 0x7f05cc15ae80 2023-10-16T14:33:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.056+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc15c200 con 0x7f05cc15ae80 2023-10-16T14:33:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.058+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05cc04e160 con 0x7f05cc15ae80 2023-10-16T14:33:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.059+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f05c4005310 con 0x7f05cc15ae80 2023-10-16T14:33:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.060+0000 7f05c97fa640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f05b006af00 0x7f05b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.060+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f05c40896f0 con 0x7f05cc15ae80 2023-10-16T14:33:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.061+0000 7f05d0baa640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f05b006af00 0x7f05b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.062+0000 7f05d0baa640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f05b006af00 0x7f05b006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009bd0 tx=0x7f05c0009240 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.064+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05c4054400 con 0x7f05cc15ae80 2023-10-16T14:33:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.300+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05cc0c2af0 con 0x7f05cc15ae80 2023-10-16T14:33:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.301+0000 7f05c97fa640 1 -- 172.21.15.29:0/3292567013 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f05c4057a20 con 0x7f05cc15ae80 2023-10-16T14:33:29.307 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:29.307 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:05.723125+0000","modified":"2023-10-16T14:33:08.790919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5926":{"gid":5926,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3812375996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3812375996},{"type":"v1","addr":"172.21.15.170:6837","nonce":3812375996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T14:33:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-16T14:33:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f05b006af00 msgr2=0x7f05b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f05b006af00 0x7f05b006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009bd0 tx=0x7f05c0009240 comp rx=0 tx=0).stop 2023-10-16T14:33:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 msgr2=0x7f05cc15cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 0x7f05cc15cef0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f05c400c6b0 tx=0x7f05c400cb70 comp rx=0 tx=0).stop 2023-10-16T14:33:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 shutdown_connections 2023-10-16T14:33:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f05b006af00 0x7f05b006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05cc15ae80 0x7f05cc15cef0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05cc15aaa0 0x7f05cc0c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 --2- 172.21.15.29:0/3292567013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05cc0ccaa0 0x7f05cc149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 >> 172.21.15.29:0/3292567013 conn(0x7f05cc0cb410 msgr2=0x7f05cc0b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 shutdown_connections 2023-10-16T14:33:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.304+0000 7f05d2e0e640 1 -- 172.21.15.29:0/3292567013 wait complete. 2023-10-16T14:33:29.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.572+0000 7f6d9c8c2640 1 Processor -- start 2023-10-16T14:33:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.573+0000 7f6d9c8c2640 1 -- start start 2023-10-16T14:33:29.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.574+0000 7f6d9c8c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 0x7f6d98164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.574+0000 7f6d9c8c2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9806f7f0 con 0x7f6d98159f00 2023-10-16T14:33:29.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.574+0000 7f6d9c8c2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9806c5e0 con 0x7f6d98159b20 2023-10-16T14:33:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.574+0000 7f6d9c8c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9806c8c0 con 0x7f6d980b0b20 2023-10-16T14:33:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d8ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6d98159f00 0x7f6d980b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58060/0 (socket says 172.21.15.29:58060) 2023-10-16T14:33:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d8ffff640 1 -- 172.21.15.29:0/2170790473 learned_addr learned my addr 172.21.15.29:0/2170790473 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d96d9d640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 0x7f6d98164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384499727 0 0) 0x7f6d9806f7f0 con 0x7f6d98159f00 2023-10-16T14:33:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d84003520 con 0x7f6d98159f00 2023-10-16T14:33:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4231903503 0 0) 0x7f6d84003520 con 0x7f6d98159f00 2023-10-16T14:33:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 msgr2=0x7f6d98164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 0x7f6d98164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 >> v1:172.21.15.170:6789/0 conn(0x7f6d98159b20 legacy=0x7f6d9806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9806cba0 con 0x7f6d98159f00 2023-10-16T14:33:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.575+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6d80001670 con 0x7f6d98159f00 2023-10-16T14:33:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 >> v1:172.21.15.29:6789/0 conn(0x7f6d98159f00 legacy=0x7f6d980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 0x7f6d840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d84004d00 0x7f6d84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9806c8c0 con 0x7f6d840039a0 2023-10-16T14:33:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8400dc70 con 0x7f6d84004d00 2023-10-16T14:33:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8400de30 con 0x7f6d980b0b20 2023-10-16T14:33:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d9659c640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d96d9d640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d84004d00 0x7f6d84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d9659c640 1 -- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 msgr2=0x7f6d840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d9659c640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980b0b20 0x7f6d840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d9659c640 1 -- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d84004d00 msgr2=0x7f6d84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d9659c640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d84004d00 0x7f6d84005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.576+0000 7f6d9659c640 1 -- 172.21.15.29:0/2170790473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84003520 con 0x7f6d840039a0 2023-10-16T14:33:29.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.577+0000 7f6d9659c640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d84003dc0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6d88001720 tx=0x7f6d840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f3e23cdd46abd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880052d0 con 0x7f6d840039a0 2023-10-16T14:33:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8802e050 con 0x7f6d840039a0 2023-10-16T14:33:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d95d9b640 1 -- 172.21.15.29:0/2170790473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880035f0 con 0x7f6d840039a0 2023-10-16T14:33:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/2170790473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 msgr2=0x7f6d84003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/2170790473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d84003dc0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6d88001720 tx=0x7f6d840057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d96d9d640 1 -- 172.21.15.29:0/2170790473 reap_dead start 2023-10-16T14:33:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/2170790473 shutdown_connections 2023-10-16T14:33:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/2170790473 >> 172.21.15.29:0/2170790473 conn(0x7f6d980cb410 msgr2=0x7f6d980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:29.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.578+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/2170790473 shutdown_connections 2023-10-16T14:33:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.579+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/2170790473 wait complete. 2023-10-16T14:33:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.580+0000 7f6d9c8c2640 1 Processor -- start 2023-10-16T14:33:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.580+0000 7f6d9c8c2640 1 -- start start 2023-10-16T14:33:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.581+0000 7f6d9c8c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d981ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.581+0000 7f6d9c8c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980ac890 0x7f6d980acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.581+0000 7f6d9c8c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d980c7840 0x7f6d980c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.581+0000 7f6d9c8c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d981ed100 con 0x7f6d840039a0 2023-10-16T14:33:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.581+0000 7f6d9c8c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d980ad250 con 0x7f6d980c7840 2023-10-16T14:33:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.581+0000 7f6d9c8c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d98156770 con 0x7f6d980ac890 2023-10-16T14:33:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d981ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d981ecbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37952/0 (socket says 172.21.15.29:37952) 2023-10-16T14:33:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 -- 172.21.15.29:0/409308384 learned_addr learned my addr 172.21.15.29:0/409308384 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d96d9d640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d980c7840 0x7f6d980c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d8ffff640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980ac890 0x7f6d980acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 -- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980ac890 msgr2=0x7f6d980acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980ac890 0x7f6d980acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 -- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d980c7840 msgr2=0x7f6d980c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d980c7840 0x7f6d980c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d9659c640 1 -- 172.21.15.29:0/409308384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d981f3830 con 0x7f6d840039a0 2023-10-16T14:33:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d96d9d640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d980c7840 0x7f6d980c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:29.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.582+0000 7f6d8ffff640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980ac890 0x7f6d980acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:29.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.583+0000 7f6d9659c640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d981ecbc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f6d88008fd0 tx=0x7f6d88001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:29.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.583+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880076c0 con 0x7f6d840039a0 2023-10-16T14:33:29.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.583+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d88007a30 con 0x7f6d840039a0 2023-10-16T14:33:29.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.583+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d88006c20 con 0x7f6d840039a0 2023-10-16T14:33:29.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.583+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d981f2f00 con 0x7f6d840039a0 2023-10-16T14:33:29.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.583+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d981f34c0 con 0x7f6d840039a0 2023-10-16T14:33:29.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.584+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d981f3680 con 0x7f6d840039a0 2023-10-16T14:33:29.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.585+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6d880035f0 con 0x7f6d840039a0 2023-10-16T14:33:29.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.586+0000 7f6d8effd640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6d7806af80 0x7f6d7806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:29.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.586+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6d880abac0 con 0x7f6d840039a0 2023-10-16T14:33:29.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.587+0000 7f6d8ffff640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6d7806af80 0x7f6d7806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:29.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.588+0000 7f6d8ffff640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6d7806af80 0x7f6d7806d440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d980ad450 tx=0x7f6d80009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:29.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.590+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d88076740 con 0x7f6d840039a0 2023-10-16T14:33:29.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.823+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d9804e160 con 0x7f6d840039a0 2023-10-16T14:33:29.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.824+0000 7f6d8effd640 1 -- 172.21.15.29:0/409308384 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f6d88079d60 con 0x7f6d840039a0 2023-10-16T14:33:29.828 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:29.828 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:05.723125+0000","modified":"2023-10-16T14:33:08.790919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5926":{"gid":5926,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6837/3812375996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3812375996},{"type":"v1","addr":"172.21.15.170:6837","nonce":3812375996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T14:33:29.829 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-16T14:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6d7806af80 msgr2=0x7f6d7806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6d7806af80 0x7f6d7806d440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d980ad450 tx=0x7f6d80009040 comp rx=0 tx=0).stop 2023-10-16T14:33:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 msgr2=0x7f6d981ecbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d981ecbc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f6d88008fd0 tx=0x7f6d88001120 comp rx=0 tx=0).stop 2023-10-16T14:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 shutdown_connections 2023-10-16T14:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f6d7806af80 0x7f6d7806d440 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d980c7840 0x7f6d980c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d980ac890 0x7f6d980acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 --2- 172.21.15.29:0/409308384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d840039a0 0x7f6d981ecbc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 >> 172.21.15.29:0/409308384 conn(0x7f6d980cb410 msgr2=0x7f6d980cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 shutdown_connections 2023-10-16T14:33:29.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:29.827+0000 7f6d9c8c2640 1 -- 172.21.15.29:0/409308384 wait complete. 2023-10-16T14:33:29.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:30.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.069+0000 7fefbe225640 1 Processor -- start 2023-10-16T14:33:30.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.069+0000 7fefbe225640 1 -- start start 2023-10-16T14:33:30.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbe225640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb8159950 0x7fefb815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbe225640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefb8070450 0x7fefb80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbe225640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb8070df0 con 0x7fefb806c100 2023-10-16T14:33:30.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbe225640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb815e490 con 0x7fefb8070450 2023-10-16T14:33:30.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbe225640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb815aef0 con 0x7fefb8159950 2023-10-16T14:33:30.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbca22640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefb8070450 0x7fefb80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbda24640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fefb806c100 0x7fefb806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58068/0 (socket says 172.21.15.29:58068) 2023-10-16T14:33:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbca22640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefb8070450 0x7fefb80708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37966/0 (socket says 172.21.15.29:37966) 2023-10-16T14:33:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.070+0000 7fefbda24640 1 -- 172.21.15.29:0/347027968 learned_addr learned my addr 172.21.15.29:0/347027968 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.071+0000 7fefbd223640 1 --2- 172.21.15.29:0/347027968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb8159950 0x7fefb815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.071+0000 7fefbca22640 1 -- 172.21.15.29:0/347027968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb8159950 msgr2=0x7fefb815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:30.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.071+0000 7fefbca22640 1 --2- 172.21.15.29:0/347027968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb8159950 0x7fefb815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.071+0000 7fefbca22640 1 -- 172.21.15.29:0/347027968 >> v1:172.21.15.29:6789/0 conn(0x7fefb806c100 legacy=0x7fefb806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.071+0000 7fefbca22640 1 -- 172.21.15.29:0/347027968 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb806cc80 con 0x7fefb8070450 2023-10-16T14:33:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.071+0000 7fefb7fff640 1 -- 172.21.15.29:0/347027968 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235636872 0 0) 0x7fefb8070df0 con 0x7fefb806c100 2023-10-16T14:33:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.072+0000 7fefbca22640 1 --2- 172.21.15.29:0/347027968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80708b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fefac0097a0 tx=0x7fefac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49494071ab61ba0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.072+0000 7fefb7fff640 1 -- 172.21.15.29:0/347027968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac02e070 con 0x7fefb8070450 2023-10-16T14:33:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.072+0000 7fefb7fff640 1 -- 172.21.15.29:0/347027968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac0315a0 con 0x7fefb8070450 2023-10-16T14:33:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.072+0000 7fefb7fff640 1 -- 172.21.15.29:0/347027968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac031930 con 0x7fefb8070450 2023-10-16T14:33:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 -- 172.21.15.29:0/347027968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 msgr2=0x7fefb80708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 --2- 172.21.15.29:0/347027968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80708b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fefac0097a0 tx=0x7fefac02b840 comp rx=0 tx=0).stop 2023-10-16T14:33:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 -- 172.21.15.29:0/347027968 shutdown_connections 2023-10-16T14:33:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 --2- 172.21.15.29:0/347027968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80708b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 --2- 172.21.15.29:0/347027968 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb8159950 0x7fefb815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 -- 172.21.15.29:0/347027968 >> 172.21.15.29:0/347027968 conn(0x7fefb80ce6a0 msgr2=0x7fefb80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 -- 172.21.15.29:0/347027968 shutdown_connections 2023-10-16T14:33:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.073+0000 7fefbe225640 1 -- 172.21.15.29:0/347027968 wait complete. 2023-10-16T14:33:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.075+0000 7fefbe225640 1 Processor -- start 2023-10-16T14:33:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.075+0000 7fefbe225640 1 -- start start 2023-10-16T14:33:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbe225640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb806c100 0x7fefb8157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbe225640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbe225640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefb8159950 0x7fefb814db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbe225640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb814e070 con 0x7fefb8070450 2023-10-16T14:33:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbe225640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c4e90 con 0x7fefb8159950 2023-10-16T14:33:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbe225640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8158320 con 0x7fefb806c100 2023-10-16T14:33:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbca22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbca22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80c3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37970/0 (socket says 172.21.15.29:37970) 2023-10-16T14:33:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.076+0000 7fefbca22640 1 -- 172.21.15.29:0/1607302387 learned_addr learned my addr 172.21.15.29:0/1607302387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbca22640 1 -- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb806c100 msgr2=0x7fefb8157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbca22640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb806c100 0x7fefb8157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbda24640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefb8159950 0x7fefb814db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbca22640 1 -- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefb8159950 msgr2=0x7fefb814db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbca22640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefb8159950 0x7fefb814db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbca22640 1 -- 172.21.15.29:0/1607302387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac009400 con 0x7fefb8070450 2023-10-16T14:33:30.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.077+0000 7fefbca22640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80c3240 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fefac034040 tx=0x7fefac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:30.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.078+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac032840 con 0x7fefb8070450 2023-10-16T14:33:30.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.078+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac0329a0 con 0x7fefb8070450 2023-10-16T14:33:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.078+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac032d50 con 0x7fefb8070450 2023-10-16T14:33:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.078+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb81f24b0 con 0x7fefb8070450 2023-10-16T14:33:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.078+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb81f2a70 con 0x7fefb8070450 2023-10-16T14:33:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.079+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb804e9b0 con 0x7fefb8070450 2023-10-16T14:33:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.080+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fefac04e430 con 0x7fefb8070450 2023-10-16T14:33:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.081+0000 7fefb5ffb640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fefa406af00 0x7fefa406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.081+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fefac0abaf0 con 0x7fefb8070450 2023-10-16T14:33:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.082+0000 7fefbd223640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fefa406af00 0x7fefa406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.083+0000 7fefbd223640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fefa406af00 0x7fefa406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefa80070b0 tx=0x7fefa8007040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.086+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefac0766f0 con 0x7fefb8070450 2023-10-16T14:33:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.279+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefb8157b00 con 0x7fefb8070450 2023-10-16T14:33:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.283+0000 7fefb5ffb640 1 -- 172.21.15.29:0/1607302387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17579 (secure 0 0 0) 0x7fefac079d10 con 0x7fefb8070450 2023-10-16T14:33:30.287 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:30.288 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":99,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:26.572859+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:03.693665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:04.756815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/136212456":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/3176118466":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/2083683232":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/1621614164":"2023-10-17T14:33:26.572781+0000","172.21.15.170:6833/3288786257":"2023-10-17T14:33:26.572781+0000","172.21.15.170:6832/3288786257":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/2674280726":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/434807034":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/503917561":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/661241058":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/565742787":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1207268771":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6833/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6832/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1288542658":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/329350515":"2023-10-17T14:33:20.443300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fefa406af00 msgr2=0x7fefa406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fefa406af00 0x7fefa406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefa80070b0 tx=0x7fefa8007040 comp rx=0 tx=0).stop 2023-10-16T14:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 msgr2=0x7fefb80c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80c3240 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fefac034040 tx=0x7fefac0046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 shutdown_connections 2023-10-16T14:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fefa406af00 0x7fefa406d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb8070450 0x7fefb80c3240 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefb8159950 0x7fefb814db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 --2- 172.21.15.29:0/1607302387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefb806c100 0x7fefb8157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 >> 172.21.15.29:0/1607302387 conn(0x7fefb80ce6a0 msgr2=0x7fefb80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:30.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 shutdown_connections 2023-10-16T14:33:30.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.286+0000 7fefbe225640 1 -- 172.21.15.29:0/1607302387 wait complete. 2023-10-16T14:33:30.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:33:30.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.525+0000 7fc2f759e640 1 Processor -- start 2023-10-16T14:33:30.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.525+0000 7fc2f759e640 1 -- start start 2023-10-16T14:33:30.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.526+0000 7fc2f759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc2f806c040 0x7fc2f806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f815f4a0 con 0x7fc2f80cda80 2023-10-16T14:33:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f806c960 con 0x7fc2f80ad970 2023-10-16T14:33:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f815b9d0 con 0x7fc2f806c040 2023-10-16T14:33:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc2f806c040 0x7fc2f806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc2f80cda80 0x7fc2f815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58072/0 (socket says 172.21.15.29:58072) 2023-10-16T14:33:30.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc2f806c040 0x7fc2f806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37976/0 (socket says 172.21.15.29:37976) 2023-10-16T14:33:30.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.527+0000 7fc2f6d9d640 1 -- 172.21.15.29:0/2960402947 learned_addr learned my addr 172.21.15.29:0/2960402947 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:30.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.528+0000 7fc2f659c640 1 -- 172.21.15.29:0/2960402947 >> v1:172.21.15.170:6789/0 conn(0x7fc2f80ad970 legacy=0x7fc2f815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:30.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.528+0000 7fc2f659c640 1 -- 172.21.15.29:0/2960402947 >> v1:172.21.15.29:6789/0 conn(0x7fc2f80cda80 legacy=0x7fc2f815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.528+0000 7fc2f659c640 1 -- 172.21.15.29:0/2960402947 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f815bcb0 con 0x7fc2f806c040 2023-10-16T14:33:30.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.529+0000 7fc2f659c640 1 --2- 172.21.15.29:0/2960402947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f806c040 0x7fc2f806c420 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80097a0 tx=0x7fc2e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1454df664eda5da3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:30.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.529+0000 7fc2f5d9b640 1 -- 172.21.15.29:0/2960402947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8031830 con 0x7fc2f806c040 2023-10-16T14:33:30.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.529+0000 7fc2f5d9b640 1 -- 172.21.15.29:0/2960402947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8031990 con 0x7fc2f806c040 2023-10-16T14:33:30.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.529+0000 7fc2f5d9b640 1 -- 172.21.15.29:0/2960402947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8031d20 con 0x7fc2f806c040 2023-10-16T14:33:30.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 -- 172.21.15.29:0/2960402947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f806c040 msgr2=0x7fc2f806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2960402947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f806c040 0x7fc2f806c420 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80097a0 tx=0x7fc2e802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 -- 172.21.15.29:0/2960402947 shutdown_connections 2023-10-16T14:33:30.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2960402947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f806c040 0x7fc2f806c420 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 -- 172.21.15.29:0/2960402947 >> 172.21.15.29:0/2960402947 conn(0x7fc2f80ce6a0 msgr2=0x7fc2f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:30.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 -- 172.21.15.29:0/2960402947 shutdown_connections 2023-10-16T14:33:30.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.530+0000 7fc2f759e640 1 -- 172.21.15.29:0/2960402947 wait complete. 2023-10-16T14:33:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.532+0000 7fc2f759e640 1 Processor -- start 2023-10-16T14:33:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.532+0000 7fc2f759e640 1 -- start start 2023-10-16T14:33:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 0x7fc2f80c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2f80cda80 0x7fc2f80ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2f8152be0 0x7fc2f80ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f81f6fe0 con 0x7fc2f80ad970 2023-10-16T14:33:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80c9e50 con 0x7fc2f80cda80 2023-10-16T14:33:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f81f60f0 con 0x7fc2f8152be0 2023-10-16T14:33:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 0x7fc2f80c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 0x7fc2f80c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37986/0 (socket says 172.21.15.29:37986) 2023-10-16T14:33:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.533+0000 7fc2f659c640 1 -- 172.21.15.29:0/2852384222 learned_addr learned my addr 172.21.15.29:0/2852384222 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f659c640 1 -- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2f8152be0 msgr2=0x7fc2f80ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f659c640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2f8152be0 0x7fc2f80ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f659c640 1 -- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2f80cda80 msgr2=0x7fc2f80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f659c640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2f80cda80 0x7fc2f80ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f659c640 1 -- 172.21.15.29:0/2852384222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8009400 con 0x7fc2f80ad970 2023-10-16T14:33:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f659c640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 0x7fc2f80c9910 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80098c0 tx=0x7fc2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e803ba70 con 0x7fc2f80ad970 2023-10-16T14:33:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e803bbf0 con 0x7fc2f80ad970 2023-10-16T14:33:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80453f0 con 0x7fc2f80ad970 2023-10-16T14:33:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f81f63d0 con 0x7fc2f80ad970 2023-10-16T14:33:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.534+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f81f6990 con 0x7fc2f80ad970 2023-10-16T14:33:30.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.536+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f80c56f0 con 0x7fc2f80ad970 2023-10-16T14:33:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.536+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc2e804e450 con 0x7fc2f80ad970 2023-10-16T14:33:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.537+0000 7fc2eeffd640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc2d806af00 0x7fc2d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.538+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc2e80abcf0 con 0x7fc2f80ad970 2023-10-16T14:33:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.538+0000 7fc2effff640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc2d806af00 0x7fc2d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.539+0000 7fc2effff640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc2d806af00 0x7fc2d806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80ad690 tx=0x7fc2e00079b0 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.542+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2e8076980 con 0x7fc2f80ad970 2023-10-16T14:33:30.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.777+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc2f806b330 con 0x7fc2f80ad970 2023-10-16T14:33:30.781 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:33:30.778+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:33:30.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.781+0000 7fc2eeffd640 1 -- 172.21.15.29:0/2852384222 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc2e8035090 con 0x7fc2f80ad970 2023-10-16T14:33:30.785 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:33:30.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc2d806af00 msgr2=0x7fc2d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc2d806af00 0x7fc2d806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80ad690 tx=0x7fc2e00079b0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 msgr2=0x7fc2f80c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 0x7fc2f80c9910 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80098c0 tx=0x7fc2e80046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 shutdown_connections 2023-10-16T14:33:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc2d806af00 0x7fc2d806d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2f8152be0 0x7fc2f80ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2f80cda80 0x7fc2f80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 --2- 172.21.15.29:0/2852384222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2f80ad970 0x7fc2f80c9910 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 >> 172.21.15.29:0/2852384222 conn(0x7fc2f80ce6a0 msgr2=0x7fc2f815ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 shutdown_connections 2023-10-16T14:33:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:30.784+0000 7fc2f759e640 1 -- 172.21.15.29:0/2852384222 wait complete. 2023-10-16T14:33:30.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:33:30.812 INFO:tasks.ceph.mds.b.smithi170.stderr: -2661> 2023-10-16T14:33:12.225+0000 7f6167adc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:33:30.893 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:33:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.034+0000 7f53c2602640 1 Processor -- start 2023-10-16T14:33:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.034+0000 7f53c2602640 1 -- start start 2023-10-16T14:33:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.035+0000 7f53c2602640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f53bc15ad70 0x7f53bc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.035+0000 7f53c2602640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53bc0caac0 0x7f53bc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.035+0000 7f53c2602640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0cb030 con 0x7f53bc0c9ca0 2023-10-16T14:33:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.035+0000 7f53c2602640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0c91c0 con 0x7f53bc0caac0 2023-10-16T14:33:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.035+0000 7f53c2602640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0c94a0 con 0x7f53bc15ad70 2023-10-16T14:33:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.035+0000 7f53b37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f53bc0c9ca0 0x7f53bc0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58086/0 (socket says 172.21.15.29:58086) 2023-10-16T14:33:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53b37fe640 1 -- 172.21.15.29:0/544139194 learned_addr learned my addr 172.21.15.29:0/544139194 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53c0b9f640 1 --2- 172.21.15.29:0/544139194 >> v2:172.21.15.29:3300/0 conn(0x7f53bc0caac0 0x7f53bc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53bbfff640 1 --2- 172.21.15.29:0/544139194 >> v2:172.21.15.170:3300/0 conn(0x7f53bc15ad70 0x7f53bc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53bb7fe640 1 -- 172.21.15.29:0/544139194 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920679393 0 0) 0x7f53bc0cb030 con 0x7f53bc0c9ca0 2023-10-16T14:33:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53bb7fe640 1 -- 172.21.15.29:0/544139194 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a8003520 con 0x7f53bc0c9ca0 2023-10-16T14:33:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53c0b9f640 1 -- 172.21.15.29:0/544139194 >> v2:172.21.15.170:3300/0 conn(0x7f53bc15ad70 msgr2=0x7f53bc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53c0b9f640 1 --2- 172.21.15.29:0/544139194 >> v2:172.21.15.170:3300/0 conn(0x7f53bc15ad70 0x7f53bc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53c0b9f640 1 -- 172.21.15.29:0/544139194 >> v1:172.21.15.29:6789/0 conn(0x7f53bc0c9ca0 legacy=0x7f53bc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.036+0000 7f53c0b9f640 1 -- 172.21.15.29:0/544139194 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0c9780 con 0x7f53bc0caac0 2023-10-16T14:33:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.037+0000 7f53c0b9f640 1 --2- 172.21.15.29:0/544139194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc0caac0 0x7f53bc06a2e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0cf780 tx=0x7f53b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bd0ec3d53859bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53bb7fe640 1 -- 172.21.15.29:0/544139194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4031830 con 0x7f53bc0caac0 2023-10-16T14:33:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53bb7fe640 1 -- 172.21.15.29:0/544139194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b4031990 con 0x7f53bc0caac0 2023-10-16T14:33:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53bb7fe640 1 -- 172.21.15.29:0/544139194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4031d20 con 0x7f53bc0caac0 2023-10-16T14:33:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 -- 172.21.15.29:0/544139194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc0caac0 msgr2=0x7f53bc06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 --2- 172.21.15.29:0/544139194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc0caac0 0x7f53bc06a2e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0cf780 tx=0x7f53b402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 -- 172.21.15.29:0/544139194 shutdown_connections 2023-10-16T14:33:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 --2- 172.21.15.29:0/544139194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc0caac0 0x7f53bc06a2e0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 --2- 172.21.15.29:0/544139194 >> v2:172.21.15.170:3300/0 conn(0x7f53bc15ad70 0x7f53bc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 -- 172.21.15.29:0/544139194 >> 172.21.15.29:0/544139194 conn(0x7f53bc0ce9b0 msgr2=0x7f53bc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 -- 172.21.15.29:0/544139194 shutdown_connections 2023-10-16T14:33:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.038+0000 7f53c2602640 1 -- 172.21.15.29:0/544139194 wait complete. 2023-10-16T14:33:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.040+0000 7f53c2602640 1 Processor -- start 2023-10-16T14:33:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.040+0000 7f53c2602640 1 -- start start 2023-10-16T14:33:31.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.041+0000 7f53c2602640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53bc0c9ca0 0x7f53bc157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.041+0000 7f53c2602640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53bc0caac0 0x7f53bc0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.041+0000 7f53c2602640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 0x7f53bc0b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.041+0000 7f53c2602640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc158060 con 0x7f53bc15ad70 2023-10-16T14:33:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.041+0000 7f53c2602640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc15c310 con 0x7f53bc0c9ca0 2023-10-16T14:33:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.041+0000 7f53c2602640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc15c5f0 con 0x7f53bc0caac0 2023-10-16T14:33:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 0x7f53bc0b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 0x7f53bc0b2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38004/0 (socket says 172.21.15.29:38004) 2023-10-16T14:33:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 -- 172.21.15.29:0/3103677529 learned_addr learned my addr 172.21.15.29:0/3103677529 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53bbfff640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53bc0c9ca0 0x7f53bc157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 -- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53bc0caac0 msgr2=0x7f53bc0b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53b37fe640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53bc0caac0 0x7f53bc0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53bc0caac0 0x7f53bc0b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 -- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53bc0c9ca0 msgr2=0x7f53bc157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53bc0c9ca0 0x7f53bc157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53c0b9f640 1 -- 172.21.15.29:0/3103677529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b4009400 con 0x7f53bc15ad70 2023-10-16T14:33:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.042+0000 7f53bbfff640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53bc0c9ca0 0x7f53bc157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.043+0000 7f53c0b9f640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 0x7f53bc0b2ec0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009890 tx=0x7f53b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.043+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b40317b0 con 0x7f53bc15ad70 2023-10-16T14:33:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.043+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b403b9f0 con 0x7f53bc15ad70 2023-10-16T14:33:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.043+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b403bda0 con 0x7f53bc15ad70 2023-10-16T14:33:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.043+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc15c8d0 con 0x7f53bc15ad70 2023-10-16T14:33:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.043+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc1582b0 con 0x7f53bc15ad70 2023-10-16T14:33:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.044+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc04e160 con 0x7f53bc15ad70 2023-10-16T14:33:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.045+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f53b40453f0 con 0x7f53bc15ad70 2023-10-16T14:33:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.046+0000 7f53b97fa640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f539c06af00 0x7f539c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.046+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f53b40abc30 con 0x7f53bc15ad70 2023-10-16T14:33:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.047+0000 7f53bbfff640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f539c06af00 0x7f539c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.048+0000 7f53bbfff640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f539c06af00 0x7f539c06d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009a80 tx=0x7f53ac0039d0 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.051+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f53b40769f0 con 0x7f53bc15ad70 2023-10-16T14:33:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.206+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f53b4076dc0 con 0x7f53bc15ad70 2023-10-16T14:33:31.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.289+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f53bc070930 con 0x7f53bc15ad70 2023-10-16T14:33:31.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.785+0000 7f53b97fa640 1 -- 172.21.15.29:0/3103677529 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f53b40508d0 con 0x7f53bc15ad70 2023-10-16T14:33:31.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.788+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f539c06af00 msgr2=0x7f539c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:31.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.788+0000 7f53c2602640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f539c06af00 0x7f539c06d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009a80 tx=0x7f53ac0039d0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.788+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 msgr2=0x7f53bc0b2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:31.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.788+0000 7f53c2602640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 0x7f53bc0b2ec0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009890 tx=0x7f53b40046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 shutdown_connections 2023-10-16T14:33:31.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f539c06af00 0x7f539c06d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53bc15ad70 0x7f53bc0b2ec0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f53bc0caac0 0x7f53bc0b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 --2- 172.21.15.29:0/3103677529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53bc0c9ca0 0x7f53bc157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:31.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 >> 172.21.15.29:0/3103677529 conn(0x7f53bc0ce9b0 msgr2=0x7f53bc0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:31.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 shutdown_connections 2023-10-16T14:33:31.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:31.789+0000 7f53c2602640 1 -- 172.21.15.29:0/3103677529 wait complete. 2023-10-16T14:33:31.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:33:32.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.048+0000 7f34b9856640 1 Processor -- start 2023-10-16T14:33:32.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.048+0000 7f34b9856640 1 -- start start 2023-10-16T14:33:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b9856640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 0x7f34b40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b9856640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b415b310 con 0x7f34b415ac10 2023-10-16T14:33:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b9856640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b406f640 con 0x7f34b40b0360 2023-10-16T14:33:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b9856640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b415f800 con 0x7f34b40cac20 2023-10-16T14:33:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f34b415ac10 0x7f34b406c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58088/0 (socket says 172.21.15.29:58088) 2023-10-16T14:33:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b2ffd640 1 -- 172.21.15.29:0/1174521526 learned_addr learned my addr 172.21.15.29:0/1174521526 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115966907 0 0) 0x7f34b415b310 con 0x7f34b415ac10 2023-10-16T14:33:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f349c003520 con 0x7f34b415ac10 2023-10-16T14:33:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.050+0000 7f34b37fe640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 0x7f34b40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 178008173 0 0) 0x7f349c003520 con 0x7f34b415ac10 2023-10-16T14:33:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 msgr2=0x7f34b40b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 0x7f34b40b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 >> v1:172.21.15.170:6789/0 conn(0x7f34b40b0360 legacy=0x7f34b40ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b415b5b0 con 0x7f34b415ac10 2023-10-16T14:33:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f34a4001670 con 0x7f34b415ac10 2023-10-16T14:33:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 >> v1:172.21.15.29:6789/0 conn(0x7f34b415ac10 legacy=0x7f34b406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f349c003a10 0x7f349c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 0x7f349c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c004d50 0x7f349c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b415f800 con 0x7f349c003a10 2023-10-16T14:33:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c00bbe0 con 0x7f349c004d50 2023-10-16T14:33:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c00bda0 con 0x7f34b40cac20 2023-10-16T14:33:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.051+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f349c003a10 0x7f349c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b27fc640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 0x7f349c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b37fe640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c004d50 0x7f349c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b2ffd640 1 -- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 msgr2=0x7f349c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40cac20 0x7f349c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b2ffd640 1 -- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c004d50 msgr2=0x7f349c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:32.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c004d50 0x7f349c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.052+0000 7f34b2ffd640 1 -- 172.21.15.29:0/1174521526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c003520 con 0x7f349c003a10 2023-10-16T14:33:32.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.053+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f349c003a10 0x7f349c003e30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c240 tx=0x7f349c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495a9be311f78a75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.053+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4005ad0 con 0x7f349c003a10 2023-10-16T14:33:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.053+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a4005c30 con 0x7f349c003a10 2023-10-16T14:33:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.053+0000 7f34b1ffb640 1 -- 172.21.15.29:0/1174521526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a402c8b0 con 0x7f349c003a10 2023-10-16T14:33:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b9856640 1 -- 172.21.15.29:0/1174521526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f349c003a10 msgr2=0x7f349c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b9856640 1 --2- 172.21.15.29:0/1174521526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f349c003a10 0x7f349c003e30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c240 tx=0x7f349c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b37fe640 1 -- 172.21.15.29:0/1174521526 reap_dead start 2023-10-16T14:33:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b9856640 1 -- 172.21.15.29:0/1174521526 shutdown_connections 2023-10-16T14:33:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b9856640 1 -- 172.21.15.29:0/1174521526 >> 172.21.15.29:0/1174521526 conn(0x7f34b40cb360 msgr2=0x7f34b40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b9856640 1 -- 172.21.15.29:0/1174521526 shutdown_connections 2023-10-16T14:33:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.054+0000 7f34b9856640 1 -- 172.21.15.29:0/1174521526 wait complete. 2023-10-16T14:33:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.056+0000 7f34b9856640 1 Processor -- start 2023-10-16T14:33:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.056+0000 7f34b9856640 1 -- start start 2023-10-16T14:33:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b9856640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c003a10 0x7f34b4159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b9856640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40b35f0 0x7f34b40b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b9856640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 0x7f34b41f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b9856640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40c4040 con 0x7f34b41f03d0 2023-10-16T14:33:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b9856640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40b2550 con 0x7f349c003a10 2023-10-16T14:33:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b9856640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40b2830 con 0x7f34b40b35f0 2023-10-16T14:33:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 0x7f34b41f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 0x7f34b41f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38012/0 (socket says 172.21.15.29:38012) 2023-10-16T14:33:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.057+0000 7f34b37fe640 1 -- 172.21.15.29:0/3268854250 learned_addr learned my addr 172.21.15.29:0/3268854250 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b27fc640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40b35f0 0x7f34b40b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c003a10 0x7f34b4159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b37fe640 1 -- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40b35f0 msgr2=0x7f34b40b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b37fe640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40b35f0 0x7f34b40b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b37fe640 1 -- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c003a10 msgr2=0x7f34b4159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b37fe640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c003a10 0x7f34b4159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b37fe640 1 -- 172.21.15.29:0/3268854250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40b2b10 con 0x7f34b41f03d0 2023-10-16T14:33:32.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c003a10 0x7f34b4159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:32.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b27fc640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40b35f0 0x7f34b40b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:32.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b37fe640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 0x7f34b41f0850 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f34b4070570 tx=0x7f34ac009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:32.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac007db0 con 0x7f34b41f03d0 2023-10-16T14:33:32.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac00f050 con 0x7f34b41f03d0 2023-10-16T14:33:32.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b40b2df0 con 0x7f34b41f03d0 2023-10-16T14:33:32.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac014650 con 0x7f34b41f03d0 2023-10-16T14:33:32.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.058+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b40c51d0 con 0x7f34b41f03d0 2023-10-16T14:33:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.059+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b40c5390 con 0x7f34b41f03d0 2023-10-16T14:33:32.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.061+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34ac0147b0 con 0x7f34b41f03d0 2023-10-16T14:33:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.062+0000 7f349bfff640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f34a006afc0 0x7f34a006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.062+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f34ac08a0b0 con 0x7f34b41f03d0 2023-10-16T14:33:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.063+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f34a006afc0 0x7f34a006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:32.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.064+0000 7f34b2ffd640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f34a006afc0 0x7f34a006d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f34a4034000 tx=0x7f34a4009030 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:32.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.066+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34ac054c20 con 0x7f34b41f03d0 2023-10-16T14:33:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.258+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f34b40b3310 con 0x7f34b41f03d0 2023-10-16T14:33:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.795+0000 7f349bfff640 1 -- 172.21.15.29:0/3268854250 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7f34ac058240 con 0x7f34b41f03d0 2023-10-16T14:33:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:33:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f34a006afc0 msgr2=0x7f34a006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f34a006afc0 0x7f34a006d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f34a4034000 tx=0x7f34a4009030 comp rx=0 tx=0).stop 2023-10-16T14:33:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 msgr2=0x7f34b41f0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 0x7f34b41f0850 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f34b4070570 tx=0x7f34ac009da0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 shutdown_connections 2023-10-16T14:33:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f34a006afc0 0x7f34a006d480 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34b41f03d0 0x7f34b41f0850 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34b40b35f0 0x7f34b40b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 --2- 172.21.15.29:0/3268854250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f349c003a10 0x7f34b4159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.799+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 >> 172.21.15.29:0/3268854250 conn(0x7f34b40cb360 msgr2=0x7f34b415e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.800+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 shutdown_connections 2023-10-16T14:33:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:32.800+0000 7f34b9856640 1 -- 172.21.15.29:0/3268854250 wait complete. 2023-10-16T14:33:32.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:33:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.035+0000 7f4776521640 1 Processor -- start 2023-10-16T14:33:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.035+0000 7f4776521640 1 -- start start 2023-10-16T14:33:33.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.037+0000 7f4776521640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 0x7f47700cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.037+0000 7f4776521640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f47700ccfb0 0x7f47700cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.037+0000 7f4776521640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477015b9d0 con 0x7f477006c040 2023-10-16T14:33:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.037+0000 7f4776521640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477015bcb0 con 0x7f47700ccfb0 2023-10-16T14:33:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.037+0000 7f4776521640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477015bf90 con 0x7f47700ad970 2023-10-16T14:33:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.037+0000 7f477551f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f477006c040 0x7f477006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58090/0 (socket says 172.21.15.29:58090) 2023-10-16T14:33:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f477551f640 1 -- 172.21.15.29:0/628602026 learned_addr learned my addr 172.21.15.29:0/628602026 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4775d20640 1 --2- 172.21.15.29:0/628602026 >> v2:172.21.15.170:3300/0 conn(0x7f47700ccfb0 0x7f47700cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4774d1e640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 0x7f47700cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184672591 0 0) 0x7f477015b9d0 con 0x7f477006c040 2023-10-16T14:33:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f475c003520 con 0x7f477006c040 2023-10-16T14:33:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786036177 0 0) 0x7f475c003520 con 0x7f477006c040 2023-10-16T14:33:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 msgr2=0x7f47700cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 0x7f47700cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 >> v2:172.21.15.170:3300/0 conn(0x7f47700ccfb0 msgr2=0x7f47700cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 --2- 172.21.15.29:0/628602026 >> v2:172.21.15.170:3300/0 conn(0x7f47700ccfb0 0x7f47700cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47700cc270 con 0x7f477006c040 2023-10-16T14:33:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f476c001670 con 0x7f477006c040 2023-10-16T14:33:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 >> v1:172.21.15.29:6789/0 conn(0x7f477006c040 legacy=0x7f477006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.038+0000 7f4767fff640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 0x7f475c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4767fff640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475c003ef0 0x7f475c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4767fff640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f475c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477015bcb0 con 0x7f475c004d00 2023-10-16T14:33:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477015bf90 con 0x7f475c003ef0 2023-10-16T14:33:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475c00b9d0 con 0x7f47700ad970 2023-10-16T14:33:33.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4775d20640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f475c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f477551f640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 0x7f475c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4774d1e640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475c003ef0 0x7f475c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4775d20640 1 -- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 msgr2=0x7f475c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:33.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4775d20640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47700ad970 0x7f475c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4775d20640 1 -- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475c003ef0 msgr2=0x7f475c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:33.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4775d20640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f475c003ef0 0x7f475c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.039+0000 7f4775d20640 1 -- 172.21.15.29:0/628602026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c003520 con 0x7f475c004d00 2023-10-16T14:33:33.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.040+0000 7f4775d20640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f475c005180 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4768009fc0 tx=0x7f475c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be9cca0354b50039 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:33.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.041+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476802cc60 con 0x7f475c004d00 2023-10-16T14:33:33.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.041+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476800f040 con 0x7f475c004d00 2023-10-16T14:33:33.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.041+0000 7f4767fff640 1 -- 172.21.15.29:0/628602026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768034440 con 0x7f475c004d00 2023-10-16T14:33:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.041+0000 7f4776521640 1 -- 172.21.15.29:0/628602026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 msgr2=0x7f475c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.041+0000 7f4776521640 1 --2- 172.21.15.29:0/628602026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f475c005180 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4768009fc0 tx=0x7f475c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.041+0000 7f4775d20640 1 -- 172.21.15.29:0/628602026 reap_dead start 2023-10-16T14:33:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.042+0000 7f4776521640 1 -- 172.21.15.29:0/628602026 shutdown_connections 2023-10-16T14:33:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.042+0000 7f4776521640 1 -- 172.21.15.29:0/628602026 >> 172.21.15.29:0/628602026 conn(0x7f47700ce6a0 msgr2=0x7f47700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:33.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.042+0000 7f4776521640 1 -- 172.21.15.29:0/628602026 shutdown_connections 2023-10-16T14:33:33.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.042+0000 7f4776521640 1 -- 172.21.15.29:0/628602026 wait complete. 2023-10-16T14:33:33.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.043+0000 7f4776521640 1 Processor -- start 2023-10-16T14:33:33.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.044+0000 7f4776521640 1 -- start start 2023-10-16T14:33:33.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4776521640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f4770151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4776521640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4770153060 0x7f477006f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4776521640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4770150fd0 0x7f4770151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4776521640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4770156fc0 con 0x7f475c004d00 2023-10-16T14:33:33.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4776521640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4770152440 con 0x7f4770150fd0 2023-10-16T14:33:33.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4776521640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4770151970 con 0x7f4770153060 2023-10-16T14:33:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f4770151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f4770151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38034/0 (socket says 172.21.15.29:38034) 2023-10-16T14:33:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 -- 172.21.15.29:0/2311813632 learned_addr learned my addr 172.21.15.29:0/2311813632 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4774d1e640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4770153060 0x7f477006f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 -- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4770153060 msgr2=0x7f477006f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4770153060 0x7f477006f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 -- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4770150fd0 msgr2=0x7f4770151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f4775d20640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4770150fd0 0x7f4770151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4770150fd0 0x7f4770151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.045+0000 7f477551f640 1 -- 172.21.15.29:0/2311813632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770156690 con 0x7f475c004d00 2023-10-16T14:33:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.046+0000 7f477551f640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f4770151f00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f476c00e820 tx=0x7f476c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.046+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c00c8d0 con 0x7f475c004d00 2023-10-16T14:33:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.046+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476c00ca30 con 0x7f475c004d00 2023-10-16T14:33:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.046+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c00cdc0 con 0x7f475c004d00 2023-10-16T14:33:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.046+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4770156970 con 0x7f475c004d00 2023-10-16T14:33:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.046+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4770151b90 con 0x7f475c004d00 2023-10-16T14:33:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.047+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477004e9b0 con 0x7f475c004d00 2023-10-16T14:33:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.048+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f476c01d460 con 0x7f475c004d00 2023-10-16T14:33:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.049+0000 7f4765ffb640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f475406afc0 0x7f475406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.049+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6045+0+0 (secure 0 0 0) 0x7f476c089950 con 0x7f475c004d00 2023-10-16T14:33:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.050+0000 7f4774d1e640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f475406afc0 0x7f475406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:33.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.051+0000 7f4774d1e640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f475406afc0 0x7f475406d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f47700c94b0 tx=0x7f4760008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:33.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.054+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f476c054790 con 0x7f475c004d00 2023-10-16T14:33:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.252+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f477006b2b0 con 0x7f475c004d00 2023-10-16T14:33:33.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.799+0000 7f4765ffb640 1 -- 172.21.15.29:0/2311813632 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f476c057db0 con 0x7f475c004d00 2023-10-16T14:33:33.803 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:33:33.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f475406afc0 msgr2=0x7f475406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:33.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f475406afc0 0x7f475406d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f47700c94b0 tx=0x7f4760008040 comp rx=0 tx=0).stop 2023-10-16T14:33:33.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 msgr2=0x7f4770151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:33.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f4770151f00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f476c00e820 tx=0x7f476c00ece0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 shutdown_connections 2023-10-16T14:33:33.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f475406afc0 0x7f475406d480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4770150fd0 0x7f4770151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4770153060 0x7f477006f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 --2- 172.21.15.29:0/2311813632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475c004d00 0x7f4770151f00 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:33.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.803+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 >> 172.21.15.29:0/2311813632 conn(0x7f47700ce6a0 msgr2=0x7f47700cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:33.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.804+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 shutdown_connections 2023-10-16T14:33:33.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:33.805+0000 7f4776521640 1 -- 172.21.15.29:0/2311813632 wait complete. 2023-10-16T14:33:33.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:33:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.057+0000 7fbaa3516640 1 Processor -- start 2023-10-16T14:33:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.058+0000 7fbaa3516640 1 -- start start 2023-10-16T14:33:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa3516640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fba9c070830 0x7fba9c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa3516640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c06c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa3516640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba9c070c10 0x7fba9c0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa3516640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9c161740 con 0x7fba9c070c10 2023-10-16T14:33:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa3516640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9c06c680 con 0x7fba9c070830 2023-10-16T14:33:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa3516640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0cdbd0 con 0x7fba9c0cf5f0 2023-10-16T14:33:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa1ab3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba9c070c10 0x7fba9c0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa1ab3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba9c070c10 0x7fba9c0cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56910/0 (socket says 172.21.15.29:56910) 2023-10-16T14:33:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.059+0000 7fbaa1ab3640 1 -- 172.21.15.29:0/329727403 learned_addr learned my addr 172.21.15.29:0/329727403 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa0ab1640 1 --2- 172.21.15.29:0/329727403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c06c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa12b2640 1 --2- 172.21.15.29:0/329727403 >> v2:172.21.15.170:3300/0 conn(0x7fba9c070830 0x7fba9c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa1ab3640 1 -- 172.21.15.29:0/329727403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 msgr2=0x7fba9c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa1ab3640 1 --2- 172.21.15.29:0/329727403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c06c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa1ab3640 1 -- 172.21.15.29:0/329727403 >> v2:172.21.15.170:3300/0 conn(0x7fba9c070830 msgr2=0x7fba9c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa1ab3640 1 --2- 172.21.15.29:0/329727403 >> v2:172.21.15.170:3300/0 conn(0x7fba9c070830 0x7fba9c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa1ab3640 1 -- 172.21.15.29:0/329727403 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c06b0f0 con 0x7fba9c070c10 2023-10-16T14:33:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.060+0000 7fbaa12b2640 1 --2- 172.21.15.29:0/329727403 >> v2:172.21.15.170:3300/0 conn(0x7fba9c070830 0x7fba9c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.061+0000 7fbaa1ab3640 1 --2- 172.21.15.29:0/329727403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070c10 0x7fba9c0cd690 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fba9c06c010 tx=0x7fba9402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34030e74997b08a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.061+0000 7fba93fff640 1 -- 172.21.15.29:0/329727403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9402c950 con 0x7fba9c070c10 2023-10-16T14:33:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.061+0000 7fba93fff640 1 -- 172.21.15.29:0/329727403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9402cab0 con 0x7fba9c070c10 2023-10-16T14:33:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.061+0000 7fba93fff640 1 -- 172.21.15.29:0/329727403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9402ce40 con 0x7fba9c070c10 2023-10-16T14:33:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 -- 172.21.15.29:0/329727403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070c10 msgr2=0x7fba9c0cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 --2- 172.21.15.29:0/329727403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070c10 0x7fba9c0cd690 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fba9c06c010 tx=0x7fba9402bdb0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 -- 172.21.15.29:0/329727403 shutdown_connections 2023-10-16T14:33:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 --2- 172.21.15.29:0/329727403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070c10 0x7fba9c0cd690 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 --2- 172.21.15.29:0/329727403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c06c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 --2- 172.21.15.29:0/329727403 >> v2:172.21.15.170:3300/0 conn(0x7fba9c070830 0x7fba9c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 -- 172.21.15.29:0/329727403 >> 172.21.15.29:0/329727403 conn(0x7fba9c0cfac0 msgr2=0x7fba9c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 -- 172.21.15.29:0/329727403 shutdown_connections 2023-10-16T14:33:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.062+0000 7fbaa3516640 1 -- 172.21.15.29:0/329727403 wait complete. 2023-10-16T14:33:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.064+0000 7fbaa3516640 1 Processor -- start 2023-10-16T14:33:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.064+0000 7fbaa3516640 1 -- start start 2023-10-16T14:33:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa3516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 0x7fba9c0c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa3516640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba9c070c10 0x7fba9c14bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa3516640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c0c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa3516640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c14c4d0 con 0x7fba9c070830 2023-10-16T14:33:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa3516640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0c8720 con 0x7fba9c070c10 2023-10-16T14:33:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa3516640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0c3880 con 0x7fba9c0cf5f0 2023-10-16T14:33:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa12b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 0x7fba9c0c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa12b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 0x7fba9c0c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56916/0 (socket says 172.21.15.29:56916) 2023-10-16T14:33:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.065+0000 7fbaa12b2640 1 -- 172.21.15.29:0/2199653216 learned_addr learned my addr 172.21.15.29:0/2199653216 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa0ab1640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba9c070c10 0x7fba9c14bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa12b2640 1 -- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 msgr2=0x7fba9c0c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa1ab3640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c0c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa12b2640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c0c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa12b2640 1 -- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba9c070c10 msgr2=0x7fba9c14bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa12b2640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba9c070c10 0x7fba9c14bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa12b2640 1 -- 172.21.15.29:0/2199653216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba94009b30 con 0x7fba9c070830 2023-10-16T14:33:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa0ab1640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba9c070c10 0x7fba9c14bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa12b2640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 0x7fba9c0c3f30 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c7f0 tx=0x7fba9800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98007820 con 0x7fba9c070830 2023-10-16T14:33:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98007980 con 0x7fba9c070830 2023-10-16T14:33:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98007d30 con 0x7fba9c070830 2023-10-16T14:33:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.066+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c0c4510 con 0x7fba9c070830 2023-10-16T14:33:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.067+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c0c89c0 con 0x7fba9c070830 2023-10-16T14:33:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.068+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c0c8bb0 con 0x7fba9c070830 2023-10-16T14:33:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.069+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fba980113f0 con 0x7fba9c070830 2023-10-16T14:33:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.070+0000 7fba91ffb640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fba8006af90 0x7fba8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.070+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fba98089b80 con 0x7fba9c070830 2023-10-16T14:33:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.070+0000 7fbaa0ab1640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fba8006af90 0x7fba8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.072+0000 7fbaa0ab1640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fba8006af90 0x7fba8006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba8c006fd0 tx=0x7fba8c008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.074+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fba980549d0 con 0x7fba9c070830 2023-10-16T14:33:34.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.269+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fba9c04e160 con 0x7fba9c070830 2023-10-16T14:33:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.275+0000 7fba91ffb640 1 -- 172.21.15.29:0/2199653216 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fba98057ff0 con 0x7fba9c070830 2023-10-16T14:33:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fba8006af90 msgr2=0x7fba8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fba8006af90 0x7fba8006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba8c006fd0 tx=0x7fba8c008040 comp rx=0 tx=0).stop 2023-10-16T14:33:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 msgr2=0x7fba9c0c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 0x7fba9c0c3f30 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c7f0 tx=0x7fba9800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 shutdown_connections 2023-10-16T14:33:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fba8006af90 0x7fba8006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba9c0cf5f0 0x7fba9c0c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba9c070c10 0x7fba9c14bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 --2- 172.21.15.29:0/2199653216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba9c070830 0x7fba9c0c3f30 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 >> 172.21.15.29:0/2199653216 conn(0x7fba9c0cfac0 msgr2=0x7fba9c0ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 shutdown_connections 2023-10-16T14:33:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.279+0000 7fbaa3516640 1 -- 172.21.15.29:0/2199653216 wait complete. 2023-10-16T14:33:34.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:33:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.517+0000 7f1597ae5640 1 Processor -- start 2023-10-16T14:33:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.517+0000 7f1597ae5640 1 -- start start 2023-10-16T14:33:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.518+0000 7f1597ae5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15901615d0 0x7f159015db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f1597ae5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159015e3c0 con 0x7f15900ccfc0 2023-10-16T14:33:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f1597ae5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159015e0e0 con 0x7f15900b0b20 2023-10-16T14:33:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f1597ae5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1590160af0 con 0x7f15901615d0 2023-10-16T14:33:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f1595881640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f15900ccfc0 0x7f15900b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38546/0 (socket says 172.21.15.29:38546) 2023-10-16T14:33:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f1595881640 1 -- 172.21.15.29:0/2940775161 learned_addr learned my addr 172.21.15.29:0/2940775161 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492531103 0 0) 0x7f159015e3c0 con 0x7f15900ccfc0 2023-10-16T14:33:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f1596082640 1 --2- 172.21.15.29:0/2940775161 >> v2:172.21.15.170:3300/0 conn(0x7f15901615d0 0x7f159015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.519+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157c003520 con 0x7f15900ccfc0 2023-10-16T14:33:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 398264863 0 0) 0x7f157c003520 con 0x7f15900ccfc0 2023-10-16T14:33:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 >> v2:172.21.15.170:3300/0 conn(0x7f15901615d0 msgr2=0x7f159015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 --2- 172.21.15.29:0/2940775161 >> v2:172.21.15.170:3300/0 conn(0x7f15901615d0 0x7f159015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 >> v1:172.21.15.170:6789/0 conn(0x7f15900b0b20 legacy=0x7f159015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590160dd0 con 0x7f15900ccfc0 2023-10-16T14:33:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f158c001670 con 0x7f15900ccfc0 2023-10-16T14:33:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 >> v1:172.21.15.29:6789/0 conn(0x7f15900ccfc0 legacy=0x7f15900b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f157c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f157c0047c0 0x7f157c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f157c0051d0 0x7f157c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1590160af0 con 0x7f157c0039c0 2023-10-16T14:33:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c00c120 con 0x7f157c0051d0 2023-10-16T14:33:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c00c2e0 con 0x7f157c0047c0 2023-10-16T14:33:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.520+0000 7f1595881640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f157c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595080640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f157c0047c0 0x7f157c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1596082640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f157c0051d0 0x7f157c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595881640 1 -- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f157c0047c0 msgr2=0x7f157c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595881640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f157c0047c0 0x7f157c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595881640 1 -- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f157c0051d0 msgr2=0x7f157c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595881640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f157c0051d0 0x7f157c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595881640 1 -- 172.21.15.29:0/2940775161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c003520 con 0x7f157c0039c0 2023-10-16T14:33:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1595080640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f157c0047c0 0x7f157c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1596082640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f157c0051d0 0x7f157c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.521+0000 7f1596082640 1 -- 172.21.15.29:0/2940775161 reap_dead start 2023-10-16T14:33:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.522+0000 7f1595881640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f157c003de0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f158c001eb0 tx=0x7f157c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9098e2ac09be1d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.522+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c005ad0 con 0x7f157c0039c0 2023-10-16T14:33:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.522+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c005c30 con 0x7f157c0039c0 2023-10-16T14:33:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.522+0000 7f159487f640 1 -- 172.21.15.29:0/2940775161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c02c9e0 con 0x7f157c0039c0 2023-10-16T14:33:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 -- 172.21.15.29:0/2940775161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 msgr2=0x7f157c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f157c003de0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f158c001eb0 tx=0x7f157c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 -- 172.21.15.29:0/2940775161 shutdown_connections 2023-10-16T14:33:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 --2- 172.21.15.29:0/2940775161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f157c003de0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 -- 172.21.15.29:0/2940775161 >> 172.21.15.29:0/2940775161 conn(0x7f15900cb410 msgr2=0x7f15900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 -- 172.21.15.29:0/2940775161 shutdown_connections 2023-10-16T14:33:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.523+0000 7f1597ae5640 1 -- 172.21.15.29:0/2940775161 wait complete. 2023-10-16T14:33:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.525+0000 7f1597ae5640 1 Processor -- start 2023-10-16T14:33:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.525+0000 7f1597ae5640 1 -- start start 2023-10-16T14:33:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1597ae5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f15901514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1597ae5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006f450 0x7f159006f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1597ae5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15900b37b0 0x7f15900b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1597ae5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159006fe10 con 0x7f157c0039c0 2023-10-16T14:33:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1597ae5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c3bc0 con 0x7f15900b37b0 2023-10-16T14:33:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1597ae5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c3ea0 con 0x7f159006f450 2023-10-16T14:33:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1595881640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f15901514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1595881640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f15901514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56932/0 (socket says 172.21.15.29:56932) 2023-10-16T14:33:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1595881640 1 -- 172.21.15.29:0/4167987785 learned_addr learned my addr 172.21.15.29:0/4167987785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.527+0000 7f1595080640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006f450 0x7f159006f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.528+0000 7f1596082640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15900b37b0 0x7f15900b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.528+0000 7f1595881640 1 -- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006f450 msgr2=0x7f159006f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.528+0000 7f1595881640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006f450 0x7f159006f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.528+0000 7f1595881640 1 -- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15900b37b0 msgr2=0x7f15900b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.528+0000 7f1595881640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15900b37b0 0x7f15900b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.528+0000 7f1595881640 1 -- 172.21.15.29:0/4167987785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15900c4180 con 0x7f157c0039c0 2023-10-16T14:33:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.529+0000 7f1595881640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f15901514d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f158c000c00 tx=0x7f158c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.530+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c03c040 con 0x7f157c0039c0 2023-10-16T14:33:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.530+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c02cd10 con 0x7f157c0039c0 2023-10-16T14:33:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.530+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c0072b0 con 0x7f157c0039c0 2023-10-16T14:33:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.530+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15900c4460 con 0x7f157c0039c0 2023-10-16T14:33:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.530+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15900b41a0 con 0x7f157c0039c0 2023-10-16T14:33:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.532+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15900c4980 con 0x7f157c0039c0 2023-10-16T14:33:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.533+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f158c032070 con 0x7f157c0039c0 2023-10-16T14:33:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.535+0000 7f15867fc640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f157406af90 0x7f157406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.535+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f158c0aaac0 con 0x7f157c0039c0 2023-10-16T14:33:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.535+0000 7f1595080640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f157406af90 0x7f157406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:34.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.537+0000 7f1595080640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f157406af90 0x7f157406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15900b3660 tx=0x7f1580009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:34.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.539+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f15900c4980 con 0x7f157c0039c0 2023-10-16T14:33:34.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.738+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f15900c4980 con 0x7f157c0039c0 2023-10-16T14:33:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.745+0000 7f15867fc640 1 -- 172.21.15.29:0/4167987785 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7f15900c4980 con 0x7f157c0039c0 2023-10-16T14:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f157406af90 msgr2=0x7f157406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f157406af90 0x7f157406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15900b3660 tx=0x7f1580009040 comp rx=0 tx=0).stop 2023-10-16T14:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 msgr2=0x7f15901514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f15901514d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f158c000c00 tx=0x7f158c02e070 comp rx=0 tx=0).stop 2023-10-16T14:33:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 shutdown_connections 2023-10-16T14:33:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f157406af90 0x7f157406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f15900b37b0 0x7f15900b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006f450 0x7f159006f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 --2- 172.21.15.29:0/4167987785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f157c0039c0 0x7f15901514d0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.749+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 >> 172.21.15.29:0/4167987785 conn(0x7f15900cb410 msgr2=0x7f159006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.750+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 shutdown_connections 2023-10-16T14:33:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:34.750+0000 7f1597ae5640 1 -- 172.21.15.29:0/4167987785 wait complete. 2023-10-16T14:33:34.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:33:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.003+0000 7f9307c39640 1 Processor -- start 2023-10-16T14:33:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.003+0000 7f9307c39640 1 -- start start 2023-10-16T14:33:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f9307c39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9300159e90 0x7f93000ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f9307c39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f930006c140 0x7f930006c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f9307c39640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930006cad0 con 0x7f9300159ab0 2023-10-16T14:33:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f9307c39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930015d890 con 0x7f930006c140 2023-10-16T14:33:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f9307c39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9300070360 con 0x7f9300159e90 2023-10-16T14:33:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f93061d6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f930006c140 0x7f930006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f93061d6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f930006c140 0x7f930006c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56946/0 (socket says 172.21.15.29:56946) 2023-10-16T14:33:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f93061d6640 1 -- 172.21.15.29:0/2734722704 learned_addr learned my addr 172.21.15.29:0/2734722704 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f93061d6640 1 -- 172.21.15.29:0/2734722704 >> v2:172.21.15.170:3300/0 conn(0x7f9300159e90 msgr2=0x7f93000ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f93061d6640 1 --2- 172.21.15.29:0/2734722704 >> v2:172.21.15.170:3300/0 conn(0x7f9300159e90 0x7f93000ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.005+0000 7f93061d6640 1 -- 172.21.15.29:0/2734722704 >> v1:172.21.15.170:6789/0 conn(0x7f9300159ab0 legacy=0x7f930006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.006+0000 7f93061d6640 1 -- 172.21.15.29:0/2734722704 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93000cd1f0 con 0x7f930006c140 2023-10-16T14:33:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.007+0000 7f93061d6640 1 --2- 172.21.15.29:0/2734722704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930006c140 0x7f930006c590 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f930006d900 tx=0x7f92f802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac499220730d9ccf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.007+0000 7f93049d3640 1 -- 172.21.15.29:0/2734722704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f802c960 con 0x7f930006c140 2023-10-16T14:33:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.007+0000 7f93049d3640 1 -- 172.21.15.29:0/2734722704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f802cac0 con 0x7f930006c140 2023-10-16T14:33:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.007+0000 7f93049d3640 1 -- 172.21.15.29:0/2734722704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f802ce50 con 0x7f930006c140 2023-10-16T14:33:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 -- 172.21.15.29:0/2734722704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930006c140 msgr2=0x7f930006c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 --2- 172.21.15.29:0/2734722704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930006c140 0x7f930006c590 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f930006d900 tx=0x7f92f802bdc0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 -- 172.21.15.29:0/2734722704 shutdown_connections 2023-10-16T14:33:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 --2- 172.21.15.29:0/2734722704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930006c140 0x7f930006c590 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 --2- 172.21.15.29:0/2734722704 >> v2:172.21.15.170:3300/0 conn(0x7f9300159e90 0x7f93000ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 -- 172.21.15.29:0/2734722704 >> 172.21.15.29:0/2734722704 conn(0x7f93000ce9b0 msgr2=0x7f93000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 -- 172.21.15.29:0/2734722704 shutdown_connections 2023-10-16T14:33:35.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.008+0000 7f9307c39640 1 -- 172.21.15.29:0/2734722704 wait complete. 2023-10-16T14:33:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.010+0000 7f9307c39640 1 Processor -- start 2023-10-16T14:33:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.010+0000 7f9307c39640 1 -- start start 2023-10-16T14:33:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f9307c39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f930006c140 0x7f93001f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f9307c39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300159ab0 0x7f930015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f9307c39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 0x7f93000ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f9307c39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93000b3590 con 0x7f9300159e90 2023-10-16T14:33:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f9307c39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93001f3880 con 0x7f9300159ab0 2023-10-16T14:33:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f9307c39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93000b1b90 con 0x7f930006c140 2023-10-16T14:33:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f93061d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 0x7f93000ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f93061d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 0x7f93000ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56960/0 (socket says 172.21.15.29:56960) 2023-10-16T14:33:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.011+0000 7f93061d6640 1 -- 172.21.15.29:0/1247162363 learned_addr learned my addr 172.21.15.29:0/1247162363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93059d5640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f930006c140 0x7f93001f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93061d6640 1 -- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f930006c140 msgr2=0x7f93001f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93061d6640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f930006c140 0x7f93001f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93061d6640 1 -- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300159ab0 msgr2=0x7f930015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93061d6640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300159ab0 0x7f930015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93061d6640 1 -- 172.21.15.29:0/1247162363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f8009b30 con 0x7f9300159e90 2023-10-16T14:33:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93059d5640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f930006c140 0x7f93001f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f93061d6640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 0x7f93000ac890 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f92f8036040 tx=0x7f92f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f803e450 con 0x7f9300159e90 2023-10-16T14:33:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f803e5b0 con 0x7f9300159e90 2023-10-16T14:33:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f803e960 con 0x7f9300159e90 2023-10-16T14:33:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.012+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93000b1e70 con 0x7f9300159e90 2023-10-16T14:33:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.013+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93000b2430 con 0x7f9300159e90 2023-10-16T14:33:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.014+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930004e160 con 0x7f9300159e90 2023-10-16T14:33:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.015+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f92f803eac0 con 0x7f9300159e90 2023-10-16T14:33:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.016+0000 7f92f67fc640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f92e406af90 0x7f92e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.016+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f92f80ac4c0 con 0x7f9300159e90 2023-10-16T14:33:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.016+0000 7f93059d5640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f92e406af90 0x7f92e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.018+0000 7f93059d5640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f92e406af90 0x7f92e406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00a810 tx=0x7f92fc008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.020+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f92f80774b0 con 0x7f9300159e90 2023-10-16T14:33:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.214+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f930006cad0 con 0x7f9300159e90 2023-10-16T14:33:35.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.221+0000 7f92f67fc640 1 -- 172.21.15.29:0/1247162363 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7f930006cad0 con 0x7f9300159e90 2023-10-16T14:33:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f92e406af90 msgr2=0x7f92e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f92e406af90 0x7f92e406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00a810 tx=0x7f92fc008040 comp rx=0 tx=0).stop 2023-10-16T14:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 msgr2=0x7f93000ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 0x7f93000ac890 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f92f8036040 tx=0x7f92f8004a40 comp rx=0 tx=0).stop 2023-10-16T14:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 shutdown_connections 2023-10-16T14:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f92e406af90 0x7f92e406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9300159e90 0x7f93000ac890 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9300159ab0 0x7f930015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 --2- 172.21.15.29:0/1247162363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f930006c140 0x7f93001f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.224+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 >> 172.21.15.29:0/1247162363 conn(0x7f93000ce9b0 msgr2=0x7f9300159010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.225+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 shutdown_connections 2023-10-16T14:33:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.225+0000 7f9307c39640 1 -- 172.21.15.29:0/1247162363 wait complete. 2023-10-16T14:33:35.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T14:33:35.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.460+0000 7fe6cf59e640 1 Processor -- start 2023-10-16T14:33:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.461+0000 7fe6cf59e640 1 -- start start 2023-10-16T14:33:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.462+0000 7fe6cf59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe6d006c040 0x7fe6d006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.462+0000 7fe6cf59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6d00ad970 0x7fe6d015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.462+0000 7fe6cf59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d006c960 con 0x7fe6d015ebc0 2023-10-16T14:33:35.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.462+0000 7fe6cf59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d015b9d0 con 0x7fe6d00ad970 2023-10-16T14:33:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.462+0000 7fe6cf59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d015bcb0 con 0x7fe6d006c040 2023-10-16T14:33:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6d00ad970 0x7fe6d015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6d00ad970 0x7fe6d015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56962/0 (socket says 172.21.15.29:56962) 2023-10-16T14:33:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6ced9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe6d015ebc0 0x7fe6d00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38556/0 (socket says 172.21.15.29:38556) 2023-10-16T14:33:35.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 -- 172.21.15.29:0/2052619035 learned_addr learned my addr 172.21.15.29:0/2052619035 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:35.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6ce59c640 1 --2- 172.21.15.29:0/2052619035 >> v2:172.21.15.170:3300/0 conn(0x7fe6d006c040 0x7fe6d006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 -- 172.21.15.29:0/2052619035 >> v2:172.21.15.170:3300/0 conn(0x7fe6d006c040 msgr2=0x7fe6d006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/2052619035 >> v2:172.21.15.170:3300/0 conn(0x7fe6d006c040 0x7fe6d006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 -- 172.21.15.29:0/2052619035 >> v1:172.21.15.29:6789/0 conn(0x7fe6d015ebc0 legacy=0x7fe6d00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6c7fff640 1 -- 172.21.15.29:0/2052619035 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d015bf90 con 0x7fe6d00ad970 2023-10-16T14:33:35.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.463+0000 7fe6cdd9b640 1 -- 172.21.15.29:0/2052619035 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000822222 0 0) 0x7fe6d006c960 con 0x7fe6d015ebc0 2023-10-16T14:33:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.465+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/2052619035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d015e680 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00cf350 tx=0x7fe6b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45158b2ffeaa704 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.465+0000 7fe6cdd9b640 1 -- 172.21.15.29:0/2052619035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8030860 con 0x7fe6d00ad970 2023-10-16T14:33:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.465+0000 7fe6cdd9b640 1 -- 172.21.15.29:0/2052619035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b80309c0 con 0x7fe6d00ad970 2023-10-16T14:33:35.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.465+0000 7fe6cdd9b640 1 -- 172.21.15.29:0/2052619035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8030d50 con 0x7fe6d00ad970 2023-10-16T14:33:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.465+0000 7fe6cf59e640 1 -- 172.21.15.29:0/2052619035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 msgr2=0x7fe6d015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.465+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/2052619035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d015e680 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00cf350 tx=0x7fe6b802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.466+0000 7fe6cf59e640 1 -- 172.21.15.29:0/2052619035 shutdown_connections 2023-10-16T14:33:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.466+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/2052619035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d015e680 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.466+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/2052619035 >> v2:172.21.15.170:3300/0 conn(0x7fe6d006c040 0x7fe6d006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.466+0000 7fe6cf59e640 1 -- 172.21.15.29:0/2052619035 >> 172.21.15.29:0/2052619035 conn(0x7fe6d00ce6a0 msgr2=0x7fe6d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.466+0000 7fe6cf59e640 1 -- 172.21.15.29:0/2052619035 shutdown_connections 2023-10-16T14:33:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.466+0000 7fe6cf59e640 1 -- 172.21.15.29:0/2052619035 wait complete. 2023-10-16T14:33:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.468+0000 7fe6cf59e640 1 Processor -- start 2023-10-16T14:33:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.468+0000 7fe6cf59e640 1 -- start start 2023-10-16T14:33:35.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6cf59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006c040 0x7fe6d0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6cf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d00afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6cf59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d015ebc0 0x7fe6d01f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6cf59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01f7260 con 0x7fe6d00ad970 2023-10-16T14:33:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6cf59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00ae360 con 0x7fe6d015ebc0 2023-10-16T14:33:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6cf59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00aff80 con 0x7fe6d006c040 2023-10-16T14:33:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d00afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d00afa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56964/0 (socket says 172.21.15.29:56964) 2023-10-16T14:33:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6c7fff640 1 -- 172.21.15.29:0/3712526715 learned_addr learned my addr 172.21.15.29:0/3712526715 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.469+0000 7fe6ced9d640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d015ebc0 0x7fe6d01f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6ce59c640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006c040 0x7fe6d0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c7fff640 1 -- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006c040 msgr2=0x7fe6d0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006c040 0x7fe6d0156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c7fff640 1 -- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d015ebc0 msgr2=0x7fe6d01f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d015ebc0 0x7fe6d01f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c7fff640 1 -- 172.21.15.29:0/3712526715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80094d0 con 0x7fe6d00ad970 2023-10-16T14:33:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6ced9d640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d015ebc0 0x7fe6d01f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d00afa40 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8000c00 tx=0x7fe6b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8039b60 con 0x7fe6d00ad970 2023-10-16T14:33:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8039ce0 con 0x7fe6d00ad970 2023-10-16T14:33:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8043650 con 0x7fe6d00ad970 2023-10-16T14:33:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.470+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d01f6ca0 con 0x7fe6d00ad970 2023-10-16T14:33:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.471+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d00adf80 con 0x7fe6d00ad970 2023-10-16T14:33:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.472+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d006b330 con 0x7fe6d00ad970 2023-10-16T14:33:35.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.473+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe6b804c3b0 con 0x7fe6d00ad970 2023-10-16T14:33:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.473+0000 7fe6c6ffd640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fe6b006af90 0x7fe6b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:35.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.474+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe6b80abf90 con 0x7fe6d00ad970 2023-10-16T14:33:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.474+0000 7fe6ce59c640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fe6b006af90 0x7fe6b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.476+0000 7fe6ce59c640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fe6b006af90 0x7fe6b006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0006fd0 tx=0x7fe6c0008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.478+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe6b8076dd0 con 0x7fe6d00ad970 2023-10-16T14:33:35.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:35.740+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fe6d00adde0 con 0x7fe6d00ad970 2023-10-16T14:33:36.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.213+0000 7fe6c6ffd640 1 -- 172.21.15.29:0/3712526715 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v643) v1 ==== 166+0+0 (secure 0 0 0) 0x7fe6d00adde0 con 0x7fe6d00ad970 2023-10-16T14:33:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fe6b006af90 msgr2=0x7fe6b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fe6b006af90 0x7fe6b006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0006fd0 tx=0x7fe6c0008040 comp rx=0 tx=0).stop 2023-10-16T14:33:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 msgr2=0x7fe6d00afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d00afa40 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8000c00 tx=0x7fe6b8004730 comp rx=0 tx=0).stop 2023-10-16T14:33:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 shutdown_connections 2023-10-16T14:33:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fe6b006af90 0x7fe6b006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d00ad970 0x7fe6d00afa40 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6d015ebc0 0x7fe6d01f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 --2- 172.21.15.29:0/3712526715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6d006c040 0x7fe6d0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.216+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 >> 172.21.15.29:0/3712526715 conn(0x7fe6d00ce6a0 msgr2=0x7fe6d00c9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.217+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 shutdown_connections 2023-10-16T14:33:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.217+0000 7fe6cf59e640 1 -- 172.21.15.29:0/3712526715 wait complete. 2023-10-16T14:33:36.243 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-16T14:33:36.244 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-16T14:33:36.244 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:33:36.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:36.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.463+0000 7f28bc92b640 1 Processor -- start 2023-10-16T14:33:36.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.464+0000 7f28bc92b640 1 -- start start 2023-10-16T14:33:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.466+0000 7f28bc92b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f28b815cd70 0x7f28b80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.466+0000 7f28bc92b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b815d150 con 0x7f28b80b8480 2023-10-16T14:33:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.466+0000 7f28bc92b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b80d1180 con 0x7f28b80cded0 2023-10-16T14:33:36.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.466+0000 7f28bc92b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b80d1460 con 0x7f28b815cd70 2023-10-16T14:33:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.466+0000 7f28affff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f28b80b8480 0x7f28b80cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38572/0 (socket says 172.21.15.29:38572) 2023-10-16T14:33:36.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.466+0000 7f28affff640 1 -- 172.21.15.29:0/1555455252 learned_addr learned my addr 172.21.15.29:0/1555455252 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952653713 0 0) 0x7f28b815d150 con 0x7f28b80b8480 2023-10-16T14:33:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a4003520 con 0x7f28b80b8480 2023-10-16T14:33:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b6d9d640 1 --2- 172.21.15.29:0/1555455252 >> v2:172.21.15.170:3300/0 conn(0x7f28b815cd70 0x7f28b80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348525828 0 0) 0x7f28a4003520 con 0x7f28b80b8480 2023-10-16T14:33:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 >> v2:172.21.15.170:3300/0 conn(0x7f28b815cd70 msgr2=0x7f28b80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 --2- 172.21.15.29:0/1555455252 >> v2:172.21.15.170:3300/0 conn(0x7f28b815cd70 0x7f28b80d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 >> v1:172.21.15.170:6789/0 conn(0x7f28b80cded0 legacy=0x7f28b80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b80d1740 con 0x7f28b80b8480 2023-10-16T14:33:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f28a0001670 con 0x7f28b80b8480 2023-10-16T14:33:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 >> v1:172.21.15.29:6789/0 conn(0x7f28b80b8480 legacy=0x7f28b80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40039c0 0x7f28a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40047c0 0x7f28a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40051d0 0x7f28a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80d1460 con 0x7f28a40047c0 2023-10-16T14:33:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a400c100 con 0x7f28a40051d0 2023-10-16T14:33:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a400c2c0 con 0x7f28a40039c0 2023-10-16T14:33:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.467+0000 7f28affff640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40047c0 0x7f28a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28b6d9d640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40039c0 0x7f28a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28b759e640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40051d0 0x7f28a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28affff640 1 -- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40039c0 msgr2=0x7f28a4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28affff640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28a40039c0 0x7f28a4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28affff640 1 -- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40051d0 msgr2=0x7f28a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28affff640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40051d0 0x7f28a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28affff640 1 -- 172.21.15.29:0/1555455252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a4003520 con 0x7f28a40047c0 2023-10-16T14:33:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28b759e640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40051d0 0x7f28a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.468+0000 7f28b759e640 1 -- 172.21.15.29:0/1555455252 reap_dead start 2023-10-16T14:33:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.469+0000 7f28affff640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40047c0 0x7f28a4004c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c160 tx=0x7f28a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a585c4b34c76821 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.469+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0005ad0 con 0x7f28a40047c0 2023-10-16T14:33:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.469+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a0005c30 con 0x7f28a40047c0 2023-10-16T14:33:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.469+0000 7f28b659c640 1 -- 172.21.15.29:0/1555455252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a002c7c0 con 0x7f28a40047c0 2023-10-16T14:33:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 -- 172.21.15.29:0/1555455252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40047c0 msgr2=0x7f28a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40047c0 0x7f28a4004c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c160 tx=0x7f28a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 -- 172.21.15.29:0/1555455252 shutdown_connections 2023-10-16T14:33:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 --2- 172.21.15.29:0/1555455252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28a40047c0 0x7f28a4004c40 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 -- 172.21.15.29:0/1555455252 >> 172.21.15.29:0/1555455252 conn(0x7f28b80cf570 msgr2=0x7f28b80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 -- 172.21.15.29:0/1555455252 shutdown_connections 2023-10-16T14:33:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.470+0000 7f28bc92b640 1 -- 172.21.15.29:0/1555455252 wait complete. 2023-10-16T14:33:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.472+0000 7f28bc92b640 1 Processor -- start 2023-10-16T14:33:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.472+0000 7f28bc92b640 1 -- start start 2023-10-16T14:33:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28bc92b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40047c0 0x7f28b80a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28bc92b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 0x7f28b80b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28bc92b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b814cb30 0x7f28b80b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28bc92b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b815f370 con 0x7f28b814c6b0 2023-10-16T14:33:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28bc92b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b814d630 con 0x7f28a40047c0 2023-10-16T14:33:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28bc92b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b814d070 con 0x7f28b814cb30 2023-10-16T14:33:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 0x7f28b80b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 0x7f28b80b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56980/0 (socket says 172.21.15.29:56980) 2023-10-16T14:33:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.473+0000 7f28affff640 1 -- 172.21.15.29:0/2248348173 learned_addr learned my addr 172.21.15.29:0/2248348173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28b6d9d640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40047c0 0x7f28b80a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28b759e640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b814cb30 0x7f28b80b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28affff640 1 -- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b814cb30 msgr2=0x7f28b80b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28affff640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b814cb30 0x7f28b80b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28affff640 1 -- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40047c0 msgr2=0x7f28b80a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28affff640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40047c0 0x7f28b80a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28affff640 1 -- 172.21.15.29:0/2248348173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b814d350 con 0x7f28b814c6b0 2023-10-16T14:33:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28b6d9d640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40047c0 0x7f28b80a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28b759e640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b814cb30 0x7f28b80b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.474+0000 7f28affff640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 0x7f28b80b7410 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c160 tx=0x7f28a002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.475+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a003c040 con 0x7f28b814c6b0 2023-10-16T14:33:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.475+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a002ccb0 con 0x7f28b814c6b0 2023-10-16T14:33:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.475+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a00072b0 con 0x7f28b814c6b0 2023-10-16T14:33:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.475+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b815ef90 con 0x7f28b814c6b0 2023-10-16T14:33:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.475+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b81f59e0 con 0x7f28b814c6b0 2023-10-16T14:33:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.476+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b804e9b0 con 0x7f28b814c6b0 2023-10-16T14:33:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.477+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f28a0032070 con 0x7f28b814c6b0 2023-10-16T14:33:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.478+0000 7f28af7fe640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f289806afc0 0x7f289806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.478+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f28a00aac90 con 0x7f28b814c6b0 2023-10-16T14:33:36.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.479+0000 7f28b6d9d640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f289806afc0 0x7f289806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.480+0000 7f28b6d9d640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f289806afc0 0x7f289806d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f28a8006fd0 tx=0x7f28a8008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:36.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.483+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28a0075e40 con 0x7f28b814c6b0 2023-10-16T14:33:36.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.677+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28b80cec70 con 0x7f28b814c6b0 2023-10-16T14:33:36.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.679+0000 7f28af7fe640 1 -- 172.21.15.29:0/2248348173 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+14002 (secure 0 0 0) 0x7f28a0046020 con 0x7f28b814c6b0 2023-10-16T14:33:36.683 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:36.683 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":102,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:33.796611+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6836/3812375996":"2023-10-17T14:33:30.779288+0000","172.21.15.170:0/136212456":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/3176118466":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/2083683232":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/1621614164":"2023-10-17T14:33:26.572781+0000","172.21.15.170:6833/3288786257":"2023-10-17T14:33:26.572781+0000","172.21.15.170:6832/3288786257":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/2674280726":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/434807034":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/503917561":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/661241058":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/565742787":"2023-10-17T14:33:20.443300+0000","172.21.15.170:0/1207268771":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6833/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6832/610139797":"2023-10-17T14:33:20.443300+0000","172.21.15.170:6837/3812375996":"2023-10-17T14:33:30.779288+0000","172.21.15.170:0/1288542658":"2023-10-17T14:33:26.572781+0000","172.21.15.170:0/329350515":"2023-10-17T14:33:20.443300+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f289806afc0 msgr2=0x7f289806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f289806afc0 0x7f289806d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f28a8006fd0 tx=0x7f28a8008040 comp rx=0 tx=0).stop 2023-10-16T14:33:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 msgr2=0x7f28b80b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 0x7f28b80b7410 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c160 tx=0x7f28a002e070 comp rx=0 tx=0).stop 2023-10-16T14:33:36.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 shutdown_connections 2023-10-16T14:33:36.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f289806afc0 0x7f289806d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b814c6b0 0x7f28b80b7410 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b814cb30 0x7f28b80b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 --2- 172.21.15.29:0/2248348173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28a40047c0 0x7f28b80a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 >> 172.21.15.29:0/2248348173 conn(0x7f28b80cf570 msgr2=0x7f28b80b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:36.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 shutdown_connections 2023-10-16T14:33:36.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.682+0000 7f28bc92b640 1 -- 172.21.15.29:0/2248348173 wait complete. 2023-10-16T14:33:36.706 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:33:36.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:33:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.932+0000 7f4272b75640 1 Processor -- start 2023-10-16T14:33:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.932+0000 7f4272b75640 1 -- start start 2023-10-16T14:33:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.934+0000 7f4272b75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 0x7f426c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.934+0000 7f4272b75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c15d0e0 con 0x7f426c15cd00 2023-10-16T14:33:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.934+0000 7f4272b75640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c0cbd80 con 0x7f426c0cb000 2023-10-16T14:33:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.934+0000 7f4272b75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0cbaa0 con 0x7f426c070a90 2023-10-16T14:33:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.934+0000 7f4271b73640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f426c15cd00 0x7f426c0cdfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38584/0 (socket says 172.21.15.29:38584) 2023-10-16T14:33:36.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.934+0000 7f4271b73640 1 -- 172.21.15.29:0/3093821901 learned_addr learned my addr 172.21.15.29:0/3093821901 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582784353 0 0) 0x7f426c15d0e0 con 0x7f426c15cd00 2023-10-16T14:33:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4258003520 con 0x7f426c15cd00 2023-10-16T14:33:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4272374640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 0x7f426c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 868103904 0 0) 0x7f4258003520 con 0x7f426c15cd00 2023-10-16T14:33:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 msgr2=0x7f426c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 0x7f426c0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 >> v1:172.21.15.170:6789/0 conn(0x7f426c0cb000 legacy=0x7f426c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c0cd500 con 0x7f426c15cd00 2023-10-16T14:33:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4268001670 con 0x7f426c15cd00 2023-10-16T14:33:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 >> v1:172.21.15.29:6789/0 conn(0x7f426c15cd00 legacy=0x7f426c0cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42580039a0 0x7f4258003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42580047c0 0x7f4258004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 0x7f4258005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0cbaa0 con 0x7f42580047c0 2023-10-16T14:33:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425800ba60 con 0x7f42580039a0 2023-10-16T14:33:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.935+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425800bc20 con 0x7f426c070a90 2023-10-16T14:33:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271372640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42580047c0 0x7f4258004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4272374640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 0x7f4258005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271b73640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42580039a0 0x7f4258003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271372640 1 -- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 msgr2=0x7f4258005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271372640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f426c070a90 0x7f4258005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271372640 1 -- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42580039a0 msgr2=0x7f4258003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271372640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42580039a0 0x7f4258003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271372640 1 -- 172.21.15.29:0/3093821901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258003520 con 0x7f42580047c0 2023-10-16T14:33:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.936+0000 7f4271b73640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42580039a0 0x7f4258003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.937+0000 7f4271372640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42580047c0 0x7f4258004c40 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f425c001720 tx=0x7f42580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb21aeb354262e1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.937+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0052d0 con 0x7f42580047c0 2023-10-16T14:33:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.937+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c02e050 con 0x7f42580047c0 2023-10-16T14:33:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.937+0000 7f4270b71640 1 -- 172.21.15.29:0/3093821901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0035f0 con 0x7f42580047c0 2023-10-16T14:33:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272b75640 1 -- 172.21.15.29:0/3093821901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42580047c0 msgr2=0x7f4258004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272b75640 1 --2- 172.21.15.29:0/3093821901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42580047c0 0x7f4258004c40 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f425c001720 tx=0x7f42580057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272374640 1 -- 172.21.15.29:0/3093821901 reap_dead start 2023-10-16T14:33:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272b75640 1 -- 172.21.15.29:0/3093821901 shutdown_connections 2023-10-16T14:33:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272b75640 1 -- 172.21.15.29:0/3093821901 >> 172.21.15.29:0/3093821901 conn(0x7f426c0cf790 msgr2=0x7f426c0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272b75640 1 -- 172.21.15.29:0/3093821901 shutdown_connections 2023-10-16T14:33:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.938+0000 7f4272b75640 1 -- 172.21.15.29:0/3093821901 wait complete. 2023-10-16T14:33:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.940+0000 7f4272b75640 1 Processor -- start 2023-10-16T14:33:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.940+0000 7f4272b75640 1 -- start start 2023-10-16T14:33:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4272b75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42580047c0 0x7f426c148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4272b75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 0x7f426c14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4272b75640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f426c148e90 0x7f426c14aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4272b75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0b3d70 con 0x7f426c0c3560 2023-10-16T14:33:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4272b75640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c14ef20 con 0x7f426c148e90 2023-10-16T14:33:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4272b75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0c39e0 con 0x7f42580047c0 2023-10-16T14:33:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4271372640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 0x7f426c14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4271372640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 0x7f426c14e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56986/0 (socket says 172.21.15.29:56986) 2023-10-16T14:33:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.941+0000 7f4271372640 1 -- 172.21.15.29:0/1735047714 learned_addr learned my addr 172.21.15.29:0/1735047714 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271b73640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42580047c0 0x7f426c148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271372640 1 -- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42580047c0 msgr2=0x7f426c148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271372640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42580047c0 0x7f426c148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271372640 1 -- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f426c148e90 msgr2=0x7f426c14aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4272374640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f426c148e90 0x7f426c14aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271372640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f426c148e90 0x7f426c14aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271372640 1 -- 172.21.15.29:0/1735047714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c14f130 con 0x7f426c0c3560 2023-10-16T14:33:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271b73640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42580047c0 0x7f426c148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.942+0000 7f4271372640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 0x7f426c14e9e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f425c008fd0 tx=0x7f425c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.943+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0076c0 con 0x7f426c0c3560 2023-10-16T14:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.943+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c007a30 con 0x7f426c0c3560 2023-10-16T14:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.943+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c006be0 con 0x7f426c0c3560 2023-10-16T14:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.943+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c0b33d0 con 0x7f426c0c3560 2023-10-16T14:33:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.943+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c0b3990 con 0x7f426c0c3560 2023-10-16T14:33:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.944+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c0c3b20 con 0x7f426c0c3560 2023-10-16T14:33:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.945+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f425c006d40 con 0x7f426c0c3560 2023-10-16T14:33:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.946+0000 7f42627fc640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f425006af90 0x7f425006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.946+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f425c0aba30 con 0x7f426c0c3560 2023-10-16T14:33:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.947+0000 7f4271b73640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f425006af90 0x7f425006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.948+0000 7f4271b73640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f425006af90 0x7f425006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f42680018e0 tx=0x7f4268009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:36.950+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f426c0c3b20 con 0x7f426c0c3560 2023-10-16T14:33:37.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.141+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f426c0c3b20 con 0x7f426c0c3560 2023-10-16T14:33:37.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.142+0000 7f42627fc640 1 -- 172.21.15.29:0/1735047714 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f426c0c3b20 con 0x7f426c0c3560 2023-10-16T14:33:37.146 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:33:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f425006af90 msgr2=0x7f425006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f425006af90 0x7f425006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f42680018e0 tx=0x7f4268009040 comp rx=0 tx=0).stop 2023-10-16T14:33:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 msgr2=0x7f426c14e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 0x7f426c14e9e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f425c008fd0 tx=0x7f425c001120 comp rx=0 tx=0).stop 2023-10-16T14:33:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 shutdown_connections 2023-10-16T14:33:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f425006af90 0x7f425006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f426c0c3560 0x7f426c14e9e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f426c148e90 0x7f426c14aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 --2- 172.21.15.29:0/1735047714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42580047c0 0x7f426c148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 >> 172.21.15.29:0/1735047714 conn(0x7f426c0cf790 msgr2=0x7f426c15fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 shutdown_connections 2023-10-16T14:33:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.145+0000 7f4272b75640 1 -- 172.21.15.29:0/1735047714 wait complete. 2023-10-16T14:33:37.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T14:33:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.396+0000 7f4e7f14c640 1 Processor -- start 2023-10-16T14:33:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.396+0000 7f4e7f14c640 1 -- start start 2023-10-16T14:33:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e7f14c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4e78159f00 0x7f4e780b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e7f14c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e7f14c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7806f7f0 con 0x7f4e78159b20 2023-10-16T14:33:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e7f14c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7806c5e0 con 0x7f4e78159f00 2023-10-16T14:33:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e7f14c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7806c8c0 con 0x7f4e7815d880 2023-10-16T14:33:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e7d6e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.398+0000 7f4e77fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4e78159f00 0x7f4e780b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7cee8640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f4e78159b20 0x7f4e7806bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:45828/0 (socket says 172.21.15.29:45828) 2023-10-16T14:33:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7d6e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:50890/0 (socket says 172.21.15.29:50890) 2023-10-16T14:33:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7cee8640 1 -- 172.21.15.29:0/3790821040 learned_addr learned my addr 172.21.15.29:0/3790821040 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:37.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7d6e9640 1 -- 172.21.15.29:0/3790821040 >> v2:172.21.15.170:3300/0 conn(0x7f4e78159f00 msgr2=0x7f4e780b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:37.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7d6e9640 1 --2- 172.21.15.29:0/3790821040 >> v2:172.21.15.170:3300/0 conn(0x7f4e78159f00 0x7f4e780b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7d6e9640 1 -- 172.21.15.29:0/3790821040 >> v1:172.21.15.170:6789/0 conn(0x7f4e78159b20 legacy=0x7f4e7806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:37.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.399+0000 7f4e7d6e9640 1 -- 172.21.15.29:0/3790821040 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7806cba0 con 0x7f4e7815d880 2023-10-16T14:33:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.401+0000 7f4e7d6e9640 1 --2- 172.21.15.29:0/3790821040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780b0950 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009d90 tx=0x7f4e7002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb16cbb35ed71d32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.401+0000 7f4e777fe640 1 -- 172.21.15.29:0/3790821040 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7002cc50 con 0x7f4e7815d880 2023-10-16T14:33:37.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.401+0000 7f4e777fe640 1 -- 172.21.15.29:0/3790821040 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7002cdb0 con 0x7f4e7815d880 2023-10-16T14:33:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e777fe640 1 -- 172.21.15.29:0/3790821040 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70034660 con 0x7f4e7815d880 2023-10-16T14:33:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e7f14c640 1 -- 172.21.15.29:0/3790821040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 msgr2=0x7f4e780b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/3790821040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780b0950 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009d90 tx=0x7f4e7002bea0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e7f14c640 1 -- 172.21.15.29:0/3790821040 shutdown_connections 2023-10-16T14:33:37.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/3790821040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780b0950 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/3790821040 >> v2:172.21.15.170:3300/0 conn(0x7f4e78159f00 0x7f4e780b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.402+0000 7f4e7f14c640 1 -- 172.21.15.29:0/3790821040 >> 172.21.15.29:0/3790821040 conn(0x7f4e780cb410 msgr2=0x7f4e780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.403+0000 7f4e7f14c640 1 -- 172.21.15.29:0/3790821040 shutdown_connections 2023-10-16T14:33:37.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.403+0000 7f4e7f14c640 1 -- 172.21.15.29:0/3790821040 wait complete. 2023-10-16T14:33:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.404+0000 7f4e7f14c640 1 Processor -- start 2023-10-16T14:33:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.404+0000 7f4e7f14c640 1 -- start start 2023-10-16T14:33:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.405+0000 7f4e7f14c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 0x7f4e780c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.405+0000 7f4e7f14c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e78159f00 0x7f4e7814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.405+0000 7f4e7f14c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.405+0000 7f4e7f14c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780c54e0 con 0x7f4e78159b20 2023-10-16T14:33:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.405+0000 7f4e7f14c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7814e780 con 0x7f4e78159f00 2023-10-16T14:33:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.405+0000 7f4e7f14c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7814ef00 con 0x7f4e7815d880 2023-10-16T14:33:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 0x7f4e780c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 0x7f4e780c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56996/0 (socket says 172.21.15.29:56996) 2023-10-16T14:33:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 -- 172.21.15.29:0/2150173164 learned_addr learned my addr 172.21.15.29:0/2150173164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7d6e9640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e77fff640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e78159f00 0x7f4e7814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 -- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 msgr2=0x7f4e780af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 -- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e78159f00 msgr2=0x7f4e7814e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e78159f00 0x7f4e7814e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.406+0000 7f4e7cee8640 1 -- 172.21.15.29:0/2150173164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009a70 con 0x7f4e78159b20 2023-10-16T14:33:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.407+0000 7f4e7cee8640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 0x7f4e780c4fa0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800c480 tx=0x7f4e6800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.407+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68005ce0 con 0x7f4e78159b20 2023-10-16T14:33:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.407+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68005e40 con 0x7f4e78159b20 2023-10-16T14:33:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.407+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680072f0 con 0x7f4e78159b20 2023-10-16T14:33:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.407+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7814f1e0 con 0x7f4e78159b20 2023-10-16T14:33:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.407+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7814f7a0 con 0x7f4e78159b20 2023-10-16T14:33:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.408+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7806ce60 con 0x7f4e78159b20 2023-10-16T14:33:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.410+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e68007510 con 0x7f4e78159b20 2023-10-16T14:33:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.410+0000 7f4e757fa640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6406af90 0x7f4e6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.411+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f4e68088c70 con 0x7f4e78159b20 2023-10-16T14:33:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.411+0000 7f4e77fff640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6406af90 0x7f4e6406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.412+0000 7f4e77fff640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6406af90 0x7f4e6406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e780cdda0 tx=0x7f4e6c008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.415+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e7806ce60 con 0x7f4e78159b20 2023-10-16T14:33:37.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:37.685+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f4e7806ce60 con 0x7f4e78159b20 2023-10-16T14:33:38.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.224+0000 7f4e757fa640 1 -- 172.21.15.29:0/2150173164 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v645) v1 ==== 186+0+0 (secure 0 0 0) 0x7f4e7806ce60 con 0x7f4e78159b20 2023-10-16T14:33:38.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6406af90 msgr2=0x7f4e6406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6406af90 0x7f4e6406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e780cdda0 tx=0x7f4e6c008040 comp rx=0 tx=0).stop 2023-10-16T14:33:38.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 msgr2=0x7f4e780c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 0x7f4e780c4fa0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800c480 tx=0x7f4e6800c940 comp rx=0 tx=0).stop 2023-10-16T14:33:38.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 shutdown_connections 2023-10-16T14:33:38.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6406af90 0x7f4e6406d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e7815d880 0x7f4e780af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e78159f00 0x7f4e7814e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 --2- 172.21.15.29:0/2150173164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e78159b20 0x7f4e780c4fa0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 >> 172.21.15.29:0/2150173164 conn(0x7f4e780cb410 msgr2=0x7f4e7806fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:38.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 shutdown_connections 2023-10-16T14:33:38.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.228+0000 7f4e7f14c640 1 -- 172.21.15.29:0/2150173164 wait complete. 2023-10-16T14:33:38.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:33:38.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.477+0000 7f48f4d4e640 1 Processor -- start 2023-10-16T14:33:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.477+0000 7f48f4d4e640 1 -- start start 2023-10-16T14:33:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.478+0000 7f48f4d4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48f00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.478+0000 7f48f4d4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f015b9d0 con 0x7f48f00ccfb0 2023-10-16T14:33:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.478+0000 7f48f4d4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f015bcb0 con 0x7f48f006c040 2023-10-16T14:33:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.478+0000 7f48f4d4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f015bf90 con 0x7f48f00ad970 2023-10-16T14:33:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f48f00ccfb0 0x7f48f00cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38594/0 (socket says 172.21.15.29:38594) 2023-10-16T14:33:38.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48effff640 1 -- 172.21.15.29:0/1972753192 learned_addr learned my addr 172.21.15.29:0/1972753192 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:38.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48eeffd640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48f00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:38.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639668602 0 0) 0x7f48f015b9d0 con 0x7f48f00ccfb0 2023-10-16T14:33:38.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48dc003520 con 0x7f48f00ccfb0 2023-10-16T14:33:38.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866438464 0 0) 0x7f48dc003520 con 0x7f48f00ccfb0 2023-10-16T14:33:38.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 msgr2=0x7f48f00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:38.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48f00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 >> v1:172.21.15.170:6789/0 conn(0x7f48f006c040 legacy=0x7f48f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f00cc270 con 0x7f48f00ccfb0 2023-10-16T14:33:38.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f48e8001670 con 0x7f48f00ccfb0 2023-10-16T14:33:38.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 >> v1:172.21.15.29:6789/0 conn(0x7f48f00ccfb0 legacy=0x7f48f00cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48dc0039a0 0x7f48dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f015bf90 con 0x7f48dc004d00 2023-10-16T14:33:38.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc00bac0 con 0x7f48dc0039a0 2023-10-16T14:33:38.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc00bc80 con 0x7f48f00ad970 2023-10-16T14:33:38.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.480+0000 7f48effff640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:38.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48ef7fe640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48dc0039a0 0x7f48dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:38.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48eeffd640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:38.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48effff640 1 -- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 msgr2=0x7f48dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:38.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48effff640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48effff640 1 -- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48dc0039a0 msgr2=0x7f48dc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:38.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48effff640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48dc0039a0 0x7f48dc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48effff640 1 -- 172.21.15.29:0/1972753192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc003520 con 0x7f48dc004d00 2023-10-16T14:33:38.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.481+0000 7f48eeffd640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00ad970 0x7f48dc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:38.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48effff640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48dc005180 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f48e800c0e0 tx=0x7f48dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f1e6fe592416d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:38.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8005ad0 con 0x7f48dc004d00 2023-10-16T14:33:38.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e8005c30 con 0x7f48dc004d00 2023-10-16T14:33:38.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48ee7fc640 1 -- 172.21.15.29:0/1972753192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e802c810 con 0x7f48dc004d00 2023-10-16T14:33:38.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48f4d4e640 1 -- 172.21.15.29:0/1972753192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 msgr2=0x7f48dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/1972753192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48dc005180 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f48e800c0e0 tx=0x7f48dc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48effff640 1 -- 172.21.15.29:0/1972753192 reap_dead start 2023-10-16T14:33:38.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48f4d4e640 1 -- 172.21.15.29:0/1972753192 shutdown_connections 2023-10-16T14:33:38.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48f4d4e640 1 -- 172.21.15.29:0/1972753192 >> 172.21.15.29:0/1972753192 conn(0x7f48f00ce6a0 msgr2=0x7f48f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:38.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48f4d4e640 1 -- 172.21.15.29:0/1972753192 shutdown_connections 2023-10-16T14:33:38.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.483+0000 7f48f4d4e640 1 -- 172.21.15.29:0/1972753192 wait complete. 2023-10-16T14:33:38.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 Processor -- start 2023-10-16T14:33:38.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 -- start start 2023-10-16T14:33:38.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48f00c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48f00acb00 0x7f48f014b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00acf60 0x7f48f014bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00cad80 con 0x7f48dc004d00 2023-10-16T14:33:38.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00c2370 con 0x7f48f00acb00 2023-10-16T14:33:38.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.486+0000 7f48f4d4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f00c2610 con 0x7f48f00acf60 2023-10-16T14:33:38.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48f00c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:38.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48f00c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57006/0 (socket says 172.21.15.29:57006) 2023-10-16T14:33:38.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 -- 172.21.15.29:0/4264095679 learned_addr learned my addr 172.21.15.29:0/4264095679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:38.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 -- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00acf60 msgr2=0x7f48f014bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00acf60 0x7f48f014bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 -- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48f00acb00 msgr2=0x7f48f014b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48f00acb00 0x7f48f014b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 -- 172.21.15.29:0/4264095679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f00cdcd0 con 0x7f48dc004d00 2023-10-16T14:33:38.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.487+0000 7f48ef7fe640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48f00c30f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f48e000e8c0 tx=0x7f48e000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:38.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.488+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0003b80 con 0x7f48dc004d00 2023-10-16T14:33:38.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.488+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e0003ce0 con 0x7f48dc004d00 2023-10-16T14:33:38.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.488+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0005310 con 0x7f48dc004d00 2023-10-16T14:33:38.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.488+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f00c28f0 con 0x7f48dc004d00 2023-10-16T14:33:38.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.488+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f00ca050 con 0x7f48dc004d00 2023-10-16T14:33:38.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.489+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f006b330 con 0x7f48dc004d00 2023-10-16T14:33:38.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.490+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f48e0005470 con 0x7f48dc004d00 2023-10-16T14:33:38.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.491+0000 7f48e7fff640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f48d006afc0 0x7f48d006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:38.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.491+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f48e00898b0 con 0x7f48dc004d00 2023-10-16T14:33:38.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.492+0000 7f48eeffd640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f48d006afc0 0x7f48d006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:38.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.493+0000 7f48eeffd640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f48d006afc0 0x7f48d006d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f48f00ad570 tx=0x7f48d8008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:38.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.495+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f48e0054770 con 0x7f48dc004d00 2023-10-16T14:33:38.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.688+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f48f004e9b0 con 0x7f48dc004d00 2023-10-16T14:33:38.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.693+0000 7f48e7fff640 1 -- 172.21.15.29:0/4264095679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48e001a310 con 0x7f48dc004d00 2023-10-16T14:33:38.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f48d006afc0 msgr2=0x7f48d006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f48d006afc0 0x7f48d006d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f48f00ad570 tx=0x7f48d8008040 comp rx=0 tx=0).stop 2023-10-16T14:33:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 msgr2=0x7f48f00c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48f00c30f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f48e000e8c0 tx=0x7f48e000ed80 comp rx=0 tx=0).stop 2023-10-16T14:33:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 shutdown_connections 2023-10-16T14:33:38.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f48d006afc0 0x7f48d006d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f48f00acf60 0x7f48f014bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f48f00acb00 0x7f48f014b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 --2- 172.21.15.29:0/4264095679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48dc004d00 0x7f48f00c30f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 >> 172.21.15.29:0/4264095679 conn(0x7f48f00ce6a0 msgr2=0x7f48f006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:38.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.696+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 shutdown_connections 2023-10-16T14:33:38.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:38.697+0000 7f48f4d4e640 1 -- 172.21.15.29:0/4264095679 wait complete. 2023-10-16T14:33:38.719 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:33:38.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:33:38.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:33:38.971 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:33:38.971 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:38.972 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:39.035 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:33:39.036 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:33:39.036 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:39.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:33:39.098 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:39.099 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:39.162 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:33:39.163 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:33:39.163 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:39.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:33:39.221 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:33:39.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.452+0000 7fed558c3640 1 Processor -- start 2023-10-16T14:33:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.452+0000 7fed558c3640 1 -- start start 2023-10-16T14:33:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.454+0000 7fed558c3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 0x7fed5015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.454+0000 7fed558c3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5015ac60 con 0x7fed500ccb00 2023-10-16T14:33:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.454+0000 7fed558c3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed50070030 con 0x7fed500adbe0 2023-10-16T14:33:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.454+0000 7fed558c3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed50070310 con 0x7fed50153740 2023-10-16T14:33:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.454+0000 7fed4ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed500ccb00 0x7fed500ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38608/0 (socket says 172.21.15.29:38608) 2023-10-16T14:33:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.454+0000 7fed4ffff640 1 -- 172.21.15.29:0/2598085651 learned_addr learned my addr 172.21.15.29:0/2598085651 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832934624 0 0) 0x7fed5015ac60 con 0x7fed500ccb00 2023-10-16T14:33:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed40003520 con 0x7fed500ccb00 2023-10-16T14:33:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed548c1640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 0x7fed5015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750150122 0 0) 0x7fed40003520 con 0x7fed500ccb00 2023-10-16T14:33:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 msgr2=0x7fed5015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 0x7fed5015a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 >> v1:172.21.15.170:6789/0 conn(0x7fed500adbe0 legacy=0x7fed5015b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed500705f0 con 0x7fed500ccb00 2023-10-16T14:33:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fed3c001670 con 0x7fed500ccb00 2023-10-16T14:33:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 >> v1:172.21.15.29:6789/0 conn(0x7fed500ccb00 legacy=0x7fed500ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed400039a0 0x7fed40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 0x7fed400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed50070310 con 0x7fed40004d00 2023-10-16T14:33:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000bb90 con 0x7fed400039a0 2023-10-16T14:33:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.455+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000df60 con 0x7fed50153740 2023-10-16T14:33:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed550c2640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed4ffff640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 0x7fed400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed548c1640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed400039a0 0x7fed40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed550c2640 1 -- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 msgr2=0x7fed400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed550c2640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50153740 0x7fed400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed550c2640 1 -- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed400039a0 msgr2=0x7fed40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed550c2640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed400039a0 0x7fed40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.456+0000 7fed550c2640 1 -- 172.21.15.29:0/2598085651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40003520 con 0x7fed40004d00 2023-10-16T14:33:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.457+0000 7fed550c2640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed40005180 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fed48004480 tx=0x7fed400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194ef9dd3ab0ee24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.457+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48030400 con 0x7fed40004d00 2023-10-16T14:33:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.457+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48030560 con 0x7fed40004d00 2023-10-16T14:33:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.457+0000 7fed4f7fe640 1 -- 172.21.15.29:0/2598085651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48030910 con 0x7fed40004d00 2023-10-16T14:33:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed558c3640 1 -- 172.21.15.29:0/2598085651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 msgr2=0x7fed40005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed558c3640 1 --2- 172.21.15.29:0/2598085651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed40005180 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fed48004480 tx=0x7fed400057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed550c2640 1 -- 172.21.15.29:0/2598085651 reap_dead start 2023-10-16T14:33:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed558c3640 1 -- 172.21.15.29:0/2598085651 shutdown_connections 2023-10-16T14:33:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed558c3640 1 -- 172.21.15.29:0/2598085651 >> 172.21.15.29:0/2598085651 conn(0x7fed500ce6a0 msgr2=0x7fed500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed558c3640 1 -- 172.21.15.29:0/2598085651 shutdown_connections 2023-10-16T14:33:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.458+0000 7fed558c3640 1 -- 172.21.15.29:0/2598085651 wait complete. 2023-10-16T14:33:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.460+0000 7fed558c3640 1 Processor -- start 2023-10-16T14:33:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.460+0000 7fed558c3640 1 -- start start 2023-10-16T14:33:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed558c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed501e19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed558c3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50150820 0x7fed50155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed558c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed50156130 0x7fed501565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed558c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed501e1f30 con 0x7fed40004d00 2023-10-16T14:33:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed558c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed50156b20 con 0x7fed50156130 2023-10-16T14:33:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed558c3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed501e2120 con 0x7fed50150820 2023-10-16T14:33:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed548c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed501e19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed548c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed501e19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57014/0 (socket says 172.21.15.29:57014) 2023-10-16T14:33:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.461+0000 7fed548c1640 1 -- 172.21.15.29:0/171944340 learned_addr learned my addr 172.21.15.29:0/171944340 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed548c1640 1 -- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50150820 msgr2=0x7fed50155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed548c1640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50150820 0x7fed50155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed548c1640 1 -- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed50156130 msgr2=0x7fed501565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed548c1640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed50156130 0x7fed501565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed548c1640 1 -- 172.21.15.29:0/171944340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed501e2400 con 0x7fed40004d00 2023-10-16T14:33:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed548c1640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed501e19f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fed4400f580 tx=0x7fed4400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.462+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44014870 con 0x7fed40004d00 2023-10-16T14:33:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.463+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed440149d0 con 0x7fed40004d00 2023-10-16T14:33:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.463+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed501e26e0 con 0x7fed40004d00 2023-10-16T14:33:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.463+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44014d80 con 0x7fed40004d00 2023-10-16T14:33:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.463+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed501e2ca0 con 0x7fed40004d00 2023-10-16T14:33:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.464+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5006fc30 con 0x7fed40004d00 2023-10-16T14:33:39.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.465+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed44015410 con 0x7fed40004d00 2023-10-16T14:33:39.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.466+0000 7fed4d7fa640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fed3406afc0 0x7fed3406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.466+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fed4408a0e0 con 0x7fed40004d00 2023-10-16T14:33:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.467+0000 7fed4ffff640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fed3406afc0 0x7fed3406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.468+0000 7fed4ffff640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fed3406afc0 0x7fed3406d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed3c001bd0 tx=0x7fed3c000f50 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.470+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fed440550e0 con 0x7fed40004d00 2023-10-16T14:33:39.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.704+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed500708a0 con 0x7fed40004d00 2023-10-16T14:33:39.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.705+0000 7fed4d7fa640 1 -- 172.21.15.29:0/171944340 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fed44058700 con 0x7fed40004d00 2023-10-16T14:33:39.708 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:39.708 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3999082943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3999082943},{"type":"v1","addr":"172.21.15.170:6835","nonce":3999082943}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5528,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/3608495586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3608495586},{"type":"v1","addr":"172.21.15.29:6834","nonce":3608495586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/133876642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":133876642},{"type":"v1","addr":"172.21.15.29:6835","nonce":133876642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1341865278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1341865278},{"type":"v1","addr":"172.21.15.170:6837","nonce":1341865278}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[]} 2023-10-16T14:33:39.709 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 44 2023-10-16T14:33:39.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fed3406afc0 msgr2=0x7fed3406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:39.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fed3406afc0 0x7fed3406d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed3c001bd0 tx=0x7fed3c000f50 comp rx=0 tx=0).stop 2023-10-16T14:33:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 msgr2=0x7fed501e19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed501e19f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fed4400f580 tx=0x7fed4400fa40 comp rx=0 tx=0).stop 2023-10-16T14:33:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 shutdown_connections 2023-10-16T14:33:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fed3406afc0 0x7fed3406d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed50156130 0x7fed501565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed50150820 0x7fed50155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 --2- 172.21.15.29:0/171944340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed40004d00 0x7fed501e19f0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 >> 172.21.15.29:0/171944340 conn(0x7fed500ce6a0 msgr2=0x7fed50152990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 shutdown_connections 2023-10-16T14:33:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:39.708+0000 7fed558c3640 1 -- 172.21.15.29:0/171944340 wait complete. 2023-10-16T14:33:39.730 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:33:39.730 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:33:39.730 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:33:39.731 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:39.731 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:33:39.731 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:33:39.732 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:33:39.732 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:39.732 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:33:39.732 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:33:39.733 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:33:39.733 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:39.733 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:33:39.733 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:33:39.734 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:33:39.734 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:33:39.751 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:33:39.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:33:39.752 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:33:39.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:33:39.755 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:33:39.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:33:39.757 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:33:39.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:33:39.804 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:33:39.805 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:39.807 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:33:39.807 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:39.813 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:33:39.813 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:39.815 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:33:39.815 DEBUG:teuthology.parallel:result is None 2023-10-16T14:33:39.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:33:39.972 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:33:39.972 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:33:39.978 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:33:39.978 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:33:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.008+0000 7fc4615b6640 1 Processor -- start 2023-10-16T14:33:40.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.008+0000 7fc4615b6640 1 -- start start 2023-10-16T14:33:40.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.009+0000 7fc4615b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 0x7fc45c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:40.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.009+0000 7fc4615b6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc45c160b90 0x7fc45c160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:40.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.009+0000 7fc4615b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0ac890 con 0x7fc45c069da0 2023-10-16T14:33:40.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.010+0000 7fc4615b6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c15d530 con 0x7fc45c160b90 2023-10-16T14:33:40.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.010+0000 7fc4615b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c161650 con 0x7fc45c15d150 2023-10-16T14:33:40.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.010+0000 7fc45affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 0x7fc45c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.010+0000 7fc45a7fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc45c160b90 0x7fc45c160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.010+0000 7fc45affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 0x7fc45c0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:51006/0 (socket says 172.21.15.29:51006) 2023-10-16T14:33:40.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.010+0000 7fc45affd640 1 -- 172.21.15.29:0/3649783302 learned_addr learned my addr 172.21.15.29:0/3649783302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:40.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.011+0000 7fc45affd640 1 -- 172.21.15.29:0/3649783302 >> v2:172.21.15.170:3300/0 conn(0x7fc45c160b90 msgr2=0x7fc45c160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:40.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.011+0000 7fc45affd640 1 --2- 172.21.15.29:0/3649783302 >> v2:172.21.15.170:3300/0 conn(0x7fc45c160b90 0x7fc45c160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:40.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.011+0000 7fc45affd640 1 -- 172.21.15.29:0/3649783302 >> v1:172.21.15.170:6789/0 conn(0x7fc45c069da0 legacy=0x7fc45c06a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:40.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.011+0000 7fc45affd640 1 -- 172.21.15.29:0/3649783302 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c161930 con 0x7fc45c15d150 2023-10-16T14:33:40.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.011+0000 7fc459ffb640 1 -- 172.21.15.29:0/3649783302 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170963409 0 0) 0x7fc45c0ac890 con 0x7fc45c069da0 2023-10-16T14:33:40.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.011+0000 7fc45a7fc640 1 --2- 172.21.15.29:0/3649783302 >> v2:172.21.15.170:3300/0 conn(0x7fc45c160b90 0x7fc45c160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:33:40.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.012+0000 7fc45affd640 1 --2- 172.21.15.29:0/3649783302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 0x7fc45c0ad350 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009870 tx=0x7fc44c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38568e0093da5ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:40.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.012+0000 7fc459ffb640 1 -- 172.21.15.29:0/3649783302 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c030840 con 0x7fc45c15d150 2023-10-16T14:33:40.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.012+0000 7fc459ffb640 1 -- 172.21.15.29:0/3649783302 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c0309a0 con 0x7fc45c15d150 2023-10-16T14:33:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.012+0000 7fc459ffb640 1 -- 172.21.15.29:0/3649783302 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c030d30 con 0x7fc45c15d150 2023-10-16T14:33:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 -- 172.21.15.29:0/3649783302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 msgr2=0x7fc45c0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 --2- 172.21.15.29:0/3649783302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 0x7fc45c0ad350 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009870 tx=0x7fc44c02b980 comp rx=0 tx=0).stop 2023-10-16T14:33:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 -- 172.21.15.29:0/3649783302 shutdown_connections 2023-10-16T14:33:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 --2- 172.21.15.29:0/3649783302 >> v2:172.21.15.170:3300/0 conn(0x7fc45c160b90 0x7fc45c160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 --2- 172.21.15.29:0/3649783302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c15d150 0x7fc45c0ad350 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 -- 172.21.15.29:0/3649783302 >> 172.21.15.29:0/3649783302 conn(0x7fc45c0ce9b0 msgr2=0x7fc45c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 -- 172.21.15.29:0/3649783302 shutdown_connections 2023-10-16T14:33:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.013+0000 7fc4615b6640 1 -- 172.21.15.29:0/3649783302 wait complete. 2023-10-16T14:33:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.014+0000 7fc4615b6640 1 Processor -- start 2023-10-16T14:33:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.015+0000 7fc4615b6640 1 -- start start 2023-10-16T14:33:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc4615b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc45c069da0 0x7fc45c14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc4615b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 0x7fc45c0b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc4615b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c160b90 0x7fc45c14ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc4615b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0c6750 con 0x7fc45c15d150 2023-10-16T14:33:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc4615b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c14b190 con 0x7fc45c069da0 2023-10-16T14:33:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc4615b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c14b960 con 0x7fc45c160b90 2023-10-16T14:33:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc45a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 0x7fc45c0b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc45a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 0x7fc45c0b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57050/0 (socket says 172.21.15.29:57050) 2023-10-16T14:33:40.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc45a7fc640 1 -- 172.21.15.29:0/4226454739 learned_addr learned my addr 172.21.15.29:0/4226454739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:40.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc45affd640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc45c069da0 0x7fc45c14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:40.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.016+0000 7fc45b7fe640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c160b90 0x7fc45c14ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:40.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45a7fc640 1 -- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c160b90 msgr2=0x7fc45c14ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:40.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45a7fc640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c160b90 0x7fc45c14ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:40.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45a7fc640 1 -- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc45c069da0 msgr2=0x7fc45c14ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:40.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45a7fc640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc45c069da0 0x7fc45c14ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:40.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45a7fc640 1 -- 172.21.15.29:0/4226454739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc450009400 con 0x7fc45c15d150 2023-10-16T14:33:40.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45affd640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc45c069da0 0x7fc45c14ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:33:40.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc45a7fc640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 0x7fc45c0b4110 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e960 tx=0x7fc45000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:40.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000ca00 con 0x7fc45c15d150 2023-10-16T14:33:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45000cb60 con 0x7fc45c15d150 2023-10-16T14:33:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450013470 con 0x7fc45c15d150 2023-10-16T14:33:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c0094d0 con 0x7fc45c15d150 2023-10-16T14:33:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.017+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c14bf20 con 0x7fc45c15d150 2023-10-16T14:33:40.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.018+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c04e160 con 0x7fc45c15d150 2023-10-16T14:33:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.020+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc4500135d0 con 0x7fc45c15d150 2023-10-16T14:33:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.021+0000 7fc4437fe640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc44806af90 0x7fc44806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.021+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc45008a080 con 0x7fc45c15d150 2023-10-16T14:33:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.022+0000 7fc45affd640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc44806af90 0x7fc44806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.022+0000 7fc45affd640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc44806af90 0x7fc44806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc44c036000 tx=0x7fc44c008030 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.024+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc450055190 con 0x7fc45c15d150 2023-10-16T14:33:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:40.232+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc45c0ade10 con 0x7fc45c15d150 2023-10-16T14:33:41.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.232+0000 7fc4437fe640 1 -- 172.21.15.29:0/4226454739 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc4500587b0 con 0x7fc45c15d150 2023-10-16T14:33:41.236 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:33:41.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc44806af90 msgr2=0x7fc44806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc44806af90 0x7fc44806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc44c036000 tx=0x7fc44c008030 comp rx=0 tx=0).stop 2023-10-16T14:33:41.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 msgr2=0x7fc45c0b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 0x7fc45c0b4110 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e960 tx=0x7fc45000ee20 comp rx=0 tx=0).stop 2023-10-16T14:33:41.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 shutdown_connections 2023-10-16T14:33:41.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc44806af90 0x7fc44806d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c15d150 0x7fc45c0b4110 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc45c160b90 0x7fc45c14ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 --2- 172.21.15.29:0/4226454739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc45c069da0 0x7fc45c14ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 >> 172.21.15.29:0/4226454739 conn(0x7fc45c0ce9b0 msgr2=0x7fc45c0ad540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:41.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 shutdown_connections 2023-10-16T14:33:41.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.237+0000 7fc4615b6640 1 -- 172.21.15.29:0/4226454739 wait complete. 2023-10-16T14:33:41.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:33:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.486+0000 7fec44860640 1 Processor -- start 2023-10-16T14:33:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.486+0000 7fec44860640 1 -- start start 2023-10-16T14:33:41.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.487+0000 7fec44860640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec40159950 0x7fec4015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:41.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.487+0000 7fec44860640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fec4006c810 0x7fec400708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:41.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.487+0000 7fec44860640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4015aef0 con 0x7fec40070450 2023-10-16T14:33:41.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.487+0000 7fec44860640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec40070e20 con 0x7fec40159950 2023-10-16T14:33:41.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec44860640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4015e490 con 0x7fec4006c810 2023-10-16T14:33:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec40159950 0x7fec4015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fec40070450 0x7fec4006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38630/0 (socket says 172.21.15.29:38630) 2023-10-16T14:33:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3effd640 1 -- 172.21.15.29:0/2467868246 learned_addr learned my addr 172.21.15.29:0/2467868246 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3f7fe640 1 -- 172.21.15.29:0/2467868246 >> v2:172.21.15.170:3300/0 conn(0x7fec4006c810 msgr2=0x7fec400708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3f7fe640 1 --2- 172.21.15.29:0/2467868246 >> v2:172.21.15.170:3300/0 conn(0x7fec4006c810 0x7fec400708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3f7fe640 1 -- 172.21.15.29:0/2467868246 >> v1:172.21.15.29:6789/0 conn(0x7fec40070450 legacy=0x7fec4006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3f7fe640 1 -- 172.21.15.29:0/2467868246 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4015d910 con 0x7fec40159950 2023-10-16T14:33:41.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.488+0000 7fec3e7fc640 1 -- 172.21.15.29:0/2467868246 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589411988 0 0) 0x7fec34001110 con 0x7fec40070450 2023-10-16T14:33:41.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.490+0000 7fec3f7fe640 1 --2- 172.21.15.29:0/2467868246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40159950 0x7fec4015a9b0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fec400716a0 tx=0x7fec3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab940a305a17bc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:41.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.490+0000 7fec3e7fc640 1 -- 172.21.15.29:0/2467868246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3002e070 con 0x7fec40159950 2023-10-16T14:33:41.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.490+0000 7fec3e7fc640 1 -- 172.21.15.29:0/2467868246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec300315a0 con 0x7fec40159950 2023-10-16T14:33:41.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.490+0000 7fec3e7fc640 1 -- 172.21.15.29:0/2467868246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30031930 con 0x7fec40159950 2023-10-16T14:33:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.490+0000 7fec44860640 1 -- 172.21.15.29:0/2467868246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40159950 msgr2=0x7fec4015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.490+0000 7fec44860640 1 --2- 172.21.15.29:0/2467868246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40159950 0x7fec4015a9b0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fec400716a0 tx=0x7fec3002b840 comp rx=0 tx=0).stop 2023-10-16T14:33:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.491+0000 7fec44860640 1 -- 172.21.15.29:0/2467868246 shutdown_connections 2023-10-16T14:33:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.491+0000 7fec44860640 1 --2- 172.21.15.29:0/2467868246 >> v2:172.21.15.170:3300/0 conn(0x7fec4006c810 0x7fec400708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.491+0000 7fec44860640 1 --2- 172.21.15.29:0/2467868246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40159950 0x7fec4015a9b0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.491+0000 7fec44860640 1 -- 172.21.15.29:0/2467868246 >> 172.21.15.29:0/2467868246 conn(0x7fec400ce6a0 msgr2=0x7fec400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.491+0000 7fec44860640 1 -- 172.21.15.29:0/2467868246 shutdown_connections 2023-10-16T14:33:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.491+0000 7fec44860640 1 -- 172.21.15.29:0/2467868246 wait complete. 2023-10-16T14:33:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.492+0000 7fec44860640 1 Processor -- start 2023-10-16T14:33:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.493+0000 7fec44860640 1 -- start start 2023-10-16T14:33:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.493+0000 7fec44860640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4006c810 0x7fec400b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.493+0000 7fec44860640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 0x7fec400c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:41.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.493+0000 7fec44860640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec40159950 0x7fec401587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:41.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec44860640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c26f0 con 0x7fec40070450 2023-10-16T14:33:41.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec44860640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec40158d20 con 0x7fec4006c810 2023-10-16T14:33:41.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec44860640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c2990 con 0x7fec40159950 2023-10-16T14:33:41.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 0x7fec400c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:41.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 0x7fec400c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57068/0 (socket says 172.21.15.29:57068) 2023-10-16T14:33:41.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 -- 172.21.15.29:0/4003949476 learned_addr learned my addr 172.21.15.29:0/4003949476 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:41.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3f7fe640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4006c810 0x7fec400b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:41.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 -- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec40159950 msgr2=0x7fec401587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec40159950 0x7fec401587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 -- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4006c810 msgr2=0x7fec400b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4006c810 0x7fec400b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.494+0000 7fec3effd640 1 -- 172.21.15.29:0/4003949476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30009400 con 0x7fec40070450 2023-10-16T14:33:41.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.495+0000 7fec3effd640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 0x7fec400c21b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fec3400c780 tx=0x7fec3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:41.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.495+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340072d0 con 0x7fec40070450 2023-10-16T14:33:41.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.495+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec34007450 con 0x7fec40070450 2023-10-16T14:33:41.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.495+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec400c2c70 con 0x7fec40070450 2023-10-16T14:33:41.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.495+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340047e0 con 0x7fec40070450 2023-10-16T14:33:41.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.495+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec400c3230 con 0x7fec40070450 2023-10-16T14:33:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.496+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400cf310 con 0x7fec40070450 2023-10-16T14:33:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.497+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec34004960 con 0x7fec40070450 2023-10-16T14:33:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.498+0000 7fec27fff640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fec2c06af90 0x7fec2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.498+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec34088670 con 0x7fec40070450 2023-10-16T14:33:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.499+0000 7fec3f7fe640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fec2c06af90 0x7fec2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:41.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.500+0000 7fec3f7fe640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fec2c06af90 0x7fec2c06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec30036000 tx=0x7fec3003d040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:41.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.502+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec340538c0 con 0x7fec40070450 2023-10-16T14:33:41.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.763+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fec4015aef0 con 0x7fec40070450 2023-10-16T14:33:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.764+0000 7fec27fff640 1 -- 172.21.15.29:0/4003949476 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fec4015aef0 con 0x7fec40070450 2023-10-16T14:33:41.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:41.768 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:33:41.769 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:33:41.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.770 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:33:41.770 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:33:41.770 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:41.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:41.772 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.772 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.774 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:33:41.774 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:33:41.775 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:41.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:41.776 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.776 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:33:41.777 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:33:41.777 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:41.778 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:41.778 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.779 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:33:41.779 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:33:41.779 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.780 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.780 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:41.780 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:41.781 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.781 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:33:41.782 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:33:41.782 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.782 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.783 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.783 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.783 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.784 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:33:41.785 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:33:41.785 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.785 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.786 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.786 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.787 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.788 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:33:41.788 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:33:41.789 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.789 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.789 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.790 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.790 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.791 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:33:41.792 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:33:41.792 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.792 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.793 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.793 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.794 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.795 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:33:41.795 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:33:41.795 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.796 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.796 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.796 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.797 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.798 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:33:41.798 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:33:41.798 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.799 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.799 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.799 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.800 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.800 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:33:41.801 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:33:41.801 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.801 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.801 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.802 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.803 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:33:41.803 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:33:41.803 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.803 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:33:41.803 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:33:41.804 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.804 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.805 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:33:41.805 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:33:41.805 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.805 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.805 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:33:41.806 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:33:41.806 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.807 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:33:41.807 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:33:41.807 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.807 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.807 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:33:41.808 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:33:41.808 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:33:41.808 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.809 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:33:41.809 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:33:41.809 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.809 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.809 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:33:41.810 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:33:41.810 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:33:41.810 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.811 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:33:41.811 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:33:41.811 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.811 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:33:41.812 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:33:41.812 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:33:41.812 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:33:41.812 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.813 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:33:41.813 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:33:41.813 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.814 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:33:41.814 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:33:41.814 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:33:41.814 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.814 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.815 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:33:41.815 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:33:41.815 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.816 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:33:41.816 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:33:41.816 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:33:41.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:41.817 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:33:41.818 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:33:41.818 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:33:41.818 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:33:41.818 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:33:41.818 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:33:41.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:41.819 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:33:41.819 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:33:41.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fec2c06af90 msgr2=0x7fec2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fec2c06af90 0x7fec2c06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec30036000 tx=0x7fec3003d040 comp rx=0 tx=0).stop 2023-10-16T14:33:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 msgr2=0x7fec400c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 0x7fec400c21b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fec3400c780 tx=0x7fec3400cc40 comp rx=0 tx=0).stop 2023-10-16T14:33:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 shutdown_connections 2023-10-16T14:33:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fec2c06af90 0x7fec2c06d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec40070450 0x7fec400c21b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec40159950 0x7fec401587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 --2- 172.21.15.29:0/4003949476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec4006c810 0x7fec400b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.767+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 >> 172.21.15.29:0/4003949476 conn(0x7fec400ce6a0 msgr2=0x7fec40159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.768+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 shutdown_connections 2023-10-16T14:33:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:41.768+0000 7fec44860640 1 -- 172.21.15.29:0/4003949476 wait complete. 2023-10-16T14:33:41.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:42.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.052+0000 7f67b7075640 1 Processor -- start 2023-10-16T14:33:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.053+0000 7f67b7075640 1 -- start start 2023-10-16T14:33:42.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.054+0000 7f67b7075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67b00ad970 0x7f67b00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.054+0000 7f67b7075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b00ccfb0 0x7f67b00cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.054+0000 7f67b7075640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b015b9d0 con 0x7f67b006c040 2023-10-16T14:33:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.054+0000 7f67b7075640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b015bcb0 con 0x7f67b00ad970 2023-10-16T14:33:42.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.054+0000 7f67b7075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b015bf90 con 0x7f67b00ccfb0 2023-10-16T14:33:42.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.054+0000 7f67b5872640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67b00ad970 0x7f67b00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5872640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67b00ad970 0x7f67b00cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57084/0 (socket says 172.21.15.29:57084) 2023-10-16T14:33:42.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5872640 1 -- 172.21.15.29:0/1637565259 learned_addr learned my addr 172.21.15.29:0/1637565259 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:42.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b6874640 1 --2- 172.21.15.29:0/1637565259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b00ccfb0 0x7f67b00cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5872640 1 -- 172.21.15.29:0/1637565259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b00ccfb0 msgr2=0x7f67b00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:42.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5872640 1 --2- 172.21.15.29:0/1637565259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b00ccfb0 0x7f67b00cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5872640 1 -- 172.21.15.29:0/1637565259 >> v1:172.21.15.29:6789/0 conn(0x7f67b006c040 legacy=0x7f67b006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5872640 1 -- 172.21.15.29:0/1637565259 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00cc270 con 0x7f67b00ad970 2023-10-16T14:33:42.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.055+0000 7f67b5071640 1 -- 172.21.15.29:0/1637565259 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168673156 0 0) 0x7f67b015b9d0 con 0x7f67b006c040 2023-10-16T14:33:42.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.056+0000 7f67b5872640 1 --2- 172.21.15.29:0/1637565259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b00cca70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009870 tx=0x7f67a002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414316c4b20c6bd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.057+0000 7f67b5071640 1 -- 172.21.15.29:0/1637565259 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0030840 con 0x7f67b00ad970 2023-10-16T14:33:42.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.057+0000 7f67b5071640 1 -- 172.21.15.29:0/1637565259 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a00309a0 con 0x7f67b00ad970 2023-10-16T14:33:42.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.057+0000 7f67b5071640 1 -- 172.21.15.29:0/1637565259 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0030d30 con 0x7f67b00ad970 2023-10-16T14:33:42.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.057+0000 7f67b7075640 1 -- 172.21.15.29:0/1637565259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 msgr2=0x7f67b00cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.057+0000 7f67b7075640 1 --2- 172.21.15.29:0/1637565259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b00cca70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009870 tx=0x7f67a002b980 comp rx=0 tx=0).stop 2023-10-16T14:33:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.058+0000 7f67b7075640 1 -- 172.21.15.29:0/1637565259 shutdown_connections 2023-10-16T14:33:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.058+0000 7f67b7075640 1 --2- 172.21.15.29:0/1637565259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b00ccfb0 0x7f67b00cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.058+0000 7f67b7075640 1 --2- 172.21.15.29:0/1637565259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b00cca70 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.058+0000 7f67b7075640 1 -- 172.21.15.29:0/1637565259 >> 172.21.15.29:0/1637565259 conn(0x7f67b00ce6a0 msgr2=0x7f67b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.058+0000 7f67b7075640 1 -- 172.21.15.29:0/1637565259 shutdown_connections 2023-10-16T14:33:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.058+0000 7f67b7075640 1 -- 172.21.15.29:0/1637565259 wait complete. 2023-10-16T14:33:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.060+0000 7f67b7075640 1 Processor -- start 2023-10-16T14:33:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.060+0000 7f67b7075640 1 -- start start 2023-10-16T14:33:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b7075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b006c040 0x7f67b015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b7075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b7075640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67b00ccfb0 0x7f67b015d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b7075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b01f3210 con 0x7f67b00ad970 2023-10-16T14:33:42.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b7075640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b015a610 con 0x7f67b00ccfb0 2023-10-16T14:33:42.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b7075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b01f6280 con 0x7f67b006c040 2023-10-16T14:33:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b5872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b5872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b015d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57096/0 (socket says 172.21.15.29:57096) 2023-10-16T14:33:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b5872640 1 -- 172.21.15.29:0/1210350834 learned_addr learned my addr 172.21.15.29:0/1210350834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b6073640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b006c040 0x7f67b015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b6874640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67b00ccfb0 0x7f67b015d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b5872640 1 -- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b006c040 msgr2=0x7f67b015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.061+0000 7f67b5872640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b006c040 0x7f67b015a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67b5872640 1 -- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67b00ccfb0 msgr2=0x7f67b015d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67b5872640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67b00ccfb0 0x7f67b015d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67b5872640 1 -- 172.21.15.29:0/1210350834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00094d0 con 0x7f67b00ad970 2023-10-16T14:33:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67b6874640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67b00ccfb0 0x7f67b015d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67b5872640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b015d1a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f67a0000c00 tx=0x7f67a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0047070 con 0x7f67b00ad970 2023-10-16T14:33:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0039a90 con 0x7f67b00ad970 2023-10-16T14:33:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0039e20 con 0x7f67b00ad970 2023-10-16T14:33:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.062+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b01f6710 con 0x7f67b00ad970 2023-10-16T14:33:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.063+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b01f5c30 con 0x7f67b00ad970 2023-10-16T14:33:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.064+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b006b330 con 0x7f67b00ad970 2023-10-16T14:33:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.065+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f67a004c410 con 0x7f67b00ad970 2023-10-16T14:33:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.065+0000 7f67a6ffd640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f679406afc0 0x7f679406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.066+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f67a003e070 con 0x7f67b00ad970 2023-10-16T14:33:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.066+0000 7f67b6073640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f679406afc0 0x7f679406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.067+0000 7f67b6073640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f679406afc0 0x7f679406d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f67ac000f60 tx=0x7f67ac009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.070+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f67a0076c90 con 0x7f67b00ad970 2023-10-16T14:33:42.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.267+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67b004e9b0 con 0x7f67b00ad970 2023-10-16T14:33:42.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.269+0000 7f67a6ffd640 1 -- 172.21.15.29:0/1210350834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+12859 (secure 0 0 0) 0x7f67a007a2b0 con 0x7f67b00ad970 2023-10-16T14:33:42.272 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:42.273 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":107,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:41.229567+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:42.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f679406afc0 msgr2=0x7f679406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f679406afc0 0x7f679406d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f67ac000f60 tx=0x7f67ac009040 comp rx=0 tx=0).stop 2023-10-16T14:33:42.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 msgr2=0x7f67b015d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b015d1a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f67a0000c00 tx=0x7f67a0004730 comp rx=0 tx=0).stop 2023-10-16T14:33:42.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 shutdown_connections 2023-10-16T14:33:42.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f679406afc0 0x7f679406d480 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67b00ad970 0x7f67b015d1a0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f67b00ccfb0 0x7f67b015d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 --2- 172.21.15.29:0/1210350834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67b006c040 0x7f67b015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.272+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 >> 172.21.15.29:0/1210350834 conn(0x7f67b00ce6a0 msgr2=0x7f67b00cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:42.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.273+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 shutdown_connections 2023-10-16T14:33:42.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.273+0000 7f67b7075640 1 -- 172.21.15.29:0/1210350834 wait complete. 2023-10-16T14:33:42.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:33:42.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:33:42.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.521+0000 7f5edbfff640 1 Processor -- start 2023-10-16T14:33:42.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.521+0000 7f5edbfff640 1 -- start start 2023-10-16T14:33:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.522+0000 7f5edbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5edc0cb000 0x7f5edc0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc070380 0x7f5edc070810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5edbfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc15d0e0 con 0x7f5edc15cd00 2023-10-16T14:33:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5edbfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5edc070d50 con 0x7f5edc0cb000 2023-10-16T14:33:42.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5edbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc0cbd80 con 0x7f5edc070380 2023-10-16T14:33:42.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5edc0cb000 0x7f5edc0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5edc0cb000 0x7f5edc0cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57110/0 (socket says 172.21.15.29:57110) 2023-10-16T14:33:42.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 -- 172.21.15.29:0/1443645029 learned_addr learned my addr 172.21.15.29:0/1443645029 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:42.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 -- 172.21.15.29:0/1443645029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc070380 msgr2=0x7f5edc070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:42.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 --2- 172.21.15.29:0/1443645029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc070380 0x7f5edc070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 -- 172.21.15.29:0/1443645029 >> v1:172.21.15.170:6789/0 conn(0x7f5edc15cd00 legacy=0x7f5edc0cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.523+0000 7f5eda7fc640 1 -- 172.21.15.29:0/1443645029 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc0cbaa0 con 0x7f5edc0cb000 2023-10-16T14:33:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.524+0000 7f5eda7fc640 1 --2- 172.21.15.29:0/1443645029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc0cb000 0x7f5edc0cb490 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009e50 tx=0x7f5ec402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ec0d6b6d1cfb62 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.525+0000 7f5ed9ffb640 1 -- 172.21.15.29:0/1443645029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec402ca80 con 0x7f5edc0cb000 2023-10-16T14:33:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.525+0000 7f5ed9ffb640 1 -- 172.21.15.29:0/1443645029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec402cbe0 con 0x7f5edc0cb000 2023-10-16T14:33:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.525+0000 7f5ed9ffb640 1 -- 172.21.15.29:0/1443645029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40343f0 con 0x7f5edc0cb000 2023-10-16T14:33:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.525+0000 7f5edbfff640 1 -- 172.21.15.29:0/1443645029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc0cb000 msgr2=0x7f5edc0cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.525+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1443645029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc0cb000 0x7f5edc0cb490 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009e50 tx=0x7f5ec402bee0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.526+0000 7f5edbfff640 1 -- 172.21.15.29:0/1443645029 shutdown_connections 2023-10-16T14:33:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.526+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1443645029 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc070380 0x7f5edc070810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.526+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1443645029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc0cb000 0x7f5edc0cb490 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.526+0000 7f5edbfff640 1 -- 172.21.15.29:0/1443645029 >> 172.21.15.29:0/1443645029 conn(0x7f5edc0cf790 msgr2=0x7f5edc0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.526+0000 7f5edbfff640 1 -- 172.21.15.29:0/1443645029 shutdown_connections 2023-10-16T14:33:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.526+0000 7f5edbfff640 1 -- 172.21.15.29:0/1443645029 wait complete. 2023-10-16T14:33:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.527+0000 7f5edbfff640 1 Processor -- start 2023-10-16T14:33:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.528+0000 7f5edbfff640 1 -- start start 2023-10-16T14:33:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.528+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 0x7f5edc0b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc0cb000 0x7f5edc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5edc15cd00 0x7f5edc1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc0b3170 con 0x7f5edc070380 2023-10-16T14:33:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edbfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc0b2bb0 con 0x7f5edc15cd00 2023-10-16T14:33:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc0b2e90 con 0x7f5edc0cb000 2023-10-16T14:33:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 0x7f5edc0b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 0x7f5edc0b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57124/0 (socket says 172.21.15.29:57124) 2023-10-16T14:33:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 -- 172.21.15.29:0/1491059277 learned_addr learned my addr 172.21.15.29:0/1491059277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5eda7fc640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc0cb000 0x7f5edc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edb7fe640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5edc15cd00 0x7f5edc1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 -- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc0cb000 msgr2=0x7f5edc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc0cb000 0x7f5edc0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 -- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5edc15cd00 msgr2=0x7f5edc1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5edc15cd00 0x7f5edc1575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.529+0000 7f5edaffd640 1 -- 172.21.15.29:0/1491059277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec4009b30 con 0x7f5edc070380 2023-10-16T14:33:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.530+0000 7f5edaffd640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 0x7f5edc0b2380 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00b3e0 tx=0x7f5ecc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.530+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011830 con 0x7f5edc070380 2023-10-16T14:33:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.530+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc0119b0 con 0x7f5edc070380 2023-10-16T14:33:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.530+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011d60 con 0x7f5edc070380 2023-10-16T14:33:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.530+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5edc0c3b30 con 0x7f5edc070380 2023-10-16T14:33:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.530+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5edc0b28c0 con 0x7f5edc070380 2023-10-16T14:33:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.531+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc0c3990 con 0x7f5edc070380 2023-10-16T14:33:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.532+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ecc01a400 con 0x7f5edc070380 2023-10-16T14:33:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.533+0000 7f5ed3fff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f5ebc06af90 0x7f5ebc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.533+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5ecc089650 con 0x7f5edc070380 2023-10-16T14:33:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.534+0000 7f5eda7fc640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f5ebc06af90 0x7f5ebc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.535+0000 7f5eda7fc640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f5ebc06af90 0x7f5ebc06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4035000 tx=0x7f5ec4030000 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.538+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5edc0c3990 con 0x7f5edc070380 2023-10-16T14:33:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.733+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5edc0c3990 con 0x7f5edc070380 2023-10-16T14:33:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.733+0000 7f5ed3fff640 1 -- 172.21.15.29:0/1491059277 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5edc0c3990 con 0x7f5edc070380 2023-10-16T14:33:42.736 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:33:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f5ebc06af90 msgr2=0x7f5ebc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f5ebc06af90 0x7f5ebc06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4035000 tx=0x7f5ec4030000 comp rx=0 tx=0).stop 2023-10-16T14:33:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 msgr2=0x7f5edc0b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 0x7f5edc0b2380 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00b3e0 tx=0x7f5ecc00b8a0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 shutdown_connections 2023-10-16T14:33:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f5ebc06af90 0x7f5ebc06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5edc15cd00 0x7f5edc1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5edc0cb000 0x7f5edc0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 --2- 172.21.15.29:0/1491059277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5edc070380 0x7f5edc0b2380 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 >> 172.21.15.29:0/1491059277 conn(0x7f5edc0cf790 msgr2=0x7f5edc15b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.736+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 shutdown_connections 2023-10-16T14:33:42.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.737+0000 7f5edbfff640 1 -- 172.21.15.29:0/1491059277 wait complete. 2023-10-16T14:33:42.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:33:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.978+0000 7f178e64d640 1 Processor -- start 2023-10-16T14:33:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.978+0000 7f178e64d640 1 -- start start 2023-10-16T14:33:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178e64d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178806c040 0x7f178806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178e64d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f17880ad970 0x7f178815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178e64d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178806c960 con 0x7f178815ebc0 2023-10-16T14:33:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178e64d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178815b9d0 con 0x7f17880ad970 2023-10-16T14:33:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178e64d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178815bcb0 con 0x7f178806c040 2023-10-16T14:33:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178ce4a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f17880ad970 0x7f178815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178de4c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f178815ebc0 0x7f17880cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38658/0 (socket says 172.21.15.29:38658) 2023-10-16T14:33:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.980+0000 7f178de4c640 1 -- 172.21.15.29:0/2766311882 learned_addr learned my addr 172.21.15.29:0/2766311882 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.981+0000 7f178d64b640 1 --2- 172.21.15.29:0/2766311882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178806c040 0x7f178806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.981+0000 7f178ce4a640 1 -- 172.21.15.29:0/2766311882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178806c040 msgr2=0x7f178806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.981+0000 7f178ce4a640 1 --2- 172.21.15.29:0/2766311882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178806c040 0x7f178806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.981+0000 7f178ce4a640 1 -- 172.21.15.29:0/2766311882 >> v1:172.21.15.29:6789/0 conn(0x7f178815ebc0 legacy=0x7f17880cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.981+0000 7f178ce4a640 1 -- 172.21.15.29:0/2766311882 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178815bf90 con 0x7f17880ad970 2023-10-16T14:33:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.981+0000 7f177ffff640 1 -- 172.21.15.29:0/2766311882 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367203993 0 0) 0x7f178806c960 con 0x7f178815ebc0 2023-10-16T14:33:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.982+0000 7f178ce4a640 1 --2- 172.21.15.29:0/2766311882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f178815e680 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f17780097d0 tx=0x7f177802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb50244367f2935 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.982+0000 7f177ffff640 1 -- 172.21.15.29:0/2766311882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778031800 con 0x7f17880ad970 2023-10-16T14:33:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f177ffff640 1 -- 172.21.15.29:0/2766311882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778031960 con 0x7f17880ad970 2023-10-16T14:33:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f177ffff640 1 -- 172.21.15.29:0/2766311882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778031cf0 con 0x7f17880ad970 2023-10-16T14:33:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 -- 172.21.15.29:0/2766311882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 msgr2=0x7f178815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 --2- 172.21.15.29:0/2766311882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f178815e680 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f17780097d0 tx=0x7f177802b8a0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 -- 172.21.15.29:0/2766311882 shutdown_connections 2023-10-16T14:33:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 --2- 172.21.15.29:0/2766311882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f178815e680 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 --2- 172.21.15.29:0/2766311882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178806c040 0x7f178806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 -- 172.21.15.29:0/2766311882 >> 172.21.15.29:0/2766311882 conn(0x7f17880ce6a0 msgr2=0x7f17880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 -- 172.21.15.29:0/2766311882 shutdown_connections 2023-10-16T14:33:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.983+0000 7f178e64d640 1 -- 172.21.15.29:0/2766311882 wait complete. 2023-10-16T14:33:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.985+0000 7f178e64d640 1 Processor -- start 2023-10-16T14:33:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.985+0000 7f178e64d640 1 -- start start 2023-10-16T14:33:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178e64d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178806c040 0x7f17880b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178e64d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f17880c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178e64d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178815ebc0 0x7f178814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178e64d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880c7e40 con 0x7f17880ad970 2023-10-16T14:33:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178e64d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880c6860 con 0x7f178806c040 2023-10-16T14:33:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178e64d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17880c6b40 con 0x7f178815ebc0 2023-10-16T14:33:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178ce4a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f17880c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178ce4a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f17880c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57138/0 (socket says 172.21.15.29:57138) 2023-10-16T14:33:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.986+0000 7f178ce4a640 1 -- 172.21.15.29:0/1339477402 learned_addr learned my addr 172.21.15.29:0/1339477402 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178ce4a640 1 -- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178815ebc0 msgr2=0x7f178814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178d64b640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178806c040 0x7f17880b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178ce4a640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178815ebc0 0x7f178814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178ce4a640 1 -- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178806c040 msgr2=0x7f17880b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178ce4a640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178806c040 0x7f17880b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178ce4a640 1 -- 172.21.15.29:0/1339477402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778009400 con 0x7f17880ad970 2023-10-16T14:33:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f178ce4a640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f17880c7900 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1778034040 tx=0x7f17780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.987+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177803ad20 con 0x7f17880ad970 2023-10-16T14:33:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.988+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778044400 con 0x7f17880ad970 2023-10-16T14:33:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.988+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177803aea0 con 0x7f17880ad970 2023-10-16T14:33:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.988+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17880c6e20 con 0x7f17880ad970 2023-10-16T14:33:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.988+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17880c73e0 con 0x7f17880ad970 2023-10-16T14:33:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.989+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178806b330 con 0x7f17880ad970 2023-10-16T14:33:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.990+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f177804d440 con 0x7f17880ad970 2023-10-16T14:33:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.991+0000 7f177dffb640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f176c06afc0 0x7f176c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.991+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f177802e070 con 0x7f17880ad970 2023-10-16T14:33:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.992+0000 7f178d64b640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f176c06afc0 0x7f176c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.993+0000 7f178d64b640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f176c06afc0 0x7f176c06d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1784006fd0 tx=0x7f1784008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:42.995+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1778076b00 con 0x7f17880ad970 2023-10-16T14:33:43.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.189+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f17880c75d0 con 0x7f17880ad970 2023-10-16T14:33:43.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.191+0000 7f177dffb640 1 -- 172.21.15.29:0/1339477402 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1778035090 con 0x7f17880ad970 2023-10-16T14:33:43.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:43.194 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:33:43.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:43.194 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:33:43.194 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:33:43.195 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:43.195 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:33:43.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f176c06afc0 msgr2=0x7f176c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f176c06afc0 0x7f176c06d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1784006fd0 tx=0x7f1784008040 comp rx=0 tx=0).stop 2023-10-16T14:33:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 msgr2=0x7f17880c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f17880c7900 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1778034040 tx=0x7f17780046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 shutdown_connections 2023-10-16T14:33:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f176c06afc0 0x7f176c06d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17880ad970 0x7f17880c7900 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f178815ebc0 0x7f178814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 --2- 172.21.15.29:0/1339477402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f178806c040 0x7f17880b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.193+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 >> 172.21.15.29:0/1339477402 conn(0x7f17880ce6a0 msgr2=0x7f17880b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.194+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 shutdown_connections 2023-10-16T14:33:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.194+0000 7f178e64d640 1 -- 172.21.15.29:0/1339477402 wait complete. 2023-10-16T14:33:43.215 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:33:43.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:33:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.445+0000 7fd3a9e75640 1 Processor -- start 2023-10-16T14:33:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.445+0000 7fd3a9e75640 1 -- start start 2023-10-16T14:33:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.447+0000 7fd3a9e75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a4165570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.447+0000 7fd3a9e75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3a40c18e0 0x7fd3a40c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.447+0000 7fd3a9e75640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a40cae70 con 0x7fd3a40adec0 2023-10-16T14:33:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.447+0000 7fd3a9e75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a4165de0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.447+0000 7fd3a9e75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a406b860 con 0x7fd3a4071460 2023-10-16T14:33:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3a40c18e0 0x7fd3a40c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3a40c18e0 0x7fd3a40c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57154/0 (socket says 172.21.15.29:57154) 2023-10-16T14:33:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 -- 172.21.15.29:0/2476381472 learned_addr learned my addr 172.21.15.29:0/2476381472 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a3fff640 1 --2- 172.21.15.29:0/2476381472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a4165570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 -- 172.21.15.29:0/2476381472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 msgr2=0x7fd3a4165570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 --2- 172.21.15.29:0/2476381472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a4165570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 -- 172.21.15.29:0/2476381472 >> v1:172.21.15.170:6789/0 conn(0x7fd3a40adec0 legacy=0x7fd3a4164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.448+0000 7fd3a9674640 1 -- 172.21.15.29:0/2476381472 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a406b580 con 0x7fd3a40c18e0 2023-10-16T14:33:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.449+0000 7fd3a9674640 1 --2- 172.21.15.29:0/2476381472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a40c8f00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009e50 tx=0x7fd39c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f5ea1eb1e49399 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.450+0000 7fd3a37fe640 1 -- 172.21.15.29:0/2476381472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c02cb00 con 0x7fd3a40c18e0 2023-10-16T14:33:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.450+0000 7fd3a37fe640 1 -- 172.21.15.29:0/2476381472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c02cc60 con 0x7fd3a40c18e0 2023-10-16T14:33:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.450+0000 7fd3a37fe640 1 -- 172.21.15.29:0/2476381472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0343f0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.450+0000 7fd3a9e75640 1 -- 172.21.15.29:0/2476381472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 msgr2=0x7fd3a40c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.450+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/2476381472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a40c8f00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009e50 tx=0x7fd39c02bf60 comp rx=0 tx=0).stop 2023-10-16T14:33:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.451+0000 7fd3a9e75640 1 -- 172.21.15.29:0/2476381472 shutdown_connections 2023-10-16T14:33:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.451+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/2476381472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a40c8f00 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.451+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/2476381472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a4165570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.451+0000 7fd3a9e75640 1 -- 172.21.15.29:0/2476381472 >> 172.21.15.29:0/2476381472 conn(0x7fd3a40ce820 msgr2=0x7fd3a40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.451+0000 7fd3a9e75640 1 -- 172.21.15.29:0/2476381472 shutdown_connections 2023-10-16T14:33:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.451+0000 7fd3a9e75640 1 -- 172.21.15.29:0/2476381472 wait complete. 2023-10-16T14:33:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.453+0000 7fd3a9e75640 1 Processor -- start 2023-10-16T14:33:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.453+0000 7fd3a9e75640 1 -- start start 2023-10-16T14:33:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.454+0000 7fd3a9e75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a40c2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.454+0000 7fd3a9e75640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a40adec0 0x7fd3a4163460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9e75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a415e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9e75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a41639a0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9e75640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a40c2c60 con 0x7fd3a40adec0 2023-10-16T14:33:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9e75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a40c2f40 con 0x7fd3a4071460 2023-10-16T14:33:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a415e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a415e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57168/0 (socket says 172.21.15.29:57168) 2023-10-16T14:33:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 -- 172.21.15.29:0/1635426240 learned_addr learned my addr 172.21.15.29:0/1635426240 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a3fff640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a40adec0 0x7fd3a4163460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a8e73640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a40c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 -- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 msgr2=0x7fd3a40c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a40c2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 -- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a40adec0 msgr2=0x7fd3a4163460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a40adec0 0x7fd3a4163460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.455+0000 7fd3a9674640 1 -- 172.21.15.29:0/1635426240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c009b30 con 0x7fd3a40c18e0 2023-10-16T14:33:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.456+0000 7fd3a9674640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a415e660 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009e50 tx=0x7fd39c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.456+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c040670 con 0x7fd3a40c18e0 2023-10-16T14:33:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.456+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c0407d0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.456+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c040b80 con 0x7fd3a40c18e0 2023-10-16T14:33:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.456+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a40c3220 con 0x7fd3a40c18e0 2023-10-16T14:33:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.456+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a40c37e0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.457+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a40c8cc0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.459+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd39c040ce0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.460+0000 7fd3a17fa640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd38806afc0 0x7fd38806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.460+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd39c0ac4f0 con 0x7fd3a40c18e0 2023-10-16T14:33:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.461+0000 7fd3a8e73640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd38806afc0 0x7fd38806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.462+0000 7fd3a8e73640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd38806afc0 0x7fd38806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40c2400 tx=0x7fd398009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.464+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd39c077860 con 0x7fd3a40c18e0 2023-10-16T14:33:43.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:43.657+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd3a404e9b0 con 0x7fd3a40c18e0 2023-10-16T14:33:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.273+0000 7fd3a17fa640 1 -- 172.21.15.29:0/1635426240 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd39c009510 con 0x7fd3a40c18e0 2023-10-16T14:33:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:33:44.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.277+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd38806afc0 msgr2=0x7fd38806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.277+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd38806afc0 0x7fd38806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40c2400 tx=0x7fd398009040 comp rx=0 tx=0).stop 2023-10-16T14:33:44.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.277+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 msgr2=0x7fd3a415e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.277+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a415e660 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009e50 tx=0x7fd39c004a40 comp rx=0 tx=0).stop 2023-10-16T14:33:44.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 shutdown_connections 2023-10-16T14:33:44.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd38806afc0 0x7fd38806d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3a40c18e0 0x7fd3a415e660 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3a40adec0 0x7fd3a4163460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 --2- 172.21.15.29:0/1635426240 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3a4071460 0x7fd3a40c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 >> 172.21.15.29:0/1635426240 conn(0x7fd3a40ce820 msgr2=0x7fd3a415f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:44.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 shutdown_connections 2023-10-16T14:33:44.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.278+0000 7fd3a9e75640 1 -- 172.21.15.29:0/1635426240 wait complete. 2023-10-16T14:33:44.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:33:44.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.522+0000 7f65f4e25640 1 Processor -- start 2023-10-16T14:33:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.523+0000 7f65f4e25640 1 -- start start 2023-10-16T14:33:44.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.524+0000 7f65f4e25640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65f00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65f4e25640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00cc850 con 0x7f65f015ac70 2023-10-16T14:33:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65f4e25640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00ccb30 con 0x7f65f00b4160 2023-10-16T14:33:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65f4e25640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00cce10 con 0x7f65f006a4b0 2023-10-16T14:33:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65ef7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65f015ac70 0x7f65f0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44632/0 (socket says 172.21.15.29:44632) 2023-10-16T14:33:44.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65ef7fe640 1 -- 172.21.15.29:0/3216036621 learned_addr learned my addr 172.21.15.29:0/3216036621 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:44.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65e6ffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65f00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.525+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845990168 0 0) 0x7f65f00cc850 con 0x7f65f015ac70 2023-10-16T14:33:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65dc003520 con 0x7f65f015ac70 2023-10-16T14:33:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704006519 0 0) 0x7f65dc003520 con 0x7f65f015ac70 2023-10-16T14:33:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 msgr2=0x7f65f00cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65f00cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 >> v1:172.21.15.170:6789/0 conn(0x7f65f00b4160 legacy=0x7f65f006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00cd0f0 con 0x7f65f015ac70 2023-10-16T14:33:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f65e0001670 con 0x7f65f015ac70 2023-10-16T14:33:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 >> v1:172.21.15.29:6789/0 conn(0x7f65f015ac70 legacy=0x7f65f0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65dc0039a0 0x7f65dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65dc0047c0 0x7f65dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00cce10 con 0x7f65dc0039a0 2023-10-16T14:33:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc00ba60 con 0x7f65dc0047c0 2023-10-16T14:33:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc00bc20 con 0x7f65f006a4b0 2023-10-16T14:33:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.526+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65dc0039a0 0x7f65dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65ef7fe640 1 -- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 msgr2=0x7f65dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65e6ffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65dc0047c0 0x7f65dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65effff640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65ef7fe640 1 -- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65dc0047c0 msgr2=0x7f65dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65dc0047c0 0x7f65dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65ef7fe640 1 -- 172.21.15.29:0/3216036621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc003520 con 0x7f65dc0039a0 2023-10-16T14:33:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65e6ffd640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65dc0047c0 0x7f65dc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.527+0000 7f65effff640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65f006a4b0 0x7f65dc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.528+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65dc0039a0 0x7f65dc003dc0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f65e0001eb0 tx=0x7f65dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947d85aea7b353cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0005ad0 con 0x7f65dc0039a0 2023-10-16T14:33:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0005c30 con 0x7f65dc0039a0 2023-10-16T14:33:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65eeffd640 1 -- 172.21.15.29:0/3216036621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e002c7a0 con 0x7f65dc0039a0 2023-10-16T14:33:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65f4e25640 1 -- 172.21.15.29:0/3216036621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65dc0039a0 msgr2=0x7f65dc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65f4e25640 1 --2- 172.21.15.29:0/3216036621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65dc0039a0 0x7f65dc003dc0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f65e0001eb0 tx=0x7f65dc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65effff640 1 -- 172.21.15.29:0/3216036621 reap_dead start 2023-10-16T14:33:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65f4e25640 1 -- 172.21.15.29:0/3216036621 shutdown_connections 2023-10-16T14:33:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65f4e25640 1 -- 172.21.15.29:0/3216036621 >> 172.21.15.29:0/3216036621 conn(0x7f65f00ce6a0 msgr2=0x7f65f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65f4e25640 1 -- 172.21.15.29:0/3216036621 shutdown_connections 2023-10-16T14:33:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.529+0000 7f65f4e25640 1 -- 172.21.15.29:0/3216036621 wait complete. 2023-10-16T14:33:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.531+0000 7f65f4e25640 1 Processor -- start 2023-10-16T14:33:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.531+0000 7f65f4e25640 1 -- start start 2023-10-16T14:33:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.532+0000 7f65f4e25640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65dc0039a0 0x7f65f0150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.532+0000 7f65f4e25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 0x7f65f00c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.532+0000 7f65f4e25640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65f00c6c20 0x7f65f00ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.532+0000 7f65f4e25640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00c7e50 con 0x7f65f00c67c0 2023-10-16T14:33:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.532+0000 7f65f4e25640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0151130 con 0x7f65f00c6c20 2023-10-16T14:33:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.532+0000 7f65f4e25640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00c71d0 con 0x7f65dc0039a0 2023-10-16T14:33:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 0x7f65f00c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 0x7f65f00c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42758/0 (socket says 172.21.15.29:42758) 2023-10-16T14:33:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 -- 172.21.15.29:0/2960304723 learned_addr learned my addr 172.21.15.29:0/2960304723 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65dc0039a0 0x7f65f0150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 -- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65dc0039a0 msgr2=0x7f65f0150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65dc0039a0 0x7f65f0150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 -- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65f00c6c20 msgr2=0x7f65f00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65f00c6c20 0x7f65f00ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 -- 172.21.15.29:0/2960304723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00c74b0 con 0x7f65f00c67c0 2023-10-16T14:33:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65dc0039a0 0x7f65f0150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.533+0000 7f65e6ffd640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 0x7f65f00c9e70 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f65d8010900 tx=0x7f65d8010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.534+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d800ebc0 con 0x7f65f00c67c0 2023-10-16T14:33:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.534+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d800ed20 con 0x7f65f00c67c0 2023-10-16T14:33:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.534+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d801c620 con 0x7f65f00c67c0 2023-10-16T14:33:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.534+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f00c7790 con 0x7f65f00c67c0 2023-10-16T14:33:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.534+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f00c3570 con 0x7f65f00c67c0 2023-10-16T14:33:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.535+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00c80c0 con 0x7f65f00c67c0 2023-10-16T14:33:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.537+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65d801c780 con 0x7f65f00c67c0 2023-10-16T14:33:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.537+0000 7f65ecff9640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f65d006af90 0x7f65d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.538+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f65d8089ae0 con 0x7f65f00c67c0 2023-10-16T14:33:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.538+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f65d006af90 0x7f65d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.539+0000 7f65ef7fe640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f65d006af90 0x7f65d006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65e0034000 tx=0x7f65e0009030 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.541+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65d8054cd0 con 0x7f65f00c67c0 2023-10-16T14:33:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:44.735+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f65f00c80c0 con 0x7f65f00c67c0 2023-10-16T14:33:45.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.278+0000 7f65ecff9640 1 -- 172.21.15.29:0/2960304723 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7f65f00c80c0 con 0x7f65f00c67c0 2023-10-16T14:33:45.282 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:33:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f65d006af90 msgr2=0x7f65d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f65d006af90 0x7f65d006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65e0034000 tx=0x7f65e0009030 comp rx=0 tx=0).stop 2023-10-16T14:33:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 msgr2=0x7f65f00c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 0x7f65f00c9e70 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f65d8010900 tx=0x7f65d8010dc0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 shutdown_connections 2023-10-16T14:33:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f65d006af90 0x7f65d006d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65f00c67c0 0x7f65f00c9e70 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65f00c6c20 0x7f65f00ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 --2- 172.21.15.29:0/2960304723 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65dc0039a0 0x7f65f0150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.282+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 >> 172.21.15.29:0/2960304723 conn(0x7f65f00ce6a0 msgr2=0x7f65f01585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:45.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.283+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 shutdown_connections 2023-10-16T14:33:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.283+0000 7f65f4e25640 1 -- 172.21.15.29:0/2960304723 wait complete. 2023-10-16T14:33:45.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:33:45.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.509+0000 7f11af93c640 1 Processor -- start 2023-10-16T14:33:45.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.510+0000 7f11af93c640 1 -- start start 2023-10-16T14:33:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.511+0000 7f11af93c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11a8159ab0 0x7f11a80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.511+0000 7f11af93c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a8159e90 0x7f11a806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.511+0000 7f11af93c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a80cd1f0 con 0x7f11a806c140 2023-10-16T14:33:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.511+0000 7f11af93c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a815d890 con 0x7f11a8159ab0 2023-10-16T14:33:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.511+0000 7f11af93c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a80cd4d0 con 0x7f11a8159e90 2023-10-16T14:33:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11a8159ab0 0x7f11a80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11a8159ab0 0x7f11a80ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42772/0 (socket says 172.21.15.29:42772) 2023-10-16T14:33:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 -- 172.21.15.29:0/241970848 learned_addr learned my addr 172.21.15.29:0/241970848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11aced7640 1 --2- 172.21.15.29:0/241970848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a8159e90 0x7f11a806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 -- 172.21.15.29:0/241970848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a8159e90 msgr2=0x7f11a806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 --2- 172.21.15.29:0/241970848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a8159e90 0x7f11a806fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 -- 172.21.15.29:0/241970848 >> v1:172.21.15.170:6789/0 conn(0x7f11a806c140 legacy=0x7f11a815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.512+0000 7f11ad6d8640 1 -- 172.21.15.29:0/241970848 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a80cd7b0 con 0x7f11a8159ab0 2023-10-16T14:33:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.514+0000 7f11ad6d8640 1 --2- 172.21.15.29:0/241970848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80ce370 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f11a40098a0 tx=0x7f11a402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f028818c26083b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.514+0000 7f119ffff640 1 -- 172.21.15.29:0/241970848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4031520 con 0x7f11a8159ab0 2023-10-16T14:33:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.514+0000 7f119ffff640 1 -- 172.21.15.29:0/241970848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a4031680 con 0x7f11a8159ab0 2023-10-16T14:33:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.514+0000 7f119ffff640 1 -- 172.21.15.29:0/241970848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4031a10 con 0x7f11a8159ab0 2023-10-16T14:33:45.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 -- 172.21.15.29:0/241970848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 msgr2=0x7f11a80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 --2- 172.21.15.29:0/241970848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80ce370 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f11a40098a0 tx=0x7f11a402b7c0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 -- 172.21.15.29:0/241970848 shutdown_connections 2023-10-16T14:33:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 --2- 172.21.15.29:0/241970848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a8159e90 0x7f11a806fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 --2- 172.21.15.29:0/241970848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80ce370 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 -- 172.21.15.29:0/241970848 >> 172.21.15.29:0/241970848 conn(0x7f11a80ce9b0 msgr2=0x7f11a80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:45.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 -- 172.21.15.29:0/241970848 shutdown_connections 2023-10-16T14:33:45.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.515+0000 7f11af93c640 1 -- 172.21.15.29:0/241970848 wait complete. 2023-10-16T14:33:45.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.517+0000 7f11af93c640 1 Processor -- start 2023-10-16T14:33:45.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.517+0000 7f11af93c640 1 -- start start 2023-10-16T14:33:45.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11af93c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a806c140 0x7f11a81581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:45.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11af93c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11af93c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a8159e90 0x7f11a8157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11af93c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a815b930 con 0x7f11a8159ab0 2023-10-16T14:33:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11af93c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a80c4b10 con 0x7f11a8159e90 2023-10-16T14:33:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11af93c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a80c4df0 con 0x7f11a806c140 2023-10-16T14:33:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11aced7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11aced7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42786/0 (socket says 172.21.15.29:42786) 2023-10-16T14:33:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.518+0000 7f11aced7640 1 -- 172.21.15.29:0/2166364741 learned_addr learned my addr 172.21.15.29:0/2166364741 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11ad6d8640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a806c140 0x7f11a81581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aded9640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a8159e90 0x7f11a8157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aced7640 1 -- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a806c140 msgr2=0x7f11a81581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aced7640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a806c140 0x7f11a81581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aced7640 1 -- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a8159e90 msgr2=0x7f11a8157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aced7640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a8159e90 0x7f11a8157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aced7640 1 -- 172.21.15.29:0/2166364741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a40094d0 con 0x7f11a8159ab0 2023-10-16T14:33:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11ad6d8640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a806c140 0x7f11a81581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aded9640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a8159e90 0x7f11a8157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f11aced7640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80c44f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f119800d800 tx=0x7f119800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.519+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119800b6c0 con 0x7f11a8159ab0 2023-10-16T14:33:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.520+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119800b820 con 0x7f11a8159ab0 2023-10-16T14:33:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.520+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a80c50d0 con 0x7f11a8159ab0 2023-10-16T14:33:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.520+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119800bbb0 con 0x7f11a8159ab0 2023-10-16T14:33:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.520+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a80c53b0 con 0x7f11a8159ab0 2023-10-16T14:33:45.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.521+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a815b780 con 0x7f11a8159ab0 2023-10-16T14:33:45.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.522+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f119800bd10 con 0x7f11a8159ab0 2023-10-16T14:33:45.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.523+0000 7f119dffb640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f118c06af90 0x7f118c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:45.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.523+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1198089d70 con 0x7f11a8159ab0 2023-10-16T14:33:45.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.524+0000 7f11ad6d8640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f118c06af90 0x7f118c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:45.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.525+0000 7f11ad6d8640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f118c06af90 0x7f118c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f11a4035000 tx=0x7f11a4030000 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:45.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.528+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1198054cd0 con 0x7f11a8159ab0 2023-10-16T14:33:45.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:45.761+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f11a804e160 con 0x7f11a8159ab0 2023-10-16T14:33:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.429+0000 7f119dffb640 1 -- 172.21.15.29:0/2166364741 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v46) v1 ==== 174+0+0 (secure 0 0 0) 0x7f11980582f0 con 0x7f11a8159ab0 2023-10-16T14:33:46.436 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 12 and data pool 13 2023-10-16T14:33:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.436+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f118c06af90 msgr2=0x7f118c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.436+0000 7f11af93c640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f118c06af90 0x7f118c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f11a4035000 tx=0x7f11a4030000 comp rx=0 tx=0).stop 2023-10-16T14:33:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.436+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 msgr2=0x7f11a80c44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.436+0000 7f11af93c640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80c44f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f119800d800 tx=0x7f119800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.437+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 shutdown_connections 2023-10-16T14:33:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.437+0000 7f11af93c640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f118c06af90 0x7f118c06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.437+0000 7f11af93c640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a8159ab0 0x7f11a80c44f0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.437+0000 7f11af93c640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a8159e90 0x7f11a8157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.437+0000 7f11af93c640 1 --2- 172.21.15.29:0/2166364741 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a806c140 0x7f11a81581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.437+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 >> 172.21.15.29:0/2166364741 conn(0x7f11a80ce9b0 msgr2=0x7f11a80704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.438+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 shutdown_connections 2023-10-16T14:33:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.438+0000 7f11af93c640 1 -- 172.21.15.29:0/2166364741 wait complete. 2023-10-16T14:33:46.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.658+0000 7f32f0410640 1 Processor -- start 2023-10-16T14:33:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.658+0000 7f32f0410640 1 -- start start 2023-10-16T14:33:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32f0410640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32e8160420 0x7f32e80ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32f0410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e8160800 0x7f32e80cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32f0410640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f32e80adc40 0x7f32e806adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32f0410640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e80cde30 con 0x7f32e8160420 2023-10-16T14:33:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32f0410640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e806e170 con 0x7f32e80adc40 2023-10-16T14:33:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32f0410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80cd460 con 0x7f32e8160800 2023-10-16T14:33:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32ee1ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32e8160420 0x7f32e80ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32ee1ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32e8160420 0x7f32e80ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42796/0 (socket says 172.21.15.29:42796) 2023-10-16T14:33:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.660+0000 7f32ee1ac640 1 -- 172.21.15.29:0/2526780022 learned_addr learned my addr 172.21.15.29:0/2526780022 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ed9ab640 1 --2- 172.21.15.29:0/2526780022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e8160800 0x7f32e80cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ee9ad640 1 --2- 172.21.15.29:0/2526780022 >> v2:172.21.15.170:3300/0 conn(0x7f32e80adc40 0x7f32e806adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ee1ac640 1 -- 172.21.15.29:0/2526780022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e8160800 msgr2=0x7f32e80cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2526780022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e8160800 0x7f32e80cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ee1ac640 1 -- 172.21.15.29:0/2526780022 >> v2:172.21.15.170:3300/0 conn(0x7f32e80adc40 msgr2=0x7f32e806adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2526780022 >> v2:172.21.15.170:3300/0 conn(0x7f32e80adc40 0x7f32e806adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.661+0000 7f32ee1ac640 1 -- 172.21.15.29:0/2526780022 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e806e3c0 con 0x7f32e8160420 2023-10-16T14:33:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.662+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2526780022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e80ae170 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f32e80b1ae0 tx=0x7f32e402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24bd7c2d957eb72 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.662+0000 7f32ed1aa640 1 -- 172.21.15.29:0/2526780022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4031860 con 0x7f32e8160420 2023-10-16T14:33:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.662+0000 7f32ed1aa640 1 -- 172.21.15.29:0/2526780022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e40319c0 con 0x7f32e8160420 2023-10-16T14:33:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.662+0000 7f32ed1aa640 1 -- 172.21.15.29:0/2526780022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4031d50 con 0x7f32e8160420 2023-10-16T14:33:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 -- 172.21.15.29:0/2526780022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 msgr2=0x7f32e80ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 --2- 172.21.15.29:0/2526780022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e80ae170 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f32e80b1ae0 tx=0x7f32e402b900 comp rx=0 tx=0).stop 2023-10-16T14:33:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 -- 172.21.15.29:0/2526780022 shutdown_connections 2023-10-16T14:33:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 --2- 172.21.15.29:0/2526780022 >> v2:172.21.15.170:3300/0 conn(0x7f32e80adc40 0x7f32e806adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 --2- 172.21.15.29:0/2526780022 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e8160800 0x7f32e80cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 --2- 172.21.15.29:0/2526780022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e80ae170 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 -- 172.21.15.29:0/2526780022 >> 172.21.15.29:0/2526780022 conn(0x7f32e80b0d30 msgr2=0x7f32e80b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 -- 172.21.15.29:0/2526780022 shutdown_connections 2023-10-16T14:33:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.663+0000 7f32f0410640 1 -- 172.21.15.29:0/2526780022 wait complete. 2023-10-16T14:33:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.665+0000 7f32f0410640 1 Processor -- start 2023-10-16T14:33:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.665+0000 7f32f0410640 1 -- start start 2023-10-16T14:33:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32f0410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e80adc40 0x7f32e80c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32f0410640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e81f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32f0410640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32e8160800 0x7f32e80c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32f0410640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81f28d0 con 0x7f32e8160420 2023-10-16T14:33:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32f0410640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80c5a40 con 0x7f32e8160800 2023-10-16T14:33:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32f0410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80c4ec0 con 0x7f32e80adc40 2023-10-16T14:33:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32ed9ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e81f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32ed9ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e81f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42808/0 (socket says 172.21.15.29:42808) 2023-10-16T14:33:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.666+0000 7f32ed9ab640 1 -- 172.21.15.29:0/2422517257 learned_addr learned my addr 172.21.15.29:0/2422517257 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e80adc40 0x7f32e80c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ed9ab640 1 -- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e80adc40 msgr2=0x7f32e80c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ed9ab640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e80adc40 0x7f32e80c4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ed9ab640 1 -- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32e8160800 msgr2=0x7f32e80c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ee9ad640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32e8160800 0x7f32e80c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ed9ab640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32e8160800 0x7f32e80c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ed9ab640 1 -- 172.21.15.29:0/2422517257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e40094d0 con 0x7f32e8160420 2023-10-16T14:33:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e80adc40 0x7f32e80c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32ed9ab640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e81f2390 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c7f0 tx=0x7f32d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004220 con 0x7f32e8160420 2023-10-16T14:33:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8004380 con 0x7f32e8160420 2023-10-16T14:33:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8007840 con 0x7f32e8160420 2023-10-16T14:33:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.667+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e80c51a0 con 0x7f32e8160420 2023-10-16T14:33:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.668+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e80c5760 con 0x7f32e8160420 2023-10-16T14:33:46.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.669+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e804e160 con 0x7f32e8160420 2023-10-16T14:33:46.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.670+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f32d8007bb0 con 0x7f32e8160420 2023-10-16T14:33:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.671+0000 7f32deffd640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f32cc06af90 0x7f32cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.672+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32d8089710 con 0x7f32e8160420 2023-10-16T14:33:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.672+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f32cc06af90 0x7f32cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.673+0000 7f32ee1ac640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f32cc06af90 0x7f32cc06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32e4036000 tx=0x7f32e403c040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:46.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.676+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32d8054590 con 0x7f32e8160420 2023-10-16T14:33:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.875+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32e80b51c0 con 0x7f32e8160420 2023-10-16T14:33:46.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.878+0000 7f32deffd640 1 -- 172.21.15.29:0/2422517257 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f32d8007e60 con 0x7f32e8160420 2023-10-16T14:33:46.881 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:46.882 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":110,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:45.762701+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:46.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f32cc06af90 msgr2=0x7f32cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f32cc06af90 0x7f32cc06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32e4036000 tx=0x7f32e403c040 comp rx=0 tx=0).stop 2023-10-16T14:33:46.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 msgr2=0x7f32e81f2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:46.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e81f2390 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c7f0 tx=0x7f32d800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 shutdown_connections 2023-10-16T14:33:46.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f32cc06af90 0x7f32cc06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32e8160420 0x7f32e81f2390 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32e8160800 0x7f32e80c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 --2- 172.21.15.29:0/2422517257 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32e80adc40 0x7f32e80c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:46.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 >> 172.21.15.29:0/2422517257 conn(0x7f32e80b0d30 msgr2=0x7f32e80af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:46.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 shutdown_connections 2023-10-16T14:33:46.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:46.881+0000 7f32f0410640 1 -- 172.21.15.29:0/2422517257 wait complete. 2023-10-16T14:33:46.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.125+0000 7f4ec1685640 1 Processor -- start 2023-10-16T14:33:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.125+0000 7f4ec1685640 1 -- start start 2023-10-16T14:33:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec1685640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc0ad970 0x7f4ebc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec1685640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ebc0ccfb0 0x7f4ebc0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec1685640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc15b9d0 con 0x7f4ebc06c040 2023-10-16T14:33:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec1685640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc15bcb0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec1685640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc15bf90 con 0x7f4ebc0ad970 2023-10-16T14:33:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec0e84640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ebc0ccfb0 0x7f4ebc0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec0e84640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ebc0ccfb0 0x7f4ebc0cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42816/0 (socket says 172.21.15.29:42816) 2023-10-16T14:33:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4ec0e84640 1 -- 172.21.15.29:0/518867461 learned_addr learned my addr 172.21.15.29:0/518867461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.127+0000 7f4eb3fff640 1 --2- 172.21.15.29:0/518867461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc0ad970 0x7f4ebc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.128+0000 7f4ec0e84640 1 -- 172.21.15.29:0/518867461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc0ad970 msgr2=0x7f4ebc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.128+0000 7f4ec0e84640 1 --2- 172.21.15.29:0/518867461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc0ad970 0x7f4ebc0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.128+0000 7f4ec0e84640 1 -- 172.21.15.29:0/518867461 >> v1:172.21.15.170:6789/0 conn(0x7f4ebc06c040 legacy=0x7f4ebc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.128+0000 7f4ec0e84640 1 -- 172.21.15.29:0/518867461 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0cc270 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.129+0000 7f4ec0e84640 1 --2- 172.21.15.29:0/518867461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc0cd440 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009e80 tx=0x7f4eb402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29508c9e297b35b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.129+0000 7f4ebb7fe640 1 -- 172.21.15.29:0/518867461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb402ca50 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.129+0000 7f4ebb7fe640 1 -- 172.21.15.29:0/518867461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb402cbb0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.129+0000 7f4ebb7fe640 1 -- 172.21.15.29:0/518867461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40343f0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 -- 172.21.15.29:0/518867461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 msgr2=0x7f4ebc0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 --2- 172.21.15.29:0/518867461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc0cd440 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009e80 tx=0x7f4eb402beb0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 -- 172.21.15.29:0/518867461 shutdown_connections 2023-10-16T14:33:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 --2- 172.21.15.29:0/518867461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc0cd440 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 --2- 172.21.15.29:0/518867461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc0ad970 0x7f4ebc0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 -- 172.21.15.29:0/518867461 >> 172.21.15.29:0/518867461 conn(0x7f4ebc0ce6a0 msgr2=0x7f4ebc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 -- 172.21.15.29:0/518867461 shutdown_connections 2023-10-16T14:33:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.130+0000 7f4ec1685640 1 -- 172.21.15.29:0/518867461 wait complete. 2023-10-16T14:33:47.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.132+0000 7f4ec1685640 1 Processor -- start 2023-10-16T14:33:47.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.132+0000 7f4ec1685640 1 -- start start 2023-10-16T14:33:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec1685640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc06c040 0x7f4ebc153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec1685640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ebc0ad970 0x7f4ebc0ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec1685640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec1685640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0ad590 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec1685640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0c6f70 con 0x7f4ebc0ad970 2023-10-16T14:33:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec1685640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc154160 con 0x7f4ebc06c040 2023-10-16T14:33:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec0e84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.133+0000 7f4ec0e84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42830/0 (socket says 172.21.15.29:42830) 2023-10-16T14:33:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 -- 172.21.15.29:0/3681864967 learned_addr learned my addr 172.21.15.29:0/3681864967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ebbfff640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc06c040 0x7f4ebc153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 -- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc06c040 msgr2=0x7f4ebc153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc06c040 0x7f4ebc153c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 -- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ebc0ad970 msgr2=0x7f4ebc0ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4eb3fff640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ebc0ad970 0x7f4ebc0ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ebc0ad970 0x7f4ebc0ac9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 -- 172.21.15.29:0/3681864967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb4009b30 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.134+0000 7f4ec0e84640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc154340 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009fa0 tx=0x7f4eb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:47.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.135+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4040670 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.135+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb40407d0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.135+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4040b80 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.135+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc0c6960 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.135+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc0c6c00 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.136+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc0c67c0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.137+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4eb4040ce0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.138+0000 7f4eb97fa640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e9c06afc0 0x7f4e9c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.139+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4eb40acb20 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.139+0000 7f4ebbfff640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e9c06afc0 0x7f4e9c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.140+0000 7f4ebbfff640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e9c06afc0 0x7f4e9c06d480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00a750 tx=0x7f4eac008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.143+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ebc0c67c0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.348+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ebc0c67c0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.350+0000 7f4eb97fa640 1 -- 172.21.15.29:0/3681864967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16571 (secure 0 0 0) 0x7f4ebc0c67c0 con 0x7f4ebc0ccfb0 2023-10-16T14:33:47.353 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:47.353 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":111,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:47.320794+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.352+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e9c06afc0 msgr2=0x7f4e9c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.352+0000 7f4ec1685640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e9c06afc0 0x7f4e9c06d480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00a750 tx=0x7f4eac008040 comp rx=0 tx=0).stop 2023-10-16T14:33:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.352+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 msgr2=0x7f4ebc154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc154340 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009fa0 tx=0x7f4eb4004a40 comp rx=0 tx=0).stop 2023-10-16T14:33:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 shutdown_connections 2023-10-16T14:33:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e9c06afc0 0x7f4e9c06d480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ebc0ccfb0 0x7f4ebc154340 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ebc0ad970 0x7f4ebc0ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 --2- 172.21.15.29:0/3681864967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ebc06c040 0x7f4ebc153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 >> 172.21.15.29:0/3681864967 conn(0x7f4ebc0ce6a0 msgr2=0x7f4ebc0b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:47.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 shutdown_connections 2023-10-16T14:33:47.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.353+0000 7f4ec1685640 1 -- 172.21.15.29:0/3681864967 wait complete. 2023-10-16T14:33:47.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:33:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.586+0000 7fbf8f14e640 1 Processor -- start 2023-10-16T14:33:47.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.587+0000 7fbf8f14e640 1 -- start start 2023-10-16T14:33:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.588+0000 7fbf8f14e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbf8806fb50 0x7fbf880cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.588+0000 7fbf8f14e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf8806c9e0 0x7fbf880ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.588+0000 7fbf8f14e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf880ca560 con 0x7fbf880c9060 2023-10-16T14:33:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.588+0000 7fbf8f14e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf880c9950 con 0x7fbf8806c9e0 2023-10-16T14:33:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.588+0000 7fbf8f14e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf880c9c30 con 0x7fbf8806fb50 2023-10-16T14:33:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e94d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf8806c9e0 0x7fbf880ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e14c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf880c9060 0x7fbf8806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44646/0 (socket says 172.21.15.29:44646) 2023-10-16T14:33:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e14c640 1 -- 172.21.15.29:0/738378269 learned_addr learned my addr 172.21.15.29:0/738378269 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e94d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf8806c9e0 0x7fbf880ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42836/0 (socket says 172.21.15.29:42836) 2023-10-16T14:33:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8d94b640 1 --2- 172.21.15.29:0/738378269 >> v2:172.21.15.170:3300/0 conn(0x7fbf8806fb50 0x7fbf880cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e94d640 1 -- 172.21.15.29:0/738378269 >> v2:172.21.15.170:3300/0 conn(0x7fbf8806fb50 msgr2=0x7fbf880cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e94d640 1 --2- 172.21.15.29:0/738378269 >> v2:172.21.15.170:3300/0 conn(0x7fbf8806fb50 0x7fbf880cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e94d640 1 -- 172.21.15.29:0/738378269 >> v1:172.21.15.29:6789/0 conn(0x7fbf880c9060 legacy=0x7fbf8806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8e94d640 1 -- 172.21.15.29:0/738378269 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf880c9f10 con 0x7fbf8806c9e0 2023-10-16T14:33:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.589+0000 7fbf8d14a640 1 -- 172.21.15.29:0/738378269 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248422250 0 0) 0x7fbf880ca560 con 0x7fbf880c9060 2023-10-16T14:33:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.591+0000 7fbf8e94d640 1 --2- 172.21.15.29:0/738378269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806c9e0 0x7fbf880ccde0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbf88164cb0 tx=0x7fbf8002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542b1eaf652c0a2a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.591+0000 7fbf8d14a640 1 -- 172.21.15.29:0/738378269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8002cc80 con 0x7fbf8806c9e0 2023-10-16T14:33:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.591+0000 7fbf8d14a640 1 -- 172.21.15.29:0/738378269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8002cde0 con 0x7fbf8806c9e0 2023-10-16T14:33:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.591+0000 7fbf8d14a640 1 -- 172.21.15.29:0/738378269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80034660 con 0x7fbf8806c9e0 2023-10-16T14:33:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.591+0000 7fbf8f14e640 1 -- 172.21.15.29:0/738378269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806c9e0 msgr2=0x7fbf880ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.591+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/738378269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806c9e0 0x7fbf880ccde0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbf88164cb0 tx=0x7fbf8002bed0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.592+0000 7fbf8f14e640 1 -- 172.21.15.29:0/738378269 shutdown_connections 2023-10-16T14:33:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.592+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/738378269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806c9e0 0x7fbf880ccde0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.592+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/738378269 >> v2:172.21.15.170:3300/0 conn(0x7fbf8806fb50 0x7fbf880cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.592+0000 7fbf8f14e640 1 -- 172.21.15.29:0/738378269 >> 172.21.15.29:0/738378269 conn(0x7fbf880ce6a0 msgr2=0x7fbf880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.592+0000 7fbf8f14e640 1 -- 172.21.15.29:0/738378269 shutdown_connections 2023-10-16T14:33:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.592+0000 7fbf8f14e640 1 -- 172.21.15.29:0/738378269 wait complete. 2023-10-16T14:33:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.593+0000 7fbf8f14e640 1 Processor -- start 2023-10-16T14:33:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.594+0000 7fbf8f14e640 1 -- start start 2023-10-16T14:33:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8f14e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8806c9e0 0x7fbf880c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8f14e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 0x7fbf8815eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8f14e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf880c9060 0x7fbf880c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8f14e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880c3300 con 0x7fbf8806fb50 2023-10-16T14:33:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8f14e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880c3ba0 con 0x7fbf880c9060 2023-10-16T14:33:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8f14e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880c6770 con 0x7fbf8806c9e0 2023-10-16T14:33:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8d94b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 0x7fbf8815eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8d94b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 0x7fbf8815eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42852/0 (socket says 172.21.15.29:42852) 2023-10-16T14:33:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8d94b640 1 -- 172.21.15.29:0/2777775895 learned_addr learned my addr 172.21.15.29:0/2777775895 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8e14c640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8806c9e0 0x7fbf880c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8e94d640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf880c9060 0x7fbf880c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8d94b640 1 -- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8806c9e0 msgr2=0x7fbf880c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8d94b640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8806c9e0 0x7fbf880c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.595+0000 7fbf8d94b640 1 -- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf880c9060 msgr2=0x7fbf880c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf8d94b640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf880c9060 0x7fbf880c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf8d94b640 1 -- 172.21.15.29:0/2777775895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80009a70 con 0x7fbf8806fb50 2023-10-16T14:33:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf8e94d640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf880c9060 0x7fbf880c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf8d94b640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 0x7fbf8815eed0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800d830 tx=0x7fbf7800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7800b6c0 con 0x7fbf8806fb50 2023-10-16T14:33:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7800b820 con 0x7fbf8806fb50 2023-10-16T14:33:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7800bbb0 con 0x7fbf8806fb50 2023-10-16T14:33:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf880c3550 con 0x7fbf8806fb50 2023-10-16T14:33:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.596+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf880b41b0 con 0x7fbf8806fb50 2023-10-16T14:33:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.598+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf880c3d30 con 0x7fbf8806fb50 2023-10-16T14:33:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.599+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf7801e460 con 0x7fbf8806fb50 2023-10-16T14:33:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.600+0000 7fbf7effd640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fbf6c06af90 0x7fbf6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.600+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf78089fb0 con 0x7fbf8806fb50 2023-10-16T14:33:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.600+0000 7fbf8e14c640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fbf6c06af90 0x7fbf6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.601+0000 7fbf8e14c640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fbf6c06af90 0x7fbf6c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf84000f60 tx=0x7fbf84007c00 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.604+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf78054eb0 con 0x7fbf8806fb50 2023-10-16T14:33:47.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:47.843+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbf8804e9b0 con 0x7fbf8806fb50 2023-10-16T14:33:47.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:33:47.851 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:33:48.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.440+0000 7fbf7effd640 1 -- 172.21.15.29:0/2777775895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbf780584d0 con 0x7fbf8806fb50 2023-10-16T14:33:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fbf6c06af90 msgr2=0x7fbf6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fbf6c06af90 0x7fbf6c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf84000f60 tx=0x7fbf84007c00 comp rx=0 tx=0).stop 2023-10-16T14:33:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 msgr2=0x7fbf8815eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:48.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 0x7fbf8815eed0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800d830 tx=0x7fbf7800dcf0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 shutdown_connections 2023-10-16T14:33:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fbf6c06af90 0x7fbf6c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8806fb50 0x7fbf8815eed0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf880c9060 0x7fbf880c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 --2- 172.21.15.29:0/2777775895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8806c9e0 0x7fbf880c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 >> 172.21.15.29:0/2777775895 conn(0x7fbf880ce6a0 msgr2=0x7fbf8806b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 shutdown_connections 2023-10-16T14:33:48.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.444+0000 7fbf8f14e640 1 -- 172.21.15.29:0/2777775895 wait complete. 2023-10-16T14:33:48.469 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:33:48.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:33:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.694+0000 7fc116d73640 1 Processor -- start 2023-10-16T14:33:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.694+0000 7fc116d73640 1 -- start start 2023-10-16T14:33:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.695+0000 7fc116d73640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc11015ad70 0x7fc110069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc116d73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1100caac0 0x7fc11006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc116d73640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1100cb030 con 0x7fc1100c9ca0 2023-10-16T14:33:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc116d73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100c91c0 con 0x7fc1100caac0 2023-10-16T14:33:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc116d73640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100c94a0 con 0x7fc11015ad70 2023-10-16T14:33:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc115310640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1100caac0 0x7fc11006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc115310640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1100caac0 0x7fc11006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42854/0 (socket says 172.21.15.29:42854) 2023-10-16T14:33:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc115310640 1 -- 172.21.15.29:0/3825025773 learned_addr learned my addr 172.21.15.29:0/3825025773 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc107fff640 1 --1- 172.21.15.29:0/3825025773 >> v1:172.21.15.29:6789/0 conn(0x7fc1100c9ca0 0x7fc1100ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44650/0 (socket says 172.21.15.29:44650) 2023-10-16T14:33:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc114b0f640 1 --2- 172.21.15.29:0/3825025773 >> v2:172.21.15.170:3300/0 conn(0x7fc11015ad70 0x7fc110069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc115310640 1 -- 172.21.15.29:0/3825025773 >> v2:172.21.15.170:3300/0 conn(0x7fc11015ad70 msgr2=0x7fc110069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc115310640 1 --2- 172.21.15.29:0/3825025773 >> v2:172.21.15.170:3300/0 conn(0x7fc11015ad70 0x7fc110069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.696+0000 7fc115310640 1 -- 172.21.15.29:0/3825025773 >> v1:172.21.15.29:6789/0 conn(0x7fc1100c9ca0 legacy=0x7fc1100ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.697+0000 7fc115310640 1 -- 172.21.15.29:0/3825025773 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100c9780 con 0x7fc1100caac0 2023-10-16T14:33:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.698+0000 7fc115310640 1 --2- 172.21.15.29:0/3825025773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11006a2e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc1100cf780 tx=0x7fc10802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c230daad220914b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.698+0000 7fc1077fe640 1 -- 172.21.15.29:0/3825025773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10802cb10 con 0x7fc1100caac0 2023-10-16T14:33:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.698+0000 7fc1077fe640 1 -- 172.21.15.29:0/3825025773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10802cc70 con 0x7fc1100caac0 2023-10-16T14:33:48.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.698+0000 7fc1077fe640 1 -- 172.21.15.29:0/3825025773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1080343f0 con 0x7fc1100caac0 2023-10-16T14:33:48.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 -- 172.21.15.29:0/3825025773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 msgr2=0x7fc11006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:48.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 --2- 172.21.15.29:0/3825025773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11006a2e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc1100cf780 tx=0x7fc10802bf70 comp rx=0 tx=0).stop 2023-10-16T14:33:48.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 -- 172.21.15.29:0/3825025773 shutdown_connections 2023-10-16T14:33:48.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 --2- 172.21.15.29:0/3825025773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11006a2e0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 --2- 172.21.15.29:0/3825025773 >> v2:172.21.15.170:3300/0 conn(0x7fc11015ad70 0x7fc110069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 -- 172.21.15.29:0/3825025773 >> 172.21.15.29:0/3825025773 conn(0x7fc1100ce9b0 msgr2=0x7fc1100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:48.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 -- 172.21.15.29:0/3825025773 shutdown_connections 2023-10-16T14:33:48.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.699+0000 7fc116d73640 1 -- 172.21.15.29:0/3825025773 wait complete. 2023-10-16T14:33:48.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.701+0000 7fc116d73640 1 Processor -- start 2023-10-16T14:33:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.701+0000 7fc116d73640 1 -- start start 2023-10-16T14:33:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc116d73640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1100c9ca0 0x7fc11014deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc116d73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc116d73640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc11015ad70 0x7fc1100b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc116d73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100c69e0 con 0x7fc1100caac0 2023-10-16T14:33:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc116d73640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11015e0e0 con 0x7fc11015ad70 2023-10-16T14:33:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc116d73640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100c5a80 con 0x7fc1100c9ca0 2023-10-16T14:33:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11015b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42866/0 (socket says 172.21.15.29:42866) 2023-10-16T14:33:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc107fff640 1 -- 172.21.15.29:0/1217899076 learned_addr learned my addr 172.21.15.29:0/1217899076 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc115310640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc11015ad70 0x7fc1100b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc114b0f640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1100c9ca0 0x7fc11014deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc107fff640 1 -- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1100c9ca0 msgr2=0x7fc11014deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.702+0000 7fc107fff640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1100c9ca0 0x7fc11014deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc107fff640 1 -- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc11015ad70 msgr2=0x7fc1100b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc107fff640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc11015ad70 0x7fc1100b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc107fff640 1 -- 172.21.15.29:0/1217899076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc108009b30 con 0x7fc1100caac0 2023-10-16T14:33:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc115310640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc11015ad70 0x7fc1100b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc107fff640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11015b450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc10000c4d0 tx=0x7fc10000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100007480 con 0x7fc1100caac0 2023-10-16T14:33:48.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1000075e0 con 0x7fc1100caac0 2023-10-16T14:33:48.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1100c5d60 con 0x7fc1100caac0 2023-10-16T14:33:48.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1000049e0 con 0x7fc1100caac0 2023-10-16T14:33:48.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.703+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1100c6320 con 0x7fc1100caac0 2023-10-16T14:33:48.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.705+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1100ad4b0 con 0x7fc1100caac0 2023-10-16T14:33:48.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.706+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc100004b40 con 0x7fc1100caac0 2023-10-16T14:33:48.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.706+0000 7fc1057fa640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc0fc06af90 0x7fc0fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:48.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.707+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc100088ce0 con 0x7fc1100caac0 2023-10-16T14:33:48.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.707+0000 7fc114b0f640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc0fc06af90 0x7fc0fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:48.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.709+0000 7fc114b0f640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc0fc06af90 0x7fc0fc06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc10c006fd0 tx=0x7fc10c008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:48.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.711+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc1100ad4b0 con 0x7fc1100caac0 2023-10-16T14:33:48.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:48.950+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc1100ad4b0 con 0x7fc1100caac0 2023-10-16T14:33:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.452+0000 7fc1057fa640 1 -- 172.21.15.29:0/1217899076 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc1100ad4b0 con 0x7fc1100caac0 2023-10-16T14:33:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc0fc06af90 msgr2=0x7fc0fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc0fc06af90 0x7fc0fc06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc10c006fd0 tx=0x7fc10c008040 comp rx=0 tx=0).stop 2023-10-16T14:33:49.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 msgr2=0x7fc11015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11015b450 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc10000c4d0 tx=0x7fc10000c990 comp rx=0 tx=0).stop 2023-10-16T14:33:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 shutdown_connections 2023-10-16T14:33:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fc0fc06af90 0x7fc0fc06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1100caac0 0x7fc11015b450 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc11015ad70 0x7fc1100b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 --2- 172.21.15.29:0/1217899076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1100c9ca0 0x7fc11014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.455+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 >> 172.21.15.29:0/1217899076 conn(0x7fc1100ce9b0 msgr2=0x7fc110070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.456+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 shutdown_connections 2023-10-16T14:33:49.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.456+0000 7fc116d73640 1 -- 172.21.15.29:0/1217899076 wait complete. 2023-10-16T14:33:49.478 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:33:49.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:49.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.704+0000 7f4f70a59640 1 Processor -- start 2023-10-16T14:33:49.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.705+0000 7f4f70a59640 1 -- start start 2023-10-16T14:33:49.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.706+0000 7f4f70a59640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f6c0ad970 0x7f4f6c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.706+0000 7f4f70a59640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c15b9d0 con 0x7f4f6c06c040 2023-10-16T14:33:49.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.706+0000 7f4f70a59640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c15bcb0 con 0x7f4f6c0ccfb0 2023-10-16T14:33:49.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.706+0000 7f4f70a59640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6c15bf90 con 0x7f4f6c0ad970 2023-10-16T14:33:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.707+0000 7f4f6b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f6c06c040 0x7f4f6c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44656/0 (socket says 172.21.15.29:44656) 2023-10-16T14:33:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.707+0000 7f4f6b7fe640 1 -- 172.21.15.29:0/1598948674 learned_addr learned my addr 172.21.15.29:0/1598948674 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.707+0000 7f4f6affd640 1 --2- 172.21.15.29:0/1598948674 >> v2:172.21.15.170:3300/0 conn(0x7f4f6c0ad970 0x7f4f6c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.707+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971306734 0 0) 0x7f4f6c15b9d0 con 0x7f4f6c06c040 2023-10-16T14:33:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.707+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54003520 con 0x7f4f6c06c040 2023-10-16T14:33:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457471107 0 0) 0x7f4f54003520 con 0x7f4f6c06c040 2023-10-16T14:33:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 >> v2:172.21.15.170:3300/0 conn(0x7f4f6c0ad970 msgr2=0x7f4f6c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 --2- 172.21.15.29:0/1598948674 >> v2:172.21.15.170:3300/0 conn(0x7f4f6c0ad970 0x7f4f6c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 >> v1:172.21.15.170:6789/0 conn(0x7f4f6c0ccfb0 legacy=0x7f4f6c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c0cc270 con 0x7f4f6c06c040 2023-10-16T14:33:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4f5c001670 con 0x7f4f6c06c040 2023-10-16T14:33:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 >> v1:172.21.15.29:6789/0 conn(0x7f4f6c06c040 legacy=0x7f4f6c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f54003970 0x7f4f54003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f540047c0 0x7f4f54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f54005180 0x7f4f5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c15bf90 con 0x7f4f540047c0 2023-10-16T14:33:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5400c020 con 0x7f4f54003970 2023-10-16T14:33:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5400c1e0 con 0x7f4f54005180 2023-10-16T14:33:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6affd640 1 --2- 172.21.15.29:0/1598948674 >> v2:172.21.15.170:3300/0 conn(0x7f4f6c0ad970 0x7f4f6c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:33:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.708+0000 7f4f6affd640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f540047c0 0x7f4f54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6b7fe640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f54003970 0x7f4f54003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6bfff640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f54005180 0x7f4f5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6affd640 1 -- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f54005180 msgr2=0x7f4f5400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6affd640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f54005180 0x7f4f5400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6affd640 1 -- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f54003970 msgr2=0x7f4f54003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6affd640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f54003970 0x7f4f54003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6affd640 1 -- 172.21.15.29:0/1598948674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54003520 con 0x7f4f540047c0 2023-10-16T14:33:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6bfff640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f54005180 0x7f4f5400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.709+0000 7f4f6bfff640 1 -- 172.21.15.29:0/1598948674 reap_dead start 2023-10-16T14:33:49.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.710+0000 7f4f6affd640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f540047c0 0x7f4f54004c40 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009c60 tx=0x7f4f540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612d456bc19dbf7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:49.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.710+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6002c8c0 con 0x7f4f540047c0 2023-10-16T14:33:49.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.710+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6002ca20 con 0x7f4f540047c0 2023-10-16T14:33:49.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.710+0000 7f4f6a7fc640 1 -- 172.21.15.29:0/1598948674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6002cdd0 con 0x7f4f540047c0 2023-10-16T14:33:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 -- 172.21.15.29:0/1598948674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f540047c0 msgr2=0x7f4f54004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f540047c0 0x7f4f54004c40 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009c60 tx=0x7f4f540057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 -- 172.21.15.29:0/1598948674 shutdown_connections 2023-10-16T14:33:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 --2- 172.21.15.29:0/1598948674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f540047c0 0x7f4f54004c40 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 -- 172.21.15.29:0/1598948674 >> 172.21.15.29:0/1598948674 conn(0x7f4f6c0ce6a0 msgr2=0x7f4f6c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 -- 172.21.15.29:0/1598948674 shutdown_connections 2023-10-16T14:33:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.711+0000 7f4f70a59640 1 -- 172.21.15.29:0/1598948674 wait complete. 2023-10-16T14:33:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.713+0000 7f4f70a59640 1 Processor -- start 2023-10-16T14:33:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.713+0000 7f4f70a59640 1 -- start start 2023-10-16T14:33:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f70a59640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f540047c0 0x7f4f6c0ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f70a59640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f6c15edf0 0x7f4f6c156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f70a59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 0x7f4f6c0c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f70a59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0c72e0 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f70a59640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c15d220 con 0x7f4f540047c0 2023-10-16T14:33:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f70a59640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c15e080 con 0x7f4f6c15edf0 2023-10-16T14:33:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 0x7f4f6c0c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 0x7f4f6c0c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42880/0 (socket says 172.21.15.29:42880) 2023-10-16T14:33:49.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.714+0000 7f4f6bfff640 1 -- 172.21.15.29:0/934236837 learned_addr learned my addr 172.21.15.29:0/934236837 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:49.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6affd640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f6c15edf0 0x7f4f6c156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6b7fe640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f540047c0 0x7f4f6c0ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6bfff640 1 -- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f6c15edf0 msgr2=0x7f4f6c156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6bfff640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f6c15edf0 0x7f4f6c156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6bfff640 1 -- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f540047c0 msgr2=0x7f4f6c0ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6bfff640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f540047c0 0x7f4f6c0ad140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6bfff640 1 -- 172.21.15.29:0/934236837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c0ad680 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6b7fe640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f540047c0 0x7f4f6c0ad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:49.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f6bfff640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 0x7f4f6c0c6da0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0cd390 tx=0x7f4f6400d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:49.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64003d80 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f640050f0 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.715+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f640054a0 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.716+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c15e510 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.716+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c15d750 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.717+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6c06b330 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.718+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f64005600 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.719+0000 7f4f53fff640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4f5806afc0 0x7f4f5806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.719+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f640891a0 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.719+0000 7f4f6b7fe640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4f5806afc0 0x7f4f5806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:49.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.721+0000 7f4f6b7fe640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4f5806afc0 0x7f4f5806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c007c60 tx=0x7f4f5c009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:49.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.723+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f64054020 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.955+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f6c04e9b0 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.956+0000 7f4f53fff640 1 -- 172.21.15.29:0/934236837 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f4f64057640 con 0x7f4f6c0c68f0 2023-10-16T14:33:49.959 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:49.959 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:45.763311+0000","modified":"2023-10-16T14:33:48.952054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7323},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7323":{"gid":7323,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6836/190249339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":190249339},{"type":"v1","addr":"172.21.15.170:6836","nonce":190249339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T14:33:49.959 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2023-10-16T14:33:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.958+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4f5806afc0 msgr2=0x7f4f5806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.958+0000 7f4f70a59640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4f5806afc0 0x7f4f5806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c007c60 tx=0x7f4f5c009040 comp rx=0 tx=0).stop 2023-10-16T14:33:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.958+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 msgr2=0x7f4f6c0c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:49.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.958+0000 7f4f70a59640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 0x7f4f6c0c6da0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0cd390 tx=0x7f4f6400d9f0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 shutdown_connections 2023-10-16T14:33:49.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4f5806afc0 0x7f4f5806d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f6c0c68f0 0x7f4f6c0c6da0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f6c15edf0 0x7f4f6c156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 --2- 172.21.15.29:0/934236837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f540047c0 0x7f4f6c0ad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:49.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 >> 172.21.15.29:0/934236837 conn(0x7f4f6c0ce6a0 msgr2=0x7f4f6c0c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:49.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 shutdown_connections 2023-10-16T14:33:49.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:49.959+0000 7f4f70a59640 1 -- 172.21.15.29:0/934236837 wait complete. 2023-10-16T14:33:49.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:33:50.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.207+0000 7fa63536a640 1 Processor -- start 2023-10-16T14:33:50.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.207+0000 7fa63536a640 1 -- start start 2023-10-16T14:33:50.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.208+0000 7fa63536a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa63006fb50 0x7fa6300cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.208+0000 7fa63536a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300ca560 con 0x7fa6300c9060 2023-10-16T14:33:50.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.208+0000 7fa63536a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300c9950 con 0x7fa63006c9e0 2023-10-16T14:33:50.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.208+0000 7fa63536a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300c9c30 con 0x7fa63006fb50 2023-10-16T14:33:50.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.209+0000 7fa62ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa6300c9060 0x7fa63006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44670/0 (socket says 172.21.15.29:44670) 2023-10-16T14:33:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.209+0000 7fa62ffff640 1 -- 172.21.15.29:0/1617491993 learned_addr learned my addr 172.21.15.29:0/1617491993 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.209+0000 7fa62f7fe640 1 --2- 172.21.15.29:0/1617491993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa63006fb50 0x7fa6300cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.210+0000 7fa62f7fe640 1 -- 172.21.15.29:0/1617491993 >> v1:172.21.15.170:6789/0 conn(0x7fa63006c9e0 legacy=0x7fa6300ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.210+0000 7fa62f7fe640 1 -- 172.21.15.29:0/1617491993 >> v1:172.21.15.29:6789/0 conn(0x7fa6300c9060 legacy=0x7fa63006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.210+0000 7fa62f7fe640 1 -- 172.21.15.29:0/1617491993 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300c9f10 con 0x7fa63006fb50 2023-10-16T14:33:50.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.211+0000 7fa62f7fe640 1 --2- 172.21.15.29:0/1617491993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa63006fb50 0x7fa6300cc8a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6240097d0 tx=0x7fa62402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ace41cfeb4e2bed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:50.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.214+0000 7fa62effd640 1 -- 172.21.15.29:0/1617491993 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624031890 con 0x7fa63006fb50 2023-10-16T14:33:50.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.214+0000 7fa62effd640 1 -- 172.21.15.29:0/1617491993 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6240319f0 con 0x7fa63006fb50 2023-10-16T14:33:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.214+0000 7fa62effd640 1 -- 172.21.15.29:0/1617491993 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624031d80 con 0x7fa63006fb50 2023-10-16T14:33:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.215+0000 7fa63536a640 1 -- 172.21.15.29:0/1617491993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa63006fb50 msgr2=0x7fa6300cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.215+0000 7fa63536a640 1 --2- 172.21.15.29:0/1617491993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa63006fb50 0x7fa6300cc8a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6240097d0 tx=0x7fa62402b930 comp rx=0 tx=0).stop 2023-10-16T14:33:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.215+0000 7fa63536a640 1 -- 172.21.15.29:0/1617491993 shutdown_connections 2023-10-16T14:33:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.215+0000 7fa63536a640 1 --2- 172.21.15.29:0/1617491993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa63006fb50 0x7fa6300cc8a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.215+0000 7fa63536a640 1 -- 172.21.15.29:0/1617491993 >> 172.21.15.29:0/1617491993 conn(0x7fa6300ce6a0 msgr2=0x7fa6300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:50.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.216+0000 7fa63536a640 1 -- 172.21.15.29:0/1617491993 shutdown_connections 2023-10-16T14:33:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.216+0000 7fa63536a640 1 -- 172.21.15.29:0/1617491993 wait complete. 2023-10-16T14:33:50.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.217+0000 7fa63536a640 1 Processor -- start 2023-10-16T14:33:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.218+0000 7fa63536a640 1 -- start start 2023-10-16T14:33:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.218+0000 7fa63536a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 0x7fa6300c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.218+0000 7fa63536a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa63006fb50 0x7fa63015f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.218+0000 7fa63536a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6300c9060 0x7fa6300c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa63536a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300c3a20 con 0x7fa63006c9e0 2023-10-16T14:33:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa63536a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63015fef0 con 0x7fa63006fb50 2023-10-16T14:33:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa63536a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300c3260 con 0x7fa6300c9060 2023-10-16T14:33:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 0x7fa6300c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 0x7fa6300c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42884/0 (socket says 172.21.15.29:42884) 2023-10-16T14:33:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 -- 172.21.15.29:0/2551580869 learned_addr learned my addr 172.21.15.29:0/2551580869 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62f7fe640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa63006fb50 0x7fa63015f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 -- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6300c9060 msgr2=0x7fa6300c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6300c9060 0x7fa6300c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 -- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa63006fb50 msgr2=0x7fa63015f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa63006fb50 0x7fa63015f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.219+0000 7fa62ffff640 1 -- 172.21.15.29:0/2551580869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624009400 con 0x7fa63006c9e0 2023-10-16T14:33:50.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.220+0000 7fa62ffff640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 0x7fa6300c58c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa62000c4a0 tx=0x7fa62000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:50.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.220+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa620005ce0 con 0x7fa63006c9e0 2023-10-16T14:33:50.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.220+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa620005e40 con 0x7fa63006c9e0 2023-10-16T14:33:50.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.220+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa620007270 con 0x7fa63006c9e0 2023-10-16T14:33:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.220+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6300c4300 con 0x7fa63006c9e0 2023-10-16T14:33:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.220+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6300c3500 con 0x7fa63006c9e0 2023-10-16T14:33:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.221+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6300c36f0 con 0x7fa63006c9e0 2023-10-16T14:33:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.222+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa6200073d0 con 0x7fa63006c9e0 2023-10-16T14:33:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.223+0000 7fa62cff9640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa61c06afc0 0x7fa61c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.223+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa620088c20 con 0x7fa63006c9e0 2023-10-16T14:33:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.224+0000 7fa62f7fe640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa61c06afc0 0x7fa61c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.225+0000 7fa62f7fe640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa61c06afc0 0x7fa61c06d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa624036000 tx=0x7fa62403c040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:50.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.228+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa620053bd0 con 0x7fa63006c9e0 2023-10-16T14:33:50.256 INFO:tasks.ceph.mds.d.smithi170.stderr:2023-10-16T14:33:50.252+0000 7f9edd03c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:33:50.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.424+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa63004e9b0 con 0x7fa63006c9e0 2023-10-16T14:33:50.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.427+0000 7fa62cff9640 1 -- 172.21.15.29:0/2551580869 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fa620017020 con 0x7fa63006c9e0 2023-10-16T14:33:50.431 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:50.431 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":111,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:33:47.320794+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:33:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.429+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa61c06afc0 msgr2=0x7fa61c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.429+0000 7fa63536a640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa61c06afc0 0x7fa61c06d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa624036000 tx=0x7fa62403c040 comp rx=0 tx=0).stop 2023-10-16T14:33:50.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.429+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 msgr2=0x7fa6300c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.429+0000 7fa63536a640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 0x7fa6300c58c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa62000c4a0 tx=0x7fa62000c960 comp rx=0 tx=0).stop 2023-10-16T14:33:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.429+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 shutdown_connections 2023-10-16T14:33:50.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa61c06afc0 0x7fa61c06d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa6300c9060 0x7fa6300c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa63006fb50 0x7fa63015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 --2- 172.21.15.29:0/2551580869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa63006c9e0 0x7fa6300c58c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 >> 172.21.15.29:0/2551580869 conn(0x7fa6300ce6a0 msgr2=0x7fa6300b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:50.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 shutdown_connections 2023-10-16T14:33:50.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.430+0000 7fa63536a640 1 -- 172.21.15.29:0/2551580869 wait complete. 2023-10-16T14:33:50.453 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:33:50.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:33:50.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.667+0000 7f4e8b26e640 1 Processor -- start 2023-10-16T14:33:50.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.668+0000 7f4e8b26e640 1 -- start start 2023-10-16T14:33:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.669+0000 7f4e8b26e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e840ad970 0x7f4e840cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.669+0000 7f4e8b26e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8415b9d0 con 0x7f4e8406c040 2023-10-16T14:33:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.669+0000 7f4e8b26e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8415bcb0 con 0x7f4e840ccfb0 2023-10-16T14:33:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.669+0000 7f4e8b26e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8415bf90 con 0x7f4e840ad970 2023-10-16T14:33:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.669+0000 7f4e8a26c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4e8406c040 0x7f4e8406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44674/0 (socket says 172.21.15.29:44674) 2023-10-16T14:33:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e8a26c640 1 -- 172.21.15.29:0/4127077061 learned_addr learned my addr 172.21.15.29:0/4127077061 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:50.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e89a6b640 1 --2- 172.21.15.29:0/4127077061 >> v2:172.21.15.29:3300/0 conn(0x7f4e840ad970 0x7f4e840cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e8926a640 1 -- 172.21.15.29:0/4127077061 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217154501 0 0) 0x7f4e8415b9d0 con 0x7f4e8406c040 2023-10-16T14:33:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e8926a640 1 -- 172.21.15.29:0/4127077061 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e70003520 con 0x7f4e8406c040 2023-10-16T14:33:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e89a6b640 1 -- 172.21.15.29:0/4127077061 >> v1:172.21.15.170:6789/0 conn(0x7f4e840ccfb0 legacy=0x7f4e840cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e89a6b640 1 -- 172.21.15.29:0/4127077061 >> v1:172.21.15.29:6789/0 conn(0x7f4e8406c040 legacy=0x7f4e8406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.670+0000 7f4e89a6b640 1 -- 172.21.15.29:0/4127077061 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840cc270 con 0x7f4e840ad970 2023-10-16T14:33:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e89a6b640 1 --2- 172.21.15.29:0/4127077061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840cca70 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4e740097d0 tx=0x7f4e7402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea6829750f91da3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8926a640 1 -- 172.21.15.29:0/4127077061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74030800 con 0x7f4e840ad970 2023-10-16T14:33:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8926a640 1 -- 172.21.15.29:0/4127077061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74030960 con 0x7f4e840ad970 2023-10-16T14:33:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8926a640 1 -- 172.21.15.29:0/4127077061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74030cd0 con 0x7f4e840ad970 2023-10-16T14:33:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8b26e640 1 -- 172.21.15.29:0/4127077061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 msgr2=0x7f4e840cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/4127077061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840cca70 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4e740097d0 tx=0x7f4e7402b940 comp rx=0 tx=0).stop 2023-10-16T14:33:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8b26e640 1 -- 172.21.15.29:0/4127077061 shutdown_connections 2023-10-16T14:33:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.672+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/4127077061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840cca70 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.673+0000 7f4e8b26e640 1 -- 172.21.15.29:0/4127077061 >> 172.21.15.29:0/4127077061 conn(0x7f4e840ce6a0 msgr2=0x7f4e840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.673+0000 7f4e8b26e640 1 -- 172.21.15.29:0/4127077061 shutdown_connections 2023-10-16T14:33:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.673+0000 7f4e8b26e640 1 -- 172.21.15.29:0/4127077061 wait complete. 2023-10-16T14:33:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.674+0000 7f4e8b26e640 1 Processor -- start 2023-10-16T14:33:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.675+0000 7f4e8b26e640 1 -- start start 2023-10-16T14:33:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8b26e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e8406c040 0x7f4e84159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8b26e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8b26e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e840ccfb0 0x7f4e841f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8b26e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840b3e40 con 0x7f4e840ad970 2023-10-16T14:33:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8b26e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840c6180 con 0x7f4e8406c040 2023-10-16T14:33:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8b26e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840c81a0 con 0x7f4e840ccfb0 2023-10-16T14:33:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e89a6b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e89a6b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42904/0 (socket says 172.21.15.29:42904) 2023-10-16T14:33:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e89a6b640 1 -- 172.21.15.29:0/191077207 learned_addr learned my addr 172.21.15.29:0/191077207 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8a26c640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e8406c040 0x7f4e84159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e89a6b640 1 -- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e840ccfb0 msgr2=0x7f4e841f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e8aa6d640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e840ccfb0 0x7f4e841f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.676+0000 7f4e89a6b640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e840ccfb0 0x7f4e841f2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e89a6b640 1 -- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e8406c040 msgr2=0x7f4e84159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e89a6b640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e8406c040 0x7f4e84159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e89a6b640 1 -- 172.21.15.29:0/191077207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e74009400 con 0x7f4e840ad970 2023-10-16T14:33:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e89a6b640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840c6940 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4e740098f0 tx=0x7f4e740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74039c10 con 0x7f4e840ad970 2023-10-16T14:33:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7403f040 con 0x7f4e840ad970 2023-10-16T14:33:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74006e10 con 0x7f4e840ad970 2023-10-16T14:33:50.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e840c8480 con 0x7f4e840ad970 2023-10-16T14:33:50.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.677+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8415a000 con 0x7f4e840ad970 2023-10-16T14:33:50.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.679+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8404e9b0 con 0x7f4e840ad970 2023-10-16T14:33:50.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.679+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e74004030 con 0x7f4e840ad970 2023-10-16T14:33:50.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.681+0000 7f4e7affd640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6806af90 0x7f4e6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:50.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.681+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e740ab8c0 con 0x7f4e840ad970 2023-10-16T14:33:50.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.682+0000 7f4e8a26c640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6806af90 0x7f4e6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:50.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.683+0000 7f4e8a26c640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6806af90 0x7f4e6806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e800011e0 tx=0x7f4e80009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:50.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.686+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e740767f0 con 0x7f4e840ad970 2023-10-16T14:33:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.876+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 --> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e840cdb70 con 0x7f4e6806af90 2023-10-16T14:33:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.905+0000 7f4e7affd640 1 -- 172.21.15.29:0/191077207 <== mgr.6076 v2:172.21.15.170:6832/2925184086 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287358 (secure 0 0 0) 0x7f4e80001c40 con 0x7f4e6806af90 2023-10-16T14:33:50.912 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:33:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6806af90 msgr2=0x7f4e6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6806af90 0x7f4e6806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e800011e0 tx=0x7f4e80009040 comp rx=0 tx=0).stop 2023-10-16T14:33:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 msgr2=0x7f4e840c6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840c6940 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4e740098f0 tx=0x7f4e740046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 shutdown_connections 2023-10-16T14:33:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f4e6806af90 0x7f4e6806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e840ad970 0x7f4e840c6940 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4e840ccfb0 0x7f4e841f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 --2- 172.21.15.29:0/191077207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4e8406c040 0x7f4e84159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 >> 172.21.15.29:0/191077207 conn(0x7f4e840ce6a0 msgr2=0x7f4e8406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 shutdown_connections 2023-10-16T14:33:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:50.911+0000 7f4e8b26e640 1 -- 172.21.15.29:0/191077207 wait complete. 2023-10-16T14:33:50.935 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T14:33:50.619222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":327,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":322320,"kb_used_data":11796,"kb_used_omap":12,"kb_used_meta":310387,"kb_avail":754652400,"statfs":{"total":773094113280,"available":772764057600,"internally_reserved":0,"allocated":12079104,"data_stored":6392504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":317836857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002336"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339122+0000","last_change":"2023-10-16T14:33:46.447246+0000","last_active":"2023-10-16T14:33:47.339122+0000","last_peered":"2023-10-16T14:33:47.339122+0000","last_clean":"2023-10-16T14:33:47.339122+0000","last_became_active":"2023-10-16T14:33:46.445501+0000","last_became_peered":"2023-10-16T14:33:46.445501+0000","last_unstale":"2023-10-16T14:33:47.339122+0000","last_undegraded":"2023-10-16T14:33:47.339122+0000","last_fullsized":"2023-10-16T14:33:47.339122+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:03.655183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339591+0000","last_change":"2023-10-16T14:33:46.453278+0000","last_active":"2023-10-16T14:33:47.339591+0000","last_peered":"2023-10-16T14:33:47.339591+0000","last_clean":"2023-10-16T14:33:47.339591+0000","last_became_active":"2023-10-16T14:33:46.452982+0000","last_became_peered":"2023-10-16T14:33:46.452982+0000","last_unstale":"2023-10-16T14:33:47.339591+0000","last_undegraded":"2023-10-16T14:33:47.339591+0000","last_fullsized":"2023-10-16T14:33:47.339591+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:00.990582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331799+0000","last_change":"2023-10-16T14:33:46.449197+0000","last_active":"2023-10-16T14:33:47.331799+0000","last_peered":"2023-10-16T14:33:47.331799+0000","last_clean":"2023-10-16T14:33:47.331799+0000","last_became_active":"2023-10-16T14:33:46.447894+0000","last_became_peered":"2023-10-16T14:33:46.447894+0000","last_unstale":"2023-10-16T14:33:47.331799+0000","last_undegraded":"2023-10-16T14:33:47.331799+0000","last_fullsized":"2023-10-16T14:33:47.331799+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:37.818459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335999+0000","last_change":"2023-10-16T14:33:46.451178+0000","last_active":"2023-10-16T14:33:47.335999+0000","last_peered":"2023-10-16T14:33:47.335999+0000","last_clean":"2023-10-16T14:33:47.335999+0000","last_became_active":"2023-10-16T14:33:46.450875+0000","last_became_peered":"2023-10-16T14:33:46.450875+0000","last_unstale":"2023-10-16T14:33:47.335999+0000","last_undegraded":"2023-10-16T14:33:47.335999+0000","last_fullsized":"2023-10-16T14:33:47.335999+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:03.428608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333139+0000","last_change":"2023-10-16T14:33:46.443085+0000","last_active":"2023-10-16T14:33:47.333139+0000","last_peered":"2023-10-16T14:33:47.333139+0000","last_clean":"2023-10-16T14:33:47.333139+0000","last_became_active":"2023-10-16T14:33:46.442795+0000","last_became_peered":"2023-10-16T14:33:46.442795+0000","last_unstale":"2023-10-16T14:33:47.333139+0000","last_undegraded":"2023-10-16T14:33:47.333139+0000","last_fullsized":"2023-10-16T14:33:47.333139+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:33.576593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338547+0000","last_change":"2023-10-16T14:33:46.442134+0000","last_active":"2023-10-16T14:33:47.338547+0000","last_peered":"2023-10-16T14:33:47.338547+0000","last_clean":"2023-10-16T14:33:47.338547+0000","last_became_active":"2023-10-16T14:33:46.441864+0000","last_became_peered":"2023-10-16T14:33:46.441864+0000","last_unstale":"2023-10-16T14:33:47.338547+0000","last_undegraded":"2023-10-16T14:33:47.338547+0000","last_fullsized":"2023-10-16T14:33:47.338547+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:55.271107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335426+0000","last_change":"2023-10-16T14:33:46.340437+0000","last_active":"2023-10-16T14:33:47.335426+0000","last_peered":"2023-10-16T14:33:47.335426+0000","last_clean":"2023-10-16T14:33:47.335426+0000","last_became_active":"2023-10-16T14:33:46.339528+0000","last_became_peered":"2023-10-16T14:33:46.339528+0000","last_unstale":"2023-10-16T14:33:47.335426+0000","last_undegraded":"2023-10-16T14:33:47.335426+0000","last_fullsized":"2023-10-16T14:33:47.335426+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:35.952492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337184+0000","last_change":"2023-10-16T14:33:46.455992+0000","last_active":"2023-10-16T14:33:47.337184+0000","last_peered":"2023-10-16T14:33:47.337184+0000","last_clean":"2023-10-16T14:33:47.337184+0000","last_became_active":"2023-10-16T14:33:46.455246+0000","last_became_peered":"2023-10-16T14:33:46.455246+0000","last_unstale":"2023-10-16T14:33:47.337184+0000","last_undegraded":"2023-10-16T14:33:47.337184+0000","last_fullsized":"2023-10-16T14:33:47.337184+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:17.956937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339064+0000","last_change":"2023-10-16T14:33:46.460172+0000","last_active":"2023-10-16T14:33:47.339064+0000","last_peered":"2023-10-16T14:33:47.339064+0000","last_clean":"2023-10-16T14:33:47.339064+0000","last_became_active":"2023-10-16T14:33:46.459764+0000","last_became_peered":"2023-10-16T14:33:46.459764+0000","last_unstale":"2023-10-16T14:33:47.339064+0000","last_undegraded":"2023-10-16T14:33:47.339064+0000","last_fullsized":"2023-10-16T14:33:47.339064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:50.040056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338888+0000","last_change":"2023-10-16T14:33:46.445061+0000","last_active":"2023-10-16T14:33:47.338888+0000","last_peered":"2023-10-16T14:33:47.338888+0000","last_clean":"2023-10-16T14:33:47.338888+0000","last_became_active":"2023-10-16T14:33:46.444753+0000","last_became_peered":"2023-10-16T14:33:46.444753+0000","last_unstale":"2023-10-16T14:33:47.338888+0000","last_undegraded":"2023-10-16T14:33:47.338888+0000","last_fullsized":"2023-10-16T14:33:47.338888+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:49.706863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340216+0000","last_change":"2023-10-16T14:33:46.441615+0000","last_active":"2023-10-16T14:33:47.340216+0000","last_peered":"2023-10-16T14:33:47.340216+0000","last_clean":"2023-10-16T14:33:47.340216+0000","last_became_active":"2023-10-16T14:33:46.441297+0000","last_became_peered":"2023-10-16T14:33:46.441297+0000","last_unstale":"2023-10-16T14:33:47.340216+0000","last_undegraded":"2023-10-16T14:33:47.340216+0000","last_fullsized":"2023-10-16T14:33:47.340216+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:25.410021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338164+0000","last_change":"2023-10-16T14:33:46.448492+0000","last_active":"2023-10-16T14:33:47.338164+0000","last_peered":"2023-10-16T14:33:47.338164+0000","last_clean":"2023-10-16T14:33:47.338164+0000","last_became_active":"2023-10-16T14:33:46.448212+0000","last_became_peered":"2023-10-16T14:33:46.448212+0000","last_unstale":"2023-10-16T14:33:47.338164+0000","last_undegraded":"2023-10-16T14:33:47.338164+0000","last_fullsized":"2023-10-16T14:33:47.338164+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:28.072982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331734+0000","last_change":"2023-10-16T14:33:46.343112+0000","last_active":"2023-10-16T14:33:47.331734+0000","last_peered":"2023-10-16T14:33:47.331734+0000","last_clean":"2023-10-16T14:33:47.331734+0000","last_became_active":"2023-10-16T14:33:46.342256+0000","last_became_peered":"2023-10-16T14:33:46.342256+0000","last_unstale":"2023-10-16T14:33:47.331734+0000","last_undegraded":"2023-10-16T14:33:47.331734+0000","last_fullsized":"2023-10-16T14:33:47.331734+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:58.146309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335944+0000","last_change":"2023-10-16T14:33:46.456701+0000","last_active":"2023-10-16T14:33:47.335944+0000","last_peered":"2023-10-16T14:33:47.335944+0000","last_clean":"2023-10-16T14:33:47.335944+0000","last_became_active":"2023-10-16T14:33:46.456187+0000","last_became_peered":"2023-10-16T14:33:46.456187+0000","last_unstale":"2023-10-16T14:33:47.335944+0000","last_undegraded":"2023-10-16T14:33:47.335944+0000","last_fullsized":"2023-10-16T14:33:47.335944+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:26.588555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330183+0000","last_change":"2023-10-16T14:33:46.441161+0000","last_active":"2023-10-16T14:33:47.330183+0000","last_peered":"2023-10-16T14:33:47.330183+0000","last_clean":"2023-10-16T14:33:47.330183+0000","last_became_active":"2023-10-16T14:33:46.440688+0000","last_became_peered":"2023-10-16T14:33:46.440688+0000","last_unstale":"2023-10-16T14:33:47.330183+0000","last_undegraded":"2023-10-16T14:33:47.330183+0000","last_fullsized":"2023-10-16T14:33:47.330183+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:57.146819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332767+0000","last_change":"2023-10-16T14:33:46.440936+0000","last_active":"2023-10-16T14:33:47.332767+0000","last_peered":"2023-10-16T14:33:47.332767+0000","last_clean":"2023-10-16T14:33:47.332767+0000","last_became_active":"2023-10-16T14:33:46.440658+0000","last_became_peered":"2023-10-16T14:33:46.440658+0000","last_unstale":"2023-10-16T14:33:47.332767+0000","last_undegraded":"2023-10-16T14:33:47.332767+0000","last_fullsized":"2023-10-16T14:33:47.332767+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:26.641874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333738+0000","last_change":"2023-10-16T14:33:46.336574+0000","last_active":"2023-10-16T14:33:47.333738+0000","last_peered":"2023-10-16T14:33:47.333738+0000","last_clean":"2023-10-16T14:33:47.333738+0000","last_became_active":"2023-10-16T14:33:46.335159+0000","last_became_peered":"2023-10-16T14:33:46.335159+0000","last_unstale":"2023-10-16T14:33:47.333738+0000","last_undegraded":"2023-10-16T14:33:47.333738+0000","last_fullsized":"2023-10-16T14:33:47.333738+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:19.542085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339084+0000","last_change":"2023-10-16T14:33:46.446834+0000","last_active":"2023-10-16T14:33:47.339084+0000","last_peered":"2023-10-16T14:33:47.339084+0000","last_clean":"2023-10-16T14:33:47.339084+0000","last_became_active":"2023-10-16T14:33:46.446514+0000","last_became_peered":"2023-10-16T14:33:46.446514+0000","last_unstale":"2023-10-16T14:33:47.339084+0000","last_undegraded":"2023-10-16T14:33:47.339084+0000","last_fullsized":"2023-10-16T14:33:47.339084+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:28.072982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337638+0000","last_change":"2023-10-16T14:33:46.446138+0000","last_active":"2023-10-16T14:33:47.337638+0000","last_peered":"2023-10-16T14:33:47.337638+0000","last_clean":"2023-10-16T14:33:47.337638+0000","last_became_active":"2023-10-16T14:33:46.445839+0000","last_became_peered":"2023-10-16T14:33:46.445839+0000","last_unstale":"2023-10-16T14:33:47.337638+0000","last_undegraded":"2023-10-16T14:33:47.337638+0000","last_fullsized":"2023-10-16T14:33:47.337638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:10.471768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336796+0000","last_change":"2023-10-16T14:33:46.461085+0000","last_active":"2023-10-16T14:33:47.336796+0000","last_peered":"2023-10-16T14:33:47.336796+0000","last_clean":"2023-10-16T14:33:47.336796+0000","last_became_active":"2023-10-16T14:33:46.460796+0000","last_became_peered":"2023-10-16T14:33:46.460796+0000","last_unstale":"2023-10-16T14:33:47.336796+0000","last_undegraded":"2023-10-16T14:33:47.336796+0000","last_fullsized":"2023-10-16T14:33:47.336796+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:50.007020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340310+0000","last_change":"2023-10-16T14:33:46.462544+0000","last_active":"2023-10-16T14:33:47.340310+0000","last_peered":"2023-10-16T14:33:47.340310+0000","last_clean":"2023-10-16T14:33:47.340310+0000","last_became_active":"2023-10-16T14:33:46.461995+0000","last_became_peered":"2023-10-16T14:33:46.461995+0000","last_unstale":"2023-10-16T14:33:47.340310+0000","last_undegraded":"2023-10-16T14:33:47.340310+0000","last_fullsized":"2023-10-16T14:33:47.340310+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:17.006796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340215+0000","last_change":"2023-10-16T14:33:46.459119+0000","last_active":"2023-10-16T14:33:47.340215+0000","last_peered":"2023-10-16T14:33:47.340215+0000","last_clean":"2023-10-16T14:33:47.340215+0000","last_became_active":"2023-10-16T14:33:46.458853+0000","last_became_peered":"2023-10-16T14:33:46.458853+0000","last_unstale":"2023-10-16T14:33:47.340215+0000","last_undegraded":"2023-10-16T14:33:47.340215+0000","last_fullsized":"2023-10-16T14:33:47.340215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:51.326358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337794+0000","last_change":"2023-10-16T14:33:46.460358+0000","last_active":"2023-10-16T14:33:47.337794+0000","last_peered":"2023-10-16T14:33:47.337794+0000","last_clean":"2023-10-16T14:33:47.337794+0000","last_became_active":"2023-10-16T14:33:46.459733+0000","last_became_peered":"2023-10-16T14:33:46.459733+0000","last_unstale":"2023-10-16T14:33:47.337794+0000","last_undegraded":"2023-10-16T14:33:47.337794+0000","last_fullsized":"2023-10-16T14:33:47.337794+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:03.428608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334937+0000","last_change":"2023-10-16T14:33:46.450285+0000","last_active":"2023-10-16T14:33:47.334937+0000","last_peered":"2023-10-16T14:33:47.334937+0000","last_clean":"2023-10-16T14:33:47.334937+0000","last_became_active":"2023-10-16T14:33:46.449970+0000","last_became_peered":"2023-10-16T14:33:46.449970+0000","last_unstale":"2023-10-16T14:33:47.334937+0000","last_undegraded":"2023-10-16T14:33:47.334937+0000","last_fullsized":"2023-10-16T14:33:47.334937+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:01.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339841+0000","last_change":"2023-10-16T14:33:46.454821+0000","last_active":"2023-10-16T14:33:47.339841+0000","last_peered":"2023-10-16T14:33:47.339841+0000","last_clean":"2023-10-16T14:33:47.339841+0000","last_became_active":"2023-10-16T14:33:46.453683+0000","last_became_peered":"2023-10-16T14:33:46.453683+0000","last_unstale":"2023-10-16T14:33:47.339841+0000","last_undegraded":"2023-10-16T14:33:47.339841+0000","last_fullsized":"2023-10-16T14:33:47.339841+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:03.655183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331496+0000","last_change":"2023-10-16T14:33:46.454153+0000","last_active":"2023-10-16T14:33:47.331496+0000","last_peered":"2023-10-16T14:33:47.331496+0000","last_clean":"2023-10-16T14:33:47.331496+0000","last_became_active":"2023-10-16T14:33:46.453682+0000","last_became_peered":"2023-10-16T14:33:46.453682+0000","last_unstale":"2023-10-16T14:33:47.331496+0000","last_undegraded":"2023-10-16T14:33:47.331496+0000","last_fullsized":"2023-10-16T14:33:47.331496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:51.182380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334917+0000","last_change":"2023-10-16T14:33:46.443435+0000","last_active":"2023-10-16T14:33:47.334917+0000","last_peered":"2023-10-16T14:33:47.334917+0000","last_clean":"2023-10-16T14:33:47.334917+0000","last_became_active":"2023-10-16T14:33:46.443132+0000","last_became_peered":"2023-10-16T14:33:46.443132+0000","last_unstale":"2023-10-16T14:33:47.334917+0000","last_undegraded":"2023-10-16T14:33:47.334917+0000","last_fullsized":"2023-10-16T14:33:47.334917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:51.812904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338865+0000","last_change":"2023-10-16T14:33:46.451427+0000","last_active":"2023-10-16T14:33:47.338865+0000","last_peered":"2023-10-16T14:33:47.338865+0000","last_clean":"2023-10-16T14:33:47.338865+0000","last_became_active":"2023-10-16T14:33:46.450955+0000","last_became_peered":"2023-10-16T14:33:46.450955+0000","last_unstale":"2023-10-16T14:33:47.338865+0000","last_undegraded":"2023-10-16T14:33:47.338865+0000","last_fullsized":"2023-10-16T14:33:47.338865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:20.311567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338214+0000","last_change":"2023-10-16T14:33:46.450726+0000","last_active":"2023-10-16T14:33:47.338214+0000","last_peered":"2023-10-16T14:33:47.338214+0000","last_clean":"2023-10-16T14:33:47.338214+0000","last_became_active":"2023-10-16T14:33:46.449233+0000","last_became_peered":"2023-10-16T14:33:46.449233+0000","last_unstale":"2023-10-16T14:33:47.338214+0000","last_undegraded":"2023-10-16T14:33:47.338214+0000","last_fullsized":"2023-10-16T14:33:47.338214+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:20.311567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340071+0000","last_change":"2023-10-16T14:33:46.449662+0000","last_active":"2023-10-16T14:33:47.340071+0000","last_peered":"2023-10-16T14:33:47.340071+0000","last_clean":"2023-10-16T14:33:47.340071+0000","last_became_active":"2023-10-16T14:33:46.449115+0000","last_became_peered":"2023-10-16T14:33:46.449115+0000","last_unstale":"2023-10-16T14:33:47.340071+0000","last_undegraded":"2023-10-16T14:33:47.340071+0000","last_fullsized":"2023-10-16T14:33:47.340071+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:04.527422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332268+0000","last_change":"2023-10-16T14:33:46.449722+0000","last_active":"2023-10-16T14:33:47.332268+0000","last_peered":"2023-10-16T14:33:47.332268+0000","last_clean":"2023-10-16T14:33:47.332268+0000","last_became_active":"2023-10-16T14:33:46.449396+0000","last_became_peered":"2023-10-16T14:33:46.449396+0000","last_unstale":"2023-10-16T14:33:47.332268+0000","last_undegraded":"2023-10-16T14:33:47.332268+0000","last_fullsized":"2023-10-16T14:33:47.332268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:45.538655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337848+0000","last_change":"2023-10-16T14:33:46.457390+0000","last_active":"2023-10-16T14:33:47.337848+0000","last_peered":"2023-10-16T14:33:47.337848+0000","last_clean":"2023-10-16T14:33:47.337848+0000","last_became_active":"2023-10-16T14:33:46.456467+0000","last_became_peered":"2023-10-16T14:33:46.456467+0000","last_unstale":"2023-10-16T14:33:47.337848+0000","last_undegraded":"2023-10-16T14:33:47.337848+0000","last_fullsized":"2023-10-16T14:33:47.337848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:02.604688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334216+0000","last_change":"2023-10-16T14:33:46.447933+0000","last_active":"2023-10-16T14:33:47.334216+0000","last_peered":"2023-10-16T14:33:47.334216+0000","last_clean":"2023-10-16T14:33:47.334216+0000","last_became_active":"2023-10-16T14:33:46.447672+0000","last_became_peered":"2023-10-16T14:33:46.447672+0000","last_unstale":"2023-10-16T14:33:47.334216+0000","last_undegraded":"2023-10-16T14:33:47.334216+0000","last_fullsized":"2023-10-16T14:33:47.334216+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:01.220894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"110'4","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339673+0000","last_change":"2023-10-16T14:33:45.311723+0000","last_active":"2023-10-16T14:33:47.339673+0000","last_peered":"2023-10-16T14:33:47.339673+0000","last_clean":"2023-10-16T14:33:47.339673+0000","last_became_active":"2023-10-16T14:33:45.310078+0000","last_became_peered":"2023-10-16T14:33:45.310078+0000","last_unstale":"2023-10-16T14:33:47.339673+0000","last_undegraded":"2023-10-16T14:33:47.339673+0000","last_fullsized":"2023-10-16T14:33:47.339673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:33.046320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339497+0000","last_change":"2023-10-16T14:33:46.454177+0000","last_active":"2023-10-16T14:33:47.339497+0000","last_peered":"2023-10-16T14:33:47.339497+0000","last_clean":"2023-10-16T14:33:47.339497+0000","last_became_active":"2023-10-16T14:33:46.453245+0000","last_became_peered":"2023-10-16T14:33:46.453245+0000","last_unstale":"2023-10-16T14:33:47.339497+0000","last_undegraded":"2023-10-16T14:33:47.339497+0000","last_fullsized":"2023-10-16T14:33:47.339497+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:04.527422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334516+0000","last_change":"2023-10-16T14:33:45.315352+0000","last_active":"2023-10-16T14:33:47.334516+0000","last_peered":"2023-10-16T14:33:47.334516+0000","last_clean":"2023-10-16T14:33:47.334516+0000","last_became_active":"2023-10-16T14:33:45.315039+0000","last_became_peered":"2023-10-16T14:33:45.315039+0000","last_unstale":"2023-10-16T14:33:47.334516+0000","last_undegraded":"2023-10-16T14:33:47.334516+0000","last_fullsized":"2023-10-16T14:33:47.334516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.038356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338581+0000","last_change":"2023-10-16T14:33:46.457095+0000","last_active":"2023-10-16T14:33:47.338581+0000","last_peered":"2023-10-16T14:33:47.338581+0000","last_clean":"2023-10-16T14:33:47.338581+0000","last_became_active":"2023-10-16T14:33:46.456797+0000","last_became_peered":"2023-10-16T14:33:46.456797+0000","last_unstale":"2023-10-16T14:33:47.338581+0000","last_undegraded":"2023-10-16T14:33:47.338581+0000","last_fullsized":"2023-10-16T14:33:47.338581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:17.006796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337733+0000","last_change":"2023-10-16T14:33:45.303207+0000","last_active":"2023-10-16T14:33:47.337733+0000","last_peered":"2023-10-16T14:33:47.337733+0000","last_clean":"2023-10-16T14:33:47.337733+0000","last_became_active":"2023-10-16T14:33:45.302786+0000","last_became_peered":"2023-10-16T14:33:45.302786+0000","last_unstale":"2023-10-16T14:33:47.337733+0000","last_undegraded":"2023-10-16T14:33:47.337733+0000","last_fullsized":"2023-10-16T14:33:47.337733+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:32.054815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332139+0000","last_change":"2023-10-16T14:33:46.343123+0000","last_active":"2023-10-16T14:33:47.332139+0000","last_peered":"2023-10-16T14:33:47.332139+0000","last_clean":"2023-10-16T14:33:47.332139+0000","last_became_active":"2023-10-16T14:33:46.342623+0000","last_became_peered":"2023-10-16T14:33:46.342623+0000","last_unstale":"2023-10-16T14:33:47.332139+0000","last_undegraded":"2023-10-16T14:33:47.332139+0000","last_fullsized":"2023-10-16T14:33:47.332139+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:58.001842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336850+0000","last_change":"2023-10-16T14:33:45.303709+0000","last_active":"2023-10-16T14:33:47.336850+0000","last_peered":"2023-10-16T14:33:47.336850+0000","last_clean":"2023-10-16T14:33:47.336850+0000","last_became_active":"2023-10-16T14:33:45.302615+0000","last_became_peered":"2023-10-16T14:33:45.302615+0000","last_unstale":"2023-10-16T14:33:47.336850+0000","last_undegraded":"2023-10-16T14:33:47.336850+0000","last_fullsized":"2023-10-16T14:33:47.336850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:09.549638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340250+0000","last_change":"2023-10-16T14:33:46.449418+0000","last_active":"2023-10-16T14:33:47.340250+0000","last_peered":"2023-10-16T14:33:47.340250+0000","last_clean":"2023-10-16T14:33:47.340250+0000","last_became_active":"2023-10-16T14:33:46.448240+0000","last_became_peered":"2023-10-16T14:33:46.448240+0000","last_unstale":"2023-10-16T14:33:47.340250+0000","last_undegraded":"2023-10-16T14:33:47.340250+0000","last_fullsized":"2023-10-16T14:33:47.340250+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:36.729192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333578+0000","last_change":"2023-10-16T14:33:45.316657+0000","last_active":"2023-10-16T14:33:47.333578+0000","last_peered":"2023-10-16T14:33:47.333578+0000","last_clean":"2023-10-16T14:33:47.333578+0000","last_became_active":"2023-10-16T14:33:45.316276+0000","last_became_peered":"2023-10-16T14:33:45.316276+0000","last_unstale":"2023-10-16T14:33:47.333578+0000","last_undegraded":"2023-10-16T14:33:47.333578+0000","last_fullsized":"2023-10-16T14:33:47.333578+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:39.586510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333284+0000","last_change":"2023-10-16T14:33:46.335786+0000","last_active":"2023-10-16T14:33:47.333284+0000","last_peered":"2023-10-16T14:33:47.333284+0000","last_clean":"2023-10-16T14:33:47.333284+0000","last_became_active":"2023-10-16T14:33:46.334510+0000","last_became_peered":"2023-10-16T14:33:46.334510+0000","last_unstale":"2023-10-16T14:33:47.333284+0000","last_undegraded":"2023-10-16T14:33:47.333284+0000","last_fullsized":"2023-10-16T14:33:47.333284+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:38.272927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333554+0000","last_change":"2023-10-16T14:33:45.314410+0000","last_active":"2023-10-16T14:33:47.333554+0000","last_peered":"2023-10-16T14:33:47.333554+0000","last_clean":"2023-10-16T14:33:47.333554+0000","last_became_active":"2023-10-16T14:33:45.314103+0000","last_became_peered":"2023-10-16T14:33:45.314103+0000","last_unstale":"2023-10-16T14:33:47.333554+0000","last_undegraded":"2023-10-16T14:33:47.333554+0000","last_fullsized":"2023-10-16T14:33:47.333554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:02.327159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332140+0000","last_change":"2023-10-16T14:33:46.450482+0000","last_active":"2023-10-16T14:33:47.332140+0000","last_peered":"2023-10-16T14:33:47.332140+0000","last_clean":"2023-10-16T14:33:47.332140+0000","last_became_active":"2023-10-16T14:33:46.450106+0000","last_became_peered":"2023-10-16T14:33:46.450106+0000","last_unstale":"2023-10-16T14:33:47.332140+0000","last_undegraded":"2023-10-16T14:33:47.332140+0000","last_fullsized":"2023-10-16T14:33:47.332140+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:12.771061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337187+0000","last_change":"2023-10-16T14:33:45.312180+0000","last_active":"2023-10-16T14:33:47.337187+0000","last_peered":"2023-10-16T14:33:47.337187+0000","last_clean":"2023-10-16T14:33:47.337187+0000","last_became_active":"2023-10-16T14:33:45.311844+0000","last_became_peered":"2023-10-16T14:33:45.311844+0000","last_unstale":"2023-10-16T14:33:47.337187+0000","last_undegraded":"2023-10-16T14:33:47.337187+0000","last_fullsized":"2023-10-16T14:33:47.337187+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:13.245531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330554+0000","last_change":"2023-10-16T14:33:46.440970+0000","last_active":"2023-10-16T14:33:47.330554+0000","last_peered":"2023-10-16T14:33:47.330554+0000","last_clean":"2023-10-16T14:33:47.330554+0000","last_became_active":"2023-10-16T14:33:46.439972+0000","last_became_peered":"2023-10-16T14:33:46.439972+0000","last_unstale":"2023-10-16T14:33:47.330554+0000","last_undegraded":"2023-10-16T14:33:47.330554+0000","last_fullsized":"2023-10-16T14:33:47.330554+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:51.182380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338046+0000","last_change":"2023-10-16T14:33:45.310523+0000","last_active":"2023-10-16T14:33:47.338046+0000","last_peered":"2023-10-16T14:33:47.338046+0000","last_clean":"2023-10-16T14:33:47.338046+0000","last_became_active":"2023-10-16T14:33:45.309548+0000","last_became_peered":"2023-10-16T14:33:45.309548+0000","last_unstale":"2023-10-16T14:33:47.338046+0000","last_undegraded":"2023-10-16T14:33:47.338046+0000","last_fullsized":"2023-10-16T14:33:47.338046+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:03.126029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331222+0000","last_change":"2023-10-16T14:33:46.457742+0000","last_active":"2023-10-16T14:33:47.331222+0000","last_peered":"2023-10-16T14:33:47.331222+0000","last_clean":"2023-10-16T14:33:47.331222+0000","last_became_active":"2023-10-16T14:33:46.457380+0000","last_became_peered":"2023-10-16T14:33:46.457380+0000","last_unstale":"2023-10-16T14:33:47.331222+0000","last_undegraded":"2023-10-16T14:33:47.331222+0000","last_fullsized":"2023-10-16T14:33:47.331222+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:45.484436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339770+0000","last_change":"2023-10-16T14:33:45.312263+0000","last_active":"2023-10-16T14:33:47.339770+0000","last_peered":"2023-10-16T14:33:47.339770+0000","last_clean":"2023-10-16T14:33:47.339770+0000","last_became_active":"2023-10-16T14:33:45.311969+0000","last_became_peered":"2023-10-16T14:33:45.311969+0000","last_unstale":"2023-10-16T14:33:47.339770+0000","last_undegraded":"2023-10-16T14:33:47.339770+0000","last_fullsized":"2023-10-16T14:33:47.339770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:09.466762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339293+0000","last_change":"2023-10-16T14:33:46.445263+0000","last_active":"2023-10-16T14:33:47.339293+0000","last_peered":"2023-10-16T14:33:47.339293+0000","last_clean":"2023-10-16T14:33:47.339293+0000","last_became_active":"2023-10-16T14:33:46.445000+0000","last_became_peered":"2023-10-16T14:33:46.445000+0000","last_unstale":"2023-10-16T14:33:47.339293+0000","last_undegraded":"2023-10-16T14:33:47.339293+0000","last_fullsized":"2023-10-16T14:33:47.339293+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:27.890965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338614+0000","last_change":"2023-10-16T14:33:45.317182+0000","last_active":"2023-10-16T14:33:47.338614+0000","last_peered":"2023-10-16T14:33:47.338614+0000","last_clean":"2023-10-16T14:33:47.338614+0000","last_became_active":"2023-10-16T14:33:45.316670+0000","last_became_peered":"2023-10-16T14:33:45.316670+0000","last_unstale":"2023-10-16T14:33:47.338614+0000","last_undegraded":"2023-10-16T14:33:47.338614+0000","last_fullsized":"2023-10-16T14:33:47.338614+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:56.029775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336666+0000","last_change":"2023-10-16T14:33:46.458454+0000","last_active":"2023-10-16T14:33:47.336666+0000","last_peered":"2023-10-16T14:33:47.336666+0000","last_clean":"2023-10-16T14:33:47.336666+0000","last_became_active":"2023-10-16T14:33:46.455052+0000","last_became_peered":"2023-10-16T14:33:46.455052+0000","last_unstale":"2023-10-16T14:33:47.336666+0000","last_undegraded":"2023-10-16T14:33:47.336666+0000","last_fullsized":"2023-10-16T14:33:47.336666+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:34.051326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338418+0000","last_change":"2023-10-16T14:33:45.318096+0000","last_active":"2023-10-16T14:33:47.338418+0000","last_peered":"2023-10-16T14:33:47.338418+0000","last_clean":"2023-10-16T14:33:47.338418+0000","last_became_active":"2023-10-16T14:33:45.316471+0000","last_became_peered":"2023-10-16T14:33:45.316471+0000","last_unstale":"2023-10-16T14:33:47.338418+0000","last_undegraded":"2023-10-16T14:33:47.338418+0000","last_fullsized":"2023-10-16T14:33:47.338418+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:47.477165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333968+0000","last_change":"2023-10-16T14:33:46.458818+0000","last_active":"2023-10-16T14:33:47.333968+0000","last_peered":"2023-10-16T14:33:47.333968+0000","last_clean":"2023-10-16T14:33:47.333968+0000","last_became_active":"2023-10-16T14:33:46.458491+0000","last_became_peered":"2023-10-16T14:33:46.458491+0000","last_unstale":"2023-10-16T14:33:47.333968+0000","last_undegraded":"2023-10-16T14:33:47.333968+0000","last_fullsized":"2023-10-16T14:33:47.333968+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:18.391780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"110'6","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337735+0000","last_change":"2023-10-16T14:33:45.310689+0000","last_active":"2023-10-16T14:33:47.337735+0000","last_peered":"2023-10-16T14:33:47.337735+0000","last_clean":"2023-10-16T14:33:47.337735+0000","last_became_active":"2023-10-16T14:33:45.310390+0000","last_became_peered":"2023-10-16T14:33:45.310390+0000","last_unstale":"2023-10-16T14:33:47.337735+0000","last_undegraded":"2023-10-16T14:33:47.337735+0000","last_fullsized":"2023-10-16T14:33:47.337735+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:04.451086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339106+0000","last_change":"2023-10-16T14:33:46.447787+0000","last_active":"2023-10-16T14:33:47.339106+0000","last_peered":"2023-10-16T14:33:47.339106+0000","last_clean":"2023-10-16T14:33:47.339106+0000","last_became_active":"2023-10-16T14:33:46.445073+0000","last_became_peered":"2023-10-16T14:33:46.445073+0000","last_unstale":"2023-10-16T14:33:47.339106+0000","last_undegraded":"2023-10-16T14:33:47.339106+0000","last_fullsized":"2023-10-16T14:33:47.339106+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:54.185890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338453+0000","last_change":"2023-10-16T14:33:45.312942+0000","last_active":"2023-10-16T14:33:47.338453+0000","last_peered":"2023-10-16T14:33:47.338453+0000","last_clean":"2023-10-16T14:33:47.338453+0000","last_became_active":"2023-10-16T14:33:45.312603+0000","last_became_peered":"2023-10-16T14:33:45.312603+0000","last_unstale":"2023-10-16T14:33:47.338453+0000","last_undegraded":"2023-10-16T14:33:47.338453+0000","last_fullsized":"2023-10-16T14:33:47.338453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.583549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331848+0000","last_change":"2023-10-16T14:33:46.450643+0000","last_active":"2023-10-16T14:33:47.331848+0000","last_peered":"2023-10-16T14:33:47.331848+0000","last_clean":"2023-10-16T14:33:47.331848+0000","last_became_active":"2023-10-16T14:33:46.450338+0000","last_became_peered":"2023-10-16T14:33:46.450338+0000","last_unstale":"2023-10-16T14:33:47.331848+0000","last_undegraded":"2023-10-16T14:33:47.331848+0000","last_fullsized":"2023-10-16T14:33:47.331848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:29.997780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330108+0000","last_change":"2023-10-16T14:33:45.312750+0000","last_active":"2023-10-16T14:33:47.330108+0000","last_peered":"2023-10-16T14:33:47.330108+0000","last_clean":"2023-10-16T14:33:47.330108+0000","last_became_active":"2023-10-16T14:33:45.312429+0000","last_became_peered":"2023-10-16T14:33:45.312429+0000","last_unstale":"2023-10-16T14:33:47.330108+0000","last_undegraded":"2023-10-16T14:33:47.330108+0000","last_fullsized":"2023-10-16T14:33:47.330108+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:06.804628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337819+0000","last_change":"2023-10-16T14:33:46.445420+0000","last_active":"2023-10-16T14:33:47.337819+0000","last_peered":"2023-10-16T14:33:47.337819+0000","last_clean":"2023-10-16T14:33:47.337819+0000","last_became_active":"2023-10-16T14:33:46.445062+0000","last_became_peered":"2023-10-16T14:33:46.445062+0000","last_unstale":"2023-10-16T14:33:47.337819+0000","last_undegraded":"2023-10-16T14:33:47.337819+0000","last_fullsized":"2023-10-16T14:33:47.337819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:17.006796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337002+0000","last_change":"2023-10-16T14:33:45.316993+0000","last_active":"2023-10-16T14:33:47.337002+0000","last_peered":"2023-10-16T14:33:47.337002+0000","last_clean":"2023-10-16T14:33:47.337002+0000","last_became_active":"2023-10-16T14:33:45.315095+0000","last_became_peered":"2023-10-16T14:33:45.315095+0000","last_unstale":"2023-10-16T14:33:47.337002+0000","last_undegraded":"2023-10-16T14:33:47.337002+0000","last_fullsized":"2023-10-16T14:33:47.337002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:31:58.892475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336756+0000","last_change":"2023-10-16T14:33:46.453305+0000","last_active":"2023-10-16T14:33:47.336756+0000","last_peered":"2023-10-16T14:33:47.336756+0000","last_clean":"2023-10-16T14:33:47.336756+0000","last_became_active":"2023-10-16T14:33:46.453010+0000","last_became_peered":"2023-10-16T14:33:46.453010+0000","last_unstale":"2023-10-16T14:33:47.336756+0000","last_undegraded":"2023-10-16T14:33:47.336756+0000","last_fullsized":"2023-10-16T14:33:47.336756+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:28.072982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"110'8","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338655+0000","last_change":"2023-10-16T14:33:45.315138+0000","last_active":"2023-10-16T14:33:47.338655+0000","last_peered":"2023-10-16T14:33:47.338655+0000","last_clean":"2023-10-16T14:33:47.338655+0000","last_became_active":"2023-10-16T14:33:45.314381+0000","last_became_peered":"2023-10-16T14:33:45.314381+0000","last_unstale":"2023-10-16T14:33:47.338655+0000","last_undegraded":"2023-10-16T14:33:47.338655+0000","last_fullsized":"2023-10-16T14:33:47.338655+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:25:43.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332149+0000","last_change":"2023-10-16T14:33:46.440969+0000","last_active":"2023-10-16T14:33:47.332149+0000","last_peered":"2023-10-16T14:33:47.332149+0000","last_clean":"2023-10-16T14:33:47.332149+0000","last_became_active":"2023-10-16T14:33:46.440649+0000","last_became_peered":"2023-10-16T14:33:46.440649+0000","last_unstale":"2023-10-16T14:33:47.332149+0000","last_undegraded":"2023-10-16T14:33:47.332149+0000","last_fullsized":"2023-10-16T14:33:47.332149+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:48.245341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339897+0000","last_change":"2023-10-16T14:33:45.317908+0000","last_active":"2023-10-16T14:33:47.339897+0000","last_peered":"2023-10-16T14:33:47.339897+0000","last_clean":"2023-10-16T14:33:47.339897+0000","last_became_active":"2023-10-16T14:33:45.317287+0000","last_became_peered":"2023-10-16T14:33:45.317287+0000","last_unstale":"2023-10-16T14:33:47.339897+0000","last_undegraded":"2023-10-16T14:33:47.339897+0000","last_fullsized":"2023-10-16T14:33:47.339897+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:53.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339645+0000","last_change":"2023-10-16T14:33:46.442081+0000","last_active":"2023-10-16T14:33:47.339645+0000","last_peered":"2023-10-16T14:33:47.339645+0000","last_clean":"2023-10-16T14:33:47.339645+0000","last_became_active":"2023-10-16T14:33:46.441343+0000","last_became_peered":"2023-10-16T14:33:46.441343+0000","last_unstale":"2023-10-16T14:33:47.339645+0000","last_undegraded":"2023-10-16T14:33:47.339645+0000","last_fullsized":"2023-10-16T14:33:47.339645+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:51.326358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337357+0000","last_change":"2023-10-16T14:33:45.315678+0000","last_active":"2023-10-16T14:33:47.337357+0000","last_peered":"2023-10-16T14:33:47.337357+0000","last_clean":"2023-10-16T14:33:47.337357+0000","last_became_active":"2023-10-16T14:33:45.313889+0000","last_became_peered":"2023-10-16T14:33:45.313889+0000","last_unstale":"2023-10-16T14:33:47.337357+0000","last_undegraded":"2023-10-16T14:33:47.337357+0000","last_fullsized":"2023-10-16T14:33:47.337357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:28.052772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334402+0000","last_change":"2023-10-16T14:33:46.437520+0000","last_active":"2023-10-16T14:33:47.334402+0000","last_peered":"2023-10-16T14:33:47.334402+0000","last_clean":"2023-10-16T14:33:47.334402+0000","last_became_active":"2023-10-16T14:33:46.436979+0000","last_became_peered":"2023-10-16T14:33:46.436979+0000","last_unstale":"2023-10-16T14:33:47.334402+0000","last_undegraded":"2023-10-16T14:33:47.334402+0000","last_fullsized":"2023-10-16T14:33:47.334402+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:28.405987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336945+0000","last_change":"2023-10-16T14:33:45.311117+0000","last_active":"2023-10-16T14:33:47.336945+0000","last_peered":"2023-10-16T14:33:47.336945+0000","last_clean":"2023-10-16T14:33:47.336945+0000","last_became_active":"2023-10-16T14:33:45.310621+0000","last_became_peered":"2023-10-16T14:33:45.310621+0000","last_unstale":"2023-10-16T14:33:47.336945+0000","last_undegraded":"2023-10-16T14:33:47.336945+0000","last_fullsized":"2023-10-16T14:33:47.336945+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:00.999541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336500+0000","last_change":"2023-10-16T14:33:46.437946+0000","last_active":"2023-10-16T14:33:47.336500+0000","last_peered":"2023-10-16T14:33:47.336500+0000","last_clean":"2023-10-16T14:33:47.336500+0000","last_became_active":"2023-10-16T14:33:46.437355+0000","last_became_peered":"2023-10-16T14:33:46.437355+0000","last_unstale":"2023-10-16T14:33:47.336500+0000","last_undegraded":"2023-10-16T14:33:47.336500+0000","last_fullsized":"2023-10-16T14:33:47.336500+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:02.604688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330084+0000","last_change":"2023-10-16T14:33:45.299128+0000","last_active":"2023-10-16T14:33:47.330084+0000","last_peered":"2023-10-16T14:33:47.330084+0000","last_clean":"2023-10-16T14:33:47.330084+0000","last_became_active":"2023-10-16T14:33:45.298810+0000","last_became_peered":"2023-10-16T14:33:45.298810+0000","last_unstale":"2023-10-16T14:33:47.330084+0000","last_undegraded":"2023-10-16T14:33:47.330084+0000","last_fullsized":"2023-10-16T14:33:47.330084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:25.967371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337476+0000","last_change":"2023-10-16T14:33:46.440724+0000","last_active":"2023-10-16T14:33:47.337476+0000","last_peered":"2023-10-16T14:33:47.337476+0000","last_clean":"2023-10-16T14:33:47.337476+0000","last_became_active":"2023-10-16T14:33:46.440436+0000","last_became_peered":"2023-10-16T14:33:46.440436+0000","last_unstale":"2023-10-16T14:33:47.337476+0000","last_undegraded":"2023-10-16T14:33:47.337476+0000","last_fullsized":"2023-10-16T14:33:47.337476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:54.721248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338195+0000","last_change":"2023-10-16T14:33:45.300358+0000","last_active":"2023-10-16T14:33:47.338195+0000","last_peered":"2023-10-16T14:33:47.338195+0000","last_clean":"2023-10-16T14:33:47.338195+0000","last_became_active":"2023-10-16T14:33:45.298693+0000","last_became_peered":"2023-10-16T14:33:45.298693+0000","last_unstale":"2023-10-16T14:33:47.338195+0000","last_undegraded":"2023-10-16T14:33:47.338195+0000","last_fullsized":"2023-10-16T14:33:47.338195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:09.466762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335261+0000","last_change":"2023-10-16T14:33:46.445380+0000","last_active":"2023-10-16T14:33:47.335261+0000","last_peered":"2023-10-16T14:33:47.335261+0000","last_clean":"2023-10-16T14:33:47.335261+0000","last_became_active":"2023-10-16T14:33:46.445061+0000","last_became_peered":"2023-10-16T14:33:46.445061+0000","last_unstale":"2023-10-16T14:33:47.335261+0000","last_undegraded":"2023-10-16T14:33:47.335261+0000","last_fullsized":"2023-10-16T14:33:47.335261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:04.131034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335332+0000","last_change":"2023-10-16T14:33:45.305623+0000","last_active":"2023-10-16T14:33:47.335332+0000","last_peered":"2023-10-16T14:33:47.335332+0000","last_clean":"2023-10-16T14:33:47.335332+0000","last_became_active":"2023-10-16T14:33:45.305312+0000","last_became_peered":"2023-10-16T14:33:45.305312+0000","last_unstale":"2023-10-16T14:33:47.335332+0000","last_undegraded":"2023-10-16T14:33:47.335332+0000","last_fullsized":"2023-10-16T14:33:47.335332+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:32.054815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336993+0000","last_change":"2023-10-16T14:33:46.456145+0000","last_active":"2023-10-16T14:33:47.336993+0000","last_peered":"2023-10-16T14:33:47.336993+0000","last_clean":"2023-10-16T14:33:47.336993+0000","last_became_active":"2023-10-16T14:33:46.455842+0000","last_became_peered":"2023-10-16T14:33:46.455842+0000","last_unstale":"2023-10-16T14:33:47.336993+0000","last_undegraded":"2023-10-16T14:33:47.336993+0000","last_fullsized":"2023-10-16T14:33:47.336993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:03.520289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.329069+0000","last_change":"2023-10-16T14:33:45.300057+0000","last_active":"2023-10-16T14:33:47.329069+0000","last_peered":"2023-10-16T14:33:47.329069+0000","last_clean":"2023-10-16T14:33:47.329069+0000","last_became_active":"2023-10-16T14:33:45.299786+0000","last_became_peered":"2023-10-16T14:33:45.299786+0000","last_unstale":"2023-10-16T14:33:47.329069+0000","last_undegraded":"2023-10-16T14:33:47.329069+0000","last_fullsized":"2023-10-16T14:33:47.329069+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:05.012883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334194+0000","last_change":"2023-10-16T14:33:46.441142+0000","last_active":"2023-10-16T14:33:47.334194+0000","last_peered":"2023-10-16T14:33:47.334194+0000","last_clean":"2023-10-16T14:33:47.334194+0000","last_became_active":"2023-10-16T14:33:46.440529+0000","last_became_peered":"2023-10-16T14:33:46.440529+0000","last_unstale":"2023-10-16T14:33:47.334194+0000","last_undegraded":"2023-10-16T14:33:47.334194+0000","last_fullsized":"2023-10-16T14:33:47.334194+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:47.739038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339858+0000","last_change":"2023-10-16T14:33:45.312168+0000","last_active":"2023-10-16T14:33:47.339858+0000","last_peered":"2023-10-16T14:33:47.339858+0000","last_clean":"2023-10-16T14:33:47.339858+0000","last_became_active":"2023-10-16T14:33:45.311844+0000","last_became_peered":"2023-10-16T14:33:45.311844+0000","last_unstale":"2023-10-16T14:33:47.339858+0000","last_undegraded":"2023-10-16T14:33:47.339858+0000","last_fullsized":"2023-10-16T14:33:47.339858+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:47.477165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332833+0000","last_change":"2023-10-16T14:33:46.448662+0000","last_active":"2023-10-16T14:33:47.332833+0000","last_peered":"2023-10-16T14:33:47.332833+0000","last_clean":"2023-10-16T14:33:47.332833+0000","last_became_active":"2023-10-16T14:33:46.447853+0000","last_became_peered":"2023-10-16T14:33:46.447853+0000","last_unstale":"2023-10-16T14:33:47.332833+0000","last_undegraded":"2023-10-16T14:33:47.332833+0000","last_fullsized":"2023-10-16T14:33:47.332833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:45.455269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339560+0000","last_change":"2023-10-16T14:33:45.313953+0000","last_active":"2023-10-16T14:33:47.339560+0000","last_peered":"2023-10-16T14:33:47.339560+0000","last_clean":"2023-10-16T14:33:47.339560+0000","last_became_active":"2023-10-16T14:33:45.313165+0000","last_became_peered":"2023-10-16T14:33:45.313165+0000","last_unstale":"2023-10-16T14:33:47.339560+0000","last_undegraded":"2023-10-16T14:33:47.339560+0000","last_fullsized":"2023-10-16T14:33:47.339560+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:01.599682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338945+0000","last_change":"2023-10-16T14:33:46.450870+0000","last_active":"2023-10-16T14:33:47.338945+0000","last_peered":"2023-10-16T14:33:47.338945+0000","last_clean":"2023-10-16T14:33:47.338945+0000","last_became_active":"2023-10-16T14:33:46.450184+0000","last_became_peered":"2023-10-16T14:33:46.450184+0000","last_unstale":"2023-10-16T14:33:47.338945+0000","last_undegraded":"2023-10-16T14:33:47.338945+0000","last_fullsized":"2023-10-16T14:33:47.338945+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:00.990582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338824+0000","last_change":"2023-10-16T14:33:45.312478+0000","last_active":"2023-10-16T14:33:47.338824+0000","last_peered":"2023-10-16T14:33:47.338824+0000","last_clean":"2023-10-16T14:33:47.338824+0000","last_became_active":"2023-10-16T14:33:45.312100+0000","last_became_peered":"2023-10-16T14:33:45.312100+0000","last_unstale":"2023-10-16T14:33:47.338824+0000","last_undegraded":"2023-10-16T14:33:47.338824+0000","last_fullsized":"2023-10-16T14:33:47.338824+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:00.834604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334739+0000","last_change":"2023-10-16T14:33:46.446921+0000","last_active":"2023-10-16T14:33:47.334739+0000","last_peered":"2023-10-16T14:33:47.334739+0000","last_clean":"2023-10-16T14:33:47.334739+0000","last_became_active":"2023-10-16T14:33:46.446634+0000","last_became_peered":"2023-10-16T14:33:46.446634+0000","last_unstale":"2023-10-16T14:33:47.334739+0000","last_undegraded":"2023-10-16T14:33:47.334739+0000","last_fullsized":"2023-10-16T14:33:47.334739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:08.967220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339061+0000","last_change":"2023-10-16T14:33:45.308907+0000","last_active":"2023-10-16T14:33:47.339061+0000","last_peered":"2023-10-16T14:33:47.339061+0000","last_clean":"2023-10-16T14:33:47.339061+0000","last_became_active":"2023-10-16T14:33:45.307907+0000","last_became_peered":"2023-10-16T14:33:45.307907+0000","last_unstale":"2023-10-16T14:33:47.339061+0000","last_undegraded":"2023-10-16T14:33:47.339061+0000","last_fullsized":"2023-10-16T14:33:47.339061+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.583549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338417+0000","last_change":"2023-10-16T14:33:46.443511+0000","last_active":"2023-10-16T14:33:47.338417+0000","last_peered":"2023-10-16T14:33:47.338417+0000","last_clean":"2023-10-16T14:33:47.338417+0000","last_became_active":"2023-10-16T14:33:46.443222+0000","last_became_peered":"2023-10-16T14:33:46.443222+0000","last_unstale":"2023-10-16T14:33:47.338417+0000","last_undegraded":"2023-10-16T14:33:47.338417+0000","last_fullsized":"2023-10-16T14:33:47.338417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:55.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338804+0000","last_change":"2023-10-16T14:33:45.311992+0000","last_active":"2023-10-16T14:33:47.338804+0000","last_peered":"2023-10-16T14:33:47.338804+0000","last_clean":"2023-10-16T14:33:47.338804+0000","last_became_active":"2023-10-16T14:33:45.311699+0000","last_became_peered":"2023-10-16T14:33:45.311699+0000","last_unstale":"2023-10-16T14:33:47.338804+0000","last_undegraded":"2023-10-16T14:33:47.338804+0000","last_fullsized":"2023-10-16T14:33:47.338804+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:12.344471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338130+0000","last_change":"2023-10-16T14:33:46.452059+0000","last_active":"2023-10-16T14:33:47.338130+0000","last_peered":"2023-10-16T14:33:47.338130+0000","last_clean":"2023-10-16T14:33:47.338130+0000","last_became_active":"2023-10-16T14:33:46.451642+0000","last_became_peered":"2023-10-16T14:33:46.451642+0000","last_unstale":"2023-10-16T14:33:47.338130+0000","last_undegraded":"2023-10-16T14:33:47.338130+0000","last_fullsized":"2023-10-16T14:33:47.338130+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:57.034780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333658+0000","last_change":"2023-10-16T14:33:45.314352+0000","last_active":"2023-10-16T14:33:47.333658+0000","last_peered":"2023-10-16T14:33:47.333658+0000","last_clean":"2023-10-16T14:33:47.333658+0000","last_became_active":"2023-10-16T14:33:45.314053+0000","last_became_peered":"2023-10-16T14:33:45.314053+0000","last_unstale":"2023-10-16T14:33:47.333658+0000","last_undegraded":"2023-10-16T14:33:47.333658+0000","last_fullsized":"2023-10-16T14:33:47.333658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:21.581893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331711+0000","last_change":"2023-10-16T14:33:46.442461+0000","last_active":"2023-10-16T14:33:47.331711+0000","last_peered":"2023-10-16T14:33:47.331711+0000","last_clean":"2023-10-16T14:33:47.331711+0000","last_became_active":"2023-10-16T14:33:46.442096+0000","last_became_peered":"2023-10-16T14:33:46.442096+0000","last_unstale":"2023-10-16T14:33:47.331711+0000","last_undegraded":"2023-10-16T14:33:47.331711+0000","last_fullsized":"2023-10-16T14:33:47.331711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:45.484436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338253+0000","last_change":"2023-10-16T14:33:45.309699+0000","last_active":"2023-10-16T14:33:47.338253+0000","last_peered":"2023-10-16T14:33:47.338253+0000","last_clean":"2023-10-16T14:33:47.338253+0000","last_became_active":"2023-10-16T14:33:45.307655+0000","last_became_peered":"2023-10-16T14:33:45.307655+0000","last_unstale":"2023-10-16T14:33:47.338253+0000","last_undegraded":"2023-10-16T14:33:47.338253+0000","last_fullsized":"2023-10-16T14:33:47.338253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:00.999541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338111+0000","last_change":"2023-10-16T14:33:46.441423+0000","last_active":"2023-10-16T14:33:47.338111+0000","last_peered":"2023-10-16T14:33:47.338111+0000","last_clean":"2023-10-16T14:33:47.338111+0000","last_became_active":"2023-10-16T14:33:46.441122+0000","last_became_peered":"2023-10-16T14:33:46.441122+0000","last_unstale":"2023-10-16T14:33:47.338111+0000","last_undegraded":"2023-10-16T14:33:47.338111+0000","last_fullsized":"2023-10-16T14:33:47.338111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:50.007020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333006+0000","last_change":"2023-10-16T14:33:45.316120+0000","last_active":"2023-10-16T14:33:47.333006+0000","last_peered":"2023-10-16T14:33:47.333006+0000","last_clean":"2023-10-16T14:33:47.333006+0000","last_became_active":"2023-10-16T14:33:45.315806+0000","last_became_peered":"2023-10-16T14:33:45.315806+0000","last_unstale":"2023-10-16T14:33:47.333006+0000","last_undegraded":"2023-10-16T14:33:47.333006+0000","last_fullsized":"2023-10-16T14:33:47.333006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:33.532860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338993+0000","last_change":"2023-10-16T14:33:46.446951+0000","last_active":"2023-10-16T14:33:47.338993+0000","last_peered":"2023-10-16T14:33:47.338993+0000","last_clean":"2023-10-16T14:33:47.338993+0000","last_became_active":"2023-10-16T14:33:46.446656+0000","last_became_peered":"2023-10-16T14:33:46.446656+0000","last_unstale":"2023-10-16T14:33:47.338993+0000","last_undegraded":"2023-10-16T14:33:47.338993+0000","last_fullsized":"2023-10-16T14:33:47.338993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:49.706863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332396+0000","last_change":"2023-10-16T14:33:45.312611+0000","last_active":"2023-10-16T14:33:47.332396+0000","last_peered":"2023-10-16T14:33:47.332396+0000","last_clean":"2023-10-16T14:33:47.332396+0000","last_became_active":"2023-10-16T14:33:45.312003+0000","last_became_peered":"2023-10-16T14:33:45.312003+0000","last_unstale":"2023-10-16T14:33:47.332396+0000","last_undegraded":"2023-10-16T14:33:47.332396+0000","last_fullsized":"2023-10-16T14:33:47.332396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:06.804628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"99'186","reported_seq":508,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T14:33:46.332592+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:33:46.332592+0000","last_peered":"2023-10-16T14:33:46.332592+0000","last_clean":"2023-10-16T14:33:46.332592+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:33:46.332592+0000","last_undegraded":"2023-10-16T14:33:46.332592+0000","last_fullsized":"2023-10-16T14:33:46.332592+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575057,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41404,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330436,"statfs":{"total":96636764160,"available":96594366464,"internally_reserved":0,"allocated":1949696,"data_stored":1240519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0640000000000001,"15min":1.147},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.42799999999999999},"max":{"1min":1.784,"5min":1.833,"15min":1.9159999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.133,"15min":1.157},"min":{"1min":0.77800000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.522,"5min":1.6220000000000001,"15min":1.673},"last":1.5720000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1599999999999999,"15min":1.1799999999999999},"min":{"1min":0.71599999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3859999999999999,"5min":1.7030000000000001,"15min":1.8320000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.22,"15min":1.2390000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":0.88500000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.298,"15min":1.296},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63500000000000001},"max":{"1min":1.512,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.23,"15min":1.234},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.24}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.298,"15min":1.2989999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.359},{"interface":"front","average":{"1min":1.294,"5min":1.3500000000000001,"15min":1.379},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2509999999999999},"last":1.399}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3580000000000001,"15min":1.3600000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.835,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.484},{"interface":"front","average":{"1min":1.236,"5min":1.25,"15min":1.258},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.9530000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.2949999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3939999999999999,"15min":1.407},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.637},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4750000000000001,"15min":1.4870000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.6719999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.2850000000000001,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.446},{"interface":"front","average":{"1min":1.488,"5min":1.504,"15min":1.512},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":2.359,"5min":2.359,"15min":2.427},"last":1.599}]}]},{"osd":6,"up_from":13,"seq":55834575056,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1363968,"data_stored":650775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.169},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.6990000000000001,"5min":1.7569999999999999,"15min":2.2160000000000002},"last":7.7210000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.82599999999999996,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.577,"5min":2.2709999999999999,"15min":3.863},"last":6.0629999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.2509999999999999},"min":{"1min":0.92400000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.581,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":7.6459999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.3,"15min":1.29},"min":{"1min":0.81799999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7849999999999999,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":10.241}]},{"osd":2,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.147,"15min":1.1990000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.59899999999999998},"max":{"1min":1.3400000000000001,"5min":1.708,"15min":3.5800000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.232,"5min":1.3180000000000001,"15min":1.3500000000000001},"min":{"1min":0.80200000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.52,"5min":1.8799999999999999,"15min":2.1429999999999998},"last":5.694}]},{"osd":3,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3300000000000001,"15min":1.353},"min":{"1min":0.85599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.8480000000000001,"5min":1.96,"15min":4.9790000000000001},"last":5.9139999999999997},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.286,"15min":1.3400000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":1.6080000000000001,"5min":1.913,"15min":5.4160000000000004},"last":2.2349999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.296,"15min":1.3},"min":{"1min":0.748,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8009999999999999,"5min":2.1070000000000002,"15min":5},"last":13.826000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3640000000000001,"15min":1.377},"min":{"1min":0.94999999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.897,"5min":2.1789999999999998,"15min":5.4640000000000004},"last":5.9790000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2949999999999999,"15min":1.3160000000000001},"min":{"1min":0.89600000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.76,"5min":1.859,"15min":3.9279999999999999},"last":3.1419999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.482,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":3.6749999999999998},"last":5.8700000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.5600000000000001},"min":{"1min":1.105,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":4.6859999999999999},"last":6.0199999999999996},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.508,"15min":1.542},"min":{"1min":1.0840000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9139999999999999,"5min":2.0459999999999998,"15min":4.883},"last":2.1779999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":1368064,"data_stored":651715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.099,"15min":1.1160000000000001},"min":{"1min":0.73599999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5760000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1990000000000001,"15min":1.1819999999999999},"min":{"1min":0.63300000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.7729999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.0880000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.444,"5min":1.726,"15min":1.726},"last":1.032},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.2430000000000001},"min":{"1min":0.81499999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5249999999999999,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.244,"15min":1.2390000000000001},"min":{"1min":0.71599999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5720000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3200000000000001,"15min":1.353},"min":{"1min":0.85799999999999998,"5min":0.625,"15min":0.625},"max":{"1min":2.0049999999999999,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.3089999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.76900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.351,"15min":1.3779999999999999},"min":{"1min":0.85699999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.6819999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.3859999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3049999999999999,"15min":1.3169999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.54700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5049999999999999,"15min":1.5029999999999999},"min":{"1min":1.073,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.853,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.5489999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3520000000000001,"15min":1.3819999999999999},"min":{"1min":0.97299999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.218},{"interface":"front","average":{"1min":1.385,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":0.77700000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.587}]},{"osd":7,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.5489999999999999},"min":{"1min":0.93400000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.956,"5min":2.641,"15min":2.641},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.579,"15min":1.603},"min":{"1min":1.246,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.1909999999999998,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.6180000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575057,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46392,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325448,"statfs":{"total":96636764160,"available":96589258752,"internally_reserved":0,"allocated":1351680,"data_stored":650081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1559999999999999,"15min":1.1479999999999999},"min":{"1min":0.58999999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7689999999999999,"5min":1.881,"15min":2.0339999999999998},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.1020000000000001,"15min":1.1639999999999999},"min":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4830000000000001,"5min":1.7589999999999999,"15min":1.9199999999999999},"last":1.256}]},{"osd":1,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1479999999999999,"15min":1.177},"min":{"1min":0.58399999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.78},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.1910000000000001,"15min":1.2330000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.716,"5min":1.782,"15min":2.3690000000000002},"last":1.329}]},{"osd":2,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3700000000000001,"15min":1.399},"min":{"1min":0.68100000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.54},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.3400000000000001,"15min":1.365},"min":{"1min":0.69399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7769999999999999,"5min":1.8799999999999999,"15min":1.8839999999999999},"last":1.5629999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.5920000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.268,"15min":1.304},"min":{"1min":0.53900000000000003,"5min":0.437,"15min":0.437},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.595}]},{"osd":5,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.306,"15min":1.288},"min":{"1min":0.78800000000000003,"5min":0.61499999999999999,"15min":0.48699999999999999},"max":{"1min":1.639,"5min":1.696,"15min":1.8220000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.3460000000000001,"15min":1.349},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.806,"5min":1.806,"15min":1.964},"last":1.472}]},{"osd":6,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.323,"15min":1.2909999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4690000000000001,"15min":1.4610000000000001},"min":{"1min":0.74199999999999999,"5min":0.71999999999999997,"15min":0.44700000000000001},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.2890000000000001},"last":1.6339999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.4730000000000001,"15min":1.4690000000000001},"min":{"1min":0.76100000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9259999999999999},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.466,"15min":1.458},"min":{"1min":0.66700000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8979999999999999,"5min":2.0579999999999998,"15min":2.29},"last":1.704}]}]},{"osd":3,"up_from":13,"seq":55834575056,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40848,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330992,"statfs":{"total":96636764160,"available":96594935808,"internally_reserved":0,"allocated":1376256,"data_stored":657993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.50800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0609999999999999,"5min":1.4419999999999999,"15min":1.6639999999999999},"last":1.696},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":1.0169999999999999},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.3320000000000001,"5min":1.913,"15min":1.982},"last":2.1819999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90000000000000002,"15min":0.94099999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.32500000000000001},"max":{"1min":1.337,"5min":1.7390000000000001,"15min":2.149},"last":2.7850000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.089,"15min":1.139},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.415,"5min":1.6499999999999999,"15min":2.1600000000000001},"last":3.024}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2050000000000001,"15min":1.3180000000000001},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.44800000000000001},"max":{"1min":1.6830000000000001,"5min":1.895,"15min":2.1259999999999999},"last":3.3420000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.331,"15min":1.4259999999999999},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.5},"max":{"1min":1.8029999999999999,"5min":1.8660000000000001,"15min":2.6299999999999999},"last":2.5609999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1910000000000001,"15min":1.2569999999999999},"min":{"1min":0.77100000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.768,"5min":1.768,"15min":1.9239999999999999},"last":2.02},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.298,"15min":1.3300000000000001},"min":{"1min":0.93999999999999995,"5min":0.629,"15min":0.629},"max":{"1min":1.7689999999999999,"5min":1.909,"15min":1.9259999999999999},"last":2.9430000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.4550000000000001,"15min":1.516},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7170000000000001,"5min":2.0070000000000001,"15min":2.23},"last":3.206},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3899999999999999,"15min":1.468},"min":{"1min":0.90700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6220000000000001,"5min":1.95,"15min":2.3570000000000002},"last":3.3809999999999998}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.361,"15min":1.421},"min":{"1min":0.95999999999999996,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.837,"5min":2.181,"15min":2.181},"last":2.2970000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4419999999999999,"15min":1.4910000000000001},"min":{"1min":0.82599999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.6799999999999999,"5min":1.9510000000000001,"15min":2.347},"last":3.0910000000000002}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.4870000000000001,"15min":1.581},"min":{"1min":1.0069999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7310000000000001,"5min":2.1589999999999998,"15min":2.395},"last":5.5119999999999996},{"interface":"front","average":{"1min":1.347,"5min":1.472,"15min":1.5509999999999999},"min":{"1min":0.749,"5min":0.629,"15min":0.629},"max":{"1min":1.8160000000000001,"5min":2.1000000000000001,"15min":2.1739999999999999},"last":2.5209999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575057,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28348,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343492,"statfs":{"total":96636764160,"available":96607735808,"internally_reserved":0,"allocated":1355776,"data_stored":650663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.4319999999999999,"5min":1.556,"15min":2.472},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96299999999999997,"15min":1.024},"min":{"1min":0.48299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3560000000000001,"5min":1.585,"15min":2.3260000000000001},"last":0.83299999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.0029999999999999},"min":{"1min":0.50800000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.454,"5min":3.234,"15min":3.234},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.1459999999999999,"15min":1.1619999999999999},"min":{"1min":0.52200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6299999999999999,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.0940000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.1830000000000001,"15min":1.234},"min":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.2629999999999999,"15min":1.2969999999999999},"min":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6879999999999999,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.151}]},{"osd":4,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.415,"15min":1.425},"min":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.766,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.3400000000000001,"15min":1.3620000000000001},"min":{"1min":0.80400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6759999999999999,"5min":3.016,"15min":3.016},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.393,"15min":1.415},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8169999999999999,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.327},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.552},"min":{"1min":0.96599999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.3039999999999998,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.585}]},{"osd":6,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.448,"15min":1.486},"min":{"1min":0.85899999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.0129999999999999,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.5,"15min":1.5289999999999999},"min":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.3610000000000002,"5min":2.5920000000000001,"15min":3.3929999999999998},"last":1.4510000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.52,"15min":1.5640000000000001},"min":{"1min":0.80700000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7789999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":0.875},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.617,"15min":1.6439999999999999},"min":{"1min":0.88,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.206,"5min":2.7519999999999998,"15min":2.944},"last":1.542}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":1363968,"data_stored":650275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.85399999999999998,"15min":0.90400000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.409,"5min":1.421,"15min":1.821},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.0149999999999999,"15min":1.0980000000000001},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.827,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.92600000000000005,"15min":0.98699999999999999},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.4330000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0680000000000001,"15min":1.1579999999999999},"min":{"1min":0.51900000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6899999999999999,"5min":1.756,"15min":1.796},"last":1.0149999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.2150000000000001,"15min":1.3160000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.627,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.748},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.129,"15min":1.1930000000000001},"min":{"1min":0.55300000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5609999999999999,"5min":1.79,"15min":1.855},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.216,"15min":1.286},"min":{"1min":0.75600000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.958,"5min":2.069,"15min":2.069},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.349,"15min":1.4219999999999999},"min":{"1min":0.74099999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.9279999999999999,"5min":2.899,"15min":2.899},"last":1.589}]},{"osd":5,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3640000000000001,"15min":1.4379999999999999},"min":{"1min":0.88700000000000001,"5min":0.623,"15min":0.623},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.054},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.347,"15min":1.427},"min":{"1min":0.71199999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.387}]},{"osd":6,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.4890000000000001,"15min":1.5509999999999999},"min":{"1min":0.80000000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.462,"15min":1.538},"min":{"1min":0.81100000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.151}]},{"osd":7,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4650000000000001,"15min":1.5580000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.0870000000000002,"5min":2.2000000000000002,"15min":2.262},"last":1.522}]}]},{"osd":0,"up_from":13,"seq":55834575057,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37252,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334588,"statfs":{"total":96636764160,"available":96598618112,"internally_reserved":0,"allocated":1949696,"data_stored":1240483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.875,"15min":0.94299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.7929999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.04,"5min":1.097,"15min":1.163},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.36399999999999999},"max":{"1min":1.508,"5min":1.833,"15min":1.867},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.204,"15min":1.2470000000000001},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1.6359999999999999,"5min":1.887,"15min":2.2970000000000002},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.093,"5min":1.159,"15min":1.2050000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6000000000000001,"5min":2.012,"15min":2.2330000000000001},"last":1.764}]},{"osd":3,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.96099999999999997,"15min":1.0029999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40100000000000002},"max":{"1min":1.3149999999999999,"5min":1.889,"15min":1.9410000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.181,"15min":1.238},"min":{"1min":0.45000000000000001,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.823,"5min":1.823,"15min":2.5339999999999998},"last":1.6120000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.266,"15min":1.302},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.61599999999999999},"max":{"1min":1.786,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4419999999999999,"15min":1.462},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.7809999999999999,"5min":2.032,"15min":2.6259999999999999},"last":1.25}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.383,"15min":1.419},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.02,"5min":2.097,"15min":2.5880000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.302,"15min":1.3600000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.67900000000000005},"max":{"1min":1.778,"5min":1.778,"15min":2.4889999999999999},"last":1.1240000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5389999999999999,"15min":1.573},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8819999999999999,"5min":2.097,"15min":2.7599999999999998},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.4770000000000001,"15min":1.5},"min":{"1min":1.02,"5min":0.98899999999999999,"15min":0.89000000000000001},"max":{"1min":1.9299999999999999,"5min":2.1459999999999999,"15min":2.5089999999999999},"last":1.3160000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5409999999999999,"15min":1.585},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1040000000000001,"5min":2.2650000000000001,"15min":2.6509999999999998},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.472,"15min":1.544},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.976},"last":1.532}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:33:50.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:33:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.204+0000 7f2f66f47640 1 Processor -- start 2023-10-16T14:33:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.204+0000 7f2f66f47640 1 -- start start 2023-10-16T14:33:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f66f47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f6006c040 0x7f2f6006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f66f47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2f600cda80 0x7f2f6015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f66f47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6006c960 con 0x7f2f600ad970 2023-10-16T14:33:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f66f47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6015b9d0 con 0x7f2f6006c040 2023-10-16T14:33:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f66f47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6015bcb0 con 0x7f2f600cda80 2023-10-16T14:33:51.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f65f45640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f6006c040 0x7f2f6006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f65f45640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f6006c040 0x7f2f6006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42920/0 (socket says 172.21.15.29:42920) 2023-10-16T14:33:51.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.206+0000 7f2f65f45640 1 -- 172.21.15.29:0/1184492954 learned_addr learned my addr 172.21.15.29:0/1184492954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:51.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.207+0000 7f2f65f45640 1 -- 172.21.15.29:0/1184492954 >> v2:172.21.15.170:3300/0 conn(0x7f2f600cda80 msgr2=0x7f2f6015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:51.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.207+0000 7f2f65f45640 1 --2- 172.21.15.29:0/1184492954 >> v2:172.21.15.170:3300/0 conn(0x7f2f600cda80 0x7f2f6015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.207+0000 7f2f65f45640 1 -- 172.21.15.29:0/1184492954 >> v1:172.21.15.170:6789/0 conn(0x7f2f600ad970 legacy=0x7f2f6015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:51.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.207+0000 7f2f65f45640 1 -- 172.21.15.29:0/1184492954 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6015bf90 con 0x7f2f6006c040 2023-10-16T14:33:51.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.208+0000 7f2f65f45640 1 --2- 172.21.15.29:0/1184492954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f6006c420 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2f600cf300 tx=0x7f2f5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38b6caffeeee45b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:51.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.208+0000 7f2f64f43640 1 -- 172.21.15.29:0/1184492954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c031870 con 0x7f2f6006c040 2023-10-16T14:33:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.208+0000 7f2f64f43640 1 -- 172.21.15.29:0/1184492954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c0319d0 con 0x7f2f6006c040 2023-10-16T14:33:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.208+0000 7f2f64f43640 1 -- 172.21.15.29:0/1184492954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c031d60 con 0x7f2f6006c040 2023-10-16T14:33:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 -- 172.21.15.29:0/1184492954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 msgr2=0x7f2f6006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 --2- 172.21.15.29:0/1184492954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f6006c420 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2f600cf300 tx=0x7f2f5c02b910 comp rx=0 tx=0).stop 2023-10-16T14:33:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 -- 172.21.15.29:0/1184492954 shutdown_connections 2023-10-16T14:33:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 --2- 172.21.15.29:0/1184492954 >> v2:172.21.15.170:3300/0 conn(0x7f2f600cda80 0x7f2f6015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 --2- 172.21.15.29:0/1184492954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f6006c420 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 -- 172.21.15.29:0/1184492954 >> 172.21.15.29:0/1184492954 conn(0x7f2f600ce6a0 msgr2=0x7f2f600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 -- 172.21.15.29:0/1184492954 shutdown_connections 2023-10-16T14:33:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.209+0000 7f2f66f47640 1 -- 172.21.15.29:0/1184492954 wait complete. 2023-10-16T14:33:51.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.211+0000 7f2f66f47640 1 Processor -- start 2023-10-16T14:33:51.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.211+0000 7f2f66f47640 1 -- start start 2023-10-16T14:33:51.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f66f47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f60153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f66f47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f600ad970 0x7f2f600c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f66f47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f600cda80 0x7f2f600c3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f66f47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f600c6490 con 0x7f2f6006c040 2023-10-16T14:33:51.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f66f47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f600c5880 con 0x7f2f600ad970 2023-10-16T14:33:51.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f66f47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f600c5b60 con 0x7f2f600cda80 2023-10-16T14:33:51.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f65f45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f60153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f65f45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f60153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42934/0 (socket says 172.21.15.29:42934) 2023-10-16T14:33:51.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.212+0000 7f2f65f45640 1 -- 172.21.15.29:0/3412058277 learned_addr learned my addr 172.21.15.29:0/3412058277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:51.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65744640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f600ad970 0x7f2f600c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65f45640 1 -- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f600cda80 msgr2=0x7f2f600c3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65f45640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f600cda80 0x7f2f600c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65f45640 1 -- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f600ad970 msgr2=0x7f2f600c3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65f45640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f600ad970 0x7f2f600c3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65f45640 1 -- 172.21.15.29:0/3412058277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0094d0 con 0x7f2f6006c040 2023-10-16T14:33:51.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65744640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f600ad970 0x7f2f600c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:51.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.213+0000 7f2f65f45640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f60153c90 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c000c00 tx=0x7f2f5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:51.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.214+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c03ad10 con 0x7f2f6006c040 2023-10-16T14:33:51.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.214+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c03ae70 con 0x7f2f6006c040 2023-10-16T14:33:51.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.214+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c044610 con 0x7f2f6006c040 2023-10-16T14:33:51.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.214+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f600c5e40 con 0x7f2f6006c040 2023-10-16T14:33:51.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.214+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f601541d0 con 0x7f2f6006c040 2023-10-16T14:33:51.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.215+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6006b330 con 0x7f2f6006c040 2023-10-16T14:33:51.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.216+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2f5c04d450 con 0x7f2f6006c040 2023-10-16T14:33:51.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.217+0000 7f2f567fc640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f2f4406af90 0x7f2f4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.217+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f5c02d040 con 0x7f2f6006c040 2023-10-16T14:33:51.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.218+0000 7f2f65744640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f2f4406af90 0x7f2f4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.219+0000 7f2f65744640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f2f4406af90 0x7f2f4406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f601e7230 tx=0x7f2f50009040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:51.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.221+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2f5c076e40 con 0x7f2f6006c040 2023-10-16T14:33:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.411+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 --> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f601543c0 con 0x7f2f4406af90 2023-10-16T14:33:51.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.434+0000 7f2f567fc640 1 -- 172.21.15.29:0/3412058277 <== mgr.6076 v2:172.21.15.170:6832/2925184086 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287358 (secure 0 0 0) 0x7f2f50005030 con 0x7f2f4406af90 2023-10-16T14:33:51.440 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:33:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f2f4406af90 msgr2=0x7f2f4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f2f4406af90 0x7f2f4406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f601e7230 tx=0x7f2f50009040 comp rx=0 tx=0).stop 2023-10-16T14:33:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 msgr2=0x7f2f60153c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f60153c90 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c000c00 tx=0x7f2f5c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 shutdown_connections 2023-10-16T14:33:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f2f4406af90 0x7f2f4406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f600cda80 0x7f2f600c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f600ad970 0x7f2f600c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 --2- 172.21.15.29:0/3412058277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f6006c040 0x7f2f60153c90 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.440+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 >> 172.21.15.29:0/3412058277 conn(0x7f2f600ce6a0 msgr2=0x7f2f6015d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.441+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 shutdown_connections 2023-10-16T14:33:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.441+0000 7f2f66f47640 1 -- 172.21.15.29:0/3412058277 wait complete. 2023-10-16T14:33:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T14:33:50.619222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":327,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":322320,"kb_used_data":11796,"kb_used_omap":12,"kb_used_meta":310387,"kb_avail":754652400,"statfs":{"total":773094113280,"available":772764057600,"internally_reserved":0,"allocated":12079104,"data_stored":6392504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":317836857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002336"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339122+0000","last_change":"2023-10-16T14:33:46.447246+0000","last_active":"2023-10-16T14:33:47.339122+0000","last_peered":"2023-10-16T14:33:47.339122+0000","last_clean":"2023-10-16T14:33:47.339122+0000","last_became_active":"2023-10-16T14:33:46.445501+0000","last_became_peered":"2023-10-16T14:33:46.445501+0000","last_unstale":"2023-10-16T14:33:47.339122+0000","last_undegraded":"2023-10-16T14:33:47.339122+0000","last_fullsized":"2023-10-16T14:33:47.339122+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:03.655183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339591+0000","last_change":"2023-10-16T14:33:46.453278+0000","last_active":"2023-10-16T14:33:47.339591+0000","last_peered":"2023-10-16T14:33:47.339591+0000","last_clean":"2023-10-16T14:33:47.339591+0000","last_became_active":"2023-10-16T14:33:46.452982+0000","last_became_peered":"2023-10-16T14:33:46.452982+0000","last_unstale":"2023-10-16T14:33:47.339591+0000","last_undegraded":"2023-10-16T14:33:47.339591+0000","last_fullsized":"2023-10-16T14:33:47.339591+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:00.990582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331799+0000","last_change":"2023-10-16T14:33:46.449197+0000","last_active":"2023-10-16T14:33:47.331799+0000","last_peered":"2023-10-16T14:33:47.331799+0000","last_clean":"2023-10-16T14:33:47.331799+0000","last_became_active":"2023-10-16T14:33:46.447894+0000","last_became_peered":"2023-10-16T14:33:46.447894+0000","last_unstale":"2023-10-16T14:33:47.331799+0000","last_undegraded":"2023-10-16T14:33:47.331799+0000","last_fullsized":"2023-10-16T14:33:47.331799+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:37.818459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335999+0000","last_change":"2023-10-16T14:33:46.451178+0000","last_active":"2023-10-16T14:33:47.335999+0000","last_peered":"2023-10-16T14:33:47.335999+0000","last_clean":"2023-10-16T14:33:47.335999+0000","last_became_active":"2023-10-16T14:33:46.450875+0000","last_became_peered":"2023-10-16T14:33:46.450875+0000","last_unstale":"2023-10-16T14:33:47.335999+0000","last_undegraded":"2023-10-16T14:33:47.335999+0000","last_fullsized":"2023-10-16T14:33:47.335999+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:03.428608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333139+0000","last_change":"2023-10-16T14:33:46.443085+0000","last_active":"2023-10-16T14:33:47.333139+0000","last_peered":"2023-10-16T14:33:47.333139+0000","last_clean":"2023-10-16T14:33:47.333139+0000","last_became_active":"2023-10-16T14:33:46.442795+0000","last_became_peered":"2023-10-16T14:33:46.442795+0000","last_unstale":"2023-10-16T14:33:47.333139+0000","last_undegraded":"2023-10-16T14:33:47.333139+0000","last_fullsized":"2023-10-16T14:33:47.333139+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:33.576593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338547+0000","last_change":"2023-10-16T14:33:46.442134+0000","last_active":"2023-10-16T14:33:47.338547+0000","last_peered":"2023-10-16T14:33:47.338547+0000","last_clean":"2023-10-16T14:33:47.338547+0000","last_became_active":"2023-10-16T14:33:46.441864+0000","last_became_peered":"2023-10-16T14:33:46.441864+0000","last_unstale":"2023-10-16T14:33:47.338547+0000","last_undegraded":"2023-10-16T14:33:47.338547+0000","last_fullsized":"2023-10-16T14:33:47.338547+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:55.271107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335426+0000","last_change":"2023-10-16T14:33:46.340437+0000","last_active":"2023-10-16T14:33:47.335426+0000","last_peered":"2023-10-16T14:33:47.335426+0000","last_clean":"2023-10-16T14:33:47.335426+0000","last_became_active":"2023-10-16T14:33:46.339528+0000","last_became_peered":"2023-10-16T14:33:46.339528+0000","last_unstale":"2023-10-16T14:33:47.335426+0000","last_undegraded":"2023-10-16T14:33:47.335426+0000","last_fullsized":"2023-10-16T14:33:47.335426+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:35.952492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337184+0000","last_change":"2023-10-16T14:33:46.455992+0000","last_active":"2023-10-16T14:33:47.337184+0000","last_peered":"2023-10-16T14:33:47.337184+0000","last_clean":"2023-10-16T14:33:47.337184+0000","last_became_active":"2023-10-16T14:33:46.455246+0000","last_became_peered":"2023-10-16T14:33:46.455246+0000","last_unstale":"2023-10-16T14:33:47.337184+0000","last_undegraded":"2023-10-16T14:33:47.337184+0000","last_fullsized":"2023-10-16T14:33:47.337184+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:17.956937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339064+0000","last_change":"2023-10-16T14:33:46.460172+0000","last_active":"2023-10-16T14:33:47.339064+0000","last_peered":"2023-10-16T14:33:47.339064+0000","last_clean":"2023-10-16T14:33:47.339064+0000","last_became_active":"2023-10-16T14:33:46.459764+0000","last_became_peered":"2023-10-16T14:33:46.459764+0000","last_unstale":"2023-10-16T14:33:47.339064+0000","last_undegraded":"2023-10-16T14:33:47.339064+0000","last_fullsized":"2023-10-16T14:33:47.339064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:50.040056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338888+0000","last_change":"2023-10-16T14:33:46.445061+0000","last_active":"2023-10-16T14:33:47.338888+0000","last_peered":"2023-10-16T14:33:47.338888+0000","last_clean":"2023-10-16T14:33:47.338888+0000","last_became_active":"2023-10-16T14:33:46.444753+0000","last_became_peered":"2023-10-16T14:33:46.444753+0000","last_unstale":"2023-10-16T14:33:47.338888+0000","last_undegraded":"2023-10-16T14:33:47.338888+0000","last_fullsized":"2023-10-16T14:33:47.338888+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:49.706863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340216+0000","last_change":"2023-10-16T14:33:46.441615+0000","last_active":"2023-10-16T14:33:47.340216+0000","last_peered":"2023-10-16T14:33:47.340216+0000","last_clean":"2023-10-16T14:33:47.340216+0000","last_became_active":"2023-10-16T14:33:46.441297+0000","last_became_peered":"2023-10-16T14:33:46.441297+0000","last_unstale":"2023-10-16T14:33:47.340216+0000","last_undegraded":"2023-10-16T14:33:47.340216+0000","last_fullsized":"2023-10-16T14:33:47.340216+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:25.410021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338164+0000","last_change":"2023-10-16T14:33:46.448492+0000","last_active":"2023-10-16T14:33:47.338164+0000","last_peered":"2023-10-16T14:33:47.338164+0000","last_clean":"2023-10-16T14:33:47.338164+0000","last_became_active":"2023-10-16T14:33:46.448212+0000","last_became_peered":"2023-10-16T14:33:46.448212+0000","last_unstale":"2023-10-16T14:33:47.338164+0000","last_undegraded":"2023-10-16T14:33:47.338164+0000","last_fullsized":"2023-10-16T14:33:47.338164+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:28.072982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331734+0000","last_change":"2023-10-16T14:33:46.343112+0000","last_active":"2023-10-16T14:33:47.331734+0000","last_peered":"2023-10-16T14:33:47.331734+0000","last_clean":"2023-10-16T14:33:47.331734+0000","last_became_active":"2023-10-16T14:33:46.342256+0000","last_became_peered":"2023-10-16T14:33:46.342256+0000","last_unstale":"2023-10-16T14:33:47.331734+0000","last_undegraded":"2023-10-16T14:33:47.331734+0000","last_fullsized":"2023-10-16T14:33:47.331734+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:58.146309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335944+0000","last_change":"2023-10-16T14:33:46.456701+0000","last_active":"2023-10-16T14:33:47.335944+0000","last_peered":"2023-10-16T14:33:47.335944+0000","last_clean":"2023-10-16T14:33:47.335944+0000","last_became_active":"2023-10-16T14:33:46.456187+0000","last_became_peered":"2023-10-16T14:33:46.456187+0000","last_unstale":"2023-10-16T14:33:47.335944+0000","last_undegraded":"2023-10-16T14:33:47.335944+0000","last_fullsized":"2023-10-16T14:33:47.335944+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:26.588555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330183+0000","last_change":"2023-10-16T14:33:46.441161+0000","last_active":"2023-10-16T14:33:47.330183+0000","last_peered":"2023-10-16T14:33:47.330183+0000","last_clean":"2023-10-16T14:33:47.330183+0000","last_became_active":"2023-10-16T14:33:46.440688+0000","last_became_peered":"2023-10-16T14:33:46.440688+0000","last_unstale":"2023-10-16T14:33:47.330183+0000","last_undegraded":"2023-10-16T14:33:47.330183+0000","last_fullsized":"2023-10-16T14:33:47.330183+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:57.146819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332767+0000","last_change":"2023-10-16T14:33:46.440936+0000","last_active":"2023-10-16T14:33:47.332767+0000","last_peered":"2023-10-16T14:33:47.332767+0000","last_clean":"2023-10-16T14:33:47.332767+0000","last_became_active":"2023-10-16T14:33:46.440658+0000","last_became_peered":"2023-10-16T14:33:46.440658+0000","last_unstale":"2023-10-16T14:33:47.332767+0000","last_undegraded":"2023-10-16T14:33:47.332767+0000","last_fullsized":"2023-10-16T14:33:47.332767+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:26.641874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333738+0000","last_change":"2023-10-16T14:33:46.336574+0000","last_active":"2023-10-16T14:33:47.333738+0000","last_peered":"2023-10-16T14:33:47.333738+0000","last_clean":"2023-10-16T14:33:47.333738+0000","last_became_active":"2023-10-16T14:33:46.335159+0000","last_became_peered":"2023-10-16T14:33:46.335159+0000","last_unstale":"2023-10-16T14:33:47.333738+0000","last_undegraded":"2023-10-16T14:33:47.333738+0000","last_fullsized":"2023-10-16T14:33:47.333738+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:19.542085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339084+0000","last_change":"2023-10-16T14:33:46.446834+0000","last_active":"2023-10-16T14:33:47.339084+0000","last_peered":"2023-10-16T14:33:47.339084+0000","last_clean":"2023-10-16T14:33:47.339084+0000","last_became_active":"2023-10-16T14:33:46.446514+0000","last_became_peered":"2023-10-16T14:33:46.446514+0000","last_unstale":"2023-10-16T14:33:47.339084+0000","last_undegraded":"2023-10-16T14:33:47.339084+0000","last_fullsized":"2023-10-16T14:33:47.339084+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:28.072982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337638+0000","last_change":"2023-10-16T14:33:46.446138+0000","last_active":"2023-10-16T14:33:47.337638+0000","last_peered":"2023-10-16T14:33:47.337638+0000","last_clean":"2023-10-16T14:33:47.337638+0000","last_became_active":"2023-10-16T14:33:46.445839+0000","last_became_peered":"2023-10-16T14:33:46.445839+0000","last_unstale":"2023-10-16T14:33:47.337638+0000","last_undegraded":"2023-10-16T14:33:47.337638+0000","last_fullsized":"2023-10-16T14:33:47.337638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:10.471768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336796+0000","last_change":"2023-10-16T14:33:46.461085+0000","last_active":"2023-10-16T14:33:47.336796+0000","last_peered":"2023-10-16T14:33:47.336796+0000","last_clean":"2023-10-16T14:33:47.336796+0000","last_became_active":"2023-10-16T14:33:46.460796+0000","last_became_peered":"2023-10-16T14:33:46.460796+0000","last_unstale":"2023-10-16T14:33:47.336796+0000","last_undegraded":"2023-10-16T14:33:47.336796+0000","last_fullsized":"2023-10-16T14:33:47.336796+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:50.007020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340310+0000","last_change":"2023-10-16T14:33:46.462544+0000","last_active":"2023-10-16T14:33:47.340310+0000","last_peered":"2023-10-16T14:33:47.340310+0000","last_clean":"2023-10-16T14:33:47.340310+0000","last_became_active":"2023-10-16T14:33:46.461995+0000","last_became_peered":"2023-10-16T14:33:46.461995+0000","last_unstale":"2023-10-16T14:33:47.340310+0000","last_undegraded":"2023-10-16T14:33:47.340310+0000","last_fullsized":"2023-10-16T14:33:47.340310+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:17.006796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340215+0000","last_change":"2023-10-16T14:33:46.459119+0000","last_active":"2023-10-16T14:33:47.340215+0000","last_peered":"2023-10-16T14:33:47.340215+0000","last_clean":"2023-10-16T14:33:47.340215+0000","last_became_active":"2023-10-16T14:33:46.458853+0000","last_became_peered":"2023-10-16T14:33:46.458853+0000","last_unstale":"2023-10-16T14:33:47.340215+0000","last_undegraded":"2023-10-16T14:33:47.340215+0000","last_fullsized":"2023-10-16T14:33:47.340215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:51.326358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337794+0000","last_change":"2023-10-16T14:33:46.460358+0000","last_active":"2023-10-16T14:33:47.337794+0000","last_peered":"2023-10-16T14:33:47.337794+0000","last_clean":"2023-10-16T14:33:47.337794+0000","last_became_active":"2023-10-16T14:33:46.459733+0000","last_became_peered":"2023-10-16T14:33:46.459733+0000","last_unstale":"2023-10-16T14:33:47.337794+0000","last_undegraded":"2023-10-16T14:33:47.337794+0000","last_fullsized":"2023-10-16T14:33:47.337794+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:03.428608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334937+0000","last_change":"2023-10-16T14:33:46.450285+0000","last_active":"2023-10-16T14:33:47.334937+0000","last_peered":"2023-10-16T14:33:47.334937+0000","last_clean":"2023-10-16T14:33:47.334937+0000","last_became_active":"2023-10-16T14:33:46.449970+0000","last_became_peered":"2023-10-16T14:33:46.449970+0000","last_unstale":"2023-10-16T14:33:47.334937+0000","last_undegraded":"2023-10-16T14:33:47.334937+0000","last_fullsized":"2023-10-16T14:33:47.334937+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:01.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339841+0000","last_change":"2023-10-16T14:33:46.454821+0000","last_active":"2023-10-16T14:33:47.339841+0000","last_peered":"2023-10-16T14:33:47.339841+0000","last_clean":"2023-10-16T14:33:47.339841+0000","last_became_active":"2023-10-16T14:33:46.453683+0000","last_became_peered":"2023-10-16T14:33:46.453683+0000","last_unstale":"2023-10-16T14:33:47.339841+0000","last_undegraded":"2023-10-16T14:33:47.339841+0000","last_fullsized":"2023-10-16T14:33:47.339841+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:03.655183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331496+0000","last_change":"2023-10-16T14:33:46.454153+0000","last_active":"2023-10-16T14:33:47.331496+0000","last_peered":"2023-10-16T14:33:47.331496+0000","last_clean":"2023-10-16T14:33:47.331496+0000","last_became_active":"2023-10-16T14:33:46.453682+0000","last_became_peered":"2023-10-16T14:33:46.453682+0000","last_unstale":"2023-10-16T14:33:47.331496+0000","last_undegraded":"2023-10-16T14:33:47.331496+0000","last_fullsized":"2023-10-16T14:33:47.331496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:51.182380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334917+0000","last_change":"2023-10-16T14:33:46.443435+0000","last_active":"2023-10-16T14:33:47.334917+0000","last_peered":"2023-10-16T14:33:47.334917+0000","last_clean":"2023-10-16T14:33:47.334917+0000","last_became_active":"2023-10-16T14:33:46.443132+0000","last_became_peered":"2023-10-16T14:33:46.443132+0000","last_unstale":"2023-10-16T14:33:47.334917+0000","last_undegraded":"2023-10-16T14:33:47.334917+0000","last_fullsized":"2023-10-16T14:33:47.334917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:51.812904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338865+0000","last_change":"2023-10-16T14:33:46.451427+0000","last_active":"2023-10-16T14:33:47.338865+0000","last_peered":"2023-10-16T14:33:47.338865+0000","last_clean":"2023-10-16T14:33:47.338865+0000","last_became_active":"2023-10-16T14:33:46.450955+0000","last_became_peered":"2023-10-16T14:33:46.450955+0000","last_unstale":"2023-10-16T14:33:47.338865+0000","last_undegraded":"2023-10-16T14:33:47.338865+0000","last_fullsized":"2023-10-16T14:33:47.338865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:20.311567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338214+0000","last_change":"2023-10-16T14:33:46.450726+0000","last_active":"2023-10-16T14:33:47.338214+0000","last_peered":"2023-10-16T14:33:47.338214+0000","last_clean":"2023-10-16T14:33:47.338214+0000","last_became_active":"2023-10-16T14:33:46.449233+0000","last_became_peered":"2023-10-16T14:33:46.449233+0000","last_unstale":"2023-10-16T14:33:47.338214+0000","last_undegraded":"2023-10-16T14:33:47.338214+0000","last_fullsized":"2023-10-16T14:33:47.338214+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:20.311567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340071+0000","last_change":"2023-10-16T14:33:46.449662+0000","last_active":"2023-10-16T14:33:47.340071+0000","last_peered":"2023-10-16T14:33:47.340071+0000","last_clean":"2023-10-16T14:33:47.340071+0000","last_became_active":"2023-10-16T14:33:46.449115+0000","last_became_peered":"2023-10-16T14:33:46.449115+0000","last_unstale":"2023-10-16T14:33:47.340071+0000","last_undegraded":"2023-10-16T14:33:47.340071+0000","last_fullsized":"2023-10-16T14:33:47.340071+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:04.527422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332268+0000","last_change":"2023-10-16T14:33:46.449722+0000","last_active":"2023-10-16T14:33:47.332268+0000","last_peered":"2023-10-16T14:33:47.332268+0000","last_clean":"2023-10-16T14:33:47.332268+0000","last_became_active":"2023-10-16T14:33:46.449396+0000","last_became_peered":"2023-10-16T14:33:46.449396+0000","last_unstale":"2023-10-16T14:33:47.332268+0000","last_undegraded":"2023-10-16T14:33:47.332268+0000","last_fullsized":"2023-10-16T14:33:47.332268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:45.538655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337848+0000","last_change":"2023-10-16T14:33:46.457390+0000","last_active":"2023-10-16T14:33:47.337848+0000","last_peered":"2023-10-16T14:33:47.337848+0000","last_clean":"2023-10-16T14:33:47.337848+0000","last_became_active":"2023-10-16T14:33:46.456467+0000","last_became_peered":"2023-10-16T14:33:46.456467+0000","last_unstale":"2023-10-16T14:33:47.337848+0000","last_undegraded":"2023-10-16T14:33:47.337848+0000","last_fullsized":"2023-10-16T14:33:47.337848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:02.604688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334216+0000","last_change":"2023-10-16T14:33:46.447933+0000","last_active":"2023-10-16T14:33:47.334216+0000","last_peered":"2023-10-16T14:33:47.334216+0000","last_clean":"2023-10-16T14:33:47.334216+0000","last_became_active":"2023-10-16T14:33:46.447672+0000","last_became_peered":"2023-10-16T14:33:46.447672+0000","last_unstale":"2023-10-16T14:33:47.334216+0000","last_undegraded":"2023-10-16T14:33:47.334216+0000","last_fullsized":"2023-10-16T14:33:47.334216+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:01.220894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"110'4","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339673+0000","last_change":"2023-10-16T14:33:45.311723+0000","last_active":"2023-10-16T14:33:47.339673+0000","last_peered":"2023-10-16T14:33:47.339673+0000","last_clean":"2023-10-16T14:33:47.339673+0000","last_became_active":"2023-10-16T14:33:45.310078+0000","last_became_peered":"2023-10-16T14:33:45.310078+0000","last_unstale":"2023-10-16T14:33:47.339673+0000","last_undegraded":"2023-10-16T14:33:47.339673+0000","last_fullsized":"2023-10-16T14:33:47.339673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:33.046320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339497+0000","last_change":"2023-10-16T14:33:46.454177+0000","last_active":"2023-10-16T14:33:47.339497+0000","last_peered":"2023-10-16T14:33:47.339497+0000","last_clean":"2023-10-16T14:33:47.339497+0000","last_became_active":"2023-10-16T14:33:46.453245+0000","last_became_peered":"2023-10-16T14:33:46.453245+0000","last_unstale":"2023-10-16T14:33:47.339497+0000","last_undegraded":"2023-10-16T14:33:47.339497+0000","last_fullsized":"2023-10-16T14:33:47.339497+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:04.527422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334516+0000","last_change":"2023-10-16T14:33:45.315352+0000","last_active":"2023-10-16T14:33:47.334516+0000","last_peered":"2023-10-16T14:33:47.334516+0000","last_clean":"2023-10-16T14:33:47.334516+0000","last_became_active":"2023-10-16T14:33:45.315039+0000","last_became_peered":"2023-10-16T14:33:45.315039+0000","last_unstale":"2023-10-16T14:33:47.334516+0000","last_undegraded":"2023-10-16T14:33:47.334516+0000","last_fullsized":"2023-10-16T14:33:47.334516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.038356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338581+0000","last_change":"2023-10-16T14:33:46.457095+0000","last_active":"2023-10-16T14:33:47.338581+0000","last_peered":"2023-10-16T14:33:47.338581+0000","last_clean":"2023-10-16T14:33:47.338581+0000","last_became_active":"2023-10-16T14:33:46.456797+0000","last_became_peered":"2023-10-16T14:33:46.456797+0000","last_unstale":"2023-10-16T14:33:47.338581+0000","last_undegraded":"2023-10-16T14:33:47.338581+0000","last_fullsized":"2023-10-16T14:33:47.338581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:17.006796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337733+0000","last_change":"2023-10-16T14:33:45.303207+0000","last_active":"2023-10-16T14:33:47.337733+0000","last_peered":"2023-10-16T14:33:47.337733+0000","last_clean":"2023-10-16T14:33:47.337733+0000","last_became_active":"2023-10-16T14:33:45.302786+0000","last_became_peered":"2023-10-16T14:33:45.302786+0000","last_unstale":"2023-10-16T14:33:47.337733+0000","last_undegraded":"2023-10-16T14:33:47.337733+0000","last_fullsized":"2023-10-16T14:33:47.337733+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:32.054815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332139+0000","last_change":"2023-10-16T14:33:46.343123+0000","last_active":"2023-10-16T14:33:47.332139+0000","last_peered":"2023-10-16T14:33:47.332139+0000","last_clean":"2023-10-16T14:33:47.332139+0000","last_became_active":"2023-10-16T14:33:46.342623+0000","last_became_peered":"2023-10-16T14:33:46.342623+0000","last_unstale":"2023-10-16T14:33:47.332139+0000","last_undegraded":"2023-10-16T14:33:47.332139+0000","last_fullsized":"2023-10-16T14:33:47.332139+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:58.001842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336850+0000","last_change":"2023-10-16T14:33:45.303709+0000","last_active":"2023-10-16T14:33:47.336850+0000","last_peered":"2023-10-16T14:33:47.336850+0000","last_clean":"2023-10-16T14:33:47.336850+0000","last_became_active":"2023-10-16T14:33:45.302615+0000","last_became_peered":"2023-10-16T14:33:45.302615+0000","last_unstale":"2023-10-16T14:33:47.336850+0000","last_undegraded":"2023-10-16T14:33:47.336850+0000","last_fullsized":"2023-10-16T14:33:47.336850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:09.549638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.340250+0000","last_change":"2023-10-16T14:33:46.449418+0000","last_active":"2023-10-16T14:33:47.340250+0000","last_peered":"2023-10-16T14:33:47.340250+0000","last_clean":"2023-10-16T14:33:47.340250+0000","last_became_active":"2023-10-16T14:33:46.448240+0000","last_became_peered":"2023-10-16T14:33:46.448240+0000","last_unstale":"2023-10-16T14:33:47.340250+0000","last_undegraded":"2023-10-16T14:33:47.340250+0000","last_fullsized":"2023-10-16T14:33:47.340250+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:36.729192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333578+0000","last_change":"2023-10-16T14:33:45.316657+0000","last_active":"2023-10-16T14:33:47.333578+0000","last_peered":"2023-10-16T14:33:47.333578+0000","last_clean":"2023-10-16T14:33:47.333578+0000","last_became_active":"2023-10-16T14:33:45.316276+0000","last_became_peered":"2023-10-16T14:33:45.316276+0000","last_unstale":"2023-10-16T14:33:47.333578+0000","last_undegraded":"2023-10-16T14:33:47.333578+0000","last_fullsized":"2023-10-16T14:33:47.333578+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:39.586510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333284+0000","last_change":"2023-10-16T14:33:46.335786+0000","last_active":"2023-10-16T14:33:47.333284+0000","last_peered":"2023-10-16T14:33:47.333284+0000","last_clean":"2023-10-16T14:33:47.333284+0000","last_became_active":"2023-10-16T14:33:46.334510+0000","last_became_peered":"2023-10-16T14:33:46.334510+0000","last_unstale":"2023-10-16T14:33:47.333284+0000","last_undegraded":"2023-10-16T14:33:47.333284+0000","last_fullsized":"2023-10-16T14:33:47.333284+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:38.272927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333554+0000","last_change":"2023-10-16T14:33:45.314410+0000","last_active":"2023-10-16T14:33:47.333554+0000","last_peered":"2023-10-16T14:33:47.333554+0000","last_clean":"2023-10-16T14:33:47.333554+0000","last_became_active":"2023-10-16T14:33:45.314103+0000","last_became_peered":"2023-10-16T14:33:45.314103+0000","last_unstale":"2023-10-16T14:33:47.333554+0000","last_undegraded":"2023-10-16T14:33:47.333554+0000","last_fullsized":"2023-10-16T14:33:47.333554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:02.327159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332140+0000","last_change":"2023-10-16T14:33:46.450482+0000","last_active":"2023-10-16T14:33:47.332140+0000","last_peered":"2023-10-16T14:33:47.332140+0000","last_clean":"2023-10-16T14:33:47.332140+0000","last_became_active":"2023-10-16T14:33:46.450106+0000","last_became_peered":"2023-10-16T14:33:46.450106+0000","last_unstale":"2023-10-16T14:33:47.332140+0000","last_undegraded":"2023-10-16T14:33:47.332140+0000","last_fullsized":"2023-10-16T14:33:47.332140+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:12.771061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337187+0000","last_change":"2023-10-16T14:33:45.312180+0000","last_active":"2023-10-16T14:33:47.337187+0000","last_peered":"2023-10-16T14:33:47.337187+0000","last_clean":"2023-10-16T14:33:47.337187+0000","last_became_active":"2023-10-16T14:33:45.311844+0000","last_became_peered":"2023-10-16T14:33:45.311844+0000","last_unstale":"2023-10-16T14:33:47.337187+0000","last_undegraded":"2023-10-16T14:33:47.337187+0000","last_fullsized":"2023-10-16T14:33:47.337187+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:13.245531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330554+0000","last_change":"2023-10-16T14:33:46.440970+0000","last_active":"2023-10-16T14:33:47.330554+0000","last_peered":"2023-10-16T14:33:47.330554+0000","last_clean":"2023-10-16T14:33:47.330554+0000","last_became_active":"2023-10-16T14:33:46.439972+0000","last_became_peered":"2023-10-16T14:33:46.439972+0000","last_unstale":"2023-10-16T14:33:47.330554+0000","last_undegraded":"2023-10-16T14:33:47.330554+0000","last_fullsized":"2023-10-16T14:33:47.330554+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:51.182380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338046+0000","last_change":"2023-10-16T14:33:45.310523+0000","last_active":"2023-10-16T14:33:47.338046+0000","last_peered":"2023-10-16T14:33:47.338046+0000","last_clean":"2023-10-16T14:33:47.338046+0000","last_became_active":"2023-10-16T14:33:45.309548+0000","last_became_peered":"2023-10-16T14:33:45.309548+0000","last_unstale":"2023-10-16T14:33:47.338046+0000","last_undegraded":"2023-10-16T14:33:47.338046+0000","last_fullsized":"2023-10-16T14:33:47.338046+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:03.126029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331222+0000","last_change":"2023-10-16T14:33:46.457742+0000","last_active":"2023-10-16T14:33:47.331222+0000","last_peered":"2023-10-16T14:33:47.331222+0000","last_clean":"2023-10-16T14:33:47.331222+0000","last_became_active":"2023-10-16T14:33:46.457380+0000","last_became_peered":"2023-10-16T14:33:46.457380+0000","last_unstale":"2023-10-16T14:33:47.331222+0000","last_undegraded":"2023-10-16T14:33:47.331222+0000","last_fullsized":"2023-10-16T14:33:47.331222+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:45.484436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339770+0000","last_change":"2023-10-16T14:33:45.312263+0000","last_active":"2023-10-16T14:33:47.339770+0000","last_peered":"2023-10-16T14:33:47.339770+0000","last_clean":"2023-10-16T14:33:47.339770+0000","last_became_active":"2023-10-16T14:33:45.311969+0000","last_became_peered":"2023-10-16T14:33:45.311969+0000","last_unstale":"2023-10-16T14:33:47.339770+0000","last_undegraded":"2023-10-16T14:33:47.339770+0000","last_fullsized":"2023-10-16T14:33:47.339770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:09.466762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339293+0000","last_change":"2023-10-16T14:33:46.445263+0000","last_active":"2023-10-16T14:33:47.339293+0000","last_peered":"2023-10-16T14:33:47.339293+0000","last_clean":"2023-10-16T14:33:47.339293+0000","last_became_active":"2023-10-16T14:33:46.445000+0000","last_became_peered":"2023-10-16T14:33:46.445000+0000","last_unstale":"2023-10-16T14:33:47.339293+0000","last_undegraded":"2023-10-16T14:33:47.339293+0000","last_fullsized":"2023-10-16T14:33:47.339293+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:27.890965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338614+0000","last_change":"2023-10-16T14:33:45.317182+0000","last_active":"2023-10-16T14:33:47.338614+0000","last_peered":"2023-10-16T14:33:47.338614+0000","last_clean":"2023-10-16T14:33:47.338614+0000","last_became_active":"2023-10-16T14:33:45.316670+0000","last_became_peered":"2023-10-16T14:33:45.316670+0000","last_unstale":"2023-10-16T14:33:47.338614+0000","last_undegraded":"2023-10-16T14:33:47.338614+0000","last_fullsized":"2023-10-16T14:33:47.338614+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:56.029775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336666+0000","last_change":"2023-10-16T14:33:46.458454+0000","last_active":"2023-10-16T14:33:47.336666+0000","last_peered":"2023-10-16T14:33:47.336666+0000","last_clean":"2023-10-16T14:33:47.336666+0000","last_became_active":"2023-10-16T14:33:46.455052+0000","last_became_peered":"2023-10-16T14:33:46.455052+0000","last_unstale":"2023-10-16T14:33:47.336666+0000","last_undegraded":"2023-10-16T14:33:47.336666+0000","last_fullsized":"2023-10-16T14:33:47.336666+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:34.051326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"110'3","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338418+0000","last_change":"2023-10-16T14:33:45.318096+0000","last_active":"2023-10-16T14:33:47.338418+0000","last_peered":"2023-10-16T14:33:47.338418+0000","last_clean":"2023-10-16T14:33:47.338418+0000","last_became_active":"2023-10-16T14:33:45.316471+0000","last_became_peered":"2023-10-16T14:33:45.316471+0000","last_unstale":"2023-10-16T14:33:47.338418+0000","last_undegraded":"2023-10-16T14:33:47.338418+0000","last_fullsized":"2023-10-16T14:33:47.338418+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:47.477165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333968+0000","last_change":"2023-10-16T14:33:46.458818+0000","last_active":"2023-10-16T14:33:47.333968+0000","last_peered":"2023-10-16T14:33:47.333968+0000","last_clean":"2023-10-16T14:33:47.333968+0000","last_became_active":"2023-10-16T14:33:46.458491+0000","last_became_peered":"2023-10-16T14:33:46.458491+0000","last_unstale":"2023-10-16T14:33:47.333968+0000","last_undegraded":"2023-10-16T14:33:47.333968+0000","last_fullsized":"2023-10-16T14:33:47.333968+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:18.391780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"110'6","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337735+0000","last_change":"2023-10-16T14:33:45.310689+0000","last_active":"2023-10-16T14:33:47.337735+0000","last_peered":"2023-10-16T14:33:47.337735+0000","last_clean":"2023-10-16T14:33:47.337735+0000","last_became_active":"2023-10-16T14:33:45.310390+0000","last_became_peered":"2023-10-16T14:33:45.310390+0000","last_unstale":"2023-10-16T14:33:47.337735+0000","last_undegraded":"2023-10-16T14:33:47.337735+0000","last_fullsized":"2023-10-16T14:33:47.337735+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:04.451086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339106+0000","last_change":"2023-10-16T14:33:46.447787+0000","last_active":"2023-10-16T14:33:47.339106+0000","last_peered":"2023-10-16T14:33:47.339106+0000","last_clean":"2023-10-16T14:33:47.339106+0000","last_became_active":"2023-10-16T14:33:46.445073+0000","last_became_peered":"2023-10-16T14:33:46.445073+0000","last_unstale":"2023-10-16T14:33:47.339106+0000","last_undegraded":"2023-10-16T14:33:47.339106+0000","last_fullsized":"2023-10-16T14:33:47.339106+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:54.185890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338453+0000","last_change":"2023-10-16T14:33:45.312942+0000","last_active":"2023-10-16T14:33:47.338453+0000","last_peered":"2023-10-16T14:33:47.338453+0000","last_clean":"2023-10-16T14:33:47.338453+0000","last_became_active":"2023-10-16T14:33:45.312603+0000","last_became_peered":"2023-10-16T14:33:45.312603+0000","last_unstale":"2023-10-16T14:33:47.338453+0000","last_undegraded":"2023-10-16T14:33:47.338453+0000","last_fullsized":"2023-10-16T14:33:47.338453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.583549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331848+0000","last_change":"2023-10-16T14:33:46.450643+0000","last_active":"2023-10-16T14:33:47.331848+0000","last_peered":"2023-10-16T14:33:47.331848+0000","last_clean":"2023-10-16T14:33:47.331848+0000","last_became_active":"2023-10-16T14:33:46.450338+0000","last_became_peered":"2023-10-16T14:33:46.450338+0000","last_unstale":"2023-10-16T14:33:47.331848+0000","last_undegraded":"2023-10-16T14:33:47.331848+0000","last_fullsized":"2023-10-16T14:33:47.331848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:29.997780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330108+0000","last_change":"2023-10-16T14:33:45.312750+0000","last_active":"2023-10-16T14:33:47.330108+0000","last_peered":"2023-10-16T14:33:47.330108+0000","last_clean":"2023-10-16T14:33:47.330108+0000","last_became_active":"2023-10-16T14:33:45.312429+0000","last_became_peered":"2023-10-16T14:33:45.312429+0000","last_unstale":"2023-10-16T14:33:47.330108+0000","last_undegraded":"2023-10-16T14:33:47.330108+0000","last_fullsized":"2023-10-16T14:33:47.330108+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:06.804628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337819+0000","last_change":"2023-10-16T14:33:46.445420+0000","last_active":"2023-10-16T14:33:47.337819+0000","last_peered":"2023-10-16T14:33:47.337819+0000","last_clean":"2023-10-16T14:33:47.337819+0000","last_became_active":"2023-10-16T14:33:46.445062+0000","last_became_peered":"2023-10-16T14:33:46.445062+0000","last_unstale":"2023-10-16T14:33:47.337819+0000","last_undegraded":"2023-10-16T14:33:47.337819+0000","last_fullsized":"2023-10-16T14:33:47.337819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:17.006796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337002+0000","last_change":"2023-10-16T14:33:45.316993+0000","last_active":"2023-10-16T14:33:47.337002+0000","last_peered":"2023-10-16T14:33:47.337002+0000","last_clean":"2023-10-16T14:33:47.337002+0000","last_became_active":"2023-10-16T14:33:45.315095+0000","last_became_peered":"2023-10-16T14:33:45.315095+0000","last_unstale":"2023-10-16T14:33:47.337002+0000","last_undegraded":"2023-10-16T14:33:47.337002+0000","last_fullsized":"2023-10-16T14:33:47.337002+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:31:58.892475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336756+0000","last_change":"2023-10-16T14:33:46.453305+0000","last_active":"2023-10-16T14:33:47.336756+0000","last_peered":"2023-10-16T14:33:47.336756+0000","last_clean":"2023-10-16T14:33:47.336756+0000","last_became_active":"2023-10-16T14:33:46.453010+0000","last_became_peered":"2023-10-16T14:33:46.453010+0000","last_unstale":"2023-10-16T14:33:47.336756+0000","last_undegraded":"2023-10-16T14:33:47.336756+0000","last_fullsized":"2023-10-16T14:33:47.336756+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:28.072982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"110'8","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338655+0000","last_change":"2023-10-16T14:33:45.315138+0000","last_active":"2023-10-16T14:33:47.338655+0000","last_peered":"2023-10-16T14:33:47.338655+0000","last_clean":"2023-10-16T14:33:47.338655+0000","last_became_active":"2023-10-16T14:33:45.314381+0000","last_became_peered":"2023-10-16T14:33:45.314381+0000","last_unstale":"2023-10-16T14:33:47.338655+0000","last_undegraded":"2023-10-16T14:33:47.338655+0000","last_fullsized":"2023-10-16T14:33:47.338655+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:25:43.161213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332149+0000","last_change":"2023-10-16T14:33:46.440969+0000","last_active":"2023-10-16T14:33:47.332149+0000","last_peered":"2023-10-16T14:33:47.332149+0000","last_clean":"2023-10-16T14:33:47.332149+0000","last_became_active":"2023-10-16T14:33:46.440649+0000","last_became_peered":"2023-10-16T14:33:46.440649+0000","last_unstale":"2023-10-16T14:33:47.332149+0000","last_undegraded":"2023-10-16T14:33:47.332149+0000","last_fullsized":"2023-10-16T14:33:47.332149+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:48.245341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339897+0000","last_change":"2023-10-16T14:33:45.317908+0000","last_active":"2023-10-16T14:33:47.339897+0000","last_peered":"2023-10-16T14:33:47.339897+0000","last_clean":"2023-10-16T14:33:47.339897+0000","last_became_active":"2023-10-16T14:33:45.317287+0000","last_became_peered":"2023-10-16T14:33:45.317287+0000","last_unstale":"2023-10-16T14:33:47.339897+0000","last_undegraded":"2023-10-16T14:33:47.339897+0000","last_fullsized":"2023-10-16T14:33:47.339897+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:53.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339645+0000","last_change":"2023-10-16T14:33:46.442081+0000","last_active":"2023-10-16T14:33:47.339645+0000","last_peered":"2023-10-16T14:33:47.339645+0000","last_clean":"2023-10-16T14:33:47.339645+0000","last_became_active":"2023-10-16T14:33:46.441343+0000","last_became_peered":"2023-10-16T14:33:46.441343+0000","last_unstale":"2023-10-16T14:33:47.339645+0000","last_undegraded":"2023-10-16T14:33:47.339645+0000","last_fullsized":"2023-10-16T14:33:47.339645+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:51.326358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337357+0000","last_change":"2023-10-16T14:33:45.315678+0000","last_active":"2023-10-16T14:33:47.337357+0000","last_peered":"2023-10-16T14:33:47.337357+0000","last_clean":"2023-10-16T14:33:47.337357+0000","last_became_active":"2023-10-16T14:33:45.313889+0000","last_became_peered":"2023-10-16T14:33:45.313889+0000","last_unstale":"2023-10-16T14:33:47.337357+0000","last_undegraded":"2023-10-16T14:33:47.337357+0000","last_fullsized":"2023-10-16T14:33:47.337357+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:28.052772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334402+0000","last_change":"2023-10-16T14:33:46.437520+0000","last_active":"2023-10-16T14:33:47.334402+0000","last_peered":"2023-10-16T14:33:47.334402+0000","last_clean":"2023-10-16T14:33:47.334402+0000","last_became_active":"2023-10-16T14:33:46.436979+0000","last_became_peered":"2023-10-16T14:33:46.436979+0000","last_unstale":"2023-10-16T14:33:47.334402+0000","last_undegraded":"2023-10-16T14:33:47.334402+0000","last_fullsized":"2023-10-16T14:33:47.334402+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:28.405987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336945+0000","last_change":"2023-10-16T14:33:45.311117+0000","last_active":"2023-10-16T14:33:47.336945+0000","last_peered":"2023-10-16T14:33:47.336945+0000","last_clean":"2023-10-16T14:33:47.336945+0000","last_became_active":"2023-10-16T14:33:45.310621+0000","last_became_peered":"2023-10-16T14:33:45.310621+0000","last_unstale":"2023-10-16T14:33:47.336945+0000","last_undegraded":"2023-10-16T14:33:47.336945+0000","last_fullsized":"2023-10-16T14:33:47.336945+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:00.999541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336500+0000","last_change":"2023-10-16T14:33:46.437946+0000","last_active":"2023-10-16T14:33:47.336500+0000","last_peered":"2023-10-16T14:33:47.336500+0000","last_clean":"2023-10-16T14:33:47.336500+0000","last_became_active":"2023-10-16T14:33:46.437355+0000","last_became_peered":"2023-10-16T14:33:46.437355+0000","last_unstale":"2023-10-16T14:33:47.336500+0000","last_undegraded":"2023-10-16T14:33:47.336500+0000","last_fullsized":"2023-10-16T14:33:47.336500+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:02.604688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.330084+0000","last_change":"2023-10-16T14:33:45.299128+0000","last_active":"2023-10-16T14:33:47.330084+0000","last_peered":"2023-10-16T14:33:47.330084+0000","last_clean":"2023-10-16T14:33:47.330084+0000","last_became_active":"2023-10-16T14:33:45.298810+0000","last_became_peered":"2023-10-16T14:33:45.298810+0000","last_unstale":"2023-10-16T14:33:47.330084+0000","last_undegraded":"2023-10-16T14:33:47.330084+0000","last_fullsized":"2023-10-16T14:33:47.330084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:25.967371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.337476+0000","last_change":"2023-10-16T14:33:46.440724+0000","last_active":"2023-10-16T14:33:47.337476+0000","last_peered":"2023-10-16T14:33:47.337476+0000","last_clean":"2023-10-16T14:33:47.337476+0000","last_became_active":"2023-10-16T14:33:46.440436+0000","last_became_peered":"2023-10-16T14:33:46.440436+0000","last_unstale":"2023-10-16T14:33:47.337476+0000","last_undegraded":"2023-10-16T14:33:47.337476+0000","last_fullsized":"2023-10-16T14:33:47.337476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:54.721248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338195+0000","last_change":"2023-10-16T14:33:45.300358+0000","last_active":"2023-10-16T14:33:47.338195+0000","last_peered":"2023-10-16T14:33:47.338195+0000","last_clean":"2023-10-16T14:33:47.338195+0000","last_became_active":"2023-10-16T14:33:45.298693+0000","last_became_peered":"2023-10-16T14:33:45.298693+0000","last_unstale":"2023-10-16T14:33:47.338195+0000","last_undegraded":"2023-10-16T14:33:47.338195+0000","last_fullsized":"2023-10-16T14:33:47.338195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:09.466762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335261+0000","last_change":"2023-10-16T14:33:46.445380+0000","last_active":"2023-10-16T14:33:47.335261+0000","last_peered":"2023-10-16T14:33:47.335261+0000","last_clean":"2023-10-16T14:33:47.335261+0000","last_became_active":"2023-10-16T14:33:46.445061+0000","last_became_peered":"2023-10-16T14:33:46.445061+0000","last_unstale":"2023-10-16T14:33:47.335261+0000","last_undegraded":"2023-10-16T14:33:47.335261+0000","last_fullsized":"2023-10-16T14:33:47.335261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:04.131034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.335332+0000","last_change":"2023-10-16T14:33:45.305623+0000","last_active":"2023-10-16T14:33:47.335332+0000","last_peered":"2023-10-16T14:33:47.335332+0000","last_clean":"2023-10-16T14:33:47.335332+0000","last_became_active":"2023-10-16T14:33:45.305312+0000","last_became_peered":"2023-10-16T14:33:45.305312+0000","last_unstale":"2023-10-16T14:33:47.335332+0000","last_undegraded":"2023-10-16T14:33:47.335332+0000","last_fullsized":"2023-10-16T14:33:47.335332+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:32.054815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.336993+0000","last_change":"2023-10-16T14:33:46.456145+0000","last_active":"2023-10-16T14:33:47.336993+0000","last_peered":"2023-10-16T14:33:47.336993+0000","last_clean":"2023-10-16T14:33:47.336993+0000","last_became_active":"2023-10-16T14:33:46.455842+0000","last_became_peered":"2023-10-16T14:33:46.455842+0000","last_unstale":"2023-10-16T14:33:47.336993+0000","last_undegraded":"2023-10-16T14:33:47.336993+0000","last_fullsized":"2023-10-16T14:33:47.336993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:03.520289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.329069+0000","last_change":"2023-10-16T14:33:45.300057+0000","last_active":"2023-10-16T14:33:47.329069+0000","last_peered":"2023-10-16T14:33:47.329069+0000","last_clean":"2023-10-16T14:33:47.329069+0000","last_became_active":"2023-10-16T14:33:45.299786+0000","last_became_peered":"2023-10-16T14:33:45.299786+0000","last_unstale":"2023-10-16T14:33:47.329069+0000","last_undegraded":"2023-10-16T14:33:47.329069+0000","last_fullsized":"2023-10-16T14:33:47.329069+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:05.012883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334194+0000","last_change":"2023-10-16T14:33:46.441142+0000","last_active":"2023-10-16T14:33:47.334194+0000","last_peered":"2023-10-16T14:33:47.334194+0000","last_clean":"2023-10-16T14:33:47.334194+0000","last_became_active":"2023-10-16T14:33:46.440529+0000","last_became_peered":"2023-10-16T14:33:46.440529+0000","last_unstale":"2023-10-16T14:33:47.334194+0000","last_undegraded":"2023-10-16T14:33:47.334194+0000","last_fullsized":"2023-10-16T14:33:47.334194+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:47.739038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"110'2","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339858+0000","last_change":"2023-10-16T14:33:45.312168+0000","last_active":"2023-10-16T14:33:47.339858+0000","last_peered":"2023-10-16T14:33:47.339858+0000","last_clean":"2023-10-16T14:33:47.339858+0000","last_became_active":"2023-10-16T14:33:45.311844+0000","last_became_peered":"2023-10-16T14:33:45.311844+0000","last_unstale":"2023-10-16T14:33:47.339858+0000","last_undegraded":"2023-10-16T14:33:47.339858+0000","last_fullsized":"2023-10-16T14:33:47.339858+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:47.477165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332833+0000","last_change":"2023-10-16T14:33:46.448662+0000","last_active":"2023-10-16T14:33:47.332833+0000","last_peered":"2023-10-16T14:33:47.332833+0000","last_clean":"2023-10-16T14:33:47.332833+0000","last_became_active":"2023-10-16T14:33:46.447853+0000","last_became_peered":"2023-10-16T14:33:46.447853+0000","last_unstale":"2023-10-16T14:33:47.332833+0000","last_undegraded":"2023-10-16T14:33:47.332833+0000","last_fullsized":"2023-10-16T14:33:47.332833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:45.455269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339560+0000","last_change":"2023-10-16T14:33:45.313953+0000","last_active":"2023-10-16T14:33:47.339560+0000","last_peered":"2023-10-16T14:33:47.339560+0000","last_clean":"2023-10-16T14:33:47.339560+0000","last_became_active":"2023-10-16T14:33:45.313165+0000","last_became_peered":"2023-10-16T14:33:45.313165+0000","last_unstale":"2023-10-16T14:33:47.339560+0000","last_undegraded":"2023-10-16T14:33:47.339560+0000","last_fullsized":"2023-10-16T14:33:47.339560+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:01.599682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338945+0000","last_change":"2023-10-16T14:33:46.450870+0000","last_active":"2023-10-16T14:33:47.338945+0000","last_peered":"2023-10-16T14:33:47.338945+0000","last_clean":"2023-10-16T14:33:47.338945+0000","last_became_active":"2023-10-16T14:33:46.450184+0000","last_became_peered":"2023-10-16T14:33:46.450184+0000","last_unstale":"2023-10-16T14:33:47.338945+0000","last_undegraded":"2023-10-16T14:33:47.338945+0000","last_fullsized":"2023-10-16T14:33:47.338945+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:00.990582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338824+0000","last_change":"2023-10-16T14:33:45.312478+0000","last_active":"2023-10-16T14:33:47.338824+0000","last_peered":"2023-10-16T14:33:47.338824+0000","last_clean":"2023-10-16T14:33:47.338824+0000","last_became_active":"2023-10-16T14:33:45.312100+0000","last_became_peered":"2023-10-16T14:33:45.312100+0000","last_unstale":"2023-10-16T14:33:47.338824+0000","last_undegraded":"2023-10-16T14:33:47.338824+0000","last_fullsized":"2023-10-16T14:33:47.338824+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:00.834604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.334739+0000","last_change":"2023-10-16T14:33:46.446921+0000","last_active":"2023-10-16T14:33:47.334739+0000","last_peered":"2023-10-16T14:33:47.334739+0000","last_clean":"2023-10-16T14:33:47.334739+0000","last_became_active":"2023-10-16T14:33:46.446634+0000","last_became_peered":"2023-10-16T14:33:46.446634+0000","last_unstale":"2023-10-16T14:33:47.334739+0000","last_undegraded":"2023-10-16T14:33:47.334739+0000","last_fullsized":"2023-10-16T14:33:47.334739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:08.967220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.339061+0000","last_change":"2023-10-16T14:33:45.308907+0000","last_active":"2023-10-16T14:33:47.339061+0000","last_peered":"2023-10-16T14:33:47.339061+0000","last_clean":"2023-10-16T14:33:47.339061+0000","last_became_active":"2023-10-16T14:33:45.307907+0000","last_became_peered":"2023-10-16T14:33:45.307907+0000","last_unstale":"2023-10-16T14:33:47.339061+0000","last_undegraded":"2023-10-16T14:33:47.339061+0000","last_fullsized":"2023-10-16T14:33:47.339061+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.583549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338417+0000","last_change":"2023-10-16T14:33:46.443511+0000","last_active":"2023-10-16T14:33:47.338417+0000","last_peered":"2023-10-16T14:33:47.338417+0000","last_clean":"2023-10-16T14:33:47.338417+0000","last_became_active":"2023-10-16T14:33:46.443222+0000","last_became_peered":"2023-10-16T14:33:46.443222+0000","last_unstale":"2023-10-16T14:33:47.338417+0000","last_undegraded":"2023-10-16T14:33:47.338417+0000","last_fullsized":"2023-10-16T14:33:47.338417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:55.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338804+0000","last_change":"2023-10-16T14:33:45.311992+0000","last_active":"2023-10-16T14:33:47.338804+0000","last_peered":"2023-10-16T14:33:47.338804+0000","last_clean":"2023-10-16T14:33:47.338804+0000","last_became_active":"2023-10-16T14:33:45.311699+0000","last_became_peered":"2023-10-16T14:33:45.311699+0000","last_unstale":"2023-10-16T14:33:47.338804+0000","last_undegraded":"2023-10-16T14:33:47.338804+0000","last_fullsized":"2023-10-16T14:33:47.338804+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:12.344471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338130+0000","last_change":"2023-10-16T14:33:46.452059+0000","last_active":"2023-10-16T14:33:47.338130+0000","last_peered":"2023-10-16T14:33:47.338130+0000","last_clean":"2023-10-16T14:33:47.338130+0000","last_became_active":"2023-10-16T14:33:46.451642+0000","last_became_peered":"2023-10-16T14:33:46.451642+0000","last_unstale":"2023-10-16T14:33:47.338130+0000","last_undegraded":"2023-10-16T14:33:47.338130+0000","last_fullsized":"2023-10-16T14:33:47.338130+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:57.034780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333658+0000","last_change":"2023-10-16T14:33:45.314352+0000","last_active":"2023-10-16T14:33:47.333658+0000","last_peered":"2023-10-16T14:33:47.333658+0000","last_clean":"2023-10-16T14:33:47.333658+0000","last_became_active":"2023-10-16T14:33:45.314053+0000","last_became_peered":"2023-10-16T14:33:45.314053+0000","last_unstale":"2023-10-16T14:33:47.333658+0000","last_undegraded":"2023-10-16T14:33:47.333658+0000","last_fullsized":"2023-10-16T14:33:47.333658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:21.581893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.331711+0000","last_change":"2023-10-16T14:33:46.442461+0000","last_active":"2023-10-16T14:33:47.331711+0000","last_peered":"2023-10-16T14:33:47.331711+0000","last_clean":"2023-10-16T14:33:47.331711+0000","last_became_active":"2023-10-16T14:33:46.442096+0000","last_became_peered":"2023-10-16T14:33:46.442096+0000","last_unstale":"2023-10-16T14:33:47.331711+0000","last_undegraded":"2023-10-16T14:33:47.331711+0000","last_fullsized":"2023-10-16T14:33:47.331711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:45.484436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338253+0000","last_change":"2023-10-16T14:33:45.309699+0000","last_active":"2023-10-16T14:33:47.338253+0000","last_peered":"2023-10-16T14:33:47.338253+0000","last_clean":"2023-10-16T14:33:47.338253+0000","last_became_active":"2023-10-16T14:33:45.307655+0000","last_became_peered":"2023-10-16T14:33:45.307655+0000","last_unstale":"2023-10-16T14:33:47.338253+0000","last_undegraded":"2023-10-16T14:33:47.338253+0000","last_fullsized":"2023-10-16T14:33:47.338253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:00.999541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338111+0000","last_change":"2023-10-16T14:33:46.441423+0000","last_active":"2023-10-16T14:33:47.338111+0000","last_peered":"2023-10-16T14:33:47.338111+0000","last_clean":"2023-10-16T14:33:47.338111+0000","last_became_active":"2023-10-16T14:33:46.441122+0000","last_became_peered":"2023-10-16T14:33:46.441122+0000","last_unstale":"2023-10-16T14:33:47.338111+0000","last_undegraded":"2023-10-16T14:33:47.338111+0000","last_fullsized":"2023-10-16T14:33:47.338111+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:50.007020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.333006+0000","last_change":"2023-10-16T14:33:45.316120+0000","last_active":"2023-10-16T14:33:47.333006+0000","last_peered":"2023-10-16T14:33:47.333006+0000","last_clean":"2023-10-16T14:33:47.333006+0000","last_became_active":"2023-10-16T14:33:45.315806+0000","last_became_peered":"2023-10-16T14:33:45.315806+0000","last_unstale":"2023-10-16T14:33:47.333006+0000","last_undegraded":"2023-10-16T14:33:47.333006+0000","last_fullsized":"2023-10-16T14:33:47.333006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:33.532860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.338993+0000","last_change":"2023-10-16T14:33:46.446951+0000","last_active":"2023-10-16T14:33:47.338993+0000","last_peered":"2023-10-16T14:33:47.338993+0000","last_clean":"2023-10-16T14:33:47.338993+0000","last_became_active":"2023-10-16T14:33:46.446656+0000","last_became_peered":"2023-10-16T14:33:46.446656+0000","last_unstale":"2023-10-16T14:33:47.338993+0000","last_undegraded":"2023-10-16T14:33:47.338993+0000","last_fullsized":"2023-10-16T14:33:47.338993+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:45.272708+0000","last_clean_scrub_stamp":"2023-10-16T14:33:45.272708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:49.706863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"110'1","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T14:33:47.332396+0000","last_change":"2023-10-16T14:33:45.312611+0000","last_active":"2023-10-16T14:33:47.332396+0000","last_peered":"2023-10-16T14:33:47.332396+0000","last_clean":"2023-10-16T14:33:47.332396+0000","last_became_active":"2023-10-16T14:33:45.312003+0000","last_became_peered":"2023-10-16T14:33:45.312003+0000","last_unstale":"2023-10-16T14:33:47.332396+0000","last_undegraded":"2023-10-16T14:33:47.332396+0000","last_fullsized":"2023-10-16T14:33:47.332396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:33:44.267703+0000","last_clean_scrub_stamp":"2023-10-16T14:33:44.267703+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:06.804628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"99'186","reported_seq":508,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T14:33:46.332592+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:33:46.332592+0000","last_peered":"2023-10-16T14:33:46.332592+0000","last_clean":"2023-10-16T14:33:46.332592+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:33:46.332592+0000","last_undegraded":"2023-10-16T14:33:46.332592+0000","last_fullsized":"2023-10-16T14:33:46.332592+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575057,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41404,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330436,"statfs":{"total":96636764160,"available":96594366464,"internally_reserved":0,"allocated":1949696,"data_stored":1240519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0640000000000001,"15min":1.147},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.42799999999999999},"max":{"1min":1.784,"5min":1.833,"15min":1.9159999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.133,"15min":1.157},"min":{"1min":0.77800000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.522,"5min":1.6220000000000001,"15min":1.673},"last":1.5720000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1599999999999999,"15min":1.1799999999999999},"min":{"1min":0.71599999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3859999999999999,"5min":1.7030000000000001,"15min":1.8320000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.22,"15min":1.2390000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":0.88500000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.298,"15min":1.296},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63500000000000001},"max":{"1min":1.512,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.23,"15min":1.234},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.24}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.298,"15min":1.2989999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.359},{"interface":"front","average":{"1min":1.294,"5min":1.3500000000000001,"15min":1.379},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2509999999999999},"last":1.399}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3580000000000001,"15min":1.3600000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.835,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.484},{"interface":"front","average":{"1min":1.236,"5min":1.25,"15min":1.258},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.9530000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.2949999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3939999999999999,"15min":1.407},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.637},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4750000000000001,"15min":1.4870000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.6719999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.2850000000000001,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.446},{"interface":"front","average":{"1min":1.488,"5min":1.504,"15min":1.512},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":2.359,"5min":2.359,"15min":2.427},"last":1.599}]}]},{"osd":6,"up_from":13,"seq":55834575056,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1363968,"data_stored":650775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.1519999999999999,"15min":1.169},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.6990000000000001,"5min":1.7569999999999999,"15min":2.2160000000000002},"last":7.7210000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.82599999999999996,"5min":0.70599999999999996,"15min":0.63300000000000001},"max":{"1min":1.577,"5min":2.2709999999999999,"15min":3.863},"last":6.0629999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.2509999999999999},"min":{"1min":0.92400000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.581,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":7.6459999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.3,"15min":1.29},"min":{"1min":0.81799999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7849999999999999,"5min":2.0449999999999999,"15min":3.7879999999999998},"last":10.241}]},{"osd":2,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.147,"15min":1.1990000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.59899999999999998},"max":{"1min":1.3400000000000001,"5min":1.708,"15min":3.5800000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.232,"5min":1.3180000000000001,"15min":1.3500000000000001},"min":{"1min":0.80200000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.52,"5min":1.8799999999999999,"15min":2.1429999999999998},"last":5.694}]},{"osd":3,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3300000000000001,"15min":1.353},"min":{"1min":0.85599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.8480000000000001,"5min":1.96,"15min":4.9790000000000001},"last":5.9139999999999997},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.286,"15min":1.3400000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":1.6080000000000001,"5min":1.913,"15min":5.4160000000000004},"last":2.2349999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.296,"15min":1.3},"min":{"1min":0.748,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8009999999999999,"5min":2.1070000000000002,"15min":5},"last":13.826000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3640000000000001,"15min":1.377},"min":{"1min":0.94999999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.897,"5min":2.1789999999999998,"15min":5.4640000000000004},"last":5.9790000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.2949999999999999,"15min":1.3160000000000001},"min":{"1min":0.89600000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.76,"5min":1.859,"15min":3.9279999999999999},"last":3.1419999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.482,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":3.6749999999999998},"last":5.8700000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.5600000000000001},"min":{"1min":1.105,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":4.6859999999999999},"last":6.0199999999999996},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.508,"15min":1.542},"min":{"1min":1.0840000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9139999999999999,"5min":2.0459999999999998,"15min":4.883},"last":2.1779999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":1368064,"data_stored":651715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.099,"15min":1.1160000000000001},"min":{"1min":0.73599999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5760000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1990000000000001,"15min":1.1819999999999999},"min":{"1min":0.63300000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.7729999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.0880000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.444,"5min":1.726,"15min":1.726},"last":1.032},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.2430000000000001},"min":{"1min":0.81499999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5249999999999999,"5min":1.7290000000000001,"15min":1.7809999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.244,"15min":1.2390000000000001},"min":{"1min":0.71599999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5720000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3200000000000001,"15min":1.353},"min":{"1min":0.85799999999999998,"5min":0.625,"15min":0.625},"max":{"1min":2.0049999999999999,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.3089999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.76900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.351,"15min":1.3779999999999999},"min":{"1min":0.85699999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.6819999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.3859999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3049999999999999,"15min":1.3169999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.54700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5049999999999999,"15min":1.5029999999999999},"min":{"1min":1.073,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.853,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":1.5489999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3520000000000001,"15min":1.3819999999999999},"min":{"1min":0.97299999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.218},{"interface":"front","average":{"1min":1.385,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":0.77700000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.587}]},{"osd":7,"last update":"Mon Oct 16 14:32:52 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.5489999999999999},"min":{"1min":0.93400000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.956,"5min":2.641,"15min":2.641},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.579,"15min":1.603},"min":{"1min":1.246,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.1909999999999998,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.6180000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575057,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46392,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325448,"statfs":{"total":96636764160,"available":96589258752,"internally_reserved":0,"allocated":1351680,"data_stored":650081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1559999999999999,"15min":1.1479999999999999},"min":{"1min":0.58999999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7689999999999999,"5min":1.881,"15min":2.0339999999999998},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.1020000000000001,"15min":1.1639999999999999},"min":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4830000000000001,"5min":1.7589999999999999,"15min":1.9199999999999999},"last":1.256}]},{"osd":1,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1479999999999999,"15min":1.177},"min":{"1min":0.58399999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.78},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.1910000000000001,"15min":1.2330000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.716,"5min":1.782,"15min":2.3690000000000002},"last":1.329}]},{"osd":2,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3700000000000001,"15min":1.399},"min":{"1min":0.68100000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.54},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.3400000000000001,"15min":1.365},"min":{"1min":0.69399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7769999999999999,"5min":1.8799999999999999,"15min":1.8839999999999999},"last":1.5629999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.5920000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.268,"15min":1.304},"min":{"1min":0.53900000000000003,"5min":0.437,"15min":0.437},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.595}]},{"osd":5,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.306,"15min":1.288},"min":{"1min":0.78800000000000003,"5min":0.61499999999999999,"15min":0.48699999999999999},"max":{"1min":1.639,"5min":1.696,"15min":1.8220000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.3460000000000001,"15min":1.349},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.806,"5min":1.806,"15min":1.964},"last":1.472}]},{"osd":6,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.323,"15min":1.2909999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4690000000000001,"15min":1.4610000000000001},"min":{"1min":0.74199999999999999,"5min":0.71999999999999997,"15min":0.44700000000000001},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.2890000000000001},"last":1.6339999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:32:58 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.4730000000000001,"15min":1.4690000000000001},"min":{"1min":0.76100000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9259999999999999},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.466,"15min":1.458},"min":{"1min":0.66700000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8979999999999999,"5min":2.0579999999999998,"15min":2.29},"last":1.704}]}]},{"osd":3,"up_from":13,"seq":55834575056,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40848,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330992,"statfs":{"total":96636764160,"available":96594935808,"internally_reserved":0,"allocated":1376256,"data_stored":657993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.50800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0609999999999999,"5min":1.4419999999999999,"15min":1.6639999999999999},"last":1.696},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":1.0169999999999999},"min":{"1min":0.5,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.3320000000000001,"5min":1.913,"15min":1.982},"last":2.1819999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90000000000000002,"15min":0.94099999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.32500000000000001},"max":{"1min":1.337,"5min":1.7390000000000001,"15min":2.149},"last":2.7850000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.089,"15min":1.139},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.415,"5min":1.6499999999999999,"15min":2.1600000000000001},"last":3.024}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2050000000000001,"15min":1.3180000000000001},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.44800000000000001},"max":{"1min":1.6830000000000001,"5min":1.895,"15min":2.1259999999999999},"last":3.3420000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.331,"15min":1.4259999999999999},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.5},"max":{"1min":1.8029999999999999,"5min":1.8660000000000001,"15min":2.6299999999999999},"last":2.5609999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1910000000000001,"15min":1.2569999999999999},"min":{"1min":0.77100000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.768,"5min":1.768,"15min":1.9239999999999999},"last":2.02},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.298,"15min":1.3300000000000001},"min":{"1min":0.93999999999999995,"5min":0.629,"15min":0.629},"max":{"1min":1.7689999999999999,"5min":1.909,"15min":1.9259999999999999},"last":2.9430000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.4550000000000001,"15min":1.516},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7170000000000001,"5min":2.0070000000000001,"15min":2.23},"last":3.206},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3899999999999999,"15min":1.468},"min":{"1min":0.90700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6220000000000001,"5min":1.95,"15min":2.3570000000000002},"last":3.3809999999999998}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.361,"15min":1.421},"min":{"1min":0.95999999999999996,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.837,"5min":2.181,"15min":2.181},"last":2.2970000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4419999999999999,"15min":1.4910000000000001},"min":{"1min":0.82599999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.6799999999999999,"5min":1.9510000000000001,"15min":2.347},"last":3.0910000000000002}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.4870000000000001,"15min":1.581},"min":{"1min":1.0069999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7310000000000001,"5min":2.1589999999999998,"15min":2.395},"last":5.5119999999999996},{"interface":"front","average":{"1min":1.347,"5min":1.472,"15min":1.5509999999999999},"min":{"1min":0.749,"5min":0.629,"15min":0.629},"max":{"1min":1.8160000000000001,"5min":2.1000000000000001,"15min":2.1739999999999999},"last":2.5209999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575057,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28348,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343492,"statfs":{"total":96636764160,"available":96607735808,"internally_reserved":0,"allocated":1355776,"data_stored":650663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.85899999999999999,"15min":0.874},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.4319999999999999,"5min":1.556,"15min":2.472},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96299999999999997,"15min":1.024},"min":{"1min":0.48299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3560000000000001,"5min":1.585,"15min":2.3260000000000001},"last":0.83299999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.0029999999999999},"min":{"1min":0.50800000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.454,"5min":3.234,"15min":3.234},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.1459999999999999,"15min":1.1619999999999999},"min":{"1min":0.52200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6299999999999999,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.0940000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.1830000000000001,"15min":1.234},"min":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4139999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.2629999999999999,"15min":1.2969999999999999},"min":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6879999999999999,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.151}]},{"osd":4,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.415,"15min":1.425},"min":{"1min":0.84299999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.766,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.3400000000000001,"15min":1.3620000000000001},"min":{"1min":0.80400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6759999999999999,"5min":3.016,"15min":3.016},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.393,"15min":1.415},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8169999999999999,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.327},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.552},"min":{"1min":0.96599999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.3039999999999998,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.585}]},{"osd":6,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.448,"15min":1.486},"min":{"1min":0.85899999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.0129999999999999,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.5,"15min":1.5289999999999999},"min":{"1min":0.71299999999999997,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.3610000000000002,"5min":2.5920000000000001,"15min":3.3929999999999998},"last":1.4510000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.52,"15min":1.5640000000000001},"min":{"1min":0.80700000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.7789999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":0.875},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.617,"15min":1.6439999999999999},"min":{"1min":0.88,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.206,"5min":2.7519999999999998,"15min":2.944},"last":1.542}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":1363968,"data_stored":650275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.85399999999999998,"15min":0.90400000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.409,"5min":1.421,"15min":1.821},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.0149999999999999,"15min":1.0980000000000001},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.827,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.92600000000000005,"15min":0.98699999999999999},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.4330000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0680000000000001,"15min":1.1579999999999999},"min":{"1min":0.51900000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6899999999999999,"5min":1.756,"15min":1.796},"last":1.0149999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.2150000000000001,"15min":1.3160000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.627,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.748},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.129,"15min":1.1930000000000001},"min":{"1min":0.55300000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5609999999999999,"5min":1.79,"15min":1.855},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.216,"15min":1.286},"min":{"1min":0.75600000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.958,"5min":2.069,"15min":2.069},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.349,"15min":1.4219999999999999},"min":{"1min":0.74099999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.9279999999999999,"5min":2.899,"15min":2.899},"last":1.589}]},{"osd":5,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3640000000000001,"15min":1.4379999999999999},"min":{"1min":0.88700000000000001,"5min":0.623,"15min":0.623},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.054},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.347,"15min":1.427},"min":{"1min":0.71199999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.387}]},{"osd":6,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.4890000000000001,"15min":1.5509999999999999},"min":{"1min":0.80000000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.462,"15min":1.538},"min":{"1min":0.81100000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.151}]},{"osd":7,"last update":"Mon Oct 16 14:33:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4650000000000001,"15min":1.5580000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.0870000000000002,"5min":2.2000000000000002,"15min":2.262},"last":1.522}]}]},{"osd":0,"up_from":13,"seq":55834575057,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37252,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334588,"statfs":{"total":96636764160,"available":96598618112,"internally_reserved":0,"allocated":1949696,"data_stored":1240483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.875,"15min":0.94299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.7929999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.04,"5min":1.097,"15min":1.163},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.36399999999999999},"max":{"1min":1.508,"5min":1.833,"15min":1.867},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.204,"15min":1.2470000000000001},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1.6359999999999999,"5min":1.887,"15min":2.2970000000000002},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.093,"5min":1.159,"15min":1.2050000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6000000000000001,"5min":2.012,"15min":2.2330000000000001},"last":1.764}]},{"osd":3,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.96099999999999997,"15min":1.0029999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40100000000000002},"max":{"1min":1.3149999999999999,"5min":1.889,"15min":1.9410000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.181,"15min":1.238},"min":{"1min":0.45000000000000001,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.823,"5min":1.823,"15min":2.5339999999999998},"last":1.6120000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.266,"15min":1.302},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.61599999999999999},"max":{"1min":1.786,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4419999999999999,"15min":1.462},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.7809999999999999,"5min":2.032,"15min":2.6259999999999999},"last":1.25}]},{"osd":5,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.383,"15min":1.419},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.02,"5min":2.097,"15min":2.5880000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.302,"15min":1.3600000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.67900000000000005},"max":{"1min":1.778,"5min":1.778,"15min":2.4889999999999999},"last":1.1240000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5389999999999999,"15min":1.573},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8819999999999999,"5min":2.097,"15min":2.7599999999999998},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.4770000000000001,"15min":1.5},"min":{"1min":1.02,"5min":0.98899999999999999,"15min":0.89000000000000001},"max":{"1min":1.9299999999999999,"5min":2.1459999999999999,"15min":2.5089999999999999},"last":1.3160000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:32:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5409999999999999,"15min":1.585},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1040000000000001,"5min":2.2650000000000001,"15min":2.6509999999999998},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.472,"15min":1.544},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.976},"last":1.532}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:33:51.469 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:33:51.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:33:51.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.743+0000 7fd0a2fed640 1 Processor -- start 2023-10-16T14:33:51.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.743+0000 7fd0a2fed640 1 -- start start 2023-10-16T14:33:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.745+0000 7fd0a2fed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd09c15cb50 0x7fd09c161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.745+0000 7fd0a2fed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd09c0b1690 0x7fd09c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.745+0000 7fd0a2fed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c0b4f00 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.745+0000 7fd0a2fed640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c1616c0 con 0x7fd09c15cb50 2023-10-16T14:33:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.745+0000 7fd0a2fed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c0cdbf0 con 0x7fd09c0b1690 2023-10-16T14:33:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.745+0000 7fd0a1feb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd09c15cb50 0x7fd09c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.746+0000 7fd0a1feb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd09c15cb50 0x7fd09c161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42946/0 (socket says 172.21.15.29:42946) 2023-10-16T14:33:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.746+0000 7fd0a1feb640 1 -- 172.21.15.29:0/1469951811 learned_addr learned my addr 172.21.15.29:0/1469951811 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.746+0000 7fd0a1feb640 1 -- 172.21.15.29:0/1469951811 >> v2:172.21.15.170:3300/0 conn(0x7fd09c0b1690 msgr2=0x7fd09c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.746+0000 7fd0a1feb640 1 --2- 172.21.15.29:0/1469951811 >> v2:172.21.15.170:3300/0 conn(0x7fd09c0b1690 0x7fd09c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.746+0000 7fd0a1feb640 1 -- 172.21.15.29:0/1469951811 >> v1:172.21.15.170:6789/0 conn(0x7fd09c0b4ab0 legacy=0x7fd09c06bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.746+0000 7fd0a1feb640 1 -- 172.21.15.29:0/1469951811 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0cde90 con 0x7fd09c15cb50 2023-10-16T14:33:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.747+0000 7fd0a1feb640 1 --2- 172.21.15.29:0/1469951811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c15cb50 0x7fd09c161180 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0b24f0 tx=0x7fd09802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192371d2677df9c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.747+0000 7fd0a0fe9640 1 -- 172.21.15.29:0/1469951811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098031750 con 0x7fd09c15cb50 2023-10-16T14:33:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.748+0000 7fd0a0fe9640 1 -- 172.21.15.29:0/1469951811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0980318b0 con 0x7fd09c15cb50 2023-10-16T14:33:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.748+0000 7fd0a0fe9640 1 -- 172.21.15.29:0/1469951811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098031c40 con 0x7fd09c15cb50 2023-10-16T14:33:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.748+0000 7fd0a2fed640 1 -- 172.21.15.29:0/1469951811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c15cb50 msgr2=0x7fd09c161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.748+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/1469951811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c15cb50 0x7fd09c161180 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0b24f0 tx=0x7fd09802b7f0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.749+0000 7fd0a2fed640 1 -- 172.21.15.29:0/1469951811 shutdown_connections 2023-10-16T14:33:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.749+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/1469951811 >> v2:172.21.15.170:3300/0 conn(0x7fd09c0b1690 0x7fd09c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.749+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/1469951811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c15cb50 0x7fd09c161180 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.749+0000 7fd0a2fed640 1 -- 172.21.15.29:0/1469951811 >> 172.21.15.29:0/1469951811 conn(0x7fd09c0cf570 msgr2=0x7fd09c0cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.749+0000 7fd0a2fed640 1 -- 172.21.15.29:0/1469951811 shutdown_connections 2023-10-16T14:33:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.749+0000 7fd0a2fed640 1 -- 172.21.15.29:0/1469951811 wait complete. 2023-10-16T14:33:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.751+0000 7fd0a2fed640 1 Processor -- start 2023-10-16T14:33:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.751+0000 7fd0a2fed640 1 -- start start 2023-10-16T14:33:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a2fed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd09c0b1690 0x7fd09c0d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a2fed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 0x7fd09c14d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a2fed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09c15cb50 0x7fd09c15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a2fed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1f5e50 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a2fed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1f6130 con 0x7fd09c0b1690 2023-10-16T14:33:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a2fed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1f6410 con 0x7fd09c15cb50 2023-10-16T14:33:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a17ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 0x7fd09c14d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a17ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 0x7fd09c14d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42950/0 (socket says 172.21.15.29:42950) 2023-10-16T14:33:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a17ea640 1 -- 172.21.15.29:0/2589380735 learned_addr learned my addr 172.21.15.29:0/2589380735 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.752+0000 7fd0a1feb640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd09c0b1690 0x7fd09c0d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a27ec640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09c15cb50 0x7fd09c15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a17ea640 1 -- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09c15cb50 msgr2=0x7fd09c15f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a17ea640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09c15cb50 0x7fd09c15f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a17ea640 1 -- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd09c0b1690 msgr2=0x7fd09c0d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a17ea640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd09c0b1690 0x7fd09c0d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a17ea640 1 -- 172.21.15.29:0/2589380735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0980094d0 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a1feb640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd09c0b1690 0x7fd09c0d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:51.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a17ea640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 0x7fd09c14d880 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b410 tx=0x7fd08c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:51.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c011830 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c0119b0 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c011d60 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.753+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c1f66f0 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.754+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c0d36f0 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.755+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c1f6910 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.756+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd08c01a400 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.757+0000 7fd0927fc640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd08006af90 0x7fd08006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.757+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd08c089920 con 0x7fd09c0b4ab0 2023-10-16T14:33:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.757+0000 7fd0a1feb640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd08006af90 0x7fd08006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.759+0000 7fd0a1feb640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd08006af90 0x7fd08006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd098035000 tx=0x7fd098008030 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:51.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:51.761+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd08c0548b0 con 0x7fd09c0b4ab0 2023-10-16T14:33:52.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.028+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd09c04e9b0 con 0x7fd09c0b4ab0 2023-10-16T14:33:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.034+0000 7fd0927fc640 1 -- 172.21.15.29:0/2589380735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd08c015020 con 0x7fd09c0b4ab0 2023-10-16T14:33:52.037 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:33:52.037 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:33:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:33:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:33:52.038 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:33:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:33:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd08006af90 msgr2=0x7fd08006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd08006af90 0x7fd08006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd098035000 tx=0x7fd098008030 comp rx=0 tx=0).stop 2023-10-16T14:33:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 msgr2=0x7fd09c14d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 0x7fd09c14d880 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b410 tx=0x7fd08c00b8d0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 shutdown_connections 2023-10-16T14:33:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd08006af90 0x7fd08006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd09c0b4ab0 0x7fd09c14d880 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd09c15cb50 0x7fd09c15f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 --2- 172.21.15.29:0/2589380735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd09c0b1690 0x7fd09c0d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.037+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 >> 172.21.15.29:0/2589380735 conn(0x7fd09c0cf570 msgr2=0x7fd09c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.038+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 shutdown_connections 2023-10-16T14:33:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.038+0000 7fd0a2fed640 1 -- 172.21.15.29:0/2589380735 wait complete. 2023-10-16T14:33:52.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:33:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.277+0000 7f91be6fc640 1 Processor -- start 2023-10-16T14:33:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.278+0000 7f91be6fc640 1 -- start start 2023-10-16T14:33:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.279+0000 7f91be6fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 0x7f91b806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.279+0000 7f91be6fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f91b80ad970 0x7f91b815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.279+0000 7f91be6fc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b806c960 con 0x7f91b815ebc0 2023-10-16T14:33:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.279+0000 7f91be6fc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b815b9d0 con 0x7f91b80ad970 2023-10-16T14:33:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.279+0000 7f91be6fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815bcb0 con 0x7f91b806c040 2023-10-16T14:33:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.280+0000 7f91bdefb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91b815ebc0 0x7f91b80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44684/0 (socket says 172.21.15.29:44684) 2023-10-16T14:33:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.280+0000 7f91bdefb640 1 -- 172.21.15.29:0/125406743 learned_addr learned my addr 172.21.15.29:0/125406743 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.280+0000 7f91bd6fa640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 0x7f91b806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.280+0000 7f91bcef9640 1 --2- 172.21.15.29:0/125406743 >> v2:172.21.15.170:3300/0 conn(0x7f91b80ad970 0x7f91b815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.280+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891896280 0 0) 0x7f91b806c960 con 0x7f91b815ebc0 2023-10-16T14:33:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.280+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4003520 con 0x7f91b815ebc0 2023-10-16T14:33:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831738874 0 0) 0x7f91a4003520 con 0x7f91b815ebc0 2023-10-16T14:33:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 msgr2=0x7f91b806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 0x7f91b806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 >> v2:172.21.15.170:3300/0 conn(0x7f91b80ad970 msgr2=0x7f91b815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 --2- 172.21.15.29:0/125406743 >> v2:172.21.15.170:3300/0 conn(0x7f91b80ad970 0x7f91b815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b815bf90 con 0x7f91b815ebc0 2023-10-16T14:33:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f91b0001670 con 0x7f91b815ebc0 2023-10-16T14:33:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 >> v1:172.21.15.29:6789/0 conn(0x7f91b815ebc0 legacy=0x7f91b80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4003970 0x7f91a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 0x7f91a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815b9d0 con 0x7f91a40047c0 2023-10-16T14:33:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815bcb0 con 0x7f91a4003970 2023-10-16T14:33:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a400b980 con 0x7f91b806c040 2023-10-16T14:33:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.281+0000 7f91bcef9640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bd6fa640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4003970 0x7f91a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bdefb640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 0x7f91a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bcef9640 1 -- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 msgr2=0x7f91a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bcef9640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b806c040 0x7f91a4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bcef9640 1 -- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4003970 msgr2=0x7f91a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:52.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bcef9640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4003970 0x7f91a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.282+0000 7f91bcef9640 1 -- 172.21.15.29:0/125406743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4003520 con 0x7f91a40047c0 2023-10-16T14:33:52.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.283+0000 7f91bcef9640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91a4004c40 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009ac0 tx=0x7f91a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1527535d3f823b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:52.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.283+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a802c6d0 con 0x7f91a40047c0 2023-10-16T14:33:52.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.283+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a802c830 con 0x7f91a40047c0 2023-10-16T14:33:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.283+0000 7f91affff640 1 -- 172.21.15.29:0/125406743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a802cbc0 con 0x7f91a40047c0 2023-10-16T14:33:52.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91be6fc640 1 -- 172.21.15.29:0/125406743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 msgr2=0x7f91a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91be6fc640 1 --2- 172.21.15.29:0/125406743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91a4004c40 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009ac0 tx=0x7f91a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91bdefb640 1 -- 172.21.15.29:0/125406743 reap_dead start 2023-10-16T14:33:52.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91be6fc640 1 -- 172.21.15.29:0/125406743 shutdown_connections 2023-10-16T14:33:52.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91be6fc640 1 -- 172.21.15.29:0/125406743 >> 172.21.15.29:0/125406743 conn(0x7f91b80ce6a0 msgr2=0x7f91b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:52.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91be6fc640 1 -- 172.21.15.29:0/125406743 shutdown_connections 2023-10-16T14:33:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.284+0000 7f91be6fc640 1 -- 172.21.15.29:0/125406743 wait complete. 2023-10-16T14:33:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.286+0000 7f91be6fc640 1 Processor -- start 2023-10-16T14:33:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.286+0000 7f91be6fc640 1 -- start start 2023-10-16T14:33:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91be6fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91b8152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91be6fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b8152b00 0x7f91b8158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91be6fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91b80c6ea0 0x7f91b80ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91be6fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b80b0730 con 0x7f91a40047c0 2023-10-16T14:33:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91be6fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b80c8600 con 0x7f91b80c6ea0 2023-10-16T14:33:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91be6fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b80b0170 con 0x7f91b8152b00 2023-10-16T14:33:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.287+0000 7f91bd6fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91b8152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91b8152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42956/0 (socket says 172.21.15.29:42956) 2023-10-16T14:33:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 -- 172.21.15.29:0/1442933179 learned_addr learned my addr 172.21.15.29:0/1442933179 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bdefb640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91b80c6ea0 0x7f91b80ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bcef9640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b8152b00 0x7f91b8158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 -- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b8152b00 msgr2=0x7f91b8158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b8152b00 0x7f91b8158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 -- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91b80c6ea0 msgr2=0x7f91b80ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91b80c6ea0 0x7f91b80ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 -- 172.21.15.29:0/1442933179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b80b0450 con 0x7f91a40047c0 2023-10-16T14:33:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bdefb640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91b80c6ea0 0x7f91b80ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.288+0000 7f91bd6fa640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91b8152500 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f91b4007f30 tx=0x7f91b4011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:52.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.289+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b400e030 con 0x7f91a40047c0 2023-10-16T14:33:52.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.289+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b4016470 con 0x7f91a40047c0 2023-10-16T14:33:52.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.289+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b4016820 con 0x7f91a40047c0 2023-10-16T14:33:52.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.289+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b80c5910 con 0x7f91a40047c0 2023-10-16T14:33:52.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.289+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b80afe00 con 0x7f91a40047c0 2023-10-16T14:33:52.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.290+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b806b330 con 0x7f91a40047c0 2023-10-16T14:33:52.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.291+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f91b4016980 con 0x7f91a40047c0 2023-10-16T14:33:52.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.292+0000 7f91adffb640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f919c06af90 0x7f919c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:52.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.292+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91b400f070 con 0x7f91a40047c0 2023-10-16T14:33:52.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.292+0000 7f91bcef9640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f919c06af90 0x7f919c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:52.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.294+0000 7f91bcef9640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f919c06af90 0x7f919c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f91a8036000 tx=0x7f91a8030000 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:52.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.296+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f91b4055640 con 0x7f91a40047c0 2023-10-16T14:33:52.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.529+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b80c5750 con 0x7f91a40047c0 2023-10-16T14:33:52.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.531+0000 7f91adffb640 1 -- 172.21.15.29:0/1442933179 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f91b4058c60 con 0x7f91a40047c0 2023-10-16T14:33:52.534 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:33:52.534 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:45.763311+0000","modified":"2023-10-16T14:33:48.952054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7323},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7323":{"gid":7323,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6836/190249339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":190249339},{"type":"v1","addr":"172.21.15.170:6836","nonce":190249339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T14:33:52.534 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2023-10-16T14:33:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.533+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f919c06af90 msgr2=0x7f919c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.533+0000 7f91be6fc640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f919c06af90 0x7f919c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f91a8036000 tx=0x7f91a8030000 comp rx=0 tx=0).stop 2023-10-16T14:33:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 msgr2=0x7f91b8152500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91b8152500 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f91b4007f30 tx=0x7f91b4011660 comp rx=0 tx=0).stop 2023-10-16T14:33:52.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 shutdown_connections 2023-10-16T14:33:52.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f919c06af90 0x7f919c06d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91b80c6ea0 0x7f91b80ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91b8152b00 0x7f91b8158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 --2- 172.21.15.29:0/1442933179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a40047c0 0x7f91b8152500 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:52.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 >> 172.21.15.29:0/1442933179 conn(0x7f91b80ce6a0 msgr2=0x7f91b80c9240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:52.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 shutdown_connections 2023-10-16T14:33:52.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:52.534+0000 7f91be6fc640 1 -- 172.21.15.29:0/1442933179 wait complete. 2023-10-16T14:33:52.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:33:45.763311+0000', 'modified': '2023-10-16T14:33:48.952054+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7323}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7323': {'gid': 7323, 'name': 'd', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.170:6836/190249339', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 190249339}, {'type': 'v1', 'addr': '172.21.15.170:6836', 'nonce': 190249339}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:33:52.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:33:52.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T14:33:53.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.196+0000 7fd423002640 1 Processor -- start 2023-10-16T14:33:53.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.196+0000 7fd423002640 1 -- start start 2023-10-16T14:33:53.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd423002640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd41c06a4b0 0x7fd41c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd423002640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd41c0c9ca0 0x7fd41c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd423002640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c0cb030 con 0x7fd41c15ad70 2023-10-16T14:33:53.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd423002640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd41c0cae30 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd423002640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd41c0c91c0 con 0x7fd41c06a4b0 2023-10-16T14:33:53.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd420d9e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd41c15ad70 0x7fd41c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44688/0 (socket says 172.21.15.29:44688) 2023-10-16T14:33:53.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd420d9e640 1 -- 172.21.15.29:0/2976974396 learned_addr learned my addr 172.21.15.29:0/2976974396 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:53.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd42159f640 1 --2- 172.21.15.29:0/2976974396 >> v2:172.21.15.29:3300/0 conn(0x7fd41c0c9ca0 0x7fd41c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.198+0000 7fd41bfff640 1 --2- 172.21.15.29:0/2976974396 >> v2:172.21.15.170:3300/0 conn(0x7fd41c06a4b0 0x7fd41c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.199+0000 7fd41b7fe640 1 -- 172.21.15.29:0/2976974396 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621811902 0 0) 0x7fd41c0cb030 con 0x7fd41c15ad70 2023-10-16T14:33:53.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.199+0000 7fd41b7fe640 1 -- 172.21.15.29:0/2976974396 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c003520 con 0x7fd41c15ad70 2023-10-16T14:33:53.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.199+0000 7fd42159f640 1 -- 172.21.15.29:0/2976974396 >> v2:172.21.15.170:3300/0 conn(0x7fd41c06a4b0 msgr2=0x7fd41c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.199+0000 7fd42159f640 1 --2- 172.21.15.29:0/2976974396 >> v2:172.21.15.170:3300/0 conn(0x7fd41c06a4b0 0x7fd41c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.199+0000 7fd42159f640 1 -- 172.21.15.29:0/2976974396 >> v1:172.21.15.29:6789/0 conn(0x7fd41c15ad70 legacy=0x7fd41c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.199+0000 7fd42159f640 1 -- 172.21.15.29:0/2976974396 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c0c94a0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.200+0000 7fd42159f640 1 --2- 172.21.15.29:0/2976974396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0ca8f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0ce870 tx=0x7fd41402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eda24468ba14003 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.200+0000 7fd41b7fe640 1 -- 172.21.15.29:0/2976974396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41402cc80 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.200+0000 7fd41b7fe640 1 -- 172.21.15.29:0/2976974396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41402cde0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.200+0000 7fd41b7fe640 1 -- 172.21.15.29:0/2976974396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414034660 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 -- 172.21.15.29:0/2976974396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 msgr2=0x7fd41c0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 --2- 172.21.15.29:0/2976974396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0ca8f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0ce870 tx=0x7fd41402bed0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 -- 172.21.15.29:0/2976974396 shutdown_connections 2023-10-16T14:33:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 --2- 172.21.15.29:0/2976974396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0ca8f0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 --2- 172.21.15.29:0/2976974396 >> v2:172.21.15.170:3300/0 conn(0x7fd41c06a4b0 0x7fd41c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 -- 172.21.15.29:0/2976974396 >> 172.21.15.29:0/2976974396 conn(0x7fd41c0ce9b0 msgr2=0x7fd41c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 -- 172.21.15.29:0/2976974396 shutdown_connections 2023-10-16T14:33:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.201+0000 7fd423002640 1 -- 172.21.15.29:0/2976974396 wait complete. 2023-10-16T14:33:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.203+0000 7fd423002640 1 Processor -- start 2023-10-16T14:33:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.203+0000 7fd423002640 1 -- start start 2023-10-16T14:33:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd423002640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd41c06a4b0 0x7fd41c0a8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd423002640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd423002640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd41c15ad70 0x7fd41c0c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd423002640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c0c78b0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd423002640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c0c6c30 con 0x7fd41c15ad70 2023-10-16T14:33:53.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd423002640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c0c6f10 con 0x7fd41c06a4b0 2023-10-16T14:33:53.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd41bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd41bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42982/0 (socket says 172.21.15.29:42982) 2023-10-16T14:33:53.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd41bfff640 1 -- 172.21.15.29:0/739348323 learned_addr learned my addr 172.21.15.29:0/739348323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd41bfff640 1 -- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd41c06a4b0 msgr2=0x7fd41c0a8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.204+0000 7fd41bfff640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd41c06a4b0 0x7fd41c0a8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd41bfff640 1 -- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd41c15ad70 msgr2=0x7fd41c0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd41bfff640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd41c15ad70 0x7fd41c0c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd41bfff640 1 -- 172.21.15.29:0/739348323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd414009a70 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd41bfff640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0c4c10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd40800d800 tx=0x7fd40800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd4197fa640 1 -- 172.21.15.29:0/739348323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40800b6c0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd4197fa640 1 -- 172.21.15.29:0/739348323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40800b820 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd4197fa640 1 -- 172.21.15.29:0/739348323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40800bbb0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41c0c71f0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.205+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41c0c86f0 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.208+0000 7fd4197fa640 1 -- 172.21.15.29:0/739348323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd40800bd10 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.209+0000 7fd4197fa640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd40006af90 0x7fd40006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.209+0000 7fd4197fa640 1 -- 172.21.15.29:0/739348323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd408089d40 con 0x7fd41c0c9ca0 2023-10-16T14:33:53.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.210+0000 7fd425b94740 1 Processor -- start 2023-10-16T14:33:53.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.210+0000 7fd420d9e640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd40006af90 0x7fd40006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.210+0000 7fd425b94740 1 -- start start 2023-10-16T14:33:53.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd420d9e640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd40006af90 0x7fd40006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd410001570 tx=0x7fd410007930 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd425b94740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002294ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002292a320 0x56002293e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x56002292cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd425b94740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002281d7a0 con 0x56002294a490 2023-10-16T14:33:53.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56002286f740 con 0x56002292a320 2023-10-16T14:33:53.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.211+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x56002286ce30 con 0x56002293e7a0 2023-10-16T14:33:53.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002294ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002294ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42994/0 (socket says 172.21.15.29:42994) 2023-10-16T14:33:53.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 -- 172.21.15.29:0/995328444 learned_addr learned my addr 172.21.15.29:0/995328444 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd41bfff640 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002292a320 0x56002293e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 -- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 msgr2=0x56002292cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x56002292cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 -- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002292a320 msgr2=0x56002293e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002292a320 0x56002293e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 -- 172.21.15.29:0/995328444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56002282c910 con 0x56002294a490 2023-10-16T14:33:53.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd41bfff640 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002292a320 0x56002293e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:53.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.212+0000 7fd420d9e640 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002294ac60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd410005c00 tx=0x7fd41000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e721cbd11b1561a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd4067fc640 1 -- 172.21.15.29:0/995328444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41001b070 con 0x56002294a490 2023-10-16T14:33:53.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd4067fc640 1 -- 172.21.15.29:0/995328444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd410007750 con 0x56002294a490 2023-10-16T14:33:53.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd4067fc640 1 -- 172.21.15.29:0/995328444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd410012870 con 0x56002294a490 2023-10-16T14:33:53.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 -- 172.21.15.29:0/995328444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 msgr2=0x56002294ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002294ac60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd410005c00 tx=0x7fd41000f000 comp rx=0 tx=0).stop 2023-10-16T14:33:53.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 -- 172.21.15.29:0/995328444 shutdown_connections 2023-10-16T14:33:53.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x56002292cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002292a320 0x56002293e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 --2- 172.21.15.29:0/995328444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002294ac60 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.213+0000 7fd425b94740 1 -- 172.21.15.29:0/995328444 >> 172.21.15.29:0/995328444 conn(0x5600228ff190 msgr2=0x560022947d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.214+0000 7fd425b94740 1 -- 172.21.15.29:0/995328444 shutdown_connections 2023-10-16T14:33:53.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.214+0000 7fd425b94740 1 -- 172.21.15.29:0/995328444 wait complete. 2023-10-16T14:33:53.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.214+0000 7fd425b94740 1 -- start start 2023-10-16T14:33:53.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.214+0000 7fd425b94740 1 Processor -- start 2023-10-16T14:33:53.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.215+0000 7fd425b94740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 0x560022926e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.215+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x56002292c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.215+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x56002292c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.215+0000 7fd425b94740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x560022868720 con 0x56002293e7a0 2023-10-16T14:33:53.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.215+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560022806970 con 0x56002294a490 2023-10-16T14:33:53.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.215+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x560022806610 con 0x56002292df00 2023-10-16T14:33:53.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd425b94740 10 client.0 _ll_register_callbacks cb 0x5600228289d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:33:53.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 0x560022926e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 0x560022926e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43008/0 (socket says 172.21.15.29:43008) 2023-10-16T14:33:53.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 -- 172.21.15.29:0/2329738776 learned_addr learned my addr 172.21.15.29:0/2329738776 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd41bfff640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x56002292c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd42159f640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x56002292c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 -- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 msgr2=0x56002292c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x56002292c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 -- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 msgr2=0x56002292c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x56002292c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.216+0000 7fd420d9e640 1 -- 172.21.15.29:0/2329738776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c003520 con 0x56002293e7a0 2023-10-16T14:33:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd420d9e640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 0x560022926e40 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x5600228fef10 tx=0x7fd4100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3a8f5efe4b7025 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd3fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:33:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/2329738776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd410025070 con 0x56002293e7a0 2023-10-16T14:33:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/2329738776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd410014640 con 0x56002293e7a0 2023-10-16T14:33:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56002288f160 con 0x56002293e7a0 2023-10-16T14:33:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/2329738776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd410014840 con 0x56002293e7a0 2023-10-16T14:33:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd425b94740 10 client.7461 fetch_fsmap learned FSMap version 50 2023-10-16T14:33:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x560022866da0 con 0x56002293e7a0 2023-10-16T14:33:53.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.217+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/2329738776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56002288f160 con 0x56002293e7a0 2023-10-16T14:33:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/2329738776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 50) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd4100149a0 con 0x56002293e7a0 2023-10-16T14:33:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd425b94740 10 client.7461 fetch_fsmap finished waiting for FSMap version 50 2023-10-16T14:33:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd425b94740 10 client.7461 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T14:33:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd425b94740 20 client.7461 populate_metadata read hostname 'smithi029' 2023-10-16T14:33:53.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd425b94740 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x56002292b980 0x560022963ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd425b94740 4 client.7461 mds_command: new command op to 7323 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:33:53.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 --> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56002283acc0 con 0x56002292b980 2023-10-16T14:33:53.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.218+0000 7fd41bfff640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x56002292b980 0x560022963ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.219+0000 7fd41bfff640 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x56002292b980 0x560022963ca0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6eea3de3ed9a6c6 server_cookie=aca20a31d21146b9 in_seq=0 out_seq=0 2023-10-16T14:33:53.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.219+0000 7fd3fb7fe640 10 client.7461 ms_handle_connect on v2:172.21.15.170:6834/190249339 2023-10-16T14:33:53.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/2329738776 <== mds.0 v2:172.21.15.170:6834/190249339 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56002283acc0 con 0x56002292b980 2023-10-16T14:33:53.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd3fb7fe640 10 client.7461 handle_command_reply: tid=0 2023-10-16T14:33:53.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 client.7461 shutdown 2023-10-16T14:33:53.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 msgr2=0x560022926e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 0x560022926e40 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x5600228fef10 tx=0x7fd4100057f0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 shutdown_connections 2023-10-16T14:33:53.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x56002292b980 0x560022963ca0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x56002292c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x56002292c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd3fb7fe640 0 client.7461 ms_handle_reset on v2:172.21.15.170:6834/190249339 2023-10-16T14:33:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 --2- 172.21.15.29:0/2329738776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002293e7a0 0x560022926e40 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.221+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 >> 172.21.15.29:0/2329738776 conn(0x56002292a320 msgr2=0x5600228fbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.222+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 shutdown_connections 2023-10-16T14:33:53.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.222+0000 7fd425b94740 1 -- 172.21.15.29:0/2329738776 wait complete. 2023-10-16T14:33:53.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.222+0000 7fd425b94740 20 client.7461 trim_cache size 0 max 16384 2023-10-16T14:33:53.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.239+0000 7fd425b94740 1 Processor -- start 2023-10-16T14:33:53.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.239+0000 7fd425b94740 1 -- start start 2023-10-16T14:33:53.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x560022934700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002293e7a0 0x560022934c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd425b94740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002292e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd425b94740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600225ccf20 con 0x56002294a490 2023-10-16T14:33:53.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600228357f0 con 0x56002293e7a0 2023-10-16T14:33:53.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x56002292ad00 con 0x56002292df00 2023-10-16T14:33:53.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd42159f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002292e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd42159f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002292e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43018/0 (socket says 172.21.15.29:43018) 2023-10-16T14:33:53.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.240+0000 7fd42159f640 1 -- 172.21.15.29:0/545151943 learned_addr learned my addr 172.21.15.29:0/545151943 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd42159f640 1 -- 172.21.15.29:0/545151943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 msgr2=0x560022934700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd42159f640 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x560022934700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd42159f640 1 -- 172.21.15.29:0/545151943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002293e7a0 msgr2=0x560022934c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd42159f640 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002293e7a0 0x560022934c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd42159f640 1 -- 172.21.15.29:0/545151943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560022805b80 con 0x56002294a490 2023-10-16T14:33:53.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd42159f640 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002292e980 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fd414009e80 tx=0x7fd414004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdb04da08cf0f3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd406ffd640 1 -- 172.21.15.29:0/545151943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41402cc80 con 0x56002294a490 2023-10-16T14:33:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd406ffd640 1 -- 172.21.15.29:0/545151943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41403f480 con 0x56002294a490 2023-10-16T14:33:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.241+0000 7fd406ffd640 1 -- 172.21.15.29:0/545151943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41403f830 con 0x56002294a490 2023-10-16T14:33:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 -- 172.21.15.29:0/545151943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 msgr2=0x56002292e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002292e980 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fd414009e80 tx=0x7fd414004ab0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 -- 172.21.15.29:0/545151943 shutdown_connections 2023-10-16T14:33:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002294a490 0x56002292e980 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002293e7a0 0x560022934c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 --2- 172.21.15.29:0/545151943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002292df00 0x560022934700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 -- 172.21.15.29:0/545151943 >> 172.21.15.29:0/545151943 conn(0x56002292a320 msgr2=0x560022949b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 -- 172.21.15.29:0/545151943 shutdown_connections 2023-10-16T14:33:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.242+0000 7fd425b94740 1 -- 172.21.15.29:0/545151943 wait complete. 2023-10-16T14:33:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.243+0000 7fd425b94740 1 -- start start 2023-10-16T14:33:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.243+0000 7fd425b94740 1 Processor -- start 2023-10-16T14:33:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 0x5600229b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x5600228fbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x5600228fc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600229b98f0 con 0x56002292df00 2023-10-16T14:33:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560022972ab0 con 0x56002294a490 2023-10-16T14:33:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x560022972fe0 con 0x56002293e7a0 2023-10-16T14:33:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd425b94740 10 client.0 _ll_register_callbacks cb 0x7fd410013a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:33:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd420d9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 0x5600229b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd420d9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 0x5600229b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43030/0 (socket says 172.21.15.29:43030) 2023-10-16T14:33:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.244+0000 7fd420d9e640 1 -- 172.21.15.29:0/1627227348 learned_addr learned my addr 172.21.15.29:0/1627227348 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd42159f640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x5600228fc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd420d9e640 1 -- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 msgr2=0x5600228fbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd41bfff640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x5600228fbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd420d9e640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x5600228fbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd420d9e640 1 -- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 msgr2=0x5600228fc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd420d9e640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x5600228fc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd420d9e640 1 -- 172.21.15.29:0/1627227348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41c0c94a0 con 0x56002292df00 2023-10-16T14:33:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd42159f640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x5600228fc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd420d9e640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 0x5600229b90a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd410000f60 tx=0x7fd410010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ead1155cdfd285 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd3fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:33:53.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/1627227348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41003a070 con 0x56002292df00 2023-10-16T14:33:53.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/1627227348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd410005040 con 0x56002292df00 2023-10-16T14:33:53.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.245+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd410007750 con 0x56002292df00 2023-10-16T14:33:53.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/1627227348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd41000c0e0 con 0x56002292df00 2023-10-16T14:33:53.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 10 client.7467 fetch_fsmap learned FSMap version 50 2023-10-16T14:33:53.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x56002282b170 con 0x56002292df00 2023-10-16T14:33:53.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/1627227348 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd410007750 con 0x56002292df00 2023-10-16T14:33:53.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/1627227348 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 50) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd41003d020 con 0x56002292df00 2023-10-16T14:33:53.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 10 client.7467 fetch_fsmap finished waiting for FSMap version 50 2023-10-16T14:33:53.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 10 client.7467 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T14:33:53.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 20 client.7467 populate_metadata read hostname 'smithi029' 2023-10-16T14:33:53.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x560022934b90 0x560022934f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 4 client.7467 mds_command: new command op to 7323 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:33:53.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.246+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 --> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560022861700 con 0x560022934b90 2023-10-16T14:33:53.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.247+0000 7fd41bfff640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x560022934b90 0x560022934f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.248+0000 7fd41bfff640 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x560022934b90 0x560022934f70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b649c95273683f0 server_cookie=e7e4ec592d219cba in_seq=0 out_seq=0 2023-10-16T14:33:53.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.248+0000 7fd3fb7fe640 10 client.7467 ms_handle_connect on v2:172.21.15.170:6834/190249339 2023-10-16T14:33:53.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.248+0000 7fd3fb7fe640 1 -- 172.21.15.29:0/1627227348 <== mds.0 v2:172.21.15.170:6834/190249339 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x560022861700 con 0x560022934b90 2023-10-16T14:33:53.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.248+0000 7fd3fb7fe640 10 client.7467 handle_command_reply: tid=0 2023-10-16T14:33:53.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.248+0000 7fd425b94740 1 client.7467 shutdown 2023-10-16T14:33:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 msgr2=0x5600229b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 0x5600229b90a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd410000f60 tx=0x7fd410010000 comp rx=0 tx=0).stop 2023-10-16T14:33:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 shutdown_connections 2023-10-16T14:33:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:6834/190249339,v1:172.21.15.170:6836/190249339] conn(0x560022934b90 0x560022934f70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56002294a490 0x5600228fc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56002293e7a0 0x5600228fbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 --2- 172.21.15.29:0/1627227348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56002292df00 0x5600229b90a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 >> 172.21.15.29:0/1627227348 conn(0x56002292a320 msgr2=0x560022929b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd3fb7fe640 0 client.7467 ms_handle_reset on v2:172.21.15.170:6834/190249339 2023-10-16T14:33:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 shutdown_connections 2023-10-16T14:33:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 1 -- 172.21.15.29:0/1627227348 wait complete. 2023-10-16T14:33:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.249+0000 7fd425b94740 20 client.7467 trim_cache size 0 max 16384 2023-10-16T14:33:53.269 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:33:53.269 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:33:53.269 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:33:53.270 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7323, 2023-10-16T14:33:53.270 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:33:53.270 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:33:53.270 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:33:53.270 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8158308569999999, 2023-10-16T14:33:53.271 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 50, 2023-10-16T14:33:53.271 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 111, 2023-10-16T14:33:53.271 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 111, 2023-10-16T14:33:53.271 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.271048275 2023-10-16T14:33:53.271 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:33:53.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd40006af90 msgr2=0x7fd40006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd40006af90 0x7fd40006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd410001570 tx=0x7fd410007930 comp rx=0 tx=0).stop 2023-10-16T14:33:53.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 msgr2=0x7fd41c0c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0c4c10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd40800d800 tx=0x7fd40800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 shutdown_connections 2023-10-16T14:33:53.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fd40006af90 0x7fd40006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd41c0c9ca0 0x7fd41c0c4c10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd41c15ad70 0x7fd41c0c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 --2- 172.21.15.29:0/739348323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd41c06a4b0 0x7fd41c0a8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 >> 172.21.15.29:0/739348323 conn(0x7fd41c0ce9b0 msgr2=0x7fd41c0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 shutdown_connections 2023-10-16T14:33:53.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.250+0000 7fd423002640 1 -- 172.21.15.29:0/739348323 wait complete. 2023-10-16T14:33:53.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:33:53.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.488+0000 7f8c918bb640 1 Processor -- start 2023-10-16T14:33:53.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.488+0000 7f8c918bb640 1 -- start start 2023-10-16T14:33:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.490+0000 7f8c918bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c8c15aad0 0x7f8c8c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.490+0000 7f8c918bb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8c8c070d70 0x7f8c8c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.490+0000 7f8c918bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8c0b4620 con 0x7f8c8c0712c0 2023-10-16T14:33:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.490+0000 7f8c918bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c0b4340 con 0x7f8c8c15aad0 2023-10-16T14:33:53.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.490+0000 7f8c918bb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c06f520 con 0x7f8c8c070d70 2023-10-16T14:33:53.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.490+0000 7f8c908b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c8c15aad0 0x7f8c8c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c908b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c8c15aad0 0x7f8c8c0cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43046/0 (socket says 172.21.15.29:43046) 2023-10-16T14:33:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c910ba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8c8c0712c0 0x7f8c8c070000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44690/0 (socket says 172.21.15.29:44690) 2023-10-16T14:33:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c908b9640 1 -- 172.21.15.29:0/2831815173 learned_addr learned my addr 172.21.15.29:0/2831815173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c8bfff640 1 --2- 172.21.15.29:0/2831815173 >> v2:172.21.15.170:3300/0 conn(0x7f8c8c070d70 0x7f8c8c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c908b9640 1 -- 172.21.15.29:0/2831815173 >> v2:172.21.15.170:3300/0 conn(0x7f8c8c070d70 msgr2=0x7f8c8c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c908b9640 1 --2- 172.21.15.29:0/2831815173 >> v2:172.21.15.170:3300/0 conn(0x7f8c8c070d70 0x7f8c8c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c908b9640 1 -- 172.21.15.29:0/2831815173 >> v1:172.21.15.29:6789/0 conn(0x7f8c8c0712c0 legacy=0x7f8c8c070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c908b9640 1 -- 172.21.15.29:0/2831815173 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c06f800 con 0x7f8c8c15aad0 2023-10-16T14:33:53.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.491+0000 7f8c8b7fe640 1 -- 172.21.15.29:0/2831815173 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352648277 0 0) 0x7f8c8c0b4620 con 0x7f8c8c0712c0 2023-10-16T14:33:53.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.493+0000 7f8c908b9640 1 --2- 172.21.15.29:0/2831815173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c15aad0 0x7f8c8c0cc270 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c06d4a0 tx=0x7f8c8002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5678457c8a542ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.493+0000 7f8c8b7fe640 1 -- 172.21.15.29:0/2831815173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8002f850 con 0x7f8c8c15aad0 2023-10-16T14:33:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.493+0000 7f8c8b7fe640 1 -- 172.21.15.29:0/2831815173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8002f9b0 con 0x7f8c8c15aad0 2023-10-16T14:33:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.493+0000 7f8c8b7fe640 1 -- 172.21.15.29:0/2831815173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8002fd40 con 0x7f8c8c15aad0 2023-10-16T14:33:53.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 -- 172.21.15.29:0/2831815173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c15aad0 msgr2=0x7f8c8c0cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 --2- 172.21.15.29:0/2831815173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c15aad0 0x7f8c8c0cc270 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c06d4a0 tx=0x7f8c8002ba30 comp rx=0 tx=0).stop 2023-10-16T14:33:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 -- 172.21.15.29:0/2831815173 shutdown_connections 2023-10-16T14:33:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 --2- 172.21.15.29:0/2831815173 >> v2:172.21.15.170:3300/0 conn(0x7f8c8c070d70 0x7f8c8c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 --2- 172.21.15.29:0/2831815173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c15aad0 0x7f8c8c0cc270 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 -- 172.21.15.29:0/2831815173 >> 172.21.15.29:0/2831815173 conn(0x7f8c8c0b0a90 msgr2=0x7f8c8c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 -- 172.21.15.29:0/2831815173 shutdown_connections 2023-10-16T14:33:53.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.494+0000 7f8c918bb640 1 -- 172.21.15.29:0/2831815173 wait complete. 2023-10-16T14:33:53.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.496+0000 7f8c918bb640 1 Processor -- start 2023-10-16T14:33:53.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.497+0000 7f8c918bb640 1 -- start start 2023-10-16T14:33:53.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c918bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 0x7f8c8c1563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c918bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c8c0712c0 0x7f8c8c0c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c918bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c8c15aad0 0x7f8c8c0c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c918bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c15b800 con 0x7f8c8c070d70 2023-10-16T14:33:53.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c918bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0b2530 con 0x7f8c8c0712c0 2023-10-16T14:33:53.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c918bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0b27e0 con 0x7f8c8c15aad0 2023-10-16T14:33:53.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c908b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 0x7f8c8c1563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c908b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 0x7f8c8c1563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43050/0 (socket says 172.21.15.29:43050) 2023-10-16T14:33:53.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c908b9640 1 -- 172.21.15.29:0/684910177 learned_addr learned my addr 172.21.15.29:0/684910177 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:53.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c8bfff640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c8c0712c0 0x7f8c8c0c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.498+0000 7f8c910ba640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c8c15aad0 0x7f8c8c0c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c908b9640 1 -- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c8c15aad0 msgr2=0x7f8c8c0c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c908b9640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c8c15aad0 0x7f8c8c0c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c908b9640 1 -- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c8c0712c0 msgr2=0x7f8c8c0c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c908b9640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c8c0712c0 0x7f8c8c0c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c908b9640 1 -- 172.21.15.29:0/684910177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c800094d0 con 0x7f8c8c070d70 2023-10-16T14:33:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c8bfff640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c8c0712c0 0x7f8c8c0c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c910ba640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c8c15aad0 0x7f8c8c0c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c908b9640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 0x7f8c8c1563f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f8c8002ba70 tx=0x7f8c80004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80037d10 con 0x7f8c8c070d70 2023-10-16T14:33:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c80037e90 con 0x7f8c8c070d70 2023-10-16T14:33:53.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80041640 con 0x7f8c8c070d70 2023-10-16T14:33:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.499+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c0b2ac0 con 0x7f8c8c070d70 2023-10-16T14:33:53.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.500+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c0b3360 con 0x7f8c8c070d70 2023-10-16T14:33:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.501+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8c04e9b0 con 0x7f8c8c070d70 2023-10-16T14:33:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.502+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8c800417a0 con 0x7f8c8c070d70 2023-10-16T14:33:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.503+0000 7f8c897fa640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f8c7006af90 0x7f8c7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:53.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.503+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c8003c070 con 0x7f8c8c070d70 2023-10-16T14:33:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.504+0000 7f8c8bfff640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f8c7006af90 0x7f8c7006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:53.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.505+0000 7f8c8bfff640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f8c7006af90 0x7f8c7006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0cc4a0 tx=0x7f8c78008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:53.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.507+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c80077500 con 0x7f8c8c070d70 2023-10-16T14:33:53.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.739+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8c8c0cf520 con 0x7f8c8c070d70 2023-10-16T14:33:53.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.740+0000 7f8c897fa640 1 -- 172.21.15.29:0/684910177 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8c8007ab20 con 0x7f8c8c070d70 2023-10-16T14:33:53.743 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:33:53.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.742+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f8c7006af90 msgr2=0x7f8c7006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.742+0000 7f8c918bb640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f8c7006af90 0x7f8c7006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0cc4a0 tx=0x7f8c78008040 comp rx=0 tx=0).stop 2023-10-16T14:33:53.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.742+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 msgr2=0x7f8c8c1563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:53.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.742+0000 7f8c918bb640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 0x7f8c8c1563f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f8c8002ba70 tx=0x7f8c80004730 comp rx=0 tx=0).stop 2023-10-16T14:33:53.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 shutdown_connections 2023-10-16T14:33:53.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f8c7006af90 0x7f8c7006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8c8c15aad0 0x7f8c8c0c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8c8c0712c0 0x7f8c8c0c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 --2- 172.21.15.29:0/684910177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c8c070d70 0x7f8c8c1563f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:53.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 >> 172.21.15.29:0/684910177 conn(0x7f8c8c0b0a90 msgr2=0x7f8c8c06bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:53.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 shutdown_connections 2023-10-16T14:33:53.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:53.743+0000 7f8c918bb640 1 -- 172.21.15.29:0/684910177 wait complete. 2023-10-16T14:33:53.764 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:33:53.765 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:33:53.765 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:33:53.765 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:33:53.766 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:53.766 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:33:53.766 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:33:53.767 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:33:53.767 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:33:53.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:53.768 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:33:53.824 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:33:53.824 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:33:53.825 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:33:53.825 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.825 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:33:53.825 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.825 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:33:53.826 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:53.826 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:33:53.826 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:33:53.826 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:53.826 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:33:53.827 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:33:53.827 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:53.827 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:33:53.827 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:33:53.827 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41291sec preferred_lft 41291sec 2023-10-16T14:33:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:33:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:33:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:33:53.829 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:33:53.829 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:33:53.829 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:33:53.829 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.829 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:33:53.830 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.830 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:33:53.830 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:33:53.830 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:33:53.830 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.831 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:33:53.831 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:33:53.831 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:33:53.831 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:33:53.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:53.833 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:33:53.931 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:33:53.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:53.937 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:33:53.998 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:33:53.998 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:33:53.999 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:33:53.999 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:33:53.999 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:33:54.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:54.001 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:33:54.001 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:33:54.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:33:54.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:33:54.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:33:54.003 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:33:54.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:54.202 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:33:54.254 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:33:54.255 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:33:54.255 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:33:54.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:54.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:33:54.356 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:33:54.357 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:54.414 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:33:54.415 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:33:54.415 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:54.474 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:33:54.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:54.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:54.625 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:54.625 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:33:54.625 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:33:54.648 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:54.649 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:33:54.649 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:33:54.649 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:33:54.650 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:33:54.675 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:33:54.676 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:33:54.676 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:33:54.676 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:33:54.676 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:33:54.677 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:33:54.677 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:33:54.677 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:33:54.678 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:33:54.679 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:33:54.679 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:33:54.679 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:33:54.679 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:33:54.683 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:33:54.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:54.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:33:54.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:33:54.779 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:33:54.862 INFO:teuthology.orchestra.run.smithi029.stdout:client7479 2023-10-16T14:33:54.863 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:33:54.863 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:33:54.928 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:33:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.156+0000 7f0c53fff640 1 Processor -- start 2023-10-16T14:33:55.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.156+0000 7f0c53fff640 1 -- start start 2023-10-16T14:33:55.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c53fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c53fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c540cda80 0x7f0c5415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c53fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5406c960 con 0x7f0c540ad970 2023-10-16T14:33:55.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c53fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5415b9d0 con 0x7f0c540cda80 2023-10-16T14:33:55.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c53fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415bcb0 con 0x7f0c5406c040 2023-10-16T14:33:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c537fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c540cda80 0x7f0c5415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c537fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c540cda80 0x7f0c5415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57782/0 (socket says 172.21.15.29:57782) 2023-10-16T14:33:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.158+0000 7f0c537fe640 1 -- 172.21.15.29:0/2869992329 learned_addr learned my addr 172.21.15.29:0/2869992329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.159+0000 7f0c52ffd640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.159+0000 7f0c537fe640 1 -- 172.21.15.29:0/2869992329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 msgr2=0x7f0c5406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.159+0000 7f0c537fe640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.159+0000 7f0c537fe640 1 -- 172.21.15.29:0/2869992329 >> v1:172.21.15.170:6789/0 conn(0x7f0c540ad970 legacy=0x7f0c5415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.159+0000 7f0c537fe640 1 -- 172.21.15.29:0/2869992329 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5415bf90 con 0x7f0c540cda80 2023-10-16T14:33:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.159+0000 7f0c52ffd640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.160+0000 7f0c537fe640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540cda80 0x7f0c5415ed90 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07e8967e4e7045b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:55.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.160+0000 7f0c51ffb640 1 -- 172.21.15.29:0/2869992329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c031640 con 0x7f0c540cda80 2023-10-16T14:33:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c51ffb640 1 -- 172.21.15.29:0/2869992329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c0317a0 con 0x7f0c540cda80 2023-10-16T14:33:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c51ffb640 1 -- 172.21.15.29:0/2869992329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c031b30 con 0x7f0c540cda80 2023-10-16T14:33:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 -- 172.21.15.29:0/2869992329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540cda80 msgr2=0x7f0c5415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540cda80 0x7f0c5415ed90 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009870 tx=0x7f0c4c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 -- 172.21.15.29:0/2869992329 shutdown_connections 2023-10-16T14:33:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540cda80 0x7f0c5415ed90 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 --2- 172.21.15.29:0/2869992329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 -- 172.21.15.29:0/2869992329 >> 172.21.15.29:0/2869992329 conn(0x7f0c540ce6a0 msgr2=0x7f0c540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:55.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 -- 172.21.15.29:0/2869992329 shutdown_connections 2023-10-16T14:33:55.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.161+0000 7f0c53fff640 1 -- 172.21.15.29:0/2869992329 wait complete. 2023-10-16T14:33:55.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.163+0000 7f0c53fff640 1 Processor -- start 2023-10-16T14:33:55.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.163+0000 7f0c53fff640 1 -- start start 2023-10-16T14:33:55.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.164+0000 7f0c53fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.164+0000 7f0c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 0x7f0c54150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.164+0000 7f0c53fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c540cda80 0x7f0c540ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.164+0000 7f0c53fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c9870 con 0x7f0c540ad970 2023-10-16T14:33:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.164+0000 7f0c53fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c90f0 con 0x7f0c540cda80 2023-10-16T14:33:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.164+0000 7f0c53fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54151660 con 0x7f0c5406c040 2023-10-16T14:33:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 0x7f0c54150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 0x7f0c54150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57794/0 (socket says 172.21.15.29:57794) 2023-10-16T14:33:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 -- 172.21.15.29:0/4166350558 learned_addr learned my addr 172.21.15.29:0/4166350558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c537fe640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c540cda80 0x7f0c540ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c52ffd640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 -- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 msgr2=0x7f0c5414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 -- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c540cda80 msgr2=0x7f0c540ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c540cda80 0x7f0c540ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.165+0000 7f0c527fc640 1 -- 172.21.15.29:0/4166350558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c0094d0 con 0x7f0c540ad970 2023-10-16T14:33:55.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.166+0000 7f0c527fc640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 0x7f0c54150ca0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00c7c0 tx=0x7f0c3c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:55.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.166+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00aa50 con 0x7f0c540ad970 2023-10-16T14:33:55.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.166+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c00abb0 con 0x7f0c540ad970 2023-10-16T14:33:55.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.166+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c012420 con 0x7f0c540ad970 2023-10-16T14:33:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.166+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c541518b0 con 0x7f0c540ad970 2023-10-16T14:33:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.166+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c54151e70 con 0x7f0c540ad970 2023-10-16T14:33:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.167+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5406b330 con 0x7f0c540ad970 2023-10-16T14:33:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.168+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c3c012580 con 0x7f0c540ad970 2023-10-16T14:33:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.170+0000 7f0c4b7fe640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f0c3406af90 0x7f0c3406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.170+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c3c089990 con 0x7f0c540ad970 2023-10-16T14:33:55.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.171+0000 7f0c52ffd640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f0c3406af90 0x7f0c3406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.172+0000 7f0c52ffd640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f0c3406af90 0x7f0c3406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400a950 tx=0x7f0c44008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:55.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.174+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c3c054920 con 0x7f0c540ad970 2023-10-16T14:33:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.396+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 --> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c540cc550 con 0x7f0c3406af90 2023-10-16T14:33:55.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.400+0000 7f0c4b7fe640 1 -- 172.21.15.29:0/4166350558 <== mgr.6076 v2:172.21.15.170:6832/2925184086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0c4400c050 con 0x7f0c3406af90 2023-10-16T14:33:55.404 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:33:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:33:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:33:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:33:55.406 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:33:55.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f0c3406af90 msgr2=0x7f0c3406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f0c3406af90 0x7f0c3406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400a950 tx=0x7f0c44008040 comp rx=0 tx=0).stop 2023-10-16T14:33:55.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 msgr2=0x7f0c54150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 0x7f0c54150ca0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00c7c0 tx=0x7f0c3c00cc80 comp rx=0 tx=0).stop 2023-10-16T14:33:55.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 shutdown_connections 2023-10-16T14:33:55.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f0c3406af90 0x7f0c3406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c540ad970 0x7f0c54150ca0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0c540cda80 0x7f0c540ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 --2- 172.21.15.29:0/4166350558 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0c5406c040 0x7f0c5414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.403+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 >> 172.21.15.29:0/4166350558 conn(0x7f0c540ce6a0 msgr2=0x7f0c5415e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:55.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.404+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 shutdown_connections 2023-10-16T14:33:55.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.404+0000 7f0c53fff640 1 -- 172.21.15.29:0/4166350558 wait complete. 2023-10-16T14:33:55.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:33:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.666+0000 7fa0f4a59640 1 Processor -- start 2023-10-16T14:33:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.666+0000 7fa0f4a59640 1 -- start start 2023-10-16T14:33:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.667+0000 7fa0f4a59640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa0f006c040 0x7fa0f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.667+0000 7fa0f4a59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0f00ad970 0x7fa0f015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0f4a59640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f006c960 con 0x7fa0f015ebc0 2023-10-16T14:33:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0f4a59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f015b9d0 con 0x7fa0f00ad970 2023-10-16T14:33:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0f4a59640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f015bcb0 con 0x7fa0f006c040 2023-10-16T14:33:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0eeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0f00ad970 0x7fa0f015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0eeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0f00ad970 0x7fa0f015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57810/0 (socket says 172.21.15.29:57810) 2023-10-16T14:33:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0eeffd640 1 -- 172.21.15.29:0/94812017 learned_addr learned my addr 172.21.15.29:0/94812017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0ef7fe640 1 --2- 172.21.15.29:0/94812017 >> v2:172.21.15.170:3300/0 conn(0x7fa0f006c040 0x7fa0f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0eeffd640 1 -- 172.21.15.29:0/94812017 >> v2:172.21.15.170:3300/0 conn(0x7fa0f006c040 msgr2=0x7fa0f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.668+0000 7fa0eeffd640 1 --2- 172.21.15.29:0/94812017 >> v2:172.21.15.170:3300/0 conn(0x7fa0f006c040 0x7fa0f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.669+0000 7fa0eeffd640 1 -- 172.21.15.29:0/94812017 >> v1:172.21.15.170:6789/0 conn(0x7fa0f015ebc0 legacy=0x7fa0f00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.669+0000 7fa0eeffd640 1 -- 172.21.15.29:0/94812017 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f015bf90 con 0x7fa0f00ad970 2023-10-16T14:33:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.669+0000 7fa0ef7fe640 1 --2- 172.21.15.29:0/94812017 >> v2:172.21.15.170:3300/0 conn(0x7fa0f006c040 0x7fa0f006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.670+0000 7fa0eeffd640 1 --2- 172.21.15.29:0/94812017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f00ad970 0x7fa0f015e680 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00cf350 tx=0x7fa0e402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2810769c01a78ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.670+0000 7fa0ee7fc640 1 -- 172.21.15.29:0/94812017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4031890 con 0x7fa0f00ad970 2023-10-16T14:33:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.670+0000 7fa0ee7fc640 1 -- 172.21.15.29:0/94812017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e40319f0 con 0x7fa0f00ad970 2023-10-16T14:33:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.670+0000 7fa0ee7fc640 1 -- 172.21.15.29:0/94812017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4031d80 con 0x7fa0f00ad970 2023-10-16T14:33:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 -- 172.21.15.29:0/94812017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f00ad970 msgr2=0x7fa0f015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/94812017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f00ad970 0x7fa0f015e680 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00cf350 tx=0x7fa0e402b930 comp rx=0 tx=0).stop 2023-10-16T14:33:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 -- 172.21.15.29:0/94812017 shutdown_connections 2023-10-16T14:33:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/94812017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f00ad970 0x7fa0f015e680 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/94812017 >> v2:172.21.15.170:3300/0 conn(0x7fa0f006c040 0x7fa0f006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 -- 172.21.15.29:0/94812017 >> 172.21.15.29:0/94812017 conn(0x7fa0f00ce6a0 msgr2=0x7fa0f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 -- 172.21.15.29:0/94812017 shutdown_connections 2023-10-16T14:33:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.671+0000 7fa0f4a59640 1 -- 172.21.15.29:0/94812017 wait complete. 2023-10-16T14:33:55.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.673+0000 7fa0f4a59640 1 Processor -- start 2023-10-16T14:33:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.673+0000 7fa0f4a59640 1 -- start start 2023-10-16T14:33:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0f4a59640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0f006c040 0x7fa0f00ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0f4a59640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0f00ad970 0x7fa0f0153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0f4a59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 0x7fa0f00b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0f4a59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f01539c0 con 0x7fa0f015ebc0 2023-10-16T14:33:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0f4a59640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00b26b0 con 0x7fa0f00ad970 2023-10-16T14:33:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0f4a59640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00b2990 con 0x7fa0f006c040 2023-10-16T14:33:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 0x7fa0f00b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 0x7fa0f00b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57824/0 (socket says 172.21.15.29:57824) 2023-10-16T14:33:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.674+0000 7fa0effff640 1 -- 172.21.15.29:0/2404352840 learned_addr learned my addr 172.21.15.29:0/2404352840 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0eeffd640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0f00ad970 0x7fa0f0153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0ef7fe640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0f006c040 0x7fa0f00ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0effff640 1 -- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0f006c040 msgr2=0x7fa0f00ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0effff640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0f006c040 0x7fa0f00ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0effff640 1 -- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0f00ad970 msgr2=0x7fa0f0153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0effff640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0f00ad970 0x7fa0f0153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0effff640 1 -- 172.21.15.29:0/2404352840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40094d0 con 0x7fa0f015ebc0 2023-10-16T14:33:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0ef7fe640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0f006c040 0x7fa0f00ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:33:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0effff640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 0x7fa0f00b1e10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800c780 tx=0x7fa0e800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:55.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8004b60 con 0x7fa0f015ebc0 2023-10-16T14:33:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e800ce90 con 0x7fa0f015ebc0 2023-10-16T14:33:55.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e800a850 con 0x7fa0f015ebc0 2023-10-16T14:33:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.675+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f00b2c70 con 0x7fa0f015ebc0 2023-10-16T14:33:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.676+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f00b3230 con 0x7fa0f015ebc0 2023-10-16T14:33:55.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.677+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f006b330 con 0x7fa0f015ebc0 2023-10-16T14:33:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.678+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0e800a9b0 con 0x7fa0f015ebc0 2023-10-16T14:33:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.679+0000 7fa0d7fff640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa0dc06af90 0x7fa0dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.679+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0e8089a10 con 0x7fa0f015ebc0 2023-10-16T14:33:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.680+0000 7fa0ef7fe640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa0dc06af90 0x7fa0dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.681+0000 7fa0ef7fe640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa0dc06af90 0x7fa0dc06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0006fd0 tx=0x7fa0e0008040 comp rx=0 tx=0).ready entity=mgr.6076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.684+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0e8054910 con 0x7fa0f015ebc0 2023-10-16T14:33:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:55.925+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa0f00b3420 con 0x7fa0f015ebc0 2023-10-16T14:33:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.483+0000 7fa0d7fff640 1 -- 172.21.15.29:0/2404352840 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v46) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa0e8057f30 con 0x7fa0f015ebc0 2023-10-16T14:33:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa0dc06af90 msgr2=0x7fa0dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa0dc06af90 0x7fa0dc06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0006fd0 tx=0x7fa0e0008040 comp rx=0 tx=0).stop 2023-10-16T14:33:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 msgr2=0x7fa0f00b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 0x7fa0f00b1e10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800c780 tx=0x7fa0e800cc40 comp rx=0 tx=0).stop 2023-10-16T14:33:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 shutdown_connections 2023-10-16T14:33:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7fa0dc06af90 0x7fa0dc06d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0f015ebc0 0x7fa0f00b1e10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa0f00ad970 0x7fa0f0153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 --2- 172.21.15.29:0/2404352840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0f006c040 0x7fa0f00ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.487+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 >> 172.21.15.29:0/2404352840 conn(0x7fa0f00ce6a0 msgr2=0x7fa0f00c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:56.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.488+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 shutdown_connections 2023-10-16T14:33:56.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.488+0000 7fa0f4a59640 1 -- 172.21.15.29:0/2404352840 wait complete. 2023-10-16T14:33:56.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:33:56.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.716+0000 7f021afd0640 1 Processor -- start 2023-10-16T14:33:56.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.716+0000 7f021afd0640 1 -- start start 2023-10-16T14:33:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f021afd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0214160530 0x7f02141609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f021afd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140ac890 con 0x7f0214069fa0 2023-10-16T14:33:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f021afd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140ae800 con 0x7f0214065ab0 2023-10-16T14:33:56.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f021afd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02140ae1f0 con 0x7f0214160530 2023-10-16T14:33:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f0218d6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0214160530 0x7f02141609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f021956d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0214069fa0 0x7f021406a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54514/0 (socket says 172.21.15.29:54514) 2023-10-16T14:33:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.718+0000 7f021956d640 1 -- 172.21.15.29:0/2557193696 learned_addr learned my addr 172.21.15.29:0/2557193696 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.719+0000 7f0218d6c640 1 -- 172.21.15.29:0/2557193696 >> v1:172.21.15.170:6789/0 conn(0x7f0214065ab0 legacy=0x7f0214160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:56.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.719+0000 7f0218d6c640 1 -- 172.21.15.29:0/2557193696 >> v1:172.21.15.29:6789/0 conn(0x7f0214069fa0 legacy=0x7f021406a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.719+0000 7f0218d6c640 1 -- 172.21.15.29:0/2557193696 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140acae0 con 0x7f0214160530 2023-10-16T14:33:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.720+0000 7f0218d6c640 1 --2- 172.21.15.29:0/2557193696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214160530 0x7f02141609f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f02080097a0 tx=0x7f020802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642e78683d0db511 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.720+0000 7f02137fe640 1 -- 172.21.15.29:0/2557193696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020802e070 con 0x7f0214160530 2023-10-16T14:33:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.720+0000 7f02137fe640 1 -- 172.21.15.29:0/2557193696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02080315a0 con 0x7f0214160530 2023-10-16T14:33:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.720+0000 7f02137fe640 1 -- 172.21.15.29:0/2557193696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208031930 con 0x7f0214160530 2023-10-16T14:33:56.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 -- 172.21.15.29:0/2557193696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214160530 msgr2=0x7f02141609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 --2- 172.21.15.29:0/2557193696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214160530 0x7f02141609f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f02080097a0 tx=0x7f020802b840 comp rx=0 tx=0).stop 2023-10-16T14:33:56.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 -- 172.21.15.29:0/2557193696 shutdown_connections 2023-10-16T14:33:56.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 --2- 172.21.15.29:0/2557193696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214160530 0x7f02141609f0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 -- 172.21.15.29:0/2557193696 >> 172.21.15.29:0/2557193696 conn(0x7f021406bc10 msgr2=0x7f021406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 -- 172.21.15.29:0/2557193696 shutdown_connections 2023-10-16T14:33:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.721+0000 7f021afd0640 1 -- 172.21.15.29:0/2557193696 wait complete. 2023-10-16T14:33:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.723+0000 7f021afd0640 1 Processor -- start 2023-10-16T14:33:56.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.723+0000 7f021afd0640 1 -- start start 2023-10-16T14:33:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.724+0000 7f021afd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 0x7f0214151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.724+0000 7f021afd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214069fa0 0x7f02140c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.724+0000 7f021afd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0214160530 0x7f02140c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.724+0000 7f021afd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02140c7780 con 0x7f0214065ab0 2023-10-16T14:33:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.724+0000 7f021afd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02140b2340 con 0x7f0214160530 2023-10-16T14:33:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.724+0000 7f021afd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02140c4480 con 0x7f0214069fa0 2023-10-16T14:33:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 0x7f0214151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 0x7f0214151bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57860/0 (socket says 172.21.15.29:57860) 2023-10-16T14:33:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 -- 172.21.15.29:0/2612121432 learned_addr learned my addr 172.21.15.29:0/2612121432 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f021956d640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0214160530 0x7f02140c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0213fff640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214069fa0 0x7f02140c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 -- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214069fa0 msgr2=0x7f02140c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214069fa0 0x7f02140c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 -- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0214160530 msgr2=0x7f02140c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0214160530 0x7f02140c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.725+0000 7f0218d6c640 1 -- 172.21.15.29:0/2612121432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0208009400 con 0x7f0214065ab0 2023-10-16T14:33:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.726+0000 7f0218d6c640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 0x7f0214151bc0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0208034040 tx=0x7f02080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.726+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208032900 con 0x7f0214065ab0 2023-10-16T14:33:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.726+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0208032a80 con 0x7f0214065ab0 2023-10-16T14:33:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.726+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208032e30 con 0x7f0214065ab0 2023-10-16T14:33:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.726+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02140c4760 con 0x7f0214065ab0 2023-10-16T14:33:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.726+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0214152a50 con 0x7f0214065ab0 2023-10-16T14:33:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.727+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02140b2820 con 0x7f0214065ab0 2023-10-16T14:33:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.728+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f020803b3f0 con 0x7f0214065ab0 2023-10-16T14:33:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.729+0000 7f02117fa640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 0x7f01f806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.730+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02080ab9d0 con 0x7f0214065ab0 2023-10-16T14:33:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.730+0000 7f0213fff640 1 -- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 msgr2=0x7f01f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.730+0000 7f0213fff640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 0x7f01f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:56.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.739+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f02140b2820 con 0x7f0214065ab0 2023-10-16T14:33:56.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.931+0000 7f0213fff640 1 -- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 msgr2=0x7f01f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:56.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.931+0000 7f0213fff640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 0x7f01f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:56.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.940+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f02140b2820 con 0x7f0214065ab0 2023-10-16T14:33:56.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.945+0000 7f02117fa640 1 -- 172.21.15.29:0/2612121432 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f02140b2820 con 0x7f0214065ab0 2023-10-16T14:33:56.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 msgr2=0x7f01f806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:56.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 0x7f01f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 msgr2=0x7f0214151bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 0x7f0214151bc0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0208034040 tx=0x7f02080046c0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 shutdown_connections 2023-10-16T14:33:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f01f806afe0 0x7f01f806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0214160530 0x7f02140c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0214069fa0 0x7f02140c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 --2- 172.21.15.29:0/2612121432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0214065ab0 0x7f0214151bc0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 >> 172.21.15.29:0/2612121432 conn(0x7f021406bc10 msgr2=0x7f0214070000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.948+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 shutdown_connections 2023-10-16T14:33:56.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:56.949+0000 7f021afd0640 1 -- 172.21.15.29:0/2612121432 wait complete. 2023-10-16T14:33:56.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:33:57.024 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.020+0000 7fd83862b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:33:57.039 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.036+0000 7f7c73a2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:33:57.077 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.073+0000 7fd83862b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:33:57.094 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.092+0000 7f7c73a2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:33:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.176+0000 7f57f9dc4640 1 Processor -- start 2023-10-16T14:33:57.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.176+0000 7f57f9dc4640 1 -- start start 2023-10-16T14:33:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.178+0000 7f57f9dc4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 0x7f57f406c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.178+0000 7f57f9dc4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f415d910 con 0x7f57f4159950 2023-10-16T14:33:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.178+0000 7f57f9dc4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f415dbf0 con 0x7f57f4070b60 2023-10-16T14:33:57.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.178+0000 7f57f9dc4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f415e490 con 0x7f57f40ca0a0 2023-10-16T14:33:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.178+0000 7f57f8dc2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f57f4159950 0x7f57f4070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54530/0 (socket says 172.21.15.29:54530) 2023-10-16T14:33:57.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.178+0000 7f57f8dc2640 1 -- 172.21.15.29:0/3538427632 learned_addr learned my addr 172.21.15.29:0/3538427632 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f95c3640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 0x7f57f406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686475054 0 0) 0x7f57f415d910 con 0x7f57f4159950 2023-10-16T14:33:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e4005730 con 0x7f57f4159950 2023-10-16T14:33:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714512780 0 0) 0x7f57e4005730 con 0x7f57f4159950 2023-10-16T14:33:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 msgr2=0x7f57f406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 0x7f57f406c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 >> v1:172.21.15.170:6789/0 conn(0x7f57f4070b60 legacy=0x7f57f406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f415ded0 con 0x7f57f4159950 2023-10-16T14:33:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f57e8001670 con 0x7f57f4159950 2023-10-16T14:33:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 >> v1:172.21.15.29:6789/0 conn(0x7f57f4159950 legacy=0x7f57f4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 0x7f57e4005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e4006130 0x7f57e40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006f10 0x7f57e4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.179+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f415e490 con 0x7f57e4006130 2023-10-16T14:33:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e400dd20 con 0x7f57e4006f10 2023-10-16T14:33:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e400dee0 con 0x7f57f40ca0a0 2023-10-16T14:33:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e4006130 0x7f57e40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f8dc2640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 0x7f57e4005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f95c3640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006f10 0x7f57e4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f3fff640 1 -- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 msgr2=0x7f57e4005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40ca0a0 0x7f57e4005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f3fff640 1 -- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006f10 msgr2=0x7f57e4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:33:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006f10 0x7f57e4007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.180+0000 7f57f3fff640 1 -- 172.21.15.29:0/3538427632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f415ded0 con 0x7f57e4006130 2023-10-16T14:33:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.181+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e4006130 0x7f57e40069d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f57e0001750 tx=0x7f57e4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c948c330eaedde1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.181+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e00052d0 con 0x7f57e4006130 2023-10-16T14:33:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e002e050 con 0x7f57e4006130 2023-10-16T14:33:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f37fe640 1 -- 172.21.15.29:0/3538427632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e00035f0 con 0x7f57e4006130 2023-10-16T14:33:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3538427632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e4006130 msgr2=0x7f57e40069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3538427632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57e4006130 0x7f57e40069d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f57e0001750 tx=0x7f57e4007a00 comp rx=0 tx=0).stop 2023-10-16T14:33:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f95c3640 1 -- 172.21.15.29:0/3538427632 reap_dead start 2023-10-16T14:33:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3538427632 shutdown_connections 2023-10-16T14:33:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3538427632 >> 172.21.15.29:0/3538427632 conn(0x7f57f40ce6a0 msgr2=0x7f57f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3538427632 shutdown_connections 2023-10-16T14:33:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.182+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3538427632 wait complete. 2023-10-16T14:33:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.184+0000 7f57f9dc4640 1 Processor -- start 2023-10-16T14:33:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.184+0000 7f57f9dc4640 1 -- start start 2023-10-16T14:33:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.185+0000 7f57f9dc4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006130 0x7f57f4154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.185+0000 7f57f9dc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 0x7f57f414d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.185+0000 7f57f9dc4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40b3950 0x7f57f40b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.185+0000 7f57f9dc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f40b28b0 con 0x7f57f40b23f0 2023-10-16T14:33:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.185+0000 7f57f9dc4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f40b2b90 con 0x7f57e4006130 2023-10-16T14:33:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.185+0000 7f57f9dc4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f40b2e70 con 0x7f57f40b3950 2023-10-16T14:33:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 0x7f57f414d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 0x7f57f414d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57870/0 (socket says 172.21.15.29:57870) 2023-10-16T14:33:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 -- 172.21.15.29:0/3451196565 learned_addr learned my addr 172.21.15.29:0/3451196565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f8dc2640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006130 0x7f57f4154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f95c3640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40b3950 0x7f57f40b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 -- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40b3950 msgr2=0x7f57f40b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40b3950 0x7f57f40b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 -- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006130 msgr2=0x7f57f4154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006130 0x7f57f4154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f3fff640 1 -- 172.21.15.29:0/3451196565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f40b3150 con 0x7f57f40b23f0 2023-10-16T14:33:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.186+0000 7f57f8dc2640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006130 0x7f57f4154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:33:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.187+0000 7f57f3fff640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 0x7f57f414d870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f57e002bc30 tx=0x7f57e0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.187+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e00076c0 con 0x7f57f40b23f0 2023-10-16T14:33:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.187+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0007a30 con 0x7f57f40b23f0 2023-10-16T14:33:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.187+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0006be0 con 0x7f57f40b23f0 2023-10-16T14:33:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.187+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f40b3430 con 0x7f57f40b23f0 2023-10-16T14:33:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.187+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f40c84b0 con 0x7f57f40b23f0 2023-10-16T14:33:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.188+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f40c86a0 con 0x7f57f40b23f0 2023-10-16T14:33:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.189+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f57e0006d40 con 0x7f57f40b23f0 2023-10-16T14:33:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.191+0000 7f57f17fa640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 0x7f57d806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.191+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57e00ab670 con 0x7f57f40b23f0 2023-10-16T14:33:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.191+0000 7f57f8dc2640 1 -- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 msgr2=0x7f57d806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.191+0000 7f57f8dc2640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 0x7f57d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.195+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f57e0076460 con 0x7f57f40b23f0 2023-10-16T14:33:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.392+0000 7f57f8dc2640 1 -- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 msgr2=0x7f57d806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.392+0000 7f57f8dc2640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 0x7f57d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:57.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.410+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f57f404e9b0 con 0x7f57f40b23f0 2023-10-16T14:33:57.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.414+0000 7f57f17fa640 1 -- 172.21.15.29:0/3451196565 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f57e0079a80 con 0x7f57f40b23f0 2023-10-16T14:33:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 msgr2=0x7f57d806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:33:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 0x7f57d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 msgr2=0x7f57f414d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 0x7f57f414d870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f57e002bc30 tx=0x7f57e0001120 comp rx=0 tx=0).stop 2023-10-16T14:33:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 shutdown_connections 2023-10-16T14:33:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f57d806afe0 0x7f57d806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57f40b23f0 0x7f57f414d870 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57f40b3950 0x7f57f40b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 --2- 172.21.15.29:0/3451196565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57e4006130 0x7f57f4154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 >> 172.21.15.29:0/3451196565 conn(0x7f57f40ce6a0 msgr2=0x7f57f406b860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 shutdown_connections 2023-10-16T14:33:57.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.417+0000 7f57f9dc4640 1 -- 172.21.15.29:0/3451196565 wait complete. 2023-10-16T14:33:57.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-10-16T14:33:57.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.523+0000 7fd83862b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:33:57.578 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.575+0000 7fd83862b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:33:57.606 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.603+0000 7f7c73a2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:33:57.628 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.624+0000 7fd83862b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:33:57.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.644+0000 7f83ff59e640 1 Processor -- start 2023-10-16T14:33:57.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.644+0000 7f83ff59e640 1 -- start start 2023-10-16T14:33:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.646+0000 7f83ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f84000b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.646+0000 7f83ff59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840015d150 con 0x7f84000b8480 2023-10-16T14:33:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.646+0000 7f83ff59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84000d1180 con 0x7f840015cd70 2023-10-16T14:33:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.646+0000 7f83ff59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84000d1460 con 0x7f84000cded0 2023-10-16T14:33:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.646+0000 7f83fdd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84000b8480 0x7f84000cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54536/0 (socket says 172.21.15.29:54536) 2023-10-16T14:33:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.646+0000 7f83fdd9b640 1 -- 172.21.15.29:0/4285905859 learned_addr learned my addr 172.21.15.29:0/4285905859 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:33:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fed9d640 1 --2- 172.21.15.29:0/4285905859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f84000b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fd59a640 1 -- 172.21.15.29:0/4285905859 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103083633 0 0) 0x7f840015d150 con 0x7f84000b8480 2023-10-16T14:33:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fd59a640 1 -- 172.21.15.29:0/4285905859 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e8003520 con 0x7f84000b8480 2023-10-16T14:33:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fed9d640 1 -- 172.21.15.29:0/4285905859 >> v1:172.21.15.170:6789/0 conn(0x7f840015cd70 legacy=0x7f84000d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fed9d640 1 -- 172.21.15.29:0/4285905859 >> v1:172.21.15.29:6789/0 conn(0x7f84000b8480 legacy=0x7f84000cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fd59a640 1 -- 172.21.15.29:0/4285905859 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646277957 0 0) 0x7f83e8003520 con 0x7f84000b8480 2023-10-16T14:33:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.647+0000 7f83fed9d640 1 -- 172.21.15.29:0/4285905859 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000d1740 con 0x7f84000cded0 2023-10-16T14:33:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.649+0000 7f83fed9d640 1 --2- 172.21.15.29:0/4285905859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f84000b5c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009dc0 tx=0x7f83f802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=addb47e977320585 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.649+0000 7f83fd59a640 1 -- 172.21.15.29:0/4285905859 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4001690 con 0x7f84000cded0 2023-10-16T14:33:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.649+0000 7f83fd59a640 1 -- 172.21.15.29:0/4285905859 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f802c910 con 0x7f84000cded0 2023-10-16T14:33:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.649+0000 7f83fd59a640 1 -- 172.21.15.29:0/4285905859 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f802cc80 con 0x7f84000cded0 2023-10-16T14:33:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 -- 172.21.15.29:0/4285905859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 msgr2=0x7f84000b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 --2- 172.21.15.29:0/4285905859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f84000b5c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009dc0 tx=0x7f83f802bde0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 -- 172.21.15.29:0/4285905859 shutdown_connections 2023-10-16T14:33:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 --2- 172.21.15.29:0/4285905859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f84000b5c90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 -- 172.21.15.29:0/4285905859 >> 172.21.15.29:0/4285905859 conn(0x7f84000cf570 msgr2=0x7f84000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:33:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 -- 172.21.15.29:0/4285905859 shutdown_connections 2023-10-16T14:33:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.650+0000 7f83ff59e640 1 -- 172.21.15.29:0/4285905859 wait complete. 2023-10-16T14:33:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.651+0000 7f83ff59e640 1 Processor -- start 2023-10-16T14:33:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.652+0000 7f83ff59e640 1 -- start start 2023-10-16T14:33:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83ff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84000b8480 0x7f84001f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f8400156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83ff59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f840015cd70 0x7f84000b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83ff59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84001567d0 con 0x7f84000b8480 2023-10-16T14:33:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83ff59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8400158060 con 0x7f840015cd70 2023-10-16T14:33:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83ff59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84000ccc80 con 0x7f84000cded0 2023-10-16T14:33:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84000b8480 0x7f84001f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84000b8480 0x7f84001f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57876/0 (socket says 172.21.15.29:57876) 2023-10-16T14:33:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 -- 172.21.15.29:0/98617099 learned_addr learned my addr 172.21.15.29:0/98617099 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:33:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fed9d640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f840015cd70 0x7f84000b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f8400156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:33:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 msgr2=0x7f8400156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f84000cded0 0x7f8400156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f840015cd70 msgr2=0x7f84000b75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:33:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f840015cd70 0x7f84000b75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:33:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.653+0000 7f83fe59c640 1 -- 172.21.15.29:0/98617099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f8009a70 con 0x7f84000b8480 2023-10-16T14:33:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.654+0000 7f83fe59c640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84000b8480 0x7f84001f1d20 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f83f000c400 tx=0x7f83f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:33:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.654+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f00052d0 con 0x7f84000b8480 2023-10-16T14:33:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.654+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f0005430 con 0x7f84000b8480 2023-10-16T14:33:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.654+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0003830 con 0x7f84000b8480 2023-10-16T14:33:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.654+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8400142c10 con 0x7f84000b8480 2023-10-16T14:33:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.654+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8400145ac0 con 0x7f84000b8480 2023-10-16T14:33:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.656+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840004e9b0 con 0x7f84000b8480 2023-10-16T14:33:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.658+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f83f0003990 con 0x7f84000b8480 2023-10-16T14:33:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.658+0000 7f83eeffd640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 0x7f83e006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:33:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.658+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83f0088d70 con 0x7f84000b8480 2023-10-16T14:33:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.659+0000 7f83fdd9b640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 msgr2=0x7f83e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.659+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 0x7f83e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:33:57.664 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.660+0000 7f7c73a2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:33:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.663+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83f0053d10 con 0x7f84000b8480 2023-10-16T14:33:57.690 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.687+0000 7fd83862b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:33:57.717 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.714+0000 7f7c73a2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:33:57.739 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.735+0000 7fd83862b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:33:57.781 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.779+0000 7f7c73a2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:33:57.833 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.830+0000 7f7c73a2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:33:57.849 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.845+0000 7fd83862b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:33:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.860+0000 7f83fdd9b640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 msgr2=0x7f83e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.860+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 0x7f83e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:33:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:57.871+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 --> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f84000d1d00 con 0x7f83e006afe0 2023-10-16T14:33:57.911 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.907+0000 7fd83862b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:57.949 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:57.946+0000 7f7c73a2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:33:57.961 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:57.957+0000 7fd83862b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:33:58.014 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.011+0000 7f7c73a2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:58.018 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.014+0000 7fd83862b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:33:58.065 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.063+0000 7f7c73a2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:33:58.067 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.063+0000 7fd83862b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:33:58.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.121+0000 7f7c73a2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:33:58.177 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.174+0000 7f7c73a2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:33:58.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:58.261+0000 7f83fdd9b640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 msgr2=0x7f83e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:58.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:58.261+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 0x7f83e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:33:58.448 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.444+0000 7fd83862b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:33:58.576 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.573+0000 7f7c73a2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:33:58.606 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.602+0000 7fd83862b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:33:58.714 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.710+0000 7fd83862b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:33:58.743 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.740+0000 7f7c73a2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:33:58.763 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.759+0000 7fd83862b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:33:58.855 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.852+0000 7f7c73a2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:33:58.868 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.865+0000 7fd83862b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:58.907 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:58.904+0000 7f7c73a2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:33:58.975 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:58.971+0000 7fd83862b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:33:59.019 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:59.017+0000 7f7c73a2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:59.062+0000 7f83fdd9b640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 msgr2=0x7f83e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/2925184086 2023-10-16T14:33:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:33:59.062+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 0x7f83e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:33:59.131 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:59.127+0000 7fd83862b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:59.132 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:59.128+0000 7f7c73a2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:33:59.180 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:59.176+0000 7fd83862b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:59.295 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:59.292+0000 7f7c73a2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:33:59.298 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:33:59.294+0000 7fd83862b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:33:59.346 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:59.343+0000 7f7c73a2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:33:59.470 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:33:59.467+0000 7f7c73a2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:34:00.123 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.119+0000 7fd83862b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:34:00.338 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:34:00.335+0000 7f7c73a2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:34:00.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.415+0000 7fd83862b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:34:00.475 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.472+0000 7fd83862b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:34:00.588 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.585+0000 7fd83862b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:34:00.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:00.609+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 47) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f83f0087290 con 0x7f84000b8480 2023-10-16T14:34:00.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:00.610+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 msgr2=0x7f83e006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:34:00.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:00.610+0000 7f83eeffd640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/2925184086,v1:172.21.15.170:6833/2925184086] conn(0x7f83e006afe0 0x7f83e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:00.653 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:34:00.648+0000 7f7c73a2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:34:00.716 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:34:00.713+0000 7f7c73a2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:34:00.789 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.785+0000 7fd7b4a24640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.785+0000 7fd7b4a24640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.785+0000 7fd7b4a24640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.785+0000 7fd7b4a24640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.785+0000 7fd7b4a24640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.791 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.785+0000 7fd7b7a6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.791 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.786+0000 7fd7b7a6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.791 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.786+0000 7fd7b7a6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.791 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.786+0000 7fd7b7a6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.791 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:34:00.786+0000 7fd7b7a6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:34:00.844 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:34:00.841+0000 7f7c73a2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:34:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.623+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 48) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f83f00540e0 con 0x7f84000b8480 2023-10-16T14:34:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.624+0000 7f83eeffd640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f83e0035880 0x7f83e0074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.624+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 --> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f83e0035c60 con 0x7f83e0035880 2023-10-16T14:34:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.628+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f83e0035880 0x7f83e0074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:01.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.639+0000 7f83fdd9b640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f83e0035880 0x7f83e0074ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f84000cc790 tx=0x7f83f4001950 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:34:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.807+0000 7f83eeffd640 1 -- 172.21.15.29:0/98617099 <== mgr.5855 v2:172.21.15.170:6832/3649833910 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f83e0035c60 con 0x7f83e0035880 2023-10-16T14:34:01.810 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /bad-path 2023-10-16T14:34:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f83e0035880 msgr2=0x7f83e0074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f83e0035880 0x7f83e0074ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f84000cc790 tx=0x7f83f4001950 comp rx=0 tx=0).stop 2023-10-16T14:34:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84000b8480 msgr2=0x7f84001f1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 --2- 172.21.15.29:0/98617099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84000b8480 0x7f84001f1d20 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f83f000c400 tx=0x7f83f000c8c0 comp rx=0 tx=0).stop 2023-10-16T14:34:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83fed9d640 1 -- 172.21.15.29:0/98617099 reap_dead start 2023-10-16T14:34:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 shutdown_connections 2023-10-16T14:34:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 >> 172.21.15.29:0/98617099 conn(0x7f84000cf570 msgr2=0x7f84000cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:34:01.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 shutdown_connections 2023-10-16T14:34:01.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:01.811+0000 7f83ff59e640 1 -- 172.21.15.29:0/98617099 wait complete. 2023-10-16T14:34:01.835 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-16T14:34:01.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T14:34:02.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.050+0000 7f3df7064640 1 Processor -- start 2023-10-16T14:34:02.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.050+0000 7f3df7064640 1 -- start start 2023-10-16T14:34:02.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.052+0000 7f3df7064640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3df0159ab0 0x7f3df00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.052+0000 7f3df7064640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df0070360 con 0x7f3df006c140 2023-10-16T14:34:02.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.052+0000 7f3df7064640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00cd1f0 con 0x7f3df0159e90 2023-10-16T14:34:02.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.052+0000 7f3df7064640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df015d890 con 0x7f3df0159ab0 2023-10-16T14:34:02.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.052+0000 7f3df5601640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3df006c140 0x7f3df015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54550/0 (socket says 172.21.15.29:54550) 2023-10-16T14:34:02.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.052+0000 7f3df5601640 1 -- 172.21.15.29:0/1540028842 learned_addr learned my addr 172.21.15.29:0/1540028842 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:34:02.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3df4e00640 1 --2- 172.21.15.29:0/1540028842 >> v2:172.21.15.170:3300/0 conn(0x7f3df0159ab0 0x7f3df00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836194247 0 0) 0x7f3df0070360 con 0x7f3df006c140 2023-10-16T14:34:02.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de0003520 con 0x7f3df006c140 2023-10-16T14:34:02.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4140405450 0 0) 0x7f3de0003520 con 0x7f3df006c140 2023-10-16T14:34:02.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 >> v2:172.21.15.170:3300/0 conn(0x7f3df0159ab0 msgr2=0x7f3df00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:34:02.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 --2- 172.21.15.29:0/1540028842 >> v2:172.21.15.170:3300/0 conn(0x7f3df0159ab0 0x7f3df00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 >> v1:172.21.15.170:6789/0 conn(0x7f3df0159e90 legacy=0x7f3df006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00cd4d0 con 0x7f3df006c140 2023-10-16T14:34:02.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3de80019f0 con 0x7f3df006c140 2023-10-16T14:34:02.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 >> v1:172.21.15.29:6789/0 conn(0x7f3df006c140 legacy=0x7f3df015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de0003970 0x7f3de0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de00047c0 0x7f3de0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3de00052a0 0x7f3de000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df015d890 con 0x7f3de0003970 2023-10-16T14:34:02.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de000c170 con 0x7f3de00047c0 2023-10-16T14:34:02.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.053+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de000c330 con 0x7f3de00052a0 2023-10-16T14:34:02.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df4e00640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de0003970 0x7f3de0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3deffff640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de00047c0 0x7f3de0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df4e00640 1 -- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3de00052a0 msgr2=0x7f3de000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:34:02.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df4e00640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3de00052a0 0x7f3de000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df4e00640 1 -- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de00047c0 msgr2=0x7f3de0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:34:02.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df4e00640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de00047c0 0x7f3de0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df4e00640 1 -- 172.21.15.29:0/1540028842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0003520 con 0x7f3de0003970 2023-10-16T14:34:02.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.054+0000 7f3df5601640 1 -- 172.21.15.29:0/1540028842 reap_dead start 2023-10-16T14:34:02.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.055+0000 7f3df4e00640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de0003970 0x7f3de0003d90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009a90 tx=0x7f3de00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13b6c7d83b4475a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:34:02.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.055+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de402c660 con 0x7f3de0003970 2023-10-16T14:34:02.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.055+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de402c7c0 con 0x7f3de0003970 2023-10-16T14:34:02.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.055+0000 7f3def7fe640 1 -- 172.21.15.29:0/1540028842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de402cb50 con 0x7f3de0003970 2023-10-16T14:34:02.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 -- 172.21.15.29:0/1540028842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de0003970 msgr2=0x7f3de0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de0003970 0x7f3de0003d90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009a90 tx=0x7f3de00057f0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 -- 172.21.15.29:0/1540028842 shutdown_connections 2023-10-16T14:34:02.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 --2- 172.21.15.29:0/1540028842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3de0003970 0x7f3de0003d90 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 -- 172.21.15.29:0/1540028842 >> 172.21.15.29:0/1540028842 conn(0x7f3df00ce9b0 msgr2=0x7f3df00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:34:02.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 -- 172.21.15.29:0/1540028842 shutdown_connections 2023-10-16T14:34:02.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.056+0000 7f3df7064640 1 -- 172.21.15.29:0/1540028842 wait complete. 2023-10-16T14:34:02.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.058+0000 7f3df7064640 1 Processor -- start 2023-10-16T14:34:02.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.058+0000 7f3df7064640 1 -- start start 2023-10-16T14:34:02.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3df7064640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de0003970 0x7f3df01ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3df7064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 0x7f3df00c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3df7064640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3df00c6c00 0x7f3df00c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3df7064640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00c7d20 con 0x7f3df014ded0 2023-10-16T14:34:02.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3df7064640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df01ef5c0 con 0x7f3de0003970 2023-10-16T14:34:02.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3df7064640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00c71a0 con 0x7f3df00c6c00 2023-10-16T14:34:02.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3deffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 0x7f3df00c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3deffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 0x7f3df00c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57894/0 (socket says 172.21.15.29:57894) 2023-10-16T14:34:02.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.059+0000 7f3deffff640 1 -- 172.21.15.29:0/176604667 learned_addr learned my addr 172.21.15.29:0/176604667 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:34:02.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3df4e00640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de0003970 0x7f3df01ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3deffff640 1 -- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3df00c6c00 msgr2=0x7f3df00c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3df5601640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3df00c6c00 0x7f3df00c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3deffff640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3df00c6c00 0x7f3df00c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3deffff640 1 -- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de0003970 msgr2=0x7f3df01ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3deffff640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de0003970 0x7f3df01ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3deffff640 1 -- 172.21.15.29:0/176604667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00c7480 con 0x7f3df014ded0 2023-10-16T14:34:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3df4e00640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de0003970 0x7f3df01ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:34:02.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3deffff640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 0x7f3df00c44f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc001eb0 tx=0x7f3ddc00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:34:02.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.060+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc005480 con 0x7f3df014ded0 2023-10-16T14:34:02.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.061+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ddc003610 con 0x7f3df014ded0 2023-10-16T14:34:02.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.061+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df00c7760 con 0x7f3df014ded0 2023-10-16T14:34:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.061+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc003980 con 0x7f3df014ded0 2023-10-16T14:34:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.061+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df01ef7c0 con 0x7f3df014ded0 2023-10-16T14:34:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.062+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df01ef950 con 0x7f3df014ded0 2023-10-16T14:34:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.063+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 48) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3ddc003ae0 con 0x7f3df014ded0 2023-10-16T14:34:02.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.064+0000 7f3ded7fa640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f3dd406ad10 0x7f3dd406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:34:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.064+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3ddc089370 con 0x7f3df014ded0 2023-10-16T14:34:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.065+0000 7f3df4e00640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f3dd406ad10 0x7f3dd406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:34:02.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.066+0000 7f3df4e00640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f3dd406ad10 0x7f3dd406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3de4036000 tx=0x7f3de4030000 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:34:02.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.068+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ddc054530 con 0x7f3df014ded0 2023-10-16T14:34:02.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.277+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 --> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3df00cda90 con 0x7f3dd406ad10 2023-10-16T14:34:02.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.293+0000 7f3ded7fa640 1 -- 172.21.15.29:0/176604667 <== mgr.5855 v2:172.21.15.170:6832/3649833910 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7f3de40041c0 con 0x7f3dd406ad10 2023-10-16T14:34:02.296 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:34:02.296 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:34:01", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T14:34:02.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.295+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f3dd406ad10 msgr2=0x7f3dd406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.295+0000 7f3df7064640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f3dd406ad10 0x7f3dd406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3de4036000 tx=0x7f3de4030000 comp rx=0 tx=0).stop 2023-10-16T14:34:02.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.295+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 msgr2=0x7f3df00c44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:34:02.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.295+0000 7f3df7064640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 0x7f3df00c44f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc001eb0 tx=0x7f3ddc00d9d0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 shutdown_connections 2023-10-16T14:34:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f3dd406ad10 0x7f3dd406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3df014ded0 0x7f3df00c44f0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3df00c6c00 0x7f3df00c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 --2- 172.21.15.29:0/176604667 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3de0003970 0x7f3df01ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:34:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 >> 172.21.15.29:0/176604667 conn(0x7f3df00ce9b0 msgr2=0x7f3df00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:34:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 shutdown_connections 2023-10-16T14:34:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:34:02.296+0000 7f3df7064640 1 -- 172.21.15.29:0/176604667 wait complete. 2023-10-16T14:34:02.317 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:34:01", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T14:34:17.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:34:17.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:34:48.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:34:48.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:35:00.835 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:00.832+0000 7fd7a7209640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:00.836 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:00.832+0000 7fd7a7209640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:00.836 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:00.832+0000 7fd7a7209640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:00.836 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:00.832+0000 7fd7a7209640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:00.837 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:00.832+0000 7fd7a7209640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:02.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T14:35:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.553+0000 7f12e4b28640 1 Processor -- start 2023-10-16T14:35:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.554+0000 7f12e4b28640 1 -- start start 2023-10-16T14:35:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.555+0000 7f12e4b28640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f12e00b8480 0x7f12e00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.555+0000 7f12e4b28640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f12e00d0be0 0x7f12e00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.555+0000 7f12e4b28640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e015d150 con 0x7f12e015cd70 2023-10-16T14:35:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.555+0000 7f12e4b28640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e00d1180 con 0x7f12e00d0be0 2023-10-16T14:35:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.555+0000 7f12e4b28640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e00d1460 con 0x7f12e00b8480 2023-10-16T14:35:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.555+0000 7f12dffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f12e00d0be0 0x7f12e00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12dffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f12e00d0be0 0x7f12e00cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33524/0 (socket says 172.21.15.29:33524) 2023-10-16T14:35:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12dffff640 1 -- 172.21.15.29:0/9615736 learned_addr learned my addr 172.21.15.29:0/9615736 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12deffd640 1 --2- 172.21.15.29:0/9615736 >> v2:172.21.15.170:3300/0 conn(0x7f12e00b8480 0x7f12e00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12dffff640 1 -- 172.21.15.29:0/9615736 >> v2:172.21.15.170:3300/0 conn(0x7f12e00b8480 msgr2=0x7f12e00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12dffff640 1 --2- 172.21.15.29:0/9615736 >> v2:172.21.15.170:3300/0 conn(0x7f12e00b8480 0x7f12e00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12dffff640 1 -- 172.21.15.29:0/9615736 >> v1:172.21.15.170:6789/0 conn(0x7f12e015cd70 legacy=0x7f12e00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.556+0000 7f12dffff640 1 -- 172.21.15.29:0/9615736 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00d1740 con 0x7f12e00d0be0 2023-10-16T14:35:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.557+0000 7f12dffff640 1 --2- 172.21.15.29:0/9615736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00d0be0 0x7f12e00cdd00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f12e0066a00 tx=0x7f12d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573b5e2f185ca6a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.557+0000 7f12de7fc640 1 -- 172.21.15.29:0/9615736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d80317e0 con 0x7f12e00d0be0 2023-10-16T14:35:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.557+0000 7f12de7fc640 1 -- 172.21.15.29:0/9615736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d8031940 con 0x7f12e00d0be0 2023-10-16T14:35:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12de7fc640 1 -- 172.21.15.29:0/9615736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8031cd0 con 0x7f12e00d0be0 2023-10-16T14:35:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 -- 172.21.15.29:0/9615736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00d0be0 msgr2=0x7f12e00cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 --2- 172.21.15.29:0/9615736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00d0be0 0x7f12e00cdd00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f12e0066a00 tx=0x7f12d802b880 comp rx=0 tx=0).stop 2023-10-16T14:35:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 -- 172.21.15.29:0/9615736 shutdown_connections 2023-10-16T14:35:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 --2- 172.21.15.29:0/9615736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00d0be0 0x7f12e00cdd00 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 --2- 172.21.15.29:0/9615736 >> v2:172.21.15.170:3300/0 conn(0x7f12e00b8480 0x7f12e00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 -- 172.21.15.29:0/9615736 >> 172.21.15.29:0/9615736 conn(0x7f12e00cf570 msgr2=0x7f12e00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 -- 172.21.15.29:0/9615736 shutdown_connections 2023-10-16T14:35:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.558+0000 7f12e4b28640 1 -- 172.21.15.29:0/9615736 wait complete. 2023-10-16T14:35:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.560+0000 7f12e4b28640 1 Processor -- start 2023-10-16T14:35:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.560+0000 7f12e4b28640 1 -- start start 2023-10-16T14:35:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12e4b28640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 0x7f12e015dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12e4b28640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12e00d0be0 0x7f12e01f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12e4b28640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12e015cd70 0x7f12e00cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12e4b28640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e015f1b0 con 0x7f12e00b8480 2023-10-16T14:35:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12e4b28640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e015e5a0 con 0x7f12e015cd70 2023-10-16T14:35:02.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12e4b28640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e015e880 con 0x7f12e00d0be0 2023-10-16T14:35:02.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.561+0000 7f12df7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 0x7f12e015dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 0x7f12e015dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33534/0 (socket says 172.21.15.29:33534) 2023-10-16T14:35:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 -- 172.21.15.29:0/1690524344 learned_addr learned my addr 172.21.15.29:0/1690524344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12dffff640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12e015cd70 0x7f12e00cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12deffd640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12e00d0be0 0x7f12e01f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 -- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12e00d0be0 msgr2=0x7f12e01f4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12e00d0be0 0x7f12e01f4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 -- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12e015cd70 msgr2=0x7f12e00cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12e015cd70 0x7f12e00cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 -- 172.21.15.29:0/1690524344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d80094d0 con 0x7f12e00b8480 2023-10-16T14:35:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.562+0000 7f12df7fe640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 0x7f12e015dd50 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f12d000c7c0 tx=0x7f12d000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.563+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d000aa50 con 0x7f12e00b8480 2023-10-16T14:35:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.563+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d000abb0 con 0x7f12e00b8480 2023-10-16T14:35:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.563+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0012420 con 0x7f12e00b8480 2023-10-16T14:35:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.563+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e015eb60 con 0x7f12e00b8480 2023-10-16T14:35:02.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.563+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e015e290 con 0x7f12e00b8480 2023-10-16T14:35:02.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.564+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e01535f0 con 0x7f12e00b8480 2023-10-16T14:35:02.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.565+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 50) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f12d0012580 con 0x7f12e00b8480 2023-10-16T14:35:02.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.566+0000 7f12c7fff640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f12cc06b0a0 0x7f12cc06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:02.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.566+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f12d0089a70 con 0x7f12e00b8480 2023-10-16T14:35:02.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.567+0000 7f12deffd640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f12cc06b0a0 0x7f12cc06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:02.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.568+0000 7f12deffd640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f12cc06b0a0 0x7f12cc06d560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006fd0 tx=0x7f12d4008040 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:02.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.571+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12d0054710 con 0x7f12e00b8480 2023-10-16T14:35:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.788+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 --> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12e00d1a20 con 0x7f12cc06b0a0 2023-10-16T14:35:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.811+0000 7f12c7fff640 1 -- 172.21.15.29:0/1690524344 <== mgr.5855 v2:172.21.15.170:6832/3649833910 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7f12d400f830 con 0x7f12cc06b0a0 2023-10-16T14:35:02.814 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:02.815 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:34:01", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T14:35:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f12cc06b0a0 msgr2=0x7f12cc06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f12cc06b0a0 0x7f12cc06d560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006fd0 tx=0x7f12d4008040 comp rx=0 tx=0).stop 2023-10-16T14:35:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 msgr2=0x7f12e015dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 0x7f12e015dd50 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f12d000c7c0 tx=0x7f12d000cc80 comp rx=0 tx=0).stop 2023-10-16T14:35:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 shutdown_connections 2023-10-16T14:35:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12e015cd70 0x7f12e00cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12e00d0be0 0x7f12e01f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f12cc06b0a0 0x7f12cc06d560 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 --2- 172.21.15.29:0/1690524344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12e00b8480 0x7f12e015dd50 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 >> 172.21.15.29:0/1690524344 conn(0x7f12e00cf570 msgr2=0x7f12e00d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 shutdown_connections 2023-10-16T14:35:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:02.814+0000 7f12e4b28640 1 -- 172.21.15.29:0/1690524344 wait complete. 2023-10-16T14:35:02.837 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:34:01", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T14:35:02.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-10-16T14:35:03.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.064+0000 7fb21ffff640 1 Processor -- start 2023-10-16T14:35:03.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.064+0000 7fb21ffff640 1 -- start start 2023-10-16T14:35:03.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 0x7fb2201654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22015ac60 con 0x7fb2200ad4d0 2023-10-16T14:35:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb220070030 con 0x7fb220153740 2023-10-16T14:35:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb220070310 con 0x7fb2201630e0 2023-10-16T14:35:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21e7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2200ad4d0 0x7fb2200ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50868/0 (socket says 172.21.15.29:50868) 2023-10-16T14:35:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21e7fc640 1 -- 172.21.15.29:0/1856239969 learned_addr learned my addr 172.21.15.29:0/1856239969 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:03.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095693831 0 0) 0x7fb22015ac60 con 0x7fb2200ad4d0 2023-10-16T14:35:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb208003520 con 0x7fb2200ad4d0 2023-10-16T14:35:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.066+0000 7fb21f7fe640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 0x7fb2201654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1883583529 0 0) 0x7fb208003520 con 0x7fb2200ad4d0 2023-10-16T14:35:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 msgr2=0x7fb2201654d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 0x7fb2201654d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 >> v1:172.21.15.170:6789/0 conn(0x7fb220153740 legacy=0x7fb22015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2200705f0 con 0x7fb2200ad4d0 2023-10-16T14:35:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fb214001670 con 0x7fb2200ad4d0 2023-10-16T14:35:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 >> v1:172.21.15.29:6789/0 conn(0x7fb2200ad4d0 legacy=0x7fb2200ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb208003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2080047c0 0x7fb208004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 0x7fb208005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb220070310 con 0x7fb2080039a0 2023-10-16T14:35:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20800bb40 con 0x7fb2080047c0 2023-10-16T14:35:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20800bd00 con 0x7fb2201630e0 2023-10-16T14:35:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.067+0000 7fb21effd640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb208003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21f7fe640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 0x7fb208005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21effd640 1 -- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 msgr2=0x7fb208005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21effd640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb2201630e0 0x7fb208005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21e7fc640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2080047c0 0x7fb208004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21effd640 1 -- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2080047c0 msgr2=0x7fb208004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21effd640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2080047c0 0x7fb208004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.068+0000 7fb21effd640 1 -- 172.21.15.29:0/1856239969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb208003520 con 0x7fb2080039a0 2023-10-16T14:35:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.069+0000 7fb21effd640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb208003dc0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb210001750 tx=0x7fb2080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3028b976c477b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.069+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100052d0 con 0x7fb2080039a0 2023-10-16T14:35:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.069+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21002e050 con 0x7fb2080039a0 2023-10-16T14:35:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.069+0000 7fb21dffb640 1 -- 172.21.15.29:0/1856239969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100035f0 con 0x7fb2080039a0 2023-10-16T14:35:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21ffff640 1 -- 172.21.15.29:0/1856239969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 msgr2=0x7fb208003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1856239969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb208003dc0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb210001750 tx=0x7fb2080057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21f7fe640 1 -- 172.21.15.29:0/1856239969 reap_dead start 2023-10-16T14:35:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21ffff640 1 -- 172.21.15.29:0/1856239969 shutdown_connections 2023-10-16T14:35:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21ffff640 1 -- 172.21.15.29:0/1856239969 >> 172.21.15.29:0/1856239969 conn(0x7fb2200ce6a0 msgr2=0x7fb2200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21ffff640 1 -- 172.21.15.29:0/1856239969 shutdown_connections 2023-10-16T14:35:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.070+0000 7fb21ffff640 1 -- 172.21.15.29:0/1856239969 wait complete. 2023-10-16T14:35:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.071+0000 7fb21ffff640 1 Processor -- start 2023-10-16T14:35:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.072+0000 7fb21ffff640 1 -- start start 2023-10-16T14:35:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.072+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb220151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb220151030 0x7fb2201f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2201f2960 0x7fb22014bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200c21d0 con 0x7fb2080039a0 2023-10-16T14:35:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200c3ba0 con 0x7fb2201f2960 2023-10-16T14:35:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200c3e20 con 0x7fb220151030 2023-10-16T14:35:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb220151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb220151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33548/0 (socket says 172.21.15.29:33548) 2023-10-16T14:35:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 -- 172.21.15.29:0/1198953522 learned_addr learned my addr 172.21.15.29:0/1198953522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 -- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb220151030 msgr2=0x7fb2201f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb220151030 0x7fb2201f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 -- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2201f2960 msgr2=0x7fb22014bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2201f2960 0x7fb22014bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.073+0000 7fb21effd640 1 -- 172.21.15.29:0/1198953522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2200c2700 con 0x7fb2080039a0 2023-10-16T14:35:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.074+0000 7fb21effd640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb220151620 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb210008fd0 tx=0x7fb210001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:03.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.074+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100076c0 con 0x7fb2080039a0 2023-10-16T14:35:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.074+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb210007a30 con 0x7fb2080039a0 2023-10-16T14:35:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.074+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210006c00 con 0x7fb2080039a0 2023-10-16T14:35:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.074+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2200c2420 con 0x7fb2080039a0 2023-10-16T14:35:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.074+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2200afba0 con 0x7fb2080039a0 2023-10-16T14:35:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.075+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2200c2150 con 0x7fb2080039a0 2023-10-16T14:35:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.076+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 50) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fb2100035f0 con 0x7fb2080039a0 2023-10-16T14:35:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.077+0000 7fb207fff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb20c06b0c0 0x7fb20c06d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.078+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fb2100abc20 con 0x7fb2080039a0 2023-10-16T14:35:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.078+0000 7fb21e7fc640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb20c06b0c0 0x7fb20c06d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.079+0000 7fb21e7fc640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb20c06b0c0 0x7fb20c06d580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb214000ff0 tx=0x7fb214009040 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.082+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb210076830 con 0x7fb2080039a0 2023-10-16T14:35:03.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.304+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 --> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2200708d0 con 0x7fb20c06b0c0 2023-10-16T14:35:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.424+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 51) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb210076c00 con 0x7fb2080039a0 2023-10-16T14:35:03.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.480+0000 7fb207fff640 1 -- 172.21.15.29:0/1198953522 <== mgr.5855 v2:172.21.15.170:6832/3649833910 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fb214001ce0 con 0x7fb20c06b0c0 2023-10-16T14:35:03.483 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /bad-path 2023-10-16T14:35:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb20c06b0c0 msgr2=0x7fb20c06d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb20c06b0c0 0x7fb20c06d580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb214000ff0 tx=0x7fb214009040 comp rx=0 tx=0).stop 2023-10-16T14:35:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 msgr2=0x7fb220151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb220151620 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb210008fd0 tx=0x7fb210001120 comp rx=0 tx=0).stop 2023-10-16T14:35:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 shutdown_connections 2023-10-16T14:35:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2201f2960 0x7fb22014bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb220151030 0x7fb2201f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb20c06b0c0 0x7fb20c06d580 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 --2- 172.21.15.29:0/1198953522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2080039a0 0x7fb220151620 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 >> 172.21.15.29:0/1198953522 conn(0x7fb2200ce6a0 msgr2=0x7fb2201589f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:03.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 shutdown_connections 2023-10-16T14:35:03.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.484+0000 7fb21ffff640 1 -- 172.21.15.29:0/1198953522 wait complete. 2023-10-16T14:35:03.507 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-16T14:35:03.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:35:03.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.737+0000 7f9e611e4640 1 Processor -- start 2023-10-16T14:35:03.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.737+0000 7f9e611e4640 1 -- start start 2023-10-16T14:35:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e611e4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e5c0ca0a0 0x7f9e5c06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e611e4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c15d910 con 0x7f9e5c070b60 2023-10-16T14:35:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e611e4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c15dbf0 con 0x7f9e5c159950 2023-10-16T14:35:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e611e4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c15e490 con 0x7f9e5c0ca0a0 2023-10-16T14:35:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e609e3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e5c0ca0a0 0x7f9e5c06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e609e3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e5c0ca0a0 0x7f9e5c06c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33562/0 (socket says 172.21.15.29:33562) 2023-10-16T14:35:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.739+0000 7f9e609e3640 1 -- 172.21.15.29:0/3642152680 learned_addr learned my addr 172.21.15.29:0/3642152680 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:03.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.740+0000 7f9e609e3640 1 -- 172.21.15.29:0/3642152680 >> v1:172.21.15.170:6789/0 conn(0x7f9e5c159950 legacy=0x7f9e5c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:03.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.740+0000 7f9e609e3640 1 -- 172.21.15.29:0/3642152680 >> v1:172.21.15.29:6789/0 conn(0x7f9e5c070b60 legacy=0x7f9e5c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.740+0000 7f9e609e3640 1 -- 172.21.15.29:0/3642152680 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c15ded0 con 0x7f9e5c0ca0a0 2023-10-16T14:35:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.741+0000 7f9e609e3640 1 --2- 172.21.15.29:0/3642152680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c0ca0a0 0x7f9e5c06c810 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009e50 tx=0x7f9e5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea631be07815b42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.741+0000 7f9e5affd640 1 -- 172.21.15.29:0/3642152680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5402ca80 con 0x7f9e5c0ca0a0 2023-10-16T14:35:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.741+0000 7f9e5affd640 1 -- 172.21.15.29:0/3642152680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5402cbe0 con 0x7f9e5c0ca0a0 2023-10-16T14:35:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.741+0000 7f9e5affd640 1 -- 172.21.15.29:0/3642152680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e540343f0 con 0x7f9e5c0ca0a0 2023-10-16T14:35:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 -- 172.21.15.29:0/3642152680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c0ca0a0 msgr2=0x7f9e5c06c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 --2- 172.21.15.29:0/3642152680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c0ca0a0 0x7f9e5c06c810 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009e50 tx=0x7f9e5402bee0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 -- 172.21.15.29:0/3642152680 shutdown_connections 2023-10-16T14:35:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 --2- 172.21.15.29:0/3642152680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c0ca0a0 0x7f9e5c06c810 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 -- 172.21.15.29:0/3642152680 >> 172.21.15.29:0/3642152680 conn(0x7f9e5c0ce6a0 msgr2=0x7f9e5c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 -- 172.21.15.29:0/3642152680 shutdown_connections 2023-10-16T14:35:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.742+0000 7f9e611e4640 1 -- 172.21.15.29:0/3642152680 wait complete. 2023-10-16T14:35:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.745+0000 7f9e611e4640 1 Processor -- start 2023-10-16T14:35:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.745+0000 7f9e611e4640 1 -- start start 2023-10-16T14:35:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.746+0000 7f9e611e4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e5c070b60 0x7f9e5c155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.746+0000 7f9e611e4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e5c0ca0a0 0x7f9e5c0af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.746+0000 7f9e611e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 0x7f9e5c1562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.746+0000 7f9e611e4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0c4d50 con 0x7f9e5c159950 2023-10-16T14:35:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.746+0000 7f9e611e4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0c5030 con 0x7f9e5c070b60 2023-10-16T14:35:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.746+0000 7f9e611e4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c1557f0 con 0x7f9e5c0ca0a0 2023-10-16T14:35:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 0x7f9e5c1562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 0x7f9e5c1562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33578/0 (socket says 172.21.15.29:33578) 2023-10-16T14:35:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 -- 172.21.15.29:0/4124937957 learned_addr learned my addr 172.21.15.29:0/4124937957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e5b7fe640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e5c0ca0a0 0x7f9e5c0af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e5bfff640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e5c070b60 0x7f9e5c155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 -- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e5c0ca0a0 msgr2=0x7f9e5c0af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e5c0ca0a0 0x7f9e5c0af790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 -- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e5c070b60 msgr2=0x7f9e5c155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e5c070b60 0x7f9e5c155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.747+0000 7f9e609e3640 1 -- 172.21.15.29:0/4124937957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009b30 con 0x7f9e5c159950 2023-10-16T14:35:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.748+0000 7f9e609e3640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 0x7f9e5c1562b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f9e54036040 tx=0x7f9e54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.748+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5403e450 con 0x7f9e5c159950 2023-10-16T14:35:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.748+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5403e5d0 con 0x7f9e5c159950 2023-10-16T14:35:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.748+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c155ad0 con 0x7f9e5c159950 2023-10-16T14:35:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.748+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5403e980 con 0x7f9e5c159950 2023-10-16T14:35:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.748+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c0afcd0 con 0x7f9e5c159950 2023-10-16T14:35:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.749+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c04e9b0 con 0x7f9e5c159950 2023-10-16T14:35:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.750+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 51) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9e5402cd50 con 0x7f9e5c159950 2023-10-16T14:35:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.751+0000 7f9e58ff9640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f9e4806b100 0x7f9e4806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.751+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f9e540ac840 con 0x7f9e5c159950 2023-10-16T14:35:03.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.752+0000 7f9e5bfff640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f9e4806b100 0x7f9e4806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:03.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.753+0000 7f9e5bfff640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f9e4806b100 0x7f9e4806d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c004ea0 tx=0x7f9e4c009040 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:03.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:03.756+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9e540775d0 con 0x7f9e5c159950 2023-10-16T14:35:04.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.000+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f9e5c0b4620 con 0x7f9e5c159950 2023-10-16T14:35:04.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.416+0000 7f9e58ff9640 1 -- 172.21.15.29:0/4124937957 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v52) v1 ==== 93+0+0 (secure 0 0 0) 0x7f9e5407abf0 con 0x7f9e5c159950 2023-10-16T14:35:04.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f9e4806b100 msgr2=0x7f9e4806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:04.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f9e4806b100 0x7f9e4806d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c004ea0 tx=0x7f9e4c009040 comp rx=0 tx=0).stop 2023-10-16T14:35:04.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 msgr2=0x7f9e5c1562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:04.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 0x7f9e5c1562b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f9e54036040 tx=0x7f9e54004a40 comp rx=0 tx=0).stop 2023-10-16T14:35:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 shutdown_connections 2023-10-16T14:35:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f9e4806b100 0x7f9e4806d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e5c159950 0x7f9e5c1562b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e5c0ca0a0 0x7f9e5c0af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 --2- 172.21.15.29:0/4124937957 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e5c070b60 0x7f9e5c155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.420+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 >> 172.21.15.29:0/4124937957 conn(0x7f9e5c0ce6a0 msgr2=0x7f9e5c070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.421+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 shutdown_connections 2023-10-16T14:35:04.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.421+0000 7f9e611e4640 1 -- 172.21.15.29:0/4124937957 wait complete. 2023-10-16T14:35:04.448 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:35:04.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:35:04.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:35:04.660 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:35:04.661 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:35:04.661 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:04.719 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:35:04.719 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:04.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:04.942 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:04.940+0000 7f038e630040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:35:04.962 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:04.960+0000 7fc456a26040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:35:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.973+0000 7f1e6fd3c640 1 Processor -- start 2023-10-16T14:35:04.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.973+0000 7f1e6fd3c640 1 -- start start 2023-10-16T14:35:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.975+0000 7f1e6fd3c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e680b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.975+0000 7f1e6fd3c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1e6815b0c0 0x7f1e6815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.975+0000 7f1e6fd3c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e680ce2d0 con 0x7f1e6806fa90 2023-10-16T14:35:04.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.975+0000 7f1e6fd3c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e680b63d0 con 0x7f1e6815b0c0 2023-10-16T14:35:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.975+0000 7f1e6fd3c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e680b66b0 con 0x7f1e6815ace0 2023-10-16T14:35:04.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.976+0000 7f1e6d2d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1e6815b0c0 0x7f1e6815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.976+0000 7f1e6e2d9640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f1e6806fa90 0x7f1e680cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:42316/0 (socket says 172.21.15.29:42316) 2023-10-16T14:35:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.976+0000 7f1e6e2d9640 1 -- 172.21.15.29:0/4231419690 learned_addr learned my addr 172.21.15.29:0/4231419690 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.976+0000 7f1e6dad8640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e680b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:04.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.977+0000 7f1e6d2d7640 1 -- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 msgr2=0x7f1e680b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:04.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.977+0000 7f1e6d2d7640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e680b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.977+0000 7f1e6d2d7640 1 -- 172.21.15.29:0/4231419690 >> v1:172.21.15.170:6789/0 conn(0x7f1e6806fa90 legacy=0x7f1e680cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:04.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.977+0000 7f1e6d2d7640 1 -- 172.21.15.29:0/4231419690 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e680b6990 con 0x7f1e6815b0c0 2023-10-16T14:35:04.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.977+0000 7f1e6dad8640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e680b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:35:04.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.978+0000 7f1e6d2d7640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6815b0c0 0x7f1e6815baa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1e580097a0 tx=0x7f1e5802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64c075ff9c2f985 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:04.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6cad6640 1 -- 172.21.15.29:0/4231419690 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5802e070 con 0x7f1e6815b0c0 2023-10-16T14:35:04.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6cad6640 1 -- 172.21.15.29:0/4231419690 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e580315a0 con 0x7f1e6815b0c0 2023-10-16T14:35:04.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6cad6640 1 -- 172.21.15.29:0/4231419690 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58031930 con 0x7f1e6815b0c0 2023-10-16T14:35:04.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6815b0c0 msgr2=0x7f1e6815baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:04.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6815b0c0 0x7f1e6815baa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1e580097a0 tx=0x7f1e5802b840 comp rx=0 tx=0).stop 2023-10-16T14:35:04.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/4231419690 shutdown_connections 2023-10-16T14:35:04.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6815b0c0 0x7f1e6815baa0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/4231419690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e680b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/4231419690 >> 172.21.15.29:0/4231419690 conn(0x7f1e6806d700 msgr2=0x7f1e6806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:04.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.979+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/4231419690 shutdown_connections 2023-10-16T14:35:04.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.980+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/4231419690 wait complete. 2023-10-16T14:35:04.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.981+0000 7f1e6fd3c640 1 Processor -- start 2023-10-16T14:35:04.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.982+0000 7f1e6fd3c640 1 -- start start 2023-10-16T14:35:04.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6fd3c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6806fa90 0x7f1e680c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:04.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6fd3c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e6806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:04.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6fd3c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 0x7f1e68150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:04.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6fd3c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6806c980 con 0x7f1e6815b0c0 2023-10-16T14:35:04.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6fd3c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6806b8e0 con 0x7f1e6806fa90 2023-10-16T14:35:04.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6fd3c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6806bbc0 con 0x7f1e6815ace0 2023-10-16T14:35:04.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6e2d9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 0x7f1e68150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:04.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6dad8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6806fa90 0x7f1e680c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:04.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6e2d9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 0x7f1e68150f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35808/0 (socket says 172.21.15.29:35808) 2023-10-16T14:35:04.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.983+0000 7f1e6e2d9640 1 -- 172.21.15.29:0/2394814638 learned_addr learned my addr 172.21.15.29:0/2394814638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:04.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6d2d7640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e6806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:04.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6e2d9640 1 -- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 msgr2=0x7f1e6806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:04.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6e2d9640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e6806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6e2d9640 1 -- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6806fa90 msgr2=0x7f1e680c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:04.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6e2d9640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6806fa90 0x7f1e680c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:04.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6e2d9640 1 -- 172.21.15.29:0/2394814638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64009400 con 0x7f1e6815b0c0 2023-10-16T14:35:04.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6d2d7640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e6806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:04.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6e2d9640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 0x7f1e68150f90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000ca80 tx=0x7f1e6000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:04.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6000ac30 con 0x7f1e6815b0c0 2023-10-16T14:35:04.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6000ad90 con 0x7f1e6815b0c0 2023-10-16T14:35:04.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60012600 con 0x7f1e6815b0c0 2023-10-16T14:35:04.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.984+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e58009400 con 0x7f1e6815b0c0 2023-10-16T14:35:04.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.985+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6806c180 con 0x7f1e6815b0c0 2023-10-16T14:35:04.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.986+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6804e160 con 0x7f1e6815b0c0 2023-10-16T14:35:04.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.987+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1e60007070 con 0x7f1e6815b0c0 2023-10-16T14:35:04.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.988+0000 7f1e5e7fc640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 0x7f1e4c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:04.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.988+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f1e60089a80 con 0x7f1e6815b0c0 2023-10-16T14:35:04.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.988+0000 7f1e6dad8640 1 -- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 msgr2=0x7f1e4c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:04.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.988+0000 7f1e6dad8640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 0x7f1e4c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:04.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:04.992+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1e60054660 con 0x7f1e6815b0c0 2023-10-16T14:35:04.995 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:04.993+0000 7f038e630040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:35:05.020 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.018+0000 7fc456a26040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:35:05.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.187+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e680736e0 con 0x7f1e6815b0c0 2023-10-16T14:35:05.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.188+0000 7f1e5e7fc640 1 -- 172.21.15.29:0/2394814638 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+17129 (secure 0 0 0) 0x7f1e60017020 con 0x7f1e6815b0c0 2023-10-16T14:35:05.192 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:05.192 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:02.383554+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.26","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2765694912":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/3476448138":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2716588410":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/261609677":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6833/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2811499332":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6832/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1477766173":"2023-10-17T14:34:00.590880+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:05.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.189+0000 7f1e6dad8640 1 -- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 msgr2=0x7f1e4c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:05.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.189+0000 7f1e6dad8640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 0x7f1e4c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 msgr2=0x7f1e4c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 0x7f1e4c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 msgr2=0x7f1e68150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 0x7f1e68150f90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000ca80 tx=0x7f1e6000cf40 comp rx=0 tx=0).stop 2023-10-16T14:35:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 shutdown_connections 2023-10-16T14:35:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f1e4c06b0b0 0x7f1e4c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e6815b0c0 0x7f1e68150f90 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e6815ace0 0x7f1e6806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 --2- 172.21.15.29:0/2394814638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e6806fa90 0x7f1e680c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 >> 172.21.15.29:0/2394814638 conn(0x7f1e6806d700 msgr2=0x7f1e6815d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 shutdown_connections 2023-10-16T14:35:05.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.191+0000 7f1e6fd3c640 1 -- 172.21.15.29:0/2394814638 wait complete. 2023-10-16T14:35:05.212 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:33:43.719274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:33:44.790872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:35:05.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:35:05.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.433+0000 7f58593a9640 1 Processor -- start 2023-10-16T14:35:05.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.433+0000 7f58593a9640 1 -- start start 2023-10-16T14:35:05.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58593a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f585415d030 0x7f58540d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58593a9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f585415d410 0x7f5854161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58593a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58540b5d90 con 0x7f5854161990 2023-10-16T14:35:05.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58593a9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58540d2b50 con 0x7f585415d410 2023-10-16T14:35:05.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58593a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5854160fc0 con 0x7f585415d030 2023-10-16T14:35:05.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58537fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5854161990 0x7f58540cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45550/0 (socket says 172.21.15.29:45550) 2023-10-16T14:35:05.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f58537fe640 1 -- 172.21.15.29:0/3387006138 learned_addr learned my addr 172.21.15.29:0/3387006138 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:05.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.435+0000 7f5852ffd640 1 --2- 172.21.15.29:0/3387006138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f585415d030 0x7f58540d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.436+0000 7f58527fc640 1 --2- 172.21.15.29:0/3387006138 >> v2:172.21.15.170:3300/0 conn(0x7f585415d410 0x7f5854161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.436+0000 7f5852ffd640 1 -- 172.21.15.29:0/3387006138 >> v2:172.21.15.170:3300/0 conn(0x7f585415d410 msgr2=0x7f5854161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:05.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.436+0000 7f5852ffd640 1 --2- 172.21.15.29:0/3387006138 >> v2:172.21.15.170:3300/0 conn(0x7f585415d410 0x7f5854161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.436+0000 7f5852ffd640 1 -- 172.21.15.29:0/3387006138 >> v1:172.21.15.29:6789/0 conn(0x7f5854161990 legacy=0x7f58540cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.436+0000 7f5852ffd640 1 -- 172.21.15.29:0/3387006138 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58540d25d0 con 0x7f585415d030 2023-10-16T14:35:05.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.438+0000 7f5852ffd640 1 --2- 172.21.15.29:0/3387006138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f585415d030 0x7f58540d1630 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f58440097a0 tx=0x7f584402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13d21c4bbf3767f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:05.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.439+0000 7f5851ffb640 1 -- 172.21.15.29:0/3387006138 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844030800 con 0x7f585415d030 2023-10-16T14:35:05.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.439+0000 7f5851ffb640 1 -- 172.21.15.29:0/3387006138 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5844030960 con 0x7f585415d030 2023-10-16T14:35:05.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.439+0000 7f5851ffb640 1 -- 172.21.15.29:0/3387006138 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844030cf0 con 0x7f585415d030 2023-10-16T14:35:05.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.439+0000 7f58593a9640 1 -- 172.21.15.29:0/3387006138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f585415d030 msgr2=0x7f58540d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.439+0000 7f58593a9640 1 --2- 172.21.15.29:0/3387006138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f585415d030 0x7f58540d1630 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f58440097a0 tx=0x7f584402b940 comp rx=0 tx=0).stop 2023-10-16T14:35:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.440+0000 7f58593a9640 1 -- 172.21.15.29:0/3387006138 shutdown_connections 2023-10-16T14:35:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.440+0000 7f58593a9640 1 --2- 172.21.15.29:0/3387006138 >> v2:172.21.15.170:3300/0 conn(0x7f585415d410 0x7f5854161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.440+0000 7f58593a9640 1 --2- 172.21.15.29:0/3387006138 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f585415d030 0x7f58540d1630 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.440+0000 7f58593a9640 1 -- 172.21.15.29:0/3387006138 >> 172.21.15.29:0/3387006138 conn(0x7f58540cf720 msgr2=0x7f58540cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.440+0000 7f58593a9640 1 -- 172.21.15.29:0/3387006138 shutdown_connections 2023-10-16T14:35:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.440+0000 7f58593a9640 1 -- 172.21.15.29:0/3387006138 wait complete. 2023-10-16T14:35:05.445 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.441+0000 7f038e630040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:35:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.442+0000 7f58593a9640 1 Processor -- start 2023-10-16T14:35:05.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.443+0000 7f58593a9640 1 -- start start 2023-10-16T14:35:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58593a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f585415d030 0x7f58541501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58593a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 0x7f58540cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58593a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5854161990 0x7f58540cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58593a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58540c96a0 con 0x7f585415d410 2023-10-16T14:35:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58593a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58540c67e0 con 0x7f585415d030 2023-10-16T14:35:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58593a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58540c6ac0 con 0x7f5854161990 2023-10-16T14:35:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58527fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 0x7f58540cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58527fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 0x7f58540cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35812/0 (socket says 172.21.15.29:35812) 2023-10-16T14:35:05.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.444+0000 7f58527fc640 1 -- 172.21.15.29:0/3583794867 learned_addr learned my addr 172.21.15.29:0/3583794867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:05.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f5852ffd640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f585415d030 0x7f58541501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58537fe640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5854161990 0x7f58540cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58527fc640 1 -- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5854161990 msgr2=0x7f58540cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58527fc640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5854161990 0x7f58540cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58527fc640 1 -- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f585415d030 msgr2=0x7f58541501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58527fc640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f585415d030 0x7f58541501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58527fc640 1 -- 172.21.15.29:0/3583794867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5844009400 con 0x7f585415d410 2023-10-16T14:35:05.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f5852ffd640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f585415d030 0x7f58541501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:05.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.445+0000 7f58527fc640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 0x7f58540cbeb0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9b0 tx=0x7f584800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.446+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848009670 con 0x7f585415d410 2023-10-16T14:35:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.446+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584800c9a0 con 0x7f585415d410 2023-10-16T14:35:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.446+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800cd50 con 0x7f585415d410 2023-10-16T14:35:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.446+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58540c6da0 con 0x7f585415d410 2023-10-16T14:35:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.446+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5854150710 con 0x7f585415d410 2023-10-16T14:35:05.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.447+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585404e160 con 0x7f585415d410 2023-10-16T14:35:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.448+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f5848013470 con 0x7f585415d410 2023-10-16T14:35:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.449+0000 7f583b7fe640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f584006b0b0 0x7f584006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.450+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f5848089ec0 con 0x7f585415d410 2023-10-16T14:35:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.450+0000 7f5852ffd640 1 -- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f584006b0b0 msgr2=0x7f584006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:05.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.450+0000 7f5852ffd640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f584006b0b0 0x7f584006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.454+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58480549f0 con 0x7f585415d410 2023-10-16T14:35:05.497 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.494+0000 7f038e630040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:35:05.502 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.500+0000 7fc456a26040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:35:05.546 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.543+0000 7f038e630040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:35:05.557 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.554+0000 7fc456a26040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:35:05.606 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.603+0000 7f038e630040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:35:05.609 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.607+0000 7fc456a26040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:35:05.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.643+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58540c9aa0 con 0x7f585415d410 2023-10-16T14:35:05.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.644+0000 7f583b7fe640 1 -- 172.21.15.29:0/3583794867 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58480170a0 con 0x7f585415d410 2023-10-16T14:35:05.647 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:35:05.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f584006b0b0 msgr2=0x7f584006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:05.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f584006b0b0 0x7f584006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 msgr2=0x7f58540cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 0x7f58540cbeb0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9b0 tx=0x7f584800ee70 comp rx=0 tx=0).stop 2023-10-16T14:35:05.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 shutdown_connections 2023-10-16T14:35:05.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f584006b0b0 0x7f584006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f585415d410 0x7f58540cbeb0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5854161990 0x7f58540cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 --2- 172.21.15.29:0/3583794867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f585415d030 0x7f58541501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 >> 172.21.15.29:0/3583794867 conn(0x7f58540cf720 msgr2=0x7f58540b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:05.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 shutdown_connections 2023-10-16T14:35:05.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.647+0000 7f58593a9640 1 -- 172.21.15.29:0/3583794867 wait complete. 2023-10-16T14:35:05.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.652+0000 7f038e630040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:35:05.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:35:05.676 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.673+0000 7fc456a26040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:35:05.728 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.725+0000 7fc456a26040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:35:05.763 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.761+0000 7f038e630040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:35:05.825 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.822+0000 7f038e630040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:05.846 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.843+0000 7fc456a26040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:35:05.873 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.871+0000 7f038e630040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:35:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.884+0000 7f7aaa032640 1 Processor -- start 2023-10-16T14:35:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.884+0000 7f7aaa032640 1 -- start start 2023-10-16T14:35:05.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7aaa032640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7aa415ad70 0x7f7aa4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7aaa032640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7aa406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7aaa032640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa40cb030 con 0x7f7aa40c9ca0 2023-10-16T14:35:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7aaa032640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa40c91c0 con 0x7f7aa415ad70 2023-10-16T14:35:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7aaa032640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c94a0 con 0x7f7aa40caac0 2023-10-16T14:35:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7a9bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7aa40c9ca0 0x7f7aa40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45564/0 (socket says 172.21.15.29:45564) 2023-10-16T14:35:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.886+0000 7f7a9bfff640 1 -- 172.21.15.29:0/3487494602 learned_addr learned my addr 172.21.15.29:0/3487494602 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa37fe640 1 --2- 172.21.15.29:0/3487494602 >> v2:172.21.15.170:3300/0 conn(0x7f7aa415ad70 0x7f7aa4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7aa406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613767086 0 0) 0x7f7aa40cb030 con 0x7f7aa40c9ca0 2023-10-16T14:35:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a90003520 con 0x7f7aa40c9ca0 2023-10-16T14:35:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2359152506 0 0) 0x7f7a90003520 con 0x7f7aa40c9ca0 2023-10-16T14:35:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 msgr2=0x7f7aa406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7aa406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 >> v2:172.21.15.170:3300/0 conn(0x7f7aa415ad70 msgr2=0x7f7aa4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 --2- 172.21.15.29:0/3487494602 >> v2:172.21.15.170:3300/0 conn(0x7f7aa415ad70 0x7f7aa4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40c9780 con 0x7f7aa40c9ca0 2023-10-16T14:35:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7a8c001670 con 0x7f7aa40c9ca0 2023-10-16T14:35:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 >> v1:172.21.15.29:6789/0 conn(0x7f7aa40c9ca0 legacy=0x7f7aa40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7a90003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.887+0000 7f7aa2ffd640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a90003ef0 0x7f7a900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa2ffd640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7a90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c91c0 con 0x7f7a90004d00 2023-10-16T14:35:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c94a0 con 0x7f7a90003ef0 2023-10-16T14:35:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9000b980 con 0x7f7aa40caac0 2023-10-16T14:35:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7a90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7a9bfff640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a90003ef0 0x7f7a900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa37fe640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7a90003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa3fff640 1 -- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 msgr2=0x7f7a90003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7a90003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa3fff640 1 -- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a90003ef0 msgr2=0x7f7a900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a90003ef0 0x7f7a900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa3fff640 1 -- 172.21.15.29:0/3487494602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90003520 con 0x7f7a90004d00 2023-10-16T14:35:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.888+0000 7f7aa37fe640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40caac0 0x7f7a90003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.890+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7a90005180 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009fc0 tx=0x7f7a900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5c356f96013a2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.890+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c02cc00 con 0x7f7a90004d00 2023-10-16T14:35:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.890+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c00f040 con 0x7f7a90004d00 2023-10-16T14:35:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.890+0000 7f7aa2ffd640 1 -- 172.21.15.29:0/3487494602 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c034440 con 0x7f7a90004d00 2023-10-16T14:35:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aaa032640 1 -- 172.21.15.29:0/3487494602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 msgr2=0x7f7a90005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aaa032640 1 --2- 172.21.15.29:0/3487494602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7a90005180 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009fc0 tx=0x7f7a900057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aa3fff640 1 -- 172.21.15.29:0/3487494602 reap_dead start 2023-10-16T14:35:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aaa032640 1 -- 172.21.15.29:0/3487494602 shutdown_connections 2023-10-16T14:35:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aaa032640 1 -- 172.21.15.29:0/3487494602 >> 172.21.15.29:0/3487494602 conn(0x7f7aa40ce9b0 msgr2=0x7f7aa40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:05.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aaa032640 1 -- 172.21.15.29:0/3487494602 shutdown_connections 2023-10-16T14:35:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.891+0000 7f7aaa032640 1 -- 172.21.15.29:0/3487494602 wait complete. 2023-10-16T14:35:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.893+0000 7f7aaa032640 1 Processor -- start 2023-10-16T14:35:05.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.893+0000 7f7aaa032640 1 -- start start 2023-10-16T14:35:05.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aaa032640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7aa4159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aaa032640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7aa40cd2b0 0x7f7aa40cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aaa032640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40cdf90 0x7f7aa40b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aaa032640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40cc580 con 0x7f7a90004d00 2023-10-16T14:35:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aaa032640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40cce20 con 0x7f7aa40cd2b0 2023-10-16T14:35:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aaa032640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40ce430 con 0x7f7aa40cdf90 2023-10-16T14:35:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.894+0000 7f7aa37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7aa4159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7aa4159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35822/0 (socket says 172.21.15.29:35822) 2023-10-16T14:35:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 -- 172.21.15.29:0/157017161 learned_addr learned my addr 172.21.15.29:0/157017161 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40cdf90 0x7f7aa40b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7a9bfff640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7aa40cd2b0 0x7f7aa40cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 -- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40cdf90 msgr2=0x7f7aa40b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40cdf90 0x7f7aa40b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 -- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7aa40cd2b0 msgr2=0x7f7aa40cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7aa40cd2b0 0x7f7aa40cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa37fe640 1 -- 172.21.15.29:0/157017161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40cc7d0 con 0x7f7a90004d00 2023-10-16T14:35:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7aa3fff640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40cdf90 0x7f7aa40b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.895+0000 7f7a9bfff640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7aa40cd2b0 0x7f7aa40cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.896+0000 7f7aa37fe640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7aa4159880 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7a94010a00 tx=0x7f7a94010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.896+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400ec20 con 0x7f7a90004d00 2023-10-16T14:35:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.896+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9400ed80 con 0x7f7a90004d00 2023-10-16T14:35:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.896+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40ccab0 con 0x7f7a90004d00 2023-10-16T14:35:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.896+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9401c620 con 0x7f7a90004d00 2023-10-16T14:35:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.896+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa40c4aa0 con 0x7f7a90004d00 2023-10-16T14:35:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.897+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa404e160 con 0x7f7a90004d00 2023-10-16T14:35:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.899+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f7a9401c780 con 0x7f7a90004d00 2023-10-16T14:35:05.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.900+0000 7f7aa0ff9640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f7a8406b0b0 0x7f7a8406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:05.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.900+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f7a9408a350 con 0x7f7a90004d00 2023-10-16T14:35:05.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.901+0000 7f7a9bfff640 1 -- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f7a8406b0b0 msgr2=0x7f7a8406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:05.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.901+0000 7f7a9bfff640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f7a8406b0b0 0x7f7a8406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:05.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:05.905+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a94054f30 con 0x7f7a90004d00 2023-10-16T14:35:05.916 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.914+0000 7fc456a26040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:05.931 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.928+0000 7f038e630040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:35:05.969 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:05.966+0000 7fc456a26040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:35:05.980 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:05.978+0000 7f038e630040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:35:06.029 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.026+0000 7fc456a26040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:35:06.081 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.079+0000 7fc456a26040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:35:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.094+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7aa415e310 con 0x7f7a90004d00 2023-10-16T14:35:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.095+0000 7f7aa0ff9640 1 -- 172.21.15.29:0/157017161 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a94058550 con 0x7f7a90004d00 2023-10-16T14:35:06.098 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:35:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.097+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f7a8406b0b0 msgr2=0x7f7a8406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.097+0000 7f7aaa032640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f7a8406b0b0 0x7f7a8406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.097+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 msgr2=0x7f7aa4159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.097+0000 7f7aaa032640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7aa4159880 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7a94010a00 tx=0x7f7a94010ec0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 shutdown_connections 2023-10-16T14:35:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7aa40cdf90 0x7f7aa40b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7aa40cd2b0 0x7f7aa40cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7f7a8406b0b0 0x7f7a8406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 --2- 172.21.15.29:0/157017161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a90004d00 0x7f7aa4159880 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 >> 172.21.15.29:0/157017161 conn(0x7f7aa40ce9b0 msgr2=0x7f7aa40af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 shutdown_connections 2023-10-16T14:35:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.098+0000 7f7aaa032640 1 -- 172.21.15.29:0/157017161 wait complete. 2023-10-16T14:35:06.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:35:06.357 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:06.354+0000 7f038e630040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:35:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.376+0000 7fea50f4c640 1 Processor -- start 2023-10-16T14:35:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.377+0000 7fea50f4c640 1 -- start start 2023-10-16T14:35:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea50f4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea4c15d150 0x7fea4c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea50f4c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fea4c160b90 0x7fea4c160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea50f4c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c0ac890 con 0x7fea4c069da0 2023-10-16T14:35:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea50f4c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4c15d530 con 0x7fea4c15d150 2023-10-16T14:35:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea50f4c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4c161650 con 0x7fea4c160b90 2023-10-16T14:35:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea4a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea4c15d150 0x7fea4c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea4ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fea4c069da0 0x7fea4c0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45580/0 (socket says 172.21.15.29:45580) 2023-10-16T14:35:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea4ad9d640 1 -- 172.21.15.29:0/808283657 learned_addr learned my addr 172.21.15.29:0/808283657 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.378+0000 7fea43fff640 1 --2- 172.21.15.29:0/808283657 >> v2:172.21.15.170:3300/0 conn(0x7fea4c160b90 0x7fea4c160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.379+0000 7fea4a59c640 1 -- 172.21.15.29:0/808283657 >> v2:172.21.15.170:3300/0 conn(0x7fea4c160b90 msgr2=0x7fea4c160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.379+0000 7fea4a59c640 1 --2- 172.21.15.29:0/808283657 >> v2:172.21.15.170:3300/0 conn(0x7fea4c160b90 0x7fea4c160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.379+0000 7fea4a59c640 1 -- 172.21.15.29:0/808283657 >> v1:172.21.15.29:6789/0 conn(0x7fea4c069da0 legacy=0x7fea4c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.379+0000 7fea49d9b640 1 -- 172.21.15.29:0/808283657 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755556876 0 0) 0x7fea4c0ac890 con 0x7fea4c069da0 2023-10-16T14:35:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.379+0000 7fea4a59c640 1 -- 172.21.15.29:0/808283657 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c161930 con 0x7fea4c15d150 2023-10-16T14:35:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.380+0000 7fea4a59c640 1 --2- 172.21.15.29:0/808283657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c15d150 0x7fea4c0ad350 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0cf990 tx=0x7fea3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbdc320b3b8003 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.380+0000 7fea49d9b640 1 -- 172.21.15.29:0/808283657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c02e070 con 0x7fea4c15d150 2023-10-16T14:35:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.380+0000 7fea49d9b640 1 -- 172.21.15.29:0/808283657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3c0315a0 con 0x7fea4c15d150 2023-10-16T14:35:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.380+0000 7fea49d9b640 1 -- 172.21.15.29:0/808283657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c031930 con 0x7fea4c15d150 2023-10-16T14:35:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 -- 172.21.15.29:0/808283657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c15d150 msgr2=0x7fea4c0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 --2- 172.21.15.29:0/808283657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c15d150 0x7fea4c0ad350 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0cf990 tx=0x7fea3c02b840 comp rx=0 tx=0).stop 2023-10-16T14:35:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 -- 172.21.15.29:0/808283657 shutdown_connections 2023-10-16T14:35:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 --2- 172.21.15.29:0/808283657 >> v2:172.21.15.170:3300/0 conn(0x7fea4c160b90 0x7fea4c160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 --2- 172.21.15.29:0/808283657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c15d150 0x7fea4c0ad350 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 -- 172.21.15.29:0/808283657 >> 172.21.15.29:0/808283657 conn(0x7fea4c0ce9b0 msgr2=0x7fea4c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 -- 172.21.15.29:0/808283657 shutdown_connections 2023-10-16T14:35:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.381+0000 7fea50f4c640 1 -- 172.21.15.29:0/808283657 wait complete. 2023-10-16T14:35:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.383+0000 7fea50f4c640 1 Processor -- start 2023-10-16T14:35:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.383+0000 7fea50f4c640 1 -- start start 2023-10-16T14:35:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea50f4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 0x7fea4c0c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea50f4c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea4c15d150 0x7fea4c155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea50f4c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c160b90 0x7fea4c0c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea50f4c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c0c3740 con 0x7fea4c069da0 2023-10-16T14:35:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea50f4c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c0c2ac0 con 0x7fea4c160b90 2023-10-16T14:35:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea50f4c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c0c2da0 con 0x7fea4c15d150 2023-10-16T14:35:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 0x7fea4c0c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 0x7fea4c0c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35836/0 (socket says 172.21.15.29:35836) 2023-10-16T14:35:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.384+0000 7fea4a59c640 1 -- 172.21.15.29:0/1130766172 learned_addr learned my addr 172.21.15.29:0/1130766172 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4ad9d640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c160b90 0x7fea4c0c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4a59c640 1 -- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea4c15d150 msgr2=0x7fea4c155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4a59c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea4c15d150 0x7fea4c155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4a59c640 1 -- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c160b90 msgr2=0x7fea4c0c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4a59c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c160b90 0x7fea4c0c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4a59c640 1 -- 172.21.15.29:0/1130766172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c009400 con 0x7fea4c069da0 2023-10-16T14:35:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea4a59c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 0x7fea4c0c24e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009890 tx=0x7fea3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c032840 con 0x7fea4c069da0 2023-10-16T14:35:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3c0329a0 con 0x7fea4c069da0 2023-10-16T14:35:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c032d50 con 0x7fea4c069da0 2023-10-16T14:35:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c0c3080 con 0x7fea4c069da0 2023-10-16T14:35:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.385+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c0c40a0 con 0x7fea4c069da0 2023-10-16T14:35:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.387+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c0c3880 con 0x7fea4c069da0 2023-10-16T14:35:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.388+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fea3c03b3f0 con 0x7fea4c069da0 2023-10-16T14:35:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.389+0000 7fea42ffd640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 0x7fea2c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.389+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fea3c0abc00 con 0x7fea4c069da0 2023-10-16T14:35:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.389+0000 7fea43fff640 1 -- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 msgr2=0x7fea2c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.389+0000 7fea43fff640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 0x7fea2c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.393+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea4c0c3880 con 0x7fea4c069da0 2023-10-16T14:35:06.493 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.490+0000 7fc456a26040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:35:06.521 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:06.518+0000 7f038e630040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:35:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.590+0000 7fea43fff640 1 -- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 msgr2=0x7fea2c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.590+0000 7fea43fff640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 0x7fea2c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.594+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea4c0c3880 con 0x7fea4c069da0 2023-10-16T14:35:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.595+0000 7fea42ffd640 1 -- 172.21.15.29:0/1130766172 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fea4c0c3880 con 0x7fea4c069da0 2023-10-16T14:35:06.598 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:35:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.598+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 msgr2=0x7fea2c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 0x7fea2c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 msgr2=0x7fea4c0c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 0x7fea4c0c24e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009890 tx=0x7fea3c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 shutdown_connections 2023-10-16T14:35:06.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea4c160b90 0x7fea4c0c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea4c15d150 0x7fea4c155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fea2c06b0b0 0x7fea2c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 --2- 172.21.15.29:0/1130766172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea4c069da0 0x7fea4c0c24e0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 >> 172.21.15.29:0/1130766172 conn(0x7fea4c0ce9b0 msgr2=0x7fea4c0cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:06.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 shutdown_connections 2023-10-16T14:35:06.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.599+0000 7fea50f4c640 1 -- 172.21.15.29:0/1130766172 wait complete. 2023-10-16T14:35:06.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:35:06.629 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:06.627+0000 7f038e630040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:35:06.661 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.659+0000 7fc456a26040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:35:06.680 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:06.677+0000 7f038e630040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:35:06.777 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.774+0000 7fc456a26040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:35:06.788 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:06.785+0000 7f038e630040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.824+0000 7fb6972ff640 1 Processor -- start 2023-10-16T14:35:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.824+0000 7fb6972ff640 1 -- start start 2023-10-16T14:35:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.826+0000 7fb6972ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6900cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.826+0000 7fb6972ff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb6900d0be0 0x7fb6900cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.826+0000 7fb6972ff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69015d150 con 0x7fb69015cd70 2023-10-16T14:35:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.826+0000 7fb6972ff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6900d1180 con 0x7fb6900d0be0 2023-10-16T14:35:06.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.826+0000 7fb6972ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900d1460 con 0x7fb6900b8480 2023-10-16T14:35:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.827+0000 7fb695afc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6900cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.827+0000 7fb696afe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb6900d0be0 0x7fb6900cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.827+0000 7fb695afc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6900cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:36490/0 (socket says 172.21.15.29:36490) 2023-10-16T14:35:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.827+0000 7fb695afc640 1 -- 172.21.15.29:0/3108585823 learned_addr learned my addr 172.21.15.29:0/3108585823 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.828+0000 7fb695afc640 1 -- 172.21.15.29:0/3108585823 >> v2:172.21.15.170:3300/0 conn(0x7fb6900d0be0 msgr2=0x7fb6900cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.828+0000 7fb695afc640 1 --2- 172.21.15.29:0/3108585823 >> v2:172.21.15.170:3300/0 conn(0x7fb6900d0be0 0x7fb6900cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.828+0000 7fb695afc640 1 -- 172.21.15.29:0/3108585823 >> v1:172.21.15.170:6789/0 conn(0x7fb69015cd70 legacy=0x7fb6900d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.828+0000 7fb695afc640 1 -- 172.21.15.29:0/3108585823 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6900d1740 con 0x7fb6900b8480 2023-10-16T14:35:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.828+0000 7fb696afe640 1 --2- 172.21.15.29:0/3108585823 >> v2:172.21.15.170:3300/0 conn(0x7fb6900d0be0 0x7fb6900cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:35:06.833 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.828+0000 7fc456a26040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:35:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.829+0000 7fb695afc640 1 --2- 172.21.15.29:0/3108585823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6900cd7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb680009870 tx=0x7fb68002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d8d8c9a79e02150 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.829+0000 7fb6952fb640 1 -- 172.21.15.29:0/3108585823 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680030840 con 0x7fb6900b8480 2023-10-16T14:35:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.829+0000 7fb6952fb640 1 -- 172.21.15.29:0/3108585823 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6800309a0 con 0x7fb6900b8480 2023-10-16T14:35:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.829+0000 7fb6952fb640 1 -- 172.21.15.29:0/3108585823 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680030d30 con 0x7fb6900b8480 2023-10-16T14:35:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.830+0000 7fb6972ff640 1 -- 172.21.15.29:0/3108585823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 msgr2=0x7fb6900cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.830+0000 7fb6972ff640 1 --2- 172.21.15.29:0/3108585823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6900cd7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb680009870 tx=0x7fb68002b980 comp rx=0 tx=0).stop 2023-10-16T14:35:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.830+0000 7fb6972ff640 1 -- 172.21.15.29:0/3108585823 shutdown_connections 2023-10-16T14:35:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.830+0000 7fb6972ff640 1 --2- 172.21.15.29:0/3108585823 >> v2:172.21.15.170:3300/0 conn(0x7fb6900d0be0 0x7fb6900cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.830+0000 7fb6972ff640 1 --2- 172.21.15.29:0/3108585823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6900cd7c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.830+0000 7fb6972ff640 1 -- 172.21.15.29:0/3108585823 >> 172.21.15.29:0/3108585823 conn(0x7fb6900cf570 msgr2=0x7fb6900cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.831+0000 7fb6972ff640 1 -- 172.21.15.29:0/3108585823 shutdown_connections 2023-10-16T14:35:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.831+0000 7fb6972ff640 1 -- 172.21.15.29:0/3108585823 wait complete. 2023-10-16T14:35:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.832+0000 7fb6972ff640 1 Processor -- start 2023-10-16T14:35:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.832+0000 7fb6972ff640 1 -- start start 2023-10-16T14:35:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.833+0000 7fb6972ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6901f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.833+0000 7fb6972ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 0x7fb6900b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.833+0000 7fb6972ff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb69015cd70 0x7fb6900b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.833+0000 7fb6972ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900ccbe0 con 0x7fb6900d0be0 2023-10-16T14:35:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.833+0000 7fb6972ff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6901f5420 con 0x7fb69015cd70 2023-10-16T14:35:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.833+0000 7fb6972ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900cbfd0 con 0x7fb6900b8480 2023-10-16T14:35:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 0x7fb6900b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 0x7fb6900b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35844/0 (socket says 172.21.15.29:35844) 2023-10-16T14:35:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 -- 172.21.15.29:0/401227014 learned_addr learned my addr 172.21.15.29:0/401227014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb6962fd640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6901f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb696afe640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb69015cd70 0x7fb6900b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 -- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 msgr2=0x7fb6901f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6901f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 -- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb69015cd70 msgr2=0x7fb6900b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb69015cd70 0x7fb6900b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:06.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 -- 172.21.15.29:0/401227014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb688009a70 con 0x7fb6900d0be0 2023-10-16T14:35:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.834+0000 7fb695afc640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 0x7fb6900b3610 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fb68002b9c0 tx=0x7fb680004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.835+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680039a90 con 0x7fb6900d0be0 2023-10-16T14:35:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.835+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb680039bf0 con 0x7fb6900d0be0 2023-10-16T14:35:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.835+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6800094d0 con 0x7fb6900d0be0 2023-10-16T14:35:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.835+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680043410 con 0x7fb6900d0be0 2023-10-16T14:35:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.835+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6900cc590 con 0x7fb6900d0be0 2023-10-16T14:35:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.836+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69004e9b0 con 0x7fb6900d0be0 2023-10-16T14:35:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.837+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb68002f390 con 0x7fb6900d0be0 2023-10-16T14:35:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.838+0000 7fb686ffd640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 0x7fb67406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.838+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fb6800abdb0 con 0x7fb6900d0be0 2023-10-16T14:35:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.839+0000 7fb6962fd640 1 -- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 msgr2=0x7fb67406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.839+0000 7fb6962fd640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 0x7fb67406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:06.842+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6800b0070 con 0x7fb6900d0be0 2023-10-16T14:35:06.897 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:06.895+0000 7f038e630040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:35:06.947 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:06.944+0000 7fc456a26040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.040+0000 7fb6962fd640 1 -- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 msgr2=0x7fb67406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:07.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.040+0000 7fb6962fd640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 0x7fb67406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.048+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb6900cece0 con 0x7fb6900d0be0 2023-10-16T14:35:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.049+0000 7fb686ffd640 1 -- 172.21.15.29:0/401227014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v114) v1 ==== 80+0+477 (secure 0 0 0) 0x7fb6800341f0 con 0x7fb6900d0be0 2023-10-16T14:35:07.052 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2765694912 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.052 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3476448138 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.052 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2716588410 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.052 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/261609677 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.053 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/2925184086 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.053 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2811499332 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.053 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/2925184086 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.053 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1477766173 2023-10-17T14:34:00.590880+0000 2023-10-16T14:35:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T14:35:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 msgr2=0x7fb67406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 0x7fb67406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 msgr2=0x7fb6900b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 0x7fb6900b3610 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fb68002b9c0 tx=0x7fb680004730 comp rx=0 tx=0).stop 2023-10-16T14:35:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 shutdown_connections 2023-10-16T14:35:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fb67406b0b0 0x7fb67406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6900d0be0 0x7fb6900b3610 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb69015cd70 0x7fb6900b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 --2- 172.21.15.29:0/401227014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6900b8480 0x7fb6901f4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 >> 172.21.15.29:0/401227014 conn(0x7fb6900cf570 msgr2=0x7fb6900b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 shutdown_connections 2023-10-16T14:35:07.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.051+0000 7fb6972ff640 1 -- 172.21.15.29:0/401227014 wait complete. 2023-10-16T14:35:07.057 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:07.052+0000 7f038e630040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:07.061 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:07.058+0000 7fc456a26040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:35:07.072 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:35:07.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:07.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:35:07.104 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:07.101+0000 7f038e630040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:07.215 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:35:07.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:07.217 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:35:07.224 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:07.221+0000 7f038e630040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:35:07.232 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:07.230+0000 7fc456a26040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:07.269 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:35:07.269 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:07.287 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:07.284+0000 7fc456a26040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:07.323 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:35:07.324 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:35:07.324 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:07.381 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:35:07.382 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:35:07.382 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:07.413 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:07.411+0000 7fc456a26040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:35:07.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:35:07.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.636+0000 7ff2d2a40640 1 Processor -- start 2023-10-16T14:35:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.636+0000 7ff2d2a40640 1 -- start start 2023-10-16T14:35:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.637+0000 7ff2d2a40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2cc065ab0 0x7ff2cc0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.637+0000 7ff2d2a40640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0acd70 con 0x7ff2cc06d220 2023-10-16T14:35:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.637+0000 7ff2d2a40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0ac4a0 con 0x7ff2cc0ce280 2023-10-16T14:35:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.637+0000 7ff2d2a40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2cc065f40 con 0x7ff2cc065ab0 2023-10-16T14:35:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.638+0000 7ff2d1a3e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff2cc06d220 0x7ff2cc0cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45584/0 (socket says 172.21.15.29:45584) 2023-10-16T14:35:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.638+0000 7ff2d1a3e640 1 -- 172.21.15.29:0/4005031364 learned_addr learned my addr 172.21.15.29:0/4005031364 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.638+0000 7ff2d123d640 1 --2- 172.21.15.29:0/4005031364 >> v2:172.21.15.170:3300/0 conn(0x7ff2cc065ab0 0x7ff2cc0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.638+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994495677 0 0) 0x7ff2cc0acd70 con 0x7ff2cc06d220 2023-10-16T14:35:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.638+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b8003520 con 0x7ff2cc06d220 2023-10-16T14:35:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212964530 0 0) 0x7ff2b8003520 con 0x7ff2cc06d220 2023-10-16T14:35:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 >> v2:172.21.15.170:3300/0 conn(0x7ff2cc065ab0 msgr2=0x7ff2cc0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 --2- 172.21.15.29:0/4005031364 >> v2:172.21.15.170:3300/0 conn(0x7ff2cc065ab0 0x7ff2cc0c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 >> v1:172.21.15.170:6789/0 conn(0x7ff2cc0ce280 legacy=0x7ff2cc0ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0ac720 con 0x7ff2cc06d220 2023-10-16T14:35:07.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7ff2c8001670 con 0x7ff2cc06d220 2023-10-16T14:35:07.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 >> v1:172.21.15.29:6789/0 conn(0x7ff2cc06d220 legacy=0x7ff2cc0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2b8003970 0x7ff2b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2b80047c0 0x7ff2b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8005180 0x7ff2b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc065f40 con 0x7ff2b8003970 2023-10-16T14:35:07.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b800c120 con 0x7ff2b8005180 2023-10-16T14:35:07.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b800c2e0 con 0x7ff2b80047c0 2023-10-16T14:35:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2b8003970 0x7ff2b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d223f640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8005180 0x7ff2b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.639+0000 7ff2d123d640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2b80047c0 0x7ff2b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.640+0000 7ff2d1a3e640 1 -- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2b80047c0 msgr2=0x7ff2b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.640+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2b80047c0 0x7ff2b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.640+0000 7ff2d1a3e640 1 -- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8005180 msgr2=0x7ff2b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.640+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8005180 0x7ff2b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.640+0000 7ff2d1a3e640 1 -- 172.21.15.29:0/4005031364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b8003520 con 0x7ff2b8003970 2023-10-16T14:35:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.640+0000 7ff2d223f640 1 -- 172.21.15.29:0/4005031364 reap_dead start 2023-10-16T14:35:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.641+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2b8003970 0x7ff2b8003d90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800c190 tx=0x7ff2b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e60bb4ee7f012 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.641+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8005ad0 con 0x7ff2b8003970 2023-10-16T14:35:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.641+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c8005c30 con 0x7ff2b8003970 2023-10-16T14:35:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.641+0000 7ff2d0a3c640 1 -- 172.21.15.29:0/4005031364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c802cae0 con 0x7ff2b8003970 2023-10-16T14:35:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4005031364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2b8003970 msgr2=0x7ff2b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2b8003970 0x7ff2b8003d90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800c190 tx=0x7ff2b80057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4005031364 shutdown_connections 2023-10-16T14:35:07.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4005031364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2b8003970 0x7ff2b8003d90 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4005031364 >> 172.21.15.29:0/4005031364 conn(0x7ff2cc0b0a80 msgr2=0x7ff2cc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4005031364 shutdown_connections 2023-10-16T14:35:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.642+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4005031364 wait complete. 2023-10-16T14:35:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.644+0000 7ff2d2a40640 1 Processor -- start 2023-10-16T14:35:07.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.644+0000 7ff2d2a40640 1 -- start start 2023-10-16T14:35:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d2a40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8003970 0x7ff2cc09a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d2a40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 0x7ff2cc06c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d2a40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2cc0b2500 0x7ff2cc0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d2a40640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc069ed0 con 0x7ff2cc06bf70 2023-10-16T14:35:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d2a40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc06a170 con 0x7ff2b8003970 2023-10-16T14:35:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d2a40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc06a450 con 0x7ff2cc0b2500 2023-10-16T14:35:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 0x7ff2cc06c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 0x7ff2cc06c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35862/0 (socket says 172.21.15.29:35862) 2023-10-16T14:35:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 -- 172.21.15.29:0/4047523185 learned_addr learned my addr 172.21.15.29:0/4047523185 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8003970 0x7ff2cc09a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 -- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2cc0b2500 msgr2=0x7ff2cc0b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d223f640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2cc0b2500 0x7ff2cc0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2cc0b2500 0x7ff2cc0b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 -- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8003970 msgr2=0x7ff2cc09a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8003970 0x7ff2cc09a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d123d640 1 -- 172.21.15.29:0/4047523185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc06a730 con 0x7ff2cc06bf70 2023-10-16T14:35:07.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.645+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8003970 0x7ff2cc09a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:07.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.646+0000 7ff2d123d640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 0x7ff2cc06c3f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0114e0 tx=0x7ff2bc0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:07.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.646+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc015870 con 0x7ff2cc06bf70 2023-10-16T14:35:07.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.646+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc0159d0 con 0x7ff2cc06bf70 2023-10-16T14:35:07.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.646+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2cc06aa10 con 0x7ff2cc06bf70 2023-10-16T14:35:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.646+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc015d40 con 0x7ff2cc06bf70 2023-10-16T14:35:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.646+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2cc09a9b0 con 0x7ff2cc06bf70 2023-10-16T14:35:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.647+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2cc04e9b0 con 0x7ff2cc06bf70 2023-10-16T14:35:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.649+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff2bc013400 con 0x7ff2cc06bf70 2023-10-16T14:35:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.650+0000 7ff2c27fc640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 0x7ff2b006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.650+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7ff2bc08a2a0 con 0x7ff2cc06bf70 2023-10-16T14:35:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.650+0000 7ff2d1a3e640 1 -- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 msgr2=0x7ff2b006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.650+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 0x7ff2b006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:07.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.658+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2bc054e60 con 0x7ff2cc06bf70 2023-10-16T14:35:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.851+0000 7ff2d1a3e640 1 -- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 msgr2=0x7ff2b006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.851+0000 7ff2d1a3e640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 0x7ff2b006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:07.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.886+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2cc06aba0 con 0x7ff2cc06bf70 2023-10-16T14:35:07.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.886+0000 7ff2c27fc640 1 -- 172.21.15.29:0/4047523185 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4915 (secure 0 0 0) 0x7ff2bc058480 con 0x7ff2cc06bf70 2023-10-16T14:35:07.890 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:07.890 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:45.763311+0000","modified":"2023-10-16T14:33:48.952054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7323},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7323":{"gid":7323,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6836/190249339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":190249339},{"type":"v1","addr":"172.21.15.170:6836","nonce":190249339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T14:35:07.890 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2023-10-16T14:35:07.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 msgr2=0x7ff2b006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:07.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 0x7ff2b006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 msgr2=0x7ff2cc06c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:07.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 0x7ff2cc06c3f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc0114e0 tx=0x7ff2bc0119a0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 shutdown_connections 2023-10-16T14:35:07.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7ff2b006b0b0 0x7ff2b006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2cc06bf70 0x7ff2cc06c3f0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2cc0b2500 0x7ff2cc0b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 --2- 172.21.15.29:0/4047523185 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2b8003970 0x7ff2cc09a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:07.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 >> 172.21.15.29:0/4047523185 conn(0x7ff2cc0b0a80 msgr2=0x7ff2cc0afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:07.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 shutdown_connections 2023-10-16T14:35:07.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:07.889+0000 7ff2d2a40640 1 -- 172.21.15.29:0/4047523185 wait complete. 2023-10-16T14:35:07.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:08.064 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.061+0000 7f038e630040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:35:08.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.095+0000 7fd6c5059640 1 Processor -- start 2023-10-16T14:35:08.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.095+0000 7fd6c5059640 1 -- start start 2023-10-16T14:35:08.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.096+0000 7fd6c5059640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6c00ad970 0x7fd6c00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6c5059640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c015b9d0 con 0x7fd6c006c040 2023-10-16T14:35:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6c5059640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c015bcb0 con 0x7fd6c00ccfb0 2023-10-16T14:35:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6c5059640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c015bf90 con 0x7fd6c00ad970 2023-10-16T14:35:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6b7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6c00ad970 0x7fd6c00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6b7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6c00ad970 0x7fd6c00cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35878/0 (socket says 172.21.15.29:35878) 2023-10-16T14:35:08.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6b7fff640 1 -- 172.21.15.29:0/2172626712 learned_addr learned my addr 172.21.15.29:0/2172626712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:08.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6bffff640 1 --1- 172.21.15.29:0/2172626712 >> v1:172.21.15.29:6789/0 conn(0x7fd6c006c040 0x7fd6c006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45592/0 (socket says 172.21.15.29:45592) 2023-10-16T14:35:08.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6b7fff640 1 -- 172.21.15.29:0/2172626712 >> v1:172.21.15.170:6789/0 conn(0x7fd6c00ccfb0 legacy=0x7fd6c00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:08.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6b7fff640 1 -- 172.21.15.29:0/2172626712 >> v1:172.21.15.29:6789/0 conn(0x7fd6c006c040 legacy=0x7fd6c006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.097+0000 7fd6b7fff640 1 -- 172.21.15.29:0/2172626712 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c00cc270 con 0x7fd6c00ad970 2023-10-16T14:35:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.099+0000 7fd6b7fff640 1 --2- 172.21.15.29:0/2172626712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c00ad970 0x7fd6c00cca70 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80098a0 tx=0x7fd6a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a305d7d51d3a1e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.099+0000 7fd6bf7fe640 1 -- 172.21.15.29:0/2172626712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8030840 con 0x7fd6c00ad970 2023-10-16T14:35:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.099+0000 7fd6bf7fe640 1 -- 172.21.15.29:0/2172626712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a80309a0 con 0x7fd6c00ad970 2023-10-16T14:35:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.099+0000 7fd6bf7fe640 1 -- 172.21.15.29:0/2172626712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8030d30 con 0x7fd6c00ad970 2023-10-16T14:35:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.099+0000 7fd6c5059640 1 -- 172.21.15.29:0/2172626712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c00ad970 msgr2=0x7fd6c00cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.099+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2172626712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c00ad970 0x7fd6c00cca70 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80098a0 tx=0x7fd6a802b980 comp rx=0 tx=0).stop 2023-10-16T14:35:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.100+0000 7fd6c5059640 1 -- 172.21.15.29:0/2172626712 shutdown_connections 2023-10-16T14:35:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.100+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2172626712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c00ad970 0x7fd6c00cca70 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.100+0000 7fd6c5059640 1 -- 172.21.15.29:0/2172626712 >> 172.21.15.29:0/2172626712 conn(0x7fd6c00ce6a0 msgr2=0x7fd6c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.100+0000 7fd6c5059640 1 -- 172.21.15.29:0/2172626712 shutdown_connections 2023-10-16T14:35:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.100+0000 7fd6c5059640 1 -- 172.21.15.29:0/2172626712 wait complete. 2023-10-16T14:35:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.101+0000 7fd6c5059640 1 Processor -- start 2023-10-16T14:35:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.101+0000 7fd6c5059640 1 -- start start 2023-10-16T14:35:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.102+0000 7fd6c5059640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 0x7fd6c01f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.102+0000 7fd6c5059640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00ad970 0x7fd6c00b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.102+0000 7fd6c5059640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00ccfb0 0x7fd6c00c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.102+0000 7fd6c5059640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00ae470 con 0x7fd6c006c040 2023-10-16T14:35:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.102+0000 7fd6c5059640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c01f28a0 con 0x7fd6c00ccfb0 2023-10-16T14:35:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.102+0000 7fd6c5059640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00b0960 con 0x7fd6c00ad970 2023-10-16T14:35:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 0x7fd6c01f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 0x7fd6c01f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35884/0 (socket says 172.21.15.29:35884) 2023-10-16T14:35:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 -- 172.21.15.29:0/2998350670 learned_addr learned my addr 172.21.15.29:0/2998350670 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 -- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00ad970 msgr2=0x7fd6c00b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00ad970 0x7fd6c00b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 -- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00ccfb0 msgr2=0x7fd6c00c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00ccfb0 0x7fd6c00c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 -- 172.21.15.29:0/2998350670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80094d0 con 0x7fd6c006c040 2023-10-16T14:35:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.103+0000 7fd6bffff640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 0x7fd6c01f2360 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c3e0 tx=0x7fd6b000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.104+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00074e0 con 0x7fd6c006c040 2023-10-16T14:35:08.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.104+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0007640 con 0x7fd6c006c040 2023-10-16T14:35:08.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.104+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00049e0 con 0x7fd6c006c040 2023-10-16T14:35:08.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.104+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c00b0c40 con 0x7fd6c006c040 2023-10-16T14:35:08.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.104+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c00ae100 con 0x7fd6c006c040 2023-10-16T14:35:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.105+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c006b330 con 0x7fd6c006c040 2023-10-16T14:35:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.106+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd6b0004b40 con 0x7fd6c006c040 2023-10-16T14:35:08.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.107+0000 7fd6bd7fa640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fd6a006b0b0 0x7fd6a006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.107+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fd6b0089530 con 0x7fd6c006c040 2023-10-16T14:35:08.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.107+0000 7fd6b7fff640 1 -- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fd6a006b0b0 msgr2=0x7fd6a006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3649833910 2023-10-16T14:35:08.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.107+0000 7fd6b7fff640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fd6a006b0b0 0x7fd6a006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.111+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6b0054190 con 0x7fd6c006c040 2023-10-16T14:35:08.272 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:08.269+0000 7fc456a26040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:35:08.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.298+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6c004e9b0 con 0x7fd6c006c040 2023-10-16T14:35:08.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.300+0000 7fd6bd7fa640 1 -- 172.21.15.29:0/2998350670 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+17129 (secure 0 0 0) 0x7fd6b00577b0 con 0x7fd6c006c040 2023-10-16T14:35:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:08.303 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:02.383554+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.26","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2765694912":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/3476448138":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2716588410":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/261609677":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6833/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2811499332":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6832/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1477766173":"2023-10-17T14:34:00.590880+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.302+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fd6a006b0b0 msgr2=0x7fd6a006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.302+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fd6a006b0b0 0x7fd6a006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 msgr2=0x7fd6c01f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 0x7fd6c01f2360 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c3e0 tx=0x7fd6b000c8a0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 shutdown_connections 2023-10-16T14:35:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6c00ccfb0 0x7fd6c00c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6c00ad970 0x7fd6c00b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.170:6832/3649833910,v1:172.21.15.170:6833/3649833910] conn(0x7fd6a006b0b0 0x7fd6a006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 --2- 172.21.15.29:0/2998350670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c006c040 0x7fd6c01f2360 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 >> 172.21.15.29:0/2998350670 conn(0x7fd6c00ce6a0 msgr2=0x7fd6c00c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 shutdown_connections 2023-10-16T14:35:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.303+0000 7fd6c5059640 1 -- 172.21.15.29:0/2998350670 wait complete. 2023-10-16T14:35:08.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:33:43.719274+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:33:44.790872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:35:08.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:35:08.357 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.355+0000 7f038e630040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:35:08.415 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.412+0000 7f038e630040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:35:08.529 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.527+0000 7f038e630040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:35:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.531+0000 7fc33658e640 1 Processor -- start 2023-10-16T14:35:08.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.531+0000 7fc33658e640 1 -- start start 2023-10-16T14:35:08.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.532+0000 7fc33658e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc3300b1ca0 0x7fc3300d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.532+0000 7fc33658e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300b2080 0x7fc3300ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.532+0000 7fc33658e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3300ce960 0x7fc3300b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.532+0000 7fc33658e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3300d0970 con 0x7fc3300ce960 2023-10-16T14:35:08.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.532+0000 7fc33658e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3300d0ea0 con 0x7fc3300b1ca0 2023-10-16T14:35:08.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.532+0000 7fc33658e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300d0bc0 con 0x7fc3300b2080 2023-10-16T14:35:08.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3300ce960 0x7fc3300b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3300ce960 0x7fc3300b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35888/0 (socket says 172.21.15.29:35888) 2023-10-16T14:35:08.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 -- 172.21.15.29:0/4186121020 learned_addr learned my addr 172.21.15.29:0/4186121020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:08.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc33558c640 1 --2- 172.21.15.29:0/4186121020 >> v2:172.21.15.170:3300/0 conn(0x7fc3300b1ca0 0x7fc3300d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc334d8b640 1 --2- 172.21.15.29:0/4186121020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300b2080 0x7fc3300ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 -- 172.21.15.29:0/4186121020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300b2080 msgr2=0x7fc3300ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:08.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 --2- 172.21.15.29:0/4186121020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300b2080 0x7fc3300ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 -- 172.21.15.29:0/4186121020 >> v2:172.21.15.170:3300/0 conn(0x7fc3300b1ca0 msgr2=0x7fc3300d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 --2- 172.21.15.29:0/4186121020 >> v2:172.21.15.170:3300/0 conn(0x7fc3300b1ca0 0x7fc3300d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc335d8d640 1 -- 172.21.15.29:0/4186121020 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3300cd940 con 0x7fc3300ce960 2023-10-16T14:35:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.533+0000 7fc33558c640 1 --2- 172.21.15.29:0/4186121020 >> v2:172.21.15.170:3300/0 conn(0x7fc3300b1ca0 0x7fc3300d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:08.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.534+0000 7fc335d8d640 1 --2- 172.21.15.29:0/4186121020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300ce960 0x7fc3300b24c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc330067210 tx=0x7fc32802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaca643e6fcfb1f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc327fff640 1 -- 172.21.15.29:0/4186121020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32802cb00 con 0x7fc3300ce960 2023-10-16T14:35:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc327fff640 1 -- 172.21.15.29:0/4186121020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32802cc60 con 0x7fc3300ce960 2023-10-16T14:35:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc327fff640 1 -- 172.21.15.29:0/4186121020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3280343f0 con 0x7fc3300ce960 2023-10-16T14:35:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 -- 172.21.15.29:0/4186121020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300ce960 msgr2=0x7fc3300b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 --2- 172.21.15.29:0/4186121020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300ce960 0x7fc3300b24c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc330067210 tx=0x7fc32802bf60 comp rx=0 tx=0).stop 2023-10-16T14:35:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 -- 172.21.15.29:0/4186121020 shutdown_connections 2023-10-16T14:35:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 --2- 172.21.15.29:0/4186121020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300ce960 0x7fc3300b24c0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 --2- 172.21.15.29:0/4186121020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300b2080 0x7fc3300ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 --2- 172.21.15.29:0/4186121020 >> v2:172.21.15.170:3300/0 conn(0x7fc3300b1ca0 0x7fc3300d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.535+0000 7fc33658e640 1 -- 172.21.15.29:0/4186121020 >> 172.21.15.29:0/4186121020 conn(0x7fc3300d2b90 msgr2=0x7fc3300d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:08.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.536+0000 7fc33658e640 1 -- 172.21.15.29:0/4186121020 shutdown_connections 2023-10-16T14:35:08.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.536+0000 7fc33658e640 1 -- 172.21.15.29:0/4186121020 wait complete. 2023-10-16T14:35:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.538+0000 7fc33658e640 1 Processor -- start 2023-10-16T14:35:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.539+0000 7fc33658e640 1 -- start start 2023-10-16T14:35:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.540+0000 7fc33658e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 0x7fc3300c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.540+0000 7fc33658e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3300b2080 0x7fc3300c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.540+0000 7fc33658e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300ce960 0x7fc33015cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.540+0000 7fc33658e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33015d0a0 con 0x7fc3300b1ca0 2023-10-16T14:35:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.540+0000 7fc33658e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300c9e00 con 0x7fc3300b2080 2023-10-16T14:35:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.540+0000 7fc33658e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300ca0e0 con 0x7fc3300ce960 2023-10-16T14:35:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 0x7fc3300c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 0x7fc3300c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35894/0 (socket says 172.21.15.29:35894) 2023-10-16T14:35:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 -- 172.21.15.29:0/90365785 learned_addr learned my addr 172.21.15.29:0/90365785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 -- 172.21.15.29:0/90365785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300ce960 msgr2=0x7fc33015cb60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:35:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300ce960 0x7fc33015cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 -- 172.21.15.29:0/90365785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3300b2080 msgr2=0x7fc3300c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3300b2080 0x7fc3300c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 -- 172.21.15.29:0/90365785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328009b30 con 0x7fc3300b1ca0 2023-10-16T14:35:08.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.541+0000 7fc33558c640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 0x7fc3300c9680 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00e7e0 tx=0x7fc32c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.543+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c004030 con 0x7fc3300b1ca0 2023-10-16T14:35:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.543+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c004190 con 0x7fc3300b1ca0 2023-10-16T14:35:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.543+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c004520 con 0x7fc3300b1ca0 2023-10-16T14:35:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.543+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3300ca3c0 con 0x7fc3300b1ca0 2023-10-16T14:35:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.544+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3301f26a0 con 0x7fc3300b1ca0 2023-10-16T14:35:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.545+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33004e9b0 con 0x7fc3300b1ca0 2023-10-16T14:35:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.545+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc32c00c6d0 con 0x7fc3300b1ca0 2023-10-16T14:35:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.546+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fc32c089080 con 0x7fc3300b1ca0 2023-10-16T14:35:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.550+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc32c054260 con 0x7fc3300b1ca0 2023-10-16T14:35:08.600 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:08.597+0000 7fc456a26040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:35:08.614 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f031006f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.614 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f031006f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.614 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f031006f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.614 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f031006f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.615 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f031006f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.615 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f0313075640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.615 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f0313075640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.615 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f0313075640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.615 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f0313075640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.616 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:08.611+0000 7f0313075640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:08.663 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:08.660+0000 7fc456a26040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:35:08.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.748+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3300ccf30 con 0x7fc3300b1ca0 2023-10-16T14:35:08.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.749+0000 7fc325ffb640 1 -- 172.21.15.29:0/90365785 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc32c057880 con 0x7fc3300b1ca0 2023-10-16T14:35:08.752 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:35:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 msgr2=0x7fc3300c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 0x7fc3300c9680 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00e7e0 tx=0x7fc32c00eca0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 shutdown_connections 2023-10-16T14:35:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc3300ce960 0x7fc33015cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3300b2080 0x7fc3300c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 --2- 172.21.15.29:0/90365785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3300b1ca0 0x7fc3300c9680 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.752+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 >> 172.21.15.29:0/90365785 conn(0x7fc3300d2b90 msgr2=0x7fc3300b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:08.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.753+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 shutdown_connections 2023-10-16T14:35:08.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.753+0000 7fc33658e640 1 -- 172.21.15.29:0/90365785 wait complete. 2023-10-16T14:35:08.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:35:08.782 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:08.780+0000 7fc456a26040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:35:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.987+0000 7f035455e640 1 Processor -- start 2023-10-16T14:35:08.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.987+0000 7f035455e640 1 -- start start 2023-10-16T14:35:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.988+0000 7f035455e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0cf710 0x7f034c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.988+0000 7f035455e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c0d18c0 con 0x7f034c0b6cd0 2023-10-16T14:35:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.989+0000 7f035455e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c0d3980 con 0x7f034c0cfaf0 2023-10-16T14:35:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.989+0000 7f035455e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0d3c60 con 0x7f034c0cf710 2023-10-16T14:35:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.989+0000 7f0352afb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f034c0b6cd0 0x7f034c0d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45604/0 (socket says 172.21.15.29:45604) 2023-10-16T14:35:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.989+0000 7f0352afb640 1 -- 172.21.15.29:0/3991873653 learned_addr learned my addr 172.21.15.29:0/3991873653 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.989+0000 7f03522fa640 1 --2- 172.21.15.29:0/3991873653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0cf710 0x7f034c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.990+0000 7f03512f8640 1 -- 172.21.15.29:0/3991873653 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736688070 0 0) 0x7f034c0d18c0 con 0x7f034c0b6cd0 2023-10-16T14:35:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.990+0000 7f03512f8640 1 -- 172.21.15.29:0/3991873653 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0338003520 con 0x7f034c0b6cd0 2023-10-16T14:35:08.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.990+0000 7f03512f8640 1 -- 172.21.15.29:0/3991873653 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367960732 0 0) 0x7f0338003520 con 0x7f034c0b6cd0 2023-10-16T14:35:08.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.990+0000 7f03522fa640 1 -- 172.21.15.29:0/3991873653 >> v1:172.21.15.170:6789/0 conn(0x7f034c0cfaf0 legacy=0x7f034c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.990+0000 7f03522fa640 1 -- 172.21.15.29:0/3991873653 >> v1:172.21.15.29:6789/0 conn(0x7f034c0b6cd0 legacy=0x7f034c0d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.990+0000 7f03522fa640 1 -- 172.21.15.29:0/3991873653 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c0ce630 con 0x7f034c0cf710 2023-10-16T14:35:08.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.991+0000 7f03522fa640 1 --2- 172.21.15.29:0/3991873653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0cf710 0x7f034c0b2d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0348009e10 tx=0x7f034802bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b04918825cc4eeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:08.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.992+0000 7f03512f8640 1 -- 172.21.15.29:0/3991873653 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344001670 con 0x7f034c0cf710 2023-10-16T14:35:08.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.992+0000 7f03512f8640 1 -- 172.21.15.29:0/3991873653 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034802c8e0 con 0x7f034c0cf710 2023-10-16T14:35:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.992+0000 7f03512f8640 1 -- 172.21.15.29:0/3991873653 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034802cc70 con 0x7f034c0cf710 2023-10-16T14:35:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 -- 172.21.15.29:0/3991873653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0cf710 msgr2=0x7f034c0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 --2- 172.21.15.29:0/3991873653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0cf710 0x7f034c0b2d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0348009e10 tx=0x7f034802bdb0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 -- 172.21.15.29:0/3991873653 shutdown_connections 2023-10-16T14:35:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 --2- 172.21.15.29:0/3991873653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0cf710 0x7f034c0b2d10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 -- 172.21.15.29:0/3991873653 >> 172.21.15.29:0/3991873653 conn(0x7f034c0d2cc0 msgr2=0x7f034c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 -- 172.21.15.29:0/3991873653 shutdown_connections 2023-10-16T14:35:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.993+0000 7f035455e640 1 -- 172.21.15.29:0/3991873653 wait complete. 2023-10-16T14:35:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.995+0000 7f035455e640 1 Processor -- start 2023-10-16T14:35:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.995+0000 7f035455e640 1 -- start start 2023-10-16T14:35:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f035455e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0b6cd0 0x7f034c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f035455e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0cf710 0x7f034c145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f035455e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 0x7f034c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f035455e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0cb890 con 0x7f034c0cfaf0 2023-10-16T14:35:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f035455e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c153cb0 con 0x7f034c0cf710 2023-10-16T14:35:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f035455e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c153f90 con 0x7f034c0b6cd0 2023-10-16T14:35:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.996+0000 7f0352afb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 0x7f034c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 0x7f034c0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35896/0 (socket says 172.21.15.29:35896) 2023-10-16T14:35:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 -- 172.21.15.29:0/484727065 learned_addr learned my addr 172.21.15.29:0/484727065 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0351af9640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0cf710 0x7f034c145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f03522fa640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0b6cd0 0x7f034c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 -- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0b6cd0 msgr2=0x7f034c0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0b6cd0 0x7f034c0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 -- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0cf710 msgr2=0x7f034c145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0cf710 0x7f034c145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 -- 172.21.15.29:0/484727065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0348009a70 con 0x7f034c0cfaf0 2023-10-16T14:35:09.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.997+0000 7f0352afb640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 0x7f034c0b02e0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f03440011e0 tx=0x7f034400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.998+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344005ad0 con 0x7f034c0cfaf0 2023-10-16T14:35:09.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.998+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0344005c30 con 0x7f034c0cfaf0 2023-10-16T14:35:09.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.998+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344007070 con 0x7f034c0cfaf0 2023-10-16T14:35:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.998+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c154270 con 0x7f034c0cfaf0 2023-10-16T14:35:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.998+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c1461a0 con 0x7f034c0cfaf0 2023-10-16T14:35:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:08.999+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c0cba60 con 0x7f034c0cfaf0 2023-10-16T14:35:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.000+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f03440071d0 con 0x7f034c0cfaf0 2023-10-16T14:35:09.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.001+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f03440890b0 con 0x7f034c0cfaf0 2023-10-16T14:35:09.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.006+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0344054130 con 0x7f034c0cfaf0 2023-10-16T14:35:09.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.201+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f034c154400 con 0x7f034c0cfaf0 2023-10-16T14:35:09.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.202+0000 7f0342ffd640 1 -- 172.21.15.29:0/484727065 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0344057750 con 0x7f034c0cfaf0 2023-10-16T14:35:09.205 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:35:09.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 msgr2=0x7f034c0b02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 0x7f034c0b02e0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f03440011e0 tx=0x7f034400cbe0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 shutdown_connections 2023-10-16T14:35:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0cfaf0 0x7f034c0b02e0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0cf710 0x7f034c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 --2- 172.21.15.29:0/484727065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c0b6cd0 0x7f034c0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.204+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 >> 172.21.15.29:0/484727065 conn(0x7f034c0d2cc0 msgr2=0x7f034c0d12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.205+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 shutdown_connections 2023-10-16T14:35:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.205+0000 7f035455e640 1 -- 172.21.15.29:0/484727065 wait complete. 2023-10-16T14:35:09.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:35:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.505+0000 7fe4df59e640 1 Processor -- start 2023-10-16T14:35:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.505+0000 7fe4df59e640 1 -- start start 2023-10-16T14:35:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4df59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4e0159950 0x7fe4e015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4df59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e006c810 0x7fe4e00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4df59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e015aef0 con 0x7fe4e0070450 2023-10-16T14:35:09.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4df59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e0070e20 con 0x7fe4e0159950 2023-10-16T14:35:09.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4df59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e015e490 con 0x7fe4e006c810 2023-10-16T14:35:09.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4de59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4e0159950 0x7fe4e015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4de59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4e0159950 0x7fe4e015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35908/0 (socket says 172.21.15.29:35908) 2023-10-16T14:35:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.507+0000 7fe4de59c640 1 -- 172.21.15.29:0/1117165061 learned_addr learned my addr 172.21.15.29:0/1117165061 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.508+0000 7fe4de59c640 1 -- 172.21.15.29:0/1117165061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e006c810 msgr2=0x7fe4e00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.508+0000 7fe4de59c640 1 --2- 172.21.15.29:0/1117165061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e006c810 0x7fe4e00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.508+0000 7fe4de59c640 1 -- 172.21.15.29:0/1117165061 >> v1:172.21.15.170:6789/0 conn(0x7fe4e0070450 legacy=0x7fe4e006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.508+0000 7fe4de59c640 1 -- 172.21.15.29:0/1117165061 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e006cc60 con 0x7fe4e0159950 2023-10-16T14:35:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.509+0000 7fe4de59c640 1 --2- 172.21.15.29:0/1117165061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159950 0x7fe4e015a9b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009870 tx=0x7fe4d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8253f0b8a0a8a06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.509+0000 7fe4dd59a640 1 -- 172.21.15.29:0/1117165061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00317c0 con 0x7fe4e0159950 2023-10-16T14:35:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.509+0000 7fe4dd59a640 1 -- 172.21.15.29:0/1117165061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d0031920 con 0x7fe4e0159950 2023-10-16T14:35:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.509+0000 7fe4dd59a640 1 -- 172.21.15.29:0/1117165061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0031cb0 con 0x7fe4e0159950 2023-10-16T14:35:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 -- 172.21.15.29:0/1117165061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159950 msgr2=0x7fe4e015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 --2- 172.21.15.29:0/1117165061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159950 0x7fe4e015a9b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009870 tx=0x7fe4d002b860 comp rx=0 tx=0).stop 2023-10-16T14:35:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 -- 172.21.15.29:0/1117165061 shutdown_connections 2023-10-16T14:35:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 --2- 172.21.15.29:0/1117165061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e006c810 0x7fe4e00708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 --2- 172.21.15.29:0/1117165061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0159950 0x7fe4e015a9b0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 -- 172.21.15.29:0/1117165061 >> 172.21.15.29:0/1117165061 conn(0x7fe4e00ce6a0 msgr2=0x7fe4e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 -- 172.21.15.29:0/1117165061 shutdown_connections 2023-10-16T14:35:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.510+0000 7fe4df59e640 1 -- 172.21.15.29:0/1117165061 wait complete. 2023-10-16T14:35:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.512+0000 7fe4df59e640 1 Processor -- start 2023-10-16T14:35:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.512+0000 7fe4df59e640 1 -- start start 2023-10-16T14:35:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.513+0000 7fe4df59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 0x7fe4e014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.513+0000 7fe4df59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4e0070450 0x7fe4e00c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.513+0000 7fe4df59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e0159950 0x7fe4e00c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.513+0000 7fe4df59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00c7910 con 0x7fe4e006c810 2023-10-16T14:35:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.513+0000 7fe4df59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00c2490 con 0x7fe4e0070450 2023-10-16T14:35:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.513+0000 7fe4df59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00c7300 con 0x7fe4e0159950 2023-10-16T14:35:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 0x7fe4e014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 0x7fe4e014eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35912/0 (socket says 172.21.15.29:35912) 2023-10-16T14:35:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 -- 172.21.15.29:0/660898630 learned_addr learned my addr 172.21.15.29:0/660898630 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4ded9d640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e0159950 0x7fe4e00c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 -- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e0159950 msgr2=0x7fe4e00c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e0159950 0x7fe4e00c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 -- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4e0070450 msgr2=0x7fe4e00c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4ddd9b640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4e0070450 0x7fe4e00c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4e0070450 0x7fe4e00c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 -- 172.21.15.29:0/660898630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00094d0 con 0x7fe4e006c810 2023-10-16T14:35:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4ded9d640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e0159950 0x7fe4e00c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.514+0000 7fe4de59c640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 0x7fe4e014eab0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0000c00 tx=0x7fe4d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.515+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0032860 con 0x7fe4e006c810 2023-10-16T14:35:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.515+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d00329e0 con 0x7fe4e006c810 2023-10-16T14:35:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.515+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0032d90 con 0x7fe4e006c810 2023-10-16T14:35:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.515+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e00c5150 con 0x7fe4e006c810 2023-10-16T14:35:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.515+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e00c75a0 con 0x7fe4e006c810 2023-10-16T14:35:09.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.516+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e004e9b0 con 0x7fe4e006c810 2023-10-16T14:35:09.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.517+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe4d004d3f0 con 0x7fe4e006c810 2023-10-16T14:35:09.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.518+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fe4d00ab1d0 con 0x7fe4e006c810 2023-10-16T14:35:09.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.523+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4d00762e0 con 0x7fe4e006c810 2023-10-16T14:35:09.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.551+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 54) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe4d0075d90 con 0x7fe4e006c810 2023-10-16T14:35:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.552+0000 7fe4ceffd640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe4c0070690 0x7fe4c0072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.553+0000 7fe4ddd9b640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe4c0070690 0x7fe4c0072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.565+0000 7fe4ddd9b640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe4c0070690 0x7fe4c0072a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0158bd0 tx=0x7fe4d4009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.729+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4e015aef0 con 0x7fe4e006c810 2023-10-16T14:35:09.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.730+0000 7fe4ceffd640 1 -- 172.21.15.29:0/660898630 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe4e015aef0 con 0x7fe4e006c810 2023-10-16T14:35:09.733 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:35:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe4c0070690 msgr2=0x7fe4c0072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe4c0070690 0x7fe4c0072a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0158bd0 tx=0x7fe4d4009040 comp rx=0 tx=0).stop 2023-10-16T14:35:09.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 msgr2=0x7fe4e014eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 0x7fe4e014eab0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0000c00 tx=0x7fe4d00046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 shutdown_connections 2023-10-16T14:35:09.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe4c0070690 0x7fe4c0072a80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4e0159950 0x7fe4e00c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4e0070450 0x7fe4e00c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 --2- 172.21.15.29:0/660898630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e006c810 0x7fe4e014eab0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.733+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 >> 172.21.15.29:0/660898630 conn(0x7fe4e00ce6a0 msgr2=0x7fe4e0159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.734+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 shutdown_connections 2023-10-16T14:35:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.734+0000 7fe4df59e640 1 -- 172.21.15.29:0/660898630 wait complete. 2023-10-16T14:35:09.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:09.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.971+0000 7f248e9b8640 1 Processor -- start 2023-10-16T14:35:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.972+0000 7f248e9b8640 1 -- start start 2023-10-16T14:35:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.973+0000 7f248e9b8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f248815cd70 0x7f24880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.973+0000 7f248e9b8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248815d150 con 0x7f24880b8480 2023-10-16T14:35:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.973+0000 7f248e9b8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880d1180 con 0x7f24880cded0 2023-10-16T14:35:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.973+0000 7f248e9b8640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24880d1460 con 0x7f248815cd70 2023-10-16T14:35:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.973+0000 7f248d1b5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f24880b8480 0x7f24880cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45612/0 (socket says 172.21.15.29:45612) 2023-10-16T14:35:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.973+0000 7f248d1b5640 1 -- 172.21.15.29:0/2600722490 learned_addr learned my addr 172.21.15.29:0/2600722490 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248d9b6640 1 --2- 172.21.15.29:0/2600722490 >> v2:172.21.15.170:3300/0 conn(0x7f248815cd70 0x7f24880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190699123 0 0) 0x7f248815d150 con 0x7f24880b8480 2023-10-16T14:35:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2470003520 con 0x7f24880b8480 2023-10-16T14:35:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322907874 0 0) 0x7f2470003520 con 0x7f24880b8480 2023-10-16T14:35:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 >> v2:172.21.15.170:3300/0 conn(0x7f248815cd70 msgr2=0x7f24880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 --2- 172.21.15.29:0/2600722490 >> v2:172.21.15.170:3300/0 conn(0x7f248815cd70 0x7f24880d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 >> v1:172.21.15.170:6789/0 conn(0x7f24880cded0 legacy=0x7f24880b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24880d1740 con 0x7f24880b8480 2023-10-16T14:35:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f247c001670 con 0x7f24880b8480 2023-10-16T14:35:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 >> v1:172.21.15.29:6789/0 conn(0x7f24880b8480 legacy=0x7f24880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470003970 0x7f2470003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24700047c0 0x7f2470004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2470005180 0x7f247000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880d1460 con 0x7f2470003970 2023-10-16T14:35:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247000c020 con 0x7f24700047c0 2023-10-16T14:35:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.974+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247000c1e0 con 0x7f2470005180 2023-10-16T14:35:09.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248d9b6640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470003970 0x7f2470003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248d9b6640 1 -- 172.21.15.29:0/2600722490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2470005180 msgr2=0x7f247000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:09.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248d9b6640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2470005180 0x7f247000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248d9b6640 1 -- 172.21.15.29:0/2600722490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24700047c0 msgr2=0x7f2470004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:09.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248d9b6640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24700047c0 0x7f2470004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248d9b6640 1 -- 172.21.15.29:0/2600722490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470003520 con 0x7f2470003970 2023-10-16T14:35:09.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.975+0000 7f248e1b7640 1 -- 172.21.15.29:0/2600722490 reap_dead start 2023-10-16T14:35:09.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.976+0000 7f248d9b6640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470003970 0x7f2470003d90 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2478009fc0 tx=0x7f24700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e49b9bcdb5cbe89 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.976+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247802cc60 con 0x7f2470003970 2023-10-16T14:35:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.976+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247800f040 con 0x7f2470003970 2023-10-16T14:35:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.976+0000 7f248c9b4640 1 -- 172.21.15.29:0/2600722490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478034440 con 0x7f2470003970 2023-10-16T14:35:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 -- 172.21.15.29:0/2600722490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470003970 msgr2=0x7f2470003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470003970 0x7f2470003d90 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2478009fc0 tx=0x7f24700057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 -- 172.21.15.29:0/2600722490 shutdown_connections 2023-10-16T14:35:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 --2- 172.21.15.29:0/2600722490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2470003970 0x7f2470003d90 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 -- 172.21.15.29:0/2600722490 >> 172.21.15.29:0/2600722490 conn(0x7f24880cf570 msgr2=0x7f24880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 -- 172.21.15.29:0/2600722490 shutdown_connections 2023-10-16T14:35:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.977+0000 7f248e9b8640 1 -- 172.21.15.29:0/2600722490 wait complete. 2023-10-16T14:35:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.979+0000 7f248e9b8640 1 Processor -- start 2023-10-16T14:35:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.979+0000 7f248e9b8640 1 -- start start 2023-10-16T14:35:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e9b8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470003970 0x7f248806f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e9b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24880cbe40 0x7f24880c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e9b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 0x7f2488158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e9b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24881faed0 con 0x7f24880cc2c0 2023-10-16T14:35:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e9b8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24881fa2c0 con 0x7f2470003970 2023-10-16T14:35:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e9b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24881fa5a0 con 0x7f24880cbe40 2023-10-16T14:35:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.980+0000 7f248e1b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 0x7f2488158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 0x7f2488158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35924/0 (socket says 172.21.15.29:35924) 2023-10-16T14:35:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 -- 172.21.15.29:0/945463736 learned_addr learned my addr 172.21.15.29:0/945463736 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248d1b5640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24880cbe40 0x7f24880c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248d9b6640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470003970 0x7f248806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 -- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24880cbe40 msgr2=0x7f24880c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24880cbe40 0x7f24880c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 -- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470003970 msgr2=0x7f248806f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470003970 0x7f248806f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 -- 172.21.15.29:0/945463736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24881fa880 con 0x7f24880cc2c0 2023-10-16T14:35:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248d9b6640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470003970 0x7f248806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.981+0000 7f248e1b7640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 0x7f2488158e60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f24880cd0f0 tx=0x7f2480001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.982+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480004220 con 0x7f24880cc2c0 2023-10-16T14:35:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.982+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24800043a0 con 0x7f24880cc2c0 2023-10-16T14:35:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.982+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480007310 con 0x7f24880cc2c0 2023-10-16T14:35:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.982+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24881fab60 con 0x7f24880cc2c0 2023-10-16T14:35:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.982+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24881f9ee0 con 0x7f24880cc2c0 2023-10-16T14:35:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.983+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248804e9b0 con 0x7f24880cc2c0 2023-10-16T14:35:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.984+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2480007470 con 0x7f24880cc2c0 2023-10-16T14:35:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.985+0000 7f24867fc640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f247406acf0 0x7f247406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.985+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f2480088510 con 0x7f24880cc2c0 2023-10-16T14:35:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.986+0000 7f248d9b6640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f247406acf0 0x7f247406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.987+0000 7f248d9b6640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f247406acf0 0x7f247406d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2478036000 tx=0x7f247802e000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:09.989+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2480053530 con 0x7f24880cc2c0 2023-10-16T14:35:10.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.219+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24880cf330 con 0x7f24880cc2c0 2023-10-16T14:35:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.220+0000 7f24867fc640 1 -- 172.21.15.29:0/945463736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f2480056b50 con 0x7f24880cc2c0 2023-10-16T14:35:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:10.224 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:45.763311+0000","modified":"2023-10-16T14:33:48.952054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7323},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7323":{"gid":7323,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6836/190249339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":190249339},{"type":"v1","addr":"172.21.15.170:6836","nonce":190249339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T14:35:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2023-10-16T14:35:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f247406acf0 msgr2=0x7f247406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f247406acf0 0x7f247406d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2478036000 tx=0x7f247802e000 comp rx=0 tx=0).stop 2023-10-16T14:35:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 msgr2=0x7f2488158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 0x7f2488158e60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f24880cd0f0 tx=0x7f2480001e70 comp rx=0 tx=0).stop 2023-10-16T14:35:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 shutdown_connections 2023-10-16T14:35:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f247406acf0 0x7f247406d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24880cc2c0 0x7f2488158e60 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24880cbe40 0x7f24880c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 --2- 172.21.15.29:0/945463736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2470003970 0x7f248806f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 >> 172.21.15.29:0/945463736 conn(0x7f24880cf570 msgr2=0x7f24880b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 shutdown_connections 2023-10-16T14:35:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.223+0000 7f248e9b8640 1 -- 172.21.15.29:0/945463736 wait complete. 2023-10-16T14:35:10.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:10.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.461+0000 7f0e030aa640 1 Processor -- start 2023-10-16T14:35:10.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.461+0000 7f0e030aa640 1 -- start start 2023-10-16T14:35:10.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e030aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e030aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0dfc0cda80 0x7f0dfc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e030aa640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc06c960 con 0x7f0dfc0ad970 2023-10-16T14:35:10.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e030aa640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc15b9d0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e030aa640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc15bcb0 con 0x7f0dfc06c040 2023-10-16T14:35:10.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e028a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0dfc0cda80 0x7f0dfc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e028a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0dfc0cda80 0x7f0dfc15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35932/0 (socket says 172.21.15.29:35932) 2023-10-16T14:35:10.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.463+0000 7f0e028a9640 1 -- 172.21.15.29:0/94507648 learned_addr learned my addr 172.21.15.29:0/94507648 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:10.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.464+0000 7f0e028a9640 1 -- 172.21.15.29:0/94507648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 msgr2=0x7f0dfc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:10.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.464+0000 7f0e020a8640 1 --2- 172.21.15.29:0/94507648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.464+0000 7f0e028a9640 1 --2- 172.21.15.29:0/94507648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.464+0000 7f0e028a9640 1 -- 172.21.15.29:0/94507648 >> v1:172.21.15.170:6789/0 conn(0x7f0dfc0ad970 legacy=0x7f0dfc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:10.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.464+0000 7f0e028a9640 1 -- 172.21.15.29:0/94507648 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc15bf90 con 0x7f0dfc0cda80 2023-10-16T14:35:10.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.465+0000 7f0e028a9640 1 --2- 172.21.15.29:0/94507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc15ed90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009870 tx=0x7f0df402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f248863b299ce6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:10.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.465+0000 7f0e010a6640 1 -- 172.21.15.29:0/94507648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4031640 con 0x7f0dfc0cda80 2023-10-16T14:35:10.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.465+0000 7f0e010a6640 1 -- 172.21.15.29:0/94507648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df40317a0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.465+0000 7f0e010a6640 1 -- 172.21.15.29:0/94507648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4031b30 con 0x7f0dfc0cda80 2023-10-16T14:35:10.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 -- 172.21.15.29:0/94507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 msgr2=0x7f0dfc15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 --2- 172.21.15.29:0/94507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc15ed90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009870 tx=0x7f0df402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 -- 172.21.15.29:0/94507648 shutdown_connections 2023-10-16T14:35:10.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 --2- 172.21.15.29:0/94507648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc15ed90 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 --2- 172.21.15.29:0/94507648 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 -- 172.21.15.29:0/94507648 >> 172.21.15.29:0/94507648 conn(0x7f0dfc0ce6a0 msgr2=0x7f0dfc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:10.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 -- 172.21.15.29:0/94507648 shutdown_connections 2023-10-16T14:35:10.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.466+0000 7f0e030aa640 1 -- 172.21.15.29:0/94507648 wait complete. 2023-10-16T14:35:10.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.468+0000 7f0e030aa640 1 Processor -- start 2023-10-16T14:35:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.468+0000 7f0e030aa640 1 -- start start 2023-10-16T14:35:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e030aa640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc1f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e030aa640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0dfc0ad970 0x7f0dfc147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e030aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc0a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e030aa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc1484d0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e030aa640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0ab370 con 0x7f0dfc0ad970 2023-10-16T14:35:10.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e030aa640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0ab650 con 0x7f0dfc06c040 2023-10-16T14:35:10.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc0a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc0a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35934/0 (socket says 172.21.15.29:35934) 2023-10-16T14:35:10.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 -- 172.21.15.29:0/2231038323 learned_addr learned my addr 172.21.15.29:0/2231038323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e020a8640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc1f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 -- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 msgr2=0x7f0dfc1f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc1f0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 -- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0dfc0ad970 msgr2=0x7f0dfc147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e018a7640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0dfc0ad970 0x7f0dfc147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.469+0000 7f0e028a9640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0dfc0ad970 0x7f0dfc147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0e028a9640 1 -- 172.21.15.29:0/2231038323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40094d0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0e028a9640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc0a8c50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0df402b930 tx=0x7f0df40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df403bc70 con 0x7f0dfc0cda80 2023-10-16T14:35:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df403bdf0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4045630 con 0x7f0dfc0cda80 2023-10-16T14:35:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc0ab930 con 0x7f0dfc0cda80 2023-10-16T14:35:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.470+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc0abef0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.472+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc06b330 con 0x7f0dfc0cda80 2023-10-16T14:35:10.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.473+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0df402f050 con 0x7f0dfc0cda80 2023-10-16T14:35:10.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.474+0000 7f0df2ffd640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f0de006acc0 0x7f0de006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.474+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f0df40ab570 con 0x7f0dfc0cda80 2023-10-16T14:35:10.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.475+0000 7f0e020a8640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f0de006acc0 0x7f0de006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.476+0000 7f0e020a8640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f0de006acc0 0x7f0de006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0df8008e50 tx=0x7f0df80083a0 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:10.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.479+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0df4076640 con 0x7f0dfc0cda80 2023-10-16T14:35:10.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.678+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dfc04e9b0 con 0x7f0dfc0cda80 2023-10-16T14:35:10.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.681+0000 7f0df2ffd640 1 -- 172.21.15.29:0/2231038323 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17763 (secure 0 0 0) 0x7f0df4031e60 con 0x7f0dfc0cda80 2023-10-16T14:35:10.684 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:10.685 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":115,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:08.530977+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.26","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2638410402":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/813256058":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/3321905509":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/3034262782":"2023-10-17T14:35:08.530942+0000","172.21.15.170:6832/3649833910":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2734992949":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/388278811":"2023-10-17T14:35:08.530942+0000","172.21.15.170:6833/3649833910":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2765694912":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/3476448138":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1360555952":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2716588410":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1873966041":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/261609677":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6833/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2811499332":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6832/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1477766173":"2023-10-17T14:34:00.590880+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:10.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.683+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f0de006acc0 msgr2=0x7f0de006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.683+0000 7f0e030aa640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f0de006acc0 0x7f0de006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0df8008e50 tx=0x7f0df80083a0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 msgr2=0x7f0dfc0a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc0a8c50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0df402b930 tx=0x7f0df40046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 shutdown_connections 2023-10-16T14:35:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f0de006acc0 0x7f0de006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dfc0cda80 0x7f0dfc0a8c50 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0dfc0ad970 0x7f0dfc147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 --2- 172.21.15.29:0/2231038323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0dfc06c040 0x7f0dfc1f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 >> 172.21.15.29:0/2231038323 conn(0x7f0dfc0ce6a0 msgr2=0x7f0dfc15d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:10.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 shutdown_connections 2023-10-16T14:35:10.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.684+0000 7f0e030aa640 1 -- 172.21.15.29:0/2231038323 wait complete. 2023-10-16T14:35:10.707 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:35:10.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:10.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.925+0000 7f415359e640 1 Processor -- start 2023-10-16T14:35:10.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.926+0000 7f415359e640 1 -- start start 2023-10-16T14:35:10.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f415359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f415415cbe0 0x7f41540d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f415359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f415415cfc0 0x7f41540b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f415359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540d1090 0x7f41540b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f415359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41540d15a0 con 0x7f415415cfc0 2023-10-16T14:35:10.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f415359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41540d1880 con 0x7f415415cbe0 2023-10-16T14:35:10.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f415359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41540d1b60 con 0x7f41540d1090 2023-10-16T14:35:10.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.927+0000 7f4151d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f415415cfc0 0x7f41540b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f415415cfc0 0x7f41540b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35942/0 (socket says 172.21.15.29:35942) 2023-10-16T14:35:10.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 -- 172.21.15.29:0/2345699989 learned_addr learned my addr 172.21.15.29:0/2345699989 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:10.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4152d9d640 1 --2- 172.21.15.29:0/2345699989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540d1090 0x7f41540b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f415259c640 1 --2- 172.21.15.29:0/2345699989 >> v2:172.21.15.170:3300/0 conn(0x7f415415cbe0 0x7f41540d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 -- 172.21.15.29:0/2345699989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540d1090 msgr2=0x7f41540b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 --2- 172.21.15.29:0/2345699989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540d1090 0x7f41540b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 -- 172.21.15.29:0/2345699989 >> v2:172.21.15.170:3300/0 conn(0x7f415415cbe0 msgr2=0x7f41540d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 --2- 172.21.15.29:0/2345699989 >> v2:172.21.15.170:3300/0 conn(0x7f415415cbe0 0x7f41540d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.928+0000 7f4151d9b640 1 -- 172.21.15.29:0/2345699989 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41540d1e40 con 0x7f415415cfc0 2023-10-16T14:35:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.929+0000 7f4151d9b640 1 --2- 172.21.15.29:0/2345699989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cfc0 0x7f41540b4570 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f41540b53f0 tx=0x7f414802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2180beb790cfad35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.929+0000 7f415159a640 1 -- 172.21.15.29:0/2345699989 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148030860 con 0x7f415415cfc0 2023-10-16T14:35:10.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.929+0000 7f415159a640 1 -- 172.21.15.29:0/2345699989 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41480309c0 con 0x7f415415cfc0 2023-10-16T14:35:10.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.929+0000 7f415159a640 1 -- 172.21.15.29:0/2345699989 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148030d50 con 0x7f415415cfc0 2023-10-16T14:35:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 -- 172.21.15.29:0/2345699989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cfc0 msgr2=0x7f41540b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 --2- 172.21.15.29:0/2345699989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cfc0 0x7f41540b4570 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f41540b53f0 tx=0x7f414802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 -- 172.21.15.29:0/2345699989 shutdown_connections 2023-10-16T14:35:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 --2- 172.21.15.29:0/2345699989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540d1090 0x7f41540b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 --2- 172.21.15.29:0/2345699989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cfc0 0x7f41540b4570 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 --2- 172.21.15.29:0/2345699989 >> v2:172.21.15.170:3300/0 conn(0x7f415415cbe0 0x7f41540d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 -- 172.21.15.29:0/2345699989 >> 172.21.15.29:0/2345699989 conn(0x7f41540cf570 msgr2=0x7f41540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 -- 172.21.15.29:0/2345699989 shutdown_connections 2023-10-16T14:35:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.930+0000 7f415359e640 1 -- 172.21.15.29:0/2345699989 wait complete. 2023-10-16T14:35:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.932+0000 7f415359e640 1 Processor -- start 2023-10-16T14:35:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.932+0000 7f415359e640 1 -- start start 2023-10-16T14:35:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f415359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540d1090 0x7f41540c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f415359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 0x7f41540cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f415359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415cfc0 0x7f41540cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f415359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c8b10 con 0x7f415415cbe0 2023-10-16T14:35:10.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f415359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c8df0 con 0x7f41540d1090 2023-10-16T14:35:10.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f415359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c90d0 con 0x7f415415cfc0 2023-10-16T14:35:10.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f4151d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 0x7f41540cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f4151d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 0x7f41540cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35954/0 (socket says 172.21.15.29:35954) 2023-10-16T14:35:10.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.933+0000 7f4151d9b640 1 -- 172.21.15.29:0/1394946006 learned_addr learned my addr 172.21.15.29:0/1394946006 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:10.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4151d9b640 1 -- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415cfc0 msgr2=0x7f41540cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4151d9b640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415cfc0 0x7f41540cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4151d9b640 1 -- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540d1090 msgr2=0x7f41540c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:10.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4151d9b640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540d1090 0x7f41540c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:10.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4151d9b640 1 -- 172.21.15.29:0/1394946006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41480094d0 con 0x7f415415cbe0 2023-10-16T14:35:10.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4151d9b640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 0x7f41540cd370 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f414802b9e0 tx=0x7f4148004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:10.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148039b60 con 0x7f415415cbe0 2023-10-16T14:35:10.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4148039ce0 con 0x7f415415cbe0 2023-10-16T14:35:10.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148043650 con 0x7f415415cbe0 2023-10-16T14:35:10.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.934+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41540c93b0 con 0x7f415415cbe0 2023-10-16T14:35:10.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.935+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41540c6770 con 0x7f415415cbe0 2023-10-16T14:35:10.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.936+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41540c6900 con 0x7f415415cbe0 2023-10-16T14:35:10.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.937+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f414802f3b0 con 0x7f415415cbe0 2023-10-16T14:35:10.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.937+0000 7f4142ffd640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413406acc0 0x7f413406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:10.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.938+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f41480ab8a0 con 0x7f415415cbe0 2023-10-16T14:35:10.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.938+0000 7f415259c640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413406acc0 0x7f413406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:10.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.939+0000 7f415259c640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413406acc0 0x7f413406d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4144006b10 tx=0x7f4144008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:10.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:10.942+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41480768c0 con 0x7f415415cbe0 2023-10-16T14:35:11.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.174+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f415404e9b0 con 0x7f415415cbe0 2023-10-16T14:35:11.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.176+0000 7f4142ffd640 1 -- 172.21.15.29:0/1394946006 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f4148034090 con 0x7f415415cbe0 2023-10-16T14:35:11.179 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:11.179 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:45.763311+0000","modified":"2023-10-16T14:33:48.952054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7323},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7323":{"gid":7323,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6836/190249339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":190249339},{"type":"v1","addr":"172.21.15.170:6836","nonce":190249339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T14:35:11.179 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2023-10-16T14:35:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413406acc0 msgr2=0x7f413406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413406acc0 0x7f413406d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4144006b10 tx=0x7f4144008040 comp rx=0 tx=0).stop 2023-10-16T14:35:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 msgr2=0x7f41540cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 0x7f41540cd370 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f414802b9e0 tx=0x7f4148004730 comp rx=0 tx=0).stop 2023-10-16T14:35:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 shutdown_connections 2023-10-16T14:35:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413406acc0 0x7f413406d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415415cbe0 0x7f41540cd370 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415cfc0 0x7f41540cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 --2- 172.21.15.29:0/1394946006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540d1090 0x7f41540c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.178+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 >> 172.21.15.29:0/1394946006 conn(0x7f41540cf570 msgr2=0x7f41540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.179+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 shutdown_connections 2023-10-16T14:35:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.179+0000 7f415359e640 1 -- 172.21.15.29:0/1394946006 wait complete. 2023-10-16T14:35:11.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.411+0000 7fae33840640 1 Processor -- start 2023-10-16T14:35:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.412+0000 7fae33840640 1 -- start start 2023-10-16T14:35:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae33840640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fae2c15aaa0 0x7fae2c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae33840640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae2c0ce390 0x7fae2c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae33840640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c15ed60 con 0x7fae2c15ae80 2023-10-16T14:35:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae33840640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2c06b810 con 0x7fae2c0ce390 2023-10-16T14:35:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae33840640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2c0707e0 con 0x7fae2c15aaa0 2023-10-16T14:35:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae31ddd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae2c0ce390 0x7fae2c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae31ddd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae2c0ce390 0x7fae2c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35956/0 (socket says 172.21.15.29:35956) 2023-10-16T14:35:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.413+0000 7fae31ddd640 1 -- 172.21.15.29:0/2166544143 learned_addr learned my addr 172.21.15.29:0/2166544143 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.414+0000 7fae315dc640 1 --2- 172.21.15.29:0/2166544143 >> v2:172.21.15.170:3300/0 conn(0x7fae2c15aaa0 0x7fae2c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.414+0000 7fae31ddd640 1 -- 172.21.15.29:0/2166544143 >> v2:172.21.15.170:3300/0 conn(0x7fae2c15aaa0 msgr2=0x7fae2c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.414+0000 7fae31ddd640 1 --2- 172.21.15.29:0/2166544143 >> v2:172.21.15.170:3300/0 conn(0x7fae2c15aaa0 0x7fae2c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.414+0000 7fae31ddd640 1 -- 172.21.15.29:0/2166544143 >> v1:172.21.15.170:6789/0 conn(0x7fae2c15ae80 legacy=0x7fae2c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.414+0000 7fae31ddd640 1 -- 172.21.15.29:0/2166544143 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c15ef90 con 0x7fae2c0ce390 2023-10-16T14:35:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.415+0000 7fae31ddd640 1 --2- 172.21.15.29:0/2166544143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c0ce390 0x7fae2c070f50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0cd990 tx=0x7fae2402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bfbf53c85656c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae23fff640 1 -- 172.21.15.29:0/2166544143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2402ca80 con 0x7fae2c0ce390 2023-10-16T14:35:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae23fff640 1 -- 172.21.15.29:0/2166544143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2402cbe0 con 0x7fae2c0ce390 2023-10-16T14:35:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae23fff640 1 -- 172.21.15.29:0/2166544143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae240343f0 con 0x7fae2c0ce390 2023-10-16T14:35:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae33840640 1 -- 172.21.15.29:0/2166544143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c0ce390 msgr2=0x7fae2c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae33840640 1 --2- 172.21.15.29:0/2166544143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c0ce390 0x7fae2c070f50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0cd990 tx=0x7fae2402bee0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae33840640 1 -- 172.21.15.29:0/2166544143 shutdown_connections 2023-10-16T14:35:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae33840640 1 --2- 172.21.15.29:0/2166544143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c0ce390 0x7fae2c070f50 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae33840640 1 --2- 172.21.15.29:0/2166544143 >> v2:172.21.15.170:3300/0 conn(0x7fae2c15aaa0 0x7fae2c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.416+0000 7fae33840640 1 -- 172.21.15.29:0/2166544143 >> 172.21.15.29:0/2166544143 conn(0x7fae2c0cb410 msgr2=0x7fae2c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.417+0000 7fae33840640 1 -- 172.21.15.29:0/2166544143 shutdown_connections 2023-10-16T14:35:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.417+0000 7fae33840640 1 -- 172.21.15.29:0/2166544143 wait complete. 2023-10-16T14:35:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.418+0000 7fae33840640 1 Processor -- start 2023-10-16T14:35:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.418+0000 7fae33840640 1 -- start start 2023-10-16T14:35:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.419+0000 7fae33840640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae2c0ce390 0x7fae2c14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.419+0000 7fae33840640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae2c15aaa0 0x7fae2c0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.419+0000 7fae33840640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 0x7fae2c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.419+0000 7fae33840640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0af320 con 0x7fae2c15ae80 2023-10-16T14:35:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.419+0000 7fae33840640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0af600 con 0x7fae2c15aaa0 2023-10-16T14:35:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.419+0000 7fae33840640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0af8e0 con 0x7fae2c0ce390 2023-10-16T14:35:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 0x7fae2c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 0x7fae2c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35966/0 (socket says 172.21.15.29:35966) 2023-10-16T14:35:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 -- 172.21.15.29:0/2102314306 learned_addr learned my addr 172.21.15.29:0/2102314306 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae30ddb640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae2c15aaa0 0x7fae2c0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 -- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae2c0ce390 msgr2=0x7fae2c14ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae2c0ce390 0x7fae2c14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 -- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae2c15aaa0 msgr2=0x7fae2c0c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae2c15aaa0 0x7fae2c0c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.420+0000 7fae31ddd640 1 -- 172.21.15.29:0/2102314306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae24009b30 con 0x7fae2c15ae80 2023-10-16T14:35:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.421+0000 7fae31ddd640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 0x7fae2c0ac890 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fae24036040 tx=0x7fae24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.421+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24009510 con 0x7fae2c15ae80 2023-10-16T14:35:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.421+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2403e450 con 0x7fae2c15ae80 2023-10-16T14:35:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.421+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2403e7e0 con 0x7fae2c15ae80 2023-10-16T14:35:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.421+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c0afbc0 con 0x7fae2c15ae80 2023-10-16T14:35:11.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.421+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c0b00e0 con 0x7fae2c15ae80 2023-10-16T14:35:11.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.422+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c0afde0 con 0x7fae2c15ae80 2023-10-16T14:35:11.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.423+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fae2402cd50 con 0x7fae2c15ae80 2023-10-16T14:35:11.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.424+0000 7fae21ffb640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fae1006af30 0x7fae1006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.424+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fae240ac210 con 0x7fae2c15ae80 2023-10-16T14:35:11.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.425+0000 7fae315dc640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fae1006af30 0x7fae1006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.426+0000 7fae315dc640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fae1006af30 0x7fae1006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fae280098f0 tx=0x7fae28008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:11.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.429+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae24076f60 con 0x7fae2c15ae80 2023-10-16T14:35:11.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.664+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae2c0af460 con 0x7fae2c15ae80 2023-10-16T14:35:11.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.665+0000 7fae21ffb640 1 -- 172.21.15.29:0/2102314306 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fae2c0af460 con 0x7fae2c15ae80 2023-10-16T14:35:11.668 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:11.669 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:33:45.763311+0000","modified":"2023-10-16T14:33:48.952054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7323},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7323":{"gid":7323,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.170:6836/190249339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":190249339},{"type":"v1","addr":"172.21.15.170:6836","nonce":190249339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T14:35:11.669 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2023-10-16T14:35:11.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fae1006af30 msgr2=0x7fae1006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fae1006af30 0x7fae1006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fae280098f0 tx=0x7fae28008040 comp rx=0 tx=0).stop 2023-10-16T14:35:11.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 msgr2=0x7fae2c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 0x7fae2c0ac890 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fae24036040 tx=0x7fae24004a40 comp rx=0 tx=0).stop 2023-10-16T14:35:11.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 shutdown_connections 2023-10-16T14:35:11.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fae1006af30 0x7fae1006d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae2c15ae80 0x7fae2c0ac890 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae2c15aaa0 0x7fae2c0c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 --2- 172.21.15.29:0/2102314306 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae2c0ce390 0x7fae2c14ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 >> 172.21.15.29:0/2102314306 conn(0x7fae2c0cb410 msgr2=0x7fae2c158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:11.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 shutdown_connections 2023-10-16T14:35:11.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.668+0000 7fae33840640 1 -- 172.21.15.29:0/2102314306 wait complete. 2023-10-16T14:35:11.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.908+0000 7f78054b4640 1 Processor -- start 2023-10-16T14:35:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.909+0000 7f78054b4640 1 -- start start 2023-10-16T14:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f78054b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f780006c040 0x7f780006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f78054b4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f78000cda80 0x7f780015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f78054b4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780006c960 con 0x7f78000ad970 2023-10-16T14:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f78054b4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f780015b9d0 con 0x7f780006c040 2023-10-16T14:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f78054b4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f780015bcb0 con 0x7f78000cda80 2023-10-16T14:35:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f77fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f780006c040 0x7f780006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f77fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f780006c040 0x7f780006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35980/0 (socket says 172.21.15.29:35980) 2023-10-16T14:35:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.910+0000 7f77fffff640 1 -- 172.21.15.29:0/2125804741 learned_addr learned my addr 172.21.15.29:0/2125804741 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.911+0000 7f77fffff640 1 -- 172.21.15.29:0/2125804741 >> v2:172.21.15.170:3300/0 conn(0x7f78000cda80 msgr2=0x7f780015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.911+0000 7f77fffff640 1 --2- 172.21.15.29:0/2125804741 >> v2:172.21.15.170:3300/0 conn(0x7f78000cda80 0x7f780015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.911+0000 7f77fffff640 1 -- 172.21.15.29:0/2125804741 >> v1:172.21.15.170:6789/0 conn(0x7f78000ad970 legacy=0x7f780015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.911+0000 7f77fffff640 1 -- 172.21.15.29:0/2125804741 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780015bf90 con 0x7f780006c040 2023-10-16T14:35:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.912+0000 7f77fffff640 1 --2- 172.21.15.29:0/2125804741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f780006c040 0x7f780006c420 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f78000cf300 tx=0x7f77f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9bc4b59ab381cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:11.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.912+0000 7f77ff7fe640 1 -- 172.21.15.29:0/2125804741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0031870 con 0x7f780006c040 2023-10-16T14:35:11.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.912+0000 7f77ff7fe640 1 -- 172.21.15.29:0/2125804741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f00319d0 con 0x7f780006c040 2023-10-16T14:35:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.912+0000 7f77ff7fe640 1 -- 172.21.15.29:0/2125804741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0031d60 con 0x7f780006c040 2023-10-16T14:35:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.913+0000 7f78054b4640 1 -- 172.21.15.29:0/2125804741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f780006c040 msgr2=0x7f780006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.913+0000 7f78054b4640 1 --2- 172.21.15.29:0/2125804741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f780006c040 0x7f780006c420 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f78000cf300 tx=0x7f77f002b910 comp rx=0 tx=0).stop 2023-10-16T14:35:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.913+0000 7f78054b4640 1 -- 172.21.15.29:0/2125804741 shutdown_connections 2023-10-16T14:35:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.913+0000 7f78054b4640 1 --2- 172.21.15.29:0/2125804741 >> v2:172.21.15.170:3300/0 conn(0x7f78000cda80 0x7f780015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.913+0000 7f78054b4640 1 --2- 172.21.15.29:0/2125804741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f780006c040 0x7f780006c420 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.913+0000 7f78054b4640 1 -- 172.21.15.29:0/2125804741 >> 172.21.15.29:0/2125804741 conn(0x7f78000ce6a0 msgr2=0x7f78000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.914+0000 7f78054b4640 1 -- 172.21.15.29:0/2125804741 shutdown_connections 2023-10-16T14:35:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.914+0000 7f78054b4640 1 -- 172.21.15.29:0/2125804741 wait complete. 2023-10-16T14:35:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.916+0000 7f78054b4640 1 Processor -- start 2023-10-16T14:35:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.916+0000 7f78054b4640 1 -- start start 2023-10-16T14:35:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.917+0000 7f78054b4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f780006c040 0x7f7800156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.917+0000 7f78054b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 0x7f78000c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.917+0000 7f78054b4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78000cda80 0x7f78000ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.917+0000 7f78054b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000c7cb0 con 0x7f78000ad970 2023-10-16T14:35:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.917+0000 7f78054b4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000c2d80 con 0x7f780006c040 2023-10-16T14:35:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.917+0000 7f78054b4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000c3060 con 0x7f78000cda80 2023-10-16T14:35:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 0x7f78000c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 0x7f78000c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35990/0 (socket says 172.21.15.29:35990) 2023-10-16T14:35:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 -- 172.21.15.29:0/1435720930 learned_addr learned my addr 172.21.15.29:0/1435720930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77fffff640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f780006c040 0x7f7800156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f7804cb3640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78000cda80 0x7f78000ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 -- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78000cda80 msgr2=0x7f78000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78000cda80 0x7f78000ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 -- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f780006c040 msgr2=0x7f7800156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f780006c040 0x7f7800156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.918+0000 7f77f7fff640 1 -- 172.21.15.29:0/1435720930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00094d0 con 0x7f78000ad970 2023-10-16T14:35:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.919+0000 7f77f7fff640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 0x7f78000c3b40 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f77e800c400 tx=0x7f77e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.919+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e80035f0 con 0x7f78000ad970 2023-10-16T14:35:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.919+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e8003750 con 0x7f78000ad970 2023-10-16T14:35:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.919+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8003b40 con 0x7f78000ad970 2023-10-16T14:35:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.919+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78000c3340 con 0x7f78000ad970 2023-10-16T14:35:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.919+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7800156db0 con 0x7f78000ad970 2023-10-16T14:35:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.920+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780006b330 con 0x7f78000ad970 2023-10-16T14:35:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.922+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f77e8003ca0 con 0x7f78000ad970 2023-10-16T14:35:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.923+0000 7f77fd7fa640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f77e006af00 0x7f77e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.923+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f77e8088c10 con 0x7f78000ad970 2023-10-16T14:35:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.924+0000 7f77fffff640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f77e006af00 0x7f77e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.925+0000 7f77fffff640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f77e006af00 0x7f77e006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f77f0036000 tx=0x7f77f003c040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:11.928+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f77e8053980 con 0x7f78000ad970 2023-10-16T14:35:12.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.123+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7800156fd0 con 0x7f78000ad970 2023-10-16T14:35:12.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.126+0000 7f77fd7fa640 1 -- 172.21.15.29:0/1435720930 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17763 (secure 0 0 0) 0x7f77e8056fa0 con 0x7f78000ad970 2023-10-16T14:35:12.130 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:12.130 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":115,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:08.530977+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:33:43.719274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T14:33:44.790872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.26","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2638410402":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/813256058":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/3321905509":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/3034262782":"2023-10-17T14:35:08.530942+0000","172.21.15.170:6832/3649833910":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2734992949":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/388278811":"2023-10-17T14:35:08.530942+0000","172.21.15.170:6833/3649833910":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2765694912":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/3476448138":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1360555952":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2716588410":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1873966041":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/261609677":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6833/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2811499332":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6832/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1477766173":"2023-10-17T14:34:00.590880+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:12.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f77e006af00 msgr2=0x7f77e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f77e006af00 0x7f77e006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f77f0036000 tx=0x7f77f003c040 comp rx=0 tx=0).stop 2023-10-16T14:35:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 msgr2=0x7f78000c3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 0x7f78000c3b40 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f77e800c400 tx=0x7f77e800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 shutdown_connections 2023-10-16T14:35:12.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f77e006af00 0x7f77e006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78000ad970 0x7f78000c3b40 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78000cda80 0x7f78000ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 --2- 172.21.15.29:0/1435720930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f780006c040 0x7f7800156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.129+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 >> 172.21.15.29:0/1435720930 conn(0x7f78000ce6a0 msgr2=0x7f780006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:12.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.130+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 shutdown_connections 2023-10-16T14:35:12.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.130+0000 7f78054b4640 1 -- 172.21.15.29:0/1435720930 wait complete. 2023-10-16T14:35:12.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:35:12.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.374+0000 7f60bb600640 1 Processor -- start 2023-10-16T14:35:12.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.374+0000 7f60bb600640 1 -- start start 2023-10-16T14:35:12.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.375+0000 7f60bb600640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d410 0x7f60b40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.375+0000 7f60bb600640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f60b40cd260 0x7f60b40cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60bb600640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40b5d90 con 0x7f60b415d030 2023-10-16T14:35:12.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60bb600640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b40d2b50 con 0x7f60b40cd260 2023-10-16T14:35:12.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60bb600640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4160fc0 con 0x7f60b415d410 2023-10-16T14:35:12.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60b939c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f60b415d030 0x7f60b4161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:42418/0 (socket says 172.21.15.29:42418) 2023-10-16T14:35:12.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60b9b9d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f60b40cd260 0x7f60b40cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60b939c640 1 -- 172.21.15.29:0/4254277227 learned_addr learned my addr 172.21.15.29:0/4254277227 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:12.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.376+0000 7f60b8b9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d410 0x7f60b40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.377+0000 7f60b8b9b640 1 -- 172.21.15.29:0/4254277227 >> v2:172.21.15.170:3300/0 conn(0x7f60b40cd260 msgr2=0x7f60b40cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:12.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.377+0000 7f60b8b9b640 1 --2- 172.21.15.29:0/4254277227 >> v2:172.21.15.170:3300/0 conn(0x7f60b40cd260 0x7f60b40cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.377+0000 7f60b8b9b640 1 -- 172.21.15.29:0/4254277227 >> v1:172.21.15.170:6789/0 conn(0x7f60b415d030 legacy=0x7f60b4161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.377+0000 7f60b8b9b640 1 -- 172.21.15.29:0/4254277227 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40d25d0 con 0x7f60b415d410 2023-10-16T14:35:12.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.377+0000 7f60abfff640 1 -- 172.21.15.29:0/4254277227 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403987081 0 0) 0x7f60b40b5d90 con 0x7f60b415d030 2023-10-16T14:35:12.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.379+0000 7f60b8b9b640 1 --2- 172.21.15.29:0/4254277227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d410 0x7f60b40d1630 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c26f6a333a2ae2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:12.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.380+0000 7f60abfff640 1 -- 172.21.15.29:0/4254277227 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4030800 con 0x7f60b415d410 2023-10-16T14:35:12.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.380+0000 7f60abfff640 1 -- 172.21.15.29:0/4254277227 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4030960 con 0x7f60b415d410 2023-10-16T14:35:12.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.380+0000 7f60abfff640 1 -- 172.21.15.29:0/4254277227 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4030cf0 con 0x7f60b415d410 2023-10-16T14:35:12.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.380+0000 7f60bb600640 1 -- 172.21.15.29:0/4254277227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d410 msgr2=0x7f60b40d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.380+0000 7f60bb600640 1 --2- 172.21.15.29:0/4254277227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d410 0x7f60b40d1630 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b940 comp rx=0 tx=0).stop 2023-10-16T14:35:12.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.381+0000 7f60bb600640 1 -- 172.21.15.29:0/4254277227 shutdown_connections 2023-10-16T14:35:12.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.381+0000 7f60bb600640 1 --2- 172.21.15.29:0/4254277227 >> v2:172.21.15.170:3300/0 conn(0x7f60b40cd260 0x7f60b40cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.381+0000 7f60bb600640 1 --2- 172.21.15.29:0/4254277227 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d410 0x7f60b40d1630 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.381+0000 7f60bb600640 1 -- 172.21.15.29:0/4254277227 >> 172.21.15.29:0/4254277227 conn(0x7f60b40cf720 msgr2=0x7f60b40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:12.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.381+0000 7f60bb600640 1 -- 172.21.15.29:0/4254277227 shutdown_connections 2023-10-16T14:35:12.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.381+0000 7f60bb600640 1 -- 172.21.15.29:0/4254277227 wait complete. 2023-10-16T14:35:12.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.383+0000 7f60bb600640 1 Processor -- start 2023-10-16T14:35:12.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.384+0000 7f60bb600640 1 -- start start 2023-10-16T14:35:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60bb600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 0x7f60b415a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60bb600640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d030 0x7f60b415e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60bb600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60b415d410 0x7f60b40c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60bb600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40c9e80 con 0x7f60b40cd260 2023-10-16T14:35:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60bb600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40ca160 con 0x7f60b415d410 2023-10-16T14:35:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60bb600640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40ca440 con 0x7f60b415d030 2023-10-16T14:35:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60b939c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 0x7f60b415a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60b939c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 0x7f60b415a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36006/0 (socket says 172.21.15.29:36006) 2023-10-16T14:35:12.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.385+0000 7f60b939c640 1 -- 172.21.15.29:0/3751790622 learned_addr learned my addr 172.21.15.29:0/3751790622 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b8b9b640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d030 0x7f60b415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b939c640 1 -- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d030 msgr2=0x7f60b415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b939c640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d030 0x7f60b415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b939c640 1 -- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60b415d410 msgr2=0x7f60b40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b9b9d640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60b415d410 0x7f60b40c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b939c640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60b415d410 0x7f60b40c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b939c640 1 -- 172.21.15.29:0/3751790622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a4009400 con 0x7f60b40cd260 2023-10-16T14:35:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b9b9d640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60b415d410 0x7f60b40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60b939c640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 0x7f60b415a3a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f60b000c4d0 tx=0x7f60b000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:12.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00074e0 con 0x7f60b40cd260 2023-10-16T14:35:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b0007660 con 0x7f60b40cd260 2023-10-16T14:35:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b40ca720 con 0x7f60b40cd260 2023-10-16T14:35:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.386+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0004a20 con 0x7f60b40cd260 2023-10-16T14:35:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.387+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b415ec40 con 0x7f60b40cd260 2023-10-16T14:35:12.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.388+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b404e160 con 0x7f60b40cd260 2023-10-16T14:35:12.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.389+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f60b0004b80 con 0x7f60b40cd260 2023-10-16T14:35:12.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.390+0000 7f60a9ffb640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f609806af00 0x7f609806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.390+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f60b0088ed0 con 0x7f60b40cd260 2023-10-16T14:35:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.390+0000 7f60b8b9b640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f609806af00 0x7f609806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.391+0000 7f60b8b9b640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f609806af00 0x7f609806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f60a4036000 tx=0x7f60a403b040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:12.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.394+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f60b0053c20 con 0x7f60b40cd260 2023-10-16T14:35:12.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.580+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f60b0053ff0 con 0x7f60b40cd260 2023-10-16T14:35:12.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.633+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f60b40c9fc0 con 0x7f60b40cd260 2023-10-16T14:35:12.637 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:35:12.634+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:35:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.638+0000 7f60a9ffb640 1 -- 172.21.15.29:0/3751790622 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7f60b40c9fc0 con 0x7f60b40cd260 2023-10-16T14:35:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:35:12.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f609806af00 msgr2=0x7f609806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f609806af00 0x7f609806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f60a4036000 tx=0x7f60a403b040 comp rx=0 tx=0).stop 2023-10-16T14:35:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 msgr2=0x7f60b415a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 0x7f60b415a3a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f60b000c4d0 tx=0x7f60b000c990 comp rx=0 tx=0).stop 2023-10-16T14:35:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 shutdown_connections 2023-10-16T14:35:12.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f609806af00 0x7f609806d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60b415d410 0x7f60b40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60b415d030 0x7f60b415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 --2- 172.21.15.29:0/3751790622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b40cd260 0x7f60b415a3a0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.640+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 >> 172.21.15.29:0/3751790622 conn(0x7f60b40cf720 msgr2=0x7f60b415c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:12.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.641+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 shutdown_connections 2023-10-16T14:35:12.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.641+0000 7f60bb600640 1 -- 172.21.15.29:0/3751790622 wait complete. 2023-10-16T14:35:12.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:35:12.680 INFO:tasks.ceph.mds.d.smithi170.stderr: -6926> 2023-10-16T14:33:50.252+0000 7f9edd03c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:35:12.752 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:35:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.893+0000 7f2ecf2a9640 1 Processor -- start 2023-10-16T14:35:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.894+0000 7f2ecf2a9640 1 -- start start 2023-10-16T14:35:12.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.895+0000 7f2ecf2a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.895+0000 7f2ecf2a9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2ec80c9150 0x7f2ec80c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.895+0000 7f2ecf2a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ec806b660 0x7f2ec806baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.895+0000 7f2ecf2a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec80c9af0 con 0x7f2ec806b660 2023-10-16T14:35:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.895+0000 7f2ecf2a9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec80cc6b0 con 0x7f2ec80c9150 2023-10-16T14:35:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.895+0000 7f2ecf2a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec806c030 con 0x7f2ec81428c0 2023-10-16T14:35:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ec806b660 0x7f2ec806baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ec806b660 0x7f2ec806baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36010/0 (socket says 172.21.15.29:36010) 2023-10-16T14:35:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 -- 172.21.15.29:0/3948576876 learned_addr learned my addr 172.21.15.29:0/3948576876 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd045640 1 --2- 172.21.15.29:0/3948576876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecc844640 1 --2- 172.21.15.29:0/3948576876 >> v2:172.21.15.170:3300/0 conn(0x7f2ec80c9150 0x7f2ec80c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 -- 172.21.15.29:0/3948576876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 msgr2=0x7f2ec8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 --2- 172.21.15.29:0/3948576876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 -- 172.21.15.29:0/3948576876 >> v2:172.21.15.170:3300/0 conn(0x7f2ec80c9150 msgr2=0x7f2ec80c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 --2- 172.21.15.29:0/3948576876 >> v2:172.21.15.170:3300/0 conn(0x7f2ec80c9150 0x7f2ec80c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.896+0000 7f2ecd846640 1 -- 172.21.15.29:0/3948576876 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec81599b0 con 0x7f2ec806b660 2023-10-16T14:35:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.898+0000 7f2ecd846640 1 --2- 172.21.15.29:0/3948576876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec806baf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80cf820 tx=0x7f2ec002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812f9539311f9d88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.898+0000 7f2ebbfff640 1 -- 172.21.15.29:0/3948576876 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec002ca70 con 0x7f2ec806b660 2023-10-16T14:35:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.898+0000 7f2ebbfff640 1 -- 172.21.15.29:0/3948576876 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec002cbd0 con 0x7f2ec806b660 2023-10-16T14:35:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.898+0000 7f2ebbfff640 1 -- 172.21.15.29:0/3948576876 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00343f0 con 0x7f2ec806b660 2023-10-16T14:35:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/3948576876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 msgr2=0x7f2ec806baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/3948576876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec806baf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80cf820 tx=0x7f2ec002bed0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/3948576876 shutdown_connections 2023-10-16T14:35:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/3948576876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec806baf0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/3948576876 >> v2:172.21.15.170:3300/0 conn(0x7f2ec80c9150 0x7f2ec80c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/3948576876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/3948576876 >> 172.21.15.29:0/3948576876 conn(0x7f2ec80ce9b0 msgr2=0x7f2ec80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/3948576876 shutdown_connections 2023-10-16T14:35:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.899+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/3948576876 wait complete. 2023-10-16T14:35:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.901+0000 7f2ecf2a9640 1 Processor -- start 2023-10-16T14:35:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.901+0000 7f2ecf2a9640 1 -- start start 2023-10-16T14:35:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecf2a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec80c0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecf2a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ec80c9150 0x7f2ec81416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecf2a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecf2a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec81507d0 con 0x7f2ec806b660 2023-10-16T14:35:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecf2a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec815aa60 con 0x7f2ec80c9150 2023-10-16T14:35:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecf2a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec815acf0 con 0x7f2ec81428c0 2023-10-16T14:35:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecd045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec80c0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecd045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec80c0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36022/0 (socket says 172.21.15.29:36022) 2023-10-16T14:35:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.903+0000 7f2ecd045640 1 -- 172.21.15.29:0/1099839365 learned_addr learned my addr 172.21.15.29:0/1099839365 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecc844640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ec80c9150 0x7f2ec81416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd846640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd045640 1 -- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 msgr2=0x7f2ec8141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd045640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd045640 1 -- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ec80c9150 msgr2=0x7f2ec81416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd045640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ec80c9150 0x7f2ec81416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd045640 1 -- 172.21.15.29:0/1099839365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0009b30 con 0x7f2ec806b660 2023-10-16T14:35:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.904+0000 7f2ecd045640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec80c0310 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00d800 tx=0x7f2ebc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.905+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00b6c0 con 0x7f2ec806b660 2023-10-16T14:35:12.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.905+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc00b820 con 0x7f2ec806b660 2023-10-16T14:35:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.905+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec815afd0 con 0x7f2ec806b660 2023-10-16T14:35:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.905+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00bbb0 con 0x7f2ec806b660 2023-10-16T14:35:12.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.905+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec815b590 con 0x7f2ec806b660 2023-10-16T14:35:12.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.906+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec804e160 con 0x7f2ec806b660 2023-10-16T14:35:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.907+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ebc00bd10 con 0x7f2ec806b660 2023-10-16T14:35:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.908+0000 7f2eb9ffb640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2eac06f3a0 0x7f2eac071860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.908+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f2ebc08a1a0 con 0x7f2ec806b660 2023-10-16T14:35:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.909+0000 7f2ecc844640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2eac06f3a0 0x7f2eac071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.910+0000 7f2ecc844640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2eac06f3a0 0x7f2eac071860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:12.914+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ebc054ca0 con 0x7f2ec806b660 2023-10-16T14:35:13.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.146+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2ec806a9e0 con 0x7f2ec806b660 2023-10-16T14:35:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.641+0000 7f2eb9ffb640 1 -- 172.21.15.29:0/1099839365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2ebc0582c0 con 0x7f2ec806b660 2023-10-16T14:35:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2eac06f3a0 msgr2=0x7f2eac071860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2eac06f3a0 0x7f2eac071860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).stop 2023-10-16T14:35:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 msgr2=0x7f2ec80c0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec80c0310 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00d800 tx=0x7f2ebc00dcc0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 shutdown_connections 2023-10-16T14:35:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2eac06f3a0 0x7f2eac071860 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ec81428c0 0x7f2ec8141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ec80c9150 0x7f2ec81416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 --2- 172.21.15.29:0/1099839365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec806b660 0x7f2ec80c0310 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.645+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 >> 172.21.15.29:0/1099839365 conn(0x7f2ec80ce9b0 msgr2=0x7f2ec80cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.646+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 shutdown_connections 2023-10-16T14:35:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.646+0000 7f2ecf2a9640 1 -- 172.21.15.29:0/1099839365 wait complete. 2023-10-16T14:35:13.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:35:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.895+0000 7f4162759640 1 Processor -- start 2023-10-16T14:35:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.895+0000 7f4162759640 1 -- start start 2023-10-16T14:35:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4162759640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4162759640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f415c0ad970 0x7f415c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4162759640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c15ebc0 0x7f415c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4162759640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415c06c960 con 0x7f415c0ad970 2023-10-16T14:35:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4162759640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415c15b9d0 con 0x7f415c06c040 2023-10-16T14:35:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4162759640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c15bcb0 con 0x7f415c15ebc0 2023-10-16T14:35:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.896+0000 7f4160f56640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f415c0ad970 0x7f415c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f415c0ad970 0x7f415c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45800/0 (socket says 172.21.15.29:45800) 2023-10-16T14:35:13.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 -- 172.21.15.29:0/1296962606 learned_addr learned my addr 172.21.15.29:0/1296962606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 -- 172.21.15.29:0/1296962606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c15ebc0 msgr2=0x7f415c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4161f58640 1 --2- 172.21.15.29:0/1296962606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c15ebc0 0x7f415c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4161757640 1 --2- 172.21.15.29:0/1296962606 >> v2:172.21.15.170:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 --2- 172.21.15.29:0/1296962606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c15ebc0 0x7f415c15f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 -- 172.21.15.29:0/1296962606 >> v2:172.21.15.170:3300/0 conn(0x7f415c06c040 msgr2=0x7f415c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 --2- 172.21.15.29:0/1296962606 >> v2:172.21.15.170:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4160f56640 1 -- 172.21.15.29:0/1296962606 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c15bf90 con 0x7f415c0ad970 2023-10-16T14:35:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.897+0000 7f4161757640 1 --2- 172.21.15.29:0/1296962606 >> v2:172.21.15.170:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.898+0000 7f4160f56640 1 --2- 172.21.15.29:0/1296962606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c0ad970 0x7f415c15e680 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f415c0cf350 tx=0x7f414c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf7771a177c1009 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4153fff640 1 -- 172.21.15.29:0/1296962606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c031870 con 0x7f415c0ad970 2023-10-16T14:35:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4153fff640 1 -- 172.21.15.29:0/1296962606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c0319d0 con 0x7f415c0ad970 2023-10-16T14:35:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4153fff640 1 -- 172.21.15.29:0/1296962606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c031d60 con 0x7f415c0ad970 2023-10-16T14:35:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 -- 172.21.15.29:0/1296962606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c0ad970 msgr2=0x7f415c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 --2- 172.21.15.29:0/1296962606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c0ad970 0x7f415c15e680 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f415c0cf350 tx=0x7f414c02b910 comp rx=0 tx=0).stop 2023-10-16T14:35:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 -- 172.21.15.29:0/1296962606 shutdown_connections 2023-10-16T14:35:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 --2- 172.21.15.29:0/1296962606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c15ebc0 0x7f415c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 --2- 172.21.15.29:0/1296962606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c0ad970 0x7f415c15e680 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 --2- 172.21.15.29:0/1296962606 >> v2:172.21.15.170:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 -- 172.21.15.29:0/1296962606 >> 172.21.15.29:0/1296962606 conn(0x7f415c0ce6a0 msgr2=0x7f415c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 -- 172.21.15.29:0/1296962606 shutdown_connections 2023-10-16T14:35:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.899+0000 7f4162759640 1 -- 172.21.15.29:0/1296962606 wait complete. 2023-10-16T14:35:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.901+0000 7f4162759640 1 Processor -- start 2023-10-16T14:35:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.901+0000 7f4162759640 1 -- start start 2023-10-16T14:35:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.902+0000 7f4162759640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 0x7f415c153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.902+0000 7f4162759640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c0ad970 0x7f415c154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.902+0000 7f4162759640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f415c15ebc0 0x7f415c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.902+0000 7f4162759640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0c7720 con 0x7f415c06c040 2023-10-16T14:35:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.902+0000 7f4162759640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0ca120 con 0x7f415c15ebc0 2023-10-16T14:35:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.902+0000 7f4162759640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0c67c0 con 0x7f415c0ad970 2023-10-16T14:35:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 0x7f415c153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 0x7f415c153d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45812/0 (socket says 172.21.15.29:45812) 2023-10-16T14:35:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 -- 172.21.15.29:0/1057211955 learned_addr learned my addr 172.21.15.29:0/1057211955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161f58640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f415c15ebc0 0x7f415c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4160f56640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c0ad970 0x7f415c154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 -- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c0ad970 msgr2=0x7f415c154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c0ad970 0x7f415c154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 -- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f415c15ebc0 msgr2=0x7f415c159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:13.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f415c15ebc0 0x7f415c159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:13.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.903+0000 7f4161757640 1 -- 172.21.15.29:0/1057211955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c0094d0 con 0x7f415c06c040 2023-10-16T14:35:13.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.904+0000 7f4161757640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 0x7f415c153d00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f415800cee0 tx=0x7f415800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:13.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.904+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415800d840 con 0x7f415c06c040 2023-10-16T14:35:13.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.904+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415800d9a0 con 0x7f415c06c040 2023-10-16T14:35:13.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.904+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415800dd30 con 0x7f415c06c040 2023-10-16T14:35:13.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.904+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c0c6aa0 con 0x7f415c06c040 2023-10-16T14:35:13.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.904+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c0c7060 con 0x7f415c06c040 2023-10-16T14:35:13.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.905+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c06b330 con 0x7f415c06c040 2023-10-16T14:35:13.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.906+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4158015440 con 0x7f415c06c040 2023-10-16T14:35:13.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.907+0000 7f4151ffb640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f414006af90 0x7f414006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:13.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.907+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f415808a3f0 con 0x7f415c06c040 2023-10-16T14:35:13.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.908+0000 7f4160f56640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f414006af90 0x7f414006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:13.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.909+0000 7f4160f56640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f414006af90 0x7f414006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f414c036000 tx=0x7f414c03c040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:13.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:13.911+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41580550b0 con 0x7f415c06c040 2023-10-16T14:35:14.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.103+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f415c04e9b0 con 0x7f415c06c040 2023-10-16T14:35:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.650+0000 7f4151ffb640 1 -- 172.21.15.29:0/1057211955 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v117) v1 ==== 181+0+0 (secure 0 0 0) 0x7f41580586d0 con 0x7f415c06c040 2023-10-16T14:35:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:35:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.653+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f414006af90 msgr2=0x7f414006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.653+0000 7f4162759640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f414006af90 0x7f414006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f414c036000 tx=0x7f414c03c040 comp rx=0 tx=0).stop 2023-10-16T14:35:14.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.653+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 msgr2=0x7f415c153d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:14.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.653+0000 7f4162759640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 0x7f415c153d00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f415800cee0 tx=0x7f415800a710 comp rx=0 tx=0).stop 2023-10-16T14:35:14.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.655+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 shutdown_connections 2023-10-16T14:35:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.655+0000 7f4162759640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f414006af90 0x7f414006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.655+0000 7f4162759640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f415c15ebc0 0x7f415c159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.655+0000 7f4162759640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415c0ad970 0x7f415c154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.655+0000 7f4162759640 1 --2- 172.21.15.29:0/1057211955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f415c06c040 0x7f415c153d00 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.655+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 >> 172.21.15.29:0/1057211955 conn(0x7f415c0ce6a0 msgr2=0x7f415c0cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.657+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 shutdown_connections 2023-10-16T14:35:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.657+0000 7f4162759640 1 -- 172.21.15.29:0/1057211955 wait complete. 2023-10-16T14:35:14.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:35:14.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.907+0000 7fb8b194e640 1 Processor -- start 2023-10-16T14:35:14.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.908+0000 7fb8b194e640 1 -- start start 2023-10-16T14:35:14.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.909+0000 7fb8b194e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac15aaa0 0x7fb8ac0b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.909+0000 7fb8b194e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb8ac0ca8a0 0x7fb8ac0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b194e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8ac070240 0x7fb8ac0706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b194e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac070be0 con 0x7fb8ac070240 2023-10-16T14:35:14.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b194e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac0cad00 con 0x7fb8ac0ca8a0 2023-10-16T14:35:14.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b194e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0b2d00 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b114d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8ac070240 0x7fb8ac0706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:14.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b114d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8ac070240 0x7fb8ac0706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45814/0 (socket says 172.21.15.29:45814) 2023-10-16T14:35:14.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b114d640 1 -- 172.21.15.29:0/3075045214 learned_addr learned my addr 172.21.15.29:0/3075045214 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:14.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8b094c640 1 --2- 172.21.15.29:0/3075045214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac15aaa0 0x7fb8ac0b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:14.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.910+0000 7fb8a3fff640 1 --2- 172.21.15.29:0/3075045214 >> v2:172.21.15.170:3300/0 conn(0x7fb8ac0ca8a0 0x7fb8ac0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.911+0000 7fb8b114d640 1 -- 172.21.15.29:0/3075045214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac15aaa0 msgr2=0x7fb8ac0b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.911+0000 7fb8b114d640 1 --2- 172.21.15.29:0/3075045214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac15aaa0 0x7fb8ac0b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.911+0000 7fb8b114d640 1 -- 172.21.15.29:0/3075045214 >> v2:172.21.15.170:3300/0 conn(0x7fb8ac0ca8a0 msgr2=0x7fb8ac0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.911+0000 7fb8b114d640 1 --2- 172.21.15.29:0/3075045214 >> v2:172.21.15.170:3300/0 conn(0x7fb8ac0ca8a0 0x7fb8ac0b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.911+0000 7fb8b114d640 1 -- 172.21.15.29:0/3075045214 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0b2fe0 con 0x7fb8ac070240 2023-10-16T14:35:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.912+0000 7fb8b114d640 1 --2- 172.21.15.29:0/3075045214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac070240 0x7fb8ac0706a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0cf560 tx=0x7fb8a402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcda2c733bd2c2cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.912+0000 7fb8a37fe640 1 -- 172.21.15.29:0/3075045214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4031860 con 0x7fb8ac070240 2023-10-16T14:35:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.912+0000 7fb8a37fe640 1 -- 172.21.15.29:0/3075045214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a40319c0 con 0x7fb8ac070240 2023-10-16T14:35:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.912+0000 7fb8a37fe640 1 -- 172.21.15.29:0/3075045214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4031d50 con 0x7fb8ac070240 2023-10-16T14:35:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 -- 172.21.15.29:0/3075045214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac070240 msgr2=0x7fb8ac0706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3075045214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac070240 0x7fb8ac0706a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0cf560 tx=0x7fb8a402b900 comp rx=0 tx=0).stop 2023-10-16T14:35:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 -- 172.21.15.29:0/3075045214 shutdown_connections 2023-10-16T14:35:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3075045214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac070240 0x7fb8ac0706a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3075045214 >> v2:172.21.15.170:3300/0 conn(0x7fb8ac0ca8a0 0x7fb8ac0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3075045214 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac15aaa0 0x7fb8ac0b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 -- 172.21.15.29:0/3075045214 >> 172.21.15.29:0/3075045214 conn(0x7fb8ac0cc230 msgr2=0x7fb8ac0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 -- 172.21.15.29:0/3075045214 shutdown_connections 2023-10-16T14:35:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.913+0000 7fb8b194e640 1 -- 172.21.15.29:0/3075045214 wait complete. 2023-10-16T14:35:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.915+0000 7fb8b194e640 1 Processor -- start 2023-10-16T14:35:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.915+0000 7fb8b194e640 1 -- start start 2023-10-16T14:35:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.916+0000 7fb8b194e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac070240 0x7fb8ac14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.916+0000 7fb8b194e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8ac0ca8a0 0x7fb8ac0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.916+0000 7fb8b194e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 0x7fb8ac0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.916+0000 7fb8b194e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac157110 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.916+0000 7fb8b194e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0b0eb0 con 0x7fb8ac0ca8a0 2023-10-16T14:35:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.916+0000 7fb8b194e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0add40 con 0x7fb8ac070240 2023-10-16T14:35:14.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 0x7fb8ac0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:14.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 0x7fb8ac0ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45824/0 (socket says 172.21.15.29:45824) 2023-10-16T14:35:14.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 -- 172.21.15.29:0/3433997614 learned_addr learned my addr 172.21.15.29:0/3433997614 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:14.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 -- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac070240 msgr2=0x7fb8ac14fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:14.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac070240 0x7fb8ac14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 -- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8ac0ca8a0 msgr2=0x7fb8ac0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:14.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8ac0ca8a0 0x7fb8ac0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:14.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 -- 172.21.15.29:0/3433997614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40094d0 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.917+0000 7fb8b114d640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 0x7fb8ac0ce810 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fb8a402b940 tx=0x7fb8a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.918+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a403ac60 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.918+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a403adc0 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.918+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4044630 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.918+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac150060 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.918+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac0ad780 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.919+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac04e9b0 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.920+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb8a4044790 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.922+0000 7fb8a17fa640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb89806af90 0x7fb89806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.922+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6160+0+0 (secure 0 0 0) 0x7fb8a40abfe0 con 0x7fb8ac15aaa0 2023-10-16T14:35:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.922+0000 7fb8b094c640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb89806af90 0x7fb89806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.924+0000 7fb8b094c640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb89806af90 0x7fb89806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80070b0 tx=0x7fb8a8007040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:14.926+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb8a4076d50 con 0x7fb8ac15aaa0 2023-10-16T14:35:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.121+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb8ac0b1020 con 0x7fb8ac15aaa0 2023-10-16T14:35:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.653+0000 7fb8a17fa640 1 -- 172.21.15.29:0/3433997614 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v118) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb8a4035070 con 0x7fb8ac15aaa0 2023-10-16T14:35:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:35:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.659+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb89806af90 msgr2=0x7fb89806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.659+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb89806af90 0x7fb89806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80070b0 tx=0x7fb8a8007040 comp rx=0 tx=0).stop 2023-10-16T14:35:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.659+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 msgr2=0x7fb8ac0ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.659+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 0x7fb8ac0ce810 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fb8a402b940 tx=0x7fb8a40046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.660+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 shutdown_connections 2023-10-16T14:35:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.660+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb89806af90 0x7fb89806d450 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80070b0 tx=0x7fb8a8007040 comp rx=0 tx=0).stop 2023-10-16T14:35:15.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.660+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8ac15aaa0 0x7fb8ac0ce810 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.660+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb8ac0ca8a0 0x7fb8ac0ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.660+0000 7fb8b194e640 1 --2- 172.21.15.29:0/3433997614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb8ac070240 0x7fb8ac14fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.660+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 >> 172.21.15.29:0/3433997614 conn(0x7fb8ac0cc230 msgr2=0x7fb8ac0c84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:15.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.661+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 shutdown_connections 2023-10-16T14:35:15.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.661+0000 7fb8b194e640 1 -- 172.21.15.29:0/3433997614 wait complete. 2023-10-16T14:35:15.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:35:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.909+0000 7fbfb6059640 1 Processor -- start 2023-10-16T14:35:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.909+0000 7fbfb6059640 1 -- start start 2023-10-16T14:35:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.911+0000 7fbfb6059640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbfb015dde0 0x7fbfb00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.911+0000 7fbfb6059640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb0160a90 con 0x7fbfb015c9f0 2023-10-16T14:35:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.911+0000 7fbfb6059640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb00cd900 con 0x7fbfb0070f20 2023-10-16T14:35:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.911+0000 7fbfb6059640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb0071350 con 0x7fbfb015dde0 2023-10-16T14:35:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.911+0000 7fbfb5057640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbfb015c9f0 0x7fbfb00c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37186/0 (socket says 172.21.15.29:37186) 2023-10-16T14:35:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.911+0000 7fbfb5057640 1 -- 172.21.15.29:0/1040697709 learned_addr learned my addr 172.21.15.29:0/1040697709 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1040697709 >> v2:172.21.15.170:3300/0 conn(0x7fbfb015dde0 0x7fbfb00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776342593 0 0) 0x7fbfb0160a90 con 0x7fbfb015c9f0 2023-10-16T14:35:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c003520 con 0x7fbfb015c9f0 2023-10-16T14:35:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556400271 0 0) 0x7fbf9c003520 con 0x7fbfb015c9f0 2023-10-16T14:35:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 >> v2:172.21.15.170:3300/0 conn(0x7fbfb015dde0 msgr2=0x7fbfb00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 --2- 172.21.15.29:0/1040697709 >> v2:172.21.15.170:3300/0 conn(0x7fbfb015dde0 0x7fbfb00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 >> v1:172.21.15.170:6789/0 conn(0x7fbfb0070f20 legacy=0x7fbfb00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb00cdbe0 con 0x7fbfb015c9f0 2023-10-16T14:35:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbfac001670 con 0x7fbfb015c9f0 2023-10-16T14:35:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 >> v1:172.21.15.29:6789/0 conn(0x7fbfb015c9f0 legacy=0x7fbfb00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf9c003970 0x7fbf9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbf9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf9c005180 0x7fbf9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb0071350 con 0x7fbf9c0047c0 2023-10-16T14:35:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c00c020 con 0x7fbf9c003970 2023-10-16T14:35:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.912+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c00c1e0 con 0x7fbf9c005180 2023-10-16T14:35:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbf9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb5057640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf9c003970 0x7fbf9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb4856640 1 -- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf9c005180 msgr2=0x7fbf9c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb5858640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf9c005180 0x7fbf9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf9c005180 0x7fbf9c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb4856640 1 -- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf9c003970 msgr2=0x7fbf9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf9c003970 0x7fbf9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb4856640 1 -- 172.21.15.29:0/1040697709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c003520 con 0x7fbf9c0047c0 2023-10-16T14:35:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb5057640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf9c003970 0x7fbf9c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.913+0000 7fbfb5858640 1 -- 172.21.15.29:0/1040697709 reap_dead start 2023-10-16T14:35:15.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.914+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbf9c004c40 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009ac0 tx=0x7fbf9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb27a00b608e713 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:15.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa002c6d0 con 0x7fbf9c0047c0 2023-10-16T14:35:15.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa002c830 con 0x7fbf9c0047c0 2023-10-16T14:35:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfa7fff640 1 -- 172.21.15.29:0/1040697709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa002cbc0 con 0x7fbf9c0047c0 2023-10-16T14:35:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 -- 172.21.15.29:0/1040697709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 msgr2=0x7fbf9c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbf9c004c40 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009ac0 tx=0x7fbf9c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 -- 172.21.15.29:0/1040697709 shutdown_connections 2023-10-16T14:35:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1040697709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbf9c004c40 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 -- 172.21.15.29:0/1040697709 >> 172.21.15.29:0/1040697709 conn(0x7fbfb00ce6a0 msgr2=0x7fbfb00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 -- 172.21.15.29:0/1040697709 shutdown_connections 2023-10-16T14:35:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.915+0000 7fbfb6059640 1 -- 172.21.15.29:0/1040697709 wait complete. 2023-10-16T14:35:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.917+0000 7fbfb6059640 1 Processor -- start 2023-10-16T14:35:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.917+0000 7fbfb6059640 1 -- start start 2023-10-16T14:35:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.918+0000 7fbfb6059640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbfb006c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.918+0000 7fbfb6059640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb00c5770 0x7fbfb00c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.918+0000 7fbfb6059640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb00c7840 0x7fbfb00c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.918+0000 7fbfb6059640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb0151120 con 0x7fbf9c0047c0 2023-10-16T14:35:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.918+0000 7fbfb6059640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb006c9b0 con 0x7fbfb00c5770 2023-10-16T14:35:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.918+0000 7fbfb6059640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb006cc50 con 0x7fbfb00c7840 2023-10-16T14:35:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbfb006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbfb006c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45842/0 (socket says 172.21.15.29:45842) 2023-10-16T14:35:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 -- 172.21.15.29:0/1919224916 learned_addr learned my addr 172.21.15.29:0/1919224916 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb00c5770 0x7fbfb00c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5858640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb00c7840 0x7fbfb00c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 -- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb00c7840 msgr2=0x7fbfb00c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb00c7840 0x7fbfb00c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 -- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb00c5770 msgr2=0x7fbfb00c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb00c5770 0x7fbfb00c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb5057640 1 -- 172.21.15.29:0/1919224916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb015faf0 con 0x7fbf9c0047c0 2023-10-16T14:35:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.919+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb00c5770 0x7fbfb00c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.920+0000 7fbfb5057640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbfb006c470 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00d700 tx=0x7fbfac00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.920+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0074e0 con 0x7fbf9c0047c0 2023-10-16T14:35:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.920+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac007640 con 0x7fbf9c0047c0 2023-10-16T14:35:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.920+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac004a00 con 0x7fbf9c0047c0 2023-10-16T14:35:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.920+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb015ed30 con 0x7fbf9c0047c0 2023-10-16T14:35:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.920+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb015f2f0 con 0x7fbf9c0047c0 2023-10-16T14:35:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.921+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb004e9b0 con 0x7fbf9c0047c0 2023-10-16T14:35:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.922+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbfac004b60 con 0x7fbf9c0047c0 2023-10-16T14:35:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.923+0000 7fbfa5ffb640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbf9406af90 0x7fbf9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.923+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fbfac089390 con 0x7fbf9c0047c0 2023-10-16T14:35:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.924+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbf9406af90 0x7fbf9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.925+0000 7fbfb4856640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbf9406af90 0x7fbf9406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0036000 tx=0x7fbfa0030000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:15.928+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbfac0542b0 con 0x7fbf9c0047c0 2023-10-16T14:35:16.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.127+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbfb00adac0 con 0x7fbf9c0047c0 2023-10-16T14:35:16.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.132+0000 7fbfa5ffb640 1 -- 172.21.15.29:0/1919224916 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbfac0578d0 con 0x7fbf9c0047c0 2023-10-16T14:35:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.135+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbf9406af90 msgr2=0x7fbf9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.135+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbf9406af90 0x7fbf9406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0036000 tx=0x7fbfa0030000 comp rx=0 tx=0).stop 2023-10-16T14:35:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.135+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 msgr2=0x7fbfb006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.135+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbfb006c470 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00d700 tx=0x7fbfac00dbc0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 shutdown_connections 2023-10-16T14:35:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbf9406af90 0x7fbf9406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb00c7840 0x7fbfb00c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb00c5770 0x7fbfb00c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 --2- 172.21.15.29:0/1919224916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c0047c0 0x7fbfb006c470 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 >> 172.21.15.29:0/1919224916 conn(0x7fbfb00ce6a0 msgr2=0x7fbfb00ad310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 shutdown_connections 2023-10-16T14:35:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.136+0000 7fbfb6059640 1 -- 172.21.15.29:0/1919224916 wait complete. 2023-10-16T14:35:16.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:35:16.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.354+0000 7fe9c71fc640 1 Processor -- start 2023-10-16T14:35:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.355+0000 7fe9c71fc640 1 -- start start 2023-10-16T14:35:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.356+0000 7fe9c71fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9c015aaa0 0x7fe9c015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.356+0000 7fe9c71fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015ae80 0x7fe9c00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.356+0000 7fe9c71fc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00707e0 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.356+0000 7fe9c71fc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c006b7a0 con 0x7fe9c015aaa0 2023-10-16T14:35:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.356+0000 7fe9c71fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c015ef20 con 0x7fe9c015ae80 2023-10-16T14:35:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9c015aaa0 0x7fe9c015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9c015aaa0 0x7fe9c015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45856/0 (socket says 172.21.15.29:45856) 2023-10-16T14:35:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 -- 172.21.15.29:0/3544346564 learned_addr learned my addr 172.21.15.29:0/3544346564 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:16.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 -- 172.21.15.29:0/3544346564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015ae80 msgr2=0x7fe9c00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:16.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9b7fff640 1 --2- 172.21.15.29:0/3544346564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015ae80 0x7fe9c00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 --2- 172.21.15.29:0/3544346564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015ae80 0x7fe9c00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 -- 172.21.15.29:0/3544346564 >> v1:172.21.15.170:6789/0 conn(0x7fe9c00ccaa0 legacy=0x7fe9c0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:16.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.357+0000 7fe9c4f98640 1 -- 172.21.15.29:0/3544346564 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c015f200 con 0x7fe9c015aaa0 2023-10-16T14:35:16.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.359+0000 7fe9c4f98640 1 --2- 172.21.15.29:0/3544346564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c015aaa0 0x7fe9c015e820 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009870 tx=0x7fe9bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a806215db75fa6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:16.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.359+0000 7fe9b77fe640 1 -- 172.21.15.29:0/3544346564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc031640 con 0x7fe9c015aaa0 2023-10-16T14:35:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.359+0000 7fe9b77fe640 1 -- 172.21.15.29:0/3544346564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0317a0 con 0x7fe9c015aaa0 2023-10-16T14:35:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.359+0000 7fe9b77fe640 1 -- 172.21.15.29:0/3544346564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc031b30 con 0x7fe9c015aaa0 2023-10-16T14:35:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 -- 172.21.15.29:0/3544346564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c015aaa0 msgr2=0x7fe9c015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/3544346564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c015aaa0 0x7fe9c015e820 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009870 tx=0x7fe9bc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 -- 172.21.15.29:0/3544346564 shutdown_connections 2023-10-16T14:35:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/3544346564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015ae80 0x7fe9c00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/3544346564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c015aaa0 0x7fe9c015e820 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 -- 172.21.15.29:0/3544346564 >> 172.21.15.29:0/3544346564 conn(0x7fe9c00cb410 msgr2=0x7fe9c00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:16.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 -- 172.21.15.29:0/3544346564 shutdown_connections 2023-10-16T14:35:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.360+0000 7fe9c71fc640 1 -- 172.21.15.29:0/3544346564 wait complete. 2023-10-16T14:35:16.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.362+0000 7fe9c71fc640 1 Processor -- start 2023-10-16T14:35:16.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.362+0000 7fe9c71fc640 1 -- start start 2023-10-16T14:35:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c71fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 0x7fe9c00c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c71fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015aaa0 0x7fe9c015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c71fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9c015ae80 0x7fe9c0144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c71fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0146510 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c71fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c01599c0 con 0x7fe9c015ae80 2023-10-16T14:35:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c71fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c01455b0 con 0x7fe9c015aaa0 2023-10-16T14:35:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c4f98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 0x7fe9c00c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c4f98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 0x7fe9c00c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45872/0 (socket says 172.21.15.29:45872) 2023-10-16T14:35:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.363+0000 7fe9c4f98640 1 -- 172.21.15.29:0/847858959 learned_addr learned my addr 172.21.15.29:0/847858959 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9b7fff640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015aaa0 0x7fe9c015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c5799640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9c015ae80 0x7fe9c0144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c4f98640 1 -- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015aaa0 msgr2=0x7fe9c015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c4f98640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015aaa0 0x7fe9c015b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c4f98640 1 -- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9c015ae80 msgr2=0x7fe9c0144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c4f98640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9c015ae80 0x7fe9c0144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c4f98640 1 -- 172.21.15.29:0/847858959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0094d0 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c5799640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9c015ae80 0x7fe9c0144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.364+0000 7fe9c4f98640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 0x7fe9c00c4ca0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000c00 tx=0x7fe9bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.365+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc03ac60 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.365+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc03ade0 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.365+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc044630 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.365+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c0145890 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.365+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c0145e50 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.366+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0071640 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.367+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe9bc02f050 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.369+0000 7fe9b57fa640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe9ac06af90 0x7fe9ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.369+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fe9bc0ab9f0 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.370+0000 7fe9b7fff640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe9ac06af90 0x7fe9ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.371+0000 7fe9b7fff640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe9ac06af90 0x7fe9ac06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.373+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe9c0071640 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.567+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe9c0071640 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.574+0000 7fe9b57fa640 1 -- 172.21.15.29:0/847858959 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v38) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe9c0071640 con 0x7fe9c00ccaa0 2023-10-16T14:35:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe9ac06af90 msgr2=0x7fe9ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe9ac06af90 0x7fe9ac06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0008040 comp rx=0 tx=0).stop 2023-10-16T14:35:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 msgr2=0x7fe9c00c4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 0x7fe9c00c4ca0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000c00 tx=0x7fe9bc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 shutdown_connections 2023-10-16T14:35:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe9ac06af90 0x7fe9ac06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9c015ae80 0x7fe9c0144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9c015aaa0 0x7fe9c015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 --2- 172.21.15.29:0/847858959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c00ccaa0 0x7fe9c00c4ca0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 >> 172.21.15.29:0/847858959 conn(0x7fe9c00cb410 msgr2=0x7fe9c006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 shutdown_connections 2023-10-16T14:35:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.577+0000 7fe9c71fc640 1 -- 172.21.15.29:0/847858959 wait complete. 2023-10-16T14:35:16.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:35:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.828+0000 7fa64e883640 1 Processor -- start 2023-10-16T14:35:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.828+0000 7fa64e883640 1 -- start start 2023-10-16T14:35:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa64e883640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa6480ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa64e883640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa6480c9ca0 0x7fa6480ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa64e883640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6480cb030 con 0x7fa64815ad70 2023-10-16T14:35:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa64e883640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6480cae30 con 0x7fa6480c9ca0 2023-10-16T14:35:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa64e883640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480c91c0 con 0x7fa64806a4b0 2023-10-16T14:35:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa647fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa64815ad70 0x7fa648069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37188/0 (socket says 172.21.15.29:37188) 2023-10-16T14:35:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.830+0000 7fa647fff640 1 -- 172.21.15.29:0/1701141637 learned_addr learned my addr 172.21.15.29:0/1701141637 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:16.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa63ffff640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa6480ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa64ce20640 1 --2- 172.21.15.29:0/1701141637 >> v2:172.21.15.170:3300/0 conn(0x7fa6480c9ca0 0x7fa6480ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395092975 0 0) 0x7fa6480cb030 con 0x7fa64815ad70 2023-10-16T14:35:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa634003520 con 0x7fa64815ad70 2023-10-16T14:35:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3411380842 0 0) 0x7fa634003520 con 0x7fa64815ad70 2023-10-16T14:35:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 msgr2=0x7fa6480ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa6480ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 >> v2:172.21.15.170:3300/0 conn(0x7fa6480c9ca0 msgr2=0x7fa6480ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 --2- 172.21.15.29:0/1701141637 >> v2:172.21.15.170:3300/0 conn(0x7fa6480c9ca0 0x7fa6480ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480c94a0 con 0x7fa64815ad70 2023-10-16T14:35:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa638001670 con 0x7fa64815ad70 2023-10-16T14:35:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 >> v1:172.21.15.29:6789/0 conn(0x7fa64815ad70 legacy=0x7fa648069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa634003970 0x7fa634003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6340047c0 0x7fa634004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480cae30 con 0x7fa6340047c0 2023-10-16T14:35:16.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480c91c0 con 0x7fa634003970 2023-10-16T14:35:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.831+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63400b980 con 0x7fa64806a4b0 2023-10-16T14:35:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa63ffff640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6340047c0 0x7fa634004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa647fff640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa634003970 0x7fa634003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa64ce20640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa63ffff640 1 -- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 msgr2=0x7fa634005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa63ffff640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa634005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa63ffff640 1 -- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa634003970 msgr2=0x7fa634003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa63ffff640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa634003970 0x7fa634003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa63ffff640 1 -- 172.21.15.29:0/1701141637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa634003520 con 0x7fa6340047c0 2023-10-16T14:35:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.832+0000 7fa64ce20640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64806a4b0 0x7fa634005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa63ffff640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6340047c0 0x7fa634004c40 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fa630009ac0 tx=0x7fa6340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55a7d8d7a6b1ebd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63002c6b0 con 0x7fa6340047c0 2023-10-16T14:35:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63002c810 con 0x7fa6340047c0 2023-10-16T14:35:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa6477fe640 1 -- 172.21.15.29:0/1701141637 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63002cba0 con 0x7fa6340047c0 2023-10-16T14:35:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa64e883640 1 -- 172.21.15.29:0/1701141637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6340047c0 msgr2=0x7fa634004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa64e883640 1 --2- 172.21.15.29:0/1701141637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6340047c0 0x7fa634004c40 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fa630009ac0 tx=0x7fa6340057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa64ce20640 1 -- 172.21.15.29:0/1701141637 reap_dead start 2023-10-16T14:35:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa64e883640 1 -- 172.21.15.29:0/1701141637 shutdown_connections 2023-10-16T14:35:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.834+0000 7fa64e883640 1 -- 172.21.15.29:0/1701141637 >> 172.21.15.29:0/1701141637 conn(0x7fa6480ce9b0 msgr2=0x7fa6480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.835+0000 7fa64e883640 1 -- 172.21.15.29:0/1701141637 shutdown_connections 2023-10-16T14:35:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.835+0000 7fa64e883640 1 -- 172.21.15.29:0/1701141637 wait complete. 2023-10-16T14:35:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.836+0000 7fa64e883640 1 Processor -- start 2023-10-16T14:35:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 -- start start 2023-10-16T14:35:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6340047c0 0x7fa6480a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64814e2a0 0x7fa6480b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 0x7fa6480aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480aa750 con 0x7fa6480a9d90 2023-10-16T14:35:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480a98c0 con 0x7fa6340047c0 2023-10-16T14:35:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.837+0000 7fa64e883640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480a8cb0 con 0x7fa64814e2a0 2023-10-16T14:35:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 0x7fa6480aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 0x7fa6480aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45888/0 (socket says 172.21.15.29:45888) 2023-10-16T14:35:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 -- 172.21.15.29:0/458617938 learned_addr learned my addr 172.21.15.29:0/458617938 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa647fff640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6340047c0 0x7fa6480a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa63ffff640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64814e2a0 0x7fa6480b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 -- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64814e2a0 msgr2=0x7fa6480b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64814e2a0 0x7fa6480b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 -- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6340047c0 msgr2=0x7fa6480a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6340047c0 0x7fa6480a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa64ce20640 1 -- 172.21.15.29:0/458617938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480a8f90 con 0x7fa6480a9d90 2023-10-16T14:35:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.838+0000 7fa647fff640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6340047c0 0x7fa6480a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.839+0000 7fa64ce20640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 0x7fa6480aa210 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fa6480cf820 tx=0x7fa64000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.839+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640016620 con 0x7fa6480a9d90 2023-10-16T14:35:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.839+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa640016780 con 0x7fa6480a9d90 2023-10-16T14:35:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.839+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640016b10 con 0x7fa6480a9d90 2023-10-16T14:35:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.839+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6480a9270 con 0x7fa6480a9d90 2023-10-16T14:35:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.839+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6480a89c0 con 0x7fa6480a9d90 2023-10-16T14:35:16.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.840+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64804e160 con 0x7fa6480a9d90 2023-10-16T14:35:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.841+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa640016c70 con 0x7fa6480a9d90 2023-10-16T14:35:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.842+0000 7fa6457fa640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fa62806af90 0x7fa62806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:16.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.842+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fa64008a780 con 0x7fa6480a9d90 2023-10-16T14:35:16.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.843+0000 7fa647fff640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fa62806af90 0x7fa62806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:16.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.844+0000 7fa647fff640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fa62806af90 0x7fa62806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa638001200 tx=0x7fa638001110 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:16.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:16.847+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa640055690 con 0x7fa6480a9d90 2023-10-16T14:35:17.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.049+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa648069e40 con 0x7fa6480a9d90 2023-10-16T14:35:17.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.054+0000 7fa6457fa640 1 -- 172.21.15.29:0/458617938 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa640058cb0 con 0x7fa6480a9d90 2023-10-16T14:35:17.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.057+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fa62806af90 msgr2=0x7fa62806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.057+0000 7fa64e883640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fa62806af90 0x7fa62806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa638001200 tx=0x7fa638001110 comp rx=0 tx=0).stop 2023-10-16T14:35:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 msgr2=0x7fa6480aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 0x7fa6480aa210 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fa6480cf820 tx=0x7fa64000f520 comp rx=0 tx=0).stop 2023-10-16T14:35:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 shutdown_connections 2023-10-16T14:35:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fa62806af90 0x7fa62806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6480a9d90 0x7fa6480aa210 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa64814e2a0 0x7fa6480b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 --2- 172.21.15.29:0/458617938 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6340047c0 0x7fa6480a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 >> 172.21.15.29:0/458617938 conn(0x7fa6480ce9b0 msgr2=0x7fa64815dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:17.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 shutdown_connections 2023-10-16T14:35:17.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.058+0000 7fa64e883640 1 -- 172.21.15.29:0/458617938 wait complete. 2023-10-16T14:35:17.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T14:35:17.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.307+0000 7fc234e1a640 1 Processor -- start 2023-10-16T14:35:17.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.307+0000 7fc234e1a640 1 -- start start 2023-10-16T14:35:17.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.308+0000 7fc234e1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc23006c040 0x7fc23006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.308+0000 7fc234e1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23015f4a0 con 0x7fc2300cda80 2023-10-16T14:35:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.308+0000 7fc234e1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23006c960 con 0x7fc2300ad970 2023-10-16T14:35:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.308+0000 7fc234e1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23015b9d0 con 0x7fc23006c040 2023-10-16T14:35:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.309+0000 7fc22ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc2300cda80 0x7fc23015ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37192/0 (socket says 172.21.15.29:37192) 2023-10-16T14:35:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.309+0000 7fc22ffff640 1 -- 172.21.15.29:0/2158919504 learned_addr learned my addr 172.21.15.29:0/2158919504 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.309+0000 7fc22f7fe640 1 --2- 172.21.15.29:0/2158919504 >> v2:172.21.15.170:3300/0 conn(0x7fc23006c040 0x7fc23006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.309+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044166266 0 0) 0x7fc23015f4a0 con 0x7fc2300cda80 2023-10-16T14:35:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.309+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21c003520 con 0x7fc2300cda80 2023-10-16T14:35:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1847797005 0 0) 0x7fc21c003520 con 0x7fc2300cda80 2023-10-16T14:35:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 >> v2:172.21.15.170:3300/0 conn(0x7fc23006c040 msgr2=0x7fc23006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 --2- 172.21.15.29:0/2158919504 >> v2:172.21.15.170:3300/0 conn(0x7fc23006c040 0x7fc23006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 >> v1:172.21.15.170:6789/0 conn(0x7fc2300ad970 legacy=0x7fc23015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23015bcb0 con 0x7fc2300cda80 2023-10-16T14:35:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc2280019f0 con 0x7fc2300cda80 2023-10-16T14:35:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 >> v1:172.21.15.29:6789/0 conn(0x7fc2300cda80 legacy=0x7fc23015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc21c003970 0x7fc21c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc21c0047c0 0x7fc21c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc21c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23015b9d0 con 0x7fc21c005180 2023-10-16T14:35:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c00c020 con 0x7fc21c0047c0 2023-10-16T14:35:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c00c1e0 con 0x7fc21c003970 2023-10-16T14:35:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc22ffff640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc21c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.310+0000 7fc227fff640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc21c0047c0 0x7fc21c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.311+0000 7fc22f7fe640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc21c003970 0x7fc21c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.311+0000 7fc22ffff640 1 -- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc21c003970 msgr2=0x7fc21c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.311+0000 7fc22ffff640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc21c003970 0x7fc21c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.311+0000 7fc22ffff640 1 -- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc21c0047c0 msgr2=0x7fc21c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.311+0000 7fc22ffff640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc21c0047c0 0x7fc21c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.311+0000 7fc22ffff640 1 -- 172.21.15.29:0/2158919504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c003520 con 0x7fc21c005180 2023-10-16T14:35:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.312+0000 7fc22ffff640 1 -- 172.21.15.29:0/2158919504 reap_dead start 2023-10-16T14:35:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.313+0000 7fc22ffff640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc21c00b980 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc22800a830 tx=0x7fc21c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad4db186da8b202 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.313+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2280072d0 con 0x7fc21c005180 2023-10-16T14:35:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.313+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22802f070 con 0x7fc21c005180 2023-10-16T14:35:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.313+0000 7fc22effd640 1 -- 172.21.15.29:0/2158919504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2280076b0 con 0x7fc21c005180 2023-10-16T14:35:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 -- 172.21.15.29:0/2158919504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 msgr2=0x7fc21c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc21c00b980 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc22800a830 tx=0x7fc21c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 -- 172.21.15.29:0/2158919504 shutdown_connections 2023-10-16T14:35:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 --2- 172.21.15.29:0/2158919504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc21c00b980 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 -- 172.21.15.29:0/2158919504 >> 172.21.15.29:0/2158919504 conn(0x7fc2300ce6a0 msgr2=0x7fc2300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 -- 172.21.15.29:0/2158919504 shutdown_connections 2023-10-16T14:35:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.314+0000 7fc234e1a640 1 -- 172.21.15.29:0/2158919504 wait complete. 2023-10-16T14:35:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.316+0000 7fc234e1a640 1 Processor -- start 2023-10-16T14:35:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.317+0000 7fc234e1a640 1 -- start start 2023-10-16T14:35:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc234e1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc2300c1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc234e1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2300c21b0 0x7fc2301433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc234e1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2300c25f0 0x7fc230143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc234e1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300c69f0 con 0x7fc21c005180 2023-10-16T14:35:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc234e1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300c1ac0 con 0x7fc2300c25f0 2023-10-16T14:35:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc234e1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300c6c60 con 0x7fc2300c21b0 2023-10-16T14:35:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc22f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc2300c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc22f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc2300c1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45908/0 (socket says 172.21.15.29:45908) 2023-10-16T14:35:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.318+0000 7fc22f7fe640 1 -- 172.21.15.29:0/1840410896 learned_addr learned my addr 172.21.15.29:0/1840410896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc227fff640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2300c21b0 0x7fc2301433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22ffff640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2300c25f0 0x7fc230143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22f7fe640 1 -- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2300c21b0 msgr2=0x7fc2301433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22f7fe640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2300c21b0 0x7fc2301433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22f7fe640 1 -- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2300c25f0 msgr2=0x7fc230143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22f7fe640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2300c25f0 0x7fc230143930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22f7fe640 1 -- 172.21.15.29:0/1840410896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2300c6f40 con 0x7fc21c005180 2023-10-16T14:35:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc227fff640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2300c21b0 0x7fc2301433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22f7fe640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc2300c1580 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc2200118e0 tx=0x7fc220011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:17.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22000faf0 con 0x7fc21c005180 2023-10-16T14:35:17.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22000fc50 con 0x7fc21c005180 2023-10-16T14:35:17.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220014440 con 0x7fc21c005180 2023-10-16T14:35:17.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.319+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2300c7220 con 0x7fc21c005180 2023-10-16T14:35:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.320+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2300c7a20 con 0x7fc21c005180 2023-10-16T14:35:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.321+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23006b330 con 0x7fc21c005180 2023-10-16T14:35:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.322+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc2200145a0 con 0x7fc21c005180 2023-10-16T14:35:17.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.323+0000 7fc22cff9640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc21006afc0 0x7fc21006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.323+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fc22008a070 con 0x7fc21c005180 2023-10-16T14:35:17.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.323+0000 7fc227fff640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc21006afc0 0x7fc21006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.325+0000 7fc227fff640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc21006afc0 0x7fc21006d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc230152a10 tx=0x7fc218009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:17.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.327+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc220055000 con 0x7fc21c005180 2023-10-16T14:35:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.589+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fc23015bf60 con 0x7fc21c005180 2023-10-16T14:35:17.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.675+0000 7fc22cff9640 1 -- 172.21.15.29:0/1840410896 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v710) v1 ==== 183+0+0 (secure 0 0 0) 0x7fc220058620 con 0x7fc21c005180 2023-10-16T14:35:17.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.678+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc21006afc0 msgr2=0x7fc21006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.678+0000 7fc234e1a640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc21006afc0 0x7fc21006d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc230152a10 tx=0x7fc218009040 comp rx=0 tx=0).stop 2023-10-16T14:35:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.678+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 msgr2=0x7fc2300c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.678+0000 7fc234e1a640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc2300c1580 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc2200118e0 tx=0x7fc220011da0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 shutdown_connections 2023-10-16T14:35:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc21006afc0 0x7fc21006d480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc2300c25f0 0x7fc230143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc2300c21b0 0x7fc2301433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 --2- 172.21.15.29:0/1840410896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21c005180 0x7fc2300c1580 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 >> 172.21.15.29:0/1840410896 conn(0x7fc2300ce6a0 msgr2=0x7fc2300cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:17.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 shutdown_connections 2023-10-16T14:35:17.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.679+0000 7fc234e1a640 1 -- 172.21.15.29:0/1840410896 wait complete. 2023-10-16T14:35:17.703 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-16T14:35:17.704 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-16T14:35:17.704 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:35:17.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:17.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.907+0000 7f1e7efd0640 1 Processor -- start 2023-10-16T14:35:17.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.907+0000 7f1e7efd0640 1 -- start start 2023-10-16T14:35:17.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.908+0000 7f1e7efd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1e780ad970 0x7f1e780cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7efd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7815b9d0 con 0x7f1e7806c040 2023-10-16T14:35:17.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7efd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7815bcb0 con 0x7f1e780ccfb0 2023-10-16T14:35:17.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7efd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7815bf90 con 0x7f1e780ad970 2023-10-16T14:35:17.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7dfce640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1e7806c040 0x7f1e7806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37196/0 (socket says 172.21.15.29:37196) 2023-10-16T14:35:17.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7dfce640 1 -- 172.21.15.29:0/1821958482 learned_addr learned my addr 172.21.15.29:0/1821958482 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616539637 0 0) 0x7f1e7815b9d0 con 0x7f1e7806c040 2023-10-16T14:35:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7d7cd640 1 --2- 172.21.15.29:0/1821958482 >> v2:172.21.15.170:3300/0 conn(0x7f1e780ad970 0x7f1e780cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.909+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e64003520 con 0x7f1e7806c040 2023-10-16T14:35:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478824511 0 0) 0x7f1e64003520 con 0x7f1e7806c040 2023-10-16T14:35:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 >> v2:172.21.15.170:3300/0 conn(0x7f1e780ad970 msgr2=0x7f1e780cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 --2- 172.21.15.29:0/1821958482 >> v2:172.21.15.170:3300/0 conn(0x7f1e780ad970 0x7f1e780cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 >> v1:172.21.15.170:6789/0 conn(0x7f1e780ccfb0 legacy=0x7f1e780cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e780cc270 con 0x7f1e7806c040 2023-10-16T14:35:17.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1e74001670 con 0x7f1e7806c040 2023-10-16T14:35:17.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 >> v1:172.21.15.29:6789/0 conn(0x7f1e7806c040 legacy=0x7f1e7806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e640039c0 0x7f1e64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e640047c0 0x7f1e64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e64005180 0x7f1e6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7815bf90 con 0x7f1e640039c0 2023-10-16T14:35:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6400c080 con 0x7f1e640047c0 2023-10-16T14:35:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6400c240 con 0x7f1e64005180 2023-10-16T14:35:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.910+0000 7f1e7dfce640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e640039c0 0x7f1e64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7d7cd640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e640047c0 0x7f1e64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7dfce640 1 -- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e64005180 msgr2=0x7f1e6400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7dfce640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e64005180 0x7f1e6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7dfce640 1 -- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e640047c0 msgr2=0x7f1e64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7dfce640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e640047c0 0x7f1e64004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7dfce640 1 -- 172.21.15.29:0/1821958482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64003520 con 0x7f1e640039c0 2023-10-16T14:35:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.911+0000 7f1e7e7cf640 1 -- 172.21.15.29:0/1821958482 reap_dead start 2023-10-16T14:35:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.912+0000 7f1e7dfce640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e640039c0 0x7f1e64003de0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1e74007fa0 tx=0x7f1e640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4357c10fdf2a08a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.912+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74005ad0 con 0x7f1e640039c0 2023-10-16T14:35:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.912+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e74005c50 con 0x7f1e640039c0 2023-10-16T14:35:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.912+0000 7f1e7cfcc640 1 -- 172.21.15.29:0/1821958482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7402c790 con 0x7f1e640039c0 2023-10-16T14:35:17.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 -- 172.21.15.29:0/1821958482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e640039c0 msgr2=0x7f1e64003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e640039c0 0x7f1e64003de0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1e74007fa0 tx=0x7f1e640057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 -- 172.21.15.29:0/1821958482 shutdown_connections 2023-10-16T14:35:17.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/1821958482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e640039c0 0x7f1e64003de0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 -- 172.21.15.29:0/1821958482 >> 172.21.15.29:0/1821958482 conn(0x7f1e780ce6a0 msgr2=0x7f1e780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:17.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 -- 172.21.15.29:0/1821958482 shutdown_connections 2023-10-16T14:35:17.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.913+0000 7f1e7efd0640 1 -- 172.21.15.29:0/1821958482 wait complete. 2023-10-16T14:35:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.915+0000 7f1e7efd0640 1 Processor -- start 2023-10-16T14:35:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.915+0000 7f1e7efd0640 1 -- start start 2023-10-16T14:35:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7efd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e640039c0 0x7f1e780c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7efd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 0x7f1e780b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7efd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e780b3520 0x7f1e7815c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7efd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780b1f80 con 0x7f1e780b3df0 2023-10-16T14:35:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7efd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780ae5b0 con 0x7f1e780b3520 2023-10-16T14:35:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7efd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780b2220 con 0x7f1e640039c0 2023-10-16T14:35:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7d7cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 0x7f1e780b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7d7cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 0x7f1e780b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45926/0 (socket says 172.21.15.29:45926) 2023-10-16T14:35:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7d7cd640 1 -- 172.21.15.29:0/561632320 learned_addr learned my addr 172.21.15.29:0/561632320 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7d7cd640 1 -- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e640039c0 msgr2=0x7f1e780c2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.916+0000 7f1e7d7cd640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e640039c0 0x7f1e780c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e7d7cd640 1 -- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e780b3520 msgr2=0x7f1e7815c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e7d7cd640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e780b3520 0x7f1e7815c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:17.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e7d7cd640 1 -- 172.21.15.29:0/561632320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e780b2500 con 0x7f1e780b3df0 2023-10-16T14:35:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e7d7cd640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 0x7f1e780b2fe0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0108d0 tx=0x7f1e6c010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c009650 con 0x7f1e780b3df0 2023-10-16T14:35:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c00eb00 con 0x7f1e780b3df0 2023-10-16T14:35:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e780b27e0 con 0x7f1e780b3df0 2023-10-16T14:35:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00ee90 con 0x7f1e780b3df0 2023-10-16T14:35:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.917+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e780c2a80 con 0x7f1e780b3df0 2023-10-16T14:35:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.918+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7806b330 con 0x7f1e780b3df0 2023-10-16T14:35:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.919+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1e6c01c400 con 0x7f1e780b3df0 2023-10-16T14:35:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.920+0000 7f1e6a7fc640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f1e5c06afc0 0x7f1e5c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.920+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f1e6c089d50 con 0x7f1e780b3df0 2023-10-16T14:35:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.921+0000 7f1e7dfce640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f1e5c06afc0 0x7f1e5c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.922+0000 7f1e7dfce640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f1e5c06afc0 0x7f1e5c06d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1e74034000 tx=0x7f1e7402e000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:17.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:17.924+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1e6c054cf0 con 0x7f1e780b3df0 2023-10-16T14:35:18.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.117+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e780cc530 con 0x7f1e780b3df0 2023-10-16T14:35:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.119+0000 7f1e6a7fc640 1 -- 172.21.15.29:0/561632320 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+14139 (secure 0 0 0) 0x7f1e780cc530 con 0x7f1e780b3df0 2023-10-16T14:35:18.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":118,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:15.651296+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6834/190249339":"2023-10-17T14:35:12.634718+0000","172.21.15.170:0/2638410402":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/813256058":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/3321905509":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/3034262782":"2023-10-17T14:35:08.530942+0000","172.21.15.170:6832/3649833910":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2734992949":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/388278811":"2023-10-17T14:35:08.530942+0000","172.21.15.170:6833/3649833910":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2765694912":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/3476448138":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1360555952":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/2716588410":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6836/190249339":"2023-10-17T14:35:12.634718+0000","172.21.15.170:0/1873966041":"2023-10-17T14:35:08.530942+0000","172.21.15.170:0/261609677":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6833/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/2811499332":"2023-10-17T14:34:00.590880+0000","172.21.15.170:6832/2925184086":"2023-10-17T14:34:00.590880+0000","172.21.15.170:0/1477766173":"2023-10-17T14:34:00.590880+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f1e5c06afc0 msgr2=0x7f1e5c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f1e5c06afc0 0x7f1e5c06d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1e74034000 tx=0x7f1e7402e000 comp rx=0 tx=0).stop 2023-10-16T14:35:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 msgr2=0x7f1e780b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 0x7f1e780b2fe0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0108d0 tx=0x7f1e6c010d90 comp rx=0 tx=0).stop 2023-10-16T14:35:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 shutdown_connections 2023-10-16T14:35:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f1e5c06afc0 0x7f1e5c06d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e780b3df0 0x7f1e780b2fe0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1e780b3520 0x7f1e7815c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 --2- 172.21.15.29:0/561632320 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1e640039c0 0x7f1e780c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 >> 172.21.15.29:0/561632320 conn(0x7f1e780ce6a0 msgr2=0x7f1e780cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:18.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 shutdown_connections 2023-10-16T14:35:18.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.122+0000 7f1e7efd0640 1 -- 172.21.15.29:0/561632320 wait complete. 2023-10-16T14:35:18.147 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:35:18.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:35:18.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:35:18.314 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:35:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.372+0000 7f2f5ba00640 1 Processor -- start 2023-10-16T14:35:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.373+0000 7f2f5ba00640 1 -- start start 2023-10-16T14:35:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.374+0000 7f2f5ba00640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2f540caf80 0x7f2f540ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.374+0000 7f2f5ba00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.374+0000 7f2f5ba00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f540b4e50 0x7f2f54069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.374+0000 7f2f5ba00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5415b1e0 con 0x7f2f540b4e50 2023-10-16T14:35:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.374+0000 7f2f5ba00640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f540cb8a0 con 0x7f2f540caf80 2023-10-16T14:35:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.374+0000 7f2f5ba00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c9930 con 0x7f2f54065ab0 2023-10-16T14:35:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f540b4e50 0x7f2f54069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f540b4e50 0x7f2f54069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45938/0 (socket says 172.21.15.29:45938) 2023-10-16T14:35:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 -- 172.21.15.29:0/3674337982 learned_addr learned my addr 172.21.15.29:0/3674337982 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:18.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f5979c640 1 --2- 172.21.15.29:0/3674337982 >> v2:172.21.15.170:3300/0 conn(0x7f2f540caf80 0x7f2f540ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f58f9b640 1 --2- 172.21.15.29:0/3674337982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 -- 172.21.15.29:0/3674337982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 msgr2=0x7f2f540cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:18.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 --2- 172.21.15.29:0/3674337982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 -- 172.21.15.29:0/3674337982 >> v2:172.21.15.170:3300/0 conn(0x7f2f540caf80 msgr2=0x7f2f540ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:18.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 --2- 172.21.15.29:0/3674337982 >> v2:172.21.15.170:3300/0 conn(0x7f2f540caf80 0x7f2f540ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.375+0000 7f2f59f9d640 1 -- 172.21.15.29:0/3674337982 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540c9c10 con 0x7f2f540b4e50 2023-10-16T14:35:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.377+0000 7f2f59f9d640 1 --2- 172.21.15.29:0/3674337982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f54069da0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009e50 tx=0x7f2f4c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3c29085e64991d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.377+0000 7f2f4bfff640 1 -- 172.21.15.29:0/3674337982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c02c9f0 con 0x7f2f540b4e50 2023-10-16T14:35:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.377+0000 7f2f4bfff640 1 -- 172.21.15.29:0/3674337982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c02cb50 con 0x7f2f540b4e50 2023-10-16T14:35:18.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.377+0000 7f2f4bfff640 1 -- 172.21.15.29:0/3674337982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0343f0 con 0x7f2f540b4e50 2023-10-16T14:35:18.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3674337982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 msgr2=0x7f2f54069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3674337982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f54069da0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009e50 tx=0x7f2f4c02be50 comp rx=0 tx=0).stop 2023-10-16T14:35:18.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3674337982 shutdown_connections 2023-10-16T14:35:18.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3674337982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f54069da0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3674337982 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3674337982 >> v2:172.21.15.170:3300/0 conn(0x7f2f540caf80 0x7f2f540ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3674337982 >> 172.21.15.29:0/3674337982 conn(0x7f2f5406bc10 msgr2=0x7f2f5406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3674337982 shutdown_connections 2023-10-16T14:35:18.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.378+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3674337982 wait complete. 2023-10-16T14:35:18.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.380+0000 7f2f5ba00640 1 Processor -- start 2023-10-16T14:35:18.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.380+0000 7f2f5ba00640 1 -- start start 2023-10-16T14:35:18.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.381+0000 7f2f5ba00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f5ba00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f541f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f5ba00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f540caf80 0x7f2f541f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f5ba00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c3b70 con 0x7f2f540b4e50 2023-10-16T14:35:18.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f5ba00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c3090 con 0x7f2f540caf80 2023-10-16T14:35:18.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f5ba00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541f6820 con 0x7f2f54065ab0 2023-10-16T14:35:18.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f541f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f541f5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45946/0 (socket says 172.21.15.29:45946) 2023-10-16T14:35:18.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 -- 172.21.15.29:0/3645816548 learned_addr learned my addr 172.21.15.29:0/3645816548 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:18.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f59f9d640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f540caf80 0x7f2f541f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f5979c640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 -- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 msgr2=0x7f2f540b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 -- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f540caf80 msgr2=0x7f2f541f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f540caf80 0x7f2f541f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.382+0000 7f2f58f9b640 1 -- 172.21.15.29:0/3645816548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c009b30 con 0x7f2f540b4e50 2023-10-16T14:35:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.383+0000 7f2f58f9b640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f541f5c10 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400d5f0 tx=0x7f2f4400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.383+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f440125f0 con 0x7f2f540b4e50 2023-10-16T14:35:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.383+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f44012750 con 0x7f2f540b4e50 2023-10-16T14:35:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.383+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f541f6b00 con 0x7f2f540b4e50 2023-10-16T14:35:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.383+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44012b00 con 0x7f2f540b4e50 2023-10-16T14:35:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.384+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f540c3790 con 0x7f2f540b4e50 2023-10-16T14:35:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.385+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f540c3cb0 con 0x7f2f540b4e50 2023-10-16T14:35:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.386+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2f44012c60 con 0x7f2f540b4e50 2023-10-16T14:35:18.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.387+0000 7f2f49ffb640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2f3806af90 0x7f2f3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.388+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f2f44089c80 con 0x7f2f540b4e50 2023-10-16T14:35:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.388+0000 7f2f5979c640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2f3806af90 0x7f2f3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.389+0000 7f2f5979c640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2f3806af90 0x7f2f3806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f500099b0 tx=0x7f2f50008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.391+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2f540c3cb0 con 0x7f2f540b4e50 2023-10-16T14:35:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.582+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f540c3cb0 con 0x7f2f540b4e50 2023-10-16T14:35:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.583+0000 7f2f49ffb640 1 -- 172.21.15.29:0/3645816548 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f540c3cb0 con 0x7f2f540b4e50 2023-10-16T14:35:18.587 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:35:18.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2f3806af90 msgr2=0x7f2f3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2f3806af90 0x7f2f3806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f500099b0 tx=0x7f2f50008040 comp rx=0 tx=0).stop 2023-10-16T14:35:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 msgr2=0x7f2f541f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f541f5c10 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400d5f0 tx=0x7f2f4400dab0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 shutdown_connections 2023-10-16T14:35:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2f3806af90 0x7f2f3806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b4e50 0x7f2f541f5c10 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f540caf80 0x7f2f541f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 --2- 172.21.15.29:0/3645816548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f54065ab0 0x7f2f540b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 >> 172.21.15.29:0/3645816548 conn(0x7f2f5406bc10 msgr2=0x7f2f5415a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 shutdown_connections 2023-10-16T14:35:18.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.586+0000 7f2f5ba00640 1 -- 172.21.15.29:0/3645816548 wait complete. 2023-10-16T14:35:18.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T14:35:18.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.833+0000 7fbcb8d1a640 1 Processor -- start 2023-10-16T14:35:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.833+0000 7fbcb8d1a640 1 -- start start 2023-10-16T14:35:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.834+0000 7fbcb8d1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbcb40cf2b0 0x7fbcb40cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.834+0000 7fbcb8d1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40d1f80 0x7fbcb40cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.834+0000 7fbcb8d1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb4160b60 con 0x7fbcb40cdae0 2023-10-16T14:35:18.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.834+0000 7fbcb8d1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb41617e0 con 0x7fbcb40cf2b0 2023-10-16T14:35:18.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.834+0000 7fbcb8d1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4160e40 con 0x7fbcb40d1f80 2023-10-16T14:35:18.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.835+0000 7fbcb2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbcb40cdae0 0x7fbcb40b4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37202/0 (socket says 172.21.15.29:37202) 2023-10-16T14:35:18.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.835+0000 7fbcb2ffd640 1 -- 172.21.15.29:0/2380618562 learned_addr learned my addr 172.21.15.29:0/2380618562 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.835+0000 7fbcb3fff640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40d1f80 0x7fbcb40cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.835+0000 7fbcb37fe640 1 --2- 172.21.15.29:0/2380618562 >> v2:172.21.15.170:3300/0 conn(0x7fbcb40cf2b0 0x7fbcb40cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.835+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517350164 0 0) 0x7fbcb4160b60 con 0x7fbcb40cdae0 2023-10-16T14:35:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.835+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c003520 con 0x7fbcb40cdae0 2023-10-16T14:35:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330090160 0 0) 0x7fbc9c003520 con 0x7fbcb40cdae0 2023-10-16T14:35:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40d1f80 msgr2=0x7fbcb40cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40d1f80 0x7fbcb40cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 >> v2:172.21.15.170:3300/0 conn(0x7fbcb40cf2b0 msgr2=0x7fbcb40cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 --2- 172.21.15.29:0/2380618562 >> v2:172.21.15.170:3300/0 conn(0x7fbcb40cf2b0 0x7fbcb40cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb4161120 con 0x7fbcb40cdae0 2023-10-16T14:35:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbca8001670 con 0x7fbcb40cdae0 2023-10-16T14:35:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 >> v1:172.21.15.29:6789/0 conn(0x7fbcb40cdae0 legacy=0x7fbcb40b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc9c003970 0x7fbc9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbc9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc9c005180 0x7fbc9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41617e0 con 0x7fbc9c0047c0 2023-10-16T14:35:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4160e40 con 0x7fbc9c003970 2023-10-16T14:35:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c00bfa0 con 0x7fbc9c005180 2023-10-16T14:35:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb3fff640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb40d1f80 0x7fbcb40cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.836+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbc9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb37fe640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc9c003970 0x7fbc9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb3fff640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc9c005180 0x7fbc9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb2ffd640 1 -- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc9c005180 msgr2=0x7fbc9c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:18.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc9c005180 0x7fbc9c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb2ffd640 1 -- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc9c003970 msgr2=0x7fbc9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:18.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc9c003970 0x7fbc9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb2ffd640 1 -- 172.21.15.29:0/2380618562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c003520 con 0x7fbc9c0047c0 2023-10-16T14:35:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb37fe640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc9c003970 0x7fbc9c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.837+0000 7fbcb3fff640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc9c005180 0x7fbc9c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.838+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbc9c004c40 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c240 tx=0x7fbc9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84a4c4cca894614 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:18.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.838+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca802cb00 con 0x7fbc9c0047c0 2023-10-16T14:35:18.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.838+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca802cc60 con 0x7fbc9c0047c0 2023-10-16T14:35:18.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.838+0000 7fbcb27fc640 1 -- 172.21.15.29:0/2380618562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8005af0 con 0x7fbc9c0047c0 2023-10-16T14:35:18.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/2380618562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 msgr2=0x7fbc9c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/2380618562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbc9c004c40 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c240 tx=0x7fbc9c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb3fff640 1 -- 172.21.15.29:0/2380618562 reap_dead start 2023-10-16T14:35:18.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/2380618562 shutdown_connections 2023-10-16T14:35:18.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/2380618562 >> 172.21.15.29:0/2380618562 conn(0x7fbcb406bc60 msgr2=0x7fbcb406c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:18.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/2380618562 shutdown_connections 2023-10-16T14:35:18.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.839+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/2380618562 wait complete. 2023-10-16T14:35:18.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.841+0000 7fbcb8d1a640 1 Processor -- start 2023-10-16T14:35:18.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.841+0000 7fbcb8d1a640 1 -- start start 2023-10-16T14:35:18.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb8d1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbcb41f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb8d1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb41441e0 0x7fbcb4151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb8d1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4151ca0 0x7fbcb4153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb8d1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4152120 con 0x7fbc9c0047c0 2023-10-16T14:35:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb8d1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb40b2f80 con 0x7fbcb4151ca0 2023-10-16T14:35:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb8d1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41f23a0 con 0x7fbcb41441e0 2023-10-16T14:35:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbcb41f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbcb41f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45966/0 (socket says 172.21.15.29:45966) 2023-10-16T14:35:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.842+0000 7fbcb37fe640 1 -- 172.21.15.29:0/4073137835 learned_addr learned my addr 172.21.15.29:0/4073137835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb41441e0 0x7fbcb4151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb3fff640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4151ca0 0x7fbcb4153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb37fe640 1 -- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb41441e0 msgr2=0x7fbcb4151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb37fe640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb41441e0 0x7fbcb4151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb37fe640 1 -- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4151ca0 msgr2=0x7fbcb4153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb37fe640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4151ca0 0x7fbcb4153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:18.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb37fe640 1 -- 172.21.15.29:0/4073137835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb4150e00 con 0x7fbc9c0047c0 2023-10-16T14:35:18.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbcb37fe640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbcb41f1e60 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fbca400f3d0 tx=0x7fbca400f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:18.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.843+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca401a070 con 0x7fbc9c0047c0 2023-10-16T14:35:18.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.844+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca4003990 con 0x7fbc9c0047c0 2023-10-16T14:35:18.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.844+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca4003d00 con 0x7fbc9c0047c0 2023-10-16T14:35:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.844+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb40b31d0 con 0x7fbc9c0047c0 2023-10-16T14:35:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.844+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb40b3790 con 0x7fbc9c0047c0 2023-10-16T14:35:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.845+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb40b3950 con 0x7fbc9c0047c0 2023-10-16T14:35:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.846+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbca4004020 con 0x7fbc9c0047c0 2023-10-16T14:35:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.847+0000 7fbc9bfff640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbca006af90 0x7fbca006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.847+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fbca408a470 con 0x7fbc9c0047c0 2023-10-16T14:35:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.848+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbca006af90 0x7fbca006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.849+0000 7fbcb2ffd640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbca006af90 0x7fbca006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbca8034000 tx=0x7fbca8007000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:18.851+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbca4055380 con 0x7fbc9c0047c0 2023-10-16T14:35:19.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.116+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fbcb415e2f0 con 0x7fbc9c0047c0 2023-10-16T14:35:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.689+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4073137835 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v712) v1 ==== 151+0+0 (secure 0 0 0) 0x7fbca4003e60 con 0x7fbc9c0047c0 2023-10-16T14:35:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbca006af90 msgr2=0x7fbca006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbca006af90 0x7fbca006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbca8034000 tx=0x7fbca8007000 comp rx=0 tx=0).stop 2023-10-16T14:35:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 msgr2=0x7fbcb41f1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbcb41f1e60 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fbca400f3d0 tx=0x7fbca400f890 comp rx=0 tx=0).stop 2023-10-16T14:35:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 shutdown_connections 2023-10-16T14:35:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbca006af90 0x7fbca006d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbcb4151ca0 0x7fbcb4153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbcb41441e0 0x7fbcb4151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 --2- 172.21.15.29:0/4073137835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c0047c0 0x7fbcb41f1e60 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 >> 172.21.15.29:0/4073137835 conn(0x7fbcb406bc60 msgr2=0x7fbcb4159e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 shutdown_connections 2023-10-16T14:35:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.692+0000 7fbcb8d1a640 1 -- 172.21.15.29:0/4073137835 wait complete. 2023-10-16T14:35:19.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:35:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.939+0000 7fada2b0c640 1 Processor -- start 2023-10-16T14:35:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.940+0000 7fada2b0c640 1 -- start start 2023-10-16T14:35:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.941+0000 7fada2b0c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fad9c06c040 0x7fad9c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.941+0000 7fada2b0c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad9c0ad970 0x7fad9c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.941+0000 7fada2b0c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c15ebc0 0x7fad9c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.941+0000 7fada2b0c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9c06c960 con 0x7fad9c0ad970 2023-10-16T14:35:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.941+0000 7fada2b0c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9c15b9d0 con 0x7fad9c06c040 2023-10-16T14:35:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.941+0000 7fada2b0c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c15bcb0 con 0x7fad9c15ebc0 2023-10-16T14:35:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad9c0ad970 0x7fad9c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad9c0ad970 0x7fad9c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45974/0 (socket says 172.21.15.29:45974) 2023-10-16T14:35:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 -- 172.21.15.29:0/3540511926 learned_addr learned my addr 172.21.15.29:0/3540511926 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada230b640 1 --2- 172.21.15.29:0/3540511926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c15ebc0 0x7fad9c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1b0a640 1 --2- 172.21.15.29:0/3540511926 >> v2:172.21.15.170:3300/0 conn(0x7fad9c06c040 0x7fad9c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 -- 172.21.15.29:0/3540511926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c15ebc0 msgr2=0x7fad9c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 --2- 172.21.15.29:0/3540511926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c15ebc0 0x7fad9c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 -- 172.21.15.29:0/3540511926 >> v2:172.21.15.170:3300/0 conn(0x7fad9c06c040 msgr2=0x7fad9c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 --2- 172.21.15.29:0/3540511926 >> v2:172.21.15.170:3300/0 conn(0x7fad9c06c040 0x7fad9c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1309640 1 -- 172.21.15.29:0/3540511926 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c15bf90 con 0x7fad9c0ad970 2023-10-16T14:35:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.942+0000 7fada1b0a640 1 --2- 172.21.15.29:0/3540511926 >> v2:172.21.15.170:3300/0 conn(0x7fad9c06c040 0x7fad9c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.943+0000 7fada1309640 1 --2- 172.21.15.29:0/3540511926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c0ad970 0x7fad9c15e680 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0cf350 tx=0x7fad8c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4690ff8018a9a9fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada0b08640 1 -- 172.21.15.29:0/3540511926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c031860 con 0x7fad9c0ad970 2023-10-16T14:35:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada0b08640 1 -- 172.21.15.29:0/3540511926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c0319c0 con 0x7fad9c0ad970 2023-10-16T14:35:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada0b08640 1 -- 172.21.15.29:0/3540511926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c031d50 con 0x7fad9c0ad970 2023-10-16T14:35:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 -- 172.21.15.29:0/3540511926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c0ad970 msgr2=0x7fad9c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 --2- 172.21.15.29:0/3540511926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c0ad970 0x7fad9c15e680 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0cf350 tx=0x7fad8c02b900 comp rx=0 tx=0).stop 2023-10-16T14:35:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 -- 172.21.15.29:0/3540511926 shutdown_connections 2023-10-16T14:35:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 --2- 172.21.15.29:0/3540511926 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c15ebc0 0x7fad9c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 --2- 172.21.15.29:0/3540511926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c0ad970 0x7fad9c15e680 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 --2- 172.21.15.29:0/3540511926 >> v2:172.21.15.170:3300/0 conn(0x7fad9c06c040 0x7fad9c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 -- 172.21.15.29:0/3540511926 >> 172.21.15.29:0/3540511926 conn(0x7fad9c0ce6a0 msgr2=0x7fad9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.944+0000 7fada2b0c640 1 -- 172.21.15.29:0/3540511926 shutdown_connections 2023-10-16T14:35:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.945+0000 7fada2b0c640 1 -- 172.21.15.29:0/3540511926 wait complete. 2023-10-16T14:35:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.946+0000 7fada2b0c640 1 Processor -- start 2023-10-16T14:35:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.946+0000 7fada2b0c640 1 -- start start 2023-10-16T14:35:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.947+0000 7fada2b0c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c06c040 0x7fad9c14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.947+0000 7fada2b0c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c0ad970 0x7fad9c0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.947+0000 7fada2b0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 0x7fad9c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.947+0000 7fada2b0c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c14dcb0 con 0x7fad9c15ebc0 2023-10-16T14:35:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.947+0000 7fada2b0c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c0b2850 con 0x7fad9c06c040 2023-10-16T14:35:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.947+0000 7fada2b0c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c0ca410 con 0x7fad9c0ad970 2023-10-16T14:35:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 0x7fad9c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 0x7fad9c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45990/0 (socket says 172.21.15.29:45990) 2023-10-16T14:35:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 -- 172.21.15.29:0/1054205059 learned_addr learned my addr 172.21.15.29:0/1054205059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada1b0a640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c06c040 0x7fad9c14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada1309640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c0ad970 0x7fad9c0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 -- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c0ad970 msgr2=0x7fad9c0c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c0ad970 0x7fad9c0c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 -- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c06c040 msgr2=0x7fad9c14d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:19.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c06c040 0x7fad9c14d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:19.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.948+0000 7fada230b640 1 -- 172.21.15.29:0/1054205059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0094d0 con 0x7fad9c15ebc0 2023-10-16T14:35:19.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.949+0000 7fada230b640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 0x7fad9c0ac890 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fad94007e60 tx=0x7fad9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:19.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.949+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94014420 con 0x7fad9c15ebc0 2023-10-16T14:35:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.949+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad940145a0 con 0x7fad9c15ebc0 2023-10-16T14:35:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.949+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94014950 con 0x7fad9c15ebc0 2023-10-16T14:35:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.949+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9c0b2d40 con 0x7fad9c15ebc0 2023-10-16T14:35:19.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.949+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9c0b3300 con 0x7fad9c15ebc0 2023-10-16T14:35:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.950+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9c06b330 con 0x7fad9c15ebc0 2023-10-16T14:35:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.951+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fad94014ab0 con 0x7fad9c15ebc0 2023-10-16T14:35:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.953+0000 7fad927fc640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fad8006af90 0x7fad8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.953+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fad9408a190 con 0x7fad9c15ebc0 2023-10-16T14:35:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.953+0000 7fada1b0a640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fad8006af90 0x7fad8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.954+0000 7fada1b0a640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fad8006af90 0x7fad8006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0c4b70 tx=0x7fad98008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:19.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:19.957+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fad94055080 con 0x7fad9c15ebc0 2023-10-16T14:35:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.152+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fad9c04e9b0 con 0x7fad9c15ebc0 2023-10-16T14:35:20.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.158+0000 7fad927fc640 1 -- 172.21.15.29:0/1054205059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad940586a0 con 0x7fad9c15ebc0 2023-10-16T14:35:20.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fad8006af90 msgr2=0x7fad8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fad8006af90 0x7fad8006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0c4b70 tx=0x7fad98008040 comp rx=0 tx=0).stop 2023-10-16T14:35:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 msgr2=0x7fad9c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 0x7fad9c0ac890 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fad94007e60 tx=0x7fad9400e4b0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 shutdown_connections 2023-10-16T14:35:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fad8006af90 0x7fad8006d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad9c15ebc0 0x7fad9c0ac890 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad9c0ad970 0x7fad9c0c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 --2- 172.21.15.29:0/1054205059 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad9c06c040 0x7fad9c14d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.160+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 >> 172.21.15.29:0/1054205059 conn(0x7fad9c0ce6a0 msgr2=0x7fad9c0cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.161+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 shutdown_connections 2023-10-16T14:35:20.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.161+0000 7fada2b0c640 1 -- 172.21.15.29:0/1054205059 wait complete. 2023-10-16T14:35:20.182 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:35:20.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:35:20.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:35:20.434 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:35:20.434 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:35:20.435 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:20.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:35:20.501 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:35:20.501 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:20.560 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:35:20.561 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:35:20.561 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:20.626 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:35:20.626 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:35:20.627 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:20.688 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:35:20.689 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:35:20.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.920+0000 7f898f8b0640 1 Processor -- start 2023-10-16T14:35:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.921+0000 7f898f8b0640 1 -- start start 2023-10-16T14:35:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.922+0000 7f898f8b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8988160420 0x7f89880ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.922+0000 7f898f8b0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8988160800 0x7f89880cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.922+0000 7f898f8b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898806e170 con 0x7f89880adc40 2023-10-16T14:35:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.922+0000 7f898f8b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89880cde30 con 0x7f8988160420 2023-10-16T14:35:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.922+0000 7f898f8b0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89880cd460 con 0x7f8988160800 2023-10-16T14:35:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898de4d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f89880adc40 0x7f898806adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37208/0 (socket says 172.21.15.29:37208) 2023-10-16T14:35:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898d64c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8988160420 0x7f89880ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898de4d640 1 -- 172.21.15.29:0/1624786307 learned_addr learned my addr 172.21.15.29:0/1624786307 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898ce4b640 1 --2- 172.21.15.29:0/1624786307 >> v2:172.21.15.170:3300/0 conn(0x7f8988160800 0x7f89880cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f897ffff640 1 -- 172.21.15.29:0/1624786307 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701979552 0 0) 0x7f898806e170 con 0x7f89880adc40 2023-10-16T14:35:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898d64c640 1 -- 172.21.15.29:0/1624786307 >> v2:172.21.15.170:3300/0 conn(0x7f8988160800 msgr2=0x7f89880cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898d64c640 1 --2- 172.21.15.29:0/1624786307 >> v2:172.21.15.170:3300/0 conn(0x7f8988160800 0x7f89880cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898d64c640 1 -- 172.21.15.29:0/1624786307 >> v1:172.21.15.29:6789/0 conn(0x7f89880adc40 legacy=0x7f898806adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.923+0000 7f898d64c640 1 -- 172.21.15.29:0/1624786307 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898806e3c0 con 0x7f8988160420 2023-10-16T14:35:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.924+0000 7f898d64c640 1 --2- 172.21.15.29:0/1624786307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880ae170 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f89880cf1a0 tx=0x7f898402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6970e824c16f7c73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f897ffff640 1 -- 172.21.15.29:0/1624786307 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984031830 con 0x7f8988160420 2023-10-16T14:35:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f897ffff640 1 -- 172.21.15.29:0/1624786307 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984031990 con 0x7f8988160420 2023-10-16T14:35:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f897ffff640 1 -- 172.21.15.29:0/1624786307 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984031d20 con 0x7f8988160420 2023-10-16T14:35:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 -- 172.21.15.29:0/1624786307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 msgr2=0x7f89880ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 --2- 172.21.15.29:0/1624786307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880ae170 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f89880cf1a0 tx=0x7f898402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 -- 172.21.15.29:0/1624786307 shutdown_connections 2023-10-16T14:35:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 --2- 172.21.15.29:0/1624786307 >> v2:172.21.15.170:3300/0 conn(0x7f8988160800 0x7f89880cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 --2- 172.21.15.29:0/1624786307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880ae170 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 -- 172.21.15.29:0/1624786307 >> 172.21.15.29:0/1624786307 conn(0x7f89880b0d30 msgr2=0x7f89880b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 -- 172.21.15.29:0/1624786307 shutdown_connections 2023-10-16T14:35:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.925+0000 7f898f8b0640 1 -- 172.21.15.29:0/1624786307 wait complete. 2023-10-16T14:35:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.927+0000 7f898f8b0640 1 Processor -- start 2023-10-16T14:35:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.927+0000 7f898f8b0640 1 -- start start 2023-10-16T14:35:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.928+0000 7f898f8b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89880adc40 0x7f89880c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.928+0000 7f898f8b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.928+0000 7f898f8b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8988160800 0x7f89880c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.928+0000 7f898f8b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898814ac10 con 0x7f8988160420 2023-10-16T14:35:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.928+0000 7f898f8b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898815e880 con 0x7f8988160800 2023-10-16T14:35:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.928+0000 7f898f8b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898815eb60 con 0x7f89880adc40 2023-10-16T14:35:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46014/0 (socket says 172.21.15.29:46014) 2023-10-16T14:35:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 -- 172.21.15.29:0/902670405 learned_addr learned my addr 172.21.15.29:0/902670405 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898de4d640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8988160800 0x7f89880c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898d64c640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89880adc40 0x7f89880c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 -- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89880adc40 msgr2=0x7f89880c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89880adc40 0x7f89880c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 -- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8988160800 msgr2=0x7f89880c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8988160800 0x7f89880c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898ce4b640 1 -- 172.21.15.29:0/902670405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984009400 con 0x7f8988160420 2023-10-16T14:35:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898de4d640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8988160800 0x7f89880c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.929+0000 7f898d64c640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89880adc40 0x7f89880c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.930+0000 7f898ce4b640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880c4c90 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f897800d5f0 tx=0x7f897800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.930+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89780125f0 con 0x7f8988160420 2023-10-16T14:35:20.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.930+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8978012750 con 0x7f8988160420 2023-10-16T14:35:20.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.930+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978012b00 con 0x7f8988160420 2023-10-16T14:35:20.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.930+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898815ee40 con 0x7f8988160420 2023-10-16T14:35:20.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.930+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898815da50 con 0x7f8988160420 2023-10-16T14:35:20.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.931+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898814ad50 con 0x7f8988160420 2023-10-16T14:35:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.932+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8978012c60 con 0x7f8988160420 2023-10-16T14:35:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.933+0000 7f897dffb640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f896c06af90 0x7f896c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.933+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f8978089d00 con 0x7f8988160420 2023-10-16T14:35:20.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.934+0000 7f898d64c640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f896c06af90 0x7f896c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:20.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.935+0000 7f898d64c640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f896c06af90 0x7f896c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8984036000 tx=0x7f8984008030 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:20.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:20.938+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8978054c00 con 0x7f8988160420 2023-10-16T14:35:21.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.167+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f898804e160 con 0x7f8988160420 2023-10-16T14:35:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.168+0000 7f897dffb640 1 -- 172.21.15.29:0/902670405 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f8978058220 con 0x7f8988160420 2023-10-16T14:35:21.174 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:21.174 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3214217600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3214217600},{"type":"v1","addr":"172.21.15.170:6837","nonce":3214217600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2003272963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2003272963},{"type":"v1","addr":"172.21.15.170:6836","nonce":2003272963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":7317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/642999863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":642999863},{"type":"v1","addr":"172.21.15.29:6834","nonce":642999863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1850262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1850262499},{"type":"v1","addr":"172.21.15.29:6835","nonce":1850262499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[]} 2023-10-16T14:35:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 52 2023-10-16T14:35:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f896c06af90 msgr2=0x7f896c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f896c06af90 0x7f896c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8984036000 tx=0x7f8984008030 comp rx=0 tx=0).stop 2023-10-16T14:35:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 msgr2=0x7f89880c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880c4c90 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f897800d5f0 tx=0x7f897800dab0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 shutdown_connections 2023-10-16T14:35:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f896c06af90 0x7f896c06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8988160420 0x7f89880c4c90 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8988160800 0x7f89880c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 --2- 172.21.15.29:0/902670405 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89880adc40 0x7f89880c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 >> 172.21.15.29:0/902670405 conn(0x7f89880b0d30 msgr2=0x7f89880cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 shutdown_connections 2023-10-16T14:35:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.171+0000 7f898f8b0640 1 -- 172.21.15.29:0/902670405 wait complete. 2023-10-16T14:35:21.195 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:35:21.195 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:35:21.195 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:35:21.196 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:35:21.196 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:35:21.196 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:35:21.196 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:35:21.197 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:35:21.197 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:35:21.197 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:35:21.198 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:35:21.198 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:35:21.198 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:35:21.198 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:35:21.199 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:35:21.199 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:35:21.216 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:35:21.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:35:21.218 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:35:21.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:35:21.219 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:35:21.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:35:21.221 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:35:21.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:35:21.265 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:35:21.266 DEBUG:teuthology.parallel:result is None 2023-10-16T14:35:21.269 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:35:21.270 DEBUG:teuthology.parallel:result is None 2023-10-16T14:35:21.272 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:35:21.273 DEBUG:teuthology.parallel:result is None 2023-10-16T14:35:21.277 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:35:21.278 DEBUG:teuthology.parallel:result is None 2023-10-16T14:35:21.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:35:21.432 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:35:21.433 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:35:21.435 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:35:21.435 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:35:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.473+0000 7f9626d0f640 1 Processor -- start 2023-10-16T14:35:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.473+0000 7f9626d0f640 1 -- start start 2023-10-16T14:35:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9626d0f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9620160ad0 0x7f96200ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9626d0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96200ae260 0x7f962006a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9626d0f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200ce800 con 0x7f96200ce350 2023-10-16T14:35:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9626d0f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96200cdec0 con 0x7f9620160ad0 2023-10-16T14:35:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9626d0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96200acdd0 con 0x7f96200ae260 2023-10-16T14:35:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9620160ad0 0x7f96200ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9620160ad0 0x7f96200ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46022/0 (socket says 172.21.15.29:46022) 2023-10-16T14:35:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 -- 172.21.15.29:0/412831993 learned_addr learned my addr 172.21.15.29:0/412831993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f96252ac640 1 --2- 172.21.15.29:0/412831993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96200ae260 0x7f962006a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 -- 172.21.15.29:0/412831993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96200ae260 msgr2=0x7f962006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 --2- 172.21.15.29:0/412831993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96200ae260 0x7f962006a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 -- 172.21.15.29:0/412831993 >> v1:172.21.15.170:6789/0 conn(0x7f96200ce350 legacy=0x7f9620160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.475+0000 7f9624aab640 1 -- 172.21.15.29:0/412831993 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200ae740 con 0x7f9620160ad0 2023-10-16T14:35:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.476+0000 7f9624aab640 1 --2- 172.21.15.29:0/412831993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9620160ad0 0x7f96200ac890 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f9614009870 tx=0x7f961402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58962bf5f3e8f893 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f961f7fe640 1 -- 172.21.15.29:0/412831993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614030840 con 0x7f9620160ad0 2023-10-16T14:35:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f961f7fe640 1 -- 172.21.15.29:0/412831993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96140309a0 con 0x7f9620160ad0 2023-10-16T14:35:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f961f7fe640 1 -- 172.21.15.29:0/412831993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614030d30 con 0x7f9620160ad0 2023-10-16T14:35:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f9626d0f640 1 -- 172.21.15.29:0/412831993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9620160ad0 msgr2=0x7f96200ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f9626d0f640 1 --2- 172.21.15.29:0/412831993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9620160ad0 0x7f96200ac890 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f9614009870 tx=0x7f961402b980 comp rx=0 tx=0).stop 2023-10-16T14:35:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f9626d0f640 1 -- 172.21.15.29:0/412831993 shutdown_connections 2023-10-16T14:35:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f9626d0f640 1 --2- 172.21.15.29:0/412831993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96200ae260 0x7f962006a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f9626d0f640 1 --2- 172.21.15.29:0/412831993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9620160ad0 0x7f96200ac890 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.477+0000 7f9626d0f640 1 -- 172.21.15.29:0/412831993 >> 172.21.15.29:0/412831993 conn(0x7f962006bc10 msgr2=0x7f962006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.478+0000 7f9626d0f640 1 -- 172.21.15.29:0/412831993 shutdown_connections 2023-10-16T14:35:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.478+0000 7f9626d0f640 1 -- 172.21.15.29:0/412831993 wait complete. 2023-10-16T14:35:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.479+0000 7f9626d0f640 1 Processor -- start 2023-10-16T14:35:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.479+0000 7f9626d0f640 1 -- start start 2023-10-16T14:35:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.480+0000 7f9626d0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 0x7f9620157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.480+0000 7f9626d0f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96200ce350 0x7f96200b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.480+0000 7f9626d0f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9620160ad0 0x7f96200aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.480+0000 7f9626d0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b4960 con 0x7f96200ae260 2023-10-16T14:35:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.480+0000 7f9626d0f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b4c40 con 0x7f96200ce350 2023-10-16T14:35:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.480+0000 7f9626d0f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96200c6420 con 0x7f9620160ad0 2023-10-16T14:35:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 0x7f9620157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 0x7f9620157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46030/0 (socket says 172.21.15.29:46030) 2023-10-16T14:35:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 -- 172.21.15.29:0/3900617864 learned_addr learned my addr 172.21.15.29:0/3900617864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f961ffff640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96200ce350 0x7f96200b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f96252ac640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9620160ad0 0x7f96200aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 -- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9620160ad0 msgr2=0x7f96200aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9620160ad0 0x7f96200aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 -- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96200ce350 msgr2=0x7f96200b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96200ce350 0x7f96200b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f9624aab640 1 -- 172.21.15.29:0/3900617864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140094d0 con 0x7f96200ae260 2023-10-16T14:35:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.481+0000 7f961ffff640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96200ce350 0x7f96200b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.482+0000 7f9624aab640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 0x7f9620157740 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f9614000c00 tx=0x7f9614004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.482+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614039a90 con 0x7f96200ae260 2023-10-16T14:35:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.482+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614039bf0 con 0x7f96200ae260 2023-10-16T14:35:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.482+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614043410 con 0x7f96200ae260 2023-10-16T14:35:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.482+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96200c6700 con 0x7f96200ae260 2023-10-16T14:35:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.482+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9620157c80 con 0x7f96200ae260 2023-10-16T14:35:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.483+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96200c6260 con 0x7f96200ae260 2023-10-16T14:35:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.484+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f961402f390 con 0x7f96200ae260 2023-10-16T14:35:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.486+0000 7f961d7fa640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f960406af90 0x7f960406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.486+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f96140abb50 con 0x7f96200ae260 2023-10-16T14:35:21.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.486+0000 7f961ffff640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f960406af90 0x7f960406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:21.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.487+0000 7f961ffff640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f960406af90 0x7f960406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f960c007fd0 tx=0x7f960c009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.490+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9614076930 con 0x7f96200ae260 2023-10-16T14:35:21.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:21.699+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f962004e160 con 0x7f96200ae260 2023-10-16T14:35:22.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.726+0000 7f961d7fa640 1 -- 172.21.15.29:0/3900617864 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v123) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9614034080 con 0x7f96200ae260 2023-10-16T14:35:22.730 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:35:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f960406af90 msgr2=0x7f960406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f960406af90 0x7f960406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f960c007fd0 tx=0x7f960c009040 comp rx=0 tx=0).stop 2023-10-16T14:35:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 msgr2=0x7f9620157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 0x7f9620157740 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f9614000c00 tx=0x7f9614004730 comp rx=0 tx=0).stop 2023-10-16T14:35:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 shutdown_connections 2023-10-16T14:35:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f960406af90 0x7f960406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9620160ad0 0x7f96200aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96200ce350 0x7f96200b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 --2- 172.21.15.29:0/3900617864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96200ae260 0x7f9620157740 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.731+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 >> 172.21.15.29:0/3900617864 conn(0x7f962006bc10 msgr2=0x7f9620070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.732+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 shutdown_connections 2023-10-16T14:35:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.732+0000 7f9626d0f640 1 -- 172.21.15.29:0/3900617864 wait complete. 2023-10-16T14:35:22.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:35:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.981+0000 7f539f290640 1 Processor -- start 2023-10-16T14:35:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.981+0000 7f539f290640 1 -- start start 2023-10-16T14:35:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539f290640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160420 0x7f53980ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539f290640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5398160800 0x7f53980cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539f290640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539806e170 con 0x7f53980adc40 2023-10-16T14:35:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539f290640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53980cde30 con 0x7f5398160800 2023-10-16T14:35:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539f290640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53980cd460 con 0x7f5398160420 2023-10-16T14:35:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539c82b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5398160800 0x7f53980cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539c82b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5398160800 0x7f53980cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46034/0 (socket says 172.21.15.29:46034) 2023-10-16T14:35:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.983+0000 7f539c82b640 1 -- 172.21.15.29:0/574536731 learned_addr learned my addr 172.21.15.29:0/574536731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.984+0000 7f539c82b640 1 -- 172.21.15.29:0/574536731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160420 msgr2=0x7f53980ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.984+0000 7f539c82b640 1 --2- 172.21.15.29:0/574536731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160420 0x7f53980ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.984+0000 7f539c82b640 1 -- 172.21.15.29:0/574536731 >> v1:172.21.15.170:6789/0 conn(0x7f53980adc40 legacy=0x7f539806adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.984+0000 7f539c82b640 1 -- 172.21.15.29:0/574536731 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539806e3c0 con 0x7f5398160800 2023-10-16T14:35:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.985+0000 7f539c82b640 1 --2- 172.21.15.29:0/574536731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160800 0x7f53980cd8f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5388009870 tx=0x7f538802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae358474be42a70f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.985+0000 7f538ffff640 1 -- 172.21.15.29:0/574536731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388031640 con 0x7f5398160800 2023-10-16T14:35:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.985+0000 7f538ffff640 1 -- 172.21.15.29:0/574536731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53880317a0 con 0x7f5398160800 2023-10-16T14:35:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.985+0000 7f538ffff640 1 -- 172.21.15.29:0/574536731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388031b30 con 0x7f5398160800 2023-10-16T14:35:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 -- 172.21.15.29:0/574536731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160800 msgr2=0x7f53980cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 --2- 172.21.15.29:0/574536731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160800 0x7f53980cd8f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5388009870 tx=0x7f538802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 -- 172.21.15.29:0/574536731 shutdown_connections 2023-10-16T14:35:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 --2- 172.21.15.29:0/574536731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160800 0x7f53980cd8f0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 --2- 172.21.15.29:0/574536731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160420 0x7f53980ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 -- 172.21.15.29:0/574536731 >> 172.21.15.29:0/574536731 conn(0x7f53980b0d30 msgr2=0x7f53980b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 -- 172.21.15.29:0/574536731 shutdown_connections 2023-10-16T14:35:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.986+0000 7f539f290640 1 -- 172.21.15.29:0/574536731 wait complete. 2023-10-16T14:35:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.988+0000 7f539f290640 1 Processor -- start 2023-10-16T14:35:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.988+0000 7f539f290640 1 -- start start 2023-10-16T14:35:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539f290640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53980adc40 0x7f53981f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539f290640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 0x7f539806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539f290640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160800 0x7f539815e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539f290640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539809e4a0 con 0x7f5398160420 2023-10-16T14:35:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539f290640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539809e740 con 0x7f53980adc40 2023-10-16T14:35:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539f290640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539809ea20 con 0x7f5398160800 2023-10-16T14:35:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.989+0000 7f539c82b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 0x7f539806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 0x7f539806a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46046/0 (socket says 172.21.15.29:46046) 2023-10-16T14:35:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 -- 172.21.15.29:0/3067632282 learned_addr learned my addr 172.21.15.29:0/3067632282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539d02c640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53980adc40 0x7f53981f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539d82d640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160800 0x7f539815e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 -- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160800 msgr2=0x7f539815e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160800 0x7f539815e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 -- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53980adc40 msgr2=0x7f53981f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53980adc40 0x7f53981f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 -- 172.21.15.29:0/3067632282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880094d0 con 0x7f5398160420 2023-10-16T14:35:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.990+0000 7f539c82b640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 0x7f539806a6b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f5388009870 tx=0x7f53880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.991+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538803ac60 con 0x7f5398160420 2023-10-16T14:35:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.991+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538803ade0 con 0x7f5398160420 2023-10-16T14:35:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.991+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388044630 con 0x7f5398160420 2023-10-16T14:35:23.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.991+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539809ed00 con 0x7f5398160420 2023-10-16T14:35:23.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.991+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53980a02a0 con 0x7f5398160420 2023-10-16T14:35:23.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.992+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539804e160 con 0x7f5398160420 2023-10-16T14:35:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.993+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f538802f050 con 0x7f5398160420 2023-10-16T14:35:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.994+0000 7f538dffb640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f537c06af90 0x7f537c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.994+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f53880ab6a0 con 0x7f5398160420 2023-10-16T14:35:23.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.995+0000 7f539d02c640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f537c06af90 0x7f537c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.996+0000 7f539d02c640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f537c06af90 0x7f537c06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5394009890 tx=0x7f5394008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:23.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:22.998+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5388076870 con 0x7f5398160420 2023-10-16T14:35:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.273+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f539809eb60 con 0x7f5398160420 2023-10-16T14:35:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.274+0000 7f538dffb640 1 -- 172.21.15.29:0/3067632282 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f539809eb60 con 0x7f5398160420 2023-10-16T14:35:23.278 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:23.279 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:35:23.279 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:35:23.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.279 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:35:23.279 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:35:23.280 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.280 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.280 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:35:23.280 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:35:23.280 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.281 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.281 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:35:23.281 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:35:23.281 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.282 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.282 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:35:23.282 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:35:23.282 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.283 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.283 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:35:23.283 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:35:23.283 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.284 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.284 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:35:23.284 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:35:23.284 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.284 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.285 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:35:23.285 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:35:23.285 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.285 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.286 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:35:23.286 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:35:23.286 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.286 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.287 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:35:23.288 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:35:23.288 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.288 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.288 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.288 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.289 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:35:23.289 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:35:23.290 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.290 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.290 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.290 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.290 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.291 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:35:23.291 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:35:23.291 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.291 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.292 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.292 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.292 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.293 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:35:23.293 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:35:23.293 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.293 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.293 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.294 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.294 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.294 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:35:23.295 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:35:23.295 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.295 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.295 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.295 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.296 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.296 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:35:23.296 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:35:23.297 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.298 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.298 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.298 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.298 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.299 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:35:23.299 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:35:23.299 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.299 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.300 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.300 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.300 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.301 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:35:23.301 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:35:23.301 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.301 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:35:23.301 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:35:23.302 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.302 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.302 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:35:23.302 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:35:23.303 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.303 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.303 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:35:23.303 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:35:23.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:35:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:35:23.304 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.305 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.305 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:35:23.305 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:35:23.305 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:35:23.305 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.306 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:35:23.306 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:35:23.306 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.306 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.307 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:35:23.307 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:35:23.307 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:35:23.309 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.310 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:35:23.310 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:35:23.310 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.310 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:35:23.310 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:35:23.311 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:35:23.311 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:35:23.311 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.312 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:35:23.312 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:35:23.312 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.312 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:35:23.312 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:35:23.313 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:35:23.313 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.313 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.314 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:35:23.314 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:35:23.314 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.314 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:35:23.314 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:35:23.315 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.315 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:35:23.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:23.315 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:35:23.316 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:35:23.316 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:35:23.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:35:23.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:35:23.316 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:35:23.316 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:23.317 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:35:23.317 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:35:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f537c06af90 msgr2=0x7f537c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f537c06af90 0x7f537c06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5394009890 tx=0x7f5394008040 comp rx=0 tx=0).stop 2023-10-16T14:35:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 msgr2=0x7f539806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 0x7f539806a6b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f5388009870 tx=0x7f53880046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 shutdown_connections 2023-10-16T14:35:23.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f537c06af90 0x7f537c06d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5398160420 0x7f539806a6b0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5398160800 0x7f539815e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 --2- 172.21.15.29:0/3067632282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f53980adc40 0x7f53981f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 >> 172.21.15.29:0/3067632282 conn(0x7f53980b0d30 msgr2=0x7f539815aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:23.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 shutdown_connections 2023-10-16T14:35:23.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.277+0000 7f539f290640 1 -- 172.21.15.29:0/3067632282 wait complete. 2023-10-16T14:35:23.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.543+0000 7fb1573b6640 1 Processor -- start 2023-10-16T14:35:23.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.544+0000 7fb1573b6640 1 -- start start 2023-10-16T14:35:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.545+0000 7fb1573b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb1501615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.545+0000 7fb1573b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb15015d3f0 0x7fb15015d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.545+0000 7fb1573b6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15015e3e0 con 0x7fb1500cd0c0 2023-10-16T14:35:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.545+0000 7fb1573b6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1500cd4a0 con 0x7fb15015d3f0 2023-10-16T14:35:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.545+0000 7fb1573b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15015e100 con 0x7fb1500b0b20 2023-10-16T14:35:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.546+0000 7fb155152640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb1500cd0c0 0x7fb1500b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37230/0 (socket says 172.21.15.29:37230) 2023-10-16T14:35:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.546+0000 7fb155152640 1 -- 172.21.15.29:0/1285393828 learned_addr learned my addr 172.21.15.29:0/1285393828 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.546+0000 7fb155953640 1 --2- 172.21.15.29:0/1285393828 >> v2:172.21.15.29:3300/0 conn(0x7fb15015d3f0 0x7fb15015d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.546+0000 7fb154951640 1 --2- 172.21.15.29:0/1285393828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb1501615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.547+0000 7fb147fff640 1 -- 172.21.15.29:0/1285393828 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326732208 0 0) 0x7fb15015e3e0 con 0x7fb1500cd0c0 2023-10-16T14:35:23.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.547+0000 7fb155953640 1 -- 172.21.15.29:0/1285393828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 msgr2=0x7fb1501615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:23.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.547+0000 7fb155953640 1 --2- 172.21.15.29:0/1285393828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb1501615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.547+0000 7fb155953640 1 -- 172.21.15.29:0/1285393828 >> v1:172.21.15.29:6789/0 conn(0x7fb1500cd0c0 legacy=0x7fb1500b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.547+0000 7fb155953640 1 -- 172.21.15.29:0/1285393828 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb150160b10 con 0x7fb15015d3f0 2023-10-16T14:35:23.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.548+0000 7fb155953640 1 --2- 172.21.15.29:0/1285393828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb15015d860 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d90 tx=0x7fb14802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754fd015c08f28f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.548+0000 7fb147fff640 1 -- 172.21.15.29:0/1285393828 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14802cbd0 con 0x7fb15015d3f0 2023-10-16T14:35:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.548+0000 7fb147fff640 1 -- 172.21.15.29:0/1285393828 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14802cd30 con 0x7fb15015d3f0 2023-10-16T14:35:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.548+0000 7fb147fff640 1 -- 172.21.15.29:0/1285393828 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14802cea0 con 0x7fb15015d3f0 2023-10-16T14:35:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.548+0000 7fb1573b6640 1 -- 172.21.15.29:0/1285393828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 msgr2=0x7fb15015d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.548+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1285393828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb15015d860 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d90 tx=0x7fb14802be20 comp rx=0 tx=0).stop 2023-10-16T14:35:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.549+0000 7fb1573b6640 1 -- 172.21.15.29:0/1285393828 shutdown_connections 2023-10-16T14:35:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.549+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1285393828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb15015d860 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.549+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1285393828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb1501615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.549+0000 7fb1573b6640 1 -- 172.21.15.29:0/1285393828 >> 172.21.15.29:0/1285393828 conn(0x7fb1500cb410 msgr2=0x7fb1500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.549+0000 7fb1573b6640 1 -- 172.21.15.29:0/1285393828 shutdown_connections 2023-10-16T14:35:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.549+0000 7fb1573b6640 1 -- 172.21.15.29:0/1285393828 wait complete. 2023-10-16T14:35:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.550+0000 7fb1573b6640 1 Processor -- start 2023-10-16T14:35:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.551+0000 7fb1573b6640 1 -- start start 2023-10-16T14:35:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb1573b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb150152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb1573b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1500cd0c0 0x7fb150154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb1573b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb1500aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb1573b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1501544b0 con 0x7fb15015d3f0 2023-10-16T14:35:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb1573b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1501538a0 con 0x7fb1500cd0c0 2023-10-16T14:35:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb1573b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb150153b80 con 0x7fb1500b0b20 2023-10-16T14:35:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb1500aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb1500aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46050/0 (socket says 172.21.15.29:46050) 2023-10-16T14:35:23.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 -- 172.21.15.29:0/1145917857 learned_addr learned my addr 172.21.15.29:0/1145917857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:23.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155152640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb150152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb154951640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1500cd0c0 0x7fb150154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 -- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 msgr2=0x7fb150152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb150152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 -- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1500cd0c0 msgr2=0x7fb150154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1500cd0c0 0x7fb150154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb155953640 1 -- 172.21.15.29:0/1145917857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148009a70 con 0x7fb15015d3f0 2023-10-16T14:35:23.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.552+0000 7fb154951640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1500cd0c0 0x7fb150154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:23.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.553+0000 7fb155953640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb1500aef10 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d90 tx=0x7fb148004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:23.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.553+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14802cea0 con 0x7fb15015d3f0 2023-10-16T14:35:23.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.553+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148040650 con 0x7fb15015d3f0 2023-10-16T14:35:23.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.553+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148040a00 con 0x7fb15015d3f0 2023-10-16T14:35:23.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.553+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb150153e60 con 0x7fb15015d3f0 2023-10-16T14:35:23.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.553+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb150153320 con 0x7fb15015d3f0 2023-10-16T14:35:23.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.554+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb150154660 con 0x7fb15015d3f0 2023-10-16T14:35:23.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.555+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb148040b60 con 0x7fb15015d3f0 2023-10-16T14:35:23.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.557+0000 7fb145ffb640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb13406af90 0x7fb13406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:23.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.557+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb1480ac070 con 0x7fb15015d3f0 2023-10-16T14:35:23.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.557+0000 7fb155152640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb13406af90 0x7fb13406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:23.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.559+0000 7fb155152640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb13406af90 0x7fb13406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb14c000f60 tx=0x7fb14c007b70 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:23.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.561+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb1480deca0 con 0x7fb15015d3f0 2023-10-16T14:35:23.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.756+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb15004e160 con 0x7fb15015d3f0 2023-10-16T14:35:23.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.757+0000 7fb145ffb640 1 -- 172.21.15.29:0/1145917857 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fb148077350 con 0x7fb15015d3f0 2023-10-16T14:35:23.761 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:23.761 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":123,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:22.723764+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.759+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb13406af90 msgr2=0x7fb13406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.759+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb13406af90 0x7fb13406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb14c000f60 tx=0x7fb14c007b70 comp rx=0 tx=0).stop 2023-10-16T14:35:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 msgr2=0x7fb1500aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:23.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb1500aef10 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d90 tx=0x7fb148004a40 comp rx=0 tx=0).stop 2023-10-16T14:35:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 shutdown_connections 2023-10-16T14:35:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fb13406af90 0x7fb13406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb15015d3f0 0x7fb1500aef10 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1500cd0c0 0x7fb150154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 --2- 172.21.15.29:0/1145917857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1500b0b20 0x7fb150152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 >> 172.21.15.29:0/1145917857 conn(0x7fb1500cb410 msgr2=0x7fb15006c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 shutdown_connections 2023-10-16T14:35:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.760+0000 7fb1573b6640 1 -- 172.21.15.29:0/1145917857 wait complete. 2023-10-16T14:35:23.784 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:35:23.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:35:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.997+0000 7fc9c9890640 1 Processor -- start 2023-10-16T14:35:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.997+0000 7fc9c9890640 1 -- start start 2023-10-16T14:35:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.998+0000 7fc9c9890640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9c40b0ea0 0x7fc9c415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.998+0000 7fc9c9890640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc9c40cd970 0x7fc9c415a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.998+0000 7fc9c9890640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c406be30 con 0x7fc9c40b3ec0 2023-10-16T14:35:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.998+0000 7fc9c9890640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c406ca00 con 0x7fc9c40b0ea0 2023-10-16T14:35:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.998+0000 7fc9c9890640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40ada10 con 0x7fc9c40cd970 2023-10-16T14:35:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c888e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc9c40b3ec0 0x7fc9c40cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32870/0 (socket says 172.21.15.29:32870) 2023-10-16T14:35:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c888e640 1 -- 172.21.15.29:0/1710897470 learned_addr learned my addr 172.21.15.29:0/1710897470 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c3fff640 1 --2- 172.21.15.29:0/1710897470 >> v2:172.21.15.29:3300/0 conn(0x7fc9c40b0ea0 0x7fc9c415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c37fe640 1 -- 172.21.15.29:0/1710897470 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621523528 0 0) 0x7fc9c406be30 con 0x7fc9c40b3ec0 2023-10-16T14:35:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c37fe640 1 -- 172.21.15.29:0/1710897470 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b4003520 con 0x7fc9c40b3ec0 2023-10-16T14:35:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c3fff640 1 -- 172.21.15.29:0/1710897470 >> v2:172.21.15.170:3300/0 conn(0x7fc9c40cd970 msgr2=0x7fc9c415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c908f640 1 --2- 172.21.15.29:0/1710897470 >> v2:172.21.15.170:3300/0 conn(0x7fc9c40cd970 0x7fc9c415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c3fff640 1 --2- 172.21.15.29:0/1710897470 >> v2:172.21.15.170:3300/0 conn(0x7fc9c40cd970 0x7fc9c415a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c3fff640 1 -- 172.21.15.29:0/1710897470 >> v1:172.21.15.29:6789/0 conn(0x7fc9c40b3ec0 legacy=0x7fc9c40cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c3fff640 1 -- 172.21.15.29:0/1710897470 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c406c060 con 0x7fc9c40b0ea0 2023-10-16T14:35:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:23.999+0000 7fc9c908f640 1 --2- 172.21.15.29:0/1710897470 >> v2:172.21.15.170:3300/0 conn(0x7fc9c40cd970 0x7fc9c415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.001+0000 7fc9c3fff640 1 --2- 172.21.15.29:0/1710897470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40b0ea0 0x7fc9c415a2f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4071b60 tx=0x7fc9b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595a4d8d89cb35a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.001+0000 7fc9c37fe640 1 -- 172.21.15.29:0/1710897470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b002e070 con 0x7fc9c40b0ea0 2023-10-16T14:35:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.001+0000 7fc9c37fe640 1 -- 172.21.15.29:0/1710897470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b00315a0 con 0x7fc9c40b0ea0 2023-10-16T14:35:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.001+0000 7fc9c37fe640 1 -- 172.21.15.29:0/1710897470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0031930 con 0x7fc9c40b0ea0 2023-10-16T14:35:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.001+0000 7fc9c9890640 1 -- 172.21.15.29:0/1710897470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40b0ea0 msgr2=0x7fc9c415a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.001+0000 7fc9c9890640 1 --2- 172.21.15.29:0/1710897470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40b0ea0 0x7fc9c415a2f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4071b60 tx=0x7fc9b002b840 comp rx=0 tx=0).stop 2023-10-16T14:35:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.002+0000 7fc9c9890640 1 -- 172.21.15.29:0/1710897470 shutdown_connections 2023-10-16T14:35:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.002+0000 7fc9c9890640 1 --2- 172.21.15.29:0/1710897470 >> v2:172.21.15.170:3300/0 conn(0x7fc9c40cd970 0x7fc9c415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.002+0000 7fc9c9890640 1 --2- 172.21.15.29:0/1710897470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40b0ea0 0x7fc9c415a2f0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.002+0000 7fc9c9890640 1 -- 172.21.15.29:0/1710897470 >> 172.21.15.29:0/1710897470 conn(0x7fc9c40b0170 msgr2=0x7fc9c40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.002+0000 7fc9c9890640 1 -- 172.21.15.29:0/1710897470 shutdown_connections 2023-10-16T14:35:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.002+0000 7fc9c9890640 1 -- 172.21.15.29:0/1710897470 wait complete. 2023-10-16T14:35:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.003+0000 7fc9c9890640 1 Processor -- start 2023-10-16T14:35:24.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.003+0000 7fc9c9890640 1 -- start start 2023-10-16T14:35:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.004+0000 7fc9c9890640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9c40b0ea0 0x7fc9c41f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.004+0000 7fc9c9890640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9c40b3ec0 0x7fc9c40c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.004+0000 7fc9c9890640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 0x7fc9c4159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.004+0000 7fc9c9890640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4158930 con 0x7fc9c40cd970 2023-10-16T14:35:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.004+0000 7fc9c9890640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4157a10 con 0x7fc9c40b3ec0 2023-10-16T14:35:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.004+0000 7fc9c9890640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4157cb0 con 0x7fc9c40b0ea0 2023-10-16T14:35:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 0x7fc9c4159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 0x7fc9c4159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51878/0 (socket says 172.21.15.29:51878) 2023-10-16T14:35:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 -- 172.21.15.29:0/3710717236 learned_addr learned my addr 172.21.15.29:0/3710717236 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c888e640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9c40b0ea0 0x7fc9c41f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c3fff640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9c40b3ec0 0x7fc9c40c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 -- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9c40b0ea0 msgr2=0x7fc9c41f0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9c40b0ea0 0x7fc9c41f0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 -- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9c40b3ec0 msgr2=0x7fc9c40c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9c40b3ec0 0x7fc9c40c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.005+0000 7fc9c908f640 1 -- 172.21.15.29:0/3710717236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b0009400 con 0x7fc9c40cd970 2023-10-16T14:35:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.006+0000 7fc9c908f640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 0x7fc9c4159810 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00cef0 tx=0x7fc9bc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.006+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00d840 con 0x7fc9c40cd970 2023-10-16T14:35:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.006+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc00d9c0 con 0x7fc9c40cd970 2023-10-16T14:35:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.006+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c4157f90 con 0x7fc9c40cd970 2023-10-16T14:35:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.006+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00dd70 con 0x7fc9c40cd970 2023-10-16T14:35:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.006+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c4158550 con 0x7fc9c40cd970 2023-10-16T14:35:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.007+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c4158a70 con 0x7fc9c40cd970 2023-10-16T14:35:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.008+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9bc01e3b0 con 0x7fc9c40cd970 2023-10-16T14:35:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.009+0000 7fc9c17fa640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc9a806af90 0x7fc9a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.009+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc9bc089f30 con 0x7fc9c40cd970 2023-10-16T14:35:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.010+0000 7fc9c888e640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc9a806af90 0x7fc9a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.011+0000 7fc9c888e640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc9a806af90 0x7fc9a806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80011e0 tx=0x7fc9b8009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.013+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9c4158a70 con 0x7fc9c40cd970 2023-10-16T14:35:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.206+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9c4158c50 con 0x7fc9c40cd970 2023-10-16T14:35:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.207+0000 7fc9c17fa640 1 -- 172.21.15.29:0/3710717236 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9c4158c50 con 0x7fc9c40cd970 2023-10-16T14:35:24.211 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:35:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc9a806af90 msgr2=0x7fc9a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc9a806af90 0x7fc9a806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80011e0 tx=0x7fc9b8009040 comp rx=0 tx=0).stop 2023-10-16T14:35:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 msgr2=0x7fc9c4159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 0x7fc9c4159810 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00cef0 tx=0x7fc9bc00a710 comp rx=0 tx=0).stop 2023-10-16T14:35:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 shutdown_connections 2023-10-16T14:35:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fc9a806af90 0x7fc9a806d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9c40cd970 0x7fc9c4159810 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9c40b3ec0 0x7fc9c40c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 --2- 172.21.15.29:0/3710717236 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9c40b0ea0 0x7fc9c41f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 >> 172.21.15.29:0/3710717236 conn(0x7fc9c40b0170 msgr2=0x7fc9c40b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.209+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 shutdown_connections 2023-10-16T14:35:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.210+0000 7fc9c9890640 1 -- 172.21.15.29:0/3710717236 wait complete. 2023-10-16T14:35:24.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:35:24.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.451+0000 7f04cd5ed640 1 Processor -- start 2023-10-16T14:35:24.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.452+0000 7f04cd5ed640 1 -- start start 2023-10-16T14:35:24.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.453+0000 7f04cd5ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04c80ad970 0x7f04c80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.453+0000 7f04cd5ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f04c80ccfb0 0x7f04c80cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.453+0000 7f04cd5ed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c815b9d0 con 0x7f04c806c040 2023-10-16T14:35:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.453+0000 7f04cd5ed640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c815bcb0 con 0x7f04c80ad970 2023-10-16T14:35:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.453+0000 7f04cd5ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c815bf90 con 0x7f04c80ccfb0 2023-10-16T14:35:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04bffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04c80ad970 0x7f04c80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04c7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f04c806c040 0x7f04c806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32872/0 (socket says 172.21.15.29:32872) 2023-10-16T14:35:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04c7fff640 1 -- 172.21.15.29:0/2703927358 learned_addr learned my addr 172.21.15.29:0/2703927358 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04ccdec640 1 --2- 172.21.15.29:0/2703927358 >> v2:172.21.15.170:3300/0 conn(0x7f04c80ccfb0 0x7f04c80cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04bffff640 1 -- 172.21.15.29:0/2703927358 >> v2:172.21.15.170:3300/0 conn(0x7f04c80ccfb0 msgr2=0x7f04c80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04bffff640 1 --2- 172.21.15.29:0/2703927358 >> v2:172.21.15.170:3300/0 conn(0x7f04c80ccfb0 0x7f04c80cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04bffff640 1 -- 172.21.15.29:0/2703927358 >> v1:172.21.15.29:6789/0 conn(0x7f04c806c040 legacy=0x7f04c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.454+0000 7f04bffff640 1 -- 172.21.15.29:0/2703927358 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80cc270 con 0x7f04c80ad970 2023-10-16T14:35:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.456+0000 7f04bffff640 1 --2- 172.21.15.29:0/2703927358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ad970 0x7f04c80cca70 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f04b00097a0 tx=0x7f04b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab41da6633ce1057 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.456+0000 7f04c77fe640 1 -- 172.21.15.29:0/2703927358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0031830 con 0x7f04c80ad970 2023-10-16T14:35:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.456+0000 7f04c77fe640 1 -- 172.21.15.29:0/2703927358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b0031990 con 0x7f04c80ad970 2023-10-16T14:35:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.456+0000 7f04c77fe640 1 -- 172.21.15.29:0/2703927358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0031d20 con 0x7f04c80ad970 2023-10-16T14:35:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.456+0000 7f04cd5ed640 1 -- 172.21.15.29:0/2703927358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ad970 msgr2=0x7f04c80cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.456+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/2703927358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ad970 0x7f04c80cca70 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f04b00097a0 tx=0x7f04b002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.457+0000 7f04cd5ed640 1 -- 172.21.15.29:0/2703927358 shutdown_connections 2023-10-16T14:35:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.457+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/2703927358 >> v2:172.21.15.170:3300/0 conn(0x7f04c80ccfb0 0x7f04c80cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.457+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/2703927358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ad970 0x7f04c80cca70 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.457+0000 7f04cd5ed640 1 -- 172.21.15.29:0/2703927358 >> 172.21.15.29:0/2703927358 conn(0x7f04c80ce6a0 msgr2=0x7f04c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.457+0000 7f04cd5ed640 1 -- 172.21.15.29:0/2703927358 shutdown_connections 2023-10-16T14:35:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.457+0000 7f04cd5ed640 1 -- 172.21.15.29:0/2703927358 wait complete. 2023-10-16T14:35:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.458+0000 7f04cd5ed640 1 Processor -- start 2023-10-16T14:35:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.459+0000 7f04cd5ed640 1 -- start start 2023-10-16T14:35:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.459+0000 7f04cd5ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04c806c040 0x7f04c81efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04cd5ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04c80ad970 0x7f04c80b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04cd5ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 0x7f04c80b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04cd5ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8154b70 con 0x7f04c80ccfb0 2023-10-16T14:35:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04cd5ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b3350 con 0x7f04c806c040 2023-10-16T14:35:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04cd5ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b3630 con 0x7f04c80ad970 2023-10-16T14:35:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 0x7f04c80b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 0x7f04c80b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51904/0 (socket says 172.21.15.29:51904) 2023-10-16T14:35:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 -- 172.21.15.29:0/1329960253 learned_addr learned my addr 172.21.15.29:0/1329960253 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04c7fff640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04c806c040 0x7f04c81efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 -- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04c80ad970 msgr2=0x7f04c80b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04c80ad970 0x7f04c80b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 -- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04c806c040 msgr2=0x7f04c81efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04c806c040 0x7f04c81efad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.460+0000 7f04ccdec640 1 -- 172.21.15.29:0/1329960253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b0009400 con 0x7f04c80ccfb0 2023-10-16T14:35:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.461+0000 7f04ccdec640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 0x7f04c80b04a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f04c000cc30 tx=0x7f04c000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.461+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000ce40 con 0x7f04c80ccfb0 2023-10-16T14:35:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.461+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c0003990 con 0x7f04c80ccfb0 2023-10-16T14:35:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.461+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c0003d20 con 0x7f04c80ccfb0 2023-10-16T14:35:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.461+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c80b3910 con 0x7f04c80ccfb0 2023-10-16T14:35:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.461+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c80ae480 con 0x7f04c80ccfb0 2023-10-16T14:35:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.462+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c806b330 con 0x7f04c80ccfb0 2023-10-16T14:35:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.463+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f04c0003e80 con 0x7f04c80ccfb0 2023-10-16T14:35:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.464+0000 7f04c57fa640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f04a806af90 0x7f04a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.464+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f04c0089990 con 0x7f04c80ccfb0 2023-10-16T14:35:24.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.465+0000 7f04c7fff640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f04a806af90 0x7f04a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.466+0000 7f04c7fff640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f04a806af90 0x7f04a806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04b8005880 tx=0x7f04b8009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.468+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f04c0054b60 con 0x7f04c80ccfb0 2023-10-16T14:35:24.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.657+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f04c80ae6a0 con 0x7f04c80ccfb0 2023-10-16T14:35:24.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.658+0000 7f04c57fa640 1 -- 172.21.15.29:0/1329960253 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) v1 ==== 84+0+65 (secure 0 0 0) 0x7f04c001d070 con 0x7f04c80ccfb0 2023-10-16T14:35:24.662 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:24.662 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:35:24.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:24.663 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:35:24.663 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:35:24.663 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:24.663 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:35:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.660+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f04a806af90 msgr2=0x7f04a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.660+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f04a806af90 0x7f04a806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04b8005880 tx=0x7f04b8009040 comp rx=0 tx=0).stop 2023-10-16T14:35:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.660+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 msgr2=0x7f04c80b04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.660+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 0x7f04c80b04a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f04c000cc30 tx=0x7f04c000a5c0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 shutdown_connections 2023-10-16T14:35:24.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f04a806af90 0x7f04a806d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04c80ccfb0 0x7f04c80b04a0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04c80ad970 0x7f04c80b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 --2- 172.21.15.29:0/1329960253 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04c806c040 0x7f04c81efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 >> 172.21.15.29:0/1329960253 conn(0x7f04c80ce6a0 msgr2=0x7f04c815c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 shutdown_connections 2023-10-16T14:35:24.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.661+0000 7f04cd5ed640 1 -- 172.21.15.29:0/1329960253 wait complete. 2023-10-16T14:35:24.684 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:35:24.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:35:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.901+0000 7f656bfff640 1 Processor -- start 2023-10-16T14:35:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.901+0000 7f656bfff640 1 -- start start 2023-10-16T14:35:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f656c15ad80 0x7f656c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f656c0ca310 0x7f656c06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0c8e70 con 0x7f656c0cae40 2023-10-16T14:35:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c0b4170 con 0x7f656c15ad80 2023-10-16T14:35:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c0adde0 con 0x7f656c0ca310 2023-10-16T14:35:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f656c15ad80 0x7f656c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f656c15ad80 0x7f656c06cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51912/0 (socket says 172.21.15.29:51912) 2023-10-16T14:35:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.903+0000 7f656affd640 1 -- 172.21.15.29:0/3064663531 learned_addr learned my addr 172.21.15.29:0/3064663531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:24.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.904+0000 7f656b7fe640 1 --2- 172.21.15.29:0/3064663531 >> v2:172.21.15.170:3300/0 conn(0x7f656c0ca310 0x7f656c06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.904+0000 7f656affd640 1 -- 172.21.15.29:0/3064663531 >> v2:172.21.15.170:3300/0 conn(0x7f656c0ca310 msgr2=0x7f656c06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.904+0000 7f656affd640 1 --2- 172.21.15.29:0/3064663531 >> v2:172.21.15.170:3300/0 conn(0x7f656c0ca310 0x7f656c06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.904+0000 7f656affd640 1 -- 172.21.15.29:0/3064663531 >> v1:172.21.15.170:6789/0 conn(0x7f656c0cae40 legacy=0x7f656c06b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.904+0000 7f656affd640 1 -- 172.21.15.29:0/3064663531 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c0b4410 con 0x7f656c15ad80 2023-10-16T14:35:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.904+0000 7f656b7fe640 1 --2- 172.21.15.29:0/3064663531 >> v2:172.21.15.170:3300/0 conn(0x7f656c0ca310 0x7f656c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.905+0000 7f656affd640 1 --2- 172.21.15.29:0/3064663531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c15ad80 0x7f656c06cb70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f656c06c5f0 tx=0x7f655c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf3f72635fe83fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.905+0000 7f6569ffb640 1 -- 172.21.15.29:0/3064663531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c031870 con 0x7f656c15ad80 2023-10-16T14:35:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.905+0000 7f6569ffb640 1 -- 172.21.15.29:0/3064663531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c0319d0 con 0x7f656c15ad80 2023-10-16T14:35:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.905+0000 7f6569ffb640 1 -- 172.21.15.29:0/3064663531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c031d60 con 0x7f656c15ad80 2023-10-16T14:35:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 -- 172.21.15.29:0/3064663531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c15ad80 msgr2=0x7f656c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 --2- 172.21.15.29:0/3064663531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c15ad80 0x7f656c06cb70 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f656c06c5f0 tx=0x7f655c02b910 comp rx=0 tx=0).stop 2023-10-16T14:35:24.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 -- 172.21.15.29:0/3064663531 shutdown_connections 2023-10-16T14:35:24.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 --2- 172.21.15.29:0/3064663531 >> v2:172.21.15.170:3300/0 conn(0x7f656c0ca310 0x7f656c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 --2- 172.21.15.29:0/3064663531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c15ad80 0x7f656c06cb70 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 -- 172.21.15.29:0/3064663531 >> 172.21.15.29:0/3064663531 conn(0x7f656c0ce820 msgr2=0x7f656c0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:24.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 -- 172.21.15.29:0/3064663531 shutdown_connections 2023-10-16T14:35:24.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.906+0000 7f656bfff640 1 -- 172.21.15.29:0/3064663531 wait complete. 2023-10-16T14:35:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.909+0000 7f656bfff640 1 Processor -- start 2023-10-16T14:35:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.909+0000 7f656bfff640 1 -- start start 2023-10-16T14:35:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f656c0ca310 0x7f656c15d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 0x7f656c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f656c15ad80 0x7f656c15dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c15c540 con 0x7f656c0cae40 2023-10-16T14:35:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c15e250 con 0x7f656c15ad80 2023-10-16T14:35:24.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c15d090 con 0x7f656c0ca310 2023-10-16T14:35:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.910+0000 7f656a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 0x7f656c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 0x7f656c15d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51920/0 (socket says 172.21.15.29:51920) 2023-10-16T14:35:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 -- 172.21.15.29:0/3207926622 learned_addr learned my addr 172.21.15.29:0/3207926622 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:24.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656affd640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f656c0ca310 0x7f656c15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656b7fe640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f656c15ad80 0x7f656c15dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 -- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f656c0ca310 msgr2=0x7f656c15d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f656c0ca310 0x7f656c15d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 -- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f656c15ad80 msgr2=0x7f656c15dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f656c15ad80 0x7f656c15dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 -- 172.21.15.29:0/3207926622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c0094d0 con 0x7f656c0cae40 2023-10-16T14:35:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.911+0000 7f656a7fc640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 0x7f656c15d7d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f656000c400 tx=0x7f656000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.912+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65600035f0 con 0x7f656c0cae40 2023-10-16T14:35:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.912+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6560003750 con 0x7f656c0cae40 2023-10-16T14:35:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.912+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560003b40 con 0x7f656c0cae40 2023-10-16T14:35:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.912+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c15bca0 con 0x7f656c0cae40 2023-10-16T14:35:24.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.912+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c15c260 con 0x7f656c0cae40 2023-10-16T14:35:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.914+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c04e9b0 con 0x7f656c0cae40 2023-10-16T14:35:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.914+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6560003ca0 con 0x7f656c0cae40 2023-10-16T14:35:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.915+0000 7f65537fe640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f655806afe0 0x7f655806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.915+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f65600888b0 con 0x7f656c0cae40 2023-10-16T14:35:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.916+0000 7f656affd640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f655806afe0 0x7f655806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.917+0000 7f656affd640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f655806afe0 0x7f655806d4a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f655c036000 tx=0x7f655c03c040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:24.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:24.919+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6560053dc0 con 0x7f656c0cae40 2023-10-16T14:35:25.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.105+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f656c06ca50 con 0x7f656c0cae40 2023-10-16T14:35:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.751+0000 7f65537fe640 1 -- 172.21.15.29:0/3207926622 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v124) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6560017020 con 0x7f656c0cae40 2023-10-16T14:35:25.755 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:35:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f655806afe0 msgr2=0x7f655806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f655806afe0 0x7f655806d4a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f655c036000 tx=0x7f655c03c040 comp rx=0 tx=0).stop 2023-10-16T14:35:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 msgr2=0x7f656c15d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 0x7f656c15d7d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f656000c400 tx=0x7f656000c8c0 comp rx=0 tx=0).stop 2023-10-16T14:35:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 shutdown_connections 2023-10-16T14:35:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f655806afe0 0x7f655806d4a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f656c0cae40 0x7f656c15d7d0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f656c15ad80 0x7f656c15dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.756+0000 7f656bfff640 1 --2- 172.21.15.29:0/3207926622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f656c0ca310 0x7f656c15d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.757+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 >> 172.21.15.29:0/3207926622 conn(0x7f656c0ce820 msgr2=0x7f656c0c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.757+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 shutdown_connections 2023-10-16T14:35:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.757+0000 7f656bfff640 1 -- 172.21.15.29:0/3207926622 wait complete. 2023-10-16T14:35:25.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:35:25.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.995+0000 7f96a6d73640 1 Processor -- start 2023-10-16T14:35:25.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.995+0000 7f96a6d73640 1 -- start start 2023-10-16T14:35:26.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.996+0000 7f96a6d73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f96a00b4160 0x7f96a00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.996+0000 7f96a6d73640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00cc850 con 0x7f96a015ac70 2023-10-16T14:35:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a6d73640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00ccb30 con 0x7f96a006a4b0 2023-10-16T14:35:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a6d73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a00cce10 con 0x7f96a00b4160 2023-10-16T14:35:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a5d71640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f96a015ac70 0x7f96a0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32882/0 (socket says 172.21.15.29:32882) 2023-10-16T14:35:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a5d71640 1 -- 172.21.15.29:0/2755633147 learned_addr learned my addr 172.21.15.29:0/2755633147 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:26.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a6572640 1 --2- 172.21.15.29:0/2755633147 >> v2:172.21.15.29:3300/0 conn(0x7f96a00b4160 0x7f96a00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a4d6f640 1 -- 172.21.15.29:0/2755633147 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467495796 0 0) 0x7f96a00cc850 con 0x7f96a015ac70 2023-10-16T14:35:26.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.997+0000 7f96a4d6f640 1 -- 172.21.15.29:0/2755633147 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f968c003520 con 0x7f96a015ac70 2023-10-16T14:35:26.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.998+0000 7f96a6572640 1 -- 172.21.15.29:0/2755633147 >> v1:172.21.15.170:6789/0 conn(0x7f96a006a4b0 legacy=0x7f96a006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.998+0000 7f96a6572640 1 -- 172.21.15.29:0/2755633147 >> v1:172.21.15.29:6789/0 conn(0x7f96a015ac70 legacy=0x7f96a0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.998+0000 7f96a6572640 1 -- 172.21.15.29:0/2755633147 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a00cd0f0 con 0x7f96a00b4160 2023-10-16T14:35:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.999+0000 7f96a6572640 1 --2- 172.21.15.29:0/2755633147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a00cc270 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f9698009d90 tx=0x7f969802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aae7c630016faa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.999+0000 7f96a4d6f640 1 -- 172.21.15.29:0/2755633147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969802cd00 con 0x7f96a00b4160 2023-10-16T14:35:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.999+0000 7f96a4d6f640 1 -- 172.21.15.29:0/2755633147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969802ce60 con 0x7f96a00b4160 2023-10-16T14:35:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:25.999+0000 7f96a4d6f640 1 -- 172.21.15.29:0/2755633147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9698034660 con 0x7f96a00b4160 2023-10-16T14:35:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 -- 172.21.15.29:0/2755633147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 msgr2=0x7f96a00cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 --2- 172.21.15.29:0/2755633147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a00cc270 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f9698009d90 tx=0x7f969802bf50 comp rx=0 tx=0).stop 2023-10-16T14:35:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 -- 172.21.15.29:0/2755633147 shutdown_connections 2023-10-16T14:35:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 --2- 172.21.15.29:0/2755633147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a00cc270 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 -- 172.21.15.29:0/2755633147 >> 172.21.15.29:0/2755633147 conn(0x7f96a00ce6a0 msgr2=0x7f96a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 -- 172.21.15.29:0/2755633147 shutdown_connections 2023-10-16T14:35:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.000+0000 7f96a6d73640 1 -- 172.21.15.29:0/2755633147 wait complete. 2023-10-16T14:35:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.002+0000 7f96a6d73640 1 Processor -- start 2023-10-16T14:35:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.002+0000 7f96a6d73640 1 -- start start 2023-10-16T14:35:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a6d73640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96a006a4b0 0x7f96a00c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a6d73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a015d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a6d73640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a015ac70 0x7f96a0070180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a6d73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00c99f0 con 0x7f96a00b4160 2023-10-16T14:35:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a6d73640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00c24e0 con 0x7f96a015ac70 2023-10-16T14:35:26.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a6d73640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00c27c0 con 0x7f96a006a4b0 2023-10-16T14:35:26.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a5570640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a015d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a5570640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a015d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51940/0 (socket says 172.21.15.29:51940) 2023-10-16T14:35:26.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.003+0000 7f96a5570640 1 -- 172.21.15.29:0/765827951 learned_addr learned my addr 172.21.15.29:0/765827951 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:26.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a6572640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a015ac70 0x7f96a0070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5d71640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96a006a4b0 0x7f96a00c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5570640 1 -- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96a006a4b0 msgr2=0x7f96a00c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5570640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96a006a4b0 0x7f96a00c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5570640 1 -- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a015ac70 msgr2=0x7f96a0070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5570640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a015ac70 0x7f96a0070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5570640 1 -- 172.21.15.29:0/765827951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698009a70 con 0x7f96a00b4160 2023-10-16T14:35:26.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a6572640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a015ac70 0x7f96a0070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:26.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96a5570640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a015d6a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f969000c450 tx=0x7f969000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:26.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.004+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96900052d0 con 0x7f96a00b4160 2023-10-16T14:35:26.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.005+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9690005430 con 0x7f96a00b4160 2023-10-16T14:35:26.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.005+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690012870 con 0x7f96a00b4160 2023-10-16T14:35:26.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.005+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a00c2aa0 con 0x7f96a00b4160 2023-10-16T14:35:26.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.005+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a00c5770 con 0x7f96a00b4160 2023-10-16T14:35:26.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.006+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a00c5990 con 0x7f96a00b4160 2023-10-16T14:35:26.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.008+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f96900129d0 con 0x7f96a00b4160 2023-10-16T14:35:26.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.009+0000 7f96967fc640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f968406af90 0x7f968406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.009+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9690089ba0 con 0x7f96a00b4160 2023-10-16T14:35:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.010+0000 7f96a5d71640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f968406af90 0x7f968406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.011+0000 7f96a5d71640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f968406af90 0x7f968406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f969c0011e0 tx=0x7f969c009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:26.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.014+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9690054d90 con 0x7f96a00b4160 2023-10-16T14:35:26.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.201+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f96a004e9b0 con 0x7f96a00b4160 2023-10-16T14:35:26.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.758+0000 7f96967fc640 1 -- 172.21.15.29:0/765827951 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v125) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9690019300 con 0x7f96a00b4160 2023-10-16T14:35:26.762 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:35:26.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f968406af90 msgr2=0x7f968406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f968406af90 0x7f968406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f969c0011e0 tx=0x7f969c009040 comp rx=0 tx=0).stop 2023-10-16T14:35:26.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 msgr2=0x7f96a015d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:26.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a015d6a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f969000c450 tx=0x7f969000c910 comp rx=0 tx=0).stop 2023-10-16T14:35:26.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 shutdown_connections 2023-10-16T14:35:26.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f968406af90 0x7f968406d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96a00b4160 0x7f96a015d6a0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96a015ac70 0x7f96a0070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 --2- 172.21.15.29:0/765827951 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96a006a4b0 0x7f96a00c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 >> 172.21.15.29:0/765827951 conn(0x7f96a00ce6a0 msgr2=0x7f96a015eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.762+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 shutdown_connections 2023-10-16T14:35:26.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.763+0000 7f96a6d73640 1 -- 172.21.15.29:0/765827951 wait complete. 2023-10-16T14:35:26.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:35:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.988+0000 7fbb44d90640 1 Processor -- start 2023-10-16T14:35:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.988+0000 7fbb44d90640 1 -- start start 2023-10-16T14:35:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb44d90640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbb4015d150 0x7fbb400ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb44d90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 0x7fbb40160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb44d90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb400ac890 con 0x7fbb40160b90 2023-10-16T14:35:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb44d90640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4015d530 con 0x7fbb4015d150 2023-10-16T14:35:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb44d90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40161650 con 0x7fbb4006a4b0 2023-10-16T14:35:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb35d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbb40160b90 0x7fbb40069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32888/0 (socket says 172.21.15.29:32888) 2023-10-16T14:35:26.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.990+0000 7fbb35d9b640 1 -- 172.21.15.29:0/2032304231 learned_addr learned my addr 172.21.15.29:0/2032304231 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3ed9d640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 0x7fbb40160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2032304231 >> v2:172.21.15.170:3300/0 conn(0x7fbb4015d150 0x7fbb400ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168974935 0 0) 0x7fbb400ac890 con 0x7fbb40160b90 2023-10-16T14:35:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c003520 con 0x7fbb40160b90 2023-10-16T14:35:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765537889 0 0) 0x7fbb2c003520 con 0x7fbb40160b90 2023-10-16T14:35:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 msgr2=0x7fbb40160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 0x7fbb40160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 >> v2:172.21.15.170:3300/0 conn(0x7fbb4015d150 msgr2=0x7fbb400ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 --2- 172.21.15.29:0/2032304231 >> v2:172.21.15.170:3300/0 conn(0x7fbb4015d150 0x7fbb400ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40161930 con 0x7fbb40160b90 2023-10-16T14:35:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbb28001670 con 0x7fbb40160b90 2023-10-16T14:35:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 >> v1:172.21.15.29:6789/0 conn(0x7fbb40160b90 legacy=0x7fbb40069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.991+0000 7fbb3dd9b640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb2c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3dd9b640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 0x7fbb2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3dd9b640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb2c004d00 0x7fbb2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4015d530 con 0x7fbb2c003970 2023-10-16T14:35:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40161650 con 0x7fbb2c004d00 2023-10-16T14:35:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c00b9b0 con 0x7fbb4006a4b0 2023-10-16T14:35:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb2c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb35d9b640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 0x7fbb2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3ed9d640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb2c004d00 0x7fbb2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 msgr2=0x7fbb2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:27.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb4006a4b0 0x7fbb2c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb2c004d00 msgr2=0x7fbb2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:27.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb2c004d00 0x7fbb2c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2032304231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c003520 con 0x7fbb2c003970 2023-10-16T14:35:27.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.992+0000 7fbb3ed9d640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb2c004d00 0x7fbb2c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:27.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.994+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb2c003d90 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009ac0 tx=0x7fbb2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c746bb92cc718cbe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:27.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.994+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3002c710 con 0x7fbb2c003970 2023-10-16T14:35:27.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.994+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3002c870 con 0x7fbb2c003970 2023-10-16T14:35:27.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.994+0000 7fbb3dd9b640 1 -- 172.21.15.29:0/2032304231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3002cc00 con 0x7fbb2c003970 2023-10-16T14:35:27.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.994+0000 7fbb44d90640 1 -- 172.21.15.29:0/2032304231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 msgr2=0x7fbb2c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:27.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.994+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2032304231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb2c003d90 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009ac0 tx=0x7fbb2c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.995+0000 7fbb3ed9d640 1 -- 172.21.15.29:0/2032304231 reap_dead start 2023-10-16T14:35:27.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.995+0000 7fbb44d90640 1 -- 172.21.15.29:0/2032304231 shutdown_connections 2023-10-16T14:35:27.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.995+0000 7fbb44d90640 1 -- 172.21.15.29:0/2032304231 >> 172.21.15.29:0/2032304231 conn(0x7fbb400ce9b0 msgr2=0x7fbb400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:27.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.995+0000 7fbb44d90640 1 -- 172.21.15.29:0/2032304231 shutdown_connections 2023-10-16T14:35:27.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.995+0000 7fbb44d90640 1 -- 172.21.15.29:0/2032304231 wait complete. 2023-10-16T14:35:27.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.997+0000 7fbb44d90640 1 Processor -- start 2023-10-16T14:35:27.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.997+0000 7fbb44d90640 1 -- start start 2023-10-16T14:35:27.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb44d90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb40143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb44d90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb400c8630 0x7fbb4015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb44d90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb40070ad0 0x7fbb40070f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb44d90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40071490 con 0x7fbb2c003970 2023-10-16T14:35:27.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb44d90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4015ebd0 con 0x7fbb400c8630 2023-10-16T14:35:27.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb44d90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4015dfc0 con 0x7fbb40070ad0 2023-10-16T14:35:27.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb40143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb40143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51952/0 (socket says 172.21.15.29:51952) 2023-10-16T14:35:27.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2732010826 learned_addr learned my addr 172.21.15.29:0/2732010826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:27.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.998+0000 7fbb35d9b640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb400c8630 0x7fbb4015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3ed9d640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb40070ad0 0x7fbb40070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb40070ad0 msgr2=0x7fbb40070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:27.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb40070ad0 0x7fbb40070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb400c8630 msgr2=0x7fbb4015d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:27.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb400c8630 0x7fbb4015d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3e59c640 1 -- 172.21.15.29:0/2732010826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4015e2a0 con 0x7fbb2c003970 2023-10-16T14:35:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb3e59c640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb40143ec0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fbb30037040 tx=0x7fbb30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30003810 con 0x7fbb2c003970 2023-10-16T14:35:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3002de90 con 0x7fbb2c003970 2023-10-16T14:35:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30003b70 con 0x7fbb2c003970 2023-10-16T14:35:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:26.999+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4015e580 con 0x7fbb2c003970 2023-10-16T14:35:27.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.000+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4015dbb0 con 0x7fbb2c003970 2023-10-16T14:35:27.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.001+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4004e160 con 0x7fbb2c003970 2023-10-16T14:35:27.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.002+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbb30003cd0 con 0x7fbb2c003970 2023-10-16T14:35:27.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.003+0000 7fbb377fe640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbb2006afc0 0x7fbb2006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:27.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.003+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fbb300abae0 con 0x7fbb2c003970 2023-10-16T14:35:27.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.004+0000 7fbb35d9b640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbb2006afc0 0x7fbb2006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:27.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.005+0000 7fbb35d9b640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbb2006afc0 0x7fbb2006d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb400c5950 tx=0x7fbb28009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:27.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.007+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbb30076b70 con 0x7fbb2c003970 2023-10-16T14:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.244+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbb400ade10 con 0x7fbb2c003970 2023-10-16T14:35:27.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.804+0000 7fbb377fe640 1 -- 172.21.15.29:0/2732010826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v54) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbb300390c0 con 0x7fbb2c003970 2023-10-16T14:35:27.808 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 14 and data pool 15 2023-10-16T14:35:27.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.813+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbb2006afc0 msgr2=0x7fbb2006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.813+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbb2006afc0 0x7fbb2006d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb400c5950 tx=0x7fbb28009040 comp rx=0 tx=0).stop 2023-10-16T14:35:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.813+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 msgr2=0x7fbb40143ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.813+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb40143ec0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fbb30037040 tx=0x7fbb30004730 comp rx=0 tx=0).stop 2023-10-16T14:35:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.817+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 shutdown_connections 2023-10-16T14:35:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.817+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbb2006afc0 0x7fbb2006d480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.817+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbb40070ad0 0x7fbb40070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.817+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbb400c8630 0x7fbb4015d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.817+0000 7fbb44d90640 1 --2- 172.21.15.29:0/2732010826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb2c003970 0x7fbb40143ec0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:27.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.817+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 >> 172.21.15.29:0/2732010826 conn(0x7fbb400ce9b0 msgr2=0x7fbb4015afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.819+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 shutdown_connections 2023-10-16T14:35:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:27.819+0000 7fbb44d90640 1 -- 172.21.15.29:0/2732010826 wait complete. 2023-10-16T14:35:27.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.039+0000 7f8bc6296640 1 Processor -- start 2023-10-16T14:35:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.039+0000 7f8bc6296640 1 -- start start 2023-10-16T14:35:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.040+0000 7f8bc6296640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8bc0159f00 0x7f8bc00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bc6296640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc006f7f0 con 0x7f8bc0159b20 2023-10-16T14:35:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bc6296640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc006c5e0 con 0x7f8bc015d880 2023-10-16T14:35:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bc6296640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc006c8c0 con 0x7f8bc0159f00 2023-10-16T14:35:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bbffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8bc0159b20 0x7f8bc006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32900/0 (socket says 172.21.15.29:32900) 2023-10-16T14:35:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bbffff640 1 -- 172.21.15.29:0/1040094750 learned_addr learned my addr 172.21.15.29:0/1040094750 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607050364 0 0) 0x7f8bc006f7f0 con 0x7f8bc0159b20 2023-10-16T14:35:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba8005730 con 0x7f8bc0159b20 2023-10-16T14:35:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.041+0000 7f8bbf7fe640 1 --2- 172.21.15.29:0/1040094750 >> v2:172.21.15.170:3300/0 conn(0x7f8bc0159f00 0x7f8bc00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423249688 0 0) 0x7f8ba8005730 con 0x7f8bc0159b20 2023-10-16T14:35:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 >> v2:172.21.15.170:3300/0 conn(0x7f8bc0159f00 msgr2=0x7f8bc00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 --2- 172.21.15.29:0/1040094750 >> v2:172.21.15.170:3300/0 conn(0x7f8bc0159f00 0x7f8bc00b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 >> v1:172.21.15.170:6789/0 conn(0x7f8bc015d880 legacy=0x7f8bc006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc006cba0 con 0x7f8bc0159b20 2023-10-16T14:35:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8bb0001670 con 0x7f8bc0159b20 2023-10-16T14:35:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 >> v1:172.21.15.29:6789/0 conn(0x7f8bc0159b20 legacy=0x7f8bc006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ba8005bd0 0x7f8ba8005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ba80069d0 0x7f8ba8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8ba800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc006c8c0 con 0x7f8ba80073e0 2023-10-16T14:35:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba800e310 con 0x7f8ba8005bd0 2023-10-16T14:35:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba800e4d0 con 0x7f8ba80069d0 2023-10-16T14:35:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.042+0000 7f8bc4833640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8ba800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bbf7fe640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ba80069d0 0x7f8ba8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bbffff640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ba8005bd0 0x7f8ba8005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bc4833640 1 -- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ba80069d0 msgr2=0x7f8ba8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bc4833640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ba80069d0 0x7f8ba8006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bc4833640 1 -- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ba8005bd0 msgr2=0x7f8ba8005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bc4833640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ba8005bd0 0x7f8ba8005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bc4833640 1 -- 172.21.15.29:0/1040094750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba8005730 con 0x7f8ba80073e0 2023-10-16T14:35:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.043+0000 7f8bbffff640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ba8005bd0 0x7f8ba8005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.044+0000 7f8bc4833640 1 -- 172.21.15.29:0/1040094750 reap_dead start 2023-10-16T14:35:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.044+0000 7f8bc4833640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8ba800db90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8004480 tx=0x7f8ba8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73570a4ce1ace68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.044+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8030400 con 0x7f8ba80073e0 2023-10-16T14:35:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.044+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb8030560 con 0x7f8ba80073e0 2023-10-16T14:35:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.044+0000 7f8bbeffd640 1 -- 172.21.15.29:0/1040094750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80308f0 con 0x7f8ba80073e0 2023-10-16T14:35:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 -- 172.21.15.29:0/1040094750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 msgr2=0x7f8ba800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8ba800db90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8004480 tx=0x7f8ba8007a00 comp rx=0 tx=0).stop 2023-10-16T14:35:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 -- 172.21.15.29:0/1040094750 shutdown_connections 2023-10-16T14:35:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1040094750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8ba800db90 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 -- 172.21.15.29:0/1040094750 >> 172.21.15.29:0/1040094750 conn(0x7f8bc00cb410 msgr2=0x7f8bc00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 -- 172.21.15.29:0/1040094750 shutdown_connections 2023-10-16T14:35:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.045+0000 7f8bc6296640 1 -- 172.21.15.29:0/1040094750 wait complete. 2023-10-16T14:35:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.047+0000 7f8bc6296640 1 Processor -- start 2023-10-16T14:35:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.047+0000 7f8bc6296640 1 -- start start 2023-10-16T14:35:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bc6296640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8bc0148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bc6296640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bc0156a10 0x7f8bc0156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bc6296640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bc015ba80 0x7f8bc015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bc6296640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc01573d0 con 0x7f8ba80073e0 2023-10-16T14:35:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bc6296640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc01f6b30 con 0x7f8bc0156a10 2023-10-16T14:35:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bc6296640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc01f6220 con 0x7f8bc015ba80 2023-10-16T14:35:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8bc0148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8bc0148ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51970/0 (socket says 172.21.15.29:51970) 2023-10-16T14:35:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bbffff640 1 -- 172.21.15.29:0/1597644484 learned_addr learned my addr 172.21.15.29:0/1597644484 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bbffff640 1 -- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bc015ba80 msgr2=0x7f8bc015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.048+0000 7f8bbffff640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bc015ba80 0x7f8bc015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbffff640 1 -- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bc0156a10 msgr2=0x7f8bc0156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbffff640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bc0156a10 0x7f8bc0156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbffff640 1 -- 172.21.15.29:0/1597644484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc015c7d0 con 0x7f8ba80073e0 2023-10-16T14:35:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbffff640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8bc0148ca0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000d7a0 tx=0x7f8bb000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb00074e0 con 0x7f8ba80073e0 2023-10-16T14:35:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb0007640 con 0x7f8ba80073e0 2023-10-16T14:35:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb00049e0 con 0x7f8ba80073e0 2023-10-16T14:35:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc01f64e0 con 0x7f8ba80073e0 2023-10-16T14:35:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.049+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc015c440 con 0x7f8ba80073e0 2023-10-16T14:35:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.050+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc004e160 con 0x7f8ba80073e0 2023-10-16T14:35:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.052+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8bb0004b40 con 0x7f8ba80073e0 2023-10-16T14:35:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.053+0000 7f8bbcff9640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f8bac06af90 0x7f8bac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.053+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bb00891c0 con 0x7f8ba80073e0 2023-10-16T14:35:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.054+0000 7f8bbf7fe640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f8bac06af90 0x7f8bac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.055+0000 7f8bbf7fe640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f8bac06af90 0x7f8bac06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006a90 tx=0x7f8bb4006a20 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.058+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8bb0053fb0 con 0x7f8ba80073e0 2023-10-16T14:35:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.251+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bc00b10b0 con 0x7f8ba80073e0 2023-10-16T14:35:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.254+0000 7f8bbcff9640 1 -- 172.21.15.29:0/1597644484 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f8bb00575d0 con 0x7f8ba80073e0 2023-10-16T14:35:28.258 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:28.258 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":126,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:27.246728+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f8bac06af90 msgr2=0x7f8bac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f8bac06af90 0x7f8bac06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006a90 tx=0x7f8bb4006a20 comp rx=0 tx=0).stop 2023-10-16T14:35:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 msgr2=0x7f8bc0148ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8bc0148ca0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000d7a0 tx=0x7f8bb000dc60 comp rx=0 tx=0).stop 2023-10-16T14:35:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 shutdown_connections 2023-10-16T14:35:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f8bac06af90 0x7f8bac06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8bc015ba80 0x7f8bc015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8bc0156a10 0x7f8bc0156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 --2- 172.21.15.29:0/1597644484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ba80073e0 0x7f8bc0148ca0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.256+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 >> 172.21.15.29:0/1597644484 conn(0x7f8bc00cb410 msgr2=0x7f8bc00ae2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.257+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 shutdown_connections 2023-10-16T14:35:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.257+0000 7f8bc6296640 1 -- 172.21.15.29:0/1597644484 wait complete. 2023-10-16T14:35:28.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:28.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.498+0000 7f3f5886e640 1 Processor -- start 2023-10-16T14:35:28.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.498+0000 7f3f5886e640 1 -- start start 2023-10-16T14:35:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.500+0000 7f3f5886e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3f540ccb00 0x7f3f540ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.500+0000 7f3f5886e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f540cd4a0 con 0x7f3f540ad4d0 2023-10-16T14:35:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.500+0000 7f3f5886e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5415ac60 con 0x7f3f54153740 2023-10-16T14:35:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.500+0000 7f3f5886e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f54070030 con 0x7f3f540ccb00 2023-10-16T14:35:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.500+0000 7f3f5359e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3f540ad4d0 0x7f3f540ad960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32908/0 (socket says 172.21.15.29:32908) 2023-10-16T14:35:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.500+0000 7f3f5359e640 1 -- 172.21.15.29:0/468310684 learned_addr learned my addr 172.21.15.29:0/468310684 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/468310684 >> v2:172.21.15.170:3300/0 conn(0x7f3f540ccb00 0x7f3f540ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f5259c640 1 -- 172.21.15.29:0/468310684 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332831351 0 0) 0x7f3f540cd4a0 con 0x7f3f540ad4d0 2023-10-16T14:35:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f5259c640 1 -- 172.21.15.29:0/468310684 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f40004630 con 0x7f3f540ad4d0 2023-10-16T14:35:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f4bfff640 1 -- 172.21.15.29:0/468310684 >> v1:172.21.15.170:6789/0 conn(0x7f3f54153740 legacy=0x7f3f5415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f4bfff640 1 -- 172.21.15.29:0/468310684 >> v1:172.21.15.29:6789/0 conn(0x7f3f540ad4d0 legacy=0x7f3f540ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f5259c640 1 -- 172.21.15.29:0/468310684 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799201142 0 0) 0x7f3f40004630 con 0x7f3f540ad4d0 2023-10-16T14:35:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.501+0000 7f3f4bfff640 1 -- 172.21.15.29:0/468310684 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f54070310 con 0x7f3f540ccb00 2023-10-16T14:35:28.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.503+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/468310684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f540ccf60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f936e52d23601c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5259c640 1 -- 172.21.15.29:0/468310684 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c0019f0 con 0x7f3f540ccb00 2023-10-16T14:35:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5259c640 1 -- 172.21.15.29:0/468310684 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c02e050 con 0x7f3f540ccb00 2023-10-16T14:35:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5259c640 1 -- 172.21.15.29:0/468310684 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c030730 con 0x7f3f540ccb00 2023-10-16T14:35:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5886e640 1 -- 172.21.15.29:0/468310684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 msgr2=0x7f3f540ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5886e640 1 --2- 172.21.15.29:0/468310684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f540ccf60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c02b790 comp rx=0 tx=0).stop 2023-10-16T14:35:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5886e640 1 -- 172.21.15.29:0/468310684 shutdown_connections 2023-10-16T14:35:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5886e640 1 --2- 172.21.15.29:0/468310684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f540ccf60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.504+0000 7f3f5886e640 1 -- 172.21.15.29:0/468310684 >> 172.21.15.29:0/468310684 conn(0x7f3f540ce6a0 msgr2=0x7f3f540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.505+0000 7f3f5886e640 1 -- 172.21.15.29:0/468310684 shutdown_connections 2023-10-16T14:35:28.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.505+0000 7f3f5886e640 1 -- 172.21.15.29:0/468310684 wait complete. 2023-10-16T14:35:28.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.506+0000 7f3f5886e640 1 Processor -- start 2023-10-16T14:35:28.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.506+0000 7f3f5886e640 1 -- start start 2023-10-16T14:35:28.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.507+0000 7f3f5886e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 0x7f3f540b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.507+0000 7f3f5886e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f54152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.507+0000 7f3f5886e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54153740 0x7f3f540af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.507+0000 7f3f5886e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f54153350 con 0x7f3f540ad4d0 2023-10-16T14:35:28.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.507+0000 7f3f5886e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f541e91e0 con 0x7f3f540ccb00 2023-10-16T14:35:28.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.507+0000 7f3f5886e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540b19c0 con 0x7f3f54153740 2023-10-16T14:35:28.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 0x7f3f540b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 0x7f3f540b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51974/0 (socket says 172.21.15.29:51974) 2023-10-16T14:35:28.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 -- 172.21.15.29:0/3779540615 learned_addr learned my addr 172.21.15.29:0/3779540615 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:28.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f54152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f5359e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54153740 0x7f3f540af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 -- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54153740 msgr2=0x7f3f540af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54153740 0x7f3f540af240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 -- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 msgr2=0x7f3f54152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f54152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.508+0000 7f3f52d9d640 1 -- 172.21.15.29:0/3779540615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c009400 con 0x7f3f540ad4d0 2023-10-16T14:35:28.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.509+0000 7f3f52d9d640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 0x7f3f540b12c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400c430 tx=0x7f3f4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.509+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440052d0 con 0x7f3f540ad4d0 2023-10-16T14:35:28.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.509+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44005430 con 0x7f3f540ad4d0 2023-10-16T14:35:28.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.509+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44003830 con 0x7f3f540ad4d0 2023-10-16T14:35:28.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.509+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f540b1ca0 con 0x7f3f540ad4d0 2023-10-16T14:35:28.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.509+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f540b2260 con 0x7f3f540ad4d0 2023-10-16T14:35:28.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.510+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f541e6f80 con 0x7f3f540ad4d0 2023-10-16T14:35:28.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.511+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f44003990 con 0x7f3f540ad4d0 2023-10-16T14:35:28.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.512+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3f3406af90 0x7f3f3406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.512+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f44088c60 con 0x7f3f540ad4d0 2023-10-16T14:35:28.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.513+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3f3406af90 0x7f3f3406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.514+0000 7f3f4bfff640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3f3406af90 0x7f3f3406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034000 tx=0x7f3f3c02f000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.516+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3f541e6f80 con 0x7f3f540ad4d0 2023-10-16T14:35:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.711+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f541e6f80 con 0x7f3f540ad4d0 2023-10-16T14:35:28.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.713+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3779540615 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f3f541e6f80 con 0x7f3f540ad4d0 2023-10-16T14:35:28.717 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:28.718 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":126,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:27.246728+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.716+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3f3406af90 msgr2=0x7f3f3406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.716+0000 7f3f5886e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3f3406af90 0x7f3f3406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034000 tx=0x7f3f3c02f000 comp rx=0 tx=0).stop 2023-10-16T14:35:28.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.716+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 msgr2=0x7f3f540b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.716+0000 7f3f5886e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 0x7f3f540b12c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400c430 tx=0x7f3f4400c8f0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 shutdown_connections 2023-10-16T14:35:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3f3406af90 0x7f3f3406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3f54153740 0x7f3f540af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3f540ccb00 0x7f3f54152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 --2- 172.21.15.29:0/3779540615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f540ad4d0 0x7f3f540b12c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 >> 172.21.15.29:0/3779540615 conn(0x7f3f540ce6a0 msgr2=0x7f3f540ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 shutdown_connections 2023-10-16T14:35:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.717+0000 7f3f5886e640 1 -- 172.21.15.29:0/3779540615 wait complete. 2023-10-16T14:35:28.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:35:28.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.958+0000 7fe2a8b00640 1 Processor -- start 2023-10-16T14:35:28.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.959+0000 7fe2a8b00640 1 -- start start 2023-10-16T14:35:28.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.960+0000 7fe2a8b00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 0x7fe2a40cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.960+0000 7fe2a8b00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40cabe0 con 0x7fe2a40b4d50 2023-10-16T14:35:28.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.960+0000 7fe2a8b00640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40b4960 con 0x7fe2a40b4580 2023-10-16T14:35:28.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.960+0000 7fe2a8b00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a406ff20 con 0x7fe2a4071bc0 2023-10-16T14:35:28.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.961+0000 7fe299d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe2a40b4d50 0x7fe2a4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32914/0 (socket says 172.21.15.29:32914) 2023-10-16T14:35:28.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.961+0000 7fe299d9b640 1 -- 172.21.15.29:0/141540694 learned_addr learned my addr 172.21.15.29:0/141540694 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:28.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.961+0000 7fe2a2d9d640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 0x7fe2a40cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.961+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830636472 0 0) 0x7fe2a40cabe0 con 0x7fe2a40b4d50 2023-10-16T14:35:28.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.961+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe290003520 con 0x7fe2a40b4d50 2023-10-16T14:35:28.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2293157754 0 0) 0x7fe290003520 con 0x7fe2a40b4d50 2023-10-16T14:35:28.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 msgr2=0x7fe2a40cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 0x7fe2a40cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 >> v1:172.21.15.170:6789/0 conn(0x7fe2a40b4580 legacy=0x7fe2a40714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a40ca2b0 con 0x7fe2a40b4d50 2023-10-16T14:35:28.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe28c001670 con 0x7fe2a40b4d50 2023-10-16T14:35:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 >> v1:172.21.15.29:6789/0 conn(0x7fe2a40b4d50 legacy=0x7fe2a4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 0x7fe2900039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290003f20 0x7fe2900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe290004d00 0x7fe290005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a406ff20 con 0x7fe290003f20 2023-10-16T14:35:28.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29000ba60 con 0x7fe290004d00 2023-10-16T14:35:28.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29000bc20 con 0x7fe2a4071bc0 2023-10-16T14:35:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.962+0000 7fe299d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290003f20 0x7fe2900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe2a259c640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 0x7fe2900039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe2a2d9d640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe290004d00 0x7fe290005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe299d9b640 1 -- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 msgr2=0x7fe2900039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe299d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2a4071bc0 0x7fe2900039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe299d9b640 1 -- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe290004d00 msgr2=0x7fe290005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe299d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe290004d00 0x7fe290005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe299d9b640 1 -- 172.21.15.29:0/141540694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe290003520 con 0x7fe290003f20 2023-10-16T14:35:28.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.963+0000 7fe2a2d9d640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe290004d00 0x7fe290005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.964+0000 7fe299d9b640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290003f20 0x7fe2900047c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c190 tx=0x7fe2900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a385e656c9ba393 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.964+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c005ad0 con 0x7fe290003f20 2023-10-16T14:35:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.964+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28c005c30 con 0x7fe290003f20 2023-10-16T14:35:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.964+0000 7fe2a1d9b640 1 -- 172.21.15.29:0/141540694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c02caf0 con 0x7fe290003f20 2023-10-16T14:35:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a8b00640 1 -- 172.21.15.29:0/141540694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290003f20 msgr2=0x7fe2900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/141540694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290003f20 0x7fe2900047c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c190 tx=0x7fe2900057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a2d9d640 1 -- 172.21.15.29:0/141540694 reap_dead start 2023-10-16T14:35:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a8b00640 1 -- 172.21.15.29:0/141540694 shutdown_connections 2023-10-16T14:35:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a8b00640 1 -- 172.21.15.29:0/141540694 >> 172.21.15.29:0/141540694 conn(0x7fe2a406bc10 msgr2=0x7fe2a406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a8b00640 1 -- 172.21.15.29:0/141540694 shutdown_connections 2023-10-16T14:35:28.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.965+0000 7fe2a8b00640 1 -- 172.21.15.29:0/141540694 wait complete. 2023-10-16T14:35:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.967+0000 7fe2a8b00640 1 Processor -- start 2023-10-16T14:35:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.967+0000 7fe2a8b00640 1 -- start start 2023-10-16T14:35:28.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a8b00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe290003f20 0x7fe2a40c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a8b00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 0x7fe2a41efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a8b00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2a40c6280 0x7fe2a40c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a8b00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a414df60 con 0x7fe2a41ef940 2023-10-16T14:35:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a8b00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a41f0300 con 0x7fe2a40c6280 2023-10-16T14:35:28.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a8b00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40c6f90 con 0x7fe290003f20 2023-10-16T14:35:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe299d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 0x7fe2a41efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe299d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 0x7fe2a41efdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51998/0 (socket says 172.21.15.29:51998) 2023-10-16T14:35:28.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe299d9b640 1 -- 172.21.15.29:0/2014282300 learned_addr learned my addr 172.21.15.29:0/2014282300 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.968+0000 7fe2a259c640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe290003f20 0x7fe2a40c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe299d9b640 1 -- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe290003f20 msgr2=0x7fe2a40c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:28.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe299d9b640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe290003f20 0x7fe2a40c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe299d9b640 1 -- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2a40c6280 msgr2=0x7fe2a40c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe299d9b640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2a40c6280 0x7fe2a40c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe299d9b640 1 -- 172.21.15.29:0/2014282300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a40c7270 con 0x7fe2a41ef940 2023-10-16T14:35:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe299d9b640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 0x7fe2a41efdc0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c190 tx=0x7fe2a415b1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c03c040 con 0x7fe2a41ef940 2023-10-16T14:35:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28c02ccc0 con 0x7fe2a41ef940 2023-10-16T14:35:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c007290 con 0x7fe2a41ef940 2023-10-16T14:35:29.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a40c7550 con 0x7fe2a41ef940 2023-10-16T14:35:29.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.969+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a40c7b10 con 0x7fe2a41ef940 2023-10-16T14:35:29.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.970+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a40c7cd0 con 0x7fe2a41ef940 2023-10-16T14:35:29.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.971+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe28c032070 con 0x7fe2a41ef940 2023-10-16T14:35:29.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.972+0000 7fe29b7fe640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:29.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.973+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe28c0aa800 con 0x7fe2a41ef940 2023-10-16T14:35:29.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.973+0000 7fe2a259c640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:29.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.974+0000 7fe2a259c640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe294007fd0 tx=0x7fe294009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:29.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:28.977+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe28c075790 con 0x7fe2a41ef940 2023-10-16T14:35:29.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.209+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe2a404e160 con 0x7fe2a41ef940 2023-10-16T14:35:29.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.815+0000 7fe29b7fe640 1 -- 172.21.15.29:0/2014282300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v57) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe28c046020 con 0x7fe2a41ef940 2023-10-16T14:35:29.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 msgr2=0x7fe28406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:29.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe294007fd0 tx=0x7fe294009040 comp rx=0 tx=0).stop 2023-10-16T14:35:29.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 msgr2=0x7fe2a41efdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:29.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 0x7fe2a41efdc0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c190 tx=0x7fe2a415b1c0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 shutdown_connections 2023-10-16T14:35:29.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2a41ef940 0x7fe2a41efdc0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2a40c6280 0x7fe2a40c6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 --2- 172.21.15.29:0/2014282300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe290003f20 0x7fe2a40c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:29.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 >> 172.21.15.29:0/2014282300 conn(0x7fe2a406bc10 msgr2=0x7fe2a415dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:29.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 shutdown_connections 2023-10-16T14:35:29.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:29.819+0000 7fe2a8b00640 1 -- 172.21.15.29:0/2014282300 wait complete. 2023-10-16T14:35:29.846 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:35:29.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:35:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.075+0000 7f9f6ccc2640 1 Processor -- start 2023-10-16T14:35:30.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.075+0000 7f9f6ccc2640 1 -- start start 2023-10-16T14:35:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.076+0000 7f9f6ccc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f6806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.076+0000 7f9f6ccc2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9f680ad970 0x7f9f6815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:30.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.076+0000 7f9f6ccc2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6806c960 con 0x7f9f6815ebc0 2023-10-16T14:35:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.076+0000 7f9f6ccc2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6815b9d0 con 0x7f9f680ad970 2023-10-16T14:35:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.076+0000 7f9f6ccc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6815bcb0 con 0x7f9f6806c040 2023-10-16T14:35:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.077+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f6806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.077+0000 7f9f677fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f6806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:58454/0 (socket says 172.21.15.29:58454) 2023-10-16T14:35:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.077+0000 7f9f5ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9f680ad970 0x7f9f6815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:30.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.077+0000 7f9f677fe640 1 -- 172.21.15.29:0/2311603024 learned_addr learned my addr 172.21.15.29:0/2311603024 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.078+0000 7f9f677fe640 1 -- 172.21.15.29:0/2311603024 >> v2:172.21.15.170:3300/0 conn(0x7f9f680ad970 msgr2=0x7f9f6815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.078+0000 7f9f677fe640 1 --2- 172.21.15.29:0/2311603024 >> v2:172.21.15.170:3300/0 conn(0x7f9f680ad970 0x7f9f6815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.078+0000 7f9f677fe640 1 -- 172.21.15.29:0/2311603024 >> v1:172.21.15.170:6789/0 conn(0x7f9f6815ebc0 legacy=0x7f9f680cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.078+0000 7f9f677fe640 1 -- 172.21.15.29:0/2311603024 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6815bf90 con 0x7f9f6806c040 2023-10-16T14:35:30.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.078+0000 7f9f5ffff640 1 --2- 172.21.15.29:0/2311603024 >> v2:172.21.15.170:3300/0 conn(0x7f9f680ad970 0x7f9f6815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:35:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.080+0000 7f9f677fe640 1 --2- 172.21.15.29:0/2311603024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f6806c420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f580098a0 tx=0x7f9f5802b970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a1943faf91773fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.080+0000 7f9f66ffd640 1 -- 172.21.15.29:0/2311603024 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58030830 con 0x7f9f6806c040 2023-10-16T14:35:30.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.080+0000 7f9f66ffd640 1 -- 172.21.15.29:0/2311603024 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58030990 con 0x7f9f6806c040 2023-10-16T14:35:30.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.080+0000 7f9f66ffd640 1 -- 172.21.15.29:0/2311603024 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58030d20 con 0x7f9f6806c040 2023-10-16T14:35:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2311603024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 msgr2=0x7f9f6806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2311603024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f6806c420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f580098a0 tx=0x7f9f5802b970 comp rx=0 tx=0).stop 2023-10-16T14:35:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2311603024 shutdown_connections 2023-10-16T14:35:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2311603024 >> v2:172.21.15.170:3300/0 conn(0x7f9f680ad970 0x7f9f6815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2311603024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f6806c420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2311603024 >> 172.21.15.29:0/2311603024 conn(0x7f9f680ce6a0 msgr2=0x7f9f680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2311603024 shutdown_connections 2023-10-16T14:35:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.081+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2311603024 wait complete. 2023-10-16T14:35:30.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.083+0000 7f9f6ccc2640 1 Processor -- start 2023-10-16T14:35:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.083+0000 7f9f6ccc2640 1 -- start start 2023-10-16T14:35:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f6ccc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f680b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f6ccc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 0x7f9f6815c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:30.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f6ccc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f6815ebc0 0x7f9f6815ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f6ccc2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680c8050 con 0x7f9f680ad970 2023-10-16T14:35:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f6ccc2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680b1ad0 con 0x7f9f6815ebc0 2023-10-16T14:35:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f6ccc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680b1d90 con 0x7f9f6806c040 2023-10-16T14:35:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.084+0000 7f9f5ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 0x7f9f6815c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:30.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 0x7f9f6815c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52000/0 (socket says 172.21.15.29:52000) 2023-10-16T14:35:30.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 -- 172.21.15.29:0/2535061292 learned_addr learned my addr 172.21.15.29:0/2535061292 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:30.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f677fe640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f680b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:30.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f67fff640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f6815ebc0 0x7f9f6815ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:30.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 -- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 msgr2=0x7f9f680b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:30.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f680b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:30.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 -- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f6815ebc0 msgr2=0x7f9f6815ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f6815ebc0 0x7f9f6815ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 -- 172.21.15.29:0/2535061292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f50009400 con 0x7f9f680ad970 2023-10-16T14:35:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.085+0000 7f9f5ffff640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 0x7f9f6815c930 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000e960 tx=0x7f9f5000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.086+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000ca00 con 0x7f9f680ad970 2023-10-16T14:35:30.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.086+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5000cb60 con 0x7f9f680ad970 2023-10-16T14:35:30.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.086+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50013470 con 0x7f9f680ad970 2023-10-16T14:35:30.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.086+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f580094d0 con 0x7f9f680ad970 2023-10-16T14:35:30.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.086+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f680b2350 con 0x7f9f680ad970 2023-10-16T14:35:30.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.087+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6806b330 con 0x7f9f680ad970 2023-10-16T14:35:30.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.088+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9f500135d0 con 0x7f9f680ad970 2023-10-16T14:35:30.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.090+0000 7f9f64ff9640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f9f4806af90 0x7f9f4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:30.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.090+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f50089a20 con 0x7f9f680ad970 2023-10-16T14:35:30.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.090+0000 7f9f677fe640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f9f4806af90 0x7f9f4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:30.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.091+0000 7f9f677fe640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f9f4806af90 0x7f9f4806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f58036000 tx=0x7f9f5803b040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.094+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9f500549b0 con 0x7f9f680ad970 2023-10-16T14:35:30.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:30.327+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9f6804e9b0 con 0x7f9f680ad970 2023-10-16T14:35:31.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.260+0000 7f9f64ff9640 1 -- 172.21.15.29:0/2535061292 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9f50057fd0 con 0x7f9f680ad970 2023-10-16T14:35:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f9f4806af90 msgr2=0x7f9f4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f9f4806af90 0x7f9f4806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f58036000 tx=0x7f9f5803b040 comp rx=0 tx=0).stop 2023-10-16T14:35:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 msgr2=0x7f9f6815c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 0x7f9f6815c930 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000e960 tx=0x7f9f5000ee20 comp rx=0 tx=0).stop 2023-10-16T14:35:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 shutdown_connections 2023-10-16T14:35:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f9f4806af90 0x7f9f4806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f680ad970 0x7f9f6815c930 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9f6815ebc0 0x7f9f6815ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 --2- 172.21.15.29:0/2535061292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9f6806c040 0x7f9f680b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.264+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 >> 172.21.15.29:0/2535061292 conn(0x7f9f680ce6a0 msgr2=0x7f9f6815d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.265+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 shutdown_connections 2023-10-16T14:35:31.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.265+0000 7f9f6ccc2640 1 -- 172.21.15.29:0/2535061292 wait complete. 2023-10-16T14:35:31.288 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:35:31.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:31.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.513+0000 7ff9dbb55640 1 Processor -- start 2023-10-16T14:35:31.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.514+0000 7ff9dbb55640 1 -- start start 2023-10-16T14:35:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.515+0000 7ff9dbb55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.515+0000 7ff9dbb55640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9d40b0b20 0x7ff9d406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.515+0000 7ff9dbb55640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d406f7f0 con 0x7ff9d4159f00 2023-10-16T14:35:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.515+0000 7ff9dbb55640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d406c5e0 con 0x7ff9d40b0b20 2023-10-16T14:35:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.515+0000 7ff9dbb55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d406c8c0 con 0x7ff9d4159b20 2023-10-16T14:35:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9da0f2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9d40b0b20 0x7ff9d406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9d90f0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff9d4159f00 0x7ff9d40b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32918/0 (socket says 172.21.15.29:32918) 2023-10-16T14:35:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9d90f0640 1 -- 172.21.15.29:0/3141142621 learned_addr learned my addr 172.21.15.29:0/3141142621 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9d98f1640 1 --2- 172.21.15.29:0/3141142621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3141142621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 msgr2=0x7ff9d406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9da0f2640 1 --2- 172.21.15.29:0/3141142621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.516+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3141142621 >> v1:172.21.15.29:6789/0 conn(0x7ff9d4159f00 legacy=0x7ff9d40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.517+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3141142621 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d406cba0 con 0x7ff9d40b0b20 2023-10-16T14:35:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.517+0000 7ff9d88ef640 1 -- 172.21.15.29:0/3141142621 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442862640 0 0) 0x7ff9d406f7f0 con 0x7ff9d4159f00 2023-10-16T14:35:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.518+0000 7ff9da0f2640 1 --2- 172.21.15.29:0/3141142621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d40b0b20 0x7ff9d406fc80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009d90 tx=0x7ff9cc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b6ca14f041e03e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.518+0000 7ff9d88ef640 1 -- 172.21.15.29:0/3141142621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc02cc50 con 0x7ff9d40b0b20 2023-10-16T14:35:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.518+0000 7ff9d88ef640 1 -- 172.21.15.29:0/3141142621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc02cdb0 con 0x7ff9d40b0b20 2023-10-16T14:35:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.518+0000 7ff9d88ef640 1 -- 172.21.15.29:0/3141142621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc034660 con 0x7ff9d40b0b20 2023-10-16T14:35:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3141142621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d40b0b20 msgr2=0x7ff9d406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3141142621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d40b0b20 0x7ff9d406fc80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009d90 tx=0x7ff9cc02bea0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3141142621 shutdown_connections 2023-10-16T14:35:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3141142621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d40b0b20 0x7ff9d406fc80 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3141142621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d406bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3141142621 >> 172.21.15.29:0/3141142621 conn(0x7ff9d40cb410 msgr2=0x7ff9d40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3141142621 shutdown_connections 2023-10-16T14:35:31.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.519+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3141142621 wait complete. 2023-10-16T14:35:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.521+0000 7ff9dbb55640 1 Processor -- start 2023-10-16T14:35:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.521+0000 7ff9dbb55640 1 -- start start 2023-10-16T14:35:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.522+0000 7ff9dbb55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d40b0b20 0x7ff9d40af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.522+0000 7ff9dbb55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d414eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.522+0000 7ff9dbb55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 0x7ff9d40c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.522+0000 7ff9dbb55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d414f400 con 0x7ff9d4159f00 2023-10-16T14:35:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.522+0000 7ff9dbb55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d415a520 con 0x7ff9d40b0b20 2023-10-16T14:35:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.522+0000 7ff9dbb55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40c3410 con 0x7ff9d4159b20 2023-10-16T14:35:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 0x7ff9d40c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 0x7ff9d40c2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52020/0 (socket says 172.21.15.29:52020) 2023-10-16T14:35:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3581195730 learned_addr learned my addr 172.21.15.29:0/3581195730 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9d98f1640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d40b0b20 0x7ff9d40af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9d90f0640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d414eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 msgr2=0x7ff9d414eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d414eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d40b0b20 msgr2=0x7ff9d40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d40b0b20 0x7ff9d40af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9da0f2640 1 -- 172.21.15.29:0/3581195730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc009a70 con 0x7ff9d4159f00 2023-10-16T14:35:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.523+0000 7ff9d90f0640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d414eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.524+0000 7ff9da0f2640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 0x7ff9d40c2ed0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009d90 tx=0x7ff9cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.524+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc040650 con 0x7ff9d4159f00 2023-10-16T14:35:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.524+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc0407d0 con 0x7ff9d4159f00 2023-10-16T14:35:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.524+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d40c3690 con 0x7ff9d4159f00 2023-10-16T14:35:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.524+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc040b80 con 0x7ff9d4159f00 2023-10-16T14:35:31.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.524+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d40c3c50 con 0x7ff9d4159f00 2023-10-16T14:35:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.525+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d404e160 con 0x7ff9d4159f00 2023-10-16T14:35:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.526+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9cc040ce0 con 0x7ff9d4159f00 2023-10-16T14:35:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.528+0000 7ff9ca7fc640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7ff9b806af90 0x7ff9b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.528+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9cc0ac560 con 0x7ff9d4159f00 2023-10-16T14:35:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.528+0000 7ff9d98f1640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7ff9b806af90 0x7ff9b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.530+0000 7ff9d98f1640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7ff9b806af90 0x7ff9b806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006fd0 tx=0x7ff9d0008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:31.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.532+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9cc077500 con 0x7ff9d4159f00 2023-10-16T14:35:31.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.762+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9d40c54f0 con 0x7ff9d4159f00 2023-10-16T14:35:31.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.764+0000 7ff9ca7fc640 1 -- 172.21.15.29:0/3581195730 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ff9cc07ab20 con 0x7ff9d4159f00 2023-10-16T14:35:31.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:31.768 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:35:27.247546+0000","modified":"2023-10-16T14:35:30.329395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7812":{"gid":7812,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2882189329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2882189329},{"type":"v1","addr":"172.21.15.29:6835","nonce":2882189329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T14:35:31.769 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2023-10-16T14:35:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.766+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7ff9b806af90 msgr2=0x7ff9b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.766+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7ff9b806af90 0x7ff9b806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006fd0 tx=0x7ff9d0008040 comp rx=0 tx=0).stop 2023-10-16T14:35:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.766+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 msgr2=0x7ff9d40c2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.766+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 0x7ff9d40c2ed0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009d90 tx=0x7ff9cc004a40 comp rx=0 tx=0).stop 2023-10-16T14:35:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 shutdown_connections 2023-10-16T14:35:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7ff9b806af90 0x7ff9b806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d4159f00 0x7ff9d40c2ed0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d4159b20 0x7ff9d414eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 --2- 172.21.15.29:0/3581195730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d40b0b20 0x7ff9d40af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 >> 172.21.15.29:0/3581195730 conn(0x7ff9d40cb410 msgr2=0x7ff9d415d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:31.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 shutdown_connections 2023-10-16T14:35:31.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:31.767+0000 7ff9dbb55640 1 -- 172.21.15.29:0/3581195730 wait complete. 2023-10-16T14:35:31.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:35:32.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.008+0000 7f554ccca640 1 Processor -- start 2023-10-16T14:35:32.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.008+0000 7f554ccca640 1 -- start start 2023-10-16T14:35:32.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.009+0000 7f554ccca640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5548070b60 0x7f554815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.009+0000 7f554ccca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f554806c100 0x7f554806c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.009+0000 7f554ccca640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554815aef0 con 0x7f5548159950 2023-10-16T14:35:32.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.009+0000 7f554ccca640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554815e490 con 0x7f5548070b60 2023-10-16T14:35:32.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.009+0000 7f554ccca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554815d910 con 0x7f554806c100 2023-10-16T14:35:32.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.010+0000 7f553ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5548070b60 0x7f554815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.010+0000 7f5547fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f554806c100 0x7f554806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.010+0000 7f553ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5548070b60 0x7f554815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:60810/0 (socket says 172.21.15.29:60810) 2023-10-16T14:35:32.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.010+0000 7f553ffff640 1 -- 172.21.15.29:0/2781833114 learned_addr learned my addr 172.21.15.29:0/2781833114 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:32.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.011+0000 7f553ffff640 1 -- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f554806c100 msgr2=0x7f554806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:32.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.011+0000 7f553ffff640 1 --2- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f554806c100 0x7f554806c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.011+0000 7f553ffff640 1 -- 172.21.15.29:0/2781833114 >> v1:172.21.15.170:6789/0 conn(0x7f5548159950 legacy=0x7f5548070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.011+0000 7f553ffff640 1 -- 172.21.15.29:0/2781833114 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554815dbf0 con 0x7f5548070b60 2023-10-16T14:35:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.012+0000 7f553ffff640 1 --2- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5548070b60 0x7f554815a9b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5530009870 tx=0x7f553002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e7720a04423fdfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.013+0000 7f5546ffd640 1 -- 172.21.15.29:0/2781833114 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530031640 con 0x7f5548070b60 2023-10-16T14:35:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.013+0000 7f5546ffd640 1 -- 172.21.15.29:0/2781833114 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55300317a0 con 0x7f5548070b60 2023-10-16T14:35:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.013+0000 7f5546ffd640 1 -- 172.21.15.29:0/2781833114 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530031b30 con 0x7f5548070b60 2023-10-16T14:35:32.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 -- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5548070b60 msgr2=0x7f554815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 --2- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5548070b60 0x7f554815a9b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5530009870 tx=0x7f553002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 -- 172.21.15.29:0/2781833114 shutdown_connections 2023-10-16T14:35:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 --2- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f554806c100 0x7f554806c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 --2- 172.21.15.29:0/2781833114 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5548070b60 0x7f554815a9b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 -- 172.21.15.29:0/2781833114 >> 172.21.15.29:0/2781833114 conn(0x7f55480ce6a0 msgr2=0x7f55480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 -- 172.21.15.29:0/2781833114 shutdown_connections 2023-10-16T14:35:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.014+0000 7f554ccca640 1 -- 172.21.15.29:0/2781833114 wait complete. 2023-10-16T14:35:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.016+0000 7f554ccca640 1 Processor -- start 2023-10-16T14:35:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.016+0000 7f554ccca640 1 -- start start 2023-10-16T14:35:32.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f554ccca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f554806c100 0x7f554814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f554ccca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 0x7f554814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f554ccca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5548159950 0x7f55480c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f554ccca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55481583e0 con 0x7f5548070b60 2023-10-16T14:35:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f554ccca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5548157f50 con 0x7f554806c100 2023-10-16T14:35:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f554ccca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5548157340 con 0x7f5548159950 2023-10-16T14:35:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f553ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 0x7f554814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f553ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 0x7f554814eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52026/0 (socket says 172.21.15.29:52026) 2023-10-16T14:35:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f553ffff640 1 -- 172.21.15.29:0/4001085592 learned_addr learned my addr 172.21.15.29:0/4001085592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f55477fe640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f554806c100 0x7f554814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.017+0000 7f5547fff640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5548159950 0x7f55480c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f553ffff640 1 -- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5548159950 msgr2=0x7f55480c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f553ffff640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5548159950 0x7f55480c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f553ffff640 1 -- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f554806c100 msgr2=0x7f554814e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f553ffff640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f554806c100 0x7f554814e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f553ffff640 1 -- 172.21.15.29:0/4001085592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55300094d0 con 0x7f5548070b60 2023-10-16T14:35:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f5547fff640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5548159950 0x7f55480c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f55477fe640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f554806c100 0x7f554814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f553ffff640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 0x7f554814eea0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5530000c00 tx=0x7f55300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553003a9f0 con 0x7f5548070b60 2023-10-16T14:35:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553003ab90 con 0x7f5548070b60 2023-10-16T14:35:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55300443f0 con 0x7f5548070b60 2023-10-16T14:35:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.018+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5548157620 con 0x7f5548070b60 2023-10-16T14:35:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.019+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5548157be0 con 0x7f5548070b60 2023-10-16T14:35:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.020+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55481580c0 con 0x7f5548070b60 2023-10-16T14:35:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.021+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f553002f050 con 0x7f5548070b60 2023-10-16T14:35:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.021+0000 7f5544ff9640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f552806afc0 0x7f552806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.022+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55300ab860 con 0x7f5548070b60 2023-10-16T14:35:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.022+0000 7f55477fe640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f552806afc0 0x7f552806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.023+0000 7f55477fe640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f552806afc0 0x7f552806d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55480708a0 tx=0x7f5538009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.026+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5530076650 con 0x7f5548070b60 2023-10-16T14:35:32.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.221+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f554804e9b0 con 0x7f5548070b60 2023-10-16T14:35:32.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.224+0000 7f5544ff9640 1 -- 172.21.15.29:0/4001085592 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f5530079c70 con 0x7f5548070b60 2023-10-16T14:35:32.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:32.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":127,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:28.804865+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:35:32.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.226+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f552806afc0 msgr2=0x7f552806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.226+0000 7f554ccca640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f552806afc0 0x7f552806d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55480708a0 tx=0x7f5538009040 comp rx=0 tx=0).stop 2023-10-16T14:35:32.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.226+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 msgr2=0x7f554814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.226+0000 7f554ccca640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 0x7f554814eea0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5530000c00 tx=0x7f55300046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 shutdown_connections 2023-10-16T14:35:32.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f552806afc0 0x7f552806d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5548070b60 0x7f554814eea0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5548159950 0x7f55480c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 --2- 172.21.15.29:0/4001085592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f554806c100 0x7f554814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 >> 172.21.15.29:0/4001085592 conn(0x7f55480ce6a0 msgr2=0x7f5548155160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:32.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 shutdown_connections 2023-10-16T14:35:32.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.227+0000 7f554ccca640 1 -- 172.21.15.29:0/4001085592 wait complete. 2023-10-16T14:35:32.250 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:35:32.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:35:32.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.461+0000 7f8736e60640 1 Processor -- start 2023-10-16T14:35:32.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.461+0000 7f8736e60640 1 -- start start 2023-10-16T14:35:32.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.463+0000 7f8736e60640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f87300cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.463+0000 7f8736e60640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f87300ccfb0 0x7f87300cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.463+0000 7f8736e60640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873015b9d0 con 0x7f873006c040 2023-10-16T14:35:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.463+0000 7f8736e60640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873015bcb0 con 0x7f87300ccfb0 2023-10-16T14:35:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.463+0000 7f8736e60640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873015bf90 con 0x7f87300ad970 2023-10-16T14:35:32.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.464+0000 7f873665f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f87300ccfb0 0x7f87300cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.464+0000 7f873565d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f87300cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.464+0000 7f8735e5e640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f873006c040 0x7f873006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:42430/0 (socket says 172.21.15.29:42430) 2023-10-16T14:35:32.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.464+0000 7f8735e5e640 1 -- 172.21.15.29:0/824398358 learned_addr learned my addr 172.21.15.29:0/824398358 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:32.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.465+0000 7f873665f640 1 -- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 msgr2=0x7f87300cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:32.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.465+0000 7f873665f640 1 --2- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f87300cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.465+0000 7f873665f640 1 -- 172.21.15.29:0/824398358 >> v1:172.21.15.170:6789/0 conn(0x7f873006c040 legacy=0x7f873006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.465+0000 7f873665f640 1 -- 172.21.15.29:0/824398358 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87300cc270 con 0x7f87300ccfb0 2023-10-16T14:35:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.466+0000 7f873665f640 1 --2- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87300ccfb0 0x7f87300cd440 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8728009dc0 tx=0x7f872802be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31dfdb5e77f77b46 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.467+0000 7f8734e5c640 1 -- 172.21.15.29:0/824398358 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872802cc30 con 0x7f87300ccfb0 2023-10-16T14:35:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.467+0000 7f8734e5c640 1 -- 172.21.15.29:0/824398358 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872802cd90 con 0x7f87300ccfb0 2023-10-16T14:35:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.467+0000 7f8734e5c640 1 -- 172.21.15.29:0/824398358 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728034660 con 0x7f87300ccfb0 2023-10-16T14:35:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.467+0000 7f8736e60640 1 -- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87300ccfb0 msgr2=0x7f87300cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.467+0000 7f8736e60640 1 --2- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87300ccfb0 0x7f87300cd440 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8728009dc0 tx=0x7f872802be80 comp rx=0 tx=0).stop 2023-10-16T14:35:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.467+0000 7f8736e60640 1 -- 172.21.15.29:0/824398358 shutdown_connections 2023-10-16T14:35:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.468+0000 7f8736e60640 1 --2- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87300ccfb0 0x7f87300cd440 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.468+0000 7f8736e60640 1 --2- 172.21.15.29:0/824398358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f87300cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.468+0000 7f8736e60640 1 -- 172.21.15.29:0/824398358 >> 172.21.15.29:0/824398358 conn(0x7f87300ce6a0 msgr2=0x7f87300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.468+0000 7f8736e60640 1 -- 172.21.15.29:0/824398358 shutdown_connections 2023-10-16T14:35:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.468+0000 7f8736e60640 1 -- 172.21.15.29:0/824398358 wait complete. 2023-10-16T14:35:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.469+0000 7f8736e60640 1 Processor -- start 2023-10-16T14:35:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.470+0000 7f8736e60640 1 -- start start 2023-10-16T14:35:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8736e60640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f873006c040 0x7f8730152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8736e60640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f873015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8736e60640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 0x7f873015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8736e60640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87300c7790 con 0x7f87300ccfb0 2023-10-16T14:35:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8736e60640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8730153110 con 0x7f873006c040 2023-10-16T14:35:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8736e60640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87300c6b50 con 0x7f87300ad970 2023-10-16T14:35:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f873665f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 0x7f873015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f873665f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 0x7f873015cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52030/0 (socket says 172.21.15.29:52030) 2023-10-16T14:35:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f873665f640 1 -- 172.21.15.29:0/3536541317 learned_addr learned my addr 172.21.15.29:0/3536541317 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f8735e5e640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f873006c040 0x7f8730152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.471+0000 7f873565d640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f873015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873665f640 1 -- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 msgr2=0x7f873015c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873665f640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f873015c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873665f640 1 -- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f873006c040 msgr2=0x7f8730152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873665f640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f873006c040 0x7f8730152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873665f640 1 -- 172.21.15.29:0/3536541317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728009a70 con 0x7f87300ccfb0 2023-10-16T14:35:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873565d640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f873015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f873665f640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 0x7f873015cd80 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8728009ee0 tx=0x7f8728004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728040650 con 0x7f87300ccfb0 2023-10-16T14:35:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87280407b0 con 0x7f87300ccfb0 2023-10-16T14:35:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87300c6df0 con 0x7f87300ccfb0 2023-10-16T14:35:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.472+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728040b60 con 0x7f87300ccfb0 2023-10-16T14:35:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.473+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87300c73b0 con 0x7f87300ccfb0 2023-10-16T14:35:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.474+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873006b330 con 0x7f87300ccfb0 2023-10-16T14:35:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.475+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8728040cc0 con 0x7f87300ccfb0 2023-10-16T14:35:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.476+0000 7f87267fc640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f871406af90 0x7f871406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.476+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87280ac4d0 con 0x7f87300ccfb0 2023-10-16T14:35:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.477+0000 7f8735e5e640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f871406af90 0x7f871406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.478+0000 7f8735e5e640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f871406af90 0x7f871406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f872c005880 tx=0x7f872c009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:32.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.481+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8728077460 con 0x7f87300ccfb0 2023-10-16T14:35:32.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.664+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 --> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87300cc550 con 0x7f871406af90 2023-10-16T14:35:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.693+0000 7f87267fc640 1 -- 172.21.15.29:0/3536541317 <== mgr.6406 v2:172.21.15.170:6832/1801486769 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286932 (secure 0 0 0) 0x7f872c001c50 con 0x7f871406af90 2023-10-16T14:35:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:35:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.698+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f871406af90 msgr2=0x7f871406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.698+0000 7f8736e60640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f871406af90 0x7f871406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f872c005880 tx=0x7f872c009040 comp rx=0 tx=0).stop 2023-10-16T14:35:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.698+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 msgr2=0x7f873015cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.698+0000 7f8736e60640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 0x7f873015cd80 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8728009ee0 tx=0x7f8728004a40 comp rx=0 tx=0).stop 2023-10-16T14:35:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 shutdown_connections 2023-10-16T14:35:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f871406af90 0x7f871406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87300ccfb0 0x7f873015cd80 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87300ad970 0x7f873015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 --2- 172.21.15.29:0/3536541317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f873006c040 0x7f8730152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 >> 172.21.15.29:0/3536541317 conn(0x7f87300ce6a0 msgr2=0x7f873006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 shutdown_connections 2023-10-16T14:35:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.699+0000 7f8736e60640 1 -- 172.21.15.29:0/3536541317 wait complete. 2023-10-16T14:35:32.721 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T14:35:32.568236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":367,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":350304,"kb_used_data":13408,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754624416,"statfs":{"total":773094113280,"available":772735401984,"internally_reserved":0,"allocated":13729792,"data_stored":7207556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":344837689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002244"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816905+0000","last_change":"2023-10-16T14:35:27.830951+0000","last_active":"2023-10-16T14:35:28.816905+0000","last_peered":"2023-10-16T14:35:28.816905+0000","last_clean":"2023-10-16T14:35:28.816905+0000","last_became_active":"2023-10-16T14:35:27.830681+0000","last_became_peered":"2023-10-16T14:35:27.830681+0000","last_unstale":"2023-10-16T14:35:28.816905+0000","last_undegraded":"2023-10-16T14:35:28.816905+0000","last_fullsized":"2023-10-16T14:35:28.816905+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:13.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921665+0000","last_change":"2023-10-16T14:35:27.827055+0000","last_active":"2023-10-16T14:35:28.921665+0000","last_peered":"2023-10-16T14:35:28.921665+0000","last_clean":"2023-10-16T14:35:28.921665+0000","last_became_active":"2023-10-16T14:35:27.826772+0000","last_became_peered":"2023-10-16T14:35:27.826772+0000","last_unstale":"2023-10-16T14:35:28.921665+0000","last_undegraded":"2023-10-16T14:35:28.921665+0000","last_fullsized":"2023-10-16T14:35:28.921665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.752080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816762+0000","last_change":"2023-10-16T14:35:27.831784+0000","last_active":"2023-10-16T14:35:28.816762+0000","last_peered":"2023-10-16T14:35:28.816762+0000","last_clean":"2023-10-16T14:35:28.816762+0000","last_became_active":"2023-10-16T14:35:27.831479+0000","last_became_peered":"2023-10-16T14:35:27.831479+0000","last_unstale":"2023-10-16T14:35:28.816762+0000","last_undegraded":"2023-10-16T14:35:28.816762+0000","last_fullsized":"2023-10-16T14:35:28.816762+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.575493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.840331+0000","last_change":"2023-10-16T14:35:27.833085+0000","last_active":"2023-10-16T14:35:27.840331+0000","last_peered":"2023-10-16T14:35:27.840331+0000","last_clean":"2023-10-16T14:35:27.840331+0000","last_became_active":"2023-10-16T14:35:27.832572+0000","last_became_peered":"2023-10-16T14:35:27.832572+0000","last_unstale":"2023-10-16T14:35:27.840331+0000","last_undegraded":"2023-10-16T14:35:27.840331+0000","last_fullsized":"2023-10-16T14:35:27.840331+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:48.160322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820612+0000","last_change":"2023-10-16T14:35:27.827982+0000","last_active":"2023-10-16T14:35:28.820612+0000","last_peered":"2023-10-16T14:35:28.820612+0000","last_clean":"2023-10-16T14:35:28.820612+0000","last_became_active":"2023-10-16T14:35:27.827430+0000","last_became_peered":"2023-10-16T14:35:27.827430+0000","last_unstale":"2023-10-16T14:35:28.820612+0000","last_undegraded":"2023-10-16T14:35:28.820612+0000","last_fullsized":"2023-10-16T14:35:28.820612+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:13.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822512+0000","last_change":"2023-10-16T14:35:27.816233+0000","last_active":"2023-10-16T14:35:28.822512+0000","last_peered":"2023-10-16T14:35:28.822512+0000","last_clean":"2023-10-16T14:35:28.822512+0000","last_became_active":"2023-10-16T14:35:27.815961+0000","last_became_peered":"2023-10-16T14:35:27.815961+0000","last_unstale":"2023-10-16T14:35:28.822512+0000","last_undegraded":"2023-10-16T14:35:28.822512+0000","last_fullsized":"2023-10-16T14:35:28.822512+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:44.909816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822128+0000","last_change":"2023-10-16T14:35:27.815743+0000","last_active":"2023-10-16T14:35:28.822128+0000","last_peered":"2023-10-16T14:35:28.822128+0000","last_clean":"2023-10-16T14:35:28.822128+0000","last_became_active":"2023-10-16T14:35:27.815455+0000","last_became_peered":"2023-10-16T14:35:27.815455+0000","last_unstale":"2023-10-16T14:35:28.822128+0000","last_undegraded":"2023-10-16T14:35:28.822128+0000","last_fullsized":"2023-10-16T14:35:28.822128+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:48.573039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920587+0000","last_change":"2023-10-16T14:35:27.816981+0000","last_active":"2023-10-16T14:35:28.920587+0000","last_peered":"2023-10-16T14:35:28.920587+0000","last_clean":"2023-10-16T14:35:28.920587+0000","last_became_active":"2023-10-16T14:35:27.815791+0000","last_became_peered":"2023-10-16T14:35:27.815791+0000","last_unstale":"2023-10-16T14:35:28.920587+0000","last_undegraded":"2023-10-16T14:35:28.920587+0000","last_fullsized":"2023-10-16T14:35:28.920587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:18.609488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922268+0000","last_change":"2023-10-16T14:35:27.835255+0000","last_active":"2023-10-16T14:35:28.922268+0000","last_peered":"2023-10-16T14:35:28.922268+0000","last_clean":"2023-10-16T14:35:28.922268+0000","last_became_active":"2023-10-16T14:35:27.833530+0000","last_became_peered":"2023-10-16T14:35:27.833530+0000","last_unstale":"2023-10-16T14:35:28.922268+0000","last_undegraded":"2023-10-16T14:35:28.922268+0000","last_fullsized":"2023-10-16T14:35:28.922268+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:42.775538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820082+0000","last_change":"2023-10-16T14:35:27.814037+0000","last_active":"2023-10-16T14:35:28.820082+0000","last_peered":"2023-10-16T14:35:28.820082+0000","last_clean":"2023-10-16T14:35:28.820082+0000","last_became_active":"2023-10-16T14:35:27.813731+0000","last_became_peered":"2023-10-16T14:35:27.813731+0000","last_unstale":"2023-10-16T14:35:28.820082+0000","last_undegraded":"2023-10-16T14:35:28.820082+0000","last_fullsized":"2023-10-16T14:35:28.820082+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.752080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822598+0000","last_change":"2023-10-16T14:35:27.821820+0000","last_active":"2023-10-16T14:35:28.822598+0000","last_peered":"2023-10-16T14:35:28.822598+0000","last_clean":"2023-10-16T14:35:28.822598+0000","last_became_active":"2023-10-16T14:35:27.821352+0000","last_became_peered":"2023-10-16T14:35:27.821352+0000","last_unstale":"2023-10-16T14:35:28.822598+0000","last_undegraded":"2023-10-16T14:35:28.822598+0000","last_fullsized":"2023-10-16T14:35:28.822598+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:25.043951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921708+0000","last_change":"2023-10-16T14:35:27.832760+0000","last_active":"2023-10-16T14:35:28.921708+0000","last_peered":"2023-10-16T14:35:28.921708+0000","last_clean":"2023-10-16T14:35:28.921708+0000","last_became_active":"2023-10-16T14:35:27.832402+0000","last_became_peered":"2023-10-16T14:35:27.832402+0000","last_unstale":"2023-10-16T14:35:28.921708+0000","last_undegraded":"2023-10-16T14:35:28.921708+0000","last_fullsized":"2023-10-16T14:35:28.921708+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:22.707752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.840403+0000","last_change":"2023-10-16T14:35:27.833431+0000","last_active":"2023-10-16T14:35:27.840403+0000","last_peered":"2023-10-16T14:35:27.840403+0000","last_clean":"2023-10-16T14:35:27.840403+0000","last_became_active":"2023-10-16T14:35:27.833085+0000","last_became_peered":"2023-10-16T14:35:27.833085+0000","last_unstale":"2023-10-16T14:35:27.840403+0000","last_undegraded":"2023-10-16T14:35:27.840403+0000","last_fullsized":"2023-10-16T14:35:27.840403+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:05.767929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.815682+0000","last_change":"2023-10-16T14:35:27.807995+0000","last_active":"2023-10-16T14:35:27.815682+0000","last_peered":"2023-10-16T14:35:27.815682+0000","last_clean":"2023-10-16T14:35:27.815682+0000","last_became_active":"2023-10-16T14:35:27.805721+0000","last_became_peered":"2023-10-16T14:35:27.805721+0000","last_unstale":"2023-10-16T14:35:27.815682+0000","last_undegraded":"2023-10-16T14:35:27.815682+0000","last_fullsized":"2023-10-16T14:35:27.815682+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:19.752829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.837378+0000","last_change":"2023-10-16T14:35:27.832078+0000","last_active":"2023-10-16T14:35:27.837378+0000","last_peered":"2023-10-16T14:35:27.837378+0000","last_clean":"2023-10-16T14:35:27.837378+0000","last_became_active":"2023-10-16T14:35:27.831781+0000","last_became_peered":"2023-10-16T14:35:27.831781+0000","last_unstale":"2023-10-16T14:35:27.837378+0000","last_undegraded":"2023-10-16T14:35:27.837378+0000","last_fullsized":"2023-10-16T14:35:27.837378+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:51.132955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820387+0000","last_change":"2023-10-16T14:35:27.820597+0000","last_active":"2023-10-16T14:35:28.820387+0000","last_peered":"2023-10-16T14:35:28.820387+0000","last_clean":"2023-10-16T14:35:28.820387+0000","last_became_active":"2023-10-16T14:35:27.819037+0000","last_became_peered":"2023-10-16T14:35:27.819037+0000","last_unstale":"2023-10-16T14:35:28.820387+0000","last_undegraded":"2023-10-16T14:35:28.820387+0000","last_fullsized":"2023-10-16T14:35:28.820387+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:08.504041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922193+0000","last_change":"2023-10-16T14:35:27.830245+0000","last_active":"2023-10-16T14:35:28.922193+0000","last_peered":"2023-10-16T14:35:28.922193+0000","last_clean":"2023-10-16T14:35:28.922193+0000","last_became_active":"2023-10-16T14:35:27.829932+0000","last_became_peered":"2023-10-16T14:35:27.829932+0000","last_unstale":"2023-10-16T14:35:28.922193+0000","last_undegraded":"2023-10-16T14:35:28.922193+0000","last_fullsized":"2023-10-16T14:35:28.922193+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:33:41.377382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.842356+0000","last_change":"2023-10-16T14:35:27.839397+0000","last_active":"2023-10-16T14:35:27.842356+0000","last_peered":"2023-10-16T14:35:27.842356+0000","last_clean":"2023-10-16T14:35:27.842356+0000","last_became_active":"2023-10-16T14:35:27.838836+0000","last_became_peered":"2023-10-16T14:35:27.838836+0000","last_unstale":"2023-10-16T14:35:27.842356+0000","last_undegraded":"2023-10-16T14:35:27.842356+0000","last_fullsized":"2023-10-16T14:35:27.842356+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:41.296118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.823927+0000","last_change":"2023-10-16T14:35:27.817745+0000","last_active":"2023-10-16T14:35:27.823927+0000","last_peered":"2023-10-16T14:35:27.823927+0000","last_clean":"2023-10-16T14:35:27.823927+0000","last_became_active":"2023-10-16T14:35:27.816201+0000","last_became_peered":"2023-10-16T14:35:27.816201+0000","last_unstale":"2023-10-16T14:35:27.823927+0000","last_undegraded":"2023-10-16T14:35:27.823927+0000","last_fullsized":"2023-10-16T14:35:27.823927+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:33.292806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922086+0000","last_change":"2023-10-16T14:35:27.836038+0000","last_active":"2023-10-16T14:35:28.922086+0000","last_peered":"2023-10-16T14:35:28.922086+0000","last_clean":"2023-10-16T14:35:28.922086+0000","last_became_active":"2023-10-16T14:35:27.835440+0000","last_became_peered":"2023-10-16T14:35:27.835440+0000","last_unstale":"2023-10-16T14:35:28.922086+0000","last_undegraded":"2023-10-16T14:35:28.922086+0000","last_fullsized":"2023-10-16T14:35:28.922086+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:46.695154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.822932+0000","last_change":"2023-10-16T14:35:27.816668+0000","last_active":"2023-10-16T14:35:27.822932+0000","last_peered":"2023-10-16T14:35:27.822932+0000","last_clean":"2023-10-16T14:35:27.822932+0000","last_became_active":"2023-10-16T14:35:27.816364+0000","last_became_peered":"2023-10-16T14:35:27.816364+0000","last_unstale":"2023-10-16T14:35:27.822932+0000","last_undegraded":"2023-10-16T14:35:27.822932+0000","last_fullsized":"2023-10-16T14:35:27.822932+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:21:08.121776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920718+0000","last_change":"2023-10-16T14:35:27.818095+0000","last_active":"2023-10-16T14:35:28.920718+0000","last_peered":"2023-10-16T14:35:28.920718+0000","last_clean":"2023-10-16T14:35:28.920718+0000","last_became_active":"2023-10-16T14:35:27.817264+0000","last_became_peered":"2023-10-16T14:35:27.817264+0000","last_unstale":"2023-10-16T14:35:28.920718+0000","last_undegraded":"2023-10-16T14:35:28.920718+0000","last_fullsized":"2023-10-16T14:35:28.920718+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:56.927052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920836+0000","last_change":"2023-10-16T14:35:27.820763+0000","last_active":"2023-10-16T14:35:28.920836+0000","last_peered":"2023-10-16T14:35:28.920836+0000","last_clean":"2023-10-16T14:35:28.920836+0000","last_became_active":"2023-10-16T14:35:27.819076+0000","last_became_peered":"2023-10-16T14:35:27.819076+0000","last_unstale":"2023-10-16T14:35:28.920836+0000","last_undegraded":"2023-10-16T14:35:28.920836+0000","last_fullsized":"2023-10-16T14:35:28.920836+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.575493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822027+0000","last_change":"2023-10-16T14:35:27.815116+0000","last_active":"2023-10-16T14:35:28.822027+0000","last_peered":"2023-10-16T14:35:28.822027+0000","last_clean":"2023-10-16T14:35:28.822027+0000","last_became_active":"2023-10-16T14:35:27.814786+0000","last_became_peered":"2023-10-16T14:35:27.814786+0000","last_unstale":"2023-10-16T14:35:28.822027+0000","last_undegraded":"2023-10-16T14:35:28.822027+0000","last_fullsized":"2023-10-16T14:35:28.822027+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:04.120607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820984+0000","last_change":"2023-10-16T14:35:27.823185+0000","last_active":"2023-10-16T14:35:28.820984+0000","last_peered":"2023-10-16T14:35:28.820984+0000","last_clean":"2023-10-16T14:35:28.820984+0000","last_became_active":"2023-10-16T14:35:27.822845+0000","last_became_peered":"2023-10-16T14:35:27.822845+0000","last_unstale":"2023-10-16T14:35:28.820984+0000","last_undegraded":"2023-10-16T14:35:28.820984+0000","last_fullsized":"2023-10-16T14:35:28.820984+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:46.935993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921928+0000","last_change":"2023-10-16T14:35:27.836048+0000","last_active":"2023-10-16T14:35:28.921928+0000","last_peered":"2023-10-16T14:35:28.921928+0000","last_clean":"2023-10-16T14:35:28.921928+0000","last_became_active":"2023-10-16T14:35:27.835614+0000","last_became_peered":"2023-10-16T14:35:27.835614+0000","last_unstale":"2023-10-16T14:35:28.921928+0000","last_undegraded":"2023-10-16T14:35:28.921928+0000","last_fullsized":"2023-10-16T14:35:28.921928+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:13.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822743+0000","last_change":"2023-10-16T14:35:27.809671+0000","last_active":"2023-10-16T14:35:28.822743+0000","last_peered":"2023-10-16T14:35:28.822743+0000","last_clean":"2023-10-16T14:35:28.822743+0000","last_became_active":"2023-10-16T14:35:27.809123+0000","last_became_peered":"2023-10-16T14:35:27.809123+0000","last_unstale":"2023-10-16T14:35:28.822743+0000","last_undegraded":"2023-10-16T14:35:28.822743+0000","last_fullsized":"2023-10-16T14:35:28.822743+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:33.189682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822112+0000","last_change":"2023-10-16T14:35:27.834295+0000","last_active":"2023-10-16T14:35:28.822112+0000","last_peered":"2023-10-16T14:35:28.822112+0000","last_clean":"2023-10-16T14:35:28.822112+0000","last_became_active":"2023-10-16T14:35:27.833973+0000","last_became_peered":"2023-10-16T14:35:27.833973+0000","last_unstale":"2023-10-16T14:35:28.822112+0000","last_undegraded":"2023-10-16T14:35:28.822112+0000","last_fullsized":"2023-10-16T14:35:28.822112+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:59.436838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921202+0000","last_change":"2023-10-16T14:35:27.821621+0000","last_active":"2023-10-16T14:35:28.921202+0000","last_peered":"2023-10-16T14:35:28.921202+0000","last_clean":"2023-10-16T14:35:28.921202+0000","last_became_active":"2023-10-16T14:35:27.821329+0000","last_became_peered":"2023-10-16T14:35:27.821329+0000","last_unstale":"2023-10-16T14:35:28.921202+0000","last_undegraded":"2023-10-16T14:35:28.921202+0000","last_fullsized":"2023-10-16T14:35:28.921202+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:12.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.822181+0000","last_change":"2023-10-16T14:35:27.816836+0000","last_active":"2023-10-16T14:35:27.822181+0000","last_peered":"2023-10-16T14:35:27.822181+0000","last_clean":"2023-10-16T14:35:27.822181+0000","last_became_active":"2023-10-16T14:35:27.816499+0000","last_became_peered":"2023-10-16T14:35:27.816499+0000","last_unstale":"2023-10-16T14:35:27.822181+0000","last_undegraded":"2023-10-16T14:35:27.822181+0000","last_fullsized":"2023-10-16T14:35:27.822181+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:27.018557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.827635+0000","last_change":"2023-10-16T14:35:27.826183+0000","last_active":"2023-10-16T14:35:27.827635+0000","last_peered":"2023-10-16T14:35:27.827635+0000","last_clean":"2023-10-16T14:35:27.827635+0000","last_became_active":"2023-10-16T14:35:27.825868+0000","last_became_peered":"2023-10-16T14:35:27.825868+0000","last_unstale":"2023-10-16T14:35:27.827635+0000","last_undegraded":"2023-10-16T14:35:27.827635+0000","last_fullsized":"2023-10-16T14:35:27.827635+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:52.173443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.815929+0000","last_change":"2023-10-16T14:35:27.818655+0000","last_active":"2023-10-16T14:35:28.815929+0000","last_peered":"2023-10-16T14:35:28.815929+0000","last_clean":"2023-10-16T14:35:28.815929+0000","last_became_active":"2023-10-16T14:35:27.818345+0000","last_became_peered":"2023-10-16T14:35:27.818345+0000","last_unstale":"2023-10-16T14:35:28.815929+0000","last_undegraded":"2023-10-16T14:35:28.815929+0000","last_fullsized":"2023-10-16T14:35:28.815929+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:18.609488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.814308+0000","last_change":"2023-10-16T14:35:27.805559+0000","last_active":"2023-10-16T14:35:27.814308+0000","last_peered":"2023-10-16T14:35:27.814308+0000","last_clean":"2023-10-16T14:35:27.814308+0000","last_became_active":"2023-10-16T14:35:27.803828+0000","last_became_peered":"2023-10-16T14:35:27.803828+0000","last_unstale":"2023-10-16T14:35:27.814308+0000","last_undegraded":"2023-10-16T14:35:27.814308+0000","last_fullsized":"2023-10-16T14:35:27.814308+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:05.964248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"126'4","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922092+0000","last_change":"2023-10-16T14:35:26.795297+0000","last_active":"2023-10-16T14:35:28.922092+0000","last_peered":"2023-10-16T14:35:28.922092+0000","last_clean":"2023-10-16T14:35:28.922092+0000","last_became_active":"2023-10-16T14:35:26.794799+0000","last_became_peered":"2023-10-16T14:35:26.794799+0000","last_unstale":"2023-10-16T14:35:28.922092+0000","last_undegraded":"2023-10-16T14:35:28.922092+0000","last_fullsized":"2023-10-16T14:35:28.922092+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:49.435538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.835148+0000","last_change":"2023-10-16T14:35:27.829039+0000","last_active":"2023-10-16T14:35:27.835148+0000","last_peered":"2023-10-16T14:35:27.835148+0000","last_clean":"2023-10-16T14:35:27.835148+0000","last_became_active":"2023-10-16T14:35:27.826166+0000","last_became_peered":"2023-10-16T14:35:27.826166+0000","last_unstale":"2023-10-16T14:35:27.835148+0000","last_undegraded":"2023-10-16T14:35:27.835148+0000","last_fullsized":"2023-10-16T14:35:27.835148+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:14.351097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823006+0000","last_change":"2023-10-16T14:35:26.788980+0000","last_active":"2023-10-16T14:35:28.823006+0000","last_peered":"2023-10-16T14:35:28.823006+0000","last_clean":"2023-10-16T14:35:28.823006+0000","last_became_active":"2023-10-16T14:35:26.788451+0000","last_became_peered":"2023-10-16T14:35:26.788451+0000","last_unstale":"2023-10-16T14:35:28.823006+0000","last_undegraded":"2023-10-16T14:35:28.823006+0000","last_fullsized":"2023-10-16T14:35:28.823006+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:09.729246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816852+0000","last_change":"2023-10-16T14:35:27.832378+0000","last_active":"2023-10-16T14:35:28.816852+0000","last_peered":"2023-10-16T14:35:28.816852+0000","last_clean":"2023-10-16T14:35:28.816852+0000","last_became_active":"2023-10-16T14:35:27.832103+0000","last_became_peered":"2023-10-16T14:35:27.832103+0000","last_unstale":"2023-10-16T14:35:28.816852+0000","last_undegraded":"2023-10-16T14:35:28.816852+0000","last_fullsized":"2023-10-16T14:35:28.816852+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.752080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921405+0000","last_change":"2023-10-16T14:35:26.791016+0000","last_active":"2023-10-16T14:35:28.921405+0000","last_peered":"2023-10-16T14:35:28.921405+0000","last_clean":"2023-10-16T14:35:28.921405+0000","last_became_active":"2023-10-16T14:35:26.790715+0000","last_became_peered":"2023-10-16T14:35:26.790715+0000","last_unstale":"2023-10-16T14:35:28.921405+0000","last_undegraded":"2023-10-16T14:35:28.921405+0000","last_fullsized":"2023-10-16T14:35:28.921405+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:18.741245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921993+0000","last_change":"2023-10-16T14:35:27.818278+0000","last_active":"2023-10-16T14:35:28.921993+0000","last_peered":"2023-10-16T14:35:28.921993+0000","last_clean":"2023-10-16T14:35:28.921993+0000","last_became_active":"2023-10-16T14:35:27.817919+0000","last_became_peered":"2023-10-16T14:35:27.817919+0000","last_unstale":"2023-10-16T14:35:28.921993+0000","last_undegraded":"2023-10-16T14:35:28.921993+0000","last_fullsized":"2023-10-16T14:35:28.921993+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:58.730925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921400+0000","last_change":"2023-10-16T14:35:26.781983+0000","last_active":"2023-10-16T14:35:28.921400+0000","last_peered":"2023-10-16T14:35:28.921400+0000","last_clean":"2023-10-16T14:35:28.921400+0000","last_became_active":"2023-10-16T14:35:26.780624+0000","last_became_peered":"2023-10-16T14:35:26.780624+0000","last_unstale":"2023-10-16T14:35:28.921400+0000","last_undegraded":"2023-10-16T14:35:28.921400+0000","last_fullsized":"2023-10-16T14:35:28.921400+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:21:07.110192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.827069+0000","last_change":"2023-10-16T14:35:27.825513+0000","last_active":"2023-10-16T14:35:27.827069+0000","last_peered":"2023-10-16T14:35:27.827069+0000","last_clean":"2023-10-16T14:35:27.827069+0000","last_became_active":"2023-10-16T14:35:27.824446+0000","last_became_peered":"2023-10-16T14:35:27.824446+0000","last_unstale":"2023-10-16T14:35:27.827069+0000","last_undegraded":"2023-10-16T14:35:27.827069+0000","last_fullsized":"2023-10-16T14:35:27.827069+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:44.951954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816616+0000","last_change":"2023-10-16T14:35:26.797232+0000","last_active":"2023-10-16T14:35:28.816616+0000","last_peered":"2023-10-16T14:35:28.816616+0000","last_clean":"2023-10-16T14:35:28.816616+0000","last_became_active":"2023-10-16T14:35:26.795817+0000","last_became_peered":"2023-10-16T14:35:26.795817+0000","last_unstale":"2023-10-16T14:35:28.816616+0000","last_undegraded":"2023-10-16T14:35:28.816616+0000","last_fullsized":"2023-10-16T14:35:28.816616+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:49.435538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822530+0000","last_change":"2023-10-16T14:35:27.806755+0000","last_active":"2023-10-16T14:35:28.822530+0000","last_peered":"2023-10-16T14:35:28.822530+0000","last_clean":"2023-10-16T14:35:28.822530+0000","last_became_active":"2023-10-16T14:35:27.803901+0000","last_became_peered":"2023-10-16T14:35:27.803901+0000","last_unstale":"2023-10-16T14:35:28.822530+0000","last_undegraded":"2023-10-16T14:35:28.822530+0000","last_fullsized":"2023-10-16T14:35:28.822530+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:18.356865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920775+0000","last_change":"2023-10-16T14:35:26.792286+0000","last_active":"2023-10-16T14:35:28.920775+0000","last_peered":"2023-10-16T14:35:28.920775+0000","last_clean":"2023-10-16T14:35:28.920775+0000","last_became_active":"2023-10-16T14:35:26.791957+0000","last_became_peered":"2023-10-16T14:35:26.791957+0000","last_unstale":"2023-10-16T14:35:28.920775+0000","last_undegraded":"2023-10-16T14:35:28.920775+0000","last_fullsized":"2023-10-16T14:35:28.920775+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:21.059833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.840749+0000","last_change":"2023-10-16T14:35:27.834920+0000","last_active":"2023-10-16T14:35:27.840749+0000","last_peered":"2023-10-16T14:35:27.840749+0000","last_clean":"2023-10-16T14:35:27.840749+0000","last_became_active":"2023-10-16T14:35:27.834643+0000","last_became_peered":"2023-10-16T14:35:27.834643+0000","last_unstale":"2023-10-16T14:35:27.840749+0000","last_undegraded":"2023-10-16T14:35:27.840749+0000","last_fullsized":"2023-10-16T14:35:27.840749+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:04.469840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820135+0000","last_change":"2023-10-16T14:35:26.798916+0000","last_active":"2023-10-16T14:35:28.820135+0000","last_peered":"2023-10-16T14:35:28.820135+0000","last_clean":"2023-10-16T14:35:28.820135+0000","last_became_active":"2023-10-16T14:35:26.798582+0000","last_became_peered":"2023-10-16T14:35:26.798582+0000","last_unstale":"2023-10-16T14:35:28.820135+0000","last_undegraded":"2023-10-16T14:35:28.820135+0000","last_fullsized":"2023-10-16T14:35:28.820135+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:58.860098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921287+0000","last_change":"2023-10-16T14:35:27.819114+0000","last_active":"2023-10-16T14:35:28.921287+0000","last_peered":"2023-10-16T14:35:28.921287+0000","last_clean":"2023-10-16T14:35:28.921287+0000","last_became_active":"2023-10-16T14:35:27.818427+0000","last_became_peered":"2023-10-16T14:35:27.818427+0000","last_unstale":"2023-10-16T14:35:28.921287+0000","last_undegraded":"2023-10-16T14:35:28.921287+0000","last_fullsized":"2023-10-16T14:35:28.921287+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:53.697631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816050+0000","last_change":"2023-10-16T14:35:26.793141+0000","last_active":"2023-10-16T14:35:28.816050+0000","last_peered":"2023-10-16T14:35:28.816050+0000","last_clean":"2023-10-16T14:35:28.816050+0000","last_became_active":"2023-10-16T14:35:26.791132+0000","last_became_peered":"2023-10-16T14:35:26.791132+0000","last_unstale":"2023-10-16T14:35:28.816050+0000","last_undegraded":"2023-10-16T14:35:28.816050+0000","last_fullsized":"2023-10-16T14:35:28.816050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:41.689212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922181+0000","last_change":"2023-10-16T14:35:27.829767+0000","last_active":"2023-10-16T14:35:28.922181+0000","last_peered":"2023-10-16T14:35:28.922181+0000","last_clean":"2023-10-16T14:35:28.922181+0000","last_became_active":"2023-10-16T14:35:27.829354+0000","last_became_peered":"2023-10-16T14:35:27.829354+0000","last_unstale":"2023-10-16T14:35:28.922181+0000","last_undegraded":"2023-10-16T14:35:28.922181+0000","last_fullsized":"2023-10-16T14:35:28.922181+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:11.265954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816958+0000","last_change":"2023-10-16T14:35:26.795925+0000","last_active":"2023-10-16T14:35:28.816958+0000","last_peered":"2023-10-16T14:35:28.816958+0000","last_clean":"2023-10-16T14:35:28.816958+0000","last_became_active":"2023-10-16T14:35:26.795636+0000","last_became_peered":"2023-10-16T14:35:26.795636+0000","last_unstale":"2023-10-16T14:35:28.816958+0000","last_undegraded":"2023-10-16T14:35:28.816958+0000","last_fullsized":"2023-10-16T14:35:28.816958+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:26.082674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922271+0000","last_change":"2023-10-16T14:35:27.817417+0000","last_active":"2023-10-16T14:35:28.922271+0000","last_peered":"2023-10-16T14:35:28.922271+0000","last_clean":"2023-10-16T14:35:28.922271+0000","last_became_active":"2023-10-16T14:35:27.817122+0000","last_became_peered":"2023-10-16T14:35:27.817122+0000","last_unstale":"2023-10-16T14:35:28.922271+0000","last_undegraded":"2023-10-16T14:35:28.922271+0000","last_fullsized":"2023-10-16T14:35:28.922271+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:56.764459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.823761+0000","last_change":"2023-10-16T14:35:26.799576+0000","last_active":"2023-10-16T14:35:27.823761+0000","last_peered":"2023-10-16T14:35:27.823761+0000","last_clean":"2023-10-16T14:35:27.823761+0000","last_became_active":"2023-10-16T14:35:26.799156+0000","last_became_peered":"2023-10-16T14:35:26.799156+0000","last_unstale":"2023-10-16T14:35:27.823761+0000","last_undegraded":"2023-10-16T14:35:27.823761+0000","last_fullsized":"2023-10-16T14:35:27.823761+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:59.413398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820736+0000","last_change":"2023-10-16T14:35:27.813368+0000","last_active":"2023-10-16T14:35:28.820736+0000","last_peered":"2023-10-16T14:35:28.820736+0000","last_clean":"2023-10-16T14:35:28.820736+0000","last_became_active":"2023-10-16T14:35:27.812418+0000","last_became_peered":"2023-10-16T14:35:27.812418+0000","last_unstale":"2023-10-16T14:35:28.820736+0000","last_undegraded":"2023-10-16T14:35:28.820736+0000","last_fullsized":"2023-10-16T14:35:28.820736+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:11.265954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"126'3","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823098+0000","last_change":"2023-10-16T14:35:26.792383+0000","last_active":"2023-10-16T14:35:28.823098+0000","last_peered":"2023-10-16T14:35:28.823098+0000","last_clean":"2023-10-16T14:35:28.823098+0000","last_became_active":"2023-10-16T14:35:26.792023+0000","last_became_peered":"2023-10-16T14:35:26.792023+0000","last_unstale":"2023-10-16T14:35:28.823098+0000","last_undegraded":"2023-10-16T14:35:28.823098+0000","last_fullsized":"2023-10-16T14:35:28.823098+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:12.786390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822939+0000","last_change":"2023-10-16T14:35:27.819653+0000","last_active":"2023-10-16T14:35:28.822939+0000","last_peered":"2023-10-16T14:35:28.822939+0000","last_clean":"2023-10-16T14:35:28.822939+0000","last_became_active":"2023-10-16T14:35:27.818418+0000","last_became_peered":"2023-10-16T14:35:27.818418+0000","last_unstale":"2023-10-16T14:35:28.822939+0000","last_undegraded":"2023-10-16T14:35:28.822939+0000","last_fullsized":"2023-10-16T14:35:28.822939+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:38.502814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"126'6","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922381+0000","last_change":"2023-10-16T14:35:26.794979+0000","last_active":"2023-10-16T14:35:28.922381+0000","last_peered":"2023-10-16T14:35:28.922381+0000","last_clean":"2023-10-16T14:35:28.922381+0000","last_became_active":"2023-10-16T14:35:26.794646+0000","last_became_peered":"2023-10-16T14:35:26.794646+0000","last_unstale":"2023-10-16T14:35:28.922381+0000","last_undegraded":"2023-10-16T14:35:28.922381+0000","last_fullsized":"2023-10-16T14:35:28.922381+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:03:25.923587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.832351+0000","last_change":"2023-10-16T14:35:27.828658+0000","last_active":"2023-10-16T14:35:27.832351+0000","last_peered":"2023-10-16T14:35:27.832351+0000","last_clean":"2023-10-16T14:35:27.832351+0000","last_became_active":"2023-10-16T14:35:27.828364+0000","last_became_peered":"2023-10-16T14:35:27.828364+0000","last_unstale":"2023-10-16T14:35:27.832351+0000","last_undegraded":"2023-10-16T14:35:27.832351+0000","last_fullsized":"2023-10-16T14:35:27.832351+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:55.404409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"126'2","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.843221+0000","last_change":"2023-10-16T14:35:26.795772+0000","last_active":"2023-10-16T14:35:27.843221+0000","last_peered":"2023-10-16T14:35:27.843221+0000","last_clean":"2023-10-16T14:35:27.843221+0000","last_became_active":"2023-10-16T14:35:26.795482+0000","last_became_peered":"2023-10-16T14:35:26.795482+0000","last_unstale":"2023-10-16T14:35:27.843221+0000","last_undegraded":"2023-10-16T14:35:27.843221+0000","last_fullsized":"2023-10-16T14:35:27.843221+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:41.689212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816571+0000","last_change":"2023-10-16T14:35:27.827455+0000","last_active":"2023-10-16T14:35:28.816571+0000","last_peered":"2023-10-16T14:35:28.816571+0000","last_clean":"2023-10-16T14:35:28.816571+0000","last_became_active":"2023-10-16T14:35:27.827188+0000","last_became_peered":"2023-10-16T14:35:27.827188+0000","last_unstale":"2023-10-16T14:35:28.816571+0000","last_undegraded":"2023-10-16T14:35:28.816571+0000","last_fullsized":"2023-10-16T14:35:28.816571+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:56.927052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.790966+0000","last_change":"2023-10-16T14:35:26.797117+0000","last_active":"2023-10-16T14:35:27.790966+0000","last_peered":"2023-10-16T14:35:27.790966+0000","last_clean":"2023-10-16T14:35:27.790966+0000","last_became_active":"2023-10-16T14:35:26.796769+0000","last_became_peered":"2023-10-16T14:35:26.796769+0000","last_unstale":"2023-10-16T14:35:27.790966+0000","last_undegraded":"2023-10-16T14:35:27.790966+0000","last_fullsized":"2023-10-16T14:35:27.790966+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:49.435538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921249+0000","last_change":"2023-10-16T14:35:27.819740+0000","last_active":"2023-10-16T14:35:28.921249+0000","last_peered":"2023-10-16T14:35:28.921249+0000","last_clean":"2023-10-16T14:35:28.921249+0000","last_became_active":"2023-10-16T14:35:27.819386+0000","last_became_peered":"2023-10-16T14:35:27.819386+0000","last_unstale":"2023-10-16T14:35:28.921249+0000","last_undegraded":"2023-10-16T14:35:28.921249+0000","last_fullsized":"2023-10-16T14:35:28.921249+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:36.933549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921440+0000","last_change":"2023-10-16T14:35:26.796589+0000","last_active":"2023-10-16T14:35:28.921440+0000","last_peered":"2023-10-16T14:35:28.921440+0000","last_clean":"2023-10-16T14:35:28.921440+0000","last_became_active":"2023-10-16T14:35:26.796266+0000","last_became_peered":"2023-10-16T14:35:26.796266+0000","last_unstale":"2023-10-16T14:35:28.921440+0000","last_undegraded":"2023-10-16T14:35:28.921440+0000","last_fullsized":"2023-10-16T14:35:28.921440+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:28.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.819201+0000","last_change":"2023-10-16T14:35:27.811373+0000","last_active":"2023-10-16T14:35:27.819201+0000","last_peered":"2023-10-16T14:35:27.819201+0000","last_clean":"2023-10-16T14:35:27.819201+0000","last_became_active":"2023-10-16T14:35:27.811033+0000","last_became_peered":"2023-10-16T14:35:27.811033+0000","last_unstale":"2023-10-16T14:35:27.819201+0000","last_undegraded":"2023-10-16T14:35:27.819201+0000","last_fullsized":"2023-10-16T14:35:27.819201+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:55.523264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"126'8","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816076+0000","last_change":"2023-10-16T14:35:26.793651+0000","last_active":"2023-10-16T14:35:28.816076+0000","last_peered":"2023-10-16T14:35:28.816076+0000","last_clean":"2023-10-16T14:35:28.816076+0000","last_became_active":"2023-10-16T14:35:26.793176+0000","last_became_peered":"2023-10-16T14:35:26.793176+0000","last_unstale":"2023-10-16T14:35:28.816076+0000","last_undegraded":"2023-10-16T14:35:28.816076+0000","last_fullsized":"2023-10-16T14:35:28.816076+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:08.874304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.821332+0000","last_change":"2023-10-16T14:35:27.823806+0000","last_active":"2023-10-16T14:35:28.821332+0000","last_peered":"2023-10-16T14:35:28.821332+0000","last_clean":"2023-10-16T14:35:28.821332+0000","last_became_active":"2023-10-16T14:35:27.823524+0000","last_became_peered":"2023-10-16T14:35:27.823524+0000","last_unstale":"2023-10-16T14:35:28.821332+0000","last_undegraded":"2023-10-16T14:35:28.821332+0000","last_fullsized":"2023-10-16T14:35:28.821332+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:00:45.000191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921731+0000","last_change":"2023-10-16T14:35:26.786569+0000","last_active":"2023-10-16T14:35:28.921731+0000","last_peered":"2023-10-16T14:35:28.921731+0000","last_clean":"2023-10-16T14:35:28.921731+0000","last_became_active":"2023-10-16T14:35:26.786272+0000","last_became_peered":"2023-10-16T14:35:26.786272+0000","last_unstale":"2023-10-16T14:35:28.921731+0000","last_undegraded":"2023-10-16T14:35:28.921731+0000","last_fullsized":"2023-10-16T14:35:28.921731+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:47.918700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.824089+0000","last_change":"2023-10-16T14:35:27.822150+0000","last_active":"2023-10-16T14:35:27.824089+0000","last_peered":"2023-10-16T14:35:27.824089+0000","last_clean":"2023-10-16T14:35:27.824089+0000","last_became_active":"2023-10-16T14:35:27.821865+0000","last_became_peered":"2023-10-16T14:35:27.821865+0000","last_unstale":"2023-10-16T14:35:27.824089+0000","last_undegraded":"2023-10-16T14:35:27.824089+0000","last_fullsized":"2023-10-16T14:35:27.824089+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:42.967527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820540+0000","last_change":"2023-10-16T14:35:26.796155+0000","last_active":"2023-10-16T14:35:28.820540+0000","last_peered":"2023-10-16T14:35:28.820540+0000","last_clean":"2023-10-16T14:35:28.820540+0000","last_became_active":"2023-10-16T14:35:26.795883+0000","last_became_peered":"2023-10-16T14:35:26.795883+0000","last_unstale":"2023-10-16T14:35:28.820540+0000","last_undegraded":"2023-10-16T14:35:28.820540+0000","last_fullsized":"2023-10-16T14:35:28.820540+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:04.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823138+0000","last_change":"2023-10-16T14:35:27.821641+0000","last_active":"2023-10-16T14:35:28.823138+0000","last_peered":"2023-10-16T14:35:28.823138+0000","last_clean":"2023-10-16T14:35:28.823138+0000","last_became_active":"2023-10-16T14:35:27.820974+0000","last_became_peered":"2023-10-16T14:35:27.820974+0000","last_unstale":"2023-10-16T14:35:28.823138+0000","last_undegraded":"2023-10-16T14:35:28.823138+0000","last_fullsized":"2023-10-16T14:35:28.823138+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:01.726571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921579+0000","last_change":"2023-10-16T14:35:26.789671+0000","last_active":"2023-10-16T14:35:28.921579+0000","last_peered":"2023-10-16T14:35:28.921579+0000","last_clean":"2023-10-16T14:35:28.921579+0000","last_became_active":"2023-10-16T14:35:26.789378+0000","last_became_peered":"2023-10-16T14:35:26.789378+0000","last_unstale":"2023-10-16T14:35:28.921579+0000","last_undegraded":"2023-10-16T14:35:28.921579+0000","last_fullsized":"2023-10-16T14:35:28.921579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:14.044911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.838219+0000","last_change":"2023-10-16T14:35:27.833883+0000","last_active":"2023-10-16T14:35:27.838219+0000","last_peered":"2023-10-16T14:35:27.838219+0000","last_clean":"2023-10-16T14:35:27.838219+0000","last_became_active":"2023-10-16T14:35:27.833516+0000","last_became_peered":"2023-10-16T14:35:27.833516+0000","last_unstale":"2023-10-16T14:35:27.838219+0000","last_undegraded":"2023-10-16T14:35:27.838219+0000","last_fullsized":"2023-10-16T14:35:27.838219+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.871682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921129+0000","last_change":"2023-10-16T14:35:26.787500+0000","last_active":"2023-10-16T14:35:28.921129+0000","last_peered":"2023-10-16T14:35:28.921129+0000","last_clean":"2023-10-16T14:35:28.921129+0000","last_became_active":"2023-10-16T14:35:26.787220+0000","last_became_peered":"2023-10-16T14:35:26.787220+0000","last_unstale":"2023-10-16T14:35:28.921129+0000","last_undegraded":"2023-10-16T14:35:28.921129+0000","last_fullsized":"2023-10-16T14:35:28.921129+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:54.511680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.824926+0000","last_change":"2023-10-16T14:35:27.822660+0000","last_active":"2023-10-16T14:35:27.824926+0000","last_peered":"2023-10-16T14:35:27.824926+0000","last_clean":"2023-10-16T14:35:27.824926+0000","last_became_active":"2023-10-16T14:35:27.813998+0000","last_became_peered":"2023-10-16T14:35:27.813998+0000","last_unstale":"2023-10-16T14:35:27.824926+0000","last_undegraded":"2023-10-16T14:35:27.824926+0000","last_fullsized":"2023-10-16T14:35:27.824926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:01.021987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.812091+0000","last_change":"2023-10-16T14:35:26.796677+0000","last_active":"2023-10-16T14:35:27.812091+0000","last_peered":"2023-10-16T14:35:27.812091+0000","last_clean":"2023-10-16T14:35:27.812091+0000","last_became_active":"2023-10-16T14:35:26.796379+0000","last_became_peered":"2023-10-16T14:35:26.796379+0000","last_unstale":"2023-10-16T14:35:27.812091+0000","last_undegraded":"2023-10-16T14:35:27.812091+0000","last_fullsized":"2023-10-16T14:35:27.812091+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:25.281210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.837720+0000","last_change":"2023-10-16T14:35:27.829731+0000","last_active":"2023-10-16T14:35:27.837720+0000","last_peered":"2023-10-16T14:35:27.837720+0000","last_clean":"2023-10-16T14:35:27.837720+0000","last_became_active":"2023-10-16T14:35:27.829446+0000","last_became_peered":"2023-10-16T14:35:27.829446+0000","last_unstale":"2023-10-16T14:35:27.837720+0000","last_undegraded":"2023-10-16T14:35:27.837720+0000","last_fullsized":"2023-10-16T14:35:27.837720+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:44.257924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.809164+0000","last_change":"2023-10-16T14:35:26.791017+0000","last_active":"2023-10-16T14:35:27.809164+0000","last_peered":"2023-10-16T14:35:27.809164+0000","last_clean":"2023-10-16T14:35:27.809164+0000","last_became_active":"2023-10-16T14:35:26.790676+0000","last_became_peered":"2023-10-16T14:35:26.790676+0000","last_unstale":"2023-10-16T14:35:27.809164+0000","last_undegraded":"2023-10-16T14:35:27.809164+0000","last_fullsized":"2023-10-16T14:35:27.809164+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:56.252738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.819716+0000","last_change":"2023-10-16T14:35:27.817891+0000","last_active":"2023-10-16T14:35:28.819716+0000","last_peered":"2023-10-16T14:35:28.819716+0000","last_clean":"2023-10-16T14:35:28.819716+0000","last_became_active":"2023-10-16T14:35:27.817532+0000","last_became_peered":"2023-10-16T14:35:27.817532+0000","last_unstale":"2023-10-16T14:35:28.819716+0000","last_undegraded":"2023-10-16T14:35:28.819716+0000","last_fullsized":"2023-10-16T14:35:28.819716+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.575493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920836+0000","last_change":"2023-10-16T14:35:26.787566+0000","last_active":"2023-10-16T14:35:28.920836+0000","last_peered":"2023-10-16T14:35:28.920836+0000","last_clean":"2023-10-16T14:35:28.920836+0000","last_became_active":"2023-10-16T14:35:26.787230+0000","last_became_peered":"2023-10-16T14:35:26.787230+0000","last_unstale":"2023-10-16T14:35:28.920836+0000","last_undegraded":"2023-10-16T14:35:28.920836+0000","last_fullsized":"2023-10-16T14:35:28.920836+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:57.995030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922265+0000","last_change":"2023-10-16T14:35:27.823065+0000","last_active":"2023-10-16T14:35:28.922265+0000","last_peered":"2023-10-16T14:35:28.922265+0000","last_clean":"2023-10-16T14:35:28.922265+0000","last_became_active":"2023-10-16T14:35:27.822798+0000","last_became_peered":"2023-10-16T14:35:27.822798+0000","last_unstale":"2023-10-16T14:35:28.922265+0000","last_undegraded":"2023-10-16T14:35:28.922265+0000","last_fullsized":"2023-10-16T14:35:28.922265+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:46.935993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"126'2","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.836760+0000","last_change":"2023-10-16T14:35:26.792930+0000","last_active":"2023-10-16T14:35:27.836760+0000","last_peered":"2023-10-16T14:35:27.836760+0000","last_clean":"2023-10-16T14:35:27.836760+0000","last_became_active":"2023-10-16T14:35:26.791961+0000","last_became_peered":"2023-10-16T14:35:26.791961+0000","last_unstale":"2023-10-16T14:35:27.836760+0000","last_undegraded":"2023-10-16T14:35:27.836760+0000","last_fullsized":"2023-10-16T14:35:27.836760+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:58.145335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816942+0000","last_change":"2023-10-16T14:35:27.825481+0000","last_active":"2023-10-16T14:35:28.816942+0000","last_peered":"2023-10-16T14:35:28.816942+0000","last_clean":"2023-10-16T14:35:28.816942+0000","last_became_active":"2023-10-16T14:35:27.825211+0000","last_became_peered":"2023-10-16T14:35:27.825211+0000","last_unstale":"2023-10-16T14:35:28.816942+0000","last_undegraded":"2023-10-16T14:35:28.816942+0000","last_fullsized":"2023-10-16T14:35:28.816942+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:11.265954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922380+0000","last_change":"2023-10-16T14:35:26.791918+0000","last_active":"2023-10-16T14:35:28.922380+0000","last_peered":"2023-10-16T14:35:28.922380+0000","last_clean":"2023-10-16T14:35:28.922380+0000","last_became_active":"2023-10-16T14:35:26.785641+0000","last_became_peered":"2023-10-16T14:35:26.785641+0000","last_unstale":"2023-10-16T14:35:28.922380+0000","last_undegraded":"2023-10-16T14:35:28.922380+0000","last_fullsized":"2023-10-16T14:35:28.922380+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:04.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816762+0000","last_change":"2023-10-16T14:35:27.813315+0000","last_active":"2023-10-16T14:35:28.816762+0000","last_peered":"2023-10-16T14:35:28.816762+0000","last_clean":"2023-10-16T14:35:28.816762+0000","last_became_active":"2023-10-16T14:35:27.813026+0000","last_became_peered":"2023-10-16T14:35:27.813026+0000","last_unstale":"2023-10-16T14:35:28.816762+0000","last_undegraded":"2023-10-16T14:35:28.816762+0000","last_fullsized":"2023-10-16T14:35:28.816762+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:12.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921545+0000","last_change":"2023-10-16T14:35:26.780074+0000","last_active":"2023-10-16T14:35:28.921545+0000","last_peered":"2023-10-16T14:35:28.921545+0000","last_clean":"2023-10-16T14:35:28.921545+0000","last_became_active":"2023-10-16T14:35:26.779707+0000","last_became_peered":"2023-10-16T14:35:26.779707+0000","last_unstale":"2023-10-16T14:35:28.921545+0000","last_undegraded":"2023-10-16T14:35:28.921545+0000","last_fullsized":"2023-10-16T14:35:28.921545+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:00.010403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820809+0000","last_change":"2023-10-16T14:35:27.835016+0000","last_active":"2023-10-16T14:35:28.820809+0000","last_peered":"2023-10-16T14:35:28.820809+0000","last_clean":"2023-10-16T14:35:28.820809+0000","last_became_active":"2023-10-16T14:35:27.833615+0000","last_became_peered":"2023-10-16T14:35:27.833615+0000","last_unstale":"2023-10-16T14:35:28.820809+0000","last_undegraded":"2023-10-16T14:35:28.820809+0000","last_fullsized":"2023-10-16T14:35:28.820809+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:33:41.377382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820951+0000","last_change":"2023-10-16T14:35:26.795023+0000","last_active":"2023-10-16T14:35:28.820951+0000","last_peered":"2023-10-16T14:35:28.820951+0000","last_clean":"2023-10-16T14:35:28.820951+0000","last_became_active":"2023-10-16T14:35:26.794718+0000","last_became_peered":"2023-10-16T14:35:26.794718+0000","last_unstale":"2023-10-16T14:35:28.820951+0000","last_undegraded":"2023-10-16T14:35:28.820951+0000","last_fullsized":"2023-10-16T14:35:28.820951+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:08.874304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.829957+0000","last_change":"2023-10-16T14:35:27.822294+0000","last_active":"2023-10-16T14:35:27.829957+0000","last_peered":"2023-10-16T14:35:27.829957+0000","last_clean":"2023-10-16T14:35:27.829957+0000","last_became_active":"2023-10-16T14:35:27.821931+0000","last_became_peered":"2023-10-16T14:35:27.821931+0000","last_unstale":"2023-10-16T14:35:27.829957+0000","last_undegraded":"2023-10-16T14:35:27.829957+0000","last_fullsized":"2023-10-16T14:35:27.829957+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:09.885888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921232+0000","last_change":"2023-10-16T14:35:26.788099+0000","last_active":"2023-10-16T14:35:28.921232+0000","last_peered":"2023-10-16T14:35:28.921232+0000","last_clean":"2023-10-16T14:35:28.921232+0000","last_became_active":"2023-10-16T14:35:26.787835+0000","last_became_peered":"2023-10-16T14:35:26.787835+0000","last_unstale":"2023-10-16T14:35:28.921232+0000","last_undegraded":"2023-10-16T14:35:28.921232+0000","last_fullsized":"2023-10-16T14:35:28.921232+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:26.006973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921929+0000","last_change":"2023-10-16T14:35:27.816861+0000","last_active":"2023-10-16T14:35:28.921929+0000","last_peered":"2023-10-16T14:35:28.921929+0000","last_clean":"2023-10-16T14:35:28.921929+0000","last_became_active":"2023-10-16T14:35:27.816555+0000","last_became_peered":"2023-10-16T14:35:27.816555+0000","last_unstale":"2023-10-16T14:35:28.921929+0000","last_undegraded":"2023-10-16T14:35:28.921929+0000","last_fullsized":"2023-10-16T14:35:28.921929+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:15.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.819882+0000","last_change":"2023-10-16T14:35:26.798514+0000","last_active":"2023-10-16T14:35:28.819882+0000","last_peered":"2023-10-16T14:35:28.819882+0000","last_clean":"2023-10-16T14:35:28.819882+0000","last_became_active":"2023-10-16T14:35:26.798125+0000","last_became_peered":"2023-10-16T14:35:26.798125+0000","last_unstale":"2023-10-16T14:35:28.819882+0000","last_undegraded":"2023-10-16T14:35:28.819882+0000","last_fullsized":"2023-10-16T14:35:28.819882+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:32.281222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.828155+0000","last_change":"2023-10-16T14:35:27.824256+0000","last_active":"2023-10-16T14:35:27.828155+0000","last_peered":"2023-10-16T14:35:27.828155+0000","last_clean":"2023-10-16T14:35:27.828155+0000","last_became_active":"2023-10-16T14:35:27.823070+0000","last_became_peered":"2023-10-16T14:35:27.823070+0000","last_unstale":"2023-10-16T14:35:27.828155+0000","last_undegraded":"2023-10-16T14:35:27.828155+0000","last_fullsized":"2023-10-16T14:35:27.828155+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:15.056495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.811650+0000","last_change":"2023-10-16T14:35:26.787307+0000","last_active":"2023-10-16T14:35:27.811650+0000","last_peered":"2023-10-16T14:35:27.811650+0000","last_clean":"2023-10-16T14:35:27.811650+0000","last_became_active":"2023-10-16T14:35:26.787011+0000","last_became_peered":"2023-10-16T14:35:26.787011+0000","last_unstale":"2023-10-16T14:35:27.811650+0000","last_undegraded":"2023-10-16T14:35:27.811650+0000","last_fullsized":"2023-10-16T14:35:27.811650+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.575406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.819770+0000","last_change":"2023-10-16T14:35:27.834344+0000","last_active":"2023-10-16T14:35:28.819770+0000","last_peered":"2023-10-16T14:35:28.819770+0000","last_clean":"2023-10-16T14:35:28.819770+0000","last_became_active":"2023-10-16T14:35:27.833984+0000","last_became_peered":"2023-10-16T14:35:27.833984+0000","last_unstale":"2023-10-16T14:35:28.819770+0000","last_undegraded":"2023-10-16T14:35:28.819770+0000","last_fullsized":"2023-10-16T14:35:28.819770+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:12.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.790306+0000","last_change":"2023-10-16T14:35:26.781471+0000","last_active":"2023-10-16T14:35:27.790306+0000","last_peered":"2023-10-16T14:35:27.790306+0000","last_clean":"2023-10-16T14:35:27.790306+0000","last_became_active":"2023-10-16T14:35:26.780245+0000","last_became_peered":"2023-10-16T14:35:26.780245+0000","last_unstale":"2023-10-16T14:35:27.790306+0000","last_undegraded":"2023-10-16T14:35:27.790306+0000","last_fullsized":"2023-10-16T14:35:27.790306+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:28.207356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.838494+0000","last_change":"2023-10-16T14:35:27.832606+0000","last_active":"2023-10-16T14:35:27.838494+0000","last_peered":"2023-10-16T14:35:27.838494+0000","last_clean":"2023-10-16T14:35:27.838494+0000","last_became_active":"2023-10-16T14:35:27.832165+0000","last_became_peered":"2023-10-16T14:35:27.832165+0000","last_unstale":"2023-10-16T14:35:27.838494+0000","last_undegraded":"2023-10-16T14:35:27.838494+0000","last_fullsized":"2023-10-16T14:35:27.838494+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:27.094258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823117+0000","last_change":"2023-10-16T14:35:26.794753+0000","last_active":"2023-10-16T14:35:28.823117+0000","last_peered":"2023-10-16T14:35:28.823117+0000","last_clean":"2023-10-16T14:35:28.823117+0000","last_became_active":"2023-10-16T14:35:26.794446+0000","last_became_peered":"2023-10-16T14:35:26.794446+0000","last_unstale":"2023-10-16T14:35:28.823117+0000","last_undegraded":"2023-10-16T14:35:28.823117+0000","last_fullsized":"2023-10-16T14:35:28.823117+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:12.262849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"115'208","reported_seq":603,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.792629+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:35:27.792629+0000","last_peered":"2023-10-16T14:35:27.792629+0000","last_clean":"2023-10-16T14:35:27.792629+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:35:27.792629+0000","last_undegraded":"2023-10-16T14:35:27.792629+0000","last_fullsized":"2023-10-16T14:35:27.792629+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575076,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2158592,"data_stored":1341655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.091,"15min":1.145},"min":{"1min":0.75,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5089999999999999,"5min":1.833,"15min":1.833},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.095,"15min":1.149},"min":{"1min":0.93799999999999994,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3049999999999999,"5min":1.891,"15min":1.891},"last":0.84999999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.82199999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.325,"5min":1.2390000000000001,"15min":1.2430000000000001},"min":{"1min":0.96699999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6850000000000001,"5min":2.048,"15min":2.048},"last":0.68799999999999994}]},{"osd":2,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.2709999999999999,"15min":1.3080000000000001},"min":{"1min":1.016,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9339999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.246,"15min":1.2370000000000001},"min":{"1min":1.073,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.4079999999999999,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.2669999999999999,"15min":1.302},"min":{"1min":0.82899999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.339,"15min":1.3700000000000001},"min":{"1min":0.98799999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.804,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.73099999999999998}]},{"osd":4,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.3720000000000001,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.495,"15min":0.495},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.274,"15min":1.2669999999999999},"min":{"1min":0.81000000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3779999999999999,"15min":1.409},"min":{"1min":0.93899999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.0950000000000002,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.482,"15min":1.498},"min":{"1min":1.286,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.97,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.4450000000000001,"15min":1.4430000000000001},"min":{"1min":1.1220000000000001,"5min":0.502,"15min":0.502},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.516,"15min":1.522},"min":{"1min":1.294,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575075,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36880,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334960,"statfs":{"total":96636764160,"available":96598999040,"internally_reserved":0,"allocated":1576960,"data_stored":758127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1950000000000001,"15min":1.1839999999999999},"min":{"1min":0.79000000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.571,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2549999999999999,"15min":1.252},"min":{"1min":0.89100000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6579999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":0.998}]},{"osd":1,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.296,"15min":1.282},"min":{"1min":0.89200000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7849999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3140000000000001,"15min":1.302},"min":{"1min":0.94999999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.679,"5min":10.241,"15min":10.241},"last":1.1160000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.155,"15min":1.202},"min":{"1min":0.95199999999999996,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5860000000000001,"5min":1.708,"15min":2.4940000000000002},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.387,"5min":1.337,"15min":1.3580000000000001},"min":{"1min":1.006,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9059999999999999,"5min":5.694,"15min":5.694},"last":1.2669999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.355,"15min":1.3520000000000001},"min":{"1min":1.012,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5580000000000001,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.296,"15min":1.329},"min":{"1min":0.95799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.8620000000000001,"5min":2.2349999999999999,"15min":2.6120000000000001},"last":1.0860000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.409,"15min":1.329},"min":{"1min":0.90100000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.9930000000000001,"5min":13.826000000000001,"15min":13.826000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.367,"15min":1.3680000000000001},"min":{"1min":0.81999999999999995,"5min":0.44,"15min":0.44},"max":{"1min":1.742,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.4139999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3149999999999999,"15min":1.3129999999999999},"min":{"1min":1,"5min":0.439,"15min":0.439},"max":{"1min":1.7789999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":1.327},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5169999999999999,"15min":1.4950000000000001},"min":{"1min":1.204,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.0819999999999999,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.6140000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.575,"15min":1.5620000000000001},"min":{"1min":1.109,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.831,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.829},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.526,"15min":1.542},"min":{"1min":1.194,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.946,"5min":2.1779999999999999,"15min":2.7250000000000001},"last":1.6559999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575075,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":751245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.107,"15min":1.1259999999999999},"min":{"1min":0.79700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.669,"5min":1.669,"15min":2.2309999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.171,"15min":1.1819999999999999},"min":{"1min":0.76200000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.7210000000000001,"5min":1.7729999999999999,"15min":1.8340000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.119,"15min":1.1699999999999999},"min":{"1min":0.95699999999999996,"5min":0.622,"15min":0.622},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.726},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.2370000000000001},"min":{"1min":0.92300000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.526,"5min":1.704,"15min":1.7809999999999999},"last":0.876}]},{"osd":2,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.202,"15min":1.246},"min":{"1min":0.93400000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.569,"5min":1.913,"15min":2.5009999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":1.3500000000000001},"min":{"1min":1.097,"5min":0.625,"15min":0.625},"max":{"1min":1.6779999999999999,"5min":2.0049999999999999,"15min":2.1200000000000001},"last":0.78200000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2470000000000001,"15min":1.28},"min":{"1min":0.997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.696,"5min":1.903,"15min":1.903},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.381,"5min":1.298,"15min":1.377},"min":{"1min":1.143,"5min":0.78400000000000003,"15min":0.76100000000000001},"max":{"1min":1.7549999999999999,"5min":1.8420000000000001,"15min":2.8029999999999999},"last":0.92800000000000005}]},{"osd":4,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.284,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.587,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4359999999999999,"15min":1.4990000000000001},"min":{"1min":0.82899999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.883,"5min":1.96,"15min":4.9480000000000004},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.345,"15min":1.3839999999999999},"min":{"1min":0.91900000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.7250000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.363,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.798,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":0.90500000000000003}]},{"osd":7,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4890000000000001,"15min":1.5529999999999999},"min":{"1min":0.95199999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.95,"5min":2.1200000000000001,"15min":2.641},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5649999999999999,"15min":1.609},"min":{"1min":1.298,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.726,"5min":2.1909999999999998,"15min":2.2010000000000001},"last":0.97599999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575076,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46624,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325216,"statfs":{"total":96636764160,"available":96589021184,"internally_reserved":0,"allocated":1589248,"data_stored":759211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.137,"15min":1.1579999999999999},"min":{"1min":0.93799999999999994,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.0339999999999998},"last":2.0259999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.071,"15min":1.1579999999999999},"min":{"1min":0.65700000000000003,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.653,"5min":1.7589999999999999,"15min":1.9199999999999999},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.117,"15min":1.177},"min":{"1min":0.70199999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.53},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.2070000000000001,"15min":1.23},"min":{"1min":0.97599999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.704,"5min":1.778,"15min":2.3690000000000002},"last":1.929}]},{"osd":2,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3640000000000001,"15min":1.399},"min":{"1min":1.0649999999999999,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.863,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.671},{"interface":"front","average":{"1min":1.383,"5min":1.3140000000000001,"15min":1.3580000000000001},"min":{"1min":0.94799999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.6990000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1379999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2569999999999999,"15min":1.3109999999999999},"min":{"1min":0.75900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":1.728,"5min":1.95,"15min":1.95},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.262,"15min":1.3080000000000001},"min":{"1min":0.78900000000000003,"5min":0.437,"15min":0.437},"max":{"1min":1.984,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.8200000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3089999999999999,"15min":1.2969999999999999},"min":{"1min":0.74299999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":2.113},{"interface":"front","average":{"1min":1.395,"5min":1.3560000000000001,"15min":1.3540000000000001},"min":{"1min":0.92700000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8440000000000001,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.228}]},{"osd":6,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.343,"15min":1.3080000000000001},"min":{"1min":0.86699999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":2.069},{"interface":"front","average":{"1min":1.516,"5min":1.476,"15min":1.464},"min":{"1min":0.998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.149,"5min":2.1960000000000002,"15min":2.2890000000000001},"last":2.198}]},{"osd":7,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.484,"15min":1.4770000000000001},"min":{"1min":1.0860000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.486,"15min":1.468},"min":{"1min":1.135,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.214,"5min":2.214,"15min":2.214},"last":1.968}]}]},{"osd":3,"up_from":13,"seq":55834575075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1572864,"data_stored":752809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.83799999999999997,"15min":0.88200000000000001},"min":{"1min":0.67300000000000004,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.5,"5min":1.696,"15min":1.696},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.121,"5min":0.93100000000000005,"15min":1.0209999999999999},"min":{"1min":0.80300000000000005,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.651,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.01}]},{"osd":1,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.88500000000000001,"15min":0.93999999999999995},"min":{"1min":0.73299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4730000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.149,"5min":1.0580000000000001,"15min":1.131},"min":{"1min":0.70799999999999996,"5min":0.314,"15min":0.314},"max":{"1min":1.736,"5min":3.024,"15min":3.024},"last":1.3400000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.1519999999999999,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.853,"5min":3.3420000000000001,"15min":3.3420000000000001},"last":1.27},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.2849999999999999,"15min":1.405},"min":{"1min":1.1419999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.75,"5min":2.5609999999999999,"15min":2.6299999999999999},"last":1.198}]},{"osd":4,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1970000000000001,"15min":1.256},"min":{"1min":0.96799999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.6990000000000001,"5min":2.02,"15min":2.02},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2689999999999999,"15min":1.331},"min":{"1min":0.85099999999999998,"5min":0.629,"15min":0.629},"max":{"1min":1.6419999999999999,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.4810000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.4350000000000001,"15min":1.518},"min":{"1min":1.036,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8620000000000001,"5min":3.206,"15min":3.206},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.385,"15min":1.452},"min":{"1min":1.026,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8380000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.7250000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.4019999999999999},"min":{"1min":1.002,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7549999999999999,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.155},{"interface":"front","average":{"1min":1.421,"5min":1.3859999999999999,"15min":1.4830000000000001},"min":{"1min":1.1020000000000001,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.9279999999999999,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.7629999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.49,"15min":1.577},"min":{"1min":1.3839999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.911,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4039999999999999,"15min":1.5329999999999999},"min":{"1min":1.125,"5min":0.629,"15min":0.629},"max":{"1min":1.869,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.526}]}]},{"osd":2,"up_from":13,"seq":55834575076,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46592,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325248,"statfs":{"total":96636764160,"available":96589053952,"internally_reserved":0,"allocated":1560576,"data_stored":751209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.874,"15min":0.874},"min":{"1min":0.58799999999999997,"5min":0.38,"15min":0.38},"max":{"1min":1.631,"5min":1.631,"15min":1.7150000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.97499999999999998,"15min":1.0249999999999999},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.32600000000000001},"max":{"1min":1.4910000000000001,"5min":1.585,"15min":2.3260000000000001},"last":1.127}]},{"osd":1,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.98999999999999999,"15min":1.0009999999999999},"min":{"1min":0.55400000000000005,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.399,"5min":3.234,"15min":3.234},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.141,"15min":1.1619999999999999},"min":{"1min":0.69399999999999995,"5min":0.51200000000000001,"15min":0.42199999999999999},"max":{"1min":1.827,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.5880000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.143,"15min":1.232},"min":{"1min":0.61299999999999999,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":1.6779999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.747},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.2789999999999999,"15min":1.3049999999999999},"min":{"1min":0.84899999999999998,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.8660000000000001,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.4810000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.409,"15min":1.4259999999999999},"min":{"1min":1.123,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8009999999999999,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.748},{"interface":"front","average":{"1min":1.282,"5min":1.29,"15min":1.343},"min":{"1min":0.94199999999999995,"5min":0.77400000000000002,"15min":0.72599999999999998},"max":{"1min":1.6559999999999999,"5min":2.9359999999999999,"15min":3.016},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.375,"15min":1.411},"min":{"1min":0.90100000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.819,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.635},{"interface":"front","average":{"1min":1.532,"5min":1.524,"15min":1.5489999999999999},"min":{"1min":0.92300000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.8380000000000001,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.677}]},{"osd":6,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3919999999999999,"15min":1.4650000000000001},"min":{"1min":0.873,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.732,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.244},{"interface":"front","average":{"1min":1.47,"5min":1.4910000000000001,"15min":1.524},"min":{"1min":1.0009999999999999,"5min":0.71299999999999997,"15min":0.65300000000000002},"max":{"1min":1.7629999999999999,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":1.399}]},{"osd":7,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.478,"15min":1.542},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.69099999999999995},"max":{"1min":1.788,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.29},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5860000000000001,"15min":1.633},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.80300000000000005},"max":{"1min":1.9750000000000001,"5min":2.7519999999999998,"15min":2.7519999999999998},"last":1.71}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1556480,"data_stored":751175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.872,"15min":0.90000000000000002},"min":{"1min":0.63200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.7749999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0149999999999999,"15min":1.0940000000000001},"min":{"1min":0.67100000000000004,"5min":0.378,"15min":0.378},"max":{"1min":1.6140000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.996}]},{"osd":2,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92300000000000004,"15min":0.98099999999999998},"min":{"1min":0.58599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.546,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.167,"5min":1.0780000000000001,"15min":1.155},"min":{"1min":0.56899999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.597,"5min":1.756,"15min":1.796},"last":1.1499999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.21,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.853,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.218},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.135,"15min":1.1930000000000001},"min":{"1min":0.51900000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6619999999999999,"5min":1.79,"15min":1.855},"last":1.111}]},{"osd":4,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2050000000000001,"15min":1.282},"min":{"1min":0.89800000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6679999999999999,"5min":2.069,"15min":2.069},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3440000000000001,"15min":1.4159999999999999},"min":{"1min":1.113,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8009999999999999,"5min":2.899,"15min":2.899},"last":1.6559999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3380000000000001,"15min":1.431},"min":{"1min":0.78500000000000003,"5min":0.623,"15min":0.623},"max":{"1min":1.6299999999999999,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.337,"15min":1.4219999999999999},"min":{"1min":1.024,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7330000000000001,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.482,"15min":1.544},"min":{"1min":1.1419999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.923,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.427,"15min":1.5329999999999999},"min":{"1min":0.88600000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6799999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.77}]},{"osd":7,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5489999999999999,"15min":1.625},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.74,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.804},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.448,"15min":1.55},"min":{"1min":1.143,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7869999999999999,"5min":2.2000000000000002,"15min":2.262},"last":1.3759999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37448,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334392,"statfs":{"total":96636764160,"available":96598417408,"internally_reserved":0,"allocated":2154496,"data_stored":1342125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.89200000000000002,"15min":0.94399999999999995},"min":{"1min":0.66700000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2569999999999999,"5min":1.5880000000000001,"15min":1.7729999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.119,"15min":1.169},"min":{"1min":0.94599999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.52,"5min":3.1120000000000001,"15min":3.1120000000000001},"last":1.4970000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.212,"15min":1.254},"min":{"1min":0.78400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.752,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1699999999999999,"15min":1.21},"min":{"1min":0.82399999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6220000000000001,"5min":1.847,"15min":2.1070000000000002},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.96599999999999997,"15min":1.004},"min":{"1min":0.80800000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.609,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.753},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.23},"min":{"1min":0.84399999999999997,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":1.512,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":1.097}]},{"osd":4,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.264,"15min":1.3089999999999999},"min":{"1min":1.044,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.012,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":2.3849999999999998},{"interface":"front","average":{"1min":1.48,"5min":1.454,"15min":1.4730000000000001},"min":{"1min":1.103,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.0569999999999999,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":2.4300000000000002}]},{"osd":5,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3819999999999999,"15min":1.431},"min":{"1min":1.012,"5min":0.79000000000000004,"15min":0.78400000000000003},"max":{"1min":1.7909999999999999,"5min":2.1930000000000001,"15min":2.5880000000000001},"last":1.069},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3240000000000001,"15min":1.371},"min":{"1min":0.91500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.895,"5min":3.7069999999999999,"15min":3.7069999999999999},"last":1.1599999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.575,"15min":1.589},"min":{"1min":1.2849999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9330000000000001,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.454,"15min":1.504},"min":{"1min":1.0980000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.958,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":2.093}]},{"osd":7,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.534,"15min":1.591},"min":{"1min":1.2649999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.117,"5min":2.335,"15min":2.6509999999999998},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.514,"15min":1.5549999999999999},"min":{"1min":1.071,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.206,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":1.6910000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:35:32.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:35:32.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.989+0000 7fbfb973c640 1 Processor -- start 2023-10-16T14:35:32.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.990+0000 7fbfb973c640 1 -- start start 2023-10-16T14:35:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.991+0000 7fbfb973c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfb40b2600 0x7fbfb40ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.991+0000 7fbfb973c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbfb4070240 0x7fbfb40706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.991+0000 7fbfb973c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40b2a60 con 0x7fbfb415aaa0 2023-10-16T14:35:32.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb973c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb4070c10 con 0x7fbfb40b2600 2023-10-16T14:35:32.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb973c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb40b2d00 con 0x7fbfb4070240 2023-10-16T14:35:32.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfb40b2600 0x7fbfb40ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfb40b2600 0x7fbfb40ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52046/0 (socket says 172.21.15.29:52046) 2023-10-16T14:35:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3806203618 learned_addr learned my addr 172.21.15.29:0/3806203618 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb8f3b640 1 --2- 172.21.15.29:0/3806203618 >> v2:172.21.15.170:3300/0 conn(0x7fbfb4070240 0x7fbfb40706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3806203618 >> v2:172.21.15.170:3300/0 conn(0x7fbfb4070240 msgr2=0x7fbfb40706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 --2- 172.21.15.29:0/3806203618 >> v2:172.21.15.170:3300/0 conn(0x7fbfb4070240 0x7fbfb40706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:32.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3806203618 >> v1:172.21.15.170:6789/0 conn(0x7fbfb415aaa0 legacy=0x7fbfb40b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3806203618 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40b2fe0 con 0x7fbfb40b2600 2023-10-16T14:35:32.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.992+0000 7fbfb8f3b640 1 --2- 172.21.15.29:0/3806203618 >> v2:172.21.15.170:3300/0 conn(0x7fbfb4070240 0x7fbfb40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:33.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.993+0000 7fbfb37fe640 1 --2- 172.21.15.29:0/3806203618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb40ca8a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fbfb40af1c0 tx=0x7fbfa802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2dcada080c8e66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.994+0000 7fbfb2ffd640 1 -- 172.21.15.29:0/3806203618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80317e0 con 0x7fbfb40b2600 2023-10-16T14:35:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.994+0000 7fbfb2ffd640 1 -- 172.21.15.29:0/3806203618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8031940 con 0x7fbfb40b2600 2023-10-16T14:35:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.994+0000 7fbfb2ffd640 1 -- 172.21.15.29:0/3806203618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8031cd0 con 0x7fbfb40b2600 2023-10-16T14:35:33.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.994+0000 7fbfb973c640 1 -- 172.21.15.29:0/3806203618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 msgr2=0x7fbfb40ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.994+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3806203618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb40ca8a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fbfb40af1c0 tx=0x7fbfa802b880 comp rx=0 tx=0).stop 2023-10-16T14:35:33.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.995+0000 7fbfb973c640 1 -- 172.21.15.29:0/3806203618 shutdown_connections 2023-10-16T14:35:33.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.995+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3806203618 >> v2:172.21.15.170:3300/0 conn(0x7fbfb4070240 0x7fbfb40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.995+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3806203618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb40ca8a0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.995+0000 7fbfb973c640 1 -- 172.21.15.29:0/3806203618 >> 172.21.15.29:0/3806203618 conn(0x7fbfb40cc230 msgr2=0x7fbfb40cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.995+0000 7fbfb973c640 1 -- 172.21.15.29:0/3806203618 shutdown_connections 2023-10-16T14:35:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.995+0000 7fbfb973c640 1 -- 172.21.15.29:0/3806203618 wait complete. 2023-10-16T14:35:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.996+0000 7fbfb973c640 1 Processor -- start 2023-10-16T14:35:33.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.997+0000 7fbfb973c640 1 -- start start 2023-10-16T14:35:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb973c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb4070240 0x7fbfb41f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb973c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb41f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb973c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb415aaa0 0x7fbfb40c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb973c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb41f3e00 con 0x7fbfb40b2600 2023-10-16T14:35:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb973c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb41f1230 con 0x7fbfb415aaa0 2023-10-16T14:35:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb973c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb41f4740 con 0x7fbfb4070240 2023-10-16T14:35:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb41f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb41f0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52052/0 (socket says 172.21.15.29:52052) 2023-10-16T14:35:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3432870475 learned_addr learned my addr 172.21.15.29:0/3432870475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb3fff640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb4070240 0x7fbfb41f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb8f3b640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb415aaa0 0x7fbfb40c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb4070240 msgr2=0x7fbfb41f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb4070240 0x7fbfb41f4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb415aaa0 msgr2=0x7fbfb40c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb415aaa0 0x7fbfb40c4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.998+0000 7fbfb37fe640 1 -- 172.21.15.29:0/3432870475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80094d0 con 0x7fbfb40b2600 2023-10-16T14:35:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.999+0000 7fbfb37fe640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb41f0300 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fbfa802b8c0 tx=0x7fbfa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.999+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8032880 con 0x7fbfb40b2600 2023-10-16T14:35:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.999+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa80329e0 con 0x7fbfb40b2600 2023-10-16T14:35:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.999+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8032d90 con 0x7fbfb40b2600 2023-10-16T14:35:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.999+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb41f0c70 con 0x7fbfb40b2600 2023-10-16T14:35:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:32.999+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb41f3a20 con 0x7fbfb40b2600 2023-10-16T14:35:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.000+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb404e9b0 con 0x7fbfb40b2600 2023-10-16T14:35:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.001+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbfa80453f0 con 0x7fbfb40b2600 2023-10-16T14:35:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.002+0000 7fbfb0ff9640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbfa006af90 0x7fbfa006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.002+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfa80abd30 con 0x7fbfb40b2600 2023-10-16T14:35:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.003+0000 7fbfb3fff640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbfa006af90 0x7fbfa006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.004+0000 7fbfb3fff640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbfa006af90 0x7fbfa006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4001db0 tx=0x7fbfa4009040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.007+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbfa8076bb0 con 0x7fbfb40b2600 2023-10-16T14:35:33.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.198+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 --> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfb40b35a0 con 0x7fbfa006af90 2023-10-16T14:35:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.221+0000 7fbfb0ff9640 1 -- 172.21.15.29:0/3432870475 <== mgr.6406 v2:172.21.15.170:6832/1801486769 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286932 (secure 0 0 0) 0x7fbfa4001a50 con 0x7fbfa006af90 2023-10-16T14:35:33.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:35:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbfa006af90 msgr2=0x7fbfa006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbfa006af90 0x7fbfa006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4001db0 tx=0x7fbfa4009040 comp rx=0 tx=0).stop 2023-10-16T14:35:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 msgr2=0x7fbfb41f0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb41f0300 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fbfa802b8c0 tx=0x7fbfa80046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 shutdown_connections 2023-10-16T14:35:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fbfa006af90 0x7fbfa006d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb40b2600 0x7fbfb41f0300 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbfb415aaa0 0x7fbfb40c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 --2- 172.21.15.29:0/3432870475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbfb4070240 0x7fbfb41f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 >> 172.21.15.29:0/3432870475 conn(0x7fbfb40cc230 msgr2=0x7fbfb40b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 shutdown_connections 2023-10-16T14:35:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.228+0000 7fbfb973c640 1 -- 172.21.15.29:0/3432870475 wait complete. 2023-10-16T14:35:33.254 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T14:35:32.568236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":367,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":350304,"kb_used_data":13408,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754624416,"statfs":{"total":773094113280,"available":772735401984,"internally_reserved":0,"allocated":13729792,"data_stored":7207556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":344837689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002244"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816905+0000","last_change":"2023-10-16T14:35:27.830951+0000","last_active":"2023-10-16T14:35:28.816905+0000","last_peered":"2023-10-16T14:35:28.816905+0000","last_clean":"2023-10-16T14:35:28.816905+0000","last_became_active":"2023-10-16T14:35:27.830681+0000","last_became_peered":"2023-10-16T14:35:27.830681+0000","last_unstale":"2023-10-16T14:35:28.816905+0000","last_undegraded":"2023-10-16T14:35:28.816905+0000","last_fullsized":"2023-10-16T14:35:28.816905+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:13.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921665+0000","last_change":"2023-10-16T14:35:27.827055+0000","last_active":"2023-10-16T14:35:28.921665+0000","last_peered":"2023-10-16T14:35:28.921665+0000","last_clean":"2023-10-16T14:35:28.921665+0000","last_became_active":"2023-10-16T14:35:27.826772+0000","last_became_peered":"2023-10-16T14:35:27.826772+0000","last_unstale":"2023-10-16T14:35:28.921665+0000","last_undegraded":"2023-10-16T14:35:28.921665+0000","last_fullsized":"2023-10-16T14:35:28.921665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.752080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816762+0000","last_change":"2023-10-16T14:35:27.831784+0000","last_active":"2023-10-16T14:35:28.816762+0000","last_peered":"2023-10-16T14:35:28.816762+0000","last_clean":"2023-10-16T14:35:28.816762+0000","last_became_active":"2023-10-16T14:35:27.831479+0000","last_became_peered":"2023-10-16T14:35:27.831479+0000","last_unstale":"2023-10-16T14:35:28.816762+0000","last_undegraded":"2023-10-16T14:35:28.816762+0000","last_fullsized":"2023-10-16T14:35:28.816762+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.575493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.840331+0000","last_change":"2023-10-16T14:35:27.833085+0000","last_active":"2023-10-16T14:35:27.840331+0000","last_peered":"2023-10-16T14:35:27.840331+0000","last_clean":"2023-10-16T14:35:27.840331+0000","last_became_active":"2023-10-16T14:35:27.832572+0000","last_became_peered":"2023-10-16T14:35:27.832572+0000","last_unstale":"2023-10-16T14:35:27.840331+0000","last_undegraded":"2023-10-16T14:35:27.840331+0000","last_fullsized":"2023-10-16T14:35:27.840331+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:48.160322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820612+0000","last_change":"2023-10-16T14:35:27.827982+0000","last_active":"2023-10-16T14:35:28.820612+0000","last_peered":"2023-10-16T14:35:28.820612+0000","last_clean":"2023-10-16T14:35:28.820612+0000","last_became_active":"2023-10-16T14:35:27.827430+0000","last_became_peered":"2023-10-16T14:35:27.827430+0000","last_unstale":"2023-10-16T14:35:28.820612+0000","last_undegraded":"2023-10-16T14:35:28.820612+0000","last_fullsized":"2023-10-16T14:35:28.820612+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:13.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822512+0000","last_change":"2023-10-16T14:35:27.816233+0000","last_active":"2023-10-16T14:35:28.822512+0000","last_peered":"2023-10-16T14:35:28.822512+0000","last_clean":"2023-10-16T14:35:28.822512+0000","last_became_active":"2023-10-16T14:35:27.815961+0000","last_became_peered":"2023-10-16T14:35:27.815961+0000","last_unstale":"2023-10-16T14:35:28.822512+0000","last_undegraded":"2023-10-16T14:35:28.822512+0000","last_fullsized":"2023-10-16T14:35:28.822512+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:44.909816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822128+0000","last_change":"2023-10-16T14:35:27.815743+0000","last_active":"2023-10-16T14:35:28.822128+0000","last_peered":"2023-10-16T14:35:28.822128+0000","last_clean":"2023-10-16T14:35:28.822128+0000","last_became_active":"2023-10-16T14:35:27.815455+0000","last_became_peered":"2023-10-16T14:35:27.815455+0000","last_unstale":"2023-10-16T14:35:28.822128+0000","last_undegraded":"2023-10-16T14:35:28.822128+0000","last_fullsized":"2023-10-16T14:35:28.822128+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:48.573039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920587+0000","last_change":"2023-10-16T14:35:27.816981+0000","last_active":"2023-10-16T14:35:28.920587+0000","last_peered":"2023-10-16T14:35:28.920587+0000","last_clean":"2023-10-16T14:35:28.920587+0000","last_became_active":"2023-10-16T14:35:27.815791+0000","last_became_peered":"2023-10-16T14:35:27.815791+0000","last_unstale":"2023-10-16T14:35:28.920587+0000","last_undegraded":"2023-10-16T14:35:28.920587+0000","last_fullsized":"2023-10-16T14:35:28.920587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:18.609488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922268+0000","last_change":"2023-10-16T14:35:27.835255+0000","last_active":"2023-10-16T14:35:28.922268+0000","last_peered":"2023-10-16T14:35:28.922268+0000","last_clean":"2023-10-16T14:35:28.922268+0000","last_became_active":"2023-10-16T14:35:27.833530+0000","last_became_peered":"2023-10-16T14:35:27.833530+0000","last_unstale":"2023-10-16T14:35:28.922268+0000","last_undegraded":"2023-10-16T14:35:28.922268+0000","last_fullsized":"2023-10-16T14:35:28.922268+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:42.775538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820082+0000","last_change":"2023-10-16T14:35:27.814037+0000","last_active":"2023-10-16T14:35:28.820082+0000","last_peered":"2023-10-16T14:35:28.820082+0000","last_clean":"2023-10-16T14:35:28.820082+0000","last_became_active":"2023-10-16T14:35:27.813731+0000","last_became_peered":"2023-10-16T14:35:27.813731+0000","last_unstale":"2023-10-16T14:35:28.820082+0000","last_undegraded":"2023-10-16T14:35:28.820082+0000","last_fullsized":"2023-10-16T14:35:28.820082+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.752080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822598+0000","last_change":"2023-10-16T14:35:27.821820+0000","last_active":"2023-10-16T14:35:28.822598+0000","last_peered":"2023-10-16T14:35:28.822598+0000","last_clean":"2023-10-16T14:35:28.822598+0000","last_became_active":"2023-10-16T14:35:27.821352+0000","last_became_peered":"2023-10-16T14:35:27.821352+0000","last_unstale":"2023-10-16T14:35:28.822598+0000","last_undegraded":"2023-10-16T14:35:28.822598+0000","last_fullsized":"2023-10-16T14:35:28.822598+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:25.043951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921708+0000","last_change":"2023-10-16T14:35:27.832760+0000","last_active":"2023-10-16T14:35:28.921708+0000","last_peered":"2023-10-16T14:35:28.921708+0000","last_clean":"2023-10-16T14:35:28.921708+0000","last_became_active":"2023-10-16T14:35:27.832402+0000","last_became_peered":"2023-10-16T14:35:27.832402+0000","last_unstale":"2023-10-16T14:35:28.921708+0000","last_undegraded":"2023-10-16T14:35:28.921708+0000","last_fullsized":"2023-10-16T14:35:28.921708+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:22.707752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.840403+0000","last_change":"2023-10-16T14:35:27.833431+0000","last_active":"2023-10-16T14:35:27.840403+0000","last_peered":"2023-10-16T14:35:27.840403+0000","last_clean":"2023-10-16T14:35:27.840403+0000","last_became_active":"2023-10-16T14:35:27.833085+0000","last_became_peered":"2023-10-16T14:35:27.833085+0000","last_unstale":"2023-10-16T14:35:27.840403+0000","last_undegraded":"2023-10-16T14:35:27.840403+0000","last_fullsized":"2023-10-16T14:35:27.840403+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:05.767929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.815682+0000","last_change":"2023-10-16T14:35:27.807995+0000","last_active":"2023-10-16T14:35:27.815682+0000","last_peered":"2023-10-16T14:35:27.815682+0000","last_clean":"2023-10-16T14:35:27.815682+0000","last_became_active":"2023-10-16T14:35:27.805721+0000","last_became_peered":"2023-10-16T14:35:27.805721+0000","last_unstale":"2023-10-16T14:35:27.815682+0000","last_undegraded":"2023-10-16T14:35:27.815682+0000","last_fullsized":"2023-10-16T14:35:27.815682+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:19.752829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.837378+0000","last_change":"2023-10-16T14:35:27.832078+0000","last_active":"2023-10-16T14:35:27.837378+0000","last_peered":"2023-10-16T14:35:27.837378+0000","last_clean":"2023-10-16T14:35:27.837378+0000","last_became_active":"2023-10-16T14:35:27.831781+0000","last_became_peered":"2023-10-16T14:35:27.831781+0000","last_unstale":"2023-10-16T14:35:27.837378+0000","last_undegraded":"2023-10-16T14:35:27.837378+0000","last_fullsized":"2023-10-16T14:35:27.837378+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:51.132955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820387+0000","last_change":"2023-10-16T14:35:27.820597+0000","last_active":"2023-10-16T14:35:28.820387+0000","last_peered":"2023-10-16T14:35:28.820387+0000","last_clean":"2023-10-16T14:35:28.820387+0000","last_became_active":"2023-10-16T14:35:27.819037+0000","last_became_peered":"2023-10-16T14:35:27.819037+0000","last_unstale":"2023-10-16T14:35:28.820387+0000","last_undegraded":"2023-10-16T14:35:28.820387+0000","last_fullsized":"2023-10-16T14:35:28.820387+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:08.504041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922193+0000","last_change":"2023-10-16T14:35:27.830245+0000","last_active":"2023-10-16T14:35:28.922193+0000","last_peered":"2023-10-16T14:35:28.922193+0000","last_clean":"2023-10-16T14:35:28.922193+0000","last_became_active":"2023-10-16T14:35:27.829932+0000","last_became_peered":"2023-10-16T14:35:27.829932+0000","last_unstale":"2023-10-16T14:35:28.922193+0000","last_undegraded":"2023-10-16T14:35:28.922193+0000","last_fullsized":"2023-10-16T14:35:28.922193+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:33:41.377382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.842356+0000","last_change":"2023-10-16T14:35:27.839397+0000","last_active":"2023-10-16T14:35:27.842356+0000","last_peered":"2023-10-16T14:35:27.842356+0000","last_clean":"2023-10-16T14:35:27.842356+0000","last_became_active":"2023-10-16T14:35:27.838836+0000","last_became_peered":"2023-10-16T14:35:27.838836+0000","last_unstale":"2023-10-16T14:35:27.842356+0000","last_undegraded":"2023-10-16T14:35:27.842356+0000","last_fullsized":"2023-10-16T14:35:27.842356+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:41.296118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.823927+0000","last_change":"2023-10-16T14:35:27.817745+0000","last_active":"2023-10-16T14:35:27.823927+0000","last_peered":"2023-10-16T14:35:27.823927+0000","last_clean":"2023-10-16T14:35:27.823927+0000","last_became_active":"2023-10-16T14:35:27.816201+0000","last_became_peered":"2023-10-16T14:35:27.816201+0000","last_unstale":"2023-10-16T14:35:27.823927+0000","last_undegraded":"2023-10-16T14:35:27.823927+0000","last_fullsized":"2023-10-16T14:35:27.823927+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:33.292806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922086+0000","last_change":"2023-10-16T14:35:27.836038+0000","last_active":"2023-10-16T14:35:28.922086+0000","last_peered":"2023-10-16T14:35:28.922086+0000","last_clean":"2023-10-16T14:35:28.922086+0000","last_became_active":"2023-10-16T14:35:27.835440+0000","last_became_peered":"2023-10-16T14:35:27.835440+0000","last_unstale":"2023-10-16T14:35:28.922086+0000","last_undegraded":"2023-10-16T14:35:28.922086+0000","last_fullsized":"2023-10-16T14:35:28.922086+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:46.695154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.822932+0000","last_change":"2023-10-16T14:35:27.816668+0000","last_active":"2023-10-16T14:35:27.822932+0000","last_peered":"2023-10-16T14:35:27.822932+0000","last_clean":"2023-10-16T14:35:27.822932+0000","last_became_active":"2023-10-16T14:35:27.816364+0000","last_became_peered":"2023-10-16T14:35:27.816364+0000","last_unstale":"2023-10-16T14:35:27.822932+0000","last_undegraded":"2023-10-16T14:35:27.822932+0000","last_fullsized":"2023-10-16T14:35:27.822932+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:21:08.121776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920718+0000","last_change":"2023-10-16T14:35:27.818095+0000","last_active":"2023-10-16T14:35:28.920718+0000","last_peered":"2023-10-16T14:35:28.920718+0000","last_clean":"2023-10-16T14:35:28.920718+0000","last_became_active":"2023-10-16T14:35:27.817264+0000","last_became_peered":"2023-10-16T14:35:27.817264+0000","last_unstale":"2023-10-16T14:35:28.920718+0000","last_undegraded":"2023-10-16T14:35:28.920718+0000","last_fullsized":"2023-10-16T14:35:28.920718+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:56.927052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920836+0000","last_change":"2023-10-16T14:35:27.820763+0000","last_active":"2023-10-16T14:35:28.920836+0000","last_peered":"2023-10-16T14:35:28.920836+0000","last_clean":"2023-10-16T14:35:28.920836+0000","last_became_active":"2023-10-16T14:35:27.819076+0000","last_became_peered":"2023-10-16T14:35:27.819076+0000","last_unstale":"2023-10-16T14:35:28.920836+0000","last_undegraded":"2023-10-16T14:35:28.920836+0000","last_fullsized":"2023-10-16T14:35:28.920836+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.575493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822027+0000","last_change":"2023-10-16T14:35:27.815116+0000","last_active":"2023-10-16T14:35:28.822027+0000","last_peered":"2023-10-16T14:35:28.822027+0000","last_clean":"2023-10-16T14:35:28.822027+0000","last_became_active":"2023-10-16T14:35:27.814786+0000","last_became_peered":"2023-10-16T14:35:27.814786+0000","last_unstale":"2023-10-16T14:35:28.822027+0000","last_undegraded":"2023-10-16T14:35:28.822027+0000","last_fullsized":"2023-10-16T14:35:28.822027+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:04.120607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820984+0000","last_change":"2023-10-16T14:35:27.823185+0000","last_active":"2023-10-16T14:35:28.820984+0000","last_peered":"2023-10-16T14:35:28.820984+0000","last_clean":"2023-10-16T14:35:28.820984+0000","last_became_active":"2023-10-16T14:35:27.822845+0000","last_became_peered":"2023-10-16T14:35:27.822845+0000","last_unstale":"2023-10-16T14:35:28.820984+0000","last_undegraded":"2023-10-16T14:35:28.820984+0000","last_fullsized":"2023-10-16T14:35:28.820984+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:46.935993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921928+0000","last_change":"2023-10-16T14:35:27.836048+0000","last_active":"2023-10-16T14:35:28.921928+0000","last_peered":"2023-10-16T14:35:28.921928+0000","last_clean":"2023-10-16T14:35:28.921928+0000","last_became_active":"2023-10-16T14:35:27.835614+0000","last_became_peered":"2023-10-16T14:35:27.835614+0000","last_unstale":"2023-10-16T14:35:28.921928+0000","last_undegraded":"2023-10-16T14:35:28.921928+0000","last_fullsized":"2023-10-16T14:35:28.921928+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:13.388689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822743+0000","last_change":"2023-10-16T14:35:27.809671+0000","last_active":"2023-10-16T14:35:28.822743+0000","last_peered":"2023-10-16T14:35:28.822743+0000","last_clean":"2023-10-16T14:35:28.822743+0000","last_became_active":"2023-10-16T14:35:27.809123+0000","last_became_peered":"2023-10-16T14:35:27.809123+0000","last_unstale":"2023-10-16T14:35:28.822743+0000","last_undegraded":"2023-10-16T14:35:28.822743+0000","last_fullsized":"2023-10-16T14:35:28.822743+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:33.189682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822112+0000","last_change":"2023-10-16T14:35:27.834295+0000","last_active":"2023-10-16T14:35:28.822112+0000","last_peered":"2023-10-16T14:35:28.822112+0000","last_clean":"2023-10-16T14:35:28.822112+0000","last_became_active":"2023-10-16T14:35:27.833973+0000","last_became_peered":"2023-10-16T14:35:27.833973+0000","last_unstale":"2023-10-16T14:35:28.822112+0000","last_undegraded":"2023-10-16T14:35:28.822112+0000","last_fullsized":"2023-10-16T14:35:28.822112+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:59.436838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921202+0000","last_change":"2023-10-16T14:35:27.821621+0000","last_active":"2023-10-16T14:35:28.921202+0000","last_peered":"2023-10-16T14:35:28.921202+0000","last_clean":"2023-10-16T14:35:28.921202+0000","last_became_active":"2023-10-16T14:35:27.821329+0000","last_became_peered":"2023-10-16T14:35:27.821329+0000","last_unstale":"2023-10-16T14:35:28.921202+0000","last_undegraded":"2023-10-16T14:35:28.921202+0000","last_fullsized":"2023-10-16T14:35:28.921202+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:12.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.822181+0000","last_change":"2023-10-16T14:35:27.816836+0000","last_active":"2023-10-16T14:35:27.822181+0000","last_peered":"2023-10-16T14:35:27.822181+0000","last_clean":"2023-10-16T14:35:27.822181+0000","last_became_active":"2023-10-16T14:35:27.816499+0000","last_became_peered":"2023-10-16T14:35:27.816499+0000","last_unstale":"2023-10-16T14:35:27.822181+0000","last_undegraded":"2023-10-16T14:35:27.822181+0000","last_fullsized":"2023-10-16T14:35:27.822181+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:27.018557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.827635+0000","last_change":"2023-10-16T14:35:27.826183+0000","last_active":"2023-10-16T14:35:27.827635+0000","last_peered":"2023-10-16T14:35:27.827635+0000","last_clean":"2023-10-16T14:35:27.827635+0000","last_became_active":"2023-10-16T14:35:27.825868+0000","last_became_peered":"2023-10-16T14:35:27.825868+0000","last_unstale":"2023-10-16T14:35:27.827635+0000","last_undegraded":"2023-10-16T14:35:27.827635+0000","last_fullsized":"2023-10-16T14:35:27.827635+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:52.173443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.815929+0000","last_change":"2023-10-16T14:35:27.818655+0000","last_active":"2023-10-16T14:35:28.815929+0000","last_peered":"2023-10-16T14:35:28.815929+0000","last_clean":"2023-10-16T14:35:28.815929+0000","last_became_active":"2023-10-16T14:35:27.818345+0000","last_became_peered":"2023-10-16T14:35:27.818345+0000","last_unstale":"2023-10-16T14:35:28.815929+0000","last_undegraded":"2023-10-16T14:35:28.815929+0000","last_fullsized":"2023-10-16T14:35:28.815929+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:18.609488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.814308+0000","last_change":"2023-10-16T14:35:27.805559+0000","last_active":"2023-10-16T14:35:27.814308+0000","last_peered":"2023-10-16T14:35:27.814308+0000","last_clean":"2023-10-16T14:35:27.814308+0000","last_became_active":"2023-10-16T14:35:27.803828+0000","last_became_peered":"2023-10-16T14:35:27.803828+0000","last_unstale":"2023-10-16T14:35:27.814308+0000","last_undegraded":"2023-10-16T14:35:27.814308+0000","last_fullsized":"2023-10-16T14:35:27.814308+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:05.964248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"126'4","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922092+0000","last_change":"2023-10-16T14:35:26.795297+0000","last_active":"2023-10-16T14:35:28.922092+0000","last_peered":"2023-10-16T14:35:28.922092+0000","last_clean":"2023-10-16T14:35:28.922092+0000","last_became_active":"2023-10-16T14:35:26.794799+0000","last_became_peered":"2023-10-16T14:35:26.794799+0000","last_unstale":"2023-10-16T14:35:28.922092+0000","last_undegraded":"2023-10-16T14:35:28.922092+0000","last_fullsized":"2023-10-16T14:35:28.922092+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:49.435538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.835148+0000","last_change":"2023-10-16T14:35:27.829039+0000","last_active":"2023-10-16T14:35:27.835148+0000","last_peered":"2023-10-16T14:35:27.835148+0000","last_clean":"2023-10-16T14:35:27.835148+0000","last_became_active":"2023-10-16T14:35:27.826166+0000","last_became_peered":"2023-10-16T14:35:27.826166+0000","last_unstale":"2023-10-16T14:35:27.835148+0000","last_undegraded":"2023-10-16T14:35:27.835148+0000","last_fullsized":"2023-10-16T14:35:27.835148+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:14.351097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823006+0000","last_change":"2023-10-16T14:35:26.788980+0000","last_active":"2023-10-16T14:35:28.823006+0000","last_peered":"2023-10-16T14:35:28.823006+0000","last_clean":"2023-10-16T14:35:28.823006+0000","last_became_active":"2023-10-16T14:35:26.788451+0000","last_became_peered":"2023-10-16T14:35:26.788451+0000","last_unstale":"2023-10-16T14:35:28.823006+0000","last_undegraded":"2023-10-16T14:35:28.823006+0000","last_fullsized":"2023-10-16T14:35:28.823006+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:09.729246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816852+0000","last_change":"2023-10-16T14:35:27.832378+0000","last_active":"2023-10-16T14:35:28.816852+0000","last_peered":"2023-10-16T14:35:28.816852+0000","last_clean":"2023-10-16T14:35:28.816852+0000","last_became_active":"2023-10-16T14:35:27.832103+0000","last_became_peered":"2023-10-16T14:35:27.832103+0000","last_unstale":"2023-10-16T14:35:28.816852+0000","last_undegraded":"2023-10-16T14:35:28.816852+0000","last_fullsized":"2023-10-16T14:35:28.816852+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:27.752080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921405+0000","last_change":"2023-10-16T14:35:26.791016+0000","last_active":"2023-10-16T14:35:28.921405+0000","last_peered":"2023-10-16T14:35:28.921405+0000","last_clean":"2023-10-16T14:35:28.921405+0000","last_became_active":"2023-10-16T14:35:26.790715+0000","last_became_peered":"2023-10-16T14:35:26.790715+0000","last_unstale":"2023-10-16T14:35:28.921405+0000","last_undegraded":"2023-10-16T14:35:28.921405+0000","last_fullsized":"2023-10-16T14:35:28.921405+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:18.741245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921993+0000","last_change":"2023-10-16T14:35:27.818278+0000","last_active":"2023-10-16T14:35:28.921993+0000","last_peered":"2023-10-16T14:35:28.921993+0000","last_clean":"2023-10-16T14:35:28.921993+0000","last_became_active":"2023-10-16T14:35:27.817919+0000","last_became_peered":"2023-10-16T14:35:27.817919+0000","last_unstale":"2023-10-16T14:35:28.921993+0000","last_undegraded":"2023-10-16T14:35:28.921993+0000","last_fullsized":"2023-10-16T14:35:28.921993+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:58.730925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921400+0000","last_change":"2023-10-16T14:35:26.781983+0000","last_active":"2023-10-16T14:35:28.921400+0000","last_peered":"2023-10-16T14:35:28.921400+0000","last_clean":"2023-10-16T14:35:28.921400+0000","last_became_active":"2023-10-16T14:35:26.780624+0000","last_became_peered":"2023-10-16T14:35:26.780624+0000","last_unstale":"2023-10-16T14:35:28.921400+0000","last_undegraded":"2023-10-16T14:35:28.921400+0000","last_fullsized":"2023-10-16T14:35:28.921400+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:21:07.110192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.827069+0000","last_change":"2023-10-16T14:35:27.825513+0000","last_active":"2023-10-16T14:35:27.827069+0000","last_peered":"2023-10-16T14:35:27.827069+0000","last_clean":"2023-10-16T14:35:27.827069+0000","last_became_active":"2023-10-16T14:35:27.824446+0000","last_became_peered":"2023-10-16T14:35:27.824446+0000","last_unstale":"2023-10-16T14:35:27.827069+0000","last_undegraded":"2023-10-16T14:35:27.827069+0000","last_fullsized":"2023-10-16T14:35:27.827069+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:44.951954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816616+0000","last_change":"2023-10-16T14:35:26.797232+0000","last_active":"2023-10-16T14:35:28.816616+0000","last_peered":"2023-10-16T14:35:28.816616+0000","last_clean":"2023-10-16T14:35:28.816616+0000","last_became_active":"2023-10-16T14:35:26.795817+0000","last_became_peered":"2023-10-16T14:35:26.795817+0000","last_unstale":"2023-10-16T14:35:28.816616+0000","last_undegraded":"2023-10-16T14:35:28.816616+0000","last_fullsized":"2023-10-16T14:35:28.816616+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:49.435538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822530+0000","last_change":"2023-10-16T14:35:27.806755+0000","last_active":"2023-10-16T14:35:28.822530+0000","last_peered":"2023-10-16T14:35:28.822530+0000","last_clean":"2023-10-16T14:35:28.822530+0000","last_became_active":"2023-10-16T14:35:27.803901+0000","last_became_peered":"2023-10-16T14:35:27.803901+0000","last_unstale":"2023-10-16T14:35:28.822530+0000","last_undegraded":"2023-10-16T14:35:28.822530+0000","last_fullsized":"2023-10-16T14:35:28.822530+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:18.356865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920775+0000","last_change":"2023-10-16T14:35:26.792286+0000","last_active":"2023-10-16T14:35:28.920775+0000","last_peered":"2023-10-16T14:35:28.920775+0000","last_clean":"2023-10-16T14:35:28.920775+0000","last_became_active":"2023-10-16T14:35:26.791957+0000","last_became_peered":"2023-10-16T14:35:26.791957+0000","last_unstale":"2023-10-16T14:35:28.920775+0000","last_undegraded":"2023-10-16T14:35:28.920775+0000","last_fullsized":"2023-10-16T14:35:28.920775+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:21.059833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.840749+0000","last_change":"2023-10-16T14:35:27.834920+0000","last_active":"2023-10-16T14:35:27.840749+0000","last_peered":"2023-10-16T14:35:27.840749+0000","last_clean":"2023-10-16T14:35:27.840749+0000","last_became_active":"2023-10-16T14:35:27.834643+0000","last_became_peered":"2023-10-16T14:35:27.834643+0000","last_unstale":"2023-10-16T14:35:27.840749+0000","last_undegraded":"2023-10-16T14:35:27.840749+0000","last_fullsized":"2023-10-16T14:35:27.840749+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:04.469840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820135+0000","last_change":"2023-10-16T14:35:26.798916+0000","last_active":"2023-10-16T14:35:28.820135+0000","last_peered":"2023-10-16T14:35:28.820135+0000","last_clean":"2023-10-16T14:35:28.820135+0000","last_became_active":"2023-10-16T14:35:26.798582+0000","last_became_peered":"2023-10-16T14:35:26.798582+0000","last_unstale":"2023-10-16T14:35:28.820135+0000","last_undegraded":"2023-10-16T14:35:28.820135+0000","last_fullsized":"2023-10-16T14:35:28.820135+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:58.860098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921287+0000","last_change":"2023-10-16T14:35:27.819114+0000","last_active":"2023-10-16T14:35:28.921287+0000","last_peered":"2023-10-16T14:35:28.921287+0000","last_clean":"2023-10-16T14:35:28.921287+0000","last_became_active":"2023-10-16T14:35:27.818427+0000","last_became_peered":"2023-10-16T14:35:27.818427+0000","last_unstale":"2023-10-16T14:35:28.921287+0000","last_undegraded":"2023-10-16T14:35:28.921287+0000","last_fullsized":"2023-10-16T14:35:28.921287+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:53.697631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816050+0000","last_change":"2023-10-16T14:35:26.793141+0000","last_active":"2023-10-16T14:35:28.816050+0000","last_peered":"2023-10-16T14:35:28.816050+0000","last_clean":"2023-10-16T14:35:28.816050+0000","last_became_active":"2023-10-16T14:35:26.791132+0000","last_became_peered":"2023-10-16T14:35:26.791132+0000","last_unstale":"2023-10-16T14:35:28.816050+0000","last_undegraded":"2023-10-16T14:35:28.816050+0000","last_fullsized":"2023-10-16T14:35:28.816050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:41.689212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922181+0000","last_change":"2023-10-16T14:35:27.829767+0000","last_active":"2023-10-16T14:35:28.922181+0000","last_peered":"2023-10-16T14:35:28.922181+0000","last_clean":"2023-10-16T14:35:28.922181+0000","last_became_active":"2023-10-16T14:35:27.829354+0000","last_became_peered":"2023-10-16T14:35:27.829354+0000","last_unstale":"2023-10-16T14:35:28.922181+0000","last_undegraded":"2023-10-16T14:35:28.922181+0000","last_fullsized":"2023-10-16T14:35:28.922181+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:11.265954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816958+0000","last_change":"2023-10-16T14:35:26.795925+0000","last_active":"2023-10-16T14:35:28.816958+0000","last_peered":"2023-10-16T14:35:28.816958+0000","last_clean":"2023-10-16T14:35:28.816958+0000","last_became_active":"2023-10-16T14:35:26.795636+0000","last_became_peered":"2023-10-16T14:35:26.795636+0000","last_unstale":"2023-10-16T14:35:28.816958+0000","last_undegraded":"2023-10-16T14:35:28.816958+0000","last_fullsized":"2023-10-16T14:35:28.816958+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:26.082674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922271+0000","last_change":"2023-10-16T14:35:27.817417+0000","last_active":"2023-10-16T14:35:28.922271+0000","last_peered":"2023-10-16T14:35:28.922271+0000","last_clean":"2023-10-16T14:35:28.922271+0000","last_became_active":"2023-10-16T14:35:27.817122+0000","last_became_peered":"2023-10-16T14:35:27.817122+0000","last_unstale":"2023-10-16T14:35:28.922271+0000","last_undegraded":"2023-10-16T14:35:28.922271+0000","last_fullsized":"2023-10-16T14:35:28.922271+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:56.764459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.823761+0000","last_change":"2023-10-16T14:35:26.799576+0000","last_active":"2023-10-16T14:35:27.823761+0000","last_peered":"2023-10-16T14:35:27.823761+0000","last_clean":"2023-10-16T14:35:27.823761+0000","last_became_active":"2023-10-16T14:35:26.799156+0000","last_became_peered":"2023-10-16T14:35:26.799156+0000","last_unstale":"2023-10-16T14:35:27.823761+0000","last_undegraded":"2023-10-16T14:35:27.823761+0000","last_fullsized":"2023-10-16T14:35:27.823761+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:59.413398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820736+0000","last_change":"2023-10-16T14:35:27.813368+0000","last_active":"2023-10-16T14:35:28.820736+0000","last_peered":"2023-10-16T14:35:28.820736+0000","last_clean":"2023-10-16T14:35:28.820736+0000","last_became_active":"2023-10-16T14:35:27.812418+0000","last_became_peered":"2023-10-16T14:35:27.812418+0000","last_unstale":"2023-10-16T14:35:28.820736+0000","last_undegraded":"2023-10-16T14:35:28.820736+0000","last_fullsized":"2023-10-16T14:35:28.820736+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:11.265954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"126'3","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823098+0000","last_change":"2023-10-16T14:35:26.792383+0000","last_active":"2023-10-16T14:35:28.823098+0000","last_peered":"2023-10-16T14:35:28.823098+0000","last_clean":"2023-10-16T14:35:28.823098+0000","last_became_active":"2023-10-16T14:35:26.792023+0000","last_became_peered":"2023-10-16T14:35:26.792023+0000","last_unstale":"2023-10-16T14:35:28.823098+0000","last_undegraded":"2023-10-16T14:35:28.823098+0000","last_fullsized":"2023-10-16T14:35:28.823098+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:12.786390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.822939+0000","last_change":"2023-10-16T14:35:27.819653+0000","last_active":"2023-10-16T14:35:28.822939+0000","last_peered":"2023-10-16T14:35:28.822939+0000","last_clean":"2023-10-16T14:35:28.822939+0000","last_became_active":"2023-10-16T14:35:27.818418+0000","last_became_peered":"2023-10-16T14:35:27.818418+0000","last_unstale":"2023-10-16T14:35:28.822939+0000","last_undegraded":"2023-10-16T14:35:28.822939+0000","last_fullsized":"2023-10-16T14:35:28.822939+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:48:38.502814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"126'6","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922381+0000","last_change":"2023-10-16T14:35:26.794979+0000","last_active":"2023-10-16T14:35:28.922381+0000","last_peered":"2023-10-16T14:35:28.922381+0000","last_clean":"2023-10-16T14:35:28.922381+0000","last_became_active":"2023-10-16T14:35:26.794646+0000","last_became_peered":"2023-10-16T14:35:26.794646+0000","last_unstale":"2023-10-16T14:35:28.922381+0000","last_undegraded":"2023-10-16T14:35:28.922381+0000","last_fullsized":"2023-10-16T14:35:28.922381+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:03:25.923587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.832351+0000","last_change":"2023-10-16T14:35:27.828658+0000","last_active":"2023-10-16T14:35:27.832351+0000","last_peered":"2023-10-16T14:35:27.832351+0000","last_clean":"2023-10-16T14:35:27.832351+0000","last_became_active":"2023-10-16T14:35:27.828364+0000","last_became_peered":"2023-10-16T14:35:27.828364+0000","last_unstale":"2023-10-16T14:35:27.832351+0000","last_undegraded":"2023-10-16T14:35:27.832351+0000","last_fullsized":"2023-10-16T14:35:27.832351+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:55.404409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"126'2","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.843221+0000","last_change":"2023-10-16T14:35:26.795772+0000","last_active":"2023-10-16T14:35:27.843221+0000","last_peered":"2023-10-16T14:35:27.843221+0000","last_clean":"2023-10-16T14:35:27.843221+0000","last_became_active":"2023-10-16T14:35:26.795482+0000","last_became_peered":"2023-10-16T14:35:26.795482+0000","last_unstale":"2023-10-16T14:35:27.843221+0000","last_undegraded":"2023-10-16T14:35:27.843221+0000","last_fullsized":"2023-10-16T14:35:27.843221+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:41.689212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816571+0000","last_change":"2023-10-16T14:35:27.827455+0000","last_active":"2023-10-16T14:35:28.816571+0000","last_peered":"2023-10-16T14:35:28.816571+0000","last_clean":"2023-10-16T14:35:28.816571+0000","last_became_active":"2023-10-16T14:35:27.827188+0000","last_became_peered":"2023-10-16T14:35:27.827188+0000","last_unstale":"2023-10-16T14:35:28.816571+0000","last_undegraded":"2023-10-16T14:35:28.816571+0000","last_fullsized":"2023-10-16T14:35:28.816571+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:56.927052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.790966+0000","last_change":"2023-10-16T14:35:26.797117+0000","last_active":"2023-10-16T14:35:27.790966+0000","last_peered":"2023-10-16T14:35:27.790966+0000","last_clean":"2023-10-16T14:35:27.790966+0000","last_became_active":"2023-10-16T14:35:26.796769+0000","last_became_peered":"2023-10-16T14:35:26.796769+0000","last_unstale":"2023-10-16T14:35:27.790966+0000","last_undegraded":"2023-10-16T14:35:27.790966+0000","last_fullsized":"2023-10-16T14:35:27.790966+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:49.435538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921249+0000","last_change":"2023-10-16T14:35:27.819740+0000","last_active":"2023-10-16T14:35:28.921249+0000","last_peered":"2023-10-16T14:35:28.921249+0000","last_clean":"2023-10-16T14:35:28.921249+0000","last_became_active":"2023-10-16T14:35:27.819386+0000","last_became_peered":"2023-10-16T14:35:27.819386+0000","last_unstale":"2023-10-16T14:35:28.921249+0000","last_undegraded":"2023-10-16T14:35:28.921249+0000","last_fullsized":"2023-10-16T14:35:28.921249+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:36.933549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921440+0000","last_change":"2023-10-16T14:35:26.796589+0000","last_active":"2023-10-16T14:35:28.921440+0000","last_peered":"2023-10-16T14:35:28.921440+0000","last_clean":"2023-10-16T14:35:28.921440+0000","last_became_active":"2023-10-16T14:35:26.796266+0000","last_became_peered":"2023-10-16T14:35:26.796266+0000","last_unstale":"2023-10-16T14:35:28.921440+0000","last_undegraded":"2023-10-16T14:35:28.921440+0000","last_fullsized":"2023-10-16T14:35:28.921440+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:28.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.819201+0000","last_change":"2023-10-16T14:35:27.811373+0000","last_active":"2023-10-16T14:35:27.819201+0000","last_peered":"2023-10-16T14:35:27.819201+0000","last_clean":"2023-10-16T14:35:27.819201+0000","last_became_active":"2023-10-16T14:35:27.811033+0000","last_became_peered":"2023-10-16T14:35:27.811033+0000","last_unstale":"2023-10-16T14:35:27.819201+0000","last_undegraded":"2023-10-16T14:35:27.819201+0000","last_fullsized":"2023-10-16T14:35:27.819201+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:55.523264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"126'8","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816076+0000","last_change":"2023-10-16T14:35:26.793651+0000","last_active":"2023-10-16T14:35:28.816076+0000","last_peered":"2023-10-16T14:35:28.816076+0000","last_clean":"2023-10-16T14:35:28.816076+0000","last_became_active":"2023-10-16T14:35:26.793176+0000","last_became_peered":"2023-10-16T14:35:26.793176+0000","last_unstale":"2023-10-16T14:35:28.816076+0000","last_undegraded":"2023-10-16T14:35:28.816076+0000","last_fullsized":"2023-10-16T14:35:28.816076+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:08.874304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.821332+0000","last_change":"2023-10-16T14:35:27.823806+0000","last_active":"2023-10-16T14:35:28.821332+0000","last_peered":"2023-10-16T14:35:28.821332+0000","last_clean":"2023-10-16T14:35:28.821332+0000","last_became_active":"2023-10-16T14:35:27.823524+0000","last_became_peered":"2023-10-16T14:35:27.823524+0000","last_unstale":"2023-10-16T14:35:28.821332+0000","last_undegraded":"2023-10-16T14:35:28.821332+0000","last_fullsized":"2023-10-16T14:35:28.821332+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:00:45.000191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921731+0000","last_change":"2023-10-16T14:35:26.786569+0000","last_active":"2023-10-16T14:35:28.921731+0000","last_peered":"2023-10-16T14:35:28.921731+0000","last_clean":"2023-10-16T14:35:28.921731+0000","last_became_active":"2023-10-16T14:35:26.786272+0000","last_became_peered":"2023-10-16T14:35:26.786272+0000","last_unstale":"2023-10-16T14:35:28.921731+0000","last_undegraded":"2023-10-16T14:35:28.921731+0000","last_fullsized":"2023-10-16T14:35:28.921731+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:47.918700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.824089+0000","last_change":"2023-10-16T14:35:27.822150+0000","last_active":"2023-10-16T14:35:27.824089+0000","last_peered":"2023-10-16T14:35:27.824089+0000","last_clean":"2023-10-16T14:35:27.824089+0000","last_became_active":"2023-10-16T14:35:27.821865+0000","last_became_peered":"2023-10-16T14:35:27.821865+0000","last_unstale":"2023-10-16T14:35:27.824089+0000","last_undegraded":"2023-10-16T14:35:27.824089+0000","last_fullsized":"2023-10-16T14:35:27.824089+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:42.967527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820540+0000","last_change":"2023-10-16T14:35:26.796155+0000","last_active":"2023-10-16T14:35:28.820540+0000","last_peered":"2023-10-16T14:35:28.820540+0000","last_clean":"2023-10-16T14:35:28.820540+0000","last_became_active":"2023-10-16T14:35:26.795883+0000","last_became_peered":"2023-10-16T14:35:26.795883+0000","last_unstale":"2023-10-16T14:35:28.820540+0000","last_undegraded":"2023-10-16T14:35:28.820540+0000","last_fullsized":"2023-10-16T14:35:28.820540+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:04.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823138+0000","last_change":"2023-10-16T14:35:27.821641+0000","last_active":"2023-10-16T14:35:28.823138+0000","last_peered":"2023-10-16T14:35:28.823138+0000","last_clean":"2023-10-16T14:35:28.823138+0000","last_became_active":"2023-10-16T14:35:27.820974+0000","last_became_peered":"2023-10-16T14:35:27.820974+0000","last_unstale":"2023-10-16T14:35:28.823138+0000","last_undegraded":"2023-10-16T14:35:28.823138+0000","last_fullsized":"2023-10-16T14:35:28.823138+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:01.726571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921579+0000","last_change":"2023-10-16T14:35:26.789671+0000","last_active":"2023-10-16T14:35:28.921579+0000","last_peered":"2023-10-16T14:35:28.921579+0000","last_clean":"2023-10-16T14:35:28.921579+0000","last_became_active":"2023-10-16T14:35:26.789378+0000","last_became_peered":"2023-10-16T14:35:26.789378+0000","last_unstale":"2023-10-16T14:35:28.921579+0000","last_undegraded":"2023-10-16T14:35:28.921579+0000","last_fullsized":"2023-10-16T14:35:28.921579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:14.044911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.838219+0000","last_change":"2023-10-16T14:35:27.833883+0000","last_active":"2023-10-16T14:35:27.838219+0000","last_peered":"2023-10-16T14:35:27.838219+0000","last_clean":"2023-10-16T14:35:27.838219+0000","last_became_active":"2023-10-16T14:35:27.833516+0000","last_became_peered":"2023-10-16T14:35:27.833516+0000","last_unstale":"2023-10-16T14:35:27.838219+0000","last_undegraded":"2023-10-16T14:35:27.838219+0000","last_fullsized":"2023-10-16T14:35:27.838219+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:59.871682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"126'2","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921129+0000","last_change":"2023-10-16T14:35:26.787500+0000","last_active":"2023-10-16T14:35:28.921129+0000","last_peered":"2023-10-16T14:35:28.921129+0000","last_clean":"2023-10-16T14:35:28.921129+0000","last_became_active":"2023-10-16T14:35:26.787220+0000","last_became_peered":"2023-10-16T14:35:26.787220+0000","last_unstale":"2023-10-16T14:35:28.921129+0000","last_undegraded":"2023-10-16T14:35:28.921129+0000","last_fullsized":"2023-10-16T14:35:28.921129+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:54.511680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.824926+0000","last_change":"2023-10-16T14:35:27.822660+0000","last_active":"2023-10-16T14:35:27.824926+0000","last_peered":"2023-10-16T14:35:27.824926+0000","last_clean":"2023-10-16T14:35:27.824926+0000","last_became_active":"2023-10-16T14:35:27.813998+0000","last_became_peered":"2023-10-16T14:35:27.813998+0000","last_unstale":"2023-10-16T14:35:27.824926+0000","last_undegraded":"2023-10-16T14:35:27.824926+0000","last_fullsized":"2023-10-16T14:35:27.824926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:01.021987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.812091+0000","last_change":"2023-10-16T14:35:26.796677+0000","last_active":"2023-10-16T14:35:27.812091+0000","last_peered":"2023-10-16T14:35:27.812091+0000","last_clean":"2023-10-16T14:35:27.812091+0000","last_became_active":"2023-10-16T14:35:26.796379+0000","last_became_peered":"2023-10-16T14:35:26.796379+0000","last_unstale":"2023-10-16T14:35:27.812091+0000","last_undegraded":"2023-10-16T14:35:27.812091+0000","last_fullsized":"2023-10-16T14:35:27.812091+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:25.281210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.837720+0000","last_change":"2023-10-16T14:35:27.829731+0000","last_active":"2023-10-16T14:35:27.837720+0000","last_peered":"2023-10-16T14:35:27.837720+0000","last_clean":"2023-10-16T14:35:27.837720+0000","last_became_active":"2023-10-16T14:35:27.829446+0000","last_became_peered":"2023-10-16T14:35:27.829446+0000","last_unstale":"2023-10-16T14:35:27.837720+0000","last_undegraded":"2023-10-16T14:35:27.837720+0000","last_fullsized":"2023-10-16T14:35:27.837720+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:44.257924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.809164+0000","last_change":"2023-10-16T14:35:26.791017+0000","last_active":"2023-10-16T14:35:27.809164+0000","last_peered":"2023-10-16T14:35:27.809164+0000","last_clean":"2023-10-16T14:35:27.809164+0000","last_became_active":"2023-10-16T14:35:26.790676+0000","last_became_peered":"2023-10-16T14:35:26.790676+0000","last_unstale":"2023-10-16T14:35:27.809164+0000","last_undegraded":"2023-10-16T14:35:27.809164+0000","last_fullsized":"2023-10-16T14:35:27.809164+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:56.252738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.819716+0000","last_change":"2023-10-16T14:35:27.817891+0000","last_active":"2023-10-16T14:35:28.819716+0000","last_peered":"2023-10-16T14:35:28.819716+0000","last_clean":"2023-10-16T14:35:28.819716+0000","last_became_active":"2023-10-16T14:35:27.817532+0000","last_became_peered":"2023-10-16T14:35:27.817532+0000","last_unstale":"2023-10-16T14:35:28.819716+0000","last_undegraded":"2023-10-16T14:35:28.819716+0000","last_fullsized":"2023-10-16T14:35:28.819716+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.575493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.920836+0000","last_change":"2023-10-16T14:35:26.787566+0000","last_active":"2023-10-16T14:35:28.920836+0000","last_peered":"2023-10-16T14:35:28.920836+0000","last_clean":"2023-10-16T14:35:28.920836+0000","last_became_active":"2023-10-16T14:35:26.787230+0000","last_became_peered":"2023-10-16T14:35:26.787230+0000","last_unstale":"2023-10-16T14:35:28.920836+0000","last_undegraded":"2023-10-16T14:35:28.920836+0000","last_fullsized":"2023-10-16T14:35:28.920836+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:57.995030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922265+0000","last_change":"2023-10-16T14:35:27.823065+0000","last_active":"2023-10-16T14:35:28.922265+0000","last_peered":"2023-10-16T14:35:28.922265+0000","last_clean":"2023-10-16T14:35:28.922265+0000","last_became_active":"2023-10-16T14:35:27.822798+0000","last_became_peered":"2023-10-16T14:35:27.822798+0000","last_unstale":"2023-10-16T14:35:28.922265+0000","last_undegraded":"2023-10-16T14:35:28.922265+0000","last_fullsized":"2023-10-16T14:35:28.922265+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:46.935993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"126'2","reported_seq":13,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.836760+0000","last_change":"2023-10-16T14:35:26.792930+0000","last_active":"2023-10-16T14:35:27.836760+0000","last_peered":"2023-10-16T14:35:27.836760+0000","last_clean":"2023-10-16T14:35:27.836760+0000","last_became_active":"2023-10-16T14:35:26.791961+0000","last_became_peered":"2023-10-16T14:35:26.791961+0000","last_unstale":"2023-10-16T14:35:27.836760+0000","last_undegraded":"2023-10-16T14:35:27.836760+0000","last_fullsized":"2023-10-16T14:35:27.836760+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:58.145335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816942+0000","last_change":"2023-10-16T14:35:27.825481+0000","last_active":"2023-10-16T14:35:28.816942+0000","last_peered":"2023-10-16T14:35:28.816942+0000","last_clean":"2023-10-16T14:35:28.816942+0000","last_became_active":"2023-10-16T14:35:27.825211+0000","last_became_peered":"2023-10-16T14:35:27.825211+0000","last_unstale":"2023-10-16T14:35:28.816942+0000","last_undegraded":"2023-10-16T14:35:28.816942+0000","last_fullsized":"2023-10-16T14:35:28.816942+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:11.265954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.922380+0000","last_change":"2023-10-16T14:35:26.791918+0000","last_active":"2023-10-16T14:35:28.922380+0000","last_peered":"2023-10-16T14:35:28.922380+0000","last_clean":"2023-10-16T14:35:28.922380+0000","last_became_active":"2023-10-16T14:35:26.785641+0000","last_became_peered":"2023-10-16T14:35:26.785641+0000","last_unstale":"2023-10-16T14:35:28.922380+0000","last_undegraded":"2023-10-16T14:35:28.922380+0000","last_fullsized":"2023-10-16T14:35:28.922380+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:04.952664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.816762+0000","last_change":"2023-10-16T14:35:27.813315+0000","last_active":"2023-10-16T14:35:28.816762+0000","last_peered":"2023-10-16T14:35:28.816762+0000","last_clean":"2023-10-16T14:35:28.816762+0000","last_became_active":"2023-10-16T14:35:27.813026+0000","last_became_peered":"2023-10-16T14:35:27.813026+0000","last_unstale":"2023-10-16T14:35:28.816762+0000","last_undegraded":"2023-10-16T14:35:28.816762+0000","last_fullsized":"2023-10-16T14:35:28.816762+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:12.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921545+0000","last_change":"2023-10-16T14:35:26.780074+0000","last_active":"2023-10-16T14:35:28.921545+0000","last_peered":"2023-10-16T14:35:28.921545+0000","last_clean":"2023-10-16T14:35:28.921545+0000","last_became_active":"2023-10-16T14:35:26.779707+0000","last_became_peered":"2023-10-16T14:35:26.779707+0000","last_unstale":"2023-10-16T14:35:28.921545+0000","last_undegraded":"2023-10-16T14:35:28.921545+0000","last_fullsized":"2023-10-16T14:35:28.921545+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:00.010403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820809+0000","last_change":"2023-10-16T14:35:27.835016+0000","last_active":"2023-10-16T14:35:28.820809+0000","last_peered":"2023-10-16T14:35:28.820809+0000","last_clean":"2023-10-16T14:35:28.820809+0000","last_became_active":"2023-10-16T14:35:27.833615+0000","last_became_peered":"2023-10-16T14:35:27.833615+0000","last_unstale":"2023-10-16T14:35:28.820809+0000","last_undegraded":"2023-10-16T14:35:28.820809+0000","last_fullsized":"2023-10-16T14:35:28.820809+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:33:41.377382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.820951+0000","last_change":"2023-10-16T14:35:26.795023+0000","last_active":"2023-10-16T14:35:28.820951+0000","last_peered":"2023-10-16T14:35:28.820951+0000","last_clean":"2023-10-16T14:35:28.820951+0000","last_became_active":"2023-10-16T14:35:26.794718+0000","last_became_peered":"2023-10-16T14:35:26.794718+0000","last_unstale":"2023-10-16T14:35:28.820951+0000","last_undegraded":"2023-10-16T14:35:28.820951+0000","last_fullsized":"2023-10-16T14:35:28.820951+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:08.874304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.829957+0000","last_change":"2023-10-16T14:35:27.822294+0000","last_active":"2023-10-16T14:35:27.829957+0000","last_peered":"2023-10-16T14:35:27.829957+0000","last_clean":"2023-10-16T14:35:27.829957+0000","last_became_active":"2023-10-16T14:35:27.821931+0000","last_became_peered":"2023-10-16T14:35:27.821931+0000","last_unstale":"2023-10-16T14:35:27.829957+0000","last_undegraded":"2023-10-16T14:35:27.829957+0000","last_fullsized":"2023-10-16T14:35:27.829957+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:09.885888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921232+0000","last_change":"2023-10-16T14:35:26.788099+0000","last_active":"2023-10-16T14:35:28.921232+0000","last_peered":"2023-10-16T14:35:28.921232+0000","last_clean":"2023-10-16T14:35:28.921232+0000","last_became_active":"2023-10-16T14:35:26.787835+0000","last_became_peered":"2023-10-16T14:35:26.787835+0000","last_unstale":"2023-10-16T14:35:28.921232+0000","last_undegraded":"2023-10-16T14:35:28.921232+0000","last_fullsized":"2023-10-16T14:35:28.921232+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:26.006973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.921929+0000","last_change":"2023-10-16T14:35:27.816861+0000","last_active":"2023-10-16T14:35:28.921929+0000","last_peered":"2023-10-16T14:35:28.921929+0000","last_clean":"2023-10-16T14:35:28.921929+0000","last_became_active":"2023-10-16T14:35:27.816555+0000","last_became_peered":"2023-10-16T14:35:27.816555+0000","last_unstale":"2023-10-16T14:35:28.921929+0000","last_undegraded":"2023-10-16T14:35:28.921929+0000","last_fullsized":"2023-10-16T14:35:28.921929+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:15.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.819882+0000","last_change":"2023-10-16T14:35:26.798514+0000","last_active":"2023-10-16T14:35:28.819882+0000","last_peered":"2023-10-16T14:35:28.819882+0000","last_clean":"2023-10-16T14:35:28.819882+0000","last_became_active":"2023-10-16T14:35:26.798125+0000","last_became_peered":"2023-10-16T14:35:26.798125+0000","last_unstale":"2023-10-16T14:35:28.819882+0000","last_undegraded":"2023-10-16T14:35:28.819882+0000","last_fullsized":"2023-10-16T14:35:28.819882+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:32.281222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.828155+0000","last_change":"2023-10-16T14:35:27.824256+0000","last_active":"2023-10-16T14:35:27.828155+0000","last_peered":"2023-10-16T14:35:27.828155+0000","last_clean":"2023-10-16T14:35:27.828155+0000","last_became_active":"2023-10-16T14:35:27.823070+0000","last_became_peered":"2023-10-16T14:35:27.823070+0000","last_unstale":"2023-10-16T14:35:27.828155+0000","last_undegraded":"2023-10-16T14:35:27.828155+0000","last_fullsized":"2023-10-16T14:35:27.828155+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:15.056495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.811650+0000","last_change":"2023-10-16T14:35:26.787307+0000","last_active":"2023-10-16T14:35:27.811650+0000","last_peered":"2023-10-16T14:35:27.811650+0000","last_clean":"2023-10-16T14:35:27.811650+0000","last_became_active":"2023-10-16T14:35:26.787011+0000","last_became_peered":"2023-10-16T14:35:26.787011+0000","last_unstale":"2023-10-16T14:35:27.811650+0000","last_undegraded":"2023-10-16T14:35:27.811650+0000","last_fullsized":"2023-10-16T14:35:27.811650+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:54.575406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.819770+0000","last_change":"2023-10-16T14:35:27.834344+0000","last_active":"2023-10-16T14:35:28.819770+0000","last_peered":"2023-10-16T14:35:28.819770+0000","last_clean":"2023-10-16T14:35:28.819770+0000","last_became_active":"2023-10-16T14:35:27.833984+0000","last_became_peered":"2023-10-16T14:35:27.833984+0000","last_unstale":"2023-10-16T14:35:28.819770+0000","last_undegraded":"2023-10-16T14:35:28.819770+0000","last_fullsized":"2023-10-16T14:35:28.819770+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:12.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.790306+0000","last_change":"2023-10-16T14:35:26.781471+0000","last_active":"2023-10-16T14:35:27.790306+0000","last_peered":"2023-10-16T14:35:27.790306+0000","last_clean":"2023-10-16T14:35:27.790306+0000","last_became_active":"2023-10-16T14:35:26.780245+0000","last_became_peered":"2023-10-16T14:35:26.780245+0000","last_unstale":"2023-10-16T14:35:27.790306+0000","last_undegraded":"2023-10-16T14:35:27.790306+0000","last_fullsized":"2023-10-16T14:35:27.790306+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:28.207356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.838494+0000","last_change":"2023-10-16T14:35:27.832606+0000","last_active":"2023-10-16T14:35:27.838494+0000","last_peered":"2023-10-16T14:35:27.838494+0000","last_clean":"2023-10-16T14:35:27.838494+0000","last_became_active":"2023-10-16T14:35:27.832165+0000","last_became_peered":"2023-10-16T14:35:27.832165+0000","last_unstale":"2023-10-16T14:35:27.838494+0000","last_undegraded":"2023-10-16T14:35:27.838494+0000","last_fullsized":"2023-10-16T14:35:27.838494+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:26.752610+0000","last_clean_scrub_stamp":"2023-10-16T14:35:26.752610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:27.094258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"126'1","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T14:35:28.823117+0000","last_change":"2023-10-16T14:35:26.794753+0000","last_active":"2023-10-16T14:35:28.823117+0000","last_peered":"2023-10-16T14:35:28.823117+0000","last_clean":"2023-10-16T14:35:28.823117+0000","last_became_active":"2023-10-16T14:35:26.794446+0000","last_became_peered":"2023-10-16T14:35:26.794446+0000","last_unstale":"2023-10-16T14:35:28.823117+0000","last_undegraded":"2023-10-16T14:35:28.823117+0000","last_fullsized":"2023-10-16T14:35:28.823117+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:35:25.741026+0000","last_clean_scrub_stamp":"2023-10-16T14:35:25.741026+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:12.262849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"115'208","reported_seq":603,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T14:35:27.792629+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:35:27.792629+0000","last_peered":"2023-10-16T14:35:27.792629+0000","last_clean":"2023-10-16T14:35:27.792629+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:35:27.792629+0000","last_undegraded":"2023-10-16T14:35:27.792629+0000","last_fullsized":"2023-10-16T14:35:27.792629+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575076,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2158592,"data_stored":1341655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.091,"15min":1.145},"min":{"1min":0.75,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5089999999999999,"5min":1.833,"15min":1.833},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.095,"15min":1.149},"min":{"1min":0.93799999999999994,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3049999999999999,"5min":1.891,"15min":1.891},"last":0.84999999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.82199999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.325,"5min":1.2390000000000001,"15min":1.2430000000000001},"min":{"1min":0.96699999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6850000000000001,"5min":2.048,"15min":2.048},"last":0.68799999999999994}]},{"osd":2,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.2709999999999999,"15min":1.3080000000000001},"min":{"1min":1.016,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9339999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.246,"15min":1.2370000000000001},"min":{"1min":1.073,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.4079999999999999,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.2669999999999999,"15min":1.302},"min":{"1min":0.82899999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.339,"15min":1.3700000000000001},"min":{"1min":0.98799999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.804,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.73099999999999998}]},{"osd":4,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.3720000000000001,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.495,"15min":0.495},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.274,"15min":1.2669999999999999},"min":{"1min":0.81000000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3779999999999999,"15min":1.409},"min":{"1min":0.93899999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.0950000000000002,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.482,"15min":1.498},"min":{"1min":1.286,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.97,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Oct 16 14:34:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.4450000000000001,"15min":1.4430000000000001},"min":{"1min":1.1220000000000001,"5min":0.502,"15min":0.502},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.516,"15min":1.522},"min":{"1min":1.294,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.6139999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575075,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36880,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334960,"statfs":{"total":96636764160,"available":96598999040,"internally_reserved":0,"allocated":1576960,"data_stored":758127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1950000000000001,"15min":1.1839999999999999},"min":{"1min":0.79000000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.571,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2549999999999999,"15min":1.252},"min":{"1min":0.89100000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6579999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":0.998}]},{"osd":1,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.296,"15min":1.282},"min":{"1min":0.89200000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7849999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3140000000000001,"15min":1.302},"min":{"1min":0.94999999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.679,"5min":10.241,"15min":10.241},"last":1.1160000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.155,"15min":1.202},"min":{"1min":0.95199999999999996,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5860000000000001,"5min":1.708,"15min":2.4940000000000002},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.387,"5min":1.337,"15min":1.3580000000000001},"min":{"1min":1.006,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9059999999999999,"5min":5.694,"15min":5.694},"last":1.2669999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.355,"15min":1.3520000000000001},"min":{"1min":1.012,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5580000000000001,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.296,"15min":1.329},"min":{"1min":0.95799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.8620000000000001,"5min":2.2349999999999999,"15min":2.6120000000000001},"last":1.0860000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.409,"15min":1.329},"min":{"1min":0.90100000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.9930000000000001,"5min":13.826000000000001,"15min":13.826000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.367,"15min":1.3680000000000001},"min":{"1min":0.81999999999999995,"5min":0.44,"15min":0.44},"max":{"1min":1.742,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.4139999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3149999999999999,"15min":1.3129999999999999},"min":{"1min":1,"5min":0.439,"15min":0.439},"max":{"1min":1.7789999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":1.327},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5169999999999999,"15min":1.4950000000000001},"min":{"1min":1.204,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.0819999999999999,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.6140000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.575,"15min":1.5620000000000001},"min":{"1min":1.109,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.831,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.829},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.526,"15min":1.542},"min":{"1min":1.194,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.946,"5min":2.1779999999999999,"15min":2.7250000000000001},"last":1.6559999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575075,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":751245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.107,"15min":1.1259999999999999},"min":{"1min":0.79700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.669,"5min":1.669,"15min":2.2309999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.171,"15min":1.1819999999999999},"min":{"1min":0.76200000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.7210000000000001,"5min":1.7729999999999999,"15min":1.8340000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.119,"15min":1.1699999999999999},"min":{"1min":0.95699999999999996,"5min":0.622,"15min":0.622},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.726},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.2370000000000001},"min":{"1min":0.92300000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.526,"5min":1.704,"15min":1.7809999999999999},"last":0.876}]},{"osd":2,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.202,"15min":1.246},"min":{"1min":0.93400000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.569,"5min":1.913,"15min":2.5009999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":1.3500000000000001},"min":{"1min":1.097,"5min":0.625,"15min":0.625},"max":{"1min":1.6779999999999999,"5min":2.0049999999999999,"15min":2.1200000000000001},"last":0.78200000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2470000000000001,"15min":1.28},"min":{"1min":0.997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.696,"5min":1.903,"15min":1.903},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.381,"5min":1.298,"15min":1.377},"min":{"1min":1.143,"5min":0.78400000000000003,"15min":0.76100000000000001},"max":{"1min":1.7549999999999999,"5min":1.8420000000000001,"15min":2.8029999999999999},"last":0.92800000000000005}]},{"osd":4,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.284,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.587,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4359999999999999,"15min":1.4990000000000001},"min":{"1min":0.82899999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.883,"5min":1.96,"15min":4.9480000000000004},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.345,"15min":1.3839999999999999},"min":{"1min":0.91900000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.7250000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.363,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.798,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":0.90500000000000003}]},{"osd":7,"last update":"Mon Oct 16 14:34:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4890000000000001,"15min":1.5529999999999999},"min":{"1min":0.95199999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.95,"5min":2.1200000000000001,"15min":2.641},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5649999999999999,"15min":1.609},"min":{"1min":1.298,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.726,"5min":2.1909999999999998,"15min":2.2010000000000001},"last":0.97599999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575076,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46624,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325216,"statfs":{"total":96636764160,"available":96589021184,"internally_reserved":0,"allocated":1589248,"data_stored":759211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.137,"15min":1.1579999999999999},"min":{"1min":0.93799999999999994,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.0339999999999998},"last":2.0259999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.071,"15min":1.1579999999999999},"min":{"1min":0.65700000000000003,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.653,"5min":1.7589999999999999,"15min":1.9199999999999999},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.117,"15min":1.177},"min":{"1min":0.70199999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.53},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.2070000000000001,"15min":1.23},"min":{"1min":0.97599999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.704,"5min":1.778,"15min":2.3690000000000002},"last":1.929}]},{"osd":2,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3640000000000001,"15min":1.399},"min":{"1min":1.0649999999999999,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.863,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.671},{"interface":"front","average":{"1min":1.383,"5min":1.3140000000000001,"15min":1.3580000000000001},"min":{"1min":0.94799999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.6990000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1379999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2569999999999999,"15min":1.3109999999999999},"min":{"1min":0.75900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":1.728,"5min":1.95,"15min":1.95},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.262,"15min":1.3080000000000001},"min":{"1min":0.78900000000000003,"5min":0.437,"15min":0.437},"max":{"1min":1.984,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.8200000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3089999999999999,"15min":1.2969999999999999},"min":{"1min":0.74299999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":2.113},{"interface":"front","average":{"1min":1.395,"5min":1.3560000000000001,"15min":1.3540000000000001},"min":{"1min":0.92700000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8440000000000001,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.228}]},{"osd":6,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.343,"15min":1.3080000000000001},"min":{"1min":0.86699999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":2.069},{"interface":"front","average":{"1min":1.516,"5min":1.476,"15min":1.464},"min":{"1min":0.998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.149,"5min":2.1960000000000002,"15min":2.2890000000000001},"last":2.198}]},{"osd":7,"last update":"Mon Oct 16 14:35:05 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.484,"15min":1.4770000000000001},"min":{"1min":1.0860000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.486,"15min":1.468},"min":{"1min":1.135,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.214,"5min":2.214,"15min":2.214},"last":1.968}]}]},{"osd":3,"up_from":13,"seq":55834575075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1572864,"data_stored":752809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.83799999999999997,"15min":0.88200000000000001},"min":{"1min":0.67300000000000004,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.5,"5min":1.696,"15min":1.696},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.121,"5min":0.93100000000000005,"15min":1.0209999999999999},"min":{"1min":0.80300000000000005,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.651,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.01}]},{"osd":1,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.88500000000000001,"15min":0.93999999999999995},"min":{"1min":0.73299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4730000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.149,"5min":1.0580000000000001,"15min":1.131},"min":{"1min":0.70799999999999996,"5min":0.314,"15min":0.314},"max":{"1min":1.736,"5min":3.024,"15min":3.024},"last":1.3400000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.1519999999999999,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.853,"5min":3.3420000000000001,"15min":3.3420000000000001},"last":1.27},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.2849999999999999,"15min":1.405},"min":{"1min":1.1419999999999999,"5min":0.5,"15min":0.5},"max":{"1min":1.75,"5min":2.5609999999999999,"15min":2.6299999999999999},"last":1.198}]},{"osd":4,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1970000000000001,"15min":1.256},"min":{"1min":0.96799999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.6990000000000001,"5min":2.02,"15min":2.02},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2689999999999999,"15min":1.331},"min":{"1min":0.85099999999999998,"5min":0.629,"15min":0.629},"max":{"1min":1.6419999999999999,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.4810000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.4350000000000001,"15min":1.518},"min":{"1min":1.036,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8620000000000001,"5min":3.206,"15min":3.206},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.385,"15min":1.452},"min":{"1min":1.026,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8380000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.7250000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.4019999999999999},"min":{"1min":1.002,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7549999999999999,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.155},{"interface":"front","average":{"1min":1.421,"5min":1.3859999999999999,"15min":1.4830000000000001},"min":{"1min":1.1020000000000001,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.9279999999999999,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.7629999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:34:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.49,"15min":1.577},"min":{"1min":1.3839999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.911,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4039999999999999,"15min":1.5329999999999999},"min":{"1min":1.125,"5min":0.629,"15min":0.629},"max":{"1min":1.869,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.526}]}]},{"osd":2,"up_from":13,"seq":55834575076,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46592,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325248,"statfs":{"total":96636764160,"available":96589053952,"internally_reserved":0,"allocated":1560576,"data_stored":751209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.874,"15min":0.874},"min":{"1min":0.58799999999999997,"5min":0.38,"15min":0.38},"max":{"1min":1.631,"5min":1.631,"15min":1.7150000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.97499999999999998,"15min":1.0249999999999999},"min":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.32600000000000001},"max":{"1min":1.4910000000000001,"5min":1.585,"15min":2.3260000000000001},"last":1.127}]},{"osd":1,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.98999999999999999,"15min":1.0009999999999999},"min":{"1min":0.55400000000000005,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.399,"5min":3.234,"15min":3.234},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.141,"15min":1.1619999999999999},"min":{"1min":0.69399999999999995,"5min":0.51200000000000001,"15min":0.42199999999999999},"max":{"1min":1.827,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.5880000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.143,"15min":1.232},"min":{"1min":0.61299999999999999,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":1.6779999999999999,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.747},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.2789999999999999,"15min":1.3049999999999999},"min":{"1min":0.84899999999999998,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.8660000000000001,"5min":3.5569999999999999,"15min":3.5569999999999999},"last":1.4810000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.409,"15min":1.4259999999999999},"min":{"1min":1.123,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8009999999999999,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":1.748},{"interface":"front","average":{"1min":1.282,"5min":1.29,"15min":1.343},"min":{"1min":0.94199999999999995,"5min":0.77400000000000002,"15min":0.72599999999999998},"max":{"1min":1.6559999999999999,"5min":2.9359999999999999,"15min":3.016},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.375,"15min":1.411},"min":{"1min":0.90100000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.819,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.635},{"interface":"front","average":{"1min":1.532,"5min":1.524,"15min":1.5489999999999999},"min":{"1min":0.92300000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.8380000000000001,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.677}]},{"osd":6,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3919999999999999,"15min":1.4650000000000001},"min":{"1min":0.873,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.732,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.244},{"interface":"front","average":{"1min":1.47,"5min":1.4910000000000001,"15min":1.524},"min":{"1min":1.0009999999999999,"5min":0.71299999999999997,"15min":0.65300000000000002},"max":{"1min":1.7629999999999999,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":1.399}]},{"osd":7,"last update":"Mon Oct 16 14:34:56 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.478,"15min":1.542},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.69099999999999995},"max":{"1min":1.788,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.29},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5860000000000001,"15min":1.633},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.80300000000000005},"max":{"1min":1.9750000000000001,"5min":2.7519999999999998,"15min":2.7519999999999998},"last":1.71}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1556480,"data_stored":751175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.872,"15min":0.90000000000000002},"min":{"1min":0.63200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.7749999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0149999999999999,"15min":1.0940000000000001},"min":{"1min":0.67100000000000004,"5min":0.378,"15min":0.378},"max":{"1min":1.6140000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.996}]},{"osd":2,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92300000000000004,"15min":0.98099999999999998},"min":{"1min":0.58599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.546,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.167,"5min":1.0780000000000001,"15min":1.155},"min":{"1min":0.56899999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.597,"5min":1.756,"15min":1.796},"last":1.1499999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.21,"15min":1.3100000000000001},"min":{"1min":0.71599999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.853,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.218},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.135,"15min":1.1930000000000001},"min":{"1min":0.51900000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6619999999999999,"5min":1.79,"15min":1.855},"last":1.111}]},{"osd":4,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2050000000000001,"15min":1.282},"min":{"1min":0.89800000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6679999999999999,"5min":2.069,"15min":2.069},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3440000000000001,"15min":1.4159999999999999},"min":{"1min":1.113,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8009999999999999,"5min":2.899,"15min":2.899},"last":1.6559999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3380000000000001,"15min":1.431},"min":{"1min":0.78500000000000003,"5min":0.623,"15min":0.623},"max":{"1min":1.6299999999999999,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.337,"15min":1.4219999999999999},"min":{"1min":1.024,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7330000000000001,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.482,"15min":1.544},"min":{"1min":1.1419999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.923,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.427,"15min":1.5329999999999999},"min":{"1min":0.88600000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6799999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.77}]},{"osd":7,"last update":"Mon Oct 16 14:34:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5489999999999999,"15min":1.625},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.74,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.804},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.448,"15min":1.55},"min":{"1min":1.143,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7869999999999999,"5min":2.2000000000000002,"15min":2.262},"last":1.3759999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37448,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334392,"statfs":{"total":96636764160,"available":96598417408,"internally_reserved":0,"allocated":2154496,"data_stored":1342125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.89200000000000002,"15min":0.94399999999999995},"min":{"1min":0.66700000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2569999999999999,"5min":1.5880000000000001,"15min":1.7729999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.119,"15min":1.169},"min":{"1min":0.94599999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.52,"5min":3.1120000000000001,"15min":3.1120000000000001},"last":1.4970000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.212,"15min":1.254},"min":{"1min":0.78400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.752,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1699999999999999,"15min":1.21},"min":{"1min":0.82399999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6220000000000001,"5min":1.847,"15min":2.1070000000000002},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.96599999999999997,"15min":1.004},"min":{"1min":0.80800000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.609,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.753},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.23},"min":{"1min":0.84399999999999997,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":1.512,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":1.097}]},{"osd":4,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.264,"15min":1.3089999999999999},"min":{"1min":1.044,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.012,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":2.3849999999999998},{"interface":"front","average":{"1min":1.48,"5min":1.454,"15min":1.4730000000000001},"min":{"1min":1.103,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.0569999999999999,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":2.4300000000000002}]},{"osd":5,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3819999999999999,"15min":1.431},"min":{"1min":1.012,"5min":0.79000000000000004,"15min":0.78400000000000003},"max":{"1min":1.7909999999999999,"5min":2.1930000000000001,"15min":2.5880000000000001},"last":1.069},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3240000000000001,"15min":1.371},"min":{"1min":0.91500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.895,"5min":3.7069999999999999,"15min":3.7069999999999999},"last":1.1599999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.575,"15min":1.589},"min":{"1min":1.2849999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9330000000000001,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.454,"15min":1.504},"min":{"1min":1.0980000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":1.958,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":2.093}]},{"osd":7,"last update":"Mon Oct 16 14:34:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.534,"15min":1.591},"min":{"1min":1.2649999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.117,"5min":2.335,"15min":2.6509999999999998},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.514,"15min":1.5549999999999999},"min":{"1min":1.071,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.206,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":1.6910000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:35:33.263 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:35:33.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:35:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.529+0000 7fd9f8bb6640 1 Processor -- start 2023-10-16T14:35:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.529+0000 7fd9f8bb6640 1 -- start start 2023-10-16T14:35:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f8bb6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9f4159b20 0x7fd9f406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f8bb6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f4159f00 0x7fd9f40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f8bb6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f406f890 con 0x7fd9f415d8c0 2023-10-16T14:35:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f8bb6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f4070d30 con 0x7fd9f4159b20 2023-10-16T14:35:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f8bb6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f406c650 con 0x7fd9f4159f00 2023-10-16T14:35:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f2d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9f415d8c0 0x7fd9f40cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32924/0 (socket says 172.21.15.29:32924) 2023-10-16T14:35:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.530+0000 7fd9f2d9d640 1 -- 172.21.15.29:0/2234765975 learned_addr learned my addr 172.21.15.29:0/2234765975 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:33.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f259c640 1 --2- 172.21.15.29:0/2234765975 >> v2:172.21.15.29:3300/0 conn(0x7fd9f4159b20 0x7fd9f406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f1d9b640 1 --2- 172.21.15.29:0/2234765975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f4159f00 0x7fd9f40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f159a640 1 -- 172.21.15.29:0/2234765975 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677410679 0 0) 0x7fd9f406f890 con 0x7fd9f415d8c0 2023-10-16T14:35:33.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f159a640 1 -- 172.21.15.29:0/2234765975 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9dc005730 con 0x7fd9f415d8c0 2023-10-16T14:35:33.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f259c640 1 -- 172.21.15.29:0/2234765975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f4159f00 msgr2=0x7fd9f40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:33.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f259c640 1 --2- 172.21.15.29:0/2234765975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f4159f00 0x7fd9f40b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f259c640 1 -- 172.21.15.29:0/2234765975 >> v1:172.21.15.29:6789/0 conn(0x7fd9f415d8c0 legacy=0x7fd9f40cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.531+0000 7fd9f259c640 1 -- 172.21.15.29:0/2234765975 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f406c930 con 0x7fd9f4159b20 2023-10-16T14:35:33.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.532+0000 7fd9f259c640 1 --2- 172.21.15.29:0/2234765975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f406bbe0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009d90 tx=0x7fd9e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08ffbc27776b425 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:33.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f159a640 1 -- 172.21.15.29:0/2234765975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e402cc60 con 0x7fd9f4159b20 2023-10-16T14:35:33.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f159a640 1 -- 172.21.15.29:0/2234765975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e402cdc0 con 0x7fd9f4159b20 2023-10-16T14:35:33.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f159a640 1 -- 172.21.15.29:0/2234765975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4034640 con 0x7fd9f4159b20 2023-10-16T14:35:33.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/2234765975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 msgr2=0x7fd9f406bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/2234765975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f406bbe0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009d90 tx=0x7fd9e402beb0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/2234765975 shutdown_connections 2023-10-16T14:35:33.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/2234765975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f4159f00 0x7fd9f40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/2234765975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f406bbe0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/2234765975 >> 172.21.15.29:0/2234765975 conn(0x7fd9f40cb410 msgr2=0x7fd9f40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:33.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.533+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/2234765975 shutdown_connections 2023-10-16T14:35:33.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.534+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/2234765975 wait complete. 2023-10-16T14:35:33.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.535+0000 7fd9f8bb6640 1 Processor -- start 2023-10-16T14:35:33.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.535+0000 7fd9f8bb6640 1 -- start start 2023-10-16T14:35:33.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.536+0000 7fd9f8bb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f40c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.536+0000 7fd9f8bb6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f4159f00 0x7fd9f4153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.536+0000 7fd9f8bb6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f415d8c0 0x7fd9f40c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.536+0000 7fd9f8bb6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c24c0 con 0x7fd9f4159b20 2023-10-16T14:35:33.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.536+0000 7fd9f8bb6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40b22a0 con 0x7fd9f4159f00 2023-10-16T14:35:33.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.536+0000 7fd9f8bb6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40b2580 con 0x7fd9f415d8c0 2023-10-16T14:35:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f40c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f40c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52068/0 (socket says 172.21.15.29:52068) 2023-10-16T14:35:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 -- 172.21.15.29:0/78899931 learned_addr learned my addr 172.21.15.29:0/78899931 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f1d9b640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f4159f00 0x7fd9f4153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f2d9d640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f415d8c0 0x7fd9f40c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 -- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f415d8c0 msgr2=0x7fd9f40c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f415d8c0 0x7fd9f40c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 -- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f4159f00 msgr2=0x7fd9f4153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f4159f00 0x7fd9f4153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f259c640 1 -- 172.21.15.29:0/78899931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4009a70 con 0x7fd9f4159b20 2023-10-16T14:35:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.537+0000 7fd9f1d9b640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f4159f00 0x7fd9f4153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:35:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.538+0000 7fd9f259c640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f40c8430 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009eb0 tx=0x7fd9e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.538+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e403e610 con 0x7fd9f4159b20 2023-10-16T14:35:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.538+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e403e770 con 0x7fd9f4159b20 2023-10-16T14:35:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.538+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e403eb20 con 0x7fd9f4159b20 2023-10-16T14:35:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.538+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f40b2860 con 0x7fd9f4159b20 2023-10-16T14:35:33.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.538+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f40b2e20 con 0x7fd9f4159b20 2023-10-16T14:35:33.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.540+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f40b3010 con 0x7fd9f4159b20 2023-10-16T14:35:33.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.540+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd9e403b050 con 0x7fd9f4159b20 2023-10-16T14:35:33.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.541+0000 7fd9e2ffd640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fd9d406afc0 0x7fd9d406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:33.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.541+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9e40ac190 con 0x7fd9f4159b20 2023-10-16T14:35:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.542+0000 7fd9f1d9b640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fd9d406afc0 0x7fd9d406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.543+0000 7fd9f1d9b640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fd9d406afc0 0x7fd9d406d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd9f406f680 tx=0x7fd9e8008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.546+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd9e4077090 con 0x7fd9f4159b20 2023-10-16T14:35:33.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.810+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd9f406cbe0 con 0x7fd9f4159b20 2023-10-16T14:35:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.815+0000 7fd9e2ffd640 1 -- 172.21.15.29:0/78899931 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd9e4009450 con 0x7fd9f4159b20 2023-10-16T14:35:33.819 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:35:33.820 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:35:33.820 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:35:33.820 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:35:33.820 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:35:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:35:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fd9d406afc0 msgr2=0x7fd9d406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fd9d406afc0 0x7fd9d406d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd9f406f680 tx=0x7fd9e8008040 comp rx=0 tx=0).stop 2023-10-16T14:35:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 msgr2=0x7fd9f40c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f40c8430 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009eb0 tx=0x7fd9e4004a40 comp rx=0 tx=0).stop 2023-10-16T14:35:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 shutdown_connections 2023-10-16T14:35:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fd9d406afc0 0x7fd9d406d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd9f415d8c0 0x7fd9f40c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9f4159f00 0x7fd9f4153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 --2- 172.21.15.29:0/78899931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9f4159b20 0x7fd9f40c8430 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 >> 172.21.15.29:0/78899931 conn(0x7fd9f40cb410 msgr2=0x7fd9f40b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.818+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 shutdown_connections 2023-10-16T14:35:33.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:33.819+0000 7fd9f8bb6640 1 -- 172.21.15.29:0/78899931 wait complete. 2023-10-16T14:35:33.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:35:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.064+0000 7f7e43ee5640 1 Processor -- start 2023-10-16T14:35:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.065+0000 7f7e43ee5640 1 -- start start 2023-10-16T14:35:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.066+0000 7f7e43ee5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e3c160420 0x7f7e3c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.066+0000 7f7e43ee5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.066+0000 7f7e43ee5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c06e170 con 0x7f7e3c0adc40 2023-10-16T14:35:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.066+0000 7f7e43ee5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c0cde30 con 0x7f7e3c160420 2023-10-16T14:35:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.066+0000 7f7e43ee5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0cd460 con 0x7f7e3c160800 2023-10-16T14:35:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.066+0000 7f7e41c81640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e3c160420 0x7f7e3c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41c81640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e3c160420 0x7f7e3c0ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37604/0 (socket says 172.21.15.29:37604) 2023-10-16T14:35:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41c81640 1 -- 172.21.15.29:0/2669312963 learned_addr learned my addr 172.21.15.29:0/2669312963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41480640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41c81640 1 -- 172.21.15.29:0/2669312963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 msgr2=0x7f7e3c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41c81640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41c81640 1 -- 172.21.15.29:0/2669312963 >> v1:172.21.15.170:6789/0 conn(0x7f7e3c0adc40 legacy=0x7f7e3c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41c81640 1 -- 172.21.15.29:0/2669312963 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c06e3c0 con 0x7f7e3c160420 2023-10-16T14:35:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.067+0000 7f7e41480640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.068+0000 7f7e41c81640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c160420 0x7f7e3c0ae170 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009870 tx=0x7f7e3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce043bfbb7ee6d02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.068+0000 7f7e40c7f640 1 -- 172.21.15.29:0/2669312963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38031640 con 0x7f7e3c160420 2023-10-16T14:35:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e40c7f640 1 -- 172.21.15.29:0/2669312963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e380317a0 con 0x7f7e3c160420 2023-10-16T14:35:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e40c7f640 1 -- 172.21.15.29:0/2669312963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38031b30 con 0x7f7e3c160420 2023-10-16T14:35:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2669312963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c160420 msgr2=0x7f7e3c0ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c160420 0x7f7e3c0ae170 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009870 tx=0x7f7e3802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2669312963 shutdown_connections 2023-10-16T14:35:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2669312963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c160420 0x7f7e3c0ae170 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2669312963 >> 172.21.15.29:0/2669312963 conn(0x7f7e3c0b0d30 msgr2=0x7f7e3c0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2669312963 shutdown_connections 2023-10-16T14:35:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.069+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2669312963 wait complete. 2023-10-16T14:35:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.071+0000 7f7e43ee5640 1 Processor -- start 2023-10-16T14:35:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.071+0000 7f7e43ee5640 1 -- start start 2023-10-16T14:35:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.072+0000 7f7e43ee5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 0x7f7e3c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.072+0000 7f7e43ee5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c160420 0x7f7e3c0c2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.072+0000 7f7e43ee5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.072+0000 7f7e43ee5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c14e5a0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.072+0000 7f7e43ee5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0c8d50 con 0x7f7e3c160420 2023-10-16T14:35:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.072+0000 7f7e43ee5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0c31f0 con 0x7f7e3c160800 2023-10-16T14:35:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 0x7f7e3c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 0x7f7e3c0c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37618/0 (socket says 172.21.15.29:37618) 2023-10-16T14:35:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 -- 172.21.15.29:0/2486263265 learned_addr learned my addr 172.21.15.29:0/2486263265 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41480640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c160420 0x7f7e3c0c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 -- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 msgr2=0x7f7e3c0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 -- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c160420 msgr2=0x7f7e3c0c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c160420 0x7f7e3c0c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41c81640 1 -- 172.21.15.29:0/2486263265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e380094d0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.073+0000 7f7e41480640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c160420 0x7f7e3c0c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.074+0000 7f7e41c81640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 0x7f7e3c0c8810 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009870 tx=0x7f7e380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.074+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3803ac60 con 0x7f7e3c0adc40 2023-10-16T14:35:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.074+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3803ade0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.074+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c0c34d0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.074+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38044630 con 0x7f7e3c0adc40 2023-10-16T14:35:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.074+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c14e1c0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.075+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c04e160 con 0x7f7e3c0adc40 2023-10-16T14:35:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.076+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e3802f050 con 0x7f7e3c0adc40 2023-10-16T14:35:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.077+0000 7f7e327fc640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f7e2006af90 0x7f7e2006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.077+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e380aba70 con 0x7f7e3c0adc40 2023-10-16T14:35:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.078+0000 7f7e41480640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f7e2006af90 0x7f7e2006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.079+0000 7f7e41480640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f7e2006af90 0x7f7e2006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c006fd0 tx=0x7f7e2c008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:34.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.081+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e380769b0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.313+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e3c06b6c0 con 0x7f7e3c0adc40 2023-10-16T14:35:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.315+0000 7f7e327fc640 1 -- 172.21.15.29:0/2486263265 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f7e38035090 con 0x7f7e3c0adc40 2023-10-16T14:35:34.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:34.319 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:35:27.247546+0000","modified":"2023-10-16T14:35:30.329395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7812":{"gid":7812,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2882189329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2882189329},{"type":"v1","addr":"172.21.15.29:6835","nonce":2882189329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T14:35:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2023-10-16T14:35:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.317+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f7e2006af90 msgr2=0x7f7e2006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.317+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f7e2006af90 0x7f7e2006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c006fd0 tx=0x7f7e2c008040 comp rx=0 tx=0).stop 2023-10-16T14:35:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.317+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 msgr2=0x7f7e3c0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.317+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 0x7f7e3c0c8810 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009870 tx=0x7f7e380046c0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 shutdown_connections 2023-10-16T14:35:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f7e2006af90 0x7f7e2006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c160800 0x7f7e3c0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c160420 0x7f7e3c0c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 --2- 172.21.15.29:0/2486263265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0adc40 0x7f7e3c0c8810 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 >> 172.21.15.29:0/2486263265 conn(0x7f7e3c0b0d30 msgr2=0x7f7e3c0b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 shutdown_connections 2023-10-16T14:35:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.318+0000 7f7e43ee5640 1 -- 172.21.15.29:0/2486263265 wait complete. 2023-10-16T14:35:34.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:35:27.247546+0000', 'modified': '2023-10-16T14:35:30.329395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7812': {'gid': 7812, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/2882189329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 2882189329}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2882189329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:35:34.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:35:34.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:35:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.985+0000 7fe29d189640 1 Processor -- start 2023-10-16T14:35:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.985+0000 7fe29d189640 1 -- start start 2023-10-16T14:35:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.986+0000 7fe29d189640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe29815d150 0x7fe2980ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.986+0000 7fe29d189640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe298160b90 0x7fe298160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.986+0000 7fe29d189640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe29806a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:34.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29d189640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29815d530 con 0x7fe298160b90 2023-10-16T14:35:34.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29d189640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29806a730 con 0x7fe29815d150 2023-10-16T14:35:34.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29d189640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980ac890 con 0x7fe298069da0 2023-10-16T14:35:34.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe298160b90 0x7fe298160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe298160b90 0x7fe298160fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37632/0 (socket says 172.21.15.29:37632) 2023-10-16T14:35:34.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29659c640 1 -- 172.21.15.29:0/4094638797 learned_addr learned my addr 172.21.15.29:0/4094638797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:34.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe296d9d640 1 --2- 172.21.15.29:0/4094638797 >> v2:172.21.15.170:3300/0 conn(0x7fe29815d150 0x7fe2980ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.987+0000 7fe29759e640 1 --2- 172.21.15.29:0/4094638797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe29806a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.988+0000 7fe29659c640 1 -- 172.21.15.29:0/4094638797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 msgr2=0x7fe29806a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.988+0000 7fe29659c640 1 --2- 172.21.15.29:0/4094638797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe29806a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.988+0000 7fe29659c640 1 -- 172.21.15.29:0/4094638797 >> v2:172.21.15.170:3300/0 conn(0x7fe29815d150 msgr2=0x7fe2980ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.988+0000 7fe29659c640 1 --2- 172.21.15.29:0/4094638797 >> v2:172.21.15.170:3300/0 conn(0x7fe29815d150 0x7fe2980ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.988+0000 7fe29659c640 1 -- 172.21.15.29:0/4094638797 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe298161650 con 0x7fe298160b90 2023-10-16T14:35:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.989+0000 7fe29659c640 1 --2- 172.21.15.29:0/4094638797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298160fe0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fe2980b15a0 tx=0x7fe28c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a87f1531be9d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.989+0000 7fe295d9b640 1 -- 172.21.15.29:0/4094638797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0317d0 con 0x7fe298160b90 2023-10-16T14:35:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.989+0000 7fe295d9b640 1 -- 172.21.15.29:0/4094638797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28c031930 con 0x7fe298160b90 2023-10-16T14:35:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.989+0000 7fe295d9b640 1 -- 172.21.15.29:0/4094638797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c031cc0 con 0x7fe298160b90 2023-10-16T14:35:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 -- 172.21.15.29:0/4094638797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 msgr2=0x7fe298160fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 --2- 172.21.15.29:0/4094638797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298160fe0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fe2980b15a0 tx=0x7fe28c02b870 comp rx=0 tx=0).stop 2023-10-16T14:35:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 -- 172.21.15.29:0/4094638797 shutdown_connections 2023-10-16T14:35:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 --2- 172.21.15.29:0/4094638797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe29806a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 --2- 172.21.15.29:0/4094638797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298160fe0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 --2- 172.21.15.29:0/4094638797 >> v2:172.21.15.170:3300/0 conn(0x7fe29815d150 0x7fe2980ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 -- 172.21.15.29:0/4094638797 >> 172.21.15.29:0/4094638797 conn(0x7fe2980ce9b0 msgr2=0x7fe2980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 -- 172.21.15.29:0/4094638797 shutdown_connections 2023-10-16T14:35:34.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.990+0000 7fe29d189640 1 -- 172.21.15.29:0/4094638797 wait complete. 2023-10-16T14:35:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.992+0000 7fe29d189640 1 Processor -- start 2023-10-16T14:35:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.992+0000 7fe29d189640 1 -- start start 2023-10-16T14:35:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29d189640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe2980aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29d189640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29815d150 0x7fe29815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29d189640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29d189640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980c24e0 con 0x7fe298160b90 2023-10-16T14:35:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29d189640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980c3020 con 0x7fe29815d150 2023-10-16T14:35:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29d189640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980c26f0 con 0x7fe298069da0 2023-10-16T14:35:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37648/0 (socket says 172.21.15.29:37648) 2023-10-16T14:35:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.993+0000 7fe29759e640 1 -- 172.21.15.29:0/2556001609 learned_addr learned my addr 172.21.15.29:0/2556001609 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29659c640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29815d150 0x7fe29815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe296d9d640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe2980aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29759e640 1 -- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 msgr2=0x7fe2980aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29759e640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe2980aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29759e640 1 -- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29815d150 msgr2=0x7fe29815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29759e640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29815d150 0x7fe29815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29759e640 1 -- 172.21.15.29:0/2556001609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c0094d0 con 0x7fe298160b90 2023-10-16T14:35:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe29759e640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298155890 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe29000eed0 tx=0x7fe29000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.994+0000 7fe27f7fe640 1 -- 172.21.15.29:0/2556001609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290010040 con 0x7fe298160b90 2023-10-16T14:35:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.995+0000 7fe27f7fe640 1 -- 172.21.15.29:0/2556001609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29000bd20 con 0x7fe298160b90 2023-10-16T14:35:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.995+0000 7fe27f7fe640 1 -- 172.21.15.29:0/2556001609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290014640 con 0x7fe298160b90 2023-10-16T14:35:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.995+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2980c29d0 con 0x7fe298160b90 2023-10-16T14:35:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.995+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29815dcc0 con 0x7fe298160b90 2023-10-16T14:35:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.997+0000 7fe27f7fe640 1 -- 172.21.15.29:0/2556001609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2900147a0 con 0x7fe298160b90 2023-10-16T14:35:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.998+0000 7fe27f7fe640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.998+0000 7fe27f7fe640 1 -- 172.21.15.29:0/2556001609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe290089fb0 con 0x7fe298160b90 2023-10-16T14:35:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.999+0000 7fe296d9d640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.999+0000 7fe29fdf2740 1 Processor -- start 2023-10-16T14:35:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:34.999+0000 7fe29fdf2740 1 -- start start 2023-10-16T14:35:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe296d9d640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe288009a30 tx=0x7fe288008040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a739be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 0x5639a73cc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a72bc7a0 con 0x5639a739c3c0 2023-10-16T14:35:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a730e740 con 0x5639a73c9360 2023-10-16T14:35:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639a730be30 con 0x5639a73c8f80 2023-10-16T14:35:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 0x5639a73cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 0x5639a73cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37664/0 (socket says 172.21.15.29:37664) 2023-10-16T14:35:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe29759e640 1 -- 172.21.15.29:0/3931124436 learned_addr learned my addr 172.21.15.29:0/3931124436 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.000+0000 7fe296d9d640 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a739be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29659c640 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29759e640 1 -- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 msgr2=0x5639a739be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29759e640 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a739be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29759e640 1 -- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 msgr2=0x5639a73cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29759e640 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29759e640 1 -- 172.21.15.29:0/3931124436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639a72cb910 con 0x5639a739c3c0 2023-10-16T14:35:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe29759e640 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 0x5639a73cc250 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe2900589f0 tx=0x7fe2900592a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb906ba77ed998af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe27cff9640 1 -- 172.21.15.29:0/3931124436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290088430 con 0x5639a739c3c0 2023-10-16T14:35:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe27cff9640 1 -- 172.21.15.29:0/3931124436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290083070 con 0x5639a739c3c0 2023-10-16T14:35:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.001+0000 7fe27cff9640 1 -- 172.21.15.29:0/3931124436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290088590 con 0x5639a739c3c0 2023-10-16T14:35:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3931124436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 msgr2=0x5639a73cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 0x5639a73cc250 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe2900589f0 tx=0x7fe2900592a0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3931124436 shutdown_connections 2023-10-16T14:35:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a739c3c0 0x5639a73cc250 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3931124436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a739be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3931124436 >> 172.21.15.29:0/3931124436 conn(0x5639a739e190 msgr2=0x5639a73e6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3931124436 shutdown_connections 2023-10-16T14:35:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.002+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3931124436 wait complete. 2023-10-16T14:35:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.003+0000 7fe29fdf2740 1 -- start start 2023-10-16T14:35:35.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.003+0000 7fe29fdf2740 1 Processor -- start 2023-10-16T14:35:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73de7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73cf4d0 0x5639a73cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a7307720 con 0x5639a73c9360 2023-10-16T14:35:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a72a5970 con 0x5639a73cf4d0 2023-10-16T14:35:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639a72a5610 con 0x5639a73c8f80 2023-10-16T14:35:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29fdf2740 10 client.0 _ll_register_callbacks cb 0x5639a72c79d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:35:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73de7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73de7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37672/0 (socket says 172.21.15.29:37672) 2023-10-16T14:35:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.004+0000 7fe29659c640 1 -- 172.21.15.29:0/3786681587 learned_addr learned my addr 172.21.15.29:0/3786681587 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe296d9d640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29759e640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73cf4d0 0x5639a73cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29659c640 1 -- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 msgr2=0x5639a73c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29659c640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29659c640 1 -- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73cf4d0 msgr2=0x5639a73cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29659c640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73cf4d0 0x5639a73cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29659c640 1 -- 172.21.15.29:0/3786681587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29815dcc0 con 0x5639a73c9360 2023-10-16T14:35:35.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29759e640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73cf4d0 0x5639a73cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29659c640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73de7d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe28c02b8b0 tx=0x7fe28c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5afa8c2282579c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:35:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe275ffb640 1 -- 172.21.15.29:0/3786681587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c032870 con 0x5639a73c9360 2023-10-16T14:35:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe275ffb640 1 -- 172.21.15.29:0/3786681587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe28c040040 con 0x5639a73c9360 2023-10-16T14:35:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.005+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639a732e160 con 0x5639a73c9360 2023-10-16T14:35:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe275ffb640 1 -- 172.21.15.29:0/3786681587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe28c031ed0 con 0x5639a73c9360 2023-10-16T14:35:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 10 client.7947 fetch_fsmap learned FSMap version 58 2023-10-16T14:35:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x5639a7305da0 con 0x5639a73c9360 2023-10-16T14:35:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe275ffb640 1 -- 172.21.15.29:0/3786681587 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5639a732e160 con 0x5639a73c9360 2023-10-16T14:35:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe275ffb640 1 -- 172.21.15.29:0/3786681587 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 58) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe28c04d060 con 0x5639a73c9360 2023-10-16T14:35:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 10 client.7947 fetch_fsmap finished waiting for FSMap version 58 2023-10-16T14:35:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 10 client.7947 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:35:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 20 client.7947 populate_metadata read hostname 'smithi029' 2023-10-16T14:35:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73e0130 0x5639a73cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 4 client.7947 mds_command: new command op to 7812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:35:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.006+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 --> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639a72d9cc0 con 0x5639a73e0130 2023-10-16T14:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.007+0000 7fe296d9d640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73e0130 0x5639a73cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.007+0000 7fe296d9d640 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73e0130 0x5639a73cda30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8adf3e6661b4559 server_cookie=fe5fc9e58c04e58c in_seq=0 out_seq=0 2023-10-16T14:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.007+0000 7fe275ffb640 10 client.7947 ms_handle_connect on v2:172.21.15.29:6833/2882189329 2023-10-16T14:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe275ffb640 1 -- 172.21.15.29:0/3786681587 <== mds.0 v2:172.21.15.29:6833/2882189329 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5639a72d9cc0 con 0x5639a73e0130 2023-10-16T14:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe275ffb640 10 client.7947 handle_command_reply: tid=0 2023-10-16T14:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 client.7947 shutdown 2023-10-16T14:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 msgr2=0x5639a73de7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73de7d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe28c02b8b0 tx=0x7fe28c004730 comp rx=0 tx=0).stop 2023-10-16T14:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 shutdown_connections 2023-10-16T14:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73e0130 0x5639a73cda30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73de7d0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73cf4d0 0x5639a73cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe275ffb640 0 client.7947 ms_handle_reset on v2:172.21.15.29:6833/2882189329 2023-10-16T14:35:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3786681587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 >> 172.21.15.29:0/3786681587 conn(0x5639a739c3c0 msgr2=0x5639a739fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.008+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 shutdown_connections 2023-10-16T14:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.009+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3786681587 wait complete. 2023-10-16T14:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.009+0000 7fe29fdf2740 20 client.7947 trim_cache size 0 max 16384 2023-10-16T14:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.025+0000 7fe29fdf2740 1 Processor -- start 2023-10-16T14:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 -- start start 2023-10-16T14:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73f6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 0x5639a73cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a706bf20 con 0x5639a73cf4d0 2023-10-16T14:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a73c6ec0 con 0x5639a73c9360 2023-10-16T14:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.026+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639a73c78c0 con 0x5639a73c8f80 2023-10-16T14:35:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 0x5639a73cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 0x5639a73cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37688/0 (socket says 172.21.15.29:37688) 2023-10-16T14:35:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 -- 172.21.15.29:0/1003340887 learned_addr learned my addr 172.21.15.29:0/1003340887 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29659c640 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe296d9d640 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 -- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 msgr2=0x5639a73f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 -- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 msgr2=0x5639a73c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe29759e640 1 -- 172.21.15.29:0/1003340887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639a72a4b80 con 0x5639a73cf4d0 2023-10-16T14:35:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.027+0000 7fe296d9d640 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe29759e640 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 0x5639a73cfaa0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe2900589a0 tx=0x7fe290089000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea27a615f97706ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe27d7fa640 1 -- 172.21.15.29:0/1003340887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2900585a0 con 0x5639a73cf4d0 2023-10-16T14:35:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe27d7fa640 1 -- 172.21.15.29:0/1003340887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290056e00 con 0x5639a73cf4d0 2023-10-16T14:35:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe27d7fa640 1 -- 172.21.15.29:0/1003340887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290055170 con 0x5639a73cf4d0 2023-10-16T14:35:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe29fdf2740 1 -- 172.21.15.29:0/1003340887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 msgr2=0x5639a73cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 0x5639a73cfaa0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe2900589a0 tx=0x7fe290089000 comp rx=0 tx=0).stop 2023-10-16T14:35:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.028+0000 7fe29fdf2740 1 -- 172.21.15.29:0/1003340887 shutdown_connections 2023-10-16T14:35:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73cf4d0 0x5639a73cfaa0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c9360 0x5639a73c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/1003340887 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73c8f80 0x5639a73f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 -- 172.21.15.29:0/1003340887 >> 172.21.15.29:0/1003340887 conn(0x5639a739c3c0 msgr2=0x5639a74091d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 -- 172.21.15.29:0/1003340887 shutdown_connections 2023-10-16T14:35:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 -- 172.21.15.29:0/1003340887 wait complete. 2023-10-16T14:35:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.029+0000 7fe29fdf2740 1 -- start start 2023-10-16T14:35:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.030+0000 7fe29fdf2740 1 Processor -- start 2023-10-16T14:35:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.030+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c8f80 0x5639a73dd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.030+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.030+0000 7fe29fdf2740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73cf4d0 0x5639a73e84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a73e3140 con 0x5639a73c9360 2023-10-16T14:35:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639a74506f0 con 0x5639a73c8f80 2023-10-16T14:35:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29fdf2740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639a73dddd0 con 0x5639a73cf4d0 2023-10-16T14:35:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29fdf2740 10 client.0 _ll_register_callbacks cb 0x7fe288009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:35:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73e7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37690/0 (socket says 172.21.15.29:37690) 2023-10-16T14:35:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 -- 172.21.15.29:0/3768319086 learned_addr learned my addr 172.21.15.29:0/3768319086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 -- 172.21.15.29:0/3768319086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73cf4d0 msgr2=0x5639a73e84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73cf4d0 0x5639a73e84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 -- 172.21.15.29:0/3768319086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c8f80 msgr2=0x5639a73dd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c8f80 0x5639a73dd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 -- 172.21.15.29:0/3768319086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639a72cb910 con 0x5639a73c9360 2023-10-16T14:35:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe29659c640 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73e7f60 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe28c031900 tx=0x7fe28c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee692fd5bde5dae6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.031+0000 7fe275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:35:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe275ffb640 1 -- 172.21.15.29:0/3768319086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c004240 con 0x5639a73c9360 2023-10-16T14:35:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe275ffb640 1 -- 172.21.15.29:0/3768319086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe28c040070 con 0x5639a73c9360 2023-10-16T14:35:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe290083070 con 0x5639a73c9360 2023-10-16T14:35:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe275ffb640 1 -- 172.21.15.29:0/3768319086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe28c032e40 con 0x5639a73c9360 2023-10-16T14:35:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 10 client.7953 fetch_fsmap learned FSMap version 58 2023-10-16T14:35:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x5639a72ca170 con 0x5639a73c9360 2023-10-16T14:35:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe275ffb640 1 -- 172.21.15.29:0/3768319086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290083070 con 0x5639a73c9360 2023-10-16T14:35:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe275ffb640 1 -- 172.21.15.29:0/3768319086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 58) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe28c0043a0 con 0x5639a73c9360 2023-10-16T14:35:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 10 client.7953 fetch_fsmap finished waiting for FSMap version 58 2023-10-16T14:35:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 10 client.7953 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:35:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 20 client.7953 populate_metadata read hostname 'smithi029' 2023-10-16T14:35:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73dca80 0x5639a73dce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 4 client.7953 mds_command: new command op to 7812 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:35:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.032+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 --> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5639a7300700 con 0x5639a73dca80 2023-10-16T14:35:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.033+0000 7fe296d9d640 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73dca80 0x5639a73dce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.033+0000 7fe296d9d640 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73dca80 0x5639a73dce60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f1a27e1400bfbd server_cookie=6269be810f0ec17b in_seq=0 out_seq=0 2023-10-16T14:35:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.033+0000 7fe275ffb640 10 client.7953 ms_handle_connect on v2:172.21.15.29:6833/2882189329 2023-10-16T14:35:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.033+0000 7fe275ffb640 1 -- 172.21.15.29:0/3768319086 <== mds.0 v2:172.21.15.29:6833/2882189329 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5639a7300700 con 0x5639a73dca80 2023-10-16T14:35:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.033+0000 7fe275ffb640 10 client.7953 handle_command_reply: tid=0 2023-10-16T14:35:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.033+0000 7fe29fdf2740 1 client.7953 shutdown 2023-10-16T14:35:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 msgr2=0x5639a73e7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73e7f60 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fe28c031900 tx=0x7fe28c031000 comp rx=0 tx=0).stop 2023-10-16T14:35:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 shutdown_connections 2023-10-16T14:35:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:6833/2882189329,v1:172.21.15.29:6835/2882189329] conn(0x5639a73dca80 0x5639a73dce60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5639a73c9360 0x5639a73e7f60 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5639a73cf4d0 0x5639a73e84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 --2- 172.21.15.29:0/3768319086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5639a73c8f80 0x5639a73dd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 >> 172.21.15.29:0/3768319086 conn(0x5639a739c3c0 msgr2=0x5639a73f6280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe275ffb640 0 client.7953 ms_handle_reset on v2:172.21.15.29:6833/2882189329 2023-10-16T14:35:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 shutdown_connections 2023-10-16T14:35:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 1 -- 172.21.15.29:0/3768319086 wait complete. 2023-10-16T14:35:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29fdf2740 20 client.7953 trim_cache size 0 max 16384 2023-10-16T14:35:35.051 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:35:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:35:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:35:35.051 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7812, 2023-10-16T14:35:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:35:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:35:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:35:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 7.2276530599999997, 2023-10-16T14:35:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 58, 2023-10-16T14:35:35.052 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 127, 2023-10-16T14:35:35.053 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 127, 2023-10-16T14:35:35.053 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.600724639999999 2023-10-16T14:35:35.053 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:35:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 msgr2=0x7fe28406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29d189640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe288009a30 tx=0x7fe288008040 comp rx=0 tx=0).stop 2023-10-16T14:35:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 msgr2=0x7fe298155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.034+0000 7fe29d189640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298155890 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe29000eed0 tx=0x7fe29000b480 comp rx=0 tx=0).stop 2023-10-16T14:35:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 shutdown_connections 2023-10-16T14:35:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fe28406af90 0x7fe28406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe298160b90 0x7fe298155890 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe29815d150 0x7fe29815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 --2- 172.21.15.29:0/2556001609 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe298069da0 0x7fe2980aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 >> 172.21.15.29:0/2556001609 conn(0x7fe2980ce9b0 msgr2=0x7fe298158c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 shutdown_connections 2023-10-16T14:35:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.035+0000 7fe29d189640 1 -- 172.21.15.29:0/2556001609 wait complete. 2023-10-16T14:35:35.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:35:35.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.260+0000 7f415a213640 1 Processor -- start 2023-10-16T14:35:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.261+0000 7f415a213640 1 -- start start 2023-10-16T14:35:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.262+0000 7f415a213640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 0x7f41540cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.262+0000 7f415a213640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415415b9d0 con 0x7f415406c040 2023-10-16T14:35:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.262+0000 7f415a213640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415415bcb0 con 0x7f41540ad970 2023-10-16T14:35:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.262+0000 7f415a213640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415415b7d0 con 0x7f415415b320 2023-10-16T14:35:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.262+0000 7f4159211640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f415406c040 0x7f415406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59694/0 (socket says 172.21.15.29:59694) 2023-10-16T14:35:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.262+0000 7f4159211640 1 -- 172.21.15.29:0/2283300024 learned_addr learned my addr 172.21.15.29:0/2283300024 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f4159a12640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 0x7f41540cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211368531 0 0) 0x7f415415b9d0 con 0x7f415406c040 2023-10-16T14:35:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140003520 con 0x7f415406c040 2023-10-16T14:35:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3724646143 0 0) 0x7f4140003520 con 0x7f415406c040 2023-10-16T14:35:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 msgr2=0x7f41540cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 0x7f41540cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 >> v1:172.21.15.170:6789/0 conn(0x7f41540ad970 legacy=0x7f41540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415415bf90 con 0x7f415406c040 2023-10-16T14:35:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.263+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f41500019f0 con 0x7f415406c040 2023-10-16T14:35:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 >> v1:172.21.15.29:6789/0 conn(0x7f415406c040 legacy=0x7f415406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41400039a0 0x7f4140003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41400047c0 0x7f4140004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 0x7f41400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415415b7d0 con 0x7f41400039a0 2023-10-16T14:35:35.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414000bae0 con 0x7f41400047c0 2023-10-16T14:35:35.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414000bca0 con 0x7f415415b320 2023-10-16T14:35:35.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159211640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41400039a0 0x7f4140003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159211640 1 -- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 msgr2=0x7f41400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159a12640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 0x7f41400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4158a10640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41400047c0 0x7f4140004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159211640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f415415b320 0x7f41400051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159211640 1 -- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41400047c0 msgr2=0x7f4140004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159211640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41400047c0 0x7f4140004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4159211640 1 -- 172.21.15.29:0/2283300024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140003520 con 0x7f41400039a0 2023-10-16T14:35:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.264+0000 7f4158a10640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41400047c0 0x7f4140004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.265+0000 7f4159211640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41400039a0 0x7f4140003dc0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f415000a830 tx=0x7f41400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2169cfe0cdf69df7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41500072d0 con 0x7f41400039a0 2023-10-16T14:35:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415002f070 con 0x7f41400039a0 2023-10-16T14:35:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f414bfff640 1 -- 172.21.15.29:0/2283300024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150007670 con 0x7f41400039a0 2023-10-16T14:35:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f415a213640 1 -- 172.21.15.29:0/2283300024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41400039a0 msgr2=0x7f4140003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f415a213640 1 --2- 172.21.15.29:0/2283300024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41400039a0 0x7f4140003dc0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f415000a830 tx=0x7f41400057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f4159a12640 1 -- 172.21.15.29:0/2283300024 reap_dead start 2023-10-16T14:35:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f415a213640 1 -- 172.21.15.29:0/2283300024 shutdown_connections 2023-10-16T14:35:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f415a213640 1 -- 172.21.15.29:0/2283300024 >> 172.21.15.29:0/2283300024 conn(0x7f41540ce6a0 msgr2=0x7f41540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f415a213640 1 -- 172.21.15.29:0/2283300024 shutdown_connections 2023-10-16T14:35:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.266+0000 7f415a213640 1 -- 172.21.15.29:0/2283300024 wait complete. 2023-10-16T14:35:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.268+0000 7f415a213640 1 Processor -- start 2023-10-16T14:35:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.269+0000 7f415a213640 1 -- start start 2023-10-16T14:35:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.269+0000 7f415a213640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41400039a0 0x7f41541edbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.269+0000 7f415a213640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540c67c0 0x7f41540c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.269+0000 7f415a213640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 0x7f41540ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f415a213640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c7fb0 con 0x7f41540ae100 2023-10-16T14:35:35.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f415a213640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41540aeac0 con 0x7f41540c67c0 2023-10-16T14:35:35.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f415a213640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c7330 con 0x7f41400039a0 2023-10-16T14:35:35.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 0x7f41540ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 0x7f41540ae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37714/0 (socket says 172.21.15.29:37714) 2023-10-16T14:35:35.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 -- 172.21.15.29:0/1602868658 learned_addr learned my addr 172.21.15.29:0/1602868658 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:35.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159211640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41400039a0 0x7f41541edbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4158a10640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540c67c0 0x7f41540c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 -- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41400039a0 msgr2=0x7f41541edbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41400039a0 0x7f41541edbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 -- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540c67c0 msgr2=0x7f41540c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540c67c0 0x7f41540c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.270+0000 7f4159a12640 1 -- 172.21.15.29:0/1602868658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41540c7610 con 0x7f41540ae100 2023-10-16T14:35:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.271+0000 7f4159a12640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 0x7f41540ae580 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f41540b1260 tx=0x7f414c00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.271+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c00c070 con 0x7f41540ae100 2023-10-16T14:35:35.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.271+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c013610 con 0x7f41540ae100 2023-10-16T14:35:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.271+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41540c78f0 con 0x7f41540ae100 2023-10-16T14:35:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.271+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c0139c0 con 0x7f41540ae100 2023-10-16T14:35:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.271+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415414e330 con 0x7f41540ae100 2023-10-16T14:35:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.272+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415406b330 con 0x7f41540ae100 2023-10-16T14:35:35.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.273+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f414c013b20 con 0x7f41540ae100 2023-10-16T14:35:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.274+0000 7f4149ffb640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413806afc0 0x7f413806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.274+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f414c088f40 con 0x7f41540ae100 2023-10-16T14:35:35.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.275+0000 7f4159211640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413806afc0 0x7f413806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:35.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.276+0000 7f4159211640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413806afc0 0x7f413806d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4150034000 tx=0x7f4150007000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:35.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.279+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f414c053dc0 con 0x7f41540ae100 2023-10-16T14:35:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.522+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f41540c80f0 con 0x7f41540ae100 2023-10-16T14:35:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.523+0000 7f4149ffb640 1 -- 172.21.15.29:0/1602868658 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v58) v1 ==== 53+0+73 (secure 0 0 0) 0x7f414c017060 con 0x7f41540ae100 2023-10-16T14:35:35.527 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:35:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413806afc0 msgr2=0x7f413806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413806afc0 0x7f413806d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4150034000 tx=0x7f4150007000 comp rx=0 tx=0).stop 2023-10-16T14:35:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 msgr2=0x7f41540ae580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 0x7f41540ae580 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f41540b1260 tx=0x7f414c00e8b0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 shutdown_connections 2023-10-16T14:35:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f413806afc0 0x7f413806d480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540ae100 0x7f41540ae580 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540c67c0 0x7f41540c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 --2- 172.21.15.29:0/1602868658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41400039a0 0x7f41541edbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:35.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.525+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 >> 172.21.15.29:0/1602868658 conn(0x7f41540ce6a0 msgr2=0x7f41540cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:35.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.526+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 shutdown_connections 2023-10-16T14:35:35.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:35.526+0000 7f415a213640 1 -- 172.21.15.29:0/1602868658 wait complete. 2023-10-16T14:35:35.548 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:35:35.549 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:35:35.549 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:35:35.549 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:35:35.550 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:35:35.550 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:35:35.550 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:35:35.551 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:35:35.551 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:35:35.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:35.552 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:35:35.607 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:35:35.607 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:35:35.608 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:35:35.608 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.608 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:35:35.608 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.608 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:35:35.609 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:35:35.609 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:35:35.609 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:35:35.609 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:35:35.609 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:35:35.610 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:35:35.610 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:35:35.610 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:35:35.610 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:35:35.610 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41189sec preferred_lft 41189sec 2023-10-16T14:35:35.611 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:35:35.611 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.611 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:35:35.611 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:35:35.611 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:35:35.612 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:35:35.612 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:35:35.612 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:35:35.612 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.612 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:35:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.613 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:35:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:35:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:35:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.614 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:35:35.614 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:35:35.614 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:35:35.614 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:35:35.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:35.616 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:35:35.717 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:35:35.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:35.722 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:35:35.781 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:35:35.782 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:35:35.782 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:35:35.782 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:35:35.782 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:35:35.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:35.784 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:35:35.784 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:35:35.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:35:35.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:35:35.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:35:35.785 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:35:35.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:35.989 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:35:36.045 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:35:36.045 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:35:36.047 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:35:36.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:36.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:35:36.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:35:36.149 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:35:36.209 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:35:36.209 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:35:36.210 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:35:36.265 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:35:36.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:36.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:35:36.420 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:35:36.421 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:35:36.421 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:35:36.443 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:35:36.443 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:35:36.443 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:35:36.443 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:35:36.444 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:35:36.470 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:35:36.471 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:35:36.471 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:35:36.471 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:35:36.471 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:35:36.471 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:35:36.472 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:35:36.472 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:35:36.475 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:35:36.476 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:35:36.476 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:35:36.476 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:35:36.476 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:35:36.481 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:35:36.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:36.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:35:36.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:36.577 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:35:36.643 INFO:teuthology.orchestra.run.smithi029.stdout:client6131 2023-10-16T14:35:36.643 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:35:36.644 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:35:36.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:35:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.967+0000 7f37808c9640 1 Processor -- start 2023-10-16T14:35:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.968+0000 7f37808c9640 1 -- start start 2023-10-16T14:35:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.969+0000 7f37808c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 0x7f377c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.969+0000 7f37808c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f377c0c9ca0 0x7f377c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.969+0000 7f37808c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0cb030 con 0x7f377c0ca3b0 2023-10-16T14:35:36.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.969+0000 7f37808c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c0c91c0 con 0x7f377c0c9ca0 2023-10-16T14:35:36.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.969+0000 7f37808c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0c94a0 con 0x7f377c15ad70 2023-10-16T14:35:36.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.970+0000 7f377a59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 0x7f377c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.970+0000 7f377a59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 0x7f377c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:52074/0 (socket says 172.21.15.29:52074) 2023-10-16T14:35:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.970+0000 7f377a59c640 1 -- 172.21.15.29:0/3875983004 learned_addr learned my addr 172.21.15.29:0/3875983004 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.970+0000 7f3773fff640 1 --2- 172.21.15.29:0/3875983004 >> v2:172.21.15.170:3300/0 conn(0x7f377c0c9ca0 0x7f377c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:36.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.971+0000 7f377a59c640 1 -- 172.21.15.29:0/3875983004 >> v2:172.21.15.170:3300/0 conn(0x7f377c0c9ca0 msgr2=0x7f377c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.971+0000 7f377a59c640 1 --2- 172.21.15.29:0/3875983004 >> v2:172.21.15.170:3300/0 conn(0x7f377c0c9ca0 0x7f377c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.971+0000 7f377a59c640 1 -- 172.21.15.29:0/3875983004 >> v1:172.21.15.170:6789/0 conn(0x7f377c0ca3b0 legacy=0x7f377c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:36.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.971+0000 7f377a59c640 1 -- 172.21.15.29:0/3875983004 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0c9780 con 0x7f377c15ad70 2023-10-16T14:35:36.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.972+0000 7f377a59c640 1 --2- 172.21.15.29:0/3875983004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 0x7f377c069da0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f376c009870 tx=0x7f376c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95069fa0f79bebf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f3779d9b640 1 -- 172.21.15.29:0/3875983004 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c030840 con 0x7f377c15ad70 2023-10-16T14:35:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f3779d9b640 1 -- 172.21.15.29:0/3875983004 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c0309a0 con 0x7f377c15ad70 2023-10-16T14:35:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f3779d9b640 1 -- 172.21.15.29:0/3875983004 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c030d30 con 0x7f377c15ad70 2023-10-16T14:35:36.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f37808c9640 1 -- 172.21.15.29:0/3875983004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 msgr2=0x7f377c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f37808c9640 1 --2- 172.21.15.29:0/3875983004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 0x7f377c069da0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f376c009870 tx=0x7f376c02b980 comp rx=0 tx=0).stop 2023-10-16T14:35:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f37808c9640 1 -- 172.21.15.29:0/3875983004 shutdown_connections 2023-10-16T14:35:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f37808c9640 1 --2- 172.21.15.29:0/3875983004 >> v2:172.21.15.170:3300/0 conn(0x7f377c0c9ca0 0x7f377c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f37808c9640 1 --2- 172.21.15.29:0/3875983004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c15ad70 0x7f377c069da0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.973+0000 7f37808c9640 1 -- 172.21.15.29:0/3875983004 >> 172.21.15.29:0/3875983004 conn(0x7f377c0ce9b0 msgr2=0x7f377c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.974+0000 7f37808c9640 1 -- 172.21.15.29:0/3875983004 shutdown_connections 2023-10-16T14:35:36.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.974+0000 7f37808c9640 1 -- 172.21.15.29:0/3875983004 wait complete. 2023-10-16T14:35:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.975+0000 7f37808c9640 1 Processor -- start 2023-10-16T14:35:36.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.976+0000 7f37808c9640 1 -- start start 2023-10-16T14:35:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f37808c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c0c9ca0 0x7f377c14a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f37808c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 0x7f377c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f37808c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f377c15ad70 0x7f377c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:36.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f37808c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1f6900 con 0x7f377c0ca3b0 2023-10-16T14:35:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f37808c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c14a6f0 con 0x7f377c15ad70 2023-10-16T14:35:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f37808c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1f5c80 con 0x7f377c0c9ca0 2023-10-16T14:35:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f3773fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 0x7f377c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f3773fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 0x7f377c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37716/0 (socket says 172.21.15.29:37716) 2023-10-16T14:35:36.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f3773fff640 1 -- 172.21.15.29:0/1106061857 learned_addr learned my addr 172.21.15.29:0/1106061857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f3773fff640 1 -- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c0c9ca0 msgr2=0x7f377c14a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f377a59c640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c0c9ca0 0x7f377c14a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f377ad9d640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f377c15ad70 0x7f377c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:36.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f3773fff640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c0c9ca0 0x7f377c14a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.977+0000 7f3773fff640 1 -- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f377c15ad70 msgr2=0x7f377c0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f3773fff640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f377c15ad70 0x7f377c0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f3773fff640 1 -- 172.21.15.29:0/1106061857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0094d0 con 0x7f377c0ca3b0 2023-10-16T14:35:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f377ad9d640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f377c15ad70 0x7f377c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f3773fff640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 0x7f377c0ac890 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f376400e9b0 tx=0x7f376400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764009670 con 0x7f377c0ca3b0 2023-10-16T14:35:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376400c9a0 con 0x7f377c0ca3b0 2023-10-16T14:35:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c1f5f60 con 0x7f377c0ca3b0 2023-10-16T14:35:36.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376400cd50 con 0x7f377c0ca3b0 2023-10-16T14:35:36.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.978+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c1f6520 con 0x7f377c0ca3b0 2023-10-16T14:35:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.979+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c1f6a40 con 0x7f377c0ca3b0 2023-10-16T14:35:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.980+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3764013470 con 0x7f377c0ca3b0 2023-10-16T14:35:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.981+0000 7f3772ffd640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f375c06af90 0x7f375c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:36.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.981+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37640899b0 con 0x7f377c0ca3b0 2023-10-16T14:35:36.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.982+0000 7f377a59c640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f375c06af90 0x7f375c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:36.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.983+0000 7f377a59c640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f375c06af90 0x7f375c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f376c036000 tx=0x7f376c03b040 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:36.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:36.986+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f377c1f6a40 con 0x7f377c0ca3b0 2023-10-16T14:35:37.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.204+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 --> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f377c0ad4d0 con 0x7f375c06af90 2023-10-16T14:35:37.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.208+0000 7f3772ffd640 1 -- 172.21.15.29:0/1106061857 <== mgr.6406 v2:172.21.15.170:6832/1801486769 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f376c030b90 con 0x7f375c06af90 2023-10-16T14:35:37.212 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:35:37.212 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:35:37.212 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:35:37.212 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:35:37.213 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:35:37.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.210+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f375c06af90 msgr2=0x7f375c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:37.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.210+0000 7f37808c9640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f375c06af90 0x7f375c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f376c036000 tx=0x7f376c03b040 comp rx=0 tx=0).stop 2023-10-16T14:35:37.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.210+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 msgr2=0x7f377c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:37.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.210+0000 7f37808c9640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 0x7f377c0ac890 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f376400e9b0 tx=0x7f376400ee70 comp rx=0 tx=0).stop 2023-10-16T14:35:37.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 shutdown_connections 2023-10-16T14:35:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f375c06af90 0x7f375c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f377c0ca3b0 0x7f377c0ac890 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f377c15ad70 0x7f377c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 --2- 172.21.15.29:0/1106061857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f377c0c9ca0 0x7f377c14a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 >> 172.21.15.29:0/1106061857 conn(0x7f377c0ce9b0 msgr2=0x7f377c15c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:37.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 shutdown_connections 2023-10-16T14:35:37.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.211+0000 7f37808c9640 1 -- 172.21.15.29:0/1106061857 wait complete. 2023-10-16T14:35:37.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:35:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.459+0000 7f2d7759e640 1 Processor -- start 2023-10-16T14:35:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.459+0000 7f2d7759e640 1 -- start start 2023-10-16T14:35:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d7759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d780ad970 0x7f2d780cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d7759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7815b9d0 con 0x7f2d7806c040 2023-10-16T14:35:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d7759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7815bcb0 con 0x7f2d780ccfb0 2023-10-16T14:35:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d7759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7815bf90 con 0x7f2d780ad970 2023-10-16T14:35:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d7659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2d7806c040 0x7f2d7806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59704/0 (socket says 172.21.15.29:59704) 2023-10-16T14:35:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d75d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2d780ad970 0x7f2d780cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.461+0000 7f2d7659c640 1 -- 172.21.15.29:0/327883633 learned_addr learned my addr 172.21.15.29:0/327883633 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.462+0000 7f2d7559a640 1 -- 172.21.15.29:0/327883633 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944294010 0 0) 0x7f2d7815b9d0 con 0x7f2d7806c040 2023-10-16T14:35:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.462+0000 7f2d75d9b640 1 -- 172.21.15.29:0/327883633 >> v1:172.21.15.170:6789/0 conn(0x7f2d780ccfb0 legacy=0x7f2d780cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.462+0000 7f2d75d9b640 1 -- 172.21.15.29:0/327883633 >> v1:172.21.15.29:6789/0 conn(0x7f2d7806c040 legacy=0x7f2d7806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.462+0000 7f2d75d9b640 1 -- 172.21.15.29:0/327883633 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d780cc270 con 0x7f2d780ad970 2023-10-16T14:35:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.463+0000 7f2d75d9b640 1 --2- 172.21.15.29:0/327883633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d780ad970 0x7f2d780cca70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0097d0 tx=0x7f2d6c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8e7ab4c9fb8d45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.463+0000 7f2d7559a640 1 -- 172.21.15.29:0/327883633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c02d040 con 0x7f2d780ad970 2023-10-16T14:35:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.463+0000 7f2d7559a640 1 -- 172.21.15.29:0/327883633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c031810 con 0x7f2d780ad970 2023-10-16T14:35:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.463+0000 7f2d7559a640 1 -- 172.21.15.29:0/327883633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c031ba0 con 0x7f2d780ad970 2023-10-16T14:35:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 -- 172.21.15.29:0/327883633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d780ad970 msgr2=0x7f2d780cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 --2- 172.21.15.29:0/327883633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d780ad970 0x7f2d780cca70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0097d0 tx=0x7f2d6c02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 -- 172.21.15.29:0/327883633 shutdown_connections 2023-10-16T14:35:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 --2- 172.21.15.29:0/327883633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d780ad970 0x7f2d780cca70 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 -- 172.21.15.29:0/327883633 >> 172.21.15.29:0/327883633 conn(0x7f2d780ce6a0 msgr2=0x7f2d780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 -- 172.21.15.29:0/327883633 shutdown_connections 2023-10-16T14:35:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.464+0000 7f2d7759e640 1 -- 172.21.15.29:0/327883633 wait complete. 2023-10-16T14:35:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.466+0000 7f2d7759e640 1 Processor -- start 2023-10-16T14:35:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.466+0000 7f2d7759e640 1 -- start start 2023-10-16T14:35:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 0x7f2d781e6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d780ad970 0x7f2d78156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d780ccfb0 0x7f2d780b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780b1290 con 0x7f2d7806c040 2023-10-16T14:35:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780b1aa0 con 0x7f2d780ccfb0 2023-10-16T14:35:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d781e6a70 con 0x7f2d780ad970 2023-10-16T14:35:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 0x7f2d781e6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 0x7f2d781e6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37736/0 (socket says 172.21.15.29:37736) 2023-10-16T14:35:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 -- 172.21.15.29:0/1465097909 learned_addr learned my addr 172.21.15.29:0/1465097909 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 -- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d780ad970 msgr2=0x7f2d78156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d780ad970 0x7f2d78156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 -- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d780ccfb0 msgr2=0x7f2d780b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d780ccfb0 0x7f2d780b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.467+0000 7f2d7659c640 1 -- 172.21.15.29:0/1465097909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c009400 con 0x7f2d7806c040 2023-10-16T14:35:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.468+0000 7f2d7659c640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 0x7f2d781e6530 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c7c0 tx=0x7f2d6800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.468+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800a640 con 0x7f2d7806c040 2023-10-16T14:35:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.468+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6800a7c0 con 0x7f2d7806c040 2023-10-16T14:35:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.468+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800ab70 con 0x7f2d7806c040 2023-10-16T14:35:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.468+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d780b3d10 con 0x7f2d7806c040 2023-10-16T14:35:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.468+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d780b17c0 con 0x7f2d7806c040 2023-10-16T14:35:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.469+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7806b330 con 0x7f2d7806c040 2023-10-16T14:35:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.471+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d6800acd0 con 0x7f2d7806c040 2023-10-16T14:35:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.471+0000 7f2d66ffd640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2d5806af90 0x7f2d5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.472+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d68088d80 con 0x7f2d7806c040 2023-10-16T14:35:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.472+0000 7f2d75d9b640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2d5806af90 0x7f2d5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.473+0000 7f2d75d9b640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2d5806af90 0x7f2d5806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c036000 tx=0x7f2d6c030000 comp rx=0 tx=0).ready entity=mgr.6406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.476+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2d68053d10 con 0x7f2d7806c040 2023-10-16T14:35:37.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:37.718+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2d7804e9b0 con 0x7f2d7806c040 2023-10-16T14:35:38.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.308+0000 7f2d66ffd640 1 -- 172.21.15.29:0/1465097909 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v57) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2d68018070 con 0x7f2d7806c040 2023-10-16T14:35:38.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2d5806af90 msgr2=0x7f2d5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2d5806af90 0x7f2d5806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c036000 tx=0x7f2d6c030000 comp rx=0 tx=0).stop 2023-10-16T14:35:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 msgr2=0x7f2d781e6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 0x7f2d781e6530 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c7c0 tx=0x7f2d6800cc80 comp rx=0 tx=0).stop 2023-10-16T14:35:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 shutdown_connections 2023-10-16T14:35:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f2d5806af90 0x7f2d5806d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d780ccfb0 0x7f2d780b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d780ad970 0x7f2d78156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 --2- 172.21.15.29:0/1465097909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d7806c040 0x7f2d781e6530 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 >> 172.21.15.29:0/1465097909 conn(0x7f2d780ce6a0 msgr2=0x7f2d7815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.312+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 shutdown_connections 2023-10-16T14:35:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.313+0000 7f2d7759e640 1 -- 172.21.15.29:0/1465097909 wait complete. 2023-10-16T14:35:38.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:35:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.545+0000 7f3e8c53c640 1 Processor -- start 2023-10-16T14:35:38.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.546+0000 7f3e8c53c640 1 -- start start 2023-10-16T14:35:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.547+0000 7f3e8c53c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e841618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.547+0000 7f3e8c53c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e840d0890 con 0x7f3e840661c0 2023-10-16T14:35:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.547+0000 7f3e8c53c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e840d05b0 con 0x7f3e8406c7d0 2023-10-16T14:35:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.547+0000 7f3e8c53c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8406b1e0 con 0x7f3e8415d330 2023-10-16T14:35:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.548+0000 7f3e8aad9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3e840661c0 0x7f3e840ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59714/0 (socket says 172.21.15.29:59714) 2023-10-16T14:35:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.548+0000 7f3e8aad9640 1 -- 172.21.15.29:0/4056196686 learned_addr learned my addr 172.21.15.29:0/4056196686 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:38.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.548+0000 7f3e8a2d8640 1 --2- 172.21.15.29:0/4056196686 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e841618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:38.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.548+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208543939 0 0) 0x7f3e840d0890 con 0x7f3e840661c0 2023-10-16T14:35:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.548+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e70003520 con 0x7f3e840661c0 2023-10-16T14:35:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.549+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156857859 0 0) 0x7f3e840d05b0 con 0x7f3e8406c7d0 2023-10-16T14:35:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.549+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e840d0890 con 0x7f3e8406c7d0 2023-10-16T14:35:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.549+0000 7f3e8a2d8640 1 -- 172.21.15.29:0/4056196686 >> v1:172.21.15.170:6789/0 conn(0x7f3e8406c7d0 legacy=0x7f3e84065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.549+0000 7f3e8a2d8640 1 -- 172.21.15.29:0/4056196686 >> v1:172.21.15.29:6789/0 conn(0x7f3e840661c0 legacy=0x7f3e840ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.549+0000 7f3e8a2d8640 1 -- 172.21.15.29:0/4056196686 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84160dd0 con 0x7f3e8415d330 2023-10-16T14:35:38.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.550+0000 7f3e8a2d8640 1 --2- 172.21.15.29:0/4056196686 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e841618b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009dc0 tx=0x7f3e8002be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebeafc9bf1349d39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:38.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.550+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8002cc30 con 0x7f3e8415d330 2023-10-16T14:35:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.551+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8002cd90 con 0x7f3e8415d330 2023-10-16T14:35:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.551+0000 7f3e892d6640 1 -- 172.21.15.29:0/4056196686 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80034640 con 0x7f3e8415d330 2023-10-16T14:35:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.551+0000 7f3e8c53c640 1 -- 172.21.15.29:0/4056196686 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 msgr2=0x7f3e841618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.551+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/4056196686 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e841618b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009dc0 tx=0x7f3e8002be80 comp rx=0 tx=0).stop 2023-10-16T14:35:38.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.552+0000 7f3e8c53c640 1 -- 172.21.15.29:0/4056196686 shutdown_connections 2023-10-16T14:35:38.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.552+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/4056196686 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e841618b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.552+0000 7f3e8c53c640 1 -- 172.21.15.29:0/4056196686 >> 172.21.15.29:0/4056196686 conn(0x7f3e840d0ab0 msgr2=0x7f3e840d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:38.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.552+0000 7f3e8c53c640 1 -- 172.21.15.29:0/4056196686 shutdown_connections 2023-10-16T14:35:38.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.552+0000 7f3e8c53c640 1 -- 172.21.15.29:0/4056196686 wait complete. 2023-10-16T14:35:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.554+0000 7f3e8c53c640 1 Processor -- start 2023-10-16T14:35:38.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.555+0000 7f3e8c53c640 1 -- start start 2023-10-16T14:35:38.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.556+0000 7f3e8c53c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e840661c0 0x7f3e84154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:38.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.556+0000 7f3e8c53c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 0x7f3e840ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:38.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.556+0000 7f3e8c53c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e84159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:38.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.556+0000 7f3e8c53c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840ad580 con 0x7f3e8406c7d0 2023-10-16T14:35:38.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.556+0000 7f3e8c53c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840ad250 con 0x7f3e840661c0 2023-10-16T14:35:38.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.556+0000 7f3e8c53c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84154850 con 0x7f3e8415d330 2023-10-16T14:35:38.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 0x7f3e840ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:38.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 0x7f3e840ada10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37772/0 (socket says 172.21.15.29:37772) 2023-10-16T14:35:38.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 -- 172.21.15.29:0/2566590913 learned_addr learned my addr 172.21.15.29:0/2566590913 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e8a2d8640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e840661c0 0x7f3e84154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e8aad9640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e84159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 -- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 msgr2=0x7f3e84159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e84159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 -- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e840661c0 msgr2=0x7f3e84154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e840661c0 0x7f3e84154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.557+0000 7f3e89ad7640 1 -- 172.21.15.29:0/2566590913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e80009a70 con 0x7f3e8406c7d0 2023-10-16T14:35:38.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.558+0000 7f3e89ad7640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 0x7f3e840ada10 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400c500 tx=0x7f3e7400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:38.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.558+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e740052d0 con 0x7f3e8406c7d0 2023-10-16T14:35:38.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.558+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e74005450 con 0x7f3e8406c7d0 2023-10-16T14:35:38.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.558+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e840ca780 con 0x7f3e8406c7d0 2023-10-16T14:35:38.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.558+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e74012870 con 0x7f3e8406c7d0 2023-10-16T14:35:38.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.558+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e840af6a0 con 0x7f3e8406c7d0 2023-10-16T14:35:38.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.559+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e840ad730 con 0x7f3e8406c7d0 2023-10-16T14:35:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.560+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3e740129d0 con 0x7f3e8406c7d0 2023-10-16T14:35:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.561+0000 7f3e7affd640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 0x7f3e6806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.561+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e74089540 con 0x7f3e8406c7d0 2023-10-16T14:35:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.562+0000 7f3e8a2d8640 1 -- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 msgr2=0x7f3e6806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:38.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.562+0000 7f3e8a2d8640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 0x7f3e6806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:38.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.566+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e840ad730 con 0x7f3e8406c7d0 2023-10-16T14:35:38.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.761+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3e841610b0 con 0x7f3e8406c7d0 2023-10-16T14:35:38.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.763+0000 7f3e8a2d8640 1 -- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 msgr2=0x7f3e6806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:38.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.763+0000 7f3e8a2d8640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 0x7f3e6806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:38.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.766+0000 7f3e7affd640 1 -- 172.21.15.29:0/2566590913 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3e74054450 con 0x7f3e8406c7d0 2023-10-16T14:35:38.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 msgr2=0x7f3e6806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:38.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 0x7f3e6806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 msgr2=0x7f3e840ada10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:38.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 0x7f3e840ada10 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400c500 tx=0x7f3e7400c9c0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 shutdown_connections 2023-10-16T14:35:38.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f3e6806afe0 0x7f3e6806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8406c7d0 0x7f3e840ada10 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8415d330 0x7f3e84159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 --2- 172.21.15.29:0/2566590913 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e840661c0 0x7f3e84154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:38.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.768+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 >> 172.21.15.29:0/2566590913 conn(0x7f3e840d0ab0 msgr2=0x7f3e84157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:38.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.769+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 shutdown_connections 2023-10-16T14:35:38.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:38.769+0000 7f3e8c53c640 1 -- 172.21.15.29:0/2566590913 wait complete. 2023-10-16T14:35:38.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:35:38.840 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:38.836+0000 7f21224b0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:35:38.860 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:38.856+0000 7fea41642040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:35:38.892 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:38.888+0000 7f21224b0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:35:38.917 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:38.914+0000 7fea41642040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:35:39.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.012+0000 7fdd1498f640 1 Processor -- start 2023-10-16T14:35:39.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.013+0000 7fdd1498f640 1 -- start start 2023-10-16T14:35:39.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd1498f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 0x7fdd100cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd1498f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdd100d0be0 0x7fdd100cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd1498f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1015d150 con 0x7fdd1015cd70 2023-10-16T14:35:39.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd1498f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd100d1180 con 0x7fdd100d0be0 2023-10-16T14:35:39.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd1498f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100d1460 con 0x7fdd100b8480 2023-10-16T14:35:39.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd0ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdd1015cd70 0x7fdd100d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59716/0 (socket says 172.21.15.29:59716) 2023-10-16T14:35:39.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd0ed9d640 1 -- 172.21.15.29:0/766191915 learned_addr learned my addr 172.21.15.29:0/766191915 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:39.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458789387 0 0) 0x7fdd1015d150 con 0x7fdd1015cd70 2023-10-16T14:35:39.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcfc003520 con 0x7fdd1015cd70 2023-10-16T14:35:39.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd0f59e640 1 --2- 172.21.15.29:0/766191915 >> v2:172.21.15.170:3300/0 conn(0x7fdd100d0be0 0x7fdd100cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.015+0000 7fdd07fff640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 0x7fdd100cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1263352627 0 0) 0x7fdcfc003520 con 0x7fdd1015cd70 2023-10-16T14:35:39.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 msgr2=0x7fdd100cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 0x7fdd100cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 >> v2:172.21.15.170:3300/0 conn(0x7fdd100d0be0 msgr2=0x7fdd100cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 --2- 172.21.15.29:0/766191915 >> v2:172.21.15.170:3300/0 conn(0x7fdd100d0be0 0x7fdd100cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd100d1740 con 0x7fdd1015cd70 2023-10-16T14:35:39.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fdd00001670 con 0x7fdd1015cd70 2023-10-16T14:35:39.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 >> v1:172.21.15.29:6789/0 conn(0x7fdd1015cd70 legacy=0x7fdd100d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc003970 0x7fdcfc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcfc0047c0 0x7fdcfc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 0x7fdcfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100d1180 con 0x7fdcfc0047c0 2023-10-16T14:35:39.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100d1460 con 0x7fdcfc003970 2023-10-16T14:35:39.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc00ba80 con 0x7fdd100b8480 2023-10-16T14:35:39.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.016+0000 7fdd07fff640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcfc0047c0 0x7fdcfc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.017+0000 7fdd07fff640 1 -- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 msgr2=0x7fdcfc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.017+0000 7fdd0f59e640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 0x7fdcfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.017+0000 7fdd07fff640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100b8480 0x7fdcfc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.017+0000 7fdd07fff640 1 -- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc003970 msgr2=0x7fdcfc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.017+0000 7fdd07fff640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc003970 0x7fdcfc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.017+0000 7fdd07fff640 1 -- 172.21.15.29:0/766191915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc003520 con 0x7fdcfc0047c0 2023-10-16T14:35:39.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.018+0000 7fdd07fff640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcfc0047c0 0x7fdcfc004c40 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800ba00 tx=0x7fdcfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f144f63187ee486d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:39.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.018+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf80325f0 con 0x7fdcfc0047c0 2023-10-16T14:35:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.018+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8032750 con 0x7fdcfc0047c0 2023-10-16T14:35:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.018+0000 7fdd0e59c640 1 -- 172.21.15.29:0/766191915 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8032ae0 con 0x7fdcfc0047c0 2023-10-16T14:35:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd1498f640 1 -- 172.21.15.29:0/766191915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcfc0047c0 msgr2=0x7fdcfc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd1498f640 1 --2- 172.21.15.29:0/766191915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcfc0047c0 0x7fdcfc004c40 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800ba00 tx=0x7fdcfc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd0f59e640 1 -- 172.21.15.29:0/766191915 reap_dead start 2023-10-16T14:35:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd1498f640 1 -- 172.21.15.29:0/766191915 shutdown_connections 2023-10-16T14:35:39.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd1498f640 1 -- 172.21.15.29:0/766191915 >> 172.21.15.29:0/766191915 conn(0x7fdd100cf570 msgr2=0x7fdd100cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:39.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd1498f640 1 -- 172.21.15.29:0/766191915 shutdown_connections 2023-10-16T14:35:39.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.019+0000 7fdd1498f640 1 -- 172.21.15.29:0/766191915 wait complete. 2023-10-16T14:35:39.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.021+0000 7fdd1498f640 1 Processor -- start 2023-10-16T14:35:39.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.021+0000 7fdd1498f640 1 -- start start 2023-10-16T14:35:39.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd1498f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc0047c0 0x7fdd10151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd1498f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 0x7fdd1015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd1498f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100cd430 0x7fdd100cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd1498f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100cddf0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd1498f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1015d340 con 0x7fdcfc0047c0 2023-10-16T14:35:39.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd1498f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10152070 con 0x7fdd100cd430 2023-10-16T14:35:39.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 0x7fdd1015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 0x7fdd1015eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37800/0 (socket says 172.21.15.29:37800) 2023-10-16T14:35:39.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.022+0000 7fdd07fff640 1 -- 172.21.15.29:0/468860121 learned_addr learned my addr 172.21.15.29:0/468860121 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:39.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd0f59e640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100cd430 0x7fdd100cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd0ed9d640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc0047c0 0x7fdd10151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd07fff640 1 -- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100cd430 msgr2=0x7fdd100cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd07fff640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100cd430 0x7fdd100cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd07fff640 1 -- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc0047c0 msgr2=0x7fdd10151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd07fff640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc0047c0 0x7fdd10151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd07fff640 1 -- 172.21.15.29:0/468860121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1015f8f0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd0ed9d640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc0047c0 0x7fdd10151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:35:39.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd07fff640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 0x7fdd1015eff0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800bb20 tx=0x7fdcf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:39.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf803ca80 con 0x7fdd101f5ab0 2023-10-16T14:35:39.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf803cbe0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1015fbd0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.023+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf80463f0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.024+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd10160190 con 0x7fdd101f5ab0 2023-10-16T14:35:39.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.025+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1004e9b0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.026+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdcf804f450 con 0x7fdd101f5ab0 2023-10-16T14:35:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.027+0000 7fdd077fe640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 0x7fdcf006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.027+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcf80abe80 con 0x7fdd101f5ab0 2023-10-16T14:35:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.027+0000 7fdd0ed9d640 1 -- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 msgr2=0x7fdcf006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.027+0000 7fdd0ed9d640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 0x7fdcf006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:39.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.031+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdcf807a7a0 con 0x7fdd101f5ab0 2023-10-16T14:35:39.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.224+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fdd101e8600 con 0x7fdd101f5ab0 2023-10-16T14:35:39.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.228+0000 7fdd0ed9d640 1 -- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 msgr2=0x7fdcf006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:39.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.228+0000 7fdd0ed9d640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 0x7fdcf006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:39.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.230+0000 7fdd077fe640 1 -- 172.21.15.29:0/468860121 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdcf807a330 con 0x7fdd101f5ab0 2023-10-16T14:35:39.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 msgr2=0x7fdcf006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:39.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 0x7fdcf006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 msgr2=0x7fdd1015eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 0x7fdd1015eff0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800bb20 tx=0x7fdcf8004730 comp rx=0 tx=0).stop 2023-10-16T14:35:39.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 shutdown_connections 2023-10-16T14:35:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7fdcf006b010 0x7fdcf006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd101f5ab0 0x7fdd1015eff0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd100cd430 0x7fdd100cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 --2- 172.21.15.29:0/468860121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcfc0047c0 0x7fdd10151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 >> 172.21.15.29:0/468860121 conn(0x7fdd100cf570 msgr2=0x7fdd100b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:39.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 shutdown_connections 2023-10-16T14:35:39.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.233+0000 7fdd1498f640 1 -- 172.21.15.29:0/468860121 wait complete. 2023-10-16T14:35:39.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:39.256 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:35:39.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T14:35:39.341 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.338+0000 7f21224b0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:35:39.393 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.390+0000 7f21224b0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:35:39.401 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.395+0000 7fea41642040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:35:39.443 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.439+0000 7f21224b0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:35:39.455 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.451+0000 7fea41642040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:35:39.503 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.500+0000 7f21224b0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:35:39.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.504+0000 7fea41642040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:35:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.538+0000 7f91930ea640 1 Processor -- start 2023-10-16T14:35:39.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.539+0000 7f91930ea640 1 -- start start 2023-10-16T14:35:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.540+0000 7f91930ea640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 0x7f918c071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.540+0000 7f91930ea640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918c06bb60 con 0x7f918c0ccdc0 2023-10-16T14:35:39.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.540+0000 7f91930ea640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918c0716b0 con 0x7f918c0cdf50 2023-10-16T14:35:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.540+0000 7f91930ea640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0d09e0 con 0x7f918c15cba0 2023-10-16T14:35:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91918e7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f918c0ccdc0 0x7f918c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59728/0 (socket says 172.21.15.29:59728) 2023-10-16T14:35:39.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91918e7640 1 -- 172.21.15.29:0/3530991261 learned_addr learned my addr 172.21.15.29:0/3530991261 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91920e8640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 0x7f918c071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355334228 0 0) 0x7f918c06bb60 con 0x7f918c0ccdc0 2023-10-16T14:35:39.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9178003520 con 0x7f918c0ccdc0 2023-10-16T14:35:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979151212 0 0) 0x7f9178003520 con 0x7f918c0ccdc0 2023-10-16T14:35:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 msgr2=0x7f918c071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 0x7f918c071bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 >> v1:172.21.15.170:6789/0 conn(0x7f918c0cdf50 legacy=0x7f918c0cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c06af50 con 0x7f918c0ccdc0 2023-10-16T14:35:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.541+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f917c001670 con 0x7f918c0ccdc0 2023-10-16T14:35:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 >> v1:172.21.15.29:6789/0 conn(0x7f918c0ccdc0 legacy=0x7f918c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91780039a0 0x7f9178003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780047c0 0x7f9178004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 0x7f9178005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0d09e0 con 0x7f91780039a0 2023-10-16T14:35:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917800ba60 con 0x7f91780047c0 2023-10-16T14:35:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917800bc20 con 0x7f918c15cba0 2023-10-16T14:35:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91920e8640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91780039a0 0x7f9178003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91918e7640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780047c0 0x7f9178004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91928e9640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 0x7f9178005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91920e8640 1 -- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 msgr2=0x7f9178005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91920e8640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c15cba0 0x7f9178005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91920e8640 1 -- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780047c0 msgr2=0x7f9178004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91920e8640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780047c0 0x7f9178004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.542+0000 7f91920e8640 1 -- 172.21.15.29:0/3530991261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9178003520 con 0x7f91780039a0 2023-10-16T14:35:39.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.544+0000 7f91920e8640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91780039a0 0x7f9178003dc0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9188009a90 tx=0x7f91780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bff04777701c598 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:39.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.544+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918802c660 con 0x7f91780039a0 2023-10-16T14:35:39.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.544+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918802c7c0 con 0x7f91780039a0 2023-10-16T14:35:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.544+0000 7f91910e6640 1 -- 172.21.15.29:0/3530991261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918802cb50 con 0x7f91780039a0 2023-10-16T14:35:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91930ea640 1 -- 172.21.15.29:0/3530991261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91780039a0 msgr2=0x7f9178003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91930ea640 1 --2- 172.21.15.29:0/3530991261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91780039a0 0x7f9178003dc0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9188009a90 tx=0x7f91780057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91928e9640 1 -- 172.21.15.29:0/3530991261 reap_dead start 2023-10-16T14:35:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91930ea640 1 -- 172.21.15.29:0/3530991261 shutdown_connections 2023-10-16T14:35:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91930ea640 1 -- 172.21.15.29:0/3530991261 >> 172.21.15.29:0/3530991261 conn(0x7f918c065ed0 msgr2=0x7f918c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91930ea640 1 -- 172.21.15.29:0/3530991261 shutdown_connections 2023-10-16T14:35:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.545+0000 7f91930ea640 1 -- 172.21.15.29:0/3530991261 wait complete. 2023-10-16T14:35:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.547+0000 7f91930ea640 1 Processor -- start 2023-10-16T14:35:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.547+0000 7f91930ea640 1 -- start start 2023-10-16T14:35:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91930ea640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780039a0 0x7f918c0b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91930ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918c159000 0x7f918c1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91930ea640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c0b5c20 0x7f918c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91930ea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0b39c0 con 0x7f918c159000 2023-10-16T14:35:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91930ea640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0b65e0 con 0x7f91780039a0 2023-10-16T14:35:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91930ea640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c1588f0 con 0x7f918c0b5c20 2023-10-16T14:35:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918c159000 0x7f918c1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918c159000 0x7f918c1583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37824/0 (socket says 172.21.15.29:37824) 2023-10-16T14:35:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 -- 172.21.15.29:0/314637304 learned_addr learned my addr 172.21.15.29:0/314637304 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c0b5c20 msgr2=0x7f918c0b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f918c0b5c20 0x7f918c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780039a0 msgr2=0x7f918c0b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91780039a0 0x7f918c0b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.548+0000 7f91918e7640 1 -- 172.21.15.29:0/314637304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0ae230 con 0x7f918c159000 2023-10-16T14:35:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.549+0000 7f91918e7640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918c159000 0x7f918c1583b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e790 tx=0x7f917c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.549+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00eea0 con 0x7f918c159000 2023-10-16T14:35:39.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.549+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c00c870 con 0x7f918c159000 2023-10-16T14:35:39.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.549+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c158d70 con 0x7f918c159000 2023-10-16T14:35:39.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.549+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00cc20 con 0x7f918c159000 2023-10-16T14:35:39.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.549+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c0adbe0 con 0x7f918c159000 2023-10-16T14:35:39.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.550+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c04e9b0 con 0x7f918c159000 2023-10-16T14:35:39.561 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.549+0000 7f21224b0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:35:39.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.551+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f917c01d460 con 0x7f918c159000 2023-10-16T14:35:39.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.552+0000 7f9182ffd640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 0x7f917006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:39.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.552+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f917c0897f0 con 0x7f918c159000 2023-10-16T14:35:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.552+0000 7f91920e8640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 msgr2=0x7f917006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.552+0000 7f91920e8640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 0x7f917006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:35:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.556+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f917c058180 con 0x7f918c159000 2023-10-16T14:35:39.574 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.571+0000 7fea41642040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:35:39.627 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.623+0000 7fea41642040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:35:39.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.660+0000 7f21224b0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:35:39.725 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.721+0000 7f21224b0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:39.744 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.740+0000 7fea41642040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:35:39.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.754+0000 7f91920e8640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 msgr2=0x7f917006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:39.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.754+0000 7f91920e8640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 0x7f917006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:35:39.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:39.769+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 --> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f918c06b510 con 0x7f917006afe0 2023-10-16T14:35:39.773 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.770+0000 7f21224b0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:35:39.810 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.806+0000 7fea41642040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:39.831 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.827+0000 7f21224b0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:35:39.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.858+0000 7fea41642040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:35:39.880 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:39.877+0000 7f21224b0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:35:39.920 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.917+0000 7fea41642040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:35:39.973 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:39.969+0000 7fea41642040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:35:40.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:40.155+0000 7f91920e8640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 msgr2=0x7f917006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:40.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:40.155+0000 7f91920e8640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 0x7f917006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:35:40.261 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.258+0000 7f21224b0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:35:40.373 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:40.369+0000 7fea41642040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:35:40.419 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.415+0000 7f21224b0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:35:40.526 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.522+0000 7f21224b0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:35:40.541 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:40.538+0000 7fea41642040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:35:40.576 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.572+0000 7f21224b0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:35:40.655 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:40.652+0000 7fea41642040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:35:40.682 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.678+0000 7f21224b0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:40.708 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:40.704+0000 7fea41642040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:35:40.787 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.784+0000 7f21224b0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:35:40.821 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:40.817+0000 7fea41642040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:40.932 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:40.928+0000 7fea41642040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:35:40.942 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.938+0000 7f21224b0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:40.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:40.956+0000 7f91920e8640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 msgr2=0x7f917006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1801486769 2023-10-16T14:35:40.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:40.957+0000 7f91920e8640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 0x7f917006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:35:40.990 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:40.986+0000 7f21224b0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:41.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:41.091+0000 7fea41642040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:35:41.107 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:41.103+0000 7f21224b0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:35:41.146 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:41.143+0000 7fea41642040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:35:41.271 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:41.267+0000 7fea41642040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:35:41.931 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:41.927+0000 7f21224b0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:35:42.144 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:42.140+0000 7fea41642040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:35:42.223 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.219+0000 7f21224b0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:35:42.280 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.276+0000 7f21224b0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:35:42.391 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.387+0000 7f21224b0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:35:42.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:42.410+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 58) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f917c054ac0 con 0x7f918c159000 2023-10-16T14:35:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:42.411+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 msgr2=0x7f917006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:35:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:42.411+0000 7f9182ffd640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/1801486769,v1:172.21.15.170:6833/1801486769] conn(0x7f917006afe0 0x7f917006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:42.466 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:42.462+0000 7fea41642040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:35:42.535 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:42.531+0000 7fea41642040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:35:42.591 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f20a273e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.592 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f20a273e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.592 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f20a273e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.592 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f20a273e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.592 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f20a273e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.593 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f209f738640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.594 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f209f738640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.594 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f209f738640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.594 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f209f738640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.594 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:35:42.588+0000 7f209f738640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:35:42.659 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:35:42.656+0000 7fea41642040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:35:43.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.424+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 59) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f917c00cd80 con 0x7f918c159000 2023-10-16T14:35:43.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.425+0000 7f9182ffd640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f9170035880 0x7f9170074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.425+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 --> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9170035c60 con 0x7f9170035880 2023-10-16T14:35:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.427+0000 7f91920e8640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f9170035880 0x7f9170074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.439+0000 7f91920e8640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f9170035880 0x7f9170074ea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9188036000 tx=0x7f918802e000 comp rx=0 tx=0).ready entity=mgr.6143 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:43.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.600+0000 7f9182ffd640 1 -- 172.21.15.29:0/314637304 <== mgr.6143 v2:172.21.15.170:6832/3633682082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9170035c60 con 0x7f9170035880 2023-10-16T14:35:43.603 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-16T14:35:43.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f9170035880 msgr2=0x7f9170074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:43.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f9170035880 0x7f9170074ea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9188036000 tx=0x7f918802e000 comp rx=0 tx=0).stop 2023-10-16T14:35:43.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918c159000 msgr2=0x7f918c1583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:43.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 --2- 172.21.15.29:0/314637304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918c159000 0x7f918c1583b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e790 tx=0x7f917c00ec50 comp rx=0 tx=0).stop 2023-10-16T14:35:43.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91928e9640 1 -- 172.21.15.29:0/314637304 reap_dead start 2023-10-16T14:35:43.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 shutdown_connections 2023-10-16T14:35:43.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 >> 172.21.15.29:0/314637304 conn(0x7f918c065ed0 msgr2=0x7f918c0b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:43.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 shutdown_connections 2023-10-16T14:35:43.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.603+0000 7f91930ea640 1 -- 172.21.15.29:0/314637304 wait complete. 2023-10-16T14:35:43.627 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T14:35:43.628 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-14_36_00 in ~17s... 2023-10-16T14:35:43.628 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T14:35:43.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T14:35:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.846+0000 7f616bfff640 1 Processor -- start 2023-10-16T14:35:43.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.846+0000 7f616bfff640 1 -- start start 2023-10-16T14:35:43.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.847+0000 7f616bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f616c15cd70 0x7f616c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.847+0000 7f616bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c15d150 con 0x7f616c0b8480 2023-10-16T14:35:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f616bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0d1180 con 0x7f616c0cded0 2023-10-16T14:35:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f616bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0d1460 con 0x7f616c15cd70 2023-10-16T14:35:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f616a7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f616c0b8480 0x7f616c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59740/0 (socket says 172.21.15.29:59740) 2023-10-16T14:35:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f616a7fc640 1 -- 172.21.15.29:0/3128424660 learned_addr learned my addr 172.21.15.29:0/3128424660 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:35:43.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f616affd640 1 --2- 172.21.15.29:0/3128424660 >> v2:172.21.15.170:3300/0 conn(0x7f616c15cd70 0x7f616c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246877792 0 0) 0x7f616c15d150 con 0x7f616c0b8480 2023-10-16T14:35:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.848+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158003520 con 0x7f616c0b8480 2023-10-16T14:35:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1275240869 0 0) 0x7f6158003520 con 0x7f616c0b8480 2023-10-16T14:35:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 >> v2:172.21.15.170:3300/0 conn(0x7f616c15cd70 msgr2=0x7f616c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 --2- 172.21.15.29:0/3128424660 >> v2:172.21.15.170:3300/0 conn(0x7f616c15cd70 0x7f616c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 >> v1:172.21.15.170:6789/0 conn(0x7f616c0cded0 legacy=0x7f616c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0d1740 con 0x7f616c0b8480 2023-10-16T14:35:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6154001670 con 0x7f616c0b8480 2023-10-16T14:35:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 >> v1:172.21.15.29:6789/0 conn(0x7f616c0b8480 legacy=0x7f616c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6158003970 0x7f6158003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61580047c0 0x7f6158004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6158005180 0x7f615800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d1460 con 0x7f6158003970 2023-10-16T14:35:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615800c000 con 0x7f61580047c0 2023-10-16T14:35:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615800c1c0 con 0x7f6158005180 2023-10-16T14:35:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f616affd640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6158003970 0x7f6158003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.849+0000 7f616a7fc640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61580047c0 0x7f6158004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.850+0000 7f616affd640 1 -- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6158005180 msgr2=0x7f615800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.850+0000 7f616affd640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6158005180 0x7f615800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.850+0000 7f616affd640 1 -- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61580047c0 msgr2=0x7f6158004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:35:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.850+0000 7f616affd640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61580047c0 0x7f6158004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.850+0000 7f616affd640 1 -- 172.21.15.29:0/3128424660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158003520 con 0x7f6158003970 2023-10-16T14:35:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.850+0000 7f616b7fe640 1 -- 172.21.15.29:0/3128424660 reap_dead start 2023-10-16T14:35:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.851+0000 7f616affd640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6158003970 0x7f6158003d70 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f615c009fc0 tx=0x7f61580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e515f2c6f397bfa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.851+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c02cc30 con 0x7f6158003970 2023-10-16T14:35:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.851+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c00f040 con 0x7f6158003970 2023-10-16T14:35:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.851+0000 7f6169ffb640 1 -- 172.21.15.29:0/3128424660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c034440 con 0x7f6158003970 2023-10-16T14:35:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 -- 172.21.15.29:0/3128424660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6158003970 msgr2=0x7f6158003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6158003970 0x7f6158003d70 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f615c009fc0 tx=0x7f61580057f0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 -- 172.21.15.29:0/3128424660 shutdown_connections 2023-10-16T14:35:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 --2- 172.21.15.29:0/3128424660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6158003970 0x7f6158003d70 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 -- 172.21.15.29:0/3128424660 >> 172.21.15.29:0/3128424660 conn(0x7f616c0cf570 msgr2=0x7f616c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 -- 172.21.15.29:0/3128424660 shutdown_connections 2023-10-16T14:35:43.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.852+0000 7f616bfff640 1 -- 172.21.15.29:0/3128424660 wait complete. 2023-10-16T14:35:43.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.854+0000 7f616bfff640 1 Processor -- start 2023-10-16T14:35:43.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.854+0000 7f616bfff640 1 -- start start 2023-10-16T14:35:43.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6158003970 0x7f616c0aa130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 0x7f616c0cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f616c0b45c0 0x7f616c14ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0caab0 con 0x7f616c0cc7a0 2023-10-16T14:35:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0cad90 con 0x7f6158003970 2023-10-16T14:35:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0cb070 con 0x7f616c0b45c0 2023-10-16T14:35:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 0x7f616c0cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 0x7f616c0cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37848/0 (socket says 172.21.15.29:37848) 2023-10-16T14:35:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616a7fc640 1 -- 172.21.15.29:0/1440911360 learned_addr learned my addr 172.21.15.29:0/1440911360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:35:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616affd640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6158003970 0x7f616c0aa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616a7fc640 1 -- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f616c0b45c0 msgr2=0x7f616c14ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.855+0000 7f616a7fc640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f616c0b45c0 0x7f616c14ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f616a7fc640 1 -- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6158003970 msgr2=0x7f616c0aa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f616a7fc640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6158003970 0x7f616c0aa130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f616a7fc640 1 -- 172.21.15.29:0/1440911360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0cb350 con 0x7f616c0cc7a0 2023-10-16T14:35:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f616a7fc640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 0x7f616c0cbb50 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f615400e8c0 tx=0x7f615400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400c870 con 0x7f616c0cc7a0 2023-10-16T14:35:43.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615400c9d0 con 0x7f616c0cc7a0 2023-10-16T14:35:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400cd40 con 0x7f616c0cc7a0 2023-10-16T14:35:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c0cb630 con 0x7f616c0cc7a0 2023-10-16T14:35:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.856+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0cc090 con 0x7f616c0cc7a0 2023-10-16T14:35:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.858+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c04e9b0 con 0x7f616c0cc7a0 2023-10-16T14:35:43.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.858+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 59) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f615401d460 con 0x7f616c0cc7a0 2023-10-16T14:35:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.859+0000 7f61637fe640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f614c06ad40 0x7f614c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:35:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.860+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6154089580 con 0x7f616c0cc7a0 2023-10-16T14:35:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.860+0000 7f616affd640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f614c06ad40 0x7f614c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:35:43.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.861+0000 7f616affd640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f614c06ad40 0x7f614c06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f615c036000 tx=0x7f615c02e000 comp rx=0 tx=0).ready entity=mgr.6143 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:35:43.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:43.864+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6154054740 con 0x7f616c0cc7a0 2023-10-16T14:35:44.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.079+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 --> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f616c0d1d00 con 0x7f614c06ad40 2023-10-16T14:35:44.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.094+0000 7f61637fe640 1 -- 172.21.15.29:0/1440911360 <== mgr.6143 v2:172.21.15.170:6832/3633682082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f615c02cd80 con 0x7f614c06ad40 2023-10-16T14:35:44.097 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:35:44.098 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T14:35:44.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f614c06ad40 msgr2=0x7f614c06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:44.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f614c06ad40 0x7f614c06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f615c036000 tx=0x7f615c02e000 comp rx=0 tx=0).stop 2023-10-16T14:35:44.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 msgr2=0x7f616c0cbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:35:44.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 0x7f616c0cbb50 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f615400e8c0 tx=0x7f615400ed80 comp rx=0 tx=0).stop 2023-10-16T14:35:44.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 shutdown_connections 2023-10-16T14:35:44.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f614c06ad40 0x7f614c06d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:44.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f616c0cc7a0 0x7f616c0cbb50 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:44.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f616c0b45c0 0x7f616c14ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:44.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 --2- 172.21.15.29:0/1440911360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6158003970 0x7f616c0aa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:35:44.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.097+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 >> 172.21.15.29:0/1440911360 conn(0x7f616c0cf570 msgr2=0x7f616c15bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:35:44.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.098+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 shutdown_connections 2023-10-16T14:35:44.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:35:44.098+0000 7f616bfff640 1 -- 172.21.15.29:0/1440911360 wait complete. 2023-10-16T14:35:44.119 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T14:35:44.120 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-10-16T14:35:44.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:44.120 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:44.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:44.181 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:44.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:45.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:45.182 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:45.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:45.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:45.254 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:46.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:46.255 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:46.322 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:46.323 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:46.323 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:47.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:47.325 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:47.392 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:47.392 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:47.392 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:48.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:48.394 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:48.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:35:48.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:35:48.463 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:48.463 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:48.464 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:49.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:49.465 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:49.532 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:49.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:49.533 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:50.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:50.535 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:50.601 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:50.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:50.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:51.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:51.604 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:51.671 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:51.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:51.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:52.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:52.674 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:52.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:52.741 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:52.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:53.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:53.746 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:53.813 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:53.814 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:53.814 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:54.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:54.816 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:54.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:54.883 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:54.883 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:55.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:55.886 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:55.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:55.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:55.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:56.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:56.955 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:57.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:57.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:57.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:58.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:58.026 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:58.092 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:58.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:58.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:35:59.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:35:59.095 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:35:59.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:35:59.163 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:35:59.163 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:36:00.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:00.164 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:36:00.232 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:36:00.233 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:36:00.233 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:36:00.628 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:00.624+0000 7f2091edd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:00.628 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:00.624+0000 7f2091edd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:00.628 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:00.624+0000 7f2091edd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:00.629 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:00.624+0000 7f2091edd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:00.629 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:00.624+0000 7f2091edd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:01.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:01.235 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:36:01.301 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_36_00_UTC 2023-10-16T14:36:01.303 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_36_00_UTC'} 2023-10-16T14:36:01.303 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-14_36_00_UTC'} 2023-10-16T14:36:01.303 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:36:01.304 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-14_36_00_UTC'] 2023-10-16T14:36:01.304 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697466943.6281424, now = 1697466961.3046238, timo = 17 2023-10-16T14:36:02.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T14:36:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.541+0000 7f99a0975640 1 Processor -- start 2023-10-16T14:36:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.542+0000 7f99a0975640 1 -- start start 2023-10-16T14:36:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f99a0975640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f999c0ad970 0x7f999c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f99a0975640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f999c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f99a0975640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999c15b9d0 con 0x7f999c06c040 2023-10-16T14:36:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f99a0975640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999c15bcb0 con 0x7f999c0ad970 2023-10-16T14:36:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f99a0975640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c15bf90 con 0x7f999c0ccfb0 2023-10-16T14:36:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f999b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f999c06c040 0x7f999c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49968/0 (socket says 172.21.15.29:49968) 2023-10-16T14:36:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.543+0000 7f999b7fe640 1 -- 172.21.15.29:0/1516436495 learned_addr learned my addr 172.21.15.29:0/1516436495 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999bfff640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f999c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999affd640 1 --2- 172.21.15.29:0/1516436495 >> v2:172.21.15.170:3300/0 conn(0x7f999c0ad970 0x7f999c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831976404 0 0) 0x7f999c15b9d0 con 0x7f999c06c040 2023-10-16T14:36:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9984003520 con 0x7f999c06c040 2023-10-16T14:36:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3083211848 0 0) 0x7f9984003520 con 0x7f999c06c040 2023-10-16T14:36:02.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 msgr2=0x7f999c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f999c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 >> v2:172.21.15.170:3300/0 conn(0x7f999c0ad970 msgr2=0x7f999c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:02.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 --2- 172.21.15.29:0/1516436495 >> v2:172.21.15.170:3300/0 conn(0x7f999c0ad970 0x7f999c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c0cc270 con 0x7f999c06c040 2023-10-16T14:36:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f998c0019f0 con 0x7f999c06c040 2023-10-16T14:36:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 >> v1:172.21.15.29:6789/0 conn(0x7f999c06c040 legacy=0x7f999c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f9984003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9984003ef0 0x7f99840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984004d00 0x7f9984005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c15bcb0 con 0x7f9984003ef0 2023-10-16T14:36:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c15bf90 con 0x7f9984004d00 2023-10-16T14:36:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.544+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998400b980 con 0x7f999c0ccfb0 2023-10-16T14:36:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999affd640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9984003ef0 0x7f99840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999b7fe640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f9984003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999affd640 1 -- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 msgr2=0x7f9984003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999bfff640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984004d00 0x7f9984005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999affd640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f9984003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999affd640 1 -- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984004d00 msgr2=0x7f9984005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999affd640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984004d00 0x7f9984005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999affd640 1 -- 172.21.15.29:0/1516436495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984003520 con 0x7f9984003ef0 2023-10-16T14:36:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.545+0000 7f999b7fe640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0ccfb0 0x7f9984003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.546+0000 7f999affd640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9984003ef0 0x7f99840047c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9990009a90 tx=0x7f99840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f28cd80e2f5d27f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999002c6b0 con 0x7f9984003ef0 2023-10-16T14:36:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999002c810 con 0x7f9984003ef0 2023-10-16T14:36:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f999a7fc640 1 -- 172.21.15.29:0/1516436495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999002cbc0 con 0x7f9984003ef0 2023-10-16T14:36:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f99a0975640 1 -- 172.21.15.29:0/1516436495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9984003ef0 msgr2=0x7f99840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f99a0975640 1 --2- 172.21.15.29:0/1516436495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9984003ef0 0x7f99840047c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9990009a90 tx=0x7f99840057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f999bfff640 1 -- 172.21.15.29:0/1516436495 reap_dead start 2023-10-16T14:36:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f99a0975640 1 -- 172.21.15.29:0/1516436495 shutdown_connections 2023-10-16T14:36:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.547+0000 7f99a0975640 1 -- 172.21.15.29:0/1516436495 >> 172.21.15.29:0/1516436495 conn(0x7f999c0ce6a0 msgr2=0x7f999c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.548+0000 7f99a0975640 1 -- 172.21.15.29:0/1516436495 shutdown_connections 2023-10-16T14:36:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.548+0000 7f99a0975640 1 -- 172.21.15.29:0/1516436495 wait complete. 2023-10-16T14:36:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.549+0000 7f99a0975640 1 Processor -- start 2023-10-16T14:36:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.550+0000 7f99a0975640 1 -- start start 2023-10-16T14:36:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f99a0975640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984003ef0 0x7f999c1555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f99a0975640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 0x7f999c0c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f99a0975640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0b0010 0x7f999c0b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f99a0975640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b09d0 con 0x7f999c0af210 2023-10-16T14:36:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f99a0975640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0c3970 con 0x7f9984003ef0 2023-10-16T14:36:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f99a0975640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0c3e00 con 0x7f999c0b0010 2023-10-16T14:36:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f999affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 0x7f999c0c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f999affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 0x7f999c0c4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58310/0 (socket says 172.21.15.29:58310) 2023-10-16T14:36:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f999affd640 1 -- 172.21.15.29:0/1489068261 learned_addr learned my addr 172.21.15.29:0/1489068261 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f999b7fe640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984003ef0 0x7f999c1555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f999bfff640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0b0010 0x7f999c0b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.551+0000 7f999affd640 1 -- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0b0010 msgr2=0x7f999c0b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f999affd640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0b0010 0x7f999c0b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f999affd640 1 -- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984003ef0 msgr2=0x7f999c1555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f999affd640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984003ef0 0x7f999c1555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f999affd640 1 -- 172.21.15.29:0/1489068261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c0c2d60 con 0x7f999c0af210 2023-10-16T14:36:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f999b7fe640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984003ef0 0x7f999c1555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f999affd640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 0x7f999c0c4960 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f99900099e0 tx=0x7f9990004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990034a80 con 0x7f999c0af210 2023-10-16T14:36:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9990034c00 con 0x7f999c0af210 2023-10-16T14:36:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999003e440 con 0x7f999c0af210 2023-10-16T14:36:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c0c3040 con 0x7f999c0af210 2023-10-16T14:36:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.552+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c0c3600 con 0x7f999c0af210 2023-10-16T14:36:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.554+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c0c37c0 con 0x7f999c0af210 2023-10-16T14:36:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.555+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 61) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f999003e5a0 con 0x7f999c0af210 2023-10-16T14:36:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.555+0000 7f9983fff640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f998806b070 0x7f998806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.556+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f99900ab990 con 0x7f999c0af210 2023-10-16T14:36:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.556+0000 7f999b7fe640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f998806b070 0x7f998806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:02.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.557+0000 7f999b7fe640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f998806b070 0x7f998806d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f998c007c40 tx=0x7f998c009040 comp rx=0 tx=0).ready entity=mgr.6143 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:02.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.560+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9990076650 con 0x7f999c0af210 2023-10-16T14:36:02.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.773+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 --> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f999c0cc550 con 0x7f998806b070 2023-10-16T14:36:02.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.949+0000 7f9983fff640 1 -- 172.21.15.29:0/1489068261 <== mgr.6143 v2:172.21.15.170:6832/3633682082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f998c00d040 con 0x7f998806b070 2023-10-16T14:36:02.953 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-16T14:36:02.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.952+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f998806b070 msgr2=0x7f998806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f998806b070 0x7f998806d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f998c007c40 tx=0x7f998c009040 comp rx=0 tx=0).stop 2023-10-16T14:36:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 msgr2=0x7f999c0c4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 0x7f999c0c4960 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f99900099e0 tx=0x7f9990004730 comp rx=0 tx=0).stop 2023-10-16T14:36:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 shutdown_connections 2023-10-16T14:36:02.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f998806b070 0x7f998806d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f999c0af210 0x7f999c0c4960 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f999c0b0010 0x7f999c0b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 --2- 172.21.15.29:0/1489068261 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9984003ef0 0x7f999c1555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 >> 172.21.15.29:0/1489068261 conn(0x7f999c0ce6a0 msgr2=0x7f999c0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 shutdown_connections 2023-10-16T14:36:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:02.953+0000 7f99a0975640 1 -- 172.21.15.29:0/1489068261 wait complete. 2023-10-16T14:36:02.979 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T14:36:02.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:02.979 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:36:03.038 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_36_00_UTC 2023-10-16T14:36:03.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_36_00_UTC 2023-10-16T14:36:03.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:03.041 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_36_00_UTC) 2023-10-16T14:36:03.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:03.112 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T14:36:03.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:36:03.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.410+0000 7f7a2f59e640 1 Processor -- start 2023-10-16T14:36:03.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.410+0000 7f7a2f59e640 1 -- start start 2023-10-16T14:36:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a2f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7a3015b320 0x7f7a300cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a2f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3015b9d0 con 0x7f7a300ad970 2023-10-16T14:36:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a2f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3015bcb0 con 0x7f7a3006c040 2023-10-16T14:36:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a2f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3015b7d0 con 0x7f7a3015b320 2023-10-16T14:36:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a27fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7a300ad970 0x7f7a300cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49974/0 (socket says 172.21.15.29:49974) 2023-10-16T14:36:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a27fff640 1 -- 172.21.15.29:0/2768503221 learned_addr learned my addr 172.21.15.29:0/2768503221 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.412+0000 7f7a2ed9d640 1 --2- 172.21.15.29:0/2768503221 >> v2:172.21.15.170:3300/0 conn(0x7f7a3015b320 0x7f7a300cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185851096 0 0) 0x7f7a3015b9d0 con 0x7f7a300ad970 2023-10-16T14:36:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1c003520 con 0x7f7a300ad970 2023-10-16T14:36:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 203869426 0 0) 0x7f7a1c003520 con 0x7f7a300ad970 2023-10-16T14:36:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 >> v2:172.21.15.170:3300/0 conn(0x7f7a3015b320 msgr2=0x7f7a300cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 --2- 172.21.15.29:0/2768503221 >> v2:172.21.15.170:3300/0 conn(0x7f7a3015b320 0x7f7a300cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 >> v1:172.21.15.170:6789/0 conn(0x7f7a3006c040 legacy=0x7f7a3006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3015bf90 con 0x7f7a300ad970 2023-10-16T14:36:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7a18001670 con 0x7f7a300ad970 2023-10-16T14:36:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 >> v1:172.21.15.29:6789/0 conn(0x7f7a300ad970 legacy=0x7f7a300cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:03.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a1c0039a0 0x7f7a1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a1c005180 0x7f7a1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3015b7d0 con 0x7f7a1c0039a0 2023-10-16T14:36:03.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c00c050 con 0x7f7a1c005180 2023-10-16T14:36:03.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.413+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c00c210 con 0x7f7a1c0047c0 2023-10-16T14:36:03.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a1c0039a0 0x7f7a1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2ed9d640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a1c005180 0x7f7a1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a27fff640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2e59c640 1 -- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0047c0 msgr2=0x7f7a1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2e59c640 1 -- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a1c005180 msgr2=0x7f7a1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a1c005180 0x7f7a1c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2e59c640 1 -- 172.21.15.29:0/2768503221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c003520 con 0x7f7a1c0039a0 2023-10-16T14:36:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.414+0000 7f7a2ed9d640 1 -- 172.21.15.29:0/2768503221 reap_dead start 2023-10-16T14:36:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.415+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a1c0039a0 0x7f7a1c003dc0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f7a20001750 tx=0x7f7a1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95136599049b0daf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.415+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200052d0 con 0x7f7a1c0039a0 2023-10-16T14:36:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.415+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2002e050 con 0x7f7a1c0039a0 2023-10-16T14:36:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.415+0000 7f7a2dd9b640 1 -- 172.21.15.29:0/2768503221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20003830 con 0x7f7a1c0039a0 2023-10-16T14:36:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2768503221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a1c0039a0 msgr2=0x7f7a1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a1c0039a0 0x7f7a1c003dc0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f7a20001750 tx=0x7f7a1c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2768503221 shutdown_connections 2023-10-16T14:36:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2768503221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a1c0039a0 0x7f7a1c003dc0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2768503221 >> 172.21.15.29:0/2768503221 conn(0x7f7a300ce6a0 msgr2=0x7f7a300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2768503221 shutdown_connections 2023-10-16T14:36:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.416+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2768503221 wait complete. 2023-10-16T14:36:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.418+0000 7f7a2f59e640 1 Processor -- start 2023-10-16T14:36:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.418+0000 7f7a2f59e640 1 -- start start 2023-10-16T14:36:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0039a0 0x7f7a3014e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a3014ecd0 0x7f7a300af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 0x7f7a30069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3014f150 con 0x7f7a300afd60 2023-10-16T14:36:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30150060 con 0x7f7a3014ecd0 2023-10-16T14:36:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301f5bc0 con 0x7f7a1c0039a0 2023-10-16T14:36:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 0x7f7a30069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 0x7f7a30069e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58332/0 (socket says 172.21.15.29:58332) 2023-10-16T14:36:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.419+0000 7f7a2ed9d640 1 -- 172.21.15.29:0/2126081268 learned_addr learned my addr 172.21.15.29:0/2126081268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a27fff640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a3014ecd0 0x7f7a300af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0039a0 0x7f7a3014e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2ed9d640 1 -- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0039a0 msgr2=0x7f7a3014e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2ed9d640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0039a0 0x7f7a3014e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2ed9d640 1 -- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a3014ecd0 msgr2=0x7f7a300af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2ed9d640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a3014ecd0 0x7f7a300af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2ed9d640 1 -- 172.21.15.29:0/2126081268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a301f5ea0 con 0x7f7a300afd60 2023-10-16T14:36:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a27fff640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a3014ecd0 0x7f7a300af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0039a0 0x7f7a3014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.420+0000 7f7a2ed9d640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 0x7f7a30069e30 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7a3015b700 tx=0x7f7a280115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.421+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2800e070 con 0x7f7a300afd60 2023-10-16T14:36:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.421+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a28011db0 con 0x7f7a300afd60 2023-10-16T14:36:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.421+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28003990 con 0x7f7a300afd60 2023-10-16T14:36:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.421+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a301f6180 con 0x7f7a300afd60 2023-10-16T14:36:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.421+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a301f6740 con 0x7f7a300afd60 2023-10-16T14:36:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.422+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3006b330 con 0x7f7a300afd60 2023-10-16T14:36:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.423+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 62) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7a28003af0 con 0x7f7a300afd60 2023-10-16T14:36:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.424+0000 7f7a26ffd640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f7a1006b130 0x7f7a1006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.424+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7a2808a450 con 0x7f7a300afd60 2023-10-16T14:36:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.425+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f7a1006b130 0x7f7a1006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.426+0000 7f7a2e59c640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f7a1006b130 0x7f7a1006d5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7a20033000 tx=0x7f7a20009030 comp rx=0 tx=0).ready entity=mgr.6143 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.428+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a28054f30 con 0x7f7a300afd60 2023-10-16T14:36:03.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:03.673+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f7a3015c1b0 con 0x7f7a300afd60 2023-10-16T14:36:04.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.318+0000 7f7a26ffd640 1 -- 172.21.15.29:0/2126081268 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v63) v1 ==== 93+0+0 (secure 0 0 0) 0x7f7a3015c1b0 con 0x7f7a300afd60 2023-10-16T14:36:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.324+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f7a1006b130 msgr2=0x7f7a1006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.324+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f7a1006b130 0x7f7a1006d5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7a20033000 tx=0x7f7a20009030 comp rx=0 tx=0).stop 2023-10-16T14:36:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.324+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 msgr2=0x7f7a30069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.324+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 0x7f7a30069e30 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7a3015b700 tx=0x7f7a280115a0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 shutdown_connections 2023-10-16T14:36:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f7a1006b130 0x7f7a1006d5f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a300afd60 0x7f7a30069e30 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a3014ecd0 0x7f7a300af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 --2- 172.21.15.29:0/2126081268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a1c0039a0 0x7f7a3014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 >> 172.21.15.29:0/2126081268 conn(0x7f7a300ce6a0 msgr2=0x7f7a300cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.325+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 shutdown_connections 2023-10-16T14:36:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.326+0000 7f7a2f59e640 1 -- 172.21.15.29:0/2126081268 wait complete. 2023-10-16T14:36:04.352 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:36:04.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:36:04.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:36:04.563 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:36:04.563 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:04.563 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:04.617 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:04.618 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:04.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:04.847 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:04.844+0000 7f07b0024040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:36:04.864 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:04.861+0000 7fd255614040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:36:04.901 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:04.897+0000 7f07b0024040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:36:04.920 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:04.918+0000 7fd255614040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:36:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.924+0000 7fdfa467c640 1 Processor -- start 2023-10-16T14:36:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.924+0000 7fdfa467c640 1 -- start start 2023-10-16T14:36:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa467c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf9c06c3f0 0x7fdf9c06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa467c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdf9c15e0c0 0x7fdf9c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa467c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0cd630 con 0x7fdf9c15e540 2023-10-16T14:36:04.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa467c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9c0ce1b0 con 0x7fdf9c06c3f0 2023-10-16T14:36:04.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa467c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9c06cd10 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa2418640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf9c06c3f0 0x7fdf9c06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:04.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa2418640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf9c06c3f0 0x7fdf9c06c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50934/0 (socket says 172.21.15.29:50934) 2023-10-16T14:36:04.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.926+0000 7fdfa2418640 1 -- 172.21.15.29:0/1339060991 learned_addr learned my addr 172.21.15.29:0/1339060991 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:04.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.927+0000 7fdfa2418640 1 -- 172.21.15.29:0/1339060991 >> v2:172.21.15.170:3300/0 conn(0x7fdf9c15e0c0 msgr2=0x7fdf9c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:04.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.927+0000 7fdfa2418640 1 --2- 172.21.15.29:0/1339060991 >> v2:172.21.15.170:3300/0 conn(0x7fdf9c15e0c0 0x7fdf9c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.927+0000 7fdfa2418640 1 -- 172.21.15.29:0/1339060991 >> v1:172.21.15.170:6789/0 conn(0x7fdf9c15e540 legacy=0x7fdf9c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:04.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.927+0000 7fdfa2418640 1 -- 172.21.15.29:0/1339060991 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c0d2170 con 0x7fdf9c06c3f0 2023-10-16T14:36:04.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.928+0000 7fdfa2418640 1 --2- 172.21.15.29:0/1339060991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c06c7d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0d3a50 tx=0x7fdf9802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc0d086d874d207 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:04.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.928+0000 7fdfa1416640 1 -- 172.21.15.29:0/1339060991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98031890 con 0x7fdf9c06c3f0 2023-10-16T14:36:04.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.928+0000 7fdfa1416640 1 -- 172.21.15.29:0/1339060991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf980319f0 con 0x7fdf9c06c3f0 2023-10-16T14:36:04.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.928+0000 7fdfa1416640 1 -- 172.21.15.29:0/1339060991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98031d80 con 0x7fdf9c06c3f0 2023-10-16T14:36:04.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 -- 172.21.15.29:0/1339060991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c06c3f0 msgr2=0x7fdf9c06c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 --2- 172.21.15.29:0/1339060991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c06c7d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0d3a50 tx=0x7fdf9802b930 comp rx=0 tx=0).stop 2023-10-16T14:36:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 -- 172.21.15.29:0/1339060991 shutdown_connections 2023-10-16T14:36:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 --2- 172.21.15.29:0/1339060991 >> v2:172.21.15.170:3300/0 conn(0x7fdf9c15e0c0 0x7fdf9c0b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 --2- 172.21.15.29:0/1339060991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c06c7d0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 -- 172.21.15.29:0/1339060991 >> 172.21.15.29:0/1339060991 conn(0x7fdf9c0d2cb0 msgr2=0x7fdf9c0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:04.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 -- 172.21.15.29:0/1339060991 shutdown_connections 2023-10-16T14:36:04.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.929+0000 7fdfa467c640 1 -- 172.21.15.29:0/1339060991 wait complete. 2023-10-16T14:36:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.931+0000 7fdfa467c640 1 Processor -- start 2023-10-16T14:36:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.931+0000 7fdfa467c640 1 -- start start 2023-10-16T14:36:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa467c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c0ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa467c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 0x7fdf9c15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa467c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf9c15e540 0x7fdf9c154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa467c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cac40 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa467c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c159f90 con 0x7fdf9c06c3f0 2023-10-16T14:36:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa467c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c159410 con 0x7fdf9c15e540 2023-10-16T14:36:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa1c17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 0x7fdf9c15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa1c17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 0x7fdf9c15eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50944/0 (socket says 172.21.15.29:50944) 2023-10-16T14:36:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa1c17640 1 -- 172.21.15.29:0/3742865560 learned_addr learned my addr 172.21.15.29:0/3742865560 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.932+0000 7fdfa2418640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c0ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa1c17640 1 -- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf9c15e540 msgr2=0x7fdf9c154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa1c17640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf9c15e540 0x7fdf9c154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa1c17640 1 -- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf9c06c3f0 msgr2=0x7fdf9c0ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:04.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa1c17640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c0ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:04.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa1c17640 1 -- 172.21.15.29:0/3742865560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980094d0 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa1c17640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 0x7fdf9c15eef0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00b4d0 tx=0x7fdf8c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:04.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c010870 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c0109d0 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c1596f0 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c010d80 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.933+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c159cb0 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.934+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c0a9d50 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.936+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fdf8c00f3c0 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.937+0000 7fdf92ffd640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 0x7fdf8006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.937+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fdf8c089b20 con 0x7fdf9c15e0c0 2023-10-16T14:36:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.937+0000 7fdfa2418640 1 -- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 msgr2=0x7fdf8006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.937+0000 7fdfa2418640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 0x7fdf8006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:04.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:04.945+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf9c0a9d50 con 0x7fdf9c15e0c0 2023-10-16T14:36:05.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.137+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf9c0a9d50 con 0x7fdf9c15e0c0 2023-10-16T14:36:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.138+0000 7fdfa2418640 1 -- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 msgr2=0x7fdf8006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.138+0000 7fdfa2418640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 0x7fdf8006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.139+0000 7fdf92ffd640 1 -- 172.21.15.29:0/3742865560 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17028 (secure 0 0 0) 0x7fdf9c0a9d50 con 0x7fdf9c15e0c0 2023-10-16T14:36:05.161 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:05.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":128,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:42.393631+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3330302692":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3494288089":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6833/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/2874036200":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3741475474":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1228162669":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1339480091":"2023-10-17T14:35:42.393479+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 msgr2=0x7fdf8006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 0x7fdf8006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 msgr2=0x7fdf9c15eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 0x7fdf9c15eef0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00b4d0 tx=0x7fdf8c00b990 comp rx=0 tx=0).stop 2023-10-16T14:36:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 shutdown_connections 2023-10-16T14:36:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdf8006b0b0 0x7fdf8006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c15e0c0 0x7fdf9c15eef0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf9c15e540 0x7fdf9c154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 --2- 172.21.15.29:0/3742865560 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf9c06c3f0 0x7fdf9c0ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 >> 172.21.15.29:0/3742865560 conn(0x7fdf9c0d2cb0 msgr2=0x7fdf9c0d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 shutdown_connections 2023-10-16T14:36:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.142+0000 7fdfa467c640 1 -- 172.21.15.29:0/3742865560 wait complete. 2023-10-16T14:36:05.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:35:25.164943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:35:26.261110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:36:05.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:36:05.346 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.343+0000 7f07b0024040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:36:05.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.376+0000 7fe24f2c9640 1 Processor -- start 2023-10-16T14:36:05.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.376+0000 7fe24f2c9640 1 -- start start 2023-10-16T14:36:05.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24f2c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2481618b0 0x7fe248065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24f2c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe248065ff0 0x7fe2480ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24f2c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe248066420 con 0x7fe24815d330 2023-10-16T14:36:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24f2c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24806b1e0 con 0x7fe2481618b0 2023-10-16T14:36:05.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24f2c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24806cee0 con 0x7fe248065ff0 2023-10-16T14:36:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24d065640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe24815d330 0x7fe24806c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54160/0 (socket says 172.21.15.29:54160) 2023-10-16T14:36:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24d065640 1 -- 172.21.15.29:0/977616334 learned_addr learned my addr 172.21.15.29:0/977616334 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:05.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24c864640 1 --2- 172.21.15.29:0/977616334 >> v2:172.21.15.29:3300/0 conn(0x7fe2481618b0 0x7fe248065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe24d866640 1 --2- 172.21.15.29:0/977616334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe248065ff0 0x7fe2480ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.378+0000 7fe23ffff640 1 -- 172.21.15.29:0/977616334 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490689190 0 0) 0x7fe248066420 con 0x7fe24815d330 2023-10-16T14:36:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.379+0000 7fe23ffff640 1 -- 172.21.15.29:0/977616334 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe234003520 con 0x7fe24815d330 2023-10-16T14:36:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.379+0000 7fe24c864640 1 -- 172.21.15.29:0/977616334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe248065ff0 msgr2=0x7fe2480ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.379+0000 7fe24c864640 1 --2- 172.21.15.29:0/977616334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe248065ff0 0x7fe2480ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.379+0000 7fe24c864640 1 -- 172.21.15.29:0/977616334 >> v1:172.21.15.29:6789/0 conn(0x7fe24815d330 legacy=0x7fe24806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.379+0000 7fe24c864640 1 -- 172.21.15.29:0/977616334 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480d0890 con 0x7fe2481618b0 2023-10-16T14:36:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.380+0000 7fe24c864640 1 --2- 172.21.15.29:0/977616334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2481618b0 0x7fe248065ab0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097a0 tx=0x7fe23802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caea623398041e04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.380+0000 7fe23ffff640 1 -- 172.21.15.29:0/977616334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238031780 con 0x7fe2481618b0 2023-10-16T14:36:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.380+0000 7fe23ffff640 1 -- 172.21.15.29:0/977616334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2380318e0 con 0x7fe2481618b0 2023-10-16T14:36:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.380+0000 7fe23ffff640 1 -- 172.21.15.29:0/977616334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238031c50 con 0x7fe2481618b0 2023-10-16T14:36:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 -- 172.21.15.29:0/977616334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2481618b0 msgr2=0x7fe248065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/977616334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2481618b0 0x7fe248065ab0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097a0 tx=0x7fe23802b820 comp rx=0 tx=0).stop 2023-10-16T14:36:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 -- 172.21.15.29:0/977616334 shutdown_connections 2023-10-16T14:36:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/977616334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe248065ff0 0x7fe2480ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/977616334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2481618b0 0x7fe248065ab0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 -- 172.21.15.29:0/977616334 >> 172.21.15.29:0/977616334 conn(0x7fe2480d0ab0 msgr2=0x7fe2480d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 -- 172.21.15.29:0/977616334 shutdown_connections 2023-10-16T14:36:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.381+0000 7fe24f2c9640 1 -- 172.21.15.29:0/977616334 wait complete. 2023-10-16T14:36:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.383+0000 7fe24f2c9640 1 Processor -- start 2023-10-16T14:36:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.383+0000 7fe24f2c9640 1 -- start start 2023-10-16T14:36:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24f2c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 0x7fe2481f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24f2c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe24815d330 0x7fe2480aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24f2c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2481618b0 0x7fe248158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24f2c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480b21c0 con 0x7fe248065ff0 2023-10-16T14:36:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24f2c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe248156850 con 0x7fe24815d330 2023-10-16T14:36:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24f2c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480b1540 con 0x7fe2481618b0 2023-10-16T14:36:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24d065640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 0x7fe2481f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24d065640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 0x7fe2481f68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50962/0 (socket says 172.21.15.29:50962) 2023-10-16T14:36:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.384+0000 7fe24d065640 1 -- 172.21.15.29:0/3973609975 learned_addr learned my addr 172.21.15.29:0/3973609975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d866640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2481618b0 0x7fe248158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24c864640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe24815d330 0x7fe2480aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d065640 1 -- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2481618b0 msgr2=0x7fe248158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d065640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2481618b0 0x7fe248158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d065640 1 -- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe24815d330 msgr2=0x7fe2480aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d065640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe24815d330 0x7fe2480aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d065640 1 -- 172.21.15.29:0/3973609975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238009400 con 0x7fe248065ff0 2023-10-16T14:36:05.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24c864640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe24815d330 0x7fe2480aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:05.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24d065640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 0x7fe2481f68f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe2440010e0 tx=0x7fe24400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:05.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244005ce0 con 0x7fe248065ff0 2023-10-16T14:36:05.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe244005e60 con 0x7fe248065ff0 2023-10-16T14:36:05.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2440072f0 con 0x7fe248065ff0 2023-10-16T14:36:05.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.385+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2480b1820 con 0x7fe248065ff0 2023-10-16T14:36:05.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.386+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2480b1de0 con 0x7fe248065ff0 2023-10-16T14:36:05.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.387+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2480b2300 con 0x7fe248065ff0 2023-10-16T14:36:05.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.388+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe244007450 con 0x7fe248065ff0 2023-10-16T14:36:05.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.389+0000 7fe23dffb640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fe22c06b0b0 0x7fe22c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.389+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe244089180 con 0x7fe248065ff0 2023-10-16T14:36:05.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.389+0000 7fe24c864640 1 -- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fe22c06b0b0 msgr2=0x7fe22c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:05.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.389+0000 7fe24c864640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fe22c06b0b0 0x7fe22c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:05.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.393+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2480b2300 con 0x7fe248065ff0 2023-10-16T14:36:05.399 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.396+0000 7f07b0024040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:36:05.405 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.403+0000 7fd255614040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:36:05.449 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.446+0000 7f07b0024040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:36:05.461 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.458+0000 7fd255614040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:36:05.510 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.506+0000 7f07b0024040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:36:05.514 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.511+0000 7fd255614040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:36:05.559 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.555+0000 7f07b0024040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:36:05.578 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.576+0000 7fd255614040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:36:05.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.586+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2480b2300 con 0x7fe248065ff0 2023-10-16T14:36:05.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.587+0000 7fe23dffb640 1 -- 172.21.15.29:0/3973609975 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2480b2300 con 0x7fe248065ff0 2023-10-16T14:36:05.589 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:36:05.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fe22c06b0b0 msgr2=0x7fe22c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:05.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fe22c06b0b0 0x7fe22c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 msgr2=0x7fe2481f68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 0x7fe2481f68f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe2440010e0 tx=0x7fe24400c890 comp rx=0 tx=0).stop 2023-10-16T14:36:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 shutdown_connections 2023-10-16T14:36:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fe22c06b0b0 0x7fe22c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2481618b0 0x7fe248158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe24815d330 0x7fe2480aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 --2- 172.21.15.29:0/3973609975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe248065ff0 0x7fe2481f68f0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.589+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 >> 172.21.15.29:0/3973609975 conn(0x7fe2480d0ab0 msgr2=0x7fe248159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.590+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 shutdown_connections 2023-10-16T14:36:05.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.590+0000 7fe24f2c9640 1 -- 172.21.15.29:0/3973609975 wait complete. 2023-10-16T14:36:05.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:36:05.632 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.629+0000 7fd255614040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:36:05.669 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.665+0000 7f07b0024040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:36:05.731 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.727+0000 7f07b0024040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:05.750 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.747+0000 7fd255614040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:36:05.779 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.776+0000 7f07b0024040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:36:05.817 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.814+0000 7fd255614040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:05.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.823+0000 7fdfdd275640 1 Processor -- start 2023-10-16T14:36:05.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.823+0000 7fdfdd275640 1 -- start start 2023-10-16T14:36:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.824+0000 7fdfdd275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 0x7fdfd80b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfdd275640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80cd790 con 0x7fdfd80b4f80 2023-10-16T14:36:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfdd275640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80ce310 con 0x7fdfd80b20f0 2023-10-16T14:36:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfdd275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80cd9e0 con 0x7fdfd80661c0 2023-10-16T14:36:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfd77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdfd80b4f80 0x7fdfd815c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54176/0 (socket says 172.21.15.29:54176) 2023-10-16T14:36:05.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfd77fe640 1 -- 172.21.15.29:0/721572263 learned_addr learned my addr 172.21.15.29:0/721572263 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:05.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfcffff640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 0x7fdfd80b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067920556 0 0) 0x7fdfd80cd790 con 0x7fdfd80b4f80 2023-10-16T14:36:05.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.825+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc4003520 con 0x7fdfd80b4f80 2023-10-16T14:36:05.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233172759 0 0) 0x7fdfc4003520 con 0x7fdfd80b4f80 2023-10-16T14:36:05.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 msgr2=0x7fdfd80b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:05.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 0x7fdfd80b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 >> v1:172.21.15.170:6789/0 conn(0x7fdfd80b20f0 legacy=0x7fdfd8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd80cdcc0 con 0x7fdfd80b4f80 2023-10-16T14:36:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fdfd00019f0 con 0x7fdfd80b4f80 2023-10-16T14:36:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 >> v1:172.21.15.29:6789/0 conn(0x7fdfd80b4f80 legacy=0x7fdfd815c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfc40039a0 0x7fdfc4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 0x7fdfc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80cd9e0 con 0x7fdfc4004d00 2023-10-16T14:36:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc400bb40 con 0x7fdfc40039a0 2023-10-16T14:36:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc400bd00 con 0x7fdfd80661c0 2023-10-16T14:36:05.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.826+0000 7fdfd77fe640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd77fe640 1 -- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 msgr2=0x7fdfc40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd77fe640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80661c0 0x7fdfc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd77fe640 1 -- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfc40039a0 msgr2=0x7fdfc4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd6ffd640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfc40039a0 0x7fdfc4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd77fe640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfc40039a0 0x7fdfc4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd77fe640 1 -- 172.21.15.29:0/721572263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4003520 con 0x7fdfc4004d00 2023-10-16T14:36:05.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.827+0000 7fdfd6ffd640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfc40039a0 0x7fdfc4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:05.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.828+0000 7fdfd77fe640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfc4005180 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000a760 tx=0x7fdfc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bc1f58406ed746 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:05.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0034630 con 0x7fdfc4004d00 2023-10-16T14:36:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd002f070 con 0x7fdfc4004d00 2023-10-16T14:36:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfd67fc640 1 -- 172.21.15.29:0/721572263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00349f0 con 0x7fdfc4004d00 2023-10-16T14:36:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfdd275640 1 -- 172.21.15.29:0/721572263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 msgr2=0x7fdfc4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfdd275640 1 --2- 172.21.15.29:0/721572263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfc4005180 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000a760 tx=0x7fdfc40057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfd77fe640 1 -- 172.21.15.29:0/721572263 reap_dead start 2023-10-16T14:36:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfdd275640 1 -- 172.21.15.29:0/721572263 shutdown_connections 2023-10-16T14:36:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfdd275640 1 -- 172.21.15.29:0/721572263 >> 172.21.15.29:0/721572263 conn(0x7fdfd80d2d10 msgr2=0x7fdfd80d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfdd275640 1 -- 172.21.15.29:0/721572263 shutdown_connections 2023-10-16T14:36:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.829+0000 7fdfdd275640 1 -- 172.21.15.29:0/721572263 wait complete. 2023-10-16T14:36:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.831+0000 7fdfdd275640 1 Processor -- start 2023-10-16T14:36:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.831+0000 7fdfdd275640 1 -- start start 2023-10-16T14:36:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.832+0000 7fdfdd275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfd81551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.832+0000 7fdfdd275640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfd80ca9d0 0x7fdfd80b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.832+0000 7fdfdd275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80b6a90 0x7fdfd81f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.832+0000 7fdfdd275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd81f3b00 con 0x7fdfc4004d00 2023-10-16T14:36:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.832+0000 7fdfdd275640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80b6f40 con 0x7fdfd80ca9d0 2023-10-16T14:36:05.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.832+0000 7fdfdd275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd815c150 con 0x7fdfd80b6a90 2023-10-16T14:36:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfd81551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfd81551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50978/0 (socket says 172.21.15.29:50978) 2023-10-16T14:36:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 -- 172.21.15.29:0/2051177142 learned_addr learned my addr 172.21.15.29:0/2051177142 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 -- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80b6a90 msgr2=0x7fdfd81f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd77fe640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80b6a90 0x7fdfd81f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80b6a90 0x7fdfd81f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfcffff640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfd80ca9d0 0x7fdfd80b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 -- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfd80ca9d0 msgr2=0x7fdfd80b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfd80ca9d0 0x7fdfd80b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 -- 172.21.15.29:0/2051177142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd8155710 con 0x7fdfc4004d00 2023-10-16T14:36:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.833+0000 7fdfd6ffd640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfd81551d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800d7a0 tx=0x7fdfc800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.834+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8017070 con 0x7fdfc4004d00 2023-10-16T14:36:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.834+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd81f3820 con 0x7fdfc4004d00 2023-10-16T14:36:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.834+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc8005070 con 0x7fdfc4004d00 2023-10-16T14:36:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.834+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8005420 con 0x7fdfc4004d00 2023-10-16T14:36:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.834+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd80ca1d0 con 0x7fdfc4004d00 2023-10-16T14:36:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.835+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd80c9d50 con 0x7fdfc4004d00 2023-10-16T14:36:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.836+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fdfc8007030 con 0x7fdfc4004d00 2023-10-16T14:36:05.848 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.834+0000 7f07b0024040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:36:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.837+0000 7fdfcf7fe640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 0x7fdfb806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.837+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fdfc8089480 con 0x7fdfc4004d00 2023-10-16T14:36:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.837+0000 7fdfcffff640 1 -- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 msgr2=0x7fdfb806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.837+0000 7fdfcffff640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 0x7fdfb806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:05.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:05.841+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdfd80c9d50 con 0x7fdfc4004d00 2023-10-16T14:36:05.879 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.876+0000 7fd255614040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:36:05.888 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:05.884+0000 7f07b0024040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:36:05.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.937+0000 7fd255614040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:36:05.993 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:05.990+0000 7fd255614040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:36:06.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.038+0000 7fdfcffff640 1 -- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 msgr2=0x7fdfb806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:06.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.038+0000 7fdfcffff640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 0x7fdfb806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:06.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.042+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdfd8155710 con 0x7fdfc4004d00 2023-10-16T14:36:06.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.043+0000 7fdfcf7fe640 1 -- 172.21.15.29:0/2051177142 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdfd8155710 con 0x7fdfc4004d00 2023-10-16T14:36:06.045 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:36:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 msgr2=0x7fdfb806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 0x7fdfb806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 msgr2=0x7fdfd81551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfd81551d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800d7a0 tx=0x7fdfc800dc60 comp rx=0 tx=0).stop 2023-10-16T14:36:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 shutdown_connections 2023-10-16T14:36:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fdfb806b0b0 0x7fdfb806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfd80b6a90 0x7fdfd81f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfd80ca9d0 0x7fdfd80b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 --2- 172.21.15.29:0/2051177142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfc4004d00 0x7fdfd81551d0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.045+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 >> 172.21.15.29:0/2051177142 conn(0x7fdfd80d2d10 msgr2=0x7fdfd815ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.046+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 shutdown_connections 2023-10-16T14:36:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.046+0000 7fdfdd275640 1 -- 172.21.15.29:0/2051177142 wait complete. 2023-10-16T14:36:06.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:36:06.265 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.262+0000 7f07b0024040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:36:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.326+0000 7fbc7f97c640 1 Processor -- start 2023-10-16T14:36:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.326+0000 7fbc7f97c640 1 -- start start 2023-10-16T14:36:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.328+0000 7fbc7f97c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbc78070830 0x7fbc7806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.328+0000 7fbc7f97c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc7806c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.328+0000 7fbc7f97c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78161740 con 0x7fbc78070c10 2023-10-16T14:36:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.328+0000 7fbc7f97c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7806c680 con 0x7fbc78070830 2023-10-16T14:36:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.328+0000 7fbc7f97c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7806b0f0 con 0x7fbc780cf5f0 2023-10-16T14:36:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.329+0000 7fbc7cf17640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc7806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.329+0000 7fbc7cf17640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc7806c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:41468/0 (socket says 172.21.15.29:41468) 2023-10-16T14:36:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.329+0000 7fbc7cf17640 1 -- 172.21.15.29:0/2911396469 learned_addr learned my addr 172.21.15.29:0/2911396469 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.329+0000 7fbc7d718640 1 --2- 172.21.15.29:0/2911396469 >> v2:172.21.15.170:3300/0 conn(0x7fbc78070830 0x7fbc7806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.330+0000 7fbc7d718640 1 -- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 msgr2=0x7fbc7806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.330+0000 7fbc7d718640 1 --2- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc7806c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.330+0000 7fbc7d718640 1 -- 172.21.15.29:0/2911396469 >> v1:172.21.15.170:6789/0 conn(0x7fbc78070c10 legacy=0x7fbc780cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.330+0000 7fbc7d718640 1 -- 172.21.15.29:0/2911396469 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc781619b0 con 0x7fbc78070830 2023-10-16T14:36:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.330+0000 7fbc7cf17640 1 --2- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc7806c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.331+0000 7fbc7d718640 1 --2- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070830 0x7fbc7806d3c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbc740097a0 tx=0x7fbc7402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d07463e3d0c5f238 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.331+0000 7fbc6ffff640 1 -- 172.21.15.29:0/2911396469 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7402f050 con 0x7fbc78070830 2023-10-16T14:36:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc6ffff640 1 -- 172.21.15.29:0/2911396469 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74031700 con 0x7fbc78070830 2023-10-16T14:36:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc6ffff640 1 -- 172.21.15.29:0/2911396469 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74031a90 con 0x7fbc78070830 2023-10-16T14:36:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 -- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070830 msgr2=0x7fbc7806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070830 0x7fbc7806d3c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbc740097a0 tx=0x7fbc7402b7a0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 -- 172.21.15.29:0/2911396469 shutdown_connections 2023-10-16T14:36:06.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc7806c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/2911396469 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070830 0x7fbc7806d3c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 -- 172.21.15.29:0/2911396469 >> 172.21.15.29:0/2911396469 conn(0x7fbc780cfac0 msgr2=0x7fbc780cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:06.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 -- 172.21.15.29:0/2911396469 shutdown_connections 2023-10-16T14:36:06.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.332+0000 7fbc7f97c640 1 -- 172.21.15.29:0/2911396469 wait complete. 2023-10-16T14:36:06.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.334+0000 7fbc7f97c640 1 Processor -- start 2023-10-16T14:36:06.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.334+0000 7fbc7f97c640 1 -- start start 2023-10-16T14:36:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7f97c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 0x7fbc780c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7f97c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070c10 0x7fbc780b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7f97c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc780b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7f97c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781f29b0 con 0x7fbc78070830 2023-10-16T14:36:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7f97c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780c7070 con 0x7fbc78070c10 2023-10-16T14:36:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7f97c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780c7350 con 0x7fbc780cf5f0 2023-10-16T14:36:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7d718640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 0x7fbc780c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7d718640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 0x7fbc780c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50988/0 (socket says 172.21.15.29:50988) 2023-10-16T14:36:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.335+0000 7fbc7d718640 1 -- 172.21.15.29:0/3140899140 learned_addr learned my addr 172.21.15.29:0/3140899140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7df19640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc780b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7cf17640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070c10 0x7fbc780b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7d718640 1 -- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 msgr2=0x7fbc780b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7d718640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc780b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7d718640 1 -- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070c10 msgr2=0x7fbc780b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7d718640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070c10 0x7fbc780b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7d718640 1 -- 172.21.15.29:0/3140899140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680094d0 con 0x7fbc78070830 2023-10-16T14:36:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7df19640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc780b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7cf17640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070c10 0x7fbc780b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc7d718640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 0x7fbc780c8110 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbc74036040 tx=0x7fbc740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.336+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74033930 con 0x7fbc78070830 2023-10-16T14:36:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.337+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc74009400 con 0x7fbc78070830 2023-10-16T14:36:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.337+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74033ab0 con 0x7fbc78070830 2023-10-16T14:36:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.337+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74033e60 con 0x7fbc78070830 2023-10-16T14:36:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.337+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc780c7910 con 0x7fbc78070830 2023-10-16T14:36:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.338+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7804e160 con 0x7fbc78070830 2023-10-16T14:36:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.339+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fbc7403a040 con 0x7fbc78070830 2023-10-16T14:36:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.340+0000 7fbc6dffb640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 0x7fbc5c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.340+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbc740ab030 con 0x7fbc78070830 2023-10-16T14:36:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.340+0000 7fbc7cf17640 1 -- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 msgr2=0x7fbc5c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:06.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.340+0000 7fbc7cf17640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 0x7fbc5c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:06.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.347+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc74075a90 con 0x7fbc78070830 2023-10-16T14:36:06.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:06.406+0000 7fd255614040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:36:06.425 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.422+0000 7f07b0024040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:36:06.534 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.530+0000 7f07b0024040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:36:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.541+0000 7fbc7cf17640 1 -- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 msgr2=0x7fbc5c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.541+0000 7fbc7cf17640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 0x7fbc5c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.542+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc780cbab0 con 0x7fbc78070830 2023-10-16T14:36:06.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.543+0000 7fbc6dffb640 1 -- 172.21.15.29:0/3140899140 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc740790b0 con 0x7fbc78070830 2023-10-16T14:36:06.546 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:36:06.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 msgr2=0x7fbc5c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:06.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 0x7fbc5c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 msgr2=0x7fbc780c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 0x7fbc780c8110 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbc74036040 tx=0x7fbc740046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 shutdown_connections 2023-10-16T14:36:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7fbc5c06b0b0 0x7fbc5c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbc780cf5f0 0x7fbc780b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc78070c10 0x7fbc780b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 --2- 172.21.15.29:0/3140899140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc78070830 0x7fbc780c8110 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 >> 172.21.15.29:0/3140899140 conn(0x7fbc780cfac0 msgr2=0x7fbc780ae4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 shutdown_connections 2023-10-16T14:36:06.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.546+0000 7fbc7f97c640 1 -- 172.21.15.29:0/3140899140 wait complete. 2023-10-16T14:36:06.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:36:06.579 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:06.577+0000 7fd255614040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:36:06.583 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.580+0000 7f07b0024040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:36:06.690 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.687+0000 7f07b0024040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:06.696 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:06.693+0000 7fd255614040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:36:06.749 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:06.747+0000 7fd255614040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:36:06.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.772+0000 7f0cc3fff640 1 Processor -- start 2023-10-16T14:36:06.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.772+0000 7f0cc3fff640 1 -- start start 2023-10-16T14:36:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc415aaa0 0x7f0cc40b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cc40ca8a0 0x7f0cc40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc3fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc40cad00 con 0x7f0cc4070240 2023-10-16T14:36:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc40b2d00 con 0x7f0cc40ca8a0 2023-10-16T14:36:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc40b2fe0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cc40ca8a0 0x7f0cc40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cc40ca8a0 0x7f0cc40b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51002/0 (socket says 172.21.15.29:51002) 2023-10-16T14:36:06.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.774+0000 7f0cc27fc640 1 -- 172.21.15.29:0/3125950149 learned_addr learned my addr 172.21.15.29:0/3125950149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:06.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.775+0000 7f0cc2ffd640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc415aaa0 0x7f0cc40b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.775+0000 7f0cc27fc640 1 -- 172.21.15.29:0/3125950149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc415aaa0 msgr2=0x7f0cc40b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.775+0000 7f0cc27fc640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc415aaa0 0x7f0cc40b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.775+0000 7f0cc27fc640 1 -- 172.21.15.29:0/3125950149 >> v1:172.21.15.170:6789/0 conn(0x7f0cc4070240 legacy=0x7f0cc40c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.775+0000 7f0cc27fc640 1 -- 172.21.15.29:0/3125950149 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc40b32c0 con 0x7f0cc40ca8a0 2023-10-16T14:36:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.775+0000 7f0cc2ffd640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc415aaa0 0x7f0cc40b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.776+0000 7f0cc27fc640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc40b3da0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009870 tx=0x7f0cb802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f78a9c917edc1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.776+0000 7f0cc1ffb640 1 -- 172.21.15.29:0/3125950149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031640 con 0x7f0cc40ca8a0 2023-10-16T14:36:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.776+0000 7f0cc1ffb640 1 -- 172.21.15.29:0/3125950149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb80317a0 con 0x7f0cc40ca8a0 2023-10-16T14:36:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.776+0000 7f0cc1ffb640 1 -- 172.21.15.29:0/3125950149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031b30 con 0x7f0cc40ca8a0 2023-10-16T14:36:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 -- 172.21.15.29:0/3125950149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc40ca8a0 msgr2=0x7f0cc40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc40b3da0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009870 tx=0x7f0cb802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 -- 172.21.15.29:0/3125950149 shutdown_connections 2023-10-16T14:36:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc40b3da0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/3125950149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc415aaa0 0x7f0cc40b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 -- 172.21.15.29:0/3125950149 >> 172.21.15.29:0/3125950149 conn(0x7f0cc40cc230 msgr2=0x7f0cc40cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 -- 172.21.15.29:0/3125950149 shutdown_connections 2023-10-16T14:36:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.777+0000 7f0cc3fff640 1 -- 172.21.15.29:0/3125950149 wait complete. 2023-10-16T14:36:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.779+0000 7f0cc3fff640 1 Processor -- start 2023-10-16T14:36:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.779+0000 7f0cc3fff640 1 -- start start 2023-10-16T14:36:06.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.780+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc4070240 0x7f0cc41f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.780+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc4152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.780+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 0x7f0cc4152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.780+0000 7f0cc3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc415b7a0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.780+0000 7f0cc3fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc415bd60 con 0x7f0cc40ca8a0 2023-10-16T14:36:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.780+0000 7f0cc3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc41f0650 con 0x7f0cc4070240 2023-10-16T14:36:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 0x7f0cc4152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 0x7f0cc4152fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51018/0 (socket says 172.21.15.29:51018) 2023-10-16T14:36:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 -- 172.21.15.29:0/1810793380 learned_addr learned my addr 172.21.15.29:0/1810793380 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc2ffd640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc4070240 0x7f0cc41f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc27fc640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc4152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:06.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 -- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc4070240 msgr2=0x7f0cc41f0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc4070240 0x7f0cc41f0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 -- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cc40ca8a0 msgr2=0x7f0cc4152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:06.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc4152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:06.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.781+0000 7f0cc37fe640 1 -- 172.21.15.29:0/1810793380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb80094d0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.782+0000 7f0cc37fe640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 0x7f0cc4152fc0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c7b0 tx=0x7f0cbc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:06.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.782+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc004b60 con 0x7f0cc415aaa0 2023-10-16T14:36:06.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.782+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc004cc0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.782+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00aa30 con 0x7f0cc415aaa0 2023-10-16T14:36:06.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.782+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc4156e80 con 0x7f0cc415aaa0 2023-10-16T14:36:06.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.782+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc415b9f0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.784+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc404e9b0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.784+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0cbc00ab90 con 0x7f0cc415aaa0 2023-10-16T14:36:06.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.785+0000 7f0cabfff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 0x7f0cb006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:06.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.785+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0cbc089d90 con 0x7f0cc415aaa0 2023-10-16T14:36:06.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.785+0000 7f0cc2ffd640 1 -- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 msgr2=0x7f0cb006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:06.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.785+0000 7f0cc2ffd640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 0x7f0cb006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:06.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.789+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0cbc0548e0 con 0x7f0cc415aaa0 2023-10-16T14:36:06.798 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.795+0000 7f07b0024040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:36:06.890 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:06.887+0000 7fd255614040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:06.955 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:06.951+0000 7f07b0024040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.986+0000 7f0cc2ffd640 1 -- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 msgr2=0x7f0cb006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:06.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:06.986+0000 7f0cc2ffd640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 0x7f0cb006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:07.003 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:07.000+0000 7fd255614040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:36:07.004 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:07.001+0000 7f07b0024040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:07.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.020+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0cc415bea0 con 0x7f0cc415aaa0 2023-10-16T14:36:07.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.021+0000 7f0cabfff640 1 -- 172.21.15.29:0/1810793380 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v128) v1 ==== 80+0+478 (secure 0 0 0) 0x7f0cbc057f00 con 0x7f0cc415aaa0 2023-10-16T14:36:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3330302692 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3494288089 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/1801486769 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2874036200 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/1801486769 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3741475474 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1228162669 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1339480091 2023-10-17T14:35:42.393479+0000 2023-10-16T14:36:07.027 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T14:36:07.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.023+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 msgr2=0x7f0cb006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 0x7f0cb006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 msgr2=0x7f0cc4152fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 0x7f0cc4152fc0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c7b0 tx=0x7f0cbc00cc70 comp rx=0 tx=0).stop 2023-10-16T14:36:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 shutdown_connections 2023-10-16T14:36:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f0cb006b0b0 0x7f0cb006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cc415aaa0 0x7f0cc4152fc0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cc40ca8a0 0x7f0cc4152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 --2- 172.21.15.29:0/1810793380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cc4070240 0x7f0cc41f0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 >> 172.21.15.29:0/1810793380 conn(0x7f0cc40cc230 msgr2=0x7f0cc40c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:07.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 shutdown_connections 2023-10-16T14:36:07.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.024+0000 7f0cc3fff640 1 -- 172.21.15.29:0/1810793380 wait complete. 2023-10-16T14:36:07.045 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:36:07.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:07.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:36:07.124 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:07.120+0000 7f07b0024040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:36:07.170 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:07.167+0000 7fd255614040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:07.194 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:36:07.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:07.195 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:36:07.226 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:07.223+0000 7fd255614040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:07.241 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:36:07.242 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:07.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:36:07.295 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:07.295 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:07.351 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:36:07.351 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:36:07.352 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:07.353 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:07.351+0000 7fd255614040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:36:07.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:36:07.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:07.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.619+0000 7f6ae8039640 1 Processor -- start 2023-10-16T14:36:07.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.620+0000 7f6ae8039640 1 -- start start 2023-10-16T14:36:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.621+0000 7f6ae8039640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 0x7f6ae015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.621+0000 7f6ae8039640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae015ed60 con 0x7f6ae015aaa0 2023-10-16T14:36:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.621+0000 7f6ae8039640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00707e0 con 0x7f6ae00ce390 2023-10-16T14:36:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.621+0000 7f6ae8039640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae006b810 con 0x7f6ae015ae80 2023-10-16T14:36:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.621+0000 7f6ae5dd5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6ae015aaa0 0x7f6ae00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54182/0 (socket says 172.21.15.29:54182) 2023-10-16T14:36:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae5dd5640 1 -- 172.21.15.29:0/4271047501 learned_addr learned my addr 172.21.15.29:0/4271047501 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:07.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 0x7f6ae015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668219827 0 0) 0x7f6ae015ed60 con 0x7f6ae015aaa0 2023-10-16T14:36:07.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6acc003520 con 0x7f6ae015aaa0 2023-10-16T14:36:07.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285532842 0 0) 0x7f6ae00707e0 con 0x7f6ae00ce390 2023-10-16T14:36:07.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae015ed60 con 0x7f6ae00ce390 2023-10-16T14:36:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 876224380 0 0) 0x7f6acc003520 con 0x7f6ae015aaa0 2023-10-16T14:36:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 msgr2=0x7f6ae015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 0x7f6ae015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.622+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 >> v1:172.21.15.170:6789/0 conn(0x7f6ae00ce390 legacy=0x7f6ae0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae015ef90 con 0x7f6ae015aaa0 2023-10-16T14:36:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6adc001670 con 0x7f6ae015aaa0 2023-10-16T14:36:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 >> v1:172.21.15.29:6789/0 conn(0x7f6ae015aaa0 legacy=0x7f6ae00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc003b30 0x7f6acc003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc0047c0 0x7f6acc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 0x7f6acc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae006b810 con 0x7f6acc0047c0 2023-10-16T14:36:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc00ba60 con 0x7f6acc003b30 2023-10-16T14:36:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc00bc20 con 0x7f6ae015ae80 2023-10-16T14:36:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc0047c0 0x7f6acc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae5dd5640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc003b30 0x7f6acc003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae65d6640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 0x7f6acc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae55d4640 1 -- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 msgr2=0x7f6acc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae015ae80 0x7f6acc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae55d4640 1 -- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc003b30 msgr2=0x7f6acc003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.623+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc003b30 0x7f6acc003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.624+0000 7f6ae55d4640 1 -- 172.21.15.29:0/4271047501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc003520 con 0x7f6acc0047c0 2023-10-16T14:36:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.624+0000 7f6ae5dd5640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc003b30 0x7f6acc003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc0047c0 0x7f6acc004c40 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009ac0 tx=0x7f6acc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2793d25818d64d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad002c6d0 con 0x7f6acc0047c0 2023-10-16T14:36:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad002c830 con 0x7f6acc0047c0 2023-10-16T14:36:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae4dd3640 1 -- 172.21.15.29:0/4271047501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad002cbc0 con 0x7f6acc0047c0 2023-10-16T14:36:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae8039640 1 -- 172.21.15.29:0/4271047501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc0047c0 msgr2=0x7f6acc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae8039640 1 --2- 172.21.15.29:0/4271047501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6acc0047c0 0x7f6acc004c40 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009ac0 tx=0x7f6acc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.625+0000 7f6ae65d6640 1 -- 172.21.15.29:0/4271047501 reap_dead start 2023-10-16T14:36:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.626+0000 7f6ae8039640 1 -- 172.21.15.29:0/4271047501 shutdown_connections 2023-10-16T14:36:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.626+0000 7f6ae8039640 1 -- 172.21.15.29:0/4271047501 >> 172.21.15.29:0/4271047501 conn(0x7f6ae00cb410 msgr2=0x7f6ae00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.626+0000 7f6ae8039640 1 -- 172.21.15.29:0/4271047501 shutdown_connections 2023-10-16T14:36:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.626+0000 7f6ae8039640 1 -- 172.21.15.29:0/4271047501 wait complete. 2023-10-16T14:36:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.627+0000 7f6ae8039640 1 Processor -- start 2023-10-16T14:36:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.628+0000 7f6ae8039640 1 -- start start 2023-10-16T14:36:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.628+0000 7f6ae8039640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc0047c0 0x7f6ae00c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.628+0000 7f6ae8039640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 0x7f6ae0158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.628+0000 7f6ae8039640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae01f2710 0x7f6ae0158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae8039640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01f6cf0 con 0x7f6ae00c43e0 2023-10-16T14:36:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae8039640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01f6170 con 0x7f6acc0047c0 2023-10-16T14:36:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae8039640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01f6450 con 0x7f6ae01f2710 2023-10-16T14:36:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 0x7f6ae0158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 0x7f6ae0158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51036/0 (socket says 172.21.15.29:51036) 2023-10-16T14:36:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 -- 172.21.15.29:0/1700244314 learned_addr learned my addr 172.21.15.29:0/1700244314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae5dd5640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc0047c0 0x7f6ae00c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae65d6640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae01f2710 0x7f6ae0158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 -- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae01f2710 msgr2=0x7f6ae0158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae01f2710 0x7f6ae0158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 -- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc0047c0 msgr2=0x7f6ae00c17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc0047c0 0x7f6ae00c17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.629+0000 7f6ae55d4640 1 -- 172.21.15.29:0/1700244314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae01f6730 con 0x7f6ae00c43e0 2023-10-16T14:36:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.630+0000 7f6ae55d4640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 0x7f6ae0158030 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0037040 tx=0x7f6ad0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.630+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0003810 con 0x7f6ae00c43e0 2023-10-16T14:36:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.630+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad0003970 con 0x7f6ae00c43e0 2023-10-16T14:36:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.630+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0003d20 con 0x7f6ae00c43e0 2023-10-16T14:36:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.630+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae01f6a10 con 0x7f6ae00c43e0 2023-10-16T14:36:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.630+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae01f5d00 con 0x7f6ae00c43e0 2023-10-16T14:36:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.631+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae004e160 con 0x7f6ae00c43e0 2023-10-16T14:36:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.632+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f6ad0034440 con 0x7f6ae00c43e0 2023-10-16T14:36:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.634+0000 7f6ad67fc640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 0x7f6ac406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.634+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6ad00abad0 con 0x7f6ae00c43e0 2023-10-16T14:36:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.634+0000 7f6ae5dd5640 1 -- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 msgr2=0x7f6ac406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.634+0000 7f6ae5dd5640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 0x7f6ac406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.638+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ad00767f0 con 0x7f6ae00c43e0 2023-10-16T14:36:07.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.835+0000 7f6ae5dd5640 1 -- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 msgr2=0x7f6ac406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:07.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.835+0000 7f6ae5dd5640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 0x7f6ac406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.870+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ae015b2c0 con 0x7f6ae00c43e0 2023-10-16T14:36:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.871+0000 7f6ad67fc640 1 -- 172.21.15.29:0/1700244314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f6ad0079e10 con 0x7f6ae00c43e0 2023-10-16T14:36:07.874 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:07.874 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:35:27.247546+0000","modified":"2023-10-16T14:35:30.329395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7812":{"gid":7812,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2882189329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2882189329},{"type":"v1","addr":"172.21.15.29:6835","nonce":2882189329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T14:36:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2023-10-16T14:36:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 msgr2=0x7f6ac406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 0x7f6ac406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 msgr2=0x7f6ae0158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 0x7f6ae0158030 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0037040 tx=0x7f6ad0004730 comp rx=0 tx=0).stop 2023-10-16T14:36:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 shutdown_connections 2023-10-16T14:36:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f6ac406b0b0 0x7f6ac406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae00c43e0 0x7f6ae0158030 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ae01f2710 0x7f6ae0158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 --2- 172.21.15.29:0/1700244314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6acc0047c0 0x7f6ae00c17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 >> 172.21.15.29:0/1700244314 conn(0x7f6ae00cb410 msgr2=0x7f6ae00b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 shutdown_connections 2023-10-16T14:36:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:07.873+0000 7f6ae8039640 1 -- 172.21.15.29:0/1700244314 wait complete. 2023-10-16T14:36:07.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:07.955 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:07.952+0000 7f07b0024040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:36:08.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.085+0000 7f16b2e72640 1 Processor -- start 2023-10-16T14:36:08.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.085+0000 7f16b2e72640 1 -- start start 2023-10-16T14:36:08.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.087+0000 7f16b2e72640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f16ac06d220 0x7f16ac0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.087+0000 7f16b2e72640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0acd70 con 0x7f16ac0ce280 2023-10-16T14:36:08.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.087+0000 7f16b2e72640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0ac4a0 con 0x7f16ac065ab0 2023-10-16T14:36:08.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.087+0000 7f16b2e72640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac065f40 con 0x7f16ac06d220 2023-10-16T14:36:08.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b2671640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16ac0ce280 0x7f16ac0ad200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54186/0 (socket says 172.21.15.29:54186) 2023-10-16T14:36:08.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b2671640 1 -- 172.21.15.29:0/3639942680 learned_addr learned my addr 172.21.15.29:0/3639942680 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:08.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3639942680 >> v2:172.21.15.170:3300/0 conn(0x7f16ac06d220 0x7f16ac0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133300397 0 0) 0x7f16ac0acd70 con 0x7f16ac0ce280 2023-10-16T14:36:08.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1698003520 con 0x7f16ac0ce280 2023-10-16T14:36:08.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783027650 0 0) 0x7f1698003520 con 0x7f16ac0ce280 2023-10-16T14:36:08.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 >> v2:172.21.15.170:3300/0 conn(0x7f16ac06d220 msgr2=0x7f16ac0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:08.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 --2- 172.21.15.29:0/3639942680 >> v2:172.21.15.170:3300/0 conn(0x7f16ac06d220 0x7f16ac0c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 >> v1:172.21.15.170:6789/0 conn(0x7f16ac065ab0 legacy=0x7f16ac0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0ac720 con 0x7f16ac0ce280 2023-10-16T14:36:08.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f16a40019f0 con 0x7f16ac0ce280 2023-10-16T14:36:08.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3639942680 >> v2:172.21.15.170:3300/0 conn(0x7f16ac06d220 0x7f16ac0c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:08.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.088+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 >> v1:172.21.15.29:6789/0 conn(0x7f16ac0ce280 legacy=0x7f16ac0ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b0e6e640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16980039c0 0x7f1698003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b0e6e640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f1698004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b0e6e640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16980052a0 0x7f169800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac065f40 con 0x7f16980047c0 2023-10-16T14:36:08.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169800c1c0 con 0x7f16980052a0 2023-10-16T14:36:08.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169800c380 con 0x7f16980039c0 2023-10-16T14:36:08.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b166f640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f1698004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b2671640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16980052a0 0x7f169800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b166f640 1 -- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16980039c0 msgr2=0x7f1698003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:08.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16980039c0 0x7f1698003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b166f640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16980039c0 0x7f1698003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b166f640 1 -- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16980052a0 msgr2=0x7f169800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:08.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b166f640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16980052a0 0x7f169800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b166f640 1 -- 172.21.15.29:0/3639942680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698003520 con 0x7f16980047c0 2023-10-16T14:36:08.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b2671640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16980052a0 0x7f169800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:08.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b2671640 1 -- 172.21.15.29:0/3639942680 reap_dead start 2023-10-16T14:36:08.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.089+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16980039c0 0x7f1698003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:08.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.091+0000 7f16b166f640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f1698004c40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f169c001750 tx=0x7f16980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74bd650ca9b3637 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.091+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c0052d0 con 0x7f16980047c0 2023-10-16T14:36:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.091+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c02e050 con 0x7f16980047c0 2023-10-16T14:36:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.091+0000 7f16b0e6e640 1 -- 172.21.15.29:0/3639942680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c003830 con 0x7f16980047c0 2023-10-16T14:36:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.091+0000 7f16b2e72640 1 -- 172.21.15.29:0/3639942680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 msgr2=0x7f1698004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.091+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f1698004c40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f169c001750 tx=0x7f16980057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.092+0000 7f16b2e72640 1 -- 172.21.15.29:0/3639942680 shutdown_connections 2023-10-16T14:36:08.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.092+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3639942680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f1698004c40 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.092+0000 7f16b2e72640 1 -- 172.21.15.29:0/3639942680 >> 172.21.15.29:0/3639942680 conn(0x7f16ac0b0a80 msgr2=0x7f16ac0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:08.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.092+0000 7f16b2e72640 1 -- 172.21.15.29:0/3639942680 shutdown_connections 2023-10-16T14:36:08.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.092+0000 7f16b2e72640 1 -- 172.21.15.29:0/3639942680 wait complete. 2023-10-16T14:36:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.093+0000 7f16b2e72640 1 Processor -- start 2023-10-16T14:36:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.094+0000 7f16b2e72640 1 -- start start 2023-10-16T14:36:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.094+0000 7f16b2e72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f16ac0b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.094+0000 7f16b2e72640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16ac0b4550 0x7f16ac09c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.094+0000 7f16b2e72640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16ac1f6270 0x7f16ac1f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.094+0000 7f16b2e72640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1517f0 con 0x7f16980047c0 2023-10-16T14:36:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.094+0000 7f16b2e72640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1f6c30 con 0x7f16ac1f6270 2023-10-16T14:36:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b2e72640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0b4ac0 con 0x7f16ac0b4550 2023-10-16T14:36:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f16ac0b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f16ac0b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51058/0 (socket says 172.21.15.29:51058) 2023-10-16T14:36:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 -- 172.21.15.29:0/3157727483 learned_addr learned my addr 172.21.15.29:0/3157727483 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b2671640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16ac1f6270 0x7f16ac1f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b166f640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16ac0b4550 0x7f16ac09c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 -- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16ac0b4550 msgr2=0x7f16ac09c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16ac0b4550 0x7f16ac09c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 -- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16ac1f6270 msgr2=0x7f16ac1f66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16ac1f6270 0x7f16ac1f66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.095+0000 7f16b1e70640 1 -- 172.21.15.29:0/3157727483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0b39f0 con 0x7f16980047c0 2023-10-16T14:36:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.096+0000 7f16b1e70640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f16ac0b34b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f16a800f4c0 tx=0x7f16a800f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.096+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8014890 con 0x7f16980047c0 2023-10-16T14:36:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.096+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a800c030 con 0x7f16980047c0 2023-10-16T14:36:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.096+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac0b26f0 con 0x7f16980047c0 2023-10-16T14:36:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.096+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8014c30 con 0x7f16980047c0 2023-10-16T14:36:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.096+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac0b2cb0 con 0x7f16980047c0 2023-10-16T14:36:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.097+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac04e9b0 con 0x7f16980047c0 2023-10-16T14:36:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.099+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 63) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f16a80133e0 con 0x7f16980047c0 2023-10-16T14:36:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.100+0000 7f16a27fc640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f169006b0e0 0x7f169006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.101+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f16a8089e80 con 0x7f16980047c0 2023-10-16T14:36:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.101+0000 7f16b166f640 1 -- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f169006b0e0 msgr2=0x7f169006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3633682082 2023-10-16T14:36:08.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.101+0000 7f16b166f640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f169006b0e0 0x7f169006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:08.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.105+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f16a8054970 con 0x7f16980047c0 2023-10-16T14:36:08.218 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:08.215+0000 7fd255614040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:36:08.250 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.246+0000 7f07b0024040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:36:08.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.295+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ac0c8f70 con 0x7f16980047c0 2023-10-16T14:36:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.298+0000 7f16a27fc640 1 -- 172.21.15.29:0/3157727483 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17028 (secure 0 0 0) 0x7f16a8022090 con 0x7f16980047c0 2023-10-16T14:36:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:08.301 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":128,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:35:42.393631+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3330302692":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3494288089":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6833/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/2874036200":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3741475474":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1228162669":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1339480091":"2023-10-17T14:35:42.393479+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.300+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f169006b0e0 msgr2=0x7f169006d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.300+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f169006b0e0 0x7f169006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.300+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 msgr2=0x7f16ac0b34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.300+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f16ac0b34b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f16a800f4c0 tx=0x7f16a800f980 comp rx=0 tx=0).stop 2023-10-16T14:36:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 shutdown_connections 2023-10-16T14:36:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:6832/3633682082,v1:172.21.15.170:6833/3633682082] conn(0x7f169006b0e0 0x7f169006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f16ac1f6270 0x7f16ac1f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16ac0b4550 0x7f16ac09c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 --2- 172.21.15.29:0/3157727483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980047c0 0x7f16ac0b34b0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 >> 172.21.15.29:0/3157727483 conn(0x7f16ac0b0a80 msgr2=0x7f16ac070400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 shutdown_connections 2023-10-16T14:36:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.301+0000 7f16b2e72640 1 -- 172.21.15.29:0/3157727483 wait complete. 2023-10-16T14:36:08.306 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.303+0000 7f07b0024040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:36:08.320 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:35:25.164943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:35:26.261110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:36:08.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:36:08.421 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.417+0000 7f07b0024040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:36:08.516 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.513+0000 7f0733f26640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.517 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.513+0000 7f0733f26640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.517 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.513+0000 7f0733f26640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.517 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.513+0000 7f0733f26640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.517 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.513+0000 7f0733f26640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.518 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.514+0000 7f0731f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.518 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.514+0000 7f0731f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.519 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.514+0000 7f0731f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.519 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.514+0000 7f0731f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.519 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:08.514+0000 7f0731f22640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.536+0000 7f1ac5dc4640 1 Processor -- start 2023-10-16T14:36:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.536+0000 7f1ac5dc4640 1 -- start start 2023-10-16T14:36:08.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.537+0000 7f1ac5dc4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1ac00b1ca0 0x7f1ac00d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac5dc4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b2080 0x7f1ac00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac5dc4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00d0970 con 0x7f1ac00ce960 2023-10-16T14:36:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac5dc4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac00d0ea0 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac5dc4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00d0bc0 con 0x7f1ac00b2080 2023-10-16T14:36:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac4dc2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1ac00b1ca0 0x7f1ac00d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1abffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b2080 0x7f1ac00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1abffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b2080 0x7f1ac00ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:41574/0 (socket says 172.21.15.29:41574) 2023-10-16T14:36:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac55c3640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f1ac00ce960 0x7f1ac00b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:36238/0 (socket says 172.21.15.29:36238) 2023-10-16T14:36:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1abffff640 1 -- 172.21.15.29:0/2365058358 learned_addr learned my addr 172.21.15.29:0/2365058358 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:08.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac4dc2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1ac00b1ca0 0x7f1ac00d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:36372/0 (socket says 172.21.15.29:36372) 2023-10-16T14:36:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.538+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b2080 msgr2=0x7f1ac00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.539+0000 7f1ac4dc2640 1 --2- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b2080 0x7f1ac00ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.539+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/2365058358 >> v1:172.21.15.170:6789/0 conn(0x7f1ac00ce960 legacy=0x7f1ac00b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.539+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/2365058358 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00cd940 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.539+0000 7f1abf7fe640 1 -- 172.21.15.29:0/2365058358 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625835605 0 0) 0x7f1ac00d0970 con 0x7f1ac00ce960 2023-10-16T14:36:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.540+0000 7f1ac4dc2640 1 --2- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b1ca0 0x7f1ac00d36b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40097a0 tx=0x7f1ab402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53fee041346fb431 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.540+0000 7f1abf7fe640 1 -- 172.21.15.29:0/2365058358 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4030610 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.540+0000 7f1abf7fe640 1 -- 172.21.15.29:0/2365058358 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab4030770 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.540+0000 7f1abf7fe640 1 -- 172.21.15.29:0/2365058358 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4030b00 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b1ca0 msgr2=0x7f1ac00d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b1ca0 0x7f1ac00d36b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40097a0 tx=0x7f1ab402b960 comp rx=0 tx=0).stop 2023-10-16T14:36:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/2365058358 shutdown_connections 2023-10-16T14:36:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b2080 0x7f1ac00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/2365058358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b1ca0 0x7f1ac00d36b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/2365058358 >> 172.21.15.29:0/2365058358 conn(0x7f1ac00d2b90 msgr2=0x7f1ac00d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/2365058358 shutdown_connections 2023-10-16T14:36:08.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.541+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/2365058358 wait complete. 2023-10-16T14:36:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.543+0000 7f1ac5dc4640 1 Processor -- start 2023-10-16T14:36:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.543+0000 7f1ac5dc4640 1 -- start start 2023-10-16T14:36:08.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac5dc4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 0x7f1ac00d1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac5dc4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b2080 0x7f1ac00afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac5dc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac00ce960 0x7f1ac01f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac5dc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b0300 con 0x7f1ac00ce960 2023-10-16T14:36:08.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac5dc4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac009ddb0 con 0x7f1ac00b2080 2023-10-16T14:36:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac5dc4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac009d480 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 0x7f1ac00d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1abffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b2080 0x7f1ac00afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 0x7f1ac00d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:41580/0 (socket says 172.21.15.29:41580) 2023-10-16T14:36:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/585331904 learned_addr learned my addr 172.21.15.29:0/585331904 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac55c3640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac00ce960 0x7f1ac01f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b2080 msgr2=0x7f1ac00afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b2080 0x7f1ac00afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/585331904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac00ce960 msgr2=0x7f1ac01f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac00ce960 0x7f1ac01f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.544+0000 7f1ac4dc2640 1 -- 172.21.15.29:0/585331904 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4009400 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.545+0000 7f1ac4dc2640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 0x7f1ac00d1710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ab402bf70 tx=0x7f1ab40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:08.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.545+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab403e070 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.554 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:08.545+0000 7fd255614040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:36:08.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.545+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac009d760 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.545+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab4030e10 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.545+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4039c30 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.545+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01f2600 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.546+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac00ce300 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.548+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 64) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1ab404b400 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.549+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f1ab40ab990 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.554+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ab4076b40 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:08.607+0000 7fd255614040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:36:08.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:08.726+0000 7fd255614040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:36:08.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.748+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ac009d5c0 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.749+0000 7f1abd7fa640 1 -- 172.21.15.29:0/585331904 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ac009d5c0 con 0x7f1ac00b1ca0 2023-10-16T14:36:08.752 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:36:08.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 msgr2=0x7f1ac00d1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 0x7f1ac00d1710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ab402bf70 tx=0x7f1ab40046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 shutdown_connections 2023-10-16T14:36:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac00ce960 0x7f1ac01f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1ac00b2080 0x7f1ac00afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 --2- 172.21.15.29:0/585331904 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1ac00b1ca0 0x7f1ac00d1710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 >> 172.21.15.29:0/585331904 conn(0x7f1ac00d2b90 msgr2=0x7f1ac015fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 shutdown_connections 2023-10-16T14:36:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.752+0000 7f1ac5dc4640 1 -- 172.21.15.29:0/585331904 wait complete. 2023-10-16T14:36:08.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:36:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.983+0000 7fc967db6640 1 Processor -- start 2023-10-16T14:36:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.983+0000 7fc967db6640 1 -- start start 2023-10-16T14:36:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc967db6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc960159ab0 0x7fc9600ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc967db6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc960159e90 0x7fc96006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc967db6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc96006c140 0x7fc96006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc967db6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96006cb10 con 0x7fc96006c140 2023-10-16T14:36:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc967db6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96015d890 con 0x7fc960159ab0 2023-10-16T14:36:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc967db6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600cd1f0 con 0x7fc960159e90 2023-10-16T14:36:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc966353640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc96006c140 0x7fc96006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc966353640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc96006c140 0x7fc96006c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51078/0 (socket says 172.21.15.29:51078) 2023-10-16T14:36:08.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.985+0000 7fc966353640 1 -- 172.21.15.29:0/984110554 learned_addr learned my addr 172.21.15.29:0/984110554 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc965351640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc960159e90 0x7fc96006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc965b52640 1 --2- 172.21.15.29:0/984110554 >> v2:172.21.15.170:3300/0 conn(0x7fc960159ab0 0x7fc9600ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc966353640 1 -- 172.21.15.29:0/984110554 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc960159e90 msgr2=0x7fc96006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:08.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc966353640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc960159e90 0x7fc96006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc966353640 1 -- 172.21.15.29:0/984110554 >> v2:172.21.15.170:3300/0 conn(0x7fc960159ab0 msgr2=0x7fc9600ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc966353640 1 --2- 172.21.15.29:0/984110554 >> v2:172.21.15.170:3300/0 conn(0x7fc960159ab0 0x7fc9600ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc966353640 1 -- 172.21.15.29:0/984110554 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600cd4d0 con 0x7fc96006c140 2023-10-16T14:36:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc965351640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc960159e90 0x7fc96006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.986+0000 7fc965b52640 1 --2- 172.21.15.29:0/984110554 >> v2:172.21.15.170:3300/0 conn(0x7fc960159ab0 0x7fc9600ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.988+0000 7fc966353640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc96006c140 0x7fc96006c5d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc958009e50 tx=0x7fc95802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829e05ae1e97ffd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.988+0000 7fc964b50640 1 -- 172.21.15.29:0/984110554 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802c960 con 0x7fc96006c140 2023-10-16T14:36:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.988+0000 7fc964b50640 1 -- 172.21.15.29:0/984110554 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95802cac0 con 0x7fc96006c140 2023-10-16T14:36:08.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.988+0000 7fc964b50640 1 -- 172.21.15.29:0/984110554 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802ce50 con 0x7fc96006c140 2023-10-16T14:36:08.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 -- 172.21.15.29:0/984110554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc96006c140 msgr2=0x7fc96006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc96006c140 0x7fc96006c5d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc958009e50 tx=0x7fc95802bdc0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 -- 172.21.15.29:0/984110554 shutdown_connections 2023-10-16T14:36:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc96006c140 0x7fc96006c5d0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 --2- 172.21.15.29:0/984110554 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc960159e90 0x7fc96006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 --2- 172.21.15.29:0/984110554 >> v2:172.21.15.170:3300/0 conn(0x7fc960159ab0 0x7fc9600ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 -- 172.21.15.29:0/984110554 >> 172.21.15.29:0/984110554 conn(0x7fc9600ce9b0 msgr2=0x7fc9600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:08.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 -- 172.21.15.29:0/984110554 shutdown_connections 2023-10-16T14:36:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.989+0000 7fc967db6640 1 -- 172.21.15.29:0/984110554 wait complete. 2023-10-16T14:36:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.991+0000 7fc967db6640 1 Processor -- start 2023-10-16T14:36:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.991+0000 7fc967db6640 1 -- start start 2023-10-16T14:36:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc967db6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc96006c140 0x7fc9600c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc967db6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc960159ab0 0x7fc96015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc967db6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 0x7fc9601e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc967db6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc960152480 con 0x7fc960159e90 2023-10-16T14:36:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc967db6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600c3410 con 0x7fc960159ab0 2023-10-16T14:36:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc967db6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601e65f0 con 0x7fc96006c140 2023-10-16T14:36:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 0x7fc9601e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 0x7fc9601e6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51084/0 (socket says 172.21.15.29:51084) 2023-10-16T14:36:08.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 -- 172.21.15.29:0/3167747387 learned_addr learned my addr 172.21.15.29:0/3167747387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:08.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc965351640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc960159ab0 0x7fc96015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:08.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 -- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc96006c140 msgr2=0x7fc9600c2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:08.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc96006c140 0x7fc9600c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 -- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc960159ab0 msgr2=0x7fc96015a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc960159ab0 0x7fc96015a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.992+0000 7fc966353640 1 -- 172.21.15.29:0/3167747387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc958009b30 con 0x7fc960159e90 2023-10-16T14:36:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.993+0000 7fc966353640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 0x7fc9601e6a80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc958036040 tx=0x7fc958004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.993+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95803e450 con 0x7fc960159e90 2023-10-16T14:36:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.993+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95803e5d0 con 0x7fc960159e90 2023-10-16T14:36:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.993+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95803e980 con 0x7fc960159e90 2023-10-16T14:36:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.993+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9600c3640 con 0x7fc960159e90 2023-10-16T14:36:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.993+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9600c8610 con 0x7fc960159e90 2023-10-16T14:36:09.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.995+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96004e160 con 0x7fc960159e90 2023-10-16T14:36:09.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.995+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc95803eae0 con 0x7fc960159e90 2023-10-16T14:36:09.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:08.996+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fc9580ac110 con 0x7fc960159e90 2023-10-16T14:36:09.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.000+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9580772d0 con 0x7fc960159e90 2023-10-16T14:36:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.192+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9600ccb60 con 0x7fc960159e90 2023-10-16T14:36:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.193+0000 7fc9567fc640 1 -- 172.21.15.29:0/3167747387 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc95807a8f0 con 0x7fc960159e90 2023-10-16T14:36:09.196 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:36:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 msgr2=0x7fc9601e6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 0x7fc9601e6a80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc958036040 tx=0x7fc958004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 shutdown_connections 2023-10-16T14:36:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc960159e90 0x7fc9601e6a80 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc960159ab0 0x7fc96015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 --2- 172.21.15.29:0/3167747387 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc96006c140 0x7fc9600c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 >> 172.21.15.29:0/3167747387 conn(0x7fc9600ce9b0 msgr2=0x7fc9600c1960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 shutdown_connections 2023-10-16T14:36:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.196+0000 7fc967db6640 1 -- 172.21.15.29:0/3167747387 wait complete. 2023-10-16T14:36:09.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:36:09.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.451+0000 7f6fafffc640 1 Processor -- start 2023-10-16T14:36:09.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.451+0000 7f6fafffc640 1 -- start start 2023-10-16T14:36:09.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.453+0000 7f6fafffc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6fa80cf710 0x7f6fa80b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.453+0000 7f6fafffc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6fa80cfaf0 0x7f6fa80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.453+0000 7f6fafffc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 0x7f6fa80d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.453+0000 7f6fafffc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa80d3c60 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.453+0000 7f6fafffc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa80d3980 con 0x7f6fa80cf710 2023-10-16T14:36:09.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.453+0000 7f6fafffc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa80ce630 con 0x7f6fa80b38e0 2023-10-16T14:36:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6fa80cfaf0 0x7f6fa80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6fa80cfaf0 0x7f6fa80b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51090/0 (socket says 172.21.15.29:51090) 2023-10-16T14:36:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 -- 172.21.15.29:0/3534791565 learned_addr learned my addr 172.21.15.29:0/3534791565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 -- 172.21.15.29:0/3534791565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 msgr2=0x7f6fa80d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 --2- 172.21.15.29:0/3534791565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 0x7f6fa80d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 -- 172.21.15.29:0/3534791565 >> v2:172.21.15.170:3300/0 conn(0x7f6fa80cf710 msgr2=0x7f6fa80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:09.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 --2- 172.21.15.29:0/3534791565 >> v2:172.21.15.170:3300/0 conn(0x7f6fa80cf710 0x7f6fa80b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.454+0000 7f6fad597640 1 -- 172.21.15.29:0/3534791565 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa80ce910 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.456+0000 7f6fad597640 1 --2- 172.21.15.29:0/3534791565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b3250 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80d2c20 tx=0x7f6fa402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf1f501e4534415 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.456+0000 7f6facd96640 1 -- 172.21.15.29:0/3534791565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40317e0 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.456+0000 7f6facd96640 1 -- 172.21.15.29:0/3534791565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa4031940 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.456+0000 7f6facd96640 1 -- 172.21.15.29:0/3534791565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4031cd0 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 -- 172.21.15.29:0/3534791565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 msgr2=0x7f6fa80b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 --2- 172.21.15.29:0/3534791565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b3250 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80d2c20 tx=0x7f6fa402b880 comp rx=0 tx=0).stop 2023-10-16T14:36:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 -- 172.21.15.29:0/3534791565 shutdown_connections 2023-10-16T14:36:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 --2- 172.21.15.29:0/3534791565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 0x7f6fa80d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 --2- 172.21.15.29:0/3534791565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b3250 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 --2- 172.21.15.29:0/3534791565 >> v2:172.21.15.170:3300/0 conn(0x7f6fa80cf710 0x7f6fa80b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 -- 172.21.15.29:0/3534791565 >> 172.21.15.29:0/3534791565 conn(0x7f6fa80d2cc0 msgr2=0x7f6fa80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.457+0000 7f6fafffc640 1 -- 172.21.15.29:0/3534791565 shutdown_connections 2023-10-16T14:36:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.458+0000 7f6fafffc640 1 -- 172.21.15.29:0/3534791565 wait complete. 2023-10-16T14:36:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.459+0000 7f6fafffc640 1 Processor -- start 2023-10-16T14:36:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.460+0000 7f6fafffc640 1 -- start start 2023-10-16T14:36:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fafffc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 0x7f6fa815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fafffc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa80cf710 0x7f6fa80b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fafffc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fafffc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa815a8a0 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fafffc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa80b0470 con 0x7f6fa80cf710 2023-10-16T14:36:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fafffc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa80b0750 con 0x7f6fa80b38e0 2023-10-16T14:36:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51102/0 (socket says 172.21.15.29:51102) 2023-10-16T14:36:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 -- 172.21.15.29:0/1397475608 learned_addr learned my addr 172.21.15.29:0/1397475608 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 -- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 msgr2=0x7f6fa815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 0x7f6fa815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 -- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa80cf710 msgr2=0x7f6fa80b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa80cf710 0x7f6fa80b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.461+0000 7f6fae599640 1 -- 172.21.15.29:0/1397475608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40094d0 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.462+0000 7f6fae599640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b31b0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000b970 tx=0x7f6fa000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.462+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000cb40 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.462+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa000ccc0 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.462+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0014680 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.462+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa80b0a30 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.462+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa80c7600 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.464+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa80b0f50 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.464+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6fa001d3b0 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.465+0000 7f6f9a7fc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f6f8c06acc0 0x7f6f8c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.465+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f6fa0089520 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.467+0000 7f6fadd98640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f6f8c06acc0 0x7f6f8c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.469+0000 7f6fadd98640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f6f8c06acc0 0x7f6f8c06d180 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80b80e0 tx=0x7f6f9c006c80 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.469+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6fa80b0f50 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.671+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6fa80b0f50 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.672+0000 7f6f9a7fc640 1 -- 172.21.15.29:0/1397475608 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6fa80b0f50 con 0x7f6fa80cfaf0 2023-10-16T14:36:09.675 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:36:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.674+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f6f8c06acc0 msgr2=0x7f6f8c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.674+0000 7f6fafffc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f6f8c06acc0 0x7f6f8c06d180 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80b80e0 tx=0x7f6f9c006c80 comp rx=0 tx=0).stop 2023-10-16T14:36:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.674+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 msgr2=0x7f6fa80b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.674+0000 7f6fafffc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b31b0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000b970 tx=0x7f6fa000be30 comp rx=0 tx=0).stop 2023-10-16T14:36:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 shutdown_connections 2023-10-16T14:36:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fa80cfaf0 0x7f6fa80b31b0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f6f8c06acc0 0x7f6f8c06d180 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fa80cf710 0x7f6fa80b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 --2- 172.21.15.29:0/1397475608 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fa80b38e0 0x7f6fa815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 >> 172.21.15.29:0/1397475608 conn(0x7f6fa80d2cc0 msgr2=0x7f6fa8160250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:09.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 shutdown_connections 2023-10-16T14:36:09.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.675+0000 7f6fafffc640 1 -- 172.21.15.29:0/1397475608 wait complete. 2023-10-16T14:36:09.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.916+0000 7fdedf59e640 1 Processor -- start 2023-10-16T14:36:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.916+0000 7fdedf59e640 1 -- start start 2023-10-16T14:36:09.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fdedf59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fdedf59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdee00cda80 0x7fdee015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fdedf59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee006c960 con 0x7fdee00ad970 2023-10-16T14:36:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fdedf59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee015b9d0 con 0x7fdee00cda80 2023-10-16T14:36:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fdedf59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee015bcb0 con 0x7fdee006c040 2023-10-16T14:36:09.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fded7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdee00ad970 0x7fdee015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54194/0 (socket says 172.21.15.29:54194) 2023-10-16T14:36:09.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fded7fff640 1 -- 172.21.15.29:0/273963318 learned_addr learned my addr 172.21.15.29:0/273963318 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.918+0000 7fdeded9d640 1 --2- 172.21.15.29:0/273963318 >> v2:172.21.15.170:3300/0 conn(0x7fdee00cda80 0x7fdee015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdede59c640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568368979 0 0) 0x7fdee006c960 con 0x7fdee00ad970 2023-10-16T14:36:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc003520 con 0x7fdee00ad970 2023-10-16T14:36:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3925815584 0 0) 0x7fdecc003520 con 0x7fdee00ad970 2023-10-16T14:36:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 msgr2=0x7fdee006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 0x7fdee006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 >> v2:172.21.15.170:3300/0 conn(0x7fdee00cda80 msgr2=0x7fdee015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 --2- 172.21.15.29:0/273963318 >> v2:172.21.15.170:3300/0 conn(0x7fdee00cda80 0x7fdee015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee015bf90 con 0x7fdee00ad970 2023-10-16T14:36:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fdec8001670 con 0x7fdee00ad970 2023-10-16T14:36:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 >> v1:172.21.15.29:6789/0 conn(0x7fdee00ad970 legacy=0x7fdee015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 0x7fdecc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdecc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdecc004d00 0x7fdecc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee015b9d0 con 0x7fdecc003ef0 2023-10-16T14:36:09.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee015bcb0 con 0x7fdecc004d00 2023-10-16T14:36:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.919+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc00b980 con 0x7fdee006c040 2023-10-16T14:36:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fded7fff640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdecc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fdeded9d640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdecc004d00 0x7fdecc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fded7fff640 1 -- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 msgr2=0x7fdecc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:09.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fded7fff640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee006c040 0x7fdecc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fded7fff640 1 -- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdecc004d00 msgr2=0x7fdecc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:09.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fded7fff640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdecc004d00 0x7fdecc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fded7fff640 1 -- 172.21.15.29:0/273963318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc003520 con 0x7fdecc003ef0 2023-10-16T14:36:09.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.920+0000 7fdeded9d640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdecc004d00 0x7fdecc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:09.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.921+0000 7fded7fff640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdecc0047c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c240 tx=0x7fdecc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d528af63ef864a0b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec802cb00 con 0x7fdecc003ef0 2023-10-16T14:36:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec802cc60 con 0x7fdecc003ef0 2023-10-16T14:36:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdeddd9b640 1 -- 172.21.15.29:0/273963318 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8005af0 con 0x7fdecc003ef0 2023-10-16T14:36:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdedf59e640 1 -- 172.21.15.29:0/273963318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 msgr2=0x7fdecc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdedf59e640 1 --2- 172.21.15.29:0/273963318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdecc0047c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c240 tx=0x7fdecc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdeded9d640 1 -- 172.21.15.29:0/273963318 reap_dead start 2023-10-16T14:36:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdedf59e640 1 -- 172.21.15.29:0/273963318 shutdown_connections 2023-10-16T14:36:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdedf59e640 1 -- 172.21.15.29:0/273963318 >> 172.21.15.29:0/273963318 conn(0x7fdee00ce6a0 msgr2=0x7fdee00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdedf59e640 1 -- 172.21.15.29:0/273963318 shutdown_connections 2023-10-16T14:36:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.922+0000 7fdedf59e640 1 -- 172.21.15.29:0/273963318 wait complete. 2023-10-16T14:36:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.924+0000 7fdedf59e640 1 Processor -- start 2023-10-16T14:36:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.924+0000 7fdedf59e640 1 -- start start 2023-10-16T14:36:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.925+0000 7fdedf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdee00c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.925+0000 7fdedf59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00b27c0 0x7fdee00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.925+0000 7fdedf59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee00c8cf0 0x7fdee00c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.925+0000 7fdedf59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00b17b0 con 0x7fdecc003ef0 2023-10-16T14:36:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.925+0000 7fdedf59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00b2330 con 0x7fdee00c8cf0 2023-10-16T14:36:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.925+0000 7fdedf59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00b1a00 con 0x7fdee00b27c0 2023-10-16T14:36:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdee00c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdee00c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51124/0 (socket says 172.21.15.29:51124) 2023-10-16T14:36:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 -- 172.21.15.29:0/1312020443 learned_addr learned my addr 172.21.15.29:0/1312020443 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdeded9d640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee00c8cf0 0x7fdee00c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 -- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00b27c0 msgr2=0x7fdee00c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fded7fff640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00b27c0 0x7fdee00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00b27c0 0x7fdee00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 -- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee00c8cf0 msgr2=0x7fdee00c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee00c8cf0 0x7fdee00c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 -- 172.21.15.29:0/1312020443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00b1ce0 con 0x7fdecc003ef0 2023-10-16T14:36:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.926+0000 7fdede59c640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdee00c6380 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fded000e7f0 tx=0x7fded000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.927+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded000c8e0 con 0x7fdecc003ef0 2023-10-16T14:36:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.927+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded000ca40 con 0x7fdecc003ef0 2023-10-16T14:36:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.927+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded000cdd0 con 0x7fdecc003ef0 2023-10-16T14:36:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.927+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee00b1fc0 con 0x7fdecc003ef0 2023-10-16T14:36:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.927+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee00c96e0 con 0x7fdecc003ef0 2023-10-16T14:36:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.928+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee006b330 con 0x7fdecc003ef0 2023-10-16T14:36:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.929+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fded0013470 con 0x7fdecc003ef0 2023-10-16T14:36:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.930+0000 7fded6ffd640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdec006acf0 0x7fdec006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.930+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7fded00895c0 con 0x7fdecc003ef0 2023-10-16T14:36:09.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.931+0000 7fded7fff640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdec006acf0 0x7fdec006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:09.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.932+0000 7fded7fff640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdec006acf0 0x7fdec006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdec8034000 tx=0x7fdec8007000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:09.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:09.934+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fded00545a0 con 0x7fdecc003ef0 2023-10-16T14:36:10.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.165+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee00ca740 con 0x7fdecc003ef0 2023-10-16T14:36:10.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.166+0000 7fded6ffd640 1 -- 172.21.15.29:0/1312020443 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fded0057bc0 con 0x7fdecc003ef0 2023-10-16T14:36:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:10.169 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:35:27.247546+0000","modified":"2023-10-16T14:35:30.329395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7812":{"gid":7812,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2882189329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2882189329},{"type":"v1","addr":"172.21.15.29:6835","nonce":2882189329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T14:36:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2023-10-16T14:36:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdec006acf0 msgr2=0x7fdec006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdec006acf0 0x7fdec006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdec8034000 tx=0x7fdec8007000 comp rx=0 tx=0).stop 2023-10-16T14:36:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 msgr2=0x7fdee00c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdee00c6380 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fded000e7f0 tx=0x7fded000ecb0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 shutdown_connections 2023-10-16T14:36:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdec006acf0 0x7fdec006d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdee00c8cf0 0x7fdee00c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdee00b27c0 0x7fdee00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 --2- 172.21.15.29:0/1312020443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdecc003ef0 0x7fdee00c6380 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 >> 172.21.15.29:0/1312020443 conn(0x7fdee00ce6a0 msgr2=0x7fdee0070830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 shutdown_connections 2023-10-16T14:36:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.169+0000 7fdedf59e640 1 -- 172.21.15.29:0/1312020443 wait complete. 2023-10-16T14:36:10.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.418+0000 7f4705010640 1 Processor -- start 2023-10-16T14:36:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.418+0000 7f4705010640 1 -- start start 2023-10-16T14:36:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.419+0000 7f4705010640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f47001615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.419+0000 7f4705010640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f470015d3f0 0x7f470015d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.419+0000 7f4705010640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470015e3e0 con 0x7f47000cd0c0 2023-10-16T14:36:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.419+0000 7f4705010640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47000cd4a0 con 0x7f470015d3f0 2023-10-16T14:36:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.419+0000 7f4705010640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470015e100 con 0x7f47000b0b20 2023-10-16T14:36:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46ff59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f470015d3f0 0x7f470015d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46fed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f47000cd0c0 0x7f47000b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54200/0 (socket says 172.21.15.29:54200) 2023-10-16T14:36:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46fed9d640 1 -- 172.21.15.29:0/3664533789 learned_addr learned my addr 172.21.15.29:0/3664533789 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46fe59c640 1 --2- 172.21.15.29:0/3664533789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f47001615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46ff59e640 1 -- 172.21.15.29:0/3664533789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 msgr2=0x7f47001615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46ff59e640 1 --2- 172.21.15.29:0/3664533789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f47001615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46fdd9b640 1 -- 172.21.15.29:0/3664533789 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300764337 0 0) 0x7f470015e3e0 con 0x7f47000cd0c0 2023-10-16T14:36:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46ff59e640 1 -- 172.21.15.29:0/3664533789 >> v1:172.21.15.29:6789/0 conn(0x7f47000cd0c0 legacy=0x7f47000b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.420+0000 7f46ff59e640 1 -- 172.21.15.29:0/3664533789 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700160b10 con 0x7f470015d3f0 2023-10-16T14:36:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.422+0000 7f46ff59e640 1 --2- 172.21.15.29:0/3664533789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f470015d860 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f46f80097a0 tx=0x7f46f802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c37426eef0324f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.422+0000 7f46fdd9b640 1 -- 172.21.15.29:0/3664533789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8031780 con 0x7f470015d3f0 2023-10-16T14:36:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.422+0000 7f46fdd9b640 1 -- 172.21.15.29:0/3664533789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f80318e0 con 0x7f470015d3f0 2023-10-16T14:36:10.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.422+0000 7f46fdd9b640 1 -- 172.21.15.29:0/3664533789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8031c70 con 0x7f470015d3f0 2023-10-16T14:36:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 -- 172.21.15.29:0/3664533789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 msgr2=0x7f470015d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 --2- 172.21.15.29:0/3664533789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f470015d860 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f46f80097a0 tx=0x7f46f802b820 comp rx=0 tx=0).stop 2023-10-16T14:36:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 -- 172.21.15.29:0/3664533789 shutdown_connections 2023-10-16T14:36:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 --2- 172.21.15.29:0/3664533789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f470015d860 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 --2- 172.21.15.29:0/3664533789 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f47001615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 -- 172.21.15.29:0/3664533789 >> 172.21.15.29:0/3664533789 conn(0x7f47000cb410 msgr2=0x7f47000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:10.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 -- 172.21.15.29:0/3664533789 shutdown_connections 2023-10-16T14:36:10.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.423+0000 7f4705010640 1 -- 172.21.15.29:0/3664533789 wait complete. 2023-10-16T14:36:10.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.425+0000 7f4705010640 1 Processor -- start 2023-10-16T14:36:10.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.425+0000 7f4705010640 1 -- start start 2023-10-16T14:36:10.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f4705010640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f4700152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f4705010640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47000cd0c0 0x7f47000ce970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f4705010640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f47000ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f4705010640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470015eaa0 con 0x7f470015d3f0 2023-10-16T14:36:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f4705010640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470015f060 con 0x7f47000cd0c0 2023-10-16T14:36:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f4705010640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47001534b0 con 0x7f47000b0b20 2023-10-16T14:36:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f46ff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f47000ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f46ff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f47000ceeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51148/0 (socket says 172.21.15.29:51148) 2023-10-16T14:36:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f46ff59e640 1 -- 172.21.15.29:0/381329680 learned_addr learned my addr 172.21.15.29:0/381329680 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f46fe59c640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47000cd0c0 0x7f47000ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f46ff59e640 1 -- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 msgr2=0x7f4700152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.426+0000 7f46fed9d640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f4700152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46ff59e640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f4700152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46ff59e640 1 -- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47000cd0c0 msgr2=0x7f47000ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46ff59e640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47000cd0c0 0x7f47000ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46ff59e640 1 -- 172.21.15.29:0/381329680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f8009400 con 0x7f470015d3f0 2023-10-16T14:36:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46fed9d640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f4700152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46ff59e640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f47000ceeb0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f46f80098c0 tx=0x7f46f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8004230 con 0x7f470015d3f0 2023-10-16T14:36:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f80043b0 con 0x7f470015d3f0 2023-10-16T14:36:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47000af9a0 con 0x7f470015d3f0 2023-10-16T14:36:10.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8004520 con 0x7f470015d3f0 2023-10-16T14:36:10.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.427+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47000af6c0 con 0x7f470015d3f0 2023-10-16T14:36:10.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.429+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470015f1a0 con 0x7f470015d3f0 2023-10-16T14:36:10.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.429+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f46f8032860 con 0x7f470015d3f0 2023-10-16T14:36:10.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.430+0000 7f46e77fe640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f46ec06acc0 0x7f46ec06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.430+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f46f80ab790 con 0x7f470015d3f0 2023-10-16T14:36:10.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.431+0000 7f46fed9d640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f46ec06acc0 0x7f46ec06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.432+0000 7f46fed9d640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f46ec06acc0 0x7f46ec06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f46f0007fd0 tx=0x7f46f0009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:10.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.435+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f470015f1a0 con 0x7f470015d3f0 2023-10-16T14:36:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.640+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f470015f1a0 con 0x7f470015d3f0 2023-10-16T14:36:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.641+0000 7f46e77fe640 1 -- 172.21.15.29:0/381329680 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+17773 (secure 0 0 0) 0x7f470015f1a0 con 0x7f470015d3f0 2023-10-16T14:36:10.644 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:10.645 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:09.428800+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/3633682082":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/2408525211":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3344036704":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/2843457870":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1110178125":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/110350219":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1384820266":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3330302692":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1968227499":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3494288089":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/3633682082":"2023-10-17T14:36:08.423291+0000","172.21.15.170:6833/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/2874036200":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3741475474":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/286310502":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1228162669":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1339480091":"2023-10-17T14:35:42.393479+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f46ec06acc0 msgr2=0x7f46ec06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f46ec06acc0 0x7f46ec06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f46f0007fd0 tx=0x7f46f0009040 comp rx=0 tx=0).stop 2023-10-16T14:36:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 msgr2=0x7f47000ceeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f47000ceeb0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f46f80098c0 tx=0x7f46f80046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 shutdown_connections 2023-10-16T14:36:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015d3f0 0x7f47000ceeb0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f46ec06acc0 0x7f46ec06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47000cd0c0 0x7f47000ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 --2- 172.21.15.29:0/381329680 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47000b0b20 0x7f4700152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 >> 172.21.15.29:0/381329680 conn(0x7f47000cb410 msgr2=0x7f470006fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 shutdown_connections 2023-10-16T14:36:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.644+0000 7f4705010640 1 -- 172.21.15.29:0/381329680 wait complete. 2023-10-16T14:36:10.668 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:36:10.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.895+0000 7f39ecb4e640 1 Processor -- start 2023-10-16T14:36:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.895+0000 7f39ecb4e640 1 -- start start 2023-10-16T14:36:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.897+0000 7f39ecb4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39e815dde0 0x7f39e80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.897+0000 7f39ecb4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e8070f20 0x7f39e80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.897+0000 7f39ecb4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e8160a90 con 0x7f39e815c9f0 2023-10-16T14:36:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.897+0000 7f39ecb4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80cdec0 con 0x7f39e815dde0 2023-10-16T14:36:10.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.897+0000 7f39ecb4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80cd900 con 0x7f39e8070f20 2023-10-16T14:36:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39e815dde0 0x7f39e80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39e815dde0 0x7f39e80709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51160/0 (socket says 172.21.15.29:51160) 2023-10-16T14:36:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 -- 172.21.15.29:0/2586164483 learned_addr learned my addr 172.21.15.29:0/2586164483 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39e7fff640 1 --2- 172.21.15.29:0/2586164483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e8070f20 0x7f39e80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 -- 172.21.15.29:0/2586164483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e8070f20 msgr2=0x7f39e80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 --2- 172.21.15.29:0/2586164483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e8070f20 0x7f39e80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 -- 172.21.15.29:0/2586164483 >> v1:172.21.15.170:6789/0 conn(0x7f39e815c9f0 legacy=0x7f39e80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.898+0000 7f39dffff640 1 -- 172.21.15.29:0/2586164483 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80cdbe0 con 0x7f39e815dde0 2023-10-16T14:36:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.900+0000 7f39dffff640 1 --2- 172.21.15.29:0/2586164483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e80709e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009870 tx=0x7f39d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcc62e7bfb3cad4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.900+0000 7f39e6ffd640 1 -- 172.21.15.29:0/2586164483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0030840 con 0x7f39e815dde0 2023-10-16T14:36:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.900+0000 7f39e6ffd640 1 -- 172.21.15.29:0/2586164483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d00309a0 con 0x7f39e815dde0 2023-10-16T14:36:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.900+0000 7f39e6ffd640 1 -- 172.21.15.29:0/2586164483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0030d30 con 0x7f39e815dde0 2023-10-16T14:36:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2586164483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 msgr2=0x7f39e80709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2586164483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e80709e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009870 tx=0x7f39d002b980 comp rx=0 tx=0).stop 2023-10-16T14:36:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2586164483 shutdown_connections 2023-10-16T14:36:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2586164483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e8070f20 0x7f39e80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2586164483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e80709e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2586164483 >> 172.21.15.29:0/2586164483 conn(0x7f39e80ce6a0 msgr2=0x7f39e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2586164483 shutdown_connections 2023-10-16T14:36:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.901+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2586164483 wait complete. 2023-10-16T14:36:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.903+0000 7f39ecb4e640 1 Processor -- start 2023-10-16T14:36:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.903+0000 7f39ecb4e640 1 -- start start 2023-10-16T14:36:10.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39ecb4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39e8070f20 0x7f39e80c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39ecb4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e815c9f0 0x7f39e81f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39ecb4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e806a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39ecb4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e806b460 con 0x7f39e815dde0 2023-10-16T14:36:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39ecb4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e806b180 con 0x7f39e8070f20 2023-10-16T14:36:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39ecb4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80c2300 con 0x7f39e815c9f0 2023-10-16T14:36:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e806a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e806a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51166/0 (socket says 172.21.15.29:51166) 2023-10-16T14:36:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.904+0000 7f39e7fff640 1 -- 172.21.15.29:0/2870930907 learned_addr learned my addr 172.21.15.29:0/2870930907 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e77fe640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39e8070f20 0x7f39e80c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39dffff640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e815c9f0 0x7f39e81f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e7fff640 1 -- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e815c9f0 msgr2=0x7f39e81f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e7fff640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e815c9f0 0x7f39e81f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e7fff640 1 -- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39e8070f20 msgr2=0x7f39e80c2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:10.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e7fff640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39e8070f20 0x7f39e80c2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:10.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e7fff640 1 -- 172.21.15.29:0/2870930907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00094d0 con 0x7f39e815dde0 2023-10-16T14:36:10.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.905+0000 7f39e7fff640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e806a930 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f39e0007e40 tx=0x7f39e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:10.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.906+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0014620 con 0x7f39e815dde0 2023-10-16T14:36:10.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.906+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e000ee90 con 0x7f39e815dde0 2023-10-16T14:36:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.906+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e00149e0 con 0x7f39e815dde0 2023-10-16T14:36:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.906+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e80c25e0 con 0x7f39e815dde0 2023-10-16T14:36:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.906+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e806ae70 con 0x7f39e815dde0 2023-10-16T14:36:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.907+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e80c2b00 con 0x7f39e815dde0 2023-10-16T14:36:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.908+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f39e0014b40 con 0x7f39e815dde0 2023-10-16T14:36:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.909+0000 7f39e4ff9640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f39c806acc0 0x7f39c806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.910+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f39e008a060 con 0x7f39e815dde0 2023-10-16T14:36:10.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.910+0000 7f39e77fe640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f39c806acc0 0x7f39c806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:10.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.911+0000 7f39e77fe640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f39c806acc0 0x7f39c806d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f39d8000f60 tx=0x7f39d8009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:10.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:10.914+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39e00bcca0 con 0x7f39e815dde0 2023-10-16T14:36:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.144+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39e8160d70 con 0x7f39e815dde0 2023-10-16T14:36:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.145+0000 7f39e4ff9640 1 -- 172.21.15.29:0/2870930907 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f39e80c2b00 con 0x7f39e815dde0 2023-10-16T14:36:11.148 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:11.148 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:35:27.247546+0000","modified":"2023-10-16T14:35:30.329395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7812":{"gid":7812,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2882189329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2882189329},{"type":"v1","addr":"172.21.15.29:6835","nonce":2882189329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T14:36:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2023-10-16T14:36:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f39c806acc0 msgr2=0x7f39c806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f39c806acc0 0x7f39c806d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f39d8000f60 tx=0x7f39d8009040 comp rx=0 tx=0).stop 2023-10-16T14:36:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 msgr2=0x7f39e806a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e806a930 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f39e0007e40 tx=0x7f39e000e4b0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 shutdown_connections 2023-10-16T14:36:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39e815dde0 0x7f39e806a930 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f39c806acc0 0x7f39c806d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39e815c9f0 0x7f39e81f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 --2- 172.21.15.29:0/2870930907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39e8070f20 0x7f39e80c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 >> 172.21.15.29:0/2870930907 conn(0x7f39e80ce6a0 msgr2=0x7f39e80af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 shutdown_connections 2023-10-16T14:36:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.148+0000 7f39ecb4e640 1 -- 172.21.15.29:0/2870930907 wait complete. 2023-10-16T14:36:11.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.394+0000 7f26c2275640 1 Processor -- start 2023-10-16T14:36:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.394+0000 7f26c2275640 1 -- start start 2023-10-16T14:36:11.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.396+0000 7f26c2275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26bc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.396+0000 7f26c2275640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26bc15f4a0 con 0x7f26bc0ad970 2023-10-16T14:36:11.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.396+0000 7f26c2275640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26bc06c960 con 0x7f26bc0cda80 2023-10-16T14:36:11.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.396+0000 7f26c2275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc15b9d0 con 0x7f26bc06c040 2023-10-16T14:36:11.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.396+0000 7f26c0a72640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f26bc0ad970 0x7f26bc15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54204/0 (socket says 172.21.15.29:54204) 2023-10-16T14:36:11.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.396+0000 7f26c0a72640 1 -- 172.21.15.29:0/3758101760 learned_addr learned my addr 172.21.15.29:0/3758101760 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:11.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26c1273640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26bc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479920971 0 0) 0x7f26bc15f4a0 con 0x7f26bc0ad970 2023-10-16T14:36:11.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a8003520 con 0x7f26bc0ad970 2023-10-16T14:36:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126957971 0 0) 0x7f26a8003520 con 0x7f26bc0ad970 2023-10-16T14:36:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 msgr2=0x7f26bc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26b3fff640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26bc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.397+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 >> v1:172.21.15.170:6789/0 conn(0x7f26bc0cda80 legacy=0x7f26bc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc15bcb0 con 0x7f26bc0ad970 2023-10-16T14:36:11.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f26ac001670 con 0x7f26bc0ad970 2023-10-16T14:36:11.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 >> v1:172.21.15.29:6789/0 conn(0x7f26bc0ad970 legacy=0x7f26bc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26a80039a0 0x7f26a8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a8004d00 0x7f26a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc15b9d0 con 0x7f26a80039a0 2023-10-16T14:36:11.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a800ba30 con 0x7f26a8004d00 2023-10-16T14:36:11.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a800bba0 con 0x7f26bc06c040 2023-10-16T14:36:11.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1273640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26a80039a0 0x7f26a8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c0a72640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1a74640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a8004d00 0x7f26a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1273640 1 -- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 msgr2=0x7f26a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:11.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1273640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26a80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1273640 1 -- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a8004d00 msgr2=0x7f26a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:11.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1273640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a8004d00 0x7f26a8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.398+0000 7f26c1273640 1 -- 172.21.15.29:0/3758101760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a8003520 con 0x7f26a80039a0 2023-10-16T14:36:11.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.399+0000 7f26c1a74640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a8004d00 0x7f26a8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:11.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.399+0000 7f26c0a72640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26bc06c040 0x7f26a80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26c1273640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26a80039a0 0x7f26a8003d80 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009ac0 tx=0x7f26a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e965b581034b3bd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b802c6b0 con 0x7f26a80039a0 2023-10-16T14:36:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b802c810 con 0x7f26a80039a0 2023-10-16T14:36:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26b3fff640 1 -- 172.21.15.29:0/3758101760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b802cba0 con 0x7f26a80039a0 2023-10-16T14:36:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26c2275640 1 -- 172.21.15.29:0/3758101760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26a80039a0 msgr2=0x7f26a8003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26c2275640 1 --2- 172.21.15.29:0/3758101760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26a80039a0 0x7f26a8003d80 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009ac0 tx=0x7f26a80057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.400+0000 7f26c1a74640 1 -- 172.21.15.29:0/3758101760 reap_dead start 2023-10-16T14:36:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.401+0000 7f26c2275640 1 -- 172.21.15.29:0/3758101760 shutdown_connections 2023-10-16T14:36:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.401+0000 7f26c2275640 1 -- 172.21.15.29:0/3758101760 >> 172.21.15.29:0/3758101760 conn(0x7f26bc0ce6a0 msgr2=0x7f26bc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.401+0000 7f26c2275640 1 -- 172.21.15.29:0/3758101760 shutdown_connections 2023-10-16T14:36:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.401+0000 7f26c2275640 1 -- 172.21.15.29:0/3758101760 wait complete. 2023-10-16T14:36:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.402+0000 7f26c2275640 1 Processor -- start 2023-10-16T14:36:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.403+0000 7f26c2275640 1 -- start start 2023-10-16T14:36:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c2275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26a80039a0 0x7f26bc0c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c2275640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26bc0c66a0 0x7f26bc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c2275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 0x7f26bc0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c2275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc0ae0c0 con 0x7f26bc0acdd0 2023-10-16T14:36:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c2275640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc0ad280 con 0x7f26bc0c66a0 2023-10-16T14:36:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c2275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc0c85c0 con 0x7f26a80039a0 2023-10-16T14:36:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 0x7f26bc0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 0x7f26bc0c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51188/0 (socket says 172.21.15.29:51188) 2023-10-16T14:36:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 -- 172.21.15.29:0/3993547208 learned_addr learned my addr 172.21.15.29:0/3993547208 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1273640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26a80039a0 0x7f26bc0c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c0a72640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26bc0c66a0 0x7f26bc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 -- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26a80039a0 msgr2=0x7f26bc0c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26a80039a0 0x7f26bc0c6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 -- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26bc0c66a0 msgr2=0x7f26bc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26bc0c66a0 0x7f26bc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1a74640 1 -- 172.21.15.29:0/3993547208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc0c8000 con 0x7f26bc0acdd0 2023-10-16T14:36:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c0a72640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26bc0c66a0 0x7f26bc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.404+0000 7f26c1273640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26a80039a0 0x7f26bc0c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.405+0000 7f26c1a74640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 0x7f26bc0c94b0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f26bc06d2e0 tx=0x7f26b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.405+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4003b70 con 0x7f26bc0acdd0 2023-10-16T14:36:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.405+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b4003cf0 con 0x7f26bc0acdd0 2023-10-16T14:36:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.405+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b40052b0 con 0x7f26bc0acdd0 2023-10-16T14:36:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.405+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26bc0c82e0 con 0x7f26bc0acdd0 2023-10-16T14:36:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.405+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26bc0b3410 con 0x7f26bc0acdd0 2023-10-16T14:36:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.406+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26bc06b330 con 0x7f26bc0acdd0 2023-10-16T14:36:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.407+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 66) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f26b4005410 con 0x7f26bc0acdd0 2023-10-16T14:36:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.408+0000 7f26b1ffb640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f26a006af30 0x7f26a006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.408+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f26b4089130 con 0x7f26bc0acdd0 2023-10-16T14:36:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.409+0000 7f26c1273640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f26a006af30 0x7f26a006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.410+0000 7f26c1273640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f26a006af30 0x7f26a006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26b8036000 tx=0x7f26b8030000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.412+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f26b4053d30 con 0x7f26bc0acdd0 2023-10-16T14:36:11.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.645+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26bc04e9b0 con 0x7f26bc0acdd0 2023-10-16T14:36:11.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.646+0000 7f26b1ffb640 1 -- 172.21.15.29:0/3993547208 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f26b4018090 con 0x7f26bc0acdd0 2023-10-16T14:36:11.649 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:11.649 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:35:27.247546+0000","modified":"2023-10-16T14:35:30.329395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7812":{"gid":7812,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2882189329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2882189329},{"type":"v1","addr":"172.21.15.29:6835","nonce":2882189329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T14:36:11.649 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2023-10-16T14:36:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.648+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f26a006af30 msgr2=0x7f26a006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.648+0000 7f26c2275640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f26a006af30 0x7f26a006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26b8036000 tx=0x7f26b8030000 comp rx=0 tx=0).stop 2023-10-16T14:36:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.648+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 msgr2=0x7f26bc0c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.648+0000 7f26c2275640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 0x7f26bc0c94b0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f26bc06d2e0 tx=0x7f26b400dc00 comp rx=0 tx=0).stop 2023-10-16T14:36:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 shutdown_connections 2023-10-16T14:36:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26bc0acdd0 0x7f26bc0c94b0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f26a006af30 0x7f26a006d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26bc0c66a0 0x7f26bc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 --2- 172.21.15.29:0/3993547208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26a80039a0 0x7f26bc0c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 >> 172.21.15.29:0/3993547208 conn(0x7f26bc0ce6a0 msgr2=0x7f26bc069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:11.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 shutdown_connections 2023-10-16T14:36:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.649+0000 7f26c2275640 1 -- 172.21.15.29:0/3993547208 wait complete. 2023-10-16T14:36:11.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:11.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.892+0000 7f593e6a3640 1 Processor -- start 2023-10-16T14:36:11.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.892+0000 7f593e6a3640 1 -- start start 2023-10-16T14:36:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.893+0000 7f593e6a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ad970 0x7f59380cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.893+0000 7f593e6a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59380ccfb0 0x7f59380cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.893+0000 7f593e6a3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593815b9d0 con 0x7f593806c040 2023-10-16T14:36:11.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.893+0000 7f593e6a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593815bcb0 con 0x7f59380ccfb0 2023-10-16T14:36:11.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.893+0000 7f593e6a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593815bf90 con 0x7f59380ad970 2023-10-16T14:36:11.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59380ccfb0 0x7f59380cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59380ccfb0 0x7f59380cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51204/0 (socket says 172.21.15.29:51204) 2023-10-16T14:36:11.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 -- 172.21.15.29:0/318419030 learned_addr learned my addr 172.21.15.29:0/318419030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:11.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593cea0640 1 --2- 172.21.15.29:0/318419030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ad970 0x7f59380cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 -- 172.21.15.29:0/318419030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ad970 msgr2=0x7f59380cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:11.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 --2- 172.21.15.29:0/318419030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ad970 0x7f59380cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 -- 172.21.15.29:0/318419030 >> v1:172.21.15.170:6789/0 conn(0x7f593806c040 legacy=0x7f593806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.894+0000 7f593dea2640 1 -- 172.21.15.29:0/318419030 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59380cc270 con 0x7f59380ccfb0 2023-10-16T14:36:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.895+0000 7f593dea2640 1 --2- 172.21.15.29:0/318419030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59380ccfb0 0x7f59380cd440 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5930009e80 tx=0x7f593002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109a9c326a9f50c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f592ffff640 1 -- 172.21.15.29:0/318419030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593002ca50 con 0x7f59380ccfb0 2023-10-16T14:36:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f592ffff640 1 -- 172.21.15.29:0/318419030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593002cbb0 con 0x7f59380ccfb0 2023-10-16T14:36:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f592ffff640 1 -- 172.21.15.29:0/318419030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300343f0 con 0x7f59380ccfb0 2023-10-16T14:36:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 -- 172.21.15.29:0/318419030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59380ccfb0 msgr2=0x7f59380cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 --2- 172.21.15.29:0/318419030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59380ccfb0 0x7f59380cd440 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5930009e80 tx=0x7f593002beb0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 -- 172.21.15.29:0/318419030 shutdown_connections 2023-10-16T14:36:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 --2- 172.21.15.29:0/318419030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59380ccfb0 0x7f59380cd440 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 --2- 172.21.15.29:0/318419030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ad970 0x7f59380cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 -- 172.21.15.29:0/318419030 >> 172.21.15.29:0/318419030 conn(0x7f59380ce6a0 msgr2=0x7f59380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 -- 172.21.15.29:0/318419030 shutdown_connections 2023-10-16T14:36:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.896+0000 7f593e6a3640 1 -- 172.21.15.29:0/318419030 wait complete. 2023-10-16T14:36:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.898+0000 7f593e6a3640 1 Processor -- start 2023-10-16T14:36:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.898+0000 7f593e6a3640 1 -- start start 2023-10-16T14:36:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.899+0000 7f593e6a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 0x7f59380c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.899+0000 7f593e6a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59380ad970 0x7f59381f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.899+0000 7f593e6a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ccfb0 0x7f59380ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.899+0000 7f593e6a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381f3230 con 0x7f593806c040 2023-10-16T14:36:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.899+0000 7f593e6a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59380c9b10 con 0x7f59380ad970 2023-10-16T14:36:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.899+0000 7f593e6a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59380c6960 con 0x7f59380ccfb0 2023-10-16T14:36:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 0x7f59380c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 0x7f59380c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51212/0 (socket says 172.21.15.29:51212) 2023-10-16T14:36:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 -- 172.21.15.29:0/2052418966 learned_addr learned my addr 172.21.15.29:0/2052418966 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593dea2640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ccfb0 0x7f59380ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593cea0640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59380ad970 0x7f59381f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 -- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ccfb0 msgr2=0x7f59380ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ccfb0 0x7f59380ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 -- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59380ad970 msgr2=0x7f59381f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59380ad970 0x7f59381f2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593d6a1640 1 -- 172.21.15.29:0/2052418966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930009b30 con 0x7f593806c040 2023-10-16T14:36:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593cea0640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59380ad970 0x7f59381f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.900+0000 7f593dea2640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ccfb0 0x7f59380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.901+0000 7f593d6a1640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 0x7f59380c9410 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f593400c460 tx=0x7f593400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.901+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934011790 con 0x7f593806c040 2023-10-16T14:36:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.901+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934011930 con 0x7f593806c040 2023-10-16T14:36:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.901+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934011ce0 con 0x7f593806c040 2023-10-16T14:36:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.901+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59380c6c40 con 0x7f593806c040 2023-10-16T14:36:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.901+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59380c7200 con 0x7f593806c040 2023-10-16T14:36:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.902+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593806b330 con 0x7f593806c040 2023-10-16T14:36:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.903+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 66) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f593401c460 con 0x7f593806c040 2023-10-16T14:36:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.904+0000 7f592dffb640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f591c06af00 0x7f591c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.904+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f59340899e0 con 0x7f593806c040 2023-10-16T14:36:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.905+0000 7f593cea0640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f591c06af00 0x7f591c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.906+0000 7f593cea0640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f591c06af00 0x7f591c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f593815ded0 tx=0x7f5928008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:11.908+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5934054660 con 0x7f593806c040 2023-10-16T14:36:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.101+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f593804e9b0 con 0x7f593806c040 2023-10-16T14:36:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.103+0000 7f592dffb640 1 -- 172.21.15.29:0/2052418966 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+17773 (secure 0 0 0) 0x7f5934057c80 con 0x7f593806c040 2023-10-16T14:36:12.107 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:12.107 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:09.428800+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:35:25.164943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T14:35:26.261110+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/3633682082":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/2408525211":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3344036704":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/2843457870":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1110178125":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/110350219":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1384820266":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3330302692":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1968227499":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3494288089":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/3633682082":"2023-10-17T14:36:08.423291+0000","172.21.15.170:6833/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/2874036200":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3741475474":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/286310502":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1228162669":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1339480091":"2023-10-17T14:35:42.393479+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f591c06af00 msgr2=0x7f591c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f591c06af00 0x7f591c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f593815ded0 tx=0x7f5928008040 comp rx=0 tx=0).stop 2023-10-16T14:36:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 msgr2=0x7f59380c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 0x7f59380c9410 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f593400c460 tx=0x7f593400c920 comp rx=0 tx=0).stop 2023-10-16T14:36:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 shutdown_connections 2023-10-16T14:36:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f591c06af00 0x7f591c06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59380ccfb0 0x7f59380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59380ad970 0x7f59381f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 --2- 172.21.15.29:0/2052418966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593806c040 0x7f59380c9410 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 >> 172.21.15.29:0/2052418966 conn(0x7f59380ce6a0 msgr2=0x7f59380ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 shutdown_connections 2023-10-16T14:36:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.106+0000 7f593e6a3640 1 -- 172.21.15.29:0/2052418966 wait complete. 2023-10-16T14:36:12.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:36:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.353+0000 7f8ef8d8e640 1 Processor -- start 2023-10-16T14:36:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.353+0000 7f8ef8d8e640 1 -- start start 2023-10-16T14:36:12.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef8d8e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8ef406c040 0x7f8ef406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef8d8e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40cda80 0x7f8ef415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef8d8e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef406c960 con 0x7f8ef40ad970 2023-10-16T14:36:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef8d8e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef415b9d0 con 0x7f8ef406c040 2023-10-16T14:36:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef8d8e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef415bcb0 con 0x7f8ef40cda80 2023-10-16T14:36:12.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8ef406c040 0x7f8ef406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8ef40ad970 0x7f8ef415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54210/0 (socket says 172.21.15.29:54210) 2023-10-16T14:36:12.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.355+0000 7f8ef2ffd640 1 -- 172.21.15.29:0/2576502622 learned_addr learned my addr 172.21.15.29:0/2576502622 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:12.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.356+0000 7f8ef3fff640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40cda80 0x7f8ef415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.356+0000 7f8ef37fe640 1 -- 172.21.15.29:0/2576502622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40cda80 msgr2=0x7f8ef415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:12.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.356+0000 7f8ef37fe640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40cda80 0x7f8ef415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.356+0000 7f8ef37fe640 1 -- 172.21.15.29:0/2576502622 >> v1:172.21.15.29:6789/0 conn(0x7f8ef40ad970 legacy=0x7f8ef415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.356+0000 7f8ef37fe640 1 -- 172.21.15.29:0/2576502622 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef415bf90 con 0x7f8ef406c040 2023-10-16T14:36:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.356+0000 7f8ef3fff640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40cda80 0x7f8ef415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.357+0000 7f8ef37fe640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef406c420 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097a0 tx=0x7f8ee402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f4e757372f1984 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:12.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.357+0000 7f8ef27fc640 1 -- 172.21.15.29:0/2576502622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee402d040 con 0x7f8ef406c040 2023-10-16T14:36:12.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.357+0000 7f8ef27fc640 1 -- 172.21.15.29:0/2576502622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee4031810 con 0x7f8ef406c040 2023-10-16T14:36:12.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.357+0000 7f8ef27fc640 1 -- 172.21.15.29:0/2576502622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4031ba0 con 0x7f8ef406c040 2023-10-16T14:36:12.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/2576502622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 msgr2=0x7f8ef406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef406c420 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097a0 tx=0x7f8ee402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/2576502622 shutdown_connections 2023-10-16T14:36:12.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40cda80 0x7f8ef415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/2576502622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef406c420 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/2576502622 >> 172.21.15.29:0/2576502622 conn(0x7f8ef40ce6a0 msgr2=0x7f8ef40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:12.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/2576502622 shutdown_connections 2023-10-16T14:36:12.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.358+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/2576502622 wait complete. 2023-10-16T14:36:12.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.360+0000 7f8ef8d8e640 1 Processor -- start 2023-10-16T14:36:12.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.360+0000 7f8ef8d8e640 1 -- start start 2023-10-16T14:36:12.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef8d8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef40b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef8d8e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40ad970 0x7f8ef40afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef8d8e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ef40cda80 0x7f8ef40c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef8d8e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40b3af0 con 0x7f8ef406c040 2023-10-16T14:36:12.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef8d8e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40b0cb0 con 0x7f8ef40cda80 2023-10-16T14:36:12.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef8d8e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40b0430 con 0x7f8ef40ad970 2023-10-16T14:36:12.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef40b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef40b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51218/0 (socket says 172.21.15.29:51218) 2023-10-16T14:36:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.361+0000 7f8ef37fe640 1 -- 172.21.15.29:0/1118837618 learned_addr learned my addr 172.21.15.29:0/1118837618 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef37fe640 1 -- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40ad970 msgr2=0x7f8ef40afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef37fe640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40ad970 0x7f8ef40afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef37fe640 1 -- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ef40cda80 msgr2=0x7f8ef40c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef37fe640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ef40cda80 0x7f8ef40c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef37fe640 1 -- 172.21.15.29:0/1118837618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee4009400 con 0x7f8ef406c040 2023-10-16T14:36:12.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef37fe640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef40b35b0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098c0 tx=0x7f8ee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:12.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee403ad60 con 0x7f8ef406c040 2023-10-16T14:36:12.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee40423f0 con 0x7f8ef406c040 2023-10-16T14:36:12.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40427a0 con 0x7f8ef406c040 2023-10-16T14:36:12.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.362+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef40b06f0 con 0x7f8ef406c040 2023-10-16T14:36:12.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.363+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef41f59b0 con 0x7f8ef406c040 2023-10-16T14:36:12.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.364+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef406b330 con 0x7f8ef406c040 2023-10-16T14:36:12.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.365+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 66) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8ee40393b0 con 0x7f8ef406c040 2023-10-16T14:36:12.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.366+0000 7f8edbfff640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8ee006af00 0x7f8ee006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.366+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f8ee40ab880 con 0x7f8ef406c040 2023-10-16T14:36:12.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.367+0000 7f8ef2ffd640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8ee006af00 0x7f8ee006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.368+0000 7f8ef2ffd640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8ee006af00 0x7f8ee006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4070900 tx=0x7f8ee8007a40 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:12.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.371+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ee4076480 con 0x7f8ef406c040 2023-10-16T14:36:12.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.467+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8ee4076850 con 0x7f8ef406c040 2023-10-16T14:36:12.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.608+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8ef404e9b0 con 0x7f8ef406c040 2023-10-16T14:36:12.612 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:36:12.609+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:36:12.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.613+0000 7f8edbfff640 1 -- 172.21.15.29:0/1118837618 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8ee407dea0 con 0x7f8ef406c040 2023-10-16T14:36:12.616 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:36:12.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8ee006af00 msgr2=0x7f8ee006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8ee006af00 0x7f8ee006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4070900 tx=0x7f8ee8007a40 comp rx=0 tx=0).stop 2023-10-16T14:36:12.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 msgr2=0x7f8ef40b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef40b35b0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098c0 tx=0x7f8ee40046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 shutdown_connections 2023-10-16T14:36:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8ee006af00 0x7f8ee006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ef40cda80 0x7f8ef40c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ef40ad970 0x7f8ef40afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 --2- 172.21.15.29:0/1118837618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ef406c040 0x7f8ef40b35b0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 >> 172.21.15.29:0/1118837618 conn(0x7f8ef40ce6a0 msgr2=0x7f8ef40c8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:12.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 shutdown_connections 2023-10-16T14:36:12.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.616+0000 7f8ef8d8e640 1 -- 172.21.15.29:0/1118837618 wait complete. 2023-10-16T14:36:12.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:36:12.709 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:36:12.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.840+0000 7fcb9bfff640 1 Processor -- start 2023-10-16T14:36:12.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.840+0000 7fcb9bfff640 1 -- start start 2023-10-16T14:36:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.841+0000 7fcb9bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ad970 0x7fcb9c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.841+0000 7fcb9bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcb9c0ccfb0 0x7fcb9c0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.841+0000 7fcb9bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c15b9d0 con 0x7fcb9c06c040 2023-10-16T14:36:12.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.841+0000 7fcb9bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c15bcb0 con 0x7fcb9c0ccfb0 2023-10-16T14:36:12.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.841+0000 7fcb9bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c15bf90 con 0x7fcb9c0ad970 2023-10-16T14:36:12.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcb9c06c040 0x7fcb9c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54222/0 (socket says 172.21.15.29:54222) 2023-10-16T14:36:12.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9affd640 1 -- 172.21.15.29:0/3493422418 learned_addr learned my addr 172.21.15.29:0/3493422418 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:12.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9b7fe640 1 --2- 172.21.15.29:0/3493422418 >> v2:172.21.15.29:3300/0 conn(0x7fcb9c0ccfb0 0x7fcb9c0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb927fc640 1 --2- 172.21.15.29:0/3493422418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ad970 0x7fcb9c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9a7fc640 1 -- 172.21.15.29:0/3493422418 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873151665 0 0) 0x7fcb9c15b9d0 con 0x7fcb9c06c040 2023-10-16T14:36:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9b7fe640 1 -- 172.21.15.29:0/3493422418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ad970 msgr2=0x7fcb9c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9b7fe640 1 --2- 172.21.15.29:0/3493422418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ad970 0x7fcb9c0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9b7fe640 1 -- 172.21.15.29:0/3493422418 >> v1:172.21.15.29:6789/0 conn(0x7fcb9c06c040 legacy=0x7fcb9c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.842+0000 7fcb9b7fe640 1 -- 172.21.15.29:0/3493422418 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c0cc270 con 0x7fcb9c0ccfb0 2023-10-16T14:36:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.843+0000 7fcb9b7fe640 1 --2- 172.21.15.29:0/3493422418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c0ccfb0 0x7fcb9c0cd440 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcb940097d0 tx=0x7fcb9402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96c4c4384576166 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9a7fc640 1 -- 172.21.15.29:0/3493422418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94031800 con 0x7fcb9c0ccfb0 2023-10-16T14:36:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9a7fc640 1 -- 172.21.15.29:0/3493422418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb94031960 con 0x7fcb9c0ccfb0 2023-10-16T14:36:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9a7fc640 1 -- 172.21.15.29:0/3493422418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94031cf0 con 0x7fcb9c0ccfb0 2023-10-16T14:36:12.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 -- 172.21.15.29:0/3493422418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c0ccfb0 msgr2=0x7fcb9c0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/3493422418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c0ccfb0 0x7fcb9c0cd440 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcb940097d0 tx=0x7fcb9402b8a0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 -- 172.21.15.29:0/3493422418 shutdown_connections 2023-10-16T14:36:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/3493422418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c0ccfb0 0x7fcb9c0cd440 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/3493422418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ad970 0x7fcb9c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 -- 172.21.15.29:0/3493422418 >> 172.21.15.29:0/3493422418 conn(0x7fcb9c0ce6a0 msgr2=0x7fcb9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 -- 172.21.15.29:0/3493422418 shutdown_connections 2023-10-16T14:36:12.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.844+0000 7fcb9bfff640 1 -- 172.21.15.29:0/3493422418 wait complete. 2023-10-16T14:36:12.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.846+0000 7fcb9bfff640 1 Processor -- start 2023-10-16T14:36:12.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.846+0000 7fcb9bfff640 1 -- start start 2023-10-16T14:36:12.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.847+0000 7fcb9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 0x7fcb9c0c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.847+0000 7fcb9bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb9c0ad970 0x7fcb9c1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.847+0000 7fcb9bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ccfb0 0x7fcb9c1efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.847+0000 7fcb9bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c151d10 con 0x7fcb9c06c040 2023-10-16T14:36:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.847+0000 7fcb9bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c1f0010 con 0x7fcb9c0ad970 2023-10-16T14:36:12.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.847+0000 7fcb9bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c151f90 con 0x7fcb9c0ccfb0 2023-10-16T14:36:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 0x7fcb9c0c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 0x7fcb9c0c7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51252/0 (socket says 172.21.15.29:51252) 2023-10-16T14:36:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 -- 172.21.15.29:0/1489968740 learned_addr learned my addr 172.21.15.29:0/1489968740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9b7fe640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ccfb0 0x7fcb9c1efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb927fc640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb9c0ad970 0x7fcb9c1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 -- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ccfb0 msgr2=0x7fcb9c1efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ccfb0 0x7fcb9c1efad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 -- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb9c0ad970 msgr2=0x7fcb9c1517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb9c0ad970 0x7fcb9c1517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 -- 172.21.15.29:0/1489968740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb94009400 con 0x7fcb9c06c040 2023-10-16T14:36:12.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb927fc640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb9c0ad970 0x7fcb9c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.848+0000 7fcb9affd640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 0x7fcb9c0c7ba0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00c4d0 tx=0x7fcb8c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.849+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0074e0 con 0x7fcb9c06c040 2023-10-16T14:36:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.849+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c007660 con 0x7fcb9c06c040 2023-10-16T14:36:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.849+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c004a20 con 0x7fcb9c06c040 2023-10-16T14:36:12.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.849+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c152270 con 0x7fcb9c06c040 2023-10-16T14:36:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.849+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c152830 con 0x7fcb9c06c040 2023-10-16T14:36:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.850+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c06b330 con 0x7fcb9c06c040 2023-10-16T14:36:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.851+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcb8c004b80 con 0x7fcb9c06c040 2023-10-16T14:36:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.852+0000 7fcb93fff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fcb7c06af90 0x7fcb7c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.852+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6597+0+0 (secure 0 0 0) 0x7fcb8c088d10 con 0x7fcb9c06c040 2023-10-16T14:36:12.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.853+0000 7fcb927fc640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fcb7c06af90 0x7fcb7c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.854+0000 7fcb927fc640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fcb7c06af90 0x7fcb7c06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c153cf0 tx=0x7fcb840074a0 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:12.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:12.856+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb8c08e020 con 0x7fcb9c06c040 2023-10-16T14:36:13.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.086+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcb9c04e9b0 con 0x7fcb9c06c040 2023-10-16T14:36:13.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.616+0000 7fcb93fff640 1 -- 172.21.15.29:0/1489968740 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcb8c0180a0 con 0x7fcb9c06c040 2023-10-16T14:36:13.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fcb7c06af90 msgr2=0x7fcb7c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fcb7c06af90 0x7fcb7c06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c153cf0 tx=0x7fcb840074a0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 msgr2=0x7fcb9c0c7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 0x7fcb9c0c7ba0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00c4d0 tx=0x7fcb8c00c990 comp rx=0 tx=0).stop 2023-10-16T14:36:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 shutdown_connections 2023-10-16T14:36:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fcb7c06af90 0x7fcb7c06d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb9c0ccfb0 0x7fcb9c1efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb9c0ad970 0x7fcb9c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 --2- 172.21.15.29:0/1489968740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb9c06c040 0x7fcb9c0c7ba0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.620+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 >> 172.21.15.29:0/1489968740 conn(0x7fcb9c0ce6a0 msgr2=0x7fcb9c15cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.621+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 shutdown_connections 2023-10-16T14:36:13.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.621+0000 7fcb9bfff640 1 -- 172.21.15.29:0/1489968740 wait complete. 2023-10-16T14:36:13.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:36:13.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.870+0000 7f421ffff640 1 Processor -- start 2023-10-16T14:36:13.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.871+0000 7f421ffff640 1 -- start start 2023-10-16T14:36:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f422015c100 0x7f42200ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42200cd4a0 con 0x7f4220153740 2023-10-16T14:36:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422015ac60 con 0x7f42200ccb00 2023-10-16T14:36:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421ffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4220070030 con 0x7f422015c100 2023-10-16T14:36:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4220153740 0x7f422015a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54228/0 (socket says 172.21.15.29:54228) 2023-10-16T14:36:13.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421effd640 1 -- 172.21.15.29:0/247376752 learned_addr learned my addr 172.21.15.29:0/247376752 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:13.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.872+0000 7f421f7fe640 1 --2- 172.21.15.29:0/247376752 >> v2:172.21.15.29:3300/0 conn(0x7f422015c100 0x7f42200ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.873+0000 7f421dffb640 1 -- 172.21.15.29:0/247376752 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602967507 0 0) 0x7f42200cd4a0 con 0x7f4220153740 2023-10-16T14:36:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.873+0000 7f421dffb640 1 -- 172.21.15.29:0/247376752 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208003520 con 0x7f4220153740 2023-10-16T14:36:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.873+0000 7f421f7fe640 1 -- 172.21.15.29:0/247376752 >> v1:172.21.15.170:6789/0 conn(0x7f42200ccb00 legacy=0x7f42200ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.873+0000 7f421f7fe640 1 -- 172.21.15.29:0/247376752 >> v1:172.21.15.29:6789/0 conn(0x7f4220153740 legacy=0x7f422015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.873+0000 7f421f7fe640 1 -- 172.21.15.29:0/247376752 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4220070310 con 0x7f422015c100 2023-10-16T14:36:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.874+0000 7f421f7fe640 1 --2- 172.21.15.29:0/247376752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422015c100 0x7f42200ccf60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f4218009d90 tx=0x7f421802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c75c1a0dc54bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.874+0000 7f421dffb640 1 -- 172.21.15.29:0/247376752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421802cd00 con 0x7f422015c100 2023-10-16T14:36:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421dffb640 1 -- 172.21.15.29:0/247376752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421802ce60 con 0x7f422015c100 2023-10-16T14:36:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421dffb640 1 -- 172.21.15.29:0/247376752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218034640 con 0x7f422015c100 2023-10-16T14:36:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 -- 172.21.15.29:0/247376752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422015c100 msgr2=0x7f42200ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 --2- 172.21.15.29:0/247376752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422015c100 0x7f42200ccf60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f4218009d90 tx=0x7f421802bf50 comp rx=0 tx=0).stop 2023-10-16T14:36:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 -- 172.21.15.29:0/247376752 shutdown_connections 2023-10-16T14:36:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 --2- 172.21.15.29:0/247376752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422015c100 0x7f42200ccf60 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 -- 172.21.15.29:0/247376752 >> 172.21.15.29:0/247376752 conn(0x7f42200ce6a0 msgr2=0x7f42200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 -- 172.21.15.29:0/247376752 shutdown_connections 2023-10-16T14:36:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.875+0000 7f421ffff640 1 -- 172.21.15.29:0/247376752 wait complete. 2023-10-16T14:36:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.877+0000 7f421ffff640 1 Processor -- start 2023-10-16T14:36:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.877+0000 7f421ffff640 1 -- start start 2023-10-16T14:36:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42200ccb00 0x7f42201479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 0x7f4220151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422015c100 0x7f4220152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422014f970 con 0x7f4220153740 2023-10-16T14:36:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422014f3b0 con 0x7f422015c100 2023-10-16T14:36:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422014f690 con 0x7f42200ccb00 2023-10-16T14:36:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.878+0000 7f421e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 0x7f4220151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 0x7f4220151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51270/0 (socket says 172.21.15.29:51270) 2023-10-16T14:36:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 -- 172.21.15.29:0/1497871642 learned_addr learned my addr 172.21.15.29:0/1497871642 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421f7fe640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422015c100 0x7f4220152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421effd640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42200ccb00 0x7f42201479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 -- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42200ccb00 msgr2=0x7f42201479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42200ccb00 0x7f42201479a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 -- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422015c100 msgr2=0x7f4220152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422015c100 0x7f4220152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 -- 172.21.15.29:0/1497871642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218009a70 con 0x7f4220153740 2023-10-16T14:36:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.879+0000 7f421e7fc640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 0x7f4220151fd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f421400c400 tx=0x7f421400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.880+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42140052d0 con 0x7f4220153740 2023-10-16T14:36:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.880+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4214005450 con 0x7f4220153740 2023-10-16T14:36:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.880+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214007260 con 0x7f4220153740 2023-10-16T14:36:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.880+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4220151ab0 con 0x7f4220153740 2023-10-16T14:36:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.880+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42200c1800 con 0x7f4220153740 2023-10-16T14:36:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.881+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42200c1a20 con 0x7f4220153740 2023-10-16T14:36:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.882+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42140073c0 con 0x7f4220153740 2023-10-16T14:36:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.883+0000 7f42077fe640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f420c06afe0 0x7f420c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.883+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f4214088ee0 con 0x7f4220153740 2023-10-16T14:36:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.884+0000 7f421effd640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f420c06afe0 0x7f420c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.885+0000 7f421effd640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f420c06afe0 0x7f420c06d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f42100011e0 tx=0x7f4210009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:13.887+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4214053b70 con 0x7f4220153740 2023-10-16T14:36:14.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.074+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4220151ca0 con 0x7f4220153740 2023-10-16T14:36:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.620+0000 7f42077fe640 1 -- 172.21.15.29:0/1497871642 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4214057190 con 0x7f4220153740 2023-10-16T14:36:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:36:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f420c06afe0 msgr2=0x7f420c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f420c06afe0 0x7f420c06d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f42100011e0 tx=0x7f4210009040 comp rx=0 tx=0).stop 2023-10-16T14:36:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 msgr2=0x7f4220151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 0x7f4220151fd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f421400c400 tx=0x7f421400c8c0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 shutdown_connections 2023-10-16T14:36:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4220153740 0x7f4220151fd0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f420c06afe0 0x7f420c06d4a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422015c100 0x7f4220152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 --2- 172.21.15.29:0/1497871642 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42200ccb00 0x7f42201479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.624+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 >> 172.21.15.29:0/1497871642 conn(0x7f42200ce6a0 msgr2=0x7f42200ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:14.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.626+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 shutdown_connections 2023-10-16T14:36:14.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.626+0000 7f421ffff640 1 -- 172.21.15.29:0/1497871642 wait complete. 2023-10-16T14:36:14.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:36:14.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.872+0000 7f6557fff640 1 Processor -- start 2023-10-16T14:36:14.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.873+0000 7f6557fff640 1 -- start start 2023-10-16T14:36:14.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.874+0000 7f6557fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f655806ce40 0x7f6558069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.874+0000 7f6557fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 0x7f65580cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.874+0000 7f6557fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655806c3f0 con 0x7f6558147470 2023-10-16T14:36:14.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.874+0000 7f6557fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655806a710 con 0x7f655806ce40 2023-10-16T14:36:14.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.874+0000 7f6557fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6558147850 con 0x7f655806a2e0 2023-10-16T14:36:14.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.875+0000 7f6556ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6558147470 0x7f6558164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41694/0 (socket says 172.21.15.29:41694) 2023-10-16T14:36:14.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.875+0000 7f6556ffd640 1 -- 172.21.15.29:0/2163941384 learned_addr learned my addr 172.21.15.29:0/2163941384 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:14.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.875+0000 7f65567fc640 1 --2- 172.21.15.29:0/2163941384 >> v2:172.21.15.170:3300/0 conn(0x7f655806ce40 0x7f6558069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.875+0000 7f65577fe640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 0x7f65580cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.875+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314944880 0 0) 0x7f655806c3f0 con 0x7f6558147470 2023-10-16T14:36:14.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.875+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6544003520 con 0x7f6558147470 2023-10-16T14:36:14.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2795354493 0 0) 0x7f6544003520 con 0x7f6558147470 2023-10-16T14:36:14.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 msgr2=0x7f65580cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:14.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 0x7f65580cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 >> v2:172.21.15.170:3300/0 conn(0x7f655806ce40 msgr2=0x7f6558069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 --2- 172.21.15.29:0/2163941384 >> v2:172.21.15.170:3300/0 conn(0x7f655806ce40 0x7f6558069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65581656a0 con 0x7f6558147470 2023-10-16T14:36:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6548001670 con 0x7f6558147470 2023-10-16T14:36:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 >> v1:172.21.15.29:6789/0 conn(0x7f6558147470 legacy=0x7f6558164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:14.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6544003970 0x7f6544003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 0x7f65440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f6544005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655806a710 con 0x7f6544004d00 2023-10-16T14:36:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6558147850 con 0x7f6544003970 2023-10-16T14:36:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654400bb10 con 0x7f655806a2e0 2023-10-16T14:36:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.876+0000 7f65577fe640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f6544005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.877+0000 7f6556ffd640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6544003970 0x7f6544003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.877+0000 7f65577fe640 1 -- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 msgr2=0x7f65440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.877+0000 7f65577fe640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f655806a2e0 0x7f65440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.877+0000 7f65577fe640 1 -- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6544003970 msgr2=0x7f6544003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:14.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.877+0000 7f65577fe640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6544003970 0x7f6544003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.877+0000 7f65577fe640 1 -- 172.21.15.29:0/2163941384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6544003520 con 0x7f6544004d00 2023-10-16T14:36:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f65577fe640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f6544005180 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f6550007360 tx=0x7f65440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e74d7c598584e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:14.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655002ccf0 con 0x7f6544004d00 2023-10-16T14:36:14.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655002ce50 con 0x7f6544004d00 2023-10-16T14:36:14.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f6555ffb640 1 -- 172.21.15.29:0/2163941384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550034660 con 0x7f6544004d00 2023-10-16T14:36:14.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f6557fff640 1 -- 172.21.15.29:0/2163941384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 msgr2=0x7f6544005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:14.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f6557fff640 1 --2- 172.21.15.29:0/2163941384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f6544005180 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f6550007360 tx=0x7f65440057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.878+0000 7f65577fe640 1 -- 172.21.15.29:0/2163941384 reap_dead start 2023-10-16T14:36:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.879+0000 7f6557fff640 1 -- 172.21.15.29:0/2163941384 shutdown_connections 2023-10-16T14:36:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.879+0000 7f6557fff640 1 -- 172.21.15.29:0/2163941384 >> 172.21.15.29:0/2163941384 conn(0x7f655806aeb0 msgr2=0x7f655806b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:14.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.879+0000 7f6557fff640 1 -- 172.21.15.29:0/2163941384 shutdown_connections 2023-10-16T14:36:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.879+0000 7f6557fff640 1 -- 172.21.15.29:0/2163941384 wait complete. 2023-10-16T14:36:14.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.881+0000 7f6557fff640 1 Processor -- start 2023-10-16T14:36:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.881+0000 7f6557fff640 1 -- start start 2023-10-16T14:36:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6557fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f655813edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6557fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65581403e0 0x7f65580ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6557fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6558140860 0x7f65580ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6557fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655813e000 con 0x7f6544004d00 2023-10-16T14:36:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6557fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655813e2e0 con 0x7f65581403e0 2023-10-16T14:36:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6557fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655813e5c0 con 0x7f6558140860 2023-10-16T14:36:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f655813edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f655813edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40130/0 (socket says 172.21.15.29:40130) 2023-10-16T14:36:14.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 -- 172.21.15.29:0/2327952764 learned_addr learned my addr 172.21.15.29:0/2327952764 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:14.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f65567fc640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65581403e0 0x7f65580ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 -- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6558140860 msgr2=0x7f65580ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:14.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6558140860 0x7f65580ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 -- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65581403e0 msgr2=0x7f65580ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:14.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65581403e0 0x7f65580ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.882+0000 7f6556ffd640 1 -- 172.21.15.29:0/2327952764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655813e8a0 con 0x7f6544004d00 2023-10-16T14:36:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.883+0000 7f6556ffd640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f655813edc0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f654800d830 tx=0x7f654800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.883+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480074e0 con 0x7f6544004d00 2023-10-16T14:36:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.883+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6548007640 con 0x7f6544004d00 2023-10-16T14:36:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.883+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480049e0 con 0x7f6544004d00 2023-10-16T14:36:14.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.883+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6558140e30 con 0x7f6544004d00 2023-10-16T14:36:14.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.883+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655815cc00 con 0x7f6544004d00 2023-10-16T14:36:14.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.884+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655804e9b0 con 0x7f6544004d00 2023-10-16T14:36:14.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.885+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6548004b40 con 0x7f6544004d00 2023-10-16T14:36:14.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.886+0000 7f654f7fe640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f653806afc0 0x7f653806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:14.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.886+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6548089420 con 0x7f6544004d00 2023-10-16T14:36:14.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.887+0000 7f65567fc640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f653806afc0 0x7f653806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:14.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.888+0000 7f65567fc640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f653806afc0 0x7f653806d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6540006a90 tx=0x7f6540008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:14.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:14.890+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65480541b0 con 0x7f6544004d00 2023-10-16T14:36:15.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.081+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f655813e140 con 0x7f6544004d00 2023-10-16T14:36:15.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.636+0000 7f654f7fe640 1 -- 172.21.15.29:0/2327952764 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) v1 ==== 169+0+0 (secure 0 0 0) 0x7f655813e140 con 0x7f6544004d00 2023-10-16T14:36:15.640 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:36:15.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.639+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f653806afc0 msgr2=0x7f653806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:15.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f653806afc0 0x7f653806d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6540006a90 tx=0x7f6540008040 comp rx=0 tx=0).stop 2023-10-16T14:36:15.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 msgr2=0x7f655813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:15.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f655813edc0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f654800d830 tx=0x7f654800dcf0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 shutdown_connections 2023-10-16T14:36:15.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f653806afc0 0x7f653806d480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6558140860 0x7f65580ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f65581403e0 0x7f65580ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 --2- 172.21.15.29:0/2327952764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6544004d00 0x7f655813edc0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 >> 172.21.15.29:0/2327952764 conn(0x7f655806aeb0 msgr2=0x7f65580cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 shutdown_connections 2023-10-16T14:36:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.640+0000 7f6557fff640 1 -- 172.21.15.29:0/2327952764 wait complete. 2023-10-16T14:36:15.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:36:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.893+0000 7fb3ac464640 1 Processor -- start 2023-10-16T14:36:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.893+0000 7fb3ac464640 1 -- start start 2023-10-16T14:36:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.894+0000 7fb3ac464640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a4160ad0 0x7fb3a40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.894+0000 7fb3ac464640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3a40ce350 0x7fb3a4160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.894+0000 7fb3ac464640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a40ce800 con 0x7fb3a40ae260 2023-10-16T14:36:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.894+0000 7fb3ac464640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a40cdec0 con 0x7fb3a40ce350 2023-10-16T14:36:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.894+0000 7fb3ac464640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40acdd0 con 0x7fb3a4160ad0 2023-10-16T14:36:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3aaa01640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb3a40ae260 0x7fb3a40ccd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41708/0 (socket says 172.21.15.29:41708) 2023-10-16T14:36:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3aaa01640 1 -- 172.21.15.29:0/1655208170 learned_addr learned my addr 172.21.15.29:0/1655208170 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3a99ff640 1 --2- 172.21.15.29:0/1655208170 >> v2:172.21.15.29:3300/0 conn(0x7fb3a40ce350 0x7fb3a4160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3aa200640 1 --2- 172.21.15.29:0/1655208170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a4160ad0 0x7fb3a40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3a99ff640 1 -- 172.21.15.29:0/1655208170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a4160ad0 msgr2=0x7fb3a40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3a99ff640 1 --2- 172.21.15.29:0/1655208170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a4160ad0 0x7fb3a40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3a99ff640 1 -- 172.21.15.29:0/1655208170 >> v1:172.21.15.29:6789/0 conn(0x7fb3a40ae260 legacy=0x7fb3a40ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.895+0000 7fb3a99ff640 1 -- 172.21.15.29:0/1655208170 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a41613f0 con 0x7fb3a40ce350 2023-10-16T14:36:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.896+0000 7fb3a99ff640 1 --2- 172.21.15.29:0/1655208170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a40ce350 0x7fb3a4160eb0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb3940097a0 tx=0x7fb39402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0419f7f27a2a8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3a91fe640 1 -- 172.21.15.29:0/1655208170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39402d040 con 0x7fb3a40ce350 2023-10-16T14:36:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3a91fe640 1 -- 172.21.15.29:0/1655208170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb394031810 con 0x7fb3a40ce350 2023-10-16T14:36:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3a91fe640 1 -- 172.21.15.29:0/1655208170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394031ba0 con 0x7fb3a40ce350 2023-10-16T14:36:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3ac464640 1 -- 172.21.15.29:0/1655208170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a40ce350 msgr2=0x7fb3a4160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3ac464640 1 --2- 172.21.15.29:0/1655208170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a40ce350 0x7fb3a4160eb0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb3940097a0 tx=0x7fb39402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3ac464640 1 -- 172.21.15.29:0/1655208170 shutdown_connections 2023-10-16T14:36:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3ac464640 1 --2- 172.21.15.29:0/1655208170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a40ce350 0x7fb3a4160eb0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3ac464640 1 --2- 172.21.15.29:0/1655208170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a4160ad0 0x7fb3a40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.897+0000 7fb3ac464640 1 -- 172.21.15.29:0/1655208170 >> 172.21.15.29:0/1655208170 conn(0x7fb3a406bc10 msgr2=0x7fb3a406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.898+0000 7fb3ac464640 1 -- 172.21.15.29:0/1655208170 shutdown_connections 2023-10-16T14:36:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.898+0000 7fb3ac464640 1 -- 172.21.15.29:0/1655208170 wait complete. 2023-10-16T14:36:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.899+0000 7fb3ac464640 1 Processor -- start 2023-10-16T14:36:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.899+0000 7fb3ac464640 1 -- start start 2023-10-16T14:36:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.900+0000 7fb3ac464640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a40ae260 0x7fb3a414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.900+0000 7fb3ac464640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3a40ce350 0x7fb3a40b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.900+0000 7fb3ac464640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 0x7fb3a40ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.900+0000 7fb3ac464640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a41f2770 con 0x7fb3a4160ad0 2023-10-16T14:36:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.900+0000 7fb3ac464640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a414f510 con 0x7fb3a40ce350 2023-10-16T14:36:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.900+0000 7fb3ac464640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40c4210 con 0x7fb3a40ae260 2023-10-16T14:36:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 0x7fb3a40ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 0x7fb3a40ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40144/0 (socket says 172.21.15.29:40144) 2023-10-16T14:36:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 -- 172.21.15.29:0/651684153 learned_addr learned my addr 172.21.15.29:0/651684153 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aa200640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a40ae260 0x7fb3a414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3a99ff640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3a40ce350 0x7fb3a40b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 -- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a40ae260 msgr2=0x7fb3a414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a40ae260 0x7fb3a414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 -- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3a40ce350 msgr2=0x7fb3a40b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3a40ce350 0x7fb3a40b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3aaa01640 1 -- 172.21.15.29:0/651684153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394009400 con 0x7fb3a4160ad0 2023-10-16T14:36:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.901+0000 7fb3a99ff640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3a40ce350 0x7fb3a40b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.902+0000 7fb3aaa01640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 0x7fb3a40ccd60 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00cb40 tx=0x7fb39c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.902+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00aca0 con 0x7fb3a4160ad0 2023-10-16T14:36:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.902+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c00ae00 con 0x7fb3a4160ad0 2023-10-16T14:36:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.902+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c012600 con 0x7fb3a4160ad0 2023-10-16T14:36:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.902+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a40c3190 con 0x7fb3a4160ad0 2023-10-16T14:36:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.902+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a40c3500 con 0x7fb3a4160ad0 2023-10-16T14:36:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.903+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a404e160 con 0x7fb3a4160ad0 2023-10-16T14:36:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.904+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb39c007070 con 0x7fb3a4160ad0 2023-10-16T14:36:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.905+0000 7fb39affd640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fb38806af90 0x7fb38806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.905+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fb39c089660 con 0x7fb3a4160ad0 2023-10-16T14:36:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.906+0000 7fb3aa200640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fb38806af90 0x7fb38806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.907+0000 7fb3aa200640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fb38806af90 0x7fb38806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006fd0 tx=0x7fb3a0008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:15.909+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb39c0545e0 con 0x7fb3a4160ad0 2023-10-16T14:36:16.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.103+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb3a4065d60 con 0x7fb3a4160ad0 2023-10-16T14:36:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.110+0000 7fb39affd640 1 -- 172.21.15.29:0/651684153 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb39c057c00 con 0x7fb3a4160ad0 2023-10-16T14:36:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.113+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fb38806af90 msgr2=0x7fb38806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.113+0000 7fb3ac464640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fb38806af90 0x7fb38806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006fd0 tx=0x7fb3a0008040 comp rx=0 tx=0).stop 2023-10-16T14:36:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.113+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 msgr2=0x7fb3a40ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.113+0000 7fb3ac464640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 0x7fb3a40ccd60 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00cb40 tx=0x7fb39c00a060 comp rx=0 tx=0).stop 2023-10-16T14:36:16.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 shutdown_connections 2023-10-16T14:36:16.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3a4160ad0 0x7fb3a40ccd60 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fb38806af90 0x7fb38806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3a40ce350 0x7fb3a40b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 --2- 172.21.15.29:0/651684153 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb3a40ae260 0x7fb3a414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 >> 172.21.15.29:0/651684153 conn(0x7fb3a406bc10 msgr2=0x7fb3a40b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 shutdown_connections 2023-10-16T14:36:16.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.114+0000 7fb3ac464640 1 -- 172.21.15.29:0/651684153 wait complete. 2023-10-16T14:36:16.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:36:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.362+0000 7f571db55640 1 Processor -- start 2023-10-16T14:36:16.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.362+0000 7f571db55640 1 -- start start 2023-10-16T14:36:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571db55640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57180cd970 0x7f571815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571db55640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571806be30 con 0x7f57180b3ec0 2023-10-16T14:36:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571db55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180ada10 con 0x7f57180b0ea0 2023-10-16T14:36:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571db55640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571806ca00 con 0x7f57180cd970 2023-10-16T14:36:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571cb53640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f57180b3ec0 0x7f57180cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41716/0 (socket says 172.21.15.29:41716) 2023-10-16T14:36:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571cb53640 1 -- 172.21.15.29:0/705100094 learned_addr learned my addr 172.21.15.29:0/705100094 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.364+0000 7f571d354640 1 --2- 172.21.15.29:0/705100094 >> v2:172.21.15.29:3300/0 conn(0x7f57180cd970 0x7f571815aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.365+0000 7f57177fe640 1 -- 172.21.15.29:0/705100094 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822440129 0 0) 0x7f571806be30 con 0x7f57180b3ec0 2023-10-16T14:36:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.365+0000 7f57177fe640 1 -- 172.21.15.29:0/705100094 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708003520 con 0x7f57180b3ec0 2023-10-16T14:36:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.365+0000 7f571d354640 1 -- 172.21.15.29:0/705100094 >> v1:172.21.15.170:6789/0 conn(0x7f57180b0ea0 legacy=0x7f571815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.365+0000 7f571d354640 1 -- 172.21.15.29:0/705100094 >> v1:172.21.15.29:6789/0 conn(0x7f57180b3ec0 legacy=0x7f57180cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.365+0000 7f571d354640 1 -- 172.21.15.29:0/705100094 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571806c060 con 0x7f57180cd970 2023-10-16T14:36:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.366+0000 7f571d354640 1 --2- 172.21.15.29:0/705100094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815aa00 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f5710009d90 tx=0x7f571002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f9bd772626082f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.366+0000 7f57177fe640 1 -- 172.21.15.29:0/705100094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571002cc60 con 0x7f57180cd970 2023-10-16T14:36:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.366+0000 7f57177fe640 1 -- 172.21.15.29:0/705100094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571002cdc0 con 0x7f57180cd970 2023-10-16T14:36:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.366+0000 7f57177fe640 1 -- 172.21.15.29:0/705100094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710034660 con 0x7f57180cd970 2023-10-16T14:36:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 -- 172.21.15.29:0/705100094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 msgr2=0x7f571815aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 --2- 172.21.15.29:0/705100094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815aa00 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f5710009d90 tx=0x7f571002beb0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 -- 172.21.15.29:0/705100094 shutdown_connections 2023-10-16T14:36:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 --2- 172.21.15.29:0/705100094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815aa00 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 -- 172.21.15.29:0/705100094 >> 172.21.15.29:0/705100094 conn(0x7f57180b0170 msgr2=0x7f57180b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 -- 172.21.15.29:0/705100094 shutdown_connections 2023-10-16T14:36:16.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.367+0000 7f571db55640 1 -- 172.21.15.29:0/705100094 wait complete. 2023-10-16T14:36:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.369+0000 7f571db55640 1 Processor -- start 2023-10-16T14:36:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.369+0000 7f571db55640 1 -- start start 2023-10-16T14:36:16.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571db55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57180b0ea0 0x7f57181f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571db55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57180b3ec0 0x7f5718151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571db55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571db55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181f4890 con 0x7f57180cd970 2023-10-16T14:36:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571db55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718158430 con 0x7f57180b3ec0 2023-10-16T14:36:16.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571db55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180c6f90 con 0x7f57180b0ea0 2023-10-16T14:36:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571d354640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571d354640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40150/0 (socket says 172.21.15.29:40150) 2023-10-16T14:36:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.370+0000 7f571d354640 1 -- 172.21.15.29:0/2470819523 learned_addr learned my addr 172.21.15.29:0/2470819523 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571cb53640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57180b0ea0 0x7f57181f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f5717fff640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57180b3ec0 0x7f5718151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571d354640 1 -- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57180b0ea0 msgr2=0x7f57181f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571d354640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57180b0ea0 0x7f57181f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571d354640 1 -- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57180b3ec0 msgr2=0x7f5718151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571d354640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57180b3ec0 0x7f5718151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571d354640 1 -- 172.21.15.29:0/2470819523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710009a70 con 0x7f57180cd970 2023-10-16T14:36:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f571d354640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815a1d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f5710009d90 tx=0x7f5710004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.371+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710040650 con 0x7f57180cd970 2023-10-16T14:36:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.372+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57100407b0 con 0x7f57180cd970 2023-10-16T14:36:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.372+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710040b60 con 0x7f57180cd970 2023-10-16T14:36:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.372+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5718157e70 con 0x7f57180cd970 2023-10-16T14:36:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.372+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5718154260 con 0x7f57180cd970 2023-10-16T14:36:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.373+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57180c6d60 con 0x7f57180cd970 2023-10-16T14:36:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.374+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5710040cc0 con 0x7f57180cd970 2023-10-16T14:36:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.375+0000 7f57157fa640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f56fc06afc0 0x7f56fc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.375+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f57100ac580 con 0x7f57180cd970 2023-10-16T14:36:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.376+0000 7f571cb53640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f56fc06afc0 0x7f56fc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.377+0000 7f571cb53640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f56fc06afc0 0x7f56fc06d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f570c0011e0 tx=0x7f570c009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.380+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f57180c6d60 con 0x7f57180cd970 2023-10-16T14:36:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.571+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f57180c6d60 con 0x7f57180cd970 2023-10-16T14:36:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.577+0000 7f57157fa640 1 -- 172.21.15.29:0/2470819523 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v44) v1 ==== 115+0+0 (secure 0 0 0) 0x7f57180c6d60 con 0x7f57180cd970 2023-10-16T14:36:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f56fc06afc0 msgr2=0x7f56fc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f56fc06afc0 0x7f56fc06d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f570c0011e0 tx=0x7f570c009040 comp rx=0 tx=0).stop 2023-10-16T14:36:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 msgr2=0x7f571815a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815a1d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f5710009d90 tx=0x7f5710004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 shutdown_connections 2023-10-16T14:36:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57180cd970 0x7f571815a1d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f56fc06afc0 0x7f56fc06d480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f57180b3ec0 0x7f5718151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 --2- 172.21.15.29:0/2470819523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f57180b0ea0 0x7f57181f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 >> 172.21.15.29:0/2470819523 conn(0x7f57180b0170 msgr2=0x7f57180b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:16.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 shutdown_connections 2023-10-16T14:36:16.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.580+0000 7f571db55640 1 -- 172.21.15.29:0/2470819523 wait complete. 2023-10-16T14:36:16.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:36:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.830+0000 7f0678421640 1 Processor -- start 2023-10-16T14:36:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.830+0000 7f0678421640 1 -- start start 2023-10-16T14:36:16.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.832+0000 7f0678421640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06700ade50 0x7f0670159110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.832+0000 7f0678421640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f067006fcb0 0x7f0670070120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.832+0000 7f0678421640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067015d400 con 0x7f067006d560 2023-10-16T14:36:16.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.832+0000 7f0678421640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067015d6e0 con 0x7f06700ade50 2023-10-16T14:36:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.832+0000 7f0678421640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067015d9c0 con 0x7f067006fcb0 2023-10-16T14:36:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.832+0000 7f06759bc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06700ade50 0x7f0670159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06759bc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06700ade50 0x7f0670159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40156/0 (socket says 172.21.15.29:40156) 2023-10-16T14:36:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06759bc640 1 -- 172.21.15.29:0/1440859591 learned_addr learned my addr 172.21.15.29:0/1440859591 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06769be640 1 --2- 172.21.15.29:0/1440859591 >> v2:172.21.15.170:3300/0 conn(0x7f067006fcb0 0x7f0670070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06759bc640 1 -- 172.21.15.29:0/1440859591 >> v2:172.21.15.170:3300/0 conn(0x7f067006fcb0 msgr2=0x7f0670070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06759bc640 1 --2- 172.21.15.29:0/1440859591 >> v2:172.21.15.170:3300/0 conn(0x7f067006fcb0 0x7f0670070120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06759bc640 1 -- 172.21.15.29:0/1440859591 >> v1:172.21.15.170:6789/0 conn(0x7f067006d560 legacy=0x7f067015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06759bc640 1 -- 172.21.15.29:0/1440859591 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067006f4b0 con 0x7f06700ade50 2023-10-16T14:36:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.833+0000 7f06769be640 1 --2- 172.21.15.29:0/1440859591 >> v2:172.21.15.170:3300/0 conn(0x7f067006fcb0 0x7f0670070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.834+0000 7f06759bc640 1 --2- 172.21.15.29:0/1440859591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06700ade50 0x7f0670159110 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f06700ce940 tx=0x7f066002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afa98522863b3ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.835+0000 7f06751bb640 1 -- 172.21.15.29:0/1440859591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660031870 con 0x7f06700ade50 2023-10-16T14:36:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.835+0000 7f06751bb640 1 -- 172.21.15.29:0/1440859591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06600319d0 con 0x7f06700ade50 2023-10-16T14:36:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.835+0000 7f06751bb640 1 -- 172.21.15.29:0/1440859591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660031d60 con 0x7f06700ade50 2023-10-16T14:36:16.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.835+0000 7f0678421640 1 -- 172.21.15.29:0/1440859591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06700ade50 msgr2=0x7f0670159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.835+0000 7f0678421640 1 --2- 172.21.15.29:0/1440859591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06700ade50 0x7f0670159110 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f06700ce940 tx=0x7f066002b910 comp rx=0 tx=0).stop 2023-10-16T14:36:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.836+0000 7f0678421640 1 -- 172.21.15.29:0/1440859591 shutdown_connections 2023-10-16T14:36:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.836+0000 7f0678421640 1 --2- 172.21.15.29:0/1440859591 >> v2:172.21.15.170:3300/0 conn(0x7f067006fcb0 0x7f0670070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.836+0000 7f0678421640 1 --2- 172.21.15.29:0/1440859591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06700ade50 0x7f0670159110 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.836+0000 7f0678421640 1 -- 172.21.15.29:0/1440859591 >> 172.21.15.29:0/1440859591 conn(0x7f06700ce9b0 msgr2=0x7f06700cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.836+0000 7f0678421640 1 -- 172.21.15.29:0/1440859591 shutdown_connections 2023-10-16T14:36:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.836+0000 7f0678421640 1 -- 172.21.15.29:0/1440859591 wait complete. 2023-10-16T14:36:16.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.838+0000 7f0678421640 1 Processor -- start 2023-10-16T14:36:16.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.838+0000 7f0678421640 1 -- start start 2023-10-16T14:36:16.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f0678421640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f067006d560 0x7f0670150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f0678421640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 0x7f06700af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f0678421640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06700ade50 0x7f06700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f0678421640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0670151420 con 0x7f067006fcb0 2023-10-16T14:36:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f0678421640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06700afc80 con 0x7f06700ade50 2023-10-16T14:36:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f0678421640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06700ae6a0 con 0x7f067006d560 2023-10-16T14:36:16.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 0x7f06700af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 0x7f06700af740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40160/0 (socket says 172.21.15.29:40160) 2023-10-16T14:36:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 -- 172.21.15.29:0/2017143367 learned_addr learned my addr 172.21.15.29:0/2017143367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 -- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f067006d560 msgr2=0x7f0670150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f067006d560 0x7f0670150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 -- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06700ade50 msgr2=0x7f06700b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06700ade50 0x7f06700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.839+0000 7f06759bc640 1 -- 172.21.15.29:0/2017143367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06600094d0 con 0x7f067006fcb0 2023-10-16T14:36:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.840+0000 7f06759bc640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 0x7f06700af740 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0660000c00 tx=0x7f06600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.840+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660031840 con 0x7f067006fcb0 2023-10-16T14:36:16.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.840+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06600443f0 con 0x7f067006fcb0 2023-10-16T14:36:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.840+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06600447a0 con 0x7f067006fcb0 2023-10-16T14:36:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.840+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06700ae980 con 0x7f067006fcb0 2023-10-16T14:36:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.840+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06700aef40 con 0x7f067006fcb0 2023-10-16T14:36:16.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.841+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067004e160 con 0x7f067006fcb0 2023-10-16T14:36:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.842+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f066004d450 con 0x7f067006fcb0 2023-10-16T14:36:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.843+0000 7f0666ffd640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f065406af90 0x7f065406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:16.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.843+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f066002d040 con 0x7f067006fcb0 2023-10-16T14:36:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.844+0000 7f06761bd640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f065406af90 0x7f065406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.845+0000 7f06761bd640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f065406af90 0x7f065406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f066c001560 tx=0x7f066c007990 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:16.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:16.848+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0660076e70 con 0x7f067006fcb0 2023-10-16T14:36:17.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.046+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f067006ca00 con 0x7f067006fcb0 2023-10-16T14:36:17.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.053+0000 7f0666ffd640 1 -- 172.21.15.29:0/2017143367 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f06600350b0 con 0x7f067006fcb0 2023-10-16T14:36:17.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f065406af90 msgr2=0x7f065406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f065406af90 0x7f065406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f066c001560 tx=0x7f066c007990 comp rx=0 tx=0).stop 2023-10-16T14:36:17.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 msgr2=0x7f06700af740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 0x7f06700af740 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0660000c00 tx=0x7f06600046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 shutdown_connections 2023-10-16T14:36:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f067006fcb0 0x7f06700af740 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f065406af90 0x7f065406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06700ade50 0x7f06700b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 --2- 172.21.15.29:0/2017143367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f067006d560 0x7f0670150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.056+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 >> 172.21.15.29:0/2017143367 conn(0x7f06700ce9b0 msgr2=0x7f06700cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.057+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 shutdown_connections 2023-10-16T14:36:17.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.057+0000 7f0678421640 1 -- 172.21.15.29:0/2017143367 wait complete. 2023-10-16T14:36:17.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T14:36:17.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.307+0000 7f4b7b375640 1 Processor -- start 2023-10-16T14:36:17.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.307+0000 7f4b7b375640 1 -- start start 2023-10-16T14:36:17.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.308+0000 7f4b7b375640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b7415aaa0 0x7f4b740b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.308+0000 7f4b7b375640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4b740ca8a0 0x7f4b740b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7b375640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b74070240 0x7f4b740706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7b375640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b74070be0 con 0x7f4b74070240 2023-10-16T14:36:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7b375640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b740cad00 con 0x7f4b740ca8a0 2023-10-16T14:36:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7b375640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740b2d00 con 0x7f4b7415aaa0 2023-10-16T14:36:17.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b74070240 0x7f4b740706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b74070240 0x7f4b740706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40164/0 (socket says 172.21.15.29:40164) 2023-10-16T14:36:17.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 -- 172.21.15.29:0/4068081513 learned_addr learned my addr 172.21.15.29:0/4068081513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:17.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 -- 172.21.15.29:0/4068081513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b7415aaa0 msgr2=0x7f4b740b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:17.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 --2- 172.21.15.29:0/4068081513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b7415aaa0 0x7f4b740b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 -- 172.21.15.29:0/4068081513 >> v2:172.21.15.170:3300/0 conn(0x7f4b740ca8a0 msgr2=0x7f4b740b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 --2- 172.21.15.29:0/4068081513 >> v2:172.21.15.170:3300/0 conn(0x7f4b740ca8a0 0x7f4b740b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.309+0000 7f4b7ab74640 1 -- 172.21.15.29:0/4068081513 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b740b2fe0 con 0x7f4b74070240 2023-10-16T14:36:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.311+0000 7f4b7ab74640 1 --2- 172.21.15.29:0/4068081513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b74070240 0x7f4b740706a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4b740cf560 tx=0x7f4b6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7304cf20b80cdd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.311+0000 7f4b79371640 1 -- 172.21.15.29:0/4068081513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c02ca80 con 0x7f4b74070240 2023-10-16T14:36:17.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.311+0000 7f4b79371640 1 -- 172.21.15.29:0/4068081513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c02cbe0 con 0x7f4b74070240 2023-10-16T14:36:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.311+0000 7f4b79371640 1 -- 172.21.15.29:0/4068081513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0343f0 con 0x7f4b74070240 2023-10-16T14:36:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 -- 172.21.15.29:0/4068081513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b74070240 msgr2=0x7f4b740706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 --2- 172.21.15.29:0/4068081513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b74070240 0x7f4b740706a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4b740cf560 tx=0x7f4b6c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 -- 172.21.15.29:0/4068081513 shutdown_connections 2023-10-16T14:36:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 --2- 172.21.15.29:0/4068081513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b74070240 0x7f4b740706a0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 --2- 172.21.15.29:0/4068081513 >> v2:172.21.15.170:3300/0 conn(0x7f4b740ca8a0 0x7f4b740b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 --2- 172.21.15.29:0/4068081513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b7415aaa0 0x7f4b740b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 -- 172.21.15.29:0/4068081513 >> 172.21.15.29:0/4068081513 conn(0x7f4b740cc230 msgr2=0x7f4b740cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 -- 172.21.15.29:0/4068081513 shutdown_connections 2023-10-16T14:36:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.312+0000 7f4b7b375640 1 -- 172.21.15.29:0/4068081513 wait complete. 2023-10-16T14:36:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.315+0000 7f4b7b375640 1 Processor -- start 2023-10-16T14:36:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.315+0000 7f4b7b375640 1 -- start start 2023-10-16T14:36:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7b375640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b74070240 0x7f4b740c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7b375640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b740ca8a0 0x7f4b74143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7b375640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 0x7f4b74143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7b375640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740c61d0 con 0x7f4b7415aaa0 2023-10-16T14:36:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7b375640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740b1bd0 con 0x7f4b74070240 2023-10-16T14:36:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7b375640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740b0e10 con 0x7f4b740ca8a0 2023-10-16T14:36:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7ab74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 0x7f4b74143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7ab74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 0x7f4b74143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40172/0 (socket says 172.21.15.29:40172) 2023-10-16T14:36:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.316+0000 7f4b7ab74640 1 -- 172.21.15.29:0/694929925 learned_addr learned my addr 172.21.15.29:0/694929925 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7ab74640 1 -- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b740ca8a0 msgr2=0x7f4b74143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7ab74640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b740ca8a0 0x7f4b74143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7a373640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b74070240 0x7f4b740c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7ab74640 1 -- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b74070240 msgr2=0x7f4b740c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7ab74640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b74070240 0x7f4b740c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7ab74640 1 -- 172.21.15.29:0/694929925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c009b30 con 0x7f4b7415aaa0 2023-10-16T14:36:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.317+0000 7f4b7ab74640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 0x7f4b74143cc0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0093e0 tx=0x7f4b6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.318+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c009510 con 0x7f4b7415aaa0 2023-10-16T14:36:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.318+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c03e450 con 0x7f4b7415aaa0 2023-10-16T14:36:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.318+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c03e7c0 con 0x7f4b7415aaa0 2023-10-16T14:36:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.318+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b740b10f0 con 0x7f4b7415aaa0 2023-10-16T14:36:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.318+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b740b16b0 con 0x7f4b7415aaa0 2023-10-16T14:36:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.319+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74146160 con 0x7f4b7415aaa0 2023-10-16T14:36:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.320+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4b6c02cd50 con 0x7f4b7415aaa0 2023-10-16T14:36:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.321+0000 7f4b6affd640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b5806af90 0x7f4b5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.321+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f4b6c0ac470 con 0x7f4b7415aaa0 2023-10-16T14:36:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.322+0000 7f4b7a373640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b5806af90 0x7f4b5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.323+0000 7f4b7a373640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b5806af90 0x7f4b5806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000a810 tx=0x7f4b70008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.326+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4b6c077350 con 0x7f4b7415aaa0 2023-10-16T14:36:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.589+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f4b7404e9b0 con 0x7f4b7415aaa0 2023-10-16T14:36:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.640+0000 7f4b6affd640 1 -- 172.21.15.29:0/694929925 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v755) v1 ==== 148+0+0 (secure 0 0 0) 0x7f4b6c07a970 con 0x7f4b7415aaa0 2023-10-16T14:36:17.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b5806af90 msgr2=0x7f4b5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b5806af90 0x7f4b5806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000a810 tx=0x7f4b70008040 comp rx=0 tx=0).stop 2023-10-16T14:36:17.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 msgr2=0x7f4b74143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 0x7f4b74143cc0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0093e0 tx=0x7f4b6c004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:17.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 shutdown_connections 2023-10-16T14:36:17.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b7415aaa0 0x7f4b74143cc0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b5806af90 0x7f4b5806d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b740ca8a0 0x7f4b74143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 --2- 172.21.15.29:0/694929925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b74070240 0x7f4b740c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.643+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 >> 172.21.15.29:0/694929925 conn(0x7f4b740cc230 msgr2=0x7f4b7415d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:17.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.644+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 shutdown_connections 2023-10-16T14:36:17.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.644+0000 7f4b7b375640 1 -- 172.21.15.29:0/694929925 wait complete. 2023-10-16T14:36:17.666 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-16T14:36:17.666 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:36:17.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.888+0000 7fca79678640 1 Processor -- start 2023-10-16T14:36:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.888+0000 7fca79678640 1 -- start start 2023-10-16T14:36:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.889+0000 7fca79678640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca7415cfc0 0x7fca740b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.889+0000 7fca79678640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca740d2640 0x7fca740b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.889+0000 7fca79678640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740d15a0 con 0x7fca7415cbe0 2023-10-16T14:36:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.889+0000 7fca79678640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740d1880 con 0x7fca740d2640 2023-10-16T14:36:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.889+0000 7fca79678640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca740d1b60 con 0x7fca7415cfc0 2023-10-16T14:36:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca73fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fca7415cbe0 0x7fca740d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41730/0 (socket says 172.21.15.29:41730) 2023-10-16T14:36:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca73fff640 1 -- 172.21.15.29:0/187715044 learned_addr learned my addr 172.21.15.29:0/187715044 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca78e77640 1 --2- 172.21.15.29:0/187715044 >> v2:172.21.15.29:3300/0 conn(0x7fca740d2640 0x7fca740b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca737fe640 1 --2- 172.21.15.29:0/187715044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca7415cfc0 0x7fca740b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca78e77640 1 -- 172.21.15.29:0/187715044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca7415cfc0 msgr2=0x7fca740b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca78e77640 1 --2- 172.21.15.29:0/187715044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca7415cfc0 0x7fca740b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca78e77640 1 -- 172.21.15.29:0/187715044 >> v1:172.21.15.29:6789/0 conn(0x7fca7415cbe0 legacy=0x7fca740d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca78e77640 1 -- 172.21.15.29:0/187715044 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740d1e40 con 0x7fca740d2640 2023-10-16T14:36:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.890+0000 7fca72ffd640 1 -- 172.21.15.29:0/187715044 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874285572 0 0) 0x7fca740d15a0 con 0x7fca7415cbe0 2023-10-16T14:36:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca78e77640 1 --2- 172.21.15.29:0/187715044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca740d2640 0x7fca740b4ab0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0097a0 tx=0x7fca6c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc69ce3cdb224736 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca72ffd640 1 -- 172.21.15.29:0/187715044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c030800 con 0x7fca740d2640 2023-10-16T14:36:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca72ffd640 1 -- 172.21.15.29:0/187715044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c030960 con 0x7fca740d2640 2023-10-16T14:36:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca72ffd640 1 -- 172.21.15.29:0/187715044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c030cf0 con 0x7fca740d2640 2023-10-16T14:36:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca79678640 1 -- 172.21.15.29:0/187715044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca740d2640 msgr2=0x7fca740b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca79678640 1 --2- 172.21.15.29:0/187715044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca740d2640 0x7fca740b4ab0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0097a0 tx=0x7fca6c02b940 comp rx=0 tx=0).stop 2023-10-16T14:36:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca79678640 1 -- 172.21.15.29:0/187715044 shutdown_connections 2023-10-16T14:36:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca79678640 1 --2- 172.21.15.29:0/187715044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca740d2640 0x7fca740b4ab0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.892+0000 7fca79678640 1 --2- 172.21.15.29:0/187715044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca7415cfc0 0x7fca740b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.893+0000 7fca79678640 1 -- 172.21.15.29:0/187715044 >> 172.21.15.29:0/187715044 conn(0x7fca740cf570 msgr2=0x7fca740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.893+0000 7fca79678640 1 -- 172.21.15.29:0/187715044 shutdown_connections 2023-10-16T14:36:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.893+0000 7fca79678640 1 -- 172.21.15.29:0/187715044 wait complete. 2023-10-16T14:36:17.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.894+0000 7fca79678640 1 Processor -- start 2023-10-16T14:36:17.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.895+0000 7fca79678640 1 -- start start 2023-10-16T14:36:17.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca79678640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca740d2640 0x7fca7414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca79678640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 0x7fca740ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca79678640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca7415cfc0 0x7fca740c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca79678640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740cbc80 con 0x7fca7415cbe0 2023-10-16T14:36:17.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca79678640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740cb070 con 0x7fca7415cfc0 2023-10-16T14:36:17.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca79678640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca740cb350 con 0x7fca740d2640 2023-10-16T14:36:17.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 0x7fca740ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 0x7fca740ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40194/0 (socket says 172.21.15.29:40194) 2023-10-16T14:36:17.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 -- 172.21.15.29:0/4036875315 learned_addr learned my addr 172.21.15.29:0/4036875315 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:17.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca73fff640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca740d2640 0x7fca7414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 -- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca740d2640 msgr2=0x7fca7414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca740d2640 0x7fca7414ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 -- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca7415cfc0 msgr2=0x7fca740c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:17.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca7415cfc0 0x7fca740c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:17.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.896+0000 7fca737fe640 1 -- 172.21.15.29:0/4036875315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c009400 con 0x7fca7415cbe0 2023-10-16T14:36:17.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.897+0000 7fca737fe640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 0x7fca740ca5d0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fca6800d5f0 tx=0x7fca6800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:17.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.897+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680125f0 con 0x7fca7415cbe0 2023-10-16T14:36:17.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.897+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca68012750 con 0x7fca7415cbe0 2023-10-16T14:36:17.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.897+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68012b00 con 0x7fca7415cbe0 2023-10-16T14:36:17.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.897+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca740cb630 con 0x7fca7415cbe0 2023-10-16T14:36:17.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.897+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7414e410 con 0x7fca7415cbe0 2023-10-16T14:36:17.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.898+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7404e9b0 con 0x7fca7415cbe0 2023-10-16T14:36:17.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.899+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fca68012c60 con 0x7fca7415cbe0 2023-10-16T14:36:17.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.900+0000 7fca70ff9640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fca6006af90 0x7fca6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:17.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.900+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fca68089db0 con 0x7fca7415cbe0 2023-10-16T14:36:17.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.901+0000 7fca73fff640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fca6006af90 0x7fca6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:17.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.902+0000 7fca73fff640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fca6006af90 0x7fca6006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca640012e0 tx=0x7fca64009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:17.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:17.904+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fca6808e070 con 0x7fca7415cbe0 2023-10-16T14:36:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.096+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca740ce9f0 con 0x7fca7415cbe0 2023-10-16T14:36:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.099+0000 7fca70ff9640 1 -- 172.21.15.29:0/4036875315 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+14184 (secure 0 0 0) 0x7fca68054bf0 con 0x7fca7415cbe0 2023-10-16T14:36:18.103 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:18.103 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":133,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:15.633307+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/3633682082":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/2408525211":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3344036704":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/2843457870":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1110178125":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/110350219":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1384820266":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3330302692":"2023-10-17T14:35:42.393479+0000","172.21.15.29:6835/2882189329":"2023-10-17T14:36:12.609718+0000","172.21.15.170:0/1968227499":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/3494288089":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/3633682082":"2023-10-17T14:36:08.423291+0000","172.21.15.170:6833/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.29:6833/2882189329":"2023-10-17T14:36:12.609718+0000","172.21.15.170:0/2874036200":"2023-10-17T14:35:42.393479+0000","172.21.15.170:6832/1801486769":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/3741475474":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/286310502":"2023-10-17T14:36:08.423291+0000","172.21.15.170:0/1228162669":"2023-10-17T14:35:42.393479+0000","172.21.15.170:0/1339480091":"2023-10-17T14:35:42.393479+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.101+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fca6006af90 msgr2=0x7fca6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.101+0000 7fca79678640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fca6006af90 0x7fca6006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca640012e0 tx=0x7fca64009040 comp rx=0 tx=0).stop 2023-10-16T14:36:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.101+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 msgr2=0x7fca740ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.101+0000 7fca79678640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 0x7fca740ca5d0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fca6800d5f0 tx=0x7fca6800dab0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 shutdown_connections 2023-10-16T14:36:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415cbe0 0x7fca740ca5d0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fca6006af90 0x7fca6006d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fca7415cfc0 0x7fca740c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 --2- 172.21.15.29:0/4036875315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fca740d2640 0x7fca7414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 >> 172.21.15.29:0/4036875315 conn(0x7fca740cf570 msgr2=0x7fca740d2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 shutdown_connections 2023-10-16T14:36:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.102+0000 7fca79678640 1 -- 172.21.15.29:0/4036875315 wait complete. 2023-10-16T14:36:18.126 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:36:18.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:36:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.358+0000 7fe0055d0640 1 Processor -- start 2023-10-16T14:36:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.358+0000 7fe0055d0640 1 -- start start 2023-10-16T14:36:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.360+0000 7fe0055d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0000cd9b0 0x7fe0000b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.360+0000 7fe0055d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00015b0b0 con 0x7fe0000afcf0 2023-10-16T14:36:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.360+0000 7fe0055d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0000b6730 con 0x7fe00015acd0 2023-10-16T14:36:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.360+0000 7fe0055d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000cc6f0 con 0x7fe0000cd9b0 2023-10-16T14:36:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.360+0000 7fe004dcf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe0000afcf0 0x7fe0000ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41746/0 (socket says 172.21.15.29:41746) 2023-10-16T14:36:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.360+0000 7fe004dcf640 1 -- 172.21.15.29:0/2774400191 learned_addr learned my addr 172.21.15.29:0/2774400191 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0000cd9b0 0x7fe0000b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464441329 0 0) 0x7fe00015b0b0 con 0x7fe0000afcf0 2023-10-16T14:36:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfec003520 con 0x7fe0000afcf0 2023-10-16T14:36:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579621784 0 0) 0x7fdfec003520 con 0x7fe0000afcf0 2023-10-16T14:36:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0000cd9b0 msgr2=0x7fe0000b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0000cd9b0 0x7fe0000b60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 >> v1:172.21.15.170:6789/0 conn(0x7fe00015acd0 legacy=0x7fe0000cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0000cc9d0 con 0x7fe0000afcf0 2023-10-16T14:36:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fdff8001670 con 0x7fe0000afcf0 2023-10-16T14:36:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 >> v1:172.21.15.29:6789/0 conn(0x7fe0000afcf0 legacy=0x7fe0000ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfec0039a0 0x7fdfec003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfec0047c0 0x7fdfec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec005180 0x7fdfec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000cc6f0 con 0x7fdfec0047c0 2023-10-16T14:36:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec00c050 con 0x7fdfec0039a0 2023-10-16T14:36:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec00c1c0 con 0x7fdfec005180 2023-10-16T14:36:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.361+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0000cd9b0 0x7fe0000b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfec0047c0 0x7fdfec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fe004dcf640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec005180 0x7fdfec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdffffff640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfec0039a0 0x7fdfec003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdff77fe640 1 -- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec005180 msgr2=0x7fdfec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec005180 0x7fdfec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdff77fe640 1 -- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfec0039a0 msgr2=0x7fdfec003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdfec0039a0 0x7fdfec003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.362+0000 7fdff77fe640 1 -- 172.21.15.29:0/2774400191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec003520 con 0x7fdfec0047c0 2023-10-16T14:36:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.364+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfec0047c0 0x7fdfec004c40 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8006fd0 tx=0x7fdfec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a9eb8e44aa62a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.364+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8032600 con 0x7fdfec0047c0 2023-10-16T14:36:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.364+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe802e050 con 0x7fdfec0047c0 2023-10-16T14:36:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.364+0000 7fdfff7fe640 1 -- 172.21.15.29:0/2774400191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8032980 con 0x7fdfec0047c0 2023-10-16T14:36:18.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe0055d0640 1 -- 172.21.15.29:0/2774400191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfec0047c0 msgr2=0x7fdfec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2774400191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfec0047c0 0x7fdfec004c40 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8006fd0 tx=0x7fdfec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe004dcf640 1 -- 172.21.15.29:0/2774400191 reap_dead start 2023-10-16T14:36:18.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe0055d0640 1 -- 172.21.15.29:0/2774400191 shutdown_connections 2023-10-16T14:36:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe0055d0640 1 -- 172.21.15.29:0/2774400191 >> 172.21.15.29:0/2774400191 conn(0x7fe0000b23b0 msgr2=0x7fe0000b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe0055d0640 1 -- 172.21.15.29:0/2774400191 shutdown_connections 2023-10-16T14:36:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.365+0000 7fe0055d0640 1 -- 172.21.15.29:0/2774400191 wait complete. 2023-10-16T14:36:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.367+0000 7fe0055d0640 1 Processor -- start 2023-10-16T14:36:18.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.367+0000 7fe0055d0640 1 -- start start 2023-10-16T14:36:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fe0055d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec0047c0 0x7fe0001f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fe0055d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 0x7fe0000ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fe0055d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0001f2250 0x7fe00006c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fe0055d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00015d1c0 con 0x7fe0001f1df0 2023-10-16T14:36:18.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fe0055d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00015c640 con 0x7fe0001f2250 2023-10-16T14:36:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fe0055d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00015c920 con 0x7fdfec0047c0 2023-10-16T14:36:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.368+0000 7fdff77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 0x7fe0000ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 0x7fe0000ae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40220/0 (socket says 172.21.15.29:40220) 2023-10-16T14:36:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 -- 172.21.15.29:0/2043761817 learned_addr learned my addr 172.21.15.29:0/2043761817 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdffffff640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec0047c0 0x7fe0001f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fe004dcf640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0001f2250 0x7fe00006c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 -- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec0047c0 msgr2=0x7fe0001f5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec0047c0 0x7fe0001f5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 -- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0001f2250 msgr2=0x7fe00006c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0001f2250 0x7fe00006c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 -- 172.21.15.29:0/2043761817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00015cc00 con 0x7fe0001f1df0 2023-10-16T14:36:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.369+0000 7fdff77fe640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 0x7fe0000ae620 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fe0000cddd0 tx=0x7fdfe8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.370+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe803d8f0 con 0x7fe0001f1df0 2023-10-16T14:36:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.370+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe803da50 con 0x7fe0001f1df0 2023-10-16T14:36:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.370+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe803dde0 con 0x7fe0001f1df0 2023-10-16T14:36:18.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.370+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00015cee0 con 0x7fe0001f1df0 2023-10-16T14:36:18.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.370+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0001f6050 con 0x7fe0001f1df0 2023-10-16T14:36:18.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.371+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00004e9b0 con 0x7fe0001f1df0 2023-10-16T14:36:18.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.372+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdfe8031420 con 0x7fe0001f1df0 2023-10-16T14:36:18.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.373+0000 7fdffd7fa640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdfe006af90 0x7fdfe006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.373+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fdfe80abdf0 con 0x7fe0001f1df0 2023-10-16T14:36:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.374+0000 7fdffffff640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdfe006af90 0x7fdfe006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.375+0000 7fdffffff640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdfe006af90 0x7fdfe006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdff0003fc0 tx=0x7fdff0009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:18.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.378+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdfe8076bb0 con 0x7fe0001f1df0 2023-10-16T14:36:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.574+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe00015d0d0 con 0x7fe0001f1df0 2023-10-16T14:36:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.576+0000 7fdffd7fa640 1 -- 172.21.15.29:0/2043761817 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdfe804a0a0 con 0x7fe0001f1df0 2023-10-16T14:36:18.580 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:36:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdfe006af90 msgr2=0x7fdfe006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdfe006af90 0x7fdfe006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdff0003fc0 tx=0x7fdff0009040 comp rx=0 tx=0).stop 2023-10-16T14:36:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 msgr2=0x7fe0000ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 0x7fe0000ae620 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fe0000cddd0 tx=0x7fdfe8004730 comp rx=0 tx=0).stop 2023-10-16T14:36:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 shutdown_connections 2023-10-16T14:36:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0001f1df0 0x7fe0000ae620 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdfe006af90 0x7fdfe006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0001f2250 0x7fe00006c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 --2- 172.21.15.29:0/2043761817 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdfec0047c0 0x7fe0001f5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.578+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 >> 172.21.15.29:0/2043761817 conn(0x7fe0000b23b0 msgr2=0x7fe0000b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.579+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 shutdown_connections 2023-10-16T14:36:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.579+0000 7fe0055d0640 1 -- 172.21.15.29:0/2043761817 wait complete. 2023-10-16T14:36:18.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:36:18.608 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:36:18.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T14:36:18.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.865+0000 7f697db0a640 1 Processor -- start 2023-10-16T14:36:18.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.865+0000 7f697db0a640 1 -- start start 2023-10-16T14:36:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f697db0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f697815dde0 0x7f69780709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f697db0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6978160a90 con 0x7f6978070f20 2023-10-16T14:36:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f697db0a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69780cd900 con 0x7f697815c9f0 2023-10-16T14:36:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f697db0a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6978071350 con 0x7f697815dde0 2023-10-16T14:36:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f6977fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f697815dde0 0x7f69780709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f697d309640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6978070f20 0x7f69780ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41748/0 (socket says 172.21.15.29:41748) 2023-10-16T14:36:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.867+0000 7f697d309640 1 -- 172.21.15.29:0/605672049 learned_addr learned my addr 172.21.15.29:0/605672049 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:18.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.868+0000 7f6977fff640 1 -- 172.21.15.29:0/605672049 >> v1:172.21.15.170:6789/0 conn(0x7f697815c9f0 legacy=0x7f69780c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.868+0000 7f6977fff640 1 -- 172.21.15.29:0/605672049 >> v1:172.21.15.29:6789/0 conn(0x7f6978070f20 legacy=0x7f69780ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.868+0000 7f6977fff640 1 -- 172.21.15.29:0/605672049 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69780cdbe0 con 0x7f697815dde0 2023-10-16T14:36:18.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.868+0000 7f69777fe640 1 -- 172.21.15.29:0/605672049 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385392017 0 0) 0x7f6978160a90 con 0x7f6978070f20 2023-10-16T14:36:18.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.869+0000 7f6977fff640 1 --2- 172.21.15.29:0/605672049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815dde0 0x7f69780709e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f69640097a0 tx=0x7f696402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857363d49f72ed5b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:18.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.869+0000 7f69777fe640 1 -- 172.21.15.29:0/605672049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964030610 con 0x7f697815dde0 2023-10-16T14:36:18.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.869+0000 7f69777fe640 1 -- 172.21.15.29:0/605672049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6964030770 con 0x7f697815dde0 2023-10-16T14:36:18.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.869+0000 7f69777fe640 1 -- 172.21.15.29:0/605672049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964030b00 con 0x7f697815dde0 2023-10-16T14:36:18.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 -- 172.21.15.29:0/605672049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815dde0 msgr2=0x7f69780709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 --2- 172.21.15.29:0/605672049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815dde0 0x7f69780709e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f69640097a0 tx=0x7f696402b960 comp rx=0 tx=0).stop 2023-10-16T14:36:18.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 -- 172.21.15.29:0/605672049 shutdown_connections 2023-10-16T14:36:18.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 --2- 172.21.15.29:0/605672049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815dde0 0x7f69780709e0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 -- 172.21.15.29:0/605672049 >> 172.21.15.29:0/605672049 conn(0x7f69780ce6a0 msgr2=0x7f69780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:18.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 -- 172.21.15.29:0/605672049 shutdown_connections 2023-10-16T14:36:18.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.870+0000 7f697db0a640 1 -- 172.21.15.29:0/605672049 wait complete. 2023-10-16T14:36:18.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.872+0000 7f697db0a640 1 Processor -- start 2023-10-16T14:36:18.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.872+0000 7f697db0a640 1 -- start start 2023-10-16T14:36:18.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f697db0a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6978070f20 0x7f6978157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f697db0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 0x7f69780c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f697db0a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815dde0 0x7f69780c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f697db0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69780afb80 con 0x7f697815c9f0 2023-10-16T14:36:18.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f697db0a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697806b060 con 0x7f6978070f20 2023-10-16T14:36:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f697db0a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697806b340 con 0x7f697815dde0 2023-10-16T14:36:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f6977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 0x7f69780c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f6977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 0x7f69780c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40232/0 (socket says 172.21.15.29:40232) 2023-10-16T14:36:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.873+0000 7f6977fff640 1 -- 172.21.15.29:0/3001159738 learned_addr learned my addr 172.21.15.29:0/3001159738 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:18.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f697cb08640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6978070f20 0x7f6978157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f6977fff640 1 -- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815dde0 msgr2=0x7f69780c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f697d309640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815dde0 0x7f69780c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f6977fff640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815dde0 0x7f69780c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f6977fff640 1 -- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6978070f20 msgr2=0x7f6978157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:18.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f6977fff640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6978070f20 0x7f6978157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:18.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f6977fff640 1 -- 172.21.15.29:0/3001159738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964009400 con 0x7f697815c9f0 2023-10-16T14:36:18.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f697cb08640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6978070f20 0x7f6978157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:18.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f697d309640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815dde0 0x7f69780c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:18.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f6977fff640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 0x7f69780c43f0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f69640098c0 tx=0x7f6964004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:18.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696403e040 con 0x7f697815c9f0 2023-10-16T14:36:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6964039cb0 con 0x7f697815c9f0 2023-10-16T14:36:18.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964044630 con 0x7f697815c9f0 2023-10-16T14:36:18.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.874+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697806b620 con 0x7f697815c9f0 2023-10-16T14:36:18.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.875+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69780c3750 con 0x7f697815c9f0 2023-10-16T14:36:18.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.876+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697806bb40 con 0x7f697815c9f0 2023-10-16T14:36:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.882+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6964043390 con 0x7f697815c9f0 2023-10-16T14:36:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.883+0000 7f69757fa640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f695c06af90 0x7f695c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.883+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f69640abd30 con 0x7f697815c9f0 2023-10-16T14:36:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.884+0000 7f697cb08640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f695c06af90 0x7f695c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.885+0000 7f697cb08640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f695c06af90 0x7f695c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f696c000f60 tx=0x7f696c009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:18.889+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f697806bb40 con 0x7f697815c9f0 2023-10-16T14:36:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.148+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f697806bb40 con 0x7f697815c9f0 2023-10-16T14:36:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.651+0000 7f69757fa640 1 -- 172.21.15.29:0/3001159738 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v757) v1 ==== 163+0+0 (secure 0 0 0) 0x7f697806bb40 con 0x7f697815c9f0 2023-10-16T14:36:19.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f695c06af90 msgr2=0x7f695c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:19.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f695c06af90 0x7f695c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f696c000f60 tx=0x7f696c009040 comp rx=0 tx=0).stop 2023-10-16T14:36:19.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 msgr2=0x7f69780c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 0x7f69780c43f0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f69640098c0 tx=0x7f6964004730 comp rx=0 tx=0).stop 2023-10-16T14:36:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 shutdown_connections 2023-10-16T14:36:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f697815c9f0 0x7f69780c43f0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f695c06af90 0x7f695c06d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815dde0 0x7f69780c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 --2- 172.21.15.29:0/3001159738 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6978070f20 0x7f6978157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 >> 172.21.15.29:0/3001159738 conn(0x7f69780ce6a0 msgr2=0x7f69780b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 shutdown_connections 2023-10-16T14:36:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.654+0000 7f697db0a640 1 -- 172.21.15.29:0/3001159738 wait complete. 2023-10-16T14:36:19.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:36:19.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.911+0000 7f8907275640 1 Processor -- start 2023-10-16T14:36:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.911+0000 7f8907275640 1 -- start start 2023-10-16T14:36:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.912+0000 7f8907275640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f890006c040 0x7f890006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.912+0000 7f8907275640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f89000ad970 0x7f890015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.912+0000 7f8907275640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890006c960 con 0x7f890015ebc0 2023-10-16T14:36:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.912+0000 7f8907275640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890015b9d0 con 0x7f89000ad970 2023-10-16T14:36:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.912+0000 7f8907275640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890015bcb0 con 0x7f890006c040 2023-10-16T14:36:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f89000ad970 0x7f890015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f89000ad970 0x7f890015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40248/0 (socket says 172.21.15.29:40248) 2023-10-16T14:36:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 -- 172.21.15.29:0/2621914733 learned_addr learned my addr 172.21.15.29:0/2621914733 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8906273640 1 --2- 172.21.15.29:0/2621914733 >> v2:172.21.15.170:3300/0 conn(0x7f890006c040 0x7f890006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 -- 172.21.15.29:0/2621914733 >> v2:172.21.15.170:3300/0 conn(0x7f890006c040 msgr2=0x7f890006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 --2- 172.21.15.29:0/2621914733 >> v2:172.21.15.170:3300/0 conn(0x7f890006c040 0x7f890006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 -- 172.21.15.29:0/2621914733 >> v1:172.21.15.170:6789/0 conn(0x7f890015ebc0 legacy=0x7f89000cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.913+0000 7f8905a72640 1 -- 172.21.15.29:0/2621914733 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890015bf90 con 0x7f89000ad970 2023-10-16T14:36:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.915+0000 7f8905a72640 1 --2- 172.21.15.29:0/2621914733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89000ad970 0x7f890015e680 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f89000cf350 tx=0x7f88f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa924413a720644a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.915+0000 7f8905271640 1 -- 172.21.15.29:0/2621914733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0031870 con 0x7f89000ad970 2023-10-16T14:36:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.915+0000 7f8905271640 1 -- 172.21.15.29:0/2621914733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f00319d0 con 0x7f89000ad970 2023-10-16T14:36:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.915+0000 7f8905271640 1 -- 172.21.15.29:0/2621914733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0031d60 con 0x7f89000ad970 2023-10-16T14:36:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.915+0000 7f8907275640 1 -- 172.21.15.29:0/2621914733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89000ad970 msgr2=0x7f890015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.915+0000 7f8907275640 1 --2- 172.21.15.29:0/2621914733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89000ad970 0x7f890015e680 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f89000cf350 tx=0x7f88f002b910 comp rx=0 tx=0).stop 2023-10-16T14:36:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.916+0000 7f8907275640 1 -- 172.21.15.29:0/2621914733 shutdown_connections 2023-10-16T14:36:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.916+0000 7f8907275640 1 --2- 172.21.15.29:0/2621914733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89000ad970 0x7f890015e680 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.916+0000 7f8907275640 1 --2- 172.21.15.29:0/2621914733 >> v2:172.21.15.170:3300/0 conn(0x7f890006c040 0x7f890006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.916+0000 7f8907275640 1 -- 172.21.15.29:0/2621914733 >> 172.21.15.29:0/2621914733 conn(0x7f89000ce6a0 msgr2=0x7f89000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.916+0000 7f8907275640 1 -- 172.21.15.29:0/2621914733 shutdown_connections 2023-10-16T14:36:19.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.916+0000 7f8907275640 1 -- 172.21.15.29:0/2621914733 wait complete. 2023-10-16T14:36:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.918+0000 7f8907275640 1 Processor -- start 2023-10-16T14:36:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.918+0000 7f8907275640 1 -- start start 2023-10-16T14:36:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8907275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 0x7f89000c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8907275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89000ad970 0x7f89000c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8907275640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f890015ebc0 0x7f89000c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8907275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001f1b50 con 0x7f890006c040 2023-10-16T14:36:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8907275640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c4e80 con 0x7f890015ebc0 2023-10-16T14:36:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8907275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000b4570 con 0x7f89000ad970 2023-10-16T14:36:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906273640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 0x7f89000c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906273640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 0x7f89000c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40258/0 (socket says 172.21.15.29:40258) 2023-10-16T14:36:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906273640 1 -- 172.21.15.29:0/3380028086 learned_addr learned my addr 172.21.15.29:0/3380028086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906273640 1 -- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89000ad970 msgr2=0x7f89000c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906273640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89000ad970 0x7f89000c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906273640 1 -- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f890015ebc0 msgr2=0x7f89000c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.920+0000 7f8906a74640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f890015ebc0 0x7f89000c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f8906273640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f890015ebc0 0x7f89000c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f8906273640 1 -- 172.21.15.29:0/3380028086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00094d0 con 0x7f890006c040 2023-10-16T14:36:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f8906273640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 0x7f89000c4940 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00d620 tx=0x7f88fc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc00b620 con 0x7f890006c040 2023-10-16T14:36:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc00b780 con 0x7f890006c040 2023-10-16T14:36:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc00bb30 con 0x7f890006c040 2023-10-16T14:36:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89000b4290 con 0x7f890006c040 2023-10-16T14:36:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.921+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89001f17e0 con 0x7f890006c040 2023-10-16T14:36:19.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.923+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89001f1970 con 0x7f890006c040 2023-10-16T14:36:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.924+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88fc00bc90 con 0x7f890006c040 2023-10-16T14:36:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.925+0000 7f88f6ffd640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f88e406af90 0x7f88e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.925+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f88fc089d80 con 0x7f890006c040 2023-10-16T14:36:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.925+0000 7f8905a72640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f88e406af90 0x7f88e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.927+0000 7f8905a72640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f88e406af90 0x7f88e406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88f0036000 tx=0x7f88f0030000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:19.929+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88fc054c50 con 0x7f890006c040 2023-10-16T14:36:20.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.126+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f890006b330 con 0x7f890006c040 2023-10-16T14:36:20.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.132+0000 7f88f6ffd640 1 -- 172.21.15.29:0/3380028086 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f88fc058270 con 0x7f890006c040 2023-10-16T14:36:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f88e406af90 msgr2=0x7f88e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f88e406af90 0x7f88e406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88f0036000 tx=0x7f88f0030000 comp rx=0 tx=0).stop 2023-10-16T14:36:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 msgr2=0x7f89000c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 0x7f89000c4940 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00d620 tx=0x7f88fc00dae0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 shutdown_connections 2023-10-16T14:36:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f88e406af90 0x7f88e406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f890015ebc0 0x7f89000c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f89000ad970 0x7f89000c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 --2- 172.21.15.29:0/3380028086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f890006c040 0x7f89000c4940 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.135+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 >> 172.21.15.29:0/3380028086 conn(0x7f89000ce6a0 msgr2=0x7f89000cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.136+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 shutdown_connections 2023-10-16T14:36:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.136+0000 7f8907275640 1 -- 172.21.15.29:0/3380028086 wait complete. 2023-10-16T14:36:20.161 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:36:20.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:36:20.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:36:20.394 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:36:20.394 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:20.395 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:20.458 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:36:20.459 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:36:20.459 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:20.519 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:36:20.520 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:20.520 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:20.589 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:36:20.590 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:36:20.590 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:20.652 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:36:20.653 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:36:20.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:20.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.888+0000 7f25b2275640 1 Processor -- start 2023-10-16T14:36:20.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.889+0000 7f25b2275640 1 -- start start 2023-10-16T14:36:20.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.890+0000 7f25b2275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 0x7f25ac070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.890+0000 7f25b2275640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0ae1a0 con 0x7f25ac160c80 2023-10-16T14:36:20.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25b2275640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0ae480 con 0x7f25ac0adc00 2023-10-16T14:36:20.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25b2275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0ae760 con 0x7f25ac0ad7a0 2023-10-16T14:36:20.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25b1273640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25ac160c80 0x7f25ac161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41752/0 (socket says 172.21.15.29:41752) 2023-10-16T14:36:20.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25b1273640 1 -- 172.21.15.29:0/442059374 learned_addr learned my addr 172.21.15.29:0/442059374 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:20.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25b0a72640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 0x7f25ac070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174678313 0 0) 0x7f25ac0ae1a0 con 0x7f25ac160c80 2023-10-16T14:36:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.891+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003520 con 0x7f25ac160c80 2023-10-16T14:36:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544391733 0 0) 0x7f2598003520 con 0x7f25ac160c80 2023-10-16T14:36:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 msgr2=0x7f25ac070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 0x7f25ac070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 >> v1:172.21.15.170:6789/0 conn(0x7f25ac0adc00 legacy=0x7f25ac140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac0aea40 con 0x7f25ac160c80 2023-10-16T14:36:20.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f25a8001670 con 0x7f25ac160c80 2023-10-16T14:36:20.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 >> v1:172.21.15.29:6789/0 conn(0x7f25ac160c80 legacy=0x7f25ac161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25980039a0 0x7f2598003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980047c0 0x7f2598004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 0x7f2598005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0ae760 con 0x7f25980039a0 2023-10-16T14:36:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259800ba30 con 0x7f25980047c0 2023-10-16T14:36:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259800bbf0 con 0x7f25ac0ad7a0 2023-10-16T14:36:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.892+0000 7f25b1273640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25980039a0 0x7f2598003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b0a72640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980047c0 0x7f2598004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b1a74640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 0x7f2598005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b1273640 1 -- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 msgr2=0x7f2598005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b1273640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac0ad7a0 0x7f2598005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b1273640 1 -- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980047c0 msgr2=0x7f2598004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b1273640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980047c0 0x7f2598004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.893+0000 7f25b1273640 1 -- 172.21.15.29:0/442059374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598003520 con 0x7f25980039a0 2023-10-16T14:36:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.894+0000 7f25b1273640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25980039a0 0x7f2598003d80 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f25a800c160 tx=0x7f25980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0b784303dbce48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.894+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a8005ad0 con 0x7f25980039a0 2023-10-16T14:36:20.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a8005c30 con 0x7f25980039a0 2023-10-16T14:36:20.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25a3fff640 1 -- 172.21.15.29:0/442059374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a802c7c0 con 0x7f25980039a0 2023-10-16T14:36:20.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b2275640 1 -- 172.21.15.29:0/442059374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25980039a0 msgr2=0x7f2598003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b2275640 1 --2- 172.21.15.29:0/442059374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25980039a0 0x7f2598003d80 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f25a800c160 tx=0x7f25980057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b1a74640 1 -- 172.21.15.29:0/442059374 reap_dead start 2023-10-16T14:36:20.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b2275640 1 -- 172.21.15.29:0/442059374 shutdown_connections 2023-10-16T14:36:20.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b2275640 1 -- 172.21.15.29:0/442059374 >> 172.21.15.29:0/442059374 conn(0x7f25ac0cc230 msgr2=0x7f25ac0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:20.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b2275640 1 -- 172.21.15.29:0/442059374 shutdown_connections 2023-10-16T14:36:20.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.895+0000 7f25b2275640 1 -- 172.21.15.29:0/442059374 wait complete. 2023-10-16T14:36:20.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.897+0000 7f25b2275640 1 Processor -- start 2023-10-16T14:36:20.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.897+0000 7f25b2275640 1 -- start start 2023-10-16T14:36:20.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.898+0000 7f25b2275640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980039a0 0x7f25ac144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.898+0000 7f25b2275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac14e0a0 0x7f25ac1f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.898+0000 7f25b2275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 0x7f25ac06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.898+0000 7f25b2275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac144a80 con 0x7f25ac1f2a50 2023-10-16T14:36:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.898+0000 7f25b2275640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac145450 con 0x7f25980039a0 2023-10-16T14:36:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.898+0000 7f25b2275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac143e70 con 0x7f25ac14e0a0 2023-10-16T14:36:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 0x7f25ac06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 0x7f25ac06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40276/0 (socket says 172.21.15.29:40276) 2023-10-16T14:36:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 -- 172.21.15.29:0/2549004600 learned_addr learned my addr 172.21.15.29:0/2549004600 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1273640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980039a0 0x7f25ac144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b0a72640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac14e0a0 0x7f25ac1f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 -- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac14e0a0 msgr2=0x7f25ac1f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac14e0a0 0x7f25ac1f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 -- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980039a0 msgr2=0x7f25ac144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980039a0 0x7f25ac144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 -- 172.21.15.29:0/2549004600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac144150 con 0x7f25ac1f2a50 2023-10-16T14:36:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.899+0000 7f25b1a74640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 0x7f25ac06b0d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f25ac073a70 tx=0x7f25a400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.900+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4003970 con 0x7f25ac1f2a50 2023-10-16T14:36:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.900+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a4003af0 con 0x7f25ac1f2a50 2023-10-16T14:36:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.900+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4003ea0 con 0x7f25ac1f2a50 2023-10-16T14:36:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.900+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac144430 con 0x7f25ac1f2a50 2023-10-16T14:36:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.900+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac14e520 con 0x7f25ac1f2a50 2023-10-16T14:36:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.901+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac04e9b0 con 0x7f25ac1f2a50 2023-10-16T14:36:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.902+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f25a40050d0 con 0x7f25ac1f2a50 2023-10-16T14:36:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.903+0000 7f25a1ffb640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f259006af90 0x7f259006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.903+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f25a4089720 con 0x7f25ac1f2a50 2023-10-16T14:36:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.904+0000 7f25b1273640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f259006af90 0x7f259006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.905+0000 7f25b1273640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f259006af90 0x7f259006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25a8034000 tx=0x7f25a8009030 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:20.908+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25a4058120 con 0x7f25ac1f2a50 2023-10-16T14:36:21.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.144+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25ac144bc0 con 0x7f25ac1f2a50 2023-10-16T14:36:21.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.145+0000 7f25a1ffb640 1 -- 172.21.15.29:0/2549004600 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f25a4057cb0 con 0x7f25ac1f2a50 2023-10-16T14:36:21.149 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:21.150 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6056,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1627263146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1627263146},{"type":"v1","addr":"172.21.15.170:6836","nonce":1627263146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6059,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1525538700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1525538700},{"type":"v1","addr":"172.21.15.170:6837","nonce":1525538700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2572195555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2572195555},{"type":"v1","addr":"172.21.15.29:6834","nonce":2572195555}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8169,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1948093472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1948093472},{"type":"v1","addr":"172.21.15.29:6835","nonce":1948093472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2023-10-16T14:36:21.150 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2023-10-16T14:36:21.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f259006af90 msgr2=0x7f259006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:21.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f259006af90 0x7f259006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25a8034000 tx=0x7f25a8009030 comp rx=0 tx=0).stop 2023-10-16T14:36:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 msgr2=0x7f25ac06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 0x7f25ac06b0d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f25ac073a70 tx=0x7f25a400eba0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 shutdown_connections 2023-10-16T14:36:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25ac1f2a50 0x7f25ac06b0d0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f259006af90 0x7f259006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25ac14e0a0 0x7f25ac1f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 --2- 172.21.15.29:0/2549004600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25980039a0 0x7f25ac144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.148+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 >> 172.21.15.29:0/2549004600 conn(0x7f25ac0cc230 msgr2=0x7f25ac0c5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:21.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.149+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 shutdown_connections 2023-10-16T14:36:21.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.149+0000 7f25b2275640 1 -- 172.21.15.29:0/2549004600 wait complete. 2023-10-16T14:36:21.171 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:36:21.172 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:36:21.172 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:36:21.173 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:36:21.173 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:36:21.173 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:36:21.174 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:36:21.174 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:36:21.174 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:36:21.175 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:36:21.175 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:36:21.175 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:36:21.176 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:36:21.176 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:36:21.176 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:36:21.177 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:36:21.191 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:36:21.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:36:21.193 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:36:21.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:36:21.199 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:36:21.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:36:21.202 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:36:21.203 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:36:21.241 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:36:21.242 DEBUG:teuthology.parallel:result is None 2023-10-16T14:36:21.248 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:36:21.249 DEBUG:teuthology.parallel:result is None 2023-10-16T14:36:21.253 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:36:21.253 DEBUG:teuthology.parallel:result is None 2023-10-16T14:36:21.260 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:36:21.261 DEBUG:teuthology.parallel:result is None 2023-10-16T14:36:21.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:36:21.409 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:36:21.410 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:36:21.415 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:36:21.421 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:36:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.456+0000 7fdd00373640 1 Processor -- start 2023-10-16T14:36:21.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.457+0000 7fdd00373640 1 -- start start 2023-10-16T14:36:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdd00373640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf81540e0 0x7fdcf80adae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdd00373640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdcf80cca00 0x7fdcf80b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdd00373640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdcf80b0df0 0x7fdcf806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdd00373640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf80b1230 con 0x7fdcf80b0df0 2023-10-16T14:36:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdd00373640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf81544c0 con 0x7fdcf80cca00 2023-10-16T14:36:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdd00373640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80ccf00 con 0x7fdcf81540e0 2023-10-16T14:36:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdcfe910640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdcf80b0df0 0x7fdcf806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdcfe910640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdcf80b0df0 0x7fdcf806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40312/0 (socket says 172.21.15.29:40312) 2023-10-16T14:36:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdcfe910640 1 -- 172.21.15.29:0/2006710274 learned_addr learned my addr 172.21.15.29:0/2006710274 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdcfd90e640 1 --2- 172.21.15.29:0/2006710274 >> v2:172.21.15.170:3300/0 conn(0x7fdcf80cca00 0x7fdcf80b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.458+0000 7fdcfe10f640 1 --2- 172.21.15.29:0/2006710274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf81540e0 0x7fdcf80adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.459+0000 7fdcfe910640 1 -- 172.21.15.29:0/2006710274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf81540e0 msgr2=0x7fdcf80adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.459+0000 7fdcfe910640 1 --2- 172.21.15.29:0/2006710274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf81540e0 0x7fdcf80adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.459+0000 7fdcfe910640 1 -- 172.21.15.29:0/2006710274 >> v2:172.21.15.170:3300/0 conn(0x7fdcf80cca00 msgr2=0x7fdcf80b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.459+0000 7fdcfe910640 1 --2- 172.21.15.29:0/2006710274 >> v2:172.21.15.170:3300/0 conn(0x7fdcf80cca00 0x7fdcf80b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.459+0000 7fdcfe910640 1 -- 172.21.15.29:0/2006710274 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80cd1e0 con 0x7fdcf80b0df0 2023-10-16T14:36:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.459+0000 7fdcfd90e640 1 --2- 172.21.15.29:0/2006710274 >> v2:172.21.15.170:3300/0 conn(0x7fdcf80cca00 0x7fdcf80b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.460+0000 7fdcfe910640 1 --2- 172.21.15.29:0/2006710274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf806b9e0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fdcf806b930 tx=0x7fdcf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6f71da297d3cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.460+0000 7fdcfd10d640 1 -- 172.21.15.29:0/2006710274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf002ca70 con 0x7fdcf80b0df0 2023-10-16T14:36:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.460+0000 7fdcfd10d640 1 -- 172.21.15.29:0/2006710274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf002cbd0 con 0x7fdcf80b0df0 2023-10-16T14:36:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.460+0000 7fdcfd10d640 1 -- 172.21.15.29:0/2006710274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00343f0 con 0x7fdcf80b0df0 2023-10-16T14:36:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 -- 172.21.15.29:0/2006710274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 msgr2=0x7fdcf806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 --2- 172.21.15.29:0/2006710274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf806b9e0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fdcf806b930 tx=0x7fdcf002bed0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 -- 172.21.15.29:0/2006710274 shutdown_connections 2023-10-16T14:36:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 --2- 172.21.15.29:0/2006710274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf806b9e0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 --2- 172.21.15.29:0/2006710274 >> v2:172.21.15.170:3300/0 conn(0x7fdcf80cca00 0x7fdcf80b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 --2- 172.21.15.29:0/2006710274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf81540e0 0x7fdcf80adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 -- 172.21.15.29:0/2006710274 >> 172.21.15.29:0/2006710274 conn(0x7fdcf80cb8b0 msgr2=0x7fdcf80cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 -- 172.21.15.29:0/2006710274 shutdown_connections 2023-10-16T14:36:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.461+0000 7fdd00373640 1 -- 172.21.15.29:0/2006710274 wait complete. 2023-10-16T14:36:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.463+0000 7fdd00373640 1 Processor -- start 2023-10-16T14:36:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.463+0000 7fdd00373640 1 -- start start 2023-10-16T14:36:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.464+0000 7fdd00373640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf8151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.464+0000 7fdd00373640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf80cca00 0x7fdcf815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.464+0000 7fdd00373640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcf81540e0 0x7fdcf815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.464+0000 7fdd00373640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80c3530 con 0x7fdcf80b0df0 2023-10-16T14:36:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.464+0000 7fdd00373640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80c2940 con 0x7fdcf81540e0 2023-10-16T14:36:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.464+0000 7fdd00373640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80c2b90 con 0x7fdcf80cca00 2023-10-16T14:36:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf8151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf8151720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40318/0 (socket says 172.21.15.29:40318) 2023-10-16T14:36:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 -- 172.21.15.29:0/1822389323 learned_addr learned my addr 172.21.15.29:0/1822389323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe910640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcf81540e0 0x7fdcf815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfd90e640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf80cca00 0x7fdcf815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 -- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf80cca00 msgr2=0x7fdcf815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf80cca00 0x7fdcf815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 -- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcf81540e0 msgr2=0x7fdcf815fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcf81540e0 0x7fdcf815fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 -- 172.21.15.29:0/1822389323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf0009b30 con 0x7fdcf80b0df0 2023-10-16T14:36:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.465+0000 7fdcfe10f640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf8151720 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdcf400d620 tx=0x7fdcf400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.466+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf400b620 con 0x7fdcf80b0df0 2023-10-16T14:36:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.466+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf400b780 con 0x7fdcf80b0df0 2023-10-16T14:36:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.466+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf400bb30 con 0x7fdcf80b0df0 2023-10-16T14:36:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.466+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf80c2e70 con 0x7fdcf80b0df0 2023-10-16T14:36:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.466+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf80c3950 con 0x7fdcf80b0df0 2023-10-16T14:36:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.467+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf80c37a0 con 0x7fdcf80b0df0 2023-10-16T14:36:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.468+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdcf400bc90 con 0x7fdcf80b0df0 2023-10-16T14:36:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.469+0000 7fdceeffd640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdcdc06afc0 0x7fdcdc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.469+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fdcf4089d00 con 0x7fdcf80b0df0 2023-10-16T14:36:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.470+0000 7fdcfd90e640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdcdc06afc0 0x7fdcdc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.470+0000 7fdcfd90e640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdcdc06afc0 0x7fdcdc06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdce8006fd0 tx=0x7fdce8008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.473+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdcf4054bd0 con 0x7fdcf80b0df0 2023-10-16T14:36:21.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:21.678+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdcf804e160 con 0x7fdcf80b0df0 2023-10-16T14:36:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.692+0000 7fdceeffd640 1 -- 172.21.15.29:0/1822389323 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v138) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdcf40581f0 con 0x7fdcf80b0df0 2023-10-16T14:36:22.696 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:36:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdcdc06afc0 msgr2=0x7fdcdc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdcdc06afc0 0x7fdcdc06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdce8006fd0 tx=0x7fdce8008040 comp rx=0 tx=0).stop 2023-10-16T14:36:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 msgr2=0x7fdcf8151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf8151720 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fdcf400d620 tx=0x7fdcf400dae0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 shutdown_connections 2023-10-16T14:36:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fdcdc06afc0 0x7fdcdc06d480 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdcf81540e0 0x7fdcf815fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdcf80cca00 0x7fdcf815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.695+0000 7fdd00373640 1 --2- 172.21.15.29:0/1822389323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf80b0df0 0x7fdcf8151720 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.696+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 >> 172.21.15.29:0/1822389323 conn(0x7fdcf80cb8b0 msgr2=0x7fdcf80ca4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.696+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 shutdown_connections 2023-10-16T14:36:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.696+0000 7fdd00373640 1 -- 172.21.15.29:0/1822389323 wait complete. 2023-10-16T14:36:22.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:36:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.943+0000 7f5a819e4640 1 Processor -- start 2023-10-16T14:36:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.943+0000 7f5a819e4640 1 -- start start 2023-10-16T14:36:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.945+0000 7f5a819e4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 0x7f5a7c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.945+0000 7f5a819e4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 0x7f5a7c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.945+0000 7f5a819e4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7c15d150 con 0x7f5a7c0b8480 2023-10-16T14:36:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.945+0000 7f5a819e4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7c0d1180 con 0x7f5a7c15cd70 2023-10-16T14:36:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.945+0000 7f5a819e4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0d1460 con 0x7f5a7c0cded0 2023-10-16T14:36:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a811e3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 0x7f5a7c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a809e2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 0x7f5a7c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a809e2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 0x7f5a7c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:56172/0 (socket says 172.21.15.29:56172) 2023-10-16T14:36:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a811e3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 0x7f5a7c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:44214/0 (socket says 172.21.15.29:44214) 2023-10-16T14:36:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a809e2640 1 -- 172.21.15.29:0/3989613897 learned_addr learned my addr 172.21.15.29:0/3989613897 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a7b7fe640 1 -- 172.21.15.29:0/3989613897 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906862374 0 0) 0x7f5a7c15d150 con 0x7f5a7c0b8480 2023-10-16T14:36:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a7b7fe640 1 -- 172.21.15.29:0/3989613897 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003520 con 0x7f5a7c0b8480 2023-10-16T14:36:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a811e3640 1 -- 172.21.15.29:0/3989613897 >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 msgr2=0x7f5a7c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.946+0000 7f5a811e3640 1 --2- 172.21.15.29:0/3989613897 >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 0x7f5a7c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.947+0000 7f5a811e3640 1 -- 172.21.15.29:0/3989613897 >> v1:172.21.15.170:6789/0 conn(0x7f5a7c0b8480 legacy=0x7f5a7c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.947+0000 7f5a811e3640 1 -- 172.21.15.29:0/3989613897 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c0d1740 con 0x7f5a7c0cded0 2023-10-16T14:36:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.947+0000 7f5a809e2640 1 --2- 172.21.15.29:0/3989613897 >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 0x7f5a7c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.948+0000 7f5a811e3640 1 --2- 172.21.15.29:0/3989613897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 0x7f5a7c0b5c90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009d90 tx=0x7f5a7402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c283bd6c24065d3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.948+0000 7f5a7b7fe640 1 -- 172.21.15.29:0/3989613897 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7402cc60 con 0x7f5a7c0cded0 2023-10-16T14:36:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.948+0000 7f5a7b7fe640 1 -- 172.21.15.29:0/3989613897 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7402cdc0 con 0x7f5a7c0cded0 2023-10-16T14:36:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.948+0000 7f5a7b7fe640 1 -- 172.21.15.29:0/3989613897 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74034660 con 0x7f5a7c0cded0 2023-10-16T14:36:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 -- 172.21.15.29:0/3989613897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 msgr2=0x7f5a7c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 --2- 172.21.15.29:0/3989613897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 0x7f5a7c0b5c90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009d90 tx=0x7f5a7402beb0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 -- 172.21.15.29:0/3989613897 shutdown_connections 2023-10-16T14:36:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 --2- 172.21.15.29:0/3989613897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0cded0 0x7f5a7c0b5c90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 --2- 172.21.15.29:0/3989613897 >> v2:172.21.15.170:3300/0 conn(0x7f5a7c15cd70 0x7f5a7c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 -- 172.21.15.29:0/3989613897 >> 172.21.15.29:0/3989613897 conn(0x7f5a7c0cf570 msgr2=0x7f5a7c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 -- 172.21.15.29:0/3989613897 shutdown_connections 2023-10-16T14:36:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.949+0000 7f5a819e4640 1 -- 172.21.15.29:0/3989613897 wait complete. 2023-10-16T14:36:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.951+0000 7f5a819e4640 1 Processor -- start 2023-10-16T14:36:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.951+0000 7f5a819e4640 1 -- start start 2023-10-16T14:36:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.952+0000 7f5a819e4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0b8480 0x7f5a7c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.952+0000 7f5a819e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 0x7f5a7c151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.952+0000 7f5a819e4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a7c15cd70 0x7f5a7c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.952+0000 7f5a819e4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0cab10 con 0x7f5a7c0cded0 2023-10-16T14:36:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.952+0000 7f5a819e4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0cadb0 con 0x7f5a7c15cd70 2023-10-16T14:36:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.952+0000 7f5a819e4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0cb090 con 0x7f5a7c0b8480 2023-10-16T14:36:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 0x7f5a7c151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 0x7f5a7c151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40320/0 (socket says 172.21.15.29:40320) 2023-10-16T14:36:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 -- 172.21.15.29:0/2167247468 learned_addr learned my addr 172.21.15.29:0/2167247468 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a811e3640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a7c15cd70 0x7f5a7c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a809e2640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0b8480 0x7f5a7c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 -- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0b8480 msgr2=0x7f5a7c15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0b8480 0x7f5a7c15e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 -- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a7c15cd70 msgr2=0x7f5a7c0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a7c15cd70 0x7f5a7c0ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 -- 172.21.15.29:0/2167247468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700094d0 con 0x7f5a7c0cded0 2023-10-16T14:36:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a809e2640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0b8480 0x7f5a7c15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.953+0000 7f5a7bfff640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 0x7f5a7c151190 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800c500 tx=0x7f5a6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:22.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.954+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a680052d0 con 0x7f5a7c0cded0 2023-10-16T14:36:22.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.954+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a68005450 con 0x7f5a7c0cded0 2023-10-16T14:36:22.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.954+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a74009a70 con 0x7f5a7c0cded0 2023-10-16T14:36:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.954+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68012870 con 0x7f5a7c0cded0 2023-10-16T14:36:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.954+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c156080 con 0x7f5a7c0cded0 2023-10-16T14:36:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.955+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c0cb5b0 con 0x7f5a7c0cded0 2023-10-16T14:36:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.956+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a680129d0 con 0x7f5a7c0cded0 2023-10-16T14:36:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.957+0000 7f5a797fa640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a6006af90 0x7f5a6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:22.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.957+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f5a68089160 con 0x7f5a7c0cded0 2023-10-16T14:36:22.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.958+0000 7f5a809e2640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a6006af90 0x7f5a6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:22.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.959+0000 7f5a809e2640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a6006af90 0x7f5a6006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009bb0 tx=0x7f5a70008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:22.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:22.961+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a68057ef0 con 0x7f5a7c0cded0 2023-10-16T14:36:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.224+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5a7c04e9b0 con 0x7f5a7c0cded0 2023-10-16T14:36:23.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.225+0000 7f5a797fa640 1 -- 172.21.15.29:0/2167247468 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5a68057a80 con 0x7f5a7c0cded0 2023-10-16T14:36:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:23.229 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:36:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:36:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:36:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:36:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:36:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:36:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:36:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:36:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:36:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:36:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:36:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:36:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:36:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:36:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:36:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:36:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:36:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:36:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:36:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:36:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:36:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:36:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:36:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:36:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:36:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:36:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:36:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:36:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:36:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:36:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:36:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:36:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:36:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:36:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:36:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:36:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:36:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:36:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:36:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:36:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:36:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:36:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:36:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:36:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:36:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:36:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:36:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:36:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:36:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:36:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:36:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:36:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:36:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:36:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:36:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:36:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:36:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:36:23.261 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.261 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:36:23.261 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:36:23.261 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:36:23.261 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.261 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.262 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:36:23.262 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:36:23.262 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.263 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:36:23.263 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:36:23.263 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.263 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:36:23.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:23.264 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:36:23.264 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:36:23.264 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:36:23.264 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:36:23.265 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:36:23.265 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:36:23.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:23.265 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:36:23.265 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:36:23.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.227+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a6006af90 msgr2=0x7f5a6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.227+0000 7f5a819e4640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a6006af90 0x7f5a6006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009bb0 tx=0x7f5a70008040 comp rx=0 tx=0).stop 2023-10-16T14:36:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.227+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 msgr2=0x7f5a7c151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.227+0000 7f5a819e4640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 0x7f5a7c151190 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800c500 tx=0x7f5a6800c9c0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 shutdown_connections 2023-10-16T14:36:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a7c0cded0 0x7f5a7c151190 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a6006af90 0x7f5a6006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a7c15cd70 0x7f5a7c0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 --2- 172.21.15.29:0/2167247468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a7c0b8480 0x7f5a7c15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 >> 172.21.15.29:0/2167247468 conn(0x7f5a7c0cf570 msgr2=0x7f5a7c15bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 shutdown_connections 2023-10-16T14:36:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.228+0000 7f5a819e4640 1 -- 172.21.15.29:0/2167247468 wait complete. 2023-10-16T14:36:23.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.500+0000 7f040966e640 1 Processor -- start 2023-10-16T14:36:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.500+0000 7f040966e640 1 -- start start 2023-10-16T14:36:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.501+0000 7f040966e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 0x7f04040709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.501+0000 7f040966e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0404160a90 con 0x7f0404070f20 2023-10-16T14:36:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.501+0000 7f040966e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04040cd900 con 0x7f040415c9f0 2023-10-16T14:36:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.501+0000 7f040966e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04040cdec0 con 0x7f040415dde0 2023-10-16T14:36:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.502+0000 7f0408e6d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0404070f20 0x7f04040ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41778/0 (socket says 172.21.15.29:41778) 2023-10-16T14:36:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.502+0000 7f0408e6d640 1 -- 172.21.15.29:0/755287193 learned_addr learned my addr 172.21.15.29:0/755287193 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.502+0000 7f04037fe640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 0x7f04040709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.502+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022852267 0 0) 0x7f0404160a90 con 0x7f0404070f20 2023-10-16T14:36:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.502+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ec003520 con 0x7f0404070f20 2023-10-16T14:36:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 69889885 0 0) 0x7f03ec003520 con 0x7f0404070f20 2023-10-16T14:36:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 msgr2=0x7f04040709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 0x7f04040709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 >> v1:172.21.15.170:6789/0 conn(0x7f040415c9f0 legacy=0x7f04040c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040cdbe0 con 0x7f0404070f20 2023-10-16T14:36:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f03fc0019f0 con 0x7f0404070f20 2023-10-16T14:36:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 >> v1:172.21.15.29:6789/0 conn(0x7f0404070f20 legacy=0x7f04040ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f03ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 0x7f03ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ec004d00 0x7f03ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec00ba30 con 0x7f03ec0039a0 2023-10-16T14:36:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec00bb70 con 0x7f03ec004d00 2023-10-16T14:36:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec00bd30 con 0x7f040415dde0 2023-10-16T14:36:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.503+0000 7f0403fff640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f03ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0403fff640 1 -- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 msgr2=0x7f03ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0403fff640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f040415dde0 0x7f03ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0403fff640 1 -- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ec004d00 msgr2=0x7f03ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0408e6d640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ec004d00 0x7f03ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0403fff640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ec004d00 0x7f03ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0403fff640 1 -- 172.21.15.29:0/755287193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec003520 con 0x7f03ec0039a0 2023-10-16T14:36:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.504+0000 7f0408e6d640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03ec004d00 0x7f03ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.505+0000 7f0403fff640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f03ec003dc0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f03f4001720 tx=0x7f03ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a224719384aa02a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.505+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40052d0 con 0x7f03ec0039a0 2023-10-16T14:36:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.505+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f402e050 con 0x7f03ec0039a0 2023-10-16T14:36:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.505+0000 7f0402ffd640 1 -- 172.21.15.29:0/755287193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40035f0 con 0x7f03ec0039a0 2023-10-16T14:36:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f040966e640 1 -- 172.21.15.29:0/755287193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 msgr2=0x7f03ec003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f040966e640 1 --2- 172.21.15.29:0/755287193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f03ec003dc0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f03f4001720 tx=0x7f03ec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f0408e6d640 1 -- 172.21.15.29:0/755287193 reap_dead start 2023-10-16T14:36:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f040966e640 1 -- 172.21.15.29:0/755287193 shutdown_connections 2023-10-16T14:36:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f040966e640 1 -- 172.21.15.29:0/755287193 >> 172.21.15.29:0/755287193 conn(0x7f04040ce6a0 msgr2=0x7f04040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f040966e640 1 -- 172.21.15.29:0/755287193 shutdown_connections 2023-10-16T14:36:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.506+0000 7f040966e640 1 -- 172.21.15.29:0/755287193 wait complete. 2023-10-16T14:36:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.508+0000 7f040966e640 1 Processor -- start 2023-10-16T14:36:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.508+0000 7f040966e640 1 -- start start 2023-10-16T14:36:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f040966e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f04041f2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f040966e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04040c7990 0x7f04041f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f040966e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04040c5770 0x7f04040c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f040966e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04040c6130 con 0x7f03ec0039a0 2023-10-16T14:36:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f040966e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04040aeaf0 con 0x7f04040c5770 2023-10-16T14:36:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f040966e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04040b0ff0 con 0x7f04040c7990 2023-10-16T14:36:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f0403fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f04041f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f0403fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f04041f2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40334/0 (socket says 172.21.15.29:40334) 2023-10-16T14:36:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.509+0000 7f0403fff640 1 -- 172.21.15.29:0/2755039662 learned_addr learned my addr 172.21.15.29:0/2755039662 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f04037fe640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04040c7990 0x7f04041f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0403fff640 1 -- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04040c7990 msgr2=0x7f04041f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0403fff640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04040c7990 0x7f04041f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0403fff640 1 -- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04040c5770 msgr2=0x7f04040c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0403fff640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04040c5770 0x7f04040c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0403fff640 1 -- 172.21.15.29:0/2755039662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040c8100 con 0x7f03ec0039a0 2023-10-16T14:36:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0403fff640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f04041f2540 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03f4005040 tx=0x7f03f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40076c0 con 0x7f03ec0039a0 2023-10-16T14:36:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f4007a30 con 0x7f03ec0039a0 2023-10-16T14:36:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4006be0 con 0x7f03ec0039a0 2023-10-16T14:36:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.510+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04040c83e0 con 0x7f03ec0039a0 2023-10-16T14:36:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.511+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04040c89a0 con 0x7f03ec0039a0 2023-10-16T14:36:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.512+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0404161670 con 0x7f03ec0039a0 2023-10-16T14:36:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.513+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03f4006d40 con 0x7f03ec0039a0 2023-10-16T14:36:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.513+0000 7f0400ff9640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f03f006afc0 0x7f03f006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.514+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f03f40ab380 con 0x7f03ec0039a0 2023-10-16T14:36:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.514+0000 7f04037fe640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f03f006afc0 0x7f03f006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.515+0000 7f04037fe640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f03f006afc0 0x7f03f006d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f04040b0a20 tx=0x7f03f8008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.518+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03f40765c0 con 0x7f03ec0039a0 2023-10-16T14:36:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.712+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f040404e9b0 con 0x7f03ec0039a0 2023-10-16T14:36:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.713+0000 7f0400ff9640 1 -- 172.21.15.29:0/2755039662 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f03f4079be0 con 0x7f03ec0039a0 2023-10-16T14:36:23.718 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:23.718 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":138,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:22.689590+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f03f006afc0 msgr2=0x7f03f006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f03f006afc0 0x7f03f006d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f04040b0a20 tx=0x7f03f8008040 comp rx=0 tx=0).stop 2023-10-16T14:36:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 msgr2=0x7f04041f2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f04041f2540 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03f4005040 tx=0x7f03f4001120 comp rx=0 tx=0).stop 2023-10-16T14:36:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 shutdown_connections 2023-10-16T14:36:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f03f006afc0 0x7f03f006d480 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04040c5770 0x7f04040c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f04040c7990 0x7f04041f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 --2- 172.21.15.29:0/2755039662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03ec0039a0 0x7f04041f2540 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.716+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 >> 172.21.15.29:0/2755039662 conn(0x7f04040ce6a0 msgr2=0x7f04040c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.717+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 shutdown_connections 2023-10-16T14:36:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.717+0000 7f040966e640 1 -- 172.21.15.29:0/2755039662 wait complete. 2023-10-16T14:36:23.741 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:36:23.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:36:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.965+0000 7f9067275640 1 Processor -- start 2023-10-16T14:36:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.965+0000 7f9067275640 1 -- start start 2023-10-16T14:36:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9067275640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9060160530 0x7f90601609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9067275640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9060065ab0 0x7f9060160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9067275640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90600ac890 con 0x7f9060161470 2023-10-16T14:36:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9067275640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600ae800 con 0x7f9060160530 2023-10-16T14:36:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9067275640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600ae1f0 con 0x7f9060065ab0 2023-10-16T14:36:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9065011640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9060160530 0x7f90601609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9065011640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9060160530 0x7f90601609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36350/0 (socket says 172.21.15.29:36350) 2023-10-16T14:36:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.967+0000 7f9065011640 1 -- 172.21.15.29:0/4239287727 learned_addr learned my addr 172.21.15.29:0/4239287727 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.968+0000 7f9064810640 1 --2- 172.21.15.29:0/4239287727 >> v2:172.21.15.170:3300/0 conn(0x7f9060065ab0 0x7f9060160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.968+0000 7f9065011640 1 -- 172.21.15.29:0/4239287727 >> v2:172.21.15.170:3300/0 conn(0x7f9060065ab0 msgr2=0x7f9060160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.968+0000 7f9065011640 1 --2- 172.21.15.29:0/4239287727 >> v2:172.21.15.170:3300/0 conn(0x7f9060065ab0 0x7f9060160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.968+0000 7f9065011640 1 -- 172.21.15.29:0/4239287727 >> v1:172.21.15.29:6789/0 conn(0x7f9060161470 legacy=0x7f90600ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.968+0000 7f9065011640 1 -- 172.21.15.29:0/4239287727 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600acae0 con 0x7f9060160530 2023-10-16T14:36:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.968+0000 7f9064810640 1 --2- 172.21.15.29:0/4239287727 >> v2:172.21.15.170:3300/0 conn(0x7f9060065ab0 0x7f9060160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.969+0000 7f9065011640 1 --2- 172.21.15.29:0/4239287727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060160530 0x7f90601609f0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f906006f640 tx=0x7f905c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247d837bc690dced server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.969+0000 7f9057fff640 1 -- 172.21.15.29:0/4239287727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c02ca80 con 0x7f9060160530 2023-10-16T14:36:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.969+0000 7f9057fff640 1 -- 172.21.15.29:0/4239287727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c02cbe0 con 0x7f9060160530 2023-10-16T14:36:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.969+0000 7f9057fff640 1 -- 172.21.15.29:0/4239287727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c0343f0 con 0x7f9060160530 2023-10-16T14:36:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 -- 172.21.15.29:0/4239287727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060160530 msgr2=0x7f90601609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 --2- 172.21.15.29:0/4239287727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060160530 0x7f90601609f0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f906006f640 tx=0x7f905c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 -- 172.21.15.29:0/4239287727 shutdown_connections 2023-10-16T14:36:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 --2- 172.21.15.29:0/4239287727 >> v2:172.21.15.170:3300/0 conn(0x7f9060065ab0 0x7f9060160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 --2- 172.21.15.29:0/4239287727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060160530 0x7f90601609f0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 -- 172.21.15.29:0/4239287727 >> 172.21.15.29:0/4239287727 conn(0x7f906006bc10 msgr2=0x7f906006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 -- 172.21.15.29:0/4239287727 shutdown_connections 2023-10-16T14:36:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.970+0000 7f9067275640 1 -- 172.21.15.29:0/4239287727 wait complete. 2023-10-16T14:36:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.972+0000 7f9067275640 1 Processor -- start 2023-10-16T14:36:23.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.972+0000 7f9067275640 1 -- start start 2023-10-16T14:36:23.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9067275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9060065ab0 0x7f90601f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9067275640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9060160530 0x7f90600cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9067275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 0x7f90600afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9067275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90600c6070 con 0x7f9060161470 2023-10-16T14:36:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9067275640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906014b100 con 0x7f9060160530 2023-10-16T14:36:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9067275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90601f6c20 con 0x7f9060065ab0 2023-10-16T14:36:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9065812640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 0x7f90600afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9065812640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 0x7f90600afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36366/0 (socket says 172.21.15.29:36366) 2023-10-16T14:36:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.973+0000 7f9065812640 1 -- 172.21.15.29:0/1653607924 learned_addr learned my addr 172.21.15.29:0/1653607924 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065011640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9060065ab0 0x7f90601f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9064810640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9060160530 0x7f90600cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065812640 1 -- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9060065ab0 msgr2=0x7f90601f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065812640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9060065ab0 0x7f90601f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065812640 1 -- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9060160530 msgr2=0x7f90600cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065812640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9060160530 0x7f90600cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065812640 1 -- 172.21.15.29:0/1653607924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c009b30 con 0x7f9060161470 2023-10-16T14:36:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9064810640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9060160530 0x7f90600cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9065812640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 0x7f90600afc90 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f905800c400 tx=0x7f905800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.974+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580052d0 con 0x7f9060161470 2023-10-16T14:36:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.975+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9058005430 con 0x7f9060161470 2023-10-16T14:36:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.975+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058003810 con 0x7f9060161470 2023-10-16T14:36:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.975+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90600c5d90 con 0x7f9060161470 2023-10-16T14:36:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.975+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906014d8a0 con 0x7f9060161470 2023-10-16T14:36:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.976+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906014d420 con 0x7f9060161470 2023-10-16T14:36:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.977+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9058003970 con 0x7f9060161470 2023-10-16T14:36:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.978+0000 7f9055ffb640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f904406af90 0x7f904406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.978+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f9058088a10 con 0x7f9060161470 2023-10-16T14:36:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.978+0000 7f9065011640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f904406af90 0x7f904406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.980+0000 7f9065011640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f904406af90 0x7f904406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f905c035000 tx=0x7f905c030000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:23.982+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f906014d420 con 0x7f9060161470 2023-10-16T14:36:24.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.177+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f906014d420 con 0x7f9060161470 2023-10-16T14:36:24.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.178+0000 7f9055ffb640 1 -- 172.21.15.29:0/1653607924 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) v1 ==== 93+0+10 (secure 0 0 0) 0x7f906014d420 con 0x7f9060161470 2023-10-16T14:36:24.182 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:36:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f904406af90 msgr2=0x7f904406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f904406af90 0x7f904406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f905c035000 tx=0x7f905c030000 comp rx=0 tx=0).stop 2023-10-16T14:36:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 msgr2=0x7f90600afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 0x7f90600afc90 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f905800c400 tx=0x7f905800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 shutdown_connections 2023-10-16T14:36:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9060161470 0x7f90600afc90 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f904406af90 0x7f904406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9060160530 0x7f90600cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 --2- 172.21.15.29:0/1653607924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9060065ab0 0x7f90601f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 >> 172.21.15.29:0/1653607924 conn(0x7f906006bc10 msgr2=0x7f9060069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 shutdown_connections 2023-10-16T14:36:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.181+0000 7f9067275640 1 -- 172.21.15.29:0/1653607924 wait complete. 2023-10-16T14:36:24.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:36:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.420+0000 7f1b2e1b8640 1 Processor -- start 2023-10-16T14:36:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.420+0000 7f1b2e1b8640 1 -- start start 2023-10-16T14:36:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2e1b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b280d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2e1b8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b280b4c80 0x7f1b280b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2e1b8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b280d15a0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2e1b8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b280d1880 con 0x7f1b280b4c80 2023-10-16T14:36:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2e1b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b280d1b60 con 0x7f1b2815cbe0 2023-10-16T14:36:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2d9b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b280b4c80 0x7f1b280b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2d9b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b280b4c80 0x7f1b280b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36374/0 (socket says 172.21.15.29:36374) 2023-10-16T14:36:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/4100825412 learned_addr learned my addr 172.21.15.29:0/4100825412 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2c9b5640 1 --1- 172.21.15.29:0/4100825412 >> v1:172.21.15.29:6789/0 conn(0x7f1b2815cfc0 0x7f1b280b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46310/0 (socket says 172.21.15.29:46310) 2023-10-16T14:36:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.422+0000 7f1b2d1b6640 1 --2- 172.21.15.29:0/4100825412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b280d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.423+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/4100825412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 msgr2=0x7f1b280d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.423+0000 7f1b2d9b7640 1 --2- 172.21.15.29:0/4100825412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b280d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.423+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/4100825412 >> v1:172.21.15.29:6789/0 conn(0x7f1b2815cfc0 legacy=0x7f1b280b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.423+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/4100825412 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b280d1e40 con 0x7f1b280b4c80 2023-10-16T14:36:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.423+0000 7f1b1ffff640 1 -- 172.21.15.29:0/4100825412 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988117947 0 0) 0x7f1b280d15a0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.424+0000 7f1b2d9b7640 1 --2- 172.21.15.29:0/4100825412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b280b4c80 0x7f1b280b5c20 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1b20009e50 tx=0x7f1b2002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b060175227d52e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.424+0000 7f1b1ffff640 1 -- 172.21.15.29:0/4100825412 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2002cb90 con 0x7f1b280b4c80 2023-10-16T14:36:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.424+0000 7f1b1ffff640 1 -- 172.21.15.29:0/4100825412 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2002ccf0 con 0x7f1b280b4c80 2023-10-16T14:36:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.424+0000 7f1b1ffff640 1 -- 172.21.15.29:0/4100825412 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20034610 con 0x7f1b280b4c80 2023-10-16T14:36:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/4100825412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b280b4c80 msgr2=0x7f1b280b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/4100825412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b280b4c80 0x7f1b280b5c20 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1b20009e50 tx=0x7f1b2002bff0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/4100825412 shutdown_connections 2023-10-16T14:36:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/4100825412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b280b4c80 0x7f1b280b5c20 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/4100825412 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b280d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/4100825412 >> 172.21.15.29:0/4100825412 conn(0x7f1b280cf570 msgr2=0x7f1b280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/4100825412 shutdown_connections 2023-10-16T14:36:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.425+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/4100825412 wait complete. 2023-10-16T14:36:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.427+0000 7f1b2e1b8640 1 Processor -- start 2023-10-16T14:36:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.427+0000 7f1b2e1b8640 1 -- start start 2023-10-16T14:36:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2e1b8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b280b4c80 0x7f1b2815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2e1b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b28151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2e1b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 0x7f1b2815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2e1b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b28152a80 con 0x7f1b2815cfc0 2023-10-16T14:36:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2e1b8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b280ca600 con 0x7f1b280b4c80 2023-10-16T14:36:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2e1b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b280ca8e0 con 0x7f1b2815cbe0 2023-10-16T14:36:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2d9b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 0x7f1b2815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2d9b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 0x7f1b2815ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36380/0 (socket says 172.21.15.29:36380) 2023-10-16T14:36:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.428+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/2148387847 learned_addr learned my addr 172.21.15.29:0/2148387847 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d1b6640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b280b4c80 0x7f1b2815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2c9b5640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b28151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 msgr2=0x7f1b28151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d9b7640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b28151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b280b4c80 msgr2=0x7f1b2815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d9b7640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b280b4c80 0x7f1b2815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d9b7640 1 -- 172.21.15.29:0/2148387847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20009b30 con 0x7f1b2815cfc0 2023-10-16T14:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d1b6640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b280b4c80 0x7f1b2815fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:24.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2c9b5640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b28151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:24.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.429+0000 7f1b2d9b7640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 0x7f1b2815ef60 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1b20036040 tx=0x7f1b20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:24.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.430+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20009510 con 0x7f1b2815cfc0 2023-10-16T14:36:24.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.430+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2003e450 con 0x7f1b2815cfc0 2023-10-16T14:36:24.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.430+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2003e7e0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.430+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b28152cd0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.430+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b280ca2b0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.431+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b280caaa0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.432+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1b2003e940 con 0x7f1b2815cfc0 2023-10-16T14:36:24.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.433+0000 7f1b1dffb640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f1b0c06af90 0x7f1b0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.433+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f1b200abff0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.434+0000 7f1b2d1b6640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f1b0c06af90 0x7f1b0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.435+0000 7f1b2d1b6640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f1b0c06af90 0x7f1b0c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b24006fd0 tx=0x7f1b24008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:24.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.437+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1b200deca0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.633+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1b281531f0 con 0x7f1b2815cfc0 2023-10-16T14:36:24.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.634+0000 7f1b1dffb640 1 -- 172.21.15.29:0/2148387847 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v138) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1b20077260 con 0x7f1b2815cfc0 2023-10-16T14:36:24.638 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:24.639 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:36:24.639 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:24.639 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:36:24.639 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:36:24.640 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:24.640 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:36:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f1b0c06af90 msgr2=0x7f1b0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f1b0c06af90 0x7f1b0c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b24006fd0 tx=0x7f1b24008040 comp rx=0 tx=0).stop 2023-10-16T14:36:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 msgr2=0x7f1b2815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 0x7f1b2815ef60 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1b20036040 tx=0x7f1b20004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 shutdown_connections 2023-10-16T14:36:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b2815cfc0 0x7f1b2815ef60 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f1b0c06af90 0x7f1b0c06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b2815cbe0 0x7f1b28151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 --2- 172.21.15.29:0/2148387847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b280b4c80 0x7f1b2815fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.637+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 >> 172.21.15.29:0/2148387847 conn(0x7f1b280cf570 msgr2=0x7f1b280b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.638+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 shutdown_connections 2023-10-16T14:36:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.638+0000 7f1b2e1b8640 1 -- 172.21.15.29:0/2148387847 wait complete. 2023-10-16T14:36:24.659 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:36:24.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:36:24.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.885+0000 7f32c97f4640 1 Processor -- start 2023-10-16T14:36:24.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.885+0000 7f32c97f4640 1 -- start start 2023-10-16T14:36:24.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c97f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c40c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c97f4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32c40cc910 0x7f32c40ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c97f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40cc480 con 0x7f32c40acc70 2023-10-16T14:36:24.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c97f4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c40adfe0 con 0x7f32c40cc910 2023-10-16T14:36:24.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c97f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c415d8c0 con 0x7f32c40ac890 2023-10-16T14:36:24.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c8ff3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32c40cc910 0x7f32c40ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c8ff3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f32c40cc910 0x7f32c40ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36386/0 (socket says 172.21.15.29:36386) 2023-10-16T14:36:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.887+0000 7f32c8ff3640 1 -- 172.21.15.29:0/992708657 learned_addr learned my addr 172.21.15.29:0/992708657 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.888+0000 7f32c3fff640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c40c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.888+0000 7f32c8ff3640 1 -- 172.21.15.29:0/992708657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 msgr2=0x7f32c40c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.888+0000 7f32c8ff3640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c40c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.888+0000 7f32c8ff3640 1 -- 172.21.15.29:0/992708657 >> v1:172.21.15.170:6789/0 conn(0x7f32c40acc70 legacy=0x7f32c415d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.888+0000 7f32c8ff3640 1 -- 172.21.15.29:0/992708657 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c415dba0 con 0x7f32c40cc910 2023-10-16T14:36:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.888+0000 7f32c3fff640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c40c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.889+0000 7f32c8ff3640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40cc910 0x7f32c40ccda0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009870 tx=0x7f32bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbe03acc0d1024e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c2ffd640 1 -- 172.21.15.29:0/992708657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc031640 con 0x7f32c40cc910 2023-10-16T14:36:24.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c2ffd640 1 -- 172.21.15.29:0/992708657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc0317a0 con 0x7f32c40cc910 2023-10-16T14:36:24.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c2ffd640 1 -- 172.21.15.29:0/992708657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc031b30 con 0x7f32c40cc910 2023-10-16T14:36:24.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 -- 172.21.15.29:0/992708657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40cc910 msgr2=0x7f32c40ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40cc910 0x7f32c40ccda0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009870 tx=0x7f32bc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 -- 172.21.15.29:0/992708657 shutdown_connections 2023-10-16T14:36:24.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40cc910 0x7f32c40ccda0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 --2- 172.21.15.29:0/992708657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c40c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 -- 172.21.15.29:0/992708657 >> 172.21.15.29:0/992708657 conn(0x7f32c40ce820 msgr2=0x7f32c40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:24.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 -- 172.21.15.29:0/992708657 shutdown_connections 2023-10-16T14:36:24.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.890+0000 7f32c97f4640 1 -- 172.21.15.29:0/992708657 wait complete. 2023-10-16T14:36:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.892+0000 7f32c97f4640 1 Processor -- start 2023-10-16T14:36:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.892+0000 7f32c97f4640 1 -- start start 2023-10-16T14:36:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.893+0000 7f32c97f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.893+0000 7f32c97f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 0x7f32c414ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.893+0000 7f32c97f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c40cc910 0x7f32c40c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.893+0000 7f32c97f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40c5fc0 con 0x7f32c40acc70 2023-10-16T14:36:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.893+0000 7f32c97f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40c5c70 con 0x7f32c40cc910 2023-10-16T14:36:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.893+0000 7f32c97f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c415c240 con 0x7f32c40ac890 2023-10-16T14:36:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 0x7f32c414ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 0x7f32c414ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36402/0 (socket says 172.21.15.29:36402) 2023-10-16T14:36:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 -- 172.21.15.29:0/2329271997 learned_addr learned my addr 172.21.15.29:0/2329271997 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c3fff640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c8ff3640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c40cc910 0x7f32c40c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 -- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 msgr2=0x7f32c415b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c415b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 -- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c40cc910 msgr2=0x7f32c40c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c40cc910 0x7f32c40c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.894+0000 7f32c37fe640 1 -- 172.21.15.29:0/2329271997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc0094d0 con 0x7f32c40acc70 2023-10-16T14:36:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.895+0000 7f32c37fe640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 0x7f32c414ca70 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c430 tx=0x7f32ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.895+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac011790 con 0x7f32c40acc70 2023-10-16T14:36:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.895+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac0118f0 con 0x7f32c40acc70 2023-10-16T14:36:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.895+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac011ce0 con 0x7f32c40acc70 2023-10-16T14:36:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.895+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c415c520 con 0x7f32c40acc70 2023-10-16T14:36:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.895+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c415bea0 con 0x7f32c40acc70 2023-10-16T14:36:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.896+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c40c6170 con 0x7f32c40acc70 2023-10-16T14:36:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.897+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f32ac01c460 con 0x7f32c40acc70 2023-10-16T14:36:24.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.898+0000 7f32c0ff9640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f32a406afc0 0x7f32a406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:24.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.898+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f32ac089400 con 0x7f32c40acc70 2023-10-16T14:36:24.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.899+0000 7f32c3fff640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f32a406afc0 0x7f32a406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:24.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.900+0000 7f32c3fff640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f32a406afc0 0x7f32a406d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32b400d620 tx=0x7f32b4007810 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:24.902+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32c40c6170 con 0x7f32c40acc70 2023-10-16T14:36:25.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.096+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32c404e9b0 con 0x7f32c40acc70 2023-10-16T14:36:25.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.707+0000 7f32c0ff9640 1 -- 172.21.15.29:0/2329271997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v139) v1 ==== 138+0+0 (secure 0 0 0) 0x7f32ac0545a0 con 0x7f32c40acc70 2023-10-16T14:36:25.714 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:36:25.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f32a406afc0 msgr2=0x7f32a406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f32a406afc0 0x7f32a406d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32b400d620 tx=0x7f32b4007810 comp rx=0 tx=0).stop 2023-10-16T14:36:25.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 msgr2=0x7f32c414ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 0x7f32c414ca70 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c430 tx=0x7f32ac00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 shutdown_connections 2023-10-16T14:36:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32c40acc70 0x7f32c414ca70 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f32a406afc0 0x7f32a406d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32c40cc910 0x7f32c40c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 --2- 172.21.15.29:0/2329271997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32c40ac890 0x7f32c415b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 >> 172.21.15.29:0/2329271997 conn(0x7f32c40ce820 msgr2=0x7f32c4158570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 shutdown_connections 2023-10-16T14:36:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.711+0000 7f32c97f4640 1 -- 172.21.15.29:0/2329271997 wait complete. 2023-10-16T14:36:25.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:36:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.947+0000 7f86a1ff4640 1 Processor -- start 2023-10-16T14:36:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.947+0000 7f86a1ff4640 1 -- start start 2023-10-16T14:36:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a1ff4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 0x7f869c167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a1ff4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0cdbf0 con 0x7f869c15cb50 2023-10-16T14:36:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a1ff4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0cde90 con 0x7f869c0b1690 2023-10-16T14:36:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a1ff4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0ce170 con 0x7f869c0b4c80 2023-10-16T14:36:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a0ff2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f869c15cb50 0x7f869c161180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46320/0 (socket says 172.21.15.29:46320) 2023-10-16T14:36:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a0ff2640 1 -- 172.21.15.29:0/865011436 learned_addr learned my addr 172.21.15.29:0/865011436 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86a17f3640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 0x7f869c167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33134295 0 0) 0x7f869c0cdbf0 con 0x7f869c15cb50 2023-10-16T14:36:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.949+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8684003520 con 0x7f869c15cb50 2023-10-16T14:36:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159290572 0 0) 0x7f8684003520 con 0x7f869c15cb50 2023-10-16T14:36:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 msgr2=0x7f869c167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 0x7f869c167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 >> v1:172.21.15.170:6789/0 conn(0x7f869c0b1690 legacy=0x7f869c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0d0f30 con 0x7f869c15cb50 2023-10-16T14:36:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f86980019f0 con 0x7f869c15cb50 2023-10-16T14:36:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 >> v1:172.21.15.29:6789/0 conn(0x7f869c15cb50 legacy=0x7f869c161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 0x7f86840039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8684003f20 0x7f86840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8684004d00 0x7f8684005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0ce170 con 0x7f8684003f20 2023-10-16T14:36:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868400bbc0 con 0x7f8684004d00 2023-10-16T14:36:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868400bd80 con 0x7f869c0b4c80 2023-10-16T14:36:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.950+0000 7f8693fff640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8684003f20 0x7f86840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f86a0ff2640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 0x7f86840039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f86a17f3640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8684004d00 0x7f8684005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f8693fff640 1 -- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 msgr2=0x7f86840039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f8693fff640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c0b4c80 0x7f86840039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f8693fff640 1 -- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8684004d00 msgr2=0x7f8684005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f8693fff640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8684004d00 0x7f8684005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.951+0000 7f8693fff640 1 -- 172.21.15.29:0/865011436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684003520 con 0x7f8684003f20 2023-10-16T14:36:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.952+0000 7f8693fff640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8684003f20 0x7f86840047c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f868c001750 tx=0x7f86840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e0412bd22f779c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.952+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0052d0 con 0x7f8684003f20 2023-10-16T14:36:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c02e050 con 0x7f8684003f20 2023-10-16T14:36:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86937fe640 1 -- 172.21.15.29:0/865011436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c003830 con 0x7f8684003f20 2023-10-16T14:36:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a1ff4640 1 -- 172.21.15.29:0/865011436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8684003f20 msgr2=0x7f86840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/865011436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8684003f20 0x7f86840047c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f868c001750 tx=0x7f86840057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a17f3640 1 -- 172.21.15.29:0/865011436 reap_dead start 2023-10-16T14:36:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a1ff4640 1 -- 172.21.15.29:0/865011436 shutdown_connections 2023-10-16T14:36:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a1ff4640 1 -- 172.21.15.29:0/865011436 >> 172.21.15.29:0/865011436 conn(0x7f869c0cf570 msgr2=0x7f869c0cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a1ff4640 1 -- 172.21.15.29:0/865011436 shutdown_connections 2023-10-16T14:36:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.953+0000 7f86a1ff4640 1 -- 172.21.15.29:0/865011436 wait complete. 2023-10-16T14:36:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.955+0000 7f86a1ff4640 1 Processor -- start 2023-10-16T14:36:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.956+0000 7f86a1ff4640 1 -- start start 2023-10-16T14:36:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a1ff4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 0x7f869c15d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a1ff4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c155b20 0x7f869c15dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a1ff4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f869c0b30c0 0x7f869c150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a1ff4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c151be0 con 0x7f869c0b2ce0 2023-10-16T14:36:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a1ff4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c151ec0 con 0x7f869c0b30c0 2023-10-16T14:36:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a1ff4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1521a0 con 0x7f869c155b20 2023-10-16T14:36:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a0ff2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 0x7f869c15d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a0ff2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 0x7f869c15d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36430/0 (socket says 172.21.15.29:36430) 2023-10-16T14:36:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a0ff2640 1 -- 172.21.15.29:0/1051214348 learned_addr learned my addr 172.21.15.29:0/1051214348 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f8693fff640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c155b20 0x7f869c15dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.957+0000 7f86a17f3640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f869c0b30c0 0x7f869c150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a0ff2640 1 -- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c155b20 msgr2=0x7f869c15dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a0ff2640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c155b20 0x7f869c15dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a0ff2640 1 -- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f869c0b30c0 msgr2=0x7f869c150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a0ff2640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f869c0b30c0 0x7f869c150bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a0ff2640 1 -- 172.21.15.29:0/1051214348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c152480 con 0x7f869c0b2ce0 2023-10-16T14:36:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f8693fff640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c155b20 0x7f869c15dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a0ff2640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 0x7f869c15d690 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8698007c40 tx=0x7f869800e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869800edf0 con 0x7f869c0b2ce0 2023-10-16T14:36:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8698013420 con 0x7f869c0b2ce0 2023-10-16T14:36:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86980137d0 con 0x7f869c0b2ce0 2023-10-16T14:36:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.958+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c152760 con 0x7f869c0b2ce0 2023-10-16T14:36:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.959+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c151110 con 0x7f869c0b2ce0 2023-10-16T14:36:25.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.960+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c04e9b0 con 0x7f869c0b2ce0 2023-10-16T14:36:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.962+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8698013930 con 0x7f869c0b2ce0 2023-10-16T14:36:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.963+0000 7f86917fa640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f868806afc0 0x7f868806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.963+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f869808a810 con 0x7f869c0b2ce0 2023-10-16T14:36:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.964+0000 7f8693fff640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f868806afc0 0x7f868806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.965+0000 7f8693fff640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f868806afc0 0x7f868806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f868c033000 tx=0x7f868c009030 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:25.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:25.967+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8698055b10 con 0x7f869c0b2ce0 2023-10-16T14:36:26.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.160+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f869c0d2710 con 0x7f869c0b2ce0 2023-10-16T14:36:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.714+0000 7f86917fa640 1 -- 172.21.15.29:0/1051214348 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v140) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8698059130 con 0x7f869c0b2ce0 2023-10-16T14:36:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:36:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.717+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f868806afc0 msgr2=0x7f868806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.717+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f868806afc0 0x7f868806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f868c033000 tx=0x7f868c009030 comp rx=0 tx=0).stop 2023-10-16T14:36:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.717+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 msgr2=0x7f869c15d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.717+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 0x7f869c15d690 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8698007c40 tx=0x7f869800e4d0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.717+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 shutdown_connections 2023-10-16T14:36:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f868806afc0 0x7f868806d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f869c0b30c0 0x7f869c150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f869c155b20 0x7f869c15dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 --2- 172.21.15.29:0/1051214348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f869c0b2ce0 0x7f869c15d690 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 >> 172.21.15.29:0/1051214348 conn(0x7f869c0cf570 msgr2=0x7f869c15b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 shutdown_connections 2023-10-16T14:36:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.718+0000 7f86a1ff4640 1 -- 172.21.15.29:0/1051214348 wait complete. 2023-10-16T14:36:26.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:36:26.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.944+0000 7f2578f8a640 1 Processor -- start 2023-10-16T14:36:26.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.944+0000 7f2578f8a640 1 -- start start 2023-10-16T14:36:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2578f8a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25740caac0 0x7f25740ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2578f8a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740cb030 con 0x7f257406a4b0 2023-10-16T14:36:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2578f8a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740c91c0 con 0x7f257415ad70 2023-10-16T14:36:26.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2578f8a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25740c94a0 con 0x7f25740caac0 2023-10-16T14:36:26.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2572d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25740caac0 0x7f25740ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:26.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2572d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25740caac0 0x7f25740ad4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36434/0 (socket says 172.21.15.29:36434) 2023-10-16T14:36:26.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2571d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f257406a4b0 0x7f25740ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46324/0 (socket says 172.21.15.29:46324) 2023-10-16T14:36:26.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.946+0000 7f2572d9d640 1 -- 172.21.15.29:0/4112713009 learned_addr learned my addr 172.21.15.29:0/4112713009 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.947+0000 7f2572d9d640 1 -- 172.21.15.29:0/4112713009 >> v1:172.21.15.170:6789/0 conn(0x7f257415ad70 legacy=0x7f2574069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.947+0000 7f2572d9d640 1 -- 172.21.15.29:0/4112713009 >> v1:172.21.15.29:6789/0 conn(0x7f257406a4b0 legacy=0x7f25740ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.947+0000 7f2572d9d640 1 -- 172.21.15.29:0/4112713009 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740c9780 con 0x7f25740caac0 2023-10-16T14:36:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.948+0000 7f2572d9d640 1 --2- 172.21.15.29:0/4112713009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740ad4d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f256c009e50 tx=0x7f256c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcdcfe1c2e295db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.948+0000 7f257159a640 1 -- 172.21.15.29:0/4112713009 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c02cb10 con 0x7f25740caac0 2023-10-16T14:36:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.948+0000 7f257159a640 1 -- 172.21.15.29:0/4112713009 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c02cc70 con 0x7f25740caac0 2023-10-16T14:36:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.948+0000 7f257159a640 1 -- 172.21.15.29:0/4112713009 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0343f0 con 0x7f25740caac0 2023-10-16T14:36:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 -- 172.21.15.29:0/4112713009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 msgr2=0x7f25740ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 --2- 172.21.15.29:0/4112713009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740ad4d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f256c009e50 tx=0x7f256c02bf70 comp rx=0 tx=0).stop 2023-10-16T14:36:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 -- 172.21.15.29:0/4112713009 shutdown_connections 2023-10-16T14:36:26.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 --2- 172.21.15.29:0/4112713009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740ad4d0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 -- 172.21.15.29:0/4112713009 >> 172.21.15.29:0/4112713009 conn(0x7f25740ce9b0 msgr2=0x7f25740cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:26.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 -- 172.21.15.29:0/4112713009 shutdown_connections 2023-10-16T14:36:26.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.949+0000 7f2578f8a640 1 -- 172.21.15.29:0/4112713009 wait complete. 2023-10-16T14:36:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.951+0000 7f2578f8a640 1 Processor -- start 2023-10-16T14:36:26.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.951+0000 7f2578f8a640 1 -- start start 2023-10-16T14:36:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2578f8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f257406a4b0 0x7f2574158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2578f8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2578f8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f257415ad70 0x7f25740c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2578f8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25740c8ec0 con 0x7f25740caac0 2023-10-16T14:36:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2578f8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25741594c0 con 0x7f257415ad70 2023-10-16T14:36:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2578f8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257415da90 con 0x7f257406a4b0 2023-10-16T14:36:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.952+0000 7f2571d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36438/0 (socket says 172.21.15.29:36438) 2023-10-16T14:36:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 -- 172.21.15.29:0/1990867155 learned_addr learned my addr 172.21.15.29:0/1990867155 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2572d9d640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f257415ad70 0x7f25740c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:26.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f257259c640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f257406a4b0 0x7f2574158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:26.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 -- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f257406a4b0 msgr2=0x7f2574158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:26.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f257406a4b0 0x7f2574158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 -- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f257415ad70 msgr2=0x7f25740c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:26.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f257415ad70 0x7f25740c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 -- 172.21.15.29:0/1990867155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c009b30 con 0x7f25740caac0 2023-10-16T14:36:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.953+0000 7f2571d9b640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740c4630 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f255c00c4d0 tx=0x7f255c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.954+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c007480 con 0x7f25740caac0 2023-10-16T14:36:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.954+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c0075e0 con 0x7f25740caac0 2023-10-16T14:36:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.954+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0049e0 con 0x7f25740caac0 2023-10-16T14:36:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.954+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257415dd70 con 0x7f25740caac0 2023-10-16T14:36:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.954+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25740acc10 con 0x7f25740caac0 2023-10-16T14:36:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.955+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257404e160 con 0x7f25740caac0 2023-10-16T14:36:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.957+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f255c004b40 con 0x7f25740caac0 2023-10-16T14:36:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.958+0000 7f256affd640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f255406afe0 0x7f255406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.958+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f255c088bd0 con 0x7f25740caac0 2023-10-16T14:36:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.958+0000 7f257259c640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f255406afe0 0x7f255406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:26.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.960+0000 7f257259c640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f255406afe0 0x7f255406d4a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2564006fd0 tx=0x7f2564008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:26.963+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f255c053c40 con 0x7f25740caac0 2023-10-16T14:36:27.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.193+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f25740700e0 con 0x7f25740caac0 2023-10-16T14:36:27.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.766+0000 7f256affd640 1 -- 172.21.15.29:0/1990867155 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v62) v1 ==== 174+0+0 (secure 0 0 0) 0x7f255c057260 con 0x7f25740caac0 2023-10-16T14:36:27.774 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 16 and data pool 17 2023-10-16T14:36:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.779+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f255406afe0 msgr2=0x7f255406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.779+0000 7f2578f8a640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f255406afe0 0x7f255406d4a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2564006fd0 tx=0x7f2564008040 comp rx=0 tx=0).stop 2023-10-16T14:36:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.779+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 msgr2=0x7f25740c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.779+0000 7f2578f8a640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740c4630 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f255c00c4d0 tx=0x7f255c00c990 comp rx=0 tx=0).stop 2023-10-16T14:36:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.780+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 shutdown_connections 2023-10-16T14:36:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.780+0000 7f2578f8a640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25740caac0 0x7f25740c4630 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.780+0000 7f2578f8a640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f255406afe0 0x7f255406d4a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.780+0000 7f2578f8a640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f257415ad70 0x7f25740c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.780+0000 7f2578f8a640 1 --2- 172.21.15.29:0/1990867155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f257406a4b0 0x7f2574158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.780+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 >> 172.21.15.29:0/1990867155 conn(0x7f25740ce9b0 msgr2=0x7f25740ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.781+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 shutdown_connections 2023-10-16T14:36:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.781+0000 7f2578f8a640 1 -- 172.21.15.29:0/1990867155 wait complete. 2023-10-16T14:36:27.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.995+0000 7fe0370a9640 1 Processor -- start 2023-10-16T14:36:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.995+0000 7fe0370a9640 1 -- start start 2023-10-16T14:36:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.996+0000 7fe0370a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe03015aaa0 0x7fe03015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.996+0000 7fe0370a9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe03015ae80 0x7fe0300cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.996+0000 7fe0370a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe030070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.996+0000 7fe0370a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03006b810 con 0x7fe03015aaa0 2023-10-16T14:36:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.996+0000 7fe0370a9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03015ed60 con 0x7fe03015ae80 2023-10-16T14:36:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.996+0000 7fe0370a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0300707e0 con 0x7fe0300ccaa0 2023-10-16T14:36:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe03015aaa0 0x7fe03015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe03015aaa0 0x7fe03015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36446/0 (socket says 172.21.15.29:36446) 2023-10-16T14:36:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 -- 172.21.15.29:0/1038796627 learned_addr learned my addr 172.21.15.29:0/1038796627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 -- 172.21.15.29:0/1038796627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 msgr2=0x7fe030070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 --2- 172.21.15.29:0/1038796627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe030070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 -- 172.21.15.29:0/1038796627 >> v2:172.21.15.170:3300/0 conn(0x7fe03015ae80 msgr2=0x7fe0300cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe02ffff640 1 --2- 172.21.15.29:0/1038796627 >> v2:172.21.15.170:3300/0 conn(0x7fe03015ae80 0x7fe0300cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 --2- 172.21.15.29:0/1038796627 >> v2:172.21.15.170:3300/0 conn(0x7fe03015ae80 0x7fe0300cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe034e45640 1 -- 172.21.15.29:0/1038796627 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03015ef90 con 0x7fe03015aaa0 2023-10-16T14:36:28.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.997+0000 7fe02ffff640 1 --2- 172.21.15.29:0/1038796627 >> v2:172.21.15.170:3300/0 conn(0x7fe03015ae80 0x7fe0300cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:28.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.998+0000 7fe034e45640 1 --2- 172.21.15.29:0/1038796627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe03015e820 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe0300c59c0 tx=0x7fe02402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f027b14b3b366927 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe02f7fe640 1 -- 172.21.15.29:0/1038796627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024031860 con 0x7fe03015aaa0 2023-10-16T14:36:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe02f7fe640 1 -- 172.21.15.29:0/1038796627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0240319c0 con 0x7fe03015aaa0 2023-10-16T14:36:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe02f7fe640 1 -- 172.21.15.29:0/1038796627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024031d50 con 0x7fe03015aaa0 2023-10-16T14:36:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 -- 172.21.15.29:0/1038796627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 msgr2=0x7fe03015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 --2- 172.21.15.29:0/1038796627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe03015e820 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe0300c59c0 tx=0x7fe02402b900 comp rx=0 tx=0).stop 2023-10-16T14:36:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 -- 172.21.15.29:0/1038796627 shutdown_connections 2023-10-16T14:36:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 --2- 172.21.15.29:0/1038796627 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe030070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 --2- 172.21.15.29:0/1038796627 >> v2:172.21.15.170:3300/0 conn(0x7fe03015ae80 0x7fe0300cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 --2- 172.21.15.29:0/1038796627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe03015e820 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 -- 172.21.15.29:0/1038796627 >> 172.21.15.29:0/1038796627 conn(0x7fe0300cb410 msgr2=0x7fe0300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 -- 172.21.15.29:0/1038796627 shutdown_connections 2023-10-16T14:36:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:27.999+0000 7fe0370a9640 1 -- 172.21.15.29:0/1038796627 wait complete. 2023-10-16T14:36:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.001+0000 7fe0370a9640 1 Processor -- start 2023-10-16T14:36:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.001+0000 7fe0370a9640 1 -- start start 2023-10-16T14:36:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe0370a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe03014bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe0370a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe0300a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe0370a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe03015ae80 0x7fe0300a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe0370a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03014e940 con 0x7fe03015aaa0 2023-10-16T14:36:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe0370a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03014f470 con 0x7fe03015ae80 2023-10-16T14:36:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe0370a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03014dd30 con 0x7fe0300ccaa0 2023-10-16T14:36:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.002+0000 7fe02ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe0300a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe0300a8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36462/0 (socket says 172.21.15.29:36462) 2023-10-16T14:36:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 -- 172.21.15.29:0/3758550581 learned_addr learned my addr 172.21.15.29:0/3758550581 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe034e45640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe03014bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe035646640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe03015ae80 0x7fe0300a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 -- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 msgr2=0x7fe03014bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe03014bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 -- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe03015ae80 msgr2=0x7fe0300a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe03015ae80 0x7fe0300a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 -- 172.21.15.29:0/3758550581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240094d0 con 0x7fe03015aaa0 2023-10-16T14:36:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe035646640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe03015ae80 0x7fe0300a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.003+0000 7fe02ffff640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe0300a8d90 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c820 tx=0x7fe01c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.004+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c007a30 con 0x7fe03015aaa0 2023-10-16T14:36:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.004+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c007b90 con 0x7fe03015aaa0 2023-10-16T14:36:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.004+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0113f0 con 0x7fe03015aaa0 2023-10-16T14:36:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.004+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03014e010 con 0x7fe03015aaa0 2023-10-16T14:36:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.004+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03014e5d0 con 0x7fe03015aaa0 2023-10-16T14:36:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.005+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03004e160 con 0x7fe03015aaa0 2023-10-16T14:36:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.006+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe01c011550 con 0x7fe03015aaa0 2023-10-16T14:36:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.007+0000 7fe02d7fa640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe01406afc0 0x7fe01406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.008+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe01c089a10 con 0x7fe03015aaa0 2023-10-16T14:36:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.008+0000 7fe034e45640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe01406afc0 0x7fe01406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.009+0000 7fe034e45640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe01406afc0 0x7fe01406d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe024036000 tx=0x7fe02403c040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.012+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe01c054860 con 0x7fe03015aaa0 2023-10-16T14:36:28.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.197+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe03015e350 con 0x7fe03015aaa0 2023-10-16T14:36:28.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.198+0000 7fe02d7fa640 1 -- 172.21.15.29:0/3758550581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16584 (secure 0 0 0) 0x7fe01c01f090 con 0x7fe03015aaa0 2023-10-16T14:36:28.202 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:28.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:27.195862+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.200+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe01406afc0 msgr2=0x7fe01406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.200+0000 7fe0370a9640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe01406afc0 0x7fe01406d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe024036000 tx=0x7fe02403c040 comp rx=0 tx=0).stop 2023-10-16T14:36:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 msgr2=0x7fe0300a8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe0300a8d90 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c820 tx=0x7fe01c00cce0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 shutdown_connections 2023-10-16T14:36:28.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03015aaa0 0x7fe0300a8d90 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe01406afc0 0x7fe01406d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe03015ae80 0x7fe0300a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 --2- 172.21.15.29:0/3758550581 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0300ccaa0 0x7fe03014bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 >> 172.21.15.29:0/3758550581 conn(0x7fe0300cb410 msgr2=0x7fe03006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 shutdown_connections 2023-10-16T14:36:28.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.201+0000 7fe0370a9640 1 -- 172.21.15.29:0/3758550581 wait complete. 2023-10-16T14:36:28.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:28.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.440+0000 7f8d57fff640 1 Processor -- start 2023-10-16T14:36:28.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.441+0000 7f8d57fff640 1 -- start start 2023-10-16T14:36:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d57fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d580ad970 0x7f8d580cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d57fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5815b9d0 con 0x7f8d5806c040 2023-10-16T14:36:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d57fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5815bcb0 con 0x7f8d580ccfb0 2023-10-16T14:36:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d57fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5815bf90 con 0x7f8d580ad970 2023-10-16T14:36:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d580ad970 0x7f8d580cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d580ad970 0x7f8d580cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36470/0 (socket says 172.21.15.29:36470) 2023-10-16T14:36:28.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d4ffff640 1 -- 172.21.15.29:0/1771422038 learned_addr learned my addr 172.21.15.29:0/1771422038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:28.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d56ffd640 1 --1- 172.21.15.29:0/1771422038 >> v1:172.21.15.29:6789/0 conn(0x7f8d5806c040 0x7f8d5806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46338/0 (socket says 172.21.15.29:46338) 2023-10-16T14:36:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.443+0000 7f8d4ffff640 1 -- 172.21.15.29:0/1771422038 >> v1:172.21.15.170:6789/0 conn(0x7f8d580ccfb0 legacy=0x7f8d580cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.444+0000 7f8d4ffff640 1 -- 172.21.15.29:0/1771422038 >> v1:172.21.15.29:6789/0 conn(0x7f8d5806c040 legacy=0x7f8d5806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.444+0000 7f8d567fc640 1 -- 172.21.15.29:0/1771422038 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435577287 0 0) 0x7f8d5815b9d0 con 0x7f8d5806c040 2023-10-16T14:36:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.444+0000 7f8d4ffff640 1 -- 172.21.15.29:0/1771422038 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580cc270 con 0x7f8d580ad970 2023-10-16T14:36:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.445+0000 7f8d4ffff640 1 --2- 172.21.15.29:0/1771422038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ad970 0x7f8d580cca70 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8d400098a0 tx=0x7f8d4002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f543d95ca1c53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.445+0000 7f8d567fc640 1 -- 172.21.15.29:0/1771422038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40030840 con 0x7f8d580ad970 2023-10-16T14:36:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.445+0000 7f8d567fc640 1 -- 172.21.15.29:0/1771422038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d400309a0 con 0x7f8d580ad970 2023-10-16T14:36:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.445+0000 7f8d567fc640 1 -- 172.21.15.29:0/1771422038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40030d30 con 0x7f8d580ad970 2023-10-16T14:36:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 -- 172.21.15.29:0/1771422038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ad970 msgr2=0x7f8d580cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1771422038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ad970 0x7f8d580cca70 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8d400098a0 tx=0x7f8d4002b980 comp rx=0 tx=0).stop 2023-10-16T14:36:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 -- 172.21.15.29:0/1771422038 shutdown_connections 2023-10-16T14:36:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1771422038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ad970 0x7f8d580cca70 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 -- 172.21.15.29:0/1771422038 >> 172.21.15.29:0/1771422038 conn(0x7f8d580ce6a0 msgr2=0x7f8d580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 -- 172.21.15.29:0/1771422038 shutdown_connections 2023-10-16T14:36:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.446+0000 7f8d57fff640 1 -- 172.21.15.29:0/1771422038 wait complete. 2023-10-16T14:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.448+0000 7f8d57fff640 1 Processor -- start 2023-10-16T14:36:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.448+0000 7f8d57fff640 1 -- start start 2023-10-16T14:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d57fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d5806c040 0x7f8d581f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d57fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d580ad970 0x7f8d580b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 0x7f8d580ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d57fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580b1c70 con 0x7f8d580ccfb0 2023-10-16T14:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d57fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580b1f50 con 0x7f8d5806c040 2023-10-16T14:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d57fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580b2230 con 0x7f8d580ad970 2023-10-16T14:36:28.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 0x7f8d580ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 0x7f8d580ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36472/0 (socket says 172.21.15.29:36472) 2023-10-16T14:36:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 -- 172.21.15.29:0/1322538622 learned_addr learned my addr 172.21.15.29:0/1322538622 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d56ffd640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d5806c040 0x7f8d581f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d4ffff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d580ad970 0x7f8d580b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 -- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d580ad970 msgr2=0x7f8d580b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d580ad970 0x7f8d580b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 -- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d5806c040 msgr2=0x7f8d581f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d5806c040 0x7f8d581f1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.449+0000 7f8d577fe640 1 -- 172.21.15.29:0/1322538622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400094d0 con 0x7f8d580ccfb0 2023-10-16T14:36:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.450+0000 7f8d577fe640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 0x7f8d580ac890 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c780 tx=0x7f8d5000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.450+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50004b60 con 0x7f8d580ccfb0 2023-10-16T14:36:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.450+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5000ce90 con 0x7f8d580ccfb0 2023-10-16T14:36:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.450+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000a850 con 0x7f8d580ccfb0 2023-10-16T14:36:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.450+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d580b2510 con 0x7f8d580ccfb0 2023-10-16T14:36:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.450+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d581f21e0 con 0x7f8d580ccfb0 2023-10-16T14:36:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.451+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5806b330 con 0x7f8d580ccfb0 2023-10-16T14:36:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.453+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8d5000a9b0 con 0x7f8d580ccfb0 2023-10-16T14:36:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.454+0000 7f8d4f7fe640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8d3806af90 0x7f8d3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.454+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8d50089980 con 0x7f8d580ccfb0 2023-10-16T14:36:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.454+0000 7f8d56ffd640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8d3806af90 0x7f8d3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.455+0000 7f8d56ffd640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8d3806af90 0x7f8d3806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8d48000f60 tx=0x7f8d48007c00 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.458+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d500548f0 con 0x7f8d580ccfb0 2023-10-16T14:36:28.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.655+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d5814f360 con 0x7f8d580ccfb0 2023-10-16T14:36:28.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.657+0000 7f8d4f7fe640 1 -- 172.21.15.29:0/1322538622 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f8d5814f360 con 0x7f8d580ccfb0 2023-10-16T14:36:28.661 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:28.661 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:27.195862+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8d3806af90 msgr2=0x7f8d3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8d3806af90 0x7f8d3806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8d48000f60 tx=0x7f8d48007c00 comp rx=0 tx=0).stop 2023-10-16T14:36:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 msgr2=0x7f8d580ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 0x7f8d580ac890 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c780 tx=0x7f8d5000cc40 comp rx=0 tx=0).stop 2023-10-16T14:36:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 shutdown_connections 2023-10-16T14:36:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d580ccfb0 0x7f8d580ac890 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f8d3806af90 0x7f8d3806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d580ad970 0x7f8d580b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 --2- 172.21.15.29:0/1322538622 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d5806c040 0x7f8d581f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 >> 172.21.15.29:0/1322538622 conn(0x7f8d580ce6a0 msgr2=0x7f8d580c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:28.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 shutdown_connections 2023-10-16T14:36:28.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.660+0000 7f8d57fff640 1 -- 172.21.15.29:0/1322538622 wait complete. 2023-10-16T14:36:28.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:36:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.904+0000 7f5a48935640 1 Processor -- start 2023-10-16T14:36:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.904+0000 7f5a48935640 1 -- start start 2023-10-16T14:36:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a48935640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 0x7f5a4415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a48935640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a44070450 0x7f5a440708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a48935640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a44070df0 con 0x7f5a4406c100 2023-10-16T14:36:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a48935640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4415e490 con 0x7f5a44070450 2023-10-16T14:36:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a48935640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4415aef0 con 0x7f5a44159950 2023-10-16T14:36:28.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a43fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5a4406c100 0x7f5a4406c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46344/0 (socket says 172.21.15.29:46344) 2023-10-16T14:36:28.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.906+0000 7f5a43fff640 1 -- 172.21.15.29:0/147536755 learned_addr learned my addr 172.21.15.29:0/147536755 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a437fe640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 0x7f5a4415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/147536755 >> v2:172.21.15.170:3300/0 conn(0x7f5a44070450 0x7f5a440708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096215013 0 0) 0x7f5a44070df0 con 0x7f5a4406c100 2023-10-16T14:36:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c003520 con 0x7f5a4406c100 2023-10-16T14:36:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437588683 0 0) 0x7f5a2c003520 con 0x7f5a4406c100 2023-10-16T14:36:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 msgr2=0x7f5a4415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 0x7f5a4415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 >> v2:172.21.15.170:3300/0 conn(0x7f5a44070450 msgr2=0x7f5a440708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 --2- 172.21.15.29:0/147536755 >> v2:172.21.15.170:3300/0 conn(0x7f5a44070450 0x7f5a440708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4406cc80 con 0x7f5a4406c100 2023-10-16T14:36:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5a3c0019f0 con 0x7f5a4406c100 2023-10-16T14:36:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 >> v1:172.21.15.29:6789/0 conn(0x7f5a4406c100 legacy=0x7f5a4406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c003970 0x7f5a2c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0047c0 0x7f5a2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 0x7f5a2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4415e490 con 0x7f5a2c0047c0 2023-10-16T14:36:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4415aef0 con 0x7f5a2c003970 2023-10-16T14:36:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.907+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c00b9d0 con 0x7f5a44159950 2023-10-16T14:36:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0047c0 0x7f5a2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a43fff640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 0x7f5a2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a42ffd640 1 -- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 msgr2=0x7f5a2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a44159950 0x7f5a2c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a42ffd640 1 -- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c003970 msgr2=0x7f5a2c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c003970 0x7f5a2c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.908+0000 7f5a42ffd640 1 -- 172.21.15.29:0/147536755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c003520 con 0x7f5a2c0047c0 2023-10-16T14:36:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.909+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0047c0 0x7f5a2c004c40 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009af0 tx=0x7f5a2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135302269018a81a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.909+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3802c710 con 0x7f5a2c0047c0 2023-10-16T14:36:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.909+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3802c870 con 0x7f5a2c0047c0 2023-10-16T14:36:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.909+0000 7f5a427fc640 1 -- 172.21.15.29:0/147536755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3802cc00 con 0x7f5a2c0047c0 2023-10-16T14:36:28.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a48935640 1 -- 172.21.15.29:0/147536755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0047c0 msgr2=0x7f5a2c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a48935640 1 --2- 172.21.15.29:0/147536755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2c0047c0 0x7f5a2c004c40 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009af0 tx=0x7f5a2c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a43fff640 1 -- 172.21.15.29:0/147536755 reap_dead start 2023-10-16T14:36:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a48935640 1 -- 172.21.15.29:0/147536755 shutdown_connections 2023-10-16T14:36:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a48935640 1 -- 172.21.15.29:0/147536755 >> 172.21.15.29:0/147536755 conn(0x7f5a440ce6a0 msgr2=0x7f5a440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a48935640 1 -- 172.21.15.29:0/147536755 shutdown_connections 2023-10-16T14:36:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.910+0000 7f5a48935640 1 -- 172.21.15.29:0/147536755 wait complete. 2023-10-16T14:36:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.912+0000 7f5a48935640 1 Processor -- start 2023-10-16T14:36:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.912+0000 7f5a48935640 1 -- start start 2023-10-16T14:36:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a48935640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c0047c0 0x7f5a440af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a48935640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a440c1260 0x7f5a440b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a48935640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 0x7f5a440b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a48935640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44150360 con 0x7f5a440b32c0 2023-10-16T14:36:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a48935640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440b3740 con 0x7f5a2c0047c0 2023-10-16T14:36:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a48935640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4414fda0 con 0x7f5a440c1260 2023-10-16T14:36:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 0x7f5a440b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 0x7f5a440b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36484/0 (socket says 172.21.15.29:36484) 2023-10-16T14:36:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.913+0000 7f5a43fff640 1 -- 172.21.15.29:0/3399898067 learned_addr learned my addr 172.21.15.29:0/3399898067 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a437fe640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c0047c0 0x7f5a440af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a440c1260 0x7f5a440b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a43fff640 1 -- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a440c1260 msgr2=0x7f5a440b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a43fff640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a440c1260 0x7f5a440b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a43fff640 1 -- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c0047c0 msgr2=0x7f5a440af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a43fff640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c0047c0 0x7f5a440af0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a43fff640 1 -- 172.21.15.29:0/3399898067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44150080 con 0x7f5a440b32c0 2023-10-16T14:36:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a42ffd640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a440c1260 0x7f5a440b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a43fff640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 0x7f5a440b12c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f5a440b1290 tx=0x7f5a3c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c004d60 con 0x7f5a440b32c0 2023-10-16T14:36:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c00bb80 con 0x7f5a440b32c0 2023-10-16T14:36:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c007070 con 0x7f5a440b32c0 2023-10-16T14:36:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a44150740 con 0x7f5a440b32c0 2023-10-16T14:36:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.914+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a440c6560 con 0x7f5a440b32c0 2023-10-16T14:36:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.915+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a441505a0 con 0x7f5a440b32c0 2023-10-16T14:36:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.916+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a3c0071d0 con 0x7f5a440b32c0 2023-10-16T14:36:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.917+0000 7f5a2bfff640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a3006afc0 0x7f5a3006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.917+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5a3c089ce0 con 0x7f5a440b32c0 2023-10-16T14:36:28.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.918+0000 7f5a437fe640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a3006afc0 0x7f5a3006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:28.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.919+0000 7f5a437fe640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a3006afc0 0x7f5a3006d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a34004770 tx=0x7f5a34008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:28.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:28.921+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a441505a0 con 0x7f5a440b32c0 2023-10-16T14:36:29.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.149+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5a44150c60 con 0x7f5a440b32c0 2023-10-16T14:36:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.783+0000 7f5a2bfff640 1 -- 172.21.15.29:0/3399898067 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5a44150c60 con 0x7f5a440b32c0 2023-10-16T14:36:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a3006afc0 msgr2=0x7f5a3006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a3006afc0 0x7f5a3006d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a34004770 tx=0x7f5a34008040 comp rx=0 tx=0).stop 2023-10-16T14:36:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 msgr2=0x7f5a440b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 0x7f5a440b12c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f5a440b1290 tx=0x7f5a3c00b4a0 comp rx=0 tx=0).stop 2023-10-16T14:36:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 shutdown_connections 2023-10-16T14:36:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a440b32c0 0x7f5a440b12c0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f5a3006afc0 0x7f5a3006d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a440c1260 0x7f5a440b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 --2- 172.21.15.29:0/3399898067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2c0047c0 0x7f5a440af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.787+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 >> 172.21.15.29:0/3399898067 conn(0x7f5a440ce6a0 msgr2=0x7f5a44070830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.788+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 shutdown_connections 2023-10-16T14:36:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:29.788+0000 7f5a48935640 1 -- 172.21.15.29:0/3399898067 wait complete. 2023-10-16T14:36:29.814 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:36:29.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:36:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.039+0000 7f03172f3640 1 Processor -- start 2023-10-16T14:36:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.039+0000 7f03172f3640 1 -- start start 2023-10-16T14:36:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.040+0000 7f03172f3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0310159950 0x7f031015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.040+0000 7f03172f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.040+0000 7f03172f3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f031006c100 0x7f031006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.040+0000 7f03172f3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031015e490 con 0x7f0310159950 2023-10-16T14:36:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.040+0000 7f03172f3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0310070df0 con 0x7f031006c100 2023-10-16T14:36:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.040+0000 7f03172f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031015aef0 con 0x7f0310070450 2023-10-16T14:36:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0310159950 0x7f031015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0310159950 0x7f031015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36488/0 (socket says 172.21.15.29:36488) 2023-10-16T14:36:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 -- 172.21.15.29:0/349378992 learned_addr learned my addr 172.21.15.29:0/349378992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 -- 172.21.15.29:0/349378992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 msgr2=0x7f03100708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 --2- 172.21.15.29:0/349378992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 -- 172.21.15.29:0/349378992 >> v2:172.21.15.170:3300/0 conn(0x7f031006c100 msgr2=0x7f031006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 --2- 172.21.15.29:0/349378992 >> v2:172.21.15.170:3300/0 conn(0x7f031006c100 0x7f031006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.041+0000 7f03162f1640 1 -- 172.21.15.29:0/349378992 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031015d910 con 0x7f0310159950 2023-10-16T14:36:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.042+0000 7f03162f1640 1 --2- 172.21.15.29:0/349378992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0310159950 0x7f031015a9b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f03100b0120 tx=0x7f030c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f55fe6fd024e98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03152ef640 1 -- 172.21.15.29:0/349378992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c0317c0 con 0x7f0310159950 2023-10-16T14:36:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03152ef640 1 -- 172.21.15.29:0/349378992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c031920 con 0x7f0310159950 2023-10-16T14:36:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03152ef640 1 -- 172.21.15.29:0/349378992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c031cb0 con 0x7f0310159950 2023-10-16T14:36:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 -- 172.21.15.29:0/349378992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0310159950 msgr2=0x7f031015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 --2- 172.21.15.29:0/349378992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0310159950 0x7f031015a9b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f03100b0120 tx=0x7f030c02b860 comp rx=0 tx=0).stop 2023-10-16T14:36:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 -- 172.21.15.29:0/349378992 shutdown_connections 2023-10-16T14:36:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 --2- 172.21.15.29:0/349378992 >> v2:172.21.15.170:3300/0 conn(0x7f031006c100 0x7f031006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 --2- 172.21.15.29:0/349378992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 --2- 172.21.15.29:0/349378992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0310159950 0x7f031015a9b0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.043+0000 7f03172f3640 1 -- 172.21.15.29:0/349378992 >> 172.21.15.29:0/349378992 conn(0x7f03100ce6a0 msgr2=0x7f03100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.044+0000 7f03172f3640 1 -- 172.21.15.29:0/349378992 shutdown_connections 2023-10-16T14:36:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.044+0000 7f03172f3640 1 -- 172.21.15.29:0/349378992 wait complete. 2023-10-16T14:36:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.045+0000 7f03172f3640 1 Processor -- start 2023-10-16T14:36:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.045+0000 7f03172f3640 1 -- start start 2023-10-16T14:36:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.046+0000 7f03172f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 0x7f03101f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.046+0000 7f03172f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.046+0000 7f03172f3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0310159950 0x7f03100a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.046+0000 7f03172f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03100c81a0 con 0x7f031006c100 2023-10-16T14:36:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.046+0000 7f03172f3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03100c7ec0 con 0x7f0310159950 2023-10-16T14:36:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.046+0000 7f03172f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0310145620 con 0x7f0310070450 2023-10-16T14:36:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 0x7f03101f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 0x7f03101f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36494/0 (socket says 172.21.15.29:36494) 2023-10-16T14:36:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 -- 172.21.15.29:0/2185897919 learned_addr learned my addr 172.21.15.29:0/2185897919 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f0316af2640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0310159950 0x7f03100a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 -- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 msgr2=0x7f03100a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f0315af0640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 -- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0310159950 msgr2=0x7f03100a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0310159950 0x7f03100a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 -- 172.21.15.29:0/2185897919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c0094d0 con 0x7f031006c100 2023-10-16T14:36:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f0316af2640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0310159950 0x7f03100a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f0315af0640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.047+0000 7f03162f1640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 0x7f03101f2220 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f030c000c00 tx=0x7f030c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.048+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c0329a0 con 0x7f031006c100 2023-10-16T14:36:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.048+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c032b20 con 0x7f031006c100 2023-10-16T14:36:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.048+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c0453f0 con 0x7f031006c100 2023-10-16T14:36:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.048+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310145900 con 0x7f031006c100 2023-10-16T14:36:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.048+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03101f2760 con 0x7f031006c100 2023-10-16T14:36:30.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.049+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0310145e20 con 0x7f031006c100 2023-10-16T14:36:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.050+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f030c04d3f0 con 0x7f031006c100 2023-10-16T14:36:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.051+0000 7f0302ffd640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f02f406af90 0x7f02f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.052+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f030c0ab760 con 0x7f031006c100 2023-10-16T14:36:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.052+0000 7f0315af0640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f02f406af90 0x7f02f406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.053+0000 7f0315af0640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f02f406af90 0x7f02f406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0304006fd0 tx=0x7f0304008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:30.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.056+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0310145e20 con 0x7f031006c100 2023-10-16T14:36:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:30.288+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0310145760 con 0x7f031006c100 2023-10-16T14:36:31.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.298+0000 7f0302ffd640 1 -- 172.21.15.29:0/2185897919 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0310145760 con 0x7f031006c100 2023-10-16T14:36:31.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f02f406af90 msgr2=0x7f02f406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f02f406af90 0x7f02f406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0304006fd0 tx=0x7f0304008040 comp rx=0 tx=0).stop 2023-10-16T14:36:31.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 msgr2=0x7f03101f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 0x7f03101f2220 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f030c000c00 tx=0x7f030c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 shutdown_connections 2023-10-16T14:36:31.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f02f406af90 0x7f02f406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0310159950 0x7f03100a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0310070450 0x7f03100a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 --2- 172.21.15.29:0/2185897919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f031006c100 0x7f03101f2220 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 >> 172.21.15.29:0/2185897919 conn(0x7f03100ce6a0 msgr2=0x7f031015a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:31.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 shutdown_connections 2023-10-16T14:36:31.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.303+0000 7f03172f3640 1 -- 172.21.15.29:0/2185897919 wait complete. 2023-10-16T14:36:31.329 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:36:31.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:31.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.551+0000 7fde790d7640 1 Processor -- start 2023-10-16T14:36:31.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.551+0000 7fde790d7640 1 -- start start 2023-10-16T14:36:31.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde790d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fde7415aaa0 0x7fde7415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde790d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde7415ae80 0x7fde740cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde790d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde74070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde790d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7406b810 con 0x7fde7415ae80 2023-10-16T14:36:31.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde790d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7415ed60 con 0x7fde7415aaa0 2023-10-16T14:36:31.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde790d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde740707e0 con 0x7fde740ccaa0 2023-10-16T14:36:31.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde6bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde7415ae80 0x7fde740cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde6bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde7415ae80 0x7fde740cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36498/0 (socket says 172.21.15.29:36498) 2023-10-16T14:36:31.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.553+0000 7fde6bfff640 1 -- 172.21.15.29:0/1660732100 learned_addr learned my addr 172.21.15.29:0/1660732100 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:31.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde72d9d640 1 --2- 172.21.15.29:0/1660732100 >> v2:172.21.15.170:3300/0 conn(0x7fde7415aaa0 0x7fde7415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde7359e640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde74070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde6bfff640 1 -- 172.21.15.29:0/1660732100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 msgr2=0x7fde74070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:31.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde6bfff640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde74070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde6bfff640 1 -- 172.21.15.29:0/1660732100 >> v2:172.21.15.170:3300/0 conn(0x7fde7415aaa0 msgr2=0x7fde7415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:31.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde6bfff640 1 --2- 172.21.15.29:0/1660732100 >> v2:172.21.15.170:3300/0 conn(0x7fde7415aaa0 0x7fde7415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde6bfff640 1 -- 172.21.15.29:0/1660732100 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7415ef90 con 0x7fde7415ae80 2023-10-16T14:36:31.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.554+0000 7fde7359e640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde74070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:31.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.555+0000 7fde6bfff640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415ae80 0x7fde740cc560 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fde740cd990 tx=0x7fde5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1dae2bc4b5f1cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:31.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.555+0000 7fde7259c640 1 -- 172.21.15.29:0/1660732100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c031870 con 0x7fde7415ae80 2023-10-16T14:36:31.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.555+0000 7fde7259c640 1 -- 172.21.15.29:0/1660732100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c0319d0 con 0x7fde7415ae80 2023-10-16T14:36:31.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.555+0000 7fde7259c640 1 -- 172.21.15.29:0/1660732100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c031d60 con 0x7fde7415ae80 2023-10-16T14:36:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 -- 172.21.15.29:0/1660732100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415ae80 msgr2=0x7fde740cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415ae80 0x7fde740cc560 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fde740cd990 tx=0x7fde5c02b910 comp rx=0 tx=0).stop 2023-10-16T14:36:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 -- 172.21.15.29:0/1660732100 shutdown_connections 2023-10-16T14:36:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde74070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 --2- 172.21.15.29:0/1660732100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415ae80 0x7fde740cc560 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 --2- 172.21.15.29:0/1660732100 >> v2:172.21.15.170:3300/0 conn(0x7fde7415aaa0 0x7fde7415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 -- 172.21.15.29:0/1660732100 >> 172.21.15.29:0/1660732100 conn(0x7fde740cb410 msgr2=0x7fde740cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 -- 172.21.15.29:0/1660732100 shutdown_connections 2023-10-16T14:36:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.556+0000 7fde790d7640 1 -- 172.21.15.29:0/1660732100 wait complete. 2023-10-16T14:36:31.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.558+0000 7fde790d7640 1 Processor -- start 2023-10-16T14:36:31.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.558+0000 7fde790d7640 1 -- start start 2023-10-16T14:36:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde790d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde741f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde790d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 0x7fde7415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde790d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde7415ae80 0x7fde741e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde790d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde741e45b0 con 0x7fde7415aaa0 2023-10-16T14:36:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde790d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde741f2c20 con 0x7fde7415ae80 2023-10-16T14:36:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde790d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde741f2f00 con 0x7fde740ccaa0 2023-10-16T14:36:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 0x7fde7415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 0x7fde7415dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36510/0 (socket says 172.21.15.29:36510) 2023-10-16T14:36:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.559+0000 7fde6bfff640 1 -- 172.21.15.29:0/2600381270 learned_addr learned my addr 172.21.15.29:0/2600381270 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6bfff640 1 -- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 msgr2=0x7fde741f6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde7359e640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde7415ae80 0x7fde741e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde72d9d640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde741f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6bfff640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde741f6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6bfff640 1 -- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde7415ae80 msgr2=0x7fde741e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6bfff640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde7415ae80 0x7fde741e7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6bfff640 1 -- 172.21.15.29:0/2600381270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c0094d0 con 0x7fde7415aaa0 2023-10-16T14:36:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6bfff640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 0x7fde7415dad0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fde5c02b950 tx=0x7fde5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c03ad30 con 0x7fde7415aaa0 2023-10-16T14:36:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c0443f0 con 0x7fde7415aaa0 2023-10-16T14:36:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0447a0 con 0x7fde7415aaa0 2023-10-16T14:36:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde741f31e0 con 0x7fde7415aaa0 2023-10-16T14:36:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.560+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde741e5f90 con 0x7fde7415aaa0 2023-10-16T14:36:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.561+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7404e160 con 0x7fde7415aaa0 2023-10-16T14:36:31.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.563+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fde5c04d450 con 0x7fde7415aaa0 2023-10-16T14:36:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.563+0000 7fde6b7fe640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fde5406af90 0x7fde5406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.564+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fde5c02d040 con 0x7fde7415aaa0 2023-10-16T14:36:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.564+0000 7fde72d9d640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fde5406af90 0x7fde5406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.565+0000 7fde72d9d640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fde5406af90 0x7fde5406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde64006fd0 tx=0x7fde64008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.568+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde5c0b0070 con 0x7fde7415aaa0 2023-10-16T14:36:31.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.802+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde740cdc60 con 0x7fde7415aaa0 2023-10-16T14:36:31.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.803+0000 7fde6b7fe640 1 -- 172.21.15.29:0/2600381270 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fde5c035070 con 0x7fde7415aaa0 2023-10-16T14:36:31.807 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:31.807 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:36:27.196747+0000","modified":"2023-10-16T14:36:30.291825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1700644562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1700644562},{"type":"v1","addr":"172.21.15.29:6835","nonce":1700644562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T14:36:31.808 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-16T14:36:31.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.805+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fde5406af90 msgr2=0x7fde5406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.805+0000 7fde790d7640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fde5406af90 0x7fde5406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde64006fd0 tx=0x7fde64008040 comp rx=0 tx=0).stop 2023-10-16T14:36:31.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.805+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 msgr2=0x7fde7415dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:31.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.805+0000 7fde790d7640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 0x7fde7415dad0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fde5c02b950 tx=0x7fde5c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 shutdown_connections 2023-10-16T14:36:31.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde7415aaa0 0x7fde7415dad0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fde5406af90 0x7fde5406d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde7415ae80 0x7fde741e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 --2- 172.21.15.29:0/2600381270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde740ccaa0 0x7fde741f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:31.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 >> 172.21.15.29:0/2600381270 conn(0x7fde740cb410 msgr2=0x7fde74070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:31.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 shutdown_connections 2023-10-16T14:36:31.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:31.806+0000 7fde790d7640 1 -- 172.21.15.29:0/2600381270 wait complete. 2023-10-16T14:36:31.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:32.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.047+0000 7fd91f7d7640 1 Processor -- start 2023-10-16T14:36:32.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.047+0000 7fd91f7d7640 1 -- start start 2023-10-16T14:36:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91f7d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd91815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91f7d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd91815ae80 0x7fd9180cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91f7d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9180ccaa0 0x7fd918070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91f7d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91806b810 con 0x7fd9180ccaa0 2023-10-16T14:36:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91f7d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91815ed60 con 0x7fd91815ae80 2023-10-16T14:36:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91f7d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180707e0 con 0x7fd91815aaa0 2023-10-16T14:36:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91dd74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9180ccaa0 0x7fd918070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91dd74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9180ccaa0 0x7fd918070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36526/0 (socket says 172.21.15.29:36526) 2023-10-16T14:36:32.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.049+0000 7fd91dd74640 1 -- 172.21.15.29:0/1802189606 learned_addr learned my addr 172.21.15.29:0/1802189606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91cd72640 1 --2- 172.21.15.29:0/1802189606 >> v2:172.21.15.170:3300/0 conn(0x7fd91815ae80 0x7fd9180cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91d573640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd91815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91dd74640 1 -- 172.21.15.29:0/1802189606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 msgr2=0x7fd91815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91dd74640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd91815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91dd74640 1 -- 172.21.15.29:0/1802189606 >> v2:172.21.15.170:3300/0 conn(0x7fd91815ae80 msgr2=0x7fd9180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91dd74640 1 --2- 172.21.15.29:0/1802189606 >> v2:172.21.15.170:3300/0 conn(0x7fd91815ae80 0x7fd9180cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91dd74640 1 -- 172.21.15.29:0/1802189606 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91815ef90 con 0x7fd9180ccaa0 2023-10-16T14:36:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91d573640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd91815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.050+0000 7fd91cd72640 1 --2- 172.21.15.29:0/1802189606 >> v2:172.21.15.170:3300/0 conn(0x7fd91815ae80 0x7fd9180cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.051+0000 7fd91dd74640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9180ccaa0 0x7fd918070f50 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd9180cb270 tx=0x7fd91002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67884a6c2b7c79a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.051+0000 7fd90ffff640 1 -- 172.21.15.29:0/1802189606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91002ca80 con 0x7fd9180ccaa0 2023-10-16T14:36:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.051+0000 7fd90ffff640 1 -- 172.21.15.29:0/1802189606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91002cbe0 con 0x7fd9180ccaa0 2023-10-16T14:36:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.051+0000 7fd90ffff640 1 -- 172.21.15.29:0/1802189606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9100343f0 con 0x7fd9180ccaa0 2023-10-16T14:36:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 -- 172.21.15.29:0/1802189606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9180ccaa0 msgr2=0x7fd918070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9180ccaa0 0x7fd918070f50 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd9180cb270 tx=0x7fd91002bee0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 -- 172.21.15.29:0/1802189606 shutdown_connections 2023-10-16T14:36:32.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9180ccaa0 0x7fd918070f50 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/1802189606 >> v2:172.21.15.170:3300/0 conn(0x7fd91815ae80 0x7fd9180cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/1802189606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd91815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 -- 172.21.15.29:0/1802189606 >> 172.21.15.29:0/1802189606 conn(0x7fd9180cb410 msgr2=0x7fd9180cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 -- 172.21.15.29:0/1802189606 shutdown_connections 2023-10-16T14:36:32.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.052+0000 7fd91f7d7640 1 -- 172.21.15.29:0/1802189606 wait complete. 2023-10-16T14:36:32.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.054+0000 7fd91f7d7640 1 Processor -- start 2023-10-16T14:36:32.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.054+0000 7fd91f7d7640 1 -- start start 2023-10-16T14:36:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91f7d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9180ccaa0 0x7fd918155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91f7d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd9181566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91f7d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 0x7fd9180c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91f7d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181564c0 con 0x7fd91815ae80 2023-10-16T14:36:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91f7d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd918159850 con 0x7fd9180ccaa0 2023-10-16T14:36:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91f7d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91815b4c0 con 0x7fd91815aaa0 2023-10-16T14:36:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91dd74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 0x7fd9180c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.055+0000 7fd91dd74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 0x7fd9180c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36536/0 (socket says 172.21.15.29:36536) 2023-10-16T14:36:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 -- 172.21.15.29:0/3407007390 learned_addr learned my addr 172.21.15.29:0/3407007390 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91d573640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9180ccaa0 0x7fd918155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91cd72640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd9181566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 -- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 msgr2=0x7fd9181566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd9181566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 -- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9180ccaa0 msgr2=0x7fd918155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9180ccaa0 0x7fd918155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 -- 172.21.15.29:0/3407007390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910009b30 con 0x7fd91815ae80 2023-10-16T14:36:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.056+0000 7fd91dd74640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 0x7fd9180c49a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd910036040 tx=0x7fd910004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.057+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910009510 con 0x7fd91815ae80 2023-10-16T14:36:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.057+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91003e450 con 0x7fd91815ae80 2023-10-16T14:36:32.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.057+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91003e800 con 0x7fd91815ae80 2023-10-16T14:36:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.057+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91815b7a0 con 0x7fd91815ae80 2023-10-16T14:36:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.057+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd918156c20 con 0x7fd91815ae80 2023-10-16T14:36:32.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.058+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91815b320 con 0x7fd91815ae80 2023-10-16T14:36:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.059+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd91002cd50 con 0x7fd91815ae80 2023-10-16T14:36:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.061+0000 7fd90dffb640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fd8fc06af90 0x7fd8fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.061+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd9100ac3e0 con 0x7fd91815ae80 2023-10-16T14:36:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.061+0000 7fd91d573640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fd8fc06af90 0x7fd8fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.062+0000 7fd91d573640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fd8fc06af90 0x7fd8fc06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd91400a980 tx=0x7fd914008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:32.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.065+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd91815b320 con 0x7fd91815ae80 2023-10-16T14:36:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.259+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91815bcc0 con 0x7fd91815ae80 2023-10-16T14:36:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.262+0000 7fd90dffb640 1 -- 172.21.15.29:0/3407007390 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fd91815bcc0 con 0x7fd91815ae80 2023-10-16T14:36:32.266 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:32.266 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":142,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:28.743002+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fd8fc06af90 msgr2=0x7fd8fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fd8fc06af90 0x7fd8fc06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd91400a980 tx=0x7fd914008040 comp rx=0 tx=0).stop 2023-10-16T14:36:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 msgr2=0x7fd9180c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 0x7fd9180c49a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd910036040 tx=0x7fd910004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 shutdown_connections 2023-10-16T14:36:32.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91815ae80 0x7fd9180c49a0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fd8fc06af90 0x7fd8fc06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd91815aaa0 0x7fd9181566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 --2- 172.21.15.29:0/3407007390 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd9180ccaa0 0x7fd918155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.265+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 >> 172.21.15.29:0/3407007390 conn(0x7fd9180cb410 msgr2=0x7fd9180b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.266+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 shutdown_connections 2023-10-16T14:36:32.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.266+0000 7fd91f7d7640 1 -- 172.21.15.29:0/3407007390 wait complete. 2023-10-16T14:36:32.288 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:36:32.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:36:32.305 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T14:36:32.301+0000 7fc09189c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:36:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.513+0000 7f3af619e640 1 Processor -- start 2023-10-16T14:36:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.513+0000 7f3af619e640 1 -- start start 2023-10-16T14:36:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.515+0000 7f3af619e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3af015aaa0 0x7f3af00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.515+0000 7f3af619e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af0070240 0x7f3af00706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.515+0000 7f3af619e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af00cad00 con 0x7f3af00ca8a0 2023-10-16T14:36:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.515+0000 7f3af619e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af0070c10 con 0x7f3af015aaa0 2023-10-16T14:36:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.515+0000 7f3af619e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00b2d00 con 0x7f3af0070240 2023-10-16T14:36:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.515+0000 7f3af519c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3af015aaa0 0x7f3af00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af519c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3af015aaa0 0x7f3af00b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36540/0 (socket says 172.21.15.29:36540) 2023-10-16T14:36:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af519c640 1 -- 172.21.15.29:0/1812734043 learned_addr learned my addr 172.21.15.29:0/1812734043 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af599d640 1 --2- 172.21.15.29:0/1812734043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af0070240 0x7f3af00706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af519c640 1 -- 172.21.15.29:0/1812734043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af0070240 msgr2=0x7f3af00706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af519c640 1 --2- 172.21.15.29:0/1812734043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af0070240 0x7f3af00706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af519c640 1 -- 172.21.15.29:0/1812734043 >> v1:172.21.15.170:6789/0 conn(0x7f3af00ca8a0 legacy=0x7f3af00b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.516+0000 7f3af519c640 1 -- 172.21.15.29:0/1812734043 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af00b2fe0 con 0x7f3af015aaa0 2023-10-16T14:36:32.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.517+0000 7f3af519c640 1 --2- 172.21.15.29:0/1812734043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af00b2600 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009e50 tx=0x7f3aec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e71b0805a53814 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:32.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.517+0000 7f3ae7fff640 1 -- 172.21.15.29:0/1812734043 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec02c9e0 con 0x7f3af015aaa0 2023-10-16T14:36:32.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3ae7fff640 1 -- 172.21.15.29:0/1812734043 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec02cb40 con 0x7f3af015aaa0 2023-10-16T14:36:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3ae7fff640 1 -- 172.21.15.29:0/1812734043 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0343f0 con 0x7f3af015aaa0 2023-10-16T14:36:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 -- 172.21.15.29:0/1812734043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 msgr2=0x7f3af00b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 --2- 172.21.15.29:0/1812734043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af00b2600 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009e50 tx=0x7f3aec02be40 comp rx=0 tx=0).stop 2023-10-16T14:36:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 -- 172.21.15.29:0/1812734043 shutdown_connections 2023-10-16T14:36:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 --2- 172.21.15.29:0/1812734043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af0070240 0x7f3af00706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 --2- 172.21.15.29:0/1812734043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af00b2600 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 -- 172.21.15.29:0/1812734043 >> 172.21.15.29:0/1812734043 conn(0x7f3af00cc230 msgr2=0x7f3af00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:32.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 -- 172.21.15.29:0/1812734043 shutdown_connections 2023-10-16T14:36:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.518+0000 7f3af619e640 1 -- 172.21.15.29:0/1812734043 wait complete. 2023-10-16T14:36:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.520+0000 7f3af619e640 1 Processor -- start 2023-10-16T14:36:32.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.520+0000 7f3af619e640 1 -- start start 2023-10-16T14:36:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af619e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3af0070240 0x7f3af014c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af619e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af00ca8a0 0x7f3af01f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af619e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af01f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af619e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00c9380 con 0x7f3af015aaa0 2023-10-16T14:36:32.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af619e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af01f3850 con 0x7f3af0070240 2023-10-16T14:36:32.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af619e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00c90a0 con 0x7f3af00ca8a0 2023-10-16T14:36:32.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.521+0000 7f3af599d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af01f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af01f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36550/0 (socket says 172.21.15.29:36550) 2023-10-16T14:36:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 -- 172.21.15.29:0/2846882486 learned_addr learned my addr 172.21.15.29:0/2846882486 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:32.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af499b640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af00ca8a0 0x7f3af01f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af519c640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3af0070240 0x7f3af014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 -- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af00ca8a0 msgr2=0x7f3af01f26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af00ca8a0 0x7f3af01f26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 -- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3af0070240 msgr2=0x7f3af014c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3af0070240 0x7f3af014c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 -- 172.21.15.29:0/2846882486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec009b30 con 0x7f3af015aaa0 2023-10-16T14:36:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af519c640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3af0070240 0x7f3af014c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.522+0000 7f3af599d640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af01f2c10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800d5d0 tx=0x7f3ae800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:32.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.523+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae80125f0 con 0x7f3af015aaa0 2023-10-16T14:36:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.523+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae8012770 con 0x7f3af015aaa0 2023-10-16T14:36:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.523+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8012b20 con 0x7f3af015aaa0 2023-10-16T14:36:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.523+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af014ca40 con 0x7f3af015aaa0 2023-10-16T14:36:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.523+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af014d000 con 0x7f3af015aaa0 2023-10-16T14:36:32.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.524+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af014c8a0 con 0x7f3af015aaa0 2023-10-16T14:36:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.525+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3ae8012c80 con 0x7f3af015aaa0 2023-10-16T14:36:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.526+0000 7f3ae5ffb640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f3ad406af90 0x7f3ad406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:32.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.526+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3ae8089d20 con 0x7f3af015aaa0 2023-10-16T14:36:32.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.527+0000 7f3af519c640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f3ad406af90 0x7f3ad406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.528+0000 7f3af519c640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f3ad406af90 0x7f3ad406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3aec035000 tx=0x7f3aec030000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:32.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.530+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3af014c8a0 con 0x7f3af015aaa0 2023-10-16T14:36:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.720+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 --> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3af00b35a0 con 0x7f3ad406af90 2023-10-16T14:36:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.749+0000 7f3ae5ffb640 1 -- 172.21.15.29:0/2846882486 <== mgr.6790 v2:172.21.15.170:6832/1905872466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287480 (secure 0 0 0) 0x7f3aec0343f0 con 0x7f3ad406af90 2023-10-16T14:36:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:36:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f3ad406af90 msgr2=0x7f3ad406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f3ad406af90 0x7f3ad406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3aec035000 tx=0x7f3aec030000 comp rx=0 tx=0).stop 2023-10-16T14:36:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 msgr2=0x7f3af01f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af01f2c10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800d5d0 tx=0x7f3ae800da90 comp rx=0 tx=0).stop 2023-10-16T14:36:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 shutdown_connections 2023-10-16T14:36:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015aaa0 0x7f3af01f2c10 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f3ad406af90 0x7f3ad406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3af00ca8a0 0x7f3af01f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 --2- 172.21.15.29:0/2846882486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3af0070240 0x7f3af014c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 >> 172.21.15.29:0/2846882486 conn(0x7f3af00cc230 msgr2=0x7f3af00c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 shutdown_connections 2023-10-16T14:36:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:32.756+0000 7f3af619e640 1 -- 172.21.15.29:0/2846882486 wait complete. 2023-10-16T14:36:32.781 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T14:36:32.473031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":407,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":351832,"kb_used_data":14936,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754622888,"statfs":{"total":773094113280,"available":772733837312,"internally_reserved":0,"allocated":15294464,"data_stored":7986180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":344837689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002281"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979063+0000","last_change":"2023-10-16T14:36:27.795757+0000","last_active":"2023-10-16T14:36:28.979063+0000","last_peered":"2023-10-16T14:36:28.979063+0000","last_clean":"2023-10-16T14:36:28.979063+0000","last_became_active":"2023-10-16T14:36:27.791202+0000","last_became_peered":"2023-10-16T14:36:27.791202+0000","last_unstale":"2023-10-16T14:36:28.979063+0000","last_undegraded":"2023-10-16T14:36:28.979063+0000","last_fullsized":"2023-10-16T14:36:28.979063+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:14.307117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754126+0000","last_change":"2023-10-16T14:36:27.788137+0000","last_active":"2023-10-16T14:36:28.754126+0000","last_peered":"2023-10-16T14:36:28.754126+0000","last_clean":"2023-10-16T14:36:28.754126+0000","last_became_active":"2023-10-16T14:36:27.787521+0000","last_became_peered":"2023-10-16T14:36:27.787521+0000","last_unstale":"2023-10-16T14:36:28.754126+0000","last_undegraded":"2023-10-16T14:36:28.754126+0000","last_fullsized":"2023-10-16T14:36:28.754126+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:23.090304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754979+0000","last_change":"2023-10-16T14:36:27.774865+0000","last_active":"2023-10-16T14:36:28.754979+0000","last_peered":"2023-10-16T14:36:28.754979+0000","last_clean":"2023-10-16T14:36:28.754979+0000","last_became_active":"2023-10-16T14:36:27.774571+0000","last_became_peered":"2023-10-16T14:36:27.774571+0000","last_unstale":"2023-10-16T14:36:28.754979+0000","last_undegraded":"2023-10-16T14:36:28.754979+0000","last_fullsized":"2023-10-16T14:36:28.754979+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:46.587562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979073+0000","last_change":"2023-10-16T14:36:27.775697+0000","last_active":"2023-10-16T14:36:28.979073+0000","last_peered":"2023-10-16T14:36:28.979073+0000","last_clean":"2023-10-16T14:36:28.979073+0000","last_became_active":"2023-10-16T14:36:27.775301+0000","last_became_peered":"2023-10-16T14:36:27.775301+0000","last_unstale":"2023-10-16T14:36:28.979073+0000","last_undegraded":"2023-10-16T14:36:28.979073+0000","last_fullsized":"2023-10-16T14:36:28.979073+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:55.543011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975826+0000","last_change":"2023-10-16T14:36:27.780715+0000","last_active":"2023-10-16T14:36:28.975826+0000","last_peered":"2023-10-16T14:36:28.975826+0000","last_clean":"2023-10-16T14:36:28.975826+0000","last_became_active":"2023-10-16T14:36:27.779102+0000","last_became_peered":"2023-10-16T14:36:27.779102+0000","last_unstale":"2023-10-16T14:36:28.975826+0000","last_undegraded":"2023-10-16T14:36:28.975826+0000","last_fullsized":"2023-10-16T14:36:28.975826+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:21.414718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977051+0000","last_change":"2023-10-16T14:36:27.785909+0000","last_active":"2023-10-16T14:36:28.977051+0000","last_peered":"2023-10-16T14:36:28.977051+0000","last_clean":"2023-10-16T14:36:28.977051+0000","last_became_active":"2023-10-16T14:36:27.785606+0000","last_became_peered":"2023-10-16T14:36:27.785606+0000","last_unstale":"2023-10-16T14:36:28.977051+0000","last_undegraded":"2023-10-16T14:36:28.977051+0000","last_fullsized":"2023-10-16T14:36:28.977051+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:42.923548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977277+0000","last_change":"2023-10-16T14:36:27.780492+0000","last_active":"2023-10-16T14:36:28.977277+0000","last_peered":"2023-10-16T14:36:28.977277+0000","last_clean":"2023-10-16T14:36:28.977277+0000","last_became_active":"2023-10-16T14:36:27.777749+0000","last_became_peered":"2023-10-16T14:36:27.777749+0000","last_unstale":"2023-10-16T14:36:28.977277+0000","last_undegraded":"2023-10-16T14:36:28.977277+0000","last_fullsized":"2023-10-16T14:36:28.977277+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:34.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752503+0000","last_change":"2023-10-16T14:36:27.791062+0000","last_active":"2023-10-16T14:36:28.752503+0000","last_peered":"2023-10-16T14:36:28.752503+0000","last_clean":"2023-10-16T14:36:28.752503+0000","last_became_active":"2023-10-16T14:36:27.790725+0000","last_became_peered":"2023-10-16T14:36:27.790725+0000","last_unstale":"2023-10-16T14:36:28.752503+0000","last_undegraded":"2023-10-16T14:36:28.752503+0000","last_fullsized":"2023-10-16T14:36:28.752503+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:48.161425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979201+0000","last_change":"2023-10-16T14:36:27.792830+0000","last_active":"2023-10-16T14:36:28.979201+0000","last_peered":"2023-10-16T14:36:28.979201+0000","last_clean":"2023-10-16T14:36:28.979201+0000","last_became_active":"2023-10-16T14:36:27.792094+0000","last_became_peered":"2023-10-16T14:36:27.792094+0000","last_unstale":"2023-10-16T14:36:28.979201+0000","last_undegraded":"2023-10-16T14:36:28.979201+0000","last_fullsized":"2023-10-16T14:36:28.979201+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:04.425861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979457+0000","last_change":"2023-10-16T14:36:27.789311+0000","last_active":"2023-10-16T14:36:28.979457+0000","last_peered":"2023-10-16T14:36:28.979457+0000","last_clean":"2023-10-16T14:36:28.979457+0000","last_became_active":"2023-10-16T14:36:27.788879+0000","last_became_peered":"2023-10-16T14:36:27.788879+0000","last_unstale":"2023-10-16T14:36:28.979457+0000","last_undegraded":"2023-10-16T14:36:28.979457+0000","last_fullsized":"2023-10-16T14:36:28.979457+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:41.252139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755181+0000","last_change":"2023-10-16T14:36:27.776590+0000","last_active":"2023-10-16T14:36:28.755181+0000","last_peered":"2023-10-16T14:36:28.755181+0000","last_clean":"2023-10-16T14:36:28.755181+0000","last_became_active":"2023-10-16T14:36:27.774290+0000","last_became_peered":"2023-10-16T14:36:27.774290+0000","last_unstale":"2023-10-16T14:36:28.755181+0000","last_undegraded":"2023-10-16T14:36:28.755181+0000","last_fullsized":"2023-10-16T14:36:28.755181+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:04.169834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979433+0000","last_change":"2023-10-16T14:36:27.788870+0000","last_active":"2023-10-16T14:36:28.979433+0000","last_peered":"2023-10-16T14:36:28.979433+0000","last_clean":"2023-10-16T14:36:28.979433+0000","last_became_active":"2023-10-16T14:36:27.788583+0000","last_became_peered":"2023-10-16T14:36:27.788583+0000","last_unstale":"2023-10-16T14:36:28.979433+0000","last_undegraded":"2023-10-16T14:36:28.979433+0000","last_fullsized":"2023-10-16T14:36:28.979433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:05.723950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753628+0000","last_change":"2023-10-16T14:36:27.783679+0000","last_active":"2023-10-16T14:36:28.753628+0000","last_peered":"2023-10-16T14:36:28.753628+0000","last_clean":"2023-10-16T14:36:28.753628+0000","last_became_active":"2023-10-16T14:36:27.783366+0000","last_became_peered":"2023-10-16T14:36:27.783366+0000","last_unstale":"2023-10-16T14:36:28.753628+0000","last_undegraded":"2023-10-16T14:36:28.753628+0000","last_fullsized":"2023-10-16T14:36:28.753628+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:00.218713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753889+0000","last_change":"2023-10-16T14:36:27.776685+0000","last_active":"2023-10-16T14:36:28.753889+0000","last_peered":"2023-10-16T14:36:28.753889+0000","last_clean":"2023-10-16T14:36:28.753889+0000","last_became_active":"2023-10-16T14:36:27.776399+0000","last_became_peered":"2023-10-16T14:36:27.776399+0000","last_unstale":"2023-10-16T14:36:28.753889+0000","last_undegraded":"2023-10-16T14:36:28.753889+0000","last_fullsized":"2023-10-16T14:36:28.753889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:51.527376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976858+0000","last_change":"2023-10-16T14:36:27.785864+0000","last_active":"2023-10-16T14:36:28.976858+0000","last_peered":"2023-10-16T14:36:28.976858+0000","last_clean":"2023-10-16T14:36:28.976858+0000","last_became_active":"2023-10-16T14:36:27.784994+0000","last_became_peered":"2023-10-16T14:36:27.784994+0000","last_unstale":"2023-10-16T14:36:28.976858+0000","last_undegraded":"2023-10-16T14:36:28.976858+0000","last_fullsized":"2023-10-16T14:36:28.976858+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:56.720480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977352+0000","last_change":"2023-10-16T14:36:27.778753+0000","last_active":"2023-10-16T14:36:28.977352+0000","last_peered":"2023-10-16T14:36:28.977352+0000","last_clean":"2023-10-16T14:36:28.977352+0000","last_became_active":"2023-10-16T14:36:27.778472+0000","last_became_peered":"2023-10-16T14:36:27.778472+0000","last_unstale":"2023-10-16T14:36:28.977352+0000","last_undegraded":"2023-10-16T14:36:28.977352+0000","last_fullsized":"2023-10-16T14:36:28.977352+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:39.400330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754267+0000","last_change":"2023-10-16T14:36:27.783159+0000","last_active":"2023-10-16T14:36:28.754267+0000","last_peered":"2023-10-16T14:36:28.754267+0000","last_clean":"2023-10-16T14:36:28.754267+0000","last_became_active":"2023-10-16T14:36:27.782820+0000","last_became_peered":"2023-10-16T14:36:27.782820+0000","last_unstale":"2023-10-16T14:36:28.754267+0000","last_undegraded":"2023-10-16T14:36:28.754267+0000","last_fullsized":"2023-10-16T14:36:28.754267+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:02.950949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977039+0000","last_change":"2023-10-16T14:36:27.784685+0000","last_active":"2023-10-16T14:36:28.977039+0000","last_peered":"2023-10-16T14:36:28.977039+0000","last_clean":"2023-10-16T14:36:28.977039+0000","last_became_active":"2023-10-16T14:36:27.784382+0000","last_became_peered":"2023-10-16T14:36:27.784382+0000","last_unstale":"2023-10-16T14:36:28.977039+0000","last_undegraded":"2023-10-16T14:36:28.977039+0000","last_fullsized":"2023-10-16T14:36:28.977039+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:46.651175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755341+0000","last_change":"2023-10-16T14:36:27.787939+0000","last_active":"2023-10-16T14:36:28.755341+0000","last_peered":"2023-10-16T14:36:28.755341+0000","last_clean":"2023-10-16T14:36:28.755341+0000","last_became_active":"2023-10-16T14:36:27.786332+0000","last_became_peered":"2023-10-16T14:36:27.786332+0000","last_unstale":"2023-10-16T14:36:28.755341+0000","last_undegraded":"2023-10-16T14:36:28.755341+0000","last_fullsized":"2023-10-16T14:36:28.755341+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:08.908055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754558+0000","last_change":"2023-10-16T14:36:27.782072+0000","last_active":"2023-10-16T14:36:28.754558+0000","last_peered":"2023-10-16T14:36:28.754558+0000","last_clean":"2023-10-16T14:36:28.754558+0000","last_became_active":"2023-10-16T14:36:27.781017+0000","last_became_peered":"2023-10-16T14:36:27.781017+0000","last_unstale":"2023-10-16T14:36:28.754558+0000","last_undegraded":"2023-10-16T14:36:28.754558+0000","last_fullsized":"2023-10-16T14:36:28.754558+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:47.470649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754484+0000","last_change":"2023-10-16T14:36:27.783008+0000","last_active":"2023-10-16T14:36:28.754484+0000","last_peered":"2023-10-16T14:36:28.754484+0000","last_clean":"2023-10-16T14:36:28.754484+0000","last_became_active":"2023-10-16T14:36:27.782503+0000","last_became_peered":"2023-10-16T14:36:27.782503+0000","last_unstale":"2023-10-16T14:36:28.754484+0000","last_undegraded":"2023-10-16T14:36:28.754484+0000","last_fullsized":"2023-10-16T14:36:28.754484+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:16.571347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749503+0000","last_change":"2023-10-16T14:36:27.777789+0000","last_active":"2023-10-16T14:36:28.749503+0000","last_peered":"2023-10-16T14:36:28.749503+0000","last_clean":"2023-10-16T14:36:28.749503+0000","last_became_active":"2023-10-16T14:36:27.777472+0000","last_became_peered":"2023-10-16T14:36:27.777472+0000","last_unstale":"2023-10-16T14:36:28.749503+0000","last_undegraded":"2023-10-16T14:36:28.749503+0000","last_fullsized":"2023-10-16T14:36:28.749503+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:20.996152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752180+0000","last_change":"2023-10-16T14:36:27.782563+0000","last_active":"2023-10-16T14:36:28.752180+0000","last_peered":"2023-10-16T14:36:28.752180+0000","last_clean":"2023-10-16T14:36:28.752180+0000","last_became_active":"2023-10-16T14:36:27.781508+0000","last_became_peered":"2023-10-16T14:36:27.781508+0000","last_unstale":"2023-10-16T14:36:28.752180+0000","last_undegraded":"2023-10-16T14:36:28.752180+0000","last_fullsized":"2023-10-16T14:36:28.752180+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:33.145703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749010+0000","last_change":"2023-10-16T14:36:27.772286+0000","last_active":"2023-10-16T14:36:28.749010+0000","last_peered":"2023-10-16T14:36:28.749010+0000","last_clean":"2023-10-16T14:36:28.749010+0000","last_became_active":"2023-10-16T14:36:27.771962+0000","last_became_peered":"2023-10-16T14:36:27.771962+0000","last_unstale":"2023-10-16T14:36:28.749010+0000","last_undegraded":"2023-10-16T14:36:28.749010+0000","last_fullsized":"2023-10-16T14:36:28.749010+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:43.534949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754769+0000","last_change":"2023-10-16T14:36:27.783580+0000","last_active":"2023-10-16T14:36:28.754769+0000","last_peered":"2023-10-16T14:36:28.754769+0000","last_clean":"2023-10-16T14:36:28.754769+0000","last_became_active":"2023-10-16T14:36:27.783239+0000","last_became_peered":"2023-10-16T14:36:27.783239+0000","last_unstale":"2023-10-16T14:36:28.754769+0000","last_undegraded":"2023-10-16T14:36:28.754769+0000","last_fullsized":"2023-10-16T14:36:28.754769+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:31.785329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979738+0000","last_change":"2023-10-16T14:36:27.783953+0000","last_active":"2023-10-16T14:36:28.979738+0000","last_peered":"2023-10-16T14:36:28.979738+0000","last_clean":"2023-10-16T14:36:28.979738+0000","last_became_active":"2023-10-16T14:36:27.783634+0000","last_became_peered":"2023-10-16T14:36:27.783634+0000","last_unstale":"2023-10-16T14:36:28.979738+0000","last_undegraded":"2023-10-16T14:36:28.979738+0000","last_fullsized":"2023-10-16T14:36:28.979738+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:54.673942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978222+0000","last_change":"2023-10-16T14:36:27.788967+0000","last_active":"2023-10-16T14:36:28.978222+0000","last_peered":"2023-10-16T14:36:28.978222+0000","last_clean":"2023-10-16T14:36:28.978222+0000","last_became_active":"2023-10-16T14:36:27.788675+0000","last_became_peered":"2023-10-16T14:36:27.788675+0000","last_unstale":"2023-10-16T14:36:28.978222+0000","last_undegraded":"2023-10-16T14:36:28.978222+0000","last_fullsized":"2023-10-16T14:36:28.978222+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:26.248815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.750084+0000","last_change":"2023-10-16T14:36:27.784199+0000","last_active":"2023-10-16T14:36:28.750084+0000","last_peered":"2023-10-16T14:36:28.750084+0000","last_clean":"2023-10-16T14:36:28.750084+0000","last_became_active":"2023-10-16T14:36:27.782900+0000","last_became_peered":"2023-10-16T14:36:27.782900+0000","last_unstale":"2023-10-16T14:36:28.750084+0000","last_undegraded":"2023-10-16T14:36:28.750084+0000","last_fullsized":"2023-10-16T14:36:28.750084+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:13.000863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753232+0000","last_change":"2023-10-16T14:36:27.782181+0000","last_active":"2023-10-16T14:36:28.753232+0000","last_peered":"2023-10-16T14:36:28.753232+0000","last_clean":"2023-10-16T14:36:28.753232+0000","last_became_active":"2023-10-16T14:36:27.781896+0000","last_became_peered":"2023-10-16T14:36:27.781896+0000","last_unstale":"2023-10-16T14:36:28.753232+0000","last_undegraded":"2023-10-16T14:36:28.753232+0000","last_fullsized":"2023-10-16T14:36:28.753232+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:01.682592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976548+0000","last_change":"2023-10-16T14:36:27.761897+0000","last_active":"2023-10-16T14:36:28.976548+0000","last_peered":"2023-10-16T14:36:28.976548+0000","last_clean":"2023-10-16T14:36:28.976548+0000","last_became_active":"2023-10-16T14:36:27.761120+0000","last_became_peered":"2023-10-16T14:36:27.761120+0000","last_unstale":"2023-10-16T14:36:28.976548+0000","last_undegraded":"2023-10-16T14:36:28.976548+0000","last_fullsized":"2023-10-16T14:36:28.976548+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:33.145703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978390+0000","last_change":"2023-10-16T14:36:27.783309+0000","last_active":"2023-10-16T14:36:28.978390+0000","last_peered":"2023-10-16T14:36:28.978390+0000","last_clean":"2023-10-16T14:36:28.978390+0000","last_became_active":"2023-10-16T14:36:27.783052+0000","last_became_peered":"2023-10-16T14:36:27.783052+0000","last_unstale":"2023-10-16T14:36:28.978390+0000","last_undegraded":"2023-10-16T14:36:28.978390+0000","last_fullsized":"2023-10-16T14:36:28.978390+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:52.129464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976909+0000","last_change":"2023-10-16T14:36:27.788383+0000","last_active":"2023-10-16T14:36:28.976909+0000","last_peered":"2023-10-16T14:36:28.976909+0000","last_clean":"2023-10-16T14:36:28.976909+0000","last_became_active":"2023-10-16T14:36:27.788090+0000","last_became_peered":"2023-10-16T14:36:27.788090+0000","last_unstale":"2023-10-16T14:36:28.976909+0000","last_undegraded":"2023-10-16T14:36:28.976909+0000","last_fullsized":"2023-10-16T14:36:28.976909+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:44.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977646+0000","last_change":"2023-10-16T14:36:27.787323+0000","last_active":"2023-10-16T14:36:28.977646+0000","last_peered":"2023-10-16T14:36:28.977646+0000","last_clean":"2023-10-16T14:36:28.977646+0000","last_became_active":"2023-10-16T14:36:27.787034+0000","last_became_peered":"2023-10-16T14:36:27.787034+0000","last_unstale":"2023-10-16T14:36:28.977646+0000","last_undegraded":"2023-10-16T14:36:28.977646+0000","last_fullsized":"2023-10-16T14:36:28.977646+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.112940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"141'4","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977834+0000","last_change":"2023-10-16T14:36:26.749938+0000","last_active":"2023-10-16T14:36:28.977834+0000","last_peered":"2023-10-16T14:36:28.977834+0000","last_clean":"2023-10-16T14:36:28.977834+0000","last_became_active":"2023-10-16T14:36:26.749655+0000","last_became_peered":"2023-10-16T14:36:26.749655+0000","last_unstale":"2023-10-16T14:36:28.977834+0000","last_undegraded":"2023-10-16T14:36:28.977834+0000","last_fullsized":"2023-10-16T14:36:28.977834+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:07.455720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977447+0000","last_change":"2023-10-16T14:36:27.787031+0000","last_active":"2023-10-16T14:36:28.977447+0000","last_peered":"2023-10-16T14:36:28.977447+0000","last_clean":"2023-10-16T14:36:28.977447+0000","last_became_active":"2023-10-16T14:36:27.786460+0000","last_became_peered":"2023-10-16T14:36:27.786460+0000","last_unstale":"2023-10-16T14:36:28.977447+0000","last_undegraded":"2023-10-16T14:36:28.977447+0000","last_fullsized":"2023-10-16T14:36:28.977447+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:56.720480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755326+0000","last_change":"2023-10-16T14:36:26.749014+0000","last_active":"2023-10-16T14:36:28.755326+0000","last_peered":"2023-10-16T14:36:28.755326+0000","last_clean":"2023-10-16T14:36:28.755326+0000","last_became_active":"2023-10-16T14:36:26.747877+0000","last_became_peered":"2023-10-16T14:36:26.747877+0000","last_unstale":"2023-10-16T14:36:28.755326+0000","last_undegraded":"2023-10-16T14:36:28.755326+0000","last_fullsized":"2023-10-16T14:36:28.755326+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:52.667891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978594+0000","last_change":"2023-10-16T14:36:27.780371+0000","last_active":"2023-10-16T14:36:28.978594+0000","last_peered":"2023-10-16T14:36:28.978594+0000","last_clean":"2023-10-16T14:36:28.978594+0000","last_became_active":"2023-10-16T14:36:27.780093+0000","last_became_peered":"2023-10-16T14:36:27.780093+0000","last_unstale":"2023-10-16T14:36:28.978594+0000","last_undegraded":"2023-10-16T14:36:28.978594+0000","last_fullsized":"2023-10-16T14:36:28.978594+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:21.414718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975687+0000","last_change":"2023-10-16T14:36:26.749842+0000","last_active":"2023-10-16T14:36:28.975687+0000","last_peered":"2023-10-16T14:36:28.975687+0000","last_clean":"2023-10-16T14:36:28.975687+0000","last_became_active":"2023-10-16T14:36:26.749568+0000","last_became_peered":"2023-10-16T14:36:26.749568+0000","last_unstale":"2023-10-16T14:36:28.975687+0000","last_undegraded":"2023-10-16T14:36:28.975687+0000","last_fullsized":"2023-10-16T14:36:28.975687+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:56.216001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755343+0000","last_change":"2023-10-16T14:36:27.785478+0000","last_active":"2023-10-16T14:36:28.755343+0000","last_peered":"2023-10-16T14:36:28.755343+0000","last_clean":"2023-10-16T14:36:28.755343+0000","last_became_active":"2023-10-16T14:36:27.785150+0000","last_became_peered":"2023-10-16T14:36:27.785150+0000","last_unstale":"2023-10-16T14:36:28.755343+0000","last_undegraded":"2023-10-16T14:36:28.755343+0000","last_fullsized":"2023-10-16T14:36:28.755343+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:52.512886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975423+0000","last_change":"2023-10-16T14:36:26.747805+0000","last_active":"2023-10-16T14:36:28.975423+0000","last_peered":"2023-10-16T14:36:28.975423+0000","last_clean":"2023-10-16T14:36:28.975423+0000","last_became_active":"2023-10-16T14:36:26.745642+0000","last_became_peered":"2023-10-16T14:36:26.745642+0000","last_unstale":"2023-10-16T14:36:28.975423+0000","last_undegraded":"2023-10-16T14:36:28.975423+0000","last_fullsized":"2023-10-16T14:36:28.975423+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:41.919206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978157+0000","last_change":"2023-10-16T14:36:27.792877+0000","last_active":"2023-10-16T14:36:28.978157+0000","last_peered":"2023-10-16T14:36:28.978157+0000","last_clean":"2023-10-16T14:36:28.978157+0000","last_became_active":"2023-10-16T14:36:27.791711+0000","last_became_peered":"2023-10-16T14:36:27.791711+0000","last_unstale":"2023-10-16T14:36:28.978157+0000","last_undegraded":"2023-10-16T14:36:28.978157+0000","last_fullsized":"2023-10-16T14:36:28.978157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:48.116343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976836+0000","last_change":"2023-10-16T14:36:26.749195+0000","last_active":"2023-10-16T14:36:28.976836+0000","last_peered":"2023-10-16T14:36:28.976836+0000","last_clean":"2023-10-16T14:36:28.976836+0000","last_became_active":"2023-10-16T14:36:26.748894+0000","last_became_peered":"2023-10-16T14:36:26.748894+0000","last_unstale":"2023-10-16T14:36:28.976836+0000","last_undegraded":"2023-10-16T14:36:28.976836+0000","last_fullsized":"2023-10-16T14:36:28.976836+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:12.749653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753633+0000","last_change":"2023-10-16T14:36:27.787874+0000","last_active":"2023-10-16T14:36:28.753633+0000","last_peered":"2023-10-16T14:36:28.753633+0000","last_clean":"2023-10-16T14:36:28.753633+0000","last_became_active":"2023-10-16T14:36:27.787587+0000","last_became_peered":"2023-10-16T14:36:27.787587+0000","last_unstale":"2023-10-16T14:36:28.753633+0000","last_undegraded":"2023-10-16T14:36:28.753633+0000","last_fullsized":"2023-10-16T14:36:28.753633+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:34.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749691+0000","last_change":"2023-10-16T14:36:26.754082+0000","last_active":"2023-10-16T14:36:28.749691+0000","last_peered":"2023-10-16T14:36:28.749691+0000","last_clean":"2023-10-16T14:36:28.749691+0000","last_became_active":"2023-10-16T14:36:26.753160+0000","last_became_peered":"2023-10-16T14:36:26.753160+0000","last_unstale":"2023-10-16T14:36:28.749691+0000","last_undegraded":"2023-10-16T14:36:28.749691+0000","last_fullsized":"2023-10-16T14:36:28.749691+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:13.163619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977146+0000","last_change":"2023-10-16T14:36:27.796683+0000","last_active":"2023-10-16T14:36:28.977146+0000","last_peered":"2023-10-16T14:36:28.977146+0000","last_clean":"2023-10-16T14:36:28.977146+0000","last_became_active":"2023-10-16T14:36:27.796386+0000","last_became_peered":"2023-10-16T14:36:27.796386+0000","last_unstale":"2023-10-16T14:36:28.977146+0000","last_undegraded":"2023-10-16T14:36:28.977146+0000","last_fullsized":"2023-10-16T14:36:28.977146+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:27.862098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752462+0000","last_change":"2023-10-16T14:36:26.753260+0000","last_active":"2023-10-16T14:36:28.752462+0000","last_peered":"2023-10-16T14:36:28.752462+0000","last_clean":"2023-10-16T14:36:28.752462+0000","last_became_active":"2023-10-16T14:36:26.752872+0000","last_became_peered":"2023-10-16T14:36:26.752872+0000","last_unstale":"2023-10-16T14:36:28.752462+0000","last_undegraded":"2023-10-16T14:36:28.752462+0000","last_fullsized":"2023-10-16T14:36:28.752462+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:17.308544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976214+0000","last_change":"2023-10-16T14:36:27.780261+0000","last_active":"2023-10-16T14:36:28.976214+0000","last_peered":"2023-10-16T14:36:28.976214+0000","last_clean":"2023-10-16T14:36:28.976214+0000","last_became_active":"2023-10-16T14:36:27.778107+0000","last_became_peered":"2023-10-16T14:36:27.778107+0000","last_unstale":"2023-10-16T14:36:28.976214+0000","last_undegraded":"2023-10-16T14:36:28.976214+0000","last_fullsized":"2023-10-16T14:36:28.976214+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:24.999972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977448+0000","last_change":"2023-10-16T14:36:26.755025+0000","last_active":"2023-10-16T14:36:28.977448+0000","last_peered":"2023-10-16T14:36:28.977448+0000","last_clean":"2023-10-16T14:36:28.977448+0000","last_became_active":"2023-10-16T14:36:26.754753+0000","last_became_peered":"2023-10-16T14:36:26.754753+0000","last_unstale":"2023-10-16T14:36:28.977448+0000","last_undegraded":"2023-10-16T14:36:28.977448+0000","last_fullsized":"2023-10-16T14:36:28.977448+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:47.781398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977065+0000","last_change":"2023-10-16T14:36:27.778669+0000","last_active":"2023-10-16T14:36:28.977065+0000","last_peered":"2023-10-16T14:36:28.977065+0000","last_clean":"2023-10-16T14:36:28.977065+0000","last_became_active":"2023-10-16T14:36:27.778370+0000","last_became_peered":"2023-10-16T14:36:27.778370+0000","last_unstale":"2023-10-16T14:36:28.977065+0000","last_undegraded":"2023-10-16T14:36:28.977065+0000","last_fullsized":"2023-10-16T14:36:28.977065+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:01.682592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975494+0000","last_change":"2023-10-16T14:36:26.750634+0000","last_active":"2023-10-16T14:36:28.975494+0000","last_peered":"2023-10-16T14:36:28.975494+0000","last_clean":"2023-10-16T14:36:28.975494+0000","last_became_active":"2023-10-16T14:36:26.750347+0000","last_became_peered":"2023-10-16T14:36:26.750347+0000","last_unstale":"2023-10-16T14:36:28.975494+0000","last_undegraded":"2023-10-16T14:36:28.975494+0000","last_fullsized":"2023-10-16T14:36:28.975494+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:58.108598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977842+0000","last_change":"2023-10-16T14:36:27.792903+0000","last_active":"2023-10-16T14:36:28.977842+0000","last_peered":"2023-10-16T14:36:28.977842+0000","last_clean":"2023-10-16T14:36:28.977842+0000","last_became_active":"2023-10-16T14:36:27.792568+0000","last_became_peered":"2023-10-16T14:36:27.792568+0000","last_unstale":"2023-10-16T14:36:28.977842+0000","last_undegraded":"2023-10-16T14:36:28.977842+0000","last_fullsized":"2023-10-16T14:36:28.977842+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:42.923548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749636+0000","last_change":"2023-10-16T14:36:26.752797+0000","last_active":"2023-10-16T14:36:28.749636+0000","last_peered":"2023-10-16T14:36:28.749636+0000","last_clean":"2023-10-16T14:36:28.749636+0000","last_became_active":"2023-10-16T14:36:26.752474+0000","last_became_peered":"2023-10-16T14:36:26.752474+0000","last_unstale":"2023-10-16T14:36:28.749636+0000","last_undegraded":"2023-10-16T14:36:28.749636+0000","last_fullsized":"2023-10-16T14:36:28.749636+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:16.781966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.750411+0000","last_change":"2023-10-16T14:36:27.775045+0000","last_active":"2023-10-16T14:36:28.750411+0000","last_peered":"2023-10-16T14:36:28.750411+0000","last_clean":"2023-10-16T14:36:28.750411+0000","last_became_active":"2023-10-16T14:36:27.774765+0000","last_became_peered":"2023-10-16T14:36:27.774765+0000","last_unstale":"2023-10-16T14:36:28.750411+0000","last_undegraded":"2023-10-16T14:36:28.750411+0000","last_fullsized":"2023-10-16T14:36:28.750411+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:05.560805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"141'3","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979012+0000","last_change":"2023-10-16T14:36:26.752645+0000","last_active":"2023-10-16T14:36:28.979012+0000","last_peered":"2023-10-16T14:36:28.979012+0000","last_clean":"2023-10-16T14:36:28.979012+0000","last_became_active":"2023-10-16T14:36:26.752282+0000","last_became_peered":"2023-10-16T14:36:26.752282+0000","last_unstale":"2023-10-16T14:36:28.979012+0000","last_undegraded":"2023-10-16T14:36:28.979012+0000","last_fullsized":"2023-10-16T14:36:28.979012+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:13.984175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976924+0000","last_change":"2023-10-16T14:36:27.767176+0000","last_active":"2023-10-16T14:36:28.976924+0000","last_peered":"2023-10-16T14:36:28.976924+0000","last_clean":"2023-10-16T14:36:28.976924+0000","last_became_active":"2023-10-16T14:36:27.766565+0000","last_became_peered":"2023-10-16T14:36:27.766565+0000","last_unstale":"2023-10-16T14:36:28.976924+0000","last_undegraded":"2023-10-16T14:36:28.976924+0000","last_fullsized":"2023-10-16T14:36:28.976924+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:38.458834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"141'6","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979908+0000","last_change":"2023-10-16T14:36:26.758644+0000","last_active":"2023-10-16T14:36:28.979908+0000","last_peered":"2023-10-16T14:36:28.979908+0000","last_clean":"2023-10-16T14:36:28.979908+0000","last_became_active":"2023-10-16T14:36:26.757865+0000","last_became_peered":"2023-10-16T14:36:26.757865+0000","last_unstale":"2023-10-16T14:36:28.979908+0000","last_undegraded":"2023-10-16T14:36:28.979908+0000","last_fullsized":"2023-10-16T14:36:28.979908+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:39.940546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749555+0000","last_change":"2023-10-16T14:36:27.763855+0000","last_active":"2023-10-16T14:36:28.749555+0000","last_peered":"2023-10-16T14:36:28.749555+0000","last_clean":"2023-10-16T14:36:28.749555+0000","last_became_active":"2023-10-16T14:36:27.761283+0000","last_became_peered":"2023-10-16T14:36:27.761283+0000","last_unstale":"2023-10-16T14:36:28.749555+0000","last_undegraded":"2023-10-16T14:36:28.749555+0000","last_fullsized":"2023-10-16T14:36:28.749555+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:33.409206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977751+0000","last_change":"2023-10-16T14:36:26.734712+0000","last_active":"2023-10-16T14:36:28.977751+0000","last_peered":"2023-10-16T14:36:28.977751+0000","last_clean":"2023-10-16T14:36:28.977751+0000","last_became_active":"2023-10-16T14:36:26.734325+0000","last_became_peered":"2023-10-16T14:36:26.734325+0000","last_unstale":"2023-10-16T14:36:28.977751+0000","last_undegraded":"2023-10-16T14:36:28.977751+0000","last_fullsized":"2023-10-16T14:36:28.977751+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:35.885229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749858+0000","last_change":"2023-10-16T14:36:27.767104+0000","last_active":"2023-10-16T14:36:28.749858+0000","last_peered":"2023-10-16T14:36:28.749858+0000","last_clean":"2023-10-16T14:36:28.749858+0000","last_became_active":"2023-10-16T14:36:27.765854+0000","last_became_peered":"2023-10-16T14:36:27.765854+0000","last_unstale":"2023-10-16T14:36:28.749858+0000","last_undegraded":"2023-10-16T14:36:28.749858+0000","last_fullsized":"2023-10-16T14:36:28.749858+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:48.161425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977352+0000","last_change":"2023-10-16T14:36:26.753817+0000","last_active":"2023-10-16T14:36:28.977352+0000","last_peered":"2023-10-16T14:36:28.977352+0000","last_clean":"2023-10-16T14:36:28.977352+0000","last_became_active":"2023-10-16T14:36:26.753473+0000","last_became_peered":"2023-10-16T14:36:26.753473+0000","last_unstale":"2023-10-16T14:36:28.977352+0000","last_undegraded":"2023-10-16T14:36:28.977352+0000","last_fullsized":"2023-10-16T14:36:28.977352+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:21.659431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749159+0000","last_change":"2023-10-16T14:36:27.774638+0000","last_active":"2023-10-16T14:36:28.749159+0000","last_peered":"2023-10-16T14:36:28.749159+0000","last_clean":"2023-10-16T14:36:28.749159+0000","last_became_active":"2023-10-16T14:36:27.774161+0000","last_became_peered":"2023-10-16T14:36:27.774161+0000","last_unstale":"2023-10-16T14:36:28.749159+0000","last_undegraded":"2023-10-16T14:36:28.749159+0000","last_fullsized":"2023-10-16T14:36:28.749159+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:34.929978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"141'8","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.974901+0000","last_change":"2023-10-16T14:36:26.754976+0000","last_active":"2023-10-16T14:36:28.974901+0000","last_peered":"2023-10-16T14:36:28.974901+0000","last_clean":"2023-10-16T14:36:28.974901+0000","last_became_active":"2023-10-16T14:36:26.754677+0000","last_became_peered":"2023-10-16T14:36:26.754677+0000","last_unstale":"2023-10-16T14:36:28.974901+0000","last_undegraded":"2023-10-16T14:36:28.974901+0000","last_fullsized":"2023-10-16T14:36:28.974901+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:45.646833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976728+0000","last_change":"2023-10-16T14:36:27.794441+0000","last_active":"2023-10-16T14:36:28.976728+0000","last_peered":"2023-10-16T14:36:28.976728+0000","last_clean":"2023-10-16T14:36:28.976728+0000","last_became_active":"2023-10-16T14:36:27.794160+0000","last_became_peered":"2023-10-16T14:36:27.794160+0000","last_unstale":"2023-10-16T14:36:28.976728+0000","last_undegraded":"2023-10-16T14:36:28.976728+0000","last_fullsized":"2023-10-16T14:36:28.976728+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:19.858100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979397+0000","last_change":"2023-10-16T14:36:26.753759+0000","last_active":"2023-10-16T14:36:28.979397+0000","last_peered":"2023-10-16T14:36:28.979397+0000","last_clean":"2023-10-16T14:36:28.979397+0000","last_became_active":"2023-10-16T14:36:26.753484+0000","last_became_peered":"2023-10-16T14:36:26.753484+0000","last_unstale":"2023-10-16T14:36:28.979397+0000","last_undegraded":"2023-10-16T14:36:28.979397+0000","last_fullsized":"2023-10-16T14:36:28.979397+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:02.386929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977483+0000","last_change":"2023-10-16T14:36:27.798590+0000","last_active":"2023-10-16T14:36:28.977483+0000","last_peered":"2023-10-16T14:36:28.977483+0000","last_clean":"2023-10-16T14:36:28.977483+0000","last_became_active":"2023-10-16T14:36:27.797341+0000","last_became_peered":"2023-10-16T14:36:27.797341+0000","last_unstale":"2023-10-16T14:36:28.977483+0000","last_undegraded":"2023-10-16T14:36:28.977483+0000","last_fullsized":"2023-10-16T14:36:28.977483+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:57.220343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976420+0000","last_change":"2023-10-16T14:36:26.743860+0000","last_active":"2023-10-16T14:36:28.976420+0000","last_peered":"2023-10-16T14:36:28.976420+0000","last_clean":"2023-10-16T14:36:28.976420+0000","last_became_active":"2023-10-16T14:36:26.743520+0000","last_became_peered":"2023-10-16T14:36:26.743520+0000","last_unstale":"2023-10-16T14:36:28.976420+0000","last_undegraded":"2023-10-16T14:36:28.976420+0000","last_fullsized":"2023-10-16T14:36:28.976420+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:25.244473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754949+0000","last_change":"2023-10-16T14:36:27.783330+0000","last_active":"2023-10-16T14:36:28.754949+0000","last_peered":"2023-10-16T14:36:28.754949+0000","last_clean":"2023-10-16T14:36:28.754949+0000","last_became_active":"2023-10-16T14:36:27.782716+0000","last_became_peered":"2023-10-16T14:36:27.782716+0000","last_unstale":"2023-10-16T14:36:28.754949+0000","last_undegraded":"2023-10-16T14:36:28.754949+0000","last_fullsized":"2023-10-16T14:36:28.754949+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:56.264242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978037+0000","last_change":"2023-10-16T14:36:26.756244+0000","last_active":"2023-10-16T14:36:28.978037+0000","last_peered":"2023-10-16T14:36:28.978037+0000","last_clean":"2023-10-16T14:36:28.978037+0000","last_became_active":"2023-10-16T14:36:26.755990+0000","last_became_peered":"2023-10-16T14:36:26.755990+0000","last_unstale":"2023-10-16T14:36:28.978037+0000","last_undegraded":"2023-10-16T14:36:28.978037+0000","last_fullsized":"2023-10-16T14:36:28.978037+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:14.654281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755640+0000","last_change":"2023-10-16T14:36:27.770592+0000","last_active":"2023-10-16T14:36:28.755640+0000","last_peered":"2023-10-16T14:36:28.755640+0000","last_clean":"2023-10-16T14:36:28.755640+0000","last_became_active":"2023-10-16T14:36:27.770236+0000","last_became_peered":"2023-10-16T14:36:27.770236+0000","last_unstale":"2023-10-16T14:36:28.755640+0000","last_undegraded":"2023-10-16T14:36:28.755640+0000","last_fullsized":"2023-10-16T14:36:28.755640+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:50:28.693112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977703+0000","last_change":"2023-10-16T14:36:26.754995+0000","last_active":"2023-10-16T14:36:28.977703+0000","last_peered":"2023-10-16T14:36:28.977703+0000","last_clean":"2023-10-16T14:36:28.977703+0000","last_became_active":"2023-10-16T14:36:26.754750+0000","last_became_peered":"2023-10-16T14:36:26.754750+0000","last_unstale":"2023-10-16T14:36:28.977703+0000","last_undegraded":"2023-10-16T14:36:28.977703+0000","last_fullsized":"2023-10-16T14:36:28.977703+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:03.072287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754250+0000","last_change":"2023-10-16T14:36:27.775061+0000","last_active":"2023-10-16T14:36:28.754250+0000","last_peered":"2023-10-16T14:36:28.754250+0000","last_clean":"2023-10-16T14:36:28.754250+0000","last_became_active":"2023-10-16T14:36:27.774684+0000","last_became_peered":"2023-10-16T14:36:27.774684+0000","last_unstale":"2023-10-16T14:36:28.754250+0000","last_undegraded":"2023-10-16T14:36:28.754250+0000","last_fullsized":"2023-10-16T14:36:28.754250+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:31.327320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977845+0000","last_change":"2023-10-16T14:36:26.743410+0000","last_active":"2023-10-16T14:36:28.977845+0000","last_peered":"2023-10-16T14:36:28.977845+0000","last_clean":"2023-10-16T14:36:28.977845+0000","last_became_active":"2023-10-16T14:36:26.742404+0000","last_became_peered":"2023-10-16T14:36:26.742404+0000","last_unstale":"2023-10-16T14:36:28.977845+0000","last_undegraded":"2023-10-16T14:36:28.977845+0000","last_fullsized":"2023-10-16T14:36:28.977845+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:52.649310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978491+0000","last_change":"2023-10-16T14:36:27.775955+0000","last_active":"2023-10-16T14:36:28.978491+0000","last_peered":"2023-10-16T14:36:28.978491+0000","last_clean":"2023-10-16T14:36:28.978491+0000","last_became_active":"2023-10-16T14:36:27.775231+0000","last_became_peered":"2023-10-16T14:36:27.775231+0000","last_unstale":"2023-10-16T14:36:28.978491+0000","last_undegraded":"2023-10-16T14:36:28.978491+0000","last_fullsized":"2023-10-16T14:36:28.978491+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:51.088976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.974931+0000","last_change":"2023-10-16T14:36:26.749567+0000","last_active":"2023-10-16T14:36:28.974931+0000","last_peered":"2023-10-16T14:36:28.974931+0000","last_clean":"2023-10-16T14:36:28.974931+0000","last_became_active":"2023-10-16T14:36:26.749280+0000","last_became_peered":"2023-10-16T14:36:26.749280+0000","last_unstale":"2023-10-16T14:36:28.974931+0000","last_undegraded":"2023-10-16T14:36:28.974931+0000","last_fullsized":"2023-10-16T14:36:28.974931+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:55.716138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752612+0000","last_change":"2023-10-16T14:36:27.780120+0000","last_active":"2023-10-16T14:36:28.752612+0000","last_peered":"2023-10-16T14:36:28.752612+0000","last_clean":"2023-10-16T14:36:28.752612+0000","last_became_active":"2023-10-16T14:36:27.777388+0000","last_became_peered":"2023-10-16T14:36:27.777388+0000","last_unstale":"2023-10-16T14:36:28.752612+0000","last_undegraded":"2023-10-16T14:36:28.752612+0000","last_fullsized":"2023-10-16T14:36:28.752612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:13.000863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976993+0000","last_change":"2023-10-16T14:36:26.741753+0000","last_active":"2023-10-16T14:36:28.976993+0000","last_peered":"2023-10-16T14:36:28.976993+0000","last_clean":"2023-10-16T14:36:28.976993+0000","last_became_active":"2023-10-16T14:36:26.741468+0000","last_became_peered":"2023-10-16T14:36:26.741468+0000","last_unstale":"2023-10-16T14:36:28.976993+0000","last_undegraded":"2023-10-16T14:36:28.976993+0000","last_fullsized":"2023-10-16T14:36:28.976993+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:57.682605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749230+0000","last_change":"2023-10-16T14:36:27.774454+0000","last_active":"2023-10-16T14:36:28.749230+0000","last_peered":"2023-10-16T14:36:28.749230+0000","last_clean":"2023-10-16T14:36:28.749230+0000","last_became_active":"2023-10-16T14:36:27.773746+0000","last_became_peered":"2023-10-16T14:36:27.773746+0000","last_unstale":"2023-10-16T14:36:28.749230+0000","last_undegraded":"2023-10-16T14:36:28.749230+0000","last_fullsized":"2023-10-16T14:36:28.749230+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:04.069407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977617+0000","last_change":"2023-10-16T14:36:26.750386+0000","last_active":"2023-10-16T14:36:28.977617+0000","last_peered":"2023-10-16T14:36:28.977617+0000","last_clean":"2023-10-16T14:36:28.977617+0000","last_became_active":"2023-10-16T14:36:26.750067+0000","last_became_peered":"2023-10-16T14:36:26.750067+0000","last_unstale":"2023-10-16T14:36:28.977617+0000","last_undegraded":"2023-10-16T14:36:28.977617+0000","last_fullsized":"2023-10-16T14:36:28.977617+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:02.110045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977378+0000","last_change":"2023-10-16T14:36:27.795812+0000","last_active":"2023-10-16T14:36:28.977378+0000","last_peered":"2023-10-16T14:36:28.977378+0000","last_clean":"2023-10-16T14:36:28.977378+0000","last_became_active":"2023-10-16T14:36:27.795286+0000","last_became_peered":"2023-10-16T14:36:27.795286+0000","last_unstale":"2023-10-16T14:36:28.977378+0000","last_undegraded":"2023-10-16T14:36:28.977378+0000","last_fullsized":"2023-10-16T14:36:28.977378+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:42.731559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749901+0000","last_change":"2023-10-16T14:36:26.754041+0000","last_active":"2023-10-16T14:36:28.749901+0000","last_peered":"2023-10-16T14:36:28.749901+0000","last_clean":"2023-10-16T14:36:28.749901+0000","last_became_active":"2023-10-16T14:36:26.753750+0000","last_became_peered":"2023-10-16T14:36:26.753750+0000","last_unstale":"2023-10-16T14:36:28.749901+0000","last_undegraded":"2023-10-16T14:36:28.749901+0000","last_fullsized":"2023-10-16T14:36:28.749901+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:21.746068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979489+0000","last_change":"2023-10-16T14:36:27.779841+0000","last_active":"2023-10-16T14:36:28.979489+0000","last_peered":"2023-10-16T14:36:28.979489+0000","last_clean":"2023-10-16T14:36:28.979489+0000","last_became_active":"2023-10-16T14:36:27.779445+0000","last_became_peered":"2023-10-16T14:36:27.779445+0000","last_unstale":"2023-10-16T14:36:28.979489+0000","last_undegraded":"2023-10-16T14:36:28.979489+0000","last_fullsized":"2023-10-16T14:36:28.979489+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:00.381003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977283+0000","last_change":"2023-10-16T14:36:26.750369+0000","last_active":"2023-10-16T14:36:28.977283+0000","last_peered":"2023-10-16T14:36:28.977283+0000","last_clean":"2023-10-16T14:36:28.977283+0000","last_became_active":"2023-10-16T14:36:26.750007+0000","last_became_peered":"2023-10-16T14:36:26.750007+0000","last_unstale":"2023-10-16T14:36:28.977283+0000","last_undegraded":"2023-10-16T14:36:28.977283+0000","last_fullsized":"2023-10-16T14:36:28.977283+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:57.682605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979808+0000","last_change":"2023-10-16T14:36:27.797078+0000","last_active":"2023-10-16T14:36:28.979808+0000","last_peered":"2023-10-16T14:36:28.979808+0000","last_clean":"2023-10-16T14:36:28.979808+0000","last_became_active":"2023-10-16T14:36:27.796736+0000","last_became_peered":"2023-10-16T14:36:27.796736+0000","last_unstale":"2023-10-16T14:36:28.979808+0000","last_undegraded":"2023-10-16T14:36:28.979808+0000","last_fullsized":"2023-10-16T14:36:28.979808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:13.230454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979050+0000","last_change":"2023-10-16T14:36:26.749885+0000","last_active":"2023-10-16T14:36:28.979050+0000","last_peered":"2023-10-16T14:36:28.979050+0000","last_clean":"2023-10-16T14:36:28.979050+0000","last_became_active":"2023-10-16T14:36:26.749580+0000","last_became_peered":"2023-10-16T14:36:26.749580+0000","last_unstale":"2023-10-16T14:36:28.979050+0000","last_undegraded":"2023-10-16T14:36:28.979050+0000","last_fullsized":"2023-10-16T14:36:28.979050+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:45.850238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977787+0000","last_change":"2023-10-16T14:36:27.787573+0000","last_active":"2023-10-16T14:36:28.977787+0000","last_peered":"2023-10-16T14:36:28.977787+0000","last_clean":"2023-10-16T14:36:28.977787+0000","last_became_active":"2023-10-16T14:36:27.787140+0000","last_became_peered":"2023-10-16T14:36:27.787140+0000","last_unstale":"2023-10-16T14:36:28.977787+0000","last_undegraded":"2023-10-16T14:36:28.977787+0000","last_fullsized":"2023-10-16T14:36:28.977787+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:44.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.750334+0000","last_change":"2023-10-16T14:36:26.750820+0000","last_active":"2023-10-16T14:36:28.750334+0000","last_peered":"2023-10-16T14:36:28.750334+0000","last_clean":"2023-10-16T14:36:28.750334+0000","last_became_active":"2023-10-16T14:36:26.750340+0000","last_became_peered":"2023-10-16T14:36:26.750340+0000","last_unstale":"2023-10-16T14:36:28.750334+0000","last_undegraded":"2023-10-16T14:36:28.750334+0000","last_fullsized":"2023-10-16T14:36:28.750334+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:52.667891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752453+0000","last_change":"2023-10-16T14:36:27.775695+0000","last_active":"2023-10-16T14:36:28.752453+0000","last_peered":"2023-10-16T14:36:28.752453+0000","last_clean":"2023-10-16T14:36:28.752453+0000","last_became_active":"2023-10-16T14:36:27.775420+0000","last_became_peered":"2023-10-16T14:36:27.775420+0000","last_unstale":"2023-10-16T14:36:28.752453+0000","last_undegraded":"2023-10-16T14:36:28.752453+0000","last_fullsized":"2023-10-16T14:36:28.752453+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:38.458834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977056+0000","last_change":"2023-10-16T14:36:26.747263+0000","last_active":"2023-10-16T14:36:28.977056+0000","last_peered":"2023-10-16T14:36:28.977056+0000","last_clean":"2023-10-16T14:36:28.977056+0000","last_became_active":"2023-10-16T14:36:26.746934+0000","last_became_peered":"2023-10-16T14:36:26.746934+0000","last_unstale":"2023-10-16T14:36:28.977056+0000","last_undegraded":"2023-10-16T14:36:28.977056+0000","last_fullsized":"2023-10-16T14:36:28.977056+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:25.447610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753982+0000","last_change":"2023-10-16T14:36:27.782764+0000","last_active":"2023-10-16T14:36:28.753982+0000","last_peered":"2023-10-16T14:36:28.753982+0000","last_clean":"2023-10-16T14:36:28.753982+0000","last_became_active":"2023-10-16T14:36:27.782079+0000","last_became_peered":"2023-10-16T14:36:27.782079+0000","last_unstale":"2023-10-16T14:36:28.753982+0000","last_undegraded":"2023-10-16T14:36:28.753982+0000","last_fullsized":"2023-10-16T14:36:28.753982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:51.150514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752672+0000","last_change":"2023-10-16T14:36:26.736614+0000","last_active":"2023-10-16T14:36:28.752672+0000","last_peered":"2023-10-16T14:36:28.752672+0000","last_clean":"2023-10-16T14:36:28.752672+0000","last_became_active":"2023-10-16T14:36:26.731128+0000","last_became_peered":"2023-10-16T14:36:26.731128+0000","last_unstale":"2023-10-16T14:36:28.752672+0000","last_undegraded":"2023-10-16T14:36:28.752672+0000","last_fullsized":"2023-10-16T14:36:28.752672+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:23.995630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752834+0000","last_change":"2023-10-16T14:36:27.777818+0000","last_active":"2023-10-16T14:36:28.752834+0000","last_peered":"2023-10-16T14:36:28.752834+0000","last_clean":"2023-10-16T14:36:28.752834+0000","last_became_active":"2023-10-16T14:36:27.776011+0000","last_became_peered":"2023-10-16T14:36:27.776011+0000","last_unstale":"2023-10-16T14:36:28.752834+0000","last_undegraded":"2023-10-16T14:36:28.752834+0000","last_fullsized":"2023-10-16T14:36:28.752834+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:05.560805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.974953+0000","last_change":"2023-10-16T14:36:26.750705+0000","last_active":"2023-10-16T14:36:28.974953+0000","last_peered":"2023-10-16T14:36:28.974953+0000","last_clean":"2023-10-16T14:36:28.974953+0000","last_became_active":"2023-10-16T14:36:26.749681+0000","last_became_peered":"2023-10-16T14:36:26.749681+0000","last_unstale":"2023-10-16T14:36:28.974953+0000","last_undegraded":"2023-10-16T14:36:28.974953+0000","last_fullsized":"2023-10-16T14:36:28.974953+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:43.903633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978331+0000","last_change":"2023-10-16T14:36:27.796281+0000","last_active":"2023-10-16T14:36:28.978331+0000","last_peered":"2023-10-16T14:36:28.978331+0000","last_clean":"2023-10-16T14:36:28.978331+0000","last_became_active":"2023-10-16T14:36:27.795993+0000","last_became_peered":"2023-10-16T14:36:27.795993+0000","last_unstale":"2023-10-16T14:36:28.978331+0000","last_undegraded":"2023-10-16T14:36:28.978331+0000","last_fullsized":"2023-10-16T14:36:28.978331+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:55.360430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977608+0000","last_change":"2023-10-16T14:36:26.750338+0000","last_active":"2023-10-16T14:36:28.977608+0000","last_peered":"2023-10-16T14:36:28.977608+0000","last_clean":"2023-10-16T14:36:28.977608+0000","last_became_active":"2023-10-16T14:36:26.749681+0000","last_became_peered":"2023-10-16T14:36:26.749681+0000","last_unstale":"2023-10-16T14:36:28.977608+0000","last_undegraded":"2023-10-16T14:36:28.977608+0000","last_fullsized":"2023-10-16T14:36:28.977608+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:02.175201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749010+0000","last_change":"2023-10-16T14:36:27.770494+0000","last_active":"2023-10-16T14:36:28.749010+0000","last_peered":"2023-10-16T14:36:28.749010+0000","last_clean":"2023-10-16T14:36:28.749010+0000","last_became_active":"2023-10-16T14:36:27.770175+0000","last_became_peered":"2023-10-16T14:36:27.770175+0000","last_unstale":"2023-10-16T14:36:28.749010+0000","last_undegraded":"2023-10-16T14:36:28.749010+0000","last_fullsized":"2023-10-16T14:36:28.749010+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:58.992728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976525+0000","last_change":"2023-10-16T14:36:26.756906+0000","last_active":"2023-10-16T14:36:28.976525+0000","last_peered":"2023-10-16T14:36:28.976525+0000","last_clean":"2023-10-16T14:36:28.976525+0000","last_became_active":"2023-10-16T14:36:26.756560+0000","last_became_peered":"2023-10-16T14:36:26.756560+0000","last_unstale":"2023-10-16T14:36:28.976525+0000","last_undegraded":"2023-10-16T14:36:28.976525+0000","last_fullsized":"2023-10-16T14:36:28.976525+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:17.308544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"130'230","reported_seq":696,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T14:36:27.759180+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:36:27.759180+0000","last_peered":"2023-10-16T14:36:27.759180+0000","last_clean":"2023-10-16T14:36:27.759180+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:36:27.759180+0000","last_undegraded":"2023-10-16T14:36:27.759180+0000","last_fullsized":"2023-10-16T14:36:27.759180+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575087,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41808,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330032,"statfs":{"total":96636764160,"available":96593952768,"internally_reserved":0,"allocated":2363392,"data_stored":1444226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.079,"15min":1.131},"min":{"1min":0.65300000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5289999999999999,"5min":1.833,"15min":1.833},"last":1.181},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.0649999999999999,"15min":1.133},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.712,"5min":1.891,"15min":1.891},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1359999999999999,"15min":1.175},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6060000000000001,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":2.8479999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.22,"15min":1.2270000000000001},"min":{"1min":0.625,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6479999999999999,"5min":2.048,"15min":2.048},"last":1.5069999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.26,"15min":1.3049999999999999},"min":{"1min":0.69199999999999995,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7509999999999999,"5min":1.8839999999999999,"15min":1.9339999999999999},"last":2.2450000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.214,"15min":1.224},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.786,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":2.1269999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.2390000000000001,"15min":1.2929999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.8620000000000001,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":2.677},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.3240000000000001,"15min":1.3560000000000001},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5680000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":2.6120000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.347,"15min":1.363},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.1749999999999998,"5min":2.52,"15min":2.52},"last":1.643},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.264,"15min":1.258},"min":{"1min":0.496,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.117,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":2.0299999999999998}]},{"osd":5,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3700000000000001,"15min":1.401},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9750000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.482,"15min":1.4870000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9910000000000001,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":2.4060000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.4199999999999999,"15min":1.4350000000000001},"min":{"1min":0.61199999999999999,"5min":0.502,"15min":0.502},"max":{"1min":2.2330000000000001,"5min":2.79,"15min":2.79},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":0.63300000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9019999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.774}]}]},{"osd":6,"up_from":13,"seq":55834575086,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1765376,"data_stored":853312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.181,"15min":1.1779999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.5860000000000001,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.081},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2809999999999999,"15min":1.256},"min":{"1min":0.67700000000000005,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.0830000000000002,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":1.6919999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3100000000000001,"15min":1.2849999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6719999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.22,"5min":1.3100000000000001,"15min":1.2969999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7310000000000001,"5min":10.241,"15min":10.241},"last":1.496}]},{"osd":2,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.1639999999999999,"15min":1.204},"min":{"1min":0.70099999999999996,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.1469999999999998,"5min":2.1469999999999998,"15min":2.4940000000000002},"last":1.177},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.343,"15min":1.3520000000000001},"min":{"1min":0.88800000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1920000000000002,"5min":5.694,"15min":5.694},"last":1.5429999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.375,"15min":1.357},"min":{"1min":1.0700000000000001,"5min":0.76300000000000001,"15min":0.71999999999999997},"max":{"1min":1.804,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":1.321},{"interface":"front","average":{"1min":1.331,"5min":1.3009999999999999,"15min":1.325},"min":{"1min":0.90300000000000002,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7709999999999999,"5min":2.2349999999999999,"15min":2.6120000000000001},"last":1.4379999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.4179999999999999,"15min":1.333},"min":{"1min":0.85499999999999998,"5min":0.54200000000000004,"15min":0.48999999999999999},"max":{"1min":1.9359999999999999,"5min":13.826000000000001,"15min":13.826000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.367,"15min":1.3640000000000001},"min":{"1min":1.0309999999999999,"5min":0.63200000000000001,"15min":0.44},"max":{"1min":2.004,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.8380000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.333,"15min":1.3109999999999999},"min":{"1min":0.88100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.857,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5209999999999999,"15min":1.494},"min":{"1min":1.05,"5min":0.59599999999999997,"15min":0.56000000000000005},"max":{"1min":2.202,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.653}]},{"osd":7,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.593,"15min":1.5660000000000001},"min":{"1min":1.135,"5min":0.84099999999999997,"15min":0.58299999999999996},"max":{"1min":2.1400000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.754},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.544,"15min":1.542},"min":{"1min":1.0780000000000001,"5min":0.67500000000000004,"15min":0.54200000000000004},"max":{"1min":2.0459999999999998,"5min":2.1779999999999999,"15min":2.7250000000000001},"last":1.7230000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575086,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1769472,"data_stored":853846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.103,"15min":1.119},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.595,"5min":1.669,"15min":2.2309999999999999},"last":1.129},{"interface":"front","average":{"1min":1.129,"5min":1.163,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.42299999999999999},"max":{"1min":1.589,"5min":1.7729999999999999,"15min":1.8340000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1200000000000001,"15min":1.165},"min":{"1min":0.69599999999999995,"5min":0.622,"15min":0.622},"max":{"1min":1.6559999999999999,"5min":1.6899999999999999,"15min":1.726},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.876,"5min":0.78600000000000003,"15min":0.60399999999999998},"max":{"1min":1.5329999999999999,"5min":1.6140000000000001,"15min":1.7809999999999999},"last":0.877}]},{"osd":2,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.83799999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8260000000000001,"5min":1.913,"15min":2.5009999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.327,"15min":1.347},"min":{"1min":0.78200000000000003,"5min":0.72199999999999998,"15min":0.625},"max":{"1min":1.6739999999999999,"5min":2.0049999999999999,"15min":2.1200000000000001},"last":1.044}]},{"osd":3,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.248,"15min":1.278},"min":{"1min":0.80700000000000005,"5min":0.76900000000000002,"15min":0.45000000000000001},"max":{"1min":1.546,"5min":1.903,"15min":1.903},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.3009999999999999,"15min":1.363},"min":{"1min":0.92800000000000005,"5min":0.78400000000000003,"15min":0.76100000000000001},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.8029999999999999},"last":1.3560000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2789999999999999,"15min":1.321},"min":{"1min":0.80100000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6359999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.391},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4430000000000001,"15min":1.4890000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":4.9480000000000004},"last":1.427}]},{"osd":6,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.327,"15min":1.377},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.47899999999999998},"max":{"1min":1.798,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.214},{"interface":"front","average":{"1min":1.359,"5min":1.3540000000000001,"15min":1.4239999999999999},"min":{"1min":0.90500000000000003,"5min":0.77700000000000002,"15min":0.55000000000000004},"max":{"1min":1.6890000000000001,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.2629999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4950000000000001,"15min":1.5469999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.641},"last":1.635},{"interface":"front","average":{"1min":1.581,"5min":1.5760000000000001,"15min":1.6080000000000001},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.58099999999999996},"max":{"1min":1.8839999999999999,"5min":2.1909999999999998,"15min":2.2010000000000001},"last":1.3120000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575087,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46792,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325048,"statfs":{"total":96636764160,"available":96588849152,"internally_reserved":0,"allocated":1761280,"data_stored":853242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.2170000000000001,"15min":1.1679999999999999},"min":{"1min":0.60799999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":7.9400000000000004,"5min":7.9400000000000004,"15min":7.9400000000000004},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.0920000000000001,"15min":1.1559999999999999},"min":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.58999999999999997},"max":{"1min":1.6379999999999999,"5min":1.7090000000000001,"15min":1.8500000000000001},"last":1.2430000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.27,"15min":1.22},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":16.959,"5min":16.959,"15min":16.959},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.764,"5min":1.3200000000000001,"15min":1.2629999999999999},"min":{"1min":0.67700000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":12.333,"5min":12.333,"15min":12.333},"last":1.5489999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.4119999999999999,"15min":1.413},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.67900000000000005},"max":{"1min":8.0939999999999994,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":1.609},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.3839999999999999,"15min":1.383},"min":{"1min":0.82399999999999995,"5min":0.59199999999999997,"15min":0.51600000000000001},"max":{"1min":8.2899999999999991,"5min":8.2899999999999991,"15min":8.2899999999999991},"last":1.52}]},{"osd":3,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2490000000000001,"15min":1.3069999999999999},"min":{"1min":0.73999999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.595,"5min":1.95,"15min":1.95},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.3049999999999999,"15min":1.3169999999999999},"min":{"1min":0.76100000000000001,"5min":0.53900000000000003,"15min":0.437},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.379,"15min":1.3220000000000001},"min":{"1min":0.627,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":7.8639999999999999,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.3839999999999999,"15min":1.3540000000000001},"min":{"1min":0.85099999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.766,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.411}]},{"osd":6,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.3759999999999999,"15min":1.325},"min":{"1min":0.60599999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.7869999999999999,"5min":1.5469999999999999,"15min":1.4850000000000001},"min":{"1min":0.72699999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":7.9900000000000002,"5min":7.9900000000000002,"15min":7.9900000000000002},"last":1.4410000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5109999999999999,"15min":1.48},"min":{"1min":0.90300000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.913,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.524,"15min":1.4830000000000001},"min":{"1min":0.69999999999999996,"5min":0.66700000000000004,"15min":0.49399999999999999},"max":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"last":1.4890000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575086,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1765376,"data_stored":848596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.40200000000000002,"5min":0.377,"15min":0.28999999999999998},"max":{"1min":1.5740000000000001,"5min":1.696,"15min":1.696},"last":1.367},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93899999999999995,"15min":1.012},"min":{"1min":0.48099999999999998,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":1.5760000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.0940000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.89700000000000002,"15min":0.93300000000000005},"min":{"1min":0.57499999999999996,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.583,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":1.169},{"interface":"front","average":{"1min":1.032,"5min":1.054,"15min":1.123},"min":{"1min":0.53200000000000003,"5min":0.45400000000000001,"15min":0.314},"max":{"1min":1.635,"5min":3.024,"15min":3.024},"last":0.95699999999999996}]},{"osd":2,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1759999999999999,"15min":1.2929999999999999},"min":{"1min":0.84599999999999997,"5min":0.54900000000000004,"15min":0.53800000000000003},"max":{"1min":1.792,"5min":3.3420000000000001,"15min":3.3420000000000001},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.306,"15min":1.3979999999999999},"min":{"1min":0.79000000000000004,"5min":0.53000000000000003,"15min":0.5},"max":{"1min":1.8200000000000001,"5min":2.5609999999999999,"15min":2.6299999999999999},"last":1.6839999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2150000000000001,"15min":1.25},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.56699999999999995},"max":{"1min":1.6890000000000001,"5min":2.02,"15min":2.02},"last":1.9930000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.262,"15min":1.3200000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.629},"max":{"1min":1.4810000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":2.516}]},{"osd":5,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.4199999999999999,"15min":1.5009999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.6200000000000001,"5min":3.206,"15min":3.206},"last":2.3090000000000002},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4079999999999999,"15min":1.4450000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.68799999999999994},"max":{"1min":2.0390000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":2.145}]},{"osd":6,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.3100000000000001,"15min":1.387},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.929,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.391,"15min":1.47},"min":{"1min":0.82699999999999996,"5min":0.82599999999999996,"15min":0.57999999999999996},"max":{"1min":1.78,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":2.391}]},{"osd":7,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5,"15min":1.5640000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.69099999999999995},"max":{"1min":2.2149999999999999,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":3.5649999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.413,"15min":1.5169999999999999},"min":{"1min":0.82099999999999995,"5min":0.749,"15min":0.629},"max":{"1min":2.1789999999999998,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":3.4239999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575087,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":1748992,"data_stored":846914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.88700000000000001,"15min":0.874},"min":{"1min":0.626,"5min":0.38,"15min":0.38},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.872},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.98999999999999999,"15min":1.0189999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":1.617,"5min":1.617,"15min":2.3260000000000001},"last":0.91800000000000004}]},{"osd":1,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98199999999999998,"15min":0.997},"min":{"1min":0.505,"5min":0.40600000000000003,"15min":0.35299999999999998},"max":{"1min":1.7649999999999999,"5min":1.8149999999999999,"15min":3.234},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.171,"5min":1.129,"15min":1.163},"min":{"1min":0.67700000000000005,"5min":0.51200000000000001,"15min":0.42199999999999999},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":3.4740000000000002},"last":1.054}]},{"osd":3,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.161,"15min":1.224},"min":{"1min":0.66700000000000004,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":2.0550000000000002,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.278,"15min":1.3069999999999999},"min":{"1min":0.84799999999999998,"5min":0.59299999999999997,"15min":0.57599999999999996},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":3.5569999999999999},"last":1.1100000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.4319999999999999,"15min":1.4330000000000001},"min":{"1min":1.0269999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.161},{"interface":"front","average":{"1min":1.331,"5min":1.294,"15min":1.343},"min":{"1min":0.754,"5min":0.754,"15min":0.72599999999999998},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":3.016},"last":1.4790000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.397,"15min":1.4139999999999999},"min":{"1min":0.84099999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.141,"5min":2.3650000000000002,"15min":2.4100000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.532,"15min":1.5529999999999999},"min":{"1min":0.995,"5min":0.90400000000000003,"15min":0.79200000000000004},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":3.3769999999999998},"last":1.508}]},{"osd":6,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3819999999999999,"15min":1.46},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.67100000000000004},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":3.0760000000000001},"last":1.25},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5009999999999999,"15min":1.526},"min":{"1min":0.93200000000000005,"5min":0.71299999999999997,"15min":0.65300000000000002},"max":{"1min":2.0230000000000001,"5min":2.5230000000000001,"15min":2.5920000000000001},"last":1.4339999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4610000000000001,"15min":1.5369999999999999},"min":{"1min":0.81299999999999994,"5min":0.70699999999999996,"15min":0.69099999999999995},"max":{"1min":2.226,"5min":2.226,"15min":3.1899999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.649,"5min":1.609,"15min":1.635},"min":{"1min":0.98999999999999999,"5min":0.81999999999999995,"15min":0.80300000000000005},"max":{"1min":2.21,"5min":2.6030000000000002,"15min":2.7519999999999998},"last":1.532}]}]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59272,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312568,"statfs":{"total":96636764160,"available":96576069632,"internally_reserved":0,"allocated":1757184,"data_stored":847038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92400000000000004,"15min":0.91300000000000003},"min":{"1min":0.48299999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.7749999999999999},"last":1.208},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0049999999999999,"15min":1.077},"min":{"1min":0.69899999999999995,"5min":0.40799999999999997,"15min":0.378},"max":{"1min":1.579,"5min":1.827,"15min":1.9370000000000001},"last":0.747}]},{"osd":2,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.93200000000000005,"15min":0.96999999999999997},"min":{"1min":0.42599999999999999,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.792,"5min":1.792,"15min":2.0259999999999998},"last":1.405},{"interface":"front","average":{"1min":1.052,"5min":1.0940000000000001,"15min":1.1499999999999999},"min":{"1min":0.53900000000000003,"5min":0.47099999999999997,"15min":0.45800000000000002},"max":{"1min":1.4339999999999999,"5min":1.6919999999999999,"15min":1.796},"last":1.6859999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2190000000000001,"15min":1.3},"min":{"1min":0.63800000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1339999999999999,"15min":1.1819999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.44600000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.855},"last":0.56999999999999995}]},{"osd":4,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2190000000000001,"15min":1.274},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.56100000000000005},"max":{"1min":1.5820000000000001,"5min":1.958,"15min":2.069},"last":1.042},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.379,"15min":1.4139999999999999},"min":{"1min":0.79500000000000004,"5min":0.74099999999999999,"15min":0.61399999999999999},"max":{"1min":1.7390000000000001,"5min":2.899,"15min":2.899},"last":1.351}]},{"osd":5,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.363,"15min":1.4239999999999999},"min":{"1min":0.92600000000000005,"5min":0.78500000000000003,"15min":0.623},"max":{"1min":2.117,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.169},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.347,"15min":1.415},"min":{"1min":0.76700000000000002,"5min":0.71199999999999997,"15min":0.59499999999999997},"max":{"1min":1.7949999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.2629999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.5209999999999999,"15min":1.5389999999999999},"min":{"1min":0.85399999999999998,"5min":0.80000000000000004,"15min":0.53100000000000003},"max":{"1min":1.839,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.4319999999999999,"15min":1.5189999999999999},"min":{"1min":0.81000000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.3220000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.55,"15min":1.6080000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.2530000000000001,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4470000000000001,"15min":1.536},"min":{"1min":0.76300000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7330000000000001,"5min":2.2000000000000002,"15min":2.2170000000000001},"last":1.508}]}]},{"osd":0,"up_from":13,"seq":55834575087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2363392,"data_stored":1439006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.91300000000000003,"15min":0.93999999999999995},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.871,"5min":3.871,"15min":3.871},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.165,"5min":1.1539999999999999,"15min":1.169},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.726,"5min":3.726,"15min":3.726},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1990000000000001,"15min":1.252},"min":{"1min":0.50900000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.351,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.747},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1599999999999999,"15min":1.1990000000000001},"min":{"1min":0.54300000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.1240000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.96699999999999997,"15min":0.995},"min":{"1min":0.55700000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.109,"5min":3.109,"15min":3.109},"last":1.036},{"interface":"front","average":{"1min":1.169,"5min":1.1850000000000001,"15min":1.2230000000000001},"min":{"1min":0.71499999999999997,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":2.5489999999999999,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.2929999999999999,"15min":1.3069999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":3.4380000000000002,"5min":3.4380000000000002,"15min":3.4380000000000002},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.77000000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.4620000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":1.5329999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4039999999999999,"15min":1.4239999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":4.2119999999999997,"5min":4.2119999999999997,"15min":4.2119999999999997},"last":1.246},{"interface":"front","average":{"1min":1.589,"5min":1.3879999999999999,"15min":1.3779999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"last":1.6100000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.6040000000000001,"15min":1.589},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.7040000000000002,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.478,"15min":1.4990000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.371,"5min":3.371,"15min":3.371},"last":1.3600000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.5469999999999999,"15min":1.583},"min":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.8570000000000002,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.556},{"interface":"front","average":{"1min":1.702,"5min":1.5609999999999999,"15min":1.5569999999999999},"min":{"1min":0.77100000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":5.6189999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"last":1.3}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:36:32.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:36:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.053+0000 7f51c9f4e640 1 Processor -- start 2023-10-16T14:36:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.053+0000 7f51c9f4e640 1 -- start start 2023-10-16T14:36:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.054+0000 7f51c9f4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c415c9f0 0x7f51c40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.054+0000 7f51c9f4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51c40c9230 0x7f51c40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.054+0000 7f51c9f4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c4160a90 con 0x7f51c4070f20 2023-10-16T14:36:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c9f4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c40cdec0 con 0x7f51c40c9230 2023-10-16T14:36:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c9f4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40cd900 con 0x7f51c415c9f0 2023-10-16T14:36:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51c40c9230 0x7f51c40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c974d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f51c4070f20 0x7f51c40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46358/0 (socket says 172.21.15.29:46358) 2023-10-16T14:36:33.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c974d640 1 -- 172.21.15.29:0/1423117611 learned_addr learned my addr 172.21.15.29:0/1423117611 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:33.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c8f4c640 1 --2- 172.21.15.29:0/1423117611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c415c9f0 0x7f51c40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c3fff640 1 -- 172.21.15.29:0/1423117611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c415c9f0 msgr2=0x7f51c40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c3fff640 1 --2- 172.21.15.29:0/1423117611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c415c9f0 0x7f51c40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.055+0000 7f51c3fff640 1 -- 172.21.15.29:0/1423117611 >> v1:172.21.15.29:6789/0 conn(0x7f51c4070f20 legacy=0x7f51c40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.056+0000 7f51c3fff640 1 -- 172.21.15.29:0/1423117611 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40cdbe0 con 0x7f51c40c9230 2023-10-16T14:36:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.057+0000 7f51c3fff640 1 --2- 172.21.15.29:0/1423117611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c40c9230 0x7f51c40709e0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f51b00097a0 tx=0x7f51b002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1c475456af3a44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.057+0000 7f51c37fe640 1 -- 172.21.15.29:0/1423117611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b002d040 con 0x7f51c40c9230 2023-10-16T14:36:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.057+0000 7f51c37fe640 1 -- 172.21.15.29:0/1423117611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b0031810 con 0x7f51c40c9230 2023-10-16T14:36:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.057+0000 7f51c37fe640 1 -- 172.21.15.29:0/1423117611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b0031ba0 con 0x7f51c40c9230 2023-10-16T14:36:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 -- 172.21.15.29:0/1423117611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c40c9230 msgr2=0x7f51c40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/1423117611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c40c9230 0x7f51c40709e0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f51b00097a0 tx=0x7f51b002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 -- 172.21.15.29:0/1423117611 shutdown_connections 2023-10-16T14:36:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/1423117611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c40c9230 0x7f51c40709e0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/1423117611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c415c9f0 0x7f51c40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 -- 172.21.15.29:0/1423117611 >> 172.21.15.29:0/1423117611 conn(0x7f51c40ce6a0 msgr2=0x7f51c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 -- 172.21.15.29:0/1423117611 shutdown_connections 2023-10-16T14:36:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.058+0000 7f51c9f4e640 1 -- 172.21.15.29:0/1423117611 wait complete. 2023-10-16T14:36:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.060+0000 7f51c9f4e640 1 Processor -- start 2023-10-16T14:36:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.060+0000 7f51c9f4e640 1 -- start start 2023-10-16T14:36:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c9f4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 0x7f51c406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c9f4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c40c9230 0x7f51c415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c9f4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c415c9f0 0x7f51c415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c9f4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c3220 con 0x7f51c4070f20 2023-10-16T14:36:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c9f4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c3da0 con 0x7f51c415c9f0 2023-10-16T14:36:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c9f4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40c3470 con 0x7f51c40c9230 2023-10-16T14:36:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c8f4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 0x7f51c406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c8f4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 0x7f51c406c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36570/0 (socket says 172.21.15.29:36570) 2023-10-16T14:36:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.061+0000 7f51c8f4c640 1 -- 172.21.15.29:0/3238961643 learned_addr learned my addr 172.21.15.29:0/3238961643 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c8f4c640 1 -- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c40c9230 msgr2=0x7f51c415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c8f4c640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c40c9230 0x7f51c415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c8f4c640 1 -- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c415c9f0 msgr2=0x7f51c415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c8f4c640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c415c9f0 0x7f51c415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c8f4c640 1 -- 172.21.15.29:0/3238961643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b0009400 con 0x7f51c4070f20 2023-10-16T14:36:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c8f4c640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 0x7f51c406c510 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f51b800d620 tx=0x7f51b800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800b620 con 0x7f51c4070f20 2023-10-16T14:36:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b800b780 con 0x7f51c4070f20 2023-10-16T14:36:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800bb30 con 0x7f51c4070f20 2023-10-16T14:36:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.062+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c40c3750 con 0x7f51c4070f20 2023-10-16T14:36:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.063+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c406ca50 con 0x7f51c4070f20 2023-10-16T14:36:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.064+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c404e9b0 con 0x7f51c4070f20 2023-10-16T14:36:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.065+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f51b800bc90 con 0x7f51c4070f20 2023-10-16T14:36:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.066+0000 7f51c17fa640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f51a806af90 0x7f51a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.066+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f51b8089c50 con 0x7f51c4070f20 2023-10-16T14:36:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.066+0000 7f51c3fff640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f51a806af90 0x7f51a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.067+0000 7f51c3fff640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f51a806af90 0x7f51a806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f51b0036000 tx=0x7f51b0030000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.071+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51b808f020 con 0x7f51c4070f20 2023-10-16T14:36:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.264+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 --> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51c40704c0 con 0x7f51a806af90 2023-10-16T14:36:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.287+0000 7f51c17fa640 1 -- 172.21.15.29:0/3238961643 <== mgr.6790 v2:172.21.15.170:6832/1905872466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287480 (secure 0 0 0) 0x7f51b003a420 con 0x7f51a806af90 2023-10-16T14:36:33.296 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:36:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.292+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f51a806af90 msgr2=0x7f51a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.292+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f51a806af90 0x7f51a806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f51b0036000 tx=0x7f51b0030000 comp rx=0 tx=0).stop 2023-10-16T14:36:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.292+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 msgr2=0x7f51c406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.292+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 0x7f51c406c510 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f51b800d620 tx=0x7f51b800dae0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 shutdown_connections 2023-10-16T14:36:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f51a806af90 0x7f51a806d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c415c9f0 0x7f51c415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c40c9230 0x7f51c415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 --2- 172.21.15.29:0/3238961643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c4070f20 0x7f51c406c510 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 >> 172.21.15.29:0/3238961643 conn(0x7f51c40ce6a0 msgr2=0x7f51c4160f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 shutdown_connections 2023-10-16T14:36:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.293+0000 7f51c9f4e640 1 -- 172.21.15.29:0/3238961643 wait complete. 2023-10-16T14:36:33.318 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T14:36:32.473031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":407,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":351832,"kb_used_data":14936,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754622888,"statfs":{"total":773094113280,"available":772733837312,"internally_reserved":0,"allocated":15294464,"data_stored":7986180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":344837689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002281"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979063+0000","last_change":"2023-10-16T14:36:27.795757+0000","last_active":"2023-10-16T14:36:28.979063+0000","last_peered":"2023-10-16T14:36:28.979063+0000","last_clean":"2023-10-16T14:36:28.979063+0000","last_became_active":"2023-10-16T14:36:27.791202+0000","last_became_peered":"2023-10-16T14:36:27.791202+0000","last_unstale":"2023-10-16T14:36:28.979063+0000","last_undegraded":"2023-10-16T14:36:28.979063+0000","last_fullsized":"2023-10-16T14:36:28.979063+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:14.307117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754126+0000","last_change":"2023-10-16T14:36:27.788137+0000","last_active":"2023-10-16T14:36:28.754126+0000","last_peered":"2023-10-16T14:36:28.754126+0000","last_clean":"2023-10-16T14:36:28.754126+0000","last_became_active":"2023-10-16T14:36:27.787521+0000","last_became_peered":"2023-10-16T14:36:27.787521+0000","last_unstale":"2023-10-16T14:36:28.754126+0000","last_undegraded":"2023-10-16T14:36:28.754126+0000","last_fullsized":"2023-10-16T14:36:28.754126+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:23.090304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754979+0000","last_change":"2023-10-16T14:36:27.774865+0000","last_active":"2023-10-16T14:36:28.754979+0000","last_peered":"2023-10-16T14:36:28.754979+0000","last_clean":"2023-10-16T14:36:28.754979+0000","last_became_active":"2023-10-16T14:36:27.774571+0000","last_became_peered":"2023-10-16T14:36:27.774571+0000","last_unstale":"2023-10-16T14:36:28.754979+0000","last_undegraded":"2023-10-16T14:36:28.754979+0000","last_fullsized":"2023-10-16T14:36:28.754979+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:46.587562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979073+0000","last_change":"2023-10-16T14:36:27.775697+0000","last_active":"2023-10-16T14:36:28.979073+0000","last_peered":"2023-10-16T14:36:28.979073+0000","last_clean":"2023-10-16T14:36:28.979073+0000","last_became_active":"2023-10-16T14:36:27.775301+0000","last_became_peered":"2023-10-16T14:36:27.775301+0000","last_unstale":"2023-10-16T14:36:28.979073+0000","last_undegraded":"2023-10-16T14:36:28.979073+0000","last_fullsized":"2023-10-16T14:36:28.979073+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:55.543011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975826+0000","last_change":"2023-10-16T14:36:27.780715+0000","last_active":"2023-10-16T14:36:28.975826+0000","last_peered":"2023-10-16T14:36:28.975826+0000","last_clean":"2023-10-16T14:36:28.975826+0000","last_became_active":"2023-10-16T14:36:27.779102+0000","last_became_peered":"2023-10-16T14:36:27.779102+0000","last_unstale":"2023-10-16T14:36:28.975826+0000","last_undegraded":"2023-10-16T14:36:28.975826+0000","last_fullsized":"2023-10-16T14:36:28.975826+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:21.414718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977051+0000","last_change":"2023-10-16T14:36:27.785909+0000","last_active":"2023-10-16T14:36:28.977051+0000","last_peered":"2023-10-16T14:36:28.977051+0000","last_clean":"2023-10-16T14:36:28.977051+0000","last_became_active":"2023-10-16T14:36:27.785606+0000","last_became_peered":"2023-10-16T14:36:27.785606+0000","last_unstale":"2023-10-16T14:36:28.977051+0000","last_undegraded":"2023-10-16T14:36:28.977051+0000","last_fullsized":"2023-10-16T14:36:28.977051+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:42.923548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977277+0000","last_change":"2023-10-16T14:36:27.780492+0000","last_active":"2023-10-16T14:36:28.977277+0000","last_peered":"2023-10-16T14:36:28.977277+0000","last_clean":"2023-10-16T14:36:28.977277+0000","last_became_active":"2023-10-16T14:36:27.777749+0000","last_became_peered":"2023-10-16T14:36:27.777749+0000","last_unstale":"2023-10-16T14:36:28.977277+0000","last_undegraded":"2023-10-16T14:36:28.977277+0000","last_fullsized":"2023-10-16T14:36:28.977277+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:34.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752503+0000","last_change":"2023-10-16T14:36:27.791062+0000","last_active":"2023-10-16T14:36:28.752503+0000","last_peered":"2023-10-16T14:36:28.752503+0000","last_clean":"2023-10-16T14:36:28.752503+0000","last_became_active":"2023-10-16T14:36:27.790725+0000","last_became_peered":"2023-10-16T14:36:27.790725+0000","last_unstale":"2023-10-16T14:36:28.752503+0000","last_undegraded":"2023-10-16T14:36:28.752503+0000","last_fullsized":"2023-10-16T14:36:28.752503+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:48.161425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979201+0000","last_change":"2023-10-16T14:36:27.792830+0000","last_active":"2023-10-16T14:36:28.979201+0000","last_peered":"2023-10-16T14:36:28.979201+0000","last_clean":"2023-10-16T14:36:28.979201+0000","last_became_active":"2023-10-16T14:36:27.792094+0000","last_became_peered":"2023-10-16T14:36:27.792094+0000","last_unstale":"2023-10-16T14:36:28.979201+0000","last_undegraded":"2023-10-16T14:36:28.979201+0000","last_fullsized":"2023-10-16T14:36:28.979201+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:04.425861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979457+0000","last_change":"2023-10-16T14:36:27.789311+0000","last_active":"2023-10-16T14:36:28.979457+0000","last_peered":"2023-10-16T14:36:28.979457+0000","last_clean":"2023-10-16T14:36:28.979457+0000","last_became_active":"2023-10-16T14:36:27.788879+0000","last_became_peered":"2023-10-16T14:36:27.788879+0000","last_unstale":"2023-10-16T14:36:28.979457+0000","last_undegraded":"2023-10-16T14:36:28.979457+0000","last_fullsized":"2023-10-16T14:36:28.979457+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:41.252139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755181+0000","last_change":"2023-10-16T14:36:27.776590+0000","last_active":"2023-10-16T14:36:28.755181+0000","last_peered":"2023-10-16T14:36:28.755181+0000","last_clean":"2023-10-16T14:36:28.755181+0000","last_became_active":"2023-10-16T14:36:27.774290+0000","last_became_peered":"2023-10-16T14:36:27.774290+0000","last_unstale":"2023-10-16T14:36:28.755181+0000","last_undegraded":"2023-10-16T14:36:28.755181+0000","last_fullsized":"2023-10-16T14:36:28.755181+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:04.169834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979433+0000","last_change":"2023-10-16T14:36:27.788870+0000","last_active":"2023-10-16T14:36:28.979433+0000","last_peered":"2023-10-16T14:36:28.979433+0000","last_clean":"2023-10-16T14:36:28.979433+0000","last_became_active":"2023-10-16T14:36:27.788583+0000","last_became_peered":"2023-10-16T14:36:27.788583+0000","last_unstale":"2023-10-16T14:36:28.979433+0000","last_undegraded":"2023-10-16T14:36:28.979433+0000","last_fullsized":"2023-10-16T14:36:28.979433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:05.723950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753628+0000","last_change":"2023-10-16T14:36:27.783679+0000","last_active":"2023-10-16T14:36:28.753628+0000","last_peered":"2023-10-16T14:36:28.753628+0000","last_clean":"2023-10-16T14:36:28.753628+0000","last_became_active":"2023-10-16T14:36:27.783366+0000","last_became_peered":"2023-10-16T14:36:27.783366+0000","last_unstale":"2023-10-16T14:36:28.753628+0000","last_undegraded":"2023-10-16T14:36:28.753628+0000","last_fullsized":"2023-10-16T14:36:28.753628+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:00.218713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753889+0000","last_change":"2023-10-16T14:36:27.776685+0000","last_active":"2023-10-16T14:36:28.753889+0000","last_peered":"2023-10-16T14:36:28.753889+0000","last_clean":"2023-10-16T14:36:28.753889+0000","last_became_active":"2023-10-16T14:36:27.776399+0000","last_became_peered":"2023-10-16T14:36:27.776399+0000","last_unstale":"2023-10-16T14:36:28.753889+0000","last_undegraded":"2023-10-16T14:36:28.753889+0000","last_fullsized":"2023-10-16T14:36:28.753889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:51.527376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976858+0000","last_change":"2023-10-16T14:36:27.785864+0000","last_active":"2023-10-16T14:36:28.976858+0000","last_peered":"2023-10-16T14:36:28.976858+0000","last_clean":"2023-10-16T14:36:28.976858+0000","last_became_active":"2023-10-16T14:36:27.784994+0000","last_became_peered":"2023-10-16T14:36:27.784994+0000","last_unstale":"2023-10-16T14:36:28.976858+0000","last_undegraded":"2023-10-16T14:36:28.976858+0000","last_fullsized":"2023-10-16T14:36:28.976858+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:56.720480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977352+0000","last_change":"2023-10-16T14:36:27.778753+0000","last_active":"2023-10-16T14:36:28.977352+0000","last_peered":"2023-10-16T14:36:28.977352+0000","last_clean":"2023-10-16T14:36:28.977352+0000","last_became_active":"2023-10-16T14:36:27.778472+0000","last_became_peered":"2023-10-16T14:36:27.778472+0000","last_unstale":"2023-10-16T14:36:28.977352+0000","last_undegraded":"2023-10-16T14:36:28.977352+0000","last_fullsized":"2023-10-16T14:36:28.977352+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:39.400330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754267+0000","last_change":"2023-10-16T14:36:27.783159+0000","last_active":"2023-10-16T14:36:28.754267+0000","last_peered":"2023-10-16T14:36:28.754267+0000","last_clean":"2023-10-16T14:36:28.754267+0000","last_became_active":"2023-10-16T14:36:27.782820+0000","last_became_peered":"2023-10-16T14:36:27.782820+0000","last_unstale":"2023-10-16T14:36:28.754267+0000","last_undegraded":"2023-10-16T14:36:28.754267+0000","last_fullsized":"2023-10-16T14:36:28.754267+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:02.950949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977039+0000","last_change":"2023-10-16T14:36:27.784685+0000","last_active":"2023-10-16T14:36:28.977039+0000","last_peered":"2023-10-16T14:36:28.977039+0000","last_clean":"2023-10-16T14:36:28.977039+0000","last_became_active":"2023-10-16T14:36:27.784382+0000","last_became_peered":"2023-10-16T14:36:27.784382+0000","last_unstale":"2023-10-16T14:36:28.977039+0000","last_undegraded":"2023-10-16T14:36:28.977039+0000","last_fullsized":"2023-10-16T14:36:28.977039+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:46.651175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755341+0000","last_change":"2023-10-16T14:36:27.787939+0000","last_active":"2023-10-16T14:36:28.755341+0000","last_peered":"2023-10-16T14:36:28.755341+0000","last_clean":"2023-10-16T14:36:28.755341+0000","last_became_active":"2023-10-16T14:36:27.786332+0000","last_became_peered":"2023-10-16T14:36:27.786332+0000","last_unstale":"2023-10-16T14:36:28.755341+0000","last_undegraded":"2023-10-16T14:36:28.755341+0000","last_fullsized":"2023-10-16T14:36:28.755341+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:08.908055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754558+0000","last_change":"2023-10-16T14:36:27.782072+0000","last_active":"2023-10-16T14:36:28.754558+0000","last_peered":"2023-10-16T14:36:28.754558+0000","last_clean":"2023-10-16T14:36:28.754558+0000","last_became_active":"2023-10-16T14:36:27.781017+0000","last_became_peered":"2023-10-16T14:36:27.781017+0000","last_unstale":"2023-10-16T14:36:28.754558+0000","last_undegraded":"2023-10-16T14:36:28.754558+0000","last_fullsized":"2023-10-16T14:36:28.754558+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:47.470649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754484+0000","last_change":"2023-10-16T14:36:27.783008+0000","last_active":"2023-10-16T14:36:28.754484+0000","last_peered":"2023-10-16T14:36:28.754484+0000","last_clean":"2023-10-16T14:36:28.754484+0000","last_became_active":"2023-10-16T14:36:27.782503+0000","last_became_peered":"2023-10-16T14:36:27.782503+0000","last_unstale":"2023-10-16T14:36:28.754484+0000","last_undegraded":"2023-10-16T14:36:28.754484+0000","last_fullsized":"2023-10-16T14:36:28.754484+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:16.571347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749503+0000","last_change":"2023-10-16T14:36:27.777789+0000","last_active":"2023-10-16T14:36:28.749503+0000","last_peered":"2023-10-16T14:36:28.749503+0000","last_clean":"2023-10-16T14:36:28.749503+0000","last_became_active":"2023-10-16T14:36:27.777472+0000","last_became_peered":"2023-10-16T14:36:27.777472+0000","last_unstale":"2023-10-16T14:36:28.749503+0000","last_undegraded":"2023-10-16T14:36:28.749503+0000","last_fullsized":"2023-10-16T14:36:28.749503+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:20.996152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752180+0000","last_change":"2023-10-16T14:36:27.782563+0000","last_active":"2023-10-16T14:36:28.752180+0000","last_peered":"2023-10-16T14:36:28.752180+0000","last_clean":"2023-10-16T14:36:28.752180+0000","last_became_active":"2023-10-16T14:36:27.781508+0000","last_became_peered":"2023-10-16T14:36:27.781508+0000","last_unstale":"2023-10-16T14:36:28.752180+0000","last_undegraded":"2023-10-16T14:36:28.752180+0000","last_fullsized":"2023-10-16T14:36:28.752180+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:33.145703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749010+0000","last_change":"2023-10-16T14:36:27.772286+0000","last_active":"2023-10-16T14:36:28.749010+0000","last_peered":"2023-10-16T14:36:28.749010+0000","last_clean":"2023-10-16T14:36:28.749010+0000","last_became_active":"2023-10-16T14:36:27.771962+0000","last_became_peered":"2023-10-16T14:36:27.771962+0000","last_unstale":"2023-10-16T14:36:28.749010+0000","last_undegraded":"2023-10-16T14:36:28.749010+0000","last_fullsized":"2023-10-16T14:36:28.749010+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:43.534949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754769+0000","last_change":"2023-10-16T14:36:27.783580+0000","last_active":"2023-10-16T14:36:28.754769+0000","last_peered":"2023-10-16T14:36:28.754769+0000","last_clean":"2023-10-16T14:36:28.754769+0000","last_became_active":"2023-10-16T14:36:27.783239+0000","last_became_peered":"2023-10-16T14:36:27.783239+0000","last_unstale":"2023-10-16T14:36:28.754769+0000","last_undegraded":"2023-10-16T14:36:28.754769+0000","last_fullsized":"2023-10-16T14:36:28.754769+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:31.785329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979738+0000","last_change":"2023-10-16T14:36:27.783953+0000","last_active":"2023-10-16T14:36:28.979738+0000","last_peered":"2023-10-16T14:36:28.979738+0000","last_clean":"2023-10-16T14:36:28.979738+0000","last_became_active":"2023-10-16T14:36:27.783634+0000","last_became_peered":"2023-10-16T14:36:27.783634+0000","last_unstale":"2023-10-16T14:36:28.979738+0000","last_undegraded":"2023-10-16T14:36:28.979738+0000","last_fullsized":"2023-10-16T14:36:28.979738+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:54.673942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978222+0000","last_change":"2023-10-16T14:36:27.788967+0000","last_active":"2023-10-16T14:36:28.978222+0000","last_peered":"2023-10-16T14:36:28.978222+0000","last_clean":"2023-10-16T14:36:28.978222+0000","last_became_active":"2023-10-16T14:36:27.788675+0000","last_became_peered":"2023-10-16T14:36:27.788675+0000","last_unstale":"2023-10-16T14:36:28.978222+0000","last_undegraded":"2023-10-16T14:36:28.978222+0000","last_fullsized":"2023-10-16T14:36:28.978222+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:26.248815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.750084+0000","last_change":"2023-10-16T14:36:27.784199+0000","last_active":"2023-10-16T14:36:28.750084+0000","last_peered":"2023-10-16T14:36:28.750084+0000","last_clean":"2023-10-16T14:36:28.750084+0000","last_became_active":"2023-10-16T14:36:27.782900+0000","last_became_peered":"2023-10-16T14:36:27.782900+0000","last_unstale":"2023-10-16T14:36:28.750084+0000","last_undegraded":"2023-10-16T14:36:28.750084+0000","last_fullsized":"2023-10-16T14:36:28.750084+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:13.000863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753232+0000","last_change":"2023-10-16T14:36:27.782181+0000","last_active":"2023-10-16T14:36:28.753232+0000","last_peered":"2023-10-16T14:36:28.753232+0000","last_clean":"2023-10-16T14:36:28.753232+0000","last_became_active":"2023-10-16T14:36:27.781896+0000","last_became_peered":"2023-10-16T14:36:27.781896+0000","last_unstale":"2023-10-16T14:36:28.753232+0000","last_undegraded":"2023-10-16T14:36:28.753232+0000","last_fullsized":"2023-10-16T14:36:28.753232+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:01.682592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976548+0000","last_change":"2023-10-16T14:36:27.761897+0000","last_active":"2023-10-16T14:36:28.976548+0000","last_peered":"2023-10-16T14:36:28.976548+0000","last_clean":"2023-10-16T14:36:28.976548+0000","last_became_active":"2023-10-16T14:36:27.761120+0000","last_became_peered":"2023-10-16T14:36:27.761120+0000","last_unstale":"2023-10-16T14:36:28.976548+0000","last_undegraded":"2023-10-16T14:36:28.976548+0000","last_fullsized":"2023-10-16T14:36:28.976548+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:33.145703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978390+0000","last_change":"2023-10-16T14:36:27.783309+0000","last_active":"2023-10-16T14:36:28.978390+0000","last_peered":"2023-10-16T14:36:28.978390+0000","last_clean":"2023-10-16T14:36:28.978390+0000","last_became_active":"2023-10-16T14:36:27.783052+0000","last_became_peered":"2023-10-16T14:36:27.783052+0000","last_unstale":"2023-10-16T14:36:28.978390+0000","last_undegraded":"2023-10-16T14:36:28.978390+0000","last_fullsized":"2023-10-16T14:36:28.978390+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:52.129464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976909+0000","last_change":"2023-10-16T14:36:27.788383+0000","last_active":"2023-10-16T14:36:28.976909+0000","last_peered":"2023-10-16T14:36:28.976909+0000","last_clean":"2023-10-16T14:36:28.976909+0000","last_became_active":"2023-10-16T14:36:27.788090+0000","last_became_peered":"2023-10-16T14:36:27.788090+0000","last_unstale":"2023-10-16T14:36:28.976909+0000","last_undegraded":"2023-10-16T14:36:28.976909+0000","last_fullsized":"2023-10-16T14:36:28.976909+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:44.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977646+0000","last_change":"2023-10-16T14:36:27.787323+0000","last_active":"2023-10-16T14:36:28.977646+0000","last_peered":"2023-10-16T14:36:28.977646+0000","last_clean":"2023-10-16T14:36:28.977646+0000","last_became_active":"2023-10-16T14:36:27.787034+0000","last_became_peered":"2023-10-16T14:36:27.787034+0000","last_unstale":"2023-10-16T14:36:28.977646+0000","last_undegraded":"2023-10-16T14:36:28.977646+0000","last_fullsized":"2023-10-16T14:36:28.977646+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.112940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"141'4","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977834+0000","last_change":"2023-10-16T14:36:26.749938+0000","last_active":"2023-10-16T14:36:28.977834+0000","last_peered":"2023-10-16T14:36:28.977834+0000","last_clean":"2023-10-16T14:36:28.977834+0000","last_became_active":"2023-10-16T14:36:26.749655+0000","last_became_peered":"2023-10-16T14:36:26.749655+0000","last_unstale":"2023-10-16T14:36:28.977834+0000","last_undegraded":"2023-10-16T14:36:28.977834+0000","last_fullsized":"2023-10-16T14:36:28.977834+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:07.455720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977447+0000","last_change":"2023-10-16T14:36:27.787031+0000","last_active":"2023-10-16T14:36:28.977447+0000","last_peered":"2023-10-16T14:36:28.977447+0000","last_clean":"2023-10-16T14:36:28.977447+0000","last_became_active":"2023-10-16T14:36:27.786460+0000","last_became_peered":"2023-10-16T14:36:27.786460+0000","last_unstale":"2023-10-16T14:36:28.977447+0000","last_undegraded":"2023-10-16T14:36:28.977447+0000","last_fullsized":"2023-10-16T14:36:28.977447+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:56.720480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755326+0000","last_change":"2023-10-16T14:36:26.749014+0000","last_active":"2023-10-16T14:36:28.755326+0000","last_peered":"2023-10-16T14:36:28.755326+0000","last_clean":"2023-10-16T14:36:28.755326+0000","last_became_active":"2023-10-16T14:36:26.747877+0000","last_became_peered":"2023-10-16T14:36:26.747877+0000","last_unstale":"2023-10-16T14:36:28.755326+0000","last_undegraded":"2023-10-16T14:36:28.755326+0000","last_fullsized":"2023-10-16T14:36:28.755326+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:52.667891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978594+0000","last_change":"2023-10-16T14:36:27.780371+0000","last_active":"2023-10-16T14:36:28.978594+0000","last_peered":"2023-10-16T14:36:28.978594+0000","last_clean":"2023-10-16T14:36:28.978594+0000","last_became_active":"2023-10-16T14:36:27.780093+0000","last_became_peered":"2023-10-16T14:36:27.780093+0000","last_unstale":"2023-10-16T14:36:28.978594+0000","last_undegraded":"2023-10-16T14:36:28.978594+0000","last_fullsized":"2023-10-16T14:36:28.978594+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:21.414718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975687+0000","last_change":"2023-10-16T14:36:26.749842+0000","last_active":"2023-10-16T14:36:28.975687+0000","last_peered":"2023-10-16T14:36:28.975687+0000","last_clean":"2023-10-16T14:36:28.975687+0000","last_became_active":"2023-10-16T14:36:26.749568+0000","last_became_peered":"2023-10-16T14:36:26.749568+0000","last_unstale":"2023-10-16T14:36:28.975687+0000","last_undegraded":"2023-10-16T14:36:28.975687+0000","last_fullsized":"2023-10-16T14:36:28.975687+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:56.216001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755343+0000","last_change":"2023-10-16T14:36:27.785478+0000","last_active":"2023-10-16T14:36:28.755343+0000","last_peered":"2023-10-16T14:36:28.755343+0000","last_clean":"2023-10-16T14:36:28.755343+0000","last_became_active":"2023-10-16T14:36:27.785150+0000","last_became_peered":"2023-10-16T14:36:27.785150+0000","last_unstale":"2023-10-16T14:36:28.755343+0000","last_undegraded":"2023-10-16T14:36:28.755343+0000","last_fullsized":"2023-10-16T14:36:28.755343+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:52.512886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975423+0000","last_change":"2023-10-16T14:36:26.747805+0000","last_active":"2023-10-16T14:36:28.975423+0000","last_peered":"2023-10-16T14:36:28.975423+0000","last_clean":"2023-10-16T14:36:28.975423+0000","last_became_active":"2023-10-16T14:36:26.745642+0000","last_became_peered":"2023-10-16T14:36:26.745642+0000","last_unstale":"2023-10-16T14:36:28.975423+0000","last_undegraded":"2023-10-16T14:36:28.975423+0000","last_fullsized":"2023-10-16T14:36:28.975423+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:41.919206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978157+0000","last_change":"2023-10-16T14:36:27.792877+0000","last_active":"2023-10-16T14:36:28.978157+0000","last_peered":"2023-10-16T14:36:28.978157+0000","last_clean":"2023-10-16T14:36:28.978157+0000","last_became_active":"2023-10-16T14:36:27.791711+0000","last_became_peered":"2023-10-16T14:36:27.791711+0000","last_unstale":"2023-10-16T14:36:28.978157+0000","last_undegraded":"2023-10-16T14:36:28.978157+0000","last_fullsized":"2023-10-16T14:36:28.978157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:48.116343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976836+0000","last_change":"2023-10-16T14:36:26.749195+0000","last_active":"2023-10-16T14:36:28.976836+0000","last_peered":"2023-10-16T14:36:28.976836+0000","last_clean":"2023-10-16T14:36:28.976836+0000","last_became_active":"2023-10-16T14:36:26.748894+0000","last_became_peered":"2023-10-16T14:36:26.748894+0000","last_unstale":"2023-10-16T14:36:28.976836+0000","last_undegraded":"2023-10-16T14:36:28.976836+0000","last_fullsized":"2023-10-16T14:36:28.976836+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:12.749653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753633+0000","last_change":"2023-10-16T14:36:27.787874+0000","last_active":"2023-10-16T14:36:28.753633+0000","last_peered":"2023-10-16T14:36:28.753633+0000","last_clean":"2023-10-16T14:36:28.753633+0000","last_became_active":"2023-10-16T14:36:27.787587+0000","last_became_peered":"2023-10-16T14:36:27.787587+0000","last_unstale":"2023-10-16T14:36:28.753633+0000","last_undegraded":"2023-10-16T14:36:28.753633+0000","last_fullsized":"2023-10-16T14:36:28.753633+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:34.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749691+0000","last_change":"2023-10-16T14:36:26.754082+0000","last_active":"2023-10-16T14:36:28.749691+0000","last_peered":"2023-10-16T14:36:28.749691+0000","last_clean":"2023-10-16T14:36:28.749691+0000","last_became_active":"2023-10-16T14:36:26.753160+0000","last_became_peered":"2023-10-16T14:36:26.753160+0000","last_unstale":"2023-10-16T14:36:28.749691+0000","last_undegraded":"2023-10-16T14:36:28.749691+0000","last_fullsized":"2023-10-16T14:36:28.749691+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:13.163619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977146+0000","last_change":"2023-10-16T14:36:27.796683+0000","last_active":"2023-10-16T14:36:28.977146+0000","last_peered":"2023-10-16T14:36:28.977146+0000","last_clean":"2023-10-16T14:36:28.977146+0000","last_became_active":"2023-10-16T14:36:27.796386+0000","last_became_peered":"2023-10-16T14:36:27.796386+0000","last_unstale":"2023-10-16T14:36:28.977146+0000","last_undegraded":"2023-10-16T14:36:28.977146+0000","last_fullsized":"2023-10-16T14:36:28.977146+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:27.862098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752462+0000","last_change":"2023-10-16T14:36:26.753260+0000","last_active":"2023-10-16T14:36:28.752462+0000","last_peered":"2023-10-16T14:36:28.752462+0000","last_clean":"2023-10-16T14:36:28.752462+0000","last_became_active":"2023-10-16T14:36:26.752872+0000","last_became_peered":"2023-10-16T14:36:26.752872+0000","last_unstale":"2023-10-16T14:36:28.752462+0000","last_undegraded":"2023-10-16T14:36:28.752462+0000","last_fullsized":"2023-10-16T14:36:28.752462+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:17.308544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976214+0000","last_change":"2023-10-16T14:36:27.780261+0000","last_active":"2023-10-16T14:36:28.976214+0000","last_peered":"2023-10-16T14:36:28.976214+0000","last_clean":"2023-10-16T14:36:28.976214+0000","last_became_active":"2023-10-16T14:36:27.778107+0000","last_became_peered":"2023-10-16T14:36:27.778107+0000","last_unstale":"2023-10-16T14:36:28.976214+0000","last_undegraded":"2023-10-16T14:36:28.976214+0000","last_fullsized":"2023-10-16T14:36:28.976214+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:24.999972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977448+0000","last_change":"2023-10-16T14:36:26.755025+0000","last_active":"2023-10-16T14:36:28.977448+0000","last_peered":"2023-10-16T14:36:28.977448+0000","last_clean":"2023-10-16T14:36:28.977448+0000","last_became_active":"2023-10-16T14:36:26.754753+0000","last_became_peered":"2023-10-16T14:36:26.754753+0000","last_unstale":"2023-10-16T14:36:28.977448+0000","last_undegraded":"2023-10-16T14:36:28.977448+0000","last_fullsized":"2023-10-16T14:36:28.977448+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:47.781398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977065+0000","last_change":"2023-10-16T14:36:27.778669+0000","last_active":"2023-10-16T14:36:28.977065+0000","last_peered":"2023-10-16T14:36:28.977065+0000","last_clean":"2023-10-16T14:36:28.977065+0000","last_became_active":"2023-10-16T14:36:27.778370+0000","last_became_peered":"2023-10-16T14:36:27.778370+0000","last_unstale":"2023-10-16T14:36:28.977065+0000","last_undegraded":"2023-10-16T14:36:28.977065+0000","last_fullsized":"2023-10-16T14:36:28.977065+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:01.682592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.975494+0000","last_change":"2023-10-16T14:36:26.750634+0000","last_active":"2023-10-16T14:36:28.975494+0000","last_peered":"2023-10-16T14:36:28.975494+0000","last_clean":"2023-10-16T14:36:28.975494+0000","last_became_active":"2023-10-16T14:36:26.750347+0000","last_became_peered":"2023-10-16T14:36:26.750347+0000","last_unstale":"2023-10-16T14:36:28.975494+0000","last_undegraded":"2023-10-16T14:36:28.975494+0000","last_fullsized":"2023-10-16T14:36:28.975494+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:58.108598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977842+0000","last_change":"2023-10-16T14:36:27.792903+0000","last_active":"2023-10-16T14:36:28.977842+0000","last_peered":"2023-10-16T14:36:28.977842+0000","last_clean":"2023-10-16T14:36:28.977842+0000","last_became_active":"2023-10-16T14:36:27.792568+0000","last_became_peered":"2023-10-16T14:36:27.792568+0000","last_unstale":"2023-10-16T14:36:28.977842+0000","last_undegraded":"2023-10-16T14:36:28.977842+0000","last_fullsized":"2023-10-16T14:36:28.977842+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:42.923548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749636+0000","last_change":"2023-10-16T14:36:26.752797+0000","last_active":"2023-10-16T14:36:28.749636+0000","last_peered":"2023-10-16T14:36:28.749636+0000","last_clean":"2023-10-16T14:36:28.749636+0000","last_became_active":"2023-10-16T14:36:26.752474+0000","last_became_peered":"2023-10-16T14:36:26.752474+0000","last_unstale":"2023-10-16T14:36:28.749636+0000","last_undegraded":"2023-10-16T14:36:28.749636+0000","last_fullsized":"2023-10-16T14:36:28.749636+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:16.781966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.750411+0000","last_change":"2023-10-16T14:36:27.775045+0000","last_active":"2023-10-16T14:36:28.750411+0000","last_peered":"2023-10-16T14:36:28.750411+0000","last_clean":"2023-10-16T14:36:28.750411+0000","last_became_active":"2023-10-16T14:36:27.774765+0000","last_became_peered":"2023-10-16T14:36:27.774765+0000","last_unstale":"2023-10-16T14:36:28.750411+0000","last_undegraded":"2023-10-16T14:36:28.750411+0000","last_fullsized":"2023-10-16T14:36:28.750411+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:05.560805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"141'3","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979012+0000","last_change":"2023-10-16T14:36:26.752645+0000","last_active":"2023-10-16T14:36:28.979012+0000","last_peered":"2023-10-16T14:36:28.979012+0000","last_clean":"2023-10-16T14:36:28.979012+0000","last_became_active":"2023-10-16T14:36:26.752282+0000","last_became_peered":"2023-10-16T14:36:26.752282+0000","last_unstale":"2023-10-16T14:36:28.979012+0000","last_undegraded":"2023-10-16T14:36:28.979012+0000","last_fullsized":"2023-10-16T14:36:28.979012+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:13.984175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976924+0000","last_change":"2023-10-16T14:36:27.767176+0000","last_active":"2023-10-16T14:36:28.976924+0000","last_peered":"2023-10-16T14:36:28.976924+0000","last_clean":"2023-10-16T14:36:28.976924+0000","last_became_active":"2023-10-16T14:36:27.766565+0000","last_became_peered":"2023-10-16T14:36:27.766565+0000","last_unstale":"2023-10-16T14:36:28.976924+0000","last_undegraded":"2023-10-16T14:36:28.976924+0000","last_fullsized":"2023-10-16T14:36:28.976924+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:38.458834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"141'6","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979908+0000","last_change":"2023-10-16T14:36:26.758644+0000","last_active":"2023-10-16T14:36:28.979908+0000","last_peered":"2023-10-16T14:36:28.979908+0000","last_clean":"2023-10-16T14:36:28.979908+0000","last_became_active":"2023-10-16T14:36:26.757865+0000","last_became_peered":"2023-10-16T14:36:26.757865+0000","last_unstale":"2023-10-16T14:36:28.979908+0000","last_undegraded":"2023-10-16T14:36:28.979908+0000","last_fullsized":"2023-10-16T14:36:28.979908+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:39.940546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749555+0000","last_change":"2023-10-16T14:36:27.763855+0000","last_active":"2023-10-16T14:36:28.749555+0000","last_peered":"2023-10-16T14:36:28.749555+0000","last_clean":"2023-10-16T14:36:28.749555+0000","last_became_active":"2023-10-16T14:36:27.761283+0000","last_became_peered":"2023-10-16T14:36:27.761283+0000","last_unstale":"2023-10-16T14:36:28.749555+0000","last_undegraded":"2023-10-16T14:36:28.749555+0000","last_fullsized":"2023-10-16T14:36:28.749555+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:33.409206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977751+0000","last_change":"2023-10-16T14:36:26.734712+0000","last_active":"2023-10-16T14:36:28.977751+0000","last_peered":"2023-10-16T14:36:28.977751+0000","last_clean":"2023-10-16T14:36:28.977751+0000","last_became_active":"2023-10-16T14:36:26.734325+0000","last_became_peered":"2023-10-16T14:36:26.734325+0000","last_unstale":"2023-10-16T14:36:28.977751+0000","last_undegraded":"2023-10-16T14:36:28.977751+0000","last_fullsized":"2023-10-16T14:36:28.977751+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:35.885229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749858+0000","last_change":"2023-10-16T14:36:27.767104+0000","last_active":"2023-10-16T14:36:28.749858+0000","last_peered":"2023-10-16T14:36:28.749858+0000","last_clean":"2023-10-16T14:36:28.749858+0000","last_became_active":"2023-10-16T14:36:27.765854+0000","last_became_peered":"2023-10-16T14:36:27.765854+0000","last_unstale":"2023-10-16T14:36:28.749858+0000","last_undegraded":"2023-10-16T14:36:28.749858+0000","last_fullsized":"2023-10-16T14:36:28.749858+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:48.161425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977352+0000","last_change":"2023-10-16T14:36:26.753817+0000","last_active":"2023-10-16T14:36:28.977352+0000","last_peered":"2023-10-16T14:36:28.977352+0000","last_clean":"2023-10-16T14:36:28.977352+0000","last_became_active":"2023-10-16T14:36:26.753473+0000","last_became_peered":"2023-10-16T14:36:26.753473+0000","last_unstale":"2023-10-16T14:36:28.977352+0000","last_undegraded":"2023-10-16T14:36:28.977352+0000","last_fullsized":"2023-10-16T14:36:28.977352+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:21.659431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749159+0000","last_change":"2023-10-16T14:36:27.774638+0000","last_active":"2023-10-16T14:36:28.749159+0000","last_peered":"2023-10-16T14:36:28.749159+0000","last_clean":"2023-10-16T14:36:28.749159+0000","last_became_active":"2023-10-16T14:36:27.774161+0000","last_became_peered":"2023-10-16T14:36:27.774161+0000","last_unstale":"2023-10-16T14:36:28.749159+0000","last_undegraded":"2023-10-16T14:36:28.749159+0000","last_fullsized":"2023-10-16T14:36:28.749159+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:34.929978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"141'8","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.974901+0000","last_change":"2023-10-16T14:36:26.754976+0000","last_active":"2023-10-16T14:36:28.974901+0000","last_peered":"2023-10-16T14:36:28.974901+0000","last_clean":"2023-10-16T14:36:28.974901+0000","last_became_active":"2023-10-16T14:36:26.754677+0000","last_became_peered":"2023-10-16T14:36:26.754677+0000","last_unstale":"2023-10-16T14:36:28.974901+0000","last_undegraded":"2023-10-16T14:36:28.974901+0000","last_fullsized":"2023-10-16T14:36:28.974901+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:45.646833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976728+0000","last_change":"2023-10-16T14:36:27.794441+0000","last_active":"2023-10-16T14:36:28.976728+0000","last_peered":"2023-10-16T14:36:28.976728+0000","last_clean":"2023-10-16T14:36:28.976728+0000","last_became_active":"2023-10-16T14:36:27.794160+0000","last_became_peered":"2023-10-16T14:36:27.794160+0000","last_unstale":"2023-10-16T14:36:28.976728+0000","last_undegraded":"2023-10-16T14:36:28.976728+0000","last_fullsized":"2023-10-16T14:36:28.976728+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:19.858100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979397+0000","last_change":"2023-10-16T14:36:26.753759+0000","last_active":"2023-10-16T14:36:28.979397+0000","last_peered":"2023-10-16T14:36:28.979397+0000","last_clean":"2023-10-16T14:36:28.979397+0000","last_became_active":"2023-10-16T14:36:26.753484+0000","last_became_peered":"2023-10-16T14:36:26.753484+0000","last_unstale":"2023-10-16T14:36:28.979397+0000","last_undegraded":"2023-10-16T14:36:28.979397+0000","last_fullsized":"2023-10-16T14:36:28.979397+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:02.386929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977483+0000","last_change":"2023-10-16T14:36:27.798590+0000","last_active":"2023-10-16T14:36:28.977483+0000","last_peered":"2023-10-16T14:36:28.977483+0000","last_clean":"2023-10-16T14:36:28.977483+0000","last_became_active":"2023-10-16T14:36:27.797341+0000","last_became_peered":"2023-10-16T14:36:27.797341+0000","last_unstale":"2023-10-16T14:36:28.977483+0000","last_undegraded":"2023-10-16T14:36:28.977483+0000","last_fullsized":"2023-10-16T14:36:28.977483+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:57.220343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976420+0000","last_change":"2023-10-16T14:36:26.743860+0000","last_active":"2023-10-16T14:36:28.976420+0000","last_peered":"2023-10-16T14:36:28.976420+0000","last_clean":"2023-10-16T14:36:28.976420+0000","last_became_active":"2023-10-16T14:36:26.743520+0000","last_became_peered":"2023-10-16T14:36:26.743520+0000","last_unstale":"2023-10-16T14:36:28.976420+0000","last_undegraded":"2023-10-16T14:36:28.976420+0000","last_fullsized":"2023-10-16T14:36:28.976420+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:25.244473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754949+0000","last_change":"2023-10-16T14:36:27.783330+0000","last_active":"2023-10-16T14:36:28.754949+0000","last_peered":"2023-10-16T14:36:28.754949+0000","last_clean":"2023-10-16T14:36:28.754949+0000","last_became_active":"2023-10-16T14:36:27.782716+0000","last_became_peered":"2023-10-16T14:36:27.782716+0000","last_unstale":"2023-10-16T14:36:28.754949+0000","last_undegraded":"2023-10-16T14:36:28.754949+0000","last_fullsized":"2023-10-16T14:36:28.754949+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:56.264242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978037+0000","last_change":"2023-10-16T14:36:26.756244+0000","last_active":"2023-10-16T14:36:28.978037+0000","last_peered":"2023-10-16T14:36:28.978037+0000","last_clean":"2023-10-16T14:36:28.978037+0000","last_became_active":"2023-10-16T14:36:26.755990+0000","last_became_peered":"2023-10-16T14:36:26.755990+0000","last_unstale":"2023-10-16T14:36:28.978037+0000","last_undegraded":"2023-10-16T14:36:28.978037+0000","last_fullsized":"2023-10-16T14:36:28.978037+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:14.654281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.755640+0000","last_change":"2023-10-16T14:36:27.770592+0000","last_active":"2023-10-16T14:36:28.755640+0000","last_peered":"2023-10-16T14:36:28.755640+0000","last_clean":"2023-10-16T14:36:28.755640+0000","last_became_active":"2023-10-16T14:36:27.770236+0000","last_became_peered":"2023-10-16T14:36:27.770236+0000","last_unstale":"2023-10-16T14:36:28.755640+0000","last_undegraded":"2023-10-16T14:36:28.755640+0000","last_fullsized":"2023-10-16T14:36:28.755640+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:50:28.693112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977703+0000","last_change":"2023-10-16T14:36:26.754995+0000","last_active":"2023-10-16T14:36:28.977703+0000","last_peered":"2023-10-16T14:36:28.977703+0000","last_clean":"2023-10-16T14:36:28.977703+0000","last_became_active":"2023-10-16T14:36:26.754750+0000","last_became_peered":"2023-10-16T14:36:26.754750+0000","last_unstale":"2023-10-16T14:36:28.977703+0000","last_undegraded":"2023-10-16T14:36:28.977703+0000","last_fullsized":"2023-10-16T14:36:28.977703+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:03.072287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.754250+0000","last_change":"2023-10-16T14:36:27.775061+0000","last_active":"2023-10-16T14:36:28.754250+0000","last_peered":"2023-10-16T14:36:28.754250+0000","last_clean":"2023-10-16T14:36:28.754250+0000","last_became_active":"2023-10-16T14:36:27.774684+0000","last_became_peered":"2023-10-16T14:36:27.774684+0000","last_unstale":"2023-10-16T14:36:28.754250+0000","last_undegraded":"2023-10-16T14:36:28.754250+0000","last_fullsized":"2023-10-16T14:36:28.754250+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:31.327320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977845+0000","last_change":"2023-10-16T14:36:26.743410+0000","last_active":"2023-10-16T14:36:28.977845+0000","last_peered":"2023-10-16T14:36:28.977845+0000","last_clean":"2023-10-16T14:36:28.977845+0000","last_became_active":"2023-10-16T14:36:26.742404+0000","last_became_peered":"2023-10-16T14:36:26.742404+0000","last_unstale":"2023-10-16T14:36:28.977845+0000","last_undegraded":"2023-10-16T14:36:28.977845+0000","last_fullsized":"2023-10-16T14:36:28.977845+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:52.649310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978491+0000","last_change":"2023-10-16T14:36:27.775955+0000","last_active":"2023-10-16T14:36:28.978491+0000","last_peered":"2023-10-16T14:36:28.978491+0000","last_clean":"2023-10-16T14:36:28.978491+0000","last_became_active":"2023-10-16T14:36:27.775231+0000","last_became_peered":"2023-10-16T14:36:27.775231+0000","last_unstale":"2023-10-16T14:36:28.978491+0000","last_undegraded":"2023-10-16T14:36:28.978491+0000","last_fullsized":"2023-10-16T14:36:28.978491+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:51.088976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.974931+0000","last_change":"2023-10-16T14:36:26.749567+0000","last_active":"2023-10-16T14:36:28.974931+0000","last_peered":"2023-10-16T14:36:28.974931+0000","last_clean":"2023-10-16T14:36:28.974931+0000","last_became_active":"2023-10-16T14:36:26.749280+0000","last_became_peered":"2023-10-16T14:36:26.749280+0000","last_unstale":"2023-10-16T14:36:28.974931+0000","last_undegraded":"2023-10-16T14:36:28.974931+0000","last_fullsized":"2023-10-16T14:36:28.974931+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:55.716138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752612+0000","last_change":"2023-10-16T14:36:27.780120+0000","last_active":"2023-10-16T14:36:28.752612+0000","last_peered":"2023-10-16T14:36:28.752612+0000","last_clean":"2023-10-16T14:36:28.752612+0000","last_became_active":"2023-10-16T14:36:27.777388+0000","last_became_peered":"2023-10-16T14:36:27.777388+0000","last_unstale":"2023-10-16T14:36:28.752612+0000","last_undegraded":"2023-10-16T14:36:28.752612+0000","last_fullsized":"2023-10-16T14:36:28.752612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:13.000863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976993+0000","last_change":"2023-10-16T14:36:26.741753+0000","last_active":"2023-10-16T14:36:28.976993+0000","last_peered":"2023-10-16T14:36:28.976993+0000","last_clean":"2023-10-16T14:36:28.976993+0000","last_became_active":"2023-10-16T14:36:26.741468+0000","last_became_peered":"2023-10-16T14:36:26.741468+0000","last_unstale":"2023-10-16T14:36:28.976993+0000","last_undegraded":"2023-10-16T14:36:28.976993+0000","last_fullsized":"2023-10-16T14:36:28.976993+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:57.682605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749230+0000","last_change":"2023-10-16T14:36:27.774454+0000","last_active":"2023-10-16T14:36:28.749230+0000","last_peered":"2023-10-16T14:36:28.749230+0000","last_clean":"2023-10-16T14:36:28.749230+0000","last_became_active":"2023-10-16T14:36:27.773746+0000","last_became_peered":"2023-10-16T14:36:27.773746+0000","last_unstale":"2023-10-16T14:36:28.749230+0000","last_undegraded":"2023-10-16T14:36:28.749230+0000","last_fullsized":"2023-10-16T14:36:28.749230+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:04.069407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977617+0000","last_change":"2023-10-16T14:36:26.750386+0000","last_active":"2023-10-16T14:36:28.977617+0000","last_peered":"2023-10-16T14:36:28.977617+0000","last_clean":"2023-10-16T14:36:28.977617+0000","last_became_active":"2023-10-16T14:36:26.750067+0000","last_became_peered":"2023-10-16T14:36:26.750067+0000","last_unstale":"2023-10-16T14:36:28.977617+0000","last_undegraded":"2023-10-16T14:36:28.977617+0000","last_fullsized":"2023-10-16T14:36:28.977617+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:02.110045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977378+0000","last_change":"2023-10-16T14:36:27.795812+0000","last_active":"2023-10-16T14:36:28.977378+0000","last_peered":"2023-10-16T14:36:28.977378+0000","last_clean":"2023-10-16T14:36:28.977378+0000","last_became_active":"2023-10-16T14:36:27.795286+0000","last_became_peered":"2023-10-16T14:36:27.795286+0000","last_unstale":"2023-10-16T14:36:28.977378+0000","last_undegraded":"2023-10-16T14:36:28.977378+0000","last_fullsized":"2023-10-16T14:36:28.977378+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:42.731559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749901+0000","last_change":"2023-10-16T14:36:26.754041+0000","last_active":"2023-10-16T14:36:28.749901+0000","last_peered":"2023-10-16T14:36:28.749901+0000","last_clean":"2023-10-16T14:36:28.749901+0000","last_became_active":"2023-10-16T14:36:26.753750+0000","last_became_peered":"2023-10-16T14:36:26.753750+0000","last_unstale":"2023-10-16T14:36:28.749901+0000","last_undegraded":"2023-10-16T14:36:28.749901+0000","last_fullsized":"2023-10-16T14:36:28.749901+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:21.746068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979489+0000","last_change":"2023-10-16T14:36:27.779841+0000","last_active":"2023-10-16T14:36:28.979489+0000","last_peered":"2023-10-16T14:36:28.979489+0000","last_clean":"2023-10-16T14:36:28.979489+0000","last_became_active":"2023-10-16T14:36:27.779445+0000","last_became_peered":"2023-10-16T14:36:27.779445+0000","last_unstale":"2023-10-16T14:36:28.979489+0000","last_undegraded":"2023-10-16T14:36:28.979489+0000","last_fullsized":"2023-10-16T14:36:28.979489+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:00.381003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977283+0000","last_change":"2023-10-16T14:36:26.750369+0000","last_active":"2023-10-16T14:36:28.977283+0000","last_peered":"2023-10-16T14:36:28.977283+0000","last_clean":"2023-10-16T14:36:28.977283+0000","last_became_active":"2023-10-16T14:36:26.750007+0000","last_became_peered":"2023-10-16T14:36:26.750007+0000","last_unstale":"2023-10-16T14:36:28.977283+0000","last_undegraded":"2023-10-16T14:36:28.977283+0000","last_fullsized":"2023-10-16T14:36:28.977283+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:57.682605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979808+0000","last_change":"2023-10-16T14:36:27.797078+0000","last_active":"2023-10-16T14:36:28.979808+0000","last_peered":"2023-10-16T14:36:28.979808+0000","last_clean":"2023-10-16T14:36:28.979808+0000","last_became_active":"2023-10-16T14:36:27.796736+0000","last_became_peered":"2023-10-16T14:36:27.796736+0000","last_unstale":"2023-10-16T14:36:28.979808+0000","last_undegraded":"2023-10-16T14:36:28.979808+0000","last_fullsized":"2023-10-16T14:36:28.979808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:13.230454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.979050+0000","last_change":"2023-10-16T14:36:26.749885+0000","last_active":"2023-10-16T14:36:28.979050+0000","last_peered":"2023-10-16T14:36:28.979050+0000","last_clean":"2023-10-16T14:36:28.979050+0000","last_became_active":"2023-10-16T14:36:26.749580+0000","last_became_peered":"2023-10-16T14:36:26.749580+0000","last_unstale":"2023-10-16T14:36:28.979050+0000","last_undegraded":"2023-10-16T14:36:28.979050+0000","last_fullsized":"2023-10-16T14:36:28.979050+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:45.850238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977787+0000","last_change":"2023-10-16T14:36:27.787573+0000","last_active":"2023-10-16T14:36:28.977787+0000","last_peered":"2023-10-16T14:36:28.977787+0000","last_clean":"2023-10-16T14:36:28.977787+0000","last_became_active":"2023-10-16T14:36:27.787140+0000","last_became_peered":"2023-10-16T14:36:27.787140+0000","last_unstale":"2023-10-16T14:36:28.977787+0000","last_undegraded":"2023-10-16T14:36:28.977787+0000","last_fullsized":"2023-10-16T14:36:28.977787+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:44.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.750334+0000","last_change":"2023-10-16T14:36:26.750820+0000","last_active":"2023-10-16T14:36:28.750334+0000","last_peered":"2023-10-16T14:36:28.750334+0000","last_clean":"2023-10-16T14:36:28.750334+0000","last_became_active":"2023-10-16T14:36:26.750340+0000","last_became_peered":"2023-10-16T14:36:26.750340+0000","last_unstale":"2023-10-16T14:36:28.750334+0000","last_undegraded":"2023-10-16T14:36:28.750334+0000","last_fullsized":"2023-10-16T14:36:28.750334+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:52.667891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752453+0000","last_change":"2023-10-16T14:36:27.775695+0000","last_active":"2023-10-16T14:36:28.752453+0000","last_peered":"2023-10-16T14:36:28.752453+0000","last_clean":"2023-10-16T14:36:28.752453+0000","last_became_active":"2023-10-16T14:36:27.775420+0000","last_became_peered":"2023-10-16T14:36:27.775420+0000","last_unstale":"2023-10-16T14:36:28.752453+0000","last_undegraded":"2023-10-16T14:36:28.752453+0000","last_fullsized":"2023-10-16T14:36:28.752453+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:38.458834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977056+0000","last_change":"2023-10-16T14:36:26.747263+0000","last_active":"2023-10-16T14:36:28.977056+0000","last_peered":"2023-10-16T14:36:28.977056+0000","last_clean":"2023-10-16T14:36:28.977056+0000","last_became_active":"2023-10-16T14:36:26.746934+0000","last_became_peered":"2023-10-16T14:36:26.746934+0000","last_unstale":"2023-10-16T14:36:28.977056+0000","last_undegraded":"2023-10-16T14:36:28.977056+0000","last_fullsized":"2023-10-16T14:36:28.977056+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:25.447610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.753982+0000","last_change":"2023-10-16T14:36:27.782764+0000","last_active":"2023-10-16T14:36:28.753982+0000","last_peered":"2023-10-16T14:36:28.753982+0000","last_clean":"2023-10-16T14:36:28.753982+0000","last_became_active":"2023-10-16T14:36:27.782079+0000","last_became_peered":"2023-10-16T14:36:27.782079+0000","last_unstale":"2023-10-16T14:36:28.753982+0000","last_undegraded":"2023-10-16T14:36:28.753982+0000","last_fullsized":"2023-10-16T14:36:28.753982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:51.150514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752672+0000","last_change":"2023-10-16T14:36:26.736614+0000","last_active":"2023-10-16T14:36:28.752672+0000","last_peered":"2023-10-16T14:36:28.752672+0000","last_clean":"2023-10-16T14:36:28.752672+0000","last_became_active":"2023-10-16T14:36:26.731128+0000","last_became_peered":"2023-10-16T14:36:26.731128+0000","last_unstale":"2023-10-16T14:36:28.752672+0000","last_undegraded":"2023-10-16T14:36:28.752672+0000","last_fullsized":"2023-10-16T14:36:28.752672+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:23.995630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.752834+0000","last_change":"2023-10-16T14:36:27.777818+0000","last_active":"2023-10-16T14:36:28.752834+0000","last_peered":"2023-10-16T14:36:28.752834+0000","last_clean":"2023-10-16T14:36:28.752834+0000","last_became_active":"2023-10-16T14:36:27.776011+0000","last_became_peered":"2023-10-16T14:36:27.776011+0000","last_unstale":"2023-10-16T14:36:28.752834+0000","last_undegraded":"2023-10-16T14:36:28.752834+0000","last_fullsized":"2023-10-16T14:36:28.752834+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:05.560805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.974953+0000","last_change":"2023-10-16T14:36:26.750705+0000","last_active":"2023-10-16T14:36:28.974953+0000","last_peered":"2023-10-16T14:36:28.974953+0000","last_clean":"2023-10-16T14:36:28.974953+0000","last_became_active":"2023-10-16T14:36:26.749681+0000","last_became_peered":"2023-10-16T14:36:26.749681+0000","last_unstale":"2023-10-16T14:36:28.974953+0000","last_undegraded":"2023-10-16T14:36:28.974953+0000","last_fullsized":"2023-10-16T14:36:28.974953+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:43.903633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.978331+0000","last_change":"2023-10-16T14:36:27.796281+0000","last_active":"2023-10-16T14:36:28.978331+0000","last_peered":"2023-10-16T14:36:28.978331+0000","last_clean":"2023-10-16T14:36:28.978331+0000","last_became_active":"2023-10-16T14:36:27.795993+0000","last_became_peered":"2023-10-16T14:36:27.795993+0000","last_unstale":"2023-10-16T14:36:28.978331+0000","last_undegraded":"2023-10-16T14:36:28.978331+0000","last_fullsized":"2023-10-16T14:36:28.978331+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:55.360430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.977608+0000","last_change":"2023-10-16T14:36:26.750338+0000","last_active":"2023-10-16T14:36:28.977608+0000","last_peered":"2023-10-16T14:36:28.977608+0000","last_clean":"2023-10-16T14:36:28.977608+0000","last_became_active":"2023-10-16T14:36:26.749681+0000","last_became_peered":"2023-10-16T14:36:26.749681+0000","last_unstale":"2023-10-16T14:36:28.977608+0000","last_undegraded":"2023-10-16T14:36:28.977608+0000","last_fullsized":"2023-10-16T14:36:28.977608+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:02.175201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.749010+0000","last_change":"2023-10-16T14:36:27.770494+0000","last_active":"2023-10-16T14:36:28.749010+0000","last_peered":"2023-10-16T14:36:28.749010+0000","last_clean":"2023-10-16T14:36:28.749010+0000","last_became_active":"2023-10-16T14:36:27.770175+0000","last_became_peered":"2023-10-16T14:36:27.770175+0000","last_unstale":"2023-10-16T14:36:28.749010+0000","last_undegraded":"2023-10-16T14:36:28.749010+0000","last_fullsized":"2023-10-16T14:36:28.749010+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:26.708631+0000","last_clean_scrub_stamp":"2023-10-16T14:36:26.708631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:58.992728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T14:36:28.976525+0000","last_change":"2023-10-16T14:36:26.756906+0000","last_active":"2023-10-16T14:36:28.976525+0000","last_peered":"2023-10-16T14:36:28.976525+0000","last_clean":"2023-10-16T14:36:28.976525+0000","last_became_active":"2023-10-16T14:36:26.756560+0000","last_became_peered":"2023-10-16T14:36:26.756560+0000","last_unstale":"2023-10-16T14:36:28.976525+0000","last_undegraded":"2023-10-16T14:36:28.976525+0000","last_fullsized":"2023-10-16T14:36:28.976525+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:36:25.704289+0000","last_clean_scrub_stamp":"2023-10-16T14:36:25.704289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:17.308544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"130'230","reported_seq":696,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T14:36:27.759180+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:36:27.759180+0000","last_peered":"2023-10-16T14:36:27.759180+0000","last_clean":"2023-10-16T14:36:27.759180+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:36:27.759180+0000","last_undegraded":"2023-10-16T14:36:27.759180+0000","last_fullsized":"2023-10-16T14:36:27.759180+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575087,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41808,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330032,"statfs":{"total":96636764160,"available":96593952768,"internally_reserved":0,"allocated":2363392,"data_stored":1444226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.079,"15min":1.131},"min":{"1min":0.65300000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5289999999999999,"5min":1.833,"15min":1.833},"last":1.181},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.0649999999999999,"15min":1.133},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.712,"5min":1.891,"15min":1.891},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1359999999999999,"15min":1.175},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6060000000000001,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":2.8479999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.22,"15min":1.2270000000000001},"min":{"1min":0.625,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6479999999999999,"5min":2.048,"15min":2.048},"last":1.5069999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.26,"15min":1.3049999999999999},"min":{"1min":0.69199999999999995,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7509999999999999,"5min":1.8839999999999999,"15min":1.9339999999999999},"last":2.2450000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.214,"15min":1.224},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.786,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":2.1269999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.2390000000000001,"15min":1.2929999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.8620000000000001,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":2.677},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.3240000000000001,"15min":1.3560000000000001},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5680000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":2.6120000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.347,"15min":1.363},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.1749999999999998,"5min":2.52,"15min":2.52},"last":1.643},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.264,"15min":1.258},"min":{"1min":0.496,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.117,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":2.0299999999999998}]},{"osd":5,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3700000000000001,"15min":1.401},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9750000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.482,"15min":1.4870000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9910000000000001,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":2.4060000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:35:54 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.4199999999999999,"15min":1.4350000000000001},"min":{"1min":0.61199999999999999,"5min":0.502,"15min":0.502},"max":{"1min":2.2330000000000001,"5min":2.79,"15min":2.79},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.498,"15min":1.5109999999999999},"min":{"1min":0.63300000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9019999999999999,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.774}]}]},{"osd":6,"up_from":13,"seq":55834575086,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1765376,"data_stored":853312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.181,"15min":1.1779999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.5860000000000001,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.081},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2809999999999999,"15min":1.256},"min":{"1min":0.67700000000000005,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.0830000000000002,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":1.6919999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3100000000000001,"15min":1.2849999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6719999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.22,"5min":1.3100000000000001,"15min":1.2969999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7310000000000001,"5min":10.241,"15min":10.241},"last":1.496}]},{"osd":2,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.1639999999999999,"15min":1.204},"min":{"1min":0.70099999999999996,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.1469999999999998,"5min":2.1469999999999998,"15min":2.4940000000000002},"last":1.177},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.343,"15min":1.3520000000000001},"min":{"1min":0.88800000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1920000000000002,"5min":5.694,"15min":5.694},"last":1.5429999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.375,"15min":1.357},"min":{"1min":1.0700000000000001,"5min":0.76300000000000001,"15min":0.71999999999999997},"max":{"1min":1.804,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":1.321},{"interface":"front","average":{"1min":1.331,"5min":1.3009999999999999,"15min":1.325},"min":{"1min":0.90300000000000002,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7709999999999999,"5min":2.2349999999999999,"15min":2.6120000000000001},"last":1.4379999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.4179999999999999,"15min":1.333},"min":{"1min":0.85499999999999998,"5min":0.54200000000000004,"15min":0.48999999999999999},"max":{"1min":1.9359999999999999,"5min":13.826000000000001,"15min":13.826000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.367,"15min":1.3640000000000001},"min":{"1min":1.0309999999999999,"5min":0.63200000000000001,"15min":0.44},"max":{"1min":2.004,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.8380000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.333,"15min":1.3109999999999999},"min":{"1min":0.88100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.857,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5209999999999999,"15min":1.494},"min":{"1min":1.05,"5min":0.59599999999999997,"15min":0.56000000000000005},"max":{"1min":2.202,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.653}]},{"osd":7,"last update":"Mon Oct 16 14:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.593,"15min":1.5660000000000001},"min":{"1min":1.135,"5min":0.84099999999999997,"15min":0.58299999999999996},"max":{"1min":2.1400000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.754},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.544,"15min":1.542},"min":{"1min":1.0780000000000001,"5min":0.67500000000000004,"15min":0.54200000000000004},"max":{"1min":2.0459999999999998,"5min":2.1779999999999999,"15min":2.7250000000000001},"last":1.7230000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575086,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1769472,"data_stored":853846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.103,"15min":1.119},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.595,"5min":1.669,"15min":2.2309999999999999},"last":1.129},{"interface":"front","average":{"1min":1.129,"5min":1.163,"15min":1.1779999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.42299999999999999},"max":{"1min":1.589,"5min":1.7729999999999999,"15min":1.8340000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1200000000000001,"15min":1.165},"min":{"1min":0.69599999999999995,"5min":0.622,"15min":0.622},"max":{"1min":1.6559999999999999,"5min":1.6899999999999999,"15min":1.726},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.876,"5min":0.78600000000000003,"15min":0.60399999999999998},"max":{"1min":1.5329999999999999,"5min":1.6140000000000001,"15min":1.7809999999999999},"last":0.877}]},{"osd":2,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.83799999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8260000000000001,"5min":1.913,"15min":2.5009999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.327,"15min":1.347},"min":{"1min":0.78200000000000003,"5min":0.72199999999999998,"15min":0.625},"max":{"1min":1.6739999999999999,"5min":2.0049999999999999,"15min":2.1200000000000001},"last":1.044}]},{"osd":3,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.248,"15min":1.278},"min":{"1min":0.80700000000000005,"5min":0.76900000000000002,"15min":0.45000000000000001},"max":{"1min":1.546,"5min":1.903,"15min":1.903},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.3009999999999999,"15min":1.363},"min":{"1min":0.92800000000000005,"5min":0.78400000000000003,"15min":0.76100000000000001},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.8029999999999999},"last":1.3560000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2789999999999999,"15min":1.321},"min":{"1min":0.80100000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6359999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.391},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4430000000000001,"15min":1.4890000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":4.9480000000000004},"last":1.427}]},{"osd":6,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.327,"15min":1.377},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.47899999999999998},"max":{"1min":1.798,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.214},{"interface":"front","average":{"1min":1.359,"5min":1.3540000000000001,"15min":1.4239999999999999},"min":{"1min":0.90500000000000003,"5min":0.77700000000000002,"15min":0.55000000000000004},"max":{"1min":1.6890000000000001,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.2629999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:35:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4950000000000001,"15min":1.5469999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.641},"last":1.635},{"interface":"front","average":{"1min":1.581,"5min":1.5760000000000001,"15min":1.6080000000000001},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.58099999999999996},"max":{"1min":1.8839999999999999,"5min":2.1909999999999998,"15min":2.2010000000000001},"last":1.3120000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575087,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46792,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325048,"statfs":{"total":96636764160,"available":96588849152,"internally_reserved":0,"allocated":1761280,"data_stored":853242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.2170000000000001,"15min":1.1679999999999999},"min":{"1min":0.60799999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":7.9400000000000004,"5min":7.9400000000000004,"15min":7.9400000000000004},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.0920000000000001,"15min":1.1559999999999999},"min":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.58999999999999997},"max":{"1min":1.6379999999999999,"5min":1.7090000000000001,"15min":1.8500000000000001},"last":1.2430000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.27,"15min":1.22},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":16.959,"5min":16.959,"15min":16.959},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.764,"5min":1.3200000000000001,"15min":1.2629999999999999},"min":{"1min":0.67700000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":12.333,"5min":12.333,"15min":12.333},"last":1.5489999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.4119999999999999,"15min":1.413},"min":{"1min":0.83299999999999996,"5min":0.68100000000000005,"15min":0.67900000000000005},"max":{"1min":8.0939999999999994,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":1.609},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.3839999999999999,"15min":1.383},"min":{"1min":0.82399999999999995,"5min":0.59199999999999997,"15min":0.51600000000000001},"max":{"1min":8.2899999999999991,"5min":8.2899999999999991,"15min":8.2899999999999991},"last":1.52}]},{"osd":3,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2490000000000001,"15min":1.3069999999999999},"min":{"1min":0.73999999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.595,"5min":1.95,"15min":1.95},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.3049999999999999,"15min":1.3169999999999999},"min":{"1min":0.76100000000000001,"5min":0.53900000000000003,"15min":0.437},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.379,"15min":1.3220000000000001},"min":{"1min":0.627,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":7.8639999999999999,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.3839999999999999,"15min":1.3540000000000001},"min":{"1min":0.85099999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.766,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.411}]},{"osd":6,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.3759999999999999,"15min":1.325},"min":{"1min":0.60599999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.7869999999999999,"5min":1.5469999999999999,"15min":1.4850000000000001},"min":{"1min":0.72699999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":7.9900000000000002,"5min":7.9900000000000002,"15min":7.9900000000000002},"last":1.4410000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:36:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5109999999999999,"15min":1.48},"min":{"1min":0.90300000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.913,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.524,"15min":1.4830000000000001},"min":{"1min":0.69999999999999996,"5min":0.66700000000000004,"15min":0.49399999999999999},"max":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"last":1.4890000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575086,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1765376,"data_stored":848596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.40200000000000002,"5min":0.377,"15min":0.28999999999999998},"max":{"1min":1.5740000000000001,"5min":1.696,"15min":1.696},"last":1.367},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93899999999999995,"15min":1.012},"min":{"1min":0.48099999999999998,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":1.5760000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.0940000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.89700000000000002,"15min":0.93300000000000005},"min":{"1min":0.57499999999999996,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.583,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":1.169},{"interface":"front","average":{"1min":1.032,"5min":1.054,"15min":1.123},"min":{"1min":0.53200000000000003,"5min":0.45400000000000001,"15min":0.314},"max":{"1min":1.635,"5min":3.024,"15min":3.024},"last":0.95699999999999996}]},{"osd":2,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1759999999999999,"15min":1.2929999999999999},"min":{"1min":0.84599999999999997,"5min":0.54900000000000004,"15min":0.53800000000000003},"max":{"1min":1.792,"5min":3.3420000000000001,"15min":3.3420000000000001},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.306,"15min":1.3979999999999999},"min":{"1min":0.79000000000000004,"5min":0.53000000000000003,"15min":0.5},"max":{"1min":1.8200000000000001,"5min":2.5609999999999999,"15min":2.6299999999999999},"last":1.6839999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2150000000000001,"15min":1.25},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.56699999999999995},"max":{"1min":1.6890000000000001,"5min":2.02,"15min":2.02},"last":1.9930000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.262,"15min":1.3200000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.629},"max":{"1min":1.4810000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":2.516}]},{"osd":5,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.4199999999999999,"15min":1.5009999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.6200000000000001,"5min":3.206,"15min":3.206},"last":2.3090000000000002},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4079999999999999,"15min":1.4450000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.68799999999999994},"max":{"1min":2.0390000000000001,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":2.145}]},{"osd":6,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.3100000000000001,"15min":1.387},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.929,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.391,"15min":1.47},"min":{"1min":0.82699999999999996,"5min":0.82599999999999996,"15min":0.57999999999999996},"max":{"1min":1.78,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":2.391}]},{"osd":7,"last update":"Mon Oct 16 14:35:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5,"15min":1.5640000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.69099999999999995},"max":{"1min":2.2149999999999999,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":3.5649999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.413,"15min":1.5169999999999999},"min":{"1min":0.82099999999999995,"5min":0.749,"15min":0.629},"max":{"1min":2.1789999999999998,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":3.4239999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575087,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":1748992,"data_stored":846914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.88700000000000001,"15min":0.874},"min":{"1min":0.626,"5min":0.38,"15min":0.38},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.872},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.98999999999999999,"15min":1.0189999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":1.617,"5min":1.617,"15min":2.3260000000000001},"last":0.91800000000000004}]},{"osd":1,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98199999999999998,"15min":0.997},"min":{"1min":0.505,"5min":0.40600000000000003,"15min":0.35299999999999998},"max":{"1min":1.7649999999999999,"5min":1.8149999999999999,"15min":3.234},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.171,"5min":1.129,"15min":1.163},"min":{"1min":0.67700000000000005,"5min":0.51200000000000001,"15min":0.42199999999999999},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":3.4740000000000002},"last":1.054}]},{"osd":3,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.161,"15min":1.224},"min":{"1min":0.66700000000000004,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":2.0550000000000002,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.278,"15min":1.3069999999999999},"min":{"1min":0.84799999999999998,"5min":0.59299999999999997,"15min":0.57599999999999996},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":3.5569999999999999},"last":1.1100000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.4319999999999999,"15min":1.4330000000000001},"min":{"1min":1.0269999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.161},{"interface":"front","average":{"1min":1.331,"5min":1.294,"15min":1.343},"min":{"1min":0.754,"5min":0.754,"15min":0.72599999999999998},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":3.016},"last":1.4790000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.397,"15min":1.4139999999999999},"min":{"1min":0.84099999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.141,"5min":2.3650000000000002,"15min":2.4100000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.532,"15min":1.5529999999999999},"min":{"1min":0.995,"5min":0.90400000000000003,"15min":0.79200000000000004},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":3.3769999999999998},"last":1.508}]},{"osd":6,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3819999999999999,"15min":1.46},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.67100000000000004},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":3.0760000000000001},"last":1.25},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5009999999999999,"15min":1.526},"min":{"1min":0.93200000000000005,"5min":0.71299999999999997,"15min":0.65300000000000002},"max":{"1min":2.0230000000000001,"5min":2.5230000000000001,"15min":2.5920000000000001},"last":1.4339999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:35:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4610000000000001,"15min":1.5369999999999999},"min":{"1min":0.81299999999999994,"5min":0.70699999999999996,"15min":0.69099999999999995},"max":{"1min":2.226,"5min":2.226,"15min":3.1899999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.649,"5min":1.609,"15min":1.635},"min":{"1min":0.98999999999999999,"5min":0.81999999999999995,"15min":0.80300000000000005},"max":{"1min":2.21,"5min":2.6030000000000002,"15min":2.7519999999999998},"last":1.532}]}]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59272,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312568,"statfs":{"total":96636764160,"available":96576069632,"internally_reserved":0,"allocated":1757184,"data_stored":847038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92400000000000004,"15min":0.91300000000000003},"min":{"1min":0.48299999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.7749999999999999},"last":1.208},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0049999999999999,"15min":1.077},"min":{"1min":0.69899999999999995,"5min":0.40799999999999997,"15min":0.378},"max":{"1min":1.579,"5min":1.827,"15min":1.9370000000000001},"last":0.747}]},{"osd":2,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.93200000000000005,"15min":0.96999999999999997},"min":{"1min":0.42599999999999999,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.792,"5min":1.792,"15min":2.0259999999999998},"last":1.405},{"interface":"front","average":{"1min":1.052,"5min":1.0940000000000001,"15min":1.1499999999999999},"min":{"1min":0.53900000000000003,"5min":0.47099999999999997,"15min":0.45800000000000002},"max":{"1min":1.4339999999999999,"5min":1.6919999999999999,"15min":1.796},"last":1.6859999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2190000000000001,"15min":1.3},"min":{"1min":0.63800000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1339999999999999,"15min":1.1819999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.44600000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.855},"last":0.56999999999999995}]},{"osd":4,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2190000000000001,"15min":1.274},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.56100000000000005},"max":{"1min":1.5820000000000001,"5min":1.958,"15min":2.069},"last":1.042},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.379,"15min":1.4139999999999999},"min":{"1min":0.79500000000000004,"5min":0.74099999999999999,"15min":0.61399999999999999},"max":{"1min":1.7390000000000001,"5min":2.899,"15min":2.899},"last":1.351}]},{"osd":5,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.363,"15min":1.4239999999999999},"min":{"1min":0.92600000000000005,"5min":0.78500000000000003,"15min":0.623},"max":{"1min":2.117,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.169},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.347,"15min":1.415},"min":{"1min":0.76700000000000002,"5min":0.71199999999999997,"15min":0.59499999999999997},"max":{"1min":1.7949999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.2629999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.5209999999999999,"15min":1.5389999999999999},"min":{"1min":0.85399999999999998,"5min":0.80000000000000004,"15min":0.53100000000000003},"max":{"1min":1.839,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.4319999999999999,"15min":1.5189999999999999},"min":{"1min":0.81000000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.3220000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:35:47 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.55,"15min":1.6080000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.2530000000000001,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.4470000000000001,"15min":1.536},"min":{"1min":0.76300000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7330000000000001,"5min":2.2000000000000002,"15min":2.2170000000000001},"last":1.508}]}]},{"osd":0,"up_from":13,"seq":55834575087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2363392,"data_stored":1439006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.91300000000000003,"15min":0.93999999999999995},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.871,"5min":3.871,"15min":3.871},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.165,"5min":1.1539999999999999,"15min":1.169},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.726,"5min":3.726,"15min":3.726},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1990000000000001,"15min":1.252},"min":{"1min":0.50900000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.351,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.747},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1599999999999999,"15min":1.1990000000000001},"min":{"1min":0.54300000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.1240000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.96699999999999997,"15min":0.995},"min":{"1min":0.55700000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.109,"5min":3.109,"15min":3.109},"last":1.036},{"interface":"front","average":{"1min":1.169,"5min":1.1850000000000001,"15min":1.2230000000000001},"min":{"1min":0.71499999999999997,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":2.5489999999999999,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.2929999999999999,"15min":1.3069999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":3.4380000000000002,"5min":3.4380000000000002,"15min":3.4380000000000002},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.77000000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.4620000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":1.5329999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4039999999999999,"15min":1.4239999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":4.2119999999999997,"5min":4.2119999999999997,"15min":4.2119999999999997},"last":1.246},{"interface":"front","average":{"1min":1.589,"5min":1.3879999999999999,"15min":1.3779999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"last":1.6100000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.6040000000000001,"15min":1.589},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.7040000000000002,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.478,"15min":1.4990000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.371,"5min":3.371,"15min":3.371},"last":1.3600000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:35:52 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.5469999999999999,"15min":1.583},"min":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.8570000000000002,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.556},{"interface":"front","average":{"1min":1.702,"5min":1.5609999999999999,"15min":1.5569999999999999},"min":{"1min":0.77100000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":5.6189999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"last":1.3}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:36:33.325 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:36:33.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:36:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.599+0000 7f4f76ae5640 1 Processor -- start 2023-10-16T14:36:33.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.599+0000 7f4f76ae5640 1 -- start start 2023-10-16T14:36:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.600+0000 7f4f76ae5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f70159b20 0x7f4f7006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.600+0000 7f4f76ae5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 0x7f4f7006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.600+0000 7f4f76ae5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7006f890 con 0x7f4f70159f00 2023-10-16T14:36:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.600+0000 7f4f76ae5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7006c650 con 0x7f4f70159b20 2023-10-16T14:36:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.600+0000 7f4f76ae5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7006c930 con 0x7f4f700b0b20 2023-10-16T14:36:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.601+0000 7f4f67fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f70159f00 0x7f4f700b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46370/0 (socket says 172.21.15.29:46370) 2023-10-16T14:36:33.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.601+0000 7f4f67fff640 1 -- 172.21.15.29:0/1485281672 learned_addr learned my addr 172.21.15.29:0/1485281672 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.601+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506963957 0 0) 0x7f4f7006f890 con 0x7f4f70159f00 2023-10-16T14:36:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.601+0000 7f4f75082640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 0x7f4f7006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.601+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f58003520 con 0x7f4f70159f00 2023-10-16T14:36:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.601+0000 7f4f74881640 1 --2- 172.21.15.29:0/1485281672 >> v2:172.21.15.170:3300/0 conn(0x7f4f70159b20 0x7f4f7006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 792382430 0 0) 0x7f4f58003520 con 0x7f4f70159f00 2023-10-16T14:36:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 msgr2=0x7f4f7006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 0x7f4f7006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 >> v2:172.21.15.170:3300/0 conn(0x7f4f70159b20 msgr2=0x7f4f7006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 --2- 172.21.15.29:0/1485281672 >> v2:172.21.15.170:3300/0 conn(0x7f4f70159b20 0x7f4f7006bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7006cc10 con 0x7f4f70159f00 2023-10-16T14:36:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4f60001670 con 0x7f4f70159f00 2023-10-16T14:36:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 >> v1:172.21.15.29:6789/0 conn(0x7f4f70159f00 legacy=0x7f4f700b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f58003970 0x7f4f58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 0x7f4f580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7006c650 con 0x7f4f58004d00 2023-10-16T14:36:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7006c930 con 0x7f4f58003970 2023-10-16T14:36:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5800ba30 con 0x7f4f700b0b20 2023-10-16T14:36:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f74881640 1 --2- 172.21.15.29:0/1485281672 >> v2:172.21.15.170:3300/0 conn(0x7f4f70159b20 0x7f4f7006bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.602+0000 7f4f75082640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.603+0000 7f4f75082640 1 -- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 msgr2=0x7f4f580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.603+0000 7f4f75082640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700b0b20 0x7f4f580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.603+0000 7f4f74881640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f58003970 0x7f4f58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.603+0000 7f4f75082640 1 -- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f58003970 msgr2=0x7f4f58003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.603+0000 7f4f75082640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f58003970 0x7f4f58003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.603+0000 7f4f75082640 1 -- 172.21.15.29:0/1485281672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58003520 con 0x7f4f58004d00 2023-10-16T14:36:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.604+0000 7f4f75082640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f58005180 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f4f7015d770 tx=0x7f4f580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b421c8f85b9e074b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.604+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6802c680 con 0x7f4f58004d00 2023-10-16T14:36:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.604+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6802c7e0 con 0x7f4f58004d00 2023-10-16T14:36:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.604+0000 7f4f677fe640 1 -- 172.21.15.29:0/1485281672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6802cb90 con 0x7f4f58004d00 2023-10-16T14:36:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f76ae5640 1 -- 172.21.15.29:0/1485281672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 msgr2=0x7f4f58005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/1485281672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f58005180 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f4f7015d770 tx=0x7f4f580057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f75082640 1 -- 172.21.15.29:0/1485281672 reap_dead start 2023-10-16T14:36:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f76ae5640 1 -- 172.21.15.29:0/1485281672 shutdown_connections 2023-10-16T14:36:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f76ae5640 1 -- 172.21.15.29:0/1485281672 >> 172.21.15.29:0/1485281672 conn(0x7f4f700cb410 msgr2=0x7f4f700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f76ae5640 1 -- 172.21.15.29:0/1485281672 shutdown_connections 2023-10-16T14:36:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.605+0000 7f4f76ae5640 1 -- 172.21.15.29:0/1485281672 wait complete. 2023-10-16T14:36:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.607+0000 7f4f76ae5640 1 Processor -- start 2023-10-16T14:36:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.607+0000 7f4f76ae5640 1 -- start start 2023-10-16T14:36:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f76ae5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f701522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f76ae5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700c7100 0x7f4f700c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f76ae5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f700aef70 0x7f4f700af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f76ae5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700af940 con 0x7f4f58004d00 2023-10-16T14:36:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f76ae5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7015a590 con 0x7f4f700aef70 2023-10-16T14:36:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f76ae5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700c7c50 con 0x7f4f700c7100 2023-10-16T14:36:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.608+0000 7f4f74881640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f701522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f701522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36592/0 (socket says 172.21.15.29:36592) 2023-10-16T14:36:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 -- 172.21.15.29:0/2121408544 learned_addr learned my addr 172.21.15.29:0/2121408544 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f67fff640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700c7100 0x7f4f700c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f75082640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f700aef70 0x7f4f700af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 -- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700c7100 msgr2=0x7f4f700c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700c7100 0x7f4f700c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 -- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f700aef70 msgr2=0x7f4f700af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f700aef70 0x7f4f700af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 -- 172.21.15.29:0/2121408544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f700c7f30 con 0x7f4f58004d00 2023-10-16T14:36:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.609+0000 7f4f74881640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f701522e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c007fa0 tx=0x7f4f6c00e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.610+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c015610 con 0x7f4f58004d00 2023-10-16T14:36:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.610+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c00f040 con 0x7f4f58004d00 2023-10-16T14:36:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.610+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f700c8210 con 0x7f4f58004d00 2023-10-16T14:36:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.610+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c0159b0 con 0x7f4f58004d00 2023-10-16T14:36:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.610+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f700c87d0 con 0x7f4f58004d00 2023-10-16T14:36:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.611+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7004e160 con 0x7f4f58004d00 2023-10-16T14:36:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.612+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f6c015b10 con 0x7f4f58004d00 2023-10-16T14:36:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.613+0000 7f4f657fa640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f5c06af90 0x7f4f5c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.613+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4f6c08a7e0 con 0x7f4f58004d00 2023-10-16T14:36:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.614+0000 7f4f67fff640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f5c06af90 0x7f4f5c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.615+0000 7f4f67fff640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f5c06af90 0x7f4f5c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f60000ff0 tx=0x7f4f60001640 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.618+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f6c055640 con 0x7f4f58004d00 2023-10-16T14:36:33.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.881+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4f700c9610 con 0x7f4f58004d00 2023-10-16T14:36:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.886+0000 7f4f657fa640 1 -- 172.21.15.29:0/2121408544 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4f6c058c60 con 0x7f4f58004d00 2023-10-16T14:36:33.890 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:36:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:36:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:36:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:36:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:36:33.892 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:36:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f5c06af90 msgr2=0x7f4f5c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f5c06af90 0x7f4f5c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f60000ff0 tx=0x7f4f60001640 comp rx=0 tx=0).stop 2023-10-16T14:36:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 msgr2=0x7f4f701522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f701522e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c007fa0 tx=0x7f4f6c00e5d0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 shutdown_connections 2023-10-16T14:36:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f5c06af90 0x7f4f5c06d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f700aef70 0x7f4f700af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f700c7100 0x7f4f700c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 --2- 172.21.15.29:0/2121408544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f58004d00 0x7f4f701522e0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 >> 172.21.15.29:0/2121408544 conn(0x7f4f700cb410 msgr2=0x7f4f7015de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 shutdown_connections 2023-10-16T14:36:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:33.889+0000 7f4f76ae5640 1 -- 172.21.15.29:0/2121408544 wait complete. 2023-10-16T14:36:33.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.131+0000 7fe574f47640 1 Processor -- start 2023-10-16T14:36:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.131+0000 7fe574f47640 1 -- start start 2023-10-16T14:36:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe574f47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe57006c040 0x7fe57006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe574f47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 0x7fe57015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe574f47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57006c960 con 0x7fe57015ebc0 2023-10-16T14:36:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe574f47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57015b9d0 con 0x7fe57006c040 2023-10-16T14:36:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe574f47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57015bcb0 con 0x7fe5700ad970 2023-10-16T14:36:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe56ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe57015ebc0 0x7fe5700cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40328/0 (socket says 172.21.15.29:40328) 2023-10-16T14:36:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe56ffff640 1 -- 172.21.15.29:0/1577338420 learned_addr learned my addr 172.21.15.29:0/1577338420 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe56f7fe640 1 --2- 172.21.15.29:0/1577338420 >> v2:172.21.15.170:3300/0 conn(0x7fe57006c040 0x7fe57006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe567fff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 0x7fe57015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.133+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523046824 0 0) 0x7fe57006c960 con 0x7fe57015ebc0 2023-10-16T14:36:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c003520 con 0x7fe57015ebc0 2023-10-16T14:36:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 691772643 0 0) 0x7fe55c003520 con 0x7fe57015ebc0 2023-10-16T14:36:34.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 msgr2=0x7fe57015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 0x7fe57015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 >> v2:172.21.15.170:3300/0 conn(0x7fe57006c040 msgr2=0x7fe57006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 --2- 172.21.15.29:0/1577338420 >> v2:172.21.15.170:3300/0 conn(0x7fe57006c040 0x7fe57006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57015bf90 con 0x7fe57015ebc0 2023-10-16T14:36:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe5680019f0 con 0x7fe57015ebc0 2023-10-16T14:36:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 >> v1:172.21.15.29:6789/0 conn(0x7fe57015ebc0 legacy=0x7fe5700cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 0x7fe55c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe55c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe55c004d00 0x7fe55c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57015b9d0 con 0x7fe55c003f40 2023-10-16T14:36:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57015bcb0 con 0x7fe55c004d00 2023-10-16T14:36:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c00b980 con 0x7fe5700ad970 2023-10-16T14:36:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.134+0000 7fe567fff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe55c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe56f7fe640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 0x7fe55c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe56ffff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe55c004d00 0x7fe55c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe567fff640 1 -- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 msgr2=0x7fe55c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe567fff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700ad970 0x7fe55c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe567fff640 1 -- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe55c004d00 msgr2=0x7fe55c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe567fff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe55c004d00 0x7fe55c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe567fff640 1 -- 172.21.15.29:0/1577338420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c003520 con 0x7fe55c003f40 2023-10-16T14:36:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.135+0000 7fe56ffff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe55c004d00 0x7fe55c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.136+0000 7fe567fff640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe55c0047c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe558009af0 tx=0x7fe55c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69a59d7049c770 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55802c6b0 con 0x7fe55c003f40 2023-10-16T14:36:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55802c810 con 0x7fe55c003f40 2023-10-16T14:36:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe56effd640 1 -- 172.21.15.29:0/1577338420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55802cba0 con 0x7fe55c003f40 2023-10-16T14:36:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe574f47640 1 -- 172.21.15.29:0/1577338420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 msgr2=0x7fe55c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe574f47640 1 --2- 172.21.15.29:0/1577338420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe55c0047c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe558009af0 tx=0x7fe55c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe56ffff640 1 -- 172.21.15.29:0/1577338420 reap_dead start 2023-10-16T14:36:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe574f47640 1 -- 172.21.15.29:0/1577338420 shutdown_connections 2023-10-16T14:36:34.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe574f47640 1 -- 172.21.15.29:0/1577338420 >> 172.21.15.29:0/1577338420 conn(0x7fe5700ce6a0 msgr2=0x7fe5700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:34.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe574f47640 1 -- 172.21.15.29:0/1577338420 shutdown_connections 2023-10-16T14:36:34.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.137+0000 7fe574f47640 1 -- 172.21.15.29:0/1577338420 wait complete. 2023-10-16T14:36:34.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.139+0000 7fe574f47640 1 Processor -- start 2023-10-16T14:36:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.140+0000 7fe574f47640 1 -- start start 2023-10-16T14:36:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.140+0000 7fe574f47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe570156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.140+0000 7fe574f47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5700c7da0 0x7fe5700c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe574f47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700c95c0 0x7fe5700c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe574f47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700c9f80 con 0x7fe55c003f40 2023-10-16T14:36:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe574f47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5701f66e0 con 0x7fe5700c7da0 2023-10-16T14:36:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe574f47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe570157f70 con 0x7fe5700c95c0 2023-10-16T14:36:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe570156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe570156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43840/0 (socket says 172.21.15.29:43840) 2023-10-16T14:36:34.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 -- 172.21.15.29:0/3314628124 learned_addr learned my addr 172.21.15.29:0/3314628124 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 -- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700c95c0 msgr2=0x7fe5700c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700c95c0 0x7fe5700c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 -- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5700c7da0 msgr2=0x7fe5700c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe567fff640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5700c7da0 0x7fe5700c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5700c7da0 0x7fe5700c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.141+0000 7fe56f7fe640 1 -- 172.21.15.29:0/3314628124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5701f5ad0 con 0x7fe55c003f40 2023-10-16T14:36:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.142+0000 7fe56f7fe640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe570156ce0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fe560011730 tx=0x7fe560011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.142+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56000f9c0 con 0x7fe55c003f40 2023-10-16T14:36:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.142+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56000fb20 con 0x7fe55c003f40 2023-10-16T14:36:34.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.142+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56000fe90 con 0x7fe55c003f40 2023-10-16T14:36:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.142+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5701f5db0 con 0x7fe55c003f40 2023-10-16T14:36:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.142+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5701f6370 con 0x7fe55c003f40 2023-10-16T14:36:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.143+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57006b330 con 0x7fe55c003f40 2023-10-16T14:36:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.144+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe560015460 con 0x7fe55c003f40 2023-10-16T14:36:34.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.145+0000 7fe56cff9640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe55006af90 0x7fe55006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:34.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.145+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe560089dd0 con 0x7fe55c003f40 2023-10-16T14:36:34.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.146+0000 7fe567fff640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe55006af90 0x7fe55006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:34.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.147+0000 7fe567fff640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe55006af90 0x7fe55006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe558036000 tx=0x7fe558030000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:34.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.149+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe560054cb0 con 0x7fe55c003f40 2023-10-16T14:36:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.381+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe57004e9b0 con 0x7fe55c003f40 2023-10-16T14:36:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.383+0000 7fe56cff9640 1 -- 172.21.15.29:0/3314628124 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe560022090 con 0x7fe55c003f40 2023-10-16T14:36:34.387 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:34.387 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:36:27.196747+0000","modified":"2023-10-16T14:36:30.291825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1700644562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1700644562},{"type":"v1","addr":"172.21.15.29:6835","nonce":1700644562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T14:36:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-16T14:36:34.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.385+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe55006af90 msgr2=0x7fe55006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe55006af90 0x7fe55006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe558036000 tx=0x7fe558030000 comp rx=0 tx=0).stop 2023-10-16T14:36:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 msgr2=0x7fe570156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe570156ce0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fe560011730 tx=0x7fe560011bf0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 shutdown_connections 2023-10-16T14:36:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe55006af90 0x7fe55006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5700c95c0 0x7fe5700c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5700c7da0 0x7fe5700c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 --2- 172.21.15.29:0/3314628124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55c003f40 0x7fe570156ce0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:34.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 >> 172.21.15.29:0/3314628124 conn(0x7fe5700ce6a0 msgr2=0x7fe5700b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:34.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 shutdown_connections 2023-10-16T14:36:34.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:34.386+0000 7fe574f47640 1 -- 172.21.15.29:0/3314628124 wait complete. 2023-10-16T14:36:34.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:36:27.196747+0000', 'modified': '2023-10-16T14:36:30.291825+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8283': {'gid': 8283, 'name': 'a', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/1700644562', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 1700644562}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 1700644562}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:36:34.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:36:34.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:36:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.048+0000 7fe9d9e9a640 1 Processor -- start 2023-10-16T14:36:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.048+0000 7fe9d9e9a640 1 -- start start 2023-10-16T14:36:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.049+0000 7fe9d9e9a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9d40cdf30 0x7fe9d4071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.049+0000 7fe9d9e9a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d40d12f0 con 0x7fe9d415cc80 2023-10-16T14:36:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.049+0000 7fe9d9e9a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d40d06e0 con 0x7fe9d4071fa0 2023-10-16T14:36:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.049+0000 7fe9d9e9a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d40d0500 con 0x7fe9d40cdf30 2023-10-16T14:36:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9d8e98640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe9d415cc80 0x7fe9d40cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40334/0 (socket says 172.21.15.29:40334) 2023-10-16T14:36:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9d8e98640 1 -- 172.21.15.29:0/4057741674 learned_addr learned my addr 172.21.15.29:0/4057741674 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/4057741674 >> v2:172.21.15.29:3300/0 conn(0x7fe9d40cdf30 0x7fe9d4071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9cb7fe640 1 -- 172.21.15.29:0/4057741674 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241484794 0 0) 0x7fe9d40d12f0 con 0x7fe9d415cc80 2023-10-16T14:36:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9cbfff640 1 -- 172.21.15.29:0/4057741674 >> v1:172.21.15.170:6789/0 conn(0x7fe9d4071fa0 legacy=0x7fe9d406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9cbfff640 1 -- 172.21.15.29:0/4057741674 >> v1:172.21.15.29:6789/0 conn(0x7fe9d415cc80 legacy=0x7fe9d40cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.050+0000 7fe9cbfff640 1 -- 172.21.15.29:0/4057741674 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d40d09c0 con 0x7fe9d40cdf30 2023-10-16T14:36:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.052+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/4057741674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4071a60 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40097a0 tx=0x7fe9c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6402e7466558140 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.052+0000 7fe9cb7fe640 1 -- 172.21.15.29:0/4057741674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4031830 con 0x7fe9d40cdf30 2023-10-16T14:36:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.052+0000 7fe9cb7fe640 1 -- 172.21.15.29:0/4057741674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c4031990 con 0x7fe9d40cdf30 2023-10-16T14:36:35.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.052+0000 7fe9cb7fe640 1 -- 172.21.15.29:0/4057741674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4031d20 con 0x7fe9d40cdf30 2023-10-16T14:36:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.052+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/4057741674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d40cdf30 msgr2=0x7fe9d4071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.052+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/4057741674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4071a60 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40097a0 tx=0x7fe9c402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.053+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/4057741674 shutdown_connections 2023-10-16T14:36:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.053+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/4057741674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4071a60 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.053+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/4057741674 >> 172.21.15.29:0/4057741674 conn(0x7fe9d4065ed0 msgr2=0x7fe9d40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.053+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/4057741674 shutdown_connections 2023-10-16T14:36:35.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.053+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/4057741674 wait complete. 2023-10-16T14:36:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.054+0000 7fe9d9e9a640 1 Processor -- start 2023-10-16T14:36:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.054+0000 7fe9d9e9a640 1 -- start start 2023-10-16T14:36:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.055+0000 7fe9d9e9a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9d4071fa0 0x7fe9d40c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.055+0000 7fe9d9e9a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.055+0000 7fe9d9e9a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 0x7fe9d40ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.055+0000 7fe9d9e9a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b5160 con 0x7fe9d415cc80 2023-10-16T14:36:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.055+0000 7fe9d9e9a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b4550 con 0x7fe9d40cdf30 2023-10-16T14:36:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.055+0000 7fe9d9e9a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b4830 con 0x7fe9d4071fa0 2023-10-16T14:36:35.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 0x7fe9d40ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 0x7fe9d40ae4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43856/0 (socket says 172.21.15.29:43856) 2023-10-16T14:36:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 -- 172.21.15.29:0/1994521292 learned_addr learned my addr 172.21.15.29:0/1994521292 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9d4071fa0 0x7fe9d40c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 -- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9d4071fa0 msgr2=0x7fe9d40c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9d4071fa0 0x7fe9d40c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 -- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d40cdf30 msgr2=0x7fe9d4151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d9699640 1 -- 172.21.15.29:0/1994521292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4009400 con 0x7fe9d415cc80 2023-10-16T14:36:35.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9d4071fa0 0x7fe9d40c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:35.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.056+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:35.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.057+0000 7fe9d9699640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 0x7fe9d40ae4c0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00c410 tx=0x7fe9cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.057+0000 7fe9c97fa640 1 -- 172.21.15.29:0/1994521292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc011790 con 0x7fe9d415cc80 2023-10-16T14:36:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.057+0000 7fe9c97fa640 1 -- 172.21.15.29:0/1994521292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0118f0 con 0x7fe9d415cc80 2023-10-16T14:36:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.057+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d40b4b10 con 0x7fe9d415cc80 2023-10-16T14:36:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.057+0000 7fe9c97fa640 1 -- 172.21.15.29:0/1994521292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc011ce0 con 0x7fe9d415cc80 2023-10-16T14:36:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.057+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d4152260 con 0x7fe9d415cc80 2023-10-16T14:36:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.059+0000 7fe9c97fa640 1 -- 172.21.15.29:0/1994521292 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe9cc01c460 con 0x7fe9d415cc80 2023-10-16T14:36:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.060+0000 7fe9c97fa640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe9c006afc0 0x7fe9c006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.060+0000 7fe9c97fa640 1 -- 172.21.15.29:0/1994521292 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe9cc0894d0 con 0x7fe9d415cc80 2023-10-16T14:36:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.061+0000 7fe9ddd66740 1 Processor -- start 2023-10-16T14:36:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.061+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe9c006afc0 0x7fe9c006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.061+0000 7fe9ddd66740 1 -- start start 2023-10-16T14:36:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4cf6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 0x5597f4d25f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4cf65b0 0x5597f4d26440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4c167a0 con 0x5597f4d23550 2023-10-16T14:36:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4c68740 con 0x5597f4cf65b0 2023-10-16T14:36:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597f4c65e30 con 0x5597f4d23170 2023-10-16T14:36:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 0x5597f4d25f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe9c006afc0 0x7fe9c006d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0007fd0 tx=0x7fe9d0009040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 0x5597f4d25f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43870/0 (socket says 172.21.15.29:43870) 2023-10-16T14:36:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.062+0000 7fe9cbfff640 1 -- 172.21.15.29:0/1134626260 learned_addr learned my addr 172.21.15.29:0/1134626260 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9d9699640 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4cf65b0 0x5597f4d26440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4cf6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9cbfff640 1 -- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 msgr2=0x5597f4cf6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4cf6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9cbfff640 1 -- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4cf65b0 msgr2=0x5597f4d26440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4cf65b0 0x5597f4d26440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9cbfff640 1 -- 172.21.15.29:0/1134626260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597f4c25910 con 0x5597f4d23550 2023-10-16T14:36:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 0x5597f4d25f00 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40098c0 tx=0x7fe9c4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8784ee12b2429f66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9ba7fc640 1 -- 172.21.15.29:0/1134626260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4040070 con 0x5597f4d23550 2023-10-16T14:36:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9ba7fc640 1 -- 172.21.15.29:0/1134626260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c4031b00 con 0x5597f4d23550 2023-10-16T14:36:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.063+0000 7fe9ba7fc640 1 -- 172.21.15.29:0/1134626260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4031e70 con 0x5597f4d23550 2023-10-16T14:36:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1134626260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 msgr2=0x5597f4d25f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 0x5597f4d25f00 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40098c0 tx=0x7fe9c4030000 comp rx=0 tx=0).stop 2023-10-16T14:36:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1134626260 shutdown_connections 2023-10-16T14:36:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d23550 0x5597f4d25f00 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4cf65b0 0x5597f4d26440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1134626260 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4cf6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1134626260 >> 172.21.15.29:0/1134626260 conn(0x5597f4cf8380 msgr2=0x5597f4d40f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1134626260 shutdown_connections 2023-10-16T14:36:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.064+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1134626260 wait complete. 2023-10-16T14:36:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.065+0000 7fe9ddd66740 1 -- start start 2023-10-16T14:36:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.065+0000 7fe9ddd66740 1 Processor -- start 2023-10-16T14:36:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d1bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4c61720 con 0x5597f4d107a0 2023-10-16T14:36:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4bff970 con 0x5597f4d23550 2023-10-16T14:36:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597f4bff610 con 0x5597f4d23170 2023-10-16T14:36:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.066+0000 7fe9ddd66740 10 client.0 _ll_register_callbacks cb 0x5597f4c219d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:36:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43872/0 (socket says 172.21.15.29:43872) 2023-10-16T14:36:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 -- 172.21.15.29:0/2581688356 learned_addr learned my addr 172.21.15.29:0/2581688356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d1bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 -- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 msgr2=0x5597f4d1bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d1bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 -- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 msgr2=0x5597f4d1da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 -- 172.21.15.29:0/2581688356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d40d09c0 con 0x5597f4d107a0 2023-10-16T14:36:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.067+0000 7fe9d9699640 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1df90 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc057d70 tx=0x7fe9cc0588b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcab6d44827b1624 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:36:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9b37fe640 1 -- 172.21.15.29:0/2581688356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc060370 con 0x5597f4d107a0 2023-10-16T14:36:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9b37fe640 1 -- 172.21.15.29:0/2581688356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe9cc05e0b0 con 0x5597f4d107a0 2023-10-16T14:36:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597f4c88160 con 0x5597f4d107a0 2023-10-16T14:36:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9b37fe640 1 -- 172.21.15.29:0/2581688356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9cc05e270 con 0x5597f4d107a0 2023-10-16T14:36:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9ddd66740 10 client.8424 fetch_fsmap learned FSMap version 66 2023-10-16T14:36:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5597f4c5fda0 con 0x5597f4d107a0 2023-10-16T14:36:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9b37fe640 1 -- 172.21.15.29:0/2581688356 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5597f4c88160 con 0x5597f4d107a0 2023-10-16T14:36:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9b37fe640 1 -- 172.21.15.29:0/2581688356 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe9cc0582d0 con 0x5597f4d107a0 2023-10-16T14:36:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9ddd66740 10 client.8424 fetch_fsmap finished waiting for FSMap version 66 2023-10-16T14:36:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9ddd66740 10 client.8424 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:36:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.068+0000 7fe9ddd66740 20 client.8424 populate_metadata read hostname 'smithi029' 2023-10-16T14:36:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.069+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d5c3f0 0x5597f4d5e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.069+0000 7fe9ddd66740 4 client.8424 mds_command: new command op to 8283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:36:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.069+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 --> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597f4c33cc0 con 0x5597f4d5c3f0 2023-10-16T14:36:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.069+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d5c3f0 0x5597f4d5e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.069+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d5c3f0 0x5597f4d5e7e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc1d471ddbddb6d2 server_cookie=b981dab79277eb04 in_seq=0 out_seq=0 2023-10-16T14:36:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.069+0000 7fe9b37fe640 10 client.8424 ms_handle_connect on v2:172.21.15.29:6833/1700644562 2023-10-16T14:36:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.070+0000 7fe9b37fe640 1 -- 172.21.15.29:0/2581688356 <== mds.0 v2:172.21.15.29:6833/1700644562 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5597f4c33cc0 con 0x5597f4d5c3f0 2023-10-16T14:36:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.070+0000 7fe9b37fe640 10 client.8424 handle_command_reply: tid=0 2023-10-16T14:36:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.070+0000 7fe9ddd66740 1 client.8424 shutdown 2023-10-16T14:36:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 msgr2=0x5597f4d1df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1df90 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc057d70 tx=0x7fe9cc0588b0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 shutdown_connections 2023-10-16T14:36:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d5c3f0 0x5597f4d5e7e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1df90 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/2581688356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d1bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9b37fe640 0 client.8424 ms_handle_reset on v2:172.21.15.29:6833/1700644562 2023-10-16T14:36:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 >> 172.21.15.29:0/2581688356 conn(0x5597f4cf65b0 msgr2=0x5597f4d44380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 shutdown_connections 2023-10-16T14:36:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 1 -- 172.21.15.29:0/2581688356 wait complete. 2023-10-16T14:36:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.071+0000 7fe9ddd66740 20 client.8424 trim_cache size 0 max 16384 2023-10-16T14:36:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.088+0000 7fe9ddd66740 1 Processor -- start 2023-10-16T14:36:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.088+0000 7fe9ddd66740 1 -- start start 2023-10-16T14:36:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d299d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f49c5f20 con 0x5597f4d107a0 2023-10-16T14:36:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4d1d1b0 con 0x5597f4d23550 2023-10-16T14:36:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597f4d23b60 con 0x5597f4d23170 2023-10-16T14:36:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9d8e98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9d8e98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43884/0 (socket says 172.21.15.29:43884) 2023-10-16T14:36:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.089+0000 7fe9d8e98640 1 -- 172.21.15.29:0/3192218181 learned_addr learned my addr 172.21.15.29:0/3192218181 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d8e98640 1 -- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 msgr2=0x5597f4d299d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d299d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d8e98640 1 -- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 msgr2=0x5597f4d1cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d9699640 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d8e98640 1 -- 172.21.15.29:0/3192218181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597f4bfeb80 con 0x5597f4d107a0 2023-10-16T14:36:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1bd20 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x5597f4d24a70 tx=0x7fe9d0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ab0c90b2e3a726 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9baffd640 1 -- 172.21.15.29:0/3192218181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00243d0 con 0x5597f4d107a0 2023-10-16T14:36:35.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9baffd640 1 -- 172.21.15.29:0/3192218181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0028070 con 0x5597f4d107a0 2023-10-16T14:36:35.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.090+0000 7fe9baffd640 1 -- 172.21.15.29:0/3192218181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0024a20 con 0x5597f4d107a0 2023-10-16T14:36:35.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 -- 172.21.15.29:0/3192218181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 msgr2=0x5597f4d1bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1bd20 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x5597f4d24a70 tx=0x7fe9d0011000 comp rx=0 tx=0).stop 2023-10-16T14:36:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 -- 172.21.15.29:0/3192218181 shutdown_connections 2023-10-16T14:36:35.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23550 0x5597f4d1cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23170 0x5597f4d299d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/3192218181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d1bd20 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 -- 172.21.15.29:0/3192218181 >> 172.21.15.29:0/3192218181 conn(0x5597f4cf65b0 msgr2=0x5597f4cf1810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 -- 172.21.15.29:0/3192218181 shutdown_connections 2023-10-16T14:36:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.091+0000 7fe9ddd66740 1 -- 172.21.15.29:0/3192218181 wait complete. 2023-10-16T14:36:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.092+0000 7fe9ddd66740 1 -- start start 2023-10-16T14:36:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.092+0000 7fe9ddd66740 1 Processor -- start 2023-10-16T14:36:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d42b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23170 0x5597f4da8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23550 0x5597f4d1ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4d53bd0 con 0x5597f4d107a0 2023-10-16T14:36:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f4d431f0 con 0x5597f4d23170 2023-10-16T14:36:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597f4d43720 con 0x5597f4d23550 2023-10-16T14:36:35.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9ddd66740 10 client.0 _ll_register_callbacks cb 0x5597f4cf6500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:36:35.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9d8e98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d42b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9d8e98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d42b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43886/0 (socket says 172.21.15.29:43886) 2023-10-16T14:36:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.093+0000 7fe9d8e98640 1 -- 172.21.15.29:0/1273185908 learned_addr learned my addr 172.21.15.29:0/1273185908 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9d8e98640 1 -- 172.21.15.29:0/1273185908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23550 msgr2=0x5597f4d1ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23550 0x5597f4d1ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9d8e98640 1 -- 172.21.15.29:0/1273185908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23170 msgr2=0x5597f4da8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23170 0x5597f4da8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9d8e98640 1 -- 172.21.15.29:0/1273185908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d001c040 con 0x5597f4d107a0 2023-10-16T14:36:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9d8e98640 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d42b60 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x5597f4d24aa0 tx=0x7fe9d0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b645f3c5fbf99164 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:36:35.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9b37fe640 1 -- 172.21.15.29:0/1273185908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0024a20 con 0x5597f4d107a0 2023-10-16T14:36:35.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9b37fe640 1 -- 172.21.15.29:0/1273185908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe9d0024b80 con 0x5597f4d107a0 2023-10-16T14:36:35.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.094+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9c4031b00 con 0x5597f4d107a0 2023-10-16T14:36:35.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9b37fe640 1 -- 172.21.15.29:0/1273185908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9d0024d40 con 0x5597f4d107a0 2023-10-16T14:36:35.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 10 client.8430 fetch_fsmap learned FSMap version 66 2023-10-16T14:36:35.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5597f4c24170 con 0x5597f4d107a0 2023-10-16T14:36:35.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9b37fe640 1 -- 172.21.15.29:0/1273185908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c4031b00 con 0x5597f4d107a0 2023-10-16T14:36:35.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9b37fe640 1 -- 172.21.15.29:0/1273185908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe9d00074a0 con 0x5597f4d107a0 2023-10-16T14:36:35.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 10 client.8430 fetch_fsmap finished waiting for FSMap version 66 2023-10-16T14:36:35.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 10 client.8430 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:36:35.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 20 client.8430 populate_metadata read hostname 'smithi029' 2023-10-16T14:36:35.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d53240 0x5597f4d53620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 4 client.8430 mds_command: new command op to 8283 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:36:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 --> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5597f4c5a700 con 0x5597f4d53240 2023-10-16T14:36:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.095+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d53240 0x5597f4d53620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.096+0000 7fe9cbfff640 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d53240 0x5597f4d53620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7c4c23d6e48fbae server_cookie=48338c073fcc69 in_seq=0 out_seq=0 2023-10-16T14:36:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.096+0000 7fe9b37fe640 10 client.8430 ms_handle_connect on v2:172.21.15.29:6833/1700644562 2023-10-16T14:36:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.096+0000 7fe9b37fe640 1 -- 172.21.15.29:0/1273185908 <== mds.0 v2:172.21.15.29:6833/1700644562 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5597f4c5a700 con 0x5597f4d53240 2023-10-16T14:36:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.096+0000 7fe9b37fe640 10 client.8430 handle_command_reply: tid=0 2023-10-16T14:36:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.096+0000 7fe9ddd66740 1 client.8430 shutdown 2023-10-16T14:36:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 msgr2=0x5597f4d42b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d42b60 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x5597f4d24aa0 tx=0x7fe9d0010000 comp rx=0 tx=0).stop 2023-10-16T14:36:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 shutdown_connections 2023-10-16T14:36:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:6833/1700644562,v1:172.21.15.29:6835/1700644562] conn(0x5597f4d53240 0x5597f4d53620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x5597f4d23550 0x5597f4d1ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5597f4d23170 0x5597f4da8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 --2- 172.21.15.29:0/1273185908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5597f4d107a0 0x5597f4d42b60 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9b37fe640 0 client.8430 ms_handle_reset on v2:172.21.15.29:6833/1700644562 2023-10-16T14:36:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 >> 172.21.15.29:0/1273185908 conn(0x5597f4cf65b0 msgr2=0x5597f4cf1810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 shutdown_connections 2023-10-16T14:36:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 1 -- 172.21.15.29:0/1273185908 wait complete. 2023-10-16T14:36:35.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.097+0000 7fe9ddd66740 20 client.8430 trim_cache size 0 max 16384 2023-10-16T14:36:35.122 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:36:35.122 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:36:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:36:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8283, 2023-10-16T14:36:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:36:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:36:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:36:35.124 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 7.3306969640000004, 2023-10-16T14:36:35.124 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 66, 2023-10-16T14:36:35.124 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 142, 2023-10-16T14:36:35.124 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 142, 2023-10-16T14:36:35.125 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.683457615 2023-10-16T14:36:35.125 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:36:35.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe9c006afc0 msgr2=0x7fe9c006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe9c006afc0 0x7fe9c006d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0007fd0 tx=0x7fe9d0009040 comp rx=0 tx=0).stop 2023-10-16T14:36:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 msgr2=0x7fe9d40ae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 0x7fe9d40ae4c0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00c410 tx=0x7fe9cc00c8d0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 shutdown_connections 2023-10-16T14:36:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d415cc80 0x7fe9d40ae4c0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fe9c006afc0 0x7fe9c006d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9d40cdf30 0x7fe9d4151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 --2- 172.21.15.29:0/1994521292 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9d4071fa0 0x7fe9d40c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 >> 172.21.15.29:0/1994521292 conn(0x7fe9d4065ed0 msgr2=0x7fe9d406b740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 shutdown_connections 2023-10-16T14:36:35.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.098+0000 7fe9d9e9a640 1 -- 172.21.15.29:0/1994521292 wait complete. 2023-10-16T14:36:35.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:36:35.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.344+0000 7f4faa505640 1 Processor -- start 2023-10-16T14:36:35.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.345+0000 7f4faa505640 1 -- start start 2023-10-16T14:36:35.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.346+0000 7f4faa505640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4fa415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.346+0000 7f4faa505640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa415ed60 con 0x7f4fa415ae80 2023-10-16T14:36:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.346+0000 7f4faa505640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa40707e0 con 0x7f4fa40ce390 2023-10-16T14:36:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.346+0000 7f4faa505640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa406b810 con 0x7f4fa415aaa0 2023-10-16T14:36:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4fa415ae80 0x7f4fa40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40344/0 (socket says 172.21.15.29:40344) 2023-10-16T14:36:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa37fe640 1 -- 172.21.15.29:0/743230818 learned_addr learned my addr 172.21.15.29:0/743230818 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:35.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa3fff640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4fa415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089612416 0 0) 0x7f4fa415ed60 con 0x7f4fa415ae80 2023-10-16T14:36:35.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f90003520 con 0x7f4fa415ae80 2023-10-16T14:36:35.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051583454 0 0) 0x7f4f90003520 con 0x7f4fa415ae80 2023-10-16T14:36:35.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 msgr2=0x7f4fa415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4fa415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 >> v1:172.21.15.170:6789/0 conn(0x7f4fa40ce390 legacy=0x7f4fa4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.347+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa415ef90 con 0x7f4fa415ae80 2023-10-16T14:36:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4f94001670 con 0x7f4fa415ae80 2023-10-16T14:36:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 >> v1:172.21.15.29:6789/0 conn(0x7f4fa415ae80 legacy=0x7f4fa40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4f900039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f90003fd0 0x7f4f900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f90004d00 0x7f4f90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa406b810 con 0x7f4f90004d00 2023-10-16T14:36:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9000bb40 con 0x7f4f90003fd0 2023-10-16T14:36:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9000bd00 con 0x7f4fa415aaa0 2023-10-16T14:36:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f90004d00 0x7f4f90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa3fff640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4f900039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa37fe640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f90003fd0 0x7f4f900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 msgr2=0x7f4f900039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4f900039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f90003fd0 msgr2=0x7f4f900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:35.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f90003fd0 0x7f4f900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.348+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/743230818 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90003520 con 0x7f4f90004d00 2023-10-16T14:36:35.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.349+0000 7f4fa3fff640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4fa415aaa0 0x7f4f900039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:35.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f90004d00 0x7f4f90005180 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c001e50 tx=0x7f4f900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17abc8bfbf62a16f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c003b70 con 0x7f4f90004d00 2023-10-16T14:36:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c003cd0 con 0x7f4f90004d00 2023-10-16T14:36:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4fa2ffd640 1 -- 172.21.15.29:0/743230818 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c005310 con 0x7f4f90004d00 2023-10-16T14:36:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4faa505640 1 -- 172.21.15.29:0/743230818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f90004d00 msgr2=0x7f4f90005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4faa505640 1 --2- 172.21.15.29:0/743230818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f90004d00 0x7f4f90005180 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c001e50 tx=0x7f4f900057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.350+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/743230818 reap_dead start 2023-10-16T14:36:35.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.351+0000 7f4faa505640 1 -- 172.21.15.29:0/743230818 shutdown_connections 2023-10-16T14:36:35.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.351+0000 7f4faa505640 1 -- 172.21.15.29:0/743230818 >> 172.21.15.29:0/743230818 conn(0x7f4fa40cb410 msgr2=0x7f4fa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.351+0000 7f4faa505640 1 -- 172.21.15.29:0/743230818 shutdown_connections 2023-10-16T14:36:35.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.351+0000 7f4faa505640 1 -- 172.21.15.29:0/743230818 wait complete. 2023-10-16T14:36:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.352+0000 7f4faa505640 1 Processor -- start 2023-10-16T14:36:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.353+0000 7f4faa505640 1 -- start start 2023-10-16T14:36:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.353+0000 7f4faa505640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90004d00 0x7f4fa4157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.353+0000 7f4faa505640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4fa4157a20 0x7f4fa40c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4faa505640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 0x7f4fa40c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4faa505640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40c5340 con 0x7f4fa40c40c0 2023-10-16T14:36:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4faa505640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40c2c20 con 0x7f4fa4157a20 2023-10-16T14:36:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4faa505640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40c5060 con 0x7f4f90004d00 2023-10-16T14:36:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 0x7f4fa40c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 0x7f4fa40c4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43904/0 (socket says 172.21.15.29:43904) 2023-10-16T14:36:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/3435785426 learned_addr learned my addr 172.21.15.29:0/3435785426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa37fe640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4fa4157a20 0x7f4fa40c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa3fff640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90004d00 0x7f4fa4157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90004d00 msgr2=0x7f4fa4157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90004d00 0x7f4fa4157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4fa4157a20 msgr2=0x7f4fa40c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4fa4157a20 0x7f4fa40c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.354+0000 7f4fa8aa2640 1 -- 172.21.15.29:0/3435785426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa40c2660 con 0x7f4fa40c40c0 2023-10-16T14:36:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.355+0000 7f4fa8aa2640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 0x7f4fa40c4550 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c001f70 tx=0x7f4f9c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.355+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c03e070 con 0x7f4fa40c40c0 2023-10-16T14:36:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.355+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c003e40 con 0x7f4fa40c40c0 2023-10-16T14:36:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.355+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c006d70 con 0x7f4fa40c40c0 2023-10-16T14:36:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.355+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa40c2940 con 0x7f4fa40c40c0 2023-10-16T14:36:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.355+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa40c4c70 con 0x7f4fa40c40c0 2023-10-16T14:36:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.357+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa40c24c0 con 0x7f4fa40c40c0 2023-10-16T14:36:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.358+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f9c007030 con 0x7f4fa40c40c0 2023-10-16T14:36:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.359+0000 7f4fa0ff9640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f8406afc0 0x7f4f8406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.359+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4f9c0aaa60 con 0x7f4fa40c40c0 2023-10-16T14:36:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.360+0000 7f4fa3fff640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f8406afc0 0x7f4f8406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.361+0000 7f4fa3fff640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f8406afc0 0x7f4f8406d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40b06a0 tx=0x7f4f8c008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.364+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4fa40c24c0 con 0x7f4fa40c40c0 2023-10-16T14:36:35.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.596+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4fa40c24c0 con 0x7f4fa40c40c0 2023-10-16T14:36:35.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.596+0000 7f4fa0ff9640 1 -- 172.21.15.29:0/3435785426 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4fa40c24c0 con 0x7f4fa40c40c0 2023-10-16T14:36:35.601 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:36:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f8406afc0 msgr2=0x7f4f8406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f8406afc0 0x7f4f8406d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40b06a0 tx=0x7f4f8c008040 comp rx=0 tx=0).stop 2023-10-16T14:36:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 msgr2=0x7f4fa40c4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 0x7f4fa40c4550 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c001f70 tx=0x7f4f9c0014a0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 shutdown_connections 2023-10-16T14:36:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fa40c40c0 0x7f4fa40c4550 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4f8406afc0 0x7f4f8406d480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4fa4157a20 0x7f4fa40c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 --2- 172.21.15.29:0/3435785426 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f90004d00 0x7f4fa4157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 >> 172.21.15.29:0/3435785426 conn(0x7f4fa40cb410 msgr2=0x7f4fa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 shutdown_connections 2023-10-16T14:36:35.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:35.599+0000 7f4faa505640 1 -- 172.21.15.29:0/3435785426 wait complete. 2023-10-16T14:36:35.623 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:36:35.623 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:36:35.624 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:36:35.624 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:36:35.624 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:36:35.625 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:36:35.625 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:36:35.625 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:36:35.626 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:36:35.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:35.627 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:36:35.682 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:36:35.683 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:36:35.683 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:36:35.683 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.683 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:36:35.684 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.684 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:36:35.684 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:36:35.684 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:36:35.684 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:36:35.685 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:36:35.685 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:36:35.685 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:36:35.685 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:36:35.686 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:36:35.686 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:36:35.686 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41129sec preferred_lft 41129sec 2023-10-16T14:36:35.686 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:36:35.686 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.687 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:36:35.687 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:36:35.687 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:36:35.687 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:36:35.687 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:36:35.687 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:36:35.688 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.688 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:36:35.688 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.688 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:36:35.688 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:36:35.689 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:36:35.689 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.689 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:36:35.689 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:36:35.689 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:36:35.690 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:36:35.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:35.691 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:36:35.796 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:36:35.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:35.802 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:36:35.859 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:36:35.859 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:36:35.859 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:36:35.860 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:36:35.860 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:36:35.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:35.861 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:36:35.861 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:36:35.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:36:35.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:36:35.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:36:35.862 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:36:36.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:36.057 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:36:36.112 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:36:36.250 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:36:36.252 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:36:36.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:36.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:36:36.369 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:36:36.370 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:36:36.430 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:36:36.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:36:36.431 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:36:36.487 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:36:36.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:36.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:36:36.646 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:36:36.646 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:36:36.646 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:36:36.669 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:36:36.669 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:36:36.669 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:36:36.670 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:36:36.670 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:36:36.694 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:36:36.694 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:36:36.695 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:36:36.695 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:36:36.695 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:36:36.695 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:36:36.695 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:36:36.696 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:36:36.696 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:36:36.696 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:36:36.697 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:36:36.697 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:36:36.697 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:36:36.700 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:36:36.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:36.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:36:36.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:36.791 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:36:36.863 INFO:teuthology.orchestra.run.smithi029.stdout:client6422 2023-10-16T14:36:36.864 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:36.864 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:36.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:36:37.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.142+0000 7f13ecc1a640 1 Processor -- start 2023-10-16T14:36:37.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.142+0000 7f13ecc1a640 1 -- start start 2023-10-16T14:36:37.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.143+0000 7f13ecc1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f13e8159950 0x7f13e815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13ecc1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e815aef0 con 0x7f13e806c810 2023-10-16T14:36:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13ecc1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e815d910 con 0x7f13e8070450 2023-10-16T14:36:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13ecc1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e815e490 con 0x7f13e8159950 2023-10-16T14:36:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13e7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f13e806c810 0x7f13e80c9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40346/0 (socket says 172.21.15.29:40346) 2023-10-16T14:36:37.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13e7fff640 1 -- 172.21.15.29:0/4090637176 learned_addr learned my addr 172.21.15.29:0/4090637176 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13e77fe640 1 --2- 172.21.15.29:0/4090637176 >> v2:172.21.15.170:3300/0 conn(0x7f13e8159950 0x7f13e815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.144+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621587843 0 0) 0x7f13e0001110 con 0x7f13e806c810 2023-10-16T14:36:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0005730 con 0x7f13e806c810 2023-10-16T14:36:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869096349 0 0) 0x7f13d0005730 con 0x7f13e806c810 2023-10-16T14:36:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 >> v2:172.21.15.170:3300/0 conn(0x7f13e8159950 msgr2=0x7f13e815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 --2- 172.21.15.29:0/4090637176 >> v2:172.21.15.170:3300/0 conn(0x7f13e8159950 0x7f13e815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 >> v1:172.21.15.170:6789/0 conn(0x7f13e8070450 legacy=0x7f13e806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e815dbf0 con 0x7f13e806c810 2023-10-16T14:36:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f13e0001800 con 0x7f13e806c810 2023-10-16T14:36:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e77fe640 1 --2- 172.21.15.29:0/4090637176 >> v2:172.21.15.170:3300/0 conn(0x7f13e8159950 0x7f13e815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 >> v1:172.21.15.29:6789/0 conn(0x7f13e806c810 legacy=0x7f13e80c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d0005bd0 0x7f13d0005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d00069d0 0x7f13d0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13d00073e0 0x7f13d000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e815e490 con 0x7f13d0005bd0 2023-10-16T14:36:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d000e330 con 0x7f13d00073e0 2023-10-16T14:36:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d000e4f0 con 0x7f13d00069d0 2023-10-16T14:36:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.145+0000 7f13e77fe640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d0005bd0 0x7f13d0005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e7fff640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13d00073e0 0x7f13d000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e6ffd640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d00069d0 0x7f13d0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e77fe640 1 -- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d00069d0 msgr2=0x7f13d0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e77fe640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d00069d0 0x7f13d0006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e77fe640 1 -- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13d00073e0 msgr2=0x7f13d000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e77fe640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13d00073e0 0x7f13d000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e77fe640 1 -- 172.21.15.29:0/4090637176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0005730 con 0x7f13d0005bd0 2023-10-16T14:36:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e7fff640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13d00073e0 0x7f13d000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.146+0000 7f13e7fff640 1 -- 172.21.15.29:0/4090637176 reap_dead start 2023-10-16T14:36:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.147+0000 7f13e77fe640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d0005bd0 0x7f13d0005ff0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f13d800e320 tx=0x7f13d0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b04412e3cbc987 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.147+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d802cd30 con 0x7f13d0005bd0 2023-10-16T14:36:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.147+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d800f040 con 0x7f13d0005bd0 2023-10-16T14:36:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.147+0000 7f13e67fc640 1 -- 172.21.15.29:0/4090637176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d8034640 con 0x7f13d0005bd0 2023-10-16T14:36:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 -- 172.21.15.29:0/4090637176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d0005bd0 msgr2=0x7f13d0005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d0005bd0 0x7f13d0005ff0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f13d800e320 tx=0x7f13d0007a00 comp rx=0 tx=0).stop 2023-10-16T14:36:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 -- 172.21.15.29:0/4090637176 shutdown_connections 2023-10-16T14:36:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/4090637176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d0005bd0 0x7f13d0005ff0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 -- 172.21.15.29:0/4090637176 >> 172.21.15.29:0/4090637176 conn(0x7f13e80ce6a0 msgr2=0x7f13e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:37.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 -- 172.21.15.29:0/4090637176 shutdown_connections 2023-10-16T14:36:37.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.148+0000 7f13ecc1a640 1 -- 172.21.15.29:0/4090637176 wait complete. 2023-10-16T14:36:37.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.150+0000 7f13ecc1a640 1 Processor -- start 2023-10-16T14:36:37.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.150+0000 7f13ecc1a640 1 -- start start 2023-10-16T14:36:37.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.151+0000 7f13ecc1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d0005bd0 0x7f13e8155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.151+0000 7f13ecc1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e8158020 0x7f13e81584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.151+0000 7f13ecc1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 0x7f13e80af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.151+0000 7f13ecc1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e81f3d30 con 0x7f13e80af1f0 2023-10-16T14:36:37.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.151+0000 7f13ecc1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80afbb0 con 0x7f13e8158020 2023-10-16T14:36:37.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.151+0000 7f13ecc1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e8155ca0 con 0x7f13d0005bd0 2023-10-16T14:36:37.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 0x7f13e80af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 0x7f13e80af670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43922/0 (socket says 172.21.15.29:43922) 2023-10-16T14:36:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 -- 172.21.15.29:0/43767897 learned_addr learned my addr 172.21.15.29:0/43767897 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e77fe640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d0005bd0 0x7f13e8155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e6ffd640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e8158020 0x7f13e81584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 -- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d0005bd0 msgr2=0x7f13e8155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d0005bd0 0x7f13e8155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 -- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e8158020 msgr2=0x7f13e81584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e8158020 0x7f13e81584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.152+0000 7f13e7fff640 1 -- 172.21.15.29:0/43767897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e80ad650 con 0x7f13e80af1f0 2023-10-16T14:36:37.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.153+0000 7f13e7fff640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 0x7f13e80af670 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f13e80b0120 tx=0x7f13e000d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.153+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e0007440 con 0x7f13e80af1f0 2023-10-16T14:36:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.153+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e00075a0 con 0x7f13e80af1f0 2023-10-16T14:36:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.153+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e00049e0 con 0x7f13e80af1f0 2023-10-16T14:36:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.153+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e81f30b0 con 0x7f13e80af1f0 2023-10-16T14:36:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.153+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e81f3670 con 0x7f13e80af1f0 2023-10-16T14:36:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.154+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e804e9b0 con 0x7f13e80af1f0 2023-10-16T14:36:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.155+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f13e0004b40 con 0x7f13e80af1f0 2023-10-16T14:36:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.156+0000 7f13cffff640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f13d406af90 0x7f13d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.156+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f13e0088d70 con 0x7f13e80af1f0 2023-10-16T14:36:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.157+0000 7f13e77fe640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f13d406af90 0x7f13d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.158+0000 7f13e77fe640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f13d406af90 0x7f13d406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f13d8036000 tx=0x7f13d802e000 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:37.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.161+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f13e0053c70 con 0x7f13e80af1f0 2023-10-16T14:36:37.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.377+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 --> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f13e815e1b0 con 0x7f13d406af90 2023-10-16T14:36:37.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.382+0000 7f13cffff640 1 -- 172.21.15.29:0/43767897 <== mgr.6790 v2:172.21.15.170:6832/1905872466 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f13d8035550 con 0x7f13d406af90 2023-10-16T14:36:37.385 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:36:37.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:36:37.385 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:36:37.385 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:36:37.386 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:36:37.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f13d406af90 msgr2=0x7f13d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f13d406af90 0x7f13d406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f13d8036000 tx=0x7f13d802e000 comp rx=0 tx=0).stop 2023-10-16T14:36:37.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 msgr2=0x7f13e80af670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 0x7f13e80af670 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f13e80b0120 tx=0x7f13e000d950 comp rx=0 tx=0).stop 2023-10-16T14:36:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 shutdown_connections 2023-10-16T14:36:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80af1f0 0x7f13e80af670 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f13d406af90 0x7f13d406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13e8158020 0x7f13e81584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 --2- 172.21.15.29:0/43767897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13d0005bd0 0x7f13e8155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 >> 172.21.15.29:0/43767897 conn(0x7f13e80ce6a0 msgr2=0x7f13e806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 shutdown_connections 2023-10-16T14:36:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.384+0000 7f13ecc1a640 1 -- 172.21.15.29:0/43767897 wait complete. 2023-10-16T14:36:37.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:36:37.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.640+0000 7faddd7ed640 1 Processor -- start 2023-10-16T14:36:37.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.640+0000 7faddd7ed640 1 -- start start 2023-10-16T14:36:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.642+0000 7faddd7ed640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fadd80ad970 0x7fadd80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.642+0000 7faddd7ed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd815b9d0 con 0x7fadd806c040 2023-10-16T14:36:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.642+0000 7faddd7ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd815bcb0 con 0x7fadd80ccfb0 2023-10-16T14:36:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.642+0000 7faddd7ed640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd815bf90 con 0x7fadd80ad970 2023-10-16T14:36:37.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.642+0000 7fadd7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fadd806c040 0x7fadd806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40362/0 (socket says 172.21.15.29:40362) 2023-10-16T14:36:37.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.642+0000 7fadd7fff640 1 -- 172.21.15.29:0/984755256 learned_addr learned my addr 172.21.15.29:0/984755256 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:37.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111872461 0 0) 0x7fadd815b9d0 con 0x7fadd806c040 2023-10-16T14:36:37.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0003520 con 0x7fadd806c040 2023-10-16T14:36:37.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd77fe640 1 --2- 172.21.15.29:0/984755256 >> v2:172.21.15.170:3300/0 conn(0x7fadd80ad970 0x7fadd80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643636835 0 0) 0x7fadc0003520 con 0x7fadd806c040 2023-10-16T14:36:37.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 >> v2:172.21.15.170:3300/0 conn(0x7fadd80ad970 msgr2=0x7fadd80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 --2- 172.21.15.29:0/984755256 >> v2:172.21.15.170:3300/0 conn(0x7fadd80ad970 0x7fadd80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 >> v1:172.21.15.170:6789/0 conn(0x7fadd80ccfb0 legacy=0x7fadd80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80cc270 con 0x7fadd806c040 2023-10-16T14:36:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fadc8001670 con 0x7fadd806c040 2023-10-16T14:36:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd77fe640 1 --2- 172.21.15.29:0/984755256 >> v2:172.21.15.170:3300/0 conn(0x7fadd80ad970 0x7fadd80cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 >> v1:172.21.15.29:6789/0 conn(0x7fadd806c040 legacy=0x7fadd806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc00047c0 0x7fadc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc00051d0 0x7fadc000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd815bf90 con 0x7fadc00039c0 2023-10-16T14:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc000c100 con 0x7fadc00051d0 2023-10-16T14:36:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc000c2c0 con 0x7fadc00047c0 2023-10-16T14:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.643+0000 7fadd7fff640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7faddcfec640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc00051d0 0x7fadc000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7fadd77fe640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc00047c0 0x7fadc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7fadd7fff640 1 -- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc00047c0 msgr2=0x7fadc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7fadd7fff640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc00047c0 0x7fadc0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7fadd7fff640 1 -- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc00051d0 msgr2=0x7fadc000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7fadd7fff640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc00051d0 0x7fadc000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7faddcfec640 1 -- 172.21.15.29:0/984755256 reap_dead start 2023-10-16T14:36:37.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.644+0000 7fadd7fff640 1 -- 172.21.15.29:0/984755256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0003520 con 0x7fadc00039c0 2023-10-16T14:36:37.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.645+0000 7fadd7fff640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadc0003dc0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c160 tx=0x7fadc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ab982e7a7a8765 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:37.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.645+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8005ad0 con 0x7fadc00039c0 2023-10-16T14:36:37.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc8005c30 con 0x7fadc00039c0 2023-10-16T14:36:37.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7fadd6ffd640 1 -- 172.21.15.29:0/984755256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc802c7e0 con 0x7fadc00039c0 2023-10-16T14:36:37.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 -- 172.21.15.29:0/984755256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 msgr2=0x7fadc0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadc0003dc0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c160 tx=0x7fadc00057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 -- 172.21.15.29:0/984755256 shutdown_connections 2023-10-16T14:36:37.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 --2- 172.21.15.29:0/984755256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadc0003dc0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 -- 172.21.15.29:0/984755256 >> 172.21.15.29:0/984755256 conn(0x7fadd80ce6a0 msgr2=0x7fadd80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:37.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 -- 172.21.15.29:0/984755256 shutdown_connections 2023-10-16T14:36:37.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.646+0000 7faddd7ed640 1 -- 172.21.15.29:0/984755256 wait complete. 2023-10-16T14:36:37.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.648+0000 7faddd7ed640 1 Processor -- start 2023-10-16T14:36:37.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.648+0000 7faddd7ed640 1 -- start start 2023-10-16T14:36:37.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.649+0000 7faddd7ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadd81f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.649+0000 7faddd7ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80ae7c0 0x7fadd80c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.649+0000 7faddd7ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd8159460 0x7fadd80c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.649+0000 7faddd7ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd815d450 con 0x7fadc00039c0 2023-10-16T14:36:37.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.649+0000 7faddd7ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ae210 con 0x7fadd80ae7c0 2023-10-16T14:36:37.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.649+0000 7faddd7ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ae4f0 con 0x7fadd8159460 2023-10-16T14:36:37.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadd81f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadd81f1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43940/0 (socket says 172.21.15.29:43940) 2023-10-16T14:36:37.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 -- 172.21.15.29:0/1964212930 learned_addr learned my addr 172.21.15.29:0/1964212930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:37.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd77fe640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80ae7c0 0x7fadd80c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7faddcfec640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd8159460 0x7fadd80c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 -- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd8159460 msgr2=0x7fadd80c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd8159460 0x7fadd80c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 -- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80ae7c0 msgr2=0x7fadd80c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80ae7c0 0x7fadd80c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd7fff640 1 -- 172.21.15.29:0/1964212930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd815ce00 con 0x7fadc00039c0 2023-10-16T14:36:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.650+0000 7fadd77fe640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80ae7c0 0x7fadd80c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.651+0000 7fadd7fff640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadd81f1c20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fadc802c380 tx=0x7fadc8001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.651+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc802cbf0 con 0x7fadc00039c0 2023-10-16T14:36:37.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.651+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc802cd50 con 0x7fadc00039c0 2023-10-16T14:36:37.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.651+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc80072b0 con 0x7fadc00039c0 2023-10-16T14:36:37.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.651+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd815d0e0 con 0x7fadc00039c0 2023-10-16T14:36:37.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.651+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd80ade50 con 0x7fadc00039c0 2023-10-16T14:36:37.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.652+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd804e9b0 con 0x7fadc00039c0 2023-10-16T14:36:37.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.653+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fadc8032070 con 0x7fadc00039c0 2023-10-16T14:36:37.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.654+0000 7fadd4ff9640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fadc406afc0 0x7fadc406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:37.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.654+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fadc803c040 con 0x7fadc00039c0 2023-10-16T14:36:37.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.655+0000 7fadd77fe640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fadc406afc0 0x7fadc406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:37.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.656+0000 7fadd77fe640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fadc406afc0 0x7fadc406d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadd81f5910 tx=0x7fadcc008040 comp rx=0 tx=0).ready entity=mgr.6790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:37.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.658+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadc80758a0 con 0x7fadc00039c0 2023-10-16T14:36:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:37.902+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fadd806b2b0 con 0x7fadc00039c0 2023-10-16T14:36:38.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.338+0000 7fadd4ff9640 1 -- 172.21.15.29:0/1964212930 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v68) v1 ==== 92+0+0 (secure 0 0 0) 0x7fadc80450a0 con 0x7fadc00039c0 2023-10-16T14:36:38.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.342+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fadc406afc0 msgr2=0x7fadc406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.342+0000 7faddd7ed640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fadc406afc0 0x7fadc406d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadd81f5910 tx=0x7fadcc008040 comp rx=0 tx=0).stop 2023-10-16T14:36:38.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.342+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 msgr2=0x7fadd81f1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.342+0000 7faddd7ed640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadd81f1c20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fadc802c380 tx=0x7fadc8001930 comp rx=0 tx=0).stop 2023-10-16T14:36:38.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.343+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 shutdown_connections 2023-10-16T14:36:38.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.343+0000 7faddd7ed640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fadc406afc0 0x7fadc406d480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.343+0000 7faddd7ed640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadd8159460 0x7fadd80c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.343+0000 7faddd7ed640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadd80ae7c0 0x7fadd80c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.343+0000 7faddd7ed640 1 --2- 172.21.15.29:0/1964212930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc00039c0 0x7fadd81f1c20 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.343+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 >> 172.21.15.29:0/1964212930 conn(0x7fadd80ce6a0 msgr2=0x7fadd80c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:38.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.344+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 shutdown_connections 2023-10-16T14:36:38.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.344+0000 7faddd7ed640 1 -- 172.21.15.29:0/1964212930 wait complete. 2023-10-16T14:36:38.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:36:38.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.574+0000 7f7f88b16640 1 Processor -- start 2023-10-16T14:36:38.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.574+0000 7f7f88b16640 1 -- start start 2023-10-16T14:36:38.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.575+0000 7f7f88b16640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f8415d030 0x7f7f840d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:38.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.575+0000 7f7f88b16640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7f8415d410 0x7f7f84161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:38.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.575+0000 7f7f88b16640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f840b5d90 con 0x7f7f84161990 2023-10-16T14:36:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f88b16640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f840d2b50 con 0x7f7f8415d030 2023-10-16T14:36:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f88b16640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f84160fc0 con 0x7f7f8415d410 2023-10-16T14:36:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f8259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f8415d030 0x7f7f840d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f8259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f8415d030 0x7f7f840d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43958/0 (socket says 172.21.15.29:43958) 2023-10-16T14:36:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f8259c640 1 -- 172.21.15.29:0/4041629135 learned_addr learned my addr 172.21.15.29:0/4041629135 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:38.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f7bfff640 1 --2- 172.21.15.29:0/4041629135 >> v2:172.21.15.170:3300/0 conn(0x7f7f8415d410 0x7f7f84161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f8259c640 1 -- 172.21.15.29:0/4041629135 >> v2:172.21.15.170:3300/0 conn(0x7f7f8415d410 msgr2=0x7f7f84161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.576+0000 7f7f8259c640 1 --2- 172.21.15.29:0/4041629135 >> v2:172.21.15.170:3300/0 conn(0x7f7f8415d410 0x7f7f84161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.577+0000 7f7f8259c640 1 -- 172.21.15.29:0/4041629135 >> v1:172.21.15.170:6789/0 conn(0x7f7f84161990 legacy=0x7f7f840cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.577+0000 7f7f8259c640 1 -- 172.21.15.29:0/4041629135 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840d25d0 con 0x7f7f8415d030 2023-10-16T14:36:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.578+0000 7f7f8259c640 1 --2- 172.21.15.29:0/4041629135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f8415d030 0x7f7f840d1630 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f7f84160cd0 tx=0x7f7f7402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4d8bed0905789c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.578+0000 7f7f81d9b640 1 -- 172.21.15.29:0/4041629135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7402f850 con 0x7f7f8415d030 2023-10-16T14:36:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.578+0000 7f7f81d9b640 1 -- 172.21.15.29:0/4041629135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7402f9b0 con 0x7f7f8415d030 2023-10-16T14:36:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.578+0000 7f7f81d9b640 1 -- 172.21.15.29:0/4041629135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7402fd40 con 0x7f7f8415d030 2023-10-16T14:36:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 -- 172.21.15.29:0/4041629135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f8415d030 msgr2=0x7f7f840d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 --2- 172.21.15.29:0/4041629135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f8415d030 0x7f7f840d1630 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f7f84160cd0 tx=0x7f7f7402ba30 comp rx=0 tx=0).stop 2023-10-16T14:36:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 -- 172.21.15.29:0/4041629135 shutdown_connections 2023-10-16T14:36:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 --2- 172.21.15.29:0/4041629135 >> v2:172.21.15.170:3300/0 conn(0x7f7f8415d410 0x7f7f84161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 --2- 172.21.15.29:0/4041629135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f8415d030 0x7f7f840d1630 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 -- 172.21.15.29:0/4041629135 >> 172.21.15.29:0/4041629135 conn(0x7f7f840cf720 msgr2=0x7f7f840cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 -- 172.21.15.29:0/4041629135 shutdown_connections 2023-10-16T14:36:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.579+0000 7f7f88b16640 1 -- 172.21.15.29:0/4041629135 wait complete. 2023-10-16T14:36:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.581+0000 7f7f88b16640 1 Processor -- start 2023-10-16T14:36:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.581+0000 7f7f88b16640 1 -- start start 2023-10-16T14:36:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f88b16640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f8415d030 0x7f7f841f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f88b16640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f8415d410 0x7f7f84155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f88b16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 0x7f7f840b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f88b16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f841f3fc0 con 0x7f7f84161990 2023-10-16T14:36:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f88b16640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c9ce0 con 0x7f7f8415d410 2023-10-16T14:36:38.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f88b16640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c9fc0 con 0x7f7f8415d030 2023-10-16T14:36:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.582+0000 7f7f82d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 0x7f7f840b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 0x7f7f840b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43966/0 (socket says 172.21.15.29:43966) 2023-10-16T14:36:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 -- 172.21.15.29:0/3650004659 learned_addr learned my addr 172.21.15.29:0/3650004659 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f7bfff640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f8415d410 0x7f7f84155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f8259c640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f8415d030 0x7f7f841f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 -- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f8415d030 msgr2=0x7f7f841f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f8415d030 0x7f7f841f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 -- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f8415d410 msgr2=0x7f7f84155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f8415d410 0x7f7f84155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f82d9d640 1 -- 172.21.15.29:0/3650004659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f740094d0 con 0x7f7f84161990 2023-10-16T14:36:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.583+0000 7f7f7bfff640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f8415d410 0x7f7f84155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.584+0000 7f7f82d9d640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 0x7f7f840b3e20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00c6b0 tx=0x7f7f7c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.584+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c003b70 con 0x7f7f84161990 2023-10-16T14:36:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.584+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c003cf0 con 0x7f7f84161990 2023-10-16T14:36:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.584+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c003ea0 con 0x7f7f84161990 2023-10-16T14:36:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.584+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f840ca2a0 con 0x7f7f84161990 2023-10-16T14:36:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.584+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f840ca860 con 0x7f7f84161990 2023-10-16T14:36:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.585+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8404e160 con 0x7f7f84161990 2023-10-16T14:36:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.587+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7f7c005310 con 0x7f7f84161990 2023-10-16T14:36:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.588+0000 7f7f7affd640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 0x7f7f6406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.588+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7f7c089510 con 0x7f7f84161990 2023-10-16T14:36:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.589+0000 7f7f8259c640 1 -- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 msgr2=0x7f7f6406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.589+0000 7f7f8259c640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 0x7f7f6406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.593+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f7c0544a0 con 0x7f7f84161990 2023-10-16T14:36:38.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.790+0000 7f7f8259c640 1 -- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 msgr2=0x7f7f6406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:38.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.790+0000 7f7f8259c640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 0x7f7f6406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:38.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.791+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f7f840c9aa0 con 0x7f7f84161990 2023-10-16T14:36:38.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.797+0000 7f7f7affd640 1 -- 172.21.15.29:0/3650004659 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7f7c057ac0 con 0x7f7f84161990 2023-10-16T14:36:38.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 msgr2=0x7f7f6406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:38.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 0x7f7f6406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 msgr2=0x7f7f840b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 0x7f7f840b3e20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00c6b0 tx=0x7f7f7c00cb70 comp rx=0 tx=0).stop 2023-10-16T14:36:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 shutdown_connections 2023-10-16T14:36:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f84161990 0x7f7f840b3e20 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f7f6406afe0 0x7f7f6406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f8415d410 0x7f7f84155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 --2- 172.21.15.29:0/3650004659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f8415d030 0x7f7f841f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 >> 172.21.15.29:0/3650004659 conn(0x7f7f840cf720 msgr2=0x7f7f840b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.800+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 shutdown_connections 2023-10-16T14:36:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:38.801+0000 7f7f88b16640 1 -- 172.21.15.29:0/3650004659 wait complete. 2023-10-16T14:36:38.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:36:38.887 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:38.883+0000 7f08d9630040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:36:38.887 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:38.885+0000 7fbfe263e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:36:38.941 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:38.939+0000 7fbfe263e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:36:38.943 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:38.940+0000 7f08d9630040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:36:39.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.032+0000 7f4b3ccc3640 1 Processor -- start 2023-10-16T14:36:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.033+0000 7f4b3ccc3640 1 -- start start 2023-10-16T14:36:39.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.034+0000 7f4b3ccc3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b3815cd70 0x7f4b380d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.034+0000 7f4b3ccc3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.034+0000 7f4b3ccc3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4b380d0be0 0x7f4b380cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.034+0000 7f4b3ccc3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3815d150 con 0x7f4b3815cd70 2023-10-16T14:36:39.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.034+0000 7f4b3ccc3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b380d1180 con 0x7f4b380d0be0 2023-10-16T14:36:39.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.034+0000 7f4b3ccc3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380d1460 con 0x7f4b380b8480 2023-10-16T14:36:39.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b3815cd70 0x7f4b380d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b3815cd70 0x7f4b380d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43976/0 (socket says 172.21.15.29:43976) 2023-10-16T14:36:39.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 -- 172.21.15.29:0/11823872 learned_addr learned my addr 172.21.15.29:0/11823872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:39.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b37fff640 1 --2- 172.21.15.29:0/11823872 >> v2:172.21.15.170:3300/0 conn(0x7f4b380d0be0 0x7f4b380cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b2ffff640 1 --2- 172.21.15.29:0/11823872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 -- 172.21.15.29:0/11823872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 msgr2=0x7f4b380cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:39.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 --2- 172.21.15.29:0/11823872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 -- 172.21.15.29:0/11823872 >> v2:172.21.15.170:3300/0 conn(0x7f4b380d0be0 msgr2=0x7f4b380cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:39.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 --2- 172.21.15.29:0/11823872 >> v2:172.21.15.170:3300/0 conn(0x7f4b380d0be0 0x7f4b380cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.035+0000 7f4b377fe640 1 -- 172.21.15.29:0/11823872 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b380d1740 con 0x7f4b3815cd70 2023-10-16T14:36:39.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.036+0000 7f4b377fe640 1 --2- 172.21.15.29:0/11823872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380d2220 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4b380b2520 tx=0x7f4b2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1af114e72a2fe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:39.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.037+0000 7f4b36ffd640 1 -- 172.21.15.29:0/11823872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28031870 con 0x7f4b3815cd70 2023-10-16T14:36:39.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.037+0000 7f4b36ffd640 1 -- 172.21.15.29:0/11823872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b280319d0 con 0x7f4b3815cd70 2023-10-16T14:36:39.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.037+0000 7f4b36ffd640 1 -- 172.21.15.29:0/11823872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28031d60 con 0x7f4b3815cd70 2023-10-16T14:36:39.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.037+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/11823872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 msgr2=0x7f4b380d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.037+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/11823872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380d2220 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4b380b2520 tx=0x7f4b2802b910 comp rx=0 tx=0).stop 2023-10-16T14:36:39.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/11823872 shutdown_connections 2023-10-16T14:36:39.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/11823872 >> v2:172.21.15.170:3300/0 conn(0x7f4b380d0be0 0x7f4b380cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/11823872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/11823872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380d2220 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/11823872 >> 172.21.15.29:0/11823872 conn(0x7f4b380cf570 msgr2=0x7f4b380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:39.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/11823872 shutdown_connections 2023-10-16T14:36:39.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.038+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/11823872 wait complete. 2023-10-16T14:36:39.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.040+0000 7f4b3ccc3640 1 Processor -- start 2023-10-16T14:36:39.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.040+0000 7f4b3ccc3640 1 -- start start 2023-10-16T14:36:39.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b3ccc3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b3ccc3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b380d0be0 0x7f4b38151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b3ccc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b3ccc3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380c7140 con 0x7f4b3815cd70 2023-10-16T14:36:39.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b3ccc3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380c7420 con 0x7f4b380d0be0 2023-10-16T14:36:39.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b3ccc3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380c9bc0 con 0x7f4b380b8480 2023-10-16T14:36:39.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43992/0 (socket says 172.21.15.29:43992) 2023-10-16T14:36:39.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b37fff640 1 -- 172.21.15.29:0/3890606727 learned_addr learned my addr 172.21.15.29:0/3890606727 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:39.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b2ffff640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b380d0be0 0x7f4b38151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.041+0000 7f4b377fe640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b37fff640 1 -- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 msgr2=0x7f4b380cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b37fff640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b37fff640 1 -- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b380d0be0 msgr2=0x7f4b38151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b37fff640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b380d0be0 0x7f4b38151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b37fff640 1 -- 172.21.15.29:0/3890606727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280094d0 con 0x7f4b3815cd70 2023-10-16T14:36:39.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b2ffff640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b380d0be0 0x7f4b38151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:39.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b37fff640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380c7e40 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4b30007d90 tx=0x7f4b3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:39.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30014420 con 0x7f4b3815cd70 2023-10-16T14:36:39.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b300145a0 con 0x7f4b3815cd70 2023-10-16T14:36:39.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30014950 con 0x7f4b3815cd70 2023-10-16T14:36:39.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.042+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b380c9ea0 con 0x7f4b3815cd70 2023-10-16T14:36:39.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.043+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b380c6e30 con 0x7f4b3815cd70 2023-10-16T14:36:39.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.044+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b380cc6b0 con 0x7f4b3815cd70 2023-10-16T14:36:39.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.046+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4b30014ab0 con 0x7f4b3815cd70 2023-10-16T14:36:39.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.047+0000 7f4b34ff9640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 0x7f4b1806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.047+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4b30089f70 con 0x7f4b3815cd70 2023-10-16T14:36:39.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.047+0000 7f4b377fe640 1 -- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 msgr2=0x7f4b1806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:39.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.047+0000 7f4b377fe640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 0x7f4b1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:39.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.055+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4b30054f90 con 0x7f4b3815cd70 2023-10-16T14:36:39.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.248+0000 7f4b377fe640 1 -- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 msgr2=0x7f4b1806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:39.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.248+0000 7f4b377fe640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 0x7f4b1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:39.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.251+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f4b380c7280 con 0x7f4b3815cd70 2023-10-16T14:36:39.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.257+0000 7f4b34ff9640 1 -- 172.21.15.29:0/3890606727 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4b380c7280 con 0x7f4b3815cd70 2023-10-16T14:36:39.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 msgr2=0x7f4b1806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:39.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 0x7f4b1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 msgr2=0x7f4b380c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380c7e40 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4b30007d90 tx=0x7f4b3000e4b0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 shutdown_connections 2023-10-16T14:36:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b3815cd70 0x7f4b380c7e40 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7f4b1806afe0 0x7f4b1806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4b380d0be0 0x7f4b38151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 --2- 172.21.15.29:0/3890606727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4b380b8480 0x7f4b380cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.260+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 >> 172.21.15.29:0/3890606727 conn(0x7f4b380cf570 msgr2=0x7f4b380b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.261+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 shutdown_connections 2023-10-16T14:36:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.261+0000 7f4b3ccc3640 1 -- 172.21.15.29:0/3890606727 wait complete. 2023-10-16T14:36:39.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:39.283 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-10-16T14:36:39.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-10-16T14:36:39.387 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.385+0000 7fbfe263e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:36:39.429 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.426+0000 7f08d9630040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:36:39.439 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.437+0000 7fbfe263e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:36:39.485 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.481+0000 7f08d9630040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:36:39.489 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.486+0000 7fbfe263e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:36:39.538 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.534+0000 7f08d9630040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:36:39.548 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.546+0000 7fbfe263e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:36:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.547+0000 7fbea001a640 1 Processor -- start 2023-10-16T14:36:39.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.547+0000 7fbea001a640 1 -- start start 2023-10-16T14:36:39.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.549+0000 7fbea001a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbe980b3420 0x7fbe980b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.549+0000 7fbea001a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.549+0000 7fbea001a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980d3980 con 0x7fbe980cf710 2023-10-16T14:36:39.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.549+0000 7fbea001a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980d3c60 con 0x7fbe980b3420 2023-10-16T14:36:39.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.549+0000 7fbea001a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980ce630 con 0x7fbe980cfaf0 2023-10-16T14:36:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.550+0000 7fbe9e5b7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.550+0000 7fbe9e5b7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:43354/0 (socket says 172.21.15.29:43354) 2023-10-16T14:36:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.550+0000 7fbe9e5b7640 1 -- 172.21.15.29:0/588934906 learned_addr learned my addr 172.21.15.29:0/588934906 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.550+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/588934906 >> v2:172.21.15.170:3300/0 conn(0x7fbe980b3420 0x7fbe980b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.551+0000 7fbe9e5b7640 1 -- 172.21.15.29:0/588934906 >> v2:172.21.15.170:3300/0 conn(0x7fbe980b3420 msgr2=0x7fbe980b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.551+0000 7fbe9e5b7640 1 --2- 172.21.15.29:0/588934906 >> v2:172.21.15.170:3300/0 conn(0x7fbe980b3420 0x7fbe980b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.551+0000 7fbe9e5b7640 1 -- 172.21.15.29:0/588934906 >> v1:172.21.15.170:6789/0 conn(0x7fbe980cf710 legacy=0x7fbe980b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.551+0000 7fbe9e5b7640 1 -- 172.21.15.29:0/588934906 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980ce910 con 0x7fbe980cfaf0 2023-10-16T14:36:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.552+0000 7fbe9e5b7640 1 --2- 172.21.15.29:0/588934906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980b6200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009e50 tx=0x7fbe9002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8e173f62f86281f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbe9cdb4640 1 -- 172.21.15.29:0/588934906 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9002ca70 con 0x7fbe980cfaf0 2023-10-16T14:36:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbe9cdb4640 1 -- 172.21.15.29:0/588934906 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9002cbd0 con 0x7fbe980cfaf0 2023-10-16T14:36:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbe9cdb4640 1 -- 172.21.15.29:0/588934906 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe900343f0 con 0x7fbe980cfaf0 2023-10-16T14:36:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 -- 172.21.15.29:0/588934906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 msgr2=0x7fbe980b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 --2- 172.21.15.29:0/588934906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980b6200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009e50 tx=0x7fbe9002bed0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 -- 172.21.15.29:0/588934906 shutdown_connections 2023-10-16T14:36:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 --2- 172.21.15.29:0/588934906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980b6200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 --2- 172.21.15.29:0/588934906 >> v2:172.21.15.170:3300/0 conn(0x7fbe980b3420 0x7fbe980b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 -- 172.21.15.29:0/588934906 >> 172.21.15.29:0/588934906 conn(0x7fbe980d2cc0 msgr2=0x7fbe980d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.553+0000 7fbea001a640 1 -- 172.21.15.29:0/588934906 shutdown_connections 2023-10-16T14:36:39.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.554+0000 7fbea001a640 1 -- 172.21.15.29:0/588934906 wait complete. 2023-10-16T14:36:39.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.555+0000 7fbea001a640 1 Processor -- start 2023-10-16T14:36:39.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.555+0000 7fbea001a640 1 -- start start 2023-10-16T14:36:39.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.556+0000 7fbea001a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe980b3420 0x7fbe9815e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.556+0000 7fbea001a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe980cf710 0x7fbe980cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.556+0000 7fbea001a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.556+0000 7fbea001a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9809d750 con 0x7fbe980b3420 2023-10-16T14:36:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.556+0000 7fbea001a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9815edc0 con 0x7fbe980cf710 2023-10-16T14:36:39.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.556+0000 7fbea001a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9809cad0 con 0x7fbe980cfaf0 2023-10-16T14:36:39.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe980b3420 0x7fbe9815e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe980b3420 0x7fbe9815e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44000/0 (socket says 172.21.15.29:44000) 2023-10-16T14:36:39.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 -- 172.21.15.29:0/3483229208 learned_addr learned my addr 172.21.15.29:0/3483229208 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:39.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9e5b7640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe980cf710 0x7fbe980cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:39.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 msgr2=0x7fbe980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe980cf710 msgr2=0x7fbe980cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe980cf710 0x7fbe980cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9ddb6640 1 -- 172.21.15.29:0/3483229208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90009b30 con 0x7fbe980b3420 2023-10-16T14:36:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.557+0000 7fbe9e5b7640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe980cfaf0 0x7fbe980cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.558+0000 7fbe9ddb6640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe980b3420 0x7fbe9815e880 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c450 tx=0x7fbe9400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.558+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe940052d0 con 0x7fbe980b3420 2023-10-16T14:36:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.558+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe94005450 con 0x7fbe980b3420 2023-10-16T14:36:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.558+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94003880 con 0x7fbe980b3420 2023-10-16T14:36:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.558+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9809cdb0 con 0x7fbe980b3420 2023-10-16T14:36:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.558+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9809d370 con 0x7fbe980b3420 2023-10-16T14:36:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.560+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9809da70 con 0x7fbe980b3420 2023-10-16T14:36:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.560+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fbe940039e0 con 0x7fbe980b3420 2023-10-16T14:36:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.561+0000 7fbe8e7fc640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 0x7fbe7c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.561+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbe940893d0 con 0x7fbe980b3420 2023-10-16T14:36:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.561+0000 7fbe9d5b5640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 msgr2=0x7fbe7c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.561+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 0x7fbe7c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.565+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe94054340 con 0x7fbe980b3420 2023-10-16T14:36:39.598 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.596+0000 7fbfe263e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:36:39.622 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.618+0000 7f08d9630040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:36:39.674 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.671+0000 7f08d9630040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:36:39.707 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.704+0000 7fbfe263e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:36:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.762+0000 7fbe9d5b5640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 msgr2=0x7fbe7c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:39.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.762+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 0x7fbe7c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:39.768 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.766+0000 7fbfe263e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:39.794 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.791+0000 7f08d9630040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:36:39.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:39.792+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 --> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe980d1d50 con 0x7fbe7c06afe0 2023-10-16T14:36:39.818 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.816+0000 7fbfe263e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:36:39.863 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.860+0000 7f08d9630040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:39.875 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.873+0000 7fbfe263e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:36:39.915 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.911+0000 7f08d9630040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:36:39.925 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:39.922+0000 7fbfe263e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:36:39.974 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:39.970+0000 7f08d9630040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:36:40.027 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.023+0000 7f08d9630040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:36:40.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:40.163+0000 7fbe9d5b5640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 msgr2=0x7fbe7c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:40.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:40.163+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 0x7fbe7c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:36:40.302 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.300+0000 7fbfe263e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:36:40.425 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.421+0000 7f08d9630040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:36:40.461 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.458+0000 7fbfe263e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:36:40.568 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.565+0000 7fbfe263e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:36:40.592 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.588+0000 7f08d9630040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:36:40.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.615+0000 7fbfe263e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:36:40.705 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.701+0000 7f08d9630040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:36:40.723 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.721+0000 7fbfe263e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:40.757 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.754+0000 7f08d9630040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:36:40.830 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.828+0000 7fbfe263e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:36:40.871 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.867+0000 7f08d9630040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:40.964+0000 7fbe9d5b5640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 msgr2=0x7fbe7c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1905872466 2023-10-16T14:36:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:40.964+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 0x7fbe7c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:36:40.982 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:40.979+0000 7f08d9630040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:36:40.986 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:40.983+0000 7fbfe263e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:41.035 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:41.032+0000 7fbfe263e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:41.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:41.143+0000 7f08d9630040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:41.153 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:41.150+0000 7fbfe263e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:36:41.199 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:41.195+0000 7f08d9630040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:41.323 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:41.319+0000 7f08d9630040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:36:41.954 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:41.951+0000 7fbfe263e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:36:42.205 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:42.201+0000 7f08d9630040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:36:42.245 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.242+0000 7fbfe263e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:36:42.302 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.299+0000 7fbfe263e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:36:42.414 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.411+0000 7fbfe263e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:36:42.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:42.434+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 69) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fbe94054710 con 0x7fbe980b3420 2023-10-16T14:36:42.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:42.435+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 msgr2=0x7fbe7c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:42.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:42.435+0000 7fbe8e7fc640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/1905872466,v1:172.21.15.170:6833/1905872466] conn(0x7fbe7c06afe0 0x7fbe7c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:42.527 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:42.523+0000 7f08d9630040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:36:42.575 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.571+0000 7fbf5f120640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.575 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.571+0000 7fbf5f120640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.576 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.571+0000 7fbf5f120640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.576 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.571+0000 7fbf5f120640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.576 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.571+0000 7fbf5f120640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.576 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.572+0000 7fbf61164640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.576 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.572+0000 7fbf61164640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.577 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.572+0000 7fbf61164640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.577 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.572+0000 7fbf61164640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.577 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:42.572+0000 7fbf61164640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:42.591 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:42.587+0000 7f08d9630040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:36:42.710 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:42.706+0000 7f08d9630040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:36:43.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.445+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 70) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fbe94076820 con 0x7fbe980b3420 2023-10-16T14:36:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.446+0000 7fbe8e7fc640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fbe7c0355a0 0x7fbe7c035980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.446+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe7c035ec0 con 0x7fbe7c0355a0 2023-10-16T14:36:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.451+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fbe7c0355a0 0x7fbe7c035980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.461+0000 7fbe9d5b5640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fbe7c0355a0 0x7fbe7c035980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe980cc340 tx=0x7fbe88008040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:43.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.601+0000 7fbe8e7fc640 1 -- 172.21.15.29:0/3483229208 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fbe7c035ec0 con 0x7fbe7c0355a0 2023-10-16T14:36:43.605 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/minutes 2023-10-16T14:36:43.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fbe7c0355a0 msgr2=0x7fbe7c035980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:43.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fbe7c0355a0 0x7fbe7c035980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe980cc340 tx=0x7fbe88008040 comp rx=0 tx=0).stop 2023-10-16T14:36:43.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe980b3420 msgr2=0x7fbe9815e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:43.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 --2- 172.21.15.29:0/3483229208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe980b3420 0x7fbe9815e880 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c450 tx=0x7fbe9400c910 comp rx=0 tx=0).stop 2023-10-16T14:36:43.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbe9e5b7640 1 -- 172.21.15.29:0/3483229208 reap_dead start 2023-10-16T14:36:43.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 shutdown_connections 2023-10-16T14:36:43.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 >> 172.21.15.29:0/3483229208 conn(0x7fbe980d2cc0 msgr2=0x7fbe980b5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:43.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 shutdown_connections 2023-10-16T14:36:43.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.604+0000 7fbea001a640 1 -- 172.21.15.29:0/3483229208 wait complete. 2023-10-16T14:36:43.632 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-10-16T14:36:43.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:43.633 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-10-16T14:36:43.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-10-16T14:36:43.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.921+0000 7f3034d90640 1 Processor -- start 2023-10-16T14:36:43.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.921+0000 7f3034d90640 1 -- start start 2023-10-16T14:36:43.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f3034d90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f3030069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f3034d90640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f30300caac0 0x7f303006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f3034d90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300cb030 con 0x7f30300c9ca0 2023-10-16T14:36:43.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f3034d90640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30300c91c0 con 0x7f30300caac0 2023-10-16T14:36:43.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f3034d90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c94a0 con 0x7f303015ad70 2023-10-16T14:36:43.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f302dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30300c9ca0 0x7f30300ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49032/0 (socket says 172.21.15.29:49032) 2023-10-16T14:36:43.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f302dd9b640 1 -- 172.21.15.29:0/1532339486 learned_addr learned my addr 172.21.15.29:0/1532339486 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:43.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f302e59c640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f3030069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.923+0000 7f302ed9d640 1 --2- 172.21.15.29:0/1532339486 >> v2:172.21.15.170:3300/0 conn(0x7f30300caac0 0x7f303006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014005317 0 0) 0x7f30300cb030 con 0x7f30300c9ca0 2023-10-16T14:36:43.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c003520 con 0x7f30300c9ca0 2023-10-16T14:36:43.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145658352 0 0) 0x7f301c003520 con 0x7f30300c9ca0 2023-10-16T14:36:43.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 msgr2=0x7f3030069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:43.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f3030069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 >> v2:172.21.15.170:3300/0 conn(0x7f30300caac0 msgr2=0x7f303006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:43.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 --2- 172.21.15.29:0/1532339486 >> v2:172.21.15.170:3300/0 conn(0x7f30300caac0 0x7f303006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300c9780 con 0x7f30300c9ca0 2023-10-16T14:36:43.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3018001670 con 0x7f30300c9ca0 2023-10-16T14:36:43.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 >> v1:172.21.15.29:6789/0 conn(0x7f30300c9ca0 legacy=0x7f30300ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:43.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f301c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301c003f40 0x7f301c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f301c004d00 0x7f301c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c91c0 con 0x7f301c003f40 2023-10-16T14:36:43.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c94a0 con 0x7f301c004d00 2023-10-16T14:36:43.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c00b980 con 0x7f303015ad70 2023-10-16T14:36:43.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.924+0000 7f302ed9d640 1 --2- 172.21.15.29:0/1532339486 >> v2:172.21.15.170:3300/0 conn(0x7f30300caac0 0x7f303006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:36:43.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302dd9b640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301c003f40 0x7f301c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302e59c640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f301c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302dd9b640 1 -- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 msgr2=0x7f301c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:43.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302dd9b640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f301c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302dd9b640 1 -- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f301c004d00 msgr2=0x7f301c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:43.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302dd9b640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f301c004d00 0x7f301c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302dd9b640 1 -- 172.21.15.29:0/1532339486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c003520 con 0x7f301c003f40 2023-10-16T14:36:43.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.925+0000 7f302e59c640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f303015ad70 0x7f301c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.926+0000 7f302dd9b640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301c003f40 0x7f301c0047c0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f301800c140 tx=0x7f301c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74f965036f07496 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:43.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.926+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018005ad0 con 0x7f301c003f40 2023-10-16T14:36:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3018005c50 con 0x7f301c003f40 2023-10-16T14:36:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f302d59a640 1 -- 172.21.15.29:0/1532339486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301802c830 con 0x7f301c003f40 2023-10-16T14:36:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f3034d90640 1 -- 172.21.15.29:0/1532339486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301c003f40 msgr2=0x7f301c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:43.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f3034d90640 1 --2- 172.21.15.29:0/1532339486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f301c003f40 0x7f301c0047c0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f301800c140 tx=0x7f301c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f302ed9d640 1 -- 172.21.15.29:0/1532339486 reap_dead start 2023-10-16T14:36:43.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f3034d90640 1 -- 172.21.15.29:0/1532339486 shutdown_connections 2023-10-16T14:36:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f3034d90640 1 -- 172.21.15.29:0/1532339486 >> 172.21.15.29:0/1532339486 conn(0x7f30300ce9b0 msgr2=0x7f30300cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:43.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f3034d90640 1 -- 172.21.15.29:0/1532339486 shutdown_connections 2023-10-16T14:36:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.927+0000 7f3034d90640 1 -- 172.21.15.29:0/1532339486 wait complete. 2023-10-16T14:36:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.929+0000 7f3034d90640 1 Processor -- start 2023-10-16T14:36:43.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.929+0000 7f3034d90640 1 -- start start 2023-10-16T14:36:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f3034d90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f301c003f40 0x7f30300c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f3034d90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 0x7f30300c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f3034d90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30300a8480 0x7f30300c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f3034d90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300ad6c0 con 0x7f30301f0170 2023-10-16T14:36:43.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f3034d90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300a9570 con 0x7f30300a8480 2023-10-16T14:36:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f3034d90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300aedc0 con 0x7f301c003f40 2023-10-16T14:36:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.930+0000 7f302dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 0x7f30300c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 0x7f30300c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58090/0 (socket says 172.21.15.29:58090) 2023-10-16T14:36:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 -- 172.21.15.29:0/2739322168 learned_addr learned my addr 172.21.15.29:0/2739322168 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302e59c640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f301c003f40 0x7f30300c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302ed9d640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30300a8480 0x7f30300c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 -- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f301c003f40 msgr2=0x7f30300c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:43.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f301c003f40 0x7f30300c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 -- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30300a8480 msgr2=0x7f30300c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30300a8480 0x7f30300c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 -- 172.21.15.29:0/2739322168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300a89f0 con 0x7f30301f0170 2023-10-16T14:36:43.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.931+0000 7f302dd9b640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 0x7f30300c7c50 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f301802c3d0 tx=0x7f3018001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:43.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.932+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301803c040 con 0x7f30301f0170 2023-10-16T14:36:43.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.932+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301802dcd0 con 0x7f30301f0170 2023-10-16T14:36:43.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.932+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180072b0 con 0x7f30301f0170 2023-10-16T14:36:43.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.932+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30300a8cd0 con 0x7f30301f0170 2023-10-16T14:36:43.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.932+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30300a9290 con 0x7f30301f0170 2023-10-16T14:36:43.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.933+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303004e160 con 0x7f30301f0170 2023-10-16T14:36:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.934+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3018032070 con 0x7f30301f0170 2023-10-16T14:36:43.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.936+0000 7f3026ffd640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f301006ad60 0x7f301006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.936+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f30180aaad0 con 0x7f30301f0170 2023-10-16T14:36:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.936+0000 7f302e59c640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f301006ad60 0x7f301006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.937+0000 7f302e59c640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f301006ad60 0x7f301006d220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3020003be0 tx=0x7f302000b040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:43.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:43.941+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3018075cf0 con 0x7f30301f0170 2023-10-16T14:36:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.149+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f3030069da0 con 0x7f301006ad60 2023-10-16T14:36:44.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.296+0000 7f3026ffd640 1 -- 172.21.15.29:0/2739322168 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f3020009bd0 con 0x7f301006ad60 2023-10-16T14:36:44.300 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/hourly 2023-10-16T14:36:44.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f301006ad60 msgr2=0x7f301006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f301006ad60 0x7f301006d220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3020003be0 tx=0x7f302000b040 comp rx=0 tx=0).stop 2023-10-16T14:36:44.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 msgr2=0x7f30300c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 0x7f30300c7c50 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f301802c3d0 tx=0x7f3018001930 comp rx=0 tx=0).stop 2023-10-16T14:36:44.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 shutdown_connections 2023-10-16T14:36:44.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f301006ad60 0x7f301006d220 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30301f0170 0x7f30300c7c50 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30300a8480 0x7f30300c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 --2- 172.21.15.29:0/2739322168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f301c003f40 0x7f30300c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 >> 172.21.15.29:0/2739322168 conn(0x7f30300ce9b0 msgr2=0x7f30300b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:44.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 shutdown_connections 2023-10-16T14:36:44.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.298+0000 7f3034d90640 1 -- 172.21.15.29:0/2739322168 wait complete. 2023-10-16T14:36:44.320 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-10-16T14:36:44.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:44.321 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-10-16T14:36:44.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-10-16T14:36:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.612+0000 7f75affff640 1 Processor -- start 2023-10-16T14:36:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.612+0000 7f75affff640 1 -- start start 2023-10-16T14:36:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.614+0000 7f75affff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f75b00ac890 0x7f75b00accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.614+0000 7f75affff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b0160a90 con 0x7f75b015dde0 2023-10-16T14:36:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.614+0000 7f75affff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b00ad200 con 0x7f75b015c9f0 2023-10-16T14:36:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.614+0000 7f75affff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b00cdec0 con 0x7f75b00ac890 2023-10-16T14:36:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.614+0000 7f75ae7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f75b015dde0 0x7f75b00709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49044/0 (socket says 172.21.15.29:49044) 2023-10-16T14:36:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.614+0000 7f75ae7fc640 1 -- 172.21.15.29:0/2712310093 learned_addr learned my addr 172.21.15.29:0/2712310093 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75af7fe640 1 --2- 172.21.15.29:0/2712310093 >> v2:172.21.15.170:3300/0 conn(0x7f75b00ac890 0x7f75b00accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368725723 0 0) 0x7f75b0160a90 con 0x7f75b015dde0 2023-10-16T14:36:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c003520 con 0x7f75b015dde0 2023-10-16T14:36:44.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072075799 0 0) 0x7f759c003520 con 0x7f75b015dde0 2023-10-16T14:36:44.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 >> v2:172.21.15.170:3300/0 conn(0x7f75b00ac890 msgr2=0x7f75b00accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:44.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 --2- 172.21.15.29:0/2712310093 >> v2:172.21.15.170:3300/0 conn(0x7f75b00ac890 0x7f75b00accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 >> v1:172.21.15.170:6789/0 conn(0x7f75b015c9f0 legacy=0x7f75b00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00cd900 con 0x7f75b015dde0 2023-10-16T14:36:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7598001670 con 0x7f75b015dde0 2023-10-16T14:36:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.615+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 >> v1:172.21.15.29:6789/0 conn(0x7f75b015dde0 legacy=0x7f75b00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75adffb640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c003970 0x7f759c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75adffb640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75adffb640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00c050 con 0x7f759c0047c0 2023-10-16T14:36:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00c190 con 0x7f759c003970 2023-10-16T14:36:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00c350 con 0x7f759c005180 2023-10-16T14:36:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75ae7fc640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75aeffd640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c003970 0x7f759c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75af7fe640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75ae7fc640 1 -- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759c005180 msgr2=0x7f759c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75ae7fc640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759c005180 0x7f759c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75ae7fc640 1 -- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c003970 msgr2=0x7f759c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.616+0000 7f75ae7fc640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c003970 0x7f759c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.617+0000 7f75ae7fc640 1 -- 172.21.15.29:0/2712310093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c003520 con 0x7f759c0047c0 2023-10-16T14:36:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.617+0000 7f75af7fe640 1 -- 172.21.15.29:0/2712310093 reap_dead start 2023-10-16T14:36:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.617+0000 7f75aeffd640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c003970 0x7f759c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.618+0000 7f75ae7fc640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759c0047c0 0x7f759c004c40 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f759800c240 tx=0x7f759c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a498694311ee6c64 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.618+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598005ad0 con 0x7f759c0047c0 2023-10-16T14:36:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.618+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598005c30 con 0x7f759c0047c0 2023-10-16T14:36:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.618+0000 7f75adffb640 1 -- 172.21.15.29:0/2712310093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759802c7c0 con 0x7f759c0047c0 2023-10-16T14:36:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.618+0000 7f75affff640 1 -- 172.21.15.29:0/2712310093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759c0047c0 msgr2=0x7f759c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.618+0000 7f75affff640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759c0047c0 0x7f759c004c40 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f759800c240 tx=0x7f759c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.619+0000 7f75affff640 1 -- 172.21.15.29:0/2712310093 shutdown_connections 2023-10-16T14:36:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.619+0000 7f75affff640 1 --2- 172.21.15.29:0/2712310093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759c0047c0 0x7f759c004c40 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.619+0000 7f75affff640 1 -- 172.21.15.29:0/2712310093 >> 172.21.15.29:0/2712310093 conn(0x7f75b00ce6a0 msgr2=0x7f75b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.619+0000 7f75affff640 1 -- 172.21.15.29:0/2712310093 shutdown_connections 2023-10-16T14:36:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.619+0000 7f75affff640 1 -- 172.21.15.29:0/2712310093 wait complete. 2023-10-16T14:36:44.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.621+0000 7f75affff640 1 Processor -- start 2023-10-16T14:36:44.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.621+0000 7f75affff640 1 -- start start 2023-10-16T14:36:44.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75affff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c0047c0 0x7f75b006c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75affff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0158e90 0x7f75b006adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 0x7f75b015a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75affff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b006c3e0 con 0x7f75b015a1d0 2023-10-16T14:36:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75affff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00c79f0 con 0x7f759c0047c0 2023-10-16T14:36:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75affff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b015ab90 con 0x7f75b0158e90 2023-10-16T14:36:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75af7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 0x7f75b015a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75af7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 0x7f75b015a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58112/0 (socket says 172.21.15.29:58112) 2023-10-16T14:36:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.622+0000 7f75af7fe640 1 -- 172.21.15.29:0/2486898423 learned_addr learned my addr 172.21.15.29:0/2486898423 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75ae7fc640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0158e90 0x7f75b006adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75aeffd640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c0047c0 0x7f75b006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75af7fe640 1 -- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0158e90 msgr2=0x7f75b006adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75af7fe640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0158e90 0x7f75b006adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75af7fe640 1 -- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c0047c0 msgr2=0x7f75b006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:44.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75af7fe640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c0047c0 0x7f75b006c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:44.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75af7fe640 1 -- 172.21.15.29:0/2486898423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b006b760 con 0x7f75b015a1d0 2023-10-16T14:36:44.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75af7fe640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 0x7f75b015a650 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f75b006d350 tx=0x7f75a800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:44.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.623+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a8016420 con 0x7f75b015a1d0 2023-10-16T14:36:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.624+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a80165a0 con 0x7f75b015a1d0 2023-10-16T14:36:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.624+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a8016950 con 0x7f75b015a1d0 2023-10-16T14:36:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.624+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b006ba40 con 0x7f75b015a1d0 2023-10-16T14:36:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.624+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b006c000 con 0x7f75b015a1d0 2023-10-16T14:36:44.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.625+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b006c520 con 0x7f75b015a1d0 2023-10-16T14:36:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.626+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f75a8016ab0 con 0x7f75b015a1d0 2023-10-16T14:36:44.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.627+0000 7f75a77fe640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f759006b010 0x7f759006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.628+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f75a808aa20 con 0x7f75b015a1d0 2023-10-16T14:36:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.628+0000 7f75aeffd640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f759006b010 0x7f759006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.629+0000 7f75aeffd640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f759006b010 0x7f759006d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007fd0 tx=0x7f75a0009040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.632+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75b006c520 con 0x7f75b015a1d0 2023-10-16T14:36:44.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:44.849+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f75b006c1c0 con 0x7f759006b010 2023-10-16T14:36:45.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.006+0000 7f75a77fe640 1 -- 172.21.15.29:0/2486898423 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f75b006c1c0 con 0x7f759006b010 2023-10-16T14:36:45.010 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/daily 2023-10-16T14:36:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.009+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f759006b010 msgr2=0x7f759006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.009+0000 7f75affff640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f759006b010 0x7f759006d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007fd0 tx=0x7f75a0009040 comp rx=0 tx=0).stop 2023-10-16T14:36:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.009+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 msgr2=0x7f75b015a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.009+0000 7f75affff640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 0x7f75b015a650 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f75b006d350 tx=0x7f75a800f520 comp rx=0 tx=0).stop 2023-10-16T14:36:45.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 shutdown_connections 2023-10-16T14:36:45.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f759006b010 0x7f759006d4d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75b015a1d0 0x7f75b015a650 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f75b0158e90 0x7f75b006adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 --2- 172.21.15.29:0/2486898423 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f759c0047c0 0x7f75b006c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 >> 172.21.15.29:0/2486898423 conn(0x7f75b00ce6a0 msgr2=0x7f75b00ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:45.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 shutdown_connections 2023-10-16T14:36:45.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.010+0000 7f75affff640 1 -- 172.21.15.29:0/2486898423 wait complete. 2023-10-16T14:36:45.035 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-10-16T14:36:45.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:45.036 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-10-16T14:36:45.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-10-16T14:36:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.321+0000 7fcdcba9b640 1 Processor -- start 2023-10-16T14:36:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.322+0000 7fcdcba9b640 1 -- start start 2023-10-16T14:36:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdcba9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcdc415ad70 0x7fcdc4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdcba9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdcba9b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc40cb030 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdcba9b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc40c91c0 con 0x7fcdc415ad70 2023-10-16T14:36:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdcba9b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc40c94a0 con 0x7fcdc40c9ca0 2023-10-16T14:36:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcdc415ad70 0x7fcdc4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcdc415ad70 0x7fcdc4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58120/0 (socket says 172.21.15.29:58120) 2023-10-16T14:36:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 -- 172.21.15.29:0/3620090197 learned_addr learned my addr 172.21.15.29:0/3620090197 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9036640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 -- 172.21.15.29:0/3620090197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 msgr2=0x7fcdc406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 -- 172.21.15.29:0/3620090197 >> v1:172.21.15.170:6789/0 conn(0x7fcdc40ca3b0 legacy=0x7fcdc40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9837640 1 -- 172.21.15.29:0/3620090197 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc40c9780 con 0x7fcdc415ad70 2023-10-16T14:36:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.324+0000 7fcdc9036640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.325+0000 7fcdc9837640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc415ad70 0x7fcdc4069da0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009e50 tx=0x7fcdc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cc7c25b8a4c81b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.326+0000 7fcdc8835640 1 -- 172.21.15.29:0/3620090197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc002ca70 con 0x7fcdc415ad70 2023-10-16T14:36:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.326+0000 7fcdc8835640 1 -- 172.21.15.29:0/3620090197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc002cbd0 con 0x7fcdc415ad70 2023-10-16T14:36:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.326+0000 7fcdc8835640 1 -- 172.21.15.29:0/3620090197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00343f0 con 0x7fcdc415ad70 2023-10-16T14:36:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.326+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3620090197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc415ad70 msgr2=0x7fcdc4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.326+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc415ad70 0x7fcdc4069da0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009e50 tx=0x7fcdc002bed0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.327+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3620090197 shutdown_connections 2023-10-16T14:36:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.327+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.327+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3620090197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc415ad70 0x7fcdc4069da0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.327+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3620090197 >> 172.21.15.29:0/3620090197 conn(0x7fcdc40ce9b0 msgr2=0x7fcdc40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.327+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3620090197 shutdown_connections 2023-10-16T14:36:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.327+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3620090197 wait complete. 2023-10-16T14:36:45.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 Processor -- start 2023-10-16T14:36:45.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 -- start start 2023-10-16T14:36:45.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:45.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 0x7fcdc4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:45.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc415ad70 0x7fcdc406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:45.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4157570 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4157850 con 0x7fcdc415ad70 2023-10-16T14:36:45.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdcba9b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4157b30 con 0x7fcdc40c9ca0 2023-10-16T14:36:45.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdc9036640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 0x7fcdc4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:45.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdc9036640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 0x7fcdc4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58132/0 (socket says 172.21.15.29:58132) 2023-10-16T14:36:45.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.330+0000 7fcdc9036640 1 -- 172.21.15.29:0/3558047596 learned_addr learned my addr 172.21.15.29:0/3558047596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:45.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdca038640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc415ad70 0x7fcdc406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:45.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9837640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:45.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9036640 1 -- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 msgr2=0x7fcdc414bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9036640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc414bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9036640 1 -- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc415ad70 msgr2=0x7fcdc406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9036640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc415ad70 0x7fcdc406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9036640 1 -- 172.21.15.29:0/3558047596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc0009b30 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdca038640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc415ad70 0x7fcdc406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:45.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdc9036640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 0x7fcdc4069da0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c7f0 tx=0x7fcdb400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:45.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4007a30 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb4007b90 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb400ce40 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.331+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc4157e10 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.332+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc4158330 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.333+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc404e160 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.334+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fcdb40113f0 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.336+0000 7fcdba7fc640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fcda806afe0 0x7fcda806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:45.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.336+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fcdb4089c70 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.336+0000 7fcdc9837640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fcda806afe0 0x7fcda806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:45.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.337+0000 7fcdc9837640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fcda806afe0 0x7fcda806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0035000 tx=0x7fcdc0030000 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:45.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.340+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcdb40548e0 con 0x7fcdc40ca3b0 2023-10-16T14:36:45.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.556+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc41579c0 con 0x7fcda806afe0 2023-10-16T14:36:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.717+0000 7fcdba7fc640 1 -- 172.21.15.29:0/3558047596 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fcdc41579c0 con 0x7fcda806afe0 2023-10-16T14:36:45.721 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/weekly 2023-10-16T14:36:45.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fcda806afe0 msgr2=0x7fcda806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fcda806afe0 0x7fcda806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0035000 tx=0x7fcdc0030000 comp rx=0 tx=0).stop 2023-10-16T14:36:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 msgr2=0x7fcdc4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:45.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 0x7fcdc4069da0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c7f0 tx=0x7fcdb400cbf0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 shutdown_connections 2023-10-16T14:36:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fcda806afe0 0x7fcda806d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdc40ca3b0 0x7fcdc4069da0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdc415ad70 0x7fcdc406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 --2- 172.21.15.29:0/3558047596 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcdc40c9ca0 0x7fcdc414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 >> 172.21.15.29:0/3558047596 conn(0x7fcdc40ce9b0 msgr2=0x7fcdc40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 shutdown_connections 2023-10-16T14:36:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:45.721+0000 7fcdcba9b640 1 -- 172.21.15.29:0/3558047596 wait complete. 2023-10-16T14:36:45.746 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-10-16T14:36:45.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:45.747 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-10-16T14:36:45.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-10-16T14:36:46.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.034+0000 7f2855d6e640 1 Processor -- start 2023-10-16T14:36:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.034+0000 7f2855d6e640 1 -- start start 2023-10-16T14:36:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.035+0000 7f2855d6e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f28500b0ea0 0x7f285015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.035+0000 7f2855d6e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 0x7f285015a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.035+0000 7f2855d6e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285006be30 con 0x7f28500b3ec0 2023-10-16T14:36:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.035+0000 7f2855d6e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285006ca00 con 0x7f28500b0ea0 2023-10-16T14:36:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.035+0000 7f2855d6e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500ada10 con 0x7f28500cd970 2023-10-16T14:36:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.036+0000 7f2854d6c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f28500b3ec0 0x7f28500cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49046/0 (socket says 172.21.15.29:49046) 2023-10-16T14:36:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.036+0000 7f2854d6c640 1 -- 172.21.15.29:0/3235680336 learned_addr learned my addr 172.21.15.29:0/3235680336 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.036+0000 7f284ffff640 1 --2- 172.21.15.29:0/3235680336 >> v2:172.21.15.170:3300/0 conn(0x7f28500b0ea0 0x7f285015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.036+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000999793 0 0) 0x7f285006be30 con 0x7f28500b3ec0 2023-10-16T14:36:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.036+0000 7f285556d640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 0x7f285015a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.036+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2840003520 con 0x7f28500b3ec0 2023-10-16T14:36:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083649895 0 0) 0x7f2840003520 con 0x7f28500b3ec0 2023-10-16T14:36:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 msgr2=0x7f285015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 0x7f285015a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 >> v2:172.21.15.170:3300/0 conn(0x7f28500b0ea0 msgr2=0x7f285015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 --2- 172.21.15.29:0/3235680336 >> v2:172.21.15.170:3300/0 conn(0x7f28500b0ea0 0x7f285015a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285006c060 con 0x7f28500b3ec0 2023-10-16T14:36:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2844001670 con 0x7f28500b3ec0 2023-10-16T14:36:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 >> v1:172.21.15.29:6789/0 conn(0x7f28500b3ec0 legacy=0x7f28500cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2840003970 0x7f2840003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28400047c0 0x7f2840004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 0x7f2840005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285006ca00 con 0x7f28400047c0 2023-10-16T14:36:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500ada10 con 0x7f2840003970 2023-10-16T14:36:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284000ff20 con 0x7f28500cd970 2023-10-16T14:36:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284ffff640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28400047c0 0x7f2840004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f2854d6c640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2840003970 0x7f2840003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f284ffff640 1 -- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 msgr2=0x7f2840005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.037+0000 7f285556d640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 0x7f2840005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.038+0000 7f284ffff640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28500cd970 0x7f2840005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.038+0000 7f284ffff640 1 -- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2840003970 msgr2=0x7f2840003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.038+0000 7f284ffff640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2840003970 0x7f2840003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.038+0000 7f284ffff640 1 -- 172.21.15.29:0/3235680336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2840003520 con 0x7f28400047c0 2023-10-16T14:36:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.038+0000 7f2854d6c640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2840003970 0x7f2840003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.039+0000 7f284ffff640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28400047c0 0x7f2840004c40 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f283c00ba00 tx=0x7f28400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cb6fc165010cc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.039+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c032620 con 0x7f28400047c0 2023-10-16T14:36:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.039+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c032780 con 0x7f28400047c0 2023-10-16T14:36:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.039+0000 7f284f7fe640 1 -- 172.21.15.29:0/3235680336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c032b10 con 0x7f28400047c0 2023-10-16T14:36:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f2855d6e640 1 -- 172.21.15.29:0/3235680336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28400047c0 msgr2=0x7f2840004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f2855d6e640 1 --2- 172.21.15.29:0/3235680336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28400047c0 0x7f2840004c40 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f283c00ba00 tx=0x7f28400057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f285556d640 1 -- 172.21.15.29:0/3235680336 reap_dead start 2023-10-16T14:36:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f2855d6e640 1 -- 172.21.15.29:0/3235680336 shutdown_connections 2023-10-16T14:36:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f2855d6e640 1 -- 172.21.15.29:0/3235680336 >> 172.21.15.29:0/3235680336 conn(0x7f28500b0170 msgr2=0x7f28500b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f2855d6e640 1 -- 172.21.15.29:0/3235680336 shutdown_connections 2023-10-16T14:36:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.040+0000 7f2855d6e640 1 -- 172.21.15.29:0/3235680336 wait complete. 2023-10-16T14:36:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.042+0000 7f2855d6e640 1 Processor -- start 2023-10-16T14:36:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.042+0000 7f2855d6e640 1 -- start start 2023-10-16T14:36:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f2855d6e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28400047c0 0x7f28500c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f2855d6e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f285015abc0 0x7f28500c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f2855d6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 0x7f28500c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f2855d6e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500c5aa0 con 0x7f28500c8cb0 2023-10-16T14:36:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f2855d6e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500c9670 con 0x7f28400047c0 2023-10-16T14:36:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f2855d6e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500c4300 con 0x7f285015abc0 2023-10-16T14:36:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f285556d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 0x7f28500c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f285556d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 0x7f28500c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58152/0 (socket says 172.21.15.29:58152) 2023-10-16T14:36:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.043+0000 7f285556d640 1 -- 172.21.15.29:0/1007886106 learned_addr learned my addr 172.21.15.29:0/1007886106 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f2854d6c640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28400047c0 0x7f28500c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f284ffff640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f285015abc0 0x7f28500c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f285556d640 1 -- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f285015abc0 msgr2=0x7f28500c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f285556d640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f285015abc0 0x7f28500c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f285556d640 1 -- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28400047c0 msgr2=0x7f28500c3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f285556d640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28400047c0 0x7f28500c3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f285556d640 1 -- 172.21.15.29:0/1007886106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500c5750 con 0x7f28500c8cb0 2023-10-16T14:36:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f285556d640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 0x7f28500c9130 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f28500cb0d0 tx=0x7f284800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.044+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284800fe30 con 0x7f28500c8cb0 2023-10-16T14:36:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.045+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28480143f0 con 0x7f28500c8cb0 2023-10-16T14:36:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.045+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28480147a0 con 0x7f28500c8cb0 2023-10-16T14:36:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.045+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28500acae0 con 0x7f28500c8cb0 2023-10-16T14:36:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.045+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28500ad0a0 con 0x7f28500c8cb0 2023-10-16T14:36:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.046+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28500ad2c0 con 0x7f28500c8cb0 2023-10-16T14:36:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.047+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 71) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f2848014900 con 0x7f28500c8cb0 2023-10-16T14:36:46.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.048+0000 7f284d7fa640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f283406b010 0x7f283406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.049+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f284808a1c0 con 0x7f28500c8cb0 2023-10-16T14:36:46.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.049+0000 7f2854d6c640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f283406b010 0x7f283406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.050+0000 7f2854d6c640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f283406b010 0x7f283406d4d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2844001200 tx=0x7f2844009040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:46.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.053+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2848054ec0 con 0x7f28500c8cb0 2023-10-16T14:36:46.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.268+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f285006c620 con 0x7f283406b010 2023-10-16T14:36:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.423+0000 7f284d7fa640 1 -- 172.21.15.29:0/1007886106 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f284400d050 con 0x7f283406b010 2023-10-16T14:36:46.426 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/monthly 2023-10-16T14:36:46.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f283406b010 msgr2=0x7f283406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f283406b010 0x7f283406d4d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2844001200 tx=0x7f2844009040 comp rx=0 tx=0).stop 2023-10-16T14:36:46.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 msgr2=0x7f28500c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 0x7f28500c9130 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f28500cb0d0 tx=0x7f284800f520 comp rx=0 tx=0).stop 2023-10-16T14:36:46.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 shutdown_connections 2023-10-16T14:36:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f283406b010 0x7f283406d4d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28500c8cb0 0x7f28500c9130 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f285015abc0 0x7f28500c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 --2- 172.21.15.29:0/1007886106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28400047c0 0x7f28500c3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 >> 172.21.15.29:0/1007886106 conn(0x7f28500b0170 msgr2=0x7f28500b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 shutdown_connections 2023-10-16T14:36:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.427+0000 7f2855d6e640 1 -- 172.21.15.29:0/1007886106 wait complete. 2023-10-16T14:36:46.451 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-10-16T14:36:46.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:46.452 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-10-16T14:36:46.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-10-16T14:36:46.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.737+0000 7f2f58925640 1 Processor -- start 2023-10-16T14:36:46.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.737+0000 7f2f58925640 1 -- start start 2023-10-16T14:36:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.739+0000 7f2f58925640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2f5406a4b0 0x7f2f540ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.739+0000 7f2f58925640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540cb030 con 0x7f2f5415ad70 2023-10-16T14:36:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.739+0000 7f2f58925640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540c91c0 con 0x7f2f540c9ca0 2023-10-16T14:36:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.739+0000 7f2f58925640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f540c94a0 con 0x7f2f5406a4b0 2023-10-16T14:36:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.739+0000 7f2f5259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2f5415ad70 0x7f2f54069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49062/0 (socket says 172.21.15.29:49062) 2023-10-16T14:36:46.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.739+0000 7f2f5259c640 1 -- 172.21.15.29:0/3906586427 learned_addr learned my addr 172.21.15.29:0/3906586427 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:46.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901307891 0 0) 0x7f2f540cb030 con 0x7f2f5415ad70 2023-10-16T14:36:46.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f3c003520 con 0x7f2f5415ad70 2023-10-16T14:36:46.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f51d9b640 1 --2- 172.21.15.29:0/3906586427 >> v2:172.21.15.170:3300/0 conn(0x7f2f5406a4b0 0x7f2f540ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3335016845 0 0) 0x7f2f3c003520 con 0x7f2f5415ad70 2023-10-16T14:36:46.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 >> v2:172.21.15.170:3300/0 conn(0x7f2f5406a4b0 msgr2=0x7f2f540ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 --2- 172.21.15.29:0/3906586427 >> v2:172.21.15.170:3300/0 conn(0x7f2f5406a4b0 0x7f2f540ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 >> v1:172.21.15.170:6789/0 conn(0x7f2f540c9ca0 legacy=0x7f2f540ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540c9780 con 0x7f2f5415ad70 2023-10-16T14:36:46.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2f44001670 con 0x7f2f5415ad70 2023-10-16T14:36:46.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 >> v1:172.21.15.29:6789/0 conn(0x7f2f5415ad70 legacy=0x7f2f54069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c003970 0x7f2f3c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3c0047c0 0x7f2f3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f3c0052a0 0x7f2f3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c94a0 con 0x7f2f3c0047c0 2023-10-16T14:36:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c00c1a0 con 0x7f2f3c003970 2023-10-16T14:36:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.740+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c00c360 con 0x7f2f3c0052a0 2023-10-16T14:36:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f51d9b640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3c0047c0 0x7f2f3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f5259c640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c003970 0x7f2f3c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f52d9d640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f3c0052a0 0x7f2f3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f51d9b640 1 -- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f3c0052a0 msgr2=0x7f2f3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f51d9b640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f3c0052a0 0x7f2f3c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f51d9b640 1 -- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c003970 msgr2=0x7f2f3c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f51d9b640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c003970 0x7f2f3c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f51d9b640 1 -- 172.21.15.29:0/3906586427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c003520 con 0x7f2f3c0047c0 2023-10-16T14:36:46.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.741+0000 7f2f52d9d640 1 -- 172.21.15.29:0/3906586427 reap_dead start 2023-10-16T14:36:46.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.742+0000 7f2f51d9b640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3c0047c0 0x7f2f3c004c40 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800a970 tx=0x7f2f3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251a4b05adfe2972 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:46.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48031660 con 0x7f2f3c0047c0 2023-10-16T14:36:46.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f480317c0 con 0x7f2f3c0047c0 2023-10-16T14:36:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f5159a640 1 -- 172.21.15.29:0/3906586427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48031bb0 con 0x7f2f3c0047c0 2023-10-16T14:36:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 -- 172.21.15.29:0/3906586427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3c0047c0 msgr2=0x7f2f3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3c0047c0 0x7f2f3c004c40 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800a970 tx=0x7f2f3c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 -- 172.21.15.29:0/3906586427 shutdown_connections 2023-10-16T14:36:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 --2- 172.21.15.29:0/3906586427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f3c0047c0 0x7f2f3c004c40 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 -- 172.21.15.29:0/3906586427 >> 172.21.15.29:0/3906586427 conn(0x7f2f540ce9b0 msgr2=0x7f2f540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 -- 172.21.15.29:0/3906586427 shutdown_connections 2023-10-16T14:36:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.743+0000 7f2f58925640 1 -- 172.21.15.29:0/3906586427 wait complete. 2023-10-16T14:36:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.745+0000 7f2f58925640 1 Processor -- start 2023-10-16T14:36:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.745+0000 7f2f58925640 1 -- start start 2023-10-16T14:36:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.746+0000 7f2f58925640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c0047c0 0x7f2f54154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.746+0000 7f2f58925640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f540c6c50 0x7f2f540b3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.746+0000 7f2f58925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 0x7f2f540c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.746+0000 7f2f58925640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540b3cf0 con 0x7f2f540b3870 2023-10-16T14:36:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.746+0000 7f2f58925640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540c67c0 con 0x7f2f3c0047c0 2023-10-16T14:36:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.746+0000 7f2f58925640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541548a0 con 0x7f2f540c6c50 2023-10-16T14:36:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 0x7f2f540c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 0x7f2f540c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58168/0 (socket says 172.21.15.29:58168) 2023-10-16T14:36:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 -- 172.21.15.29:0/3268467587 learned_addr learned my addr 172.21.15.29:0/3268467587 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f5259c640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c0047c0 0x7f2f54154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f51d9b640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f540c6c50 0x7f2f540b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 -- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f540c6c50 msgr2=0x7f2f540b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f540c6c50 0x7f2f540b3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 -- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c0047c0 msgr2=0x7f2f54154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c0047c0 0x7f2f54154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.747+0000 7f2f52d9d640 1 -- 172.21.15.29:0/3268467587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540c5bb0 con 0x7f2f540b3870 2023-10-16T14:36:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.748+0000 7f2f52d9d640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 0x7f2f540c7290 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e570 tx=0x7f2f4c00ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.748+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c003df0 con 0x7f2f540b3870 2023-10-16T14:36:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.748+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c013420 con 0x7f2f540b3870 2023-10-16T14:36:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.748+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0137d0 con 0x7f2f540b3870 2023-10-16T14:36:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.748+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f540c5e90 con 0x7f2f540b3870 2023-10-16T14:36:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.748+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f540c6450 con 0x7f2f540b3870 2023-10-16T14:36:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.749+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f540c6610 con 0x7f2f540b3870 2023-10-16T14:36:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.750+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f2f4c013930 con 0x7f2f540b3870 2023-10-16T14:36:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.752+0000 7f2f42ffd640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f2f3406b090 0x7f2f3406d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.752+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f2f4c089b00 con 0x7f2f540b3870 2023-10-16T14:36:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.752+0000 7f2f5259c640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f2f3406b090 0x7f2f3406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.754+0000 7f2f5259c640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f2f3406b090 0x7f2f3406d550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f44001cb0 tx=0x7f2f44009040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.756+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2f4c054840 con 0x7f2f540b3870 2023-10-16T14:36:46.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:46.976+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f54069da0 con 0x7f2f3406b090 2023-10-16T14:36:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.125+0000 7f2f42ffd640 1 -- 172.21.15.29:0/3268467587 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f2f440047e0 con 0x7f2f3406b090 2023-10-16T14:36:47.128 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/yearly 2023-10-16T14:36:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f2f3406b090 msgr2=0x7f2f3406d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f2f3406b090 0x7f2f3406d550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f44001cb0 tx=0x7f2f44009040 comp rx=0 tx=0).stop 2023-10-16T14:36:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 msgr2=0x7f2f540c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 0x7f2f540c7290 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e570 tx=0x7f2f4c00ea30 comp rx=0 tx=0).stop 2023-10-16T14:36:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 shutdown_connections 2023-10-16T14:36:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f2f3406b090 0x7f2f3406d550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f540b3870 0x7f2f540c7290 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2f540c6c50 0x7f2f540b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 --2- 172.21.15.29:0/3268467587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2f3c0047c0 0x7f2f54154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.128+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 >> 172.21.15.29:0/3268467587 conn(0x7f2f540ce9b0 msgr2=0x7f2f540ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.129+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 shutdown_connections 2023-10-16T14:36:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.129+0000 7f2f58925640 1 -- 172.21.15.29:0/3268467587 wait complete. 2023-10-16T14:36:47.154 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-10-16T14:36:47.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:47.155 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-10-16T14:36:47.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-10-16T14:36:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.435+0000 7fb7f73f0640 1 Processor -- start 2023-10-16T14:36:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.435+0000 7fb7f73f0640 1 -- start start 2023-10-16T14:36:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.436+0000 7fb7f73f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7f0159e90 0x7fb7f00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f73f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f006c140 0x7fb7f006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f73f0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f0070360 con 0x7fb7f0159ab0 2023-10-16T14:36:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f73f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f006cb10 con 0x7fb7f0159e90 2023-10-16T14:36:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f73f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f015d890 con 0x7fb7f006c140 2023-10-16T14:36:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f518c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb7f0159ab0 0x7fb7f006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49078/0 (socket says 172.21.15.29:49078) 2023-10-16T14:36:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f518c640 1 -- 172.21.15.29:0/1035872112 learned_addr learned my addr 172.21.15.29:0/1035872112 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7f498b640 1 --2- 172.21.15.29:0/1035872112 >> v2:172.21.15.29:3300/0 conn(0x7fb7f0159e90 0x7fb7f00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7e7fff640 1 -- 172.21.15.29:0/1035872112 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991498565 0 0) 0x7fb7f0070360 con 0x7fb7f0159ab0 2023-10-16T14:36:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.437+0000 7fb7e7fff640 1 -- 172.21.15.29:0/1035872112 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc003520 con 0x7fb7f0159ab0 2023-10-16T14:36:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.438+0000 7fb7f598d640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f006c140 0x7fb7f006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.438+0000 7fb7f498b640 1 -- 172.21.15.29:0/1035872112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f006c140 msgr2=0x7fb7f006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.438+0000 7fb7f498b640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f006c140 0x7fb7f006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.438+0000 7fb7f498b640 1 -- 172.21.15.29:0/1035872112 >> v1:172.21.15.29:6789/0 conn(0x7fb7f0159ab0 legacy=0x7fb7f006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.438+0000 7fb7f498b640 1 -- 172.21.15.29:0/1035872112 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f00cd1f0 con 0x7fb7f0159e90 2023-10-16T14:36:47.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.438+0000 7fb7f598d640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f006c140 0x7fb7f006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.439+0000 7fb7f498b640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f00ce370 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00097a0 tx=0x7fb7e002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444a4d4043036125 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.439+0000 7fb7e7fff640 1 -- 172.21.15.29:0/1035872112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e002f050 con 0x7fb7f0159e90 2023-10-16T14:36:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.439+0000 7fb7e7fff640 1 -- 172.21.15.29:0/1035872112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e0031700 con 0x7fb7f0159e90 2023-10-16T14:36:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.439+0000 7fb7e7fff640 1 -- 172.21.15.29:0/1035872112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0031a90 con 0x7fb7f0159e90 2023-10-16T14:36:47.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 -- 172.21.15.29:0/1035872112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 msgr2=0x7fb7f00ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f00ce370 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00097a0 tx=0x7fb7e002b7a0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 -- 172.21.15.29:0/1035872112 shutdown_connections 2023-10-16T14:36:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f006c140 0x7fb7f006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 --2- 172.21.15.29:0/1035872112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f00ce370 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 -- 172.21.15.29:0/1035872112 >> 172.21.15.29:0/1035872112 conn(0x7fb7f00ce9b0 msgr2=0x7fb7f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 -- 172.21.15.29:0/1035872112 shutdown_connections 2023-10-16T14:36:47.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.440+0000 7fb7f73f0640 1 -- 172.21.15.29:0/1035872112 wait complete. 2023-10-16T14:36:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.442+0000 7fb7f73f0640 1 Processor -- start 2023-10-16T14:36:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.442+0000 7fb7f73f0640 1 -- start start 2023-10-16T14:36:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f73f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7f006c140 0x7fb7f00b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f73f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f0159ab0 0x7fb7f00b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f73f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f73f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0152730 con 0x7fb7f0159e90 2023-10-16T14:36:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f73f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00b21e0 con 0x7fb7f006c140 2023-10-16T14:36:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f73f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00b3160 con 0x7fb7f0159ab0 2023-10-16T14:36:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f598d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f598d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f006b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58190/0 (socket says 172.21.15.29:58190) 2023-10-16T14:36:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f598d640 1 -- 172.21.15.29:0/2973312072 learned_addr learned my addr 172.21.15.29:0/2973312072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.443+0000 7fb7f498b640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f0159ab0 0x7fb7f00b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f598d640 1 -- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f0159ab0 msgr2=0x7fb7f00b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f518c640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7f006c140 0x7fb7f00b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f598d640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb7f0159ab0 0x7fb7f00b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f598d640 1 -- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7f006c140 msgr2=0x7fb7f00b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f598d640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7f006c140 0x7fb7f00b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f598d640 1 -- 172.21.15.29:0/2973312072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e0009400 con 0x7fb7f0159e90 2023-10-16T14:36:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f598d640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7f0159e90 0x7fb7f006b3a0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800cd50 tx=0x7fb7e800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e800f680 con 0x7fb7f0159e90 2023-10-16T14:36:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e800f800 con 0x7fb7f0159e90 2023-10-16T14:36:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e800fbb0 con 0x7fb7f0159e90 2023-10-16T14:36:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f73f0640 1 -- 172.21.15.29:0/2973312072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f00b2e80 con 0x7fb7f0159e90 2023-10-16T14:36:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.444+0000 7fb7f73f0640 1 -- 172.21.15.29:0/2973312072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f01f5ce0 con 0x7fb7f0159e90 2023-10-16T14:36:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.446+0000 7fb7f73f0640 1 -- 172.21.15.29:0/2973312072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f01f5860 con 0x7fb7f0159e90 2023-10-16T14:36:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.447+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fb7e800b040 con 0x7fb7f0159e90 2023-10-16T14:36:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.448+0000 7fb7e5ffb640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb7d406b070 0x7fb7d406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.448+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb7e808a590 con 0x7fb7f0159e90 2023-10-16T14:36:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.449+0000 7fb7f518c640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb7d406b070 0x7fb7d406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.450+0000 7fb7f518c640 1 --2- 172.21.15.29:0/2973312072 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb7d406b070 0x7fb7d406d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0011e0 tx=0x7fb7ec009040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.453+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb7f01f5860 con 0x7fb7f0159e90 2023-10-16T14:36:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.668+0000 7fb7f73f0640 1 -- 172.21.15.29:0/2973312072 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7f00cd7b0 con 0x7fb7d406b070 2023-10-16T14:36:47.674 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:47.670+0000 7fbf69174640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T14:36:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.671+0000 7fb7e5ffb640 1 -- 172.21.15.29:0/2973312072 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7fb7ec0058d0 con 0x7fb7d406b070 2023-10-16T14:36:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T14:36:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:36:47.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-10-16T14:36:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.919+0000 7fd0bf99e640 1 Processor -- start 2023-10-16T14:36:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.920+0000 7fd0bf99e640 1 -- start start 2023-10-16T14:36:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.921+0000 7fd0bf99e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b8159e90 0x7fd0b80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.921+0000 7fd0bf99e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b8070360 con 0x7fd0b806c140 2023-10-16T14:36:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.921+0000 7fd0bf99e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b80cd1f0 con 0x7fd0b8159ab0 2023-10-16T14:36:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.921+0000 7fd0bf99e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b815d890 con 0x7fd0b8159e90 2023-10-16T14:36:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.921+0000 7fd0bcf39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b8159e90 0x7fd0b80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.922+0000 7fd0bcf39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b8159e90 0x7fd0b80ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58196/0 (socket says 172.21.15.29:58196) 2023-10-16T14:36:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.922+0000 7fd0bdf3b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd0b806c140 0x7fd0b815dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49092/0 (socket says 172.21.15.29:49092) 2023-10-16T14:36:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.922+0000 7fd0bcf39640 1 -- 172.21.15.29:0/3219624556 learned_addr learned my addr 172.21.15.29:0/3219624556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.922+0000 7fd0bcf39640 1 -- 172.21.15.29:0/3219624556 >> v1:172.21.15.170:6789/0 conn(0x7fd0b8159ab0 legacy=0x7fd0b806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.922+0000 7fd0bcf39640 1 -- 172.21.15.29:0/3219624556 >> v1:172.21.15.29:6789/0 conn(0x7fd0b806c140 legacy=0x7fd0b815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.922+0000 7fd0bcf39640 1 -- 172.21.15.29:0/3219624556 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b80cd4d0 con 0x7fd0b8159e90 2023-10-16T14:36:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.923+0000 7fd0bcf39640 1 --2- 172.21.15.29:0/3219624556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159e90 0x7fd0b80ce370 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009870 tx=0x7fd0ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c03ea09d9e857f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.923+0000 7fd0abfff640 1 -- 172.21.15.29:0/3219624556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0317e0 con 0x7fd0b8159e90 2023-10-16T14:36:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0abfff640 1 -- 172.21.15.29:0/3219624556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac031940 con 0x7fd0b8159e90 2023-10-16T14:36:47.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0abfff640 1 -- 172.21.15.29:0/3219624556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac031cd0 con 0x7fd0b8159e90 2023-10-16T14:36:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 -- 172.21.15.29:0/3219624556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159e90 msgr2=0x7fd0b80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 --2- 172.21.15.29:0/3219624556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159e90 0x7fd0b80ce370 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009870 tx=0x7fd0ac02b880 comp rx=0 tx=0).stop 2023-10-16T14:36:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 -- 172.21.15.29:0/3219624556 shutdown_connections 2023-10-16T14:36:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 --2- 172.21.15.29:0/3219624556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159e90 0x7fd0b80ce370 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 -- 172.21.15.29:0/3219624556 >> 172.21.15.29:0/3219624556 conn(0x7fd0b80ce9b0 msgr2=0x7fd0b80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 -- 172.21.15.29:0/3219624556 shutdown_connections 2023-10-16T14:36:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.924+0000 7fd0bf99e640 1 -- 172.21.15.29:0/3219624556 wait complete. 2023-10-16T14:36:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.926+0000 7fd0bf99e640 1 Processor -- start 2023-10-16T14:36:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.926+0000 7fd0bf99e640 1 -- start start 2023-10-16T14:36:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bf99e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b806c140 0x7fd0b80b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bf99e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159ab0 0x7fd0b80c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bf99e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b8159e90 0x7fd0b8158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bf99e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80b4540 con 0x7fd0b8159ab0 2023-10-16T14:36:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bf99e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80b3ca0 con 0x7fd0b806c140 2023-10-16T14:36:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bf99e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80b3f80 con 0x7fd0b8159e90 2023-10-16T14:36:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.927+0000 7fd0bcf39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159ab0 0x7fd0b80c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159ab0 0x7fd0b80c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58202/0 (socket says 172.21.15.29:58202) 2023-10-16T14:36:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 -- 172.21.15.29:0/1442598473 learned_addr learned my addr 172.21.15.29:0/1442598473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bd73a640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b806c140 0x7fd0b80b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bdf3b640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b8159e90 0x7fd0b8158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 -- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b8159e90 msgr2=0x7fd0b8158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b8159e90 0x7fd0b8158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 -- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b806c140 msgr2=0x7fd0b80b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b806c140 0x7fd0b80b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 -- 172.21.15.29:0/1442598473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac0094d0 con 0x7fd0b8159ab0 2023-10-16T14:36:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bd73a640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b806c140 0x7fd0b80b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bdf3b640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b8159e90 0x7fd0b8158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.928+0000 7fd0bcf39640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b8159ab0 0x7fd0b80c24e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac02b8c0 tx=0x7fd0ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.929+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac032980 con 0x7fd0b8159ab0 2023-10-16T14:36:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.929+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac032ae0 con 0x7fd0b8159ab0 2023-10-16T14:36:47.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.929+0000 7fd0bf99e640 1 -- 172.21.15.29:0/1442598473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b80b4260 con 0x7fd0b8159ab0 2023-10-16T14:36:47.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.929+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac032e90 con 0x7fd0b8159ab0 2023-10-16T14:36:47.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.929+0000 7fd0bf99e640 1 -- 172.21.15.29:0/1442598473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b8158640 con 0x7fd0b8159ab0 2023-10-16T14:36:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.930+0000 7fd0bf99e640 1 -- 172.21.15.29:0/1442598473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b804e160 con 0x7fd0b8159ab0 2023-10-16T14:36:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.931+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fd0ac04e450 con 0x7fd0b8159ab0 2023-10-16T14:36:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.932+0000 7fd0a9ffb640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fd09c06b070 0x7fd09c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.932+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd0ac0ac030 con 0x7fd0b8159ab0 2023-10-16T14:36:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.933+0000 7fd0bd73a640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fd09c06b070 0x7fd09c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.934+0000 7fd0bd73a640 1 --2- 172.21.15.29:0/1442598473 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fd09c06b070 0x7fd09c06d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4003eb0 tx=0x7fd0b4009040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:47.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:47.938+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0ac076be0 con 0x7fd0b8159ab0 2023-10-16T14:36:48.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.150+0000 7fd0bf99e640 1 -- 172.21.15.29:0/1442598473 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b80cda90 con 0x7fd09c06b070 2023-10-16T14:36:48.155 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:48.152+0000 7fbf69174640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.153+0000 7fd0a9ffb640 1 -- 172.21.15.29:0/1442598473 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd0b4005030 con 0x7fd09c06b070 2023-10-16T14:36:48.156 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:36:48.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-10-16T14:36:48.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.404+0000 7f79fe924640 1 Processor -- start 2023-10-16T14:36:48.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.404+0000 7f79fe924640 1 -- start start 2023-10-16T14:36:48.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.405+0000 7f79fe924640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f79f815d150 0x7f79f80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.405+0000 7f79fe924640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f80ac890 con 0x7f79f806a4b0 2023-10-16T14:36:48.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.406+0000 7f79fe924640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f8161650 con 0x7f79f8160b90 2023-10-16T14:36:48.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.406+0000 7f79fe924640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f815d530 con 0x7f79f815d150 2023-10-16T14:36:48.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.406+0000 7f79fcec1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f79f806a4b0 0x7f79f80c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49108/0 (socket says 172.21.15.29:49108) 2023-10-16T14:36:48.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.406+0000 7f79fcec1640 1 -- 172.21.15.29:0/1065629826 learned_addr learned my addr 172.21.15.29:0/1065629826 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.406+0000 7f79f7fff640 1 --2- 172.21.15.29:0/1065629826 >> v2:172.21.15.170:3300/0 conn(0x7f79f815d150 0x7f79f80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.406+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550876781 0 0) 0x7f79f80ac890 con 0x7f79f806a4b0 2023-10-16T14:36:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e4003520 con 0x7f79f806a4b0 2023-10-16T14:36:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082158382 0 0) 0x7f79e4003520 con 0x7f79f806a4b0 2023-10-16T14:36:48.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 >> v2:172.21.15.170:3300/0 conn(0x7f79f815d150 msgr2=0x7f79f80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 --2- 172.21.15.29:0/1065629826 >> v2:172.21.15.170:3300/0 conn(0x7f79f815d150 0x7f79f80ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 >> v1:172.21.15.170:6789/0 conn(0x7f79f8160b90 legacy=0x7f79f8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f8161930 con 0x7f79f806a4b0 2023-10-16T14:36:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f79f00019f0 con 0x7f79f806a4b0 2023-10-16T14:36:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 >> v1:172.21.15.29:6789/0 conn(0x7f79f806a4b0 legacy=0x7f79f80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79e40039c0 0x7f79e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79e40047c0 0x7f79e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e4005180 0x7f79e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f815d530 con 0x7f79e40047c0 2023-10-16T14:36:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e400c080 con 0x7f79e4005180 2023-10-16T14:36:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e400c240 con 0x7f79e40039c0 2023-10-16T14:36:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.407+0000 7f79ef7fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79e40047c0 0x7f79e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79f7fff640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79e40039c0 0x7f79e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79fcec1640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e4005180 0x7f79e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79ef7fe640 1 -- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79e40039c0 msgr2=0x7f79e4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79ef7fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79e40039c0 0x7f79e4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79ef7fe640 1 -- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e4005180 msgr2=0x7f79e400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79ef7fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e4005180 0x7f79e400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79ef7fe640 1 -- 172.21.15.29:0/1065629826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e4003520 con 0x7f79e40047c0 2023-10-16T14:36:48.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79fcec1640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e4005180 0x7f79e400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:48.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.408+0000 7f79fcec1640 1 -- 172.21.15.29:0/1065629826 reap_dead start 2023-10-16T14:36:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.409+0000 7f79ef7fe640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79e40047c0 0x7f79e4004c40 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f79e0001750 tx=0x7f79e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24495054017fa34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.409+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e00052d0 con 0x7f79e40047c0 2023-10-16T14:36:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.409+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e002e050 con 0x7f79e40047c0 2023-10-16T14:36:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.409+0000 7f79f77fe640 1 -- 172.21.15.29:0/1065629826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0003830 con 0x7f79e40047c0 2023-10-16T14:36:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 -- 172.21.15.29:0/1065629826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79e40047c0 msgr2=0x7f79e4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79e40047c0 0x7f79e4004c40 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f79e0001750 tx=0x7f79e40057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 -- 172.21.15.29:0/1065629826 shutdown_connections 2023-10-16T14:36:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 --2- 172.21.15.29:0/1065629826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79e40047c0 0x7f79e4004c40 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 -- 172.21.15.29:0/1065629826 >> 172.21.15.29:0/1065629826 conn(0x7f79f80ce9b0 msgr2=0x7f79f80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 -- 172.21.15.29:0/1065629826 shutdown_connections 2023-10-16T14:36:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.410+0000 7f79fe924640 1 -- 172.21.15.29:0/1065629826 wait complete. 2023-10-16T14:36:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.412+0000 7f79fe924640 1 Processor -- start 2023-10-16T14:36:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.412+0000 7f79fe924640 1 -- start start 2023-10-16T14:36:48.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fe924640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e40047c0 0x7f79f8149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fe924640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f80b2e30 0x7f79f80b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fe924640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f815ec90 0x7f79f815f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fe924640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80b37f0 con 0x7f79f815ec90 2023-10-16T14:36:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fe924640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8149650 con 0x7f79e40047c0 2023-10-16T14:36:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fe924640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f815fa30 con 0x7f79f80b2e30 2023-10-16T14:36:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fcec1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f815ec90 0x7f79f815f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fcec1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f815ec90 0x7f79f815f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58218/0 (socket says 172.21.15.29:58218) 2023-10-16T14:36:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.413+0000 7f79fcec1640 1 -- 172.21.15.29:0/413386337 learned_addr learned my addr 172.21.15.29:0/413386337 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79f7fff640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e40047c0 0x7f79f8149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79ef7fe640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f80b2e30 0x7f79f80b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fcec1640 1 -- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f80b2e30 msgr2=0x7f79f80b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fcec1640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79f80b2e30 0x7f79f80b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fcec1640 1 -- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e40047c0 msgr2=0x7f79f8149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fcec1640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e40047c0 0x7f79f8149110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fcec1640 1 -- 172.21.15.29:0/413386337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f815fd10 con 0x7f79f815ec90 2023-10-16T14:36:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79f7fff640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79e40047c0 0x7f79f8149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fcec1640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f815ec90 0x7f79f815f110 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f79f80cf870 tx=0x7f79f000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0004d60 con 0x7f79f815ec90 2023-10-16T14:36:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f000bb80 con 0x7f79f815ec90 2023-10-16T14:36:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0007070 con 0x7f79f815ec90 2023-10-16T14:36:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.414+0000 7f79fe924640 1 -- 172.21.15.29:0/413386337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f815fff0 con 0x7f79f815ec90 2023-10-16T14:36:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.415+0000 7f79fe924640 1 -- 172.21.15.29:0/413386337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f81605b0 con 0x7f79f815ec90 2023-10-16T14:36:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.416+0000 7f79fe924640 1 -- 172.21.15.29:0/413386337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f804e160 con 0x7f79f815ec90 2023-10-16T14:36:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.417+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f79f00071d0 con 0x7f79f815ec90 2023-10-16T14:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.418+0000 7f79f57fa640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f79d806b070 0x7f79d806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.418+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f79f008a150 con 0x7f79f815ec90 2023-10-16T14:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.419+0000 7f79f7fff640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f79d806b070 0x7f79d806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.420+0000 7f79f7fff640 1 --2- 172.21.15.29:0/413386337 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f79d806b070 0x7f79d806d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009a80 tx=0x7f79e8008040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.422+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f79f0054c50 con 0x7f79f815ec90 2023-10-16T14:36:48.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.639+0000 7f79fe924640 1 -- 172.21.15.29:0/413386337 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f79f80acb50 con 0x7f79d806b070 2023-10-16T14:36:48.645 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:48.642+0000 7fbf69174640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.643+0000 7f79f57fa640 1 -- 172.21.15.29:0/413386337 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f79e8012b00 con 0x7f79d806b070 2023-10-16T14:36:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:36:48.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-10-16T14:36:48.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:36:48.738 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:36:48.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.888+0000 7f3cd5826640 1 Processor -- start 2023-10-16T14:36:48.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.888+0000 7f3cd5826640 1 -- start start 2023-10-16T14:36:48.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cd5826640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3cd00cda70 0x7f3cd006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cd5826640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cd006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cd5826640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00c3610 con 0x7f3cd0159e40 2023-10-16T14:36:48.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cd5826640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd006b880 con 0x7f3cd00cda70 2023-10-16T14:36:48.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cd5826640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00cdeb0 con 0x7f3cd006b3f0 2023-10-16T14:36:48.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cceffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3cd0159e40 0x7f3cd00ca780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49112/0 (socket says 172.21.15.29:49112) 2023-10-16T14:36:48.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3cceffd640 1 -- 172.21.15.29:0/2911009147 learned_addr learned my addr 172.21.15.29:0/2911009147 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:48.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.890+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351211680 0 0) 0x7f3cd00c3610 con 0x7f3cd0159e40 2023-10-16T14:36:48.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc003520 con 0x7f3cd0159e40 2023-10-16T14:36:48.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3cce7fc640 1 --2- 172.21.15.29:0/2911009147 >> v2:172.21.15.170:3300/0 conn(0x7f3cd00cda70 0x7f3cd006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccf7fe640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cd006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124707343 0 0) 0x7f3cbc003520 con 0x7f3cd0159e40 2023-10-16T14:36:48.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 msgr2=0x7f3cd006ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cd006ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 >> v2:172.21.15.170:3300/0 conn(0x7f3cd00cda70 msgr2=0x7f3cd006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 --2- 172.21.15.29:0/2911009147 >> v2:172.21.15.170:3300/0 conn(0x7f3cd00cda70 0x7f3cd006aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd015a220 con 0x7f3cd0159e40 2023-10-16T14:36:48.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3cc00019f0 con 0x7f3cd0159e40 2023-10-16T14:36:48.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccf7fe640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cd006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:48.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 >> v1:172.21.15.29:6789/0 conn(0x7f3cd0159e40 legacy=0x7f3cd00ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cbc0047c0 0x7f3cbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cbc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd006b880 con 0x7f3cbc003970 2023-10-16T14:36:48.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00cdeb0 con 0x7f3cbc0047c0 2023-10-16T14:36:48.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.891+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc00dbc0 con 0x7f3cd006b3f0 2023-10-16T14:36:48.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3ccf7fe640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cbc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cce7fc640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cbc0047c0 0x7f3cbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cceffd640 1 -- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 msgr2=0x7f3cbc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cbc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cceffd640 1 -- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cbc0047c0 msgr2=0x7f3cbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:48.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cbc0047c0 0x7f3cbc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cceffd640 1 -- 172.21.15.29:0/2911009147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc003520 con 0x7f3cbc003970 2023-10-16T14:36:48.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3cce7fc640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cbc0047c0 0x7f3cbc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.892+0000 7f3ccf7fe640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd006b3f0 0x7f3cbc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.893+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cbc003d90 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000a790 tx=0x7f3cbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197dadf4ae59039f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.893+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0034630 con 0x7f3cbc003970 2023-10-16T14:36:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.893+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc002f070 con 0x7f3cbc003970 2023-10-16T14:36:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.893+0000 7f3ccdffb640 1 -- 172.21.15.29:0/2911009147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0034a10 con 0x7f3cbc003970 2023-10-16T14:36:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3cd5826640 1 -- 172.21.15.29:0/2911009147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 msgr2=0x7f3cbc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3cd5826640 1 --2- 172.21.15.29:0/2911009147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cbc003d90 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000a790 tx=0x7f3cbc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3ccf7fe640 1 -- 172.21.15.29:0/2911009147 reap_dead start 2023-10-16T14:36:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3cd5826640 1 -- 172.21.15.29:0/2911009147 shutdown_connections 2023-10-16T14:36:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3cd5826640 1 -- 172.21.15.29:0/2911009147 >> 172.21.15.29:0/2911009147 conn(0x7f3cd00cfac0 msgr2=0x7f3cd00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3cd5826640 1 -- 172.21.15.29:0/2911009147 shutdown_connections 2023-10-16T14:36:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.894+0000 7f3cd5826640 1 -- 172.21.15.29:0/2911009147 wait complete. 2023-10-16T14:36:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.896+0000 7f3cd5826640 1 Processor -- start 2023-10-16T14:36:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.896+0000 7f3cd5826640 1 -- start start 2023-10-16T14:36:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cd5826640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cd015cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cd5826640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cd015c4b0 0x7f3cd00c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cd5826640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd015c910 0x7f3cd00c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cd5826640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd015d320 con 0x7f3cbc003970 2023-10-16T14:36:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cd5826640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd014d620 con 0x7f3cd015c4b0 2023-10-16T14:36:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cd5826640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00c6480 con 0x7f3cd015c910 2023-10-16T14:36:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cd015cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cd015cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58232/0 (socket says 172.21.15.29:58232) 2023-10-16T14:36:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.897+0000 7f3cceffd640 1 -- 172.21.15.29:0/2405107170 learned_addr learned my addr 172.21.15.29:0/2405107170 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cceffd640 1 -- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd015c910 msgr2=0x7f3cd00c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3cd015c910 0x7f3cd00c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cceffd640 1 -- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cd015c4b0 msgr2=0x7f3cd00c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3cd015c4b0 0x7f3cd00c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cceffd640 1 -- 172.21.15.29:0/2405107170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00b27a0 con 0x7f3cbc003970 2023-10-16T14:36:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cceffd640 1 --2- 172.21.15.29:0/2405107170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cbc003970 0x7f3cd015cde0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3cc002cba0 tx=0x7f3cc0001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0043070 con 0x7f3cbc003970 2023-10-16T14:36:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc0034700 con 0x7f3cbc003970 2023-10-16T14:36:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cd5826640 1 -- 172.21.15.29:0/2405107170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd014d060 con 0x7f3cbc003970 2023-10-16T14:36:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.898+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0034a90 con 0x7f3cbc003970 2023-10-16T14:36:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.899+0000 7f3cd5826640 1 -- 172.21.15.29:0/2405107170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd00b2a40 con 0x7f3cbc003970 2023-10-16T14:36:48.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.900+0000 7f3cd5826640 1 -- 172.21.15.29:0/2405107170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd00b2600 con 0x7f3cbc003970 2023-10-16T14:36:48.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.901+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f3cc0031030 con 0x7f3cbc003970 2023-10-16T14:36:48.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.902+0000 7f3cc7fff640 1 --2- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f3cb006b0a0 0x7f3cb006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:48.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.902+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3cc00ab640 con 0x7f3cbc003970 2023-10-16T14:36:48.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.902+0000 7f3cce7fc640 1 --2- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f3cb006b0a0 0x7f3cb006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:48.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.904+0000 7f3cce7fc640 1 --2- 172.21.15.29:0/2405107170 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f3cb006b0a0 0x7f3cb006d560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3cd014ae60 tx=0x7f3cb8008040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:48.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:48.906+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3cd00b2600 con 0x7f3cbc003970 2023-10-16T14:36:49.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.122+0000 7f3cd5826640 1 -- 172.21.15.29:0/2405107170 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cd00cb2b0 con 0x7f3cb006b0a0 2023-10-16T14:36:49.128 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:49.124+0000 7fbf69174640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:49.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.126+0000 7f3cc7fff640 1 -- 172.21.15.29:0/2405107170 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3cb8006bd0 con 0x7f3cb006b0a0 2023-10-16T14:36:49.128 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:36:49.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-10-16T14:36:49.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.418+0000 7fec78a2e640 1 Processor -- start 2023-10-16T14:36:49.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.419+0000 7fec78a2e640 1 -- start start 2023-10-16T14:36:49.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.420+0000 7fec78a2e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fec7415cd70 0x7fec740d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.420+0000 7fec78a2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec740b8480 0x7fec740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.420+0000 7fec78a2e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7415d150 con 0x7fec740d0be0 2023-10-16T14:36:49.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.420+0000 7fec78a2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec740d1180 con 0x7fec740b8480 2023-10-16T14:36:49.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.420+0000 7fec78a2e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec740d1460 con 0x7fec7415cd70 2023-10-16T14:36:49.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec6affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec740b8480 0x7fec740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec6affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec740b8480 0x7fec740cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58236/0 (socket says 172.21.15.29:58236) 2023-10-16T14:36:49.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec6affd640 1 -- 172.21.15.29:0/1118100444 learned_addr learned my addr 172.21.15.29:0/1118100444 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:49.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec737fe640 1 --2- 172.21.15.29:0/1118100444 >> v2:172.21.15.170:3300/0 conn(0x7fec7415cd70 0x7fec740d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec6affd640 1 -- 172.21.15.29:0/1118100444 >> v2:172.21.15.170:3300/0 conn(0x7fec7415cd70 msgr2=0x7fec740d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:49.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec6affd640 1 --2- 172.21.15.29:0/1118100444 >> v2:172.21.15.170:3300/0 conn(0x7fec7415cd70 0x7fec740d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.421+0000 7fec6affd640 1 -- 172.21.15.29:0/1118100444 >> v1:172.21.15.170:6789/0 conn(0x7fec740d0be0 legacy=0x7fec740b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:49.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.422+0000 7fec6affd640 1 -- 172.21.15.29:0/1118100444 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740d1740 con 0x7fec740b8480 2023-10-16T14:36:49.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.423+0000 7fec6affd640 1 --2- 172.21.15.29:0/1118100444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740b8480 0x7fec740cd7c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fec740b56a0 tx=0x7fec5c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6aa5b71da0d420c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:49.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.423+0000 7fec72ffd640 1 -- 172.21.15.29:0/1118100444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0317e0 con 0x7fec740b8480 2023-10-16T14:36:49.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.423+0000 7fec72ffd640 1 -- 172.21.15.29:0/1118100444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c031940 con 0x7fec740b8480 2023-10-16T14:36:49.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.423+0000 7fec72ffd640 1 -- 172.21.15.29:0/1118100444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c031cd0 con 0x7fec740b8480 2023-10-16T14:36:49.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.423+0000 7fec78a2e640 1 -- 172.21.15.29:0/1118100444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740b8480 msgr2=0x7fec740cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.423+0000 7fec78a2e640 1 --2- 172.21.15.29:0/1118100444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740b8480 0x7fec740cd7c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fec740b56a0 tx=0x7fec5c02b880 comp rx=0 tx=0).stop 2023-10-16T14:36:49.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.424+0000 7fec78a2e640 1 -- 172.21.15.29:0/1118100444 shutdown_connections 2023-10-16T14:36:49.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.424+0000 7fec78a2e640 1 --2- 172.21.15.29:0/1118100444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740b8480 0x7fec740cd7c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.424+0000 7fec78a2e640 1 --2- 172.21.15.29:0/1118100444 >> v2:172.21.15.170:3300/0 conn(0x7fec7415cd70 0x7fec740d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.424+0000 7fec78a2e640 1 -- 172.21.15.29:0/1118100444 >> 172.21.15.29:0/1118100444 conn(0x7fec740cf570 msgr2=0x7fec740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:49.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.424+0000 7fec78a2e640 1 -- 172.21.15.29:0/1118100444 shutdown_connections 2023-10-16T14:36:49.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.424+0000 7fec78a2e640 1 -- 172.21.15.29:0/1118100444 wait complete. 2023-10-16T14:36:49.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.426+0000 7fec78a2e640 1 Processor -- start 2023-10-16T14:36:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.426+0000 7fec78a2e640 1 -- start start 2023-10-16T14:36:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec78a2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec740b8480 0x7fec74156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec78a2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740d0be0 0x7fec7415f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec78a2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec7415cd70 0x7fec7415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec78a2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74159550 con 0x7fec740d0be0 2023-10-16T14:36:49.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec78a2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74158bb0 con 0x7fec7415cd70 2023-10-16T14:36:49.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec78a2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec74158e90 con 0x7fec740b8480 2023-10-16T14:36:49.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740d0be0 0x7fec7415f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740d0be0 0x7fec7415f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58240/0 (socket says 172.21.15.29:58240) 2023-10-16T14:36:49.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 -- 172.21.15.29:0/3880031314 learned_addr learned my addr 172.21.15.29:0/3880031314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 -- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec740b8480 msgr2=0x7fec74156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 --2- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec740b8480 0x7fec74156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 -- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec7415cd70 msgr2=0x7fec7415fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 --2- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec7415cd70 0x7fec7415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.427+0000 7fec6affd640 1 -- 172.21.15.29:0/3880031314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c0094d0 con 0x7fec740d0be0 2023-10-16T14:36:49.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.428+0000 7fec6affd640 1 --2- 172.21.15.29:0/3880031314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec740d0be0 0x7fec7415f870 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fec5c02b8c0 tx=0x7fec5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:49.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.428+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c032880 con 0x7fec740d0be0 2023-10-16T14:36:49.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.428+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c0329e0 con 0x7fec740d0be0 2023-10-16T14:36:49.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.428+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c032d90 con 0x7fec740d0be0 2023-10-16T14:36:49.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.428+0000 7fec78a2e640 1 -- 172.21.15.29:0/3880031314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74159170 con 0x7fec740d0be0 2023-10-16T14:36:49.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.428+0000 7fec78a2e640 1 -- 172.21.15.29:0/3880031314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74156aa0 con 0x7fec740d0be0 2023-10-16T14:36:49.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.429+0000 7fec78a2e640 1 -- 172.21.15.29:0/3880031314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7404e9b0 con 0x7fec740d0be0 2023-10-16T14:36:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.430+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fec5c04e450 con 0x7fec740d0be0 2023-10-16T14:36:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.431+0000 7fec70ff9640 1 --2- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fec5406b0a0 0x7fec5406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.431+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fec5c0abd00 con 0x7fec740d0be0 2023-10-16T14:36:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.432+0000 7fec737fe640 1 --2- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fec5406b0a0 0x7fec5406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.433+0000 7fec737fe640 1 --2- 172.21.15.29:0/3880031314 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fec5406b0a0 0x7fec5406d560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fec640098f0 tx=0x7fec64008040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:49.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.438+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec5c076940 con 0x7fec740d0be0 2023-10-16T14:36:49.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.665+0000 7fec78a2e640 1 -- 172.21.15.29:0/3880031314 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fec740d1d00 con 0x7fec5406b0a0 2023-10-16T14:36:49.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:49.667+0000 7fbf69174640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T14:36:49.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.668+0000 7fec70ff9640 1 -- 172.21.15.29:0/3880031314 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7fec64009e70 con 0x7fec5406b0a0 2023-10-16T14:36:49.671 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T14:36:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:36:49.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-10-16T14:36:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.915+0000 7fea1759e640 1 Processor -- start 2023-10-16T14:36:49.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.915+0000 7fea1759e640 1 -- start start 2023-10-16T14:36:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.916+0000 7fea1759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 0x7fea180cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.916+0000 7fea1759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1815b9d0 con 0x7fea1806c040 2023-10-16T14:36:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.916+0000 7fea1759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1815bcb0 con 0x7fea180ad970 2023-10-16T14:36:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.916+0000 7fea1759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1815b7d0 con 0x7fea1815b320 2023-10-16T14:36:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fea1806c040 0x7fea1806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49118/0 (socket says 172.21.15.29:49118) 2023-10-16T14:36:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1659c640 1 -- 172.21.15.29:0/4062660906 learned_addr learned my addr 172.21.15.29:0/4062660906 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235387620 0 0) 0x7fea1815b9d0 con 0x7fea1806c040 2023-10-16T14:36:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea04003520 con 0x7fea1806c040 2023-10-16T14:36:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea16d9d640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 0x7fea180cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3402406584 0 0) 0x7fea04003520 con 0x7fea1806c040 2023-10-16T14:36:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 msgr2=0x7fea180cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 0x7fea180cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 >> v1:172.21.15.170:6789/0 conn(0x7fea180ad970 legacy=0x7fea180cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1815bf90 con 0x7fea1806c040 2023-10-16T14:36:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.917+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fea080019f0 con 0x7fea1806c040 2023-10-16T14:36:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 >> v1:172.21.15.29:6789/0 conn(0x7fea1806c040 legacy=0x7fea1806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea040039a0 0x7fea04003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 0x7fea040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04004d00 0x7fea04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1815b7d0 con 0x7fea04004d00 2023-10-16T14:36:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0400bae0 con 0x7fea040039a0 2023-10-16T14:36:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0400bca0 con 0x7fea1815b320 2023-10-16T14:36:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea16d9d640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04004d00 0x7fea04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea1659c640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea040039a0 0x7fea04003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.918+0000 7fea15d9b640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 0x7fea040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.919+0000 7fea16d9d640 1 -- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 msgr2=0x7fea040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.919+0000 7fea16d9d640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea1815b320 0x7fea040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.919+0000 7fea16d9d640 1 -- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea040039a0 msgr2=0x7fea04003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.919+0000 7fea16d9d640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea040039a0 0x7fea04003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.919+0000 7fea16d9d640 1 -- 172.21.15.29:0/4062660906 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04003520 con 0x7fea04004d00 2023-10-16T14:36:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.920+0000 7fea16d9d640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04004d00 0x7fea04005180 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fea1000ba30 tx=0x7fea040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cc1832876442ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.920+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10032640 con 0x7fea04004d00 2023-10-16T14:36:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.920+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea100327a0 con 0x7fea04004d00 2023-10-16T14:36:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.920+0000 7fea1559a640 1 -- 172.21.15.29:0/4062660906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10032b30 con 0x7fea04004d00 2023-10-16T14:36:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea1759e640 1 -- 172.21.15.29:0/4062660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04004d00 msgr2=0x7fea04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea1759e640 1 --2- 172.21.15.29:0/4062660906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04004d00 0x7fea04005180 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fea1000ba30 tx=0x7fea040057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea16d9d640 1 -- 172.21.15.29:0/4062660906 reap_dead start 2023-10-16T14:36:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea1759e640 1 -- 172.21.15.29:0/4062660906 shutdown_connections 2023-10-16T14:36:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea1759e640 1 -- 172.21.15.29:0/4062660906 >> 172.21.15.29:0/4062660906 conn(0x7fea180ce6a0 msgr2=0x7fea180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea1759e640 1 -- 172.21.15.29:0/4062660906 shutdown_connections 2023-10-16T14:36:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.921+0000 7fea1759e640 1 -- 172.21.15.29:0/4062660906 wait complete. 2023-10-16T14:36:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.923+0000 7fea1759e640 1 Processor -- start 2023-10-16T14:36:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.923+0000 7fea1759e640 1 -- start start 2023-10-16T14:36:49.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea1759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea04004d00 0x7fea18159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea1759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea181f1fc0 0x7fea181580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea1759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea181585e0 0x7fea180c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea1759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea18158b80 con 0x7fea181585e0 2023-10-16T14:36:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea1759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea18158e60 con 0x7fea04004d00 2023-10-16T14:36:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea1759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea18159140 con 0x7fea181f1fc0 2023-10-16T14:36:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea16d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea181585e0 0x7fea180c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea16d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea181585e0 0x7fea180c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58260/0 (socket says 172.21.15.29:58260) 2023-10-16T14:36:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.924+0000 7fea16d9d640 1 -- 172.21.15.29:0/3774439693 learned_addr learned my addr 172.21.15.29:0/3774439693 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea15d9b640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea181f1fc0 0x7fea181580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea1659c640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea04004d00 0x7fea18159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea16d9d640 1 -- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea181f1fc0 msgr2=0x7fea181580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea16d9d640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea181f1fc0 0x7fea181580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea16d9d640 1 -- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea04004d00 msgr2=0x7fea18159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea16d9d640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea04004d00 0x7fea18159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea16d9d640 1 -- 172.21.15.29:0/3774439693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18159420 con 0x7fea181585e0 2023-10-16T14:36:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea1659c640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea04004d00 0x7fea18159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea16d9d640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea181585e0 0x7fea180c43a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fea1000ba30 tx=0x7fea10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.925+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10004220 con 0x7fea181585e0 2023-10-16T14:36:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.926+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea100043a0 con 0x7fea181585e0 2023-10-16T14:36:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.926+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1003b930 con 0x7fea181585e0 2023-10-16T14:36:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.926+0000 7fea1759e640 1 -- 172.21.15.29:0/3774439693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea18159700 con 0x7fea181585e0 2023-10-16T14:36:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.926+0000 7fea1759e640 1 -- 172.21.15.29:0/3774439693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea18156710 con 0x7fea181585e0 2023-10-16T14:36:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.927+0000 7fea1759e640 1 -- 172.21.15.29:0/3774439693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1806b330 con 0x7fea181585e0 2023-10-16T14:36:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.928+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fea1003ba90 con 0x7fea181585e0 2023-10-16T14:36:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.929+0000 7fea0effd640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe9f806b0a0 0x7fe9f806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.929+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fea100ac220 con 0x7fea181585e0 2023-10-16T14:36:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.930+0000 7fea1659c640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe9f806b0a0 0x7fe9f806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.931+0000 7fea1659c640 1 --2- 172.21.15.29:0/3774439693 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe9f806b0a0 0x7fe9f806d560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fea08002230 tx=0x7fea08002100 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:49.933+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea10076e40 con 0x7fea181585e0 2023-10-16T14:36:50.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.150+0000 7fea1759e640 1 -- 172.21.15.29:0/3774439693 --> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fea180cc550 con 0x7fe9f806b0a0 2023-10-16T14:36:50.156 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:50.153+0000 7fbf69174640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.154+0000 7fea0effd640 1 -- 172.21.15.29:0/3774439693 <== mgr.6431 v2:172.21.15.170:6832/230171195 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fea08005e00 con 0x7fe9f806b0a0 2023-10-16T14:36:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T14:36:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T14:36:50.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.173 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-10-16T14:36:50.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.233 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-10-16T14:36:50.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.292 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-10-16T14:36:50.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.353 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-10-16T14:36:50.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.413 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-10-16T14:36:50.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.475 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-10-16T14:36:50.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:50.537 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-10-16T14:36:50.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:36:50.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.834+0000 7fb9a1ed7640 1 Processor -- start 2023-10-16T14:36:50.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.834+0000 7fb9a1ed7640 1 -- start start 2023-10-16T14:36:50.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.835+0000 7fb9a1ed7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb99c070b60 0x7fb99c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:50.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.835+0000 7fb9a1ed7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb99c06c100 0x7fb99c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:50.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.835+0000 7fb9a1ed7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c15aef0 con 0x7fb99c159950 2023-10-16T14:36:50.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb9a1ed7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c15e490 con 0x7fb99c070b60 2023-10-16T14:36:50.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb9a1ed7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c15d910 con 0x7fb99c06c100 2023-10-16T14:36:50.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb99bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb99c070b60 0x7fb99c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:50.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb99bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb99c070b60 0x7fb99c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58264/0 (socket says 172.21.15.29:58264) 2023-10-16T14:36:50.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb99bfff640 1 -- 172.21.15.29:0/734997460 learned_addr learned my addr 172.21.15.29:0/734997460 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:50.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb9a16d6640 1 --2- 172.21.15.29:0/734997460 >> v2:172.21.15.170:3300/0 conn(0x7fb99c06c100 0x7fb99c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:50.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb99bfff640 1 -- 172.21.15.29:0/734997460 >> v2:172.21.15.170:3300/0 conn(0x7fb99c06c100 msgr2=0x7fb99c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:50.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb99bfff640 1 --2- 172.21.15.29:0/734997460 >> v2:172.21.15.170:3300/0 conn(0x7fb99c06c100 0x7fb99c06c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:50.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.836+0000 7fb99bfff640 1 -- 172.21.15.29:0/734997460 >> v1:172.21.15.170:6789/0 conn(0x7fb99c159950 legacy=0x7fb99c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:50.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.837+0000 7fb99bfff640 1 -- 172.21.15.29:0/734997460 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c15dbf0 con 0x7fb99c070b60 2023-10-16T14:36:50.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.838+0000 7fb99bfff640 1 --2- 172.21.15.29:0/734997460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c070b60 0x7fb99c15a9b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0ade70 tx=0x7fb98802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8a66bab09634b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:50.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.838+0000 7fb99b7fe640 1 -- 172.21.15.29:0/734997460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9880317e0 con 0x7fb99c070b60 2023-10-16T14:36:50.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.838+0000 7fb99b7fe640 1 -- 172.21.15.29:0/734997460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb988031940 con 0x7fb99c070b60 2023-10-16T14:36:50.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb99b7fe640 1 -- 172.21.15.29:0/734997460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988031cd0 con 0x7fb99c070b60 2023-10-16T14:36:50.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/734997460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c070b60 msgr2=0x7fb99c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/734997460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c070b60 0x7fb99c15a9b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0ade70 tx=0x7fb98802b880 comp rx=0 tx=0).stop 2023-10-16T14:36:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/734997460 shutdown_connections 2023-10-16T14:36:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/734997460 >> v2:172.21.15.170:3300/0 conn(0x7fb99c06c100 0x7fb99c06c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/734997460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c070b60 0x7fb99c15a9b0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/734997460 >> 172.21.15.29:0/734997460 conn(0x7fb99c0ce6a0 msgr2=0x7fb99c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:50.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.839+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/734997460 shutdown_connections 2023-10-16T14:36:50.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.840+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/734997460 wait complete. 2023-10-16T14:36:50.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.841+0000 7fb9a1ed7640 1 Processor -- start 2023-10-16T14:36:50.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.841+0000 7fb9a1ed7640 1 -- start start 2023-10-16T14:36:50.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.842+0000 7fb9a1ed7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb99c06c100 0x7fb99c154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:50.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.842+0000 7fb9a1ed7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb99c070b60 0x7fb99c146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:50.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a1ed7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 0x7fb99c15c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:50.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a1ed7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c155a80 con 0x7fb99c159950 2023-10-16T14:36:50.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a1ed7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c154e70 con 0x7fb99c06c100 2023-10-16T14:36:50.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a1ed7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c155150 con 0x7fb99c070b60 2023-10-16T14:36:50.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 0x7fb99c15c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:50.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 0x7fb99c15c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58266/0 (socket says 172.21.15.29:58266) 2023-10-16T14:36:50.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 -- 172.21.15.29:0/1018457472 learned_addr learned my addr 172.21.15.29:0/1018457472 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:50.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a0ed5640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb99c06c100 0x7fb99c154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:50.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb99bfff640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb99c070b60 0x7fb99c146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:50.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 -- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb99c070b60 msgr2=0x7fb99c146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:50.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb99c070b60 0x7fb99c146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:50.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 -- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb99c06c100 msgr2=0x7fb99c154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb99c06c100 0x7fb99c154790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb9a16d6640 1 -- 172.21.15.29:0/1018457472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9880094d0 con 0x7fb99c159950 2023-10-16T14:36:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.843+0000 7fb99bfff640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb99c070b60 0x7fb99c146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.844+0000 7fb9a16d6640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 0x7fb99c15c350 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb994007e40 tx=0x7fb99400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.844+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb994014620 con 0x7fb99c159950 2023-10-16T14:36:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.844+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99400ee90 con 0x7fb99c159950 2023-10-16T14:36:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.844+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9940149e0 con 0x7fb99c159950 2023-10-16T14:36:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.844+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c155430 con 0x7fb99c159950 2023-10-16T14:36:50.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.844+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c146a60 con 0x7fb99c159950 2023-10-16T14:36:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.845+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c04e9b0 con 0x7fb99c159950 2023-10-16T14:36:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.846+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 72) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fb994014b40 con 0x7fb99c159950 2023-10-16T14:36:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.847+0000 7fb9997fa640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb98006b070 0x7fb98006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:50.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.847+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb99408a660 con 0x7fb99c159950 2023-10-16T14:36:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.848+0000 7fb9a0ed5640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb98006b070 0x7fb98006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.849+0000 7fb9a0ed5640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb98006b070 0x7fb98006d530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb990006fd0 tx=0x7fb990008040 comp rx=0 tx=0).ready entity=mgr.6431 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:50.852+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb9940551e0 con 0x7fb99c159950 2023-10-16T14:36:51.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.094+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb99c0b4620 con 0x7fb99c159950 2023-10-16T14:36:51.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.311+0000 7fb9997fa640 1 -- 172.21.15.29:0/1018457472 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v73) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb994058800 con 0x7fb99c159950 2023-10-16T14:36:51.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.315+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb98006b070 msgr2=0x7fb98006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:51.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.315+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb98006b070 0x7fb98006d530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb990006fd0 tx=0x7fb990008040 comp rx=0 tx=0).stop 2023-10-16T14:36:51.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.315+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 msgr2=0x7fb99c15c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:51.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.315+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 0x7fb99c15c350 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb994007e40 tx=0x7fb99400e4b0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 shutdown_connections 2023-10-16T14:36:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fb98006b070 0x7fb98006d530 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb99c159950 0x7fb99c15c350 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb99c070b60 0x7fb99c146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 --2- 172.21.15.29:0/1018457472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb99c06c100 0x7fb99c154790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 >> 172.21.15.29:0/1018457472 conn(0x7fb99c0ce6a0 msgr2=0x7fb99c070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.316+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 shutdown_connections 2023-10-16T14:36:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.317+0000 7fb9a1ed7640 1 -- 172.21.15.29:0/1018457472 wait complete. 2023-10-16T14:36:51.341 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:36:51.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:36:51.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:36:51.553 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:36:51.554 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:51.554 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:51.609 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:51.610 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:51.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:51.858 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:51.855+0000 7fa049e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:36:51.892 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:51.888+0000 7fa1a840f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:36:51.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.907+0000 7f1d3c075640 1 Processor -- start 2023-10-16T14:36:51.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.907+0000 7f1d3c075640 1 -- start start 2023-10-16T14:36:51.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.908+0000 7f1d3c075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d3415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.908+0000 7f1d3c075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1d340ce390 0x7f1d34070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.908+0000 7f1d3c075640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3415ed60 con 0x7f1d3415aaa0 2023-10-16T14:36:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.908+0000 7f1d3c075640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3406b810 con 0x7f1d340ce390 2023-10-16T14:36:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.908+0000 7f1d3c075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340707e0 con 0x7f1d3415ae80 2023-10-16T14:36:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1d340ce390 0x7f1d34070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:51.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1d340ce390 0x7f1d34070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58300/0 (socket says 172.21.15.29:58300) 2023-10-16T14:36:51.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 -- 172.21.15.29:0/3071075729 learned_addr learned my addr 172.21.15.29:0/3071075729 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:51.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d39e11640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d3415aaa0 0x7f1d340cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49136/0 (socket says 172.21.15.29:49136) 2023-10-16T14:36:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d39610640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d3415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 -- 172.21.15.29:0/3071075729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 msgr2=0x7f1d3415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d3415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 -- 172.21.15.29:0/3071075729 >> v1:172.21.15.29:6789/0 conn(0x7f1d3415aaa0 legacy=0x7f1d340cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d3a612640 1 -- 172.21.15.29:0/3071075729 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3415ef90 con 0x7f1d340ce390 2023-10-16T14:36:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.909+0000 7f1d39610640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d3415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.910+0000 7f1d3a612640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d340ce390 0x7f1d34070f50 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009e50 tx=0x7f1d2c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67dc715953bb0181 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d38e0f640 1 -- 172.21.15.29:0/3071075729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c02cb00 con 0x7f1d340ce390 2023-10-16T14:36:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d38e0f640 1 -- 172.21.15.29:0/3071075729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c02cc60 con 0x7f1d340ce390 2023-10-16T14:36:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d38e0f640 1 -- 172.21.15.29:0/3071075729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0343f0 con 0x7f1d340ce390 2023-10-16T14:36:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d3c075640 1 -- 172.21.15.29:0/3071075729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d340ce390 msgr2=0x7f1d34070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d3c075640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d340ce390 0x7f1d34070f50 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009e50 tx=0x7f1d2c02bf60 comp rx=0 tx=0).stop 2023-10-16T14:36:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d3c075640 1 -- 172.21.15.29:0/3071075729 shutdown_connections 2023-10-16T14:36:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d3c075640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d340ce390 0x7f1d34070f50 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d3c075640 1 --2- 172.21.15.29:0/3071075729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d3415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.911+0000 7f1d3c075640 1 -- 172.21.15.29:0/3071075729 >> 172.21.15.29:0/3071075729 conn(0x7f1d340cb410 msgr2=0x7f1d340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.912+0000 7f1d3c075640 1 -- 172.21.15.29:0/3071075729 shutdown_connections 2023-10-16T14:36:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.912+0000 7f1d3c075640 1 -- 172.21.15.29:0/3071075729 wait complete. 2023-10-16T14:36:51.920 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:51.912+0000 7fa049e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:36:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.913+0000 7f1d3c075640 1 Processor -- start 2023-10-16T14:36:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.914+0000 7f1d3c075640 1 -- start start 2023-10-16T14:36:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.914+0000 7f1d3c075640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d340ce390 0x7f1d340ad120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d3c075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 0x7f1d341f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d3c075640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d340c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d3c075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340c9350 con 0x7f1d3415aaa0 2023-10-16T14:36:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d3c075640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341f24b0 con 0x7f1d340ce390 2023-10-16T14:36:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d3c075640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340c4c40 con 0x7f1d3415ae80 2023-10-16T14:36:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 0x7f1d341f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 0x7f1d341f1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58308/0 (socket says 172.21.15.29:58308) 2023-10-16T14:36:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 -- 172.21.15.29:0/349523613 learned_addr learned my addr 172.21.15.29:0/349523613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 -- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 msgr2=0x7f1d340c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d340c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 -- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d340ce390 msgr2=0x7f1d340ad120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d340ce390 0x7f1d340ad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.915+0000 7f1d39610640 1 -- 172.21.15.29:0/349523613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c009b30 con 0x7f1d3415aaa0 2023-10-16T14:36:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.916+0000 7f1d39610640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 0x7f1d341f1f70 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c7f0 tx=0x7f1d2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.916+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d24007a30 con 0x7f1d3415aaa0 2023-10-16T14:36:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.916+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d24007b90 con 0x7f1d3415aaa0 2023-10-16T14:36:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.916+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400ce40 con 0x7f1d3415aaa0 2023-10-16T14:36:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.916+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d340c4f20 con 0x7f1d3415aaa0 2023-10-16T14:36:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.916+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d340c8fe0 con 0x7f1d3415aaa0 2023-10-16T14:36:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.917+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3404e160 con 0x7f1d3415aaa0 2023-10-16T14:36:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.918+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f1d240113f0 con 0x7f1d3415aaa0 2023-10-16T14:36:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.919+0000 7f1d2a7fc640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 0x7f1d1806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.919+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1d24089d60 con 0x7f1d3415aaa0 2023-10-16T14:36:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.919+0000 7f1d39e11640 1 -- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 msgr2=0x7f1d1806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.919+0000 7f1d39e11640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 0x7f1d1806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:51.924+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1d24054900 con 0x7f1d3415aaa0 2023-10-16T14:36:51.945 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:51.941+0000 7fa1a840f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:36:52.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.116+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d340cdc60 con 0x7f1d3415aaa0 2023-10-16T14:36:52.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.119+0000 7f1d2a7fc640 1 -- 172.21.15.29:0/349523613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+17079 (secure 0 0 0) 0x7f1d2401f090 con 0x7f1d3415aaa0 2023-10-16T14:36:52.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:52.123 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":143,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:42.416458+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2774967038":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2333704669":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1712463277":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/782892922":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6833/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6832/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1119524146":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1398880520":"2023-10-17T14:36:42.416389+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:52.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.120+0000 7f1d39e11640 1 -- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 msgr2=0x7f1d1806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:52.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.120+0000 7f1d39e11640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 0x7f1d1806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.121+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 msgr2=0x7f1d1806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.121+0000 7f1d3c075640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 0x7f1d1806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 msgr2=0x7f1d341f1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 0x7f1d341f1f70 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c7f0 tx=0x7f1d2400cbf0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 shutdown_connections 2023-10-16T14:36:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f1d1806b020 0x7f1d1806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3415aaa0 0x7f1d341f1f70 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d3415ae80 0x7f1d340c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 --2- 172.21.15.29:0/349523613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d340ce390 0x7f1d340ad120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 >> 172.21.15.29:0/349523613 conn(0x7f1d340cb410 msgr2=0x7f1d340b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 shutdown_connections 2023-10-16T14:36:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.122+0000 7f1d3c075640 1 -- 172.21.15.29:0/349523613 wait complete. 2023-10-16T14:36:52.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:36:25.153525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:36:26.216898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:36:52.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:36:52.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.354+0000 7fc4db047640 1 Processor -- start 2023-10-16T14:36:52.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.355+0000 7fc4db047640 1 -- start start 2023-10-16T14:36:52.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4db047640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4d40b4540 0x7fc4d4164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4db047640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d415e410 con 0x7fc4d4159c00 2023-10-16T14:36:52.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4db047640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d40b4c00 con 0x7fc4d40cad80 2023-10-16T14:36:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4db047640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d40b49f0 con 0x7fc4d40b4540 2023-10-16T14:36:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4da045640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc4d4159c00 0x7fc4d415d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49152/0 (socket says 172.21.15.29:49152) 2023-10-16T14:36:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4da045640 1 -- 172.21.15.29:0/2586148022 learned_addr learned my addr 172.21.15.29:0/2586148022 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.356+0000 7fc4da846640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4d40b4540 0x7fc4d4164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.357+0000 7fc4da846640 1 -- 172.21.15.29:0/2586148022 >> v1:172.21.15.170:6789/0 conn(0x7fc4d40cad80 legacy=0x7fc4d415dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.357+0000 7fc4da846640 1 -- 172.21.15.29:0/2586148022 >> v1:172.21.15.29:6789/0 conn(0x7fc4d4159c00 legacy=0x7fc4d415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.357+0000 7fc4da846640 1 -- 172.21.15.29:0/2586148022 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d40c9fc0 con 0x7fc4d40b4540 2023-10-16T14:36:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.358+0000 7fc4da846640 1 --2- 172.21.15.29:0/2586148022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d4164920 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009d90 tx=0x7fc4cc02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5373d6256612738a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.358+0000 7fc4d9043640 1 -- 172.21.15.29:0/2586148022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc02ee70 con 0x7fc4d40b4540 2023-10-16T14:36:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.358+0000 7fc4d9043640 1 -- 172.21.15.29:0/2586148022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4cc036440 con 0x7fc4d40b4540 2023-10-16T14:36:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.358+0000 7fc4d9043640 1 -- 172.21.15.29:0/2586148022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0367d0 con 0x7fc4d40b4540 2023-10-16T14:36:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 -- 172.21.15.29:0/2586148022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 msgr2=0x7fc4d4164920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 --2- 172.21.15.29:0/2586148022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d4164920 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009d90 tx=0x7fc4cc02e0c0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 -- 172.21.15.29:0/2586148022 shutdown_connections 2023-10-16T14:36:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 --2- 172.21.15.29:0/2586148022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d4164920 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 -- 172.21.15.29:0/2586148022 >> 172.21.15.29:0/2586148022 conn(0x7fc4d40b0a50 msgr2=0x7fc4d40b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 -- 172.21.15.29:0/2586148022 shutdown_connections 2023-10-16T14:36:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.359+0000 7fc4db047640 1 -- 172.21.15.29:0/2586148022 wait complete. 2023-10-16T14:36:52.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.361+0000 7fc4db047640 1 Processor -- start 2023-10-16T14:36:52.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.361+0000 7fc4db047640 1 -- start start 2023-10-16T14:36:52.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4db047640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d41f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4db047640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4d40cad80 0x7fc4d415b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4db047640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4159c00 0x7fc4d415b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4db047640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d41f5b80 con 0x7fc4d40b4540 2023-10-16T14:36:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4db047640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d415cc90 con 0x7fc4d4159c00 2023-10-16T14:36:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4db047640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4158200 con 0x7fc4d40cad80 2023-10-16T14:36:52.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4da045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d41f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4da045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d41f5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58314/0 (socket says 172.21.15.29:58314) 2023-10-16T14:36:52.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4da045640 1 -- 172.21.15.29:0/3243168031 learned_addr learned my addr 172.21.15.29:0/3243168031 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:52.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.362+0000 7fc4d9844640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4d40cad80 0x7fc4d415b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da846640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4159c00 0x7fc4d415b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da045640 1 -- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4d40cad80 msgr2=0x7fc4d415b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da045640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4d40cad80 0x7fc4d415b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da045640 1 -- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4159c00 msgr2=0x7fc4d415b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da045640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4159c00 0x7fc4d415b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da045640 1 -- 172.21.15.29:0/3243168031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc009a70 con 0x7fc4d40b4540 2023-10-16T14:36:52.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da846640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4159c00 0x7fc4d415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:52.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4da045640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d41f5640 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c4e0 tx=0x7fc4d000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:52.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0005ce0 con 0x7fc4d40b4540 2023-10-16T14:36:52.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0005e40 con 0x7fc4d40b4540 2023-10-16T14:36:52.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0007270 con 0x7fc4d40b4540 2023-10-16T14:36:52.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.363+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d415bbf0 con 0x7fc4d40b4540 2023-10-16T14:36:52.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.364+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d415c1b0 con 0x7fc4d40b4540 2023-10-16T14:36:52.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.365+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d404e9b0 con 0x7fc4d40b4540 2023-10-16T14:36:52.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.366+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fc4d00073d0 con 0x7fc4d40b4540 2023-10-16T14:36:52.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.367+0000 7fc4caffd640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc4b806b020 0x7fc4b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.367+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fc4d0088dd0 con 0x7fc4d40b4540 2023-10-16T14:36:52.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.367+0000 7fc4d9844640 1 -- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc4b806b020 msgr2=0x7fc4b806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:52.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.367+0000 7fc4d9844640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc4b806b020 0x7fc4b806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:52.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.371+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc4d0053bc0 con 0x7fc4d40b4540 2023-10-16T14:36:52.393 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.389+0000 7fa1a840f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:36:52.399 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.396+0000 7fa049e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:36:52.445 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.442+0000 7fa1a840f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:36:52.454 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.452+0000 7fa049e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:36:52.495 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.491+0000 7fa1a840f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:36:52.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.505+0000 7fa049e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:36:52.556 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.553+0000 7fa1a840f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:36:52.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.564+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4d409ca40 con 0x7fc4d40b4540 2023-10-16T14:36:52.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.564+0000 7fc4caffd640 1 -- 172.21.15.29:0/3243168031 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4d008a0a0 con 0x7fc4d40b4540 2023-10-16T14:36:52.568 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:36:52.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.567+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc4b806b020 msgr2=0x7fc4b806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:52.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.567+0000 7fc4db047640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc4b806b020 0x7fc4b806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.567+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 msgr2=0x7fc4d41f5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.567+0000 7fc4db047640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d41f5640 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c4e0 tx=0x7fc4d000c9a0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.567+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 shutdown_connections 2023-10-16T14:36:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.567+0000 7fc4db047640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc4b806b020 0x7fc4b806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.568+0000 7fc4db047640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc4d4159c00 0x7fc4d415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.568+0000 7fc4db047640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc4d40cad80 0x7fc4d415b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.568+0000 7fc4db047640 1 --2- 172.21.15.29:0/3243168031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4d40b4540 0x7fc4d41f5640 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.568+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 >> 172.21.15.29:0/3243168031 conn(0x7fc4d40b0a50 msgr2=0x7fc4d40b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.568+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 shutdown_connections 2023-10-16T14:36:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.568+0000 7fc4db047640 1 -- 172.21.15.29:0/3243168031 wait complete. 2023-10-16T14:36:52.573 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.571+0000 7fa049e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:36:52.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:36:52.605 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.601+0000 7fa1a840f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:36:52.627 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.624+0000 7fa049e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:36:52.715 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.711+0000 7fa1a840f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:36:52.745 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.743+0000 7fa049e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:36:52.776 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.773+0000 7fa1a840f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:52.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.804+0000 7fc705ded640 1 Processor -- start 2023-10-16T14:36:52.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.804+0000 7fc705ded640 1 -- start start 2023-10-16T14:36:52.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.805+0000 7fc705ded640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 0x7fc7000d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.805+0000 7fc705ded640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70015d150 con 0x7fc7000cded0 2023-10-16T14:36:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.805+0000 7fc705ded640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7000d1180 con 0x7fc7000b8480 2023-10-16T14:36:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.805+0000 7fc705ded640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000d1460 con 0x7fc70015cd70 2023-10-16T14:36:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.806+0000 7fc7055ec640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc7000cded0 0x7fc7000b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49164/0 (socket says 172.21.15.29:49164) 2023-10-16T14:36:52.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.806+0000 7fc7055ec640 1 -- 172.21.15.29:0/1964071421 learned_addr learned my addr 172.21.15.29:0/1964071421 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.806+0000 7fc704deb640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 0x7fc7000d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.806+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290237941 0 0) 0x7fc70015d150 con 0x7fc7000cded0 2023-10-16T14:36:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.806+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0003520 con 0x7fc7000cded0 2023-10-16T14:36:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209088569 0 0) 0x7fc6f0003520 con 0x7fc7000cded0 2023-10-16T14:36:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 msgr2=0x7fc7000d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:52.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 0x7fc7000d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 >> v1:172.21.15.170:6789/0 conn(0x7fc7000b8480 legacy=0x7fc7000cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000d1740 con 0x7fc7000cded0 2023-10-16T14:36:52.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc6f80019f0 con 0x7fc7000cded0 2023-10-16T14:36:52.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 >> v1:172.21.15.29:6789/0 conn(0x7fc7000cded0 legacy=0x7fc7000b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 0x7fc6f00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6f0003f70 0x7fc6f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc6f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000d1460 con 0x7fc6f0004d00 2023-10-16T14:36:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f000bac0 con 0x7fc6f0003f70 2023-10-16T14:36:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f000bc80 con 0x7fc70015cd70 2023-10-16T14:36:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.807+0000 7fc7055ec640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc6f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc704deb640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 0x7fc6f00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc6fffff640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6f0003f70 0x7fc6f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc7055ec640 1 -- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 msgr2=0x7fc6f00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:52.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc7055ec640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc70015cd70 0x7fc6f00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc7055ec640 1 -- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6f0003f70 msgr2=0x7fc6f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc7055ec640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6f0003f70 0x7fc6f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.808+0000 7fc7055ec640 1 -- 172.21.15.29:0/1964071421 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f0003520 con 0x7fc6f0004d00 2023-10-16T14:36:52.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.809+0000 7fc7055ec640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc6f0005180 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800a760 tx=0x7fc6f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd56a423110d72c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.809+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8034630 con 0x7fc6f0004d00 2023-10-16T14:36:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.809+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f802f070 con 0x7fc6f0004d00 2023-10-16T14:36:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.809+0000 7fc6ff7fe640 1 -- 172.21.15.29:0/1964071421 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80349f0 con 0x7fc6f0004d00 2023-10-16T14:36:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc705ded640 1 -- 172.21.15.29:0/1964071421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 msgr2=0x7fc6f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc705ded640 1 --2- 172.21.15.29:0/1964071421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc6f0005180 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800a760 tx=0x7fc6f00057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc7055ec640 1 -- 172.21.15.29:0/1964071421 reap_dead start 2023-10-16T14:36:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc705ded640 1 -- 172.21.15.29:0/1964071421 shutdown_connections 2023-10-16T14:36:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc705ded640 1 -- 172.21.15.29:0/1964071421 >> 172.21.15.29:0/1964071421 conn(0x7fc7000cf570 msgr2=0x7fc7000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:52.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc705ded640 1 -- 172.21.15.29:0/1964071421 shutdown_connections 2023-10-16T14:36:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.810+0000 7fc705ded640 1 -- 172.21.15.29:0/1964071421 wait complete. 2023-10-16T14:36:52.820 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.811+0000 7fa049e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:52.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.812+0000 7fc705ded640 1 Processor -- start 2023-10-16T14:36:52.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.812+0000 7fc705ded640 1 -- start start 2023-10-16T14:36:52.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc705ded640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc700153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc705ded640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc700152850 0x7fc700151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc705ded640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc700156ea0 0x7fc700157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc705ded640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000c9bc0 con 0x7fc6f0004d00 2023-10-16T14:36:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc705ded640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000c9ea0 con 0x7fc700156ea0 2023-10-16T14:36:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc705ded640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000ca180 con 0x7fc700152850 2023-10-16T14:36:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc704deb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc700153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc704deb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc700153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58334/0 (socket says 172.21.15.29:58334) 2023-10-16T14:36:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.814+0000 7fc704deb640 1 -- 172.21.15.29:0/2792345585 learned_addr learned my addr 172.21.15.29:0/2792345585 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc7055ec640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc700156ea0 0x7fc700157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc6fffff640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc700152850 0x7fc700151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc704deb640 1 -- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc700152850 msgr2=0x7fc700151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc704deb640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc700152850 0x7fc700151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc704deb640 1 -- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc700156ea0 msgr2=0x7fc700157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc704deb640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc700156ea0 0x7fc700157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc704deb640 1 -- 172.21.15.29:0/2792345585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000ca460 con 0x7fc6f0004d00 2023-10-16T14:36:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc6fffff640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc700152850 0x7fc700151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.815+0000 7fc704deb640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc700153740 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4011600 tx=0x7fc6f4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.816+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f400f920 con 0x7fc6f0004d00 2023-10-16T14:36:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.816+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f400fa80 con 0x7fc6f0004d00 2023-10-16T14:36:52.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.816+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7000ca740 con 0x7fc6f0004d00 2023-10-16T14:36:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.816+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f400fdf0 con 0x7fc6f0004d00 2023-10-16T14:36:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.816+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7001524c0 con 0x7fc6f0004d00 2023-10-16T14:36:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.818+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70004e9b0 con 0x7fc6f0004d00 2023-10-16T14:36:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.820+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fc6f4015460 con 0x7fc6f0004d00 2023-10-16T14:36:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.821+0000 7fc6fd7fa640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc6e406b050 0x7fc6e406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.821+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fc6f408a430 con 0x7fc6f0004d00 2023-10-16T14:36:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.821+0000 7fc6fffff640 1 -- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc6e406b050 msgr2=0x7fc6e406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:52.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.821+0000 7fc6fffff640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc6e406b050 0x7fc6e406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:52.829 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.825+0000 7fa1a840f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:36:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:52.826+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc6f4055040 con 0x7fc6f0004d00 2023-10-16T14:36:52.871 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.869+0000 7fa049e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:36:52.886 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.883+0000 7fa1a840f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:36:52.932 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.930+0000 7fa049e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:36:52.936 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:52.933+0000 7fa1a840f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:36:52.986 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:52.983+0000 7fa049e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:36:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.017+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7000d1d00 con 0x7fc6f0004d00 2023-10-16T14:36:53.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.018+0000 7fc6fd7fa640 1 -- 172.21.15.29:0/2792345585 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc6f4058660 con 0x7fc6f0004d00 2023-10-16T14:36:53.021 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:36:53.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.020+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc6e406b050 msgr2=0x7fc6e406d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:53.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.020+0000 7fc705ded640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc6e406b050 0x7fc6e406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.020+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 msgr2=0x7fc700153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.020+0000 7fc705ded640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc700153740 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4011600 tx=0x7fc6f4011ac0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.020+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 shutdown_connections 2023-10-16T14:36:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fc6e406b050 0x7fc6e406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc700156ea0 0x7fc700157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc700152850 0x7fc700151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 --2- 172.21.15.29:0/2792345585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6f0004d00 0x7fc700153740 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 >> 172.21.15.29:0/2792345585 conn(0x7fc7000cf570 msgr2=0x7fc7000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:53.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 shutdown_connections 2023-10-16T14:36:53.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.021+0000 7fc705ded640 1 -- 172.21.15.29:0/2792345585 wait complete. 2023-10-16T14:36:53.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:36:53.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.303+0000 7f8cf0c90640 1 Processor -- start 2023-10-16T14:36:53.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.303+0000 7f8cf0c90640 1 -- start start 2023-10-16T14:36:53.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.304+0000 7f8cf0c90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cec0cf710 0x7f8cec0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.304+0000 7f8cf0c90640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8cec0b6cd0 0x7f8cec0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.304+0000 7f8cf0c90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0d3980 con 0x7f8cec0cfaf0 2023-10-16T14:36:53.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cf0c90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0d3c60 con 0x7f8cec0cf710 2023-10-16T14:36:53.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cf0c90640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0ce630 con 0x7f8cec0b6cd0 2023-10-16T14:36:53.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cea59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cec0cf710 0x7f8cec0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cea59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cec0cf710 0x7f8cec0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58350/0 (socket says 172.21.15.29:58350) 2023-10-16T14:36:53.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cea59c640 1 -- 172.21.15.29:0/1325142629 learned_addr learned my addr 172.21.15.29:0/1325142629 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:53.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8ce3fff640 1 --1- 172.21.15.29:0/1325142629 >> v1:172.21.15.29:6789/0 conn(0x7f8cec0cfaf0 0x7f8cec0d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49178/0 (socket says 172.21.15.29:49178) 2023-10-16T14:36:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cead9d640 1 --2- 172.21.15.29:0/1325142629 >> v2:172.21.15.170:3300/0 conn(0x7f8cec0b6cd0 0x7f8cec0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cea59c640 1 -- 172.21.15.29:0/1325142629 >> v2:172.21.15.170:3300/0 conn(0x7f8cec0b6cd0 msgr2=0x7f8cec0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cea59c640 1 --2- 172.21.15.29:0/1325142629 >> v2:172.21.15.170:3300/0 conn(0x7f8cec0b6cd0 0x7f8cec0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8ce9d9b640 1 -- 172.21.15.29:0/1325142629 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273929647 0 0) 0x7f8cec0d3980 con 0x7f8cec0cfaf0 2023-10-16T14:36:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.305+0000 7f8cea59c640 1 -- 172.21.15.29:0/1325142629 >> v1:172.21.15.29:6789/0 conn(0x7f8cec0cfaf0 legacy=0x7f8cec0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.306+0000 7f8cea59c640 1 -- 172.21.15.29:0/1325142629 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0ce910 con 0x7f8cec0cf710 2023-10-16T14:36:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.307+0000 7f8cea59c640 1 --2- 172.21.15.29:0/1325142629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec0b2d10 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0b5890 tx=0x7f8cdc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93035417b300b7a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.307+0000 7f8ce9d9b640 1 -- 172.21.15.29:0/1325142629 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc004230 con 0x7f8cec0cf710 2023-10-16T14:36:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.307+0000 7f8ce9d9b640 1 -- 172.21.15.29:0/1325142629 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc004390 con 0x7f8cec0cf710 2023-10-16T14:36:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.307+0000 7f8ce9d9b640 1 -- 172.21.15.29:0/1325142629 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc003810 con 0x7f8cec0cf710 2023-10-16T14:36:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 -- 172.21.15.29:0/1325142629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 msgr2=0x7f8cec0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/1325142629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec0b2d10 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0b5890 tx=0x7f8cdc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 -- 172.21.15.29:0/1325142629 shutdown_connections 2023-10-16T14:36:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/1325142629 >> v2:172.21.15.170:3300/0 conn(0x7f8cec0b6cd0 0x7f8cec0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/1325142629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec0b2d10 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 -- 172.21.15.29:0/1325142629 >> 172.21.15.29:0/1325142629 conn(0x7f8cec0d2cc0 msgr2=0x7f8cec0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 -- 172.21.15.29:0/1325142629 shutdown_connections 2023-10-16T14:36:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.308+0000 7f8cf0c90640 1 -- 172.21.15.29:0/1325142629 wait complete. 2023-10-16T14:36:53.315 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.310+0000 7fa1a840f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:36:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.311+0000 7f8cf0c90640 1 Processor -- start 2023-10-16T14:36:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.311+0000 7f8cf0c90640 1 -- start start 2023-10-16T14:36:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.312+0000 7f8cf0c90640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cec0b6cd0 0x7f8cec15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.312+0000 7f8cf0c90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.312+0000 7f8cf0c90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cec0cfaf0 0x7f8cec0cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8cf0c90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec15aeb0 con 0x7f8cec0cf710 2023-10-16T14:36:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8cf0c90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec15bcd0 con 0x7f8cec0cfaf0 2023-10-16T14:36:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8cf0c90640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec15dd50 con 0x7f8cec0b6cd0 2023-10-16T14:36:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58356/0 (socket says 172.21.15.29:58356) 2023-10-16T14:36:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 -- 172.21.15.29:0/3878008760 learned_addr learned my addr 172.21.15.29:0/3878008760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8cead9d640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cec0cfaf0 0x7f8cec0cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 -- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cec0b6cd0 msgr2=0x7f8cec15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cec0b6cd0 0x7f8cec15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 -- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cec0cfaf0 msgr2=0x7f8cec0cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cec0cfaf0 0x7f8cec0cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.313+0000 7f8ce3fff640 1 -- 172.21.15.29:0/3878008760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc009460 con 0x7f8cec0cf710 2023-10-16T14:36:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.314+0000 7f8ce3fff640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec159cd0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400c4d0 tx=0x7f8cd400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.316+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd40074e0 con 0x7f8cec0cf710 2023-10-16T14:36:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.316+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd4007660 con 0x7f8cec0cf710 2023-10-16T14:36:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.316+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4004a20 con 0x7f8cec0cf710 2023-10-16T14:36:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.316+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec0b0ba0 con 0x7f8cec0cf710 2023-10-16T14:36:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.317+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec15b3a0 con 0x7f8cec0cf710 2023-10-16T14:36:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.318+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec15be40 con 0x7f8cec0cf710 2023-10-16T14:36:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.319+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f8cd4004b80 con 0x7f8cec0cf710 2023-10-16T14:36:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.320+0000 7f8ce2ffd640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f8ccc06b050 0x7f8ccc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.320+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8cd4088c80 con 0x7f8cec0cf710 2023-10-16T14:36:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.321+0000 7f8cea59c640 1 -- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f8ccc06b050 msgr2=0x7f8ccc06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.321+0000 7f8cea59c640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f8ccc06b050 0x7f8ccc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.324+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8cd40538b0 con 0x7f8cec0cf710 2023-10-16T14:36:53.399 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:53.396+0000 7fa049e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:36:53.474 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.471+0000 7fa1a840f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:36:53.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.514+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8cec04e160 con 0x7f8cec0cf710 2023-10-16T14:36:53.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.515+0000 7f8ce2ffd640 1 -- 172.21.15.29:0/3878008760 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8cd4056ed0 con 0x7f8cec0cf710 2023-10-16T14:36:53.518 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:36:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f8ccc06b050 msgr2=0x7f8ccc06d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f8ccc06b050 0x7f8ccc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 msgr2=0x7f8cec159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec159cd0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400c4d0 tx=0x7f8cd400c990 comp rx=0 tx=0).stop 2023-10-16T14:36:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 shutdown_connections 2023-10-16T14:36:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f8ccc06b050 0x7f8ccc06d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cec0cf710 0x7f8cec159cd0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cec0cfaf0 0x7f8cec0cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 --2- 172.21.15.29:0/3878008760 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cec0b6cd0 0x7f8cec15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 >> 172.21.15.29:0/3878008760 conn(0x7f8cec0d2cc0 msgr2=0x7f8cec0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 shutdown_connections 2023-10-16T14:36:53.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.518+0000 7f8cf0c90640 1 -- 172.21.15.29:0/3878008760 wait complete. 2023-10-16T14:36:53.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:36:53.571 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:53.568+0000 7fa049e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:36:53.582 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.579+0000 7fa1a840f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:36:53.632 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.628+0000 7fa1a840f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:36:53.687 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:53.684+0000 7fa049e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:36:53.738 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.734+0000 7fa1a840f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:53.740 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:53.738+0000 7fa049e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:36:53.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.743+0000 7f55c65a5640 1 Processor -- start 2023-10-16T14:36:53.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.744+0000 7f55c65a5640 1 -- start start 2023-10-16T14:36:53.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.745+0000 7f55c65a5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.745+0000 7f55c65a5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55c0070c10 0x7f55c00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.745+0000 7f55c65a5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c0161740 con 0x7f55c00cf5f0 2023-10-16T14:36:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.745+0000 7f55c65a5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c00cdbd0 con 0x7f55c0070c10 2023-10-16T14:36:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.745+0000 7f55c65a5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c006b0f0 con 0x7f55c0070830 2023-10-16T14:36:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55c0070c10 0x7f55c00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55c0070c10 0x7f55c00cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58368/0 (socket says 172.21.15.29:58368) 2023-10-16T14:36:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 -- 172.21.15.29:0/2256604060 learned_addr learned my addr 172.21.15.29:0/2256604060 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:53.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55bffff640 1 --2- 172.21.15.29:0/2256604060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 -- 172.21.15.29:0/2256604060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 msgr2=0x7f55c006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 --2- 172.21.15.29:0/2256604060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 -- 172.21.15.29:0/2256604060 >> v1:172.21.15.170:6789/0 conn(0x7f55c00cf5f0 legacy=0x7f55c006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.746+0000 7f55c4b42640 1 -- 172.21.15.29:0/2256604060 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c01619b0 con 0x7f55c0070c10 2023-10-16T14:36:53.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.748+0000 7f55c4b42640 1 --2- 172.21.15.29:0/2256604060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c0070c10 0x7f55c00cd690 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009e50 tx=0x7f55b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f3c812d5dba4de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:53.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.748+0000 7f55bf7fe640 1 -- 172.21.15.29:0/2256604060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b802c9e0 con 0x7f55c0070c10 2023-10-16T14:36:53.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.748+0000 7f55bf7fe640 1 -- 172.21.15.29:0/2256604060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b802cb40 con 0x7f55c0070c10 2023-10-16T14:36:53.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.748+0000 7f55bf7fe640 1 -- 172.21.15.29:0/2256604060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80343f0 con 0x7f55c0070c10 2023-10-16T14:36:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 -- 172.21.15.29:0/2256604060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c0070c10 msgr2=0x7f55c00cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 --2- 172.21.15.29:0/2256604060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c0070c10 0x7f55c00cd690 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009e50 tx=0x7f55b802be40 comp rx=0 tx=0).stop 2023-10-16T14:36:53.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 -- 172.21.15.29:0/2256604060 shutdown_connections 2023-10-16T14:36:53.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 --2- 172.21.15.29:0/2256604060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c0070c10 0x7f55c00cd690 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 --2- 172.21.15.29:0/2256604060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 -- 172.21.15.29:0/2256604060 >> 172.21.15.29:0/2256604060 conn(0x7f55c00cfac0 msgr2=0x7f55c00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:53.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.749+0000 7f55c65a5640 1 -- 172.21.15.29:0/2256604060 shutdown_connections 2023-10-16T14:36:53.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.750+0000 7f55c65a5640 1 -- 172.21.15.29:0/2256604060 wait complete. 2023-10-16T14:36:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.752+0000 7f55c65a5640 1 Processor -- start 2023-10-16T14:36:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.752+0000 7f55c65a5640 1 -- start start 2023-10-16T14:36:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.753+0000 7f55c65a5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c014deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.753+0000 7f55c65a5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c0070c10 0x7f55c00af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.753+0000 7f55c65a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 0x7f55c00afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.753+0000 7f55c65a5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c00c4680 con 0x7f55c00cf5f0 2023-10-16T14:36:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.753+0000 7f55c65a5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c00c6230 con 0x7f55c0070c10 2023-10-16T14:36:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.753+0000 7f55c65a5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c00c5620 con 0x7f55c0070830 2023-10-16T14:36:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 0x7f55c00afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 0x7f55c00afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58384/0 (socket says 172.21.15.29:58384) 2023-10-16T14:36:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 -- 172.21.15.29:0/1194224523 learned_addr learned my addr 172.21.15.29:0/1194224523 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55b7fff640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c0070c10 0x7f55c00af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55bffff640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c014deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 -- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 msgr2=0x7f55c014deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c014deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 -- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c0070c10 msgr2=0x7f55c00af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c0070c10 0x7f55c00af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55c4b42640 1 -- 172.21.15.29:0/1194224523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8009b30 con 0x7f55c00cf5f0 2023-10-16T14:36:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.754+0000 7f55bffff640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.755+0000 7f55c4b42640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 0x7f55c00afa60 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009e50 tx=0x7f55b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.755+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b803e450 con 0x7f55c00cf5f0 2023-10-16T14:36:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.755+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b803e5f0 con 0x7f55c00cf5f0 2023-10-16T14:36:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.755+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b803e9a0 con 0x7f55c00cf5f0 2023-10-16T14:36:53.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.755+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c00c5900 con 0x7f55c00cf5f0 2023-10-16T14:36:53.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.755+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c00c5ec0 con 0x7f55c00cf5f0 2023-10-16T14:36:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.756+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c00c63c0 con 0x7f55c00cf5f0 2023-10-16T14:36:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.757+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f55b803eb00 con 0x7f55c00cf5f0 2023-10-16T14:36:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.759+0000 7f55bd7fa640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 0x7f55a006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.759+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f55b80ac7f0 con 0x7f55c00cf5f0 2023-10-16T14:36:53.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.759+0000 7f55bffff640 1 -- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 msgr2=0x7f55a006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:53.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.759+0000 7f55bffff640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 0x7f55a006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:53.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.763+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f55b8077400 con 0x7f55c00cf5f0 2023-10-16T14:36:53.846 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.843+0000 7fa1a840f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:36:53.859 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:53.856+0000 7fa049e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.960+0000 7f55bffff640 1 -- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 msgr2=0x7f55a006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.960+0000 7f55bffff640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 0x7f55a006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:53.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.968+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f55c004e160 con 0x7f55c00cf5f0 2023-10-16T14:36:53.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.969+0000 7f55bd7fa640 1 -- 172.21.15.29:0/1194224523 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v143) v1 ==== 80+0+477 (secure 0 0 0) 0x7f55b807aa20 con 0x7f55c00cf5f0 2023-10-16T14:36:53.972 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2774967038 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.972 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2333704669 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.972 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1712463277 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/782892922 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/1905872466 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/1905872466 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1119524146 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1398880520 2023-10-17T14:36:42.416389+0000 2023-10-16T14:36:53.974 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T14:36:53.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 msgr2=0x7f55a006d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:53.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 0x7f55a006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 msgr2=0x7f55c00afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:53.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 0x7f55c00afa60 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009e50 tx=0x7f55b8004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:53.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 shutdown_connections 2023-10-16T14:36:53.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f55a006b050 0x7f55a006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c00cf5f0 0x7f55c00afa60 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55c0070c10 0x7f55c00af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 --2- 172.21.15.29:0/1194224523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55c0070830 0x7f55c014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:53.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.971+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 >> 172.21.15.29:0/1194224523 conn(0x7f55c00cfac0 msgr2=0x7f55c00ca940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:53.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.972+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 shutdown_connections 2023-10-16T14:36:53.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:53.972+0000 7f55c65a5640 1 -- 172.21.15.29:0/1194224523 wait complete. 2023-10-16T14:36:53.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:53.972+0000 7fa049e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:36:53.991 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:36:53.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:53.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:36:54.003 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:53.999+0000 7fa1a840f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:54.052 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:54.048+0000 7fa1a840f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:54.134 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T14:36:54.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:36:54.135 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:36:54.148 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:54.146+0000 7fa049e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:36:54.170 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:54.166+0000 7fa1a840f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:36:54.181 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:36:54.181 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:54.205 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:54.200+0000 7fa049e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:36:54.235 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:36:54.236 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:36:54.236 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:54.291 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:36:54.292 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:36:54.292 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:36:54.331 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:54.328+0000 7fa049e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:36:54.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:36:54.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.559+0000 7f0124e1a640 1 Processor -- start 2023-10-16T14:36:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.559+0000 7f0124e1a640 1 -- start start 2023-10-16T14:36:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f0124e1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f012015cd70 0x7f01200d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f0124e1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f01200b8480 0x7f01200cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f0124e1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012015d150 con 0x7f01200d0be0 2023-10-16T14:36:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f0124e1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01200d1180 con 0x7f012015cd70 2023-10-16T14:36:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f0124e1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01200d1460 con 0x7f01200b8480 2023-10-16T14:36:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f011f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f012015cd70 0x7f01200d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f011f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f012015cd70 0x7f01200d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40360/0 (socket says 172.21.15.29:40360) 2023-10-16T14:36:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f011ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f01200d0be0 0x7f01200b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48366/0 (socket says 172.21.15.29:48366) 2023-10-16T14:36:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f011f7fe640 1 -- 172.21.15.29:0/4120303674 learned_addr learned my addr 172.21.15.29:0/4120303674 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.561+0000 7f0116ffd640 1 --2- 172.21.15.29:0/4120303674 >> v2:172.21.15.170:3300/0 conn(0x7f01200b8480 0x7f01200cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.562+0000 7f011f7fe640 1 -- 172.21.15.29:0/4120303674 >> v2:172.21.15.170:3300/0 conn(0x7f01200b8480 msgr2=0x7f01200cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.562+0000 7f011f7fe640 1 --2- 172.21.15.29:0/4120303674 >> v2:172.21.15.170:3300/0 conn(0x7f01200b8480 0x7f01200cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.562+0000 7f011f7fe640 1 -- 172.21.15.29:0/4120303674 >> v1:172.21.15.29:6789/0 conn(0x7f01200d0be0 legacy=0x7f01200b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.562+0000 7f011f7fe640 1 -- 172.21.15.29:0/4120303674 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01200d1740 con 0x7f012015cd70 2023-10-16T14:36:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.563+0000 7f011f7fe640 1 --2- 172.21.15.29:0/4120303674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f012015cd70 0x7f01200d2220 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0120066a00 tx=0x7f011002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222cf9db64dbf0a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.563+0000 7f011effd640 1 -- 172.21.15.29:0/4120303674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110031870 con 0x7f012015cd70 2023-10-16T14:36:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.563+0000 7f011effd640 1 -- 172.21.15.29:0/4120303674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100319d0 con 0x7f012015cd70 2023-10-16T14:36:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.563+0000 7f011effd640 1 -- 172.21.15.29:0/4120303674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110031d60 con 0x7f012015cd70 2023-10-16T14:36:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 -- 172.21.15.29:0/4120303674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f012015cd70 msgr2=0x7f01200d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 --2- 172.21.15.29:0/4120303674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f012015cd70 0x7f01200d2220 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0120066a00 tx=0x7f011002b910 comp rx=0 tx=0).stop 2023-10-16T14:36:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 -- 172.21.15.29:0/4120303674 shutdown_connections 2023-10-16T14:36:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 --2- 172.21.15.29:0/4120303674 >> v2:172.21.15.170:3300/0 conn(0x7f01200b8480 0x7f01200cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 --2- 172.21.15.29:0/4120303674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f012015cd70 0x7f01200d2220 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 -- 172.21.15.29:0/4120303674 >> 172.21.15.29:0/4120303674 conn(0x7f01200cf570 msgr2=0x7f01200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 -- 172.21.15.29:0/4120303674 shutdown_connections 2023-10-16T14:36:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.564+0000 7f0124e1a640 1 -- 172.21.15.29:0/4120303674 wait complete. 2023-10-16T14:36:54.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.565+0000 7f0124e1a640 1 Processor -- start 2023-10-16T14:36:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.566+0000 7f0124e1a640 1 -- start start 2023-10-16T14:36:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.566+0000 7f0124e1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01200b8480 0x7f0120157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.566+0000 7f0124e1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 0x7f012015afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.566+0000 7f0124e1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012015cd70 0x7f01200cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0124e1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0120155ba0 con 0x7f01200d0be0 2023-10-16T14:36:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0124e1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01201581f0 con 0x7f012015cd70 2023-10-16T14:36:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0124e1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01201558c0 con 0x7f01200b8480 2023-10-16T14:36:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 0x7f012015afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 0x7f012015afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40376/0 (socket says 172.21.15.29:40376) 2023-10-16T14:36:54.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 -- 172.21.15.29:0/191784407 learned_addr learned my addr 172.21.15.29:0/191784407 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:54.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f011ffff640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012015cd70 0x7f01200cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:54.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 -- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01200b8480 msgr2=0x7f0120157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:54.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01200b8480 0x7f0120157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 -- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012015cd70 msgr2=0x7f01200cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:54.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012015cd70 0x7f01200cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 -- 172.21.15.29:0/191784407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01100094d0 con 0x7f01200d0be0 2023-10-16T14:36:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f011ffff640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012015cd70 0x7f01200cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.567+0000 7f0116ffd640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 0x7f012015afa0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f010800d620 tx=0x7f010800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.568+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800b620 con 0x7f01200d0be0 2023-10-16T14:36:54.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.568+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010800b780 con 0x7f01200d0be0 2023-10-16T14:36:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.568+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01201571d0 con 0x7f01200d0be0 2023-10-16T14:36:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.568+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800bb30 con 0x7f01200d0be0 2023-10-16T14:36:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.568+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0120157790 con 0x7f01200d0be0 2023-10-16T14:36:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.569+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0120157950 con 0x7f01200d0be0 2023-10-16T14:36:54.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.570+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f010800bc90 con 0x7f01200d0be0 2023-10-16T14:36:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.571+0000 7f011cff9640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 0x7f010006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:54.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.571+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f0108089fd0 con 0x7f01200d0be0 2023-10-16T14:36:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.571+0000 7f011f7fe640 1 -- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 msgr2=0x7f010006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.571+0000 7f011f7fe640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 0x7f010006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:54.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.575+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0108054be0 con 0x7f01200d0be0 2023-10-16T14:36:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.772+0000 7f011f7fe640 1 -- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 msgr2=0x7f010006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.772+0000 7f011f7fe640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 0x7f010006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:36:54.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.803+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0120157030 con 0x7f01200d0be0 2023-10-16T14:36:54.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.804+0000 7f011cff9640 1 -- 172.21.15.29:0/191784407 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f0120157030 con 0x7f01200d0be0 2023-10-16T14:36:54.807 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:54.808 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:36:27.196747+0000","modified":"2023-10-16T14:36:30.291825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1700644562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1700644562},{"type":"v1","addr":"172.21.15.29:6835","nonce":1700644562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T14:36:54.808 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-16T14:36:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.806+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 msgr2=0x7f010006d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.806+0000 7f0124e1a640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 0x7f010006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 msgr2=0x7f012015afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 0x7f012015afa0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f010800d620 tx=0x7f010800dae0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 shutdown_connections 2023-10-16T14:36:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f010006b050 0x7f010006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01200d0be0 0x7f012015afa0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f012015cd70 0x7f01200cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 --2- 172.21.15.29:0/191784407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f01200b8480 0x7f0120157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:54.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 >> 172.21.15.29:0/191784407 conn(0x7f01200cf570 msgr2=0x7f012015fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 shutdown_connections 2023-10-16T14:36:54.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:54.807+0000 7f0124e1a640 1 -- 172.21.15.29:0/191784407 wait complete. 2023-10-16T14:36:54.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:55.010 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.006+0000 7fa1a840f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:36:55.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.009+0000 7f7ead897640 1 Processor -- start 2023-10-16T14:36:55.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.010+0000 7f7ead897640 1 -- start start 2023-10-16T14:36:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.011+0000 7f7ead897640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7ea815aaa0 0x7f7ea815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.011+0000 7f7ead897640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 0x7f7ea8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.011+0000 7f7ead897640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea815ed60 con 0x7f7ea815ae80 2023-10-16T14:36:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.011+0000 7f7ead897640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea806b810 con 0x7f7ea815aaa0 2023-10-16T14:36:55.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.011+0000 7f7ead897640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80707e0 con 0x7f7ea80ce390 2023-10-16T14:36:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.012+0000 7f7ea6ffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7ea815aaa0 0x7f7ea815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.012+0000 7f7ea77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 0x7f7ea8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.012+0000 7f7ea77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 0x7f7ea8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59478/0 (socket says 172.21.15.29:59478) 2023-10-16T14:36:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.012+0000 7f7ea77fe640 1 -- 172.21.15.29:0/1851897991 learned_addr learned my addr 172.21.15.29:0/1851897991 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.013+0000 7f7ea77fe640 1 -- 172.21.15.29:0/1851897991 >> v2:172.21.15.170:3300/0 conn(0x7f7ea815aaa0 msgr2=0x7f7ea815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.013+0000 7f7ea77fe640 1 --2- 172.21.15.29:0/1851897991 >> v2:172.21.15.170:3300/0 conn(0x7f7ea815aaa0 0x7f7ea815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.013+0000 7f7ea77fe640 1 -- 172.21.15.29:0/1851897991 >> v1:172.21.15.170:6789/0 conn(0x7f7ea815ae80 legacy=0x7f7ea80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.013+0000 7f7ea77fe640 1 -- 172.21.15.29:0/1851897991 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea815ef90 con 0x7f7ea80ce390 2023-10-16T14:36:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.014+0000 7f7ea77fe640 1 --2- 172.21.15.29:0/1851897991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 0x7f7ea8070f50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009870 tx=0x7f7ea002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee245fff73e96f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.014+0000 7f7ea5ffb640 1 -- 172.21.15.29:0/1851897991 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0030840 con 0x7f7ea80ce390 2023-10-16T14:36:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.014+0000 7f7ea5ffb640 1 -- 172.21.15.29:0/1851897991 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea00309a0 con 0x7f7ea80ce390 2023-10-16T14:36:55.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.014+0000 7f7ea5ffb640 1 -- 172.21.15.29:0/1851897991 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0030d30 con 0x7f7ea80ce390 2023-10-16T14:36:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 -- 172.21.15.29:0/1851897991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 msgr2=0x7f7ea8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 --2- 172.21.15.29:0/1851897991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 0x7f7ea8070f50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009870 tx=0x7f7ea002b980 comp rx=0 tx=0).stop 2023-10-16T14:36:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 -- 172.21.15.29:0/1851897991 shutdown_connections 2023-10-16T14:36:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 --2- 172.21.15.29:0/1851897991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea80ce390 0x7f7ea8070f50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 --2- 172.21.15.29:0/1851897991 >> v2:172.21.15.170:3300/0 conn(0x7f7ea815aaa0 0x7f7ea815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 -- 172.21.15.29:0/1851897991 >> 172.21.15.29:0/1851897991 conn(0x7f7ea80cb410 msgr2=0x7f7ea80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 -- 172.21.15.29:0/1851897991 shutdown_connections 2023-10-16T14:36:55.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.015+0000 7f7ead897640 1 -- 172.21.15.29:0/1851897991 wait complete. 2023-10-16T14:36:55.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.017+0000 7f7ead897640 1 Processor -- start 2023-10-16T14:36:55.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.017+0000 7f7ead897640 1 -- start start 2023-10-16T14:36:55.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.018+0000 7f7ead897640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ea80ce390 0x7f7ea80c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.018+0000 7f7ead897640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 0x7f7ea814eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.018+0000 7f7ead897640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea815ae80 0x7f7ea80c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.018+0000 7f7ead897640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80c93f0 con 0x7f7ea815aaa0 2023-10-16T14:36:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.018+0000 7f7ead897640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80c5fa0 con 0x7f7ea80ce390 2023-10-16T14:36:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.018+0000 7f7ead897640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80c9080 con 0x7f7ea815ae80 2023-10-16T14:36:55.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 0x7f7ea814eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 0x7f7ea814eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40388/0 (socket says 172.21.15.29:40388) 2023-10-16T14:36:55.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 -- 172.21.15.29:0/3400358333 learned_addr learned my addr 172.21.15.29:0/3400358333 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:55.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea6ffd640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ea80ce390 0x7f7ea80c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea77fe640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea815ae80 0x7f7ea80c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 -- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea815ae80 msgr2=0x7f7ea80c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea815ae80 0x7f7ea80c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 -- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ea80ce390 msgr2=0x7f7ea80c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ea80ce390 0x7f7ea80c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 -- 172.21.15.29:0/3400358333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea00094d0 con 0x7f7ea815aaa0 2023-10-16T14:36:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.019+0000 7f7ea67fc640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 0x7f7ea814eda0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c3f0 tx=0x7f7e9000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:55.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.020+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900052d0 con 0x7f7ea815aaa0 2023-10-16T14:36:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.020+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e90005430 con 0x7f7ea815aaa0 2023-10-16T14:36:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.020+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90003850 con 0x7f7ea815aaa0 2023-10-16T14:36:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.020+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea814f2e0 con 0x7f7ea815aaa0 2023-10-16T14:36:55.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.020+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea80c64b0 con 0x7f7ea815aaa0 2023-10-16T14:36:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.021+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea80c9530 con 0x7f7ea815aaa0 2023-10-16T14:36:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.022+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f7e900039b0 con 0x7f7ea815aaa0 2023-10-16T14:36:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.023+0000 7f7e9ffff640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f7e8806b020 0x7f7e8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.023+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f7e90089000 con 0x7f7ea815aaa0 2023-10-16T14:36:55.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.023+0000 7f7ea6ffd640 1 -- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f7e8806b020 msgr2=0x7f7e8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:55.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.023+0000 7f7ea6ffd640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f7e8806b020 0x7f7e8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:55.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.027+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ea80c9530 con 0x7f7ea815aaa0 2023-10-16T14:36:55.184 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:55.181+0000 7fa049e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:36:55.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.220+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ea80c9530 con 0x7f7ea815aaa0 2023-10-16T14:36:55.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.221+0000 7f7e9ffff640 1 -- 172.21.15.29:0/3400358333 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+17079 (secure 0 0 0) 0x7f7ea80c9530 con 0x7f7ea815aaa0 2023-10-16T14:36:55.224 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:55.224 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":143,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:42.416458+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2774967038":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2333704669":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1712463277":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/782892922":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6833/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6832/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1119524146":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1398880520":"2023-10-17T14:36:42.416389+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:55.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.223+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f7e8806b020 msgr2=0x7f7e8806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:55.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.223+0000 7f7ead897640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f7e8806b020 0x7f7e8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.223+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 msgr2=0x7f7ea814eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.223+0000 7f7ead897640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 0x7f7ea814eda0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c3f0 tx=0x7f7e9000c8b0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 shutdown_connections 2023-10-16T14:36:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7f7e8806b020 0x7f7e8806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ea815aaa0 0x7f7ea814eda0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ea815ae80 0x7f7ea80c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 --2- 172.21.15.29:0/3400358333 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ea80ce390 0x7f7ea80c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 >> 172.21.15.29:0/3400358333 conn(0x7f7ea80cb410 msgr2=0x7f7ea80b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 shutdown_connections 2023-10-16T14:36:55.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.224+0000 7f7ead897640 1 -- 172.21.15.29:0/3400358333 wait complete. 2023-10-16T14:36:55.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:36:25.153525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:36:26.216898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:36:55.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:36:55.304 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.301+0000 7fa1a840f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:36:55.363 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.360+0000 7fa1a840f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:36:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.458+0000 7fe67759e640 1 Processor -- start 2023-10-16T14:36:55.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.458+0000 7fe67759e640 1 -- start start 2023-10-16T14:36:55.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.459+0000 7fe67759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe67815e350 0x7fe6780cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.459+0000 7fe67759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67806fbc0 con 0x7fe67815f090 2023-10-16T14:36:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.459+0000 7fe67759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780c9920 con 0x7fe678070970 2023-10-16T14:36:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.459+0000 7fe67759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67815e730 con 0x7fe67815e350 2023-10-16T14:36:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe67659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe67815e350 0x7fe6780cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe67659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe67815e350 0x7fe6780cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40394/0 (socket says 172.21.15.29:40394) 2023-10-16T14:36:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe67659c640 1 -- 172.21.15.29:0/3448149264 learned_addr learned my addr 172.21.15.29:0/3448149264 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe676d9d640 1 --1- 172.21.15.29:0/3448149264 >> v1:172.21.15.29:6789/0 conn(0x7fe67815f090 0x7fe6780ca9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48376/0 (socket says 172.21.15.29:48376) 2023-10-16T14:36:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe67659c640 1 -- 172.21.15.29:0/3448149264 >> v1:172.21.15.170:6789/0 conn(0x7fe678070970 legacy=0x7fe67815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe67659c640 1 -- 172.21.15.29:0/3448149264 >> v1:172.21.15.29:6789/0 conn(0x7fe67815f090 legacy=0x7fe6780ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.460+0000 7fe67659c640 1 -- 172.21.15.29:0/3448149264 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780c9c00 con 0x7fe67815e350 2023-10-16T14:36:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.461+0000 7fe67659c640 1 --2- 172.21.15.29:0/3448149264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780cdb40 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fe668009870 tx=0x7fe66802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e35484f1f6d693 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.461+0000 7fe67559a640 1 -- 172.21.15.29:0/3448149264 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668031870 con 0x7fe67815e350 2023-10-16T14:36:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67559a640 1 -- 172.21.15.29:0/3448149264 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6680319d0 con 0x7fe67815e350 2023-10-16T14:36:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67559a640 1 -- 172.21.15.29:0/3448149264 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668031d60 con 0x7fe67815e350 2023-10-16T14:36:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 -- 172.21.15.29:0/3448149264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 msgr2=0x7fe6780cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 --2- 172.21.15.29:0/3448149264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780cdb40 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fe668009870 tx=0x7fe66802b910 comp rx=0 tx=0).stop 2023-10-16T14:36:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 -- 172.21.15.29:0/3448149264 shutdown_connections 2023-10-16T14:36:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 --2- 172.21.15.29:0/3448149264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780cdb40 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 -- 172.21.15.29:0/3448149264 >> 172.21.15.29:0/3448149264 conn(0x7fe6780b0a50 msgr2=0x7fe6780b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 -- 172.21.15.29:0/3448149264 shutdown_connections 2023-10-16T14:36:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.462+0000 7fe67759e640 1 -- 172.21.15.29:0/3448149264 wait complete. 2023-10-16T14:36:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.464+0000 7fe67759e640 1 Processor -- start 2023-10-16T14:36:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.464+0000 7fe67759e640 1 -- start start 2023-10-16T14:36:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe67759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe678070970 0x7fe678152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe67759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe67759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe67815f090 0x7fe6781508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe67759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780c3e00 con 0x7fe67815e350 2023-10-16T14:36:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe67759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678151920 con 0x7fe678070970 2023-10-16T14:36:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe67759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780c2060 con 0x7fe67815f090 2023-10-16T14:36:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe675d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe675d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780c1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40402/0 (socket says 172.21.15.29:40402) 2023-10-16T14:36:55.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.465+0000 7fe675d9b640 1 -- 172.21.15.29:0/1146805280 learned_addr learned my addr 172.21.15.29:0/1146805280 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:55.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe67659c640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe678070970 0x7fe678152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe675d9b640 1 -- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe67815f090 msgr2=0x7fe6781508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe675d9b640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe67815f090 0x7fe6781508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe675d9b640 1 -- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe678070970 msgr2=0x7fe678152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe675d9b640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe678070970 0x7fe678152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe675d9b640 1 -- 172.21.15.29:0/1146805280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6680094d0 con 0x7fe67815e350 2023-10-16T14:36:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe675d9b640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780c1b20 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c430 tx=0x7fe66c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c011790 con 0x7fe67815e350 2023-10-16T14:36:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c0118f0 con 0x7fe67815e350 2023-10-16T14:36:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c011ca0 con 0x7fe67815e350 2023-10-16T14:36:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.466+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe678099cd0 con 0x7fe67815e350 2023-10-16T14:36:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.467+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe678150e20 con 0x7fe67815e350 2023-10-16T14:36:55.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.468+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67806e500 con 0x7fe67815e350 2023-10-16T14:36:55.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.469+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fe66c01c460 con 0x7fe67815e350 2023-10-16T14:36:55.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.470+0000 7fe666ffd640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.470+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe66c089a90 con 0x7fe67815e350 2023-10-16T14:36:55.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.470+0000 7fe67659c640 1 -- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe65806b020 msgr2=0x7fe65806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/230171195 2023-10-16T14:36:55.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.470+0000 7fe67659c640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:36:55.477 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.473+0000 7fa1a840f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:36:55.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.479+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe66c0546a0 con 0x7fe67815e350 2023-10-16T14:36:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.490+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 74) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe66c054a70 con 0x7fe67815e350 2023-10-16T14:36:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.491+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe65806b020 msgr2=0x7fe65806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:36:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.491+0000 7fe666ffd640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.517 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:55.514+0000 7fa049e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:36:55.572 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.569+0000 7fa12d2ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.573 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.569+0000 7fa12d2ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.573 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.569+0000 7fa12d2ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.573 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.569+0000 7fa12d2ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.573 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.569+0000 7fa12d2ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.574 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.570+0000 7fa12a2e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.574 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.570+0000 7fa12a2e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.574 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.570+0000 7fa12a2e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.574 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.570+0000 7fa12a2e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.575 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:36:55.570+0000 7fa12a2e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:36:55.581 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:55.578+0000 7fa049e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:36:55.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.687+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe67804e9b0 con 0x7fe67815e350 2023-10-16T14:36:55.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.688+0000 7fe666ffd640 1 -- 172.21.15.29:0/1146805280 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe66c0714f0 con 0x7fe67815e350 2023-10-16T14:36:55.691 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:36:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.690+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 msgr2=0x7fe6780c1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.690+0000 7fe67759e640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780c1b20 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c430 tx=0x7fe66c00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 shutdown_connections 2023-10-16T14:36:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:6832/230171195,v1:172.21.15.170:6833/230171195] conn(0x7fe65806b020 0x7fe65806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe67815e350 0x7fe6780c1b20 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe67815f090 0x7fe6781508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 --2- 172.21.15.29:0/1146805280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe678070970 0x7fe678152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 >> 172.21.15.29:0/1146805280 conn(0x7fe6780b0a50 msgr2=0x7fe678158e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 shutdown_connections 2023-10-16T14:36:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.691+0000 7fe67759e640 1 -- 172.21.15.29:0/1146805280 wait complete. 2023-10-16T14:36:55.702 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:36:55.699+0000 7fa049e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:36:55.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:36:55.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.919+0000 7feab48b3640 1 Processor -- start 2023-10-16T14:36:55.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.919+0000 7feab48b3640 1 -- start start 2023-10-16T14:36:55.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feab48b3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab006c040 0x7feab006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feab48b3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7feab00cda80 0x7feab015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feab48b3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab006c960 con 0x7feab00ad970 2023-10-16T14:36:55.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feab48b3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab015b9d0 con 0x7feab00cda80 2023-10-16T14:36:55.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feab48b3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab015bcb0 con 0x7feab006c040 2023-10-16T14:36:55.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feaa7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feab00ad970 0x7feab015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48388/0 (socket says 172.21.15.29:48388) 2023-10-16T14:36:55.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.921+0000 7feaa7fff640 1 -- 172.21.15.29:0/464486053 learned_addr learned my addr 172.21.15.29:0/464486053 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:55.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.922+0000 7feaaf59e640 1 --2- 172.21.15.29:0/464486053 >> v2:172.21.15.170:3300/0 conn(0x7feab00cda80 0x7feab015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.922+0000 7feaaed9d640 1 --2- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab006c040 0x7feab006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.922+0000 7feaae59c640 1 -- 172.21.15.29:0/464486053 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174078345 0 0) 0x7feab006c960 con 0x7feab00ad970 2023-10-16T14:36:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.922+0000 7feaae59c640 1 -- 172.21.15.29:0/464486053 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea9c003520 con 0x7feab00ad970 2023-10-16T14:36:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.922+0000 7feaae59c640 1 -- 172.21.15.29:0/464486053 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869692548 0 0) 0x7fea9c003520 con 0x7feab00ad970 2023-10-16T14:36:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.923+0000 7feaaf59e640 1 -- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab006c040 msgr2=0x7feab006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.923+0000 7feaaf59e640 1 --2- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab006c040 0x7feab006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.923+0000 7feaaf59e640 1 -- 172.21.15.29:0/464486053 >> v1:172.21.15.29:6789/0 conn(0x7feab00ad970 legacy=0x7feab015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.923+0000 7feaaf59e640 1 -- 172.21.15.29:0/464486053 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab015bf90 con 0x7feab00cda80 2023-10-16T14:36:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.924+0000 7feaaf59e640 1 --2- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab00cda80 0x7feab015ed90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009d90 tx=0x7feaa802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=922ddf833aa54800 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feaae59c640 1 -- 172.21.15.29:0/464486053 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98001670 con 0x7feab00cda80 2023-10-16T14:36:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feaae59c640 1 -- 172.21.15.29:0/464486053 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa802c8a0 con 0x7feab00cda80 2023-10-16T14:36:55.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feaae59c640 1 -- 172.21.15.29:0/464486053 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa802cc30 con 0x7feab00cda80 2023-10-16T14:36:55.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feab48b3640 1 -- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab00cda80 msgr2=0x7feab015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feab48b3640 1 --2- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab00cda80 0x7feab015ed90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009d90 tx=0x7feaa802bd70 comp rx=0 tx=0).stop 2023-10-16T14:36:55.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feab48b3640 1 -- 172.21.15.29:0/464486053 shutdown_connections 2023-10-16T14:36:55.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.925+0000 7feab48b3640 1 --2- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab00cda80 0x7feab015ed90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.926+0000 7feab48b3640 1 --2- 172.21.15.29:0/464486053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab006c040 0x7feab006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.926+0000 7feab48b3640 1 -- 172.21.15.29:0/464486053 >> 172.21.15.29:0/464486053 conn(0x7feab00ce6a0 msgr2=0x7feab00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:55.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.926+0000 7feab48b3640 1 -- 172.21.15.29:0/464486053 shutdown_connections 2023-10-16T14:36:55.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.926+0000 7feab48b3640 1 -- 172.21.15.29:0/464486053 wait complete. 2023-10-16T14:36:55.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.928+0000 7feab48b3640 1 Processor -- start 2023-10-16T14:36:55.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.928+0000 7feab48b3640 1 -- start start 2023-10-16T14:36:55.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.929+0000 7feab48b3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab006c040 0x7feab0156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.929+0000 7feab48b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 0x7feab01f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.929+0000 7feab48b3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab00cda80 0x7feab00c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:55.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.929+0000 7feab48b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab0150c80 con 0x7feab00ad970 2023-10-16T14:36:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.929+0000 7feab48b3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab01f6920 con 0x7feab006c040 2023-10-16T14:36:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.929+0000 7feab48b3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab01f6080 con 0x7feab00cda80 2023-10-16T14:36:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 0x7feab01f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 0x7feab01f59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40412/0 (socket says 172.21.15.29:40412) 2023-10-16T14:36:55.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 -- 172.21.15.29:0/2593640438 learned_addr learned my addr 172.21.15.29:0/2593640438 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaaed9d640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab006c040 0x7feab0156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaaf59e640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab00cda80 0x7feab00c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 -- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab00cda80 msgr2=0x7feab00c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab00cda80 0x7feab00c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 -- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab006c040 msgr2=0x7feab0156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab006c040 0x7feab0156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 -- 172.21.15.29:0/2593640438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8009a70 con 0x7feab00ad970 2023-10-16T14:36:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.930+0000 7feaa7fff640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 0x7feab01f59c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fea9800c790 tx=0x7fea9800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.931+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98005ad0 con 0x7feab00ad970 2023-10-16T14:36:55.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.931+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea98005c30 con 0x7feab00ad970 2023-10-16T14:36:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.931+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98007070 con 0x7feab00ad970 2023-10-16T14:36:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.931+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab01f6360 con 0x7feab00ad970 2023-10-16T14:36:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.931+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab014cc90 con 0x7feab00ad970 2023-10-16T14:36:55.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.932+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab006b330 con 0x7feab00ad970 2023-10-16T14:36:55.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.933+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fea980071d0 con 0x7feab00ad970 2023-10-16T14:36:55.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.934+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fea980891f0 con 0x7feab00ad970 2023-10-16T14:36:55.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:55.938+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea98054300 con 0x7feab00ad970 2023-10-16T14:36:56.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.129+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feab004e9b0 con 0x7feab00ad970 2023-10-16T14:36:56.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.130+0000 7feaa77fe640 1 -- 172.21.15.29:0/2593640438 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) v1 ==== 104+0+11 (secure 0 0 0) 0x7fea98057920 con 0x7feab00ad970 2023-10-16T14:36:56.133 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T14:36:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 msgr2=0x7feab01f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 0x7feab01f59c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fea9800c790 tx=0x7fea9800cc50 comp rx=0 tx=0).stop 2023-10-16T14:36:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 shutdown_connections 2023-10-16T14:36:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feab00ad970 0x7feab01f59c0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feab00cda80 0x7feab00c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 --2- 172.21.15.29:0/2593640438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feab006c040 0x7feab0156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 >> 172.21.15.29:0/2593640438 conn(0x7feab00ce6a0 msgr2=0x7feab0069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 shutdown_connections 2023-10-16T14:36:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.133+0000 7feab48b3640 1 -- 172.21.15.29:0/2593640438 wait complete. 2023-10-16T14:36:56.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:36:56.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.375+0000 7fc7ee4ad640 1 Processor -- start 2023-10-16T14:36:56.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.375+0000 7fc7ee4ad640 1 -- start start 2023-10-16T14:36:56.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.376+0000 7fc7ee4ad640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc7e806c040 0x7fc7e806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.376+0000 7fc7ee4ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7e80cda80 0x7fc7e815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.376+0000 7fc7ee4ad640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e806c960 con 0x7fc7e80ad970 2023-10-16T14:36:56.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.376+0000 7fc7ee4ad640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e815b9d0 con 0x7fc7e80cda80 2023-10-16T14:36:56.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.376+0000 7fc7ee4ad640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e815bcb0 con 0x7fc7e806c040 2023-10-16T14:36:56.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7e80cda80 0x7fc7e815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7e80cda80 0x7fc7e815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40424/0 (socket says 172.21.15.29:40424) 2023-10-16T14:36:56.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 -- 172.21.15.29:0/916654289 learned_addr learned my addr 172.21.15.29:0/916654289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:56.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7ed4ab640 1 --2- 172.21.15.29:0/916654289 >> v2:172.21.15.170:3300/0 conn(0x7fc7e806c040 0x7fc7e806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 -- 172.21.15.29:0/916654289 >> v2:172.21.15.170:3300/0 conn(0x7fc7e806c040 msgr2=0x7fc7e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 --2- 172.21.15.29:0/916654289 >> v2:172.21.15.170:3300/0 conn(0x7fc7e806c040 0x7fc7e806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 -- 172.21.15.29:0/916654289 >> v1:172.21.15.170:6789/0 conn(0x7fc7e80ad970 legacy=0x7fc7e815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7edcac640 1 -- 172.21.15.29:0/916654289 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e815bf90 con 0x7fc7e80cda80 2023-10-16T14:36:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.377+0000 7fc7ed4ab640 1 --2- 172.21.15.29:0/916654289 >> v2:172.21.15.170:3300/0 conn(0x7fc7e806c040 0x7fc7e806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:56.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.378+0000 7fc7edcac640 1 --2- 172.21.15.29:0/916654289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80cda80 0x7fc7e815ed90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80cf350 tx=0x7fc7e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c9267304d355dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:56.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7dffff640 1 -- 172.21.15.29:0/916654289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e002ca80 con 0x7fc7e80cda80 2023-10-16T14:36:56.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7dffff640 1 -- 172.21.15.29:0/916654289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e002cbe0 con 0x7fc7e80cda80 2023-10-16T14:36:56.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7dffff640 1 -- 172.21.15.29:0/916654289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00343f0 con 0x7fc7e80cda80 2023-10-16T14:36:56.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/916654289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80cda80 msgr2=0x7fc7e815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/916654289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80cda80 0x7fc7e815ed90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80cf350 tx=0x7fc7e002bee0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/916654289 shutdown_connections 2023-10-16T14:36:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/916654289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80cda80 0x7fc7e815ed90 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/916654289 >> v2:172.21.15.170:3300/0 conn(0x7fc7e806c040 0x7fc7e806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/916654289 >> 172.21.15.29:0/916654289 conn(0x7fc7e80ce6a0 msgr2=0x7fc7e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/916654289 shutdown_connections 2023-10-16T14:36:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.379+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/916654289 wait complete. 2023-10-16T14:36:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.381+0000 7fc7ee4ad640 1 Processor -- start 2023-10-16T14:36:56.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.381+0000 7fc7ee4ad640 1 -- start start 2023-10-16T14:36:56.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7ee4ad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7e806c040 0x7fc7e80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7ee4ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 0x7fc7e80aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7ee4ad640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7e80cda80 0x7fc7e80aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7ee4ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e80c67c0 con 0x7fc7e80ad970 2023-10-16T14:36:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7ee4ad640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e81f0810 con 0x7fc7e80cda80 2023-10-16T14:36:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7ee4ad640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e81f0af0 con 0x7fc7e806c040 2023-10-16T14:36:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7eccaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 0x7fc7e80aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7eccaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 0x7fc7e80aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40434/0 (socket says 172.21.15.29:40434) 2023-10-16T14:36:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.382+0000 7fc7eccaa640 1 -- 172.21.15.29:0/4015995126 learned_addr learned my addr 172.21.15.29:0/4015995126 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7eccaa640 1 -- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7e806c040 msgr2=0x7fc7e80ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7eccaa640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7e806c040 0x7fc7e80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7eccaa640 1 -- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7e80cda80 msgr2=0x7fc7e80aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7eccaa640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7e80cda80 0x7fc7e80aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7eccaa640 1 -- 172.21.15.29:0/4015995126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e0009b30 con 0x7fc7e80ad970 2023-10-16T14:36:56.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7eccaa640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 0x7fc7e80aea80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800c400 tx=0x7fc7d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:56.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8011790 con 0x7fc7e80ad970 2023-10-16T14:36:56.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d8011930 con 0x7fc7e80ad970 2023-10-16T14:36:56.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8011ce0 con 0x7fc7e80ad970 2023-10-16T14:36:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.383+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e81f0dd0 con 0x7fc7e80ad970 2023-10-16T14:36:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.384+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e80ae640 con 0x7fc7e80ad970 2023-10-16T14:36:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.385+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e806b330 con 0x7fc7e80ad970 2023-10-16T14:36:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.386+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc7d801c460 con 0x7fc7e80ad970 2023-10-16T14:36:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.387+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fc7d80893e0 con 0x7fc7e80ad970 2023-10-16T14:36:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.391+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7d80544f0 con 0x7fc7e80ad970 2023-10-16T14:36:56.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.504+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 75) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc7d8053fa0 con 0x7fc7e80ad970 2023-10-16T14:36:56.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.505+0000 7fc7ddffb640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc7cc070640 0x7fc7cc072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.509+0000 7fc7ed4ab640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc7cc070640 0x7fc7cc072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.520+0000 7fc7ed4ab640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc7cc070640 0x7fc7cc072a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4006fd0 tx=0x7fc7e4008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.596+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7e80c6900 con 0x7fc7e80ad970 2023-10-16T14:36:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.597+0000 7fc7ddffb640 1 -- 172.21.15.29:0/4015995126 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc7d8070e20 con 0x7fc7e80ad970 2023-10-16T14:36:56.600 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T14:36:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.599+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc7cc070640 msgr2=0x7fc7cc072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.599+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc7cc070640 0x7fc7cc072a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4006fd0 tx=0x7fc7e4008040 comp rx=0 tx=0).stop 2023-10-16T14:36:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.599+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 msgr2=0x7fc7e80aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.599+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 0x7fc7e80aea80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800c400 tx=0x7fc7d800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 shutdown_connections 2023-10-16T14:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc7cc070640 0x7fc7cc072a30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7e80ad970 0x7fc7e80aea80 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc7e80cda80 0x7fc7e80aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 --2- 172.21.15.29:0/4015995126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc7e806c040 0x7fc7e80ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 >> 172.21.15.29:0/4015995126 conn(0x7fc7e80ce6a0 msgr2=0x7fc7e815d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 shutdown_connections 2023-10-16T14:36:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.600+0000 7fc7ee4ad640 1 -- 172.21.15.29:0/4015995126 wait complete. 2023-10-16T14:36:56.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:56.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.842+0000 7f51048c3640 1 Processor -- start 2023-10-16T14:36:56.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.843+0000 7f51048c3640 1 -- start start 2023-10-16T14:36:56.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.844+0000 7f51048c3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f510015cfc0 0x7f51000b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.844+0000 7f51048c3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51000d2640 0x7f51000b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.844+0000 7f51048c3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000d15a0 con 0x7f510015cbe0 2023-10-16T14:36:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.844+0000 7f51048c3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000d1880 con 0x7f51000d2640 2023-10-16T14:36:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.844+0000 7f51048c3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000d1b60 con 0x7f510015cfc0 2023-10-16T14:36:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51000d2640 0x7f51000b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51000d2640 0x7f51000b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40436/0 (socket says 172.21.15.29:40436) 2023-10-16T14:36:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 -- 172.21.15.29:0/2455070694 learned_addr learned my addr 172.21.15.29:0/2455070694 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50f7fff640 1 --2- 172.21.15.29:0/2455070694 >> v2:172.21.15.170:3300/0 conn(0x7f510015cfc0 0x7f51000b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 -- 172.21.15.29:0/2455070694 >> v2:172.21.15.170:3300/0 conn(0x7f510015cfc0 msgr2=0x7f51000b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:56.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 --2- 172.21.15.29:0/2455070694 >> v2:172.21.15.170:3300/0 conn(0x7f510015cfc0 0x7f51000b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 -- 172.21.15.29:0/2455070694 >> v1:172.21.15.170:6789/0 conn(0x7f510015cbe0 legacy=0x7f51000d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.845+0000 7f50fffff640 1 -- 172.21.15.29:0/2455070694 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000d1e40 con 0x7f51000d2640 2023-10-16T14:36:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.846+0000 7f50fffff640 1 --2- 172.21.15.29:0/2455070694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51000d2640 0x7f51000b4ab0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f51000cf4b0 tx=0x7f50f802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d9d180452fd6a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f50feffd640 1 -- 172.21.15.29:0/2455070694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f802cb20 con 0x7f51000d2640 2023-10-16T14:36:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f50feffd640 1 -- 172.21.15.29:0/2455070694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f802cc80 con 0x7f51000d2640 2023-10-16T14:36:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f50feffd640 1 -- 172.21.15.29:0/2455070694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f80343f0 con 0x7f51000d2640 2023-10-16T14:36:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 -- 172.21.15.29:0/2455070694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51000d2640 msgr2=0x7f51000b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 --2- 172.21.15.29:0/2455070694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51000d2640 0x7f51000b4ab0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f51000cf4b0 tx=0x7f50f802bf80 comp rx=0 tx=0).stop 2023-10-16T14:36:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 -- 172.21.15.29:0/2455070694 shutdown_connections 2023-10-16T14:36:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 --2- 172.21.15.29:0/2455070694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51000d2640 0x7f51000b4ab0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 --2- 172.21.15.29:0/2455070694 >> v2:172.21.15.170:3300/0 conn(0x7f510015cfc0 0x7f51000b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 -- 172.21.15.29:0/2455070694 >> 172.21.15.29:0/2455070694 conn(0x7f51000cf570 msgr2=0x7f51000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 -- 172.21.15.29:0/2455070694 shutdown_connections 2023-10-16T14:36:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.847+0000 7f51048c3640 1 -- 172.21.15.29:0/2455070694 wait complete. 2023-10-16T14:36:56.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.849+0000 7f51048c3640 1 Processor -- start 2023-10-16T14:36:56.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.849+0000 7f51048c3640 1 -- start start 2023-10-16T14:36:56.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.850+0000 7f51048c3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51000d2640 0x7f51000cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.850+0000 7f51048c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 0x7f51000b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.850+0000 7f51048c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f510015cfc0 0x7f5100156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.850+0000 7f51048c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000c6fe0 con 0x7f510015cbe0 2023-10-16T14:36:56.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.850+0000 7f51048c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000b74c0 con 0x7f510015cfc0 2023-10-16T14:36:56.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.850+0000 7f51048c3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51000c7ac0 con 0x7f51000d2640 2023-10-16T14:36:56.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 0x7f51000b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 0x7f51000b6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40450/0 (socket says 172.21.15.29:40450) 2023-10-16T14:36:56.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 -- 172.21.15.29:0/2518953917 learned_addr learned my addr 172.21.15.29:0/2518953917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:56.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50ff7fe640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51000d2640 0x7f51000cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50fffff640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f510015cfc0 0x7f5100156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 -- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51000d2640 msgr2=0x7f51000cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51000d2640 0x7f51000cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 -- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f510015cfc0 msgr2=0x7f5100156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f510015cfc0 0x7f5100156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 -- 172.21.15.29:0/2518953917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f8009b30 con 0x7f510015cbe0 2023-10-16T14:36:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50ff7fe640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51000d2640 0x7f51000cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.851+0000 7f50f7fff640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 0x7f51000b6c70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f50e800d5f0 tx=0x7f50e800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.852+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e80125f0 con 0x7f510015cbe0 2023-10-16T14:36:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.852+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e8012750 con 0x7f510015cbe0 2023-10-16T14:36:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.852+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8012b00 con 0x7f510015cbe0 2023-10-16T14:36:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.852+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51000c71d0 con 0x7f510015cbe0 2023-10-16T14:36:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.852+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51000c7750 con 0x7f510015cbe0 2023-10-16T14:36:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.853+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51000d11e0 con 0x7f510015cbe0 2023-10-16T14:36:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.854+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f50e8012c60 con 0x7f510015cbe0 2023-10-16T14:36:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.855+0000 7f50fcff9640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f50e006acf0 0x7f50e006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.855+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f50e80899c0 con 0x7f510015cbe0 2023-10-16T14:36:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.856+0000 7f50ff7fe640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f50e006acf0 0x7f50e006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.857+0000 7f50ff7fe640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f50e006acf0 0x7f50e006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f50f0003eb0 tx=0x7f50f0009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:56.860+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f50e8054ab0 con 0x7f510015cbe0 2023-10-16T14:36:57.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.090+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f510004e9b0 con 0x7f510015cbe0 2023-10-16T14:36:57.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.091+0000 7f50fcff9640 1 -- 172.21.15.29:0/2518953917 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f50e80580d0 con 0x7f510015cbe0 2023-10-16T14:36:57.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:57.095 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:36:27.196747+0000","modified":"2023-10-16T14:36:30.291825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1700644562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1700644562},{"type":"v1","addr":"172.21.15.29:6835","nonce":1700644562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T14:36:57.095 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-16T14:36:57.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.094+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f50e006acf0 msgr2=0x7f50e006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.094+0000 7f51048c3640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f50e006acf0 0x7f50e006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f50f0003eb0 tx=0x7f50f0009040 comp rx=0 tx=0).stop 2023-10-16T14:36:57.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.094+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 msgr2=0x7f51000b6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.094+0000 7f51048c3640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 0x7f51000b6c70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f50e800d5f0 tx=0x7f50e800dab0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 shutdown_connections 2023-10-16T14:36:57.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f50e006acf0 0x7f50e006d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f510015cbe0 0x7f51000b6c70 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f510015cfc0 0x7f5100156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 --2- 172.21.15.29:0/2518953917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51000d2640 0x7f51000cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 >> 172.21.15.29:0/2518953917 conn(0x7f51000cf570 msgr2=0x7f51000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:57.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 shutdown_connections 2023-10-16T14:36:57.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.095+0000 7f51048c3640 1 -- 172.21.15.29:0/2518953917 wait complete. 2023-10-16T14:36:57.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.343+0000 7fa44ed2e640 1 Processor -- start 2023-10-16T14:36:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.343+0000 7fa44ed2e640 1 -- start start 2023-10-16T14:36:57.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44ed2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 0x7fa44806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44ed2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4480cda80 0x7fa44815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44ed2e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44806c960 con 0x7fa4480ad970 2023-10-16T14:36:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44ed2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44815b9d0 con 0x7fa4480cda80 2023-10-16T14:36:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44ed2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44815bcb0 con 0x7fa44806c040 2023-10-16T14:36:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44e52d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4480cda80 0x7fa44815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44e52d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4480cda80 0x7fa44815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40458/0 (socket says 172.21.15.29:40458) 2023-10-16T14:36:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44e52d640 1 -- 172.21.15.29:0/1505613393 learned_addr learned my addr 172.21.15.29:0/1505613393 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44e52d640 1 -- 172.21.15.29:0/1505613393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 msgr2=0x7fa44806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.345+0000 7fa44e52d640 1 --2- 172.21.15.29:0/1505613393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 0x7fa44806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.346+0000 7fa44e52d640 1 -- 172.21.15.29:0/1505613393 >> v1:172.21.15.170:6789/0 conn(0x7fa4480ad970 legacy=0x7fa44815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.346+0000 7fa44e52d640 1 -- 172.21.15.29:0/1505613393 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44815bf90 con 0x7fa4480cda80 2023-10-16T14:36:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.347+0000 7fa44e52d640 1 --2- 172.21.15.29:0/1505613393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa44815ed90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fa440009e50 tx=0x7fa44002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7f6834052915aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.347+0000 7fa44cd2a640 1 -- 172.21.15.29:0/1505613393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44002ca80 con 0x7fa4480cda80 2023-10-16T14:36:57.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.347+0000 7fa44cd2a640 1 -- 172.21.15.29:0/1505613393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44002cbe0 con 0x7fa4480cda80 2023-10-16T14:36:57.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.347+0000 7fa44cd2a640 1 -- 172.21.15.29:0/1505613393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4400343f0 con 0x7fa4480cda80 2023-10-16T14:36:57.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 -- 172.21.15.29:0/1505613393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 msgr2=0x7fa44815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/1505613393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa44815ed90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fa440009e50 tx=0x7fa44002bee0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 -- 172.21.15.29:0/1505613393 shutdown_connections 2023-10-16T14:36:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/1505613393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa44815ed90 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/1505613393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 0x7fa44806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 -- 172.21.15.29:0/1505613393 >> 172.21.15.29:0/1505613393 conn(0x7fa4480ce6a0 msgr2=0x7fa4480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 -- 172.21.15.29:0/1505613393 shutdown_connections 2023-10-16T14:36:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.348+0000 7fa44ed2e640 1 -- 172.21.15.29:0/1505613393 wait complete. 2023-10-16T14:36:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.350+0000 7fa44ed2e640 1 Processor -- start 2023-10-16T14:36:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.350+0000 7fa44ed2e640 1 -- start start 2023-10-16T14:36:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44ed2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 0x7fa4481e4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44ed2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4480ad970 0x7fa448148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44ed2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa4480c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44ed2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa448148970 con 0x7fa4480cda80 2023-10-16T14:36:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44ed2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4481458f0 con 0x7fa4480ad970 2023-10-16T14:36:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44ed2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa448145b90 con 0x7fa44806c040 2023-10-16T14:36:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44e52d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa4480c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44e52d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa4480c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40468/0 (socket says 172.21.15.29:40468) 2023-10-16T14:36:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44e52d640 1 -- 172.21.15.29:0/2498203942 learned_addr learned my addr 172.21.15.29:0/2498203942 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44d52b640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4480ad970 0x7fa448148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44e52d640 1 -- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 msgr2=0x7fa4481e4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.351+0000 7fa44e52d640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 0x7fa4481e4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa44e52d640 1 -- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4480ad970 msgr2=0x7fa448148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa44e52d640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4480ad970 0x7fa448148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa44e52d640 1 -- 172.21.15.29:0/2498203942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa440009b30 con 0x7fa4480cda80 2023-10-16T14:36:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa44d52b640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4480ad970 0x7fa448148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa44e52d640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa4480c4940 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fa440036040 tx=0x7fa440004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440040670 con 0x7fa4480cda80 2023-10-16T14:36:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4400407f0 con 0x7fa4480cda80 2023-10-16T14:36:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440040ba0 con 0x7fa4480cda80 2023-10-16T14:36:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.352+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa448145e70 con 0x7fa4480cda80 2023-10-16T14:36:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.353+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa448146430 con 0x7fa4480cda80 2023-10-16T14:36:57.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.354+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44806b330 con 0x7fa4480cda80 2023-10-16T14:36:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.355+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa44002cd50 con 0x7fa4480cda80 2023-10-16T14:36:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.356+0000 7fa43e7fc640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa42c06acc0 0x7fa42c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.356+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fa4400ac410 con 0x7fa4480cda80 2023-10-16T14:36:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.356+0000 7fa44dd2c640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa42c06acc0 0x7fa42c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.358+0000 7fa44dd2c640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa42c06acc0 0x7fa42c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa444009670 tx=0x7fa444006bf0 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.360+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa440077500 con 0x7fa4480cda80 2023-10-16T14:36:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.512+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 76) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa4400778d0 con 0x7fa4480cda80 2023-10-16T14:36:57.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.562+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa44804e9b0 con 0x7fa4480cda80 2023-10-16T14:36:57.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.565+0000 7fa43e7fc640 1 -- 172.21.15.29:0/2498203942 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+17649 (secure 0 0 0) 0x7fa44008ef10 con 0x7fa4480cda80 2023-10-16T14:36:57.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:57.569 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":144,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:55.479234+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3813450646":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1911194233":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/3477374804":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/601419330":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1163999179":"2023-10-17T14:36:55.479187+0000","172.21.15.170:6833/230171195":"2023-10-17T14:36:55.479187+0000","172.21.15.170:6832/230171195":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1143303444":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/2774967038":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2333704669":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1712463277":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2908562666":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/782892922":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6833/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6832/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1119524146":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1398880520":"2023-10-17T14:36:42.416389+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:57.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa42c06acc0 msgr2=0x7fa42c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa42c06acc0 0x7fa42c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa444009670 tx=0x7fa444006bf0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 msgr2=0x7fa4480c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa4480c4940 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fa440036040 tx=0x7fa440004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:57.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 shutdown_connections 2023-10-16T14:36:57.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa42c06acc0 0x7fa42c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4480cda80 0x7fa4480c4940 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4480ad970 0x7fa448148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 --2- 172.21.15.29:0/2498203942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44806c040 0x7fa4481e4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 >> 172.21.15.29:0/2498203942 conn(0x7fa4480ce6a0 msgr2=0x7fa448159ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 shutdown_connections 2023-10-16T14:36:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.568+0000 7fa44ed2e640 1 -- 172.21.15.29:0/2498203942 wait complete. 2023-10-16T14:36:57.590 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:36:57.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:57.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.803+0000 7fe0d9d1a640 1 Processor -- start 2023-10-16T14:36:57.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.803+0000 7fe0d9d1a640 1 -- start start 2023-10-16T14:36:57.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d9d1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe0d415c9f0 0x7fe0d40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d9d1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4160a90 con 0x7fe0d40c9230 2023-10-16T14:36:57.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d9d1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40cd900 con 0x7fe0d415dde0 2023-10-16T14:36:57.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d9d1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d40cdec0 con 0x7fe0d415c9f0 2023-10-16T14:36:57.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d8d18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe0d415c9f0 0x7fe0d40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d8d18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe0d415c9f0 0x7fe0d40c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40478/0 (socket says 172.21.15.29:40478) 2023-10-16T14:36:57.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.805+0000 7fe0d8d18640 1 -- 172.21.15.29:0/91208357 learned_addr learned my addr 172.21.15.29:0/91208357 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.806+0000 7fe0d8d18640 1 -- 172.21.15.29:0/91208357 >> v1:172.21.15.170:6789/0 conn(0x7fe0d415dde0 legacy=0x7fe0d40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.806+0000 7fe0d8d18640 1 -- 172.21.15.29:0/91208357 >> v1:172.21.15.29:6789/0 conn(0x7fe0d40c9230 legacy=0x7fe0d40709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.806+0000 7fe0d8d18640 1 -- 172.21.15.29:0/91208357 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40cdbe0 con 0x7fe0d415c9f0 2023-10-16T14:36:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.807+0000 7fe0d8d18640 1 --2- 172.21.15.29:0/91208357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415c9f0 0x7fe0d40c8cf0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009870 tx=0x7fe0c802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56fc483d3de5ba1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.807+0000 7fe0d37fe640 1 -- 172.21.15.29:0/91208357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8030850 con 0x7fe0d415c9f0 2023-10-16T14:36:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.807+0000 7fe0d37fe640 1 -- 172.21.15.29:0/91208357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c80309b0 con 0x7fe0d415c9f0 2023-10-16T14:36:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.807+0000 7fe0d37fe640 1 -- 172.21.15.29:0/91208357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8030d40 con 0x7fe0d415c9f0 2023-10-16T14:36:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/91208357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415c9f0 msgr2=0x7fe0d40c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/91208357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415c9f0 0x7fe0d40c8cf0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009870 tx=0x7fe0c802b990 comp rx=0 tx=0).stop 2023-10-16T14:36:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/91208357 shutdown_connections 2023-10-16T14:36:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/91208357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415c9f0 0x7fe0d40c8cf0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/91208357 >> 172.21.15.29:0/91208357 conn(0x7fe0d40ce6a0 msgr2=0x7fe0d40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/91208357 shutdown_connections 2023-10-16T14:36:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.808+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/91208357 wait complete. 2023-10-16T14:36:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.810+0000 7fe0d9d1a640 1 Processor -- start 2023-10-16T14:36:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.811+0000 7fe0d9d1a640 1 -- start start 2023-10-16T14:36:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9d1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d40c9230 0x7fe0d4153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9d1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d415c9f0 0x7fe0d4157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9d1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 0x7fe0d4157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9d1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d40c2af0 con 0x7fe0d415dde0 2023-10-16T14:36:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9d1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d41f2930 con 0x7fe0d415c9f0 2023-10-16T14:36:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9d1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d40c24a0 con 0x7fe0d40c9230 2023-10-16T14:36:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 0x7fe0d4157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 0x7fe0d4157f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40494/0 (socket says 172.21.15.29:40494) 2023-10-16T14:36:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.812+0000 7fe0d9519640 1 -- 172.21.15.29:0/332850152 learned_addr learned my addr 172.21.15.29:0/332850152 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d8d18640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d40c9230 0x7fe0d4153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d3fff640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d415c9f0 0x7fe0d4157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d9519640 1 -- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d40c9230 msgr2=0x7fe0d4153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d9519640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d40c9230 0x7fe0d4153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d9519640 1 -- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d415c9f0 msgr2=0x7fe0d4157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d9519640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d415c9f0 0x7fe0d4157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d9519640 1 -- 172.21.15.29:0/332850152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80094d0 con 0x7fe0d415dde0 2023-10-16T14:36:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.813+0000 7fe0d9519640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 0x7fe0d4157f90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00b960 tx=0x7fe0cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:57.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.814+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00cb20 con 0x7fe0d415dde0 2023-10-16T14:36:57.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.814+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00cc80 con 0x7fe0d415dde0 2023-10-16T14:36:57.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.814+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc014440 con 0x7fe0d415dde0 2023-10-16T14:36:57.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.814+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d41541e0 con 0x7fe0d415dde0 2023-10-16T14:36:57.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.814+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d4153c60 con 0x7fe0d415dde0 2023-10-16T14:36:57.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.815+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d40c2c30 con 0x7fe0d415dde0 2023-10-16T14:36:57.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.816+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe0cc01d3b0 con 0x7fe0d415dde0 2023-10-16T14:36:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.817+0000 7fe0d17fa640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe0b806af00 0x7fe0b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.818+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fe0cc0898d0 con 0x7fe0d415dde0 2023-10-16T14:36:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.818+0000 7fe0d8d18640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe0b806af00 0x7fe0b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.819+0000 7fe0d8d18640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe0b806af00 0x7fe0b806d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8036000 tx=0x7fe0c8008030 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:57.822+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe0d40c2c30 con 0x7fe0d415dde0 2023-10-16T14:36:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.050+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0d40c2c30 con 0x7fe0d415dde0 2023-10-16T14:36:58.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.051+0000 7fe0d17fa640 1 -- 172.21.15.29:0/332850152 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe0d40c2c30 con 0x7fe0d415dde0 2023-10-16T14:36:58.054 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:58.054 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:36:27.196747+0000","modified":"2023-10-16T14:36:30.291825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1700644562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1700644562},{"type":"v1","addr":"172.21.15.29:6835","nonce":1700644562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T14:36:58.055 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-16T14:36:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.053+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe0b806af00 msgr2=0x7fe0b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.053+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe0b806af00 0x7fe0b806d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8036000 tx=0x7fe0c8008030 comp rx=0 tx=0).stop 2023-10-16T14:36:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.053+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 msgr2=0x7fe0d4157f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.053+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 0x7fe0d4157f90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00b960 tx=0x7fe0cc00be20 comp rx=0 tx=0).stop 2023-10-16T14:36:58.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 shutdown_connections 2023-10-16T14:36:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe0b806af00 0x7fe0b806d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0d415dde0 0x7fe0d4157f90 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0d415c9f0 0x7fe0d4157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 --2- 172.21.15.29:0/332850152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0d40c9230 0x7fe0d4153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 >> 172.21.15.29:0/332850152 conn(0x7fe0d40ce6a0 msgr2=0x7fe0d40ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 shutdown_connections 2023-10-16T14:36:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.054+0000 7fe0d9d1a640 1 -- 172.21.15.29:0/332850152 wait complete. 2023-10-16T14:36:58.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:36:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.294+0000 7f4cc6b4e640 1 Processor -- start 2023-10-16T14:36:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.295+0000 7f4cc6b4e640 1 -- start start 2023-10-16T14:36:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.296+0000 7f4cc6b4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4cc00cd0c0 0x7f4cc01615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.296+0000 7f4cc6b4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4cc015db00 0x7f4cc00b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.296+0000 7f4cc6b4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc015e3e0 con 0x7f4cc00b0410 2023-10-16T14:36:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.296+0000 7f4cc6b4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc00b0de0 con 0x7f4cc00cd0c0 2023-10-16T14:36:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.296+0000 7f4cc6b4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc00cd4a0 con 0x7f4cc015db00 2023-10-16T14:36:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4cc00cd0c0 0x7f4cc01615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4cc00cd0c0 0x7f4cc01615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40504/0 (socket says 172.21.15.29:40504) 2023-10-16T14:36:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 -- 172.21.15.29:0/3922359560 learned_addr learned my addr 172.21.15.29:0/3922359560 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc50eb640 1 --2- 172.21.15.29:0/3922359560 >> v2:172.21.15.170:3300/0 conn(0x7f4cc015db00 0x7f4cc00b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 -- 172.21.15.29:0/3922359560 >> v2:172.21.15.170:3300/0 conn(0x7f4cc015db00 msgr2=0x7f4cc00b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 --2- 172.21.15.29:0/3922359560 >> v2:172.21.15.170:3300/0 conn(0x7f4cc015db00 0x7f4cc00b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 -- 172.21.15.29:0/3922359560 >> v1:172.21.15.170:6789/0 conn(0x7f4cc00b0410 legacy=0x7f4cc015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc48ea640 1 -- 172.21.15.29:0/3922359560 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc015e100 con 0x7f4cc00cd0c0 2023-10-16T14:36:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.297+0000 7f4cc50eb640 1 --2- 172.21.15.29:0/3922359560 >> v2:172.21.15.170:3300/0 conn(0x7f4cc015db00 0x7f4cc00b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.298+0000 7f4cc48ea640 1 --2- 172.21.15.29:0/3922359560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc01615f0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f4cc015afc0 tx=0x7f4cb402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61eb07659313ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.298+0000 7f4cbf7fe640 1 -- 172.21.15.29:0/3922359560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40317e0 con 0x7f4cc00cd0c0 2023-10-16T14:36:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.298+0000 7f4cbf7fe640 1 -- 172.21.15.29:0/3922359560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4031940 con 0x7f4cc00cd0c0 2023-10-16T14:36:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.298+0000 7f4cbf7fe640 1 -- 172.21.15.29:0/3922359560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4031cd0 con 0x7f4cc00cd0c0 2023-10-16T14:36:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/3922359560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc00cd0c0 msgr2=0x7f4cc01615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/3922359560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc01615f0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f4cc015afc0 tx=0x7f4cb402b880 comp rx=0 tx=0).stop 2023-10-16T14:36:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/3922359560 shutdown_connections 2023-10-16T14:36:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/3922359560 >> v2:172.21.15.170:3300/0 conn(0x7f4cc015db00 0x7f4cc00b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/3922359560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc01615f0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/3922359560 >> 172.21.15.29:0/3922359560 conn(0x7f4cc00cb410 msgr2=0x7f4cc00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/3922359560 shutdown_connections 2023-10-16T14:36:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.299+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/3922359560 wait complete. 2023-10-16T14:36:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.301+0000 7f4cc6b4e640 1 Processor -- start 2023-10-16T14:36:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.301+0000 7f4cc6b4e640 1 -- start start 2023-10-16T14:36:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc6b4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cc00b0410 0x7f4cc0155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc6b4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc00ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc6b4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 0x7f4cc00ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc6b4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc01f3230 con 0x7f4cc015db00 2023-10-16T14:36:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc6b4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc00b2160 con 0x7f4cc00cd0c0 2023-10-16T14:36:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc6b4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc01f41e0 con 0x7f4cc00b0410 2023-10-16T14:36:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc50eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 0x7f4cc00ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc50eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 0x7f4cc00ca550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40514/0 (socket says 172.21.15.29:40514) 2023-10-16T14:36:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.302+0000 7f4cc50eb640 1 -- 172.21.15.29:0/1641727479 learned_addr learned my addr 172.21.15.29:0/1641727479 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cbffff640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc00ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc48ea640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cc00b0410 0x7f4cc0155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc50eb640 1 -- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cc00b0410 msgr2=0x7f4cc0155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc50eb640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cc00b0410 0x7f4cc0155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc50eb640 1 -- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc00cd0c0 msgr2=0x7f4cc00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc50eb640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc00ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc50eb640 1 -- 172.21.15.29:0/1641727479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40094d0 con 0x7f4cc015db00 2023-10-16T14:36:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc48ea640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cc00b0410 0x7f4cc0155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cc50eb640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 0x7f4cc00ca550 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800cef0 tx=0x7f4cb800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb800d840 con 0x7f4cc015db00 2023-10-16T14:36:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb800d9c0 con 0x7f4cc015db00 2023-10-16T14:36:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.303+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb800dd70 con 0x7f4cc015db00 2023-10-16T14:36:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.304+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc00c4c80 con 0x7f4cc015db00 2023-10-16T14:36:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.304+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc01f3700 con 0x7f4cc015db00 2023-10-16T14:36:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.305+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc004e160 con 0x7f4cc015db00 2023-10-16T14:36:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.306+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4cb801e3b0 con 0x7f4cc015db00 2023-10-16T14:36:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.307+0000 7f4cbd7fa640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4ca406af00 0x7f4ca406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.307+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4cb808a340 con 0x7f4cc015db00 2023-10-16T14:36:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.308+0000 7f4cc48ea640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4ca406af00 0x7f4ca406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.309+0000 7f4cc48ea640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4ca406af00 0x7f4ca406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4036000 tx=0x7f4cb403d040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.311+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4cb8055250 con 0x7f4cc015db00 2023-10-16T14:36:58.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.545+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc00b4910 con 0x7f4cc015db00 2023-10-16T14:36:58.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.547+0000 7f4cbd7fa640 1 -- 172.21.15.29:0/1641727479 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f4cb8058870 con 0x7f4cc015db00 2023-10-16T14:36:58.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:58.550 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:36:27.196747+0000","modified":"2023-10-16T14:36:30.291825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8283":{"gid":8283,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/1700644562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1700644562},{"type":"v1","addr":"172.21.15.29:6835","nonce":1700644562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T14:36:58.550 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-16T14:36:58.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.549+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4ca406af00 msgr2=0x7f4ca406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.549+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4ca406af00 0x7f4ca406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4036000 tx=0x7f4cb403d040 comp rx=0 tx=0).stop 2023-10-16T14:36:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.549+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 msgr2=0x7f4cc00ca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.549+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 0x7f4cc00ca550 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800cef0 tx=0x7f4cb800a710 comp rx=0 tx=0).stop 2023-10-16T14:36:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 shutdown_connections 2023-10-16T14:36:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4ca406af00 0x7f4ca406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cc015db00 0x7f4cc00ca550 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cc00cd0c0 0x7f4cc00ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 --2- 172.21.15.29:0/1641727479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cc00b0410 0x7f4cc0155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 >> 172.21.15.29:0/1641727479 conn(0x7f4cc00cb410 msgr2=0x7f4cc00ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 shutdown_connections 2023-10-16T14:36:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.550+0000 7f4cc6b4e640 1 -- 172.21.15.29:0/1641727479 wait complete. 2023-10-16T14:36:58.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:36:58.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.795+0000 7f3e9e8e5640 1 Processor -- start 2023-10-16T14:36:58.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.795+0000 7f3e9e8e5640 1 -- start start 2023-10-16T14:36:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9e8e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e9815aaa0 0x7f3e9815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9e8e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e980ce390 0x7f3e98070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9e8e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9815ed60 con 0x7f3e9815ae80 2023-10-16T14:36:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9e8e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9806b810 con 0x7f3e980ce390 2023-10-16T14:36:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9e8e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e980707e0 con 0x7f3e9815aaa0 2023-10-16T14:36:58.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9ce82640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e980ce390 0x7f3e98070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9ce82640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e980ce390 0x7f3e98070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40516/0 (socket says 172.21.15.29:40516) 2023-10-16T14:36:58.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9ce82640 1 -- 172.21.15.29:0/2270583139 learned_addr learned my addr 172.21.15.29:0/2270583139 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:58.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9ce82640 1 -- 172.21.15.29:0/2270583139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e9815aaa0 msgr2=0x7f3e9815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:58.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9ce82640 1 --2- 172.21.15.29:0/2270583139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e9815aaa0 0x7f3e9815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.797+0000 7f3e9ce82640 1 -- 172.21.15.29:0/2270583139 >> v1:172.21.15.170:6789/0 conn(0x7f3e9815ae80 legacy=0x7f3e980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:58.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.798+0000 7f3e9ce82640 1 -- 172.21.15.29:0/2270583139 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9815ef90 con 0x7f3e980ce390 2023-10-16T14:36:58.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.799+0000 7f3e9ce82640 1 --2- 172.21.15.29:0/2270583139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e980ce390 0x7f3e98070f50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009e50 tx=0x7f3e9002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3d3c3b69889a97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:58.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.799+0000 7f3e977fe640 1 -- 172.21.15.29:0/2270583139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9002ca70 con 0x7f3e980ce390 2023-10-16T14:36:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.799+0000 7f3e977fe640 1 -- 172.21.15.29:0/2270583139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9002cbd0 con 0x7f3e980ce390 2023-10-16T14:36:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.799+0000 7f3e977fe640 1 -- 172.21.15.29:0/2270583139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900343f0 con 0x7f3e980ce390 2023-10-16T14:36:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/2270583139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e980ce390 msgr2=0x7f3e98070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/2270583139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e980ce390 0x7f3e98070f50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009e50 tx=0x7f3e9002bed0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/2270583139 shutdown_connections 2023-10-16T14:36:58.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/2270583139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e980ce390 0x7f3e98070f50 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/2270583139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e9815aaa0 0x7f3e9815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/2270583139 >> 172.21.15.29:0/2270583139 conn(0x7f3e980cb410 msgr2=0x7f3e980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:58.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/2270583139 shutdown_connections 2023-10-16T14:36:58.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.800+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/2270583139 wait complete. 2023-10-16T14:36:58.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.802+0000 7f3e9e8e5640 1 Processor -- start 2023-10-16T14:36:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.802+0000 7f3e9e8e5640 1 -- start start 2023-10-16T14:36:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9e8e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e980ce390 0x7f3e981585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9e8e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e9815aaa0 0x7f3e98158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9e8e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 0x7f3e980c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9e8e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e980c3fc0 con 0x7f3e9815ae80 2023-10-16T14:36:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9e8e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9815e200 con 0x7f3e9815aaa0 2023-10-16T14:36:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9e8e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e981f2ae0 con 0x7f3e980ce390 2023-10-16T14:36:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9ce82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 0x7f3e980c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9ce82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 0x7f3e980c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40518/0 (socket says 172.21.15.29:40518) 2023-10-16T14:36:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.803+0000 7f3e9ce82640 1 -- 172.21.15.29:0/1725490808 learned_addr learned my addr 172.21.15.29:0/1725490808 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:58.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e97fff640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e980ce390 0x7f3e981585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e8ffff640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e9815aaa0 0x7f3e98158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9ce82640 1 -- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e980ce390 msgr2=0x7f3e981585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9ce82640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e980ce390 0x7f3e981585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9ce82640 1 -- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e9815aaa0 msgr2=0x7f3e98158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9ce82640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e9815aaa0 0x7f3e98158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9ce82640 1 -- 172.21.15.29:0/1725490808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e90009b30 con 0x7f3e9815ae80 2023-10-16T14:36:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e97fff640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e980ce390 0x7f3e981585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e8ffff640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e9815aaa0 0x7f3e98158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9ce82640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 0x7f3e980c43e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f3e90036040 tx=0x7f3e90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9003e450 con 0x7f3e9815ae80 2023-10-16T14:36:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9003e5d0 con 0x7f3e9815ae80 2023-10-16T14:36:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9003e980 con 0x7f3e9815ae80 2023-10-16T14:36:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.804+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e980c3620 con 0x7f3e9815ae80 2023-10-16T14:36:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.805+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e980c3be0 con 0x7f3e9815ae80 2023-10-16T14:36:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.806+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e980c4100 con 0x7f3e9815ae80 2023-10-16T14:36:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.807+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3e9003eae0 con 0x7f3e9815ae80 2023-10-16T14:36:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.807+0000 7f3e957fa640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3e7806af00 0x7f3e7806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.808+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f3e900ac8f0 con 0x7f3e9815ae80 2023-10-16T14:36:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.808+0000 7f3e97fff640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3e7806af00 0x7f3e7806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.809+0000 7f3e97fff640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3e7806af00 0x7f3e7806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e88006fd0 tx=0x7f3e88008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:58.812+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e980c4100 con 0x7f3e9815ae80 2023-10-16T14:36:59.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.008+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e980c4100 con 0x7f3e9815ae80 2023-10-16T14:36:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.011+0000 7f3e957fa640 1 -- 172.21.15.29:0/1725490808 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+17649 (secure 0 0 0) 0x7f3e980c4100 con 0x7f3e9815ae80 2023-10-16T14:36:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:36:59.015 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":144,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:36:55.479234+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:36:25.153525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T14:36:26.216898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3813450646":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1911194233":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/3477374804":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/601419330":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1163999179":"2023-10-17T14:36:55.479187+0000","172.21.15.170:6833/230171195":"2023-10-17T14:36:55.479187+0000","172.21.15.170:6832/230171195":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1143303444":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/2774967038":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2333704669":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1712463277":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2908562666":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/782892922":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6833/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6832/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1119524146":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1398880520":"2023-10-17T14:36:42.416389+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:36:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3e7806af00 msgr2=0x7f3e7806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3e7806af00 0x7f3e7806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e88006fd0 tx=0x7f3e88008040 comp rx=0 tx=0).stop 2023-10-16T14:36:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 msgr2=0x7f3e980c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 0x7f3e980c43e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f3e90036040 tx=0x7f3e90004a40 comp rx=0 tx=0).stop 2023-10-16T14:36:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 shutdown_connections 2023-10-16T14:36:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3e7806af00 0x7f3e7806d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e9815ae80 0x7f3e980c43e0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e9815aaa0 0x7f3e98158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 --2- 172.21.15.29:0/1725490808 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e980ce390 0x7f3e981585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 >> 172.21.15.29:0/1725490808 conn(0x7f3e980cb410 msgr2=0x7f3e980acaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 shutdown_connections 2023-10-16T14:36:59.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.014+0000 7f3e9e8e5640 1 -- 172.21.15.29:0/1725490808 wait complete. 2023-10-16T14:36:59.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:36:59.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.261+0000 7f3c951a7640 1 Processor -- start 2023-10-16T14:36:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.261+0000 7f3c951a7640 1 -- start start 2023-10-16T14:36:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c951a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 0x7f3c900cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c951a7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9015b9d0 con 0x7f3c900ccfb0 2023-10-16T14:36:59.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c951a7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9015bcb0 con 0x7f3c9006c040 2023-10-16T14:36:59.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c951a7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9015bf90 con 0x7f3c900ad970 2023-10-16T14:36:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c949a6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3c900ccfb0 0x7f3c900cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48400/0 (socket says 172.21.15.29:48400) 2023-10-16T14:36:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c949a6640 1 -- 172.21.15.29:0/3505730699 learned_addr learned my addr 172.21.15.29:0/3505730699 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.263+0000 7f3c87fff640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 0x7f3c900cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928897989 0 0) 0x7f3c9015b9d0 con 0x7f3c900ccfb0 2023-10-16T14:36:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c7c003520 con 0x7f3c900ccfb0 2023-10-16T14:36:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656002951 0 0) 0x7f3c7c003520 con 0x7f3c900ccfb0 2023-10-16T14:36:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 msgr2=0x7f3c900cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 0x7f3c900cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 >> v1:172.21.15.170:6789/0 conn(0x7f3c9006c040 legacy=0x7f3c9006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900cc270 con 0x7f3c900ccfb0 2023-10-16T14:36:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3c880019f0 con 0x7f3c900ccfb0 2023-10-16T14:36:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 >> v1:172.21.15.29:6789/0 conn(0x7f3c900ccfb0 legacy=0x7f3c900cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 0x7f3c7c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c7c003f20 0x7f3c7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c004d00 0x7f3c7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9015bf90 con 0x7f3c7c004d00 2023-10-16T14:36:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c00ba60 con 0x7f3c7c003f20 2023-10-16T14:36:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.264+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c00bc20 con 0x7f3c900ad970 2023-10-16T14:36:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c949a6640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c004d00 0x7f3c7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c87fff640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c7c003f20 0x7f3c7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c8ffff640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 0x7f3c7c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c949a6640 1 -- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 msgr2=0x7f3c7c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c949a6640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c900ad970 0x7f3c7c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c949a6640 1 -- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c7c003f20 msgr2=0x7f3c7c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c949a6640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c7c003f20 0x7f3c7c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c949a6640 1 -- 172.21.15.29:0/3505730699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c003520 con 0x7f3c7c004d00 2023-10-16T14:36:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.265+0000 7f3c87fff640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c7c003f20 0x7f3c7c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.266+0000 7f3c949a6640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c004d00 0x7f3c7c005180 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800a830 tx=0x7f3c7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d779b853929ab90 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.266+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c880072d0 con 0x7f3c7c004d00 2023-10-16T14:36:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.266+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8802f070 con 0x7f3c7c004d00 2023-10-16T14:36:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.266+0000 7f3c8f7fe640 1 -- 172.21.15.29:0/3505730699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c880076f0 con 0x7f3c7c004d00 2023-10-16T14:36:59.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c951a7640 1 -- 172.21.15.29:0/3505730699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c004d00 msgr2=0x7f3c7c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c951a7640 1 --2- 172.21.15.29:0/3505730699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c7c004d00 0x7f3c7c005180 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800a830 tx=0x7f3c7c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c949a6640 1 -- 172.21.15.29:0/3505730699 reap_dead start 2023-10-16T14:36:59.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c951a7640 1 -- 172.21.15.29:0/3505730699 shutdown_connections 2023-10-16T14:36:59.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c951a7640 1 -- 172.21.15.29:0/3505730699 >> 172.21.15.29:0/3505730699 conn(0x7f3c900ce6a0 msgr2=0x7f3c900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:59.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c951a7640 1 -- 172.21.15.29:0/3505730699 shutdown_connections 2023-10-16T14:36:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.267+0000 7f3c951a7640 1 -- 172.21.15.29:0/3505730699 wait complete. 2023-10-16T14:36:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.269+0000 7f3c951a7640 1 Processor -- start 2023-10-16T14:36:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.269+0000 7f3c951a7640 1 -- start start 2023-10-16T14:36:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.270+0000 7f3c951a7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c901ed990 0x7f3c9014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.270+0000 7f3c951a7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c90158d00 0x7f3c90151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.270+0000 7f3c951a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 0x7f3c90152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.270+0000 7f3c951a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901591c0 con 0x7f3c901edd70 2023-10-16T14:36:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.270+0000 7f3c951a7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901594a0 con 0x7f3c90158d00 2023-10-16T14:36:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.270+0000 7f3c951a7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90159780 con 0x7f3c901ed990 2023-10-16T14:36:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 0x7f3c90152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 0x7f3c90152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40536/0 (socket says 172.21.15.29:40536) 2023-10-16T14:36:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 -- 172.21.15.29:0/2748027846 learned_addr learned my addr 172.21.15.29:0/2748027846 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c8ffff640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c901ed990 0x7f3c9014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c87fff640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c90158d00 0x7f3c90151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 -- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c901ed990 msgr2=0x7f3c9014ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c901ed990 0x7f3c9014ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 -- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c90158d00 msgr2=0x7f3c90151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c90158d00 0x7f3c90151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 -- 172.21.15.29:0/2748027846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900cdcd0 con 0x7f3c901edd70 2023-10-16T14:36:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.271+0000 7f3c949a6640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 0x7f3c90152c20 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f3c8802cb70 tx=0x7f3c88001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.272+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88044070 con 0x7f3c901edd70 2023-10-16T14:36:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.272+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c88001250 con 0x7f3c901edd70 2023-10-16T14:36:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.272+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c880074e0 con 0x7f3c901edd70 2023-10-16T14:36:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.272+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c90159a60 con 0x7f3c901edd70 2023-10-16T14:36:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.272+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c90152090 con 0x7f3c901edd70 2023-10-16T14:36:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.273+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9006b330 con 0x7f3c901edd70 2023-10-16T14:36:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.274+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3c88031030 con 0x7f3c901edd70 2023-10-16T14:36:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.275+0000 7f3c8d7fa640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3c7006af00 0x7f3c7006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.275+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f3c880aba60 con 0x7f3c901edd70 2023-10-16T14:36:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.276+0000 7f3c8ffff640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3c7006af00 0x7f3c7006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.277+0000 7f3c8ffff640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3c7006af00 0x7f3c7006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c80007fd0 tx=0x7f3c80009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.279+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c88076730 con 0x7f3c901edd70 2023-10-16T14:36:59.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.514+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c9006b330 con 0x7f3c901edd70 2023-10-16T14:36:59.518 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T14:36:59.515+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:36:59.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.519+0000 7f3c8d7fa640 1 -- 172.21.15.29:0/2748027846 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3c88079d50 con 0x7f3c901edd70 2023-10-16T14:36:59.522 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:36:59.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.522+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3c7006af00 msgr2=0x7f3c7006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.522+0000 7f3c951a7640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3c7006af00 0x7f3c7006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c80007fd0 tx=0x7f3c80009040 comp rx=0 tx=0).stop 2023-10-16T14:36:59.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.522+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 msgr2=0x7f3c90152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.522+0000 7f3c951a7640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 0x7f3c90152c20 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f3c8802cb70 tx=0x7f3c88001d80 comp rx=0 tx=0).stop 2023-10-16T14:36:59.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 shutdown_connections 2023-10-16T14:36:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3c7006af00 0x7f3c7006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c901edd70 0x7f3c90152c20 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c90158d00 0x7f3c90151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 --2- 172.21.15.29:0/2748027846 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c901ed990 0x7f3c9014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 >> 172.21.15.29:0/2748027846 conn(0x7f3c900ce6a0 msgr2=0x7f3c9006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 shutdown_connections 2023-10-16T14:36:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.523+0000 7f3c951a7640 1 -- 172.21.15.29:0/2748027846 wait complete. 2023-10-16T14:36:59.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:36:59.547 INFO:tasks.ceph.mds.a.smithi029.stderr: -8804> 2023-10-16T14:36:32.301+0000 7fc09189c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:36:59.615 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:36:59.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.740+0000 7f934bfff640 1 Processor -- start 2023-10-16T14:36:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.740+0000 7f934bfff640 1 -- start start 2023-10-16T14:36:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.741+0000 7f934bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f934c06c040 0x7f934c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f934c0ad970 0x7f934c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c06c960 con 0x7f934c15ebc0 2023-10-16T14:36:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c15b9d0 con 0x7f934c06c040 2023-10-16T14:36:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c15bcb0 con 0x7f934c0ad970 2023-10-16T14:36:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f934c15ebc0 0x7f934c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48420/0 (socket says 172.21.15.29:48420) 2023-10-16T14:36:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934b7fe640 1 -- 172.21.15.29:0/4076081135 learned_addr learned my addr 172.21.15.29:0/4076081135 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:36:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f934affd640 1 --2- 172.21.15.29:0/4076081135 >> v2:172.21.15.29:3300/0 conn(0x7f934c06c040 0x7f934c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.742+0000 7f9343fff640 1 --2- 172.21.15.29:0/4076081135 >> v2:172.21.15.170:3300/0 conn(0x7f934c0ad970 0x7f934c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f934a7fc640 1 -- 172.21.15.29:0/4076081135 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63004686 0 0) 0x7f934c06c960 con 0x7f934c15ebc0 2023-10-16T14:36:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f934a7fc640 1 -- 172.21.15.29:0/4076081135 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9338003520 con 0x7f934c15ebc0 2023-10-16T14:36:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f934affd640 1 -- 172.21.15.29:0/4076081135 >> v2:172.21.15.170:3300/0 conn(0x7f934c0ad970 msgr2=0x7f934c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:36:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f934affd640 1 --2- 172.21.15.29:0/4076081135 >> v2:172.21.15.170:3300/0 conn(0x7f934c0ad970 0x7f934c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f934affd640 1 -- 172.21.15.29:0/4076081135 >> v1:172.21.15.29:6789/0 conn(0x7f934c15ebc0 legacy=0x7f934c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f934affd640 1 -- 172.21.15.29:0/4076081135 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c15bf90 con 0x7f934c06c040 2023-10-16T14:36:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.743+0000 7f9343fff640 1 --2- 172.21.15.29:0/4076081135 >> v2:172.21.15.170:3300/0 conn(0x7f934c0ad970 0x7f934c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:36:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.744+0000 7f934affd640 1 --2- 172.21.15.29:0/4076081135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c06c040 0x7f934c06c420 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f934c0cb060 tx=0x7f933c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b8b623409ae5e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.744+0000 7f934a7fc640 1 -- 172.21.15.29:0/4076081135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c031830 con 0x7f934c06c040 2023-10-16T14:36:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.744+0000 7f934a7fc640 1 -- 172.21.15.29:0/4076081135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933c031990 con 0x7f934c06c040 2023-10-16T14:36:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.744+0000 7f934a7fc640 1 -- 172.21.15.29:0/4076081135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c031d00 con 0x7f934c06c040 2023-10-16T14:36:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 -- 172.21.15.29:0/4076081135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c06c040 msgr2=0x7f934c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 --2- 172.21.15.29:0/4076081135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c06c040 0x7f934c06c420 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f934c0cb060 tx=0x7f933c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 -- 172.21.15.29:0/4076081135 shutdown_connections 2023-10-16T14:36:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 --2- 172.21.15.29:0/4076081135 >> v2:172.21.15.170:3300/0 conn(0x7f934c0ad970 0x7f934c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 --2- 172.21.15.29:0/4076081135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c06c040 0x7f934c06c420 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 -- 172.21.15.29:0/4076081135 >> 172.21.15.29:0/4076081135 conn(0x7f934c0ce6a0 msgr2=0x7f934c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:36:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 -- 172.21.15.29:0/4076081135 shutdown_connections 2023-10-16T14:36:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.745+0000 7f934bfff640 1 -- 172.21.15.29:0/4076081135 wait complete. 2023-10-16T14:36:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.747+0000 7f934bfff640 1 Processor -- start 2023-10-16T14:36:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.747+0000 7f934bfff640 1 -- start start 2023-10-16T14:36:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f934c06c040 0x7f934c0ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 0x7f934c0acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934c15ebc0 0x7f934c1f0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0ae5f0 con 0x7f934c0ad970 2023-10-16T14:36:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0a8480 con 0x7f934c06c040 2023-10-16T14:36:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c1f0f90 con 0x7f934c15ebc0 2023-10-16T14:36:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f9343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 0x7f934c0acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f9343fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 0x7f934c0acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40566/0 (socket says 172.21.15.29:40566) 2023-10-16T14:36:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f9343fff640 1 -- 172.21.15.29:0/606342961 learned_addr learned my addr 172.21.15.29:0/606342961 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:36:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934affd640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f934c06c040 0x7f934c0ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f934b7fe640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934c15ebc0 0x7f934c1f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.748+0000 7f9343fff640 1 -- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934c15ebc0 msgr2=0x7f934c1f0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f9343fff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934c15ebc0 0x7f934c1f0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f9343fff640 1 -- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f934c06c040 msgr2=0x7f934c0ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:36:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f9343fff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f934c06c040 0x7f934c0ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:36:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f9343fff640 1 -- 172.21.15.29:0/606342961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c009400 con 0x7f934c0ad970 2023-10-16T14:36:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f934affd640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f934c06c040 0x7f934c0ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:36:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f9343fff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 0x7f934c0acda0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f933400d7d0 tx=0x7f933400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340096a0 con 0x7f934c0ad970 2023-10-16T14:36:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334009800 con 0x7f934c0ad970 2023-10-16T14:36:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334009bb0 con 0x7f934c0ad970 2023-10-16T14:36:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c1f1270 con 0x7f934c0ad970 2023-10-16T14:36:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.749+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c0a8a00 con 0x7f934c0ad970 2023-10-16T14:36:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.750+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c06b330 con 0x7f934c0ad970 2023-10-16T14:36:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.751+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9334009d10 con 0x7f934c0ad970 2023-10-16T14:36:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.752+0000 7f93437fe640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932c06af90 0x7f932c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:36:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.752+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6554+0+0 (secure 0 0 0) 0x7f9334089fa0 con 0x7f934c0ad970 2023-10-16T14:36:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.753+0000 7f934affd640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932c06af90 0x7f932c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:36:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.754+0000 7f934affd640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932c06af90 0x7f932c06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f933c036000 tx=0x7f933c03c040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:36:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.757+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9334054be0 con 0x7f934c0ad970 2023-10-16T14:36:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:36:59.981+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f934c0a8b90 con 0x7f934c0ad970 2023-10-16T14:37:00.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.535+0000 7f93437fe640 1 -- 172.21.15.29:0/606342961 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9334058200 con 0x7f934c0ad970 2023-10-16T14:37:00.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.538+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932c06af90 msgr2=0x7f932c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:00.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.538+0000 7f934bfff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932c06af90 0x7f932c06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f933c036000 tx=0x7f933c03c040 comp rx=0 tx=0).stop 2023-10-16T14:37:00.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.538+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 msgr2=0x7f934c0acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:00.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.538+0000 7f934bfff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 0x7f934c0acda0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f933400d7d0 tx=0x7f933400dc90 comp rx=0 tx=0).stop 2023-10-16T14:37:00.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 shutdown_connections 2023-10-16T14:37:00.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932c06af90 0x7f932c06d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f934c0ad970 0x7f934c0acda0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f934c15ebc0 0x7f934c1f0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 --2- 172.21.15.29:0/606342961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f934c06c040 0x7f934c0ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 >> 172.21.15.29:0/606342961 conn(0x7f934c0ce6a0 msgr2=0x7f934c0cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:00.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 shutdown_connections 2023-10-16T14:37:00.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.539+0000 7f934bfff640 1 -- 172.21.15.29:0/606342961 wait complete. 2023-10-16T14:37:00.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:37:00.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.787+0000 7f9b6ad21640 1 Processor -- start 2023-10-16T14:37:00.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.787+0000 7f9b6ad21640 1 -- start start 2023-10-16T14:37:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6ad21640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b6415b320 0x7f9b640cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6ad21640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6415b9d0 con 0x7f9b640ad970 2023-10-16T14:37:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6ad21640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6415bcb0 con 0x7f9b6406c040 2023-10-16T14:37:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6ad21640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b6415b7d0 con 0x7f9b6415b320 2023-10-16T14:37:00.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6951e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b640ad970 0x7f9b640cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48436/0 (socket says 172.21.15.29:48436) 2023-10-16T14:37:00.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6951e640 1 -- 172.21.15.29:0/942866250 learned_addr learned my addr 172.21.15.29:0/942866250 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:00.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.789+0000 7f9b6a520640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b6415b320 0x7f9b640cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:00.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.790+0000 7f9b6a520640 1 -- 172.21.15.29:0/942866250 >> v1:172.21.15.170:6789/0 conn(0x7f9b6406c040 legacy=0x7f9b6406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:00.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.790+0000 7f9b68d1d640 1 -- 172.21.15.29:0/942866250 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292857443 0 0) 0x7f9b6415b9d0 con 0x7f9b640ad970 2023-10-16T14:37:00.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.790+0000 7f9b6a520640 1 -- 172.21.15.29:0/942866250 >> v1:172.21.15.29:6789/0 conn(0x7f9b640ad970 legacy=0x7f9b640cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:00.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.790+0000 7f9b6a520640 1 -- 172.21.15.29:0/942866250 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6415bf90 con 0x7f9b6415b320 2023-10-16T14:37:00.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.791+0000 7f9b6a520640 1 --2- 172.21.15.29:0/942866250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b6415b320 0x7f9b640cd180 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009dc0 tx=0x7f9b5c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87cc0245a6248f6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:00.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.791+0000 7f9b68d1d640 1 -- 172.21.15.29:0/942866250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c02cc40 con 0x7f9b6415b320 2023-10-16T14:37:00.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.791+0000 7f9b68d1d640 1 -- 172.21.15.29:0/942866250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5c02cda0 con 0x7f9b6415b320 2023-10-16T14:37:00.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.791+0000 7f9b68d1d640 1 -- 172.21.15.29:0/942866250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c034660 con 0x7f9b6415b320 2023-10-16T14:37:00.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 -- 172.21.15.29:0/942866250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b6415b320 msgr2=0x7f9b640cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:00.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/942866250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b6415b320 0x7f9b640cd180 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009dc0 tx=0x7f9b5c02be90 comp rx=0 tx=0).stop 2023-10-16T14:37:00.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 -- 172.21.15.29:0/942866250 shutdown_connections 2023-10-16T14:37:00.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/942866250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b6415b320 0x7f9b640cd180 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 -- 172.21.15.29:0/942866250 >> 172.21.15.29:0/942866250 conn(0x7f9b640ce6a0 msgr2=0x7f9b640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:00.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 -- 172.21.15.29:0/942866250 shutdown_connections 2023-10-16T14:37:00.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.792+0000 7f9b6ad21640 1 -- 172.21.15.29:0/942866250 wait complete. 2023-10-16T14:37:00.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.794+0000 7f9b6ad21640 1 Processor -- start 2023-10-16T14:37:00.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.794+0000 7f9b6ad21640 1 -- start start 2023-10-16T14:37:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6ad21640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b6406c040 0x7f9b64156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6ad21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 0x7f9b640afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6ad21640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b6415b320 0x7f9b640aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6ad21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64156ec0 con 0x7f9b640ad970 2023-10-16T14:37:00.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6ad21640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640ae7e0 con 0x7f9b6406c040 2023-10-16T14:37:00.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6ad21640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640b01b0 con 0x7f9b6415b320 2023-10-16T14:37:00.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6951e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 0x7f9b640afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:00.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6951e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 0x7f9b640afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40582/0 (socket says 172.21.15.29:40582) 2023-10-16T14:37:00.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.795+0000 7f9b6951e640 1 -- 172.21.15.29:0/181335277 learned_addr learned my addr 172.21.15.29:0/181335277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:00.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6951e640 1 -- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b6415b320 msgr2=0x7f9b640aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:00.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b69d1f640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b6406c040 0x7f9b64156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:00.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6951e640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b6415b320 0x7f9b640aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6951e640 1 -- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b6406c040 msgr2=0x7f9b64156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:00.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6951e640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b6406c040 0x7f9b64156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:00.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6951e640 1 -- 172.21.15.29:0/181335277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c009a70 con 0x7f9b640ad970 2023-10-16T14:37:00.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6951e640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 0x7f9b640afc70 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400c4e0 tx=0x7f9b5400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:00.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b540074e0 con 0x7f9b640ad970 2023-10-16T14:37:00.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b54007660 con 0x7f9b640ad970 2023-10-16T14:37:00.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54004a20 con 0x7f9b640ad970 2023-10-16T14:37:00.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.796+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b640adef0 con 0x7f9b640ad970 2023-10-16T14:37:00.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.797+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b640ae470 con 0x7f9b640ad970 2023-10-16T14:37:00.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.798+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6406b330 con 0x7f9b640ad970 2023-10-16T14:37:00.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.799+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b54004b80 con 0x7f9b640ad970 2023-10-16T14:37:00.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.799+0000 7f9b5a7fc640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f9b4806af90 0x7f9b4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:00.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.800+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6554+0+0 (secure 0 0 0) 0x7f9b54088cd0 con 0x7f9b640ad970 2023-10-16T14:37:00.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.800+0000 7f9b69d1f640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f9b4806af90 0x7f9b4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:00.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.801+0000 7f9b69d1f640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f9b4806af90 0x7f9b4806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000a750 tx=0x7f9b60008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:00.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:00.804+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b54053a70 con 0x7f9b640ad970 2023-10-16T14:37:01.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.002+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9b640ae990 con 0x7f9b640ad970 2023-10-16T14:37:01.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.557+0000 7f9b5a7fc640 1 -- 172.21.15.29:0/181335277 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v146) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9b640ae990 con 0x7f9b640ad970 2023-10-16T14:37:01.560 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:37:01.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.561+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f9b4806af90 msgr2=0x7f9b4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.561+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f9b4806af90 0x7f9b4806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000a750 tx=0x7f9b60008040 comp rx=0 tx=0).stop 2023-10-16T14:37:01.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.561+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 msgr2=0x7f9b640afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.561+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 0x7f9b640afc70 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400c4e0 tx=0x7f9b5400c9a0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 shutdown_connections 2023-10-16T14:37:01.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f9b4806af90 0x7f9b4806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b640ad970 0x7f9b640afc70 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9b6415b320 0x7f9b640aec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 --2- 172.21.15.29:0/181335277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9b6406c040 0x7f9b64156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 >> 172.21.15.29:0/181335277 conn(0x7f9b640ce6a0 msgr2=0x7f9b640cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:01.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 shutdown_connections 2023-10-16T14:37:01.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.563+0000 7f9b6ad21640 1 -- 172.21.15.29:0/181335277 wait complete. 2023-10-16T14:37:01.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:37:01.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.819+0000 7fb6d0ac3640 1 Processor -- start 2023-10-16T14:37:01.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.820+0000 7fb6d0ac3640 1 -- start start 2023-10-16T14:37:01.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.821+0000 7fb6d0ac3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6cc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.821+0000 7fb6d0ac3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc15b9d0 con 0x7fb6cc0ad970 2023-10-16T14:37:01.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.821+0000 7fb6d0ac3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc15bcb0 con 0x7fb6cc06c040 2023-10-16T14:37:01.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.821+0000 7fb6d0ac3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc15b7d0 con 0x7fb6cc15b320 2023-10-16T14:37:01.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.821+0000 7fb6c3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6cc0ad970 0x7fb6cc0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48438/0 (socket says 172.21.15.29:48438) 2023-10-16T14:37:01.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.821+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3536080226 learned_addr learned my addr 172.21.15.29:0/3536080226 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:01.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002985207 0 0) 0x7fb6cc15b9d0 con 0x7fb6cc0ad970 2023-10-16T14:37:01.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6cbfff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6cc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b8003520 con 0x7fb6cc0ad970 2023-10-16T14:37:01.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1797891102 0 0) 0x7fb6b8003520 con 0x7fb6cc0ad970 2023-10-16T14:37:01.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 msgr2=0x7fb6cc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:01.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6cc0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 >> v1:172.21.15.170:6789/0 conn(0x7fb6cc06c040 legacy=0x7fb6cc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc15bf90 con 0x7fb6cc0ad970 2023-10-16T14:37:01.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fb6b4001670 con 0x7fb6cc0ad970 2023-10-16T14:37:01.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.822+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 >> v1:172.21.15.29:6789/0 conn(0x7fb6cc0ad970 legacy=0x7fb6cc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6caffd640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6b80039a0 0x7fb6b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6caffd640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80047c0 0x7fb6b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6caffd640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc15b7d0 con 0x7fb6b80047c0 2023-10-16T14:37:01.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b800bae0 con 0x7fb6b80039a0 2023-10-16T14:37:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b800bca0 con 0x7fb6cc15b320 2023-10-16T14:37:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80047c0 0x7fb6b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6cb7fe640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6b80039a0 0x7fb6b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6cbfff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 msgr2=0x7fb6b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6b80039a0 msgr2=0x7fb6b8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6b80039a0 0x7fb6b8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3536080226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8003520 con 0x7fb6b80047c0 2023-10-16T14:37:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.823+0000 7fb6cbfff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6cc15b320 0x7fb6b80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80047c0 0x7fb6b8004c40 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb6b400c210 tx=0x7fb6b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cec8d5a08b7a1fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4005ad0 con 0x7fb6b80047c0 2023-10-16T14:37:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b4005c30 con 0x7fb6b80047c0 2023-10-16T14:37:01.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6caffd640 1 -- 172.21.15.29:0/3536080226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b402c890 con 0x7fb6b80047c0 2023-10-16T14:37:01.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3536080226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80047c0 msgr2=0x7fb6b8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3536080226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6b80047c0 0x7fb6b8004c40 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb6b400c210 tx=0x7fb6b80057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.825+0000 7fb6cbfff640 1 -- 172.21.15.29:0/3536080226 reap_dead start 2023-10-16T14:37:01.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.826+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3536080226 shutdown_connections 2023-10-16T14:37:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.826+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3536080226 >> 172.21.15.29:0/3536080226 conn(0x7fb6cc0ce6a0 msgr2=0x7fb6cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.826+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3536080226 shutdown_connections 2023-10-16T14:37:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.826+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3536080226 wait complete. 2023-10-16T14:37:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.827+0000 7fb6d0ac3640 1 Processor -- start 2023-10-16T14:37:01.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 -- start start 2023-10-16T14:37:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6b80047c0 0x7fb6cc14ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 0x7fb6cc0c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc157c90 0x7fb6cc158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc158680 con 0x7fb6cc0c6060 2023-10-16T14:37:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0c6a20 con 0x7fb6cc157c90 2023-10-16T14:37:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.828+0000 7fb6d0ac3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc15cd80 con 0x7fb6b80047c0 2023-10-16T14:37:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 0x7fb6cc0c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 0x7fb6cc0c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40598/0 (socket says 172.21.15.29:40598) 2023-10-16T14:37:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3021406521 learned_addr learned my addr 172.21.15.29:0/3021406521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6cbfff640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc157c90 0x7fb6cc158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6b80047c0 msgr2=0x7fb6cc14ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6b80047c0 0x7fb6cc14ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc157c90 msgr2=0x7fb6cc158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc157c90 0x7fb6cc158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.829+0000 7fb6c3fff640 1 -- 172.21.15.29:0/3021406521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc15d060 con 0x7fb6cc0c6060 2023-10-16T14:37:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.830+0000 7fb6c3fff640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 0x7fb6cc0c64e0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb6b402c430 tx=0x7fb6b4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.830+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4039030 con 0x7fb6cc0c6060 2023-10-16T14:37:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.830+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b402cca0 con 0x7fb6cc0c6060 2023-10-16T14:37:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.830+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b403c040 con 0x7fb6cc0c6060 2023-10-16T14:37:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.830+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc15d340 con 0x7fb6cc0c6060 2023-10-16T14:37:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.830+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc15d900 con 0x7fb6cc0c6060 2023-10-16T14:37:01.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.831+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc06b330 con 0x7fb6cc0c6060 2023-10-16T14:37:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.832+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb6b4032070 con 0x7fb6cc0c6060 2023-10-16T14:37:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.833+0000 7fb6c8ff9640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb6ac06afc0 0x7fb6ac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.834+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fb6b40aae40 con 0x7fb6cc0c6060 2023-10-16T14:37:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.834+0000 7fb6cb7fe640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb6ac06afc0 0x7fb6ac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.835+0000 7fb6cb7fe640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb6ac06afc0 0x7fb6ac06d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc007fd0 tx=0x7fb6bc009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:01.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:01.838+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6b4075ba0 con 0x7fb6cc0c6060 2023-10-16T14:37:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.034+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb6cc04e9b0 con 0x7fb6cc0c6060 2023-10-16T14:37:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.560+0000 7fb6c8ff9640 1 -- 172.21.15.29:0/3021406521 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v147) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb6b40791c0 con 0x7fb6cc0c6060 2023-10-16T14:37:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T14:37:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.566+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb6ac06afc0 msgr2=0x7fb6ac06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.566+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb6ac06afc0 0x7fb6ac06d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc007fd0 tx=0x7fb6bc009040 comp rx=0 tx=0).stop 2023-10-16T14:37:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.566+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 msgr2=0x7fb6cc0c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.566+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 0x7fb6cc0c64e0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb6b402c430 tx=0x7fb6b4001a00 comp rx=0 tx=0).stop 2023-10-16T14:37:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 shutdown_connections 2023-10-16T14:37:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb6ac06afc0 0x7fb6ac06d480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6cc0c6060 0x7fb6cc0c64e0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb6cc157c90 0x7fb6cc158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 --2- 172.21.15.29:0/3021406521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6b80047c0 0x7fb6cc14ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 >> 172.21.15.29:0/3021406521 conn(0x7fb6cc0ce6a0 msgr2=0x7fb6cc06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 shutdown_connections 2023-10-16T14:37:02.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.567+0000 7fb6d0ac3640 1 -- 172.21.15.29:0/3021406521 wait complete. 2023-10-16T14:37:02.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:37:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.812+0000 7f08fd7d0640 1 Processor -- start 2023-10-16T14:37:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.812+0000 7f08fd7d0640 1 -- start start 2023-10-16T14:37:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fd7d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f08f806c040 0x7f08f806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fd7d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f08f80ad970 0x7f08f815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fd7d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f806c960 con 0x7f08f815ebc0 2023-10-16T14:37:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fd7d0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f815b9d0 con 0x7f08f80ad970 2023-10-16T14:37:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fd7d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f815bcb0 con 0x7f08f806c040 2023-10-16T14:37:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08effff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f08f80ad970 0x7f08f815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fcfcf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f08f815ebc0 0x7f08f80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48442/0 (socket says 172.21.15.29:48442) 2023-10-16T14:37:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.814+0000 7f08fcfcf640 1 -- 172.21.15.29:0/3103984711 learned_addr learned my addr 172.21.15.29:0/3103984711 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.815+0000 7f08f7fff640 1 --2- 172.21.15.29:0/3103984711 >> v2:172.21.15.170:3300/0 conn(0x7f08f806c040 0x7f08f806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.815+0000 7f08effff640 1 -- 172.21.15.29:0/3103984711 >> v2:172.21.15.170:3300/0 conn(0x7f08f806c040 msgr2=0x7f08f806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.815+0000 7f08effff640 1 --2- 172.21.15.29:0/3103984711 >> v2:172.21.15.170:3300/0 conn(0x7f08f806c040 0x7f08f806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.815+0000 7f08effff640 1 -- 172.21.15.29:0/3103984711 >> v1:172.21.15.29:6789/0 conn(0x7f08f815ebc0 legacy=0x7f08f80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.815+0000 7f08effff640 1 -- 172.21.15.29:0/3103984711 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f815bf90 con 0x7f08f80ad970 2023-10-16T14:37:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.815+0000 7f08f7fff640 1 --2- 172.21.15.29:0/3103984711 >> v2:172.21.15.170:3300/0 conn(0x7f08f806c040 0x7f08f806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.816+0000 7f08effff640 1 --2- 172.21.15.29:0/3103984711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f80ad970 0x7f08f815e680 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f08f80cf350 tx=0x7f08e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abcf3aecb5629a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08f77fe640 1 -- 172.21.15.29:0/3103984711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0031830 con 0x7f08f80ad970 2023-10-16T14:37:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08f77fe640 1 -- 172.21.15.29:0/3103984711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e0031990 con 0x7f08f80ad970 2023-10-16T14:37:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08f77fe640 1 -- 172.21.15.29:0/3103984711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0031d20 con 0x7f08f80ad970 2023-10-16T14:37:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 -- 172.21.15.29:0/3103984711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f80ad970 msgr2=0x7f08f815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/3103984711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f80ad970 0x7f08f815e680 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f08f80cf350 tx=0x7f08e002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 -- 172.21.15.29:0/3103984711 shutdown_connections 2023-10-16T14:37:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/3103984711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f80ad970 0x7f08f815e680 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/3103984711 >> v2:172.21.15.170:3300/0 conn(0x7f08f806c040 0x7f08f806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 -- 172.21.15.29:0/3103984711 >> 172.21.15.29:0/3103984711 conn(0x7f08f80ce6a0 msgr2=0x7f08f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 -- 172.21.15.29:0/3103984711 shutdown_connections 2023-10-16T14:37:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.817+0000 7f08fd7d0640 1 -- 172.21.15.29:0/3103984711 wait complete. 2023-10-16T14:37:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.819+0000 7f08fd7d0640 1 Processor -- start 2023-10-16T14:37:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.819+0000 7f08fd7d0640 1 -- start start 2023-10-16T14:37:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.820+0000 7f08fd7d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 0x7f08f8154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.820+0000 7f08fd7d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f80ad970 0x7f08f80ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.820+0000 7f08fd7d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08f815ebc0 0x7f08f80aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.820+0000 7f08fd7d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80b0320 con 0x7f08f806c040 2023-10-16T14:37:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.820+0000 7f08fd7d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80af4e0 con 0x7f08f80ad970 2023-10-16T14:37:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.820+0000 7f08fd7d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80af7a0 con 0x7f08f815ebc0 2023-10-16T14:37:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 0x7f08f8154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 0x7f08f8154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40608/0 (socket says 172.21.15.29:40608) 2023-10-16T14:37:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 -- 172.21.15.29:0/2812199126 learned_addr learned my addr 172.21.15.29:0/2812199126 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 -- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08f815ebc0 msgr2=0x7f08f80aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08f815ebc0 0x7f08f80aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 -- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f80ad970 msgr2=0x7f08f80ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f80ad970 0x7f08f80ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 -- 172.21.15.29:0/2812199126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e0009400 con 0x7f08f806c040 2023-10-16T14:37:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.821+0000 7f08f7fff640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 0x7f08f8154d30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f08e800d7d0 tx=0x7f08e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.822+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e8004030 con 0x7f08f806c040 2023-10-16T14:37:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.822+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e8004190 con 0x7f08f806c040 2023-10-16T14:37:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.822+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e80096a0 con 0x7f08f806c040 2023-10-16T14:37:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.822+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f80afa80 con 0x7f08f806c040 2023-10-16T14:37:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.822+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f80b0040 con 0x7f08f806c040 2023-10-16T14:37:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.823+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f806b330 con 0x7f08f806c040 2023-10-16T14:37:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.824+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08e8009800 con 0x7f08f806c040 2023-10-16T14:37:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.825+0000 7f08f57fa640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f08d806af90 0x7f08d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.825+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f08e8089980 con 0x7f08f806c040 2023-10-16T14:37:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.826+0000 7f08effff640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f08d806af90 0x7f08d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.827+0000 7f08effff640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f08d806af90 0x7f08d806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f08e0036000 tx=0x7f08e0030000 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:02.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:02.829+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08e8054910 con 0x7f08f806c040 2023-10-16T14:37:03.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.022+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f08f80cc550 con 0x7f08f806c040 2023-10-16T14:37:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.029+0000 7f08f57fa640 1 -- 172.21.15.29:0/2812199126 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7f08e8057f30 con 0x7f08f806c040 2023-10-16T14:37:03.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f08d806af90 msgr2=0x7f08d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f08d806af90 0x7f08d806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f08e0036000 tx=0x7f08e0030000 comp rx=0 tx=0).stop 2023-10-16T14:37:03.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 msgr2=0x7f08f8154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 0x7f08f8154d30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f08e800d7d0 tx=0x7f08e800dc90 comp rx=0 tx=0).stop 2023-10-16T14:37:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 shutdown_connections 2023-10-16T14:37:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f08d806af90 0x7f08d806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f08f815ebc0 0x7f08f80aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08f80ad970 0x7f08f80ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 --2- 172.21.15.29:0/2812199126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08f806c040 0x7f08f8154d30 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 >> 172.21.15.29:0/2812199126 conn(0x7f08f80ce6a0 msgr2=0x7f08f80c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.032+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 shutdown_connections 2023-10-16T14:37:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.033+0000 7f08fd7d0640 1 -- 172.21.15.29:0/2812199126 wait complete. 2023-10-16T14:37:03.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:37:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.283+0000 7f74cb59e640 1 Processor -- start 2023-10-16T14:37:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.283+0000 7f74cb59e640 1 -- start start 2023-10-16T14:37:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74cb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74cc0b3ec0 0x7f74cc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b0ea0 0x7f74cc0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74cb59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f74cc0b43c0 0x7f74cc15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74cb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc06ca00 con 0x7f74cc0b3ec0 2023-10-16T14:37:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74cb59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc06be30 con 0x7f74cc0b43c0 2023-10-16T14:37:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74cb59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0ada10 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.285+0000 7f74ca59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74cc0b3ec0 0x7f74cc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74cc0b3ec0 0x7f74cc0cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40618/0 (socket says 172.21.15.29:40618) 2023-10-16T14:37:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 -- 172.21.15.29:0/143151792 learned_addr learned my addr 172.21.15.29:0/143151792 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74cad9d640 1 --2- 172.21.15.29:0/143151792 >> v2:172.21.15.170:3300/0 conn(0x7f74cc0b43c0 0x7f74cc15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 -- 172.21.15.29:0/143151792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b0ea0 msgr2=0x7f74cc0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 --2- 172.21.15.29:0/143151792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b0ea0 0x7f74cc0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 -- 172.21.15.29:0/143151792 >> v2:172.21.15.170:3300/0 conn(0x7f74cc0b43c0 msgr2=0x7f74cc15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 --2- 172.21.15.29:0/143151792 >> v2:172.21.15.170:3300/0 conn(0x7f74cc0b43c0 0x7f74cc15a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.286+0000 7f74ca59c640 1 -- 172.21.15.29:0/143151792 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc06c060 con 0x7f74cc0b3ec0 2023-10-16T14:37:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.287+0000 7f74ca59c640 1 --2- 172.21.15.29:0/143151792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b3ec0 0x7f74cc0cd260 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f74cc06d700 tx=0x7f74bc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8bd984f6bc1802b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.287+0000 7f74c9d9b640 1 -- 172.21.15.29:0/143151792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc02c9f0 con 0x7f74cc0b3ec0 2023-10-16T14:37:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74c9d9b640 1 -- 172.21.15.29:0/143151792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc02cb50 con 0x7f74cc0b3ec0 2023-10-16T14:37:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74c9d9b640 1 -- 172.21.15.29:0/143151792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0343f0 con 0x7f74cc0b3ec0 2023-10-16T14:37:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 -- 172.21.15.29:0/143151792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b3ec0 msgr2=0x7f74cc0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 --2- 172.21.15.29:0/143151792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b3ec0 0x7f74cc0cd260 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f74cc06d700 tx=0x7f74bc02be50 comp rx=0 tx=0).stop 2023-10-16T14:37:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 -- 172.21.15.29:0/143151792 shutdown_connections 2023-10-16T14:37:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 --2- 172.21.15.29:0/143151792 >> v2:172.21.15.170:3300/0 conn(0x7f74cc0b43c0 0x7f74cc15a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 --2- 172.21.15.29:0/143151792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b0ea0 0x7f74cc0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 --2- 172.21.15.29:0/143151792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b3ec0 0x7f74cc0cd260 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 -- 172.21.15.29:0/143151792 >> 172.21.15.29:0/143151792 conn(0x7f74cc0b0170 msgr2=0x7f74cc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.288+0000 7f74cb59e640 1 -- 172.21.15.29:0/143151792 shutdown_connections 2023-10-16T14:37:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.289+0000 7f74cb59e640 1 -- 172.21.15.29:0/143151792 wait complete. 2023-10-16T14:37:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.291+0000 7f74cb59e640 1 Processor -- start 2023-10-16T14:37:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.291+0000 7f74cb59e640 1 -- start start 2023-10-16T14:37:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 0x7f74cc15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b3ec0 0x7f74cc158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74cc0b43c0 0x7f74cc150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74cb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc158cd0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74cb59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0b1ad0 con 0x7f74cc0b43c0 2023-10-16T14:37:03.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74cb59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc151180 con 0x7f74cc0b3ec0 2023-10-16T14:37:03.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.292+0000 7f74ca59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 0x7f74cc15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 0x7f74cc15b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40628/0 (socket says 172.21.15.29:40628) 2023-10-16T14:37:03.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 -- 172.21.15.29:0/3543649564 learned_addr learned my addr 172.21.15.29:0/3543649564 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:03.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74c3fff640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b3ec0 0x7f74cc158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74cad9d640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74cc0b43c0 0x7f74cc150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 -- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b3ec0 msgr2=0x7f74cc158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b3ec0 0x7f74cc158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 -- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74cc0b43c0 msgr2=0x7f74cc150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74cc0b43c0 0x7f74cc150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 -- 172.21.15.29:0/3543649564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc009b30 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.293+0000 7f74ca59c640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 0x7f74cc15b650 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f74bc000f80 tx=0x7f74bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.294+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc03e450 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.294+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc03e5f0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.294+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc03e9a0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.294+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc0b1d20 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.294+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc0b22e0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.295+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc04e9b0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.297+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74bc03eb00 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.298+0000 7f74c2ffd640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f74ac06af90 0x7f74ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.298+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f74bc0ac6b0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.298+0000 7f74c3fff640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f74ac06af90 0x7f74ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.299+0000 7f74c3fff640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f74ac06af90 0x7f74ac06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74cc1f5cc0 tx=0x7f74b4008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:03.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.302+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74bc077820 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.512+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f74cc06abd0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.516+0000 7f74c2ffd640 1 -- 172.21.15.29:0/3543649564 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v50) v1 ==== 115+0+0 (secure 0 0 0) 0x7f74bc03edb0 con 0x7f74cc0b0ea0 2023-10-16T14:37:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f74ac06af90 msgr2=0x7f74ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f74ac06af90 0x7f74ac06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74cc1f5cc0 tx=0x7f74b4008040 comp rx=0 tx=0).stop 2023-10-16T14:37:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 msgr2=0x7f74cc15b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 0x7f74cc15b650 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f74bc000f80 tx=0x7f74bc004a40 comp rx=0 tx=0).stop 2023-10-16T14:37:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 shutdown_connections 2023-10-16T14:37:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f74ac06af90 0x7f74ac06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74cc0b43c0 0x7f74cc150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74cc0b3ec0 0x7f74cc158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 --2- 172.21.15.29:0/3543649564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74cc0b0ea0 0x7f74cc15b650 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.519+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 >> 172.21.15.29:0/3543649564 conn(0x7f74cc0b0170 msgr2=0x7f74cc0af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.520+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 shutdown_connections 2023-10-16T14:37:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.520+0000 7f74cb59e640 1 -- 172.21.15.29:0/3543649564 wait complete. 2023-10-16T14:37:03.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:37:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.766+0000 7f17d5860640 1 Processor -- start 2023-10-16T14:37:03.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.766+0000 7f17d5860640 1 -- start start 2023-10-16T14:37:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d5860640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d015cd70 0x7f17d00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d5860640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f17d00b8480 0x7f17d00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d5860640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f17d00d0be0 0x7f17d00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d5860640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d015d150 con 0x7f17d00d0be0 2023-10-16T14:37:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d5860640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d00d1180 con 0x7f17d00b8480 2023-10-16T14:37:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d5860640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00d1460 con 0x7f17d015cd70 2023-10-16T14:37:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d505f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f17d00d0be0 0x7f17d00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d505f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f17d00d0be0 0x7f17d00cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40634/0 (socket says 172.21.15.29:40634) 2023-10-16T14:37:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d505f640 1 -- 172.21.15.29:0/3474920076 learned_addr learned my addr 172.21.15.29:0/3474920076 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17d485e640 1 --2- 172.21.15.29:0/3474920076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d015cd70 0x7f17d00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.768+0000 7f17c7fff640 1 --2- 172.21.15.29:0/3474920076 >> v2:172.21.15.170:3300/0 conn(0x7f17d00b8480 0x7f17d00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.769+0000 7f17d505f640 1 -- 172.21.15.29:0/3474920076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d015cd70 msgr2=0x7f17d00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.769+0000 7f17d505f640 1 --2- 172.21.15.29:0/3474920076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d015cd70 0x7f17d00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.769+0000 7f17d505f640 1 -- 172.21.15.29:0/3474920076 >> v2:172.21.15.170:3300/0 conn(0x7f17d00b8480 msgr2=0x7f17d00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.769+0000 7f17d505f640 1 --2- 172.21.15.29:0/3474920076 >> v2:172.21.15.170:3300/0 conn(0x7f17d00b8480 0x7f17d00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.769+0000 7f17d505f640 1 -- 172.21.15.29:0/3474920076 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d00d1740 con 0x7f17d00d0be0 2023-10-16T14:37:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.770+0000 7f17d505f640 1 --2- 172.21.15.29:0/3474920076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d00d0be0 0x7f17d00cdd00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f17d0066ab0 tx=0x7f17c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9946ab82f83405a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.770+0000 7f17c77fe640 1 -- 172.21.15.29:0/3474920076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c802ca70 con 0x7f17d00d0be0 2023-10-16T14:37:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.770+0000 7f17c77fe640 1 -- 172.21.15.29:0/3474920076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c802cbd0 con 0x7f17d00d0be0 2023-10-16T14:37:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.770+0000 7f17c77fe640 1 -- 172.21.15.29:0/3474920076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c80343f0 con 0x7f17d00d0be0 2023-10-16T14:37:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 -- 172.21.15.29:0/3474920076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d00d0be0 msgr2=0x7f17d00cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 --2- 172.21.15.29:0/3474920076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d00d0be0 0x7f17d00cdd00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f17d0066ab0 tx=0x7f17c802bed0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 -- 172.21.15.29:0/3474920076 shutdown_connections 2023-10-16T14:37:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 --2- 172.21.15.29:0/3474920076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d00d0be0 0x7f17d00cdd00 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 --2- 172.21.15.29:0/3474920076 >> v2:172.21.15.170:3300/0 conn(0x7f17d00b8480 0x7f17d00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 --2- 172.21.15.29:0/3474920076 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d015cd70 0x7f17d00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 -- 172.21.15.29:0/3474920076 >> 172.21.15.29:0/3474920076 conn(0x7f17d00cf570 msgr2=0x7f17d00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 -- 172.21.15.29:0/3474920076 shutdown_connections 2023-10-16T14:37:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.771+0000 7f17d5860640 1 -- 172.21.15.29:0/3474920076 wait complete. 2023-10-16T14:37:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.773+0000 7f17d5860640 1 Processor -- start 2023-10-16T14:37:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.773+0000 7f17d5860640 1 -- start start 2023-10-16T14:37:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.774+0000 7f17d5860640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17d00b8480 0x7f17d015f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.774+0000 7f17d5860640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d00d0be0 0x7f17d0151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.774+0000 7f17d5860640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 0x7f17d00c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.774+0000 7f17d5860640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d015e590 con 0x7f17d015cd70 2023-10-16T14:37:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.774+0000 7f17d5860640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d015e870 con 0x7f17d00b8480 2023-10-16T14:37:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.774+0000 7f17d5860640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d015eb50 con 0x7f17d00d0be0 2023-10-16T14:37:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 0x7f17d00c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 0x7f17d00c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40644/0 (socket says 172.21.15.29:40644) 2023-10-16T14:37:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 -- 172.21.15.29:0/1839249338 learned_addr learned my addr 172.21.15.29:0/1839249338 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d485e640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17d00b8480 0x7f17d015f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17c7fff640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d00d0be0 0x7f17d0151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 -- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d00d0be0 msgr2=0x7f17d0151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d00d0be0 0x7f17d0151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 -- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17d00b8480 msgr2=0x7f17d015f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17d00b8480 0x7f17d015f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d505f640 1 -- 172.21.15.29:0/1839249338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8009b30 con 0x7f17d015cd70 2023-10-16T14:37:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17d485e640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17d00b8480 0x7f17d015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.775+0000 7f17c7fff640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d00d0be0 0x7f17d0151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.776+0000 7f17d505f640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 0x7f17d00c8c90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f17c8036040 tx=0x7f17c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.776+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8009510 con 0x7f17d015cd70 2023-10-16T14:37:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.776+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c803e450 con 0x7f17d015cd70 2023-10-16T14:37:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.776+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c803e800 con 0x7f17d015cd70 2023-10-16T14:37:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.776+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d015ee30 con 0x7f17d015cd70 2023-10-16T14:37:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.776+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d01517d0 con 0x7f17d015cd70 2023-10-16T14:37:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.777+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d015f350 con 0x7f17d015cd70 2023-10-16T14:37:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.779+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f17c803e960 con 0x7f17d015cd70 2023-10-16T14:37:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.779+0000 7f17c57fa640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f17bc06af90 0x7f17bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.780+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f17c80ac590 con 0x7f17d015cd70 2023-10-16T14:37:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.780+0000 7f17d485e640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f17bc06af90 0x7f17bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.782+0000 7f17d485e640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f17bc06af90 0x7f17bc06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f17cc006fd0 tx=0x7f17cc008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.784+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f17d015f350 con 0x7f17d015cd70 2023-10-16T14:37:03.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.980+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f17d015f350 con 0x7f17d015cd70 2023-10-16T14:37:03.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.987+0000 7f17c57fa640 1 -- 172.21.15.29:0/1839249338 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7f17d015f350 con 0x7f17d015cd70 2023-10-16T14:37:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f17bc06af90 msgr2=0x7f17bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f17bc06af90 0x7f17bc06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f17cc006fd0 tx=0x7f17cc008040 comp rx=0 tx=0).stop 2023-10-16T14:37:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 msgr2=0x7f17d00c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 0x7f17d00c8c90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f17c8036040 tx=0x7f17c8004a40 comp rx=0 tx=0).stop 2023-10-16T14:37:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 shutdown_connections 2023-10-16T14:37:03.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f17bc06af90 0x7f17bc06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17d015cd70 0x7f17d00c8c90 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f17d00d0be0 0x7f17d0151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 --2- 172.21.15.29:0/1839249338 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17d00b8480 0x7f17d015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:03.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.990+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 >> 172.21.15.29:0/1839249338 conn(0x7f17d00cf570 msgr2=0x7f17d00b6540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:03.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.991+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 shutdown_connections 2023-10-16T14:37:03.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:03.991+0000 7f17d5860640 1 -- 172.21.15.29:0/1839249338 wait complete. 2023-10-16T14:37:04.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T14:37:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.226+0000 7fc16359e640 1 Processor -- start 2023-10-16T14:37:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.226+0000 7fc16359e640 1 -- start start 2023-10-16T14:37:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc1640d2120 0x7fc1640b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc1640b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640d1080 con 0x7fc16415d130 2023-10-16T14:37:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16415d510 con 0x7fc1640d2120 2023-10-16T14:37:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640d1360 con 0x7fc1640cdff0 2023-10-16T14:37:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc16415d130 0x7fc1640cd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37962/0 (socket says 172.21.15.29:37962) 2023-10-16T14:37:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.227+0000 7fc16259c640 1 -- 172.21.15.29:0/3673738468 learned_addr learned my addr 172.21.15.29:0/3673738468 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc1640b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc161d9b640 1 --2- 172.21.15.29:0/3673738468 >> v2:172.21.15.170:3300/0 conn(0x7fc1640d2120 0x7fc1640b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058462070 0 0) 0x7fc1640d1080 con 0x7fc16415d130 2023-10-16T14:37:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc150003520 con 0x7fc16415d130 2023-10-16T14:37:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4006011901 0 0) 0x7fc150003520 con 0x7fc16415d130 2023-10-16T14:37:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 msgr2=0x7fc1640b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc1640b61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 >> v2:172.21.15.170:3300/0 conn(0x7fc1640d2120 msgr2=0x7fc1640b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 --2- 172.21.15.29:0/3673738468 >> v2:172.21.15.170:3300/0 conn(0x7fc1640d2120 0x7fc1640b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.228+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640d1640 con 0x7fc16415d130 2023-10-16T14:37:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc1540019f0 con 0x7fc16415d130 2023-10-16T14:37:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 >> v1:172.21.15.29:6789/0 conn(0x7fc16415d130 legacy=0x7fc1640cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc150003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150003ef0 0x7fc1500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150004d00 0x7fc150005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16415d510 con 0x7fc150004d00 2023-10-16T14:37:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640d1360 con 0x7fc150003ef0 2023-10-16T14:37:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15000b980 con 0x7fc1640cdff0 2023-10-16T14:37:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150004d00 0x7fc150005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16259c640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc150003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc161d9b640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150003ef0 0x7fc1500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc162d9d640 1 -- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 msgr2=0x7fc150003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc150003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc162d9d640 1 -- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150003ef0 msgr2=0x7fc1500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150003ef0 0x7fc1500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc162d9d640 1 -- 172.21.15.29:0/3673738468 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150003520 con 0x7fc150004d00 2023-10-16T14:37:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc161d9b640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc150003ef0 0x7fc1500047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.229+0000 7fc16259c640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1640cdff0 0x7fc150003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150004d00 0x7fc150005180 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fc1640cd330 tx=0x7fc1500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df929ca0a68b1f5c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c02c650 con 0x7fc150004d00 2023-10-16T14:37:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c02c7b0 con 0x7fc150004d00 2023-10-16T14:37:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc16159a640 1 -- 172.21.15.29:0/3673738468 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c02cb40 con 0x7fc150004d00 2023-10-16T14:37:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc16359e640 1 -- 172.21.15.29:0/3673738468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150004d00 msgr2=0x7fc150005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc16359e640 1 --2- 172.21.15.29:0/3673738468 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc150004d00 0x7fc150005180 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fc1640cd330 tx=0x7fc1500057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.231+0000 7fc162d9d640 1 -- 172.21.15.29:0/3673738468 reap_dead start 2023-10-16T14:37:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.232+0000 7fc16359e640 1 -- 172.21.15.29:0/3673738468 shutdown_connections 2023-10-16T14:37:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.232+0000 7fc16359e640 1 -- 172.21.15.29:0/3673738468 >> 172.21.15.29:0/3673738468 conn(0x7fc16406bc60 msgr2=0x7fc16406c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.232+0000 7fc16359e640 1 -- 172.21.15.29:0/3673738468 shutdown_connections 2023-10-16T14:37:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.232+0000 7fc16359e640 1 -- 172.21.15.29:0/3673738468 wait complete. 2023-10-16T14:37:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.234+0000 7fc16359e640 1 Processor -- start 2023-10-16T14:37:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.234+0000 7fc16359e640 1 -- start start 2023-10-16T14:37:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc16359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc16414f050 0x7fc1640c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc16359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 0x7fc1640c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc16359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc16415e130 0x7fc16415e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc16359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c8f10 con 0x7fc16414f430 2023-10-16T14:37:04.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc16359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164160130 con 0x7fc16414f050 2023-10-16T14:37:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc16359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc164160410 con 0x7fc16415e130 2023-10-16T14:37:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc161d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 0x7fc1640c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.235+0000 7fc161d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 0x7fc1640c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50330/0 (socket says 172.21.15.29:50330) 2023-10-16T14:37:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 -- 172.21.15.29:0/3139125500 learned_addr learned my addr 172.21.15.29:0/3139125500 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc16415e130 0x7fc16415e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 -- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc16415e130 msgr2=0x7fc16415e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc16415e130 0x7fc16415e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc16259c640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc16414f050 0x7fc1640c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 -- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc16414f050 msgr2=0x7fc1640c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc16414f050 0x7fc1640c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 -- 172.21.15.29:0/3139125500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16415d650 con 0x7fc16414f430 2023-10-16T14:37:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc162d9d640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc16415e130 0x7fc16415e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc161d9b640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 0x7fc1640c82d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0109d0 tx=0x7fc14c010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.236+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c00ec20 con 0x7fc16414f430 2023-10-16T14:37:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.237+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c00ed80 con 0x7fc16414f430 2023-10-16T14:37:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.237+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c01c620 con 0x7fc16414f430 2023-10-16T14:37:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.237+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16415d930 con 0x7fc16414f430 2023-10-16T14:37:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.237+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16415fd50 con 0x7fc16414f430 2023-10-16T14:37:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.238+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16404e9b0 con 0x7fc16414f430 2023-10-16T14:37:04.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.239+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc14c01c780 con 0x7fc16414f430 2023-10-16T14:37:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.240+0000 7fc15affd640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc14406afe0 0x7fc14406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.240+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7fc14c089d60 con 0x7fc16414f430 2023-10-16T14:37:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.241+0000 7fc16259c640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc14406afe0 0x7fc14406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.242+0000 7fc16259c640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc14406afe0 0x7fc14406d4a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc15400e320 tx=0x7fc154009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:04.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.244+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc14c054e90 con 0x7fc16414f430 2023-10-16T14:37:04.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.504+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fc164067e30 con 0x7fc16414f430 2023-10-16T14:37:04.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.583+0000 7fc15affd640 1 -- 172.21.15.29:0/3139125500 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v796) v1 ==== 160+0+0 (secure 0 0 0) 0x7fc14c0584b0 con 0x7fc16414f430 2023-10-16T14:37:04.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc14406afe0 msgr2=0x7fc14406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc14406afe0 0x7fc14406d4a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc15400e320 tx=0x7fc154009040 comp rx=0 tx=0).stop 2023-10-16T14:37:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 msgr2=0x7fc1640c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 0x7fc1640c82d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0109d0 tx=0x7fc14c010e90 comp rx=0 tx=0).stop 2023-10-16T14:37:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 shutdown_connections 2023-10-16T14:37:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fc14406afe0 0x7fc14406d4a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc16414f430 0x7fc1640c82d0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc16415e130 0x7fc16415e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 --2- 172.21.15.29:0/3139125500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc16414f050 0x7fc1640c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 >> 172.21.15.29:0/3139125500 conn(0x7fc16406bc60 msgr2=0x7fc1640c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:04.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 shutdown_connections 2023-10-16T14:37:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.587+0000 7fc16359e640 1 -- 172.21.15.29:0/3139125500 wait complete. 2023-10-16T14:37:04.608 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-10-16T14:37:04.609 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-16T14:37:04.610 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:37:04.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:37:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.822+0000 7f5d723b8640 1 Processor -- start 2023-10-16T14:37:04.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.822+0000 7f5d723b8640 1 -- start start 2023-10-16T14:37:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d723b8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d6c06c040 0x7f5d6c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d723b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d723b8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c06c960 con 0x7f5d6c0ad970 2023-10-16T14:37:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d723b8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d6c15b9d0 con 0x7f5d6c06c040 2023-10-16T14:37:04.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d723b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c15bcb0 con 0x7f5d6c0cda80 2023-10-16T14:37:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d713b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d6c06c040 0x7f5d6c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.824+0000 7f5d70bb5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5d6c0ad970 0x7f5d6c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37968/0 (socket says 172.21.15.29:37968) 2023-10-16T14:37:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d70bb5640 1 -- 172.21.15.29:0/3629110548 learned_addr learned my addr 172.21.15.29:0/3629110548 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d71bb7640 1 --2- 172.21.15.29:0/3629110548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d63fff640 1 -- 172.21.15.29:0/3629110548 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980286166 0 0) 0x7f5d6c06c960 con 0x7f5d6c0ad970 2023-10-16T14:37:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d713b6640 1 -- 172.21.15.29:0/3629110548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 msgr2=0x7f5d6c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d713b6640 1 --2- 172.21.15.29:0/3629110548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d713b6640 1 -- 172.21.15.29:0/3629110548 >> v1:172.21.15.29:6789/0 conn(0x7f5d6c0ad970 legacy=0x7f5d6c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.825+0000 7f5d713b6640 1 -- 172.21.15.29:0/3629110548 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c15bf90 con 0x7f5d6c06c040 2023-10-16T14:37:04.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.826+0000 7f5d713b6640 1 --2- 172.21.15.29:0/3629110548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c06c040 0x7f5d6c06c420 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f5d680097a0 tx=0x7f5d6802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f0a37889395124 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d63fff640 1 -- 172.21.15.29:0/3629110548 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6802d040 con 0x7f5d6c06c040 2023-10-16T14:37:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d63fff640 1 -- 172.21.15.29:0/3629110548 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d68031810 con 0x7f5d6c06c040 2023-10-16T14:37:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d63fff640 1 -- 172.21.15.29:0/3629110548 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d68031ba0 con 0x7f5d6c06c040 2023-10-16T14:37:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 -- 172.21.15.29:0/3629110548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c06c040 msgr2=0x7f5d6c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 --2- 172.21.15.29:0/3629110548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c06c040 0x7f5d6c06c420 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f5d680097a0 tx=0x7f5d6802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 -- 172.21.15.29:0/3629110548 shutdown_connections 2023-10-16T14:37:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 --2- 172.21.15.29:0/3629110548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 --2- 172.21.15.29:0/3629110548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c06c040 0x7f5d6c06c420 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 -- 172.21.15.29:0/3629110548 >> 172.21.15.29:0/3629110548 conn(0x7f5d6c0ce6a0 msgr2=0x7f5d6c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 -- 172.21.15.29:0/3629110548 shutdown_connections 2023-10-16T14:37:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.827+0000 7f5d723b8640 1 -- 172.21.15.29:0/3629110548 wait complete. 2023-10-16T14:37:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.829+0000 7f5d723b8640 1 Processor -- start 2023-10-16T14:37:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.829+0000 7f5d723b8640 1 -- start start 2023-10-16T14:37:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.830+0000 7f5d723b8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d6c06c040 0x7f5d6c0c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.830+0000 7f5d723b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 0x7f5d6c1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.830+0000 7f5d723b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.830+0000 7f5d723b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c1f2a10 con 0x7f5d6c0ad970 2023-10-16T14:37:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.830+0000 7f5d723b8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c1f2cf0 con 0x7f5d6c06c040 2023-10-16T14:37:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.830+0000 7f5d723b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c1f2fd0 con 0x7f5d6c0cda80 2023-10-16T14:37:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 0x7f5d6c1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 0x7f5d6c1f2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50338/0 (socket says 172.21.15.29:50338) 2023-10-16T14:37:04.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 -- 172.21.15.29:0/2841167752 learned_addr learned my addr 172.21.15.29:0/2841167752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d71bb7640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d713b6640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d6c06c040 0x7f5d6c0c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 -- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 msgr2=0x7f5d6c0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c0ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 -- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d6c06c040 msgr2=0x7f5d6c0c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:04.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d6c06c040 0x7f5d6c0c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:04.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 -- 172.21.15.29:0/2841167752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68009400 con 0x7f5d6c0ad970 2023-10-16T14:37:04.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d713b6640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d6c06c040 0x7f5d6c0c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:04.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.831+0000 7f5d70bb5640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 0x7f5d6c1f2150 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c410 tx=0x7f5d5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:04.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.832+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c0074e0 con 0x7f5d6c0ad970 2023-10-16T14:37:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.832+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c007660 con 0x7f5d6c0ad970 2023-10-16T14:37:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.832+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c004a20 con 0x7f5d6c0ad970 2023-10-16T14:37:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.832+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c0b1de0 con 0x7f5d6c0ad970 2023-10-16T14:37:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.832+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c1f2690 con 0x7f5d6c0ad970 2023-10-16T14:37:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.833+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c06b330 con 0x7f5d6c0ad970 2023-10-16T14:37:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.834+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d5c004b80 con 0x7f5d6c0ad970 2023-10-16T14:37:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.835+0000 7f5d61ffb640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5d5006afc0 0x7f5d5006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.835+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f5d5c0895d0 con 0x7f5d6c0ad970 2023-10-16T14:37:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.835+0000 7f5d713b6640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5d5006afc0 0x7f5d5006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.837+0000 7f5d713b6640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5d5006afc0 0x7f5d5006d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5d68036000 tx=0x7f5d6803c040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:04.839+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5d5c057f60 con 0x7f5d6c0ad970 2023-10-16T14:37:05.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.032+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d6c04e9b0 con 0x7f5d6c0ad970 2023-10-16T14:37:05.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.033+0000 7f5d61ffb640 1 -- 172.21.15.29:0/2841167752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+14119 (secure 0 0 0) 0x7f5d5c057af0 con 0x7f5d6c0ad970 2023-10-16T14:37:05.037 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:05.037 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":147,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:37:02.557181+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/1700644562":"2023-10-17T14:36:59.515954+0000","172.21.15.170:0/3813450646":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1911194233":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/3477374804":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/601419330":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1163999179":"2023-10-17T14:36:55.479187+0000","172.21.15.170:6833/230171195":"2023-10-17T14:36:55.479187+0000","172.21.15.170:6832/230171195":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/1143303444":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/2774967038":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2333704669":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1712463277":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/2908562666":"2023-10-17T14:36:55.479187+0000","172.21.15.170:0/782892922":"2023-10-17T14:36:42.416389+0000","172.21.15.29:6833/1700644562":"2023-10-17T14:36:59.515954+0000","172.21.15.170:6833/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:6832/1905872466":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1119524146":"2023-10-17T14:36:42.416389+0000","172.21.15.170:0/1398880520":"2023-10-17T14:36:42.416389+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:37:05.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5d5006afc0 msgr2=0x7f5d5006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5d5006afc0 0x7f5d5006d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5d68036000 tx=0x7f5d6803c040 comp rx=0 tx=0).stop 2023-10-16T14:37:05.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 msgr2=0x7f5d6c1f2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 0x7f5d6c1f2150 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c410 tx=0x7f5d5c00c8d0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 shutdown_connections 2023-10-16T14:37:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5d5006afc0 0x7f5d5006d480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d6c0ad970 0x7f5d6c1f2150 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d6c0cda80 0x7f5d6c0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 --2- 172.21.15.29:0/2841167752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d6c06c040 0x7f5d6c0c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 >> 172.21.15.29:0/2841167752 conn(0x7f5d6c0ce6a0 msgr2=0x7f5d6c0acf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 shutdown_connections 2023-10-16T14:37:05.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.036+0000 7f5d723b8640 1 -- 172.21.15.29:0/2841167752 wait complete. 2023-10-16T14:37:05.056 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:37:05.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:37:05.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.254+0000 7f91ab210640 1 Processor -- start 2023-10-16T14:37:05.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.254+0000 7f91ab210640 1 -- start start 2023-10-16T14:37:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91ab210640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91a4159e90 0x7f91a40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91ab210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a406c140 0x7f91a406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91ab210640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a4070360 con 0x7f91a4159ab0 2023-10-16T14:37:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91ab210640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a406cb10 con 0x7f91a4159e90 2023-10-16T14:37:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91ab210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a415d890 con 0x7f91a406c140 2023-10-16T14:37:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91a3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91a4159e90 0x7f91a40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91a3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91a4159e90 0x7f91a40ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50352/0 (socket says 172.21.15.29:50352) 2023-10-16T14:37:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.256+0000 7f91a3fff640 1 -- 172.21.15.29:0/508368131 learned_addr learned my addr 172.21.15.29:0/508368131 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:05.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.257+0000 7f91a97ad640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a406c140 0x7f91a406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.257+0000 7f91a3fff640 1 -- 172.21.15.29:0/508368131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a406c140 msgr2=0x7f91a406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.257+0000 7f91a3fff640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a406c140 0x7f91a406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.257+0000 7f91a3fff640 1 -- 172.21.15.29:0/508368131 >> v1:172.21.15.170:6789/0 conn(0x7f91a4159ab0 legacy=0x7f91a406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.257+0000 7f91a3fff640 1 -- 172.21.15.29:0/508368131 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a40cd1f0 con 0x7f91a4159e90 2023-10-16T14:37:05.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.257+0000 7f91a97ad640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a406c140 0x7f91a406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:05.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.258+0000 7f91a3fff640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a4159e90 0x7f91a40ce370 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f919000bb10 tx=0x7f919002dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32b488c10a5f527 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:05.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.258+0000 7f91a37fe640 1 -- 172.21.15.29:0/508368131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919002ed50 con 0x7f91a4159e90 2023-10-16T14:37:05.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.258+0000 7f91a37fe640 1 -- 172.21.15.29:0/508368131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190036440 con 0x7f91a4159e90 2023-10-16T14:37:05.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.258+0000 7f91a37fe640 1 -- 172.21.15.29:0/508368131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91900367d0 con 0x7f91a4159e90 2023-10-16T14:37:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 -- 172.21.15.29:0/508368131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a4159e90 msgr2=0x7f91a40ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a4159e90 0x7f91a40ce370 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f919000bb10 tx=0x7f919002dfa0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 -- 172.21.15.29:0/508368131 shutdown_connections 2023-10-16T14:37:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a406c140 0x7f91a406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 --2- 172.21.15.29:0/508368131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a4159e90 0x7f91a40ce370 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 -- 172.21.15.29:0/508368131 >> 172.21.15.29:0/508368131 conn(0x7f91a40ce9b0 msgr2=0x7f91a40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:05.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 -- 172.21.15.29:0/508368131 shutdown_connections 2023-10-16T14:37:05.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.259+0000 7f91ab210640 1 -- 172.21.15.29:0/508368131 wait complete. 2023-10-16T14:37:05.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.261+0000 7f91ab210640 1 Processor -- start 2023-10-16T14:37:05.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.261+0000 7f91ab210640 1 -- start start 2023-10-16T14:37:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.262+0000 7f91ab210640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 0x7f91a4151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.262+0000 7f91ab210640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4159ab0 0x7f91a40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.262+0000 7f91ab210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a4159e90 0x7f91a4069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.262+0000 7f91ab210640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a40c61e0 con 0x7f91a406c140 2023-10-16T14:37:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.262+0000 7f91ab210640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a415a860 con 0x7f91a4159ab0 2023-10-16T14:37:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.262+0000 7f91ab210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a415ab40 con 0x7f91a4159e90 2023-10-16T14:37:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 0x7f91a4151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 0x7f91a4151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50358/0 (socket says 172.21.15.29:50358) 2023-10-16T14:37:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 -- 172.21.15.29:0/4094674107 learned_addr learned my addr 172.21.15.29:0/4094674107 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:05.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a97ad640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a4159e90 0x7f91a4069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a3fff640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4159ab0 0x7f91a40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 -- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a4159e90 msgr2=0x7f91a4069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a4159e90 0x7f91a4069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 -- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4159ab0 msgr2=0x7f91a40c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4159ab0 0x7f91a40c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 -- 172.21.15.29:0/4094674107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919000b770 con 0x7f91a406c140 2023-10-16T14:37:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a3fff640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4159ab0 0x7f91a40c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.263+0000 7f91a8fac640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 0x7f91a4151d80 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f919800b3f0 tx=0x7f919800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.264+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198011830 con 0x7f91a406c140 2023-10-16T14:37:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.264+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198011990 con 0x7f91a406c140 2023-10-16T14:37:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.264+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198011d40 con 0x7f91a406c140 2023-10-16T14:37:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.264+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a415ae20 con 0x7f91a406c140 2023-10-16T14:37:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.264+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a41522c0 con 0x7f91a406c140 2023-10-16T14:37:05.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.265+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a40c6350 con 0x7f91a406c140 2023-10-16T14:37:05.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.266+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f919801a400 con 0x7f91a406c140 2023-10-16T14:37:05.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.267+0000 7f91a17fa640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918806af90 0x7f918806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.267+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f91980896b0 con 0x7f91a406c140 2023-10-16T14:37:05.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.268+0000 7f91a3fff640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918806af90 0x7f918806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.269+0000 7f91a3fff640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918806af90 0x7f918806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9190038000 tx=0x7f9190008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:05.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.272+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f91a40c6350 con 0x7f91a406c140 2023-10-16T14:37:05.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.457+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91a40c6320 con 0x7f91a406c140 2023-10-16T14:37:05.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.458+0000 7f91a17fa640 1 -- 172.21.15.29:0/4094674107 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91980545b0 con 0x7f91a406c140 2023-10-16T14:37:05.461 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:37:05.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918806af90 msgr2=0x7f918806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918806af90 0x7f918806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9190038000 tx=0x7f9190008040 comp rx=0 tx=0).stop 2023-10-16T14:37:05.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 msgr2=0x7f91a4151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 0x7f91a4151d80 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f919800b3f0 tx=0x7f919800b8b0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 shutdown_connections 2023-10-16T14:37:05.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918806af90 0x7f918806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91a4159e90 0x7f91a4069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a4159ab0 0x7f91a40c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 --2- 172.21.15.29:0/4094674107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a406c140 0x7f91a4151d80 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.460+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 >> 172.21.15.29:0/4094674107 conn(0x7f91a40ce9b0 msgr2=0x7f91a40c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:05.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.461+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 shutdown_connections 2023-10-16T14:37:05.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.461+0000 7f91ab210640 1 -- 172.21.15.29:0/4094674107 wait complete. 2023-10-16T14:37:05.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T14:37:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.702+0000 7f933aa4f640 1 Processor -- start 2023-10-16T14:37:05.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.702+0000 7f933aa4f640 1 -- start start 2023-10-16T14:37:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f933aa4f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f93341593b0 0x7f933406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f933aa4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f933406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f933aa4f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340c9b20 con 0x7f93340caa10 2023-10-16T14:37:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f933aa4f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9334159840 con 0x7f93341593b0 2023-10-16T14:37:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f933aa4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93340b0b90 con 0x7f933406fa90 2023-10-16T14:37:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f9338fec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f933406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.704+0000 7f93337fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f93341593b0 0x7f933406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f9338fec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f933406ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:58906/0 (socket says 172.21.15.29:58906) 2023-10-16T14:37:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f9338fec640 1 -- 172.21.15.29:0/2267027315 learned_addr learned my addr 172.21.15.29:0/2267027315 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f9338fec640 1 -- 172.21.15.29:0/2267027315 >> v2:172.21.15.170:3300/0 conn(0x7f93341593b0 msgr2=0x7f933406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f9338fec640 1 --2- 172.21.15.29:0/2267027315 >> v2:172.21.15.170:3300/0 conn(0x7f93341593b0 0x7f933406f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f9338fec640 1 -- 172.21.15.29:0/2267027315 >> v1:172.21.15.170:6789/0 conn(0x7f93340caa10 legacy=0x7f93340b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f9338fec640 1 -- 172.21.15.29:0/2267027315 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340c9e00 con 0x7f933406fa90 2023-10-16T14:37:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.705+0000 7f93337fe640 1 --2- 172.21.15.29:0/2267027315 >> v2:172.21.15.170:3300/0 conn(0x7f93341593b0 0x7f933406f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:37:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.707+0000 7f9338fec640 1 --2- 172.21.15.29:0/2267027315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f933406ff20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f932c009e50 tx=0x7f932c02be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad44d7d0d0591456 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.707+0000 7f9332ffd640 1 -- 172.21.15.29:0/2267027315 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c02c9f0 con 0x7f933406fa90 2023-10-16T14:37:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.707+0000 7f9332ffd640 1 -- 172.21.15.29:0/2267027315 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c02cb50 con 0x7f933406fa90 2023-10-16T14:37:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.707+0000 7f9332ffd640 1 -- 172.21.15.29:0/2267027315 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0343f0 con 0x7f933406fa90 2023-10-16T14:37:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 -- 172.21.15.29:0/2267027315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 msgr2=0x7f933406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 --2- 172.21.15.29:0/2267027315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f933406ff20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f932c009e50 tx=0x7f932c02be50 comp rx=0 tx=0).stop 2023-10-16T14:37:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 -- 172.21.15.29:0/2267027315 shutdown_connections 2023-10-16T14:37:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 --2- 172.21.15.29:0/2267027315 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f933406ff20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 --2- 172.21.15.29:0/2267027315 >> v2:172.21.15.170:3300/0 conn(0x7f93341593b0 0x7f933406f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 -- 172.21.15.29:0/2267027315 >> 172.21.15.29:0/2267027315 conn(0x7f93340cb410 msgr2=0x7f93340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 -- 172.21.15.29:0/2267027315 shutdown_connections 2023-10-16T14:37:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.708+0000 7f933aa4f640 1 -- 172.21.15.29:0/2267027315 wait complete. 2023-10-16T14:37:05.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.710+0000 7f933aa4f640 1 Processor -- start 2023-10-16T14:37:05.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.710+0000 7f933aa4f640 1 -- start start 2023-10-16T14:37:05.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.711+0000 7f933aa4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f9334154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.711+0000 7f933aa4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93340caa10 0x7f933415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.711+0000 7f933aa4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 0x7f93340cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.711+0000 7f933aa4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c39b0 con 0x7f93341593b0 2023-10-16T14:37:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.711+0000 7f933aa4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c3e40 con 0x7f93340caa10 2023-10-16T14:37:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.711+0000 7f933aa4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c2da0 con 0x7f933406fa90 2023-10-16T14:37:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 0x7f93340cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 0x7f93340cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50360/0 (socket says 172.21.15.29:50360) 2023-10-16T14:37:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 -- 172.21.15.29:0/3143610130 learned_addr learned my addr 172.21.15.29:0/3143610130 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9333fff640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f9334154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 -- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 msgr2=0x7f9334154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f9334154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 -- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93340caa10 msgr2=0x7f933415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93340caa10 0x7f933415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.712+0000 7f9338fec640 1 -- 172.21.15.29:0/3143610130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328009400 con 0x7f93341593b0 2023-10-16T14:37:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.713+0000 7f9338fec640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 0x7f93340cc920 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f932c009f70 tx=0x7f932c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.713+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c009510 con 0x7f93341593b0 2023-10-16T14:37:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.713+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c03e450 con 0x7f93341593b0 2023-10-16T14:37:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.713+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c03e820 con 0x7f93341593b0 2023-10-16T14:37:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.713+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c009b30 con 0x7f93341593b0 2023-10-16T14:37:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.713+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93340c3360 con 0x7f93341593b0 2023-10-16T14:37:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.714+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93340c3b60 con 0x7f93341593b0 2023-10-16T14:37:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.715+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f932c03e980 con 0x7f93341593b0 2023-10-16T14:37:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.716+0000 7f9330ff9640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932006af90 0x7f932006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.716+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f932c0ac170 con 0x7f93341593b0 2023-10-16T14:37:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.717+0000 7f9333fff640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932006af90 0x7f932006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.718+0000 7f9333fff640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932006af90 0x7f932006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9324007fd0 tx=0x7f9324009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.721+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f93340c3b60 con 0x7f93341593b0 2023-10-16T14:37:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:05.987+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f933404e160 con 0x7f93341593b0 2023-10-16T14:37:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.589+0000 7f9330ff9640 1 -- 172.21.15.29:0/3143610130 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v798) v1 ==== 181+0+0 (secure 0 0 0) 0x7f932c077100 con 0x7f93341593b0 2023-10-16T14:37:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932006af90 msgr2=0x7f932006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932006af90 0x7f932006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9324007fd0 tx=0x7f9324009040 comp rx=0 tx=0).stop 2023-10-16T14:37:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 msgr2=0x7f93340cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 0x7f93340cc920 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f932c009f70 tx=0x7f932c004a40 comp rx=0 tx=0).stop 2023-10-16T14:37:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 shutdown_connections 2023-10-16T14:37:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f932006af90 0x7f932006d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93341593b0 0x7f93340cc920 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93340caa10 0x7f933415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 --2- 172.21.15.29:0/3143610130 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f933406fa90 0x7f9334154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.592+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 >> 172.21.15.29:0/3143610130 conn(0x7f93340cb410 msgr2=0x7f933415bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.593+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 shutdown_connections 2023-10-16T14:37:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.593+0000 7f933aa4f640 1 -- 172.21.15.29:0/3143610130 wait complete. 2023-10-16T14:37:06.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:37:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.842+0000 7f4cef59e640 1 Processor -- start 2023-10-16T14:37:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.842+0000 7f4cef59e640 1 -- start start 2023-10-16T14:37:06.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4cef59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4cf0159950 0x7f4cf015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4cef59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4cf006c810 0x7f4cf00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4cef59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf015aef0 con 0x7f4cf0070450 2023-10-16T14:37:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4cef59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf0070e20 con 0x7f4cf0159950 2023-10-16T14:37:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4cef59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf015e490 con 0x7f4cf006c810 2023-10-16T14:37:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4cee59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4cf0159950 0x7f4cf015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.844+0000 7f4ce7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4cf0070450 0x7f4cf006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37970/0 (socket says 172.21.15.29:37970) 2023-10-16T14:37:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.845+0000 7f4ce7fff640 1 -- 172.21.15.29:0/3857768613 learned_addr learned my addr 172.21.15.29:0/3857768613 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.845+0000 7f4cee59c640 1 -- 172.21.15.29:0/3857768613 >> v2:172.21.15.170:3300/0 conn(0x7f4cf006c810 msgr2=0x7f4cf00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.845+0000 7f4cee59c640 1 --2- 172.21.15.29:0/3857768613 >> v2:172.21.15.170:3300/0 conn(0x7f4cf006c810 0x7f4cf00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.845+0000 7f4cee59c640 1 -- 172.21.15.29:0/3857768613 >> v1:172.21.15.29:6789/0 conn(0x7f4cf0070450 legacy=0x7f4cf006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:06.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.845+0000 7f4cee59c640 1 -- 172.21.15.29:0/3857768613 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf015d910 con 0x7f4cf0159950 2023-10-16T14:37:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.846+0000 7f4cee59c640 1 --2- 172.21.15.29:0/3857768613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf0159950 0x7f4cf015a9b0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00716a0 tx=0x7f4ce002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c8d4764ec823f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.846+0000 7f4cedd9b640 1 -- 172.21.15.29:0/3857768613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce002e070 con 0x7f4cf0159950 2023-10-16T14:37:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cedd9b640 1 -- 172.21.15.29:0/3857768613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce00315a0 con 0x7f4cf0159950 2023-10-16T14:37:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cedd9b640 1 -- 172.21.15.29:0/3857768613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0031930 con 0x7f4cf0159950 2023-10-16T14:37:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 -- 172.21.15.29:0/3857768613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf0159950 msgr2=0x7f4cf015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3857768613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf0159950 0x7f4cf015a9b0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00716a0 tx=0x7f4ce002b840 comp rx=0 tx=0).stop 2023-10-16T14:37:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 -- 172.21.15.29:0/3857768613 shutdown_connections 2023-10-16T14:37:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3857768613 >> v2:172.21.15.170:3300/0 conn(0x7f4cf006c810 0x7f4cf00708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3857768613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf0159950 0x7f4cf015a9b0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 -- 172.21.15.29:0/3857768613 >> 172.21.15.29:0/3857768613 conn(0x7f4cf00ce6a0 msgr2=0x7f4cf00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 -- 172.21.15.29:0/3857768613 shutdown_connections 2023-10-16T14:37:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.847+0000 7f4cef59e640 1 -- 172.21.15.29:0/3857768613 wait complete. 2023-10-16T14:37:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.849+0000 7f4cef59e640 1 Processor -- start 2023-10-16T14:37:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.849+0000 7f4cef59e640 1 -- start start 2023-10-16T14:37:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.850+0000 7f4cef59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 0x7f4cf014fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.850+0000 7f4cef59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cf0070450 0x7f4cf00c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cef59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cf0159950 0x7f4cf00c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cef59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf01f5b70 con 0x7f4cf006c810 2023-10-16T14:37:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cef59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf01f5dc0 con 0x7f4cf0159950 2023-10-16T14:37:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cef59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf01f60a0 con 0x7f4cf0070450 2023-10-16T14:37:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 0x7f4cf014fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 0x7f4cf014fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50370/0 (socket says 172.21.15.29:50370) 2023-10-16T14:37:06.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 -- 172.21.15.29:0/3104556167 learned_addr learned my addr 172.21.15.29:0/3104556167 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 -- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cf0070450 msgr2=0x7f4cf00c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4ce7fff640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cf0070450 0x7f4cf00c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4ceed9d640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cf0159950 0x7f4cf00c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cf0070450 0x7f4cf00c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 -- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cf0159950 msgr2=0x7f4cf00c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cf0159950 0x7f4cf00c9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.851+0000 7f4cee59c640 1 -- 172.21.15.29:0/3104556167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0009400 con 0x7f4cf006c810 2023-10-16T14:37:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.852+0000 7f4cee59c640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 0x7f4cf014fae0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009890 tx=0x7f4ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.852+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0032900 con 0x7f4cf006c810 2023-10-16T14:37:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.852+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce0032a60 con 0x7f4cf006c810 2023-10-16T14:37:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.852+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf01f6380 con 0x7f4cf006c810 2023-10-16T14:37:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.852+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0032e10 con 0x7f4cf006c810 2023-10-16T14:37:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.852+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf0150020 con 0x7f4cf006c810 2023-10-16T14:37:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.853+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf01f68a0 con 0x7f4cf006c810 2023-10-16T14:37:06.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.854+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4ce004d3f0 con 0x7f4cf006c810 2023-10-16T14:37:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.855+0000 7f4ce6ffd640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4cd006af90 0x7f4cd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.855+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f4ce00ab480 con 0x7f4cf006c810 2023-10-16T14:37:06.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.856+0000 7f4ce7fff640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4cd006af90 0x7f4cd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.857+0000 7f4ce7fff640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4cd006af90 0x7f4cd006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00ca460 tx=0x7f4cd8009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:06.859+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ce0076310 con 0x7f4cf006c810 2023-10-16T14:37:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.051+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4cf004e9b0 con 0x7f4cf006c810 2023-10-16T14:37:07.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.056+0000 7f4ce6ffd640 1 -- 172.21.15.29:0/3104556167 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4ce007ee60 con 0x7f4cf006c810 2023-10-16T14:37:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4cd006af90 msgr2=0x7f4cd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4cd006af90 0x7f4cd006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00ca460 tx=0x7f4cd8009040 comp rx=0 tx=0).stop 2023-10-16T14:37:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 msgr2=0x7f4cf014fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:07.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 0x7f4cf014fae0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009890 tx=0x7f4ce00046c0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 shutdown_connections 2023-10-16T14:37:07.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4cd006af90 0x7f4cd006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4cf0159950 0x7f4cf00c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4cf0070450 0x7f4cf00c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 --2- 172.21.15.29:0/3104556167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf006c810 0x7f4cf014fae0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 >> 172.21.15.29:0/3104556167 conn(0x7f4cf00ce6a0 msgr2=0x7f4cf0072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 shutdown_connections 2023-10-16T14:37:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.059+0000 7f4cef59e640 1 -- 172.21.15.29:0/3104556167 wait complete. 2023-10-16T14:37:07.082 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:37:07.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:37:07.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:37:07.328 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:37:07.367 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:37:07.368 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:37:07.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:37:07.432 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:37:07.433 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:37:07.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:37:07.495 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:37:07.495 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:37:07.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:37:07.559 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T14:37:07.560 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:37:07.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:37:07.618 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:37:07.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:37:07.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.855+0000 7f7b389f0640 1 Processor -- start 2023-10-16T14:37:07.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.855+0000 7f7b389f0640 1 -- start start 2023-10-16T14:37:07.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.856+0000 7f7b389f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b34159950 0x7f7b3415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.856+0000 7f7b389f0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7b34070450 0x7f7b340708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.856+0000 7f7b389f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b3406c100 0x7f7b3406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.856+0000 7f7b389f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b34070e20 con 0x7f7b34159950 2023-10-16T14:37:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.856+0000 7f7b389f0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b3415e490 con 0x7f7b34070450 2023-10-16T14:37:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.856+0000 7f7b389f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3415aef0 con 0x7f7b3406c100 2023-10-16T14:37:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b34159950 0x7f7b3415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b34159950 0x7f7b3415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50384/0 (socket says 172.21.15.29:50384) 2023-10-16T14:37:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 -- 172.21.15.29:0/1563757363 learned_addr learned my addr 172.21.15.29:0/1563757363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b32ffd640 1 --2- 172.21.15.29:0/1563757363 >> v2:172.21.15.170:3300/0 conn(0x7f7b34070450 0x7f7b340708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b33fff640 1 --2- 172.21.15.29:0/1563757363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b3406c100 0x7f7b3406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 -- 172.21.15.29:0/1563757363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b3406c100 msgr2=0x7f7b3406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1563757363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b3406c100 0x7f7b3406c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 -- 172.21.15.29:0/1563757363 >> v2:172.21.15.170:3300/0 conn(0x7f7b34070450 msgr2=0x7f7b340708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:07.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1563757363 >> v2:172.21.15.170:3300/0 conn(0x7f7b34070450 0x7f7b340708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.857+0000 7f7b337fe640 1 -- 172.21.15.29:0/1563757363 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3415d910 con 0x7f7b34159950 2023-10-16T14:37:07.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.858+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1563757363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b3415a9b0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f7b340cab70 tx=0x7f7b2402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af01821edc083aaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:07.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.859+0000 7f7b327fc640 1 -- 172.21.15.29:0/1563757363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b240317c0 con 0x7f7b34159950 2023-10-16T14:37:07.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.859+0000 7f7b327fc640 1 -- 172.21.15.29:0/1563757363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b24031920 con 0x7f7b34159950 2023-10-16T14:37:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.859+0000 7f7b327fc640 1 -- 172.21.15.29:0/1563757363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24031cb0 con 0x7f7b34159950 2023-10-16T14:37:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.859+0000 7f7b389f0640 1 -- 172.21.15.29:0/1563757363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 msgr2=0x7f7b3415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.859+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1563757363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b3415a9b0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f7b340cab70 tx=0x7f7b2402b860 comp rx=0 tx=0).stop 2023-10-16T14:37:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 -- 172.21.15.29:0/1563757363 shutdown_connections 2023-10-16T14:37:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1563757363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b3406c100 0x7f7b3406c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1563757363 >> v2:172.21.15.170:3300/0 conn(0x7f7b34070450 0x7f7b340708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1563757363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b3415a9b0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 -- 172.21.15.29:0/1563757363 >> 172.21.15.29:0/1563757363 conn(0x7f7b340ce6a0 msgr2=0x7f7b340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 -- 172.21.15.29:0/1563757363 shutdown_connections 2023-10-16T14:37:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.860+0000 7f7b389f0640 1 -- 172.21.15.29:0/1563757363 wait complete. 2023-10-16T14:37:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.861+0000 7f7b389f0640 1 Processor -- start 2023-10-16T14:37:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.862+0000 7f7b389f0640 1 -- start start 2023-10-16T14:37:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.862+0000 7f7b389f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b3406c100 0x7f7b3414a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.862+0000 7f7b389f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b34070450 0x7f7b340c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.862+0000 7f7b389f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b340c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b389f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3414ac80 con 0x7f7b34159950 2023-10-16T14:37:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b389f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340b2680 con 0x7f7b3406c100 2023-10-16T14:37:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b389f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340c60a0 con 0x7f7b34070450 2023-10-16T14:37:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b340c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b340c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50400/0 (socket says 172.21.15.29:50400) 2023-10-16T14:37:07.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 -- 172.21.15.29:0/1274015915 learned_addr learned my addr 172.21.15.29:0/1274015915 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:07.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b3406c100 0x7f7b3414a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b32ffd640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b34070450 0x7f7b340c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 -- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b34070450 msgr2=0x7f7b340c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:07.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b34070450 0x7f7b340c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 -- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b3406c100 msgr2=0x7f7b3414a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b3406c100 0x7f7b3414a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b33fff640 1 -- 172.21.15.29:0/1274015915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240094d0 con 0x7f7b34159950 2023-10-16T14:37:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.863+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b3406c100 0x7f7b3414a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.864+0000 7f7b33fff640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b340c8f20 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00cc30 tx=0x7f7b2c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.864+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c00ce40 con 0x7f7b34159950 2023-10-16T14:37:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.864+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c003990 con 0x7f7b34159950 2023-10-16T14:37:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.864+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c003d20 con 0x7f7b34159950 2023-10-16T14:37:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.864+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b340c6380 con 0x7f7b34159950 2023-10-16T14:37:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.864+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b340b2310 con 0x7f7b34159950 2023-10-16T14:37:07.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.866+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b340b27f0 con 0x7f7b34159950 2023-10-16T14:37:07.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.866+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b2c003e80 con 0x7f7b34159950 2023-10-16T14:37:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.867+0000 7f7b1bfff640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f7b2006af90 0x7f7b2006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.867+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f7b2c089c90 con 0x7f7b34159950 2023-10-16T14:37:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.868+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f7b2006af90 0x7f7b2006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.869+0000 7f7b337fe640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f7b2006af90 0x7f7b2006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7b24036000 tx=0x7f7b2403d040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:07.871+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b2c054b20 con 0x7f7b34159950 2023-10-16T14:37:08.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.099+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b3404e9b0 con 0x7f7b34159950 2023-10-16T14:37:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.100+0000 7f7b1bfff640 1 -- 172.21.15.29:0/1274015915 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7b2c01d070 con 0x7f7b34159950 2023-10-16T14:37:08.103 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:08.103 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/4195261155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":4195261155},{"type":"v1","addr":"172.21.15.170:6837","nonce":4195261155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/4145291367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":4145291367},{"type":"v1","addr":"172.21.15.170:6835","nonce":4145291367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8274,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1286043585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1286043585},{"type":"v1","addr":"172.21.15.29:6834","nonce":1286043585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":8742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1267892919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":1267892919},{"type":"v1","addr":"172.21.15.29:6835","nonce":1267892919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[]} 2023-10-16T14:37:08.104 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-16T14:37:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f7b2006af90 msgr2=0x7f7b2006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f7b2006af90 0x7f7b2006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7b24036000 tx=0x7f7b2403d040 comp rx=0 tx=0).stop 2023-10-16T14:37:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 msgr2=0x7f7b340c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b340c8f20 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00cc30 tx=0x7f7b2c00a5c0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 shutdown_connections 2023-10-16T14:37:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f7b2006af90 0x7f7b2006d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34159950 0x7f7b340c8f20 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b34070450 0x7f7b340c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 --2- 172.21.15.29:0/1274015915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b3406c100 0x7f7b3414a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 >> 172.21.15.29:0/1274015915 conn(0x7f7b340ce6a0 msgr2=0x7f7b3415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 shutdown_connections 2023-10-16T14:37:08.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.103+0000 7f7b389f0640 1 -- 172.21.15.29:0/1274015915 wait complete. 2023-10-16T14:37:08.126 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:37:08.126 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:37:08.126 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:37:08.127 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:37:08.127 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:37:08.127 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:37:08.127 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:37:08.128 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:37:08.128 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:37:08.128 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:37:08.129 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:37:08.129 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:37:08.129 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:37:08.129 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:37:08.130 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:37:08.130 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:37:08.143 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:37:08.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:37:08.146 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:37:08.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:37:08.150 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:37:08.150 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:37:08.151 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:37:08.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:37:08.195 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:37:08.196 DEBUG:teuthology.parallel:result is None 2023-10-16T14:37:08.200 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:37:08.201 DEBUG:teuthology.parallel:result is None 2023-10-16T14:37:08.203 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:37:08.203 DEBUG:teuthology.parallel:result is None 2023-10-16T14:37:08.208 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:37:08.208 DEBUG:teuthology.parallel:result is None 2023-10-16T14:37:08.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:37:08.352 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T14:37:08.354 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T14:37:08.358 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T14:37:08.371 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T14:37:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.388+0000 7f6a6133c640 1 Processor -- start 2023-10-16T14:37:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.388+0000 7f6a6133c640 1 -- start start 2023-10-16T14:37:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.389+0000 7f6a6133c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15d150 0x7f6a5c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.389+0000 7f6a6133c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c0ac890 con 0x7f6a5c160b90 2023-10-16T14:37:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.389+0000 7f6a6133c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c161650 con 0x7f6a5c06a4b0 2023-10-16T14:37:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.389+0000 7f6a6133c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c15d530 con 0x7f6a5c15d150 2023-10-16T14:37:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.390+0000 7f6a5a7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6a5c160b90 0x7f6a5c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37984/0 (socket says 172.21.15.29:37984) 2023-10-16T14:37:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.390+0000 7f6a5affd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15d150 0x7f6a5c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.390+0000 7f6a5a7fc640 1 -- 172.21.15.29:0/953886287 learned_addr learned my addr 172.21.15.29:0/953886287 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.390+0000 7f6a5affd640 1 -- 172.21.15.29:0/953886287 >> v1:172.21.15.170:6789/0 conn(0x7f6a5c06a4b0 legacy=0x7f6a5c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.390+0000 7f6a5affd640 1 -- 172.21.15.29:0/953886287 >> v1:172.21.15.29:6789/0 conn(0x7f6a5c160b90 legacy=0x7f6a5c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.390+0000 7f6a5affd640 1 -- 172.21.15.29:0/953886287 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c161930 con 0x7f6a5c15d150 2023-10-16T14:37:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.391+0000 7f6a5affd640 1 --2- 172.21.15.29:0/953886287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15d150 0x7f6a5c0ad350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0097d0 tx=0x7f6a4c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4fa6749f33f8038 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.391+0000 7f6a59ffb640 1 -- 172.21.15.29:0/953886287 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c030800 con 0x7f6a5c15d150 2023-10-16T14:37:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a59ffb640 1 -- 172.21.15.29:0/953886287 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c030960 con 0x7f6a5c15d150 2023-10-16T14:37:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a59ffb640 1 -- 172.21.15.29:0/953886287 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c030cf0 con 0x7f6a5c15d150 2023-10-16T14:37:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 -- 172.21.15.29:0/953886287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15d150 msgr2=0x7f6a5c0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 --2- 172.21.15.29:0/953886287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15d150 0x7f6a5c0ad350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0097d0 tx=0x7f6a4c02b940 comp rx=0 tx=0).stop 2023-10-16T14:37:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 -- 172.21.15.29:0/953886287 shutdown_connections 2023-10-16T14:37:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 --2- 172.21.15.29:0/953886287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c15d150 0x7f6a5c0ad350 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 -- 172.21.15.29:0/953886287 >> 172.21.15.29:0/953886287 conn(0x7f6a5c0ce9b0 msgr2=0x7f6a5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 -- 172.21.15.29:0/953886287 shutdown_connections 2023-10-16T14:37:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.392+0000 7f6a6133c640 1 -- 172.21.15.29:0/953886287 wait complete. 2023-10-16T14:37:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.394+0000 7f6a6133c640 1 Processor -- start 2023-10-16T14:37:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.394+0000 7f6a6133c640 1 -- start start 2023-10-16T14:37:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a6133c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06a4b0 0x7f6a5c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a6133c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 0x7f6a5c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a6133c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c160b90 0x7f6a5c1599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a6133c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c15fc50 con 0x7f6a5c15d150 2023-10-16T14:37:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a6133c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c160460 con 0x7f6a5c160b90 2023-10-16T14:37:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a6133c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0c5f20 con 0x7f6a5c06a4b0 2023-10-16T14:37:08.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a5a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 0x7f6a5c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:08.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a5a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 0x7f6a5c154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50432/0 (socket says 172.21.15.29:50432) 2023-10-16T14:37:08.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a5a7fc640 1 -- 172.21.15.29:0/3634459324 learned_addr learned my addr 172.21.15.29:0/3634459324 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:08.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a5affd640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06a4b0 0x7f6a5c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:08.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.395+0000 7f6a5b7fe640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c160b90 0x7f6a5c1599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5a7fc640 1 -- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06a4b0 msgr2=0x7f6a5c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5a7fc640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06a4b0 0x7f6a5c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5a7fc640 1 -- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c160b90 msgr2=0x7f6a5c1599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5a7fc640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c160b90 0x7f6a5c1599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5a7fc640 1 -- 172.21.15.29:0/3634459324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c009400 con 0x7f6a5c15d150 2023-10-16T14:37:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5affd640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06a4b0 0x7f6a5c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a5a7fc640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 0x7f6a5c154a60 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000c4d0 tx=0x7f6a5000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50005ce0 con 0x7f6a5c15d150 2023-10-16T14:37:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a50005e40 con 0x7f6a5c15d150 2023-10-16T14:37:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50007270 con 0x7f6a5c15d150 2023-10-16T14:37:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5c15fea0 con 0x7f6a5c15d150 2023-10-16T14:37:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.396+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5c0c5bb0 con 0x7f6a5c15d150 2023-10-16T14:37:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.397+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5c0cdf30 con 0x7f6a5c15d150 2023-10-16T14:37:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.398+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a500073d0 con 0x7f6a5c15d150 2023-10-16T14:37:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.399+0000 7f6a437fe640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f6a4806afc0 0x7f6a4806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.399+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f6a50088b90 con 0x7f6a5c15d150 2023-10-16T14:37:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.400+0000 7f6a5affd640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f6a4806afc0 0x7f6a4806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.401+0000 7f6a5affd640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f6a4806afc0 0x7f6a4806d480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c036000 tx=0x7f6a4c03b040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.403+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a50053bd0 con 0x7f6a5c15d150 2023-10-16T14:37:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:08.602+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f6a5c04e160 con 0x7f6a5c15d150 2023-10-16T14:37:09.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.080+0000 7f6a437fe640 1 -- 172.21.15.29:0/3634459324 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v149) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6a50017020 con 0x7f6a5c15d150 2023-10-16T14:37:09.084 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T14:37:09.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f6a4806afc0 msgr2=0x7f6a4806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f6a4806afc0 0x7f6a4806d480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c036000 tx=0x7f6a4c03b040 comp rx=0 tx=0).stop 2023-10-16T14:37:09.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 msgr2=0x7f6a5c154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 0x7f6a5c154a60 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000c4d0 tx=0x7f6a5000c990 comp rx=0 tx=0).stop 2023-10-16T14:37:09.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 shutdown_connections 2023-10-16T14:37:09.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f6a4806afc0 0x7f6a4806d480 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a5c15d150 0x7f6a5c154a60 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a5c160b90 0x7f6a5c1599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 --2- 172.21.15.29:0/3634459324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a5c06a4b0 0x7f6a5c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 >> 172.21.15.29:0/3634459324 conn(0x7f6a5c0ce9b0 msgr2=0x7f6a5c0ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:09.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.084+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 shutdown_connections 2023-10-16T14:37:09.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.085+0000 7f6a6133c640 1 -- 172.21.15.29:0/3634459324 wait complete. 2023-10-16T14:37:09.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:37:09.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.322+0000 7fb5cd8a3640 1 Processor -- start 2023-10-16T14:37:09.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.322+0000 7fb5cd8a3640 1 -- start start 2023-10-16T14:37:09.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cd8a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cd8a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5c80ccfb0 0x7fb5c80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cd8a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c815b9d0 con 0x7fb5c806c040 2023-10-16T14:37:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cd8a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c815bcb0 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cd8a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c815bf90 con 0x7fb5c80ad970 2023-10-16T14:37:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cd0a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5c80ccfb0 0x7fb5c80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cc8a1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb5c806c040 0x7fb5c806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37994/0 (socket says 172.21.15.29:37994) 2023-10-16T14:37:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.324+0000 7fb5cc8a1640 1 -- 172.21.15.29:0/3000424678 learned_addr learned my addr 172.21.15.29:0/3000424678 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.325+0000 7fb5c7fff640 1 --2- 172.21.15.29:0/3000424678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.325+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/3000424678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 msgr2=0x7fb5c80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.325+0000 7fb5cd0a2640 1 --2- 172.21.15.29:0/3000424678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.325+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/3000424678 >> v1:172.21.15.29:6789/0 conn(0x7fb5c806c040 legacy=0x7fb5c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.325+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/3000424678 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80cc270 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.326+0000 7fb5cd0a2640 1 --2- 172.21.15.29:0/3000424678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80cd440 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009dc0 tx=0x7fb5c002be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d364c9b3ae71da9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5c77fe640 1 -- 172.21.15.29:0/3000424678 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c002cc30 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5c77fe640 1 -- 172.21.15.29:0/3000424678 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c002cd90 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5c77fe640 1 -- 172.21.15.29:0/3000424678 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0034660 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/3000424678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 msgr2=0x7fb5c80cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/3000424678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80cd440 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009dc0 tx=0x7fb5c002be80 comp rx=0 tx=0).stop 2023-10-16T14:37:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/3000424678 shutdown_connections 2023-10-16T14:37:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/3000424678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80cd440 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/3000424678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/3000424678 >> 172.21.15.29:0/3000424678 conn(0x7fb5c80ce6a0 msgr2=0x7fb5c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/3000424678 shutdown_connections 2023-10-16T14:37:09.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.327+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/3000424678 wait complete. 2023-10-16T14:37:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.329+0000 7fb5cd8a3640 1 Processor -- start 2023-10-16T14:37:09.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.329+0000 7fb5cd8a3640 1 -- start start 2023-10-16T14:37:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.330+0000 7fb5cd8a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c806c040 0x7fb5c8156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.330+0000 7fb5cd8a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.330+0000 7fb5cd8a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.330+0000 7fb5cd8a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80ae580 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.330+0000 7fb5cd8a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80c8100 con 0x7fb5c806c040 2023-10-16T14:37:09.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.330+0000 7fb5cd8a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80c7b40 con 0x7fb5c80ad970 2023-10-16T14:37:09.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50436/0 (socket says 172.21.15.29:50436) 2023-10-16T14:37:09.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/2165709120 learned_addr learned my addr 172.21.15.29:0/2165709120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:09.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cc8a1640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c806c040 0x7fb5c8156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5c7fff640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 msgr2=0x7fb5c80c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c806c040 msgr2=0x7fb5c8156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c806c040 0x7fb5c8156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.331+0000 7fb5cd0a2640 1 -- 172.21.15.29:0/2165709120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0009a70 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.332+0000 7fb5cd0a2640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80c95c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009eb0 tx=0x7fb5c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:09.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.332+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c003e610 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.332+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c003e770 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.332+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c003eae0 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.332+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c80c7e20 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.332+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c8156fd0 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.333+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c806b330 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.334+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5c003b050 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.335+0000 7fb5c57fa640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb5ac06af90 0x7fb5ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.335+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fb5c00abf00 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.336+0000 7fb5cc8a1640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb5ac06af90 0x7fb5ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.337+0000 7fb5cc8a1640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb5ac06af90 0x7fb5ac06d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc000f60 tx=0x7fb5bc009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:09.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.339+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb5c00772e0 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.604+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb5c80ae730 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.606+0000 7fb5c57fa640 1 -- 172.21.15.29:0/2165709120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb5c80ae730 con 0x7fb5c80ccfb0 2023-10-16T14:37:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:09.609 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:37:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T14:37:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T14:37:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T14:37:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:37:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:37:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T14:37:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T14:37:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:37:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:37:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T14:37:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T14:37:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:37:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:37:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T14:37:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T14:37:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:37:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:37:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T14:37:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T14:37:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T14:37:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T14:37:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T14:37:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T14:37:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T14:37:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T14:37:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T14:37:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T14:37:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T14:37:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T14:37:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T14:37:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T14:37:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T14:37:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T14:37:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T14:37:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T14:37:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T14:37:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T14:37:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T14:37:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:37:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T14:37:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T14:37:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:37:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:37:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:37:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T14:37:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T14:37:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:37:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:37:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:37:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T14:37:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T14:37:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T14:37:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T14:37:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T14:37:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T14:37:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T14:37:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T14:37:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:37:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T14:37:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T14:37:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T14:37:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T14:37:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:37:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:37:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T14:37:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T14:37:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T14:37:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T14:37:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T14:37:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T14:37:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T14:37:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T14:37:09.643 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:37:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.608+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb5ac06af90 msgr2=0x7fb5ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb5ac06af90 0x7fb5ac06d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc000f60 tx=0x7fb5bc009040 comp rx=0 tx=0).stop 2023-10-16T14:37:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 msgr2=0x7fb5c80c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80c95c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009eb0 tx=0x7fb5c0004a40 comp rx=0 tx=0).stop 2023-10-16T14:37:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 shutdown_connections 2023-10-16T14:37:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb5ac06af90 0x7fb5ac06d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5c80ccfb0 0x7fb5c80c95c0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb5c80ad970 0x7fb5c80c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 --2- 172.21.15.29:0/2165709120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb5c806c040 0x7fb5c8156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 >> 172.21.15.29:0/2165709120 conn(0x7fb5c80ce6a0 msgr2=0x7fb5c80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:09.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 shutdown_connections 2023-10-16T14:37:09.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.609+0000 7fb5cd8a3640 1 -- 172.21.15.29:0/2165709120 wait complete. 2023-10-16T14:37:09.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:37:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.875+0000 7f03efea9640 1 Processor -- start 2023-10-16T14:37:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.875+0000 7f03efea9640 1 -- start start 2023-10-16T14:37:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.877+0000 7f03efea9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e80ce390 0x7f03e815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.877+0000 7f03efea9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e815ed60 con 0x7f03e815ae80 2023-10-16T14:37:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.877+0000 7f03efea9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e80707e0 con 0x7f03e815aaa0 2023-10-16T14:37:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.877+0000 7f03efea9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e806b810 con 0x7f03e80ce390 2023-10-16T14:37:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.877+0000 7f03ed444640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f03e815ae80 0x7f03e8070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38008/0 (socket says 172.21.15.29:38008) 2023-10-16T14:37:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.877+0000 7f03ed444640 1 -- 172.21.15.29:0/1387534475 learned_addr learned my addr 172.21.15.29:0/1387534475 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.878+0000 7f03ee446640 1 --2- 172.21.15.29:0/1387534475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e80ce390 0x7f03e815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.878+0000 7f03ecc43640 1 -- 172.21.15.29:0/1387534475 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672813070 0 0) 0x7f03e815ed60 con 0x7f03e815ae80 2023-10-16T14:37:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.878+0000 7f03ecc43640 1 -- 172.21.15.29:0/1387534475 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d4003520 con 0x7f03e815ae80 2023-10-16T14:37:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.878+0000 7f03ee446640 1 -- 172.21.15.29:0/1387534475 >> v1:172.21.15.170:6789/0 conn(0x7f03e815aaa0 legacy=0x7f03e80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.878+0000 7f03ee446640 1 -- 172.21.15.29:0/1387534475 >> v1:172.21.15.29:6789/0 conn(0x7f03e815ae80 legacy=0x7f03e8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.878+0000 7f03ee446640 1 -- 172.21.15.29:0/1387534475 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e815ef90 con 0x7f03e80ce390 2023-10-16T14:37:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.880+0000 7f03ee446640 1 --2- 172.21.15.29:0/1387534475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e80ce390 0x7f03e815e820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009dc0 tx=0x7f03e002bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77aac0d278093757 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.880+0000 7f03ecc43640 1 -- 172.21.15.29:0/1387534475 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e002ccc0 con 0x7f03e80ce390 2023-10-16T14:37:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.880+0000 7f03ecc43640 1 -- 172.21.15.29:0/1387534475 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e002ce20 con 0x7f03e80ce390 2023-10-16T14:37:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.880+0000 7f03ecc43640 1 -- 172.21.15.29:0/1387534475 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0034660 con 0x7f03e80ce390 2023-10-16T14:37:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 -- 172.21.15.29:0/1387534475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e80ce390 msgr2=0x7f03e815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 --2- 172.21.15.29:0/1387534475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e80ce390 0x7f03e815e820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009dc0 tx=0x7f03e002bf10 comp rx=0 tx=0).stop 2023-10-16T14:37:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 -- 172.21.15.29:0/1387534475 shutdown_connections 2023-10-16T14:37:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 --2- 172.21.15.29:0/1387534475 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e80ce390 0x7f03e815e820 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 -- 172.21.15.29:0/1387534475 >> 172.21.15.29:0/1387534475 conn(0x7f03e80cb410 msgr2=0x7f03e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:09.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 -- 172.21.15.29:0/1387534475 shutdown_connections 2023-10-16T14:37:09.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.881+0000 7f03efea9640 1 -- 172.21.15.29:0/1387534475 wait complete. 2023-10-16T14:37:09.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.884+0000 7f03efea9640 1 Processor -- start 2023-10-16T14:37:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.884+0000 7f03efea9640 1 -- start start 2023-10-16T14:37:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03efea9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 0x7f03e80b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03efea9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e815aaa0 0x7f03e814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03efea9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e815ae80 0x7f03e80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03efea9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80af570 con 0x7f03e80ce390 2023-10-16T14:37:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03efea9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80afe10 con 0x7f03e815ae80 2023-10-16T14:37:09.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03efea9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80b0cc0 con 0x7f03e815aaa0 2023-10-16T14:37:09.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03edc45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 0x7f03e80b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03edc45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 0x7f03e80b02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50448/0 (socket says 172.21.15.29:50448) 2023-10-16T14:37:09.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.885+0000 7f03edc45640 1 -- 172.21.15.29:0/1658448606 learned_addr learned my addr 172.21.15.29:0/1658448606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:09.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03ed444640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e815aaa0 0x7f03e814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03edc45640 1 -- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e815aaa0 msgr2=0x7f03e814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03ee446640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e815ae80 0x7f03e80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03edc45640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e815aaa0 0x7f03e814ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03edc45640 1 -- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e815ae80 msgr2=0x7f03e80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03edc45640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e815ae80 0x7f03e80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03edc45640 1 -- 172.21.15.29:0/1658448606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e0009a70 con 0x7f03e80ce390 2023-10-16T14:37:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03ee446640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e815ae80 0x7f03e80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.886+0000 7f03edc45640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 0x7f03e80b02a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c480 tx=0x7f03e400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.887+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e40052d0 con 0x7f03e80ce390 2023-10-16T14:37:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.887+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e4005430 con 0x7f03e80ce390 2023-10-16T14:37:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.887+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4012870 con 0x7f03e80ce390 2023-10-16T14:37:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.887+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e80c86c0 con 0x7f03e80ce390 2023-10-16T14:37:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.887+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e80afaa0 con 0x7f03e80ce390 2023-10-16T14:37:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.888+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e804e160 con 0x7f03e80ce390 2023-10-16T14:37:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.889+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03e40129d0 con 0x7f03e80ce390 2023-10-16T14:37:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.890+0000 7f03de7fc640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f03cc06af90 0x7f03cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.890+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f03e4088ff0 con 0x7f03e80ce390 2023-10-16T14:37:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.890+0000 7f03ed444640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f03cc06af90 0x7f03cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.891+0000 7f03ed444640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f03cc06af90 0x7f03cc06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03e8159d20 tx=0x7f03d8009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:09.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:09.894+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03e40bbca0 con 0x7f03e80ce390 2023-10-16T14:37:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.088+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03e80ce010 con 0x7f03e80ce390 2023-10-16T14:37:10.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.092+0000 7f03de7fc640 1 -- 172.21.15.29:0/1658448606 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f03e40542f0 con 0x7f03e80ce390 2023-10-16T14:37:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:10.095 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":151,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:37:09.092640+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:37:10.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.094+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f03cc06af90 msgr2=0x7f03cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.094+0000 7f03efea9640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f03cc06af90 0x7f03cc06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03e8159d20 tx=0x7f03d8009040 comp rx=0 tx=0).stop 2023-10-16T14:37:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 msgr2=0x7f03e80b02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 0x7f03e80b02a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c480 tx=0x7f03e400c940 comp rx=0 tx=0).stop 2023-10-16T14:37:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 shutdown_connections 2023-10-16T14:37:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f03cc06af90 0x7f03cc06d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03e815ae80 0x7f03e80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03e815aaa0 0x7f03e814ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 --2- 172.21.15.29:0/1658448606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03e80ce390 0x7f03e80b02a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 >> 172.21.15.29:0/1658448606 conn(0x7f03e80cb410 msgr2=0x7f03e806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:10.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 shutdown_connections 2023-10-16T14:37:10.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.095+0000 7f03efea9640 1 -- 172.21.15.29:0/1658448606 wait complete. 2023-10-16T14:37:10.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:37:10.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:37:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.348+0000 7fad2944b640 1 Processor -- start 2023-10-16T14:37:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.348+0000 7fad2944b640 1 -- start start 2023-10-16T14:37:10.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.349+0000 7fad2944b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159ab0 0x7fad240ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad2944b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fad24159e90 0x7fad2406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad2944b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad240cd1f0 con 0x7fad2406c140 2023-10-16T14:37:10.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad2944b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2415d890 con 0x7fad24159e90 2023-10-16T14:37:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad2944b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad240cd4d0 con 0x7fad24159ab0 2023-10-16T14:37:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad237fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fad2406c140 0x7fad2415dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38022/0 (socket says 172.21.15.29:38022) 2023-10-16T14:37:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad237fe640 1 -- 172.21.15.29:0/952996434 learned_addr learned my addr 172.21.15.29:0/952996434 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad227fc640 1 --2- 172.21.15.29:0/952996434 >> v2:172.21.15.29:3300/0 conn(0x7fad24159e90 0x7fad2406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.350+0000 7fad22ffd640 1 --2- 172.21.15.29:0/952996434 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159ab0 0x7fad240ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.351+0000 7fad21ffb640 1 -- 172.21.15.29:0/952996434 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071863149 0 0) 0x7fad240cd1f0 con 0x7fad2406c140 2023-10-16T14:37:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.351+0000 7fad227fc640 1 -- 172.21.15.29:0/952996434 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159ab0 msgr2=0x7fad240ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.351+0000 7fad227fc640 1 --2- 172.21.15.29:0/952996434 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159ab0 0x7fad240ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.351+0000 7fad227fc640 1 -- 172.21.15.29:0/952996434 >> v1:172.21.15.29:6789/0 conn(0x7fad2406c140 legacy=0x7fad2415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.351+0000 7fad227fc640 1 -- 172.21.15.29:0/952996434 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad240cd7b0 con 0x7fad24159e90 2023-10-16T14:37:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.352+0000 7fad227fc640 1 --2- 172.21.15.29:0/952996434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159e90 0x7fad2406fc50 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fad180097d0 tx=0x7fad1802d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab49b12c38f9779b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.352+0000 7fad21ffb640 1 -- 172.21.15.29:0/952996434 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18032860 con 0x7fad24159e90 2023-10-16T14:37:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.352+0000 7fad21ffb640 1 -- 172.21.15.29:0/952996434 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad180329c0 con 0x7fad24159e90 2023-10-16T14:37:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.352+0000 7fad21ffb640 1 -- 172.21.15.29:0/952996434 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18032d50 con 0x7fad24159e90 2023-10-16T14:37:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 -- 172.21.15.29:0/952996434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159e90 msgr2=0x7fad2406fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 --2- 172.21.15.29:0/952996434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159e90 0x7fad2406fc50 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fad180097d0 tx=0x7fad1802d9a0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 -- 172.21.15.29:0/952996434 shutdown_connections 2023-10-16T14:37:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 --2- 172.21.15.29:0/952996434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159e90 0x7fad2406fc50 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 --2- 172.21.15.29:0/952996434 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159ab0 0x7fad240ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 -- 172.21.15.29:0/952996434 >> 172.21.15.29:0/952996434 conn(0x7fad240ce9b0 msgr2=0x7fad240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 -- 172.21.15.29:0/952996434 shutdown_connections 2023-10-16T14:37:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.353+0000 7fad2944b640 1 -- 172.21.15.29:0/952996434 wait complete. 2023-10-16T14:37:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.355+0000 7fad2944b640 1 Processor -- start 2023-10-16T14:37:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.355+0000 7fad2944b640 1 -- start start 2023-10-16T14:37:10.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.356+0000 7fad2944b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad2406c140 0x7fad240c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.356+0000 7fad2944b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 0x7fad2414df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.356+0000 7fad2944b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159e90 0x7fad2406c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.356+0000 7fad2944b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad240c4590 con 0x7fad24159ab0 2023-10-16T14:37:10.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.356+0000 7fad2944b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad241527a0 con 0x7fad2406c140 2023-10-16T14:37:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.356+0000 7fad2944b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad240c3a10 con 0x7fad24159e90 2023-10-16T14:37:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 0x7fad2414df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 0x7fad2414df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50480/0 (socket says 172.21.15.29:50480) 2023-10-16T14:37:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 -- 172.21.15.29:0/4181338219 learned_addr learned my addr 172.21.15.29:0/4181338219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:10.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad237fe640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159e90 0x7fad2406c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad22ffd640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad2406c140 0x7fad240c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 -- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159e90 msgr2=0x7fad2406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159e90 0x7fad2406c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 -- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad2406c140 msgr2=0x7fad240c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad2406c140 0x7fad240c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.357+0000 7fad227fc640 1 -- 172.21.15.29:0/4181338219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad18009400 con 0x7fad24159ab0 2023-10-16T14:37:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.358+0000 7fad227fc640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 0x7fad2414df40 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fad1802dfb0 tx=0x7fad180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.358+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1803cca0 con 0x7fad24159ab0 2023-10-16T14:37:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.358+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1803ce20 con 0x7fad24159ab0 2023-10-16T14:37:10.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.358+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18046630 con 0x7fad24159ab0 2023-10-16T14:37:10.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.358+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad240c3cf0 con 0x7fad24159ab0 2023-10-16T14:37:10.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.358+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad240c42b0 con 0x7fad24159ab0 2023-10-16T14:37:10.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.359+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2404e160 con 0x7fad24159ab0 2023-10-16T14:37:10.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.360+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fad18046790 con 0x7fad24159ab0 2023-10-16T14:37:10.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.361+0000 7fad0b7fe640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fad1006af90 0x7fad1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.361+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fad180ac070 con 0x7fad24159ab0 2023-10-16T14:37:10.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.362+0000 7fad22ffd640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fad1006af90 0x7fad1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.363+0000 7fad22ffd640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fad1006af90 0x7fad1006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fad140099c0 tx=0x7fad14003990 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:10.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.366+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fad180772c0 con 0x7fad24159ab0 2023-10-16T14:37:10.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.562+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad240ced90 con 0x7fad24159ab0 2023-10-16T14:37:10.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.563+0000 7fad0b7fe640 1 -- 172.21.15.29:0/4181338219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v151) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad1807a8e0 con 0x7fad24159ab0 2023-10-16T14:37:10.566 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T14:37:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fad1006af90 msgr2=0x7fad1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fad1006af90 0x7fad1006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fad140099c0 tx=0x7fad14003990 comp rx=0 tx=0).stop 2023-10-16T14:37:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 msgr2=0x7fad2414df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 0x7fad2414df40 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fad1802dfb0 tx=0x7fad180046c0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 shutdown_connections 2023-10-16T14:37:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fad1006af90 0x7fad1006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fad24159ab0 0x7fad2414df40 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fad24159e90 0x7fad2406c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 --2- 172.21.15.29:0/4181338219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fad2406c140 0x7fad240c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 >> 172.21.15.29:0/4181338219 conn(0x7fad240ce9b0 msgr2=0x7fad240b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 shutdown_connections 2023-10-16T14:37:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.566+0000 7fad2944b640 1 -- 172.21.15.29:0/4181338219 wait complete. 2023-10-16T14:37:10.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:37:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.809+0000 7f105bfff640 1 Processor -- start 2023-10-16T14:37:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.809+0000 7f105bfff640 1 -- start start 2023-10-16T14:37:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.811+0000 7f105bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f105c15b320 0x7f105c0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.811+0000 7f105bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c15b9d0 con 0x7f105c06c040 2023-10-16T14:37:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.811+0000 7f105bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c15bcb0 con 0x7f105c0ad970 2023-10-16T14:37:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.811+0000 7f105bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105c15b7d0 con 0x7f105c15b320 2023-10-16T14:37:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.811+0000 7f105affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f105c06c040 0x7f105c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38028/0 (socket says 172.21.15.29:38028) 2023-10-16T14:37:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.811+0000 7f105affd640 1 -- 172.21.15.29:0/3759282021 learned_addr learned my addr 172.21.15.29:0/3759282021 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267817162 0 0) 0x7f105c15b9d0 con 0x7f105c06c040 2023-10-16T14:37:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1044003520 con 0x7f105c06c040 2023-10-16T14:37:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f105b7fe640 1 --2- 172.21.15.29:0/3759282021 >> v2:172.21.15.170:3300/0 conn(0x7f105c15b320 0x7f105c0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383309165 0 0) 0x7f1044003520 con 0x7f105c06c040 2023-10-16T14:37:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 >> v2:172.21.15.170:3300/0 conn(0x7f105c15b320 msgr2=0x7f105c0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 --2- 172.21.15.29:0/3759282021 >> v2:172.21.15.170:3300/0 conn(0x7f105c15b320 0x7f105c0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 >> v1:172.21.15.170:6789/0 conn(0x7f105c0ad970 legacy=0x7f105c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c15bf90 con 0x7f105c06c040 2023-10-16T14:37:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f104c001670 con 0x7f105c06c040 2023-10-16T14:37:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 >> v1:172.21.15.29:6789/0 conn(0x7f105c06c040 legacy=0x7f105c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10440039f0 0x7f1044003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440047c0 0x7f1044004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10440052a0 0x7f104400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c15b7d0 con 0x7f10440039f0 2023-10-16T14:37:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104400c150 con 0x7f10440052a0 2023-10-16T14:37:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.812+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104400c310 con 0x7f10440047c0 2023-10-16T14:37:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105affd640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10440039f0 0x7f1044003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105a7fc640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440047c0 0x7f1044004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105b7fe640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10440052a0 0x7f104400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105affd640 1 -- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440047c0 msgr2=0x7f1044004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105affd640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440047c0 0x7f1044004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105affd640 1 -- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10440052a0 msgr2=0x7f104400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105affd640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f10440052a0 0x7f104400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105affd640 1 -- 172.21.15.29:0/3759282021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044003520 con 0x7f10440039f0 2023-10-16T14:37:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105b7fe640 1 -- 172.21.15.29:0/3759282021 reap_dead start 2023-10-16T14:37:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.813+0000 7f105a7fc640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440047c0 0x7f1044004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.815+0000 7f105affd640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10440039f0 0x7f1044003e10 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f104c001eb0 tx=0x7f10440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5049a51be4c8670c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:10.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.815+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c005ad0 con 0x7f10440039f0 2023-10-16T14:37:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.815+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c005c30 con 0x7f10440039f0 2023-10-16T14:37:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.815+0000 7f1059ffb640 1 -- 172.21.15.29:0/3759282021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c02ca00 con 0x7f10440039f0 2023-10-16T14:37:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 -- 172.21.15.29:0/3759282021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10440039f0 msgr2=0x7f1044003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10440039f0 0x7f1044003e10 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f104c001eb0 tx=0x7f10440057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 -- 172.21.15.29:0/3759282021 shutdown_connections 2023-10-16T14:37:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 --2- 172.21.15.29:0/3759282021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10440039f0 0x7f1044003e10 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 -- 172.21.15.29:0/3759282021 >> 172.21.15.29:0/3759282021 conn(0x7f105c0ce6a0 msgr2=0x7f105c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 -- 172.21.15.29:0/3759282021 shutdown_connections 2023-10-16T14:37:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.816+0000 7f105bfff640 1 -- 172.21.15.29:0/3759282021 wait complete. 2023-10-16T14:37:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.818+0000 7f105bfff640 1 Processor -- start 2023-10-16T14:37:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.818+0000 7f105bfff640 1 -- start start 2023-10-16T14:37:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440039f0 0x7f105c1f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f105c0c7340 0x7f105c0c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 0x7f105c0b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0c7d00 con 0x7f105c0b2d60 2023-10-16T14:37:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0b3700 con 0x7f105c0c7340 2023-10-16T14:37:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0c8e90 con 0x7f10440039f0 2023-10-16T14:37:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 0x7f105c0b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 0x7f105c0b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50492/0 (socket says 172.21.15.29:50492) 2023-10-16T14:37:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.819+0000 7f105b7fe640 1 -- 172.21.15.29:0/404779241 learned_addr learned my addr 172.21.15.29:0/404779241 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105a7fc640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f105c0c7340 0x7f105c0c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105affd640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440039f0 0x7f105c1f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105b7fe640 1 -- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440039f0 msgr2=0x7f105c1f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105b7fe640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440039f0 0x7f105c1f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105b7fe640 1 -- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f105c0c7340 msgr2=0x7f105c0c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105b7fe640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f105c0c7340 0x7f105c0c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105b7fe640 1 -- 172.21.15.29:0/404779241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c0c9170 con 0x7f105c0b2d60 2023-10-16T14:37:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.820+0000 7f105b7fe640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 0x7f105c0b31c0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f105c15b700 tx=0x7f1054007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.821+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054007ea0 con 0x7f105c0b2d60 2023-10-16T14:37:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.821+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105400f050 con 0x7f105c0b2d60 2023-10-16T14:37:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.821+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054014650 con 0x7f105c0b2d60 2023-10-16T14:37:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.821+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c0c9450 con 0x7f105c0b2d60 2023-10-16T14:37:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.821+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c0c9a10 con 0x7f105c0b2d60 2023-10-16T14:37:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.822+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105c06b330 con 0x7f105c0b2d60 2023-10-16T14:37:10.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.823+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10540147b0 con 0x7f105c0b2d60 2023-10-16T14:37:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.824+0000 7f10437fe640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f104806afc0 0x7f104806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.824+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f1054089b30 con 0x7f105c0b2d60 2023-10-16T14:37:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.825+0000 7f105affd640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f104806afc0 0x7f104806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.826+0000 7f105affd640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f104806afc0 0x7f104806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f104c034000 tx=0x7f104c009030 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:10.829+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1054054c50 con 0x7f105c0b2d60 2023-10-16T14:37:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.030+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f105c04e9b0 con 0x7f105c0b2d60 2023-10-16T14:37:11.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.030+0000 7f10437fe640 1 -- 172.21.15.29:0/404779241 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v151) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1054058270 con 0x7f105c0b2d60 2023-10-16T14:37:11.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:11.034 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:37:11.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:11.034 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T14:37:11.035 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T14:37:11.035 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:11.035 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:37:11.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f104806afc0 msgr2=0x7f104806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:11.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f104806afc0 0x7f104806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f104c034000 tx=0x7f104c009030 comp rx=0 tx=0).stop 2023-10-16T14:37:11.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 msgr2=0x7f105c0b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:11.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 0x7f105c0b31c0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f105c15b700 tx=0x7f1054007700 comp rx=0 tx=0).stop 2023-10-16T14:37:11.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 shutdown_connections 2023-10-16T14:37:11.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f104806afc0 0x7f104806d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f105c0b2d60 0x7f105c0b31c0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f105c0c7340 0x7f105c0c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 --2- 172.21.15.29:0/404779241 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f10440039f0 0x7f105c1f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 >> 172.21.15.29:0/404779241 conn(0x7f105c0ce6a0 msgr2=0x7f105c0c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:11.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 shutdown_connections 2023-10-16T14:37:11.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.033+0000 7f105bfff640 1 -- 172.21.15.29:0/404779241 wait complete. 2023-10-16T14:37:11.055 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:37:11.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:37:11.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.273+0000 7f5f67fff640 1 Processor -- start 2023-10-16T14:37:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.273+0000 7f5f67fff640 1 -- start start 2023-10-16T14:37:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.274+0000 7f5f67fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 0x7f5f680cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.274+0000 7f5f67fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5f680b4ca0 0x7f5f680d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:11.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.274+0000 7f5f67fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f680b8430 con 0x7f5f6815d100 2023-10-16T14:37:11.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.274+0000 7f5f67fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f680d2160 con 0x7f5f680b4ca0 2023-10-16T14:37:11.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.274+0000 7f5f67fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f680ce0b0 con 0x7f5f6815cd20 2023-10-16T14:37:11.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.275+0000 7f5f66ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 0x7f5f680cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:11.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.275+0000 7f5f66ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 0x7f5f680cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:59072/0 (socket says 172.21.15.29:59072) 2023-10-16T14:37:11.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.275+0000 7f5f66ffd640 1 -- 172.21.15.29:0/2736281699 learned_addr learned my addr 172.21.15.29:0/2736281699 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:11.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.275+0000 7f5f677fe640 1 --2- 172.21.15.29:0/2736281699 >> v2:172.21.15.170:3300/0 conn(0x7f5f680b4ca0 0x7f5f680d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:11.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.276+0000 7f5f66ffd640 1 -- 172.21.15.29:0/2736281699 >> v2:172.21.15.170:3300/0 conn(0x7f5f680b4ca0 msgr2=0x7f5f680d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:11.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.276+0000 7f5f66ffd640 1 --2- 172.21.15.29:0/2736281699 >> v2:172.21.15.170:3300/0 conn(0x7f5f680b4ca0 0x7f5f680d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.276+0000 7f5f66ffd640 1 -- 172.21.15.29:0/2736281699 >> v1:172.21.15.170:6789/0 conn(0x7f5f6815d100 legacy=0x7f5f680b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:11.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.276+0000 7f5f66ffd640 1 -- 172.21.15.29:0/2736281699 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f680b8680 con 0x7f5f6815cd20 2023-10-16T14:37:11.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.277+0000 7f5f66ffd640 1 --2- 172.21.15.29:0/2736281699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 0x7f5f680cdb70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009e50 tx=0x7f5f5802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d384e5cea60682a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:11.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f65ffb640 1 -- 172.21.15.29:0/2736281699 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5802cb90 con 0x7f5f6815cd20 2023-10-16T14:37:11.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f65ffb640 1 -- 172.21.15.29:0/2736281699 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5802ccf0 con 0x7f5f6815cd20 2023-10-16T14:37:11.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f65ffb640 1 -- 172.21.15.29:0/2736281699 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58034610 con 0x7f5f6815cd20 2023-10-16T14:37:11.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f67fff640 1 -- 172.21.15.29:0/2736281699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 msgr2=0x7f5f680cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f67fff640 1 --2- 172.21.15.29:0/2736281699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 0x7f5f680cdb70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009e50 tx=0x7f5f5802bff0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f67fff640 1 -- 172.21.15.29:0/2736281699 shutdown_connections 2023-10-16T14:37:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f67fff640 1 --2- 172.21.15.29:0/2736281699 >> v2:172.21.15.170:3300/0 conn(0x7f5f680b4ca0 0x7f5f680d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f67fff640 1 --2- 172.21.15.29:0/2736281699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815cd20 0x7f5f680cdb70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.278+0000 7f5f67fff640 1 -- 172.21.15.29:0/2736281699 >> 172.21.15.29:0/2736281699 conn(0x7f5f680cf590 msgr2=0x7f5f680cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.279+0000 7f5f67fff640 1 -- 172.21.15.29:0/2736281699 shutdown_connections 2023-10-16T14:37:11.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.279+0000 7f5f67fff640 1 -- 172.21.15.29:0/2736281699 wait complete. 2023-10-16T14:37:11.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.280+0000 7f5f67fff640 1 Processor -- start 2023-10-16T14:37:11.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.280+0000 7f5f67fff640 1 -- start start 2023-10-16T14:37:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.281+0000 7f5f67fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f680b4ca0 0x7f5f680b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.281+0000 7f5f67fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 0x7f5f6815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.281+0000 7f5f67fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815d100 0x7f5f6815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.281+0000 7f5f67fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6815f090 con 0x7f5f6815cd20 2023-10-16T14:37:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.281+0000 7f5f67fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f680c9be0 con 0x7f5f680b4ca0 2023-10-16T14:37:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.281+0000 7f5f67fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f680c9e30 con 0x7f5f6815d100 2023-10-16T14:37:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 0x7f5f6815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 0x7f5f6815e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50508/0 (socket says 172.21.15.29:50508) 2023-10-16T14:37:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 -- 172.21.15.29:0/1778838228 learned_addr learned my addr 172.21.15.29:0/1778838228 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f66ffd640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f680b4ca0 0x7f5f680b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f677fe640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815d100 0x7f5f6815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 -- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815d100 msgr2=0x7f5f6815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815d100 0x7f5f6815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 -- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f680b4ca0 msgr2=0x7f5f680b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f680b4ca0 0x7f5f680b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f667fc640 1 -- 172.21.15.29:0/1778838228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f58009b30 con 0x7f5f6815cd20 2023-10-16T14:37:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.282+0000 7f5f677fe640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815d100 0x7f5f6815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.283+0000 7f5f667fc640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 0x7f5f6815e610 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c400 tx=0x7f5f5000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:11.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.283+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500052d0 con 0x7f5f6815cd20 2023-10-16T14:37:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.283+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f50005430 con 0x7f5f6815cd20 2023-10-16T14:37:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.283+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50003830 con 0x7f5f6815cd20 2023-10-16T14:37:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.283+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f680ca110 con 0x7f5f6815cd20 2023-10-16T14:37:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.283+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f680ca6d0 con 0x7f5f6815cd20 2023-10-16T14:37:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.284+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f680ca890 con 0x7f5f6815cd20 2023-10-16T14:37:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.285+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f50003990 con 0x7f5f6815cd20 2023-10-16T14:37:11.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.286+0000 7f5f5ffff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5f4806afc0 0x7f5f4806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:11.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.286+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f5f500888d0 con 0x7f5f6815cd20 2023-10-16T14:37:11.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.287+0000 7f5f66ffd640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5f4806afc0 0x7f5f4806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:11.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.288+0000 7f5f66ffd640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5f4806afc0 0x7f5f4806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5f58035000 tx=0x7f5f58030000 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:11.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.290+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f50053c50 con 0x7f5f6815cd20 2023-10-16T14:37:11.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:11.485+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5f6804e9b0 con 0x7f5f6815cd20 2023-10-16T14:37:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.104+0000 7f5f5ffff640 1 -- 172.21.15.29:0/1778838228 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v152) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5f50057270 con 0x7f5f6815cd20 2023-10-16T14:37:12.108 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T14:37:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5f4806afc0 msgr2=0x7f5f4806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5f4806afc0 0x7f5f4806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5f58035000 tx=0x7f5f58030000 comp rx=0 tx=0).stop 2023-10-16T14:37:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 msgr2=0x7f5f6815e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 0x7f5f6815e610 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c400 tx=0x7f5f5000c8c0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 shutdown_connections 2023-10-16T14:37:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f5f4806afc0 0x7f5f4806d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f6815cd20 0x7f5f6815e610 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f6815d100 0x7f5f6815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 --2- 172.21.15.29:0/1778838228 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f680b4ca0 0x7f5f680b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 >> 172.21.15.29:0/1778838228 conn(0x7f5f680cf590 msgr2=0x7f5f6815ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.108+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 shutdown_connections 2023-10-16T14:37:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.109+0000 7f5f67fff640 1 -- 172.21.15.29:0/1778838228 wait complete. 2023-10-16T14:37:12.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:37:12.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.342+0000 7fbe81311640 1 Processor -- start 2023-10-16T14:37:12.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.342+0000 7fbe81311640 1 -- start start 2023-10-16T14:37:12.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.343+0000 7fbe81311640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe7c0ca1a0 0x7fbe7c0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:12.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.343+0000 7fbe81311640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbe7c06f940 0x7fbe7c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:12.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.343+0000 7fbe81311640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7c161760 con 0x7fbe7c0ccb90 2023-10-16T14:37:12.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.343+0000 7fbe81311640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7c0cd010 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.343+0000 7fbe81311640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7c0cd2f0 con 0x7fbe7c06f940 2023-10-16T14:37:12.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe7c0ca1a0 0x7fbe7c0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:12.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe7c0ca1a0 0x7fbe7c0cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50514/0 (socket says 172.21.15.29:50514) 2023-10-16T14:37:12.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 -- 172.21.15.29:0/3870842927 learned_addr learned my addr 172.21.15.29:0/3870842927 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:12.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7b7fe640 1 --1- 172.21.15.29:0/3870842927 >> v1:172.21.15.29:6789/0 conn(0x7fbe7c0ccb90 0x7fbe7c06f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38044/0 (socket says 172.21.15.29:38044) 2023-10-16T14:37:12.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe80b10640 1 --2- 172.21.15.29:0/3870842927 >> v2:172.21.15.170:3300/0 conn(0x7fbe7c06f940 0x7fbe7c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:12.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 -- 172.21.15.29:0/3870842927 >> v2:172.21.15.170:3300/0 conn(0x7fbe7c06f940 msgr2=0x7fbe7c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:12.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 --2- 172.21.15.29:0/3870842927 >> v2:172.21.15.170:3300/0 conn(0x7fbe7c06f940 0x7fbe7c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 -- 172.21.15.29:0/3870842927 >> v1:172.21.15.29:6789/0 conn(0x7fbe7c0ccb90 legacy=0x7fbe7c06f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:12.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.344+0000 7fbe7bfff640 1 -- 172.21.15.29:0/3870842927 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c0cd5d0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.345+0000 7fbe7bfff640 1 --2- 172.21.15.29:0/3870842927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c0cc650 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0addd0 tx=0x7fbe6c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531d6ef8964c992f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:12.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.346+0000 7fbe7affd640 1 -- 172.21.15.29:0/3870842927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c02f850 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.346+0000 7fbe7affd640 1 -- 172.21.15.29:0/3870842927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c02f9b0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.346+0000 7fbe7affd640 1 -- 172.21.15.29:0/3870842927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c02fd40 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.346+0000 7fbe81311640 1 -- 172.21.15.29:0/3870842927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 msgr2=0x7fbe7c0cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:12.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.346+0000 7fbe81311640 1 --2- 172.21.15.29:0/3870842927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c0cc650 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0addd0 tx=0x7fbe6c02ba30 comp rx=0 tx=0).stop 2023-10-16T14:37:12.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.347+0000 7fbe81311640 1 -- 172.21.15.29:0/3870842927 shutdown_connections 2023-10-16T14:37:12.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.347+0000 7fbe81311640 1 --2- 172.21.15.29:0/3870842927 >> v2:172.21.15.170:3300/0 conn(0x7fbe7c06f940 0x7fbe7c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.347+0000 7fbe81311640 1 --2- 172.21.15.29:0/3870842927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c0cc650 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.347+0000 7fbe81311640 1 -- 172.21.15.29:0/3870842927 >> 172.21.15.29:0/3870842927 conn(0x7fbe7c0ce6a0 msgr2=0x7fbe7c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:12.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.347+0000 7fbe81311640 1 -- 172.21.15.29:0/3870842927 shutdown_connections 2023-10-16T14:37:12.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.347+0000 7fbe81311640 1 -- 172.21.15.29:0/3870842927 wait complete. 2023-10-16T14:37:12.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.348+0000 7fbe81311640 1 Processor -- start 2023-10-16T14:37:12.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.349+0000 7fbe81311640 1 -- start start 2023-10-16T14:37:12.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.349+0000 7fbe81311640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe7c06f940 0x7fbe7c0c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:12.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.349+0000 7fbe81311640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe81311640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe7c0ccb90 0x7fbe7c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe81311640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0b2bf0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe81311640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0c52f0 con 0x7fbe7c0ccb90 2023-10-16T14:37:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe81311640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0b2e90 con 0x7fbe7c06f940 2023-10-16T14:37:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:12.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50526/0 (socket says 172.21.15.29:50526) 2023-10-16T14:37:12.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 -- 172.21.15.29:0/4176542919 learned_addr learned my addr 172.21.15.29:0/4176542919 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:12.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe80b10640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe7c0ccb90 0x7fbe7c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:12.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7bfff640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe7c06f940 0x7fbe7c0c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:12.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 -- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe7c06f940 msgr2=0x7fbe7c0c4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe7c06f940 0x7fbe7c0c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 -- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe7c0ccb90 msgr2=0x7fbe7c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe7c0ccb90 0x7fbe7c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.350+0000 7fbe7b7fe640 1 -- 172.21.15.29:0/4176542919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c0094d0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.351+0000 7fbe7b7fe640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c155d80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fbe7000c3c0 tx=0x7fbe7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:12.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.351+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700074e0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.351+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70007640 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.351+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70004a00 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.351+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7c0b3170 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.351+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7c0b3730 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.352+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7c04e9b0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.354+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe70004b60 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.355+0000 7fbe78ff9640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbe6806af90 0x7fbe6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.355+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fbe70088fe0 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.356+0000 7fbe7bfff640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbe6806af90 0x7fbe6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:12.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.357+0000 7fbe7bfff640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbe6806af90 0x7fbe6806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c036000 tx=0x7fbe6c008030 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:12.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.360+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe70054170 con 0x7fbe7c0ca1a0 2023-10-16T14:37:12.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:12.554+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbe7c0cee00 con 0x7fbe7c0ca1a0 2023-10-16T14:37:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.122+0000 7fbe78ff9640 1 -- 172.21.15.29:0/4176542919 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v153) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbe70057790 con 0x7fbe7c0ca1a0 2023-10-16T14:37:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T14:37:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.126+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbe6806af90 msgr2=0x7fbe6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.126+0000 7fbe81311640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbe6806af90 0x7fbe6806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c036000 tx=0x7fbe6c008030 comp rx=0 tx=0).stop 2023-10-16T14:37:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.126+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 msgr2=0x7fbe7c155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.126+0000 7fbe81311640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c155d80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fbe7000c3c0 tx=0x7fbe7000c880 comp rx=0 tx=0).stop 2023-10-16T14:37:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.127+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 shutdown_connections 2023-10-16T14:37:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.127+0000 7fbe81311640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbe6806af90 0x7fbe6806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.127+0000 7fbe81311640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe7c0ca1a0 0x7fbe7c155d80 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.127+0000 7fbe81311640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe7c0ccb90 0x7fbe7c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.127+0000 7fbe81311640 1 --2- 172.21.15.29:0/4176542919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe7c06f940 0x7fbe7c0c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.127+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 >> 172.21.15.29:0/4176542919 conn(0x7fbe7c0ce6a0 msgr2=0x7fbe7c163940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:13.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.128+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 shutdown_connections 2023-10-16T14:37:13.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.128+0000 7fbe81311640 1 -- 172.21.15.29:0/4176542919 wait complete. 2023-10-16T14:37:13.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:37:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.354+0000 7f0fefc46640 1 Processor -- start 2023-10-16T14:37:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.354+0000 7f0fefc46640 1 -- start start 2023-10-16T14:37:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.355+0000 7f0fefc46640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 0x7f0fe80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.355+0000 7f0fefc46640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe8070360 con 0x7f0fe806c140 2023-10-16T14:37:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.356+0000 7f0fefc46640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe80cd1f0 con 0x7f0fe8159ab0 2023-10-16T14:37:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.356+0000 7f0fefc46640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe815d890 con 0x7f0fe8159e90 2023-10-16T14:37:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.356+0000 7f0fee1e3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0fe806c140 0x7f0fe815dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38050/0 (socket says 172.21.15.29:38050) 2023-10-16T14:37:13.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.356+0000 7f0fee1e3640 1 -- 172.21.15.29:0/3783402799 learned_addr learned my addr 172.21.15.29:0/3783402799 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:13.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.356+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 0x7f0fe80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.356+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065758889 0 0) 0x7f0fe8070360 con 0x7f0fe806c140 2023-10-16T14:37:13.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4005730 con 0x7f0fe806c140 2023-10-16T14:37:13.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197730704 0 0) 0x7f0fe80cd1f0 con 0x7f0fe8159ab0 2023-10-16T14:37:13.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe8070360 con 0x7f0fe8159ab0 2023-10-16T14:37:13.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3752718720 0 0) 0x7f0fd4005730 con 0x7f0fe806c140 2023-10-16T14:37:13.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 msgr2=0x7f0fe80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:13.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 0x7f0fe80ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 >> v1:172.21.15.170:6789/0 conn(0x7f0fe8159ab0 legacy=0x7f0fe806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80cd4d0 con 0x7f0fe806c140 2023-10-16T14:37:13.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0fe0001670 con 0x7f0fe806c140 2023-10-16T14:37:13.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 >> v1:172.21.15.29:6789/0 conn(0x7f0fe806c140 legacy=0x7f0fe815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 0x7f0fd4005d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40062a0 0x7f0fd40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd4006f10 0x7f0fd4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe815d890 con 0x7f0fd40062a0 2023-10-16T14:37:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd400dda0 con 0x7f0fd4006f10 2023-10-16T14:37:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd400df60 con 0x7f0fe8159e90 2023-10-16T14:37:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.357+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40062a0 0x7f0fd40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fee1e3640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd4006f10 0x7f0fd4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fed9e2640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 0x7f0fd4005d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 msgr2=0x7f0fd4005d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:13.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fe8159e90 0x7f0fd4005d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd4006f10 msgr2=0x7f0fd4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:13.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fd4006f10 0x7f0fd4007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.358+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3783402799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd4005730 con 0x7f0fd40062a0 2023-10-16T14:37:13.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.359+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40062a0 0x7f0fd40069d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009fc0 tx=0x7f0fd4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39a9b32e81f6fb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:13.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.359+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe402cc20 con 0x7f0fd40062a0 2023-10-16T14:37:13.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe400f040 con 0x7f0fd40062a0 2023-10-16T14:37:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fec9e0640 1 -- 172.21.15.29:0/3783402799 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4034440 con 0x7f0fd40062a0 2023-10-16T14:37:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fefc46640 1 -- 172.21.15.29:0/3783402799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40062a0 msgr2=0x7f0fd40069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3783402799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40062a0 0x7f0fd40069d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009fc0 tx=0x7f0fd4007a00 comp rx=0 tx=0).stop 2023-10-16T14:37:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fee1e3640 1 -- 172.21.15.29:0/3783402799 reap_dead start 2023-10-16T14:37:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fefc46640 1 -- 172.21.15.29:0/3783402799 shutdown_connections 2023-10-16T14:37:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fefc46640 1 -- 172.21.15.29:0/3783402799 >> 172.21.15.29:0/3783402799 conn(0x7f0fe80ce9b0 msgr2=0x7f0fe80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.360+0000 7f0fefc46640 1 -- 172.21.15.29:0/3783402799 shutdown_connections 2023-10-16T14:37:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.361+0000 7f0fefc46640 1 -- 172.21.15.29:0/3783402799 wait complete. 2023-10-16T14:37:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.362+0000 7f0fefc46640 1 Processor -- start 2023-10-16T14:37:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.362+0000 7f0fefc46640 1 -- start start 2023-10-16T14:37:13.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.363+0000 7f0fefc46640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd40062a0 0x7f0fe8151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.363+0000 7f0fefc46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 0x7f0fe81520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.363+0000 7f0fefc46640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fe815ceb0 0x7f0fe8155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.363+0000 7f0fefc46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe815b710 con 0x7f0fe814dfd0 2023-10-16T14:37:13.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.363+0000 7f0fefc46640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8154230 con 0x7f0fe815ceb0 2023-10-16T14:37:13.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.363+0000 7f0fefc46640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe815ade0 con 0x7f0fd40062a0 2023-10-16T14:37:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 0x7f0fe81520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 0x7f0fe81520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50548/0 (socket says 172.21.15.29:50548) 2023-10-16T14:37:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3575924422 learned_addr learned my addr 172.21.15.29:0/3575924422 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed9e2640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd40062a0 0x7f0fe8151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fee1e3640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fe815ceb0 0x7f0fe8155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd40062a0 msgr2=0x7f0fe8151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd40062a0 0x7f0fe8151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fe815ceb0 msgr2=0x7f0fe8155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fe815ceb0 0x7f0fe8155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed1e1640 1 -- 172.21.15.29:0/3575924422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe815b0c0 con 0x7f0fe814dfd0 2023-10-16T14:37:13.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.364+0000 7f0fed9e2640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd40062a0 0x7f0fe8151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:13.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.365+0000 7f0fed1e1640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 0x7f0fe81520b0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400e230 tx=0x7f0fe4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:13.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.365+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4034bd0 con 0x7f0fe814dfd0 2023-10-16T14:37:13.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.365+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4034d50 con 0x7f0fe814dfd0 2023-10-16T14:37:13.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.365+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4003bd0 con 0x7f0fe814dfd0 2023-10-16T14:37:13.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.365+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe815b3a0 con 0x7f0fe814dfd0 2023-10-16T14:37:13.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.365+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe81527f0 con 0x7f0fe814dfd0 2023-10-16T14:37:13.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.366+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe80adf30 con 0x7f0fe814dfd0 2023-10-16T14:37:13.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.368+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0fe4053480 con 0x7f0fe814dfd0 2023-10-16T14:37:13.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.369+0000 7f0fda7fc640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f0fcc06afc0 0x7f0fcc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:13.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.369+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f0fe40ac620 con 0x7f0fe814dfd0 2023-10-16T14:37:13.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.370+0000 7f0fed9e2640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f0fcc06afc0 0x7f0fcc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.371+0000 7f0fed9e2640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f0fcc06afc0 0x7f0fcc06d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0010e0 tx=0x7f0fdc009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:13.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.373+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0fe4077800 con 0x7f0fe814dfd0 2023-10-16T14:37:13.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:13.607+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0fe804e160 con 0x7f0fe814dfd0 2023-10-16T14:37:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.147+0000 7f0fda7fc640 1 -- 172.21.15.29:0/3575924422 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0fe407ae20 con 0x7f0fe814dfd0 2023-10-16T14:37:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 18 and data pool 19 2023-10-16T14:37:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.154+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f0fcc06afc0 msgr2=0x7f0fcc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.154+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f0fcc06afc0 0x7f0fcc06d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0010e0 tx=0x7f0fdc009040 comp rx=0 tx=0).stop 2023-10-16T14:37:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.154+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 msgr2=0x7f0fe81520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.154+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 0x7f0fe81520b0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400e230 tx=0x7f0fe4009750 comp rx=0 tx=0).stop 2023-10-16T14:37:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.155+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 shutdown_connections 2023-10-16T14:37:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.155+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f0fcc06afc0 0x7f0fcc06d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.155+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fe814dfd0 0x7f0fe81520b0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.155+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fe815ceb0 0x7f0fe8155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.155+0000 7f0fefc46640 1 --2- 172.21.15.29:0/3575924422 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fd40062a0 0x7f0fe8151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.155+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 >> 172.21.15.29:0/3575924422 conn(0x7f0fe80ce9b0 msgr2=0x7f0fe80c1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:14.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.158+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 shutdown_connections 2023-10-16T14:37:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.159+0000 7f0fefc46640 1 -- 172.21.15.29:0/3575924422 wait complete. 2023-10-16T14:37:14.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:37:14.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.378+0000 7feb112fc640 1 Processor -- start 2023-10-16T14:37:14.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.379+0000 7feb112fc640 1 -- start start 2023-10-16T14:37:14.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb112fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb112fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb0c0ce7f0 0x7feb0c0cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb112fc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0cc940 con 0x7feb0c15f220 2023-10-16T14:37:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb112fc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c0d1560 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb112fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0cbd30 con 0x7feb0c0cd510 2023-10-16T14:37:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb0a7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feb0c15f220 0x7feb0c0cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43784/0 (socket says 172.21.15.29:43784) 2023-10-16T14:37:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.380+0000 7feb0a7fc640 1 -- 172.21.15.29:0/3724579559 learned_addr learned my addr 172.21.15.29:0/3724579559 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:14.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb0b7fe640 1 --2- 172.21.15.29:0/3724579559 >> v2:172.21.15.29:3300/0 conn(0x7feb0c0ce7f0 0x7feb0c0cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb0affd640 1 --2- 172.21.15.29:0/3724579559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb0b7fe640 1 -- 172.21.15.29:0/3724579559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 msgr2=0x7feb0c0cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:14.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb0b7fe640 1 --2- 172.21.15.29:0/3724579559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb09ffb640 1 -- 172.21.15.29:0/3724579559 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197811284 0 0) 0x7feb0c0cc940 con 0x7feb0c15f220 2023-10-16T14:37:14.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb0b7fe640 1 -- 172.21.15.29:0/3724579559 >> v1:172.21.15.29:6789/0 conn(0x7feb0c15f220 legacy=0x7feb0c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.381+0000 7feb0b7fe640 1 -- 172.21.15.29:0/3724579559 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c0cc010 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.382+0000 7feb0b7fe640 1 --2- 172.21.15.29:0/3724579559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0cec80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7feb040097a0 tx=0x7feb0402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0e5ace5e9fa34d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:14.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.382+0000 7feb09ffb640 1 -- 172.21.15.29:0/3724579559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0402d040 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb09ffb640 1 -- 172.21.15.29:0/3724579559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb04031810 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb09ffb640 1 -- 172.21.15.29:0/3724579559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04031ba0 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 -- 172.21.15.29:0/3724579559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 msgr2=0x7feb0c0cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 --2- 172.21.15.29:0/3724579559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0cec80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7feb040097a0 tx=0x7feb0402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 -- 172.21.15.29:0/3724579559 shutdown_connections 2023-10-16T14:37:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 --2- 172.21.15.29:0/3724579559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0cec80 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 --2- 172.21.15.29:0/3724579559 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 -- 172.21.15.29:0/3724579559 >> 172.21.15.29:0/3724579559 conn(0x7feb0c099ea0 msgr2=0x7feb0c0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 -- 172.21.15.29:0/3724579559 shutdown_connections 2023-10-16T14:37:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.383+0000 7feb112fc640 1 -- 172.21.15.29:0/3724579559 wait complete. 2023-10-16T14:37:14.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.385+0000 7feb112fc640 1 Processor -- start 2023-10-16T14:37:14.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.385+0000 7feb112fc640 1 -- start start 2023-10-16T14:37:14.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb112fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb112fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb112fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c15f220 0x7feb0c06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb112fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0b3f90 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb112fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0b4270 con 0x7feb0c15f220 2023-10-16T14:37:14.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb112fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0b4550 con 0x7feb0c0cd510 2023-10-16T14:37:14.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb0a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb0a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35148/0 (socket says 172.21.15.29:35148) 2023-10-16T14:37:14.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.386+0000 7feb0a7fc640 1 -- 172.21.15.29:0/3375766611 learned_addr learned my addr 172.21.15.29:0/3375766611 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:14.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0affd640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0b7fe640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c15f220 0x7feb0c06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0a7fc640 1 -- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 msgr2=0x7feb0c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0a7fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0a7fc640 1 -- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c15f220 msgr2=0x7feb0c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0a7fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c15f220 0x7feb0c06c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0a7fc640 1 -- 172.21.15.29:0/3375766611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb04009400 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feb0a7fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0b5b00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7feb0000c7c0 tx=0x7feb0000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:14.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.387+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000a640 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.388+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0000a7c0 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.388+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c0b4830 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.388+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000ab70 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.388+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c0b5030 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.389+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c0b4d50 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.390+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb0000acd0 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.391+0000 7feaf37fe640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7feaf806af90 0x7feaf806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.392+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feb00088da0 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.392+0000 7feb0affd640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7feaf806af90 0x7feaf806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.394+0000 7feb0affd640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7feaf806af90 0x7feaf806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feafc009770 tx=0x7feafc008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:14.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.397+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb0c0b4d50 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.589+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb0c0b4d50 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.592+0000 7feaf37fe640 1 -- 172.21.15.29:0/3375766611 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16636 (secure 0 0 0) 0x7feb0c0b4d50 con 0x7feb0c0ce7f0 2023-10-16T14:37:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:14.595 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":154,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:37:13.609013+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:37:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7feaf806af90 msgr2=0x7feaf806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7feaf806af90 0x7feaf806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feafc009770 tx=0x7feafc008040 comp rx=0 tx=0).stop 2023-10-16T14:37:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 msgr2=0x7feb0c0b5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0b5b00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7feb0000c7c0 tx=0x7feb0000cc80 comp rx=0 tx=0).stop 2023-10-16T14:37:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 shutdown_connections 2023-10-16T14:37:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7feaf806af90 0x7feaf806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0c0ce7f0 0x7feb0c0b5b00 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb0c15f220 0x7feb0c06c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 --2- 172.21.15.29:0/3375766611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb0c0cd510 0x7feb0c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.594+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 >> 172.21.15.29:0/3375766611 conn(0x7feb0c099ea0 msgr2=0x7feb0c0c6660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.595+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 shutdown_connections 2023-10-16T14:37:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.595+0000 7feb112fc640 1 -- 172.21.15.29:0/3375766611 wait complete. 2023-10-16T14:37:14.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:37:14.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.838+0000 7f4dd8932640 1 Processor -- start 2023-10-16T14:37:14.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.838+0000 7f4dd8932640 1 -- start start 2023-10-16T14:37:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.840+0000 7f4dd8932640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dd415cd70 0x7f4dd40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.840+0000 7f4dd8932640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40cded0 0x7f4dd40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.840+0000 7f4dd8932640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd415d150 con 0x7f4dd40b8480 2023-10-16T14:37:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.840+0000 7f4dd8932640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40d1180 con 0x7f4dd415cd70 2023-10-16T14:37:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.840+0000 7f4dd8932640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40d1460 con 0x7f4dd40cded0 2023-10-16T14:37:14.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.840+0000 7f4dd2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dd415cd70 0x7f4dd40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4dd415cd70 0x7f4dd40d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35160/0 (socket says 172.21.15.29:35160) 2023-10-16T14:37:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd2d9d640 1 -- 172.21.15.29:0/1056353936 learned_addr learned my addr 172.21.15.29:0/1056353936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd359e640 1 --2- 172.21.15.29:0/1056353936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40cded0 0x7f4dd40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd2d9d640 1 -- 172.21.15.29:0/1056353936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40cded0 msgr2=0x7f4dd40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd2d9d640 1 --2- 172.21.15.29:0/1056353936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40cded0 0x7f4dd40b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd2d9d640 1 -- 172.21.15.29:0/1056353936 >> v1:172.21.15.170:6789/0 conn(0x7f4dd40b8480 legacy=0x7f4dd40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.841+0000 7f4dd2d9d640 1 -- 172.21.15.29:0/1056353936 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40d1740 con 0x7f4dd415cd70 2023-10-16T14:37:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.843+0000 7f4dd2d9d640 1 --2- 172.21.15.29:0/1056353936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd40d2220 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009870 tx=0x7f4dc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2accf7eb4275d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.844+0000 7f4dd259c640 1 -- 172.21.15.29:0/1056353936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4030840 con 0x7f4dd415cd70 2023-10-16T14:37:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.844+0000 7f4dd259c640 1 -- 172.21.15.29:0/1056353936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc40309a0 con 0x7f4dd415cd70 2023-10-16T14:37:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.844+0000 7f4dd259c640 1 -- 172.21.15.29:0/1056353936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4030d30 con 0x7f4dd415cd70 2023-10-16T14:37:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 -- 172.21.15.29:0/1056353936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 msgr2=0x7f4dd40d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 --2- 172.21.15.29:0/1056353936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd40d2220 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009870 tx=0x7f4dc402b980 comp rx=0 tx=0).stop 2023-10-16T14:37:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 -- 172.21.15.29:0/1056353936 shutdown_connections 2023-10-16T14:37:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 --2- 172.21.15.29:0/1056353936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40cded0 0x7f4dd40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 --2- 172.21.15.29:0/1056353936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd40d2220 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 -- 172.21.15.29:0/1056353936 >> 172.21.15.29:0/1056353936 conn(0x7f4dd40cf570 msgr2=0x7f4dd40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 -- 172.21.15.29:0/1056353936 shutdown_connections 2023-10-16T14:37:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.845+0000 7f4dd8932640 1 -- 172.21.15.29:0/1056353936 wait complete. 2023-10-16T14:37:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.847+0000 7f4dd8932640 1 Processor -- start 2023-10-16T14:37:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.847+0000 7f4dd8932640 1 -- start start 2023-10-16T14:37:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd8932640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40b8480 0x7f4dd40d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd8932640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dd40cded0 0x7f4dd41f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd8932640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd41f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd8932640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd41f3470 con 0x7f4dd415cd70 2023-10-16T14:37:14.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd8932640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd41f39f0 con 0x7f4dd40cded0 2023-10-16T14:37:14.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd8932640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd41f2de0 con 0x7f4dd40b8480 2023-10-16T14:37:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd41f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd41f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35166/0 (socket says 172.21.15.29:35166) 2023-10-16T14:37:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.848+0000 7f4dd359e640 1 -- 172.21.15.29:0/779157777 learned_addr learned my addr 172.21.15.29:0/779157777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd2d9d640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40b8480 0x7f4dd40d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dcbfff640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dd40cded0 0x7f4dd41f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd359e640 1 -- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40b8480 msgr2=0x7f4dd40d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd359e640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40b8480 0x7f4dd40d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd359e640 1 -- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dd40cded0 msgr2=0x7f4dd41f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd359e640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dd40cded0 0x7f4dd41f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd359e640 1 -- 172.21.15.29:0/779157777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40094d0 con 0x7f4dd415cd70 2023-10-16T14:37:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dcbfff640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dd40cded0 0x7f4dd41f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dd359e640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd41f27f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00efc0 tx=0x7f4dcc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00cd20 con 0x7f4dd415cd70 2023-10-16T14:37:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.849+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc00f040 con 0x7f4dd415cd70 2023-10-16T14:37:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.850+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd41f30c0 con 0x7f4dd415cd70 2023-10-16T14:37:14.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.850+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00cea0 con 0x7f4dd415cd70 2023-10-16T14:37:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.850+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd40d3230 con 0x7f4dd415cd70 2023-10-16T14:37:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.851+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd40d3450 con 0x7f4dd415cd70 2023-10-16T14:37:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.852+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4dcc0148d0 con 0x7f4dd415cd70 2023-10-16T14:37:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.853+0000 7f4dcb7fe640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4db406af90 0x7f4db406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.853+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4dcc08aa40 con 0x7f4dd415cd70 2023-10-16T14:37:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.854+0000 7f4dd2d9d640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4db406af90 0x7f4db406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.855+0000 7f4dd2d9d640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4db406af90 0x7f4db406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4036000 tx=0x7f4dc403b040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:14.858+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4dcc08ce60 con 0x7f4dd415cd70 2023-10-16T14:37:15.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.057+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dd404e9b0 con 0x7f4dd415cd70 2023-10-16T14:37:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.060+0000 7f4dcb7fe640 1 -- 172.21.15.29:0/779157777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16636 (secure 0 0 0) 0x7f4dcc018020 con 0x7f4dd415cd70 2023-10-16T14:37:15.063 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:15.064 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":154,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:37:13.609013+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:37:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4db406af90 msgr2=0x7f4db406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4db406af90 0x7f4db406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4036000 tx=0x7f4dc403b040 comp rx=0 tx=0).stop 2023-10-16T14:37:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 msgr2=0x7f4dd41f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd41f27f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00efc0 tx=0x7f4dcc00c490 comp rx=0 tx=0).stop 2023-10-16T14:37:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 shutdown_connections 2023-10-16T14:37:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f4db406af90 0x7f4db406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4dd415cd70 0x7f4dd41f27f0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dd40cded0 0x7f4dd41f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 --2- 172.21.15.29:0/779157777 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4dd40b8480 0x7f4dd40d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 >> 172.21.15.29:0/779157777 conn(0x7f4dd40cf570 msgr2=0x7f4dd40b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 shutdown_connections 2023-10-16T14:37:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.063+0000 7f4dd8932640 1 -- 172.21.15.29:0/779157777 wait complete. 2023-10-16T14:37:15.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:37:15.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.300+0000 7fbacbfff640 1 Processor -- start 2023-10-16T14:37:15.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.300+0000 7fbacbfff640 1 -- start start 2023-10-16T14:37:15.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.301+0000 7fbacbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 0x7fbacc0af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbacbfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0b3730 con 0x7fbacc0cc9f0 2023-10-16T14:37:15.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbacbfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0b3980 con 0x7fbacc0afd60 2023-10-16T14:37:15.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbacbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0b3c60 con 0x7fbacc15aa60 2023-10-16T14:37:15.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbacb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbacc0cc9f0 0x7fbacc0b4740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43790/0 (socket says 172.21.15.29:43790) 2023-10-16T14:37:15.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbacb7fe640 1 -- 172.21.15.29:0/692181365 learned_addr learned my addr 172.21.15.29:0/692181365 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:15.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbacaffd640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 0x7fbacc0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.302+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582931680 0 0) 0x7fbacc0b3730 con 0x7fbacc0cc9f0 2023-10-16T14:37:15.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8003520 con 0x7fbacc0cc9f0 2023-10-16T14:37:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436938168 0 0) 0x7fbab8003520 con 0x7fbacc0cc9f0 2023-10-16T14:37:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 msgr2=0x7fbacc0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 0x7fbacc0af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 >> v1:172.21.15.170:6789/0 conn(0x7fbacc0afd60 legacy=0x7fbacc0cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0b3f40 con 0x7fbacc0cc9f0 2023-10-16T14:37:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbac40019f0 con 0x7fbacc0cc9f0 2023-10-16T14:37:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 >> v1:172.21.15.29:6789/0 conn(0x7fbacc0cc9f0 legacy=0x7fbacc0b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 0x7fbab80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003f20 0x7fbab80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8004d00 0x7fbab8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0b3c60 con 0x7fbab8004d00 2023-10-16T14:37:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab800bb10 con 0x7fbab8003f20 2023-10-16T14:37:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab800bcd0 con 0x7fbacc15aa60 2023-10-16T14:37:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.303+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8004d00 0x7fbab8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbacaffd640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 0x7fbab80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbac3fff640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003f20 0x7fbab80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbacb7fe640 1 -- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 msgr2=0x7fbab80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc15aa60 0x7fbab80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbacb7fe640 1 -- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003f20 msgr2=0x7fbab80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003f20 0x7fbab80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.304+0000 7fbacb7fe640 1 -- 172.21.15.29:0/692181365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8003520 con 0x7fbab8004d00 2023-10-16T14:37:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.305+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8004d00 0x7fbab8005180 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fbac400a830 tx=0x7fbab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cee260d5a51a07 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.305+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40072d0 con 0x7fbab8004d00 2023-10-16T14:37:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.305+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac402f070 con 0x7fbab8004d00 2023-10-16T14:37:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.305+0000 7fbaca7fc640 1 -- 172.21.15.29:0/692181365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4007690 con 0x7fbab8004d00 2023-10-16T14:37:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacbfff640 1 -- 172.21.15.29:0/692181365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8004d00 msgr2=0x7fbab8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacbfff640 1 --2- 172.21.15.29:0/692181365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab8004d00 0x7fbab8005180 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fbac400a830 tx=0x7fbab80057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacb7fe640 1 -- 172.21.15.29:0/692181365 reap_dead start 2023-10-16T14:37:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacbfff640 1 -- 172.21.15.29:0/692181365 shutdown_connections 2023-10-16T14:37:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacbfff640 1 -- 172.21.15.29:0/692181365 >> 172.21.15.29:0/692181365 conn(0x7fbacc065ed0 msgr2=0x7fbacc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacbfff640 1 -- 172.21.15.29:0/692181365 shutdown_connections 2023-10-16T14:37:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.306+0000 7fbacbfff640 1 -- 172.21.15.29:0/692181365 wait complete. 2023-10-16T14:37:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.308+0000 7fbacbfff640 1 Processor -- start 2023-10-16T14:37:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.308+0000 7fbacbfff640 1 -- start start 2023-10-16T14:37:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.309+0000 7fbacbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8004d00 0x7fbacc15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.309+0000 7fbacbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc14fb50 0x7fbacc0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.309+0000 7fbacbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 0x7fbacc15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.309+0000 7fbacbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc15dfe0 con 0x7fbacc15af20 2023-10-16T14:37:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.309+0000 7fbacbfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc15b8e0 con 0x7fbab8004d00 2023-10-16T14:37:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.309+0000 7fbacbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc15d740 con 0x7fbacc14fb50 2023-10-16T14:37:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 0x7fbacc15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 0x7fbacc15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35186/0 (socket says 172.21.15.29:35186) 2023-10-16T14:37:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 -- 172.21.15.29:0/3211247640 learned_addr learned my addr 172.21.15.29:0/3211247640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacaffd640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8004d00 0x7fbacc15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbac3fff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc14fb50 0x7fbacc0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 -- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc14fb50 msgr2=0x7fbacc0c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc14fb50 0x7fbacc0c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 -- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8004d00 msgr2=0x7fbacc15d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8004d00 0x7fbacc15d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 -- 172.21.15.29:0/3211247640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc15da20 con 0x7fbacc15af20 2023-10-16T14:37:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.310+0000 7fbacb7fe640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 0x7fbacc15b3a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fbac400a950 tx=0x7fbac4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.311+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4044070 con 0x7fbacc15af20 2023-10-16T14:37:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.311+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac40072e0 con 0x7fbacc15af20 2023-10-16T14:37:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.311+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4007650 con 0x7fbacc15af20 2023-10-16T14:37:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.311+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc15dd00 con 0x7fbacc15af20 2023-10-16T14:37:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.311+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc150270 con 0x7fbacc15af20 2023-10-16T14:37:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.312+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc0b0da0 con 0x7fbacc15af20 2023-10-16T14:37:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.313+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbac4031030 con 0x7fbacc15af20 2023-10-16T14:37:15.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.314+0000 7fbac37fe640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbaac06afc0 0x7fbaac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:15.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.314+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbac40ab5f0 con 0x7fbacc15af20 2023-10-16T14:37:15.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.315+0000 7fbacaffd640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbaac06afc0 0x7fbaac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:15.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.316+0000 7fbacaffd640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbaac06afc0 0x7fbaac06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009a10 tx=0x7fbabc008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:15.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.319+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbacc0b0da0 con 0x7fbacc15af20 2023-10-16T14:37:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:15.555+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbacc0b1180 con 0x7fbacc15af20 2023-10-16T14:37:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.310+0000 7fbac37fe640 1 -- 172.21.15.29:0/3211247640 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v74) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbacc0b1180 con 0x7fbacc15af20 2023-10-16T14:37:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbaac06afc0 msgr2=0x7fbaac06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbaac06afc0 0x7fbaac06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009a10 tx=0x7fbabc008040 comp rx=0 tx=0).stop 2023-10-16T14:37:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 msgr2=0x7fbacc15b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 0x7fbacc15b3a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fbac400a950 tx=0x7fbac4001d80 comp rx=0 tx=0).stop 2023-10-16T14:37:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 shutdown_connections 2023-10-16T14:37:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fbaac06afc0 0x7fbaac06d480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbacc15af20 0x7fbacc15b3a0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbacc14fb50 0x7fbacc0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 --2- 172.21.15.29:0/3211247640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8004d00 0x7fbacc15d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 >> 172.21.15.29:0/3211247640 conn(0x7fbacc065ed0 msgr2=0x7fbacc0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 shutdown_connections 2023-10-16T14:37:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.314+0000 7fbacbfff640 1 -- 172.21.15.29:0/3211247640 wait complete. 2023-10-16T14:37:16.339 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:37:16.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:37:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.573+0000 7f3dad01a640 1 Processor -- start 2023-10-16T14:37:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.573+0000 7f3dad01a640 1 -- start start 2023-10-16T14:37:16.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.574+0000 7f3dad01a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3da80cded0 0x7f3da80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:16.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.574+0000 7f3dad01a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da815d150 con 0x7f3da80b8480 2023-10-16T14:37:16.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dad01a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da80d1180 con 0x7f3da815cd70 2023-10-16T14:37:16.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dad01a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da80d1460 con 0x7f3da80cded0 2023-10-16T14:37:16.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dac819640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3da80cded0 0x7f3da80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:16.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dac819640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3da80cded0 0x7f3da80b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35202/0 (socket says 172.21.15.29:35202) 2023-10-16T14:37:16.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dac819640 1 -- 172.21.15.29:0/2815499428 learned_addr learned my addr 172.21.15.29:0/2815499428 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:16.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3da77fe640 1 --1- 172.21.15.29:0/2815499428 >> v1:172.21.15.29:6789/0 conn(0x7f3da80b8480 0x7f3da80cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43798/0 (socket says 172.21.15.29:43798) 2023-10-16T14:37:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dac819640 1 -- 172.21.15.29:0/2815499428 >> v1:172.21.15.170:6789/0 conn(0x7f3da815cd70 legacy=0x7f3da80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dac819640 1 -- 172.21.15.29:0/2815499428 >> v1:172.21.15.29:6789/0 conn(0x7f3da80b8480 legacy=0x7f3da80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3dac819640 1 -- 172.21.15.29:0/2815499428 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da80d1740 con 0x7f3da80cded0 2023-10-16T14:37:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.575+0000 7f3da6ffd640 1 -- 172.21.15.29:0/2815499428 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712083075 0 0) 0x7f3da815d150 con 0x7f3da80b8480 2023-10-16T14:37:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.577+0000 7f3dac819640 1 --2- 172.21.15.29:0/2815499428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da80cded0 0x7f3da80b5c90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009870 tx=0x7f3da002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf871ea421ebc50 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:16.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.577+0000 7f3da6ffd640 1 -- 172.21.15.29:0/2815499428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da002f840 con 0x7f3da80cded0 2023-10-16T14:37:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.577+0000 7f3da6ffd640 1 -- 172.21.15.29:0/2815499428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da002f9a0 con 0x7f3da80cded0 2023-10-16T14:37:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.577+0000 7f3da6ffd640 1 -- 172.21.15.29:0/2815499428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da002fd30 con 0x7f3da80cded0 2023-10-16T14:37:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 -- 172.21.15.29:0/2815499428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da80cded0 msgr2=0x7f3da80b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 --2- 172.21.15.29:0/2815499428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da80cded0 0x7f3da80b5c90 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009870 tx=0x7f3da002ba20 comp rx=0 tx=0).stop 2023-10-16T14:37:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 -- 172.21.15.29:0/2815499428 shutdown_connections 2023-10-16T14:37:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 --2- 172.21.15.29:0/2815499428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da80cded0 0x7f3da80b5c90 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 -- 172.21.15.29:0/2815499428 >> 172.21.15.29:0/2815499428 conn(0x7f3da80cf570 msgr2=0x7f3da80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 -- 172.21.15.29:0/2815499428 shutdown_connections 2023-10-16T14:37:16.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.578+0000 7f3dad01a640 1 -- 172.21.15.29:0/2815499428 wait complete. 2023-10-16T14:37:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.580+0000 7f3dad01a640 1 Processor -- start 2023-10-16T14:37:16.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.581+0000 7f3dad01a640 1 -- start start 2023-10-16T14:37:16.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dad01a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3da80b8480 0x7f3da8158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:16.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dad01a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3da80cded0 0x7f3da81f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:16.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dad01a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 0x7f3da81f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:16.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dad01a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da8158990 con 0x7f3da815cd70 2023-10-16T14:37:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dad01a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da8158b80 con 0x7f3da80b8480 2023-10-16T14:37:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dad01a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da8158e60 con 0x7f3da80cded0 2023-10-16T14:37:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dac819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 0x7f3da81f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dac819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 0x7f3da81f5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35204/0 (socket says 172.21.15.29:35204) 2023-10-16T14:37:16.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.582+0000 7f3dac819640 1 -- 172.21.15.29:0/4032584605 learned_addr learned my addr 172.21.15.29:0/4032584605 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3da7fff640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3da80b8480 0x7f3da8158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3da77fe640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3da80cded0 0x7f3da81f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3dac819640 1 -- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3da80cded0 msgr2=0x7f3da81f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3dac819640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3da80cded0 0x7f3da81f5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3dac819640 1 -- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3da80b8480 msgr2=0x7f3da8158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3dac819640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3da80b8480 0x7f3da8158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3dac819640 1 -- 172.21.15.29:0/4032584605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da00094d0 con 0x7f3da815cd70 2023-10-16T14:37:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3da77fe640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3da80cded0 0x7f3da81f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:16.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.583+0000 7f3dac819640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 0x7f3da81f5c50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3da002ba60 tx=0x7f3da0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:16.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.584+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0037dd0 con 0x7f3da815cd70 2023-10-16T14:37:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.584+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da0041400 con 0x7f3da815cd70 2023-10-16T14:37:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.584+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da00417b0 con 0x7f3da815cd70 2023-10-16T14:37:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.584+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da8159140 con 0x7f3da815cd70 2023-10-16T14:37:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.584+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da8159700 con 0x7f3da815cd70 2023-10-16T14:37:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.585+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da8159890 con 0x7f3da815cd70 2023-10-16T14:37:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.586+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3da0041910 con 0x7f3da815cd70 2023-10-16T14:37:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.587+0000 7f3da4ff9640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3d9406af90 0x7f3d9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.588+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3da003c070 con 0x7f3da815cd70 2023-10-16T14:37:16.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.588+0000 7f3da7fff640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3d9406af90 0x7f3d9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:16.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.589+0000 7f3da7fff640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3d9406af90 0x7f3d9406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000f60 tx=0x7f3d98009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:16.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.592+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3da0077670 con 0x7f3da815cd70 2023-10-16T14:37:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:16.824+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3da804e9b0 con 0x7f3da815cd70 2023-10-16T14:37:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.320+0000 7f3da4ff9640 1 -- 172.21.15.29:0/4032584605 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3da007ac90 con 0x7f3da815cd70 2023-10-16T14:37:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.324+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3d9406af90 msgr2=0x7f3d9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.324+0000 7f3dad01a640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3d9406af90 0x7f3d9406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000f60 tx=0x7f3d98009040 comp rx=0 tx=0).stop 2023-10-16T14:37:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 msgr2=0x7f3da81f5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 0x7f3da81f5c50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3da002ba60 tx=0x7f3da0004730 comp rx=0 tx=0).stop 2023-10-16T14:37:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 shutdown_connections 2023-10-16T14:37:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f3d9406af90 0x7f3d9406d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3da815cd70 0x7f3da81f5c50 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3da80cded0 0x7f3da81f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 --2- 172.21.15.29:0/4032584605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3da80b8480 0x7f3da8158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 >> 172.21.15.29:0/4032584605 conn(0x7f3da80cf570 msgr2=0x7f3da80b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 shutdown_connections 2023-10-16T14:37:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.325+0000 7f3dad01a640 1 -- 172.21.15.29:0/4032584605 wait complete. 2023-10-16T14:37:17.347 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:37:17.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:37:17.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.575+0000 7f15948f3640 1 Processor -- start 2023-10-16T14:37:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.575+0000 7f15948f3640 1 -- start start 2023-10-16T14:37:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.576+0000 7f15948f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 0x7f159006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.576+0000 7f15948f3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15900ad970 0x7f159015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:17.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.576+0000 7f15948f3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159006c960 con 0x7f159015ebc0 2023-10-16T14:37:17.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f15948f3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159015b9d0 con 0x7f15900ad970 2023-10-16T14:37:17.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f15948f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159015bcb0 con 0x7f159006c040 2023-10-16T14:37:17.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 0x7f159006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:17.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f1587fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f15900ad970 0x7f159015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:17.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f158f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 0x7f159006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:37612/0 (socket says 172.21.15.29:37612) 2023-10-16T14:37:17.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f158ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f159015ebc0 0x7f15900cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:55404/0 (socket says 172.21.15.29:55404) 2023-10-16T14:37:17.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.577+0000 7f158f7fe640 1 -- 172.21.15.29:0/2649035861 learned_addr learned my addr 172.21.15.29:0/2649035861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:17.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.578+0000 7f158f7fe640 1 -- 172.21.15.29:0/2649035861 >> v2:172.21.15.170:3300/0 conn(0x7f15900ad970 msgr2=0x7f159015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:17.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.578+0000 7f158effd640 1 -- 172.21.15.29:0/2649035861 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416104646 0 0) 0x7f159006c960 con 0x7f159015ebc0 2023-10-16T14:37:17.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.578+0000 7f158f7fe640 1 --2- 172.21.15.29:0/2649035861 >> v2:172.21.15.170:3300/0 conn(0x7f15900ad970 0x7f159015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.578+0000 7f158f7fe640 1 -- 172.21.15.29:0/2649035861 >> v1:172.21.15.170:6789/0 conn(0x7f159015ebc0 legacy=0x7f15900cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.578+0000 7f158f7fe640 1 -- 172.21.15.29:0/2649035861 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159015bf90 con 0x7f159006c040 2023-10-16T14:37:17.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.578+0000 7f1587fff640 1 --2- 172.21.15.29:0/2649035861 >> v2:172.21.15.170:3300/0 conn(0x7f15900ad970 0x7f159015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:37:17.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.580+0000 7f158f7fe640 1 --2- 172.21.15.29:0/2649035861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 0x7f159006c420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f15800098a0 tx=0x7f158002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f02ab7e748c9f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:17.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.580+0000 7f158effd640 1 -- 172.21.15.29:0/2649035861 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580030840 con 0x7f159006c040 2023-10-16T14:37:17.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.580+0000 7f158effd640 1 -- 172.21.15.29:0/2649035861 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15800309a0 con 0x7f159006c040 2023-10-16T14:37:17.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.580+0000 7f158effd640 1 -- 172.21.15.29:0/2649035861 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580030d30 con 0x7f159006c040 2023-10-16T14:37:17.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 -- 172.21.15.29:0/2649035861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 msgr2=0x7f159006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 --2- 172.21.15.29:0/2649035861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 0x7f159006c420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f15800098a0 tx=0x7f158002b980 comp rx=0 tx=0).stop 2023-10-16T14:37:17.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 -- 172.21.15.29:0/2649035861 shutdown_connections 2023-10-16T14:37:17.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 --2- 172.21.15.29:0/2649035861 >> v2:172.21.15.170:3300/0 conn(0x7f15900ad970 0x7f159015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 --2- 172.21.15.29:0/2649035861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159006c040 0x7f159006c420 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 -- 172.21.15.29:0/2649035861 >> 172.21.15.29:0/2649035861 conn(0x7f15900ce6a0 msgr2=0x7f15900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 -- 172.21.15.29:0/2649035861 shutdown_connections 2023-10-16T14:37:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.581+0000 7f15948f3640 1 -- 172.21.15.29:0/2649035861 wait complete. 2023-10-16T14:37:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.583+0000 7f15948f3640 1 Processor -- start 2023-10-16T14:37:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.583+0000 7f15948f3640 1 -- start start 2023-10-16T14:37:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f15948f3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f159006c040 0x7f15900c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f15948f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 0x7f15900ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f15948f3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159015ebc0 0x7f15900aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f15948f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1590153660 con 0x7f15900ad970 2023-10-16T14:37:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f15948f3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900af490 con 0x7f159006c040 2023-10-16T14:37:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f15948f3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c6880 con 0x7f159015ebc0 2023-10-16T14:37:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f1587fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 0x7f15900ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f1587fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 0x7f15900ca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35208/0 (socket says 172.21.15.29:35208) 2023-10-16T14:37:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.584+0000 7f1587fff640 1 -- 172.21.15.29:0/3701124347 learned_addr learned my addr 172.21.15.29:0/3701124347 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f1587fff640 1 -- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159015ebc0 msgr2=0x7f15900aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f1587fff640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159015ebc0 0x7f15900aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f1587fff640 1 -- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f159006c040 msgr2=0x7f15900c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f1587fff640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f159006c040 0x7f15900c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f1587fff640 1 -- 172.21.15.29:0/3701124347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578009400 con 0x7f15900ad970 2023-10-16T14:37:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f1587fff640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 0x7f15900ca430 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f157800e960 tx=0x7f157800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157800ca00 con 0x7f15900ad970 2023-10-16T14:37:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157800cb60 con 0x7f15900ad970 2023-10-16T14:37:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578013470 con 0x7f15900ad970 2023-10-16T14:37:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15800094d0 con 0x7f15900ad970 2023-10-16T14:37:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.585+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15901571d0 con 0x7f15900ad970 2023-10-16T14:37:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.587+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159006b330 con 0x7f15900ad970 2023-10-16T14:37:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.587+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f15780135d0 con 0x7f15900ad970 2023-10-16T14:37:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.589+0000 7f158cff9640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f157006af90 0x7f157006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.589+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1578089a50 con 0x7f15900ad970 2023-10-16T14:37:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.589+0000 7f158f7fe640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f157006af90 0x7f157006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.591+0000 7f158f7fe640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f157006af90 0x7f157006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1580036000 tx=0x7f158003b040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:17.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.593+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1578054930 con 0x7f15900ad970 2023-10-16T14:37:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.815+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f159004e9b0 con 0x7f15900ad970 2023-10-16T14:37:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.816+0000 7f158cff9640 1 -- 172.21.15.29:0/3701124347 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f1578057f50 con 0x7f15900ad970 2023-10-16T14:37:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:17.819 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:37:13.609698+0000","modified":"2023-10-16T14:37:16.825159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8838},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8838":{"gid":8838,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3542022450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3542022450},{"type":"v1","addr":"172.21.15.29:6833","nonce":3542022450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T14:37:17.820 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 75 2023-10-16T14:37:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f157006af90 msgr2=0x7f157006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f157006af90 0x7f157006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1580036000 tx=0x7f158003b040 comp rx=0 tx=0).stop 2023-10-16T14:37:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 msgr2=0x7f15900ca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 0x7f15900ca430 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f157800e960 tx=0x7f157800ee20 comp rx=0 tx=0).stop 2023-10-16T14:37:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 shutdown_connections 2023-10-16T14:37:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f157006af90 0x7f157006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15900ad970 0x7f15900ca430 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f159015ebc0 0x7f15900aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.818+0000 7f15948f3640 1 --2- 172.21.15.29:0/3701124347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f159006c040 0x7f15900c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:17.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.819+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 >> 172.21.15.29:0/3701124347 conn(0x7f15900ce6a0 msgr2=0x7f15900cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:17.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.819+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 shutdown_connections 2023-10-16T14:37:17.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:17.819+0000 7f15948f3640 1 -- 172.21.15.29:0/3701124347 wait complete. 2023-10-16T14:37:17.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:37:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.067+0000 7efef5df4640 1 Processor -- start 2023-10-16T14:37:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.068+0000 7efef5df4640 1 -- start start 2023-10-16T14:37:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.069+0000 7efef5df4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.069+0000 7efef5df4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.069+0000 7efef5df4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef006c960 con 0x7efef015ebc0 2023-10-16T14:37:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.069+0000 7efef5df4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef015b9d0 con 0x7efef00ad970 2023-10-16T14:37:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.069+0000 7efef5df4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef015bcb0 con 0x7efef006c040 2023-10-16T14:37:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.070+0000 7efef55f3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efef015ebc0 0x7efef00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43814/0 (socket says 172.21.15.29:43814) 2023-10-16T14:37:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.070+0000 7efef55f3640 1 -- 172.21.15.29:0/4027226980 learned_addr learned my addr 172.21.15.29:0/4027226980 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.070+0000 7efef4df2640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.070+0000 7efee7fff640 1 --2- 172.21.15.29:0/4027226980 >> v2:172.21.15.170:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.070+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274714176 0 0) 0x7efef006c960 con 0x7efef015ebc0 2023-10-16T14:37:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.070+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed8003520 con 0x7efef015ebc0 2023-10-16T14:37:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 132798092 0 0) 0x7efed8003520 con 0x7efef015ebc0 2023-10-16T14:37:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 msgr2=0x7efef006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 >> v2:172.21.15.170:3300/0 conn(0x7efef00ad970 msgr2=0x7efef015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 --2- 172.21.15.29:0/4027226980 >> v2:172.21.15.170:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef015bf90 con 0x7efef015ebc0 2023-10-16T14:37:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7efee8001670 con 0x7efef015ebc0 2023-10-16T14:37:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 >> v1:172.21.15.29:6789/0 conn(0x7efef015ebc0 legacy=0x7efef00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efed8003970 0x7efed8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed80047c0 0x7efed8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 0x7efed8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef015b9d0 con 0x7efed8003970 2023-10-16T14:37:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef015bcb0 con 0x7efed80047c0 2023-10-16T14:37:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed800b980 con 0x7efef006c040 2023-10-16T14:37:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.071+0000 7efef4df2640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efed8003970 0x7efed8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.072+0000 7efef4df2640 1 -- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 msgr2=0x7efed8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.072+0000 7efef4df2640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efef006c040 0x7efed8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.072+0000 7efef4df2640 1 -- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed80047c0 msgr2=0x7efed8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.072+0000 7efef4df2640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efed80047c0 0x7efed8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.072+0000 7efef4df2640 1 -- 172.21.15.29:0/4027226980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed8003520 con 0x7efed8003970 2023-10-16T14:37:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.073+0000 7efef4df2640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efed8003970 0x7efed8003d90 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7efeec009fc0 tx=0x7efed80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de11f453614af6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.073+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec02cc40 con 0x7efed8003970 2023-10-16T14:37:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.073+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec00f040 con 0x7efed8003970 2023-10-16T14:37:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.073+0000 7efee77fe640 1 -- 172.21.15.29:0/4027226980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec034440 con 0x7efed8003970 2023-10-16T14:37:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef5df4640 1 -- 172.21.15.29:0/4027226980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efed8003970 msgr2=0x7efed8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef5df4640 1 --2- 172.21.15.29:0/4027226980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efed8003970 0x7efed8003d90 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7efeec009fc0 tx=0x7efed80057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef55f3640 1 -- 172.21.15.29:0/4027226980 reap_dead start 2023-10-16T14:37:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef5df4640 1 -- 172.21.15.29:0/4027226980 shutdown_connections 2023-10-16T14:37:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef5df4640 1 -- 172.21.15.29:0/4027226980 >> 172.21.15.29:0/4027226980 conn(0x7efef00ce6a0 msgr2=0x7efef00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef5df4640 1 -- 172.21.15.29:0/4027226980 shutdown_connections 2023-10-16T14:37:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.074+0000 7efef5df4640 1 -- 172.21.15.29:0/4027226980 wait complete. 2023-10-16T14:37:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.076+0000 7efef5df4640 1 Processor -- start 2023-10-16T14:37:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.076+0000 7efef5df4640 1 -- start start 2023-10-16T14:37:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef5df4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed8003970 0x7efef00c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef5df4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efef0158bd0 0x7efef01ef720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef5df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 0x7efef01efc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef5df4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef01f01a0 con 0x7efef0159030 2023-10-16T14:37:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef5df4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00c66c0 con 0x7efef0158bd0 2023-10-16T14:37:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef5df4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef014c650 con 0x7efed8003970 2023-10-16T14:37:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef55f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 0x7efef01efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef55f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 0x7efef01efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35228/0 (socket says 172.21.15.29:35228) 2023-10-16T14:37:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.077+0000 7efef55f3640 1 -- 172.21.15.29:0/2078874895 learned_addr learned my addr 172.21.15.29:0/2078874895 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef55f3640 1 -- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed8003970 msgr2=0x7efef00c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef4df2640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed8003970 0x7efef00c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efee7fff640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efef0158bd0 0x7efef01ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef55f3640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed8003970 0x7efef00c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef55f3640 1 -- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efef0158bd0 msgr2=0x7efef01ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef55f3640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efef0158bd0 0x7efef01ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef55f3640 1 -- 172.21.15.29:0/2078874895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0148790 con 0x7efef0159030 2023-10-16T14:37:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef55f3640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 0x7efef01efc60 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7efef006d2e0 tx=0x7efee800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8004bf0 con 0x7efef0159030 2023-10-16T14:37:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee8004d70 con 0x7efef0159030 2023-10-16T14:37:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8007270 con 0x7efef0159030 2023-10-16T14:37:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.078+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef0148a70 con 0x7efef0159030 2023-10-16T14:37:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.079+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0149030 con 0x7efef0159030 2023-10-16T14:37:18.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.080+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef006b330 con 0x7efef0159030 2023-10-16T14:37:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.080+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efee80073d0 con 0x7efef0159030 2023-10-16T14:37:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.081+0000 7efee57fa640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7efedc06afc0 0x7efedc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.082+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efee8088880 con 0x7efef0159030 2023-10-16T14:37:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.082+0000 7efef4df2640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7efedc06afc0 0x7efedc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.083+0000 7efef4df2640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7efedc06afc0 0x7efedc06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efeec036000 tx=0x7efeec02e000 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.086+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efee8053800 con 0x7efef0159030 2023-10-16T14:37:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.279+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efef004e9b0 con 0x7efef0159030 2023-10-16T14:37:18.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.281+0000 7efee57fa640 1 -- 172.21.15.29:0/2078874895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16623 (secure 0 0 0) 0x7efee8056e20 con 0x7efef0159030 2023-10-16T14:37:18.285 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:18.286 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":155,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T14:37:15.144674+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:37:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.284+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7efedc06afc0 msgr2=0x7efedc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.284+0000 7efef5df4640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7efedc06afc0 0x7efedc06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efeec036000 tx=0x7efeec02e000 comp rx=0 tx=0).stop 2023-10-16T14:37:18.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.284+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 msgr2=0x7efef01efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.284+0000 7efef5df4640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 0x7efef01efc60 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7efef006d2e0 tx=0x7efee800c8e0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 shutdown_connections 2023-10-16T14:37:18.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7efedc06afc0 0x7efedc06d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef0159030 0x7efef01efc60 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efef0158bd0 0x7efef01ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 --2- 172.21.15.29:0/2078874895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efed8003970 0x7efef00c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 >> 172.21.15.29:0/2078874895 conn(0x7efef00ce6a0 msgr2=0x7efef00c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:18.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 shutdown_connections 2023-10-16T14:37:18.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.285+0000 7efef5df4640 1 -- 172.21.15.29:0/2078874895 wait complete. 2023-10-16T14:37:18.308 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:37:18.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:37:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.529+0000 7fce8f554640 1 Processor -- start 2023-10-16T14:37:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.529+0000 7fce8f554640 1 -- start start 2023-10-16T14:37:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8f554640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce8815cd70 0x7fce880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8f554640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce880b8480 0x7fce880cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8f554640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8815d150 con 0x7fce880d0be0 2023-10-16T14:37:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8f554640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880d1180 con 0x7fce880b8480 2023-10-16T14:37:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8f554640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce880d1460 con 0x7fce8815cd70 2023-10-16T14:37:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8dd51640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce880b8480 0x7fce880cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8dd51640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce880b8480 0x7fce880cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35236/0 (socket says 172.21.15.29:35236) 2023-10-16T14:37:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.531+0000 7fce8dd51640 1 -- 172.21.15.29:0/171667326 learned_addr learned my addr 172.21.15.29:0/171667326 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.532+0000 7fce8e552640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce8815cd70 0x7fce880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.532+0000 7fce8dd51640 1 -- 172.21.15.29:0/171667326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce8815cd70 msgr2=0x7fce880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.532+0000 7fce8dd51640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce8815cd70 0x7fce880d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.532+0000 7fce8dd51640 1 -- 172.21.15.29:0/171667326 >> v1:172.21.15.29:6789/0 conn(0x7fce880d0be0 legacy=0x7fce880b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.532+0000 7fce8dd51640 1 -- 172.21.15.29:0/171667326 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880d1740 con 0x7fce880b8480 2023-10-16T14:37:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.532+0000 7fce8e552640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce8815cd70 0x7fce880d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:18.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.533+0000 7fce8dd51640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce880b8480 0x7fce880cd7c0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fce78009870 tx=0x7fce7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313f9fcccb2ace13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:18.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.533+0000 7fce8d550640 1 -- 172.21.15.29:0/171667326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78030840 con 0x7fce880b8480 2023-10-16T14:37:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.533+0000 7fce8d550640 1 -- 172.21.15.29:0/171667326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce780309a0 con 0x7fce880b8480 2023-10-16T14:37:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.533+0000 7fce8d550640 1 -- 172.21.15.29:0/171667326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78030d30 con 0x7fce880b8480 2023-10-16T14:37:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 -- 172.21.15.29:0/171667326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce880b8480 msgr2=0x7fce880cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce880b8480 0x7fce880cd7c0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fce78009870 tx=0x7fce7802b980 comp rx=0 tx=0).stop 2023-10-16T14:37:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 -- 172.21.15.29:0/171667326 shutdown_connections 2023-10-16T14:37:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce880b8480 0x7fce880cd7c0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 --2- 172.21.15.29:0/171667326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce8815cd70 0x7fce880d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 -- 172.21.15.29:0/171667326 >> 172.21.15.29:0/171667326 conn(0x7fce880cf570 msgr2=0x7fce880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 -- 172.21.15.29:0/171667326 shutdown_connections 2023-10-16T14:37:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.534+0000 7fce8f554640 1 -- 172.21.15.29:0/171667326 wait complete. 2023-10-16T14:37:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.536+0000 7fce8f554640 1 Processor -- start 2023-10-16T14:37:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.536+0000 7fce8f554640 1 -- start start 2023-10-16T14:37:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8f554640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce880b8480 0x7fce881f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8f554640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce880d0be0 0x7fce8814d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8f554640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 0x7fce880b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8f554640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8814d770 con 0x7fce8815cd70 2023-10-16T14:37:18.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8f554640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8815f080 con 0x7fce880b8480 2023-10-16T14:37:18.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8f554640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8815ea40 con 0x7fce880d0be0 2023-10-16T14:37:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8ed53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 0x7fce880b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8ed53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 0x7fce880b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35246/0 (socket says 172.21.15.29:35246) 2023-10-16T14:37:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.537+0000 7fce8ed53640 1 -- 172.21.15.29:0/4212116895 learned_addr learned my addr 172.21.15.29:0/4212116895 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8ed53640 1 -- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce880d0be0 msgr2=0x7fce8814d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8ed53640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce880d0be0 0x7fce8814d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8ed53640 1 -- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce880b8480 msgr2=0x7fce881f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8ed53640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce880b8480 0x7fce881f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8ed53640 1 -- 172.21.15.29:0/4212116895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780094d0 con 0x7fce8815cd70 2023-10-16T14:37:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8ed53640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 0x7fce880b4400 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fce8000c730 tx=0x7fce8000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80003b70 con 0x7fce8815cd70 2023-10-16T14:37:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce80003cf0 con 0x7fce8815cd70 2023-10-16T14:37:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce880c78e0 con 0x7fce8815cd70 2023-10-16T14:37:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80005310 con 0x7fce8815cd70 2023-10-16T14:37:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.538+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8815ed10 con 0x7fce8815cd70 2023-10-16T14:37:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.539+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce880c7e00 con 0x7fce8815cd70 2023-10-16T14:37:18.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.541+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fce80005680 con 0x7fce8815cd70 2023-10-16T14:37:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.541+0000 7fce7effd640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fce6c06af90 0x7fce6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.542+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fce80089850 con 0x7fce8815cd70 2023-10-16T14:37:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.542+0000 7fce8e552640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fce6c06af90 0x7fce6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.543+0000 7fce8e552640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fce6c06af90 0x7fce6c06d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce84006fd0 tx=0x7fce84008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:18.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.546+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce80054730 con 0x7fce8815cd70 2023-10-16T14:37:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.732+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 --> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce880d1d00 con 0x7fce6c06af90 2023-10-16T14:37:18.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.761+0000 7fce7effd640 1 -- 172.21.15.29:0/4212116895 <== mgr.7138 v2:172.21.15.170:6832/1985319551 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287487 (secure 0 0 0) 0x7fce8400ae40 con 0x7fce6c06af90 2023-10-16T14:37:18.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:37:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.767+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fce6c06af90 msgr2=0x7fce6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.767+0000 7fce8f554640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fce6c06af90 0x7fce6c06d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce84006fd0 tx=0x7fce84008040 comp rx=0 tx=0).stop 2023-10-16T14:37:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 msgr2=0x7fce880b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 0x7fce880b4400 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fce8000c730 tx=0x7fce8000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 shutdown_connections 2023-10-16T14:37:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fce6c06af90 0x7fce6c06d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce8815cd70 0x7fce880b4400 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce880d0be0 0x7fce8814d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 --2- 172.21.15.29:0/4212116895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce880b8480 0x7fce881f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 >> 172.21.15.29:0/4212116895 conn(0x7fce880cf570 msgr2=0x7fce880b7570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 shutdown_connections 2023-10-16T14:37:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:18.768+0000 7fce8f554640 1 -- 172.21.15.29:0/4212116895 wait complete. 2023-10-16T14:37:18.791 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T14:37:17.519820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":447,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":361512,"kb_used_data":16296,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754613208,"statfs":{"total":773094113280,"available":772723924992,"internally_reserved":0,"allocated":16687104,"data_stored":8657148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":353357369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412879"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376721+0000","last_change":"2023-10-16T14:37:14.164040+0000","last_active":"2023-10-16T14:37:15.376721+0000","last_peered":"2023-10-16T14:37:15.376721+0000","last_clean":"2023-10-16T14:37:15.376721+0000","last_became_active":"2023-10-16T14:37:14.162846+0000","last_became_peered":"2023-10-16T14:37:14.162846+0000","last_unstale":"2023-10-16T14:37:15.376721+0000","last_undegraded":"2023-10-16T14:37:15.376721+0000","last_fullsized":"2023-10-16T14:37:15.376721+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:11.397048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158335+0000","last_change":"2023-10-16T14:37:14.169501+0000","last_active":"2023-10-16T14:37:15.158335+0000","last_peered":"2023-10-16T14:37:15.158335+0000","last_clean":"2023-10-16T14:37:15.158335+0000","last_became_active":"2023-10-16T14:37:14.164348+0000","last_became_peered":"2023-10-16T14:37:14.164348+0000","last_unstale":"2023-10-16T14:37:15.158335+0000","last_undegraded":"2023-10-16T14:37:15.158335+0000","last_fullsized":"2023-10-16T14:37:15.158335+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.696246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378892+0000","last_change":"2023-10-16T14:37:14.160361+0000","last_active":"2023-10-16T14:37:15.378892+0000","last_peered":"2023-10-16T14:37:15.378892+0000","last_clean":"2023-10-16T14:37:15.378892+0000","last_became_active":"2023-10-16T14:37:14.160027+0000","last_became_peered":"2023-10-16T14:37:14.160027+0000","last_unstale":"2023-10-16T14:37:15.378892+0000","last_undegraded":"2023-10-16T14:37:15.378892+0000","last_fullsized":"2023-10-16T14:37:15.378892+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.324235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376659+0000","last_change":"2023-10-16T14:37:14.165967+0000","last_active":"2023-10-16T14:37:15.376659+0000","last_peered":"2023-10-16T14:37:15.376659+0000","last_clean":"2023-10-16T14:37:15.376659+0000","last_became_active":"2023-10-16T14:37:14.165594+0000","last_became_peered":"2023-10-16T14:37:14.165594+0000","last_unstale":"2023-10-16T14:37:15.376659+0000","last_undegraded":"2023-10-16T14:37:15.376659+0000","last_fullsized":"2023-10-16T14:37:15.376659+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:31.262913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162461+0000","last_change":"2023-10-16T14:37:14.163370+0000","last_active":"2023-10-16T14:37:15.162461+0000","last_peered":"2023-10-16T14:37:15.162461+0000","last_clean":"2023-10-16T14:37:15.162461+0000","last_became_active":"2023-10-16T14:37:14.163070+0000","last_became_peered":"2023-10-16T14:37:14.163070+0000","last_unstale":"2023-10-16T14:37:15.162461+0000","last_undegraded":"2023-10-16T14:37:15.162461+0000","last_fullsized":"2023-10-16T14:37:15.162461+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:54.313790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164102+0000","last_change":"2023-10-16T14:37:14.154272+0000","last_active":"2023-10-16T14:37:15.164102+0000","last_peered":"2023-10-16T14:37:15.164102+0000","last_clean":"2023-10-16T14:37:15.164102+0000","last_became_active":"2023-10-16T14:37:14.151846+0000","last_became_peered":"2023-10-16T14:37:14.151846+0000","last_unstale":"2023-10-16T14:37:15.164102+0000","last_undegraded":"2023-10-16T14:37:15.164102+0000","last_fullsized":"2023-10-16T14:37:15.164102+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:07.393228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163398+0000","last_change":"2023-10-16T14:37:14.159875+0000","last_active":"2023-10-16T14:37:15.163398+0000","last_peered":"2023-10-16T14:37:15.163398+0000","last_clean":"2023-10-16T14:37:15.163398+0000","last_became_active":"2023-10-16T14:37:14.159357+0000","last_became_peered":"2023-10-16T14:37:14.159357+0000","last_unstale":"2023-10-16T14:37:15.163398+0000","last_undegraded":"2023-10-16T14:37:15.163398+0000","last_fullsized":"2023-10-16T14:37:15.163398+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:50.466482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378252+0000","last_change":"2023-10-16T14:37:14.169747+0000","last_active":"2023-10-16T14:37:15.378252+0000","last_peered":"2023-10-16T14:37:15.378252+0000","last_clean":"2023-10-16T14:37:15.378252+0000","last_became_active":"2023-10-16T14:37:14.169357+0000","last_became_peered":"2023-10-16T14:37:14.169357+0000","last_unstale":"2023-10-16T14:37:15.378252+0000","last_undegraded":"2023-10-16T14:37:15.378252+0000","last_fullsized":"2023-10-16T14:37:15.378252+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:57.093926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379060+0000","last_change":"2023-10-16T14:37:14.162024+0000","last_active":"2023-10-16T14:37:15.379060+0000","last_peered":"2023-10-16T14:37:15.379060+0000","last_clean":"2023-10-16T14:37:15.379060+0000","last_became_active":"2023-10-16T14:37:14.161748+0000","last_became_peered":"2023-10-16T14:37:14.161748+0000","last_unstale":"2023-10-16T14:37:15.379060+0000","last_undegraded":"2023-10-16T14:37:15.379060+0000","last_fullsized":"2023-10-16T14:37:15.379060+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:54.857137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158518+0000","last_change":"2023-10-16T14:37:14.163076+0000","last_active":"2023-10-16T14:37:15.158518+0000","last_peered":"2023-10-16T14:37:15.158518+0000","last_clean":"2023-10-16T14:37:15.158518+0000","last_became_active":"2023-10-16T14:37:14.162573+0000","last_became_peered":"2023-10-16T14:37:14.162573+0000","last_unstale":"2023-10-16T14:37:15.158518+0000","last_undegraded":"2023-10-16T14:37:15.158518+0000","last_fullsized":"2023-10-16T14:37:15.158518+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:37.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379807+0000","last_change":"2023-10-16T14:37:14.161916+0000","last_active":"2023-10-16T14:37:15.379807+0000","last_peered":"2023-10-16T14:37:15.379807+0000","last_clean":"2023-10-16T14:37:15.379807+0000","last_became_active":"2023-10-16T14:37:14.161600+0000","last_became_peered":"2023-10-16T14:37:14.161600+0000","last_unstale":"2023-10-16T14:37:15.379807+0000","last_undegraded":"2023-10-16T14:37:15.379807+0000","last_fullsized":"2023-10-16T14:37:15.379807+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:45.257306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378563+0000","last_change":"2023-10-16T14:37:14.165627+0000","last_active":"2023-10-16T14:37:15.378563+0000","last_peered":"2023-10-16T14:37:15.378563+0000","last_clean":"2023-10-16T14:37:15.378563+0000","last_became_active":"2023-10-16T14:37:14.165337+0000","last_became_peered":"2023-10-16T14:37:14.165337+0000","last_unstale":"2023-10-16T14:37:15.378563+0000","last_undegraded":"2023-10-16T14:37:15.378563+0000","last_fullsized":"2023-10-16T14:37:15.378563+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:00.151071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376998+0000","last_change":"2023-10-16T14:37:14.159699+0000","last_active":"2023-10-16T14:37:15.376998+0000","last_peered":"2023-10-16T14:37:15.376998+0000","last_clean":"2023-10-16T14:37:15.376998+0000","last_became_active":"2023-10-16T14:37:14.159391+0000","last_became_peered":"2023-10-16T14:37:14.159391+0000","last_unstale":"2023-10-16T14:37:15.376998+0000","last_undegraded":"2023-10-16T14:37:15.376998+0000","last_fullsized":"2023-10-16T14:37:15.376998+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:34.926136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158019+0000","last_change":"2023-10-16T14:37:14.158520+0000","last_active":"2023-10-16T14:37:15.158019+0000","last_peered":"2023-10-16T14:37:15.158019+0000","last_clean":"2023-10-16T14:37:15.158019+0000","last_became_active":"2023-10-16T14:37:14.158197+0000","last_became_peered":"2023-10-16T14:37:14.158197+0000","last_unstale":"2023-10-16T14:37:15.158019+0000","last_undegraded":"2023-10-16T14:37:15.158019+0000","last_fullsized":"2023-10-16T14:37:15.158019+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:19.542779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.375603+0000","last_change":"2023-10-16T14:37:14.173364+0000","last_active":"2023-10-16T14:37:15.375603+0000","last_peered":"2023-10-16T14:37:15.375603+0000","last_clean":"2023-10-16T14:37:15.375603+0000","last_became_active":"2023-10-16T14:37:14.172894+0000","last_became_peered":"2023-10-16T14:37:14.172894+0000","last_unstale":"2023-10-16T14:37:15.375603+0000","last_undegraded":"2023-10-16T14:37:15.375603+0000","last_fullsized":"2023-10-16T14:37:15.375603+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:12.849027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378711+0000","last_change":"2023-10-16T14:37:14.172943+0000","last_active":"2023-10-16T14:37:15.378711+0000","last_peered":"2023-10-16T14:37:15.378711+0000","last_clean":"2023-10-16T14:37:15.378711+0000","last_became_active":"2023-10-16T14:37:14.172645+0000","last_became_peered":"2023-10-16T14:37:14.172645+0000","last_unstale":"2023-10-16T14:37:15.378711+0000","last_undegraded":"2023-10-16T14:37:15.378711+0000","last_fullsized":"2023-10-16T14:37:15.378711+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:45.747808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164881+0000","last_change":"2023-10-16T14:37:14.161475+0000","last_active":"2023-10-16T14:37:15.164881+0000","last_peered":"2023-10-16T14:37:15.164881+0000","last_clean":"2023-10-16T14:37:15.164881+0000","last_became_active":"2023-10-16T14:37:14.160829+0000","last_became_peered":"2023-10-16T14:37:14.160829+0000","last_unstale":"2023-10-16T14:37:15.164881+0000","last_undegraded":"2023-10-16T14:37:15.164881+0000","last_fullsized":"2023-10-16T14:37:15.164881+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:59.397938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379292+0000","last_change":"2023-10-16T14:37:14.162528+0000","last_active":"2023-10-16T14:37:15.379292+0000","last_peered":"2023-10-16T14:37:15.379292+0000","last_clean":"2023-10-16T14:37:15.379292+0000","last_became_active":"2023-10-16T14:37:14.162271+0000","last_became_peered":"2023-10-16T14:37:14.162271+0000","last_unstale":"2023-10-16T14:37:15.379292+0000","last_undegraded":"2023-10-16T14:37:15.379292+0000","last_fullsized":"2023-10-16T14:37:15.379292+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:27.947212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380157+0000","last_change":"2023-10-16T14:37:14.150190+0000","last_active":"2023-10-16T14:37:15.380157+0000","last_peered":"2023-10-16T14:37:15.380157+0000","last_clean":"2023-10-16T14:37:15.380157+0000","last_became_active":"2023-10-16T14:37:14.149360+0000","last_became_peered":"2023-10-16T14:37:14.149360+0000","last_unstale":"2023-10-16T14:37:15.380157+0000","last_undegraded":"2023-10-16T14:37:15.380157+0000","last_fullsized":"2023-10-16T14:37:15.380157+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:25.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379937+0000","last_change":"2023-10-16T14:37:14.164957+0000","last_active":"2023-10-16T14:37:15.379937+0000","last_peered":"2023-10-16T14:37:15.379937+0000","last_clean":"2023-10-16T14:37:15.379937+0000","last_became_active":"2023-10-16T14:37:14.164649+0000","last_became_peered":"2023-10-16T14:37:14.164649+0000","last_unstale":"2023-10-16T14:37:15.379937+0000","last_undegraded":"2023-10-16T14:37:15.379937+0000","last_fullsized":"2023-10-16T14:37:15.379937+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:20.616678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162349+0000","last_change":"2023-10-16T14:37:14.164718+0000","last_active":"2023-10-16T14:37:15.162349+0000","last_peered":"2023-10-16T14:37:15.162349+0000","last_clean":"2023-10-16T14:37:15.162349+0000","last_became_active":"2023-10-16T14:37:14.163197+0000","last_became_peered":"2023-10-16T14:37:14.163197+0000","last_unstale":"2023-10-16T14:37:15.162349+0000","last_undegraded":"2023-10-16T14:37:15.162349+0000","last_fullsized":"2023-10-16T14:37:15.162349+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:35.111846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378568+0000","last_change":"2023-10-16T14:37:14.164657+0000","last_active":"2023-10-16T14:37:15.378568+0000","last_peered":"2023-10-16T14:37:15.378568+0000","last_clean":"2023-10-16T14:37:15.378568+0000","last_became_active":"2023-10-16T14:37:14.163329+0000","last_became_peered":"2023-10-16T14:37:14.163329+0000","last_unstale":"2023-10-16T14:37:15.378568+0000","last_undegraded":"2023-10-16T14:37:15.378568+0000","last_fullsized":"2023-10-16T14:37:15.378568+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:05.110417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378505+0000","last_change":"2023-10-16T14:37:14.162269+0000","last_active":"2023-10-16T14:37:15.378505+0000","last_peered":"2023-10-16T14:37:15.378505+0000","last_clean":"2023-10-16T14:37:15.378505+0000","last_became_active":"2023-10-16T14:37:14.161780+0000","last_became_peered":"2023-10-16T14:37:14.161780+0000","last_unstale":"2023-10-16T14:37:15.378505+0000","last_undegraded":"2023-10-16T14:37:15.378505+0000","last_fullsized":"2023-10-16T14:37:15.378505+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:13.308843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377650+0000","last_change":"2023-10-16T14:37:14.155853+0000","last_active":"2023-10-16T14:37:15.377650+0000","last_peered":"2023-10-16T14:37:15.377650+0000","last_clean":"2023-10-16T14:37:15.377650+0000","last_became_active":"2023-10-16T14:37:14.155552+0000","last_became_peered":"2023-10-16T14:37:14.155552+0000","last_unstale":"2023-10-16T14:37:15.377650+0000","last_undegraded":"2023-10-16T14:37:15.377650+0000","last_fullsized":"2023-10-16T14:37:15.377650+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:57.197725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158723+0000","last_change":"2023-10-16T14:37:14.167518+0000","last_active":"2023-10-16T14:37:15.158723+0000","last_peered":"2023-10-16T14:37:15.158723+0000","last_clean":"2023-10-16T14:37:15.158723+0000","last_became_active":"2023-10-16T14:37:14.167186+0000","last_became_peered":"2023-10-16T14:37:14.167186+0000","last_unstale":"2023-10-16T14:37:15.158723+0000","last_undegraded":"2023-10-16T14:37:15.158723+0000","last_fullsized":"2023-10-16T14:37:15.158723+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.348025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379789+0000","last_change":"2023-10-16T14:37:14.174870+0000","last_active":"2023-10-16T14:37:15.379789+0000","last_peered":"2023-10-16T14:37:15.379789+0000","last_clean":"2023-10-16T14:37:15.379789+0000","last_became_active":"2023-10-16T14:37:14.174602+0000","last_became_peered":"2023-10-16T14:37:14.174602+0000","last_unstale":"2023-10-16T14:37:15.379789+0000","last_undegraded":"2023-10-16T14:37:15.379789+0000","last_fullsized":"2023-10-16T14:37:15.379789+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:05.422010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377408+0000","last_change":"2023-10-16T14:37:14.158672+0000","last_active":"2023-10-16T14:37:15.377408+0000","last_peered":"2023-10-16T14:37:15.377408+0000","last_clean":"2023-10-16T14:37:15.377408+0000","last_became_active":"2023-10-16T14:37:14.158128+0000","last_became_peered":"2023-10-16T14:37:14.158128+0000","last_unstale":"2023-10-16T14:37:15.377408+0000","last_undegraded":"2023-10-16T14:37:15.377408+0000","last_fullsized":"2023-10-16T14:37:15.377408+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:48.079668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175535+0000","last_change":"2023-10-16T14:37:14.163583+0000","last_active":"2023-10-16T14:37:15.175535+0000","last_peered":"2023-10-16T14:37:15.175535+0000","last_clean":"2023-10-16T14:37:15.175535+0000","last_became_active":"2023-10-16T14:37:14.162784+0000","last_became_peered":"2023-10-16T14:37:14.162784+0000","last_unstale":"2023-10-16T14:37:15.175535+0000","last_undegraded":"2023-10-16T14:37:15.175535+0000","last_fullsized":"2023-10-16T14:37:15.175535+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.565036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376688+0000","last_change":"2023-10-16T14:37:14.167823+0000","last_active":"2023-10-16T14:37:15.376688+0000","last_peered":"2023-10-16T14:37:15.376688+0000","last_clean":"2023-10-16T14:37:15.376688+0000","last_became_active":"2023-10-16T14:37:14.167138+0000","last_became_peered":"2023-10-16T14:37:14.167138+0000","last_unstale":"2023-10-16T14:37:15.376688+0000","last_undegraded":"2023-10-16T14:37:15.376688+0000","last_fullsized":"2023-10-16T14:37:15.376688+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:04.709961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.375804+0000","last_change":"2023-10-16T14:37:14.170081+0000","last_active":"2023-10-16T14:37:15.375804+0000","last_peered":"2023-10-16T14:37:15.375804+0000","last_clean":"2023-10-16T14:37:15.375804+0000","last_became_active":"2023-10-16T14:37:14.169211+0000","last_became_peered":"2023-10-16T14:37:14.169211+0000","last_unstale":"2023-10-16T14:37:15.375804+0000","last_undegraded":"2023-10-16T14:37:15.375804+0000","last_fullsized":"2023-10-16T14:37:15.375804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:54.857137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162360+0000","last_change":"2023-10-16T14:37:14.165925+0000","last_active":"2023-10-16T14:37:15.162360+0000","last_peered":"2023-10-16T14:37:15.162360+0000","last_clean":"2023-10-16T14:37:15.162360+0000","last_became_active":"2023-10-16T14:37:14.164385+0000","last_became_peered":"2023-10-16T14:37:14.164385+0000","last_unstale":"2023-10-16T14:37:15.162360+0000","last_undegraded":"2023-10-16T14:37:15.162360+0000","last_fullsized":"2023-10-16T14:37:15.162360+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:06.826826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162718+0000","last_change":"2023-10-16T14:37:14.167773+0000","last_active":"2023-10-16T14:37:15.162718+0000","last_peered":"2023-10-16T14:37:15.162718+0000","last_clean":"2023-10-16T14:37:15.162718+0000","last_became_active":"2023-10-16T14:37:14.167452+0000","last_became_peered":"2023-10-16T14:37:14.167452+0000","last_unstale":"2023-10-16T14:37:15.162718+0000","last_undegraded":"2023-10-16T14:37:15.162718+0000","last_fullsized":"2023-10-16T14:37:15.162718+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:21.327054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380163+0000","last_change":"2023-10-16T14:37:14.159713+0000","last_active":"2023-10-16T14:37:15.380163+0000","last_peered":"2023-10-16T14:37:15.380163+0000","last_clean":"2023-10-16T14:37:15.380163+0000","last_became_active":"2023-10-16T14:37:14.159400+0000","last_became_peered":"2023-10-16T14:37:14.159400+0000","last_unstale":"2023-10-16T14:37:15.380163+0000","last_undegraded":"2023-10-16T14:37:15.380163+0000","last_fullsized":"2023-10-16T14:37:15.380163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:27.218231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"154'4","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163361+0000","last_change":"2023-10-16T14:37:13.159947+0000","last_active":"2023-10-16T14:37:15.163361+0000","last_peered":"2023-10-16T14:37:15.163361+0000","last_clean":"2023-10-16T14:37:15.163361+0000","last_became_active":"2023-10-16T14:37:13.158935+0000","last_became_peered":"2023-10-16T14:37:13.158935+0000","last_unstale":"2023-10-16T14:37:15.163361+0000","last_undegraded":"2023-10-16T14:37:15.163361+0000","last_fullsized":"2023-10-16T14:37:15.163361+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:46.628347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379282+0000","last_change":"2023-10-16T14:37:14.172137+0000","last_active":"2023-10-16T14:37:15.379282+0000","last_peered":"2023-10-16T14:37:15.379282+0000","last_clean":"2023-10-16T14:37:15.379282+0000","last_became_active":"2023-10-16T14:37:14.171809+0000","last_became_peered":"2023-10-16T14:37:14.171809+0000","last_unstale":"2023-10-16T14:37:15.379282+0000","last_undegraded":"2023-10-16T14:37:15.379282+0000","last_fullsized":"2023-10-16T14:37:15.379282+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:50.473704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175765+0000","last_change":"2023-10-16T14:37:13.157211+0000","last_active":"2023-10-16T14:37:15.175765+0000","last_peered":"2023-10-16T14:37:15.175765+0000","last_clean":"2023-10-16T14:37:15.175765+0000","last_became_active":"2023-10-16T14:37:13.156941+0000","last_became_peered":"2023-10-16T14:37:13.156941+0000","last_unstale":"2023-10-16T14:37:15.175765+0000","last_undegraded":"2023-10-16T14:37:15.175765+0000","last_fullsized":"2023-10-16T14:37:15.175765+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:08.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163223+0000","last_change":"2023-10-16T14:37:14.155082+0000","last_active":"2023-10-16T14:37:15.163223+0000","last_peered":"2023-10-16T14:37:15.163223+0000","last_clean":"2023-10-16T14:37:15.163223+0000","last_became_active":"2023-10-16T14:37:14.154796+0000","last_became_peered":"2023-10-16T14:37:14.154796+0000","last_unstale":"2023-10-16T14:37:15.163223+0000","last_undegraded":"2023-10-16T14:37:15.163223+0000","last_fullsized":"2023-10-16T14:37:15.163223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:50.620680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380235+0000","last_change":"2023-10-16T14:37:13.154326+0000","last_active":"2023-10-16T14:37:15.380235+0000","last_peered":"2023-10-16T14:37:15.380235+0000","last_clean":"2023-10-16T14:37:15.380235+0000","last_became_active":"2023-10-16T14:37:13.154006+0000","last_became_peered":"2023-10-16T14:37:13.154006+0000","last_unstale":"2023-10-16T14:37:15.380235+0000","last_undegraded":"2023-10-16T14:37:15.380235+0000","last_fullsized":"2023-10-16T14:37:15.380235+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:57:17.168445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162684+0000","last_change":"2023-10-16T14:37:14.158133+0000","last_active":"2023-10-16T14:37:15.162684+0000","last_peered":"2023-10-16T14:37:15.162684+0000","last_clean":"2023-10-16T14:37:15.162684+0000","last_became_active":"2023-10-16T14:37:14.157848+0000","last_became_peered":"2023-10-16T14:37:14.157848+0000","last_unstale":"2023-10-16T14:37:15.162684+0000","last_undegraded":"2023-10-16T14:37:15.162684+0000","last_fullsized":"2023-10-16T14:37:15.162684+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:48:29.049308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158238+0000","last_change":"2023-10-16T14:37:13.150181+0000","last_active":"2023-10-16T14:37:15.158238+0000","last_peered":"2023-10-16T14:37:15.158238+0000","last_clean":"2023-10-16T14:37:15.158238+0000","last_became_active":"2023-10-16T14:37:13.149921+0000","last_became_peered":"2023-10-16T14:37:13.149921+0000","last_unstale":"2023-10-16T14:37:15.158238+0000","last_undegraded":"2023-10-16T14:37:15.158238+0000","last_fullsized":"2023-10-16T14:37:15.158238+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:23.905986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379139+0000","last_change":"2023-10-16T14:37:14.157362+0000","last_active":"2023-10-16T14:37:15.379139+0000","last_peered":"2023-10-16T14:37:15.379139+0000","last_clean":"2023-10-16T14:37:15.379139+0000","last_became_active":"2023-10-16T14:37:14.157036+0000","last_became_peered":"2023-10-16T14:37:14.157036+0000","last_unstale":"2023-10-16T14:37:15.379139+0000","last_undegraded":"2023-10-16T14:37:15.379139+0000","last_fullsized":"2023-10-16T14:37:15.379139+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:18.632217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157985+0000","last_change":"2023-10-16T14:37:13.152419+0000","last_active":"2023-10-16T14:37:15.157985+0000","last_peered":"2023-10-16T14:37:15.157985+0000","last_clean":"2023-10-16T14:37:15.157985+0000","last_became_active":"2023-10-16T14:37:13.152138+0000","last_became_peered":"2023-10-16T14:37:13.152138+0000","last_unstale":"2023-10-16T14:37:15.157985+0000","last_undegraded":"2023-10-16T14:37:15.157985+0000","last_fullsized":"2023-10-16T14:37:15.157985+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:40.057057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164245+0000","last_change":"2023-10-16T14:37:14.156987+0000","last_active":"2023-10-16T14:37:15.164245+0000","last_peered":"2023-10-16T14:37:15.164245+0000","last_clean":"2023-10-16T14:37:15.164245+0000","last_became_active":"2023-10-16T14:37:14.156684+0000","last_became_peered":"2023-10-16T14:37:14.156684+0000","last_unstale":"2023-10-16T14:37:15.164245+0000","last_undegraded":"2023-10-16T14:37:15.164245+0000","last_fullsized":"2023-10-16T14:37:15.164245+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:19.806281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379008+0000","last_change":"2023-10-16T14:37:13.145105+0000","last_active":"2023-10-16T14:37:15.379008+0000","last_peered":"2023-10-16T14:37:15.379008+0000","last_clean":"2023-10-16T14:37:15.379008+0000","last_became_active":"2023-10-16T14:37:13.144824+0000","last_became_peered":"2023-10-16T14:37:13.144824+0000","last_unstale":"2023-10-16T14:37:15.379008+0000","last_undegraded":"2023-10-16T14:37:15.379008+0000","last_fullsized":"2023-10-16T14:37:15.379008+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:40.115179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378334+0000","last_change":"2023-10-16T14:37:14.169627+0000","last_active":"2023-10-16T14:37:15.378334+0000","last_peered":"2023-10-16T14:37:15.378334+0000","last_clean":"2023-10-16T14:37:15.378334+0000","last_became_active":"2023-10-16T14:37:14.168342+0000","last_became_peered":"2023-10-16T14:37:14.168342+0000","last_unstale":"2023-10-16T14:37:15.378334+0000","last_undegraded":"2023-10-16T14:37:15.378334+0000","last_fullsized":"2023-10-16T14:37:15.378334+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:24.919946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163637+0000","last_change":"2023-10-16T14:37:13.156656+0000","last_active":"2023-10-16T14:37:15.163637+0000","last_peered":"2023-10-16T14:37:15.163637+0000","last_clean":"2023-10-16T14:37:15.163637+0000","last_became_active":"2023-10-16T14:37:13.156323+0000","last_became_peered":"2023-10-16T14:37:13.156323+0000","last_unstale":"2023-10-16T14:37:15.163637+0000","last_undegraded":"2023-10-16T14:37:15.163637+0000","last_fullsized":"2023-10-16T14:37:15.163637+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.551075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157780+0000","last_change":"2023-10-16T14:37:14.163939+0000","last_active":"2023-10-16T14:37:15.157780+0000","last_peered":"2023-10-16T14:37:15.157780+0000","last_clean":"2023-10-16T14:37:15.157780+0000","last_became_active":"2023-10-16T14:37:14.163415+0000","last_became_peered":"2023-10-16T14:37:14.163415+0000","last_unstale":"2023-10-16T14:37:15.157780+0000","last_undegraded":"2023-10-16T14:37:15.157780+0000","last_fullsized":"2023-10-16T14:37:15.157780+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:04.709961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157887+0000","last_change":"2023-10-16T14:37:13.152494+0000","last_active":"2023-10-16T14:37:15.157887+0000","last_peered":"2023-10-16T14:37:15.157887+0000","last_clean":"2023-10-16T14:37:15.157887+0000","last_became_active":"2023-10-16T14:37:13.152207+0000","last_became_peered":"2023-10-16T14:37:13.152207+0000","last_unstale":"2023-10-16T14:37:15.157887+0000","last_undegraded":"2023-10-16T14:37:15.157887+0000","last_fullsized":"2023-10-16T14:37:15.157887+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:33.912175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377408+0000","last_change":"2023-10-16T14:37:14.161603+0000","last_active":"2023-10-16T14:37:15.377408+0000","last_peered":"2023-10-16T14:37:15.377408+0000","last_clean":"2023-10-16T14:37:15.377408+0000","last_became_active":"2023-10-16T14:37:14.161293+0000","last_became_peered":"2023-10-16T14:37:14.161293+0000","last_unstale":"2023-10-16T14:37:15.377408+0000","last_undegraded":"2023-10-16T14:37:15.377408+0000","last_fullsized":"2023-10-16T14:37:15.377408+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.696246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.176368+0000","last_change":"2023-10-16T14:37:13.152057+0000","last_active":"2023-10-16T14:37:15.176368+0000","last_peered":"2023-10-16T14:37:15.176368+0000","last_clean":"2023-10-16T14:37:15.176368+0000","last_became_active":"2023-10-16T14:37:13.151714+0000","last_became_peered":"2023-10-16T14:37:13.151714+0000","last_unstale":"2023-10-16T14:37:15.176368+0000","last_undegraded":"2023-10-16T14:37:15.176368+0000","last_fullsized":"2023-10-16T14:37:15.176368+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:45:00.542986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175995+0000","last_change":"2023-10-16T14:37:14.162024+0000","last_active":"2023-10-16T14:37:15.175995+0000","last_peered":"2023-10-16T14:37:15.175995+0000","last_clean":"2023-10-16T14:37:15.175995+0000","last_became_active":"2023-10-16T14:37:14.161727+0000","last_became_peered":"2023-10-16T14:37:14.161727+0000","last_unstale":"2023-10-16T14:37:15.175995+0000","last_undegraded":"2023-10-16T14:37:15.175995+0000","last_fullsized":"2023-10-16T14:37:15.175995+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:29.932025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380395+0000","last_change":"2023-10-16T14:37:13.158699+0000","last_active":"2023-10-16T14:37:15.380395+0000","last_peered":"2023-10-16T14:37:15.380395+0000","last_clean":"2023-10-16T14:37:15.380395+0000","last_became_active":"2023-10-16T14:37:13.158077+0000","last_became_peered":"2023-10-16T14:37:13.158077+0000","last_unstale":"2023-10-16T14:37:15.380395+0000","last_undegraded":"2023-10-16T14:37:15.380395+0000","last_fullsized":"2023-10-16T14:37:15.380395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:09.048184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163600+0000","last_change":"2023-10-16T14:37:14.160997+0000","last_active":"2023-10-16T14:37:15.163600+0000","last_peered":"2023-10-16T14:37:15.163600+0000","last_clean":"2023-10-16T14:37:15.163600+0000","last_became_active":"2023-10-16T14:37:14.160048+0000","last_became_peered":"2023-10-16T14:37:14.160048+0000","last_unstale":"2023-10-16T14:37:15.163600+0000","last_undegraded":"2023-10-16T14:37:15.163600+0000","last_fullsized":"2023-10-16T14:37:15.163600+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:31:25.508509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"154'3","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175644+0000","last_change":"2023-10-16T14:37:13.153804+0000","last_active":"2023-10-16T14:37:15.175644+0000","last_peered":"2023-10-16T14:37:15.175644+0000","last_clean":"2023-10-16T14:37:15.175644+0000","last_became_active":"2023-10-16T14:37:13.153539+0000","last_became_peered":"2023-10-16T14:37:13.153539+0000","last_unstale":"2023-10-16T14:37:15.175644+0000","last_undegraded":"2023-10-16T14:37:15.175644+0000","last_fullsized":"2023-10-16T14:37:15.175644+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:28.195462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163047+0000","last_change":"2023-10-16T14:37:14.165061+0000","last_active":"2023-10-16T14:37:15.163047+0000","last_peered":"2023-10-16T14:37:15.163047+0000","last_clean":"2023-10-16T14:37:15.163047+0000","last_became_active":"2023-10-16T14:37:14.162497+0000","last_became_peered":"2023-10-16T14:37:14.162497+0000","last_unstale":"2023-10-16T14:37:15.163047+0000","last_undegraded":"2023-10-16T14:37:15.163047+0000","last_fullsized":"2023-10-16T14:37:15.163047+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:02.539156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"154'6","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378521+0000","last_change":"2023-10-16T14:37:13.154199+0000","last_active":"2023-10-16T14:37:15.378521+0000","last_peered":"2023-10-16T14:37:15.378521+0000","last_clean":"2023-10-16T14:37:15.378521+0000","last_became_active":"2023-10-16T14:37:13.153918+0000","last_became_peered":"2023-10-16T14:37:13.153918+0000","last_unstale":"2023-10-16T14:37:15.378521+0000","last_undegraded":"2023-10-16T14:37:15.378521+0000","last_fullsized":"2023-10-16T14:37:15.378521+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:49.808976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377022+0000","last_change":"2023-10-16T14:37:14.171955+0000","last_active":"2023-10-16T14:37:15.377022+0000","last_peered":"2023-10-16T14:37:15.377022+0000","last_clean":"2023-10-16T14:37:15.377022+0000","last_became_active":"2023-10-16T14:37:14.171689+0000","last_became_peered":"2023-10-16T14:37:14.171689+0000","last_unstale":"2023-10-16T14:37:15.377022+0000","last_undegraded":"2023-10-16T14:37:15.377022+0000","last_fullsized":"2023-10-16T14:37:15.377022+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:24.855910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.375900+0000","last_change":"2023-10-16T14:37:13.150557+0000","last_active":"2023-10-16T14:37:15.375900+0000","last_peered":"2023-10-16T14:37:15.375900+0000","last_clean":"2023-10-16T14:37:15.375900+0000","last_became_active":"2023-10-16T14:37:13.149432+0000","last_became_peered":"2023-10-16T14:37:13.149432+0000","last_unstale":"2023-10-16T14:37:15.375900+0000","last_undegraded":"2023-10-16T14:37:15.375900+0000","last_fullsized":"2023-10-16T14:37:15.375900+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:59.137110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157763+0000","last_change":"2023-10-16T14:37:14.168758+0000","last_active":"2023-10-16T14:37:15.157763+0000","last_peered":"2023-10-16T14:37:15.157763+0000","last_clean":"2023-10-16T14:37:15.157763+0000","last_became_active":"2023-10-16T14:37:14.168458+0000","last_became_peered":"2023-10-16T14:37:14.168458+0000","last_unstale":"2023-10-16T14:37:15.157763+0000","last_undegraded":"2023-10-16T14:37:15.157763+0000","last_fullsized":"2023-10-16T14:37:15.157763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:31.262913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157864+0000","last_change":"2023-10-16T14:37:13.154504+0000","last_active":"2023-10-16T14:37:15.157864+0000","last_peered":"2023-10-16T14:37:15.157864+0000","last_clean":"2023-10-16T14:37:15.157864+0000","last_became_active":"2023-10-16T14:37:13.154246+0000","last_became_peered":"2023-10-16T14:37:13.154246+0000","last_unstale":"2023-10-16T14:37:15.157864+0000","last_undegraded":"2023-10-16T14:37:15.157864+0000","last_fullsized":"2023-10-16T14:37:15.157864+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:05.706662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378332+0000","last_change":"2023-10-16T14:37:14.164130+0000","last_active":"2023-10-16T14:37:15.378332+0000","last_peered":"2023-10-16T14:37:15.378332+0000","last_clean":"2023-10-16T14:37:15.378332+0000","last_became_active":"2023-10-16T14:37:14.163863+0000","last_became_peered":"2023-10-16T14:37:14.163863+0000","last_unstale":"2023-10-16T14:37:15.378332+0000","last_undegraded":"2023-10-16T14:37:15.378332+0000","last_fullsized":"2023-10-16T14:37:15.378332+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:29.639359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"154'8","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378619+0000","last_change":"2023-10-16T14:37:13.156769+0000","last_active":"2023-10-16T14:37:15.378619+0000","last_peered":"2023-10-16T14:37:15.378619+0000","last_clean":"2023-10-16T14:37:15.378619+0000","last_became_active":"2023-10-16T14:37:13.156444+0000","last_became_peered":"2023-10-16T14:37:13.156444+0000","last_unstale":"2023-10-16T14:37:15.378619+0000","last_undegraded":"2023-10-16T14:37:15.378619+0000","last_fullsized":"2023-10-16T14:37:15.378619+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:26.635254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378941+0000","last_change":"2023-10-16T14:37:14.172468+0000","last_active":"2023-10-16T14:37:15.378941+0000","last_peered":"2023-10-16T14:37:15.378941+0000","last_clean":"2023-10-16T14:37:15.378941+0000","last_became_active":"2023-10-16T14:37:14.172178+0000","last_became_peered":"2023-10-16T14:37:14.172178+0000","last_unstale":"2023-10-16T14:37:15.378941+0000","last_undegraded":"2023-10-16T14:37:15.378941+0000","last_fullsized":"2023-10-16T14:37:15.378941+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:12.849027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.176418+0000","last_change":"2023-10-16T14:37:13.153936+0000","last_active":"2023-10-16T14:37:15.176418+0000","last_peered":"2023-10-16T14:37:15.176418+0000","last_clean":"2023-10-16T14:37:15.176418+0000","last_became_active":"2023-10-16T14:37:13.153629+0000","last_became_peered":"2023-10-16T14:37:13.153629+0000","last_unstale":"2023-10-16T14:37:15.176418+0000","last_undegraded":"2023-10-16T14:37:15.176418+0000","last_fullsized":"2023-10-16T14:37:15.176418+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:14.613465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.176119+0000","last_change":"2023-10-16T14:37:14.164956+0000","last_active":"2023-10-16T14:37:15.176119+0000","last_peered":"2023-10-16T14:37:15.176119+0000","last_clean":"2023-10-16T14:37:15.176119+0000","last_became_active":"2023-10-16T14:37:14.164351+0000","last_became_peered":"2023-10-16T14:37:14.164351+0000","last_unstale":"2023-10-16T14:37:15.176119+0000","last_undegraded":"2023-10-16T14:37:15.176119+0000","last_fullsized":"2023-10-16T14:37:15.176119+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:21.327054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380505+0000","last_change":"2023-10-16T14:37:13.159746+0000","last_active":"2023-10-16T14:37:15.380505+0000","last_peered":"2023-10-16T14:37:15.380505+0000","last_clean":"2023-10-16T14:37:15.380505+0000","last_became_active":"2023-10-16T14:37:13.159395+0000","last_became_peered":"2023-10-16T14:37:13.159395+0000","last_unstale":"2023-10-16T14:37:15.380505+0000","last_undegraded":"2023-10-16T14:37:15.380505+0000","last_fullsized":"2023-10-16T14:37:15.380505+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:10.570994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380012+0000","last_change":"2023-10-16T14:37:14.173360+0000","last_active":"2023-10-16T14:37:15.380012+0000","last_peered":"2023-10-16T14:37:15.380012+0000","last_clean":"2023-10-16T14:37:15.380012+0000","last_became_active":"2023-10-16T14:37:14.172893+0000","last_became_peered":"2023-10-16T14:37:14.172893+0000","last_unstale":"2023-10-16T14:37:15.380012+0000","last_undegraded":"2023-10-16T14:37:15.380012+0000","last_fullsized":"2023-10-16T14:37:15.380012+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:31.945899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164779+0000","last_change":"2023-10-16T14:37:13.152582+0000","last_active":"2023-10-16T14:37:15.164779+0000","last_peered":"2023-10-16T14:37:15.164779+0000","last_clean":"2023-10-16T14:37:15.164779+0000","last_became_active":"2023-10-16T14:37:13.152311+0000","last_became_peered":"2023-10-16T14:37:13.152311+0000","last_unstale":"2023-10-16T14:37:15.164779+0000","last_undegraded":"2023-10-16T14:37:15.164779+0000","last_fullsized":"2023-10-16T14:37:15.164779+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:44.375843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158555+0000","last_change":"2023-10-16T14:37:14.162468+0000","last_active":"2023-10-16T14:37:15.158555+0000","last_peered":"2023-10-16T14:37:15.158555+0000","last_clean":"2023-10-16T14:37:15.158555+0000","last_became_active":"2023-10-16T14:37:14.162206+0000","last_became_peered":"2023-10-16T14:37:14.162206+0000","last_unstale":"2023-10-16T14:37:15.158555+0000","last_undegraded":"2023-10-16T14:37:15.158555+0000","last_fullsized":"2023-10-16T14:37:15.158555+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:09.487380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175588+0000","last_change":"2023-10-16T14:37:13.155921+0000","last_active":"2023-10-16T14:37:15.175588+0000","last_peered":"2023-10-16T14:37:15.175588+0000","last_clean":"2023-10-16T14:37:15.175588+0000","last_became_active":"2023-10-16T14:37:13.155569+0000","last_became_peered":"2023-10-16T14:37:13.155569+0000","last_unstale":"2023-10-16T14:37:15.175588+0000","last_undegraded":"2023-10-16T14:37:15.175588+0000","last_fullsized":"2023-10-16T14:37:15.175588+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:03.169423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376009+0000","last_change":"2023-10-16T14:37:14.161567+0000","last_active":"2023-10-16T14:37:15.376009+0000","last_peered":"2023-10-16T14:37:15.376009+0000","last_clean":"2023-10-16T14:37:15.376009+0000","last_became_active":"2023-10-16T14:37:14.159001+0000","last_became_peered":"2023-10-16T14:37:14.159001+0000","last_unstale":"2023-10-16T14:37:15.376009+0000","last_undegraded":"2023-10-16T14:37:15.376009+0000","last_fullsized":"2023-10-16T14:37:15.376009+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:50.473704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175502+0000","last_change":"2023-10-16T14:37:13.153284+0000","last_active":"2023-10-16T14:37:15.175502+0000","last_peered":"2023-10-16T14:37:15.175502+0000","last_clean":"2023-10-16T14:37:15.175502+0000","last_became_active":"2023-10-16T14:37:13.152985+0000","last_became_peered":"2023-10-16T14:37:13.152985+0000","last_unstale":"2023-10-16T14:37:15.175502+0000","last_undegraded":"2023-10-16T14:37:15.175502+0000","last_fullsized":"2023-10-16T14:37:15.175502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:02.863281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164513+0000","last_change":"2023-10-16T14:37:14.164130+0000","last_active":"2023-10-16T14:37:15.164513+0000","last_peered":"2023-10-16T14:37:15.164513+0000","last_clean":"2023-10-16T14:37:15.164513+0000","last_became_active":"2023-10-16T14:37:14.162356+0000","last_became_peered":"2023-10-16T14:37:14.162356+0000","last_unstale":"2023-10-16T14:37:15.164513+0000","last_undegraded":"2023-10-16T14:37:15.164513+0000","last_fullsized":"2023-10-16T14:37:15.164513+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:34.558500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379234+0000","last_change":"2023-10-16T14:37:13.154608+0000","last_active":"2023-10-16T14:37:15.379234+0000","last_peered":"2023-10-16T14:37:15.379234+0000","last_clean":"2023-10-16T14:37:15.379234+0000","last_became_active":"2023-10-16T14:37:13.154337+0000","last_became_peered":"2023-10-16T14:37:13.154337+0000","last_unstale":"2023-10-16T14:37:15.379234+0000","last_undegraded":"2023-10-16T14:37:15.379234+0000","last_fullsized":"2023-10-16T14:37:15.379234+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:26:57.739325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158145+0000","last_change":"2023-10-16T14:37:14.165480+0000","last_active":"2023-10-16T14:37:15.158145+0000","last_peered":"2023-10-16T14:37:15.158145+0000","last_clean":"2023-10-16T14:37:15.158145+0000","last_became_active":"2023-10-16T14:37:14.165214+0000","last_became_peered":"2023-10-16T14:37:14.165214+0000","last_unstale":"2023-10-16T14:37:15.158145+0000","last_undegraded":"2023-10-16T14:37:15.158145+0000","last_fullsized":"2023-10-16T14:37:15.158145+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:24.855910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378445+0000","last_change":"2023-10-16T14:37:13.152198+0000","last_active":"2023-10-16T14:37:15.378445+0000","last_peered":"2023-10-16T14:37:15.378445+0000","last_clean":"2023-10-16T14:37:15.378445+0000","last_became_active":"2023-10-16T14:37:13.151780+0000","last_became_peered":"2023-10-16T14:37:13.151780+0000","last_unstale":"2023-10-16T14:37:15.378445+0000","last_undegraded":"2023-10-16T14:37:15.378445+0000","last_fullsized":"2023-10-16T14:37:15.378445+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:35.538081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.161958+0000","last_change":"2023-10-16T14:37:14.158257+0000","last_active":"2023-10-16T14:37:15.161958+0000","last_peered":"2023-10-16T14:37:15.161958+0000","last_clean":"2023-10-16T14:37:15.161958+0000","last_became_active":"2023-10-16T14:37:14.157947+0000","last_became_peered":"2023-10-16T14:37:14.157947+0000","last_unstale":"2023-10-16T14:37:15.161958+0000","last_undegraded":"2023-10-16T14:37:15.161958+0000","last_fullsized":"2023-10-16T14:37:15.161958+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:46.458141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377741+0000","last_change":"2023-10-16T14:37:13.160975+0000","last_active":"2023-10-16T14:37:15.377741+0000","last_peered":"2023-10-16T14:37:15.377741+0000","last_clean":"2023-10-16T14:37:15.377741+0000","last_became_active":"2023-10-16T14:37:13.158969+0000","last_became_peered":"2023-10-16T14:37:13.158969+0000","last_unstale":"2023-10-16T14:37:15.377741+0000","last_undegraded":"2023-10-16T14:37:15.377741+0000","last_fullsized":"2023-10-16T14:37:15.377741+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:19.682285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.165223+0000","last_change":"2023-10-16T14:37:14.161420+0000","last_active":"2023-10-16T14:37:15.165223+0000","last_peered":"2023-10-16T14:37:15.165223+0000","last_clean":"2023-10-16T14:37:15.165223+0000","last_became_active":"2023-10-16T14:37:14.160311+0000","last_became_peered":"2023-10-16T14:37:14.160311+0000","last_unstale":"2023-10-16T14:37:15.165223+0000","last_undegraded":"2023-10-16T14:37:15.165223+0000","last_fullsized":"2023-10-16T14:37:15.165223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:46.615789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376954+0000","last_change":"2023-10-16T14:37:13.158627+0000","last_active":"2023-10-16T14:37:15.376954+0000","last_peered":"2023-10-16T14:37:15.376954+0000","last_clean":"2023-10-16T14:37:15.376954+0000","last_became_active":"2023-10-16T14:37:13.158291+0000","last_became_peered":"2023-10-16T14:37:13.158291+0000","last_unstale":"2023-10-16T14:37:15.376954+0000","last_undegraded":"2023-10-16T14:37:15.376954+0000","last_fullsized":"2023-10-16T14:37:15.376954+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:56.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376857+0000","last_change":"2023-10-16T14:37:14.160760+0000","last_active":"2023-10-16T14:37:15.376857+0000","last_peered":"2023-10-16T14:37:15.376857+0000","last_clean":"2023-10-16T14:37:15.376857+0000","last_became_active":"2023-10-16T14:37:14.160418+0000","last_became_peered":"2023-10-16T14:37:14.160418+0000","last_unstale":"2023-10-16T14:37:15.376857+0000","last_undegraded":"2023-10-16T14:37:15.376857+0000","last_fullsized":"2023-10-16T14:37:15.376857+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:19.542779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162893+0000","last_change":"2023-10-16T14:37:13.161101+0000","last_active":"2023-10-16T14:37:15.162893+0000","last_peered":"2023-10-16T14:37:15.162893+0000","last_clean":"2023-10-16T14:37:15.162893+0000","last_became_active":"2023-10-16T14:37:13.159488+0000","last_became_peered":"2023-10-16T14:37:13.159488+0000","last_unstale":"2023-10-16T14:37:15.162893+0000","last_undegraded":"2023-10-16T14:37:15.162893+0000","last_fullsized":"2023-10-16T14:37:15.162893+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:40.115179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158206+0000","last_change":"2023-10-16T14:37:14.160037+0000","last_active":"2023-10-16T14:37:15.158206+0000","last_peered":"2023-10-16T14:37:15.158206+0000","last_clean":"2023-10-16T14:37:15.158206+0000","last_became_active":"2023-10-16T14:37:14.159756+0000","last_became_peered":"2023-10-16T14:37:14.159756+0000","last_unstale":"2023-10-16T14:37:15.158206+0000","last_undegraded":"2023-10-16T14:37:15.158206+0000","last_fullsized":"2023-10-16T14:37:15.158206+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:48.079668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158473+0000","last_change":"2023-10-16T14:37:13.153896+0000","last_active":"2023-10-16T14:37:15.158473+0000","last_peered":"2023-10-16T14:37:15.158473+0000","last_clean":"2023-10-16T14:37:15.158473+0000","last_became_active":"2023-10-16T14:37:13.152833+0000","last_became_peered":"2023-10-16T14:37:13.152833+0000","last_unstale":"2023-10-16T14:37:15.158473+0000","last_undegraded":"2023-10-16T14:37:15.158473+0000","last_fullsized":"2023-10-16T14:37:15.158473+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:56.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.165004+0000","last_change":"2023-10-16T14:37:14.167657+0000","last_active":"2023-10-16T14:37:15.165004+0000","last_peered":"2023-10-16T14:37:15.165004+0000","last_clean":"2023-10-16T14:37:15.165004+0000","last_became_active":"2023-10-16T14:37:14.167326+0000","last_became_peered":"2023-10-16T14:37:14.167326+0000","last_unstale":"2023-10-16T14:37:15.165004+0000","last_undegraded":"2023-10-16T14:37:15.165004+0000","last_fullsized":"2023-10-16T14:37:15.165004+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:51.957881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379183+0000","last_change":"2023-10-16T14:37:13.157117+0000","last_active":"2023-10-16T14:37:15.379183+0000","last_peered":"2023-10-16T14:37:15.379183+0000","last_clean":"2023-10-16T14:37:15.379183+0000","last_became_active":"2023-10-16T14:37:13.156496+0000","last_became_peered":"2023-10-16T14:37:13.156496+0000","last_unstale":"2023-10-16T14:37:15.379183+0000","last_undegraded":"2023-10-16T14:37:15.379183+0000","last_fullsized":"2023-10-16T14:37:15.379183+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:08.473419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.161831+0000","last_change":"2023-10-16T14:37:14.162514+0000","last_active":"2023-10-16T14:37:15.161831+0000","last_peered":"2023-10-16T14:37:15.161831+0000","last_clean":"2023-10-16T14:37:15.161831+0000","last_became_active":"2023-10-16T14:37:14.161428+0000","last_became_peered":"2023-10-16T14:37:14.161428+0000","last_unstale":"2023-10-16T14:37:15.161831+0000","last_undegraded":"2023-10-16T14:37:15.161831+0000","last_fullsized":"2023-10-16T14:37:15.161831+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:27.218231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158068+0000","last_change":"2023-10-16T14:37:13.153872+0000","last_active":"2023-10-16T14:37:15.158068+0000","last_peered":"2023-10-16T14:37:15.158068+0000","last_clean":"2023-10-16T14:37:15.158068+0000","last_became_active":"2023-10-16T14:37:13.153613+0000","last_became_peered":"2023-10-16T14:37:13.153613+0000","last_unstale":"2023-10-16T14:37:15.158068+0000","last_undegraded":"2023-10-16T14:37:15.158068+0000","last_fullsized":"2023-10-16T14:37:15.158068+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:56.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175873+0000","last_change":"2023-10-16T14:37:14.166930+0000","last_active":"2023-10-16T14:37:15.175873+0000","last_peered":"2023-10-16T14:37:15.175873+0000","last_clean":"2023-10-16T14:37:15.175873+0000","last_became_active":"2023-10-16T14:37:14.166339+0000","last_became_peered":"2023-10-16T14:37:14.166339+0000","last_unstale":"2023-10-16T14:37:15.175873+0000","last_undegraded":"2023-10-16T14:37:15.175873+0000","last_fullsized":"2023-10-16T14:37:15.175873+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:45.389804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162020+0000","last_change":"2023-10-16T14:37:13.159323+0000","last_active":"2023-10-16T14:37:15.162020+0000","last_peered":"2023-10-16T14:37:15.162020+0000","last_clean":"2023-10-16T14:37:15.162020+0000","last_became_active":"2023-10-16T14:37:13.158383+0000","last_became_peered":"2023-10-16T14:37:13.158383+0000","last_unstale":"2023-10-16T14:37:15.162020+0000","last_undegraded":"2023-10-16T14:37:15.162020+0000","last_fullsized":"2023-10-16T14:37:15.162020+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:17.618256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378837+0000","last_change":"2023-10-16T14:37:14.161380+0000","last_active":"2023-10-16T14:37:15.378837+0000","last_peered":"2023-10-16T14:37:15.378837+0000","last_clean":"2023-10-16T14:37:15.378837+0000","last_became_active":"2023-10-16T14:37:14.160851+0000","last_became_peered":"2023-10-16T14:37:14.160851+0000","last_unstale":"2023-10-16T14:37:15.378837+0000","last_undegraded":"2023-10-16T14:37:15.378837+0000","last_fullsized":"2023-10-16T14:37:15.378837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:39.617735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378550+0000","last_change":"2023-10-16T14:37:13.158945+0000","last_active":"2023-10-16T14:37:15.378550+0000","last_peered":"2023-10-16T14:37:15.378550+0000","last_clean":"2023-10-16T14:37:15.378550+0000","last_became_active":"2023-10-16T14:37:13.158389+0000","last_became_peered":"2023-10-16T14:37:13.158389+0000","last_unstale":"2023-10-16T14:37:15.378550+0000","last_undegraded":"2023-10-16T14:37:15.378550+0000","last_fullsized":"2023-10-16T14:37:15.378550+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:32.920579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157836+0000","last_change":"2023-10-16T14:37:14.167927+0000","last_active":"2023-10-16T14:37:15.157836+0000","last_peered":"2023-10-16T14:37:15.157836+0000","last_clean":"2023-10-16T14:37:15.157836+0000","last_became_active":"2023-10-16T14:37:14.167600+0000","last_became_peered":"2023-10-16T14:37:14.167600+0000","last_unstale":"2023-10-16T14:37:15.157836+0000","last_undegraded":"2023-10-16T14:37:15.157836+0000","last_fullsized":"2023-10-16T14:37:15.157836+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:11.397048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377829+0000","last_change":"2023-10-16T14:37:13.156247+0000","last_active":"2023-10-16T14:37:15.377829+0000","last_peered":"2023-10-16T14:37:15.377829+0000","last_clean":"2023-10-16T14:37:15.377829+0000","last_became_active":"2023-10-16T14:37:13.155946+0000","last_became_peered":"2023-10-16T14:37:13.155946+0000","last_unstale":"2023-10-16T14:37:15.377829+0000","last_undegraded":"2023-10-16T14:37:15.377829+0000","last_fullsized":"2023-10-16T14:37:15.377829+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:48.334064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.161782+0000","last_change":"2023-10-16T14:37:14.160716+0000","last_active":"2023-10-16T14:37:15.161782+0000","last_peered":"2023-10-16T14:37:15.161782+0000","last_clean":"2023-10-16T14:37:15.161782+0000","last_became_active":"2023-10-16T14:37:14.159818+0000","last_became_peered":"2023-10-16T14:37:14.159818+0000","last_unstale":"2023-10-16T14:37:15.161782+0000","last_undegraded":"2023-10-16T14:37:15.161782+0000","last_fullsized":"2023-10-16T14:37:15.161782+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:31.945899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162605+0000","last_change":"2023-10-16T14:37:13.159184+0000","last_active":"2023-10-16T14:37:15.162605+0000","last_peered":"2023-10-16T14:37:15.162605+0000","last_clean":"2023-10-16T14:37:15.162605+0000","last_became_active":"2023-10-16T14:37:13.156882+0000","last_became_peered":"2023-10-16T14:37:13.156882+0000","last_unstale":"2023-10-16T14:37:15.162605+0000","last_undegraded":"2023-10-16T14:37:15.162605+0000","last_fullsized":"2023-10-16T14:37:15.162605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:26.933252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"144'252","reported_seq":788,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-16T14:37:14.136793+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:37:14.136793+0000","last_peered":"2023-10-16T14:37:14.136793+0000","last_clean":"2023-10-16T14:37:14.136793+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:37:14.136793+0000","last_undegraded":"2023-10-16T14:37:14.136793+0000","last_fullsized":"2023-10-16T14:37:14.136793+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575095,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41992,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329848,"statfs":{"total":96636764160,"available":96593764352,"internally_reserved":0,"allocated":2547712,"data_stored":1526567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.0860000000000001,"15min":1.121},"min":{"1min":0.752,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0449999999999999,"15min":1.1200000000000001},"min":{"1min":0.77800000000000002,"5min":0.442,"15min":0.442},"max":{"1min":1.389,"5min":1.891,"15min":1.891},"last":1.155}]},{"osd":1,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.1419999999999999,"15min":1.181},"min":{"1min":0.66000000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":1.04},{"interface":"front","average":{"1min":1.236,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.71599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.627,"5min":2.048,"15min":2.048},"last":0.998}]},{"osd":2,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.2669999999999999,"15min":1.302},"min":{"1min":1.0600000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1839999999999999,"15min":1.2170000000000001},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.1269999999999998,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":1.125}]},{"osd":3,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.2410000000000001,"15min":1.29},"min":{"1min":0.85999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.3280000000000001,"15min":1.3600000000000001},"min":{"1min":0.77700000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.1970000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3400000000000001,"15min":1.359},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.8420000000000001,"5min":2.52,"15min":2.52},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3009999999999999,"15min":1.2609999999999999},"min":{"1min":0.60399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.0299999999999998,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":1.262}]},{"osd":5,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.3819999999999999,"15min":1.3959999999999999},"min":{"1min":0.79800000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.913,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.498,"15min":1.4890000000000001},"min":{"1min":0.84499999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":1.55}]},{"osd":6,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.4379999999999999,"15min":1.4390000000000001},"min":{"1min":0.996,"5min":0.502,"15min":0.502},"max":{"1min":2.2010000000000001,"5min":2.79,"15min":2.79},"last":1.508},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.518,"15min":1.5109999999999999},"min":{"1min":0.89800000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.1360000000000001,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.458}]}]},{"osd":6,"up_from":13,"seq":55834575094,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1941504,"data_stored":935561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1539999999999999,"15min":1.1759999999999999},"min":{"1min":0.68899999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.4239999999999999,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.264},{"interface":"front","average":{"1min":1.228,"5min":1.2709999999999999,"15min":1.2569999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.6919999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":1.5960000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2909999999999999,"15min":1.284},"min":{"1min":0.78800000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.637,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.278,"15min":1.29},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.603,"5min":10.241,"15min":10.241},"last":1.2030000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.141,"15min":1.1990000000000001},"min":{"1min":0.65700000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.4670000000000001,"5min":2.1469999999999998,"15min":2.4940000000000002},"last":1.524},{"interface":"front","average":{"1min":1.28,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.76500000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.647,"5min":5.694,"15min":5.694},"last":1.1419999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3580000000000001,"15min":1.355},"min":{"1min":0.85699999999999998,"5min":0.76300000000000001,"15min":0.71999999999999997},"max":{"1min":1.6930000000000001,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":1.474},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.2609999999999999,"15min":1.3160000000000001},"min":{"1min":0.84599999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.617,"5min":2.2349999999999999,"15min":2.6120000000000001},"last":1.4339999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.413,"15min":1.343},"min":{"1min":0.82499999999999996,"5min":0.54200000000000004,"15min":0.48999999999999999},"max":{"1min":1.7849999999999999,"5min":13.826000000000001,"15min":13.826000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.3859999999999999,"15min":1.3660000000000001},"min":{"1min":1.01,"5min":0.63200000000000001,"15min":0.44},"max":{"1min":1.8380000000000001,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.6850000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.351,"15min":1.3129999999999999},"min":{"1min":0.77600000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.897,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":1.306},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.518,"15min":1.5},"min":{"1min":0.89400000000000002,"5min":0.59599999999999997,"15min":0.56000000000000005},"max":{"1min":1.8959999999999999,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.585,"15min":1.5629999999999999},"min":{"1min":1.1120000000000001,"5min":0.84099999999999997,"15min":0.58299999999999996},"max":{"1min":1.8400000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5349999999999999,"15min":1.536},"min":{"1min":1.004,"5min":0.67500000000000004,"15min":0.54200000000000004},"max":{"1min":1.9810000000000001,"5min":2.1779999999999999,"15min":2.7250000000000001},"last":1.625}]}]},{"osd":5,"up_from":13,"seq":55834575094,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":934061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.079,"15min":1.109},"min":{"1min":0.65000000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3779999999999999,"5min":1.669,"15min":2.2309999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.111,"5min":1.131,"15min":1.171},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.42299999999999999},"max":{"1min":1.569,"5min":1.7729999999999999,"15min":1.8340000000000001},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.099,"15min":1.1559999999999999},"min":{"1min":0.71099999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.5169999999999999,"5min":1.6899999999999999,"15min":1.726},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.218,"15min":1.238},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.60399999999999998},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7809999999999999},"last":1.4319999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.2330000000000001},"min":{"1min":0.75600000000000001,"5min":0.71599999999999997,"15min":0.65400000000000003},"max":{"1min":1.5760000000000001,"5min":1.913,"15min":2.5009999999999999},"last":1.232},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.3069999999999999,"15min":1.345},"min":{"1min":0.94299999999999995,"5min":0.72199999999999998,"15min":0.625},"max":{"1min":1.5569999999999999,"5min":2.0049999999999999,"15min":2.1200000000000001},"last":1.6799999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.22,"15min":1.266},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.45000000000000001},"max":{"1min":1.482,"5min":1.903,"15min":1.903},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2709999999999999,"15min":1.345},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.611,"5min":2.2429999999999999,"15min":2.8029999999999999},"last":1.371}]},{"osd":4,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2949999999999999,"15min":1.3200000000000001},"min":{"1min":0.81000000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6459999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4359999999999999,"15min":1.486},"min":{"1min":0.78000000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.782,"5min":2.1379999999999999,"15min":4.9480000000000004},"last":1.607}]},{"osd":6,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.349,"15min":1.377},"min":{"1min":0.86299999999999999,"5min":0.73499999999999999,"15min":0.47899999999999998},"max":{"1min":1.708,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3520000000000001,"15min":1.4099999999999999},"min":{"1min":0.97099999999999997,"5min":0.77700000000000002,"15min":0.55000000000000004},"max":{"1min":1.6639999999999999,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.5029999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.492,"15min":1.534},"min":{"1min":0.96399999999999997,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":1.78,"5min":2.3399999999999999,"15min":2.641},"last":1.74},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.569,"15min":1.605},"min":{"1min":0.999,"5min":0.97599999999999998,"15min":0.58099999999999996},"max":{"1min":1.9390000000000001,"5min":2.1909999999999998,"15min":2.2010000000000001},"last":2.0150000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575095,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51124,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320716,"statfs":{"total":96636764160,"available":96584413184,"internally_reserved":0,"allocated":1933312,"data_stored":934019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1970000000000001,"15min":1.165},"min":{"1min":0.65500000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.569,"5min":7.9400000000000004,"15min":7.9400000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.71899999999999997,"5min":0.60699999999999998,"15min":0.58999999999999997},"max":{"1min":1.5609999999999999,"5min":1.7090000000000001,"15min":1.8500000000000001},"last":0.84399999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.2529999999999999,"15min":1.2130000000000001},"min":{"1min":0.73299999999999998,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":1.6739999999999999,"5min":16.959,"15min":16.959},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.123,"5min":1.2929999999999999,"15min":1.2549999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.8069999999999999,"5min":12.333,"15min":12.333},"last":0.96999999999999997}]},{"osd":2,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.377,"15min":1.3979999999999999},"min":{"1min":0.78400000000000003,"5min":0.68100000000000005,"15min":0.67900000000000005},"max":{"1min":1.7130000000000001,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":1.323},{"interface":"front","average":{"1min":1.345,"5min":1.3799999999999999,"15min":1.381},"min":{"1min":0.91800000000000004,"5min":0.59199999999999997,"15min":0.51600000000000001},"max":{"1min":1.669,"5min":8.2899999999999991,"15min":8.2899999999999991},"last":1.054}]},{"osd":3,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.224,"15min":1.2969999999999999},"min":{"1min":0.79800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.762,"5min":1.95,"15min":1.95},"last":1.246},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2949999999999999,"15min":1.3080000000000001},"min":{"1min":0.83599999999999997,"5min":0.53900000000000003,"15min":0.437},"max":{"1min":1.5740000000000001,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.361,"15min":1.3160000000000001},"min":{"1min":0.71299999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.9870000000000001,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3859999999999999,"15min":1.351},"min":{"1min":1.026,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.786,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.157}]},{"osd":6,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.329},"min":{"1min":0.92600000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.657,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.5229999999999999,"15min":1.476},"min":{"1min":0.76000000000000001,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8819999999999999,"5min":7.9900000000000002,"15min":7.9900000000000002},"last":1.401}]},{"osd":7,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5109999999999999,"15min":1.4850000000000001},"min":{"1min":0.93200000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.925,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.534,"15min":1.4830000000000001},"min":{"1min":1.0509999999999999,"5min":0.66700000000000004,"15min":0.49399999999999999},"max":{"1min":1.9330000000000001,"5min":3.8090000000000002,"15min":3.8090000000000002},"last":1.3720000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575094,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45552,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326288,"statfs":{"total":96636764160,"available":96590118912,"internally_reserved":0,"allocated":1937408,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":44693961},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.85399999999999998,"15min":0.87},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.28999999999999998},"max":{"1min":1.367,"5min":1.696,"15min":1.696},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.91300000000000003,"15min":1.002},"min":{"1min":0.45800000000000002,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":1.4610000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.0489999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.92200000000000004},"min":{"1min":0.44500000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4430000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.1060000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.314},"max":{"1min":1.585,"5min":3.024,"15min":3.024},"last":1.331}]},{"osd":2,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.157,"15min":1.2789999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.0590000000000002,"5min":3.3420000000000001,"15min":3.3420000000000001},"last":1.149},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2589999999999999,"15min":1.377},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.5},"max":{"1min":1.843,"5min":2.5609999999999999,"15min":2.6299999999999999},"last":1.468}]},{"osd":4,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2150000000000001,"15min":1.248},"min":{"1min":0.79400000000000004,"5min":0.64300000000000002,"15min":0.56699999999999995},"max":{"1min":1.9930000000000001,"5min":2.02,"15min":2.02},"last":1.498},{"interface":"front","average":{"1min":1.347,"5min":1.274,"15min":1.3169999999999999},"min":{"1min":0.76000000000000001,"5min":0.67800000000000005,"15min":0.629},"max":{"1min":2.516,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.536}]},{"osd":5,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.4239999999999999,"15min":1.4990000000000001},"min":{"1min":0.96799999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3090000000000002,"5min":3.206,"15min":3.206},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4159999999999999,"15min":1.4450000000000001},"min":{"1min":0.92600000000000005,"5min":0.754,"15min":0.68799999999999994},"max":{"1min":2.145,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.673}]},{"osd":6,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3149999999999999,"15min":1.3779999999999999},"min":{"1min":0.85999999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.2789999999999999,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.714},{"interface":"front","average":{"1min":1.474,"5min":1.395,"15min":1.468},"min":{"1min":0.84699999999999998,"5min":0.82599999999999996,"15min":0.57999999999999996},"max":{"1min":2.391,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.4239999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.5089999999999999,"15min":1.5580000000000001},"min":{"1min":0.91800000000000004,"5min":0.85299999999999998,"15min":0.69099999999999995},"max":{"1min":3.5649999999999999,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":1.601},{"interface":"front","average":{"1min":1.51,"5min":1.4219999999999999,"15min":1.5109999999999999},"min":{"1min":0.84899999999999998,"5min":0.749,"15min":0.629},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.782}]}]},{"osd":2,"up_from":13,"seq":55834575095,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46968,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324872,"statfs":{"total":96636764160,"available":96588668928,"internally_reserved":0,"allocated":1937408,"data_stored":934565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.88900000000000001,"15min":0.874},"min":{"1min":0.46999999999999997,"5min":0.38,"15min":0.38},"max":{"1min":1.454,"5min":1.835,"15min":1.835},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.92300000000000004,"15min":1.0029999999999999},"min":{"1min":0.436,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":1.3400000000000001,"5min":1.617,"15min":2.3260000000000001},"last":1.034}]},{"osd":1,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.98199999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.35299999999999998},"max":{"1min":1.49,"5min":1.7649999999999999,"15min":3.234},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.115,"5min":1.1180000000000001,"15min":1.161},"min":{"1min":0.624,"5min":0.51200000000000001,"15min":0.42199999999999999},"max":{"1min":1.716,"5min":2.0910000000000002,"15min":3.4740000000000002},"last":1.123}]},{"osd":3,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.087,"15min":1.2070000000000001},"min":{"1min":0.48499999999999999,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":1.6990000000000001,"5min":2.0550000000000002,"15min":2.1640000000000001},"last":1.242},{"interface":"front","average":{"1min":1.137,"5min":1.238,"15min":1.294},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.8169999999999999,"5min":2.2610000000000001,"15min":3.5569999999999999},"last":1.5780000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3939999999999999,"15min":1.425},"min":{"1min":0.80400000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8680000000000001,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.363},{"interface":"front","average":{"1min":1.268,"5min":1.258,"15min":1.337},"min":{"1min":0.76800000000000002,"5min":0.754,"15min":0.72599999999999998},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":3.016},"last":1.6739999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3520000000000001,"15min":1.4039999999999999},"min":{"1min":0.89800000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.913,"5min":2.3170000000000002,"15min":2.4100000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5,"15min":1.5429999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.79200000000000004},"max":{"1min":1.9850000000000001,"5min":2.3239999999999998,"15min":3.3769999999999998},"last":1.46}]},{"osd":6,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.323,"15min":1.4450000000000001},"min":{"1min":0.83999999999999997,"5min":0.69399999999999995,"15min":0.67100000000000004},"max":{"1min":1.8600000000000001,"5min":2.0529999999999999,"15min":3.0760000000000001},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.4430000000000001,"15min":1.5109999999999999},"min":{"1min":0.81899999999999995,"5min":0.71299999999999997,"15min":0.65300000000000002},"max":{"1min":1.8799999999999999,"5min":2.4830000000000001,"15min":2.5920000000000001},"last":1.411}]},{"osd":7,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4219999999999999,"15min":1.5289999999999999},"min":{"1min":0.96599999999999997,"5min":0.70699999999999996,"15min":0.69099999999999995},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":3.1899999999999999},"last":1.869},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.538,"15min":1.6200000000000001},"min":{"1min":0.83699999999999997,"5min":0.81999999999999995,"15min":0.80300000000000005},"max":{"1min":2.0259999999999998,"5min":2.6030000000000002,"15min":2.7519999999999998},"last":1.6299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59436,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312404,"statfs":{"total":96636764160,"available":96575901696,"internally_reserved":0,"allocated":1929216,"data_stored":933949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.90600000000000003,"15min":0.91000000000000003},"min":{"1min":0.56899999999999995,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.28},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.95699999999999996,"15min":1.056},"min":{"1min":0.64000000000000001,"5min":0.40799999999999997,"15min":0.378},"max":{"1min":1.611,"5min":1.827,"15min":1.9370000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.91200000000000003,"15min":0.96799999999999997},"min":{"1min":0.55200000000000005,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.6479999999999999,"5min":1.792,"15min":2.0259999999999998},"last":1.2},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.099,"15min":1.1419999999999999},"min":{"1min":0.59299999999999997,"5min":0.47099999999999997,"15min":0.45800000000000002},"max":{"1min":1.6859999999999999,"5min":1.6919999999999999,"15min":1.756},"last":0.85599999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.1990000000000001,"15min":1.2889999999999999},"min":{"1min":0.67200000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.837,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.995},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.091,"15min":1.1739999999999999},"min":{"1min":0.46899999999999997,"5min":0.45700000000000002,"15min":0.44600000000000001},"max":{"1min":1.587,"5min":1.778,"15min":1.855},"last":2.395}]},{"osd":4,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.198,"15min":1.2629999999999999},"min":{"1min":0.754,"5min":0.71299999999999997,"15min":0.56100000000000005},"max":{"1min":1.5009999999999999,"5min":1.958,"15min":2.069},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.61399999999999999},"max":{"1min":1.7929999999999999,"5min":2.899,"15min":2.899},"last":1.393}]},{"osd":5,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3340000000000001,"15min":1.4079999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.623},"max":{"1min":1.7030000000000001,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.339},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.335,"15min":1.411},"min":{"1min":0.82899999999999996,"5min":0.71199999999999997,"15min":0.59499999999999997},"max":{"1min":1.9059999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.964}]},{"osd":6,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.506,"15min":1.526},"min":{"1min":0.89800000000000002,"5min":0.80000000000000004,"15min":0.53100000000000003},"max":{"1min":2.008,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.403,"15min":1.5049999999999999},"min":{"1min":0.96199999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.798,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.9099999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.522,"15min":1.6000000000000001},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.923,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":2.2080000000000002},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.4410000000000001,"15min":1.5309999999999999},"min":{"1min":0.98399999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.008,"5min":2.2000000000000002,"15min":2.2170000000000001},"last":2.085}]}]},{"osd":0,"up_from":13,"seq":55834575095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37804,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334036,"statfs":{"total":96636764160,"available":96598052864,"internally_reserved":0,"allocated":2523136,"data_stored":1524317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.91800000000000004,"15min":0.93600000000000005},"min":{"1min":0.48699999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.274,"5min":3.871,"15min":3.871},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.119,"15min":1.159},"min":{"1min":0.54500000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7290000000000001,"5min":3.726,"15min":3.726},"last":0.93500000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.1579999999999999,"15min":1.236},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.7589999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1419999999999999,"15min":1.1819999999999999},"min":{"1min":0.60399999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9259999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95699999999999996,"15min":0.99399999999999999},"min":{"1min":0.63200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.3700000000000001,"5min":3.109,"15min":3.109},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.181,"5min":1.1819999999999999,"15min":1.218},"min":{"1min":0.59999999999999998,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":2.04,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":1.1200000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2889999999999999,"15min":1.306},"min":{"1min":0.88900000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.865,"5min":3.4380000000000002,"15min":3.4380000000000002},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.4750000000000001,"15min":1.478},"min":{"1min":0.997,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1829999999999998,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":1.2470000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3899999999999999,"15min":1.419},"min":{"1min":0.97399999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.877,"5min":4.2119999999999997,"15min":4.2119999999999997},"last":1.496},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3939999999999999,"15min":1.3759999999999999},"min":{"1min":0.84099999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.978,"5min":5.7510000000000003,"15min":5.7510000000000003},"last":1.1759999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.589,"15min":1.585},"min":{"1min":1.179,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.0190000000000001,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":1.2},{"interface":"front","average":{"1min":1.512,"5min":1.4850000000000001,"15min":1.502},"min":{"1min":1.018,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.0110000000000001,"5min":3.371,"15min":3.371},"last":1.3859999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5509999999999999,"15min":1.583},"min":{"1min":0.93000000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1920000000000002,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.595},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.579,"15min":1.5600000000000001},"min":{"1min":1.0960000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.113,"5min":5.6189999999999998,"15min":5.6189999999999998},"last":1.081}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:37:18.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:37:18.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:37:18.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:37:19.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.060+0000 7f1fe7d4c640 1 Processor -- start 2023-10-16T14:37:19.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.061+0000 7f1fe7d4c640 1 -- start start 2023-10-16T14:37:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.062+0000 7f1fe7d4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fe00b3070 0x7f1fe00cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.062+0000 7f1fe7d4c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00b3940 con 0x7f1fe00d1b10 2023-10-16T14:37:19.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.062+0000 7f1fe7d4c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00b45e0 con 0x7f1fe00b3e70 2023-10-16T14:37:19.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.062+0000 7f1fe7d4c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00b3b90 con 0x7f1fe00b3070 2023-10-16T14:37:19.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe5ae8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fe00b3070 0x7f1fe00cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe62e9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1fe00d1b10 0x7f1fe00c6f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43844/0 (socket says 172.21.15.29:43844) 2023-10-16T14:37:19.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe62e9640 1 -- 172.21.15.29:0/1842566165 learned_addr learned my addr 172.21.15.29:0/1842566165 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe5ae8640 1 -- 172.21.15.29:0/1842566165 >> v1:172.21.15.170:6789/0 conn(0x7f1fe00b3e70 legacy=0x7f1fe00c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe5ae8640 1 -- 172.21.15.29:0/1842566165 >> v1:172.21.15.29:6789/0 conn(0x7f1fe00d1b10 legacy=0x7f1fe00c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe5ae8640 1 -- 172.21.15.29:0/1842566165 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00d3840 con 0x7f1fe00b3070 2023-10-16T14:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.063+0000 7f1fe4ae6640 1 -- 172.21.15.29:0/1842566165 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56862194 0 0) 0x7f1fe00b3940 con 0x7f1fe00d1b10 2023-10-16T14:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.065+0000 7f1fe5ae8640 1 --2- 172.21.15.29:0/1842566165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cf500 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009790 tx=0x7f1fdc02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25231b84b33b6b31 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.065+0000 7f1fe4ae6640 1 -- 172.21.15.29:0/1842566165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc030800 con 0x7f1fe00b3070 2023-10-16T14:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.065+0000 7f1fe4ae6640 1 -- 172.21.15.29:0/1842566165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc030960 con 0x7f1fe00b3070 2023-10-16T14:37:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.065+0000 7f1fe4ae6640 1 -- 172.21.15.29:0/1842566165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc030cf0 con 0x7f1fe00b3070 2023-10-16T14:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/1842566165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3070 msgr2=0x7f1fe00cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/1842566165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cf500 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009790 tx=0x7f1fdc02b940 comp rx=0 tx=0).stop 2023-10-16T14:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/1842566165 shutdown_connections 2023-10-16T14:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/1842566165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cf500 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/1842566165 >> 172.21.15.29:0/1842566165 conn(0x7f1fe00d2d20 msgr2=0x7f1fe00d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/1842566165 shutdown_connections 2023-10-16T14:37:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.066+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/1842566165 wait complete. 2023-10-16T14:37:19.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.068+0000 7f1fe7d4c640 1 Processor -- start 2023-10-16T14:37:19.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.068+0000 7f1fe7d4c640 1 -- start start 2023-10-16T14:37:19.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe7d4c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe7d4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 0x7f1fe00ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe7d4c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1fe00d1b10 0x7f1fe00ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe7d4c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0160a80 con 0x7f1fe00b3e70 2023-10-16T14:37:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe7d4c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe015a800 con 0x7f1fe00b3070 2023-10-16T14:37:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe7d4c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00ccc70 con 0x7f1fe00d1b10 2023-10-16T14:37:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe52e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 0x7f1fe00ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe52e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 0x7f1fe00ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35270/0 (socket says 172.21.15.29:35270) 2023-10-16T14:37:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.069+0000 7f1fe52e7640 1 -- 172.21.15.29:0/318928803 learned_addr learned my addr 172.21.15.29:0/318928803 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:19.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe62e9640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1fe00d1b10 0x7f1fe00ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe52e7640 1 -- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1fe00d1b10 msgr2=0x7f1fe00ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe5ae8640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe52e7640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1fe00d1b10 0x7f1fe00ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe52e7640 1 -- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1fe00b3070 msgr2=0x7f1fe00cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe52e7640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe52e7640 1 -- 172.21.15.29:0/318928803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0093a0 con 0x7f1fe00b3e70 2023-10-16T14:37:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe5ae8640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fe52e7640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 0x7f1fe00ca470 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c450 tx=0x7f1fd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00052d0 con 0x7f1fe00b3e70 2023-10-16T14:37:19.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0005430 con 0x7f1fe00b3e70 2023-10-16T14:37:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.070+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0003830 con 0x7f1fe00b3e70 2023-10-16T14:37:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.071+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe00ccf50 con 0x7f1fe00b3e70 2023-10-16T14:37:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.071+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe015a490 con 0x7f1fe00b3e70 2023-10-16T14:37:19.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.072+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe004e160 con 0x7f1fe00b3e70 2023-10-16T14:37:19.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.073+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1fd0003990 con 0x7f1fe00b3e70 2023-10-16T14:37:19.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.074+0000 7f1fd67fc640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f1fc406af90 0x7f1fc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.074+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1fd0088df0 con 0x7f1fe00b3e70 2023-10-16T14:37:19.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.075+0000 7f1fe5ae8640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f1fc406af90 0x7f1fc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.076+0000 7f1fe5ae8640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f1fc406af90 0x7f1fc406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc036000 tx=0x7f1fdc03b040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:19.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.078+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1fd0053cd0 con 0x7f1fe00b3e70 2023-10-16T14:37:19.082 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T14:37:19.079+0000 7fb3f679a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:37:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.273+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 --> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1fe00ce4f0 con 0x7f1fc406af90 2023-10-16T14:37:19.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.296+0000 7f1fd67fc640 1 -- 172.21.15.29:0/318928803 <== mgr.7138 v2:172.21.15.170:6832/1985319551 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287487 (secure 0 0 0) 0x7f1fdc03f040 con 0x7f1fc406af90 2023-10-16T14:37:19.303 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:19.305 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T14:37:19.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f1fc406af90 msgr2=0x7f1fc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f1fc406af90 0x7f1fc406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc036000 tx=0x7f1fdc03b040 comp rx=0 tx=0).stop 2023-10-16T14:37:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 msgr2=0x7f1fe00ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 0x7f1fe00ca470 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c450 tx=0x7f1fd000c910 comp rx=0 tx=0).stop 2023-10-16T14:37:19.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 shutdown_connections 2023-10-16T14:37:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f1fc406af90 0x7f1fc406d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe00b3e70 0x7f1fe00ca470 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1fe00d1b10 0x7f1fe00ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 --2- 172.21.15.29:0/318928803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1fe00b3070 0x7f1fe00cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.301+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 >> 172.21.15.29:0/318928803 conn(0x7f1fe00d2d20 msgr2=0x7f1fe00cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:19.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.302+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 shutdown_connections 2023-10-16T14:37:19.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.302+0000 7f1fe7d4c640 1 -- 172.21.15.29:0/318928803 wait complete. 2023-10-16T14:37:19.324 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T14:37:17.519820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":447,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":361512,"kb_used_data":16296,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754613208,"statfs":{"total":773094113280,"available":772723924992,"internally_reserved":0,"allocated":16687104,"data_stored":8657148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":353357369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412879"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376721+0000","last_change":"2023-10-16T14:37:14.164040+0000","last_active":"2023-10-16T14:37:15.376721+0000","last_peered":"2023-10-16T14:37:15.376721+0000","last_clean":"2023-10-16T14:37:15.376721+0000","last_became_active":"2023-10-16T14:37:14.162846+0000","last_became_peered":"2023-10-16T14:37:14.162846+0000","last_unstale":"2023-10-16T14:37:15.376721+0000","last_undegraded":"2023-10-16T14:37:15.376721+0000","last_fullsized":"2023-10-16T14:37:15.376721+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:11.397048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158335+0000","last_change":"2023-10-16T14:37:14.169501+0000","last_active":"2023-10-16T14:37:15.158335+0000","last_peered":"2023-10-16T14:37:15.158335+0000","last_clean":"2023-10-16T14:37:15.158335+0000","last_became_active":"2023-10-16T14:37:14.164348+0000","last_became_peered":"2023-10-16T14:37:14.164348+0000","last_unstale":"2023-10-16T14:37:15.158335+0000","last_undegraded":"2023-10-16T14:37:15.158335+0000","last_fullsized":"2023-10-16T14:37:15.158335+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.696246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378892+0000","last_change":"2023-10-16T14:37:14.160361+0000","last_active":"2023-10-16T14:37:15.378892+0000","last_peered":"2023-10-16T14:37:15.378892+0000","last_clean":"2023-10-16T14:37:15.378892+0000","last_became_active":"2023-10-16T14:37:14.160027+0000","last_became_peered":"2023-10-16T14:37:14.160027+0000","last_unstale":"2023-10-16T14:37:15.378892+0000","last_undegraded":"2023-10-16T14:37:15.378892+0000","last_fullsized":"2023-10-16T14:37:15.378892+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.324235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376659+0000","last_change":"2023-10-16T14:37:14.165967+0000","last_active":"2023-10-16T14:37:15.376659+0000","last_peered":"2023-10-16T14:37:15.376659+0000","last_clean":"2023-10-16T14:37:15.376659+0000","last_became_active":"2023-10-16T14:37:14.165594+0000","last_became_peered":"2023-10-16T14:37:14.165594+0000","last_unstale":"2023-10-16T14:37:15.376659+0000","last_undegraded":"2023-10-16T14:37:15.376659+0000","last_fullsized":"2023-10-16T14:37:15.376659+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:31.262913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162461+0000","last_change":"2023-10-16T14:37:14.163370+0000","last_active":"2023-10-16T14:37:15.162461+0000","last_peered":"2023-10-16T14:37:15.162461+0000","last_clean":"2023-10-16T14:37:15.162461+0000","last_became_active":"2023-10-16T14:37:14.163070+0000","last_became_peered":"2023-10-16T14:37:14.163070+0000","last_unstale":"2023-10-16T14:37:15.162461+0000","last_undegraded":"2023-10-16T14:37:15.162461+0000","last_fullsized":"2023-10-16T14:37:15.162461+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:54.313790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164102+0000","last_change":"2023-10-16T14:37:14.154272+0000","last_active":"2023-10-16T14:37:15.164102+0000","last_peered":"2023-10-16T14:37:15.164102+0000","last_clean":"2023-10-16T14:37:15.164102+0000","last_became_active":"2023-10-16T14:37:14.151846+0000","last_became_peered":"2023-10-16T14:37:14.151846+0000","last_unstale":"2023-10-16T14:37:15.164102+0000","last_undegraded":"2023-10-16T14:37:15.164102+0000","last_fullsized":"2023-10-16T14:37:15.164102+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:07.393228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163398+0000","last_change":"2023-10-16T14:37:14.159875+0000","last_active":"2023-10-16T14:37:15.163398+0000","last_peered":"2023-10-16T14:37:15.163398+0000","last_clean":"2023-10-16T14:37:15.163398+0000","last_became_active":"2023-10-16T14:37:14.159357+0000","last_became_peered":"2023-10-16T14:37:14.159357+0000","last_unstale":"2023-10-16T14:37:15.163398+0000","last_undegraded":"2023-10-16T14:37:15.163398+0000","last_fullsized":"2023-10-16T14:37:15.163398+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:50.466482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378252+0000","last_change":"2023-10-16T14:37:14.169747+0000","last_active":"2023-10-16T14:37:15.378252+0000","last_peered":"2023-10-16T14:37:15.378252+0000","last_clean":"2023-10-16T14:37:15.378252+0000","last_became_active":"2023-10-16T14:37:14.169357+0000","last_became_peered":"2023-10-16T14:37:14.169357+0000","last_unstale":"2023-10-16T14:37:15.378252+0000","last_undegraded":"2023-10-16T14:37:15.378252+0000","last_fullsized":"2023-10-16T14:37:15.378252+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:57.093926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379060+0000","last_change":"2023-10-16T14:37:14.162024+0000","last_active":"2023-10-16T14:37:15.379060+0000","last_peered":"2023-10-16T14:37:15.379060+0000","last_clean":"2023-10-16T14:37:15.379060+0000","last_became_active":"2023-10-16T14:37:14.161748+0000","last_became_peered":"2023-10-16T14:37:14.161748+0000","last_unstale":"2023-10-16T14:37:15.379060+0000","last_undegraded":"2023-10-16T14:37:15.379060+0000","last_fullsized":"2023-10-16T14:37:15.379060+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:54.857137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158518+0000","last_change":"2023-10-16T14:37:14.163076+0000","last_active":"2023-10-16T14:37:15.158518+0000","last_peered":"2023-10-16T14:37:15.158518+0000","last_clean":"2023-10-16T14:37:15.158518+0000","last_became_active":"2023-10-16T14:37:14.162573+0000","last_became_peered":"2023-10-16T14:37:14.162573+0000","last_unstale":"2023-10-16T14:37:15.158518+0000","last_undegraded":"2023-10-16T14:37:15.158518+0000","last_fullsized":"2023-10-16T14:37:15.158518+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:37.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379807+0000","last_change":"2023-10-16T14:37:14.161916+0000","last_active":"2023-10-16T14:37:15.379807+0000","last_peered":"2023-10-16T14:37:15.379807+0000","last_clean":"2023-10-16T14:37:15.379807+0000","last_became_active":"2023-10-16T14:37:14.161600+0000","last_became_peered":"2023-10-16T14:37:14.161600+0000","last_unstale":"2023-10-16T14:37:15.379807+0000","last_undegraded":"2023-10-16T14:37:15.379807+0000","last_fullsized":"2023-10-16T14:37:15.379807+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:45.257306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378563+0000","last_change":"2023-10-16T14:37:14.165627+0000","last_active":"2023-10-16T14:37:15.378563+0000","last_peered":"2023-10-16T14:37:15.378563+0000","last_clean":"2023-10-16T14:37:15.378563+0000","last_became_active":"2023-10-16T14:37:14.165337+0000","last_became_peered":"2023-10-16T14:37:14.165337+0000","last_unstale":"2023-10-16T14:37:15.378563+0000","last_undegraded":"2023-10-16T14:37:15.378563+0000","last_fullsized":"2023-10-16T14:37:15.378563+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:00.151071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376998+0000","last_change":"2023-10-16T14:37:14.159699+0000","last_active":"2023-10-16T14:37:15.376998+0000","last_peered":"2023-10-16T14:37:15.376998+0000","last_clean":"2023-10-16T14:37:15.376998+0000","last_became_active":"2023-10-16T14:37:14.159391+0000","last_became_peered":"2023-10-16T14:37:14.159391+0000","last_unstale":"2023-10-16T14:37:15.376998+0000","last_undegraded":"2023-10-16T14:37:15.376998+0000","last_fullsized":"2023-10-16T14:37:15.376998+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:34.926136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158019+0000","last_change":"2023-10-16T14:37:14.158520+0000","last_active":"2023-10-16T14:37:15.158019+0000","last_peered":"2023-10-16T14:37:15.158019+0000","last_clean":"2023-10-16T14:37:15.158019+0000","last_became_active":"2023-10-16T14:37:14.158197+0000","last_became_peered":"2023-10-16T14:37:14.158197+0000","last_unstale":"2023-10-16T14:37:15.158019+0000","last_undegraded":"2023-10-16T14:37:15.158019+0000","last_fullsized":"2023-10-16T14:37:15.158019+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:19.542779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.375603+0000","last_change":"2023-10-16T14:37:14.173364+0000","last_active":"2023-10-16T14:37:15.375603+0000","last_peered":"2023-10-16T14:37:15.375603+0000","last_clean":"2023-10-16T14:37:15.375603+0000","last_became_active":"2023-10-16T14:37:14.172894+0000","last_became_peered":"2023-10-16T14:37:14.172894+0000","last_unstale":"2023-10-16T14:37:15.375603+0000","last_undegraded":"2023-10-16T14:37:15.375603+0000","last_fullsized":"2023-10-16T14:37:15.375603+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:12.849027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378711+0000","last_change":"2023-10-16T14:37:14.172943+0000","last_active":"2023-10-16T14:37:15.378711+0000","last_peered":"2023-10-16T14:37:15.378711+0000","last_clean":"2023-10-16T14:37:15.378711+0000","last_became_active":"2023-10-16T14:37:14.172645+0000","last_became_peered":"2023-10-16T14:37:14.172645+0000","last_unstale":"2023-10-16T14:37:15.378711+0000","last_undegraded":"2023-10-16T14:37:15.378711+0000","last_fullsized":"2023-10-16T14:37:15.378711+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:45.747808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164881+0000","last_change":"2023-10-16T14:37:14.161475+0000","last_active":"2023-10-16T14:37:15.164881+0000","last_peered":"2023-10-16T14:37:15.164881+0000","last_clean":"2023-10-16T14:37:15.164881+0000","last_became_active":"2023-10-16T14:37:14.160829+0000","last_became_peered":"2023-10-16T14:37:14.160829+0000","last_unstale":"2023-10-16T14:37:15.164881+0000","last_undegraded":"2023-10-16T14:37:15.164881+0000","last_fullsized":"2023-10-16T14:37:15.164881+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:59.397938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379292+0000","last_change":"2023-10-16T14:37:14.162528+0000","last_active":"2023-10-16T14:37:15.379292+0000","last_peered":"2023-10-16T14:37:15.379292+0000","last_clean":"2023-10-16T14:37:15.379292+0000","last_became_active":"2023-10-16T14:37:14.162271+0000","last_became_peered":"2023-10-16T14:37:14.162271+0000","last_unstale":"2023-10-16T14:37:15.379292+0000","last_undegraded":"2023-10-16T14:37:15.379292+0000","last_fullsized":"2023-10-16T14:37:15.379292+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:27.947212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380157+0000","last_change":"2023-10-16T14:37:14.150190+0000","last_active":"2023-10-16T14:37:15.380157+0000","last_peered":"2023-10-16T14:37:15.380157+0000","last_clean":"2023-10-16T14:37:15.380157+0000","last_became_active":"2023-10-16T14:37:14.149360+0000","last_became_peered":"2023-10-16T14:37:14.149360+0000","last_unstale":"2023-10-16T14:37:15.380157+0000","last_undegraded":"2023-10-16T14:37:15.380157+0000","last_fullsized":"2023-10-16T14:37:15.380157+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:25.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379937+0000","last_change":"2023-10-16T14:37:14.164957+0000","last_active":"2023-10-16T14:37:15.379937+0000","last_peered":"2023-10-16T14:37:15.379937+0000","last_clean":"2023-10-16T14:37:15.379937+0000","last_became_active":"2023-10-16T14:37:14.164649+0000","last_became_peered":"2023-10-16T14:37:14.164649+0000","last_unstale":"2023-10-16T14:37:15.379937+0000","last_undegraded":"2023-10-16T14:37:15.379937+0000","last_fullsized":"2023-10-16T14:37:15.379937+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:20.616678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162349+0000","last_change":"2023-10-16T14:37:14.164718+0000","last_active":"2023-10-16T14:37:15.162349+0000","last_peered":"2023-10-16T14:37:15.162349+0000","last_clean":"2023-10-16T14:37:15.162349+0000","last_became_active":"2023-10-16T14:37:14.163197+0000","last_became_peered":"2023-10-16T14:37:14.163197+0000","last_unstale":"2023-10-16T14:37:15.162349+0000","last_undegraded":"2023-10-16T14:37:15.162349+0000","last_fullsized":"2023-10-16T14:37:15.162349+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:35.111846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378568+0000","last_change":"2023-10-16T14:37:14.164657+0000","last_active":"2023-10-16T14:37:15.378568+0000","last_peered":"2023-10-16T14:37:15.378568+0000","last_clean":"2023-10-16T14:37:15.378568+0000","last_became_active":"2023-10-16T14:37:14.163329+0000","last_became_peered":"2023-10-16T14:37:14.163329+0000","last_unstale":"2023-10-16T14:37:15.378568+0000","last_undegraded":"2023-10-16T14:37:15.378568+0000","last_fullsized":"2023-10-16T14:37:15.378568+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:05.110417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378505+0000","last_change":"2023-10-16T14:37:14.162269+0000","last_active":"2023-10-16T14:37:15.378505+0000","last_peered":"2023-10-16T14:37:15.378505+0000","last_clean":"2023-10-16T14:37:15.378505+0000","last_became_active":"2023-10-16T14:37:14.161780+0000","last_became_peered":"2023-10-16T14:37:14.161780+0000","last_unstale":"2023-10-16T14:37:15.378505+0000","last_undegraded":"2023-10-16T14:37:15.378505+0000","last_fullsized":"2023-10-16T14:37:15.378505+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:13.308843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377650+0000","last_change":"2023-10-16T14:37:14.155853+0000","last_active":"2023-10-16T14:37:15.377650+0000","last_peered":"2023-10-16T14:37:15.377650+0000","last_clean":"2023-10-16T14:37:15.377650+0000","last_became_active":"2023-10-16T14:37:14.155552+0000","last_became_peered":"2023-10-16T14:37:14.155552+0000","last_unstale":"2023-10-16T14:37:15.377650+0000","last_undegraded":"2023-10-16T14:37:15.377650+0000","last_fullsized":"2023-10-16T14:37:15.377650+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:57.197725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158723+0000","last_change":"2023-10-16T14:37:14.167518+0000","last_active":"2023-10-16T14:37:15.158723+0000","last_peered":"2023-10-16T14:37:15.158723+0000","last_clean":"2023-10-16T14:37:15.158723+0000","last_became_active":"2023-10-16T14:37:14.167186+0000","last_became_peered":"2023-10-16T14:37:14.167186+0000","last_unstale":"2023-10-16T14:37:15.158723+0000","last_undegraded":"2023-10-16T14:37:15.158723+0000","last_fullsized":"2023-10-16T14:37:15.158723+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.348025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379789+0000","last_change":"2023-10-16T14:37:14.174870+0000","last_active":"2023-10-16T14:37:15.379789+0000","last_peered":"2023-10-16T14:37:15.379789+0000","last_clean":"2023-10-16T14:37:15.379789+0000","last_became_active":"2023-10-16T14:37:14.174602+0000","last_became_peered":"2023-10-16T14:37:14.174602+0000","last_unstale":"2023-10-16T14:37:15.379789+0000","last_undegraded":"2023-10-16T14:37:15.379789+0000","last_fullsized":"2023-10-16T14:37:15.379789+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:05.422010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377408+0000","last_change":"2023-10-16T14:37:14.158672+0000","last_active":"2023-10-16T14:37:15.377408+0000","last_peered":"2023-10-16T14:37:15.377408+0000","last_clean":"2023-10-16T14:37:15.377408+0000","last_became_active":"2023-10-16T14:37:14.158128+0000","last_became_peered":"2023-10-16T14:37:14.158128+0000","last_unstale":"2023-10-16T14:37:15.377408+0000","last_undegraded":"2023-10-16T14:37:15.377408+0000","last_fullsized":"2023-10-16T14:37:15.377408+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:48.079668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175535+0000","last_change":"2023-10-16T14:37:14.163583+0000","last_active":"2023-10-16T14:37:15.175535+0000","last_peered":"2023-10-16T14:37:15.175535+0000","last_clean":"2023-10-16T14:37:15.175535+0000","last_became_active":"2023-10-16T14:37:14.162784+0000","last_became_peered":"2023-10-16T14:37:14.162784+0000","last_unstale":"2023-10-16T14:37:15.175535+0000","last_undegraded":"2023-10-16T14:37:15.175535+0000","last_fullsized":"2023-10-16T14:37:15.175535+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.565036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376688+0000","last_change":"2023-10-16T14:37:14.167823+0000","last_active":"2023-10-16T14:37:15.376688+0000","last_peered":"2023-10-16T14:37:15.376688+0000","last_clean":"2023-10-16T14:37:15.376688+0000","last_became_active":"2023-10-16T14:37:14.167138+0000","last_became_peered":"2023-10-16T14:37:14.167138+0000","last_unstale":"2023-10-16T14:37:15.376688+0000","last_undegraded":"2023-10-16T14:37:15.376688+0000","last_fullsized":"2023-10-16T14:37:15.376688+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:04.709961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.375804+0000","last_change":"2023-10-16T14:37:14.170081+0000","last_active":"2023-10-16T14:37:15.375804+0000","last_peered":"2023-10-16T14:37:15.375804+0000","last_clean":"2023-10-16T14:37:15.375804+0000","last_became_active":"2023-10-16T14:37:14.169211+0000","last_became_peered":"2023-10-16T14:37:14.169211+0000","last_unstale":"2023-10-16T14:37:15.375804+0000","last_undegraded":"2023-10-16T14:37:15.375804+0000","last_fullsized":"2023-10-16T14:37:15.375804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:54.857137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162360+0000","last_change":"2023-10-16T14:37:14.165925+0000","last_active":"2023-10-16T14:37:15.162360+0000","last_peered":"2023-10-16T14:37:15.162360+0000","last_clean":"2023-10-16T14:37:15.162360+0000","last_became_active":"2023-10-16T14:37:14.164385+0000","last_became_peered":"2023-10-16T14:37:14.164385+0000","last_unstale":"2023-10-16T14:37:15.162360+0000","last_undegraded":"2023-10-16T14:37:15.162360+0000","last_fullsized":"2023-10-16T14:37:15.162360+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:06.826826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162718+0000","last_change":"2023-10-16T14:37:14.167773+0000","last_active":"2023-10-16T14:37:15.162718+0000","last_peered":"2023-10-16T14:37:15.162718+0000","last_clean":"2023-10-16T14:37:15.162718+0000","last_became_active":"2023-10-16T14:37:14.167452+0000","last_became_peered":"2023-10-16T14:37:14.167452+0000","last_unstale":"2023-10-16T14:37:15.162718+0000","last_undegraded":"2023-10-16T14:37:15.162718+0000","last_fullsized":"2023-10-16T14:37:15.162718+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:21.327054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380163+0000","last_change":"2023-10-16T14:37:14.159713+0000","last_active":"2023-10-16T14:37:15.380163+0000","last_peered":"2023-10-16T14:37:15.380163+0000","last_clean":"2023-10-16T14:37:15.380163+0000","last_became_active":"2023-10-16T14:37:14.159400+0000","last_became_peered":"2023-10-16T14:37:14.159400+0000","last_unstale":"2023-10-16T14:37:15.380163+0000","last_undegraded":"2023-10-16T14:37:15.380163+0000","last_fullsized":"2023-10-16T14:37:15.380163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:27.218231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"154'4","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163361+0000","last_change":"2023-10-16T14:37:13.159947+0000","last_active":"2023-10-16T14:37:15.163361+0000","last_peered":"2023-10-16T14:37:15.163361+0000","last_clean":"2023-10-16T14:37:15.163361+0000","last_became_active":"2023-10-16T14:37:13.158935+0000","last_became_peered":"2023-10-16T14:37:13.158935+0000","last_unstale":"2023-10-16T14:37:15.163361+0000","last_undegraded":"2023-10-16T14:37:15.163361+0000","last_fullsized":"2023-10-16T14:37:15.163361+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:46.628347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379282+0000","last_change":"2023-10-16T14:37:14.172137+0000","last_active":"2023-10-16T14:37:15.379282+0000","last_peered":"2023-10-16T14:37:15.379282+0000","last_clean":"2023-10-16T14:37:15.379282+0000","last_became_active":"2023-10-16T14:37:14.171809+0000","last_became_peered":"2023-10-16T14:37:14.171809+0000","last_unstale":"2023-10-16T14:37:15.379282+0000","last_undegraded":"2023-10-16T14:37:15.379282+0000","last_fullsized":"2023-10-16T14:37:15.379282+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:50.473704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175765+0000","last_change":"2023-10-16T14:37:13.157211+0000","last_active":"2023-10-16T14:37:15.175765+0000","last_peered":"2023-10-16T14:37:15.175765+0000","last_clean":"2023-10-16T14:37:15.175765+0000","last_became_active":"2023-10-16T14:37:13.156941+0000","last_became_peered":"2023-10-16T14:37:13.156941+0000","last_unstale":"2023-10-16T14:37:15.175765+0000","last_undegraded":"2023-10-16T14:37:15.175765+0000","last_fullsized":"2023-10-16T14:37:15.175765+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:08.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163223+0000","last_change":"2023-10-16T14:37:14.155082+0000","last_active":"2023-10-16T14:37:15.163223+0000","last_peered":"2023-10-16T14:37:15.163223+0000","last_clean":"2023-10-16T14:37:15.163223+0000","last_became_active":"2023-10-16T14:37:14.154796+0000","last_became_peered":"2023-10-16T14:37:14.154796+0000","last_unstale":"2023-10-16T14:37:15.163223+0000","last_undegraded":"2023-10-16T14:37:15.163223+0000","last_fullsized":"2023-10-16T14:37:15.163223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:50.620680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380235+0000","last_change":"2023-10-16T14:37:13.154326+0000","last_active":"2023-10-16T14:37:15.380235+0000","last_peered":"2023-10-16T14:37:15.380235+0000","last_clean":"2023-10-16T14:37:15.380235+0000","last_became_active":"2023-10-16T14:37:13.154006+0000","last_became_peered":"2023-10-16T14:37:13.154006+0000","last_unstale":"2023-10-16T14:37:15.380235+0000","last_undegraded":"2023-10-16T14:37:15.380235+0000","last_fullsized":"2023-10-16T14:37:15.380235+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:57:17.168445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162684+0000","last_change":"2023-10-16T14:37:14.158133+0000","last_active":"2023-10-16T14:37:15.162684+0000","last_peered":"2023-10-16T14:37:15.162684+0000","last_clean":"2023-10-16T14:37:15.162684+0000","last_became_active":"2023-10-16T14:37:14.157848+0000","last_became_peered":"2023-10-16T14:37:14.157848+0000","last_unstale":"2023-10-16T14:37:15.162684+0000","last_undegraded":"2023-10-16T14:37:15.162684+0000","last_fullsized":"2023-10-16T14:37:15.162684+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:48:29.049308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158238+0000","last_change":"2023-10-16T14:37:13.150181+0000","last_active":"2023-10-16T14:37:15.158238+0000","last_peered":"2023-10-16T14:37:15.158238+0000","last_clean":"2023-10-16T14:37:15.158238+0000","last_became_active":"2023-10-16T14:37:13.149921+0000","last_became_peered":"2023-10-16T14:37:13.149921+0000","last_unstale":"2023-10-16T14:37:15.158238+0000","last_undegraded":"2023-10-16T14:37:15.158238+0000","last_fullsized":"2023-10-16T14:37:15.158238+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:23.905986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379139+0000","last_change":"2023-10-16T14:37:14.157362+0000","last_active":"2023-10-16T14:37:15.379139+0000","last_peered":"2023-10-16T14:37:15.379139+0000","last_clean":"2023-10-16T14:37:15.379139+0000","last_became_active":"2023-10-16T14:37:14.157036+0000","last_became_peered":"2023-10-16T14:37:14.157036+0000","last_unstale":"2023-10-16T14:37:15.379139+0000","last_undegraded":"2023-10-16T14:37:15.379139+0000","last_fullsized":"2023-10-16T14:37:15.379139+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:18.632217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157985+0000","last_change":"2023-10-16T14:37:13.152419+0000","last_active":"2023-10-16T14:37:15.157985+0000","last_peered":"2023-10-16T14:37:15.157985+0000","last_clean":"2023-10-16T14:37:15.157985+0000","last_became_active":"2023-10-16T14:37:13.152138+0000","last_became_peered":"2023-10-16T14:37:13.152138+0000","last_unstale":"2023-10-16T14:37:15.157985+0000","last_undegraded":"2023-10-16T14:37:15.157985+0000","last_fullsized":"2023-10-16T14:37:15.157985+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:40.057057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164245+0000","last_change":"2023-10-16T14:37:14.156987+0000","last_active":"2023-10-16T14:37:15.164245+0000","last_peered":"2023-10-16T14:37:15.164245+0000","last_clean":"2023-10-16T14:37:15.164245+0000","last_became_active":"2023-10-16T14:37:14.156684+0000","last_became_peered":"2023-10-16T14:37:14.156684+0000","last_unstale":"2023-10-16T14:37:15.164245+0000","last_undegraded":"2023-10-16T14:37:15.164245+0000","last_fullsized":"2023-10-16T14:37:15.164245+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:19.806281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379008+0000","last_change":"2023-10-16T14:37:13.145105+0000","last_active":"2023-10-16T14:37:15.379008+0000","last_peered":"2023-10-16T14:37:15.379008+0000","last_clean":"2023-10-16T14:37:15.379008+0000","last_became_active":"2023-10-16T14:37:13.144824+0000","last_became_peered":"2023-10-16T14:37:13.144824+0000","last_unstale":"2023-10-16T14:37:15.379008+0000","last_undegraded":"2023-10-16T14:37:15.379008+0000","last_fullsized":"2023-10-16T14:37:15.379008+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:40.115179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378334+0000","last_change":"2023-10-16T14:37:14.169627+0000","last_active":"2023-10-16T14:37:15.378334+0000","last_peered":"2023-10-16T14:37:15.378334+0000","last_clean":"2023-10-16T14:37:15.378334+0000","last_became_active":"2023-10-16T14:37:14.168342+0000","last_became_peered":"2023-10-16T14:37:14.168342+0000","last_unstale":"2023-10-16T14:37:15.378334+0000","last_undegraded":"2023-10-16T14:37:15.378334+0000","last_fullsized":"2023-10-16T14:37:15.378334+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:24.919946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163637+0000","last_change":"2023-10-16T14:37:13.156656+0000","last_active":"2023-10-16T14:37:15.163637+0000","last_peered":"2023-10-16T14:37:15.163637+0000","last_clean":"2023-10-16T14:37:15.163637+0000","last_became_active":"2023-10-16T14:37:13.156323+0000","last_became_peered":"2023-10-16T14:37:13.156323+0000","last_unstale":"2023-10-16T14:37:15.163637+0000","last_undegraded":"2023-10-16T14:37:15.163637+0000","last_fullsized":"2023-10-16T14:37:15.163637+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.551075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157780+0000","last_change":"2023-10-16T14:37:14.163939+0000","last_active":"2023-10-16T14:37:15.157780+0000","last_peered":"2023-10-16T14:37:15.157780+0000","last_clean":"2023-10-16T14:37:15.157780+0000","last_became_active":"2023-10-16T14:37:14.163415+0000","last_became_peered":"2023-10-16T14:37:14.163415+0000","last_unstale":"2023-10-16T14:37:15.157780+0000","last_undegraded":"2023-10-16T14:37:15.157780+0000","last_fullsized":"2023-10-16T14:37:15.157780+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:04.709961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157887+0000","last_change":"2023-10-16T14:37:13.152494+0000","last_active":"2023-10-16T14:37:15.157887+0000","last_peered":"2023-10-16T14:37:15.157887+0000","last_clean":"2023-10-16T14:37:15.157887+0000","last_became_active":"2023-10-16T14:37:13.152207+0000","last_became_peered":"2023-10-16T14:37:13.152207+0000","last_unstale":"2023-10-16T14:37:15.157887+0000","last_undegraded":"2023-10-16T14:37:15.157887+0000","last_fullsized":"2023-10-16T14:37:15.157887+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:33.912175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377408+0000","last_change":"2023-10-16T14:37:14.161603+0000","last_active":"2023-10-16T14:37:15.377408+0000","last_peered":"2023-10-16T14:37:15.377408+0000","last_clean":"2023-10-16T14:37:15.377408+0000","last_became_active":"2023-10-16T14:37:14.161293+0000","last_became_peered":"2023-10-16T14:37:14.161293+0000","last_unstale":"2023-10-16T14:37:15.377408+0000","last_undegraded":"2023-10-16T14:37:15.377408+0000","last_fullsized":"2023-10-16T14:37:15.377408+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.696246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.176368+0000","last_change":"2023-10-16T14:37:13.152057+0000","last_active":"2023-10-16T14:37:15.176368+0000","last_peered":"2023-10-16T14:37:15.176368+0000","last_clean":"2023-10-16T14:37:15.176368+0000","last_became_active":"2023-10-16T14:37:13.151714+0000","last_became_peered":"2023-10-16T14:37:13.151714+0000","last_unstale":"2023-10-16T14:37:15.176368+0000","last_undegraded":"2023-10-16T14:37:15.176368+0000","last_fullsized":"2023-10-16T14:37:15.176368+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:45:00.542986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175995+0000","last_change":"2023-10-16T14:37:14.162024+0000","last_active":"2023-10-16T14:37:15.175995+0000","last_peered":"2023-10-16T14:37:15.175995+0000","last_clean":"2023-10-16T14:37:15.175995+0000","last_became_active":"2023-10-16T14:37:14.161727+0000","last_became_peered":"2023-10-16T14:37:14.161727+0000","last_unstale":"2023-10-16T14:37:15.175995+0000","last_undegraded":"2023-10-16T14:37:15.175995+0000","last_fullsized":"2023-10-16T14:37:15.175995+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:29.932025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380395+0000","last_change":"2023-10-16T14:37:13.158699+0000","last_active":"2023-10-16T14:37:15.380395+0000","last_peered":"2023-10-16T14:37:15.380395+0000","last_clean":"2023-10-16T14:37:15.380395+0000","last_became_active":"2023-10-16T14:37:13.158077+0000","last_became_peered":"2023-10-16T14:37:13.158077+0000","last_unstale":"2023-10-16T14:37:15.380395+0000","last_undegraded":"2023-10-16T14:37:15.380395+0000","last_fullsized":"2023-10-16T14:37:15.380395+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:09.048184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163600+0000","last_change":"2023-10-16T14:37:14.160997+0000","last_active":"2023-10-16T14:37:15.163600+0000","last_peered":"2023-10-16T14:37:15.163600+0000","last_clean":"2023-10-16T14:37:15.163600+0000","last_became_active":"2023-10-16T14:37:14.160048+0000","last_became_peered":"2023-10-16T14:37:14.160048+0000","last_unstale":"2023-10-16T14:37:15.163600+0000","last_undegraded":"2023-10-16T14:37:15.163600+0000","last_fullsized":"2023-10-16T14:37:15.163600+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:31:25.508509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"154'3","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175644+0000","last_change":"2023-10-16T14:37:13.153804+0000","last_active":"2023-10-16T14:37:15.175644+0000","last_peered":"2023-10-16T14:37:15.175644+0000","last_clean":"2023-10-16T14:37:15.175644+0000","last_became_active":"2023-10-16T14:37:13.153539+0000","last_became_peered":"2023-10-16T14:37:13.153539+0000","last_unstale":"2023-10-16T14:37:15.175644+0000","last_undegraded":"2023-10-16T14:37:15.175644+0000","last_fullsized":"2023-10-16T14:37:15.175644+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:28.195462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.163047+0000","last_change":"2023-10-16T14:37:14.165061+0000","last_active":"2023-10-16T14:37:15.163047+0000","last_peered":"2023-10-16T14:37:15.163047+0000","last_clean":"2023-10-16T14:37:15.163047+0000","last_became_active":"2023-10-16T14:37:14.162497+0000","last_became_peered":"2023-10-16T14:37:14.162497+0000","last_unstale":"2023-10-16T14:37:15.163047+0000","last_undegraded":"2023-10-16T14:37:15.163047+0000","last_fullsized":"2023-10-16T14:37:15.163047+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:02.539156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"154'6","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378521+0000","last_change":"2023-10-16T14:37:13.154199+0000","last_active":"2023-10-16T14:37:15.378521+0000","last_peered":"2023-10-16T14:37:15.378521+0000","last_clean":"2023-10-16T14:37:15.378521+0000","last_became_active":"2023-10-16T14:37:13.153918+0000","last_became_peered":"2023-10-16T14:37:13.153918+0000","last_unstale":"2023-10-16T14:37:15.378521+0000","last_undegraded":"2023-10-16T14:37:15.378521+0000","last_fullsized":"2023-10-16T14:37:15.378521+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:49.808976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377022+0000","last_change":"2023-10-16T14:37:14.171955+0000","last_active":"2023-10-16T14:37:15.377022+0000","last_peered":"2023-10-16T14:37:15.377022+0000","last_clean":"2023-10-16T14:37:15.377022+0000","last_became_active":"2023-10-16T14:37:14.171689+0000","last_became_peered":"2023-10-16T14:37:14.171689+0000","last_unstale":"2023-10-16T14:37:15.377022+0000","last_undegraded":"2023-10-16T14:37:15.377022+0000","last_fullsized":"2023-10-16T14:37:15.377022+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:24.855910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.375900+0000","last_change":"2023-10-16T14:37:13.150557+0000","last_active":"2023-10-16T14:37:15.375900+0000","last_peered":"2023-10-16T14:37:15.375900+0000","last_clean":"2023-10-16T14:37:15.375900+0000","last_became_active":"2023-10-16T14:37:13.149432+0000","last_became_peered":"2023-10-16T14:37:13.149432+0000","last_unstale":"2023-10-16T14:37:15.375900+0000","last_undegraded":"2023-10-16T14:37:15.375900+0000","last_fullsized":"2023-10-16T14:37:15.375900+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:59.137110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157763+0000","last_change":"2023-10-16T14:37:14.168758+0000","last_active":"2023-10-16T14:37:15.157763+0000","last_peered":"2023-10-16T14:37:15.157763+0000","last_clean":"2023-10-16T14:37:15.157763+0000","last_became_active":"2023-10-16T14:37:14.168458+0000","last_became_peered":"2023-10-16T14:37:14.168458+0000","last_unstale":"2023-10-16T14:37:15.157763+0000","last_undegraded":"2023-10-16T14:37:15.157763+0000","last_fullsized":"2023-10-16T14:37:15.157763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:31.262913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157864+0000","last_change":"2023-10-16T14:37:13.154504+0000","last_active":"2023-10-16T14:37:15.157864+0000","last_peered":"2023-10-16T14:37:15.157864+0000","last_clean":"2023-10-16T14:37:15.157864+0000","last_became_active":"2023-10-16T14:37:13.154246+0000","last_became_peered":"2023-10-16T14:37:13.154246+0000","last_unstale":"2023-10-16T14:37:15.157864+0000","last_undegraded":"2023-10-16T14:37:15.157864+0000","last_fullsized":"2023-10-16T14:37:15.157864+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:05.706662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378332+0000","last_change":"2023-10-16T14:37:14.164130+0000","last_active":"2023-10-16T14:37:15.378332+0000","last_peered":"2023-10-16T14:37:15.378332+0000","last_clean":"2023-10-16T14:37:15.378332+0000","last_became_active":"2023-10-16T14:37:14.163863+0000","last_became_peered":"2023-10-16T14:37:14.163863+0000","last_unstale":"2023-10-16T14:37:15.378332+0000","last_undegraded":"2023-10-16T14:37:15.378332+0000","last_fullsized":"2023-10-16T14:37:15.378332+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:29.639359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"154'8","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378619+0000","last_change":"2023-10-16T14:37:13.156769+0000","last_active":"2023-10-16T14:37:15.378619+0000","last_peered":"2023-10-16T14:37:15.378619+0000","last_clean":"2023-10-16T14:37:15.378619+0000","last_became_active":"2023-10-16T14:37:13.156444+0000","last_became_peered":"2023-10-16T14:37:13.156444+0000","last_unstale":"2023-10-16T14:37:15.378619+0000","last_undegraded":"2023-10-16T14:37:15.378619+0000","last_fullsized":"2023-10-16T14:37:15.378619+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:26.635254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378941+0000","last_change":"2023-10-16T14:37:14.172468+0000","last_active":"2023-10-16T14:37:15.378941+0000","last_peered":"2023-10-16T14:37:15.378941+0000","last_clean":"2023-10-16T14:37:15.378941+0000","last_became_active":"2023-10-16T14:37:14.172178+0000","last_became_peered":"2023-10-16T14:37:14.172178+0000","last_unstale":"2023-10-16T14:37:15.378941+0000","last_undegraded":"2023-10-16T14:37:15.378941+0000","last_fullsized":"2023-10-16T14:37:15.378941+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:12.849027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.176418+0000","last_change":"2023-10-16T14:37:13.153936+0000","last_active":"2023-10-16T14:37:15.176418+0000","last_peered":"2023-10-16T14:37:15.176418+0000","last_clean":"2023-10-16T14:37:15.176418+0000","last_became_active":"2023-10-16T14:37:13.153629+0000","last_became_peered":"2023-10-16T14:37:13.153629+0000","last_unstale":"2023-10-16T14:37:15.176418+0000","last_undegraded":"2023-10-16T14:37:15.176418+0000","last_fullsized":"2023-10-16T14:37:15.176418+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:14.613465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.176119+0000","last_change":"2023-10-16T14:37:14.164956+0000","last_active":"2023-10-16T14:37:15.176119+0000","last_peered":"2023-10-16T14:37:15.176119+0000","last_clean":"2023-10-16T14:37:15.176119+0000","last_became_active":"2023-10-16T14:37:14.164351+0000","last_became_peered":"2023-10-16T14:37:14.164351+0000","last_unstale":"2023-10-16T14:37:15.176119+0000","last_undegraded":"2023-10-16T14:37:15.176119+0000","last_fullsized":"2023-10-16T14:37:15.176119+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:21.327054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380505+0000","last_change":"2023-10-16T14:37:13.159746+0000","last_active":"2023-10-16T14:37:15.380505+0000","last_peered":"2023-10-16T14:37:15.380505+0000","last_clean":"2023-10-16T14:37:15.380505+0000","last_became_active":"2023-10-16T14:37:13.159395+0000","last_became_peered":"2023-10-16T14:37:13.159395+0000","last_unstale":"2023-10-16T14:37:15.380505+0000","last_undegraded":"2023-10-16T14:37:15.380505+0000","last_fullsized":"2023-10-16T14:37:15.380505+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:10.570994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.380012+0000","last_change":"2023-10-16T14:37:14.173360+0000","last_active":"2023-10-16T14:37:15.380012+0000","last_peered":"2023-10-16T14:37:15.380012+0000","last_clean":"2023-10-16T14:37:15.380012+0000","last_became_active":"2023-10-16T14:37:14.172893+0000","last_became_peered":"2023-10-16T14:37:14.172893+0000","last_unstale":"2023-10-16T14:37:15.380012+0000","last_undegraded":"2023-10-16T14:37:15.380012+0000","last_fullsized":"2023-10-16T14:37:15.380012+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:31.945899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"154'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164779+0000","last_change":"2023-10-16T14:37:13.152582+0000","last_active":"2023-10-16T14:37:15.164779+0000","last_peered":"2023-10-16T14:37:15.164779+0000","last_clean":"2023-10-16T14:37:15.164779+0000","last_became_active":"2023-10-16T14:37:13.152311+0000","last_became_peered":"2023-10-16T14:37:13.152311+0000","last_unstale":"2023-10-16T14:37:15.164779+0000","last_undegraded":"2023-10-16T14:37:15.164779+0000","last_fullsized":"2023-10-16T14:37:15.164779+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:44.375843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158555+0000","last_change":"2023-10-16T14:37:14.162468+0000","last_active":"2023-10-16T14:37:15.158555+0000","last_peered":"2023-10-16T14:37:15.158555+0000","last_clean":"2023-10-16T14:37:15.158555+0000","last_became_active":"2023-10-16T14:37:14.162206+0000","last_became_peered":"2023-10-16T14:37:14.162206+0000","last_unstale":"2023-10-16T14:37:15.158555+0000","last_undegraded":"2023-10-16T14:37:15.158555+0000","last_fullsized":"2023-10-16T14:37:15.158555+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:09.487380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175588+0000","last_change":"2023-10-16T14:37:13.155921+0000","last_active":"2023-10-16T14:37:15.175588+0000","last_peered":"2023-10-16T14:37:15.175588+0000","last_clean":"2023-10-16T14:37:15.175588+0000","last_became_active":"2023-10-16T14:37:13.155569+0000","last_became_peered":"2023-10-16T14:37:13.155569+0000","last_unstale":"2023-10-16T14:37:15.175588+0000","last_undegraded":"2023-10-16T14:37:15.175588+0000","last_fullsized":"2023-10-16T14:37:15.175588+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:03.169423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376009+0000","last_change":"2023-10-16T14:37:14.161567+0000","last_active":"2023-10-16T14:37:15.376009+0000","last_peered":"2023-10-16T14:37:15.376009+0000","last_clean":"2023-10-16T14:37:15.376009+0000","last_became_active":"2023-10-16T14:37:14.159001+0000","last_became_peered":"2023-10-16T14:37:14.159001+0000","last_unstale":"2023-10-16T14:37:15.376009+0000","last_undegraded":"2023-10-16T14:37:15.376009+0000","last_fullsized":"2023-10-16T14:37:15.376009+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:50.473704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175502+0000","last_change":"2023-10-16T14:37:13.153284+0000","last_active":"2023-10-16T14:37:15.175502+0000","last_peered":"2023-10-16T14:37:15.175502+0000","last_clean":"2023-10-16T14:37:15.175502+0000","last_became_active":"2023-10-16T14:37:13.152985+0000","last_became_peered":"2023-10-16T14:37:13.152985+0000","last_unstale":"2023-10-16T14:37:15.175502+0000","last_undegraded":"2023-10-16T14:37:15.175502+0000","last_fullsized":"2023-10-16T14:37:15.175502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:02.863281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.164513+0000","last_change":"2023-10-16T14:37:14.164130+0000","last_active":"2023-10-16T14:37:15.164513+0000","last_peered":"2023-10-16T14:37:15.164513+0000","last_clean":"2023-10-16T14:37:15.164513+0000","last_became_active":"2023-10-16T14:37:14.162356+0000","last_became_peered":"2023-10-16T14:37:14.162356+0000","last_unstale":"2023-10-16T14:37:15.164513+0000","last_undegraded":"2023-10-16T14:37:15.164513+0000","last_fullsized":"2023-10-16T14:37:15.164513+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:34.558500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379234+0000","last_change":"2023-10-16T14:37:13.154608+0000","last_active":"2023-10-16T14:37:15.379234+0000","last_peered":"2023-10-16T14:37:15.379234+0000","last_clean":"2023-10-16T14:37:15.379234+0000","last_became_active":"2023-10-16T14:37:13.154337+0000","last_became_peered":"2023-10-16T14:37:13.154337+0000","last_unstale":"2023-10-16T14:37:15.379234+0000","last_undegraded":"2023-10-16T14:37:15.379234+0000","last_fullsized":"2023-10-16T14:37:15.379234+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:26:57.739325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158145+0000","last_change":"2023-10-16T14:37:14.165480+0000","last_active":"2023-10-16T14:37:15.158145+0000","last_peered":"2023-10-16T14:37:15.158145+0000","last_clean":"2023-10-16T14:37:15.158145+0000","last_became_active":"2023-10-16T14:37:14.165214+0000","last_became_peered":"2023-10-16T14:37:14.165214+0000","last_unstale":"2023-10-16T14:37:15.158145+0000","last_undegraded":"2023-10-16T14:37:15.158145+0000","last_fullsized":"2023-10-16T14:37:15.158145+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:24.855910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378445+0000","last_change":"2023-10-16T14:37:13.152198+0000","last_active":"2023-10-16T14:37:15.378445+0000","last_peered":"2023-10-16T14:37:15.378445+0000","last_clean":"2023-10-16T14:37:15.378445+0000","last_became_active":"2023-10-16T14:37:13.151780+0000","last_became_peered":"2023-10-16T14:37:13.151780+0000","last_unstale":"2023-10-16T14:37:15.378445+0000","last_undegraded":"2023-10-16T14:37:15.378445+0000","last_fullsized":"2023-10-16T14:37:15.378445+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:35.538081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.161958+0000","last_change":"2023-10-16T14:37:14.158257+0000","last_active":"2023-10-16T14:37:15.161958+0000","last_peered":"2023-10-16T14:37:15.161958+0000","last_clean":"2023-10-16T14:37:15.161958+0000","last_became_active":"2023-10-16T14:37:14.157947+0000","last_became_peered":"2023-10-16T14:37:14.157947+0000","last_unstale":"2023-10-16T14:37:15.161958+0000","last_undegraded":"2023-10-16T14:37:15.161958+0000","last_fullsized":"2023-10-16T14:37:15.161958+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:46.458141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377741+0000","last_change":"2023-10-16T14:37:13.160975+0000","last_active":"2023-10-16T14:37:15.377741+0000","last_peered":"2023-10-16T14:37:15.377741+0000","last_clean":"2023-10-16T14:37:15.377741+0000","last_became_active":"2023-10-16T14:37:13.158969+0000","last_became_peered":"2023-10-16T14:37:13.158969+0000","last_unstale":"2023-10-16T14:37:15.377741+0000","last_undegraded":"2023-10-16T14:37:15.377741+0000","last_fullsized":"2023-10-16T14:37:15.377741+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:19.682285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.165223+0000","last_change":"2023-10-16T14:37:14.161420+0000","last_active":"2023-10-16T14:37:15.165223+0000","last_peered":"2023-10-16T14:37:15.165223+0000","last_clean":"2023-10-16T14:37:15.165223+0000","last_became_active":"2023-10-16T14:37:14.160311+0000","last_became_peered":"2023-10-16T14:37:14.160311+0000","last_unstale":"2023-10-16T14:37:15.165223+0000","last_undegraded":"2023-10-16T14:37:15.165223+0000","last_fullsized":"2023-10-16T14:37:15.165223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:46.615789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376954+0000","last_change":"2023-10-16T14:37:13.158627+0000","last_active":"2023-10-16T14:37:15.376954+0000","last_peered":"2023-10-16T14:37:15.376954+0000","last_clean":"2023-10-16T14:37:15.376954+0000","last_became_active":"2023-10-16T14:37:13.158291+0000","last_became_peered":"2023-10-16T14:37:13.158291+0000","last_unstale":"2023-10-16T14:37:15.376954+0000","last_undegraded":"2023-10-16T14:37:15.376954+0000","last_fullsized":"2023-10-16T14:37:15.376954+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:56.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.376857+0000","last_change":"2023-10-16T14:37:14.160760+0000","last_active":"2023-10-16T14:37:15.376857+0000","last_peered":"2023-10-16T14:37:15.376857+0000","last_clean":"2023-10-16T14:37:15.376857+0000","last_became_active":"2023-10-16T14:37:14.160418+0000","last_became_peered":"2023-10-16T14:37:14.160418+0000","last_unstale":"2023-10-16T14:37:15.376857+0000","last_undegraded":"2023-10-16T14:37:15.376857+0000","last_fullsized":"2023-10-16T14:37:15.376857+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:19.542779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162893+0000","last_change":"2023-10-16T14:37:13.161101+0000","last_active":"2023-10-16T14:37:15.162893+0000","last_peered":"2023-10-16T14:37:15.162893+0000","last_clean":"2023-10-16T14:37:15.162893+0000","last_became_active":"2023-10-16T14:37:13.159488+0000","last_became_peered":"2023-10-16T14:37:13.159488+0000","last_unstale":"2023-10-16T14:37:15.162893+0000","last_undegraded":"2023-10-16T14:37:15.162893+0000","last_fullsized":"2023-10-16T14:37:15.162893+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:40.115179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158206+0000","last_change":"2023-10-16T14:37:14.160037+0000","last_active":"2023-10-16T14:37:15.158206+0000","last_peered":"2023-10-16T14:37:15.158206+0000","last_clean":"2023-10-16T14:37:15.158206+0000","last_became_active":"2023-10-16T14:37:14.159756+0000","last_became_peered":"2023-10-16T14:37:14.159756+0000","last_unstale":"2023-10-16T14:37:15.158206+0000","last_undegraded":"2023-10-16T14:37:15.158206+0000","last_fullsized":"2023-10-16T14:37:15.158206+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:48.079668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158473+0000","last_change":"2023-10-16T14:37:13.153896+0000","last_active":"2023-10-16T14:37:15.158473+0000","last_peered":"2023-10-16T14:37:15.158473+0000","last_clean":"2023-10-16T14:37:15.158473+0000","last_became_active":"2023-10-16T14:37:13.152833+0000","last_became_peered":"2023-10-16T14:37:13.152833+0000","last_unstale":"2023-10-16T14:37:15.158473+0000","last_undegraded":"2023-10-16T14:37:15.158473+0000","last_fullsized":"2023-10-16T14:37:15.158473+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:56.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.165004+0000","last_change":"2023-10-16T14:37:14.167657+0000","last_active":"2023-10-16T14:37:15.165004+0000","last_peered":"2023-10-16T14:37:15.165004+0000","last_clean":"2023-10-16T14:37:15.165004+0000","last_became_active":"2023-10-16T14:37:14.167326+0000","last_became_peered":"2023-10-16T14:37:14.167326+0000","last_unstale":"2023-10-16T14:37:15.165004+0000","last_undegraded":"2023-10-16T14:37:15.165004+0000","last_fullsized":"2023-10-16T14:37:15.165004+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:51.957881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.379183+0000","last_change":"2023-10-16T14:37:13.157117+0000","last_active":"2023-10-16T14:37:15.379183+0000","last_peered":"2023-10-16T14:37:15.379183+0000","last_clean":"2023-10-16T14:37:15.379183+0000","last_became_active":"2023-10-16T14:37:13.156496+0000","last_became_peered":"2023-10-16T14:37:13.156496+0000","last_unstale":"2023-10-16T14:37:15.379183+0000","last_undegraded":"2023-10-16T14:37:15.379183+0000","last_fullsized":"2023-10-16T14:37:15.379183+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:08.473419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.161831+0000","last_change":"2023-10-16T14:37:14.162514+0000","last_active":"2023-10-16T14:37:15.161831+0000","last_peered":"2023-10-16T14:37:15.161831+0000","last_clean":"2023-10-16T14:37:15.161831+0000","last_became_active":"2023-10-16T14:37:14.161428+0000","last_became_peered":"2023-10-16T14:37:14.161428+0000","last_unstale":"2023-10-16T14:37:15.161831+0000","last_undegraded":"2023-10-16T14:37:15.161831+0000","last_fullsized":"2023-10-16T14:37:15.161831+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:27.218231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.158068+0000","last_change":"2023-10-16T14:37:13.153872+0000","last_active":"2023-10-16T14:37:15.158068+0000","last_peered":"2023-10-16T14:37:15.158068+0000","last_clean":"2023-10-16T14:37:15.158068+0000","last_became_active":"2023-10-16T14:37:13.153613+0000","last_became_peered":"2023-10-16T14:37:13.153613+0000","last_unstale":"2023-10-16T14:37:15.158068+0000","last_undegraded":"2023-10-16T14:37:15.158068+0000","last_fullsized":"2023-10-16T14:37:15.158068+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:56.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.175873+0000","last_change":"2023-10-16T14:37:14.166930+0000","last_active":"2023-10-16T14:37:15.175873+0000","last_peered":"2023-10-16T14:37:15.175873+0000","last_clean":"2023-10-16T14:37:15.175873+0000","last_became_active":"2023-10-16T14:37:14.166339+0000","last_became_peered":"2023-10-16T14:37:14.166339+0000","last_unstale":"2023-10-16T14:37:15.175873+0000","last_undegraded":"2023-10-16T14:37:15.175873+0000","last_fullsized":"2023-10-16T14:37:15.175873+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:45.389804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162020+0000","last_change":"2023-10-16T14:37:13.159323+0000","last_active":"2023-10-16T14:37:15.162020+0000","last_peered":"2023-10-16T14:37:15.162020+0000","last_clean":"2023-10-16T14:37:15.162020+0000","last_became_active":"2023-10-16T14:37:13.158383+0000","last_became_peered":"2023-10-16T14:37:13.158383+0000","last_unstale":"2023-10-16T14:37:15.162020+0000","last_undegraded":"2023-10-16T14:37:15.162020+0000","last_fullsized":"2023-10-16T14:37:15.162020+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:17.618256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378837+0000","last_change":"2023-10-16T14:37:14.161380+0000","last_active":"2023-10-16T14:37:15.378837+0000","last_peered":"2023-10-16T14:37:15.378837+0000","last_clean":"2023-10-16T14:37:15.378837+0000","last_became_active":"2023-10-16T14:37:14.160851+0000","last_became_peered":"2023-10-16T14:37:14.160851+0000","last_unstale":"2023-10-16T14:37:15.378837+0000","last_undegraded":"2023-10-16T14:37:15.378837+0000","last_fullsized":"2023-10-16T14:37:15.378837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:39.617735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.378550+0000","last_change":"2023-10-16T14:37:13.158945+0000","last_active":"2023-10-16T14:37:15.378550+0000","last_peered":"2023-10-16T14:37:15.378550+0000","last_clean":"2023-10-16T14:37:15.378550+0000","last_became_active":"2023-10-16T14:37:13.158389+0000","last_became_peered":"2023-10-16T14:37:13.158389+0000","last_unstale":"2023-10-16T14:37:15.378550+0000","last_undegraded":"2023-10-16T14:37:15.378550+0000","last_fullsized":"2023-10-16T14:37:15.378550+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:32.920579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.157836+0000","last_change":"2023-10-16T14:37:14.167927+0000","last_active":"2023-10-16T14:37:15.157836+0000","last_peered":"2023-10-16T14:37:15.157836+0000","last_clean":"2023-10-16T14:37:15.157836+0000","last_became_active":"2023-10-16T14:37:14.167600+0000","last_became_peered":"2023-10-16T14:37:14.167600+0000","last_unstale":"2023-10-16T14:37:15.157836+0000","last_undegraded":"2023-10-16T14:37:15.157836+0000","last_fullsized":"2023-10-16T14:37:15.157836+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:11.397048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.377829+0000","last_change":"2023-10-16T14:37:13.156247+0000","last_active":"2023-10-16T14:37:15.377829+0000","last_peered":"2023-10-16T14:37:15.377829+0000","last_clean":"2023-10-16T14:37:15.377829+0000","last_became_active":"2023-10-16T14:37:13.155946+0000","last_became_peered":"2023-10-16T14:37:13.155946+0000","last_unstale":"2023-10-16T14:37:15.377829+0000","last_undegraded":"2023-10-16T14:37:15.377829+0000","last_fullsized":"2023-10-16T14:37:15.377829+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:48.334064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.161782+0000","last_change":"2023-10-16T14:37:14.160716+0000","last_active":"2023-10-16T14:37:15.161782+0000","last_peered":"2023-10-16T14:37:15.161782+0000","last_clean":"2023-10-16T14:37:15.161782+0000","last_became_active":"2023-10-16T14:37:14.159818+0000","last_became_peered":"2023-10-16T14:37:14.159818+0000","last_unstale":"2023-10-16T14:37:15.161782+0000","last_undegraded":"2023-10-16T14:37:15.161782+0000","last_fullsized":"2023-10-16T14:37:15.161782+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:13.105707+0000","last_clean_scrub_stamp":"2023-10-16T14:37:13.105707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:31.945899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"154'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T14:37:15.162605+0000","last_change":"2023-10-16T14:37:13.159184+0000","last_active":"2023-10-16T14:37:15.162605+0000","last_peered":"2023-10-16T14:37:15.162605+0000","last_clean":"2023-10-16T14:37:15.162605+0000","last_became_active":"2023-10-16T14:37:13.156882+0000","last_became_peered":"2023-10-16T14:37:13.156882+0000","last_unstale":"2023-10-16T14:37:15.162605+0000","last_undegraded":"2023-10-16T14:37:15.162605+0000","last_fullsized":"2023-10-16T14:37:15.162605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:37:12.091746+0000","last_clean_scrub_stamp":"2023-10-16T14:37:12.091746+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:26.933252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"144'252","reported_seq":788,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-16T14:37:14.136793+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T14:37:14.136793+0000","last_peered":"2023-10-16T14:37:14.136793+0000","last_clean":"2023-10-16T14:37:14.136793+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T14:37:14.136793+0000","last_undegraded":"2023-10-16T14:37:14.136793+0000","last_fullsized":"2023-10-16T14:37:14.136793+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575095,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41992,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329848,"statfs":{"total":96636764160,"available":96593764352,"internally_reserved":0,"allocated":2547712,"data_stored":1526567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40434123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.0860000000000001,"15min":1.121},"min":{"1min":0.752,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0449999999999999,"15min":1.1200000000000001},"min":{"1min":0.77800000000000002,"5min":0.442,"15min":0.442},"max":{"1min":1.389,"5min":1.891,"15min":1.891},"last":1.155}]},{"osd":1,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.1419999999999999,"15min":1.181},"min":{"1min":0.66000000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":1.04},{"interface":"front","average":{"1min":1.236,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.71599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.627,"5min":2.048,"15min":2.048},"last":0.998}]},{"osd":2,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.2669999999999999,"15min":1.302},"min":{"1min":1.0600000000000001,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1839999999999999,"15min":1.2170000000000001},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.1269999999999998,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":1.125}]},{"osd":3,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.2410000000000001,"15min":1.29},"min":{"1min":0.85999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.3280000000000001,"15min":1.3600000000000001},"min":{"1min":0.77700000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.1970000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3400000000000001,"15min":1.359},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.8420000000000001,"5min":2.52,"15min":2.52},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3009999999999999,"15min":1.2609999999999999},"min":{"1min":0.60399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.0299999999999998,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":1.262}]},{"osd":5,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.3819999999999999,"15min":1.3959999999999999},"min":{"1min":0.79800000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.913,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.498,"15min":1.4890000000000001},"min":{"1min":0.84499999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":1.55}]},{"osd":6,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.4379999999999999,"15min":1.4390000000000001},"min":{"1min":0.996,"5min":0.502,"15min":0.502},"max":{"1min":2.2010000000000001,"5min":2.79,"15min":2.79},"last":1.508},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.518,"15min":1.5109999999999999},"min":{"1min":0.89800000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.1360000000000001,"5min":2.6139999999999999,"15min":2.6139999999999999},"last":1.458}]}]},{"osd":6,"up_from":13,"seq":55834575094,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1941504,"data_stored":935561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1539999999999999,"15min":1.1759999999999999},"min":{"1min":0.68899999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.4239999999999999,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.264},{"interface":"front","average":{"1min":1.228,"5min":1.2709999999999999,"15min":1.2569999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.6919999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":1.5960000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2909999999999999,"15min":1.284},"min":{"1min":0.78800000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.637,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.278,"15min":1.29},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.603,"5min":10.241,"15min":10.241},"last":1.2030000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.141,"15min":1.1990000000000001},"min":{"1min":0.65700000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.4670000000000001,"5min":2.1469999999999998,"15min":2.4940000000000002},"last":1.524},{"interface":"front","average":{"1min":1.28,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.76500000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.647,"5min":5.694,"15min":5.694},"last":1.1419999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3580000000000001,"15min":1.355},"min":{"1min":0.85699999999999998,"5min":0.76300000000000001,"15min":0.71999999999999997},"max":{"1min":1.6930000000000001,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":1.474},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.2609999999999999,"15min":1.3160000000000001},"min":{"1min":0.84599999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.617,"5min":2.2349999999999999,"15min":2.6120000000000001},"last":1.4339999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.413,"15min":1.343},"min":{"1min":0.82499999999999996,"5min":0.54200000000000004,"15min":0.48999999999999999},"max":{"1min":1.7849999999999999,"5min":13.826000000000001,"15min":13.826000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.3859999999999999,"15min":1.3660000000000001},"min":{"1min":1.01,"5min":0.63200000000000001,"15min":0.44},"max":{"1min":1.8380000000000001,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.6850000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.351,"15min":1.3129999999999999},"min":{"1min":0.77600000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.897,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":1.306},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.518,"15min":1.5},"min":{"1min":0.89400000000000002,"5min":0.59599999999999997,"15min":0.56000000000000005},"max":{"1min":1.8959999999999999,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.585,"15min":1.5629999999999999},"min":{"1min":1.1120000000000001,"5min":0.84099999999999997,"15min":0.58299999999999996},"max":{"1min":1.8400000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5349999999999999,"15min":1.536},"min":{"1min":1.004,"5min":0.67500000000000004,"15min":0.54200000000000004},"max":{"1min":1.9810000000000001,"5min":2.1779999999999999,"15min":2.7250000000000001},"last":1.625}]}]},{"osd":5,"up_from":13,"seq":55834575094,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":934061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.079,"15min":1.109},"min":{"1min":0.65000000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3779999999999999,"5min":1.669,"15min":2.2309999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.111,"5min":1.131,"15min":1.171},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.42299999999999999},"max":{"1min":1.569,"5min":1.7729999999999999,"15min":1.8340000000000001},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.099,"15min":1.1559999999999999},"min":{"1min":0.71099999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.5169999999999999,"5min":1.6899999999999999,"15min":1.726},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.218,"15min":1.238},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.60399999999999998},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7809999999999999},"last":1.4319999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.2330000000000001},"min":{"1min":0.75600000000000001,"5min":0.71599999999999997,"15min":0.65400000000000003},"max":{"1min":1.5760000000000001,"5min":1.913,"15min":2.5009999999999999},"last":1.232},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.3069999999999999,"15min":1.345},"min":{"1min":0.94299999999999995,"5min":0.72199999999999998,"15min":0.625},"max":{"1min":1.5569999999999999,"5min":2.0049999999999999,"15min":2.1200000000000001},"last":1.6799999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.22,"15min":1.266},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.45000000000000001},"max":{"1min":1.482,"5min":1.903,"15min":1.903},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2709999999999999,"15min":1.345},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.611,"5min":2.2429999999999999,"15min":2.8029999999999999},"last":1.371}]},{"osd":4,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2949999999999999,"15min":1.3200000000000001},"min":{"1min":0.81000000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6459999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4359999999999999,"15min":1.486},"min":{"1min":0.78000000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.782,"5min":2.1379999999999999,"15min":4.9480000000000004},"last":1.607}]},{"osd":6,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.349,"15min":1.377},"min":{"1min":0.86299999999999999,"5min":0.73499999999999999,"15min":0.47899999999999998},"max":{"1min":1.708,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3520000000000001,"15min":1.4099999999999999},"min":{"1min":0.97099999999999997,"5min":0.77700000000000002,"15min":0.55000000000000004},"max":{"1min":1.6639999999999999,"5min":2.1400000000000001,"15min":2.1949999999999998},"last":1.5029999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:37:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.492,"15min":1.534},"min":{"1min":0.96399999999999997,"5min":0.76100000000000001,"15min":0.71399999999999997},"max":{"1min":1.78,"5min":2.3399999999999999,"15min":2.641},"last":1.74},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.569,"15min":1.605},"min":{"1min":0.999,"5min":0.97599999999999998,"15min":0.58099999999999996},"max":{"1min":1.9390000000000001,"5min":2.1909999999999998,"15min":2.2010000000000001},"last":2.0150000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575095,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51124,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320716,"statfs":{"total":96636764160,"available":96584413184,"internally_reserved":0,"allocated":1933312,"data_stored":934019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1970000000000001,"15min":1.165},"min":{"1min":0.65500000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.569,"5min":7.9400000000000004,"15min":7.9400000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.71899999999999997,"5min":0.60699999999999998,"15min":0.58999999999999997},"max":{"1min":1.5609999999999999,"5min":1.7090000000000001,"15min":1.8500000000000001},"last":0.84399999999999997}]},{"osd":1,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.2529999999999999,"15min":1.2130000000000001},"min":{"1min":0.73299999999999998,"5min":0.55600000000000005,"15min":0.53800000000000003},"max":{"1min":1.6739999999999999,"5min":16.959,"15min":16.959},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.123,"5min":1.2929999999999999,"15min":1.2549999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.8069999999999999,"5min":12.333,"15min":12.333},"last":0.96999999999999997}]},{"osd":2,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.377,"15min":1.3979999999999999},"min":{"1min":0.78400000000000003,"5min":0.68100000000000005,"15min":0.67900000000000005},"max":{"1min":1.7130000000000001,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":1.323},{"interface":"front","average":{"1min":1.345,"5min":1.3799999999999999,"15min":1.381},"min":{"1min":0.91800000000000004,"5min":0.59199999999999997,"15min":0.51600000000000001},"max":{"1min":1.669,"5min":8.2899999999999991,"15min":8.2899999999999991},"last":1.054}]},{"osd":3,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.224,"15min":1.2969999999999999},"min":{"1min":0.79800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.762,"5min":1.95,"15min":1.95},"last":1.246},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2949999999999999,"15min":1.3080000000000001},"min":{"1min":0.83599999999999997,"5min":0.53900000000000003,"15min":0.437},"max":{"1min":1.5740000000000001,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.361,"15min":1.3160000000000001},"min":{"1min":0.71299999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.9870000000000001,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3859999999999999,"15min":1.351},"min":{"1min":1.026,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.786,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.157}]},{"osd":6,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.329},"min":{"1min":0.92600000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.657,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.5229999999999999,"15min":1.476},"min":{"1min":0.76000000000000001,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8819999999999999,"5min":7.9900000000000002,"15min":7.9900000000000002},"last":1.401}]},{"osd":7,"last update":"Mon Oct 16 14:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5109999999999999,"15min":1.4850000000000001},"min":{"1min":0.93200000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.925,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.534,"15min":1.4830000000000001},"min":{"1min":1.0509999999999999,"5min":0.66700000000000004,"15min":0.49399999999999999},"max":{"1min":1.9330000000000001,"5min":3.8090000000000002,"15min":3.8090000000000002},"last":1.3720000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575094,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45552,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326288,"statfs":{"total":96636764160,"available":96590118912,"internally_reserved":0,"allocated":1937408,"data_stored":934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":44693961},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.85399999999999998,"15min":0.87},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.28999999999999998},"max":{"1min":1.367,"5min":1.696,"15min":1.696},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.91300000000000003,"15min":1.002},"min":{"1min":0.45800000000000002,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":1.4610000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.0489999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.92200000000000004},"min":{"1min":0.44500000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4430000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.1060000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.314},"max":{"1min":1.585,"5min":3.024,"15min":3.024},"last":1.331}]},{"osd":2,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.157,"15min":1.2789999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.0590000000000002,"5min":3.3420000000000001,"15min":3.3420000000000001},"last":1.149},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2589999999999999,"15min":1.377},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.5},"max":{"1min":1.843,"5min":2.5609999999999999,"15min":2.6299999999999999},"last":1.468}]},{"osd":4,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2150000000000001,"15min":1.248},"min":{"1min":0.79400000000000004,"5min":0.64300000000000002,"15min":0.56699999999999995},"max":{"1min":1.9930000000000001,"5min":2.02,"15min":2.02},"last":1.498},{"interface":"front","average":{"1min":1.347,"5min":1.274,"15min":1.3169999999999999},"min":{"1min":0.76000000000000001,"5min":0.67800000000000005,"15min":0.629},"max":{"1min":2.516,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.536}]},{"osd":5,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.4239999999999999,"15min":1.4990000000000001},"min":{"1min":0.96799999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.3090000000000002,"5min":3.206,"15min":3.206},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4159999999999999,"15min":1.4450000000000001},"min":{"1min":0.92600000000000005,"5min":0.754,"15min":0.68799999999999994},"max":{"1min":2.145,"5min":3.3809999999999998,"15min":3.3809999999999998},"last":1.673}]},{"osd":6,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3149999999999999,"15min":1.3779999999999999},"min":{"1min":0.85999999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.2789999999999999,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.714},{"interface":"front","average":{"1min":1.474,"5min":1.395,"15min":1.468},"min":{"1min":0.84699999999999998,"5min":0.82599999999999996,"15min":0.57999999999999996},"max":{"1min":2.391,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.4239999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:36:58 2023","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.5089999999999999,"15min":1.5580000000000001},"min":{"1min":0.91800000000000004,"5min":0.85299999999999998,"15min":0.69099999999999995},"max":{"1min":3.5649999999999999,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":1.601},{"interface":"front","average":{"1min":1.51,"5min":1.4219999999999999,"15min":1.5109999999999999},"min":{"1min":0.84899999999999998,"5min":0.749,"15min":0.629},"max":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":1.782}]}]},{"osd":2,"up_from":13,"seq":55834575095,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46968,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324872,"statfs":{"total":96636764160,"available":96588668928,"internally_reserved":0,"allocated":1937408,"data_stored":934565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.88900000000000001,"15min":0.874},"min":{"1min":0.46999999999999997,"5min":0.38,"15min":0.38},"max":{"1min":1.454,"5min":1.835,"15min":1.835},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.92300000000000004,"15min":1.0029999999999999},"min":{"1min":0.436,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":1.3400000000000001,"5min":1.617,"15min":2.3260000000000001},"last":1.034}]},{"osd":1,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.98199999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.35299999999999998},"max":{"1min":1.49,"5min":1.7649999999999999,"15min":3.234},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.115,"5min":1.1180000000000001,"15min":1.161},"min":{"1min":0.624,"5min":0.51200000000000001,"15min":0.42199999999999999},"max":{"1min":1.716,"5min":2.0910000000000002,"15min":3.4740000000000002},"last":1.123}]},{"osd":3,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.087,"15min":1.2070000000000001},"min":{"1min":0.48499999999999999,"5min":0.42999999999999999,"15min":0.42299999999999999},"max":{"1min":1.6990000000000001,"5min":2.0550000000000002,"15min":2.1640000000000001},"last":1.242},{"interface":"front","average":{"1min":1.137,"5min":1.238,"15min":1.294},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.8169999999999999,"5min":2.2610000000000001,"15min":3.5569999999999999},"last":1.5780000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3939999999999999,"15min":1.425},"min":{"1min":0.80400000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8680000000000001,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.363},{"interface":"front","average":{"1min":1.268,"5min":1.258,"15min":1.337},"min":{"1min":0.76800000000000002,"5min":0.754,"15min":0.72599999999999998},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":3.016},"last":1.6739999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3520000000000001,"15min":1.4039999999999999},"min":{"1min":0.89800000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.913,"5min":2.3170000000000002,"15min":2.4100000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5,"15min":1.5429999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.79200000000000004},"max":{"1min":1.9850000000000001,"5min":2.3239999999999998,"15min":3.3769999999999998},"last":1.46}]},{"osd":6,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.323,"15min":1.4450000000000001},"min":{"1min":0.83999999999999997,"5min":0.69399999999999995,"15min":0.67100000000000004},"max":{"1min":1.8600000000000001,"5min":2.0529999999999999,"15min":3.0760000000000001},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.4430000000000001,"15min":1.5109999999999999},"min":{"1min":0.81899999999999995,"5min":0.71299999999999997,"15min":0.65300000000000002},"max":{"1min":1.8799999999999999,"5min":2.4830000000000001,"15min":2.5920000000000001},"last":1.411}]},{"osd":7,"last update":"Mon Oct 16 14:37:02 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4219999999999999,"15min":1.5289999999999999},"min":{"1min":0.96599999999999997,"5min":0.70699999999999996,"15min":0.69099999999999995},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":3.1899999999999999},"last":1.869},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.538,"15min":1.6200000000000001},"min":{"1min":0.83699999999999997,"5min":0.81999999999999995,"15min":0.80300000000000005},"max":{"1min":2.0259999999999998,"5min":2.6030000000000002,"15min":2.7519999999999998},"last":1.6299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575095,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59436,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312404,"statfs":{"total":96636764160,"available":96575901696,"internally_reserved":0,"allocated":1929216,"data_stored":933949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.90600000000000003,"15min":0.91000000000000003},"min":{"1min":0.56899999999999995,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.28},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.95699999999999996,"15min":1.056},"min":{"1min":0.64000000000000001,"5min":0.40799999999999997,"15min":0.378},"max":{"1min":1.611,"5min":1.827,"15min":1.9370000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.91200000000000003,"15min":0.96799999999999997},"min":{"1min":0.55200000000000005,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.6479999999999999,"5min":1.792,"15min":2.0259999999999998},"last":1.2},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.099,"15min":1.1419999999999999},"min":{"1min":0.59299999999999997,"5min":0.47099999999999997,"15min":0.45800000000000002},"max":{"1min":1.6859999999999999,"5min":1.6919999999999999,"15min":1.756},"last":0.85599999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.1990000000000001,"15min":1.2889999999999999},"min":{"1min":0.67200000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.837,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.995},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.091,"15min":1.1739999999999999},"min":{"1min":0.46899999999999997,"5min":0.45700000000000002,"15min":0.44600000000000001},"max":{"1min":1.587,"5min":1.778,"15min":1.855},"last":2.395}]},{"osd":4,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.198,"15min":1.2629999999999999},"min":{"1min":0.754,"5min":0.71299999999999997,"15min":0.56100000000000005},"max":{"1min":1.5009999999999999,"5min":1.958,"15min":2.069},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.61399999999999999},"max":{"1min":1.7929999999999999,"5min":2.899,"15min":2.899},"last":1.393}]},{"osd":5,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3340000000000001,"15min":1.4079999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.623},"max":{"1min":1.7030000000000001,"5min":2.1309999999999998,"15min":2.3439999999999999},"last":1.339},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.335,"15min":1.411},"min":{"1min":0.82899999999999996,"5min":0.71199999999999997,"15min":0.59499999999999997},"max":{"1min":1.9059999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.964}]},{"osd":6,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.506,"15min":1.526},"min":{"1min":0.89800000000000002,"5min":0.80000000000000004,"15min":0.53100000000000003},"max":{"1min":2.008,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.403,"15min":1.5049999999999999},"min":{"1min":0.96199999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.798,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.9099999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.522,"15min":1.6000000000000001},"min":{"1min":1.0569999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.923,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":2.2080000000000002},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.4410000000000001,"15min":1.5309999999999999},"min":{"1min":0.98399999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.008,"5min":2.2000000000000002,"15min":2.2170000000000001},"last":2.085}]}]},{"osd":0,"up_from":13,"seq":55834575095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37804,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334036,"statfs":{"total":96636764160,"available":96598052864,"internally_reserved":0,"allocated":2523136,"data_stored":1524317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.91800000000000004,"15min":0.93600000000000005},"min":{"1min":0.48699999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.274,"5min":3.871,"15min":3.871},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.119,"15min":1.159},"min":{"1min":0.54500000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7290000000000001,"5min":3.726,"15min":3.726},"last":0.93500000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.1579999999999999,"15min":1.236},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.7589999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1419999999999999,"15min":1.1819999999999999},"min":{"1min":0.60399999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9259999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95699999999999996,"15min":0.99399999999999999},"min":{"1min":0.63200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.3700000000000001,"5min":3.109,"15min":3.109},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.181,"5min":1.1819999999999999,"15min":1.218},"min":{"1min":0.59999999999999998,"5min":0.45000000000000001,"15min":0.437},"max":{"1min":2.04,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":1.1200000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2889999999999999,"15min":1.306},"min":{"1min":0.88900000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.865,"5min":3.4380000000000002,"15min":3.4380000000000002},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.4750000000000001,"15min":1.478},"min":{"1min":0.997,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1829999999999998,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":1.2470000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3899999999999999,"15min":1.419},"min":{"1min":0.97399999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.877,"5min":4.2119999999999997,"15min":4.2119999999999997},"last":1.496},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3939999999999999,"15min":1.3759999999999999},"min":{"1min":0.84099999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.978,"5min":5.7510000000000003,"15min":5.7510000000000003},"last":1.1759999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.589,"15min":1.585},"min":{"1min":1.179,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.0190000000000001,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":1.2},{"interface":"front","average":{"1min":1.512,"5min":1.4850000000000001,"15min":1.502},"min":{"1min":1.018,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.0110000000000001,"5min":3.371,"15min":3.371},"last":1.3859999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:36:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5509999999999999,"15min":1.583},"min":{"1min":0.93000000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1920000000000002,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.595},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.579,"15min":1.5600000000000001},"min":{"1min":1.0960000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.113,"5min":5.6189999999999998,"15min":5.6189999999999998},"last":1.081}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:37:19.334 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:37:19.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:37:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.604+0000 7f427cbd7640 1 Processor -- start 2023-10-16T14:37:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.604+0000 7f427cbd7640 1 -- start start 2023-10-16T14:37:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.605+0000 7f427cbd7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42780b3070 0x7f42780cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.605+0000 7f427cbd7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780d1b10 0x7f42780cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.605+0000 7f427cbd7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42780b3940 con 0x7f42780b3e70 2023-10-16T14:37:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.605+0000 7f427cbd7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42780b45e0 con 0x7f42780b3070 2023-10-16T14:37:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.605+0000 7f427cbd7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42780b3b90 con 0x7f42780d1b10 2023-10-16T14:37:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f427659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42780b3070 0x7f42780cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f426dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42780b3e70 0x7f42780c6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43860/0 (socket says 172.21.15.29:43860) 2023-10-16T14:37:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f427659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42780b3070 0x7f42780cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35286/0 (socket says 172.21.15.29:35286) 2023-10-16T14:37:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f426dd9b640 1 -- 172.21.15.29:0/2438598336 learned_addr learned my addr 172.21.15.29:0/2438598336 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f4276d9d640 1 --2- 172.21.15.29:0/2438598336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780d1b10 0x7f42780cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f427659c640 1 -- 172.21.15.29:0/2438598336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780d1b10 msgr2=0x7f42780cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f427659c640 1 --2- 172.21.15.29:0/2438598336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780d1b10 0x7f42780cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f427659c640 1 -- 172.21.15.29:0/2438598336 >> v1:172.21.15.29:6789/0 conn(0x7f42780b3e70 legacy=0x7f42780c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f427659c640 1 -- 172.21.15.29:0/2438598336 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42780d3840 con 0x7f42780b3070 2023-10-16T14:37:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.606+0000 7f4275d9b640 1 -- 172.21.15.29:0/2438598336 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805146794 0 0) 0x7f42780b3940 con 0x7f42780b3e70 2023-10-16T14:37:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.608+0000 7f427659c640 1 --2- 172.21.15.29:0/2438598336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780b3070 0x7f42780cf500 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f4268009790 tx=0x7f426802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc7fdeea6c22d58 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.608+0000 7f4275d9b640 1 -- 172.21.15.29:0/2438598336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268031800 con 0x7f42780b3070 2023-10-16T14:37:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.608+0000 7f4275d9b640 1 -- 172.21.15.29:0/2438598336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4268031960 con 0x7f42780b3070 2023-10-16T14:37:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.608+0000 7f4275d9b640 1 -- 172.21.15.29:0/2438598336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268031cf0 con 0x7f42780b3070 2023-10-16T14:37:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.608+0000 7f427cbd7640 1 -- 172.21.15.29:0/2438598336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780b3070 msgr2=0x7f42780cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.608+0000 7f427cbd7640 1 --2- 172.21.15.29:0/2438598336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780b3070 0x7f42780cf500 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f4268009790 tx=0x7f426802b8a0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.609+0000 7f427cbd7640 1 -- 172.21.15.29:0/2438598336 shutdown_connections 2023-10-16T14:37:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.609+0000 7f427cbd7640 1 --2- 172.21.15.29:0/2438598336 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780d1b10 0x7f42780cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.609+0000 7f427cbd7640 1 --2- 172.21.15.29:0/2438598336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780b3070 0x7f42780cf500 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.609+0000 7f427cbd7640 1 -- 172.21.15.29:0/2438598336 >> 172.21.15.29:0/2438598336 conn(0x7f42780d2d20 msgr2=0x7f42780d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.609+0000 7f427cbd7640 1 -- 172.21.15.29:0/2438598336 shutdown_connections 2023-10-16T14:37:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.609+0000 7f427cbd7640 1 -- 172.21.15.29:0/2438598336 wait complete. 2023-10-16T14:37:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.611+0000 7f427cbd7640 1 Processor -- start 2023-10-16T14:37:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.611+0000 7f427cbd7640 1 -- start start 2023-10-16T14:37:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f427cbd7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42780b3070 0x7f42780b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f427cbd7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780b3e70 0x7f42780b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f427cbd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 0x7f42780b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f427cbd7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42780b6510 con 0x7f42780d1b10 2023-10-16T14:37:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f427cbd7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42780b8510 con 0x7f42780b3070 2023-10-16T14:37:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f427cbd7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42780b87f0 con 0x7f42780b3e70 2023-10-16T14:37:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f4276d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 0x7f42780b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f4276d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 0x7f42780b7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35302/0 (socket says 172.21.15.29:35302) 2023-10-16T14:37:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f4276d9d640 1 -- 172.21.15.29:0/1716005626 learned_addr learned my addr 172.21.15.29:0/1716005626 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f4276d9d640 1 -- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780b3e70 msgr2=0x7f42780b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.612+0000 7f4276d9d640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780b3e70 0x7f42780b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f4276d9d640 1 -- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42780b3070 msgr2=0x7f42780b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f4276d9d640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42780b3070 0x7f42780b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f4276d9d640 1 -- 172.21.15.29:0/1716005626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680093a0 con 0x7f42780d1b10 2023-10-16T14:37:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f4276d9d640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 0x7f42780b7eb0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f427000d5d0 tx=0x7f427000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42700125f0 con 0x7f42780d1b10 2023-10-16T14:37:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4270012770 con 0x7f42780d1b10 2023-10-16T14:37:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270012b20 con 0x7f42780d1b10 2023-10-16T14:37:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42780b00e0 con 0x7f42780d1b10 2023-10-16T14:37:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.613+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42780b06a0 con 0x7f42780d1b10 2023-10-16T14:37:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.615+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42780b08c0 con 0x7f42780d1b10 2023-10-16T14:37:19.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.616+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4270012c80 con 0x7f42780d1b10 2023-10-16T14:37:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.617+0000 7f426f7fe640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f425806af90 0x7f425806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.617+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4270089d20 con 0x7f42780d1b10 2023-10-16T14:37:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.617+0000 7f427659c640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f425806af90 0x7f425806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.618+0000 7f427659c640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f425806af90 0x7f425806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4268036000 tx=0x7f4268030000 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.621+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4270054c10 con 0x7f42780d1b10 2023-10-16T14:37:19.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.889+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f427804e160 con 0x7f42780d1b10 2023-10-16T14:37:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.895+0000 7f426f7fe640 1 -- 172.21.15.29:0/1716005626 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4270058230 con 0x7f42780d1b10 2023-10-16T14:37:19.899 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T14:37:19.899 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T14:37:19.900 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T14:37:19.900 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T14:37:19.900 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:37:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T14:37:19.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f425806af90 msgr2=0x7f425806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f425806af90 0x7f425806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4268036000 tx=0x7f4268030000 comp rx=0 tx=0).stop 2023-10-16T14:37:19.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 msgr2=0x7f42780b7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:19.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 0x7f42780b7eb0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f427000d5d0 tx=0x7f427000da90 comp rx=0 tx=0).stop 2023-10-16T14:37:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 shutdown_connections 2023-10-16T14:37:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f425806af90 0x7f425806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42780d1b10 0x7f42780b7eb0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42780b3e70 0x7f42780b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 --2- 172.21.15.29:0/1716005626 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42780b3070 0x7f42780b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 >> 172.21.15.29:0/1716005626 conn(0x7f42780d2d20 msgr2=0x7f4278160660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 shutdown_connections 2023-10-16T14:37:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:19.899+0000 7f427cbd7640 1 -- 172.21.15.29:0/1716005626 wait complete. 2023-10-16T14:37:19.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:37:20.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.139+0000 7f0350a20640 1 Processor -- start 2023-10-16T14:37:20.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.139+0000 7f0350a20640 1 -- start start 2023-10-16T14:37:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f0350a20640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f0350a20640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f034c0ad970 0x7f034c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f0350a20640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f034c15ebc0 0x7f034c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f0350a20640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c06c960 con 0x7f034c0ad970 2023-10-16T14:37:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f0350a20640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c15b9d0 con 0x7f034c15ebc0 2023-10-16T14:37:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f0350a20640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c15bcb0 con 0x7f034c06c040 2023-10-16T14:37:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f034a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f034c0ad970 0x7f034c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f034a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f034c0ad970 0x7f034c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35310/0 (socket says 172.21.15.29:35310) 2023-10-16T14:37:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.141+0000 7f034a59c640 1 -- 172.21.15.29:0/4283013860 learned_addr learned my addr 172.21.15.29:0/4283013860 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:20.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034a59c640 1 -- 172.21.15.29:0/4283013860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 msgr2=0x7f034c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:20.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034b59e640 1 --2- 172.21.15.29:0/4283013860 >> v2:172.21.15.170:3300/0 conn(0x7f034c15ebc0 0x7f034c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:20.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034ad9d640 1 --2- 172.21.15.29:0/4283013860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:20.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034a59c640 1 --2- 172.21.15.29:0/4283013860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034a59c640 1 -- 172.21.15.29:0/4283013860 >> v2:172.21.15.170:3300/0 conn(0x7f034c15ebc0 msgr2=0x7f034c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:20.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034a59c640 1 --2- 172.21.15.29:0/4283013860 >> v2:172.21.15.170:3300/0 conn(0x7f034c15ebc0 0x7f034c15f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034a59c640 1 -- 172.21.15.29:0/4283013860 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c15bf90 con 0x7f034c0ad970 2023-10-16T14:37:20.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.142+0000 7f034b59e640 1 --2- 172.21.15.29:0/4283013860 >> v2:172.21.15.170:3300/0 conn(0x7f034c15ebc0 0x7f034c15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:20.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.143+0000 7f034a59c640 1 --2- 172.21.15.29:0/4283013860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0ad970 0x7f034c15e680 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0340009870 tx=0x7f034002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576be2977a246690 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:20.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.143+0000 7f0349d9b640 1 -- 172.21.15.29:0/4283013860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340031640 con 0x7f034c0ad970 2023-10-16T14:37:20.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.143+0000 7f0349d9b640 1 -- 172.21.15.29:0/4283013860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03400317a0 con 0x7f034c0ad970 2023-10-16T14:37:20.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.143+0000 7f0349d9b640 1 -- 172.21.15.29:0/4283013860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340031b30 con 0x7f034c0ad970 2023-10-16T14:37:20.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 -- 172.21.15.29:0/4283013860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0ad970 msgr2=0x7f034c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 --2- 172.21.15.29:0/4283013860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0ad970 0x7f034c15e680 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0340009870 tx=0x7f034002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 -- 172.21.15.29:0/4283013860 shutdown_connections 2023-10-16T14:37:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 --2- 172.21.15.29:0/4283013860 >> v2:172.21.15.170:3300/0 conn(0x7f034c15ebc0 0x7f034c15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 --2- 172.21.15.29:0/4283013860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c0ad970 0x7f034c15e680 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 --2- 172.21.15.29:0/4283013860 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 -- 172.21.15.29:0/4283013860 >> 172.21.15.29:0/4283013860 conn(0x7f034c0ce6a0 msgr2=0x7f034c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:20.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 -- 172.21.15.29:0/4283013860 shutdown_connections 2023-10-16T14:37:20.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.144+0000 7f0350a20640 1 -- 172.21.15.29:0/4283013860 wait complete. 2023-10-16T14:37:20.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.146+0000 7f0350a20640 1 Processor -- start 2023-10-16T14:37:20.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.146+0000 7f0350a20640 1 -- start start 2023-10-16T14:37:20.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f0350a20640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c1f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f0350a20640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0ad970 0x7f034c0b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f0350a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 0x7f034c0c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f0350a20640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0b0a40 con 0x7f034c15ebc0 2023-10-16T14:37:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f0350a20640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0b0f70 con 0x7f034c0ad970 2023-10-16T14:37:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f0350a20640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0c8170 con 0x7f034c06c040 2023-10-16T14:37:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f034b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 0x7f034c0c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f034b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 0x7f034c0c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35314/0 (socket says 172.21.15.29:35314) 2023-10-16T14:37:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.147+0000 7f034b59e640 1 -- 172.21.15.29:0/680780458 learned_addr learned my addr 172.21.15.29:0/680780458 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f034b59e640 1 -- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 msgr2=0x7f034c1f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f034b59e640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c1f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f034b59e640 1 -- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0ad970 msgr2=0x7f034c0b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f034b59e640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0ad970 0x7f034c0b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f034b59e640 1 -- 172.21.15.29:0/680780458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400094d0 con 0x7f034c15ebc0 2023-10-16T14:37:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f034b59e640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 0x7f034c0c97e0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f034400cef0 tx=0x7f034400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034400d840 con 0x7f034c15ebc0 2023-10-16T14:37:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034400d9c0 con 0x7f034c15ebc0 2023-10-16T14:37:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c0c8450 con 0x7f034c15ebc0 2023-10-16T14:37:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.148+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034400dd70 con 0x7f034c15ebc0 2023-10-16T14:37:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.149+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c0c9d20 con 0x7f034c15ebc0 2023-10-16T14:37:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.150+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c06b330 con 0x7f034c15ebc0 2023-10-16T14:37:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.151+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f034401e3b0 con 0x7f034c15ebc0 2023-10-16T14:37:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.151+0000 7f03337fe640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f033806afe0 0x7f033806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.152+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f034408a260 con 0x7f034c15ebc0 2023-10-16T14:37:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.152+0000 7f034ad9d640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f033806afe0 0x7f033806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.153+0000 7f034ad9d640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f033806afe0 0x7f033806d4a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f033c00a770 tx=0x7f033c008040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.156+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0344055250 con 0x7f034c15ebc0 2023-10-16T14:37:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.391+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f034c0c9f10 con 0x7f034c15ebc0 2023-10-16T14:37:20.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.393+0000 7f03337fe640 1 -- 172.21.15.29:0/680780458 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f0344058870 con 0x7f034c15ebc0 2023-10-16T14:37:20.397 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:20.397 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:37:13.609698+0000","modified":"2023-10-16T14:37:16.825159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8838},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8838":{"gid":8838,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3542022450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3542022450},{"type":"v1","addr":"172.21.15.29:6833","nonce":3542022450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T14:37:20.397 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 75 2023-10-16T14:37:20.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.395+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f033806afe0 msgr2=0x7f033806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:20.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.395+0000 7f0350a20640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f033806afe0 0x7f033806d4a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f033c00a770 tx=0x7f033c008040 comp rx=0 tx=0).stop 2023-10-16T14:37:20.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.395+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 msgr2=0x7f034c0c97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:20.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.395+0000 7f0350a20640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 0x7f034c0c97e0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f034400cef0 tx=0x7f034400a710 comp rx=0 tx=0).stop 2023-10-16T14:37:20.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 shutdown_connections 2023-10-16T14:37:20.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f033806afe0 0x7f033806d4a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f034c15ebc0 0x7f034c0c97e0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f034c0ad970 0x7f034c0b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 --2- 172.21.15.29:0/680780458 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f034c06c040 0x7f034c1f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:20.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 >> 172.21.15.29:0/680780458 conn(0x7f034c0ce6a0 msgr2=0x7f034c0ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:20.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 shutdown_connections 2023-10-16T14:37:20.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:20.396+0000 7f0350a20640 1 -- 172.21.15.29:0/680780458 wait complete. 2023-10-16T14:37:20.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:37:13.609698+0000', 'modified': '2023-10-16T14:37:16.825159+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8838}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8838': {'gid': 8838, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6833/3542022450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 3542022450}, {'type': 'v1', 'addr': '172.21.15.29:6833', 'nonce': 3542022450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:37:20.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:37:20.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:37:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.047+0000 7fb892f80640 1 Processor -- start 2023-10-16T14:37:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.047+0000 7fb892f80640 1 -- start start 2023-10-16T14:37:21.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.049+0000 7fb892f80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c0af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.049+0000 7fb892f80640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb88c0afd60 0x7fb88c0cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.049+0000 7fb892f80640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88c0b3730 con 0x7fb88c0cc820 2023-10-16T14:37:21.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.049+0000 7fb892f80640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb88c0b3980 con 0x7fb88c0afd60 2023-10-16T14:37:21.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.049+0000 7fb892f80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c0b3c60 con 0x7fb88c15aa60 2023-10-16T14:37:21.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.049+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89277f640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb88c0cc820 0x7fb88c0b2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:55416/0 (socket says 172.21.15.29:55416) 2023-10-16T14:37:21.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89277f640 1 -- 172.21.15.29:0/1423572820 learned_addr learned my addr 172.21.15.29:0/1423572820 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:21.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89177d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb88c0afd60 0x7fb88c0cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89177d640 1 -- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 msgr2=0x7fb88c0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89177d640 1 --2- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c0af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89177d640 1 -- 172.21.15.29:0/1423572820 >> v1:172.21.15.170:6789/0 conn(0x7fb88c0cc820 legacy=0x7fb88c0b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb89177d640 1 -- 172.21.15.29:0/1423572820 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0b3f40 con 0x7fb88c0afd60 2023-10-16T14:37:21.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.050+0000 7fb891f7e640 1 --2- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c0af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:37:21.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.052+0000 7fb89177d640 1 --2- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0afd60 0x7fb88c0cc2e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0097a0 tx=0x7fb87c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=213f0f99613672a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.052+0000 7fb890f7c640 1 -- 172.21.15.29:0/1423572820 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02d040 con 0x7fb88c0afd60 2023-10-16T14:37:21.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.052+0000 7fb890f7c640 1 -- 172.21.15.29:0/1423572820 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c031810 con 0x7fb88c0afd60 2023-10-16T14:37:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.052+0000 7fb890f7c640 1 -- 172.21.15.29:0/1423572820 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c031ba0 con 0x7fb88c0afd60 2023-10-16T14:37:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.053+0000 7fb892f80640 1 -- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0afd60 msgr2=0x7fb88c0cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.053+0000 7fb892f80640 1 --2- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0afd60 0x7fb88c0cc2e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0097a0 tx=0x7fb87c02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.053+0000 7fb892f80640 1 -- 172.21.15.29:0/1423572820 shutdown_connections 2023-10-16T14:37:21.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.053+0000 7fb892f80640 1 --2- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0afd60 0x7fb88c0cc2e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.053+0000 7fb892f80640 1 --2- 172.21.15.29:0/1423572820 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c0af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.053+0000 7fb892f80640 1 -- 172.21.15.29:0/1423572820 >> 172.21.15.29:0/1423572820 conn(0x7fb88c065ed0 msgr2=0x7fb88c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.054+0000 7fb892f80640 1 -- 172.21.15.29:0/1423572820 shutdown_connections 2023-10-16T14:37:21.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.054+0000 7fb892f80640 1 -- 172.21.15.29:0/1423572820 wait complete. 2023-10-16T14:37:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.056+0000 7fb892f80640 1 Processor -- start 2023-10-16T14:37:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.056+0000 7fb892f80640 1 -- start start 2023-10-16T14:37:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb892f80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 0x7fb88c156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb892f80640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0cc820 0x7fb88c158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb892f80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb892f80640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c15dc70 con 0x7fb88c0afd60 2023-10-16T14:37:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb892f80640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c0b0660 con 0x7fb88c0cc820 2023-10-16T14:37:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb892f80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c0c44c0 con 0x7fb88c15aa60 2023-10-16T14:37:21.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 0x7fb88c156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 0x7fb88c156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35320/0 (socket says 172.21.15.29:35320) 2023-10-16T14:37:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.058+0000 7fb891f7e640 1 -- 172.21.15.29:0/3368514335 learned_addr learned my addr 172.21.15.29:0/3368514335 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb89177d640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0cc820 0x7fb88c158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb89277f640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb891f7e640 1 -- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 msgr2=0x7fb88c158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb891f7e640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb891f7e640 1 -- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0cc820 msgr2=0x7fb88c158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb891f7e640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0cc820 0x7fb88c158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb891f7e640 1 -- 172.21.15.29:0/3368514335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888009400 con 0x7fb88c0afd60 2023-10-16T14:37:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb89277f640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb891f7e640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 0x7fb88c156ee0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fb88800e960 tx=0x7fb88800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb8827fc640 1 -- 172.21.15.29:0/3368514335 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88800ca00 con 0x7fb88c0afd60 2023-10-16T14:37:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb8827fc640 1 -- 172.21.15.29:0/3368514335 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88800cb60 con 0x7fb88c0afd60 2023-10-16T14:37:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.059+0000 7fb8827fc640 1 -- 172.21.15.29:0/3368514335 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888013470 con 0x7fb88c0afd60 2023-10-16T14:37:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.060+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c009400 con 0x7fb88c0afd60 2023-10-16T14:37:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.060+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c0c4760 con 0x7fb88c0afd60 2023-10-16T14:37:21.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.062+0000 7fb8827fc640 1 -- 172.21.15.29:0/3368514335 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb8880135d0 con 0x7fb88c0afd60 2023-10-16T14:37:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.063+0000 7fb8827fc640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb87006af90 0x7fb87006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.063+0000 7fb8827fc640 1 -- 172.21.15.29:0/3368514335 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb888089ac0 con 0x7fb88c0afd60 2023-10-16T14:37:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.064+0000 7fb896d73740 1 Processor -- start 2023-10-16T14:37:21.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.064+0000 7fb89177d640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb87006af90 0x7fb87006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.064+0000 7fb896d73740 1 -- start start 2023-10-16T14:37:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.065+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.065+0000 7fb896d73740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.065+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45b8760 0x55aac45e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.065+0000 7fb896d73740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac44d37a0 con 0x55aac45e0530 2023-10-16T14:37:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.065+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac4525740 con 0x55aac45b8760 2023-10-16T14:37:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.065+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aac4522e30 con 0x55aac4600cf0 2023-10-16T14:37:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb87006af90 0x7fb87006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb87c036000 tx=0x7fb87c03c040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35332/0 (socket says 172.21.15.29:35332) 2023-10-16T14:37:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 -- 172.21.15.29:0/1273159003 learned_addr learned my addr 172.21.15.29:0/1273159003 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb891f7e640 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89277f640 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45b8760 0x55aac45e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 -- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 msgr2=0x55aac45fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 -- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45b8760 msgr2=0x55aac45e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45b8760 0x55aac45e2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 -- 172.21.15.29:0/1273159003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aac44e2910 con 0x55aac45e0530 2023-10-16T14:37:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.066+0000 7fb89177d640 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b8220 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fb87c02bec0 tx=0x7fb87c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e970e77ac11a5a7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.067+0000 7fb86f7fe640 1 -- 172.21.15.29:0/1273159003 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03ac70 con 0x55aac45e0530 2023-10-16T14:37:21.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.067+0000 7fb86f7fe640 1 -- 172.21.15.29:0/1273159003 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c04a040 con 0x55aac45e0530 2023-10-16T14:37:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.067+0000 7fb86f7fe640 1 -- 172.21.15.29:0/1273159003 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c044400 con 0x55aac45e0530 2023-10-16T14:37:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.067+0000 7fb896d73740 1 -- 172.21.15.29:0/1273159003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 msgr2=0x55aac45b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.067+0000 7fb896d73740 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b8220 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fb87c02bec0 tx=0x7fb87c030000 comp rx=0 tx=0).stop 2023-10-16T14:37:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 -- 172.21.15.29:0/1273159003 shutdown_connections 2023-10-16T14:37:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b8220 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45b8760 0x55aac45e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 --2- 172.21.15.29:0/1273159003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 -- 172.21.15.29:0/1273159003 >> 172.21.15.29:0/1273159003 conn(0x55aac45b5380 msgr2=0x55aac45fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 -- 172.21.15.29:0/1273159003 shutdown_connections 2023-10-16T14:37:21.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 -- 172.21.15.29:0/1273159003 wait complete. 2023-10-16T14:37:21.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.068+0000 7fb896d73740 1 -- start start 2023-10-16T14:37:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.069+0000 7fb896d73740 1 Processor -- start 2023-10-16T14:37:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45e5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac4600cf0 0x55aac45e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac45cd6f0 0x55aac45e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac451e720 con 0x55aac45e0530 2023-10-16T14:37:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac44bc970 con 0x55aac4600cf0 2023-10-16T14:37:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aac44bc610 con 0x55aac45cd6f0 2023-10-16T14:37:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb896d73740 10 client.0 _ll_register_callbacks cb 0x55aac44de9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:37:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45e5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35338/0 (socket says 172.21.15.29:35338) 2023-10-16T14:37:21.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 -- 172.21.15.29:0/335247187 learned_addr learned my addr 172.21.15.29:0/335247187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 -- 172.21.15.29:0/335247187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac45cd6f0 msgr2=0x55aac45e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac45cd6f0 0x55aac45e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 -- 172.21.15.29:0/335247187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac4600cf0 msgr2=0x55aac45e5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac4600cf0 0x55aac45e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.070+0000 7fb891f7e640 1 -- 172.21.15.29:0/335247187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c0b3f40 con 0x55aac45e0530 2023-10-16T14:37:21.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb891f7e640 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45e5930 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fb88800c520 tx=0x7fb888058720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d8a52555235019 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb86cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:37:21.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb86cff9640 1 -- 172.21.15.29:0/335247187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8880606d0 con 0x55aac45e0530 2023-10-16T14:37:21.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb86cff9640 1 -- 172.21.15.29:0/335247187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb888054450 con 0x55aac45e0530 2023-10-16T14:37:21.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aac4545160 con 0x55aac45e0530 2023-10-16T14:37:21.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb86cff9640 1 -- 172.21.15.29:0/335247187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb888054610 con 0x55aac45e0530 2023-10-16T14:37:21.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb896d73740 10 client.8985 fetch_fsmap learned FSMap version 75 2023-10-16T14:37:21.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x55aac451cda0 con 0x55aac45e0530 2023-10-16T14:37:21.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.071+0000 7fb86cff9640 1 -- 172.21.15.29:0/335247187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55aac4545160 con 0x55aac45e0530 2023-10-16T14:37:21.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb86cff9640 1 -- 172.21.15.29:0/335247187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 75) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb888011070 con 0x55aac45e0530 2023-10-16T14:37:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb896d73740 10 client.8985 fetch_fsmap finished waiting for FSMap version 75 2023-10-16T14:37:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb896d73740 10 client.8985 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:37:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb896d73740 20 client.8985 populate_metadata read hostname 'smithi029' 2023-10-16T14:37:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb896d73740 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac45dc140 0x55aac45e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb896d73740 4 client.8985 mds_command: new command op to 8838 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:37:21.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 --> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aac44f0cc0 con 0x55aac45dc140 2023-10-16T14:37:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.072+0000 7fb89177d640 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac45dc140 0x55aac45e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.073+0000 7fb89177d640 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac45dc140 0x55aac45e40e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccb0b5db98274666 server_cookie=3c24d09db965e8aa in_seq=0 out_seq=0 2023-10-16T14:37:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.073+0000 7fb86cff9640 10 client.8985 ms_handle_connect on v2:172.21.15.29:6832/3542022450 2023-10-16T14:37:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.073+0000 7fb86cff9640 1 -- 172.21.15.29:0/335247187 <== mds.0 v2:172.21.15.29:6832/3542022450 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aac44f0cc0 con 0x55aac45dc140 2023-10-16T14:37:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.073+0000 7fb86cff9640 10 client.8985 handle_command_reply: tid=0 2023-10-16T14:37:21.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 client.8985 shutdown 2023-10-16T14:37:21.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 msgr2=0x55aac45e5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45e5930 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fb88800c520 tx=0x7fb888058720 comp rx=0 tx=0).stop 2023-10-16T14:37:21.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 shutdown_connections 2023-10-16T14:37:21.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac45dc140 0x55aac45e40e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac45cd6f0 0x55aac45e1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac4600cf0 0x55aac45e5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb86cff9640 0 client.8985 ms_handle_reset on v2:172.21.15.29:6832/3542022450 2023-10-16T14:37:21.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 --2- 172.21.15.29:0/335247187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45e5930 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 >> 172.21.15.29:0/335247187 conn(0x55aac45b8760 msgr2=0x55aac45fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 shutdown_connections 2023-10-16T14:37:21.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 1 -- 172.21.15.29:0/335247187 wait complete. 2023-10-16T14:37:21.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.074+0000 7fb896d73740 20 client.8985 trim_cache size 0 max 16384 2023-10-16T14:37:21.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.091+0000 7fb896d73740 1 Processor -- start 2023-10-16T14:37:21.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.091+0000 7fb896d73740 1 -- start start 2023-10-16T14:37:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.092+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45cd6f0 0x55aac45b9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.092+0000 7fb896d73740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.092+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.092+0000 7fb896d73740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac4282f20 con 0x55aac45e0530 2023-10-16T14:37:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.092+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac45b9060 con 0x55aac45cd6f0 2023-10-16T14:37:21.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.092+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aac45daf20 con 0x55aac4600cf0 2023-10-16T14:37:21.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35352/0 (socket says 172.21.15.29:35352) 2023-10-16T14:37:21.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 -- 172.21.15.29:0/1986947152 learned_addr learned my addr 172.21.15.29:0/1986947152 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89277f640 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb891f7e640 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45cd6f0 0x55aac45b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 -- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 msgr2=0x55aac45e6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 -- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45cd6f0 msgr2=0x55aac45b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45cd6f0 0x55aac45b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.093+0000 7fb89177d640 1 -- 172.21.15.29:0/1986947152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c06cf30 con 0x55aac45e0530 2023-10-16T14:37:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb89177d640 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b9ed0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0098c0 tx=0x7fb87c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a303e24d9befc74e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb86ffff640 1 -- 172.21.15.29:0/1986947152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c05d040 con 0x55aac45e0530 2023-10-16T14:37:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb86ffff640 1 -- 172.21.15.29:0/1986947152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c03aa60 con 0x55aac45e0530 2023-10-16T14:37:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb86ffff640 1 -- 172.21.15.29:0/1986947152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03adf0 con 0x55aac45e0530 2023-10-16T14:37:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb896d73740 1 -- 172.21.15.29:0/1986947152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 msgr2=0x55aac45b9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb896d73740 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b9ed0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0098c0 tx=0x7fb87c009190 comp rx=0 tx=0).stop 2023-10-16T14:37:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb896d73740 1 -- 172.21.15.29:0/1986947152 shutdown_connections 2023-10-16T14:37:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.094+0000 7fb896d73740 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45e0530 0x55aac45b9ed0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.095+0000 7fb896d73740 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.095+0000 7fb896d73740 1 --2- 172.21.15.29:0/1986947152 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45cd6f0 0x55aac45b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.095+0000 7fb896d73740 1 -- 172.21.15.29:0/1986947152 >> 172.21.15.29:0/1986947152 conn(0x55aac45b8760 msgr2=0x55aac45ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.095+0000 7fb896d73740 1 -- 172.21.15.29:0/1986947152 shutdown_connections 2023-10-16T14:37:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.095+0000 7fb896d73740 1 -- 172.21.15.29:0/1986947152 wait complete. 2023-10-16T14:37:21.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.095+0000 7fb896d73740 1 -- start start 2023-10-16T14:37:21.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.096+0000 7fb896d73740 1 Processor -- start 2023-10-16T14:37:21.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 0x55aac4601be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45e0530 0x55aac466a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac466a6c0 con 0x55aac45cd6f0 2023-10-16T14:37:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aac45ff7b0 con 0x55aac45e0530 2023-10-16T14:37:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aac4608cc0 con 0x55aac4600cf0 2023-10-16T14:37:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb896d73740 10 client.0 _ll_register_callbacks cb 0x7fb8880549a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:37:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 0x55aac4601be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 0x55aac4601be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35356/0 (socket says 172.21.15.29:35356) 2023-10-16T14:37:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 -- 172.21.15.29:0/940236477 learned_addr learned my addr 172.21.15.29:0/940236477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 -- 172.21.15.29:0/940236477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 msgr2=0x55aac45e4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 -- 172.21.15.29:0/940236477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45e0530 msgr2=0x55aac466a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45e0530 0x55aac466a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.097+0000 7fb891f7e640 1 -- 172.21.15.29:0/940236477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aac44e2910 con 0x55aac45cd6f0 2023-10-16T14:37:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb891f7e640 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 0x55aac4601be0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fb888058310 tx=0x7fb88801e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8116143e9cc02585 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb86cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T14:37:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb86cff9640 1 -- 172.21.15.29:0/940236477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88805c7b0 con 0x55aac45cd6f0 2023-10-16T14:37:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb86cff9640 1 -- 172.21.15.29:0/940236477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb888076b10 con 0x55aac45cd6f0 2023-10-16T14:37:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb87c04a040 con 0x55aac45cd6f0 2023-10-16T14:37:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb86cff9640 1 -- 172.21.15.29:0/940236477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb888076cd0 con 0x55aac45cd6f0 2023-10-16T14:37:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 10 client.8991 fetch_fsmap learned FSMap version 75 2023-10-16T14:37:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x55aac44bbb80 con 0x55aac45cd6f0 2023-10-16T14:37:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb86cff9640 1 -- 172.21.15.29:0/940236477 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88807e070 con 0x55aac45cd6f0 2023-10-16T14:37:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb86cff9640 1 -- 172.21.15.29:0/940236477 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 75) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb888011040 con 0x55aac45cd6f0 2023-10-16T14:37:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 10 client.8991 fetch_fsmap finished waiting for FSMap version 75 2023-10-16T14:37:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 10 client.8991 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:37:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 20 client.8991 populate_metadata read hostname 'smithi029' 2023-10-16T14:37:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.098+0000 7fb896d73740 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac4611410 0x55aac46117f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.099+0000 7fb896d73740 4 client.8991 mds_command: new command op to 8838 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:37:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.099+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 --> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aac4517700 con 0x55aac4611410 2023-10-16T14:37:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.099+0000 7fb89177d640 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac4611410 0x55aac46117f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.099+0000 7fb89177d640 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac4611410 0x55aac46117f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd34114995e950c8 server_cookie=667a951559ba886e in_seq=0 out_seq=0 2023-10-16T14:37:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.099+0000 7fb86cff9640 10 client.8991 ms_handle_connect on v2:172.21.15.29:6832/3542022450 2023-10-16T14:37:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb86cff9640 1 -- 172.21.15.29:0/940236477 <== mds.0 v2:172.21.15.29:6832/3542022450 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55aac4517700 con 0x55aac4611410 2023-10-16T14:37:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb86cff9640 10 client.8991 handle_command_reply: tid=0 2023-10-16T14:37:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 client.8991 shutdown 2023-10-16T14:37:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 msgr2=0x55aac4601be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 0x55aac4601be0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fb888058310 tx=0x7fb88801e070 comp rx=0 tx=0).stop 2023-10-16T14:37:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 shutdown_connections 2023-10-16T14:37:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:6832/3542022450,v1:172.21.15.29:6833/3542022450] conn(0x55aac4611410 0x55aac46117f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55aac4600cf0 0x55aac45e4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55aac45e0530 0x55aac466a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 --2- 172.21.15.29:0/940236477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aac45cd6f0 0x55aac4601be0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb86cff9640 0 client.8991 ms_handle_reset on v2:172.21.15.29:6832/3542022450 2023-10-16T14:37:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 >> 172.21.15.29:0/940236477 conn(0x55aac45b8760 msgr2=0x55aac45ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 shutdown_connections 2023-10-16T14:37:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 1 -- 172.21.15.29:0/940236477 wait complete. 2023-10-16T14:37:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.100+0000 7fb896d73740 20 client.8991 trim_cache size 0 max 16384 2023-10-16T14:37:21.117 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T14:37:21.117 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T14:37:21.117 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T14:37:21.117 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8838, 2023-10-16T14:37:21.118 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T14:37:21.118 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T14:37:21.118 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T14:37:21.118 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.9525033030000003, 2023-10-16T14:37:21.118 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 75, 2023-10-16T14:37:21.119 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 155, 2023-10-16T14:37:21.119 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 155, 2023-10-16T14:37:21.119 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.746686578 2023-10-16T14:37:21.119 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T14:37:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb87006af90 msgr2=0x7fb87006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb87006af90 0x7fb87006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb87c036000 tx=0x7fb87c03c040 comp rx=0 tx=0).stop 2023-10-16T14:37:21.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 msgr2=0x7fb88c156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 0x7fb88c156ee0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fb88800e960 tx=0x7fb88800ee20 comp rx=0 tx=0).stop 2023-10-16T14:37:21.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 shutdown_connections 2023-10-16T14:37:21.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fb87006af90 0x7fb87006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb88c15aa60 0x7fb88c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c0cc820 0x7fb88c158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 --2- 172.21.15.29:0/3368514335 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb88c0afd60 0x7fb88c156ee0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 >> 172.21.15.29:0/3368514335 conn(0x7fb88c065ed0 msgr2=0x7fb88c0b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 shutdown_connections 2023-10-16T14:37:21.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.101+0000 7fb892f80640 1 -- 172.21.15.29:0/3368514335 wait complete. 2023-10-16T14:37:21.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:37:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.323+0000 7fa4f2e40640 1 Processor -- start 2023-10-16T14:37:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.323+0000 7fa4f2e40640 1 -- start start 2023-10-16T14:37:21.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f2e40640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4ec15aad0 0x7fa4ec0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f2e40640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa4ec0712c0 0x7fa4ec070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f2e40640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ec0b4620 con 0x7fa4ec070d70 2023-10-16T14:37:21.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f2e40640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec0b4340 con 0x7fa4ec15aad0 2023-10-16T14:37:21.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f2e40640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec06f520 con 0x7fa4ec0712c0 2023-10-16T14:37:21.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f1e3e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4ec15aad0 0x7fa4ec0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f1e3e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa4ec15aad0 0x7fa4ec0cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35360/0 (socket says 172.21.15.29:35360) 2023-10-16T14:37:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f163d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa4ec070d70 0x7fa4ec06bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43876/0 (socket says 172.21.15.29:43876) 2023-10-16T14:37:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.325+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/981358987 learned_addr learned my addr 172.21.15.29:0/981358987 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.326+0000 7fa4f263f640 1 --2- 172.21.15.29:0/981358987 >> v2:172.21.15.170:3300/0 conn(0x7fa4ec0712c0 0x7fa4ec070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.326+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/981358987 >> v2:172.21.15.170:3300/0 conn(0x7fa4ec0712c0 msgr2=0x7fa4ec070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.326+0000 7fa4f1e3e640 1 --2- 172.21.15.29:0/981358987 >> v2:172.21.15.170:3300/0 conn(0x7fa4ec0712c0 0x7fa4ec070000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.326+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/981358987 >> v1:172.21.15.29:6789/0 conn(0x7fa4ec070d70 legacy=0x7fa4ec06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.326+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/981358987 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec06f800 con 0x7fa4ec15aad0 2023-10-16T14:37:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.326+0000 7fa4f0e3c640 1 -- 172.21.15.29:0/981358987 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620616791 0 0) 0x7fa4ec0b4620 con 0x7fa4ec070d70 2023-10-16T14:37:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.327+0000 7fa4f1e3e640 1 --2- 172.21.15.29:0/981358987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec15aad0 0x7fa4ec0cc270 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0b3f20 tx=0x7fa4e802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5f36d60097c9da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f0e3c640 1 -- 172.21.15.29:0/981358987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e802f850 con 0x7fa4ec15aad0 2023-10-16T14:37:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f0e3c640 1 -- 172.21.15.29:0/981358987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e802f9b0 con 0x7fa4ec15aad0 2023-10-16T14:37:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f0e3c640 1 -- 172.21.15.29:0/981358987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e802fd40 con 0x7fa4ec15aad0 2023-10-16T14:37:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f2e40640 1 -- 172.21.15.29:0/981358987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec15aad0 msgr2=0x7fa4ec0cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/981358987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec15aad0 0x7fa4ec0cc270 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0b3f20 tx=0x7fa4e802ba30 comp rx=0 tx=0).stop 2023-10-16T14:37:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f2e40640 1 -- 172.21.15.29:0/981358987 shutdown_connections 2023-10-16T14:37:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/981358987 >> v2:172.21.15.170:3300/0 conn(0x7fa4ec0712c0 0x7fa4ec070000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/981358987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec15aad0 0x7fa4ec0cc270 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.328+0000 7fa4f2e40640 1 -- 172.21.15.29:0/981358987 >> 172.21.15.29:0/981358987 conn(0x7fa4ec0b0a90 msgr2=0x7fa4ec0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.329+0000 7fa4f2e40640 1 -- 172.21.15.29:0/981358987 shutdown_connections 2023-10-16T14:37:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.329+0000 7fa4f2e40640 1 -- 172.21.15.29:0/981358987 wait complete. 2023-10-16T14:37:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.330+0000 7fa4f2e40640 1 Processor -- start 2023-10-16T14:37:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.330+0000 7fa4f2e40640 1 -- start start 2023-10-16T14:37:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.331+0000 7fa4f2e40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 0x7fa4ec155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.331+0000 7fa4f2e40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4ec0712c0 0x7fa4ec0c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.331+0000 7fa4f2e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4ec15aad0 0x7fa4ec1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.331+0000 7fa4f2e40640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec156510 con 0x7fa4ec070d70 2023-10-16T14:37:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.331+0000 7fa4f2e40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0c5770 con 0x7fa4ec0712c0 2023-10-16T14:37:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.331+0000 7fa4f2e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0c5ca0 con 0x7fa4ec15aad0 2023-10-16T14:37:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 0x7fa4ec155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 0x7fa4ec155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35368/0 (socket says 172.21.15.29:35368) 2023-10-16T14:37:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/107193328 learned_addr learned my addr 172.21.15.29:0/107193328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f263f640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4ec15aad0 0x7fa4ec1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f163d640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4ec0712c0 0x7fa4ec0c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4ec15aad0 msgr2=0x7fa4ec1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4ec15aad0 0x7fa4ec1566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4ec0712c0 msgr2=0x7fa4ec0c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4ec0712c0 0x7fa4ec0c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.332+0000 7fa4f1e3e640 1 -- 172.21.15.29:0/107193328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e80094d0 con 0x7fa4ec070d70 2023-10-16T14:37:21.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.333+0000 7fa4f1e3e640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 0x7fa4ec155fd0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80090b0 tx=0x7fa4e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.333+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e803b040 con 0x7fa4ec070d70 2023-10-16T14:37:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.333+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e8037c60 con 0x7fa4ec070d70 2023-10-16T14:37:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.333+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8004030 con 0x7fa4ec070d70 2023-10-16T14:37:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.333+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec0c70c0 con 0x7fa4ec070d70 2023-10-16T14:37:21.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.333+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec0c60a0 con 0x7fa4ec070d70 2023-10-16T14:37:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.334+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec04e9b0 con 0x7fa4ec070d70 2023-10-16T14:37:21.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.335+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa4e8004190 con 0x7fa4ec070d70 2023-10-16T14:37:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.336+0000 7fa4e27fc640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa4d006af90 0x7fa4d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.336+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa4e80ac7d0 con 0x7fa4ec070d70 2023-10-16T14:37:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.337+0000 7fa4f163d640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa4d006af90 0x7fa4d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:21.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.338+0000 7fa4f163d640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa4d006af90 0x7fa4d006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec159200 tx=0x7fa4dc007b50 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:21.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.341+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa4e8077750 con 0x7fa4ec070d70 2023-10-16T14:37:21.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.575+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa4ec0cf520 con 0x7fa4ec070d70 2023-10-16T14:37:21.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.576+0000 7fa4e27fc640 1 -- 172.21.15.29:0/107193328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa4e807ad70 con 0x7fa4ec070d70 2023-10-16T14:37:21.580 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:37:21.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa4d006af90 msgr2=0x7fa4d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa4d006af90 0x7fa4d006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec159200 tx=0x7fa4dc007b50 comp rx=0 tx=0).stop 2023-10-16T14:37:21.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 msgr2=0x7fa4ec155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:21.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 0x7fa4ec155fd0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80090b0 tx=0x7fa4e80046c0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 shutdown_connections 2023-10-16T14:37:21.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fa4d006af90 0x7fa4d006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa4ec15aad0 0x7fa4ec1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4ec0712c0 0x7fa4ec0c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 --2- 172.21.15.29:0/107193328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4ec070d70 0x7fa4ec155fd0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:21.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 >> 172.21.15.29:0/107193328 conn(0x7fa4ec0b0a90 msgr2=0x7fa4ec0ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:21.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 shutdown_connections 2023-10-16T14:37:21.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:21.579+0000 7fa4f2e40640 1 -- 172.21.15.29:0/107193328 wait complete. 2023-10-16T14:37:21.601 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:37:21.601 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:37:21.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T14:37:21.602 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:37:21.602 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:37:21.602 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:37:21.603 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:37:21.603 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:37:21.603 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:37:21.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:21.604 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T14:37:21.659 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:37:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:37:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:37:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T14:37:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.661 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:37:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:37:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T14:37:21.661 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:37:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:37:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T14:37:21.662 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:37:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T14:37:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T14:37:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:37:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41083sec preferred_lft 41083sec 2023-10-16T14:37:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T14:37:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.663 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:37:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T14:37:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T14:37:21.664 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:37:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:37:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:37:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T14:37:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.665 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:37:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:37:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T14:37:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.665 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:37:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:37:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T14:37:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:37:21.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:21.667 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:37:21.767 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T14:37:21.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:21.772 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T14:37:21.828 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T14:37:21.828 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:37:21.828 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:37:21.828 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:37:21.828 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:37:21.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:21.830 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T14:37:21.830 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T14:37:21.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:37:21.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:37:21.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:37:21.832 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T14:37:22.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:22.020 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T14:37:22.074 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:37:22.074 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:37:22.075 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:37:22.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:22.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T14:37:22.185 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:37:22.185 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:37:22.238 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:37:22.238 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:37:22.238 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:37:22.290 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T14:37:22.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:22.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:37:22.438 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:37:22.439 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:37:22.439 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:37:22.462 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:37:22.462 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:37:22.462 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:37:22.462 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:37:22.463 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:37:22.490 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T14:37:22.490 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T14:37:22.490 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T14:37:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T14:37:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T14:37:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T14:37:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T14:37:22.491 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T14:37:22.492 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:37:22.492 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:37:22.493 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:37:22.493 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:37:22.493 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:37:22.497 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:37:22.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:22.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:37:22.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:22.592 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:37:22.663 INFO:teuthology.orchestra.run.smithi029.stdout:client6710 2023-10-16T14:37:22.663 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T14:37:22.664 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:37:22.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:37:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.944+0000 7fe9bec47640 1 Processor -- start 2023-10-16T14:37:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.944+0000 7fe9bec47640 1 -- start start 2023-10-16T14:37:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bec47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9b80ccb00 0x7fe9b80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bec47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe9b80ad4d0 0x7fe9b80ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bec47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b80cd4a0 con 0x7fe9b8153740 2023-10-16T14:37:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bec47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b815ac60 con 0x7fe9b80ccb00 2023-10-16T14:37:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bec47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b8070030 con 0x7fe9b80ad4d0 2023-10-16T14:37:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bdc45640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe9b8153740 0x7fe9b815a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43892/0 (socket says 172.21.15.29:43892) 2023-10-16T14:37:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bdc45640 1 -- 172.21.15.29:0/2121979493 learned_addr learned my addr 172.21.15.29:0/2121979493 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bd444640 1 --2- 172.21.15.29:0/2121979493 >> v2:172.21.15.29:3300/0 conn(0x7fe9b80ccb00 0x7fe9b80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bcc43640 1 -- 172.21.15.29:0/2121979493 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017889908 0 0) 0x7fe9b80cd4a0 con 0x7fe9b8153740 2023-10-16T14:37:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bd444640 1 -- 172.21.15.29:0/2121979493 >> v2:172.21.15.170:3300/0 conn(0x7fe9b80ad4d0 msgr2=0x7fe9b80ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bd444640 1 --2- 172.21.15.29:0/2121979493 >> v2:172.21.15.170:3300/0 conn(0x7fe9b80ad4d0 0x7fe9b80ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.946+0000 7fe9bd444640 1 -- 172.21.15.29:0/2121979493 >> v1:172.21.15.29:6789/0 conn(0x7fe9b8153740 legacy=0x7fe9b815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.947+0000 7fe9bd444640 1 -- 172.21.15.29:0/2121979493 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8070310 con 0x7fe9b80ccb00 2023-10-16T14:37:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.948+0000 7fe9bd444640 1 --2- 172.21.15.29:0/2121979493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80ccf60 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8157330 tx=0x7fe9a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ddaa31422b9059 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.948+0000 7fe9bcc43640 1 -- 172.21.15.29:0/2121979493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8031830 con 0x7fe9b80ccb00 2023-10-16T14:37:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.948+0000 7fe9bcc43640 1 -- 172.21.15.29:0/2121979493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a8031990 con 0x7fe9b80ccb00 2023-10-16T14:37:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.948+0000 7fe9bcc43640 1 -- 172.21.15.29:0/2121979493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8031d20 con 0x7fe9b80ccb00 2023-10-16T14:37:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.948+0000 7fe9bec47640 1 -- 172.21.15.29:0/2121979493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 msgr2=0x7fe9b80ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.948+0000 7fe9bec47640 1 --2- 172.21.15.29:0/2121979493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80ccf60 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8157330 tx=0x7fe9a802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:37:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.949+0000 7fe9bec47640 1 -- 172.21.15.29:0/2121979493 shutdown_connections 2023-10-16T14:37:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.949+0000 7fe9bec47640 1 --2- 172.21.15.29:0/2121979493 >> v2:172.21.15.170:3300/0 conn(0x7fe9b80ad4d0 0x7fe9b80ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.949+0000 7fe9bec47640 1 --2- 172.21.15.29:0/2121979493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80ccf60 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.949+0000 7fe9bec47640 1 -- 172.21.15.29:0/2121979493 >> 172.21.15.29:0/2121979493 conn(0x7fe9b80ce6a0 msgr2=0x7fe9b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.949+0000 7fe9bec47640 1 -- 172.21.15.29:0/2121979493 shutdown_connections 2023-10-16T14:37:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.949+0000 7fe9bec47640 1 -- 172.21.15.29:0/2121979493 wait complete. 2023-10-16T14:37:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.950+0000 7fe9bec47640 1 Processor -- start 2023-10-16T14:37:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.951+0000 7fe9bec47640 1 -- start start 2023-10-16T14:37:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.951+0000 7fe9bec47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9b80ad4d0 0x7fe9b80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.951+0000 7fe9bec47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.951+0000 7fe9bec47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9b8153740 0x7fe9b80c3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bec47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b814ecd0 con 0x7fe9b80ccb00 2023-10-16T14:37:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bec47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b80b3a00 con 0x7fe9b8153740 2023-10-16T14:37:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bec47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b80b3ce0 con 0x7fe9b80ad4d0 2023-10-16T14:37:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35382/0 (socket says 172.21.15.29:35382) 2023-10-16T14:37:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 -- 172.21.15.29:0/4221708730 learned_addr learned my addr 172.21.15.29:0/4221708730 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bdc45640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9b80ad4d0 0x7fe9b80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 -- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9b80ad4d0 msgr2=0x7fe9b80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9b80ad4d0 0x7fe9b80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 -- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9b8153740 msgr2=0x7fe9b80c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9be446640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9b8153740 0x7fe9b80c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9b8153740 0x7fe9b80c3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.952+0000 7fe9bd444640 1 -- 172.21.15.29:0/4221708730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8009400 con 0x7fe9b80ccb00 2023-10-16T14:37:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.953+0000 7fe9bd444640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80b2f40 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009890 tx=0x7fe9a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.953+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a803ba70 con 0x7fe9b80ccb00 2023-10-16T14:37:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.953+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a803bbd0 con 0x7fe9b80ccb00 2023-10-16T14:37:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.953+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b80b3fc0 con 0x7fe9b80ccb00 2023-10-16T14:37:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.953+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a80453f0 con 0x7fe9b80ccb00 2023-10-16T14:37:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.953+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b814e5b0 con 0x7fe9b80ccb00 2023-10-16T14:37:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.954+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b80b4200 con 0x7fe9b80ccb00 2023-10-16T14:37:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.955+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe9a8045550 con 0x7fe9b80ccb00 2023-10-16T14:37:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.957+0000 7fe9ae7fc640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe99c06afc0 0x7fe99c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.958+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe9a80abbf0 con 0x7fe9b80ccb00 2023-10-16T14:37:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.958+0000 7fe9bdc45640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe99c06afc0 0x7fe99c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.959+0000 7fe9bdc45640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe99c06afc0 0x7fe99c06d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4000f60 tx=0x7fe9b4009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:22.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:22.962+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe9b80b4200 con 0x7fe9b80ccb00 2023-10-16T14:37:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.183+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 --> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9b80b3e50 con 0x7fe99c06afc0 2023-10-16T14:37:23.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.187+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/4221708730 <== mgr.7138 v2:172.21.15.170:6832/1985319551 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe9b80b3e50 con 0x7fe99c06afc0 2023-10-16T14:37:23.191 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T14:37:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T14:37:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T14:37:23.192 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T14:37:23.192 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T14:37:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe99c06afc0 msgr2=0x7fe99c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe99c06afc0 0x7fe99c06d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4000f60 tx=0x7fe9b4009040 comp rx=0 tx=0).stop 2023-10-16T14:37:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 msgr2=0x7fe9b80b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80b2f40 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009890 tx=0x7fe9a80046c0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 shutdown_connections 2023-10-16T14:37:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7fe99c06afc0 0x7fe99c06d480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b80ccb00 0x7fe9b80b2f40 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe9b8153740 0x7fe9b80c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 --2- 172.21.15.29:0/4221708730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe9b80ad4d0 0x7fe9b80b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.190+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 >> 172.21.15.29:0/4221708730 conn(0x7fe9b80ce6a0 msgr2=0x7fe9b8152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.191+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 shutdown_connections 2023-10-16T14:37:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.191+0000 7fe9bec47640 1 -- 172.21.15.29:0/4221708730 wait complete. 2023-10-16T14:37:23.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:37:23.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.440+0000 7f28b55b8640 1 Processor -- start 2023-10-16T14:37:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.440+0000 7f28b55b8640 1 -- start start 2023-10-16T14:37:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28b55b8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f28b0159950 0x7f28b015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28b55b8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28b006c810 0x7f28b00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28b55b8640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b015aef0 con 0x7f28b0070450 2023-10-16T14:37:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28b55b8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b0070e20 con 0x7f28b006c810 2023-10-16T14:37:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28b55b8640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b015e490 con 0x7f28b0159950 2023-10-16T14:37:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28b4db7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28b006c810 0x7f28b00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28a7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f28b0070450 0x7f28b006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43894/0 (socket says 172.21.15.29:43894) 2023-10-16T14:37:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.441+0000 7f28a7fff640 1 -- 172.21.15.29:0/96063890 learned_addr learned my addr 172.21.15.29:0/96063890 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.442+0000 7f28b4db7640 1 -- 172.21.15.29:0/96063890 >> v2:172.21.15.170:3300/0 conn(0x7f28b0159950 msgr2=0x7f28b015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.442+0000 7f28b4db7640 1 --2- 172.21.15.29:0/96063890 >> v2:172.21.15.170:3300/0 conn(0x7f28b0159950 0x7f28b015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.442+0000 7f28b4db7640 1 -- 172.21.15.29:0/96063890 >> v1:172.21.15.29:6789/0 conn(0x7f28b0070450 legacy=0x7f28b006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.442+0000 7f28b4db7640 1 -- 172.21.15.29:0/96063890 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b015d910 con 0x7f28b006c810 2023-10-16T14:37:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.443+0000 7f28b4db7640 1 --2- 172.21.15.29:0/96063890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b00708e0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f28b00ca950 tx=0x7f28a802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c892eea02a98387a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.443+0000 7f28af7fe640 1 -- 172.21.15.29:0/96063890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a802cbe0 con 0x7f28b006c810 2023-10-16T14:37:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.443+0000 7f28af7fe640 1 -- 172.21.15.29:0/96063890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a802cd40 con 0x7f28b006c810 2023-10-16T14:37:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.443+0000 7f28af7fe640 1 -- 172.21.15.29:0/96063890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8034660 con 0x7f28b006c810 2023-10-16T14:37:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 -- 172.21.15.29:0/96063890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 msgr2=0x7f28b00708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 --2- 172.21.15.29:0/96063890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b00708e0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f28b00ca950 tx=0x7f28a802be30 comp rx=0 tx=0).stop 2023-10-16T14:37:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 -- 172.21.15.29:0/96063890 shutdown_connections 2023-10-16T14:37:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 --2- 172.21.15.29:0/96063890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b00708e0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 --2- 172.21.15.29:0/96063890 >> v2:172.21.15.170:3300/0 conn(0x7f28b0159950 0x7f28b015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 -- 172.21.15.29:0/96063890 >> 172.21.15.29:0/96063890 conn(0x7f28b00ce6a0 msgr2=0x7f28b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 -- 172.21.15.29:0/96063890 shutdown_connections 2023-10-16T14:37:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.444+0000 7f28b55b8640 1 -- 172.21.15.29:0/96063890 wait complete. 2023-10-16T14:37:23.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.446+0000 7f28b55b8640 1 Processor -- start 2023-10-16T14:37:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.446+0000 7f28b55b8640 1 -- start start 2023-10-16T14:37:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28b55b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b01545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28b55b8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28b0070450 0x7f28b015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28b55b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b0159950 0x7f28b0153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28b55b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00c34f0 con 0x7f28b006c810 2023-10-16T14:37:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28b55b8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b015cb00 con 0x7f28b0070450 2023-10-16T14:37:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28b55b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00c2ea0 con 0x7f28b0159950 2023-10-16T14:37:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b01545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b01545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35394/0 (socket says 172.21.15.29:35394) 2023-10-16T14:37:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.447+0000 7f28affff640 1 -- 172.21.15.29:0/3863651453 learned_addr learned my addr 172.21.15.29:0/3863651453 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28affff640 1 -- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b0159950 msgr2=0x7f28b0153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28affff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b0159950 0x7f28b0153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28affff640 1 -- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28b0070450 msgr2=0x7f28b015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28a7fff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28b0070450 0x7f28b015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28affff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28b0070450 0x7f28b015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28affff640 1 -- 172.21.15.29:0/3863651453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8009a70 con 0x7f28b006c810 2023-10-16T14:37:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28a7fff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28b0070450 0x7f28b015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28affff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b01545b0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f28a000b410 tx=0x7f28a000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.448+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0011830 con 0x7f28b006c810 2023-10-16T14:37:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.449+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a00119b0 con 0x7f28b006c810 2023-10-16T14:37:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.449+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0011d60 con 0x7f28b006c810 2023-10-16T14:37:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.449+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b00c3180 con 0x7f28b006c810 2023-10-16T14:37:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.449+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b00c2b70 con 0x7f28b006c810 2023-10-16T14:37:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.450+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b004e9b0 con 0x7f28b006c810 2023-10-16T14:37:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.451+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f28a001a400 con 0x7f28b006c810 2023-10-16T14:37:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.452+0000 7f28ad7fa640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f289006af90 0x7f289006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.452+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f28a0089960 con 0x7f28b006c810 2023-10-16T14:37:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.453+0000 7f28a7fff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f289006af90 0x7f289006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.454+0000 7f28a7fff640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f289006af90 0x7f289006d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f28b00b44c0 tx=0x7f2898009040 comp rx=0 tx=0).ready entity=mgr.7138 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:23.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.457+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28a00548d0 con 0x7f28b006c810 2023-10-16T14:37:23.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:23.702+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f28b00b4620 con 0x7f28b006c810 2023-10-16T14:37:24.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.363+0000 7f28ad7fa640 1 -- 172.21.15.29:0/3863651453 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v78) v1 ==== 92+0+0 (secure 0 0 0) 0x7f28a0015020 con 0x7f28b006c810 2023-10-16T14:37:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f289006af90 msgr2=0x7f289006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f289006af90 0x7f289006d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f28b00b44c0 tx=0x7f2898009040 comp rx=0 tx=0).stop 2023-10-16T14:37:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 msgr2=0x7f28b01545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b01545b0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f28a000b410 tx=0x7f28a000b8d0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 shutdown_connections 2023-10-16T14:37:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f289006af90 0x7f289006d450 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f28b0159950 0x7f28b0153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f28b0070450 0x7f28b015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 --2- 172.21.15.29:0/3863651453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28b006c810 0x7f28b01545b0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.367+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 >> 172.21.15.29:0/3863651453 conn(0x7f28b00ce6a0 msgr2=0x7f28b00ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.368+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 shutdown_connections 2023-10-16T14:37:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.368+0000 7f28b55b8640 1 -- 172.21.15.29:0/3863651453 wait complete. 2023-10-16T14:37:24.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:37:24.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.600+0000 7ff2e9bb0640 1 Processor -- start 2023-10-16T14:37:24.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.601+0000 7ff2e9bb0640 1 -- start start 2023-10-16T14:37:24.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.602+0000 7ff2e9bb0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2e41618b0 0x7ff2e4065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:24.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.602+0000 7ff2e9bb0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2e4065ff0 0x7ff2e40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:24.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.602+0000 7ff2e9bb0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e406cee0 con 0x7ff2e415d330 2023-10-16T14:37:24.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.602+0000 7ff2e9bb0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e406b1e0 con 0x7ff2e41618b0 2023-10-16T14:37:24.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.602+0000 7ff2e9bb0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e40663d0 con 0x7ff2e4065ff0 2023-10-16T14:37:24.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2e41618b0 0x7ff2e4065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:24.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2e41618b0 0x7ff2e4065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55594/0 (socket says 172.21.15.29:55594) 2023-10-16T14:37:24.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 -- 172.21.15.29:0/3650532408 learned_addr learned my addr 172.21.15.29:0/3650532408 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:24.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e37fe640 1 --1- 172.21.15.29:0/3650532408 >> v1:172.21.15.29:6789/0 conn(0x7ff2e415d330 0x7ff2e406c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49908/0 (socket says 172.21.15.29:49908) 2023-10-16T14:37:24.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e3fff640 1 --2- 172.21.15.29:0/3650532408 >> v2:172.21.15.170:3300/0 conn(0x7ff2e4065ff0 0x7ff2e40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:24.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 -- 172.21.15.29:0/3650532408 >> v2:172.21.15.170:3300/0 conn(0x7ff2e4065ff0 msgr2=0x7ff2e40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 --2- 172.21.15.29:0/3650532408 >> v2:172.21.15.170:3300/0 conn(0x7ff2e4065ff0 0x7ff2e40ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 -- 172.21.15.29:0/3650532408 >> v1:172.21.15.29:6789/0 conn(0x7ff2e415d330 legacy=0x7ff2e406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e27fc640 1 -- 172.21.15.29:0/3650532408 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038245100 0 0) 0x7ff2e406cee0 con 0x7ff2e415d330 2023-10-16T14:37:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.603+0000 7ff2e2ffd640 1 -- 172.21.15.29:0/3650532408 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40d0890 con 0x7ff2e41618b0 2023-10-16T14:37:24.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.604+0000 7ff2e2ffd640 1 --2- 172.21.15.29:0/3650532408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4065ab0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ff2e406f340 tx=0x7ff2cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f823103fafc33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:24.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.605+0000 7ff2e27fc640 1 -- 172.21.15.29:0/3650532408 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc031870 con 0x7ff2e41618b0 2023-10-16T14:37:24.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.605+0000 7ff2e27fc640 1 -- 172.21.15.29:0/3650532408 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc0319d0 con 0x7ff2e41618b0 2023-10-16T14:37:24.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.605+0000 7ff2e27fc640 1 -- 172.21.15.29:0/3650532408 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc031d60 con 0x7ff2e41618b0 2023-10-16T14:37:24.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/3650532408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 msgr2=0x7ff2e4065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/3650532408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4065ab0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ff2e406f340 tx=0x7ff2cc02b910 comp rx=0 tx=0).stop 2023-10-16T14:37:24.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/3650532408 shutdown_connections 2023-10-16T14:37:24.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/3650532408 >> v2:172.21.15.170:3300/0 conn(0x7ff2e4065ff0 0x7ff2e40ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/3650532408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4065ab0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/3650532408 >> 172.21.15.29:0/3650532408 conn(0x7ff2e40d0ab0 msgr2=0x7ff2e40d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:24.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/3650532408 shutdown_connections 2023-10-16T14:37:24.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.606+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/3650532408 wait complete. 2023-10-16T14:37:24.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.608+0000 7ff2e9bb0640 1 Processor -- start 2023-10-16T14:37:24.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.608+0000 7ff2e9bb0640 1 -- start start 2023-10-16T14:37:24.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e9bb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2e4065ff0 0x7ff2e40c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:24.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e9bb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2e415d330 0x7ff2e41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:24.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e9bb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:24.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e9bb0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40caa80 con 0x7ff2e41618b0 2023-10-16T14:37:24.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e9bb0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40ca7a0 con 0x7ff2e4065ff0 2023-10-16T14:37:24.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e9bb0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40c6200 con 0x7ff2e415d330 2023-10-16T14:37:24.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:24.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55598/0 (socket says 172.21.15.29:55598) 2023-10-16T14:37:24.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e3fff640 1 -- 172.21.15.29:0/39925918 learned_addr learned my addr 172.21.15.29:0/39925918 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:24.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.609+0000 7ff2e2ffd640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2e415d330 0x7ff2e41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:24.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e37fe640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2e4065ff0 0x7ff2e40c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:24.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e3fff640 1 -- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2e415d330 msgr2=0x7ff2e41543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e3fff640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2e415d330 0x7ff2e41543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e3fff640 1 -- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2e4065ff0 msgr2=0x7ff2e40c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e3fff640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2e4065ff0 0x7ff2e40c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e3fff640 1 -- 172.21.15.29:0/39925918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0094d0 con 0x7ff2e41618b0 2023-10-16T14:37:24.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e37fe640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2e4065ff0 0x7ff2e40c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:37:24.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.610+0000 7ff2e3fff640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4159a80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc007e60 tx=0x7ff2dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:24.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.611+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc014420 con 0x7ff2e41618b0 2023-10-16T14:37:24.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.611+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc0145a0 con 0x7ff2e41618b0 2023-10-16T14:37:24.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.611+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc014950 con 0x7ff2e41618b0 2023-10-16T14:37:24.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.611+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e40c64e0 con 0x7ff2e41618b0 2023-10-16T14:37:24.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.611+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e4155340 con 0x7ff2e41618b0 2023-10-16T14:37:24.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.612+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e40c6060 con 0x7ff2e41618b0 2023-10-16T14:37:24.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.613+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff2dc014ab0 con 0x7ff2e41618b0 2023-10-16T14:37:24.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.614+0000 7ff2dbfff640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 0x7ff2c406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:24.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.614+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff2dc08a0a0 con 0x7ff2e41618b0 2023-10-16T14:37:24.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.614+0000 7ff2e37fe640 1 -- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 msgr2=0x7ff2c406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:24.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.614+0000 7ff2e37fe640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 0x7ff2c406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:37:24.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.618+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2e40c6060 con 0x7ff2e41618b0 2023-10-16T14:37:24.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.813+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff2e40c6060 con 0x7ff2e41618b0 2023-10-16T14:37:24.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.815+0000 7ff2e37fe640 1 -- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 msgr2=0x7ff2c406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:24.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.815+0000 7ff2e37fe640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 0x7ff2c406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:37:24.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.819+0000 7ff2dbfff640 1 -- 172.21.15.29:0/39925918 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff2e40c6060 con 0x7ff2e41618b0 2023-10-16T14:37:24.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.821+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 msgr2=0x7ff2c406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:37:24.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.821+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 0x7ff2c406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.821+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 msgr2=0x7ff2e4159a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:24.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.821+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4159a80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc007e60 tx=0x7ff2dc00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 shutdown_connections 2023-10-16T14:37:24.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7ff2c406afe0 0x7ff2c406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2e41618b0 0x7ff2e4159a80 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2e415d330 0x7ff2e41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 --2- 172.21.15.29:0/39925918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2e4065ff0 0x7ff2e40c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 >> 172.21.15.29:0/39925918 conn(0x7ff2e40d0ab0 msgr2=0x7ff2e415be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 shutdown_connections 2023-10-16T14:37:24.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:24.822+0000 7ff2e9bb0640 1 -- 172.21.15.29:0/39925918 wait complete. 2023-10-16T14:37:24.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:37:24.904 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:24.901+0000 7fbf28256040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:37:24.920 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:24.917+0000 7f6dd5613040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:37:24.956 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:24.954+0000 7fbf28256040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:37:24.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:24.973+0000 7f6dd5613040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:37:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.053+0000 7f91a59f4640 1 Processor -- start 2023-10-16T14:37:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.054+0000 7f91a59f4640 1 -- start start 2023-10-16T14:37:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.055+0000 7f91a59f4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f91a015cd70 0x7f91a00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.055+0000 7f91a59f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a015d150 con 0x7f91a00b8480 2023-10-16T14:37:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.055+0000 7f91a59f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00d1180 con 0x7f91a00cded0 2023-10-16T14:37:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.055+0000 7f91a59f4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00d1460 con 0x7f91a015cd70 2023-10-16T14:37:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91a00b8480 0x7f91a00cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49916/0 (socket says 172.21.15.29:49916) 2023-10-16T14:37:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919ffff640 1 -- 172.21.15.29:0/975952174 learned_addr learned my addr 172.21.15.29:0/975952174 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f91a49f2640 1 --2- 172.21.15.29:0/975952174 >> v2:172.21.15.170:3300/0 conn(0x7f91a015cd70 0x7f91a00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549890088 0 0) 0x7f91a015d150 con 0x7f91a00b8480 2023-10-16T14:37:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9190003520 con 0x7f91a00b8480 2023-10-16T14:37:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345750749 0 0) 0x7f9190003520 con 0x7f91a00b8480 2023-10-16T14:37:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 >> v2:172.21.15.170:3300/0 conn(0x7f91a015cd70 msgr2=0x7f91a00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.056+0000 7f919f7fe640 1 --2- 172.21.15.29:0/975952174 >> v2:172.21.15.170:3300/0 conn(0x7f91a015cd70 0x7f91a00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 >> v1:172.21.15.170:6789/0 conn(0x7f91a00cded0 legacy=0x7f91a00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a00d1740 con 0x7f91a00b8480 2023-10-16T14:37:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f918c001670 con 0x7f91a00b8480 2023-10-16T14:37:25.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 >> v1:172.21.15.29:6789/0 conn(0x7f91a00b8480 legacy=0x7f91a00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9190003970 0x7f9190003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91900047c0 0x7f9190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9190005180 0x7f919000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00d1460 con 0x7f91900047c0 2023-10-16T14:37:25.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919000c020 con 0x7f9190005180 2023-10-16T14:37:25.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919000c1e0 con 0x7f9190003970 2023-10-16T14:37:25.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919ffff640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91900047c0 0x7f9190004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f91a49f2640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9190003970 0x7f9190003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f91a51f3640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9190005180 0x7f919000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919ffff640 1 -- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9190003970 msgr2=0x7f9190003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919ffff640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9190003970 0x7f9190003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919ffff640 1 -- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9190005180 msgr2=0x7f919000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919ffff640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9190005180 0x7f919000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f919ffff640 1 -- 172.21.15.29:0/975952174 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190003520 con 0x7f91900047c0 2023-10-16T14:37:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.057+0000 7f91a51f3640 1 -- 172.21.15.29:0/975952174 reap_dead start 2023-10-16T14:37:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.059+0000 7f919ffff640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91900047c0 0x7f9190004c40 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f918c00c240 tx=0x7f91900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5da40151ffe663 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.059+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c005ad0 con 0x7f91900047c0 2023-10-16T14:37:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.059+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c005c30 con 0x7f91900047c0 2023-10-16T14:37:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.059+0000 7f919f7fe640 1 -- 172.21.15.29:0/975952174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c02caf0 con 0x7f91900047c0 2023-10-16T14:37:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 -- 172.21.15.29:0/975952174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91900047c0 msgr2=0x7f9190004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91900047c0 0x7f9190004c40 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f918c00c240 tx=0x7f91900057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 -- 172.21.15.29:0/975952174 shutdown_connections 2023-10-16T14:37:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 --2- 172.21.15.29:0/975952174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91900047c0 0x7f9190004c40 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 -- 172.21.15.29:0/975952174 >> 172.21.15.29:0/975952174 conn(0x7f91a00cf570 msgr2=0x7f91a00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 -- 172.21.15.29:0/975952174 shutdown_connections 2023-10-16T14:37:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.060+0000 7f91a59f4640 1 -- 172.21.15.29:0/975952174 wait complete. 2023-10-16T14:37:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.062+0000 7f91a59f4640 1 Processor -- start 2023-10-16T14:37:25.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.062+0000 7f91a59f4640 1 -- start start 2023-10-16T14:37:25.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a59f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91900047c0 0x7f91a015d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a59f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 0x7f91a01f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a59f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a01f2710 0x7f91a01f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a59f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01f3100 con 0x7f91a015d9a0 2023-10-16T14:37:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a59f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a015edd0 con 0x7f91a01f2710 2023-10-16T14:37:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a59f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a015eaf0 con 0x7f91900047c0 2023-10-16T14:37:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f919ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 0x7f91a01f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f919ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 0x7f91a01f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55618/0 (socket says 172.21.15.29:55618) 2023-10-16T14:37:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f919ffff640 1 -- 172.21.15.29:0/175589504 learned_addr learned my addr 172.21.15.29:0/175589504 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a49f2640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91900047c0 0x7f91a015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.063+0000 7f91a51f3640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a01f2710 0x7f91a01f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919ffff640 1 -- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91900047c0 msgr2=0x7f91a015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919ffff640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91900047c0 0x7f91a015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919ffff640 1 -- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a01f2710 msgr2=0x7f91a01f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919ffff640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a01f2710 0x7f91a01f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919ffff640 1 -- 172.21.15.29:0/175589504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a015abc0 con 0x7f91a015d9a0 2023-10-16T14:37:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919ffff640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 0x7f91a01f21d0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f918c001930 tx=0x7f918c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c039030 con 0x7f91a015d9a0 2023-10-16T14:37:25.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c02ccc0 con 0x7f91a015d9a0 2023-10-16T14:37:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c03c040 con 0x7f91a015d9a0 2023-10-16T14:37:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.064+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a015e1c0 con 0x7f91a015d9a0 2023-10-16T14:37:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.065+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a015e780 con 0x7f91a015d9a0 2023-10-16T14:37:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.066+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a01f3390 con 0x7f91a015d9a0 2023-10-16T14:37:25.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.066+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f918c032070 con 0x7f91a015d9a0 2023-10-16T14:37:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.067+0000 7f919d7fa640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 0x7f918406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.068+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f918c0aab80 con 0x7f91a015d9a0 2023-10-16T14:37:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.068+0000 7f91a49f2640 1 -- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 msgr2=0x7f918406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.068+0000 7f91a49f2640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 0x7f918406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:37:25.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.076+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f91a01f3390 con 0x7f91a015d9a0 2023-10-16T14:37:25.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.266+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f91a015e940 con 0x7f91a015d9a0 2023-10-16T14:37:25.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.269+0000 7f91a49f2640 1 -- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 msgr2=0x7f918406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:25.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.269+0000 7f91a49f2640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 0x7f918406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:37:25.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.272+0000 7f919d7fa640 1 -- 172.21.15.29:0/175589504 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7f918c0aae60 con 0x7f91a015d9a0 2023-10-16T14:37:25.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 msgr2=0x7f918406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:37:25.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 0x7f918406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 msgr2=0x7f91a01f21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 0x7f91a01f21d0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f918c001930 tx=0x7f918c001a00 comp rx=0 tx=0).stop 2023-10-16T14:37:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 shutdown_connections 2023-10-16T14:37:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f918406b010 0x7f918406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91a015d9a0 0x7f91a01f21d0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f91a01f2710 0x7f91a01f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 --2- 172.21.15.29:0/175589504 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91900047c0 0x7f91a015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 >> 172.21.15.29:0/175589504 conn(0x7f91a00cf570 msgr2=0x7f91a00b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 shutdown_connections 2023-10-16T14:37:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.274+0000 7f91a59f4640 1 -- 172.21.15.29:0/175589504 wait complete. 2023-10-16T14:37:25.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:37:25.297 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:37:25.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-10-16T14:37:25.403 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.400+0000 7fbf28256040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:37:25.455 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.452+0000 7fbf28256040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:37:25.460 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.456+0000 7f6dd5613040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:37:25.505 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.503+0000 7fbf28256040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:37:25.515 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.512+0000 7f6dd5613040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:37:25.565 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.562+0000 7fbf28256040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:37:25.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.563+0000 7f85f41b6640 1 Processor -- start 2023-10-16T14:37:25.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.563+0000 7f85f41b6640 1 -- start start 2023-10-16T14:37:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f41b6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f85ec15d020 0x7f85ec160a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f41b6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec0b5dd0 con 0x7f85ec0b89e0 2023-10-16T14:37:25.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f41b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec161980 con 0x7f85ec0cfdc0 2023-10-16T14:37:25.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f41b6640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec1613c0 con 0x7f85ec15d020 2023-10-16T14:37:25.569 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f6dd5613040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:37:25.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f1751640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f85ec0b89e0 0x7f85ec0cf6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49918/0 (socket says 172.21.15.29:49918) 2023-10-16T14:37:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f1751640 1 -- 172.21.15.29:0/2596754805 learned_addr learned my addr 172.21.15.29:0/2596754805 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f1f52640 1 --2- 172.21.15.29:0/2596754805 >> v2:172.21.15.170:3300/0 conn(0x7f85ec15d020 0x7f85ec160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220290900 0 0) 0x7f85ec0b5dd0 con 0x7f85ec0b89e0 2023-10-16T14:37:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.565+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d8003520 con 0x7f85ec0b89e0 2023-10-16T14:37:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3932551443 0 0) 0x7f85d8003520 con 0x7f85ec0b89e0 2023-10-16T14:37:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 >> v2:172.21.15.170:3300/0 conn(0x7f85ec15d020 msgr2=0x7f85ec160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 --2- 172.21.15.29:0/2596754805 >> v2:172.21.15.170:3300/0 conn(0x7f85ec15d020 0x7f85ec160a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 >> v1:172.21.15.170:6789/0 conn(0x7f85ec0cfdc0 legacy=0x7f85ec0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec1616a0 con 0x7f85ec0b89e0 2023-10-16T14:37:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f85dc001670 con 0x7f85ec0b89e0 2023-10-16T14:37:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 >> v1:172.21.15.29:6789/0 conn(0x7f85ec0b89e0 legacy=0x7f85ec0cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85d8003970 0x7f85d8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85d8005180 0x7f85d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec1613c0 con 0x7f85d80047c0 2023-10-16T14:37:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d800c000 con 0x7f85d8003970 2023-10-16T14:37:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d800c1c0 con 0x7f85d8005180 2023-10-16T14:37:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.566+0000 7f85f1751640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f2753640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85d8005180 0x7f85d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f1f52640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85d8003970 0x7f85d8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f1751640 1 -- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85d8005180 msgr2=0x7f85d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f1751640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85d8005180 0x7f85d800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f1751640 1 -- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85d8003970 msgr2=0x7f85d8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f1751640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85d8003970 0x7f85d8003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f1751640 1 -- 172.21.15.29:0/2596754805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d8003520 con 0x7f85d80047c0 2023-10-16T14:37:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f2753640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85d8005180 0x7f85d800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.567+0000 7f85f2753640 1 -- 172.21.15.29:0/2596754805 reap_dead start 2023-10-16T14:37:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.568+0000 7f85f1751640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85d8004c40 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c210 tx=0x7f85d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2a8092ee152588 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.568+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc005ad0 con 0x7f85d80047c0 2023-10-16T14:37:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.568+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc005c50 con 0x7f85d80047c0 2023-10-16T14:37:25.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.568+0000 7f85f0f50640 1 -- 172.21.15.29:0/2596754805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc02c8d0 con 0x7f85d80047c0 2023-10-16T14:37:25.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 -- 172.21.15.29:0/2596754805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 msgr2=0x7f85d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85d8004c40 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c210 tx=0x7f85d80057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 -- 172.21.15.29:0/2596754805 shutdown_connections 2023-10-16T14:37:25.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 --2- 172.21.15.29:0/2596754805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85d8004c40 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 -- 172.21.15.29:0/2596754805 >> 172.21.15.29:0/2596754805 conn(0x7f85ec06a320 msgr2=0x7f85ec06a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:25.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 -- 172.21.15.29:0/2596754805 shutdown_connections 2023-10-16T14:37:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.569+0000 7f85f41b6640 1 -- 172.21.15.29:0/2596754805 wait complete. 2023-10-16T14:37:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.571+0000 7f85f41b6640 1 Processor -- start 2023-10-16T14:37:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.571+0000 7f85f41b6640 1 -- start start 2023-10-16T14:37:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f41b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85ec15ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f41b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85ec150ed0 0x7f85ec15c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f41b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85ec152470 0x7f85ec152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f41b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec151330 con 0x7f85d80047c0 2023-10-16T14:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f41b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec15c850 con 0x7f85ec152470 2023-10-16T14:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f41b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec15fa70 con 0x7f85ec150ed0 2023-10-16T14:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85ec15ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85ec15ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55638/0 (socket says 172.21.15.29:55638) 2023-10-16T14:37:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 -- 172.21.15.29:0/3874679894 learned_addr learned my addr 172.21.15.29:0/3874679894 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85ec150ed0 0x7f85ec15c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f2753640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85ec152470 0x7f85ec152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85ec150ed0 msgr2=0x7f85ec15c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f85ec150ed0 0x7f85ec15c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85ec152470 msgr2=0x7f85ec152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f85ec152470 0x7f85ec152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.572+0000 7f85f1f52640 1 -- 172.21.15.29:0/3874679894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec15f4b0 con 0x7f85d80047c0 2023-10-16T14:37:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.573+0000 7f85f1f52640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85ec15ee20 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f85e80108e0 tx=0x7f85e8010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.573+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e800eb60 con 0x7f85d80047c0 2023-10-16T14:37:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.573+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e800ecc0 con 0x7f85d80047c0 2023-10-16T14:37:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.573+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e801c400 con 0x7f85d80047c0 2023-10-16T14:37:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.573+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec15f790 con 0x7f85d80047c0 2023-10-16T14:37:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.573+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec15d880 con 0x7f85d80047c0 2023-10-16T14:37:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.574+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec15d400 con 0x7f85d80047c0 2023-10-16T14:37:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.575+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f85e801c560 con 0x7f85d80047c0 2023-10-16T14:37:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.576+0000 7f85e27fc640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 0x7f85d006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.576+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f85e8089f00 con 0x7f85d80047c0 2023-10-16T14:37:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.577+0000 7f85f1751640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 msgr2=0x7f85d006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.577+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 0x7f85d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:37:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.581+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f85ec15d400 con 0x7f85d80047c0 2023-10-16T14:37:25.615 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.613+0000 7fbf28256040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:37:25.635 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.631+0000 7f6dd5613040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:37:25.688 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.684+0000 7f6dd5613040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:37:25.726 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.723+0000 7fbf28256040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:37:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.778+0000 7f85f1751640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 msgr2=0x7f85d006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.778+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 0x7f85d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:37:25.787 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.784+0000 7fbf28256040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:37:25.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:25.796+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 --> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f85ec0d1380 con 0x7f85d006afe0 2023-10-16T14:37:25.804 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.800+0000 7f6dd5613040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:37:25.835 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.832+0000 7fbf28256040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:37:25.870 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.866+0000 7f6dd5613040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:37:25.893 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.889+0000 7fbf28256040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:37:25.921 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.918+0000 7f6dd5613040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:37:25.942 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:25.939+0000 7fbf28256040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:37:25.982 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:25.979+0000 7f6dd5613040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:37:26.035 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.031+0000 7f6dd5613040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:37:26.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:26.179+0000 7f85f1751640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 msgr2=0x7f85d006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:26.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:26.179+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 0x7f85d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:37:26.326 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:26.323+0000 7fbf28256040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:37:26.436 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.432+0000 7f6dd5613040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:37:26.484 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:26.481+0000 7fbf28256040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:37:26.591 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:26.588+0000 7fbf28256040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:37:26.603 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.600+0000 7f6dd5613040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:37:26.641 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:26.637+0000 7fbf28256040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:37:26.717 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.713+0000 7f6dd5613040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:37:26.747 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:26.744+0000 7fbf28256040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:37:26.769 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.765+0000 7f6dd5613040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:37:26.852 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:26.848+0000 7fbf28256040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:37:26.882 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.878+0000 7f6dd5613040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:37:26.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:26.980+0000 7f85f1751640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 msgr2=0x7f85d006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1985319551 2023-10-16T14:37:26.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:26.980+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 0x7f85d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:37:26.993 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:26.990+0000 7f6dd5613040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:37:27.006 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:27.002+0000 7fbf28256040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:37:27.055 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:27.051+0000 7fbf28256040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:37:27.158 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:27.154+0000 7f6dd5613040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:37:27.173 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:27.169+0000 7fbf28256040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:37:27.210 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:27.206+0000 7f6dd5613040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:37:27.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:27.332+0000 7f6dd5613040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:37:27.993 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:27.990+0000 7fbf28256040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:37:28.227 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:28.224+0000 7f6dd5613040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:37:28.284 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.280+0000 7fbf28256040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:37:28.342 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.338+0000 7fbf28256040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:37:28.462 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.452+0000 7fbf28256040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:37:28.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:28.471+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 79) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f85e80552f0 con 0x7f85d80047c0 2023-10-16T14:37:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:28.472+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 msgr2=0x7f85d006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:37:28.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:28.472+0000 7f85e27fc640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/1985319551,v1:172.21.15.170:6833/1985319551] conn(0x7f85d006afe0 0x7f85d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:28.550 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:28.546+0000 7f6dd5613040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:37:28.611 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.607+0000 7fbea4de3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.611 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.608+0000 7fbea4de3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.611 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.608+0000 7fbea4de3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.611 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.608+0000 7fbea4de3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.612 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.608+0000 7fbea4de3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.612 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:28.609+0000 7f6dd5613040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:37:28.616 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.609+0000 7fbea7628640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.616 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.609+0000 7fbea7628640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.609+0000 7fbea7628640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.609+0000 7fbea7628640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:37:28.609+0000 7fbea7628640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:37:28.736 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T14:37:28.732+0000 7f6dd5613040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:37:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.483+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 80) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f85e8088360 con 0x7f85d80047c0 2023-10-16T14:37:29.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.484+0000 7f85e27fc640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f85d00358e0 0x7f85d0074f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.484+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 --> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f85d0035cc0 con 0x7f85d00358e0 2023-10-16T14:37:29.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.489+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f85d00358e0 0x7f85d0074f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.495+0000 7f85f1751640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f85d00358e0 0x7f85d0074f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f85dc034000 tx=0x7f85dc009030 comp rx=0 tx=0).ready entity=mgr.6722 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:29.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.654+0000 7f85e27fc640 1 -- 172.21.15.29:0/3874679894 <== mgr.6722 v2:172.21.15.170:6832/3272758872 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f85d0035cc0 con 0x7f85d00358e0 2023-10-16T14:37:29.658 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1 2023-10-16T14:37:29.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.657+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f85d00358e0 msgr2=0x7f85d0074f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:29.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.657+0000 7f85f41b6640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f85d00358e0 0x7f85d0074f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f85dc034000 tx=0x7f85dc009030 comp rx=0 tx=0).stop 2023-10-16T14:37:29.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.657+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 msgr2=0x7f85ec15ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:29.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.657+0000 7f85f41b6640 1 --2- 172.21.15.29:0/3874679894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d80047c0 0x7f85ec15ee20 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f85e80108e0 tx=0x7f85e8010da0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.658+0000 7f85f2753640 1 -- 172.21.15.29:0/3874679894 reap_dead start 2023-10-16T14:37:29.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.658+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 shutdown_connections 2023-10-16T14:37:29.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.658+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 >> 172.21.15.29:0/3874679894 conn(0x7f85ec06a320 msgr2=0x7f85ec0c67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:29.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.658+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 shutdown_connections 2023-10-16T14:37:29.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.658+0000 7f85f41b6640 1 -- 172.21.15.29:0/3874679894 wait complete. 2023-10-16T14:37:29.683 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-16T14:37:29.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-10-16T14:37:29.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.912+0000 7f5c23a61640 1 Processor -- start 2023-10-16T14:37:29.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.912+0000 7f5c23a61640 1 -- start start 2023-10-16T14:37:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.914+0000 7f5c23a61640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5c1c15af90 0x7f5c1c0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.914+0000 7f5c23a61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 0x7f5c1c0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.914+0000 7f5c23a61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0cb890 con 0x7f5c1c0cf3d0 2023-10-16T14:37:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.914+0000 7f5c23a61640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c0cef80 con 0x7f5c1c15af90 2023-10-16T14:37:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.914+0000 7f5c23a61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0cbac0 con 0x7f5c1c0cd320 2023-10-16T14:37:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.915+0000 7f5c21ffe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5c1c0cf3d0 0x7f5c1c0afeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49922/0 (socket says 172.21.15.29:49922) 2023-10-16T14:37:29.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.915+0000 7f5c21ffe640 1 -- 172.21.15.29:0/1799638301 learned_addr learned my addr 172.21.15.29:0/1799638301 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T14:37:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.915+0000 7f5c20ffc640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 0x7f5c1c0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.915+0000 7f5c217fd640 1 --2- 172.21.15.29:0/1799638301 >> v2:172.21.15.170:3300/0 conn(0x7f5c1c15af90 0x7f5c1c0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.915+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330494508 0 0) 0x7f5c1c0cb890 con 0x7f5c1c0cf3d0 2023-10-16T14:37:29.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.915+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c08003520 con 0x7f5c1c0cf3d0 2023-10-16T14:37:29.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2083855477 0 0) 0x7f5c08003520 con 0x7f5c1c0cf3d0 2023-10-16T14:37:29.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 msgr2=0x7f5c1c0cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:29.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 0x7f5c1c0cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 >> v2:172.21.15.170:3300/0 conn(0x7f5c1c15af90 msgr2=0x7f5c1c0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 --2- 172.21.15.29:0/1799638301 >> v2:172.21.15.170:3300/0 conn(0x7f5c1c15af90 0x7f5c1c0ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0cdcc0 con 0x7f5c1c0cf3d0 2023-10-16T14:37:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5c14001670 con 0x7f5c1c0cf3d0 2023-10-16T14:37:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 >> v1:172.21.15.29:6789/0 conn(0x7f5c1c0cf3d0 legacy=0x7f5c1c0afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c080039e0 0x7f5c08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 0x7f5c080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0cef80 con 0x7f5c08004d00 2023-10-16T14:37:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0cbac0 con 0x7f5c080039e0 2023-10-16T14:37:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0800b980 con 0x7f5c1c0cd320 2023-10-16T14:37:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c21ffe640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c217fd640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c080039e0 0x7f5c08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.916+0000 7f5c20ffc640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 0x7f5c080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.917+0000 7f5c21ffe640 1 -- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 msgr2=0x7f5c080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.917+0000 7f5c21ffe640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c0cd320 0x7f5c080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.917+0000 7f5c21ffe640 1 -- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c080039e0 msgr2=0x7f5c08003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.917+0000 7f5c21ffe640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c080039e0 0x7f5c08003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.917+0000 7f5c21ffe640 1 -- 172.21.15.29:0/1799638301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08003520 con 0x7f5c08004d00 2023-10-16T14:37:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.918+0000 7f5c21ffe640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c08005180 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400c0e0 tx=0x7f5c080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82513c84bd759d04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.918+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14005ad0 con 0x7f5c08004d00 2023-10-16T14:37:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.918+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14005c50 con 0x7f5c08004d00 2023-10-16T14:37:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.918+0000 7f5c13fff640 1 -- 172.21.15.29:0/1799638301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1402c7f0 con 0x7f5c08004d00 2023-10-16T14:37:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c23a61640 1 -- 172.21.15.29:0/1799638301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 msgr2=0x7f5c08005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1799638301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c08005180 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400c0e0 tx=0x7f5c080057f0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c21ffe640 1 -- 172.21.15.29:0/1799638301 reap_dead start 2023-10-16T14:37:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c23a61640 1 -- 172.21.15.29:0/1799638301 shutdown_connections 2023-10-16T14:37:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c23a61640 1 -- 172.21.15.29:0/1799638301 >> 172.21.15.29:0/1799638301 conn(0x7f5c1c099e30 msgr2=0x7f5c1c0d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c23a61640 1 -- 172.21.15.29:0/1799638301 shutdown_connections 2023-10-16T14:37:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.919+0000 7f5c23a61640 1 -- 172.21.15.29:0/1799638301 wait complete. 2023-10-16T14:37:29.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.921+0000 7f5c23a61640 1 Processor -- start 2023-10-16T14:37:29.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.921+0000 7f5c23a61640 1 -- start start 2023-10-16T14:37:29.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c23a61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c1c156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c23a61640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c1c0b5960 0x7f5c1c15b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c23a61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c06c590 0x7f5c1c15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c23a61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0b2550 con 0x7f5c08004d00 2023-10-16T14:37:29.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c23a61640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0b5050 con 0x7f5c1c0b5960 2023-10-16T14:37:29.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c23a61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c157260 con 0x7f5c1c06c590 2023-10-16T14:37:29.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c217fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c1c156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c217fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c1c156d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55644/0 (socket says 172.21.15.29:55644) 2023-10-16T14:37:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.922+0000 7f5c217fd640 1 -- 172.21.15.29:0/1649904026 learned_addr learned my addr 172.21.15.29:0/1649904026 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c21ffe640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c06c590 0x7f5c1c15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c217fd640 1 -- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c06c590 msgr2=0x7f5c1c15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c217fd640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c06c590 0x7f5c1c15b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c20ffc640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c1c0b5960 0x7f5c1c15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c217fd640 1 -- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c1c0b5960 msgr2=0x7f5c1c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c217fd640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c1c0b5960 0x7f5c1c15b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c217fd640 1 -- 172.21.15.29:0/1649904026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0b4bc0 con 0x7f5c08004d00 2023-10-16T14:37:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c217fd640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c1c156d20 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f5c18007ed0 tx=0x7f5c180115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800e030 con 0x7f5c08004d00 2023-10-16T14:37:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18016470 con 0x7f5c08004d00 2023-10-16T14:37:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c0b4290 con 0x7f5c08004d00 2023-10-16T14:37:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18016800 con 0x7f5c08004d00 2023-10-16T14:37:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.923+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c0b4850 con 0x7f5c08004d00 2023-10-16T14:37:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.925+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c04e160 con 0x7f5c08004d00 2023-10-16T14:37:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.926+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5c18016960 con 0x7f5c08004d00 2023-10-16T14:37:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.927+0000 7f5c11ffb640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f5c0006ad40 0x7f5c0006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.927+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5c1800f070 con 0x7f5c08004d00 2023-10-16T14:37:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.928+0000 7f5c20ffc640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f5c0006ad40 0x7f5c0006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.929+0000 7f5c20ffc640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f5c0006ad40 0x7f5c0006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c15a780 tx=0x7f5c0c008040 comp rx=0 tx=0).ready entity=mgr.6722 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:29.931+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5c18055610 con 0x7f5c08004d00 2023-10-16T14:37:30.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.146+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 --> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c1c0cc080 con 0x7f5c0006ad40 2023-10-16T14:37:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.293+0000 7f5c11ffb640 1 -- 172.21.15.29:0/1649904026 <== mgr.6722 v2:172.21.15.170:6832/3272758872 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f5c0c0095c0 con 0x7f5c0006ad40 2023-10-16T14:37:30.297 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path /snap_test_dir1 2023-10-16T14:37:30.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f5c0006ad40 msgr2=0x7f5c0006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f5c0006ad40 0x7f5c0006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c15a780 tx=0x7f5c0c008040 comp rx=0 tx=0).stop 2023-10-16T14:37:30.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 msgr2=0x7f5c1c156d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c1c156d20 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f5c18007ed0 tx=0x7f5c180115a0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 shutdown_connections 2023-10-16T14:37:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f5c0006ad40 0x7f5c0006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5c1c06c590 0x7f5c1c15b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5c1c0b5960 0x7f5c1c15b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 --2- 172.21.15.29:0/1649904026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c08004d00 0x7f5c1c156d20 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 >> 172.21.15.29:0/1649904026 conn(0x7f5c1c099e30 msgr2=0x7f5c1c0b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:30.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.296+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 shutdown_connections 2023-10-16T14:37:30.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.297+0000 7f5c23a61640 1 -- 172.21.15.29:0/1649904026 wait complete. 2023-10-16T14:37:30.324 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-16T14:37:30.325 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T14:37:30.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-10-16T14:37:30.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.558+0000 7f215c80e640 1 Processor -- start 2023-10-16T14:37:30.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.558+0000 7f215c80e640 1 -- start start 2023-10-16T14:37:30.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f215c80e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f215815ad70 0x7f2158069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:30.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f215c80e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21580caac0 0x7f215806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:30.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f215c80e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580cb030 con 0x7f21580c9ca0 2023-10-16T14:37:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f215c80e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21580c91c0 con 0x7f21580caac0 2023-10-16T14:37:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f215c80e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21580c94a0 con 0x7f215815ad70 2023-10-16T14:37:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f2156d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21580caac0 0x7f215806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f2156d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21580caac0 0x7f215806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55654/0 (socket says 172.21.15.29:55654) 2023-10-16T14:37:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f2156d9d640 1 -- 172.21.15.29:0/2370194289 learned_addr learned my addr 172.21.15.29:0/2370194289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:30.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f215659c640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f215815ad70 0x7f2158069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:30.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f2156d9d640 1 -- 172.21.15.29:0/2370194289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f215815ad70 msgr2=0x7f2158069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:30.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f2156d9d640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f215815ad70 0x7f2158069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.560+0000 7f2156d9d640 1 -- 172.21.15.29:0/2370194289 >> v1:172.21.15.170:6789/0 conn(0x7f21580c9ca0 legacy=0x7f21580ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:37:30.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.561+0000 7f2156d9d640 1 -- 172.21.15.29:0/2370194289 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21580c9780 con 0x7f21580caac0 2023-10-16T14:37:30.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.561+0000 7f215659c640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f215815ad70 0x7f2158069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:30.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.562+0000 7f2156d9d640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580caac0 0x7f215806a2e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f2150009e50 tx=0x7f215002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6021f84cabe29677 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:30.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.562+0000 7f2155d9b640 1 -- 172.21.15.29:0/2370194289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215002ca70 con 0x7f21580caac0 2023-10-16T14:37:30.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.562+0000 7f2155d9b640 1 -- 172.21.15.29:0/2370194289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215002cbd0 con 0x7f21580caac0 2023-10-16T14:37:30.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.562+0000 7f2155d9b640 1 -- 172.21.15.29:0/2370194289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500343f0 con 0x7f21580caac0 2023-10-16T14:37:30.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 -- 172.21.15.29:0/2370194289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580caac0 msgr2=0x7f215806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580caac0 0x7f215806a2e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f2150009e50 tx=0x7f215002bed0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 -- 172.21.15.29:0/2370194289 shutdown_connections 2023-10-16T14:37:30.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580caac0 0x7f215806a2e0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 --2- 172.21.15.29:0/2370194289 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f215815ad70 0x7f2158069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 -- 172.21.15.29:0/2370194289 >> 172.21.15.29:0/2370194289 conn(0x7f21580ce9b0 msgr2=0x7f21580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 -- 172.21.15.29:0/2370194289 shutdown_connections 2023-10-16T14:37:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.563+0000 7f215c80e640 1 -- 172.21.15.29:0/2370194289 wait complete. 2023-10-16T14:37:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.565+0000 7f215c80e640 1 Processor -- start 2023-10-16T14:37:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.565+0000 7f215c80e640 1 -- start start 2023-10-16T14:37:30.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215c80e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 0x7f215815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:30.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215c80e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21580caac0 0x7f21581f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:30.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215c80e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215815ad70 0x7f21581f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:30.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215c80e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2158157780 con 0x7f21580c9ca0 2023-10-16T14:37:30.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215c80e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21581f1360 con 0x7f215815ad70 2023-10-16T14:37:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215c80e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215815ce60 con 0x7f21580caac0 2023-10-16T14:37:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 0x7f215815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 0x7f215815ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55656/0 (socket says 172.21.15.29:55656) 2023-10-16T14:37:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f215659c640 1 -- 172.21.15.29:0/993680110 learned_addr learned my addr 172.21.15.29:0/993680110 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T14:37:30.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.566+0000 7f2156d9d640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215815ad70 0x7f21581f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215659c640 1 -- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21580caac0 msgr2=0x7f21581f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f214ffff640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21580caac0 0x7f21581f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215659c640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21580caac0 0x7f21581f14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215659c640 1 -- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215815ad70 msgr2=0x7f21581f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215659c640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215815ad70 0x7f21581f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215659c640 1 -- 172.21.15.29:0/993680110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150009b30 con 0x7f21580c9ca0 2023-10-16T14:37:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f2156d9d640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215815ad70 0x7f21581f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:37:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215659c640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 0x7f215815ddc0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f214800c7f0 tx=0x7f214800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148007820 con 0x7f21580c9ca0 2023-10-16T14:37:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2148007980 con 0x7f21580c9ca0 2023-10-16T14:37:30.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148007cf0 con 0x7f21580c9ca0 2023-10-16T14:37:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.567+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215815e300 con 0x7f21580c9ca0 2023-10-16T14:37:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.568+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21580c7610 con 0x7f21580c9ca0 2023-10-16T14:37:30.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.569+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215804e160 con 0x7f21580c9ca0 2023-10-16T14:37:30.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.570+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f21480113f0 con 0x7f21580c9ca0 2023-10-16T14:37:30.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.570+0000 7f214effd640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f213c06ad40 0x7f213c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:37:30.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.571+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f2148089780 con 0x7f21580c9ca0 2023-10-16T14:37:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.571+0000 7f214ffff640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f213c06ad40 0x7f213c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:37:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.572+0000 7f214ffff640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f213c06ad40 0x7f213c06d200 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f21580c59a0 tx=0x7f2140009040 comp rx=0 tx=0).ready entity=mgr.6722 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:37:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.575+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2148054920 con 0x7f21580c9ca0 2023-10-16T14:37:30.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.790+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 --> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f215806c720 con 0x7f213c06ad40 2023-10-16T14:37:30.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.808+0000 7f214effd640 1 -- 172.21.15.29:0/993680110 <== mgr.6722 v2:172.21.15.170:6832/3272758872 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f215806c720 con 0x7f213c06ad40 2023-10-16T14:37:30.812 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T14:37:30.812 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T14:37:30.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f213c06ad40 msgr2=0x7f213c06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f213c06ad40 0x7f213c06d200 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f21580c59a0 tx=0x7f2140009040 comp rx=0 tx=0).stop 2023-10-16T14:37:30.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 msgr2=0x7f215815ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:37:30.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 0x7f215815ddc0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f214800c7f0 tx=0x7f214800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 shutdown_connections 2023-10-16T14:37:30.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f213c06ad40 0x7f213c06d200 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f215815ad70 0x7f21581f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f21580caac0 0x7f21581f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 --2- 172.21.15.29:0/993680110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21580c9ca0 0x7f215815ddc0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:37:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 >> 172.21.15.29:0/993680110 conn(0x7f21580ce9b0 msgr2=0x7f215815a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:37:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.810+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 shutdown_connections 2023-10-16T14:37:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T14:37:30.811+0000 7f215c80e640 1 -- 172.21.15.29:0/993680110 wait complete. 2023-10-16T14:37:30.832 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T14:37:30.832 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-10-16T14:37:49.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:37:49.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:38:00.319 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:38:00.316+0000 7fbe96dc7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:38:00.319 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:38:00.316+0000 7fbe96dc7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:38:00.319 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:38:00.316+0000 7fbe96dc7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:38:00.319 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:38:00.316+0000 7fbe96dc7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:38:00.320 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T14:38:00.316+0000 7fbe96dc7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:38:19.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:38:19.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:38:49.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:38:49.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:39:19.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:39:19.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:39:49.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:39:49.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:40:19.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:40:19.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:40:50.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:40:50.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:41:20.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:41:20.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:41:50.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:41:50.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:42:20.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:42:20.501 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:42:50.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:42:50.661 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:43:20.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:43:20.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:43:50.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:43:50.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:44:21.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:44:21.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:44:51.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:44:51.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:45:21.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:45:21.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:45:51.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:45:51.616 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:46:21.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:46:21.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:46:51.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:46:51.918 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:47:22.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:47:22.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:47:52.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:47:52.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:48:22.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:48:22.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:48:52.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:48:52.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:49:22.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:49:22.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:49:52.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:49:52.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:50:23.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:50:23.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:50:53.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:50:53.188 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:51:23.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:51:23.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:51:53.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:51:53.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:52:23.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:52:23.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:52:54.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:52:54.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:53:24.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:53:24.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:53:54.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:53:54.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:54:24.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:54:24.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:54:54.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:54:54.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:55:24.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:55:24.932 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:55:55.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:55:55.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:56:25.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:56:25.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:56:55.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:56:55.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:57:25.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:57:25.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:57:55.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:57:55.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:58:26.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:58:26.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:58:56.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:58:56.291 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:59:26.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:59:26.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:59:56.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:59:56.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:00:26.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:00:26.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:00:57.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:00:57.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:01:27.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:01:27.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:01:57.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:01:57.476 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:02:27.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:02:27.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:02:57.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:02:57.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:03:28.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:03:28.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:03:58.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:03:58.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:04:28.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:04:28.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:04:58.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:04:58.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:05:28.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:05:28.944 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:05:59.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:05:59.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:06:29.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:06:29.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:06:59.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:06:59.467 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:07:29.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:07:29.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:07:59.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:08:00.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:08:30.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:08:30.143 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:09:00.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:09:00.305 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:09:30.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:09:30.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:10:00.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:10:00.698 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:10:30.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:10:30.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:11:01.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:11:01.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:11:31.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:11:31.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:12:01.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:12:01.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:12:31.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:12:31.748 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:13:01.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:13:01.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:13:32.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:13:32.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:14:02.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:14:02.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:14:32.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:14:32.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:15:02.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:15:02.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:15:32.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:15:32.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:16:03.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:16:03.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:16:33.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:16:33.341 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:17:03.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:17:03.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:17:33.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:17:33.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:18:03.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:18:03.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:18:34.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:18:34.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:19:04.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:19:04.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:19:34.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:19:34.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:20:04.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:20:04.716 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:20:34.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:20:34.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:21:05.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:21:05.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:21:35.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:21:35.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:22:05.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:22:05.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:22:35.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:22:35.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:23:05.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:23:05.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:23:36.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:23:36.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:24:06.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:24:06.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:24:36.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:24:36.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:25:06.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:25:06.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:25:36.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:25:36.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:26:07.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:26:07.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:26:37.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:26:37.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:27:07.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:27:07.491 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:27:37.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:27:37.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:28:07.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:28:07.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:28:38.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:28:38.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:29:08.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:29:08.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:29:38.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:29:38.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:30:08.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:30:08.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:30:38.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:30:38.870 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:31:09.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:31:09.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:31:39.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:31:39.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:32:09.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:32:09.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:32:39.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:32:39.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:33:09.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:33:09.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:33:15.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:15.834 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:33:15.907 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_39_00_UTC 2023-10-16T15:33:15.908 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_40_00_UTC 2023-10-16T15:33:15.908 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_41_00_UTC 2023-10-16T15:33:15.908 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_42_00_UTC 2023-10-16T15:33:15.908 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_43_00_UTC 2023-10-16T15:33:15.908 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_44_00_UTC 2023-10-16T15:33:15.909 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_45_00_UTC 2023-10-16T15:33:15.909 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_46_00_UTC 2023-10-16T15:33:15.909 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_47_00_UTC 2023-10-16T15:33:15.909 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_48_00_UTC 2023-10-16T15:33:15.909 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_49_00_UTC 2023-10-16T15:33:15.910 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_50_00_UTC 2023-10-16T15:33:15.910 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_51_00_UTC 2023-10-16T15:33:15.910 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_52_00_UTC 2023-10-16T15:33:15.910 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_53_00_UTC 2023-10-16T15:33:15.910 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_54_00_UTC 2023-10-16T15:33:15.911 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_55_01_UTC 2023-10-16T15:33:15.911 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_56_00_UTC 2023-10-16T15:33:15.911 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_57_00_UTC 2023-10-16T15:33:15.911 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_58_00_UTC 2023-10-16T15:33:15.911 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_59_00_UTC 2023-10-16T15:33:15.911 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_00_00_UTC 2023-10-16T15:33:15.912 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_01_00_UTC 2023-10-16T15:33:15.912 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_02_00_UTC 2023-10-16T15:33:15.912 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_03_00_UTC 2023-10-16T15:33:15.912 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_04_00_UTC 2023-10-16T15:33:15.912 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_05_00_UTC 2023-10-16T15:33:15.913 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_06_00_UTC 2023-10-16T15:33:15.913 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_07_00_UTC 2023-10-16T15:33:15.913 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_08_00_UTC 2023-10-16T15:33:15.913 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_09_00_UTC 2023-10-16T15:33:15.913 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_10_00_UTC 2023-10-16T15:33:15.913 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_11_00_UTC 2023-10-16T15:33:15.914 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_12_00_UTC 2023-10-16T15:33:15.914 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_13_00_UTC 2023-10-16T15:33:15.914 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_14_00_UTC 2023-10-16T15:33:15.914 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_15_00_UTC 2023-10-16T15:33:15.914 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_16_00_UTC 2023-10-16T15:33:15.915 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_17_00_UTC 2023-10-16T15:33:15.915 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_18_00_UTC 2023-10-16T15:33:15.915 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_19_00_UTC 2023-10-16T15:33:15.915 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_20_00_UTC 2023-10-16T15:33:15.915 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_21_00_UTC 2023-10-16T15:33:15.916 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_22_00_UTC 2023-10-16T15:33:15.916 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_23_00_UTC 2023-10-16T15:33:15.916 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_24_00_UTC 2023-10-16T15:33:15.916 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_25_00_UTC 2023-10-16T15:33:15.916 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_26_00_UTC 2023-10-16T15:33:15.916 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_27_00_UTC 2023-10-16T15:33:15.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_28_00_UTC 2023-10-16T15:33:15.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_29_00_UTC 2023-10-16T15:33:15.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_30_00_UTC 2023-10-16T15:33:15.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_31_00_UTC 2023-10-16T15:33:15.917 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_32_00_UTC 2023-10-16T15:33:15.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_33_00_UTC 2023-10-16T15:33:15.919 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-14_39_00_UTC', 'scheduled-2023-10-16-14_40_00_UTC', 'scheduled-2023-10-16-14_41_00_UTC', 'scheduled-2023-10-16-14_42_00_UTC', 'scheduled-2023-10-16-14_43_00_UTC', 'scheduled-2023-10-16-14_44_00_UTC', 'scheduled-2023-10-16-14_45_00_UTC', 'scheduled-2023-10-16-14_46_00_UTC', 'scheduled-2023-10-16-14_47_00_UTC', 'scheduled-2023-10-16-14_48_00_UTC', 'scheduled-2023-10-16-14_49_00_UTC', 'scheduled-2023-10-16-14_50_00_UTC', 'scheduled-2023-10-16-14_51_00_UTC', 'scheduled-2023-10-16-14_52_00_UTC', 'scheduled-2023-10-16-14_53_00_UTC', 'scheduled-2023-10-16-14_54_00_UTC', 'scheduled-2023-10-16-14_55_01_UTC', 'scheduled-2023-10-16-14_56_00_UTC', 'scheduled-2023-10-16-14_57_00_UTC', 'scheduled-2023-10-16-14_58_00_UTC', 'scheduled-2023-10-16-14_59_00_UTC', 'scheduled-2023-10-16-15_00_00_UTC', 'scheduled-2023-10-16-15_01_00_UTC', 'scheduled-2023-10-16-15_02_00_UTC', 'scheduled-2023-10-16-15_03_00_UTC', 'scheduled-2023-10-16-15_04_00_UTC', 'scheduled-2023-10-16-15_05_00_UTC', 'scheduled-2023-10-16-15_06_00_UTC', 'scheduled-2023-10-16-15_07_00_UTC', 'scheduled-2023-10-16-15_08_00_UTC', 'scheduled-2023-10-16-15_09_00_UTC', 'scheduled-2023-10-16-15_10_00_UTC', 'scheduled-2023-10-16-15_11_00_UTC', 'scheduled-2023-10-16-15_12_00_UTC', 'scheduled-2023-10-16-15_13_00_UTC', 'scheduled-2023-10-16-15_14_00_UTC', 'scheduled-2023-10-16-15_15_00_UTC', 'scheduled-2023-10-16-15_16_00_UTC', 'scheduled-2023-10-16-15_17_00_UTC', 'scheduled-2023-10-16-15_18_00_UTC', 'scheduled-2023-10-16-15_19_00_UTC', 'scheduled-2023-10-16-15_20_00_UTC', 'scheduled-2023-10-16-15_21_00_UTC', 'scheduled-2023-10-16-15_22_00_UTC', 'scheduled-2023-10-16-15_23_00_UTC', 'scheduled-2023-10-16-15_24_00_UTC', 'scheduled-2023-10-16-15_25_00_UTC', 'scheduled-2023-10-16-15_26_00_UTC', 'scheduled-2023-10-16-15_27_00_UTC', 'scheduled-2023-10-16-15_28_00_UTC', 'scheduled-2023-10-16-15_29_00_UTC', 'scheduled-2023-10-16-15_30_00_UTC', 'scheduled-2023-10-16-15_31_00_UTC', 'scheduled-2023-10-16-15_32_00_UTC', 'scheduled-2023-10-16-15_33_00_UTC'] 2023-10-16T15:33:15.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-10-16T15:33:16.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.142+0000 7f4be10bb640 1 Processor -- start 2023-10-16T15:33:16.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.142+0000 7f4be10bb640 1 -- start start 2023-10-16T15:33:16.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4be10bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 0x7f4bdc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4be10bb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4bdc0cda80 0x7f4bdc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4be10bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc06c960 con 0x7f4bdc0ad970 2023-10-16T15:33:16.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4be10bb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc15b9d0 con 0x7f4bdc0cda80 2023-10-16T15:33:16.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4be10bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc15bcb0 con 0x7f4bdc06c040 2023-10-16T15:33:16.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4bdb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4bdc0ad970 0x7f4bdc15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44944/0 (socket says 172.21.15.29:44944) 2023-10-16T15:33:16.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4bdb7fe640 1 -- 172.21.15.29:0/1162881242 learned_addr learned my addr 172.21.15.29:0/1162881242 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4bdbfff640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 0x7f4bdc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.144+0000 7f4be08ba640 1 --2- 172.21.15.29:0/1162881242 >> v2:172.21.15.170:3300/0 conn(0x7f4bdc0cda80 0x7f4bdc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821610556 0 0) 0x7f4bdc06c960 con 0x7f4bdc0ad970 2023-10-16T15:33:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc4003520 con 0x7f4bdc0ad970 2023-10-16T15:33:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3644256923 0 0) 0x7f4bc4003520 con 0x7f4bdc0ad970 2023-10-16T15:33:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 msgr2=0x7f4bdc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 0x7f4bdc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 >> v2:172.21.15.170:3300/0 conn(0x7f4bdc0cda80 msgr2=0x7f4bdc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 --2- 172.21.15.29:0/1162881242 >> v2:172.21.15.170:3300/0 conn(0x7f4bdc0cda80 0x7f4bdc15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc15bf90 con 0x7f4bdc0ad970 2023-10-16T15:33:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4bd0001670 con 0x7f4bdc0ad970 2023-10-16T15:33:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 >> v1:172.21.15.29:6789/0 conn(0x7f4bdc0ad970 legacy=0x7f4bdc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 0x7f4bc4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4003ef0 0x7f4bc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc4004d00 0x7f4bc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc15b9d0 con 0x7f4bc4003ef0 2023-10-16T15:33:16.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc15bcb0 con 0x7f4bc4004d00 2023-10-16T15:33:16.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc400b9d0 con 0x7f4bdc06c040 2023-10-16T15:33:16.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.145+0000 7f4bdb7fe640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4003ef0 0x7f4bc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4bdbfff640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 0x7f4bc4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4be08ba640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc4004d00 0x7f4bc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4bdb7fe640 1 -- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 msgr2=0x7f4bc4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4bdb7fe640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bdc06c040 0x7f4bc4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4bdb7fe640 1 -- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc4004d00 msgr2=0x7f4bc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4bdb7fe640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bc4004d00 0x7f4bc4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.146+0000 7f4bdb7fe640 1 -- 172.21.15.29:0/1162881242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4003520 con 0x7f4bc4003ef0 2023-10-16T15:33:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.147+0000 7f4bdb7fe640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4003ef0 0x7f4bc40047c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000c240 tx=0x7f4bc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cb8726b4fe6b9e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:16.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.147+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd002cb00 con 0x7f4bc4003ef0 2023-10-16T15:33:16.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.147+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd002cc60 con 0x7f4bc4003ef0 2023-10-16T15:33:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.147+0000 7f4bdaffd640 1 -- 172.21.15.29:0/1162881242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0005af0 con 0x7f4bc4003ef0 2023-10-16T15:33:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be10bb640 1 -- 172.21.15.29:0/1162881242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4003ef0 msgr2=0x7f4bc40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be10bb640 1 --2- 172.21.15.29:0/1162881242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4003ef0 0x7f4bc40047c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000c240 tx=0x7f4bc40057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be08ba640 1 -- 172.21.15.29:0/1162881242 reap_dead start 2023-10-16T15:33:16.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be10bb640 1 -- 172.21.15.29:0/1162881242 shutdown_connections 2023-10-16T15:33:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be10bb640 1 -- 172.21.15.29:0/1162881242 >> 172.21.15.29:0/1162881242 conn(0x7f4bdc0ce6a0 msgr2=0x7f4bdc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be10bb640 1 -- 172.21.15.29:0/1162881242 shutdown_connections 2023-10-16T15:33:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.148+0000 7f4be10bb640 1 -- 172.21.15.29:0/1162881242 wait complete. 2023-10-16T15:33:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.150+0000 7f4be10bb640 1 Processor -- start 2023-10-16T15:33:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.150+0000 7f4be10bb640 1 -- start start 2023-10-16T15:33:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.151+0000 7f4be10bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bc4003ef0 0x7f4bdc14fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.151+0000 7f4be10bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bdc0ae730 0x7f4bdc0aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.151+0000 7f4be10bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 0x7f4bdc0b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.151+0000 7f4be10bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0c67b0 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.151+0000 7f4be10bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0af0f0 con 0x7f4bdc0ae730 2023-10-16T15:33:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.151+0000 7f4be10bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc154220 con 0x7f4bc4003ef0 2023-10-16T15:33:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 0x7f4bdc0b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 0x7f4bdc0b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44766/0 (socket says 172.21.15.29:44766) 2023-10-16T15:33:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 -- 172.21.15.29:0/53171188 learned_addr learned my addr 172.21.15.29:0/53171188 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4bdbfff640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bc4003ef0 0x7f4bdc14fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4bdb7fe640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bdc0ae730 0x7f4bdc0aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 -- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bc4003ef0 msgr2=0x7f4bdc14fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bc4003ef0 0x7f4bdc14fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 -- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bdc0ae730 msgr2=0x7f4bdc0aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bdc0ae730 0x7f4bdc0aebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 -- 172.21.15.29:0/53171188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0c6fe0 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.152+0000 7f4be08ba640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 0x7f4bdc0b3010 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc06d2e0 tx=0x7f4bd4007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.153+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4009e70 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.153+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd400f050 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.153+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4014670 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.153+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc0b3910 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.153+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc0c6c70 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.154+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc06b330 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.155+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 84) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f4bd40147d0 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.157+0000 7f4bd8ff9640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f4bc806afe0 0x7f4bc806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.157+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f4bd4089fc0 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.157+0000 7f4bdbfff640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f4bc806afe0 0x7f4bc806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.158+0000 7f4bdbfff640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f4bc806afe0 0x7f4bc806d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009930 tx=0x7f4bcc008040 comp rx=0 tx=0).ready entity=mgr.6722 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:16.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.162+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4bd40bcca0 con 0x7f4bdc0b2b90 2023-10-16T15:33:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.378+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 --> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4bdc0cc550 con 0x7f4bc806afe0 2023-10-16T15:33:16.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.399+0000 7f4bd8ff9640 1 -- 172.21.15.29:0/53171188 <== mgr.6722 v2:172.21.15.170:6832/3272758872 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f4bcc0095c0 con 0x7f4bc806afe0 2023-10-16T15:33:16.401 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:16.401 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:37:29", "first": "2023-10-16T14:38:00", "last": "2023-10-16T15:33:00", "last_pruned": "2023-10-16T15:33:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T15:33:16.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.400+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f4bc806afe0 msgr2=0x7f4bc806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f4bc806afe0 0x7f4bc806d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009930 tx=0x7f4bcc008040 comp rx=0 tx=0).stop 2023-10-16T15:33:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 msgr2=0x7f4bdc0b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 0x7f4bdc0b3010 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc06d2e0 tx=0x7f4bd4007700 comp rx=0 tx=0).stop 2023-10-16T15:33:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 shutdown_connections 2023-10-16T15:33:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f4bc806afe0 0x7f4bc806d4a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bdc0b2b90 0x7f4bdc0b3010 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4bdc0ae730 0x7f4bdc0aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 --2- 172.21.15.29:0/53171188 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4bc4003ef0 0x7f4bdc14fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 >> 172.21.15.29:0/53171188 conn(0x7f4bdc0ce6a0 msgr2=0x7f4bdc15c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 shutdown_connections 2023-10-16T15:33:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.401+0000 7f4be10bb640 1 -- 172.21.15.29:0/53171188 wait complete. 2023-10-16T15:33:16.423 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:37:29", "first": "2023-10-16T14:38:00", "last": "2023-10-16T15:33:00", "last_pruned": "2023-10-16T15:33:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T15:33:16.424 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T14:37:29', 'first': '2023-10-16T14:38:00', 'last': '2023-10-16T15:33:00', 'last_pruned': '2023-10-16T15:33:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-16T15:33:16.424 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-16T15:33:16.424 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-16T15:33:16.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-10-16T15:33:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.648+0000 7fb996d55640 1 Processor -- start 2023-10-16T15:33:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.648+0000 7fb996d55640 1 -- start start 2023-10-16T15:33:16.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb996d55640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb9900cf710 0x7fb9900b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb996d55640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9900cfaf0 0x7fb9900b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb996d55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900d3980 con 0x7fb9900b38e0 2023-10-16T15:33:16.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb996d55640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9900d3c60 con 0x7fb9900cfaf0 2023-10-16T15:33:16.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb996d55640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9900ce630 con 0x7fb9900cf710 2023-10-16T15:33:16.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb98ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9900cfaf0 0x7fb9900b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb98ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9900cfaf0 0x7fb9900b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44774/0 (socket says 172.21.15.29:44774) 2023-10-16T15:33:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.650+0000 7fb98ffff640 1 -- 172.21.15.29:0/1491658345 learned_addr learned my addr 172.21.15.29:0/1491658345 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.651+0000 7fb994af1640 1 --2- 172.21.15.29:0/1491658345 >> v2:172.21.15.170:3300/0 conn(0x7fb9900cf710 0x7fb9900b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.651+0000 7fb98ffff640 1 -- 172.21.15.29:0/1491658345 >> v2:172.21.15.170:3300/0 conn(0x7fb9900cf710 msgr2=0x7fb9900b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.651+0000 7fb98ffff640 1 --2- 172.21.15.29:0/1491658345 >> v2:172.21.15.170:3300/0 conn(0x7fb9900cf710 0x7fb9900b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.651+0000 7fb98ffff640 1 -- 172.21.15.29:0/1491658345 >> v1:172.21.15.170:6789/0 conn(0x7fb9900b38e0 legacy=0x7fb9900d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.651+0000 7fb98ffff640 1 -- 172.21.15.29:0/1491658345 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900ce910 con 0x7fb9900cfaf0 2023-10-16T15:33:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.652+0000 7fb98ffff640 1 --2- 172.21.15.29:0/1491658345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cfaf0 0x7fb9900b3250 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fb9900d2c20 tx=0x7fb97c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933791267d590b2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.652+0000 7fb98f7fe640 1 -- 172.21.15.29:0/1491658345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0317e0 con 0x7fb9900cfaf0 2023-10-16T15:33:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb98f7fe640 1 -- 172.21.15.29:0/1491658345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c031940 con 0x7fb9900cfaf0 2023-10-16T15:33:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb98f7fe640 1 -- 172.21.15.29:0/1491658345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c031cd0 con 0x7fb9900cfaf0 2023-10-16T15:33:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 -- 172.21.15.29:0/1491658345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cfaf0 msgr2=0x7fb9900b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 --2- 172.21.15.29:0/1491658345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cfaf0 0x7fb9900b3250 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fb9900d2c20 tx=0x7fb97c02b880 comp rx=0 tx=0).stop 2023-10-16T15:33:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 -- 172.21.15.29:0/1491658345 shutdown_connections 2023-10-16T15:33:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 --2- 172.21.15.29:0/1491658345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cfaf0 0x7fb9900b3250 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 --2- 172.21.15.29:0/1491658345 >> v2:172.21.15.170:3300/0 conn(0x7fb9900cf710 0x7fb9900b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 -- 172.21.15.29:0/1491658345 >> 172.21.15.29:0/1491658345 conn(0x7fb9900d2cc0 msgr2=0x7fb9900d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 -- 172.21.15.29:0/1491658345 shutdown_connections 2023-10-16T15:33:16.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.653+0000 7fb996d55640 1 -- 172.21.15.29:0/1491658345 wait complete. 2023-10-16T15:33:16.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.655+0000 7fb996d55640 1 Processor -- start 2023-10-16T15:33:16.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.655+0000 7fb996d55640 1 -- start start 2023-10-16T15:33:16.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb996d55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9900b38e0 0x7fb9901f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb996d55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 0x7fb9900c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb996d55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9900cfaf0 0x7fb9900cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb996d55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9901f28d0 con 0x7fb9900cf710 2023-10-16T15:33:16.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb996d55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900c84f0 con 0x7fb9900cfaf0 2023-10-16T15:33:16.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb996d55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900c78e0 con 0x7fb9900b38e0 2023-10-16T15:33:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.656+0000 7fb98ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 0x7fb9900c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 0x7fb9900c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44778/0 (socket says 172.21.15.29:44778) 2023-10-16T15:33:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 -- 172.21.15.29:0/3389131071 learned_addr learned my addr 172.21.15.29:0/3389131071 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 -- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9900b38e0 msgr2=0x7fb9901f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb994af1640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9900b38e0 0x7fb9901f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb9952f2640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9900cfaf0 0x7fb9900cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9900b38e0 0x7fb9901f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 -- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9900cfaf0 msgr2=0x7fb9900cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9900cfaf0 0x7fb9900cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 -- 172.21.15.29:0/3389131071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c0094d0 con 0x7fb9900cf710 2023-10-16T15:33:16.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.657+0000 7fb98ffff640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 0x7fb9900c8980 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fb97c02b8c0 tx=0x7fb97c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.658+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c032880 con 0x7fb9900cf710 2023-10-16T15:33:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.658+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c0329e0 con 0x7fb9900cf710 2023-10-16T15:33:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.658+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c032d70 con 0x7fb9900cf710 2023-10-16T15:33:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.658+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9900c7bc0 con 0x7fb9900cf710 2023-10-16T15:33:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.658+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9900c8180 con 0x7fb9900cf710 2023-10-16T15:33:16.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.659+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99004e160 con 0x7fb9900cf710 2023-10-16T15:33:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.660+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 84) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fb97c04e450 con 0x7fb9900cf710 2023-10-16T15:33:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.661+0000 7fb98d7fa640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fb97406afe0 0x7fb97406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.662+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fb97c0abd50 con 0x7fb9900cf710 2023-10-16T15:33:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.662+0000 7fb994af1640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fb97406afe0 0x7fb97406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:16.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.664+0000 7fb994af1640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fb97406afe0 0x7fb97406d4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9900b7b70 tx=0x7fb984008040 comp rx=0 tx=0).ready entity=mgr.6722 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:16.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.667+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb97c076a70 con 0x7fb9900cf710 2023-10-16T15:33:16.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:16.887+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 --> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9900cae30 con 0x7fb97406afe0 2023-10-16T15:33:17.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.074+0000 7fb98d7fa640 1 -- 172.21.15.29:0/3389131071 <== mgr.6722 v2:172.21.15.170:6832/3272758872 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7fb9900cae30 con 0x7fb97406afe0 2023-10-16T15:33:17.076 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1 2023-10-16T15:33:17.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fb97406afe0 msgr2=0x7fb97406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:17.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fb97406afe0 0x7fb97406d4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9900b7b70 tx=0x7fb984008040 comp rx=0 tx=0).stop 2023-10-16T15:33:17.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 msgr2=0x7fb9900c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:17.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 0x7fb9900c8980 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fb97c02b8c0 tx=0x7fb97c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:17.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 shutdown_connections 2023-10-16T15:33:17.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fb97406afe0 0x7fb97406d4a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:17.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9900cf710 0x7fb9900c8980 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:17.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9900cfaf0 0x7fb9900cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:17.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 --2- 172.21.15.29:0/3389131071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb9900b38e0 0x7fb9901f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:17.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.077+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 >> 172.21.15.29:0/3389131071 conn(0x7fb9900d2cc0 msgr2=0x7fb9900b7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:17.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.078+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 shutdown_connections 2023-10-16T15:33:17.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:17.078+0000 7fb996d55640 1 -- 172.21.15.29:0/3389131071 wait complete. 2023-10-16T15:33:17.100 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-16T15:33:17.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.101 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:33:17.160 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_39_00_UTC 2023-10-16T15:33:17.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_40_00_UTC 2023-10-16T15:33:17.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_41_00_UTC 2023-10-16T15:33:17.161 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_42_00_UTC 2023-10-16T15:33:17.162 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_43_00_UTC 2023-10-16T15:33:17.162 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_44_00_UTC 2023-10-16T15:33:17.162 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_45_00_UTC 2023-10-16T15:33:17.162 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_46_00_UTC 2023-10-16T15:33:17.162 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_47_00_UTC 2023-10-16T15:33:17.163 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_48_00_UTC 2023-10-16T15:33:17.163 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_49_00_UTC 2023-10-16T15:33:17.163 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_50_00_UTC 2023-10-16T15:33:17.163 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_51_00_UTC 2023-10-16T15:33:17.163 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_52_00_UTC 2023-10-16T15:33:17.164 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_53_00_UTC 2023-10-16T15:33:17.164 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_54_00_UTC 2023-10-16T15:33:17.164 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_55_01_UTC 2023-10-16T15:33:17.164 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_56_00_UTC 2023-10-16T15:33:17.164 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_57_00_UTC 2023-10-16T15:33:17.165 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_58_00_UTC 2023-10-16T15:33:17.165 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-14_59_00_UTC 2023-10-16T15:33:17.165 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_00_00_UTC 2023-10-16T15:33:17.165 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_01_00_UTC 2023-10-16T15:33:17.165 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_02_00_UTC 2023-10-16T15:33:17.165 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_03_00_UTC 2023-10-16T15:33:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_04_00_UTC 2023-10-16T15:33:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_05_00_UTC 2023-10-16T15:33:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_06_00_UTC 2023-10-16T15:33:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_07_00_UTC 2023-10-16T15:33:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_08_00_UTC 2023-10-16T15:33:17.167 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_09_00_UTC 2023-10-16T15:33:17.167 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_10_00_UTC 2023-10-16T15:33:17.167 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_11_00_UTC 2023-10-16T15:33:17.167 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_12_00_UTC 2023-10-16T15:33:17.167 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_13_00_UTC 2023-10-16T15:33:17.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_14_00_UTC 2023-10-16T15:33:17.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_15_00_UTC 2023-10-16T15:33:17.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_16_00_UTC 2023-10-16T15:33:17.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_17_00_UTC 2023-10-16T15:33:17.168 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_18_00_UTC 2023-10-16T15:33:17.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_19_00_UTC 2023-10-16T15:33:17.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_20_00_UTC 2023-10-16T15:33:17.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_21_00_UTC 2023-10-16T15:33:17.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_22_00_UTC 2023-10-16T15:33:17.169 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_23_00_UTC 2023-10-16T15:33:17.170 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_24_00_UTC 2023-10-16T15:33:17.170 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_25_00_UTC 2023-10-16T15:33:17.170 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_26_00_UTC 2023-10-16T15:33:17.170 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_27_00_UTC 2023-10-16T15:33:17.170 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_28_00_UTC 2023-10-16T15:33:17.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_29_00_UTC 2023-10-16T15:33:17.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_30_00_UTC 2023-10-16T15:33:17.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_31_00_UTC 2023-10-16T15:33:17.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_32_00_UTC 2023-10-16T15:33:17.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_33_00_UTC 2023-10-16T15:33:17.172 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_39_00_UTC 2023-10-16T15:33:17.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.173 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_39_00_UTC) 2023-10-16T15:33:17.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_40_00_UTC 2023-10-16T15:33:17.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.253 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_40_00_UTC) 2023-10-16T15:33:17.334 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_41_00_UTC 2023-10-16T15:33:17.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.334 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_41_00_UTC) 2023-10-16T15:33:17.410 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_42_00_UTC 2023-10-16T15:33:17.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.411 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_42_00_UTC) 2023-10-16T15:33:17.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_43_00_UTC 2023-10-16T15:33:17.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.490 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_43_00_UTC) 2023-10-16T15:33:17.565 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_44_00_UTC 2023-10-16T15:33:17.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.566 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_44_00_UTC) 2023-10-16T15:33:17.641 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_45_00_UTC 2023-10-16T15:33:17.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.642 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_45_00_UTC) 2023-10-16T15:33:17.723 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_46_00_UTC 2023-10-16T15:33:17.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.724 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_46_00_UTC) 2023-10-16T15:33:17.803 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_47_00_UTC 2023-10-16T15:33:17.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.804 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_47_00_UTC) 2023-10-16T15:33:17.880 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_48_00_UTC 2023-10-16T15:33:17.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.881 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_48_00_UTC) 2023-10-16T15:33:17.957 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_49_00_UTC 2023-10-16T15:33:17.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:17.958 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_49_00_UTC) 2023-10-16T15:33:18.032 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_50_00_UTC 2023-10-16T15:33:18.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.033 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_50_00_UTC) 2023-10-16T15:33:18.109 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_51_00_UTC 2023-10-16T15:33:18.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.110 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_51_00_UTC) 2023-10-16T15:33:18.187 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_52_00_UTC 2023-10-16T15:33:18.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.188 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_52_00_UTC) 2023-10-16T15:33:18.265 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_53_00_UTC 2023-10-16T15:33:18.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.266 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_53_00_UTC) 2023-10-16T15:33:18.343 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_54_00_UTC 2023-10-16T15:33:18.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.343 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_54_00_UTC) 2023-10-16T15:33:18.417 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_55_01_UTC 2023-10-16T15:33:18.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.418 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_55_01_UTC) 2023-10-16T15:33:18.495 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_56_00_UTC 2023-10-16T15:33:18.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.496 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_56_00_UTC) 2023-10-16T15:33:18.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_57_00_UTC 2023-10-16T15:33:18.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.574 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_57_00_UTC) 2023-10-16T15:33:18.649 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_58_00_UTC 2023-10-16T15:33:18.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.650 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_58_00_UTC) 2023-10-16T15:33:18.723 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_59_00_UTC 2023-10-16T15:33:18.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.724 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_59_00_UTC) 2023-10-16T15:33:18.805 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_00_00_UTC 2023-10-16T15:33:18.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.806 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_00_00_UTC) 2023-10-16T15:33:18.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_01_00_UTC 2023-10-16T15:33:18.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.896 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_01_00_UTC) 2023-10-16T15:33:18.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_02_00_UTC 2023-10-16T15:33:18.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:18.977 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_02_00_UTC) 2023-10-16T15:33:19.035 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_03_00_UTC 2023-10-16T15:33:19.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.036 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_03_00_UTC) 2023-10-16T15:33:19.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_04_00_UTC 2023-10-16T15:33:19.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.095 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_04_00_UTC) 2023-10-16T15:33:19.148 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_05_00_UTC 2023-10-16T15:33:19.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.149 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_05_00_UTC) 2023-10-16T15:33:19.235 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_06_00_UTC 2023-10-16T15:33:19.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.236 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_06_00_UTC) 2023-10-16T15:33:19.289 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_07_00_UTC 2023-10-16T15:33:19.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.290 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_07_00_UTC) 2023-10-16T15:33:19.345 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_08_00_UTC 2023-10-16T15:33:19.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.346 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_08_00_UTC) 2023-10-16T15:33:19.404 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_09_00_UTC 2023-10-16T15:33:19.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.406 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_09_00_UTC) 2023-10-16T15:33:19.475 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_10_00_UTC 2023-10-16T15:33:19.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.476 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_10_00_UTC) 2023-10-16T15:33:19.547 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_11_00_UTC 2023-10-16T15:33:19.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.547 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_11_00_UTC) 2023-10-16T15:33:19.624 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_12_00_UTC 2023-10-16T15:33:19.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.625 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_12_00_UTC) 2023-10-16T15:33:19.701 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_13_00_UTC 2023-10-16T15:33:19.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.702 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_13_00_UTC) 2023-10-16T15:33:19.780 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_14_00_UTC 2023-10-16T15:33:19.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.781 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_14_00_UTC) 2023-10-16T15:33:19.857 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_15_00_UTC 2023-10-16T15:33:19.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.858 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_15_00_UTC) 2023-10-16T15:33:19.946 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_16_00_UTC 2023-10-16T15:33:19.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:19.947 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_16_00_UTC) 2023-10-16T15:33:20.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_17_00_UTC 2023-10-16T15:33:20.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.010 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_17_00_UTC) 2023-10-16T15:33:20.085 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_18_00_UTC 2023-10-16T15:33:20.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.086 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_18_00_UTC) 2023-10-16T15:33:20.139 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_19_00_UTC 2023-10-16T15:33:20.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.140 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_19_00_UTC) 2023-10-16T15:33:20.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_20_00_UTC 2023-10-16T15:33:20.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.188 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_20_00_UTC) 2023-10-16T15:33:20.295 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_21_00_UTC 2023-10-16T15:33:20.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.296 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_21_00_UTC) 2023-10-16T15:33:20.362 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_22_00_UTC 2023-10-16T15:33:20.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.363 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_22_00_UTC) 2023-10-16T15:33:20.423 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_23_00_UTC 2023-10-16T15:33:20.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.424 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_23_00_UTC) 2023-10-16T15:33:20.482 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_24_00_UTC 2023-10-16T15:33:20.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.483 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_24_00_UTC) 2023-10-16T15:33:20.552 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_25_00_UTC 2023-10-16T15:33:20.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.553 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_25_00_UTC) 2023-10-16T15:33:20.626 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_26_00_UTC 2023-10-16T15:33:20.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.627 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_26_00_UTC) 2023-10-16T15:33:20.698 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_27_00_UTC 2023-10-16T15:33:20.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.699 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_27_00_UTC) 2023-10-16T15:33:20.770 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_28_00_UTC 2023-10-16T15:33:20.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.771 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_28_00_UTC) 2023-10-16T15:33:20.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_29_00_UTC 2023-10-16T15:33:20.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.841 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_29_00_UTC) 2023-10-16T15:33:20.923 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_30_00_UTC 2023-10-16T15:33:20.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:20.924 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_30_00_UTC) 2023-10-16T15:33:21.003 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_31_00_UTC 2023-10-16T15:33:21.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:21.004 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_31_00_UTC) 2023-10-16T15:33:21.069 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_32_00_UTC 2023-10-16T15:33:21.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:21.070 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_32_00_UTC) 2023-10-16T15:33:21.118 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_33_00_UTC 2023-10-16T15:33:21.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:21.119 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_33_00_UTC) 2023-10-16T15:33:21.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:21.187 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T15:33:21.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:33:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.492+0000 7fccdeb55640 1 Processor -- start 2023-10-16T15:33:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.492+0000 7fccdeb55640 1 -- start start 2023-10-16T15:33:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccdeb55640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fccd8070b60 0x7fccd815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccdeb55640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccd806c100 0x7fccd806c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccdeb55640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd815aef0 con 0x7fccd8159950 2023-10-16T15:33:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccdeb55640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd815e490 con 0x7fccd806c100 2023-10-16T15:33:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccdeb55640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd815d910 con 0x7fccd8070b60 2023-10-16T15:33:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccde354640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccd806c100 0x7fccd806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.494+0000 7fccde354640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccd806c100 0x7fccd806c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44794/0 (socket says 172.21.15.29:44794) 2023-10-16T15:33:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.495+0000 7fccde354640 1 -- 172.21.15.29:0/3077691277 learned_addr learned my addr 172.21.15.29:0/3077691277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.495+0000 7fccdd352640 1 --2- 172.21.15.29:0/3077691277 >> v2:172.21.15.170:3300/0 conn(0x7fccd8070b60 0x7fccd815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.495+0000 7fccde354640 1 -- 172.21.15.29:0/3077691277 >> v2:172.21.15.170:3300/0 conn(0x7fccd8070b60 msgr2=0x7fccd815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.495+0000 7fccde354640 1 --2- 172.21.15.29:0/3077691277 >> v2:172.21.15.170:3300/0 conn(0x7fccd8070b60 0x7fccd815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:21.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.495+0000 7fccde354640 1 -- 172.21.15.29:0/3077691277 >> v1:172.21.15.170:6789/0 conn(0x7fccd8159950 legacy=0x7fccd8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:21.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.495+0000 7fccde354640 1 -- 172.21.15.29:0/3077691277 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd815dbf0 con 0x7fccd806c100 2023-10-16T15:33:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.496+0000 7fccde354640 1 --2- 172.21.15.29:0/3077691277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd806c100 0x7fccd806c510 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fccd80716a0 tx=0x7fccd002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283508a042bf0aca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:21.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.496+0000 7fccdcb51640 1 -- 172.21.15.29:0/3077691277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd002ec00 con 0x7fccd806c100 2023-10-16T15:33:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.496+0000 7fccdcb51640 1 -- 172.21.15.29:0/3077691277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd002ed60 con 0x7fccd806c100 2023-10-16T15:33:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.496+0000 7fccdcb51640 1 -- 172.21.15.29:0/3077691277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0036610 con 0x7fccd806c100 2023-10-16T15:33:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 -- 172.21.15.29:0/3077691277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd806c100 msgr2=0x7fccd806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 --2- 172.21.15.29:0/3077691277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd806c100 0x7fccd806c510 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fccd80716a0 tx=0x7fccd002e060 comp rx=0 tx=0).stop 2023-10-16T15:33:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 -- 172.21.15.29:0/3077691277 shutdown_connections 2023-10-16T15:33:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 --2- 172.21.15.29:0/3077691277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd806c100 0x7fccd806c510 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 --2- 172.21.15.29:0/3077691277 >> v2:172.21.15.170:3300/0 conn(0x7fccd8070b60 0x7fccd815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 -- 172.21.15.29:0/3077691277 >> 172.21.15.29:0/3077691277 conn(0x7fccd80ce6a0 msgr2=0x7fccd80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 -- 172.21.15.29:0/3077691277 shutdown_connections 2023-10-16T15:33:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.497+0000 7fccdeb55640 1 -- 172.21.15.29:0/3077691277 wait complete. 2023-10-16T15:33:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.499+0000 7fccdeb55640 1 Processor -- start 2023-10-16T15:33:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.499+0000 7fccdeb55640 1 -- start start 2023-10-16T15:33:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccdeb55640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccd806c100 0x7fccd81f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccdeb55640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccd8070b60 0x7fccd80c3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccdeb55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 0x7fccd8146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccdeb55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c4810 con 0x7fccd8159950 2023-10-16T15:33:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccdeb55640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8144750 con 0x7fccd8070b60 2023-10-16T15:33:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccdeb55640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c4250 con 0x7fccd806c100 2023-10-16T15:33:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccde354640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 0x7fccd8146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccde354640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 0x7fccd8146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44798/0 (socket says 172.21.15.29:44798) 2023-10-16T15:33:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.500+0000 7fccde354640 1 -- 172.21.15.29:0/4047454999 learned_addr learned my addr 172.21.15.29:0/4047454999 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccdd352640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccd8070b60 0x7fccd80c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccde354640 1 -- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccd806c100 msgr2=0x7fccd81f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccde354640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccd806c100 0x7fccd81f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccde354640 1 -- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccd8070b60 msgr2=0x7fccd80c3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccde354640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccd8070b60 0x7fccd80c3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccde354640 1 -- 172.21.15.29:0/4047454999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd0009b30 con 0x7fccd8159950 2023-10-16T15:33:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccde354640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 0x7fccd8146340 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fccd0038040 tx=0x7fccd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd002ec00 con 0x7fccd8159950 2023-10-16T15:33:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd0009510 con 0x7fccd8159950 2023-10-16T15:33:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd80c4530 con 0x7fccd8159950 2023-10-16T15:33:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd003f6b0 con 0x7fccd8159950 2023-10-16T15:33:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.501+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd81470a0 con 0x7fccd8159950 2023-10-16T15:33:21.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.503+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd804e9b0 con 0x7fccd8159950 2023-10-16T15:33:21.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.504+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 84) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fccd003f810 con 0x7fccd8159950 2023-10-16T15:33:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.505+0000 7fccce7fc640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fccbc06afe0 0x7fccbc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.505+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fccd00abff0 con 0x7fccd8159950 2023-10-16T15:33:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.505+0000 7fccddb53640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fccbc06afe0 0x7fccbc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.507+0000 7fccddb53640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fccbc06afe0 0x7fccbc06d4a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fccd4006fd0 tx=0x7fccd4006bc0 comp rx=0 tx=0).ready entity=mgr.6722 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.509+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fccd0076e10 con 0x7fccd8159950 2023-10-16T15:33:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:21.752+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fccd815aef0 con 0x7fccd8159950 2023-10-16T15:33:22.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.719+0000 7fccce7fc640 1 -- 172.21.15.29:0/4047454999 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v85) v1 ==== 93+0+0 (secure 0 0 0) 0x7fccd815aef0 con 0x7fccd8159950 2023-10-16T15:33:22.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.723+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fccbc06afe0 msgr2=0x7fccbc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:22.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.723+0000 7fccdeb55640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fccbc06afe0 0x7fccbc06d4a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fccd4006fd0 tx=0x7fccd4006bc0 comp rx=0 tx=0).stop 2023-10-16T15:33:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.723+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 msgr2=0x7fccd8146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:22.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.723+0000 7fccdeb55640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 0x7fccd8146340 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fccd0038040 tx=0x7fccd0004a40 comp rx=0 tx=0).stop 2023-10-16T15:33:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.724+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 shutdown_connections 2023-10-16T15:33:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.724+0000 7fccdeb55640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fccbc06afe0 0x7fccbc06d4a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.724+0000 7fccdeb55640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccd8159950 0x7fccd8146340 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:22.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.724+0000 7fccdeb55640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccd8070b60 0x7fccd80c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.724+0000 7fccdeb55640 1 --2- 172.21.15.29:0/4047454999 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccd806c100 0x7fccd81f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.724+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 >> 172.21.15.29:0/4047454999 conn(0x7fccd80ce6a0 msgr2=0x7fccd8070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.725+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 shutdown_connections 2023-10-16T15:33:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:22.725+0000 7fccdeb55640 1 -- 172.21.15.29:0/4047454999 wait complete. 2023-10-16T15:33:22.751 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:33:22.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:33:22.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:33:22.966 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:33:22.966 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:33:22.967 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:23.015 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:33:23.015 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:23.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:23.316 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:23.314+0000 7f0d88a30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:33:23.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.321+0000 7f209ff8a640 1 Processor -- start 2023-10-16T15:33:23.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.321+0000 7f209ff8a640 1 -- start start 2023-10-16T15:33:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.322+0000 7f209ff8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20981615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.322+0000 7f209ff8a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f209815d3f0 0x7f209815d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.322+0000 7f209ff8a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209815e3e0 con 0x7f20980cd0c0 2023-10-16T15:33:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.322+0000 7f209ff8a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20980cd4a0 con 0x7f209815d3f0 2023-10-16T15:33:23.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.322+0000 7f209ff8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209815e100 con 0x7f20980b0b20 2023-10-16T15:33:23.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.322+0000 7f209e527640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f209815d3f0 0x7f209815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209e527640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f209815d3f0 0x7f209815d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44814/0 (socket says 172.21.15.29:44814) 2023-10-16T15:33:23.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209e527640 1 -- 172.21.15.29:0/3838037967 learned_addr learned my addr 172.21.15.29:0/3838037967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:23.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209dd26640 1 --1- 172.21.15.29:0/3838037967 >> v1:172.21.15.29:6789/0 conn(0x7f20980cd0c0 0x7f20980b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44956/0 (socket says 172.21.15.29:44956) 2023-10-16T15:33:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209e527640 1 -- 172.21.15.29:0/3838037967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 msgr2=0x7f20981615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209e527640 1 --2- 172.21.15.29:0/3838037967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20981615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209e527640 1 -- 172.21.15.29:0/3838037967 >> v1:172.21.15.29:6789/0 conn(0x7f20980cd0c0 legacy=0x7f20980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.323+0000 7f209e527640 1 -- 172.21.15.29:0/3838037967 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2098160b10 con 0x7f209815d3f0 2023-10-16T15:33:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.324+0000 7f209e527640 1 --2- 172.21.15.29:0/3838037967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f209815d3f0 0x7f209815d860 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f2090009e50 tx=0x7f209002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442753565304655d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.324+0000 7f209cd24640 1 -- 172.21.15.29:0/3838037967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209002ca80 con 0x7f209815d3f0 2023-10-16T15:33:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209cd24640 1 -- 172.21.15.29:0/3838037967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209002cbe0 con 0x7f209815d3f0 2023-10-16T15:33:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209cd24640 1 -- 172.21.15.29:0/3838037967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900343f0 con 0x7f209815d3f0 2023-10-16T15:33:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 -- 172.21.15.29:0/3838037967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f209815d3f0 msgr2=0x7f209815d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 --2- 172.21.15.29:0/3838037967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f209815d3f0 0x7f209815d860 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f2090009e50 tx=0x7f209002bee0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 -- 172.21.15.29:0/3838037967 shutdown_connections 2023-10-16T15:33:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 --2- 172.21.15.29:0/3838037967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f209815d3f0 0x7f209815d860 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 --2- 172.21.15.29:0/3838037967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20981615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 -- 172.21.15.29:0/3838037967 >> 172.21.15.29:0/3838037967 conn(0x7f20980cb410 msgr2=0x7f20980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 -- 172.21.15.29:0/3838037967 shutdown_connections 2023-10-16T15:33:23.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.325+0000 7f209ff8a640 1 -- 172.21.15.29:0/3838037967 wait complete. 2023-10-16T15:33:23.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.327+0000 7f209ff8a640 1 Processor -- start 2023-10-16T15:33:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.327+0000 7f209ff8a640 1 -- start start 2023-10-16T15:33:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209ff8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20980aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209ff8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 0x7f20981548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209ff8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f209815d3f0 0x7f2098154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209ff8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20980af110 con 0x7f20980cd0c0 2023-10-16T15:33:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209ff8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20980af3f0 con 0x7f209815d3f0 2023-10-16T15:33:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209ff8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980af6d0 con 0x7f20980b0b20 2023-10-16T15:33:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209d525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 0x7f20981548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209d525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 0x7f20981548b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44822/0 (socket says 172.21.15.29:44822) 2023-10-16T15:33:23.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.328+0000 7f209d525640 1 -- 172.21.15.29:0/2877943256 learned_addr learned my addr 172.21.15.29:0/2877943256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209dd26640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20980aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209e527640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f209815d3f0 0x7f2098154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209d525640 1 -- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 msgr2=0x7f20980aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209d525640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20980aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209d525640 1 -- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f209815d3f0 msgr2=0x7f2098154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209d525640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f209815d3f0 0x7f2098154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209d525640 1 -- 172.21.15.29:0/2877943256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090009b30 con 0x7f20980cd0c0 2023-10-16T15:33:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209dd26640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20980aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209e527640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f209815d3f0 0x7f2098154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:23.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f209d525640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 0x7f20981548b0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f208800b410 tx=0x7f208800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.329+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088011830 con 0x7f20980cd0c0 2023-10-16T15:33:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.330+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20880119b0 con 0x7f20980cd0c0 2023-10-16T15:33:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.330+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20980af9b0 con 0x7f20980cd0c0 2023-10-16T15:33:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.330+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088011d60 con 0x7f20980cd0c0 2023-10-16T15:33:23.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.330+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2098155590 con 0x7f20980cd0c0 2023-10-16T15:33:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.331+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209804e160 con 0x7f20980cd0c0 2023-10-16T15:33:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.332+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f208801a400 con 0x7f20980cd0c0 2023-10-16T15:33:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.334+0000 7f208e7fc640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 0x7f207c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.334+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f2088089b30 con 0x7f20980cd0c0 2023-10-16T15:33:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.334+0000 7f209dd26640 1 -- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 msgr2=0x7f207c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.334+0000 7f209dd26640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 0x7f207c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.338+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f20880548b0 con 0x7f20980cd0c0 2023-10-16T15:33:23.348 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:23.347+0000 7f441a830040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:33:23.373 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:23.371+0000 7f0d88a30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:33:23.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:23.408+0000 7f441a830040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:33:23.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.535+0000 7f209dd26640 1 -- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 msgr2=0x7f207c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:23.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.535+0000 7f209dd26640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 0x7f207c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:23.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.536+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20980b4910 con 0x7f20980cd0c0 2023-10-16T15:33:23.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.538+0000 7f208e7fc640 1 -- 172.21.15.29:0/2877943256 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+17336 (secure 0 0 0) 0x7f2088015020 con 0x7f20980cd0c0 2023-10-16T15:33:23.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":165,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:22.905947+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.3f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/91040253":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/999637714":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/907915415":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6833/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6832/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3571017587":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1946312558":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1694874697":"2023-10-17T14:37:28.457671+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":52,"length":5},{"begin":108,"length":5}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 msgr2=0x7f207c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 0x7f207c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 msgr2=0x7f20981548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 0x7f20981548b0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f208800b410 tx=0x7f208800b8d0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 shutdown_connections 2023-10-16T15:33:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f207c06af90 0x7f207c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20980cd0c0 0x7f20981548b0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f209815d3f0 0x7f2098154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 --2- 172.21.15.29:0/2877943256 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20980b0b20 0x7f20980aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 >> 172.21.15.29:0/2877943256 conn(0x7f20980cb410 msgr2=0x7f209806fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 shutdown_connections 2023-10-16T15:33:23.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.540+0000 7f209ff8a640 1 -- 172.21.15.29:0/2877943256 wait complete. 2023-10-16T15:33:23.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:37:11.546312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:37:12.609923+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:33:23.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:33:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.785+0000 7f3a1c7f5640 1 Processor -- start 2023-10-16T15:33:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.786+0000 7f3a1c7f5640 1 -- start start 2023-10-16T15:33:23.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1c7f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a1415d330 0x7f3a141618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1c7f5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3a140661c0 0x7f3a140ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1c7f5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140d0890 con 0x7f3a1406c7d0 2023-10-16T15:33:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1c7f5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1406b1e0 con 0x7f3a1415d330 2023-10-16T15:33:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1c7f5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a140d05b0 con 0x7f3a140661c0 2023-10-16T15:33:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1a591640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a1415d330 0x7f3a141618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1a591640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a1415d330 0x7f3a141618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44828/0 (socket says 172.21.15.29:44828) 2023-10-16T15:33:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a1a591640 1 -- 172.21.15.29:0/2446530584 learned_addr learned my addr 172.21.15.29:0/2446530584 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.787+0000 7f3a19d90640 1 --1- 172.21.15.29:0/2446530584 >> v1:172.21.15.29:6789/0 conn(0x7f3a1406c7d0 0x7f3a14065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44966/0 (socket says 172.21.15.29:44966) 2023-10-16T15:33:23.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.788+0000 7f3a1ad92640 1 --2- 172.21.15.29:0/2446530584 >> v2:172.21.15.170:3300/0 conn(0x7f3a140661c0 0x7f3a140ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.788+0000 7f3a1a591640 1 -- 172.21.15.29:0/2446530584 >> v2:172.21.15.170:3300/0 conn(0x7f3a140661c0 msgr2=0x7f3a140ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:23.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.788+0000 7f3a1a591640 1 --2- 172.21.15.29:0/2446530584 >> v2:172.21.15.170:3300/0 conn(0x7f3a140661c0 0x7f3a140ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.788+0000 7f3a1a591640 1 -- 172.21.15.29:0/2446530584 >> v1:172.21.15.29:6789/0 conn(0x7f3a1406c7d0 legacy=0x7f3a14065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.788+0000 7f3a1a591640 1 -- 172.21.15.29:0/2446530584 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14160dd0 con 0x7f3a1415d330 2023-10-16T15:33:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.788+0000 7f3a1ad92640 1 --2- 172.21.15.29:0/2446530584 >> v2:172.21.15.170:3300/0 conn(0x7f3a140661c0 0x7f3a140ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.789+0000 7f3a1a591640 1 --2- 172.21.15.29:0/2446530584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1415d330 0x7f3a141618b0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3a1406f340 tx=0x7f3a1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39537d8b4b616aed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.789+0000 7f3a1958f640 1 -- 172.21.15.29:0/2446530584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10031870 con 0x7f3a1415d330 2023-10-16T15:33:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.789+0000 7f3a1958f640 1 -- 172.21.15.29:0/2446530584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a100319d0 con 0x7f3a1415d330 2023-10-16T15:33:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.789+0000 7f3a1958f640 1 -- 172.21.15.29:0/2446530584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10031d60 con 0x7f3a1415d330 2023-10-16T15:33:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/2446530584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1415d330 msgr2=0x7f3a141618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/2446530584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1415d330 0x7f3a141618b0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3a1406f340 tx=0x7f3a1002b910 comp rx=0 tx=0).stop 2023-10-16T15:33:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/2446530584 shutdown_connections 2023-10-16T15:33:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/2446530584 >> v2:172.21.15.170:3300/0 conn(0x7f3a140661c0 0x7f3a140ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/2446530584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1415d330 0x7f3a141618b0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/2446530584 >> 172.21.15.29:0/2446530584 conn(0x7f3a140d0ab0 msgr2=0x7f3a140d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:23.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/2446530584 shutdown_connections 2023-10-16T15:33:23.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.790+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/2446530584 wait complete. 2023-10-16T15:33:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.792+0000 7f3a1c7f5640 1 Processor -- start 2023-10-16T15:33:23.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.792+0000 7f3a1c7f5640 1 -- start start 2023-10-16T15:33:23.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1c7f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 0x7f3a140b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1c7f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a1406c7d0 0x7f3a140b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1c7f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a1415d330 0x7f3a140ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1c7f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b0700 con 0x7f3a140661c0 2023-10-16T15:33:23.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1c7f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b4620 con 0x7f3a1415d330 2023-10-16T15:33:23.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1c7f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b4900 con 0x7f3a1406c7d0 2023-10-16T15:33:23.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1a591640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 0x7f3a140b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1a591640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 0x7f3a140b4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44840/0 (socket says 172.21.15.29:44840) 2023-10-16T15:33:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.793+0000 7f3a1a591640 1 -- 172.21.15.29:0/4067517752 learned_addr learned my addr 172.21.15.29:0/4067517752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a19d90640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a1406c7d0 0x7f3a140b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1ad92640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a1415d330 0x7f3a140ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1a591640 1 -- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a1406c7d0 msgr2=0x7f3a140b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1a591640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a1406c7d0 0x7f3a140b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1a591640 1 -- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a1415d330 msgr2=0x7f3a140ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:23.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1a591640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a1415d330 0x7f3a140ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:23.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1a591640 1 -- 172.21.15.29:0/4067517752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100094d0 con 0x7f3a140661c0 2023-10-16T15:33:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1ad92640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a1415d330 0x7f3a140ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1a591640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 0x7f3a140b4040 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3a1002b950 tx=0x7f3a100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10031860 con 0x7f3a140661c0 2023-10-16T15:33:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a100443f0 con 0x7f3a140661c0 2023-10-16T15:33:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a140b4be0 con 0x7f3a140661c0 2023-10-16T15:33:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.794+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a100447a0 con 0x7f3a140661c0 2023-10-16T15:33:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.795+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a140aee00 con 0x7f3a140661c0 2023-10-16T15:33:23.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.796+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a140b0540 con 0x7f3a140661c0 2023-10-16T15:33:23.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.797+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f3a1004d450 con 0x7f3a140661c0 2023-10-16T15:33:23.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.797+0000 7f3a0affd640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 0x7f39f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.798+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f3a1002d040 con 0x7f3a140661c0 2023-10-16T15:33:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.798+0000 7f3a19d90640 1 -- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 msgr2=0x7f39f806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.798+0000 7f3a19d90640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 0x7f39f806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.806+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a140b0540 con 0x7f3a140661c0 2023-10-16T15:33:23.818 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:23.816+0000 7f0d88a30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:33:23.869 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:23.867+0000 7f0d88a30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:33:23.900 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:23.899+0000 7f441a830040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:33:23.919 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:23.917+0000 7f0d88a30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:33:23.956 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:23.954+0000 7f441a830040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:33:23.981 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:23.979+0000 7f0d88a30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:33:24.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.999+0000 7f3a19d90640 1 -- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 msgr2=0x7f39f806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:23.999+0000 7f3a19d90640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 0x7f39f806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.001+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a140b0540 con 0x7f3a140661c0 2023-10-16T15:33:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.002+0000 7f3a0affd640 1 -- 172.21.15.29:0/4067517752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a140b0540 con 0x7f3a140661c0 2023-10-16T15:33:24.004 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:33:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 msgr2=0x7f39f806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 0x7f39f806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 msgr2=0x7f3a140b4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 0x7f3a140b4040 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3a1002b950 tx=0x7f3a100046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 shutdown_connections 2023-10-16T15:33:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f39f806af90 0x7f39f806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a1415d330 0x7f3a140ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a1406c7d0 0x7f3a140b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 --2- 172.21.15.29:0/4067517752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a140661c0 0x7f3a140b4040 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 >> 172.21.15.29:0/4067517752 conn(0x7f3a140d0ab0 msgr2=0x7f3a140c4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:24.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 shutdown_connections 2023-10-16T15:33:24.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.005+0000 7f3a1c7f5640 1 -- 172.21.15.29:0/4067517752 wait complete. 2023-10-16T15:33:24.010 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.008+0000 7f441a830040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:33:24.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:33:24.030 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.028+0000 7f0d88a30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:33:24.075 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.074+0000 7f441a830040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:33:24.128 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.127+0000 7f441a830040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:33:24.141 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.139+0000 7f0d88a30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:33:24.203 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.201+0000 7f0d88a30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:24.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.239+0000 7f5261743640 1 Processor -- start 2023-10-16T15:33:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.239+0000 7f5261743640 1 -- start start 2023-10-16T15:33:24.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.240+0000 7f5261743640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f525c15cd70 0x7f525c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.240+0000 7f5261743640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f525c0b8480 0x7f525c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f5261743640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c15d150 con 0x7f525c0d0be0 2023-10-16T15:33:24.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f5261743640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c0d1180 con 0x7f525c0b8480 2023-10-16T15:33:24.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f5261743640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c0d1460 con 0x7f525c15cd70 2023-10-16T15:33:24.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f525b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f525c0b8480 0x7f525c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f525b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f525c0b8480 0x7f525c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44552/0 (socket says 172.21.15.29:44552) 2023-10-16T15:33:24.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f525b7fe640 1 -- 172.21.15.29:0/2242665066 learned_addr learned my addr 172.21.15.29:0/2242665066 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:24.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.241+0000 7f525bfff640 1 --2- 172.21.15.29:0/2242665066 >> v2:172.21.15.170:3300/0 conn(0x7f525c15cd70 0x7f525c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.242+0000 7f525b7fe640 1 -- 172.21.15.29:0/2242665066 >> v2:172.21.15.170:3300/0 conn(0x7f525c15cd70 msgr2=0x7f525c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:24.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.242+0000 7f525b7fe640 1 --2- 172.21.15.29:0/2242665066 >> v2:172.21.15.170:3300/0 conn(0x7f525c15cd70 0x7f525c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.242+0000 7f525b7fe640 1 -- 172.21.15.29:0/2242665066 >> v1:172.21.15.170:6789/0 conn(0x7f525c0d0be0 legacy=0x7f525c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:24.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.242+0000 7f525b7fe640 1 -- 172.21.15.29:0/2242665066 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c0d1740 con 0x7f525c0b8480 2023-10-16T15:33:24.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.243+0000 7f525b7fe640 1 --2- 172.21.15.29:0/2242665066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0b8480 0x7f525c0cd7c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f525c0b56a0 tx=0x7f525002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d760138cebca3b2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:24.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.243+0000 7f525affd640 1 -- 172.21.15.29:0/2242665066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52500317e0 con 0x7f525c0b8480 2023-10-16T15:33:24.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.243+0000 7f525affd640 1 -- 172.21.15.29:0/2242665066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250031940 con 0x7f525c0b8480 2023-10-16T15:33:24.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.243+0000 7f525affd640 1 -- 172.21.15.29:0/2242665066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250031cd0 con 0x7f525c0b8480 2023-10-16T15:33:24.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 -- 172.21.15.29:0/2242665066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0b8480 msgr2=0x7f525c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 --2- 172.21.15.29:0/2242665066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0b8480 0x7f525c0cd7c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f525c0b56a0 tx=0x7f525002b880 comp rx=0 tx=0).stop 2023-10-16T15:33:24.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 -- 172.21.15.29:0/2242665066 shutdown_connections 2023-10-16T15:33:24.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 --2- 172.21.15.29:0/2242665066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0b8480 0x7f525c0cd7c0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 --2- 172.21.15.29:0/2242665066 >> v2:172.21.15.170:3300/0 conn(0x7f525c15cd70 0x7f525c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 -- 172.21.15.29:0/2242665066 >> 172.21.15.29:0/2242665066 conn(0x7f525c0cf570 msgr2=0x7f525c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:24.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 -- 172.21.15.29:0/2242665066 shutdown_connections 2023-10-16T15:33:24.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.244+0000 7f5261743640 1 -- 172.21.15.29:0/2242665066 wait complete. 2023-10-16T15:33:24.249 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.245+0000 7f441a830040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:33:24.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.246+0000 7f5261743640 1 Processor -- start 2023-10-16T15:33:24.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.246+0000 7f5261743640 1 -- start start 2023-10-16T15:33:24.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.247+0000 7f5261743640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f525c0b8480 0x7f525c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.247+0000 7f5261743640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 0x7f525c0a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.247+0000 7f5261743640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525c15cd70 0x7f525c0a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.247+0000 7f5261743640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0cb550 con 0x7f525c0d0be0 2023-10-16T15:33:24.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.247+0000 7f5261743640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0ca8d0 con 0x7f525c0b8480 2023-10-16T15:33:24.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.247+0000 7f5261743640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0cabb0 con 0x7f525c15cd70 2023-10-16T15:33:24.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 0x7f525c0a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 0x7f525c0a94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44562/0 (socket says 172.21.15.29:44562) 2023-10-16T15:33:24.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 -- 172.21.15.29:0/508617602 learned_addr learned my addr 172.21.15.29:0/508617602 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:24.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525bfff640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f525c0b8480 0x7f525c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f5260f42640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525c15cd70 0x7f525c0a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 -- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525c15cd70 msgr2=0x7f525c0a9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525c15cd70 0x7f525c0a9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 -- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f525c0b8480 msgr2=0x7f525c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f525c0b8480 0x7f525c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.248+0000 7f525b7fe640 1 -- 172.21.15.29:0/508617602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500094d0 con 0x7f525c0d0be0 2023-10-16T15:33:24.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.249+0000 7f525b7fe640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 0x7f525c0a94f0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f525002b8c0 tx=0x7f52500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:24.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.249+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250032880 con 0x7f525c0d0be0 2023-10-16T15:33:24.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.249+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250032a20 con 0x7f525c0d0be0 2023-10-16T15:33:24.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.249+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c0cae90 con 0x7f525c0d0be0 2023-10-16T15:33:24.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.249+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250032dd0 con 0x7f525c0d0be0 2023-10-16T15:33:24.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.249+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c158640 con 0x7f525c0d0be0 2023-10-16T15:33:24.256 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.252+0000 7f0d88a30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:33:24.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.250+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c0d28f0 con 0x7f525c0d0be0 2023-10-16T15:33:24.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.251+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f52500453f0 con 0x7f525c0d0be0 2023-10-16T15:33:24.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.252+0000 7f5258ff9640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f524806af90 0x7f524806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.252+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f52500abc30 con 0x7f525c0d0be0 2023-10-16T15:33:24.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.252+0000 7f525bfff640 1 -- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f524806af90 msgr2=0x7f524806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.252+0000 7f525bfff640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f524806af90 0x7f524806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.257+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5250076810 con 0x7f525c0d0be0 2023-10-16T15:33:24.311 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.310+0000 7f0d88a30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:33:24.314 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.313+0000 7f441a830040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:24.361 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.359+0000 7f0d88a30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:33:24.366 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.365+0000 7f441a830040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:33:24.427 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.425+0000 7f441a830040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:33:24.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.448+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f525c0cb690 con 0x7f525c0d0be0 2023-10-16T15:33:24.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.449+0000 7f5258ff9640 1 -- 172.21.15.29:0/508617602 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v165) v1 ==== 104+0+11 (secure 0 0 0) 0x7f525c0cb690 con 0x7f525c0d0be0 2023-10-16T15:33:24.451 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:33:24.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f524806af90 msgr2=0x7f524806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f524806af90 0x7f524806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 msgr2=0x7f525c0a94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 0x7f525c0a94f0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f525002b8c0 tx=0x7f52500046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 shutdown_connections 2023-10-16T15:33:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f524806af90 0x7f524806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525c0d0be0 0x7f525c0a94f0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525c15cd70 0x7f525c0a9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 --2- 172.21.15.29:0/508617602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f525c0b8480 0x7f525c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 >> 172.21.15.29:0/508617602 conn(0x7f525c0cf570 msgr2=0x7f525c0cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:24.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 shutdown_connections 2023-10-16T15:33:24.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.452+0000 7f5261743640 1 -- 172.21.15.29:0/508617602 wait complete. 2023-10-16T15:33:24.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:33:24.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.482+0000 7f441a830040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:33:24.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.684+0000 7f74bca52640 1 Processor -- start 2023-10-16T15:33:24.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.685+0000 7f74bca52640 1 -- start start 2023-10-16T15:33:24.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.686+0000 7f74bca52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 0x7f74b80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.686+0000 7f74bca52640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f74b80b8480 0x7f74b80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.686+0000 7f74bca52640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b815d150 con 0x7f74b80d0be0 2023-10-16T15:33:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.686+0000 7f74bca52640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b80d1180 con 0x7f74b80b8480 2023-10-16T15:33:24.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.686+0000 7f74bca52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80d1460 con 0x7f74b815cd70 2023-10-16T15:33:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.687+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 0x7f74b80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.687+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 0x7f74b80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:47838/0 (socket says 172.21.15.29:47838) 2023-10-16T15:33:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.687+0000 7f74affff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f74b80b8480 0x7f74b80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.687+0000 7f74b77fe640 1 -- 172.21.15.29:0/2472400940 learned_addr learned my addr 172.21.15.29:0/2472400940 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.688+0000 7f74b77fe640 1 -- 172.21.15.29:0/2472400940 >> v2:172.21.15.170:3300/0 conn(0x7f74b80b8480 msgr2=0x7f74b80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.688+0000 7f74b77fe640 1 --2- 172.21.15.29:0/2472400940 >> v2:172.21.15.170:3300/0 conn(0x7f74b80b8480 0x7f74b80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.688+0000 7f74b77fe640 1 -- 172.21.15.29:0/2472400940 >> v1:172.21.15.170:6789/0 conn(0x7f74b80d0be0 legacy=0x7f74b80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.688+0000 7f74b77fe640 1 -- 172.21.15.29:0/2472400940 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b80d1740 con 0x7f74b815cd70 2023-10-16T15:33:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.689+0000 7f74b77fe640 1 --2- 172.21.15.29:0/2472400940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 0x7f74b80d2220 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009870 tx=0x7f74a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8181d894e195465 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74b6ffd640 1 -- 172.21.15.29:0/2472400940 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8030840 con 0x7f74b815cd70 2023-10-16T15:33:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74b6ffd640 1 -- 172.21.15.29:0/2472400940 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a80309a0 con 0x7f74b815cd70 2023-10-16T15:33:24.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74b6ffd640 1 -- 172.21.15.29:0/2472400940 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8030d30 con 0x7f74b815cd70 2023-10-16T15:33:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74bca52640 1 -- 172.21.15.29:0/2472400940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 msgr2=0x7f74b80d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74bca52640 1 --2- 172.21.15.29:0/2472400940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 0x7f74b80d2220 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009870 tx=0x7f74a802b980 comp rx=0 tx=0).stop 2023-10-16T15:33:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74bca52640 1 -- 172.21.15.29:0/2472400940 shutdown_connections 2023-10-16T15:33:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74bca52640 1 --2- 172.21.15.29:0/2472400940 >> v2:172.21.15.170:3300/0 conn(0x7f74b80b8480 0x7f74b80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74bca52640 1 --2- 172.21.15.29:0/2472400940 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b815cd70 0x7f74b80d2220 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.690+0000 7f74bca52640 1 -- 172.21.15.29:0/2472400940 >> 172.21.15.29:0/2472400940 conn(0x7f74b80cf570 msgr2=0x7f74b80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.691+0000 7f74bca52640 1 -- 172.21.15.29:0/2472400940 shutdown_connections 2023-10-16T15:33:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.691+0000 7f74bca52640 1 -- 172.21.15.29:0/2472400940 wait complete. 2023-10-16T15:33:24.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.692+0000 7f74bca52640 1 Processor -- start 2023-10-16T15:33:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 -- start start 2023-10-16T15:33:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b80b8480 0x7f74b80c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b80d0be0 0x7f74b8151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 0x7f74b815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b81f1ef0 con 0x7f74b815cd70 2023-10-16T15:33:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80c7d20 con 0x7f74b80d0be0 2023-10-16T15:33:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.693+0000 7f74bca52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80c8000 con 0x7f74b80b8480 2023-10-16T15:33:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 0x7f74b815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 0x7f74b815edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44566/0 (socket says 172.21.15.29:44566) 2023-10-16T15:33:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 -- 172.21.15.29:0/613167806 learned_addr learned my addr 172.21.15.29:0/613167806 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b77fe640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b80b8480 0x7f74b80c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74affff640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b80d0be0 0x7f74b8151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 -- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b80b8480 msgr2=0x7f74b80c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b80b8480 0x7f74b80c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 -- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b80d0be0 msgr2=0x7f74b8151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b80d0be0 0x7f74b8151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74b7fff640 1 -- 172.21.15.29:0/613167806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80094d0 con 0x7f74b815cd70 2023-10-16T15:33:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.694+0000 7f74affff640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b80d0be0 0x7f74b8151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:24.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.695+0000 7f74b7fff640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 0x7f74b815edc0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c730 tx=0x7f74b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.695+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0003b70 con 0x7f74b815cd70 2023-10-16T15:33:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.695+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0003cf0 con 0x7f74b815cd70 2023-10-16T15:33:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.695+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0003ea0 con 0x7f74b815cd70 2023-10-16T15:33:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.695+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b80c82e0 con 0x7f74b815cd70 2023-10-16T15:33:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.695+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b80c88a0 con 0x7f74b815cd70 2023-10-16T15:33:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.697+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b80c8ac0 con 0x7f74b815cd70 2023-10-16T15:33:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.697+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f74b0005520 con 0x7f74b815cd70 2023-10-16T15:33:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.698+0000 7f74b4ff9640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f749806af90 0x7f749806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.698+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f74b00898b0 con 0x7f74b815cd70 2023-10-16T15:33:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.699+0000 7f74b77fe640 1 -- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f749806af90 msgr2=0x7f749806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:24.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.699+0000 7f74b77fe640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f749806af90 0x7f749806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:24.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.703+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74b0054640 con 0x7f74b815cd70 2023-10-16T15:33:24.742 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.740+0000 7f0d88a30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:33:24.890 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:24.889+0000 7f441a830040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:33:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.893+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74b804e9b0 con 0x7f74b815cd70 2023-10-16T15:33:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.894+0000 7f74b4ff9640 1 -- 172.21.15.29:0/613167806 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v165) v1 ==== 100+0+11 (secure 0 0 0) 0x7f74b0057c60 con 0x7f74b815cd70 2023-10-16T15:33:24.896 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:33:24.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f749806af90 msgr2=0x7f749806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f749806af90 0x7f749806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 msgr2=0x7f74b815edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 0x7f74b815edc0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c730 tx=0x7f74b000cbf0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 shutdown_connections 2023-10-16T15:33:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f749806af90 0x7f749806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74b815cd70 0x7f74b815edc0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74b80d0be0 0x7f74b8151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 --2- 172.21.15.29:0/613167806 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f74b80b8480 0x7f74b80c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 >> 172.21.15.29:0/613167806 conn(0x7f74b80cf570 msgr2=0x7f74b80b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 shutdown_connections 2023-10-16T15:33:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:24.897+0000 7f74bca52640 1 -- 172.21.15.29:0/613167806 wait complete. 2023-10-16T15:33:24.901 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:24.897+0000 7f0d88a30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:33:24.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:33:25.007 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.005+0000 7f0d88a30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:33:25.056 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.054+0000 7f0d88a30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:33:25.064 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.062+0000 7f441a830040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:33:25.162 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.160+0000 7f0d88a30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:25.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.163+0000 7f6c5c82e640 1 Processor -- start 2023-10-16T15:33:25.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.164+0000 7f6c5c82e640 1 -- start start 2023-10-16T15:33:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.165+0000 7f6c5c82e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c580b8480 0x7f6c580cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.165+0000 7f6c5c82e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6c580d0be0 0x7f6c580cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.165+0000 7f6c5c82e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5815d150 con 0x7f6c5815cd70 2023-10-16T15:33:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.165+0000 7f6c5c82e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c580d1180 con 0x7f6c580b8480 2023-10-16T15:33:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.165+0000 7f6c5c82e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c580d1460 con 0x7f6c580d0be0 2023-10-16T15:33:25.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.165+0000 7f6c4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c580b8480 0x7f6c580cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c580b8480 0x7f6c580cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44578/0 (socket says 172.21.15.29:44578) 2023-10-16T15:33:25.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c4ffff640 1 -- 172.21.15.29:0/1631302740 learned_addr learned my addr 172.21.15.29:0/1631302740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:25.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c57fff640 1 --2- 172.21.15.29:0/1631302740 >> v2:172.21.15.170:3300/0 conn(0x7f6c580d0be0 0x7f6c580cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c4ffff640 1 -- 172.21.15.29:0/1631302740 >> v2:172.21.15.170:3300/0 conn(0x7f6c580d0be0 msgr2=0x7f6c580cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c4ffff640 1 --2- 172.21.15.29:0/1631302740 >> v2:172.21.15.170:3300/0 conn(0x7f6c580d0be0 0x7f6c580cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c4ffff640 1 -- 172.21.15.29:0/1631302740 >> v1:172.21.15.170:6789/0 conn(0x7f6c5815cd70 legacy=0x7f6c580d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.166+0000 7f6c4ffff640 1 -- 172.21.15.29:0/1631302740 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c580d1740 con 0x7f6c580b8480 2023-10-16T15:33:25.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.168+0000 7f6c4ffff640 1 --2- 172.21.15.29:0/1631302740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580b8480 0x7f6c580cd7c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f6c580b8910 tx=0x7f6c4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72de085ad2ae5f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:25.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.168+0000 7f6c56ffd640 1 -- 172.21.15.29:0/1631302740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40031870 con 0x7f6c580b8480 2023-10-16T15:33:25.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.168+0000 7f6c56ffd640 1 -- 172.21.15.29:0/1631302740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c400319d0 con 0x7f6c580b8480 2023-10-16T15:33:25.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.168+0000 7f6c56ffd640 1 -- 172.21.15.29:0/1631302740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40031d60 con 0x7f6c580b8480 2023-10-16T15:33:25.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 -- 172.21.15.29:0/1631302740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580b8480 msgr2=0x7f6c580cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/1631302740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580b8480 0x7f6c580cd7c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f6c580b8910 tx=0x7f6c4002b910 comp rx=0 tx=0).stop 2023-10-16T15:33:25.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 -- 172.21.15.29:0/1631302740 shutdown_connections 2023-10-16T15:33:25.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/1631302740 >> v2:172.21.15.170:3300/0 conn(0x7f6c580d0be0 0x7f6c580cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/1631302740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580b8480 0x7f6c580cd7c0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 -- 172.21.15.29:0/1631302740 >> 172.21.15.29:0/1631302740 conn(0x7f6c580cf570 msgr2=0x7f6c580cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:25.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 -- 172.21.15.29:0/1631302740 shutdown_connections 2023-10-16T15:33:25.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.169+0000 7f6c5c82e640 1 -- 172.21.15.29:0/1631302740 wait complete. 2023-10-16T15:33:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.171+0000 7f6c5c82e640 1 Processor -- start 2023-10-16T15:33:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.171+0000 7f6c5c82e640 1 -- start start 2023-10-16T15:33:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c5c82e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c580b8480 0x7f6c58155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c5c82e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 0x7f6c58145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c5c82e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c5815cd70 0x7f6c58146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c5c82e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c58154f80 con 0x7f6c580d0be0 2023-10-16T15:33:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c5c82e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c58154370 con 0x7f6c580b8480 2023-10-16T15:33:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c5c82e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c58154650 con 0x7f6c5815cd70 2023-10-16T15:33:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 0x7f6c58145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 0x7f6c58145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44584/0 (socket says 172.21.15.29:44584) 2023-10-16T15:33:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c4ffff640 1 -- 172.21.15.29:0/2407888992 learned_addr learned my addr 172.21.15.29:0/2407888992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c57fff640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c5815cd70 0x7f6c58146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.172+0000 7f6c577fe640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c580b8480 0x7f6c58155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c4ffff640 1 -- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c5815cd70 msgr2=0x7f6c58146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c4ffff640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c5815cd70 0x7f6c58146190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c4ffff640 1 -- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c580b8480 msgr2=0x7f6c58155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c4ffff640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c580b8480 0x7f6c58155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c4ffff640 1 -- 172.21.15.29:0/2407888992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c400094d0 con 0x7f6c580d0be0 2023-10-16T15:33:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c4ffff640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 0x7f6c58145c50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f6c4002b950 tx=0x7f6c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:25.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4003ad30 con 0x7f6c580d0be0 2023-10-16T15:33:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.173+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c400443f0 con 0x7f6c580d0be0 2023-10-16T15:33:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.174+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c400447a0 con 0x7f6c580d0be0 2023-10-16T15:33:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.174+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c58154930 con 0x7f6c580d0be0 2023-10-16T15:33:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.174+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c58148950 con 0x7f6c580d0be0 2023-10-16T15:33:25.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.175+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c58148b40 con 0x7f6c580d0be0 2023-10-16T15:33:25.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.177+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f6c4004d450 con 0x7f6c580d0be0 2023-10-16T15:33:25.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.178+0000 7f6c54ff9640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 0x7f6c3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.178+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f6c4002d040 con 0x7f6c580d0be0 2023-10-16T15:33:25.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.178+0000 7f6c577fe640 1 -- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 msgr2=0x7f6c3806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:25.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.178+0000 7f6c577fe640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 0x7f6c3806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:25.182 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.181+0000 7f441a830040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:33:25.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.183+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c58148b40 con 0x7f6c580d0be0 2023-10-16T15:33:25.236 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.234+0000 7f441a830040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:33:25.268 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.266+0000 7f0d88a30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:33:25.350 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.348+0000 7f441a830040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:25.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.379+0000 7f6c577fe640 1 -- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 msgr2=0x7f6c3806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:25.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.379+0000 7f6c577fe640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 0x7f6c3806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:25.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.388+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6c58148b40 con 0x7f6c580d0be0 2023-10-16T15:33:25.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.389+0000 7f6c54ff9640 1 -- 172.21.15.29:0/2407888992 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v165) v1 ==== 80+0+474 (secure 0 0 0) 0x7f6c58148b40 con 0x7f6c580d0be0 2023-10-16T15:33:25.390 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/91040253 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.391 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/999637714 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.391 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/907915415 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.391 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/1985319551 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.391 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/1985319551 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.391 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3571017587 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.392 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1946312558 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.392 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1694874697 2023-10-17T14:37:28.457671+0000 2023-10-16T15:33:25.393 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T15:33:25.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 msgr2=0x7f6c3806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:25.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 0x7f6c3806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 msgr2=0x7f6c58145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 0x7f6c58145c50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f6c4002b950 tx=0x7f6c400046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 shutdown_connections 2023-10-16T15:33:25.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7f6c3806af90 0x7f6c3806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c580d0be0 0x7f6c58145c50 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c5815cd70 0x7f6c58146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 --2- 172.21.15.29:0/2407888992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c580b8480 0x7f6c58155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 >> 172.21.15.29:0/2407888992 conn(0x7f6c580cf570 msgr2=0x7f6c580b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:25.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.391+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 shutdown_connections 2023-10-16T15:33:25.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.392+0000 7f6c5c82e640 1 -- 172.21.15.29:0/2407888992 wait complete. 2023-10-16T15:33:25.411 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:33:25.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:25.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:33:25.424 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.422+0000 7f0d88a30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:25.466 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.464+0000 7f441a830040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:33:25.472 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.471+0000 7f0d88a30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:25.513 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:33:25.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:25.513 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:33:25.563 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:33:25.564 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:25.591 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:25.590+0000 7f0d88a30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:33:25.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:33:25.620 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:33:25.621 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:25.638 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.636+0000 7f441a830040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:25.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:33:25.679 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:33:25.679 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:25.692 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.691+0000 7f441a830040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:25.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:33:25.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:25.820 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:25.819+0000 7f441a830040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:33:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.949+0000 7fa451640640 1 Processor -- start 2023-10-16T15:33:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.949+0000 7fa451640640 1 -- start start 2023-10-16T15:33:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.950+0000 7fa451640640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c15cd70 0x7fa44c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa451640640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa44c0b8480 0x7fa44c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa451640640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c15d150 con 0x7fa44c0d0be0 2023-10-16T15:33:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa451640640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c0d1180 con 0x7fa44c0b8480 2023-10-16T15:33:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa451640640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0d1460 con 0x7fa44c15cd70 2023-10-16T15:33:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa4437fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa44c0b8480 0x7fa44c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa4437fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa44c0b8480 0x7fa44c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42464/0 (socket says 172.21.15.29:42464) 2023-10-16T15:33:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa4437fe640 1 -- 172.21.15.29:0/3045030432 learned_addr learned my addr 172.21.15.29:0/3045030432 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.951+0000 7fa44bfff640 1 --2- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c15cd70 0x7fa44c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.952+0000 7fa4437fe640 1 -- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c15cd70 msgr2=0x7fa44c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.952+0000 7fa4437fe640 1 --2- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c15cd70 0x7fa44c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.952+0000 7fa4437fe640 1 -- 172.21.15.29:0/3045030432 >> v1:172.21.15.170:6789/0 conn(0x7fa44c0d0be0 legacy=0x7fa44c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.952+0000 7fa4437fe640 1 -- 172.21.15.29:0/3045030432 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0d1740 con 0x7fa44c0b8480 2023-10-16T15:33:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.954+0000 7fa4437fe640 1 --2- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0b8480 0x7fa44c0cd7c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa434009870 tx=0x7fa43402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7e3783fee4d427f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.954+0000 7fa44b7fe640 1 -- 172.21.15.29:0/3045030432 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434030840 con 0x7fa44c0b8480 2023-10-16T15:33:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.954+0000 7fa44b7fe640 1 -- 172.21.15.29:0/3045030432 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4340309a0 con 0x7fa44c0b8480 2023-10-16T15:33:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.954+0000 7fa44b7fe640 1 -- 172.21.15.29:0/3045030432 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434030d30 con 0x7fa44c0b8480 2023-10-16T15:33:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 -- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0b8480 msgr2=0x7fa44c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 --2- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0b8480 0x7fa44c0cd7c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa434009870 tx=0x7fa43402b980 comp rx=0 tx=0).stop 2023-10-16T15:33:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 -- 172.21.15.29:0/3045030432 shutdown_connections 2023-10-16T15:33:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 --2- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0b8480 0x7fa44c0cd7c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 --2- 172.21.15.29:0/3045030432 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c15cd70 0x7fa44c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 -- 172.21.15.29:0/3045030432 >> 172.21.15.29:0/3045030432 conn(0x7fa44c0cf570 msgr2=0x7fa44c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 -- 172.21.15.29:0/3045030432 shutdown_connections 2023-10-16T15:33:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.955+0000 7fa451640640 1 -- 172.21.15.29:0/3045030432 wait complete. 2023-10-16T15:33:25.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.957+0000 7fa451640640 1 Processor -- start 2023-10-16T15:33:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.957+0000 7fa451640640 1 -- start start 2023-10-16T15:33:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.957+0000 7fa451640640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c0b8480 0x7fa44c0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa451640640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0d0be0 0x7fa44c151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa451640640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 0x7fa44c1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa451640640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c0c9260 con 0x7fa44c15cd70 2023-10-16T15:33:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa451640640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c15d960 con 0x7fa44c0d0be0 2023-10-16T15:33:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa451640640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c15dc40 con 0x7fa44c0b8480 2023-10-16T15:33:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 0x7fa44c1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 0x7fa44c1520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44592/0 (socket says 172.21.15.29:44592) 2023-10-16T15:33:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 -- 172.21.15.29:0/282199297 learned_addr learned my addr 172.21.15.29:0/282199297 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa4437fe640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0d0be0 0x7fa44c151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa44bfff640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c0b8480 0x7fa44c0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 -- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c0b8480 msgr2=0x7fa44c0c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c0b8480 0x7fa44c0c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 -- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0d0be0 msgr2=0x7fa44c151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0d0be0 0x7fa44c151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.958+0000 7fa450e3f640 1 -- 172.21.15.29:0/282199297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340094d0 con 0x7fa44c15cd70 2023-10-16T15:33:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.959+0000 7fa450e3f640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 0x7fa44c1520b0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c3c0 tx=0x7fa44400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.959+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4440052d0 con 0x7fa44c15cd70 2023-10-16T15:33:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.959+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa444005430 con 0x7fa44c15cd70 2023-10-16T15:33:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.959+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444003830 con 0x7fa44c15cd70 2023-10-16T15:33:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.959+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c15df20 con 0x7fa44c15cd70 2023-10-16T15:33:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.959+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c0c6c90 con 0x7fa44c15cd70 2023-10-16T15:33:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.960+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c15d7c0 con 0x7fa44c15cd70 2023-10-16T15:33:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.962+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa444003990 con 0x7fa44c15cd70 2023-10-16T15:33:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.963+0000 7fa4497fa640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 0x7fa42c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.963+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fa444088ea0 con 0x7fa44c15cd70 2023-10-16T15:33:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.963+0000 7fa44bfff640 1 -- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 msgr2=0x7fa42c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.963+0000 7fa44bfff640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 0x7fa42c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:25.967+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa44c15d7c0 con 0x7fa44c15cd70 2023-10-16T15:33:26.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.164+0000 7fa44bfff640 1 -- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 msgr2=0x7fa42c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:26.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.164+0000 7fa44bfff640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 0x7fa42c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:26.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.197+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa44c15d7c0 con 0x7fa44c15cd70 2023-10-16T15:33:26.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.198+0000 7fa4497fa640 1 -- 172.21.15.29:0/282199297 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fa44c15d7c0 con 0x7fa44c15cd70 2023-10-16T15:33:26.199 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:26.200 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:37:13.609698+0000","modified":"2023-10-16T14:37:16.825159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8838},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8838":{"gid":8838,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3542022450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3542022450},{"type":"v1","addr":"172.21.15.29:6833","nonce":3542022450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T15:33:26.200 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 75 2023-10-16T15:33:26.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 msgr2=0x7fa42c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 0x7fa42c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 msgr2=0x7fa44c1520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 0x7fa44c1520b0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c3c0 tx=0x7fa44400c880 comp rx=0 tx=0).stop 2023-10-16T15:33:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 shutdown_connections 2023-10-16T15:33:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa42c06af90 0x7fa42c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa44c15cd70 0x7fa44c1520b0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa44c0d0be0 0x7fa44c151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.200+0000 7fa451640640 1 --2- 172.21.15.29:0/282199297 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa44c0b8480 0x7fa44c0c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.201+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 >> 172.21.15.29:0/282199297 conn(0x7fa44c0cf570 msgr2=0x7fa44c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.201+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 shutdown_connections 2023-10-16T15:33:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.201+0000 7fa451640640 1 -- 172.21.15.29:0/282199297 wait complete. 2023-10-16T15:33:26.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.422+0000 7fa66c664640 1 Processor -- start 2023-10-16T15:33:26.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.422+0000 7fa66c664640 1 -- start start 2023-10-16T15:33:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66c664640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa6640cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66c664640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa6640b0b90 0x7fa66415d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66c664640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6640cf940 con 0x7fa66406bca0 2023-10-16T15:33:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66c664640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6640ce990 con 0x7fa6640b0b90 2023-10-16T15:33:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66c664640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640d0310 con 0x7fa66406b8c0 2023-10-16T15:33:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66ac01640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa6640b0b90 0x7fa66415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66ac01640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa6640b0b90 0x7fa66415d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42478/0 (socket says 172.21.15.29:42478) 2023-10-16T15:33:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66ac01640 1 -- 172.21.15.29:0/441715487 learned_addr learned my addr 172.21.15.29:0/441715487 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.424+0000 7fa66a400640 1 --2- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa6640cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.425+0000 7fa66ac01640 1 -- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 msgr2=0x7fa6640cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:26.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.425+0000 7fa66ac01640 1 --2- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa6640cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.425+0000 7fa66ac01640 1 -- 172.21.15.29:0/441715487 >> v1:172.21.15.170:6789/0 conn(0x7fa66406bca0 legacy=0x7fa6640b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.425+0000 7fa66ac01640 1 -- 172.21.15.29:0/441715487 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66415e290 con 0x7fa6640b0b90 2023-10-16T15:33:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.426+0000 7fa66ac01640 1 --2- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6640b0b90 0x7fa66415d180 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009e50 tx=0x7fa65c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c49df01b278892f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.427+0000 7fa6693fe640 1 -- 172.21.15.29:0/441715487 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c02cb00 con 0x7fa6640b0b90 2023-10-16T15:33:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.427+0000 7fa6693fe640 1 -- 172.21.15.29:0/441715487 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c02cc60 con 0x7fa6640b0b90 2023-10-16T15:33:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.427+0000 7fa6693fe640 1 -- 172.21.15.29:0/441715487 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0343f0 con 0x7fa6640b0b90 2023-10-16T15:33:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.427+0000 7fa66c664640 1 -- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6640b0b90 msgr2=0x7fa66415d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.427+0000 7fa66c664640 1 --2- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6640b0b90 0x7fa66415d180 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009e50 tx=0x7fa65c02bf60 comp rx=0 tx=0).stop 2023-10-16T15:33:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.428+0000 7fa66c664640 1 -- 172.21.15.29:0/441715487 shutdown_connections 2023-10-16T15:33:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.428+0000 7fa66c664640 1 --2- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6640b0b90 0x7fa66415d180 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.428+0000 7fa66c664640 1 --2- 172.21.15.29:0/441715487 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa6640cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.428+0000 7fa66c664640 1 -- 172.21.15.29:0/441715487 >> 172.21.15.29:0/441715487 conn(0x7fa664099ea0 msgr2=0x7fa6640d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:26.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.428+0000 7fa66c664640 1 -- 172.21.15.29:0/441715487 shutdown_connections 2023-10-16T15:33:26.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.428+0000 7fa66c664640 1 -- 172.21.15.29:0/441715487 wait complete. 2023-10-16T15:33:26.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.429+0000 7fa66c664640 1 Processor -- start 2023-10-16T15:33:26.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 -- start start 2023-10-16T15:33:26.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa664155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66406bca0 0x7fa6640cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 0x7fa6640cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6641f3470 con 0x7fa6640b0b90 2023-10-16T15:33:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6641f28c0 con 0x7fa66406bca0 2023-10-16T15:33:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.430+0000 7fa66c664640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6641f3890 con 0x7fa66406b8c0 2023-10-16T15:33:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 0x7fa6640cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 0x7fa6640cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44604/0 (socket says 172.21.15.29:44604) 2023-10-16T15:33:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 -- 172.21.15.29:0/2924787629 learned_addr learned my addr 172.21.15.29:0/2924787629 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66a400640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa664155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa669bff640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66406bca0 0x7fa6640cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 -- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 msgr2=0x7fa664155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa664155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 -- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66406bca0 msgr2=0x7fa6640cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.431+0000 7fa66ac01640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66406bca0 0x7fa6640cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa66ac01640 1 -- 172.21.15.29:0/2924787629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c009b30 con 0x7fa6640b0b90 2023-10-16T15:33:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa66ac01640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 0x7fa6640cc6d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fa65c036040 tx=0x7fa65c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c009510 con 0x7fa6640b0b90 2023-10-16T15:33:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c03e450 con 0x7fa6640b0b90 2023-10-16T15:33:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c03e7e0 con 0x7fa6640b0b90 2023-10-16T15:33:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6641f2ad0 con 0x7fa6640b0b90 2023-10-16T15:33:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.432+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6641f3090 con 0x7fa6640b0b90 2023-10-16T15:33:26.439 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:26.432+0000 7f0d88a30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:33:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.434+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6641f35b0 con 0x7fa6640b0b90 2023-10-16T15:33:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.435+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa65c03e940 con 0x7fa6640b0b90 2023-10-16T15:33:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.436+0000 7fa65affd640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 0x7fa64806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.436+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fa65c0ac4d0 con 0x7fa6640b0b90 2023-10-16T15:33:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.436+0000 7fa66a400640 1 -- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 msgr2=0x7fa64806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.436+0000 7fa66a400640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 0x7fa64806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.440+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa65c0773a0 con 0x7fa6640b0b90 2023-10-16T15:33:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.633+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa66404e160 con 0x7fa6640b0b90 2023-10-16T15:33:26.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.635+0000 7fa65affd640 1 -- 172.21.15.29:0/2924787629 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+17336 (secure 0 0 0) 0x7fa65c07a9c0 con 0x7fa6640b0b90 2023-10-16T15:33:26.637 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:26.637 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":165,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:22.905947+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"165","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":165,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.3f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/91040253":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/999637714":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/907915415":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6833/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6832/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3571017587":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1946312558":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1694874697":"2023-10-17T14:37:28.457671+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":52,"length":5},{"begin":108,"length":5}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:26.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.637+0000 7fa66a400640 1 -- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 msgr2=0x7fa64806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:26.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.637+0000 7fa66a400640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 0x7fa64806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:26.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 msgr2=0x7fa64806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 0x7fa64806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 msgr2=0x7fa6640cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 0x7fa6640cc6d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fa65c036040 tx=0x7fa65c004a40 comp rx=0 tx=0).stop 2023-10-16T15:33:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 shutdown_connections 2023-10-16T15:33:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fa64806af90 0x7fa64806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6640b0b90 0x7fa6640cc6d0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa66406bca0 0x7fa6640cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 --2- 172.21.15.29:0/2924787629 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa66406b8c0 0x7fa664155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 >> 172.21.15.29:0/2924787629 conn(0x7fa664099ea0 msgr2=0x7fa66415c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 shutdown_connections 2023-10-16T15:33:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.638+0000 7fa66c664640 1 -- 172.21.15.29:0/2924787629 wait complete. 2023-10-16T15:33:26.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:37:11.546312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:37:12.609923+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 165, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:33:26.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:33:26.686 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:26.685+0000 7f441a830040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:33:26.738 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:26.737+0000 7f0d88a30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:33:26.803 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:26.801+0000 7f0d88a30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:33:26.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.867+0000 7fe48b59e640 1 Processor -- start 2023-10-16T15:33:26.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.867+0000 7fe48b59e640 1 -- start start 2023-10-16T15:33:26.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe48b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe48c0cb000 0x7fe48c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe48b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c15d0e0 con 0x7fe48c070380 2023-10-16T15:33:26.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe48b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c0cbd80 con 0x7fe48c15cd00 2023-10-16T15:33:26.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe48b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0cbaa0 con 0x7fe48c0cb000 2023-10-16T15:33:26.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe48ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe48c070380 0x7fe48c0ad920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47572/0 (socket says 172.21.15.29:47572) 2023-10-16T15:33:26.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe48ad9d640 1 -- 172.21.15.29:0/373708617 learned_addr learned my addr 172.21.15.29:0/373708617 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:26.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.869+0000 7fe483fff640 1 --2- 172.21.15.29:0/373708617 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0cb000 0x7fe48c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360512244 0 0) 0x7fe48c15d0e0 con 0x7fe48c070380 2023-10-16T15:33:26.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe478003520 con 0x7fe48c070380 2023-10-16T15:33:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3504463734 0 0) 0x7fe478003520 con 0x7fe48c070380 2023-10-16T15:33:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0cb000 msgr2=0x7fe48c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 --2- 172.21.15.29:0/373708617 >> v2:172.21.15.170:3300/0 conn(0x7fe48c0cb000 0x7fe48c0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 >> v1:172.21.15.170:6789/0 conn(0x7fe48c15cd00 legacy=0x7fe48c0cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0cd500 con 0x7fe48c070380 2023-10-16T15:33:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe4840019f0 con 0x7fe48c070380 2023-10-16T15:33:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 >> v1:172.21.15.29:6789/0 conn(0x7fe48c070380 legacy=0x7fe48c0ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478003970 0x7fe478003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4780047c0 0x7fe478004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478005180 0x7fe47800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0cbaa0 con 0x7fe478005180 2023-10-16T15:33:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47800c020 con 0x7fe478003970 2023-10-16T15:33:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.870+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47800c1e0 con 0x7fe4780047c0 2023-10-16T15:33:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478005180 0x7fe47800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48a59c640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478003970 0x7fe478003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe483fff640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4780047c0 0x7fe478004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48ad9d640 1 -- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4780047c0 msgr2=0x7fe478004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4780047c0 0x7fe478004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48ad9d640 1 -- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478003970 msgr2=0x7fe478003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478003970 0x7fe478003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe48ad9d640 1 -- 172.21.15.29:0/373708617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478003520 con 0x7fe478005180 2023-10-16T15:33:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.871+0000 7fe483fff640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4780047c0 0x7fe478004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:26.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.872+0000 7fe48ad9d640 1 -- 172.21.15.29:0/373708617 reap_dead start 2023-10-16T15:33:26.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.873+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478005180 0x7fe47800b980 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe48400a830 tx=0x7fe4780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e00258a9f554bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.873+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840072d0 con 0x7fe478005180 2023-10-16T15:33:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.873+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48402f070 con 0x7fe478005180 2023-10-16T15:33:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.873+0000 7fe489d9b640 1 -- 172.21.15.29:0/373708617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840076b0 con 0x7fe478005180 2023-10-16T15:33:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 -- 172.21.15.29:0/373708617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478005180 msgr2=0x7fe47800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478005180 0x7fe47800b980 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe48400a830 tx=0x7fe4780057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 -- 172.21.15.29:0/373708617 shutdown_connections 2023-10-16T15:33:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 --2- 172.21.15.29:0/373708617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478005180 0x7fe47800b980 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 -- 172.21.15.29:0/373708617 >> 172.21.15.29:0/373708617 conn(0x7fe48c0cf790 msgr2=0x7fe48c0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 -- 172.21.15.29:0/373708617 shutdown_connections 2023-10-16T15:33:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.874+0000 7fe48b59e640 1 -- 172.21.15.29:0/373708617 wait complete. 2023-10-16T15:33:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.876+0000 7fe48b59e640 1 Processor -- start 2023-10-16T15:33:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.876+0000 7fe48b59e640 1 -- start start 2023-10-16T15:33:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478005180 0x7fe48c0c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c14bac0 0x7fe48c06ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 0x7fe48c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c4c70 con 0x7fe48c06bf50 2023-10-16T15:33:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c069da0 con 0x7fe478005180 2023-10-16T15:33:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c06a5b0 con 0x7fe48c14bac0 2023-10-16T15:33:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 0x7fe48c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 0x7fe48c0c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44624/0 (socket says 172.21.15.29:44624) 2023-10-16T15:33:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48ad9d640 1 -- 172.21.15.29:0/332639586 learned_addr learned my addr 172.21.15.29:0/332639586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe48a59c640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478005180 0x7fe48c0c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.877+0000 7fe483fff640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c14bac0 0x7fe48c06ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48ad9d640 1 -- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c14bac0 msgr2=0x7fe48c06ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c14bac0 0x7fe48c06ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48ad9d640 1 -- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478005180 msgr2=0x7fe48c0c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478005180 0x7fe48c0c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48ad9d640 1 -- 172.21.15.29:0/332639586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c069ff0 con 0x7fe48c06bf50 2023-10-16T15:33:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48a59c640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478005180 0x7fe48c0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 0x7fe48c0c5090 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fe48400a950 tx=0x7fe484001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484001930 con 0x7fe48c06bf50 2023-10-16T15:33:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484001250 con 0x7fe48c06bf50 2023-10-16T15:33:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484004da0 con 0x7fe48c06bf50 2023-10-16T15:33:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.878+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c06a2d0 con 0x7fe48c06bf50 2023-10-16T15:33:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.879+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c159920 con 0x7fe48c06bf50 2023-10-16T15:33:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.880+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0c46f0 con 0x7fe48c06bf50 2023-10-16T15:33:26.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.882+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe484031030 con 0x7fe48c06bf50 2023-10-16T15:33:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.882+0000 7fe482ffd640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.883+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fe4840ab630 con 0x7fe48c06bf50 2023-10-16T15:33:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.883+0000 7fe48a59c640 1 -- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fe46c06af90 msgr2=0x7fe46c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3272758872 2023-10-16T15:33:26.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.883+0000 7fe48a59c640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:26.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.887+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe48c0c46f0 con 0x7fe48c06bf50 2023-10-16T15:33:26.920 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:26.918+0000 7f0d88a30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:33:26.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.951+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 86) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe4840765d0 con 0x7fe48c06bf50 2023-10-16T15:33:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.952+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fe46c06af90 msgr2=0x7fe46c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:26.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:26.952+0000 7fe482ffd640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.018 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.017+0000 7f0d0cd39640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.019 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.017+0000 7f0d0cd39640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.019 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.017+0000 7f0d0cd39640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.019 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.017+0000 7f0d0cd39640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.020 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.017+0000 7f0d0cd39640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.020 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.018+0000 7f0d0a534640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.021 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.018+0000 7f0d0a534640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.021 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.018+0000 7f0d0a534640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.021 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.018+0000 7f0d0a534640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.021 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:27.018+0000 7f0d0a534640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:33:27.039 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:27.038+0000 7f441a830040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:33:27.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.098+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe48c0c4db0 con 0x7fe48c06bf50 2023-10-16T15:33:27.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.099+0000 7fe482ffd640 1 -- 172.21.15.29:0/332639586 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe48c0c4db0 con 0x7fe48c06bf50 2023-10-16T15:33:27.101 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:33:27.101 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:27.100+0000 7f441a830040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:33:27.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.102+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 msgr2=0x7fe48c0c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.102+0000 7fe48b59e640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 0x7fe48c0c5090 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fe48400a950 tx=0x7fe484001d80 comp rx=0 tx=0).stop 2023-10-16T15:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.102+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 shutdown_connections 2023-10-16T15:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:6832/3272758872,v1:172.21.15.170:6833/3272758872] conn(0x7fe46c06af90 0x7fe46c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c06bf50 0x7fe48c0c5090 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe48c14bac0 0x7fe48c06ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 --2- 172.21.15.29:0/332639586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe478005180 0x7fe48c0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 >> 172.21.15.29:0/332639586 conn(0x7fe48c0cf790 msgr2=0x7fe48c0b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:27.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 shutdown_connections 2023-10-16T15:33:27.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.103+0000 7fe48b59e640 1 -- 172.21.15.29:0/332639586 wait complete. 2023-10-16T15:33:27.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:33:27.227 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:27.226+0000 7f441a830040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:33:27.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.380+0000 7f561a4e5640 1 Processor -- start 2023-10-16T15:33:27.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.380+0000 7f561a4e5640 1 -- start start 2023-10-16T15:33:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.382+0000 7f561a4e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 0x7f56140ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.382+0000 7f561a4e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140cb030 con 0x7f561415ad70 2023-10-16T15:33:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.382+0000 7f561a4e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140c91c0 con 0x7f56140c9ca0 2023-10-16T15:33:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.382+0000 7f561a4e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56140c94a0 con 0x7f561406a4b0 2023-10-16T15:33:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.382+0000 7f5613fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f561415ad70 0x7f5614069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47580/0 (socket says 172.21.15.29:47580) 2023-10-16T15:33:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.382+0000 7f5613fff640 1 -- 172.21.15.29:0/3500966429 learned_addr learned my addr 172.21.15.29:0/3500966429 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:27.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f56137fe640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 0x7f56140ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187260330 0 0) 0x7f56140cb030 con 0x7f561415ad70 2023-10-16T15:33:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc003520 con 0x7f561415ad70 2023-10-16T15:33:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588412659 0 0) 0x7f55fc003520 con 0x7f561415ad70 2023-10-16T15:33:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 msgr2=0x7f56140ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 0x7f56140ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 >> v1:172.21.15.170:6789/0 conn(0x7f56140c9ca0 legacy=0x7f56140ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140c9780 con 0x7f561415ad70 2023-10-16T15:33:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5604001670 con 0x7f561415ad70 2023-10-16T15:33:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 >> v1:172.21.15.29:6789/0 conn(0x7f561415ad70 legacy=0x7f5614069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 0x7f55fc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc003f70 0x7f55fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55fc004d00 0x7f55fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56140c94a0 con 0x7f55fc004d00 2023-10-16T15:33:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc00bac0 con 0x7f55fc003f70 2023-10-16T15:33:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc00bc80 con 0x7f561406a4b0 2023-10-16T15:33:27.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.383+0000 7f5618a82640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55fc004d00 0x7f55fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f56137fe640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc003f70 0x7f55fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f5613fff640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 0x7f55fc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f5618a82640 1 -- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 msgr2=0x7f55fc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:27.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f5618a82640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f561406a4b0 0x7f55fc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f5618a82640 1 -- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc003f70 msgr2=0x7f55fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f5618a82640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc003f70 0x7f55fc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.384+0000 7f5618a82640 1 -- 172.21.15.29:0/3500966429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc003520 con 0x7f55fc004d00 2023-10-16T15:33:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.385+0000 7f5618a82640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55fc004d00 0x7f55fc005180 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f560c001e50 tx=0x7f55fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28733d80a9fece0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.385+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c003b70 con 0x7f55fc004d00 2023-10-16T15:33:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.385+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c003cd0 con 0x7f55fc004d00 2023-10-16T15:33:27.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.385+0000 7f5612ffd640 1 -- 172.21.15.29:0/3500966429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c005310 con 0x7f55fc004d00 2023-10-16T15:33:27.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f561a4e5640 1 -- 172.21.15.29:0/3500966429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55fc004d00 msgr2=0x7f55fc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f561a4e5640 1 --2- 172.21.15.29:0/3500966429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55fc004d00 0x7f55fc005180 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f560c001e50 tx=0x7f55fc0057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f5618a82640 1 -- 172.21.15.29:0/3500966429 reap_dead start 2023-10-16T15:33:27.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f561a4e5640 1 -- 172.21.15.29:0/3500966429 shutdown_connections 2023-10-16T15:33:27.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f561a4e5640 1 -- 172.21.15.29:0/3500966429 >> 172.21.15.29:0/3500966429 conn(0x7f56140ce9b0 msgr2=0x7f56140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f561a4e5640 1 -- 172.21.15.29:0/3500966429 shutdown_connections 2023-10-16T15:33:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.386+0000 7f561a4e5640 1 -- 172.21.15.29:0/3500966429 wait complete. 2023-10-16T15:33:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.388+0000 7f561a4e5640 1 Processor -- start 2023-10-16T15:33:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.388+0000 7f561a4e5640 1 -- start start 2023-10-16T15:33:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f561a4e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc004d00 0x7f5614159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f561a4e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56140c74e0 0x7f561415d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f561a4e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 0x7f56141e7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f561a4e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561415e050 con 0x7f561415dba0 2023-10-16T15:33:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f561a4e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56141e70f0 con 0x7f55fc004d00 2023-10-16T15:33:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f561a4e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56140c7a80 con 0x7f56140c74e0 2023-10-16T15:33:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f5618a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 0x7f56141e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f5618a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 0x7f56141e7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44636/0 (socket says 172.21.15.29:44636) 2023-10-16T15:33:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.389+0000 7f5618a82640 1 -- 172.21.15.29:0/524974042 learned_addr learned my addr 172.21.15.29:0/524974042 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5613fff640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc004d00 0x7f5614159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f56137fe640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56140c74e0 0x7f561415d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5618a82640 1 -- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56140c74e0 msgr2=0x7f561415d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5618a82640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56140c74e0 0x7f561415d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5618a82640 1 -- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc004d00 msgr2=0x7f5614159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5618a82640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc004d00 0x7f5614159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5618a82640 1 -- 172.21.15.29:0/524974042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140c7d60 con 0x7f561415dba0 2023-10-16T15:33:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5618a82640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 0x7f56141e7580 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f560c005040 tx=0x7f560c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:27.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c03e070 con 0x7f561415dba0 2023-10-16T15:33:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c003e40 con 0x7f561415dba0 2023-10-16T15:33:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c02f670 con 0x7f561415dba0 2023-10-16T15:33:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.390+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56140c8040 con 0x7f561415dba0 2023-10-16T15:33:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.391+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561406fdb0 con 0x7f561415dba0 2023-10-16T15:33:27.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.392+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56141e7260 con 0x7f561415dba0 2023-10-16T15:33:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.393+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f560c02f7d0 con 0x7f561415dba0 2023-10-16T15:33:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.393+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f560c0aa840 con 0x7f561415dba0 2023-10-16T15:33:27.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.398+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f560c0758f0 con 0x7f561415dba0 2023-10-16T15:33:27.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.590+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f561404e160 con 0x7f561415dba0 2023-10-16T15:33:27.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.591+0000 7f5610ff9640 1 -- 172.21.15.29:0/524974042 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v167) v1 ==== 104+0+11 (secure 0 0 0) 0x7f560c03b0a0 con 0x7f561415dba0 2023-10-16T15:33:27.592 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:33:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.593+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 msgr2=0x7f56141e7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.593+0000 7f561a4e5640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 0x7f56141e7580 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f560c005040 tx=0x7f560c0014a0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 shutdown_connections 2023-10-16T15:33:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561415dba0 0x7f56141e7580 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f56140c74e0 0x7f561415d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 --2- 172.21.15.29:0/524974042 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55fc004d00 0x7f5614159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 >> 172.21.15.29:0/524974042 conn(0x7f56140ce9b0 msgr2=0x7f56140cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:27.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 shutdown_connections 2023-10-16T15:33:27.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.594+0000 7f561a4e5640 1 -- 172.21.15.29:0/524974042 wait complete. 2023-10-16T15:33:27.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:33:27.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.836+0000 7f2c1df43640 1 Processor -- start 2023-10-16T15:33:27.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.836+0000 7f2c1df43640 1 -- start start 2023-10-16T15:33:27.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.837+0000 7f2c1df43640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c180b8480 0x7f2c180cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.837+0000 7f2c1df43640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c180d0be0 0x7f2c180cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.837+0000 7f2c1df43640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1815d150 con 0x7f2c1815cd70 2023-10-16T15:33:27.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.837+0000 7f2c1df43640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c180d1180 con 0x7f2c180b8480 2023-10-16T15:33:27.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.837+0000 7f2c1df43640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180d1460 con 0x7f2c180d0be0 2023-10-16T15:33:27.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c180b8480 0x7f2c180cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c180b8480 0x7f2c180cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44640/0 (socket says 172.21.15.29:44640) 2023-10-16T15:33:27.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 -- 172.21.15.29:0/1956802286 learned_addr learned my addr 172.21.15.29:0/1956802286 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c1d742640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c180d0be0 0x7f2c180cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 -- 172.21.15.29:0/1956802286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c180d0be0 msgr2=0x7f2c180cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c180d0be0 0x7f2c180cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 -- 172.21.15.29:0/1956802286 >> v1:172.21.15.170:6789/0 conn(0x7f2c1815cd70 legacy=0x7f2c180d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:27.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c0ffff640 1 -- 172.21.15.29:0/1956802286 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180d1740 con 0x7f2c180b8480 2023-10-16T15:33:27.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.838+0000 7f2c1d742640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c180d0be0 0x7f2c180cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:27.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.839+0000 7f2c0ffff640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180b8480 0x7f2c180cd7c0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009870 tx=0x7f2c0802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4ca6276730a9c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c0f7fe640 1 -- 172.21.15.29:0/1956802286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08031640 con 0x7f2c180b8480 2023-10-16T15:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c0f7fe640 1 -- 172.21.15.29:0/1956802286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c080317a0 con 0x7f2c180b8480 2023-10-16T15:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c0f7fe640 1 -- 172.21.15.29:0/1956802286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08031b30 con 0x7f2c180b8480 2023-10-16T15:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 -- 172.21.15.29:0/1956802286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180b8480 msgr2=0x7f2c180cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180b8480 0x7f2c180cd7c0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009870 tx=0x7f2c0802b8f0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 -- 172.21.15.29:0/1956802286 shutdown_connections 2023-10-16T15:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c180d0be0 0x7f2c180cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 --2- 172.21.15.29:0/1956802286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180b8480 0x7f2c180cd7c0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 -- 172.21.15.29:0/1956802286 >> 172.21.15.29:0/1956802286 conn(0x7f2c180cf570 msgr2=0x7f2c180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 -- 172.21.15.29:0/1956802286 shutdown_connections 2023-10-16T15:33:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.840+0000 7f2c1df43640 1 -- 172.21.15.29:0/1956802286 wait complete. 2023-10-16T15:33:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.842+0000 7f2c1df43640 1 Processor -- start 2023-10-16T15:33:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.842+0000 7f2c1df43640 1 -- start start 2023-10-16T15:33:27.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.843+0000 7f2c1df43640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c180b8480 0x7f2c18159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.843+0000 7f2c1df43640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 0x7f2c18157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.843+0000 7f2c1df43640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1815cd70 0x7f2c1815e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.843+0000 7f2c1df43640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18156df0 con 0x7f2c180d0be0 2023-10-16T15:33:27.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.843+0000 7f2c1df43640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18157040 con 0x7f2c180b8480 2023-10-16T15:33:27.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.843+0000 7f2c1df43640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18157320 con 0x7f2c1815cd70 2023-10-16T15:33:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 0x7f2c18157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 0x7f2c18157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44644/0 (socket says 172.21.15.29:44644) 2023-10-16T15:33:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 -- 172.21.15.29:0/18260021 learned_addr learned my addr 172.21.15.29:0/18260021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 -- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1815cd70 msgr2=0x7f2c1815e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1815cd70 0x7f2c1815e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 -- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c180b8480 msgr2=0x7f2c18159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c180b8480 0x7f2c18159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 -- 172.21.15.29:0/18260021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080094d0 con 0x7f2c180d0be0 2023-10-16T15:33:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.844+0000 7f2c0ffff640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 0x7f2c18157680 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009870 tx=0x7f2c080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.845+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0803ac60 con 0x7f2c180d0be0 2023-10-16T15:33:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.845+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0803adc0 con 0x7f2c180d0be0 2023-10-16T15:33:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.845+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08044630 con 0x7f2c180d0be0 2023-10-16T15:33:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.845+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c180cb900 con 0x7f2c180d0be0 2023-10-16T15:33:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.845+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18159d60 con 0x7f2c180d0be0 2023-10-16T15:33:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.846+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c180cbb20 con 0x7f2c180d0be0 2023-10-16T15:33:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.847+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 86) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f2c0802f050 con 0x7f2c180d0be0 2023-10-16T15:33:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.849+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f2c080ab2c0 con 0x7f2c180d0be0 2023-10-16T15:33:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.853+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2c08076370 con 0x7f2c180d0be0 2023-10-16T15:33:27.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.951+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 87) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2c08075e20 con 0x7f2c180d0be0 2023-10-16T15:33:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.953+0000 7f2c0d7fa640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2c040707d0 0x7f2c04072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:27.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.962+0000 7f2c1cf41640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2c040707d0 0x7f2c04072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:27.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:27.966+0000 7f2c1cf41640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2c040707d0 0x7f2c04072bc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2c14006fd0 tx=0x7f2c14006bc0 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.058+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2c18159d60 con 0x7f2c180d0be0 2023-10-16T15:33:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.059+0000 7f2c0d7fa640 1 -- 172.21.15.29:0/18260021 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v167) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2c18159d60 con 0x7f2c180d0be0 2023-10-16T15:33:28.062 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:33:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2c040707d0 msgr2=0x7f2c04072bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2c040707d0 0x7f2c04072bc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2c14006fd0 tx=0x7f2c14006bc0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 msgr2=0x7f2c18157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 0x7f2c18157680 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009870 tx=0x7f2c080046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 shutdown_connections 2023-10-16T15:33:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2c040707d0 0x7f2c04072bc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c180d0be0 0x7f2c18157680 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2c1815cd70 0x7f2c1815e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 --2- 172.21.15.29:0/18260021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2c180b8480 0x7f2c18159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 >> 172.21.15.29:0/18260021 conn(0x7f2c180cf570 msgr2=0x7f2c180b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 shutdown_connections 2023-10-16T15:33:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.062+0000 7f2c1df43640 1 -- 172.21.15.29:0/18260021 wait complete. 2023-10-16T15:33:28.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:28.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.303+0000 7ff6ee54e640 1 Processor -- start 2023-10-16T15:33:28.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.303+0000 7ff6ee54e640 1 -- start start 2023-10-16T15:33:28.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ee54e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff6e806c040 0x7ff6e806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ee54e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e815f4a0 con 0x7ff6e80ad970 2023-10-16T15:33:28.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ee54e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e806c960 con 0x7ff6e80cda80 2023-10-16T15:33:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ee54e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e815b9d0 con 0x7ff6e806c040 2023-10-16T15:33:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ecd4b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6e80ad970 0x7ff6e815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47596/0 (socket says 172.21.15.29:47596) 2023-10-16T15:33:28.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ecd4b640 1 -- 172.21.15.29:0/2760255587 learned_addr learned my addr 172.21.15.29:0/2760255587 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:28.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.305+0000 7ff6ed54c640 1 --2- 172.21.15.29:0/2760255587 >> v2:172.21.15.170:3300/0 conn(0x7ff6e806c040 0x7ff6e806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980688247 0 0) 0x7ff6e815f4a0 con 0x7ff6e80ad970 2023-10-16T15:33:28.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4003520 con 0x7ff6e80ad970 2023-10-16T15:33:28.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752785115 0 0) 0x7ff6d4003520 con 0x7ff6e80ad970 2023-10-16T15:33:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 >> v2:172.21.15.170:3300/0 conn(0x7ff6e806c040 msgr2=0x7ff6e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 --2- 172.21.15.29:0/2760255587 >> v2:172.21.15.170:3300/0 conn(0x7ff6e806c040 0x7ff6e806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 >> v1:172.21.15.170:6789/0 conn(0x7ff6e80cda80 legacy=0x7ff6e815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e815bcb0 con 0x7ff6e80ad970 2023-10-16T15:33:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7ff6d8001670 con 0x7ff6e80ad970 2023-10-16T15:33:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 >> v1:172.21.15.29:6789/0 conn(0x7ff6e80ad970 legacy=0x7ff6e815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40039c0 0x7ff6d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d40047c0 0x7ff6d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4005180 0x7ff6d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e815b9d0 con 0x7ff6d4005180 2023-10-16T15:33:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d400c080 con 0x7ff6d40047c0 2023-10-16T15:33:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.306+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d400c240 con 0x7ff6d40039c0 2023-10-16T15:33:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4005180 0x7ff6d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6ed54c640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40039c0 0x7ff6d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6ecd4b640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d40047c0 0x7ff6d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6edd4d640 1 -- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40039c0 msgr2=0x7ff6d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d40039c0 0x7ff6d4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6edd4d640 1 -- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d40047c0 msgr2=0x7ff6d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d40047c0 0x7ff6d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.307+0000 7ff6edd4d640 1 -- 172.21.15.29:0/2760255587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4003520 con 0x7ff6d4005180 2023-10-16T15:33:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.308+0000 7ff6edd4d640 1 -- 172.21.15.29:0/2760255587 reap_dead start 2023-10-16T15:33:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.308+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4005180 0x7ff6d400b980 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0001e50 tx=0x7ff6d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437d22338aad8936 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.308+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0003b70 con 0x7ff6d4005180 2023-10-16T15:33:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.308+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0003cf0 con 0x7ff6d4005180 2023-10-16T15:33:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6dffff640 1 -- 172.21.15.29:0/2760255587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0005310 con 0x7ff6d4005180 2023-10-16T15:33:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 -- 172.21.15.29:0/2760255587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4005180 msgr2=0x7ff6d400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4005180 0x7ff6d400b980 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0001e50 tx=0x7ff6d40057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 -- 172.21.15.29:0/2760255587 shutdown_connections 2023-10-16T15:33:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/2760255587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6d4005180 0x7ff6d400b980 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 -- 172.21.15.29:0/2760255587 >> 172.21.15.29:0/2760255587 conn(0x7ff6e80ce6a0 msgr2=0x7ff6e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 -- 172.21.15.29:0/2760255587 shutdown_connections 2023-10-16T15:33:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.309+0000 7ff6ee54e640 1 -- 172.21.15.29:0/2760255587 wait complete. 2023-10-16T15:33:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.311+0000 7ff6ee54e640 1 Processor -- start 2023-10-16T15:33:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.311+0000 7ff6ee54e640 1 -- start start 2023-10-16T15:33:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.312+0000 7ff6ee54e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d4005180 0x7ff6e80c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.312+0000 7ff6ee54e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6e80af450 0x7ff6e814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.312+0000 7ff6ee54e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 0x7ff6e815d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.312+0000 7ff6ee54e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e80c9ac0 con 0x7ff6e815d220 2023-10-16T15:33:28.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.312+0000 7ff6ee54e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e815dbf0 con 0x7ff6e80af450 2023-10-16T15:33:28.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.312+0000 7ff6ee54e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e81f6c10 con 0x7ff6d4005180 2023-10-16T15:33:28.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 0x7ff6e815d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 0x7ff6e815d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44650/0 (socket says 172.21.15.29:44650) 2023-10-16T15:33:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 -- 172.21.15.29:0/3816198490 learned_addr learned my addr 172.21.15.29:0/3816198490 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6ecd4b640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6e80af450 0x7ff6e814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6ed54c640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d4005180 0x7ff6e80c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 -- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d4005180 msgr2=0x7ff6e80c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d4005180 0x7ff6e80c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 -- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6e80af450 msgr2=0x7ff6e814fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6e80af450 0x7ff6e814fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.313+0000 7ff6edd4d640 1 -- 172.21.15.29:0/3816198490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e81f6000 con 0x7ff6e815d220 2023-10-16T15:33:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.314+0000 7ff6edd4d640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 0x7ff6e815d6b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0001f40 tx=0x7ff6e00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.314+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0003b30 con 0x7ff6e815d220 2023-10-16T15:33:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.314+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0039480 con 0x7ff6e815d220 2023-10-16T15:33:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.314+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00397f0 con 0x7ff6e815d220 2023-10-16T15:33:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.314+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e81f62e0 con 0x7ff6e815d220 2023-10-16T15:33:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.314+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e81f68a0 con 0x7ff6e815d220 2023-10-16T15:33:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.315+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e806b330 con 0x7ff6e815d220 2023-10-16T15:33:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.316+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 87) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff6e0039950 con 0x7ff6e815d220 2023-10-16T15:33:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.317+0000 7ff6ddffb640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff6cc06acf0 0x7ff6cc06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.317+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7ff6e00ab680 con 0x7ff6e815d220 2023-10-16T15:33:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.318+0000 7ff6ed54c640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff6cc06acf0 0x7ff6cc06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.319+0000 7ff6ed54c640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff6cc06acf0 0x7ff6cc06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6e81539f0 tx=0x7ff6e4008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.323+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6e00765c0 con 0x7ff6e815d220 2023-10-16T15:33:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.556+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6e81f6a30 con 0x7ff6e815d220 2023-10-16T15:33:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.557+0000 7ff6ddffb640 1 -- 172.21.15.29:0/3816198490 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ff6e0003e60 con 0x7ff6e815d220 2023-10-16T15:33:28.559 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:28.559 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:37:13.609698+0000","modified":"2023-10-16T14:37:16.825159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8838},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8838":{"gid":8838,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3542022450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3542022450},{"type":"v1","addr":"172.21.15.29:6833","nonce":3542022450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T15:33:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 75 2023-10-16T15:33:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff6cc06acf0 msgr2=0x7ff6cc06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff6cc06acf0 0x7ff6cc06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6e81539f0 tx=0x7ff6e4008040 comp rx=0 tx=0).stop 2023-10-16T15:33:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 msgr2=0x7ff6e815d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 0x7ff6e815d6b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0001f40 tx=0x7ff6e00014a0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 shutdown_connections 2023-10-16T15:33:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff6cc06acf0 0x7ff6cc06d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6e815d220 0x7ff6e815d6b0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6e80af450 0x7ff6e814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 --2- 172.21.15.29:0/3816198490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6d4005180 0x7ff6e80c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 >> 172.21.15.29:0/3816198490 conn(0x7ff6e80ce6a0 msgr2=0x7ff6e8159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.560+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 shutdown_connections 2023-10-16T15:33:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.561+0000 7ff6ee54e640 1 -- 172.21.15.29:0/3816198490 wait complete. 2023-10-16T15:33:28.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:28.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.809+0000 7f697e47c640 1 Processor -- start 2023-10-16T15:33:28.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.810+0000 7f697e47c640 1 -- start start 2023-10-16T15:33:28.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.811+0000 7f697e47c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f69780cd0c0 0x7f69781615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.811+0000 7f697e47c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815db00 0x7f69780b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.811+0000 7f697e47c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697815e3e0 con 0x7f69780b0410 2023-10-16T15:33:28.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.811+0000 7f697e47c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69780b0de0 con 0x7f69780cd0c0 2023-10-16T15:33:28.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.811+0000 7f697e47c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69780cd4a0 con 0x7f697815db00 2023-10-16T15:33:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f69780cd0c0 0x7f69781615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f69780cd0c0 0x7f69781615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:42604/0 (socket says 172.21.15.29:42604) 2023-10-16T15:33:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 -- 172.21.15.29:0/1727174282 learned_addr learned my addr 172.21.15.29:0/1727174282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f697ca19640 1 --2- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815db00 0x7f69780b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 -- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815db00 msgr2=0x7f69780b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:28.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 --2- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815db00 0x7f69780b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 -- 172.21.15.29:0/1727174282 >> v1:172.21.15.170:6789/0 conn(0x7f69780b0410 legacy=0x7f697815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.812+0000 7f6977fff640 1 -- 172.21.15.29:0/1727174282 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697815e100 con 0x7f69780cd0c0 2023-10-16T15:33:28.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.814+0000 7f6977fff640 1 --2- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69780cd0c0 0x7f69781615f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6968009870 tx=0x7f696802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87c6d4a37e4b7894 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.814+0000 7f69777fe640 1 -- 172.21.15.29:0/1727174282 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968031640 con 0x7f69780cd0c0 2023-10-16T15:33:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.814+0000 7f69777fe640 1 -- 172.21.15.29:0/1727174282 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69680317a0 con 0x7f69780cd0c0 2023-10-16T15:33:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.814+0000 7f69777fe640 1 -- 172.21.15.29:0/1727174282 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968031b30 con 0x7f69780cd0c0 2023-10-16T15:33:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 -- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69780cd0c0 msgr2=0x7f69781615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 --2- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69780cd0c0 0x7f69781615f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6968009870 tx=0x7f696802b8f0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 -- 172.21.15.29:0/1727174282 shutdown_connections 2023-10-16T15:33:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 --2- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f697815db00 0x7f69780b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 --2- 172.21.15.29:0/1727174282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f69780cd0c0 0x7f69781615f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 -- 172.21.15.29:0/1727174282 >> 172.21.15.29:0/1727174282 conn(0x7f69780cb410 msgr2=0x7f69780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 -- 172.21.15.29:0/1727174282 shutdown_connections 2023-10-16T15:33:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.815+0000 7f697e47c640 1 -- 172.21.15.29:0/1727174282 wait complete. 2023-10-16T15:33:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.817+0000 7f697e47c640 1 Processor -- start 2023-10-16T15:33:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.817+0000 7f697e47c640 1 -- start start 2023-10-16T15:33:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f697e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 0x7f69780ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f697e47c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697815db00 0x7f69780c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f697e47c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69781529f0 0x7f6978152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f697e47c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69780c9810 con 0x7f69780b0410 2023-10-16T15:33:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f697e47c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69780c9af0 con 0x7f697815db00 2023-10-16T15:33:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f697e47c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69780c9dd0 con 0x7f69781529f0 2023-10-16T15:33:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f6977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 0x7f69780ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f6977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 0x7f69780ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44666/0 (socket says 172.21.15.29:44666) 2023-10-16T15:33:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f6977fff640 1 -- 172.21.15.29:0/2275384703 learned_addr learned my addr 172.21.15.29:0/2275384703 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.818+0000 7f696ffff640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697815db00 0x7f69780c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f697ca19640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69781529f0 0x7f6978152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f6977fff640 1 -- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69781529f0 msgr2=0x7f6978152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f6977fff640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69781529f0 0x7f6978152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f6977fff640 1 -- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697815db00 msgr2=0x7f69780c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:28.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f6977fff640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697815db00 0x7f69780c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:28.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f6977fff640 1 -- 172.21.15.29:0/2275384703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69680094d0 con 0x7f69780b0410 2023-10-16T15:33:28.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f6977fff640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 0x7f69780ca8b0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f6968000c00 tx=0x7f69680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.819+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696803a9b0 con 0x7f69780b0410 2023-10-16T15:33:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.820+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696803ab10 con 0x7f69780b0410 2023-10-16T15:33:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.820+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680443f0 con 0x7f69780b0410 2023-10-16T15:33:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.820+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69780ca0b0 con 0x7f69780b0410 2023-10-16T15:33:28.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.820+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69780c9500 con 0x7f69780b0410 2023-10-16T15:33:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.821+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69780ca5d0 con 0x7f69780b0410 2023-10-16T15:33:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.822+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 87) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f696802f050 con 0x7f69780b0410 2023-10-16T15:33:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.823+0000 7f69757fa640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f695806acc0 0x7f695806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.823+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f69680ab480 con 0x7f69780b0410 2023-10-16T15:33:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.823+0000 7f696ffff640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f695806acc0 0x7f695806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:28.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.825+0000 7f696ffff640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f695806acc0 0x7f695806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f69780cf550 tx=0x7f6960009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.827+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69780ca5d0 con 0x7f69780b0410 2023-10-16T15:33:28.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:28.961+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 88) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6968076810 con 0x7f69780b0410 2023-10-16T15:33:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.033+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69780ca5d0 con 0x7f69780b0410 2023-10-16T15:33:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.036+0000 7f69757fa640 1 -- 172.21.15.29:0/2275384703 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+17845 (secure 0 0 0) 0x7f69780ca5d0 con 0x7f69780b0410 2023-10-16T15:33:29.038 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:29.039 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":167,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:26.923317+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":167,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.3f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/246679773":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/951615953":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/3176243662":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/3888018037":"2023-10-17T15:33:26.923263+0000","172.21.15.170:6833/3272758872":"2023-10-17T15:33:26.923263+0000","172.21.15.170:6832/3272758872":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/91040253":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/999637714":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3650756237":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/907915415":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6833/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6832/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3571017587":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1946312558":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1765158227":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/1694874697":"2023-10-17T14:37:28.457671+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":18,"length":40},{"begin":74,"length":40}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.039+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f695806acc0 msgr2=0x7f695806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.039+0000 7f697e47c640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f695806acc0 0x7f695806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f69780cf550 tx=0x7f6960009040 comp rx=0 tx=0).stop 2023-10-16T15:33:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.039+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 msgr2=0x7f69780ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.039+0000 7f697e47c640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 0x7f69780ca8b0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f6968000c00 tx=0x7f69680046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 shutdown_connections 2023-10-16T15:33:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f695806acc0 0x7f695806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f69781529f0 0x7f6978152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f697815db00 0x7f69780c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 --2- 172.21.15.29:0/2275384703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780b0410 0x7f69780ca8b0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 >> 172.21.15.29:0/2275384703 conn(0x7f69780cb410 msgr2=0x7f697815b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 shutdown_connections 2023-10-16T15:33:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.040+0000 7f697e47c640 1 -- 172.21.15.29:0/2275384703 wait complete. 2023-10-16T15:33:29.062 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:33:29.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:29.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.276+0000 7f86e91b8640 1 Processor -- start 2023-10-16T15:33:29.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.276+0000 7f86e91b8640 1 -- start start 2023-10-16T15:33:29.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.278+0000 7f86e91b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 0x7f86e406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.278+0000 7f86e91b8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f86e40ad970 0x7f86e415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.278+0000 7f86e91b8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e406c960 con 0x7f86e415ebc0 2023-10-16T15:33:29.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.278+0000 7f86e91b8640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e415b9d0 con 0x7f86e40ad970 2023-10-16T15:33:29.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.278+0000 7f86e91b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e415bcb0 con 0x7f86e406c040 2023-10-16T15:33:29.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 0x7f86e406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 0x7f86e406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:47982/0 (socket says 172.21.15.29:47982) 2023-10-16T15:33:29.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 -- 172.21.15.29:0/4014093033 learned_addr learned my addr 172.21.15.29:0/4014093033 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:29.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e37fe640 1 --2- 172.21.15.29:0/4014093033 >> v2:172.21.15.170:3300/0 conn(0x7f86e40ad970 0x7f86e415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 -- 172.21.15.29:0/4014093033 >> v2:172.21.15.170:3300/0 conn(0x7f86e40ad970 msgr2=0x7f86e415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:29.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 --2- 172.21.15.29:0/4014093033 >> v2:172.21.15.170:3300/0 conn(0x7f86e40ad970 0x7f86e415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 -- 172.21.15.29:0/4014093033 >> v1:172.21.15.170:6789/0 conn(0x7f86e415ebc0 legacy=0x7f86e40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.279+0000 7f86e3fff640 1 -- 172.21.15.29:0/4014093033 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e415bf90 con 0x7f86e406c040 2023-10-16T15:33:29.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.281+0000 7f86e3fff640 1 --2- 172.21.15.29:0/4014093033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 0x7f86e406c420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098a0 tx=0x7f86d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da42a627b24defef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:29.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.281+0000 7f86e2ffd640 1 -- 172.21.15.29:0/4014093033 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4030840 con 0x7f86e406c040 2023-10-16T15:33:29.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.281+0000 7f86e2ffd640 1 -- 172.21.15.29:0/4014093033 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d40309a0 con 0x7f86e406c040 2023-10-16T15:33:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.281+0000 7f86e2ffd640 1 -- 172.21.15.29:0/4014093033 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4030d30 con 0x7f86e406c040 2023-10-16T15:33:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 -- 172.21.15.29:0/4014093033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 msgr2=0x7f86e406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 --2- 172.21.15.29:0/4014093033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 0x7f86e406c420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098a0 tx=0x7f86d402b980 comp rx=0 tx=0).stop 2023-10-16T15:33:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 -- 172.21.15.29:0/4014093033 shutdown_connections 2023-10-16T15:33:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 --2- 172.21.15.29:0/4014093033 >> v2:172.21.15.170:3300/0 conn(0x7f86e40ad970 0x7f86e415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 --2- 172.21.15.29:0/4014093033 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e406c040 0x7f86e406c420 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 -- 172.21.15.29:0/4014093033 >> 172.21.15.29:0/4014093033 conn(0x7f86e40ce6a0 msgr2=0x7f86e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 -- 172.21.15.29:0/4014093033 shutdown_connections 2023-10-16T15:33:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.282+0000 7f86e91b8640 1 -- 172.21.15.29:0/4014093033 wait complete. 2023-10-16T15:33:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.284+0000 7f86e91b8640 1 Processor -- start 2023-10-16T15:33:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.284+0000 7f86e91b8640 1 -- start start 2023-10-16T15:33:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.285+0000 7f86e91b8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406c040 0x7f86e4153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.285+0000 7f86e91b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 0x7f86e41e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.285+0000 7f86e91b8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e415ebc0 0x7f86e4154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.285+0000 7f86e91b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e41e66f0 con 0x7f86e40ad970 2023-10-16T15:33:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.285+0000 7f86e91b8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e41540d0 con 0x7f86e406c040 2023-10-16T15:33:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.285+0000 7f86e91b8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40c5770 con 0x7f86e415ebc0 2023-10-16T15:33:29.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 0x7f86e41e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 0x7f86e41e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44680/0 (socket says 172.21.15.29:44680) 2023-10-16T15:33:29.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 -- 172.21.15.29:0/1022117535 learned_addr learned my addr 172.21.15.29:0/1022117535 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:29.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e3fff640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406c040 0x7f86e4153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e89b7640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e415ebc0 0x7f86e4154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 -- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e415ebc0 msgr2=0x7f86e4154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e415ebc0 0x7f86e4154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 -- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406c040 msgr2=0x7f86e4153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406c040 0x7f86e4153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.286+0000 7f86e37fe640 1 -- 172.21.15.29:0/1022117535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40094d0 con 0x7f86e40ad970 2023-10-16T15:33:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.287+0000 7f86e37fe640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 0x7f86e41e4e90 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f86d800d5f0 tx=0x7f86d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.287+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d80125f0 con 0x7f86e40ad970 2023-10-16T15:33:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.287+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d8012750 con 0x7f86e40ad970 2023-10-16T15:33:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.287+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e40c5a30 con 0x7f86e40ad970 2023-10-16T15:33:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.287+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8012b00 con 0x7f86e40ad970 2023-10-16T15:33:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.287+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e40c5ff0 con 0x7f86e40ad970 2023-10-16T15:33:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.288+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e406b330 con 0x7f86e40ad970 2023-10-16T15:33:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.289+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f86d8012c60 con 0x7f86e40ad970 2023-10-16T15:33:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.291+0000 7f86e0ff9640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86d006af00 0x7f86d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.291+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f86d8089f60 con 0x7f86e40ad970 2023-10-16T15:33:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.292+0000 7f86e3fff640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86d006af00 0x7f86d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.293+0000 7f86e3fff640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86d006af00 0x7f86d006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f86d4036000 tx=0x7f86d403b040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.296+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86d8054bc0 con 0x7f86e40ad970 2023-10-16T15:33:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.537+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86e404e9b0 con 0x7f86e40ad970 2023-10-16T15:33:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.538+0000 7f86e0ff9640 1 -- 172.21.15.29:0/1022117535 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f86d80581e0 con 0x7f86e40ad970 2023-10-16T15:33:29.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:29.540 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:37:13.609698+0000","modified":"2023-10-16T14:37:16.825159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8838},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8838":{"gid":8838,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3542022450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3542022450},{"type":"v1","addr":"172.21.15.29:6833","nonce":3542022450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T15:33:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 75 2023-10-16T15:33:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86d006af00 msgr2=0x7f86d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86d006af00 0x7f86d006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f86d4036000 tx=0x7f86d403b040 comp rx=0 tx=0).stop 2023-10-16T15:33:29.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 msgr2=0x7f86e41e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 0x7f86e41e4e90 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f86d800d5f0 tx=0x7f86d800dab0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 shutdown_connections 2023-10-16T15:33:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86d006af00 0x7f86d006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86e40ad970 0x7f86e41e4e90 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86e415ebc0 0x7f86e4154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 --2- 172.21.15.29:0/1022117535 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86e406c040 0x7f86e4153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 >> 172.21.15.29:0/1022117535 conn(0x7f86e40ce6a0 msgr2=0x7f86e415ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:29.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 shutdown_connections 2023-10-16T15:33:29.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.541+0000 7f86e91b8640 1 -- 172.21.15.29:0/1022117535 wait complete. 2023-10-16T15:33:29.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.788+0000 7f2a05743640 1 Processor -- start 2023-10-16T15:33:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.788+0000 7f2a05743640 1 -- start start 2023-10-16T15:33:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.790+0000 7f2a05743640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 0x7f2a0015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.790+0000 7f2a05743640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2a0006c100 0x7f2a0006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.790+0000 7f2a05743640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0015aef0 con 0x7f2a00159950 2023-10-16T15:33:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.790+0000 7f2a05743640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0015e490 con 0x7f2a0006c100 2023-10-16T15:33:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.790+0000 7f2a05743640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0015d910 con 0x7f2a00070b60 2023-10-16T15:33:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.790+0000 7f29fffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a00159950 0x7f2a00070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47610/0 (socket says 172.21.15.29:47610) 2023-10-16T15:33:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f29fffff640 1 -- 172.21.15.29:0/3301001179 learned_addr learned my addr 172.21.15.29:0/3301001179 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 0x7f2a0015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f2a04f42640 1 --2- 172.21.15.29:0/3301001179 >> v2:172.21.15.170:3300/0 conn(0x7f2a0006c100 0x7f2a0006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035872080 0 0) 0x7f29f0001110 con 0x7f2a00159950 2023-10-16T15:33:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003520 con 0x7f2a00159950 2023-10-16T15:33:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804050977 0 0) 0x7f29e8003520 con 0x7f2a00159950 2023-10-16T15:33:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.791+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 msgr2=0x7f2a0015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 0x7f2a0015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 >> v2:172.21.15.170:3300/0 conn(0x7f2a0006c100 msgr2=0x7f2a0006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 --2- 172.21.15.29:0/3301001179 >> v2:172.21.15.170:3300/0 conn(0x7f2a0006c100 0x7f2a0006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0015dbf0 con 0x7f2a00159950 2023-10-16T15:33:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f29f0001800 con 0x7f2a00159950 2023-10-16T15:33:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 >> v1:172.21.15.29:6789/0 conn(0x7f2a00159950 legacy=0x7f2a00070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 0x7f29e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8003f70 0x7f29e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8004d00 0x7f29e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0015e490 con 0x7f29e8003f70 2023-10-16T15:33:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0015d910 con 0x7f29e8004d00 2023-10-16T15:33:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800ba60 con 0x7f2a00070b60 2023-10-16T15:33:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8003f70 0x7f29e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29ff7fe640 1 -- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 msgr2=0x7f29e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a00070b60 0x7f29e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29ff7fe640 1 -- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8004d00 msgr2=0x7f29e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29e8004d00 0x7f29e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.792+0000 7f29ff7fe640 1 -- 172.21.15.29:0/3301001179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8003520 con 0x7f29e8003f70 2023-10-16T15:33:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8003f70 0x7f29e80047c0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009ac0 tx=0x7f29e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0e182d48740c1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f402c6f0 con 0x7f29e8003f70 2023-10-16T15:33:29.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f402c850 con 0x7f29e8003f70 2023-10-16T15:33:29.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f29feffd640 1 -- 172.21.15.29:0/3301001179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f402cbe0 con 0x7f29e8003f70 2023-10-16T15:33:29.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f2a05743640 1 -- 172.21.15.29:0/3301001179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8003f70 msgr2=0x7f29e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f2a05743640 1 --2- 172.21.15.29:0/3301001179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29e8003f70 0x7f29e80047c0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009ac0 tx=0x7f29e80057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.794+0000 7f2a04f42640 1 -- 172.21.15.29:0/3301001179 reap_dead start 2023-10-16T15:33:29.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.795+0000 7f2a05743640 1 -- 172.21.15.29:0/3301001179 shutdown_connections 2023-10-16T15:33:29.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.795+0000 7f2a05743640 1 -- 172.21.15.29:0/3301001179 >> 172.21.15.29:0/3301001179 conn(0x7f2a000ce6a0 msgr2=0x7f2a000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:29.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.795+0000 7f2a05743640 1 -- 172.21.15.29:0/3301001179 shutdown_connections 2023-10-16T15:33:29.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.795+0000 7f2a05743640 1 -- 172.21.15.29:0/3301001179 wait complete. 2023-10-16T15:33:29.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.797+0000 7f2a05743640 1 Processor -- start 2023-10-16T15:33:29.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.797+0000 7f2a05743640 1 -- start start 2023-10-16T15:33:29.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a05743640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e8003f70 0x7f2a00154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a05743640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 0x7f2a0014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a05743640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a0006ae30 0x7f2a0006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a05743640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0006b7f0 con 0x7f2a00154010 2023-10-16T15:33:29.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a05743640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00151360 con 0x7f2a0006ae30 2023-10-16T15:33:29.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a05743640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000c81e0 con 0x7f29e8003f70 2023-10-16T15:33:29.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 0x7f2a0014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 0x7f2a0014d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44702/0 (socket says 172.21.15.29:44702) 2023-10-16T15:33:29.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 -- 172.21.15.29:0/290430620 learned_addr learned my addr 172.21.15.29:0/290430620 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:29.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29fffff640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e8003f70 0x7f2a00154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f2a04f42640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a0006ae30 0x7f2a0006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 -- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e8003f70 msgr2=0x7f2a00154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e8003f70 0x7f2a00154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 -- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a0006ae30 msgr2=0x7f2a0006b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:29.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a0006ae30 0x7f2a0006b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:29.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.798+0000 7f29ff7fe640 1 -- 172.21.15.29:0/290430620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a000b0430 con 0x7f2a00154010 2023-10-16T15:33:29.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.799+0000 7f29ff7fe640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 0x7f2a0014d800 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f29f402c290 tx=0x7f29f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:29.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.799+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4003810 con 0x7f2a00154010 2023-10-16T15:33:29.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.799+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f4003970 con 0x7f2a00154010 2023-10-16T15:33:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.799+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a000b0710 con 0x7f2a00154010 2023-10-16T15:33:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.799+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4003ce0 con 0x7f2a00154010 2023-10-16T15:33:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.799+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a000c84c0 con 0x7f2a00154010 2023-10-16T15:33:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.800+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a000b0290 con 0x7f2a00154010 2023-10-16T15:33:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.801+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f29f4034440 con 0x7f2a00154010 2023-10-16T15:33:29.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.802+0000 7f29fcff9640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f29ec06af00 0x7f29ec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:29.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.802+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f29f40aba40 con 0x7f2a00154010 2023-10-16T15:33:29.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.803+0000 7f29fffff640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f29ec06af00 0x7f29ec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:29.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.804+0000 7f29fffff640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f29ec06af00 0x7f29ec06d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f29f00012b0 tx=0x7f29f0001c00 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:29.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:29.807+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2a000b0290 con 0x7f2a00154010 2023-10-16T15:33:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.040+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a000b0290 con 0x7f2a00154010 2023-10-16T15:33:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.041+0000 7f29fcff9640 1 -- 172.21.15.29:0/290430620 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f2a000b0290 con 0x7f2a00154010 2023-10-16T15:33:30.043 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:30.043 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:37:13.609698+0000","modified":"2023-10-16T14:37:16.825159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8838},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8838":{"gid":8838,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3542022450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3542022450},{"type":"v1","addr":"172.21.15.29:6833","nonce":3542022450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T15:33:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 75 2023-10-16T15:33:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f29ec06af00 msgr2=0x7f29ec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f29ec06af00 0x7f29ec06d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f29f00012b0 tx=0x7f29f0001c00 comp rx=0 tx=0).stop 2023-10-16T15:33:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 msgr2=0x7f2a0014d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 0x7f2a0014d800 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f29f402c290 tx=0x7f29f4004730 comp rx=0 tx=0).stop 2023-10-16T15:33:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 shutdown_connections 2023-10-16T15:33:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f29ec06af00 0x7f29ec06d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a00154010 0x7f2a0014d800 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a0006ae30 0x7f2a0006b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 --2- 172.21.15.29:0/290430620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f29e8003f70 0x7f2a00154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 >> 172.21.15.29:0/290430620 conn(0x7f2a000ce6a0 msgr2=0x7f2a000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 shutdown_connections 2023-10-16T15:33:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.044+0000 7f2a05743640 1 -- 172.21.15.29:0/290430620 wait complete. 2023-10-16T15:33:30.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.282+0000 7f1b22fc4640 1 Processor -- start 2023-10-16T15:33:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.282+0000 7f1b22fc4640 1 -- start start 2023-10-16T15:33:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.283+0000 7f1b22fc4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b1c0ad970 0x7f1b1c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.283+0000 7f1b22fc4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1b1c0ccfb0 0x7f1b1c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.283+0000 7f1b22fc4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1c15b9d0 con 0x7f1b1c06c040 2023-10-16T15:33:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.283+0000 7f1b22fc4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1c15bcb0 con 0x7f1b1c0ad970 2023-10-16T15:33:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.283+0000 7f1b22fc4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1c15bf90 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b1c0ad970 0x7f1b1c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b1c0ad970 0x7f1b1c0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44708/0 (socket says 172.21.15.29:44708) 2023-10-16T15:33:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 -- 172.21.15.29:0/3704672386 learned_addr learned my addr 172.21.15.29:0/3704672386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 -- 172.21.15.29:0/3704672386 >> v2:172.21.15.170:3300/0 conn(0x7f1b1c0ccfb0 msgr2=0x7f1b1c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 --2- 172.21.15.29:0/3704672386 >> v2:172.21.15.170:3300/0 conn(0x7f1b1c0ccfb0 0x7f1b1c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 -- 172.21.15.29:0/3704672386 >> v1:172.21.15.29:6789/0 conn(0x7f1b1c06c040 legacy=0x7f1b1c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.284+0000 7f1b217c1640 1 -- 172.21.15.29:0/3704672386 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c0cc270 con 0x7f1b1c0ad970 2023-10-16T15:33:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.285+0000 7f1b217c1640 1 --2- 172.21.15.29:0/3704672386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ad970 0x7f1b1c0cca70 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009870 tx=0x7f1b0c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c2095822df0e47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b20fc0640 1 -- 172.21.15.29:0/3704672386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c030850 con 0x7f1b1c0ad970 2023-10-16T15:33:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b20fc0640 1 -- 172.21.15.29:0/3704672386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0c0309b0 con 0x7f1b1c0ad970 2023-10-16T15:33:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b20fc0640 1 -- 172.21.15.29:0/3704672386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c030d40 con 0x7f1b1c0ad970 2023-10-16T15:33:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 -- 172.21.15.29:0/3704672386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ad970 msgr2=0x7f1b1c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/3704672386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ad970 0x7f1b1c0cca70 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009870 tx=0x7f1b0c02b990 comp rx=0 tx=0).stop 2023-10-16T15:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 -- 172.21.15.29:0/3704672386 shutdown_connections 2023-10-16T15:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/3704672386 >> v2:172.21.15.170:3300/0 conn(0x7f1b1c0ccfb0 0x7f1b1c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/3704672386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ad970 0x7f1b1c0cca70 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 -- 172.21.15.29:0/3704672386 >> 172.21.15.29:0/3704672386 conn(0x7f1b1c0ce6a0 msgr2=0x7f1b1c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 -- 172.21.15.29:0/3704672386 shutdown_connections 2023-10-16T15:33:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.286+0000 7f1b22fc4640 1 -- 172.21.15.29:0/3704672386 wait complete. 2023-10-16T15:33:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.288+0000 7f1b22fc4640 1 Processor -- start 2023-10-16T15:33:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.288+0000 7f1b22fc4640 1 -- start start 2023-10-16T15:33:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b22fc4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b1c06c040 0x7f1b1c0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b22fc4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b1c0ad970 0x7f1b1c153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b22fc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 0x7f1b1c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b22fc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c0c96c0 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b22fc4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c154140 con 0x7f1b1c06c040 2023-10-16T15:33:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b22fc4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c0b42a0 con 0x7f1b1c0ad970 2023-10-16T15:33:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b227c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 0x7f1b1c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b227c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 0x7f1b1c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44724/0 (socket says 172.21.15.29:44724) 2023-10-16T15:33:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.289+0000 7f1b227c3640 1 -- 172.21.15.29:0/1633993147 learned_addr learned my addr 172.21.15.29:0/1633993147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b21fc2640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b1c06c040 0x7f1b1c0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b217c1640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b1c0ad970 0x7f1b1c153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b227c3640 1 -- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b1c0ad970 msgr2=0x7f1b1c153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b227c3640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b1c0ad970 0x7f1b1c153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b227c3640 1 -- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b1c06c040 msgr2=0x7f1b1c0ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b227c3640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b1c06c040 0x7f1b1c0ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b227c3640 1 -- 172.21.15.29:0/1633993147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0094d0 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b21fc2640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b1c06c040 0x7f1b1c0ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b227c3640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 0x7f1b1c0ac890 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400b9a0 tx=0x7f1b1400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400cba0 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1400cd20 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14014680 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.290+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1c0c8d90 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.291+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1c0c9350 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.292+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1c06b330 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.293+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1b1401d3b0 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.294+0000 7f1b127fc640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1b0006af00 0x7f1b0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.294+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f1b14089930 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.295+0000 7f1b21fc2640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1b0006af00 0x7f1b0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.296+0000 7f1b21fc2640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1b0006af00 0x7f1b0006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1b18000f60 tx=0x7f1b18009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:30.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.298+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1b14054690 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.492+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b1c0c9870 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.495+0000 7f1b127fc640 1 -- 172.21.15.29:0/1633993147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+17845 (secure 0 0 0) 0x7f1b1c0c9870 con 0x7f1b1c0ccfb0 2023-10-16T15:33:30.497 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:30.497 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":167,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:26.923317+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:37:11.546312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T14:37:12.609923+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":167,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.12","mappings":[{"from":2,"to":0}]},{"pgid":"19.3f","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/246679773":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/951615953":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/3176243662":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/3888018037":"2023-10-17T15:33:26.923263+0000","172.21.15.170:6833/3272758872":"2023-10-17T15:33:26.923263+0000","172.21.15.170:6832/3272758872":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/91040253":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/999637714":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3650756237":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/907915415":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6833/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6832/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3571017587":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1946312558":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1765158227":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/1694874697":"2023-10-17T14:37:28.457671+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":18,"length":40},{"begin":74,"length":40}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.497+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1b0006af00 msgr2=0x7f1b0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.497+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1b0006af00 0x7f1b0006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1b18000f60 tx=0x7f1b18009040 comp rx=0 tx=0).stop 2023-10-16T15:33:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.497+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 msgr2=0x7f1b1c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.497+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 0x7f1b1c0ac890 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400b9a0 tx=0x7f1b1400be60 comp rx=0 tx=0).stop 2023-10-16T15:33:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 shutdown_connections 2023-10-16T15:33:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1b0006af00 0x7f1b0006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b1c0ccfb0 0x7f1b1c0ac890 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1b1c0ad970 0x7f1b1c153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 --2- 172.21.15.29:0/1633993147 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b1c06c040 0x7f1b1c0ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 >> 172.21.15.29:0/1633993147 conn(0x7f1b1c0ce6a0 msgr2=0x7f1b1c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:30.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 shutdown_connections 2023-10-16T15:33:30.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.498+0000 7f1b22fc4640 1 -- 172.21.15.29:0/1633993147 wait complete. 2023-10-16T15:33:30.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:33:30.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.741+0000 7f258b0e5640 1 Processor -- start 2023-10-16T15:33:30.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.741+0000 7f258b0e5640 1 -- start start 2023-10-16T15:33:30.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.742+0000 7f258b0e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f258415ad70 0x7f2584069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.742+0000 7f258b0e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25840c9ca0 0x7f258406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.742+0000 7f258b0e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25840cb030 con 0x7f25840ca3b0 2023-10-16T15:33:30.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.742+0000 7f258b0e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840c91c0 con 0x7f25840c9ca0 2023-10-16T15:33:30.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.742+0000 7f258b0e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840c94a0 con 0x7f258415ad70 2023-10-16T15:33:30.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2583fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25840c9ca0 0x7f258406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2589682640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25840ca3b0 0x7f25840ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47632/0 (socket says 172.21.15.29:47632) 2023-10-16T15:33:30.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2589682640 1 -- 172.21.15.29:0/2828544461 learned_addr learned my addr 172.21.15.29:0/2828544461 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:30.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2588e81640 1 --2- 172.21.15.29:0/2828544461 >> v2:172.21.15.170:3300/0 conn(0x7f258415ad70 0x7f2584069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2583fff640 1 -- 172.21.15.29:0/2828544461 >> v2:172.21.15.170:3300/0 conn(0x7f258415ad70 msgr2=0x7f2584069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:30.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2583fff640 1 --2- 172.21.15.29:0/2828544461 >> v2:172.21.15.170:3300/0 conn(0x7f258415ad70 0x7f2584069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2583fff640 1 -- 172.21.15.29:0/2828544461 >> v1:172.21.15.29:6789/0 conn(0x7f25840ca3b0 legacy=0x7f25840ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f25837fe640 1 -- 172.21.15.29:0/2828544461 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778904340 0 0) 0x7f25840cb030 con 0x7f25840ca3b0 2023-10-16T15:33:30.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.743+0000 7f2583fff640 1 -- 172.21.15.29:0/2828544461 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840c9780 con 0x7f25840c9ca0 2023-10-16T15:33:30.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.745+0000 7f2583fff640 1 --2- 172.21.15.29:0/2828544461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25840c9ca0 0x7f258406a2e0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f25840cf780 tx=0x7f257002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2912622c478d9225 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:30.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.745+0000 7f25837fe640 1 -- 172.21.15.29:0/2828544461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570031830 con 0x7f25840c9ca0 2023-10-16T15:33:30.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.745+0000 7f25837fe640 1 -- 172.21.15.29:0/2828544461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2570031990 con 0x7f25840c9ca0 2023-10-16T15:33:30.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.745+0000 7f25837fe640 1 -- 172.21.15.29:0/2828544461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570031d20 con 0x7f25840c9ca0 2023-10-16T15:33:30.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 -- 172.21.15.29:0/2828544461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25840c9ca0 msgr2=0x7f258406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2828544461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25840c9ca0 0x7f258406a2e0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f25840cf780 tx=0x7f257002b8d0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 -- 172.21.15.29:0/2828544461 shutdown_connections 2023-10-16T15:33:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2828544461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25840c9ca0 0x7f258406a2e0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2828544461 >> v2:172.21.15.170:3300/0 conn(0x7f258415ad70 0x7f2584069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 -- 172.21.15.29:0/2828544461 >> 172.21.15.29:0/2828544461 conn(0x7f25840ce9b0 msgr2=0x7f25840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 -- 172.21.15.29:0/2828544461 shutdown_connections 2023-10-16T15:33:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.746+0000 7f258b0e5640 1 -- 172.21.15.29:0/2828544461 wait complete. 2023-10-16T15:33:30.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.748+0000 7f258b0e5640 1 Processor -- start 2023-10-16T15:33:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.748+0000 7f258b0e5640 1 -- start start 2023-10-16T15:33:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f258b0e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25840c9ca0 0x7f25841575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f258b0e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25840ca3b0 0x7f258415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f258b0e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 0x7f25841e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f258b0e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584158270 con 0x7f258415ad70 2023-10-16T15:33:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f258b0e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258415b450 con 0x7f25840c9ca0 2023-10-16T15:33:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f258b0e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258415b730 con 0x7f25840ca3b0 2023-10-16T15:33:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f2589682640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 0x7f25841e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f2589682640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 0x7f25841e4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44740/0 (socket says 172.21.15.29:44740) 2023-10-16T15:33:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f2589682640 1 -- 172.21.15.29:0/2712807996 learned_addr learned my addr 172.21.15.29:0/2712807996 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.749+0000 7f2588e81640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25840c9ca0 0x7f25841575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2583fff640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25840ca3b0 0x7f258415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2589682640 1 -- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25840ca3b0 msgr2=0x7f258415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2589682640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25840ca3b0 0x7f258415ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2589682640 1 -- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25840c9ca0 msgr2=0x7f25841575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2589682640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25840c9ca0 0x7f25841575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2589682640 1 -- 172.21.15.29:0/2712807996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570009400 con 0x7f258415ad70 2023-10-16T15:33:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f2589682640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 0x7f25841e4c10 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f257c00c7e0 tx=0x7f257c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c00a640 con 0x7f258415ad70 2023-10-16T15:33:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c00a7c0 con 0x7f258415ad70 2023-10-16T15:33:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c00ab70 con 0x7f258415ad70 2023-10-16T15:33:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258415ba10 con 0x7f258415ad70 2023-10-16T15:33:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.750+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258415bfd0 con 0x7f258415ad70 2023-10-16T15:33:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.752+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25840c5a80 con 0x7f258415ad70 2023-10-16T15:33:30.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.752+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f257c00acd0 con 0x7f258415ad70 2023-10-16T15:33:30.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.753+0000 7f25817fa640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f256806af00 0x7f256806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:30.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.754+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f257c088fb0 con 0x7f258415ad70 2023-10-16T15:33:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.754+0000 7f2588e81640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f256806af00 0x7f256806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.755+0000 7f2588e81640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f256806af00 0x7f256806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2578006fd0 tx=0x7f2578008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.758+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f257c053d10 con 0x7f258415ad70 2023-10-16T15:33:30.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.982+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f257c0540e0 con 0x7f258415ad70 2023-10-16T15:33:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:30.996+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f258404e160 con 0x7f258415ad70 2023-10-16T15:33:30.998 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:33:30.997+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:33:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.000+0000 7f25817fa640 1 -- 172.21.15.29:0/2712807996 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v76) v1 ==== 161+0+0 (secure 0 0 0) 0x7f257c072790 con 0x7f258415ad70 2023-10-16T15:33:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:33:31.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f256806af00 msgr2=0x7f256806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:31.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f256806af00 0x7f256806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2578006fd0 tx=0x7f2578008040 comp rx=0 tx=0).stop 2023-10-16T15:33:31.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 msgr2=0x7f25841e4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:31.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 0x7f25841e4c10 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f257c00c7e0 tx=0x7f257c00cca0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 shutdown_connections 2023-10-16T15:33:31.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f256806af00 0x7f256806d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258415ad70 0x7f25841e4c10 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f25840ca3b0 0x7f258415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 --2- 172.21.15.29:0/2712807996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25840c9ca0 0x7f25841575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.003+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 >> 172.21.15.29:0/2712807996 conn(0x7f25840ce9b0 msgr2=0x7f25840b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:31.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.004+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 shutdown_connections 2023-10-16T15:33:31.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.004+0000 7f258b0e5640 1 -- 172.21.15.29:0/2712807996 wait complete. 2023-10-16T15:33:31.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:33:31.091 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:33:31.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.221+0000 7f2e9df47640 1 Processor -- start 2023-10-16T15:33:31.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.222+0000 7f2e9df47640 1 -- start start 2023-10-16T15:33:31.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.223+0000 7f2e9df47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e980ad970 0x7f2e980cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:31.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.223+0000 7f2e9df47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2e980ccfb0 0x7f2e980cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:31.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.223+0000 7f2e9df47640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9815b9d0 con 0x7f2e9806c040 2023-10-16T15:33:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.223+0000 7f2e9df47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9815bcb0 con 0x7f2e980ad970 2023-10-16T15:33:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.223+0000 7f2e9df47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9815bf90 con 0x7f2e980ccfb0 2023-10-16T15:33:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.223+0000 7f2e97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e980ad970 0x7f2e980cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e980ad970 0x7f2e980cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44750/0 (socket says 172.21.15.29:44750) 2023-10-16T15:33:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e97fff640 1 -- 172.21.15.29:0/2012533835 learned_addr learned my addr 172.21.15.29:0/2012533835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e9d746640 1 --2- 172.21.15.29:0/2012533835 >> v2:172.21.15.170:3300/0 conn(0x7f2e980ccfb0 0x7f2e980cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e97fff640 1 -- 172.21.15.29:0/2012533835 >> v2:172.21.15.170:3300/0 conn(0x7f2e980ccfb0 msgr2=0x7f2e980cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e97fff640 1 --2- 172.21.15.29:0/2012533835 >> v2:172.21.15.170:3300/0 conn(0x7f2e980ccfb0 0x7f2e980cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e97fff640 1 -- 172.21.15.29:0/2012533835 >> v1:172.21.15.170:6789/0 conn(0x7f2e9806c040 legacy=0x7f2e9806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:31.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.224+0000 7f2e97fff640 1 -- 172.21.15.29:0/2012533835 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980cc270 con 0x7f2e980ad970 2023-10-16T15:33:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.225+0000 7f2e97fff640 1 --2- 172.21.15.29:0/2012533835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e980ad970 0x7f2e980cca70 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009870 tx=0x7f2e8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f1a95179804c2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.225+0000 7f2e977fe640 1 -- 172.21.15.29:0/2012533835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80031870 con 0x7f2e980ad970 2023-10-16T15:33:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e977fe640 1 -- 172.21.15.29:0/2012533835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e800319d0 con 0x7f2e980ad970 2023-10-16T15:33:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e977fe640 1 -- 172.21.15.29:0/2012533835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80031d60 con 0x7f2e980ad970 2023-10-16T15:33:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 -- 172.21.15.29:0/2012533835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e980ad970 msgr2=0x7f2e980cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 --2- 172.21.15.29:0/2012533835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e980ad970 0x7f2e980cca70 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009870 tx=0x7f2e8002b910 comp rx=0 tx=0).stop 2023-10-16T15:33:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 -- 172.21.15.29:0/2012533835 shutdown_connections 2023-10-16T15:33:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 --2- 172.21.15.29:0/2012533835 >> v2:172.21.15.170:3300/0 conn(0x7f2e980ccfb0 0x7f2e980cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 --2- 172.21.15.29:0/2012533835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e980ad970 0x7f2e980cca70 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 -- 172.21.15.29:0/2012533835 >> 172.21.15.29:0/2012533835 conn(0x7f2e980ce6a0 msgr2=0x7f2e980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 -- 172.21.15.29:0/2012533835 shutdown_connections 2023-10-16T15:33:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.226+0000 7f2e9df47640 1 -- 172.21.15.29:0/2012533835 wait complete. 2023-10-16T15:33:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.228+0000 7f2e9df47640 1 Processor -- start 2023-10-16T15:33:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.228+0000 7f2e9df47640 1 -- start start 2023-10-16T15:33:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9df47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 0x7f2e98153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9df47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980ad970 0x7f2e980b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9df47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980ccfb0 0x7f2e980c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9df47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e981f6bc0 con 0x7f2e9806c040 2023-10-16T15:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9df47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980ca5d0 con 0x7f2e980ad970 2023-10-16T15:33:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9df47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980af770 con 0x7f2e980ccfb0 2023-10-16T15:33:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9cf45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 0x7f2e98153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9cf45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 0x7f2e98153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44758/0 (socket says 172.21.15.29:44758) 2023-10-16T15:33:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.229+0000 7f2e9cf45640 1 -- 172.21.15.29:0/3294977762 learned_addr learned my addr 172.21.15.29:0/3294977762 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e9cf45640 1 -- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980ccfb0 msgr2=0x7f2e980c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e9cf45640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980ccfb0 0x7f2e980c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e9cf45640 1 -- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980ad970 msgr2=0x7f2e980b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e9cf45640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980ad970 0x7f2e980b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e9cf45640 1 -- 172.21.15.29:0/3294977762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e800094d0 con 0x7f2e9806c040 2023-10-16T15:33:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e9cf45640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 0x7f2e98153b90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c7c0 tx=0x7f2e8800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800aa50 con 0x7f2e9806c040 2023-10-16T15:33:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.230+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8800abb0 con 0x7f2e9806c040 2023-10-16T15:33:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.231+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88012420 con 0x7f2e9806c040 2023-10-16T15:33:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.231+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e980afa50 con 0x7f2e9806c040 2023-10-16T15:33:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.231+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e980b0010 con 0x7f2e9806c040 2023-10-16T15:33:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.232+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9806b330 con 0x7f2e9806c040 2023-10-16T15:33:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.233+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2e88012580 con 0x7f2e9806c040 2023-10-16T15:33:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.234+0000 7f2e957fa640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2e7806af90 0x7f2e7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.234+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6613+0+0 (secure 0 0 0) 0x7f2e88089b20 con 0x7f2e9806c040 2023-10-16T15:33:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.235+0000 7f2e97fff640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2e7806af90 0x7f2e7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.236+0000 7f2e97fff640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2e7806af90 0x7f2e7806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2e80036000 tx=0x7f2e80030000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.238+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2e88054730 con 0x7f2e9806c040 2023-10-16T15:33:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:31.474+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2e9804e9b0 con 0x7f2e9806c040 2023-10-16T15:33:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.003+0000 7f2e957fa640 1 -- 172.21.15.29:0/3294977762 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v77) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2e88057d50 con 0x7f2e9806c040 2023-10-16T15:33:32.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.007+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2e7806af90 msgr2=0x7f2e7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:32.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.007+0000 7f2e9df47640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2e7806af90 0x7f2e7806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2e80036000 tx=0x7f2e80030000 comp rx=0 tx=0).stop 2023-10-16T15:33:32.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.007+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 msgr2=0x7f2e98153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:32.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.007+0000 7f2e9df47640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 0x7f2e98153b90 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c7c0 tx=0x7f2e8800cc80 comp rx=0 tx=0).stop 2023-10-16T15:33:32.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 shutdown_connections 2023-10-16T15:33:32.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2e7806af90 0x7f2e7806d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2e980ccfb0 0x7f2e980c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2e980ad970 0x7f2e980b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 --2- 172.21.15.29:0/3294977762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e9806c040 0x7f2e98153b90 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 >> 172.21.15.29:0/3294977762 conn(0x7f2e980ce6a0 msgr2=0x7f2e980c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 shutdown_connections 2023-10-16T15:33:32.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.008+0000 7f2e9df47640 1 -- 172.21.15.29:0/3294977762 wait complete. 2023-10-16T15:33:32.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:33:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.250+0000 7f1964802640 1 Processor -- start 2023-10-16T15:33:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.251+0000 7f1964802640 1 -- start start 2023-10-16T15:33:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f1964802640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f1964802640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19600d0be0 0x7f19600cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f1964802640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196015d150 con 0x7f196015cd70 2023-10-16T15:33:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f1964802640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19600d1180 con 0x7f19600d0be0 2023-10-16T15:33:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f1964802640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19600d1460 con 0x7f19600b8480 2023-10-16T15:33:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f195f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f196015cd70 0x7f19600d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47638/0 (socket says 172.21.15.29:47638) 2023-10-16T15:33:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f195f7fe640 1 -- 172.21.15.29:0/1595343283 learned_addr learned my addr 172.21.15.29:0/1595343283 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f195ffff640 1 --2- 172.21.15.29:0/1595343283 >> v2:172.21.15.29:3300/0 conn(0x7f19600d0be0 0x7f19600cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f1957fff640 1 --2- 172.21.15.29:0/1595343283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:32.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.253+0000 7f195effd640 1 -- 172.21.15.29:0/1595343283 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974764854 0 0) 0x7f196015d150 con 0x7f196015cd70 2023-10-16T15:33:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.254+0000 7f195ffff640 1 -- 172.21.15.29:0/1595343283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 msgr2=0x7f19600cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.254+0000 7f195ffff640 1 --2- 172.21.15.29:0/1595343283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.254+0000 7f195ffff640 1 -- 172.21.15.29:0/1595343283 >> v1:172.21.15.29:6789/0 conn(0x7f196015cd70 legacy=0x7f19600d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.254+0000 7f195ffff640 1 -- 172.21.15.29:0/1595343283 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600d1740 con 0x7f19600d0be0 2023-10-16T15:33:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.255+0000 7f195ffff640 1 --2- 172.21.15.29:0/1595343283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600cdd00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f1958009d90 tx=0x7f195802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e933601f330d674d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.255+0000 7f195effd640 1 -- 172.21.15.29:0/1595343283 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195802cc60 con 0x7f19600d0be0 2023-10-16T15:33:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.255+0000 7f195effd640 1 -- 172.21.15.29:0/1595343283 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195802cdc0 con 0x7f19600d0be0 2023-10-16T15:33:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.255+0000 7f195effd640 1 -- 172.21.15.29:0/1595343283 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958034660 con 0x7f19600d0be0 2023-10-16T15:33:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 -- 172.21.15.29:0/1595343283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 msgr2=0x7f19600cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 --2- 172.21.15.29:0/1595343283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600cdd00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f1958009d90 tx=0x7f195802beb0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 -- 172.21.15.29:0/1595343283 shutdown_connections 2023-10-16T15:33:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 --2- 172.21.15.29:0/1595343283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600cdd00 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 --2- 172.21.15.29:0/1595343283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 -- 172.21.15.29:0/1595343283 >> 172.21.15.29:0/1595343283 conn(0x7f19600cf570 msgr2=0x7f19600cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 -- 172.21.15.29:0/1595343283 shutdown_connections 2023-10-16T15:33:32.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.256+0000 7f1964802640 1 -- 172.21.15.29:0/1595343283 wait complete. 2023-10-16T15:33:32.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.258+0000 7f1964802640 1 Processor -- start 2023-10-16T15:33:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.258+0000 7f1964802640 1 -- start start 2023-10-16T15:33:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1964802640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1964802640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:32.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1964802640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f196015cd70 0x7f19601f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1964802640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196015a2f0 con 0x7f19600d0be0 2023-10-16T15:33:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1964802640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960150e20 con 0x7f196015cd70 2023-10-16T15:33:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1964802640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960150210 con 0x7f19600b8480 2023-10-16T15:33:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44782/0 (socket says 172.21.15.29:44782) 2023-10-16T15:33:32.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.259+0000 7f1957fff640 1 -- 172.21.15.29:0/1780833345 learned_addr learned my addr 172.21.15.29:0/1780833345 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1957fff640 1 -- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 msgr2=0x7f19600b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1957fff640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1957fff640 1 -- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f196015cd70 msgr2=0x7f19601f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1957fff640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f196015cd70 0x7f19601f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1957fff640 1 -- 172.21.15.29:0/1780833345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958009a70 con 0x7f19600d0be0 2023-10-16T15:33:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1957fff640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600b4250 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f194800d5f0 tx=0x7f194800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480125f0 con 0x7f19600d0be0 2023-10-16T15:33:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1948012750 con 0x7f19600d0be0 2023-10-16T15:33:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948012b00 con 0x7f19600d0be0 2023-10-16T15:33:32.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19601504f0 con 0x7f19600d0be0 2023-10-16T15:33:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.260+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1960150ab0 con 0x7f19600d0be0 2023-10-16T15:33:32.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.261+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196004e9b0 con 0x7f19600d0be0 2023-10-16T15:33:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.262+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1948012c60 con 0x7f19600d0be0 2023-10-16T15:33:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.263+0000 7f195cff9640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f194006af90 0x7f194006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.263+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6613+0+0 (secure 0 0 0) 0x7f194808a080 con 0x7f19600d0be0 2023-10-16T15:33:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.264+0000 7f195f7fe640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f194006af90 0x7f194006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:32.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.265+0000 7f195f7fe640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f194006af90 0x7f194006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1950007fd0 tx=0x7f1950007920 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:32.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.268+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f19480bcca0 con 0x7f19600d0be0 2023-10-16T15:33:32.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:32.461+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1960150f90 con 0x7f19600d0be0 2023-10-16T15:33:33.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.007+0000 7f195cff9640 1 -- 172.21.15.29:0/1780833345 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v170) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1948054bf0 con 0x7f19600d0be0 2023-10-16T15:33:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:33:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.010+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f194006af90 msgr2=0x7f194006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.010+0000 7f1964802640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f194006af90 0x7f194006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1950007fd0 tx=0x7f1950007920 comp rx=0 tx=0).stop 2023-10-16T15:33:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.010+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 msgr2=0x7f19600b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.010+0000 7f1964802640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600b4250 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f194800d5f0 tx=0x7f194800dab0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 shutdown_connections 2023-10-16T15:33:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f194006af90 0x7f194006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19600d0be0 0x7f19600b4250 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f196015cd70 0x7f19601f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 --2- 172.21.15.29:0/1780833345 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19600b8480 0x7f19600b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 >> 172.21.15.29:0/1780833345 conn(0x7f19600cf570 msgr2=0x7f19600cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 shutdown_connections 2023-10-16T15:33:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.011+0000 7f1964802640 1 -- 172.21.15.29:0/1780833345 wait complete. 2023-10-16T15:33:33.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:33:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.261+0000 7f2efb6da640 1 Processor -- start 2023-10-16T15:33:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.261+0000 7f2efb6da640 1 -- start start 2023-10-16T15:33:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2efb6da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ef40d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2efb6da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40d1420 con 0x7f2ef415cf50 2023-10-16T15:33:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2efb6da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef415d330 con 0x7f2ef40b8a10 2023-10-16T15:33:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2efb6da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40d1680 con 0x7f2ef40b4e60 2023-10-16T15:33:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2ef9476640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2ef415cf50 0x7f2ef40d0d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47654/0 (socket says 172.21.15.29:47654) 2023-10-16T15:33:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2ef9476640 1 -- 172.21.15.29:0/1547731330 learned_addr learned my addr 172.21.15.29:0/1547731330 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.263+0000 7f2ef9c77640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ef40d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378673941 0 0) 0x7f2ef40d1420 con 0x7f2ef415cf50 2023-10-16T15:33:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0003520 con 0x7f2ef415cf50 2023-10-16T15:33:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3993383348 0 0) 0x7f2ee0003520 con 0x7f2ef415cf50 2023-10-16T15:33:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 msgr2=0x7f2ef40d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ef40d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 >> v1:172.21.15.170:6789/0 conn(0x7f2ef40b8a10 legacy=0x7f2ef40b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40d1960 con 0x7f2ef415cf50 2023-10-16T15:33:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2ef0001670 con 0x7f2ef415cf50 2023-10-16T15:33:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 >> v1:172.21.15.29:6789/0 conn(0x7f2ef415cf50 legacy=0x7f2ef40d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee0003a10 0x7f2ee0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee00047c0 0x7f2ee0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ee0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40d1680 con 0x7f2ee00047c0 2023-10-16T15:33:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000bab0 con 0x7f2ee0003a10 2023-10-16T15:33:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.264+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000bc70 con 0x7f2ef40b4e60 2023-10-16T15:33:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee00047c0 0x7f2ee0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef9476640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee0003a10 0x7f2ee0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef9c77640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ee0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 msgr2=0x7f2ee0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ee0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee0003a10 msgr2=0x7f2ee0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee0003a10 0x7f2ee0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1547731330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee0003520 con 0x7f2ee00047c0 2023-10-16T15:33:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef9476640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee0003a10 0x7f2ee0003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.265+0000 7f2ef9c77640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40b4e60 0x7f2ee0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.266+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee00047c0 0x7f2ee0004c40 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4001750 tx=0x7f2ee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fb7748b0bd3f98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40052d0 con 0x7f2ee00047c0 2023-10-16T15:33:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee402e050 con 0x7f2ee00047c0 2023-10-16T15:33:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2eebfff640 1 -- 172.21.15.29:0/1547731330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40035f0 con 0x7f2ee00047c0 2023-10-16T15:33:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2efb6da640 1 -- 172.21.15.29:0/1547731330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee00047c0 msgr2=0x7f2ee0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1547731330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ee00047c0 0x7f2ee0004c40 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4001750 tx=0x7f2ee00057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2ef9c77640 1 -- 172.21.15.29:0/1547731330 reap_dead start 2023-10-16T15:33:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2efb6da640 1 -- 172.21.15.29:0/1547731330 shutdown_connections 2023-10-16T15:33:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2efb6da640 1 -- 172.21.15.29:0/1547731330 >> 172.21.15.29:0/1547731330 conn(0x7f2ef40cf750 msgr2=0x7f2ef40cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2efb6da640 1 -- 172.21.15.29:0/1547731330 shutdown_connections 2023-10-16T15:33:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.267+0000 7f2efb6da640 1 -- 172.21.15.29:0/1547731330 wait complete. 2023-10-16T15:33:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.269+0000 7f2efb6da640 1 Processor -- start 2023-10-16T15:33:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.269+0000 7f2efb6da640 1 -- start start 2023-10-16T15:33:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.270+0000 7f2efb6da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee00047c0 0x7f2ef41512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.270+0000 7f2efb6da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 0x7f2ef40cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.270+0000 7f2efb6da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef41f5b60 0x7f2ef41f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.270+0000 7f2efb6da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef415d520 con 0x7f2ef40caff0 2023-10-16T15:33:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.270+0000 7f2efb6da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40cadd0 con 0x7f2ee00047c0 2023-10-16T15:33:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.270+0000 7f2efb6da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40cbb60 con 0x7f2ef41f5b60 2023-10-16T15:33:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 0x7f2ef40cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 0x7f2ef40cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44796/0 (socket says 172.21.15.29:44796) 2023-10-16T15:33:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1315741876 learned_addr learned my addr 172.21.15.29:0/1315741876 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef9476640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee00047c0 0x7f2ef41512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef9c77640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef41f5b60 0x7f2ef41f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef41f5b60 msgr2=0x7f2ef41f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef41f5b60 0x7f2ef41f5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee00047c0 msgr2=0x7f2ef41512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee00047c0 0x7f2ef41512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef8c75640 1 -- 172.21.15.29:0/1315741876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef415d770 con 0x7f2ef40caff0 2023-10-16T15:33:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.271+0000 7f2ef9476640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee00047c0 0x7f2ef41512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.272+0000 7f2ef8c75640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 0x7f2ef40cb470 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4008fd0 tx=0x7f2ee4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.272+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40076c0 con 0x7f2ef40caff0 2023-10-16T15:33:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.272+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4007a30 con 0x7f2ef40caff0 2023-10-16T15:33:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.272+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4006be0 con 0x7f2ef40caff0 2023-10-16T15:33:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.272+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef415d520 con 0x7f2ef40caff0 2023-10-16T15:33:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.272+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef415dd30 con 0x7f2ef40caff0 2023-10-16T15:33:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.273+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef4072350 con 0x7f2ef40caff0 2023-10-16T15:33:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.274+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ee4006d40 con 0x7f2ef40caff0 2023-10-16T15:33:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.275+0000 7f2ee9ffb640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2ed806af90 0x7f2ed806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.275+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6131+0+0 (secure 0 0 0) 0x7f2ee40abbc0 con 0x7f2ef40caff0 2023-10-16T15:33:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.276+0000 7f2ef9476640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2ed806af90 0x7f2ed806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.277+0000 7f2ef9476640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2ed806af90 0x7f2ed806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00018e0 tx=0x7f2ef0009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.279+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ee4076800 con 0x7f2ef40caff0 2023-10-16T15:33:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:33.479+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ef404e160 con 0x7f2ef40caff0 2023-10-16T15:33:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.027+0000 7f2ee9ffb640 1 -- 172.21.15.29:0/1315741876 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v171) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2ee404b0d0 con 0x7f2ef40caff0 2023-10-16T15:33:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:33:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.032+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2ed806af90 msgr2=0x7f2ed806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.032+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2ed806af90 0x7f2ed806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00018e0 tx=0x7f2ef0009040 comp rx=0 tx=0).stop 2023-10-16T15:33:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.032+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 msgr2=0x7f2ef40cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.032+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 0x7f2ef40cb470 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4008fd0 tx=0x7f2ee4001120 comp rx=0 tx=0).stop 2023-10-16T15:33:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 shutdown_connections 2023-10-16T15:33:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f2ed806af90 0x7f2ed806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40caff0 0x7f2ef40cb470 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef41f5b60 0x7f2ef41f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 --2- 172.21.15.29:0/1315741876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ee00047c0 0x7f2ef41512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 >> 172.21.15.29:0/1315741876 conn(0x7f2ef40cf750 msgr2=0x7f2ef4157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.033+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 shutdown_connections 2023-10-16T15:33:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.034+0000 7f2efb6da640 1 -- 172.21.15.29:0/1315741876 wait complete. 2023-10-16T15:33:34.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:33:34.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.273+0000 7f88bf59e640 1 Processor -- start 2023-10-16T15:33:34.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.273+0000 7f88bf59e640 1 -- start start 2023-10-16T15:33:34.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.274+0000 7f88bf59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c0164660 0x7f88c006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.274+0000 7f88bf59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f88c006d520 0x7f88c00c1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88bf59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c0164e80 con 0x7f88c00c24a0 2023-10-16T15:33:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88bf59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c006d960 con 0x7f88c0164660 2023-10-16T15:33:34.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88bf59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c00c28b0 con 0x7f88c006d520 2023-10-16T15:33:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c0164660 0x7f88c006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c0164660 0x7f88c006cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41954/0 (socket says 172.21.15.29:41954) 2023-10-16T15:33:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 -- 172.21.15.29:0/3246733333 learned_addr learned my addr 172.21.15.29:0/3246733333 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 -- 172.21.15.29:0/3246733333 >> v2:172.21.15.170:3300/0 conn(0x7f88c006d520 msgr2=0x7f88c00c1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 --2- 172.21.15.29:0/3246733333 >> v2:172.21.15.170:3300/0 conn(0x7f88c006d520 0x7f88c00c1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 -- 172.21.15.29:0/3246733333 >> v1:172.21.15.170:6789/0 conn(0x7f88c00c24a0 legacy=0x7f88c00cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:34.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.275+0000 7f88be59c640 1 -- 172.21.15.29:0/3246733333 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c01650d0 con 0x7f88c0164660 2023-10-16T15:33:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.276+0000 7f88be59c640 1 --2- 172.21.15.29:0/3246733333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c006cfe0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f88c006f730 tx=0x7f88b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad6db895474466f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.277+0000 7f88bd59a640 1 -- 172.21.15.29:0/3246733333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0031870 con 0x7f88c0164660 2023-10-16T15:33:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.277+0000 7f88bd59a640 1 -- 172.21.15.29:0/3246733333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b00319d0 con 0x7f88c0164660 2023-10-16T15:33:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.277+0000 7f88bd59a640 1 -- 172.21.15.29:0/3246733333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0031d60 con 0x7f88c0164660 2023-10-16T15:33:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.277+0000 7f88bf59e640 1 -- 172.21.15.29:0/3246733333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 msgr2=0x7f88c006cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.277+0000 7f88bf59e640 1 --2- 172.21.15.29:0/3246733333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c006cfe0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f88c006f730 tx=0x7f88b002b910 comp rx=0 tx=0).stop 2023-10-16T15:33:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.278+0000 7f88bf59e640 1 -- 172.21.15.29:0/3246733333 shutdown_connections 2023-10-16T15:33:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.278+0000 7f88bf59e640 1 --2- 172.21.15.29:0/3246733333 >> v2:172.21.15.170:3300/0 conn(0x7f88c006d520 0x7f88c00c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.278+0000 7f88bf59e640 1 --2- 172.21.15.29:0/3246733333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c006cfe0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.278+0000 7f88bf59e640 1 -- 172.21.15.29:0/3246733333 >> 172.21.15.29:0/3246733333 conn(0x7f88c00b0a80 msgr2=0x7f88c00b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.278+0000 7f88bf59e640 1 -- 172.21.15.29:0/3246733333 shutdown_connections 2023-10-16T15:33:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.278+0000 7f88bf59e640 1 -- 172.21.15.29:0/3246733333 wait complete. 2023-10-16T15:33:34.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.280+0000 7f88bf59e640 1 Processor -- start 2023-10-16T15:33:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.280+0000 7f88bf59e640 1 -- start start 2023-10-16T15:33:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bf59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88c006d520 0x7f88c00c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bf59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88c00c24a0 0x7f88c0158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c00c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bf59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0099a10 con 0x7f88c0164660 2023-10-16T15:33:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bf59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00c2ac0 con 0x7f88c00c24a0 2023-10-16T15:33:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bf59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0158da0 con 0x7f88c006d520 2023-10-16T15:33:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c00c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c00c7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41970/0 (socket says 172.21.15.29:41970) 2023-10-16T15:33:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bed9d640 1 -- 172.21.15.29:0/514990300 learned_addr learned my addr 172.21.15.29:0/514990300 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88bdd9b640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88c00c24a0 0x7f88c0158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.281+0000 7f88be59c640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88c006d520 0x7f88c00c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bed9d640 1 -- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88c006d520 msgr2=0x7f88c00c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bed9d640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88c006d520 0x7f88c00c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bed9d640 1 -- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88c00c24a0 msgr2=0x7f88c0158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bed9d640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88c00c24a0 0x7f88c0158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bed9d640 1 -- 172.21.15.29:0/514990300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00094d0 con 0x7f88c0164660 2023-10-16T15:33:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bed9d640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c00c7a60 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c780 tx=0x7f88b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8003d80 con 0x7f88c0164660 2023-10-16T15:33:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b800ce90 con 0x7f88c0164660 2023-10-16T15:33:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80052b0 con 0x7f88c0164660 2023-10-16T15:33:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.282+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c00c6640 con 0x7f88c0164660 2023-10-16T15:33:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.283+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c00c6c00 con 0x7f88c0164660 2023-10-16T15:33:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.284+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c00c2c30 con 0x7f88c0164660 2023-10-16T15:33:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.285+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88b8005410 con 0x7f88c0164660 2023-10-16T15:33:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.286+0000 7f88b6ffd640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f88a006af90 0x7f88a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.286+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f88b8088c10 con 0x7f88c0164660 2023-10-16T15:33:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.286+0000 7f88be59c640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f88a006af90 0x7f88a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.287+0000 7f88be59c640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f88a006af90 0x7f88a006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88b0036000 tx=0x7f88b003c040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.293+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88b8053be0 con 0x7f88c0164660 2023-10-16T15:33:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.501+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f88c004e9b0 con 0x7f88c0164660 2023-10-16T15:33:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.507+0000 7f88b6ffd640 1 -- 172.21.15.29:0/514990300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f88b80170a0 con 0x7f88c0164660 2023-10-16T15:33:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f88a006af90 msgr2=0x7f88a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f88a006af90 0x7f88a006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88b0036000 tx=0x7f88b003c040 comp rx=0 tx=0).stop 2023-10-16T15:33:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 msgr2=0x7f88c00c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c00c7a60 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c780 tx=0x7f88b800cc40 comp rx=0 tx=0).stop 2023-10-16T15:33:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 shutdown_connections 2023-10-16T15:33:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f88a006af90 0x7f88a006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c0164660 0x7f88c00c7a60 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88c00c24a0 0x7f88c0158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 --2- 172.21.15.29:0/514990300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88c006d520 0x7f88c00c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 >> 172.21.15.29:0/514990300 conn(0x7f88c00b0a80 msgr2=0x7f88c00ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 shutdown_connections 2023-10-16T15:33:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.510+0000 7f88bf59e640 1 -- 172.21.15.29:0/514990300 wait complete. 2023-10-16T15:33:34.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:33:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.742+0000 7fcfbffff640 1 Processor -- start 2023-10-16T15:33:34.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.742+0000 7fcfbffff640 1 -- start start 2023-10-16T15:33:34.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.743+0000 7fcfbffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fcfc00cd970 0x7fcfc015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.743+0000 7fcfbffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc006be30 con 0x7fcfc00b3ec0 2023-10-16T15:33:34.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.743+0000 7fcfbffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc00ada10 con 0x7fcfc00b0ea0 2023-10-16T15:33:34.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.743+0000 7fcfbffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc006ca00 con 0x7fcfc00cd970 2023-10-16T15:33:34.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.744+0000 7fcfbeffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcfc00b3ec0 0x7fcfc00cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54714/0 (socket says 172.21.15.29:54714) 2023-10-16T15:33:34.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.744+0000 7fcfbeffd640 1 -- 172.21.15.29:0/817188602 learned_addr learned my addr 172.21.15.29:0/817188602 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.744+0000 7fcfbf7fe640 1 --2- 172.21.15.29:0/817188602 >> v2:172.21.15.170:3300/0 conn(0x7fcfc00cd970 0x7fcfc015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.744+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46960335 0 0) 0x7fcfc006be30 con 0x7fcfc00b3ec0 2023-10-16T15:33:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.744+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa8003520 con 0x7fcfc00b3ec0 2023-10-16T15:33:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1721194417 0 0) 0x7fcfa8003520 con 0x7fcfc00b3ec0 2023-10-16T15:33:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 >> v2:172.21.15.170:3300/0 conn(0x7fcfc00cd970 msgr2=0x7fcfc015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 --2- 172.21.15.29:0/817188602 >> v2:172.21.15.170:3300/0 conn(0x7fcfc00cd970 0x7fcfc015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 >> v1:172.21.15.170:6789/0 conn(0x7fcfc00b0ea0 legacy=0x7fcfc015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc006c060 con 0x7fcfc00b3ec0 2023-10-16T15:33:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fcfb0001670 con 0x7fcfc00b3ec0 2023-10-16T15:33:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 >> v1:172.21.15.29:6789/0 conn(0x7fcfc00b3ec0 legacy=0x7fcfc00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa80039c0 0x7fcfa8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80047c0 0x7fcfa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfa80051d0 0x7fcfa800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc006ca00 con 0x7fcfa80051d0 2023-10-16T15:33:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa800c120 con 0x7fcfa80039c0 2023-10-16T15:33:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa800c2e0 con 0x7fcfa80047c0 2023-10-16T15:33:34.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.745+0000 7fcfbf7fe640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfa80051d0 0x7fcfa800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbeffd640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa80039c0 0x7fcfa8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbe7fc640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80047c0 0x7fcfa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbf7fe640 1 -- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80047c0 msgr2=0x7fcfa8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbf7fe640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80047c0 0x7fcfa8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbf7fe640 1 -- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa80039c0 msgr2=0x7fcfa8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbf7fe640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa80039c0 0x7fcfa8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.746+0000 7fcfbf7fe640 1 -- 172.21.15.29:0/817188602 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8003520 con 0x7fcfa80051d0 2023-10-16T15:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.747+0000 7fcfbf7fe640 1 -- 172.21.15.29:0/817188602 reap_dead start 2023-10-16T15:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.747+0000 7fcfbf7fe640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfa80051d0 0x7fcfa800b980 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009f90 tx=0x7fcfa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aed294e04b22e00 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.747+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb802cbf0 con 0x7fcfa80051d0 2023-10-16T15:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.747+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb800f040 con 0x7fcfa80051d0 2023-10-16T15:33:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.747+0000 7fcfbdffb640 1 -- 172.21.15.29:0/817188602 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8003990 con 0x7fcfa80051d0 2023-10-16T15:33:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 -- 172.21.15.29:0/817188602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfa80051d0 msgr2=0x7fcfa800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfa80051d0 0x7fcfa800b980 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009f90 tx=0x7fcfa80057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 -- 172.21.15.29:0/817188602 shutdown_connections 2023-10-16T15:33:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 --2- 172.21.15.29:0/817188602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfa80051d0 0x7fcfa800b980 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 -- 172.21.15.29:0/817188602 >> 172.21.15.29:0/817188602 conn(0x7fcfc00b0170 msgr2=0x7fcfc00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 -- 172.21.15.29:0/817188602 shutdown_connections 2023-10-16T15:33:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.748+0000 7fcfbffff640 1 -- 172.21.15.29:0/817188602 wait complete. 2023-10-16T15:33:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.750+0000 7fcfbffff640 1 Processor -- start 2023-10-16T15:33:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.750+0000 7fcfbffff640 1 -- start start 2023-10-16T15:33:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80051d0 0x7fcfc0148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 0x7fcfc00acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc0154440 0x7fcfc01548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00ad250 con 0x7fcfc00ac890 2023-10-16T15:33:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc0155170 con 0x7fcfc0154440 2023-10-16T15:33:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc0149000 con 0x7fcfa80051d0 2023-10-16T15:33:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbe7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 0x7fcfc00acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbe7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 0x7fcfc00acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41992/0 (socket says 172.21.15.29:41992) 2023-10-16T15:33:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.751+0000 7fcfbe7fc640 1 -- 172.21.15.29:0/2903968175 learned_addr learned my addr 172.21.15.29:0/2903968175 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbe7fc640 1 -- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80051d0 msgr2=0x7fcfc0148ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbe7fc640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80051d0 0x7fcfc0148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbe7fc640 1 -- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc0154440 msgr2=0x7fcfc01548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbe7fc640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc0154440 0x7fcfc01548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbe7fc640 1 -- 172.21.15.29:0/2903968175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc01568d0 con 0x7fcfc00ac890 2023-10-16T15:33:34.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbe7fc640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 0x7fcfc00acd10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400e480 tx=0x7fcfb400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:34.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40052e0 con 0x7fcfc00ac890 2023-10-16T15:33:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4005440 con 0x7fcfc00ac890 2023-10-16T15:33:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc0156bb0 con 0x7fcfc00ac890 2023-10-16T15:33:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.752+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4003830 con 0x7fcfc00ac890 2023-10-16T15:33:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.753+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc0157170 con 0x7fcfc00ac890 2023-10-16T15:33:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.754+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc0157330 con 0x7fcfc00ac890 2023-10-16T15:33:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.755+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcfb4003990 con 0x7fcfc00ac890 2023-10-16T15:33:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.756+0000 7fcfa7fff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcfac06af90 0x7fcfac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.756+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcfb40894b0 con 0x7fcfc00ac890 2023-10-16T15:33:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.756+0000 7fcfbeffd640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcfac06af90 0x7fcfac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.758+0000 7fcfbeffd640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcfac06af90 0x7fcfac06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0001050 tx=0x7fcfb0000f60 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.761+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcfb4054510 con 0x7fcfc00ac890 2023-10-16T15:33:34.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.957+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fcfc004e9b0 con 0x7fcfc00ac890 2023-10-16T15:33:34.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.963+0000 7fcfa7fff640 1 -- 172.21.15.29:0/2903968175 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v56) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcfb4057b30 con 0x7fcfc00ac890 2023-10-16T15:33:34.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcfac06af90 msgr2=0x7fcfac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcfac06af90 0x7fcfac06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0001050 tx=0x7fcfb0000f60 comp rx=0 tx=0).stop 2023-10-16T15:33:34.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 msgr2=0x7fcfc00acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:34.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 0x7fcfc00acd10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400e480 tx=0x7fcfb400e940 comp rx=0 tx=0).stop 2023-10-16T15:33:34.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 shutdown_connections 2023-10-16T15:33:34.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcfac06af90 0x7fcfac06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfc00ac890 0x7fcfc00acd10 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfc0154440 0x7fcfc01548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 --2- 172.21.15.29:0/2903968175 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcfa80051d0 0x7fcfc0148ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:34.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 >> 172.21.15.29:0/2903968175 conn(0x7fcfc00b0170 msgr2=0x7fcfc00b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:34.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 shutdown_connections 2023-10-16T15:33:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:34.966+0000 7fcfbffff640 1 -- 172.21.15.29:0/2903968175 wait complete. 2023-10-16T15:33:34.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:33:35.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.213+0000 7fafce8a3640 1 Processor -- start 2023-10-16T15:33:35.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.214+0000 7fafce8a3640 1 -- start start 2023-10-16T15:33:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.215+0000 7fafce8a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafc815c9f0 0x7fafc80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.215+0000 7fafce8a3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fafc815dde0 0x7fafc80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.215+0000 7fafce8a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc8160a90 con 0x7fafc80c9230 2023-10-16T15:33:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.215+0000 7fafce8a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc80cdec0 con 0x7fafc815c9f0 2023-10-16T15:33:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.215+0000 7fafce8a3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc80cd900 con 0x7fafc815dde0 2023-10-16T15:33:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.215+0000 7fafcd8a1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafc815c9f0 0x7fafc80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafcd8a1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafc815c9f0 0x7fafc80c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42000/0 (socket says 172.21.15.29:42000) 2023-10-16T15:33:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafcd8a1640 1 -- 172.21.15.29:0/1536794857 learned_addr learned my addr 172.21.15.29:0/1536794857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafce0a2640 1 --2- 172.21.15.29:0/1536794857 >> v2:172.21.15.170:3300/0 conn(0x7fafc815dde0 0x7fafc80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafcd8a1640 1 -- 172.21.15.29:0/1536794857 >> v2:172.21.15.170:3300/0 conn(0x7fafc815dde0 msgr2=0x7fafc80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafcd8a1640 1 --2- 172.21.15.29:0/1536794857 >> v2:172.21.15.170:3300/0 conn(0x7fafc815dde0 0x7fafc80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafcd8a1640 1 -- 172.21.15.29:0/1536794857 >> v1:172.21.15.29:6789/0 conn(0x7fafc80c9230 legacy=0x7fafc80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.216+0000 7fafcd8a1640 1 -- 172.21.15.29:0/1536794857 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80cdbe0 con 0x7fafc815c9f0 2023-10-16T15:33:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.217+0000 7fafcd8a1640 1 --2- 172.21.15.29:0/1536794857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815c9f0 0x7fafc80c8cf0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fafc80b0120 tx=0x7fafc402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6804556848f69b00 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafcc89f640 1 -- 172.21.15.29:0/1536794857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4030870 con 0x7fafc815c9f0 2023-10-16T15:33:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafcc89f640 1 -- 172.21.15.29:0/1536794857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc40309d0 con 0x7fafc815c9f0 2023-10-16T15:33:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafcc89f640 1 -- 172.21.15.29:0/1536794857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4030d60 con 0x7fafc815c9f0 2023-10-16T15:33:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafce8a3640 1 -- 172.21.15.29:0/1536794857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815c9f0 msgr2=0x7fafc80c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafce8a3640 1 --2- 172.21.15.29:0/1536794857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815c9f0 0x7fafc80c8cf0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fafc80b0120 tx=0x7fafc402b9b0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafce8a3640 1 -- 172.21.15.29:0/1536794857 shutdown_connections 2023-10-16T15:33:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafce8a3640 1 --2- 172.21.15.29:0/1536794857 >> v2:172.21.15.170:3300/0 conn(0x7fafc815dde0 0x7fafc80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafce8a3640 1 --2- 172.21.15.29:0/1536794857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815c9f0 0x7fafc80c8cf0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.218+0000 7fafce8a3640 1 -- 172.21.15.29:0/1536794857 >> 172.21.15.29:0/1536794857 conn(0x7fafc80ce6a0 msgr2=0x7fafc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.219+0000 7fafce8a3640 1 -- 172.21.15.29:0/1536794857 shutdown_connections 2023-10-16T15:33:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.219+0000 7fafce8a3640 1 -- 172.21.15.29:0/1536794857 wait complete. 2023-10-16T15:33:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.220+0000 7fafce8a3640 1 Processor -- start 2023-10-16T15:33:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 -- start start 2023-10-16T15:33:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafc80c9230 0x7fafc806aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815c9f0 0x7fafc81f66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 0x7fafc8150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81511a0 con 0x7fafc815dde0 2023-10-16T15:33:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81f5c70 con 0x7fafc80c9230 2023-10-16T15:33:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.221+0000 7fafce8a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81f6230 con 0x7fafc815c9f0 2023-10-16T15:33:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 0x7fafc8150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 0x7fafc8150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42012/0 (socket says 172.21.15.29:42012) 2023-10-16T15:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 -- 172.21.15.29:0/533189753 learned_addr learned my addr 172.21.15.29:0/533189753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafcd8a1640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafc80c9230 0x7fafc806aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 -- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815c9f0 msgr2=0x7fafc81f66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815c9f0 0x7fafc81f66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 -- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafc80c9230 msgr2=0x7fafc806aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafc80c9230 0x7fafc806aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafce0a2640 1 -- 172.21.15.29:0/533189753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40094d0 con 0x7fafc815dde0 2023-10-16T15:33:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.222+0000 7fafcd8a1640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafc80c9230 0x7fafc806aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.223+0000 7fafce0a2640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 0x7fafc8150c60 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fafc0007d70 tx=0x7fafc000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.223+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0014420 con 0x7fafc815dde0 2023-10-16T15:33:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.223+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc00145a0 con 0x7fafc815dde0 2023-10-16T15:33:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.223+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0014950 con 0x7fafc815dde0 2023-10-16T15:33:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.223+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc80c4e90 con 0x7fafc815dde0 2023-10-16T15:33:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.223+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc81f5ec0 con 0x7fafc815dde0 2023-10-16T15:33:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.224+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc81f63c0 con 0x7fafc815dde0 2023-10-16T15:33:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.225+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fafc0014ab0 con 0x7fafc815dde0 2023-10-16T15:33:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.226+0000 7fafbe7fc640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fafac06b010 0x7fafac06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.226+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fafc0089fd0 con 0x7fafc815dde0 2023-10-16T15:33:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.227+0000 7fafcd8a1640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fafac06b010 0x7fafac06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.228+0000 7fafcd8a1640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fafac06b010 0x7fafac06d4d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fafc4036000 tx=0x7fafc403b040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.230+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fafc0055010 con 0x7fafc815dde0 2023-10-16T15:33:35.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.424+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fafc804e9b0 con 0x7fafc815dde0 2023-10-16T15:33:35.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.429+0000 7fafbe7fc640 1 -- 172.21.15.29:0/533189753 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7fafc0058630 con 0x7fafc815dde0 2023-10-16T15:33:35.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fafac06b010 msgr2=0x7fafac06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fafac06b010 0x7fafac06d4d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fafc4036000 tx=0x7fafc403b040 comp rx=0 tx=0).stop 2023-10-16T15:33:35.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 msgr2=0x7fafc8150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 0x7fafc8150c60 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fafc0007d70 tx=0x7fafc000e4b0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 shutdown_connections 2023-10-16T15:33:35.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fafac06b010 0x7fafac06d4d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafc815dde0 0x7fafc8150c60 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fafc815c9f0 0x7fafc81f66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 --2- 172.21.15.29:0/533189753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fafc80c9230 0x7fafc806aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 >> 172.21.15.29:0/533189753 conn(0x7fafc80ce6a0 msgr2=0x7fafc80ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:35.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.432+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 shutdown_connections 2023-10-16T15:33:35.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.433+0000 7fafce8a3640 1 -- 172.21.15.29:0/533189753 wait complete. 2023-10-16T15:33:35.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T15:33:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.662+0000 7fe4cbfff640 1 Processor -- start 2023-10-16T15:33:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.662+0000 7fe4cbfff640 1 -- start start 2023-10-16T15:33:35.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.663+0000 7fe4cbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4cc1599c0 0x7fe4cc0ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.663+0000 7fe4cbfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc06bab0 con 0x7fe4cc0cda90 2023-10-16T15:33:35.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.663+0000 7fe4cbfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc06bd90 con 0x7fe4cc0ca6f0 2023-10-16T15:33:35.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.663+0000 7fe4cbfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4cc06f300 con 0x7fe4cc1599c0 2023-10-16T15:33:35.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4caffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4cc1599c0 0x7fe4cc0ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4caffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4cc1599c0 0x7fe4cc0ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42020/0 (socket says 172.21.15.29:42020) 2023-10-16T15:33:35.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4cb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe4cc0cda90 0x7fe4cc06f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54734/0 (socket says 172.21.15.29:54734) 2023-10-16T15:33:35.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4caffd640 1 -- 172.21.15.29:0/1079637387 learned_addr learned my addr 172.21.15.29:0/1079637387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4caffd640 1 -- 172.21.15.29:0/1079637387 >> v1:172.21.15.170:6789/0 conn(0x7fe4cc0ca6f0 legacy=0x7fe4cc0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4caffd640 1 -- 172.21.15.29:0/1079637387 >> v1:172.21.15.29:6789/0 conn(0x7fe4cc0cda90 legacy=0x7fe4cc06f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.664+0000 7fe4caffd640 1 -- 172.21.15.29:0/1079637387 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc06c070 con 0x7fe4cc1599c0 2023-10-16T15:33:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.665+0000 7fe4caffd640 1 --2- 172.21.15.29:0/1079637387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0ca1b0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009870 tx=0x7fe4bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b28004e91c9ad6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.665+0000 7fe4ca7fc640 1 -- 172.21.15.29:0/1079637387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc031870 con 0x7fe4cc1599c0 2023-10-16T15:33:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4ca7fc640 1 -- 172.21.15.29:0/1079637387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0319d0 con 0x7fe4cc1599c0 2023-10-16T15:33:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4ca7fc640 1 -- 172.21.15.29:0/1079637387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc031d60 con 0x7fe4cc1599c0 2023-10-16T15:33:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 -- 172.21.15.29:0/1079637387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc1599c0 msgr2=0x7fe4cc0ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/1079637387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0ca1b0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009870 tx=0x7fe4bc02b910 comp rx=0 tx=0).stop 2023-10-16T15:33:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 -- 172.21.15.29:0/1079637387 shutdown_connections 2023-10-16T15:33:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/1079637387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0ca1b0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 -- 172.21.15.29:0/1079637387 >> 172.21.15.29:0/1079637387 conn(0x7fe4cc0b0a90 msgr2=0x7fe4cc0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:35.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 -- 172.21.15.29:0/1079637387 shutdown_connections 2023-10-16T15:33:35.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.666+0000 7fe4cbfff640 1 -- 172.21.15.29:0/1079637387 wait complete. 2023-10-16T15:33:35.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.668+0000 7fe4cbfff640 1 Processor -- start 2023-10-16T15:33:35.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.668+0000 7fe4cbfff640 1 -- start start 2023-10-16T15:33:35.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.669+0000 7fe4cbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4cc0ca6f0 0x7fe4cc14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.669+0000 7fe4cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 0x7fe4cc0c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.669+0000 7fe4cbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.669+0000 7fe4cbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc1497e0 con 0x7fe4cc0cda90 2023-10-16T15:33:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.669+0000 7fe4cbfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc149f80 con 0x7fe4cc1599c0 2023-10-16T15:33:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.669+0000 7fe4cbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc14a260 con 0x7fe4cc0ca6f0 2023-10-16T15:33:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 0x7fe4cc0c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 0x7fe4cc0c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42030/0 (socket says 172.21.15.29:42030) 2023-10-16T15:33:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 -- 172.21.15.29:0/2414430578 learned_addr learned my addr 172.21.15.29:0/2414430578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4caffd640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4cc0ca6f0 0x7fe4cc14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4cb7fe640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 -- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4cc0ca6f0 msgr2=0x7fe4cc14b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4cc0ca6f0 0x7fe4cc14b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 -- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4cc1599c0 msgr2=0x7fe4cc0c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 -- 172.21.15.29:0/2414430578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc0094d0 con 0x7fe4cc0cda90 2023-10-16T15:33:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4cb7fe640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.670+0000 7fe4c27fc640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 0x7fe4cc0c58d0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400c430 tx=0x7fe4b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.671+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4011790 con 0x7fe4cc0cda90 2023-10-16T15:33:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.671+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b40118f0 con 0x7fe4cc0cda90 2023-10-16T15:33:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.671+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4011ce0 con 0x7fe4cc0cda90 2023-10-16T15:33:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.671+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc14a540 con 0x7fe4cc0cda90 2023-10-16T15:33:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.671+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc14ab00 con 0x7fe4cc0cda90 2023-10-16T15:33:35.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.672+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4cc06c330 con 0x7fe4cc0cda90 2023-10-16T15:33:35.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.673+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe4b401c460 con 0x7fe4cc0cda90 2023-10-16T15:33:35.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.674+0000 7fe4c8ff9640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fe4ac06af90 0x7fe4ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.674+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fe4b40897e0 con 0x7fe4cc0cda90 2023-10-16T15:33:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.674+0000 7fe4caffd640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fe4ac06af90 0x7fe4ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.676+0000 7fe4caffd640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fe4ac06af90 0x7fe4ac06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc036000 tx=0x7fe4bc03c040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.678+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4cc06c330 con 0x7fe4cc0cda90 2023-10-16T15:33:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:35.939+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fe4cc06c330 con 0x7fe4cc0cda90 2023-10-16T15:33:36.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.032+0000 7fe4c8ff9640 1 -- 172.21.15.29:0/2414430578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2516) v1 ==== 178+0+0 (secure 0 0 0) 0x7fe4cc06c330 con 0x7fe4cc0cda90 2023-10-16T15:33:36.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fe4ac06af90 msgr2=0x7fe4ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fe4ac06af90 0x7fe4ac06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc036000 tx=0x7fe4bc03c040 comp rx=0 tx=0).stop 2023-10-16T15:33:36.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 msgr2=0x7fe4cc0c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 0x7fe4cc0c58d0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400c430 tx=0x7fe4b400c8f0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 shutdown_connections 2023-10-16T15:33:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fe4ac06af90 0x7fe4ac06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4cc0cda90 0x7fe4cc0c58d0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4cc1599c0 0x7fe4cc0c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 --2- 172.21.15.29:0/2414430578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe4cc0ca6f0 0x7fe4cc14b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.035+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 >> 172.21.15.29:0/2414430578 conn(0x7fe4cc0b0a90 msgr2=0x7fe4cc15b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.036+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 shutdown_connections 2023-10-16T15:33:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.036+0000 7fe4cbfff640 1 -- 172.21.15.29:0/2414430578 wait complete. 2023-10-16T15:33:36.056 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-16T15:33:36.056 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-16T15:33:36.056 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T15:33:36.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:36.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.279+0000 7f7b52316640 1 Processor -- start 2023-10-16T15:33:36.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.279+0000 7f7b52316640 1 -- start start 2023-10-16T15:33:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.281+0000 7f7b52316640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7b4c160a40 0x7f7b4c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.281+0000 7f7b52316640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c0cd060 con 0x7f7b4c06c800 2023-10-16T15:33:36.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.281+0000 7f7b52316640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c0cdbe0 con 0x7f7b4c15d080 2023-10-16T15:33:36.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.281+0000 7f7b52316640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c0cd2b0 con 0x7f7b4c160a40 2023-10-16T15:33:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.281+0000 7f7b4bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7b4c06c800 0x7f7b4c15c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54742/0 (socket says 172.21.15.29:54742) 2023-10-16T15:33:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.281+0000 7f7b4bfff640 1 -- 172.21.15.29:0/1300845754 learned_addr learned my addr 172.21.15.29:0/1300845754 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:36.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4b7fe640 1 --2- 172.21.15.29:0/1300845754 >> v2:172.21.15.170:3300/0 conn(0x7f7b4c160a40 0x7f7b4c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846360923 0 0) 0x7f7b4c0cd060 con 0x7f7b4c06c800 2023-10-16T15:33:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b34003520 con 0x7f7b4c06c800 2023-10-16T15:33:36.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2259537718 0 0) 0x7f7b34003520 con 0x7f7b4c06c800 2023-10-16T15:33:36.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 >> v2:172.21.15.170:3300/0 conn(0x7f7b4c160a40 msgr2=0x7f7b4c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 --2- 172.21.15.29:0/1300845754 >> v2:172.21.15.170:3300/0 conn(0x7f7b4c160a40 0x7f7b4c0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 >> v1:172.21.15.170:6789/0 conn(0x7f7b4c15d080 legacy=0x7f7b4c0cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c0cd590 con 0x7f7b4c06c800 2023-10-16T15:33:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7b3c001670 con 0x7f7b4c06c800 2023-10-16T15:33:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 >> v1:172.21.15.29:6789/0 conn(0x7f7b4c06c800 legacy=0x7f7b4c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b340039c0 0x7f7b34003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b340047c0 0x7f7b34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34005180 0x7f7b3400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0cd2b0 con 0x7f7b34005180 2023-10-16T15:33:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3400c080 con 0x7f7b340039c0 2023-10-16T15:33:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.282+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3400c240 con 0x7f7b340047c0 2023-10-16T15:33:36.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b508b3640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34005180 0x7f7b3400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b4b7fe640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b340047c0 0x7f7b34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b508b3640 1 -- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b340047c0 msgr2=0x7f7b34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b508b3640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b340047c0 0x7f7b34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b508b3640 1 -- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b340039c0 msgr2=0x7f7b34003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b508b3640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b340039c0 0x7f7b34003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b508b3640 1 -- 172.21.15.29:0/1300845754 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b34003520 con 0x7f7b34005180 2023-10-16T15:33:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.283+0000 7f7b4b7fe640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b340047c0 0x7f7b34004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.284+0000 7f7b508b3640 1 -- 172.21.15.29:0/1300845754 reap_dead start 2023-10-16T15:33:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.284+0000 7f7b508b3640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34005180 0x7f7b3400b980 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f7b44001e20 tx=0x7f7b340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435c4d9a34658835 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.284+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44003b70 con 0x7f7b34005180 2023-10-16T15:33:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.284+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b44003cd0 con 0x7f7b34005180 2023-10-16T15:33:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.284+0000 7f7b4affd640 1 -- 172.21.15.29:0/1300845754 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44005310 con 0x7f7b34005180 2023-10-16T15:33:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 -- 172.21.15.29:0/1300845754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34005180 msgr2=0x7f7b3400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34005180 0x7f7b3400b980 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f7b44001e20 tx=0x7f7b340057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 -- 172.21.15.29:0/1300845754 shutdown_connections 2023-10-16T15:33:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 --2- 172.21.15.29:0/1300845754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b34005180 0x7f7b3400b980 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 -- 172.21.15.29:0/1300845754 >> 172.21.15.29:0/1300845754 conn(0x7f7b4c06d700 msgr2=0x7f7b4c06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 -- 172.21.15.29:0/1300845754 shutdown_connections 2023-10-16T15:33:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.285+0000 7f7b52316640 1 -- 172.21.15.29:0/1300845754 wait complete. 2023-10-16T15:33:36.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.287+0000 7f7b52316640 1 Processor -- start 2023-10-16T15:33:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.287+0000 7f7b52316640 1 -- start start 2023-10-16T15:33:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b52316640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b34005180 0x7f7b4c152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b52316640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 0x7f7b4c0b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b52316640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b4c0cc150 0x7f7b4c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b52316640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0ccb40 con 0x7f7b4c0b0670 2023-10-16T15:33:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b52316640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0c7cd0 con 0x7f7b34005180 2023-10-16T15:33:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b52316640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0c7fb0 con 0x7f7b4c0cc150 2023-10-16T15:33:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 0x7f7b4c0b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 0x7f7b4c0b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42046/0 (socket says 172.21.15.29:42046) 2023-10-16T15:33:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.288+0000 7f7b4b7fe640 1 -- 172.21.15.29:0/2181737631 learned_addr learned my addr 172.21.15.29:0/2181737631 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b4b7fe640 1 -- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b4c0cc150 msgr2=0x7f7b4c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b4b7fe640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b4c0cc150 0x7f7b4c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b4b7fe640 1 -- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b34005180 msgr2=0x7f7b4c152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b4b7fe640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b34005180 0x7f7b4c152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b4b7fe640 1 -- 172.21.15.29:0/2181737631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c0c8290 con 0x7f7b4c0b0670 2023-10-16T15:33:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b4b7fe640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 0x7f7b4c0b0af0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7b400108b0 tx=0x7f7b40010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000eb60 con 0x7f7b4c0b0670 2023-10-16T15:33:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4000ecc0 con 0x7f7b4c0b0670 2023-10-16T15:33:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4001c400 con 0x7f7b4c0b0670 2023-10-16T15:33:36.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.289+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c0c8570 con 0x7f7b4c0b0670 2023-10-16T15:33:36.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.290+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c1525a0 con 0x7f7b4c0b0670 2023-10-16T15:33:36.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.290+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c04e160 con 0x7f7b4c0b0670 2023-10-16T15:33:36.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.291+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b4001c560 con 0x7f7b4c0b0670 2023-10-16T15:33:36.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.292+0000 7f7b48ff9640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b3806af90 0x7f7b3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.292+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f7b40089d70 con 0x7f7b4c0b0670 2023-10-16T15:33:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.293+0000 7f7b4bfff640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b3806af90 0x7f7b3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.294+0000 7f7b4bfff640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b3806af90 0x7f7b3806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c000ff0 tx=0x7f7b3c001540 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.297+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b40054de0 con 0x7f7b4c0b0670 2023-10-16T15:33:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.490+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b4c0b6a50 con 0x7f7b4c0b0670 2023-10-16T15:33:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.492+0000 7f7b48ff9640 1 -- 172.21.15.29:0/2181737631 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+14054 (secure 0 0 0) 0x7f7b40058400 con 0x7f7b4c0b0670 2023-10-16T15:33:36.493 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:36.494 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":171,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:34.023583+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6833/3542022450":"2023-10-17T15:33:30.997230+0000","172.21.15.170:0/246679773":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/951615953":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/3176243662":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/3888018037":"2023-10-17T15:33:26.923263+0000","172.21.15.29:6832/3542022450":"2023-10-17T15:33:30.997230+0000","172.21.15.170:6833/3272758872":"2023-10-17T15:33:26.923263+0000","172.21.15.170:6832/3272758872":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/91040253":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/999637714":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3650756237":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/907915415":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6833/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:6832/1985319551":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/3571017587":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1946312558":"2023-10-17T14:37:28.457671+0000","172.21.15.170:0/1765158227":"2023-10-17T15:33:26.923263+0000","172.21.15.170:0/1694874697":"2023-10-17T14:37:28.457671+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b3806af90 msgr2=0x7f7b3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b3806af90 0x7f7b3806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c000ff0 tx=0x7f7b3c001540 comp rx=0 tx=0).stop 2023-10-16T15:33:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 msgr2=0x7f7b4c0b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 0x7f7b4c0b0af0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f7b400108b0 tx=0x7f7b40010d70 comp rx=0 tx=0).stop 2023-10-16T15:33:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 shutdown_connections 2023-10-16T15:33:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b3806af90 0x7f7b3806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b4c0b0670 0x7f7b4c0b0af0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b4c0cc150 0x7f7b4c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 --2- 172.21.15.29:0/2181737631 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b34005180 0x7f7b4c152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.494+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 >> 172.21.15.29:0/2181737631 conn(0x7f7b4c06d700 msgr2=0x7f7b4c15d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.495+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 shutdown_connections 2023-10-16T15:33:36.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.495+0000 7f7b52316640 1 -- 172.21.15.29:0/2181737631 wait complete. 2023-10-16T15:33:36.515 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:33:36.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:33:36.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.750+0000 7f248407c640 1 Processor -- start 2023-10-16T15:33:36.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.751+0000 7f248407c640 1 -- start start 2023-10-16T15:33:36.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.752+0000 7f248407c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f247c15ad70 0x7f247c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.752+0000 7f248407c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f247c0c9ca0 0x7f247c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.752+0000 7f248407c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.752+0000 7f248407c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c0cad80 con 0x7f247c15ad70 2023-10-16T15:33:36.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.752+0000 7f248407c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c0cb030 con 0x7f247c0c9ca0 2023-10-16T15:33:36.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.752+0000 7f248407c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0c91c0 con 0x7f247c0ca3b0 2023-10-16T15:33:36.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f247c15ad70 0x7f247c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f247c15ad70 0x7f247c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42056/0 (socket says 172.21.15.29:42056) 2023-10-16T15:33:36.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 -- 172.21.15.29:0/877312268 learned_addr learned my addr 172.21.15.29:0/877312268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:36.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2482619640 1 --2- 172.21.15.29:0/877312268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481617640 1 --2- 172.21.15.29:0/877312268 >> v2:172.21.15.170:3300/0 conn(0x7f247c0c9ca0 0x7f247c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 -- 172.21.15.29:0/877312268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 msgr2=0x7f247c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 --2- 172.21.15.29:0/877312268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 -- 172.21.15.29:0/877312268 >> v2:172.21.15.170:3300/0 conn(0x7f247c0c9ca0 msgr2=0x7f247c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 --2- 172.21.15.29:0/877312268 >> v2:172.21.15.170:3300/0 conn(0x7f247c0c9ca0 0x7f247c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.753+0000 7f2481e18640 1 -- 172.21.15.29:0/877312268 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0c94a0 con 0x7f247c15ad70 2023-10-16T15:33:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.754+0000 7f2481e18640 1 --2- 172.21.15.29:0/877312268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c069da0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f247c0ce870 tx=0x7f246c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b7e4a2612aaa2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.755+0000 7f2480e16640 1 -- 172.21.15.29:0/877312268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c02ca70 con 0x7f247c15ad70 2023-10-16T15:33:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.755+0000 7f2480e16640 1 -- 172.21.15.29:0/877312268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c02cbd0 con 0x7f247c15ad70 2023-10-16T15:33:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.755+0000 7f2480e16640 1 -- 172.21.15.29:0/877312268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c0343f0 con 0x7f247c15ad70 2023-10-16T15:33:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.755+0000 7f248407c640 1 -- 172.21.15.29:0/877312268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 msgr2=0x7f247c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.755+0000 7f248407c640 1 --2- 172.21.15.29:0/877312268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c069da0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f247c0ce870 tx=0x7f246c02bed0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 -- 172.21.15.29:0/877312268 shutdown_connections 2023-10-16T15:33:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 --2- 172.21.15.29:0/877312268 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 --2- 172.21.15.29:0/877312268 >> v2:172.21.15.170:3300/0 conn(0x7f247c0c9ca0 0x7f247c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 --2- 172.21.15.29:0/877312268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c069da0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 -- 172.21.15.29:0/877312268 >> 172.21.15.29:0/877312268 conn(0x7f247c0ce9b0 msgr2=0x7f247c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:36.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 -- 172.21.15.29:0/877312268 shutdown_connections 2023-10-16T15:33:36.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.756+0000 7f248407c640 1 -- 172.21.15.29:0/877312268 wait complete. 2023-10-16T15:33:36.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.757+0000 7f248407c640 1 Processor -- start 2023-10-16T15:33:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.758+0000 7f248407c640 1 -- start start 2023-10-16T15:33:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.758+0000 7f248407c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f247c0c9ca0 0x7f247c0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f248407c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c15d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f248407c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c1fa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f248407c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0c5240 con 0x7f247c15ad70 2023-10-16T15:33:36.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f248407c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0c88e0 con 0x7f247c0c9ca0 2023-10-16T15:33:36.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f248407c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c146200 con 0x7f247c0ca3b0 2023-10-16T15:33:36.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c1fa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c1fa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42060/0 (socket says 172.21.15.29:42060) 2023-10-16T15:33:36.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 -- 172.21.15.29:0/2945793779 learned_addr learned my addr 172.21.15.29:0/2945793779 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2481e18640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f247c0c9ca0 0x7f247c0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2481617640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c15d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 -- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 msgr2=0x7f247c15d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c15d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 -- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f247c0c9ca0 msgr2=0x7f247c0c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f247c0c9ca0 0x7f247c0c7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2482619640 1 -- 172.21.15.29:0/2945793779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c009b30 con 0x7f247c15ad70 2023-10-16T15:33:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.759+0000 7f2481617640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c15d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.760+0000 7f2482619640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c1fa3a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f247400d800 tx=0x7f247400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.760+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247400b6c0 con 0x7f247c15ad70 2023-10-16T15:33:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.760+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247400b840 con 0x7f247c15ad70 2023-10-16T15:33:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.760+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247400bbf0 con 0x7f247c15ad70 2023-10-16T15:33:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.760+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c0c8320 con 0x7f247c15ad70 2023-10-16T15:33:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.760+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c0c5b90 con 0x7f247c15ad70 2023-10-16T15:33:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.761+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c04e160 con 0x7f247c15ad70 2023-10-16T15:33:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.762+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f247400bd50 con 0x7f247c15ad70 2023-10-16T15:33:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.763+0000 7f247a7fc640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f246806af90 0x7f246806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.763+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2474089d30 con 0x7f247c15ad70 2023-10-16T15:33:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.764+0000 7f2481e18640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f246806af90 0x7f246806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.765+0000 7f2481e18640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f246806af90 0x7f246806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f246c035000 tx=0x7f246c030000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:36.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.768+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2474054c60 con 0x7f247c15ad70 2023-10-16T15:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.953+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f247c0706a0 con 0x7f247c15ad70 2023-10-16T15:33:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.954+0000 7f247a7fc640 1 -- 172.21.15.29:0/2945793779 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2474058280 con 0x7f247c15ad70 2023-10-16T15:33:36.956 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:33:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f246806af90 msgr2=0x7f246806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f246806af90 0x7f246806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f246c035000 tx=0x7f246c030000 comp rx=0 tx=0).stop 2023-10-16T15:33:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 msgr2=0x7f247c1fa3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c1fa3a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f247400d800 tx=0x7f247400dcc0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 shutdown_connections 2023-10-16T15:33:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f246806af90 0x7f246806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247c15ad70 0x7f247c1fa3a0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247c0ca3b0 0x7f247c15d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 --2- 172.21.15.29:0/2945793779 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f247c0c9ca0 0x7f247c0c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 >> 172.21.15.29:0/2945793779 conn(0x7f247c0ce9b0 msgr2=0x7f247c0b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 shutdown_connections 2023-10-16T15:33:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:36.957+0000 7f248407c640 1 -- 172.21.15.29:0/2945793779 wait complete. 2023-10-16T15:33:36.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T15:33:37.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.182+0000 7f90b16ec640 1 Processor -- start 2023-10-16T15:33:37.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.182+0000 7f90b16ec640 1 -- start start 2023-10-16T15:33:37.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.184+0000 7f90b16ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15aaa0 0x7f90ac15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.184+0000 7f90b16ec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f90ac15ae80 0x7f90ac0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.184+0000 7f90b16ec640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f90ac0ccaa0 0x7f90ac070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.184+0000 7f90b16ec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac06b810 con 0x7f90ac15ae80 2023-10-16T15:33:37.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.184+0000 7f90b16ec640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac15ed60 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.184+0000 7f90b16ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0707e0 con 0x7f90ac15aaa0 2023-10-16T15:33:37.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f90ac15ae80 0x7f90ac0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:37.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f90ac15ae80 0x7f90ac0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42070/0 (socket says 172.21.15.29:42070) 2023-10-16T15:33:37.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 -- 172.21.15.29:0/2310324727 learned_addr learned my addr 172.21.15.29:0/2310324727 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:37.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90aaffd640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15aaa0 0x7f90ac15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:37.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 -- 172.21.15.29:0/2310324727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15aaa0 msgr2=0x7f90ac15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:37.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15aaa0 0x7f90ac15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90ab7fe640 1 --2- 172.21.15.29:0/2310324727 >> v2:172.21.15.170:3300/0 conn(0x7f90ac0ccaa0 0x7f90ac070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:37.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 -- 172.21.15.29:0/2310324727 >> v2:172.21.15.170:3300/0 conn(0x7f90ac0ccaa0 msgr2=0x7f90ac070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:37.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 --2- 172.21.15.29:0/2310324727 >> v2:172.21.15.170:3300/0 conn(0x7f90ac0ccaa0 0x7f90ac070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90a27fc640 1 -- 172.21.15.29:0/2310324727 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac15ef90 con 0x7f90ac15ae80 2023-10-16T15:33:37.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.185+0000 7f90aaffd640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15aaa0 0x7f90ac15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:37.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.186+0000 7f90a27fc640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac15ae80 0x7f90ac0cc560 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0ae3c0 tx=0x7f909402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dc2348c1b1edcc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:37.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.187+0000 7f90aa7fc640 1 -- 172.21.15.29:0/2310324727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094031880 con 0x7f90ac15ae80 2023-10-16T15:33:37.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.187+0000 7f90aa7fc640 1 -- 172.21.15.29:0/2310324727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90940319e0 con 0x7f90ac15ae80 2023-10-16T15:33:37.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.187+0000 7f90aa7fc640 1 -- 172.21.15.29:0/2310324727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094031d70 con 0x7f90ac15ae80 2023-10-16T15:33:37.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 -- 172.21.15.29:0/2310324727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac15ae80 msgr2=0x7f90ac0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:37.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac15ae80 0x7f90ac0cc560 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0ae3c0 tx=0x7f909402b920 comp rx=0 tx=0).stop 2023-10-16T15:33:37.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 -- 172.21.15.29:0/2310324727 shutdown_connections 2023-10-16T15:33:37.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 --2- 172.21.15.29:0/2310324727 >> v2:172.21.15.170:3300/0 conn(0x7f90ac0ccaa0 0x7f90ac070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac15ae80 0x7f90ac0cc560 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 --2- 172.21.15.29:0/2310324727 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15aaa0 0x7f90ac15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 -- 172.21.15.29:0/2310324727 >> 172.21.15.29:0/2310324727 conn(0x7f90ac0cb410 msgr2=0x7f90ac0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:37.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 -- 172.21.15.29:0/2310324727 shutdown_connections 2023-10-16T15:33:37.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.188+0000 7f90b16ec640 1 -- 172.21.15.29:0/2310324727 wait complete. 2023-10-16T15:33:37.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.190+0000 7f90b16ec640 1 Processor -- start 2023-10-16T15:33:37.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.190+0000 7f90b16ec640 1 -- start start 2023-10-16T15:33:37.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90b16ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 0x7f90ac0c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90b16ec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90ac15aaa0 0x7f90ac0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90b16ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15ae80 0x7f90ac159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90b16ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac15c100 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90b16ec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac15bdd0 con 0x7f90ac15aaa0 2023-10-16T15:33:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90b16ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0a8620 con 0x7f90ac15ae80 2023-10-16T15:33:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 0x7f90ac0c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:37.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 0x7f90ac0c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42076/0 (socket says 172.21.15.29:42076) 2023-10-16T15:33:37.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.191+0000 7f90aaffd640 1 -- 172.21.15.29:0/973177782 learned_addr learned my addr 172.21.15.29:0/973177782 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:37.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90a27fc640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90ac15aaa0 0x7f90ac0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:37.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90aaffd640 1 -- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15ae80 msgr2=0x7f90ac159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:37.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90aaffd640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15ae80 0x7f90ac159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90aaffd640 1 -- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90ac15aaa0 msgr2=0x7f90ac0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:37.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90aaffd640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90ac15aaa0 0x7f90ac0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:37.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90aaffd640 1 -- 172.21.15.29:0/973177782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940094d0 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90a27fc640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90ac15aaa0 0x7f90ac0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:37.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90aaffd640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 0x7f90ac0c4c00 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f909c00ce00 tx=0x7f909c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:37.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00f890 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c00f9f0 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00fdc0 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.192+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac0a8900 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.193+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac0ab420 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.194+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac04e160 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.195+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f909c00b040 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.196+0000 7f90a3fff640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f908c06afc0 0x7f908c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.196+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f909c08a2f0 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.197+0000 7f90a27fc640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f908c06afc0 0x7f908c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.198+0000 7f90a27fc640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f908c06afc0 0x7f908c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9094036000 tx=0x7f9094008030 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:37.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.201+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f909c055110 con 0x7f90ac0ccaa0 2023-10-16T15:33:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:37.472+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f90ac0cdc60 con 0x7f90ac0ccaa0 2023-10-16T15:33:38.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.037+0000 7f90a3fff640 1 -- 172.21.15.29:0/973177782 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2518) v1 ==== 177+0+0 (secure 0 0 0) 0x7f909c058730 con 0x7f90ac0ccaa0 2023-10-16T15:33:38.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f908c06afc0 msgr2=0x7f908c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f908c06afc0 0x7f908c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9094036000 tx=0x7f9094008030 comp rx=0 tx=0).stop 2023-10-16T15:33:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 msgr2=0x7f90ac0c4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 0x7f90ac0c4c00 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f909c00ce00 tx=0x7f909c00a650 comp rx=0 tx=0).stop 2023-10-16T15:33:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 shutdown_connections 2023-10-16T15:33:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f908c06afc0 0x7f908c06d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90ac15ae80 0x7f90ac159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90ac15aaa0 0x7f90ac0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 --2- 172.21.15.29:0/973177782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90ac0ccaa0 0x7f90ac0c4c00 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 >> 172.21.15.29:0/973177782 conn(0x7f90ac0cb410 msgr2=0x7f90ac070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 shutdown_connections 2023-10-16T15:33:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.042+0000 7f90b16ec640 1 -- 172.21.15.29:0/973177782 wait complete. 2023-10-16T15:33:38.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T15:33:38.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.302+0000 7f0658d85640 1 Processor -- start 2023-10-16T15:33:38.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.302+0000 7f0658d85640 1 -- start start 2023-10-16T15:33:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.304+0000 7f0658d85640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 0x7f065415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.304+0000 7f0658d85640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065415aef0 con 0x7f0654159950 2023-10-16T15:33:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.304+0000 7f0658d85640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065406cc60 con 0x7f065406c100 2023-10-16T15:33:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.304+0000 7f0658d85640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065415e490 con 0x7f0654070b60 2023-10-16T15:33:38.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.304+0000 7f06537fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0654159950 0x7f0654070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54750/0 (socket says 172.21.15.29:54750) 2023-10-16T15:33:38.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.304+0000 7f06537fe640 1 -- 172.21.15.29:0/4001513531 learned_addr learned my addr 172.21.15.29:0/4001513531 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:38.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f0652ffd640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 0x7f065415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160213466 0 0) 0x7f0644001490 con 0x7f0654159950 2023-10-16T15:33:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0640003520 con 0x7f0654159950 2023-10-16T15:33:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589816890 0 0) 0x7f0640003520 con 0x7f0654159950 2023-10-16T15:33:38.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 msgr2=0x7f065415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 0x7f065415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 >> v1:172.21.15.170:6789/0 conn(0x7f065406c100 legacy=0x7f065406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065415d910 con 0x7f0654159950 2023-10-16T15:33:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0644001b80 con 0x7f0654159950 2023-10-16T15:33:38.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f0652ffd640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 0x7f065415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:33:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.305+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 >> v1:172.21.15.29:6789/0 conn(0x7f0654159950 legacy=0x7f0654070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06527fc640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06400039a0 0x7f0640003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06527fc640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400047c0 0x7f0640004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06527fc640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 0x7f0640005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065415e490 con 0x7f06400039a0 2023-10-16T15:33:38.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064000bab0 con 0x7f06400047c0 2023-10-16T15:33:38.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064000bc70 con 0x7f0654070b60 2023-10-16T15:33:38.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06537fe640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06400039a0 0x7f0640003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:38.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f0652ffd640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400047c0 0x7f0640004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:38.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06537fe640 1 -- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 msgr2=0x7f0640005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06537fe640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0654070b60 0x7f0640005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06537fe640 1 -- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400047c0 msgr2=0x7f0640004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06537fe640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400047c0 0x7f0640004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.306+0000 7f06537fe640 1 -- 172.21.15.29:0/4001513531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009400 con 0x7f06400039a0 2023-10-16T15:33:38.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f06537fe640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06400039a0 0x7f0640003dc0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f064400a5b0 tx=0x7f06400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6f431c5d8366b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644035620 con 0x7f06400039a0 2023-10-16T15:33:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644031070 con 0x7f06400039a0 2023-10-16T15:33:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f06527fc640 1 -- 172.21.15.29:0/4001513531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06440359e0 con 0x7f06400039a0 2023-10-16T15:33:38.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f0658d85640 1 -- 172.21.15.29:0/4001513531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06400039a0 msgr2=0x7f0640003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f0658d85640 1 --2- 172.21.15.29:0/4001513531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06400039a0 0x7f0640003dc0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f064400a5b0 tx=0x7f06400057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f0653fff640 1 -- 172.21.15.29:0/4001513531 reap_dead start 2023-10-16T15:33:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f0658d85640 1 -- 172.21.15.29:0/4001513531 shutdown_connections 2023-10-16T15:33:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.308+0000 7f0658d85640 1 -- 172.21.15.29:0/4001513531 >> 172.21.15.29:0/4001513531 conn(0x7f06540ce6a0 msgr2=0x7f06540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.309+0000 7f0658d85640 1 -- 172.21.15.29:0/4001513531 shutdown_connections 2023-10-16T15:33:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.309+0000 7f0658d85640 1 -- 172.21.15.29:0/4001513531 wait complete. 2023-10-16T15:33:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.310+0000 7f0658d85640 1 Processor -- start 2023-10-16T15:33:38.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 -- start start 2023-10-16T15:33:38.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400039a0 0x7f0654158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 0x7f06540b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06540c76b0 0x7f06540c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0654159190 con 0x7f06540c6180 2023-10-16T15:33:38.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c6610 con 0x7f06400039a0 2023-10-16T15:33:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.311+0000 7f0658d85640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c68f0 con 0x7f06540c76b0 2023-10-16T15:33:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 0x7f06540b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 0x7f06540b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42108/0 (socket says 172.21.15.29:42108) 2023-10-16T15:33:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 -- 172.21.15.29:0/650659323 learned_addr learned my addr 172.21.15.29:0/650659323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 -- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06540c76b0 msgr2=0x7f06540c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06540c76b0 0x7f06540c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 -- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400039a0 msgr2=0x7f0654158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400039a0 0x7f0654158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.312+0000 7f0652ffd640 1 -- 172.21.15.29:0/650659323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540c6bd0 con 0x7f06540c6180 2023-10-16T15:33:38.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.313+0000 7f0652ffd640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 0x7f06540b38b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f063c0108f0 tx=0x7f063c010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:38.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.313+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c00eb60 con 0x7f06540c6180 2023-10-16T15:33:38.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.313+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c00ecc0 con 0x7f06540c6180 2023-10-16T15:33:38.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.313+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c01c400 con 0x7f06540c6180 2023-10-16T15:33:38.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.313+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06540c6eb0 con 0x7f06540c6180 2023-10-16T15:33:38.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.313+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06540b3df0 con 0x7f06540c6180 2023-10-16T15:33:38.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.314+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065404e9b0 con 0x7f06540c6180 2023-10-16T15:33:38.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.315+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f063c0143f0 con 0x7f06540c6180 2023-10-16T15:33:38.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.317+0000 7f064bfff640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f063406afe0 0x7f063406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:38.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.317+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f063c089ca0 con 0x7f06540c6180 2023-10-16T15:33:38.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.317+0000 7f06537fe640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f063406afe0 0x7f063406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:38.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.318+0000 7f06537fe640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f063406afe0 0x7f063406d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0644037000 tx=0x7f0644009030 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:38.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.321+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f063c08e070 con 0x7f06540c6180 2023-10-16T15:33:38.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.518+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f06540b4620 con 0x7f06540c6180 2023-10-16T15:33:38.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.524+0000 7f064bfff640 1 -- 172.21.15.29:0/650659323 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f063c054d50 con 0x7f06540c6180 2023-10-16T15:33:38.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f063406afe0 msgr2=0x7f063406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f063406afe0 0x7f063406d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0644037000 tx=0x7f0644009030 comp rx=0 tx=0).stop 2023-10-16T15:33:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 msgr2=0x7f06540b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 0x7f06540b38b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f063c0108f0 tx=0x7f063c010db0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 shutdown_connections 2023-10-16T15:33:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f063406afe0 0x7f063406d4a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06540c6180 0x7f06540b38b0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06540c76b0 0x7f06540c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 --2- 172.21.15.29:0/650659323 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06400039a0 0x7f0654158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 >> 172.21.15.29:0/650659323 conn(0x7f06540ce6a0 msgr2=0x7f0654070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:38.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 shutdown_connections 2023-10-16T15:33:38.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:38.527+0000 7f0658d85640 1 -- 172.21.15.29:0/650659323 wait complete. 2023-10-16T15:33:38.550 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:33:38.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:33:38.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:33:38.794 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:33:38.794 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:33:38.795 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:38.861 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:33:38.861 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:33:38.862 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:38.920 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:33:38.921 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:33:38.921 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:38.979 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:33:38.979 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:33:38.980 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:39.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:33:39.039 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T15:33:39.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:39.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.270+0000 7fee5b94e640 1 Processor -- start 2023-10-16T15:33:39.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.271+0000 7fee5b94e640 1 -- start start 2023-10-16T15:33:39.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.272+0000 7fee5b94e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fee5415aaa0 0x7fee5415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.272+0000 7fee5b94e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5415ed60 con 0x7fee5415ae80 2023-10-16T15:33:39.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.272+0000 7fee5b94e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee540707e0 con 0x7fee540ce390 2023-10-16T15:33:39.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.272+0000 7fee5b94e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5406b810 con 0x7fee5415aaa0 2023-10-16T15:33:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.273+0000 7fee58ee9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fee5415ae80 0x7fee540cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54764/0 (socket says 172.21.15.29:54764) 2023-10-16T15:33:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.273+0000 7fee58ee9640 1 -- 172.21.15.29:0/468397574 learned_addr learned my addr 172.21.15.29:0/468397574 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.273+0000 7fee596ea640 1 --2- 172.21.15.29:0/468397574 >> v2:172.21.15.170:3300/0 conn(0x7fee5415aaa0 0x7fee5415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.273+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857689785 0 0) 0x7fee5415ed60 con 0x7fee5415ae80 2023-10-16T15:33:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.273+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee40003520 con 0x7fee5415ae80 2023-10-16T15:33:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453611390 0 0) 0x7fee40003520 con 0x7fee5415ae80 2023-10-16T15:33:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 >> v2:172.21.15.170:3300/0 conn(0x7fee5415aaa0 msgr2=0x7fee5415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 --2- 172.21.15.29:0/468397574 >> v2:172.21.15.170:3300/0 conn(0x7fee5415aaa0 0x7fee5415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 >> v1:172.21.15.170:6789/0 conn(0x7fee540ce390 legacy=0x7fee54070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5415ef90 con 0x7fee5415ae80 2023-10-16T15:33:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fee44001670 con 0x7fee5415ae80 2023-10-16T15:33:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 >> v1:172.21.15.29:6789/0 conn(0x7fee5415ae80 legacy=0x7fee540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40003970 0x7fee40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee400047c0 0x7fee40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee40005180 0x7fee4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5406b810 con 0x7fee40005180 2023-10-16T15:33:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4000c0d0 con 0x7fee400047c0 2023-10-16T15:33:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4000c290 con 0x7fee40003970 2023-10-16T15:33:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee596ea640 1 --2- 172.21.15.29:0/468397574 >> v2:172.21.15.170:3300/0 conn(0x7fee5415aaa0 0x7fee5415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.274+0000 7fee59eeb640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee40005180 0x7fee4000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee58ee9640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee400047c0 0x7fee40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee59eeb640 1 -- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40003970 msgr2=0x7fee40003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee59eeb640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40003970 0x7fee40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee59eeb640 1 -- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee400047c0 msgr2=0x7fee40004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee59eeb640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee400047c0 0x7fee40004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee59eeb640 1 -- 172.21.15.29:0/468397574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee40003520 con 0x7fee40005180 2023-10-16T15:33:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.275+0000 7fee58ee9640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee400047c0 0x7fee40004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.276+0000 7fee59eeb640 1 -- 172.21.15.29:0/468397574 reap_dead start 2023-10-16T15:33:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.276+0000 7fee59eeb640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee40005180 0x7fee4000b980 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001e50 tx=0x7fee400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e39add0a0b2c3f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.276+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c003b70 con 0x7fee40005180 2023-10-16T15:33:39.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.276+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c003d10 con 0x7fee40005180 2023-10-16T15:33:39.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.276+0000 7fee4bfff640 1 -- 172.21.15.29:0/468397574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c005310 con 0x7fee40005180 2023-10-16T15:33:39.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 -- 172.21.15.29:0/468397574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee40005180 msgr2=0x7fee4000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee40005180 0x7fee4000b980 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001e50 tx=0x7fee400057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 -- 172.21.15.29:0/468397574 shutdown_connections 2023-10-16T15:33:39.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 --2- 172.21.15.29:0/468397574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee40005180 0x7fee4000b980 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 -- 172.21.15.29:0/468397574 >> 172.21.15.29:0/468397574 conn(0x7fee540cb410 msgr2=0x7fee540cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:39.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 -- 172.21.15.29:0/468397574 shutdown_connections 2023-10-16T15:33:39.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.277+0000 7fee5b94e640 1 -- 172.21.15.29:0/468397574 wait complete. 2023-10-16T15:33:39.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.279+0000 7fee5b94e640 1 Processor -- start 2023-10-16T15:33:39.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.279+0000 7fee5b94e640 1 -- start start 2023-10-16T15:33:39.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee5b94e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40005180 0x7fee5406f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee5b94e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 0x7fee54155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee5b94e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee541f3e20 0x7fee54156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee5b94e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee541f3840 con 0x7fee541f39a0 2023-10-16T15:33:39.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee5b94e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee541f3560 con 0x7fee541f3e20 2023-10-16T15:33:39.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee5b94e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee541567f0 con 0x7fee40005180 2023-10-16T15:33:39.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee58ee9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 0x7fee54155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee58ee9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 0x7fee54155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42134/0 (socket says 172.21.15.29:42134) 2023-10-16T15:33:39.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.280+0000 7fee58ee9640 1 -- 172.21.15.29:0/3234074673 learned_addr learned my addr 172.21.15.29:0/3234074673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:39.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee596ea640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40005180 0x7fee5406f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee59eeb640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee541f3e20 0x7fee54156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee58ee9640 1 -- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40005180 msgr2=0x7fee5406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee58ee9640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40005180 0x7fee5406f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee58ee9640 1 -- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee541f3e20 msgr2=0x7fee54156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee58ee9640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee541f3e20 0x7fee54156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee58ee9640 1 -- 172.21.15.29:0/3234074673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee54156ad0 con 0x7fee541f39a0 2023-10-16T15:33:39.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee58ee9640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 0x7fee54155ac0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fee44000f60 tx=0x7fee4400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:39.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440074e0 con 0x7fee541f39a0 2023-10-16T15:33:39.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44007640 con 0x7fee541f39a0 2023-10-16T15:33:39.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440049e0 con 0x7fee541f39a0 2023-10-16T15:33:39.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.281+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee54156db0 con 0x7fee541f39a0 2023-10-16T15:33:39.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.282+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5406fde0 con 0x7fee541f39a0 2023-10-16T15:33:39.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.283+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee54156650 con 0x7fee541f39a0 2023-10-16T15:33:39.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.284+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee44004b40 con 0x7fee541f39a0 2023-10-16T15:33:39.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.285+0000 7fee49ffb640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fee3806af90 0x7fee3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.285+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fee44089060 con 0x7fee541f39a0 2023-10-16T15:33:39.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.285+0000 7fee596ea640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fee3806af90 0x7fee3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.287+0000 7fee596ea640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fee3806af90 0x7fee3806d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fee50006fd0 tx=0x7fee50008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:39.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.289+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee54156650 con 0x7fee541f39a0 2023-10-16T15:33:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.524+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee5406fde0 con 0x7fee541f39a0 2023-10-16T15:33:39.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.528+0000 7fee49ffb640 1 -- 172.21.15.29:0/3234074673 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fee5406fde0 con 0x7fee541f39a0 2023-10-16T15:33:39.529 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:39.530 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1034939100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1034939100},{"type":"v1","addr":"172.21.15.29:6835","nonce":1034939100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/3141038280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":3141038280},{"type":"v1","addr":"172.21.15.170:6835","nonce":3141038280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3321496230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":3321496230},{"type":"v1","addr":"172.21.15.170:6837","nonce":3321496230}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2746214286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2746214286},{"type":"v1","addr":"172.21.15.29:6833","nonce":2746214286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[]} 2023-10-16T15:33:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 77 2023-10-16T15:33:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fee3806af90 msgr2=0x7fee3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fee3806af90 0x7fee3806d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fee50006fd0 tx=0x7fee50008040 comp rx=0 tx=0).stop 2023-10-16T15:33:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 msgr2=0x7fee54155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 0x7fee54155ac0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fee44000f60 tx=0x7fee4400dd40 comp rx=0 tx=0).stop 2023-10-16T15:33:39.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 shutdown_connections 2023-10-16T15:33:39.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fee3806af90 0x7fee3806d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee541f39a0 0x7fee54155ac0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee541f3e20 0x7fee54156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 --2- 172.21.15.29:0/3234074673 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee40005180 0x7fee5406f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 >> 172.21.15.29:0/3234074673 conn(0x7fee540cb410 msgr2=0x7fee540b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.530+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 shutdown_connections 2023-10-16T15:33:39.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.531+0000 7fee5b94e640 1 -- 172.21.15.29:0/3234074673 wait complete. 2023-10-16T15:33:39.551 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T15:33:39.552 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T15:33:39.552 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:33:39.552 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:33:39.553 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T15:33:39.553 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T15:33:39.553 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:33:39.553 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:33:39.554 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T15:33:39.554 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T15:33:39.554 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:33:39.554 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:33:39.555 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T15:33:39.555 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T15:33:39.555 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:33:39.556 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:33:39.568 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:33:39.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T15:33:39.579 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:33:39.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T15:33:39.580 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:33:39.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T15:33:39.585 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:33:39.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T15:33:39.628 INFO:tasks.ceph.mds.a:Started 2023-10-16T15:33:39.629 DEBUG:teuthology.parallel:result is None 2023-10-16T15:33:39.632 INFO:tasks.ceph.mds.d:Started 2023-10-16T15:33:39.633 DEBUG:teuthology.parallel:result is None 2023-10-16T15:33:39.636 INFO:tasks.ceph.mds.c:Started 2023-10-16T15:33:39.636 DEBUG:teuthology.parallel:result is None 2023-10-16T15:33:39.640 INFO:tasks.ceph.mds.b:Started 2023-10-16T15:33:39.641 DEBUG:teuthology.parallel:result is None 2023-10-16T15:33:39.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T15:33:39.792 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:33:39.797 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:33:39.800 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:33:39.800 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T15:33:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.840+0000 7f4223f39640 1 Processor -- start 2023-10-16T15:33:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.840+0000 7f4223f39640 1 -- start start 2023-10-16T15:33:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.841+0000 7f4223f39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c15ade0 0x7f421c070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.841+0000 7f4223f39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f421c071430 0x7f421c0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.841+0000 7f4223f39640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c071cc0 con 0x7f421c0cb170 2023-10-16T15:33:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.841+0000 7f4223f39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421c0cb600 con 0x7f421c071430 2023-10-16T15:33:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.841+0000 7f4223f39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421c15bbc0 con 0x7f421c15ade0 2023-10-16T15:33:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42224d6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f421c0cb170 0x7f421c0c9480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54780/0 (socket says 172.21.15.29:54780) 2023-10-16T15:33:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42224d6640 1 -- 172.21.15.29:0/972521361 learned_addr learned my addr 172.21.15.29:0/972521361 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42214d4640 1 --2- 172.21.15.29:0/972521361 >> v2:172.21.15.29:3300/0 conn(0x7f421c071430 0x7f421c0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f4221cd5640 1 --2- 172.21.15.29:0/972521361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c15ade0 0x7f421c070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42214d4640 1 -- 172.21.15.29:0/972521361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c15ade0 msgr2=0x7f421c070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42214d4640 1 --2- 172.21.15.29:0/972521361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c15ade0 0x7f421c070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42214d4640 1 -- 172.21.15.29:0/972521361 >> v1:172.21.15.29:6789/0 conn(0x7f421c0cb170 legacy=0x7f421c0c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f42214d4640 1 -- 172.21.15.29:0/972521361 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0719e0 con 0x7f421c071430 2023-10-16T15:33:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.842+0000 7f4220cd3640 1 -- 172.21.15.29:0/972521361 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908189972 0 0) 0x7f4214001490 con 0x7f421c0cb170 2023-10-16T15:33:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.844+0000 7f42214d4640 1 --2- 172.21.15.29:0/972521361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c0cac30 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f420c0097a0 tx=0x7f420c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39847658e8da7f06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.844+0000 7f4220cd3640 1 -- 172.21.15.29:0/972521361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c02d040 con 0x7f421c071430 2023-10-16T15:33:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.844+0000 7f4220cd3640 1 -- 172.21.15.29:0/972521361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c031810 con 0x7f421c071430 2023-10-16T15:33:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.844+0000 7f4220cd3640 1 -- 172.21.15.29:0/972521361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c031ba0 con 0x7f421c071430 2023-10-16T15:33:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.844+0000 7f4223f39640 1 -- 172.21.15.29:0/972521361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 msgr2=0x7f421c0cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.844+0000 7f4223f39640 1 --2- 172.21.15.29:0/972521361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c0cac30 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f420c0097a0 tx=0x7f420c02b8b0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.845+0000 7f4223f39640 1 -- 172.21.15.29:0/972521361 shutdown_connections 2023-10-16T15:33:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.845+0000 7f4223f39640 1 --2- 172.21.15.29:0/972521361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c0cac30 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.845+0000 7f4223f39640 1 --2- 172.21.15.29:0/972521361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c15ade0 0x7f421c070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.845+0000 7f4223f39640 1 -- 172.21.15.29:0/972521361 >> 172.21.15.29:0/972521361 conn(0x7f421c0cb8b0 msgr2=0x7f421c0cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.845+0000 7f4223f39640 1 -- 172.21.15.29:0/972521361 shutdown_connections 2023-10-16T15:33:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.845+0000 7f4223f39640 1 -- 172.21.15.29:0/972521361 wait complete. 2023-10-16T15:33:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.846+0000 7f4223f39640 1 Processor -- start 2023-10-16T15:33:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.847+0000 7f4223f39640 1 -- start start 2023-10-16T15:33:39.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4223f39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4223f39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c0cb170 0x7f421c0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4223f39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f421c15ade0 0x7f421c0b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4223f39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c0b2d70 con 0x7f421c071430 2023-10-16T15:33:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4223f39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c0b2fc0 con 0x7f421c15ade0 2023-10-16T15:33:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4223f39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421c0b32a0 con 0x7f421c0cb170 2023-10-16T15:33:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42196/0 (socket says 172.21.15.29:42196) 2023-10-16T15:33:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 -- 172.21.15.29:0/2632796095 learned_addr learned my addr 172.21.15.29:0/2632796095 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f42214d4640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c0cb170 0x7f421c0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f42224d6640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f421c15ade0 0x7f421c0b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 -- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c0cb170 msgr2=0x7f421c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c0cb170 0x7f421c0b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 -- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f421c15ade0 msgr2=0x7f421c0b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f421c15ade0 0x7f421c0b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.848+0000 7f4221cd5640 1 -- 172.21.15.29:0/2632796095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c009400 con 0x7f421c071430 2023-10-16T15:33:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.849+0000 7f4221cd5640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c155b20 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f421800d5f0 tx=0x7f421800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.849+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180125f0 con 0x7f421c071430 2023-10-16T15:33:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.849+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4218012750 con 0x7f421c071430 2023-10-16T15:33:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.849+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218012b00 con 0x7f421c071430 2023-10-16T15:33:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.849+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c0b3870 con 0x7f421c071430 2023-10-16T15:33:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.849+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c156060 con 0x7f421c071430 2023-10-16T15:33:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.850+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c0b36d0 con 0x7f421c071430 2023-10-16T15:33:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.851+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4218012c60 con 0x7f421c071430 2023-10-16T15:33:39.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.852+0000 7f42127fc640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f420006af90 0x7f420006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:39.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.852+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4218089cd0 con 0x7f421c071430 2023-10-16T15:33:39.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.853+0000 7f42214d4640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f420006af90 0x7f420006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:39.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.853+0000 7f42214d4640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f420006af90 0x7f420006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f420c036000 tx=0x7f420c03c040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:39.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:39.857+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f421c0b36d0 con 0x7f421c071430 2023-10-16T15:33:40.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.063+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f421c0b36d0 con 0x7f421c071430 2023-10-16T15:33:40.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:33:40.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:33:40.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.549+0000 7f42127fc640 1 -- 172.21.15.29:0/2632796095 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v173) v1 ==== 97+0+0 (secure 0 0 0) 0x7f421c0b36d0 con 0x7f421c071430 2023-10-16T15:33:40.551 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T15:33:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f420006af90 msgr2=0x7f420006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f420006af90 0x7f420006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f420c036000 tx=0x7f420c03c040 comp rx=0 tx=0).stop 2023-10-16T15:33:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 msgr2=0x7f421c155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c155b20 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f421800d5f0 tx=0x7f421800dab0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 shutdown_connections 2023-10-16T15:33:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f420006af90 0x7f420006d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f421c15ade0 0x7f421c0b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421c0cb170 0x7f421c0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 --2- 172.21.15.29:0/2632796095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421c071430 0x7f421c155b20 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 >> 172.21.15.29:0/2632796095 conn(0x7f421c0cb8b0 msgr2=0x7f421c15f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 shutdown_connections 2023-10-16T15:33:40.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.553+0000 7f4223f39640 1 -- 172.21.15.29:0/2632796095 wait complete. 2023-10-16T15:33:40.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T15:33:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.806+0000 7f4baf47c640 1 Processor -- start 2023-10-16T15:33:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.806+0000 7f4baf47c640 1 -- start start 2023-10-16T15:33:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.808+0000 7f4baf47c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4ba815aaa0 0x7f4ba815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.808+0000 7f4baf47c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815ae80 0x7f4ba80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.808+0000 7f4baf47c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ba80ccaa0 0x7f4ba8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.808+0000 7f4baf47c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba806b810 con 0x7f4ba80ccaa0 2023-10-16T15:33:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.808+0000 7f4baf47c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba815ed60 con 0x7f4ba815aaa0 2023-10-16T15:33:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.808+0000 7f4baf47c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80707e0 con 0x7f4ba815ae80 2023-10-16T15:33:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ba80ccaa0 0x7f4ba8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ba80ccaa0 0x7f4ba8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42198/0 (socket says 172.21.15.29:42198) 2023-10-16T15:33:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 -- 172.21.15.29:0/2387325206 learned_addr learned my addr 172.21.15.29:0/2387325206 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4baca17640 1 --2- 172.21.15.29:0/2387325206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815ae80 0x7f4ba80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bad218640 1 --2- 172.21.15.29:0/2387325206 >> v2:172.21.15.170:3300/0 conn(0x7f4ba815aaa0 0x7f4ba815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 -- 172.21.15.29:0/2387325206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815ae80 msgr2=0x7f4ba80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 --2- 172.21.15.29:0/2387325206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815ae80 0x7f4ba80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 -- 172.21.15.29:0/2387325206 >> v2:172.21.15.170:3300/0 conn(0x7f4ba815aaa0 msgr2=0x7f4ba815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 --2- 172.21.15.29:0/2387325206 >> v2:172.21.15.170:3300/0 conn(0x7f4ba815aaa0 0x7f4ba815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.809+0000 7f4bada19640 1 -- 172.21.15.29:0/2387325206 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba815ef90 con 0x7f4ba80ccaa0 2023-10-16T15:33:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.810+0000 7f4bada19640 1 --2- 172.21.15.29:0/2387325206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba80ccaa0 0x7f4ba8070f50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80cd990 tx=0x7f4ba002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808064485870242a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4b9ffff640 1 -- 172.21.15.29:0/2387325206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba002ca80 con 0x7f4ba80ccaa0 2023-10-16T15:33:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4b9ffff640 1 -- 172.21.15.29:0/2387325206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba002cbe0 con 0x7f4ba80ccaa0 2023-10-16T15:33:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4b9ffff640 1 -- 172.21.15.29:0/2387325206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00343f0 con 0x7f4ba80ccaa0 2023-10-16T15:33:40.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 -- 172.21.15.29:0/2387325206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba80ccaa0 msgr2=0x7f4ba8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:40.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 --2- 172.21.15.29:0/2387325206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba80ccaa0 0x7f4ba8070f50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80cd990 tx=0x7f4ba002bee0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 -- 172.21.15.29:0/2387325206 shutdown_connections 2023-10-16T15:33:40.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 --2- 172.21.15.29:0/2387325206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba80ccaa0 0x7f4ba8070f50 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 --2- 172.21.15.29:0/2387325206 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815ae80 0x7f4ba80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 --2- 172.21.15.29:0/2387325206 >> v2:172.21.15.170:3300/0 conn(0x7f4ba815aaa0 0x7f4ba815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 -- 172.21.15.29:0/2387325206 >> 172.21.15.29:0/2387325206 conn(0x7f4ba80cb410 msgr2=0x7f4ba80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 -- 172.21.15.29:0/2387325206 shutdown_connections 2023-10-16T15:33:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.811+0000 7f4baf47c640 1 -- 172.21.15.29:0/2387325206 wait complete. 2023-10-16T15:33:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.813+0000 7f4baf47c640 1 Processor -- start 2023-10-16T15:33:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.813+0000 7f4baf47c640 1 -- start start 2023-10-16T15:33:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.814+0000 7f4baf47c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ba80ccaa0 0x7f4ba8150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.814+0000 7f4baf47c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815aaa0 0x7f4ba80c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.814+0000 7f4baf47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 0x7f4ba80c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.814+0000 7f4baf47c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba815cee0 con 0x7f4ba815ae80 2023-10-16T15:33:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.814+0000 7f4baf47c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8151390 con 0x7f4ba80ccaa0 2023-10-16T15:33:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.814+0000 7f4baf47c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba815c260 con 0x7f4ba815aaa0 2023-10-16T15:33:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 0x7f4ba80c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 0x7f4ba80c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42214/0 (socket says 172.21.15.29:42214) 2023-10-16T15:33:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 -- 172.21.15.29:0/3172661036 learned_addr learned my addr 172.21.15.29:0/3172661036 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4baca17640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815aaa0 0x7f4ba80c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 -- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815aaa0 msgr2=0x7f4ba80c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:40.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815aaa0 0x7f4ba80c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 -- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ba80ccaa0 msgr2=0x7f4ba8150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:40.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ba80ccaa0 0x7f4ba8150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:40.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 -- 172.21.15.29:0/3172661036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba0009b30 con 0x7f4ba815ae80 2023-10-16T15:33:40.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.815+0000 7f4bada19640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 0x7f4ba80c5a60 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0036040 tx=0x7f4ba0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.816+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0009510 con 0x7f4ba815ae80 2023-10-16T15:33:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.816+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba003e450 con 0x7f4ba815ae80 2023-10-16T15:33:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.816+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba003e800 con 0x7f4ba815ae80 2023-10-16T15:33:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.816+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba815c540 con 0x7f4ba815ae80 2023-10-16T15:33:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.816+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba815cb00 con 0x7f4ba815ae80 2023-10-16T15:33:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.817+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba804e160 con 0x7f4ba815ae80 2023-10-16T15:33:40.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.818+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4ba002cd50 con 0x7f4ba815ae80 2023-10-16T15:33:40.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.819+0000 7f4b9dffb640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4b8c06afc0 0x7f4b8c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:40.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.819+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f4ba00abf20 con 0x7f4ba815ae80 2023-10-16T15:33:40.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.820+0000 7f4bad218640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4b8c06afc0 0x7f4b8c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:40.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.821+0000 7f4bad218640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4b8c06afc0 0x7f4b8c06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400b7f0 tx=0x7f4ba4008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:40.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:40.825+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ba00771f0 con 0x7f4ba815ae80 2023-10-16T15:33:41.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.086+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4ba815ccc0 con 0x7f4ba815ae80 2023-10-16T15:33:41.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.087+0000 7f4b9dffb640 1 -- 172.21.15.29:0/3172661036 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4ba007a810 con 0x7f4ba815ae80 2023-10-16T15:33:41.088 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:41.089 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:33:41.089 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T15:33:41.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.089 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T15:33:41.090 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T15:33:41.090 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.090 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.090 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:33:41.090 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:33:41.091 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.091 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.091 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T15:33:41.092 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T15:33:41.092 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.092 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.092 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:33:41.092 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:33:41.093 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.093 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.093 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T15:33:41.094 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T15:33:41.094 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.094 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.094 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:33:41.094 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:33:41.095 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.095 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.095 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T15:33:41.096 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T15:33:41.096 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.096 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.096 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:33:41.096 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:33:41.096 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.097 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.097 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T15:33:41.097 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T15:33:41.098 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.098 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.098 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.098 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.098 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.099 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T15:33:41.099 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T15:33:41.099 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.100 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.100 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.100 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.100 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.101 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T15:33:41.101 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T15:33:41.101 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.101 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.101 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.102 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.102 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.103 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T15:33:41.103 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T15:33:41.103 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.103 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.103 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.103 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.104 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.104 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T15:33:41.104 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T15:33:41.105 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.105 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.105 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.105 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.105 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.106 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T15:33:41.106 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T15:33:41.106 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.107 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.107 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.107 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.108 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T15:33:41.108 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T15:33:41.108 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.108 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.108 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.108 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T15:33:41.109 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T15:33:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:33:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:33:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.110 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T15:33:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T15:33:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T15:33:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T15:33:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.112 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T15:33:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T15:33:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:33:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:33:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:33:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T15:33:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T15:33:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:33:41.115 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:33:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:33:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.116 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T15:33:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T15:33:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:33:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:33:41.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:33:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:33:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.118 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T15:33:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T15:33:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:33:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:33:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:33:41.119 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T15:33:41.120 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T15:33:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:33:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:33:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:33:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T15:33:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T15:33:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:33:41.122 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:33:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:33:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:33:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:41.123 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T15:33:41.124 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:33:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4b8c06afc0 msgr2=0x7f4b8c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4b8c06afc0 0x7f4b8c06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400b7f0 tx=0x7f4ba4008040 comp rx=0 tx=0).stop 2023-10-16T15:33:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 msgr2=0x7f4ba80c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 0x7f4ba80c5a60 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0036040 tx=0x7f4ba0004a40 comp rx=0 tx=0).stop 2023-10-16T15:33:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 shutdown_connections 2023-10-16T15:33:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4b8c06afc0 0x7f4b8c06d480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ba815ae80 0x7f4ba80c5a60 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ba815aaa0 0x7f4ba80c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 --2- 172.21.15.29:0/3172661036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ba80ccaa0 0x7f4ba8150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 >> 172.21.15.29:0/3172661036 conn(0x7f4ba80cb410 msgr2=0x7f4ba80b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 shutdown_connections 2023-10-16T15:33:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.090+0000 7f4baf47c640 1 -- 172.21.15.29:0/3172661036 wait complete. 2023-10-16T15:33:41.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:41.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.360+0000 7f7b93fff640 1 Processor -- start 2023-10-16T15:33:41.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.360+0000 7f7b93fff640 1 -- start start 2023-10-16T15:33:41.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b93fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7b9415c9f0 0x7f7b940c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b93fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 0x7f7b940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b93fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b94160a90 con 0x7f7b940c9230 2023-10-16T15:33:41.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b93fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b940cdec0 con 0x7f7b9415c9f0 2023-10-16T15:33:41.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b93fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940cd900 con 0x7f7b9415dde0 2023-10-16T15:33:41.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b927fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7b940c9230 0x7f7b940709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54788/0 (socket says 172.21.15.29:54788) 2023-10-16T15:33:41.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.362+0000 7f7b927fc640 1 -- 172.21.15.29:0/1153711232 learned_addr learned my addr 172.21.15.29:0/1153711232 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b937fe640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 0x7f7b940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244789381 0 0) 0x7f7b94160a90 con 0x7f7b940c9230 2023-10-16T15:33:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b92ffd640 1 --2- 172.21.15.29:0/1153711232 >> v2:172.21.15.170:3300/0 conn(0x7f7b9415c9f0 0x7f7b940c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b80003520 con 0x7f7b940c9230 2023-10-16T15:33:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2937010225 0 0) 0x7f7b80003520 con 0x7f7b940c9230 2023-10-16T15:33:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 msgr2=0x7f7b940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 0x7f7b940ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 >> v2:172.21.15.170:3300/0 conn(0x7f7b9415c9f0 msgr2=0x7f7b940c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 --2- 172.21.15.29:0/1153711232 >> v2:172.21.15.170:3300/0 conn(0x7f7b9415c9f0 0x7f7b940c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940cdbe0 con 0x7f7b940c9230 2023-10-16T15:33:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7b84001670 con 0x7f7b940c9230 2023-10-16T15:33:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 >> v1:172.21.15.29:6789/0 conn(0x7f7b940c9230 legacy=0x7f7b940709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 0x7f7b80003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b80004d00 0x7f7b80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940cd900 con 0x7f7b80003ef0 2023-10-16T15:33:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8000b980 con 0x7f7b80004d00 2023-10-16T15:33:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8000bb10 con 0x7f7b9415dde0 2023-10-16T15:33:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.363+0000 7f7b92ffd640 1 --2- 172.21.15.29:0/1153711232 >> v2:172.21.15.170:3300/0 conn(0x7f7b9415c9f0 0x7f7b940c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.364+0000 7f7b927fc640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.364+0000 7f7b927fc640 1 -- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 msgr2=0x7f7b80003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.364+0000 7f7b927fc640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415dde0 0x7f7b80003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.364+0000 7f7b927fc640 1 -- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b80004d00 msgr2=0x7f7b80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.364+0000 7f7b927fc640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b80004d00 0x7f7b80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.364+0000 7f7b927fc640 1 -- 172.21.15.29:0/1153711232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80003520 con 0x7f7b80003ef0 2023-10-16T15:33:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.365+0000 7f7b927fc640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b800047c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400c240 tx=0x7f7b800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44563367dd74d8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:41.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.365+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8402cb00 con 0x7f7b80003ef0 2023-10-16T15:33:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8402cc60 con 0x7f7b80003ef0 2023-10-16T15:33:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b91ffb640 1 -- 172.21.15.29:0/1153711232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84005af0 con 0x7f7b80003ef0 2023-10-16T15:33:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b93fff640 1 -- 172.21.15.29:0/1153711232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 msgr2=0x7f7b800047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b93fff640 1 --2- 172.21.15.29:0/1153711232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b800047c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400c240 tx=0x7f7b800057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b937fe640 1 -- 172.21.15.29:0/1153711232 reap_dead start 2023-10-16T15:33:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b93fff640 1 -- 172.21.15.29:0/1153711232 shutdown_connections 2023-10-16T15:33:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b93fff640 1 -- 172.21.15.29:0/1153711232 >> 172.21.15.29:0/1153711232 conn(0x7f7b940ce6a0 msgr2=0x7f7b940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b93fff640 1 -- 172.21.15.29:0/1153711232 shutdown_connections 2023-10-16T15:33:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.366+0000 7f7b93fff640 1 -- 172.21.15.29:0/1153711232 wait complete. 2023-10-16T15:33:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.368+0000 7f7b93fff640 1 Processor -- start 2023-10-16T15:33:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.368+0000 7f7b93fff640 1 -- start start 2023-10-16T15:33:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.369+0000 7f7b93fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b941f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.369+0000 7f7b93fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9406baf0 0x7f7b9406bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.369+0000 7f7b93fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415e280 0x7f7b9415e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.369+0000 7f7b93fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940c4a50 con 0x7f7b80003ef0 2023-10-16T15:33:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.369+0000 7f7b93fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940c4440 con 0x7f7b9406baf0 2023-10-16T15:33:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.369+0000 7f7b93fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9406c900 con 0x7f7b9415e280 2023-10-16T15:33:41.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b941f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b941f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42230/0 (socket says 172.21.15.29:42230) 2023-10-16T15:33:41.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 -- 172.21.15.29:0/105495365 learned_addr learned my addr 172.21.15.29:0/105495365 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:41.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b937fe640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415e280 0x7f7b9415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 -- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415e280 msgr2=0x7f7b9415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415e280 0x7f7b9415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 -- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9406baf0 msgr2=0x7f7b9406bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b927fc640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9406baf0 0x7f7b9406bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9406baf0 0x7f7b9406bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.370+0000 7f7b92ffd640 1 -- 172.21.15.29:0/105495365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940c46e0 con 0x7f7b80003ef0 2023-10-16T15:33:41.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.371+0000 7f7b92ffd640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b941f1ab0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e600 tx=0x7f7b7c00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:41.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.371+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c004230 con 0x7f7b80003ef0 2023-10-16T15:33:41.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.371+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c004390 con 0x7f7b80003ef0 2023-10-16T15:33:41.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.371+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c003850 con 0x7f7b80003ef0 2023-10-16T15:33:41.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.371+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9406cbe0 con 0x7f7b80003ef0 2023-10-16T15:33:41.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.371+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b940c50c0 con 0x7f7b80003ef0 2023-10-16T15:33:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.372+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9404e9b0 con 0x7f7b80003ef0 2023-10-16T15:33:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.373+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b7c0039b0 con 0x7f7b80003ef0 2023-10-16T15:33:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.374+0000 7f7b8b7fe640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b7406afe0 0x7f7b7406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.374+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f7b7c088fe0 con 0x7f7b80003ef0 2023-10-16T15:33:41.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.375+0000 7f7b927fc640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b7406afe0 0x7f7b7406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.376+0000 7f7b927fc640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b7406afe0 0x7f7b7406d4a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b84034000 tx=0x7f7b84007000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:41.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.378+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b7c0541a0 con 0x7f7b80003ef0 2023-10-16T15:33:41.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.576+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b940c50c0 con 0x7f7b80003ef0 2023-10-16T15:33:41.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.578+0000 7f7b8b7fe640 1 -- 172.21.15.29:0/105495365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f7b940c50c0 con 0x7f7b80003ef0 2023-10-16T15:33:41.579 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:41.580 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":175,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:40.564173+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:41.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.580+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b7406afe0 msgr2=0x7f7b7406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.580+0000 7f7b93fff640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b7406afe0 0x7f7b7406d4a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b84034000 tx=0x7f7b84007000 comp rx=0 tx=0).stop 2023-10-16T15:33:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 msgr2=0x7f7b941f1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b941f1ab0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e600 tx=0x7f7b7c00eac0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 shutdown_connections 2023-10-16T15:33:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f7b7406afe0 0x7f7b7406d4a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7b9415e280 0x7f7b9415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7b9406baf0 0x7f7b9406bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 --2- 172.21.15.29:0/105495365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b80003ef0 0x7f7b941f1ab0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 >> 172.21.15.29:0/105495365 conn(0x7f7b940ce6a0 msgr2=0x7f7b940cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 shutdown_connections 2023-10-16T15:33:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.581+0000 7f7b93fff640 1 -- 172.21.15.29:0/105495365 wait complete. 2023-10-16T15:33:41.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:33:41.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.831+0000 7f980ac1a640 1 Processor -- start 2023-10-16T15:33:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.831+0000 7f980ac1a640 1 -- start start 2023-10-16T15:33:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.832+0000 7f980ac1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f980415ad70 0x7f9804069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.832+0000 7f980ac1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98040caac0 0x7f980406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.832+0000 7f980ac1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98040cb030 con 0x7f98040c9ca0 2023-10-16T15:33:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.832+0000 7f980ac1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040c91c0 con 0x7f98040caac0 2023-10-16T15:33:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.832+0000 7f980ac1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040c94a0 con 0x7f980415ad70 2023-10-16T15:33:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98040caac0 0x7f980406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98040caac0 0x7f980406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42238/0 (socket says 172.21.15.29:42238) 2023-10-16T15:33:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 -- 172.21.15.29:0/1666266541 learned_addr learned my addr 172.21.15.29:0/1666266541 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98089b6640 1 --2- 172.21.15.29:0/1666266541 >> v2:172.21.15.170:3300/0 conn(0x7f980415ad70 0x7f9804069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 -- 172.21.15.29:0/1666266541 >> v2:172.21.15.170:3300/0 conn(0x7f980415ad70 msgr2=0x7f9804069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 --2- 172.21.15.29:0/1666266541 >> v2:172.21.15.170:3300/0 conn(0x7f980415ad70 0x7f9804069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 -- 172.21.15.29:0/1666266541 >> v1:172.21.15.170:6789/0 conn(0x7f98040c9ca0 legacy=0x7f98040ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98091b7640 1 -- 172.21.15.29:0/1666266541 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040c9780 con 0x7f98040caac0 2023-10-16T15:33:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.833+0000 7f98089b6640 1 --2- 172.21.15.29:0/1666266541 >> v2:172.21.15.170:3300/0 conn(0x7f980415ad70 0x7f9804069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:41.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.835+0000 7f98091b7640 1 --2- 172.21.15.29:0/1666266541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040caac0 0x7f980406a2e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f98040cf780 tx=0x7f97fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad3af5d13eeaba5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:41.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.835+0000 7f98037fe640 1 -- 172.21.15.29:0/1666266541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc031870 con 0x7f98040caac0 2023-10-16T15:33:41.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.835+0000 7f98037fe640 1 -- 172.21.15.29:0/1666266541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc0319d0 con 0x7f98040caac0 2023-10-16T15:33:41.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.835+0000 7f98037fe640 1 -- 172.21.15.29:0/1666266541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc031d60 con 0x7f98040caac0 2023-10-16T15:33:41.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 -- 172.21.15.29:0/1666266541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040caac0 msgr2=0x7f980406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 --2- 172.21.15.29:0/1666266541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040caac0 0x7f980406a2e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f98040cf780 tx=0x7f97fc02b910 comp rx=0 tx=0).stop 2023-10-16T15:33:41.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 -- 172.21.15.29:0/1666266541 shutdown_connections 2023-10-16T15:33:41.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 --2- 172.21.15.29:0/1666266541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040caac0 0x7f980406a2e0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 --2- 172.21.15.29:0/1666266541 >> v2:172.21.15.170:3300/0 conn(0x7f980415ad70 0x7f9804069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 -- 172.21.15.29:0/1666266541 >> 172.21.15.29:0/1666266541 conn(0x7f98040ce9b0 msgr2=0x7f98040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:41.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 -- 172.21.15.29:0/1666266541 shutdown_connections 2023-10-16T15:33:41.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.836+0000 7f980ac1a640 1 -- 172.21.15.29:0/1666266541 wait complete. 2023-10-16T15:33:41.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.838+0000 7f980ac1a640 1 Processor -- start 2023-10-16T15:33:41.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.838+0000 7f980ac1a640 1 -- start start 2023-10-16T15:33:41.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f980ac1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 0x7f98040c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f980ac1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98040caac0 0x7f98040b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f980ac1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980415ad70 0x7f98040b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f980ac1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980415cb30 con 0x7f98040c9ca0 2023-10-16T15:33:41.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f980ac1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980415bfb0 con 0x7f98040caac0 2023-10-16T15:33:41.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f980ac1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980415c290 con 0x7f980415ad70 2023-10-16T15:33:41.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f98089b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 0x7f98040c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f98089b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 0x7f98040c84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42244/0 (socket says 172.21.15.29:42244) 2023-10-16T15:33:41.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.839+0000 7f98089b6640 1 -- 172.21.15.29:0/2188404651 learned_addr learned my addr 172.21.15.29:0/2188404651 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:41.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98091b7640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980415ad70 0x7f98040b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98089b6640 1 -- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980415ad70 msgr2=0x7f98040b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98089b6640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980415ad70 0x7f98040b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98089b6640 1 -- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98040caac0 msgr2=0x7f98040b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:41.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f9803fff640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98040caac0 0x7f98040b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98089b6640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98040caac0 0x7f98040b2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:41.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98089b6640 1 -- 172.21.15.29:0/2188404651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0094d0 con 0x7f98040c9ca0 2023-10-16T15:33:41.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98089b6640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 0x7f98040c84d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f97f400c7f0 tx=0x7f97f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:41.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.840+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4007820 con 0x7f98040c9ca0 2023-10-16T15:33:41.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.841+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f4007980 con 0x7f98040c9ca0 2023-10-16T15:33:41.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.841+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4007cf0 con 0x7f98040c9ca0 2023-10-16T15:33:41.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.841+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980415c570 con 0x7f98040c9ca0 2023-10-16T15:33:41.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.841+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98040c5c20 con 0x7f98040c9ca0 2023-10-16T15:33:41.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.842+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98040c5a60 con 0x7f98040c9ca0 2023-10-16T15:33:41.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.843+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97f40113f0 con 0x7f98040c9ca0 2023-10-16T15:33:41.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.844+0000 7f98017fa640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f97f006af90 0x7f97f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:41.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.844+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f97f40897f0 con 0x7f98040c9ca0 2023-10-16T15:33:41.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.845+0000 7f9803fff640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f97f006af90 0x7f97f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:41.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.846+0000 7f9803fff640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f97f006af90 0x7f97f006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97f8007fd0 tx=0x7f97f8009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:41.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:41.849+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f97f4054a40 con 0x7f98040c9ca0 2023-10-16T15:33:42.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.044+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f980404e160 con 0x7f98040c9ca0 2023-10-16T15:33:42.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.045+0000 7f98017fa640 1 -- 172.21.15.29:0/2188404651 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v175) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97f401f0a0 con 0x7f98040c9ca0 2023-10-16T15:33:42.047 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:33:42.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f97f006af90 msgr2=0x7f97f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f97f006af90 0x7f97f006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97f8007fd0 tx=0x7f97f8009040 comp rx=0 tx=0).stop 2023-10-16T15:33:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 msgr2=0x7f98040c84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 0x7f98040c84d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f97f400c7f0 tx=0x7f97f400ccb0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 shutdown_connections 2023-10-16T15:33:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f97f006af90 0x7f97f006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f980415ad70 0x7f98040b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98040caac0 0x7f98040b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 --2- 172.21.15.29:0/2188404651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98040c9ca0 0x7f98040c84d0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 >> 172.21.15.29:0/2188404651 conn(0x7f98040ce9b0 msgr2=0x7f98040acc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 shutdown_connections 2023-10-16T15:33:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.048+0000 7f980ac1a640 1 -- 172.21.15.29:0/2188404651 wait complete. 2023-10-16T15:33:42.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:33:42.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.284+0000 7f211bfff640 1 Processor -- start 2023-10-16T15:33:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.284+0000 7f211bfff640 1 -- start start 2023-10-16T15:33:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 0x7f211c0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f211c071b90 0x7f211c161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c0cafd0 con 0x7f211c162480 2023-10-16T15:33:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211c0cbc50 con 0x7f211c071b90 2023-10-16T15:33:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0ccf10 con 0x7f211c161b60 2023-10-16T15:33:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f211c162480 0x7f211c0cd800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54790/0 (socket says 172.21.15.29:54790) 2023-10-16T15:33:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.286+0000 7f211b7fe640 1 -- 172.21.15.29:0/3159019918 learned_addr learned my addr 172.21.15.29:0/3159019918 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f211a7fc640 1 --2- 172.21.15.29:0/3159019918 >> v2:172.21.15.170:3300/0 conn(0x7f211c071b90 0x7f211c161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f211affd640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 0x7f211c0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718693727 0 0) 0x7f211c0cafd0 con 0x7f211c162480 2023-10-16T15:33:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2108003520 con 0x7f211c162480 2023-10-16T15:33:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113816296 0 0) 0x7f2108003520 con 0x7f211c162480 2023-10-16T15:33:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 msgr2=0x7f211c0cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 0x7f211c0cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 >> v2:172.21.15.170:3300/0 conn(0x7f211c071b90 msgr2=0x7f211c161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 --2- 172.21.15.29:0/3159019918 >> v2:172.21.15.170:3300/0 conn(0x7f211c071b90 0x7f211c161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0cb2b0 con 0x7f211c162480 2023-10-16T15:33:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2114001670 con 0x7f211c162480 2023-10-16T15:33:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 >> v1:172.21.15.29:6789/0 conn(0x7f211c162480 legacy=0x7f211c0cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f2108003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 0x7f21080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2108004d00 0x7f2108005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0cbc50 con 0x7f2108003970 2023-10-16T15:33:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0ccf10 con 0x7f2108004d00 2023-10-16T15:33:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.287+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210800b980 con 0x7f211c161b60 2023-10-16T15:33:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.288+0000 7f211affd640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f2108003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.288+0000 7f211affd640 1 -- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 msgr2=0x7f21080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:42.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.288+0000 7f211affd640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c161b60 0x7f21080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.288+0000 7f211affd640 1 -- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2108004d00 msgr2=0x7f2108005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:42.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.288+0000 7f211affd640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2108004d00 0x7f2108005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.288+0000 7f211affd640 1 -- 172.21.15.29:0/3159019918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108003520 con 0x7f2108003970 2023-10-16T15:33:42.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.289+0000 7f211affd640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f2108003d90 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f210c009ac0 tx=0x7f21080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f845b5c6702eefae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:42.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.289+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c02c710 con 0x7f2108003970 2023-10-16T15:33:42.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.289+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c02c870 con 0x7f2108003970 2023-10-16T15:33:42.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.289+0000 7f2119ffb640 1 -- 172.21.15.29:0/3159019918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c02cc00 con 0x7f2108003970 2023-10-16T15:33:42.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211bfff640 1 -- 172.21.15.29:0/3159019918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 msgr2=0x7f2108003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211bfff640 1 --2- 172.21.15.29:0/3159019918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f2108003d90 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f210c009ac0 tx=0x7f21080057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211b7fe640 1 -- 172.21.15.29:0/3159019918 reap_dead start 2023-10-16T15:33:42.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211bfff640 1 -- 172.21.15.29:0/3159019918 shutdown_connections 2023-10-16T15:33:42.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211bfff640 1 -- 172.21.15.29:0/3159019918 >> 172.21.15.29:0/3159019918 conn(0x7f211c065ed0 msgr2=0x7f211c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:42.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211bfff640 1 -- 172.21.15.29:0/3159019918 shutdown_connections 2023-10-16T15:33:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.290+0000 7f211bfff640 1 -- 172.21.15.29:0/3159019918 wait complete. 2023-10-16T15:33:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.292+0000 7f211bfff640 1 Processor -- start 2023-10-16T15:33:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.292+0000 7f211bfff640 1 -- start start 2023-10-16T15:33:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f211c0b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f211c0c59c0 0x7f211c0b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c0b1d10 0x7f211c0c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0b2190 con 0x7f2108003970 2023-10-16T15:33:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0c6b90 con 0x7f211c0c59c0 2023-10-16T15:33:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0ca5f0 con 0x7f211c0b1d10 2023-10-16T15:33:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f211c0b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f211c0b59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42264/0 (socket says 172.21.15.29:42264) 2023-10-16T15:33:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.293+0000 7f211affd640 1 -- 172.21.15.29:0/4109224310 learned_addr learned my addr 172.21.15.29:0/4109224310 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211affd640 1 -- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c0b1d10 msgr2=0x7f211c0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211affd640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c0b1d10 0x7f211c0c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211affd640 1 -- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f211c0c59c0 msgr2=0x7f211c0b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211affd640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f211c0c59c0 0x7f211c0b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211affd640 1 -- 172.21.15.29:0/4109224310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0c5f80 con 0x7f2108003970 2023-10-16T15:33:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211affd640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f211c0b59c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f210c037040 tx=0x7f210c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c003810 con 0x7f2108003970 2023-10-16T15:33:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c02de90 con 0x7f2108003970 2023-10-16T15:33:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c0c6260 con 0x7f2108003970 2023-10-16T15:33:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.294+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c003b90 con 0x7f2108003970 2023-10-16T15:33:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.295+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c0c6820 con 0x7f2108003970 2023-10-16T15:33:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.296+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c0b2130 con 0x7f2108003970 2023-10-16T15:33:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.297+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f210c003cf0 con 0x7f2108003970 2023-10-16T15:33:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.298+0000 7f2113fff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f20fc06af90 0x7f20fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.298+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f210c0ab3c0 con 0x7f2108003970 2023-10-16T15:33:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.299+0000 7f211a7fc640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f20fc06af90 0x7f20fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.300+0000 7f211a7fc640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f20fc06af90 0x7f20fc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f21040070b0 tx=0x7f210400b040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.302+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f211c0b2130 con 0x7f2108003970 2023-10-16T15:33:42.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.497+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f211c0b2130 con 0x7f2108003970 2023-10-16T15:33:42.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.498+0000 7f2113fff640 1 -- 172.21.15.29:0/4109224310 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v175) v1 ==== 84+0+65 (secure 0 0 0) 0x7f211c0b2130 con 0x7f2108003970 2023-10-16T15:33:42.500 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:42.500 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:33:42.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:42.500 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:33:42.501 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:33:42.501 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:42.501 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:33:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.500+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f20fc06af90 msgr2=0x7f20fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f20fc06af90 0x7f20fc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f21040070b0 tx=0x7f210400b040 comp rx=0 tx=0).stop 2023-10-16T15:33:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 msgr2=0x7f211c0b59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f211c0b59c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f210c037040 tx=0x7f210c004730 comp rx=0 tx=0).stop 2023-10-16T15:33:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 shutdown_connections 2023-10-16T15:33:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f20fc06af90 0x7f20fc06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f211c0b1d10 0x7f211c0c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f211c0c59c0 0x7f211c0b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 --2- 172.21.15.29:0/4109224310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2108003970 0x7f211c0b59c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 >> 172.21.15.29:0/4109224310 conn(0x7f211c065ed0 msgr2=0x7f211c0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 shutdown_connections 2023-10-16T15:33:42.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.501+0000 7f211bfff640 1 -- 172.21.15.29:0/4109224310 wait complete. 2023-10-16T15:33:42.522 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T15:33:42.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T15:33:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.743+0000 7feec9e39640 1 Processor -- start 2023-10-16T15:33:42.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.744+0000 7feec9e39640 1 -- start start 2023-10-16T15:33:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.745+0000 7feec9e39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feec415cf60 0x7feec406f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.745+0000 7feec9e39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7feec406f9d0 0x7feec40c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.745+0000 7feec9e39640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec40c9620 con 0x7feec40c9ff0 2023-10-16T15:33:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.745+0000 7feec9e39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec415d340 con 0x7feec415cf60 2023-10-16T15:33:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.745+0000 7feec9e39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec40cc860 con 0x7feec406f9d0 2023-10-16T15:33:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.745+0000 7feec37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feec415cf60 0x7feec406f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feec415cf60 0x7feec406f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42266/0 (socket says 172.21.15.29:42266) 2023-10-16T15:33:42.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec37fe640 1 -- 172.21.15.29:0/1036840181 learned_addr learned my addr 172.21.15.29:0/1036840181 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feebbfff640 1 --2- 172.21.15.29:0/1036840181 >> v2:172.21.15.170:3300/0 conn(0x7feec406f9d0 0x7feec40c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec37fe640 1 -- 172.21.15.29:0/1036840181 >> v2:172.21.15.170:3300/0 conn(0x7feec406f9d0 msgr2=0x7feec40c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec37fe640 1 --2- 172.21.15.29:0/1036840181 >> v2:172.21.15.170:3300/0 conn(0x7feec406f9d0 0x7feec40c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec2ffd640 1 -- 172.21.15.29:0/1036840181 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083308954 0 0) 0x7feec40c9620 con 0x7feec40c9ff0 2023-10-16T15:33:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec37fe640 1 -- 172.21.15.29:0/1036840181 >> v1:172.21.15.29:6789/0 conn(0x7feec40c9ff0 legacy=0x7feec40b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.746+0000 7feec37fe640 1 -- 172.21.15.29:0/1036840181 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec40ad4a0 con 0x7feec415cf60 2023-10-16T15:33:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.747+0000 7feec37fe640 1 --2- 172.21.15.29:0/1036840181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec415cf60 0x7feec406f490 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009870 tx=0x7feeb402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ed1ef49b65668f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec2ffd640 1 -- 172.21.15.29:0/1036840181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4030860 con 0x7feec415cf60 2023-10-16T15:33:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec2ffd640 1 -- 172.21.15.29:0/1036840181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb40309c0 con 0x7feec415cf60 2023-10-16T15:33:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec2ffd640 1 -- 172.21.15.29:0/1036840181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4030d50 con 0x7feec415cf60 2023-10-16T15:33:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 -- 172.21.15.29:0/1036840181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec415cf60 msgr2=0x7feec406f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 --2- 172.21.15.29:0/1036840181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec415cf60 0x7feec406f490 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009870 tx=0x7feeb402b9a0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 -- 172.21.15.29:0/1036840181 shutdown_connections 2023-10-16T15:33:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 --2- 172.21.15.29:0/1036840181 >> v2:172.21.15.170:3300/0 conn(0x7feec406f9d0 0x7feec40c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 --2- 172.21.15.29:0/1036840181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec415cf60 0x7feec406f490 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 -- 172.21.15.29:0/1036840181 >> 172.21.15.29:0/1036840181 conn(0x7feec40cb360 msgr2=0x7feec40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:42.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 -- 172.21.15.29:0/1036840181 shutdown_connections 2023-10-16T15:33:42.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.748+0000 7feec9e39640 1 -- 172.21.15.29:0/1036840181 wait complete. 2023-10-16T15:33:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.750+0000 7feec9e39640 1 Processor -- start 2023-10-16T15:33:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.750+0000 7feec9e39640 1 -- start start 2023-10-16T15:33:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.751+0000 7feec9e39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feec406f9d0 0x7feec40afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.751+0000 7feec9e39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 0x7feec40b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.751+0000 7feec9e39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec415cf60 0x7feec40b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.751+0000 7feec9e39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec40c6460 con 0x7feec40c9ff0 2023-10-16T15:33:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.751+0000 7feec9e39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec40b2ec0 con 0x7feec415cf60 2023-10-16T15:33:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.751+0000 7feec9e39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec415a4b0 con 0x7feec406f9d0 2023-10-16T15:33:42.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 0x7feec40b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 0x7feec40b2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42280/0 (socket says 172.21.15.29:42280) 2023-10-16T15:33:42.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 -- 172.21.15.29:0/2638857273 learned_addr learned my addr 172.21.15.29:0/2638857273 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:42.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feec37fe640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feec406f9d0 0x7feec40afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feec3fff640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec415cf60 0x7feec40b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 -- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feec406f9d0 msgr2=0x7feec40afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feec406f9d0 0x7feec40afcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 -- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec415cf60 msgr2=0x7feec40b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec415cf60 0x7feec40b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feebbfff640 1 -- 172.21.15.29:0/2638857273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb40094d0 con 0x7feec40c9ff0 2023-10-16T15:33:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.752+0000 7feec37fe640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feec406f9d0 0x7feec40afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.753+0000 7feebbfff640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 0x7feec40b2300 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7feeac00c820 tx=0x7feeac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.753+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac004020 con 0x7feec40c9ff0 2023-10-16T15:33:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.753+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac004180 con 0x7feec40c9ff0 2023-10-16T15:33:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.753+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac007830 con 0x7feec40c9ff0 2023-10-16T15:33:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.753+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec40c68f0 con 0x7feec40c9ff0 2023-10-16T15:33:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.753+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec40c5e10 con 0x7feec40c9ff0 2023-10-16T15:33:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.754+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec40c6610 con 0x7feec40c9ff0 2023-10-16T15:33:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.755+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feeac007990 con 0x7feec40c9ff0 2023-10-16T15:33:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.756+0000 7feec0ff9640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feea406b010 0x7feea406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.756+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4885+0+0 (secure 0 0 0) 0x7feeac089150 con 0x7feec40c9ff0 2023-10-16T15:33:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.757+0000 7feec37fe640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feea406b010 0x7feea406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.758+0000 7feec37fe640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feea406b010 0x7feea406d4d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feeb4036000 tx=0x7feeb403b040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.761+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feec40c6610 con 0x7feec40c9ff0 2023-10-16T15:33:42.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:42.959+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7feec40c6610 con 0x7feec40c9ff0 2023-10-16T15:33:43.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.573+0000 7feec0ff9640 1 -- 172.21.15.29:0/2638857273 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v176) v1 ==== 138+0+0 (secure 0 0 0) 0x7feec40c6610 con 0x7feec40c9ff0 2023-10-16T15:33:43.575 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T15:33:43.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.577+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feea406b010 msgr2=0x7feea406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:43.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.577+0000 7feec9e39640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feea406b010 0x7feea406d4d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feeb4036000 tx=0x7feeb403b040 comp rx=0 tx=0).stop 2023-10-16T15:33:43.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.577+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 msgr2=0x7feec40b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:43.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.577+0000 7feec9e39640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 0x7feec40b2300 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7feeac00c820 tx=0x7feeac00cce0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 shutdown_connections 2023-10-16T15:33:43.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feea406b010 0x7feea406d4d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec40c9ff0 0x7feec40b2300 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feec415cf60 0x7feec40b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 --2- 172.21.15.29:0/2638857273 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feec406f9d0 0x7feec40afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 >> 172.21.15.29:0/2638857273 conn(0x7feec40cb360 msgr2=0x7feec406c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:43.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 shutdown_connections 2023-10-16T15:33:43.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.578+0000 7feec9e39640 1 -- 172.21.15.29:0/2638857273 wait complete. 2023-10-16T15:33:43.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:33:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.822+0000 7fd0b9de2640 1 Processor -- start 2023-10-16T15:33:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.822+0000 7fd0b9de2640 1 -- start start 2023-10-16T15:33:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.824+0000 7fd0b9de2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b415ac70 0x7fd0b40cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.824+0000 7fd0b9de2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0b40b4160 0x7fd0b40cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.824+0000 7fd0b9de2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4069da0 0x7fd0b406a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.824+0000 7fd0b9de2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b406a770 con 0x7fd0b415ac70 2023-10-16T15:33:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.824+0000 7fd0b9de2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b40cc850 con 0x7fd0b40b4160 2023-10-16T15:33:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.824+0000 7fd0b9de2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40ccb30 con 0x7fd0b4069da0 2023-10-16T15:33:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b415ac70 0x7fd0b40cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0b415ac70 0x7fd0b40cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42284/0 (socket says 172.21.15.29:42284) 2023-10-16T15:33:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 -- 172.21.15.29:0/3050320546 learned_addr learned my addr 172.21.15.29:0/3050320546 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b3fff640 1 --2- 172.21.15.29:0/3050320546 >> v2:172.21.15.170:3300/0 conn(0x7fd0b40b4160 0x7fd0b40cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b95e1640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4069da0 0x7fd0b406a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 -- 172.21.15.29:0/3050320546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4069da0 msgr2=0x7fd0b406a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4069da0 0x7fd0b406a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 -- 172.21.15.29:0/3050320546 >> v2:172.21.15.170:3300/0 conn(0x7fd0b40b4160 msgr2=0x7fd0b40cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 --2- 172.21.15.29:0/3050320546 >> v2:172.21.15.170:3300/0 conn(0x7fd0b40b4160 0x7fd0b40cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b8de0640 1 -- 172.21.15.29:0/3050320546 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b40cce10 con 0x7fd0b415ac70 2023-10-16T15:33:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.825+0000 7fd0b95e1640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4069da0 0x7fd0b406a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.826+0000 7fd0b8de0640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40cc270 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4073ca0 tx=0x7fd0a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f64d51eac36e701 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b37fe640 1 -- 172.21.15.29:0/3050320546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8031870 con 0x7fd0b415ac70 2023-10-16T15:33:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b37fe640 1 -- 172.21.15.29:0/3050320546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a80319d0 con 0x7fd0b415ac70 2023-10-16T15:33:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b37fe640 1 -- 172.21.15.29:0/3050320546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8031d60 con 0x7fd0b415ac70 2023-10-16T15:33:43.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3050320546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 msgr2=0x7fd0b40cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:43.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40cc270 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4073ca0 tx=0x7fd0a802b910 comp rx=0 tx=0).stop 2023-10-16T15:33:43.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3050320546 shutdown_connections 2023-10-16T15:33:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b4069da0 0x7fd0b406a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3050320546 >> v2:172.21.15.170:3300/0 conn(0x7fd0b40b4160 0x7fd0b40cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3050320546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40cc270 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.827+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3050320546 >> 172.21.15.29:0/3050320546 conn(0x7fd0b40ce6a0 msgr2=0x7fd0b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.828+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3050320546 shutdown_connections 2023-10-16T15:33:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.828+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3050320546 wait complete. 2023-10-16T15:33:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.829+0000 7fd0b9de2640 1 Processor -- start 2023-10-16T15:33:43.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.829+0000 7fd0b9de2640 1 -- start start 2023-10-16T15:33:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.830+0000 7fd0b9de2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b4069da0 0x7fd0b40c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.830+0000 7fd0b9de2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b40b4160 0x7fd0b40ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.830+0000 7fd0b9de2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.830+0000 7fd0b9de2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40ca920 con 0x7fd0b415ac70 2023-10-16T15:33:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.830+0000 7fd0b9de2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40c9340 con 0x7fd0b4069da0 2023-10-16T15:33:43.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.830+0000 7fd0b9de2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40c9620 con 0x7fd0b40b4160 2023-10-16T15:33:43.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40c7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42288/0 (socket says 172.21.15.29:42288) 2023-10-16T15:33:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 -- 172.21.15.29:0/3632777205 learned_addr learned my addr 172.21.15.29:0/3632777205 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b3fff640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b40b4160 0x7fd0b40ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b8de0640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b4069da0 0x7fd0b40c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 -- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b40b4160 msgr2=0x7fd0b40ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:43.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b40b4160 0x7fd0b40ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 -- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b4069da0 msgr2=0x7fd0b40c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:43.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b4069da0 0x7fd0b40c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b95e1640 1 -- 172.21.15.29:0/3632777205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80094d0 con 0x7fd0b415ac70 2023-10-16T15:33:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.831+0000 7fd0b3fff640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b40b4160 0x7fd0b40ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.832+0000 7fd0b95e1640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40c7ea0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00cd10 tx=0x7fd0ac00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.832+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac00f680 con 0x7fd0b415ac70 2023-10-16T15:33:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.832+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac00f800 con 0x7fd0b415ac70 2023-10-16T15:33:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.832+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b40c9900 con 0x7fd0b415ac70 2023-10-16T15:33:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.832+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac00fbb0 con 0x7fd0b415ac70 2023-10-16T15:33:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.832+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b40c9ec0 con 0x7fd0b415ac70 2023-10-16T15:33:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.833+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b40ca0b0 con 0x7fd0b415ac70 2023-10-16T15:33:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.835+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd0ac00b040 con 0x7fd0b415ac70 2023-10-16T15:33:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.836+0000 7fd0b17fa640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fd09806af90 0x7fd09806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.836+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fd0ac08a040 con 0x7fd0b415ac70 2023-10-16T15:33:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.836+0000 7fd0b8de0640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fd09806af90 0x7fd09806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.838+0000 7fd0b8de0640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fd09806af90 0x7fd09806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8036000 tx=0x7fd0a803c040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:43.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:43.841+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0ac055120 con 0x7fd0b415ac70 2023-10-16T15:33:44.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.035+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd0b404e9b0 con 0x7fd0b415ac70 2023-10-16T15:33:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.592+0000 7fd0b17fa640 1 -- 172.21.15.29:0/3632777205 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v177) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd0ac058740 con 0x7fd0b415ac70 2023-10-16T15:33:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T15:33:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.598+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fd09806af90 msgr2=0x7fd09806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.598+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fd09806af90 0x7fd09806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8036000 tx=0x7fd0a803c040 comp rx=0 tx=0).stop 2023-10-16T15:33:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.598+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 msgr2=0x7fd0b40c7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.598+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40c7ea0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00cd10 tx=0x7fd0ac00a650 comp rx=0 tx=0).stop 2023-10-16T15:33:44.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.601+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 shutdown_connections 2023-10-16T15:33:44.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.601+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fd09806af90 0x7fd09806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.601+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b415ac70 0x7fd0b40c7ea0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.601+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0b40b4160 0x7fd0b40ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.601+0000 7fd0b9de2640 1 --2- 172.21.15.29:0/3632777205 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0b4069da0 0x7fd0b40c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.601+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 >> 172.21.15.29:0/3632777205 conn(0x7fd0b40ce6a0 msgr2=0x7fd0b40cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:44.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.604+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 shutdown_connections 2023-10-16T15:33:44.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.604+0000 7fd0b9de2640 1 -- 172.21.15.29:0/3632777205 wait complete. 2023-10-16T15:33:44.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T15:33:44.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.832+0000 7fae45705640 1 Processor -- start 2023-10-16T15:33:44.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.832+0000 7fae45705640 1 -- start start 2023-10-16T15:33:44.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.834+0000 7fae45705640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:44.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.834+0000 7fae45705640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae4006a4b0 0x7fae401650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:44.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.834+0000 7fae45705640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae400cb140 con 0x7fae400c9000 2023-10-16T15:33:44.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.834+0000 7fae45705640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae40161a50 con 0x7fae4006a4b0 2023-10-16T15:33:44.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.834+0000 7fae45705640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4006b030 con 0x7fae400addb0 2023-10-16T15:33:44.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae4006a4b0 0x7fae401650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:44.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae4006a4b0 0x7fae401650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46244/0 (socket says 172.21.15.29:46244) 2023-10-16T15:33:44.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3094613553 learned_addr learned my addr 172.21.15.29:0/3094613553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:44.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3effd640 1 --2- 172.21.15.29:0/3094613553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:44.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3094613553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 msgr2=0x7fae400caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:44.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 --2- 172.21.15.29:0/3094613553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3094613553 >> v1:172.21.15.170:6789/0 conn(0x7fae400c9000 legacy=0x7fae40069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:44.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.835+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3094613553 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4006b310 con 0x7fae4006a4b0 2023-10-16T15:33:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.836+0000 7fae3f7fe640 1 --2- 172.21.15.29:0/3094613553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae4006a4b0 0x7fae401650d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fae38009e50 tx=0x7fae3802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e37f597feda78a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae3dffb640 1 -- 172.21.15.29:0/3094613553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3802cb00 con 0x7fae4006a4b0 2023-10-16T15:33:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae3dffb640 1 -- 172.21.15.29:0/3094613553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3802cc60 con 0x7fae4006a4b0 2023-10-16T15:33:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae3dffb640 1 -- 172.21.15.29:0/3094613553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae380343f0 con 0x7fae4006a4b0 2023-10-16T15:33:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae45705640 1 -- 172.21.15.29:0/3094613553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae4006a4b0 msgr2=0x7fae401650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:44.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae45705640 1 --2- 172.21.15.29:0/3094613553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae4006a4b0 0x7fae401650d0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fae38009e50 tx=0x7fae3802bf60 comp rx=0 tx=0).stop 2023-10-16T15:33:44.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae45705640 1 -- 172.21.15.29:0/3094613553 shutdown_connections 2023-10-16T15:33:44.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae45705640 1 --2- 172.21.15.29:0/3094613553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae4006a4b0 0x7fae401650d0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.837+0000 7fae45705640 1 --2- 172.21.15.29:0/3094613553 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.838+0000 7fae45705640 1 -- 172.21.15.29:0/3094613553 >> 172.21.15.29:0/3094613553 conn(0x7fae400ce9b0 msgr2=0x7fae400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:44.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.838+0000 7fae45705640 1 -- 172.21.15.29:0/3094613553 shutdown_connections 2023-10-16T15:33:44.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.838+0000 7fae45705640 1 -- 172.21.15.29:0/3094613553 wait complete. 2023-10-16T15:33:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.839+0000 7fae45705640 1 Processor -- start 2023-10-16T15:33:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.839+0000 7fae45705640 1 -- start start 2023-10-16T15:33:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.840+0000 7fae45705640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4006a4b0 0x7fae400c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.840+0000 7fae45705640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.840+0000 7fae45705640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 0x7fae400b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae45705640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae401624c0 con 0x7fae400c9000 2023-10-16T15:33:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae45705640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae400b3e90 con 0x7fae4006a4b0 2023-10-16T15:33:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae45705640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae400b24a0 con 0x7fae400addb0 2023-10-16T15:33:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 0x7fae400b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 0x7fae400b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46252/0 (socket says 172.21.15.29:46252) 2023-10-16T15:33:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3209294664 learned_addr learned my addr 172.21.15.29:0/3209294664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3effd640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4006a4b0 0x7fae400c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 msgr2=0x7fae400b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3e7fc640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4006a4b0 msgr2=0x7fae400c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4006a4b0 0x7fae400c2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3f7fe640 1 -- 172.21.15.29:0/3209294664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009b30 con 0x7fae400c9000 2023-10-16T15:33:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.841+0000 7fae3e7fc640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.842+0000 7fae3f7fe640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 0x7fae400b1f60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fae38009e50 tx=0x7fae38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.842+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38009510 con 0x7fae400c9000 2023-10-16T15:33:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.842+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3803e450 con 0x7fae400c9000 2023-10-16T15:33:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.842+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae40161bd0 con 0x7fae400c9000 2023-10-16T15:33:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.842+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3803e800 con 0x7fae400c9000 2023-10-16T15:33:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.842+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae40161e70 con 0x7fae400c9000 2023-10-16T15:33:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.843+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae40162630 con 0x7fae400c9000 2023-10-16T15:33:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.846+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae3803e960 con 0x7fae400c9000 2023-10-16T15:33:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.846+0000 7fae277fe640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fae2c06af90 0x7fae2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:44.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.847+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fae380acb10 con 0x7fae400c9000 2023-10-16T15:33:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.847+0000 7fae3effd640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fae2c06af90 0x7fae2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.848+0000 7fae3effd640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fae2c06af90 0x7fae2c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fae30006fd0 tx=0x7fae30008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:44.850+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae38077c80 con 0x7fae400c9000 2023-10-16T15:33:45.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.079+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fae4004e160 con 0x7fae400c9000 2023-10-16T15:33:45.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.597+0000 7fae277fe640 1 -- 172.21.15.29:0/3209294664 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v80) v1 ==== 174+0+0 (secure 0 0 0) 0x7fae3807b2a0 con 0x7fae400c9000 2023-10-16T15:33:45.599 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 20 and data pool 21 2023-10-16T15:33:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.602+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fae2c06af90 msgr2=0x7fae2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.602+0000 7fae45705640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fae2c06af90 0x7fae2c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fae30006fd0 tx=0x7fae30008040 comp rx=0 tx=0).stop 2023-10-16T15:33:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.602+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 msgr2=0x7fae400b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:45.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.602+0000 7fae45705640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 0x7fae400b1f60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fae38009e50 tx=0x7fae38004a40 comp rx=0 tx=0).stop 2023-10-16T15:33:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.605+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 shutdown_connections 2023-10-16T15:33:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.605+0000 7fae45705640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fae2c06af90 0x7fae2c06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.605+0000 7fae45705640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae400c9000 0x7fae400b1f60 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.605+0000 7fae45705640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae400addb0 0x7fae400b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.605+0000 7fae45705640 1 --2- 172.21.15.29:0/3209294664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae4006a4b0 0x7fae400c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.605+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 >> 172.21.15.29:0/3209294664 conn(0x7fae400ce9b0 msgr2=0x7fae400aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.608+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 shutdown_connections 2023-10-16T15:33:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.609+0000 7fae45705640 1 -- 172.21.15.29:0/3209294664 wait complete. 2023-10-16T15:33:45.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:45.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.831+0000 7f870b34c640 1 Processor -- start 2023-10-16T15:33:45.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.832+0000 7f870b34c640 1 -- start start 2023-10-16T15:33:45.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f870b34c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f87041615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f870b34c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f870415d3f0 0x7f870415d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f870b34c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870415e3e0 con 0x7f87040cd0c0 2023-10-16T15:33:45.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f870b34c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87040cd4a0 con 0x7f870415d3f0 2023-10-16T15:33:45.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f870b34c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870415e100 con 0x7f87040b0b20 2023-10-16T15:33:45.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f87090e8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f87040cd0c0 0x7f87040b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44814/0 (socket says 172.21.15.29:44814) 2023-10-16T15:33:45.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.833+0000 7f87090e8640 1 -- 172.21.15.29:0/2755609285 learned_addr learned my addr 172.21.15.29:0/2755609285 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:45.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549857495 0 0) 0x7f870415e3e0 con 0x7f87040cd0c0 2023-10-16T15:33:45.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f87088e7640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f87041615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0003520 con 0x7f87040cd0c0 2023-10-16T15:33:45.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f87098e9640 1 --2- 172.21.15.29:0/2755609285 >> v2:172.21.15.170:3300/0 conn(0x7f870415d3f0 0x7f870415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870034787 0 0) 0x7f86f0003520 con 0x7f87040cd0c0 2023-10-16T15:33:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 msgr2=0x7f87041615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f87041615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 >> v2:172.21.15.170:3300/0 conn(0x7f870415d3f0 msgr2=0x7f870415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 --2- 172.21.15.29:0/2755609285 >> v2:172.21.15.170:3300/0 conn(0x7f870415d3f0 0x7f870415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8704160b10 con 0x7f87040cd0c0 2023-10-16T15:33:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8700001670 con 0x7f87040cd0c0 2023-10-16T15:33:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f87098e9640 1 --2- 172.21.15.29:0/2755609285 >> v2:172.21.15.170:3300/0 conn(0x7f870415d3f0 0x7f870415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 >> v1:172.21.15.29:6789/0 conn(0x7f87040cd0c0 legacy=0x7f87040b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0003970 0x7f86f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f86f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f86f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87040cd4a0 con 0x7f86f0004d00 2023-10-16T15:33:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870415e100 con 0x7f86f0003970 2023-10-16T15:33:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.834+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f000bb10 con 0x7f87040b0b20 2023-10-16T15:33:45.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87098e9640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f86f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87088e7640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f86f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87090e8640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0003970 0x7f86f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87098e9640 1 -- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 msgr2=0x7f86f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87098e9640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f86f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87098e9640 1 -- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0003970 msgr2=0x7f86f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87098e9640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0003970 0x7f86f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87098e9640 1 -- 172.21.15.29:0/2755609285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0003520 con 0x7f86f0004d00 2023-10-16T15:33:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87090e8640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f86f0003970 0x7f86f0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.835+0000 7f87088e7640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87040b0b20 0x7f86f00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.836+0000 7f87098e9640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f86f0005180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0073e0 tx=0x7f86f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809ba4ebd2647e09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc02cdc0 con 0x7f86f0004d00 2023-10-16T15:33:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc034440 con 0x7f86f0004d00 2023-10-16T15:33:45.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f86fbfff640 1 -- 172.21.15.29:0/2755609285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc0347b0 con 0x7f86f0004d00 2023-10-16T15:33:45.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f870b34c640 1 -- 172.21.15.29:0/2755609285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 msgr2=0x7f86f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f870b34c640 1 --2- 172.21.15.29:0/2755609285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f86f0005180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0073e0 tx=0x7f86f00057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f87098e9640 1 -- 172.21.15.29:0/2755609285 reap_dead start 2023-10-16T15:33:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f870b34c640 1 -- 172.21.15.29:0/2755609285 shutdown_connections 2023-10-16T15:33:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.837+0000 7f870b34c640 1 -- 172.21.15.29:0/2755609285 >> 172.21.15.29:0/2755609285 conn(0x7f87040cb410 msgr2=0x7f87040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.838+0000 7f870b34c640 1 -- 172.21.15.29:0/2755609285 shutdown_connections 2023-10-16T15:33:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.838+0000 7f870b34c640 1 -- 172.21.15.29:0/2755609285 wait complete. 2023-10-16T15:33:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.839+0000 7f870b34c640 1 Processor -- start 2023-10-16T15:33:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.839+0000 7f870b34c640 1 -- start start 2023-10-16T15:33:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.840+0000 7f870b34c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f870414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.840+0000 7f870b34c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8704156c20 0x7f87040b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.840+0000 7f870b34c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f870406bd20 0x7f870406c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.840+0000 7f870b34c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87040c6230 con 0x7f86f0004d00 2023-10-16T15:33:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.840+0000 7f870b34c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870406c6e0 con 0x7f870406bd20 2023-10-16T15:33:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.840+0000 7f870b34c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87040c64d0 con 0x7f8704156c20 2023-10-16T15:33:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f870414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f870414cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46258/0 (socket says 172.21.15.29:46258) 2023-10-16T15:33:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 -- 172.21.15.29:0/636370841 learned_addr learned my addr 172.21.15.29:0/636370841 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 -- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8704156c20 msgr2=0x7f87040b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8704156c20 0x7f87040b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 -- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f870406bd20 msgr2=0x7f870406c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f870406bd20 0x7f870406c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 -- 172.21.15.29:0/636370841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040b3fc0 con 0x7f86f0004d00 2023-10-16T15:33:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.841+0000 7f87090e8640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f870414cbe0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f870000d830 tx=0x7f870000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.842+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000074e0 con 0x7f86f0004d00 2023-10-16T15:33:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.842+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700007640 con 0x7f86f0004d00 2023-10-16T15:33:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.842+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87040b42a0 con 0x7f86f0004d00 2023-10-16T15:33:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.842+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000049e0 con 0x7f86f0004d00 2023-10-16T15:33:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.842+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870414d120 con 0x7f86f0004d00 2023-10-16T15:33:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.843+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870404e160 con 0x7f86f0004d00 2023-10-16T15:33:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.844+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8700004b40 con 0x7f86f0004d00 2023-10-16T15:33:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.845+0000 7f86f9ffb640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86e806af90 0x7f86e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.845+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f87000895e0 con 0x7f86f0004d00 2023-10-16T15:33:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.845+0000 7f87088e7640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86e806af90 0x7f86e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.847+0000 7f87088e7640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86e806af90 0x7f86e806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8704156bd0 tx=0x7f86f4009350 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:45.849+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87000543b0 con 0x7f86f0004d00 2023-10-16T15:33:46.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.045+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87040c8ca0 con 0x7f86f0004d00 2023-10-16T15:33:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.049+0000 7f86f9ffb640 1 -- 172.21.15.29:0/636370841 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16636 (secure 0 0 0) 0x7f87000579d0 con 0x7f86f0004d00 2023-10-16T15:33:46.051 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:46.051 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":178,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:45.081033+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.051+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86e806af90 msgr2=0x7f86e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.051+0000 7f870b34c640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86e806af90 0x7f86e806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8704156bd0 tx=0x7f86f4009350 comp rx=0 tx=0).stop 2023-10-16T15:33:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.051+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 msgr2=0x7f870414cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.051+0000 7f870b34c640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f870414cbe0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f870000d830 tx=0x7f870000dcf0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 shutdown_connections 2023-10-16T15:33:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f86e806af90 0x7f86e806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f870406bd20 0x7f870406c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8704156c20 0x7f87040b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 --2- 172.21.15.29:0/636370841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86f0004d00 0x7f870414cbe0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 >> 172.21.15.29:0/636370841 conn(0x7f87040cb410 msgr2=0x7f870415b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 shutdown_connections 2023-10-16T15:33:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.052+0000 7f870b34c640 1 -- 172.21.15.29:0/636370841 wait complete. 2023-10-16T15:33:46.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:46.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.298+0000 7f823ffff640 1 Processor -- start 2023-10-16T15:33:46.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.298+0000 7f823ffff640 1 -- start start 2023-10-16T15:33:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.299+0000 7f823ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82400c9060 0x7f824006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.299+0000 7f823ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 0x7f824006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.299+0000 7f823ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82400ca560 con 0x7f82400c9440 2023-10-16T15:33:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.299+0000 7f823ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82400c9950 con 0x7f82400c9060 2023-10-16T15:33:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.299+0000 7f823ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400c9c30 con 0x7f824006c9e0 2023-10-16T15:33:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.300+0000 7f823e7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 0x7f824006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.300+0000 7f823e7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 0x7f824006f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:48944/0 (socket says 172.21.15.29:48944) 2023-10-16T15:33:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.300+0000 7f823effd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82400c9060 0x7f824006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.300+0000 7f823e7fc640 1 -- 172.21.15.29:0/110186725 learned_addr learned my addr 172.21.15.29:0/110186725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:46.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.300+0000 7f823f7fe640 1 --1- 172.21.15.29:0/110186725 >> v1:172.21.15.170:6789/0 conn(0x7f82400c9440 0x7f8240164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:46846/0 (socket says 172.21.15.29:46846) 2023-10-16T15:33:46.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.301+0000 7f823effd640 1 -- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 msgr2=0x7f824006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.301+0000 7f823effd640 1 --2- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 0x7f824006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.301+0000 7f823effd640 1 -- 172.21.15.29:0/110186725 >> v1:172.21.15.170:6789/0 conn(0x7f82400c9440 legacy=0x7f8240164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.301+0000 7f823effd640 1 -- 172.21.15.29:0/110186725 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400c9f10 con 0x7f82400c9060 2023-10-16T15:33:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.301+0000 7f823e7fc640 1 --2- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 0x7f824006f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:33:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.302+0000 7f823effd640 1 --2- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9060 0x7f824006f440 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f82300097a0 tx=0x7f823002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1f8b1d274e58637 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.303+0000 7f823dffb640 1 -- 172.21.15.29:0/110186725 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823002d040 con 0x7f82400c9060 2023-10-16T15:33:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.303+0000 7f823dffb640 1 -- 172.21.15.29:0/110186725 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230031810 con 0x7f82400c9060 2023-10-16T15:33:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.303+0000 7f823dffb640 1 -- 172.21.15.29:0/110186725 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230031ba0 con 0x7f82400c9060 2023-10-16T15:33:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 -- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9060 msgr2=0x7f824006f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 --2- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9060 0x7f824006f440 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f82300097a0 tx=0x7f823002b8b0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 -- 172.21.15.29:0/110186725 shutdown_connections 2023-10-16T15:33:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 --2- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f824006c9e0 0x7f824006f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 --2- 172.21.15.29:0/110186725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9060 0x7f824006f440 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 -- 172.21.15.29:0/110186725 >> 172.21.15.29:0/110186725 conn(0x7f82400ce6a0 msgr2=0x7f82400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 -- 172.21.15.29:0/110186725 shutdown_connections 2023-10-16T15:33:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.304+0000 7f823ffff640 1 -- 172.21.15.29:0/110186725 wait complete. 2023-10-16T15:33:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.306+0000 7f823ffff640 1 Processor -- start 2023-10-16T15:33:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.306+0000 7f823ffff640 1 -- start start 2023-10-16T15:33:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 0x7f82400c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400c9060 0x7f82400c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9440 0x7f82400b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82400c8b10 con 0x7f824006c9e0 2023-10-16T15:33:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82400ae1a0 con 0x7f82400c9440 2023-10-16T15:33:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400c8870 con 0x7f82400c9060 2023-10-16T15:33:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 0x7f82400c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 0x7f82400c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46272/0 (socket says 172.21.15.29:46272) 2023-10-16T15:33:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.307+0000 7f823effd640 1 -- 172.21.15.29:0/144340790 learned_addr learned my addr 172.21.15.29:0/144340790 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823e7fc640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400c9060 0x7f82400c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823f7fe640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9440 0x7f82400b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823effd640 1 -- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400c9060 msgr2=0x7f82400c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823effd640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400c9060 0x7f82400c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823effd640 1 -- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9440 msgr2=0x7f82400b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823effd640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9440 0x7f82400b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823effd640 1 -- 172.21.15.29:0/144340790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228009b30 con 0x7f824006c9e0 2023-10-16T15:33:46.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.308+0000 7f823effd640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 0x7f82400c8330 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f823002b8f0 tx=0x7f82300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.309+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230006e10 con 0x7f824006c9e0 2023-10-16T15:33:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.309+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823003ad60 con 0x7f824006c9e0 2023-10-16T15:33:46.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.309+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230044630 con 0x7f824006c9e0 2023-10-16T15:33:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.309+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8230009400 con 0x7f824006c9e0 2023-10-16T15:33:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.309+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82400b2640 con 0x7f824006c9e0 2023-10-16T15:33:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.310+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82400b1620 con 0x7f824006c9e0 2023-10-16T15:33:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.311+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f823004d450 con 0x7f824006c9e0 2023-10-16T15:33:46.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.312+0000 7f8237fff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f822006afe0 0x7f822006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.312+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f82300aba20 con 0x7f824006c9e0 2023-10-16T15:33:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.313+0000 7f823e7fc640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f822006afe0 0x7f822006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.314+0000 7f823e7fc640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f822006afe0 0x7f822006d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f82400c41f0 tx=0x7f822800b040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:46.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.316+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8230076970 con 0x7f824006c9e0 2023-10-16T15:33:46.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.507+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f824004e9b0 con 0x7f824006c9e0 2023-10-16T15:33:46.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.509+0000 7f8237fff640 1 -- 172.21.15.29:0/144340790 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16636 (secure 0 0 0) 0x7f8230035090 con 0x7f824006c9e0 2023-10-16T15:33:46.510 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:46.511 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":178,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:45.081033+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f822006afe0 msgr2=0x7f822006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f822006afe0 0x7f822006d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f82400c41f0 tx=0x7f822800b040 comp rx=0 tx=0).stop 2023-10-16T15:33:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 msgr2=0x7f82400c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 0x7f82400c8330 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f823002b8f0 tx=0x7f82300046c0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 shutdown_connections 2023-10-16T15:33:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f822006afe0 0x7f822006d4a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82400c9440 0x7f82400b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82400c9060 0x7f82400c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 --2- 172.21.15.29:0/144340790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f824006c9e0 0x7f82400c8330 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.511+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 >> 172.21.15.29:0/144340790 conn(0x7f82400ce6a0 msgr2=0x7f8240160c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:46.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.512+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 shutdown_connections 2023-10-16T15:33:46.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.512+0000 7f823ffff640 1 -- 172.21.15.29:0/144340790 wait complete. 2023-10-16T15:33:46.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T15:33:46.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.745+0000 7fc392a7c640 1 Processor -- start 2023-10-16T15:33:46.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.745+0000 7fc392a7c640 1 -- start start 2023-10-16T15:33:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc392a7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 0x7fc38c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc392a7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0cb030 con 0x7fc38c06a4b0 2023-10-16T15:33:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc392a7c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0c91c0 con 0x7fc38c15ad70 2023-10-16T15:33:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc392a7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0c94a0 con 0x7fc38c0caac0 2023-10-16T15:33:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc38bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc38c06a4b0 0x7fc38c0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44830/0 (socket says 172.21.15.29:44830) 2023-10-16T15:33:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc38bfff640 1 -- 172.21.15.29:0/3823427956 learned_addr learned my addr 172.21.15.29:0/3823427956 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.747+0000 7fc391019640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 0x7fc38c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921206224 0 0) 0x7fc38c0cb030 con 0x7fc38c06a4b0 2023-10-16T15:33:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc374003520 con 0x7fc38c06a4b0 2023-10-16T15:33:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474335982 0 0) 0x7fc374003520 con 0x7fc38c06a4b0 2023-10-16T15:33:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 msgr2=0x7fc38c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 0x7fc38c0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 >> v1:172.21.15.170:6789/0 conn(0x7fc38c15ad70 legacy=0x7fc38c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0c9780 con 0x7fc38c06a4b0 2023-10-16T15:33:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc380001670 con 0x7fc38c06a4b0 2023-10-16T15:33:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 >> v1:172.21.15.29:6789/0 conn(0x7fc38c06a4b0 legacy=0x7fc38c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 0x7fc3740039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc374003f70 0x7fc3740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc374004d00 0x7fc374005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0c94a0 con 0x7fc374004d00 2023-10-16T15:33:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37400bac0 con 0x7fc374003f70 2023-10-16T15:33:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37400bc80 con 0x7fc38c0caac0 2023-10-16T15:33:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.748+0000 7fc391019640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc374004d00 0x7fc374005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc38bfff640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc374003f70 0x7fc3740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc390818640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 0x7fc3740039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc391019640 1 -- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 msgr2=0x7fc3740039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc391019640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc38c0caac0 0x7fc3740039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc391019640 1 -- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc374003f70 msgr2=0x7fc3740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:46.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc391019640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc374003f70 0x7fc3740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.749+0000 7fc391019640 1 -- 172.21.15.29:0/3823427956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374003520 con 0x7fc374004d00 2023-10-16T15:33:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.750+0000 7fc391019640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc374004d00 0x7fc374005180 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc384009ac0 tx=0x7fc3740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d067cc2adb7e6bb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.750+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38402c6b0 con 0x7fc374004d00 2023-10-16T15:33:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.750+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38402c810 con 0x7fc374004d00 2023-10-16T15:33:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.750+0000 7fc38b7fe640 1 -- 172.21.15.29:0/3823427956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38402cba0 con 0x7fc374004d00 2023-10-16T15:33:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc392a7c640 1 -- 172.21.15.29:0/3823427956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc374004d00 msgr2=0x7fc374005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3823427956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc374004d00 0x7fc374005180 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc384009ac0 tx=0x7fc3740057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc391019640 1 -- 172.21.15.29:0/3823427956 reap_dead start 2023-10-16T15:33:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc392a7c640 1 -- 172.21.15.29:0/3823427956 shutdown_connections 2023-10-16T15:33:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc392a7c640 1 -- 172.21.15.29:0/3823427956 >> 172.21.15.29:0/3823427956 conn(0x7fc38c0ce9b0 msgr2=0x7fc38c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc392a7c640 1 -- 172.21.15.29:0/3823427956 shutdown_connections 2023-10-16T15:33:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.751+0000 7fc392a7c640 1 -- 172.21.15.29:0/3823427956 wait complete. 2023-10-16T15:33:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.753+0000 7fc392a7c640 1 Processor -- start 2023-10-16T15:33:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.753+0000 7fc392a7c640 1 -- start start 2023-10-16T15:33:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc392a7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc374004d00 0x7fc38c0a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc392a7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc38c0c24a0 0x7fc38c0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc392a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 0x7fc38c157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc392a7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c158400 con 0x7fc38c157a40 2023-10-16T15:33:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc392a7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0a93a0 con 0x7fc38c0c24a0 2023-10-16T15:33:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc392a7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0c3b70 con 0x7fc374004d00 2023-10-16T15:33:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc391019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 0x7fc38c157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc391019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 0x7fc38c157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46296/0 (socket says 172.21.15.29:46296) 2023-10-16T15:33:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.754+0000 7fc391019640 1 -- 172.21.15.29:0/3655311992 learned_addr learned my addr 172.21.15.29:0/3655311992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc390818640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc374004d00 0x7fc38c0a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc38bfff640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc38c0c24a0 0x7fc38c0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc391019640 1 -- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc374004d00 msgr2=0x7fc38c0a8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc391019640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc374004d00 0x7fc38c0a8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc391019640 1 -- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc38c0c24a0 msgr2=0x7fc38c0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc391019640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc38c0c24a0 0x7fc38c0c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc391019640 1 -- 172.21.15.29:0/3655311992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0c2f60 con 0x7fc38c157a40 2023-10-16T15:33:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc391019640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 0x7fc38c157ec0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc384009ac0 tx=0x7fc384004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384003810 con 0x7fc38c157a40 2023-10-16T15:33:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384003970 con 0x7fc38c157a40 2023-10-16T15:33:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384003ce0 con 0x7fc38c157a40 2023-10-16T15:33:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c0c3240 con 0x7fc38c157a40 2023-10-16T15:33:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.755+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c0c3800 con 0x7fc38c157a40 2023-10-16T15:33:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.757+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c04e160 con 0x7fc38c157a40 2023-10-16T15:33:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.758+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc38403e440 con 0x7fc38c157a40 2023-10-16T15:33:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.759+0000 7fc3897fa640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fc37806afc0 0x7fc37806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.759+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc3840ab120 con 0x7fc38c157a40 2023-10-16T15:33:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.760+0000 7fc390818640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fc37806afc0 0x7fc37806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.761+0000 7fc390818640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fc37806afc0 0x7fc37806d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc38c15c630 tx=0x7fc37c009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.764+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc384076000 con 0x7fc38c157a40 2023-10-16T15:33:46.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:46.993+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc38c0c3cb0 con 0x7fc38c157a40 2023-10-16T15:33:47.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.730+0000 7fc3897fa640 1 -- 172.21.15.29:0/3655311992 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v83) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc3840044e0 con 0x7fc38c157a40 2023-10-16T15:33:47.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.734+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fc37806afc0 msgr2=0x7fc37806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:47.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.734+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fc37806afc0 0x7fc37806d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc38c15c630 tx=0x7fc37c009040 comp rx=0 tx=0).stop 2023-10-16T15:33:47.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.734+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 msgr2=0x7fc38c157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:47.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.734+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 0x7fc38c157ec0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc384009ac0 tx=0x7fc384004730 comp rx=0 tx=0).stop 2023-10-16T15:33:47.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 shutdown_connections 2023-10-16T15:33:47.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fc37806afc0 0x7fc37806d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc38c157a40 0x7fc38c157ec0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc38c0c24a0 0x7fc38c0c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 --2- 172.21.15.29:0/3655311992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc374004d00 0x7fc38c0a8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 >> 172.21.15.29:0/3655311992 conn(0x7fc38c0ce9b0 msgr2=0x7fc38c15a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 shutdown_connections 2023-10-16T15:33:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.735+0000 7fc392a7c640 1 -- 172.21.15.29:0/3655311992 wait complete. 2023-10-16T15:33:47.758 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:33:47.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T15:33:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.982+0000 7f1cf86c9640 1 Processor -- start 2023-10-16T15:33:47.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.982+0000 7f1cf86c9640 1 -- start start 2023-10-16T15:33:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.984+0000 7f1cf86c9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1cf0071b30 0x7f1cf006fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.984+0000 7f1cf86c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf00c9f10 0x7f1cf006b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.984+0000 7f1cf86c9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf00cdc30 con 0x7f1cf006ade0 2023-10-16T15:33:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.984+0000 7f1cf86c9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf006b780 con 0x7f1cf0071b30 2023-10-16T15:33:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.984+0000 7f1cf86c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf00ccd80 con 0x7f1cf00c9f10 2023-10-16T15:33:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.985+0000 7f1cf6c66640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf00c9f10 0x7f1cf006b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.985+0000 7f1cf6465640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1cf0071b30 0x7f1cf006fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.985+0000 7f1cf5c64640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f1cf006ade0 0x7f1cf00c9800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:46860/0 (socket says 172.21.15.29:46860) 2023-10-16T15:33:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.985+0000 7f1cf5c64640 1 -- 172.21.15.29:0/1331324676 learned_addr learned my addr 172.21.15.29:0/1331324676 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.986+0000 7f1cf5463640 1 -- 172.21.15.29:0/1331324676 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238472500 0 0) 0x7f1cf00cdc30 con 0x7f1cf006ade0 2023-10-16T15:33:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.986+0000 7f1cf5463640 1 -- 172.21.15.29:0/1331324676 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cdc003520 con 0x7f1cf006ade0 2023-10-16T15:33:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.986+0000 7f1cf6465640 1 -- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf00c9f10 msgr2=0x7f1cf006b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.986+0000 7f1cf6465640 1 --2- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf00c9f10 0x7f1cf006b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.986+0000 7f1cf6465640 1 -- 172.21.15.29:0/1331324676 >> v1:172.21.15.170:6789/0 conn(0x7f1cf006ade0 legacy=0x7f1cf00c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.986+0000 7f1cf6465640 1 -- 172.21.15.29:0/1331324676 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf00cd020 con 0x7f1cf0071b30 2023-10-16T15:33:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.988+0000 7f1cf6465640 1 --2- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf0071b30 0x7f1cf006fd40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0097a0 tx=0x7f1cec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37b55d45e92884ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.988+0000 7f1cf5463640 1 -- 172.21.15.29:0/1331324676 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec031830 con 0x7f1cf0071b30 2023-10-16T15:33:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.988+0000 7f1cf5463640 1 -- 172.21.15.29:0/1331324676 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec031990 con 0x7f1cf0071b30 2023-10-16T15:33:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.988+0000 7f1cf5463640 1 -- 172.21.15.29:0/1331324676 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec031d20 con 0x7f1cf0071b30 2023-10-16T15:33:47.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.989+0000 7f1cf86c9640 1 -- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf0071b30 msgr2=0x7f1cf006fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.989+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf0071b30 0x7f1cf006fd40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0097a0 tx=0x7f1cec02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.989+0000 7f1cf86c9640 1 -- 172.21.15.29:0/1331324676 shutdown_connections 2023-10-16T15:33:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.989+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf00c9f10 0x7f1cf006b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.989+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/1331324676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf0071b30 0x7f1cf006fd40 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.989+0000 7f1cf86c9640 1 -- 172.21.15.29:0/1331324676 >> 172.21.15.29:0/1331324676 conn(0x7f1cf006bc10 msgr2=0x7f1cf006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.990+0000 7f1cf86c9640 1 -- 172.21.15.29:0/1331324676 shutdown_connections 2023-10-16T15:33:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.990+0000 7f1cf86c9640 1 -- 172.21.15.29:0/1331324676 wait complete. 2023-10-16T15:33:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.992+0000 7f1cf86c9640 1 Processor -- start 2023-10-16T15:33:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.992+0000 7f1cf86c9640 1 -- start start 2023-10-16T15:33:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.993+0000 7f1cf86c9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf006ade0 0x7f1cf00bef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.993+0000 7f1cf86c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 0x7f1cf014d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.993+0000 7f1cf86c9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf00c9f10 0x7f1cf00c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:47.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf86c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf00bf440 con 0x7f1cf0071b30 2023-10-16T15:33:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf86c9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf00bf8c0 con 0x7f1cf00c9f10 2023-10-16T15:33:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf86c9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf014d430 con 0x7f1cf006ade0 2023-10-16T15:33:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf5c64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 0x7f1cf014d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf5c64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 0x7f1cf014d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46306/0 (socket says 172.21.15.29:46306) 2023-10-16T15:33:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf5c64640 1 -- 172.21.15.29:0/2575101326 learned_addr learned my addr 172.21.15.29:0/2575101326 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf6465640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf006ade0 0x7f1cf00bef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.994+0000 7f1cf6c66640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf00c9f10 0x7f1cf00c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf5c64640 1 -- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf006ade0 msgr2=0x7f1cf00bef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf5c64640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf006ade0 0x7f1cf00bef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf5c64640 1 -- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf00c9f10 msgr2=0x7f1cf00c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf5c64640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf00c9f10 0x7f1cf00c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf5c64640 1 -- 172.21.15.29:0/2575101326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec009400 con 0x7f1cf0071b30 2023-10-16T15:33:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf5c64640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 0x7f1cf014d8c0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000c440 tx=0x7f1ce000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0005ce0 con 0x7f1cf0071b30 2023-10-16T15:33:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce0005e40 con 0x7f1cf0071b30 2023-10-16T15:33:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce00072b0 con 0x7f1cf0071b30 2023-10-16T15:33:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.995+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf00c5aa0 con 0x7f1cf0071b30 2023-10-16T15:33:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.996+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf00c5d10 con 0x7f1cf0071b30 2023-10-16T15:33:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.997+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf0073080 con 0x7f1cf0071b30 2023-10-16T15:33:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.998+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1ce0007410 con 0x7f1cf0071b30 2023-10-16T15:33:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.999+0000 7f1ce6ffd640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1cd406af90 0x7f1cd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.999+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1ce0088c80 con 0x7f1cf0071b30 2023-10-16T15:33:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:47.999+0000 7f1cf6465640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1cd406af90 0x7f1cd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.001+0000 7f1cf6465640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1cd406af90 0x7f1cd406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1cec036000 tx=0x7f1cec03c040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:48.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.005+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ce0053d60 con 0x7f1cf0071b30 2023-10-16T15:33:48.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.234+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1cf006b4b0 con 0x7f1cf0071b30 2023-10-16T15:33:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.733+0000 7f1ce6ffd640 1 -- 172.21.15.29:0/2575101326 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v84) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ce0017020 con 0x7f1cf0071b30 2023-10-16T15:33:48.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.736+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1cd406af90 msgr2=0x7f1cd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:48.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.736+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1cd406af90 0x7f1cd406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1cec036000 tx=0x7f1cec03c040 comp rx=0 tx=0).stop 2023-10-16T15:33:48.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 msgr2=0x7f1cf014d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:48.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 0x7f1cf014d8c0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000c440 tx=0x7f1ce000c900 comp rx=0 tx=0).stop 2023-10-16T15:33:48.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 shutdown_connections 2023-10-16T15:33:48.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f1cd406af90 0x7f1cd406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:48.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cf0071b30 0x7f1cf014d8c0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:48.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cf00c9f10 0x7f1cf00c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:48.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 --2- 172.21.15.29:0/2575101326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1cf006ade0 0x7f1cf00bef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:48.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 >> 172.21.15.29:0/2575101326 conn(0x7f1cf006bc10 msgr2=0x7f1cf00aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:48.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 shutdown_connections 2023-10-16T15:33:48.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.737+0000 7f1cf86c9640 1 -- 172.21.15.29:0/2575101326 wait complete. 2023-10-16T15:33:48.759 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T15:33:48.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.989+0000 7f5f2d755640 1 Processor -- start 2023-10-16T15:33:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.989+0000 7f5f2d755640 1 -- start start 2023-10-16T15:33:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.991+0000 7f5f2d755640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 0x7f5f280b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.991+0000 7f5f2d755640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5f2815d880 0x7f5f280b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.991+0000 7f5f2d755640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2806f7f0 con 0x7f5f28159b20 2023-10-16T15:33:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.991+0000 7f5f2d755640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2806c5e0 con 0x7f5f2815d880 2023-10-16T15:33:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.991+0000 7f5f2d755640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2806c8c0 con 0x7f5f28159f00 2023-10-16T15:33:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.992+0000 7f5f26ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f28159b20 0x7f5f2806bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44844/0 (socket says 172.21.15.29:44844) 2023-10-16T15:33:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.992+0000 7f5f26ffd640 1 -- 172.21.15.29:0/2188377255 learned_addr learned my addr 172.21.15.29:0/2188377255 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.992+0000 7f5f1e7fc640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 0x7f5f280b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.992+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687797965 0 0) 0x7f5f2806f7f0 con 0x7f5f28159b20 2023-10-16T15:33:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.992+0000 7f5f277fe640 1 --2- 172.21.15.29:0/2188377255 >> v2:172.21.15.170:3300/0 conn(0x7f5f2815d880 0x7f5f280b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.992+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f14005730 con 0x7f5f28159b20 2023-10-16T15:33:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455413259 0 0) 0x7f5f14005730 con 0x7f5f28159b20 2023-10-16T15:33:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 msgr2=0x7f5f280b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 0x7f5f280b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 >> v2:172.21.15.170:3300/0 conn(0x7f5f2815d880 msgr2=0x7f5f280b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 --2- 172.21.15.29:0/2188377255 >> v2:172.21.15.170:3300/0 conn(0x7f5f2815d880 0x7f5f280b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2806cba0 con 0x7f5f28159b20 2023-10-16T15:33:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5f18001670 con 0x7f5f28159b20 2023-10-16T15:33:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 >> v1:172.21.15.29:6789/0 conn(0x7f5f28159b20 legacy=0x7f5f2806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f14005b80 0x7f5f14005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 0x7f5f140069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f14006f10 0x7f5f14007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2806c5e0 con 0x7f5f14005b80 2023-10-16T15:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2806c8c0 con 0x7f5f14006f10 2023-10-16T15:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1400db90 con 0x7f5f28159f00 2023-10-16T15:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.993+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f14005b80 0x7f5f14005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.994+0000 7f5f26ffd640 1 -- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 msgr2=0x7f5f140069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.994+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f28159f00 0x7f5f140069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.994+0000 7f5f26ffd640 1 -- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f14006f10 msgr2=0x7f5f14007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.994+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f14006f10 0x7f5f14007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.994+0000 7f5f26ffd640 1 -- 172.21.15.29:0/2188377255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f14005730 con 0x7f5f14005b80 2023-10-16T15:33:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.995+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f14005b80 0x7f5f14005fa0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800c270 tx=0x7f5f14007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a72ff123dc4f21a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.995+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1802cae0 con 0x7f5f14005b80 2023-10-16T15:33:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.995+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1802cc40 con 0x7f5f14005b80 2023-10-16T15:33:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.995+0000 7f5f267fc640 1 -- 172.21.15.29:0/2188377255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180058d0 con 0x7f5f14005b80 2023-10-16T15:33:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.995+0000 7f5f2d755640 1 -- 172.21.15.29:0/2188377255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f14005b80 msgr2=0x7f5f14005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.995+0000 7f5f2d755640 1 --2- 172.21.15.29:0/2188377255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f14005b80 0x7f5f14005fa0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800c270 tx=0x7f5f14007a00 comp rx=0 tx=0).stop 2023-10-16T15:33:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.996+0000 7f5f277fe640 1 -- 172.21.15.29:0/2188377255 reap_dead start 2023-10-16T15:33:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.996+0000 7f5f2d755640 1 -- 172.21.15.29:0/2188377255 shutdown_connections 2023-10-16T15:33:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.996+0000 7f5f2d755640 1 -- 172.21.15.29:0/2188377255 >> 172.21.15.29:0/2188377255 conn(0x7f5f280cb410 msgr2=0x7f5f280cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.996+0000 7f5f2d755640 1 -- 172.21.15.29:0/2188377255 shutdown_connections 2023-10-16T15:33:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.996+0000 7f5f2d755640 1 -- 172.21.15.29:0/2188377255 wait complete. 2023-10-16T15:33:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.998+0000 7f5f2d755640 1 Processor -- start 2023-10-16T15:33:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.998+0000 7f5f2d755640 1 -- start start 2023-10-16T15:33:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f2d755640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f14005b80 0x7f5f28156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f2d755640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28157310 0x7f5f280b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f2d755640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 0x7f5f280b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f2d755640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280b1c20 con 0x7f5f280b2cc0 2023-10-16T15:33:49.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f2d755640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280b1f00 con 0x7f5f28157310 2023-10-16T15:33:49.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f2d755640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280b21e0 con 0x7f5f14005b80 2023-10-16T15:33:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 0x7f5f280b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 0x7f5f280b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46330/0 (socket says 172.21.15.29:46330) 2023-10-16T15:33:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f277fe640 1 -- 172.21.15.29:0/3935490403 learned_addr learned my addr 172.21.15.29:0/3935490403 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f1e7fc640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28157310 0x7f5f280b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f14005b80 0x7f5f28156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:48.999+0000 7f5f277fe640 1 -- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f14005b80 msgr2=0x7f5f28156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f277fe640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f14005b80 0x7f5f28156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f277fe640 1 -- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28157310 msgr2=0x7f5f280b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f277fe640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28157310 0x7f5f280b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f277fe640 1 -- 172.21.15.29:0/3935490403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f280b24c0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f277fe640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 0x7f5f280b3140 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5f2815dc60 tx=0x7f5f2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2000ed30 con 0x7f5f280b2cc0 2023-10-16T15:33:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2000eeb0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f280b27a0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20014640 con 0x7f5f280b2cc0 2023-10-16T15:33:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.000+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f280b19f0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.001+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f280b29e0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.002+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f200147a0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.003+0000 7f5f1ffff640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f5f0806afc0 0x7f5f0806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.004+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5f200899e0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.004+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f5f0806afc0 0x7f5f0806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.005+0000 7f5f26ffd640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f5f0806afc0 0x7f5f0806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5f18034000 tx=0x7f5f18007000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.008+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f280b29e0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.239+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f280b29e0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.240+0000 7f5f1ffff640 1 -- 172.21.15.29:0/3935490403 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f5f280b29e0 con 0x7f5f280b2cc0 2023-10-16T15:33:49.241 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:49.242 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:33:45.081584+0000","modified":"2023-10-16T15:33:48.236102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9342":{"gid":9342,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3104953668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3104953668},{"type":"v1","addr":"172.21.15.29:6833","nonce":3104953668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T15:33:49.242 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 84 2023-10-16T15:33:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f5f0806afc0 msgr2=0x7f5f0806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f5f0806afc0 0x7f5f0806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5f18034000 tx=0x7f5f18007000 comp rx=0 tx=0).stop 2023-10-16T15:33:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 msgr2=0x7f5f280b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 0x7f5f280b3140 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5f2815dc60 tx=0x7f5f2000e4b0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 shutdown_connections 2023-10-16T15:33:49.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f5f0806afc0 0x7f5f0806d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f280b2cc0 0x7f5f280b3140 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f28157310 0x7f5f280b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 --2- 172.21.15.29:0/3935490403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f14005b80 0x7f5f28156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 >> 172.21.15.29:0/3935490403 conn(0x7f5f280cb410 msgr2=0x7f5f280b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:49.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 shutdown_connections 2023-10-16T15:33:49.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.243+0000 7f5f2d755640 1 -- 172.21.15.29:0/3935490403 wait complete. 2023-10-16T15:33:49.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:33:49.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.484+0000 7fccb22b6640 1 Processor -- start 2023-10-16T15:33:49.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.485+0000 7fccb22b6640 1 -- start start 2023-10-16T15:33:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.486+0000 7fccb22b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccac15cbe0 0x7fccac0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.486+0000 7fccb22b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac0b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb22b6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0d15a0 con 0x7fccac15cfc0 2023-10-16T15:33:49.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb22b6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccac0d1880 con 0x7fccac15cbe0 2023-10-16T15:33:49.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb22b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0d1b60 con 0x7fccac0b4c80 2023-10-16T15:33:49.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccac15cbe0 0x7fccac0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccac15cbe0 0x7fccac0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46346/0 (socket says 172.21.15.29:46346) 2023-10-16T15:33:49.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb0ab3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fccac15cfc0 0x7fccac0b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44854/0 (socket says 172.21.15.29:44854) 2023-10-16T15:33:49.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 -- 172.21.15.29:0/978974749 learned_addr learned my addr 172.21.15.29:0/978974749 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:49.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 -- 172.21.15.29:0/978974749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 msgr2=0x7fccac0b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:49.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 --2- 172.21.15.29:0/978974749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac0b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 -- 172.21.15.29:0/978974749 >> v1:172.21.15.29:6789/0 conn(0x7fccac15cfc0 legacy=0x7fccac0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fccb12b4640 1 -- 172.21.15.29:0/978974749 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0d1e40 con 0x7fccac15cbe0 2023-10-16T15:33:49.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.487+0000 7fcca3fff640 1 -- 172.21.15.29:0/978974749 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263914064 0 0) 0x7fccac0d15a0 con 0x7fccac15cfc0 2023-10-16T15:33:49.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.488+0000 7fccb12b4640 1 --2- 172.21.15.29:0/978974749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cbe0 0x7fccac0d0b50 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009e50 tx=0x7fcca802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e87bf85aa558ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.489+0000 7fcca3fff640 1 -- 172.21.15.29:0/978974749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca802cb90 con 0x7fccac15cbe0 2023-10-16T15:33:49.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.489+0000 7fcca3fff640 1 -- 172.21.15.29:0/978974749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca802ccf0 con 0x7fccac15cbe0 2023-10-16T15:33:49.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.489+0000 7fcca3fff640 1 -- 172.21.15.29:0/978974749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8034610 con 0x7fccac15cbe0 2023-10-16T15:33:49.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.489+0000 7fccb22b6640 1 -- 172.21.15.29:0/978974749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cbe0 msgr2=0x7fccac0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.489+0000 7fccb22b6640 1 --2- 172.21.15.29:0/978974749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cbe0 0x7fccac0d0b50 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009e50 tx=0x7fcca802bff0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.490+0000 7fccb22b6640 1 -- 172.21.15.29:0/978974749 shutdown_connections 2023-10-16T15:33:49.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.490+0000 7fccb22b6640 1 --2- 172.21.15.29:0/978974749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac0b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.490+0000 7fccb22b6640 1 --2- 172.21.15.29:0/978974749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cbe0 0x7fccac0d0b50 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.490+0000 7fccb22b6640 1 -- 172.21.15.29:0/978974749 >> 172.21.15.29:0/978974749 conn(0x7fccac0cf570 msgr2=0x7fccac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:49.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.490+0000 7fccb22b6640 1 -- 172.21.15.29:0/978974749 shutdown_connections 2023-10-16T15:33:49.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.490+0000 7fccb22b6640 1 -- 172.21.15.29:0/978974749 wait complete. 2023-10-16T15:33:49.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.491+0000 7fccb22b6640 1 Processor -- start 2023-10-16T15:33:49.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.491+0000 7fccb22b6640 1 -- start start 2023-10-16T15:33:49.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.492+0000 7fccb22b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac15f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.492+0000 7fccb22b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccac15cbe0 0x7fccac151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.492+0000 7fccb22b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 0x7fccac1e6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.492+0000 7fccb22b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac15f2b0 con 0x7fccac15cfc0 2023-10-16T15:33:49.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.492+0000 7fccb22b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac1542c0 con 0x7fccac15cbe0 2023-10-16T15:33:49.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.492+0000 7fccb22b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac152070 con 0x7fccac0b4c80 2023-10-16T15:33:49.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 0x7fccac1e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 0x7fccac1e6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46354/0 (socket says 172.21.15.29:46354) 2023-10-16T15:33:49.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 -- 172.21.15.29:0/696640161 learned_addr learned my addr 172.21.15.29:0/696640161 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:49.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb0ab3640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccac15cbe0 0x7fccac151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb12b4640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac15f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 -- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 msgr2=0x7fccac15f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac15f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 -- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccac15cbe0 msgr2=0x7fccac151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccac15cbe0 0x7fccac151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb1ab5640 1 -- 172.21.15.29:0/696640161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8009b30 con 0x7fccac15cfc0 2023-10-16T15:33:49.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.493+0000 7fccb12b4640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:49.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.494+0000 7fccb1ab5640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 0x7fccac1e6370 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fcca400b410 tx=0x7fcca4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.494+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4011830 con 0x7fccac15cfc0 2023-10-16T15:33:49.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.494+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca4011990 con 0x7fccac15cfc0 2023-10-16T15:33:49.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.494+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4011d20 con 0x7fccac15cfc0 2023-10-16T15:33:49.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.494+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac15e980 con 0x7fccac15cfc0 2023-10-16T15:33:49.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.494+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac15ef40 con 0x7fccac15cfc0 2023-10-16T15:33:49.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.495+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac15f460 con 0x7fccac15cfc0 2023-10-16T15:33:49.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.496+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcca401a400 con 0x7fccac15cfc0 2023-10-16T15:33:49.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.497+0000 7fcca1ffb640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc9006af90 0x7fcc9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.497+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcca4089870 con 0x7fccac15cfc0 2023-10-16T15:33:49.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.498+0000 7fccb12b4640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc9006af90 0x7fcc9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.499+0000 7fccb12b4640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc9006af90 0x7fcc9006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcca8035000 tx=0x7fcca8030000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.501+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fccac15f460 con 0x7fccac15cfc0 2023-10-16T15:33:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.687+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccac15f460 con 0x7fccac15cfc0 2023-10-16T15:33:49.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.690+0000 7fcca1ffb640 1 -- 172.21.15.29:0/696640161 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16623 (secure 0 0 0) 0x7fccac15f460 con 0x7fccac15cfc0 2023-10-16T15:33:49.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:49.692 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":179,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:46.597174+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:33:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc9006af90 msgr2=0x7fcc9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc9006af90 0x7fcc9006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcca8035000 tx=0x7fcca8030000 comp rx=0 tx=0).stop 2023-10-16T15:33:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 msgr2=0x7fccac1e6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 0x7fccac1e6370 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fcca400b410 tx=0x7fcca4009f90 comp rx=0 tx=0).stop 2023-10-16T15:33:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 shutdown_connections 2023-10-16T15:33:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc9006af90 0x7fcc9006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccac15cfc0 0x7fccac1e6370 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccac15cbe0 0x7fccac151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 --2- 172.21.15.29:0/696640161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fccac0b4c80 0x7fccac15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 >> 172.21.15.29:0/696640161 conn(0x7fccac0cf570 msgr2=0x7fccac0b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 shutdown_connections 2023-10-16T15:33:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.693+0000 7fccb22b6640 1 -- 172.21.15.29:0/696640161 wait complete. 2023-10-16T15:33:49.714 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:33:49.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:33:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.935+0000 7f81ea739640 1 Processor -- start 2023-10-16T15:33:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.935+0000 7f81ea739640 1 -- start start 2023-10-16T15:33:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.936+0000 7f81ea739640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415a970 0x7f81e406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.936+0000 7f81ea739640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f81e415d430 0x7f81e4070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81ea739640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81e4071360 0x7f81e40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81ea739640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e4070660 con 0x7f81e4071360 2023-10-16T15:33:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81ea739640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e415dc60 con 0x7f81e415d430 2023-10-16T15:33:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81ea739640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e415df40 con 0x7f81e415a970 2023-10-16T15:33:49.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81e8cd6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81e4071360 0x7f81e40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81e8cd6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81e4071360 0x7f81e40cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46364/0 (socket says 172.21.15.29:46364) 2023-10-16T15:33:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81e8cd6640 1 -- 172.21.15.29:0/1231374461 learned_addr learned my addr 172.21.15.29:0/1231374461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81e3fff640 1 --2- 172.21.15.29:0/1231374461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415a970 0x7f81e406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.937+0000 7f81e37fe640 1 --2- 172.21.15.29:0/1231374461 >> v2:172.21.15.170:3300/0 conn(0x7f81e415d430 0x7f81e4070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.938+0000 7f81e8cd6640 1 -- 172.21.15.29:0/1231374461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415a970 msgr2=0x7f81e406ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.938+0000 7f81e8cd6640 1 --2- 172.21.15.29:0/1231374461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415a970 0x7f81e406ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.938+0000 7f81e8cd6640 1 -- 172.21.15.29:0/1231374461 >> v2:172.21.15.170:3300/0 conn(0x7f81e415d430 msgr2=0x7f81e4070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.938+0000 7f81e8cd6640 1 --2- 172.21.15.29:0/1231374461 >> v2:172.21.15.170:3300/0 conn(0x7f81e415d430 0x7f81e4070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.938+0000 7f81e8cd6640 1 -- 172.21.15.29:0/1231374461 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e415e220 con 0x7f81e4071360 2023-10-16T15:33:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.939+0000 7f81e8cd6640 1 --2- 172.21.15.29:0/1231374461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e4071360 0x7f81e40cd880 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f81e4073af0 tx=0x7f81dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c53d0fc743297b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.939+0000 7f81e2ffd640 1 -- 172.21.15.29:0/1231374461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc031870 con 0x7f81e4071360 2023-10-16T15:33:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.939+0000 7f81e2ffd640 1 -- 172.21.15.29:0/1231374461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc0319d0 con 0x7f81e4071360 2023-10-16T15:33:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.939+0000 7f81e2ffd640 1 -- 172.21.15.29:0/1231374461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc031d60 con 0x7f81e4071360 2023-10-16T15:33:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 -- 172.21.15.29:0/1231374461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e4071360 msgr2=0x7f81e40cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 --2- 172.21.15.29:0/1231374461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e4071360 0x7f81e40cd880 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f81e4073af0 tx=0x7f81dc02b910 comp rx=0 tx=0).stop 2023-10-16T15:33:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 -- 172.21.15.29:0/1231374461 shutdown_connections 2023-10-16T15:33:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 --2- 172.21.15.29:0/1231374461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e4071360 0x7f81e40cd880 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 --2- 172.21.15.29:0/1231374461 >> v2:172.21.15.170:3300/0 conn(0x7f81e415d430 0x7f81e4070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 --2- 172.21.15.29:0/1231374461 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415a970 0x7f81e406ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 -- 172.21.15.29:0/1231374461 >> 172.21.15.29:0/1231374461 conn(0x7f81e40cfaa0 msgr2=0x7f81e40cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 -- 172.21.15.29:0/1231374461 shutdown_connections 2023-10-16T15:33:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.940+0000 7f81ea739640 1 -- 172.21.15.29:0/1231374461 wait complete. 2023-10-16T15:33:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.942+0000 7f81ea739640 1 Processor -- start 2023-10-16T15:33:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.942+0000 7f81ea739640 1 -- start start 2023-10-16T15:33:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81ea739640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 0x7f81e4157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81ea739640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415d430 0x7f81e4158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81ea739640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81e4158690 0x7f81e40c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81ea739640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40c4550 con 0x7f81e415a970 2023-10-16T15:33:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81ea739640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40c38d0 con 0x7f81e4158690 2023-10-16T15:33:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81ea739640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40c3bb0 con 0x7f81e415d430 2023-10-16T15:33:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 0x7f81e4157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 0x7f81e4157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46368/0 (socket says 172.21.15.29:46368) 2023-10-16T15:33:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 -- 172.21.15.29:0/1745922348 learned_addr learned my addr 172.21.15.29:0/1745922348 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 -- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415d430 msgr2=0x7f81e4158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415d430 0x7f81e4158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 -- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81e4158690 msgr2=0x7f81e40c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81e4158690 0x7f81e40c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.943+0000 7f81e3fff640 1 -- 172.21.15.29:0/1745922348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc0094d0 con 0x7f81e415a970 2023-10-16T15:33:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.944+0000 7f81e3fff640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 0x7f81e4157940 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f81d4007ef0 tx=0x7f81d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.944+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4014620 con 0x7f81e415a970 2023-10-16T15:33:49.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.944+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d4014780 con 0x7f81e415a970 2023-10-16T15:33:49.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.944+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4014b30 con 0x7f81e415a970 2023-10-16T15:33:49.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.944+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e40c3e90 con 0x7f81e415a970 2023-10-16T15:33:49.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.944+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e4157e80 con 0x7f81e415a970 2023-10-16T15:33:49.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.946+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e404e160 con 0x7f81e415a970 2023-10-16T15:33:49.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.946+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f81d4014c90 con 0x7f81e415a970 2023-10-16T15:33:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.947+0000 7f81e0ff9640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f81d006af90 0x7f81d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.947+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f81d408a150 con 0x7f81e415a970 2023-10-16T15:33:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.948+0000 7f81e37fe640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f81d006af90 0x7f81d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.949+0000 7f81e37fe640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f81d006af90 0x7f81d006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81e415a670 tx=0x7f81d8008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:49.951+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f81d4055160 con 0x7f81e415a970 2023-10-16T15:33:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.144+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 --> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81e415ad50 con 0x7f81d006af90 2023-10-16T15:33:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.172+0000 7f81e0ff9640 1 -- 172.21.15.29:0/1745922348 <== mgr.7456 v2:172.21.15.170:6832/1981883088 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286787 (secure 0 0 0) 0x7f81d8009dc0 con 0x7f81d006af90 2023-10-16T15:33:50.178 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:50.179 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:33:50.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f81d006af90 msgr2=0x7f81d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f81d006af90 0x7f81d006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81e415a670 tx=0x7f81d8008040 comp rx=0 tx=0).stop 2023-10-16T15:33:50.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 msgr2=0x7f81e4157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 0x7f81e4157940 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f81d4007ef0 tx=0x7f81d400e4b0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 shutdown_connections 2023-10-16T15:33:50.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f81d006af90 0x7f81d006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81e4158690 0x7f81e40c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81e415d430 0x7f81e4158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 --2- 172.21.15.29:0/1745922348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e415a970 0x7f81e4157940 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 >> 172.21.15.29:0/1745922348 conn(0x7f81e40cfaa0 msgr2=0x7f81e40ae8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:50.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 shutdown_connections 2023-10-16T15:33:50.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.178+0000 7f81ea739640 1 -- 172.21.15.29:0/1745922348 wait complete. 2023-10-16T15:33:50.200 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T15:33:48.979404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":487,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":409700,"kb_used_data":18652,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754565020,"statfs":{"total":773094113280,"available":772674580480,"internally_reserved":0,"allocated":19099648,"data_stored":9895796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":400281145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.402515"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051822+0000","last_change":"2023-10-16T15:33:45.620840+0000","last_active":"2023-10-16T15:33:47.051822+0000","last_peered":"2023-10-16T15:33:47.051822+0000","last_clean":"2023-10-16T15:33:47.051822+0000","last_became_active":"2023-10-16T15:33:45.620462+0000","last_became_peered":"2023-10-16T15:33:45.620462+0000","last_unstale":"2023-10-16T15:33:47.051822+0000","last_undegraded":"2023-10-16T15:33:47.051822+0000","last_fullsized":"2023-10-16T15:33:47.051822+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:24.818698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047884+0000","last_change":"2023-10-16T15:33:45.624971+0000","last_active":"2023-10-16T15:33:47.047884+0000","last_peered":"2023-10-16T15:33:47.047884+0000","last_clean":"2023-10-16T15:33:47.047884+0000","last_became_active":"2023-10-16T15:33:45.624675+0000","last_became_peered":"2023-10-16T15:33:45.624675+0000","last_unstale":"2023-10-16T15:33:47.047884+0000","last_undegraded":"2023-10-16T15:33:47.047884+0000","last_fullsized":"2023-10-16T15:33:47.047884+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:07.955810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609963+0000","last_change":"2023-10-16T15:33:45.644805+0000","last_active":"2023-10-16T15:33:46.609963+0000","last_peered":"2023-10-16T15:33:46.609963+0000","last_clean":"2023-10-16T15:33:46.609963+0000","last_became_active":"2023-10-16T15:33:45.643700+0000","last_became_peered":"2023-10-16T15:33:45.643700+0000","last_unstale":"2023-10-16T15:33:46.609963+0000","last_undegraded":"2023-10-16T15:33:46.609963+0000","last_fullsized":"2023-10-16T15:33:46.609963+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:34:52.840049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050931+0000","last_change":"2023-10-16T15:33:45.648807+0000","last_active":"2023-10-16T15:33:47.050931+0000","last_peered":"2023-10-16T15:33:47.050931+0000","last_clean":"2023-10-16T15:33:47.050931+0000","last_became_active":"2023-10-16T15:33:45.647862+0000","last_became_peered":"2023-10-16T15:33:45.647862+0000","last_unstale":"2023-10-16T15:33:47.050931+0000","last_undegraded":"2023-10-16T15:33:47.050931+0000","last_fullsized":"2023-10-16T15:33:47.050931+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:08.341026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050503+0000","last_change":"2023-10-16T15:33:45.641700+0000","last_active":"2023-10-16T15:33:47.050503+0000","last_peered":"2023-10-16T15:33:47.050503+0000","last_clean":"2023-10-16T15:33:47.050503+0000","last_became_active":"2023-10-16T15:33:45.641387+0000","last_became_peered":"2023-10-16T15:33:45.641387+0000","last_unstale":"2023-10-16T15:33:47.050503+0000","last_undegraded":"2023-10-16T15:33:47.050503+0000","last_fullsized":"2023-10-16T15:33:47.050503+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:56.138784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050883+0000","last_change":"2023-10-16T15:33:45.645610+0000","last_active":"2023-10-16T15:33:47.050883+0000","last_peered":"2023-10-16T15:33:47.050883+0000","last_clean":"2023-10-16T15:33:47.050883+0000","last_became_active":"2023-10-16T15:33:45.645286+0000","last_became_peered":"2023-10-16T15:33:45.645286+0000","last_unstale":"2023-10-16T15:33:47.050883+0000","last_undegraded":"2023-10-16T15:33:47.050883+0000","last_fullsized":"2023-10-16T15:33:47.050883+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:24.613523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602439+0000","last_change":"2023-10-16T15:33:45.634981+0000","last_active":"2023-10-16T15:33:46.602439+0000","last_peered":"2023-10-16T15:33:46.602439+0000","last_clean":"2023-10-16T15:33:46.602439+0000","last_became_active":"2023-10-16T15:33:45.634720+0000","last_became_peered":"2023-10-16T15:33:45.634720+0000","last_unstale":"2023-10-16T15:33:46.602439+0000","last_undegraded":"2023-10-16T15:33:46.602439+0000","last_fullsized":"2023-10-16T15:33:46.602439+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.915061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049182+0000","last_change":"2023-10-16T15:33:45.631222+0000","last_active":"2023-10-16T15:33:47.049182+0000","last_peered":"2023-10-16T15:33:47.049182+0000","last_clean":"2023-10-16T15:33:47.049182+0000","last_became_active":"2023-10-16T15:33:45.630924+0000","last_became_peered":"2023-10-16T15:33:45.630924+0000","last_unstale":"2023-10-16T15:33:47.049182+0000","last_undegraded":"2023-10-16T15:33:47.049182+0000","last_fullsized":"2023-10-16T15:33:47.049182+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:02.588048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048491+0000","last_change":"2023-10-16T15:33:45.612573+0000","last_active":"2023-10-16T15:33:47.048491+0000","last_peered":"2023-10-16T15:33:47.048491+0000","last_clean":"2023-10-16T15:33:47.048491+0000","last_became_active":"2023-10-16T15:33:45.611766+0000","last_became_peered":"2023-10-16T15:33:45.611766+0000","last_unstale":"2023-10-16T15:33:47.048491+0000","last_undegraded":"2023-10-16T15:33:47.048491+0000","last_fullsized":"2023-10-16T15:33:47.048491+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:28.106963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608068+0000","last_change":"2023-10-16T15:33:45.636767+0000","last_active":"2023-10-16T15:33:46.608068+0000","last_peered":"2023-10-16T15:33:46.608068+0000","last_clean":"2023-10-16T15:33:46.608068+0000","last_became_active":"2023-10-16T15:33:45.636202+0000","last_became_peered":"2023-10-16T15:33:45.636202+0000","last_unstale":"2023-10-16T15:33:46.608068+0000","last_undegraded":"2023-10-16T15:33:46.608068+0000","last_fullsized":"2023-10-16T15:33:46.608068+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:33.012654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049872+0000","last_change":"2023-10-16T15:33:45.631274+0000","last_active":"2023-10-16T15:33:47.049872+0000","last_peered":"2023-10-16T15:33:47.049872+0000","last_clean":"2023-10-16T15:33:47.049872+0000","last_became_active":"2023-10-16T15:33:45.630963+0000","last_became_peered":"2023-10-16T15:33:45.630963+0000","last_unstale":"2023-10-16T15:33:47.049872+0000","last_undegraded":"2023-10-16T15:33:47.049872+0000","last_fullsized":"2023-10-16T15:33:47.049872+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:40.098746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606755+0000","last_change":"2023-10-16T15:33:45.634976+0000","last_active":"2023-10-16T15:33:46.606755+0000","last_peered":"2023-10-16T15:33:46.606755+0000","last_clean":"2023-10-16T15:33:46.606755+0000","last_became_active":"2023-10-16T15:33:45.633765+0000","last_became_peered":"2023-10-16T15:33:45.633765+0000","last_unstale":"2023-10-16T15:33:46.606755+0000","last_undegraded":"2023-10-16T15:33:46.606755+0000","last_fullsized":"2023-10-16T15:33:46.606755+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:13.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050527+0000","last_change":"2023-10-16T15:33:45.639765+0000","last_active":"2023-10-16T15:33:47.050527+0000","last_peered":"2023-10-16T15:33:47.050527+0000","last_clean":"2023-10-16T15:33:47.050527+0000","last_became_active":"2023-10-16T15:33:45.639423+0000","last_became_peered":"2023-10-16T15:33:45.639423+0000","last_unstale":"2023-10-16T15:33:47.050527+0000","last_undegraded":"2023-10-16T15:33:47.050527+0000","last_fullsized":"2023-10-16T15:33:47.050527+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:49.442507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049564+0000","last_change":"2023-10-16T15:33:45.623414+0000","last_active":"2023-10-16T15:33:47.049564+0000","last_peered":"2023-10-16T15:33:47.049564+0000","last_clean":"2023-10-16T15:33:47.049564+0000","last_became_active":"2023-10-16T15:33:45.623076+0000","last_became_peered":"2023-10-16T15:33:45.623076+0000","last_unstale":"2023-10-16T15:33:47.049564+0000","last_undegraded":"2023-10-16T15:33:47.049564+0000","last_fullsized":"2023-10-16T15:33:47.049564+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:39.209070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608911+0000","last_change":"2023-10-16T15:33:45.621392+0000","last_active":"2023-10-16T15:33:46.608911+0000","last_peered":"2023-10-16T15:33:46.608911+0000","last_clean":"2023-10-16T15:33:46.608911+0000","last_became_active":"2023-10-16T15:33:45.620962+0000","last_became_peered":"2023-10-16T15:33:45.620962+0000","last_unstale":"2023-10-16T15:33:46.608911+0000","last_undegraded":"2023-10-16T15:33:46.608911+0000","last_fullsized":"2023-10-16T15:33:46.608911+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:32.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048700+0000","last_change":"2023-10-16T15:33:45.623274+0000","last_active":"2023-10-16T15:33:47.048700+0000","last_peered":"2023-10-16T15:33:47.048700+0000","last_clean":"2023-10-16T15:33:47.048700+0000","last_became_active":"2023-10-16T15:33:45.623000+0000","last_became_peered":"2023-10-16T15:33:45.623000+0000","last_unstale":"2023-10-16T15:33:47.048700+0000","last_undegraded":"2023-10-16T15:33:47.048700+0000","last_fullsized":"2023-10-16T15:33:47.048700+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:29.637498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609501+0000","last_change":"2023-10-16T15:33:45.636374+0000","last_active":"2023-10-16T15:33:46.609501+0000","last_peered":"2023-10-16T15:33:46.609501+0000","last_clean":"2023-10-16T15:33:46.609501+0000","last_became_active":"2023-10-16T15:33:45.635989+0000","last_became_peered":"2023-10-16T15:33:45.635989+0000","last_unstale":"2023-10-16T15:33:46.609501+0000","last_undegraded":"2023-10-16T15:33:46.609501+0000","last_fullsized":"2023-10-16T15:33:46.609501+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:46.404483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607689+0000","last_change":"2023-10-16T15:33:45.644254+0000","last_active":"2023-10-16T15:33:46.607689+0000","last_peered":"2023-10-16T15:33:46.607689+0000","last_clean":"2023-10-16T15:33:46.607689+0000","last_became_active":"2023-10-16T15:33:45.643724+0000","last_became_peered":"2023-10-16T15:33:45.643724+0000","last_unstale":"2023-10-16T15:33:46.607689+0000","last_undegraded":"2023-10-16T15:33:46.607689+0000","last_fullsized":"2023-10-16T15:33:46.607689+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:18.019829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051358+0000","last_change":"2023-10-16T15:33:45.628013+0000","last_active":"2023-10-16T15:33:47.051358+0000","last_peered":"2023-10-16T15:33:47.051358+0000","last_clean":"2023-10-16T15:33:47.051358+0000","last_became_active":"2023-10-16T15:33:45.627787+0000","last_became_peered":"2023-10-16T15:33:45.627787+0000","last_unstale":"2023-10-16T15:33:47.051358+0000","last_undegraded":"2023-10-16T15:33:47.051358+0000","last_fullsized":"2023-10-16T15:33:47.051358+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:20.580101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050726+0000","last_change":"2023-10-16T15:33:45.617309+0000","last_active":"2023-10-16T15:33:47.050726+0000","last_peered":"2023-10-16T15:33:47.050726+0000","last_clean":"2023-10-16T15:33:47.050726+0000","last_became_active":"2023-10-16T15:33:45.616992+0000","last_became_peered":"2023-10-16T15:33:45.616992+0000","last_unstale":"2023-10-16T15:33:47.050726+0000","last_undegraded":"2023-10-16T15:33:47.050726+0000","last_fullsized":"2023-10-16T15:33:47.050726+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:40.098746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608418+0000","last_change":"2023-10-16T15:33:45.635822+0000","last_active":"2023-10-16T15:33:46.608418+0000","last_peered":"2023-10-16T15:33:46.608418+0000","last_clean":"2023-10-16T15:33:46.608418+0000","last_became_active":"2023-10-16T15:33:45.635242+0000","last_became_peered":"2023-10-16T15:33:45.635242+0000","last_unstale":"2023-10-16T15:33:46.608418+0000","last_undegraded":"2023-10-16T15:33:46.608418+0000","last_fullsized":"2023-10-16T15:33:46.608418+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:19.641458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049920+0000","last_change":"2023-10-16T15:33:45.623835+0000","last_active":"2023-10-16T15:33:47.049920+0000","last_peered":"2023-10-16T15:33:47.049920+0000","last_clean":"2023-10-16T15:33:47.049920+0000","last_became_active":"2023-10-16T15:33:45.615916+0000","last_became_peered":"2023-10-16T15:33:45.615916+0000","last_unstale":"2023-10-16T15:33:47.049920+0000","last_undegraded":"2023-10-16T15:33:47.049920+0000","last_fullsized":"2023-10-16T15:33:47.049920+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:12.220344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049273+0000","last_change":"2023-10-16T15:33:45.624991+0000","last_active":"2023-10-16T15:33:47.049273+0000","last_peered":"2023-10-16T15:33:47.049273+0000","last_clean":"2023-10-16T15:33:47.049273+0000","last_became_active":"2023-10-16T15:33:45.624667+0000","last_became_peered":"2023-10-16T15:33:45.624667+0000","last_unstale":"2023-10-16T15:33:47.049273+0000","last_undegraded":"2023-10-16T15:33:47.049273+0000","last_fullsized":"2023-10-16T15:33:47.049273+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:35.237056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606555+0000","last_change":"2023-10-16T15:33:45.634598+0000","last_active":"2023-10-16T15:33:46.606555+0000","last_peered":"2023-10-16T15:33:46.606555+0000","last_clean":"2023-10-16T15:33:46.606555+0000","last_became_active":"2023-10-16T15:33:45.634245+0000","last_became_peered":"2023-10-16T15:33:45.634245+0000","last_unstale":"2023-10-16T15:33:46.606555+0000","last_undegraded":"2023-10-16T15:33:46.606555+0000","last_fullsized":"2023-10-16T15:33:46.606555+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:26:22.781035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049710+0000","last_change":"2023-10-16T15:33:45.622747+0000","last_active":"2023-10-16T15:33:47.049710+0000","last_peered":"2023-10-16T15:33:47.049710+0000","last_clean":"2023-10-16T15:33:47.049710+0000","last_became_active":"2023-10-16T15:33:45.622454+0000","last_became_peered":"2023-10-16T15:33:45.622454+0000","last_unstale":"2023-10-16T15:33:47.049710+0000","last_undegraded":"2023-10-16T15:33:47.049710+0000","last_fullsized":"2023-10-16T15:33:47.049710+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:56.987561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051510+0000","last_change":"2023-10-16T15:33:45.620154+0000","last_active":"2023-10-16T15:33:47.051510+0000","last_peered":"2023-10-16T15:33:47.051510+0000","last_clean":"2023-10-16T15:33:47.051510+0000","last_became_active":"2023-10-16T15:33:45.619856+0000","last_became_peered":"2023-10-16T15:33:45.619856+0000","last_unstale":"2023-10-16T15:33:47.051510+0000","last_undegraded":"2023-10-16T15:33:47.051510+0000","last_fullsized":"2023-10-16T15:33:47.051510+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:58.057873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050337+0000","last_change":"2023-10-16T15:33:45.644367+0000","last_active":"2023-10-16T15:33:47.050337+0000","last_peered":"2023-10-16T15:33:47.050337+0000","last_clean":"2023-10-16T15:33:47.050337+0000","last_became_active":"2023-10-16T15:33:45.644011+0000","last_became_peered":"2023-10-16T15:33:45.644011+0000","last_unstale":"2023-10-16T15:33:47.050337+0000","last_undegraded":"2023-10-16T15:33:47.050337+0000","last_fullsized":"2023-10-16T15:33:47.050337+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:17.818210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602649+0000","last_change":"2023-10-16T15:33:45.648341+0000","last_active":"2023-10-16T15:33:46.602649+0000","last_peered":"2023-10-16T15:33:46.602649+0000","last_clean":"2023-10-16T15:33:46.602649+0000","last_became_active":"2023-10-16T15:33:45.647984+0000","last_became_peered":"2023-10-16T15:33:45.647984+0000","last_unstale":"2023-10-16T15:33:46.602649+0000","last_undegraded":"2023-10-16T15:33:46.602649+0000","last_fullsized":"2023-10-16T15:33:46.602649+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:43.968735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050204+0000","last_change":"2023-10-16T15:33:45.647365+0000","last_active":"2023-10-16T15:33:47.050204+0000","last_peered":"2023-10-16T15:33:47.050204+0000","last_clean":"2023-10-16T15:33:47.050204+0000","last_became_active":"2023-10-16T15:33:45.647012+0000","last_became_peered":"2023-10-16T15:33:45.647012+0000","last_unstale":"2023-10-16T15:33:47.050204+0000","last_undegraded":"2023-10-16T15:33:47.050204+0000","last_fullsized":"2023-10-16T15:33:47.050204+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:38.801957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050271+0000","last_change":"2023-10-16T15:33:45.618092+0000","last_active":"2023-10-16T15:33:47.050271+0000","last_peered":"2023-10-16T15:33:47.050271+0000","last_clean":"2023-10-16T15:33:47.050271+0000","last_became_active":"2023-10-16T15:33:45.615285+0000","last_became_peered":"2023-10-16T15:33:45.615285+0000","last_unstale":"2023-10-16T15:33:47.050271+0000","last_undegraded":"2023-10-16T15:33:47.050271+0000","last_fullsized":"2023-10-16T15:33:47.050271+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:48.196062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047339+0000","last_change":"2023-10-16T15:33:45.633670+0000","last_active":"2023-10-16T15:33:47.047339+0000","last_peered":"2023-10-16T15:33:47.047339+0000","last_clean":"2023-10-16T15:33:47.047339+0000","last_became_active":"2023-10-16T15:33:45.633331+0000","last_became_peered":"2023-10-16T15:33:45.633331+0000","last_unstale":"2023-10-16T15:33:47.047339+0000","last_undegraded":"2023-10-16T15:33:47.047339+0000","last_fullsized":"2023-10-16T15:33:47.047339+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:30.580311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049044+0000","last_change":"2023-10-16T15:33:45.634674+0000","last_active":"2023-10-16T15:33:47.049044+0000","last_peered":"2023-10-16T15:33:47.049044+0000","last_clean":"2023-10-16T15:33:47.049044+0000","last_became_active":"2023-10-16T15:33:45.630512+0000","last_became_peered":"2023-10-16T15:33:45.630512+0000","last_unstale":"2023-10-16T15:33:47.049044+0000","last_undegraded":"2023-10-16T15:33:47.049044+0000","last_fullsized":"2023-10-16T15:33:47.049044+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:18.424954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608382+0000","last_change":"2023-10-16T15:33:45.635898+0000","last_active":"2023-10-16T15:33:46.608382+0000","last_peered":"2023-10-16T15:33:46.608382+0000","last_clean":"2023-10-16T15:33:46.608382+0000","last_became_active":"2023-10-16T15:33:45.634370+0000","last_became_peered":"2023-10-16T15:33:45.634370+0000","last_unstale":"2023-10-16T15:33:46.608382+0000","last_undegraded":"2023-10-16T15:33:46.608382+0000","last_fullsized":"2023-10-16T15:33:46.608382+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:49.417462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"178'4","reported_seq":16,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609495+0000","last_change":"2023-10-16T15:33:44.624815+0000","last_active":"2023-10-16T15:33:46.609495+0000","last_peered":"2023-10-16T15:33:46.609495+0000","last_clean":"2023-10-16T15:33:46.609495+0000","last_became_active":"2023-10-16T15:33:44.624498+0000","last_became_peered":"2023-10-16T15:33:44.624498+0000","last_unstale":"2023-10-16T15:33:46.609495+0000","last_undegraded":"2023-10-16T15:33:46.609495+0000","last_fullsized":"2023-10-16T15:33:46.609495+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:44.322839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048765+0000","last_change":"2023-10-16T15:33:45.631693+0000","last_active":"2023-10-16T15:33:47.048765+0000","last_peered":"2023-10-16T15:33:47.048765+0000","last_clean":"2023-10-16T15:33:47.048765+0000","last_became_active":"2023-10-16T15:33:45.628106+0000","last_became_peered":"2023-10-16T15:33:45.628106+0000","last_unstale":"2023-10-16T15:33:47.048765+0000","last_undegraded":"2023-10-16T15:33:47.048765+0000","last_fullsized":"2023-10-16T15:33:47.048765+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:35.644993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048123+0000","last_change":"2023-10-16T15:33:44.623092+0000","last_active":"2023-10-16T15:33:47.048123+0000","last_peered":"2023-10-16T15:33:47.048123+0000","last_clean":"2023-10-16T15:33:47.048123+0000","last_became_active":"2023-10-16T15:33:44.622718+0000","last_became_peered":"2023-10-16T15:33:44.622718+0000","last_unstale":"2023-10-16T15:33:47.048123+0000","last_undegraded":"2023-10-16T15:33:47.048123+0000","last_fullsized":"2023-10-16T15:33:47.048123+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:28.342455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049996+0000","last_change":"2023-10-16T15:33:45.625971+0000","last_active":"2023-10-16T15:33:47.049996+0000","last_peered":"2023-10-16T15:33:47.049996+0000","last_clean":"2023-10-16T15:33:47.049996+0000","last_became_active":"2023-10-16T15:33:45.625706+0000","last_became_peered":"2023-10-16T15:33:45.625706+0000","last_unstale":"2023-10-16T15:33:47.049996+0000","last_undegraded":"2023-10-16T15:33:47.049996+0000","last_fullsized":"2023-10-16T15:33:47.049996+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:34:01.964835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050821+0000","last_change":"2023-10-16T15:33:44.614459+0000","last_active":"2023-10-16T15:33:47.050821+0000","last_peered":"2023-10-16T15:33:47.050821+0000","last_clean":"2023-10-16T15:33:47.050821+0000","last_became_active":"2023-10-16T15:33:44.614040+0000","last_became_peered":"2023-10-16T15:33:44.614040+0000","last_unstale":"2023-10-16T15:33:47.050821+0000","last_undegraded":"2023-10-16T15:33:47.050821+0000","last_fullsized":"2023-10-16T15:33:47.050821+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:57.041308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606767+0000","last_change":"2023-10-16T15:33:45.633494+0000","last_active":"2023-10-16T15:33:46.606767+0000","last_peered":"2023-10-16T15:33:46.606767+0000","last_clean":"2023-10-16T15:33:46.606767+0000","last_became_active":"2023-10-16T15:33:45.633107+0000","last_became_peered":"2023-10-16T15:33:45.633107+0000","last_unstale":"2023-10-16T15:33:46.606767+0000","last_undegraded":"2023-10-16T15:33:46.606767+0000","last_fullsized":"2023-10-16T15:33:46.606767+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:49.106596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050915+0000","last_change":"2023-10-16T15:33:44.615061+0000","last_active":"2023-10-16T15:33:47.050915+0000","last_peered":"2023-10-16T15:33:47.050915+0000","last_clean":"2023-10-16T15:33:47.050915+0000","last_became_active":"2023-10-16T15:33:44.613917+0000","last_became_peered":"2023-10-16T15:33:44.613917+0000","last_unstale":"2023-10-16T15:33:47.050915+0000","last_undegraded":"2023-10-16T15:33:47.050915+0000","last_fullsized":"2023-10-16T15:33:47.050915+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:25.823449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047595+0000","last_change":"2023-10-16T15:33:45.645650+0000","last_active":"2023-10-16T15:33:47.047595+0000","last_peered":"2023-10-16T15:33:47.047595+0000","last_clean":"2023-10-16T15:33:47.047595+0000","last_became_active":"2023-10-16T15:33:45.645163+0000","last_became_peered":"2023-10-16T15:33:45.645163+0000","last_unstale":"2023-10-16T15:33:47.047595+0000","last_undegraded":"2023-10-16T15:33:47.047595+0000","last_fullsized":"2023-10-16T15:33:47.047595+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:20.012447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049345+0000","last_change":"2023-10-16T15:33:44.617095+0000","last_active":"2023-10-16T15:33:47.049345+0000","last_peered":"2023-10-16T15:33:47.049345+0000","last_clean":"2023-10-16T15:33:47.049345+0000","last_became_active":"2023-10-16T15:33:44.616809+0000","last_became_peered":"2023-10-16T15:33:44.616809+0000","last_unstale":"2023-10-16T15:33:47.049345+0000","last_undegraded":"2023-10-16T15:33:47.049345+0000","last_fullsized":"2023-10-16T15:33:47.049345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:31.600826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050023+0000","last_change":"2023-10-16T15:33:45.646207+0000","last_active":"2023-10-16T15:33:47.050023+0000","last_peered":"2023-10-16T15:33:47.050023+0000","last_clean":"2023-10-16T15:33:47.050023+0000","last_became_active":"2023-10-16T15:33:45.645849+0000","last_became_peered":"2023-10-16T15:33:45.645849+0000","last_unstale":"2023-10-16T15:33:47.050023+0000","last_undegraded":"2023-10-16T15:33:47.050023+0000","last_fullsized":"2023-10-16T15:33:47.050023+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:23.278967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049789+0000","last_change":"2023-10-16T15:33:44.631064+0000","last_active":"2023-10-16T15:33:47.049789+0000","last_peered":"2023-10-16T15:33:47.049789+0000","last_clean":"2023-10-16T15:33:47.049789+0000","last_became_active":"2023-10-16T15:33:44.630427+0000","last_became_peered":"2023-10-16T15:33:44.630427+0000","last_unstale":"2023-10-16T15:33:47.049789+0000","last_undegraded":"2023-10-16T15:33:47.049789+0000","last_fullsized":"2023-10-16T15:33:47.049789+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:28.625614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047318+0000","last_change":"2023-10-16T15:33:45.634611+0000","last_active":"2023-10-16T15:33:47.047318+0000","last_peered":"2023-10-16T15:33:47.047318+0000","last_clean":"2023-10-16T15:33:47.047318+0000","last_became_active":"2023-10-16T15:33:45.634328+0000","last_became_peered":"2023-10-16T15:33:45.634328+0000","last_unstale":"2023-10-16T15:33:47.047318+0000","last_undegraded":"2023-10-16T15:33:47.047318+0000","last_fullsized":"2023-10-16T15:33:47.047318+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:59.726928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049055+0000","last_change":"2023-10-16T15:33:44.625645+0000","last_active":"2023-10-16T15:33:47.049055+0000","last_peered":"2023-10-16T15:33:47.049055+0000","last_clean":"2023-10-16T15:33:47.049055+0000","last_became_active":"2023-10-16T15:33:44.625186+0000","last_became_peered":"2023-10-16T15:33:44.625186+0000","last_unstale":"2023-10-16T15:33:47.049055+0000","last_undegraded":"2023-10-16T15:33:47.049055+0000","last_fullsized":"2023-10-16T15:33:47.049055+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:55.122219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049149+0000","last_change":"2023-10-16T15:33:45.637588+0000","last_active":"2023-10-16T15:33:47.049149+0000","last_peered":"2023-10-16T15:33:47.049149+0000","last_clean":"2023-10-16T15:33:47.049149+0000","last_became_active":"2023-10-16T15:33:45.637172+0000","last_became_peered":"2023-10-16T15:33:45.637172+0000","last_unstale":"2023-10-16T15:33:47.049149+0000","last_undegraded":"2023-10-16T15:33:47.049149+0000","last_fullsized":"2023-10-16T15:33:47.049149+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:22.745923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048918+0000","last_change":"2023-10-16T15:33:44.624433+0000","last_active":"2023-10-16T15:33:47.048918+0000","last_peered":"2023-10-16T15:33:47.048918+0000","last_clean":"2023-10-16T15:33:47.048918+0000","last_became_active":"2023-10-16T15:33:44.624058+0000","last_became_peered":"2023-10-16T15:33:44.624058+0000","last_unstale":"2023-10-16T15:33:47.048918+0000","last_undegraded":"2023-10-16T15:33:47.048918+0000","last_fullsized":"2023-10-16T15:33:47.048918+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:04.068506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051184+0000","last_change":"2023-10-16T15:33:45.636497+0000","last_active":"2023-10-16T15:33:47.051184+0000","last_peered":"2023-10-16T15:33:47.051184+0000","last_clean":"2023-10-16T15:33:47.051184+0000","last_became_active":"2023-10-16T15:33:45.634140+0000","last_became_peered":"2023-10-16T15:33:45.634140+0000","last_unstale":"2023-10-16T15:33:47.051184+0000","last_undegraded":"2023-10-16T15:33:47.051184+0000","last_fullsized":"2023-10-16T15:33:47.051184+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:04.527775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048916+0000","last_change":"2023-10-16T15:33:44.612045+0000","last_active":"2023-10-16T15:33:47.048916+0000","last_peered":"2023-10-16T15:33:47.048916+0000","last_clean":"2023-10-16T15:33:47.048916+0000","last_became_active":"2023-10-16T15:33:44.611769+0000","last_became_peered":"2023-10-16T15:33:44.611769+0000","last_unstale":"2023-10-16T15:33:47.048916+0000","last_undegraded":"2023-10-16T15:33:47.048916+0000","last_fullsized":"2023-10-16T15:33:47.048916+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:22.479954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609228+0000","last_change":"2023-10-16T15:33:45.636660+0000","last_active":"2023-10-16T15:33:46.609228+0000","last_peered":"2023-10-16T15:33:46.609228+0000","last_clean":"2023-10-16T15:33:46.609228+0000","last_became_active":"2023-10-16T15:33:45.636376+0000","last_became_peered":"2023-10-16T15:33:45.636376+0000","last_unstale":"2023-10-16T15:33:46.609228+0000","last_undegraded":"2023-10-16T15:33:46.609228+0000","last_fullsized":"2023-10-16T15:33:46.609228+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:10.710971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609004+0000","last_change":"2023-10-16T15:33:44.629352+0000","last_active":"2023-10-16T15:33:46.609004+0000","last_peered":"2023-10-16T15:33:46.609004+0000","last_clean":"2023-10-16T15:33:46.609004+0000","last_became_active":"2023-10-16T15:33:44.629032+0000","last_became_peered":"2023-10-16T15:33:44.629032+0000","last_unstale":"2023-10-16T15:33:46.609004+0000","last_undegraded":"2023-10-16T15:33:46.609004+0000","last_fullsized":"2023-10-16T15:33:46.609004+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:57.812460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050711+0000","last_change":"2023-10-16T15:33:45.633788+0000","last_active":"2023-10-16T15:33:47.050711+0000","last_peered":"2023-10-16T15:33:47.050711+0000","last_clean":"2023-10-16T15:33:47.050711+0000","last_became_active":"2023-10-16T15:33:45.631665+0000","last_became_peered":"2023-10-16T15:33:45.631665+0000","last_unstale":"2023-10-16T15:33:47.050711+0000","last_undegraded":"2023-10-16T15:33:47.050711+0000","last_fullsized":"2023-10-16T15:33:47.050711+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:02.588048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"178'6","reported_seq":18,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602924+0000","last_change":"2023-10-16T15:33:44.614425+0000","last_active":"2023-10-16T15:33:46.602924+0000","last_peered":"2023-10-16T15:33:46.602924+0000","last_clean":"2023-10-16T15:33:46.602924+0000","last_became_active":"2023-10-16T15:33:44.614156+0000","last_became_peered":"2023-10-16T15:33:44.614156+0000","last_unstale":"2023-10-16T15:33:46.602924+0000","last_undegraded":"2023-10-16T15:33:46.602924+0000","last_fullsized":"2023-10-16T15:33:46.602924+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:40.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608120+0000","last_change":"2023-10-16T15:33:45.644560+0000","last_active":"2023-10-16T15:33:46.608120+0000","last_peered":"2023-10-16T15:33:46.608120+0000","last_clean":"2023-10-16T15:33:46.608120+0000","last_became_active":"2023-10-16T15:33:45.643116+0000","last_became_peered":"2023-10-16T15:33:45.643116+0000","last_unstale":"2023-10-16T15:33:46.608120+0000","last_undegraded":"2023-10-16T15:33:46.608120+0000","last_fullsized":"2023-10-16T15:33:46.608120+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:37.009681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609548+0000","last_change":"2023-10-16T15:33:44.628463+0000","last_active":"2023-10-16T15:33:46.609548+0000","last_peered":"2023-10-16T15:33:46.609548+0000","last_clean":"2023-10-16T15:33:46.609548+0000","last_became_active":"2023-10-16T15:33:44.626707+0000","last_became_peered":"2023-10-16T15:33:44.626707+0000","last_unstale":"2023-10-16T15:33:46.609548+0000","last_undegraded":"2023-10-16T15:33:46.609548+0000","last_fullsized":"2023-10-16T15:33:46.609548+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:27.705500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047560+0000","last_change":"2023-10-16T15:33:45.625564+0000","last_active":"2023-10-16T15:33:47.047560+0000","last_peered":"2023-10-16T15:33:47.047560+0000","last_clean":"2023-10-16T15:33:47.047560+0000","last_became_active":"2023-10-16T15:33:45.623750+0000","last_became_peered":"2023-10-16T15:33:45.623750+0000","last_unstale":"2023-10-16T15:33:47.047560+0000","last_undegraded":"2023-10-16T15:33:47.047560+0000","last_fullsized":"2023-10-16T15:33:47.047560+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:54.807078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608235+0000","last_change":"2023-10-16T15:33:44.629646+0000","last_active":"2023-10-16T15:33:46.608235+0000","last_peered":"2023-10-16T15:33:46.608235+0000","last_clean":"2023-10-16T15:33:46.608235+0000","last_became_active":"2023-10-16T15:33:44.629246+0000","last_became_peered":"2023-10-16T15:33:44.629246+0000","last_unstale":"2023-10-16T15:33:46.608235+0000","last_undegraded":"2023-10-16T15:33:46.608235+0000","last_fullsized":"2023-10-16T15:33:46.608235+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:06.207602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049809+0000","last_change":"2023-10-16T15:33:45.643396+0000","last_active":"2023-10-16T15:33:47.049809+0000","last_peered":"2023-10-16T15:33:47.049809+0000","last_clean":"2023-10-16T15:33:47.049809+0000","last_became_active":"2023-10-16T15:33:45.643031+0000","last_became_peered":"2023-10-16T15:33:45.643031+0000","last_unstale":"2023-10-16T15:33:47.049809+0000","last_undegraded":"2023-10-16T15:33:47.049809+0000","last_fullsized":"2023-10-16T15:33:47.049809+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:38.332204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049972+0000","last_change":"2023-10-16T15:33:44.612338+0000","last_active":"2023-10-16T15:33:47.049972+0000","last_peered":"2023-10-16T15:33:47.049972+0000","last_clean":"2023-10-16T15:33:47.049972+0000","last_became_active":"2023-10-16T15:33:44.612061+0000","last_became_peered":"2023-10-16T15:33:44.612061+0000","last_unstale":"2023-10-16T15:33:47.049972+0000","last_undegraded":"2023-10-16T15:33:47.049972+0000","last_fullsized":"2023-10-16T15:33:47.049972+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:48.425942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608616+0000","last_change":"2023-10-16T15:33:45.642815+0000","last_active":"2023-10-16T15:33:46.608616+0000","last_peered":"2023-10-16T15:33:46.608616+0000","last_clean":"2023-10-16T15:33:46.608616+0000","last_became_active":"2023-10-16T15:33:45.642550+0000","last_became_peered":"2023-10-16T15:33:45.642550+0000","last_unstale":"2023-10-16T15:33:46.608616+0000","last_undegraded":"2023-10-16T15:33:46.608616+0000","last_fullsized":"2023-10-16T15:33:46.608616+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:29.642179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"178'8","reported_seq":20,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049207+0000","last_change":"2023-10-16T15:33:44.626796+0000","last_active":"2023-10-16T15:33:47.049207+0000","last_peered":"2023-10-16T15:33:47.049207+0000","last_clean":"2023-10-16T15:33:47.049207+0000","last_became_active":"2023-10-16T15:33:44.626067+0000","last_became_peered":"2023-10-16T15:33:44.626067+0000","last_unstale":"2023-10-16T15:33:47.049207+0000","last_undegraded":"2023-10-16T15:33:47.049207+0000","last_fullsized":"2023-10-16T15:33:47.049207+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:48.425942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608629+0000","last_change":"2023-10-16T15:33:45.634025+0000","last_active":"2023-10-16T15:33:46.608629+0000","last_peered":"2023-10-16T15:33:46.608629+0000","last_clean":"2023-10-16T15:33:46.608629+0000","last_became_active":"2023-10-16T15:33:45.632514+0000","last_became_peered":"2023-10-16T15:33:45.632514+0000","last_unstale":"2023-10-16T15:33:46.608629+0000","last_undegraded":"2023-10-16T15:33:46.608629+0000","last_fullsized":"2023-10-16T15:33:46.608629+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:58:56.393005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049904+0000","last_change":"2023-10-16T15:33:44.630760+0000","last_active":"2023-10-16T15:33:47.049904+0000","last_peered":"2023-10-16T15:33:47.049904+0000","last_clean":"2023-10-16T15:33:47.049904+0000","last_became_active":"2023-10-16T15:33:44.629787+0000","last_became_peered":"2023-10-16T15:33:44.629787+0000","last_unstale":"2023-10-16T15:33:47.049904+0000","last_undegraded":"2023-10-16T15:33:47.049904+0000","last_fullsized":"2023-10-16T15:33:47.049904+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:50.586092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049973+0000","last_change":"2023-10-16T15:33:45.628893+0000","last_active":"2023-10-16T15:33:47.049973+0000","last_peered":"2023-10-16T15:33:47.049973+0000","last_clean":"2023-10-16T15:33:47.049973+0000","last_became_active":"2023-10-16T15:33:45.628446+0000","last_became_peered":"2023-10-16T15:33:45.628446+0000","last_unstale":"2023-10-16T15:33:47.049973+0000","last_undegraded":"2023-10-16T15:33:47.049973+0000","last_fullsized":"2023-10-16T15:33:47.049973+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:28.325081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048367+0000","last_change":"2023-10-16T15:33:44.625058+0000","last_active":"2023-10-16T15:33:47.048367+0000","last_peered":"2023-10-16T15:33:47.048367+0000","last_clean":"2023-10-16T15:33:47.048367+0000","last_became_active":"2023-10-16T15:33:44.624777+0000","last_became_peered":"2023-10-16T15:33:44.624777+0000","last_unstale":"2023-10-16T15:33:47.048367+0000","last_undegraded":"2023-10-16T15:33:47.048367+0000","last_fullsized":"2023-10-16T15:33:47.048367+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:04.520105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609116+0000","last_change":"2023-10-16T15:33:45.643776+0000","last_active":"2023-10-16T15:33:46.609116+0000","last_peered":"2023-10-16T15:33:46.609116+0000","last_clean":"2023-10-16T15:33:46.609116+0000","last_became_active":"2023-10-16T15:33:45.643234+0000","last_became_peered":"2023-10-16T15:33:45.643234+0000","last_unstale":"2023-10-16T15:33:46.609116+0000","last_undegraded":"2023-10-16T15:33:46.609116+0000","last_fullsized":"2023-10-16T15:33:46.609116+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:51.602657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050944+0000","last_change":"2023-10-16T15:33:44.623120+0000","last_active":"2023-10-16T15:33:47.050944+0000","last_peered":"2023-10-16T15:33:47.050944+0000","last_clean":"2023-10-16T15:33:47.050944+0000","last_became_active":"2023-10-16T15:33:44.622823+0000","last_became_peered":"2023-10-16T15:33:44.622823+0000","last_unstale":"2023-10-16T15:33:47.050944+0000","last_undegraded":"2023-10-16T15:33:47.050944+0000","last_fullsized":"2023-10-16T15:33:47.050944+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:24.543095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050103+0000","last_change":"2023-10-16T15:33:45.641454+0000","last_active":"2023-10-16T15:33:47.050103+0000","last_peered":"2023-10-16T15:33:47.050103+0000","last_clean":"2023-10-16T15:33:47.050103+0000","last_became_active":"2023-10-16T15:33:45.641165+0000","last_became_peered":"2023-10-16T15:33:45.641165+0000","last_unstale":"2023-10-16T15:33:47.050103+0000","last_undegraded":"2023-10-16T15:33:47.050103+0000","last_fullsized":"2023-10-16T15:33:47.050103+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:10.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047917+0000","last_change":"2023-10-16T15:33:44.611242+0000","last_active":"2023-10-16T15:33:47.047917+0000","last_peered":"2023-10-16T15:33:47.047917+0000","last_clean":"2023-10-16T15:33:47.047917+0000","last_became_active":"2023-10-16T15:33:44.610289+0000","last_became_peered":"2023-10-16T15:33:44.610289+0000","last_unstale":"2023-10-16T15:33:47.047917+0000","last_undegraded":"2023-10-16T15:33:47.047917+0000","last_fullsized":"2023-10-16T15:33:47.047917+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:39.260215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050657+0000","last_change":"2023-10-16T15:33:45.617665+0000","last_active":"2023-10-16T15:33:47.050657+0000","last_peered":"2023-10-16T15:33:47.050657+0000","last_clean":"2023-10-16T15:33:47.050657+0000","last_became_active":"2023-10-16T15:33:45.614269+0000","last_became_peered":"2023-10-16T15:33:45.614269+0000","last_unstale":"2023-10-16T15:33:47.050657+0000","last_undegraded":"2023-10-16T15:33:47.050657+0000","last_fullsized":"2023-10-16T15:33:47.050657+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:53.204545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602441+0000","last_change":"2023-10-16T15:33:44.626790+0000","last_active":"2023-10-16T15:33:46.602441+0000","last_peered":"2023-10-16T15:33:46.602441+0000","last_clean":"2023-10-16T15:33:46.602441+0000","last_became_active":"2023-10-16T15:33:44.626331+0000","last_became_peered":"2023-10-16T15:33:44.626331+0000","last_unstale":"2023-10-16T15:33:46.602441+0000","last_undegraded":"2023-10-16T15:33:46.602441+0000","last_fullsized":"2023-10-16T15:33:46.602441+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:40.435220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602298+0000","last_change":"2023-10-16T15:33:45.630435+0000","last_active":"2023-10-16T15:33:46.602298+0000","last_peered":"2023-10-16T15:33:46.602298+0000","last_clean":"2023-10-16T15:33:46.602298+0000","last_became_active":"2023-10-16T15:33:45.630128+0000","last_became_peered":"2023-10-16T15:33:45.630128+0000","last_unstale":"2023-10-16T15:33:46.602298+0000","last_undegraded":"2023-10-16T15:33:46.602298+0000","last_fullsized":"2023-10-16T15:33:46.602298+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:11.925522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607757+0000","last_change":"2023-10-16T15:33:44.624040+0000","last_active":"2023-10-16T15:33:46.607757+0000","last_peered":"2023-10-16T15:33:46.607757+0000","last_clean":"2023-10-16T15:33:46.607757+0000","last_became_active":"2023-10-16T15:33:44.623740+0000","last_became_peered":"2023-10-16T15:33:44.623740+0000","last_unstale":"2023-10-16T15:33:46.607757+0000","last_undegraded":"2023-10-16T15:33:46.607757+0000","last_fullsized":"2023-10-16T15:33:46.607757+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:48.735149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049149+0000","last_change":"2023-10-16T15:33:45.627875+0000","last_active":"2023-10-16T15:33:47.049149+0000","last_peered":"2023-10-16T15:33:47.049149+0000","last_clean":"2023-10-16T15:33:47.049149+0000","last_became_active":"2023-10-16T15:33:45.624042+0000","last_became_peered":"2023-10-16T15:33:45.624042+0000","last_unstale":"2023-10-16T15:33:47.049149+0000","last_undegraded":"2023-10-16T15:33:47.049149+0000","last_fullsized":"2023-10-16T15:33:47.049149+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:28.112469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607557+0000","last_change":"2023-10-16T15:33:44.626226+0000","last_active":"2023-10-16T15:33:46.607557+0000","last_peered":"2023-10-16T15:33:46.607557+0000","last_clean":"2023-10-16T15:33:46.607557+0000","last_became_active":"2023-10-16T15:33:44.625346+0000","last_became_peered":"2023-10-16T15:33:44.625346+0000","last_unstale":"2023-10-16T15:33:46.607557+0000","last_undegraded":"2023-10-16T15:33:46.607557+0000","last_fullsized":"2023-10-16T15:33:46.607557+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:11.383757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602285+0000","last_change":"2023-10-16T15:33:45.636311+0000","last_active":"2023-10-16T15:33:46.602285+0000","last_peered":"2023-10-16T15:33:46.602285+0000","last_clean":"2023-10-16T15:33:46.602285+0000","last_became_active":"2023-10-16T15:33:45.635197+0000","last_became_peered":"2023-10-16T15:33:45.635197+0000","last_unstale":"2023-10-16T15:33:46.602285+0000","last_undegraded":"2023-10-16T15:33:46.602285+0000","last_fullsized":"2023-10-16T15:33:46.602285+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:37.440753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607394+0000","last_change":"2023-10-16T15:33:44.624605+0000","last_active":"2023-10-16T15:33:46.607394+0000","last_peered":"2023-10-16T15:33:46.607394+0000","last_clean":"2023-10-16T15:33:46.607394+0000","last_became_active":"2023-10-16T15:33:44.623368+0000","last_became_peered":"2023-10-16T15:33:44.623368+0000","last_unstale":"2023-10-16T15:33:46.607394+0000","last_undegraded":"2023-10-16T15:33:46.607394+0000","last_fullsized":"2023-10-16T15:33:46.607394+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:42.262105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051033+0000","last_change":"2023-10-16T15:33:45.641590+0000","last_active":"2023-10-16T15:33:47.051033+0000","last_peered":"2023-10-16T15:33:47.051033+0000","last_clean":"2023-10-16T15:33:47.051033+0000","last_became_active":"2023-10-16T15:33:45.641326+0000","last_became_peered":"2023-10-16T15:33:45.641326+0000","last_unstale":"2023-10-16T15:33:47.051033+0000","last_undegraded":"2023-10-16T15:33:47.051033+0000","last_fullsized":"2023-10-16T15:33:47.051033+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:13:24.431946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609677+0000","last_change":"2023-10-16T15:33:44.632364+0000","last_active":"2023-10-16T15:33:46.609677+0000","last_peered":"2023-10-16T15:33:46.609677+0000","last_clean":"2023-10-16T15:33:46.609677+0000","last_became_active":"2023-10-16T15:33:44.632030+0000","last_became_peered":"2023-10-16T15:33:44.632030+0000","last_unstale":"2023-10-16T15:33:46.609677+0000","last_undegraded":"2023-10-16T15:33:46.609677+0000","last_fullsized":"2023-10-16T15:33:46.609677+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:05.462762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.610142+0000","last_change":"2023-10-16T15:33:45.648232+0000","last_active":"2023-10-16T15:33:46.610142+0000","last_peered":"2023-10-16T15:33:46.610142+0000","last_clean":"2023-10-16T15:33:46.610142+0000","last_became_active":"2023-10-16T15:33:45.647853+0000","last_became_peered":"2023-10-16T15:33:45.647853+0000","last_unstale":"2023-10-16T15:33:46.610142+0000","last_undegraded":"2023-10-16T15:33:46.610142+0000","last_fullsized":"2023-10-16T15:33:46.610142+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:22.745923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051623+0000","last_change":"2023-10-16T15:33:44.624586+0000","last_active":"2023-10-16T15:33:47.051623+0000","last_peered":"2023-10-16T15:33:47.051623+0000","last_clean":"2023-10-16T15:33:47.051623+0000","last_became_active":"2023-10-16T15:33:44.624048+0000","last_became_peered":"2023-10-16T15:33:44.624048+0000","last_unstale":"2023-10-16T15:33:47.051623+0000","last_undegraded":"2023-10-16T15:33:47.051623+0000","last_fullsized":"2023-10-16T15:33:47.051623+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:27.308516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047882+0000","last_change":"2023-10-16T15:33:45.630012+0000","last_active":"2023-10-16T15:33:47.047882+0000","last_peered":"2023-10-16T15:33:47.047882+0000","last_clean":"2023-10-16T15:33:47.047882+0000","last_became_active":"2023-10-16T15:33:45.629624+0000","last_became_peered":"2023-10-16T15:33:45.629624+0000","last_unstale":"2023-10-16T15:33:47.047882+0000","last_undegraded":"2023-10-16T15:33:47.047882+0000","last_fullsized":"2023-10-16T15:33:47.047882+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:15.110341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051500+0000","last_change":"2023-10-16T15:33:44.613942+0000","last_active":"2023-10-16T15:33:47.051500+0000","last_peered":"2023-10-16T15:33:47.051500+0000","last_clean":"2023-10-16T15:33:47.051500+0000","last_became_active":"2023-10-16T15:33:44.613627+0000","last_became_peered":"2023-10-16T15:33:44.613627+0000","last_unstale":"2023-10-16T15:33:47.051500+0000","last_undegraded":"2023-10-16T15:33:47.051500+0000","last_fullsized":"2023-10-16T15:33:47.051500+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:55.970996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051360+0000","last_change":"2023-10-16T15:33:45.615849+0000","last_active":"2023-10-16T15:33:47.051360+0000","last_peered":"2023-10-16T15:33:47.051360+0000","last_clean":"2023-10-16T15:33:47.051360+0000","last_became_active":"2023-10-16T15:33:45.615499+0000","last_became_peered":"2023-10-16T15:33:45.615499+0000","last_unstale":"2023-10-16T15:33:47.051360+0000","last_undegraded":"2023-10-16T15:33:47.051360+0000","last_fullsized":"2023-10-16T15:33:47.051360+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:32.384456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050397+0000","last_change":"2023-10-16T15:33:44.614110+0000","last_active":"2023-10-16T15:33:47.050397+0000","last_peered":"2023-10-16T15:33:47.050397+0000","last_clean":"2023-10-16T15:33:47.050397+0000","last_became_active":"2023-10-16T15:33:44.613801+0000","last_became_peered":"2023-10-16T15:33:44.613801+0000","last_unstale":"2023-10-16T15:33:47.050397+0000","last_undegraded":"2023-10-16T15:33:47.050397+0000","last_fullsized":"2023-10-16T15:33:47.050397+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:31.600826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048886+0000","last_change":"2023-10-16T15:33:45.629201+0000","last_active":"2023-10-16T15:33:47.048886+0000","last_peered":"2023-10-16T15:33:47.048886+0000","last_clean":"2023-10-16T15:33:47.048886+0000","last_became_active":"2023-10-16T15:33:45.628878+0000","last_became_peered":"2023-10-16T15:33:45.628878+0000","last_unstale":"2023-10-16T15:33:47.048886+0000","last_undegraded":"2023-10-16T15:33:47.048886+0000","last_fullsized":"2023-10-16T15:33:47.048886+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:19:51.985498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602451+0000","last_change":"2023-10-16T15:33:44.615954+0000","last_active":"2023-10-16T15:33:46.602451+0000","last_peered":"2023-10-16T15:33:46.602451+0000","last_clean":"2023-10-16T15:33:46.602451+0000","last_became_active":"2023-10-16T15:33:44.615222+0000","last_became_peered":"2023-10-16T15:33:44.615222+0000","last_unstale":"2023-10-16T15:33:46.602451+0000","last_undegraded":"2023-10-16T15:33:46.602451+0000","last_fullsized":"2023-10-16T15:33:46.602451+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:41.981941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050847+0000","last_change":"2023-10-16T15:33:45.648133+0000","last_active":"2023-10-16T15:33:47.050847+0000","last_peered":"2023-10-16T15:33:47.050847+0000","last_clean":"2023-10-16T15:33:47.050847+0000","last_became_active":"2023-10-16T15:33:45.647470+0000","last_became_peered":"2023-10-16T15:33:45.647470+0000","last_unstale":"2023-10-16T15:33:47.050847+0000","last_undegraded":"2023-10-16T15:33:47.050847+0000","last_fullsized":"2023-10-16T15:33:47.050847+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:45.847934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607977+0000","last_change":"2023-10-16T15:33:44.632417+0000","last_active":"2023-10-16T15:33:46.607977+0000","last_peered":"2023-10-16T15:33:46.607977+0000","last_clean":"2023-10-16T15:33:46.607977+0000","last_became_active":"2023-10-16T15:33:44.632102+0000","last_became_peered":"2023-10-16T15:33:44.632102+0000","last_unstale":"2023-10-16T15:33:46.607977+0000","last_undegraded":"2023-10-16T15:33:46.607977+0000","last_fullsized":"2023-10-16T15:33:46.607977+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:29.302375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606560+0000","last_change":"2023-10-16T15:33:45.622628+0000","last_active":"2023-10-16T15:33:46.606560+0000","last_peered":"2023-10-16T15:33:46.606560+0000","last_clean":"2023-10-16T15:33:46.606560+0000","last_became_active":"2023-10-16T15:33:45.622146+0000","last_became_peered":"2023-10-16T15:33:45.622146+0000","last_unstale":"2023-10-16T15:33:46.606560+0000","last_undegraded":"2023-10-16T15:33:46.606560+0000","last_fullsized":"2023-10-16T15:33:46.606560+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:03.101744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608836+0000","last_change":"2023-10-16T15:33:44.621647+0000","last_active":"2023-10-16T15:33:46.608836+0000","last_peered":"2023-10-16T15:33:46.608836+0000","last_clean":"2023-10-16T15:33:46.608836+0000","last_became_active":"2023-10-16T15:33:44.621337+0000","last_became_peered":"2023-10-16T15:33:44.621337+0000","last_unstale":"2023-10-16T15:33:46.608836+0000","last_undegraded":"2023-10-16T15:33:46.608836+0000","last_fullsized":"2023-10-16T15:33:46.608836+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:32:51.641585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049427+0000","last_change":"2023-10-16T15:33:45.636035+0000","last_active":"2023-10-16T15:33:47.049427+0000","last_peered":"2023-10-16T15:33:47.049427+0000","last_clean":"2023-10-16T15:33:47.049427+0000","last_became_active":"2023-10-16T15:33:45.634716+0000","last_became_peered":"2023-10-16T15:33:45.634716+0000","last_unstale":"2023-10-16T15:33:47.049427+0000","last_undegraded":"2023-10-16T15:33:47.049427+0000","last_fullsized":"2023-10-16T15:33:47.049427+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:53.880114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047296+0000","last_change":"2023-10-16T15:33:44.616284+0000","last_active":"2023-10-16T15:33:47.047296+0000","last_peered":"2023-10-16T15:33:47.047296+0000","last_clean":"2023-10-16T15:33:47.047296+0000","last_became_active":"2023-10-16T15:33:44.615654+0000","last_became_peered":"2023-10-16T15:33:44.615654+0000","last_unstale":"2023-10-16T15:33:47.047296+0000","last_undegraded":"2023-10-16T15:33:47.047296+0000","last_fullsized":"2023-10-16T15:33:47.047296+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:20.679846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602817+0000","last_change":"2023-10-16T15:33:45.634927+0000","last_active":"2023-10-16T15:33:46.602817+0000","last_peered":"2023-10-16T15:33:46.602817+0000","last_clean":"2023-10-16T15:33:46.602817+0000","last_became_active":"2023-10-16T15:33:45.634638+0000","last_became_peered":"2023-10-16T15:33:45.634638+0000","last_unstale":"2023-10-16T15:33:46.602817+0000","last_undegraded":"2023-10-16T15:33:46.602817+0000","last_fullsized":"2023-10-16T15:33:46.602817+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:50.175946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"178'3","reported_seq":15,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609833+0000","last_change":"2023-10-16T15:33:44.622869+0000","last_active":"2023-10-16T15:33:46.609833+0000","last_peered":"2023-10-16T15:33:46.609833+0000","last_clean":"2023-10-16T15:33:46.609833+0000","last_became_active":"2023-10-16T15:33:44.622580+0000","last_became_peered":"2023-10-16T15:33:44.622580+0000","last_unstale":"2023-10-16T15:33:46.609833+0000","last_undegraded":"2023-10-16T15:33:46.609833+0000","last_fullsized":"2023-10-16T15:33:46.609833+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:05.272611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"167'274","reported_seq":918,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-16T15:33:45.610197+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:33:45.610197+0000","last_peered":"2023-10-16T15:33:45.610197+0000","last_clean":"2023-10-16T15:33:45.610197+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:33:45.610197+0000","last_undegraded":"2023-10-16T15:33:45.610197+0000","last_fullsized":"2023-10-16T15:33:45.610197+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575772,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56164,"kb_used_data":2768,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315676,"statfs":{"total":96636764160,"available":96579252224,"internally_reserved":0,"allocated":2834432,"data_stored":1680591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1990000000000001,"15min":1.2110000000000001},"min":{"1min":0.86099999999999999,"5min":0.753,"15min":0.69099999999999995},"max":{"1min":1.6659999999999999,"5min":1.752,"15min":1.7849999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.159,"15min":1.1619999999999999},"min":{"1min":0.86799999999999999,"5min":0.78300000000000003,"15min":0.56999999999999995},"max":{"1min":1.867,"5min":1.867,"15min":1.905},"last":1.2829999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2390000000000001,"15min":1.224},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.80000000000000004},"max":{"1min":1.6879999999999999,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3660000000000001,"15min":1.337},"min":{"1min":1.0069999999999999,"5min":0.86299999999999999,"15min":0.78200000000000003},"max":{"1min":1.881,"5min":2.282,"15min":2.282},"last":1.3380000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.319,"15min":1.357},"min":{"1min":0.96199999999999997,"5min":0.91000000000000003,"15min":0.80200000000000005},"max":{"1min":1.7549999999999999,"5min":1.784,"15min":1.9039999999999999},"last":1.216},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3540000000000001,"15min":1.3320000000000001},"min":{"1min":0.97299999999999998,"5min":0.91700000000000004,"15min":0.82599999999999996},"max":{"1min":1.925,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.752}]},{"osd":3,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3640000000000001,"15min":1.3500000000000001},"min":{"1min":1.093,"5min":0.93400000000000005,"15min":0.90700000000000003},"max":{"1min":1.7390000000000001,"5min":2.528,"15min":2.528},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4710000000000001,"15min":1.4590000000000001},"min":{"1min":1.032,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.9410000000000001,"5min":1.956,"15min":1.962},"last":1.96}]},{"osd":4,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.4039999999999999,"15min":1.395},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.81699999999999995},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.954},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3919999999999999,"15min":1.3320000000000001},"min":{"1min":0.75900000000000001,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":1.9099999999999999,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":2.0329999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.502,"15min":1.4710000000000001},"min":{"1min":1.024,"5min":0.97599999999999998,"15min":0.89000000000000001},"max":{"1min":2.032,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.6240000000000001,"15min":1.5720000000000001},"min":{"1min":1.1850000000000001,"5min":0.92700000000000005,"15min":0.88500000000000001},"max":{"1min":2.0049999999999999,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":2.0880000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5209999999999999,"15min":1.4850000000000001},"min":{"1min":1.212,"5min":1.099,"15min":0.90700000000000003},"max":{"1min":2.133,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":2.1389999999999998},{"interface":"front","average":{"1min":1.673,"5min":1.645,"15min":1.6020000000000001},"min":{"1min":1.3140000000000001,"5min":1.0629999999999999,"15min":0.95099999999999996},"max":{"1min":2.0720000000000001,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":2.1920000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29216,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342624,"statfs":{"total":96636764160,"available":96606846976,"internally_reserved":0,"allocated":2244608,"data_stored":1089717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1890000000000001,"15min":1.1850000000000001},"min":{"1min":0.88400000000000001,"5min":0.77200000000000002,"15min":0.71899999999999997},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.556},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.282,"15min":1.264},"min":{"1min":0.84599999999999997,"5min":0.82099999999999995,"15min":0.80800000000000005},"max":{"1min":1.823,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.165}]},{"osd":1,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.252,"15min":1.2889999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.68600000000000005},"max":{"1min":1.653,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.335,"15min":1.3280000000000001},"min":{"1min":0.93100000000000005,"5min":0.874,"15min":0.85099999999999998},"max":{"1min":1.7,"5min":1.843,"15min":2.738},"last":1.843}]},{"osd":2,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2649999999999999,"15min":1.2829999999999999},"min":{"1min":0.93100000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.823,"5min":1.823,"15min":2.4039999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.405,"15min":1.3859999999999999},"min":{"1min":1.109,"5min":0.90400000000000003,"15min":0.89400000000000002},"max":{"1min":1.756,"5min":1.756,"15min":1.9139999999999999},"last":0.90200000000000002}]},{"osd":3,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3919999999999999,"15min":1.401},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.6100000000000001,"5min":1.8580000000000001,"15min":3.9820000000000002},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.383,"15min":1.379},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.92100000000000004},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.5489999999999999},"last":1.2210000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3160000000000001,"15min":1.3129999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":1.6879999999999999,"5min":1.9019999999999999,"15min":1.948},"last":1.766},{"interface":"front","average":{"1min":1.339,"5min":1.381,"15min":1.3799999999999999},"min":{"1min":0.88900000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7629999999999999,"5min":1.9159999999999999,"15min":4.6609999999999996},"last":1.4690000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.371,"15min":1.3540000000000001},"min":{"1min":1.0269999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.7130000000000001,"5min":1.9430000000000001,"15min":3.8450000000000002},"last":1.419},{"interface":"front","average":{"1min":1.508,"5min":1.5369999999999999,"15min":1.518},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.86299999999999999},"max":{"1min":1.897,"5min":2.02,"15min":2.8639999999999999},"last":1.677}]},{"osd":7,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5860000000000001,"15min":1.5920000000000001},"min":{"1min":1.359,"5min":1.0249999999999999,"15min":0.84799999999999998},"max":{"1min":1.8819999999999999,"5min":1.9610000000000001,"15min":4.5359999999999996},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.615,"15min":1.591},"min":{"1min":1.335,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.9330000000000001,"5min":2.1970000000000001,"15min":2.6619999999999999},"last":1.613}]}]},{"osd":5,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41688,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330152,"statfs":{"total":96636764160,"available":96594075648,"internally_reserved":0,"allocated":2236416,"data_stored":1089593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1990000000000001,"15min":1.1830000000000001},"min":{"1min":0.82299999999999995,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.254,"15min":1.252},"min":{"1min":0.85099999999999998,"5min":0.82899999999999996,"15min":0.78400000000000003},"max":{"1min":1.8240000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.131}]},{"osd":1,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.246,"15min":1.2370000000000001},"min":{"1min":0.97499999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3149999999999999,"15min":1.298},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.83699999999999997},"max":{"1min":1.7050000000000001,"5min":1.7949999999999999,"15min":1.905},"last":1.327}]},{"osd":2,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.278,"15min":1.2669999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.83099999999999996},"max":{"1min":1.883,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.399,"15min":1.399},"min":{"1min":1.0509999999999999,"5min":0.95599999999999996,"15min":0.91700000000000004},"max":{"1min":1.6850000000000001,"5min":2.073,"15min":2.3130000000000002},"last":0.96399999999999997}]},{"osd":3,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.327,"15min":1.325},"min":{"1min":0.997,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.9410000000000001,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.405,"5min":1.387,"15min":1.377},"min":{"1min":1.0700000000000001,"5min":0.94999999999999996,"15min":0.91800000000000004},"max":{"1min":1.673,"5min":2.1800000000000002,"15min":2.1800000000000002},"last":1.1930000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":1.032,"5min":0.78800000000000003,"15min":0.78500000000000003},"max":{"1min":1.5720000000000001,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.034},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4690000000000001,"15min":1.456},"min":{"1min":0.93999999999999995,"5min":0.83199999999999996,"15min":0.78400000000000003},"max":{"1min":1.7849999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.623}]},{"osd":6,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3740000000000001,"15min":1.3560000000000001},"min":{"1min":0.89000000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.7290000000000001,"5min":2.4569999999999999,"15min":2.508},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.452,"15min":1.4330000000000001},"min":{"1min":1.121,"5min":0.91300000000000003,"15min":0.89600000000000002},"max":{"1min":1.7749999999999999,"5min":2.9209999999999998,"15min":2.9790000000000001},"last":1.3660000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.5249999999999999,"15min":1.506},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.89600000000000002},"max":{"1min":1.734,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.6040000000000001,"15min":1.601},"min":{"1min":1.359,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":2.0150000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.399}]}]},{"osd":4,"up_from":13,"seq":55834575772,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59736,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312104,"statfs":{"total":96636764160,"available":96575594496,"internally_reserved":0,"allocated":2232320,"data_stored":1089571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2010000000000001,"15min":1.1930000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6220000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.1639999999999999,"15min":1.1830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82199999999999995,"15min":0.745},"max":{"1min":1.458,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.91800000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2729999999999999,"15min":1.2350000000000001},"min":{"1min":0.93000000000000005,"5min":0.88900000000000001,"15min":0.80400000000000005},"max":{"1min":1.583,"5min":1.78,"15min":1.948},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.272,"15min":1.276},"min":{"1min":0.83399999999999996,"5min":0.82099999999999995,"15min":0.80500000000000005},"max":{"1min":1.6910000000000001,"5min":1.7230000000000001,"15min":1.895},"last":1.119}]},{"osd":2,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4379999999999999,"15min":1.423},"min":{"1min":1.202,"5min":0.88,"15min":0.88},"max":{"1min":1.681,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.331},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4379999999999999,"15min":1.4430000000000001},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.835,"5min":2,"15min":2},"last":1.6559999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":1.0109999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.778,"5min":1.8149999999999999,"15min":2.093},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.375,"15min":1.3779999999999999},"min":{"1min":1.0700000000000001,"5min":1.0369999999999999,"15min":0.93100000000000005},"max":{"1min":1.756,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.4179999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.345,"15min":1.337},"min":{"1min":0.93700000000000006,"5min":0.89300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8120000000000001,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.236},{"interface":"front","average":{"1min":1.335,"5min":1.381,"15min":1.4059999999999999},"min":{"1min":0.93500000000000005,"5min":0.86799999999999999,"15min":0.86299999999999999},"max":{"1min":1.7949999999999999,"5min":1.881,"15min":2.097},"last":1.2729999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.375,"15min":1.351},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.903,"15min":1.903},"last":1.601},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.518},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.833,"5min":2.359,"15min":2.359},"last":1.536}]},{"osd":7,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5329999999999999,"15min":1.556},"min":{"1min":1.248,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":1.875,"5min":1.9059999999999999,"15min":2.1600000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.54,"15min":1.5409999999999999},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":0.97499999999999998},"max":{"1min":1.887,"5min":2.0720000000000001,"15min":2.3330000000000002},"last":1.7969999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51416,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320424,"statfs":{"total":96636764160,"available":96584114176,"internally_reserved":0,"allocated":2236416,"data_stored":1089661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.94399999999999995,"15min":0.92500000000000004},"min":{"1min":0.69299999999999995,"5min":0.67700000000000005,"15min":0.64800000000000002},"max":{"1min":1.194,"5min":1.569,"15min":1.7230000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.054,"5min":1.099,"15min":1.0880000000000001},"min":{"1min":0.72399999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4059999999999999,"5min":1.843,"15min":1.9430000000000001},"last":1.2130000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98299999999999998,"15min":0.998},"min":{"1min":0.73099999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.2789999999999999,"5min":1.704,"15min":1.704},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.236,"5min":1.242,"15min":1.2130000000000001},"min":{"1min":0.93700000000000006,"5min":0.86499999999999999,"15min":0.81399999999999995},"max":{"1min":1.5920000000000001,"5min":1.7210000000000001,"15min":1.823},"last":0.91100000000000003}]},{"osd":2,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4119999999999999,"15min":1.4219999999999999},"min":{"1min":0.88200000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.7729999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.552},{"interface":"front","average":{"1min":1.522,"5min":1.546,"15min":1.5309999999999999},"min":{"1min":1.157,"5min":1.1559999999999999,"15min":1.091},"max":{"1min":1.9059999999999999,"5min":2.278,"15min":2.278},"last":1.5840000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3660000000000001,"15min":1.331},"min":{"1min":1.0249999999999999,"5min":0.93200000000000005,"15min":0.90800000000000003},"max":{"1min":1.78,"5min":1.8560000000000001,"15min":1.877},"last":1.123},{"interface":"front","average":{"1min":1.381,"5min":1.379,"15min":1.399},"min":{"1min":1.109,"5min":1.05,"15min":1.01},"max":{"1min":1.536,"5min":2.0089999999999999,"15min":2.1840000000000002},"last":1.0369999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5429999999999999,"15min":1.5669999999999999},"min":{"1min":1.071,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":1.7829999999999999,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.6200000000000001,"15min":1.5660000000000001},"min":{"1min":1.173,"5min":1.0640000000000001,"15min":1.0640000000000001},"max":{"1min":2.3479999999999999,"5min":2.774,"15min":2.774},"last":1.1870000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.542,"15min":1.526},"min":{"1min":1.0009999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.036,"5min":2.3050000000000002,"15min":2.3719999999999999},"last":1.375},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.605,"15min":1.5960000000000001},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":1.8999999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.1579999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.71,"15min":1.671},"min":{"1min":1.2989999999999999,"5min":1.147,"15min":1.147},"max":{"1min":2.5150000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.05},"max":{"1min":2.1259999999999999,"5min":2.496,"15min":2.496},"last":1.4730000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575772,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51420,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320420,"statfs":{"total":96636764160,"available":96584110080,"internally_reserved":0,"allocated":2236416,"data_stored":1084925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.95999999999999996,"15min":0.92800000000000005},"min":{"1min":0.70099999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1439999999999999,"5min":1.849,"15min":1.849},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.105,"15min":1.1120000000000001},"min":{"1min":0.79200000000000004,"5min":0.70699999999999996,"15min":0.61699999999999999},"max":{"1min":1.5780000000000001,"5min":1.6890000000000001,"15min":1.768},"last":0.69299999999999995}]},{"osd":1,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.0409999999999999,"15min":1.0269999999999999},"min":{"1min":0.82099999999999995,"5min":0.753,"15min":0.72399999999999998},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.8140000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.256,"15min":1.24},"min":{"1min":0.88900000000000001,"5min":0.88100000000000001,"15min":0.80700000000000005},"max":{"1min":1.6479999999999999,"5min":1.8300000000000001,"15min":1.9299999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3300000000000001,"15min":1.331},"min":{"1min":1.0640000000000001,"5min":0.875,"15min":0.79900000000000004},"max":{"1min":1.6279999999999999,"5min":1.8420000000000001,"15min":1.992},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.381},"min":{"1min":0.93000000000000005,"5min":0.91800000000000004,"15min":0.67200000000000004},"max":{"1min":1.7689999999999999,"5min":1.921,"15min":1.921},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.498,"15min":1.484},"min":{"1min":1.179,"5min":1.113,"15min":0.96999999999999997},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.149},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.395,"15min":1.409},"min":{"1min":1.097,"5min":0.93899999999999995,"15min":0.92400000000000004},"max":{"1min":1.986,"5min":1.986,"15min":2.6909999999999998},"last":1.3779999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.542,"15min":1.488},"min":{"1min":1.095,"5min":1.0389999999999999,"15min":1.028},"max":{"1min":2.661,"5min":2.661,"15min":2.661},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.635,"15min":1.6259999999999999},"min":{"1min":1.3939999999999999,"5min":1.173,"15min":1.141},"max":{"1min":1.871,"5min":2.3730000000000002,"15min":3.3149999999999999},"last":1.4379999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.508,"15min":1.5149999999999999},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":0.92400000000000004},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.573},"last":1.464},{"interface":"front","average":{"1min":1.673,"5min":1.641,"15min":1.6060000000000001},"min":{"1min":1.403,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":2.3900000000000001,"5min":2.423,"15min":2.423},"last":1.492}]},{"osd":7,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.593,"15min":1.6100000000000001},"min":{"1min":1.129,"5min":1.099,"15min":1.0049999999999999},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.552},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.7430000000000001,"15min":1.7170000000000001},"min":{"1min":1.379,"5min":1.286,"15min":1.234},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.585}]}]},{"osd":1,"up_from":13,"seq":55834575772,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59732,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312108,"statfs":{"total":96636764160,"available":96575598592,"internally_reserved":0,"allocated":2236416,"data_stored":1090153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91400000000000003,"15min":0.96099999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.6830000000000001,"5min":1.716,"15min":1.8440000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1599999999999999,"15min":1.1259999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2010000000000001,"15min":1.22},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.53,"5min":1.7410000000000001,"15min":1.9430000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.371,"15min":1.379},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.012,"5min":2.012,"15min":2.02},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.258,"15min":1.2729999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.73499999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.385,"15min":1.361},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.883,"5min":1.8879999999999999,"15min":1.9990000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.427,"5min":1.4730000000000001,"15min":1.4930000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.79,"5min":1.9059999999999999,"15min":2.3500000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.532,"15min":1.4970000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.0619999999999998,"5min":2.2770000000000001,"15min":2.2770000000000001},"last":1.073},{"interface":"front","average":{"1min":1.429,"5min":1.472,"15min":1.462},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.889,"5min":2.1059999999999999,"15min":2.1200000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.629,"15min":1.639},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6399999999999999,"15min":1.611},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.1579999999999999,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":0.89600000000000002}]},{"osd":7,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.756,"15min":1.7170000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":2.274,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.125},{"interface":"front","average":{"1min":1.621,"5min":1.615,"15min":1.6080000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.036,"5min":2.0409999999999999,"15min":2.3039999999999998},"last":0.85799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60328,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":2842624,"data_stored":1681585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.997,"15min":0.97699999999999998},"min":{"1min":0.72499999999999998,"5min":0.65400000000000003,"15min":0.61299999999999999},"max":{"1min":1.5229999999999999,"5min":1.903,"15min":1.903},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2609999999999999,"15min":1.2490000000000001},"min":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.82299999999999995},"max":{"1min":1.6639999999999999,"5min":1.851,"15min":1.9450000000000001},"last":1.248}]},{"osd":2,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.3480000000000001,"15min":1.3180000000000001},"min":{"1min":0.90600000000000003,"5min":0.81100000000000005,"15min":0.70099999999999996},"max":{"1min":1.73,"5min":1.8939999999999999,"15min":2.0640000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.286,"15min":1.2789999999999999},"min":{"1min":0.84499999999999997,"5min":0.79500000000000004,"15min":0.64000000000000001},"max":{"1min":1.6319999999999999,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.028}]},{"osd":3,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0549999999999999,"15min":1.028},"min":{"1min":0.77000000000000002,"5min":0.75700000000000001,"15min":0.753},"max":{"1min":1.212,"5min":1.732,"15min":1.732},"last":1.212},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.95199999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8500000000000001,"5min":2.262,"15min":2.262},"last":1.3720000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3520000000000001,"15min":1.3340000000000001},"min":{"1min":1.117,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":1.6100000000000001,"5min":2.109,"15min":2.109},"last":1.403},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.554,"15min":1.5289999999999999},"min":{"1min":1.093,"5min":1.093,"15min":1.0029999999999999},"max":{"1min":1.853,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.4970000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.5029999999999999,"15min":1.494},"min":{"1min":1.1819999999999999,"5min":1.075,"15min":0.96599999999999997},"max":{"1min":1.776,"5min":2.411,"15min":2.4239999999999999},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4450000000000001,"15min":1.4419999999999999},"min":{"1min":1.151,"5min":1.069,"15min":1.0609999999999999},"max":{"1min":1.6559999999999999,"5min":2.0880000000000001,"15min":2.1899999999999999},"last":1.534}]},{"osd":6,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6990000000000001,"15min":1.6539999999999999},"min":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2250000000000001},"max":{"1min":1.988,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":2.3130000000000002},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5940000000000001,"15min":1.5820000000000001},"min":{"1min":1.1639999999999999,"5min":1.0800000000000001,"15min":1.0169999999999999},"max":{"1min":1.851,"5min":2.4340000000000002,"15min":2.5009999999999999},"last":2.0459999999999998}]},{"osd":7,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.706,"15min":1.6910000000000001},"min":{"1min":1.302,"5min":1.1379999999999999,"15min":1.087},"max":{"1min":1.911,"5min":2.419,"15min":2.5710000000000002},"last":3.089},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.6319999999999999,"15min":1.6040000000000001},"min":{"1min":1.038,"5min":1.038,"15min":1.0189999999999999},"max":{"1min":1.772,"5min":2.774,"15min":2.774},"last":1.355}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:33:50.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:33:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.472+0000 7fbf7da4f640 1 Processor -- start 2023-10-16T15:33:50.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.472+0000 7fbf7da4f640 1 -- start start 2023-10-16T15:33:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.474+0000 7fbf7da4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7815aaa0 0x7fbf7815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.474+0000 7fbf7da4f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbf780ce390 0x7fbf78070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.474+0000 7fbf7da4f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7815ed60 con 0x7fbf7815ae80 2023-10-16T15:33:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.474+0000 7fbf7da4f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7806b810 con 0x7fbf780ce390 2023-10-16T15:33:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.474+0000 7fbf7da4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780707e0 con 0x7fbf7815aaa0 2023-10-16T15:33:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.475+0000 7fbf76ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7815aaa0 0x7fbf7815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.475+0000 7fbf6ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fbf7815ae80 0x7fbf780cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:46876/0 (socket says 172.21.15.29:46876) 2023-10-16T15:33:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.475+0000 7fbf6ffff640 1 -- 172.21.15.29:0/2101502575 learned_addr learned my addr 172.21.15.29:0/2101502575 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.475+0000 7fbf777fe640 1 --2- 172.21.15.29:0/2101502575 >> v2:172.21.15.170:3300/0 conn(0x7fbf780ce390 0x7fbf78070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:50.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.476+0000 7fbf767fc640 1 -- 172.21.15.29:0/2101502575 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248012239 0 0) 0x7fbf7815ed60 con 0x7fbf7815ae80 2023-10-16T15:33:50.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.476+0000 7fbf76ffd640 1 -- 172.21.15.29:0/2101502575 >> v2:172.21.15.170:3300/0 conn(0x7fbf780ce390 msgr2=0x7fbf78070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.476+0000 7fbf76ffd640 1 --2- 172.21.15.29:0/2101502575 >> v2:172.21.15.170:3300/0 conn(0x7fbf780ce390 0x7fbf78070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.476+0000 7fbf76ffd640 1 -- 172.21.15.29:0/2101502575 >> v1:172.21.15.170:6789/0 conn(0x7fbf7815ae80 legacy=0x7fbf780cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.476+0000 7fbf76ffd640 1 -- 172.21.15.29:0/2101502575 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7815ef90 con 0x7fbf7815aaa0 2023-10-16T15:33:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.476+0000 7fbf777fe640 1 --2- 172.21.15.29:0/2101502575 >> v2:172.21.15.170:3300/0 conn(0x7fbf780ce390 0x7fbf78070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:33:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.478+0000 7fbf76ffd640 1 --2- 172.21.15.29:0/2101502575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7815aaa0 0x7fbf7815e820 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbf680097a0 tx=0x7fbf6802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc83a3771ab8a55 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.479+0000 7fbf767fc640 1 -- 172.21.15.29:0/2101502575 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6802d040 con 0x7fbf7815aaa0 2023-10-16T15:33:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.479+0000 7fbf767fc640 1 -- 172.21.15.29:0/2101502575 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68031810 con 0x7fbf7815aaa0 2023-10-16T15:33:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.479+0000 7fbf767fc640 1 -- 172.21.15.29:0/2101502575 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68031ba0 con 0x7fbf7815aaa0 2023-10-16T15:33:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 -- 172.21.15.29:0/2101502575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7815aaa0 msgr2=0x7fbf7815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/2101502575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7815aaa0 0x7fbf7815e820 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbf680097a0 tx=0x7fbf6802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 -- 172.21.15.29:0/2101502575 shutdown_connections 2023-10-16T15:33:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/2101502575 >> v2:172.21.15.170:3300/0 conn(0x7fbf780ce390 0x7fbf78070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/2101502575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf7815aaa0 0x7fbf7815e820 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 -- 172.21.15.29:0/2101502575 >> 172.21.15.29:0/2101502575 conn(0x7fbf780cb410 msgr2=0x7fbf780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 -- 172.21.15.29:0/2101502575 shutdown_connections 2023-10-16T15:33:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.480+0000 7fbf7da4f640 1 -- 172.21.15.29:0/2101502575 wait complete. 2023-10-16T15:33:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.483+0000 7fbf7da4f640 1 Processor -- start 2023-10-16T15:33:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.483+0000 7fbf7da4f640 1 -- start start 2023-10-16T15:33:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf7da4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf780ce390 0x7fbf7815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf7da4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 0x7fbf780caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf7da4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7815ae80 0x7fbf78158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf7da4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780c90e0 con 0x7fbf7815aaa0 2023-10-16T15:33:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf7da4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780caff0 con 0x7fbf7815ae80 2023-10-16T15:33:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf7da4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7815c950 con 0x7fbf780ce390 2023-10-16T15:33:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 0x7fbf780caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 0x7fbf780caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46378/0 (socket says 172.21.15.29:46378) 2023-10-16T15:33:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf6ffff640 1 -- 172.21.15.29:0/3646625823 learned_addr learned my addr 172.21.15.29:0/3646625823 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.485+0000 7fbf777fe640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7815ae80 0x7fbf78158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf6ffff640 1 -- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf780ce390 msgr2=0x7fbf7815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf6ffff640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf780ce390 0x7fbf7815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf6ffff640 1 -- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7815ae80 msgr2=0x7fbf78158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf6ffff640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7815ae80 0x7fbf78158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf6ffff640 1 -- 172.21.15.29:0/3646625823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70009a70 con 0x7fbf7815aaa0 2023-10-16T15:33:50.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf6ffff640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 0x7fbf780caab0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c4d0 tx=0x7fbf6000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:50.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf600074e0 con 0x7fbf7815aaa0 2023-10-16T15:33:50.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf60007660 con 0x7fbf7815aaa0 2023-10-16T15:33:50.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60004a20 con 0x7fbf7815aaa0 2023-10-16T15:33:50.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.486+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf68009400 con 0x7fbf7815aaa0 2023-10-16T15:33:50.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.487+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7815cf10 con 0x7fbf7815aaa0 2023-10-16T15:33:50.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.488+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7804e160 con 0x7fbf7815aaa0 2023-10-16T15:33:50.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.489+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf60004b80 con 0x7fbf7815aaa0 2023-10-16T15:33:50.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.490+0000 7fbf74ff9640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fbf5806af90 0x7fbf5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:50.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.490+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbf600889f0 con 0x7fbf7815aaa0 2023-10-16T15:33:50.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.490+0000 7fbf76ffd640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fbf5806af90 0x7fbf5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:50.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.492+0000 7fbf76ffd640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fbf5806af90 0x7fbf5806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf68036000 tx=0x7fbf68030000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:50.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.494+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf600538d0 con 0x7fbf7815aaa0 2023-10-16T15:33:50.550 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T15:33:50.548+0000 7f4279978640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:33:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.683+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 --> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf7815f550 con 0x7fbf5806af90 2023-10-16T15:33:50.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.706+0000 7fbf74ff9640 1 -- 172.21.15.29:0/3646625823 <== mgr.7456 v2:172.21.15.170:6832/1981883088 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286787 (secure 0 0 0) 0x7fbf6803a420 con 0x7fbf5806af90 2023-10-16T15:33:50.713 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:50.714 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:33:50.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fbf5806af90 msgr2=0x7fbf5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fbf5806af90 0x7fbf5806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf68036000 tx=0x7fbf68030000 comp rx=0 tx=0).stop 2023-10-16T15:33:50.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 msgr2=0x7fbf780caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:50.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 0x7fbf780caab0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c4d0 tx=0x7fbf6000c990 comp rx=0 tx=0).stop 2023-10-16T15:33:50.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 shutdown_connections 2023-10-16T15:33:50.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fbf5806af90 0x7fbf5806d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf7815aaa0 0x7fbf780caab0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf7815ae80 0x7fbf78158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 --2- 172.21.15.29:0/3646625823 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf780ce390 0x7fbf7815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:50.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 >> 172.21.15.29:0/3646625823 conn(0x7fbf780cb410 msgr2=0x7fbf7806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:50.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 shutdown_connections 2023-10-16T15:33:50.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:50.712+0000 7fbf7da4f640 1 -- 172.21.15.29:0/3646625823 wait complete. 2023-10-16T15:33:50.733 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T15:33:48.979404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":487,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":409700,"kb_used_data":18652,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754565020,"statfs":{"total":773094113280,"available":772674580480,"internally_reserved":0,"allocated":19099648,"data_stored":9895796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":400281145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.402515"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051822+0000","last_change":"2023-10-16T15:33:45.620840+0000","last_active":"2023-10-16T15:33:47.051822+0000","last_peered":"2023-10-16T15:33:47.051822+0000","last_clean":"2023-10-16T15:33:47.051822+0000","last_became_active":"2023-10-16T15:33:45.620462+0000","last_became_peered":"2023-10-16T15:33:45.620462+0000","last_unstale":"2023-10-16T15:33:47.051822+0000","last_undegraded":"2023-10-16T15:33:47.051822+0000","last_fullsized":"2023-10-16T15:33:47.051822+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:24.818698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047884+0000","last_change":"2023-10-16T15:33:45.624971+0000","last_active":"2023-10-16T15:33:47.047884+0000","last_peered":"2023-10-16T15:33:47.047884+0000","last_clean":"2023-10-16T15:33:47.047884+0000","last_became_active":"2023-10-16T15:33:45.624675+0000","last_became_peered":"2023-10-16T15:33:45.624675+0000","last_unstale":"2023-10-16T15:33:47.047884+0000","last_undegraded":"2023-10-16T15:33:47.047884+0000","last_fullsized":"2023-10-16T15:33:47.047884+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:07.955810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609963+0000","last_change":"2023-10-16T15:33:45.644805+0000","last_active":"2023-10-16T15:33:46.609963+0000","last_peered":"2023-10-16T15:33:46.609963+0000","last_clean":"2023-10-16T15:33:46.609963+0000","last_became_active":"2023-10-16T15:33:45.643700+0000","last_became_peered":"2023-10-16T15:33:45.643700+0000","last_unstale":"2023-10-16T15:33:46.609963+0000","last_undegraded":"2023-10-16T15:33:46.609963+0000","last_fullsized":"2023-10-16T15:33:46.609963+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:34:52.840049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050931+0000","last_change":"2023-10-16T15:33:45.648807+0000","last_active":"2023-10-16T15:33:47.050931+0000","last_peered":"2023-10-16T15:33:47.050931+0000","last_clean":"2023-10-16T15:33:47.050931+0000","last_became_active":"2023-10-16T15:33:45.647862+0000","last_became_peered":"2023-10-16T15:33:45.647862+0000","last_unstale":"2023-10-16T15:33:47.050931+0000","last_undegraded":"2023-10-16T15:33:47.050931+0000","last_fullsized":"2023-10-16T15:33:47.050931+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:08.341026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050503+0000","last_change":"2023-10-16T15:33:45.641700+0000","last_active":"2023-10-16T15:33:47.050503+0000","last_peered":"2023-10-16T15:33:47.050503+0000","last_clean":"2023-10-16T15:33:47.050503+0000","last_became_active":"2023-10-16T15:33:45.641387+0000","last_became_peered":"2023-10-16T15:33:45.641387+0000","last_unstale":"2023-10-16T15:33:47.050503+0000","last_undegraded":"2023-10-16T15:33:47.050503+0000","last_fullsized":"2023-10-16T15:33:47.050503+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:56.138784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050883+0000","last_change":"2023-10-16T15:33:45.645610+0000","last_active":"2023-10-16T15:33:47.050883+0000","last_peered":"2023-10-16T15:33:47.050883+0000","last_clean":"2023-10-16T15:33:47.050883+0000","last_became_active":"2023-10-16T15:33:45.645286+0000","last_became_peered":"2023-10-16T15:33:45.645286+0000","last_unstale":"2023-10-16T15:33:47.050883+0000","last_undegraded":"2023-10-16T15:33:47.050883+0000","last_fullsized":"2023-10-16T15:33:47.050883+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:24.613523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602439+0000","last_change":"2023-10-16T15:33:45.634981+0000","last_active":"2023-10-16T15:33:46.602439+0000","last_peered":"2023-10-16T15:33:46.602439+0000","last_clean":"2023-10-16T15:33:46.602439+0000","last_became_active":"2023-10-16T15:33:45.634720+0000","last_became_peered":"2023-10-16T15:33:45.634720+0000","last_unstale":"2023-10-16T15:33:46.602439+0000","last_undegraded":"2023-10-16T15:33:46.602439+0000","last_fullsized":"2023-10-16T15:33:46.602439+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.915061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049182+0000","last_change":"2023-10-16T15:33:45.631222+0000","last_active":"2023-10-16T15:33:47.049182+0000","last_peered":"2023-10-16T15:33:47.049182+0000","last_clean":"2023-10-16T15:33:47.049182+0000","last_became_active":"2023-10-16T15:33:45.630924+0000","last_became_peered":"2023-10-16T15:33:45.630924+0000","last_unstale":"2023-10-16T15:33:47.049182+0000","last_undegraded":"2023-10-16T15:33:47.049182+0000","last_fullsized":"2023-10-16T15:33:47.049182+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:02.588048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048491+0000","last_change":"2023-10-16T15:33:45.612573+0000","last_active":"2023-10-16T15:33:47.048491+0000","last_peered":"2023-10-16T15:33:47.048491+0000","last_clean":"2023-10-16T15:33:47.048491+0000","last_became_active":"2023-10-16T15:33:45.611766+0000","last_became_peered":"2023-10-16T15:33:45.611766+0000","last_unstale":"2023-10-16T15:33:47.048491+0000","last_undegraded":"2023-10-16T15:33:47.048491+0000","last_fullsized":"2023-10-16T15:33:47.048491+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:28.106963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608068+0000","last_change":"2023-10-16T15:33:45.636767+0000","last_active":"2023-10-16T15:33:46.608068+0000","last_peered":"2023-10-16T15:33:46.608068+0000","last_clean":"2023-10-16T15:33:46.608068+0000","last_became_active":"2023-10-16T15:33:45.636202+0000","last_became_peered":"2023-10-16T15:33:45.636202+0000","last_unstale":"2023-10-16T15:33:46.608068+0000","last_undegraded":"2023-10-16T15:33:46.608068+0000","last_fullsized":"2023-10-16T15:33:46.608068+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:33.012654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049872+0000","last_change":"2023-10-16T15:33:45.631274+0000","last_active":"2023-10-16T15:33:47.049872+0000","last_peered":"2023-10-16T15:33:47.049872+0000","last_clean":"2023-10-16T15:33:47.049872+0000","last_became_active":"2023-10-16T15:33:45.630963+0000","last_became_peered":"2023-10-16T15:33:45.630963+0000","last_unstale":"2023-10-16T15:33:47.049872+0000","last_undegraded":"2023-10-16T15:33:47.049872+0000","last_fullsized":"2023-10-16T15:33:47.049872+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:40.098746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606755+0000","last_change":"2023-10-16T15:33:45.634976+0000","last_active":"2023-10-16T15:33:46.606755+0000","last_peered":"2023-10-16T15:33:46.606755+0000","last_clean":"2023-10-16T15:33:46.606755+0000","last_became_active":"2023-10-16T15:33:45.633765+0000","last_became_peered":"2023-10-16T15:33:45.633765+0000","last_unstale":"2023-10-16T15:33:46.606755+0000","last_undegraded":"2023-10-16T15:33:46.606755+0000","last_fullsized":"2023-10-16T15:33:46.606755+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:13.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050527+0000","last_change":"2023-10-16T15:33:45.639765+0000","last_active":"2023-10-16T15:33:47.050527+0000","last_peered":"2023-10-16T15:33:47.050527+0000","last_clean":"2023-10-16T15:33:47.050527+0000","last_became_active":"2023-10-16T15:33:45.639423+0000","last_became_peered":"2023-10-16T15:33:45.639423+0000","last_unstale":"2023-10-16T15:33:47.050527+0000","last_undegraded":"2023-10-16T15:33:47.050527+0000","last_fullsized":"2023-10-16T15:33:47.050527+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:49.442507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049564+0000","last_change":"2023-10-16T15:33:45.623414+0000","last_active":"2023-10-16T15:33:47.049564+0000","last_peered":"2023-10-16T15:33:47.049564+0000","last_clean":"2023-10-16T15:33:47.049564+0000","last_became_active":"2023-10-16T15:33:45.623076+0000","last_became_peered":"2023-10-16T15:33:45.623076+0000","last_unstale":"2023-10-16T15:33:47.049564+0000","last_undegraded":"2023-10-16T15:33:47.049564+0000","last_fullsized":"2023-10-16T15:33:47.049564+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:39.209070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608911+0000","last_change":"2023-10-16T15:33:45.621392+0000","last_active":"2023-10-16T15:33:46.608911+0000","last_peered":"2023-10-16T15:33:46.608911+0000","last_clean":"2023-10-16T15:33:46.608911+0000","last_became_active":"2023-10-16T15:33:45.620962+0000","last_became_peered":"2023-10-16T15:33:45.620962+0000","last_unstale":"2023-10-16T15:33:46.608911+0000","last_undegraded":"2023-10-16T15:33:46.608911+0000","last_fullsized":"2023-10-16T15:33:46.608911+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:32.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048700+0000","last_change":"2023-10-16T15:33:45.623274+0000","last_active":"2023-10-16T15:33:47.048700+0000","last_peered":"2023-10-16T15:33:47.048700+0000","last_clean":"2023-10-16T15:33:47.048700+0000","last_became_active":"2023-10-16T15:33:45.623000+0000","last_became_peered":"2023-10-16T15:33:45.623000+0000","last_unstale":"2023-10-16T15:33:47.048700+0000","last_undegraded":"2023-10-16T15:33:47.048700+0000","last_fullsized":"2023-10-16T15:33:47.048700+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:29.637498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609501+0000","last_change":"2023-10-16T15:33:45.636374+0000","last_active":"2023-10-16T15:33:46.609501+0000","last_peered":"2023-10-16T15:33:46.609501+0000","last_clean":"2023-10-16T15:33:46.609501+0000","last_became_active":"2023-10-16T15:33:45.635989+0000","last_became_peered":"2023-10-16T15:33:45.635989+0000","last_unstale":"2023-10-16T15:33:46.609501+0000","last_undegraded":"2023-10-16T15:33:46.609501+0000","last_fullsized":"2023-10-16T15:33:46.609501+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:46.404483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607689+0000","last_change":"2023-10-16T15:33:45.644254+0000","last_active":"2023-10-16T15:33:46.607689+0000","last_peered":"2023-10-16T15:33:46.607689+0000","last_clean":"2023-10-16T15:33:46.607689+0000","last_became_active":"2023-10-16T15:33:45.643724+0000","last_became_peered":"2023-10-16T15:33:45.643724+0000","last_unstale":"2023-10-16T15:33:46.607689+0000","last_undegraded":"2023-10-16T15:33:46.607689+0000","last_fullsized":"2023-10-16T15:33:46.607689+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:18.019829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051358+0000","last_change":"2023-10-16T15:33:45.628013+0000","last_active":"2023-10-16T15:33:47.051358+0000","last_peered":"2023-10-16T15:33:47.051358+0000","last_clean":"2023-10-16T15:33:47.051358+0000","last_became_active":"2023-10-16T15:33:45.627787+0000","last_became_peered":"2023-10-16T15:33:45.627787+0000","last_unstale":"2023-10-16T15:33:47.051358+0000","last_undegraded":"2023-10-16T15:33:47.051358+0000","last_fullsized":"2023-10-16T15:33:47.051358+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:30:20.580101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050726+0000","last_change":"2023-10-16T15:33:45.617309+0000","last_active":"2023-10-16T15:33:47.050726+0000","last_peered":"2023-10-16T15:33:47.050726+0000","last_clean":"2023-10-16T15:33:47.050726+0000","last_became_active":"2023-10-16T15:33:45.616992+0000","last_became_peered":"2023-10-16T15:33:45.616992+0000","last_unstale":"2023-10-16T15:33:47.050726+0000","last_undegraded":"2023-10-16T15:33:47.050726+0000","last_fullsized":"2023-10-16T15:33:47.050726+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:40.098746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608418+0000","last_change":"2023-10-16T15:33:45.635822+0000","last_active":"2023-10-16T15:33:46.608418+0000","last_peered":"2023-10-16T15:33:46.608418+0000","last_clean":"2023-10-16T15:33:46.608418+0000","last_became_active":"2023-10-16T15:33:45.635242+0000","last_became_peered":"2023-10-16T15:33:45.635242+0000","last_unstale":"2023-10-16T15:33:46.608418+0000","last_undegraded":"2023-10-16T15:33:46.608418+0000","last_fullsized":"2023-10-16T15:33:46.608418+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:19.641458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049920+0000","last_change":"2023-10-16T15:33:45.623835+0000","last_active":"2023-10-16T15:33:47.049920+0000","last_peered":"2023-10-16T15:33:47.049920+0000","last_clean":"2023-10-16T15:33:47.049920+0000","last_became_active":"2023-10-16T15:33:45.615916+0000","last_became_peered":"2023-10-16T15:33:45.615916+0000","last_unstale":"2023-10-16T15:33:47.049920+0000","last_undegraded":"2023-10-16T15:33:47.049920+0000","last_fullsized":"2023-10-16T15:33:47.049920+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:12.220344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049273+0000","last_change":"2023-10-16T15:33:45.624991+0000","last_active":"2023-10-16T15:33:47.049273+0000","last_peered":"2023-10-16T15:33:47.049273+0000","last_clean":"2023-10-16T15:33:47.049273+0000","last_became_active":"2023-10-16T15:33:45.624667+0000","last_became_peered":"2023-10-16T15:33:45.624667+0000","last_unstale":"2023-10-16T15:33:47.049273+0000","last_undegraded":"2023-10-16T15:33:47.049273+0000","last_fullsized":"2023-10-16T15:33:47.049273+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:35.237056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606555+0000","last_change":"2023-10-16T15:33:45.634598+0000","last_active":"2023-10-16T15:33:46.606555+0000","last_peered":"2023-10-16T15:33:46.606555+0000","last_clean":"2023-10-16T15:33:46.606555+0000","last_became_active":"2023-10-16T15:33:45.634245+0000","last_became_peered":"2023-10-16T15:33:45.634245+0000","last_unstale":"2023-10-16T15:33:46.606555+0000","last_undegraded":"2023-10-16T15:33:46.606555+0000","last_fullsized":"2023-10-16T15:33:46.606555+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:26:22.781035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049710+0000","last_change":"2023-10-16T15:33:45.622747+0000","last_active":"2023-10-16T15:33:47.049710+0000","last_peered":"2023-10-16T15:33:47.049710+0000","last_clean":"2023-10-16T15:33:47.049710+0000","last_became_active":"2023-10-16T15:33:45.622454+0000","last_became_peered":"2023-10-16T15:33:45.622454+0000","last_unstale":"2023-10-16T15:33:47.049710+0000","last_undegraded":"2023-10-16T15:33:47.049710+0000","last_fullsized":"2023-10-16T15:33:47.049710+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:56.987561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051510+0000","last_change":"2023-10-16T15:33:45.620154+0000","last_active":"2023-10-16T15:33:47.051510+0000","last_peered":"2023-10-16T15:33:47.051510+0000","last_clean":"2023-10-16T15:33:47.051510+0000","last_became_active":"2023-10-16T15:33:45.619856+0000","last_became_peered":"2023-10-16T15:33:45.619856+0000","last_unstale":"2023-10-16T15:33:47.051510+0000","last_undegraded":"2023-10-16T15:33:47.051510+0000","last_fullsized":"2023-10-16T15:33:47.051510+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:58.057873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050337+0000","last_change":"2023-10-16T15:33:45.644367+0000","last_active":"2023-10-16T15:33:47.050337+0000","last_peered":"2023-10-16T15:33:47.050337+0000","last_clean":"2023-10-16T15:33:47.050337+0000","last_became_active":"2023-10-16T15:33:45.644011+0000","last_became_peered":"2023-10-16T15:33:45.644011+0000","last_unstale":"2023-10-16T15:33:47.050337+0000","last_undegraded":"2023-10-16T15:33:47.050337+0000","last_fullsized":"2023-10-16T15:33:47.050337+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:17.818210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602649+0000","last_change":"2023-10-16T15:33:45.648341+0000","last_active":"2023-10-16T15:33:46.602649+0000","last_peered":"2023-10-16T15:33:46.602649+0000","last_clean":"2023-10-16T15:33:46.602649+0000","last_became_active":"2023-10-16T15:33:45.647984+0000","last_became_peered":"2023-10-16T15:33:45.647984+0000","last_unstale":"2023-10-16T15:33:46.602649+0000","last_undegraded":"2023-10-16T15:33:46.602649+0000","last_fullsized":"2023-10-16T15:33:46.602649+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:43.968735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050204+0000","last_change":"2023-10-16T15:33:45.647365+0000","last_active":"2023-10-16T15:33:47.050204+0000","last_peered":"2023-10-16T15:33:47.050204+0000","last_clean":"2023-10-16T15:33:47.050204+0000","last_became_active":"2023-10-16T15:33:45.647012+0000","last_became_peered":"2023-10-16T15:33:45.647012+0000","last_unstale":"2023-10-16T15:33:47.050204+0000","last_undegraded":"2023-10-16T15:33:47.050204+0000","last_fullsized":"2023-10-16T15:33:47.050204+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:38.801957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050271+0000","last_change":"2023-10-16T15:33:45.618092+0000","last_active":"2023-10-16T15:33:47.050271+0000","last_peered":"2023-10-16T15:33:47.050271+0000","last_clean":"2023-10-16T15:33:47.050271+0000","last_became_active":"2023-10-16T15:33:45.615285+0000","last_became_peered":"2023-10-16T15:33:45.615285+0000","last_unstale":"2023-10-16T15:33:47.050271+0000","last_undegraded":"2023-10-16T15:33:47.050271+0000","last_fullsized":"2023-10-16T15:33:47.050271+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:48.196062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047339+0000","last_change":"2023-10-16T15:33:45.633670+0000","last_active":"2023-10-16T15:33:47.047339+0000","last_peered":"2023-10-16T15:33:47.047339+0000","last_clean":"2023-10-16T15:33:47.047339+0000","last_became_active":"2023-10-16T15:33:45.633331+0000","last_became_peered":"2023-10-16T15:33:45.633331+0000","last_unstale":"2023-10-16T15:33:47.047339+0000","last_undegraded":"2023-10-16T15:33:47.047339+0000","last_fullsized":"2023-10-16T15:33:47.047339+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:30.580311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049044+0000","last_change":"2023-10-16T15:33:45.634674+0000","last_active":"2023-10-16T15:33:47.049044+0000","last_peered":"2023-10-16T15:33:47.049044+0000","last_clean":"2023-10-16T15:33:47.049044+0000","last_became_active":"2023-10-16T15:33:45.630512+0000","last_became_peered":"2023-10-16T15:33:45.630512+0000","last_unstale":"2023-10-16T15:33:47.049044+0000","last_undegraded":"2023-10-16T15:33:47.049044+0000","last_fullsized":"2023-10-16T15:33:47.049044+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:18.424954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608382+0000","last_change":"2023-10-16T15:33:45.635898+0000","last_active":"2023-10-16T15:33:46.608382+0000","last_peered":"2023-10-16T15:33:46.608382+0000","last_clean":"2023-10-16T15:33:46.608382+0000","last_became_active":"2023-10-16T15:33:45.634370+0000","last_became_peered":"2023-10-16T15:33:45.634370+0000","last_unstale":"2023-10-16T15:33:46.608382+0000","last_undegraded":"2023-10-16T15:33:46.608382+0000","last_fullsized":"2023-10-16T15:33:46.608382+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:49.417462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"178'4","reported_seq":16,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609495+0000","last_change":"2023-10-16T15:33:44.624815+0000","last_active":"2023-10-16T15:33:46.609495+0000","last_peered":"2023-10-16T15:33:46.609495+0000","last_clean":"2023-10-16T15:33:46.609495+0000","last_became_active":"2023-10-16T15:33:44.624498+0000","last_became_peered":"2023-10-16T15:33:44.624498+0000","last_unstale":"2023-10-16T15:33:46.609495+0000","last_undegraded":"2023-10-16T15:33:46.609495+0000","last_fullsized":"2023-10-16T15:33:46.609495+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:44.322839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048765+0000","last_change":"2023-10-16T15:33:45.631693+0000","last_active":"2023-10-16T15:33:47.048765+0000","last_peered":"2023-10-16T15:33:47.048765+0000","last_clean":"2023-10-16T15:33:47.048765+0000","last_became_active":"2023-10-16T15:33:45.628106+0000","last_became_peered":"2023-10-16T15:33:45.628106+0000","last_unstale":"2023-10-16T15:33:47.048765+0000","last_undegraded":"2023-10-16T15:33:47.048765+0000","last_fullsized":"2023-10-16T15:33:47.048765+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:35.644993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048123+0000","last_change":"2023-10-16T15:33:44.623092+0000","last_active":"2023-10-16T15:33:47.048123+0000","last_peered":"2023-10-16T15:33:47.048123+0000","last_clean":"2023-10-16T15:33:47.048123+0000","last_became_active":"2023-10-16T15:33:44.622718+0000","last_became_peered":"2023-10-16T15:33:44.622718+0000","last_unstale":"2023-10-16T15:33:47.048123+0000","last_undegraded":"2023-10-16T15:33:47.048123+0000","last_fullsized":"2023-10-16T15:33:47.048123+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:28.342455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049996+0000","last_change":"2023-10-16T15:33:45.625971+0000","last_active":"2023-10-16T15:33:47.049996+0000","last_peered":"2023-10-16T15:33:47.049996+0000","last_clean":"2023-10-16T15:33:47.049996+0000","last_became_active":"2023-10-16T15:33:45.625706+0000","last_became_peered":"2023-10-16T15:33:45.625706+0000","last_unstale":"2023-10-16T15:33:47.049996+0000","last_undegraded":"2023-10-16T15:33:47.049996+0000","last_fullsized":"2023-10-16T15:33:47.049996+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:34:01.964835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050821+0000","last_change":"2023-10-16T15:33:44.614459+0000","last_active":"2023-10-16T15:33:47.050821+0000","last_peered":"2023-10-16T15:33:47.050821+0000","last_clean":"2023-10-16T15:33:47.050821+0000","last_became_active":"2023-10-16T15:33:44.614040+0000","last_became_peered":"2023-10-16T15:33:44.614040+0000","last_unstale":"2023-10-16T15:33:47.050821+0000","last_undegraded":"2023-10-16T15:33:47.050821+0000","last_fullsized":"2023-10-16T15:33:47.050821+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:57.041308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606767+0000","last_change":"2023-10-16T15:33:45.633494+0000","last_active":"2023-10-16T15:33:46.606767+0000","last_peered":"2023-10-16T15:33:46.606767+0000","last_clean":"2023-10-16T15:33:46.606767+0000","last_became_active":"2023-10-16T15:33:45.633107+0000","last_became_peered":"2023-10-16T15:33:45.633107+0000","last_unstale":"2023-10-16T15:33:46.606767+0000","last_undegraded":"2023-10-16T15:33:46.606767+0000","last_fullsized":"2023-10-16T15:33:46.606767+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:49.106596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050915+0000","last_change":"2023-10-16T15:33:44.615061+0000","last_active":"2023-10-16T15:33:47.050915+0000","last_peered":"2023-10-16T15:33:47.050915+0000","last_clean":"2023-10-16T15:33:47.050915+0000","last_became_active":"2023-10-16T15:33:44.613917+0000","last_became_peered":"2023-10-16T15:33:44.613917+0000","last_unstale":"2023-10-16T15:33:47.050915+0000","last_undegraded":"2023-10-16T15:33:47.050915+0000","last_fullsized":"2023-10-16T15:33:47.050915+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:25.823449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047595+0000","last_change":"2023-10-16T15:33:45.645650+0000","last_active":"2023-10-16T15:33:47.047595+0000","last_peered":"2023-10-16T15:33:47.047595+0000","last_clean":"2023-10-16T15:33:47.047595+0000","last_became_active":"2023-10-16T15:33:45.645163+0000","last_became_peered":"2023-10-16T15:33:45.645163+0000","last_unstale":"2023-10-16T15:33:47.047595+0000","last_undegraded":"2023-10-16T15:33:47.047595+0000","last_fullsized":"2023-10-16T15:33:47.047595+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:20.012447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049345+0000","last_change":"2023-10-16T15:33:44.617095+0000","last_active":"2023-10-16T15:33:47.049345+0000","last_peered":"2023-10-16T15:33:47.049345+0000","last_clean":"2023-10-16T15:33:47.049345+0000","last_became_active":"2023-10-16T15:33:44.616809+0000","last_became_peered":"2023-10-16T15:33:44.616809+0000","last_unstale":"2023-10-16T15:33:47.049345+0000","last_undegraded":"2023-10-16T15:33:47.049345+0000","last_fullsized":"2023-10-16T15:33:47.049345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:31.600826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050023+0000","last_change":"2023-10-16T15:33:45.646207+0000","last_active":"2023-10-16T15:33:47.050023+0000","last_peered":"2023-10-16T15:33:47.050023+0000","last_clean":"2023-10-16T15:33:47.050023+0000","last_became_active":"2023-10-16T15:33:45.645849+0000","last_became_peered":"2023-10-16T15:33:45.645849+0000","last_unstale":"2023-10-16T15:33:47.050023+0000","last_undegraded":"2023-10-16T15:33:47.050023+0000","last_fullsized":"2023-10-16T15:33:47.050023+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:23.278967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049789+0000","last_change":"2023-10-16T15:33:44.631064+0000","last_active":"2023-10-16T15:33:47.049789+0000","last_peered":"2023-10-16T15:33:47.049789+0000","last_clean":"2023-10-16T15:33:47.049789+0000","last_became_active":"2023-10-16T15:33:44.630427+0000","last_became_peered":"2023-10-16T15:33:44.630427+0000","last_unstale":"2023-10-16T15:33:47.049789+0000","last_undegraded":"2023-10-16T15:33:47.049789+0000","last_fullsized":"2023-10-16T15:33:47.049789+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:28.625614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047318+0000","last_change":"2023-10-16T15:33:45.634611+0000","last_active":"2023-10-16T15:33:47.047318+0000","last_peered":"2023-10-16T15:33:47.047318+0000","last_clean":"2023-10-16T15:33:47.047318+0000","last_became_active":"2023-10-16T15:33:45.634328+0000","last_became_peered":"2023-10-16T15:33:45.634328+0000","last_unstale":"2023-10-16T15:33:47.047318+0000","last_undegraded":"2023-10-16T15:33:47.047318+0000","last_fullsized":"2023-10-16T15:33:47.047318+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:59.726928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049055+0000","last_change":"2023-10-16T15:33:44.625645+0000","last_active":"2023-10-16T15:33:47.049055+0000","last_peered":"2023-10-16T15:33:47.049055+0000","last_clean":"2023-10-16T15:33:47.049055+0000","last_became_active":"2023-10-16T15:33:44.625186+0000","last_became_peered":"2023-10-16T15:33:44.625186+0000","last_unstale":"2023-10-16T15:33:47.049055+0000","last_undegraded":"2023-10-16T15:33:47.049055+0000","last_fullsized":"2023-10-16T15:33:47.049055+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:55.122219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049149+0000","last_change":"2023-10-16T15:33:45.637588+0000","last_active":"2023-10-16T15:33:47.049149+0000","last_peered":"2023-10-16T15:33:47.049149+0000","last_clean":"2023-10-16T15:33:47.049149+0000","last_became_active":"2023-10-16T15:33:45.637172+0000","last_became_peered":"2023-10-16T15:33:45.637172+0000","last_unstale":"2023-10-16T15:33:47.049149+0000","last_undegraded":"2023-10-16T15:33:47.049149+0000","last_fullsized":"2023-10-16T15:33:47.049149+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:22.745923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048918+0000","last_change":"2023-10-16T15:33:44.624433+0000","last_active":"2023-10-16T15:33:47.048918+0000","last_peered":"2023-10-16T15:33:47.048918+0000","last_clean":"2023-10-16T15:33:47.048918+0000","last_became_active":"2023-10-16T15:33:44.624058+0000","last_became_peered":"2023-10-16T15:33:44.624058+0000","last_unstale":"2023-10-16T15:33:47.048918+0000","last_undegraded":"2023-10-16T15:33:47.048918+0000","last_fullsized":"2023-10-16T15:33:47.048918+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:04.068506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051184+0000","last_change":"2023-10-16T15:33:45.636497+0000","last_active":"2023-10-16T15:33:47.051184+0000","last_peered":"2023-10-16T15:33:47.051184+0000","last_clean":"2023-10-16T15:33:47.051184+0000","last_became_active":"2023-10-16T15:33:45.634140+0000","last_became_peered":"2023-10-16T15:33:45.634140+0000","last_unstale":"2023-10-16T15:33:47.051184+0000","last_undegraded":"2023-10-16T15:33:47.051184+0000","last_fullsized":"2023-10-16T15:33:47.051184+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:04.527775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048916+0000","last_change":"2023-10-16T15:33:44.612045+0000","last_active":"2023-10-16T15:33:47.048916+0000","last_peered":"2023-10-16T15:33:47.048916+0000","last_clean":"2023-10-16T15:33:47.048916+0000","last_became_active":"2023-10-16T15:33:44.611769+0000","last_became_peered":"2023-10-16T15:33:44.611769+0000","last_unstale":"2023-10-16T15:33:47.048916+0000","last_undegraded":"2023-10-16T15:33:47.048916+0000","last_fullsized":"2023-10-16T15:33:47.048916+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:22.479954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609228+0000","last_change":"2023-10-16T15:33:45.636660+0000","last_active":"2023-10-16T15:33:46.609228+0000","last_peered":"2023-10-16T15:33:46.609228+0000","last_clean":"2023-10-16T15:33:46.609228+0000","last_became_active":"2023-10-16T15:33:45.636376+0000","last_became_peered":"2023-10-16T15:33:45.636376+0000","last_unstale":"2023-10-16T15:33:46.609228+0000","last_undegraded":"2023-10-16T15:33:46.609228+0000","last_fullsized":"2023-10-16T15:33:46.609228+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:10.710971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609004+0000","last_change":"2023-10-16T15:33:44.629352+0000","last_active":"2023-10-16T15:33:46.609004+0000","last_peered":"2023-10-16T15:33:46.609004+0000","last_clean":"2023-10-16T15:33:46.609004+0000","last_became_active":"2023-10-16T15:33:44.629032+0000","last_became_peered":"2023-10-16T15:33:44.629032+0000","last_unstale":"2023-10-16T15:33:46.609004+0000","last_undegraded":"2023-10-16T15:33:46.609004+0000","last_fullsized":"2023-10-16T15:33:46.609004+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:57.812460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050711+0000","last_change":"2023-10-16T15:33:45.633788+0000","last_active":"2023-10-16T15:33:47.050711+0000","last_peered":"2023-10-16T15:33:47.050711+0000","last_clean":"2023-10-16T15:33:47.050711+0000","last_became_active":"2023-10-16T15:33:45.631665+0000","last_became_peered":"2023-10-16T15:33:45.631665+0000","last_unstale":"2023-10-16T15:33:47.050711+0000","last_undegraded":"2023-10-16T15:33:47.050711+0000","last_fullsized":"2023-10-16T15:33:47.050711+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:02.588048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"178'6","reported_seq":18,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602924+0000","last_change":"2023-10-16T15:33:44.614425+0000","last_active":"2023-10-16T15:33:46.602924+0000","last_peered":"2023-10-16T15:33:46.602924+0000","last_clean":"2023-10-16T15:33:46.602924+0000","last_became_active":"2023-10-16T15:33:44.614156+0000","last_became_peered":"2023-10-16T15:33:44.614156+0000","last_unstale":"2023-10-16T15:33:46.602924+0000","last_undegraded":"2023-10-16T15:33:46.602924+0000","last_fullsized":"2023-10-16T15:33:46.602924+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:40.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608120+0000","last_change":"2023-10-16T15:33:45.644560+0000","last_active":"2023-10-16T15:33:46.608120+0000","last_peered":"2023-10-16T15:33:46.608120+0000","last_clean":"2023-10-16T15:33:46.608120+0000","last_became_active":"2023-10-16T15:33:45.643116+0000","last_became_peered":"2023-10-16T15:33:45.643116+0000","last_unstale":"2023-10-16T15:33:46.608120+0000","last_undegraded":"2023-10-16T15:33:46.608120+0000","last_fullsized":"2023-10-16T15:33:46.608120+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:37.009681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609548+0000","last_change":"2023-10-16T15:33:44.628463+0000","last_active":"2023-10-16T15:33:46.609548+0000","last_peered":"2023-10-16T15:33:46.609548+0000","last_clean":"2023-10-16T15:33:46.609548+0000","last_became_active":"2023-10-16T15:33:44.626707+0000","last_became_peered":"2023-10-16T15:33:44.626707+0000","last_unstale":"2023-10-16T15:33:46.609548+0000","last_undegraded":"2023-10-16T15:33:46.609548+0000","last_fullsized":"2023-10-16T15:33:46.609548+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:27.705500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047560+0000","last_change":"2023-10-16T15:33:45.625564+0000","last_active":"2023-10-16T15:33:47.047560+0000","last_peered":"2023-10-16T15:33:47.047560+0000","last_clean":"2023-10-16T15:33:47.047560+0000","last_became_active":"2023-10-16T15:33:45.623750+0000","last_became_peered":"2023-10-16T15:33:45.623750+0000","last_unstale":"2023-10-16T15:33:47.047560+0000","last_undegraded":"2023-10-16T15:33:47.047560+0000","last_fullsized":"2023-10-16T15:33:47.047560+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:54.807078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608235+0000","last_change":"2023-10-16T15:33:44.629646+0000","last_active":"2023-10-16T15:33:46.608235+0000","last_peered":"2023-10-16T15:33:46.608235+0000","last_clean":"2023-10-16T15:33:46.608235+0000","last_became_active":"2023-10-16T15:33:44.629246+0000","last_became_peered":"2023-10-16T15:33:44.629246+0000","last_unstale":"2023-10-16T15:33:46.608235+0000","last_undegraded":"2023-10-16T15:33:46.608235+0000","last_fullsized":"2023-10-16T15:33:46.608235+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:06.207602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049809+0000","last_change":"2023-10-16T15:33:45.643396+0000","last_active":"2023-10-16T15:33:47.049809+0000","last_peered":"2023-10-16T15:33:47.049809+0000","last_clean":"2023-10-16T15:33:47.049809+0000","last_became_active":"2023-10-16T15:33:45.643031+0000","last_became_peered":"2023-10-16T15:33:45.643031+0000","last_unstale":"2023-10-16T15:33:47.049809+0000","last_undegraded":"2023-10-16T15:33:47.049809+0000","last_fullsized":"2023-10-16T15:33:47.049809+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:38.332204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049972+0000","last_change":"2023-10-16T15:33:44.612338+0000","last_active":"2023-10-16T15:33:47.049972+0000","last_peered":"2023-10-16T15:33:47.049972+0000","last_clean":"2023-10-16T15:33:47.049972+0000","last_became_active":"2023-10-16T15:33:44.612061+0000","last_became_peered":"2023-10-16T15:33:44.612061+0000","last_unstale":"2023-10-16T15:33:47.049972+0000","last_undegraded":"2023-10-16T15:33:47.049972+0000","last_fullsized":"2023-10-16T15:33:47.049972+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:48.425942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608616+0000","last_change":"2023-10-16T15:33:45.642815+0000","last_active":"2023-10-16T15:33:46.608616+0000","last_peered":"2023-10-16T15:33:46.608616+0000","last_clean":"2023-10-16T15:33:46.608616+0000","last_became_active":"2023-10-16T15:33:45.642550+0000","last_became_peered":"2023-10-16T15:33:45.642550+0000","last_unstale":"2023-10-16T15:33:46.608616+0000","last_undegraded":"2023-10-16T15:33:46.608616+0000","last_fullsized":"2023-10-16T15:33:46.608616+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:29.642179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"178'8","reported_seq":20,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049207+0000","last_change":"2023-10-16T15:33:44.626796+0000","last_active":"2023-10-16T15:33:47.049207+0000","last_peered":"2023-10-16T15:33:47.049207+0000","last_clean":"2023-10-16T15:33:47.049207+0000","last_became_active":"2023-10-16T15:33:44.626067+0000","last_became_peered":"2023-10-16T15:33:44.626067+0000","last_unstale":"2023-10-16T15:33:47.049207+0000","last_undegraded":"2023-10-16T15:33:47.049207+0000","last_fullsized":"2023-10-16T15:33:47.049207+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:48.425942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608629+0000","last_change":"2023-10-16T15:33:45.634025+0000","last_active":"2023-10-16T15:33:46.608629+0000","last_peered":"2023-10-16T15:33:46.608629+0000","last_clean":"2023-10-16T15:33:46.608629+0000","last_became_active":"2023-10-16T15:33:45.632514+0000","last_became_peered":"2023-10-16T15:33:45.632514+0000","last_unstale":"2023-10-16T15:33:46.608629+0000","last_undegraded":"2023-10-16T15:33:46.608629+0000","last_fullsized":"2023-10-16T15:33:46.608629+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:58:56.393005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049904+0000","last_change":"2023-10-16T15:33:44.630760+0000","last_active":"2023-10-16T15:33:47.049904+0000","last_peered":"2023-10-16T15:33:47.049904+0000","last_clean":"2023-10-16T15:33:47.049904+0000","last_became_active":"2023-10-16T15:33:44.629787+0000","last_became_peered":"2023-10-16T15:33:44.629787+0000","last_unstale":"2023-10-16T15:33:47.049904+0000","last_undegraded":"2023-10-16T15:33:47.049904+0000","last_fullsized":"2023-10-16T15:33:47.049904+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:50.586092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049973+0000","last_change":"2023-10-16T15:33:45.628893+0000","last_active":"2023-10-16T15:33:47.049973+0000","last_peered":"2023-10-16T15:33:47.049973+0000","last_clean":"2023-10-16T15:33:47.049973+0000","last_became_active":"2023-10-16T15:33:45.628446+0000","last_became_peered":"2023-10-16T15:33:45.628446+0000","last_unstale":"2023-10-16T15:33:47.049973+0000","last_undegraded":"2023-10-16T15:33:47.049973+0000","last_fullsized":"2023-10-16T15:33:47.049973+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:28.325081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048367+0000","last_change":"2023-10-16T15:33:44.625058+0000","last_active":"2023-10-16T15:33:47.048367+0000","last_peered":"2023-10-16T15:33:47.048367+0000","last_clean":"2023-10-16T15:33:47.048367+0000","last_became_active":"2023-10-16T15:33:44.624777+0000","last_became_peered":"2023-10-16T15:33:44.624777+0000","last_unstale":"2023-10-16T15:33:47.048367+0000","last_undegraded":"2023-10-16T15:33:47.048367+0000","last_fullsized":"2023-10-16T15:33:47.048367+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:04.520105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609116+0000","last_change":"2023-10-16T15:33:45.643776+0000","last_active":"2023-10-16T15:33:46.609116+0000","last_peered":"2023-10-16T15:33:46.609116+0000","last_clean":"2023-10-16T15:33:46.609116+0000","last_became_active":"2023-10-16T15:33:45.643234+0000","last_became_peered":"2023-10-16T15:33:45.643234+0000","last_unstale":"2023-10-16T15:33:46.609116+0000","last_undegraded":"2023-10-16T15:33:46.609116+0000","last_fullsized":"2023-10-16T15:33:46.609116+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:51.602657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"178'1","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050944+0000","last_change":"2023-10-16T15:33:44.623120+0000","last_active":"2023-10-16T15:33:47.050944+0000","last_peered":"2023-10-16T15:33:47.050944+0000","last_clean":"2023-10-16T15:33:47.050944+0000","last_became_active":"2023-10-16T15:33:44.622823+0000","last_became_peered":"2023-10-16T15:33:44.622823+0000","last_unstale":"2023-10-16T15:33:47.050944+0000","last_undegraded":"2023-10-16T15:33:47.050944+0000","last_fullsized":"2023-10-16T15:33:47.050944+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:24.543095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050103+0000","last_change":"2023-10-16T15:33:45.641454+0000","last_active":"2023-10-16T15:33:47.050103+0000","last_peered":"2023-10-16T15:33:47.050103+0000","last_clean":"2023-10-16T15:33:47.050103+0000","last_became_active":"2023-10-16T15:33:45.641165+0000","last_became_peered":"2023-10-16T15:33:45.641165+0000","last_unstale":"2023-10-16T15:33:47.050103+0000","last_undegraded":"2023-10-16T15:33:47.050103+0000","last_fullsized":"2023-10-16T15:33:47.050103+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:10.670739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047917+0000","last_change":"2023-10-16T15:33:44.611242+0000","last_active":"2023-10-16T15:33:47.047917+0000","last_peered":"2023-10-16T15:33:47.047917+0000","last_clean":"2023-10-16T15:33:47.047917+0000","last_became_active":"2023-10-16T15:33:44.610289+0000","last_became_peered":"2023-10-16T15:33:44.610289+0000","last_unstale":"2023-10-16T15:33:47.047917+0000","last_undegraded":"2023-10-16T15:33:47.047917+0000","last_fullsized":"2023-10-16T15:33:47.047917+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:39.260215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050657+0000","last_change":"2023-10-16T15:33:45.617665+0000","last_active":"2023-10-16T15:33:47.050657+0000","last_peered":"2023-10-16T15:33:47.050657+0000","last_clean":"2023-10-16T15:33:47.050657+0000","last_became_active":"2023-10-16T15:33:45.614269+0000","last_became_peered":"2023-10-16T15:33:45.614269+0000","last_unstale":"2023-10-16T15:33:47.050657+0000","last_undegraded":"2023-10-16T15:33:47.050657+0000","last_fullsized":"2023-10-16T15:33:47.050657+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:53.204545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602441+0000","last_change":"2023-10-16T15:33:44.626790+0000","last_active":"2023-10-16T15:33:46.602441+0000","last_peered":"2023-10-16T15:33:46.602441+0000","last_clean":"2023-10-16T15:33:46.602441+0000","last_became_active":"2023-10-16T15:33:44.626331+0000","last_became_peered":"2023-10-16T15:33:44.626331+0000","last_unstale":"2023-10-16T15:33:46.602441+0000","last_undegraded":"2023-10-16T15:33:46.602441+0000","last_fullsized":"2023-10-16T15:33:46.602441+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:40.435220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602298+0000","last_change":"2023-10-16T15:33:45.630435+0000","last_active":"2023-10-16T15:33:46.602298+0000","last_peered":"2023-10-16T15:33:46.602298+0000","last_clean":"2023-10-16T15:33:46.602298+0000","last_became_active":"2023-10-16T15:33:45.630128+0000","last_became_peered":"2023-10-16T15:33:45.630128+0000","last_unstale":"2023-10-16T15:33:46.602298+0000","last_undegraded":"2023-10-16T15:33:46.602298+0000","last_fullsized":"2023-10-16T15:33:46.602298+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:11.925522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607757+0000","last_change":"2023-10-16T15:33:44.624040+0000","last_active":"2023-10-16T15:33:46.607757+0000","last_peered":"2023-10-16T15:33:46.607757+0000","last_clean":"2023-10-16T15:33:46.607757+0000","last_became_active":"2023-10-16T15:33:44.623740+0000","last_became_peered":"2023-10-16T15:33:44.623740+0000","last_unstale":"2023-10-16T15:33:46.607757+0000","last_undegraded":"2023-10-16T15:33:46.607757+0000","last_fullsized":"2023-10-16T15:33:46.607757+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:48.735149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049149+0000","last_change":"2023-10-16T15:33:45.627875+0000","last_active":"2023-10-16T15:33:47.049149+0000","last_peered":"2023-10-16T15:33:47.049149+0000","last_clean":"2023-10-16T15:33:47.049149+0000","last_became_active":"2023-10-16T15:33:45.624042+0000","last_became_peered":"2023-10-16T15:33:45.624042+0000","last_unstale":"2023-10-16T15:33:47.049149+0000","last_undegraded":"2023-10-16T15:33:47.049149+0000","last_fullsized":"2023-10-16T15:33:47.049149+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:28.112469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607557+0000","last_change":"2023-10-16T15:33:44.626226+0000","last_active":"2023-10-16T15:33:46.607557+0000","last_peered":"2023-10-16T15:33:46.607557+0000","last_clean":"2023-10-16T15:33:46.607557+0000","last_became_active":"2023-10-16T15:33:44.625346+0000","last_became_peered":"2023-10-16T15:33:44.625346+0000","last_unstale":"2023-10-16T15:33:46.607557+0000","last_undegraded":"2023-10-16T15:33:46.607557+0000","last_fullsized":"2023-10-16T15:33:46.607557+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:11.383757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602285+0000","last_change":"2023-10-16T15:33:45.636311+0000","last_active":"2023-10-16T15:33:46.602285+0000","last_peered":"2023-10-16T15:33:46.602285+0000","last_clean":"2023-10-16T15:33:46.602285+0000","last_became_active":"2023-10-16T15:33:45.635197+0000","last_became_peered":"2023-10-16T15:33:45.635197+0000","last_unstale":"2023-10-16T15:33:46.602285+0000","last_undegraded":"2023-10-16T15:33:46.602285+0000","last_fullsized":"2023-10-16T15:33:46.602285+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:37.440753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607394+0000","last_change":"2023-10-16T15:33:44.624605+0000","last_active":"2023-10-16T15:33:46.607394+0000","last_peered":"2023-10-16T15:33:46.607394+0000","last_clean":"2023-10-16T15:33:46.607394+0000","last_became_active":"2023-10-16T15:33:44.623368+0000","last_became_peered":"2023-10-16T15:33:44.623368+0000","last_unstale":"2023-10-16T15:33:46.607394+0000","last_undegraded":"2023-10-16T15:33:46.607394+0000","last_fullsized":"2023-10-16T15:33:46.607394+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:42.262105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051033+0000","last_change":"2023-10-16T15:33:45.641590+0000","last_active":"2023-10-16T15:33:47.051033+0000","last_peered":"2023-10-16T15:33:47.051033+0000","last_clean":"2023-10-16T15:33:47.051033+0000","last_became_active":"2023-10-16T15:33:45.641326+0000","last_became_peered":"2023-10-16T15:33:45.641326+0000","last_unstale":"2023-10-16T15:33:47.051033+0000","last_undegraded":"2023-10-16T15:33:47.051033+0000","last_fullsized":"2023-10-16T15:33:47.051033+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:13:24.431946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"178'2","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609677+0000","last_change":"2023-10-16T15:33:44.632364+0000","last_active":"2023-10-16T15:33:46.609677+0000","last_peered":"2023-10-16T15:33:46.609677+0000","last_clean":"2023-10-16T15:33:46.609677+0000","last_became_active":"2023-10-16T15:33:44.632030+0000","last_became_peered":"2023-10-16T15:33:44.632030+0000","last_unstale":"2023-10-16T15:33:46.609677+0000","last_undegraded":"2023-10-16T15:33:46.609677+0000","last_fullsized":"2023-10-16T15:33:46.609677+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:05.462762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.610142+0000","last_change":"2023-10-16T15:33:45.648232+0000","last_active":"2023-10-16T15:33:46.610142+0000","last_peered":"2023-10-16T15:33:46.610142+0000","last_clean":"2023-10-16T15:33:46.610142+0000","last_became_active":"2023-10-16T15:33:45.647853+0000","last_became_peered":"2023-10-16T15:33:45.647853+0000","last_unstale":"2023-10-16T15:33:46.610142+0000","last_undegraded":"2023-10-16T15:33:46.610142+0000","last_fullsized":"2023-10-16T15:33:46.610142+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:22.745923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051623+0000","last_change":"2023-10-16T15:33:44.624586+0000","last_active":"2023-10-16T15:33:47.051623+0000","last_peered":"2023-10-16T15:33:47.051623+0000","last_clean":"2023-10-16T15:33:47.051623+0000","last_became_active":"2023-10-16T15:33:44.624048+0000","last_became_peered":"2023-10-16T15:33:44.624048+0000","last_unstale":"2023-10-16T15:33:47.051623+0000","last_undegraded":"2023-10-16T15:33:47.051623+0000","last_fullsized":"2023-10-16T15:33:47.051623+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:27.308516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047882+0000","last_change":"2023-10-16T15:33:45.630012+0000","last_active":"2023-10-16T15:33:47.047882+0000","last_peered":"2023-10-16T15:33:47.047882+0000","last_clean":"2023-10-16T15:33:47.047882+0000","last_became_active":"2023-10-16T15:33:45.629624+0000","last_became_peered":"2023-10-16T15:33:45.629624+0000","last_unstale":"2023-10-16T15:33:47.047882+0000","last_undegraded":"2023-10-16T15:33:47.047882+0000","last_fullsized":"2023-10-16T15:33:47.047882+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:15.110341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051500+0000","last_change":"2023-10-16T15:33:44.613942+0000","last_active":"2023-10-16T15:33:47.051500+0000","last_peered":"2023-10-16T15:33:47.051500+0000","last_clean":"2023-10-16T15:33:47.051500+0000","last_became_active":"2023-10-16T15:33:44.613627+0000","last_became_peered":"2023-10-16T15:33:44.613627+0000","last_unstale":"2023-10-16T15:33:47.051500+0000","last_undegraded":"2023-10-16T15:33:47.051500+0000","last_fullsized":"2023-10-16T15:33:47.051500+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:55.970996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.051360+0000","last_change":"2023-10-16T15:33:45.615849+0000","last_active":"2023-10-16T15:33:47.051360+0000","last_peered":"2023-10-16T15:33:47.051360+0000","last_clean":"2023-10-16T15:33:47.051360+0000","last_became_active":"2023-10-16T15:33:45.615499+0000","last_became_peered":"2023-10-16T15:33:45.615499+0000","last_unstale":"2023-10-16T15:33:47.051360+0000","last_undegraded":"2023-10-16T15:33:47.051360+0000","last_fullsized":"2023-10-16T15:33:47.051360+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:32.384456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050397+0000","last_change":"2023-10-16T15:33:44.614110+0000","last_active":"2023-10-16T15:33:47.050397+0000","last_peered":"2023-10-16T15:33:47.050397+0000","last_clean":"2023-10-16T15:33:47.050397+0000","last_became_active":"2023-10-16T15:33:44.613801+0000","last_became_peered":"2023-10-16T15:33:44.613801+0000","last_unstale":"2023-10-16T15:33:47.050397+0000","last_undegraded":"2023-10-16T15:33:47.050397+0000","last_fullsized":"2023-10-16T15:33:47.050397+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:31.600826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.048886+0000","last_change":"2023-10-16T15:33:45.629201+0000","last_active":"2023-10-16T15:33:47.048886+0000","last_peered":"2023-10-16T15:33:47.048886+0000","last_clean":"2023-10-16T15:33:47.048886+0000","last_became_active":"2023-10-16T15:33:45.628878+0000","last_became_peered":"2023-10-16T15:33:45.628878+0000","last_unstale":"2023-10-16T15:33:47.048886+0000","last_undegraded":"2023-10-16T15:33:47.048886+0000","last_fullsized":"2023-10-16T15:33:47.048886+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:19:51.985498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602451+0000","last_change":"2023-10-16T15:33:44.615954+0000","last_active":"2023-10-16T15:33:46.602451+0000","last_peered":"2023-10-16T15:33:46.602451+0000","last_clean":"2023-10-16T15:33:46.602451+0000","last_became_active":"2023-10-16T15:33:44.615222+0000","last_became_peered":"2023-10-16T15:33:44.615222+0000","last_unstale":"2023-10-16T15:33:46.602451+0000","last_undegraded":"2023-10-16T15:33:46.602451+0000","last_fullsized":"2023-10-16T15:33:46.602451+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:41.981941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.050847+0000","last_change":"2023-10-16T15:33:45.648133+0000","last_active":"2023-10-16T15:33:47.050847+0000","last_peered":"2023-10-16T15:33:47.050847+0000","last_clean":"2023-10-16T15:33:47.050847+0000","last_became_active":"2023-10-16T15:33:45.647470+0000","last_became_peered":"2023-10-16T15:33:45.647470+0000","last_unstale":"2023-10-16T15:33:47.050847+0000","last_undegraded":"2023-10-16T15:33:47.050847+0000","last_fullsized":"2023-10-16T15:33:47.050847+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:45.847934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.607977+0000","last_change":"2023-10-16T15:33:44.632417+0000","last_active":"2023-10-16T15:33:46.607977+0000","last_peered":"2023-10-16T15:33:46.607977+0000","last_clean":"2023-10-16T15:33:46.607977+0000","last_became_active":"2023-10-16T15:33:44.632102+0000","last_became_peered":"2023-10-16T15:33:44.632102+0000","last_unstale":"2023-10-16T15:33:46.607977+0000","last_undegraded":"2023-10-16T15:33:46.607977+0000","last_fullsized":"2023-10-16T15:33:46.607977+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:29.302375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.606560+0000","last_change":"2023-10-16T15:33:45.622628+0000","last_active":"2023-10-16T15:33:46.606560+0000","last_peered":"2023-10-16T15:33:46.606560+0000","last_clean":"2023-10-16T15:33:46.606560+0000","last_became_active":"2023-10-16T15:33:45.622146+0000","last_became_peered":"2023-10-16T15:33:45.622146+0000","last_unstale":"2023-10-16T15:33:46.606560+0000","last_undegraded":"2023-10-16T15:33:46.606560+0000","last_fullsized":"2023-10-16T15:33:46.606560+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:03.101744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.608836+0000","last_change":"2023-10-16T15:33:44.621647+0000","last_active":"2023-10-16T15:33:46.608836+0000","last_peered":"2023-10-16T15:33:46.608836+0000","last_clean":"2023-10-16T15:33:46.608836+0000","last_became_active":"2023-10-16T15:33:44.621337+0000","last_became_peered":"2023-10-16T15:33:44.621337+0000","last_unstale":"2023-10-16T15:33:46.608836+0000","last_undegraded":"2023-10-16T15:33:46.608836+0000","last_fullsized":"2023-10-16T15:33:46.608836+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:32:51.641585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.049427+0000","last_change":"2023-10-16T15:33:45.636035+0000","last_active":"2023-10-16T15:33:47.049427+0000","last_peered":"2023-10-16T15:33:47.049427+0000","last_clean":"2023-10-16T15:33:47.049427+0000","last_became_active":"2023-10-16T15:33:45.634716+0000","last_became_peered":"2023-10-16T15:33:45.634716+0000","last_unstale":"2023-10-16T15:33:47.049427+0000","last_undegraded":"2023-10-16T15:33:47.049427+0000","last_fullsized":"2023-10-16T15:33:47.049427+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:53.880114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:47.047296+0000","last_change":"2023-10-16T15:33:44.616284+0000","last_active":"2023-10-16T15:33:47.047296+0000","last_peered":"2023-10-16T15:33:47.047296+0000","last_clean":"2023-10-16T15:33:47.047296+0000","last_became_active":"2023-10-16T15:33:44.615654+0000","last_became_peered":"2023-10-16T15:33:44.615654+0000","last_unstale":"2023-10-16T15:33:47.047296+0000","last_undegraded":"2023-10-16T15:33:47.047296+0000","last_fullsized":"2023-10-16T15:33:47.047296+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:20.679846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.602817+0000","last_change":"2023-10-16T15:33:45.634927+0000","last_active":"2023-10-16T15:33:46.602817+0000","last_peered":"2023-10-16T15:33:46.602817+0000","last_clean":"2023-10-16T15:33:46.602817+0000","last_became_active":"2023-10-16T15:33:45.634638+0000","last_became_peered":"2023-10-16T15:33:45.634638+0000","last_unstale":"2023-10-16T15:33:46.602817+0000","last_undegraded":"2023-10-16T15:33:46.602817+0000","last_fullsized":"2023-10-16T15:33:46.602817+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:44.575459+0000","last_clean_scrub_stamp":"2023-10-16T15:33:44.575459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:50.175946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"178'3","reported_seq":15,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-10-16T15:33:46.609833+0000","last_change":"2023-10-16T15:33:44.622869+0000","last_active":"2023-10-16T15:33:46.609833+0000","last_peered":"2023-10-16T15:33:46.609833+0000","last_clean":"2023-10-16T15:33:46.609833+0000","last_became_active":"2023-10-16T15:33:44.622580+0000","last_became_peered":"2023-10-16T15:33:44.622580+0000","last_unstale":"2023-10-16T15:33:46.609833+0000","last_undegraded":"2023-10-16T15:33:46.609833+0000","last_fullsized":"2023-10-16T15:33:46.609833+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:33:43.558894+0000","last_clean_scrub_stamp":"2023-10-16T15:33:43.558894+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:05.272611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"167'274","reported_seq":918,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-16T15:33:45.610197+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:33:45.610197+0000","last_peered":"2023-10-16T15:33:45.610197+0000","last_clean":"2023-10-16T15:33:45.610197+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:33:45.610197+0000","last_undegraded":"2023-10-16T15:33:45.610197+0000","last_fullsized":"2023-10-16T15:33:45.610197+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575772,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56164,"kb_used_data":2768,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315676,"statfs":{"total":96636764160,"available":96579252224,"internally_reserved":0,"allocated":2834432,"data_stored":1680591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1990000000000001,"15min":1.2110000000000001},"min":{"1min":0.86099999999999999,"5min":0.753,"15min":0.69099999999999995},"max":{"1min":1.6659999999999999,"5min":1.752,"15min":1.7849999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.159,"15min":1.1619999999999999},"min":{"1min":0.86799999999999999,"5min":0.78300000000000003,"15min":0.56999999999999995},"max":{"1min":1.867,"5min":1.867,"15min":1.905},"last":1.2829999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2390000000000001,"15min":1.224},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.80000000000000004},"max":{"1min":1.6879999999999999,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3660000000000001,"15min":1.337},"min":{"1min":1.0069999999999999,"5min":0.86299999999999999,"15min":0.78200000000000003},"max":{"1min":1.881,"5min":2.282,"15min":2.282},"last":1.3380000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.319,"15min":1.357},"min":{"1min":0.96199999999999997,"5min":0.91000000000000003,"15min":0.80200000000000005},"max":{"1min":1.7549999999999999,"5min":1.784,"15min":1.9039999999999999},"last":1.216},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3540000000000001,"15min":1.3320000000000001},"min":{"1min":0.97299999999999998,"5min":0.91700000000000004,"15min":0.82599999999999996},"max":{"1min":1.925,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.752}]},{"osd":3,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3640000000000001,"15min":1.3500000000000001},"min":{"1min":1.093,"5min":0.93400000000000005,"15min":0.90700000000000003},"max":{"1min":1.7390000000000001,"5min":2.528,"15min":2.528},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4710000000000001,"15min":1.4590000000000001},"min":{"1min":1.032,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.9410000000000001,"5min":1.956,"15min":1.962},"last":1.96}]},{"osd":4,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.4039999999999999,"15min":1.395},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.81699999999999995},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.954},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3919999999999999,"15min":1.3320000000000001},"min":{"1min":0.75900000000000001,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":1.9099999999999999,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":2.0329999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.502,"15min":1.4710000000000001},"min":{"1min":1.024,"5min":0.97599999999999998,"15min":0.89000000000000001},"max":{"1min":2.032,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.6240000000000001,"15min":1.5720000000000001},"min":{"1min":1.1850000000000001,"5min":0.92700000000000005,"15min":0.88500000000000001},"max":{"1min":2.0049999999999999,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":2.0880000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:33:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5209999999999999,"15min":1.4850000000000001},"min":{"1min":1.212,"5min":1.099,"15min":0.90700000000000003},"max":{"1min":2.133,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":2.1389999999999998},{"interface":"front","average":{"1min":1.673,"5min":1.645,"15min":1.6020000000000001},"min":{"1min":1.3140000000000001,"5min":1.0629999999999999,"15min":0.95099999999999996},"max":{"1min":2.0720000000000001,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":2.1920000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29216,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342624,"statfs":{"total":96636764160,"available":96606846976,"internally_reserved":0,"allocated":2244608,"data_stored":1089717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1890000000000001,"15min":1.1850000000000001},"min":{"1min":0.88400000000000001,"5min":0.77200000000000002,"15min":0.71899999999999997},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.556},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.282,"15min":1.264},"min":{"1min":0.84599999999999997,"5min":0.82099999999999995,"15min":0.80800000000000005},"max":{"1min":1.823,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.165}]},{"osd":1,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.252,"15min":1.2889999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.68600000000000005},"max":{"1min":1.653,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.335,"15min":1.3280000000000001},"min":{"1min":0.93100000000000005,"5min":0.874,"15min":0.85099999999999998},"max":{"1min":1.7,"5min":1.843,"15min":2.738},"last":1.843}]},{"osd":2,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2649999999999999,"15min":1.2829999999999999},"min":{"1min":0.93100000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.823,"5min":1.823,"15min":2.4039999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.405,"15min":1.3859999999999999},"min":{"1min":1.109,"5min":0.90400000000000003,"15min":0.89400000000000002},"max":{"1min":1.756,"5min":1.756,"15min":1.9139999999999999},"last":0.90200000000000002}]},{"osd":3,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3919999999999999,"15min":1.401},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.6100000000000001,"5min":1.8580000000000001,"15min":3.9820000000000002},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.383,"15min":1.379},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.92100000000000004},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.5489999999999999},"last":1.2210000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3160000000000001,"15min":1.3129999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":1.6879999999999999,"5min":1.9019999999999999,"15min":1.948},"last":1.766},{"interface":"front","average":{"1min":1.339,"5min":1.381,"15min":1.3799999999999999},"min":{"1min":0.88900000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7629999999999999,"5min":1.9159999999999999,"15min":4.6609999999999996},"last":1.4690000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.371,"15min":1.3540000000000001},"min":{"1min":1.0269999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.7130000000000001,"5min":1.9430000000000001,"15min":3.8450000000000002},"last":1.419},{"interface":"front","average":{"1min":1.508,"5min":1.5369999999999999,"15min":1.518},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.86299999999999999},"max":{"1min":1.897,"5min":2.02,"15min":2.8639999999999999},"last":1.677}]},{"osd":7,"last update":"Mon Oct 16 15:32:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5860000000000001,"15min":1.5920000000000001},"min":{"1min":1.359,"5min":1.0249999999999999,"15min":0.84799999999999998},"max":{"1min":1.8819999999999999,"5min":1.9610000000000001,"15min":4.5359999999999996},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.615,"15min":1.591},"min":{"1min":1.335,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.9330000000000001,"5min":2.1970000000000001,"15min":2.6619999999999999},"last":1.613}]}]},{"osd":5,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41688,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330152,"statfs":{"total":96636764160,"available":96594075648,"internally_reserved":0,"allocated":2236416,"data_stored":1089593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1990000000000001,"15min":1.1830000000000001},"min":{"1min":0.82299999999999995,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.254,"15min":1.252},"min":{"1min":0.85099999999999998,"5min":0.82899999999999996,"15min":0.78400000000000003},"max":{"1min":1.8240000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.131}]},{"osd":1,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.246,"15min":1.2370000000000001},"min":{"1min":0.97499999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3149999999999999,"15min":1.298},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.83699999999999997},"max":{"1min":1.7050000000000001,"5min":1.7949999999999999,"15min":1.905},"last":1.327}]},{"osd":2,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.278,"15min":1.2669999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.83099999999999996},"max":{"1min":1.883,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.399,"15min":1.399},"min":{"1min":1.0509999999999999,"5min":0.95599999999999996,"15min":0.91700000000000004},"max":{"1min":1.6850000000000001,"5min":2.073,"15min":2.3130000000000002},"last":0.96399999999999997}]},{"osd":3,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.327,"15min":1.325},"min":{"1min":0.997,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.9410000000000001,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.405,"5min":1.387,"15min":1.377},"min":{"1min":1.0700000000000001,"5min":0.94999999999999996,"15min":0.91800000000000004},"max":{"1min":1.673,"5min":2.1800000000000002,"15min":2.1800000000000002},"last":1.1930000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3100000000000001,"15min":1.3180000000000001},"min":{"1min":1.032,"5min":0.78800000000000003,"15min":0.78500000000000003},"max":{"1min":1.5720000000000001,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.034},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4690000000000001,"15min":1.456},"min":{"1min":0.93999999999999995,"5min":0.83199999999999996,"15min":0.78400000000000003},"max":{"1min":1.7849999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.623}]},{"osd":6,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3740000000000001,"15min":1.3560000000000001},"min":{"1min":0.89000000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.7290000000000001,"5min":2.4569999999999999,"15min":2.508},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.452,"15min":1.4330000000000001},"min":{"1min":1.121,"5min":0.91300000000000003,"15min":0.89600000000000002},"max":{"1min":1.7749999999999999,"5min":2.9209999999999998,"15min":2.9790000000000001},"last":1.3660000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:33:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.5249999999999999,"15min":1.506},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.89600000000000002},"max":{"1min":1.734,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.6040000000000001,"15min":1.601},"min":{"1min":1.359,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":2.0150000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.399}]}]},{"osd":4,"up_from":13,"seq":55834575772,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59736,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312104,"statfs":{"total":96636764160,"available":96575594496,"internally_reserved":0,"allocated":2232320,"data_stored":1089571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2010000000000001,"15min":1.1930000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6220000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.1639999999999999,"15min":1.1830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82199999999999995,"15min":0.745},"max":{"1min":1.458,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.91800000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2729999999999999,"15min":1.2350000000000001},"min":{"1min":0.93000000000000005,"5min":0.88900000000000001,"15min":0.80400000000000005},"max":{"1min":1.583,"5min":1.78,"15min":1.948},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.272,"15min":1.276},"min":{"1min":0.83399999999999996,"5min":0.82099999999999995,"15min":0.80500000000000005},"max":{"1min":1.6910000000000001,"5min":1.7230000000000001,"15min":1.895},"last":1.119}]},{"osd":2,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4379999999999999,"15min":1.423},"min":{"1min":1.202,"5min":0.88,"15min":0.88},"max":{"1min":1.681,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.331},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4379999999999999,"15min":1.4430000000000001},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.835,"5min":2,"15min":2},"last":1.6559999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":1.0109999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.778,"5min":1.8149999999999999,"15min":2.093},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.375,"15min":1.3779999999999999},"min":{"1min":1.0700000000000001,"5min":1.0369999999999999,"15min":0.93100000000000005},"max":{"1min":1.756,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.4179999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.345,"15min":1.337},"min":{"1min":0.93700000000000006,"5min":0.89300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8120000000000001,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.236},{"interface":"front","average":{"1min":1.335,"5min":1.381,"15min":1.4059999999999999},"min":{"1min":0.93500000000000005,"5min":0.86799999999999999,"15min":0.86299999999999999},"max":{"1min":1.7949999999999999,"5min":1.881,"15min":2.097},"last":1.2729999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.375,"15min":1.351},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.903,"15min":1.903},"last":1.601},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.518},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.833,"5min":2.359,"15min":2.359},"last":1.536}]},{"osd":7,"last update":"Mon Oct 16 15:33:04 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5329999999999999,"15min":1.556},"min":{"1min":1.248,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":1.875,"5min":1.9059999999999999,"15min":2.1600000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.54,"15min":1.5409999999999999},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":0.97499999999999998},"max":{"1min":1.887,"5min":2.0720000000000001,"15min":2.3330000000000002},"last":1.7969999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51416,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320424,"statfs":{"total":96636764160,"available":96584114176,"internally_reserved":0,"allocated":2236416,"data_stored":1089661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.94399999999999995,"15min":0.92500000000000004},"min":{"1min":0.69299999999999995,"5min":0.67700000000000005,"15min":0.64800000000000002},"max":{"1min":1.194,"5min":1.569,"15min":1.7230000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.054,"5min":1.099,"15min":1.0880000000000001},"min":{"1min":0.72399999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4059999999999999,"5min":1.843,"15min":1.9430000000000001},"last":1.2130000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.98299999999999998,"15min":0.998},"min":{"1min":0.73099999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.2789999999999999,"5min":1.704,"15min":1.704},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.236,"5min":1.242,"15min":1.2130000000000001},"min":{"1min":0.93700000000000006,"5min":0.86499999999999999,"15min":0.81399999999999995},"max":{"1min":1.5920000000000001,"5min":1.7210000000000001,"15min":1.823},"last":0.91100000000000003}]},{"osd":2,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4119999999999999,"15min":1.4219999999999999},"min":{"1min":0.88200000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.7729999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.552},{"interface":"front","average":{"1min":1.522,"5min":1.546,"15min":1.5309999999999999},"min":{"1min":1.157,"5min":1.1559999999999999,"15min":1.091},"max":{"1min":1.9059999999999999,"5min":2.278,"15min":2.278},"last":1.5840000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3660000000000001,"15min":1.331},"min":{"1min":1.0249999999999999,"5min":0.93200000000000005,"15min":0.90800000000000003},"max":{"1min":1.78,"5min":1.8560000000000001,"15min":1.877},"last":1.123},{"interface":"front","average":{"1min":1.381,"5min":1.379,"15min":1.399},"min":{"1min":1.109,"5min":1.05,"15min":1.01},"max":{"1min":1.536,"5min":2.0089999999999999,"15min":2.1840000000000002},"last":1.0369999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5429999999999999,"15min":1.5669999999999999},"min":{"1min":1.071,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":1.7829999999999999,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.6200000000000001,"15min":1.5660000000000001},"min":{"1min":1.173,"5min":1.0640000000000001,"15min":1.0640000000000001},"max":{"1min":2.3479999999999999,"5min":2.774,"15min":2.774},"last":1.1870000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.542,"15min":1.526},"min":{"1min":1.0009999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.036,"5min":2.3050000000000002,"15min":2.3719999999999999},"last":1.375},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.605,"15min":1.5960000000000001},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":1.8999999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.1579999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:32:54 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.71,"15min":1.671},"min":{"1min":1.2989999999999999,"5min":1.147,"15min":1.147},"max":{"1min":2.5150000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.05},"max":{"1min":2.1259999999999999,"5min":2.496,"15min":2.496},"last":1.4730000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575772,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51420,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320420,"statfs":{"total":96636764160,"available":96584110080,"internally_reserved":0,"allocated":2236416,"data_stored":1084925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.95999999999999996,"15min":0.92800000000000005},"min":{"1min":0.70099999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1439999999999999,"5min":1.849,"15min":1.849},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.105,"15min":1.1120000000000001},"min":{"1min":0.79200000000000004,"5min":0.70699999999999996,"15min":0.61699999999999999},"max":{"1min":1.5780000000000001,"5min":1.6890000000000001,"15min":1.768},"last":0.69299999999999995}]},{"osd":1,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.0409999999999999,"15min":1.0269999999999999},"min":{"1min":0.82099999999999995,"5min":0.753,"15min":0.72399999999999998},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.8140000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.256,"15min":1.24},"min":{"1min":0.88900000000000001,"5min":0.88100000000000001,"15min":0.80700000000000005},"max":{"1min":1.6479999999999999,"5min":1.8300000000000001,"15min":1.9299999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3300000000000001,"15min":1.331},"min":{"1min":1.0640000000000001,"5min":0.875,"15min":0.79900000000000004},"max":{"1min":1.6279999999999999,"5min":1.8420000000000001,"15min":1.992},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.381},"min":{"1min":0.93000000000000005,"5min":0.91800000000000004,"15min":0.67200000000000004},"max":{"1min":1.7689999999999999,"5min":1.921,"15min":1.921},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.498,"15min":1.484},"min":{"1min":1.179,"5min":1.113,"15min":0.96999999999999997},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.149},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.395,"15min":1.409},"min":{"1min":1.097,"5min":0.93899999999999995,"15min":0.92400000000000004},"max":{"1min":1.986,"5min":1.986,"15min":2.6909999999999998},"last":1.3779999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.542,"15min":1.488},"min":{"1min":1.095,"5min":1.0389999999999999,"15min":1.028},"max":{"1min":2.661,"5min":2.661,"15min":2.661},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.635,"15min":1.6259999999999999},"min":{"1min":1.3939999999999999,"5min":1.173,"15min":1.141},"max":{"1min":1.871,"5min":2.3730000000000002,"15min":3.3149999999999999},"last":1.4379999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.508,"15min":1.5149999999999999},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":0.92400000000000004},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.573},"last":1.464},{"interface":"front","average":{"1min":1.673,"5min":1.641,"15min":1.6060000000000001},"min":{"1min":1.403,"5min":1.1000000000000001,"15min":1.1000000000000001},"max":{"1min":2.3900000000000001,"5min":2.423,"15min":2.423},"last":1.492}]},{"osd":7,"last update":"Mon Oct 16 15:32:51 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.593,"15min":1.6100000000000001},"min":{"1min":1.129,"5min":1.099,"15min":1.0049999999999999},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.552},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.7430000000000001,"15min":1.7170000000000001},"min":{"1min":1.379,"5min":1.286,"15min":1.234},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.585}]}]},{"osd":1,"up_from":13,"seq":55834575772,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59732,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312108,"statfs":{"total":96636764160,"available":96575598592,"internally_reserved":0,"allocated":2236416,"data_stored":1090153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91400000000000003,"15min":0.96099999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.6830000000000001,"5min":1.716,"15min":1.8440000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1599999999999999,"15min":1.1259999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2010000000000001,"15min":1.22},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.53,"5min":1.7410000000000001,"15min":1.9430000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.371,"15min":1.379},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.012,"5min":2.012,"15min":2.02},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.258,"15min":1.2729999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.73499999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.385,"15min":1.361},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.883,"5min":1.8879999999999999,"15min":1.9990000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.427,"5min":1.4730000000000001,"15min":1.4930000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.79,"5min":1.9059999999999999,"15min":2.3500000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.532,"15min":1.4970000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.0619999999999998,"5min":2.2770000000000001,"15min":2.2770000000000001},"last":1.073},{"interface":"front","average":{"1min":1.429,"5min":1.472,"15min":1.462},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.889,"5min":2.1059999999999999,"15min":2.1200000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.629,"15min":1.639},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6399999999999999,"15min":1.611},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.1579999999999999,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":0.89600000000000002}]},{"osd":7,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.756,"15min":1.7170000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":2.274,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.125},{"interface":"front","average":{"1min":1.621,"5min":1.615,"15min":1.6080000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.036,"5min":2.0409999999999999,"15min":2.3039999999999998},"last":0.85799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60328,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":2842624,"data_stored":1681585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.997,"15min":0.97699999999999998},"min":{"1min":0.72499999999999998,"5min":0.65400000000000003,"15min":0.61299999999999999},"max":{"1min":1.5229999999999999,"5min":1.903,"15min":1.903},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2609999999999999,"15min":1.2490000000000001},"min":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.82299999999999995},"max":{"1min":1.6639999999999999,"5min":1.851,"15min":1.9450000000000001},"last":1.248}]},{"osd":2,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.3480000000000001,"15min":1.3180000000000001},"min":{"1min":0.90600000000000003,"5min":0.81100000000000005,"15min":0.70099999999999996},"max":{"1min":1.73,"5min":1.8939999999999999,"15min":2.0640000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.286,"15min":1.2789999999999999},"min":{"1min":0.84499999999999997,"5min":0.79500000000000004,"15min":0.64000000000000001},"max":{"1min":1.6319999999999999,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.028}]},{"osd":3,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0549999999999999,"15min":1.028},"min":{"1min":0.77000000000000002,"5min":0.75700000000000001,"15min":0.753},"max":{"1min":1.212,"5min":1.732,"15min":1.732},"last":1.212},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.95199999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8500000000000001,"5min":2.262,"15min":2.262},"last":1.3720000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3520000000000001,"15min":1.3340000000000001},"min":{"1min":1.117,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":1.6100000000000001,"5min":2.109,"15min":2.109},"last":1.403},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.554,"15min":1.5289999999999999},"min":{"1min":1.093,"5min":1.093,"15min":1.0029999999999999},"max":{"1min":1.853,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.4970000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.5029999999999999,"15min":1.494},"min":{"1min":1.1819999999999999,"5min":1.075,"15min":0.96599999999999997},"max":{"1min":1.776,"5min":2.411,"15min":2.4239999999999999},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4450000000000001,"15min":1.4419999999999999},"min":{"1min":1.151,"5min":1.069,"15min":1.0609999999999999},"max":{"1min":1.6559999999999999,"5min":2.0880000000000001,"15min":2.1899999999999999},"last":1.534}]},{"osd":6,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6990000000000001,"15min":1.6539999999999999},"min":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2250000000000001},"max":{"1min":1.988,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":2.3130000000000002},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5940000000000001,"15min":1.5820000000000001},"min":{"1min":1.1639999999999999,"5min":1.0800000000000001,"15min":1.0169999999999999},"max":{"1min":1.851,"5min":2.4340000000000002,"15min":2.5009999999999999},"last":2.0459999999999998}]},{"osd":7,"last update":"Mon Oct 16 15:33:06 2023","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.706,"15min":1.6910000000000001},"min":{"1min":1.302,"5min":1.1379999999999999,"15min":1.087},"max":{"1min":1.911,"5min":2.419,"15min":2.5710000000000002},"last":3.089},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.6319999999999999,"15min":1.6040000000000001},"min":{"1min":1.038,"5min":1.038,"15min":1.0189999999999999},"max":{"1min":1.772,"5min":2.774,"15min":2.774},"last":1.355}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:33:50.738 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:33:50.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T15:33:51.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.016+0000 7fcc60a38640 1 Processor -- start 2023-10-16T15:33:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.016+0000 7fcc60a38640 1 -- start start 2023-10-16T15:33:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc60a38640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc5c070bf0 0x7fcc5c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc60a38640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c15e5c0 con 0x7fcc5c159ac0 2023-10-16T15:33:51.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc60a38640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c06b810 con 0x7fcc5c0cc270 2023-10-16T15:33:51.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc60a38640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5c0cc810 con 0x7fcc5c070bf0 2023-10-16T15:33:51.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc5ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcc5c159ac0 0x7fcc5c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44860/0 (socket says 172.21.15.29:44860) 2023-10-16T15:33:51.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc5ad9d640 1 -- 172.21.15.29:0/1081276283 learned_addr learned my addr 172.21.15.29:0/1081276283 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:51.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc5b59e640 1 --2- 172.21.15.29:0/1081276283 >> v2:172.21.15.29:3300/0 conn(0x7fcc5c070bf0 0x7fcc5c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.018+0000 7fcc5a59c640 1 -- 172.21.15.29:0/1081276283 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249361098 0 0) 0x7fcc4c001110 con 0x7fcc5c159ac0 2023-10-16T15:33:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.019+0000 7fcc5a59c640 1 -- 172.21.15.29:0/1081276283 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48003520 con 0x7fcc5c159ac0 2023-10-16T15:33:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.019+0000 7fcc5b59e640 1 -- 172.21.15.29:0/1081276283 >> v1:172.21.15.170:6789/0 conn(0x7fcc5c0cc270 legacy=0x7fcc5c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.019+0000 7fcc5b59e640 1 -- 172.21.15.29:0/1081276283 >> v1:172.21.15.29:6789/0 conn(0x7fcc5c159ac0 legacy=0x7fcc5c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.019+0000 7fcc5b59e640 1 -- 172.21.15.29:0/1081276283 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c06b1c0 con 0x7fcc5c070bf0 2023-10-16T15:33:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.020+0000 7fcc5b59e640 1 --2- 172.21.15.29:0/1081276283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c15e080 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009d90 tx=0x7fcc5402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36528d4f722d4280 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:51.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.020+0000 7fcc5a59c640 1 -- 172.21.15.29:0/1081276283 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5402ccf0 con 0x7fcc5c070bf0 2023-10-16T15:33:51.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.020+0000 7fcc5a59c640 1 -- 172.21.15.29:0/1081276283 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5402ce50 con 0x7fcc5c070bf0 2023-10-16T15:33:51.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.020+0000 7fcc5a59c640 1 -- 172.21.15.29:0/1081276283 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54034660 con 0x7fcc5c070bf0 2023-10-16T15:33:51.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 -- 172.21.15.29:0/1081276283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c070bf0 msgr2=0x7fcc5c15e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 --2- 172.21.15.29:0/1081276283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c15e080 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009d90 tx=0x7fcc5402bf40 comp rx=0 tx=0).stop 2023-10-16T15:33:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 -- 172.21.15.29:0/1081276283 shutdown_connections 2023-10-16T15:33:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 --2- 172.21.15.29:0/1081276283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c15e080 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 -- 172.21.15.29:0/1081276283 >> 172.21.15.29:0/1081276283 conn(0x7fcc5c0ce6a0 msgr2=0x7fcc5c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 -- 172.21.15.29:0/1081276283 shutdown_connections 2023-10-16T15:33:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.021+0000 7fcc60a38640 1 -- 172.21.15.29:0/1081276283 wait complete. 2023-10-16T15:33:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.023+0000 7fcc60a38640 1 Processor -- start 2023-10-16T15:33:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.023+0000 7fcc60a38640 1 -- start start 2023-10-16T15:33:51.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc60a38640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c14fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc60a38640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc5c0cc270 0x7fcc5c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc60a38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 0x7fcc5c0c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc60a38640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0ad990 con 0x7fcc5c159ac0 2023-10-16T15:33:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc60a38640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0afc00 con 0x7fcc5c070bf0 2023-10-16T15:33:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc60a38640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0afee0 con 0x7fcc5c0cc270 2023-10-16T15:33:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc5b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 0x7fcc5c0c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc5b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 0x7fcc5c0c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46402/0 (socket says 172.21.15.29:46402) 2023-10-16T15:33:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.024+0000 7fcc5b59e640 1 -- 172.21.15.29:0/3622487699 learned_addr learned my addr 172.21.15.29:0/3622487699 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc53fff640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc5c0cc270 0x7fcc5c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5ad9d640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c14fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5b59e640 1 -- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc5c0cc270 msgr2=0x7fcc5c158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5b59e640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc5c0cc270 0x7fcc5c158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5b59e640 1 -- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc5c070bf0 msgr2=0x7fcc5c14fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5b59e640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c14fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5b59e640 1 -- 172.21.15.29:0/3622487699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54009a70 con 0x7fcc5c159ac0 2023-10-16T15:33:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.025+0000 7fcc5b59e640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 0x7fcc5c0c7940 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fcc54037040 tx=0x7fcc54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.026+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5403d600 con 0x7fcc5c159ac0 2023-10-16T15:33:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.026+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5403d760 con 0x7fcc5c159ac0 2023-10-16T15:33:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.026+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5403db10 con 0x7fcc5c159ac0 2023-10-16T15:33:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.026+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c0ad340 con 0x7fcc5c159ac0 2023-10-16T15:33:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.026+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c14ff60 con 0x7fcc5c159ac0 2023-10-16T15:33:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.027+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c04e9b0 con 0x7fcc5c159ac0 2023-10-16T15:33:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.028+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc5403b040 con 0x7fcc5c159ac0 2023-10-16T15:33:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.029+0000 7fcc537fe640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc3c06afc0 0x7fcc3c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.029+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcc540abd70 con 0x7fcc5c159ac0 2023-10-16T15:33:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.030+0000 7fcc5ad9d640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc3c06afc0 0x7fcc3c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.031+0000 7fcc5ad9d640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc3c06afc0 0x7fcc3c06d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c007fd0 tx=0x7fcc4c009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:51.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.034+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcc54076c50 con 0x7fcc5c159ac0 2023-10-16T15:33:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.297+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcc5c0b4620 con 0x7fcc5c159ac0 2023-10-16T15:33:51.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.303+0000 7fcc537fe640 1 -- 172.21.15.29:0/3622487699 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcc540390b0 con 0x7fcc5c159ac0 2023-10-16T15:33:51.306 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T15:33:51.306 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T15:33:51.306 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T15:33:51.306 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T15:33:51.307 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T15:33:51.307 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T15:33:51.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc3c06afc0 msgr2=0x7fcc3c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc3c06afc0 0x7fcc3c06d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c007fd0 tx=0x7fcc4c009040 comp rx=0 tx=0).stop 2023-10-16T15:33:51.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 msgr2=0x7fcc5c0c7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 0x7fcc5c0c7940 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fcc54037040 tx=0x7fcc54004a40 comp rx=0 tx=0).stop 2023-10-16T15:33:51.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 shutdown_connections 2023-10-16T15:33:51.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcc3c06afc0 0x7fcc3c06d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc5c159ac0 0x7fcc5c0c7940 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc5c0cc270 0x7fcc5c158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 --2- 172.21.15.29:0/3622487699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc5c070bf0 0x7fcc5c14fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 >> 172.21.15.29:0/3622487699 conn(0x7fcc5c0ce6a0 msgr2=0x7fcc5c0cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:51.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 shutdown_connections 2023-10-16T15:33:51.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.306+0000 7fcc60a38640 1 -- 172.21.15.29:0/3622487699 wait complete. 2023-10-16T15:33:51.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:33:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.534+0000 7f4543363640 1 Processor -- start 2023-10-16T15:33:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.535+0000 7f4543363640 1 -- start start 2023-10-16T15:33:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.536+0000 7f4543363640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f453c159950 0x7f453c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.536+0000 7f4543363640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f453c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.536+0000 7f4543363640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c070df0 con 0x7f453c06c100 2023-10-16T15:33:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.536+0000 7f4543363640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f453c15e490 con 0x7f453c159950 2023-10-16T15:33:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.536+0000 7f4543363640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453c15aef0 con 0x7f453c070450 2023-10-16T15:33:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f4542b62640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f453c06c100 0x7f453c06c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44862/0 (socket says 172.21.15.29:44862) 2023-10-16T15:33:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f4542b62640 1 -- 172.21.15.29:0/506625588 learned_addr learned my addr 172.21.15.29:0/506625588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f4541b60640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f453c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f4542361640 1 --2- 172.21.15.29:0/506625588 >> v2:172.21.15.170:3300/0 conn(0x7f453c159950 0x7f453c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721308931 0 0) 0x7f453c070df0 con 0x7f453c06c100 2023-10-16T15:33:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4528003520 con 0x7f453c06c100 2023-10-16T15:33:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2843347358 0 0) 0x7f4528003520 con 0x7f453c06c100 2023-10-16T15:33:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 msgr2=0x7f453c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f453c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 >> v2:172.21.15.170:3300/0 conn(0x7f453c159950 msgr2=0x7f453c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 --2- 172.21.15.29:0/506625588 >> v2:172.21.15.170:3300/0 conn(0x7f453c159950 0x7f453c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.537+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c06cc80 con 0x7f453c06c100 2023-10-16T15:33:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f45340019f0 con 0x7f453c06c100 2023-10-16T15:33:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 >> v1:172.21.15.29:6789/0 conn(0x7f453c06c100 legacy=0x7f453c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f4528003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f45280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4528004d00 0x7f4528005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c15e490 con 0x7f4528003970 2023-10-16T15:33:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c15aef0 con 0x7f4528004d00 2023-10-16T15:33:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452800ba00 con 0x7f453c070450 2023-10-16T15:33:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4542361640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f4528003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4541b60640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f45280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4542361640 1 -- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 msgr2=0x7f45280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4542361640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f45280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4542361640 1 -- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4528004d00 msgr2=0x7f4528005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4542361640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4528004d00 0x7f4528005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4542361640 1 -- 172.21.15.29:0/506625588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4528003520 con 0x7f4528003970 2023-10-16T15:33:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.538+0000 7f4541b60640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c070450 0x7f45280047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f4542361640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f4528003d90 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f4538009ac0 tx=0x7f45280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a116e5774bd09f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453802c6f0 con 0x7f4528003970 2023-10-16T15:33:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453802c850 con 0x7f4528003970 2023-10-16T15:33:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f454135f640 1 -- 172.21.15.29:0/506625588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453802cbe0 con 0x7f4528003970 2023-10-16T15:33:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f4543363640 1 -- 172.21.15.29:0/506625588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 msgr2=0x7f4528003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f4543363640 1 --2- 172.21.15.29:0/506625588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f4528003d90 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f4538009ac0 tx=0x7f45280057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f4542b62640 1 -- 172.21.15.29:0/506625588 reap_dead start 2023-10-16T15:33:51.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f4543363640 1 -- 172.21.15.29:0/506625588 shutdown_connections 2023-10-16T15:33:51.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.540+0000 7f4543363640 1 -- 172.21.15.29:0/506625588 >> 172.21.15.29:0/506625588 conn(0x7f453c0ce6a0 msgr2=0x7f453c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:51.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.541+0000 7f4543363640 1 -- 172.21.15.29:0/506625588 shutdown_connections 2023-10-16T15:33:51.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.541+0000 7f4543363640 1 -- 172.21.15.29:0/506625588 wait complete. 2023-10-16T15:33:51.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.542+0000 7f4543363640 1 Processor -- start 2023-10-16T15:33:51.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 -- start start 2023-10-16T15:33:51.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f453c154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f453c1559a0 0x7f453c0c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c0c4850 0x7f453c0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c0c2b60 con 0x7f4528003970 2023-10-16T15:33:51.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c155150 con 0x7f453c1559a0 2023-10-16T15:33:51.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.543+0000 7f4543363640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453c0b0ea0 con 0x7f453c0c4850 2023-10-16T15:33:51.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f453c154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f453c154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46416/0 (socket says 172.21.15.29:46416) 2023-10-16T15:33:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 -- 172.21.15.29:0/1837261071 learned_addr learned my addr 172.21.15.29:0/1837261071 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 -- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c0c4850 msgr2=0x7f453c0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542b62640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c0c4850 0x7f453c0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4541b60640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f453c1559a0 0x7f453c0c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c0c4850 0x7f453c0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 -- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f453c1559a0 msgr2=0x7f453c0c2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f453c1559a0 0x7f453c0c2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542361640 1 -- 172.21.15.29:0/1837261071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c0c4d50 con 0x7f4528003970 2023-10-16T15:33:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4542b62640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c0c4850 0x7f453c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.544+0000 7f4541b60640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f453c1559a0 0x7f453c0c2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.545+0000 7f4542361640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f453c154c10 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f453802c290 tx=0x7f4538004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:51.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.545+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538003810 con 0x7f4528003970 2023-10-16T15:33:51.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.545+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4538003970 con 0x7f4528003970 2023-10-16T15:33:51.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.545+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538003ce0 con 0x7f4528003970 2023-10-16T15:33:51.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.545+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c0b08e0 con 0x7f4528003970 2023-10-16T15:33:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.545+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c155330 con 0x7f4528003970 2023-10-16T15:33:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.546+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453c04e9b0 con 0x7f4528003970 2023-10-16T15:33:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.547+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4538040450 con 0x7f4528003970 2023-10-16T15:33:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.548+0000 7f4532ffd640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f452006af20 0x7f452006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.548+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f45380ab740 con 0x7f4528003970 2023-10-16T15:33:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.549+0000 7f4541b60640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f452006af20 0x7f452006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.550+0000 7f4541b60640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f452006af20 0x7f452006d3e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f453c155d80 tx=0x7f452c008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.552+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4538076740 con 0x7f4528003970 2023-10-16T15:33:51.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.776+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f453c0c2930 con 0x7f4528003970 2023-10-16T15:33:51.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.777+0000 7f4532ffd640 1 -- 172.21.15.29:0/1837261071 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f4538079d60 con 0x7f4528003970 2023-10-16T15:33:51.780 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:33:51.780 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:33:45.081584+0000","modified":"2023-10-16T15:33:48.236102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9342":{"gid":9342,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3104953668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3104953668},{"type":"v1","addr":"172.21.15.29:6833","nonce":3104953668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T15:33:51.780 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 84 2023-10-16T15:33:51.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f452006af20 msgr2=0x7f452006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f452006af20 0x7f452006d3e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f453c155d80 tx=0x7f452c008040 comp rx=0 tx=0).stop 2023-10-16T15:33:51.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 msgr2=0x7f453c154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:51.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f453c154c10 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f453802c290 tx=0x7f4538004730 comp rx=0 tx=0).stop 2023-10-16T15:33:51.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 shutdown_connections 2023-10-16T15:33:51.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f452006af20 0x7f452006d3e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f453c0c4850 0x7f453c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f453c1559a0 0x7f453c0c2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 --2- 172.21.15.29:0/1837261071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4528003970 0x7f453c154c10 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 >> 172.21.15.29:0/1837261071 conn(0x7f453c0ce6a0 msgr2=0x7f453c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 shutdown_connections 2023-10-16T15:33:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:51.780+0000 7f4543363640 1 -- 172.21.15.29:0/1837261071 wait complete. 2023-10-16T15:33:52.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:33:45.081584+0000', 'modified': '2023-10-16T15:33:48.236102+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9342}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9342': {'gid': 9342, 'name': 'a', 'rank': 0, 'incarnation': 80, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6833/3104953668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 3104953668}, {'type': 'v1', 'addr': '172.21.15.29:6833', 'nonce': 3104953668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:33:52.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:33:52.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T15:33:52.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.439+0000 7fcd83fff640 1 Processor -- start 2023-10-16T15:33:52.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.439+0000 7fcd83fff640 1 -- start start 2023-10-16T15:33:52.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd83fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 0x7fcd840ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd83fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd840cd4a0 con 0x7fcd84153740 2023-10-16T15:33:52.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd83fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8415ac60 con 0x7fcd840ccb00 2023-10-16T15:33:52.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd83fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84070030 con 0x7fcd8415c100 2023-10-16T15:33:52.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd82ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcd84153740 0x7fcd8415a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44870/0 (socket says 172.21.15.29:44870) 2023-10-16T15:33:52.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd82ffd640 1 -- 172.21.15.29:0/1795532657 learned_addr learned my addr 172.21.15.29:0/1795532657 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:52.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324742591 0 0) 0x7fcd840cd4a0 con 0x7fcd84153740 2023-10-16T15:33:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd6c003520 con 0x7fcd84153740 2023-10-16T15:33:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.441+0000 7fcd837fe640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 0x7fcd840ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4140598364 0 0) 0x7fcd6c003520 con 0x7fcd84153740 2023-10-16T15:33:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 msgr2=0x7fcd840ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 0x7fcd840ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 >> v1:172.21.15.170:6789/0 conn(0x7fcd840ccb00 legacy=0x7fcd840ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84070310 con 0x7fcd84153740 2023-10-16T15:33:52.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fcd740019f0 con 0x7fcd84153740 2023-10-16T15:33:52.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 >> v1:172.21.15.29:6789/0 conn(0x7fcd84153740 legacy=0x7fcd8415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c0039a0 0x7fcd6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 0x7fcd6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84070030 con 0x7fcd6c004d00 2023-10-16T15:33:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c00bb10 con 0x7fcd6c0039a0 2023-10-16T15:33:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c00dee0 con 0x7fcd8415c100 2023-10-16T15:33:52.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.442+0000 7fcd837fe640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd827fc640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 0x7fcd6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c0039a0 0x7fcd6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd837fe640 1 -- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 msgr2=0x7fcd6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd837fe640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd8415c100 0x7fcd6c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd837fe640 1 -- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c0039a0 msgr2=0x7fcd6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd837fe640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c0039a0 0x7fcd6c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd837fe640 1 -- 172.21.15.29:0/1795532657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c003520 con 0x7fcd6c004d00 2023-10-16T15:33:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.443+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c0039a0 0x7fcd6c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.444+0000 7fcd837fe640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00ba10 tx=0x7fcd6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a931e6cd746847 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.444+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c032630 con 0x7fcd6c004d00 2023-10-16T15:33:52.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.444+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c032790 con 0x7fcd6c004d00 2023-10-16T15:33:52.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.444+0000 7fcd81ffb640 1 -- 172.21.15.29:0/1795532657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c032b20 con 0x7fcd6c004d00 2023-10-16T15:33:52.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd83fff640 1 -- 172.21.15.29:0/1795532657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c004d00 msgr2=0x7fcd6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd83fff640 1 --2- 172.21.15.29:0/1795532657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00ba10 tx=0x7fcd6c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd837fe640 1 -- 172.21.15.29:0/1795532657 reap_dead start 2023-10-16T15:33:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd83fff640 1 -- 172.21.15.29:0/1795532657 shutdown_connections 2023-10-16T15:33:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd83fff640 1 -- 172.21.15.29:0/1795532657 >> 172.21.15.29:0/1795532657 conn(0x7fcd840ce6a0 msgr2=0x7fcd840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd83fff640 1 -- 172.21.15.29:0/1795532657 shutdown_connections 2023-10-16T15:33:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.445+0000 7fcd83fff640 1 -- 172.21.15.29:0/1795532657 wait complete. 2023-10-16T15:33:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.447+0000 7fcd83fff640 1 Processor -- start 2023-10-16T15:33:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.447+0000 7fcd83fff640 1 -- start start 2023-10-16T15:33:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd83fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c004d00 0x7fcd84144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 0x7fcd84156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd83fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd84156a10 0x7fcd840b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd83fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84155cf0 con 0x7fcd84154b70 2023-10-16T15:33:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd83fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84156650 con 0x7fcd6c004d00 2023-10-16T15:33:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd83fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84155070 con 0x7fcd84156a10 2023-10-16T15:33:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 0x7fcd84156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 0x7fcd84156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46436/0 (socket says 172.21.15.29:46436) 2023-10-16T15:33:52.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.448+0000 7fcd827fc640 1 -- 172.21.15.29:0/629166419 learned_addr learned my addr 172.21.15.29:0/629166419 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd827fc640 1 -- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd84156a10 msgr2=0x7fcd840b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd827fc640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd84156a10 0x7fcd840b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd827fc640 1 -- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c004d00 msgr2=0x7fcd84144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd827fc640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c004d00 0x7fcd84144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd827fc640 1 -- 172.21.15.29:0/629166419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84155350 con 0x7fcd84154b70 2023-10-16T15:33:52.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd827fc640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 0x7fcd84156110 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800e4d0 tx=0x7fcd7800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd6b7fe640 1 -- 172.21.15.29:0/629166419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd780053a0 con 0x7fcd84154b70 2023-10-16T15:33:52.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd6b7fe640 1 -- 172.21.15.29:0/629166419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd780035f0 con 0x7fcd84154b70 2023-10-16T15:33:52.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd6b7fe640 1 -- 172.21.15.29:0/629166419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd780039a0 con 0x7fcd84154b70 2023-10-16T15:33:52.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd84155630 con 0x7fcd84154b70 2023-10-16T15:33:52.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.449+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd84156ec0 con 0x7fcd84154b70 2023-10-16T15:33:52.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.452+0000 7fcd6b7fe640 1 -- 172.21.15.29:0/629166419 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcd78003b00 con 0x7fcd84154b70 2023-10-16T15:33:52.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.453+0000 7fcd6b7fe640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcd7006afc0 0x7fcd7006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.453+0000 7fcd6b7fe640 1 -- 172.21.15.29:0/629166419 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcd780895b0 con 0x7fcd84154b70 2023-10-16T15:33:52.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.453+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcd7006afc0 0x7fcd7006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.454+0000 7fcd8bfde740 1 Processor -- start 2023-10-16T15:33:52.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.454+0000 7fcd8bfde740 1 -- start start 2023-10-16T15:33:52.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb71595070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715955b0 0x55bb715c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb714b57a0 con 0x55bb715c2550 2023-10-16T15:33:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb71507740 con 0x55bb715955b0 2023-10-16T15:33:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb71504e30 con 0x55bb715c2170 2023-10-16T15:33:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcd7006afc0 0x7fcd7006d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcd74007c40 tx=0x7fcd74009040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46450/0 (socket says 172.21.15.29:46450) 2023-10-16T15:33:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd827fc640 1 -- 172.21.15.29:0/110702399 learned_addr learned my addr 172.21.15.29:0/110702399 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.455+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb71595070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd837fe640 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715955b0 0x55bb715c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd827fc640 1 -- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 msgr2=0x55bb71595070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd827fc640 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb71595070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd827fc640 1 -- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715955b0 msgr2=0x55bb715c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd827fc640 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715955b0 0x55bb715c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd827fc640 1 -- 172.21.15.29:0/110702399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb714c4910 con 0x55bb715c2550 2023-10-16T15:33:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd837fe640 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715955b0 0x55bb715c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:33:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd827fc640 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715c4f00 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fcd78058050 tx=0x7fcd78057fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cf62a12b6e4402 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd68ff9640 1 -- 172.21.15.29:0/110702399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7806bea0 con 0x55bb715c2550 2023-10-16T15:33:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd68ff9640 1 -- 172.21.15.29:0/110702399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd780580c0 con 0x55bb715c2550 2023-10-16T15:33:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.456+0000 7fcd68ff9640 1 -- 172.21.15.29:0/110702399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7805c210 con 0x55bb715c2550 2023-10-16T15:33:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 -- 172.21.15.29:0/110702399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 msgr2=0x55bb715c4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715c4f00 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fcd78058050 tx=0x7fcd78057fc0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 -- 172.21.15.29:0/110702399 shutdown_connections 2023-10-16T15:33:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715c4f00 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715955b0 0x55bb715c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/110702399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb71595070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 -- 172.21.15.29:0/110702399 >> 172.21.15.29:0/110702399 conn(0x55bb71597380 msgr2=0x55bb715dff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 -- 172.21.15.29:0/110702399 shutdown_connections 2023-10-16T15:33:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.457+0000 7fcd8bfde740 1 -- 172.21.15.29:0/110702399 wait complete. 2023-10-16T15:33:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.458+0000 7fcd8bfde740 1 -- start start 2023-10-16T15:33:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.458+0000 7fcd8bfde740 1 Processor -- start 2023-10-16T15:33:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb715c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb715dfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb71500720 con 0x55bb715c2550 2023-10-16T15:33:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb7149e970 con 0x55bb715df990 2023-10-16T15:33:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb7149e610 con 0x55bb715c2170 2023-10-16T15:33:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd8bfde740 10 client.0 _ll_register_callbacks cb 0x55bb714c09d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:33:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.459+0000 7fcd827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46462/0 (socket says 172.21.15.29:46462) 2023-10-16T15:33:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 -- 172.21.15.29:0/661650254 learned_addr learned my addr 172.21.15.29:0/661650254 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 -- 172.21.15.29:0/661650254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 msgr2=0x55bb715c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb715c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 -- 172.21.15.29:0/661650254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 msgr2=0x55bb715dfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb715dfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 -- 172.21.15.29:0/661650254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd84156ec0 con 0x55bb715c2550 2023-10-16T15:33:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd827fc640 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715f4da0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fcd780545c0 tx=0x7fcd7805c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530c5fad568acc70 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:33:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd61ffb640 1 -- 172.21.15.29:0/661650254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd78087070 con 0x55bb715c2550 2023-10-16T15:33:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.460+0000 7fcd61ffb640 1 -- 172.21.15.29:0/661650254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcd78060450 con 0x55bb715c2550 2023-10-16T15:33:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb71527160 con 0x55bb715c2550 2023-10-16T15:33:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd61ffb640 1 -- 172.21.15.29:0/661650254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd7806bea0 con 0x55bb715c2550 2023-10-16T15:33:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 10 client.9474 fetch_fsmap learned FSMap version 84 2023-10-16T15:33:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55bb714feda0 con 0x55bb715c2550 2023-10-16T15:33:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd61ffb640 1 -- 172.21.15.29:0/661650254 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bb71527160 con 0x55bb715c2550 2023-10-16T15:33:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd61ffb640 1 -- 172.21.15.29:0/661650254 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcd780813f0 con 0x55bb715c2550 2023-10-16T15:33:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 10 client.9474 fetch_fsmap finished waiting for FSMap version 84 2023-10-16T15:33:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 10 client.9474 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T15:33:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 20 client.9474 populate_metadata read hostname 'smithi029' 2023-10-16T15:33:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb715c63c0 0x55bb715c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 4 client.9474 mds_command: new command op to 9342 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:33:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.461+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 --> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb714d2cc0 con 0x55bb715c63c0 2023-10-16T15:33:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.462+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb715c63c0 0x55bb715c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.462+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb715c63c0 0x55bb715c67a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd9c75a41f70a2dc server_cookie=b772160ae844c98a in_seq=0 out_seq=0 2023-10-16T15:33:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.462+0000 7fcd61ffb640 10 client.9474 ms_handle_connect on v2:172.21.15.29:6832/3104953668 2023-10-16T15:33:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd61ffb640 1 -- 172.21.15.29:0/661650254 <== mds.0 v2:172.21.15.29:6832/3104953668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bb714d2cc0 con 0x55bb715c63c0 2023-10-16T15:33:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd61ffb640 10 client.9474 handle_command_reply: tid=0 2023-10-16T15:33:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 client.9474 shutdown 2023-10-16T15:33:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 msgr2=0x55bb715f4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715f4da0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fcd780545c0 tx=0x7fcd7805c000 comp rx=0 tx=0).stop 2023-10-16T15:33:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 shutdown_connections 2023-10-16T15:33:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb715c63c0 0x55bb715c67a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2550 0x55bb715f4da0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb715dfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/661650254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2170 0x55bb715c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd61ffb640 0 client.9474 ms_handle_reset on v2:172.21.15.29:6832/3104953668 2023-10-16T15:33:52.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.463+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 >> 172.21.15.29:0/661650254 conn(0x55bb715955b0 msgr2=0x55bb715e3410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.464+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 shutdown_connections 2023-10-16T15:33:52.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.464+0000 7fcd8bfde740 1 -- 172.21.15.29:0/661650254 wait complete. 2023-10-16T15:33:52.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.464+0000 7fcd8bfde740 20 client.9474 trim_cache size 0 max 16384 2023-10-16T15:33:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.481+0000 7fcd8bfde740 1 Processor -- start 2023-10-16T15:33:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.481+0000 7fcd8bfde740 1 -- start start 2023-10-16T15:33:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb715b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 0x55bb71596050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb71598c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb71264f20 con 0x55bb715c2170 2023-10-16T15:33:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb71599b30 con 0x55bb715df990 2023-10-16T15:33:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb715c8240 con 0x55bb715c2550 2023-10-16T15:33:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb715b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb715b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46472/0 (socket says 172.21.15.29:46472) 2023-10-16T15:33:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 -- 172.21.15.29:0/42983483 learned_addr learned my addr 172.21.15.29:0/42983483 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 -- 172.21.15.29:0/42983483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 msgr2=0x55bb71596050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 0x55bb71596050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 -- 172.21.15.29:0/42983483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 msgr2=0x55bb71598c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb71598c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.482+0000 7fcd82ffd640 1 -- 172.21.15.29:0/42983483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb7149db80 con 0x55bb715c2170 2023-10-16T15:33:52.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.483+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb715b8af0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fcd74005d80 tx=0x7fcd74010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aed5b42be5214d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.483+0000 7fcd697fa640 1 -- 172.21.15.29:0/42983483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd740079b0 con 0x55bb715c2170 2023-10-16T15:33:52.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.483+0000 7fcd697fa640 1 -- 172.21.15.29:0/42983483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd740223b0 con 0x55bb715c2170 2023-10-16T15:33:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.483+0000 7fcd697fa640 1 -- 172.21.15.29:0/42983483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74004b60 con 0x55bb715c2170 2023-10-16T15:33:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.483+0000 7fcd8bfde740 1 -- 172.21.15.29:0/42983483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 msgr2=0x55bb715b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.483+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb715b8af0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fcd74005d80 tx=0x7fcd74010000 comp rx=0 tx=0).stop 2023-10-16T15:33:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 -- 172.21.15.29:0/42983483 shutdown_connections 2023-10-16T15:33:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb71598c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 0x55bb71596050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/42983483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb715b8af0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 -- 172.21.15.29:0/42983483 >> 172.21.15.29:0/42983483 conn(0x55bb715955b0 msgr2=0x55bb715d8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 -- 172.21.15.29:0/42983483 shutdown_connections 2023-10-16T15:33:52.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 -- 172.21.15.29:0/42983483 wait complete. 2023-10-16T15:33:52.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.484+0000 7fcd8bfde740 1 -- start start 2023-10-16T15:33:52.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.485+0000 7fcd8bfde740 1 Processor -- start 2023-10-16T15:33:52.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb7164b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 0x55bb715f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb7159a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb71605940 con 0x55bb715c2170 2023-10-16T15:33:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb7164bc40 con 0x55bb715df990 2023-10-16T15:33:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb7159aca0 con 0x55bb715c2550 2023-10-16T15:33:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd8bfde740 10 client.0 _ll_register_callbacks cb 0x7fcd78057c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:33:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb7164b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb7164b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46482/0 (socket says 172.21.15.29:46482) 2023-10-16T15:33:52.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 -- 172.21.15.29:0/2072808151 learned_addr learned my addr 172.21.15.29:0/2072808151 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 -- 172.21.15.29:0/2072808151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 msgr2=0x55bb715f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 0x55bb715f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 -- 172.21.15.29:0/2072808151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 msgr2=0x55bb7159a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb7159a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.486+0000 7fcd82ffd640 1 -- 172.21.15.29:0/2072808151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7401c040 con 0x55bb715c2170 2023-10-16T15:33:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd82ffd640 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb7164b700 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009010 tx=0x7fcd740012c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bc8ff51c2eb1fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:33:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2072808151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74018030 con 0x55bb715c2170 2023-10-16T15:33:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2072808151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcd74022cb0 con 0x55bb715c2170 2023-10-16T15:33:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd780580c0 con 0x55bb715c2170 2023-10-16T15:33:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2072808151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd74022e70 con 0x55bb715c2170 2023-10-16T15:33:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd8bfde740 10 client.9480 fetch_fsmap learned FSMap version 84 2023-10-16T15:33:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55bb714c3170 con 0x55bb715c2170 2023-10-16T15:33:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.487+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2072808151 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd780580c0 con 0x55bb715c2170 2023-10-16T15:33:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2072808151 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcd74004b60 con 0x55bb715c2170 2023-10-16T15:33:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd8bfde740 10 client.9480 fetch_fsmap finished waiting for FSMap version 84 2023-10-16T15:33:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd8bfde740 10 client.9480 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T15:33:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd8bfde740 20 client.9480 populate_metadata read hostname 'smithi029' 2023-10-16T15:33:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb71598be0 0x55bb71598fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd8bfde740 4 client.9480 mds_command: new command op to 9342 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:33:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 --> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb714f9700 con 0x55bb71598be0 2023-10-16T15:33:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd827fc640 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb71598be0 0x55bb71598fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd827fc640 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb71598be0 0x55bb71598fc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f8560405ccf3621 server_cookie=15dc9416318aaa15 in_seq=0 out_seq=0 2023-10-16T15:33:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.488+0000 7fcd61ffb640 10 client.9480 ms_handle_connect on v2:172.21.15.29:6832/3104953668 2023-10-16T15:33:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2072808151 <== mds.0 v2:172.21.15.29:6832/3104953668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55bb714f9700 con 0x55bb71598be0 2023-10-16T15:33:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd61ffb640 10 client.9480 handle_command_reply: tid=0 2023-10-16T15:33:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 client.9480 shutdown 2023-10-16T15:33:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 msgr2=0x55bb7164b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb7164b700 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009010 tx=0x7fcd740012c0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 shutdown_connections 2023-10-16T15:33:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:6832/3104953668,v1:172.21.15.29:6833/3104953668] conn(0x55bb71598be0 0x55bb71598fc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb715df990 0x55bb7159a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb715c2550 0x55bb715f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 --2- 172.21.15.29:0/2072808151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb715c2170 0x55bb7164b700 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd61ffb640 0 client.9480 ms_handle_reset on v2:172.21.15.29:6832/3104953668 2023-10-16T15:33:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 >> 172.21.15.29:0/2072808151 conn(0x55bb715955b0 msgr2=0x55bb715d8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 shutdown_connections 2023-10-16T15:33:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 1 -- 172.21.15.29:0/2072808151 wait complete. 2023-10-16T15:33:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.489+0000 7fcd8bfde740 20 client.9480 trim_cache size 0 max 16384 2023-10-16T15:33:52.503 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:33:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:33:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:33:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 9342, 2023-10-16T15:33:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:33:52.504 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:33:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T15:33:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8908426540000001, 2023-10-16T15:33:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 84, 2023-10-16T15:33:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 179, 2023-10-16T15:33:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 179, 2023-10-16T15:33:52.505 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.693941966000001 2023-10-16T15:33:52.506 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:33:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcd7006afc0 msgr2=0x7fcd7006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcd7006afc0 0x7fcd7006d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcd74007c40 tx=0x7fcd74009040 comp rx=0 tx=0).stop 2023-10-16T15:33:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 msgr2=0x7fcd84156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 0x7fcd84156110 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800e4d0 tx=0x7fcd7800e990 comp rx=0 tx=0).stop 2023-10-16T15:33:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 shutdown_connections 2023-10-16T15:33:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7fcd7006afc0 0x7fcd7006d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd84154b70 0x7fcd84156110 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcd84156a10 0x7fcd840b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 --2- 172.21.15.29:0/629166419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd6c004d00 0x7fcd84144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 >> 172.21.15.29:0/629166419 conn(0x7fcd840ce6a0 msgr2=0x7fcd8406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 shutdown_connections 2023-10-16T15:33:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.490+0000 7fcd83fff640 1 -- 172.21.15.29:0/629166419 wait complete. 2023-10-16T15:33:52.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T15:33:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.726+0000 7f4c689ef640 1 Processor -- start 2023-10-16T15:33:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.726+0000 7f4c689ef640 1 -- start start 2023-10-16T15:33:52.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c689ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c641599c0 0x7f4c640ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c689ef640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c640cda90 0x7f4c6406f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c689ef640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6406bab0 con 0x7f4c640ca6f0 2023-10-16T15:33:52.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c689ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6406f300 con 0x7f4c641599c0 2023-10-16T15:33:52.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c689ef640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6406bd90 con 0x7f4c640cda90 2023-10-16T15:33:52.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c62d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c641599c0 0x7f4c640ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c62d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c641599c0 0x7f4c640ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46484/0 (socket says 172.21.15.29:46484) 2023-10-16T15:33:52.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.728+0000 7f4c62d9d640 1 -- 172.21.15.29:0/2860692659 learned_addr learned my addr 172.21.15.29:0/2860692659 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.729+0000 7f4c6359e640 1 --2- 172.21.15.29:0/2860692659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c640cda90 0x7f4c6406f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.729+0000 7f4c62d9d640 1 -- 172.21.15.29:0/2860692659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c640cda90 msgr2=0x7f4c6406f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:52.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.729+0000 7f4c62d9d640 1 --2- 172.21.15.29:0/2860692659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c640cda90 0x7f4c6406f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.729+0000 7f4c62d9d640 1 -- 172.21.15.29:0/2860692659 >> v1:172.21.15.170:6789/0 conn(0x7f4c640ca6f0 legacy=0x7f4c640cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.729+0000 7f4c62d9d640 1 -- 172.21.15.29:0/2860692659 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6406c070 con 0x7f4c641599c0 2023-10-16T15:33:52.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.730+0000 7f4c62d9d640 1 --2- 172.21.15.29:0/2860692659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c641599c0 0x7f4c640ca1b0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009870 tx=0x7f4c5402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75160b35a01d5e4f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.730+0000 7f4c61d9b640 1 -- 172.21.15.29:0/2860692659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54031640 con 0x7f4c641599c0 2023-10-16T15:33:52.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.730+0000 7f4c61d9b640 1 -- 172.21.15.29:0/2860692659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c540317a0 con 0x7f4c641599c0 2023-10-16T15:33:52.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c61d9b640 1 -- 172.21.15.29:0/2860692659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54031b30 con 0x7f4c641599c0 2023-10-16T15:33:52.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 -- 172.21.15.29:0/2860692659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c641599c0 msgr2=0x7f4c640ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2860692659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c641599c0 0x7f4c640ca1b0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009870 tx=0x7f4c5402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 -- 172.21.15.29:0/2860692659 shutdown_connections 2023-10-16T15:33:52.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2860692659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c640cda90 0x7f4c6406f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2860692659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c641599c0 0x7f4c640ca1b0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 -- 172.21.15.29:0/2860692659 >> 172.21.15.29:0/2860692659 conn(0x7f4c640b0a90 msgr2=0x7f4c640ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 -- 172.21.15.29:0/2860692659 shutdown_connections 2023-10-16T15:33:52.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.731+0000 7f4c689ef640 1 -- 172.21.15.29:0/2860692659 wait complete. 2023-10-16T15:33:52.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.733+0000 7f4c689ef640 1 Processor -- start 2023-10-16T15:33:52.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.733+0000 7f4c689ef640 1 -- start start 2023-10-16T15:33:52.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c689ef640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c640ca6f0 0x7f4c640b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c689ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 0x7f4c6415b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c689ef640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c641599c0 0x7f4c641f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c689ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b37a0 con 0x7f4c640cda90 2023-10-16T15:33:52.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c689ef640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b34c0 con 0x7f4c640ca6f0 2023-10-16T15:33:52.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c689ef640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c641f5830 con 0x7f4c641599c0 2023-10-16T15:33:52.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c6259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 0x7f4c6415b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c6259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 0x7f4c6415b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46490/0 (socket says 172.21.15.29:46490) 2023-10-16T15:33:52.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.734+0000 7f4c6259c640 1 -- 172.21.15.29:0/2180606578 learned_addr learned my addr 172.21.15.29:0/2180606578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:52.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6359e640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c641599c0 0x7f4c641f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c62d9d640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c640ca6f0 0x7f4c640b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6259c640 1 -- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c641599c0 msgr2=0x7f4c641f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6259c640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c641599c0 0x7f4c641f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6259c640 1 -- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c640ca6f0 msgr2=0x7f4c640b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6259c640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c640ca6f0 0x7f4c640b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6259c640 1 -- 172.21.15.29:0/2180606578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540094d0 con 0x7f4c640cda90 2023-10-16T15:33:52.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.735+0000 7f4c6259c640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 0x7f4c6415b4b0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800c4d0 tx=0x7f4c5800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.736+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c580052d0 con 0x7f4c640cda90 2023-10-16T15:33:52.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.736+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58005430 con 0x7f4c640cda90 2023-10-16T15:33:52.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.736+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58012850 con 0x7f4c640cda90 2023-10-16T15:33:52.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.736+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c641f5b10 con 0x7f4c640cda90 2023-10-16T15:33:52.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.736+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c64150830 con 0x7f4c640cda90 2023-10-16T15:33:52.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.737+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c641f6030 con 0x7f4c640cda90 2023-10-16T15:33:52.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.738+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4c580129b0 con 0x7f4c640cda90 2023-10-16T15:33:52.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.739+0000 7f4c4b7fe640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4c5006af90 0x7f4c5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:52.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.739+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4c58089250 con 0x7f4c640cda90 2023-10-16T15:33:52.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.740+0000 7f4c62d9d640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4c5006af90 0x7f4c5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:52.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.741+0000 7f4c62d9d640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4c5006af90 0x7f4c5006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4c54036000 tx=0x7f4c5403c040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:52.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.744+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4c641f6030 con 0x7f4c640cda90 2023-10-16T15:33:52.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.983+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4c641f6030 con 0x7f4c640cda90 2023-10-16T15:33:52.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.984+0000 7f4c4b7fe640 1 -- 172.21.15.29:0/2180606578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v84) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4c641f6030 con 0x7f4c640cda90 2023-10-16T15:33:52.987 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T15:33:52.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4c5006af90 msgr2=0x7f4c5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4c5006af90 0x7f4c5006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4c54036000 tx=0x7f4c5403c040 comp rx=0 tx=0).stop 2023-10-16T15:33:52.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 msgr2=0x7f4c6415b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:52.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 0x7f4c6415b4b0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800c4d0 tx=0x7f4c5800c990 comp rx=0 tx=0).stop 2023-10-16T15:33:52.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 shutdown_connections 2023-10-16T15:33:52.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f4c5006af90 0x7f4c5006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c640cda90 0x7f4c6415b4b0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c641599c0 0x7f4c641f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 --2- 172.21.15.29:0/2180606578 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c640ca6f0 0x7f4c640b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:52.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 >> 172.21.15.29:0/2180606578 conn(0x7f4c640b0a90 msgr2=0x7f4c6415c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:52.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 shutdown_connections 2023-10-16T15:33:52.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:52.987+0000 7f4c689ef640 1 -- 172.21.15.29:0/2180606578 wait complete. 2023-10-16T15:33:53.008 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T15:33:53.009 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T15:33:53.009 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T15:33:53.010 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T15:33:53.010 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T15:33:53.010 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T15:33:53.011 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T15:33:53.011 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T15:33:53.011 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:33:53.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.012 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T15:33:53.067 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T15:33:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T15:33:53.067 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T15:33:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T15:33:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.068 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:33:53.068 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:33:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T15:33:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:33:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:33:53.069 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T15:33:53.069 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T15:33:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T15:33:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T15:33:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T15:33:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 37692sec preferred_lft 37692sec 2023-10-16T15:33:53.070 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T15:33:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:33:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T15:33:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T15:33:53.071 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T15:33:53.071 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:33:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T15:33:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T15:33:53.072 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.072 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:33:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:33:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T15:33:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.073 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:33:53.073 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:33:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T15:33:53.074 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:33:53.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.075 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T15:33:53.173 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T15:33:53.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.178 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T15:33:53.237 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T15:33:53.237 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T15:33:53.238 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T15:33:53.238 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T15:33:53.238 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T15:33:53.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.239 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T15:33:53.239 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T15:33:53.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T15:33:53.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T15:33:53.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T15:33:53.241 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T15:33:53.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.445 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:33:53.501 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:33:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:33:53.503 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:33:53.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T15:33:53.611 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:33:53.612 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T15:33:53.671 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:33:53.672 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:33:53.672 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:33:53.725 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T15:33:53.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:33:53.874 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:33:53.874 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:33:53.875 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:33:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:33:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:33:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:33:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:33:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:33:53.925 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:33:53.925 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:33:53.926 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:33:53.926 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:33:53.926 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:33:53.926 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T15:33:53.927 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T15:33:53.927 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T15:33:53.930 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T15:33:53.930 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T15:33:53.931 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T15:33:53.931 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T15:33:53.931 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T15:33:53.934 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T15:33:53.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:53.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:33:54.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:54.029 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T15:33:54.098 INFO:teuthology.orchestra.run.smithi029.stdout:client6995 2023-10-16T15:33:54.099 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:33:54.099 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:33:54.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T15:33:54.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.383+0000 7ff6af13c640 1 Processor -- start 2023-10-16T15:33:54.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.383+0000 7ff6af13c640 1 -- start start 2023-10-16T15:33:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6af13c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff6a815c9f0 0x7ff6a80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6af13c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6a815dde0 0x7ff6a80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6af13c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a8160a90 con 0x7ff6a80c9230 2023-10-16T15:33:54.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6af13c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a80cdec0 con 0x7ff6a815dde0 2023-10-16T15:33:54.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6af13c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a80cd900 con 0x7ff6a815c9f0 2023-10-16T15:33:54.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6ae93b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6a815dde0 0x7ff6a80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6ad939640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6a80c9230 0x7ff6a80709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36260/0 (socket says 172.21.15.29:36260) 2023-10-16T15:33:54.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6ad939640 1 -- 172.21.15.29:0/936925594 learned_addr learned my addr 172.21.15.29:0/936925594 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:54.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.385+0000 7ff6ae13a640 1 --2- 172.21.15.29:0/936925594 >> v2:172.21.15.170:3300/0 conn(0x7ff6a815c9f0 0x7ff6a80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.386+0000 7ff6ae93b640 1 -- 172.21.15.29:0/936925594 >> v2:172.21.15.170:3300/0 conn(0x7ff6a815c9f0 msgr2=0x7ff6a80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:54.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.386+0000 7ff6ae93b640 1 --2- 172.21.15.29:0/936925594 >> v2:172.21.15.170:3300/0 conn(0x7ff6a815c9f0 0x7ff6a80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.386+0000 7ff6ae93b640 1 -- 172.21.15.29:0/936925594 >> v1:172.21.15.29:6789/0 conn(0x7ff6a80c9230 legacy=0x7ff6a80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.386+0000 7ff6ae93b640 1 -- 172.21.15.29:0/936925594 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a80cdbe0 con 0x7ff6a815dde0 2023-10-16T15:33:54.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.387+0000 7ff6ae93b640 1 --2- 172.21.15.29:0/936925594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80ac890 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80ad4e0 tx=0x7ff6a002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef79279a3862d47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:54.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.387+0000 7ff6ad138640 1 -- 172.21.15.29:0/936925594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a002cc70 con 0x7ff6a815dde0 2023-10-16T15:33:54.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.387+0000 7ff6ad138640 1 -- 172.21.15.29:0/936925594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a002cdd0 con 0x7ff6a815dde0 2023-10-16T15:33:54.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.387+0000 7ff6ad138640 1 -- 172.21.15.29:0/936925594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0034660 con 0x7ff6a815dde0 2023-10-16T15:33:54.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 -- 172.21.15.29:0/936925594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 msgr2=0x7ff6a80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 --2- 172.21.15.29:0/936925594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80ac890 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80ad4e0 tx=0x7ff6a002bec0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 -- 172.21.15.29:0/936925594 shutdown_connections 2023-10-16T15:33:54.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 --2- 172.21.15.29:0/936925594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80ac890 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 --2- 172.21.15.29:0/936925594 >> v2:172.21.15.170:3300/0 conn(0x7ff6a815c9f0 0x7ff6a80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 -- 172.21.15.29:0/936925594 >> 172.21.15.29:0/936925594 conn(0x7ff6a80ce6a0 msgr2=0x7ff6a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:54.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 -- 172.21.15.29:0/936925594 shutdown_connections 2023-10-16T15:33:54.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.388+0000 7ff6af13c640 1 -- 172.21.15.29:0/936925594 wait complete. 2023-10-16T15:33:54.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.390+0000 7ff6af13c640 1 Processor -- start 2023-10-16T15:33:54.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.390+0000 7ff6af13c640 1 -- start start 2023-10-16T15:33:54.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.391+0000 7ff6af13c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6a80c9230 0x7ff6a80c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.391+0000 7ff6af13c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6a815c9f0 0x7ff6a80c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.391+0000 7ff6af13c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.391+0000 7ff6af13c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a80b1080 con 0x7ff6a815dde0 2023-10-16T15:33:54.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.391+0000 7ff6af13c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a80b0da0 con 0x7ff6a815c9f0 2023-10-16T15:33:54.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.391+0000 7ff6af13c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a81f3600 con 0x7ff6a80c9230 2023-10-16T15:33:54.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48662/0 (socket says 172.21.15.29:48662) 2023-10-16T15:33:54.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 -- 172.21.15.29:0/3899684810 learned_addr learned my addr 172.21.15.29:0/3899684810 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:54.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ad939640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6a815c9f0 0x7ff6a80c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae13a640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6a80c9230 0x7ff6a80c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 -- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6a80c9230 msgr2=0x7ff6a80c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6a80c9230 0x7ff6a80c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 -- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6a815c9f0 msgr2=0x7ff6a80c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6a815c9f0 0x7ff6a80c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 -- 172.21.15.29:0/3899684810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a0009a70 con 0x7ff6a815dde0 2023-10-16T15:33:54.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.392+0000 7ff6ae93b640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80aff70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009e80 tx=0x7ff6a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:54.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.393+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0040650 con 0x7ff6a815dde0 2023-10-16T15:33:54.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.393+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a00407b0 con 0x7ff6a815dde0 2023-10-16T15:33:54.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.393+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0040b60 con 0x7ff6a815dde0 2023-10-16T15:33:54.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.393+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a81f38e0 con 0x7ff6a815dde0 2023-10-16T15:33:54.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.393+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a80b04b0 con 0x7ff6a815dde0 2023-10-16T15:33:54.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.394+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a81f3e00 con 0x7ff6a815dde0 2023-10-16T15:33:54.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.395+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6a0040cc0 con 0x7ff6a815dde0 2023-10-16T15:33:54.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.396+0000 7ff69effd640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff68c06af90 0x7ff68c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.396+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff6a00ac510 con 0x7ff6a815dde0 2023-10-16T15:33:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.397+0000 7ff6ae13a640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff68c06af90 0x7ff68c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.398+0000 7ff6ae13a640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff68c06af90 0x7ff68c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff6a81f5860 tx=0x7ff6a4008040 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.401+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6a81f3e00 con 0x7ff6a815dde0 2023-10-16T15:33:54.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.616+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 --> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6a80704c0 con 0x7ff68c06af90 2023-10-16T15:33:54.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.621+0000 7ff69effd640 1 -- 172.21.15.29:0/3899684810 <== mgr.7456 v2:172.21.15.170:6832/1981883088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff6a4009820 con 0x7ff68c06af90 2023-10-16T15:33:54.623 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:33:54.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:33:54.623 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T15:33:54.624 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:33:54.624 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:33:54.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff68c06af90 msgr2=0x7ff68c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff68c06af90 0x7ff68c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff6a81f5860 tx=0x7ff6a4008040 comp rx=0 tx=0).stop 2023-10-16T15:33:54.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 msgr2=0x7ff6a80aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80aff70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009e80 tx=0x7ff6a0004a40 comp rx=0 tx=0).stop 2023-10-16T15:33:54.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 shutdown_connections 2023-10-16T15:33:54.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7ff68c06af90 0x7ff68c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6a815dde0 0x7ff6a80aff70 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6a815c9f0 0x7ff6a80c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 --2- 172.21.15.29:0/3899684810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6a80c9230 0x7ff6a80c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 >> 172.21.15.29:0/3899684810 conn(0x7ff6a80ce6a0 msgr2=0x7ff6a80cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:54.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 shutdown_connections 2023-10-16T15:33:54.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.623+0000 7ff6af13c640 1 -- 172.21.15.29:0/3899684810 wait complete. 2023-10-16T15:33:54.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T15:33:54.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.881+0000 7f3148eea640 1 Processor -- start 2023-10-16T15:33:54.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.881+0000 7f3148eea640 1 -- start start 2023-10-16T15:33:54.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f3148eea640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 0x7f31440ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f3148eea640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31440ca560 con 0x7f31440c9060 2023-10-16T15:33:54.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f3148eea640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31440c9950 con 0x7f314406fb50 2023-10-16T15:33:54.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f3148eea640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31440c9c30 con 0x7f314406c9e0 2023-10-16T15:33:54.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f31437fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31440c9060 0x7f314406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36262/0 (socket says 172.21.15.29:36262) 2023-10-16T15:33:54.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f31437fe640 1 -- 172.21.15.29:0/516435790 learned_addr learned my addr 172.21.15.29:0/516435790 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:54.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.883+0000 7f3143fff640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 0x7f31440ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856217685 0 0) 0x7f31440ca560 con 0x7f31440c9060 2023-10-16T15:33:54.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3130003520 con 0x7f31440c9060 2023-10-16T15:33:54.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284771011 0 0) 0x7f3130003520 con 0x7f31440c9060 2023-10-16T15:33:54.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 msgr2=0x7f31440ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:54.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 0x7f31440ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 >> v1:172.21.15.170:6789/0 conn(0x7f314406fb50 legacy=0x7f31440cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31440c9f10 con 0x7f31440c9060 2023-10-16T15:33:54.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f31340019f0 con 0x7f31440c9060 2023-10-16T15:33:54.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 >> v1:172.21.15.29:6789/0 conn(0x7f31440c9060 legacy=0x7f314406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31300039a0 0x7f3130003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300047c0 0x7f3130004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 0x7f3130005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31440c9c30 con 0x7f31300039a0 2023-10-16T15:33:54.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313000ba60 con 0x7f31300047c0 2023-10-16T15:33:54.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313000bc20 con 0x7f314406c9e0 2023-10-16T15:33:54.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.884+0000 7f31437fe640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31300039a0 0x7f3130003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f313affd640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300047c0 0x7f3130004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f3143fff640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 0x7f3130005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f31437fe640 1 -- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 msgr2=0x7f3130005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f31437fe640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f314406c9e0 0x7f3130005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f31437fe640 1 -- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300047c0 msgr2=0x7f3130004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:54.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f31437fe640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300047c0 0x7f3130004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.885+0000 7f31437fe640 1 -- 172.21.15.29:0/516435790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130003520 con 0x7f31300039a0 2023-10-16T15:33:54.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.886+0000 7f31437fe640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31300039a0 0x7f3130003dc0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f3134002230 tx=0x7f31300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacee820009ad9c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:54.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.886+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134034630 con 0x7f31300039a0 2023-10-16T15:33:54.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.886+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313402f070 con 0x7f31300039a0 2023-10-16T15:33:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.886+0000 7f3142ffd640 1 -- 172.21.15.29:0/516435790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134034a10 con 0x7f31300039a0 2023-10-16T15:33:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3148eea640 1 -- 172.21.15.29:0/516435790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31300039a0 msgr2=0x7f3130003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3148eea640 1 --2- 172.21.15.29:0/516435790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31300039a0 0x7f3130003dc0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f3134002230 tx=0x7f31300057f0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3143fff640 1 -- 172.21.15.29:0/516435790 reap_dead start 2023-10-16T15:33:54.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3148eea640 1 -- 172.21.15.29:0/516435790 shutdown_connections 2023-10-16T15:33:54.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3148eea640 1 -- 172.21.15.29:0/516435790 >> 172.21.15.29:0/516435790 conn(0x7f31440ce6a0 msgr2=0x7f31440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:54.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3148eea640 1 -- 172.21.15.29:0/516435790 shutdown_connections 2023-10-16T15:33:54.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.887+0000 7f3148eea640 1 -- 172.21.15.29:0/516435790 wait complete. 2023-10-16T15:33:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.889+0000 7f3148eea640 1 Processor -- start 2023-10-16T15:33:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.889+0000 7f3148eea640 1 -- start start 2023-10-16T15:33:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f3148eea640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300039a0 0x7f3144156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f3148eea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 0x7f31440c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f3148eea640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31440c47a0 0x7f31440aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f3148eea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31440b3ab0 con 0x7f31440b3ed0 2023-10-16T15:33:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f3148eea640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31440c4c20 con 0x7f31300039a0 2023-10-16T15:33:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f3148eea640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3144157300 con 0x7f31440c47a0 2023-10-16T15:33:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.890+0000 7f313affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 0x7f31440c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 0x7f31440c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48686/0 (socket says 172.21.15.29:48686) 2023-10-16T15:33:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 -- 172.21.15.29:0/3737461649 learned_addr learned my addr 172.21.15.29:0/3737461649 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f31437fe640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300039a0 0x7f3144156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f3143fff640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31440c47a0 0x7f31440aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 -- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31440c47a0 msgr2=0x7f31440aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31440c47a0 0x7f31440aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 -- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300039a0 msgr2=0x7f3144156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300039a0 0x7f3144156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 -- 172.21.15.29:0/3737461649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31440b2e30 con 0x7f31440b3ed0 2023-10-16T15:33:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.891+0000 7f313affd640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 0x7f31440c4260 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f312c00e500 tx=0x7f312c00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.892+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c0053a0 con 0x7f31440b3ed0 2023-10-16T15:33:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.892+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c0035f0 con 0x7f31440b3ed0 2023-10-16T15:33:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.892+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31440b3110 con 0x7f31440b3ed0 2023-10-16T15:33:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.892+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c003980 con 0x7f31440b3ed0 2023-10-16T15:33:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.892+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31440b36d0 con 0x7f31440b3ed0 2023-10-16T15:33:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.893+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314404e9b0 con 0x7f31440b3ed0 2023-10-16T15:33:54.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.894+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f312c003ae0 con 0x7f31440b3ed0 2023-10-16T15:33:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.895+0000 7f3140ff9640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f312406af90 0x7f312406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.895+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f312c0894f0 con 0x7f31440b3ed0 2023-10-16T15:33:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.895+0000 7f31437fe640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f312406af90 0x7f312406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:54.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.897+0000 7f31437fe640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f312406af90 0x7f312406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3134034000 tx=0x7f313402f000 comp rx=0 tx=0).ready entity=mgr.7456 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:54.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:54.899+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f312c054480 con 0x7f31440b3ed0 2023-10-16T15:33:55.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.145+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f31440b4620 con 0x7f31440b3ed0 2023-10-16T15:33:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.773+0000 7f3140ff9640 1 -- 172.21.15.29:0/3737461649 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v90) v1 ==== 92+0+0 (secure 0 0 0) 0x7f312c057aa0 con 0x7f31440b3ed0 2023-10-16T15:33:55.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f312406af90 msgr2=0x7f312406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:55.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f312406af90 0x7f312406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3134034000 tx=0x7f313402f000 comp rx=0 tx=0).stop 2023-10-16T15:33:55.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 msgr2=0x7f31440c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:55.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 0x7f31440c4260 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f312c00e500 tx=0x7f312c00e9c0 comp rx=0 tx=0).stop 2023-10-16T15:33:55.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 shutdown_connections 2023-10-16T15:33:55.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f312406af90 0x7f312406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:55.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31440b3ed0 0x7f31440c4260 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:55.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31440c47a0 0x7f31440aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:55.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 --2- 172.21.15.29:0/3737461649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31300039a0 0x7f3144156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:55.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 >> 172.21.15.29:0/3737461649 conn(0x7f31440ce6a0 msgr2=0x7f314406b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:55.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.777+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 shutdown_connections 2023-10-16T15:33:55.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:55.778+0000 7f3148eea640 1 -- 172.21.15.29:0/3737461649 wait complete. 2023-10-16T15:33:55.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T15:33:56.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.015+0000 7f61ff59e640 1 Processor -- start 2023-10-16T15:33:56.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.015+0000 7f61ff59e640 1 -- start start 2023-10-16T15:33:56.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f620015cd70 0x7f62000d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61ff59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62000cded0 0x7f62000b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61ff59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620015d150 con 0x7f62000b8480 2023-10-16T15:33:56.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61ff59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62000d1180 con 0x7f62000cded0 2023-10-16T15:33:56.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61ff59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62000d1460 con 0x7f620015cd70 2023-10-16T15:33:56.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61fed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62000cded0 0x7f62000b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f62000b8480 0x7f62000cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36270/0 (socket says 172.21.15.29:36270) 2023-10-16T15:33:56.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.017+0000 7f61effff640 1 -- 172.21.15.29:0/3665153168 learned_addr learned my addr 172.21.15.29:0/3665153168 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:56.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.018+0000 7f61fe59c640 1 --2- 172.21.15.29:0/3665153168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f620015cd70 0x7f62000d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.018+0000 7f61fed9d640 1 -- 172.21.15.29:0/3665153168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f620015cd70 msgr2=0x7f62000d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:56.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.018+0000 7f61fed9d640 1 --2- 172.21.15.29:0/3665153168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f620015cd70 0x7f62000d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.018+0000 7f61fed9d640 1 -- 172.21.15.29:0/3665153168 >> v1:172.21.15.29:6789/0 conn(0x7f62000b8480 legacy=0x7f62000cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.018+0000 7f61fed9d640 1 -- 172.21.15.29:0/3665153168 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62000d1740 con 0x7f62000cded0 2023-10-16T15:33:56.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.018+0000 7f61fdd9b640 1 -- 172.21.15.29:0/3665153168 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902379557 0 0) 0x7f620015d150 con 0x7f62000b8480 2023-10-16T15:33:56.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.019+0000 7f61fed9d640 1 --2- 172.21.15.29:0/3665153168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000cded0 0x7f62000b5c90 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f61f80097a0 tx=0x7f61f802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63b42d9805a6e14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:56.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.019+0000 7f61fdd9b640 1 -- 172.21.15.29:0/3665153168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f802d040 con 0x7f62000cded0 2023-10-16T15:33:56.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.019+0000 7f61fdd9b640 1 -- 172.21.15.29:0/3665153168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8031810 con 0x7f62000cded0 2023-10-16T15:33:56.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.019+0000 7f61fdd9b640 1 -- 172.21.15.29:0/3665153168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8031ba0 con 0x7f62000cded0 2023-10-16T15:33:56.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 -- 172.21.15.29:0/3665153168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000cded0 msgr2=0x7f62000b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 --2- 172.21.15.29:0/3665153168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000cded0 0x7f62000b5c90 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f61f80097a0 tx=0x7f61f802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 -- 172.21.15.29:0/3665153168 shutdown_connections 2023-10-16T15:33:56.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 --2- 172.21.15.29:0/3665153168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000cded0 0x7f62000b5c90 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 --2- 172.21.15.29:0/3665153168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f620015cd70 0x7f62000d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 -- 172.21.15.29:0/3665153168 >> 172.21.15.29:0/3665153168 conn(0x7f62000cf570 msgr2=0x7f62000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:56.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 -- 172.21.15.29:0/3665153168 shutdown_connections 2023-10-16T15:33:56.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.020+0000 7f61ff59e640 1 -- 172.21.15.29:0/3665153168 wait complete. 2023-10-16T15:33:56.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.022+0000 7f61ff59e640 1 Processor -- start 2023-10-16T15:33:56.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.022+0000 7f61ff59e640 1 -- start start 2023-10-16T15:33:56.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61ff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 0x7f62000b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61ff59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62000cded0 0x7f620015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61ff59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f620015cd70 0x7f620015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61ff59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620015ec40 con 0x7f62000b8480 2023-10-16T15:33:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61ff59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620015ddc0 con 0x7f620015cd70 2023-10-16T15:33:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61ff59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620015e030 con 0x7f62000cded0 2023-10-16T15:33:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61fe59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 0x7f62000b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61fe59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 0x7f62000b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48736/0 (socket says 172.21.15.29:48736) 2023-10-16T15:33:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.023+0000 7f61fe59c640 1 -- 172.21.15.29:0/4009684436 learned_addr learned my addr 172.21.15.29:0/4009684436 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:56.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fed9d640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f620015cd70 0x7f620015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61effff640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62000cded0 0x7f620015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fe59c640 1 -- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62000cded0 msgr2=0x7f620015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fe59c640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62000cded0 0x7f620015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fe59c640 1 -- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f620015cd70 msgr2=0x7f620015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fe59c640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f620015cd70 0x7f620015d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fe59c640 1 -- 172.21.15.29:0/4009684436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8009400 con 0x7f62000b8480 2023-10-16T15:33:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.024+0000 7f61fe59c640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 0x7f62000b5ec0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f61f0007e70 tx=0x7f61f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.025+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0014620 con 0x7f62000b8480 2023-10-16T15:33:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.025+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0014780 con 0x7f62000b8480 2023-10-16T15:33:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.025+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0014b30 con 0x7f62000b8480 2023-10-16T15:33:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.025+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620015e310 con 0x7f62000b8480 2023-10-16T15:33:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.025+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620015e8d0 con 0x7f62000b8480 2023-10-16T15:33:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.026+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620004e9b0 con 0x7f62000b8480 2023-10-16T15:33:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.027+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f61f0014c90 con 0x7f62000b8480 2023-10-16T15:33:56.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.029+0000 7f61eeffd640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 0x7f61e006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.029+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f61f008a200 con 0x7f62000b8480 2023-10-16T15:33:56.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.029+0000 7f61effff640 1 -- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 msgr2=0x7f61e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:56.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.029+0000 7f61effff640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 0x7f61e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:56.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.033+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f61f00550e0 con 0x7f62000b8480 2023-10-16T15:33:56.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.230+0000 7f61effff640 1 -- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 msgr2=0x7f61e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.230+0000 7f61effff640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 0x7f61e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.231+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f62000c90d0 con 0x7f62000b8480 2023-10-16T15:33:56.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.236+0000 7f61eeffd640 1 -- 172.21.15.29:0/4009684436 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f61f0058700 con 0x7f62000b8480 2023-10-16T15:33:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 msgr2=0x7f61e006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 0x7f61e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 msgr2=0x7f62000b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 0x7f62000b5ec0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f61f0007e70 tx=0x7f61f000e4b0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 shutdown_connections 2023-10-16T15:33:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f61e006afe0 0x7f61e006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f620015cd70 0x7f620015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62000cded0 0x7f620015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 --2- 172.21.15.29:0/4009684436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62000b8480 0x7f62000b5ec0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 >> 172.21.15.29:0/4009684436 conn(0x7f62000cf570 msgr2=0x7f62000b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 shutdown_connections 2023-10-16T15:33:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.239+0000 7f61ff59e640 1 -- 172.21.15.29:0/4009684436 wait complete. 2023-10-16T15:33:56.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T15:33:56.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:56.287+0000 7f932b213040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:33:56.342 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:56.340+0000 7f932b213040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:33:56.342 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:56.340+0000 7fa9facb6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:33:56.403 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:56.400+0000 7fa9facb6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:33:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.466+0000 7f32aacb4640 1 Processor -- start 2023-10-16T15:33:56.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.467+0000 7f32aacb4640 1 -- start start 2023-10-16T15:33:56.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.468+0000 7f32aacb4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f32a40cf710 0x7f32a40b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.468+0000 7f32aacb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40cfaf0 0x7f32a40b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.468+0000 7f32aacb4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40d3980 con 0x7f32a40b38e0 2023-10-16T15:33:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.468+0000 7f32aacb4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a40d3c60 con 0x7f32a40cf710 2023-10-16T15:33:56.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.468+0000 7f32aacb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40ce630 con 0x7f32a40cfaf0 2023-10-16T15:33:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.469+0000 7f32a8a50640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f32a40cf710 0x7f32a40b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.469+0000 7f32a9251640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32a40b38e0 0x7f32a40d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36282/0 (socket says 172.21.15.29:36282) 2023-10-16T15:33:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.469+0000 7f32a9251640 1 -- 172.21.15.29:0/4016613362 learned_addr learned my addr 172.21.15.29:0/4016613362 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.469+0000 7f329bfff640 1 --2- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40cfaf0 0x7f32a40b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.469+0000 7f329b7fe640 1 -- 172.21.15.29:0/4016613362 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034448819 0 0) 0x7f32a40d3980 con 0x7f32a40b38e0 2023-10-16T15:33:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.469+0000 7f329b7fe640 1 -- 172.21.15.29:0/4016613362 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c003520 con 0x7f32a40b38e0 2023-10-16T15:33:56.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.470+0000 7f32a8a50640 1 -- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40cfaf0 msgr2=0x7f32a40b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.470+0000 7f32a8a50640 1 --2- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40cfaf0 0x7f32a40b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.470+0000 7f32a8a50640 1 -- 172.21.15.29:0/4016613362 >> v1:172.21.15.29:6789/0 conn(0x7f32a40b38e0 legacy=0x7f32a40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.470+0000 7f32a8a50640 1 -- 172.21.15.29:0/4016613362 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a40ce910 con 0x7f32a40cf710 2023-10-16T15:33:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.471+0000 7f32a8a50640 1 --2- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40b2d10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009de0 tx=0x7f32a002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e70cbdddcedaf9a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:56.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.471+0000 7f329b7fe640 1 -- 172.21.15.29:0/4016613362 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a002cc00 con 0x7f32a40cf710 2023-10-16T15:33:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.471+0000 7f329b7fe640 1 -- 172.21.15.29:0/4016613362 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a002cd60 con 0x7f32a40cf710 2023-10-16T15:33:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.471+0000 7f329b7fe640 1 -- 172.21.15.29:0/4016613362 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0034660 con 0x7f32a40cf710 2023-10-16T15:33:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 -- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 msgr2=0x7f32a40b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 --2- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40b2d10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009de0 tx=0x7f32a002be50 comp rx=0 tx=0).stop 2023-10-16T15:33:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 -- 172.21.15.29:0/4016613362 shutdown_connections 2023-10-16T15:33:56.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 --2- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40cfaf0 0x7f32a40b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 --2- 172.21.15.29:0/4016613362 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40b2d10 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 -- 172.21.15.29:0/4016613362 >> 172.21.15.29:0/4016613362 conn(0x7f32a40d2cc0 msgr2=0x7f32a40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 -- 172.21.15.29:0/4016613362 shutdown_connections 2023-10-16T15:33:56.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.472+0000 7f32aacb4640 1 -- 172.21.15.29:0/4016613362 wait complete. 2023-10-16T15:33:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.475+0000 7f32aacb4640 1 Processor -- start 2023-10-16T15:33:56.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.475+0000 7f32aacb4640 1 -- start start 2023-10-16T15:33:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32aacb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40b38e0 0x7f32a40c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32aacb4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32aacb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 0x7f32a40d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32aacb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40c92b0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32aacb4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40b15a0 con 0x7f32a40cf710 2023-10-16T15:33:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32aacb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40b0d00 con 0x7f32a40b38e0 2023-10-16T15:33:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.476+0000 7f32a9251640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 0x7f32a40d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 0x7f32a40d0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48752/0 (socket says 172.21.15.29:48752) 2023-10-16T15:33:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 -- 172.21.15.29:0/2657829656 learned_addr learned my addr 172.21.15.29:0/2657829656 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f329bfff640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a8a50640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40b38e0 0x7f32a40c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 -- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40b38e0 msgr2=0x7f32a40c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40b38e0 0x7f32a40c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 -- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 msgr2=0x7f32a40cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 -- 172.21.15.29:0/2657829656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0009a70 con 0x7f32a40cfaf0 2023-10-16T15:33:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f329bfff640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a8a50640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40b38e0 0x7f32a40c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:33:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.477+0000 7f32a9251640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 0x7f32a40d0870 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f329c00c390 tx=0x7f329c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.478+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c005ce0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.478+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c005e40 con 0x7f32a40cfaf0 2023-10-16T15:33:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.478+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c007310 con 0x7f32a40cfaf0 2023-10-16T15:33:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.478+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a40b0fe0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.478+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a40ca2d0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.479+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a404e160 con 0x7f32a40cfaf0 2023-10-16T15:33:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.480+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f329c007470 con 0x7f32a40cfaf0 2023-10-16T15:33:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.481+0000 7f32997fa640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 0x7f329006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.481+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f329c088e40 con 0x7f32a40cfaf0 2023-10-16T15:33:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.481+0000 7f32a8a50640 1 -- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 msgr2=0x7f329006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.481+0000 7f32a8a50640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 0x7f329006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.485+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f329c053db0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.676+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f32a40cb2b0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.681+0000 7f32997fa640 1 -- 172.21.15.29:0/2657829656 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f329c0170c0 con 0x7f32a40cfaf0 2023-10-16T15:33:56.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.682+0000 7f32a8a50640 1 -- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 msgr2=0x7f329006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:56.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.682+0000 7f32a8a50640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 0x7f329006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 msgr2=0x7f329006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 0x7f329006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 msgr2=0x7f32a40d0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 0x7f32a40d0870 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f329c00c390 tx=0x7f329c00c850 comp rx=0 tx=0).stop 2023-10-16T15:33:56.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 shutdown_connections 2023-10-16T15:33:56.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7f329006afe0 0x7f329006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32a40cfaf0 0x7f32a40d0870 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32a40cf710 0x7f32a40cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 --2- 172.21.15.29:0/2657829656 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32a40b38e0 0x7f32a40c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.683+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 >> 172.21.15.29:0/2657829656 conn(0x7f32a40d2cc0 msgr2=0x7f32a40d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.684+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 shutdown_connections 2023-10-16T15:33:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.684+0000 7f32aacb4640 1 -- 172.21.15.29:0/2657829656 wait complete. 2023-10-16T15:33:56.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:33:56.703 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T15:33:56.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-10-16T15:33:56.789 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:56.787+0000 7f932b213040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:33:56.843 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:56.841+0000 7f932b213040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:33:56.883 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:56.880+0000 7fa9facb6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:33:56.892 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:56.891+0000 7f932b213040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:33:56.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:56.936+0000 7fa9facb6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:33:56.952 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:56.950+0000 7f932b213040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:33:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.956+0000 7feb3bf05640 1 Processor -- start 2023-10-16T15:33:56.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.956+0000 7feb3bf05640 1 -- start start 2023-10-16T15:33:56.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3bf05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3415d330 0x7feb341618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3bf05640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb3406c7d0 0x7feb34065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3bf05640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb340d0890 con 0x7feb3406cc80 2023-10-16T15:33:56.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3bf05640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3406b1e0 con 0x7feb3406c7d0 2023-10-16T15:33:56.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3bf05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340d05b0 con 0x7feb3415d330 2023-10-16T15:33:56.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3a4a2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feb3406cc80 0x7feb340ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36290/0 (socket says 172.21.15.29:36290) 2023-10-16T15:33:56.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.958+0000 7feb3a4a2640 1 -- 172.21.15.29:0/259130133 learned_addr learned my addr 172.21.15.29:0/259130133 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:33:56.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb394a0640 1 --2- 172.21.15.29:0/259130133 >> v2:172.21.15.29:3300/0 conn(0x7feb3406c7d0 0x7feb34065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb39ca1640 1 --2- 172.21.15.29:0/259130133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3415d330 0x7feb341618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb38c9f640 1 -- 172.21.15.29:0/259130133 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397746839 0 0) 0x7feb340d0890 con 0x7feb3406cc80 2023-10-16T15:33:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb38c9f640 1 -- 172.21.15.29:0/259130133 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20003520 con 0x7feb3406cc80 2023-10-16T15:33:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb394a0640 1 -- 172.21.15.29:0/259130133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3415d330 msgr2=0x7feb341618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:33:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb394a0640 1 --2- 172.21.15.29:0/259130133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3415d330 0x7feb341618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb394a0640 1 -- 172.21.15.29:0/259130133 >> v1:172.21.15.29:6789/0 conn(0x7feb3406cc80 legacy=0x7feb340ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.959+0000 7feb394a0640 1 -- 172.21.15.29:0/259130133 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb34160dd0 con 0x7feb3406c7d0 2023-10-16T15:33:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.960+0000 7feb394a0640 1 --2- 172.21.15.29:0/259130133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3406c7d0 0x7feb34065ab0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7feb240097a0 tx=0x7feb2402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c9d201a8d156a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb38c9f640 1 -- 172.21.15.29:0/259130133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24031780 con 0x7feb3406c7d0 2023-10-16T15:33:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb38c9f640 1 -- 172.21.15.29:0/259130133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb240318e0 con 0x7feb3406c7d0 2023-10-16T15:33:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb38c9f640 1 -- 172.21.15.29:0/259130133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24031c50 con 0x7feb3406c7d0 2023-10-16T15:33:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 -- 172.21.15.29:0/259130133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3406c7d0 msgr2=0x7feb34065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 --2- 172.21.15.29:0/259130133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3406c7d0 0x7feb34065ab0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7feb240097a0 tx=0x7feb2402b820 comp rx=0 tx=0).stop 2023-10-16T15:33:56.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 -- 172.21.15.29:0/259130133 shutdown_connections 2023-10-16T15:33:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 --2- 172.21.15.29:0/259130133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3406c7d0 0x7feb34065ab0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 --2- 172.21.15.29:0/259130133 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3415d330 0x7feb341618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 -- 172.21.15.29:0/259130133 >> 172.21.15.29:0/259130133 conn(0x7feb340d0ab0 msgr2=0x7feb340d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:33:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 -- 172.21.15.29:0/259130133 shutdown_connections 2023-10-16T15:33:56.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.961+0000 7feb3bf05640 1 -- 172.21.15.29:0/259130133 wait complete. 2023-10-16T15:33:56.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.963+0000 7feb3bf05640 1 Processor -- start 2023-10-16T15:33:56.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.963+0000 7feb3bf05640 1 -- start start 2023-10-16T15:33:56.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3bf05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3406c7d0 0x7feb340adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3bf05640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb3406cc80 0x7feb34158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3bf05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3415d330 0x7feb340b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3bf05640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340b3410 con 0x7feb3415d330 2023-10-16T15:33:56.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3bf05640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340acf00 con 0x7feb3406cc80 2023-10-16T15:33:56.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3bf05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340ad1e0 con 0x7feb3406c7d0 2023-10-16T15:33:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3a4a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3415d330 0x7feb340b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3a4a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3415d330 0x7feb340b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48770/0 (socket says 172.21.15.29:48770) 2023-10-16T15:33:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.964+0000 7feb3a4a2640 1 -- 172.21.15.29:0/4037916712 learned_addr learned my addr 172.21.15.29:0/4037916712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:33:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3406c7d0 0x7feb340adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:33:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3a4a2640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3406c7d0 msgr2=0x7feb340adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3a4a2640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feb3406c7d0 0x7feb340adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3a4a2640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb3406cc80 msgr2=0x7feb34158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:33:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3a4a2640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb3406cc80 0x7feb34158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3a4a2640 1 -- 172.21.15.29:0/4037916712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24009400 con 0x7feb3415d330 2023-10-16T15:33:56.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3a4a2640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3415d330 0x7feb340b2a70 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00ca70 tx=0x7feb2c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:33:56.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00abd0 con 0x7feb3415d330 2023-10-16T15:33:56.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c00ad30 con 0x7feb3415d330 2023-10-16T15:33:56.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c012600 con 0x7feb3415d330 2023-10-16T15:33:56.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.965+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb340ad4c0 con 0x7feb3415d330 2023-10-16T15:33:56.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.966+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb341594f0 con 0x7feb3415d330 2023-10-16T15:33:56.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.967+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb340ad040 con 0x7feb3415d330 2023-10-16T15:33:56.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.968+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 89348+0+0 (secure 0 0 0) 0x7feb2c007070 con 0x7feb3415d330 2023-10-16T15:33:56.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.968+0000 7feb2a7fc640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 0x7feb1806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:33:56.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.969+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feb2c089680 con 0x7feb3415d330 2023-10-16T15:33:56.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.969+0000 7feb39ca1640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 msgr2=0x7feb1806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:56.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.969+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 0x7feb1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:33:56.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:56.973+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb340ad040 con 0x7feb3415d330 2023-10-16T15:33:56.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:56.992+0000 7fa9facb6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:33:57.001 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.000+0000 7f932b213040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:33:57.059 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.056+0000 7fa9facb6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:33:57.110 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.108+0000 7f932b213040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:33:57.111 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.108+0000 7fa9facb6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:33:57.171 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.169+0000 7f932b213040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:57.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:57.170+0000 7feb39ca1640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 msgr2=0x7feb1806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:57.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:57.170+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 0x7feb1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:33:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:57.188+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 --> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7feb34161390 con 0x7feb1806afe0 2023-10-16T15:33:57.219 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.217+0000 7f932b213040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:33:57.227 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.224+0000 7fa9facb6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:33:57.277 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.275+0000 7f932b213040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:33:57.293 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.290+0000 7fa9facb6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:57.326 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.325+0000 7f932b213040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:33:57.344 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.342+0000 7fa9facb6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:33:57.404 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.402+0000 7fa9facb6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:33:57.457 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.455+0000 7fa9facb6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:33:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:57.571+0000 7feb39ca1640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 msgr2=0x7feb1806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:57.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:57.571+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 0x7feb1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T15:33:57.701 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.699+0000 7f932b213040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:33:57.858 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:57.855+0000 7fa9facb6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:33:57.860 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.858+0000 7f932b213040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:33:57.967 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:57.965+0000 7f932b213040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:33:58.016 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:58.014+0000 7f932b213040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:33:58.025 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.022+0000 7fa9facb6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:33:58.123 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:58.121+0000 7f932b213040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:58.138 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.135+0000 7fa9facb6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:33:58.190 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.188+0000 7fa9facb6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:33:58.229 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:58.227+0000 7f932b213040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:33:58.302 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.300+0000 7fa9facb6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:58.372+0000 7feb39ca1640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 msgr2=0x7feb1806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1981883088 2023-10-16T15:33:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:58.372+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 0x7feb1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T15:33:58.384 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:58.382+0000 7f932b213040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:58.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.412+0000 7fa9facb6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:33:58.432 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:58.430+0000 7f932b213040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:58.549 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:58.547+0000 7f932b213040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:33:58.579 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.576+0000 7fa9facb6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:33:58.630 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.628+0000 7fa9facb6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:33:58.755 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:58.752+0000 7fa9facb6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:33:59.368 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:59.366+0000 7f932b213040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:33:59.645 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:59.642+0000 7fa9facb6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:33:59.662 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:59.660+0000 7f932b213040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:33:59.718 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:59.716+0000 7f932b213040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:33:59.832 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:33:59.830+0000 7f932b213040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:33:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:59.853+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 91) v1 ==== 88957+0+0 (secure 0 0 0) 0x7feb2c054a70 con 0x7feb3415d330 2023-10-16T15:33:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:59.854+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 msgr2=0x7feb1806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:33:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:33:59.854+0000 7feb2a7fc640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/1981883088,v1:172.21.15.170:6833/1981883088] conn(0x7feb1806afe0 0x7feb1806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:33:59.973 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:33:59.970+0000 7fa9facb6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:34:00.037 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.035+0000 7f92a789b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.037 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.035+0000 7f92a789b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.037 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.035+0000 7f92a789b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.037 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.035+0000 7f92a789b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.038 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.035+0000 7f92a789b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.038 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.036+0000 7f92aa8e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.038 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.036+0000 7f92aa8e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.039 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.036+0000 7f92aa8e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.039 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.036+0000 7f92aa8e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.039 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:00.036+0000 7f92aa8e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:00.041 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:00.038+0000 7fa9facb6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:34:00.162 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:00.160+0000 7fa9facb6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:34:00.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:00.863+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 92) v1 ==== 89084+0+0 (secure 0 0 0) 0x7feb2c019cd0 con 0x7feb3415d330 2023-10-16T15:34:00.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:00.864+0000 7feb2a7fc640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feb18035880 0x7feb18074f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:00.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:00.864+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 --> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7feb18035c60 con 0x7feb18035880 2023-10-16T15:34:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:00.870+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feb18035880 0x7feb18074f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:00.878+0000 7feb39ca1640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feb18035880 0x7feb18074f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feb300093d0 tx=0x7feb30008040 comp rx=0 tx=0).ready entity=mgr.7750 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.052+0000 7feb2a7fc640 1 -- 172.21.15.29:0/4037916712 <== mgr.7750 v2:172.21.15.170:6832/3772833058 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7feb18035c60 con 0x7feb18035880 2023-10-16T15:34:01.055 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-16T15:34:01.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.055+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feb18035880 msgr2=0x7feb18074f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.055+0000 7feb3bf05640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feb18035880 0x7feb18074f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feb300093d0 tx=0x7feb30008040 comp rx=0 tx=0).stop 2023-10-16T15:34:01.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.055+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3415d330 msgr2=0x7feb340b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.055+0000 7feb3bf05640 1 --2- 172.21.15.29:0/4037916712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb3415d330 0x7feb340b2a70 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00ca70 tx=0x7feb2c00cf30 comp rx=0 tx=0).stop 2023-10-16T15:34:01.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.056+0000 7feb3a4a2640 1 -- 172.21.15.29:0/4037916712 reap_dead start 2023-10-16T15:34:01.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.056+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 shutdown_connections 2023-10-16T15:34:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.056+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 >> 172.21.15.29:0/4037916712 conn(0x7feb340d0ab0 msgr2=0x7feb3415f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.056+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 shutdown_connections 2023-10-16T15:34:01.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.056+0000 7feb3bf05640 1 -- 172.21.15.29:0/4037916712 wait complete. 2023-10-16T15:34:01.081 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T15:34:01.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T15:34:01.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.310+0000 7f6e20a52640 1 Processor -- start 2023-10-16T15:34:01.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.311+0000 7f6e20a52640 1 -- start start 2023-10-16T15:34:01.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.312+0000 7f6e20a52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 0x7f6e1c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.312+0000 7f6e20a52640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c15aef0 con 0x7f6e1c070450 2023-10-16T15:34:01.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.312+0000 7f6e20a52640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c06cc80 con 0x7f6e1c06c810 2023-10-16T15:34:01.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.312+0000 7f6e20a52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c15e490 con 0x7f6e1c159950 2023-10-16T15:34:01.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.312+0000 7f6e1affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6e1c070450 0x7f6e1c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36300/0 (socket says 172.21.15.29:36300) 2023-10-16T15:34:01.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.312+0000 7f6e1affd640 1 -- 172.21.15.29:0/1029444976 learned_addr learned my addr 172.21.15.29:0/1029444976 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:01.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 0x7f6e1c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784053295 0 0) 0x7f6e04001490 con 0x7f6e1c070450 2023-10-16T15:34:01.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08003520 con 0x7f6e1c070450 2023-10-16T15:34:01.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475357590 0 0) 0x7f6e08003520 con 0x7f6e1c070450 2023-10-16T15:34:01.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 msgr2=0x7f6e1c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:01.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 0x7f6e1c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 >> v1:172.21.15.170:6789/0 conn(0x7f6e1c06c810 legacy=0x7f6e1c0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c15d910 con 0x7f6e1c070450 2023-10-16T15:34:01.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6e04001b80 con 0x7f6e1c070450 2023-10-16T15:34:01.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 >> v1:172.21.15.29:6789/0 conn(0x7f6e1c070450 legacy=0x7f6e1c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e08003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e080047c0 0x7f6e08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 0x7f6e080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c15e490 con 0x7f6e080039a0 2023-10-16T15:34:01.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0800bab0 con 0x7f6e080047c0 2023-10-16T15:34:01.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.313+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0800de80 con 0x7f6e1c159950 2023-10-16T15:34:01.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.314+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e08003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.314+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 msgr2=0x7f6e080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:01.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.314+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c159950 0x7f6e080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.314+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e080047c0 msgr2=0x7f6e08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:01.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.314+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e080047c0 0x7f6e08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.314+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/1029444976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08003520 con 0x7f6e080039a0 2023-10-16T15:34:01.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.315+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e08003dc0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009ac0 tx=0x7f6e080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd20c4e6f047ef4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c02c710 con 0x7f6e080039a0 2023-10-16T15:34:01.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c02c870 con 0x7f6e080039a0 2023-10-16T15:34:01.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e1a7fc640 1 -- 172.21.15.29:0/1029444976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c02cc00 con 0x7f6e080039a0 2023-10-16T15:34:01.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e20a52640 1 -- 172.21.15.29:0/1029444976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 msgr2=0x7f6e08003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e20a52640 1 --2- 172.21.15.29:0/1029444976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e08003dc0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009ac0 tx=0x7f6e080057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e1bfff640 1 -- 172.21.15.29:0/1029444976 reap_dead start 2023-10-16T15:34:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e20a52640 1 -- 172.21.15.29:0/1029444976 shutdown_connections 2023-10-16T15:34:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e20a52640 1 -- 172.21.15.29:0/1029444976 >> 172.21.15.29:0/1029444976 conn(0x7f6e1c0ce6a0 msgr2=0x7f6e1c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:01.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e20a52640 1 -- 172.21.15.29:0/1029444976 shutdown_connections 2023-10-16T15:34:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.316+0000 7f6e20a52640 1 -- 172.21.15.29:0/1029444976 wait complete. 2023-10-16T15:34:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.318+0000 7f6e20a52640 1 Processor -- start 2023-10-16T15:34:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.318+0000 7f6e20a52640 1 -- start start 2023-10-16T15:34:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.319+0000 7f6e20a52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e1c1548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.319+0000 7f6e20a52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c0c4700 0x7f6e1c0b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.319+0000 7f6e20a52640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e1c0b2940 0x7f6e1c0aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.319+0000 7f6e20a52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0c36b0 con 0x7f6e080039a0 2023-10-16T15:34:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.319+0000 7f6e20a52640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c154e20 con 0x7f6e1c0b2940 2023-10-16T15:34:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.319+0000 7f6e20a52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0c33d0 con 0x7f6e1c0c4700 2023-10-16T15:34:01.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e1c1548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e1c1548e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48792/0 (socket says 172.21.15.29:48792) 2023-10-16T15:34:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/941582282 learned_addr learned my addr 172.21.15.29:0/941582282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1bfff640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e1c0b2940 0x7f6e1c0aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1affd640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c0c4700 0x7f6e1c0b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c0c4700 msgr2=0x7f6e1c0b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c0c4700 0x7f6e1c0b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e1c0b2940 msgr2=0x7f6e1c0aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e1c0b2940 0x7f6e1c0aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1b7fe640 1 -- 172.21.15.29:0/941582282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0ae1a0 con 0x7f6e080039a0 2023-10-16T15:34:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1bfff640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e1c0b2940 0x7f6e1c0aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:01.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.320+0000 7f6e1affd640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c0c4700 0x7f6e1c0b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:01.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.321+0000 7f6e1b7fe640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e1c1548e0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c02c2b0 tx=0x7f6e0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.321+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c003810 con 0x7f6e080039a0 2023-10-16T15:34:01.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.321+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c02de90 con 0x7f6e080039a0 2023-10-16T15:34:01.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.321+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c003b70 con 0x7f6e080039a0 2023-10-16T15:34:01.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.321+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c0ae480 con 0x7f6e080039a0 2023-10-16T15:34:01.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.321+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c0b2df0 con 0x7f6e080039a0 2023-10-16T15:34:01.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.322+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c04e9b0 con 0x7f6e080039a0 2023-10-16T15:34:01.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.323+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f6e0c003cd0 con 0x7f6e080039a0 2023-10-16T15:34:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.324+0000 7f6e13fff640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f6dfc06ad40 0x7f6dfc06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.324+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6e0c0ab2e0 con 0x7f6e080039a0 2023-10-16T15:34:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.325+0000 7f6e1affd640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f6dfc06ad40 0x7f6dfc06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.326+0000 7f6e1affd640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f6dfc06ad40 0x7f6dfc06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6e04001cb0 tx=0x7f6e04009040 comp rx=0 tx=0).ready entity=mgr.7750 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.329+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e0c0765a0 con 0x7f6e080039a0 2023-10-16T15:34:01.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.546+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 --> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e1c15ded0 con 0x7f6dfc06ad40 2023-10-16T15:34:01.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.713+0000 7f6e13fff640 1 -- 172.21.15.29:0/941582282 <== mgr.7750 v2:172.21.15.170:6832/3772833058 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6e0400dd00 con 0x7f6dfc06ad40 2023-10-16T15:34:01.716 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-16T15:34:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f6dfc06ad40 msgr2=0x7f6dfc06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f6dfc06ad40 0x7f6dfc06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6e04001cb0 tx=0x7f6e04009040 comp rx=0 tx=0).stop 2023-10-16T15:34:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 msgr2=0x7f6e1c1548e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e1c1548e0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c02c2b0 tx=0x7f6e0c004730 comp rx=0 tx=0).stop 2023-10-16T15:34:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 shutdown_connections 2023-10-16T15:34:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f6dfc06ad40 0x7f6dfc06d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e1c0b2940 0x7f6e1c0aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e1c0c4700 0x7f6e1c0b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 --2- 172.21.15.29:0/941582282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e080039a0 0x7f6e1c1548e0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 >> 172.21.15.29:0/941582282 conn(0x7f6e1c0ce6a0 msgr2=0x7f6e1c15b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 shutdown_connections 2023-10-16T15:34:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.717+0000 7f6e20a52640 1 -- 172.21.15.29:0/941582282 wait complete. 2023-10-16T15:34:01.743 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T15:34:01.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T15:34:01.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.974+0000 7faa390f1640 1 Processor -- start 2023-10-16T15:34:01.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.974+0000 7faa390f1640 1 -- start start 2023-10-16T15:34:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.975+0000 7faa390f1640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faa34159950 0x7faa3415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa390f1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa34070450 0x7faa340708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa390f1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa3406c100 0x7faa3406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa390f1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa34070e20 con 0x7faa34070450 2023-10-16T15:34:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa390f1640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3415e490 con 0x7faa34159950 2023-10-16T15:34:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa390f1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3415aef0 con 0x7faa3406c100 2023-10-16T15:34:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa337fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa34070450 0x7faa340708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa33fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faa34159950 0x7faa3415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa337fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa34070450 0x7faa340708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48794/0 (socket says 172.21.15.29:48794) 2023-10-16T15:34:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa337fe640 1 -- 172.21.15.29:0/3420878768 learned_addr learned my addr 172.21.15.29:0/3420878768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.976+0000 7faa388f0640 1 --2- 172.21.15.29:0/3420878768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa3406c100 0x7faa3406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.977+0000 7faa337fe640 1 -- 172.21.15.29:0/3420878768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa3406c100 msgr2=0x7faa3406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:01.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.977+0000 7faa337fe640 1 --2- 172.21.15.29:0/3420878768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa3406c100 0x7faa3406c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.977+0000 7faa337fe640 1 -- 172.21.15.29:0/3420878768 >> v2:172.21.15.170:3300/0 conn(0x7faa34159950 msgr2=0x7faa3415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.977+0000 7faa337fe640 1 --2- 172.21.15.29:0/3420878768 >> v2:172.21.15.170:3300/0 conn(0x7faa34159950 0x7faa3415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.977+0000 7faa337fe640 1 -- 172.21.15.29:0/3420878768 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3415d910 con 0x7faa34070450 2023-10-16T15:34:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.978+0000 7faa337fe640 1 --2- 172.21.15.29:0/3420878768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa34070450 0x7faa340708e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7faa340ca950 tx=0x7faa2802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9749f79a1517c73e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.978+0000 7faa32ffd640 1 -- 172.21.15.29:0/3420878768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa280317c0 con 0x7faa34070450 2023-10-16T15:34:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.978+0000 7faa32ffd640 1 -- 172.21.15.29:0/3420878768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa28031920 con 0x7faa34070450 2023-10-16T15:34:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.978+0000 7faa32ffd640 1 -- 172.21.15.29:0/3420878768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28031cb0 con 0x7faa34070450 2023-10-16T15:34:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 -- 172.21.15.29:0/3420878768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa34070450 msgr2=0x7faa340708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 --2- 172.21.15.29:0/3420878768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa34070450 0x7faa340708e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7faa340ca950 tx=0x7faa2802b860 comp rx=0 tx=0).stop 2023-10-16T15:34:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 -- 172.21.15.29:0/3420878768 shutdown_connections 2023-10-16T15:34:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 --2- 172.21.15.29:0/3420878768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa3406c100 0x7faa3406c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 --2- 172.21.15.29:0/3420878768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa34070450 0x7faa340708e0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 --2- 172.21.15.29:0/3420878768 >> v2:172.21.15.170:3300/0 conn(0x7faa34159950 0x7faa3415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 -- 172.21.15.29:0/3420878768 >> 172.21.15.29:0/3420878768 conn(0x7faa340ce6a0 msgr2=0x7faa340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 -- 172.21.15.29:0/3420878768 shutdown_connections 2023-10-16T15:34:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.979+0000 7faa390f1640 1 -- 172.21.15.29:0/3420878768 wait complete. 2023-10-16T15:34:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.981+0000 7faa390f1640 1 Processor -- start 2023-10-16T15:34:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.981+0000 7faa390f1640 1 -- start start 2023-10-16T15:34:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.982+0000 7faa390f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa3406c100 0x7faa340c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.982+0000 7faa390f1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa34070450 0x7faa340c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.982+0000 7faa390f1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa34159950 0x7faa340c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.982+0000 7faa390f1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa340c2ed0 con 0x7faa3406c100 2023-10-16T15:34:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.982+0000 7faa390f1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa340c2250 con 0x7faa34159950 2023-10-16T15:34:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.982+0000 7faa390f1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa340c2530 con 0x7faa34070450 2023-10-16T15:34:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa3406c100 0x7faa340c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa3406c100 0x7faa340c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48798/0 (socket says 172.21.15.29:48798) 2023-10-16T15:34:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 -- 172.21.15.29:0/3475135359 learned_addr learned my addr 172.21.15.29:0/3475135359 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa337fe640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa34070450 0x7faa340c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa388f0640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa34159950 0x7faa340c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 -- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa34070450 msgr2=0x7faa340c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa34070450 0x7faa340c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 -- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa34159950 msgr2=0x7faa340c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa34159950 0x7faa340c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 -- 172.21.15.29:0/3475135359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa280094d0 con 0x7faa3406c100 2023-10-16T15:34:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa337fe640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa34070450 0x7faa340c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.983+0000 7faa33fff640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa3406c100 0x7faa340c46a0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7faa2400be00 tx=0x7faa24013490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.984+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24015040 con 0x7faa3406c100 2023-10-16T15:34:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.984+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa24019600 con 0x7faa3406c100 2023-10-16T15:34:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.984+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24010a60 con 0x7faa3406c100 2023-10-16T15:34:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.984+0000 7faa390f1640 1 -- 172.21.15.29:0/3475135359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa340c2810 con 0x7faa3406c100 2023-10-16T15:34:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.984+0000 7faa390f1640 1 -- 172.21.15.29:0/3475135359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa340c4210 con 0x7faa3406c100 2023-10-16T15:34:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.985+0000 7faa390f1640 1 -- 172.21.15.29:0/3475135359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3404e9b0 con 0x7faa3406c100 2023-10-16T15:34:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.986+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 89084+0+0 (secure 0 0 0) 0x7faa240199b0 con 0x7faa3406c100 2023-10-16T15:34:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.987+0000 7faa30ff9640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7faa2006ad10 0x7faa2006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.987+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7faa2408dd30 con 0x7faa3406c100 2023-10-16T15:34:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.988+0000 7faa337fe640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7faa2006ad10 0x7faa2006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.989+0000 7faa337fe640 1 --2- 172.21.15.29:0/3475135359 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7faa2006ad10 0x7faa2006d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa28036000 tx=0x7faa2803d040 comp rx=0 tx=0).ready entity=mgr.7750 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:01.992+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa24058ed0 con 0x7faa3406c100 2023-10-16T15:34:02.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.205+0000 7faa390f1640 1 -- 172.21.15.29:0/3475135359 --> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7faa3415dbf0 con 0x7faa2006ad10 2023-10-16T15:34:02.225 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:02.223+0000 7f92b18ef640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T15:34:02.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.223+0000 7faa30ff9640 1 -- 172.21.15.29:0/3475135359 <== mgr.7750 v2:172.21.15.170:6832/3772833058 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7faa28031b90 con 0x7faa2006ad10 2023-10-16T15:34:02.226 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T15:34:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T15:34:02.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:02.241 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T15:34:02.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:34:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.535+0000 7efcbc975640 1 Processor -- start 2023-10-16T15:34:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.535+0000 7efcbc975640 1 -- start start 2023-10-16T15:34:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.536+0000 7efcbc975640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efcb80ad7a0 0x7efcb8070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcbc975640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80ae1a0 con 0x7efcb80adc30 2023-10-16T15:34:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcbc975640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80ae480 con 0x7efcb8160c80 2023-10-16T15:34:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcbc975640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb80ae760 con 0x7efcb80ad7a0 2023-10-16T15:34:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcb7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efcb80adc30 0x7efcb8140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36302/0 (socket says 172.21.15.29:36302) 2023-10-16T15:34:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcb7fff640 1 -- 172.21.15.29:0/3125341542 learned_addr learned my addr 172.21.15.29:0/3125341542 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcaffff640 1 --2- 172.21.15.29:0/3125341542 >> v2:172.21.15.170:3300/0 conn(0x7efcb80ad7a0 0x7efcb8070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260173502 0 0) 0x7efcb80ae1a0 con 0x7efcb80adc30 2023-10-16T15:34:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.537+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca4003520 con 0x7efcb80adc30 2023-10-16T15:34:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661112996 0 0) 0x7efca4003520 con 0x7efcb80adc30 2023-10-16T15:34:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 >> v2:172.21.15.170:3300/0 conn(0x7efcb80ad7a0 msgr2=0x7efcb8070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 --2- 172.21.15.29:0/3125341542 >> v2:172.21.15.170:3300/0 conn(0x7efcb80ad7a0 0x7efcb8070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 >> v1:172.21.15.170:6789/0 conn(0x7efcb8160c80 legacy=0x7efcb8161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb80aea40 con 0x7efcb80adc30 2023-10-16T15:34:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7efcb00019f0 con 0x7efcb80adc30 2023-10-16T15:34:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 >> v1:172.21.15.29:6789/0 conn(0x7efcb80adc30 legacy=0x7efcb8140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40039c0 0x7efca4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40047c0 0x7efca4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca4005180 0x7efca400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80ae760 con 0x7efca40047c0 2023-10-16T15:34:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca400c050 con 0x7efca4005180 2023-10-16T15:34:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca400c210 con 0x7efca40039c0 2023-10-16T15:34:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.538+0000 7efcaffff640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40047c0 0x7efca4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcb77fe640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40039c0 0x7efca4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcb7fff640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca4005180 0x7efca400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcaffff640 1 -- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40039c0 msgr2=0x7efca4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcaffff640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40039c0 0x7efca4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcaffff640 1 -- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca4005180 msgr2=0x7efca400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcaffff640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca4005180 0x7efca400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcaffff640 1 -- 172.21.15.29:0/3125341542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca4003520 con 0x7efca40047c0 2023-10-16T15:34:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcb7fff640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efca4005180 0x7efca400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.539+0000 7efcb7fff640 1 -- 172.21.15.29:0/3125341542 reap_dead start 2023-10-16T15:34:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.540+0000 7efcaffff640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40047c0 0x7efca4004c40 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7efca0009ac0 tx=0x7efca40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dbf8367f9c6e10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.540+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca002c6b0 con 0x7efca40047c0 2023-10-16T15:34:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.540+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca002c810 con 0x7efca40047c0 2023-10-16T15:34:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.540+0000 7efcb6ffd640 1 -- 172.21.15.29:0/3125341542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca002cba0 con 0x7efca40047c0 2023-10-16T15:34:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 -- 172.21.15.29:0/3125341542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40047c0 msgr2=0x7efca4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40047c0 0x7efca4004c40 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7efca0009ac0 tx=0x7efca40057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 -- 172.21.15.29:0/3125341542 shutdown_connections 2023-10-16T15:34:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 --2- 172.21.15.29:0/3125341542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca40047c0 0x7efca4004c40 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 -- 172.21.15.29:0/3125341542 >> 172.21.15.29:0/3125341542 conn(0x7efcb80cc230 msgr2=0x7efcb80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 -- 172.21.15.29:0/3125341542 shutdown_connections 2023-10-16T15:34:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.541+0000 7efcbc975640 1 -- 172.21.15.29:0/3125341542 wait complete. 2023-10-16T15:34:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.543+0000 7efcbc975640 1 Processor -- start 2023-10-16T15:34:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.543+0000 7efcbc975640 1 -- start start 2023-10-16T15:34:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcbc975640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40047c0 0x7efcb81448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcbc975640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 0x7efcb806b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcbc975640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efcb81f3360 0x7efcb81f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcbc975640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8143830 con 0x7efcb8143330 2023-10-16T15:34:02.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcbc975640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8143b10 con 0x7efcb81f3360 2023-10-16T15:34:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcbc975640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8143df0 con 0x7efca40047c0 2023-10-16T15:34:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 0x7efcb806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 0x7efcb806b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48824/0 (socket says 172.21.15.29:48824) 2023-10-16T15:34:02.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.544+0000 7efcaffff640 1 -- 172.21.15.29:0/1171701456 learned_addr learned my addr 172.21.15.29:0/1171701456 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcb7fff640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efcb81f3360 0x7efcb81f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcb77fe640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40047c0 0x7efcb81448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcaffff640 1 -- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40047c0 msgr2=0x7efcb81448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcaffff640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40047c0 0x7efcb81448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcaffff640 1 -- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efcb81f3360 msgr2=0x7efcb81f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcaffff640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efcb81f3360 0x7efcb81f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcaffff640 1 -- 172.21.15.29:0/1171701456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb81440d0 con 0x7efcb8143330 2023-10-16T15:34:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcaffff640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 0x7efcb806b300 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7efca002c250 tx=0x7efca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.545+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0003810 con 0x7efcb8143330 2023-10-16T15:34:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.546+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0003970 con 0x7efcb8143330 2023-10-16T15:34:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.546+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0003d00 con 0x7efcb8143330 2023-10-16T15:34:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.546+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb81443b0 con 0x7efcb8143330 2023-10-16T15:34:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.546+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb81f2fe0 con 0x7efcb8143330 2023-10-16T15:34:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.547+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb8144540 con 0x7efcb8143330 2023-10-16T15:34:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.548+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 89084+0+0 (secure 0 0 0) 0x7efca0047450 con 0x7efcb8143330 2023-10-16T15:34:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.549+0000 7efcb4ff9640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7efc9806ad40 0x7efc9806d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.549+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7efca00ab730 con 0x7efcb8143330 2023-10-16T15:34:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.549+0000 7efcb77fe640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7efc9806ad40 0x7efc9806d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.550+0000 7efcb77fe640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7efc9806ad40 0x7efc9806d200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efca8004e40 tx=0x7efca8009040 comp rx=0 tx=0).ready entity=mgr.7750 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.553+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efca0076b70 con 0x7efcb8143330 2023-10-16T15:34:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.736+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 93) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efca0076f40 con 0x7efcb8143330 2023-10-16T15:34:02.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:02.802+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7efcb804e9b0 con 0x7efcb8143330 2023-10-16T15:34:03.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.737+0000 7efcb4ff9640 1 -- 172.21.15.29:0/1171701456 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v94) v1 ==== 93+0+0 (secure 0 0 0) 0x7efca007de90 con 0x7efcb8143330 2023-10-16T15:34:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.741+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7efc9806ad40 msgr2=0x7efc9806d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:03.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.741+0000 7efcbc975640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7efc9806ad40 0x7efc9806d200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efca8004e40 tx=0x7efca8009040 comp rx=0 tx=0).stop 2023-10-16T15:34:03.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.741+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 msgr2=0x7efcb806b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:03.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.741+0000 7efcbc975640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 0x7efcb806b300 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7efca002c250 tx=0x7efca0004730 comp rx=0 tx=0).stop 2023-10-16T15:34:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 shutdown_connections 2023-10-16T15:34:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcb8143330 0x7efcb806b300 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7efc9806ad40 0x7efc9806d200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efcb81f3360 0x7efcb81f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 --2- 172.21.15.29:0/1171701456 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efca40047c0 0x7efcb81448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 >> 172.21.15.29:0/1171701456 conn(0x7efcb80cc230 msgr2=0x7efcb815f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 shutdown_connections 2023-10-16T15:34:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:03.743+0000 7efcbc975640 1 -- 172.21.15.29:0/1171701456 wait complete. 2023-10-16T15:34:03.772 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:34:03.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:34:03.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:34:03.980 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:34:03.981 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:34:03.981 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:04.034 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:34:04.034 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:04.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:04.271 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.269+0000 7f70ed22b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:34:04.322 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:04.319+0000 7fbf1b811040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:34:04.323 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.322+0000 7f70ed22b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:34:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.335+0000 7fc8f231a640 1 Processor -- start 2023-10-16T15:34:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.335+0000 7fc8f231a640 1 -- start start 2023-10-16T15:34:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.337+0000 7fc8f231a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 0x7fc8ec15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.337+0000 7fc8f231a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc8ec070450 0x7fc8ec0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.337+0000 7fc8f231a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec070df0 con 0x7fc8ec06c100 2023-10-16T15:34:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.337+0000 7fc8f231a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec15e490 con 0x7fc8ec070450 2023-10-16T15:34:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.337+0000 7fc8f231a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec15aef0 con 0x7fc8ec159950 2023-10-16T15:34:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f1318640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 0x7fc8ec15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f1318640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 0x7fc8ec15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:47286/0 (socket says 172.21.15.29:47286) 2023-10-16T15:34:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f1318640 1 -- 172.21.15.29:0/809063396 learned_addr learned my addr 172.21.15.29:0/809063396 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f0b17640 1 --2- 172.21.15.29:0/809063396 >> v2:172.21.15.170:3300/0 conn(0x7fc8ec070450 0x7fc8ec0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f1318640 1 -- 172.21.15.29:0/809063396 >> v2:172.21.15.170:3300/0 conn(0x7fc8ec070450 msgr2=0x7fc8ec0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f1318640 1 --2- 172.21.15.29:0/809063396 >> v2:172.21.15.170:3300/0 conn(0x7fc8ec070450 0x7fc8ec0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.338+0000 7fc8f1318640 1 -- 172.21.15.29:0/809063396 >> v1:172.21.15.170:6789/0 conn(0x7fc8ec06c100 legacy=0x7fc8ec06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.339+0000 7fc8f1318640 1 -- 172.21.15.29:0/809063396 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec06cc80 con 0x7fc8ec159950 2023-10-16T15:34:04.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.339+0000 7fc8f0b17640 1 --2- 172.21.15.29:0/809063396 >> v2:172.21.15.170:3300/0 conn(0x7fc8ec070450 0x7fc8ec0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.340+0000 7fc8f1318640 1 --2- 172.21.15.29:0/809063396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 0x7fc8ec15a9b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800bb10 tx=0x7fc8e802e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628fed86a42829b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.340+0000 7fc8e3fff640 1 -- 172.21.15.29:0/809063396 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e802ee70 con 0x7fc8ec159950 2023-10-16T15:34:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.340+0000 7fc8e3fff640 1 -- 172.21.15.29:0/809063396 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8036440 con 0x7fc8ec159950 2023-10-16T15:34:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.340+0000 7fc8e3fff640 1 -- 172.21.15.29:0/809063396 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e80367d0 con 0x7fc8ec159950 2023-10-16T15:34:04.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 -- 172.21.15.29:0/809063396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 msgr2=0x7fc8ec15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 --2- 172.21.15.29:0/809063396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 0x7fc8ec15a9b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800bb10 tx=0x7fc8e802e0c0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 -- 172.21.15.29:0/809063396 shutdown_connections 2023-10-16T15:34:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 --2- 172.21.15.29:0/809063396 >> v2:172.21.15.170:3300/0 conn(0x7fc8ec070450 0x7fc8ec0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 --2- 172.21.15.29:0/809063396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec159950 0x7fc8ec15a9b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 -- 172.21.15.29:0/809063396 >> 172.21.15.29:0/809063396 conn(0x7fc8ec0ce6a0 msgr2=0x7fc8ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:04.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 -- 172.21.15.29:0/809063396 shutdown_connections 2023-10-16T15:34:04.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.341+0000 7fc8f231a640 1 -- 172.21.15.29:0/809063396 wait complete. 2023-10-16T15:34:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.343+0000 7fc8f231a640 1 Processor -- start 2023-10-16T15:34:04.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.343+0000 7fc8f231a640 1 -- start start 2023-10-16T15:34:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f231a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec06c100 0x7fc8ec1f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f231a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 0x7fc8ec156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f231a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8ec0c3950 0x7fc8ec0c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f231a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec1566b0 con 0x7fc8ec070450 2023-10-16T15:34:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f231a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c8ae0 con 0x7fc8ec0c3950 2023-10-16T15:34:04.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f231a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c5160 con 0x7fc8ec06c100 2023-10-16T15:34:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f0b17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 0x7fc8ec156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f0b17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 0x7fc8ec156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59944/0 (socket says 172.21.15.29:59944) 2023-10-16T15:34:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.344+0000 7fc8f0b17640 1 -- 172.21.15.29:0/2647334408 learned_addr learned my addr 172.21.15.29:0/2647334408 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f1318640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec06c100 0x7fc8ec1f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f1b19640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8ec0c3950 0x7fc8ec0c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f0b17640 1 -- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec06c100 msgr2=0x7fc8ec1f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f0b17640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec06c100 0x7fc8ec1f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f0b17640 1 -- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8ec0c3950 msgr2=0x7fc8ec0c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f0b17640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8ec0c3950 0x7fc8ec0c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f0b17640 1 -- 172.21.15.29:0/2647334408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e800b770 con 0x7fc8ec070450 2023-10-16T15:34:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f1318640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec06c100 0x7fc8ec1f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8f0b17640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 0x7fc8ec156170 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00e730 tx=0x7fc8dc00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.345+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc004230 con 0x7fc8ec070450 2023-10-16T15:34:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.346+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc004390 con 0x7fc8ec070450 2023-10-16T15:34:04.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.346+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec0c45e0 con 0x7fc8ec070450 2023-10-16T15:34:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.346+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00c8f0 con 0x7fc8ec070450 2023-10-16T15:34:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.346+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec0c4ba0 con 0x7fc8ec070450 2023-10-16T15:34:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.347+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec04e9b0 con 0x7fc8ec070450 2023-10-16T15:34:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.348+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc8dc00ca50 con 0x7fc8ec070450 2023-10-16T15:34:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.349+0000 7fc8e1ffb640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.349+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fc8dc0896a0 con 0x7fc8ec070450 2023-10-16T15:34:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.349+0000 7fc8f1318640 1 -- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 msgr2=0x7fc8d006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:04.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.349+0000 7fc8f1318640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:04.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.353+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8dc054320 con 0x7fc8ec070450 2023-10-16T15:34:04.379 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:04.377+0000 7fbf1b811040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:34:04.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.548+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8ec0b4620 con 0x7fc8ec070450 2023-10-16T15:34:04.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.549+0000 7fc8e1ffb640 1 -- 172.21.15.29:0/2647334408 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17131 (secure 0 0 0) 0x7fc8dc057940 con 0x7fc8ec070450 2023-10-16T15:34:04.552 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:04.553 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":180,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:59.836071+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2172765430":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2009585372":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/838551000":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6833/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6832/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3011898787":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2177136748":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2616191068":"2023-10-17T15:33:59.835992+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:04.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.550+0000 7fc8f1318640 1 -- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 msgr2=0x7fc8d006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:04.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.550+0000 7fc8f1318640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 msgr2=0x7fc8d006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 msgr2=0x7fc8ec156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 0x7fc8ec156170 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00e730 tx=0x7fc8dc00ebf0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 shutdown_connections 2023-10-16T15:34:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8ec070450 0x7fc8ec156170 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8ec0c3950 0x7fc8ec0c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 --2- 172.21.15.29:0/2647334408 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8ec06c100 0x7fc8ec1f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.552+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 >> 172.21.15.29:0/2647334408 conn(0x7fc8ec0ce6a0 msgr2=0x7fc8ec06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.553+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 shutdown_connections 2023-10-16T15:34:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.553+0000 7fc8f231a640 1 -- 172.21.15.29:0/2647334408 wait complete. 2023-10-16T15:34:04.574 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:33:43.022528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:33:44.097016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:34:04.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:34:04.770 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.768+0000 7f70ed22b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:34:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.786+0000 7f8d9269a640 1 Processor -- start 2023-10-16T15:34:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.787+0000 7f8d9269a640 1 -- start start 2023-10-16T15:34:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d9269a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c0ce250 0x7f8d8c06fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d9269a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d8c0b4400 0x7f8d8c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d9269a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8c0b2d80 con 0x7f8d8c15c7a0 2023-10-16T15:34:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d9269a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c0709a0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d9269a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0b2fb0 con 0x7f8d8c0ce250 2023-10-16T15:34:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d91e99640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d8c0b4400 0x7f8d8c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d91e99640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d8c0b4400 0x7f8d8c06ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59948/0 (socket says 172.21.15.29:59948) 2023-10-16T15:34:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.788+0000 7f8d91e99640 1 -- 172.21.15.29:0/423053630 learned_addr learned my addr 172.21.15.29:0/423053630 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.789+0000 7f8d90e97640 1 --2- 172.21.15.29:0/423053630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c0ce250 0x7f8d8c06fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.789+0000 7f8d91e99640 1 -- 172.21.15.29:0/423053630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c0ce250 msgr2=0x7f8d8c06fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.789+0000 7f8d91e99640 1 --2- 172.21.15.29:0/423053630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c0ce250 0x7f8d8c06fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.789+0000 7f8d91e99640 1 -- 172.21.15.29:0/423053630 >> v1:172.21.15.170:6789/0 conn(0x7f8d8c15c7a0 legacy=0x7f8d8c0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.789+0000 7f8d91e99640 1 -- 172.21.15.29:0/423053630 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0b3290 con 0x7f8d8c0b4400 2023-10-16T15:34:04.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.791+0000 7f8d91e99640 1 --2- 172.21.15.29:0/423053630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c06ff40 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009e50 tx=0x7f8d8402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f830fbc8cf0473d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.791+0000 7f8d83fff640 1 -- 172.21.15.29:0/423053630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8402cb00 con 0x7f8d8c0b4400 2023-10-16T15:34:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.791+0000 7f8d83fff640 1 -- 172.21.15.29:0/423053630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8402cc60 con 0x7f8d8c0b4400 2023-10-16T15:34:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.791+0000 7f8d83fff640 1 -- 172.21.15.29:0/423053630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d840343f0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 -- 172.21.15.29:0/423053630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 msgr2=0x7f8d8c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 --2- 172.21.15.29:0/423053630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c06ff40 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009e50 tx=0x7f8d8402bf60 comp rx=0 tx=0).stop 2023-10-16T15:34:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 -- 172.21.15.29:0/423053630 shutdown_connections 2023-10-16T15:34:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 --2- 172.21.15.29:0/423053630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c06ff40 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 --2- 172.21.15.29:0/423053630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c0ce250 0x7f8d8c06fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 -- 172.21.15.29:0/423053630 >> 172.21.15.29:0/423053630 conn(0x7f8d8c0b0a50 msgr2=0x7f8d8c0b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 -- 172.21.15.29:0/423053630 shutdown_connections 2023-10-16T15:34:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.792+0000 7f8d9269a640 1 -- 172.21.15.29:0/423053630 wait complete. 2023-10-16T15:34:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.793+0000 7f8d9269a640 1 Processor -- start 2023-10-16T15:34:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.794+0000 7f8d9269a640 1 -- start start 2023-10-16T15:34:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d9269a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c0c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d9269a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8c0ce250 0x7f8d8c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d9269a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c15c7a0 0x7f8d8c1526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d9269a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c151cd0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d9269a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0c2f20 con 0x7f8d8c0ce250 2023-10-16T15:34:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d9269a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0c3200 con 0x7f8d8c15c7a0 2023-10-16T15:34:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c0c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c0c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59964/0 (socket says 172.21.15.29:59964) 2023-10-16T15:34:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 -- 172.21.15.29:0/3579330231 learned_addr learned my addr 172.21.15.29:0/3579330231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 -- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c15c7a0 msgr2=0x7f8d8c1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c15c7a0 0x7f8d8c1526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 -- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8c0ce250 msgr2=0x7f8d8c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8c0ce250 0x7f8d8c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.795+0000 7f8d91698640 1 -- 172.21.15.29:0/3579330231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d84009b30 con 0x7f8d8c0b4400 2023-10-16T15:34:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.796+0000 7f8d91698640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c0c50b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800c450 tx=0x7f8d8800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.796+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d880052d0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.796+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d88005430 con 0x7f8d8c0b4400 2023-10-16T15:34:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.796+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88012870 con 0x7f8d8c0b4400 2023-10-16T15:34:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.796+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c0c34e0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.796+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c0c55f0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.797+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c0c57e0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.798+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8d880129d0 con 0x7f8d8c0b4400 2023-10-16T15:34:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.799+0000 7f8d81ffb640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f8d7006af90 0x7f8d7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.800+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8d88089450 con 0x7f8d8c0b4400 2023-10-16T15:34:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.800+0000 7f8d90e97640 1 -- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f8d7006af90 msgr2=0x7f8d7006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.800+0000 7f8d90e97640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f8d7006af90 0x7f8d7006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.804+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d88054270 con 0x7f8d8c0b4400 2023-10-16T15:34:04.822 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.821+0000 7f70ed22b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:34:04.860 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:04.858+0000 7fbf1b811040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:34:04.872 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.870+0000 7f70ed22b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:34:04.915 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:04.913+0000 7fbf1b811040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:34:04.933 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.931+0000 7f70ed22b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:34:04.968 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:04.966+0000 7fbf1b811040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:34:04.981 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:04.980+0000 7f70ed22b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:34:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.996+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d8c0b3830 con 0x7f8d8c0b4400 2023-10-16T15:34:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.997+0000 7f8d81ffb640 1 -- 172.21.15.29:0/3579330231 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v180) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d8c0b3830 con 0x7f8d8c0b4400 2023-10-16T15:34:04.999 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:34:05.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.999+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f8d7006af90 msgr2=0x7f8d7006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:05.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.999+0000 7f8d9269a640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f8d7006af90 0x7f8d7006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.999+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 msgr2=0x7f8d8c0c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:04.999+0000 7f8d9269a640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c0c50b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800c450 tx=0x7f8d8800c910 comp rx=0 tx=0).stop 2023-10-16T15:34:05.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 shutdown_connections 2023-10-16T15:34:05.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f8d7006af90 0x7f8d7006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d8c15c7a0 0x7f8d8c1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d8c0ce250 0x7f8d8c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 --2- 172.21.15.29:0/3579330231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d8c0b4400 0x7f8d8c0c50b0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 >> 172.21.15.29:0/3579330231 conn(0x7f8d8c0b0a50 msgr2=0x7f8d8c09f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:05.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 shutdown_connections 2023-10-16T15:34:05.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.000+0000 7f8d9269a640 1 -- 172.21.15.29:0/3579330231 wait complete. 2023-10-16T15:34:05.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:34:05.033 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.031+0000 7fbf1b811040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:34:05.085 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.083+0000 7fbf1b811040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:34:05.091 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.089+0000 7f70ed22b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:34:05.153 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.151+0000 7f70ed22b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:05.203 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.201+0000 7f70ed22b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:34:05.204 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.201+0000 7fbf1b811040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:34:05.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.219+0000 7fd401610640 1 Processor -- start 2023-10-16T15:34:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.219+0000 7fd401610640 1 -- start start 2023-10-16T15:34:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.220+0000 7fd401610640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3fc1618b0 0x7fd3fc065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd401610640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc06cee0 con 0x7fd3fc15d330 2023-10-16T15:34:05.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd401610640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc0663d0 con 0x7fd3fc065ff0 2023-10-16T15:34:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd401610640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3fc06b1e0 con 0x7fd3fc1618b0 2023-10-16T15:34:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3fa7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd3fc1618b0 0x7fd3fc065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3faffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd3fc15d330 0x7fd3fc06c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48418/0 (socket says 172.21.15.29:48418) 2023-10-16T15:34:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3faffd640 1 -- 172.21.15.29:0/3686513192 learned_addr learned my addr 172.21.15.29:0/3686513192 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:05.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3fa7fc640 1 -- 172.21.15.29:0/3686513192 >> v1:172.21.15.170:6789/0 conn(0x7fd3fc065ff0 legacy=0x7fd3fc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3fa7fc640 1 -- 172.21.15.29:0/3686513192 >> v1:172.21.15.29:6789/0 conn(0x7fd3fc15d330 legacy=0x7fd3fc06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3fa7fc640 1 -- 172.21.15.29:0/3686513192 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0d0890 con 0x7fd3fc1618b0 2023-10-16T15:34:05.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.221+0000 7fd3f9ffb640 1 -- 172.21.15.29:0/3686513192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330822188 0 0) 0x7fd3fc06cee0 con 0x7fd3fc15d330 2023-10-16T15:34:05.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.223+0000 7fd3fa7fc640 1 --2- 172.21.15.29:0/3686513192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc065ab0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00097a0 tx=0x7fd3f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3d4a043c9b242d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:05.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.223+0000 7fd3f9ffb640 1 -- 172.21.15.29:0/3686513192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0031830 con 0x7fd3fc1618b0 2023-10-16T15:34:05.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.223+0000 7fd3f9ffb640 1 -- 172.21.15.29:0/3686513192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f0031990 con 0x7fd3fc1618b0 2023-10-16T15:34:05.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.223+0000 7fd3f9ffb640 1 -- 172.21.15.29:0/3686513192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0031d20 con 0x7fd3fc1618b0 2023-10-16T15:34:05.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 -- 172.21.15.29:0/3686513192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc1618b0 msgr2=0x7fd3fc065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 --2- 172.21.15.29:0/3686513192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc065ab0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00097a0 tx=0x7fd3f002b8d0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 -- 172.21.15.29:0/3686513192 shutdown_connections 2023-10-16T15:34:05.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 --2- 172.21.15.29:0/3686513192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc065ab0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 -- 172.21.15.29:0/3686513192 >> 172.21.15.29:0/3686513192 conn(0x7fd3fc0d0ab0 msgr2=0x7fd3fc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 -- 172.21.15.29:0/3686513192 shutdown_connections 2023-10-16T15:34:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.224+0000 7fd401610640 1 -- 172.21.15.29:0/3686513192 wait complete. 2023-10-16T15:34:05.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.226+0000 7fd401610640 1 Processor -- start 2023-10-16T15:34:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.226+0000 7fd401610640 1 -- start start 2023-10-16T15:34:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd401610640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 0x7fd3fc154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd401610640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3fc15d330 0x7fd3fc15f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd401610640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd401610640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc15fae0 con 0x7fd3fc065ff0 2023-10-16T15:34:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd401610640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0c7af0 con 0x7fd3fc1618b0 2023-10-16T15:34:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd401610640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc15fd20 con 0x7fd3fc15d330 2023-10-16T15:34:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd3faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 0x7fd3fc154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd3faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 0x7fd3fc154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59994/0 (socket says 172.21.15.29:59994) 2023-10-16T15:34:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.227+0000 7fd3faffd640 1 -- 172.21.15.29:0/3137649651 learned_addr learned my addr 172.21.15.29:0/3137649651 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3fa7fc640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3fc15d330 0x7fd3fc15f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3fb7fe640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3faffd640 1 -- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3fc15d330 msgr2=0x7fd3fc15f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3faffd640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3fc15d330 0x7fd3fc15f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3faffd640 1 -- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc1618b0 msgr2=0x7fd3fc0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3faffd640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc0c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3faffd640 1 -- 172.21.15.29:0/3137649651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0009400 con 0x7fd3fc065ff0 2023-10-16T15:34:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3fa7fc640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3fc15d330 0x7fd3fc15f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3fb7fe640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3faffd640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 0x7fd3fc154fc0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec00c3f0 tx=0x7fd3ec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec0074e0 con 0x7fd3fc065ff0 2023-10-16T15:34:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec007640 con 0x7fd3fc065ff0 2023-10-16T15:34:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec004a00 con 0x7fd3fc065ff0 2023-10-16T15:34:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.228+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc0c7d40 con 0x7fd3fc065ff0 2023-10-16T15:34:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.229+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc0c8300 con 0x7fd3fc065ff0 2023-10-16T15:34:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.230+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc04e160 con 0x7fd3fc065ff0 2023-10-16T15:34:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.231+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd3ec004b60 con 0x7fd3fc065ff0 2023-10-16T15:34:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.232+0000 7fd3e37fe640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd3e806af90 0x7fd3e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.232+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd3ec089680 con 0x7fd3fc065ff0 2023-10-16T15:34:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.232+0000 7fd3fa7fc640 1 -- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd3e806af90 msgr2=0x7fd3e806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.232+0000 7fd3fa7fc640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd3e806af90 0x7fd3e806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:05.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.236+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3ec054530 con 0x7fd3fc065ff0 2023-10-16T15:34:05.261 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.259+0000 7f70ed22b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:34:05.273 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.270+0000 7fbf1b811040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:05.310 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.308+0000 7f70ed22b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:34:05.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.323+0000 7fbf1b811040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:34:05.385 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.383+0000 7fbf1b811040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:34:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.426+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3fc0c42f0 con 0x7fd3fc065ff0 2023-10-16T15:34:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.427+0000 7fd3e37fe640 1 -- 172.21.15.29:0/3137649651 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v180) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3ec057b50 con 0x7fd3fc065ff0 2023-10-16T15:34:05.430 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:34:05.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd3e806af90 msgr2=0x7fd3e806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:05.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd3e806af90 0x7fd3e806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 msgr2=0x7fd3fc154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 0x7fd3fc154fc0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec00c3f0 tx=0x7fd3ec00c8b0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 shutdown_connections 2023-10-16T15:34:05.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd3e806af90 0x7fd3e806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3fc1618b0 0x7fd3fc0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd3fc15d330 0x7fd3fc15f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 --2- 172.21.15.29:0/3137649651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3fc065ff0 0x7fd3fc154fc0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 >> 172.21.15.29:0/3137649651 conn(0x7fd3fc0d0ab0 msgr2=0x7fd3fc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:05.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 shutdown_connections 2023-10-16T15:34:05.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.430+0000 7fd401610640 1 -- 172.21.15.29:0/3137649651 wait complete. 2023-10-16T15:34:05.439 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.437+0000 7fbf1b811040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:34:05.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:34:05.687 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.685+0000 7f70ed22b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:34:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.704+0000 7fead31d7640 1 Processor -- start 2023-10-16T15:34:05.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.705+0000 7fead31d7640 1 -- start start 2023-10-16T15:34:05.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead31d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feacc15ad70 0x7feacc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead31d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7feacc0c9ca0 0x7feacc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead31d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0ca3b0 0x7feacc0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead31d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc0cad80 con 0x7feacc15ad70 2023-10-16T15:34:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead31d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc0cb030 con 0x7feacc0c9ca0 2023-10-16T15:34:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead31d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0c91c0 con 0x7feacc0ca3b0 2023-10-16T15:34:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead0f73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feacc15ad70 0x7feacc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead0f73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feacc15ad70 0x7feacc069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60008/0 (socket says 172.21.15.29:60008) 2023-10-16T15:34:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.706+0000 7fead0f73640 1 -- 172.21.15.29:0/3835130675 learned_addr learned my addr 172.21.15.29:0/3835130675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7feacbfff640 1 --2- 172.21.15.29:0/3835130675 >> v2:172.21.15.170:3300/0 conn(0x7feacc0c9ca0 0x7feacc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7fead1774640 1 --2- 172.21.15.29:0/3835130675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0ca3b0 0x7feacc0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7fead0f73640 1 -- 172.21.15.29:0/3835130675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0ca3b0 msgr2=0x7feacc0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7fead0f73640 1 --2- 172.21.15.29:0/3835130675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0ca3b0 0x7feacc0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7fead0f73640 1 -- 172.21.15.29:0/3835130675 >> v2:172.21.15.170:3300/0 conn(0x7feacc0c9ca0 msgr2=0x7feacc06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7fead0f73640 1 --2- 172.21.15.29:0/3835130675 >> v2:172.21.15.170:3300/0 conn(0x7feacc0c9ca0 0x7feacc06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.707+0000 7fead0f73640 1 -- 172.21.15.29:0/3835130675 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc0c94a0 con 0x7feacc15ad70 2023-10-16T15:34:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.708+0000 7fead0f73640 1 --2- 172.21.15.29:0/3835130675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc15ad70 0x7feacc069da0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7feacc0ce870 tx=0x7feac002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca1e6733fbc3482 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.708+0000 7feacb7fe640 1 -- 172.21.15.29:0/3835130675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac0031870 con 0x7feacc15ad70 2023-10-16T15:34:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.708+0000 7feacb7fe640 1 -- 172.21.15.29:0/3835130675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac00319d0 con 0x7feacc15ad70 2023-10-16T15:34:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.708+0000 7feacb7fe640 1 -- 172.21.15.29:0/3835130675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac0031d60 con 0x7feacc15ad70 2023-10-16T15:34:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 -- 172.21.15.29:0/3835130675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc15ad70 msgr2=0x7feacc069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 --2- 172.21.15.29:0/3835130675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc15ad70 0x7feacc069da0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7feacc0ce870 tx=0x7feac002b910 comp rx=0 tx=0).stop 2023-10-16T15:34:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 -- 172.21.15.29:0/3835130675 shutdown_connections 2023-10-16T15:34:05.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 --2- 172.21.15.29:0/3835130675 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0ca3b0 0x7feacc0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 --2- 172.21.15.29:0/3835130675 >> v2:172.21.15.170:3300/0 conn(0x7feacc0c9ca0 0x7feacc06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 --2- 172.21.15.29:0/3835130675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc15ad70 0x7feacc069da0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 -- 172.21.15.29:0/3835130675 >> 172.21.15.29:0/3835130675 conn(0x7feacc0ce9b0 msgr2=0x7feacc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:05.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 -- 172.21.15.29:0/3835130675 shutdown_connections 2023-10-16T15:34:05.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.709+0000 7fead31d7640 1 -- 172.21.15.29:0/3835130675 wait complete. 2023-10-16T15:34:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.711+0000 7fead31d7640 1 Processor -- start 2023-10-16T15:34:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.711+0000 7fead31d7640 1 -- start start 2023-10-16T15:34:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.712+0000 7fead31d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0c9ca0 0x7feacc0c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.712+0000 7fead31d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 0x7feacc15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.712+0000 7fead31d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feacc15ad70 0x7feacc158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.712+0000 7fead31d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0c5a80 con 0x7feacc0ca3b0 2023-10-16T15:34:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.712+0000 7fead31d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0c86f0 con 0x7feacc15ad70 2023-10-16T15:34:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.712+0000 7fead31d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc15ca80 con 0x7feacc0c9ca0 2023-10-16T15:34:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 0x7feacc15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 0x7feacc15dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60014/0 (socket says 172.21.15.29:60014) 2023-10-16T15:34:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 -- 172.21.15.29:0/2788695825 learned_addr learned my addr 172.21.15.29:0/2788695825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 -- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0c9ca0 msgr2=0x7feacc0c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7fead0f73640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0c9ca0 0x7feacc0c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7fead1774640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feacc15ad70 0x7feacc158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0c9ca0 0x7feacc0c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 -- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feacc15ad70 msgr2=0x7feacc158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feacc15ad70 0x7feacc158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7feacbfff640 1 -- 172.21.15.29:0/2788695825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac00094d0 con 0x7feacc0ca3b0 2023-10-16T15:34:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.713+0000 7fead1774640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feacc15ad70 0x7feacc158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.714+0000 7feacbfff640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 0x7feacc15dd70 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7feab800d800 tx=0x7feab800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.714+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab80096a0 con 0x7feacc0ca3b0 2023-10-16T15:34:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.714+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab8009800 con 0x7feacc0ca3b0 2023-10-16T15:34:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.714+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8009b90 con 0x7feacc0ca3b0 2023-10-16T15:34:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.714+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc15cd60 con 0x7feacc0ca3b0 2023-10-16T15:34:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.714+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc15d320 con 0x7feacc0ca3b0 2023-10-16T15:34:05.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.715+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc04e160 con 0x7feacc0ca3b0 2023-10-16T15:34:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.716+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7feab8009cf0 con 0x7feacc0ca3b0 2023-10-16T15:34:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.717+0000 7feac97fa640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feab006af90 0x7feab006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.717+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7feab808a150 con 0x7feacc0ca3b0 2023-10-16T15:34:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.717+0000 7fead0f73640 1 -- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feab006af90 msgr2=0x7feab006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.717+0000 7fead0f73640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feab006af90 0x7feab006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.721+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feab8054ee0 con 0x7feacc0ca3b0 2023-10-16T15:34:05.846 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:05.843+0000 7fbf1b811040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:34:05.848 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.846+0000 7f70ed22b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:34:05.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.914+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feacc0c4df0 con 0x7feacc0ca3b0 2023-10-16T15:34:05.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.915+0000 7feac97fa640 1 -- 172.21.15.29:0/2788695825 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v180) v1 ==== 100+0+11 (secure 0 0 0) 0x7feab800b440 con 0x7feacc0ca3b0 2023-10-16T15:34:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:34:05.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.917+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feab006af90 msgr2=0x7feab006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:05.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.917+0000 7fead31d7640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feab006af90 0x7feab006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.917+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 msgr2=0x7feacc15dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:05.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.917+0000 7fead31d7640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 0x7feacc15dd70 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7feab800d800 tx=0x7feab800dcc0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 shutdown_connections 2023-10-16T15:34:05.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0ca3b0 0x7feacc15dd70 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7feab006af90 0x7feab006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feacc15ad70 0x7feacc158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 --2- 172.21.15.29:0/2788695825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feacc0c9ca0 0x7feacc0c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:05.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 >> 172.21.15.29:0/2788695825 conn(0x7feacc0ce9b0 msgr2=0x7feacc06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:05.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 shutdown_connections 2023-10-16T15:34:05.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:05.918+0000 7fead31d7640 1 -- 172.21.15.29:0/2788695825 wait complete. 2023-10-16T15:34:05.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:34:05.954 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:05.952+0000 7f70ed22b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:34:06.003 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:06.002+0000 7f70ed22b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:34:06.016 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.014+0000 7fbf1b811040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:34:06.110 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:06.108+0000 7f70ed22b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:06.131 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.129+0000 7fbf1b811040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:34:06.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.145+0000 7fea71275640 1 Processor -- start 2023-10-16T15:34:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.145+0000 7fea71275640 1 -- start start 2023-10-16T15:34:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.146+0000 7fea71275640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea6c15ad70 0x7fea6c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.146+0000 7fea71275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c0c9ca0 0x7fea6c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea71275640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fea6c0ca3b0 0x7fea6c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea71275640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c0cad80 con 0x7fea6c15ad70 2023-10-16T15:34:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea71275640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c0cb030 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea71275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0c91c0 con 0x7fea6c0c9ca0 2023-10-16T15:34:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea6c15ad70 0x7fea6c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fea6c15ad70 0x7fea6c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60024/0 (socket says 172.21.15.29:60024) 2023-10-16T15:34:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 -- 172.21.15.29:0/618589237 learned_addr learned my addr 172.21.15.29:0/618589237 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:06.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 -- 172.21.15.29:0/618589237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c0c9ca0 msgr2=0x7fea6c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:06.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 --2- 172.21.15.29:0/618589237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c0c9ca0 0x7fea6c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 -- 172.21.15.29:0/618589237 >> v2:172.21.15.170:3300/0 conn(0x7fea6c0ca3b0 msgr2=0x7fea6c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 --2- 172.21.15.29:0/618589237 >> v2:172.21.15.170:3300/0 conn(0x7fea6c0ca3b0 0x7fea6c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.147+0000 7fea6affd640 1 -- 172.21.15.29:0/618589237 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c0c94a0 con 0x7fea6c15ad70 2023-10-16T15:34:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.148+0000 7fea6affd640 1 --2- 172.21.15.29:0/618589237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c15ad70 0x7fea6c069da0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0cf820 tx=0x7fea5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e47e228a997d33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.149+0000 7fea69ffb640 1 -- 172.21.15.29:0/618589237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c031870 con 0x7fea6c15ad70 2023-10-16T15:34:06.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.149+0000 7fea69ffb640 1 -- 172.21.15.29:0/618589237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c0319d0 con 0x7fea6c15ad70 2023-10-16T15:34:06.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.149+0000 7fea69ffb640 1 -- 172.21.15.29:0/618589237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c031d60 con 0x7fea6c15ad70 2023-10-16T15:34:06.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 -- 172.21.15.29:0/618589237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c15ad70 msgr2=0x7fea6c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 --2- 172.21.15.29:0/618589237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c15ad70 0x7fea6c069da0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0cf820 tx=0x7fea5c02b910 comp rx=0 tx=0).stop 2023-10-16T15:34:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 -- 172.21.15.29:0/618589237 shutdown_connections 2023-10-16T15:34:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 --2- 172.21.15.29:0/618589237 >> v2:172.21.15.170:3300/0 conn(0x7fea6c0ca3b0 0x7fea6c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 --2- 172.21.15.29:0/618589237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c0c9ca0 0x7fea6c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 --2- 172.21.15.29:0/618589237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c15ad70 0x7fea6c069da0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 -- 172.21.15.29:0/618589237 >> 172.21.15.29:0/618589237 conn(0x7fea6c0ce9b0 msgr2=0x7fea6c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 -- 172.21.15.29:0/618589237 shutdown_connections 2023-10-16T15:34:06.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.150+0000 7fea71275640 1 -- 172.21.15.29:0/618589237 wait complete. 2023-10-16T15:34:06.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.151+0000 7fea71275640 1 Processor -- start 2023-10-16T15:34:06.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.152+0000 7fea71275640 1 -- start start 2023-10-16T15:34:06.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.152+0000 7fea71275640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea6c0c9ca0 0x7fea6c154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea71275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 0x7fea6c153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea71275640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c15ad70 0x7fea6c1f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea71275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c1f1250 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea71275640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0b0640 con 0x7fea6c0c9ca0 2023-10-16T15:34:06.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea71275640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c153bb0 con 0x7fea6c15ad70 2023-10-16T15:34:06.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 0x7fea6c153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 0x7fea6c153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60026/0 (socket says 172.21.15.29:60026) 2023-10-16T15:34:06.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 -- 172.21.15.29:0/421192067 learned_addr learned my addr 172.21.15.29:0/421192067 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:06.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 -- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c15ad70 msgr2=0x7fea6c1f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6affd640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea6c0c9ca0 0x7fea6c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6b7fe640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c15ad70 0x7fea6c1f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c15ad70 0x7fea6c1f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 -- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea6c0c9ca0 msgr2=0x7fea6c154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea6c0c9ca0 0x7fea6c154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6a7fc640 1 -- 172.21.15.29:0/421192067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c0094d0 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6b7fe640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c15ad70 0x7fea6c1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:06.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.153+0000 7fea6affd640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea6c0c9ca0 0x7fea6c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:06.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.154+0000 7fea6a7fc640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 0x7fea6c153670 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fea6000b4d0 tx=0x7fea6000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:06.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.154+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60010870 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.154+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea600109d0 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.154+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60010d80 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.154+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c1f0fb0 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.154+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c0af0c0 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.156+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c04e160 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.156+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fea6000f3c0 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.157+0000 7fea537fe640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 0x7fea5806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.157+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fea60089830 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.158+0000 7fea6affd640 1 -- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 msgr2=0x7fea5806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:06.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.158+0000 7fea6affd640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 0x7fea5806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:06.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.161+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea60054640 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.186 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.184+0000 7fbf1b811040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:34:06.216 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:06.214+0000 7f70ed22b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:34:06.300 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.298+0000 7fbf1b811040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:06.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.359+0000 7fea6affd640 1 -- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 msgr2=0x7fea5806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:06.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.359+0000 7fea6affd640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 0x7fea5806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:06.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.365+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fea6c0af950 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.366+0000 7fea537fe640 1 -- 172.21.15.29:0/421192067 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v180) v1 ==== 80+0+477 (secure 0 0 0) 0x7fea60057c60 con 0x7fea6c0ca3b0 2023-10-16T15:34:06.369 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2172765430 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.369 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2009585372 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.369 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/838551000 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.369 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/1981883088 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/1981883088 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3011898787 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2177136748 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2616191068 2023-10-17T15:33:59.835992+0000 2023-10-16T15:34:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T15:34:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.368+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 msgr2=0x7fea5806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.368+0000 7fea71275640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 0x7fea5806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.368+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 msgr2=0x7fea6c153670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.368+0000 7fea71275640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 0x7fea6c153670 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fea6000b4d0 tx=0x7fea6000b990 comp rx=0 tx=0).stop 2023-10-16T15:34:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 shutdown_connections 2023-10-16T15:34:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea6c0ca3b0 0x7fea6c153670 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fea5806afc0 0x7fea5806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fea6c15ad70 0x7fea6c1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 --2- 172.21.15.29:0/421192067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fea6c0c9ca0 0x7fea6c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 >> 172.21.15.29:0/421192067 conn(0x7fea6c0ce9b0 msgr2=0x7fea6c0c8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 shutdown_connections 2023-10-16T15:34:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.369+0000 7fea71275640 1 -- 172.21.15.29:0/421192067 wait complete. 2023-10-16T15:34:06.374 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:06.369+0000 7f70ed22b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:06.388 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:34:06.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:06.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:34:06.416 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.413+0000 7fbf1b811040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:34:06.419 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:06.418+0000 7f70ed22b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:06.490 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:34:06.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:06.491 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:34:06.537 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:06.535+0000 7f70ed22b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:34:06.582 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:34:06.582 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:06.585 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.582+0000 7fbf1b811040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:06.636 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:34:06.637 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:34:06.637 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:06.638 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.635+0000 7fbf1b811040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:06.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:34:06.693 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:34:06.693 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:06.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:34:06.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:06.771 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:06.768+0000 7fbf1b811040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:34:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.954+0000 7fd7e3ac2640 1 Processor -- start 2023-10-16T15:34:06.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.954+0000 7fd7e3ac2640 1 -- start start 2023-10-16T15:34:06.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e3ac2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15aaa0 0x7fd7dc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e3ac2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7dc15ae80 0x7fd7dc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e3ac2640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc15ed60 con 0x7fd7dc0ccaa0 2023-10-16T15:34:06.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e3ac2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7dc06b810 con 0x7fd7dc15ae80 2023-10-16T15:34:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e3ac2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0707e0 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e105d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7dc15ae80 0x7fd7dc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e105d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7dc15ae80 0x7fd7dc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60028/0 (socket says 172.21.15.29:60028) 2023-10-16T15:34:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.956+0000 7fd7e105d640 1 -- 172.21.15.29:0/37355595 learned_addr learned my addr 172.21.15.29:0/37355595 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.957+0000 7fd7e185e640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15aaa0 0x7fd7dc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.957+0000 7fd7e105d640 1 -- 172.21.15.29:0/37355595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15aaa0 msgr2=0x7fd7dc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.957+0000 7fd7e105d640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15aaa0 0x7fd7dc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.957+0000 7fd7e105d640 1 -- 172.21.15.29:0/37355595 >> v1:172.21.15.170:6789/0 conn(0x7fd7dc0ccaa0 legacy=0x7fd7dc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.957+0000 7fd7e105d640 1 -- 172.21.15.29:0/37355595 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc15ef90 con 0x7fd7dc15ae80 2023-10-16T15:34:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.957+0000 7fd7e185e640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15aaa0 0x7fd7dc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.958+0000 7fd7e105d640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15ae80 0x7fd7dc0cc560 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009870 tx=0x7fd7d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca767b2b2749fb98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.958+0000 7fd7e085c640 1 -- 172.21.15.29:0/37355595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0031640 con 0x7fd7dc15ae80 2023-10-16T15:34:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.958+0000 7fd7e085c640 1 -- 172.21.15.29:0/37355595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d00317a0 con 0x7fd7dc15ae80 2023-10-16T15:34:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.958+0000 7fd7e085c640 1 -- 172.21.15.29:0/37355595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0031b30 con 0x7fd7dc15ae80 2023-10-16T15:34:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/37355595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15ae80 msgr2=0x7fd7dc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15ae80 0x7fd7dc0cc560 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009870 tx=0x7fd7d002b8f0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/37355595 shutdown_connections 2023-10-16T15:34:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15ae80 0x7fd7dc0cc560 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/37355595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15aaa0 0x7fd7dc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/37355595 >> 172.21.15.29:0/37355595 conn(0x7fd7dc0cb410 msgr2=0x7fd7dc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/37355595 shutdown_connections 2023-10-16T15:34:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.959+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/37355595 wait complete. 2023-10-16T15:34:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.961+0000 7fd7e3ac2640 1 Processor -- start 2023-10-16T15:34:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.961+0000 7fd7e3ac2640 1 -- start start 2023-10-16T15:34:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e3ac2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7dc0ccaa0 0x7fd7dc0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e3ac2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 0x7fd7dc159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e3ac2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15ae80 0x7fd7dc0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e3ac2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0c41e0 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e3ac2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0c89a0 con 0x7fd7dc0ccaa0 2023-10-16T15:34:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e3ac2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc1597a0 con 0x7fd7dc15ae80 2023-10-16T15:34:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e105d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 0x7fd7dc159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e105d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 0x7fd7dc159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60030/0 (socket says 172.21.15.29:60030) 2023-10-16T15:34:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e105d640 1 -- 172.21.15.29:0/1899559955 learned_addr learned my addr 172.21.15.29:0/1899559955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.962+0000 7fd7e205f640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15ae80 0x7fd7dc0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e185e640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7dc0ccaa0 0x7fd7dc0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e105d640 1 -- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15ae80 msgr2=0x7fd7dc0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e105d640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15ae80 0x7fd7dc0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e105d640 1 -- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7dc0ccaa0 msgr2=0x7fd7dc0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e105d640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7dc0ccaa0 0x7fd7dc0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e105d640 1 -- 172.21.15.29:0/1899559955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d00094d0 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e105d640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 0x7fd7dc159260 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fd7d00090b0 tx=0x7fd7d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d003bc70 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d003bdd0 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc0c3840 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.963+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0045630 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.964+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc0c3e00 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.965+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc0c4320 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.966+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd7d002f050 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.967+0000 7fd7da7fc640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 0x7fd7c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.967+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd7d00abad0 con 0x7fd7dc15aaa0 2023-10-16T15:34:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.968+0000 7fd7e185e640 1 -- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 msgr2=0x7fd7c806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.968+0000 7fd7e185e640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 0x7fd7c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:06.972+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7dc0c4320 con 0x7fd7dc15aaa0 2023-10-16T15:34:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.169+0000 7fd7e185e640 1 -- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 msgr2=0x7fd7c806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.169+0000 7fd7e185e640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 0x7fd7c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.199+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7dc0c4320 con 0x7fd7dc15aaa0 2023-10-16T15:34:07.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.201+0000 7fd7da7fc640 1 -- 172.21.15.29:0/1899559955 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4917 (secure 0 0 0) 0x7fd7dc0c4320 con 0x7fd7dc15aaa0 2023-10-16T15:34:07.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:07.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:33:45.081584+0000","modified":"2023-10-16T15:33:48.236102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9342":{"gid":9342,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3104953668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3104953668},{"type":"v1","addr":"172.21.15.29:6833","nonce":3104953668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T15:34:07.204 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 84 2023-10-16T15:34:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 msgr2=0x7fd7c806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 0x7fd7c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 msgr2=0x7fd7dc159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 0x7fd7dc159260 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fd7d00090b0 tx=0x7fd7d00046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 shutdown_connections 2023-10-16T15:34:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7dc15aaa0 0x7fd7dc159260 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7fd7c806af90 0x7fd7c806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7dc15ae80 0x7fd7dc0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 --2- 172.21.15.29:0/1899559955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7dc0ccaa0 0x7fd7dc0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 >> 172.21.15.29:0/1899559955 conn(0x7fd7dc0cb410 msgr2=0x7fd7dc0b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 shutdown_connections 2023-10-16T15:34:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.203+0000 7fd7e3ac2640 1 -- 172.21.15.29:0/1899559955 wait complete. 2023-10-16T15:34:07.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:07.379 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.377+0000 7f70ed22b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:34:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.460+0000 7f1aa6897640 1 Processor -- start 2023-10-16T15:34:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.460+0000 7f1aa6897640 1 -- start start 2023-10-16T15:34:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.462+0000 7f1aa6897640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.462+0000 7f1aa6897640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1aa015e0c0 0x7f1aa00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.462+0000 7f1aa6897640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00cd630 con 0x7f1aa015e540 2023-10-16T15:34:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.462+0000 7f1aa6897640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00ce1b0 con 0x7f1aa015e0c0 2023-10-16T15:34:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.462+0000 7f1aa6897640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa006cd10 con 0x7f1aa006c3f0 2023-10-16T15:34:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.462+0000 7f1a97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1aa015e0c0 0x7f1aa00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1aa015e0c0 0x7f1aa00b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60044/0 (socket says 172.21.15.29:60044) 2023-10-16T15:34:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a97fff640 1 -- 172.21.15.29:0/3775399469 learned_addr learned my addr 172.21.15.29:0/3775399469 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a9ffff640 1 --2- 172.21.15.29:0/3775399469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a97fff640 1 -- 172.21.15.29:0/3775399469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 msgr2=0x7f1aa006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a97fff640 1 --2- 172.21.15.29:0/3775399469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a97fff640 1 -- 172.21.15.29:0/3775399469 >> v1:172.21.15.170:6789/0 conn(0x7f1aa015e540 legacy=0x7f1aa00b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.463+0000 7f1a97fff640 1 -- 172.21.15.29:0/3775399469 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00d2170 con 0x7f1aa015e0c0 2023-10-16T15:34:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.464+0000 7f1a97fff640 1 --2- 172.21.15.29:0/3775399469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e0c0 0x7f1aa00b4730 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009870 tx=0x7f1a8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23037e08e8bc28b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.465+0000 7f1a9f7fe640 1 -- 172.21.15.29:0/3775399469 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88030840 con 0x7f1aa015e0c0 2023-10-16T15:34:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.465+0000 7f1a9f7fe640 1 -- 172.21.15.29:0/3775399469 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a880309a0 con 0x7f1aa015e0c0 2023-10-16T15:34:07.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.465+0000 7f1a9f7fe640 1 -- 172.21.15.29:0/3775399469 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88030d30 con 0x7f1aa015e0c0 2023-10-16T15:34:07.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 -- 172.21.15.29:0/3775399469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e0c0 msgr2=0x7f1aa00b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 --2- 172.21.15.29:0/3775399469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e0c0 0x7f1aa00b4730 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009870 tx=0x7f1a8802b980 comp rx=0 tx=0).stop 2023-10-16T15:34:07.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 -- 172.21.15.29:0/3775399469 shutdown_connections 2023-10-16T15:34:07.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 --2- 172.21.15.29:0/3775399469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e0c0 0x7f1aa00b4730 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 --2- 172.21.15.29:0/3775399469 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 -- 172.21.15.29:0/3775399469 >> 172.21.15.29:0/3775399469 conn(0x7f1aa00d2cb0 msgr2=0x7f1aa00d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:07.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 -- 172.21.15.29:0/3775399469 shutdown_connections 2023-10-16T15:34:07.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.466+0000 7f1aa6897640 1 -- 172.21.15.29:0/3775399469 wait complete. 2023-10-16T15:34:07.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.468+0000 7f1aa6897640 1 Processor -- start 2023-10-16T15:34:07.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.468+0000 7f1aa6897640 1 -- start start 2023-10-16T15:34:07.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa6897640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa01f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa6897640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1aa015e0c0 0x7f1aa0154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa6897640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 0x7f1aa015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa6897640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa01f3760 con 0x7f1aa015e540 2023-10-16T15:34:07.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa6897640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa01552d0 con 0x7f1aa015e0c0 2023-10-16T15:34:07.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa6897640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00c7b40 con 0x7f1aa006c3f0 2023-10-16T15:34:07.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa4e34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 0x7f1aa015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa4e34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 0x7f1aa015bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60060/0 (socket says 172.21.15.29:60060) 2023-10-16T15:34:07.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1aa4e34640 1 -- 172.21.15.29:0/1979735962 learned_addr learned my addr 172.21.15.29:0/1979735962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:07.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.469+0000 7f1a9ffff640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa01f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1a97fff640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1aa015e0c0 0x7f1aa0154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1aa4e34640 1 -- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 msgr2=0x7f1aa01f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1aa4e34640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa01f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1aa4e34640 1 -- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1aa015e0c0 msgr2=0x7f1aa0154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1aa4e34640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1aa015e0c0 0x7f1aa0154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1aa4e34640 1 -- 172.21.15.29:0/1979735962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880094d0 con 0x7f1aa015e540 2023-10-16T15:34:07.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.470+0000 7f1aa4e34640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 0x7f1aa015bff0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800c730 tx=0x7f1a9800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:07.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.471+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98003b70 con 0x7f1aa015e540 2023-10-16T15:34:07.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.471+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a98003cd0 con 0x7f1aa015e540 2023-10-16T15:34:07.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.471+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800aa90 con 0x7f1aa015e540 2023-10-16T15:34:07.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.471+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa00c7e20 con 0x7f1aa015e540 2023-10-16T15:34:07.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.471+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa00c83e0 con 0x7f1aa015e540 2023-10-16T15:34:07.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.472+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa004e160 con 0x7f1aa015e540 2023-10-16T15:34:07.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.473+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1a9800abf0 con 0x7f1aa015e540 2023-10-16T15:34:07.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.474+0000 7f1a9d7fa640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 0x7f1a8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.474+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1a98089670 con 0x7f1aa015e540 2023-10-16T15:34:07.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.474+0000 7f1a9ffff640 1 -- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 msgr2=0x7f1a8006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:07.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.474+0000 7f1a9ffff640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 0x7f1a8006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:07.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.478+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a98054480 con 0x7f1aa015e540 2023-10-16T15:34:07.625 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:07.623+0000 7fbf1b811040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:34:07.670 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.668+0000 7f70ed22b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:34:07.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.675+0000 7f1a9ffff640 1 -- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 msgr2=0x7f1a8006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3772833058 2023-10-16T15:34:07.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.675+0000 7f1a9ffff640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 0x7f1a8006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.683+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aa00d0c80 con 0x7f1aa015e540 2023-10-16T15:34:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.684+0000 7f1a9d7fa640 1 -- 172.21.15.29:0/1979735962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17131 (secure 0 0 0) 0x7f1a98057aa0 con 0x7f1aa015e540 2023-10-16T15:34:07.687 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:07.688 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":180,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:33:59.836071+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2172765430":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2009585372":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/838551000":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6833/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6832/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3011898787":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2177136748":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2616191068":"2023-10-17T15:33:59.835992+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.687+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 msgr2=0x7f1a8006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.687+0000 7f1aa6897640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 0x7f1a8006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.687+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 msgr2=0x7f1aa015bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.687+0000 7f1aa6897640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 0x7f1aa015bff0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800c730 tx=0x7f1a9800cbf0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 shutdown_connections 2023-10-16T15:34:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aa015e540 0x7f1aa015bff0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:6832/3772833058,v1:172.21.15.170:6833/3772833058] conn(0x7f1a8006af90 0x7f1a8006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1aa015e0c0 0x7f1aa0154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 --2- 172.21.15.29:0/1979735962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1aa006c3f0 0x7f1aa01f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 >> 172.21.15.29:0/1979735962 conn(0x7f1aa00d2cb0 msgr2=0x7f1aa00b81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 shutdown_connections 2023-10-16T15:34:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.688+0000 7f1aa6897640 1 -- 172.21.15.29:0/1979735962 wait complete. 2023-10-16T15:34:07.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:33:43.022528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:33:44.097016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:34:07.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:34:07.725 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.724+0000 7f70ed22b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:34:07.840 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.837+0000 7f70ed22b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:34:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.932+0000 7ff219ad0640 1 Processor -- start 2023-10-16T15:34:07.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.933+0000 7ff219ad0640 1 -- start start 2023-10-16T15:34:07.936 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.933+0000 7f706ee58640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.936 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.933+0000 7f706ee58640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.936 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.933+0000 7f706ee58640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.937 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.933+0000 7f706ee58640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.937 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.933+0000 7f706ee58640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.937 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.934+0000 7f707165d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.937 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.934+0000 7f707165d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.937 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.934+0000 7f707165d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.938 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.934+0000 7f707165d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.938 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:07.934+0000 7f707165d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:07.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.934+0000 7ff219ad0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff21406a4b0 0x7ff2140ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.934+0000 7ff219ad0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 0x7ff2140ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.934+0000 7ff219ad0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2140cb030 con 0x7ff21415ad70 2023-10-16T15:34:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.934+0000 7ff219ad0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140cae30 con 0x7ff21406a4b0 2023-10-16T15:34:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.934+0000 7ff219ad0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140c91c0 con 0x7ff2140c9ca0 2023-10-16T15:34:07.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff20bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff21406a4b0 0x7ff2140ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff213fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 0x7ff2140ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff213fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 0x7ff2140ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:52170/0 (socket says 172.21.15.29:52170) 2023-10-16T15:34:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff20bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff21406a4b0 0x7ff2140ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60064/0 (socket says 172.21.15.29:60064) 2023-10-16T15:34:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff213fff640 1 -- 172.21.15.29:0/2902631108 learned_addr learned my addr 172.21.15.29:0/2902631108 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:07.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff2137fe640 1 --1- 172.21.15.29:0/2902631108 >> v1:172.21.15.170:6789/0 conn(0x7ff21415ad70 0x7ff214069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:49458/0 (socket says 172.21.15.29:49458) 2023-10-16T15:34:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff20bfff640 1 -- 172.21.15.29:0/2902631108 >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 msgr2=0x7ff2140ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff20bfff640 1 --2- 172.21.15.29:0/2902631108 >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 0x7ff2140ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff20bfff640 1 -- 172.21.15.29:0/2902631108 >> v1:172.21.15.170:6789/0 conn(0x7ff21415ad70 legacy=0x7ff214069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff20bfff640 1 -- 172.21.15.29:0/2902631108 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140c94a0 con 0x7ff21406a4b0 2023-10-16T15:34:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.935+0000 7ff213fff640 1 --2- 172.21.15.29:0/2902631108 >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 0x7ff2140ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:34:07.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.937+0000 7ff20bfff640 1 --2- 172.21.15.29:0/2902631108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff21406a4b0 0x7ff2140ca3b0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7ff21406f270 tx=0x7ff1fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46bddcf99af6dbc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:07.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.937+0000 7ff212ffd640 1 -- 172.21.15.29:0/2902631108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc031830 con 0x7ff21406a4b0 2023-10-16T15:34:07.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.937+0000 7ff212ffd640 1 -- 172.21.15.29:0/2902631108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc031990 con 0x7ff21406a4b0 2023-10-16T15:34:07.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.937+0000 7ff212ffd640 1 -- 172.21.15.29:0/2902631108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc031d20 con 0x7ff21406a4b0 2023-10-16T15:34:07.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.938+0000 7ff219ad0640 1 -- 172.21.15.29:0/2902631108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff21406a4b0 msgr2=0x7ff2140ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.938+0000 7ff219ad0640 1 --2- 172.21.15.29:0/2902631108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff21406a4b0 0x7ff2140ca3b0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7ff21406f270 tx=0x7ff1fc02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.938+0000 7ff219ad0640 1 -- 172.21.15.29:0/2902631108 shutdown_connections 2023-10-16T15:34:07.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.938+0000 7ff219ad0640 1 --2- 172.21.15.29:0/2902631108 >> v2:172.21.15.170:3300/0 conn(0x7ff2140c9ca0 0x7ff2140ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.938+0000 7ff219ad0640 1 --2- 172.21.15.29:0/2902631108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff21406a4b0 0x7ff2140ca3b0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.938+0000 7ff219ad0640 1 -- 172.21.15.29:0/2902631108 >> 172.21.15.29:0/2902631108 conn(0x7ff2140ce9b0 msgr2=0x7ff2140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:07.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.939+0000 7ff219ad0640 1 -- 172.21.15.29:0/2902631108 shutdown_connections 2023-10-16T15:34:07.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.939+0000 7ff219ad0640 1 -- 172.21.15.29:0/2902631108 wait complete. 2023-10-16T15:34:07.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.941+0000 7ff219ad0640 1 Processor -- start 2023-10-16T15:34:07.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.942+0000 7ff219ad0640 1 -- start start 2023-10-16T15:34:07.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.943+0000 7ff219ad0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 0x7ff21415b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.943+0000 7ff219ad0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2140c9ca0 0x7ff2140b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.943+0000 7ff219ad0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff21415ad70 0x7ff2140c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:07.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.943+0000 7ff219ad0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21415bd80 con 0x7ff2140c9ca0 2023-10-16T15:34:07.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.943+0000 7ff219ad0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21415c060 con 0x7ff21406a4b0 2023-10-16T15:34:07.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.943+0000 7ff219ad0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21415c340 con 0x7ff21415ad70 2023-10-16T15:34:07.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 0x7ff21415b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2140c9ca0 0x7ff2140b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 0x7ff21415b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:52186/0 (socket says 172.21.15.29:52186) 2023-10-16T15:34:07.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 -- 172.21.15.29:0/3283123929 learned_addr learned my addr 172.21.15.29:0/3283123929 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:07.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff213fff640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff21415ad70 0x7ff2140c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:07.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 -- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff21415ad70 msgr2=0x7ff2140c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff21415ad70 0x7ff2140c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 -- 172.21.15.29:0/3283123929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2140c9ca0 msgr2=0x7ff2140b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:07.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2140c9ca0 0x7ff2140b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 -- 172.21.15.29:0/3283123929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c009b30 con 0x7ff21406a4b0 2023-10-16T15:34:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff20bfff640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2140c9ca0 0x7ff2140b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:34:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.944+0000 7ff2137fe640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 0x7ff21415b480 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff20400c3e0 tx=0x7ff20400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.945+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040074e0 con 0x7ff21406a4b0 2023-10-16T15:34:07.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.945+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204007640 con 0x7ff21406a4b0 2023-10-16T15:34:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.945+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040049e0 con 0x7ff21406a4b0 2023-10-16T15:34:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.945+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc009400 con 0x7ff21406a4b0 2023-10-16T15:34:07.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.945+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21415c620 con 0x7ff21406a4b0 2023-10-16T15:34:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.946+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2140c7a30 con 0x7ff21406a4b0 2023-10-16T15:34:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.947+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff204004b40 con 0x7ff21406a4b0 2023-10-16T15:34:07.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.948+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ff204088cd0 con 0x7ff21406a4b0 2023-10-16T15:34:07.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:07.953+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2140c7a30 con 0x7ff21406a4b0 2023-10-16T15:34:07.957 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:07.955+0000 7fbf1b811040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:34:08.017 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:08.015+0000 7fbf1b811040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:34:08.138 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:08.135+0000 7fbf1b811040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:34:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.155+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2140c7a30 con 0x7ff21406a4b0 2023-10-16T15:34:08.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.157+0000 7ff210ff9640 1 -- 172.21.15.29:0/3283123929 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v181) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff2140c7a30 con 0x7ff21406a4b0 2023-10-16T15:34:08.159 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:34:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.159+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 msgr2=0x7ff21415b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.159+0000 7ff219ad0640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 0x7ff21415b480 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff20400c3e0 tx=0x7ff20400c8a0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.159+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 shutdown_connections 2023-10-16T15:34:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.159+0000 7ff219ad0640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2140c9ca0 0x7ff2140b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.159+0000 7ff219ad0640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff21415ad70 0x7ff2140c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.159+0000 7ff219ad0640 1 --2- 172.21.15.29:0/3283123929 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff21406a4b0 0x7ff21415b480 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.160+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 >> 172.21.15.29:0/3283123929 conn(0x7ff2140ce9b0 msgr2=0x7ff2140c44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.160+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 shutdown_connections 2023-10-16T15:34:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.160+0000 7ff219ad0640 1 -- 172.21.15.29:0/3283123929 wait complete. 2023-10-16T15:34:08.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:34:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.388+0000 7f793010f640 1 Processor -- start 2023-10-16T15:34:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.388+0000 7f793010f640 1 -- start start 2023-10-16T15:34:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f793010f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f792815ad70 0x7f7928069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f793010f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79280c9ca0 0x7f792806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f793010f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79280cb030 con 0x7f79280ca3b0 2023-10-16T15:34:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f793010f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280c91c0 con 0x7f79280c9ca0 2023-10-16T15:34:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f793010f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280c94a0 con 0x7f792815ad70 2023-10-16T15:34:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f792d6aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79280c9ca0 0x7f792806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f792d6aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79280c9ca0 0x7f792806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60086/0 (socket says 172.21.15.29:60086) 2023-10-16T15:34:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.390+0000 7f792d6aa640 1 -- 172.21.15.29:0/2771284026 learned_addr learned my addr 172.21.15.29:0/2771284026 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.391+0000 7f792deab640 1 --2- 172.21.15.29:0/2771284026 >> v2:172.21.15.170:3300/0 conn(0x7f792815ad70 0x7f7928069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.391+0000 7f792d6aa640 1 -- 172.21.15.29:0/2771284026 >> v2:172.21.15.170:3300/0 conn(0x7f792815ad70 msgr2=0x7f7928069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.391+0000 7f792d6aa640 1 --2- 172.21.15.29:0/2771284026 >> v2:172.21.15.170:3300/0 conn(0x7f792815ad70 0x7f7928069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.391+0000 7f792d6aa640 1 -- 172.21.15.29:0/2771284026 >> v1:172.21.15.29:6789/0 conn(0x7f79280ca3b0 legacy=0x7f79280ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.391+0000 7f792d6aa640 1 -- 172.21.15.29:0/2771284026 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280c9780 con 0x7f79280c9ca0 2023-10-16T15:34:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.392+0000 7f792d6aa640 1 --2- 172.21.15.29:0/2771284026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280c9ca0 0x7f792806a2e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f79280cf780 tx=0x7f791802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f98f0ad655e8b7a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.392+0000 7f792cea9640 1 -- 172.21.15.29:0/2771284026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918030870 con 0x7f79280c9ca0 2023-10-16T15:34:08.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.392+0000 7f792cea9640 1 -- 172.21.15.29:0/2771284026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180309d0 con 0x7f79280c9ca0 2023-10-16T15:34:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.392+0000 7f792cea9640 1 -- 172.21.15.29:0/2771284026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918030d60 con 0x7f79280c9ca0 2023-10-16T15:34:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 -- 172.21.15.29:0/2771284026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280c9ca0 msgr2=0x7f792806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 --2- 172.21.15.29:0/2771284026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280c9ca0 0x7f792806a2e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f79280cf780 tx=0x7f791802b9b0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 -- 172.21.15.29:0/2771284026 shutdown_connections 2023-10-16T15:34:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 --2- 172.21.15.29:0/2771284026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280c9ca0 0x7f792806a2e0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 --2- 172.21.15.29:0/2771284026 >> v2:172.21.15.170:3300/0 conn(0x7f792815ad70 0x7f7928069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 -- 172.21.15.29:0/2771284026 >> 172.21.15.29:0/2771284026 conn(0x7f79280ce9b0 msgr2=0x7f79280cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 -- 172.21.15.29:0/2771284026 shutdown_connections 2023-10-16T15:34:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.393+0000 7f793010f640 1 -- 172.21.15.29:0/2771284026 wait complete. 2023-10-16T15:34:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.395+0000 7f793010f640 1 Processor -- start 2023-10-16T15:34:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.395+0000 7f793010f640 1 -- start start 2023-10-16T15:34:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f793010f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79280c9ca0 0x7f792815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f793010f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79280ca3b0 0x7f792815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f793010f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 0x7f792815c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f793010f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815b7c0 con 0x7f792815ad70 2023-10-16T15:34:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f793010f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815cf50 con 0x7f79280c9ca0 2023-10-16T15:34:08.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f793010f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79280c5b90 con 0x7f79280ca3b0 2023-10-16T15:34:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.396+0000 7f792e6ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 0x7f792815c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 0x7f792815c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60090/0 (socket says 172.21.15.29:60090) 2023-10-16T15:34:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 -- 172.21.15.29:0/2346905149 learned_addr learned my addr 172.21.15.29:0/2346905149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792d6aa640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79280ca3b0 0x7f792815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792deab640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79280c9ca0 0x7f792815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 -- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79280ca3b0 msgr2=0x7f792815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79280ca3b0 0x7f792815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 -- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79280c9ca0 msgr2=0x7f792815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79280c9ca0 0x7f792815d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 -- 172.21.15.29:0/2346905149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180094d0 con 0x7f792815ad70 2023-10-16T15:34:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792d6aa640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79280ca3b0 0x7f792815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:08.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f792e6ac640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 0x7f792815c8b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f792000c7a0 tx=0x7f792000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:08.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.397+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920003b70 con 0x7f792815ad70 2023-10-16T15:34:08.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.398+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7920003cf0 con 0x7f792815ad70 2023-10-16T15:34:08.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.398+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79280c5e70 con 0x7f792815ad70 2023-10-16T15:34:08.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.398+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920003ea0 con 0x7f792815ad70 2023-10-16T15:34:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.398+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79280c6430 con 0x7f792815ad70 2023-10-16T15:34:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.399+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79280cb220 con 0x7f792815ad70 2023-10-16T15:34:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.400+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f792000aa90 con 0x7f792815ad70 2023-10-16T15:34:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.401+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7920089280 con 0x7f792815ad70 2023-10-16T15:34:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.405+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7920054470 con 0x7f792815ad70 2023-10-16T15:34:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.597+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f792804e160 con 0x7f792815ad70 2023-10-16T15:34:08.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.598+0000 7f791e7fc640 1 -- 172.21.15.29:0/2346905149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v181) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7920057a90 con 0x7f792815ad70 2023-10-16T15:34:08.601 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:34:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 msgr2=0x7f792815c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 0x7f792815c8b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f792000c7a0 tx=0x7f792000cc60 comp rx=0 tx=0).stop 2023-10-16T15:34:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 shutdown_connections 2023-10-16T15:34:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f792815ad70 0x7f792815c8b0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79280ca3b0 0x7f792815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 --2- 172.21.15.29:0/2346905149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79280c9ca0 0x7f792815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 >> 172.21.15.29:0/2346905149 conn(0x7f79280ce9b0 msgr2=0x7f79280acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 shutdown_connections 2023-10-16T15:34:08.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.601+0000 7f793010f640 1 -- 172.21.15.29:0/2346905149 wait complete. 2023-10-16T15:34:08.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:34:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.846+0000 7f58d6968640 1 Processor -- start 2023-10-16T15:34:08.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.846+0000 7f58d6968640 1 -- start start 2023-10-16T15:34:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.847+0000 7f58d6968640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58d015ad70 0x7f58d0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.847+0000 7f58d6968640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00c9ca0 0x7f58d006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58d6968640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d00cb030 con 0x7f58d00ca3b0 2023-10-16T15:34:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58d6968640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d00c91c0 con 0x7f58d015ad70 2023-10-16T15:34:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58d6968640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00c94a0 con 0x7f58d00c9ca0 2023-10-16T15:34:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58cffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58d015ad70 0x7f58d0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58c77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00c9ca0 0x7f58d006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58cffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58d015ad70 0x7f58d0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:52218/0 (socket says 172.21.15.29:52218) 2023-10-16T15:34:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.848+0000 7f58cffff640 1 -- 172.21.15.29:0/3306731907 learned_addr learned my addr 172.21.15.29:0/3306731907 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.849+0000 7f58cffff640 1 -- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00c9ca0 msgr2=0x7f58d006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.849+0000 7f58cffff640 1 --2- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00c9ca0 0x7f58d006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.849+0000 7f58cffff640 1 -- 172.21.15.29:0/3306731907 >> v1:172.21.15.170:6789/0 conn(0x7f58d00ca3b0 legacy=0x7f58d00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.849+0000 7f58cffff640 1 -- 172.21.15.29:0/3306731907 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00c9780 con 0x7f58d015ad70 2023-10-16T15:34:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.851+0000 7f58cffff640 1 --2- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d015ad70 0x7f58d0069da0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009870 tx=0x7f58c002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=373931793ada3591 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.853+0000 7f58cf7fe640 1 -- 172.21.15.29:0/3306731907 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c0030840 con 0x7f58d015ad70 2023-10-16T15:34:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.853+0000 7f58cf7fe640 1 -- 172.21.15.29:0/3306731907 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c00309a0 con 0x7f58d015ad70 2023-10-16T15:34:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.853+0000 7f58cf7fe640 1 -- 172.21.15.29:0/3306731907 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c0030d30 con 0x7f58d015ad70 2023-10-16T15:34:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 -- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d015ad70 msgr2=0x7f58d0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 --2- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d015ad70 0x7f58d0069da0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009870 tx=0x7f58c002b980 comp rx=0 tx=0).stop 2023-10-16T15:34:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 -- 172.21.15.29:0/3306731907 shutdown_connections 2023-10-16T15:34:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 --2- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00c9ca0 0x7f58d006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 --2- 172.21.15.29:0/3306731907 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d015ad70 0x7f58d0069da0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 -- 172.21.15.29:0/3306731907 >> 172.21.15.29:0/3306731907 conn(0x7f58d00ce9b0 msgr2=0x7f58d00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:08.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 -- 172.21.15.29:0/3306731907 shutdown_connections 2023-10-16T15:34:08.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.854+0000 7f58d6968640 1 -- 172.21.15.29:0/3306731907 wait complete. 2023-10-16T15:34:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.856+0000 7f58d6968640 1 Processor -- start 2023-10-16T15:34:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.856+0000 7f58d6968640 1 -- start start 2023-10-16T15:34:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.857+0000 7f58d6968640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d00c9ca0 0x7f58d00cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.857+0000 7f58d6968640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00ca3b0 0x7f58d00c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.857+0000 7f58d6968640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 0x7f58d014c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.857+0000 7f58d6968640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00c80f0 con 0x7f58d015ad70 2023-10-16T15:34:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.857+0000 7f58d6968640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d014d030 con 0x7f58d00c9ca0 2023-10-16T15:34:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.857+0000 7f58d6968640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d014d310 con 0x7f58d00ca3b0 2023-10-16T15:34:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.858+0000 7f58d4f05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 0x7f58d014c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.858+0000 7f58d4f05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 0x7f58d014c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60092/0 (socket says 172.21.15.29:60092) 2023-10-16T15:34:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.858+0000 7f58d4f05640 1 -- 172.21.15.29:0/1529211072 learned_addr learned my addr 172.21.15.29:0/1529211072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58cffff640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d00c9ca0 0x7f58d00cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58d4f05640 1 -- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00ca3b0 msgr2=0x7f58d00c8580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T15:34:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58c77fe640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00ca3b0 0x7f58d00c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58d4f05640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00ca3b0 0x7f58d00c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58d4f05640 1 -- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d00c9ca0 msgr2=0x7f58d00cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58d4f05640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d00c9ca0 0x7f58d00cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58d4f05640 1 -- 172.21.15.29:0/1529211072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00094d0 con 0x7f58d015ad70 2023-10-16T15:34:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.859+0000 7f58d4f05640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 0x7f58d014c5b0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f58c800c7a0 tx=0x7f58c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:08.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.861+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8003b70 con 0x7f58d015ad70 2023-10-16T15:34:08.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.861+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d00c7d80 con 0x7f58d015ad70 2023-10-16T15:34:08.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.861+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8001830 con 0x7f58d015ad70 2023-10-16T15:34:08.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.861+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d00c7a30 con 0x7f58d015ad70 2023-10-16T15:34:08.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.861+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c800a850 con 0x7f58d015ad70 2023-10-16T15:34:08.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.863+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b4005250 con 0x7f58d015ad70 2023-10-16T15:34:08.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.864+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f58c800aa70 con 0x7f58d015ad70 2023-10-16T15:34:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.865+0000 7f58cd7fa640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f58b006acc0 0x7f58b006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.865+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f58c8089360 con 0x7f58d015ad70 2023-10-16T15:34:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.866+0000 7f58cffff640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f58b006acc0 0x7f58b006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.870+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58c8054400 con 0x7f58d015ad70 2023-10-16T15:34:08.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:08.876+0000 7f58cffff640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f58b006acc0 0x7f58b006d180 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58c0036000 tx=0x7f58c003b040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.055+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58b4005a80 con 0x7f58d015ad70 2023-10-16T15:34:09.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.056+0000 7f58cd7fa640 1 -- 172.21.15.29:0/1529211072 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v181) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58c8057a20 con 0x7f58d015ad70 2023-10-16T15:34:09.059 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:34:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.060+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f58b006acc0 msgr2=0x7f58b006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.060+0000 7f58d6968640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f58b006acc0 0x7f58b006d180 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58c0036000 tx=0x7f58c003b040 comp rx=0 tx=0).stop 2023-10-16T15:34:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.060+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 msgr2=0x7f58d014c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.060+0000 7f58d6968640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 0x7f58d014c5b0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f58c800c7a0 tx=0x7f58c800cc60 comp rx=0 tx=0).stop 2023-10-16T15:34:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 shutdown_connections 2023-10-16T15:34:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f58b006acc0 0x7f58b006d180 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015ad70 0x7f58d014c5b0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58d00ca3b0 0x7f58d00c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 --2- 172.21.15.29:0/1529211072 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58d00c9ca0 0x7f58d00cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 >> 172.21.15.29:0/1529211072 conn(0x7f58d00ce9b0 msgr2=0x7f58d00adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 shutdown_connections 2023-10-16T15:34:09.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.061+0000 7f58d6968640 1 -- 172.21.15.29:0/1529211072 wait complete. 2023-10-16T15:34:09.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.287+0000 7fb78f2a3640 1 Processor -- start 2023-10-16T15:34:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.287+0000 7fb78f2a3640 1 -- start start 2023-10-16T15:34:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.288+0000 7fb78f2a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb78815cbe0 0x7fb7880d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.288+0000 7fb78f2a3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb7880b4c80 0x7fb7880b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.288+0000 7fb78f2a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7880d15a0 con 0x7fb78815cfc0 2023-10-16T15:34:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.288+0000 7fb78f2a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7880d1880 con 0x7fb78815cbe0 2023-10-16T15:34:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.288+0000 7fb78f2a3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7880d1b60 con 0x7fb7880b4c80 2023-10-16T15:34:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb78815cbe0 0x7fb7880d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb78815cbe0 0x7fb7880d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60108/0 (socket says 172.21.15.29:60108) 2023-10-16T15:34:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 -- 172.21.15.29:0/3887033785 learned_addr learned my addr 172.21.15.29:0/3887033785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 -- 172.21.15.29:0/3887033785 >> v2:172.21.15.170:3300/0 conn(0x7fb7880b4c80 msgr2=0x7fb7880b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 --2- 172.21.15.29:0/3887033785 >> v2:172.21.15.170:3300/0 conn(0x7fb7880b4c80 0x7fb7880b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 -- 172.21.15.29:0/3887033785 >> v1:172.21.15.29:6789/0 conn(0x7fb78815cfc0 legacy=0x7fb7880b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.289+0000 7fb78e2a1640 1 -- 172.21.15.29:0/3887033785 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7880d1e40 con 0x7fb78815cbe0 2023-10-16T15:34:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.290+0000 7fb78e2a1640 1 --2- 172.21.15.29:0/3887033785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cbe0 0x7fb7880d0b50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fb7880b5b70 tx=0x7fb78402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308505661837768d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.291+0000 7fb78d29f640 1 -- 172.21.15.29:0/3887033785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784030870 con 0x7fb78815cbe0 2023-10-16T15:34:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.291+0000 7fb78d29f640 1 -- 172.21.15.29:0/3887033785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7840309d0 con 0x7fb78815cbe0 2023-10-16T15:34:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.291+0000 7fb78d29f640 1 -- 172.21.15.29:0/3887033785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784030d60 con 0x7fb78815cbe0 2023-10-16T15:34:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.291+0000 7fb78f2a3640 1 -- 172.21.15.29:0/3887033785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cbe0 msgr2=0x7fb7880d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.291+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/3887033785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cbe0 0x7fb7880d0b50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fb7880b5b70 tx=0x7fb78402b9b0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.292+0000 7fb78f2a3640 1 -- 172.21.15.29:0/3887033785 shutdown_connections 2023-10-16T15:34:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.292+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/3887033785 >> v2:172.21.15.170:3300/0 conn(0x7fb7880b4c80 0x7fb7880b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.292+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/3887033785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cbe0 0x7fb7880d0b50 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.292+0000 7fb78f2a3640 1 -- 172.21.15.29:0/3887033785 >> 172.21.15.29:0/3887033785 conn(0x7fb7880cf570 msgr2=0x7fb7880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.292+0000 7fb78f2a3640 1 -- 172.21.15.29:0/3887033785 shutdown_connections 2023-10-16T15:34:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.292+0000 7fb78f2a3640 1 -- 172.21.15.29:0/3887033785 wait complete. 2023-10-16T15:34:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.293+0000 7fb78f2a3640 1 Processor -- start 2023-10-16T15:34:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.294+0000 7fb78f2a3640 1 -- start start 2023-10-16T15:34:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78f2a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7880b4c80 0x7fb788152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78f2a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb78815cbe0 0x7fb7880c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78f2a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 0x7fb788153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78f2a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788152550 con 0x7fb78815cfc0 2023-10-16T15:34:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78f2a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788153c40 con 0x7fb7880b4c80 2023-10-16T15:34:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78f2a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb788153f20 con 0x7fb78815cbe0 2023-10-16T15:34:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 0x7fb788153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 0x7fb788153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60122/0 (socket says 172.21.15.29:60122) 2023-10-16T15:34:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 -- 172.21.15.29:0/924793753 learned_addr learned my addr 172.21.15.29:0/924793753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78e2a1640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7880b4c80 0x7fb788152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78daa0640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb78815cbe0 0x7fb7880c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 -- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb78815cbe0 msgr2=0x7fb7880c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb78815cbe0 0x7fb7880c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 -- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7880b4c80 msgr2=0x7fb788152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7880b4c80 0x7fb788152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78eaa2640 1 -- 172.21.15.29:0/924793753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7840094d0 con 0x7fb78815cfc0 2023-10-16T15:34:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78e2a1640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7880b4c80 0x7fb788152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.295+0000 7fb78daa0640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb78815cbe0 0x7fb7880c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.296+0000 7fb78eaa2640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 0x7fb788153160 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fb78000b9e0 tx=0x7fb78000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.296+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78000caa0 con 0x7fb78815cfc0 2023-10-16T15:34:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.296+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78000cc20 con 0x7fb78815cfc0 2023-10-16T15:34:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.296+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb788154200 con 0x7fb78815cfc0 2023-10-16T15:34:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.296+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780014440 con 0x7fb78815cfc0 2023-10-16T15:34:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.296+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78815e700 con 0x7fb78815cfc0 2023-10-16T15:34:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.297+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78815e920 con 0x7fb78815cfc0 2023-10-16T15:34:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.299+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb78001d3b0 con 0x7fb78815cfc0 2023-10-16T15:34:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.300+0000 7fb77effd640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb76c06acc0 0x7fb76c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.300+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fb7800894f0 con 0x7fb78815cfc0 2023-10-16T15:34:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.300+0000 7fb78e2a1640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb76c06acc0 0x7fb76c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.302+0000 7fb78e2a1640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb76c06acc0 0x7fb76c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb784036000 tx=0x7fb78403b040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.304+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb780054540 con 0x7fb78815cfc0 2023-10-16T15:34:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.539+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78804e9b0 con 0x7fb78815cfc0 2023-10-16T15:34:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.540+0000 7fb77effd640 1 -- 172.21.15.29:0/924793753 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4917 (secure 0 0 0) 0x7fb780019090 con 0x7fb78815cfc0 2023-10-16T15:34:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:09.543 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:33:45.081584+0000","modified":"2023-10-16T15:33:48.236102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9342":{"gid":9342,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3104953668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3104953668},{"type":"v1","addr":"172.21.15.29:6833","nonce":3104953668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T15:34:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 84 2023-10-16T15:34:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb76c06acc0 msgr2=0x7fb76c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb76c06acc0 0x7fb76c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb784036000 tx=0x7fb78403b040 comp rx=0 tx=0).stop 2023-10-16T15:34:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 msgr2=0x7fb788153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 0x7fb788153160 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fb78000b9e0 tx=0x7fb78000bea0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 shutdown_connections 2023-10-16T15:34:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb76c06acc0 0x7fb76c06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb78815cfc0 0x7fb788153160 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb78815cbe0 0x7fb7880c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 --2- 172.21.15.29:0/924793753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb7880b4c80 0x7fb788152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 >> 172.21.15.29:0/924793753 conn(0x7fb7880cf570 msgr2=0x7fb7880b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 shutdown_connections 2023-10-16T15:34:09.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.543+0000 7fb78f2a3640 1 -- 172.21.15.29:0/924793753 wait complete. 2023-10-16T15:34:09.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:09.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.788+0000 7f42fc35e640 1 Processor -- start 2023-10-16T15:34:09.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.788+0000 7f42fc35e640 1 -- start start 2023-10-16T15:34:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.789+0000 7f42fc35e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f4159f00 0x7f42f40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.789+0000 7f42fc35e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f406f7f0 con 0x7f42f4159b20 2023-10-16T15:34:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.789+0000 7f42fc35e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f406c5e0 con 0x7f42f415d880 2023-10-16T15:34:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.789+0000 7f42fc35e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f406c8c0 con 0x7f42f4159f00 2023-10-16T15:34:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.790+0000 7f42f98f9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f4159f00 0x7f42f40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.790+0000 7f42f98f9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f4159f00 0x7f42f40b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60136/0 (socket says 172.21.15.29:60136) 2023-10-16T15:34:09.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.790+0000 7f42f98f9640 1 -- 172.21.15.29:0/122129435 learned_addr learned my addr 172.21.15.29:0/122129435 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.790+0000 7f42f98f9640 1 -- 172.21.15.29:0/122129435 >> v1:172.21.15.170:6789/0 conn(0x7f42f415d880 legacy=0x7f42f406fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.790+0000 7f42f98f9640 1 -- 172.21.15.29:0/122129435 >> v1:172.21.15.29:6789/0 conn(0x7f42f4159b20 legacy=0x7f42f406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.790+0000 7f42f98f9640 1 -- 172.21.15.29:0/122129435 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f406cba0 con 0x7f42f4159f00 2023-10-16T15:34:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.791+0000 7f42f98f9640 1 --2- 172.21.15.29:0/122129435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40b0410 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009870 tx=0x7f42e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b571b513a5daa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42f90f8640 1 -- 172.21.15.29:0/122129435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4030860 con 0x7f42f4159f00 2023-10-16T15:34:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42f90f8640 1 -- 172.21.15.29:0/122129435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e40309c0 con 0x7f42f4159f00 2023-10-16T15:34:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42f90f8640 1 -- 172.21.15.29:0/122129435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4030d50 con 0x7f42f4159f00 2023-10-16T15:34:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42fc35e640 1 -- 172.21.15.29:0/122129435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 msgr2=0x7f42f40b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42fc35e640 1 --2- 172.21.15.29:0/122129435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40b0410 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009870 tx=0x7f42e402b9a0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42fc35e640 1 -- 172.21.15.29:0/122129435 shutdown_connections 2023-10-16T15:34:09.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42fc35e640 1 --2- 172.21.15.29:0/122129435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40b0410 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42fc35e640 1 -- 172.21.15.29:0/122129435 >> 172.21.15.29:0/122129435 conn(0x7f42f40cb410 msgr2=0x7f42f40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:09.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.792+0000 7f42fc35e640 1 -- 172.21.15.29:0/122129435 shutdown_connections 2023-10-16T15:34:09.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.793+0000 7f42fc35e640 1 -- 172.21.15.29:0/122129435 wait complete. 2023-10-16T15:34:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.794+0000 7f42fc35e640 1 Processor -- start 2023-10-16T15:34:09.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.794+0000 7f42fc35e640 1 -- start start 2023-10-16T15:34:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.795+0000 7f42fc35e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f4159b20 0x7f42f40c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.795+0000 7f42fc35e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.795+0000 7f42fc35e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f415d880 0x7f42f40c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.795+0000 7f42fc35e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40c6f30 con 0x7f42f4159f00 2023-10-16T15:34:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.795+0000 7f42fc35e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40c62b0 con 0x7f42f4159b20 2023-10-16T15:34:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.795+0000 7f42fc35e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40c6590 con 0x7f42f415d880 2023-10-16T15:34:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60140/0 (socket says 172.21.15.29:60140) 2023-10-16T15:34:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 -- 172.21.15.29:0/1130906014 learned_addr learned my addr 172.21.15.29:0/1130906014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:09.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42fa0fa640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f4159b20 0x7f42f40c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42fa8fb640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f415d880 0x7f42f40c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 -- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f415d880 msgr2=0x7f42f40c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f415d880 0x7f42f40c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 -- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f4159b20 msgr2=0x7f42f40c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:09.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f4159b20 0x7f42f40c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 -- 172.21.15.29:0/1130906014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40094d0 con 0x7f42f4159f00 2023-10-16T15:34:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.796+0000 7f42f98f9640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40c5a60 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009870 tx=0x7f42e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.797+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4030810 con 0x7f42f4159f00 2023-10-16T15:34:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.797+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4043400 con 0x7f42f4159f00 2023-10-16T15:34:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.797+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40437b0 con 0x7f42f4159f00 2023-10-16T15:34:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.797+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f40c6870 con 0x7f42f4159f00 2023-10-16T15:34:09.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.797+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f40c5fa0 con 0x7f42f4159f00 2023-10-16T15:34:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.798+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f40c22b0 con 0x7f42f4159f00 2023-10-16T15:34:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.799+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f42e4043910 con 0x7f42f4159f00 2023-10-16T15:34:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.801+0000 7f42eaffd640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f42d806acc0 0x7f42d806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.801+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f42e403e070 con 0x7f42f4159f00 2023-10-16T15:34:09.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.802+0000 7f42fa0fa640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f42d806acc0 0x7f42d806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.805+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42e4077a80 con 0x7f42f4159f00 2023-10-16T15:34:09.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.805+0000 7f42fa0fa640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f42d806acc0 0x7f42d806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f42f40b0e20 tx=0x7f42f0009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:09.872+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f42e4077e50 con 0x7f42f4159f00 2023-10-16T15:34:10.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.005+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42f40c7070 con 0x7f42f4159f00 2023-10-16T15:34:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.007+0000 7f42eaffd640 1 -- 172.21.15.29:0/1130906014 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+17641 (secure 0 0 0) 0x7f42f40c7070 con 0x7f42f4159f00 2023-10-16T15:34:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:10.010 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":181,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:07.842963+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/1288768965":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2519294017":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/1563871353":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2058472509":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2172765430":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2009585372":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/838551000":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6833/3772833058":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6833/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/1144059255":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6832/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3011898787":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3298516092":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6832/3772833058":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2177136748":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2616191068":"2023-10-17T15:33:59.835992+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:10.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f42d806acc0 msgr2=0x7f42d806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f42d806acc0 0x7f42d806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f42f40b0e20 tx=0x7f42f0009040 comp rx=0 tx=0).stop 2023-10-16T15:34:10.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 msgr2=0x7f42f40c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40c5a60 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009870 tx=0x7f42e4004730 comp rx=0 tx=0).stop 2023-10-16T15:34:10.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 shutdown_connections 2023-10-16T15:34:10.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f42d806acc0 0x7f42d806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f4159f00 0x7f42f40c5a60 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f415d880 0x7f42f40c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 --2- 172.21.15.29:0/1130906014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f4159b20 0x7f42f40c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 >> 172.21.15.29:0/1130906014 conn(0x7f42f40cb410 msgr2=0x7f42f40cdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:10.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 shutdown_connections 2023-10-16T15:34:10.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.009+0000 7f42fc35e640 1 -- 172.21.15.29:0/1130906014 wait complete. 2023-10-16T15:34:10.030 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:34:10.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:10.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.247+0000 7fb0c359e640 1 Processor -- start 2023-10-16T15:34:10.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.247+0000 7fb0c359e640 1 -- start start 2023-10-16T15:34:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.248+0000 7fb0c359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0c40ad970 0x7fb0c40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.248+0000 7fb0c359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0c40ccfb0 0x7fb0c40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.248+0000 7fb0c359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c415b9d0 con 0x7fb0c406c040 2023-10-16T15:34:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.248+0000 7fb0c359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c415bcb0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.248+0000 7fb0c359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c415bf90 con 0x7fb0c40ad970 2023-10-16T15:34:10.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0c40ccfb0 0x7fb0c40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0c40ccfb0 0x7fb0c40cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60150/0 (socket says 172.21.15.29:60150) 2023-10-16T15:34:10.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1764961941 learned_addr learned my addr 172.21.15.29:0/1764961941 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:10.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0bbfff640 1 --2- 172.21.15.29:0/1764961941 >> v2:172.21.15.170:3300/0 conn(0x7fb0c40ad970 0x7fb0c40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1764961941 >> v2:172.21.15.170:3300/0 conn(0x7fb0c40ad970 msgr2=0x7fb0c40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 --2- 172.21.15.29:0/1764961941 >> v2:172.21.15.170:3300/0 conn(0x7fb0c40ad970 0x7fb0c40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1764961941 >> v1:172.21.15.170:6789/0 conn(0x7fb0c406c040 legacy=0x7fb0c406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.249+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1764961941 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40cc270 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.250+0000 7fb0c2d9d640 1 --2- 172.21.15.29:0/1764961941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40cd460 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009e50 tx=0x7fb0bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57454968ae995378 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c1d9b640 1 -- 172.21.15.29:0/1764961941 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc02ca80 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c1d9b640 1 -- 172.21.15.29:0/1764961941 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc02cbe0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c1d9b640 1 -- 172.21.15.29:0/1764961941 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0343f0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 -- 172.21.15.29:0/1764961941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 msgr2=0x7fb0c40cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1764961941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40cd460 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009e50 tx=0x7fb0bc02bee0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 -- 172.21.15.29:0/1764961941 shutdown_connections 2023-10-16T15:34:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1764961941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40cd460 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1764961941 >> v2:172.21.15.170:3300/0 conn(0x7fb0c40ad970 0x7fb0c40cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 -- 172.21.15.29:0/1764961941 >> 172.21.15.29:0/1764961941 conn(0x7fb0c40ce6a0 msgr2=0x7fb0c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:10.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.251+0000 7fb0c359e640 1 -- 172.21.15.29:0/1764961941 shutdown_connections 2023-10-16T15:34:10.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.252+0000 7fb0c359e640 1 -- 172.21.15.29:0/1764961941 wait complete. 2023-10-16T15:34:10.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.253+0000 7fb0c359e640 1 Processor -- start 2023-10-16T15:34:10.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.253+0000 7fb0c359e640 1 -- start start 2023-10-16T15:34:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.254+0000 7fb0c359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0c406c040 0x7fb0c40b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.254+0000 7fb0c359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0c40ad970 0x7fb0c40b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.254+0000 7fb0c359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.254+0000 7fb0c359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40b10f0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.254+0000 7fb0c359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40ae100 con 0x7fb0c40ad970 2023-10-16T15:34:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.254+0000 7fb0c359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40b23a0 con 0x7fb0c406c040 2023-10-16T15:34:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60162/0 (socket says 172.21.15.29:60162) 2023-10-16T15:34:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1149382650 learned_addr learned my addr 172.21.15.29:0/1149382650 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c259c640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0c406c040 0x7fb0c40b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0bbfff640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0c40ad970 0x7fb0c40b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0c406c040 msgr2=0x7fb0c40b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0c406c040 0x7fb0c40b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0c40ad970 msgr2=0x7fb0c40b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0c40ad970 0x7fb0c40b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.255+0000 7fb0c2d9d640 1 -- 172.21.15.29:0/1149382650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc009b30 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.256+0000 7fb0c2d9d640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40b3300 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc036040 tx=0x7fb0bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.256+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc040670 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.256+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0407f0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.256+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c40ae3c0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.256+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc040ba0 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.256+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c40ae980 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.257+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c406b330 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:34:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.258+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb0bc02cd50 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.259+0000 7fb0baffd640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb0a406af00 0x7fb0a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.259+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fb0bc0ac770 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.260+0000 7fb0c259c640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb0a406af00 0x7fb0a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.261+0000 7fb0c259c640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb0a406af00 0x7fb0a406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4004ea0 tx=0x7fb0b4009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:10.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.263+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0bc077560 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:34:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.500+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c40aeb70 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.501+0000 7fb0baffd640 1 -- 172.21.15.29:0/1149382650 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4917 (secure 0 0 0) 0x7fb0bc009510 con 0x7fb0c40ccfb0 2023-10-16T15:34:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:10.504 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:33:45.081584+0000","modified":"2023-10-16T15:33:48.236102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9342":{"gid":9342,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3104953668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3104953668},{"type":"v1","addr":"172.21.15.29:6833","nonce":3104953668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T15:34:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 84 2023-10-16T15:34:10.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb0a406af00 msgr2=0x7fb0a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb0a406af00 0x7fb0a406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4004ea0 tx=0x7fb0b4009040 comp rx=0 tx=0).stop 2023-10-16T15:34:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 msgr2=0x7fb0c40b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40b3300 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc036040 tx=0x7fb0bc004a40 comp rx=0 tx=0).stop 2023-10-16T15:34:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 shutdown_connections 2023-10-16T15:34:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fb0a406af00 0x7fb0a406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0c40ccfb0 0x7fb0c40b3300 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0c40ad970 0x7fb0c40b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 --2- 172.21.15.29:0/1149382650 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0c406c040 0x7fb0c40b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 >> 172.21.15.29:0/1149382650 conn(0x7fb0c40ce6a0 msgr2=0x7fb0c40c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 shutdown_connections 2023-10-16T15:34:10.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.504+0000 7fb0c359e640 1 -- 172.21.15.29:0/1149382650 wait complete. 2023-10-16T15:34:10.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:10.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.749+0000 7f127efa7640 1 Processor -- start 2023-10-16T15:34:10.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.749+0000 7f127efa7640 1 -- start start 2023-10-16T15:34:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.751+0000 7f127efa7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 0x7f127806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.751+0000 7f127efa7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127815d910 con 0x7f1278070b60 2023-10-16T15:34:10.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.751+0000 7f127efa7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127815dbf0 con 0x7f1278159950 2023-10-16T15:34:10.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.751+0000 7f127efa7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127815e490 con 0x7f12780ca0a0 2023-10-16T15:34:10.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.751+0000 7f127d7a4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1278070b60 0x7f127806c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48452/0 (socket says 172.21.15.29:48452) 2023-10-16T15:34:10.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.751+0000 7f127d7a4640 1 -- 172.21.15.29:0/268025453 learned_addr learned my addr 172.21.15.29:0/268025453 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:10.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127e7a6640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 0x7f127806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263063309 0 0) 0x7f127815d910 con 0x7f1278070b60 2023-10-16T15:34:10.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1264003520 con 0x7f1278070b60 2023-10-16T15:34:10.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1775772815 0 0) 0x7f1264003520 con 0x7f1278070b60 2023-10-16T15:34:10.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 msgr2=0x7f127806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:10.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 0x7f127806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 >> v1:172.21.15.170:6789/0 conn(0x7f1278159950 legacy=0x7f1278070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127815ded0 con 0x7f1278070b60 2023-10-16T15:34:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1268001670 con 0x7f1278070b60 2023-10-16T15:34:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 >> v1:172.21.15.29:6789/0 conn(0x7f1278070b60 legacy=0x7f127806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.752+0000 7f127cfa3640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12640039a0 0x7f1264003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127cfa3640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640047c0 0x7f1264004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127cfa3640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 0x7f1264005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127815e490 con 0x7f12640039a0 2023-10-16T15:34:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126400bb10 con 0x7f12640047c0 2023-10-16T15:34:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126400bcd0 con 0x7f12780ca0a0 2023-10-16T15:34:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127dfa5640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12640039a0 0x7f1264003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127d7a4640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640047c0 0x7f1264004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127dfa5640 1 -- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 msgr2=0x7f1264005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127dfa5640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12780ca0a0 0x7f1264005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127dfa5640 1 -- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640047c0 msgr2=0x7f1264004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127dfa5640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640047c0 0x7f1264004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127dfa5640 1 -- 172.21.15.29:0/268025453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264003520 con 0x7f12640039a0 2023-10-16T15:34:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.753+0000 7f127d7a4640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640047c0 0x7f1264004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.754+0000 7f127dfa5640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12640039a0 0x7f1264003dc0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f1274001750 tx=0x7f12640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa778e4c74732b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12740052d0 con 0x7f12640039a0 2023-10-16T15:34:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127402e050 con 0x7f12640039a0 2023-10-16T15:34:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127cfa3640 1 -- 172.21.15.29:0/268025453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12740035f0 con 0x7f12640039a0 2023-10-16T15:34:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127efa7640 1 -- 172.21.15.29:0/268025453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12640039a0 msgr2=0x7f1264003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127efa7640 1 --2- 172.21.15.29:0/268025453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12640039a0 0x7f1264003dc0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f1274001750 tx=0x7f12640057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127e7a6640 1 -- 172.21.15.29:0/268025453 reap_dead start 2023-10-16T15:34:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127efa7640 1 -- 172.21.15.29:0/268025453 shutdown_connections 2023-10-16T15:34:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127efa7640 1 -- 172.21.15.29:0/268025453 >> 172.21.15.29:0/268025453 conn(0x7f12780ce6a0 msgr2=0x7f12780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127efa7640 1 -- 172.21.15.29:0/268025453 shutdown_connections 2023-10-16T15:34:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.755+0000 7f127efa7640 1 -- 172.21.15.29:0/268025453 wait complete. 2023-10-16T15:34:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.757+0000 7f127efa7640 1 Processor -- start 2023-10-16T15:34:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.757+0000 7f127efa7640 1 -- start start 2023-10-16T15:34:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.758+0000 7f127efa7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640039a0 0x7f12780b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.758+0000 7f127efa7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 0x7f127815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.758+0000 7f127efa7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12781547e0 0x7f1278154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.758+0000 7f127efa7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12780c85b0 con 0x7f12780b1b70 2023-10-16T15:34:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.758+0000 7f127efa7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12780b1ff0 con 0x7f12781547e0 2023-10-16T15:34:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.758+0000 7f127efa7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12781551a0 con 0x7f12640039a0 2023-10-16T15:34:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 0x7f127815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 0x7f127815cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60184/0 (socket says 172.21.15.29:60184) 2023-10-16T15:34:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 -- 172.21.15.29:0/2179544324 learned_addr learned my addr 172.21.15.29:0/2179544324 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127dfa5640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640039a0 0x7f12780b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127e7a6640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12781547e0 0x7f1278154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 -- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640039a0 msgr2=0x7f12780b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640039a0 0x7f12780b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 -- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12781547e0 msgr2=0x7f1278154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12781547e0 0x7f1278154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.759+0000 7f127d7a4640 1 -- 172.21.15.29:0/2179544324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780b2270 con 0x7f12780b1b70 2023-10-16T15:34:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.760+0000 7f127d7a4640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 0x7f127815cad0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f126800d810 tx=0x7f126800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.760+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680074e0 con 0x7f12780b1b70 2023-10-16T15:34:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.760+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268007640 con 0x7f12780b1b70 2023-10-16T15:34:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.760+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680049e0 con 0x7f12780b1b70 2023-10-16T15:34:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.760+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12780c7190 con 0x7f12780b1b70 2023-10-16T15:34:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.760+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12780c7750 con 0x7f12780b1b70 2023-10-16T15:34:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.761+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127804e9b0 con 0x7f12780b1b70 2023-10-16T15:34:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.762+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1268004b40 con 0x7f12780b1b70 2023-10-16T15:34:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.763+0000 7f126e7fc640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f125c06af30 0x7f125c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.763+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f12680893b0 con 0x7f12780b1b70 2023-10-16T15:34:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.764+0000 7f127dfa5640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f125c06af30 0x7f125c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.765+0000 7f127dfa5640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f125c06af30 0x7f125c06d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1274001750 tx=0x7f1274009030 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:10.768+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1268054020 con 0x7f12780b1b70 2023-10-16T15:34:11.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.003+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12780b4620 con 0x7f12780b1b70 2023-10-16T15:34:11.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.004+0000 7f126e7fc640 1 -- 172.21.15.29:0/2179544324 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f1268057640 con 0x7f12780b1b70 2023-10-16T15:34:11.007 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:11.007 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:33:45.081584+0000","modified":"2023-10-16T15:33:48.236102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9342":{"gid":9342,"name":"a","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/3104953668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3104953668},{"type":"v1","addr":"172.21.15.29:6833","nonce":3104953668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T15:34:11.008 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 84 2023-10-16T15:34:11.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f125c06af30 msgr2=0x7f125c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f125c06af30 0x7f125c06d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1274001750 tx=0x7f1274009030 comp rx=0 tx=0).stop 2023-10-16T15:34:11.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 msgr2=0x7f127815cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 0x7f127815cad0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f126800d810 tx=0x7f126800dcd0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 shutdown_connections 2023-10-16T15:34:11.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f125c06af30 0x7f125c06d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12780b1b70 0x7f127815cad0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12781547e0 0x7f1278154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 --2- 172.21.15.29:0/2179544324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640039a0 0x7f12780b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 >> 172.21.15.29:0/2179544324 conn(0x7f12780ce6a0 msgr2=0x7f12780ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:11.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.007+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 shutdown_connections 2023-10-16T15:34:11.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.008+0000 7f127efa7640 1 -- 172.21.15.29:0/2179544324 wait complete. 2023-10-16T15:34:11.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.246+0000 7ff46ee1a640 1 Processor -- start 2023-10-16T15:34:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.246+0000 7ff46ee1a640 1 -- start start 2023-10-16T15:34:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46ee1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff46806c040 0x7ff46806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46ee1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4680ad970 0x7ff46815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46ee1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46806c960 con 0x7ff46815ebc0 2023-10-16T15:34:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46ee1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46815b9d0 con 0x7ff46806c040 2023-10-16T15:34:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46ee1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46815bcb0 con 0x7ff4680ad970 2023-10-16T15:34:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46de18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff46806c040 0x7ff46806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46de18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff46806c040 0x7ff46806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60188/0 (socket says 172.21.15.29:60188) 2023-10-16T15:34:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.248+0000 7ff46de18640 1 -- 172.21.15.29:0/2563271613 learned_addr learned my addr 172.21.15.29:0/2563271613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.249+0000 7ff46de18640 1 -- 172.21.15.29:0/2563271613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4680ad970 msgr2=0x7ff46815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.249+0000 7ff46de18640 1 --2- 172.21.15.29:0/2563271613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4680ad970 0x7ff46815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.249+0000 7ff46de18640 1 -- 172.21.15.29:0/2563271613 >> v1:172.21.15.170:6789/0 conn(0x7ff46815ebc0 legacy=0x7ff4680cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:11.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.249+0000 7ff46de18640 1 -- 172.21.15.29:0/2563271613 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46815bf90 con 0x7ff46806c040 2023-10-16T15:34:11.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.250+0000 7ff46de18640 1 --2- 172.21.15.29:0/2563271613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46806c040 0x7ff46806c420 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7ff4640098a0 tx=0x7ff46402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5715acaec3a2d783 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.250+0000 7ff46ce16640 1 -- 172.21.15.29:0/2563271613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464031640 con 0x7ff46806c040 2023-10-16T15:34:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.250+0000 7ff46ce16640 1 -- 172.21.15.29:0/2563271613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4640317a0 con 0x7ff46806c040 2023-10-16T15:34:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.250+0000 7ff46ce16640 1 -- 172.21.15.29:0/2563271613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464031b30 con 0x7ff46806c040 2023-10-16T15:34:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 -- 172.21.15.29:0/2563271613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46806c040 msgr2=0x7ff46806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/2563271613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46806c040 0x7ff46806c420 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7ff4640098a0 tx=0x7ff46402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 -- 172.21.15.29:0/2563271613 shutdown_connections 2023-10-16T15:34:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/2563271613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4680ad970 0x7ff46815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/2563271613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46806c040 0x7ff46806c420 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 -- 172.21.15.29:0/2563271613 >> 172.21.15.29:0/2563271613 conn(0x7ff4680ce6a0 msgr2=0x7ff4680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 -- 172.21.15.29:0/2563271613 shutdown_connections 2023-10-16T15:34:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.251+0000 7ff46ee1a640 1 -- 172.21.15.29:0/2563271613 wait complete. 2023-10-16T15:34:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.253+0000 7ff46ee1a640 1 Processor -- start 2023-10-16T15:34:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.253+0000 7ff46ee1a640 1 -- start start 2023-10-16T15:34:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46ee1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff46806c040 0x7ff468153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46ee1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4680ad970 0x7ff4680c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46ee1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 0x7ff4680ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46ee1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680c97b0 con 0x7ff46815ebc0 2023-10-16T15:34:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46ee1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680c9a90 con 0x7ff4680ad970 2023-10-16T15:34:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46ee1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680c9d70 con 0x7ff46806c040 2023-10-16T15:34:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46e619640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 0x7ff4680ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46e619640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 0x7ff4680ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60192/0 (socket says 172.21.15.29:60192) 2023-10-16T15:34:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.254+0000 7ff46e619640 1 -- 172.21.15.29:0/1460648989 learned_addr learned my addr 172.21.15.29:0/1460648989 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46de18640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff46806c040 0x7ff468153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46d617640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4680ad970 0x7ff4680c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46e619640 1 -- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff46806c040 msgr2=0x7ff468153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46e619640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff46806c040 0x7ff468153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46e619640 1 -- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4680ad970 msgr2=0x7ff4680c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46e619640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4680ad970 0x7ff4680c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46e619640 1 -- 172.21.15.29:0/1460648989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640094d0 con 0x7ff46815ebc0 2023-10-16T15:34:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46d617640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4680ad970 0x7ff4680c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46e619640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 0x7ff4680ac890 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7ff46000c780 tx=0x7ff46000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460003d80 con 0x7ff46815ebc0 2023-10-16T15:34:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46000ce90 con 0x7ff46815ebc0 2023-10-16T15:34:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4600052b0 con 0x7ff46815ebc0 2023-10-16T15:34:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.255+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4680ca050 con 0x7ff46815ebc0 2023-10-16T15:34:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.256+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4680c6980 con 0x7ff46815ebc0 2023-10-16T15:34:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.257+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46806b330 con 0x7ff46815ebc0 2023-10-16T15:34:11.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.258+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff460005410 con 0x7ff46815ebc0 2023-10-16T15:34:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.259+0000 7ff45e7fc640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff44c06af00 0x7ff44c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.259+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ff460088eb0 con 0x7ff46815ebc0 2023-10-16T15:34:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.259+0000 7ff46de18640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff44c06af00 0x7ff44c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.261+0000 7ff46de18640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff44c06af00 0x7ff44c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff464036000 tx=0x7ff46403c040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:11.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.263+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff460053ba0 con 0x7ff46815ebc0 2023-10-16T15:34:11.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.456+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4680ca570 con 0x7ff46815ebc0 2023-10-16T15:34:11.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.458+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1460648989 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+17641 (secure 0 0 0) 0x7ff4680ca570 con 0x7ff46815ebc0 2023-10-16T15:34:11.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:11.461 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":181,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:07.842963+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:33:43.022528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T15:33:44.097016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/1288768965":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2519294017":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/1563871353":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2058472509":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2172765430":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2009585372":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/838551000":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6833/3772833058":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6833/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/1144059255":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6832/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3011898787":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3298516092":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6832/3772833058":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2177136748":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2616191068":"2023-10-17T15:33:59.835992+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff44c06af00 msgr2=0x7ff44c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff44c06af00 0x7ff44c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff464036000 tx=0x7ff46403c040 comp rx=0 tx=0).stop 2023-10-16T15:34:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 msgr2=0x7ff4680ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 0x7ff4680ac890 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7ff46000c780 tx=0x7ff46000cc40 comp rx=0 tx=0).stop 2023-10-16T15:34:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 shutdown_connections 2023-10-16T15:34:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff44c06af00 0x7ff44c06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46815ebc0 0x7ff4680ac890 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4680ad970 0x7ff4680c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 --2- 172.21.15.29:0/1460648989 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff46806c040 0x7ff468153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 >> 172.21.15.29:0/1460648989 conn(0x7ff4680ce6a0 msgr2=0x7ff46815d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 shutdown_connections 2023-10-16T15:34:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.461+0000 7ff46ee1a640 1 -- 172.21.15.29:0/1460648989 wait complete. 2023-10-16T15:34:11.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:34:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.701+0000 7f90570d7640 1 Processor -- start 2023-10-16T15:34:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.702+0000 7f90570d7640 1 -- start start 2023-10-16T15:34:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f90570d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9050069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f90570d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f90500c9ca0 0x7f905006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f90570d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90500cb030 con 0x7f90500ca3b0 2023-10-16T15:34:11.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f90570d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90500c91c0 con 0x7f90500c9ca0 2023-10-16T15:34:11.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f90570d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90500c94a0 con 0x7f905015ad70 2023-10-16T15:34:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f9055674640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f90500ca3b0 0x7f90500ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48466/0 (socket says 172.21.15.29:48466) 2023-10-16T15:34:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.703+0000 7f9055674640 1 -- 172.21.15.29:0/1574350859 learned_addr learned my addr 172.21.15.29:0/1574350859 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904ffff640 1 --2- 172.21.15.29:0/1574350859 >> v2:172.21.15.170:3300/0 conn(0x7f90500c9ca0 0x7f905006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f9054e73640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9050069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220270936 0 0) 0x7f90500cb030 con 0x7f90500ca3b0 2023-10-16T15:34:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9038003520 con 0x7f90500ca3b0 2023-10-16T15:34:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339521418 0 0) 0x7f9038003520 con 0x7f90500ca3b0 2023-10-16T15:34:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 msgr2=0x7f9050069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9050069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 >> v2:172.21.15.170:3300/0 conn(0x7f90500c9ca0 msgr2=0x7f905006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 --2- 172.21.15.29:0/1574350859 >> v2:172.21.15.170:3300/0 conn(0x7f90500c9ca0 0x7f905006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90500c9780 con 0x7f90500ca3b0 2023-10-16T15:34:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9048001670 con 0x7f90500ca3b0 2023-10-16T15:34:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 >> v1:172.21.15.29:6789/0 conn(0x7f90500ca3b0 legacy=0x7f90500ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9038003970 0x7f9038003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f9038004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9038005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500c91c0 con 0x7f90380047c0 2023-10-16T15:34:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500c94a0 con 0x7f9038003970 2023-10-16T15:34:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.704+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903800b980 con 0x7f905015ad70 2023-10-16T15:34:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f904ffff640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f9038004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f9054e73640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9038003970 0x7f9038003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f9055674640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9038005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f904ffff640 1 -- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 msgr2=0x7f9038005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f904ffff640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9038005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f904ffff640 1 -- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9038003970 msgr2=0x7f9038003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f904ffff640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9038003970 0x7f9038003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f904ffff640 1 -- 172.21.15.29:0/1574350859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9038003520 con 0x7f90380047c0 2023-10-16T15:34:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f9055674640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905015ad70 0x7f9038005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:11.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.705+0000 7f9054e73640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9038003970 0x7f9038003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:11.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.706+0000 7f904ffff640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f9038004c40 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f9044009ac0 tx=0x7f90380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7641b2b1cfb2c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:11.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.706+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904402c710 con 0x7f90380047c0 2023-10-16T15:34:11.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904402c870 con 0x7f90380047c0 2023-10-16T15:34:11.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f904f7fe640 1 -- 172.21.15.29:0/1574350859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904402cc00 con 0x7f90380047c0 2023-10-16T15:34:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f90570d7640 1 -- 172.21.15.29:0/1574350859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 msgr2=0x7f9038004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f90570d7640 1 --2- 172.21.15.29:0/1574350859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f9038004c40 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f9044009ac0 tx=0x7f90380057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f9055674640 1 -- 172.21.15.29:0/1574350859 reap_dead start 2023-10-16T15:34:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f90570d7640 1 -- 172.21.15.29:0/1574350859 shutdown_connections 2023-10-16T15:34:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f90570d7640 1 -- 172.21.15.29:0/1574350859 >> 172.21.15.29:0/1574350859 conn(0x7f90500ce9b0 msgr2=0x7f90500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f90570d7640 1 -- 172.21.15.29:0/1574350859 shutdown_connections 2023-10-16T15:34:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.707+0000 7f90570d7640 1 -- 172.21.15.29:0/1574350859 wait complete. 2023-10-16T15:34:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.709+0000 7f90570d7640 1 Processor -- start 2023-10-16T15:34:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.709+0000 7f90570d7640 1 -- start start 2023-10-16T15:34:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f90570d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f905014f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f90570d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90501f5b30 0x7f90501f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f90570d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90501f1650 0x7f90501f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f90570d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90501f64d0 con 0x7f90380047c0 2023-10-16T15:34:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f90570d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90500cc8d0 con 0x7f90501f5b30 2023-10-16T15:34:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f90570d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90500c5db0 con 0x7f90501f1650 2023-10-16T15:34:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.710+0000 7f9054e73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f905014f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f905014f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60204/0 (socket says 172.21.15.29:60204) 2023-10-16T15:34:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 -- 172.21.15.29:0/3856679556 learned_addr learned my addr 172.21.15.29:0/3856679556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9055674640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90501f1650 0x7f90501f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f904ffff640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90501f5b30 0x7f90501f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 -- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90501f1650 msgr2=0x7f90501f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90501f1650 0x7f90501f1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 -- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90501f5b30 msgr2=0x7f90501f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90501f5b30 0x7f90501f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 -- 172.21.15.29:0/3856679556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90500cc580 con 0x7f90380047c0 2023-10-16T15:34:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9055674640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90501f1650 0x7f90501f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f904ffff640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90501f5b30 0x7f90501f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.711+0000 7f9054e73640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f905014f420 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f904000f350 tx=0x7f904000f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.712+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040003b90 con 0x7f90380047c0 2023-10-16T15:34:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.712+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9040011040 con 0x7f90380047c0 2023-10-16T15:34:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.712+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040004020 con 0x7f90380047c0 2023-10-16T15:34:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.712+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90500c81f0 con 0x7f90380047c0 2023-10-16T15:34:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.712+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90500c87b0 con 0x7f90380047c0 2023-10-16T15:34:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.713+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905004e160 con 0x7f90380047c0 2023-10-16T15:34:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.714+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9040004180 con 0x7f90380047c0 2023-10-16T15:34:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.715+0000 7f904d7fa640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f903c06af00 0x7f903c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.715+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f904008a190 con 0x7f90380047c0 2023-10-16T15:34:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.716+0000 7f904ffff640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f903c06af00 0x7f903c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.717+0000 7f904ffff640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f903c06af00 0x7f903c06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9044036000 tx=0x7f9044030000 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.719+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9040054d70 con 0x7f90380047c0 2023-10-16T15:34:11.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.961+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f905006c690 con 0x7f90380047c0 2023-10-16T15:34:11.964 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:34:11.962+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:34:11.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.966+0000 7f904d7fa640 1 -- 172.21.15.29:0/3856679556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9040058390 con 0x7f90380047c0 2023-10-16T15:34:11.968 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:34:11.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f903c06af00 msgr2=0x7f903c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f903c06af00 0x7f903c06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9044036000 tx=0x7f9044030000 comp rx=0 tx=0).stop 2023-10-16T15:34:11.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 msgr2=0x7f905014f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:11.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f905014f420 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f904000f350 tx=0x7f904000f810 comp rx=0 tx=0).stop 2023-10-16T15:34:11.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 shutdown_connections 2023-10-16T15:34:11.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f903c06af00 0x7f903c06d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f90501f1650 0x7f90501f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90501f5b30 0x7f90501f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 --2- 172.21.15.29:0/3856679556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90380047c0 0x7f905014f420 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:11.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 >> 172.21.15.29:0/3856679556 conn(0x7f90500ce9b0 msgr2=0x7f905015d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:11.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 shutdown_connections 2023-10-16T15:34:11.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:11.970+0000 7f90570d7640 1 -- 172.21.15.29:0/3856679556 wait complete. 2023-10-16T15:34:11.985 INFO:tasks.ceph.mds.a.smithi029.stderr: -3846> 2023-10-16T15:33:50.548+0000 7f4279978640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:34:12.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:34:12.043 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:34:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.200+0000 7f8d22e40640 1 Processor -- start 2023-10-16T15:34:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.200+0000 7f8d22e40640 1 -- start start 2023-10-16T15:34:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.201+0000 7f8d22e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c06c040 0x7f8d1c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.201+0000 7f8d22e40640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d1c0ad970 0x7f8d1c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d22e40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c06c960 con 0x7f8d1c15ebc0 2023-10-16T15:34:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d22e40640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1c15b9d0 con 0x7f8d1c0ad970 2023-10-16T15:34:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d22e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c15bcb0 con 0x7f8d1c06c040 2023-10-16T15:34:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d2163d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d1c0ad970 0x7f8d1c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d2163d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d1c0ad970 0x7f8d1c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60236/0 (socket says 172.21.15.29:60236) 2023-10-16T15:34:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d2163d640 1 -- 172.21.15.29:0/2411899496 learned_addr learned my addr 172.21.15.29:0/2411899496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.202+0000 7f8d21e3e640 1 --2- 172.21.15.29:0/2411899496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c06c040 0x7f8d1c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.203+0000 7f8d2163d640 1 -- 172.21.15.29:0/2411899496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c06c040 msgr2=0x7f8d1c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.203+0000 7f8d2163d640 1 --2- 172.21.15.29:0/2411899496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c06c040 0x7f8d1c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.203+0000 7f8d2163d640 1 -- 172.21.15.29:0/2411899496 >> v1:172.21.15.170:6789/0 conn(0x7f8d1c15ebc0 legacy=0x7f8d1c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.203+0000 7f8d2163d640 1 -- 172.21.15.29:0/2411899496 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c15bf90 con 0x7f8d1c0ad970 2023-10-16T15:34:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.204+0000 7f8d2163d640 1 --2- 172.21.15.29:0/2411899496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c0ad970 0x7f8d1c15e680 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0098a0 tx=0x7f8d0c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd20a721a13109f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.204+0000 7f8d20e3c640 1 -- 172.21.15.29:0/2411899496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c030840 con 0x7f8d1c0ad970 2023-10-16T15:34:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.204+0000 7f8d20e3c640 1 -- 172.21.15.29:0/2411899496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c0309a0 con 0x7f8d1c0ad970 2023-10-16T15:34:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.204+0000 7f8d20e3c640 1 -- 172.21.15.29:0/2411899496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c030d30 con 0x7f8d1c0ad970 2023-10-16T15:34:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 -- 172.21.15.29:0/2411899496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c0ad970 msgr2=0x7f8d1c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 --2- 172.21.15.29:0/2411899496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c0ad970 0x7f8d1c15e680 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0098a0 tx=0x7f8d0c02b980 comp rx=0 tx=0).stop 2023-10-16T15:34:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 -- 172.21.15.29:0/2411899496 shutdown_connections 2023-10-16T15:34:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 --2- 172.21.15.29:0/2411899496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c0ad970 0x7f8d1c15e680 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 --2- 172.21.15.29:0/2411899496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c06c040 0x7f8d1c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 -- 172.21.15.29:0/2411899496 >> 172.21.15.29:0/2411899496 conn(0x7f8d1c0ce6a0 msgr2=0x7f8d1c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 -- 172.21.15.29:0/2411899496 shutdown_connections 2023-10-16T15:34:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.205+0000 7f8d22e40640 1 -- 172.21.15.29:0/2411899496 wait complete. 2023-10-16T15:34:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.207+0000 7f8d22e40640 1 Processor -- start 2023-10-16T15:34:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.207+0000 7f8d22e40640 1 -- start start 2023-10-16T15:34:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d22e40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 0x7f8d1c069f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d22e40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c0ad970 0x7f8d1c15ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d22e40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c15ebc0 0x7f8d1c1f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d22e40640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c1f33f0 con 0x7f8d1c06c040 2023-10-16T15:34:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d22e40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c1f3a10 con 0x7f8d1c0ad970 2023-10-16T15:34:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d22e40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0c8200 con 0x7f8d1c15ebc0 2023-10-16T15:34:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d21e3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 0x7f8d1c069f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.208+0000 7f8d21e3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 0x7f8d1c069f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60252/0 (socket says 172.21.15.29:60252) 2023-10-16T15:34:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 -- 172.21.15.29:0/1158102886 learned_addr learned my addr 172.21.15.29:0/1158102886 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 -- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c15ebc0 msgr2=0x7f8d1c1f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c15ebc0 0x7f8d1c1f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 -- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c0ad970 msgr2=0x7f8d1c15ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c0ad970 0x7f8d1c15ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 -- 172.21.15.29:0/1158102886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0094d0 con 0x7f8d1c06c040 2023-10-16T15:34:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d21e3e640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 0x7f8d1c069f70 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800d5f0 tx=0x7f8d1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180125f0 con 0x7f8d1c06c040 2023-10-16T15:34:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.209+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d18012750 con 0x7f8d1c06c040 2023-10-16T15:34:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.210+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1c1f3cb0 con 0x7f8d1c06c040 2023-10-16T15:34:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.210+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18012ae0 con 0x7f8d1c06c040 2023-10-16T15:34:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.210+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c0c84e0 con 0x7f8d1c06c040 2023-10-16T15:34:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.211+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c06b330 con 0x7f8d1c06c040 2023-10-16T15:34:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.212+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8d18012c40 con 0x7f8d1c06c040 2023-10-16T15:34:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.213+0000 7f8d127fc640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8d0006af00 0x7f8d0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.213+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f8d18089e30 con 0x7f8d1c06c040 2023-10-16T15:34:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.213+0000 7f8d2163d640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8d0006af00 0x7f8d0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.215+0000 7f8d2163d640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8d0006af00 0x7f8d0006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c036000 tx=0x7f8d0c03b040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.218+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d18054bd0 con 0x7f8d1c06c040 2023-10-16T15:34:12.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.449+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8d1c0cc250 con 0x7f8d1c06c040 2023-10-16T15:34:12.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.741+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8d18054fa0 con 0x7f8d1c06c040 2023-10-16T15:34:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.969+0000 7f8d127fc640 1 -- 172.21.15.29:0/1158102886 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8d18087360 con 0x7f8d1c06c040 2023-10-16T15:34:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8d0006af00 msgr2=0x7f8d0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8d0006af00 0x7f8d0006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c036000 tx=0x7f8d0c03b040 comp rx=0 tx=0).stop 2023-10-16T15:34:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 msgr2=0x7f8d1c069f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 0x7f8d1c069f70 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800d5f0 tx=0x7f8d1800dab0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 shutdown_connections 2023-10-16T15:34:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8d0006af00 0x7f8d0006d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d1c15ebc0 0x7f8d1c1f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d1c0ad970 0x7f8d1c15ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 --2- 172.21.15.29:0/1158102886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d1c06c040 0x7f8d1c069f70 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:12.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 >> 172.21.15.29:0/1158102886 conn(0x7f8d1c0ce6a0 msgr2=0x7f8d1c15da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:12.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 shutdown_connections 2023-10-16T15:34:12.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:12.973+0000 7f8d22e40640 1 -- 172.21.15.29:0/1158102886 wait complete. 2023-10-16T15:34:12.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:34:13.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.273+0000 7f799e935640 1 Processor -- start 2023-10-16T15:34:13.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.273+0000 7f799e935640 1 -- start start 2023-10-16T15:34:13.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.274+0000 7f799e935640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f799815cd70 0x7f79980d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.274+0000 7f799e935640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980b8480 0x7f79980cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.274+0000 7f799e935640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79980d0be0 0x7f79980cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.274+0000 7f799e935640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799815d150 con 0x7f79980d0be0 2023-10-16T15:34:13.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.274+0000 7f799e935640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79980d1180 con 0x7f799815cd70 2023-10-16T15:34:13.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.274+0000 7f799e935640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79980d1460 con 0x7f79980b8480 2023-10-16T15:34:13.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79980d0be0 0x7f79980cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79980d0be0 0x7f79980cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60258/0 (socket says 172.21.15.29:60258) 2023-10-16T15:34:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 -- 172.21.15.29:0/2191876254 learned_addr learned my addr 172.21.15.29:0/2191876254 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:13.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799d132640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980b8480 0x7f79980cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799d933640 1 --2- 172.21.15.29:0/2191876254 >> v2:172.21.15.170:3300/0 conn(0x7f799815cd70 0x7f79980d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:13.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 -- 172.21.15.29:0/2191876254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980b8480 msgr2=0x7f79980cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980b8480 0x7f79980cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 -- 172.21.15.29:0/2191876254 >> v2:172.21.15.170:3300/0 conn(0x7f799815cd70 msgr2=0x7f79980d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 --2- 172.21.15.29:0/2191876254 >> v2:172.21.15.170:3300/0 conn(0x7f799815cd70 0x7f79980d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799e134640 1 -- 172.21.15.29:0/2191876254 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980d1740 con 0x7f79980d0be0 2023-10-16T15:34:13.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.275+0000 7f799d132640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980b8480 0x7f79980cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:13.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.277+0000 7f799e134640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79980cdd00 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f79980b56a0 tx=0x7f799002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1782d212bea9b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:13.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.277+0000 7f799c931640 1 -- 172.21.15.29:0/2191876254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990031870 con 0x7f79980d0be0 2023-10-16T15:34:13.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.277+0000 7f799c931640 1 -- 172.21.15.29:0/2191876254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79900319d0 con 0x7f79980d0be0 2023-10-16T15:34:13.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.277+0000 7f799c931640 1 -- 172.21.15.29:0/2191876254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990031d60 con 0x7f79980d0be0 2023-10-16T15:34:13.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.277+0000 7f799e935640 1 -- 172.21.15.29:0/2191876254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 msgr2=0x7f79980cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:13.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.277+0000 7f799e935640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79980cdd00 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f79980b56a0 tx=0x7f799002b910 comp rx=0 tx=0).stop 2023-10-16T15:34:13.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 -- 172.21.15.29:0/2191876254 shutdown_connections 2023-10-16T15:34:13.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79980cdd00 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 --2- 172.21.15.29:0/2191876254 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79980b8480 0x7f79980cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 --2- 172.21.15.29:0/2191876254 >> v2:172.21.15.170:3300/0 conn(0x7f799815cd70 0x7f79980d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 -- 172.21.15.29:0/2191876254 >> 172.21.15.29:0/2191876254 conn(0x7f79980cf570 msgr2=0x7f79980cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:13.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 -- 172.21.15.29:0/2191876254 shutdown_connections 2023-10-16T15:34:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.278+0000 7f799e935640 1 -- 172.21.15.29:0/2191876254 wait complete. 2023-10-16T15:34:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.279+0000 7f799e935640 1 Processor -- start 2023-10-16T15:34:13.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.280+0000 7f799e935640 1 -- start start 2023-10-16T15:34:13.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799e935640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79980b8480 0x7f799815e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799e935640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79981500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799e935640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f799815cd70 0x7f79980c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799e935640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799815f6d0 con 0x7f79980d0be0 2023-10-16T15:34:13.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799e935640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7998150630 con 0x7f79980b8480 2023-10-16T15:34:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799e935640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799815eac0 con 0x7f799815cd70 2023-10-16T15:34:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79981500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:13.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79981500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60266/0 (socket says 172.21.15.29:60266) 2023-10-16T15:34:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 -- 172.21.15.29:0/1677684353 learned_addr learned my addr 172.21.15.29:0/1677684353 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:13.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d933640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79980b8480 0x7f799815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:13.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 -- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f799815cd70 msgr2=0x7f79980c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:13.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f799815cd70 0x7f79980c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 -- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79980b8480 msgr2=0x7f799815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:13.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79980b8480 0x7f799815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.281+0000 7f799d132640 1 -- 172.21.15.29:0/1677684353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79900094d0 con 0x7f79980d0be0 2023-10-16T15:34:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.282+0000 7f799d132640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79981500f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f798800c7f0 tx=0x7f798800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.282+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988007a30 con 0x7f79980d0be0 2023-10-16T15:34:13.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.282+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7988007b90 con 0x7f79980d0be0 2023-10-16T15:34:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.282+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799815eda0 con 0x7f79980d0be0 2023-10-16T15:34:13.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.282+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798800ce40 con 0x7f79980d0be0 2023-10-16T15:34:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.282+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799815f360 con 0x7f79980d0be0 2023-10-16T15:34:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.283+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799815f880 con 0x7f79980d0be0 2023-10-16T15:34:13.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.285+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79880113f0 con 0x7f79980d0be0 2023-10-16T15:34:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.286+0000 7f798e7fc640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f797c06afc0 0x7f797c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.286+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f7988089cc0 con 0x7f79980d0be0 2023-10-16T15:34:13.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.286+0000 7f799d933640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f797c06afc0 0x7f797c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:13.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.288+0000 7f799d933640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f797c06afc0 0x7f797c06d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7994009940 tx=0x7f7994003cd0 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:13.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.290+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f799815f880 con 0x7f79980d0be0 2023-10-16T15:34:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.488+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f799804e9b0 con 0x7f79980d0be0 2023-10-16T15:34:13.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.973+0000 7f798e7fc640 1 -- 172.21.15.29:0/1677684353 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v183) v1 ==== 181+0+0 (secure 0 0 0) 0x7f798801f0d0 con 0x7f79980d0be0 2023-10-16T15:34:13.976 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:34:13.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f797c06afc0 msgr2=0x7f797c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:13.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f797c06afc0 0x7f797c06d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7994009940 tx=0x7f7994003cd0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 msgr2=0x7f79981500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:13.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79981500f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f798800c7f0 tx=0x7f798800cbf0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 shutdown_connections 2023-10-16T15:34:13.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f797c06afc0 0x7f797c06d480 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79980d0be0 0x7f79981500f0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f799815cd70 0x7f79980c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 --2- 172.21.15.29:0/1677684353 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79980b8480 0x7f799815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:13.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.976+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 >> 172.21.15.29:0/1677684353 conn(0x7f79980cf570 msgr2=0x7f79980b7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:13.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.977+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 shutdown_connections 2023-10-16T15:34:13.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:13.977+0000 7f799e935640 1 -- 172.21.15.29:0/1677684353 wait complete. 2023-10-16T15:34:14.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:34:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.235+0000 7f972c5a9640 1 Processor -- start 2023-10-16T15:34:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.236+0000 7f972c5a9640 1 -- start start 2023-10-16T15:34:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.237+0000 7f972c5a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f972406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.237+0000 7f972c5a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972415b310 con 0x7f97240b0360 2023-10-16T15:34:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.237+0000 7f972c5a9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972406f640 con 0x7f97240cac20 2023-10-16T15:34:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.237+0000 7f972c5a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97240cb050 con 0x7f972415ac10 2023-10-16T15:34:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.237+0000 7f9729b44640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f97240b0360 0x7f97240ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55292/0 (socket says 172.21.15.29:55292) 2023-10-16T15:34:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729b44640 1 -- 172.21.15.29:0/3474339878 learned_addr learned my addr 172.21.15.29:0/3474339878 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f972a345640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f972406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323070883 0 0) 0x7f972415b310 con 0x7f97240b0360 2023-10-16T15:34:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9710003520 con 0x7f97240b0360 2023-10-16T15:34:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3001043855 0 0) 0x7f9710003520 con 0x7f97240b0360 2023-10-16T15:34:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 msgr2=0x7f972406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f972406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 >> v1:172.21.15.170:6789/0 conn(0x7f97240cac20 legacy=0x7f972415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.238+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972415b5b0 con 0x7f97240b0360 2023-10-16T15:34:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9714001670 con 0x7f97240b0360 2023-10-16T15:34:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 >> v1:172.21.15.29:6789/0 conn(0x7f97240b0360 legacy=0x7f97240ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9710003a10 0x7f9710003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f97100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f9710005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97240cb050 con 0x7f9710004d00 2023-10-16T15:34:14.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971000bb10 con 0x7f9710003a10 2023-10-16T15:34:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971000bcd0 con 0x7f972415ac10 2023-10-16T15:34:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972ab46640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f9710005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972a345640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9710003a10 0x7f9710003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729b44640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f97100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972ab46640 1 -- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 msgr2=0x7f97100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972ab46640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f97100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972ab46640 1 -- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9710003a10 msgr2=0x7f9710003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:14.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972ab46640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9710003a10 0x7f9710003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f972ab46640 1 -- 172.21.15.29:0/3474339878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9710003520 con 0x7f9710004d00 2023-10-16T15:34:14.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.239+0000 7f9729b44640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415ac10 0x7f97100047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:14.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.240+0000 7f972ab46640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f9710005180 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f971c001750 tx=0x7f97100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11da9a860680dab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:14.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c0052d0 con 0x7f9710004d00 2023-10-16T15:34:14.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c02e050 con 0x7f9710004d00 2023-10-16T15:34:14.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f9729343640 1 -- 172.21.15.29:0/3474339878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c0035f0 con 0x7f9710004d00 2023-10-16T15:34:14.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f972c5a9640 1 -- 172.21.15.29:0/3474339878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 msgr2=0x7f9710005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f972c5a9640 1 --2- 172.21.15.29:0/3474339878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f9710005180 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f971c001750 tx=0x7f97100057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f972ab46640 1 -- 172.21.15.29:0/3474339878 reap_dead start 2023-10-16T15:34:14.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f972c5a9640 1 -- 172.21.15.29:0/3474339878 shutdown_connections 2023-10-16T15:34:14.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.241+0000 7f972c5a9640 1 -- 172.21.15.29:0/3474339878 >> 172.21.15.29:0/3474339878 conn(0x7f97240cb360 msgr2=0x7f97240cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:14.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.242+0000 7f972c5a9640 1 -- 172.21.15.29:0/3474339878 shutdown_connections 2023-10-16T15:34:14.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.242+0000 7f972c5a9640 1 -- 172.21.15.29:0/3474339878 wait complete. 2023-10-16T15:34:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.243+0000 7f972c5a9640 1 Processor -- start 2023-10-16T15:34:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.243+0000 7f972c5a9640 1 -- start start 2023-10-16T15:34:14.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.244+0000 7f972c5a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f97240c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.244+0000 7f972c5a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9724156880 0x7f9724155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.244+0000 7f972c5a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415d3f0 0x7f972415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.244+0000 7f972c5a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972415dde0 con 0x7f9710004d00 2023-10-16T15:34:14.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.244+0000 7f972c5a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97240c8180 con 0x7f9724156880 2023-10-16T15:34:14.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.244+0000 7f972c5a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97240c59b0 con 0x7f972415d3f0 2023-10-16T15:34:14.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f97240c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f97240c2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35454/0 (socket says 172.21.15.29:35454) 2023-10-16T15:34:14.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 -- 172.21.15.29:0/1378695638 learned_addr learned my addr 172.21.15.29:0/1378695638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f9729b44640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9724156880 0x7f9724155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 -- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415d3f0 msgr2=0x7f972415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972ab46640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415d3f0 0x7f972415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415d3f0 0x7f972415d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 -- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9724156880 msgr2=0x7f9724155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9724156880 0x7f9724155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f972a345640 1 -- 172.21.15.29:0/1378695638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97240b3170 con 0x7f9710004d00 2023-10-16T15:34:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.245+0000 7f9729b44640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9724156880 0x7f9724155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.246+0000 7f972a345640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f97240c2e20 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9720007f30 tx=0x7f9720011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.246+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972000e030 con 0x7f9710004d00 2023-10-16T15:34:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.246+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9720016470 con 0x7f9710004d00 2023-10-16T15:34:14.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.246+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97240c8390 con 0x7f9710004d00 2023-10-16T15:34:14.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.246+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720016820 con 0x7f9710004d00 2023-10-16T15:34:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.246+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9724156430 con 0x7f9710004d00 2023-10-16T15:34:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.247+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97240b09f0 con 0x7f9710004d00 2023-10-16T15:34:14.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.248+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9720016980 con 0x7f9710004d00 2023-10-16T15:34:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.249+0000 7f971affd640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f970806af90 0x7f970806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.249+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f972000f070 con 0x7f9710004d00 2023-10-16T15:34:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.250+0000 7f9729b44640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f970806af90 0x7f970806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.252+0000 7f9729b44640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f970806af90 0x7f970806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f97140017a0 tx=0x7f9714001470 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.254+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9720055680 con 0x7f9710004d00 2023-10-16T15:34:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.446+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f972404e160 con 0x7f9710004d00 2023-10-16T15:34:14.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.990+0000 7f971affd640 1 -- 172.21.15.29:0/1378695638 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v184) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9720058ca0 con 0x7f9710004d00 2023-10-16T15:34:14.993 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:34:14.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f970806af90 msgr2=0x7f970806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f970806af90 0x7f970806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f97140017a0 tx=0x7f9714001470 comp rx=0 tx=0).stop 2023-10-16T15:34:14.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 msgr2=0x7f97240c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:14.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f97240c2e20 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9720007f30 tx=0x7f9720011660 comp rx=0 tx=0).stop 2023-10-16T15:34:14.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 shutdown_connections 2023-10-16T15:34:14.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f970806af90 0x7f970806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f972415d3f0 0x7f972415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9724156880 0x7f9724155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 --2- 172.21.15.29:0/1378695638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9710004d00 0x7f97240c2e20 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:14.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.994+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 >> 172.21.15.29:0/1378695638 conn(0x7f97240cb360 msgr2=0x7f97240afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:14.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.996+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 shutdown_connections 2023-10-16T15:34:14.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:14.997+0000 7f972c5a9640 1 -- 172.21.15.29:0/1378695638 wait complete. 2023-10-16T15:34:15.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:34:15.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.241+0000 7f8eb759e640 1 Processor -- start 2023-10-16T15:34:15.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.242+0000 7f8eb759e640 1 -- start start 2023-10-16T15:34:15.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.243+0000 7f8eb759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8eb80ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.243+0000 7f8eb759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb80ca560 con 0x7f8eb80c9060 2023-10-16T15:34:15.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.243+0000 7f8eb759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb80c9950 con 0x7f8eb806fb50 2023-10-16T15:34:15.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.243+0000 7f8eb759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80c9c30 con 0x7f8eb806c9e0 2023-10-16T15:34:15.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.244+0000 7f8eb659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8eb80c9060 0x7f8eb806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55306/0 (socket says 172.21.15.29:55306) 2023-10-16T15:34:15.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.244+0000 7f8eb659c640 1 -- 172.21.15.29:0/221431056 learned_addr learned my addr 172.21.15.29:0/221431056 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.244+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629328967 0 0) 0x7f8eb80ca560 con 0x7f8eb80c9060 2023-10-16T15:34:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.244+0000 7f8eb6d9d640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8eb80ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.244+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea4003520 con 0x7f8eb80c9060 2023-10-16T15:34:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817889354 0 0) 0x7f8ea4003520 con 0x7f8eb80c9060 2023-10-16T15:34:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 msgr2=0x7f8eb80ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8eb80ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 >> v1:172.21.15.170:6789/0 conn(0x7f8eb806fb50 legacy=0x7f8eb80cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb80c9f10 con 0x7f8eb80c9060 2023-10-16T15:34:15.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8ea8001670 con 0x7f8eb80c9060 2023-10-16T15:34:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 >> v1:172.21.15.29:6789/0 conn(0x7f8eb80c9060 legacy=0x7f8eb806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8ea40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea4003f70 0x7f8ea40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4004d00 0x7f8ea4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80c9c30 con 0x7f8ea4003f70 2023-10-16T15:34:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea400bac0 con 0x7f8ea4004d00 2023-10-16T15:34:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea400bc80 con 0x7f8eb806c9e0 2023-10-16T15:34:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eaffff640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea4003f70 0x7f8ea40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.245+0000 7f8eb659c640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8ea40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.246+0000 7f8eaffff640 1 -- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 msgr2=0x7f8ea40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:15.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.246+0000 7f8eaffff640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8ea40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.246+0000 7f8eaffff640 1 -- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4004d00 msgr2=0x7f8ea4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:15.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.246+0000 7f8eaffff640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4004d00 0x7f8ea4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.246+0000 7f8eaffff640 1 -- 172.21.15.29:0/221431056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4003520 con 0x7f8ea4003f70 2023-10-16T15:34:15.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.246+0000 7f8eb659c640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb806c9e0 0x7f8ea40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:15.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.247+0000 7f8eaffff640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea4003f70 0x7f8ea40047c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0001750 tx=0x7f8ea40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715245075b1c8727 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:15.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.247+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea00052d0 con 0x7f8ea4003f70 2023-10-16T15:34:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.247+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea002e050 con 0x7f8ea4003f70 2023-10-16T15:34:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.247+0000 7f8eb5d9b640 1 -- 172.21.15.29:0/221431056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0003830 con 0x7f8ea4003f70 2023-10-16T15:34:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb759e640 1 -- 172.21.15.29:0/221431056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea4003f70 msgr2=0x7f8ea40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb759e640 1 --2- 172.21.15.29:0/221431056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea4003f70 0x7f8ea40047c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0001750 tx=0x7f8ea40057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb6d9d640 1 -- 172.21.15.29:0/221431056 reap_dead start 2023-10-16T15:34:15.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb759e640 1 -- 172.21.15.29:0/221431056 shutdown_connections 2023-10-16T15:34:15.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb759e640 1 -- 172.21.15.29:0/221431056 >> 172.21.15.29:0/221431056 conn(0x7f8eb80ce6a0 msgr2=0x7f8eb80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:15.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb759e640 1 -- 172.21.15.29:0/221431056 shutdown_connections 2023-10-16T15:34:15.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.248+0000 7f8eb759e640 1 -- 172.21.15.29:0/221431056 wait complete. 2023-10-16T15:34:15.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.250+0000 7f8eb759e640 1 Processor -- start 2023-10-16T15:34:15.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.250+0000 7f8eb759e640 1 -- start start 2023-10-16T15:34:15.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eb759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4003f70 0x7f8eb8156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eb759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 0x7f8eb80c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eb759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80b1450 0x7f8eb80b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eb759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80c41a0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eb759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80b1e10 con 0x7f8ea4003f70 2023-10-16T15:34:15.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eb759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb81fb200 con 0x7f8eb80b1450 2023-10-16T15:34:15.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 0x7f8eb80c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 0x7f8eb80c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35464/0 (socket says 172.21.15.29:35464) 2023-10-16T15:34:15.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.251+0000 7f8eaffff640 1 -- 172.21.15.29:0/1791843041 learned_addr learned my addr 172.21.15.29:0/1791843041 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:15.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eb659c640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4003f70 0x7f8eb8156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eb6d9d640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80b1450 0x7f8eb80b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaffff640 1 -- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80b1450 msgr2=0x7f8eb80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaffff640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80b1450 0x7f8eb80b18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaffff640 1 -- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4003f70 msgr2=0x7f8eb8156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaffff640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4003f70 0x7f8eb8156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaffff640 1 -- 172.21.15.29:0/1791843041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb8157200 con 0x7f8eb80c36c0 2023-10-16T15:34:15.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaffff640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 0x7f8eb80c3b40 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0008fd0 tx=0x7f8ea0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:15.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea00069e0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea0006b40 con 0x7f8eb80c36c0 2023-10-16T15:34:15.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.252+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea00035f0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.253+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb81fa440 con 0x7f8eb80c36c0 2023-10-16T15:34:15.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.253+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb81faa00 con 0x7f8eb80c36c0 2023-10-16T15:34:15.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.254+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb80c41a0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.255+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8ea0003750 con 0x7f8eb80c36c0 2023-10-16T15:34:15.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.256+0000 7f8eaeffd640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8e9806af90 0x7f8e9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.256+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8ea00ab760 con 0x7f8eb80c36c0 2023-10-16T15:34:15.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.257+0000 7f8eb659c640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8e9806af90 0x7f8e9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.258+0000 7f8eb659c640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8e9806af90 0x7f8e9806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8001200 tx=0x7f8ea8009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:15.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.260+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8eb80c41a0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.456+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8eb80c41a0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.463+0000 7f8eaeffd640 1 -- 172.21.15.29:0/1791843041 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8eb80c41a0 con 0x7f8eb80c36c0 2023-10-16T15:34:15.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8e9806af90 msgr2=0x7f8e9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8e9806af90 0x7f8e9806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8001200 tx=0x7f8ea8009040 comp rx=0 tx=0).stop 2023-10-16T15:34:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 msgr2=0x7f8eb80c3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 0x7f8eb80c3b40 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0008fd0 tx=0x7f8ea0001120 comp rx=0 tx=0).stop 2023-10-16T15:34:15.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 shutdown_connections 2023-10-16T15:34:15.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8e9806af90 0x7f8e9806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80c36c0 0x7f8eb80c3b40 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80b1450 0x7f8eb80b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 --2- 172.21.15.29:0/1791843041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ea4003f70 0x7f8eb8156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 >> 172.21.15.29:0/1791843041 conn(0x7f8eb80ce6a0 msgr2=0x7f8eb8160ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:15.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 shutdown_connections 2023-10-16T15:34:15.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.466+0000 7f8eb759e640 1 -- 172.21.15.29:0/1791843041 wait complete. 2023-10-16T15:34:15.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:34:15.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.707+0000 7f24c54c3640 1 Processor -- start 2023-10-16T15:34:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.707+0000 7f24c54c3640 1 -- start start 2023-10-16T15:34:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.709+0000 7f24c54c3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f24c0159ad0 0x7f24c0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.709+0000 7f24c54c3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24c015d650 0x7f24c006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.709+0000 7f24c54c3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c00cd260 con 0x7f24c006b3e0 2023-10-16T15:34:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.709+0000 7f24c54c3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c0071470 con 0x7f24c015d650 2023-10-16T15:34:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.709+0000 7f24c54c3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c015de30 con 0x7f24c0159ad0 2023-10-16T15:34:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.709+0000 7f24b7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24c015d650 0x7f24c006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24b7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f24c015d650 0x7f24c006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35480/0 (socket says 172.21.15.29:35480) 2023-10-16T15:34:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24b7fff640 1 -- 172.21.15.29:0/1520883975 learned_addr learned my addr 172.21.15.29:0/1520883975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24bffff640 1 --2- 172.21.15.29:0/1520883975 >> v2:172.21.15.170:3300/0 conn(0x7f24c0159ad0 0x7f24c0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24b7fff640 1 -- 172.21.15.29:0/1520883975 >> v2:172.21.15.170:3300/0 conn(0x7f24c0159ad0 msgr2=0x7f24c0070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24b7fff640 1 --2- 172.21.15.29:0/1520883975 >> v2:172.21.15.170:3300/0 conn(0x7f24c0159ad0 0x7f24c0070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24b7fff640 1 -- 172.21.15.29:0/1520883975 >> v1:172.21.15.29:6789/0 conn(0x7f24c006b3e0 legacy=0x7f24c00c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24bf7fe640 1 -- 172.21.15.29:0/1520883975 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099880867 0 0) 0x7f24c00cd260 con 0x7f24c006b3e0 2023-10-16T15:34:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.710+0000 7f24b7fff640 1 -- 172.21.15.29:0/1520883975 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c015e110 con 0x7f24c015d650 2023-10-16T15:34:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.711+0000 7f24b7fff640 1 --2- 172.21.15.29:0/1520883975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c015d650 0x7f24c006aea0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f24c006d010 tx=0x7f24a802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77e36da809063a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24bf7fe640 1 -- 172.21.15.29:0/1520883975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8030860 con 0x7f24c015d650 2023-10-16T15:34:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24bf7fe640 1 -- 172.21.15.29:0/1520883975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a80309c0 con 0x7f24c015d650 2023-10-16T15:34:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24bf7fe640 1 -- 172.21.15.29:0/1520883975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8030d50 con 0x7f24c015d650 2023-10-16T15:34:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 -- 172.21.15.29:0/1520883975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c015d650 msgr2=0x7f24c006aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 --2- 172.21.15.29:0/1520883975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c015d650 0x7f24c006aea0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f24c006d010 tx=0x7f24a802b9a0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 -- 172.21.15.29:0/1520883975 shutdown_connections 2023-10-16T15:34:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 --2- 172.21.15.29:0/1520883975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c015d650 0x7f24c006aea0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 --2- 172.21.15.29:0/1520883975 >> v2:172.21.15.170:3300/0 conn(0x7f24c0159ad0 0x7f24c0070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 -- 172.21.15.29:0/1520883975 >> 172.21.15.29:0/1520883975 conn(0x7f24c00b0170 msgr2=0x7f24c00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 -- 172.21.15.29:0/1520883975 shutdown_connections 2023-10-16T15:34:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.712+0000 7f24c54c3640 1 -- 172.21.15.29:0/1520883975 wait complete. 2023-10-16T15:34:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.714+0000 7f24c54c3640 1 Processor -- start 2023-10-16T15:34:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.714+0000 7f24c54c3640 1 -- start start 2023-10-16T15:34:15.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.715+0000 7f24c54c3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c006b3e0 0x7f24c01efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.715+0000 7f24c54c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 0x7f24c0157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.715+0000 7f24c54c3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24c015d650 0x7f24c00b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.715+0000 7f24c54c3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c01f0230 con 0x7f24c0159ad0 2023-10-16T15:34:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.715+0000 7f24c54c3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c01f38a0 con 0x7f24c015d650 2023-10-16T15:34:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.715+0000 7f24c54c3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c0157df0 con 0x7f24c006b3e0 2023-10-16T15:34:15.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 0x7f24c0157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 0x7f24c0157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35494/0 (socket says 172.21.15.29:35494) 2023-10-16T15:34:15.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 -- 172.21.15.29:0/2422938536 learned_addr learned my addr 172.21.15.29:0/2422938536 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:15.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 -- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c006b3e0 msgr2=0x7f24c01efcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c006b3e0 0x7f24c01efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 -- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24c015d650 msgr2=0x7f24c00b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24c015d650 0x7f24c00b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 -- 172.21.15.29:0/2422938536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80094d0 con 0x7f24c0159ad0 2023-10-16T15:34:15.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.716+0000 7f24b7fff640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 0x7f24c0157430 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f24a8000c00 tx=0x7f24a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:15.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.717+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a803e070 con 0x7f24c0159ad0 2023-10-16T15:34:15.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.717+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a8004030 con 0x7f24c0159ad0 2023-10-16T15:34:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.717+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a80043c0 con 0x7f24c0159ad0 2023-10-16T15:34:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.717+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c01f3af0 con 0x7f24c0159ad0 2023-10-16T15:34:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.717+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c01f40b0 con 0x7f24c0159ad0 2023-10-16T15:34:15.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.718+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c004e9b0 con 0x7f24c0159ad0 2023-10-16T15:34:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.719+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f24a8042070 con 0x7f24c0159ad0 2023-10-16T15:34:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.720+0000 7f24bd7fa640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f24a006af90 0x7f24a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.720+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f24a80ab6f0 con 0x7f24c0159ad0 2023-10-16T15:34:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.721+0000 7f24bffff640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f24a006af90 0x7f24a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:15.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.722+0000 7f24bffff640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f24a006af90 0x7f24a006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098f0 tx=0x7f24b0008040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.724+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f24a8076640 con 0x7f24c0159ad0 2023-10-16T15:34:15.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.921+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f24c00b4600 con 0x7f24c0159ad0 2023-10-16T15:34:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.928+0000 7f24bd7fa640 1 -- 172.21.15.29:0/2422938536 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v62) v1 ==== 115+0+0 (secure 0 0 0) 0x7f24a80340b0 con 0x7f24c0159ad0 2023-10-16T15:34:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f24a006af90 msgr2=0x7f24a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f24a006af90 0x7f24a006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098f0 tx=0x7f24b0008040 comp rx=0 tx=0).stop 2023-10-16T15:34:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 msgr2=0x7f24c0157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 0x7f24c0157430 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f24a8000c00 tx=0x7f24a80046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 shutdown_connections 2023-10-16T15:34:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f24a006af90 0x7f24a006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24c0159ad0 0x7f24c0157430 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24c015d650 0x7f24c00b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 --2- 172.21.15.29:0/2422938536 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24c006b3e0 0x7f24c01efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 >> 172.21.15.29:0/2422938536 conn(0x7f24c00b0170 msgr2=0x7f24c00ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 shutdown_connections 2023-10-16T15:34:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:15.932+0000 7f24c54c3640 1 -- 172.21.15.29:0/2422938536 wait complete. 2023-10-16T15:34:15.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:34:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.182+0000 7f131df54640 1 Processor -- start 2023-10-16T15:34:16.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.182+0000 7f131df54640 1 -- start start 2023-10-16T15:34:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.184+0000 7f131df54640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f131806c040 0x7f131806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.184+0000 7f131df54640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131815f4a0 con 0x7f13180cda80 2023-10-16T15:34:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.184+0000 7f131df54640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131806c960 con 0x7f13180ad970 2023-10-16T15:34:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.184+0000 7f131df54640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131815b9d0 con 0x7f131806c040 2023-10-16T15:34:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.185+0000 7f131d753640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f13180cda80 0x7f131815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55326/0 (socket says 172.21.15.29:55326) 2023-10-16T15:34:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.185+0000 7f131d753640 1 -- 172.21.15.29:0/1347791529 learned_addr learned my addr 172.21.15.29:0/1347791529 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.185+0000 7f131cf52640 1 --2- 172.21.15.29:0/1347791529 >> v2:172.21.15.170:3300/0 conn(0x7f131806c040 0x7f131806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.185+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969993267 0 0) 0x7f131815f4a0 con 0x7f13180cda80 2023-10-16T15:34:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.185+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1308003520 con 0x7f13180cda80 2023-10-16T15:34:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3056761664 0 0) 0x7f1308003520 con 0x7f13180cda80 2023-10-16T15:34:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 >> v2:172.21.15.170:3300/0 conn(0x7f131806c040 msgr2=0x7f131806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 --2- 172.21.15.29:0/1347791529 >> v2:172.21.15.170:3300/0 conn(0x7f131806c040 0x7f131806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 >> v1:172.21.15.170:6789/0 conn(0x7f13180ad970 legacy=0x7f131815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131815bcb0 con 0x7f13180cda80 2023-10-16T15:34:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f13100019f0 con 0x7f13180cda80 2023-10-16T15:34:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 >> v1:172.21.15.29:6789/0 conn(0x7f13180cda80 legacy=0x7f131815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13080039c0 0x7f1308003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13080047c0 0x7f1308004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1308005180 0x7f130800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131815b9d0 con 0x7f13080039c0 2023-10-16T15:34:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130800c080 con 0x7f13080047c0 2023-10-16T15:34:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1308010650 con 0x7f1308005180 2023-10-16T15:34:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.186+0000 7f131cf52640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13080039c0 0x7f1308003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f1317fff640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13080047c0 0x7f1308004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131cf52640 1 -- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1308005180 msgr2=0x7f130800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131d753640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1308005180 0x7f130800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131cf52640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1308005180 0x7f130800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131cf52640 1 -- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13080047c0 msgr2=0x7f1308004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131cf52640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13080047c0 0x7f1308004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131cf52640 1 -- 172.21.15.29:0/1347791529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1308003520 con 0x7f13080039c0 2023-10-16T15:34:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f1317fff640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13080047c0 0x7f1308004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.187+0000 7f131d753640 1 -- 172.21.15.29:0/1347791529 reap_dead start 2023-10-16T15:34:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.188+0000 7f131cf52640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13080039c0 0x7f1308003de0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f130c009ac0 tx=0x7f13080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690a7186b51e585e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.188+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c02c6b0 con 0x7f13080039c0 2023-10-16T15:34:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.188+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c02c810 con 0x7f13080039c0 2023-10-16T15:34:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.188+0000 7f13177fe640 1 -- 172.21.15.29:0/1347791529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c02cba0 con 0x7f13080039c0 2023-10-16T15:34:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 -- 172.21.15.29:0/1347791529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13080039c0 msgr2=0x7f1308003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13080039c0 0x7f1308003de0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f130c009ac0 tx=0x7f13080057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 -- 172.21.15.29:0/1347791529 shutdown_connections 2023-10-16T15:34:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 --2- 172.21.15.29:0/1347791529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13080039c0 0x7f1308003de0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 -- 172.21.15.29:0/1347791529 >> 172.21.15.29:0/1347791529 conn(0x7f13180ce6a0 msgr2=0x7f13180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 -- 172.21.15.29:0/1347791529 shutdown_connections 2023-10-16T15:34:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.189+0000 7f131df54640 1 -- 172.21.15.29:0/1347791529 wait complete. 2023-10-16T15:34:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.191+0000 7f131df54640 1 Processor -- start 2023-10-16T15:34:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.191+0000 7f131df54640 1 -- start start 2023-10-16T15:34:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131df54640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13080039c0 0x7f13180c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131df54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 0x7f1318148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131df54640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1318154b40 0x7f1318148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131df54640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1318155b60 con 0x7f13180c5770 2023-10-16T15:34:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131df54640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13180c21d0 con 0x7f1318154b40 2023-10-16T15:34:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131df54640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13180c6580 con 0x7f13080039c0 2023-10-16T15:34:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f1317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 0x7f1318148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f1317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 0x7f1318148420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35516/0 (socket says 172.21.15.29:35516) 2023-10-16T15:34:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f1317fff640 1 -- 172.21.15.29:0/1645350749 learned_addr learned my addr 172.21.15.29:0/1645350749 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131cf52640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13080039c0 0x7f13180c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.192+0000 7f131d753640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1318154b40 0x7f1318148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f1317fff640 1 -- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13080039c0 msgr2=0x7f13180c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f1317fff640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13080039c0 0x7f13180c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f1317fff640 1 -- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1318154b40 msgr2=0x7f1318148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f1317fff640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1318154b40 0x7f1318148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f1317fff640 1 -- 172.21.15.29:0/1645350749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318155810 con 0x7f13180c5770 2023-10-16T15:34:16.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f131cf52640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13080039c0 0x7f13180c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:16.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f1317fff640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 0x7f1318148420 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f130400d500 tx=0x7f130400d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:16.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304003610 con 0x7f13180c5770 2023-10-16T15:34:16.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304003770 con 0x7f13180c5770 2023-10-16T15:34:16.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304003b20 con 0x7f13180c5770 2023-10-16T15:34:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1318148ea0 con 0x7f13180c5770 2023-10-16T15:34:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.193+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13180c5fc0 con 0x7f13180c5770 2023-10-16T15:34:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.194+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131806b330 con 0x7f13180c5770 2023-10-16T15:34:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.195+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1304003c80 con 0x7f13180c5770 2023-10-16T15:34:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.196+0000 7f13157fa640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f12fc06af90 0x7f12fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.197+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1304089670 con 0x7f13180c5770 2023-10-16T15:34:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.197+0000 7f131cf52640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f12fc06af90 0x7f12fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.198+0000 7f131cf52640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f12fc06af90 0x7f12fc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f130c036000 tx=0x7f130c030000 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.201+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1304054680 con 0x7f13180c5770 2023-10-16T15:34:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.394+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1318155ca0 con 0x7f13180c5770 2023-10-16T15:34:16.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.399+0000 7f13157fa640 1 -- 172.21.15.29:0/1645350749 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1318155ca0 con 0x7f13180c5770 2023-10-16T15:34:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.402+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f12fc06af90 msgr2=0x7f12fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.402+0000 7f131df54640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f12fc06af90 0x7f12fc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f130c036000 tx=0x7f130c030000 comp rx=0 tx=0).stop 2023-10-16T15:34:16.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.402+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 msgr2=0x7f1318148420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.402+0000 7f131df54640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 0x7f1318148420 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f130400d500 tx=0x7f130400d9c0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.402+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 shutdown_connections 2023-10-16T15:34:16.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f12fc06af90 0x7f12fc06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13180c5770 0x7f1318148420 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1318154b40 0x7f1318148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 --2- 172.21.15.29:0/1645350749 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13080039c0 0x7f13180c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 >> 172.21.15.29:0/1645350749 conn(0x7f13180ce6a0 msgr2=0x7f13180cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:16.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 shutdown_connections 2023-10-16T15:34:16.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.403+0000 7f131df54640 1 -- 172.21.15.29:0/1645350749 wait complete. 2023-10-16T15:34:16.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T15:34:16.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.620+0000 7f278f59e640 1 Processor -- start 2023-10-16T15:34:16.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.620+0000 7f278f59e640 1 -- start start 2023-10-16T15:34:16.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.621+0000 7f278f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f27900b8480 0x7f27900cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.621+0000 7f278f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 0x7f27900cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.621+0000 7f278f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279015d150 con 0x7f279015cd70 2023-10-16T15:34:16.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.621+0000 7f278f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27900d1180 con 0x7f27900b8480 2023-10-16T15:34:16.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.621+0000 7f278f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27900d1460 con 0x7f27900d0be0 2023-10-16T15:34:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.622+0000 7f278e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f279015cd70 0x7f27900d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55338/0 (socket says 172.21.15.29:55338) 2023-10-16T15:34:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.622+0000 7f278e59c640 1 -- 172.21.15.29:0/2786602036 learned_addr learned my addr 172.21.15.29:0/2786602036 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.622+0000 7f278ed9d640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 0x7f27900cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.622+0000 7f278dd9b640 1 --2- 172.21.15.29:0/2786602036 >> v2:172.21.15.170:3300/0 conn(0x7f27900b8480 0x7f27900cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.622+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855298005 0 0) 0x7f279015d150 con 0x7f279015cd70 2023-10-16T15:34:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.622+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2778003520 con 0x7f279015cd70 2023-10-16T15:34:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997295974 0 0) 0x7f2778003520 con 0x7f279015cd70 2023-10-16T15:34:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 msgr2=0x7f27900cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 0x7f27900cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 >> v2:172.21.15.170:3300/0 conn(0x7f27900b8480 msgr2=0x7f27900cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 --2- 172.21.15.29:0/2786602036 >> v2:172.21.15.170:3300/0 conn(0x7f27900b8480 0x7f27900cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27900d1740 con 0x7f279015cd70 2023-10-16T15:34:16.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2780001670 con 0x7f279015cd70 2023-10-16T15:34:16.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 >> v1:172.21.15.29:6789/0 conn(0x7f279015cd70 legacy=0x7f27900d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2778003970 0x7f2778003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 0x7f27780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2778004d00 0x7f2778005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27900d1180 con 0x7f2778003970 2023-10-16T15:34:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27900d1460 con 0x7f2778004d00 2023-10-16T15:34:16.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277800b980 con 0x7f27900d0be0 2023-10-16T15:34:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.623+0000 7f278e59c640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2778003970 0x7f2778003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.624+0000 7f278ed9d640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2778004d00 0x7f2778005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.624+0000 7f278e59c640 1 -- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 msgr2=0x7f27780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.624+0000 7f278e59c640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f27900d0be0 0x7f27780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.624+0000 7f278e59c640 1 -- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2778004d00 msgr2=0x7f2778005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.624+0000 7f278e59c640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2778004d00 0x7f2778005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.624+0000 7f278e59c640 1 -- 172.21.15.29:0/2786602036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778003520 con 0x7f2778003970 2023-10-16T15:34:16.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.625+0000 7f278e59c640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2778003970 0x7f2778003d90 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f278000c110 tx=0x7f27780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510091cf34c86c85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:16.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.625+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780005ad0 con 0x7f2778003970 2023-10-16T15:34:16.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.625+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2780005c50 con 0x7f2778003970 2023-10-16T15:34:16.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.625+0000 7f278d59a640 1 -- 172.21.15.29:0/2786602036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278002c830 con 0x7f2778003970 2023-10-16T15:34:16.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278f59e640 1 -- 172.21.15.29:0/2786602036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2778003970 msgr2=0x7f2778003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278f59e640 1 --2- 172.21.15.29:0/2786602036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2778003970 0x7f2778003d90 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f278000c110 tx=0x7f27780057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278ed9d640 1 -- 172.21.15.29:0/2786602036 reap_dead start 2023-10-16T15:34:16.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278f59e640 1 -- 172.21.15.29:0/2786602036 shutdown_connections 2023-10-16T15:34:16.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278f59e640 1 -- 172.21.15.29:0/2786602036 >> 172.21.15.29:0/2786602036 conn(0x7f27900cf570 msgr2=0x7f27900cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:16.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278f59e640 1 -- 172.21.15.29:0/2786602036 shutdown_connections 2023-10-16T15:34:16.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.626+0000 7f278f59e640 1 -- 172.21.15.29:0/2786602036 wait complete. 2023-10-16T15:34:16.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.628+0000 7f278f59e640 1 Processor -- start 2023-10-16T15:34:16.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.628+0000 7f278f59e640 1 -- start start 2023-10-16T15:34:16.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2778003970 0x7f27900cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 0x7f27900aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f279015ea00 0x7f27900aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901f6370 con 0x7f279015e580 2023-10-16T15:34:16.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27900aabf0 con 0x7f279015ea00 2023-10-16T15:34:16.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27901f6090 con 0x7f2778003970 2023-10-16T15:34:16.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 0x7f27900aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 0x7f27900aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35532/0 (socket says 172.21.15.29:35532) 2023-10-16T15:34:16.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.629+0000 7f278dd9b640 1 -- 172.21.15.29:0/904759876 learned_addr learned my addr 172.21.15.29:0/904759876 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:16.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278e59c640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2778003970 0x7f27900cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278ed9d640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f279015ea00 0x7f27900aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278dd9b640 1 -- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2778003970 msgr2=0x7f27900cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278dd9b640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2778003970 0x7f27900cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278dd9b640 1 -- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f279015ea00 msgr2=0x7f27900aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:16.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278dd9b640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f279015ea00 0x7f27900aa6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:16.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278dd9b640 1 -- 172.21.15.29:0/904759876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27900a9690 con 0x7f279015e580 2023-10-16T15:34:16.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278e59c640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2778003970 0x7f27900cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278dd9b640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 0x7f27900aa170 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f278400e860 tx=0x7f278400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784009600 con 0x7f279015e580 2023-10-16T15:34:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278400c940 con 0x7f279015e580 2023-10-16T15:34:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278400ccd0 con 0x7f279015e580 2023-10-16T15:34:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.630+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27900a9970 con 0x7f279015e580 2023-10-16T15:34:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.631+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27900cc920 con 0x7f279015e580 2023-10-16T15:34:16.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.632+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27900a94f0 con 0x7f279015e580 2023-10-16T15:34:16.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.633+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2784013470 con 0x7f279015e580 2023-10-16T15:34:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.634+0000 7f277effd640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f277006af90 0x7f277006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.634+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2784089880 con 0x7f279015e580 2023-10-16T15:34:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.635+0000 7f278e59c640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f277006af90 0x7f277006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.636+0000 7f278e59c640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f277006af90 0x7f277006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2780034000 tx=0x7f2780007000 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:16.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.639+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f27900a94f0 con 0x7f279015e580 2023-10-16T15:34:16.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.903+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f27900a94f0 con 0x7f279015e580 2023-10-16T15:34:16.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.995+0000 7f277effd640 1 -- 172.21.15.29:0/904759876 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2553) v1 ==== 174+0+0 (secure 0 0 0) 0x7f27900a94f0 con 0x7f279015e580 2023-10-16T15:34:17.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f277006af90 msgr2=0x7f277006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f277006af90 0x7f277006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2780034000 tx=0x7f2780007000 comp rx=0 tx=0).stop 2023-10-16T15:34:17.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 msgr2=0x7f27900aa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 0x7f27900aa170 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f278400e860 tx=0x7f278400ed20 comp rx=0 tx=0).stop 2023-10-16T15:34:17.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 shutdown_connections 2023-10-16T15:34:17.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f277006af90 0x7f277006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f279015e580 0x7f27900aa170 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f279015ea00 0x7f27900aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 --2- 172.21.15.29:0/904759876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2778003970 0x7f27900cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 >> 172.21.15.29:0/904759876 conn(0x7f27900cf570 msgr2=0x7f27900d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 shutdown_connections 2023-10-16T15:34:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:16.999+0000 7f278f59e640 1 -- 172.21.15.29:0/904759876 wait complete. 2023-10-16T15:34:17.023 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-16T15:34:17.023 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-16T15:34:17.024 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T15:34:17.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.242+0000 7f8b6f59e640 1 Processor -- start 2023-10-16T15:34:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.242+0000 7f8b6f59e640 1 -- start start 2023-10-16T15:34:17.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b700c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b7015dde0 0x7f8b700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b70160a90 con 0x7f8b700c9230 2023-10-16T15:34:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b700cdec0 con 0x7f8b7015dde0 2023-10-16T15:34:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700cd900 con 0x7f8b7015c9f0 2023-10-16T15:34:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b7015dde0 0x7f8b700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b7015dde0 0x7f8b700ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35546/0 (socket says 172.21.15.29:35546) 2023-10-16T15:34:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8b700c9230 0x7f8b700709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55340/0 (socket says 172.21.15.29:55340) 2023-10-16T15:34:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.244+0000 7f8b6ed9d640 1 -- 172.21.15.29:0/1534822481 learned_addr learned my addr 172.21.15.29:0/1534822481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6e59c640 1 --2- 172.21.15.29:0/1534822481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b700c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6d59a640 1 -- 172.21.15.29:0/1534822481 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081983186 0 0) 0x7f8b70160a90 con 0x7f8b700c9230 2023-10-16T15:34:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6d59a640 1 -- 172.21.15.29:0/1534822481 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5c003520 con 0x7f8b700c9230 2023-10-16T15:34:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6ed9d640 1 -- 172.21.15.29:0/1534822481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 msgr2=0x7f8b700c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6ed9d640 1 --2- 172.21.15.29:0/1534822481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b700c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6ed9d640 1 -- 172.21.15.29:0/1534822481 >> v1:172.21.15.29:6789/0 conn(0x7f8b700c9230 legacy=0x7f8b700709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.245+0000 7f8b6ed9d640 1 -- 172.21.15.29:0/1534822481 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b700cdbe0 con 0x7f8b7015dde0 2023-10-16T15:34:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.246+0000 7f8b6ed9d640 1 --2- 172.21.15.29:0/1534822481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7015dde0 0x7f8b700ac890 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009870 tx=0x7f8b6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2d26d25feba0f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6d59a640 1 -- 172.21.15.29:0/1534822481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68030840 con 0x7f8b7015dde0 2023-10-16T15:34:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6d59a640 1 -- 172.21.15.29:0/1534822481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b680309a0 con 0x7f8b7015dde0 2023-10-16T15:34:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6d59a640 1 -- 172.21.15.29:0/1534822481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68030d30 con 0x7f8b7015dde0 2023-10-16T15:34:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 -- 172.21.15.29:0/1534822481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7015dde0 msgr2=0x7f8b700ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/1534822481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7015dde0 0x7f8b700ac890 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009870 tx=0x7f8b6802b980 comp rx=0 tx=0).stop 2023-10-16T15:34:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 -- 172.21.15.29:0/1534822481 shutdown_connections 2023-10-16T15:34:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/1534822481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7015dde0 0x7f8b700ac890 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/1534822481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b700c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 -- 172.21.15.29:0/1534822481 >> 172.21.15.29:0/1534822481 conn(0x7f8b700ce6a0 msgr2=0x7f8b700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 -- 172.21.15.29:0/1534822481 shutdown_connections 2023-10-16T15:34:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.247+0000 7f8b6f59e640 1 -- 172.21.15.29:0/1534822481 wait complete. 2023-10-16T15:34:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.249+0000 7f8b6f59e640 1 Processor -- start 2023-10-16T15:34:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.249+0000 7f8b6f59e640 1 -- start start 2023-10-16T15:34:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.250+0000 7f8b6f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 0x7f8b700c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.250+0000 7f8b6f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b7006bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.250+0000 7f8b6f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7015dde0 0x7f8b7006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.250+0000 7f8b6f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b701f2c20 con 0x7f8b700c9230 2023-10-16T15:34:17.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.250+0000 7f8b6f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b701f22f0 con 0x7f8b7015dde0 2023-10-16T15:34:17.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.250+0000 7f8b6f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b701f25d0 con 0x7f8b7015c9f0 2023-10-16T15:34:17.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 0x7f8b700c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 0x7f8b700c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35552/0 (socket says 172.21.15.29:35552) 2023-10-16T15:34:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 -- 172.21.15.29:0/3628626226 learned_addr learned my addr 172.21.15.29:0/3628626226 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6dd9b640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b7006bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6ed9d640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7015dde0 0x7f8b7006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 -- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 msgr2=0x7f8b7006bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b7006bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 -- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7015dde0 msgr2=0x7f8b7006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7015dde0 0x7f8b7006c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 -- 172.21.15.29:0/3628626226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b680094d0 con 0x7f8b700c9230 2023-10-16T15:34:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6dd9b640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b7006bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.251+0000 7f8b6e59c640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 0x7f8b700c7100 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000e9b0 tx=0x7f8b6000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.252+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b60009670 con 0x7f8b700c9230 2023-10-16T15:34:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.252+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6000c9a0 con 0x7f8b700c9230 2023-10-16T15:34:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.252+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000cd50 con 0x7f8b700c9230 2023-10-16T15:34:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.252+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b701f28b0 con 0x7f8b700c9230 2023-10-16T15:34:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.252+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b70070f20 con 0x7f8b700c9230 2023-10-16T15:34:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.253+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b701f2a40 con 0x7f8b700c9230 2023-10-16T15:34:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.254+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b60013470 con 0x7f8b700c9230 2023-10-16T15:34:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.255+0000 7f8b66ffd640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8b5006af90 0x7f8b5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.255+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8b60089ab0 con 0x7f8b700c9230 2023-10-16T15:34:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.256+0000 7f8b6dd9b640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8b5006af90 0x7f8b5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.257+0000 7f8b6dd9b640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8b5006af90 0x7f8b5006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8b700c5720 tx=0x7f8b58009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:17.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.260+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b600549e0 con 0x7f8b700c9230 2023-10-16T15:34:17.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.455+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b7004e9b0 con 0x7f8b700c9230 2023-10-16T15:34:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.456+0000 7f8b66ffd640 1 -- 172.21.15.29:0/3628626226 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+14059 (secure 0 0 0) 0x7f8b60017090 con 0x7f8b700c9230 2023-10-16T15:34:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:17.459 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":184,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:14.987789+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6833/3104953668":"2023-10-17T15:34:11.964045+0000","172.21.15.29:6832/3104953668":"2023-10-17T15:34:11.964045+0000","172.21.15.170:0/1288768965":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2519294017":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/1563871353":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2058472509":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2172765430":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2009585372":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/838551000":"2023-10-17T15:33:59.835992+0000","172.21.15.170:6833/3772833058":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6833/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/1144059255":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6832/1981883088":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3011898787":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/3298516092":"2023-10-17T15:34:07.842840+0000","172.21.15.170:6832/3772833058":"2023-10-17T15:34:07.842840+0000","172.21.15.170:0/2177136748":"2023-10-17T15:33:59.835992+0000","172.21.15.170:0/2616191068":"2023-10-17T15:33:59.835992+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8b5006af90 msgr2=0x7f8b5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8b5006af90 0x7f8b5006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8b700c5720 tx=0x7f8b58009040 comp rx=0 tx=0).stop 2023-10-16T15:34:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 msgr2=0x7f8b700c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 0x7f8b700c7100 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000e9b0 tx=0x7f8b6000ee70 comp rx=0 tx=0).stop 2023-10-16T15:34:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 shutdown_connections 2023-10-16T15:34:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f8b5006af90 0x7f8b5006d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b7015dde0 0x7f8b7006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b7015c9f0 0x7f8b7006bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 --2- 172.21.15.29:0/3628626226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b700c9230 0x7f8b700c7100 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 >> 172.21.15.29:0/3628626226 conn(0x7f8b700ce6a0 msgr2=0x7f8b7015a410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 shutdown_connections 2023-10-16T15:34:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.459+0000 7f8b6f59e640 1 -- 172.21.15.29:0/3628626226 wait complete. 2023-10-16T15:34:17.481 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:34:17.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:34:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.712+0000 7fed4de52640 1 Processor -- start 2023-10-16T15:34:17.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.713+0000 7fed4de52640 1 -- start start 2023-10-16T15:34:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.714+0000 7fed4de52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed480c9de0 0x7fed480ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.714+0000 7fed4de52640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed480cd260 con 0x7fed48070f30 2023-10-16T15:34:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.714+0000 7fed4de52640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4815de30 con 0x7fed48159ad0 2023-10-16T15:34:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.714+0000 7fed4de52640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed480ca790 con 0x7fed480c9de0 2023-10-16T15:34:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.715+0000 7fed4d651640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed480c9de0 0x7fed480ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.715+0000 7fed3ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed48070f30 0x7fed4806aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55350/0 (socket says 172.21.15.29:55350) 2023-10-16T15:34:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.715+0000 7fed3ffff640 1 -- 172.21.15.29:0/3105555464 learned_addr learned my addr 172.21.15.29:0/3105555464 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.715+0000 7fed4d651640 1 -- 172.21.15.29:0/3105555464 >> v1:172.21.15.170:6789/0 conn(0x7fed48159ad0 legacy=0x7fed4815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.715+0000 7fed4d651640 1 -- 172.21.15.29:0/3105555464 >> v1:172.21.15.29:6789/0 conn(0x7fed48070f30 legacy=0x7fed4806aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.715+0000 7fed4d651640 1 -- 172.21.15.29:0/3105555464 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4815e110 con 0x7fed480c9de0 2023-10-16T15:34:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.716+0000 7fed4d651640 1 --2- 172.21.15.29:0/3105555464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed480c9de0 0x7fed480ca250 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fed40009d90 tx=0x7fed4002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa78557bfa9aa18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.717+0000 7fed3f7fe640 1 -- 172.21.15.29:0/3105555464 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4002cc60 con 0x7fed480c9de0 2023-10-16T15:34:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.717+0000 7fed3f7fe640 1 -- 172.21.15.29:0/3105555464 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4002cdc0 con 0x7fed480c9de0 2023-10-16T15:34:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.717+0000 7fed3f7fe640 1 -- 172.21.15.29:0/3105555464 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40034660 con 0x7fed480c9de0 2023-10-16T15:34:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.717+0000 7fed4de52640 1 -- 172.21.15.29:0/3105555464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed480c9de0 msgr2=0x7fed480ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.717+0000 7fed4de52640 1 --2- 172.21.15.29:0/3105555464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed480c9de0 0x7fed480ca250 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fed40009d90 tx=0x7fed4002beb0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.718+0000 7fed4de52640 1 -- 172.21.15.29:0/3105555464 shutdown_connections 2023-10-16T15:34:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.718+0000 7fed4de52640 1 --2- 172.21.15.29:0/3105555464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed480c9de0 0x7fed480ca250 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.718+0000 7fed4de52640 1 -- 172.21.15.29:0/3105555464 >> 172.21.15.29:0/3105555464 conn(0x7fed480b0170 msgr2=0x7fed480b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.718+0000 7fed4de52640 1 -- 172.21.15.29:0/3105555464 shutdown_connections 2023-10-16T15:34:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.718+0000 7fed4de52640 1 -- 172.21.15.29:0/3105555464 wait complete. 2023-10-16T15:34:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.719+0000 7fed4de52640 1 Processor -- start 2023-10-16T15:34:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.720+0000 7fed4de52640 1 -- start start 2023-10-16T15:34:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.720+0000 7fed4de52640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed48070f30 0x7fed480b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.720+0000 7fed4de52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed480c9de0 0x7fed48156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4de52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 0x7fed481e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4de52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480b32d0 con 0x7fed48159ad0 2023-10-16T15:34:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4de52640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480b2670 con 0x7fed48070f30 2023-10-16T15:34:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4de52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480b2930 con 0x7fed480c9de0 2023-10-16T15:34:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 0x7fed481e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 0x7fed481e7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35564/0 (socket says 172.21.15.29:35564) 2023-10-16T15:34:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 -- 172.21.15.29:0/2845647199 learned_addr learned my addr 172.21.15.29:0/2845647199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 -- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed480c9de0 msgr2=0x7fed48156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed480c9de0 0x7fed48156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 -- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed48070f30 msgr2=0x7fed480b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed48070f30 0x7fed480b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.721+0000 7fed4d651640 1 -- 172.21.15.29:0/2845647199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40009a70 con 0x7fed48159ad0 2023-10-16T15:34:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.722+0000 7fed4d651640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 0x7fed481e7a60 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fed40009d90 tx=0x7fed40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.722+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40040450 con 0x7fed48159ad0 2023-10-16T15:34:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.722+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed400405d0 con 0x7fed48159ad0 2023-10-16T15:34:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.722+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40040980 con 0x7fed48159ad0 2023-10-16T15:34:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.722+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed480b2c10 con 0x7fed48159ad0 2023-10-16T15:34:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.722+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed48156980 con 0x7fed48159ad0 2023-10-16T15:34:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.723+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48156b40 con 0x7fed48159ad0 2023-10-16T15:34:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.724+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed40040ae0 con 0x7fed48159ad0 2023-10-16T15:34:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.725+0000 7fed3d7fa640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fed3406af90 0x7fed3406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.725+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fed400ac2e0 con 0x7fed48159ad0 2023-10-16T15:34:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.726+0000 7fed4ce50640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fed3406af90 0x7fed3406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.727+0000 7fed4ce50640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fed3406af90 0x7fed3406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed44006fd0 tx=0x7fed44006bc0 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:17.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.730+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fed400772d0 con 0x7fed48159ad0 2023-10-16T15:34:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.923+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed4804e9b0 con 0x7fed48159ad0 2023-10-16T15:34:17.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.923+0000 7fed3d7fa640 1 -- 172.21.15.29:0/2845647199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed4007a8f0 con 0x7fed48159ad0 2023-10-16T15:34:17.926 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:34:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fed3406af90 msgr2=0x7fed3406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fed3406af90 0x7fed3406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed44006fd0 tx=0x7fed44006bc0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 msgr2=0x7fed481e7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 0x7fed481e7a60 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fed40009d90 tx=0x7fed40004a40 comp rx=0 tx=0).stop 2023-10-16T15:34:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 shutdown_connections 2023-10-16T15:34:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fed3406af90 0x7fed3406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed48159ad0 0x7fed481e7a60 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed480c9de0 0x7fed48156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 --2- 172.21.15.29:0/2845647199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed48070f30 0x7fed480b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 >> 172.21.15.29:0/2845647199 conn(0x7fed480b0170 msgr2=0x7fed480c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 shutdown_connections 2023-10-16T15:34:17.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:17.926+0000 7fed4de52640 1 -- 172.21.15.29:0/2845647199 wait complete. 2023-10-16T15:34:17.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T15:34:18.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.169+0000 7f7c2eb59640 1 Processor -- start 2023-10-16T15:34:18.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.169+0000 7f7c2eb59640 1 -- start start 2023-10-16T15:34:18.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2eb59640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c280cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2eb59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c280ccfb0 0x7f7c280cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2eb59640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2815b9d0 con 0x7f7c2806c040 2023-10-16T15:34:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2eb59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2815bcb0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2eb59640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2815bf90 con 0x7f7c280ad970 2023-10-16T15:34:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2e358640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c280ccfb0 0x7f7c280cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:18.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2e358640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c280ccfb0 0x7f7c280cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35566/0 (socket says 172.21.15.29:35566) 2023-10-16T15:34:18.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.170+0000 7f7c2e358640 1 -- 172.21.15.29:0/1652352454 learned_addr learned my addr 172.21.15.29:0/1652352454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:18.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.171+0000 7f7c2d356640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c280cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.171+0000 7f7c2e358640 1 -- 172.21.15.29:0/1652352454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 msgr2=0x7f7c280cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.171+0000 7f7c2e358640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c280cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.171+0000 7f7c2e358640 1 -- 172.21.15.29:0/1652352454 >> v1:172.21.15.170:6789/0 conn(0x7f7c2806c040 legacy=0x7f7c2806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.171+0000 7f7c2e358640 1 -- 172.21.15.29:0/1652352454 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c280cc270 con 0x7f7c280ccfb0 2023-10-16T15:34:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.171+0000 7f7c2d356640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c280cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.172+0000 7f7c2e358640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c280cd440 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009e80 tx=0x7f7c2002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5af7abf50a0006 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.172+0000 7f7c2cb55640 1 -- 172.21.15.29:0/1652352454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2002ca50 con 0x7f7c280ccfb0 2023-10-16T15:34:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.172+0000 7f7c2cb55640 1 -- 172.21.15.29:0/1652352454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2002cbb0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.172+0000 7f7c2cb55640 1 -- 172.21.15.29:0/1652352454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c200343f0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1652352454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 msgr2=0x7f7c280cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c280cd440 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009e80 tx=0x7f7c2002beb0 comp rx=0 tx=0).stop 2023-10-16T15:34:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1652352454 shutdown_connections 2023-10-16T15:34:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c280cd440 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1652352454 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c280cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1652352454 >> 172.21.15.29:0/1652352454 conn(0x7f7c280ce6a0 msgr2=0x7f7c280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1652352454 shutdown_connections 2023-10-16T15:34:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.173+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1652352454 wait complete. 2023-10-16T15:34:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.175+0000 7f7c2eb59640 1 Processor -- start 2023-10-16T15:34:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.175+0000 7f7c2eb59640 1 -- start start 2023-10-16T15:34:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2eb59640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c2806c040 0x7f7c280ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2eb59640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c28145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2eb59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c281f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2eb59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c280c6dc0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2eb59640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c280c6a70 con 0x7f7c2806c040 2023-10-16T15:34:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2eb59640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281f0780 con 0x7f7c280ad970 2023-10-16T15:34:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2e358640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c281f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2e358640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c281f0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35578/0 (socket says 172.21.15.29:35578) 2023-10-16T15:34:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2e358640 1 -- 172.21.15.29:0/1142468952 learned_addr learned my addr 172.21.15.29:0/1142468952 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2d356640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c28145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.176+0000 7f7c2db57640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c2806c040 0x7f7c280ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2e358640 1 -- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 msgr2=0x7f7c28145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2e358640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c28145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2e358640 1 -- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c2806c040 msgr2=0x7f7c280ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2e358640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c2806c040 0x7f7c280ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2e358640 1 -- 172.21.15.29:0/1142468952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c20009b30 con 0x7f7c280ccfb0 2023-10-16T15:34:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2e358640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c281f0240 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f7c2002c4c0 tx=0x7f7c20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20040670 con 0x7f7c280ccfb0 2023-10-16T15:34:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c200407f0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20040ba0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.177+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c281f0a50 con 0x7f7c280ccfb0 2023-10-16T15:34:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.178+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c281f1010 con 0x7f7c280ccfb0 2023-10-16T15:34:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.179+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2806b330 con 0x7f7c280ccfb0 2023-10-16T15:34:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.180+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7c20040d00 con 0x7f7c280ccfb0 2023-10-16T15:34:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.181+0000 7f7c1e7fc640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7c0c06af90 0x7f7c0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.181+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f7c200ac8a0 con 0x7f7c280ccfb0 2023-10-16T15:34:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.181+0000 7f7c2db57640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7c0c06af90 0x7f7c0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.182+0000 7f7c2db57640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7c0c06af90 0x7f7c0c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c28158bf0 tx=0x7f7c24009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.185+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7c20077870 con 0x7f7c280ccfb0 2023-10-16T15:34:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:18.451+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f7c280c6f00 con 0x7f7c280ccfb0 2023-10-16T15:34:19.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.007+0000 7f7c1e7fc640 1 -- 172.21.15.29:0/1142468952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2555) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7c280c6f00 con 0x7f7c280ccfb0 2023-10-16T15:34:19.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.012+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7c0c06af90 msgr2=0x7f7c0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.012+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7c0c06af90 0x7f7c0c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c28158bf0 tx=0x7f7c24009040 comp rx=0 tx=0).stop 2023-10-16T15:34:19.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.012+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 msgr2=0x7f7c281f0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.012+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c281f0240 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f7c2002c4c0 tx=0x7f7c20004a40 comp rx=0 tx=0).stop 2023-10-16T15:34:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 shutdown_connections 2023-10-16T15:34:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7c0c06af90 0x7f7c0c06d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c280ccfb0 0x7f7c281f0240 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c280ad970 0x7f7c28145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 --2- 172.21.15.29:0/1142468952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c2806c040 0x7f7c280ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 >> 172.21.15.29:0/1142468952 conn(0x7f7c280ce6a0 msgr2=0x7f7c280b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 shutdown_connections 2023-10-16T15:34:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.013+0000 7f7c2eb59640 1 -- 172.21.15.29:0/1142468952 wait complete. 2023-10-16T15:34:19.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T15:34:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.270+0000 7f9ca0e4f640 1 Processor -- start 2023-10-16T15:34:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.271+0000 7f9ca0e4f640 1 -- start start 2023-10-16T15:34:19.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.272+0000 7f9ca0e4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 0x7f9c9c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.272+0000 7f9ca0e4f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0cb030 con 0x7f9c9c0c9ca0 2023-10-16T15:34:19.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.272+0000 7f9ca0e4f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0c91c0 con 0x7f9c9c15ad70 2023-10-16T15:34:19.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.272+0000 7f9ca0e4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0c94a0 con 0x7f9c9c06a4b0 2023-10-16T15:34:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.273+0000 7f9c9ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9c9c0c9ca0 0x7f9c9c0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55352/0 (socket says 172.21.15.29:55352) 2023-10-16T15:34:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.273+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/3682315247 learned_addr learned my addr 172.21.15.29:0/3682315247 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.273+0000 7f9c93fff640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 0x7f9c9c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.273+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398898621 0 0) 0x7f9c9c0cb030 con 0x7f9c9c0c9ca0 2023-10-16T15:34:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.273+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c88003520 con 0x7f9c9c0c9ca0 2023-10-16T15:34:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133205064 0 0) 0x7f9c88003520 con 0x7f9c9c0c9ca0 2023-10-16T15:34:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 msgr2=0x7f9c9c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 0x7f9c9c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 >> v1:172.21.15.170:6789/0 conn(0x7f9c9c15ad70 legacy=0x7f9c9c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0c9780 con 0x7f9c9c0c9ca0 2023-10-16T15:34:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9c940019f0 con 0x7f9c9c0c9ca0 2023-10-16T15:34:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 >> v1:172.21.15.29:6789/0 conn(0x7f9c9c0c9ca0 legacy=0x7f9c9c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c880039a0 0x7f9c88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 0x7f9c880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88004d00 0x7f9c88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0c94a0 con 0x7f9c88004d00 2023-10-16T15:34:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8800ba30 con 0x7f9c880039a0 2023-10-16T15:34:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8800bbf0 con 0x7f9c9c06a4b0 2023-10-16T15:34:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88004d00 0x7f9c88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.274+0000 7f9c9a59c640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c880039a0 0x7f9c88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.275+0000 7f9c93fff640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 0x7f9c880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.275+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 msgr2=0x7f9c880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.275+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c9c06a4b0 0x7f9c880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.275+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c880039a0 msgr2=0x7f9c88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.275+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c880039a0 0x7f9c88003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.275+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/3682315247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88003520 con 0x7f9c88004d00 2023-10-16T15:34:19.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.276+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88004d00 0x7f9c88005180 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400a7f0 tx=0x7f9c880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294ddd2248aaff7e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.276+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c940072d0 con 0x7f9c88004d00 2023-10-16T15:34:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.276+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9402f070 con 0x7f9c88004d00 2023-10-16T15:34:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.276+0000 7f9c99d9b640 1 -- 172.21.15.29:0/3682315247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c940076b0 con 0x7f9c88004d00 2023-10-16T15:34:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/3682315247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88004d00 msgr2=0x7f9c88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/3682315247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88004d00 0x7f9c88005180 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400a7f0 tx=0x7f9c880057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/3682315247 reap_dead start 2023-10-16T15:34:19.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/3682315247 shutdown_connections 2023-10-16T15:34:19.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/3682315247 >> 172.21.15.29:0/3682315247 conn(0x7f9c9c0ce9b0 msgr2=0x7f9c9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:19.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/3682315247 shutdown_connections 2023-10-16T15:34:19.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.277+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/3682315247 wait complete. 2023-10-16T15:34:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.279+0000 7f9ca0e4f640 1 Processor -- start 2023-10-16T15:34:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.279+0000 7f9ca0e4f640 1 -- start start 2023-10-16T15:34:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9ca0e4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c88004d00 0x7f9c9c14aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9ca0e4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c9c0a85d0 0x7f9c9c0b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9ca0e4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 0x7f9c9c0b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9ca0e4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1f1c90 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9ca0e4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c14b3e0 con 0x7f9c9c0a85d0 2023-10-16T15:34:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9ca0e4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0a9020 con 0x7f9c88004d00 2023-10-16T15:34:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9c9ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 0x7f9c9c0b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9c9ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 0x7f9c9c0b1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35600/0 (socket says 172.21.15.29:35600) 2023-10-16T15:34:19.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.280+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/2789664001 learned_addr learned my addr 172.21.15.29:0/2789664001 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:19.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c93fff640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c9c0a85d0 0x7f9c9c0b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9a59c640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c88004d00 0x7f9c9c14aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c88004d00 msgr2=0x7f9c9c14aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c88004d00 0x7f9c9c14aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c9c0a85d0 msgr2=0x7f9c9c0b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c9c0a85d0 0x7f9c9c0b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9ad9d640 1 -- 172.21.15.29:0/2789664001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c1f1360 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9a59c640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c88004d00 0x7f9c9c14aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:19.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c93fff640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c9c0a85d0 0x7f9c9c0b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:19.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c9ad9d640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 0x7f9c9c0b1b10 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f9c94001cb0 tx=0x7f9c9c0ca130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:19.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.281+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94004d60 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.282+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94001250 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.282+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c940072d0 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.282+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c1f1640 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.282+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c1f44b0 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.283+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c04e160 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.284+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9c94031030 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.285+0000 7f9c92ffd640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f9c7c06af90 0x7f9c7c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.285+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f9c940ab620 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.285+0000 7f9c9a59c640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f9c7c06af90 0x7f9c7c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.287+0000 7f9c9a59c640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f9c7c06af90 0x7f9c7c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0016b0 tx=0x7f9c8c009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.289+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c94079ed0 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.481+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9c9c1f1dd0 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.489+0000 7f9c92ffd640 1 -- 172.21.15.29:0/2789664001 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9c94079a60 con 0x7f9c9c0a8a50 2023-10-16T15:34:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.491+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f9c7c06af90 msgr2=0x7f9c7c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.491+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f9c7c06af90 0x7f9c7c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0016b0 tx=0x7f9c8c009040 comp rx=0 tx=0).stop 2023-10-16T15:34:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.491+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 msgr2=0x7f9c9c0b1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.491+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 0x7f9c9c0b1b10 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f9c94001cb0 tx=0x7f9c9c0ca130 comp rx=0 tx=0).stop 2023-10-16T15:34:19.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 shutdown_connections 2023-10-16T15:34:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f9c7c06af90 0x7f9c7c06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c9c0a8a50 0x7f9c9c0b1b10 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c9c0a85d0 0x7f9c9c0b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 --2- 172.21.15.29:0/2789664001 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c88004d00 0x7f9c9c14aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 >> 172.21.15.29:0/2789664001 conn(0x7f9c9c0ce9b0 msgr2=0x7f9c9c0ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 shutdown_connections 2023-10-16T15:34:19.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:19.492+0000 7f9ca0e4f640 1 -- 172.21.15.29:0/2789664001 wait complete. 2023-10-16T15:34:19.514 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:34:19.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:34:19.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:34:19.772 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:34:19.773 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:34:19.773 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:19.838 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:34:19.838 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:34:19.839 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:19.900 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:34:19.900 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:34:19.901 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:19.963 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:34:19.964 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:34:19.964 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:20.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:34:20.026 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T15:34:20.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:20.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.266+0000 7fd459090640 1 Processor -- start 2023-10-16T15:34:20.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.266+0000 7fd459090640 1 -- start start 2023-10-16T15:34:20.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.267+0000 7fd459090640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd45406c040 0x7fd45406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.267+0000 7fd459090640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4540cda80 0x7fd45415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.267+0000 7fd459090640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45406c960 con 0x7fd4540ad970 2023-10-16T15:34:20.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.267+0000 7fd459090640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45415b9d0 con 0x7fd4540cda80 2023-10-16T15:34:20.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.267+0000 7fd459090640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45415bcb0 con 0x7fd45406c040 2023-10-16T15:34:20.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4540cda80 0x7fd45415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4540cda80 0x7fd45415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35616/0 (socket says 172.21.15.29:35616) 2023-10-16T15:34:20.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 -- 172.21.15.29:0/4115318709 learned_addr learned my addr 172.21.15.29:0/4115318709 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:20.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd453fff640 1 --2- 172.21.15.29:0/4115318709 >> v2:172.21.15.170:3300/0 conn(0x7fd45406c040 0x7fd45406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 -- 172.21.15.29:0/4115318709 >> v2:172.21.15.170:3300/0 conn(0x7fd45406c040 msgr2=0x7fd45406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:20.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 --2- 172.21.15.29:0/4115318709 >> v2:172.21.15.170:3300/0 conn(0x7fd45406c040 0x7fd45406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 -- 172.21.15.29:0/4115318709 >> v1:172.21.15.170:6789/0 conn(0x7fd4540ad970 legacy=0x7fd45415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:20.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd45888f640 1 -- 172.21.15.29:0/4115318709 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45415bf90 con 0x7fd4540cda80 2023-10-16T15:34:20.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.268+0000 7fd453fff640 1 --2- 172.21.15.29:0/4115318709 >> v2:172.21.15.170:3300/0 conn(0x7fd45406c040 0x7fd45406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:20.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.269+0000 7fd45888f640 1 --2- 172.21.15.29:0/4115318709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415ed90 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fd4540cf350 tx=0x7fd44c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7668d4ccdf342bf6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:20.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.270+0000 7fd452ffd640 1 -- 172.21.15.29:0/4115318709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c031870 con 0x7fd4540cda80 2023-10-16T15:34:20.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.270+0000 7fd452ffd640 1 -- 172.21.15.29:0/4115318709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c0319d0 con 0x7fd4540cda80 2023-10-16T15:34:20.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.270+0000 7fd452ffd640 1 -- 172.21.15.29:0/4115318709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c031d60 con 0x7fd4540cda80 2023-10-16T15:34:20.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 -- 172.21.15.29:0/4115318709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 msgr2=0x7fd45415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 --2- 172.21.15.29:0/4115318709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415ed90 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fd4540cf350 tx=0x7fd44c02b910 comp rx=0 tx=0).stop 2023-10-16T15:34:20.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 -- 172.21.15.29:0/4115318709 shutdown_connections 2023-10-16T15:34:20.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 --2- 172.21.15.29:0/4115318709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415ed90 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 --2- 172.21.15.29:0/4115318709 >> v2:172.21.15.170:3300/0 conn(0x7fd45406c040 0x7fd45406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 -- 172.21.15.29:0/4115318709 >> 172.21.15.29:0/4115318709 conn(0x7fd4540ce6a0 msgr2=0x7fd4540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 -- 172.21.15.29:0/4115318709 shutdown_connections 2023-10-16T15:34:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.271+0000 7fd459090640 1 -- 172.21.15.29:0/4115318709 wait complete. 2023-10-16T15:34:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.273+0000 7fd459090640 1 Processor -- start 2023-10-16T15:34:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.274+0000 7fd459090640 1 -- start start 2023-10-16T15:34:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd459090640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd45406c040 0x7fd4541531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd459090640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4540ad970 0x7fd454157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd459090640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd459090640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540ae6c0 con 0x7fd4540cda80 2023-10-16T15:34:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd459090640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540ae9a0 con 0x7fd4540ad970 2023-10-16T15:34:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd459090640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540aec80 con 0x7fd45406c040 2023-10-16T15:34:20.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd45888f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd45888f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35620/0 (socket says 172.21.15.29:35620) 2023-10-16T15:34:20.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.275+0000 7fd45888f640 1 -- 172.21.15.29:0/9342867 learned_addr learned my addr 172.21.15.29:0/9342867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:20.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd4537fe640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4540ad970 0x7fd454157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd453fff640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd45406c040 0x7fd4541531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd45888f640 1 -- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd45406c040 msgr2=0x7fd4541531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd45888f640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd45406c040 0x7fd4541531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd45888f640 1 -- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4540ad970 msgr2=0x7fd454157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd45888f640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4540ad970 0x7fd454157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd45888f640 1 -- 172.21.15.29:0/9342867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c0094d0 con 0x7fd4540cda80 2023-10-16T15:34:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd453fff640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd45406c040 0x7fd4541531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.276+0000 7fd45888f640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415d290 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fd44c0090b0 tx=0x7fd44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:20.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.277+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c03adf0 con 0x7fd4540cda80 2023-10-16T15:34:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.277+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c0443f0 con 0x7fd4540cda80 2023-10-16T15:34:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.277+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c0447a0 con 0x7fd4540cda80 2023-10-16T15:34:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.277+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4540aef60 con 0x7fd4540cda80 2023-10-16T15:34:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.277+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd454153710 con 0x7fd4540cda80 2023-10-16T15:34:20.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.278+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45406b330 con 0x7fd4540cda80 2023-10-16T15:34:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.279+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd44c04d450 con 0x7fd4540cda80 2023-10-16T15:34:20.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.280+0000 7fd450ff9640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd44006af90 0x7fd44006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.280+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd44c02d040 con 0x7fd4540cda80 2023-10-16T15:34:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.281+0000 7fd453fff640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd44006af90 0x7fd44006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.282+0000 7fd453fff640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd44006af90 0x7fd44006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd45415e060 tx=0x7fd4440076e0 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:20.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.284+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd44c076d50 con 0x7fd4540cda80 2023-10-16T15:34:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.520+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4540af480 con 0x7fd4540cda80 2023-10-16T15:34:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.521+0000 7fd450ff9640 1 -- 172.21.15.29:0/9342867 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fd4540af480 con 0x7fd4540cda80 2023-10-16T15:34:20.523 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:20.523 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/255822296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":255822296},{"type":"v1","addr":"172.21.15.29:6835","nonce":255822296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2071189203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2071189203},{"type":"v1","addr":"172.21.15.170:6836","nonce":2071189203}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":7651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/404976181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":404976181},{"type":"v1","addr":"172.21.15.170:6837","nonce":404976181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":9714,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2797981829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2797981829},{"type":"v1","addr":"172.21.15.29:6833","nonce":2797981829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[]} 2023-10-16T15:34:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 86 2023-10-16T15:34:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd44006af90 msgr2=0x7fd44006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd44006af90 0x7fd44006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd45415e060 tx=0x7fd4440076e0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 msgr2=0x7fd45415d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415d290 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fd44c0090b0 tx=0x7fd44c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 shutdown_connections 2023-10-16T15:34:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd44006af90 0x7fd44006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4540cda80 0x7fd45415d290 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4540ad970 0x7fd454157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 --2- 172.21.15.29:0/9342867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd45406c040 0x7fd4541531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 >> 172.21.15.29:0/9342867 conn(0x7fd4540ce6a0 msgr2=0x7fd4540ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.524+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 shutdown_connections 2023-10-16T15:34:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.525+0000 7fd459090640 1 -- 172.21.15.29:0/9342867 wait complete. 2023-10-16T15:34:20.547 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T15:34:20.548 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T15:34:20.548 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:34:20.548 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:34:20.548 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T15:34:20.549 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T15:34:20.549 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:34:20.549 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:34:20.550 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T15:34:20.550 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T15:34:20.550 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:34:20.550 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:34:20.551 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T15:34:20.551 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T15:34:20.551 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:34:20.552 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:34:20.564 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:34:20.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T15:34:20.571 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:34:20.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T15:34:20.573 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:34:20.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T15:34:20.574 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:34:20.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T15:34:20.619 INFO:tasks.ceph.mds.a:Started 2023-10-16T15:34:20.619 DEBUG:teuthology.parallel:result is None 2023-10-16T15:34:20.622 INFO:tasks.ceph.mds.b:Started 2023-10-16T15:34:20.622 DEBUG:teuthology.parallel:result is None 2023-10-16T15:34:20.626 INFO:tasks.ceph.mds.c:Started 2023-10-16T15:34:20.626 DEBUG:teuthology.parallel:result is None 2023-10-16T15:34:20.629 INFO:tasks.ceph.mds.d:Started 2023-10-16T15:34:20.629 DEBUG:teuthology.parallel:result is None 2023-10-16T15:34:20.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T15:34:20.783 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:34:20.784 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:34:20.784 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T15:34:20.786 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:34:20.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.818+0000 7fcb4a475640 1 Processor -- start 2023-10-16T15:34:20.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.819+0000 7fcb4a475640 1 -- start start 2023-10-16T15:34:20.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb4a475640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcb4415ad70 0x7fcb44069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb4a475640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb440caac0 0x7fcb4406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb4a475640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb440cb030 con 0x7fcb440c9ca0 2023-10-16T15:34:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb4a475640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440c91c0 con 0x7fcb4415ad70 2023-10-16T15:34:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb4a475640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440c94a0 con 0x7fcb440caac0 2023-10-16T15:34:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb3bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcb440c9ca0 0x7fcb440ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55382/0 (socket says 172.21.15.29:55382) 2023-10-16T15:34:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb3bfff640 1 -- 172.21.15.29:0/1731595385 learned_addr learned my addr 172.21.15.29:0/1731595385 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.820+0000 7fcb43fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcb4415ad70 0x7fcb44069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb48a12640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb440caac0 0x7fcb4406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb43fff640 1 -- 172.21.15.29:0/1731595385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb440caac0 msgr2=0x7fcb4406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb43fff640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb440caac0 0x7fcb4406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb43fff640 1 -- 172.21.15.29:0/1731595385 >> v1:172.21.15.29:6789/0 conn(0x7fcb440c9ca0 legacy=0x7fcb440ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb43fff640 1 -- 172.21.15.29:0/1731595385 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440c9780 con 0x7fcb4415ad70 2023-10-16T15:34:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb437fe640 1 -- 172.21.15.29:0/1731595385 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774772872 0 0) 0x7fcb440cb030 con 0x7fcb440c9ca0 2023-10-16T15:34:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.821+0000 7fcb48a12640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb440caac0 0x7fcb4406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:20.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.822+0000 7fcb43fff640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb4415ad70 0x7fcb44069da0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fcb340097a0 tx=0x7fcb3402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1437a9ced3fc593a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:20.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.822+0000 7fcb437fe640 1 -- 172.21.15.29:0/1731595385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3402d040 con 0x7fcb4415ad70 2023-10-16T15:34:20.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.822+0000 7fcb437fe640 1 -- 172.21.15.29:0/1731595385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34031810 con 0x7fcb4415ad70 2023-10-16T15:34:20.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.822+0000 7fcb437fe640 1 -- 172.21.15.29:0/1731595385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34031ba0 con 0x7fcb4415ad70 2023-10-16T15:34:20.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 -- 172.21.15.29:0/1731595385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb4415ad70 msgr2=0x7fcb44069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb4415ad70 0x7fcb44069da0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fcb340097a0 tx=0x7fcb3402b8b0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 -- 172.21.15.29:0/1731595385 shutdown_connections 2023-10-16T15:34:20.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb440caac0 0x7fcb4406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 --2- 172.21.15.29:0/1731595385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb4415ad70 0x7fcb44069da0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 -- 172.21.15.29:0/1731595385 >> 172.21.15.29:0/1731595385 conn(0x7fcb440ce9b0 msgr2=0x7fcb440cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:20.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 -- 172.21.15.29:0/1731595385 shutdown_connections 2023-10-16T15:34:20.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.823+0000 7fcb4a475640 1 -- 172.21.15.29:0/1731595385 wait complete. 2023-10-16T15:34:20.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.825+0000 7fcb4a475640 1 Processor -- start 2023-10-16T15:34:20.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.825+0000 7fcb4a475640 1 -- start start 2023-10-16T15:34:20.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb4a475640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 0x7fcb441f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb4a475640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb440caac0 0x7fcb440cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb4a475640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb4415ad70 0x7fcb4415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb4a475640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440c7a10 con 0x7fcb440c9ca0 2023-10-16T15:34:20.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb4a475640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440c6e90 con 0x7fcb440caac0 2023-10-16T15:34:20.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb4a475640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440c7170 con 0x7fcb4415ad70 2023-10-16T15:34:20.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb43fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 0x7fcb441f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb43fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 0x7fcb441f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35692/0 (socket says 172.21.15.29:35692) 2023-10-16T15:34:20.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.826+0000 7fcb43fff640 1 -- 172.21.15.29:0/4260199225 learned_addr learned my addr 172.21.15.29:0/4260199225 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:20.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb3bfff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb440caac0 0x7fcb440cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb48a12640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb4415ad70 0x7fcb4415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb43fff640 1 -- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb4415ad70 msgr2=0x7fcb4415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb43fff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb4415ad70 0x7fcb4415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb43fff640 1 -- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb440caac0 msgr2=0x7fcb440cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb43fff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb440caac0 0x7fcb440cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb43fff640 1 -- 172.21.15.29:0/4260199225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb34009400 con 0x7fcb440c9ca0 2023-10-16T15:34:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb48a12640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb4415ad70 0x7fcb4415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb3bfff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb440caac0 0x7fcb440cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb43fff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 0x7fcb441f1c80 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fcb34000c00 tx=0x7fcb340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3403aa50 con 0x7fcb440c9ca0 2023-10-16T15:34:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3403abb0 con 0x7fcb440c9ca0 2023-10-16T15:34:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34004030 con 0x7fcb440c9ca0 2023-10-16T15:34:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.827+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb440c7450 con 0x7fcb440c9ca0 2023-10-16T15:34:20.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.828+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb441f21c0 con 0x7fcb440c9ca0 2023-10-16T15:34:20.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.829+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb440c72b0 con 0x7fcb440c9ca0 2023-10-16T15:34:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.830+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcb34042070 con 0x7fcb440c9ca0 2023-10-16T15:34:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.830+0000 7fcb417fa640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fcb2406af90 0x7fcb2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.831+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcb340ab5d0 con 0x7fcb440c9ca0 2023-10-16T15:34:20.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.831+0000 7fcb3bfff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fcb2406af90 0x7fcb2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:20.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.832+0000 7fcb3bfff640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fcb2406af90 0x7fcb2406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcb4406d010 tx=0x7fcb2c009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:20.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:20.835+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb440c72b0 con 0x7fcb440c9ca0 2023-10-16T15:34:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.041+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcb440c72b0 con 0x7fcb440c9ca0 2023-10-16T15:34:21.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.513+0000 7fcb417fa640 1 -- 172.21.15.29:0/4260199225 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v186) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcb440c72b0 con 0x7fcb440c9ca0 2023-10-16T15:34:21.515 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T15:34:21.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fcb2406af90 msgr2=0x7fcb2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fcb2406af90 0x7fcb2406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcb4406d010 tx=0x7fcb2c009040 comp rx=0 tx=0).stop 2023-10-16T15:34:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 msgr2=0x7fcb441f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 0x7fcb441f1c80 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fcb34000c00 tx=0x7fcb340046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 shutdown_connections 2023-10-16T15:34:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fcb2406af90 0x7fcb2406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcb4415ad70 0x7fcb4415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb440caac0 0x7fcb440cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 --2- 172.21.15.29:0/4260199225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb440c9ca0 0x7fcb441f1c80 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 >> 172.21.15.29:0/4260199225 conn(0x7fcb440ce9b0 msgr2=0x7fcb44070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 shutdown_connections 2023-10-16T15:34:21.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.517+0000 7fcb4a475640 1 -- 172.21.15.29:0/4260199225 wait complete. 2023-10-16T15:34:21.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T15:34:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.754+0000 7f26de94e640 1 Processor -- start 2023-10-16T15:34:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.754+0000 7f26de94e640 1 -- start start 2023-10-16T15:34:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26de94e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26d8070b60 0x7f26d815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26de94e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d815aef0 con 0x7f26d806c100 2023-10-16T15:34:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26de94e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d815d910 con 0x7f26d8159950 2023-10-16T15:34:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26de94e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d815e490 con 0x7f26d8070b60 2023-10-16T15:34:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26dd14b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26d8070b60 0x7f26d815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26dd14b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26d8070b60 0x7f26d815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35696/0 (socket says 172.21.15.29:35696) 2023-10-16T15:34:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26dd14b640 1 -- 172.21.15.29:0/1186246158 learned_addr learned my addr 172.21.15.29:0/1186246158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.756+0000 7f26de14d640 1 --1- 172.21.15.29:0/1186246158 >> v1:172.21.15.29:6789/0 conn(0x7f26d806c100 0x7f26d806c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55392/0 (socket says 172.21.15.29:55392) 2023-10-16T15:34:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.757+0000 7f26dc94a640 1 -- 172.21.15.29:0/1186246158 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901151461 0 0) 0x7f26d0001490 con 0x7f26d806c100 2023-10-16T15:34:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.757+0000 7f26dc94a640 1 -- 172.21.15.29:0/1186246158 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c4003520 con 0x7f26d806c100 2023-10-16T15:34:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.757+0000 7f26dd14b640 1 -- 172.21.15.29:0/1186246158 >> v1:172.21.15.170:6789/0 conn(0x7f26d8159950 legacy=0x7f26d8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.757+0000 7f26dd14b640 1 -- 172.21.15.29:0/1186246158 >> v1:172.21.15.29:6789/0 conn(0x7f26d806c100 legacy=0x7f26d806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.757+0000 7f26dd14b640 1 -- 172.21.15.29:0/1186246158 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d815dbf0 con 0x7f26d8070b60 2023-10-16T15:34:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.758+0000 7f26dd14b640 1 --2- 172.21.15.29:0/1186246158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d8070b60 0x7f26d815a9b0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f26c8009870 tx=0x7f26c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6219e9521de41493 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.758+0000 7f26dc94a640 1 -- 172.21.15.29:0/1186246158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8031870 con 0x7f26d8070b60 2023-10-16T15:34:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.758+0000 7f26dc94a640 1 -- 172.21.15.29:0/1186246158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c80319d0 con 0x7f26d8070b60 2023-10-16T15:34:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.758+0000 7f26dc94a640 1 -- 172.21.15.29:0/1186246158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8031d40 con 0x7f26d8070b60 2023-10-16T15:34:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 -- 172.21.15.29:0/1186246158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d8070b60 msgr2=0x7f26d815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 --2- 172.21.15.29:0/1186246158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d8070b60 0x7f26d815a9b0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f26c8009870 tx=0x7f26c802b910 comp rx=0 tx=0).stop 2023-10-16T15:34:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 -- 172.21.15.29:0/1186246158 shutdown_connections 2023-10-16T15:34:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 --2- 172.21.15.29:0/1186246158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d8070b60 0x7f26d815a9b0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 -- 172.21.15.29:0/1186246158 >> 172.21.15.29:0/1186246158 conn(0x7f26d80ce6a0 msgr2=0x7f26d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 -- 172.21.15.29:0/1186246158 shutdown_connections 2023-10-16T15:34:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.759+0000 7f26de94e640 1 -- 172.21.15.29:0/1186246158 wait complete. 2023-10-16T15:34:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.761+0000 7f26de94e640 1 Processor -- start 2023-10-16T15:34:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.761+0000 7f26de94e640 1 -- start start 2023-10-16T15:34:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26de94e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 0x7f26d80c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26de94e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26d8070b60 0x7f26d8150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26de94e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26d8159950 0x7f26d80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26de94e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d80c4e50 con 0x7f26d806c100 2023-10-16T15:34:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26de94e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d815b250 con 0x7f26d8159950 2023-10-16T15:34:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26de94e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d8150ff0 con 0x7f26d8070b60 2023-10-16T15:34:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26dd94c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 0x7f26d80c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26dd94c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 0x7f26d80c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35712/0 (socket says 172.21.15.29:35712) 2023-10-16T15:34:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.762+0000 7f26dd94c640 1 -- 172.21.15.29:0/3214677885 learned_addr learned my addr 172.21.15.29:0/3214677885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26dd94c640 1 -- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26d8070b60 msgr2=0x7f26d8150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26dd94c640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26d8070b60 0x7f26d8150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26dd94c640 1 -- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26d8159950 msgr2=0x7f26d80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26dd94c640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26d8159950 0x7f26d80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26dd94c640 1 -- 172.21.15.29:0/3214677885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c80094d0 con 0x7f26d806c100 2023-10-16T15:34:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26dd94c640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 0x7f26d80c4910 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f26d400b3f0 tx=0x7f26d400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4011830 con 0x7f26d806c100 2023-10-16T15:34:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.763+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d4011990 con 0x7f26d806c100 2023-10-16T15:34:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.764+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d80cda00 con 0x7f26d806c100 2023-10-16T15:34:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.764+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4011d40 con 0x7f26d806c100 2023-10-16T15:34:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.764+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d80cd720 con 0x7f26d806c100 2023-10-16T15:34:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.765+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d804e9b0 con 0x7f26d806c100 2023-10-16T15:34:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.766+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f26d401a400 con 0x7f26d806c100 2023-10-16T15:34:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.767+0000 7f26ce7fc640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f26bc06af90 0x7f26bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.767+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f26d40892a0 con 0x7f26d806c100 2023-10-16T15:34:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.768+0000 7f26dd14b640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f26bc06af90 0x7f26bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.769+0000 7f26dd14b640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f26bc06af90 0x7f26bc06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f26c8036000 tx=0x7f26c8030000 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:21.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:21.771+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f26d4054560 con 0x7f26d806c100 2023-10-16T15:34:22.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.038+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f26d80b4620 con 0x7f26d806c100 2023-10-16T15:34:22.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.039+0000 7f26ce7fc640 1 -- 172.21.15.29:0/3214677885 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f26d4057b80 con 0x7f26d806c100 2023-10-16T15:34:22.041 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:22.041 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:34:22.041 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T15:34:22.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.041 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T15:34:22.042 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T15:34:22.042 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.042 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.042 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:34:22.042 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:34:22.043 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.043 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.043 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T15:34:22.044 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T15:34:22.044 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.044 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.044 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:34:22.044 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:34:22.044 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.045 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.045 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T15:34:22.045 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T15:34:22.046 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.046 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.046 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:34:22.046 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:34:22.046 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.047 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.047 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T15:34:22.047 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T15:34:22.048 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.048 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.048 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:34:22.048 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:34:22.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.049 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.049 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T15:34:22.049 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T15:34:22.049 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.050 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.050 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.050 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.050 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.051 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T15:34:22.051 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T15:34:22.051 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.051 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.052 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.052 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.053 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T15:34:22.053 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T15:34:22.053 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.053 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.053 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.054 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T15:34:22.055 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T15:34:22.055 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.055 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.055 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.055 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.056 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T15:34:22.056 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T15:34:22.056 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.057 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.057 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.058 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T15:34:22.058 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T15:34:22.058 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.058 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.058 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.059 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.059 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T15:34:22.060 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T15:34:22.060 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.060 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.060 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.060 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.060 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.061 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T15:34:22.061 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T15:34:22.061 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.062 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:34:22.062 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:34:22.062 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.062 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.063 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T15:34:22.063 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T15:34:22.063 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.063 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.064 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T15:34:22.064 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T15:34:22.064 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.065 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T15:34:22.065 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T15:34:22.065 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.065 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.065 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:34:22.065 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:34:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:34:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T15:34:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T15:34:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:34:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:34:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:34:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T15:34:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T15:34:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:34:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:34:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:34:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:34:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T15:34:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T15:34:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:34:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:34:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:34:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T15:34:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T15:34:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:34:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:34:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:34:22.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:22.074 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T15:34:22.074 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T15:34:22.074 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:34:22.074 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:34:22.075 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:34:22.075 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:34:22.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:22.075 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T15:34:22.076 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:34:22.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.041+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f26bc06af90 msgr2=0x7f26bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.041+0000 7f26de94e640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f26bc06af90 0x7f26bc06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f26c8036000 tx=0x7f26c8030000 comp rx=0 tx=0).stop 2023-10-16T15:34:22.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 msgr2=0x7f26d80c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 0x7f26d80c4910 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f26d400b3f0 tx=0x7f26d400b8b0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 shutdown_connections 2023-10-16T15:34:22.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f26bc06af90 0x7f26bc06d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26d8159950 0x7f26d80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f26d8070b60 0x7f26d8150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 --2- 172.21.15.29:0/3214677885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26d806c100 0x7f26d80c4910 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 >> 172.21.15.29:0/3214677885 conn(0x7f26d80ce6a0 msgr2=0x7f26d806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 shutdown_connections 2023-10-16T15:34:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.042+0000 7f26de94e640 1 -- 172.21.15.29:0/3214677885 wait complete. 2023-10-16T15:34:22.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.307+0000 7f5163fff640 1 Processor -- start 2023-10-16T15:34:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.308+0000 7f5163fff640 1 -- start start 2023-10-16T15:34:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.309+0000 7f5163fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f516406c040 0x7f516406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.309+0000 7f5163fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51640cda80 0x7f516415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.309+0000 7f5163fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516406c960 con 0x7f51640ad970 2023-10-16T15:34:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.309+0000 7f5163fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516415b9d0 con 0x7f51640cda80 2023-10-16T15:34:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.309+0000 7f5163fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516415bcb0 con 0x7f516406c040 2023-10-16T15:34:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.309+0000 7f51637fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51640cda80 0x7f516415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f51637fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51640cda80 0x7f516415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35728/0 (socket says 172.21.15.29:35728) 2023-10-16T15:34:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f51637fe640 1 -- 172.21.15.29:0/3382191373 learned_addr learned my addr 172.21.15.29:0/3382191373 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f5162ffd640 1 --2- 172.21.15.29:0/3382191373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f516406c040 0x7f516406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f51637fe640 1 -- 172.21.15.29:0/3382191373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f516406c040 msgr2=0x7f516406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:22.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f51637fe640 1 --2- 172.21.15.29:0/3382191373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f516406c040 0x7f516406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f51637fe640 1 -- 172.21.15.29:0/3382191373 >> v1:172.21.15.170:6789/0 conn(0x7f51640ad970 legacy=0x7f516415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.310+0000 7f51637fe640 1 -- 172.21.15.29:0/3382191373 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516415bf90 con 0x7f51640cda80 2023-10-16T15:34:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.311+0000 7f51637fe640 1 --2- 172.21.15.29:0/3382191373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51640cda80 0x7f516415ed90 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f515c009870 tx=0x7f515c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d5df0c16f7d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f51627fc640 1 -- 172.21.15.29:0/3382191373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c030840 con 0x7f51640cda80 2023-10-16T15:34:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f51627fc640 1 -- 172.21.15.29:0/3382191373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0309a0 con 0x7f51640cda80 2023-10-16T15:34:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f51627fc640 1 -- 172.21.15.29:0/3382191373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c030d30 con 0x7f51640cda80 2023-10-16T15:34:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 -- 172.21.15.29:0/3382191373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51640cda80 msgr2=0x7f516415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 --2- 172.21.15.29:0/3382191373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51640cda80 0x7f516415ed90 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f515c009870 tx=0x7f515c02b980 comp rx=0 tx=0).stop 2023-10-16T15:34:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 -- 172.21.15.29:0/3382191373 shutdown_connections 2023-10-16T15:34:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 --2- 172.21.15.29:0/3382191373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51640cda80 0x7f516415ed90 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 --2- 172.21.15.29:0/3382191373 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f516406c040 0x7f516406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 -- 172.21.15.29:0/3382191373 >> 172.21.15.29:0/3382191373 conn(0x7f51640ce6a0 msgr2=0x7f51640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 -- 172.21.15.29:0/3382191373 shutdown_connections 2023-10-16T15:34:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.312+0000 7f5163fff640 1 -- 172.21.15.29:0/3382191373 wait complete. 2023-10-16T15:34:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.314+0000 7f5163fff640 1 Processor -- start 2023-10-16T15:34:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.314+0000 7f5163fff640 1 -- start start 2023-10-16T15:34:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5163fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 0x7f51641551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5163fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51640ad970 0x7f51640acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5163fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51640cda80 0x7f5164153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5163fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51641556e0 con 0x7f516406c040 2023-10-16T15:34:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5163fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5164157160 con 0x7f51640ad970 2023-10-16T15:34:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5163fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51641560c0 con 0x7f51640cda80 2023-10-16T15:34:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.315+0000 7f5162ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 0x7f51641551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 0x7f51641551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35730/0 (socket says 172.21.15.29:35730) 2023-10-16T15:34:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 -- 172.21.15.29:0/3933523356 learned_addr learned my addr 172.21.15.29:0/3933523356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 -- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51640cda80 msgr2=0x7f5164153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51640cda80 0x7f5164153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 -- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51640ad970 msgr2=0x7f51640acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51640ad970 0x7f51640acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 -- 172.21.15.29:0/3933523356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0094d0 con 0x7f516406c040 2023-10-16T15:34:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5162ffd640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 0x7f51641551a0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f5154007e60 tx=0x7f515400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.316+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154014620 con 0x7f516406c040 2023-10-16T15:34:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.317+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515400ee90 con 0x7f516406c040 2023-10-16T15:34:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.317+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51540149e0 con 0x7f516406c040 2023-10-16T15:34:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.317+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51641563a0 con 0x7f516406c040 2023-10-16T15:34:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.317+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5164156960 con 0x7f516406c040 2023-10-16T15:34:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.318+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516406b330 con 0x7f516406c040 2023-10-16T15:34:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.319+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5154014b40 con 0x7f516406c040 2023-10-16T15:34:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.320+0000 7f5160ff9640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f514406af90 0x7f514406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.320+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f5154089ed0 con 0x7f516406c040 2023-10-16T15:34:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.320+0000 7f515a7fc640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f514406af90 0x7f514406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.322+0000 7f515a7fc640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f514406af90 0x7f514406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f514c000f60 tx=0x7f514c0079b0 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.324+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5154055100 con 0x7f516406c040 2023-10-16T15:34:22.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.517+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51641f3e90 con 0x7f516406c040 2023-10-16T15:34:22.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.521+0000 7f5160ff9640 1 -- 172.21.15.29:0/3933523356 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f5154058720 con 0x7f516406c040 2023-10-16T15:34:22.523 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:22.524 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":188,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:21.526080+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f514406af90 msgr2=0x7f514406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f514406af90 0x7f514406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f514c000f60 tx=0x7f514c0079b0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 msgr2=0x7f51641551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 0x7f51641551a0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f5154007e60 tx=0x7f515400e4b0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 shutdown_connections 2023-10-16T15:34:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f514406af90 0x7f514406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51640cda80 0x7f5164153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51640ad970 0x7f51640acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 --2- 172.21.15.29:0/3933523356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f516406c040 0x7f51641551a0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.524+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 >> 172.21.15.29:0/3933523356 conn(0x7f51640ce6a0 msgr2=0x7f516415cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.525+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 shutdown_connections 2023-10-16T15:34:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.525+0000 7f5163fff640 1 -- 172.21.15.29:0/3933523356 wait complete. 2023-10-16T15:34:22.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:34:22.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:34:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.761+0000 7fd682d4e640 1 Processor -- start 2023-10-16T15:34:22.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.762+0000 7fd682d4e640 1 -- start start 2023-10-16T15:34:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.763+0000 7fd682d4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c06a4b0 0x7fd67c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.763+0000 7fd682d4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd67c0c9ca0 0x7fd67c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.763+0000 7fd682d4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67c0cb030 con 0x7fd67c15ad70 2023-10-16T15:34:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.763+0000 7fd682d4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c0cae30 con 0x7fd67c0c9ca0 2023-10-16T15:34:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.763+0000 7fd682d4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0c91c0 con 0x7fd67c06a4b0 2023-10-16T15:34:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.764+0000 7fd680aea640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd67c15ad70 0x7fd67c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:57736/0 (socket says 172.21.15.29:57736) 2023-10-16T15:34:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.764+0000 7fd680aea640 1 -- 172.21.15.29:0/2486750335 learned_addr learned my addr 172.21.15.29:0/2486750335 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.764+0000 7fd6812eb640 1 --2- 172.21.15.29:0/2486750335 >> v2:172.21.15.170:3300/0 conn(0x7fd67c0c9ca0 0x7fd67c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.764+0000 7fd673fff640 1 --2- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c06a4b0 0x7fd67c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.765+0000 7fd6812eb640 1 -- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c06a4b0 msgr2=0x7fd67c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.765+0000 7fd6812eb640 1 --2- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c06a4b0 0x7fd67c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.765+0000 7fd6812eb640 1 -- 172.21.15.29:0/2486750335 >> v1:172.21.15.170:6789/0 conn(0x7fd67c15ad70 legacy=0x7fd67c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.765+0000 7fd6812eb640 1 -- 172.21.15.29:0/2486750335 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0c94a0 con 0x7fd67c0c9ca0 2023-10-16T15:34:22.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.767+0000 7fd6812eb640 1 --2- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c0c9ca0 0x7fd67c0ca8f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd674009d90 tx=0x7fd67402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddb8b3cf9b8e419 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:22.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd6737fe640 1 -- 172.21.15.29:0/2486750335 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67402cc60 con 0x7fd67c0c9ca0 2023-10-16T15:34:22.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd6737fe640 1 -- 172.21.15.29:0/2486750335 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67402cdc0 con 0x7fd67c0c9ca0 2023-10-16T15:34:22.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd6737fe640 1 -- 172.21.15.29:0/2486750335 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd674034660 con 0x7fd67c0c9ca0 2023-10-16T15:34:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd682d4e640 1 -- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c0c9ca0 msgr2=0x7fd67c0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd682d4e640 1 --2- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c0c9ca0 0x7fd67c0ca8f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd674009d90 tx=0x7fd67402beb0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd682d4e640 1 -- 172.21.15.29:0/2486750335 shutdown_connections 2023-10-16T15:34:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd682d4e640 1 --2- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c0c9ca0 0x7fd67c0ca8f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.768+0000 7fd682d4e640 1 --2- 172.21.15.29:0/2486750335 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c06a4b0 0x7fd67c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.769+0000 7fd682d4e640 1 -- 172.21.15.29:0/2486750335 >> 172.21.15.29:0/2486750335 conn(0x7fd67c0ce9b0 msgr2=0x7fd67c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.769+0000 7fd682d4e640 1 -- 172.21.15.29:0/2486750335 shutdown_connections 2023-10-16T15:34:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.769+0000 7fd682d4e640 1 -- 172.21.15.29:0/2486750335 wait complete. 2023-10-16T15:34:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.770+0000 7fd682d4e640 1 Processor -- start 2023-10-16T15:34:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.771+0000 7fd682d4e640 1 -- start start 2023-10-16T15:34:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd682d4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 0x7fd67c0c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd682d4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c0c9ca0 0x7fd67c0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd682d4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c15ad70 0x7fd67c14e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd682d4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0cc8b0 con 0x7fd67c06a4b0 2023-10-16T15:34:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd682d4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0ccb90 con 0x7fd67c15ad70 2023-10-16T15:34:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd682d4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0cce70 con 0x7fd67c0c9ca0 2023-10-16T15:34:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd680aea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 0x7fd67c0c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd680aea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 0x7fd67c0c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35734/0 (socket says 172.21.15.29:35734) 2023-10-16T15:34:22.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd680aea640 1 -- 172.21.15.29:0/1636816302 learned_addr learned my addr 172.21.15.29:0/1636816302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd673fff640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c0c9ca0 0x7fd67c0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd6812eb640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c15ad70 0x7fd67c14e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd680aea640 1 -- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c0c9ca0 msgr2=0x7fd67c0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.772+0000 7fd680aea640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c0c9ca0 0x7fd67c0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd680aea640 1 -- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c15ad70 msgr2=0x7fd67c14e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd680aea640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c15ad70 0x7fd67c14e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd680aea640 1 -- 172.21.15.29:0/1636816302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674009a70 con 0x7fd67c06a4b0 2023-10-16T15:34:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd6812eb640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c15ad70 0x7fd67c14e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:22.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd680aea640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 0x7fd67c0c7610 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c480 tx=0x7fd67800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:22.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678005ce0 con 0x7fd67c06a4b0 2023-10-16T15:34:22.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd678005e40 con 0x7fd67c06a4b0 2023-10-16T15:34:22.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780072f0 con 0x7fd67c06a4b0 2023-10-16T15:34:22.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c0cd150 con 0x7fd67c06a4b0 2023-10-16T15:34:22.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.773+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c0cd430 con 0x7fd67c06a4b0 2023-10-16T15:34:22.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.775+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c04e160 con 0x7fd67c06a4b0 2023-10-16T15:34:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.775+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd678007510 con 0x7fd67c06a4b0 2023-10-16T15:34:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.776+0000 7fd6717fa640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd66806af90 0x7fd66806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.777+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fd678088930 con 0x7fd67c06a4b0 2023-10-16T15:34:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.777+0000 7fd673fff640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd66806af90 0x7fd66806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.779+0000 7fd673fff640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd66806af90 0x7fd66806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd67c152c50 tx=0x7fd66c008040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:22.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.781+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd678053ca0 con 0x7fd67c06a4b0 2023-10-16T15:34:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.973+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd67c0ac890 con 0x7fd67c06a4b0 2023-10-16T15:34:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.974+0000 7fd6717fa640 1 -- 172.21.15.29:0/1636816302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd678017020 con 0x7fd67c06a4b0 2023-10-16T15:34:22.976 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:34:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd66806af90 msgr2=0x7fd66806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd66806af90 0x7fd66806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd67c152c50 tx=0x7fd66c008040 comp rx=0 tx=0).stop 2023-10-16T15:34:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 msgr2=0x7fd67c0c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 0x7fd67c0c7610 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c480 tx=0x7fd67800c940 comp rx=0 tx=0).stop 2023-10-16T15:34:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 shutdown_connections 2023-10-16T15:34:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fd66806af90 0x7fd66806d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd67c15ad70 0x7fd67c14e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd67c0c9ca0 0x7fd67c0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 --2- 172.21.15.29:0/1636816302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd67c06a4b0 0x7fd67c0c7610 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 >> 172.21.15.29:0/1636816302 conn(0x7fd67c0ce9b0 msgr2=0x7fd67c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 shutdown_connections 2023-10-16T15:34:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:22.977+0000 7fd682d4e640 1 -- 172.21.15.29:0/1636816302 wait complete. 2023-10-16T15:34:22.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:34:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.226+0000 7f0d11c75640 1 Processor -- start 2023-10-16T15:34:23.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.226+0000 7f0d11c75640 1 -- start start 2023-10-16T15:34:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d11c75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0d0c06d220 0x7f0d0c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d11c75640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0d0c0ce280 0x7f0d0c0ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d11c75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c0acd70 con 0x7f0d0c065ab0 2023-10-16T15:34:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d11c75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c0ac4a0 con 0x7f0d0c06d220 2023-10-16T15:34:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d11c75640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c065f40 con 0x7f0d0c0ce280 2023-10-16T15:34:23.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d10c73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0d0c06d220 0x7f0d0c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d0bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0d0c065ab0 0x7f0d0c0cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55400/0 (socket says 172.21.15.29:55400) 2023-10-16T15:34:23.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d10c73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0d0c06d220 0x7f0d0c0c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35736/0 (socket says 172.21.15.29:35736) 2023-10-16T15:34:23.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.228+0000 7f0d0bfff640 1 -- 172.21.15.29:0/835904238 learned_addr learned my addr 172.21.15.29:0/835904238 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:23.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.229+0000 7f0d11474640 1 --2- 172.21.15.29:0/835904238 >> v2:172.21.15.170:3300/0 conn(0x7f0d0c0ce280 0x7f0d0c0ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.229+0000 7f0d10c73640 1 -- 172.21.15.29:0/835904238 >> v2:172.21.15.170:3300/0 conn(0x7f0d0c0ce280 msgr2=0x7f0d0c0ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.229+0000 7f0d10c73640 1 --2- 172.21.15.29:0/835904238 >> v2:172.21.15.170:3300/0 conn(0x7f0d0c0ce280 0x7f0d0c0ad200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.229+0000 7f0d10c73640 1 -- 172.21.15.29:0/835904238 >> v1:172.21.15.29:6789/0 conn(0x7f0d0c065ab0 legacy=0x7f0d0c0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.229+0000 7f0d10c73640 1 -- 172.21.15.29:0/835904238 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0ac720 con 0x7f0d0c06d220 2023-10-16T15:34:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.229+0000 7f0d0b7fe640 1 -- 172.21.15.29:0/835904238 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963559325 0 0) 0x7f0d0c0acd70 con 0x7f0d0c065ab0 2023-10-16T15:34:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.230+0000 7f0d10c73640 1 --2- 172.21.15.29:0/835904238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c06d220 0x7f0d0c0c95e0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c071fd0 tx=0x7f0d0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e5aedf90e4ab1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.230+0000 7f0d0b7fe640 1 -- 172.21.15.29:0/835904238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0002e070 con 0x7f0d0c06d220 2023-10-16T15:34:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.230+0000 7f0d0b7fe640 1 -- 172.21.15.29:0/835904238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d000315a0 con 0x7f0d0c06d220 2023-10-16T15:34:23.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.230+0000 7f0d0b7fe640 1 -- 172.21.15.29:0/835904238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00031930 con 0x7f0d0c06d220 2023-10-16T15:34:23.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 -- 172.21.15.29:0/835904238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c06d220 msgr2=0x7f0d0c0c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 --2- 172.21.15.29:0/835904238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c06d220 0x7f0d0c0c95e0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c071fd0 tx=0x7f0d0002b840 comp rx=0 tx=0).stop 2023-10-16T15:34:23.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 -- 172.21.15.29:0/835904238 shutdown_connections 2023-10-16T15:34:23.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 --2- 172.21.15.29:0/835904238 >> v2:172.21.15.170:3300/0 conn(0x7f0d0c0ce280 0x7f0d0c0ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 --2- 172.21.15.29:0/835904238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c06d220 0x7f0d0c0c95e0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 -- 172.21.15.29:0/835904238 >> 172.21.15.29:0/835904238 conn(0x7f0d0c0b0a80 msgr2=0x7f0d0c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:23.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 -- 172.21.15.29:0/835904238 shutdown_connections 2023-10-16T15:34:23.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.231+0000 7f0d11c75640 1 -- 172.21.15.29:0/835904238 wait complete. 2023-10-16T15:34:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.233+0000 7f0d11c75640 1 Processor -- start 2023-10-16T15:34:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.233+0000 7f0d11c75640 1 -- start start 2023-10-16T15:34:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11c75640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d0c06d220 0x7f0d0c0c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11c75640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d0c0ce280 0x7f0d0c1f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11c75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 0x7f0d0c09ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11c75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c7480 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11c75640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c7de0 con 0x7f0d0c0ce280 2023-10-16T15:34:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11c75640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c88c0 con 0x7f0d0c06d220 2023-10-16T15:34:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11474640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 0x7f0d0c09ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11474640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 0x7f0d0c09ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35744/0 (socket says 172.21.15.29:35744) 2023-10-16T15:34:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.234+0000 7f0d11474640 1 -- 172.21.15.29:0/644582531 learned_addr learned my addr 172.21.15.29:0/644582531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d0bfff640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d0c0ce280 0x7f0d0c1f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d10c73640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d0c06d220 0x7f0d0c0c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d11474640 1 -- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d0c06d220 msgr2=0x7f0d0c0c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d11474640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d0c06d220 0x7f0d0c0c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d11474640 1 -- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d0c0ce280 msgr2=0x7f0d0c1f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d11474640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d0c0ce280 0x7f0d0c1f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d11474640 1 -- 172.21.15.29:0/644582531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00009400 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d0bfff640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d0c0ce280 0x7f0d0c1f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:23.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d11474640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 0x7f0d0c09ae60 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f0d04007e40 tx=0x7f0d0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.235+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04014620 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.236+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0400ee90 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.236+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d040149e0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.236+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c0c8ba0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.236+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c0c70a0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.237+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c0c75c0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.238+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0d04014b40 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.239+0000 7f0d097fa640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cf806af90 0x7f0cf806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.239+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f0d04089eb0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.240+0000 7f0d10c73640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cf806af90 0x7f0cf806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.241+0000 7f0d10c73640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cf806af90 0x7f0cf806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d00036000 tx=0x7f0d0003d040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.243+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0d0c0c75c0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.442+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0d0c0c75c0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.443+0000 7f0d097fa640 1 -- 172.21.15.29:0/644582531 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v188) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0d0c0c75c0 con 0x7f0d0c0b1bb0 2023-10-16T15:34:23.445 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:23.445 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:34:23.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:23.445 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:34:23.445 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:34:23.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:23.446 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:34:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cf806af90 msgr2=0x7f0cf806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cf806af90 0x7f0cf806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d00036000 tx=0x7f0d0003d040 comp rx=0 tx=0).stop 2023-10-16T15:34:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 msgr2=0x7f0d0c09ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 0x7f0d0c09ae60 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f0d04007e40 tx=0x7f0d0400e4b0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 shutdown_connections 2023-10-16T15:34:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cf806af90 0x7f0cf806d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d0c0b1bb0 0x7f0d0c09ae60 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d0c0ce280 0x7f0d0c1f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 --2- 172.21.15.29:0/644582531 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d0c06d220 0x7f0d0c0c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 >> 172.21.15.29:0/644582531 conn(0x7f0d0c0b0a80 msgr2=0x7f0d0c0ad200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 shutdown_connections 2023-10-16T15:34:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.446+0000 7f0d11c75640 1 -- 172.21.15.29:0/644582531 wait complete. 2023-10-16T15:34:23.468 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T15:34:23.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T15:34:23.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.701+0000 7f7ff2593640 1 Processor -- start 2023-10-16T15:34:23.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.701+0000 7f7ff2593640 1 -- start start 2023-10-16T15:34:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.703+0000 7f7ff2593640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7fec06b3e0 0x7f7fec06cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.703+0000 7f7ff2593640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fec06d730 con 0x7f7fec159bd0 2023-10-16T15:34:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.703+0000 7f7ff2593640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fec15a810 con 0x7f7fec06d2c0 2023-10-16T15:34:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.703+0000 7f7ff2593640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec0c5740 con 0x7f7fec06b3e0 2023-10-16T15:34:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.703+0000 7f7ff1591640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7fec159bd0 0x7f7fec159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55402/0 (socket says 172.21.15.29:55402) 2023-10-16T15:34:23.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.703+0000 7f7ff1591640 1 -- 172.21.15.29:0/1708152759 learned_addr learned my addr 172.21.15.29:0/1708152759 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7ff0d90640 1 --2- 172.21.15.29:0/1708152759 >> v2:172.21.15.170:3300/0 conn(0x7f7fec06b3e0 0x7f7fec06cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064139698 0 0) 0x7f7fec06d730 con 0x7f7fec159bd0 2023-10-16T15:34:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd8003520 con 0x7f7fec159bd0 2023-10-16T15:34:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024305334 0 0) 0x7f7fd8003520 con 0x7f7fec159bd0 2023-10-16T15:34:23.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 >> v2:172.21.15.170:3300/0 conn(0x7f7fec06b3e0 msgr2=0x7f7fec06cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 --2- 172.21.15.29:0/1708152759 >> v2:172.21.15.170:3300/0 conn(0x7f7fec06b3e0 0x7f7fec06cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 >> v1:172.21.15.170:6789/0 conn(0x7f7fec06d2c0 legacy=0x7f7fec070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec15aaf0 con 0x7f7fec159bd0 2023-10-16T15:34:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7fe80019f0 con 0x7f7fec159bd0 2023-10-16T15:34:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 >> v1:172.21.15.29:6789/0 conn(0x7f7fec159bd0 legacy=0x7f7fec159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd8003a30 0x7f7fd8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fd80047c0 0x7f7fd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fd80052a0 0x7f7fd800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c5740 con 0x7f7fd80052a0 2023-10-16T15:34:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd800e380 con 0x7f7fd8003a30 2023-10-16T15:34:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.704+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd800e540 con 0x7f7fd80047c0 2023-10-16T15:34:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fd80052a0 0x7f7fd800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1591640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd8003a30 0x7f7fd8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff0d90640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fd80047c0 0x7f7fd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1d92640 1 -- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fd80047c0 msgr2=0x7f7fd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fd80047c0 0x7f7fd8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1d92640 1 -- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd8003a30 msgr2=0x7f7fd8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd8003a30 0x7f7fd8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.705+0000 7f7ff1d92640 1 -- 172.21.15.29:0/1708152759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd8003520 con 0x7f7fd80052a0 2023-10-16T15:34:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.706+0000 7f7ff1d92640 1 -- 172.21.15.29:0/1708152759 reap_dead start 2023-10-16T15:34:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.706+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fd80052a0 0x7f7fd800db90 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4001750 tx=0x7f7fd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a181c83498f96f88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40052d0 con 0x7f7fd80052a0 2023-10-16T15:34:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe402e050 con 0x7f7fd80052a0 2023-10-16T15:34:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7fe3fff640 1 -- 172.21.15.29:0/1708152759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4003830 con 0x7f7fd80052a0 2023-10-16T15:34:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7ff2593640 1 -- 172.21.15.29:0/1708152759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fd80052a0 msgr2=0x7f7fd800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7ff2593640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fd80052a0 0x7f7fd800db90 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4001750 tx=0x7f7fd80057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7ff2593640 1 -- 172.21.15.29:0/1708152759 shutdown_connections 2023-10-16T15:34:23.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7ff2593640 1 --2- 172.21.15.29:0/1708152759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fd80052a0 0x7f7fd800db90 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.707+0000 7f7ff2593640 1 -- 172.21.15.29:0/1708152759 >> 172.21.15.29:0/1708152759 conn(0x7f7fec0ce820 msgr2=0x7f7fec0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.708+0000 7f7ff2593640 1 -- 172.21.15.29:0/1708152759 shutdown_connections 2023-10-16T15:34:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.708+0000 7f7ff2593640 1 -- 172.21.15.29:0/1708152759 wait complete. 2023-10-16T15:34:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.710+0000 7f7ff2593640 1 Processor -- start 2023-10-16T15:34:23.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.710+0000 7f7ff2593640 1 -- start start 2023-10-16T15:34:23.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff2593640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd80052a0 0x7f7fec0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff2593640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fec0c49d0 0x7f7fec154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff2593640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 0x7f7fec15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff2593640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec15bb00 con 0x7f7fec15b140 2023-10-16T15:34:23.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff2593640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c51d0 con 0x7f7fd80052a0 2023-10-16T15:34:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff2593640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0b2b10 con 0x7f7fec0c49d0 2023-10-16T15:34:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff1d92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 0x7f7fec15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff1d92640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 0x7f7fec15b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35766/0 (socket says 172.21.15.29:35766) 2023-10-16T15:34:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff1d92640 1 -- 172.21.15.29:0/67996967 learned_addr learned my addr 172.21.15.29:0/67996967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.711+0000 7f7ff0d90640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fec0c49d0 0x7f7fec154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1591640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd80052a0 0x7f7fec0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1d92640 1 -- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fec0c49d0 msgr2=0x7f7fec154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fec0c49d0 0x7f7fec154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1d92640 1 -- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd80052a0 msgr2=0x7f7fec0c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd80052a0 0x7f7fec0c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1d92640 1 -- 172.21.15.29:0/67996967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0b2df0 con 0x7f7fec15b140 2023-10-16T15:34:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff1d92640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 0x7f7fec15b5c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4001870 tx=0x7f7fe4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4000ed0 con 0x7f7fec15b140 2023-10-16T15:34:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe4007030 con 0x7f7fec15b140 2023-10-16T15:34:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40073a0 con 0x7f7fec15b140 2023-10-16T15:34:23.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec0b30d0 con 0x7f7fec15b140 2023-10-16T15:34:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.712+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec0c4e50 con 0x7f7fec15b140 2023-10-16T15:34:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.713+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fec06ee60 con 0x7f7fec15b140 2023-10-16T15:34:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.714+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7fe403a070 con 0x7f7fec15b140 2023-10-16T15:34:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.715+0000 7f7fe1ffb640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7fd006afc0 0x7f7fd006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.716+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f7fe4037070 con 0x7f7fec15b140 2023-10-16T15:34:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.716+0000 7f7ff1591640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7fd006afc0 0x7f7fd006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.717+0000 7f7ff1591640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7fd006afc0 0x7f7fd006d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80013d0 tx=0x7f7fe8009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.720+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7fe40756b0 con 0x7f7fec15b140 2023-10-16T15:34:23.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:23.916+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7fec04e9b0 con 0x7f7fec15b140 2023-10-16T15:34:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.535+0000 7f7fe1ffb640 1 -- 172.21.15.29:0/67996967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v189) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7fe4078cd0 con 0x7f7fec15b140 2023-10-16T15:34:24.537 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T15:34:24.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.540+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7fd006afc0 msgr2=0x7f7fd006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.540+0000 7f7ff2593640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7fd006afc0 0x7f7fd006d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80013d0 tx=0x7f7fe8009040 comp rx=0 tx=0).stop 2023-10-16T15:34:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.540+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 msgr2=0x7f7fec15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:24.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.540+0000 7f7ff2593640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 0x7f7fec15b5c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4001870 tx=0x7f7fe4001120 comp rx=0 tx=0).stop 2023-10-16T15:34:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 shutdown_connections 2023-10-16T15:34:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f7fd006afc0 0x7f7fd006d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fec15b140 0x7f7fec15b5c0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7fec0c49d0 0x7f7fec154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 --2- 172.21.15.29:0/67996967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fd80052a0 0x7f7fec0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 >> 172.21.15.29:0/67996967 conn(0x7f7fec0ce820 msgr2=0x7f7fec0cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 shutdown_connections 2023-10-16T15:34:24.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.541+0000 7f7ff2593640 1 -- 172.21.15.29:0/67996967 wait complete. 2023-10-16T15:34:24.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:34:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.796+0000 7fa41bfff640 1 Processor -- start 2023-10-16T15:34:24.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.797+0000 7fa41bfff640 1 -- start start 2023-10-16T15:34:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.798+0000 7fa41bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c15cc80 0x7fa41c0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.798+0000 7fa41bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa41c0cdd60 0x7fa41c071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.798+0000 7fa41bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa41c071fa0 0x7fa41c1607e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa41bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0d0500 con 0x7fa41c0cdd60 2023-10-16T15:34:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa41bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0ce1c0 con 0x7fa41c071fa0 2023-10-16T15:34:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa41bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0d12f0 con 0x7fa41c15cc80 2023-10-16T15:34:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa41c0cdd60 0x7fa41c071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa41c0cdd60 0x7fa41c071a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33282/0 (socket says 172.21.15.29:33282) 2023-10-16T15:34:24.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 -- 172.21.15.29:0/7983375 learned_addr learned my addr 172.21.15.29:0/7983375 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:24.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa41affd640 1 --2- 172.21.15.29:0/7983375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c15cc80 0x7fa41c0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa41b7fe640 1 --2- 172.21.15.29:0/7983375 >> v2:172.21.15.170:3300/0 conn(0x7fa41c071fa0 0x7fa41c1607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 -- 172.21.15.29:0/7983375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c15cc80 msgr2=0x7fa41c0cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 --2- 172.21.15.29:0/7983375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c15cc80 0x7fa41c0cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 -- 172.21.15.29:0/7983375 >> v2:172.21.15.170:3300/0 conn(0x7fa41c071fa0 msgr2=0x7fa41c1607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.799+0000 7fa413fff640 1 --2- 172.21.15.29:0/7983375 >> v2:172.21.15.170:3300/0 conn(0x7fa41c071fa0 0x7fa41c1607e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.800+0000 7fa413fff640 1 -- 172.21.15.29:0/7983375 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c072410 con 0x7fa41c0cdd60 2023-10-16T15:34:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.801+0000 7fa413fff640 1 --2- 172.21.15.29:0/7983375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c071a60 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0b27f0 tx=0x7fa40402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07838cc27d424e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.801+0000 7fa41a7fc640 1 -- 172.21.15.29:0/7983375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404031870 con 0x7fa41c0cdd60 2023-10-16T15:34:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.801+0000 7fa41a7fc640 1 -- 172.21.15.29:0/7983375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4040319d0 con 0x7fa41c0cdd60 2023-10-16T15:34:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.801+0000 7fa41a7fc640 1 -- 172.21.15.29:0/7983375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404031d60 con 0x7fa41c0cdd60 2023-10-16T15:34:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 -- 172.21.15.29:0/7983375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 msgr2=0x7fa41c071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 --2- 172.21.15.29:0/7983375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c071a60 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0b27f0 tx=0x7fa40402b910 comp rx=0 tx=0).stop 2023-10-16T15:34:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 -- 172.21.15.29:0/7983375 shutdown_connections 2023-10-16T15:34:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 --2- 172.21.15.29:0/7983375 >> v2:172.21.15.170:3300/0 conn(0x7fa41c071fa0 0x7fa41c1607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 --2- 172.21.15.29:0/7983375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c071a60 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 --2- 172.21.15.29:0/7983375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c15cc80 0x7fa41c0cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 -- 172.21.15.29:0/7983375 >> 172.21.15.29:0/7983375 conn(0x7fa41c065ed0 msgr2=0x7fa41c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:24.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 -- 172.21.15.29:0/7983375 shutdown_connections 2023-10-16T15:34:24.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.802+0000 7fa41bfff640 1 -- 172.21.15.29:0/7983375 wait complete. 2023-10-16T15:34:24.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.804+0000 7fa41bfff640 1 Processor -- start 2023-10-16T15:34:24.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.804+0000 7fa41bfff640 1 -- start start 2023-10-16T15:34:24.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa41bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c071fa0 0x7fa41c0c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa41bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa41bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa41c15cc80 0x7fa41c148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa41bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0c6d50 con 0x7fa41c0cdd60 2023-10-16T15:34:24.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa41bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c15f9a0 con 0x7fa41c15cc80 2023-10-16T15:34:24.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa41bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0b4aa0 con 0x7fa41c071fa0 2023-10-16T15:34:24.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.805+0000 7fa413fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33292/0 (socket says 172.21.15.29:33292) 2023-10-16T15:34:24.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 -- 172.21.15.29:0/1242192378 learned_addr learned my addr 172.21.15.29:0/1242192378 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa41affd640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c071fa0 0x7fa41c0c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa41b7fe640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa41c15cc80 0x7fa41c148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 -- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c071fa0 msgr2=0x7fa41c0c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c071fa0 0x7fa41c0c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 -- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa41c15cc80 msgr2=0x7fa41c148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa41c15cc80 0x7fa41c148ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 -- 172.21.15.29:0/1242192378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4040094d0 con 0x7fa41c0cdd60 2023-10-16T15:34:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.806+0000 7fa413fff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c148980 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fa40402b950 tx=0x7fa4040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.807+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40403ad30 con 0x7fa41c0cdd60 2023-10-16T15:34:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.807+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4040443f0 con 0x7fa41c0cdd60 2023-10-16T15:34:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.807+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4040447a0 con 0x7fa41c0cdd60 2023-10-16T15:34:24.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.807+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c0b4d80 con 0x7fa41c0cdd60 2023-10-16T15:34:24.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.807+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c1468f0 con 0x7fa41c0cdd60 2023-10-16T15:34:24.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.808+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c0b52a0 con 0x7fa41c0cdd60 2023-10-16T15:34:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.809+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa40404d450 con 0x7fa41c0cdd60 2023-10-16T15:34:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.810+0000 7fa418ff9640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fa3fc06af90 0x7fa3fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.810+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fa40402d040 con 0x7fa41c0cdd60 2023-10-16T15:34:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.811+0000 7fa41affd640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fa3fc06af90 0x7fa3fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:24.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.812+0000 7fa41affd640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fa3fc06af90 0x7fa3fc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa40c006fd0 tx=0x7fa40c008040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:24.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:24.816+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa41c0b52a0 con 0x7fa41c0cdd60 2023-10-16T15:34:25.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.012+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa41c0b52a0 con 0x7fa41c0cdd60 2023-10-16T15:34:25.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.543+0000 7fa418ff9640 1 -- 172.21.15.29:0/1242192378 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v190) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa41c0b52a0 con 0x7fa41c0cdd60 2023-10-16T15:34:25.545 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T15:34:25.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fa3fc06af90 msgr2=0x7fa3fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:25.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fa3fc06af90 0x7fa3fc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa40c006fd0 tx=0x7fa40c008040 comp rx=0 tx=0).stop 2023-10-16T15:34:25.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 msgr2=0x7fa41c148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:25.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c148980 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fa40402b950 tx=0x7fa4040046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 shutdown_connections 2023-10-16T15:34:25.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fa3fc06af90 0x7fa3fc06d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa41c0cdd60 0x7fa41c148980 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa41c15cc80 0x7fa41c148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 --2- 172.21.15.29:0/1242192378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa41c071fa0 0x7fa41c0c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.546+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 >> 172.21.15.29:0/1242192378 conn(0x7fa41c065ed0 msgr2=0x7fa41c0b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:25.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.547+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 shutdown_connections 2023-10-16T15:34:25.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.547+0000 7fa41bfff640 1 -- 172.21.15.29:0/1242192378 wait complete. 2023-10-16T15:34:25.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T15:34:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.771+0000 7f4f9bfff640 1 Processor -- start 2023-10-16T15:34:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.771+0000 7f4f9bfff640 1 -- start start 2023-10-16T15:34:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.773+0000 7f4f9bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f9c15cd70 0x7f4f9c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.773+0000 7f4f9bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f9c0b8480 0x7f4f9c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.773+0000 7f4f9bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9c15d150 con 0x7f4f9c0d0be0 2023-10-16T15:34:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.773+0000 7f4f9bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c0d1180 con 0x7f4f9c0b8480 2023-10-16T15:34:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.773+0000 7f4f9bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c0d1460 con 0x7f4f9c15cd70 2023-10-16T15:34:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f9c0b8480 0x7f4f9c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f9c0b8480 0x7f4f9c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33308/0 (socket says 172.21.15.29:33308) 2023-10-16T15:34:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 -- 172.21.15.29:0/2234182793 learned_addr learned my addr 172.21.15.29:0/2234182793 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9affd640 1 --2- 172.21.15.29:0/2234182793 >> v2:172.21.15.170:3300/0 conn(0x7f4f9c15cd70 0x7f4f9c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:25.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 -- 172.21.15.29:0/2234182793 >> v2:172.21.15.170:3300/0 conn(0x7f4f9c15cd70 msgr2=0x7f4f9c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:25.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 --2- 172.21.15.29:0/2234182793 >> v2:172.21.15.170:3300/0 conn(0x7f4f9c15cd70 0x7f4f9c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 -- 172.21.15.29:0/2234182793 >> v1:172.21.15.29:6789/0 conn(0x7f4f9c0d0be0 legacy=0x7f4f9c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:25.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.774+0000 7f4f9a7fc640 1 -- 172.21.15.29:0/2234182793 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c0d1740 con 0x7f4f9c0b8480 2023-10-16T15:34:25.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.775+0000 7f4f9a7fc640 1 --2- 172.21.15.29:0/2234182793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0cd7c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0b56a0 tx=0x7f4f8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb290ce6bb94f58 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:25.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.776+0000 7f4f99ffb640 1 -- 172.21.15.29:0/2234182793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84031870 con 0x7f4f9c0b8480 2023-10-16T15:34:25.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.776+0000 7f4f99ffb640 1 -- 172.21.15.29:0/2234182793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f840319d0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.776+0000 7f4f99ffb640 1 -- 172.21.15.29:0/2234182793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84031d60 con 0x7f4f9c0b8480 2023-10-16T15:34:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.776+0000 7f4f9bfff640 1 -- 172.21.15.29:0/2234182793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 msgr2=0x7f4f9c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.776+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/2234182793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0cd7c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0b56a0 tx=0x7f4f8402b910 comp rx=0 tx=0).stop 2023-10-16T15:34:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.777+0000 7f4f9bfff640 1 -- 172.21.15.29:0/2234182793 shutdown_connections 2023-10-16T15:34:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.777+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/2234182793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0cd7c0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.777+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/2234182793 >> v2:172.21.15.170:3300/0 conn(0x7f4f9c15cd70 0x7f4f9c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.777+0000 7f4f9bfff640 1 -- 172.21.15.29:0/2234182793 >> 172.21.15.29:0/2234182793 conn(0x7f4f9c0cf570 msgr2=0x7f4f9c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.777+0000 7f4f9bfff640 1 -- 172.21.15.29:0/2234182793 shutdown_connections 2023-10-16T15:34:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.777+0000 7f4f9bfff640 1 -- 172.21.15.29:0/2234182793 wait complete. 2023-10-16T15:34:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.779+0000 7f4f9bfff640 1 Processor -- start 2023-10-16T15:34:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.779+0000 7f4f9bfff640 1 -- start start 2023-10-16T15:34:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.780+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.780+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f9c0d0be0 0x7f4f9c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.780+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f9c15cd70 0x7f4f9c0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:25.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.780+0000 7f4f9bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c15e790 con 0x7f4f9c0b8480 2023-10-16T15:34:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.780+0000 7f4f9bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c15d8c0 con 0x7f4f9c0d0be0 2023-10-16T15:34:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.780+0000 7f4f9bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c15db10 con 0x7f4f9c15cd70 2023-10-16T15:34:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33310/0 (socket says 172.21.15.29:33310) 2023-10-16T15:34:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 -- 172.21.15.29:0/4038446646 learned_addr learned my addr 172.21.15.29:0/4038446646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 -- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f9c15cd70 msgr2=0x7f4f9c0cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f9c15cd70 0x7f4f9c0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 -- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f9c0d0be0 msgr2=0x7f4f9c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f9c0d0be0 0x7f4f9c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.781+0000 7f4f9affd640 1 -- 172.21.15.29:0/4038446646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f840094d0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.782+0000 7f4f9affd640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0c8c90 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00d800 tx=0x7f4f8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:25.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.782+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00b6c0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.782+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c00b820 con 0x7f4f9c0b8480 2023-10-16T15:34:25.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.782+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00bbb0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.782+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c15ddf0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.782+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c15e3b0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.784+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c04e9b0 con 0x7f4f9c0b8480 2023-10-16T15:34:25.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.785+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f8c00bd10 con 0x7f4f9c0b8480 2023-10-16T15:34:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.787+0000 7f4f93fff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4f7c06af90 0x7f4f7c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.787+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f4f8c08a790 con 0x7f4f9c0b8480 2023-10-16T15:34:25.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.787+0000 7f4f9a7fc640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4f7c06af90 0x7f4f7c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:25.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.788+0000 7f4f9a7fc640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4f7c06af90 0x7f4f7c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f84036000 tx=0x7f4f84030000 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:25.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:25.790+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f8c0557e0 con 0x7f4f9c0b8480 2023-10-16T15:34:26.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.022+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4f9c0b50d0 con 0x7f4f9c0b8480 2023-10-16T15:34:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.569+0000 7f4f93fff640 1 -- 172.21.15.29:0/4038446646 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v89) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4f8c058e00 con 0x7f4f9c0b8480 2023-10-16T15:34:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 22 and data pool 23 2023-10-16T15:34:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.573+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4f7c06af90 msgr2=0x7f4f7c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.573+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4f7c06af90 0x7f4f7c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f84036000 tx=0x7f4f84030000 comp rx=0 tx=0).stop 2023-10-16T15:34:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.573+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 msgr2=0x7f4f9c0c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.573+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0c8c90 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00d800 tx=0x7f4f8c00dcc0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.574+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 shutdown_connections 2023-10-16T15:34:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.574+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4f7c06af90 0x7f4f7c06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.574+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f9c15cd70 0x7f4f9c0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.574+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f9c0d0be0 0x7f4f9c0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.574+0000 7f4f9bfff640 1 --2- 172.21.15.29:0/4038446646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f9c0b8480 0x7f4f9c0c8c90 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.574+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 >> 172.21.15.29:0/4038446646 conn(0x7f4f9c0cf570 msgr2=0x7f4f9c15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:26.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.575+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 shutdown_connections 2023-10-16T15:34:26.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.575+0000 7f4f9bfff640 1 -- 172.21.15.29:0/4038446646 wait complete. 2023-10-16T15:34:26.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.807+0000 7f73542d7640 1 Processor -- start 2023-10-16T15:34:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.807+0000 7f73542d7640 1 -- start start 2023-10-16T15:34:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.809+0000 7f73542d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f734c15aaa0 0x7f734c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.809+0000 7f73542d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c15ed60 con 0x7f734c15ae80 2023-10-16T15:34:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.809+0000 7f73542d7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0707e0 con 0x7f734c0ce390 2023-10-16T15:34:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.809+0000 7f73542d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734c06b810 con 0x7f734c15aaa0 2023-10-16T15:34:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.810+0000 7f7351872640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f734c15ae80 0x7f734c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50052/0 (socket says 172.21.15.29:50052) 2023-10-16T15:34:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.810+0000 7f7351872640 1 -- 172.21.15.29:0/4254890877 learned_addr learned my addr 172.21.15.29:0/4254890877 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.810+0000 7f7352073640 1 --2- 172.21.15.29:0/4254890877 >> v2:172.21.15.170:3300/0 conn(0x7f734c15aaa0 0x7f734c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.811+0000 7f7352073640 1 -- 172.21.15.29:0/4254890877 >> v1:172.21.15.170:6789/0 conn(0x7f734c0ce390 legacy=0x7f734c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.811+0000 7f7352073640 1 -- 172.21.15.29:0/4254890877 >> v1:172.21.15.29:6789/0 conn(0x7f734c15ae80 legacy=0x7f734c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.811+0000 7f7351071640 1 -- 172.21.15.29:0/4254890877 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478234169 0 0) 0x7f734c15ed60 con 0x7f734c15ae80 2023-10-16T15:34:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.811+0000 7f7352073640 1 -- 172.21.15.29:0/4254890877 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c15ef90 con 0x7f734c15aaa0 2023-10-16T15:34:26.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.812+0000 7f7352073640 1 --2- 172.21.15.29:0/4254890877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15aaa0 0x7f734c15e820 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f73480097a0 tx=0x7f734802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58e264c90b055edf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:26.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.812+0000 7f7351071640 1 -- 172.21.15.29:0/4254890877 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348030610 con 0x7f734c15aaa0 2023-10-16T15:34:26.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.812+0000 7f7351071640 1 -- 172.21.15.29:0/4254890877 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7348030770 con 0x7f734c15aaa0 2023-10-16T15:34:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.812+0000 7f7351071640 1 -- 172.21.15.29:0/4254890877 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348030b00 con 0x7f734c15aaa0 2023-10-16T15:34:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 -- 172.21.15.29:0/4254890877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15aaa0 msgr2=0x7f734c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 --2- 172.21.15.29:0/4254890877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15aaa0 0x7f734c15e820 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f73480097a0 tx=0x7f734802b960 comp rx=0 tx=0).stop 2023-10-16T15:34:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 -- 172.21.15.29:0/4254890877 shutdown_connections 2023-10-16T15:34:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 --2- 172.21.15.29:0/4254890877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15aaa0 0x7f734c15e820 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 -- 172.21.15.29:0/4254890877 >> 172.21.15.29:0/4254890877 conn(0x7f734c0cb410 msgr2=0x7f734c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 -- 172.21.15.29:0/4254890877 shutdown_connections 2023-10-16T15:34:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.813+0000 7f73542d7640 1 -- 172.21.15.29:0/4254890877 wait complete. 2023-10-16T15:34:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.815+0000 7f73542d7640 1 Processor -- start 2023-10-16T15:34:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.815+0000 7f73542d7640 1 -- start start 2023-10-16T15:34:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f73542d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ce390 0x7f734c1f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f73542d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 0x7f734c15cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f73542d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15ae80 0x7f734c15d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f73542d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0c3360 con 0x7f734c15aaa0 2023-10-16T15:34:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f73542d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0c2490 con 0x7f734c15ae80 2023-10-16T15:34:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f73542d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0c26e0 con 0x7f734c0ce390 2023-10-16T15:34:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.816+0000 7f7351872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 0x7f734c15cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 0x7f734c15cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33316/0 (socket says 172.21.15.29:33316) 2023-10-16T15:34:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 -- 172.21.15.29:0/4290395756 learned_addr learned my addr 172.21.15.29:0/4290395756 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7352073640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ce390 0x7f734c1f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7352874640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15ae80 0x7f734c15d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 -- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ce390 msgr2=0x7f734c1f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ce390 0x7f734c1f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 -- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15ae80 msgr2=0x7f734c15d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15ae80 0x7f734c15d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 -- 172.21.15.29:0/4290395756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7348009400 con 0x7f734c15aaa0 2023-10-16T15:34:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7352073640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ce390 0x7f734c1f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.817+0000 7f7351872640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 0x7f734c15cfe0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c3c0 tx=0x7f733c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.818+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c0074e0 con 0x7f734c15aaa0 2023-10-16T15:34:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.818+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c007640 con 0x7f734c15aaa0 2023-10-16T15:34:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.818+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c0c29c0 con 0x7f734c15aaa0 2023-10-16T15:34:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.818+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c004a20 con 0x7f734c15aaa0 2023-10-16T15:34:26.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.818+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c0c2f80 con 0x7f734c15aaa0 2023-10-16T15:34:26.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.819+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c0c34a0 con 0x7f734c15aaa0 2023-10-16T15:34:26.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.820+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f733c004b80 con 0x7f734c15aaa0 2023-10-16T15:34:26.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.821+0000 7f7342ffd640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f733006af90 0x7f733006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:26.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.821+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f733c089680 con 0x7f734c15aaa0 2023-10-16T15:34:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.822+0000 7f7352073640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f733006af90 0x7f733006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.823+0000 7f7352073640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f733006af90 0x7f733006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7348036000 tx=0x7f734803b040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:26.826+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f733c054560 con 0x7f734c15aaa0 2023-10-16T15:34:27.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.012+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f734c04e160 con 0x7f734c15aaa0 2023-10-16T15:34:27.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.015+0000 7f7342ffd640 1 -- 172.21.15.29:0/4290395756 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16628 (secure 0 0 0) 0x7f733c057b80 con 0x7f734c15aaa0 2023-10-16T15:34:27.017 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:27.018 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":191,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:26.023937+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:27.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f733006af90 msgr2=0x7f733006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f733006af90 0x7f733006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7348036000 tx=0x7f734803b040 comp rx=0 tx=0).stop 2023-10-16T15:34:27.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 msgr2=0x7f734c15cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 0x7f734c15cfe0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c3c0 tx=0x7f733c00c880 comp rx=0 tx=0).stop 2023-10-16T15:34:27.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 shutdown_connections 2023-10-16T15:34:27.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f733006af90 0x7f733006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c15aaa0 0x7f734c15cfe0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c15ae80 0x7f734c15d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 --2- 172.21.15.29:0/4290395756 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ce390 0x7f734c1f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 >> 172.21.15.29:0/4290395756 conn(0x7f734c0cb410 msgr2=0x7f734c0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:27.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 shutdown_connections 2023-10-16T15:34:27.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.018+0000 7f73542d7640 1 -- 172.21.15.29:0/4290395756 wait complete. 2023-10-16T15:34:27.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:27.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.263+0000 7ff5b4d06640 1 Processor -- start 2023-10-16T15:34:27.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.263+0000 7ff5b4d06640 1 -- start start 2023-10-16T15:34:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5b4d06640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5b00ad970 0x7ff5b00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5b4d06640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b015b9d0 con 0x7ff5b00ccfb0 2023-10-16T15:34:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5b4d06640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b015bcb0 con 0x7ff5b006c040 2023-10-16T15:34:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5b4d06640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b015bf90 con 0x7ff5b00ad970 2023-10-16T15:34:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5a7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5b00ad970 0x7ff5b00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5a7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5b00ad970 0x7ff5b00cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33324/0 (socket says 172.21.15.29:33324) 2023-10-16T15:34:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.265+0000 7ff5a7fff640 1 -- 172.21.15.29:0/921967608 learned_addr learned my addr 172.21.15.29:0/921967608 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.266+0000 7ff5a7fff640 1 -- 172.21.15.29:0/921967608 >> v1:172.21.15.170:6789/0 conn(0x7ff5b006c040 legacy=0x7ff5b006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.266+0000 7ff5a7fff640 1 -- 172.21.15.29:0/921967608 >> v1:172.21.15.29:6789/0 conn(0x7ff5b00ccfb0 legacy=0x7ff5b00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.266+0000 7ff5a7fff640 1 -- 172.21.15.29:0/921967608 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b00cc270 con 0x7ff5b00ad970 2023-10-16T15:34:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.267+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/921967608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b00cca70 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff5980098a0 tx=0x7ff59802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8573826f2f24d584 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5aeffd640 1 -- 172.21.15.29:0/921967608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59802f830 con 0x7ff5b00ad970 2023-10-16T15:34:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5aeffd640 1 -- 172.21.15.29:0/921967608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59802f990 con 0x7ff5b00ad970 2023-10-16T15:34:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5aeffd640 1 -- 172.21.15.29:0/921967608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59802fd20 con 0x7ff5b00ad970 2023-10-16T15:34:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 -- 172.21.15.29:0/921967608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 msgr2=0x7ff5b00cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/921967608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b00cca70 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff5980098a0 tx=0x7ff59802ba10 comp rx=0 tx=0).stop 2023-10-16T15:34:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 -- 172.21.15.29:0/921967608 shutdown_connections 2023-10-16T15:34:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/921967608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b00cca70 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 -- 172.21.15.29:0/921967608 >> 172.21.15.29:0/921967608 conn(0x7ff5b00ce6a0 msgr2=0x7ff5b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 -- 172.21.15.29:0/921967608 shutdown_connections 2023-10-16T15:34:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.268+0000 7ff5b4d06640 1 -- 172.21.15.29:0/921967608 wait complete. 2023-10-16T15:34:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.270+0000 7ff5b4d06640 1 Processor -- start 2023-10-16T15:34:27.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.270+0000 7ff5b4d06640 1 -- start start 2023-10-16T15:34:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.271+0000 7ff5b4d06640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5b006c040 0x7ff5b00b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.271+0000 7ff5b4d06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b01493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.271+0000 7ff5b4d06640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5b00ccfb0 0x7ff5b00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.271+0000 7ff5b4d06640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00b31d0 con 0x7ff5b00ad970 2023-10-16T15:34:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.271+0000 7ff5b4d06640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00b3470 con 0x7ff5b00ccfb0 2023-10-16T15:34:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.271+0000 7ff5b4d06640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00b3750 con 0x7ff5b006c040 2023-10-16T15:34:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b01493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b01493e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33334/0 (socket says 172.21.15.29:33334) 2023-10-16T15:34:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 -- 172.21.15.29:0/4118335884 learned_addr learned my addr 172.21.15.29:0/4118335884 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 -- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5b006c040 msgr2=0x7ff5b00b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5b006c040 0x7ff5b00b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 -- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5b00ccfb0 msgr2=0x7ff5b00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5b00ccfb0 0x7ff5b00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.272+0000 7ff5a7fff640 1 -- 172.21.15.29:0/4118335884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5980094d0 con 0x7ff5b00ad970 2023-10-16T15:34:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.273+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b01493e0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ff598000c00 tx=0x7ff598004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.273+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59802f800 con 0x7ff5b00ad970 2023-10-16T15:34:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.273+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff598041400 con 0x7ff5b00ad970 2023-10-16T15:34:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.273+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5980417d0 con 0x7ff5b00ad970 2023-10-16T15:34:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.273+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b00b3a30 con 0x7ff5b00ad970 2023-10-16T15:34:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.273+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b00acdd0 con 0x7ff5b00ad970 2023-10-16T15:34:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.274+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b006b330 con 0x7ff5b00ad970 2023-10-16T15:34:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.275+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff598041930 con 0x7ff5b00ad970 2023-10-16T15:34:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.276+0000 7ff5acff9640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff59006af90 0x7ff59006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.277+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff59803c070 con 0x7ff5b00ad970 2023-10-16T15:34:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.277+0000 7ff5af7fe640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff59006af90 0x7ff59006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.278+0000 7ff5af7fe640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff59006af90 0x7ff59006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0000f60 tx=0x7ff5a00079b0 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:27.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.280+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff598077680 con 0x7ff5b00ad970 2023-10-16T15:34:27.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.475+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b004e9b0 con 0x7ff5b00ad970 2023-10-16T15:34:27.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.478+0000 7ff5acff9640 1 -- 172.21.15.29:0/4118335884 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16628 (secure 0 0 0) 0x7ff59807aca0 con 0x7ff5b00ad970 2023-10-16T15:34:27.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:27.480 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":191,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:26.023937+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:27.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff59006af90 msgr2=0x7ff59006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff59006af90 0x7ff59006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0000f60 tx=0x7ff5a00079b0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 msgr2=0x7ff5b01493e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b01493e0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ff598000c00 tx=0x7ff598004730 comp rx=0 tx=0).stop 2023-10-16T15:34:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 shutdown_connections 2023-10-16T15:34:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7ff59006af90 0x7ff59006d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5b00ad970 0x7ff5b01493e0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5b00ccfb0 0x7ff5b00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 --2- 172.21.15.29:0/4118335884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5b006c040 0x7ff5b00b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 >> 172.21.15.29:0/4118335884 conn(0x7ff5b00ce6a0 msgr2=0x7ff5b006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:27.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 shutdown_connections 2023-10-16T15:34:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.481+0000 7ff5b4d06640 1 -- 172.21.15.29:0/4118335884 wait complete. 2023-10-16T15:34:27.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T15:34:27.528 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-16T15:34:27.527+0000 7f9dbdf29640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:34:27.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.728+0000 7f4ecc75e640 1 Processor -- start 2023-10-16T15:34:27.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.728+0000 7f4ecc75e640 1 -- start start 2023-10-16T15:34:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4ecc75e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ec415ad70 0x7f4ec4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4ecc75e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40cb030 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4ecc75e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40c91c0 con 0x7f4ec40caac0 2023-10-16T15:34:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4ecc75e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec40c94a0 con 0x7f4ec415ad70 2023-10-16T15:34:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4eca4fa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ec415ad70 0x7f4ec4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4ec9cf9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4ec40c9ca0 0x7f4ec40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50064/0 (socket says 172.21.15.29:50064) 2023-10-16T15:34:27.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.730+0000 7f4ec9cf9640 1 -- 172.21.15.29:0/2931415869 learned_addr learned my addr 172.21.15.29:0/2931415869 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:27.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.731+0000 7f4eca4fa640 1 -- 172.21.15.29:0/2931415869 >> v1:172.21.15.170:6789/0 conn(0x7f4ec40caac0 legacy=0x7f4ec40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:27.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.731+0000 7f4ec94f8640 1 -- 172.21.15.29:0/2931415869 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808449119 0 0) 0x7f4ec40cb030 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.731+0000 7f4eca4fa640 1 -- 172.21.15.29:0/2931415869 >> v1:172.21.15.29:6789/0 conn(0x7f4ec40c9ca0 legacy=0x7f4ec40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.731+0000 7f4eca4fa640 1 -- 172.21.15.29:0/2931415869 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40c9780 con 0x7f4ec415ad70 2023-10-16T15:34:27.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.732+0000 7f4eca4fa640 1 --2- 172.21.15.29:0/2931415869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec415ad70 0x7f4ec4069da0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1987fafbf9ddc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:27.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ec94f8640 1 -- 172.21.15.29:0/2931415869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0031830 con 0x7f4ec415ad70 2023-10-16T15:34:27.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ec94f8640 1 -- 172.21.15.29:0/2931415869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0031990 con 0x7f4ec415ad70 2023-10-16T15:34:27.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ec94f8640 1 -- 172.21.15.29:0/2931415869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0031d20 con 0x7f4ec415ad70 2023-10-16T15:34:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 -- 172.21.15.29:0/2931415869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec415ad70 msgr2=0x7f4ec4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/2931415869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec415ad70 0x7f4ec4069da0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec002b8d0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 -- 172.21.15.29:0/2931415869 shutdown_connections 2023-10-16T15:34:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/2931415869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec415ad70 0x7f4ec4069da0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 -- 172.21.15.29:0/2931415869 >> 172.21.15.29:0/2931415869 conn(0x7f4ec40ce9b0 msgr2=0x7f4ec40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:27.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 -- 172.21.15.29:0/2931415869 shutdown_connections 2023-10-16T15:34:27.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.733+0000 7f4ecc75e640 1 -- 172.21.15.29:0/2931415869 wait complete. 2023-10-16T15:34:27.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.735+0000 7f4ecc75e640 1 Processor -- start 2023-10-16T15:34:27.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.735+0000 7f4ecc75e640 1 -- start start 2023-10-16T15:34:27.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4ecc75e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 0x7f4ec41f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4ecc75e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ec40caac0 0x7f4ec40cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4ecc75e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ec415ad70 0x7f4ec4159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4ecc75e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40c58c0 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4ecc75e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40ccac0 con 0x7f4ec415ad70 2023-10-16T15:34:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4ecc75e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec4159650 con 0x7f4ec40caac0 2023-10-16T15:34:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4eca4fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 0x7f4ec41f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4eca4fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 0x7f4ec41f2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33350/0 (socket says 172.21.15.29:33350) 2023-10-16T15:34:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.736+0000 7f4eca4fa640 1 -- 172.21.15.29:0/3810885886 learned_addr learned my addr 172.21.15.29:0/3810885886 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4ecacfb640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ec415ad70 0x7f4ec4159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4ec9cf9640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ec40caac0 0x7f4ec40cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4eca4fa640 1 -- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ec40caac0 msgr2=0x7f4ec40cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4eca4fa640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ec40caac0 0x7f4ec40cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4eca4fa640 1 -- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ec415ad70 msgr2=0x7f4ec4159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4eca4fa640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ec415ad70 0x7f4ec4159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4eca4fa640 1 -- 172.21.15.29:0/3810885886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec0009400 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4ecacfb640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ec415ad70 0x7f4ec4159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.737+0000 7f4eca4fa640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 0x7f4ec41f2000 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.738+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00317d0 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.738+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0004030 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.738+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00043e0 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.738+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec40cccf0 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.738+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec40cd2b0 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.739+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec404e160 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.740+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4ec0043070 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.741+0000 7f4ebaffd640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4ea806af90 0x7f4ea806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:27.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.741+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4ec00ab6a0 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.742+0000 7f4ec9cf9640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4ea806af90 0x7f4ea806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.743+0000 7f4ec9cf9640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4ea806af90 0x7f4ea806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4007fd0 tx=0x7f4eb4009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.745+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ec0076470 con 0x7f4ec40c9ca0 2023-10-16T15:34:27.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:27.978+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4ec406fcc0 con 0x7f4ec40c9ca0 2023-10-16T15:34:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.606+0000 7f4ebaffd640 1 -- 172.21.15.29:0/3810885886 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4ec0079a90 con 0x7f4ec40c9ca0 2023-10-16T15:34:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4ea806af90 msgr2=0x7f4ea806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4ea806af90 0x7f4ea806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4007fd0 tx=0x7f4eb4009040 comp rx=0 tx=0).stop 2023-10-16T15:34:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 msgr2=0x7f4ec41f2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 0x7f4ec41f2000 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec00046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 shutdown_connections 2023-10-16T15:34:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f4ea806af90 0x7f4ea806d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ec415ad70 0x7f4ec4159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4ec40caac0 0x7f4ec40cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 --2- 172.21.15.29:0/3810885886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ec40c9ca0 0x7f4ec41f2000 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.611+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 >> 172.21.15.29:0/3810885886 conn(0x7f4ec40ce9b0 msgr2=0x7f4ec406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.612+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 shutdown_connections 2023-10-16T15:34:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.612+0000 7f4ecc75e640 1 -- 172.21.15.29:0/3810885886 wait complete. 2023-10-16T15:34:28.636 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:34:28.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T15:34:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.920+0000 7fe28759e640 1 Processor -- start 2023-10-16T15:34:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.921+0000 7fe28759e640 1 -- start start 2023-10-16T15:34:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe28759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28815cd70 0x7fe2880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe28759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2880cded0 0x7fe2880b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe28759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28815d150 con 0x7fe2880b8480 2023-10-16T15:34:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe28759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2880d1180 con 0x7fe2880cded0 2023-10-16T15:34:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe28759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2880d1460 con 0x7fe28815cd70 2023-10-16T15:34:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe286d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2880cded0 0x7fe2880b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe286d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2880cded0 0x7fe2880b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33360/0 (socket says 172.21.15.29:33360) 2023-10-16T15:34:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.922+0000 7fe286d9d640 1 -- 172.21.15.29:0/2144735592 learned_addr learned my addr 172.21.15.29:0/2144735592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.923+0000 7fe28659c640 1 --2- 172.21.15.29:0/2144735592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28815cd70 0x7fe2880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.923+0000 7fe286d9d640 1 -- 172.21.15.29:0/2144735592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28815cd70 msgr2=0x7fe2880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.923+0000 7fe286d9d640 1 --2- 172.21.15.29:0/2144735592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28815cd70 0x7fe2880d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.923+0000 7fe286d9d640 1 -- 172.21.15.29:0/2144735592 >> v1:172.21.15.29:6789/0 conn(0x7fe2880b8480 legacy=0x7fe2880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.923+0000 7fe286d9d640 1 -- 172.21.15.29:0/2144735592 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2880d1740 con 0x7fe2880cded0 2023-10-16T15:34:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.924+0000 7fe286d9d640 1 --2- 172.21.15.29:0/2144735592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe2880b5c90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fe280009870 tx=0x7fe28002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7526c2ab32d3c87d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.924+0000 7fe28559a640 1 -- 172.21.15.29:0/2144735592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe280030840 con 0x7fe2880cded0 2023-10-16T15:34:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.924+0000 7fe28559a640 1 -- 172.21.15.29:0/2144735592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2800309a0 con 0x7fe2880cded0 2023-10-16T15:34:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.924+0000 7fe28559a640 1 -- 172.21.15.29:0/2144735592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe280030d30 con 0x7fe2880cded0 2023-10-16T15:34:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 -- 172.21.15.29:0/2144735592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 msgr2=0x7fe2880b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 --2- 172.21.15.29:0/2144735592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe2880b5c90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fe280009870 tx=0x7fe28002b980 comp rx=0 tx=0).stop 2023-10-16T15:34:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 -- 172.21.15.29:0/2144735592 shutdown_connections 2023-10-16T15:34:28.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 --2- 172.21.15.29:0/2144735592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe2880b5c90 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 --2- 172.21.15.29:0/2144735592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe28815cd70 0x7fe2880d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 -- 172.21.15.29:0/2144735592 >> 172.21.15.29:0/2144735592 conn(0x7fe2880cf570 msgr2=0x7fe2880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 -- 172.21.15.29:0/2144735592 shutdown_connections 2023-10-16T15:34:28.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.925+0000 7fe28759e640 1 -- 172.21.15.29:0/2144735592 wait complete. 2023-10-16T15:34:28.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.927+0000 7fe28759e640 1 Processor -- start 2023-10-16T15:34:28.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.927+0000 7fe28759e640 1 -- start start 2023-10-16T15:34:28.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe28759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2880b8480 0x7fe28815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:28.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe28759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe28815de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:28.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe28759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28815cd70 0x7fe2881591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:28.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe28759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2880c7ea0 con 0x7fe2880cded0 2023-10-16T15:34:28.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe28759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28815e380 con 0x7fe28815cd70 2023-10-16T15:34:28.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe28759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2881f2270 con 0x7fe2880b8480 2023-10-16T15:34:28.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe285d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe28815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:28.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe285d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe28815de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33362/0 (socket says 172.21.15.29:33362) 2023-10-16T15:34:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.928+0000 7fe285d9b640 1 -- 172.21.15.29:0/1513238923 learned_addr learned my addr 172.21.15.29:0/1513238923 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe286d9d640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28815cd70 0x7fe2881591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe285d9b640 1 -- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2880b8480 msgr2=0x7fe28815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe28659c640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2880b8480 0x7fe28815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe285d9b640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2880b8480 0x7fe28815d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe285d9b640 1 -- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28815cd70 msgr2=0x7fe2881591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe285d9b640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28815cd70 0x7fe2881591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe285d9b640 1 -- 172.21.15.29:0/1513238923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800094d0 con 0x7fe2880cded0 2023-10-16T15:34:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe286d9d640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28815cd70 0x7fe2881591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe285d9b640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe28815de40 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c450 tx=0x7fe27000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2700052d0 con 0x7fe2880cded0 2023-10-16T15:34:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe270005430 con 0x7fe2880cded0 2023-10-16T15:34:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270012870 con 0x7fe2880cded0 2023-10-16T15:34:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28815d9d0 con 0x7fe2880cded0 2023-10-16T15:34:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.929+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2880c77e0 con 0x7fe2880cded0 2023-10-16T15:34:28.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.931+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28804e9b0 con 0x7fe2880cded0 2023-10-16T15:34:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.932+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2700129d0 con 0x7fe2880cded0 2023-10-16T15:34:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.932+0000 7fe27effd640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe26806af90 0x7fe26806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.933+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe270089250 con 0x7fe2880cded0 2023-10-16T15:34:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.933+0000 7fe28659c640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe26806af90 0x7fe26806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.934+0000 7fe28659c640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe26806af90 0x7fe26806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe278009e90 tx=0x7fe27800b040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:28.937+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe270054260 con 0x7fe2880cded0 2023-10-16T15:34:29.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.166+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe2880cd1f0 con 0x7fe2880cded0 2023-10-16T15:34:29.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.612+0000 7fe27effd640 1 -- 172.21.15.29:0/1513238923 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v92) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe270057880 con 0x7fe2880cded0 2023-10-16T15:34:29.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe26806af90 msgr2=0x7fe26806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:29.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe26806af90 0x7fe26806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe278009e90 tx=0x7fe27800b040 comp rx=0 tx=0).stop 2023-10-16T15:34:29.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 msgr2=0x7fe28815de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:29.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe28815de40 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c450 tx=0x7fe27000c910 comp rx=0 tx=0).stop 2023-10-16T15:34:29.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 shutdown_connections 2023-10-16T15:34:29.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe26806af90 0x7fe26806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2880cded0 0x7fe28815de40 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe28815cd70 0x7fe2881591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 --2- 172.21.15.29:0/1513238923 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2880b8480 0x7fe28815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 >> 172.21.15.29:0/1513238923 conn(0x7fe2880cf570 msgr2=0x7fe2880b7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:29.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 shutdown_connections 2023-10-16T15:34:29.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.615+0000 7fe28759e640 1 -- 172.21.15.29:0/1513238923 wait complete. 2023-10-16T15:34:29.636 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T15:34:29.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:29.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.863+0000 7f55f7fff640 1 Processor -- start 2023-10-16T15:34:29.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.864+0000 7f55f7fff640 1 -- start start 2023-10-16T15:34:29.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.865+0000 7f55f7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55f806c040 0x7f55f806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.865+0000 7f55f7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f55f80ad970 0x7f55f815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f7fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f806c960 con 0x7f55f815ebc0 2023-10-16T15:34:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f7fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f815b9d0 con 0x7f55f806c040 2023-10-16T15:34:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f7fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f815bcb0 con 0x7f55f80ad970 2023-10-16T15:34:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55f806c040 0x7f55f806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55f806c040 0x7f55f806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33376/0 (socket says 172.21.15.29:33376) 2023-10-16T15:34:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 -- 172.21.15.29:0/4011947259 learned_addr learned my addr 172.21.15.29:0/4011947259 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 -- 172.21.15.29:0/4011947259 >> v2:172.21.15.170:3300/0 conn(0x7f55f80ad970 msgr2=0x7f55f815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 --2- 172.21.15.29:0/4011947259 >> v2:172.21.15.170:3300/0 conn(0x7f55f80ad970 0x7f55f815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 -- 172.21.15.29:0/4011947259 >> v1:172.21.15.170:6789/0 conn(0x7f55f815ebc0 legacy=0x7f55f80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:29.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.866+0000 7f55f6ffd640 1 -- 172.21.15.29:0/4011947259 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f815bf90 con 0x7f55f806c040 2023-10-16T15:34:29.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.868+0000 7f55f6ffd640 1 --2- 172.21.15.29:0/4011947259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f806c040 0x7f55f806c420 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f55f80cb060 tx=0x7f55e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c30a3e60a1ace4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:29.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.868+0000 7f55f67fc640 1 -- 172.21.15.29:0/4011947259 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8031870 con 0x7f55f806c040 2023-10-16T15:34:29.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.868+0000 7f55f67fc640 1 -- 172.21.15.29:0/4011947259 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e80319d0 con 0x7f55f806c040 2023-10-16T15:34:29.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.868+0000 7f55f67fc640 1 -- 172.21.15.29:0/4011947259 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8031d60 con 0x7f55f806c040 2023-10-16T15:34:29.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 -- 172.21.15.29:0/4011947259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f806c040 msgr2=0x7f55f806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:29.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 --2- 172.21.15.29:0/4011947259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f806c040 0x7f55f806c420 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f55f80cb060 tx=0x7f55e802b910 comp rx=0 tx=0).stop 2023-10-16T15:34:29.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 -- 172.21.15.29:0/4011947259 shutdown_connections 2023-10-16T15:34:29.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 --2- 172.21.15.29:0/4011947259 >> v2:172.21.15.170:3300/0 conn(0x7f55f80ad970 0x7f55f815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 --2- 172.21.15.29:0/4011947259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f806c040 0x7f55f806c420 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 -- 172.21.15.29:0/4011947259 >> 172.21.15.29:0/4011947259 conn(0x7f55f80ce6a0 msgr2=0x7f55f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:29.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 -- 172.21.15.29:0/4011947259 shutdown_connections 2023-10-16T15:34:29.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.869+0000 7f55f7fff640 1 -- 172.21.15.29:0/4011947259 wait complete. 2023-10-16T15:34:29.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.871+0000 7f55f7fff640 1 Processor -- start 2023-10-16T15:34:29.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.871+0000 7f55f7fff640 1 -- start start 2023-10-16T15:34:29.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55f806c040 0x7f55f80b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55f80ad970 0x7f55f80ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 0x7f55f8151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80b0c90 con 0x7f55f815ebc0 2023-10-16T15:34:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f7fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f81521b0 con 0x7f55f80ad970 2023-10-16T15:34:29.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f7fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80ae330 con 0x7f55f806c040 2023-10-16T15:34:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 0x7f55f8151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 0x7f55f8151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33390/0 (socket says 172.21.15.29:33390) 2023-10-16T15:34:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.873+0000 7f55f77fe640 1 -- 172.21.15.29:0/2622792316 learned_addr learned my addr 172.21.15.29:0/2622792316 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f6ffd640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55f806c040 0x7f55f80b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:29.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55effff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55f80ad970 0x7f55f80ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f77fe640 1 -- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55f806c040 msgr2=0x7f55f80b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f77fe640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55f806c040 0x7f55f80b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f77fe640 1 -- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55f80ad970 msgr2=0x7f55f80ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f77fe640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55f80ad970 0x7f55f80ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f77fe640 1 -- 172.21.15.29:0/2622792316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e80094d0 con 0x7f55f815ebc0 2023-10-16T15:34:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f77fe640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 0x7f55f8151c70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c7b0 tx=0x7f55f000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:29.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0003d80 con 0x7f55f815ebc0 2023-10-16T15:34:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f0005070 con 0x7f55f815ebc0 2023-10-16T15:34:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f80b3430 con 0x7f55f815ebc0 2023-10-16T15:34:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.874+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0005420 con 0x7f55f815ebc0 2023-10-16T15:34:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.875+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f80b39f0 con 0x7f55f815ebc0 2023-10-16T15:34:29.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.876+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f806b330 con 0x7f55f815ebc0 2023-10-16T15:34:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.877+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f55f0005580 con 0x7f55f815ebc0 2023-10-16T15:34:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.878+0000 7f55ef7fe640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f55d806af90 0x7f55d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:29.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.878+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f55f0088fe0 con 0x7f55f815ebc0 2023-10-16T15:34:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.879+0000 7f55f6ffd640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f55d806af90 0x7f55d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.880+0000 7f55f6ffd640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f55d806af90 0x7f55d806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e803c040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:29.883+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f55f0053f90 con 0x7f55f815ebc0 2023-10-16T15:34:30.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.119+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f80ae4a0 con 0x7f55f815ebc0 2023-10-16T15:34:30.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.120+0000 7f55ef7fe640 1 -- 172.21.15.29:0/2622792316 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f55f80ae4a0 con 0x7f55f815ebc0 2023-10-16T15:34:30.122 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:30.122 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:34:26.024272+0000","modified":"2023-10-16T15:34:29.168102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2122514336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122514336},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122514336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T15:34:30.123 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 92 2023-10-16T15:34:30.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f55d806af90 msgr2=0x7f55d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f55d806af90 0x7f55d806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e803c040 comp rx=0 tx=0).stop 2023-10-16T15:34:30.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 msgr2=0x7f55f8151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 0x7f55f8151c70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c7b0 tx=0x7f55f000cc70 comp rx=0 tx=0).stop 2023-10-16T15:34:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 shutdown_connections 2023-10-16T15:34:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f55d806af90 0x7f55d806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f815ebc0 0x7f55f8151c70 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f55f80ad970 0x7f55f80ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 --2- 172.21.15.29:0/2622792316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f55f806c040 0x7f55f80b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 >> 172.21.15.29:0/2622792316 conn(0x7f55f80ce6a0 msgr2=0x7f55f80c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:30.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.123+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 shutdown_connections 2023-10-16T15:34:30.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.124+0000 7f55f7fff640 1 -- 172.21.15.29:0/2622792316 wait complete. 2023-10-16T15:34:30.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:34:30.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.374+0000 7f2478ed0640 1 Processor -- start 2023-10-16T15:34:30.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.374+0000 7f2478ed0640 1 -- start start 2023-10-16T15:34:30.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.376+0000 7f2478ed0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 0x7f247415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.376+0000 7f2478ed0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24740ce390 0x7f2474070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.376+0000 7f2478ed0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247415ed60 con 0x7f247415aaa0 2023-10-16T15:34:30.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.376+0000 7f2478ed0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247406b810 con 0x7f247415ae80 2023-10-16T15:34:30.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.376+0000 7f2478ed0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24740707e0 con 0x7f24740ce390 2023-10-16T15:34:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.376+0000 7f246bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 0x7f247415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f246bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 0x7f247415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59674/0 (socket says 172.21.15.29:59674) 2023-10-16T15:34:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f246bfff640 1 -- 172.21.15.29:0/2019234535 learned_addr learned my addr 172.21.15.29:0/2019234535 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f2472d9d640 1 --2- 172.21.15.29:0/2019234535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24740ce390 0x7f2474070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f2472d9d640 1 -- 172.21.15.29:0/2019234535 >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 msgr2=0x7f247415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f2472d9d640 1 --2- 172.21.15.29:0/2019234535 >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 0x7f247415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f2472d9d640 1 -- 172.21.15.29:0/2019234535 >> v1:172.21.15.170:6789/0 conn(0x7f247415aaa0 legacy=0x7f24740cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f2472d9d640 1 -- 172.21.15.29:0/2019234535 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247415ef90 con 0x7f24740ce390 2023-10-16T15:34:30.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.377+0000 7f246bfff640 1 --2- 172.21.15.29:0/2019234535 >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 0x7f247415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:34:30.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.379+0000 7f2472d9d640 1 --2- 172.21.15.29:0/2019234535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24740ce390 0x7f2474070f50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f246c009d90 tx=0x7f246c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e6e8e17fed0780 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.379+0000 7f2471d9b640 1 -- 172.21.15.29:0/2019234535 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c02cc60 con 0x7f24740ce390 2023-10-16T15:34:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.379+0000 7f2471d9b640 1 -- 172.21.15.29:0/2019234535 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c02cdc0 con 0x7f24740ce390 2023-10-16T15:34:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.379+0000 7f2471d9b640 1 -- 172.21.15.29:0/2019234535 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c034660 con 0x7f24740ce390 2023-10-16T15:34:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 -- 172.21.15.29:0/2019234535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24740ce390 msgr2=0x7f2474070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 --2- 172.21.15.29:0/2019234535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24740ce390 0x7f2474070f50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f246c009d90 tx=0x7f246c02beb0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 -- 172.21.15.29:0/2019234535 shutdown_connections 2023-10-16T15:34:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 --2- 172.21.15.29:0/2019234535 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f24740ce390 0x7f2474070f50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 --2- 172.21.15.29:0/2019234535 >> v2:172.21.15.170:3300/0 conn(0x7f247415ae80 0x7f247415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 -- 172.21.15.29:0/2019234535 >> 172.21.15.29:0/2019234535 conn(0x7f24740cb410 msgr2=0x7f24740cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 -- 172.21.15.29:0/2019234535 shutdown_connections 2023-10-16T15:34:30.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.380+0000 7f2478ed0640 1 -- 172.21.15.29:0/2019234535 wait complete. 2023-10-16T15:34:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.382+0000 7f2478ed0640 1 Processor -- start 2023-10-16T15:34:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.382+0000 7f2478ed0640 1 -- start start 2023-10-16T15:34:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f2478ed0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24740ce390 0x7f24741523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f2478ed0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 0x7f247415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f2478ed0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247415ae80 0x7f247414cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f2478ed0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247414d7d0 con 0x7f247415aaa0 2023-10-16T15:34:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f2478ed0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247414dab0 con 0x7f24740ce390 2023-10-16T15:34:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f2478ed0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247414dd90 con 0x7f247415ae80 2023-10-16T15:34:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f246bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 0x7f247415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f246bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 0x7f247415bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33402/0 (socket says 172.21.15.29:33402) 2023-10-16T15:34:30.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f246bfff640 1 -- 172.21.15.29:0/298923572 learned_addr learned my addr 172.21.15.29:0/298923572 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.383+0000 7f247259c640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24740ce390 0x7f24741523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f2472d9d640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247415ae80 0x7f247414cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246bfff640 1 -- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247415ae80 msgr2=0x7f247414cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246bfff640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247415ae80 0x7f247414cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246bfff640 1 -- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24740ce390 msgr2=0x7f24741523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246bfff640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24740ce390 0x7f24741523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246bfff640 1 -- 172.21.15.29:0/298923572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c0094d0 con 0x7f247415aaa0 2023-10-16T15:34:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f2472d9d640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247415ae80 0x7f247414cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246bfff640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 0x7f247415bb50 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f245c00e9a0 tx=0x7f245c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00ca00 con 0x7f247415aaa0 2023-10-16T15:34:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c00cb60 con 0x7f247415aaa0 2023-10-16T15:34:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c013470 con 0x7f247415aaa0 2023-10-16T15:34:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c009a70 con 0x7f247415aaa0 2023-10-16T15:34:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.384+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247414e350 con 0x7f247415aaa0 2023-10-16T15:34:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.386+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247404e160 con 0x7f247415aaa0 2023-10-16T15:34:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.387+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f245c0135d0 con 0x7f247415aaa0 2023-10-16T15:34:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.388+0000 7f246affd640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f245406af90 0x7f245406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.388+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f245c089a40 con 0x7f247415aaa0 2023-10-16T15:34:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.389+0000 7f247259c640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f245406af90 0x7f245406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.390+0000 7f247259c640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f245406af90 0x7f245406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2464000f60 tx=0x7f2464009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:30.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.393+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f245c08be60 con 0x7f247415aaa0 2023-10-16T15:34:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.585+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2474070a00 con 0x7f247415aaa0 2023-10-16T15:34:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.588+0000 7f246affd640 1 -- 172.21.15.29:0/298923572 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f245c021090 con 0x7f247415aaa0 2023-10-16T15:34:30.590 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:30.590 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":192,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:34:27.569352+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:34:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f245406af90 msgr2=0x7f245406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f245406af90 0x7f245406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2464000f60 tx=0x7f2464009040 comp rx=0 tx=0).stop 2023-10-16T15:34:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 msgr2=0x7f247415bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 0x7f247415bb50 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f245c00e9a0 tx=0x7f245c00ee60 comp rx=0 tx=0).stop 2023-10-16T15:34:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 shutdown_connections 2023-10-16T15:34:30.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f245406af90 0x7f245406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f247415aaa0 0x7f247415bb50 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f247415ae80 0x7f247414cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 --2- 172.21.15.29:0/298923572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24740ce390 0x7f24741523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.590+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 >> 172.21.15.29:0/298923572 conn(0x7f24740cb410 msgr2=0x7f247406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.591+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 shutdown_connections 2023-10-16T15:34:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.591+0000 7f2478ed0640 1 -- 172.21.15.29:0/298923572 wait complete. 2023-10-16T15:34:30.615 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:34:30.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:34:30.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.841+0000 7f036e3d0640 1 Processor -- start 2023-10-16T15:34:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.841+0000 7f036e3d0640 1 -- start start 2023-10-16T15:34:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.842+0000 7f036e3d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f036815aaa0 0x7f036815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.842+0000 7f036e3d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f036815ae80 0x7f03680cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.842+0000 7f036e3d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036815ed60 con 0x7f03680ccaa0 2023-10-16T15:34:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.842+0000 7f036e3d0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036806b810 con 0x7f036815ae80 2023-10-16T15:34:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.842+0000 7f036e3d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680707e0 con 0x7f036815aaa0 2023-10-16T15:34:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.843+0000 7f03677fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f036815ae80 0x7f03680cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.843+0000 7f03677fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f036815ae80 0x7f03680cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33408/0 (socket says 172.21.15.29:33408) 2023-10-16T15:34:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.843+0000 7f03677fe640 1 -- 172.21.15.29:0/1175660616 learned_addr learned my addr 172.21.15.29:0/1175660616 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.843+0000 7f0367fff640 1 --2- 172.21.15.29:0/1175660616 >> v2:172.21.15.170:3300/0 conn(0x7f036815aaa0 0x7f036815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.843+0000 7f03677fe640 1 -- 172.21.15.29:0/1175660616 >> v2:172.21.15.170:3300/0 conn(0x7f036815aaa0 msgr2=0x7f036815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.843+0000 7f03677fe640 1 --2- 172.21.15.29:0/1175660616 >> v2:172.21.15.170:3300/0 conn(0x7f036815aaa0 0x7f036815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.844+0000 7f03677fe640 1 -- 172.21.15.29:0/1175660616 >> v1:172.21.15.170:6789/0 conn(0x7f03680ccaa0 legacy=0x7f0368070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.844+0000 7f03677fe640 1 -- 172.21.15.29:0/1175660616 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036815ef90 con 0x7f036815ae80 2023-10-16T15:34:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.845+0000 7f03677fe640 1 --2- 172.21.15.29:0/1175660616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f036815ae80 0x7f03680cc560 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f03680cd990 tx=0x7f035002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c125a00815daaaf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.845+0000 7f0366ffd640 1 -- 172.21.15.29:0/1175660616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350031870 con 0x7f036815ae80 2023-10-16T15:34:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.845+0000 7f0366ffd640 1 -- 172.21.15.29:0/1175660616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03500319d0 con 0x7f036815ae80 2023-10-16T15:34:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.845+0000 7f0366ffd640 1 -- 172.21.15.29:0/1175660616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350031d60 con 0x7f036815ae80 2023-10-16T15:34:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 -- 172.21.15.29:0/1175660616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f036815ae80 msgr2=0x7f03680cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1175660616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f036815ae80 0x7f03680cc560 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f03680cd990 tx=0x7f035002b910 comp rx=0 tx=0).stop 2023-10-16T15:34:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 -- 172.21.15.29:0/1175660616 shutdown_connections 2023-10-16T15:34:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1175660616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f036815ae80 0x7f03680cc560 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1175660616 >> v2:172.21.15.170:3300/0 conn(0x7f036815aaa0 0x7f036815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 -- 172.21.15.29:0/1175660616 >> 172.21.15.29:0/1175660616 conn(0x7f03680cb410 msgr2=0x7f03680cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 -- 172.21.15.29:0/1175660616 shutdown_connections 2023-10-16T15:34:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.846+0000 7f036e3d0640 1 -- 172.21.15.29:0/1175660616 wait complete. 2023-10-16T15:34:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.848+0000 7f036e3d0640 1 Processor -- start 2023-10-16T15:34:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.848+0000 7f036e3d0640 1 -- start start 2023-10-16T15:34:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f036e3d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 0x7f03681f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f036e3d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f036815aaa0 0x7f03680c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f036e3d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f036815ae80 0x7f03680ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f036e3d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680c6630 con 0x7f03680ccaa0 2023-10-16T15:34:30.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f036e3d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680c6910 con 0x7f036815aaa0 2023-10-16T15:34:30.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f036e3d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680c6bf0 con 0x7f036815ae80 2023-10-16T15:34:30.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f0367fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 0x7f03681f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f0367fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 0x7f03681f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33424/0 (socket says 172.21.15.29:33424) 2023-10-16T15:34:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.849+0000 7f0367fff640 1 -- 172.21.15.29:0/1550101058 learned_addr learned my addr 172.21.15.29:0/1550101058 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f03677fe640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f036815aaa0 0x7f03680c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f036c96d640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f036815ae80 0x7f03680ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f0367fff640 1 -- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f036815ae80 msgr2=0x7f03680ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f0367fff640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f036815ae80 0x7f03680ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f0367fff640 1 -- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f036815aaa0 msgr2=0x7f03680c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f0367fff640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f036815aaa0 0x7f03680c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f0367fff640 1 -- 172.21.15.29:0/1550101058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03500094d0 con 0x7f03680ccaa0 2023-10-16T15:34:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f036c96d640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f036815ae80 0x7f03680ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.850+0000 7f0367fff640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 0x7f03681f2840 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0358007ef0 tx=0x7f035800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.851+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358014620 con 0x7f03680ccaa0 2023-10-16T15:34:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.851+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0358014780 con 0x7f03680ccaa0 2023-10-16T15:34:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.851+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03680c6ed0 con 0x7f03680ccaa0 2023-10-16T15:34:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.851+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358014b30 con 0x7f03680ccaa0 2023-10-16T15:34:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.851+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0368156250 con 0x7f03680ccaa0 2023-10-16T15:34:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.852+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03680c7110 con 0x7f03680ccaa0 2023-10-16T15:34:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.853+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0358014c90 con 0x7f03680ccaa0 2023-10-16T15:34:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.854+0000 7f0364ff9640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f034806af90 0x7f034806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.854+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f035808a250 con 0x7f03680ccaa0 2023-10-16T15:34:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.854+0000 7f03677fe640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f034806af90 0x7f034806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.856+0000 7f03677fe640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f034806af90 0x7f034806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0350036000 tx=0x7f035003c040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:30.858+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03680c7110 con 0x7f03680ccaa0 2023-10-16T15:34:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.052+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 --> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f036815f550 con 0x7f034806af90 2023-10-16T15:34:31.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.082+0000 7f0364ff9640 1 -- 172.21.15.29:0/1550101058 <== mgr.7085 v2:172.21.15.170:6832/3694096961 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286822 (secure 0 0 0) 0x7f0350031c40 con 0x7f034806af90 2023-10-16T15:34:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:31.089 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:34:31.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f034806af90 msgr2=0x7f034806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f034806af90 0x7f034806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0350036000 tx=0x7f035003c040 comp rx=0 tx=0).stop 2023-10-16T15:34:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 msgr2=0x7f03681f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 0x7f03681f2840 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0358007ef0 tx=0x7f035800e4b0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 shutdown_connections 2023-10-16T15:34:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f034806af90 0x7f034806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f036815ae80 0x7f03680ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f036815aaa0 0x7f03680c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 --2- 172.21.15.29:0/1550101058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03680ccaa0 0x7f03681f2840 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 >> 172.21.15.29:0/1550101058 conn(0x7f03680cb410 msgr2=0x7f03680b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:31.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 shutdown_connections 2023-10-16T15:34:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.089+0000 7f036e3d0640 1 -- 172.21.15.29:0/1550101058 wait complete. 2023-10-16T15:34:31.113 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T15:34:29.887441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":527,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":334,"ondisk_log_size":334,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":415160,"kb_used_data":19976,"kb_used_omap":12,"kb_used_meta":395059,"kb_avail":754559560,"statfs":{"total":773094113280,"available":772668989440,"internally_reserved":0,"allocated":20455424,"data_stored":10545436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":404540985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.348808"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580494+0000","last_change":"2023-10-16T15:34:26.617126+0000","last_active":"2023-10-16T15:34:27.580494+0000","last_peered":"2023-10-16T15:34:27.580494+0000","last_clean":"2023-10-16T15:34:27.580494+0000","last_became_active":"2023-10-16T15:34:26.616712+0000","last_became_peered":"2023-10-16T15:34:26.616712+0000","last_unstale":"2023-10-16T15:34:27.580494+0000","last_undegraded":"2023-10-16T15:34:27.580494+0000","last_fullsized":"2023-10-16T15:34:27.580494+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:24.179525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786121+0000","last_change":"2023-10-16T15:34:26.620617+0000","last_active":"2023-10-16T15:34:27.786121+0000","last_peered":"2023-10-16T15:34:27.786121+0000","last_clean":"2023-10-16T15:34:27.786121+0000","last_became_active":"2023-10-16T15:34:26.619652+0000","last_became_peered":"2023-10-16T15:34:26.619652+0000","last_unstale":"2023-10-16T15:34:27.786121+0000","last_undegraded":"2023-10-16T15:34:27.786121+0000","last_fullsized":"2023-10-16T15:34:27.786121+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:36.340635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787849+0000","last_change":"2023-10-16T15:34:26.611209+0000","last_active":"2023-10-16T15:34:27.787849+0000","last_peered":"2023-10-16T15:34:27.787849+0000","last_clean":"2023-10-16T15:34:27.787849+0000","last_became_active":"2023-10-16T15:34:26.610937+0000","last_became_peered":"2023-10-16T15:34:26.610937+0000","last_unstale":"2023-10-16T15:34:27.787849+0000","last_undegraded":"2023-10-16T15:34:27.787849+0000","last_fullsized":"2023-10-16T15:34:27.787849+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:08.014086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787118+0000","last_change":"2023-10-16T15:34:26.594443+0000","last_active":"2023-10-16T15:34:27.787118+0000","last_peered":"2023-10-16T15:34:27.787118+0000","last_clean":"2023-10-16T15:34:27.787118+0000","last_became_active":"2023-10-16T15:34:26.593172+0000","last_became_peered":"2023-10-16T15:34:26.593172+0000","last_unstale":"2023-10-16T15:34:27.787118+0000","last_undegraded":"2023-10-16T15:34:27.787118+0000","last_fullsized":"2023-10-16T15:34:27.787118+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:11.805397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785471+0000","last_change":"2023-10-16T15:34:26.606917+0000","last_active":"2023-10-16T15:34:27.785471+0000","last_peered":"2023-10-16T15:34:27.785471+0000","last_clean":"2023-10-16T15:34:27.785471+0000","last_became_active":"2023-10-16T15:34:26.606629+0000","last_became_peered":"2023-10-16T15:34:26.606629+0000","last_unstale":"2023-10-16T15:34:27.785471+0000","last_undegraded":"2023-10-16T15:34:27.785471+0000","last_fullsized":"2023-10-16T15:34:27.785471+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:14.043314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786225+0000","last_change":"2023-10-16T15:34:26.596158+0000","last_active":"2023-10-16T15:34:27.786225+0000","last_peered":"2023-10-16T15:34:27.786225+0000","last_clean":"2023-10-16T15:34:27.786225+0000","last_became_active":"2023-10-16T15:34:26.595918+0000","last_became_peered":"2023-10-16T15:34:26.595918+0000","last_unstale":"2023-10-16T15:34:27.786225+0000","last_undegraded":"2023-10-16T15:34:27.786225+0000","last_fullsized":"2023-10-16T15:34:27.786225+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:33.630041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576144+0000","last_change":"2023-10-16T15:34:26.595098+0000","last_active":"2023-10-16T15:34:27.576144+0000","last_peered":"2023-10-16T15:34:27.576144+0000","last_clean":"2023-10-16T15:34:27.576144+0000","last_became_active":"2023-10-16T15:34:26.594512+0000","last_became_peered":"2023-10-16T15:34:26.594512+0000","last_unstale":"2023-10-16T15:34:27.576144+0000","last_undegraded":"2023-10-16T15:34:27.576144+0000","last_fullsized":"2023-10-16T15:34:27.576144+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:09.285127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784955+0000","last_change":"2023-10-16T15:34:26.615957+0000","last_active":"2023-10-16T15:34:27.784955+0000","last_peered":"2023-10-16T15:34:27.784955+0000","last_clean":"2023-10-16T15:34:27.784955+0000","last_became_active":"2023-10-16T15:34:26.611184+0000","last_became_peered":"2023-10-16T15:34:26.611184+0000","last_unstale":"2023-10-16T15:34:27.784955+0000","last_undegraded":"2023-10-16T15:34:27.784955+0000","last_fullsized":"2023-10-16T15:34:27.784955+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:46.606847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787107+0000","last_change":"2023-10-16T15:34:26.612683+0000","last_active":"2023-10-16T15:34:27.787107+0000","last_peered":"2023-10-16T15:34:27.787107+0000","last_clean":"2023-10-16T15:34:27.787107+0000","last_became_active":"2023-10-16T15:34:26.612408+0000","last_became_peered":"2023-10-16T15:34:26.612408+0000","last_unstale":"2023-10-16T15:34:27.787107+0000","last_undegraded":"2023-10-16T15:34:27.787107+0000","last_fullsized":"2023-10-16T15:34:27.787107+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:09.072515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786570+0000","last_change":"2023-10-16T15:34:26.609261+0000","last_active":"2023-10-16T15:34:27.786570+0000","last_peered":"2023-10-16T15:34:27.786570+0000","last_clean":"2023-10-16T15:34:27.786570+0000","last_became_active":"2023-10-16T15:34:26.608923+0000","last_became_peered":"2023-10-16T15:34:26.608923+0000","last_unstale":"2023-10-16T15:34:27.786570+0000","last_undegraded":"2023-10-16T15:34:27.786570+0000","last_fullsized":"2023-10-16T15:34:27.786570+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:15:00.216820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785969+0000","last_change":"2023-10-16T15:34:26.611404+0000","last_active":"2023-10-16T15:34:27.785969+0000","last_peered":"2023-10-16T15:34:27.785969+0000","last_clean":"2023-10-16T15:34:27.785969+0000","last_became_active":"2023-10-16T15:34:26.611131+0000","last_became_peered":"2023-10-16T15:34:26.611131+0000","last_unstale":"2023-10-16T15:34:27.785969+0000","last_undegraded":"2023-10-16T15:34:27.785969+0000","last_fullsized":"2023-10-16T15:34:27.785969+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:55.349752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785999+0000","last_change":"2023-10-16T15:34:26.612640+0000","last_active":"2023-10-16T15:34:27.785999+0000","last_peered":"2023-10-16T15:34:27.785999+0000","last_clean":"2023-10-16T15:34:27.785999+0000","last_became_active":"2023-10-16T15:34:26.612425+0000","last_became_peered":"2023-10-16T15:34:26.612425+0000","last_unstale":"2023-10-16T15:34:27.785999+0000","last_undegraded":"2023-10-16T15:34:27.785999+0000","last_fullsized":"2023-10-16T15:34:27.785999+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:07.800060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787607+0000","last_change":"2023-10-16T15:34:26.610895+0000","last_active":"2023-10-16T15:34:27.787607+0000","last_peered":"2023-10-16T15:34:27.787607+0000","last_clean":"2023-10-16T15:34:27.787607+0000","last_became_active":"2023-10-16T15:34:26.610567+0000","last_became_peered":"2023-10-16T15:34:26.610567+0000","last_unstale":"2023-10-16T15:34:27.787607+0000","last_undegraded":"2023-10-16T15:34:27.787607+0000","last_fullsized":"2023-10-16T15:34:27.787607+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:34.164591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786828+0000","last_change":"2023-10-16T15:34:26.616427+0000","last_active":"2023-10-16T15:34:27.786828+0000","last_peered":"2023-10-16T15:34:27.786828+0000","last_clean":"2023-10-16T15:34:27.786828+0000","last_became_active":"2023-10-16T15:34:26.614700+0000","last_became_peered":"2023-10-16T15:34:26.614700+0000","last_unstale":"2023-10-16T15:34:27.786828+0000","last_undegraded":"2023-10-16T15:34:27.786828+0000","last_fullsized":"2023-10-16T15:34:27.786828+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.239014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.578699+0000","last_change":"2023-10-16T15:34:26.597864+0000","last_active":"2023-10-16T15:34:27.578699+0000","last_peered":"2023-10-16T15:34:27.578699+0000","last_clean":"2023-10-16T15:34:27.578699+0000","last_became_active":"2023-10-16T15:34:26.597586+0000","last_became_peered":"2023-10-16T15:34:26.597586+0000","last_unstale":"2023-10-16T15:34:27.578699+0000","last_undegraded":"2023-10-16T15:34:27.578699+0000","last_fullsized":"2023-10-16T15:34:27.578699+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:14.612765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.575772+0000","last_change":"2023-10-16T15:34:26.587033+0000","last_active":"2023-10-16T15:34:27.575772+0000","last_peered":"2023-10-16T15:34:27.575772+0000","last_clean":"2023-10-16T15:34:27.575772+0000","last_became_active":"2023-10-16T15:34:26.586560+0000","last_became_peered":"2023-10-16T15:34:26.586560+0000","last_unstale":"2023-10-16T15:34:27.575772+0000","last_undegraded":"2023-10-16T15:34:27.575772+0000","last_fullsized":"2023-10-16T15:34:27.575772+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:59.385000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787621+0000","last_change":"2023-10-16T15:34:26.618489+0000","last_active":"2023-10-16T15:34:27.787621+0000","last_peered":"2023-10-16T15:34:27.787621+0000","last_clean":"2023-10-16T15:34:27.787621+0000","last_became_active":"2023-10-16T15:34:26.618127+0000","last_became_peered":"2023-10-16T15:34:26.618127+0000","last_unstale":"2023-10-16T15:34:27.787621+0000","last_undegraded":"2023-10-16T15:34:27.787621+0000","last_fullsized":"2023-10-16T15:34:27.787621+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:07.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784691+0000","last_change":"2023-10-16T15:34:26.605198+0000","last_active":"2023-10-16T15:34:27.784691+0000","last_peered":"2023-10-16T15:34:27.784691+0000","last_clean":"2023-10-16T15:34:27.784691+0000","last_became_active":"2023-10-16T15:34:26.604735+0000","last_became_peered":"2023-10-16T15:34:26.604735+0000","last_unstale":"2023-10-16T15:34:27.784691+0000","last_undegraded":"2023-10-16T15:34:27.784691+0000","last_fullsized":"2023-10-16T15:34:27.784691+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:04.516745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786927+0000","last_change":"2023-10-16T15:34:26.607556+0000","last_active":"2023-10-16T15:34:27.786927+0000","last_peered":"2023-10-16T15:34:27.786927+0000","last_clean":"2023-10-16T15:34:27.786927+0000","last_became_active":"2023-10-16T15:34:26.607232+0000","last_became_peered":"2023-10-16T15:34:26.607232+0000","last_unstale":"2023-10-16T15:34:27.786927+0000","last_undegraded":"2023-10-16T15:34:27.786927+0000","last_fullsized":"2023-10-16T15:34:27.786927+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:46.070157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787824+0000","last_change":"2023-10-16T15:34:26.620697+0000","last_active":"2023-10-16T15:34:27.787824+0000","last_peered":"2023-10-16T15:34:27.787824+0000","last_clean":"2023-10-16T15:34:27.787824+0000","last_became_active":"2023-10-16T15:34:26.619362+0000","last_became_peered":"2023-10-16T15:34:26.619362+0000","last_unstale":"2023-10-16T15:34:27.787824+0000","last_undegraded":"2023-10-16T15:34:27.787824+0000","last_fullsized":"2023-10-16T15:34:27.787824+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:36.566128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786090+0000","last_change":"2023-10-16T15:34:26.612004+0000","last_active":"2023-10-16T15:34:27.786090+0000","last_peered":"2023-10-16T15:34:27.786090+0000","last_clean":"2023-10-16T15:34:27.786090+0000","last_became_active":"2023-10-16T15:34:26.611713+0000","last_became_peered":"2023-10-16T15:34:26.611713+0000","last_unstale":"2023-10-16T15:34:27.786090+0000","last_undegraded":"2023-10-16T15:34:27.786090+0000","last_fullsized":"2023-10-16T15:34:27.786090+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:48:10.065535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785495+0000","last_change":"2023-10-16T15:34:26.619978+0000","last_active":"2023-10-16T15:34:27.785495+0000","last_peered":"2023-10-16T15:34:27.785495+0000","last_clean":"2023-10-16T15:34:27.785495+0000","last_became_active":"2023-10-16T15:34:26.618384+0000","last_became_peered":"2023-10-16T15:34:26.618384+0000","last_unstale":"2023-10-16T15:34:27.785495+0000","last_undegraded":"2023-10-16T15:34:27.785495+0000","last_fullsized":"2023-10-16T15:34:27.785495+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:24.777311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785832+0000","last_change":"2023-10-16T15:34:26.605438+0000","last_active":"2023-10-16T15:34:27.785832+0000","last_peered":"2023-10-16T15:34:27.785832+0000","last_clean":"2023-10-16T15:34:27.785832+0000","last_became_active":"2023-10-16T15:34:26.605108+0000","last_became_peered":"2023-10-16T15:34:26.605108+0000","last_unstale":"2023-10-16T15:34:27.785832+0000","last_undegraded":"2023-10-16T15:34:27.785832+0000","last_fullsized":"2023-10-16T15:34:27.785832+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:58.646193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785138+0000","last_change":"2023-10-16T15:34:26.619182+0000","last_active":"2023-10-16T15:34:27.785138+0000","last_peered":"2023-10-16T15:34:27.785138+0000","last_clean":"2023-10-16T15:34:27.785138+0000","last_became_active":"2023-10-16T15:34:26.618805+0000","last_became_peered":"2023-10-16T15:34:26.618805+0000","last_unstale":"2023-10-16T15:34:27.785138+0000","last_undegraded":"2023-10-16T15:34:27.785138+0000","last_fullsized":"2023-10-16T15:34:27.785138+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:46.045117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788353+0000","last_change":"2023-10-16T15:34:26.617033+0000","last_active":"2023-10-16T15:34:27.788353+0000","last_peered":"2023-10-16T15:34:27.788353+0000","last_clean":"2023-10-16T15:34:27.788353+0000","last_became_active":"2023-10-16T15:34:26.611442+0000","last_became_peered":"2023-10-16T15:34:26.611442+0000","last_unstale":"2023-10-16T15:34:27.788353+0000","last_undegraded":"2023-10-16T15:34:27.788353+0000","last_fullsized":"2023-10-16T15:34:27.788353+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:21:41.256837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576653+0000","last_change":"2023-10-16T15:34:26.591545+0000","last_active":"2023-10-16T15:34:27.576653+0000","last_peered":"2023-10-16T15:34:27.576653+0000","last_clean":"2023-10-16T15:34:27.576653+0000","last_became_active":"2023-10-16T15:34:26.591092+0000","last_became_peered":"2023-10-16T15:34:26.591092+0000","last_unstale":"2023-10-16T15:34:27.576653+0000","last_undegraded":"2023-10-16T15:34:27.576653+0000","last_fullsized":"2023-10-16T15:34:27.576653+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:21.058793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787957+0000","last_change":"2023-10-16T15:34:26.617274+0000","last_active":"2023-10-16T15:34:27.787957+0000","last_peered":"2023-10-16T15:34:27.787957+0000","last_clean":"2023-10-16T15:34:27.787957+0000","last_became_active":"2023-10-16T15:34:26.616426+0000","last_became_peered":"2023-10-16T15:34:26.616426+0000","last_unstale":"2023-10-16T15:34:27.787957+0000","last_undegraded":"2023-10-16T15:34:27.787957+0000","last_fullsized":"2023-10-16T15:34:27.787957+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:01.713175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787259+0000","last_change":"2023-10-16T15:34:26.604541+0000","last_active":"2023-10-16T15:34:27.787259+0000","last_peered":"2023-10-16T15:34:27.787259+0000","last_clean":"2023-10-16T15:34:27.787259+0000","last_became_active":"2023-10-16T15:34:26.604284+0000","last_became_peered":"2023-10-16T15:34:26.604284+0000","last_unstale":"2023-10-16T15:34:27.787259+0000","last_undegraded":"2023-10-16T15:34:27.787259+0000","last_fullsized":"2023-10-16T15:34:27.787259+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:55.827479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787645+0000","last_change":"2023-10-16T15:34:26.619978+0000","last_active":"2023-10-16T15:34:27.787645+0000","last_peered":"2023-10-16T15:34:27.787645+0000","last_clean":"2023-10-16T15:34:27.787645+0000","last_became_active":"2023-10-16T15:34:26.615076+0000","last_became_peered":"2023-10-16T15:34:26.615076+0000","last_unstale":"2023-10-16T15:34:27.787645+0000","last_undegraded":"2023-10-16T15:34:27.787645+0000","last_fullsized":"2023-10-16T15:34:27.787645+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:49.301073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581132+0000","last_change":"2023-10-16T15:34:26.588117+0000","last_active":"2023-10-16T15:34:27.581132+0000","last_peered":"2023-10-16T15:34:27.581132+0000","last_clean":"2023-10-16T15:34:27.581132+0000","last_became_active":"2023-10-16T15:34:26.587546+0000","last_became_peered":"2023-10-16T15:34:26.587546+0000","last_unstale":"2023-10-16T15:34:27.581132+0000","last_undegraded":"2023-10-16T15:34:27.581132+0000","last_fullsized":"2023-10-16T15:34:27.581132+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:24:15.426000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581265+0000","last_change":"2023-10-16T15:34:26.604743+0000","last_active":"2023-10-16T15:34:27.581265+0000","last_peered":"2023-10-16T15:34:27.581265+0000","last_clean":"2023-10-16T15:34:27.581265+0000","last_became_active":"2023-10-16T15:34:26.604464+0000","last_became_peered":"2023-10-16T15:34:26.604464+0000","last_unstale":"2023-10-16T15:34:27.581265+0000","last_undegraded":"2023-10-16T15:34:27.581265+0000","last_fullsized":"2023-10-16T15:34:27.581265+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.456565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.783425+0000","last_change":"2023-10-16T15:34:26.605397+0000","last_active":"2023-10-16T15:34:27.783425+0000","last_peered":"2023-10-16T15:34:27.783425+0000","last_clean":"2023-10-16T15:34:27.783425+0000","last_became_active":"2023-10-16T15:34:26.605129+0000","last_became_peered":"2023-10-16T15:34:26.605129+0000","last_unstale":"2023-10-16T15:34:27.783425+0000","last_undegraded":"2023-10-16T15:34:27.783425+0000","last_fullsized":"2023-10-16T15:34:27.783425+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:13.909012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576782+0000","last_change":"2023-10-16T15:34:26.591362+0000","last_active":"2023-10-16T15:34:27.576782+0000","last_peered":"2023-10-16T15:34:27.576782+0000","last_clean":"2023-10-16T15:34:27.576782+0000","last_became_active":"2023-10-16T15:34:26.590704+0000","last_became_peered":"2023-10-16T15:34:26.590704+0000","last_unstale":"2023-10-16T15:34:27.576782+0000","last_undegraded":"2023-10-16T15:34:27.576782+0000","last_fullsized":"2023-10-16T15:34:27.576782+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:34:42.924882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"191'4","reported_seq":16,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580739+0000","last_change":"2023-10-16T15:34:25.576361+0000","last_active":"2023-10-16T15:34:27.580739+0000","last_peered":"2023-10-16T15:34:27.580739+0000","last_clean":"2023-10-16T15:34:27.580739+0000","last_became_active":"2023-10-16T15:34:25.576037+0000","last_became_peered":"2023-10-16T15:34:25.576037+0000","last_unstale":"2023-10-16T15:34:27.580739+0000","last_undegraded":"2023-10-16T15:34:27.580739+0000","last_fullsized":"2023-10-16T15:34:27.580739+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:03:24.352960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582403+0000","last_change":"2023-10-16T15:34:26.596538+0000","last_active":"2023-10-16T15:34:27.582403+0000","last_peered":"2023-10-16T15:34:27.582403+0000","last_clean":"2023-10-16T15:34:27.582403+0000","last_became_active":"2023-10-16T15:34:26.593505+0000","last_became_peered":"2023-10-16T15:34:26.593505+0000","last_unstale":"2023-10-16T15:34:27.582403+0000","last_undegraded":"2023-10-16T15:34:27.582403+0000","last_fullsized":"2023-10-16T15:34:27.582403+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:46.935125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582038+0000","last_change":"2023-10-16T15:34:25.581803+0000","last_active":"2023-10-16T15:34:27.582038+0000","last_peered":"2023-10-16T15:34:27.582038+0000","last_clean":"2023-10-16T15:34:27.582038+0000","last_became_active":"2023-10-16T15:34:25.581390+0000","last_became_peered":"2023-10-16T15:34:25.581390+0000","last_unstale":"2023-10-16T15:34:27.582038+0000","last_undegraded":"2023-10-16T15:34:27.582038+0000","last_fullsized":"2023-10-16T15:34:27.582038+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:06.834486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787224+0000","last_change":"2023-10-16T15:34:26.596019+0000","last_active":"2023-10-16T15:34:27.787224+0000","last_peered":"2023-10-16T15:34:27.787224+0000","last_clean":"2023-10-16T15:34:27.787224+0000","last_became_active":"2023-10-16T15:34:26.595724+0000","last_became_peered":"2023-10-16T15:34:26.595724+0000","last_unstale":"2023-10-16T15:34:27.787224+0000","last_undegraded":"2023-10-16T15:34:27.787224+0000","last_fullsized":"2023-10-16T15:34:27.787224+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:52.166652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786299+0000","last_change":"2023-10-16T15:34:25.581693+0000","last_active":"2023-10-16T15:34:27.786299+0000","last_peered":"2023-10-16T15:34:27.786299+0000","last_clean":"2023-10-16T15:34:27.786299+0000","last_became_active":"2023-10-16T15:34:25.580966+0000","last_became_peered":"2023-10-16T15:34:25.580966+0000","last_unstale":"2023-10-16T15:34:27.786299+0000","last_undegraded":"2023-10-16T15:34:27.786299+0000","last_fullsized":"2023-10-16T15:34:27.786299+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:16.172910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788169+0000","last_change":"2023-10-16T15:34:26.611117+0000","last_active":"2023-10-16T15:34:27.788169+0000","last_peered":"2023-10-16T15:34:27.788169+0000","last_clean":"2023-10-16T15:34:27.788169+0000","last_became_active":"2023-10-16T15:34:26.610674+0000","last_became_peered":"2023-10-16T15:34:26.610674+0000","last_unstale":"2023-10-16T15:34:27.788169+0000","last_undegraded":"2023-10-16T15:34:27.788169+0000","last_fullsized":"2023-10-16T15:34:27.788169+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:20.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787470+0000","last_change":"2023-10-16T15:34:25.575595+0000","last_active":"2023-10-16T15:34:27.787470+0000","last_peered":"2023-10-16T15:34:27.787470+0000","last_clean":"2023-10-16T15:34:27.787470+0000","last_became_active":"2023-10-16T15:34:25.574665+0000","last_became_peered":"2023-10-16T15:34:25.574665+0000","last_unstale":"2023-10-16T15:34:27.787470+0000","last_undegraded":"2023-10-16T15:34:27.787470+0000","last_fullsized":"2023-10-16T15:34:27.787470+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:43.630060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576313+0000","last_change":"2023-10-16T15:34:26.586892+0000","last_active":"2023-10-16T15:34:27.576313+0000","last_peered":"2023-10-16T15:34:27.576313+0000","last_clean":"2023-10-16T15:34:27.576313+0000","last_became_active":"2023-10-16T15:34:26.586542+0000","last_became_peered":"2023-10-16T15:34:26.586542+0000","last_unstale":"2023-10-16T15:34:27.576313+0000","last_undegraded":"2023-10-16T15:34:27.576313+0000","last_fullsized":"2023-10-16T15:34:27.576313+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:58.778256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787868+0000","last_change":"2023-10-16T15:34:25.577870+0000","last_active":"2023-10-16T15:34:27.787868+0000","last_peered":"2023-10-16T15:34:27.787868+0000","last_clean":"2023-10-16T15:34:27.787868+0000","last_became_active":"2023-10-16T15:34:25.577585+0000","last_became_peered":"2023-10-16T15:34:25.577585+0000","last_unstale":"2023-10-16T15:34:27.787868+0000","last_undegraded":"2023-10-16T15:34:27.787868+0000","last_fullsized":"2023-10-16T15:34:27.787868+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:15.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786110+0000","last_change":"2023-10-16T15:34:26.599145+0000","last_active":"2023-10-16T15:34:27.786110+0000","last_peered":"2023-10-16T15:34:27.786110+0000","last_clean":"2023-10-16T15:34:27.786110+0000","last_became_active":"2023-10-16T15:34:26.598654+0000","last_became_peered":"2023-10-16T15:34:26.598654+0000","last_unstale":"2023-10-16T15:34:27.786110+0000","last_undegraded":"2023-10-16T15:34:27.786110+0000","last_fullsized":"2023-10-16T15:34:27.786110+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:07.850011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786487+0000","last_change":"2023-10-16T15:34:25.580915+0000","last_active":"2023-10-16T15:34:27.786487+0000","last_peered":"2023-10-16T15:34:27.786487+0000","last_clean":"2023-10-16T15:34:27.786487+0000","last_became_active":"2023-10-16T15:34:25.580623+0000","last_became_peered":"2023-10-16T15:34:25.580623+0000","last_unstale":"2023-10-16T15:34:27.786487+0000","last_undegraded":"2023-10-16T15:34:27.786487+0000","last_fullsized":"2023-10-16T15:34:27.786487+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:11.660719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785163+0000","last_change":"2023-10-16T15:34:26.605735+0000","last_active":"2023-10-16T15:34:27.785163+0000","last_peered":"2023-10-16T15:34:27.785163+0000","last_clean":"2023-10-16T15:34:27.785163+0000","last_became_active":"2023-10-16T15:34:26.605465+0000","last_became_peered":"2023-10-16T15:34:26.605465+0000","last_unstale":"2023-10-16T15:34:27.785163+0000","last_undegraded":"2023-10-16T15:34:27.785163+0000","last_fullsized":"2023-10-16T15:34:27.785163+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:28.182594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785208+0000","last_change":"2023-10-16T15:34:25.581000+0000","last_active":"2023-10-16T15:34:27.785208+0000","last_peered":"2023-10-16T15:34:27.785208+0000","last_clean":"2023-10-16T15:34:27.785208+0000","last_became_active":"2023-10-16T15:34:25.580661+0000","last_became_peered":"2023-10-16T15:34:25.580661+0000","last_unstale":"2023-10-16T15:34:27.785208+0000","last_undegraded":"2023-10-16T15:34:27.785208+0000","last_fullsized":"2023-10-16T15:34:27.785208+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:00.524622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786111+0000","last_change":"2023-10-16T15:34:26.596019+0000","last_active":"2023-10-16T15:34:27.786111+0000","last_peered":"2023-10-16T15:34:27.786111+0000","last_clean":"2023-10-16T15:34:27.786111+0000","last_became_active":"2023-10-16T15:34:26.595707+0000","last_became_peered":"2023-10-16T15:34:26.595707+0000","last_unstale":"2023-10-16T15:34:27.786111+0000","last_undegraded":"2023-10-16T15:34:27.786111+0000","last_fullsized":"2023-10-16T15:34:27.786111+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:38.684237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.575989+0000","last_change":"2023-10-16T15:34:25.575196+0000","last_active":"2023-10-16T15:34:27.575989+0000","last_peered":"2023-10-16T15:34:27.575989+0000","last_clean":"2023-10-16T15:34:27.575989+0000","last_became_active":"2023-10-16T15:34:25.574926+0000","last_became_peered":"2023-10-16T15:34:25.574926+0000","last_unstale":"2023-10-16T15:34:27.575989+0000","last_undegraded":"2023-10-16T15:34:27.575989+0000","last_fullsized":"2023-10-16T15:34:27.575989+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:08.627856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784746+0000","last_change":"2023-10-16T15:34:26.608119+0000","last_active":"2023-10-16T15:34:27.784746+0000","last_peered":"2023-10-16T15:34:27.784746+0000","last_clean":"2023-10-16T15:34:27.784746+0000","last_became_active":"2023-10-16T15:34:26.607830+0000","last_became_peered":"2023-10-16T15:34:26.607830+0000","last_unstale":"2023-10-16T15:34:27.784746+0000","last_undegraded":"2023-10-16T15:34:27.784746+0000","last_fullsized":"2023-10-16T15:34:27.784746+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:54.493055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787685+0000","last_change":"2023-10-16T15:34:25.577417+0000","last_active":"2023-10-16T15:34:27.787685+0000","last_peered":"2023-10-16T15:34:27.787685+0000","last_clean":"2023-10-16T15:34:27.787685+0000","last_became_active":"2023-10-16T15:34:25.575680+0000","last_became_peered":"2023-10-16T15:34:25.575680+0000","last_unstale":"2023-10-16T15:34:27.787685+0000","last_undegraded":"2023-10-16T15:34:27.787685+0000","last_fullsized":"2023-10-16T15:34:27.787685+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:14.823512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784272+0000","last_change":"2023-10-16T15:34:26.597698+0000","last_active":"2023-10-16T15:34:27.784272+0000","last_peered":"2023-10-16T15:34:27.784272+0000","last_clean":"2023-10-16T15:34:27.784272+0000","last_became_active":"2023-10-16T15:34:26.597341+0000","last_became_peered":"2023-10-16T15:34:26.597341+0000","last_unstale":"2023-10-16T15:34:27.784272+0000","last_undegraded":"2023-10-16T15:34:27.784272+0000","last_fullsized":"2023-10-16T15:34:27.784272+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:45.778251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"191'3","reported_seq":15,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787045+0000","last_change":"2023-10-16T15:34:25.574979+0000","last_active":"2023-10-16T15:34:27.787045+0000","last_peered":"2023-10-16T15:34:27.787045+0000","last_clean":"2023-10-16T15:34:27.787045+0000","last_became_active":"2023-10-16T15:34:25.574471+0000","last_became_peered":"2023-10-16T15:34:25.574471+0000","last_unstale":"2023-10-16T15:34:27.787045+0000","last_undegraded":"2023-10-16T15:34:27.787045+0000","last_fullsized":"2023-10-16T15:34:27.787045+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:18.710219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582232+0000","last_change":"2023-10-16T15:34:26.613954+0000","last_active":"2023-10-16T15:34:27.582232+0000","last_peered":"2023-10-16T15:34:27.582232+0000","last_clean":"2023-10-16T15:34:27.582232+0000","last_became_active":"2023-10-16T15:34:26.612933+0000","last_became_peered":"2023-10-16T15:34:26.612933+0000","last_unstale":"2023-10-16T15:34:27.582232+0000","last_undegraded":"2023-10-16T15:34:27.582232+0000","last_fullsized":"2023-10-16T15:34:27.582232+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:26.456602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"191'6","reported_seq":18,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581710+0000","last_change":"2023-10-16T15:34:25.573020+0000","last_active":"2023-10-16T15:34:27.581710+0000","last_peered":"2023-10-16T15:34:27.581710+0000","last_clean":"2023-10-16T15:34:27.581710+0000","last_became_active":"2023-10-16T15:34:25.572716+0000","last_became_peered":"2023-10-16T15:34:25.572716+0000","last_unstale":"2023-10-16T15:34:27.581710+0000","last_undegraded":"2023-10-16T15:34:27.581710+0000","last_fullsized":"2023-10-16T15:34:27.581710+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:10.789872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784216+0000","last_change":"2023-10-16T15:34:26.603626+0000","last_active":"2023-10-16T15:34:27.784216+0000","last_peered":"2023-10-16T15:34:27.784216+0000","last_clean":"2023-10-16T15:34:27.784216+0000","last_became_active":"2023-10-16T15:34:26.603327+0000","last_became_peered":"2023-10-16T15:34:26.603327+0000","last_unstale":"2023-10-16T15:34:27.784216+0000","last_undegraded":"2023-10-16T15:34:27.784216+0000","last_fullsized":"2023-10-16T15:34:27.784216+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:27.364529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786356+0000","last_change":"2023-10-16T15:34:25.583201+0000","last_active":"2023-10-16T15:34:27.786356+0000","last_peered":"2023-10-16T15:34:27.786356+0000","last_clean":"2023-10-16T15:34:27.786356+0000","last_became_active":"2023-10-16T15:34:25.582902+0000","last_became_peered":"2023-10-16T15:34:25.582902+0000","last_unstale":"2023-10-16T15:34:27.786356+0000","last_undegraded":"2023-10-16T15:34:27.786356+0000","last_fullsized":"2023-10-16T15:34:27.786356+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:05.367800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576866+0000","last_change":"2023-10-16T15:34:26.614735+0000","last_active":"2023-10-16T15:34:27.576866+0000","last_peered":"2023-10-16T15:34:27.576866+0000","last_clean":"2023-10-16T15:34:27.576866+0000","last_became_active":"2023-10-16T15:34:26.614392+0000","last_became_peered":"2023-10-16T15:34:26.614392+0000","last_unstale":"2023-10-16T15:34:27.576866+0000","last_undegraded":"2023-10-16T15:34:27.576866+0000","last_fullsized":"2023-10-16T15:34:27.576866+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:51.630786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.783946+0000","last_change":"2023-10-16T15:34:25.572332+0000","last_active":"2023-10-16T15:34:27.783946+0000","last_peered":"2023-10-16T15:34:27.783946+0000","last_clean":"2023-10-16T15:34:27.783946+0000","last_became_active":"2023-10-16T15:34:25.572012+0000","last_became_peered":"2023-10-16T15:34:25.572012+0000","last_unstale":"2023-10-16T15:34:27.783946+0000","last_undegraded":"2023-10-16T15:34:27.783946+0000","last_fullsized":"2023-10-16T15:34:27.783946+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:12.957176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785086+0000","last_change":"2023-10-16T15:34:26.605192+0000","last_active":"2023-10-16T15:34:27.785086+0000","last_peered":"2023-10-16T15:34:27.785086+0000","last_clean":"2023-10-16T15:34:27.785086+0000","last_became_active":"2023-10-16T15:34:26.604934+0000","last_became_peered":"2023-10-16T15:34:26.604934+0000","last_unstale":"2023-10-16T15:34:27.785086+0000","last_undegraded":"2023-10-16T15:34:27.785086+0000","last_fullsized":"2023-10-16T15:34:27.785086+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.171786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785450+0000","last_change":"2023-10-16T15:34:25.582542+0000","last_active":"2023-10-16T15:34:27.785450+0000","last_peered":"2023-10-16T15:34:27.785450+0000","last_clean":"2023-10-16T15:34:27.785450+0000","last_became_active":"2023-10-16T15:34:25.581999+0000","last_became_peered":"2023-10-16T15:34:25.581999+0000","last_unstale":"2023-10-16T15:34:27.785450+0000","last_undegraded":"2023-10-16T15:34:27.785450+0000","last_fullsized":"2023-10-16T15:34:27.785450+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786878+0000","last_change":"2023-10-16T15:34:26.617167+0000","last_active":"2023-10-16T15:34:27.786878+0000","last_peered":"2023-10-16T15:34:27.786878+0000","last_clean":"2023-10-16T15:34:27.786878+0000","last_became_active":"2023-10-16T15:34:26.611654+0000","last_became_peered":"2023-10-16T15:34:26.611654+0000","last_unstale":"2023-10-16T15:34:27.786878+0000","last_undegraded":"2023-10-16T15:34:27.786878+0000","last_fullsized":"2023-10-16T15:34:27.786878+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:53.180390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"191'8","reported_seq":20,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580659+0000","last_change":"2023-10-16T15:34:25.569283+0000","last_active":"2023-10-16T15:34:27.580659+0000","last_peered":"2023-10-16T15:34:27.580659+0000","last_clean":"2023-10-16T15:34:27.580659+0000","last_became_active":"2023-10-16T15:34:25.568974+0000","last_became_peered":"2023-10-16T15:34:25.568974+0000","last_unstale":"2023-10-16T15:34:27.580659+0000","last_undegraded":"2023-10-16T15:34:27.580659+0000","last_fullsized":"2023-10-16T15:34:27.580659+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:16.156261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787495+0000","last_change":"2023-10-16T15:34:26.600619+0000","last_active":"2023-10-16T15:34:27.787495+0000","last_peered":"2023-10-16T15:34:27.787495+0000","last_clean":"2023-10-16T15:34:27.787495+0000","last_became_active":"2023-10-16T15:34:26.600320+0000","last_became_peered":"2023-10-16T15:34:26.600320+0000","last_unstale":"2023-10-16T15:34:27.787495+0000","last_undegraded":"2023-10-16T15:34:27.787495+0000","last_fullsized":"2023-10-16T15:34:27.787495+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:24:15.426000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788488+0000","last_change":"2023-10-16T15:34:25.582687+0000","last_active":"2023-10-16T15:34:27.788488+0000","last_peered":"2023-10-16T15:34:27.788488+0000","last_clean":"2023-10-16T15:34:27.788488+0000","last_became_active":"2023-10-16T15:34:25.582392+0000","last_became_peered":"2023-10-16T15:34:25.582392+0000","last_unstale":"2023-10-16T15:34:27.788488+0000","last_undegraded":"2023-10-16T15:34:27.788488+0000","last_fullsized":"2023-10-16T15:34:27.788488+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:08.056990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785402+0000","last_change":"2023-10-16T15:34:26.599860+0000","last_active":"2023-10-16T15:34:27.785402+0000","last_peered":"2023-10-16T15:34:27.785402+0000","last_clean":"2023-10-16T15:34:27.785402+0000","last_became_active":"2023-10-16T15:34:26.599375+0000","last_became_peered":"2023-10-16T15:34:26.599375+0000","last_unstale":"2023-10-16T15:34:27.785402+0000","last_undegraded":"2023-10-16T15:34:27.785402+0000","last_fullsized":"2023-10-16T15:34:27.785402+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:58.623926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787787+0000","last_change":"2023-10-16T15:34:25.579171+0000","last_active":"2023-10-16T15:34:27.787787+0000","last_peered":"2023-10-16T15:34:27.787787+0000","last_clean":"2023-10-16T15:34:27.787787+0000","last_became_active":"2023-10-16T15:34:25.578533+0000","last_became_peered":"2023-10-16T15:34:25.578533+0000","last_unstale":"2023-10-16T15:34:27.787787+0000","last_undegraded":"2023-10-16T15:34:27.787787+0000","last_fullsized":"2023-10-16T15:34:27.787787+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:45.029593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787448+0000","last_change":"2023-10-16T15:34:26.598573+0000","last_active":"2023-10-16T15:34:27.787448+0000","last_peered":"2023-10-16T15:34:27.787448+0000","last_clean":"2023-10-16T15:34:27.787448+0000","last_became_active":"2023-10-16T15:34:26.596504+0000","last_became_peered":"2023-10-16T15:34:26.596504+0000","last_unstale":"2023-10-16T15:34:27.787448+0000","last_undegraded":"2023-10-16T15:34:27.787448+0000","last_fullsized":"2023-10-16T15:34:27.787448+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:46.553519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.579413+0000","last_change":"2023-10-16T15:34:25.572405+0000","last_active":"2023-10-16T15:34:27.579413+0000","last_peered":"2023-10-16T15:34:27.579413+0000","last_clean":"2023-10-16T15:34:27.579413+0000","last_became_active":"2023-10-16T15:34:25.572069+0000","last_became_peered":"2023-10-16T15:34:25.572069+0000","last_unstale":"2023-10-16T15:34:27.579413+0000","last_undegraded":"2023-10-16T15:34:27.579413+0000","last_fullsized":"2023-10-16T15:34:27.579413+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:19.823037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576519+0000","last_change":"2023-10-16T15:34:26.593722+0000","last_active":"2023-10-16T15:34:27.576519+0000","last_peered":"2023-10-16T15:34:27.576519+0000","last_clean":"2023-10-16T15:34:27.576519+0000","last_became_active":"2023-10-16T15:34:26.591933+0000","last_became_peered":"2023-10-16T15:34:26.591933+0000","last_unstale":"2023-10-16T15:34:27.576519+0000","last_undegraded":"2023-10-16T15:34:27.576519+0000","last_fullsized":"2023-10-16T15:34:27.576519+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:43.548094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580783+0000","last_change":"2023-10-16T15:34:25.582452+0000","last_active":"2023-10-16T15:34:27.580783+0000","last_peered":"2023-10-16T15:34:27.580783+0000","last_clean":"2023-10-16T15:34:27.580783+0000","last_became_active":"2023-10-16T15:34:25.581439+0000","last_became_peered":"2023-10-16T15:34:25.581439+0000","last_unstale":"2023-10-16T15:34:27.580783+0000","last_undegraded":"2023-10-16T15:34:27.580783+0000","last_fullsized":"2023-10-16T15:34:27.580783+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:23.151449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786909+0000","last_change":"2023-10-16T15:34:26.620200+0000","last_active":"2023-10-16T15:34:27.786909+0000","last_peered":"2023-10-16T15:34:27.786909+0000","last_clean":"2023-10-16T15:34:27.786909+0000","last_became_active":"2023-10-16T15:34:26.619592+0000","last_became_peered":"2023-10-16T15:34:26.619592+0000","last_unstale":"2023-10-16T15:34:27.786909+0000","last_undegraded":"2023-10-16T15:34:27.786909+0000","last_fullsized":"2023-10-16T15:34:27.786909+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:29.156108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786697+0000","last_change":"2023-10-16T15:34:25.579748+0000","last_active":"2023-10-16T15:34:27.786697+0000","last_peered":"2023-10-16T15:34:27.786697+0000","last_clean":"2023-10-16T15:34:27.786697+0000","last_became_active":"2023-10-16T15:34:25.579275+0000","last_became_peered":"2023-10-16T15:34:25.579275+0000","last_unstale":"2023-10-16T15:34:27.786697+0000","last_undegraded":"2023-10-16T15:34:27.786697+0000","last_fullsized":"2023-10-16T15:34:27.786697+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:41.474966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576883+0000","last_change":"2023-10-16T15:34:26.591413+0000","last_active":"2023-10-16T15:34:27.576883+0000","last_peered":"2023-10-16T15:34:27.576883+0000","last_clean":"2023-10-16T15:34:27.576883+0000","last_became_active":"2023-10-16T15:34:26.591155+0000","last_became_peered":"2023-10-16T15:34:26.591155+0000","last_unstale":"2023-10-16T15:34:27.576883+0000","last_undegraded":"2023-10-16T15:34:27.576883+0000","last_fullsized":"2023-10-16T15:34:27.576883+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:05.573570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.578866+0000","last_change":"2023-10-16T15:34:25.575452+0000","last_active":"2023-10-16T15:34:27.578866+0000","last_peered":"2023-10-16T15:34:27.578866+0000","last_clean":"2023-10-16T15:34:27.578866+0000","last_became_active":"2023-10-16T15:34:25.575154+0000","last_became_peered":"2023-10-16T15:34:25.575154+0000","last_unstale":"2023-10-16T15:34:27.578866+0000","last_undegraded":"2023-10-16T15:34:27.578866+0000","last_fullsized":"2023-10-16T15:34:27.578866+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:59.950670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.579100+0000","last_change":"2023-10-16T15:34:26.609750+0000","last_active":"2023-10-16T15:34:27.579100+0000","last_peered":"2023-10-16T15:34:27.579100+0000","last_clean":"2023-10-16T15:34:27.579100+0000","last_became_active":"2023-10-16T15:34:26.607638+0000","last_became_peered":"2023-10-16T15:34:26.607638+0000","last_unstale":"2023-10-16T15:34:27.579100+0000","last_undegraded":"2023-10-16T15:34:27.579100+0000","last_fullsized":"2023-10-16T15:34:27.579100+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:29.454834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785548+0000","last_change":"2023-10-16T15:34:25.584285+0000","last_active":"2023-10-16T15:34:27.785548+0000","last_peered":"2023-10-16T15:34:27.785548+0000","last_clean":"2023-10-16T15:34:27.785548+0000","last_became_active":"2023-10-16T15:34:25.583887+0000","last_became_peered":"2023-10-16T15:34:25.583887+0000","last_unstale":"2023-10-16T15:34:27.785548+0000","last_undegraded":"2023-10-16T15:34:27.785548+0000","last_fullsized":"2023-10-16T15:34:27.785548+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:06.125994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785978+0000","last_change":"2023-10-16T15:34:26.602762+0000","last_active":"2023-10-16T15:34:27.785978+0000","last_peered":"2023-10-16T15:34:27.785978+0000","last_clean":"2023-10-16T15:34:27.785978+0000","last_became_active":"2023-10-16T15:34:26.602467+0000","last_became_peered":"2023-10-16T15:34:26.602467+0000","last_unstale":"2023-10-16T15:34:27.785978+0000","last_undegraded":"2023-10-16T15:34:27.785978+0000","last_fullsized":"2023-10-16T15:34:27.785978+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:45.821024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.579297+0000","last_change":"2023-10-16T15:34:25.572530+0000","last_active":"2023-10-16T15:34:27.579297+0000","last_peered":"2023-10-16T15:34:27.579297+0000","last_clean":"2023-10-16T15:34:27.579297+0000","last_became_active":"2023-10-16T15:34:25.571988+0000","last_became_peered":"2023-10-16T15:34:25.571988+0000","last_unstale":"2023-10-16T15:34:27.579297+0000","last_undegraded":"2023-10-16T15:34:27.579297+0000","last_fullsized":"2023-10-16T15:34:27.579297+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:58.217428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785188+0000","last_change":"2023-10-16T15:34:26.608941+0000","last_active":"2023-10-16T15:34:27.785188+0000","last_peered":"2023-10-16T15:34:27.785188+0000","last_clean":"2023-10-16T15:34:27.785188+0000","last_became_active":"2023-10-16T15:34:26.608457+0000","last_became_peered":"2023-10-16T15:34:26.608457+0000","last_unstale":"2023-10-16T15:34:27.785188+0000","last_undegraded":"2023-10-16T15:34:27.785188+0000","last_fullsized":"2023-10-16T15:34:27.785188+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:37.869891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786479+0000","last_change":"2023-10-16T15:34:25.583233+0000","last_active":"2023-10-16T15:34:27.786479+0000","last_peered":"2023-10-16T15:34:27.786479+0000","last_clean":"2023-10-16T15:34:27.786479+0000","last_became_active":"2023-10-16T15:34:25.582900+0000","last_became_peered":"2023-10-16T15:34:25.582900+0000","last_unstale":"2023-10-16T15:34:27.786479+0000","last_undegraded":"2023-10-16T15:34:27.786479+0000","last_fullsized":"2023-10-16T15:34:27.786479+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:59.585980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576871+0000","last_change":"2023-10-16T15:34:26.603059+0000","last_active":"2023-10-16T15:34:27.576871+0000","last_peered":"2023-10-16T15:34:27.576871+0000","last_clean":"2023-10-16T15:34:27.576871+0000","last_became_active":"2023-10-16T15:34:26.602776+0000","last_became_peered":"2023-10-16T15:34:26.602776+0000","last_unstale":"2023-10-16T15:34:27.576871+0000","last_undegraded":"2023-10-16T15:34:27.576871+0000","last_fullsized":"2023-10-16T15:34:27.576871+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:19.762004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788156+0000","last_change":"2023-10-16T15:34:25.579357+0000","last_active":"2023-10-16T15:34:27.788156+0000","last_peered":"2023-10-16T15:34:27.788156+0000","last_clean":"2023-10-16T15:34:27.788156+0000","last_became_active":"2023-10-16T15:34:25.578914+0000","last_became_peered":"2023-10-16T15:34:25.578914+0000","last_unstale":"2023-10-16T15:34:27.788156+0000","last_undegraded":"2023-10-16T15:34:27.788156+0000","last_fullsized":"2023-10-16T15:34:27.788156+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:06.784535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787357+0000","last_change":"2023-10-16T15:34:26.601817+0000","last_active":"2023-10-16T15:34:27.787357+0000","last_peered":"2023-10-16T15:34:27.787357+0000","last_clean":"2023-10-16T15:34:27.787357+0000","last_became_active":"2023-10-16T15:34:26.600054+0000","last_became_peered":"2023-10-16T15:34:26.600054+0000","last_unstale":"2023-10-16T15:34:27.787357+0000","last_undegraded":"2023-10-16T15:34:27.787357+0000","last_fullsized":"2023-10-16T15:34:27.787357+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:13.873261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786171+0000","last_change":"2023-10-16T15:34:25.582691+0000","last_active":"2023-10-16T15:34:27.786171+0000","last_peered":"2023-10-16T15:34:27.786171+0000","last_clean":"2023-10-16T15:34:27.786171+0000","last_became_active":"2023-10-16T15:34:25.582042+0000","last_became_peered":"2023-10-16T15:34:25.582042+0000","last_unstale":"2023-10-16T15:34:27.786171+0000","last_undegraded":"2023-10-16T15:34:27.786171+0000","last_fullsized":"2023-10-16T15:34:27.786171+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:12.328978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784629+0000","last_change":"2023-10-16T15:34:26.590931+0000","last_active":"2023-10-16T15:34:27.784629+0000","last_peered":"2023-10-16T15:34:27.784629+0000","last_clean":"2023-10-16T15:34:27.784629+0000","last_became_active":"2023-10-16T15:34:26.590138+0000","last_became_peered":"2023-10-16T15:34:26.590138+0000","last_unstale":"2023-10-16T15:34:27.784629+0000","last_undegraded":"2023-10-16T15:34:27.784629+0000","last_fullsized":"2023-10-16T15:34:27.784629+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:24.346565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784309+0000","last_change":"2023-10-16T15:34:25.583239+0000","last_active":"2023-10-16T15:34:27.784309+0000","last_peered":"2023-10-16T15:34:27.784309+0000","last_clean":"2023-10-16T15:34:27.784309+0000","last_became_active":"2023-10-16T15:34:25.582754+0000","last_became_peered":"2023-10-16T15:34:25.582754+0000","last_unstale":"2023-10-16T15:34:27.784309+0000","last_undegraded":"2023-10-16T15:34:27.784309+0000","last_fullsized":"2023-10-16T15:34:27.784309+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:29.051118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.783850+0000","last_change":"2023-10-16T15:34:26.600386+0000","last_active":"2023-10-16T15:34:27.783850+0000","last_peered":"2023-10-16T15:34:27.783850+0000","last_clean":"2023-10-16T15:34:27.783850+0000","last_became_active":"2023-10-16T15:34:26.600111+0000","last_became_peered":"2023-10-16T15:34:26.600111+0000","last_unstale":"2023-10-16T15:34:27.783850+0000","last_undegraded":"2023-10-16T15:34:27.783850+0000","last_fullsized":"2023-10-16T15:34:27.783850+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:51.671018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581581+0000","last_change":"2023-10-16T15:34:25.574712+0000","last_active":"2023-10-16T15:34:27.581581+0000","last_peered":"2023-10-16T15:34:27.581581+0000","last_clean":"2023-10-16T15:34:27.581581+0000","last_became_active":"2023-10-16T15:34:25.574385+0000","last_became_peered":"2023-10-16T15:34:25.574385+0000","last_unstale":"2023-10-16T15:34:27.581581+0000","last_undegraded":"2023-10-16T15:34:27.581581+0000","last_fullsized":"2023-10-16T15:34:27.581581+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:54.811954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581692+0000","last_change":"2023-10-16T15:34:26.604076+0000","last_active":"2023-10-16T15:34:27.581692+0000","last_peered":"2023-10-16T15:34:27.581692+0000","last_clean":"2023-10-16T15:34:27.581692+0000","last_became_active":"2023-10-16T15:34:26.603160+0000","last_became_peered":"2023-10-16T15:34:26.603160+0000","last_unstale":"2023-10-16T15:34:27.581692+0000","last_undegraded":"2023-10-16T15:34:27.581692+0000","last_fullsized":"2023-10-16T15:34:27.581692+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:10.319066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786595+0000","last_change":"2023-10-16T15:34:25.577691+0000","last_active":"2023-10-16T15:34:27.786595+0000","last_peered":"2023-10-16T15:34:27.786595+0000","last_clean":"2023-10-16T15:34:27.786595+0000","last_became_active":"2023-10-16T15:34:25.577412+0000","last_became_peered":"2023-10-16T15:34:25.577412+0000","last_unstale":"2023-10-16T15:34:27.786595+0000","last_undegraded":"2023-10-16T15:34:27.786595+0000","last_fullsized":"2023-10-16T15:34:27.786595+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:01.365525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.575897+0000","last_change":"2023-10-16T15:34:26.586849+0000","last_active":"2023-10-16T15:34:27.575897+0000","last_peered":"2023-10-16T15:34:27.575897+0000","last_clean":"2023-10-16T15:34:27.575897+0000","last_became_active":"2023-10-16T15:34:26.586154+0000","last_became_peered":"2023-10-16T15:34:26.586154+0000","last_unstale":"2023-10-16T15:34:27.575897+0000","last_undegraded":"2023-10-16T15:34:27.575897+0000","last_fullsized":"2023-10-16T15:34:27.575897+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:37.947607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785391+0000","last_change":"2023-10-16T15:34:25.584579+0000","last_active":"2023-10-16T15:34:27.785391+0000","last_peered":"2023-10-16T15:34:27.785391+0000","last_clean":"2023-10-16T15:34:27.785391+0000","last_became_active":"2023-10-16T15:34:25.584090+0000","last_became_peered":"2023-10-16T15:34:25.584090+0000","last_unstale":"2023-10-16T15:34:27.785391+0000","last_undegraded":"2023-10-16T15:34:27.785391+0000","last_fullsized":"2023-10-16T15:34:27.785391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:38.002395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786811+0000","last_change":"2023-10-16T15:34:26.611760+0000","last_active":"2023-10-16T15:34:27.786811+0000","last_peered":"2023-10-16T15:34:27.786811+0000","last_clean":"2023-10-16T15:34:27.786811+0000","last_became_active":"2023-10-16T15:34:26.611452+0000","last_became_peered":"2023-10-16T15:34:26.611452+0000","last_unstale":"2023-10-16T15:34:27.786811+0000","last_undegraded":"2023-10-16T15:34:27.786811+0000","last_fullsized":"2023-10-16T15:34:27.786811+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:20.169116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580630+0000","last_change":"2023-10-16T15:34:25.566911+0000","last_active":"2023-10-16T15:34:27.580630+0000","last_peered":"2023-10-16T15:34:27.580630+0000","last_clean":"2023-10-16T15:34:27.580630+0000","last_became_active":"2023-10-16T15:34:25.566596+0000","last_became_peered":"2023-10-16T15:34:25.566596+0000","last_unstale":"2023-10-16T15:34:27.580630+0000","last_undegraded":"2023-10-16T15:34:27.580630+0000","last_fullsized":"2023-10-16T15:34:27.580630+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:27.167069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785667+0000","last_change":"2023-10-16T15:34:26.609121+0000","last_active":"2023-10-16T15:34:27.785667+0000","last_peered":"2023-10-16T15:34:27.785667+0000","last_clean":"2023-10-16T15:34:27.785667+0000","last_became_active":"2023-10-16T15:34:26.608869+0000","last_became_peered":"2023-10-16T15:34:26.608869+0000","last_unstale":"2023-10-16T15:34:27.785667+0000","last_undegraded":"2023-10-16T15:34:27.785667+0000","last_fullsized":"2023-10-16T15:34:27.785667+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:22.317917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582548+0000","last_change":"2023-10-16T15:34:25.577730+0000","last_active":"2023-10-16T15:34:27.582548+0000","last_peered":"2023-10-16T15:34:27.582548+0000","last_clean":"2023-10-16T15:34:27.582548+0000","last_became_active":"2023-10-16T15:34:25.576880+0000","last_became_peered":"2023-10-16T15:34:25.576880+0000","last_unstale":"2023-10-16T15:34:27.582548+0000","last_undegraded":"2023-10-16T15:34:27.582548+0000","last_fullsized":"2023-10-16T15:34:27.582548+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:45.537995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"181'296","reported_seq":1009,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-16T15:34:26.582788+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:34:26.582788+0000","last_peered":"2023-10-16T15:34:26.582788+0000","last_clean":"2023-10-16T15:34:26.582788+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:34:26.582788+0000","last_undegraded":"2023-10-16T15:34:26.582788+0000","last_fullsized":"2023-10-16T15:34:26.582788+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575779,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56340,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315500,"statfs":{"total":96636764160,"available":96579072000,"internally_reserved":0,"allocated":3018752,"data_stored":1762707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1799999999999999,"15min":1.206},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6990000000000001,"5min":1.752,"15min":1.7849999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.071,"5min":1.1419999999999999,"15min":1.1599999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.56999999999999995},"max":{"1min":1.4179999999999999,"5min":1.867,"15min":1.905},"last":1.087}]},{"osd":1,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.8480000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.252},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.339,"15min":1.3360000000000001},"min":{"1min":0.94099999999999995,"5min":0.91200000000000003,"15min":0.78200000000000003},"max":{"1min":1.583,"5min":2.282,"15min":2.282},"last":1.454}]},{"osd":2,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.282,"15min":1.3460000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6619999999999999,"5min":1.7549999999999999,"15min":1.9039999999999999},"last":1.323},{"interface":"front","average":{"1min":1.27,"5min":1.3440000000000001,"15min":1.3240000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.82599999999999996},"max":{"1min":2.081,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.397}]},{"osd":3,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3580000000000001,"15min":1.345},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.6599999999999999,"5min":2.528,"15min":2.528},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.452,"15min":1.456},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.96,"5min":1.96,"15min":1.962},"last":1.833}]},{"osd":4,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.411,"15min":1.387},"min":{"1min":1.036,"5min":0.84499999999999997,"15min":0.81699999999999995},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.502},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.387,"15min":1.337},"min":{"1min":0.93600000000000005,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":2.0329999999999999,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.5329999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.502,"15min":1.4770000000000001},"min":{"1min":1.089,"5min":0.97599999999999998,"15min":0.90500000000000003},"max":{"1min":1.929,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.5740000000000001},"min":{"1min":1.1659999999999999,"5min":0.98999999999999999,"15min":0.88500000000000001},"max":{"1min":2.0880000000000001,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":1.6419999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.5289999999999999,"15min":1.4890000000000001},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":0.90700000000000003},"max":{"1min":2.1389999999999998,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.6459999999999999,"15min":1.601},"min":{"1min":1.1299999999999999,"5min":1.0629999999999999,"15min":0.95099999999999996},"max":{"1min":2.1920000000000002,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":1.9590000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575778,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29368,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342472,"statfs":{"total":96636764160,"available":96606691328,"internally_reserved":0,"allocated":2404352,"data_stored":1170007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.823,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.913},{"interface":"front","average":{"1min":1.151,"5min":1.254,"15min":1.2529999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.80800000000000005},"max":{"1min":1.7250000000000001,"5min":1.823,"15min":1.9550000000000001},"last":6.6509999999999998}]},{"osd":1,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.238,"15min":1.286},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8799999999999999},"last":2.1320000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.3560000000000001,"15min":1.3300000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.843,"5min":1.843,"15min":2.738},"last":7.1210000000000004}]},{"osd":2,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.256,"15min":1.2749999999999999},"min":{"1min":0.89300000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.633,"5min":1.823,"15min":2.4039999999999999},"last":3.8109999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.389,"15min":1.389},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.89400000000000002},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":2.0289999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.411,"15min":1.407},"min":{"1min":1.139,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":3.9820000000000002},"last":6.798},{"interface":"front","average":{"1min":1.379,"5min":1.3740000000000001,"15min":1.3799999999999999},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.92100000000000004},"max":{"1min":2.0019999999999998,"5min":2.0539999999999998,"15min":2.5489999999999999},"last":4.1100000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.375,"15min":1.331},"min":{"1min":0.92200000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":4.835},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.429,"15min":1.3959999999999999},"min":{"1min":1.0860000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":4.6609999999999996},"last":6.9039999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3720000000000001,"15min":1.3620000000000001},"min":{"1min":1.121,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.6830000000000001,"5min":1.754,"15min":3.8450000000000002},"last":4.2160000000000002},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5429999999999999,"15min":1.524},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.044,"5min":2.044,"15min":2.8639999999999999},"last":3.101}]},{"osd":7,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.6140000000000001,"15min":1.6020000000000001},"min":{"1min":1.1479999999999999,"5min":1.0249999999999999,"15min":0.84799999999999998},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":4.5359999999999996},"last":7.0049999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.6160000000000001,"15min":1.6000000000000001},"min":{"1min":1.3280000000000001,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.925,"5min":1.9570000000000001,"15min":2.6619999999999999},"last":5.6360000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41852,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329988,"statfs":{"total":96636764160,"available":96593907712,"internally_reserved":0,"allocated":2408448,"data_stored":1170589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.2070000000000001,"15min":1.1850000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.4159999999999999,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.319},{"interface":"front","average":{"1min":1.246,"5min":1.25,"15min":1.258},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7589999999999999,"5min":1.845,"15min":1.9570000000000001},"last":1.2529999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.234,"15min":1.224},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.80900000000000005},"max":{"1min":1.6419999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.2949999999999999,"15min":1.302},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.7230000000000001,"5min":1.738,"15min":1.905},"last":1.6839999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.27,"15min":1.264},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.9330000000000001,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.196},{"interface":"front","average":{"1min":1.387,"5min":1.401,"15min":1.401},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.835,"5min":2.073,"15min":2.3130000000000002},"last":1.5600000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2929999999999999,"15min":1.3180000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.5800000000000001,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.367},{"interface":"front","average":{"1min":1.349,"5min":1.3799999999999999,"15min":1.375},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":1.498}]},{"osd":4,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3100000000000001,"15min":1.3089999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.7070000000000001,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.474,"15min":1.4610000000000001},"min":{"1min":0.88700000000000001,"5min":0.83199999999999996,"15min":0.81000000000000005},"max":{"1min":2.1030000000000002,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.603}]},{"osd":6,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3720000000000001,"15min":1.361},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.508},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4370000000000001,"15min":1.4319999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.764,"5min":2.9209999999999998,"15min":2.9790000000000001},"last":1.5249999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5049999999999999,"15min":1.504},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.3719999999999999,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.5920000000000001,"15min":1.601},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.3420000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.6319999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575779,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59912,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311928,"statfs":{"total":96636764160,"available":96575414272,"internally_reserved":0,"allocated":2416640,"data_stored":1171667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.194,"15min":1.1879999999999999},"min":{"1min":0.77600000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.458,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.754},{"interface":"front","average":{"1min":1.145,"5min":1.159,"15min":1.177},"min":{"1min":0.749,"5min":0.749,"15min":0.745},"max":{"1min":1.494,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.198}]},{"osd":1,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.244,"15min":1.2330000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.6659999999999999,"5min":1.78,"15min":1.948},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.268,"15min":1.268},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.7050000000000001,"5min":1.712,"15min":1.895},"last":1.0049999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.4139999999999999,"15min":1.417},"min":{"1min":0.91100000000000003,"5min":0.88,"15min":0.88},"max":{"1min":1.5900000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.409,"15min":1.431},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6830000000000001,"5min":2,"15min":2},"last":0.85699999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3520000000000001,"15min":1.373},"min":{"1min":0.98399999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.5760000000000001,"5min":1.8149999999999999,"15min":2.093},"last":1.546},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.353,"15min":1.3720000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":1.825,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.2430000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.333,"15min":1.3400000000000001},"min":{"1min":1.048,"5min":0.89300000000000002,"15min":0.77900000000000003},"max":{"1min":1.7,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.653},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.361,"15min":1.413},"min":{"1min":0.89800000000000002,"5min":0.86799999999999999,"15min":0.86299999999999999},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":2.097},"last":1.7310000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3400000000000001,"15min":1.349},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.627,"5min":1.903,"15min":1.903},"last":1.095},{"interface":"front","average":{"1min":1.49,"5min":1.512,"15min":1.5209999999999999},"min":{"1min":1.1699999999999999,"5min":1.026,"15min":1.026},"max":{"1min":1.8240000000000001,"5min":2.359,"15min":2.359},"last":1.7829999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.518,"15min":1.5580000000000001},"min":{"1min":1.123,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":1.778,"5min":1.9059999999999999,"15min":2.1600000000000001},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.5249999999999999,"15min":1.5409999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.7969999999999999,"5min":2.0720000000000001,"15min":2.3330000000000002},"last":1.605}]}]},{"osd":3,"up_from":13,"seq":55834575778,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51576,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320264,"statfs":{"total":96636764160,"available":96583950336,"internally_reserved":0,"allocated":2404352,"data_stored":1170019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.92100000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3120000000000001,"5min":1.569,"15min":1.7230000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.069,"15min":1.077},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.52,"5min":1.577,"15min":1.9430000000000001},"last":1.081}]},{"osd":1,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.97399999999999998,"15min":0.999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.4730000000000001,"5min":1.5600000000000001,"15min":1.704},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.214,"15min":1.206},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.46}]},{"osd":2,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.353,"15min":1.4099999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6120000000000001,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4950000000000001,"15min":1.514},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8300000000000001,"5min":2.278,"15min":2.278},"last":0.96399999999999997}]},{"osd":4,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.3540000000000001,"15min":1.327},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.583},{"interface":"front","average":{"1min":1.399,"5min":1.3939999999999999,"15min":1.401},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1840000000000002},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5449999999999999,"15min":1.5649999999999999},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":2.0350000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.244},{"interface":"front","average":{"1min":1.518,"5min":1.6040000000000001,"15min":1.5620000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.956,"5min":2.621,"15min":2.774},"last":1.196}]},{"osd":6,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.5069999999999999,"15min":1.51},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.1230000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.1190000000000002,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.409}]},{"osd":7,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.6739999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.04,"5min":2.7309999999999999,"15min":2.8900000000000001},"last":1.347},{"interface":"front","average":{"1min":1.488,"5min":1.633,"15min":1.6439999999999999},"min":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.05},"max":{"1min":1.8819999999999999,"5min":2.1869999999999998,"15min":2.496},"last":1.4950000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575779,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51576,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320264,"statfs":{"total":96636764160,"available":96583950336,"internally_reserved":0,"allocated":2404352,"data_stored":1170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.95099999999999996,"15min":0.92800000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5880000000000001,"5min":1.643,"15min":1.849},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.099},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6299999999999999,"5min":1.6799999999999999,"15min":1.7490000000000001},"last":1.0049999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.018,"15min":1.024},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.585,"5min":1.6419999999999999,"15min":1.8140000000000001},"last":1.121},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.216,"15min":1.2230000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.702,"5min":1.8300000000000001,"15min":1.9299999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.28,"15min":1.3169999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.462,"5min":1.8120000000000001,"15min":1.992},"last":1.323},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.351,"15min":1.3620000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.643,"5min":1.921,"15min":1.921},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.454,"15min":1.472},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.7709999999999999,"5min":2.1150000000000002,"15min":2.149},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.9159999999999999,"5min":1.986,"15min":2.6909999999999998},"last":1.1040000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5089999999999999,"15min":1.4830000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9810000000000001,"5min":2.661,"15min":2.661},"last":1.419},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.591,"15min":1.611},"min":{"1min":1.032,"5min":1.032,"15min":1.032},"max":{"1min":1.831,"5min":2.3730000000000002,"15min":3.3149999999999999},"last":1.7}]},{"osd":6,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4790000000000001,"15min":1.5},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":2.0489999999999999,"5min":2.1429999999999998,"15min":2.573},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.613,"15min":1.6040000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.948,"5min":2.423,"15min":2.423},"last":1.5149999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5640000000000001,"15min":1.595},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.1549999999999998,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.663,"5min":1.7210000000000001,"15min":1.7150000000000001},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.0979999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.6220000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575779,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64064,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307776,"statfs":{"total":96636764160,"available":96571162624,"internally_reserved":0,"allocated":2404352,"data_stored":1170019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91400000000000003,"15min":0.96099999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.6830000000000001,"5min":1.716,"15min":1.8440000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1599999999999999,"15min":1.1259999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.337}]},{"osd":2,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2010000000000001,"15min":1.22},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.53,"5min":1.7410000000000001,"15min":1.9430000000000001},"last":1.653}]},{"osd":3,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.371,"15min":1.379},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.012,"5min":2.012,"15min":2.02},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.258,"15min":1.2729999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.048}]},{"osd":4,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.385,"15min":1.361},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.883,"5min":1.8879999999999999,"15min":1.9990000000000001},"last":1.236},{"interface":"front","average":{"1min":1.427,"5min":1.4730000000000001,"15min":1.4930000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.79,"5min":1.9059999999999999,"15min":2.3500000000000001},"last":1.7829999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.532,"15min":1.4970000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.0619999999999998,"5min":2.2770000000000001,"15min":2.2770000000000001},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.472,"15min":1.462},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.889,"5min":2.1059999999999999,"15min":2.1200000000000001},"last":1.46}]},{"osd":6,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.629,"15min":1.639},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":2.5230000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6399999999999999,"15min":1.611},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.1579999999999999,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.718}]},{"osd":7,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.756,"15min":1.7170000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":2.274,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.621,"5min":1.615,"15min":1.6080000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.036,"5min":2.0409999999999999,"15min":2.3039999999999998},"last":1.5960000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575779,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60472,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311368,"statfs":{"total":96636764160,"available":96574840832,"internally_reserved":0,"allocated":2994176,"data_stored":1760353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":0.97199999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2050000000000001,"5min":1.903,"15min":1.903},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6910000000000001,"5min":1.851,"15min":1.9450000000000001},"last":1.2549999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.29,"15min":1.304},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.913,"5min":1.913,"15min":2.0640000000000001},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.2330000000000001,"15min":1.26},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8340000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0489999999999999,"15min":1.0249999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.306,"5min":1.732,"15min":1.732},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.137,"5min":1.282,"15min":1.2849999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8360000000000001,"5min":2.262,"15min":2.262},"last":1.3540000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3440000000000001,"15min":1.3280000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7529999999999999,"5min":2.109,"15min":2.109},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.516,"15min":1.522},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.9590000000000001,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.6539999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.484,"15min":1.492},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.7869999999999999,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4430000000000001,"15min":1.4399999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.4319999999999999,"5min":3.4319999999999999,"15min":3.4319999999999999},"last":1.208}]},{"osd":6,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.665,"15min":1.649},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.3130000000000002,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":1.859},{"interface":"front","average":{"1min":1.462,"5min":1.5940000000000001,"15min":1.5760000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0680000000000001,"5min":2.4340000000000002,"15min":2.5009999999999999},"last":1.099}]},{"osd":7,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.6799999999999999,"15min":1.6830000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.089,"5min":3.089,"15min":3.089},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.5940000000000001,"15min":1.5960000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9650000000000001,"5min":2.774,"15min":2.774},"last":1.5289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:34:31.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:34:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.389+0000 7fe766b1a640 1 Processor -- start 2023-10-16T15:34:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.389+0000 7fe766b1a640 1 -- start start 2023-10-16T15:34:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.391+0000 7fe766b1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe760070b60 0x7fe76015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.391+0000 7fe766b1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe76006c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.391+0000 7fe766b1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76015aef0 con 0x7fe760159950 2023-10-16T15:34:31.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.391+0000 7fe766b1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76015e490 con 0x7fe760070b60 2023-10-16T15:34:31.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.391+0000 7fe766b1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76015d910 con 0x7fe76006c100 2023-10-16T15:34:31.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.392+0000 7fe766319640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe76006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.392+0000 7fe765317640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe760070b60 0x7fe76015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.392+0000 7fe766319640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe76006c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:33664/0 (socket says 172.21.15.29:33664) 2023-10-16T15:34:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.392+0000 7fe765317640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe760070b60 0x7fe76015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59720/0 (socket says 172.21.15.29:59720) 2023-10-16T15:34:31.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.392+0000 7fe766319640 1 -- 172.21.15.29:0/2521940250 learned_addr learned my addr 172.21.15.29:0/2521940250 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:31.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.392+0000 7fe765317640 1 -- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 msgr2=0x7fe76006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:31.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.393+0000 7fe765317640 1 --2- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe76006c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.393+0000 7fe765317640 1 -- 172.21.15.29:0/2521940250 >> v1:172.21.15.170:6789/0 conn(0x7fe760159950 legacy=0x7fe760070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.393+0000 7fe765317640 1 -- 172.21.15.29:0/2521940250 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76015dbf0 con 0x7fe760070b60 2023-10-16T15:34:31.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.393+0000 7fe766319640 1 --2- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe76006c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:34:31.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.394+0000 7fe765317640 1 --2- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe76015a9b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7500097a0 tx=0x7fe75002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20fe35071e9e8e2a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.395+0000 7fe764b16640 1 -- 172.21.15.29:0/2521940250 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750031780 con 0x7fe760070b60 2023-10-16T15:34:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.395+0000 7fe764b16640 1 -- 172.21.15.29:0/2521940250 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7500318e0 con 0x7fe760070b60 2023-10-16T15:34:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.395+0000 7fe764b16640 1 -- 172.21.15.29:0/2521940250 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750031c70 con 0x7fe760070b60 2023-10-16T15:34:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.395+0000 7fe766b1a640 1 -- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 msgr2=0x7fe76015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.395+0000 7fe766b1a640 1 --2- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe76015a9b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7500097a0 tx=0x7fe75002b820 comp rx=0 tx=0).stop 2023-10-16T15:34:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.396+0000 7fe766b1a640 1 -- 172.21.15.29:0/2521940250 shutdown_connections 2023-10-16T15:34:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.396+0000 7fe766b1a640 1 --2- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe76006c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.396+0000 7fe766b1a640 1 --2- 172.21.15.29:0/2521940250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe76015a9b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.396+0000 7fe766b1a640 1 -- 172.21.15.29:0/2521940250 >> 172.21.15.29:0/2521940250 conn(0x7fe7600ce6a0 msgr2=0x7fe7600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.396+0000 7fe766b1a640 1 -- 172.21.15.29:0/2521940250 shutdown_connections 2023-10-16T15:34:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.396+0000 7fe766b1a640 1 -- 172.21.15.29:0/2521940250 wait complete. 2023-10-16T15:34:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.398+0000 7fe766b1a640 1 Processor -- start 2023-10-16T15:34:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.398+0000 7fe766b1a640 1 -- start start 2023-10-16T15:34:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766b1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe7601f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766b1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe7600acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766b1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 0x7fe760157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766b1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600c3200 con 0x7fe760159950 2023-10-16T15:34:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766b1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760157cc0 con 0x7fe760070b60 2023-10-16T15:34:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766b1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600bdf40 con 0x7fe76006c100 2023-10-16T15:34:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766319640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 0x7fe760157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766319640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 0x7fe760157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33440/0 (socket says 172.21.15.29:33440) 2023-10-16T15:34:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe766319640 1 -- 172.21.15.29:0/671633425 learned_addr learned my addr 172.21.15.29:0/671633425 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe765317640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe7600acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.400+0000 7fe765b18640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe7601f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766319640 1 -- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 msgr2=0x7fe7601f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766319640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe7601f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766319640 1 -- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 msgr2=0x7fe7600acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766319640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe7600acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766319640 1 -- 172.21.15.29:0/671633425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe758009b30 con 0x7fe760159950 2023-10-16T15:34:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766319640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 0x7fe760157780 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fe758009e70 tx=0x7fe75800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758018070 con 0x7fe760159950 2023-10-16T15:34:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75800f040 con 0x7fe760159950 2023-10-16T15:34:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800ce00 con 0x7fe760159950 2023-10-16T15:34:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.401+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe750009400 con 0x7fe760159950 2023-10-16T15:34:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.402+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7600c3730 con 0x7fe760159950 2023-10-16T15:34:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.403+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76004e9b0 con 0x7fe760159950 2023-10-16T15:34:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.404+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe758014480 con 0x7fe760159950 2023-10-16T15:34:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.405+0000 7fe7567fc640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe74406af90 0x7fe74406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.405+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe758089b20 con 0x7fe760159950 2023-10-16T15:34:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.405+0000 7fe765b18640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe74406af90 0x7fe74406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.406+0000 7fe765b18640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe74406af90 0x7fe74406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7601f99d0 tx=0x7fe75c009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.411+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe758054a20 con 0x7fe760159950 2023-10-16T15:34:31.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.602+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 --> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe76015e1b0 con 0x7fe74406af90 2023-10-16T15:34:31.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.625+0000 7fe7567fc640 1 -- 172.21.15.29:0/671633425 <== mgr.7085 v2:172.21.15.170:6832/3694096961 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286822 (secure 0 0 0) 0x7fe75c001c70 con 0x7fe74406af90 2023-10-16T15:34:31.633 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:31.636 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:34:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe74406af90 msgr2=0x7fe74406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe74406af90 0x7fe74406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7601f99d0 tx=0x7fe75c009040 comp rx=0 tx=0).stop 2023-10-16T15:34:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 msgr2=0x7fe760157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 0x7fe760157780 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fe758009e70 tx=0x7fe75800c490 comp rx=0 tx=0).stop 2023-10-16T15:34:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 shutdown_connections 2023-10-16T15:34:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fe74406af90 0x7fe74406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe760159950 0x7fe760157780 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe760070b60 0x7fe7600acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 --2- 172.21.15.29:0/671633425 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe76006c100 0x7fe7601f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 >> 172.21.15.29:0/671633425 conn(0x7fe7600ce6a0 msgr2=0x7fe760070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 shutdown_connections 2023-10-16T15:34:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.632+0000 7fe766b1a640 1 -- 172.21.15.29:0/671633425 wait complete. 2023-10-16T15:34:31.653 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T15:34:29.887441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":527,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":334,"ondisk_log_size":334,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":415160,"kb_used_data":19976,"kb_used_omap":12,"kb_used_meta":395059,"kb_avail":754559560,"statfs":{"total":773094113280,"available":772668989440,"internally_reserved":0,"allocated":20455424,"data_stored":10545436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":404540985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.348808"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580494+0000","last_change":"2023-10-16T15:34:26.617126+0000","last_active":"2023-10-16T15:34:27.580494+0000","last_peered":"2023-10-16T15:34:27.580494+0000","last_clean":"2023-10-16T15:34:27.580494+0000","last_became_active":"2023-10-16T15:34:26.616712+0000","last_became_peered":"2023-10-16T15:34:26.616712+0000","last_unstale":"2023-10-16T15:34:27.580494+0000","last_undegraded":"2023-10-16T15:34:27.580494+0000","last_fullsized":"2023-10-16T15:34:27.580494+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:24.179525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786121+0000","last_change":"2023-10-16T15:34:26.620617+0000","last_active":"2023-10-16T15:34:27.786121+0000","last_peered":"2023-10-16T15:34:27.786121+0000","last_clean":"2023-10-16T15:34:27.786121+0000","last_became_active":"2023-10-16T15:34:26.619652+0000","last_became_peered":"2023-10-16T15:34:26.619652+0000","last_unstale":"2023-10-16T15:34:27.786121+0000","last_undegraded":"2023-10-16T15:34:27.786121+0000","last_fullsized":"2023-10-16T15:34:27.786121+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:36.340635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787849+0000","last_change":"2023-10-16T15:34:26.611209+0000","last_active":"2023-10-16T15:34:27.787849+0000","last_peered":"2023-10-16T15:34:27.787849+0000","last_clean":"2023-10-16T15:34:27.787849+0000","last_became_active":"2023-10-16T15:34:26.610937+0000","last_became_peered":"2023-10-16T15:34:26.610937+0000","last_unstale":"2023-10-16T15:34:27.787849+0000","last_undegraded":"2023-10-16T15:34:27.787849+0000","last_fullsized":"2023-10-16T15:34:27.787849+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:08.014086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787118+0000","last_change":"2023-10-16T15:34:26.594443+0000","last_active":"2023-10-16T15:34:27.787118+0000","last_peered":"2023-10-16T15:34:27.787118+0000","last_clean":"2023-10-16T15:34:27.787118+0000","last_became_active":"2023-10-16T15:34:26.593172+0000","last_became_peered":"2023-10-16T15:34:26.593172+0000","last_unstale":"2023-10-16T15:34:27.787118+0000","last_undegraded":"2023-10-16T15:34:27.787118+0000","last_fullsized":"2023-10-16T15:34:27.787118+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:11.805397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785471+0000","last_change":"2023-10-16T15:34:26.606917+0000","last_active":"2023-10-16T15:34:27.785471+0000","last_peered":"2023-10-16T15:34:27.785471+0000","last_clean":"2023-10-16T15:34:27.785471+0000","last_became_active":"2023-10-16T15:34:26.606629+0000","last_became_peered":"2023-10-16T15:34:26.606629+0000","last_unstale":"2023-10-16T15:34:27.785471+0000","last_undegraded":"2023-10-16T15:34:27.785471+0000","last_fullsized":"2023-10-16T15:34:27.785471+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:14.043314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786225+0000","last_change":"2023-10-16T15:34:26.596158+0000","last_active":"2023-10-16T15:34:27.786225+0000","last_peered":"2023-10-16T15:34:27.786225+0000","last_clean":"2023-10-16T15:34:27.786225+0000","last_became_active":"2023-10-16T15:34:26.595918+0000","last_became_peered":"2023-10-16T15:34:26.595918+0000","last_unstale":"2023-10-16T15:34:27.786225+0000","last_undegraded":"2023-10-16T15:34:27.786225+0000","last_fullsized":"2023-10-16T15:34:27.786225+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:33.630041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576144+0000","last_change":"2023-10-16T15:34:26.595098+0000","last_active":"2023-10-16T15:34:27.576144+0000","last_peered":"2023-10-16T15:34:27.576144+0000","last_clean":"2023-10-16T15:34:27.576144+0000","last_became_active":"2023-10-16T15:34:26.594512+0000","last_became_peered":"2023-10-16T15:34:26.594512+0000","last_unstale":"2023-10-16T15:34:27.576144+0000","last_undegraded":"2023-10-16T15:34:27.576144+0000","last_fullsized":"2023-10-16T15:34:27.576144+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:09.285127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784955+0000","last_change":"2023-10-16T15:34:26.615957+0000","last_active":"2023-10-16T15:34:27.784955+0000","last_peered":"2023-10-16T15:34:27.784955+0000","last_clean":"2023-10-16T15:34:27.784955+0000","last_became_active":"2023-10-16T15:34:26.611184+0000","last_became_peered":"2023-10-16T15:34:26.611184+0000","last_unstale":"2023-10-16T15:34:27.784955+0000","last_undegraded":"2023-10-16T15:34:27.784955+0000","last_fullsized":"2023-10-16T15:34:27.784955+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:46.606847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787107+0000","last_change":"2023-10-16T15:34:26.612683+0000","last_active":"2023-10-16T15:34:27.787107+0000","last_peered":"2023-10-16T15:34:27.787107+0000","last_clean":"2023-10-16T15:34:27.787107+0000","last_became_active":"2023-10-16T15:34:26.612408+0000","last_became_peered":"2023-10-16T15:34:26.612408+0000","last_unstale":"2023-10-16T15:34:27.787107+0000","last_undegraded":"2023-10-16T15:34:27.787107+0000","last_fullsized":"2023-10-16T15:34:27.787107+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:09.072515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786570+0000","last_change":"2023-10-16T15:34:26.609261+0000","last_active":"2023-10-16T15:34:27.786570+0000","last_peered":"2023-10-16T15:34:27.786570+0000","last_clean":"2023-10-16T15:34:27.786570+0000","last_became_active":"2023-10-16T15:34:26.608923+0000","last_became_peered":"2023-10-16T15:34:26.608923+0000","last_unstale":"2023-10-16T15:34:27.786570+0000","last_undegraded":"2023-10-16T15:34:27.786570+0000","last_fullsized":"2023-10-16T15:34:27.786570+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:15:00.216820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785969+0000","last_change":"2023-10-16T15:34:26.611404+0000","last_active":"2023-10-16T15:34:27.785969+0000","last_peered":"2023-10-16T15:34:27.785969+0000","last_clean":"2023-10-16T15:34:27.785969+0000","last_became_active":"2023-10-16T15:34:26.611131+0000","last_became_peered":"2023-10-16T15:34:26.611131+0000","last_unstale":"2023-10-16T15:34:27.785969+0000","last_undegraded":"2023-10-16T15:34:27.785969+0000","last_fullsized":"2023-10-16T15:34:27.785969+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:55.349752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785999+0000","last_change":"2023-10-16T15:34:26.612640+0000","last_active":"2023-10-16T15:34:27.785999+0000","last_peered":"2023-10-16T15:34:27.785999+0000","last_clean":"2023-10-16T15:34:27.785999+0000","last_became_active":"2023-10-16T15:34:26.612425+0000","last_became_peered":"2023-10-16T15:34:26.612425+0000","last_unstale":"2023-10-16T15:34:27.785999+0000","last_undegraded":"2023-10-16T15:34:27.785999+0000","last_fullsized":"2023-10-16T15:34:27.785999+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:07.800060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787607+0000","last_change":"2023-10-16T15:34:26.610895+0000","last_active":"2023-10-16T15:34:27.787607+0000","last_peered":"2023-10-16T15:34:27.787607+0000","last_clean":"2023-10-16T15:34:27.787607+0000","last_became_active":"2023-10-16T15:34:26.610567+0000","last_became_peered":"2023-10-16T15:34:26.610567+0000","last_unstale":"2023-10-16T15:34:27.787607+0000","last_undegraded":"2023-10-16T15:34:27.787607+0000","last_fullsized":"2023-10-16T15:34:27.787607+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:34.164591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786828+0000","last_change":"2023-10-16T15:34:26.616427+0000","last_active":"2023-10-16T15:34:27.786828+0000","last_peered":"2023-10-16T15:34:27.786828+0000","last_clean":"2023-10-16T15:34:27.786828+0000","last_became_active":"2023-10-16T15:34:26.614700+0000","last_became_peered":"2023-10-16T15:34:26.614700+0000","last_unstale":"2023-10-16T15:34:27.786828+0000","last_undegraded":"2023-10-16T15:34:27.786828+0000","last_fullsized":"2023-10-16T15:34:27.786828+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.239014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.578699+0000","last_change":"2023-10-16T15:34:26.597864+0000","last_active":"2023-10-16T15:34:27.578699+0000","last_peered":"2023-10-16T15:34:27.578699+0000","last_clean":"2023-10-16T15:34:27.578699+0000","last_became_active":"2023-10-16T15:34:26.597586+0000","last_became_peered":"2023-10-16T15:34:26.597586+0000","last_unstale":"2023-10-16T15:34:27.578699+0000","last_undegraded":"2023-10-16T15:34:27.578699+0000","last_fullsized":"2023-10-16T15:34:27.578699+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:14.612765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.575772+0000","last_change":"2023-10-16T15:34:26.587033+0000","last_active":"2023-10-16T15:34:27.575772+0000","last_peered":"2023-10-16T15:34:27.575772+0000","last_clean":"2023-10-16T15:34:27.575772+0000","last_became_active":"2023-10-16T15:34:26.586560+0000","last_became_peered":"2023-10-16T15:34:26.586560+0000","last_unstale":"2023-10-16T15:34:27.575772+0000","last_undegraded":"2023-10-16T15:34:27.575772+0000","last_fullsized":"2023-10-16T15:34:27.575772+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:59.385000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787621+0000","last_change":"2023-10-16T15:34:26.618489+0000","last_active":"2023-10-16T15:34:27.787621+0000","last_peered":"2023-10-16T15:34:27.787621+0000","last_clean":"2023-10-16T15:34:27.787621+0000","last_became_active":"2023-10-16T15:34:26.618127+0000","last_became_peered":"2023-10-16T15:34:26.618127+0000","last_unstale":"2023-10-16T15:34:27.787621+0000","last_undegraded":"2023-10-16T15:34:27.787621+0000","last_fullsized":"2023-10-16T15:34:27.787621+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:07.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784691+0000","last_change":"2023-10-16T15:34:26.605198+0000","last_active":"2023-10-16T15:34:27.784691+0000","last_peered":"2023-10-16T15:34:27.784691+0000","last_clean":"2023-10-16T15:34:27.784691+0000","last_became_active":"2023-10-16T15:34:26.604735+0000","last_became_peered":"2023-10-16T15:34:26.604735+0000","last_unstale":"2023-10-16T15:34:27.784691+0000","last_undegraded":"2023-10-16T15:34:27.784691+0000","last_fullsized":"2023-10-16T15:34:27.784691+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:04.516745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786927+0000","last_change":"2023-10-16T15:34:26.607556+0000","last_active":"2023-10-16T15:34:27.786927+0000","last_peered":"2023-10-16T15:34:27.786927+0000","last_clean":"2023-10-16T15:34:27.786927+0000","last_became_active":"2023-10-16T15:34:26.607232+0000","last_became_peered":"2023-10-16T15:34:26.607232+0000","last_unstale":"2023-10-16T15:34:27.786927+0000","last_undegraded":"2023-10-16T15:34:27.786927+0000","last_fullsized":"2023-10-16T15:34:27.786927+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:46.070157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787824+0000","last_change":"2023-10-16T15:34:26.620697+0000","last_active":"2023-10-16T15:34:27.787824+0000","last_peered":"2023-10-16T15:34:27.787824+0000","last_clean":"2023-10-16T15:34:27.787824+0000","last_became_active":"2023-10-16T15:34:26.619362+0000","last_became_peered":"2023-10-16T15:34:26.619362+0000","last_unstale":"2023-10-16T15:34:27.787824+0000","last_undegraded":"2023-10-16T15:34:27.787824+0000","last_fullsized":"2023-10-16T15:34:27.787824+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:36.566128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786090+0000","last_change":"2023-10-16T15:34:26.612004+0000","last_active":"2023-10-16T15:34:27.786090+0000","last_peered":"2023-10-16T15:34:27.786090+0000","last_clean":"2023-10-16T15:34:27.786090+0000","last_became_active":"2023-10-16T15:34:26.611713+0000","last_became_peered":"2023-10-16T15:34:26.611713+0000","last_unstale":"2023-10-16T15:34:27.786090+0000","last_undegraded":"2023-10-16T15:34:27.786090+0000","last_fullsized":"2023-10-16T15:34:27.786090+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:48:10.065535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785495+0000","last_change":"2023-10-16T15:34:26.619978+0000","last_active":"2023-10-16T15:34:27.785495+0000","last_peered":"2023-10-16T15:34:27.785495+0000","last_clean":"2023-10-16T15:34:27.785495+0000","last_became_active":"2023-10-16T15:34:26.618384+0000","last_became_peered":"2023-10-16T15:34:26.618384+0000","last_unstale":"2023-10-16T15:34:27.785495+0000","last_undegraded":"2023-10-16T15:34:27.785495+0000","last_fullsized":"2023-10-16T15:34:27.785495+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:24.777311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785832+0000","last_change":"2023-10-16T15:34:26.605438+0000","last_active":"2023-10-16T15:34:27.785832+0000","last_peered":"2023-10-16T15:34:27.785832+0000","last_clean":"2023-10-16T15:34:27.785832+0000","last_became_active":"2023-10-16T15:34:26.605108+0000","last_became_peered":"2023-10-16T15:34:26.605108+0000","last_unstale":"2023-10-16T15:34:27.785832+0000","last_undegraded":"2023-10-16T15:34:27.785832+0000","last_fullsized":"2023-10-16T15:34:27.785832+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:58.646193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785138+0000","last_change":"2023-10-16T15:34:26.619182+0000","last_active":"2023-10-16T15:34:27.785138+0000","last_peered":"2023-10-16T15:34:27.785138+0000","last_clean":"2023-10-16T15:34:27.785138+0000","last_became_active":"2023-10-16T15:34:26.618805+0000","last_became_peered":"2023-10-16T15:34:26.618805+0000","last_unstale":"2023-10-16T15:34:27.785138+0000","last_undegraded":"2023-10-16T15:34:27.785138+0000","last_fullsized":"2023-10-16T15:34:27.785138+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:46.045117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788353+0000","last_change":"2023-10-16T15:34:26.617033+0000","last_active":"2023-10-16T15:34:27.788353+0000","last_peered":"2023-10-16T15:34:27.788353+0000","last_clean":"2023-10-16T15:34:27.788353+0000","last_became_active":"2023-10-16T15:34:26.611442+0000","last_became_peered":"2023-10-16T15:34:26.611442+0000","last_unstale":"2023-10-16T15:34:27.788353+0000","last_undegraded":"2023-10-16T15:34:27.788353+0000","last_fullsized":"2023-10-16T15:34:27.788353+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:21:41.256837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576653+0000","last_change":"2023-10-16T15:34:26.591545+0000","last_active":"2023-10-16T15:34:27.576653+0000","last_peered":"2023-10-16T15:34:27.576653+0000","last_clean":"2023-10-16T15:34:27.576653+0000","last_became_active":"2023-10-16T15:34:26.591092+0000","last_became_peered":"2023-10-16T15:34:26.591092+0000","last_unstale":"2023-10-16T15:34:27.576653+0000","last_undegraded":"2023-10-16T15:34:27.576653+0000","last_fullsized":"2023-10-16T15:34:27.576653+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:21.058793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787957+0000","last_change":"2023-10-16T15:34:26.617274+0000","last_active":"2023-10-16T15:34:27.787957+0000","last_peered":"2023-10-16T15:34:27.787957+0000","last_clean":"2023-10-16T15:34:27.787957+0000","last_became_active":"2023-10-16T15:34:26.616426+0000","last_became_peered":"2023-10-16T15:34:26.616426+0000","last_unstale":"2023-10-16T15:34:27.787957+0000","last_undegraded":"2023-10-16T15:34:27.787957+0000","last_fullsized":"2023-10-16T15:34:27.787957+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:01.713175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787259+0000","last_change":"2023-10-16T15:34:26.604541+0000","last_active":"2023-10-16T15:34:27.787259+0000","last_peered":"2023-10-16T15:34:27.787259+0000","last_clean":"2023-10-16T15:34:27.787259+0000","last_became_active":"2023-10-16T15:34:26.604284+0000","last_became_peered":"2023-10-16T15:34:26.604284+0000","last_unstale":"2023-10-16T15:34:27.787259+0000","last_undegraded":"2023-10-16T15:34:27.787259+0000","last_fullsized":"2023-10-16T15:34:27.787259+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:55.827479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787645+0000","last_change":"2023-10-16T15:34:26.619978+0000","last_active":"2023-10-16T15:34:27.787645+0000","last_peered":"2023-10-16T15:34:27.787645+0000","last_clean":"2023-10-16T15:34:27.787645+0000","last_became_active":"2023-10-16T15:34:26.615076+0000","last_became_peered":"2023-10-16T15:34:26.615076+0000","last_unstale":"2023-10-16T15:34:27.787645+0000","last_undegraded":"2023-10-16T15:34:27.787645+0000","last_fullsized":"2023-10-16T15:34:27.787645+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:49.301073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581132+0000","last_change":"2023-10-16T15:34:26.588117+0000","last_active":"2023-10-16T15:34:27.581132+0000","last_peered":"2023-10-16T15:34:27.581132+0000","last_clean":"2023-10-16T15:34:27.581132+0000","last_became_active":"2023-10-16T15:34:26.587546+0000","last_became_peered":"2023-10-16T15:34:26.587546+0000","last_unstale":"2023-10-16T15:34:27.581132+0000","last_undegraded":"2023-10-16T15:34:27.581132+0000","last_fullsized":"2023-10-16T15:34:27.581132+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:24:15.426000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581265+0000","last_change":"2023-10-16T15:34:26.604743+0000","last_active":"2023-10-16T15:34:27.581265+0000","last_peered":"2023-10-16T15:34:27.581265+0000","last_clean":"2023-10-16T15:34:27.581265+0000","last_became_active":"2023-10-16T15:34:26.604464+0000","last_became_peered":"2023-10-16T15:34:26.604464+0000","last_unstale":"2023-10-16T15:34:27.581265+0000","last_undegraded":"2023-10-16T15:34:27.581265+0000","last_fullsized":"2023-10-16T15:34:27.581265+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:04.456565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.783425+0000","last_change":"2023-10-16T15:34:26.605397+0000","last_active":"2023-10-16T15:34:27.783425+0000","last_peered":"2023-10-16T15:34:27.783425+0000","last_clean":"2023-10-16T15:34:27.783425+0000","last_became_active":"2023-10-16T15:34:26.605129+0000","last_became_peered":"2023-10-16T15:34:26.605129+0000","last_unstale":"2023-10-16T15:34:27.783425+0000","last_undegraded":"2023-10-16T15:34:27.783425+0000","last_fullsized":"2023-10-16T15:34:27.783425+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:13.909012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576782+0000","last_change":"2023-10-16T15:34:26.591362+0000","last_active":"2023-10-16T15:34:27.576782+0000","last_peered":"2023-10-16T15:34:27.576782+0000","last_clean":"2023-10-16T15:34:27.576782+0000","last_became_active":"2023-10-16T15:34:26.590704+0000","last_became_peered":"2023-10-16T15:34:26.590704+0000","last_unstale":"2023-10-16T15:34:27.576782+0000","last_undegraded":"2023-10-16T15:34:27.576782+0000","last_fullsized":"2023-10-16T15:34:27.576782+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:34:42.924882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"191'4","reported_seq":16,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580739+0000","last_change":"2023-10-16T15:34:25.576361+0000","last_active":"2023-10-16T15:34:27.580739+0000","last_peered":"2023-10-16T15:34:27.580739+0000","last_clean":"2023-10-16T15:34:27.580739+0000","last_became_active":"2023-10-16T15:34:25.576037+0000","last_became_peered":"2023-10-16T15:34:25.576037+0000","last_unstale":"2023-10-16T15:34:27.580739+0000","last_undegraded":"2023-10-16T15:34:27.580739+0000","last_fullsized":"2023-10-16T15:34:27.580739+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:03:24.352960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582403+0000","last_change":"2023-10-16T15:34:26.596538+0000","last_active":"2023-10-16T15:34:27.582403+0000","last_peered":"2023-10-16T15:34:27.582403+0000","last_clean":"2023-10-16T15:34:27.582403+0000","last_became_active":"2023-10-16T15:34:26.593505+0000","last_became_peered":"2023-10-16T15:34:26.593505+0000","last_unstale":"2023-10-16T15:34:27.582403+0000","last_undegraded":"2023-10-16T15:34:27.582403+0000","last_fullsized":"2023-10-16T15:34:27.582403+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:46.935125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582038+0000","last_change":"2023-10-16T15:34:25.581803+0000","last_active":"2023-10-16T15:34:27.582038+0000","last_peered":"2023-10-16T15:34:27.582038+0000","last_clean":"2023-10-16T15:34:27.582038+0000","last_became_active":"2023-10-16T15:34:25.581390+0000","last_became_peered":"2023-10-16T15:34:25.581390+0000","last_unstale":"2023-10-16T15:34:27.582038+0000","last_undegraded":"2023-10-16T15:34:27.582038+0000","last_fullsized":"2023-10-16T15:34:27.582038+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:06.834486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787224+0000","last_change":"2023-10-16T15:34:26.596019+0000","last_active":"2023-10-16T15:34:27.787224+0000","last_peered":"2023-10-16T15:34:27.787224+0000","last_clean":"2023-10-16T15:34:27.787224+0000","last_became_active":"2023-10-16T15:34:26.595724+0000","last_became_peered":"2023-10-16T15:34:26.595724+0000","last_unstale":"2023-10-16T15:34:27.787224+0000","last_undegraded":"2023-10-16T15:34:27.787224+0000","last_fullsized":"2023-10-16T15:34:27.787224+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:52.166652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786299+0000","last_change":"2023-10-16T15:34:25.581693+0000","last_active":"2023-10-16T15:34:27.786299+0000","last_peered":"2023-10-16T15:34:27.786299+0000","last_clean":"2023-10-16T15:34:27.786299+0000","last_became_active":"2023-10-16T15:34:25.580966+0000","last_became_peered":"2023-10-16T15:34:25.580966+0000","last_unstale":"2023-10-16T15:34:27.786299+0000","last_undegraded":"2023-10-16T15:34:27.786299+0000","last_fullsized":"2023-10-16T15:34:27.786299+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:16.172910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788169+0000","last_change":"2023-10-16T15:34:26.611117+0000","last_active":"2023-10-16T15:34:27.788169+0000","last_peered":"2023-10-16T15:34:27.788169+0000","last_clean":"2023-10-16T15:34:27.788169+0000","last_became_active":"2023-10-16T15:34:26.610674+0000","last_became_peered":"2023-10-16T15:34:26.610674+0000","last_unstale":"2023-10-16T15:34:27.788169+0000","last_undegraded":"2023-10-16T15:34:27.788169+0000","last_fullsized":"2023-10-16T15:34:27.788169+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:20.315749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787470+0000","last_change":"2023-10-16T15:34:25.575595+0000","last_active":"2023-10-16T15:34:27.787470+0000","last_peered":"2023-10-16T15:34:27.787470+0000","last_clean":"2023-10-16T15:34:27.787470+0000","last_became_active":"2023-10-16T15:34:25.574665+0000","last_became_peered":"2023-10-16T15:34:25.574665+0000","last_unstale":"2023-10-16T15:34:27.787470+0000","last_undegraded":"2023-10-16T15:34:27.787470+0000","last_fullsized":"2023-10-16T15:34:27.787470+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:43.630060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576313+0000","last_change":"2023-10-16T15:34:26.586892+0000","last_active":"2023-10-16T15:34:27.576313+0000","last_peered":"2023-10-16T15:34:27.576313+0000","last_clean":"2023-10-16T15:34:27.576313+0000","last_became_active":"2023-10-16T15:34:26.586542+0000","last_became_peered":"2023-10-16T15:34:26.586542+0000","last_unstale":"2023-10-16T15:34:27.576313+0000","last_undegraded":"2023-10-16T15:34:27.576313+0000","last_fullsized":"2023-10-16T15:34:27.576313+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:58.778256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787868+0000","last_change":"2023-10-16T15:34:25.577870+0000","last_active":"2023-10-16T15:34:27.787868+0000","last_peered":"2023-10-16T15:34:27.787868+0000","last_clean":"2023-10-16T15:34:27.787868+0000","last_became_active":"2023-10-16T15:34:25.577585+0000","last_became_peered":"2023-10-16T15:34:25.577585+0000","last_unstale":"2023-10-16T15:34:27.787868+0000","last_undegraded":"2023-10-16T15:34:27.787868+0000","last_fullsized":"2023-10-16T15:34:27.787868+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:15.181577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786110+0000","last_change":"2023-10-16T15:34:26.599145+0000","last_active":"2023-10-16T15:34:27.786110+0000","last_peered":"2023-10-16T15:34:27.786110+0000","last_clean":"2023-10-16T15:34:27.786110+0000","last_became_active":"2023-10-16T15:34:26.598654+0000","last_became_peered":"2023-10-16T15:34:26.598654+0000","last_unstale":"2023-10-16T15:34:27.786110+0000","last_undegraded":"2023-10-16T15:34:27.786110+0000","last_fullsized":"2023-10-16T15:34:27.786110+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:07.850011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786487+0000","last_change":"2023-10-16T15:34:25.580915+0000","last_active":"2023-10-16T15:34:27.786487+0000","last_peered":"2023-10-16T15:34:27.786487+0000","last_clean":"2023-10-16T15:34:27.786487+0000","last_became_active":"2023-10-16T15:34:25.580623+0000","last_became_peered":"2023-10-16T15:34:25.580623+0000","last_unstale":"2023-10-16T15:34:27.786487+0000","last_undegraded":"2023-10-16T15:34:27.786487+0000","last_fullsized":"2023-10-16T15:34:27.786487+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:11.660719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785163+0000","last_change":"2023-10-16T15:34:26.605735+0000","last_active":"2023-10-16T15:34:27.785163+0000","last_peered":"2023-10-16T15:34:27.785163+0000","last_clean":"2023-10-16T15:34:27.785163+0000","last_became_active":"2023-10-16T15:34:26.605465+0000","last_became_peered":"2023-10-16T15:34:26.605465+0000","last_unstale":"2023-10-16T15:34:27.785163+0000","last_undegraded":"2023-10-16T15:34:27.785163+0000","last_fullsized":"2023-10-16T15:34:27.785163+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:28.182594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785208+0000","last_change":"2023-10-16T15:34:25.581000+0000","last_active":"2023-10-16T15:34:27.785208+0000","last_peered":"2023-10-16T15:34:27.785208+0000","last_clean":"2023-10-16T15:34:27.785208+0000","last_became_active":"2023-10-16T15:34:25.580661+0000","last_became_peered":"2023-10-16T15:34:25.580661+0000","last_unstale":"2023-10-16T15:34:27.785208+0000","last_undegraded":"2023-10-16T15:34:27.785208+0000","last_fullsized":"2023-10-16T15:34:27.785208+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:00.524622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786111+0000","last_change":"2023-10-16T15:34:26.596019+0000","last_active":"2023-10-16T15:34:27.786111+0000","last_peered":"2023-10-16T15:34:27.786111+0000","last_clean":"2023-10-16T15:34:27.786111+0000","last_became_active":"2023-10-16T15:34:26.595707+0000","last_became_peered":"2023-10-16T15:34:26.595707+0000","last_unstale":"2023-10-16T15:34:27.786111+0000","last_undegraded":"2023-10-16T15:34:27.786111+0000","last_fullsized":"2023-10-16T15:34:27.786111+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:38.684237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.575989+0000","last_change":"2023-10-16T15:34:25.575196+0000","last_active":"2023-10-16T15:34:27.575989+0000","last_peered":"2023-10-16T15:34:27.575989+0000","last_clean":"2023-10-16T15:34:27.575989+0000","last_became_active":"2023-10-16T15:34:25.574926+0000","last_became_peered":"2023-10-16T15:34:25.574926+0000","last_unstale":"2023-10-16T15:34:27.575989+0000","last_undegraded":"2023-10-16T15:34:27.575989+0000","last_fullsized":"2023-10-16T15:34:27.575989+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:08.627856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784746+0000","last_change":"2023-10-16T15:34:26.608119+0000","last_active":"2023-10-16T15:34:27.784746+0000","last_peered":"2023-10-16T15:34:27.784746+0000","last_clean":"2023-10-16T15:34:27.784746+0000","last_became_active":"2023-10-16T15:34:26.607830+0000","last_became_peered":"2023-10-16T15:34:26.607830+0000","last_unstale":"2023-10-16T15:34:27.784746+0000","last_undegraded":"2023-10-16T15:34:27.784746+0000","last_fullsized":"2023-10-16T15:34:27.784746+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:54.493055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787685+0000","last_change":"2023-10-16T15:34:25.577417+0000","last_active":"2023-10-16T15:34:27.787685+0000","last_peered":"2023-10-16T15:34:27.787685+0000","last_clean":"2023-10-16T15:34:27.787685+0000","last_became_active":"2023-10-16T15:34:25.575680+0000","last_became_peered":"2023-10-16T15:34:25.575680+0000","last_unstale":"2023-10-16T15:34:27.787685+0000","last_undegraded":"2023-10-16T15:34:27.787685+0000","last_fullsized":"2023-10-16T15:34:27.787685+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:14.823512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784272+0000","last_change":"2023-10-16T15:34:26.597698+0000","last_active":"2023-10-16T15:34:27.784272+0000","last_peered":"2023-10-16T15:34:27.784272+0000","last_clean":"2023-10-16T15:34:27.784272+0000","last_became_active":"2023-10-16T15:34:26.597341+0000","last_became_peered":"2023-10-16T15:34:26.597341+0000","last_unstale":"2023-10-16T15:34:27.784272+0000","last_undegraded":"2023-10-16T15:34:27.784272+0000","last_fullsized":"2023-10-16T15:34:27.784272+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:45.778251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"191'3","reported_seq":15,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787045+0000","last_change":"2023-10-16T15:34:25.574979+0000","last_active":"2023-10-16T15:34:27.787045+0000","last_peered":"2023-10-16T15:34:27.787045+0000","last_clean":"2023-10-16T15:34:27.787045+0000","last_became_active":"2023-10-16T15:34:25.574471+0000","last_became_peered":"2023-10-16T15:34:25.574471+0000","last_unstale":"2023-10-16T15:34:27.787045+0000","last_undegraded":"2023-10-16T15:34:27.787045+0000","last_fullsized":"2023-10-16T15:34:27.787045+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:18.710219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582232+0000","last_change":"2023-10-16T15:34:26.613954+0000","last_active":"2023-10-16T15:34:27.582232+0000","last_peered":"2023-10-16T15:34:27.582232+0000","last_clean":"2023-10-16T15:34:27.582232+0000","last_became_active":"2023-10-16T15:34:26.612933+0000","last_became_peered":"2023-10-16T15:34:26.612933+0000","last_unstale":"2023-10-16T15:34:27.582232+0000","last_undegraded":"2023-10-16T15:34:27.582232+0000","last_fullsized":"2023-10-16T15:34:27.582232+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:26.456602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"191'6","reported_seq":18,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581710+0000","last_change":"2023-10-16T15:34:25.573020+0000","last_active":"2023-10-16T15:34:27.581710+0000","last_peered":"2023-10-16T15:34:27.581710+0000","last_clean":"2023-10-16T15:34:27.581710+0000","last_became_active":"2023-10-16T15:34:25.572716+0000","last_became_peered":"2023-10-16T15:34:25.572716+0000","last_unstale":"2023-10-16T15:34:27.581710+0000","last_undegraded":"2023-10-16T15:34:27.581710+0000","last_fullsized":"2023-10-16T15:34:27.581710+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:10.789872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784216+0000","last_change":"2023-10-16T15:34:26.603626+0000","last_active":"2023-10-16T15:34:27.784216+0000","last_peered":"2023-10-16T15:34:27.784216+0000","last_clean":"2023-10-16T15:34:27.784216+0000","last_became_active":"2023-10-16T15:34:26.603327+0000","last_became_peered":"2023-10-16T15:34:26.603327+0000","last_unstale":"2023-10-16T15:34:27.784216+0000","last_undegraded":"2023-10-16T15:34:27.784216+0000","last_fullsized":"2023-10-16T15:34:27.784216+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:27.364529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786356+0000","last_change":"2023-10-16T15:34:25.583201+0000","last_active":"2023-10-16T15:34:27.786356+0000","last_peered":"2023-10-16T15:34:27.786356+0000","last_clean":"2023-10-16T15:34:27.786356+0000","last_became_active":"2023-10-16T15:34:25.582902+0000","last_became_peered":"2023-10-16T15:34:25.582902+0000","last_unstale":"2023-10-16T15:34:27.786356+0000","last_undegraded":"2023-10-16T15:34:27.786356+0000","last_fullsized":"2023-10-16T15:34:27.786356+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:05.367800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576866+0000","last_change":"2023-10-16T15:34:26.614735+0000","last_active":"2023-10-16T15:34:27.576866+0000","last_peered":"2023-10-16T15:34:27.576866+0000","last_clean":"2023-10-16T15:34:27.576866+0000","last_became_active":"2023-10-16T15:34:26.614392+0000","last_became_peered":"2023-10-16T15:34:26.614392+0000","last_unstale":"2023-10-16T15:34:27.576866+0000","last_undegraded":"2023-10-16T15:34:27.576866+0000","last_fullsized":"2023-10-16T15:34:27.576866+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:51.630786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.783946+0000","last_change":"2023-10-16T15:34:25.572332+0000","last_active":"2023-10-16T15:34:27.783946+0000","last_peered":"2023-10-16T15:34:27.783946+0000","last_clean":"2023-10-16T15:34:27.783946+0000","last_became_active":"2023-10-16T15:34:25.572012+0000","last_became_peered":"2023-10-16T15:34:25.572012+0000","last_unstale":"2023-10-16T15:34:27.783946+0000","last_undegraded":"2023-10-16T15:34:27.783946+0000","last_fullsized":"2023-10-16T15:34:27.783946+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:12.957176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785086+0000","last_change":"2023-10-16T15:34:26.605192+0000","last_active":"2023-10-16T15:34:27.785086+0000","last_peered":"2023-10-16T15:34:27.785086+0000","last_clean":"2023-10-16T15:34:27.785086+0000","last_became_active":"2023-10-16T15:34:26.604934+0000","last_became_peered":"2023-10-16T15:34:26.604934+0000","last_unstale":"2023-10-16T15:34:27.785086+0000","last_undegraded":"2023-10-16T15:34:27.785086+0000","last_fullsized":"2023-10-16T15:34:27.785086+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.171786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785450+0000","last_change":"2023-10-16T15:34:25.582542+0000","last_active":"2023-10-16T15:34:27.785450+0000","last_peered":"2023-10-16T15:34:27.785450+0000","last_clean":"2023-10-16T15:34:27.785450+0000","last_became_active":"2023-10-16T15:34:25.581999+0000","last_became_peered":"2023-10-16T15:34:25.581999+0000","last_unstale":"2023-10-16T15:34:27.785450+0000","last_undegraded":"2023-10-16T15:34:27.785450+0000","last_fullsized":"2023-10-16T15:34:27.785450+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786878+0000","last_change":"2023-10-16T15:34:26.617167+0000","last_active":"2023-10-16T15:34:27.786878+0000","last_peered":"2023-10-16T15:34:27.786878+0000","last_clean":"2023-10-16T15:34:27.786878+0000","last_became_active":"2023-10-16T15:34:26.611654+0000","last_became_peered":"2023-10-16T15:34:26.611654+0000","last_unstale":"2023-10-16T15:34:27.786878+0000","last_undegraded":"2023-10-16T15:34:27.786878+0000","last_fullsized":"2023-10-16T15:34:27.786878+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:53.180390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"191'8","reported_seq":20,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580659+0000","last_change":"2023-10-16T15:34:25.569283+0000","last_active":"2023-10-16T15:34:27.580659+0000","last_peered":"2023-10-16T15:34:27.580659+0000","last_clean":"2023-10-16T15:34:27.580659+0000","last_became_active":"2023-10-16T15:34:25.568974+0000","last_became_peered":"2023-10-16T15:34:25.568974+0000","last_unstale":"2023-10-16T15:34:27.580659+0000","last_undegraded":"2023-10-16T15:34:27.580659+0000","last_fullsized":"2023-10-16T15:34:27.580659+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:16.156261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787495+0000","last_change":"2023-10-16T15:34:26.600619+0000","last_active":"2023-10-16T15:34:27.787495+0000","last_peered":"2023-10-16T15:34:27.787495+0000","last_clean":"2023-10-16T15:34:27.787495+0000","last_became_active":"2023-10-16T15:34:26.600320+0000","last_became_peered":"2023-10-16T15:34:26.600320+0000","last_unstale":"2023-10-16T15:34:27.787495+0000","last_undegraded":"2023-10-16T15:34:27.787495+0000","last_fullsized":"2023-10-16T15:34:27.787495+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:24:15.426000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788488+0000","last_change":"2023-10-16T15:34:25.582687+0000","last_active":"2023-10-16T15:34:27.788488+0000","last_peered":"2023-10-16T15:34:27.788488+0000","last_clean":"2023-10-16T15:34:27.788488+0000","last_became_active":"2023-10-16T15:34:25.582392+0000","last_became_peered":"2023-10-16T15:34:25.582392+0000","last_unstale":"2023-10-16T15:34:27.788488+0000","last_undegraded":"2023-10-16T15:34:27.788488+0000","last_fullsized":"2023-10-16T15:34:27.788488+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:08.056990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785402+0000","last_change":"2023-10-16T15:34:26.599860+0000","last_active":"2023-10-16T15:34:27.785402+0000","last_peered":"2023-10-16T15:34:27.785402+0000","last_clean":"2023-10-16T15:34:27.785402+0000","last_became_active":"2023-10-16T15:34:26.599375+0000","last_became_peered":"2023-10-16T15:34:26.599375+0000","last_unstale":"2023-10-16T15:34:27.785402+0000","last_undegraded":"2023-10-16T15:34:27.785402+0000","last_fullsized":"2023-10-16T15:34:27.785402+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:58.623926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787787+0000","last_change":"2023-10-16T15:34:25.579171+0000","last_active":"2023-10-16T15:34:27.787787+0000","last_peered":"2023-10-16T15:34:27.787787+0000","last_clean":"2023-10-16T15:34:27.787787+0000","last_became_active":"2023-10-16T15:34:25.578533+0000","last_became_peered":"2023-10-16T15:34:25.578533+0000","last_unstale":"2023-10-16T15:34:27.787787+0000","last_undegraded":"2023-10-16T15:34:27.787787+0000","last_fullsized":"2023-10-16T15:34:27.787787+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:45.029593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787448+0000","last_change":"2023-10-16T15:34:26.598573+0000","last_active":"2023-10-16T15:34:27.787448+0000","last_peered":"2023-10-16T15:34:27.787448+0000","last_clean":"2023-10-16T15:34:27.787448+0000","last_became_active":"2023-10-16T15:34:26.596504+0000","last_became_peered":"2023-10-16T15:34:26.596504+0000","last_unstale":"2023-10-16T15:34:27.787448+0000","last_undegraded":"2023-10-16T15:34:27.787448+0000","last_fullsized":"2023-10-16T15:34:27.787448+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:46.553519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.579413+0000","last_change":"2023-10-16T15:34:25.572405+0000","last_active":"2023-10-16T15:34:27.579413+0000","last_peered":"2023-10-16T15:34:27.579413+0000","last_clean":"2023-10-16T15:34:27.579413+0000","last_became_active":"2023-10-16T15:34:25.572069+0000","last_became_peered":"2023-10-16T15:34:25.572069+0000","last_unstale":"2023-10-16T15:34:27.579413+0000","last_undegraded":"2023-10-16T15:34:27.579413+0000","last_fullsized":"2023-10-16T15:34:27.579413+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:19.823037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576519+0000","last_change":"2023-10-16T15:34:26.593722+0000","last_active":"2023-10-16T15:34:27.576519+0000","last_peered":"2023-10-16T15:34:27.576519+0000","last_clean":"2023-10-16T15:34:27.576519+0000","last_became_active":"2023-10-16T15:34:26.591933+0000","last_became_peered":"2023-10-16T15:34:26.591933+0000","last_unstale":"2023-10-16T15:34:27.576519+0000","last_undegraded":"2023-10-16T15:34:27.576519+0000","last_fullsized":"2023-10-16T15:34:27.576519+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:43.548094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580783+0000","last_change":"2023-10-16T15:34:25.582452+0000","last_active":"2023-10-16T15:34:27.580783+0000","last_peered":"2023-10-16T15:34:27.580783+0000","last_clean":"2023-10-16T15:34:27.580783+0000","last_became_active":"2023-10-16T15:34:25.581439+0000","last_became_peered":"2023-10-16T15:34:25.581439+0000","last_unstale":"2023-10-16T15:34:27.580783+0000","last_undegraded":"2023-10-16T15:34:27.580783+0000","last_fullsized":"2023-10-16T15:34:27.580783+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:23.151449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786909+0000","last_change":"2023-10-16T15:34:26.620200+0000","last_active":"2023-10-16T15:34:27.786909+0000","last_peered":"2023-10-16T15:34:27.786909+0000","last_clean":"2023-10-16T15:34:27.786909+0000","last_became_active":"2023-10-16T15:34:26.619592+0000","last_became_peered":"2023-10-16T15:34:26.619592+0000","last_unstale":"2023-10-16T15:34:27.786909+0000","last_undegraded":"2023-10-16T15:34:27.786909+0000","last_fullsized":"2023-10-16T15:34:27.786909+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:29.156108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786697+0000","last_change":"2023-10-16T15:34:25.579748+0000","last_active":"2023-10-16T15:34:27.786697+0000","last_peered":"2023-10-16T15:34:27.786697+0000","last_clean":"2023-10-16T15:34:27.786697+0000","last_became_active":"2023-10-16T15:34:25.579275+0000","last_became_peered":"2023-10-16T15:34:25.579275+0000","last_unstale":"2023-10-16T15:34:27.786697+0000","last_undegraded":"2023-10-16T15:34:27.786697+0000","last_fullsized":"2023-10-16T15:34:27.786697+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:41.474966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576883+0000","last_change":"2023-10-16T15:34:26.591413+0000","last_active":"2023-10-16T15:34:27.576883+0000","last_peered":"2023-10-16T15:34:27.576883+0000","last_clean":"2023-10-16T15:34:27.576883+0000","last_became_active":"2023-10-16T15:34:26.591155+0000","last_became_peered":"2023-10-16T15:34:26.591155+0000","last_unstale":"2023-10-16T15:34:27.576883+0000","last_undegraded":"2023-10-16T15:34:27.576883+0000","last_fullsized":"2023-10-16T15:34:27.576883+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:05.573570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.578866+0000","last_change":"2023-10-16T15:34:25.575452+0000","last_active":"2023-10-16T15:34:27.578866+0000","last_peered":"2023-10-16T15:34:27.578866+0000","last_clean":"2023-10-16T15:34:27.578866+0000","last_became_active":"2023-10-16T15:34:25.575154+0000","last_became_peered":"2023-10-16T15:34:25.575154+0000","last_unstale":"2023-10-16T15:34:27.578866+0000","last_undegraded":"2023-10-16T15:34:27.578866+0000","last_fullsized":"2023-10-16T15:34:27.578866+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:59.950670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.579100+0000","last_change":"2023-10-16T15:34:26.609750+0000","last_active":"2023-10-16T15:34:27.579100+0000","last_peered":"2023-10-16T15:34:27.579100+0000","last_clean":"2023-10-16T15:34:27.579100+0000","last_became_active":"2023-10-16T15:34:26.607638+0000","last_became_peered":"2023-10-16T15:34:26.607638+0000","last_unstale":"2023-10-16T15:34:27.579100+0000","last_undegraded":"2023-10-16T15:34:27.579100+0000","last_fullsized":"2023-10-16T15:34:27.579100+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:29.454834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785548+0000","last_change":"2023-10-16T15:34:25.584285+0000","last_active":"2023-10-16T15:34:27.785548+0000","last_peered":"2023-10-16T15:34:27.785548+0000","last_clean":"2023-10-16T15:34:27.785548+0000","last_became_active":"2023-10-16T15:34:25.583887+0000","last_became_peered":"2023-10-16T15:34:25.583887+0000","last_unstale":"2023-10-16T15:34:27.785548+0000","last_undegraded":"2023-10-16T15:34:27.785548+0000","last_fullsized":"2023-10-16T15:34:27.785548+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:06.125994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785978+0000","last_change":"2023-10-16T15:34:26.602762+0000","last_active":"2023-10-16T15:34:27.785978+0000","last_peered":"2023-10-16T15:34:27.785978+0000","last_clean":"2023-10-16T15:34:27.785978+0000","last_became_active":"2023-10-16T15:34:26.602467+0000","last_became_peered":"2023-10-16T15:34:26.602467+0000","last_unstale":"2023-10-16T15:34:27.785978+0000","last_undegraded":"2023-10-16T15:34:27.785978+0000","last_fullsized":"2023-10-16T15:34:27.785978+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:45.821024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.579297+0000","last_change":"2023-10-16T15:34:25.572530+0000","last_active":"2023-10-16T15:34:27.579297+0000","last_peered":"2023-10-16T15:34:27.579297+0000","last_clean":"2023-10-16T15:34:27.579297+0000","last_became_active":"2023-10-16T15:34:25.571988+0000","last_became_peered":"2023-10-16T15:34:25.571988+0000","last_unstale":"2023-10-16T15:34:27.579297+0000","last_undegraded":"2023-10-16T15:34:27.579297+0000","last_fullsized":"2023-10-16T15:34:27.579297+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:58.217428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785188+0000","last_change":"2023-10-16T15:34:26.608941+0000","last_active":"2023-10-16T15:34:27.785188+0000","last_peered":"2023-10-16T15:34:27.785188+0000","last_clean":"2023-10-16T15:34:27.785188+0000","last_became_active":"2023-10-16T15:34:26.608457+0000","last_became_peered":"2023-10-16T15:34:26.608457+0000","last_unstale":"2023-10-16T15:34:27.785188+0000","last_undegraded":"2023-10-16T15:34:27.785188+0000","last_fullsized":"2023-10-16T15:34:27.785188+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:37.869891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786479+0000","last_change":"2023-10-16T15:34:25.583233+0000","last_active":"2023-10-16T15:34:27.786479+0000","last_peered":"2023-10-16T15:34:27.786479+0000","last_clean":"2023-10-16T15:34:27.786479+0000","last_became_active":"2023-10-16T15:34:25.582900+0000","last_became_peered":"2023-10-16T15:34:25.582900+0000","last_unstale":"2023-10-16T15:34:27.786479+0000","last_undegraded":"2023-10-16T15:34:27.786479+0000","last_fullsized":"2023-10-16T15:34:27.786479+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:59.585980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.576871+0000","last_change":"2023-10-16T15:34:26.603059+0000","last_active":"2023-10-16T15:34:27.576871+0000","last_peered":"2023-10-16T15:34:27.576871+0000","last_clean":"2023-10-16T15:34:27.576871+0000","last_became_active":"2023-10-16T15:34:26.602776+0000","last_became_peered":"2023-10-16T15:34:26.602776+0000","last_unstale":"2023-10-16T15:34:27.576871+0000","last_undegraded":"2023-10-16T15:34:27.576871+0000","last_fullsized":"2023-10-16T15:34:27.576871+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:19.762004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.788156+0000","last_change":"2023-10-16T15:34:25.579357+0000","last_active":"2023-10-16T15:34:27.788156+0000","last_peered":"2023-10-16T15:34:27.788156+0000","last_clean":"2023-10-16T15:34:27.788156+0000","last_became_active":"2023-10-16T15:34:25.578914+0000","last_became_peered":"2023-10-16T15:34:25.578914+0000","last_unstale":"2023-10-16T15:34:27.788156+0000","last_undegraded":"2023-10-16T15:34:27.788156+0000","last_fullsized":"2023-10-16T15:34:27.788156+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:06.784535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.787357+0000","last_change":"2023-10-16T15:34:26.601817+0000","last_active":"2023-10-16T15:34:27.787357+0000","last_peered":"2023-10-16T15:34:27.787357+0000","last_clean":"2023-10-16T15:34:27.787357+0000","last_became_active":"2023-10-16T15:34:26.600054+0000","last_became_peered":"2023-10-16T15:34:26.600054+0000","last_unstale":"2023-10-16T15:34:27.787357+0000","last_undegraded":"2023-10-16T15:34:27.787357+0000","last_fullsized":"2023-10-16T15:34:27.787357+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:13.873261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786171+0000","last_change":"2023-10-16T15:34:25.582691+0000","last_active":"2023-10-16T15:34:27.786171+0000","last_peered":"2023-10-16T15:34:27.786171+0000","last_clean":"2023-10-16T15:34:27.786171+0000","last_became_active":"2023-10-16T15:34:25.582042+0000","last_became_peered":"2023-10-16T15:34:25.582042+0000","last_unstale":"2023-10-16T15:34:27.786171+0000","last_undegraded":"2023-10-16T15:34:27.786171+0000","last_fullsized":"2023-10-16T15:34:27.786171+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:12.328978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784629+0000","last_change":"2023-10-16T15:34:26.590931+0000","last_active":"2023-10-16T15:34:27.784629+0000","last_peered":"2023-10-16T15:34:27.784629+0000","last_clean":"2023-10-16T15:34:27.784629+0000","last_became_active":"2023-10-16T15:34:26.590138+0000","last_became_peered":"2023-10-16T15:34:26.590138+0000","last_unstale":"2023-10-16T15:34:27.784629+0000","last_undegraded":"2023-10-16T15:34:27.784629+0000","last_fullsized":"2023-10-16T15:34:27.784629+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:24.346565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.784309+0000","last_change":"2023-10-16T15:34:25.583239+0000","last_active":"2023-10-16T15:34:27.784309+0000","last_peered":"2023-10-16T15:34:27.784309+0000","last_clean":"2023-10-16T15:34:27.784309+0000","last_became_active":"2023-10-16T15:34:25.582754+0000","last_became_peered":"2023-10-16T15:34:25.582754+0000","last_unstale":"2023-10-16T15:34:27.784309+0000","last_undegraded":"2023-10-16T15:34:27.784309+0000","last_fullsized":"2023-10-16T15:34:27.784309+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:29.051118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.783850+0000","last_change":"2023-10-16T15:34:26.600386+0000","last_active":"2023-10-16T15:34:27.783850+0000","last_peered":"2023-10-16T15:34:27.783850+0000","last_clean":"2023-10-16T15:34:27.783850+0000","last_became_active":"2023-10-16T15:34:26.600111+0000","last_became_peered":"2023-10-16T15:34:26.600111+0000","last_unstale":"2023-10-16T15:34:27.783850+0000","last_undegraded":"2023-10-16T15:34:27.783850+0000","last_fullsized":"2023-10-16T15:34:27.783850+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:51.671018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581581+0000","last_change":"2023-10-16T15:34:25.574712+0000","last_active":"2023-10-16T15:34:27.581581+0000","last_peered":"2023-10-16T15:34:27.581581+0000","last_clean":"2023-10-16T15:34:27.581581+0000","last_became_active":"2023-10-16T15:34:25.574385+0000","last_became_peered":"2023-10-16T15:34:25.574385+0000","last_unstale":"2023-10-16T15:34:27.581581+0000","last_undegraded":"2023-10-16T15:34:27.581581+0000","last_fullsized":"2023-10-16T15:34:27.581581+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:54.811954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.581692+0000","last_change":"2023-10-16T15:34:26.604076+0000","last_active":"2023-10-16T15:34:27.581692+0000","last_peered":"2023-10-16T15:34:27.581692+0000","last_clean":"2023-10-16T15:34:27.581692+0000","last_became_active":"2023-10-16T15:34:26.603160+0000","last_became_peered":"2023-10-16T15:34:26.603160+0000","last_unstale":"2023-10-16T15:34:27.581692+0000","last_undegraded":"2023-10-16T15:34:27.581692+0000","last_fullsized":"2023-10-16T15:34:27.581692+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:10.319066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786595+0000","last_change":"2023-10-16T15:34:25.577691+0000","last_active":"2023-10-16T15:34:27.786595+0000","last_peered":"2023-10-16T15:34:27.786595+0000","last_clean":"2023-10-16T15:34:27.786595+0000","last_became_active":"2023-10-16T15:34:25.577412+0000","last_became_peered":"2023-10-16T15:34:25.577412+0000","last_unstale":"2023-10-16T15:34:27.786595+0000","last_undegraded":"2023-10-16T15:34:27.786595+0000","last_fullsized":"2023-10-16T15:34:27.786595+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:01.365525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.575897+0000","last_change":"2023-10-16T15:34:26.586849+0000","last_active":"2023-10-16T15:34:27.575897+0000","last_peered":"2023-10-16T15:34:27.575897+0000","last_clean":"2023-10-16T15:34:27.575897+0000","last_became_active":"2023-10-16T15:34:26.586154+0000","last_became_peered":"2023-10-16T15:34:26.586154+0000","last_unstale":"2023-10-16T15:34:27.575897+0000","last_undegraded":"2023-10-16T15:34:27.575897+0000","last_fullsized":"2023-10-16T15:34:27.575897+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:37.947607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785391+0000","last_change":"2023-10-16T15:34:25.584579+0000","last_active":"2023-10-16T15:34:27.785391+0000","last_peered":"2023-10-16T15:34:27.785391+0000","last_clean":"2023-10-16T15:34:27.785391+0000","last_became_active":"2023-10-16T15:34:25.584090+0000","last_became_peered":"2023-10-16T15:34:25.584090+0000","last_unstale":"2023-10-16T15:34:27.785391+0000","last_undegraded":"2023-10-16T15:34:27.785391+0000","last_fullsized":"2023-10-16T15:34:27.785391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:38.002395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.786811+0000","last_change":"2023-10-16T15:34:26.611760+0000","last_active":"2023-10-16T15:34:27.786811+0000","last_peered":"2023-10-16T15:34:27.786811+0000","last_clean":"2023-10-16T15:34:27.786811+0000","last_became_active":"2023-10-16T15:34:26.611452+0000","last_became_peered":"2023-10-16T15:34:26.611452+0000","last_unstale":"2023-10-16T15:34:27.786811+0000","last_undegraded":"2023-10-16T15:34:27.786811+0000","last_fullsized":"2023-10-16T15:34:27.786811+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:20.169116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.580630+0000","last_change":"2023-10-16T15:34:25.566911+0000","last_active":"2023-10-16T15:34:27.580630+0000","last_peered":"2023-10-16T15:34:27.580630+0000","last_clean":"2023-10-16T15:34:27.580630+0000","last_became_active":"2023-10-16T15:34:25.566596+0000","last_became_peered":"2023-10-16T15:34:25.566596+0000","last_unstale":"2023-10-16T15:34:27.580630+0000","last_undegraded":"2023-10-16T15:34:27.580630+0000","last_fullsized":"2023-10-16T15:34:27.580630+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:27.167069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.785667+0000","last_change":"2023-10-16T15:34:26.609121+0000","last_active":"2023-10-16T15:34:27.785667+0000","last_peered":"2023-10-16T15:34:27.785667+0000","last_clean":"2023-10-16T15:34:27.785667+0000","last_became_active":"2023-10-16T15:34:26.608869+0000","last_became_peered":"2023-10-16T15:34:26.608869+0000","last_unstale":"2023-10-16T15:34:27.785667+0000","last_undegraded":"2023-10-16T15:34:27.785667+0000","last_fullsized":"2023-10-16T15:34:27.785667+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:25.535505+0000","last_clean_scrub_stamp":"2023-10-16T15:34:25.535505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:22.317917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-16T15:34:27.582548+0000","last_change":"2023-10-16T15:34:25.577730+0000","last_active":"2023-10-16T15:34:27.582548+0000","last_peered":"2023-10-16T15:34:27.582548+0000","last_clean":"2023-10-16T15:34:27.582548+0000","last_became_active":"2023-10-16T15:34:25.576880+0000","last_became_peered":"2023-10-16T15:34:25.576880+0000","last_unstale":"2023-10-16T15:34:27.582548+0000","last_undegraded":"2023-10-16T15:34:27.582548+0000","last_fullsized":"2023-10-16T15:34:27.582548+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:34:24.519981+0000","last_clean_scrub_stamp":"2023-10-16T15:34:24.519981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:45.537995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"181'296","reported_seq":1009,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-16T15:34:26.582788+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:34:26.582788+0000","last_peered":"2023-10-16T15:34:26.582788+0000","last_clean":"2023-10-16T15:34:26.582788+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:34:26.582788+0000","last_undegraded":"2023-10-16T15:34:26.582788+0000","last_fullsized":"2023-10-16T15:34:26.582788+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575779,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56340,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315500,"statfs":{"total":96636764160,"available":96579072000,"internally_reserved":0,"allocated":3018752,"data_stored":1762707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1799999999999999,"15min":1.206},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.6990000000000001,"5min":1.752,"15min":1.7849999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.071,"5min":1.1419999999999999,"15min":1.1599999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.56999999999999995},"max":{"1min":1.4179999999999999,"5min":1.867,"15min":1.905},"last":1.087}]},{"osd":1,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.8480000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.252},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.339,"15min":1.3360000000000001},"min":{"1min":0.94099999999999995,"5min":0.91200000000000003,"15min":0.78200000000000003},"max":{"1min":1.583,"5min":2.282,"15min":2.282},"last":1.454}]},{"osd":2,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.282,"15min":1.3460000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6619999999999999,"5min":1.7549999999999999,"15min":1.9039999999999999},"last":1.323},{"interface":"front","average":{"1min":1.27,"5min":1.3440000000000001,"15min":1.3240000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.82599999999999996},"max":{"1min":2.081,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.397}]},{"osd":3,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3580000000000001,"15min":1.345},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.6599999999999999,"5min":2.528,"15min":2.528},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.452,"15min":1.456},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.96,"5min":1.96,"15min":1.962},"last":1.833}]},{"osd":4,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.411,"15min":1.387},"min":{"1min":1.036,"5min":0.84499999999999997,"15min":0.81699999999999995},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.502},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.387,"15min":1.337},"min":{"1min":0.93600000000000005,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":2.0329999999999999,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.5329999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.502,"15min":1.4770000000000001},"min":{"1min":1.089,"5min":0.97599999999999998,"15min":0.90500000000000003},"max":{"1min":1.929,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.5740000000000001},"min":{"1min":1.1659999999999999,"5min":0.98999999999999999,"15min":0.88500000000000001},"max":{"1min":2.0880000000000001,"5min":2.6309999999999998,"15min":2.6309999999999998},"last":1.6419999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:34:04 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.5289999999999999,"15min":1.4890000000000001},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":0.90700000000000003},"max":{"1min":2.1389999999999998,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.6459999999999999,"15min":1.601},"min":{"1min":1.1299999999999999,"5min":1.0629999999999999,"15min":0.95099999999999996},"max":{"1min":2.1920000000000002,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":1.9590000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575778,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29368,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342472,"statfs":{"total":96636764160,"available":96606691328,"internally_reserved":0,"allocated":2404352,"data_stored":1170007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.823,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.913},{"interface":"front","average":{"1min":1.151,"5min":1.254,"15min":1.2529999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.80800000000000005},"max":{"1min":1.7250000000000001,"5min":1.823,"15min":1.9550000000000001},"last":6.6509999999999998}]},{"osd":1,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.238,"15min":1.286},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8799999999999999},"last":2.1320000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.3560000000000001,"15min":1.3300000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.843,"5min":1.843,"15min":2.738},"last":7.1210000000000004}]},{"osd":2,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.256,"15min":1.2749999999999999},"min":{"1min":0.89300000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.633,"5min":1.823,"15min":2.4039999999999999},"last":3.8109999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.389,"15min":1.389},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.89400000000000002},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":2.0289999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.411,"15min":1.407},"min":{"1min":1.139,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":3.9820000000000002},"last":6.798},{"interface":"front","average":{"1min":1.379,"5min":1.3740000000000001,"15min":1.3799999999999999},"min":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.92100000000000004},"max":{"1min":2.0019999999999998,"5min":2.0539999999999998,"15min":2.5489999999999999},"last":4.1100000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.375,"15min":1.331},"min":{"1min":0.92200000000000004,"5min":0.79600000000000004,"15min":0.76200000000000001},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":4.835},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.429,"15min":1.3959999999999999},"min":{"1min":1.0860000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":4.6609999999999996},"last":6.9039999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3720000000000001,"15min":1.3620000000000001},"min":{"1min":1.121,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.6830000000000001,"5min":1.754,"15min":3.8450000000000002},"last":4.2160000000000002},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5429999999999999,"15min":1.524},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.044,"5min":2.044,"15min":2.8639999999999999},"last":3.101}]},{"osd":7,"last update":"Mon Oct 16 15:33:51 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.6140000000000001,"15min":1.6020000000000001},"min":{"1min":1.1479999999999999,"5min":1.0249999999999999,"15min":0.84799999999999998},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":4.5359999999999996},"last":7.0049999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.6160000000000001,"15min":1.6000000000000001},"min":{"1min":1.3280000000000001,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.925,"5min":1.9570000000000001,"15min":2.6619999999999999},"last":5.6360000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41852,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329988,"statfs":{"total":96636764160,"available":96593907712,"internally_reserved":0,"allocated":2408448,"data_stored":1170589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.2070000000000001,"15min":1.1850000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.4159999999999999,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.319},{"interface":"front","average":{"1min":1.246,"5min":1.25,"15min":1.258},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7589999999999999,"5min":1.845,"15min":1.9570000000000001},"last":1.2529999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.234,"15min":1.224},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.80900000000000005},"max":{"1min":1.6419999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.2949999999999999,"15min":1.302},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.7230000000000001,"5min":1.738,"15min":1.905},"last":1.6839999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.27,"15min":1.264},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.9330000000000001,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.196},{"interface":"front","average":{"1min":1.387,"5min":1.401,"15min":1.401},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.835,"5min":2.073,"15min":2.3130000000000002},"last":1.5600000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2929999999999999,"15min":1.3180000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.5800000000000001,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.367},{"interface":"front","average":{"1min":1.349,"5min":1.3799999999999999,"15min":1.375},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":1.498}]},{"osd":4,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3100000000000001,"15min":1.3089999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.7070000000000001,"5min":2.6909999999999998,"15min":2.6909999999999998},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.474,"15min":1.4610000000000001},"min":{"1min":0.88700000000000001,"5min":0.83199999999999996,"15min":0.81000000000000005},"max":{"1min":2.1030000000000002,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.603}]},{"osd":6,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3720000000000001,"15min":1.361},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.508},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4370000000000001,"15min":1.4319999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.764,"5min":2.9209999999999998,"15min":2.9790000000000001},"last":1.5249999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:34:01 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5049999999999999,"15min":1.504},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.3719999999999999,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.5920000000000001,"15min":1.601},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.3420000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.6319999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575779,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59912,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311928,"statfs":{"total":96636764160,"available":96575414272,"internally_reserved":0,"allocated":2416640,"data_stored":1171667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.194,"15min":1.1879999999999999},"min":{"1min":0.77600000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.458,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.754},{"interface":"front","average":{"1min":1.145,"5min":1.159,"15min":1.177},"min":{"1min":0.749,"5min":0.749,"15min":0.745},"max":{"1min":1.494,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.198}]},{"osd":1,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.244,"15min":1.2330000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.6659999999999999,"5min":1.78,"15min":1.948},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.268,"15min":1.268},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.7050000000000001,"5min":1.712,"15min":1.895},"last":1.0049999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.4139999999999999,"15min":1.417},"min":{"1min":0.91100000000000003,"5min":0.88,"15min":0.88},"max":{"1min":1.5900000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.409,"15min":1.431},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6830000000000001,"5min":2,"15min":2},"last":0.85699999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3520000000000001,"15min":1.373},"min":{"1min":0.98399999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.5760000000000001,"5min":1.8149999999999999,"15min":2.093},"last":1.546},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.353,"15min":1.3720000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":1.825,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.2430000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.333,"15min":1.3400000000000001},"min":{"1min":1.048,"5min":0.89300000000000002,"15min":0.77900000000000003},"max":{"1min":1.7,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.653},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.361,"15min":1.413},"min":{"1min":0.89800000000000002,"5min":0.86799999999999999,"15min":0.86299999999999999},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":2.097},"last":1.7310000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3400000000000001,"15min":1.349},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.627,"5min":1.903,"15min":1.903},"last":1.095},{"interface":"front","average":{"1min":1.49,"5min":1.512,"15min":1.5209999999999999},"min":{"1min":1.1699999999999999,"5min":1.026,"15min":1.026},"max":{"1min":1.8240000000000001,"5min":2.359,"15min":2.359},"last":1.7829999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:34:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.518,"15min":1.5580000000000001},"min":{"1min":1.123,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":1.778,"5min":1.9059999999999999,"15min":2.1600000000000001},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.5249999999999999,"15min":1.5409999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":1.7969999999999999,"5min":2.0720000000000001,"15min":2.3330000000000002},"last":1.605}]}]},{"osd":3,"up_from":13,"seq":55834575778,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51576,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320264,"statfs":{"total":96636764160,"available":96583950336,"internally_reserved":0,"allocated":2404352,"data_stored":1170019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.92100000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3120000000000001,"5min":1.569,"15min":1.7230000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.069,"15min":1.077},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.52,"5min":1.577,"15min":1.9430000000000001},"last":1.081}]},{"osd":1,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.97399999999999998,"15min":0.999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.4730000000000001,"5min":1.5600000000000001,"15min":1.704},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.214,"15min":1.206},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.46}]},{"osd":2,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.353,"15min":1.4099999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6120000000000001,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4950000000000001,"15min":1.514},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8300000000000001,"5min":2.278,"15min":2.278},"last":0.96399999999999997}]},{"osd":4,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.3540000000000001,"15min":1.327},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.583},{"interface":"front","average":{"1min":1.399,"5min":1.3939999999999999,"15min":1.401},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1840000000000002},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5449999999999999,"15min":1.5649999999999999},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":2.0350000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.244},{"interface":"front","average":{"1min":1.518,"5min":1.6040000000000001,"15min":1.5620000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.956,"5min":2.621,"15min":2.774},"last":1.196}]},{"osd":6,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.5069999999999999,"15min":1.51},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.1230000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.1190000000000002,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.409}]},{"osd":7,"last update":"Mon Oct 16 15:33:57 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.6739999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.04,"5min":2.7309999999999999,"15min":2.8900000000000001},"last":1.347},{"interface":"front","average":{"1min":1.488,"5min":1.633,"15min":1.6439999999999999},"min":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.05},"max":{"1min":1.8819999999999999,"5min":2.1869999999999998,"15min":2.496},"last":1.4950000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575779,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51576,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320264,"statfs":{"total":96636764160,"available":96583950336,"internally_reserved":0,"allocated":2404352,"data_stored":1170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.95099999999999996,"15min":0.92800000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5880000000000001,"5min":1.643,"15min":1.849},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.099},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6299999999999999,"5min":1.6799999999999999,"15min":1.7490000000000001},"last":1.0049999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.018,"15min":1.024},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.585,"5min":1.6419999999999999,"15min":1.8140000000000001},"last":1.121},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.216,"15min":1.2230000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.702,"5min":1.8300000000000001,"15min":1.9299999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.28,"15min":1.3169999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.462,"5min":1.8120000000000001,"15min":1.992},"last":1.323},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.351,"15min":1.3620000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.643,"5min":1.921,"15min":1.921},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.454,"15min":1.472},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.7709999999999999,"5min":2.1150000000000002,"15min":2.149},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.371,"15min":1.3999999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.9159999999999999,"5min":1.986,"15min":2.6909999999999998},"last":1.1040000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5089999999999999,"15min":1.4830000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.9810000000000001,"5min":2.661,"15min":2.661},"last":1.419},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.591,"15min":1.611},"min":{"1min":1.032,"5min":1.032,"15min":1.032},"max":{"1min":1.831,"5min":2.3730000000000002,"15min":3.3149999999999999},"last":1.7}]},{"osd":6,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4790000000000001,"15min":1.5},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":2.0489999999999999,"5min":2.1429999999999998,"15min":2.573},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.613,"15min":1.6040000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.948,"5min":2.423,"15min":2.423},"last":1.5149999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:33:52 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5640000000000001,"15min":1.595},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.1549999999999998,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.663,"5min":1.7210000000000001,"15min":1.7150000000000001},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.0979999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.6220000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575779,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64064,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307776,"statfs":{"total":96636764160,"available":96571162624,"internally_reserved":0,"allocated":2404352,"data_stored":1170019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91400000000000003,"15min":0.96099999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.6830000000000001,"5min":1.716,"15min":1.8440000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1599999999999999,"15min":1.1259999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.337}]},{"osd":2,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0269999999999999,"15min":1.0429999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2010000000000001,"15min":1.22},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.53,"5min":1.7410000000000001,"15min":1.9430000000000001},"last":1.653}]},{"osd":3,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.371,"15min":1.379},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.012,"5min":2.012,"15min":2.02},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.258,"15min":1.2729999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.048}]},{"osd":4,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.385,"15min":1.361},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.883,"5min":1.8879999999999999,"15min":1.9990000000000001},"last":1.236},{"interface":"front","average":{"1min":1.427,"5min":1.4730000000000001,"15min":1.4930000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.79,"5min":1.9059999999999999,"15min":2.3500000000000001},"last":1.7829999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.532,"15min":1.4970000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.0619999999999998,"5min":2.2770000000000001,"15min":2.2770000000000001},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.472,"15min":1.462},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.889,"5min":2.1059999999999999,"15min":2.1200000000000001},"last":1.46}]},{"osd":6,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.629,"15min":1.639},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":2.5230000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6399999999999999,"15min":1.611},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.1579999999999999,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.718}]},{"osd":7,"last update":"Mon Oct 16 15:33:36 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.756,"15min":1.7170000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":2.274,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.621,"5min":1.615,"15min":1.6080000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.036,"5min":2.0409999999999999,"15min":2.3039999999999998},"last":1.5960000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575779,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60472,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311368,"statfs":{"total":96636764160,"available":96574840832,"internally_reserved":0,"allocated":2994176,"data_stored":1760353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":0.97199999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2050000000000001,"5min":1.903,"15min":1.903},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6910000000000001,"5min":1.851,"15min":1.9450000000000001},"last":1.2549999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.29,"15min":1.304},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.913,"5min":1.913,"15min":2.0640000000000001},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.2330000000000001,"15min":1.26},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8340000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0489999999999999,"15min":1.0249999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.306,"5min":1.732,"15min":1.732},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.137,"5min":1.282,"15min":1.2849999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8360000000000001,"5min":2.262,"15min":2.262},"last":1.3540000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3440000000000001,"15min":1.3280000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7529999999999999,"5min":2.109,"15min":2.109},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.516,"15min":1.522},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.9590000000000001,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.6539999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.484,"15min":1.492},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.7869999999999999,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4430000000000001,"15min":1.4399999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.4319999999999999,"5min":3.4319999999999999,"15min":3.4319999999999999},"last":1.208}]},{"osd":6,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.665,"15min":1.649},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.3130000000000002,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":1.859},{"interface":"front","average":{"1min":1.462,"5min":1.5940000000000001,"15min":1.5760000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0680000000000001,"5min":2.4340000000000002,"15min":2.5009999999999999},"last":1.099}]},{"osd":7,"last update":"Mon Oct 16 15:34:07 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.6799999999999999,"15min":1.6830000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.089,"5min":3.089,"15min":3.089},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.5940000000000001,"15min":1.5960000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9650000000000001,"5min":2.774,"15min":2.774},"last":1.5289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:34:31.663 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:34:31.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T15:34:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.934+0000 7f3db81b7640 1 Processor -- start 2023-10-16T15:34:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.935+0000 7f3db81b7640 1 -- start start 2023-10-16T15:34:31.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.936+0000 7f3db81b7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3db00b0b20 0x7f3db015d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.936+0000 7f3db81b7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3db01615d0 0x7f3db015d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.936+0000 7f3db81b7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db015e3c0 con 0x7f3db00ccfc0 2023-10-16T15:34:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.936+0000 7f3db81b7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db015de70 con 0x7f3db01615d0 2023-10-16T15:34:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.936+0000 7f3db81b7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db015e0e0 con 0x7f3db00b0b20 2023-10-16T15:34:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3db01615d0 0x7f3db015d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3db01615d0 0x7f3db015d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33444/0 (socket says 172.21.15.29:33444) 2023-10-16T15:34:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 -- 172.21.15.29:0/2631825802 learned_addr learned my addr 172.21.15.29:0/2631825802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db5752640 1 --2- 172.21.15.29:0/2631825802 >> v2:172.21.15.170:3300/0 conn(0x7f3db00b0b20 0x7f3db015d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 -- 172.21.15.29:0/2631825802 >> v2:172.21.15.170:3300/0 conn(0x7f3db00b0b20 msgr2=0x7f3db015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 --2- 172.21.15.29:0/2631825802 >> v2:172.21.15.170:3300/0 conn(0x7f3db00b0b20 0x7f3db015d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 -- 172.21.15.29:0/2631825802 >> v1:172.21.15.170:6789/0 conn(0x7f3db00ccfc0 legacy=0x7f3db00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.937+0000 7f3db6754640 1 -- 172.21.15.29:0/2631825802 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db0160af0 con 0x7f3db01615d0 2023-10-16T15:34:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.939+0000 7f3db6754640 1 --2- 172.21.15.29:0/2631825802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01615d0 0x7f3db015d930 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009e50 tx=0x7f3da802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5888c7df15d244df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.939+0000 7f3db4f51640 1 -- 172.21.15.29:0/2631825802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da802c9f0 con 0x7f3db01615d0 2023-10-16T15:34:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.939+0000 7f3db4f51640 1 -- 172.21.15.29:0/2631825802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da802cb50 con 0x7f3db01615d0 2023-10-16T15:34:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.939+0000 7f3db4f51640 1 -- 172.21.15.29:0/2631825802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da80343f0 con 0x7f3db01615d0 2023-10-16T15:34:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 -- 172.21.15.29:0/2631825802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01615d0 msgr2=0x7f3db015d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 --2- 172.21.15.29:0/2631825802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01615d0 0x7f3db015d930 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009e50 tx=0x7f3da802be50 comp rx=0 tx=0).stop 2023-10-16T15:34:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 -- 172.21.15.29:0/2631825802 shutdown_connections 2023-10-16T15:34:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 --2- 172.21.15.29:0/2631825802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db01615d0 0x7f3db015d930 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 --2- 172.21.15.29:0/2631825802 >> v2:172.21.15.170:3300/0 conn(0x7f3db00b0b20 0x7f3db015d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 -- 172.21.15.29:0/2631825802 >> 172.21.15.29:0/2631825802 conn(0x7f3db00cb410 msgr2=0x7f3db00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 -- 172.21.15.29:0/2631825802 shutdown_connections 2023-10-16T15:34:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.940+0000 7f3db81b7640 1 -- 172.21.15.29:0/2631825802 wait complete. 2023-10-16T15:34:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.942+0000 7f3db81b7640 1 Processor -- start 2023-10-16T15:34:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.943+0000 7f3db81b7640 1 -- start start 2023-10-16T15:34:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db81b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 0x7f3db01f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db81b7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00ccfc0 0x7f3db0153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db81b7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db01615d0 0x7f3db0155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db81b7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00b1550 con 0x7f3db00b0b20 2023-10-16T15:34:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db81b7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00b1810 con 0x7f3db01615d0 2023-10-16T15:34:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db81b7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00b1af0 con 0x7f3db00ccfc0 2023-10-16T15:34:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db5f53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 0x7f3db01f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db5f53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 0x7f3db01f2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33458/0 (socket says 172.21.15.29:33458) 2023-10-16T15:34:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.944+0000 7f3db5f53640 1 -- 172.21.15.29:0/3055580053 learned_addr learned my addr 172.21.15.29:0/3055580053 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db6754640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db01615d0 0x7f3db0155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5752640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00ccfc0 0x7f3db0153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5f53640 1 -- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00ccfc0 msgr2=0x7f3db0153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5f53640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00ccfc0 0x7f3db0153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5f53640 1 -- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db01615d0 msgr2=0x7f3db0155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5f53640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db01615d0 0x7f3db0155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5f53640 1 -- 172.21.15.29:0/3055580053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da8009b30 con 0x7f3db00b0b20 2023-10-16T15:34:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db6754640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db01615d0 0x7f3db0155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3db5f53640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 0x7f3db01f2cd0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00c430 tx=0x7f3dac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.945+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac011790 con 0x7f3db00b0b20 2023-10-16T15:34:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.946+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac0118f0 con 0x7f3db00b0b20 2023-10-16T15:34:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.946+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac011ce0 con 0x7f3db00b0b20 2023-10-16T15:34:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.946+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db00b1dd0 con 0x7f3db00b0b20 2023-10-16T15:34:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.946+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db0154170 con 0x7f3db00b0b20 2023-10-16T15:34:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.947+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db00b22f0 con 0x7f3db00b0b20 2023-10-16T15:34:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.948+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3dac01c460 con 0x7f3db00b0b20 2023-10-16T15:34:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.950+0000 7f3da67fc640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f3d9406af90 0x7f3d9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.950+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3dac089720 con 0x7f3db00b0b20 2023-10-16T15:34:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.950+0000 7f3db5752640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f3d9406af90 0x7f3d9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.951+0000 7f3db5752640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f3d9406af90 0x7f3d9406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3da0006fd0 tx=0x7f3da0008040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:31.955+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3db00b22f0 con 0x7f3db00b0b20 2023-10-16T15:34:32.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.224+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3db00b22f0 con 0x7f3db00b0b20 2023-10-16T15:34:32.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.229+0000 7f3da67fc640 1 -- 172.21.15.29:0/3055580053 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3db00b22f0 con 0x7f3db00b0b20 2023-10-16T15:34:32.231 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T15:34:32.231 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T15:34:32.231 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T15:34:32.231 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T15:34:32.231 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T15:34:32.232 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T15:34:32.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f3d9406af90 msgr2=0x7f3d9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f3d9406af90 0x7f3d9406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3da0006fd0 tx=0x7f3da0008040 comp rx=0 tx=0).stop 2023-10-16T15:34:32.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 msgr2=0x7f3db01f2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 0x7f3db01f2cd0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00c430 tx=0x7f3dac00c8f0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 shutdown_connections 2023-10-16T15:34:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f3d9406af90 0x7f3d9406d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3db01615d0 0x7f3db0155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3db00ccfc0 0x7f3db0153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 --2- 172.21.15.29:0/3055580053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db00b0b20 0x7f3db01f2cd0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 >> 172.21.15.29:0/3055580053 conn(0x7f3db00cb410 msgr2=0x7f3db00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 shutdown_connections 2023-10-16T15:34:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.232+0000 7f3db81b7640 1 -- 172.21.15.29:0/3055580053 wait complete. 2023-10-16T15:34:32.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:34:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.481+0000 7f416e6ad640 1 Processor -- start 2023-10-16T15:34:32.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.482+0000 7f416e6ad640 1 -- start start 2023-10-16T15:34:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.483+0000 7f416e6ad640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f416806c9e0 0x7f41680ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.483+0000 7f416e6ad640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41680ca560 con 0x7f41680c9060 2023-10-16T15:34:32.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.483+0000 7f416e6ad640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41680c9950 con 0x7f416806fb50 2023-10-16T15:34:32.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.483+0000 7f416e6ad640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41680c9c30 con 0x7f416806c9e0 2023-10-16T15:34:32.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.484+0000 7f416d6ab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41680c9060 0x7f416806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50088/0 (socket says 172.21.15.29:50088) 2023-10-16T15:34:32.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.484+0000 7f416d6ab640 1 -- 172.21.15.29:0/4185497729 learned_addr learned my addr 172.21.15.29:0/4185497729 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:32.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.484+0000 7f416deac640 1 --2- 172.21.15.29:0/4185497729 >> v2:172.21.15.170:3300/0 conn(0x7f416806c9e0 0x7f41680ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.484+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646443111 0 0) 0x7f41680ca560 con 0x7f41680c9060 2023-10-16T15:34:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.484+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154003520 con 0x7f41680c9060 2023-10-16T15:34:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2221170295 0 0) 0x7f4154003520 con 0x7f41680c9060 2023-10-16T15:34:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 >> v2:172.21.15.170:3300/0 conn(0x7f416806c9e0 msgr2=0x7f41680ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 --2- 172.21.15.29:0/4185497729 >> v2:172.21.15.170:3300/0 conn(0x7f416806c9e0 0x7f41680ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 >> v1:172.21.15.170:6789/0 conn(0x7f416806fb50 legacy=0x7f41680cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41680c9f10 con 0x7f41680c9060 2023-10-16T15:34:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4164001670 con 0x7f41680c9060 2023-10-16T15:34:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 >> v1:172.21.15.29:6789/0 conn(0x7f41680c9060 legacy=0x7f416806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540039c0 0x7f4154003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540047c0 0x7f4154004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154005180 0x7f415400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41680c9c30 con 0x7f41540039c0 2023-10-16T15:34:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415400c080 con 0x7f41540047c0 2023-10-16T15:34:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415400c240 con 0x7f4154005180 2023-10-16T15:34:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.485+0000 7f416d6ab640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540039c0 0x7f4154003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416ceaa640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540047c0 0x7f4154004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416d6ab640 1 -- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154005180 msgr2=0x7f415400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416deac640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154005180 0x7f415400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416d6ab640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154005180 0x7f415400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416d6ab640 1 -- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540047c0 msgr2=0x7f4154004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416d6ab640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41540047c0 0x7f4154004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416d6ab640 1 -- 172.21.15.29:0/4185497729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154003520 con 0x7f41540039c0 2023-10-16T15:34:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416deac640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4154005180 0x7f415400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.486+0000 7f416deac640 1 -- 172.21.15.29:0/4185497729 reap_dead start 2023-10-16T15:34:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.487+0000 7f416d6ab640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540039c0 0x7f4154003de0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f416400c0e0 tx=0x7f41540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd4da31b7327ce8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.487+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4164005ad0 con 0x7f41540039c0 2023-10-16T15:34:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.487+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4164005c30 con 0x7f41540039c0 2023-10-16T15:34:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.487+0000 7f415ffff640 1 -- 172.21.15.29:0/4185497729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416402c9d0 con 0x7f41540039c0 2023-10-16T15:34:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 -- 172.21.15.29:0/4185497729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540039c0 msgr2=0x7f4154003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540039c0 0x7f4154003de0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f416400c0e0 tx=0x7f41540057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 -- 172.21.15.29:0/4185497729 shutdown_connections 2023-10-16T15:34:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 --2- 172.21.15.29:0/4185497729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41540039c0 0x7f4154003de0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 -- 172.21.15.29:0/4185497729 >> 172.21.15.29:0/4185497729 conn(0x7f41680ce6a0 msgr2=0x7f41680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 -- 172.21.15.29:0/4185497729 shutdown_connections 2023-10-16T15:34:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.488+0000 7f416e6ad640 1 -- 172.21.15.29:0/4185497729 wait complete. 2023-10-16T15:34:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.490+0000 7f416e6ad640 1 Processor -- start 2023-10-16T15:34:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.490+0000 7f416e6ad640 1 -- start start 2023-10-16T15:34:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416e6ad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540039c0 0x7f41681f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416e6ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 0x7f41680b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416e6ad640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41681f9dc0 0x7f41681fa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416e6ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41680b4470 con 0x7f41680b3060 2023-10-16T15:34:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416e6ad640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41681fa780 con 0x7f41681f9dc0 2023-10-16T15:34:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416e6ad640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4168162790 con 0x7f41540039c0 2023-10-16T15:34:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416ceaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 0x7f41680b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416ceaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 0x7f41680b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33464/0 (socket says 172.21.15.29:33464) 2023-10-16T15:34:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.491+0000 7f416ceaa640 1 -- 172.21.15.29:0/3819058329 learned_addr learned my addr 172.21.15.29:0/3819058329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416deac640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41681f9dc0 0x7f41681fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416d6ab640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540039c0 0x7f41681f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416ceaa640 1 -- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540039c0 msgr2=0x7f41681f7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416ceaa640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540039c0 0x7f41681f7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416ceaa640 1 -- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41681f9dc0 msgr2=0x7f41681fa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416ceaa640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41681f9dc0 0x7f41681fa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416ceaa640 1 -- 172.21.15.29:0/3819058329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416814a3a0 con 0x7f41680b3060 2023-10-16T15:34:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416deac640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41681f9dc0 0x7f41681fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.492+0000 7f416ceaa640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 0x7f41680b34e0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f415800e440 tx=0x7f415800e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.493+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580053a0 con 0x7f41680b3060 2023-10-16T15:34:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.493+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41580035f0 con 0x7f41680b3060 2023-10-16T15:34:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.493+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158003960 con 0x7f41680b3060 2023-10-16T15:34:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.493+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41680b3bd0 con 0x7f41680b3060 2023-10-16T15:34:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.493+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41680b4190 con 0x7f41680b3060 2023-10-16T15:34:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.494+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416804e9b0 con 0x7f41680b3060 2023-10-16T15:34:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.495+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4158003ac0 con 0x7f41680b3060 2023-10-16T15:34:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.496+0000 7f415dffb640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f414c06af90 0x7f414c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.496+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4158088e60 con 0x7f41680b3060 2023-10-16T15:34:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.497+0000 7f416d6ab640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f414c06af90 0x7f414c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.498+0000 7f416d6ab640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f414c06af90 0x7f414c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4164034000 tx=0x7f4164009030 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.501+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4158053e80 con 0x7f41680b3060 2023-10-16T15:34:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.732+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41681628d0 con 0x7f41680b3060 2023-10-16T15:34:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.733+0000 7f415dffb640 1 -- 172.21.15.29:0/3819058329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f4158019020 con 0x7f41680b3060 2023-10-16T15:34:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:34:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:34:26.024272+0000","modified":"2023-10-16T15:34:29.168102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2122514336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122514336},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122514336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T15:34:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 92 2023-10-16T15:34:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f414c06af90 msgr2=0x7f414c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f414c06af90 0x7f414c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4164034000 tx=0x7f4164009030 comp rx=0 tx=0).stop 2023-10-16T15:34:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 msgr2=0x7f41680b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 0x7f41680b34e0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f415800e440 tx=0x7f415800e900 comp rx=0 tx=0).stop 2023-10-16T15:34:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 shutdown_connections 2023-10-16T15:34:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f414c06af90 0x7f414c06d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41680b3060 0x7f41680b34e0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41681f9dc0 0x7f41681fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 --2- 172.21.15.29:0/3819058329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41540039c0 0x7f41681f7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.736+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 >> 172.21.15.29:0/3819058329 conn(0x7f41680ce6a0 msgr2=0x7f41680cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.737+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 shutdown_connections 2023-10-16T15:34:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:32.737+0000 7f416e6ad640 1 -- 172.21.15.29:0/3819058329 wait complete. 2023-10-16T15:34:33.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:34:26.024272+0000', 'modified': '2023-10-16T15:34:29.168102+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9816}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9816': {'gid': 9816, 'name': 'c', 'rank': 0, 'incarnation': 89, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/2122514336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 2122514336}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2122514336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:34:33.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:34:33.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T15:34:33.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.396+0000 7efd40f1a640 1 Processor -- start 2023-10-16T15:34:33.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.396+0000 7efd40f1a640 1 -- start start 2023-10-16T15:34:33.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd40f1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efd3c0ad970 0x7efd3c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd40f1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c15b9d0 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd40f1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c15bcb0 con 0x7efd3c06c040 2023-10-16T15:34:33.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd40f1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3c15bf90 con 0x7efd3c0ad970 2023-10-16T15:34:33.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd3bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efd3c0ccfb0 0x7efd3c0cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50096/0 (socket says 172.21.15.29:50096) 2023-10-16T15:34:33.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd3bfff640 1 -- 172.21.15.29:0/424205418 learned_addr learned my addr 172.21.15.29:0/424205418 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:33.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.398+0000 7efd3affd640 1 --2- 172.21.15.29:0/424205418 >> v2:172.21.15.170:3300/0 conn(0x7efd3c0ad970 0x7efd3c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.399+0000 7efd3a7fc640 1 -- 172.21.15.29:0/424205418 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597682677 0 0) 0x7efd3c15b9d0 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.399+0000 7efd3a7fc640 1 -- 172.21.15.29:0/424205418 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd24003520 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.399+0000 7efd3a7fc640 1 -- 172.21.15.29:0/424205418 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 931356599 0 0) 0x7efd24003520 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.399+0000 7efd3affd640 1 -- 172.21.15.29:0/424205418 >> v1:172.21.15.170:6789/0 conn(0x7efd3c06c040 legacy=0x7efd3c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.399+0000 7efd3affd640 1 -- 172.21.15.29:0/424205418 >> v1:172.21.15.29:6789/0 conn(0x7efd3c0ccfb0 legacy=0x7efd3c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.399+0000 7efd3affd640 1 -- 172.21.15.29:0/424205418 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0cc270 con 0x7efd3c0ad970 2023-10-16T15:34:33.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.400+0000 7efd3affd640 1 --2- 172.21.15.29:0/424205418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0ad970 0x7efd3c0cca70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7efd300097d0 tx=0x7efd3002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f34c7464f3ad1fe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.401+0000 7efd3a7fc640 1 -- 172.21.15.29:0/424205418 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3002f050 con 0x7efd3c0ad970 2023-10-16T15:34:33.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.401+0000 7efd3a7fc640 1 -- 172.21.15.29:0/424205418 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd30031500 con 0x7efd3c0ad970 2023-10-16T15:34:33.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.401+0000 7efd3a7fc640 1 -- 172.21.15.29:0/424205418 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340019f0 con 0x7efd3c0ad970 2023-10-16T15:34:33.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.401+0000 7efd40f1a640 1 -- 172.21.15.29:0/424205418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0ad970 msgr2=0x7efd3c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.401+0000 7efd40f1a640 1 --2- 172.21.15.29:0/424205418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0ad970 0x7efd3c0cca70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7efd300097d0 tx=0x7efd3002b820 comp rx=0 tx=0).stop 2023-10-16T15:34:33.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.402+0000 7efd40f1a640 1 -- 172.21.15.29:0/424205418 shutdown_connections 2023-10-16T15:34:33.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.402+0000 7efd40f1a640 1 --2- 172.21.15.29:0/424205418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0ad970 0x7efd3c0cca70 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.402+0000 7efd40f1a640 1 -- 172.21.15.29:0/424205418 >> 172.21.15.29:0/424205418 conn(0x7efd3c0ce6a0 msgr2=0x7efd3c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.402+0000 7efd40f1a640 1 -- 172.21.15.29:0/424205418 shutdown_connections 2023-10-16T15:34:33.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.402+0000 7efd40f1a640 1 -- 172.21.15.29:0/424205418 wait complete. 2023-10-16T15:34:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.403+0000 7efd40f1a640 1 Processor -- start 2023-10-16T15:34:33.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.404+0000 7efd40f1a640 1 -- start start 2023-10-16T15:34:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.404+0000 7efd40f1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd3c06c040 0x7efd3c0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.404+0000 7efd40f1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 0x7efd3c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd40f1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0b2850 0x7efd3c0ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd40f1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0adde0 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd40f1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0b2d00 con 0x7efd3c0b2850 2023-10-16T15:34:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd40f1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0b2f90 con 0x7efd3c06c040 2023-10-16T15:34:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 0x7efd3c0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 0x7efd3c0b2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33480/0 (socket says 172.21.15.29:33480) 2023-10-16T15:34:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 -- 172.21.15.29:0/596899053 learned_addr learned my addr 172.21.15.29:0/596899053 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:33.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3bfff640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0b2850 0x7efd3c0ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd3c06c040 0x7efd3c0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 -- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd3c06c040 msgr2=0x7efd3c0acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd3c06c040 0x7efd3c0acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 -- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0b2850 msgr2=0x7efd3c0ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0b2850 0x7efd3c0ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3affd640 1 -- 172.21.15.29:0/596899053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd30009400 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.405+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd3c06c040 0x7efd3c0acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.406+0000 7efd3affd640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 0x7efd3c0b2310 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7efd300098f0 tx=0x7efd300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.406+0000 7efd23fff640 1 -- 172.21.15.29:0/596899053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3003d880 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.406+0000 7efd23fff640 1 -- 172.21.15.29:0/596899053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3003d9e0 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.406+0000 7efd23fff640 1 -- 172.21.15.29:0/596899053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3003dd70 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.406+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c0b3270 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.406+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c0ae310 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.408+0000 7efd23fff640 1 -- 172.21.15.29:0/596899053 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd3003a040 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.409+0000 7efd23fff640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7efd2806af90 0x7efd2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.409+0000 7efd23fff640 1 -- 172.21.15.29:0/596899053 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efd300ab480 con 0x7efd3c0ccfb0 2023-10-16T15:34:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.410+0000 7efd44de6740 1 Processor -- start 2023-10-16T15:34:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.410+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7efd2806af90 0x7efd2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.410+0000 7efd44de6740 1 -- start start 2023-10-16T15:34:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.411+0000 7efd44de6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 0x559fca847070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.411+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874550 0x559fca876f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.411+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca8475b0 0x559fca877440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.411+0000 7efd44de6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca7677a0 con 0x559fca874170 2023-10-16T15:34:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.411+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca7b9740 con 0x559fca8475b0 2023-10-16T15:34:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.411+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fca7b6e30 con 0x559fca874550 2023-10-16T15:34:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 0x559fca847070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 0x559fca847070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33486/0 (socket says 172.21.15.29:33486) 2023-10-16T15:34:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 -- 172.21.15.29:0/4179370922 learned_addr learned my addr 172.21.15.29:0/4179370922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7efd2806af90 0x7efd2806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd2c000f60 tx=0x7efd2c009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3bfff640 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca8475b0 0x559fca877440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3affd640 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874550 0x559fca876f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 -- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874550 msgr2=0x559fca876f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874550 0x559fca876f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 -- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca8475b0 msgr2=0x559fca877440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca8475b0 0x559fca877440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.412+0000 7efd3b7fe640 1 -- 172.21.15.29:0/4179370922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fca776910 con 0x559fca874170 2023-10-16T15:34:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.413+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 0x559fca847070 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00d930 tx=0x7efd2c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609e91139214957d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.413+0000 7efd217fa640 1 -- 172.21.15.29:0/4179370922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c014ba0 con 0x559fca874170 2023-10-16T15:34:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.413+0000 7efd217fa640 1 -- 172.21.15.29:0/4179370922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c014d00 con 0x559fca874170 2023-10-16T15:34:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.413+0000 7efd217fa640 1 -- 172.21.15.29:0/4179370922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c01f620 con 0x559fca874170 2023-10-16T15:34:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.413+0000 7efd44de6740 1 -- 172.21.15.29:0/4179370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 msgr2=0x559fca847070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.413+0000 7efd44de6740 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 0x559fca847070 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00d930 tx=0x7efd2c010000 comp rx=0 tx=0).stop 2023-10-16T15:34:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 -- 172.21.15.29:0/4179370922 shutdown_connections 2023-10-16T15:34:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca8475b0 0x559fca877440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874550 0x559fca876f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 --2- 172.21.15.29:0/4179370922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874170 0x559fca847070 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 -- 172.21.15.29:0/4179370922 >> 172.21.15.29:0/4179370922 conn(0x559fca849380 msgr2=0x559fca891f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 -- 172.21.15.29:0/4179370922 shutdown_connections 2023-10-16T15:34:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 -- 172.21.15.29:0/4179370922 wait complete. 2023-10-16T15:34:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.414+0000 7efd44de6740 1 -- start start 2023-10-16T15:34:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.415+0000 7efd44de6740 1 Processor -- start 2023-10-16T15:34:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca85b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca870240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca878f40 0x559fca870780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca7b2720 con 0x559fca874550 2023-10-16T15:34:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca750970 con 0x559fca878f40 2023-10-16T15:34:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fca750610 con 0x559fca874170 2023-10-16T15:34:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd44de6740 10 client.0 _ll_register_callbacks cb 0x559fca7729d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:34:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca870240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca870240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33502/0 (socket says 172.21.15.29:33502) 2023-10-16T15:34:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd3affd640 1 -- 172.21.15.29:0/2370009388 learned_addr learned my addr 172.21.15.29:0/2370009388 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.416+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca85b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd3affd640 1 -- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 msgr2=0x559fca85b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd3affd640 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca85b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd3affd640 1 -- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca878f40 msgr2=0x559fca870780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd3affd640 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca878f40 0x559fca870780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd3affd640 1 -- 172.21.15.29:0/2370009388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c15bcb0 con 0x559fca874550 2023-10-16T15:34:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd3affd640 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca870240 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7efd30079cb0 tx=0x7efd30079c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7992fa275c1057 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd1a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:34:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd1a7fc640 1 -- 172.21.15.29:0/2370009388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3002f020 con 0x559fca874550 2023-10-16T15:34:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd1a7fc640 1 -- 172.21.15.29:0/2370009388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efd3007ee40 con 0x559fca874550 2023-10-16T15:34:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fca7d9160 con 0x559fca874550 2023-10-16T15:34:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd1a7fc640 1 -- 172.21.15.29:0/2370009388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd300822b0 con 0x559fca874550 2023-10-16T15:34:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd44de6740 10 client.9948 fetch_fsmap learned FSMap version 92 2023-10-16T15:34:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x559fca7b0da0 con 0x559fca874550 2023-10-16T15:34:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.417+0000 7efd1a7fc640 1 -- 172.21.15.29:0/2370009388 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559fca7d9160 con 0x559fca874550 2023-10-16T15:34:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd1a7fc640 1 -- 172.21.15.29:0/2370009388 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 92) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efd300ab070 con 0x559fca874550 2023-10-16T15:34:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd44de6740 10 client.9948 fetch_fsmap finished waiting for FSMap version 92 2023-10-16T15:34:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd44de6740 10 client.9948 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:34:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd44de6740 20 client.9948 populate_metadata read hostname 'smithi029' 2023-10-16T15:34:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd44de6740 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca872620 0x559fca872a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd44de6740 4 client.9948 mds_command: new command op to 9816 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:34:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 --> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fca784cc0 con 0x559fca872620 2023-10-16T15:34:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.418+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca872620 0x559fca872a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.419+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca872620 0x559fca872a20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f9f559fcce15afe server_cookie=ae3e23eaaa5a2134 in_seq=0 out_seq=0 2023-10-16T15:34:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.419+0000 7efd1a7fc640 10 client.9948 ms_handle_connect on v2:172.21.15.29:6834/2122514336 2023-10-16T15:34:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.419+0000 7efd1a7fc640 1 -- 172.21.15.29:0/2370009388 <== mds.0 v2:172.21.15.29:6834/2122514336 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559fca784cc0 con 0x559fca872620 2023-10-16T15:34:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.419+0000 7efd1a7fc640 10 client.9948 handle_command_reply: tid=0 2023-10-16T15:34:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 client.9948 shutdown 2023-10-16T15:34:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 msgr2=0x559fca870240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca870240 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7efd30079cb0 tx=0x7efd30079c00 comp rx=0 tx=0).stop 2023-10-16T15:34:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 shutdown_connections 2023-10-16T15:34:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca872620 0x559fca872a20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca870240 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca878f40 0x559fca870780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 --2- 172.21.15.29:0/2370009388 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca85b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd1a7fc640 0 client.9948 ms_handle_reset on v2:172.21.15.29:6834/2122514336 2023-10-16T15:34:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 >> 172.21.15.29:0/2370009388 conn(0x559fca8475b0 msgr2=0x559fca895410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 shutdown_connections 2023-10-16T15:34:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 1 -- 172.21.15.29:0/2370009388 wait complete. 2023-10-16T15:34:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.420+0000 7efd44de6740 20 client.9948 trim_cache size 0 max 16384 2023-10-16T15:34:33.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.437+0000 7efd44de6740 1 Processor -- start 2023-10-16T15:34:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.437+0000 7efd44de6740 1 -- start start 2023-10-16T15:34:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874550 0x559fca84ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd44de6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 0x559fca895600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd44de6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca516f20 con 0x559fca878f40 2023-10-16T15:34:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca894840 con 0x559fca874550 2023-10-16T15:34:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fca84e5e0 con 0x559fca874170 2023-10-16T15:34:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.438+0000 7efd3bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 0x559fca895600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 0x559fca895600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33514/0 (socket says 172.21.15.29:33514) 2023-10-16T15:34:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 -- 172.21.15.29:0/4157004916 learned_addr learned my addr 172.21.15.29:0/4157004916 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 -- 172.21.15.29:0/4157004916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 msgr2=0x559fca84d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 -- 172.21.15.29:0/4157004916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874550 msgr2=0x559fca84ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874550 0x559fca84ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 -- 172.21.15.29:0/4157004916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fca74fb80 con 0x559fca878f40 2023-10-16T15:34:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd3bfff640 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 0x559fca895600 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7efd34001940 tx=0x7efd34007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4c0f8e17f5f0a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd21ffb640 1 -- 172.21.15.29:0/4157004916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34014400 con 0x559fca878f40 2023-10-16T15:34:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.439+0000 7efd21ffb640 1 -- 172.21.15.29:0/4157004916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34014560 con 0x559fca878f40 2023-10-16T15:34:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd21ffb640 1 -- 172.21.15.29:0/4157004916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340148f0 con 0x559fca878f40 2023-10-16T15:34:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 -- 172.21.15.29:0/4157004916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 msgr2=0x559fca895600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 0x559fca895600 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7efd34001940 tx=0x7efd34007420 comp rx=0 tx=0).stop 2023-10-16T15:34:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 -- 172.21.15.29:0/4157004916 shutdown_connections 2023-10-16T15:34:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca878f40 0x559fca895600 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874550 0x559fca84ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 --2- 172.21.15.29:0/4157004916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 -- 172.21.15.29:0/4157004916 >> 172.21.15.29:0/4157004916 conn(0x559fca8475b0 msgr2=0x559fca842810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 -- 172.21.15.29:0/4157004916 shutdown_connections 2023-10-16T15:34:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.440+0000 7efd44de6740 1 -- 172.21.15.29:0/4157004916 wait complete. 2023-10-16T15:34:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.441+0000 7efd44de6740 1 -- start start 2023-10-16T15:34:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.441+0000 7efd44de6740 1 Processor -- start 2023-10-16T15:34:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca876d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca878f40 0x559fca8f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca8fb360 con 0x559fca874550 2023-10-16T15:34:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fca877280 con 0x559fca874170 2023-10-16T15:34:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x559fca86fa70 con 0x559fca878f40 2023-10-16T15:34:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd44de6740 10 client.0 _ll_register_callbacks cb 0x7efd300762f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:34:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca876d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca876d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33518/0 (socket says 172.21.15.29:33518) 2023-10-16T15:34:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.442+0000 7efd3affd640 1 -- 172.21.15.29:0/3551296015 learned_addr learned my addr 172.21.15.29:0/3551296015 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3bfff640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca878f40 0x559fca8f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3affd640 1 -- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca878f40 msgr2=0x559fca8f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3affd640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca878f40 0x559fca8f7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3affd640 1 -- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874170 msgr2=0x559fca84d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3affd640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3affd640 1 -- 172.21.15.29:0/3551296015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3400f040 con 0x559fca874550 2023-10-16T15:34:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd3affd640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca876d40 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7efd30081d80 tx=0x7efd30032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22185877504f1dab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd1a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:34:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd1a7fc640 1 -- 172.21.15.29:0/3551296015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3002f030 con 0x559fca874550 2023-10-16T15:34:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd1a7fc640 1 -- 172.21.15.29:0/3551296015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efd3007ee40 con 0x559fca874550 2023-10-16T15:34:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd2c014d00 con 0x559fca874550 2023-10-16T15:34:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.443+0000 7efd1a7fc640 1 -- 172.21.15.29:0/3551296015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd300822b0 con 0x559fca874550 2023-10-16T15:34:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 10 client.9954 fetch_fsmap learned FSMap version 92 2023-10-16T15:34:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x559fca775170 con 0x559fca874550 2023-10-16T15:34:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd1a7fc640 1 -- 172.21.15.29:0/3551296015 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c014d00 con 0x559fca874550 2023-10-16T15:34:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd1a7fc640 1 -- 172.21.15.29:0/3551296015 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 92) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efd30075da0 con 0x559fca874550 2023-10-16T15:34:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 10 client.9954 fetch_fsmap finished waiting for FSMap version 92 2023-10-16T15:34:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 10 client.9954 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:34:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 20 client.9954 populate_metadata read hostname 'smithi029' 2023-10-16T15:34:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca8a48d0 0x559fca8a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 4 client.9954 mds_command: new command op to 9816 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:34:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 --> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559fca7ab700 con 0x559fca8a48d0 2023-10-16T15:34:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.444+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca8a48d0 0x559fca8a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd3b7fe640 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca8a48d0 0x559fca8a4cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eb326868c819fc3 server_cookie=95f6d63ba8fd06b7 in_seq=0 out_seq=0 2023-10-16T15:34:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd1a7fc640 10 client.9954 ms_handle_connect on v2:172.21.15.29:6834/2122514336 2023-10-16T15:34:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd1a7fc640 1 -- 172.21.15.29:0/3551296015 <== mds.0 v2:172.21.15.29:6834/2122514336 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x559fca7ab700 con 0x559fca8a48d0 2023-10-16T15:34:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd1a7fc640 10 client.9954 handle_command_reply: tid=0 2023-10-16T15:34:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 client.9954 shutdown 2023-10-16T15:34:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 msgr2=0x559fca876d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca876d40 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7efd30081d80 tx=0x7efd30032000 comp rx=0 tx=0).stop 2023-10-16T15:34:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 shutdown_connections 2023-10-16T15:34:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:6834/2122514336,v1:172.21.15.29:6835/2122514336] conn(0x559fca8a48d0 0x559fca8a4cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559fca874550 0x559fca876d40 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd1a7fc640 0 client.9954 ms_handle_reset on v2:172.21.15.29:6834/2122514336 2023-10-16T15:34:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x559fca878f40 0x559fca8f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 --2- 172.21.15.29:0/3551296015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559fca874170 0x559fca84d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.445+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 >> 172.21.15.29:0/3551296015 conn(0x559fca8475b0 msgr2=0x559fca8460f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 shutdown_connections 2023-10-16T15:34:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd44de6740 1 -- 172.21.15.29:0/3551296015 wait complete. 2023-10-16T15:34:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd44de6740 20 client.9954 trim_cache size 0 max 16384 2023-10-16T15:34:33.458 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:34:33.458 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:34:33.458 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:34:33.458 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 9816, 2023-10-16T15:34:33.459 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:34:33.459 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:34:33.459 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T15:34:33.459 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8740177439999997, 2023-10-16T15:34:33.459 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 92, 2023-10-16T15:34:33.459 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 192, 2023-10-16T15:34:33.460 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 192, 2023-10-16T15:34:33.460 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.655745875999999 2023-10-16T15:34:33.460 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:34:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7efd2806af90 msgr2=0x7efd2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7efd2806af90 0x7efd2806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd2c000f60 tx=0x7efd2c009040 comp rx=0 tx=0).stop 2023-10-16T15:34:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 msgr2=0x7efd3c0b2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 0x7efd3c0b2310 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7efd300098f0 tx=0x7efd300046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 shutdown_connections 2023-10-16T15:34:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7efd2806af90 0x7efd2806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd3c0ccfb0 0x7efd3c0b2310 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd3c0b2850 0x7efd3c0ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 --2- 172.21.15.29:0/596899053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efd3c06c040 0x7efd3c0acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.446+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 >> 172.21.15.29:0/596899053 conn(0x7efd3c0ce6a0 msgr2=0x7efd3c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.447+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 shutdown_connections 2023-10-16T15:34:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.447+0000 7efd40f1a640 1 -- 172.21.15.29:0/596899053 wait complete. 2023-10-16T15:34:33.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T15:34:33.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.673+0000 7f0d1b59e640 1 Processor -- start 2023-10-16T15:34:33.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.673+0000 7f0d1b59e640 1 -- start start 2023-10-16T15:34:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.675+0000 7f0d1b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0d1c0cd810 0x7f0d1c06fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.675+0000 7f0d1b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0cdca0 con 0x7f0d1c0cd380 2023-10-16T15:34:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.675+0000 7f0d1b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c06bab0 con 0x7f0d1c1599c0 2023-10-16T15:34:33.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.675+0000 7f0d1b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c06f300 con 0x7f0d1c0cd810 2023-10-16T15:34:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.675+0000 7f0d13fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0d1c0cd380 0x7f0d1c06f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50106/0 (socket says 172.21.15.29:50106) 2023-10-16T15:34:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.675+0000 7f0d13fff640 1 -- 172.21.15.29:0/4047212300 learned_addr learned my addr 172.21.15.29:0/4047212300 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d1ad9d640 1 --2- 172.21.15.29:0/4047212300 >> v2:172.21.15.170:3300/0 conn(0x7f0d1c0cd810 0x7f0d1c06fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d19d9b640 1 -- 172.21.15.29:0/4047212300 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454192872 0 0) 0x7f0d1c0cdca0 con 0x7f0d1c0cd380 2023-10-16T15:34:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d19d9b640 1 -- 172.21.15.29:0/4047212300 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d08005730 con 0x7f0d1c0cd380 2023-10-16T15:34:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d19d9b640 1 -- 172.21.15.29:0/4047212300 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086410175 0 0) 0x7f0d08005730 con 0x7f0d1c0cd380 2023-10-16T15:34:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d1ad9d640 1 -- 172.21.15.29:0/4047212300 >> v1:172.21.15.170:6789/0 conn(0x7f0d1c1599c0 legacy=0x7f0d1c0ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d1ad9d640 1 -- 172.21.15.29:0/4047212300 >> v1:172.21.15.29:6789/0 conn(0x7f0d1c0cd380 legacy=0x7f0d1c06f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.676+0000 7f0d1ad9d640 1 -- 172.21.15.29:0/4047212300 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c06bd90 con 0x7f0d1c0cd810 2023-10-16T15:34:33.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.678+0000 7f0d1ad9d640 1 --2- 172.21.15.29:0/4047212300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c0cd810 0x7f0d1c06fe60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009d90 tx=0x7f0d1402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0530b9436f17af5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.678+0000 7f0d19d9b640 1 -- 172.21.15.29:0/4047212300 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04001670 con 0x7f0d1c0cd810 2023-10-16T15:34:33.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.678+0000 7f0d19d9b640 1 -- 172.21.15.29:0/4047212300 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1402c8b0 con 0x7f0d1c0cd810 2023-10-16T15:34:33.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.678+0000 7f0d19d9b640 1 -- 172.21.15.29:0/4047212300 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1402cc40 con 0x7f0d1c0cd810 2023-10-16T15:34:33.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.679+0000 7f0d1b59e640 1 -- 172.21.15.29:0/4047212300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c0cd810 msgr2=0x7f0d1c06fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.679+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/4047212300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c0cd810 0x7f0d1c06fe60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009d90 tx=0x7f0d1402bd80 comp rx=0 tx=0).stop 2023-10-16T15:34:33.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.679+0000 7f0d1b59e640 1 -- 172.21.15.29:0/4047212300 shutdown_connections 2023-10-16T15:34:33.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.679+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/4047212300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c0cd810 0x7f0d1c06fe60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.679+0000 7f0d1b59e640 1 -- 172.21.15.29:0/4047212300 >> 172.21.15.29:0/4047212300 conn(0x7f0d1c0b0a90 msgr2=0x7f0d1c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.680+0000 7f0d1b59e640 1 -- 172.21.15.29:0/4047212300 shutdown_connections 2023-10-16T15:34:33.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.680+0000 7f0d1b59e640 1 -- 172.21.15.29:0/4047212300 wait complete. 2023-10-16T15:34:33.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.681+0000 7f0d1b59e640 1 Processor -- start 2023-10-16T15:34:33.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.682+0000 7f0d1b59e640 1 -- start start 2023-10-16T15:34:33.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.682+0000 7f0d1b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 0x7f0d1c157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d1c0cd810 0x7f0d1c0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c1599c0 0x7f0d1c0a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c157ca0 con 0x7f0d1c0cd380 2023-10-16T15:34:33.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0b3780 con 0x7f0d1c1599c0 2023-10-16T15:34:33.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0b2b70 con 0x7f0d1c0cd810 2023-10-16T15:34:33.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 0x7f0d1c157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 0x7f0d1c157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33526/0 (socket says 172.21.15.29:33526) 2023-10-16T15:34:33.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 -- 172.21.15.29:0/3434087588 learned_addr learned my addr 172.21.15.29:0/3434087588 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:33.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 -- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d1c0cd810 msgr2=0x7f0d1c0b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d1c0cd810 0x7f0d1c0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 -- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c1599c0 msgr2=0x7f0d1c0a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1ad9d640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c1599c0 0x7f0d1c0a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c1599c0 0x7f0d1c0a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.683+0000 7f0d1a59c640 1 -- 172.21.15.29:0/3434087588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d14009a70 con 0x7f0d1c0cd380 2023-10-16T15:34:33.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.684+0000 7f0d1a59c640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 0x7f0d1c157760 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c450 tx=0x7f0d0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.684+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c0052d0 con 0x7f0d1c0cd380 2023-10-16T15:34:33.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.684+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c005430 con 0x7f0d1c0cd380 2023-10-16T15:34:33.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.684+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c012870 con 0x7f0d1c0cd380 2023-10-16T15:34:33.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.684+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c0b2e50 con 0x7f0d1c0cd380 2023-10-16T15:34:33.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.684+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c0b3410 con 0x7f0d1c0cd380 2023-10-16T15:34:33.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.685+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c06c050 con 0x7f0d1c0cd380 2023-10-16T15:34:33.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.686+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0d0c0129d0 con 0x7f0d1c0cd380 2023-10-16T15:34:33.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.687+0000 7f0d12ffd640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cfc06af90 0x7f0cfc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:33.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.687+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0d0c089390 con 0x7f0d1c0cd380 2023-10-16T15:34:33.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.688+0000 7f0d13fff640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cfc06af90 0x7f0cfc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:33.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.689+0000 7f0d13fff640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cfc06af90 0x7f0cfc06d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0a8630 tx=0x7f0d040010a0 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:33.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.691+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0d1c06c050 con 0x7f0d1c0cd380 2023-10-16T15:34:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.926+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0d1c06c050 con 0x7f0d1c0cd380 2023-10-16T15:34:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.927+0000 7f0d12ffd640 1 -- 172.21.15.29:0/3434087588 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0d1c06c050 con 0x7f0d1c0cd380 2023-10-16T15:34:33.929 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T15:34:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cfc06af90 msgr2=0x7f0cfc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cfc06af90 0x7f0cfc06d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0a8630 tx=0x7f0d040010a0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 msgr2=0x7f0d1c157760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 0x7f0d1c157760 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c450 tx=0x7f0d0c00c910 comp rx=0 tx=0).stop 2023-10-16T15:34:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 shutdown_connections 2023-10-16T15:34:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f0cfc06af90 0x7f0cfc06d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0d1c1599c0 0x7f0d1c0a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0d1c0cd810 0x7f0d1c0b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 --2- 172.21.15.29:0/3434087588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d1c0cd380 0x7f0d1c157760 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 >> 172.21.15.29:0/3434087588 conn(0x7f0d1c0b0a90 msgr2=0x7f0d1c070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 shutdown_connections 2023-10-16T15:34:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:33.930+0000 7f0d1b59e640 1 -- 172.21.15.29:0/3434087588 wait complete. 2023-10-16T15:34:33.951 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T15:34:33.952 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T15:34:33.952 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T15:34:33.952 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T15:34:33.953 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T15:34:33.953 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T15:34:33.953 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T15:34:33.954 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T15:34:33.954 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:34:33.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:33.955 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T15:34:34.010 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T15:34:34.010 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T15:34:34.010 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T15:34:34.011 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.011 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T15:34:34.011 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.011 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:34:34.011 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:34:34.012 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T15:34:34.012 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:34:34.012 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:34:34.012 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T15:34:34.012 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T15:34:34.012 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T15:34:34.013 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T15:34:34.013 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T15:34:34.013 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 37651sec preferred_lft 37651sec 2023-10-16T15:34:34.013 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T15:34:34.013 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.014 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:34:34.014 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T15:34:34.014 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T15:34:34.014 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T15:34:34.014 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:34:34.015 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T15:34:34.015 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.015 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T15:34:34.015 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.015 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:34:34.016 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:34:34.016 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T15:34:34.016 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.016 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:34:34.016 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:34:34.017 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T15:34:34.017 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:34:34.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.018 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T15:34:34.121 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T15:34:34.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.127 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T15:34:34.186 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T15:34:34.186 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T15:34:34.187 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T15:34:34.187 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T15:34:34.187 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T15:34:34.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.188 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T15:34:34.188 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T15:34:34.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T15:34:34.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T15:34:34.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T15:34:34.190 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T15:34:34.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.389 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:34:34.444 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:34:34.445 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:34:34.446 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:34:34.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T15:34:34.553 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:34:34.554 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T15:34:34.613 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:34:34.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:34:34.614 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:34:34.663 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T15:34:34.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:34:34.863 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:34:34.864 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:34:34.864 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:34:34.886 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:34:34.886 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:34:34.887 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:34:34.887 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:34:34.887 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:34:34.912 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:34:34.912 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:34:34.912 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:34:34.912 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:34:34.913 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:34:34.913 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T15:34:34.913 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T15:34:34.913 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T15:34:34.914 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T15:34:34.914 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T15:34:34.914 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T15:34:34.915 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T15:34:34.915 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T15:34:34.919 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T15:34:34.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:34.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:34:35.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:35.018 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T15:34:35.086 INFO:teuthology.orchestra.run.smithi029.stdout:client9963 2023-10-16T15:34:35.087 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:34:35.087 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:34:35.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T15:34:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.374+0000 7fbf16490640 1 Processor -- start 2023-10-16T15:34:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.374+0000 7fbf16490640 1 -- start start 2023-10-16T15:34:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf16490640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf1006a4b0 0x7fbf100ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf16490640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf100c9ca0 0x7fbf100ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf16490640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100cb030 con 0x7fbf1015ad70 2023-10-16T15:34:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf16490640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100cae30 con 0x7fbf1006a4b0 2023-10-16T15:34:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf16490640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c91c0 con 0x7fbf100c9ca0 2023-10-16T15:34:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf1006a4b0 0x7fbf100ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.375+0000 7fbf0ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf1015ad70 0x7fbf10069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55570/0 (socket says 172.21.15.29:55570) 2023-10-16T15:34:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf0ffff640 1 -- 172.21.15.29:0/3913268403 learned_addr learned my addr 172.21.15.29:0/3913268403 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:35.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf14a2d640 1 --2- 172.21.15.29:0/3913268403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf100c9ca0 0x7fbf100ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf07fff640 1 -- 172.21.15.29:0/3913268403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf100c9ca0 msgr2=0x7fbf100ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf07fff640 1 --2- 172.21.15.29:0/3913268403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf100c9ca0 0x7fbf100ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf07fff640 1 -- 172.21.15.29:0/3913268403 >> v1:172.21.15.29:6789/0 conn(0x7fbf1015ad70 legacy=0x7fbf10069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf0f7fe640 1 -- 172.21.15.29:0/3913268403 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391909204 0 0) 0x7fbf100cb030 con 0x7fbf1015ad70 2023-10-16T15:34:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.376+0000 7fbf07fff640 1 -- 172.21.15.29:0/3913268403 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100c94a0 con 0x7fbf1006a4b0 2023-10-16T15:34:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.377+0000 7fbf07fff640 1 --2- 172.21.15.29:0/3913268403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1006a4b0 0x7fbf100ca3b0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fbef80097a0 tx=0x7fbef802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2ac5c2d1915988 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf0f7fe640 1 -- 172.21.15.29:0/3913268403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef802d040 con 0x7fbf1006a4b0 2023-10-16T15:34:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf0f7fe640 1 -- 172.21.15.29:0/3913268403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8031810 con 0x7fbf1006a4b0 2023-10-16T15:34:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf0f7fe640 1 -- 172.21.15.29:0/3913268403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8031ba0 con 0x7fbf1006a4b0 2023-10-16T15:34:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf16490640 1 -- 172.21.15.29:0/3913268403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1006a4b0 msgr2=0x7fbf100ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf16490640 1 --2- 172.21.15.29:0/3913268403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1006a4b0 0x7fbf100ca3b0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fbef80097a0 tx=0x7fbef802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf16490640 1 -- 172.21.15.29:0/3913268403 shutdown_connections 2023-10-16T15:34:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf16490640 1 --2- 172.21.15.29:0/3913268403 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf100c9ca0 0x7fbf100ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf16490640 1 --2- 172.21.15.29:0/3913268403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1006a4b0 0x7fbf100ca3b0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.378+0000 7fbf16490640 1 -- 172.21.15.29:0/3913268403 >> 172.21.15.29:0/3913268403 conn(0x7fbf100ce9b0 msgr2=0x7fbf100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.379+0000 7fbf16490640 1 -- 172.21.15.29:0/3913268403 shutdown_connections 2023-10-16T15:34:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.379+0000 7fbf16490640 1 -- 172.21.15.29:0/3913268403 wait complete. 2023-10-16T15:34:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.380+0000 7fbf16490640 1 Processor -- start 2023-10-16T15:34:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.380+0000 7fbf16490640 1 -- start start 2023-10-16T15:34:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.381+0000 7fbf16490640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf1006a4b0 0x7fbf1015df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.381+0000 7fbf16490640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf100c9ca0 0x7fbf10158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.381+0000 7fbf16490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 0x7fbf100c4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.381+0000 7fbf16490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100ae9b0 con 0x7fbf1015ad70 2023-10-16T15:34:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.381+0000 7fbf16490640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c4fc0 con 0x7fbf100c9ca0 2023-10-16T15:34:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.381+0000 7fbf16490640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c4dd0 con 0x7fbf1006a4b0 2023-10-16T15:34:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 0x7fbf100c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 0x7fbf100c4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48414/0 (socket says 172.21.15.29:48414) 2023-10-16T15:34:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 -- 172.21.15.29:0/847768522 learned_addr learned my addr 172.21.15.29:0/847768522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf0ffff640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf1006a4b0 0x7fbf1015df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf07fff640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf100c9ca0 0x7fbf10158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 -- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf1006a4b0 msgr2=0x7fbf1015df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf1006a4b0 0x7fbf1015df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 -- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf100c9ca0 msgr2=0x7fbf10158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf100c9ca0 0x7fbf10158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.382+0000 7fbf14a2d640 1 -- 172.21.15.29:0/847768522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009400 con 0x7fbf1015ad70 2023-10-16T15:34:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.383+0000 7fbf14a2d640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 0x7fbf100c4890 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fbf08007e70 tx=0x7fbf0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.383+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf08014620 con 0x7fbf1015ad70 2023-10-16T15:34:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.383+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf080147a0 con 0x7fbf1015ad70 2023-10-16T15:34:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.383+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf08014b50 con 0x7fbf1015ad70 2023-10-16T15:34:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.383+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf100ae010 con 0x7fbf1015ad70 2023-10-16T15:34:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.383+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf100ae5d0 con 0x7fbf1015ad70 2023-10-16T15:34:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.384+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf100aeaf0 con 0x7fbf1015ad70 2023-10-16T15:34:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.385+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf08014cb0 con 0x7fbf1015ad70 2023-10-16T15:34:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.386+0000 7fbf0d7fa640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fbef006af90 0x7fbef006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.387+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbf0808a220 con 0x7fbf1015ad70 2023-10-16T15:34:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.387+0000 7fbf0ffff640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fbef006af90 0x7fbef006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.388+0000 7fbf0ffff640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fbef006af90 0x7fbef006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf00000f60 tx=0x7fbf00009040 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.391+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf100aeaf0 con 0x7fbf1015ad70 2023-10-16T15:34:35.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.608+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 --> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf100c9780 con 0x7fbef006af90 2023-10-16T15:34:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.611+0000 7fbf0d7fa640 1 -- 172.21.15.29:0/847768522 <== mgr.7085 v2:172.21.15.170:6832/3694096961 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbf000070d0 con 0x7fbef006af90 2023-10-16T15:34:35.613 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:34:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:34:35.613 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T15:34:35.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:34:35.614 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:34:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fbef006af90 msgr2=0x7fbef006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fbef006af90 0x7fbef006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf00000f60 tx=0x7fbf00009040 comp rx=0 tx=0).stop 2023-10-16T15:34:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 msgr2=0x7fbf100c4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 0x7fbf100c4890 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fbf08007e70 tx=0x7fbf0800e4b0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 shutdown_connections 2023-10-16T15:34:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7fbef006af90 0x7fbef006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf1015ad70 0x7fbf100c4890 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.614+0000 7fbf16490640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf100c9ca0 0x7fbf10158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.615+0000 7fbf16490640 1 --2- 172.21.15.29:0/847768522 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf1006a4b0 0x7fbf1015df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.615+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 >> 172.21.15.29:0/847768522 conn(0x7fbf100ce9b0 msgr2=0x7fbf100b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.615+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 shutdown_connections 2023-10-16T15:34:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.615+0000 7fbf16490640 1 -- 172.21.15.29:0/847768522 wait complete. 2023-10-16T15:34:35.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T15:34:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.855+0000 7f5965706640 1 Processor -- start 2023-10-16T15:34:35.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.855+0000 7f5965706640 1 -- start start 2023-10-16T15:34:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.856+0000 7f5965706640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59600ad970 0x7f59600cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.856+0000 7f5965706640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59600ccfb0 0x7f59600cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.856+0000 7f5965706640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596015b9d0 con 0x7f596006c040 2023-10-16T15:34:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f5965706640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596015bcb0 con 0x7f59600ccfb0 2023-10-16T15:34:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f5965706640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596015bf90 con 0x7f59600ad970 2023-10-16T15:34:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f595ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f596006c040 0x7f596006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55584/0 (socket says 172.21.15.29:55584) 2023-10-16T15:34:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f595ffff640 1 -- 172.21.15.29:0/1947794037 learned_addr learned my addr 172.21.15.29:0/1947794037 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f5964f05640 1 --2- 172.21.15.29:0/1947794037 >> v2:172.21.15.29:3300/0 conn(0x7f59600ccfb0 0x7f59600cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f595f7fe640 1 --2- 172.21.15.29:0/1947794037 >> v2:172.21.15.170:3300/0 conn(0x7f59600ad970 0x7f59600cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f595effd640 1 -- 172.21.15.29:0/1947794037 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946725956 0 0) 0x7f596015b9d0 con 0x7f596006c040 2023-10-16T15:34:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.857+0000 7f595effd640 1 -- 172.21.15.29:0/1947794037 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948003520 con 0x7f596006c040 2023-10-16T15:34:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.858+0000 7f5964f05640 1 -- 172.21.15.29:0/1947794037 >> v2:172.21.15.170:3300/0 conn(0x7f59600ad970 msgr2=0x7f59600cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.858+0000 7f5964f05640 1 --2- 172.21.15.29:0/1947794037 >> v2:172.21.15.170:3300/0 conn(0x7f59600ad970 0x7f59600cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.858+0000 7f5964f05640 1 -- 172.21.15.29:0/1947794037 >> v1:172.21.15.29:6789/0 conn(0x7f596006c040 legacy=0x7f596006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.858+0000 7f5964f05640 1 -- 172.21.15.29:0/1947794037 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59600cc270 con 0x7f59600ccfb0 2023-10-16T15:34:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.858+0000 7f595f7fe640 1 --2- 172.21.15.29:0/1947794037 >> v2:172.21.15.170:3300/0 conn(0x7f59600ad970 0x7f59600cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.859+0000 7f5964f05640 1 --2- 172.21.15.29:0/1947794037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ccfb0 0x7f59600cd460 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f5958009d90 tx=0x7f595802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0c260f7effbb02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.859+0000 7f595effd640 1 -- 172.21.15.29:0/1947794037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595802cc70 con 0x7f59600ccfb0 2023-10-16T15:34:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.859+0000 7f595effd640 1 -- 172.21.15.29:0/1947794037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595802cdd0 con 0x7f59600ccfb0 2023-10-16T15:34:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.859+0000 7f595effd640 1 -- 172.21.15.29:0/1947794037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958034660 con 0x7f59600ccfb0 2023-10-16T15:34:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 -- 172.21.15.29:0/1947794037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ccfb0 msgr2=0x7f59600cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 --2- 172.21.15.29:0/1947794037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ccfb0 0x7f59600cd460 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f5958009d90 tx=0x7f595802bec0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 -- 172.21.15.29:0/1947794037 shutdown_connections 2023-10-16T15:34:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 --2- 172.21.15.29:0/1947794037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ccfb0 0x7f59600cd460 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 --2- 172.21.15.29:0/1947794037 >> v2:172.21.15.170:3300/0 conn(0x7f59600ad970 0x7f59600cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 -- 172.21.15.29:0/1947794037 >> 172.21.15.29:0/1947794037 conn(0x7f59600ce6a0 msgr2=0x7f59600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 -- 172.21.15.29:0/1947794037 shutdown_connections 2023-10-16T15:34:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.860+0000 7f5965706640 1 -- 172.21.15.29:0/1947794037 wait complete. 2023-10-16T15:34:35.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.862+0000 7f5965706640 1 Processor -- start 2023-10-16T15:34:35.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.862+0000 7f5965706640 1 -- start start 2023-10-16T15:34:35.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5965706640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f596006c040 0x7f59601efd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5965706640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 0x7f59600c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5965706640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59600ccfb0 0x7f59600c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5965706640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59601f0280 con 0x7f59600ad970 2023-10-16T15:34:35.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5965706640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b3e20 con 0x7f59600ccfb0 2023-10-16T15:34:35.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5965706640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b42b0 con 0x7f596006c040 2023-10-16T15:34:35.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f595f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 0x7f59600c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f595f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 0x7f59600c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48430/0 (socket says 172.21.15.29:48430) 2023-10-16T15:34:35.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f595f7fe640 1 -- 172.21.15.29:0/2026758769 learned_addr learned my addr 172.21.15.29:0/2026758769 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:35.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f5964f05640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59600ccfb0 0x7f59600c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f595f7fe640 1 -- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f596006c040 msgr2=0x7f59601efd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.863+0000 7f595f7fe640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f596006c040 0x7f59601efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595f7fe640 1 -- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59600ccfb0 msgr2=0x7f59600c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:35.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595f7fe640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59600ccfb0 0x7f59600c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:35.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595f7fe640 1 -- 172.21.15.29:0/2026758769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5958009a70 con 0x7f59600ad970 2023-10-16T15:34:35.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595f7fe640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 0x7f59600c74f0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f595400e7e0 tx=0x7f595400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:35.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595400c6d0 con 0x7f59600ad970 2023-10-16T15:34:35.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595400c830 con 0x7f59600ad970 2023-10-16T15:34:35.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595400cbe0 con 0x7f59600ad970 2023-10-16T15:34:35.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59600b3210 con 0x7f59600ad970 2023-10-16T15:34:35.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.864+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59600b37d0 con 0x7f59600ad970 2023-10-16T15:34:35.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.865+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596006b330 con 0x7f59600ad970 2023-10-16T15:34:35.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.867+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f595400cd40 con 0x7f59600ad970 2023-10-16T15:34:35.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.867+0000 7f595cff9640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f594c06af90 0x7f594c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:35.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.868+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f595408a0c0 con 0x7f59600ad970 2023-10-16T15:34:35.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.868+0000 7f595ffff640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f594c06af90 0x7f594c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:35.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.869+0000 7f595ffff640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f594c06af90 0x7f594c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5950001610 tx=0x7f5950007930 comp rx=0 tx=0).ready entity=mgr.7085 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:35.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:35.871+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5954054f10 con 0x7f59600ad970 2023-10-16T15:34:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.116+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f596004e9b0 con 0x7f59600ad970 2023-10-16T15:34:36.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.774+0000 7f595cff9640 1 -- 172.21.15.29:0/2026758769 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v99) v1 ==== 92+0+0 (secure 0 0 0) 0x7f59540210a0 con 0x7f59600ad970 2023-10-16T15:34:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f594c06af90 msgr2=0x7f594c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f594c06af90 0x7f594c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5950001610 tx=0x7f5950007930 comp rx=0 tx=0).stop 2023-10-16T15:34:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 msgr2=0x7f59600c74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 0x7f59600c74f0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f595400e7e0 tx=0x7f595400eca0 comp rx=0 tx=0).stop 2023-10-16T15:34:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 shutdown_connections 2023-10-16T15:34:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f594c06af90 0x7f594c06d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59600ad970 0x7f59600c74f0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59600ccfb0 0x7f59600c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 --2- 172.21.15.29:0/2026758769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f596006c040 0x7f59601efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.778+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 >> 172.21.15.29:0/2026758769 conn(0x7f59600ce6a0 msgr2=0x7f596006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.779+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 shutdown_connections 2023-10-16T15:34:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:36.780+0000 7f5965706640 1 -- 172.21.15.29:0/2026758769 wait complete. 2023-10-16T15:34:36.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T15:34:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.008+0000 7f2ed33b0640 1 Processor -- start 2023-10-16T15:34:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.009+0000 7f2ed33b0640 1 -- start start 2023-10-16T15:34:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.010+0000 7f2ed33b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 0x7f2ecc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.010+0000 7f2ed33b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ecc161900 con 0x7f2ecc06a780 2023-10-16T15:34:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.010+0000 7f2ed33b0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ecc0acdd0 con 0x7f2ecc0ae260 2023-10-16T15:34:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.010+0000 7f2ed33b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc0ce8a0 con 0x7f2ecc160ad0 2023-10-16T15:34:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.011+0000 7f2ed194d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2ecc06a780 0x7f2ecc06ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55604/0 (socket says 172.21.15.29:55604) 2023-10-16T15:34:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.011+0000 7f2ed194d640 1 -- 172.21.15.29:0/666902733 learned_addr learned my addr 172.21.15.29:0/666902733 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:34:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.011+0000 7f2ed114c640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 0x7f2ecc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.011+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892128941 0 0) 0x7f2ecc161900 con 0x7f2ecc06a780 2023-10-16T15:34:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.011+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb8003520 con 0x7f2ecc06a780 2023-10-16T15:34:37.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.011+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2420230504 0 0) 0x7f2eb8003520 con 0x7f2ecc06a780 2023-10-16T15:34:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 msgr2=0x7f2ecc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 0x7f2ecc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 >> v1:172.21.15.170:6789/0 conn(0x7f2ecc0ae260 legacy=0x7f2ecc160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc0ceb80 con 0x7f2ecc06a780 2023-10-16T15:34:37.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2ec40019f0 con 0x7f2ecc06a780 2023-10-16T15:34:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 >> v1:172.21.15.29:6789/0 conn(0x7f2ecc06a780 legacy=0x7f2ecc06ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb80039a0 0x7f2eb8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 0x7f2eb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb8004d00 0x7f2eb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc0ce8a0 con 0x7f2eb80039a0 2023-10-16T15:34:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb800ba30 con 0x7f2eb8004d00 2023-10-16T15:34:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb800bba0 con 0x7f2ecc160ad0 2023-10-16T15:34:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ed114c640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb80039a0 0x7f2eb8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ed094b640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 0x7f2eb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.012+0000 7f2ed194d640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb8004d00 0x7f2eb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.013+0000 7f2ed114c640 1 -- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 msgr2=0x7f2eb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.013+0000 7f2ed114c640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc160ad0 0x7f2eb80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.013+0000 7f2ed114c640 1 -- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb8004d00 msgr2=0x7f2eb8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.013+0000 7f2ed114c640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb8004d00 0x7f2eb8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.013+0000 7f2ed114c640 1 -- 172.21.15.29:0/666902733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8003520 con 0x7f2eb80039a0 2023-10-16T15:34:37.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.014+0000 7f2ed114c640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb80039a0 0x7f2eb8003d80 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009ac0 tx=0x7f2eb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e950d1cf4cbd544b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.014+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec802c6b0 con 0x7f2eb80039a0 2023-10-16T15:34:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.014+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec802c810 con 0x7f2eb80039a0 2023-10-16T15:34:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.014+0000 7f2ec3fff640 1 -- 172.21.15.29:0/666902733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec802cba0 con 0x7f2eb80039a0 2023-10-16T15:34:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed33b0640 1 -- 172.21.15.29:0/666902733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb80039a0 msgr2=0x7f2eb8003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/666902733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb80039a0 0x7f2eb8003d80 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009ac0 tx=0x7f2eb80057f0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed194d640 1 -- 172.21.15.29:0/666902733 reap_dead start 2023-10-16T15:34:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed33b0640 1 -- 172.21.15.29:0/666902733 shutdown_connections 2023-10-16T15:34:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed33b0640 1 -- 172.21.15.29:0/666902733 >> 172.21.15.29:0/666902733 conn(0x7f2ecc06bc10 msgr2=0x7f2ecc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed33b0640 1 -- 172.21.15.29:0/666902733 shutdown_connections 2023-10-16T15:34:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.015+0000 7f2ed33b0640 1 -- 172.21.15.29:0/666902733 wait complete. 2023-10-16T15:34:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.017+0000 7f2ed33b0640 1 Processor -- start 2023-10-16T15:34:37.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.017+0000 7f2ed33b0640 1 -- start start 2023-10-16T15:34:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.018+0000 7f2ed33b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb80039a0 0x7f2ecc15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.018+0000 7f2ed33b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 0x7f2ecc0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.018+0000 7f2ed33b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc0c2cc0 0x7f2ecc0c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.018+0000 7f2ed33b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc15f5d0 con 0x7f2ecc071180 2023-10-16T15:34:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.018+0000 7f2ed33b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc0719e0 con 0x7f2eb80039a0 2023-10-16T15:34:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.018+0000 7f2ed33b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc15e950 con 0x7f2ecc0c2cc0 2023-10-16T15:34:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 0x7f2ecc0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 0x7f2ecc0c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48466/0 (socket says 172.21.15.29:48466) 2023-10-16T15:34:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 -- 172.21.15.29:0/1997165326 learned_addr learned my addr 172.21.15.29:0/1997165326 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed114c640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb80039a0 0x7f2ecc15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed194d640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc0c2cc0 0x7f2ecc0c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 -- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc0c2cc0 msgr2=0x7f2ecc0c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc0c2cc0 0x7f2ecc0c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 -- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb80039a0 msgr2=0x7f2ecc15ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb80039a0 0x7f2ecc15ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed094b640 1 -- 172.21.15.29:0/1997165326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc15ec30 con 0x7f2ecc071180 2023-10-16T15:34:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.019+0000 7f2ed114c640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb80039a0 0x7f2ecc15ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.020+0000 7f2ed094b640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 0x7f2ecc0c47d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00f740 tx=0x7f2ebc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.020+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00fe50 con 0x7f2ecc071180 2023-10-16T15:34:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.020+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0052e0 con 0x7f2ecc071180 2023-10-16T15:34:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.020+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00d730 con 0x7f2ecc071180 2023-10-16T15:34:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.020+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ecc15ef10 con 0x7f2ecc071180 2023-10-16T15:34:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.020+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ecc15f9f0 con 0x7f2ecc071180 2023-10-16T15:34:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.021+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ecc15f710 con 0x7f2ecc071180 2023-10-16T15:34:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.023+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2ebc00d890 con 0x7f2ecc071180 2023-10-16T15:34:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.024+0000 7f2ec1ffb640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 0x7f2eb006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.024+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2ebc0897b0 con 0x7f2ecc071180 2023-10-16T15:34:37.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.024+0000 7f2ed114c640 1 -- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 msgr2=0x7f2eb006d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.025+0000 7f2ed114c640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 0x7f2eb006d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:37.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.029+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ecc15f710 con 0x7f2ecc071180 2023-10-16T15:34:37.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.226+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2ecc15f710 con 0x7f2ecc071180 2023-10-16T15:34:37.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.226+0000 7f2ed114c640 1 -- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 msgr2=0x7f2eb006d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:37.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.226+0000 7f2ed114c640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 0x7f2eb006d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.230+0000 7f2ec1ffb640 1 -- 172.21.15.29:0/1997165326 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2ecc15f710 con 0x7f2ecc071180 2023-10-16T15:34:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 msgr2=0x7f2eb006d4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 0x7f2eb006d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 msgr2=0x7f2ecc0c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 0x7f2ecc0c47d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00f740 tx=0x7f2ebc00fc00 comp rx=0 tx=0).stop 2023-10-16T15:34:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 shutdown_connections 2023-10-16T15:34:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f2eb006b030 0x7f2eb006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ecc071180 0x7f2ecc0c47d0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ecc0c2cc0 0x7f2ecc0c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 --2- 172.21.15.29:0/1997165326 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2eb80039a0 0x7f2ecc15ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 >> 172.21.15.29:0/1997165326 conn(0x7f2ecc06bc10 msgr2=0x7f2ecc0ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 shutdown_connections 2023-10-16T15:34:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.233+0000 7f2ed33b0640 1 -- 172.21.15.29:0/1997165326 wait complete. 2023-10-16T15:34:37.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T15:34:37.324 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:37.323+0000 7f313e613040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:34:37.348 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:37.346+0000 7fa870214040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:34:37.379 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:37.377+0000 7f313e613040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:34:37.406 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:37.405+0000 7fa870214040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:34:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.467+0000 7f968ac5d640 1 Processor -- start 2023-10-16T15:34:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.468+0000 7f968ac5d640 1 -- start start 2023-10-16T15:34:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.469+0000 7f968ac5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f968415ad70 0x7f9684069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.469+0000 7f968ac5d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f96840c9ca0 0x7f968406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.469+0000 7f968ac5d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96840ca3b0 0x7f96840ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.469+0000 7f968ac5d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96840cad80 con 0x7f968415ad70 2023-10-16T15:34:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.469+0000 7f968ac5d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96840cb030 con 0x7f96840c9ca0 2023-10-16T15:34:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.469+0000 7f968ac5d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840c91c0 con 0x7f96840ca3b0 2023-10-16T15:34:37.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f968415ad70 0x7f9684069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f968415ad70 0x7f9684069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48476/0 (socket says 172.21.15.29:48476) 2023-10-16T15:34:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 -- 172.21.15.29:0/2051811496 learned_addr learned my addr 172.21.15.29:0/2051811496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f9683fff640 1 --2- 172.21.15.29:0/2051811496 >> v2:172.21.15.170:3300/0 conn(0x7f96840c9ca0 0x7f968406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96891fa640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96840ca3b0 0x7f96840ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 -- 172.21.15.29:0/2051811496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96840ca3b0 msgr2=0x7f96840ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96840ca3b0 0x7f96840ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 -- 172.21.15.29:0/2051811496 >> v2:172.21.15.170:3300/0 conn(0x7f96840c9ca0 msgr2=0x7f968406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 --2- 172.21.15.29:0/2051811496 >> v2:172.21.15.170:3300/0 conn(0x7f96840c9ca0 0x7f968406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96889f9640 1 -- 172.21.15.29:0/2051811496 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96840c94a0 con 0x7f968415ad70 2023-10-16T15:34:37.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.470+0000 7f96891fa640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96840ca3b0 0x7f96840ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:37.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.471+0000 7f96889f9640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f968415ad70 0x7f9684069da0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f96840ce870 tx=0x7f967802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200aab5e38c74756 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.472+0000 7f96837fe640 1 -- 172.21.15.29:0/2051811496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678031860 con 0x7f968415ad70 2023-10-16T15:34:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.472+0000 7f96837fe640 1 -- 172.21.15.29:0/2051811496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96780319c0 con 0x7f968415ad70 2023-10-16T15:34:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.472+0000 7f96837fe640 1 -- 172.21.15.29:0/2051811496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678031d50 con 0x7f968415ad70 2023-10-16T15:34:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.472+0000 7f968ac5d640 1 -- 172.21.15.29:0/2051811496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f968415ad70 msgr2=0x7f9684069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.472+0000 7f968ac5d640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f968415ad70 0x7f9684069da0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f96840ce870 tx=0x7f967802b900 comp rx=0 tx=0).stop 2023-10-16T15:34:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 -- 172.21.15.29:0/2051811496 shutdown_connections 2023-10-16T15:34:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96840ca3b0 0x7f96840ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 --2- 172.21.15.29:0/2051811496 >> v2:172.21.15.170:3300/0 conn(0x7f96840c9ca0 0x7f968406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 --2- 172.21.15.29:0/2051811496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f968415ad70 0x7f9684069da0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 -- 172.21.15.29:0/2051811496 >> 172.21.15.29:0/2051811496 conn(0x7f96840ce9b0 msgr2=0x7f96840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 -- 172.21.15.29:0/2051811496 shutdown_connections 2023-10-16T15:34:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.473+0000 7f968ac5d640 1 -- 172.21.15.29:0/2051811496 wait complete. 2023-10-16T15:34:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.475+0000 7f968ac5d640 1 Processor -- start 2023-10-16T15:34:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.476+0000 7f968ac5d640 1 -- start start 2023-10-16T15:34:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.477+0000 7f968ac5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 0x7f96841593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.477+0000 7f968ac5d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96840ca3b0 0x7f96841f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.477+0000 7f968ac5d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f968415ad70 0x7f968415d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.477+0000 7f968ac5d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96841598f0 con 0x7f96840c9ca0 2023-10-16T15:34:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.477+0000 7f968ac5d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840cca10 con 0x7f96840ca3b0 2023-10-16T15:34:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.477+0000 7f968ac5d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840cccf0 con 0x7f968415ad70 2023-10-16T15:34:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 0x7f96841593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 0x7f96841593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48486/0 (socket says 172.21.15.29:48486) 2023-10-16T15:34:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 -- 172.21.15.29:0/4262585813 learned_addr learned my addr 172.21.15.29:0/4262585813 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f9683fff640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96840ca3b0 0x7f96841f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96891fa640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f968415ad70 0x7f968415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 -- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f968415ad70 msgr2=0x7f968415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f968415ad70 0x7f968415d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 -- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96840ca3b0 msgr2=0x7f96841f23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96840ca3b0 0x7f96841f23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 -- 172.21.15.29:0/4262585813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780094d0 con 0x7f96840c9ca0 2023-10-16T15:34:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.478+0000 7f96889f9640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 0x7f96841593b0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f967802b940 tx=0x7f96780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.479+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967803ad30 con 0x7f96840c9ca0 2023-10-16T15:34:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.479+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96780443f0 con 0x7f96840c9ca0 2023-10-16T15:34:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.479+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96840ccfd0 con 0x7f96840c9ca0 2023-10-16T15:34:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.479+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96780447a0 con 0x7f96840c9ca0 2023-10-16T15:34:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.479+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96840cd590 con 0x7f96840c9ca0 2023-10-16T15:34:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.480+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968404e160 con 0x7f96840c9ca0 2023-10-16T15:34:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.481+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f967804d450 con 0x7f96840c9ca0 2023-10-16T15:34:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.482+0000 7f96817fa640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 0x7f966806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.483+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f96780abf40 con 0x7f96840c9ca0 2023-10-16T15:34:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.483+0000 7f9683fff640 1 -- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 msgr2=0x7f966806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.483+0000 7f9683fff640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 0x7f966806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.486+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9678076d10 con 0x7f96840c9ca0 2023-10-16T15:34:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.684+0000 7f9683fff640 1 -- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 msgr2=0x7f966806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.684+0000 7f9683fff640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 0x7f966806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:37.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.699+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f96840703c0 con 0x7f96840c9ca0 2023-10-16T15:34:37.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.704+0000 7f96817fa640 1 -- 172.21.15.29:0/4262585813 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9678035070 con 0x7f96840c9ca0 2023-10-16T15:34:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 msgr2=0x7f966806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 0x7f966806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 msgr2=0x7f96841593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 0x7f96841593b0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f967802b940 tx=0x7f96780046c0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 shutdown_connections 2023-10-16T15:34:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f966806afe0 0x7f966806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.708+0000 7f968ac5d640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f968415ad70 0x7f968415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.709+0000 7f968ac5d640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96840ca3b0 0x7f96841f23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.709+0000 7f968ac5d640 1 --2- 172.21.15.29:0/4262585813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96840c9ca0 0x7f96841593b0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:37.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.709+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 >> 172.21.15.29:0/4262585813 conn(0x7f96840ce9b0 msgr2=0x7f968415dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:37.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.709+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 shutdown_connections 2023-10-16T15:34:37.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:37.709+0000 7f968ac5d640 1 -- 172.21.15.29:0/4262585813 wait complete. 2023-10-16T15:34:37.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:37.732 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T15:34:37.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:34:37.794 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-16T15:34:37.831 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:37.829+0000 7f313e613040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:34:37.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-10-16T15:34:37.883 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:37.881+0000 7f313e613040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:34:37.916 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:37.915+0000 7fa870214040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:34:37.932 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:37.930+0000 7f313e613040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:34:37.974 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:37.973+0000 7fa870214040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:34:37.992 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:37.991+0000 7f313e613040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:34:38.028 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.026+0000 7fa870214040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:34:38.042 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.040+0000 7f313e613040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:34:38.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.060+0000 7f1a1359e640 1 Processor -- start 2023-10-16T15:34:38.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.060+0000 7f1a1359e640 1 -- start start 2023-10-16T15:34:38.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a1359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a14153740 0x7f1a1415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a1359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a140ccb00 0x7f1a140ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a1359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1a140ad4d0 0x7f1a140ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a1359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a140cd4a0 con 0x7f1a140ccb00 2023-10-16T15:34:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a1359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1415ac60 con 0x7f1a140ad4d0 2023-10-16T15:34:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a1359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a14070030 con 0x7f1a14153740 2023-10-16T15:34:38.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a0bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a140ccb00 0x7f1a140ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:38.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a0bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a140ccb00 0x7f1a140ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48498/0 (socket says 172.21.15.29:48498) 2023-10-16T15:34:38.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.062+0000 7f1a0bfff640 1 -- 172.21.15.29:0/1148467512 learned_addr learned my addr 172.21.15.29:0/1148467512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a1259c640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a14153740 0x7f1a1415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a12d9d640 1 --2- 172.21.15.29:0/1148467512 >> v2:172.21.15.170:3300/0 conn(0x7f1a140ad4d0 0x7f1a140ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a0bfff640 1 -- 172.21.15.29:0/1148467512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a14153740 msgr2=0x7f1a1415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:38.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a0bfff640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a14153740 0x7f1a1415a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a0bfff640 1 -- 172.21.15.29:0/1148467512 >> v2:172.21.15.170:3300/0 conn(0x7f1a140ad4d0 msgr2=0x7f1a140ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a0bfff640 1 --2- 172.21.15.29:0/1148467512 >> v2:172.21.15.170:3300/0 conn(0x7f1a140ad4d0 0x7f1a140ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a0bfff640 1 -- 172.21.15.29:0/1148467512 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a14070310 con 0x7f1a140ccb00 2023-10-16T15:34:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a1259c640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a14153740 0x7f1a1415a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.063+0000 7f1a12d9d640 1 --2- 172.21.15.29:0/1148467512 >> v2:172.21.15.170:3300/0 conn(0x7f1a140ad4d0 0x7f1a140ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.064+0000 7f1a0bfff640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a140ccb00 0x7f1a140ccf60 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f1a140cf460 tx=0x7f19fc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ba5f783ebf14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.064+0000 7f1a11d9b640 1 -- 172.21.15.29:0/1148467512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc031880 con 0x7f1a140ccb00 2023-10-16T15:34:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a11d9b640 1 -- 172.21.15.29:0/1148467512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc0319e0 con 0x7f1a140ccb00 2023-10-16T15:34:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a11d9b640 1 -- 172.21.15.29:0/1148467512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc031d70 con 0x7f1a140ccb00 2023-10-16T15:34:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 -- 172.21.15.29:0/1148467512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a140ccb00 msgr2=0x7f1a140ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:38.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a140ccb00 0x7f1a140ccf60 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f1a140cf460 tx=0x7f19fc02b920 comp rx=0 tx=0).stop 2023-10-16T15:34:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 -- 172.21.15.29:0/1148467512 shutdown_connections 2023-10-16T15:34:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 --2- 172.21.15.29:0/1148467512 >> v2:172.21.15.170:3300/0 conn(0x7f1a140ad4d0 0x7f1a140ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a140ccb00 0x7f1a140ccf60 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 --2- 172.21.15.29:0/1148467512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a14153740 0x7f1a1415a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.065+0000 7f1a1359e640 1 -- 172.21.15.29:0/1148467512 >> 172.21.15.29:0/1148467512 conn(0x7f1a140ce6a0 msgr2=0x7f1a140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.066+0000 7f1a1359e640 1 -- 172.21.15.29:0/1148467512 shutdown_connections 2023-10-16T15:34:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.066+0000 7f1a1359e640 1 -- 172.21.15.29:0/1148467512 wait complete. 2023-10-16T15:34:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.067+0000 7f1a1359e640 1 Processor -- start 2023-10-16T15:34:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.067+0000 7f1a1359e640 1 -- start start 2023-10-16T15:34:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.068+0000 7f1a1359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a140ad4d0 0x7f1a14152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.068+0000 7f1a1359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a140ccb00 0x7f1a140ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.068+0000 7f1a1359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a14153740 0x7f1a140acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.068+0000 7f1a1359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a14152c70 con 0x7f1a14153740 2023-10-16T15:34:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.068+0000 7f1a1359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a141f25c0 con 0x7f1a140ccb00 2023-10-16T15:34:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.068+0000 7f1a1359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a141f28a0 con 0x7f1a140ad4d0 2023-10-16T15:34:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a14153740 0x7f1a140acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a14153740 0x7f1a140acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48510/0 (socket says 172.21.15.29:48510) 2023-10-16T15:34:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 -- 172.21.15.29:0/3558942145 learned_addr learned my addr 172.21.15.29:0/3558942145 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a140ad4d0 0x7f1a14152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a0bfff640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a140ccb00 0x7f1a140ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a140ad4d0 msgr2=0x7f1a14152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a140ad4d0 0x7f1a14152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a140ccb00 msgr2=0x7f1a140ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a140ccb00 0x7f1a140ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:38.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a12d9d640 1 -- 172.21.15.29:0/3558942145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0094d0 con 0x7f1a14153740 2023-10-16T15:34:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a140ad4d0 0x7f1a14152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.069+0000 7f1a0bfff640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a140ccb00 0x7f1a140ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:34:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.070+0000 7f1a12d9d640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a14153740 0x7f1a140acdd0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00cef0 tx=0x7f1a0c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.070+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00d840 con 0x7f1a14153740 2023-10-16T15:34:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.070+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c00d9a0 con 0x7f1a14153740 2023-10-16T15:34:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.070+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00dd50 con 0x7f1a14153740 2023-10-16T15:34:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.070+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a141f2b80 con 0x7f1a14153740 2023-10-16T15:34:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.070+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a141f3140 con 0x7f1a14153740 2023-10-16T15:34:38.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.071+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a141f3330 con 0x7f1a14153740 2023-10-16T15:34:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.073+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1a0c01e3b0 con 0x7f1a14153740 2023-10-16T15:34:38.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.074+0000 7f1a0affd640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 0x7f19f406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.074+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1a0c08a270 con 0x7f1a14153740 2023-10-16T15:34:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.074+0000 7f1a1259c640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 msgr2=0x7f19f406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.074+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 0x7f19f406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:34:38.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.079+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a0c0551e0 con 0x7f1a14153740 2023-10-16T15:34:38.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.094+0000 7fa870214040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:34:38.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.146+0000 7fa870214040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:34:38.154 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.152+0000 7f313e613040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:34:38.216 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.215+0000 7f313e613040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:38.264 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.263+0000 7fa870214040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:34:38.266 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.265+0000 7f313e613040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:34:38.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.275+0000 7f1a1259c640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 msgr2=0x7f19f406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:38.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.275+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 0x7f19f406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:34:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.292+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 --> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a140adea0 con 0x7f19f406afe0 2023-10-16T15:34:38.323 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.321+0000 7f313e613040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:34:38.330 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.329+0000 7fa870214040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:38.373 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.371+0000 7f313e613040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:34:38.382 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.381+0000 7fa870214040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:34:38.441 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.440+0000 7fa870214040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:34:38.493 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.492+0000 7fa870214040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:34:38.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.676+0000 7f1a1259c640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 msgr2=0x7f19f406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:38.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:38.676+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 0x7f19f406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T15:34:38.751 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.749+0000 7f313e613040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:34:38.891 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:38.889+0000 7fa870214040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:34:38.910 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:38.908+0000 7f313e613040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:34:39.018 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.016+0000 7f313e613040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:34:39.058 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.057+0000 7fa870214040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:34:39.067 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.066+0000 7f313e613040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:34:39.171 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.169+0000 7fa870214040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:34:39.176 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.174+0000 7f313e613040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:39.223 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.222+0000 7fa870214040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:34:39.284 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.282+0000 7f313e613040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:34:39.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.335+0000 7fa870214040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:39.442 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.440+0000 7f313e613040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:39.450 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.448+0000 7fa870214040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:34:39.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:39.477+0000 7f1a1259c640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 msgr2=0x7f19f406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3694096961 2023-10-16T15:34:39.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:39.477+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 0x7f19f406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T15:34:39.492 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.490+0000 7f313e613040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:39.611 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:39.609+0000 7f313e613040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:34:39.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.613+0000 7fa870214040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:34:39.666 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.665+0000 7fa870214040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:34:39.794 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:39.792+0000 7fa870214040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:34:40.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:34:40.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:34:40.446 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:40.445+0000 7f313e613040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:34:40.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:40.672+0000 7fa870214040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:34:40.764 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:40.763+0000 7f313e613040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:34:40.823 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:40.821+0000 7f313e613040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:34:40.938 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:40.936+0000 7f313e613040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:34:40.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:40.955+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 100) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f1a0c0555b0 con 0x7f1a14153740 2023-10-16T15:34:40.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:40.956+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 msgr2=0x7f19f406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:34:40.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:40.956+0000 7f1a0affd640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3694096961,v1:172.21.15.170:6833/3694096961] conn(0x7f19f406afe0 0x7f19f406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:40.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:40.993+0000 7fa870214040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:34:41.057 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:41.056+0000 7fa870214040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:34:41.087 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.084+0000 7f30bdce1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.087 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bdce1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.087 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bdce1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.087 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bdce1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.088 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bdce1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.088 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bac9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.088 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bac9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.088 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bac9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.088 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bac9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.089 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:34:41.085+0000 7f30bac9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:34:41.177 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:34:41.176+0000 7fa870214040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:34:41.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:41.968+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 101) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1a0c01ed90 con 0x7f1a14153740 2023-10-16T15:34:41.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:41.969+0000 7f1a0affd640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f19f40355a0 0x7f19f4035980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:41.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:41.969+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 --> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f19f4035ec0 con 0x7f19f40355a0 2023-10-16T15:34:41.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:41.975+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f19f40355a0 0x7f19f4035980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:41.978+0000 7f1a1259c640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f19f40355a0 0x7f19f4035980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a04004690 tx=0x7f1a04008040 comp rx=0 tx=0).ready entity=mgr.7268 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:42.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.121+0000 7f1a0affd640 1 -- 172.21.15.29:0/3558942145 <== mgr.7268 v2:172.21.15.170:6832/3536897498 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f19f4035ec0 con 0x7f19f40355a0 2023-10-16T15:34:42.122 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-16T15:34:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.124+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f19f40355a0 msgr2=0x7f19f4035980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.124+0000 7f1a1359e640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f19f40355a0 0x7f19f4035980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a04004690 tx=0x7f1a04008040 comp rx=0 tx=0).stop 2023-10-16T15:34:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.124+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a14153740 msgr2=0x7f1a140acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.124+0000 7f1a1359e640 1 --2- 172.21.15.29:0/3558942145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a14153740 0x7f1a140acdd0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00cef0 tx=0x7f1a0c00a710 comp rx=0 tx=0).stop 2023-10-16T15:34:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.125+0000 7f1a12d9d640 1 -- 172.21.15.29:0/3558942145 reap_dead start 2023-10-16T15:34:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.125+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 shutdown_connections 2023-10-16T15:34:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.125+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 >> 172.21.15.29:0/3558942145 conn(0x7f1a140ce6a0 msgr2=0x7f1a140af800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.125+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 shutdown_connections 2023-10-16T15:34:42.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.125+0000 7f1a1359e640 1 -- 172.21.15.29:0/3558942145 wait complete. 2023-10-16T15:34:42.147 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-16T15:34:42.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-10-16T15:34:42.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.380+0000 7f525da10640 1 Processor -- start 2023-10-16T15:34:42.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.380+0000 7f525da10640 1 -- start start 2023-10-16T15:34:42.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f525da10640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f525806a4b0 0x7f52580ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:42.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f525da10640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52580c9ca0 0x7f52580ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f525da10640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52580cb030 con 0x7f525815ad70 2023-10-16T15:34:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f525da10640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52580cae30 con 0x7f52580c9ca0 2023-10-16T15:34:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f525da10640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52580c91c0 con 0x7f525806a4b0 2023-10-16T15:34:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f52577fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52580c9ca0 0x7f52580ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:42.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f52577fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52580c9ca0 0x7f52580ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48514/0 (socket says 172.21.15.29:48514) 2023-10-16T15:34:42.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f52577fe640 1 -- 172.21.15.29:0/1308953846 learned_addr learned my addr 172.21.15.29:0/1308953846 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:42.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.382+0000 7f52567fc640 1 --2- 172.21.15.29:0/1308953846 >> v2:172.21.15.170:3300/0 conn(0x7f525806a4b0 0x7f52580ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.383+0000 7f52577fe640 1 -- 172.21.15.29:0/1308953846 >> v2:172.21.15.170:3300/0 conn(0x7f525806a4b0 msgr2=0x7f52580ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:34:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.383+0000 7f52577fe640 1 --2- 172.21.15.29:0/1308953846 >> v2:172.21.15.170:3300/0 conn(0x7f525806a4b0 0x7f52580ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.383+0000 7f52577fe640 1 -- 172.21.15.29:0/1308953846 >> v1:172.21.15.29:6789/0 conn(0x7f525815ad70 legacy=0x7f5258069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.383+0000 7f52577fe640 1 -- 172.21.15.29:0/1308953846 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52580c94a0 con 0x7f52580c9ca0 2023-10-16T15:34:42.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.384+0000 7f52577fe640 1 --2- 172.21.15.29:0/1308953846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52580c9ca0 0x7f52580ca8f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f52580ce870 tx=0x7f525002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7330754c18098a5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.384+0000 7f5255ffb640 1 -- 172.21.15.29:0/1308953846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250030870 con 0x7f52580c9ca0 2023-10-16T15:34:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.384+0000 7f5255ffb640 1 -- 172.21.15.29:0/1308953846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52500309d0 con 0x7f52580c9ca0 2023-10-16T15:34:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.384+0000 7f5255ffb640 1 -- 172.21.15.29:0/1308953846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250030d60 con 0x7f52580c9ca0 2023-10-16T15:34:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 -- 172.21.15.29:0/1308953846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52580c9ca0 msgr2=0x7f52580ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 --2- 172.21.15.29:0/1308953846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52580c9ca0 0x7f52580ca8f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f52580ce870 tx=0x7f525002b9b0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 -- 172.21.15.29:0/1308953846 shutdown_connections 2023-10-16T15:34:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 --2- 172.21.15.29:0/1308953846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52580c9ca0 0x7f52580ca8f0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 --2- 172.21.15.29:0/1308953846 >> v2:172.21.15.170:3300/0 conn(0x7f525806a4b0 0x7f52580ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 -- 172.21.15.29:0/1308953846 >> 172.21.15.29:0/1308953846 conn(0x7f52580ce9b0 msgr2=0x7f52580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 -- 172.21.15.29:0/1308953846 shutdown_connections 2023-10-16T15:34:42.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.385+0000 7f525da10640 1 -- 172.21.15.29:0/1308953846 wait complete. 2023-10-16T15:34:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.387+0000 7f525da10640 1 Processor -- start 2023-10-16T15:34:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.387+0000 7f525da10640 1 -- start start 2023-10-16T15:34:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f525da10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525806a4b0 0x7f5258158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f525da10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52580c9ca0 0x7f52580aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f525da10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 0x7f52580af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f525da10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52580ae8f0 con 0x7f525815ad70 2023-10-16T15:34:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f525da10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258158f50 con 0x7f52580c9ca0 2023-10-16T15:34:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f525da10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52580ad8e0 con 0x7f525806a4b0 2023-10-16T15:34:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f52577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 0x7f52580af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f52577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 0x7f52580af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48528/0 (socket says 172.21.15.29:48528) 2023-10-16T15:34:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.388+0000 7f52577fe640 1 -- 172.21.15.29:0/790101149 learned_addr learned my addr 172.21.15.29:0/790101149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:34:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52567fc640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52580c9ca0 0x7f52580aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f5256ffd640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525806a4b0 0x7f5258158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52577fe640 1 -- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525806a4b0 msgr2=0x7f5258158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52577fe640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525806a4b0 0x7f5258158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52577fe640 1 -- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52580c9ca0 msgr2=0x7f52580aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52577fe640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52580c9ca0 0x7f52580aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52577fe640 1 -- 172.21.15.29:0/790101149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500094d0 con 0x7f525815ad70 2023-10-16T15:34:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f5256ffd640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525806a4b0 0x7f5258158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:34:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f52577fe640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 0x7f52580af1c0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f525002b9f0 tx=0x7f5250004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250039c90 con 0x7f525815ad70 2023-10-16T15:34:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250039df0 con 0x7f525815ad70 2023-10-16T15:34:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.389+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52580c6740 con 0x7f525815ad70 2023-10-16T15:34:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.390+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52500446c0 con 0x7f525815ad70 2023-10-16T15:34:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.390+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52580ade10 con 0x7f525815ad70 2023-10-16T15:34:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.391+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525804e160 con 0x7f525815ad70 2023-10-16T15:34:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.392+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f525002f3c0 con 0x7f525815ad70 2023-10-16T15:34:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.393+0000 7f524ffff640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f523806ad10 0x7f523806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:34:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.393+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f525003e070 con 0x7f525815ad70 2023-10-16T15:34:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.394+0000 7f5256ffd640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f523806ad10 0x7f523806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:34:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.395+0000 7f5256ffd640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f523806ad10 0x7f523806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52580cf070 tx=0x7f5248009040 comp rx=0 tx=0).ready entity=mgr.7268 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:34:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.398+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5250077570 con 0x7f525815ad70 2023-10-16T15:34:42.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.621+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 --> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f52580c9780 con 0x7f523806ad10 2023-10-16T15:34:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.733+0000 7f524ffff640 1 -- 172.21.15.29:0/790101149 <== mgr.7268 v2:172.21.15.170:6832/3536897498 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f5248003990 con 0x7f523806ad10 2023-10-16T15:34:42.735 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T15:34:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f523806ad10 msgr2=0x7f523806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f523806ad10 0x7f523806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52580cf070 tx=0x7f5248009040 comp rx=0 tx=0).stop 2023-10-16T15:34:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 msgr2=0x7f52580af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:34:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 0x7f52580af1c0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f525002b9f0 tx=0x7f5250004730 comp rx=0 tx=0).stop 2023-10-16T15:34:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 shutdown_connections 2023-10-16T15:34:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f523806ad10 0x7f523806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f525815ad70 0x7f52580af1c0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f52580c9ca0 0x7f52580aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 --2- 172.21.15.29:0/790101149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f525806a4b0 0x7f5258158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:34:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 >> 172.21.15.29:0/790101149 conn(0x7f52580ce9b0 msgr2=0x7f52580cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:34:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 shutdown_connections 2023-10-16T15:34:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:34:42.736+0000 7f525da10640 1 -- 172.21.15.29:0/790101149 wait complete. 2023-10-16T15:34:42.760 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T15:35:00.759 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:35:00.758+0000 7f30acc7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:35:00.759 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:35:00.758+0000 7f30acc7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:35:00.760 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:35:00.758+0000 7f30acc7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:35:00.760 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:35:00.758+0000 7f30acc7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:35:00.760 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:35:00.758+0000 7f30acc7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:35:10.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:35:10.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:35:40.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:35:40.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:36:10.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:36:10.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:36:41.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:36:41.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:37:11.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:37:11.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:37:41.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:37:41.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:38:11.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:38:11.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:38:41.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:38:41.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:39:12.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:39:12.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:39:42.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:39:42.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:40:12.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:40:12.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:40:42.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:40:42.731 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:41:12.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:41:12.951 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:41:43.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:41:43.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:42:13.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:42:13.356 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:42:43.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:42:43.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:43:13.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:43:13.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:43:44.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:43:44.021 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:44:14.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:44:14.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:44:44.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:44:44.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:45:14.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:45:14.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:45:15.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:45:15.761 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T15:45:15.830 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_35_00_UTC 2023-10-16T15:45:15.830 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_36_00_UTC 2023-10-16T15:45:15.830 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_37_00_UTC 2023-10-16T15:45:15.830 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_38_00_UTC 2023-10-16T15:45:15.831 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_39_00_UTC 2023-10-16T15:45:15.831 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_40_00_UTC 2023-10-16T15:45:15.831 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_41_00_UTC 2023-10-16T15:45:15.831 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_42_00_UTC 2023-10-16T15:45:15.831 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_43_01_UTC 2023-10-16T15:45:15.831 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_44_00_UTC 2023-10-16T15:45:15.832 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_45_00_UTC 2023-10-16T15:45:15.833 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-15_35_00_UTC', 'scheduled-2023-10-16-15_36_00_UTC', 'scheduled-2023-10-16-15_37_00_UTC', 'scheduled-2023-10-16-15_38_00_UTC', 'scheduled-2023-10-16-15_39_00_UTC', 'scheduled-2023-10-16-15_40_00_UTC', 'scheduled-2023-10-16-15_41_00_UTC', 'scheduled-2023-10-16-15_42_00_UTC', 'scheduled-2023-10-16-15_43_01_UTC', 'scheduled-2023-10-16-15_44_00_UTC', 'scheduled-2023-10-16-15_45_00_UTC'] 2023-10-16T15:45:15.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T15:45:16.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.059+0000 7f7d70ad1640 1 Processor -- start 2023-10-16T15:45:16.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.059+0000 7f7d70ad1640 1 -- start start 2023-10-16T15:45:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d70ad1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7d6c0cd800 0x7f7d6c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d70ad1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0ce180 con 0x7f7d6c15cba0 2023-10-16T15:45:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d70ad1640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0d0500 con 0x7f7d6c06aeb0 2023-10-16T15:45:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d70ad1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6c0d07c0 con 0x7f7d6c0cd800 2023-10-16T15:45:16.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d6b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7d6c15cba0 0x7f7d6c071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56498/0 (socket says 172.21.15.29:56498) 2023-10-16T15:45:16.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d6b7fe640 1 -- 172.21.15.29:0/2947757609 learned_addr learned my addr 172.21.15.29:0/2947757609 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:45:16.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d63fff640 1 --2- 172.21.15.29:0/2947757609 >> v2:172.21.15.29:3300/0 conn(0x7f7d6c0cd800 0x7f7d6c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.061+0000 7f7d6affd640 1 -- 172.21.15.29:0/2947757609 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734417431 0 0) 0x7f7d6c0ce180 con 0x7f7d6c15cba0 2023-10-16T15:45:16.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.062+0000 7f7d6affd640 1 -- 172.21.15.29:0/2947757609 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d58003520 con 0x7f7d6c15cba0 2023-10-16T15:45:16.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.062+0000 7f7d63fff640 1 -- 172.21.15.29:0/2947757609 >> v1:172.21.15.170:6789/0 conn(0x7f7d6c06aeb0 legacy=0x7f7d6c06b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:45:16.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.062+0000 7f7d63fff640 1 -- 172.21.15.29:0/2947757609 >> v1:172.21.15.29:6789/0 conn(0x7f7d6c15cba0 legacy=0x7f7d6c071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.062+0000 7f7d63fff640 1 -- 172.21.15.29:0/2947757609 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0d0aa0 con 0x7f7d6c0cd800 2023-10-16T15:45:16.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.063+0000 7f7d63fff640 1 --2- 172.21.15.29:0/2947757609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0cdc40 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f7d540097a0 tx=0x7f7d5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c42d7c0bfe0216 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:45:16.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.063+0000 7f7d6affd640 1 -- 172.21.15.29:0/2947757609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031830 con 0x7f7d6c0cd800 2023-10-16T15:45:16.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.063+0000 7f7d6affd640 1 -- 172.21.15.29:0/2947757609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54031990 con 0x7f7d6c0cd800 2023-10-16T15:45:16.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.063+0000 7f7d6affd640 1 -- 172.21.15.29:0/2947757609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031d20 con 0x7f7d6c0cd800 2023-10-16T15:45:16.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 -- 172.21.15.29:0/2947757609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 msgr2=0x7f7d6c0cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/2947757609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0cdc40 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f7d540097a0 tx=0x7f7d5402b8d0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 -- 172.21.15.29:0/2947757609 shutdown_connections 2023-10-16T15:45:16.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/2947757609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0cdc40 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 -- 172.21.15.29:0/2947757609 >> 172.21.15.29:0/2947757609 conn(0x7f7d6c065ed0 msgr2=0x7f7d6c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:45:16.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 -- 172.21.15.29:0/2947757609 shutdown_connections 2023-10-16T15:45:16.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.064+0000 7f7d70ad1640 1 -- 172.21.15.29:0/2947757609 wait complete. 2023-10-16T15:45:16.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.066+0000 7f7d70ad1640 1 Processor -- start 2023-10-16T15:45:16.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.066+0000 7f7d70ad1640 1 -- start start 2023-10-16T15:45:16.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d70ad1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d6c06aeb0 0x7f7d6c1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d70ad1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d70ad1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d6c15cba0 0x7f7d6c0ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d70ad1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c1f2960 con 0x7f7d6c0cd800 2023-10-16T15:45:16.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d70ad1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0b08e0 con 0x7f7d6c15cba0 2023-10-16T15:45:16.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d70ad1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0b0320 con 0x7f7d6c06aeb0 2023-10-16T15:45:16.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d63fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d63fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34128/0 (socket says 172.21.15.29:34128) 2023-10-16T15:45:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.067+0000 7f7d63fff640 1 -- 172.21.15.29:0/693274867 learned_addr learned my addr 172.21.15.29:0/693274867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:45:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d6b7fe640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d6c06aeb0 0x7f7d6c1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d6bfff640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d6c15cba0 0x7f7d6c0ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d63fff640 1 -- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d6c06aeb0 msgr2=0x7f7d6c1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d63fff640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d6c06aeb0 0x7f7d6c1591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d63fff640 1 -- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d6c15cba0 msgr2=0x7f7d6c0ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d63fff640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d6c15cba0 0x7f7d6c0ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d63fff640 1 -- 172.21.15.29:0/693274867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54009400 con 0x7f7d6c0cd800 2023-10-16T15:45:16.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.068+0000 7f7d63fff640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0ad2a0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f7d5402bee0 tx=0x7f7d540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:45:16.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.069+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5403a6f0 con 0x7f7d6c0cd800 2023-10-16T15:45:16.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.069+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5403a870 con 0x7f7d6c0cd800 2023-10-16T15:45:16.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.069+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c0b0600 con 0x7f7d6c0cd800 2023-10-16T15:45:16.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.069+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5403ac20 con 0x7f7d6c0cd800 2023-10-16T15:45:16.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.069+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c1f2e30 con 0x7f7d6c0cd800 2023-10-16T15:45:16.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.070+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6c1f3020 con 0x7f7d6c0cd800 2023-10-16T15:45:16.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.071+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 105) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f7d5404c3b0 con 0x7f7d6c0cd800 2023-10-16T15:45:16.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.072+0000 7f7d68ff9640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f7d4c06afe0 0x7f7d4c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.072+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f7d540ab830 con 0x7f7d6c0cd800 2023-10-16T15:45:16.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.073+0000 7f7d6b7fe640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f7d4c06afe0 0x7f7d4c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.074+0000 7f7d6b7fe640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f7d4c06afe0 0x7f7d4c06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0011e0 tx=0x7f7d5c009040 comp rx=0 tx=0).ready entity=mgr.7268 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:45:16.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.077+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7d540764a0 con 0x7f7d6c0cd800 2023-10-16T15:45:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.291+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 --> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d6c0d1060 con 0x7f7d4c06afe0 2023-10-16T15:45:16.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.310+0000 7f7d68ff9640 1 -- 172.21.15.29:0/693274867 <== mgr.7268 v2:172.21.15.170:6832/3536897498 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f7d5c0012d0 con 0x7f7d4c06afe0 2023-10-16T15:45:16.312 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:45:16.312 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T15:34:41", "first": "2023-10-16T15:35:00", "last": "2023-10-16T15:45:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T15:45:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f7d4c06afe0 msgr2=0x7f7d4c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f7d4c06afe0 0x7f7d4c06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0011e0 tx=0x7f7d5c009040 comp rx=0 tx=0).stop 2023-10-16T15:45:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 msgr2=0x7f7d6c0ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0ad2a0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f7d5402bee0 tx=0x7f7d540046c0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 shutdown_connections 2023-10-16T15:45:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f7d4c06afe0 0x7f7d4c06d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d6c0cd800 0x7f7d6c0ad2a0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d6c15cba0 0x7f7d6c0ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 --2- 172.21.15.29:0/693274867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d6c06aeb0 0x7f7d6c1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.312+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 >> 172.21.15.29:0/693274867 conn(0x7f7d6c065ed0 msgr2=0x7f7d6c0c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:45:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.313+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 shutdown_connections 2023-10-16T15:45:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.313+0000 7f7d70ad1640 1 -- 172.21.15.29:0/693274867 wait complete. 2023-10-16T15:45:16.334 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T15:34:41", "first": "2023-10-16T15:35:00", "last": "2023-10-16T15:45:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T15:45:16.335 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T15:34:41', 'first': '2023-10-16T15:35:00', 'last': '2023-10-16T15:45:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-16T15:45:16.335 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T15:45:16.335 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T15:45:16.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T15:45:16.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.556+0000 7f5e8f4fc640 1 Processor -- start 2023-10-16T15:45:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.557+0000 7f5e8f4fc640 1 -- start start 2023-10-16T15:45:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.558+0000 7f5e8f4fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5e8806c040 0x7f5e8806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.558+0000 7f5e8f4fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e8815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.558+0000 7f5e8f4fc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8806c960 con 0x7f5e880ad970 2023-10-16T15:45:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.558+0000 7f5e8f4fc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8815b9d0 con 0x7f5e8806c040 2023-10-16T15:45:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.558+0000 7f5e8f4fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8815bcb0 con 0x7f5e880cda80 2023-10-16T15:45:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.559+0000 7f5e8ecfb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e8815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.559+0000 7f5e8ecfb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e8815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:58528/0 (socket says 172.21.15.29:58528) 2023-10-16T15:45:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.559+0000 7f5e8ecfb640 1 -- 172.21.15.29:0/3563123230 learned_addr learned my addr 172.21.15.29:0/3563123230 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:45:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.559+0000 7f5e8e4fa640 1 --2- 172.21.15.29:0/3563123230 >> v2:172.21.15.170:3300/0 conn(0x7f5e8806c040 0x7f5e8806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.560+0000 7f5e8ecfb640 1 -- 172.21.15.29:0/3563123230 >> v2:172.21.15.170:3300/0 conn(0x7f5e8806c040 msgr2=0x7f5e8806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:45:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.560+0000 7f5e8ecfb640 1 --2- 172.21.15.29:0/3563123230 >> v2:172.21.15.170:3300/0 conn(0x7f5e8806c040 0x7f5e8806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.560+0000 7f5e8ecfb640 1 -- 172.21.15.29:0/3563123230 >> v1:172.21.15.170:6789/0 conn(0x7f5e880ad970 legacy=0x7f5e8815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.560+0000 7f5e8ecfb640 1 -- 172.21.15.29:0/3563123230 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8815bf90 con 0x7f5e880cda80 2023-10-16T15:45:16.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.561+0000 7f5e8ecfb640 1 --2- 172.21.15.29:0/3563123230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e8815ed90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009e50 tx=0x7f5e8002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=603943d71e464c18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:45:16.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.562+0000 7f5e8d4f8640 1 -- 172.21.15.29:0/3563123230 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8002cb00 con 0x7f5e880cda80 2023-10-16T15:45:16.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.562+0000 7f5e8d4f8640 1 -- 172.21.15.29:0/3563123230 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8002cc60 con 0x7f5e880cda80 2023-10-16T15:45:16.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.562+0000 7f5e8d4f8640 1 -- 172.21.15.29:0/3563123230 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e800343f0 con 0x7f5e880cda80 2023-10-16T15:45:16.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.562+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/3563123230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 msgr2=0x7f5e8815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.562+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/3563123230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e8815ed90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009e50 tx=0x7f5e8002bf60 comp rx=0 tx=0).stop 2023-10-16T15:45:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.563+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/3563123230 shutdown_connections 2023-10-16T15:45:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.563+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/3563123230 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e8815ed90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.563+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/3563123230 >> v2:172.21.15.170:3300/0 conn(0x7f5e8806c040 0x7f5e8806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.563+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/3563123230 >> 172.21.15.29:0/3563123230 conn(0x7f5e880ce6a0 msgr2=0x7f5e880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:45:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.563+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/3563123230 shutdown_connections 2023-10-16T15:45:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.563+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/3563123230 wait complete. 2023-10-16T15:45:16.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.565+0000 7f5e8f4fc640 1 Processor -- start 2023-10-16T15:45:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.565+0000 7f5e8f4fc640 1 -- start start 2023-10-16T15:45:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8f4fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e8806c040 0x7f5e880c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8f4fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 0x7f5e880c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8f4fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e881534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8f4fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880c76e0 con 0x7f5e880ad970 2023-10-16T15:45:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8f4fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880c6ad0 con 0x7f5e8806c040 2023-10-16T15:45:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8f4fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880c6db0 con 0x7f5e880cda80 2023-10-16T15:45:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8dcf9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 0x7f5e880c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8dcf9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 0x7f5e880c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34144/0 (socket says 172.21.15.29:34144) 2023-10-16T15:45:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8dcf9640 1 -- 172.21.15.29:0/4108348693 learned_addr learned my addr 172.21.15.29:0/4108348693 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:45:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8e4fa640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e8806c040 0x7f5e880c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.566+0000 7f5e8ecfb640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e881534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8dcf9640 1 -- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 msgr2=0x7f5e881534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8dcf9640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e881534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8dcf9640 1 -- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e8806c040 msgr2=0x7f5e880c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8dcf9640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e8806c040 0x7f5e880c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8dcf9640 1 -- 172.21.15.29:0/4108348693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e80009b30 con 0x7f5e880ad970 2023-10-16T15:45:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8dcf9640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 0x7f5e880c9830 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800c440 tx=0x7f5e7800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:45:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780052d0 con 0x7f5e880ad970 2023-10-16T15:45:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e78005430 con 0x7f5e880ad970 2023-10-16T15:45:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78007260 con 0x7f5e880ad970 2023-10-16T15:45:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.567+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e880c7090 con 0x7f5e880ad970 2023-10-16T15:45:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.568+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e880c66c0 con 0x7f5e880ad970 2023-10-16T15:45:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.569+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8806b330 con 0x7f5e880ad970 2023-10-16T15:45:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.570+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 105) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f5e780073c0 con 0x7f5e880ad970 2023-10-16T15:45:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.571+0000 7f5e7effd640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f5e6c06afe0 0x7f5e6c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:45:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.571+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5e78088cc0 con 0x7f5e880ad970 2023-10-16T15:45:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.571+0000 7f5e8e4fa640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f5e6c06afe0 0x7f5e6c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:45:16.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.573+0000 7f5e8e4fa640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f5e6c06afe0 0x7f5e6c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009a30 tx=0x7f5e84008040 comp rx=0 tx=0).ready entity=mgr.7268 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:45:16.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.575+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e78053ae0 con 0x7f5e880ad970 2023-10-16T15:45:16.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.818+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5e880c6850 con 0x7f5e880ad970 2023-10-16T15:45:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.827+0000 7f5e7effd640 1 -- 172.21.15.29:0/4108348693 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v105) v1 ==== 74+0+171247 (secure 0 0 0) 0x7f5e78057100 con 0x7f5e880ad970 2023-10-16T15:45:16.831 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:45:16.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f5e6c06afe0 msgr2=0x7f5e6c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f5e6c06afe0 0x7f5e6c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009a30 tx=0x7f5e84008040 comp rx=0 tx=0).stop 2023-10-16T15:45:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 msgr2=0x7f5e880c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:45:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 0x7f5e880c9830 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800c440 tx=0x7f5e7800c900 comp rx=0 tx=0).stop 2023-10-16T15:45:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 shutdown_connections 2023-10-16T15:45:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:6832/3536897498,v1:172.21.15.170:6833/3536897498] conn(0x7f5e6c06afe0 0x7f5e6c06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e880ad970 0x7f5e880c9830 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e880cda80 0x7f5e881534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 --2- 172.21.15.29:0/4108348693 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e8806c040 0x7f5e880c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:45:16.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 >> 172.21.15.29:0/4108348693 conn(0x7f5e880ce6a0 msgr2=0x7f5e880c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:45:16.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.832+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 shutdown_connections 2023-10-16T15:45:16.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:45:16.833+0000 7f5e8f4fc640 1 -- 172.21.15.29:0/4108348693 wait complete. 2023-10-16T15:45:16.854 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":105,"active_gid":7268,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":3536897498},{"type":"v1","addr":"172.21.15.170:6833","nonce":3536897498}]},"active_addr":"172.21.15.170:6833/3536897498","active_change":"2023-10-16T15:34:40.941484+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":7265,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":193,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":1363341747}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":1525143382}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":3590601432}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":3372934420}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":3770772716}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":2183916363}]}]} 2023-10-16T15:45:16.859 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-10-16T15:45:16.859 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T15:45:16.860 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-10-16T15:45:16.860 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T15:45:16.860 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:45:16.936 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T15:45:16.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T15:45:16.942 INFO:tasks.ceph.mgr.x:Started 2023-10-16T15:45:17.541 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:17.539+0000 7f3120e12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:45:17.594 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:17.592+0000 7f3120e12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:45:18.035 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.033+0000 7f3120e12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:45:18.087 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.085+0000 7f3120e12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:45:18.136 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.134+0000 7f3120e12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:45:18.195 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.194+0000 7f3120e12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:45:18.244 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.242+0000 7f3120e12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:45:18.356 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.354+0000 7f3120e12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:45:18.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.416+0000 7f3120e12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:45:18.467 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.465+0000 7f3120e12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:45:18.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.521+0000 7f3120e12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:45:18.572 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.570+0000 7f3120e12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:45:18.951 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:18.949+0000 7f3120e12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:45:19.109 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.107+0000 7f3120e12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:45:19.216 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.214+0000 7f3120e12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:45:19.265 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.264+0000 7f3120e12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:45:19.373 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.372+0000 7f3120e12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:45:19.480 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.478+0000 7f3120e12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:45:19.636 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.633+0000 7f3120e12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:45:19.685 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.683+0000 7f3120e12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:45:19.804 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:19.802+0000 7f3120e12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:45:20.631 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:20.629+0000 7f3120e12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:45:20.926 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:20.924+0000 7f3120e12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:45:20.982 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:20.980+0000 7f3120e12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:45:21.096 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.095+0000 7f3120e12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:45:21.288 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309cc5f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.288 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309cc5f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309cc5f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309cc5f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309cc5f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309fca5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309fca5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.290 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309fca5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.290 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309fca5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:21.290 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:45:21.286+0000 7f309fca5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:45:44.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:45:44.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:46:00.291 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:46:00.288+0000 7f30a7cb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:46:00.291 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:46:00.289+0000 7f30a7cb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:46:00.291 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:46:00.289+0000 7f30a7cb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:46:00.291 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:46:00.289+0000 7f30a7cb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:46:00.292 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:46:00.289+0000 7f30a7cb5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:46:14.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:46:14.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:46:45.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:46:45.193 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:47:15.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:47:15.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:47:45.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:47:45.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:48:15.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:48:15.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:48:45.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:48:46.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:49:16.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:49:16.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:49:46.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:49:46.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:50:16.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:50:16.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:50:16.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-10-16T15:50:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.185+0000 7fa80b410640 1 Processor -- start 2023-10-16T15:50:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.185+0000 7fa80b410640 1 -- start start 2023-10-16T15:50:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa80b410640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8040ce200 0x7fa8040c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa80b410640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80406b860 con 0x7fa8040ade50 2023-10-16T15:50:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa80b410640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80406bb40 con 0x7fa8040c1a70 2023-10-16T15:50:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa80b410640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80406be20 con 0x7fa8040ce200 2023-10-16T15:50:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa8091ac640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8040ade50 0x7fa80406c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42988/0 (socket says 172.21.15.29:42988) 2023-10-16T15:50:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa8091ac640 1 -- 172.21.15.29:0/757914470 learned_addr learned my addr 172.21.15.29:0/757914470 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:17.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa8099ad640 1 --2- 172.21.15.29:0/757914470 >> v2:172.21.15.29:3300/0 conn(0x7fa8040ce200 0x7fa8040c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa8099ad640 1 -- 172.21.15.29:0/757914470 >> v1:172.21.15.170:6789/0 conn(0x7fa8040c1a70 legacy=0x7fa8040cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa7fbfff640 1 -- 172.21.15.29:0/757914470 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394218333 0 0) 0x7fa80406b860 con 0x7fa8040ade50 2023-10-16T15:50:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.187+0000 7fa8099ad640 1 -- 172.21.15.29:0/757914470 >> v1:172.21.15.29:6789/0 conn(0x7fa8040ade50 legacy=0x7fa80406c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.188+0000 7fa8099ad640 1 -- 172.21.15.29:0/757914470 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80406c100 con 0x7fa8040ce200 2023-10-16T15:50:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.189+0000 7fa8099ad640 1 --2- 172.21.15.29:0/757914470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040ce200 0x7fa8040c8e90 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0097a0 tx=0x7fa7fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2156a68c7a466df6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.189+0000 7fa7fbfff640 1 -- 172.21.15.29:0/757914470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc031830 con 0x7fa8040ce200 2023-10-16T15:50:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.189+0000 7fa7fbfff640 1 -- 172.21.15.29:0/757914470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc031990 con 0x7fa8040ce200 2023-10-16T15:50:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.189+0000 7fa7fbfff640 1 -- 172.21.15.29:0/757914470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc031d20 con 0x7fa8040ce200 2023-10-16T15:50:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 -- 172.21.15.29:0/757914470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040ce200 msgr2=0x7fa8040c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 --2- 172.21.15.29:0/757914470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040ce200 0x7fa8040c8e90 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0097a0 tx=0x7fa7fc02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 -- 172.21.15.29:0/757914470 shutdown_connections 2023-10-16T15:50:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 --2- 172.21.15.29:0/757914470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040ce200 0x7fa8040c8e90 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 -- 172.21.15.29:0/757914470 >> 172.21.15.29:0/757914470 conn(0x7fa8040ce840 msgr2=0x7fa8040cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 -- 172.21.15.29:0/757914470 shutdown_connections 2023-10-16T15:50:17.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.190+0000 7fa80b410640 1 -- 172.21.15.29:0/757914470 wait complete. 2023-10-16T15:50:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.192+0000 7fa80b410640 1 Processor -- start 2023-10-16T15:50:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.192+0000 7fa80b410640 1 -- start start 2023-10-16T15:50:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa80b410640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8040ade50 0x7fa8041f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa80b410640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 0x7fa8040b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa80b410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8040ce200 0x7fa8041e9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa80b410640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040c8460 con 0x7fa8040c1a70 2023-10-16T15:50:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa80b410640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040c7850 con 0x7fa8040ade50 2023-10-16T15:50:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa80b410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040c7b30 con 0x7fa8040ce200 2023-10-16T15:50:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa8089ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 0x7fa8040b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa8089ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 0x7fa8040b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38744/0 (socket says 172.21.15.29:38744) 2023-10-16T15:50:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.193+0000 7fa8089ab640 1 -- 172.21.15.29:0/1122277614 learned_addr learned my addr 172.21.15.29:0/1122277614 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8091ac640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8040ade50 0x7fa8041f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8099ad640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8040ce200 0x7fa8041e9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8089ab640 1 -- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8040ce200 msgr2=0x7fa8041e9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8089ab640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8040ce200 0x7fa8041e9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8089ab640 1 -- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8040ade50 msgr2=0x7fa8041f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8089ab640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8040ade50 0x7fa8041f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8089ab640 1 -- 172.21.15.29:0/1122277614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc009400 con 0x7fa8040c1a70 2023-10-16T15:50:17.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.194+0000 7fa8089ab640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 0x7fa8040b3b10 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400c410 tx=0x7fa7f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:17.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.195+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40035f0 con 0x7fa8040c1a70 2023-10-16T15:50:17.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.195+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f4003750 con 0x7fa8040c1a70 2023-10-16T15:50:17.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.195+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8040c7e10 con 0x7fa8040c1a70 2023-10-16T15:50:17.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.195+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4003b40 con 0x7fa8040c1a70 2023-10-16T15:50:17.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.195+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8040c5bb0 con 0x7fa8040c1a70 2023-10-16T15:50:17.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.196+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8040cab80 con 0x7fa8040c1a70 2023-10-16T15:50:17.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.197+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 111) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fa7f4003ca0 con 0x7fa8040c1a70 2023-10-16T15:50:17.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.198+0000 7fa7f9ffb640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fa7e806afe0 0x7fa7e806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.198+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fa7f4088d60 con 0x7fa8040c1a70 2023-10-16T15:50:17.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.199+0000 7fa8091ac640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fa7e806afe0 0x7fa7e806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.200+0000 7fa8091ac640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fa7e806afe0 0x7fa7e806d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa8000012e0 tx=0x7fa800007ef0 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:17.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.204+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7f4053990 con 0x7fa8040c1a70 2023-10-16T15:50:17.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.420+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 --> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa80406c3e0 con 0x7fa7e806afe0 2023-10-16T15:50:17.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.567+0000 7fa7f9ffb640 1 -- 172.21.15.29:0/1122277614 <== mgr.8176 v2:172.21.15.170:6832/23414019 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa800005c70 con 0x7fa7e806afe0 2023-10-16T15:50:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T15:50:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.569+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fa7e806afe0 msgr2=0x7fa7e806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.569+0000 7fa80b410640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fa7e806afe0 0x7fa7e806d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa8000012e0 tx=0x7fa800007ef0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.569+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 msgr2=0x7fa8040b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.569+0000 7fa80b410640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 0x7fa8040b3b10 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400c410 tx=0x7fa7f400c8d0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 shutdown_connections 2023-10-16T15:50:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fa7e806afe0 0x7fa7e806d4a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8040c1a70 0x7fa8040b3b10 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8040ce200 0x7fa8041e9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 --2- 172.21.15.29:0/1122277614 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8040ade50 0x7fa8041f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 >> 172.21.15.29:0/1122277614 conn(0x7fa8040ce840 msgr2=0x7fa804160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 shutdown_connections 2023-10-16T15:50:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.570+0000 7fa80b410640 1 -- 172.21.15.29:0/1122277614 wait complete. 2023-10-16T15:50:17.594 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T15:50:17.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:17.595 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T15:50:17.660 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_35_00_UTC 2023-10-16T15:50:17.660 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_36_00_UTC 2023-10-16T15:50:17.660 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_37_00_UTC 2023-10-16T15:50:17.660 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_38_00_UTC 2023-10-16T15:50:17.660 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_39_00_UTC 2023-10-16T15:50:17.661 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_40_00_UTC 2023-10-16T15:50:17.661 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_41_00_UTC 2023-10-16T15:50:17.661 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_42_00_UTC 2023-10-16T15:50:17.661 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_43_01_UTC 2023-10-16T15:50:17.661 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_44_00_UTC 2023-10-16T15:50:17.661 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_45_00_UTC 2023-10-16T15:50:17.662 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_46_00_UTC 2023-10-16T15:50:17.662 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_47_00_UTC 2023-10-16T15:50:17.662 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_48_00_UTC 2023-10-16T15:50:17.662 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_49_00_UTC 2023-10-16T15:50:17.662 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_50_00_UTC 2023-10-16T15:50:17.663 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-15_35_00_UTC', 'scheduled-2023-10-16-15_36_00_UTC', 'scheduled-2023-10-16-15_37_00_UTC', 'scheduled-2023-10-16-15_38_00_UTC', 'scheduled-2023-10-16-15_39_00_UTC', 'scheduled-2023-10-16-15_40_00_UTC', 'scheduled-2023-10-16-15_41_00_UTC', 'scheduled-2023-10-16-15_42_00_UTC', 'scheduled-2023-10-16-15_43_01_UTC', 'scheduled-2023-10-16-15_44_00_UTC', 'scheduled-2023-10-16-15_45_00_UTC', 'scheduled-2023-10-16-15_46_00_UTC', 'scheduled-2023-10-16-15_47_00_UTC', 'scheduled-2023-10-16-15_48_00_UTC', 'scheduled-2023-10-16-15_49_00_UTC', 'scheduled-2023-10-16-15_50_00_UTC'] 2023-10-16T15:50:17.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T15:50:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.878+0000 7f8a292d7640 1 Processor -- start 2023-10-16T15:50:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.878+0000 7f8a292d7640 1 -- start start 2023-10-16T15:50:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a292d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a2415cd70 0x7f8a240d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a292d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a240b8480 0x7f8a240cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a292d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8a240d0be0 0x7f8a240cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a292d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2415d150 con 0x7f8a240b8480 2023-10-16T15:50:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a292d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240d1180 con 0x7f8a240d0be0 2023-10-16T15:50:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a292d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240d1460 con 0x7f8a2415cd70 2023-10-16T15:50:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a240b8480 0x7f8a240cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a240b8480 0x7f8a240cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38754/0 (socket says 172.21.15.29:38754) 2023-10-16T15:50:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a1bfff640 1 -- 172.21.15.29:0/2867396848 learned_addr learned my addr 172.21.15.29:0/2867396848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a1bfff640 1 -- 172.21.15.29:0/2867396848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a2415cd70 msgr2=0x7f8a240d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a1bfff640 1 --2- 172.21.15.29:0/2867396848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a2415cd70 0x7f8a240d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a1bfff640 1 -- 172.21.15.29:0/2867396848 >> v2:172.21.15.170:3300/0 conn(0x7f8a240d0be0 msgr2=0x7f8a240cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.880+0000 7f8a28ad6640 1 --2- 172.21.15.29:0/2867396848 >> v2:172.21.15.170:3300/0 conn(0x7f8a240d0be0 0x7f8a240cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.881+0000 7f8a1bfff640 1 --2- 172.21.15.29:0/2867396848 >> v2:172.21.15.170:3300/0 conn(0x7f8a240d0be0 0x7f8a240cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.881+0000 7f8a1bfff640 1 -- 172.21.15.29:0/2867396848 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240d1740 con 0x7f8a240b8480 2023-10-16T15:50:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.881+0000 7f8a28ad6640 1 --2- 172.21.15.29:0/2867396848 >> v2:172.21.15.170:3300/0 conn(0x7f8a240d0be0 0x7f8a240cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.882+0000 7f8a1bfff640 1 --2- 172.21.15.29:0/2867396848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a240b8480 0x7f8a240cd7c0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f8a240b2520 tx=0x7f8a0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c41fc62ef3b02d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.882+0000 7f8a237fe640 1 -- 172.21.15.29:0/2867396848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c031870 con 0x7f8a240b8480 2023-10-16T15:50:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.882+0000 7f8a237fe640 1 -- 172.21.15.29:0/2867396848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c0319d0 con 0x7f8a240b8480 2023-10-16T15:50:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.882+0000 7f8a237fe640 1 -- 172.21.15.29:0/2867396848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c031d60 con 0x7f8a240b8480 2023-10-16T15:50:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 -- 172.21.15.29:0/2867396848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a240b8480 msgr2=0x7f8a240cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 --2- 172.21.15.29:0/2867396848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a240b8480 0x7f8a240cd7c0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f8a240b2520 tx=0x7f8a0c02b910 comp rx=0 tx=0).stop 2023-10-16T15:50:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 -- 172.21.15.29:0/2867396848 shutdown_connections 2023-10-16T15:50:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 --2- 172.21.15.29:0/2867396848 >> v2:172.21.15.170:3300/0 conn(0x7f8a240d0be0 0x7f8a240cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 --2- 172.21.15.29:0/2867396848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a240b8480 0x7f8a240cd7c0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 --2- 172.21.15.29:0/2867396848 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a2415cd70 0x7f8a240d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 -- 172.21.15.29:0/2867396848 >> 172.21.15.29:0/2867396848 conn(0x7f8a240cf570 msgr2=0x7f8a240cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 -- 172.21.15.29:0/2867396848 shutdown_connections 2023-10-16T15:50:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.883+0000 7f8a292d7640 1 -- 172.21.15.29:0/2867396848 wait complete. 2023-10-16T15:50:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.885+0000 7f8a292d7640 1 Processor -- start 2023-10-16T15:50:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.885+0000 7f8a292d7640 1 -- start start 2023-10-16T15:50:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a292d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a240b8480 0x7f8a241529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a292d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a240d0be0 0x7f8a24154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a292d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 0x7f8a24154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a292d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2415e6f0 con 0x7f8a2415cd70 2023-10-16T15:50:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a292d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2415ecb0 con 0x7f8a240b8480 2023-10-16T15:50:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a292d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2415f140 con 0x7f8a240d0be0 2023-10-16T15:50:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a28ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 0x7f8a24154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a28ad6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 0x7f8a24154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38760/0 (socket says 172.21.15.29:38760) 2023-10-16T15:50:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a28ad6640 1 -- 172.21.15.29:0/1846799189 learned_addr learned my addr 172.21.15.29:0/1846799189 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a23fff640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a240b8480 0x7f8a241529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.886+0000 7f8a1bfff640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a240d0be0 0x7f8a24154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a28ad6640 1 -- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a240d0be0 msgr2=0x7f8a24154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a28ad6640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a240d0be0 0x7f8a24154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a28ad6640 1 -- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a240b8480 msgr2=0x7f8a241529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a28ad6640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a240b8480 0x7f8a241529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a28ad6640 1 -- 172.21.15.29:0/1846799189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c0094d0 con 0x7f8a2415cd70 2023-10-16T15:50:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a1bfff640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a240d0be0 0x7f8a24154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a28ad6640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 0x7f8a24154b70 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00ce70 tx=0x7f8a1c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00d840 con 0x7f8a2415cd70 2023-10-16T15:50:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c00d9a0 con 0x7f8a2415cd70 2023-10-16T15:50:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24152560 con 0x7f8a2415cd70 2023-10-16T15:50:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00dd50 con 0x7f8a2415cd70 2023-10-16T15:50:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.887+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2415fe90 con 0x7f8a2415cd70 2023-10-16T15:50:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.889+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2404e9b0 con 0x7f8a2415cd70 2023-10-16T15:50:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.889+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 111) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f8a1c01e3b0 con 0x7f8a2415cd70 2023-10-16T15:50:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.890+0000 7f8a217fa640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8a0406b010 0x7f8a0406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.891+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8a1c08a4a0 con 0x7f8a2415cd70 2023-10-16T15:50:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.891+0000 7f8a23fff640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8a0406b010 0x7f8a0406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.892+0000 7f8a23fff640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8a0406b010 0x7f8a0406d4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8a240b6a90 tx=0x7f8a14008040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:17.895+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8a1c055160 con 0x7f8a2415cd70 2023-10-16T15:50:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.110+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 --> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a240d1d00 con 0x7f8a0406b010 2023-10-16T15:50:18.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.128+0000 7f8a217fa640 1 -- 172.21.15.29:0/1846799189 <== mgr.8176 v2:172.21.15.170:6832/23414019 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f8a14009480 con 0x7f8a0406b010 2023-10-16T15:50:18.130 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:18.130 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T15:34:41", "first": "2023-10-16T15:35:00", "last": "2023-10-16T15:50:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T15:50:18.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.130+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8a0406b010 msgr2=0x7f8a0406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.130+0000 7f8a292d7640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8a0406b010 0x7f8a0406d4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8a240b6a90 tx=0x7f8a14008040 comp rx=0 tx=0).stop 2023-10-16T15:50:18.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.130+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 msgr2=0x7f8a24154b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.130+0000 7f8a292d7640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 0x7f8a24154b70 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00ce70 tx=0x7f8a1c00a710 comp rx=0 tx=0).stop 2023-10-16T15:50:18.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 shutdown_connections 2023-10-16T15:50:18.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8a0406b010 0x7f8a0406d4d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a2415cd70 0x7f8a24154b70 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8a240d0be0 0x7f8a24154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 --2- 172.21.15.29:0/1846799189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8a240b8480 0x7f8a241529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 >> 172.21.15.29:0/1846799189 conn(0x7f8a240cf570 msgr2=0x7f8a240b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:18.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 shutdown_connections 2023-10-16T15:50:18.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.131+0000 7f8a292d7640 1 -- 172.21.15.29:0/1846799189 wait complete. 2023-10-16T15:50:18.153 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T15:34:41", "first": "2023-10-16T15:35:00", "last": "2023-10-16T15:50:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T15:50:18.154 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T15:34:41', 'first': '2023-10-16T15:35:00', 'last': '2023-10-16T15:50:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-16T15:50:18.154 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-16T15:50:18.154 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-16T15:50:18.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-10-16T15:50:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.392+0000 7f8f817f0640 1 Processor -- start 2023-10-16T15:50:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.393+0000 7f8f817f0640 1 -- start start 2023-10-16T15:50:18.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.394+0000 7f8f817f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8f7c0b8480 0x7f8f7c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:18.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.394+0000 7f8f817f0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8f7c0d0be0 0x7f8f7c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:18.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.394+0000 7f8f817f0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c15d150 con 0x7f8f7c15cd70 2023-10-16T15:50:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.394+0000 7f8f817f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7c0d1180 con 0x7f8f7c0b8480 2023-10-16T15:50:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.394+0000 7f8f817f0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7c0d1460 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8f7c0b8480 0x7f8f7c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8f7c0b8480 0x7f8f7c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38772/0 (socket says 172.21.15.29:38772) 2023-10-16T15:50:18.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/3419948102 learned_addr learned my addr 172.21.15.29:0/3419948102 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:18.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f80fef640 1 --2- 172.21.15.29:0/3419948102 >> v2:172.21.15.170:3300/0 conn(0x7f8f7c0d0be0 0x7f8f7c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:18.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/3419948102 >> v2:172.21.15.170:3300/0 conn(0x7f8f7c0d0be0 msgr2=0x7f8f7c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:18.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 --2- 172.21.15.29:0/3419948102 >> v2:172.21.15.170:3300/0 conn(0x7f8f7c0d0be0 0x7f8f7c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/3419948102 >> v1:172.21.15.170:6789/0 conn(0x7f8f7c15cd70 legacy=0x7f8f7c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:18.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/3419948102 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0d1740 con 0x7f8f7c0b8480 2023-10-16T15:50:18.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.395+0000 7f8f80fef640 1 --2- 172.21.15.29:0/3419948102 >> v2:172.21.15.170:3300/0 conn(0x7f8f7c0d0be0 0x7f8f7c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.396+0000 7f8f7b7fe640 1 --2- 172.21.15.29:0/3419948102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0b8480 0x7f8f7c0cd7c0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0b8910 tx=0x7f8f7002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56022fc8fb1709ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.397+0000 7f8f7affd640 1 -- 172.21.15.29:0/3419948102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700317e0 con 0x7f8f7c0b8480 2023-10-16T15:50:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.397+0000 7f8f7affd640 1 -- 172.21.15.29:0/3419948102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f70031940 con 0x7f8f7c0b8480 2023-10-16T15:50:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.397+0000 7f8f7affd640 1 -- 172.21.15.29:0/3419948102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70031cd0 con 0x7f8f7c0b8480 2023-10-16T15:50:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.397+0000 7f8f817f0640 1 -- 172.21.15.29:0/3419948102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0b8480 msgr2=0x7f8f7c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.397+0000 7f8f817f0640 1 --2- 172.21.15.29:0/3419948102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0b8480 0x7f8f7c0cd7c0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0b8910 tx=0x7f8f7002b880 comp rx=0 tx=0).stop 2023-10-16T15:50:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.398+0000 7f8f817f0640 1 -- 172.21.15.29:0/3419948102 shutdown_connections 2023-10-16T15:50:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.398+0000 7f8f817f0640 1 --2- 172.21.15.29:0/3419948102 >> v2:172.21.15.170:3300/0 conn(0x7f8f7c0d0be0 0x7f8f7c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.398+0000 7f8f817f0640 1 --2- 172.21.15.29:0/3419948102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0b8480 0x7f8f7c0cd7c0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.398+0000 7f8f817f0640 1 -- 172.21.15.29:0/3419948102 >> 172.21.15.29:0/3419948102 conn(0x7f8f7c0cf570 msgr2=0x7f8f7c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.398+0000 7f8f817f0640 1 -- 172.21.15.29:0/3419948102 shutdown_connections 2023-10-16T15:50:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.398+0000 7f8f817f0640 1 -- 172.21.15.29:0/3419948102 wait complete. 2023-10-16T15:50:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.399+0000 7f8f817f0640 1 Processor -- start 2023-10-16T15:50:18.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.399+0000 7f8f817f0640 1 -- start start 2023-10-16T15:50:18.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.400+0000 7f8f817f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f7c0b8480 0x7f8f7c0ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:18.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.400+0000 7f8f817f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 0x7f8f7c0cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:18.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.400+0000 7f8f817f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f7c15cd70 0x7f8f7c0ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:18.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.400+0000 7f8f817f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0c9bc0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.400+0000 7f8f817f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0aa320 con 0x7f8f7c15cd70 2023-10-16T15:50:18.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.400+0000 7f8f817f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0aa600 con 0x7f8f7c0b8480 2023-10-16T15:50:18.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 0x7f8f7c0cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:18.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 0x7f8f7c0cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38782/0 (socket says 172.21.15.29:38782) 2023-10-16T15:50:18.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/2652542585 learned_addr learned my addr 172.21.15.29:0/2652542585 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:18.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f80fef640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f7c15cd70 0x7f8f7c0ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:18.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7bfff640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f7c0b8480 0x7f8f7c0ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:18.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f7c0b8480 msgr2=0x7f8f7c0ab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f7c0b8480 0x7f8f7c0ab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f7c15cd70 msgr2=0x7f8f7c0ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f7c15cd70 0x7f8f7c0ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f7b7fe640 1 -- 172.21.15.29:0/2652542585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f700094d0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.401+0000 7f8f80fef640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f7c15cd70 0x7f8f7c0ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:18.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.402+0000 7f8f7b7fe640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 0x7f8f7c0cc490 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8f7002b8c0 tx=0x7f8f700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:18.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.402+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700329c0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.402+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f70032b40 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.402+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f700453f0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.402+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c0aa8e0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.402+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c0a9f00 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.403+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c0aa0f0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.404+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 111) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f8f7004d3f0 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.405+0000 7f8f78ff9640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8f6806b010 0x7f8f6806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:18.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.405+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8f700aba40 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.406+0000 7f8f7bfff640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8f6806b010 0x7f8f6806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:18.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.407+0000 7f8f7bfff640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8f6806b010 0x7f8f6806d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c001db0 tx=0x7f8f6c009040 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:18.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.410+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f700ade60 con 0x7f8f7c0d0be0 2023-10-16T15:50:18.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.625+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 --> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f7c0d1d00 con 0x7f8f6806b010 2023-10-16T15:50:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.794+0000 7f8f78ff9640 1 -- 172.21.15.29:0/2652542585 <== mgr.8176 v2:172.21.15.170:6832/23414019 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f8f6c0019a0 con 0x7f8f6806b010 2023-10-16T15:50:18.796 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T15:50:18.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8f6806b010 msgr2=0x7f8f6806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8f6806b010 0x7f8f6806d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c001db0 tx=0x7f8f6c009040 comp rx=0 tx=0).stop 2023-10-16T15:50:18.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 msgr2=0x7f8f7c0cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:18.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 0x7f8f7c0cc490 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8f7002b8c0 tx=0x7f8f700046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 shutdown_connections 2023-10-16T15:50:18.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f8f6806b010 0x7f8f6806d4d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f7c0d0be0 0x7f8f7c0cc490 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f7c15cd70 0x7f8f7c0ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 --2- 172.21.15.29:0/2652542585 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f7c0b8480 0x7f8f7c0ab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:18.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 >> 172.21.15.29:0/2652542585 conn(0x7f8f7c0cf570 msgr2=0x7f8f7c0cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:18.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 shutdown_connections 2023-10-16T15:50:18.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:18.798+0000 7f8f817f0640 1 -- 172.21.15.29:0/2652542585 wait complete. 2023-10-16T15:50:18.821 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T15:50:18.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:18.821 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T15:50:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_35_00_UTC 2023-10-16T15:50:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_36_00_UTC 2023-10-16T15:50:18.880 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_37_00_UTC 2023-10-16T15:50:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_38_00_UTC 2023-10-16T15:50:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_39_00_UTC 2023-10-16T15:50:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_40_00_UTC 2023-10-16T15:50:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_41_00_UTC 2023-10-16T15:50:18.881 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_42_00_UTC 2023-10-16T15:50:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_43_01_UTC 2023-10-16T15:50:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_44_00_UTC 2023-10-16T15:50:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_45_00_UTC 2023-10-16T15:50:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_46_00_UTC 2023-10-16T15:50:18.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_47_00_UTC 2023-10-16T15:50:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_48_00_UTC 2023-10-16T15:50:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_49_00_UTC 2023-10-16T15:50:18.883 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_50_00_UTC 2023-10-16T15:50:18.884 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_35_00_UTC 2023-10-16T15:50:18.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:18.885 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_35_00_UTC) 2023-10-16T15:50:18.956 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_36_00_UTC 2023-10-16T15:50:18.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:18.957 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_36_00_UTC) 2023-10-16T15:50:19.028 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_37_00_UTC 2023-10-16T15:50:19.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.028 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_37_00_UTC) 2023-10-16T15:50:19.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_38_00_UTC 2023-10-16T15:50:19.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.095 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_38_00_UTC) 2023-10-16T15:50:19.167 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_39_00_UTC 2023-10-16T15:50:19.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.168 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_39_00_UTC) 2023-10-16T15:50:19.241 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_40_00_UTC 2023-10-16T15:50:19.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.242 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_40_00_UTC) 2023-10-16T15:50:19.316 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_41_00_UTC 2023-10-16T15:50:19.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.317 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_41_00_UTC) 2023-10-16T15:50:19.388 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_42_00_UTC 2023-10-16T15:50:19.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.389 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_42_00_UTC) 2023-10-16T15:50:19.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_43_01_UTC 2023-10-16T15:50:19.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.464 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_43_01_UTC) 2023-10-16T15:50:19.535 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_44_00_UTC 2023-10-16T15:50:19.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.537 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_44_00_UTC) 2023-10-16T15:50:19.614 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_45_00_UTC 2023-10-16T15:50:19.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.615 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_45_00_UTC) 2023-10-16T15:50:19.688 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_46_00_UTC 2023-10-16T15:50:19.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.689 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_46_00_UTC) 2023-10-16T15:50:19.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_47_00_UTC 2023-10-16T15:50:19.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.765 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_47_00_UTC) 2023-10-16T15:50:19.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_48_00_UTC 2023-10-16T15:50:19.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.841 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_48_00_UTC) 2023-10-16T15:50:19.916 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_49_00_UTC 2023-10-16T15:50:19.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.916 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_49_00_UTC) 2023-10-16T15:50:19.991 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_50_00_UTC 2023-10-16T15:50:19.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:19.992 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-15_50_00_UTC) 2023-10-16T15:50:20.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:20.067 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-16T15:50:20.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:50:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.367+0000 7efe48cf3640 1 Processor -- start 2023-10-16T15:50:20.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.368+0000 7efe48cf3640 1 -- start start 2023-10-16T15:50:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.369+0000 7efe48cf3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efe44159950 0x7efe4415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.369+0000 7efe48cf3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe44070450 0x7efe440708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.369+0000 7efe48cf3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe44070df0 con 0x7efe4406c100 2023-10-16T15:50:20.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.369+0000 7efe48cf3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4415e490 con 0x7efe44159950 2023-10-16T15:50:20.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.369+0000 7efe48cf3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4415aef0 con 0x7efe44070450 2023-10-16T15:50:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.370+0000 7efe42ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe44070450 0x7efe440708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.370+0000 7efe437fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efe44159950 0x7efe4415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.370+0000 7efe437fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efe44159950 0x7efe4415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:45054/0 (socket says 172.21.15.29:45054) 2023-10-16T15:50:20.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.370+0000 7efe437fe640 1 -- 172.21.15.29:0/1408576550 learned_addr learned my addr 172.21.15.29:0/1408576550 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.371+0000 7efe437fe640 1 -- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe44070450 msgr2=0x7efe440708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.371+0000 7efe437fe640 1 --2- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe44070450 0x7efe440708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.371+0000 7efe437fe640 1 -- 172.21.15.29:0/1408576550 >> v1:172.21.15.170:6789/0 conn(0x7efe4406c100 legacy=0x7efe4406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:20.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.371+0000 7efe437fe640 1 -- 172.21.15.29:0/1408576550 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4406cc80 con 0x7efe44159950 2023-10-16T15:50:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.371+0000 7efe42ffd640 1 --2- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe44070450 0x7efe440708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:50:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.373+0000 7efe437fe640 1 --2- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe4415a9b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7efe34009e50 tx=0x7efe3402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bfc21085bcaad61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe427fc640 1 -- 172.21.15.29:0/1408576550 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3402ca80 con 0x7efe44159950 2023-10-16T15:50:20.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe427fc640 1 -- 172.21.15.29:0/1408576550 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3402cbe0 con 0x7efe44159950 2023-10-16T15:50:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe427fc640 1 -- 172.21.15.29:0/1408576550 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe340343f0 con 0x7efe44159950 2023-10-16T15:50:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe48cf3640 1 -- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 msgr2=0x7efe4415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe4415a9b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7efe34009e50 tx=0x7efe3402bee0 comp rx=0 tx=0).stop 2023-10-16T15:50:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe48cf3640 1 -- 172.21.15.29:0/1408576550 shutdown_connections 2023-10-16T15:50:20.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe44070450 0x7efe440708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:20.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1408576550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe4415a9b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:20.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.374+0000 7efe48cf3640 1 -- 172.21.15.29:0/1408576550 >> 172.21.15.29:0/1408576550 conn(0x7efe440ce6a0 msgr2=0x7efe440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:20.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.375+0000 7efe48cf3640 1 -- 172.21.15.29:0/1408576550 shutdown_connections 2023-10-16T15:50:20.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.375+0000 7efe48cf3640 1 -- 172.21.15.29:0/1408576550 wait complete. 2023-10-16T15:50:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.376+0000 7efe48cf3640 1 Processor -- start 2023-10-16T15:50:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.377+0000 7efe48cf3640 1 -- start start 2023-10-16T15:50:20.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe48cf3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe4406c100 0x7efe440b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe48cf3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 0x7efe44157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe48cf3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe440ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe48cf3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe441567e0 con 0x7efe44070450 2023-10-16T15:50:20.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe48cf3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe440b4240 con 0x7efe44159950 2023-10-16T15:50:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe48cf3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe44158580 con 0x7efe4406c100 2023-10-16T15:50:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe42ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 0x7efe44157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe42ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 0x7efe44157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38788/0 (socket says 172.21.15.29:38788) 2023-10-16T15:50:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe42ffd640 1 -- 172.21.15.29:0/1213410934 learned_addr learned my addr 172.21.15.29:0/1213410934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:20.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe43fff640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe440ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe437fe640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe4406c100 0x7efe440b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe42ffd640 1 -- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe4406c100 msgr2=0x7efe440b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.378+0000 7efe42ffd640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe4406c100 0x7efe440b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:20.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe42ffd640 1 -- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 msgr2=0x7efe440ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:20.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe42ffd640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe440ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:20.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe42ffd640 1 -- 172.21.15.29:0/1213410934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe38009400 con 0x7efe44070450 2023-10-16T15:50:20.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe437fe640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe4406c100 0x7efe440b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:20.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe43fff640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe440ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:20.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe42ffd640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 0x7efe44157c10 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7efe3800e990 tx=0x7efe3800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:20.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3800c9e0 con 0x7efe44070450 2023-10-16T15:50:20.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3800cb40 con 0x7efe44070450 2023-10-16T15:50:20.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3800ced0 con 0x7efe44070450 2023-10-16T15:50:20.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe34009b30 con 0x7efe44070450 2023-10-16T15:50:20.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.379+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe44158b40 con 0x7efe44070450 2023-10-16T15:50:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.381+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe44156a50 con 0x7efe44070450 2023-10-16T15:50:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.382+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 111) v1 ==== 89360+0+0 (secure 0 0 0) 0x7efe38013470 con 0x7efe44070450 2023-10-16T15:50:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.383+0000 7efe2bfff640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7efe3006afe0 0x7efe3006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.383+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6425+0+0 (secure 0 0 0) 0x7efe38089d00 con 0x7efe44070450 2023-10-16T15:50:20.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.383+0000 7efe437fe640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7efe3006afe0 0x7efe3006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.384+0000 7efe437fe640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7efe3006afe0 0x7efe3006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe34035000 tx=0x7efe34030000 comp rx=0 tx=0).ready entity=mgr.8176 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:20.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.387+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efe38054a60 con 0x7efe44070450 2023-10-16T15:50:20.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:20.633+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7efe4404e9b0 con 0x7efe44070450 2023-10-16T15:50:21.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.177+0000 7efe2bfff640 1 -- 172.21.15.29:0/1213410934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v112) v1 ==== 93+0+0 (secure 0 0 0) 0x7efe38058080 con 0x7efe44070450 2023-10-16T15:50:21.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.183+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7efe3006afe0 msgr2=0x7efe3006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:21.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.183+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7efe3006afe0 0x7efe3006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe34035000 tx=0x7efe34030000 comp rx=0 tx=0).stop 2023-10-16T15:50:21.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.183+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 msgr2=0x7efe44157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:21.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.183+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 0x7efe44157c10 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7efe3800e990 tx=0x7efe3800ee50 comp rx=0 tx=0).stop 2023-10-16T15:50:21.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.185+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 shutdown_connections 2023-10-16T15:50:21.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.185+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7efe3006afe0 0x7efe3006d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.185+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe44070450 0x7efe44157c10 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.185+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe44159950 0x7efe440ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.185+0000 7efe48cf3640 1 --2- 172.21.15.29:0/1213410934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe4406c100 0x7efe440b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.185+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 >> 172.21.15.29:0/1213410934 conn(0x7efe440ce6a0 msgr2=0x7efe44072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:21.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.186+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 shutdown_connections 2023-10-16T15:50:21.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.186+0000 7efe48cf3640 1 -- 172.21.15.29:0/1213410934 wait complete. 2023-10-16T15:50:21.222 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:50:21.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:50:21.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:50:21.437 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:50:21.438 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:50:21.438 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:21.478 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:50:21.478 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:21.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:21.810 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:21.809+0000 7fde53c1a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:50:21.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:21.839+0000 7f06c261f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:50:21.868 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:21.866+0000 7fde53c1a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:50:21.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.869+0000 7fe3e0d18640 1 Processor -- start 2023-10-16T15:50:21.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.869+0000 7fe3e0d18640 1 -- start start 2023-10-16T15:50:21.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3e0d18640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe3dc159b20 0x7fe3dc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:21.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3e0d18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3dc0b0b20 0x7fe3dc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:21.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3e0d18640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc06f7f0 con 0x7fe3dc159f00 2023-10-16T15:50:21.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3e0d18640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc06c5e0 con 0x7fe3dc0b0b20 2023-10-16T15:50:21.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3e0d18640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc06c8c0 con 0x7fe3dc159b20 2023-10-16T15:50:21.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3dad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3dc0b0b20 0x7fe3dc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:21.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3dad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3dc0b0b20 0x7fe3dc06fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38816/0 (socket says 172.21.15.29:38816) 2023-10-16T15:50:21.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.870+0000 7fe3dad9d640 1 -- 172.21.15.29:0/760253751 learned_addr learned my addr 172.21.15.29:0/760253751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:21.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.871+0000 7fe3da59c640 1 --2- 172.21.15.29:0/760253751 >> v2:172.21.15.170:3300/0 conn(0x7fe3dc159b20 0x7fe3dc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:21.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.871+0000 7fe3dad9d640 1 -- 172.21.15.29:0/760253751 >> v2:172.21.15.170:3300/0 conn(0x7fe3dc159b20 msgr2=0x7fe3dc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:21.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.871+0000 7fe3dad9d640 1 --2- 172.21.15.29:0/760253751 >> v2:172.21.15.170:3300/0 conn(0x7fe3dc159b20 0x7fe3dc06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.871+0000 7fe3dad9d640 1 -- 172.21.15.29:0/760253751 >> v1:172.21.15.170:6789/0 conn(0x7fe3dc159f00 legacy=0x7fe3dc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:21.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.871+0000 7fe3dad9d640 1 -- 172.21.15.29:0/760253751 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc06cba0 con 0x7fe3dc0b0b20 2023-10-16T15:50:21.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.871+0000 7fe3da59c640 1 --2- 172.21.15.29:0/760253751 >> v2:172.21.15.170:3300/0 conn(0x7fe3dc159b20 0x7fe3dc06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:21.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.872+0000 7fe3dad9d640 1 --2- 172.21.15.29:0/760253751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0b20 0x7fe3dc06fc80 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0c9340 tx=0x7fe3d402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71550c1647c6be80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:21.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.872+0000 7fe3d959a640 1 -- 172.21.15.29:0/760253751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d402ea90 con 0x7fe3dc0b0b20 2023-10-16T15:50:21.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.872+0000 7fe3d959a640 1 -- 172.21.15.29:0/760253751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d402ebf0 con 0x7fe3dc0b0b20 2023-10-16T15:50:21.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.872+0000 7fe3d959a640 1 -- 172.21.15.29:0/760253751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4009600 con 0x7fe3dc0b0b20 2023-10-16T15:50:21.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 -- 172.21.15.29:0/760253751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0b20 msgr2=0x7fe3dc06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:21.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/760253751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0b20 0x7fe3dc06fc80 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0c9340 tx=0x7fe3d402e0f0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 -- 172.21.15.29:0/760253751 shutdown_connections 2023-10-16T15:50:21.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/760253751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc0b0b20 0x7fe3dc06fc80 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/760253751 >> v2:172.21.15.170:3300/0 conn(0x7fe3dc159b20 0x7fe3dc06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 -- 172.21.15.29:0/760253751 >> 172.21.15.29:0/760253751 conn(0x7fe3dc0cb410 msgr2=0x7fe3dc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:21.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 -- 172.21.15.29:0/760253751 shutdown_connections 2023-10-16T15:50:21.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.873+0000 7fe3e0d18640 1 -- 172.21.15.29:0/760253751 wait complete. 2023-10-16T15:50:21.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.876+0000 7fe3e0d18640 1 Processor -- start 2023-10-16T15:50:21.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.876+0000 7fe3e0d18640 1 -- start start 2023-10-16T15:50:21.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3e0d18640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3dc0b0b20 0x7fe3dc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3e0d18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 0x7fe3dc0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3e0d18640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3dc159f00 0x7fe3dc157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3e0d18640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0b0210 con 0x7fe3dc159b20 2023-10-16T15:50:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3e0d18640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0af600 con 0x7fe3dc159f00 2023-10-16T15:50:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3e0d18640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0af8e0 con 0x7fe3dc0b0b20 2023-10-16T15:50:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3d9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 0x7fe3dc0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3d9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 0x7fe3dc0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38830/0 (socket says 172.21.15.29:38830) 2023-10-16T15:50:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.878+0000 7fe3d9d9b640 1 -- 172.21.15.29:0/3594650698 learned_addr learned my addr 172.21.15.29:0/3594650698 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3dad9d640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3dc159f00 0x7fe3dc157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3d9d9b640 1 -- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3dc0b0b20 msgr2=0x7fe3dc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3d9d9b640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3dc0b0b20 0x7fe3dc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3d9d9b640 1 -- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3dc159f00 msgr2=0x7fe3dc157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3d9d9b640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3dc159f00 0x7fe3dc157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3d9d9b640 1 -- 172.21.15.29:0/3594650698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d400bd40 con 0x7fe3dc159b20 2023-10-16T15:50:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3dad9d640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3dc159f00 0x7fe3dc157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3d9d9b640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 0x7fe3dc0acdd0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000c400 tx=0x7fe3d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d00035f0 con 0x7fe3dc159b20 2023-10-16T15:50:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d0003750 con 0x7fe3dc159b20 2023-10-16T15:50:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc0afbc0 con 0x7fe3dc159b20 2023-10-16T15:50:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0003b40 con 0x7fe3dc159b20 2023-10-16T15:50:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.879+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc15a310 con 0x7fe3dc159b20 2023-10-16T15:50:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.881+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc0b03c0 con 0x7fe3dc159b20 2023-10-16T15:50:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.882+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe3d0003ca0 con 0x7fe3dc159b20 2023-10-16T15:50:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.883+0000 7fe3caffd640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fe3bc06af90 0x7fe3bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.883+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fe3d0088c70 con 0x7fe3dc159b20 2023-10-16T15:50:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.883+0000 7fe3da59c640 1 -- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fe3bc06af90 msgr2=0x7fe3bc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:21.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.884+0000 7fe3da59c640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fe3bc06af90 0x7fe3bc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:21.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:21.887+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3dc0b03c0 con 0x7fe3dc159b20 2023-10-16T15:50:21.902 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:21.900+0000 7f06c261f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:50:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.077+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3dc15a4a0 con 0x7fe3dc159b20 2023-10-16T15:50:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.081+0000 7fe3caffd640 1 -- 172.21.15.29:0/3594650698 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+17797 (secure 0 0 0) 0x7fe3d0017090 con 0x7fe3dc159b20 2023-10-16T15:50:22.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:22.083 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":195,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:21.163792+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":32,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:6832/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3590601432":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1525143382":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/2183916363":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3770772716":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3372934420":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3161548522":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1363341747":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1893102802":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1719808847":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/345956683":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6833/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6832/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/4127022397":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/479239966":"2023-10-17T15:34:40.941061+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fe3bc06af90 msgr2=0x7fe3bc06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fe3bc06af90 0x7fe3bc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 msgr2=0x7fe3dc0acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 0x7fe3dc0acdd0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000c400 tx=0x7fe3d000c8c0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 shutdown_connections 2023-10-16T15:50:22.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fe3bc06af90 0x7fe3bc06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3dc159b20 0x7fe3dc0acdd0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3dc159f00 0x7fe3dc157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 --2- 172.21.15.29:0/3594650698 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3dc0b0b20 0x7fe3dc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 >> 172.21.15.29:0/3594650698 conn(0x7fe3dc0cb410 msgr2=0x7fe3dc0ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:22.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 shutdown_connections 2023-10-16T15:50:22.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.083+0000 7fe3e0d18640 1 -- 172.21.15.29:0/3594650698 wait complete. 2023-10-16T15:50:22.104 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:34:23.977164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:34:25.069119+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 195, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:50:22.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:50:22.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.302+0000 7f9c0759e640 1 Processor -- start 2023-10-16T15:50:22.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.302+0000 7f9c0759e640 1 -- start start 2023-10-16T15:50:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c080cd9b0 0x7f9c080b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9c080afcf0 0x7f9c080ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0815b0b0 con 0x7f9c0815acd0 2023-10-16T15:50:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c080b6730 con 0x7f9c080afcf0 2023-10-16T15:50:22.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080cc6f0 con 0x7f9c080cd9b0 2023-10-16T15:50:22.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0659c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f9c0815acd0 0x7f9c080cd2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:57464/0 (socket says 172.21.15.29:57464) 2023-10-16T15:50:22.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c0659c640 1 -- 172.21.15.29:0/925113924 learned_addr learned my addr 172.21.15.29:0/925113924 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:22.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.304+0000 7f9c06d9d640 1 --2- 172.21.15.29:0/925113924 >> v2:172.21.15.170:3300/0 conn(0x7f9c080afcf0 0x7f9c080ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9bfffff640 1 --2- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c080cd9b0 0x7f9c080b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9c05d9b640 1 -- 172.21.15.29:0/925113924 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70416820 0 0) 0x7f9c0815b0b0 con 0x7f9c0815acd0 2023-10-16T15:50:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9c05d9b640 1 -- 172.21.15.29:0/925113924 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4003520 con 0x7f9c0815acd0 2023-10-16T15:50:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9c06d9d640 1 -- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c080cd9b0 msgr2=0x7f9c080b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9c06d9d640 1 --2- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c080cd9b0 0x7f9c080b60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9c06d9d640 1 -- 172.21.15.29:0/925113924 >> v1:172.21.15.170:6789/0 conn(0x7f9c0815acd0 legacy=0x7f9c080cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.305+0000 7f9c06d9d640 1 -- 172.21.15.29:0/925113924 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c080cc9d0 con 0x7f9c080afcf0 2023-10-16T15:50:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.307+0000 7f9c06d9d640 1 --2- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080afcf0 0x7f9c080ce490 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009d90 tx=0x7f9c0002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba0263d47dc964ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c05d9b640 1 -- 172.21.15.29:0/925113924 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0002cc50 con 0x7f9c080afcf0 2023-10-16T15:50:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c05d9b640 1 -- 172.21.15.29:0/925113924 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0002cdb0 con 0x7f9c080afcf0 2023-10-16T15:50:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c05d9b640 1 -- 172.21.15.29:0/925113924 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00034660 con 0x7f9c080afcf0 2023-10-16T15:50:22.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c0759e640 1 -- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080afcf0 msgr2=0x7f9c080ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c0759e640 1 --2- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080afcf0 0x7f9c080ce490 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009d90 tx=0x7f9c0002bea0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c0759e640 1 -- 172.21.15.29:0/925113924 shutdown_connections 2023-10-16T15:50:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c0759e640 1 --2- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080afcf0 0x7f9c080ce490 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c0759e640 1 --2- 172.21.15.29:0/925113924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c080cd9b0 0x7f9c080b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.308+0000 7f9c0759e640 1 -- 172.21.15.29:0/925113924 >> 172.21.15.29:0/925113924 conn(0x7f9c080b23b0 msgr2=0x7f9c080b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:22.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.309+0000 7f9c0759e640 1 -- 172.21.15.29:0/925113924 shutdown_connections 2023-10-16T15:50:22.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.309+0000 7f9c0759e640 1 -- 172.21.15.29:0/925113924 wait complete. 2023-10-16T15:50:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.310+0000 7f9c0759e640 1 Processor -- start 2023-10-16T15:50:22.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.311+0000 7f9c0759e640 1 -- start start 2023-10-16T15:50:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 0x7f9c0815d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080cd9b0 0x7f9c080ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c0815acd0 0x7f9c08158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0815c050 con 0x7f9c080afcf0 2023-10-16T15:50:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080ae9a0 con 0x7f9c080cd9b0 2023-10-16T15:50:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0815c280 con 0x7f9c0815acd0 2023-10-16T15:50:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 0x7f9c0815d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 0x7f9c0815d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38836/0 (socket says 172.21.15.29:38836) 2023-10-16T15:50:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c0659c640 1 -- 172.21.15.29:0/1740929746 learned_addr learned my addr 172.21.15.29:0/1740929746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9bfffff640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080cd9b0 0x7f9c080ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.312+0000 7f9c06d9d640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c0815acd0 0x7f9c08158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0659c640 1 -- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c0815acd0 msgr2=0x7f9c08158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0659c640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c0815acd0 0x7f9c08158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0659c640 1 -- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080cd9b0 msgr2=0x7f9c080ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0659c640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080cd9b0 0x7f9c080ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0659c640 1 -- 172.21.15.29:0/1740929746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c00009a70 con 0x7f9c080afcf0 2023-10-16T15:50:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0659c640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 0x7f9c0815d040 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800c500 tx=0x7f9bf800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8005ce0 con 0x7f9c080afcf0 2023-10-16T15:50:22.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf8005e40 con 0x7f9c080afcf0 2023-10-16T15:50:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8007330 con 0x7f9c080afcf0 2023-10-16T15:50:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0815c560 con 0x7f9c080afcf0 2023-10-16T15:50:22.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.313+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0815cb20 con 0x7f9c080afcf0 2023-10-16T15:50:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.314+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0804e9b0 con 0x7f9c080afcf0 2023-10-16T15:50:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.316+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f9bf8007490 con 0x7f9c080afcf0 2023-10-16T15:50:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.317+0000 7f9bfeffd640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 0x7f9be806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.317+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f9bf80890a0 con 0x7f9c080afcf0 2023-10-16T15:50:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.317+0000 7f9bfffff640 1 -- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 msgr2=0x7f9be806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.317+0000 7f9bfffff640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 0x7f9be806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:22.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.321+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9bf8053e80 con 0x7f9c080afcf0 2023-10-16T15:50:22.324 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.322+0000 7fde53c1a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:50:22.381 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.379+0000 7fde53c1a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:50:22.389 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.386+0000 7f06c261f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:50:22.431 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.429+0000 7fde53c1a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:50:22.443 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.441+0000 7f06c261f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:50:22.492 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.490+0000 7fde53c1a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:50:22.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.493+0000 7f06c261f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:50:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.518+0000 7f9bfffff640 1 -- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 msgr2=0x7f9be806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.518+0000 7f9bfffff640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 0x7f9be806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.518+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c080cf5b0 con 0x7f9c080afcf0 2023-10-16T15:50:22.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.519+0000 7f9bfeffd640 1 -- 172.21.15.29:0/1740929746 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bf80170a0 con 0x7f9c080afcf0 2023-10-16T15:50:22.521 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:50:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 msgr2=0x7f9be806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 0x7f9be806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 msgr2=0x7f9c0815d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 0x7f9c0815d040 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800c500 tx=0x7f9bf800c9c0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 shutdown_connections 2023-10-16T15:50:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f9be806afc0 0x7f9be806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c0815acd0 0x7f9c08158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c080cd9b0 0x7f9c080ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 --2- 172.21.15.29:0/1740929746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c080afcf0 0x7f9c0815d040 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 >> 172.21.15.29:0/1740929746 conn(0x7f9c080b23b0 msgr2=0x7f9c08159600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 shutdown_connections 2023-10-16T15:50:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.522+0000 7f9c0759e640 1 -- 172.21.15.29:0/1740929746 wait complete. 2023-10-16T15:50:22.541 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.539+0000 7fde53c1a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:50:22.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:50:22.569 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.567+0000 7f06c261f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:50:22.624 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.623+0000 7f06c261f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:50:22.658 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.657+0000 7fde53c1a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:50:22.722 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.720+0000 7fde53c1a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:22.743 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.741+0000 7f06c261f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:50:22.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.742+0000 7f13acbf4640 1 Processor -- start 2023-10-16T15:50:22.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.743+0000 7f13acbf4640 1 -- start start 2023-10-16T15:50:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.744+0000 7f13acbf4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.744+0000 7f13acbf4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f13a80d0be0 0x7f13a80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.744+0000 7f13acbf4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a815d150 con 0x7f13a815cd70 2023-10-16T15:50:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.744+0000 7f13acbf4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a80d1180 con 0x7f13a80d0be0 2023-10-16T15:50:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.744+0000 7f13acbf4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80d1460 con 0x7f13a80b8480 2023-10-16T15:50:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.744+0000 7f139effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f13a7fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f13a80d0be0 0x7f13a80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f139effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:40880/0 (socket says 172.21.15.29:40880) 2023-10-16T15:50:22.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f139effd640 1 -- 172.21.15.29:0/1082079255 learned_addr learned my addr 172.21.15.29:0/1082079255 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:22.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f13a7fff640 1 -- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 msgr2=0x7f13a80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:22.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f13a7fff640 1 --2- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f13a7fff640 1 -- 172.21.15.29:0/1082079255 >> v1:172.21.15.170:6789/0 conn(0x7f13a815cd70 legacy=0x7f13a80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.745+0000 7f13a7fff640 1 -- 172.21.15.29:0/1082079255 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80d1740 con 0x7f13a80d0be0 2023-10-16T15:50:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.747+0000 7f13a7fff640 1 --2- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a80d0be0 0x7f13a80cdd00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009d90 tx=0x7f13a002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2700a7221bb76122 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.747+0000 7f13a6ffd640 1 -- 172.21.15.29:0/1082079255 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a002cc50 con 0x7f13a80d0be0 2023-10-16T15:50:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.747+0000 7f13a6ffd640 1 -- 172.21.15.29:0/1082079255 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a002cdb0 con 0x7f13a80d0be0 2023-10-16T15:50:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.747+0000 7f13a6ffd640 1 -- 172.21.15.29:0/1082079255 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0034660 con 0x7f13a80d0be0 2023-10-16T15:50:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 -- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a80d0be0 msgr2=0x7f13a80cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a80d0be0 0x7f13a80cdd00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009d90 tx=0x7f13a002bea0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 -- 172.21.15.29:0/1082079255 shutdown_connections 2023-10-16T15:50:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a80d0be0 0x7f13a80cdd00 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1082079255 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 -- 172.21.15.29:0/1082079255 >> 172.21.15.29:0/1082079255 conn(0x7f13a80cf570 msgr2=0x7f13a80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 -- 172.21.15.29:0/1082079255 shutdown_connections 2023-10-16T15:50:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.748+0000 7f13acbf4640 1 -- 172.21.15.29:0/1082079255 wait complete. 2023-10-16T15:50:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.749+0000 7f13acbf4640 1 Processor -- start 2023-10-16T15:50:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.750+0000 7f13acbf4640 1 -- start start 2023-10-16T15:50:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.750+0000 7f13acbf4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.750+0000 7f13acbf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 0x7f13a815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f13acbf4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a815cd70 0x7f13a815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f13acbf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8151f10 con 0x7f13a80d0be0 2023-10-16T15:50:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f13acbf4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80cafb0 con 0x7f13a815cd70 2023-10-16T15:50:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f13acbf4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80ca100 con 0x7f13a80b8480 2023-10-16T15:50:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 0x7f13a815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 0x7f13a815d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38844/0 (socket says 172.21.15.29:38844) 2023-10-16T15:50:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 -- 172.21.15.29:0/1772960932 learned_addr learned my addr 172.21.15.29:0/1772960932 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f13a7fff640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a815cd70 0x7f13a815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f13a77fe640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 -- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 msgr2=0x7f13a80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 -- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a815cd70 msgr2=0x7f13a815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a815cd70 0x7f13a815d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.751+0000 7f139effd640 1 -- 172.21.15.29:0/1772960932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0009a70 con 0x7f13a80d0be0 2023-10-16T15:50:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.752+0000 7f139effd640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 0x7f13a815d360 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f139000e840 tx=0x7f139000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.752+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139000c8e0 con 0x7f13a80d0be0 2023-10-16T15:50:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.752+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139000ca40 con 0x7f13a80d0be0 2023-10-16T15:50:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.752+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a80ca3a0 con 0x7f13a80d0be0 2023-10-16T15:50:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.752+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139000cdf0 con 0x7f13a80d0be0 2023-10-16T15:50:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.752+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a80ca960 con 0x7f13a80d0be0 2023-10-16T15:50:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.753+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a804e9b0 con 0x7f13a80d0be0 2023-10-16T15:50:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.755+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1390013470 con 0x7f13a80d0be0 2023-10-16T15:50:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.755+0000 7f13a4ff9640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f138c06afc0 0x7f138c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.756+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f1390089b80 con 0x7f13a80d0be0 2023-10-16T15:50:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.756+0000 7f13a77fe640 1 -- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f138c06afc0 msgr2=0x7f138c06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.756+0000 7f13a77fe640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f138c06afc0 0x7f138c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.760+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1390054760 con 0x7f13a80d0be0 2023-10-16T15:50:22.773 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.771+0000 7fde53c1a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:50:22.812 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.810+0000 7f06c261f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:22.829 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.827+0000 7fde53c1a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:50:22.864 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.862+0000 7f06c261f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:50:22.880 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:22.878+0000 7fde53c1a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:50:22.923 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.921+0000 7f06c261f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:50:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.951+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f13a815c2c0 con 0x7f13a80d0be0 2023-10-16T15:50:22.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.952+0000 7f13a4ff9640 1 -- 172.21.15.29:0/1772960932 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v196) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1390017020 con 0x7f13a80d0be0 2023-10-16T15:50:22.954 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:50:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f138c06afc0 msgr2=0x7f138c06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f138c06afc0 0x7f138c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 msgr2=0x7f13a815d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 0x7f13a815d360 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f139000e840 tx=0x7f139000ed00 comp rx=0 tx=0).stop 2023-10-16T15:50:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 shutdown_connections 2023-10-16T15:50:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f138c06afc0 0x7f138c06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13a80d0be0 0x7f13a815d360 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f13a815cd70 0x7f13a815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 --2- 172.21.15.29:0/1772960932 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f13a80b8480 0x7f13a80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 >> 172.21.15.29:0/1772960932 conn(0x7f13a80cf570 msgr2=0x7f13a80d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 shutdown_connections 2023-10-16T15:50:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:22.955+0000 7f13acbf4640 1 -- 172.21.15.29:0/1772960932 wait complete. 2023-10-16T15:50:22.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:50:22.978 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:22.975+0000 7f06c261f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:50:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.193+0000 7f58e1597640 1 Processor -- start 2023-10-16T15:50:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.194+0000 7f58e1597640 1 -- start start 2023-10-16T15:50:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58e1597640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58dc160ad0 0x7f58dc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58e1597640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ce350 0x7f58dc160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58e1597640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0ce800 con 0x7f58dc0ae260 2023-10-16T15:50:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58e1597640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc0cdec0 con 0x7f58dc160ad0 2023-10-16T15:50:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58e1597640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0acdd0 con 0x7f58dc0ce350 2023-10-16T15:50:23.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58daffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58dc160ad0 0x7f58dc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.195+0000 7f58daffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58dc160ad0 0x7f58dc0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:45162/0 (socket says 172.21.15.29:45162) 2023-10-16T15:50:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.196+0000 7f58daffd640 1 -- 172.21.15.29:0/3051034364 learned_addr learned my addr 172.21.15.29:0/3051034364 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.196+0000 7f58da7fc640 1 --2- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ce350 0x7f58dc160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.196+0000 7f58daffd640 1 -- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ce350 msgr2=0x7f58dc160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.196+0000 7f58daffd640 1 --2- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ce350 0x7f58dc160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.196+0000 7f58daffd640 1 -- 172.21.15.29:0/3051034364 >> v1:172.21.15.170:6789/0 conn(0x7f58dc0ae260 legacy=0x7f58dc0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.196+0000 7f58daffd640 1 -- 172.21.15.29:0/3051034364 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc1613f0 con 0x7f58dc160ad0 2023-10-16T15:50:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.197+0000 7f58daffd640 1 --2- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc160ad0 0x7f58dc0ac890 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009870 tx=0x7f58cc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e393f74880535008 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.198+0000 7f58d9ffb640 1 -- 172.21.15.29:0/3051034364 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc030840 con 0x7f58dc160ad0 2023-10-16T15:50:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.198+0000 7f58d9ffb640 1 -- 172.21.15.29:0/3051034364 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc0309a0 con 0x7f58dc160ad0 2023-10-16T15:50:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.198+0000 7f58d9ffb640 1 -- 172.21.15.29:0/3051034364 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc030d30 con 0x7f58dc160ad0 2023-10-16T15:50:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.198+0000 7f58e1597640 1 -- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc160ad0 msgr2=0x7f58dc0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.198+0000 7f58e1597640 1 --2- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc160ad0 0x7f58dc0ac890 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009870 tx=0x7f58cc02b980 comp rx=0 tx=0).stop 2023-10-16T15:50:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.199+0000 7f58e1597640 1 -- 172.21.15.29:0/3051034364 shutdown_connections 2023-10-16T15:50:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.199+0000 7f58e1597640 1 --2- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ce350 0x7f58dc160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.199+0000 7f58e1597640 1 --2- 172.21.15.29:0/3051034364 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc160ad0 0x7f58dc0ac890 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.199+0000 7f58e1597640 1 -- 172.21.15.29:0/3051034364 >> 172.21.15.29:0/3051034364 conn(0x7f58dc06bc10 msgr2=0x7f58dc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.199+0000 7f58e1597640 1 -- 172.21.15.29:0/3051034364 shutdown_connections 2023-10-16T15:50:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.199+0000 7f58e1597640 1 -- 172.21.15.29:0/3051034364 wait complete. 2023-10-16T15:50:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.200+0000 7f58e1597640 1 Processor -- start 2023-10-16T15:50:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.201+0000 7f58e1597640 1 -- start start 2023-10-16T15:50:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58e1597640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ae260 0x7f58dc159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58e1597640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc0ce350 0x7f58dc15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58e1597640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 0x7f58dc0c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58e1597640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0b2ec0 con 0x7f58dc160ad0 2023-10-16T15:50:23.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58e1597640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0b36a0 con 0x7f58dc0ce350 2023-10-16T15:50:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58e1597640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0b3980 con 0x7f58dc0ae260 2023-10-16T15:50:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 0x7f58dc0c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 0x7f58dc0c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38852/0 (socket says 172.21.15.29:38852) 2023-10-16T15:50:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58db7fe640 1 -- 172.21.15.29:0/1349702197 learned_addr learned my addr 172.21.15.29:0/1349702197 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58da7fc640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc0ce350 0x7f58dc15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.202+0000 7f58daffd640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ae260 0x7f58dc159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58db7fe640 1 -- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ae260 msgr2=0x7f58dc159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58db7fe640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ae260 0x7f58dc159af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58db7fe640 1 -- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc0ce350 msgr2=0x7f58dc15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58db7fe640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc0ce350 0x7f58dc15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58db7fe640 1 -- 172.21.15.29:0/1349702197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc0094d0 con 0x7f58dc160ad0 2023-10-16T15:50:23.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58db7fe640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 0x7f58dc0c3c70 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c730 tx=0x7f58d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:23.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4003b70 con 0x7f58dc160ad0 2023-10-16T15:50:23.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d4003cd0 con 0x7f58dc160ad0 2023-10-16T15:50:23.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc0b3c60 con 0x7f58dc160ad0 2023-10-16T15:50:23.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400aa90 con 0x7f58dc160ad0 2023-10-16T15:50:23.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.203+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc15a030 con 0x7f58dc160ad0 2023-10-16T15:50:23.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.204+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc0b4180 con 0x7f58dc160ad0 2023-10-16T15:50:23.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.206+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f58d400abf0 con 0x7f58dc160ad0 2023-10-16T15:50:23.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.207+0000 7f58c37fe640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.207+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f58d408a1b0 con 0x7f58dc160ad0 2023-10-16T15:50:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.207+0000 7f58daffd640 1 -- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 msgr2=0x7f58c806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.207+0000 7f58daffd640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.214+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58d4054e10 con 0x7f58dc160ad0 2023-10-16T15:50:23.263 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.261+0000 7fde53c1a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:50:23.385 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:23.383+0000 7f06c261f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:50:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.407+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58dc0b4180 con 0x7f58dc160ad0 2023-10-16T15:50:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.408+0000 7f58daffd640 1 -- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 msgr2=0x7f58c806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.408+0000 7f58daffd640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.408+0000 7f58c37fe640 1 -- 172.21.15.29:0/1349702197 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v196) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58dc0b4180 con 0x7f58dc160ad0 2023-10-16T15:50:23.411 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:50:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 msgr2=0x7f58c806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 msgr2=0x7f58dc0c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 0x7f58dc0c3c70 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c730 tx=0x7f58d400cbf0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 shutdown_connections 2023-10-16T15:50:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58dc160ad0 0x7f58dc0c3c70 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58dc0ce350 0x7f58dc15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 --2- 172.21.15.29:0/1349702197 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58dc0ae260 0x7f58dc159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.411+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 >> 172.21.15.29:0/1349702197 conn(0x7f58dc06bc10 msgr2=0x7f58dc06fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.412+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 shutdown_connections 2023-10-16T15:50:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.412+0000 7f58e1597640 1 -- 172.21.15.29:0/1349702197 wait complete. 2023-10-16T15:50:23.422 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.420+0000 7fde53c1a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:50:23.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:50:23.530 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.528+0000 7fde53c1a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:50:23.558 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:23.556+0000 7f06c261f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:50:23.580 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.578+0000 7fde53c1a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:50:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.632+0000 7f22389b6640 1 Processor -- start 2023-10-16T15:50:23.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.632+0000 7f22389b6640 1 -- start start 2023-10-16T15:50:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.633+0000 7f22389b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2234159ab0 0x7f22340ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.633+0000 7f22389b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f223406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f22389b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340cd1f0 con 0x7f223406c140 2023-10-16T15:50:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f22389b6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223415d890 con 0x7f2234159ab0 2023-10-16T15:50:23.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f22389b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340cd4d0 con 0x7f2234159e90 2023-10-16T15:50:23.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f223259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2234159ab0 0x7f22340ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f223259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2234159ab0 0x7f22340ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38854/0 (socket says 172.21.15.29:38854) 2023-10-16T15:50:23.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f223259c640 1 -- 172.21.15.29:0/2134954119 learned_addr learned my addr 172.21.15.29:0/2134954119 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:23.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f2231d9b640 1 --2- 172.21.15.29:0/2134954119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f223406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f223259c640 1 -- 172.21.15.29:0/2134954119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 msgr2=0x7f223406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f223259c640 1 --2- 172.21.15.29:0/2134954119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f223406fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.634+0000 7f223259c640 1 -- 172.21.15.29:0/2134954119 >> v1:172.21.15.170:6789/0 conn(0x7f223406c140 legacy=0x7f223415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.635+0000 7f223259c640 1 -- 172.21.15.29:0/2134954119 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340cd7b0 con 0x7f2234159ab0 2023-10-16T15:50:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.636+0000 7f223259c640 1 --2- 172.21.15.29:0/2134954119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2234159ab0 0x7f22340ce370 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f22240098a0 tx=0x7f222402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca3f5efee3fa8e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.636+0000 7f223159a640 1 -- 172.21.15.29:0/2134954119 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240317c0 con 0x7f2234159ab0 2023-10-16T15:50:23.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.636+0000 7f223159a640 1 -- 172.21.15.29:0/2134954119 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2224031920 con 0x7f2234159ab0 2023-10-16T15:50:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.636+0000 7f223159a640 1 -- 172.21.15.29:0/2134954119 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224031cb0 con 0x7f2234159ab0 2023-10-16T15:50:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 -- 172.21.15.29:0/2134954119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2234159ab0 msgr2=0x7f22340ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 --2- 172.21.15.29:0/2134954119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2234159ab0 0x7f22340ce370 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f22240098a0 tx=0x7f222402b860 comp rx=0 tx=0).stop 2023-10-16T15:50:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 -- 172.21.15.29:0/2134954119 shutdown_connections 2023-10-16T15:50:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 --2- 172.21.15.29:0/2134954119 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f223406fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 --2- 172.21.15.29:0/2134954119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2234159ab0 0x7f22340ce370 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 -- 172.21.15.29:0/2134954119 >> 172.21.15.29:0/2134954119 conn(0x7f22340ce9b0 msgr2=0x7f22340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 -- 172.21.15.29:0/2134954119 shutdown_connections 2023-10-16T15:50:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.637+0000 7f22389b6640 1 -- 172.21.15.29:0/2134954119 wait complete. 2023-10-16T15:50:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.639+0000 7f22389b6640 1 Processor -- start 2023-10-16T15:50:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.639+0000 7f22389b6640 1 -- start start 2023-10-16T15:50:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f22389b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 0x7f223414ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f22389b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2234159ab0 0x7f22340c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f22389b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f22340c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f22389b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c9100 con 0x7f223406c140 2023-10-16T15:50:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f22389b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c8b40 con 0x7f2234159ab0 2023-10-16T15:50:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f22389b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c8e20 con 0x7f2234159e90 2023-10-16T15:50:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f223259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 0x7f223414ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f223259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 0x7f223414ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38866/0 (socket says 172.21.15.29:38866) 2023-10-16T15:50:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f223259c640 1 -- 172.21.15.29:0/1389314056 learned_addr learned my addr 172.21.15.29:0/1389314056 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.640+0000 7f2232d9d640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f22340c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f223259c640 1 -- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 msgr2=0x7f22340c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f2231d9b640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2234159ab0 0x7f22340c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f223259c640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f22340c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f223259c640 1 -- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2234159ab0 msgr2=0x7f22340c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f223259c640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2234159ab0 0x7f22340c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f223259c640 1 -- 172.21.15.29:0/1389314056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22240094d0 con 0x7f223406c140 2023-10-16T15:50:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f2232d9d640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f22340c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f2231d9b640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2234159ab0 0x7f22340c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f223259c640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 0x7f223414ff20 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f2224000c00 tx=0x7f22240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224032860 con 0x7f223406c140 2023-10-16T15:50:23.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22240329e0 con 0x7f223406c140 2023-10-16T15:50:23.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224032d90 con 0x7f223406c140 2023-10-16T15:50:23.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.641+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223415b220 con 0x7f223406c140 2023-10-16T15:50:23.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.642+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223415d0b0 con 0x7f223406c140 2023-10-16T15:50:23.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.643+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223415b080 con 0x7f223406c140 2023-10-16T15:50:23.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.644+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f222404d3f0 con 0x7f223406c140 2023-10-16T15:50:23.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.645+0000 7f2222ffd640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 0x7f221406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.645+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f22240ab990 con 0x7f223406c140 2023-10-16T15:50:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.645+0000 7f2231d9b640 1 -- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 msgr2=0x7f221406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.645+0000 7f2231d9b640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 0x7f221406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.649+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f223415b080 con 0x7f223406c140 2023-10-16T15:50:23.675 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:23.673+0000 7f06c261f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:50:23.688 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.686+0000 7fde53c1a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:23.727 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:23.725+0000 7f06c261f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:50:23.793 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.791+0000 7fde53c1a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:50:23.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:23.839+0000 7f06c261f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.846+0000 7f2231d9b640 1 -- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 msgr2=0x7f221406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:23.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.846+0000 7f2231d9b640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 0x7f221406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:23.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.854+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f223415b080 con 0x7f223406c140 2023-10-16T15:50:23.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.854+0000 7f2222ffd640 1 -- 172.21.15.29:0/1389314056 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v196) v1 ==== 81+0+954 (secure 0 0 0) 0x7f223415b080 con 0x7f223406c140 2023-10-16T15:50:23.857 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/3536897498 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.857 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/3536897498 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.857 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3590601432 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.857 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1525143382 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.858 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2183916363 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.858 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3770772716 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.858 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3372934420 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.858 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3161548522 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.858 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1363341747 2023-10-17T15:45:21.100294+0000 2023-10-16T15:50:23.858 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1893102802 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.859 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1719808847 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.859 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/345956683 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.859 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/3694096961 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.859 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/3694096961 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.859 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/4127022397 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.860 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/479239966 2023-10-17T15:34:40.941061+0000 2023-10-16T15:50:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:listed 16 entries 2023-10-16T15:50:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 msgr2=0x7f221406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 0x7f221406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 msgr2=0x7f223414ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 0x7f223414ff20 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f2224000c00 tx=0x7f22240046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 shutdown_connections 2023-10-16T15:50:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f221406af90 0x7f221406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2234159e90 0x7f22340c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2234159ab0 0x7f22340c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 --2- 172.21.15.29:0/1389314056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f223406c140 0x7f223414ff20 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.857+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 >> 172.21.15.29:0/1389314056 conn(0x7f22340ce9b0 msgr2=0x7f22340b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.858+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 shutdown_connections 2023-10-16T15:50:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:23.858+0000 7f22389b6640 1 -- 172.21.15.29:0/1389314056 wait complete. 2023-10-16T15:50:23.876 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:50:23.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:23.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:50:23.949 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.946+0000 7fde53c1a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:23.958 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:23.956+0000 7f06c261f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:50:23.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:50:23.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:23.975 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:50:24.001 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:23.997+0000 7fde53c1a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:24.024 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:50:24.025 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:24.082 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:50:24.083 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:50:24.083 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:24.122 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:24.120+0000 7fde53c1a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:50:24.125 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:24.122+0000 7f06c261f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:24.140 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:50:24.140 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:50:24.141 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:24.176 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:24.174+0000 7f06c261f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:24.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:50:24.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:24.303 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:24.301+0000 7f06c261f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:50:24.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.414+0000 7f11a5db0640 1 Processor -- start 2023-10-16T15:50:24.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.414+0000 7f11a5db0640 1 -- start start 2023-10-16T15:50:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f11a5db0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 0x7f11a00ca860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f11a5db0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a00aca30 con 0x7f11a00cc780 2023-10-16T15:50:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f11a5db0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a0161750 con 0x7f11a00cada0 2023-10-16T15:50:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f11a5db0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0160b40 con 0x7f11a01577c0 2023-10-16T15:50:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f119ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f11a00cc780 0x7f11a006c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39006/0 (socket says 172.21.15.29:39006) 2023-10-16T15:50:24.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f119ffff640 1 -- 172.21.15.29:0/386798098 learned_addr learned my addr 172.21.15.29:0/386798098 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f119f7fe640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 0x7f11a00ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.416+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512394738 0 0) 0x7f11a00aca30 con 0x7f11a00cc780 2023-10-16T15:50:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c003520 con 0x7f11a00cc780 2023-10-16T15:50:24.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438259840 0 0) 0x7f118c003520 con 0x7f11a00cc780 2023-10-16T15:50:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 msgr2=0x7f11a00ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 0x7f11a00ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 >> v1:172.21.15.170:6789/0 conn(0x7f11a00cada0 legacy=0x7f11a00cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0160e20 con 0x7f11a00cc780 2023-10-16T15:50:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f11980019f0 con 0x7f11a00cc780 2023-10-16T15:50:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 >> v1:172.21.15.29:6789/0 conn(0x7f11a00cc780 legacy=0x7f11a006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c0039a0 0x7f118c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c0047c0 0x7f118c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 0x7f118c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0160b40 con 0x7f118c0039a0 2023-10-16T15:50:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c00bab0 con 0x7f118c0047c0 2023-10-16T15:50:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c00bc70 con 0x7f11a01577c0 2023-10-16T15:50:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.417+0000 7f119f7fe640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c0039a0 0x7f118c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f119ffff640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 0x7f118c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f1197fff640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c0047c0 0x7f118c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f119f7fe640 1 -- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 msgr2=0x7f118c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f119f7fe640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11a01577c0 0x7f118c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f119f7fe640 1 -- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c0047c0 msgr2=0x7f118c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f119f7fe640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c0047c0 0x7f118c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f119f7fe640 1 -- 172.21.15.29:0/386798098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c003520 con 0x7f118c0039a0 2023-10-16T15:50:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.418+0000 7f1197fff640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f118c0047c0 0x7f118c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.420+0000 7f119f7fe640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c0039a0 0x7f118c003dc0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f119000ba30 tx=0x7f118c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff73a4d884cc89e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.420+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190032640 con 0x7f118c0039a0 2023-10-16T15:50:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.420+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11900327a0 con 0x7f118c0039a0 2023-10-16T15:50:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.420+0000 7f119effd640 1 -- 172.21.15.29:0/386798098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190032b30 con 0x7f118c0039a0 2023-10-16T15:50:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f11a5db0640 1 -- 172.21.15.29:0/386798098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c0039a0 msgr2=0x7f118c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f11a5db0640 1 --2- 172.21.15.29:0/386798098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c0039a0 0x7f118c003dc0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f119000ba30 tx=0x7f118c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f119ffff640 1 -- 172.21.15.29:0/386798098 reap_dead start 2023-10-16T15:50:24.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f11a5db0640 1 -- 172.21.15.29:0/386798098 shutdown_connections 2023-10-16T15:50:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f11a5db0640 1 -- 172.21.15.29:0/386798098 >> 172.21.15.29:0/386798098 conn(0x7f11a006aeb0 msgr2=0x7f11a006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f11a5db0640 1 -- 172.21.15.29:0/386798098 shutdown_connections 2023-10-16T15:50:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.421+0000 7f11a5db0640 1 -- 172.21.15.29:0/386798098 wait complete. 2023-10-16T15:50:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.423+0000 7f11a5db0640 1 Processor -- start 2023-10-16T15:50:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.424+0000 7f11a5db0640 1 -- start start 2023-10-16T15:50:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.425+0000 7f11a5db0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f118c0039a0 0x7f11a00cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.425+0000 7f11a5db0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 0x7f11a00c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.425+0000 7f11a5db0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a014c8a0 0x7f11a00c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.425+0000 7f11a5db0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a01e75b0 con 0x7f11a014ced0 2023-10-16T15:50:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.425+0000 7f11a5db0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a01e7830 con 0x7f11a014c8a0 2023-10-16T15:50:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.425+0000 7f11a5db0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00cdec0 con 0x7f118c0039a0 2023-10-16T15:50:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 0x7f11a00c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f119f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f118c0039a0 0x7f11a00cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 0x7f11a00c4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50370/0 (socket says 172.21.15.29:50370) 2023-10-16T15:50:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 -- 172.21.15.29:0/2045489761 learned_addr learned my addr 172.21.15.29:0/2045489761 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f119ffff640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a014c8a0 0x7f11a00c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 -- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f118c0039a0 msgr2=0x7f11a00cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f118c0039a0 0x7f11a00cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 -- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a014c8a0 msgr2=0x7f11a00c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a014c8a0 0x7f11a00c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.426+0000 7f1197fff640 1 -- 172.21.15.29:0/2045489761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a00ce1a0 con 0x7f11a014ced0 2023-10-16T15:50:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.427+0000 7f1197fff640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 0x7f11a00c4430 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f118800d7a0 tx=0x7f118800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.427+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188017070 con 0x7f11a014ced0 2023-10-16T15:50:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.427+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1188005070 con 0x7f11a014ced0 2023-10-16T15:50:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.427+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a00ce480 con 0x7f11a014ced0 2023-10-16T15:50:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.427+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188005420 con 0x7f11a014ced0 2023-10-16T15:50:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.427+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a00d04d0 con 0x7f11a014ced0 2023-10-16T15:50:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.428+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a004e160 con 0x7f11a014ced0 2023-10-16T15:50:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.429+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1188007030 con 0x7f11a014ced0 2023-10-16T15:50:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.431+0000 7f119cff9640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 0x7f118006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.431+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f1188089490 con 0x7f11a014ced0 2023-10-16T15:50:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.431+0000 7f119f7fe640 1 -- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 msgr2=0x7f118006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.431+0000 7f119f7fe640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 0x7f118006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.434+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1188054090 con 0x7f11a014ced0 2023-10-16T15:50:24.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.632+0000 7f119f7fe640 1 -- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 msgr2=0x7f118006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:24.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.632+0000 7f119f7fe640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 0x7f118006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:24.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.663+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a01e7970 con 0x7f11a014ced0 2023-10-16T15:50:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.664+0000 7f119cff9640 1 -- 172.21.15.29:0/2045489761 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f11880576b0 con 0x7f11a014ced0 2023-10-16T15:50:24.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:24.667 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:34:26.024272+0000","modified":"2023-10-16T15:34:29.168102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2122514336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122514336},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122514336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T15:50:24.667 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 92 2023-10-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 msgr2=0x7f118006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 0x7f118006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 msgr2=0x7f11a00c4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 0x7f11a00c4430 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f118800d7a0 tx=0x7f118800dc60 comp rx=0 tx=0).stop 2023-10-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 shutdown_connections 2023-10-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f118006afe0 0x7f118006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11a014ced0 0x7f11a00c4430 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11a014c8a0 0x7f11a00c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 --2- 172.21.15.29:0/2045489761 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f118c0039a0 0x7f11a00cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 >> 172.21.15.29:0/2045489761 conn(0x7f11a006aeb0 msgr2=0x7f11a00ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 shutdown_connections 2023-10-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.667+0000 7f11a5db0640 1 -- 172.21.15.29:0/2045489761 wait complete. 2023-10-16T15:50:24.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:24.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.886+0000 7fec87db6640 1 Processor -- start 2023-10-16T15:50:24.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.886+0000 7fec87db6640 1 -- start start 2023-10-16T15:50:24.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.887+0000 7fec87db6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec80071380 0x7fec800cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.887+0000 7fec87db6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800b4380 con 0x7fec800ad990 2023-10-16T15:50:24.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.887+0000 7fec87db6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800b4620 con 0x7fec8015adc0 2023-10-16T15:50:24.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.887+0000 7fec87db6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8015e9b0 con 0x7fec80071380 2023-10-16T15:50:24.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec86353640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec80071380 0x7fec800cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec85351640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fec800ad990 0x7fec800cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39008/0 (socket says 172.21.15.29:39008) 2023-10-16T15:50:24.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec85351640 1 -- 172.21.15.29:0/3210458463 learned_addr learned my addr 172.21.15.29:0/3210458463 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:24.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec84b50640 1 -- 172.21.15.29:0/3210458463 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824621278 0 0) 0x7fec800b4380 con 0x7fec800ad990 2023-10-16T15:50:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec86353640 1 -- 172.21.15.29:0/3210458463 >> v1:172.21.15.170:6789/0 conn(0x7fec8015adc0 legacy=0x7fec80070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec86353640 1 -- 172.21.15.29:0/3210458463 >> v1:172.21.15.29:6789/0 conn(0x7fec800ad990 legacy=0x7fec800cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.888+0000 7fec86353640 1 -- 172.21.15.29:0/3210458463 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8015ec90 con 0x7fec80071380 2023-10-16T15:50:24.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.889+0000 7fec86353640 1 --2- 172.21.15.29:0/3210458463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80071380 0x7fec800cf380 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fec78009d90 tx=0x7fec7802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ba17d11ffee854 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.889+0000 7fec84b50640 1 -- 172.21.15.29:0/3210458463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7802cd00 con 0x7fec80071380 2023-10-16T15:50:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec84b50640 1 -- 172.21.15.29:0/3210458463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7802ce60 con 0x7fec80071380 2023-10-16T15:50:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec84b50640 1 -- 172.21.15.29:0/3210458463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78034660 con 0x7fec80071380 2023-10-16T15:50:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 -- 172.21.15.29:0/3210458463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80071380 msgr2=0x7fec800cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 --2- 172.21.15.29:0/3210458463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80071380 0x7fec800cf380 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fec78009d90 tx=0x7fec7802bf50 comp rx=0 tx=0).stop 2023-10-16T15:50:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 -- 172.21.15.29:0/3210458463 shutdown_connections 2023-10-16T15:50:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 --2- 172.21.15.29:0/3210458463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80071380 0x7fec800cf380 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 -- 172.21.15.29:0/3210458463 >> 172.21.15.29:0/3210458463 conn(0x7fec800ce860 msgr2=0x7fec800cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 -- 172.21.15.29:0/3210458463 shutdown_connections 2023-10-16T15:50:24.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.890+0000 7fec87db6640 1 -- 172.21.15.29:0/3210458463 wait complete. 2023-10-16T15:50:24.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.892+0000 7fec87db6640 1 Processor -- start 2023-10-16T15:50:24.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.892+0000 7fec87db6640 1 -- start start 2023-10-16T15:50:24.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec87db6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec80071380 0x7fec801585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec87db6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec800ad990 0x7fec800ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec87db6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 0x7fec800acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec87db6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801f5d30 con 0x7fec8015adc0 2023-10-16T15:50:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec87db6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801f6b90 con 0x7fec80071380 2023-10-16T15:50:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec87db6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec801f5f80 con 0x7fec800ad990 2023-10-16T15:50:24.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec86353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 0x7fec800acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec86353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 0x7fec800acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50388/0 (socket says 172.21.15.29:50388) 2023-10-16T15:50:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.893+0000 7fec86353640 1 -- 172.21.15.29:0/1707795459 learned_addr learned my addr 172.21.15.29:0/1707795459 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec85b52640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec80071380 0x7fec801585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec86353640 1 -- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec800ad990 msgr2=0x7fec800ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec86353640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec800ad990 0x7fec800ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec86353640 1 -- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec80071380 msgr2=0x7fec801585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec86353640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec80071380 0x7fec801585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec86353640 1 -- 172.21.15.29:0/1707795459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec78009a70 con 0x7fec8015adc0 2023-10-16T15:50:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec85b52640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec80071380 0x7fec801585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec86353640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 0x7fec800acdd0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fec78037040 tx=0x7fec78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7803d600 con 0x7fec8015adc0 2023-10-16T15:50:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7803d780 con 0x7fec8015adc0 2023-10-16T15:50:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8015b320 con 0x7fec8015adc0 2023-10-16T15:50:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.894+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7803db30 con 0x7fec8015adc0 2023-10-16T15:50:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.895+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec801f6540 con 0x7fec8015adc0 2023-10-16T15:50:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.896+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8004e160 con 0x7fec8015adc0 2023-10-16T15:50:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.897+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fec7803b040 con 0x7fec8015adc0 2023-10-16T15:50:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.897+0000 7fec767fc640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fec6406af90 0x7fec6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.898+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fec780abb10 con 0x7fec8015adc0 2023-10-16T15:50:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.898+0000 7fec85b52640 1 -- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fec6406af90 msgr2=0x7fec6406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:24.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.898+0000 7fec85b52640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fec6406af90 0x7fec6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:24.902+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec780768a0 con 0x7fec8015adc0 2023-10-16T15:50:24.966 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:24.964+0000 7fde53c1a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:50:25.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.094+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec800c5450 con 0x7fec8015adc0 2023-10-16T15:50:25.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.096+0000 7fec767fc640 1 -- 172.21.15.29:0/1707795459 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+17771 (secure 0 0 0) 0x7fec78039090 con 0x7fec8015adc0 2023-10-16T15:50:25.099 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:25.099 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":196,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:22.177240+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:6832/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3590601432":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1525143382":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/2183916363":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3770772716":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3372934420":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3161548522":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1363341747":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1893102802":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1719808847":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/345956683":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6833/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6832/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/4127022397":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/479239966":"2023-10-17T15:34:40.941061+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:25.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.098+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fec6406af90 msgr2=0x7fec6406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:25.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.098+0000 7fec87db6640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fec6406af90 0x7fec6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 msgr2=0x7fec800acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 0x7fec800acdd0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fec78037040 tx=0x7fec78004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:25.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 shutdown_connections 2023-10-16T15:50:25.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7fec6406af90 0x7fec6406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015adc0 0x7fec800acdd0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fec800ad990 0x7fec800ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 --2- 172.21.15.29:0/1707795459 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fec80071380 0x7fec801585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 >> 172.21.15.29:0/1707795459 conn(0x7fec800ce860 msgr2=0x7fec801560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 shutdown_connections 2023-10-16T15:50:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.099+0000 7fec87db6640 1 -- 172.21.15.29:0/1707795459 wait complete. 2023-10-16T15:50:25.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:34:23.977164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:34:25.069119+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 196, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:50:25.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:50:25.144 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:25.142+0000 7f06c261f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:50:25.266 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.264+0000 7fde53c1a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:50:25.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.319+0000 7f0907fff640 1 Processor -- start 2023-10-16T15:50:25.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.319+0000 7f0907fff640 1 -- start start 2023-10-16T15:50:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.320+0000 7f0907fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09080b1ca0 0x7f09080d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.320+0000 7f0907fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080d0970 con 0x7f09080b2080 2023-10-16T15:50:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.320+0000 7f0907fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080d0ea0 con 0x7f09080cfc40 2023-10-16T15:50:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.320+0000 7f0907fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09080d0bc0 con 0x7f09080b1ca0 2023-10-16T15:50:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0906ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09080b1ca0 0x7f09080d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0906ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09080b1ca0 0x7f09080d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50400/0 (socket says 172.21.15.29:50400) 2023-10-16T15:50:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f09067fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f09080b2080 0x7f09080ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39022/0 (socket says 172.21.15.29:39022) 2023-10-16T15:50:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0906ffd640 1 -- 172.21.15.29:0/954334030 learned_addr learned my addr 172.21.15.29:0/954334030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:25.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0906ffd640 1 -- 172.21.15.29:0/954334030 >> v1:172.21.15.170:6789/0 conn(0x7f09080cfc40 legacy=0x7f09080b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0906ffd640 1 -- 172.21.15.29:0/954334030 >> v1:172.21.15.29:6789/0 conn(0x7f09080b2080 legacy=0x7f09080ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.325 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.321+0000 7fde53c1a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:50:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0906ffd640 1 -- 172.21.15.29:0/954334030 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080cd940 con 0x7f09080b1ca0 2023-10-16T15:50:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.321+0000 7f0905ffb640 1 -- 172.21.15.29:0/954334030 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090094757 0 0) 0x7f09080d0970 con 0x7f09080b2080 2023-10-16T15:50:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.322+0000 7f0906ffd640 1 --2- 172.21.15.29:0/954334030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080b1ca0 0x7f09080d36b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009870 tx=0x7f08f802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bc19f08ba7e2aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.323+0000 7f0905ffb640 1 -- 172.21.15.29:0/954334030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f802f850 con 0x7f09080b1ca0 2023-10-16T15:50:25.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.323+0000 7f0905ffb640 1 -- 172.21.15.29:0/954334030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f802f9b0 con 0x7f09080b1ca0 2023-10-16T15:50:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.323+0000 7f0905ffb640 1 -- 172.21.15.29:0/954334030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f802fd40 con 0x7f09080b1ca0 2023-10-16T15:50:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.323+0000 7f0907fff640 1 -- 172.21.15.29:0/954334030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080b1ca0 msgr2=0x7f09080d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.323+0000 7f0907fff640 1 --2- 172.21.15.29:0/954334030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080b1ca0 0x7f09080d36b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009870 tx=0x7f08f802ba30 comp rx=0 tx=0).stop 2023-10-16T15:50:25.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.323+0000 7f0907fff640 1 -- 172.21.15.29:0/954334030 shutdown_connections 2023-10-16T15:50:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.324+0000 7f0907fff640 1 --2- 172.21.15.29:0/954334030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080b1ca0 0x7f09080d36b0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.324+0000 7f0907fff640 1 -- 172.21.15.29:0/954334030 >> 172.21.15.29:0/954334030 conn(0x7f09080d2b90 msgr2=0x7f09080d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.324+0000 7f0907fff640 1 -- 172.21.15.29:0/954334030 shutdown_connections 2023-10-16T15:50:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.324+0000 7f0907fff640 1 -- 172.21.15.29:0/954334030 wait complete. 2023-10-16T15:50:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.325+0000 7f0907fff640 1 Processor -- start 2023-10-16T15:50:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.325+0000 7f0907fff640 1 -- start start 2023-10-16T15:50:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.326+0000 7f0907fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09080b1ca0 0x7f090814ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.326+0000 7f0907fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09080b2080 0x7f09080cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.326+0000 7f0907fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 0x7f0908150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.326+0000 7f0907fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090814f1b0 con 0x7f09080cfc40 2023-10-16T15:50:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.326+0000 7f0907fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090814ffb0 con 0x7f09080b2080 2023-10-16T15:50:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.326+0000 7f0907fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090814f3a0 con 0x7f09080b1ca0 2023-10-16T15:50:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 0x7f0908150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 0x7f0908150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50404/0 (socket says 172.21.15.29:50404) 2023-10-16T15:50:25.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 -- 172.21.15.29:0/3144408235 learned_addr learned my addr 172.21.15.29:0/3144408235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09067fc640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09080b2080 0x7f09080cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f0906ffd640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09080b1ca0 0x7f090814ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 -- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09080b1ca0 msgr2=0x7f090814ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09080b1ca0 0x7f090814ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 -- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09080b2080 msgr2=0x7f09080cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09080b2080 0x7f09080cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.327+0000 7f09077fe640 1 -- 172.21.15.29:0/3144408235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80094d0 con 0x7f09080cfc40 2023-10-16T15:50:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.328+0000 7f09077fe640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 0x7f0908150440 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f090000b9a0 tx=0x7f090000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.328+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090000cba0 con 0x7f09080cfc40 2023-10-16T15:50:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.328+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090000cd00 con 0x7f09080cfc40 2023-10-16T15:50:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.328+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900014680 con 0x7f09080cfc40 2023-10-16T15:50:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.328+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090814f680 con 0x7f09080cfc40 2023-10-16T15:50:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.328+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090814fc40 con 0x7f09080cfc40 2023-10-16T15:50:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.330+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0908150160 con 0x7f09080cfc40 2023-10-16T15:50:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.330+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 112) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f090001d3b0 con 0x7f09080cfc40 2023-10-16T15:50:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.331+0000 7f08ef7fe640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f08f406afc0 0x7f08f406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.331+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f0900089960 con 0x7f09080cfc40 2023-10-16T15:50:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.331+0000 7f0906ffd640 1 -- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f08f406afc0 msgr2=0x7f08f406d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/23414019 2023-10-16T15:50:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.331+0000 7f0906ffd640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f08f406afc0 0x7f08f406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:25.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.335+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0908150160 con 0x7f09080cfc40 2023-10-16T15:50:25.438 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.436+0000 7fde53c1a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:50:25.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.455+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 113) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0900054910 con 0x7f09080cfc40 2023-10-16T15:50:25.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.455+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f08f406afc0 msgr2=0x7f08f406d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:25.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.455+0000 7f08ef7fe640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f08f406afc0 0x7f08f406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.490 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:25.488+0000 7f06c261f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:50:25.526 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd8a8d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd8a8d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd8a8d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd8a8d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd8a8d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd5a87640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.527 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd5a87640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.528 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd5a87640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.528 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd5a87640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.528 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:25.525+0000 7fddd5a87640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:25.554 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:25.552+0000 7f06c261f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:50:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.554+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f090804e9b0 con 0x7f09080cfc40 2023-10-16T15:50:25.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.555+0000 7f08ef7fe640 1 -- 172.21.15.29:0/3144408235 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09000736c0 con 0x7f09080cfc40 2023-10-16T15:50:25.557 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:50:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 msgr2=0x7f0908150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 0x7f0908150440 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f090000b9a0 tx=0x7f090000be60 comp rx=0 tx=0).stop 2023-10-16T15:50:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 shutdown_connections 2023-10-16T15:50:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:6832/23414019,v1:172.21.15.170:6833/23414019] conn(0x7f08f406afc0 0x7f08f406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09080cfc40 0x7f0908150440 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f09080b2080 0x7f09080cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 --2- 172.21.15.29:0/3144408235 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f09080b1ca0 0x7f090814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.557+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 >> 172.21.15.29:0/3144408235 conn(0x7f09080d2b90 msgr2=0x7f09080b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.558+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 shutdown_connections 2023-10-16T15:50:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.558+0000 7f0907fff640 1 -- 172.21.15.29:0/3144408235 wait complete. 2023-10-16T15:50:25.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:50:25.684 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:25.682+0000 7f06c261f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:50:25.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.836+0000 7fc96e5f4640 1 Processor -- start 2023-10-16T15:50:25.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.836+0000 7fc96e5f4640 1 -- start start 2023-10-16T15:50:25.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.838+0000 7fc96e5f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 0x7fc968070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.838+0000 7fc96e5f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96806d900 con 0x7fc96806d520 2023-10-16T15:50:25.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.838+0000 7fc96e5f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680b24f0 con 0x7fc96806fa20 2023-10-16T15:50:25.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.838+0000 7fc96e5f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96815b1d0 con 0x7fc9680ace60 2023-10-16T15:50:25.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.838+0000 7fc96d5f2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc96806d520 0x7fc9680b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39034/0 (socket says 172.21.15.29:39034) 2023-10-16T15:50:25.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.838+0000 7fc96d5f2640 1 -- 172.21.15.29:0/1511206145 learned_addr learned my addr 172.21.15.29:0/1511206145 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677892339 0 0) 0x7fc96806d900 con 0x7fc96806d520 2023-10-16T15:50:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc96ddf3640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 0x7fc968070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954003520 con 0x7fc96806d520 2023-10-16T15:50:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1253733516 0 0) 0x7fc954003520 con 0x7fc96806d520 2023-10-16T15:50:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 msgr2=0x7fc968070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 0x7fc968070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 >> v1:172.21.15.170:6789/0 conn(0x7fc96806fa20 legacy=0x7fc9680ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96806feb0 con 0x7fc96806d520 2023-10-16T15:50:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc964001670 con 0x7fc96806d520 2023-10-16T15:50:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 >> v1:172.21.15.29:6789/0 conn(0x7fc96806d520 legacy=0x7fc9680b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9540047c0 0x7fc954004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 0x7fc95400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96815b1d0 con 0x7fc9540047c0 2023-10-16T15:50:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540055f0 con 0x7fc9540039a0 2023-10-16T15:50:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.839+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95400bec0 con 0x7fc9680ace60 2023-10-16T15:50:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9540047c0 0x7fc954004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96ddf3640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 0x7fc95400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96d5f2640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96cdf1640 1 -- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 msgr2=0x7fc95400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:25.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9680ace60 0x7fc95400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96cdf1640 1 -- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9540039a0 msgr2=0x7fc954003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9540039a0 0x7fc954003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.840+0000 7fc96cdf1640 1 -- 172.21.15.29:0/1511206145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954003520 con 0x7fc9540047c0 2023-10-16T15:50:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.841+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9540047c0 0x7fc954004c40 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fc958001750 tx=0x7fc9540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e507fa3bc95e84c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.841+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580052d0 con 0x7fc9540047c0 2023-10-16T15:50:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.841+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95802e050 con 0x7fc9540047c0 2023-10-16T15:50:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.841+0000 7fc95ffff640 1 -- 172.21.15.29:0/1511206145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580035f0 con 0x7fc9540047c0 2023-10-16T15:50:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96e5f4640 1 -- 172.21.15.29:0/1511206145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9540047c0 msgr2=0x7fc954004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96e5f4640 1 --2- 172.21.15.29:0/1511206145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9540047c0 0x7fc954004c40 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fc958001750 tx=0x7fc9540057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96ddf3640 1 -- 172.21.15.29:0/1511206145 reap_dead start 2023-10-16T15:50:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96e5f4640 1 -- 172.21.15.29:0/1511206145 shutdown_connections 2023-10-16T15:50:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96e5f4640 1 -- 172.21.15.29:0/1511206145 >> 172.21.15.29:0/1511206145 conn(0x7fc9680b0a80 msgr2=0x7fc9680b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96e5f4640 1 -- 172.21.15.29:0/1511206145 shutdown_connections 2023-10-16T15:50:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.842+0000 7fc96e5f4640 1 -- 172.21.15.29:0/1511206145 wait complete. 2023-10-16T15:50:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.844+0000 7fc96e5f4640 1 Processor -- start 2023-10-16T15:50:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.844+0000 7fc96e5f4640 1 -- start start 2023-10-16T15:50:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96e5f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9540047c0 0x7fc968150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96e5f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 0x7fc9680712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96e5f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9680c7530 0x7fc9680c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96e5f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680cb090 con 0x7fc968070e50 2023-10-16T15:50:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96e5f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc968071810 con 0x7fc9680c7530 2023-10-16T15:50:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96e5f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680ca430 con 0x7fc9540047c0 2023-10-16T15:50:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96cdf1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 0x7fc9680712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96cdf1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 0x7fc9680712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50414/0 (socket says 172.21.15.29:50414) 2023-10-16T15:50:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.845+0000 7fc96cdf1640 1 -- 172.21.15.29:0/3783349231 learned_addr learned my addr 172.21.15.29:0/3783349231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96d5f2640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9540047c0 0x7fc968150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96cdf1640 1 -- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9540047c0 msgr2=0x7fc968150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9540047c0 0x7fc968150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96cdf1640 1 -- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9680c7530 msgr2=0x7fc9680c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9680c7530 0x7fc9680c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96cdf1640 1 -- 172.21.15.29:0/3783349231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680ca6f0 con 0x7fc968070e50 2023-10-16T15:50:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96cdf1640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 0x7fc9680712d0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fc958008fd0 tx=0x7fc958001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:25.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580076c0 con 0x7fc968070e50 2023-10-16T15:50:25.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc958007a30 con 0x7fc968070e50 2023-10-16T15:50:25.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958006c00 con 0x7fc968070e50 2023-10-16T15:50:25.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.846+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9680ca9d0 con 0x7fc968070e50 2023-10-16T15:50:25.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.847+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9680cb4b0 con 0x7fc968070e50 2023-10-16T15:50:25.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.848+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96804e9b0 con 0x7fc968070e50 2023-10-16T15:50:25.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.849+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 113) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc9580035f0 con 0x7fc968070e50 2023-10-16T15:50:25.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.850+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fc9580ab5a0 con 0x7fc968070e50 2023-10-16T15:50:25.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:25.854+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc958076570 con 0x7fc968070e50 2023-10-16T15:50:26.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.050+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9680ac530 con 0x7fc968070e50 2023-10-16T15:50:26.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.052+0000 7fc95dffb640 1 -- 172.21.15.29:0/3783349231 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc958079b90 con 0x7fc968070e50 2023-10-16T15:50:26.054 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:50:26.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 msgr2=0x7fc9680712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 0x7fc9680712d0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fc958008fd0 tx=0x7fc958001120 comp rx=0 tx=0).stop 2023-10-16T15:50:26.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 shutdown_connections 2023-10-16T15:50:26.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc968070e50 0x7fc9680712d0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc9680c7530 0x7fc9680c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 --2- 172.21.15.29:0/3783349231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc9540047c0 0x7fc968150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.054+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 >> 172.21.15.29:0/3783349231 conn(0x7fc9680b0a80 msgr2=0x7fc9680ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.055+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 shutdown_connections 2023-10-16T15:50:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.055+0000 7fc96e5f4640 1 -- 172.21.15.29:0/3783349231 wait complete. 2023-10-16T15:50:26.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:50:26.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.309+0000 7fa00d4da640 1 Processor -- start 2023-10-16T15:50:26.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.309+0000 7fa00d4da640 1 -- start start 2023-10-16T15:50:26.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.310+0000 7fa00d4da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 0x7fa0080b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.310+0000 7fa00d4da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0080d18c0 con 0x7fa0080b6cd0 2023-10-16T15:50:26.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.310+0000 7fa00d4da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0080d3980 con 0x7fa0080cfaf0 2023-10-16T15:50:26.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.310+0000 7fa00d4da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0080d3c60 con 0x7fa0080cf710 2023-10-16T15:50:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.311+0000 7fa0077fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa0080b6cd0 0x7fa0080d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39050/0 (socket says 172.21.15.29:39050) 2023-10-16T15:50:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.311+0000 7fa0077fe640 1 -- 172.21.15.29:0/4046294330 learned_addr learned my addr 172.21.15.29:0/4046294330 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.311+0000 7fa006ffd640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 0x7fa0080b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.311+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476965465 0 0) 0x7fa0080d18c0 con 0x7fa0080b6cd0 2023-10-16T15:50:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.311+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff4003520 con 0x7fa0080b6cd0 2023-10-16T15:50:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861048373 0 0) 0x7f9ff4003520 con 0x7fa0080b6cd0 2023-10-16T15:50:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 msgr2=0x7fa0080b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 0x7fa0080b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 >> v1:172.21.15.170:6789/0 conn(0x7fa0080cfaf0 legacy=0x7fa0080d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0080ce630 con 0x7fa0080b6cd0 2023-10-16T15:50:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa000001670 con 0x7fa0080b6cd0 2023-10-16T15:50:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 >> v1:172.21.15.29:6789/0 conn(0x7fa0080b6cd0 legacy=0x7fa0080d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff40039a0 0x7f9ff4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 0x7f9ff40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff4004d00 0x7f9ff4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0080d3c60 con 0x7f9ff4004d00 2023-10-16T15:50:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff400bb10 con 0x7f9ff40039a0 2023-10-16T15:50:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff400bcd0 con 0x7fa0080cf710 2023-10-16T15:50:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa0077fe640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff4004d00 0x7f9ff4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa006ffd640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff40039a0 0x7f9ff4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.312+0000 7fa0067fc640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 0x7f9ff40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.313+0000 7fa0077fe640 1 -- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 msgr2=0x7f9ff40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.313+0000 7fa0077fe640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa0080cf710 0x7f9ff40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.313+0000 7fa0077fe640 1 -- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff40039a0 msgr2=0x7f9ff4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.313+0000 7fa0077fe640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff40039a0 0x7f9ff4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.313+0000 7fa0077fe640 1 -- 172.21.15.29:0/4046294330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4003520 con 0x7f9ff4004d00 2023-10-16T15:50:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.314+0000 7fa0077fe640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff4004d00 0x7f9ff4005180 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fa0080b7160 tx=0x7f9ff40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465222565b513d96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.314+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000005ad0 con 0x7f9ff4004d00 2023-10-16T15:50:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.314+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa000005c30 con 0x7f9ff4004d00 2023-10-16T15:50:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.314+0000 7fa005ffb640 1 -- 172.21.15.29:0/4046294330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00002c7c0 con 0x7f9ff4004d00 2023-10-16T15:50:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa00d4da640 1 -- 172.21.15.29:0/4046294330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff4004d00 msgr2=0x7f9ff4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa00d4da640 1 --2- 172.21.15.29:0/4046294330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ff4004d00 0x7f9ff4005180 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fa0080b7160 tx=0x7f9ff40057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa0077fe640 1 -- 172.21.15.29:0/4046294330 reap_dead start 2023-10-16T15:50:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa00d4da640 1 -- 172.21.15.29:0/4046294330 shutdown_connections 2023-10-16T15:50:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa00d4da640 1 -- 172.21.15.29:0/4046294330 >> 172.21.15.29:0/4046294330 conn(0x7fa0080d2cc0 msgr2=0x7fa0080d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa00d4da640 1 -- 172.21.15.29:0/4046294330 shutdown_connections 2023-10-16T15:50:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.315+0000 7fa00d4da640 1 -- 172.21.15.29:0/4046294330 wait complete. 2023-10-16T15:50:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.317+0000 7fa00d4da640 1 Processor -- start 2023-10-16T15:50:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.317+0000 7fa00d4da640 1 -- start start 2023-10-16T15:50:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa00d4da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff4004d00 0x7fa0081f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa00d4da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 0x7fa00815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa00d4da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa00815f510 0x7fa00815f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa00d4da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00815fed0 con 0x7fa0080a0570 2023-10-16T15:50:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa00d4da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0080af930 con 0x7f9ff4004d00 2023-10-16T15:50:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa00d4da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0081f2b00 con 0x7fa00815f510 2023-10-16T15:50:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa0067fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 0x7fa00815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa0067fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 0x7fa00815b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50426/0 (socket says 172.21.15.29:50426) 2023-10-16T15:50:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.318+0000 7fa0067fc640 1 -- 172.21.15.29:0/3444408215 learned_addr learned my addr 172.21.15.29:0/3444408215 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0077fe640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa00815f510 0x7fa00815f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0067fc640 1 -- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa00815f510 msgr2=0x7fa00815f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0067fc640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa00815f510 0x7fa00815f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa006ffd640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff4004d00 0x7fa0081f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0067fc640 1 -- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff4004d00 msgr2=0x7fa0081f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0067fc640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff4004d00 0x7fa0081f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0067fc640 1 -- 172.21.15.29:0/3444408215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0081f2de0 con 0x7fa0080a0570 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa006ffd640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff4004d00 0x7fa0081f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa0067fc640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 0x7fa00815b1e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f9ff000e580 tx=0x7f9ff000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00052e0 con 0x7fa0080a0570 2023-10-16T15:50:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff0005440 con 0x7fa0080a0570 2023-10-16T15:50:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081f30c0 con 0x7fa0080a0570 2023-10-16T15:50:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0003810 con 0x7fa0080a0570 2023-10-16T15:50:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.319+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0080b35d0 con 0x7fa0080a0570 2023-10-16T15:50:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.320+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00804e160 con 0x7fa0080a0570 2023-10-16T15:50:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.321+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 113) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9ff0003970 con 0x7fa0080a0570 2023-10-16T15:50:26.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.322+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f9ff00892e0 con 0x7fa0080a0570 2023-10-16T15:50:26.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.326+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ff0054320 con 0x7fa0080a0570 2023-10-16T15:50:26.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.462+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 114) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9ff0053dd0 con 0x7fa0080a0570 2023-10-16T15:50:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.463+0000 7f9fff7fe640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9fe8070900 0x7f9fe8072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.466+0000 7fa006ffd640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9fe8070900 0x7f9fe8072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.471+0000 7fa006ffd640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9fe8070900 0x7f9fe8072cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8007440 tx=0x7f9ff800b040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:26.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.527+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa00806c1c0 con 0x7fa0080a0570 2023-10-16T15:50:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.528+0000 7f9fff7fe640 1 -- 172.21.15.29:0/3444408215 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v197) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9ff0074310 con 0x7fa0080a0570 2023-10-16T15:50:26.530 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:50:26.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9fe8070900 msgr2=0x7f9fe8072cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9fe8070900 0x7f9fe8072cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8007440 tx=0x7f9ff800b040 comp rx=0 tx=0).stop 2023-10-16T15:50:26.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 msgr2=0x7fa00815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 0x7fa00815b1e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f9ff000e580 tx=0x7f9ff000ea40 comp rx=0 tx=0).stop 2023-10-16T15:50:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 shutdown_connections 2023-10-16T15:50:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9fe8070900 0x7f9fe8072cf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0080a0570 0x7fa00815b1e0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa00815f510 0x7fa00815f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 --2- 172.21.15.29:0/3444408215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ff4004d00 0x7fa0081f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 >> 172.21.15.29:0/3444408215 conn(0x7fa0080d2cc0 msgr2=0x7fa0080cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:26.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 shutdown_connections 2023-10-16T15:50:26.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.531+0000 7fa00d4da640 1 -- 172.21.15.29:0/3444408215 wait complete. 2023-10-16T15:50:26.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.779+0000 7f41ccfa0640 1 Processor -- start 2023-10-16T15:50:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.779+0000 7f41ccfa0640 1 -- start start 2023-10-16T15:50:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41ccfa0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 0x7f41c815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41ccfa0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f41c806c100 0x7f41c806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41ccfa0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c815aef0 con 0x7f41c8159950 2023-10-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41ccfa0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c815e490 con 0x7f41c806c100 2023-10-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41ccfa0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c815d910 con 0x7f41c8070b60 2023-10-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41c77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41c8159950 0x7f41c8070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39054/0 (socket says 172.21.15.29:39054) 2023-10-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.781+0000 7f41c77fe640 1 -- 172.21.15.29:0/3292723970 learned_addr learned my addr 172.21.15.29:0/3292723970 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3292723970 >> v2:172.21.15.170:3300/0 conn(0x7f41c806c100 0x7f41c806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533844831 0 0) 0x7f41b8001110 con 0x7f41c8159950 2023-10-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41bffff640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 0x7f41c815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b4005730 con 0x7f41c8159950 2023-10-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218890083 0 0) 0x7f41b4005730 con 0x7f41c8159950 2023-10-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 msgr2=0x7f41c815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 0x7f41c815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 >> v2:172.21.15.170:3300/0 conn(0x7f41c806c100 msgr2=0x7f41c806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 --2- 172.21.15.29:0/3292723970 >> v2:172.21.15.170:3300/0 conn(0x7f41c806c100 0x7f41c806c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c815dbf0 con 0x7f41c8159950 2023-10-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f41b8001800 con 0x7f41c8159950 2023-10-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 >> v1:172.21.15.29:6789/0 conn(0x7f41c8159950 legacy=0x7f41c8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b4005b80 0x7f41b4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 0x7f41b40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41b4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c815e490 con 0x7f41b4006f10 2023-10-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c815d910 con 0x7f41b4005b80 2023-10-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.782+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b400dbc0 con 0x7f41c8070b60 2023-10-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41b4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41bffff640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 0x7f41b40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c7fff640 1 -- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 msgr2=0x7f41b40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c8070b60 0x7f41b40069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c7fff640 1 -- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b4005b80 msgr2=0x7f41b4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c77fe640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b4005b80 0x7f41b4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41b4005b80 0x7f41b4005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.783+0000 7f41c7fff640 1 -- 172.21.15.29:0/3292723970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b4005730 con 0x7f41b4006f10 2023-10-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.784+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41b4007390 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fc0 tx=0x7f41b4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27a479d0a00be3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c002cc60 con 0x7f41b4006f10 2023-10-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c000f040 con 0x7f41b4006f10 2023-10-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41c6ffd640 1 -- 172.21.15.29:0/3292723970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0034440 con 0x7f41b4006f10 2023-10-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41ccfa0640 1 -- 172.21.15.29:0/3292723970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 msgr2=0x7f41b4007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/3292723970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41b4007390 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fc0 tx=0x7f41b4007a00 comp rx=0 tx=0).stop 2023-10-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41c7fff640 1 -- 172.21.15.29:0/3292723970 reap_dead start 2023-10-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41ccfa0640 1 -- 172.21.15.29:0/3292723970 shutdown_connections 2023-10-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.785+0000 7f41ccfa0640 1 -- 172.21.15.29:0/3292723970 >> 172.21.15.29:0/3292723970 conn(0x7f41c80ce6a0 msgr2=0x7f41c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.786+0000 7f41ccfa0640 1 -- 172.21.15.29:0/3292723970 shutdown_connections 2023-10-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.786+0000 7f41ccfa0640 1 -- 172.21.15.29:0/3292723970 wait complete. 2023-10-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.787+0000 7f41ccfa0640 1 Processor -- start 2023-10-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 -- start start 2023-10-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41c80c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c8155650 0x7f41c80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80c68f0 0x7f41c80c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80c2660 con 0x7f41b4006f10 2023-10-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80c89a0 con 0x7f41c8155650 2023-10-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.788+0000 7f41ccfa0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80c1d30 con 0x7f41c80c68f0 2023-10-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41c80c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41c80c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50436/0 (socket says 172.21.15.29:50436) 2023-10-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 -- 172.21.15.29:0/1589026836 learned_addr learned my addr 172.21.15.29:0/1589026836 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41bffff640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c8155650 0x7f41c80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 -- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80c68f0 msgr2=0x7f41c80c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80c68f0 0x7f41c80c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 -- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c8155650 msgr2=0x7f41c80acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c8155650 0x7f41c80acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.789+0000 7f41c77fe640 1 -- 172.21.15.29:0/1589026836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c80c2010 con 0x7f41b4006f10 2023-10-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.790+0000 7f41c77fe640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41c80c83c0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f41b800d790 tx=0x7f41b800dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.790+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b800dea0 con 0x7f41b4006f10 2023-10-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.790+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b80074e0 con 0x7f41b4006f10 2023-10-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.790+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8004a00 con 0x7f41b4006f10 2023-10-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.790+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c80c22f0 con 0x7f41b4006f10 2023-10-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.790+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c80ad040 con 0x7f41b4006f10 2023-10-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.791+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c804e9b0 con 0x7f41b4006f10 2023-10-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.792+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 114) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f41b8004b60 con 0x7f41b4006f10 2023-10-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.793+0000 7f41c4ff9640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41a806acf0 0x7f41a806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.793+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f41b8088e20 con 0x7f41b4006f10 2023-10-16T15:50:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.794+0000 7f41bffff640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41a806acf0 0x7f41a806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.795+0000 7f41bffff640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41a806acf0 0x7f41a806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41c81562a0 tx=0x7f41b0008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:26.798+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41b8057860 con 0x7f41b4006f10 2023-10-16T15:50:27.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.032+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41c815aef0 con 0x7f41b4006f10 2023-10-16T15:50:27.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.033+0000 7f41c4ff9640 1 -- 172.21.15.29:0/1589026836 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f41c815aef0 con 0x7f41b4006f10 2023-10-16T15:50:27.036 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:27.036 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:34:26.024272+0000","modified":"2023-10-16T15:34:29.168102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2122514336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122514336},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122514336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T15:50:27.036 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 92 2023-10-16T15:50:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41a806acf0 msgr2=0x7f41a806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41a806acf0 0x7f41a806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41c81562a0 tx=0x7f41b0008040 comp rx=0 tx=0).stop 2023-10-16T15:50:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 msgr2=0x7f41c80c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41c80c83c0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f41b800d790 tx=0x7f41b800dc50 comp rx=0 tx=0).stop 2023-10-16T15:50:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 shutdown_connections 2023-10-16T15:50:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41a806acf0 0x7f41a806d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80c68f0 0x7f41c80c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c8155650 0x7f41c80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 --2- 172.21.15.29:0/1589026836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41b4006f10 0x7f41c80c83c0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 >> 172.21.15.29:0/1589026836 conn(0x7f41c80ce6a0 msgr2=0x7f41c80cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 shutdown_connections 2023-10-16T15:50:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.036+0000 7f41ccfa0640 1 -- 172.21.15.29:0/1589026836 wait complete. 2023-10-16T15:50:27.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:27.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.282+0000 7fe2fb59e640 1 Processor -- start 2023-10-16T15:50:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.283+0000 7fe2fb59e640 1 -- start start 2023-10-16T15:50:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.284+0000 7fe2fb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2fc065ab0 0x7fe2fc0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.284+0000 7fe2fb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0acd70 con 0x7fe2fc0ce280 2023-10-16T15:50:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.284+0000 7fe2fb59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0ac4a0 con 0x7fe2fc06d220 2023-10-16T15:50:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.284+0000 7fe2fb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc065f40 con 0x7fe2fc065ab0 2023-10-16T15:50:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2f1d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2fc065ab0 0x7fe2fc0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2fad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe2fc0ce280 0x7fe2fc0ad200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39064/0 (socket says 172.21.15.29:39064) 2023-10-16T15:50:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2fad9d640 1 -- 172.21.15.29:0/3831827045 learned_addr learned my addr 172.21.15.29:0/3831827045 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/3831827045 >> v1:172.21.15.170:6789/0 conn(0x7fe2fc06d220 legacy=0x7fe2fc0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/3831827045 >> v1:172.21.15.29:6789/0 conn(0x7fe2fc0ce280 legacy=0x7fe2fc0ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2f9d9b640 1 -- 172.21.15.29:0/3831827045 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757742476 0 0) 0x7fe2fc0acd70 con 0x7fe2fc0ce280 2023-10-16T15:50:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.285+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/3831827045 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc0ac720 con 0x7fe2fc065ab0 2023-10-16T15:50:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.286+0000 7fe2f1d9b640 1 --2- 172.21.15.29:0/3831827045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc065ab0 0x7fe2fc0c95e0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40097a0 tx=0x7fe2e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e6a3a6bc6a7ac9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2f9d9b640 1 -- 172.21.15.29:0/3831827045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e402e070 con 0x7fe2fc065ab0 2023-10-16T15:50:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2f9d9b640 1 -- 172.21.15.29:0/3831827045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e40315a0 con 0x7fe2fc065ab0 2023-10-16T15:50:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2f9d9b640 1 -- 172.21.15.29:0/3831827045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4031930 con 0x7fe2fc065ab0 2023-10-16T15:50:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2fb59e640 1 -- 172.21.15.29:0/3831827045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc065ab0 msgr2=0x7fe2fc0c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/3831827045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc065ab0 0x7fe2fc0c95e0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40097a0 tx=0x7fe2e402b840 comp rx=0 tx=0).stop 2023-10-16T15:50:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2fb59e640 1 -- 172.21.15.29:0/3831827045 shutdown_connections 2023-10-16T15:50:27.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/3831827045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc065ab0 0x7fe2fc0c95e0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.287+0000 7fe2fb59e640 1 -- 172.21.15.29:0/3831827045 >> 172.21.15.29:0/3831827045 conn(0x7fe2fc0b0a80 msgr2=0x7fe2fc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:27.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.288+0000 7fe2fb59e640 1 -- 172.21.15.29:0/3831827045 shutdown_connections 2023-10-16T15:50:27.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.288+0000 7fe2fb59e640 1 -- 172.21.15.29:0/3831827045 wait complete. 2023-10-16T15:50:27.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.289+0000 7fe2fb59e640 1 Processor -- start 2023-10-16T15:50:27.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.289+0000 7fe2fb59e640 1 -- start start 2023-10-16T15:50:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.290+0000 7fe2fb59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2fc06d220 0x7fe2fc158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.290+0000 7fe2fb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 0x7fe2fc0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.290+0000 7fe2fb59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2fc06b7c0 0x7fe2fc15e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.290+0000 7fe2fb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc15e380 con 0x7fe2fc0ce280 2023-10-16T15:50:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.290+0000 7fe2fb59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc15d770 con 0x7fe2fc06d220 2023-10-16T15:50:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.290+0000 7fe2fb59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc15da50 con 0x7fe2fc06b7c0 2023-10-16T15:50:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 0x7fe2fc0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 0x7fe2fc0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50452/0 (socket says 172.21.15.29:50452) 2023-10-16T15:50:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/1810719419 learned_addr learned my addr 172.21.15.29:0/1810719419 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2fa59c640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2fc06d220 0x7fe2fc158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2fad9d640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2fc06b7c0 0x7fe2fc15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2fc06b7c0 msgr2=0x7fe2fc15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2fc06b7c0 0x7fe2fc15e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2fc06d220 msgr2=0x7fe2fc158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2fc06d220 0x7fe2fc158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.291+0000 7fe2f1d9b640 1 -- 172.21.15.29:0/1810719419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e4009400 con 0x7fe2fc0ce280 2023-10-16T15:50:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.292+0000 7fe2f1d9b640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 0x7fe2fc0b1d50 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40097a0 tx=0x7fe2e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.292+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4032840 con 0x7fe2fc0ce280 2023-10-16T15:50:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.292+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e40329a0 con 0x7fe2fc0ce280 2023-10-16T15:50:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.292+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4032d50 con 0x7fe2fc0ce280 2023-10-16T15:50:27.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.292+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc15dd30 con 0x7fe2fc0ce280 2023-10-16T15:50:27.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.292+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc15d400 con 0x7fe2fc0ce280 2023-10-16T15:50:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.293+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc15e4f0 con 0x7fe2fc0ce280 2023-10-16T15:50:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.294+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 114) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe2e403b3f0 con 0x7fe2fc0ce280 2023-10-16T15:50:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.295+0000 7fe2f37fe640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fe2dc06acc0 0x7fe2dc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.295+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fe2e40ab860 con 0x7fe2fc0ce280 2023-10-16T15:50:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.296+0000 7fe2fa59c640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fe2dc06acc0 0x7fe2dc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.297+0000 7fe2fa59c640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fe2dc06acc0 0x7fe2dc06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec004ea0 tx=0x7fe2ec009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:27.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.300+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2e4076740 con 0x7fe2fc0ce280 2023-10-16T15:50:27.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.496+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2fc04e9b0 con 0x7fe2fc0ce280 2023-10-16T15:50:27.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.497+0000 7fe2f37fe640 1 -- 172.21.15.29:0/1810719419 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18207 (secure 0 0 0) 0x7fe2e4079d60 con 0x7fe2fc0ce280 2023-10-16T15:50:27.500 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:27.500 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":197,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:25.441769+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/1503925164":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/1571191706":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6833/23414019":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/2654668130":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6832/23414019":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6833/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:6832/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1849741434":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/3590601432":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1525143382":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/2183916363":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3770772716":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3372934420":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/759794124":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/3161548522":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1363341747":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1893102802":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1719808847":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/345956683":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6833/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6832/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/3541452828":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/4127022397":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/479239966":"2023-10-17T15:34:40.941061+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:27.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fe2dc06acc0 msgr2=0x7fe2dc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fe2dc06acc0 0x7fe2dc06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec004ea0 tx=0x7fe2ec009040 comp rx=0 tx=0).stop 2023-10-16T15:50:27.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 msgr2=0x7fe2fc0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 0x7fe2fc0b1d50 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40097a0 tx=0x7fe2e40046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 shutdown_connections 2023-10-16T15:50:27.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fe2dc06acc0 0x7fe2dc06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2fc0ce280 0x7fe2fc0b1d50 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2fc06b7c0 0x7fe2fc15e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 --2- 172.21.15.29:0/1810719419 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2fc06d220 0x7fe2fc158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 >> 172.21.15.29:0/1810719419 conn(0x7fe2fc0b0a80 msgr2=0x7fe2fc15c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 shutdown_connections 2023-10-16T15:50:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.501+0000 7fe2fb59e640 1 -- 172.21.15.29:0/1810719419 wait complete. 2023-10-16T15:50:27.524 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:50:27.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:27.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.750+0000 7fbad6cd7640 1 Processor -- start 2023-10-16T15:50:27.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.750+0000 7fbad6cd7640 1 -- start start 2023-10-16T15:50:27.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.751+0000 7fbad6cd7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbad015cbe0 0x7fbad00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.751+0000 7fbad6cd7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 0x7fbad00b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.751+0000 7fbad6cd7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad00d15a0 con 0x7fbad015cfc0 2023-10-16T15:50:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.751+0000 7fbad6cd7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad00d1880 con 0x7fbad015cbe0 2023-10-16T15:50:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.751+0000 7fbad6cd7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00d1b60 con 0x7fbad00b4c80 2023-10-16T15:50:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.752+0000 7fbad54d4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbad015cfc0 0x7fbad00b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39070/0 (socket says 172.21.15.29:39070) 2023-10-16T15:50:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.752+0000 7fbad54d4640 1 -- 172.21.15.29:0/264410538 learned_addr learned my addr 172.21.15.29:0/264410538 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.752+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136757031 0 0) 0x7fbad00d15a0 con 0x7fbad015cfc0 2023-10-16T15:50:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.752+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8003520 con 0x7fbad015cfc0 2023-10-16T15:50:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.752+0000 7fbad5cd5640 1 --2- 172.21.15.29:0/264410538 >> v2:172.21.15.170:3300/0 conn(0x7fbad015cbe0 0x7fbad00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.752+0000 7fbad64d6640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 0x7fbad00b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553368380 0 0) 0x7fbab8003520 con 0x7fbad015cfc0 2023-10-16T15:50:27.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 msgr2=0x7fbad00b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:27.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 0x7fbad00b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 >> v2:172.21.15.170:3300/0 conn(0x7fbad015cbe0 msgr2=0x7fbad00d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:27.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 --2- 172.21.15.29:0/264410538 >> v2:172.21.15.170:3300/0 conn(0x7fbad015cbe0 0x7fbad00d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad00d1e40 con 0x7fbad015cfc0 2023-10-16T15:50:27.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbac4001670 con 0x7fbad015cfc0 2023-10-16T15:50:27.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 >> v1:172.21.15.29:6789/0 conn(0x7fbad015cfc0 legacy=0x7fbad00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 0x7fbab8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00d1880 con 0x7fbab80047c0 2023-10-16T15:50:27.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00d1b60 con 0x7fbab8003970 2023-10-16T15:50:27.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab800bb10 con 0x7fbad00b4c80 2023-10-16T15:50:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.753+0000 7fbad54d4640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad54d4640 1 -- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 msgr2=0x7fbab8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad54d4640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbad00b4c80 0x7fbab8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad54d4640 1 -- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003970 msgr2=0x7fbab8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad5cd5640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad54d4640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad54d4640 1 -- 172.21.15.29:0/264410538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8003520 con 0x7fbab80047c0 2023-10-16T15:50:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.754+0000 7fbad5cd5640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.755+0000 7fbad54d4640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fbad00b2500 tx=0x7fbab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845040f5d072baba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.755+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4005ad0 con 0x7fbab80047c0 2023-10-16T15:50:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.755+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4005c50 con 0x7fbab80047c0 2023-10-16T15:50:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.755+0000 7fbad4cd3640 1 -- 172.21.15.29:0/264410538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac402c7a0 con 0x7fbab80047c0 2023-10-16T15:50:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad6cd7640 1 -- 172.21.15.29:0/264410538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab80047c0 msgr2=0x7fbab8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/264410538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fbad00b2500 tx=0x7fbab80057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad64d6640 1 -- 172.21.15.29:0/264410538 reap_dead start 2023-10-16T15:50:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad6cd7640 1 -- 172.21.15.29:0/264410538 shutdown_connections 2023-10-16T15:50:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad6cd7640 1 -- 172.21.15.29:0/264410538 >> 172.21.15.29:0/264410538 conn(0x7fbad00cf570 msgr2=0x7fbad00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad6cd7640 1 -- 172.21.15.29:0/264410538 shutdown_connections 2023-10-16T15:50:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.756+0000 7fbad6cd7640 1 -- 172.21.15.29:0/264410538 wait complete. 2023-10-16T15:50:27.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.758+0000 7fbad6cd7640 1 Processor -- start 2023-10-16T15:50:27.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.758+0000 7fbad6cd7640 1 -- start start 2023-10-16T15:50:27.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad6cd7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbab80047c0 0x7fbad014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad6cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 0x7fbad015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad6cd7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbad00cbba0 0x7fbad00cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad6cd7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad015efd0 con 0x7fbad015fff0 2023-10-16T15:50:27.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad6cd7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00c9e60 con 0x7fbad00cbba0 2023-10-16T15:50:27.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad6cd7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00cc540 con 0x7fbab80047c0 2023-10-16T15:50:27.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad54d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 0x7fbad015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad54d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 0x7fbad015dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50460/0 (socket says 172.21.15.29:50460) 2023-10-16T15:50:27.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.759+0000 7fbad54d4640 1 -- 172.21.15.29:0/2531405549 learned_addr learned my addr 172.21.15.29:0/2531405549 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:27.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.760+0000 7fbad54d4640 1 -- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbab80047c0 msgr2=0x7fbad014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.760+0000 7fbad54d4640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbab80047c0 0x7fbad014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.760+0000 7fbad54d4640 1 -- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbad00cbba0 msgr2=0x7fbad00cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:27.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.760+0000 7fbad54d4640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbad00cbba0 0x7fbad00cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:27.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.760+0000 7fbad54d4640 1 -- 172.21.15.29:0/2531405549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad015e730 con 0x7fbad015fff0 2023-10-16T15:50:27.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.760+0000 7fbad54d4640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 0x7fbad015dde0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fbac400c1d0 tx=0x7fbac4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:27.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.761+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac402dcc0 con 0x7fbad015fff0 2023-10-16T15:50:27.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.761+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac402de20 con 0x7fbad015fff0 2023-10-16T15:50:27.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.761+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40072b0 con 0x7fbad015fff0 2023-10-16T15:50:27.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.761+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad015ea10 con 0x7fbad015fff0 2023-10-16T15:50:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.761+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad014b450 con 0x7fbad015fff0 2023-10-16T15:50:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.762+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad00c9ba0 con 0x7fbad015fff0 2023-10-16T15:50:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.763+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 114) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbac4032070 con 0x7fbad015fff0 2023-10-16T15:50:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.764+0000 7fbace7fc640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fbabc06acf0 0x7fbabc06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.764+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7fbac403c040 con 0x7fbad015fff0 2023-10-16T15:50:27.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.765+0000 7fbad5cd5640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fbabc06acf0 0x7fbabc06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:27.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.766+0000 7fbad5cd5640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fbabc06acf0 0x7fbabc06d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbac00094c0 tx=0x7fbac000b040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:27.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:27.768+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbac4076130 con 0x7fbad015fff0 2023-10-16T15:50:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.001+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbad004e9b0 con 0x7fbad015fff0 2023-10-16T15:50:28.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.002+0000 7fbace7fc640 1 -- 172.21.15.29:0/2531405549 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fbac4079750 con 0x7fbad015fff0 2023-10-16T15:50:28.005 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:28.005 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:34:26.024272+0000","modified":"2023-10-16T15:34:29.168102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2122514336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122514336},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122514336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T15:50:28.005 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 92 2023-10-16T15:50:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fbabc06acf0 msgr2=0x7fbabc06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fbabc06acf0 0x7fbabc06d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbac00094c0 tx=0x7fbac000b040 comp rx=0 tx=0).stop 2023-10-16T15:50:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 msgr2=0x7fbad015dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 0x7fbad015dde0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fbac400c1d0 tx=0x7fbac4001930 comp rx=0 tx=0).stop 2023-10-16T15:50:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 shutdown_connections 2023-10-16T15:50:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fbabc06acf0 0x7fbabc06d1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbad015fff0 0x7fbad015dde0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbad00cbba0 0x7fbad00cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 --2- 172.21.15.29:0/2531405549 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbab80047c0 0x7fbad014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 >> 172.21.15.29:0/2531405549 conn(0x7fbad00cf570 msgr2=0x7fbad00b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 shutdown_connections 2023-10-16T15:50:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.005+0000 7fbad6cd7640 1 -- 172.21.15.29:0/2531405549 wait complete. 2023-10-16T15:50:28.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.249+0000 7f2a52b83640 1 Processor -- start 2023-10-16T15:50:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.249+0000 7f2a52b83640 1 -- start start 2023-10-16T15:50:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.250+0000 7f2a52b83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a4c0cd930 0x7f2a4c164240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a52b83640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c159790 con 0x7f2a4c0caa10 2023-10-16T15:50:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a52b83640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c0c9b20 con 0x7f2a4c1593b0 2023-10-16T15:50:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a52b83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c070370 con 0x7f2a4c0cd930 2023-10-16T15:50:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a51120640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a4c0cd930 0x7f2a4c164240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a4bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a4c0caa10 0x7f2a4c06fc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39076/0 (socket says 172.21.15.29:39076) 2023-10-16T15:50:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a4bfff640 1 -- 172.21.15.29:0/2717334514 learned_addr learned my addr 172.21.15.29:0/2717334514 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a51120640 1 -- 172.21.15.29:0/2717334514 >> v1:172.21.15.170:6789/0 conn(0x7f2a4c1593b0 legacy=0x7f2a4c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a51120640 1 -- 172.21.15.29:0/2717334514 >> v1:172.21.15.29:6789/0 conn(0x7f2a4c0caa10 legacy=0x7f2a4c06fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.251+0000 7f2a51120640 1 -- 172.21.15.29:0/2717334514 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c0c9e00 con 0x7f2a4c0cd930 2023-10-16T15:50:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.252+0000 7f2a51120640 1 --2- 172.21.15.29:0/2717334514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0cd930 0x7f2a4c164240 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f2a440097a0 tx=0x7f2a4402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55564f59d58f5c26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.253+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/2717334514 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4402f050 con 0x7f2a4c0cd930 2023-10-16T15:50:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.253+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/2717334514 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44031700 con 0x7f2a4c0cd930 2023-10-16T15:50:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.253+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/2717334514 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44031a90 con 0x7f2a4c0cd930 2023-10-16T15:50:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.253+0000 7f2a52b83640 1 -- 172.21.15.29:0/2717334514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0cd930 msgr2=0x7f2a4c164240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.253+0000 7f2a52b83640 1 --2- 172.21.15.29:0/2717334514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0cd930 0x7f2a4c164240 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f2a440097a0 tx=0x7f2a4402b7a0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.254+0000 7f2a52b83640 1 -- 172.21.15.29:0/2717334514 shutdown_connections 2023-10-16T15:50:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.254+0000 7f2a52b83640 1 --2- 172.21.15.29:0/2717334514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0cd930 0x7f2a4c164240 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.254+0000 7f2a52b83640 1 -- 172.21.15.29:0/2717334514 >> 172.21.15.29:0/2717334514 conn(0x7f2a4c0cb410 msgr2=0x7f2a4c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.254+0000 7f2a52b83640 1 -- 172.21.15.29:0/2717334514 shutdown_connections 2023-10-16T15:50:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.254+0000 7f2a52b83640 1 -- 172.21.15.29:0/2717334514 wait complete. 2023-10-16T15:50:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.256+0000 7f2a52b83640 1 Processor -- start 2023-10-16T15:50:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.256+0000 7f2a52b83640 1 -- start start 2023-10-16T15:50:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.257+0000 7f2a52b83640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0caa10 0x7f2a4c0ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.257+0000 7f2a52b83640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0cd930 0x7f2a4c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.257+0000 7f2a52b83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 0x7f2a4c0c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.257+0000 7f2a52b83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c6330 con 0x7f2a4c1593b0 2023-10-16T15:50:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.257+0000 7f2a52b83640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c159b30 con 0x7f2a4c0caa10 2023-10-16T15:50:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.257+0000 7f2a52b83640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c1440e0 con 0x7f2a4c0cd930 2023-10-16T15:50:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 0x7f2a4c0c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 0x7f2a4c0c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50474/0 (socket says 172.21.15.29:50474) 2023-10-16T15:50:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 -- 172.21.15.29:0/3329067279 learned_addr learned my addr 172.21.15.29:0/3329067279 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a4bfff640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0cd930 0x7f2a4c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a5091f640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0caa10 0x7f2a4c0ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 -- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0cd930 msgr2=0x7f2a4c15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0cd930 0x7f2a4c15af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 -- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0caa10 msgr2=0x7f2a4c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0caa10 0x7f2a4c0ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.258+0000 7f2a51120640 1 -- 172.21.15.29:0/3329067279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44009400 con 0x7f2a4c1593b0 2023-10-16T15:50:28.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.259+0000 7f2a51120640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 0x7f2a4c0c7ee0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f2a440098c0 tx=0x7f2a440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:28.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.259+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44033bb0 con 0x7f2a4c1593b0 2023-10-16T15:50:28.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.259+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44033d10 con 0x7f2a4c1593b0 2023-10-16T15:50:28.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.259+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4c0c6050 con 0x7f2a4c1593b0 2023-10-16T15:50:28.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.259+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4403c640 con 0x7f2a4c1593b0 2023-10-16T15:50:28.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.259+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c0c76e0 con 0x7f2a4c1593b0 2023-10-16T15:50:28.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.261+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4c0c7260 con 0x7f2a4c1593b0 2023-10-16T15:50:28.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.261+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 115) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2a4403a040 con 0x7f2a4c1593b0 2023-10-16T15:50:28.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.262+0000 7f2a497fa640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3806af00 0x7f2a3806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.262+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f2a440ab500 con 0x7f2a4c1593b0 2023-10-16T15:50:28.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.263+0000 7f2a5091f640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3806af00 0x7f2a3806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.264+0000 7f2a5091f640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3806af00 0x7f2a3806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c158170 tx=0x7f2a3c0094d0 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:28.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.267+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2a4c0c7260 con 0x7f2a4c1593b0 2023-10-16T15:50:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.502+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a4c0c7260 con 0x7f2a4c1593b0 2023-10-16T15:50:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.503+0000 7f2a497fa640 1 -- 172.21.15.29:0/3329067279 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f2a4c0c7260 con 0x7f2a4c1593b0 2023-10-16T15:50:28.505 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:28.505 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:34:26.024272+0000","modified":"2023-10-16T15:34:29.168102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":89,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2122514336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122514336},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122514336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T15:50:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 92 2023-10-16T15:50:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3806af00 msgr2=0x7f2a3806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3806af00 0x7f2a3806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c158170 tx=0x7f2a3c0094d0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 msgr2=0x7f2a4c0c7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 0x7f2a4c0c7ee0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f2a440098c0 tx=0x7f2a440046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 shutdown_connections 2023-10-16T15:50:28.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3806af00 0x7f2a3806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c1593b0 0x7f2a4c0c7ee0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0cd930 0x7f2a4c15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 --2- 172.21.15.29:0/3329067279 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0caa10 0x7f2a4c0ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.505+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 >> 172.21.15.29:0/3329067279 conn(0x7f2a4c0cb410 msgr2=0x7f2a4c0cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.506+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 shutdown_connections 2023-10-16T15:50:28.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.506+0000 7f2a52b83640 1 -- 172.21.15.29:0/3329067279 wait complete. 2023-10-16T15:50:28.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.754+0000 7f2213c9b640 1 Processor -- start 2023-10-16T15:50:28.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.754+0000 7f2213c9b640 1 -- start start 2023-10-16T15:50:28.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.756+0000 7f2213c9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0b0b20 0x7f220c164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.756+0000 7f2213c9b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c071450 con 0x7f220c0adb50 2023-10-16T15:50:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.756+0000 7f2213c9b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c0cc600 con 0x7f220c070fd0 2023-10-16T15:50:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.756+0000 7f2213c9b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0cc8e0 con 0x7f220c0b0b20 2023-10-16T15:50:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.756+0000 7f2211a37640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f220c0adb50 0x7f220c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39080/0 (socket says 172.21.15.29:39080) 2023-10-16T15:50:28.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.756+0000 7f2211a37640 1 -- 172.21.15.29:0/1998241773 learned_addr learned my addr 172.21.15.29:0/1998241773 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2212238640 1 --2- 172.21.15.29:0/1998241773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0b0b20 0x7f220c164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2210a35640 1 -- 172.21.15.29:0/1998241773 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924024289 0 0) 0x7f220c071450 con 0x7f220c0adb50 2023-10-16T15:50:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2210a35640 1 -- 172.21.15.29:0/1998241773 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f8003520 con 0x7f220c0adb50 2023-10-16T15:50:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2210a35640 1 -- 172.21.15.29:0/1998241773 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012624872 0 0) 0x7f220c0cc600 con 0x7f220c070fd0 2023-10-16T15:50:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2212238640 1 -- 172.21.15.29:0/1998241773 >> v1:172.21.15.170:6789/0 conn(0x7f220c070fd0 legacy=0x7f220c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2212238640 1 -- 172.21.15.29:0/1998241773 >> v1:172.21.15.29:6789/0 conn(0x7f220c0adb50 legacy=0x7f220c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.757+0000 7f2212238640 1 -- 172.21.15.29:0/1998241773 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c0ccbc0 con 0x7f220c0b0b20 2023-10-16T15:50:28.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.759+0000 7f2212238640 1 --2- 172.21.15.29:0/1998241773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0b0b20 0x7f220c164f60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2204009dc0 tx=0x7f220402bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb79c1c3ff5c1c26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.759+0000 7f2210a35640 1 -- 172.21.15.29:0/1998241773 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208001670 con 0x7f220c0b0b20 2023-10-16T15:50:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.759+0000 7f2210a35640 1 -- 172.21.15.29:0/1998241773 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220402c840 con 0x7f220c0b0b20 2023-10-16T15:50:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.759+0000 7f2210a35640 1 -- 172.21.15.29:0/1998241773 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220402cbd0 con 0x7f220c0b0b20 2023-10-16T15:50:28.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.760+0000 7f2213c9b640 1 -- 172.21.15.29:0/1998241773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0b0b20 msgr2=0x7f220c164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.760+0000 7f2213c9b640 1 --2- 172.21.15.29:0/1998241773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0b0b20 0x7f220c164f60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2204009dc0 tx=0x7f220402bd10 comp rx=0 tx=0).stop 2023-10-16T15:50:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.760+0000 7f2213c9b640 1 -- 172.21.15.29:0/1998241773 shutdown_connections 2023-10-16T15:50:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.760+0000 7f2213c9b640 1 --2- 172.21.15.29:0/1998241773 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0b0b20 0x7f220c164f60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.760+0000 7f2213c9b640 1 -- 172.21.15.29:0/1998241773 >> 172.21.15.29:0/1998241773 conn(0x7f220c0cb410 msgr2=0x7f220c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.760+0000 7f2213c9b640 1 -- 172.21.15.29:0/1998241773 shutdown_connections 2023-10-16T15:50:28.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.761+0000 7f2213c9b640 1 -- 172.21.15.29:0/1998241773 wait complete. 2023-10-16T15:50:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.763+0000 7f2213c9b640 1 Processor -- start 2023-10-16T15:50:28.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.763+0000 7f2213c9b640 1 -- start start 2023-10-16T15:50:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2213c9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 0x7f220c0c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2213c9b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0adb50 0x7f220c1f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2213c9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f220c0b0b20 0x7f220c15ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2213c9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0b3b50 con 0x7f220c070fd0 2023-10-16T15:50:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2213c9b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c164700 con 0x7f220c0b0b20 2023-10-16T15:50:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2213c9b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0b3d60 con 0x7f220c0adb50 2023-10-16T15:50:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2211a37640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 0x7f220c0c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2211a37640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 0x7f220c0c22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50476/0 (socket says 172.21.15.29:50476) 2023-10-16T15:50:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2211a37640 1 -- 172.21.15.29:0/3101875812 learned_addr learned my addr 172.21.15.29:0/3101875812 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.765+0000 7f2211236640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0adb50 0x7f220c1f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2212238640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f220c0b0b20 0x7f220c15ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2211a37640 1 -- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0adb50 msgr2=0x7f220c1f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2211a37640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0adb50 0x7f220c1f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2211a37640 1 -- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f220c0b0b20 msgr2=0x7f220c15ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2211a37640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f220c0b0b20 0x7f220c15ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2211a37640 1 -- 172.21.15.29:0/3101875812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204009a70 con 0x7f220c070fd0 2023-10-16T15:50:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2211a37640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 0x7f220c0c22d0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f220800c840 tx=0x7f220800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080074e0 con 0x7f220c070fd0 2023-10-16T15:50:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2208007660 con 0x7f220c070fd0 2023-10-16T15:50:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208004a00 con 0x7f220c070fd0 2023-10-16T15:50:28.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.766+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c0b4040 con 0x7f220c070fd0 2023-10-16T15:50:28.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.767+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c06d460 con 0x7f220c070fd0 2023-10-16T15:50:28.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.768+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c04e160 con 0x7f220c070fd0 2023-10-16T15:50:28.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.769+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 115) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2208004b60 con 0x7f220c070fd0 2023-10-16T15:50:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.771+0000 7f22027fc640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f21f006af00 0x7f21f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.771+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7f22080890b0 con 0x7f220c070fd0 2023-10-16T15:50:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.772+0000 7f2211236640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f21f006af00 0x7f21f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.773+0000 7f2211236640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f21f006af00 0x7f21f006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f21fc007fd0 tx=0x7f21fc009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.776+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2208053c30 con 0x7f220c070fd0 2023-10-16T15:50:28.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.971+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f220c0b4910 con 0x7f220c070fd0 2023-10-16T15:50:28.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.975+0000 7f22027fc640 1 -- 172.21.15.29:0/3101875812 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18207 (secure 0 0 0) 0x7f2208018090 con 0x7f220c070fd0 2023-10-16T15:50:28.977 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:28.977 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":197,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:25.441769+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:34:23.977164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T15:34:25.069119+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/1503925164":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/1571191706":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6833/23414019":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/2654668130":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6832/23414019":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6833/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:6832/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1849741434":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/3590601432":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1525143382":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/2183916363":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3770772716":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3372934420":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/759794124":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/3161548522":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1363341747":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1893102802":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1719808847":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/345956683":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6833/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6832/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/3541452828":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/4127022397":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/479239966":"2023-10-17T15:34:40.941061+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.977+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f21f006af00 msgr2=0x7f21f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.977+0000 7f2213c9b640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f21f006af00 0x7f21f006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f21fc007fd0 tx=0x7f21fc009040 comp rx=0 tx=0).stop 2023-10-16T15:50:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.977+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 msgr2=0x7f220c0c22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.977+0000 7f2213c9b640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 0x7f220c0c22d0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f220800c840 tx=0x7f220800cd00 comp rx=0 tx=0).stop 2023-10-16T15:50:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 shutdown_connections 2023-10-16T15:50:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f21f006af00 0x7f21f006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f220c0b0b20 0x7f220c15ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f220c0adb50 0x7f220c1f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 --2- 172.21.15.29:0/3101875812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f220c070fd0 0x7f220c0c22d0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 >> 172.21.15.29:0/3101875812 conn(0x7f220c0cb410 msgr2=0x7f220c0ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 shutdown_connections 2023-10-16T15:50:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:28.978+0000 7f2213c9b640 1 -- 172.21.15.29:0/3101875812 wait complete. 2023-10-16T15:50:28.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:50:29.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.217+0000 7ff9d6163640 1 Processor -- start 2023-10-16T15:50:29.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.217+0000 7ff9d6163640 1 -- start start 2023-10-16T15:50:29.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.218+0000 7ff9d6163640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00ad7a0 0x7ff9d0070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d6163640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9d00adc00 0x7ff9d0070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d6163640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d00ae1a0 con 0x7ff9d0160c80 2023-10-16T15:50:29.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d6163640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d00ae480 con 0x7ff9d00adc00 2023-10-16T15:50:29.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d6163640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00ae760 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d5962640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9d00adc00 0x7ff9d0070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d5161640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff9d0160c80 0x7ff9d0161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39090/0 (socket says 172.21.15.29:39090) 2023-10-16T15:50:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d5161640 1 -- 172.21.15.29:0/1499964396 learned_addr learned my addr 172.21.15.29:0/1499964396 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.219+0000 7ff9d4960640 1 --2- 172.21.15.29:0/1499964396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00ad7a0 0x7ff9d0070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.220+0000 7ff9d5962640 1 -- 172.21.15.29:0/1499964396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00ad7a0 msgr2=0x7ff9d0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.220+0000 7ff9d5962640 1 --2- 172.21.15.29:0/1499964396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00ad7a0 0x7ff9d0070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.220+0000 7ff9d5962640 1 -- 172.21.15.29:0/1499964396 >> v1:172.21.15.29:6789/0 conn(0x7ff9d0160c80 legacy=0x7ff9d0161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.220+0000 7ff9d5962640 1 -- 172.21.15.29:0/1499964396 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00aea40 con 0x7ff9d00adc00 2023-10-16T15:50:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.220+0000 7ff9c3fff640 1 -- 172.21.15.29:0/1499964396 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863369778 0 0) 0x7ff9d00ae1a0 con 0x7ff9d0160c80 2023-10-16T15:50:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.221+0000 7ff9d5962640 1 --2- 172.21.15.29:0/1499964396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00adc00 0x7ff9d0070780 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009d90 tx=0x7ff9c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b253298a1a6b1ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:29.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.221+0000 7ff9c3fff640 1 -- 172.21.15.29:0/1499964396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c802cc50 con 0x7ff9d00adc00 2023-10-16T15:50:29.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.221+0000 7ff9c3fff640 1 -- 172.21.15.29:0/1499964396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c802cdb0 con 0x7ff9d00adc00 2023-10-16T15:50:29.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.221+0000 7ff9c3fff640 1 -- 172.21.15.29:0/1499964396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8034660 con 0x7ff9d00adc00 2023-10-16T15:50:29.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 -- 172.21.15.29:0/1499964396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00adc00 msgr2=0x7ff9d0070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 --2- 172.21.15.29:0/1499964396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00adc00 0x7ff9d0070780 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009d90 tx=0x7ff9c802bea0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 -- 172.21.15.29:0/1499964396 shutdown_connections 2023-10-16T15:50:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 --2- 172.21.15.29:0/1499964396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00adc00 0x7ff9d0070780 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 --2- 172.21.15.29:0/1499964396 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00ad7a0 0x7ff9d0070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 -- 172.21.15.29:0/1499964396 >> 172.21.15.29:0/1499964396 conn(0x7ff9d00cc230 msgr2=0x7ff9d00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 -- 172.21.15.29:0/1499964396 shutdown_connections 2023-10-16T15:50:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.222+0000 7ff9d6163640 1 -- 172.21.15.29:0/1499964396 wait complete. 2023-10-16T15:50:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.224+0000 7ff9d6163640 1 Processor -- start 2023-10-16T15:50:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.224+0000 7ff9d6163640 1 -- start start 2023-10-16T15:50:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d6163640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 0x7ff9d015be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d6163640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00adc00 0x7ff9d015c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d6163640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d0160c80 0x7ff9d0143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d6163640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01e7950 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d6163640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01e7c30 con 0x7ff9d0160c80 2023-10-16T15:50:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d6163640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01e7f10 con 0x7ff9d00adc00 2023-10-16T15:50:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d5161640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 0x7ff9d015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d5161640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 0x7ff9d015be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50488/0 (socket says 172.21.15.29:50488) 2023-10-16T15:50:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.225+0000 7ff9d5161640 1 -- 172.21.15.29:0/974619233 learned_addr learned my addr 172.21.15.29:0/974619233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5161640 1 -- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00adc00 msgr2=0x7ff9d015c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5161640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00adc00 0x7ff9d015c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5161640 1 -- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d0160c80 msgr2=0x7ff9d0143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5962640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d0160c80 0x7ff9d0143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5161640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d0160c80 0x7ff9d0143a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5161640 1 -- 172.21.15.29:0/974619233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8009a70 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d5161640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 0x7ff9d015be70 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00c430 tx=0x7ff9cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc0074e0 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc007660 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d01e81f0 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.226+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc004a20 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.227+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d0144b00 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.228+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d004e9b0 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.229+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 115) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff9cc004b80 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.230+0000 7ff9c1ffb640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff9b406af00 0x7ff9b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.230+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6797+0+0 (secure 0 0 0) 0x7ff9cc0898d0 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.231+0000 7ff9d4960640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff9b406af00 0x7ff9b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.232+0000 7ff9d4960640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff9b406af00 0x7ff9b406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00ce470 tx=0x7ff9c4008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.234+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9cc0545e0 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.468+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff9d01e7a90 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.471 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:50:29.469+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:50:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.473+0000 7ff9c1ffb640 1 -- 172.21.15.29:0/974619233 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v93) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff9d01e7a90 con 0x7ff9d00ad7a0 2023-10-16T15:50:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:50:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff9b406af00 msgr2=0x7ff9b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff9b406af00 0x7ff9b406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00ce470 tx=0x7ff9c4008040 comp rx=0 tx=0).stop 2023-10-16T15:50:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 msgr2=0x7ff9d015be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 0x7ff9d015be70 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00c430 tx=0x7ff9cc00c8f0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 shutdown_connections 2023-10-16T15:50:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff9b406af00 0x7ff9b406d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9d0160c80 0x7ff9d0143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff9d00adc00 0x7ff9d015c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 --2- 172.21.15.29:0/974619233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d00ad7a0 0x7ff9d015be70 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 >> 172.21.15.29:0/974619233 conn(0x7ff9d00cc230 msgr2=0x7ff9d015e250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 shutdown_connections 2023-10-16T15:50:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.477+0000 7ff9d6163640 1 -- 172.21.15.29:0/974619233 wait complete. 2023-10-16T15:50:29.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:50:29.613 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:50:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.711+0000 7f0aedc7c640 1 Processor -- start 2023-10-16T15:50:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.711+0000 7f0aedc7c640 1 -- start start 2023-10-16T15:50:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0aedc7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae806c040 0x7f0ae806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0aedc7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ae80ad970 0x7f0ae815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0aedc7c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0ae815ebc0 0x7f0ae815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0aedc7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae806c960 con 0x7f0ae80ad970 2023-10-16T15:50:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0aedc7c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae815b9d0 con 0x7f0ae815ebc0 2023-10-16T15:50:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0aedc7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae815bcb0 con 0x7f0ae806c040 2023-10-16T15:50:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0ae7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ae80ad970 0x7f0ae815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0ae7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ae80ad970 0x7f0ae815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50500/0 (socket says 172.21.15.29:50500) 2023-10-16T15:50:29.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.713+0000 7f0ae7fff640 1 -- 172.21.15.29:0/447508897 learned_addr learned my addr 172.21.15.29:0/447508897 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:29.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0aed47b640 1 --2- 172.21.15.29:0/447508897 >> v2:172.21.15.170:3300/0 conn(0x7f0ae815ebc0 0x7f0ae815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0aecc7a640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae806c040 0x7f0ae806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0ae7fff640 1 -- 172.21.15.29:0/447508897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae806c040 msgr2=0x7f0ae806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:29.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0ae7fff640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae806c040 0x7f0ae806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0ae7fff640 1 -- 172.21.15.29:0/447508897 >> v2:172.21.15.170:3300/0 conn(0x7f0ae815ebc0 msgr2=0x7f0ae815f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:29.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0ae7fff640 1 --2- 172.21.15.29:0/447508897 >> v2:172.21.15.170:3300/0 conn(0x7f0ae815ebc0 0x7f0ae815f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0ae7fff640 1 -- 172.21.15.29:0/447508897 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae815bf90 con 0x7f0ae80ad970 2023-10-16T15:50:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.714+0000 7f0aecc7a640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae806c040 0x7f0ae806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.715+0000 7f0ae7fff640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae80ad970 0x7f0ae815e680 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009870 tx=0x7f0ad402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0c0eeb8721a0a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.715+0000 7f0ae77fe640 1 -- 172.21.15.29:0/447508897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4031640 con 0x7f0ae80ad970 2023-10-16T15:50:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.715+0000 7f0ae77fe640 1 -- 172.21.15.29:0/447508897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad40317a0 con 0x7f0ae80ad970 2023-10-16T15:50:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.715+0000 7f0ae77fe640 1 -- 172.21.15.29:0/447508897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4031b30 con 0x7f0ae80ad970 2023-10-16T15:50:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 -- 172.21.15.29:0/447508897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae80ad970 msgr2=0x7f0ae815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae80ad970 0x7f0ae815e680 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009870 tx=0x7f0ad402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 -- 172.21.15.29:0/447508897 shutdown_connections 2023-10-16T15:50:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/447508897 >> v2:172.21.15.170:3300/0 conn(0x7f0ae815ebc0 0x7f0ae815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae80ad970 0x7f0ae815e680 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/447508897 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae806c040 0x7f0ae806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 -- 172.21.15.29:0/447508897 >> 172.21.15.29:0/447508897 conn(0x7f0ae80ce6a0 msgr2=0x7f0ae80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 -- 172.21.15.29:0/447508897 shutdown_connections 2023-10-16T15:50:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.716+0000 7f0aedc7c640 1 -- 172.21.15.29:0/447508897 wait complete. 2023-10-16T15:50:29.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.718+0000 7f0aedc7c640 1 Processor -- start 2023-10-16T15:50:29.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.718+0000 7f0aedc7c640 1 -- start start 2023-10-16T15:50:29.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aedc7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 0x7f0ae80b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aedc7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae80ad970 0x7f0ae815a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aedc7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae815ebc0 0x7f0ae815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aedc7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae81f0ee0 con 0x7f0ae806c040 2023-10-16T15:50:29.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aedc7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae81f3920 con 0x7f0ae815ebc0 2023-10-16T15:50:29.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aedc7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae81f3c00 con 0x7f0ae80ad970 2023-10-16T15:50:29.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aecc7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 0x7f0ae80b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aecc7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 0x7f0ae80b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50514/0 (socket says 172.21.15.29:50514) 2023-10-16T15:50:29.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.719+0000 7f0aecc7a640 1 -- 172.21.15.29:0/2080372873 learned_addr learned my addr 172.21.15.29:0/2080372873 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:29.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aecc7a640 1 -- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae80ad970 msgr2=0x7f0ae815a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aed47b640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae815ebc0 0x7f0ae815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aecc7a640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae80ad970 0x7f0ae815a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aecc7a640 1 -- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae815ebc0 msgr2=0x7f0ae815a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aecc7a640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae815ebc0 0x7f0ae815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:29.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aecc7a640 1 -- 172.21.15.29:0/2080372873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40094d0 con 0x7f0ae806c040 2023-10-16T15:50:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aecc7a640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 0x7f0ae80b0a00 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00c7f0 tx=0x7f0adc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc007820 con 0x7f0ae806c040 2023-10-16T15:50:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0adc007980 con 0x7f0ae806c040 2023-10-16T15:50:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc007d30 con 0x7f0ae806c040 2023-10-16T15:50:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.720+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae81f3ee0 con 0x7f0ae806c040 2023-10-16T15:50:29.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.721+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81f4400 con 0x7f0ae806c040 2023-10-16T15:50:29.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.722+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae806b330 con 0x7f0ae806c040 2023-10-16T15:50:29.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.723+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0adc0113f0 con 0x7f0ae806c040 2023-10-16T15:50:29.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.724+0000 7f0ae57fa640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0acc06af90 0x7f0acc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:29.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.724+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6883+0+0 (secure 0 0 0) 0x7f0adc089fd0 con 0x7f0ae806c040 2023-10-16T15:50:29.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.725+0000 7f0ae7fff640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0acc06af90 0x7f0acc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:29.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.726+0000 7f0ae7fff640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0acc06af90 0x7f0acc06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4036000 tx=0x7f0ad4030000 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:29.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.729+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0adc0549d0 con 0x7f0ae806c040 2023-10-16T15:50:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:29.959+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0ae81f4120 con 0x7f0ae806c040 2023-10-16T15:50:30.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.503+0000 7f0ae57fa640 1 -- 172.21.15.29:0/2080372873 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v94) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0adc057ff0 con 0x7f0ae806c040 2023-10-16T15:50:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0acc06af90 msgr2=0x7f0acc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0acc06af90 0x7f0acc06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4036000 tx=0x7f0ad4030000 comp rx=0 tx=0).stop 2023-10-16T15:50:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 msgr2=0x7f0ae80b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 0x7f0ae80b0a00 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00c7f0 tx=0x7f0adc00ccb0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 shutdown_connections 2023-10-16T15:50:30.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0acc06af90 0x7f0acc06d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ae815ebc0 0x7f0ae815a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ae80ad970 0x7f0ae815a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 --2- 172.21.15.29:0/2080372873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae806c040 0x7f0ae80b0a00 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 >> 172.21.15.29:0/2080372873 conn(0x7f0ae80ce6a0 msgr2=0x7f0ae80cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 shutdown_connections 2023-10-16T15:50:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.507+0000 7f0aedc7c640 1 -- 172.21.15.29:0/2080372873 wait complete. 2023-10-16T15:50:30.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:50:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.756+0000 7f4a8cc02640 1 Processor -- start 2023-10-16T15:50:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.757+0000 7f4a8cc02640 1 -- start start 2023-10-16T15:50:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.758+0000 7f4a8cc02640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4a880b8480 0x7f4a880cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.758+0000 7f4a8cc02640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8815d150 con 0x7f4a8815cd70 2023-10-16T15:50:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.758+0000 7f4a8cc02640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a880d1180 con 0x7f4a880d0be0 2023-10-16T15:50:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.758+0000 7f4a8cc02640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a880d1460 con 0x7f4a880b8480 2023-10-16T15:50:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.759+0000 7f4a877fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4a8815cd70 0x7f4a880d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39112/0 (socket says 172.21.15.29:39112) 2023-10-16T15:50:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.759+0000 7f4a877fe640 1 -- 172.21.15.29:0/242309572 learned_addr learned my addr 172.21.15.29:0/242309572 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.759+0000 7f4a7ffff640 1 --2- 172.21.15.29:0/242309572 >> v2:172.21.15.170:3300/0 conn(0x7f4a880b8480 0x7f4a880cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.759+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302091880 0 0) 0x7f4a8815d150 con 0x7f4a8815cd70 2023-10-16T15:50:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.759+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a74003520 con 0x7f4a8815cd70 2023-10-16T15:50:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752953073 0 0) 0x7f4a74003520 con 0x7f4a8815cd70 2023-10-16T15:50:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 >> v2:172.21.15.170:3300/0 conn(0x7f4a880b8480 msgr2=0x7f4a880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 --2- 172.21.15.29:0/242309572 >> v2:172.21.15.170:3300/0 conn(0x7f4a880b8480 0x7f4a880cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 >> v1:172.21.15.170:6789/0 conn(0x7f4a880d0be0 legacy=0x7f4a880b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880d1740 con 0x7f4a8815cd70 2023-10-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4a78001670 con 0x7f4a8815cd70 2023-10-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 >> v1:172.21.15.29:6789/0 conn(0x7f4a8815cd70 legacy=0x7f4a880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a74003970 0x7f4a74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a740047c0 0x7f4a74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74005180 0x7f4a7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880d1460 con 0x7f4a74003970 2023-10-16T15:50:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7400c120 con 0x7f4a740047c0 2023-10-16T15:50:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7400c2e0 con 0x7f4a74005180 2023-10-16T15:50:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.760+0000 7f4a877fe640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a74003970 0x7f4a74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a7ffff640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a740047c0 0x7f4a74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a877fe640 1 -- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74005180 msgr2=0x7f4a7400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a877fe640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74005180 0x7f4a7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a877fe640 1 -- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a740047c0 msgr2=0x7f4a74004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a877fe640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a740047c0 0x7f4a74004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a877fe640 1 -- 172.21.15.29:0/242309572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74003520 con 0x7f4a74003970 2023-10-16T15:50:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.761+0000 7f4a87fff640 1 -- 172.21.15.29:0/242309572 reap_dead start 2023-10-16T15:50:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.762+0000 7f4a877fe640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a74003970 0x7f4a74003d90 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c240 tx=0x7f4a740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dbf5fba2767ad9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.762+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78005ad0 con 0x7f4a74003970 2023-10-16T15:50:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.762+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a78005c30 con 0x7f4a74003970 2023-10-16T15:50:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.762+0000 7f4a86ffd640 1 -- 172.21.15.29:0/242309572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7802caf0 con 0x7f4a74003970 2023-10-16T15:50:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 -- 172.21.15.29:0/242309572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a74003970 msgr2=0x7f4a74003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a74003970 0x7f4a74003d90 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c240 tx=0x7f4a740057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 -- 172.21.15.29:0/242309572 shutdown_connections 2023-10-16T15:50:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/242309572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a74003970 0x7f4a74003d90 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 -- 172.21.15.29:0/242309572 >> 172.21.15.29:0/242309572 conn(0x7f4a880cf570 msgr2=0x7f4a880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 -- 172.21.15.29:0/242309572 shutdown_connections 2023-10-16T15:50:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.763+0000 7f4a8cc02640 1 -- 172.21.15.29:0/242309572 wait complete. 2023-10-16T15:50:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.765+0000 7f4a8cc02640 1 Processor -- start 2023-10-16T15:50:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.765+0000 7f4a8cc02640 1 -- start start 2023-10-16T15:50:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.766+0000 7f4a8cc02640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74003970 0x7f4a881f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.766+0000 7f4a8cc02640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a880cad90 0x7f4a880cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.766+0000 7f4a8cc02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 0x7f4a8815e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.766+0000 7f4a8cc02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880cc580 con 0x7f4a8815dd70 2023-10-16T15:50:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.766+0000 7f4a8cc02640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880c9cf0 con 0x7f4a880cad90 2023-10-16T15:50:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.766+0000 7f4a8cc02640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880c9fd0 con 0x7f4a74003970 2023-10-16T15:50:30.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 0x7f4a8815e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:30.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 0x7f4a8815e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50522/0 (socket says 172.21.15.29:50522) 2023-10-16T15:50:30.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 -- 172.21.15.29:0/3529445407 learned_addr learned my addr 172.21.15.29:0/3529445407 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:30.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a7ffff640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a880cad90 0x7f4a880cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 -- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74003970 msgr2=0x7f4a881f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74003970 0x7f4a881f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 -- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a880cad90 msgr2=0x7f4a880cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a880cad90 0x7f4a880cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 -- 172.21.15.29:0/3529445407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880ca2b0 con 0x7f4a8815dd70 2023-10-16T15:50:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.767+0000 7f4a87fff640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 0x7f4a8815e1f0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f4a8808b0b0 tx=0x7f4a8000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.768+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80003970 con 0x7f4a8815dd70 2023-10-16T15:50:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.768+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a80003af0 con 0x7f4a8815dd70 2023-10-16T15:50:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.768+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a880ca590 con 0x7f4a8815dd70 2023-10-16T15:50:30.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.768+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80003ea0 con 0x7f4a8815dd70 2023-10-16T15:50:30.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.768+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a881f2070 con 0x7f4a8815dd70 2023-10-16T15:50:30.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.769+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a881f2230 con 0x7f4a8815dd70 2023-10-16T15:50:30.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.770+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a80004000 con 0x7f4a8815dd70 2023-10-16T15:50:30.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.771+0000 7f4a84ff9640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4a6806af90 0x7f4a6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:30.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.771+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6883+0+0 (secure 0 0 0) 0x7f4a80088f40 con 0x7f4a8815dd70 2023-10-16T15:50:30.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.772+0000 7f4a877fe640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4a6806af90 0x7f4a6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:30.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.773+0000 7f4a877fe640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4a6806af90 0x7f4a6806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4a78034000 tx=0x7f4a78039040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:30.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.775+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a80053a30 con 0x7f4a8815dd70 2023-10-16T15:50:30.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:30.969+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4a8804e9b0 con 0x7f4a8815dd70 2023-10-16T15:50:31.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.515+0000 7f4a84ff9640 1 -- 172.21.15.29:0/3529445407 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v199) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4a80057050 con 0x7f4a8815dd70 2023-10-16T15:50:31.518 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:50:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4a6806af90 msgr2=0x7f4a6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4a6806af90 0x7f4a6806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4a78034000 tx=0x7f4a78039040 comp rx=0 tx=0).stop 2023-10-16T15:50:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 msgr2=0x7f4a8815e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 0x7f4a8815e1f0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f4a8808b0b0 tx=0x7f4a8000c960 comp rx=0 tx=0).stop 2023-10-16T15:50:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 shutdown_connections 2023-10-16T15:50:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4a6806af90 0x7f4a6806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815dd70 0x7f4a8815e1f0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a880cad90 0x7f4a880cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 --2- 172.21.15.29:0/3529445407 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a74003970 0x7f4a881f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.521+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 >> 172.21.15.29:0/3529445407 conn(0x7f4a880cf570 msgr2=0x7f4a880cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:31.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.522+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 shutdown_connections 2023-10-16T15:50:31.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.522+0000 7f4a8cc02640 1 -- 172.21.15.29:0/3529445407 wait complete. 2023-10-16T15:50:31.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:50:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.761+0000 7fdc5f59e640 1 Processor -- start 2023-10-16T15:50:31.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.762+0000 7fdc5f59e640 1 -- start start 2023-10-16T15:50:31.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.763+0000 7fdc5f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc6015b320 0x7fdc600cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:31.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.763+0000 7fdc5f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6015b9d0 con 0x7fdc600ad970 2023-10-16T15:50:31.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.763+0000 7fdc5f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6015bcb0 con 0x7fdc6006c040 2023-10-16T15:50:31.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.763+0000 7fdc5f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6015b7d0 con 0x7fdc6015b320 2023-10-16T15:50:31.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.764+0000 7fdc5ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc6015b320 0x7fdc600cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:31.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.764+0000 7fdc57fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdc600ad970 0x7fdc600cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39116/0 (socket says 172.21.15.29:39116) 2023-10-16T15:50:31.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.764+0000 7fdc57fff640 1 -- 172.21.15.29:0/1034765538 learned_addr learned my addr 172.21.15.29:0/1034765538 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:31.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.764+0000 7fdc5ed9d640 1 -- 172.21.15.29:0/1034765538 >> v1:172.21.15.170:6789/0 conn(0x7fdc6006c040 legacy=0x7fdc6006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:31.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.764+0000 7fdc5ed9d640 1 -- 172.21.15.29:0/1034765538 >> v1:172.21.15.29:6789/0 conn(0x7fdc600ad970 legacy=0x7fdc600cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:31.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.764+0000 7fdc5ed9d640 1 -- 172.21.15.29:0/1034765538 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6015bf90 con 0x7fdc6015b320 2023-10-16T15:50:31.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.765+0000 7fdc5ed9d640 1 --2- 172.21.15.29:0/1034765538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6015b320 0x7fdc600cd180 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009dc0 tx=0x7fdc5802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915513260b665b9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:31.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.766+0000 7fdc5dd9b640 1 -- 172.21.15.29:0/1034765538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5802cc40 con 0x7fdc6015b320 2023-10-16T15:50:31.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.766+0000 7fdc5dd9b640 1 -- 172.21.15.29:0/1034765538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5802cda0 con 0x7fdc6015b320 2023-10-16T15:50:31.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.766+0000 7fdc5dd9b640 1 -- 172.21.15.29:0/1034765538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58034660 con 0x7fdc6015b320 2023-10-16T15:50:31.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 -- 172.21.15.29:0/1034765538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6015b320 msgr2=0x7fdc600cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:31.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/1034765538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6015b320 0x7fdc600cd180 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009dc0 tx=0x7fdc5802be90 comp rx=0 tx=0).stop 2023-10-16T15:50:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 -- 172.21.15.29:0/1034765538 shutdown_connections 2023-10-16T15:50:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/1034765538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6015b320 0x7fdc600cd180 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 -- 172.21.15.29:0/1034765538 >> 172.21.15.29:0/1034765538 conn(0x7fdc600ce6a0 msgr2=0x7fdc600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 -- 172.21.15.29:0/1034765538 shutdown_connections 2023-10-16T15:50:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.767+0000 7fdc5f59e640 1 -- 172.21.15.29:0/1034765538 wait complete. 2023-10-16T15:50:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.769+0000 7fdc5f59e640 1 Processor -- start 2023-10-16T15:50:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.769+0000 7fdc5f59e640 1 -- start start 2023-10-16T15:50:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 0x7fdc601e4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:31.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc600ad970 0x7fdc600acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:31.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6015b320 0x7fdc600ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:31.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc601e5d80 con 0x7fdc6006c040 2023-10-16T15:50:31.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc600ca1a0 con 0x7fdc6015b320 2023-10-16T15:50:31.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc601e54f0 con 0x7fdc600ad970 2023-10-16T15:50:31.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 0x7fdc601e4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:31.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 0x7fdc601e4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50548/0 (socket says 172.21.15.29:50548) 2023-10-16T15:50:31.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.771+0000 7fdc5e59c640 1 -- 172.21.15.29:0/2583798565 learned_addr learned my addr 172.21.15.29:0/2583798565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:31.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5ed9d640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6015b320 0x7fdc600ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:31.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5e59c640 1 -- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc600ad970 msgr2=0x7fdc600acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:31.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc57fff640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc600ad970 0x7fdc600acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:31.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5e59c640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc600ad970 0x7fdc600acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5e59c640 1 -- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6015b320 msgr2=0x7fdc600ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:31.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5e59c640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6015b320 0x7fdc600ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:31.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5e59c640 1 -- 172.21.15.29:0/2583798565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc58009a70 con 0x7fdc6006c040 2023-10-16T15:50:31.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5ed9d640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6015b320 0x7fdc600ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:31.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc57fff640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc600ad970 0x7fdc600acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc5e59c640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 0x7fdc601e4fb0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b420 tx=0x7fdc5000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.772+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50011830 con 0x7fdc6006c040 2023-10-16T15:50:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.773+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc50011990 con 0x7fdc6006c040 2023-10-16T15:50:31.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.773+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50011d40 con 0x7fdc6006c040 2023-10-16T15:50:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.773+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc601e57c0 con 0x7fdc6006c040 2023-10-16T15:50:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.773+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc600c9e30 con 0x7fdc6006c040 2023-10-16T15:50:31.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.774+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6004e9b0 con 0x7fdc6006c040 2023-10-16T15:50:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.775+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdc5001b460 con 0x7fdc6006c040 2023-10-16T15:50:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.776+0000 7fdc56ffd640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdc4006afc0 0x7fdc4006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:31.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.776+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fdc50089690 con 0x7fdc6006c040 2023-10-16T15:50:31.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.777+0000 7fdc57fff640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdc4006afc0 0x7fdc4006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:31.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.778+0000 7fdc57fff640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdc4006afc0 0x7fdc4006d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdc60158b30 tx=0x7fdc48009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:31.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.782+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdc500542b0 con 0x7fdc6006c040 2023-10-16T15:50:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:31.981+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc6006b2b0 con 0x7fdc6006c040 2023-10-16T15:50:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.139+0000 7fdc56ffd640 1 -- 172.21.15.29:0/2583798565 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v200) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdc500578d0 con 0x7fdc6006c040 2023-10-16T15:50:32.141 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:50:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.142+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdc4006afc0 msgr2=0x7fdc4006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.142+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdc4006afc0 0x7fdc4006d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdc60158b30 tx=0x7fdc48009040 comp rx=0 tx=0).stop 2023-10-16T15:50:32.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.142+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 msgr2=0x7fdc601e4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.142+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 0x7fdc601e4fb0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b420 tx=0x7fdc5000b8e0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 shutdown_connections 2023-10-16T15:50:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdc4006afc0 0x7fdc4006d480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdc6015b320 0x7fdc600ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdc600ad970 0x7fdc600acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 --2- 172.21.15.29:0/2583798565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006c040 0x7fdc601e4fb0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 >> 172.21.15.29:0/2583798565 conn(0x7fdc600ce6a0 msgr2=0x7fdc6015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.143+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 shutdown_connections 2023-10-16T15:50:32.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.145+0000 7fdc5f59e640 1 -- 172.21.15.29:0/2583798565 wait complete. 2023-10-16T15:50:32.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:50:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.395+0000 7fae65256640 1 Processor -- start 2023-10-16T15:50:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.395+0000 7fae65256640 1 -- start start 2023-10-16T15:50:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae65256640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae6015ad70 0x7fae60069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae65256640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600c9ca0 0x7fae6006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae65256640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae600cb030 con 0x7fae600ca3b0 2023-10-16T15:50:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae65256640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae600c91c0 con 0x7fae6015ad70 2023-10-16T15:50:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae65256640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae600c94a0 con 0x7fae600c9ca0 2023-10-16T15:50:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae5ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae6015ad70 0x7fae60069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae5ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae6015ad70 0x7fae60069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50558/0 (socket says 172.21.15.29:50558) 2023-10-16T15:50:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.397+0000 7fae5ed9d640 1 -- 172.21.15.29:0/4133103993 learned_addr learned my addr 172.21.15.29:0/4133103993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.398+0000 7fae5ed9d640 1 -- 172.21.15.29:0/4133103993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600c9ca0 msgr2=0x7fae6006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:32.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.398+0000 7fae5ed9d640 1 --2- 172.21.15.29:0/4133103993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600c9ca0 0x7fae6006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.398+0000 7fae5ed9d640 1 -- 172.21.15.29:0/4133103993 >> v1:172.21.15.170:6789/0 conn(0x7fae600ca3b0 legacy=0x7fae600ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:32.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.398+0000 7fae5ed9d640 1 -- 172.21.15.29:0/4133103993 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae600c9780 con 0x7fae6015ad70 2023-10-16T15:50:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.399+0000 7fae5ed9d640 1 --2- 172.21.15.29:0/4133103993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae6015ad70 0x7fae60069da0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fae50009870 tx=0x7fae5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f524687e80b92e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.399+0000 7fae5e59c640 1 -- 172.21.15.29:0/4133103993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50031640 con 0x7fae6015ad70 2023-10-16T15:50:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.399+0000 7fae5e59c640 1 -- 172.21.15.29:0/4133103993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae500317a0 con 0x7fae6015ad70 2023-10-16T15:50:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.399+0000 7fae5e59c640 1 -- 172.21.15.29:0/4133103993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50031b30 con 0x7fae6015ad70 2023-10-16T15:50:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 -- 172.21.15.29:0/4133103993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae6015ad70 msgr2=0x7fae60069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 --2- 172.21.15.29:0/4133103993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae6015ad70 0x7fae60069da0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fae50009870 tx=0x7fae5002b8f0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 -- 172.21.15.29:0/4133103993 shutdown_connections 2023-10-16T15:50:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 --2- 172.21.15.29:0/4133103993 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600c9ca0 0x7fae6006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 --2- 172.21.15.29:0/4133103993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae6015ad70 0x7fae60069da0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 -- 172.21.15.29:0/4133103993 >> 172.21.15.29:0/4133103993 conn(0x7fae600ce9b0 msgr2=0x7fae600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:32.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 -- 172.21.15.29:0/4133103993 shutdown_connections 2023-10-16T15:50:32.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.400+0000 7fae65256640 1 -- 172.21.15.29:0/4133103993 wait complete. 2023-10-16T15:50:32.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.402+0000 7fae65256640 1 Processor -- start 2023-10-16T15:50:32.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.402+0000 7fae65256640 1 -- start start 2023-10-16T15:50:32.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae65256640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 0x7fae60159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae65256640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600ca3b0 0x7fae600b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae65256640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae6015ad70 0x7fae600a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae65256640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600b3030 con 0x7fae600c9ca0 2023-10-16T15:50:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae65256640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600b3310 con 0x7fae6015ad70 2023-10-16T15:50:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae65256640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae600b35f0 con 0x7fae600ca3b0 2023-10-16T15:50:32.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae5ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 0x7fae60159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae5ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 0x7fae60159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50564/0 (socket says 172.21.15.29:50564) 2023-10-16T15:50:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.403+0000 7fae5ed9d640 1 -- 172.21.15.29:0/1709252757 learned_addr learned my addr 172.21.15.29:0/1709252757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae5ed9d640 1 -- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600ca3b0 msgr2=0x7fae600b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae5ed9d640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600ca3b0 0x7fae600b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae5ed9d640 1 -- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae6015ad70 msgr2=0x7fae600a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae5ed9d640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae6015ad70 0x7fae600a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae5ed9d640 1 -- 172.21.15.29:0/1709252757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae500094d0 con 0x7fae600c9ca0 2023-10-16T15:50:32.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae5ed9d640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 0x7fae60159520 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fae50009870 tx=0x7fae500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:32.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5003ac60 con 0x7fae600c9ca0 2023-10-16T15:50:32.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5003adc0 con 0x7fae600c9ca0 2023-10-16T15:50:32.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae600b38d0 con 0x7fae600c9ca0 2023-10-16T15:50:32.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50044630 con 0x7fae600c9ca0 2023-10-16T15:50:32.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.404+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae60159a60 con 0x7fae600c9ca0 2023-10-16T15:50:32.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.406+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6004e160 con 0x7fae600c9ca0 2023-10-16T15:50:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.407+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae5002f050 con 0x7fae600c9ca0 2023-10-16T15:50:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.407+0000 7fae577fe640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fae4006afc0 0x7fae4006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.408+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fae500ab980 con 0x7fae600c9ca0 2023-10-16T15:50:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.408+0000 7fae57fff640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fae4006afc0 0x7fae4006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.409+0000 7fae57fff640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fae4006afc0 0x7fae4006d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fae600ad4a0 tx=0x7fae48006c90 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:32.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.412+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae500766d0 con 0x7fae600c9ca0 2023-10-16T15:50:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.607+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fae6015a400 con 0x7fae600c9ca0 2023-10-16T15:50:32.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.613+0000 7fae577fe640 1 -- 172.21.15.29:0/1709252757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7fae50031e60 con 0x7fae600c9ca0 2023-10-16T15:50:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.616+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fae4006afc0 msgr2=0x7fae4006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.616+0000 7fae65256640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fae4006afc0 0x7fae4006d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fae600ad4a0 tx=0x7fae48006c90 comp rx=0 tx=0).stop 2023-10-16T15:50:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.616+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 msgr2=0x7fae60159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.616+0000 7fae65256640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 0x7fae60159520 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fae50009870 tx=0x7fae500046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 shutdown_connections 2023-10-16T15:50:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fae4006afc0 0x7fae4006d480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae6015ad70 0x7fae600a90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fae600ca3b0 0x7fae600b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 --2- 172.21.15.29:0/1709252757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae600c9ca0 0x7fae60159520 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 >> 172.21.15.29:0/1709252757 conn(0x7fae600ce9b0 msgr2=0x7fae600ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 shutdown_connections 2023-10-16T15:50:32.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.617+0000 7fae65256640 1 -- 172.21.15.29:0/1709252757 wait complete. 2023-10-16T15:50:32.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:50:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.872+0000 7f66e4fa9640 1 Processor -- start 2023-10-16T15:50:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.872+0000 7f66e4fa9640 1 -- start start 2023-10-16T15:50:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.873+0000 7f66e4fa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e015be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.873+0000 7f66e4fa9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66e006adb0 0x7f66e006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.873+0000 7f66e4fa9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f66e015e950 0x7f66e015ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.873+0000 7f66e4fa9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e006dc00 con 0x7f66e006adb0 2023-10-16T15:50:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.873+0000 7f66e4fa9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e006b750 con 0x7f66e015e950 2023-10-16T15:50:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.873+0000 7f66e4fa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e015c3a0 con 0x7f66e015adf0 2023-10-16T15:50:32.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66e006adb0 0x7f66e006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66e006adb0 0x7f66e006b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50566/0 (socket says 172.21.15.29:50566) 2023-10-16T15:50:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 -- 172.21.15.29:0/746762610 learned_addr learned my addr 172.21.15.29:0/746762610 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66ded9d640 1 --2- 172.21.15.29:0/746762610 >> v2:172.21.15.170:3300/0 conn(0x7f66e015e950 0x7f66e015ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66de59c640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 -- 172.21.15.29:0/746762610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 msgr2=0x7f66e015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e015be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 -- 172.21.15.29:0/746762610 >> v2:172.21.15.170:3300/0 conn(0x7f66e015e950 msgr2=0x7f66e015ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 --2- 172.21.15.29:0/746762610 >> v2:172.21.15.170:3300/0 conn(0x7f66e015e950 0x7f66e015ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66d5d9b640 1 -- 172.21.15.29:0/746762610 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e006cf80 con 0x7f66e006adb0 2023-10-16T15:50:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.874+0000 7f66ded9d640 1 --2- 172.21.15.29:0/746762610 >> v2:172.21.15.170:3300/0 conn(0x7f66e015e950 0x7f66e015ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.875+0000 7f66de59c640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.876+0000 7f66d5d9b640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e006adb0 0x7f66e006b210 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f66e00cf1f0 tx=0x7f66c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f9935aa48e90bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.876+0000 7f66ddd9b640 1 -- 172.21.15.29:0/746762610 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8031870 con 0x7f66e006adb0 2023-10-16T15:50:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.876+0000 7f66ddd9b640 1 -- 172.21.15.29:0/746762610 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c80319d0 con 0x7f66e006adb0 2023-10-16T15:50:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.876+0000 7f66ddd9b640 1 -- 172.21.15.29:0/746762610 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8031d60 con 0x7f66e006adb0 2023-10-16T15:50:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 -- 172.21.15.29:0/746762610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e006adb0 msgr2=0x7f66e006b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e006adb0 0x7f66e006b210 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f66e00cf1f0 tx=0x7f66c802b910 comp rx=0 tx=0).stop 2023-10-16T15:50:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 -- 172.21.15.29:0/746762610 shutdown_connections 2023-10-16T15:50:32.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/746762610 >> v2:172.21.15.170:3300/0 conn(0x7f66e015e950 0x7f66e015ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e006adb0 0x7f66e006b210 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/746762610 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 -- 172.21.15.29:0/746762610 >> 172.21.15.29:0/746762610 conn(0x7f66e00b0d30 msgr2=0x7f66e00b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 -- 172.21.15.29:0/746762610 shutdown_connections 2023-10-16T15:50:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.877+0000 7f66e4fa9640 1 -- 172.21.15.29:0/746762610 wait complete. 2023-10-16T15:50:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.879+0000 7f66e4fa9640 1 Processor -- start 2023-10-16T15:50:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.879+0000 7f66e4fa9640 1 -- start start 2023-10-16T15:50:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66e4fa9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e006adb0 0x7f66e00b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66e4fa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e009ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66e4fa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 0x7f66e0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66e4fa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e009e810 con 0x7f66e015e950 2023-10-16T15:50:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66e4fa9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e009eaf0 con 0x7f66e006adb0 2023-10-16T15:50:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66e4fa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e009edd0 con 0x7f66e015adf0 2023-10-16T15:50:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 0x7f66e0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 0x7f66e0159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50582/0 (socket says 172.21.15.29:50582) 2023-10-16T15:50:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.880+0000 7f66ded9d640 1 -- 172.21.15.29:0/131996525 learned_addr learned my addr 172.21.15.29:0/131996525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:32.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66de59c640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e006adb0 0x7f66e00b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66d5d9b640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e009ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66ded9d640 1 -- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 msgr2=0x7f66e009ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66ded9d640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e009ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66ded9d640 1 -- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e006adb0 msgr2=0x7f66e00b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66ded9d640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e006adb0 0x7f66e00b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66ded9d640 1 -- 172.21.15.29:0/131996525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80094d0 con 0x7f66e015e950 2023-10-16T15:50:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.881+0000 7f66ded9d640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 0x7f66e0159d70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f66d800c7c0 tx=0x7f66d800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.882+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8004230 con 0x7f66e015e950 2023-10-16T15:50:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.882+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d8004390 con 0x7f66e015e950 2023-10-16T15:50:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.882+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8007860 con 0x7f66e015e950 2023-10-16T15:50:32.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.882+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e009f0b0 con 0x7f66e015e950 2023-10-16T15:50:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.882+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e009f8b0 con 0x7f66e015e950 2023-10-16T15:50:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.883+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e009f5d0 con 0x7f66e015e950 2023-10-16T15:50:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.884+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f66d80079c0 con 0x7f66e015e950 2023-10-16T15:50:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.885+0000 7f66d77fe640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f66c006af90 0x7f66c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:32.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.885+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f66d8089be0 con 0x7f66e015e950 2023-10-16T15:50:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.886+0000 7f66de59c640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f66c006af90 0x7f66c006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.887+0000 7f66de59c640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f66c006af90 0x7f66c006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66d00075b0 tx=0x7f66d0007540 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:32.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:32.890+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66e009f5d0 con 0x7f66e015e950 2023-10-16T15:50:33.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.085+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f66e009f5d0 con 0x7f66e015e950 2023-10-16T15:50:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.092+0000 7f66d77fe640 1 -- 172.21.15.29:0/131996525 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v68) v1 ==== 115+0+0 (secure 0 0 0) 0x7f66e009f5d0 con 0x7f66e015e950 2023-10-16T15:50:33.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f66c006af90 msgr2=0x7f66c006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f66c006af90 0x7f66c006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66d00075b0 tx=0x7f66d0007540 comp rx=0 tx=0).stop 2023-10-16T15:50:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 msgr2=0x7f66e0159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 0x7f66e0159d70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f66d800c7c0 tx=0x7f66d800cc80 comp rx=0 tx=0).stop 2023-10-16T15:50:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 shutdown_connections 2023-10-16T15:50:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f66c006af90 0x7f66c006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66e015e950 0x7f66e0159d70 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66e015adf0 0x7f66e009ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 --2- 172.21.15.29:0/131996525 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e006adb0 0x7f66e00b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 >> 172.21.15.29:0/131996525 conn(0x7f66e00b0d30 msgr2=0x7f66e0070370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 shutdown_connections 2023-10-16T15:50:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.095+0000 7f66e4fa9640 1 -- 172.21.15.29:0/131996525 wait complete. 2023-10-16T15:50:33.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:50:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.341+0000 7fade0efc640 1 Processor -- start 2023-10-16T15:50:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.342+0000 7fade0efc640 1 -- start start 2023-10-16T15:50:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.343+0000 7fade0efc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faddc15d030 0x7faddc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.343+0000 7fade0efc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 0x7faddc0cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.343+0000 7fade0efc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0b5d90 con 0x7faddc15d410 2023-10-16T15:50:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.343+0000 7fade0efc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faddc0d2b50 con 0x7faddc15d030 2023-10-16T15:50:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.343+0000 7fade0efc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0cdbf0 con 0x7faddc0cd260 2023-10-16T15:50:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.344+0000 7faddad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 0x7faddc0cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.344+0000 7faddad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 0x7faddc0cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:55468/0 (socket says 172.21.15.29:55468) 2023-10-16T15:50:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.344+0000 7faddad9d640 1 -- 172.21.15.29:0/3976086430 learned_addr learned my addr 172.21.15.29:0/3976086430 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.344+0000 7fadda59c640 1 --2- 172.21.15.29:0/3976086430 >> v2:172.21.15.170:3300/0 conn(0x7faddc15d030 0x7faddc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.345+0000 7faddad9d640 1 -- 172.21.15.29:0/3976086430 >> v2:172.21.15.170:3300/0 conn(0x7faddc15d030 msgr2=0x7faddc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.345+0000 7faddad9d640 1 --2- 172.21.15.29:0/3976086430 >> v2:172.21.15.170:3300/0 conn(0x7faddc15d030 0x7faddc0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.345+0000 7faddad9d640 1 -- 172.21.15.29:0/3976086430 >> v1:172.21.15.170:6789/0 conn(0x7faddc15d410 legacy=0x7faddc161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.345+0000 7faddad9d640 1 -- 172.21.15.29:0/3976086430 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc160fc0 con 0x7faddc0cd260 2023-10-16T15:50:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.346+0000 7faddad9d640 1 --2- 172.21.15.29:0/3976086430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 0x7faddc0cd6b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009e50 tx=0x7fadd402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc5e8184cf9feb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.347+0000 7fadd9d9b640 1 -- 172.21.15.29:0/3976086430 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd402cb90 con 0x7faddc0cd260 2023-10-16T15:50:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.347+0000 7fadd9d9b640 1 -- 172.21.15.29:0/3976086430 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd402ccf0 con 0x7faddc0cd260 2023-10-16T15:50:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.347+0000 7fadd9d9b640 1 -- 172.21.15.29:0/3976086430 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4034610 con 0x7faddc0cd260 2023-10-16T15:50:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.347+0000 7fade0efc640 1 -- 172.21.15.29:0/3976086430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 msgr2=0x7faddc0cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.347+0000 7fade0efc640 1 --2- 172.21.15.29:0/3976086430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 0x7faddc0cd6b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009e50 tx=0x7fadd402bff0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.348+0000 7fade0efc640 1 -- 172.21.15.29:0/3976086430 shutdown_connections 2023-10-16T15:50:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.348+0000 7fade0efc640 1 --2- 172.21.15.29:0/3976086430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc0cd260 0x7faddc0cd6b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.348+0000 7fade0efc640 1 --2- 172.21.15.29:0/3976086430 >> v2:172.21.15.170:3300/0 conn(0x7faddc15d030 0x7faddc0d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.348+0000 7fade0efc640 1 -- 172.21.15.29:0/3976086430 >> 172.21.15.29:0/3976086430 conn(0x7faddc0cf720 msgr2=0x7faddc0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.348+0000 7fade0efc640 1 -- 172.21.15.29:0/3976086430 shutdown_connections 2023-10-16T15:50:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.348+0000 7fade0efc640 1 -- 172.21.15.29:0/3976086430 wait complete. 2023-10-16T15:50:33.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.350+0000 7fade0efc640 1 Processor -- start 2023-10-16T15:50:33.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.350+0000 7fade0efc640 1 -- start start 2023-10-16T15:50:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fade0efc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 0x7faddc0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fade0efc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faddc15d030 0x7faddc151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fade0efc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc15d410 0x7faddc158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fade0efc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc1515b0 con 0x7faddc0cd260 2023-10-16T15:50:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fade0efc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0c8f60 con 0x7faddc15d030 2023-10-16T15:50:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fade0efc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc15d930 con 0x7faddc15d410 2023-10-16T15:50:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 0x7faddc0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 0x7faddc0c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50592/0 (socket says 172.21.15.29:50592) 2023-10-16T15:50:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 -- 172.21.15.29:0/349341356 learned_addr learned my addr 172.21.15.29:0/349341356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadd3fff640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faddc15d030 0x7faddc151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 -- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc15d410 msgr2=0x7faddc158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc15d410 0x7faddc158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 -- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faddc15d030 msgr2=0x7faddc151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faddc15d030 0x7faddc151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.352+0000 7fadda59c640 1 -- 172.21.15.29:0/349341356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4009b30 con 0x7faddc0cd260 2023-10-16T15:50:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fadd3fff640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faddc15d030 0x7faddc151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fadda59c640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 0x7faddc0c9eb0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e9b0 tx=0x7fadcc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc009670 con 0x7faddc0cd260 2023-10-16T15:50:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc00c9a0 con 0x7faddc0cd260 2023-10-16T15:50:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00cd50 con 0x7faddc0cd260 2023-10-16T15:50:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc15dc10 con 0x7faddc0cd260 2023-10-16T15:50:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.353+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc0cca60 con 0x7faddc0cd260 2023-10-16T15:50:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.355+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc0d1ef0 con 0x7faddc0cd260 2023-10-16T15:50:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.356+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fadcc013470 con 0x7faddc0cd260 2023-10-16T15:50:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.357+0000 7fadd2ffd640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fadbc06af90 0x7fadbc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.357+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fadcc089b20 con 0x7faddc0cd260 2023-10-16T15:50:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.358+0000 7fadd3fff640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fadbc06af90 0x7fadbc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.359+0000 7fadd3fff640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fadbc06af90 0x7fadbc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faddc1e6c70 tx=0x7fadc4009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:33.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.363+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadcc054980 con 0x7faddc0cd260 2023-10-16T15:50:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.558+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7faddc04e160 con 0x7faddc0cd260 2023-10-16T15:50:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.564+0000 7fadd2ffd640 1 -- 172.21.15.29:0/349341356 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7fadcc057fa0 con 0x7faddc0cd260 2023-10-16T15:50:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fadbc06af90 msgr2=0x7fadbc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fadbc06af90 0x7fadbc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faddc1e6c70 tx=0x7fadc4009040 comp rx=0 tx=0).stop 2023-10-16T15:50:33.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 msgr2=0x7faddc0c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 0x7faddc0c9eb0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e9b0 tx=0x7fadcc00ee70 comp rx=0 tx=0).stop 2023-10-16T15:50:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 shutdown_connections 2023-10-16T15:50:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fadbc06af90 0x7fadbc06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faddc15d410 0x7faddc158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faddc15d030 0x7faddc151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 --2- 172.21.15.29:0/349341356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faddc0cd260 0x7faddc0c9eb0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 >> 172.21.15.29:0/349341356 conn(0x7faddc0cf720 msgr2=0x7faddc15c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 shutdown_connections 2023-10-16T15:50:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.568+0000 7fade0efc640 1 -- 172.21.15.29:0/349341356 wait complete. 2023-10-16T15:50:33.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T15:50:33.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.815+0000 7fa12e121640 1 Processor -- start 2023-10-16T15:50:33.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.815+0000 7fa12e121640 1 -- start start 2023-10-16T15:50:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.817+0000 7fa12e121640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa12815c9f0 0x7fa1280c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.817+0000 7fa12e121640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 0x7fa1280ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.817+0000 7fa12e121640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa128160a90 con 0x7fa1280c9230 2023-10-16T15:50:33.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.817+0000 7fa12e121640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1280cdec0 con 0x7fa12815c9f0 2023-10-16T15:50:33.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.817+0000 7fa12e121640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cd900 con 0x7fa12815dde0 2023-10-16T15:50:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.818+0000 7fa12d920640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 0x7fa1280ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.818+0000 7fa12d920640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 0x7fa1280ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:55484/0 (socket says 172.21.15.29:55484) 2023-10-16T15:50:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.818+0000 7fa12d920640 1 -- 172.21.15.29:0/4293177745 learned_addr learned my addr 172.21.15.29:0/4293177745 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.818+0000 7fa12d11f640 1 --2- 172.21.15.29:0/4293177745 >> v2:172.21.15.170:3300/0 conn(0x7fa12815c9f0 0x7fa1280c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.819+0000 7fa12d920640 1 -- 172.21.15.29:0/4293177745 >> v2:172.21.15.170:3300/0 conn(0x7fa12815c9f0 msgr2=0x7fa1280c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:33.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.819+0000 7fa12d920640 1 --2- 172.21.15.29:0/4293177745 >> v2:172.21.15.170:3300/0 conn(0x7fa12815c9f0 0x7fa1280c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.819+0000 7fa12d920640 1 -- 172.21.15.29:0/4293177745 >> v1:172.21.15.170:6789/0 conn(0x7fa1280c9230 legacy=0x7fa1280709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.819+0000 7fa12d920640 1 -- 172.21.15.29:0/4293177745 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280cdbe0 con 0x7fa12815dde0 2023-10-16T15:50:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.820+0000 7fa12d920640 1 --2- 172.21.15.29:0/4293177745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 0x7fa1280ac890 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa120009e50 tx=0x7fa12002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=446006af672ec851 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:33.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.821+0000 7fa11bfff640 1 -- 172.21.15.29:0/4293177745 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12002cb00 con 0x7fa12815dde0 2023-10-16T15:50:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.821+0000 7fa11bfff640 1 -- 172.21.15.29:0/4293177745 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12002cc60 con 0x7fa12815dde0 2023-10-16T15:50:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.821+0000 7fa11bfff640 1 -- 172.21.15.29:0/4293177745 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1200343f0 con 0x7fa12815dde0 2023-10-16T15:50:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 -- 172.21.15.29:0/4293177745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 msgr2=0x7fa1280ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 --2- 172.21.15.29:0/4293177745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 0x7fa1280ac890 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa120009e50 tx=0x7fa12002bf60 comp rx=0 tx=0).stop 2023-10-16T15:50:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 -- 172.21.15.29:0/4293177745 shutdown_connections 2023-10-16T15:50:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 --2- 172.21.15.29:0/4293177745 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa12815dde0 0x7fa1280ac890 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 --2- 172.21.15.29:0/4293177745 >> v2:172.21.15.170:3300/0 conn(0x7fa12815c9f0 0x7fa1280c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 -- 172.21.15.29:0/4293177745 >> 172.21.15.29:0/4293177745 conn(0x7fa1280ce6a0 msgr2=0x7fa1280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:33.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 -- 172.21.15.29:0/4293177745 shutdown_connections 2023-10-16T15:50:33.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.822+0000 7fa12e121640 1 -- 172.21.15.29:0/4293177745 wait complete. 2023-10-16T15:50:33.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.824+0000 7fa12e121640 1 Processor -- start 2023-10-16T15:50:33.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.824+0000 7fa12e121640 1 -- start start 2023-10-16T15:50:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12e121640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1280c9230 0x7fa1280c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12e121640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa12815c9f0 0x7fa12806b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12e121640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 0x7fa12806b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12e121640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12806be10 con 0x7fa12815dde0 2023-10-16T15:50:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12e121640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280c75e0 con 0x7fa12815c9f0 2023-10-16T15:50:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12e121640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280c77f0 con 0x7fa1280c9230 2023-10-16T15:50:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12d920640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 0x7fa12806b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12d920640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 0x7fa12806b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50606/0 (socket says 172.21.15.29:50606) 2023-10-16T15:50:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.825+0000 7fa12d920640 1 -- 172.21.15.29:0/2590785753 learned_addr learned my addr 172.21.15.29:0/2590785753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12c91e640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa12815c9f0 0x7fa12806b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d11f640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1280c9230 0x7fa1280c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d920640 1 -- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1280c9230 msgr2=0x7fa1280c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d920640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1280c9230 0x7fa1280c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d920640 1 -- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa12815c9f0 msgr2=0x7fa12806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d920640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa12815c9f0 0x7fa12806b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d920640 1 -- 172.21.15.29:0/2590785753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa120009b30 con 0x7fa12815dde0 2023-10-16T15:50:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12d920640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 0x7fa12806b8d0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fa120009e50 tx=0x7fa120004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120040670 con 0x7fa12815dde0 2023-10-16T15:50:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa120040810 con 0x7fa12815dde0 2023-10-16T15:50:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1280c7ad0 con 0x7fa12815dde0 2023-10-16T15:50:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.826+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120040bc0 con 0x7fa12815dde0 2023-10-16T15:50:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.827+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1280c8090 con 0x7fa12815dde0 2023-10-16T15:50:33.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.828+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12804e9b0 con 0x7fa12815dde0 2023-10-16T15:50:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.829+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa120040d20 con 0x7fa12815dde0 2023-10-16T15:50:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.830+0000 7fa119ffb640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa10c06af90 0x7fa10c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.830+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa1200ac8b0 con 0x7fa12815dde0 2023-10-16T15:50:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.831+0000 7fa12d11f640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa10c06af90 0x7fa10c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.832+0000 7fa12d11f640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa10c06af90 0x7fa10c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa124009ad0 tx=0x7fa124008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:33.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:33.834+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa120077820 con 0x7fa12815dde0 2023-10-16T15:50:34.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.102+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fa1280adac0 con 0x7fa12815dde0 2023-10-16T15:50:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.524+0000 7fa119ffb640 1 -- 172.21.15.29:0/2590785753 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3060) v1 ==== 165+0+0 (secure 0 0 0) 0x7fa120009510 con 0x7fa12815dde0 2023-10-16T15:50:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa10c06af90 msgr2=0x7fa10c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa10c06af90 0x7fa10c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa124009ad0 tx=0x7fa124008040 comp rx=0 tx=0).stop 2023-10-16T15:50:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 msgr2=0x7fa12806b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 0x7fa12806b8d0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fa120009e50 tx=0x7fa120004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 shutdown_connections 2023-10-16T15:50:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa10c06af90 0x7fa10c06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa12815dde0 0x7fa12806b8d0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa12815c9f0 0x7fa12806b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 --2- 172.21.15.29:0/2590785753 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1280c9230 0x7fa1280c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 >> 172.21.15.29:0/2590785753 conn(0x7fa1280ce6a0 msgr2=0x7fa1280c16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.528+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 shutdown_connections 2023-10-16T15:50:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.529+0000 7fa12e121640 1 -- 172.21.15.29:0/2590785753 wait complete. 2023-10-16T15:50:34.554 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-16T15:50:34.554 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-16T15:50:34.554 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T15:50:34.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.786+0000 7fd0d1eda640 1 Processor -- start 2023-10-16T15:50:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.786+0000 7fd0d1eda640 1 -- start start 2023-10-16T15:50:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.788+0000 7fd0d1eda640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0cc15aaa0 0x7fd0cc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.788+0000 7fd0d1eda640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.788+0000 7fd0d1eda640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc15ed60 con 0x7fd0cc0ccaa0 2023-10-16T15:50:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.788+0000 7fd0d1eda640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc06b810 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.788+0000 7fd0d1eda640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc0707e0 con 0x7fd0cc15ae80 2023-10-16T15:50:34.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0cc15aaa0 0x7fd0cc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:34.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0caffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:34.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd0cc15aaa0 0x7fd0cc15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47576/0 (socket says 172.21.15.29:47576) 2023-10-16T15:50:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 -- 172.21.15.29:0/2492479030 learned_addr learned my addr 172.21.15.29:0/2492479030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0ca7fc640 1 -- 172.21.15.29:0/2492479030 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247315636 0 0) 0x7fd0cc15ed60 con 0x7fd0cc0ccaa0 2023-10-16T15:50:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0ca7fc640 1 -- 172.21.15.29:0/2492479030 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b4003520 con 0x7fd0cc0ccaa0 2023-10-16T15:50:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 -- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 msgr2=0x7fd0cc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:34.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 --2- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 -- 172.21.15.29:0/2492479030 >> v1:172.21.15.29:6789/0 conn(0x7fd0cc0ccaa0 legacy=0x7fd0cc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0cb7fe640 1 -- 172.21.15.29:0/2492479030 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc15ef90 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.789+0000 7fd0caffd640 1 --2- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:50:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.791+0000 7fd0cb7fe640 1 --2- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc15e820 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009870 tx=0x7fd0bc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5562c24d37fb3116 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.791+0000 7fd0ca7fc640 1 -- 172.21.15.29:0/2492479030 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc030840 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.791+0000 7fd0ca7fc640 1 -- 172.21.15.29:0/2492479030 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc0309a0 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.791+0000 7fd0ca7fc640 1 -- 172.21.15.29:0/2492479030 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc030d30 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc15aaa0 msgr2=0x7fd0cc15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc15e820 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009870 tx=0x7fd0bc02b980 comp rx=0 tx=0).stop 2023-10-16T15:50:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2492479030 shutdown_connections 2023-10-16T15:50:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2492479030 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc15e820 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2492479030 >> 172.21.15.29:0/2492479030 conn(0x7fd0cc0cb410 msgr2=0x7fd0cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2492479030 shutdown_connections 2023-10-16T15:50:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.792+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2492479030 wait complete. 2023-10-16T15:50:34.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.794+0000 7fd0d1eda640 1 Processor -- start 2023-10-16T15:50:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.794+0000 7fd0d1eda640 1 -- start start 2023-10-16T15:50:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0d1eda640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc0ccaa0 0x7fd0cc14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0d1eda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc0c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0d1eda640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:34.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0d1eda640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc14bd20 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0d1eda640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc14ba50 con 0x7fd0cc0ccaa0 2023-10-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0d1eda640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc0c4af0 con 0x7fd0cc15ae80 2023-10-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc0c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc0c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40914/0 (socket says 172.21.15.29:40914) 2023-10-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.795+0000 7fd0caffd640 1 -- 172.21.15.29:0/2659178825 learned_addr learned my addr 172.21.15.29:0/2659178825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0cbfff640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0caffd640 1 -- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 msgr2=0x7fd0cc0c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0cb7fe640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc0ccaa0 0x7fd0cc14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0caffd640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0caffd640 1 -- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc0ccaa0 msgr2=0x7fd0cc14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0caffd640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc0ccaa0 0x7fd0cc14b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0caffd640 1 -- 172.21.15.29:0/2659178825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c0009400 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0cbfff640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0caffd640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc0c4030 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000ec10 tx=0x7fd0c000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0009760 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c000cc30 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0013470 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.796+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc0094d0 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.797+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc14bf30 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.798+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc06be20 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.799+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd0c00135d0 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.800+0000 7fd0b3fff640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd0b806af90 0x7fd0b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:34.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.800+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fd0c0089c30 con 0x7fd0cc15aaa0 2023-10-16T15:50:34.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.801+0000 7fd0cb7fe640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd0b806af90 0x7fd0b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:34.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.802+0000 7fd0cb7fe640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd0b806af90 0x7fd0b806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc036000 tx=0x7fd0bc03b040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:34.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:34.804+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0c0054a90 con 0x7fd0cc15aaa0 2023-10-16T15:50:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.002+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0cc15f240 con 0x7fd0cc15aaa0 2023-10-16T15:50:35.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.004+0000 7fd0b3fff640 1 -- 172.21.15.29:0/2659178825 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) v1 ==== 74+0+14608 (secure 0 0 0) 0x7fd0c0017090 con 0x7fd0cc15aaa0 2023-10-16T15:50:35.007 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:35.007 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":200,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:32.135582+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6834/2122514336":"2023-10-17T15:50:29.470732+0000","172.21.15.170:0/1503925164":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/1571191706":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6833/23414019":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/2654668130":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6832/23414019":"2023-10-17T15:50:25.441716+0000","172.21.15.170:6833/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:6832/3536897498":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1849741434":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/3590601432":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1525143382":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/2183916363":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3770772716":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/3372934420":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/759794124":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/3161548522":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/1363341747":"2023-10-17T15:45:21.100294+0000","172.21.15.170:0/1893102802":"2023-10-17T15:34:40.941061+0000","172.21.15.29:6835/2122514336":"2023-10-17T15:50:29.470732+0000","172.21.15.170:0/1719808847":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/345956683":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6833/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:6832/3694096961":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/3541452828":"2023-10-17T15:50:25.441716+0000","172.21.15.170:0/4127022397":"2023-10-17T15:34:40.941061+0000","172.21.15.170:0/479239966":"2023-10-17T15:34:40.941061+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":32}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd0b806af90 msgr2=0x7fd0b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd0b806af90 0x7fd0b806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc036000 tx=0x7fd0bc03b040 comp rx=0 tx=0).stop 2023-10-16T15:50:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 msgr2=0x7fd0cc0c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc0c4030 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000ec10 tx=0x7fd0c000c520 comp rx=0 tx=0).stop 2023-10-16T15:50:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 shutdown_connections 2023-10-16T15:50:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd0b806af90 0x7fd0b806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0cc15aaa0 0x7fd0cc0c4030 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc0c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 --2- 172.21.15.29:0/2659178825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc0ccaa0 0x7fd0cc14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 >> 172.21.15.29:0/2659178825 conn(0x7fd0cc0cb410 msgr2=0x7fd0cc0b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 shutdown_connections 2023-10-16T15:50:35.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.007+0000 7fd0d1eda640 1 -- 172.21.15.29:0/2659178825 wait complete. 2023-10-16T15:50:35.030 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:50:35.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:50:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.261+0000 7f0ca359e640 1 Processor -- start 2023-10-16T15:50:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.261+0000 7f0ca359e640 1 -- start start 2023-10-16T15:50:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b0ea0 0x7f0ca415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ca40cd970 0x7f0ca415a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca406be30 con 0x7f0ca40b3ec0 2023-10-16T15:50:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca406ca00 con 0x7f0ca40cd970 2023-10-16T15:50:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40ada10 con 0x7f0ca40b0ea0 2023-10-16T15:50:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ca40cd970 0x7f0ca415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ca40cd970 0x7f0ca415a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40916/0 (socket says 172.21.15.29:40916) 2023-10-16T15:50:35.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/1581529574 learned_addr learned my addr 172.21.15.29:0/1581529574 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.263+0000 7f0ca1d9b640 1 --2- 172.21.15.29:0/1581529574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b0ea0 0x7f0ca415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.264+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/1581529574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b0ea0 msgr2=0x7f0ca415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.264+0000 7f0ca2d9d640 1 --2- 172.21.15.29:0/1581529574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b0ea0 0x7f0ca415a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.264+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/1581529574 >> v1:172.21.15.170:6789/0 conn(0x7f0ca40b3ec0 legacy=0x7f0ca40cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.264+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/1581529574 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca406c060 con 0x7f0ca40cd970 2023-10-16T15:50:35.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.266+0000 7f0ca2d9d640 1 --2- 172.21.15.29:0/1581529574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca415a830 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009e50 tx=0x7f0c9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6dad646b720132 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.266+0000 7f0ca159a640 1 -- 172.21.15.29:0/1581529574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c02ca70 con 0x7f0ca40cd970 2023-10-16T15:50:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.266+0000 7f0ca159a640 1 -- 172.21.15.29:0/1581529574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c02cbd0 con 0x7f0ca40cd970 2023-10-16T15:50:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.266+0000 7f0ca159a640 1 -- 172.21.15.29:0/1581529574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c0343f0 con 0x7f0ca40cd970 2023-10-16T15:50:35.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 -- 172.21.15.29:0/1581529574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 msgr2=0x7f0ca415a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 --2- 172.21.15.29:0/1581529574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca415a830 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009e50 tx=0x7f0c9c02bed0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 -- 172.21.15.29:0/1581529574 shutdown_connections 2023-10-16T15:50:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 --2- 172.21.15.29:0/1581529574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca415a830 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 --2- 172.21.15.29:0/1581529574 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b0ea0 0x7f0ca415a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 -- 172.21.15.29:0/1581529574 >> 172.21.15.29:0/1581529574 conn(0x7f0ca40b0170 msgr2=0x7f0ca40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 -- 172.21.15.29:0/1581529574 shutdown_connections 2023-10-16T15:50:35.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.267+0000 7f0ca359e640 1 -- 172.21.15.29:0/1581529574 wait complete. 2023-10-16T15:50:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.269+0000 7f0ca359e640 1 Processor -- start 2023-10-16T15:50:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.269+0000 7f0ca359e640 1 -- start start 2023-10-16T15:50:35.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ca40b0ea0 0x7f0ca415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b3ec0 0x7f0ca415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca41e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40c8810 con 0x7f0ca40cd970 2023-10-16T15:50:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40c7c90 con 0x7f0ca40b0ea0 2023-10-16T15:50:35.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40c7f70 con 0x7f0ca40b3ec0 2023-10-16T15:50:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca41e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca41e4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40926/0 (socket says 172.21.15.29:40926) 2023-10-16T15:50:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/3270178962 learned_addr learned my addr 172.21.15.29:0/3270178962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca259c640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ca40b0ea0 0x7f0ca415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca1d9b640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b3ec0 0x7f0ca415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.270+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b3ec0 msgr2=0x7f0ca415bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca2d9d640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b3ec0 0x7f0ca415bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ca40b0ea0 msgr2=0x7f0ca415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca2d9d640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ca40b0ea0 0x7f0ca415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca2d9d640 1 -- 172.21.15.29:0/3270178962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c009b30 con 0x7f0ca40cd970 2023-10-16T15:50:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca1d9b640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b3ec0 0x7f0ca415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:35.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca2d9d640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca41e4f30 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009e50 tx=0x7f0c9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:35.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c03e450 con 0x7f0ca40cd970 2023-10-16T15:50:35.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c03e5b0 con 0x7f0ca40cd970 2023-10-16T15:50:35.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c03e960 con 0x7f0ca40cd970 2023-10-16T15:50:35.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca40c8250 con 0x7f0ca40cd970 2023-10-16T15:50:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.271+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca415c1f0 con 0x7f0ca40cd970 2023-10-16T15:50:35.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.273+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca406a870 con 0x7f0ca40cd970 2023-10-16T15:50:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.273+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c9c03eac0 con 0x7f0ca40cd970 2023-10-16T15:50:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.274+0000 7f0c92ffd640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0c8406afe0 0x7f0c8406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.274+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f0c9c0ac4c0 con 0x7f0ca40cd970 2023-10-16T15:50:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.275+0000 7f0ca259c640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0c8406afe0 0x7f0c8406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.276+0000 7f0ca259c640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0c8406afe0 0x7f0c8406d4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c94000f60 tx=0x7f0c94009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:35.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.279+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c9c077490 con 0x7f0ca40cd970 2023-10-16T15:50:35.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.471+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ca40c57e0 con 0x7f0ca40cd970 2023-10-16T15:50:35.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.472+0000 7f0c92ffd640 1 -- 172.21.15.29:0/3270178962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v200) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c9c07aab0 con 0x7f0ca40cd970 2023-10-16T15:50:35.474 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:50:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0c8406afe0 msgr2=0x7f0c8406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0c8406afe0 0x7f0c8406d4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c94000f60 tx=0x7f0c94009040 comp rx=0 tx=0).stop 2023-10-16T15:50:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 msgr2=0x7f0ca41e4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca41e4f30 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009e50 tx=0x7f0c9c004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 shutdown_connections 2023-10-16T15:50:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f0c8406afe0 0x7f0c8406d4a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ca40cd970 0x7f0ca41e4f30 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ca40b3ec0 0x7f0ca415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 --2- 172.21.15.29:0/3270178962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ca40b0ea0 0x7f0ca415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 >> 172.21.15.29:0/3270178962 conn(0x7f0ca40b0170 msgr2=0x7f0ca40af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 shutdown_connections 2023-10-16T15:50:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.475+0000 7f0ca359e640 1 -- 172.21.15.29:0/3270178962 wait complete. 2023-10-16T15:50:35.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T15:50:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.698+0000 7fa2ae34e640 1 Processor -- start 2023-10-16T15:50:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.698+0000 7fa2ae34e640 1 -- start start 2023-10-16T15:50:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa2ae34e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2a81615f0 0x7fa2a815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa2ae34e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a815e3e0 con 0x7fa2a80b0b20 2023-10-16T15:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa2ae34e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a80cd4a0 con 0x7fa2a80cd0c0 2023-10-16T15:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa2ae34e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a815e100 con 0x7fa2a81615f0 2023-10-16T15:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa29ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa2a80b0b20 0x7fa2a815d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51408/0 (socket says 172.21.15.29:51408) 2023-10-16T15:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa29ffff640 1 -- 172.21.15.29:0/2811005935 learned_addr learned my addr 172.21.15.29:0/2811005935 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.700+0000 7fa2ac8eb640 1 --2- 172.21.15.29:0/2811005935 >> v2:172.21.15.29:3300/0 conn(0x7fa2a81615f0 0x7fa2a815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.701+0000 7fa2a77fe640 1 -- 172.21.15.29:0/2811005935 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525516320 0 0) 0x7fa2a815e3e0 con 0x7fa2a80b0b20 2023-10-16T15:50:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.701+0000 7fa2a77fe640 1 -- 172.21.15.29:0/2811005935 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa294003520 con 0x7fa2a80b0b20 2023-10-16T15:50:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.701+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/2811005935 >> v1:172.21.15.170:6789/0 conn(0x7fa2a80cd0c0 legacy=0x7fa2a80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.701+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/2811005935 >> v1:172.21.15.29:6789/0 conn(0x7fa2a80b0b20 legacy=0x7fa2a815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.701+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/2811005935 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a8160b10 con 0x7fa2a81615f0 2023-10-16T15:50:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.702+0000 7fa2ac8eb640 1 --2- 172.21.15.29:0/2811005935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a815db00 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009d90 tx=0x7fa2a002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c10378ac92a97e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:35.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2a77fe640 1 -- 172.21.15.29:0/2811005935 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a002cc60 con 0x7fa2a81615f0 2023-10-16T15:50:35.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2a77fe640 1 -- 172.21.15.29:0/2811005935 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a002cdc0 con 0x7fa2a81615f0 2023-10-16T15:50:35.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2a77fe640 1 -- 172.21.15.29:0/2811005935 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0034640 con 0x7fa2a81615f0 2023-10-16T15:50:35.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 -- 172.21.15.29:0/2811005935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 msgr2=0x7fa2a815db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/2811005935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a815db00 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009d90 tx=0x7fa2a002beb0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 -- 172.21.15.29:0/2811005935 shutdown_connections 2023-10-16T15:50:35.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/2811005935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a815db00 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 -- 172.21.15.29:0/2811005935 >> 172.21.15.29:0/2811005935 conn(0x7fa2a80cb410 msgr2=0x7fa2a80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:35.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 -- 172.21.15.29:0/2811005935 shutdown_connections 2023-10-16T15:50:35.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.703+0000 7fa2ae34e640 1 -- 172.21.15.29:0/2811005935 wait complete. 2023-10-16T15:50:35.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.705+0000 7fa2ae34e640 1 Processor -- start 2023-10-16T15:50:35.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.705+0000 7fa2ae34e640 1 -- start start 2023-10-16T15:50:35.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.706+0000 7fa2ae34e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2a80b0b20 0x7fa2a8160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.706+0000 7fa2ae34e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2a80cd0c0 0x7fa2a806cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.706+0000 7fa2ae34e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a806c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.706+0000 7fa2ae34e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a81501c0 con 0x7fa2a81615f0 2023-10-16T15:50:35.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.706+0000 7fa2ae34e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a815f5a0 con 0x7fa2a80cd0c0 2023-10-16T15:50:35.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.706+0000 7fa2ae34e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a815f880 con 0x7fa2a80b0b20 2023-10-16T15:50:35.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a806c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40938/0 (socket says 172.21.15.29:40938) 2023-10-16T15:50:35.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/1351521237 learned_addr learned my addr 172.21.15.29:0/1351521237 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:35.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2a7fff640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2a80b0b20 0x7fa2a8160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa29ffff640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2a80cd0c0 0x7fa2a806cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2a80b0b20 msgr2=0x7fa2a8160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2a80b0b20 0x7fa2a8160080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2a80cd0c0 msgr2=0x7fa2a806cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:35.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2a80cd0c0 0x7fa2a806cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:35.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 -- 172.21.15.29:0/1351521237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a0009a70 con 0x7fa2a81615f0 2023-10-16T15:50:35.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.707+0000 7fa2ac8eb640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a806c640 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009d90 tx=0x7fa2a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:35.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.708+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0003b90 con 0x7fa2a81615f0 2023-10-16T15:50:35.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.708+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a0003d10 con 0x7fa2a81615f0 2023-10-16T15:50:35.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.708+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0004270 con 0x7fa2a81615f0 2023-10-16T15:50:35.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.708+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a815fb60 con 0x7fa2a81615f0 2023-10-16T15:50:35.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.708+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a806cb80 con 0x7fa2a81615f0 2023-10-16T15:50:35.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.709+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a806cd70 con 0x7fa2a81615f0 2023-10-16T15:50:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.710+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa2a003b050 con 0x7fa2a81615f0 2023-10-16T15:50:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.711+0000 7fa2a57fa640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa28806afe0 0x7fa28806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.711+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa2a00abdd0 con 0x7fa2a81615f0 2023-10-16T15:50:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.712+0000 7fa2a7fff640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa28806afe0 0x7fa28806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.713+0000 7fa2a7fff640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa28806afe0 0x7fa28806d4a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa298000f60 tx=0x7fa298009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.715+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2a0076c10 con 0x7fa2a81615f0 2023-10-16T15:50:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:35.980+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fa2a815f3f0 con 0x7fa2a81615f0 2023-10-16T15:50:36.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.539+0000 7fa2a57fa640 1 -- 172.21.15.29:0/1351521237 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3062) v1 ==== 166+0+0 (secure 0 0 0) 0x7fa2a00390e0 con 0x7fa2a81615f0 2023-10-16T15:50:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.542+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa28806afe0 msgr2=0x7fa28806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:36.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.542+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa28806afe0 0x7fa28806d4a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa298000f60 tx=0x7fa298009040 comp rx=0 tx=0).stop 2023-10-16T15:50:36.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.542+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 msgr2=0x7fa2a806c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:36.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.542+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a806c640 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009d90 tx=0x7fa2a0004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:36.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 shutdown_connections 2023-10-16T15:50:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fa28806afe0 0x7fa28806d4a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a81615f0 0x7fa2a806c640 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2a80cd0c0 0x7fa2a806cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 --2- 172.21.15.29:0/1351521237 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa2a80b0b20 0x7fa2a8160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 >> 172.21.15.29:0/1351521237 conn(0x7fa2a80cb410 msgr2=0x7fa2a80c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 shutdown_connections 2023-10-16T15:50:36.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.543+0000 7fa2ae34e640 1 -- 172.21.15.29:0/1351521237 wait complete. 2023-10-16T15:50:36.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T15:50:36.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.797+0000 7f3d0c831640 1 Processor -- start 2023-10-16T15:50:36.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.797+0000 7f3d0c831640 1 -- start start 2023-10-16T15:50:36.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.798+0000 7f3d0c831640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d080c9060 0x7f3d0806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.798+0000 7f3d0c831640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d0806c9e0 0x7f3d0806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.798+0000 7f3d0c831640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3d080c9440 0x7f3d08164d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0c831640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d080ca560 con 0x7f3d0806c9e0 2023-10-16T15:50:36.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0c831640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d080c9950 con 0x7f3d080c9440 2023-10-16T15:50:36.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0c831640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c9c30 con 0x7f3d080c9060 2023-10-16T15:50:36.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d0806c9e0 0x7f3d0806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d0806c9e0 0x7f3d0806f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40952/0 (socket says 172.21.15.29:40952) 2023-10-16T15:50:36.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 -- 172.21.15.29:0/2495947210 learned_addr learned my addr 172.21.15.29:0/2495947210 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:36.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d06d9d640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d080c9060 0x7f3d0806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0759e640 1 --2- 172.21.15.29:0/2495947210 >> v2:172.21.15.170:3300/0 conn(0x7f3d080c9440 0x7f3d08164d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 -- 172.21.15.29:0/2495947210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d080c9060 msgr2=0x7f3d0806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d080c9060 0x7f3d0806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 -- 172.21.15.29:0/2495947210 >> v2:172.21.15.170:3300/0 conn(0x7f3d080c9440 msgr2=0x7f3d08164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 --2- 172.21.15.29:0/2495947210 >> v2:172.21.15.170:3300/0 conn(0x7f3d080c9440 0x7f3d08164d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.799+0000 7f3d0659c640 1 -- 172.21.15.29:0/2495947210 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080c9f10 con 0x7f3d0806c9e0 2023-10-16T15:50:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.800+0000 7f3d06d9d640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d080c9060 0x7f3d0806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:36.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.800+0000 7f3d0759e640 1 --2- 172.21.15.29:0/2495947210 >> v2:172.21.15.170:3300/0 conn(0x7f3d080c9440 0x7f3d08164d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:36.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.801+0000 7f3d0659c640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0806c9e0 0x7f3d0806f980 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f3d080ca4d0 tx=0x7f3cfc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a450f5bced2933b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:36.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.801+0000 7f3d05d9b640 1 -- 172.21.15.29:0/2495947210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc031860 con 0x7f3d0806c9e0 2023-10-16T15:50:36.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.801+0000 7f3d05d9b640 1 -- 172.21.15.29:0/2495947210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc0319c0 con 0x7f3d0806c9e0 2023-10-16T15:50:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.801+0000 7f3d05d9b640 1 -- 172.21.15.29:0/2495947210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc031d50 con 0x7f3d0806c9e0 2023-10-16T15:50:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 -- 172.21.15.29:0/2495947210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0806c9e0 msgr2=0x7f3d0806f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0806c9e0 0x7f3d0806f980 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f3d080ca4d0 tx=0x7f3cfc02b900 comp rx=0 tx=0).stop 2023-10-16T15:50:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 -- 172.21.15.29:0/2495947210 shutdown_connections 2023-10-16T15:50:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 --2- 172.21.15.29:0/2495947210 >> v2:172.21.15.170:3300/0 conn(0x7f3d080c9440 0x7f3d08164d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0806c9e0 0x7f3d0806f980 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 --2- 172.21.15.29:0/2495947210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d080c9060 0x7f3d0806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 -- 172.21.15.29:0/2495947210 >> 172.21.15.29:0/2495947210 conn(0x7f3d080ce6a0 msgr2=0x7f3d080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 -- 172.21.15.29:0/2495947210 shutdown_connections 2023-10-16T15:50:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.802+0000 7f3d0c831640 1 -- 172.21.15.29:0/2495947210 wait complete. 2023-10-16T15:50:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.804+0000 7f3d0c831640 1 Processor -- start 2023-10-16T15:50:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.804+0000 7f3d0c831640 1 -- start start 2023-10-16T15:50:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0c831640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d0806c9e0 0x7f3d081fa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0c831640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 0x7f3d080c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0c831640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d080c9440 0x7f3d080c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0c831640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c6160 con 0x7f3d080c9060 2023-10-16T15:50:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0c831640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c86d0 con 0x7f3d080c9440 2023-10-16T15:50:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0c831640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d081628f0 con 0x7f3d0806c9e0 2023-10-16T15:50:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 0x7f3d080c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 0x7f3d080c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40968/0 (socket says 172.21.15.29:40968) 2023-10-16T15:50:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.805+0000 7f3d0659c640 1 -- 172.21.15.29:0/3231118952 learned_addr learned my addr 172.21.15.29:0/3231118952 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d06d9d640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d0806c9e0 0x7f3d081fa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0759e640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d080c9440 0x7f3d080c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0659c640 1 -- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d0806c9e0 msgr2=0x7f3d081fa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0659c640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d0806c9e0 0x7f3d081fa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0659c640 1 -- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d080c9440 msgr2=0x7f3d080c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0659c640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d080c9440 0x7f3d080c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0659c640 1 -- 172.21.15.29:0/3231118952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0094d0 con 0x7f3d080c9060 2023-10-16T15:50:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0759e640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d080c9440 0x7f3d080c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3d0659c640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 0x7f3d080c7c50 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc000c00 tx=0x7f3cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.806+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc031850 con 0x7f3d080c9060 2023-10-16T15:50:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.807+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc044400 con 0x7f3d080c9060 2023-10-16T15:50:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.807+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc0447b0 con 0x7f3d080c9060 2023-10-16T15:50:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.807+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d080c63b0 con 0x7f3d080c9060 2023-10-16T15:50:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.807+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d08162f90 con 0x7f3d080c9060 2023-10-16T15:50:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.808+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d08162b00 con 0x7f3d080c9060 2023-10-16T15:50:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.809+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3cfc04d440 con 0x7f3d080c9060 2023-10-16T15:50:36.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.810+0000 7f3cef7fe640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f3cf406af90 0x7f3cf406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:36.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.810+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3cfc0ac1a0 con 0x7f3d080c9060 2023-10-16T15:50:36.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.811+0000 7f3d06d9d640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f3cf406af90 0x7f3cf406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:36.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.812+0000 7f3d06d9d640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f3cf406af90 0x7f3cf406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d080b3e70 tx=0x7f3cf8008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:36.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:36.815+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3cfc076ee0 con 0x7f3d080c9060 2023-10-16T15:50:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.009+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3d0804e9b0 con 0x7f3d080c9060 2023-10-16T15:50:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.015+0000 7f3cef7fe640 1 -- 172.21.15.29:0/3231118952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3cfc07a500 con 0x7f3d080c9060 2023-10-16T15:50:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f3cf406af90 msgr2=0x7f3cf406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:37.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f3cf406af90 0x7f3cf406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d080b3e70 tx=0x7f3cf8008040 comp rx=0 tx=0).stop 2023-10-16T15:50:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 msgr2=0x7f3d080c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 0x7f3d080c7c50 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc000c00 tx=0x7f3cfc0046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 shutdown_connections 2023-10-16T15:50:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f3cf406af90 0x7f3cf406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d080c9060 0x7f3d080c7c50 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3d080c9440 0x7f3d080c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 --2- 172.21.15.29:0/3231118952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3d0806c9e0 0x7f3d081fa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 >> 172.21.15.29:0/3231118952 conn(0x7f3d080ce6a0 msgr2=0x7f3d080cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 shutdown_connections 2023-10-16T15:50:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.018+0000 7f3d0c831640 1 -- 172.21.15.29:0/3231118952 wait complete. 2023-10-16T15:50:37.040 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:50:37.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:50:37.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:50:37.312 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:50:37.313 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:50:37.313 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:37.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:50:37.377 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:50:37.377 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:37.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:50:37.437 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:50:37.437 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:37.501 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:50:37.501 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:50:37.501 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:37.562 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:50:37.562 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T15:50:37.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:37.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.799+0000 7ff0f3247640 1 Processor -- start 2023-10-16T15:50:37.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.799+0000 7ff0f3247640 1 -- start start 2023-10-16T15:50:37.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.800+0000 7ff0f3247640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec06c040 0x7ff0ec06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:37.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.800+0000 7ff0f3247640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0ec0cda80 0x7ff0ec15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:37.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.800+0000 7ff0f3247640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec06c960 con 0x7ff0ec0ad970 2023-10-16T15:50:37.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.800+0000 7ff0f3247640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ec15b9d0 con 0x7ff0ec0cda80 2023-10-16T15:50:37.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.800+0000 7ff0f3247640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec15bcb0 con 0x7ff0ec06c040 2023-10-16T15:50:37.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f1a44640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff0ec0ad970 0x7ff0ec15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51422/0 (socket says 172.21.15.29:51422) 2023-10-16T15:50:37.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f1a44640 1 -- 172.21.15.29:0/1377451721 learned_addr learned my addr 172.21.15.29:0/1377451721 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:37.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f2a46640 1 --2- 172.21.15.29:0/1377451721 >> v2:172.21.15.29:3300/0 conn(0x7ff0ec0cda80 0x7ff0ec15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:37.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f2245640 1 --2- 172.21.15.29:0/1377451721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec06c040 0x7ff0ec06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:37.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f1243640 1 -- 172.21.15.29:0/1377451721 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573380186 0 0) 0x7ff0ec06c960 con 0x7ff0ec0ad970 2023-10-16T15:50:37.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f2a46640 1 -- 172.21.15.29:0/1377451721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec06c040 msgr2=0x7ff0ec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:37.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f2a46640 1 --2- 172.21.15.29:0/1377451721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec06c040 0x7ff0ec06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f2a46640 1 -- 172.21.15.29:0/1377451721 >> v1:172.21.15.29:6789/0 conn(0x7ff0ec0ad970 legacy=0x7ff0ec15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:37.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.801+0000 7ff0f2a46640 1 -- 172.21.15.29:0/1377451721 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec15bf90 con 0x7ff0ec0cda80 2023-10-16T15:50:37.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.802+0000 7ff0f2a46640 1 --2- 172.21.15.29:0/1377451721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec0cda80 0x7ff0ec15ed90 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009d90 tx=0x7ff0e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c58e6a6cfbad210 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:37.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f1243640 1 -- 172.21.15.29:0/1377451721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e402cc60 con 0x7ff0ec0cda80 2023-10-16T15:50:37.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f1243640 1 -- 172.21.15.29:0/1377451721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e402cdc0 con 0x7ff0ec0cda80 2023-10-16T15:50:37.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f1243640 1 -- 172.21.15.29:0/1377451721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4034660 con 0x7ff0ec0cda80 2023-10-16T15:50:37.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f3247640 1 -- 172.21.15.29:0/1377451721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec0cda80 msgr2=0x7ff0ec15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:37.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f3247640 1 --2- 172.21.15.29:0/1377451721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec0cda80 0x7ff0ec15ed90 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009d90 tx=0x7ff0e402beb0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f3247640 1 -- 172.21.15.29:0/1377451721 shutdown_connections 2023-10-16T15:50:37.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f3247640 1 --2- 172.21.15.29:0/1377451721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec0cda80 0x7ff0ec15ed90 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f3247640 1 --2- 172.21.15.29:0/1377451721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec06c040 0x7ff0ec06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.803+0000 7ff0f3247640 1 -- 172.21.15.29:0/1377451721 >> 172.21.15.29:0/1377451721 conn(0x7ff0ec0ce6a0 msgr2=0x7ff0ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:37.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.804+0000 7ff0f3247640 1 -- 172.21.15.29:0/1377451721 shutdown_connections 2023-10-16T15:50:37.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.804+0000 7ff0f3247640 1 -- 172.21.15.29:0/1377451721 wait complete. 2023-10-16T15:50:37.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.805+0000 7ff0f3247640 1 Processor -- start 2023-10-16T15:50:37.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.806+0000 7ff0f3247640 1 -- start start 2023-10-16T15:50:37.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f3247640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 0x7ff0ec14ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:37.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f3247640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0ec0ad970 0x7ff0ec0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:37.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f3247640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec0cda80 0x7ff0ec0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:37.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f3247640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec158690 con 0x7ff0ec06c040 2023-10-16T15:50:37.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f3247640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec0c8eb0 con 0x7ff0ec0ad970 2023-10-16T15:50:37.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f3247640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec0c9190 con 0x7ff0ec0cda80 2023-10-16T15:50:37.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 0x7ff0ec14ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:37.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 0x7ff0ec14ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40994/0 (socket says 172.21.15.29:40994) 2023-10-16T15:50:37.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 -- 172.21.15.29:0/2025286194 learned_addr learned my addr 172.21.15.29:0/2025286194 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:37.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 -- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec0cda80 msgr2=0x7ff0ec0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:37.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec0cda80 0x7ff0ec0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 -- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0ec0ad970 msgr2=0x7ff0ec0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:37.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0ec0ad970 0x7ff0ec0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:37.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.807+0000 7ff0f2245640 1 -- 172.21.15.29:0/2025286194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4009a70 con 0x7ff0ec06c040 2023-10-16T15:50:37.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.808+0000 7ff0f2245640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 0x7ff0ec14ff00 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800d800 tx=0x7ff0e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:37.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.808+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800b6c0 con 0x7ff0ec06c040 2023-10-16T15:50:37.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.808+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e800b820 con 0x7ff0ec06c040 2023-10-16T15:50:37.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.808+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800bb90 con 0x7ff0ec06c040 2023-10-16T15:50:37.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.808+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ec0c9470 con 0x7ff0ec06c040 2023-10-16T15:50:37.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.808+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec1579d0 con 0x7ff0ec06c040 2023-10-16T15:50:37.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.809+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec06b330 con 0x7ff0ec06c040 2023-10-16T15:50:37.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.811+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff0e800bcf0 con 0x7ff0ec06c040 2023-10-16T15:50:37.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.812+0000 7ff0e2ffd640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff0d006af90 0x7ff0d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:37.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.812+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff0e8089e60 con 0x7ff0ec06c040 2023-10-16T15:50:37.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.812+0000 7ff0f1a44640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff0d006af90 0x7ff0d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:37.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.813+0000 7ff0f1a44640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff0d006af90 0x7ff0d006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc007fd0 tx=0x7ff0dc007920 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:37.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:37.816+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0e8054cc0 con 0x7ff0ec06c040 2023-10-16T15:50:38.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.046+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0ec04e9b0 con 0x7ff0ec06c040 2023-10-16T15:50:38.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.047+0000 7ff0e2ffd640 1 -- 172.21.15.29:0/2025286194 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+3703 (secure 0 0 0) 0x7ff0e80582e0 con 0x7ff0ec06c040 2023-10-16T15:50:38.049 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:38.049 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":94,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/258565231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":258565231},{"type":"v1","addr":"172.21.15.170:6836","nonce":258565231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3678430365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3678430365},{"type":"v1","addr":"172.21.15.170:6837","nonce":3678430365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88},{"gid":7978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2371983587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2371983587},{"type":"v1","addr":"172.21.15.29:6833","nonce":2371983587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2662587997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2662587997},{"type":"v1","addr":"172.21.15.29:6835","nonce":2662587997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[]} 2023-10-16T15:50:38.050 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 94 2023-10-16T15:50:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.049+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff0d006af90 msgr2=0x7ff0d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.049+0000 7ff0f3247640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff0d006af90 0x7ff0d006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc007fd0 tx=0x7ff0dc007920 comp rx=0 tx=0).stop 2023-10-16T15:50:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.049+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 msgr2=0x7ff0ec14ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.049+0000 7ff0f3247640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 0x7ff0ec14ff00 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800d800 tx=0x7ff0e800dcc0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 shutdown_connections 2023-10-16T15:50:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff0d006af90 0x7ff0d006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0ec0cda80 0x7ff0ec0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0ec0ad970 0x7ff0ec0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 --2- 172.21.15.29:0/2025286194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0ec06c040 0x7ff0ec14ff00 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 >> 172.21.15.29:0/2025286194 conn(0x7ff0ec0ce6a0 msgr2=0x7ff0ec15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 shutdown_connections 2023-10-16T15:50:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.050+0000 7ff0f3247640 1 -- 172.21.15.29:0/2025286194 wait complete. 2023-10-16T15:50:38.072 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T15:50:38.072 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T15:50:38.073 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:50:38.073 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:50:38.073 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T15:50:38.074 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T15:50:38.074 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:50:38.074 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:50:38.074 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T15:50:38.075 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T15:50:38.075 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:50:38.075 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:50:38.076 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T15:50:38.076 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T15:50:38.076 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:50:38.076 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:50:38.094 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:50:38.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T15:50:38.098 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:50:38.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T15:50:38.099 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:50:38.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T15:50:38.103 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:50:38.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T15:50:38.148 INFO:tasks.ceph.mds.b:Started 2023-10-16T15:50:38.148 DEBUG:teuthology.parallel:result is None 2023-10-16T15:50:38.152 INFO:tasks.ceph.mds.a:Started 2023-10-16T15:50:38.152 DEBUG:teuthology.parallel:result is None 2023-10-16T15:50:38.155 INFO:tasks.ceph.mds.d:Started 2023-10-16T15:50:38.156 DEBUG:teuthology.parallel:result is None 2023-10-16T15:50:38.160 INFO:tasks.ceph.mds.c:Started 2023-10-16T15:50:38.160 DEBUG:teuthology.parallel:result is None 2023-10-16T15:50:38.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T15:50:38.317 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:50:38.319 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T15:50:38.322 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:50:38.324 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:50:38.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.358+0000 7f3165bd0640 1 Processor -- start 2023-10-16T15:50:38.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.358+0000 7f3165bd0640 1 -- start start 2023-10-16T15:50:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.359+0000 7f3165bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f316015ad70 0x7f3160069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.359+0000 7f3165bd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31600c9ca0 0x7f316006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:38.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.359+0000 7f3165bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600cb030 con 0x7f31600ca3b0 2023-10-16T15:50:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.359+0000 7f3165bd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31600c91c0 con 0x7f31600c9ca0 2023-10-16T15:50:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.359+0000 7f3165bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31600c94a0 con 0x7f316015ad70 2023-10-16T15:50:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f316015ad70 0x7f3160069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315effd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f31600c9ca0 0x7f316006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315f7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f316015ad70 0x7f3160069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:38566/0 (socket says 172.21.15.29:38566) 2023-10-16T15:50:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315f7fe640 1 -- 172.21.15.29:0/988899108 learned_addr learned my addr 172.21.15.29:0/988899108 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f31600ca3b0 0x7f31600ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:56446/0 (socket says 172.21.15.29:56446) 2023-10-16T15:50:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315effd640 1 -- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f316015ad70 msgr2=0x7f3160069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315effd640 1 --2- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f316015ad70 0x7f3160069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315effd640 1 -- 172.21.15.29:0/988899108 >> v1:172.21.15.170:6789/0 conn(0x7f31600ca3b0 legacy=0x7f31600ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315effd640 1 -- 172.21.15.29:0/988899108 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600c9780 con 0x7f31600c9ca0 2023-10-16T15:50:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.360+0000 7f315e7fc640 1 -- 172.21.15.29:0/988899108 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836014995 0 0) 0x7f31600cb030 con 0x7f31600ca3b0 2023-10-16T15:50:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.361+0000 7f315effd640 1 --2- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f316006a2e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f31540097a0 tx=0x7f315402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc422f97897fbab6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f315e7fc640 1 -- 172.21.15.29:0/988899108 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315402d040 con 0x7f31600c9ca0 2023-10-16T15:50:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f315e7fc640 1 -- 172.21.15.29:0/988899108 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3154031810 con 0x7f31600c9ca0 2023-10-16T15:50:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f315e7fc640 1 -- 172.21.15.29:0/988899108 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154031ba0 con 0x7f31600c9ca0 2023-10-16T15:50:38.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 -- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 msgr2=0x7f316006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 --2- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f316006a2e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f31540097a0 tx=0x7f315402b8b0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 -- 172.21.15.29:0/988899108 shutdown_connections 2023-10-16T15:50:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 --2- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f316006a2e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 --2- 172.21.15.29:0/988899108 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f316015ad70 0x7f3160069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 -- 172.21.15.29:0/988899108 >> 172.21.15.29:0/988899108 conn(0x7f31600ce9b0 msgr2=0x7f31600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.362+0000 7f3165bd0640 1 -- 172.21.15.29:0/988899108 shutdown_connections 2023-10-16T15:50:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.363+0000 7f3165bd0640 1 -- 172.21.15.29:0/988899108 wait complete. 2023-10-16T15:50:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.364+0000 7f3165bd0640 1 Processor -- start 2023-10-16T15:50:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.364+0000 7f3165bd0640 1 -- start start 2023-10-16T15:50:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.365+0000 7f3165bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f31600c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:38.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.365+0000 7f3165bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31600ca3b0 0x7f31601f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.365+0000 7f3165bd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 0x7f31600cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.365+0000 7f3165bd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31600c5fc0 con 0x7f316015ad70 2023-10-16T15:50:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.365+0000 7f3165bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31601f6af0 con 0x7f31600c9ca0 2023-10-16T15:50:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.365+0000 7f3165bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31600c7490 con 0x7f31600ca3b0 2023-10-16T15:50:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 0x7f31600cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:38.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315f7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f31600c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315effd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31600ca3b0 0x7f31601f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 0x7f31600cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41032/0 (socket says 172.21.15.29:41032) 2023-10-16T15:50:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315ffff640 1 -- 172.21.15.29:0/3403952769 learned_addr learned my addr 172.21.15.29:0/3403952769 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315f7fe640 1 -- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31600ca3b0 msgr2=0x7f31601f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:38.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315f7fe640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31600ca3b0 0x7f31601f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315f7fe640 1 -- 172.21.15.29:0/3403952769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 msgr2=0x7f31600cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315f7fe640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 0x7f31600cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315f7fe640 1 -- 172.21.15.29:0/3403952769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154009400 con 0x7f31600c9ca0 2023-10-16T15:50:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.366+0000 7f315ffff640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 0x7f31600cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:50:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.367+0000 7f315f7fe640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f31600c5a80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f315000e870 tx=0x7f315000ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.367+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000c730 con 0x7f31600c9ca0 2023-10-16T15:50:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.367+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315000c890 con 0x7f31600c9ca0 2023-10-16T15:50:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.367+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000cc20 con 0x7f31600c9ca0 2023-10-16T15:50:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.367+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31601f2aa0 con 0x7f31600c9ca0 2023-10-16T15:50:38.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.368+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31601f3060 con 0x7f31600c9ca0 2023-10-16T15:50:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.369+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316006cdd0 con 0x7f31600c9ca0 2023-10-16T15:50:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.370+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3150013470 con 0x7f31600c9ca0 2023-10-16T15:50:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.371+0000 7f3147fff640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f314c06af90 0x7f314c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.371+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f31500897b0 con 0x7f31600c9ca0 2023-10-16T15:50:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.371+0000 7f315effd640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f314c06af90 0x7f314c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.372+0000 7f315effd640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f314c06af90 0x7f314c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3154036000 tx=0x7f315403c040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.377+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f31500546b0 con 0x7f31600c9ca0 2023-10-16T15:50:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:38.589+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f316004e160 con 0x7f31600c9ca0 2023-10-16T15:50:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.583+0000 7f3147fff640 1 -- 172.21.15.29:0/3403952769 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v205) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3150057cd0 con 0x7f31600c9ca0 2023-10-16T15:50:39.586 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T15:50:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f314c06af90 msgr2=0x7f314c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f314c06af90 0x7f314c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3154036000 tx=0x7f315403c040 comp rx=0 tx=0).stop 2023-10-16T15:50:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 msgr2=0x7f31600c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f31600c5a80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f315000e870 tx=0x7f315000ec70 comp rx=0 tx=0).stop 2023-10-16T15:50:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 shutdown_connections 2023-10-16T15:50:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f314c06af90 0x7f314c06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f316015ad70 0x7f31600cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31600ca3b0 0x7f31601f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 --2- 172.21.15.29:0/3403952769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31600c9ca0 0x7f31600c5a80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 >> 172.21.15.29:0/3403952769 conn(0x7f31600ce9b0 msgr2=0x7f31600ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 shutdown_connections 2023-10-16T15:50:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.588+0000 7f3165bd0640 1 -- 172.21.15.29:0/3403952769 wait complete. 2023-10-16T15:50:39.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T15:50:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.820+0000 7fc547439640 1 Processor -- start 2023-10-16T15:50:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.821+0000 7fc547439640 1 -- start start 2023-10-16T15:50:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.822+0000 7fc547439640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc54006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.822+0000 7fc547439640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54006f7f0 con 0x7fc5400b0b20 2023-10-16T15:50:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.822+0000 7fc547439640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54006c5e0 con 0x7fc540159f00 2023-10-16T15:50:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.822+0000 7fc547439640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54006c8c0 con 0x7fc540159b20 2023-10-16T15:50:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc5459d6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5400b0b20 0x7fc54006fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51436/0 (socket says 172.21.15.29:51436) 2023-10-16T15:50:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc5459d6640 1 -- 172.21.15.29:0/179575156 learned_addr learned my addr 172.21.15.29:0/179575156 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc5451d5640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc54006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780629849 0 0) 0x7fc54006f7f0 con 0x7fc5400b0b20 2023-10-16T15:50:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c003520 con 0x7fc5400b0b20 2023-10-16T15:50:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752879652 0 0) 0x7fc52c003520 con 0x7fc5400b0b20 2023-10-16T15:50:39.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 msgr2=0x7fc54006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc537fff640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc54006bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.823+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 >> v1:172.21.15.170:6789/0 conn(0x7fc540159f00 legacy=0x7fc5400b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54006cba0 con 0x7fc5400b0b20 2023-10-16T15:50:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc538001670 con 0x7fc5400b0b20 2023-10-16T15:50:39.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 >> v1:172.21.15.29:6789/0 conn(0x7fc5400b0b20 legacy=0x7fc54006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc52c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc52c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc52c004d00 0x7fc52c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54006c8c0 con 0x7fc52c0039a0 2023-10-16T15:50:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c00dc70 con 0x7fc52c004d00 2023-10-16T15:50:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c00de30 con 0x7fc540159b20 2023-10-16T15:50:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5451d5640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc52c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5449d4640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc52c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5451d5640 1 -- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 msgr2=0x7fc52c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:39.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5451d5640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc52c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5451d5640 1 -- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc52c004d00 msgr2=0x7fc52c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:39.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5459d6640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc52c004d00 0x7fc52c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5451d5640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc52c004d00 0x7fc52c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5451d5640 1 -- 172.21.15.29:0/179575156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c003520 con 0x7fc52c0039a0 2023-10-16T15:50:39.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.824+0000 7fc5449d4640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc540159b20 0x7fc52c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:39.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc5451d5640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc52c003dc0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009f90 tx=0x7fc52c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ad44eed65cf88c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:39.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c02cbd0 con 0x7fc52c0039a0 2023-10-16T15:50:39.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c00f040 con 0x7fc52c0039a0 2023-10-16T15:50:39.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc537fff640 1 -- 172.21.15.29:0/179575156 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c034440 con 0x7fc52c0039a0 2023-10-16T15:50:39.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc547439640 1 -- 172.21.15.29:0/179575156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 msgr2=0x7fc52c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc547439640 1 --2- 172.21.15.29:0/179575156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc52c003dc0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009f90 tx=0x7fc52c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.826+0000 7fc5459d6640 1 -- 172.21.15.29:0/179575156 reap_dead start 2023-10-16T15:50:39.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.827+0000 7fc547439640 1 -- 172.21.15.29:0/179575156 shutdown_connections 2023-10-16T15:50:39.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.827+0000 7fc547439640 1 -- 172.21.15.29:0/179575156 >> 172.21.15.29:0/179575156 conn(0x7fc5400cb410 msgr2=0x7fc5400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:39.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.827+0000 7fc547439640 1 -- 172.21.15.29:0/179575156 shutdown_connections 2023-10-16T15:50:39.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.827+0000 7fc547439640 1 -- 172.21.15.29:0/179575156 wait complete. 2023-10-16T15:50:39.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.828+0000 7fc547439640 1 Processor -- start 2023-10-16T15:50:39.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.829+0000 7fc547439640 1 -- start start 2023-10-16T15:50:39.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.829+0000 7fc547439640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc5401f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.829+0000 7fc547439640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5401578f0 0x7fc5400b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.829+0000 7fc547439640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc540155d90 0x7fc5401561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc547439640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc540157660 con 0x7fc52c0039a0 2023-10-16T15:50:39.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc547439640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5401f2940 con 0x7fc540155d90 2023-10-16T15:50:39.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc547439640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc540157d50 con 0x7fc5401578f0 2023-10-16T15:50:39.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc5401f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc5401f5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41048/0 (socket says 172.21.15.29:41048) 2023-10-16T15:50:39.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 -- 172.21.15.29:0/3977135503 learned_addr learned my addr 172.21.15.29:0/3977135503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:39.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 -- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5401578f0 msgr2=0x7fc5400b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5401578f0 0x7fc5400b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5459d6640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc540155d90 0x7fc5401561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 -- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc540155d90 msgr2=0x7fc5401561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc540155d90 0x7fc5401561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5451d5640 1 -- 172.21.15.29:0/3977135503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5400b37c0 con 0x7fc52c0039a0 2023-10-16T15:50:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.830+0000 7fc5459d6640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc540155d90 0x7fc5401561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.831+0000 7fc5451d5640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc5401f5d90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fc53c02c570 tx=0x7fc53c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.831+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c034d60 con 0x7fc52c0039a0 2023-10-16T15:50:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.831+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c040450 con 0x7fc52c0039a0 2023-10-16T15:50:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.831+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c040800 con 0x7fc52c0039a0 2023-10-16T15:50:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.831+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5401f6c10 con 0x7fc52c0039a0 2023-10-16T15:50:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.831+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5401f6650 con 0x7fc52c0039a0 2023-10-16T15:50:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.832+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54004e160 con 0x7fc52c0039a0 2023-10-16T15:50:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.833+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc53c040960 con 0x7fc52c0039a0 2023-10-16T15:50:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.834+0000 7fc535ffb640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc52406af90 0x7fc52406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.834+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fc53c0abcd0 con 0x7fc52c0039a0 2023-10-16T15:50:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.835+0000 7fc5449d4640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc52406af90 0x7fc52406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.836+0000 7fc5449d4640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc52406af90 0x7fc52406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5400ce550 tx=0x7fc530009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:39.838+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc53c077020 con 0x7fc52c0039a0 2023-10-16T15:50:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.100+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc5400c9610 con 0x7fc52c0039a0 2023-10-16T15:50:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.101+0000 7fc535ffb640 1 -- 172.21.15.29:0/3977135503 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v47) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc53c07a640 con 0x7fc52c0039a0 2023-10-16T15:50:40.104 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:40.104 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:50:40.104 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T15:50:40.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.104 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T15:50:40.105 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T15:50:40.105 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.105 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.105 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:50:40.105 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:50:40.106 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.106 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.106 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T15:50:40.107 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T15:50:40.107 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.107 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.107 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:50:40.107 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:50:40.107 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.108 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.108 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T15:50:40.108 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T15:50:40.108 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.109 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.109 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:50:40.109 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:50:40.109 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.109 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.110 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T15:50:40.110 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T15:50:40.110 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:50:40.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:50:40.111 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.112 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T15:50:40.112 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T15:50:40.112 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.112 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.112 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.113 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.113 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.113 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T15:50:40.114 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T15:50:40.114 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.114 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.114 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.114 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.115 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T15:50:40.115 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T15:50:40.115 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.116 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.116 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.116 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.116 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.117 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T15:50:40.117 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T15:50:40.117 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.117 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.118 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T15:50:40.118 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T15:50:40.119 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.119 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.119 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T15:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T15:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.121 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.121 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.121 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.122 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T15:50:40.122 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T15:50:40.122 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.122 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.122 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.122 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.123 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.123 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T15:50:40.123 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T15:50:40.124 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.124 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:50:40.124 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:50:40.124 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.125 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T15:50:40.125 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T15:50:40.125 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.125 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.125 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T15:50:40.126 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T15:50:40.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.126 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T15:50:40.127 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T15:50:40.127 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.127 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.127 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:50:40.127 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:50:40.128 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:50:40.128 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.128 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T15:50:40.128 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T15:50:40.129 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.129 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.129 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:50:40.129 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:50:40.129 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:50:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T15:50:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T15:50:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.131 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:50:40.131 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:50:40.131 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:50:40.131 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:50:40.131 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.132 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T15:50:40.132 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T15:50:40.132 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.133 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:50:40.133 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:50:40.133 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:50:40.133 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T15:50:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T15:50:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:50:40.134 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:50:40.135 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.135 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:50:40.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:40.135 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T15:50:40.136 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T15:50:40.136 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:50:40.136 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:50:40.136 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:50:40.136 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:50:40.136 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:40.137 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T15:50:40.137 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:50:40.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc52406af90 msgr2=0x7fc52406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc52406af90 0x7fc52406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5400ce550 tx=0x7fc530009040 comp rx=0 tx=0).stop 2023-10-16T15:50:40.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 msgr2=0x7fc5401f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc5401f5d90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fc53c02c570 tx=0x7fc53c009750 comp rx=0 tx=0).stop 2023-10-16T15:50:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 shutdown_connections 2023-10-16T15:50:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc52406af90 0x7fc52406d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc540155d90 0x7fc5401561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc5401578f0 0x7fc5400b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 --2- 172.21.15.29:0/3977135503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc52c0039a0 0x7fc5401f5d90 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 >> 172.21.15.29:0/3977135503 conn(0x7fc5400cb410 msgr2=0x7fc54015d910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.104+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 shutdown_connections 2023-10-16T15:50:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.105+0000 7fc547439640 1 -- 172.21.15.29:0/3977135503 wait complete. 2023-10-16T15:50:40.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.367+0000 7ff38dc47640 1 Processor -- start 2023-10-16T15:50:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.367+0000 7ff38dc47640 1 -- start start 2023-10-16T15:50:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.368+0000 7ff38dc47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff38806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.368+0000 7ff38dc47640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff3880cda80 0x7ff38815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.368+0000 7ff38dc47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38806c960 con 0x7ff3880ad970 2023-10-16T15:50:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff38dc47640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38815b9d0 con 0x7ff3880cda80 2023-10-16T15:50:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff38dc47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815bcb0 con 0x7ff38806c040 2023-10-16T15:50:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff387fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff3880ad970 0x7ff38815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51448/0 (socket says 172.21.15.29:51448) 2023-10-16T15:50:40.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff387fff640 1 -- 172.21.15.29:0/61204769 learned_addr learned my addr 172.21.15.29:0/61204769 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:40.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff38d446640 1 --2- 172.21.15.29:0/61204769 >> v2:172.21.15.170:3300/0 conn(0x7ff3880cda80 0x7ff38815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff38cc45640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff38806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547599611 0 0) 0x7ff38806c960 con 0x7ff3880ad970 2023-10-16T15:50:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.369+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff378003520 con 0x7ff3880ad970 2023-10-16T15:50:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330302559 0 0) 0x7ff378003520 con 0x7ff3880ad970 2023-10-16T15:50:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 msgr2=0x7ff38806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff38806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 >> v2:172.21.15.170:3300/0 conn(0x7ff3880cda80 msgr2=0x7ff38815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 --2- 172.21.15.29:0/61204769 >> v2:172.21.15.170:3300/0 conn(0x7ff3880cda80 0x7ff38815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38815bf90 con 0x7ff3880ad970 2023-10-16T15:50:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7ff374001670 con 0x7ff3880ad970 2023-10-16T15:50:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 >> v1:172.21.15.29:6789/0 conn(0x7ff3880ad970 legacy=0x7ff38815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff378003970 0x7ff378003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff3780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff378004d00 0x7ff378005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815b9d0 con 0x7ff378004d00 2023-10-16T15:50:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815bcb0 con 0x7ff378003970 2023-10-16T15:50:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37800fd90 con 0x7ff38806c040 2023-10-16T15:50:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.370+0000 7ff38d446640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff378004d00 0x7ff378005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38cc45640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff378003970 0x7ff378003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff387fff640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff3780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38d446640 1 -- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 msgr2=0x7ff3780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38d446640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff3780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38d446640 1 -- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff378003970 msgr2=0x7ff378003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38d446640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff378003970 0x7ff378003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38d446640 1 -- 172.21.15.29:0/61204769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378003520 con 0x7ff378004d00 2023-10-16T15:50:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff38cc45640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff378003970 0x7ff378003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.371+0000 7ff387fff640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff38806c040 0x7ff3780047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:40.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.372+0000 7ff38d446640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff378004d00 0x7ff378005180 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e170 tx=0x7ff3780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7785aba3f37a4c40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:40.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.372+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38002cc20 con 0x7ff378004d00 2023-10-16T15:50:40.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38000f040 con 0x7ff378004d00 2023-10-16T15:50:40.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff3877fe640 1 -- 172.21.15.29:0/61204769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380034440 con 0x7ff378004d00 2023-10-16T15:50:40.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38dc47640 1 -- 172.21.15.29:0/61204769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff378004d00 msgr2=0x7ff378005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38dc47640 1 --2- 172.21.15.29:0/61204769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff378004d00 0x7ff378005180 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e170 tx=0x7ff3780057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38d446640 1 -- 172.21.15.29:0/61204769 reap_dead start 2023-10-16T15:50:40.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38dc47640 1 -- 172.21.15.29:0/61204769 shutdown_connections 2023-10-16T15:50:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38dc47640 1 -- 172.21.15.29:0/61204769 >> 172.21.15.29:0/61204769 conn(0x7ff3880ce6a0 msgr2=0x7ff3880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38dc47640 1 -- 172.21.15.29:0/61204769 shutdown_connections 2023-10-16T15:50:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.373+0000 7ff38dc47640 1 -- 172.21.15.29:0/61204769 wait complete. 2023-10-16T15:50:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.375+0000 7ff38dc47640 1 Processor -- start 2023-10-16T15:50:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.375+0000 7ff38dc47640 1 -- start start 2023-10-16T15:50:40.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff38dc47640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3880b3790 0x7ff38815d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff38dc47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 0x7ff3880afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff38dc47640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3881f1830 0x7ff3881f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff38dc47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881f2220 con 0x7ff3881ef400 2023-10-16T15:50:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff38dc47640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881ef880 con 0x7ff3880b3790 2023-10-16T15:50:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff38dc47640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880affb0 con 0x7ff3881f1830 2023-10-16T15:50:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff387fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 0x7ff3880afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff387fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 0x7ff3880afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41066/0 (socket says 172.21.15.29:41066) 2023-10-16T15:50:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.376+0000 7ff387fff640 1 -- 172.21.15.29:0/1431267040 learned_addr learned my addr 172.21.15.29:0/1431267040 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff38cc45640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3880b3790 0x7ff38815d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff38d446640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3881f1830 0x7ff3881f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff387fff640 1 -- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3881f1830 msgr2=0x7ff3881f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff387fff640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3881f1830 0x7ff3881f1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff387fff640 1 -- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3880b3790 msgr2=0x7ff38815d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff387fff640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3880b3790 0x7ff38815d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff387fff640 1 -- 172.21.15.29:0/1431267040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880b0250 con 0x7ff3881ef400 2023-10-16T15:50:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff38cc45640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3880b3790 0x7ff38815d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff387fff640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 0x7ff3880afa70 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7ff37400db10 tx=0x7ff374007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374004cb0 con 0x7ff3881ef400 2023-10-16T15:50:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374004e10 con 0x7ff3881ef400 2023-10-16T15:50:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.377+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3740072d0 con 0x7ff3881ef400 2023-10-16T15:50:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.378+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3880c5770 con 0x7ff3881ef400 2023-10-16T15:50:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.378+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3880c5d30 con 0x7ff3881ef400 2023-10-16T15:50:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.379+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38806b330 con 0x7ff3881ef400 2023-10-16T15:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.380+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff374007430 con 0x7ff3881ef400 2023-10-16T15:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.380+0000 7ff3857fa640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff36c06afc0 0x7ff36c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.381+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 4885+0+0 (secure 0 0 0) 0x7ff374089020 con 0x7ff3881ef400 2023-10-16T15:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.381+0000 7ff38cc45640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff36c06afc0 0x7ff36c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.382+0000 7ff38cc45640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff36c06afc0 0x7ff36c06d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff37c006fd0 tx=0x7ff37c008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.385+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff3740541e0 con 0x7ff3881ef400 2023-10-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.580+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff38804e9b0 con 0x7ff3881ef400 2023-10-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.586+0000 7ff3857fa640 1 -- 172.21.15.29:0/1431267040 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+12911 (secure 0 0 0) 0x7ff374057800 con 0x7ff3881ef400 2023-10-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":205,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:39.574372+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff36c06afc0 msgr2=0x7ff36c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff36c06afc0 0x7ff36c06d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff37c006fd0 tx=0x7ff37c008040 comp rx=0 tx=0).stop 2023-10-16T15:50:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 msgr2=0x7ff3880afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 0x7ff3880afa70 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7ff37400db10 tx=0x7ff374007e30 comp rx=0 tx=0).stop 2023-10-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 shutdown_connections 2023-10-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7ff36c06afc0 0x7ff36c06d480 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3881ef400 0x7ff3880afa70 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3881f1830 0x7ff3881f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 --2- 172.21.15.29:0/1431267040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff3880b3790 0x7ff38815d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.588+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 >> 172.21.15.29:0/1431267040 conn(0x7ff3880ce6a0 msgr2=0x7ff38815b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.589+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 shutdown_connections 2023-10-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.589+0000 7ff38dc47640 1 -- 172.21.15.29:0/1431267040 wait complete. 2023-10-16T15:50:40.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:50:40.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:50:40.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.835+0000 7f6a4874c640 1 Processor -- start 2023-10-16T15:50:40.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.835+0000 7f6a4874c640 1 -- start start 2023-10-16T15:50:40.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a4874c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a4015d150 0x7f6a400ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a4874c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a4006a4b0 0x7f6a40160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a4874c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400ac890 con 0x7f6a40160b90 2023-10-16T15:50:40.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a4874c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4015d530 con 0x7f6a4006a4b0 2023-10-16T15:50:40.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a4874c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a40161650 con 0x7f6a4015d150 2023-10-16T15:50:40.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a46ce9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a4006a4b0 0x7f6a40160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a46ce9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a4006a4b0 0x7f6a40160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41082/0 (socket says 172.21.15.29:41082) 2023-10-16T15:50:40.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.837+0000 7f6a46ce9640 1 -- 172.21.15.29:0/623091149 learned_addr learned my addr 172.21.15.29:0/623091149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:40.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.838+0000 7f6a464e8640 1 --2- 172.21.15.29:0/623091149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a4015d150 0x7f6a400ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.838+0000 7f6a46ce9640 1 -- 172.21.15.29:0/623091149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a4015d150 msgr2=0x7f6a400ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:40.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.838+0000 7f6a46ce9640 1 --2- 172.21.15.29:0/623091149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a4015d150 0x7f6a400ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.838+0000 7f6a46ce9640 1 -- 172.21.15.29:0/623091149 >> v1:172.21.15.170:6789/0 conn(0x7f6a40160b90 legacy=0x7f6a40069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:40.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.838+0000 7f6a46ce9640 1 -- 172.21.15.29:0/623091149 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a40161930 con 0x7f6a4006a4b0 2023-10-16T15:50:40.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.839+0000 7f6a46ce9640 1 --2- 172.21.15.29:0/623091149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4006a4b0 0x7f6a40160fb0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009e50 tx=0x7f6a3802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f298238cfa2f38bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:40.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.839+0000 7f6a454e6640 1 -- 172.21.15.29:0/623091149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3802ca80 con 0x7f6a4006a4b0 2023-10-16T15:50:40.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.839+0000 7f6a454e6640 1 -- 172.21.15.29:0/623091149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3802cbe0 con 0x7f6a4006a4b0 2023-10-16T15:50:40.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a454e6640 1 -- 172.21.15.29:0/623091149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a380343f0 con 0x7f6a4006a4b0 2023-10-16T15:50:40.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 -- 172.21.15.29:0/623091149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4006a4b0 msgr2=0x7f6a40160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 --2- 172.21.15.29:0/623091149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4006a4b0 0x7f6a40160fb0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009e50 tx=0x7f6a3802bee0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 -- 172.21.15.29:0/623091149 shutdown_connections 2023-10-16T15:50:40.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 --2- 172.21.15.29:0/623091149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4006a4b0 0x7f6a40160fb0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 --2- 172.21.15.29:0/623091149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a4015d150 0x7f6a400ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 -- 172.21.15.29:0/623091149 >> 172.21.15.29:0/623091149 conn(0x7f6a400ce9b0 msgr2=0x7f6a400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:40.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 -- 172.21.15.29:0/623091149 shutdown_connections 2023-10-16T15:50:40.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.840+0000 7f6a4874c640 1 -- 172.21.15.29:0/623091149 wait complete. 2023-10-16T15:50:40.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.842+0000 7f6a4874c640 1 Processor -- start 2023-10-16T15:50:40.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.842+0000 7f6a4874c640 1 -- start start 2023-10-16T15:50:40.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a4874c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a4006a4b0 0x7f6a401600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a4874c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 0x7f6a400afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a4874c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a40160b90 0x7f6a400ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a4874c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4014e9b0 con 0x7f6a4015d150 2023-10-16T15:50:40.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a4874c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4014edf0 con 0x7f6a4006a4b0 2023-10-16T15:50:40.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a4874c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400aedb0 con 0x7f6a40160b90 2023-10-16T15:50:40.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.843+0000 7f6a45ce7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 0x7f6a400afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 0x7f6a400afe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41090/0 (socket says 172.21.15.29:41090) 2023-10-16T15:50:40.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 -- 172.21.15.29:0/1232976168 learned_addr learned my addr 172.21.15.29:0/1232976168 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:40.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a464e8640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a4006a4b0 0x7f6a401600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a46ce9640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a40160b90 0x7f6a400ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 -- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a40160b90 msgr2=0x7f6a400ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a40160b90 0x7f6a400ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 -- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a4006a4b0 msgr2=0x7f6a401600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:40.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a4006a4b0 0x7f6a401600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:40.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 -- 172.21.15.29:0/1232976168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38009b30 con 0x7f6a4015d150 2023-10-16T15:50:40.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.844+0000 7f6a45ce7640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 0x7f6a400afe50 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000b3f0 tx=0x7f6a3000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:40.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.845+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011830 con 0x7f6a4015d150 2023-10-16T15:50:40.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.845+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a30011990 con 0x7f6a4015d150 2023-10-16T15:50:40.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.845+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011d40 con 0x7f6a4015d150 2023-10-16T15:50:40.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.845+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400af090 con 0x7f6a4015d150 2023-10-16T15:50:40.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.845+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400af650 con 0x7f6a4015d150 2023-10-16T15:50:40.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.846+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4004e160 con 0x7f6a4015d150 2023-10-16T15:50:40.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.847+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a3001a400 con 0x7f6a4015d150 2023-10-16T15:50:40.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.848+0000 7f6a36ffd640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6a2406af90 0x7f6a2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:40.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.848+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f6a300893d0 con 0x7f6a4015d150 2023-10-16T15:50:40.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.849+0000 7f6a464e8640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6a2406af90 0x7f6a2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:40.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.850+0000 7f6a464e8640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6a2406af90 0x7f6a2406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c006fd0 tx=0x7f6a3c008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:40.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:40.852+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a30054570 con 0x7f6a4015d150 2023-10-16T15:50:41.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.043+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a400ade10 con 0x7f6a4015d150 2023-10-16T15:50:41.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.044+0000 7f6a36ffd640 1 -- 172.21.15.29:0/1232976168 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a30057b90 con 0x7f6a4015d150 2023-10-16T15:50:41.046 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:50:41.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.046+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6a2406af90 msgr2=0x7f6a2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.046+0000 7f6a4874c640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6a2406af90 0x7f6a2406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c006fd0 tx=0x7f6a3c008040 comp rx=0 tx=0).stop 2023-10-16T15:50:41.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.046+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 msgr2=0x7f6a400afe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.046+0000 7f6a4874c640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 0x7f6a400afe50 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000b3f0 tx=0x7f6a3000b8b0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 shutdown_connections 2023-10-16T15:50:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6a2406af90 0x7f6a2406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a4015d150 0x7f6a400afe50 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a40160b90 0x7f6a400ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 --2- 172.21.15.29:0/1232976168 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a4006a4b0 0x7f6a401600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 >> 172.21.15.29:0/1232976168 conn(0x7f6a400ce9b0 msgr2=0x7f6a400b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 shutdown_connections 2023-10-16T15:50:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.047+0000 7f6a4874c640 1 -- 172.21.15.29:0/1232976168 wait complete. 2023-10-16T15:50:41.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:50:41.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.285+0000 7fb1d5410640 1 Processor -- start 2023-10-16T15:50:41.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.285+0000 7fb1d5410640 1 -- start start 2023-10-16T15:50:41.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.287+0000 7fb1d5410640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 0x7fb1d00ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.287+0000 7fb1d5410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d006b4c0 0x7fb1d00cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.287+0000 7fb1d5410640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d006e170 con 0x7fb1d0160800 2023-10-16T15:50:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.287+0000 7fb1d5410640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d00cde30 con 0x7fb1d0160420 2023-10-16T15:50:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.287+0000 7fb1d5410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00cd460 con 0x7fb1d006b4c0 2023-10-16T15:50:41.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.288+0000 7fb1ceffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 0x7fb1d00ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.288+0000 7fb1cf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d006b4c0 0x7fb1d00cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.288+0000 7fb1ceffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 0x7fb1d00ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47712/0 (socket says 172.21.15.29:47712) 2023-10-16T15:50:41.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.288+0000 7fb1ceffd640 1 -- 172.21.15.29:0/1208339742 learned_addr learned my addr 172.21.15.29:0/1208339742 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:41.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.289+0000 7fb1cf7fe640 1 -- 172.21.15.29:0/1208339742 >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 msgr2=0x7fb1d00ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:41.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.289+0000 7fb1cf7fe640 1 --2- 172.21.15.29:0/1208339742 >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 0x7fb1d00ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.289+0000 7fb1cf7fe640 1 -- 172.21.15.29:0/1208339742 >> v1:172.21.15.170:6789/0 conn(0x7fb1d0160800 legacy=0x7fb1d006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.289+0000 7fb1cf7fe640 1 -- 172.21.15.29:0/1208339742 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d006e3c0 con 0x7fb1d006b4c0 2023-10-16T15:50:41.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.289+0000 7fb1ceffd640 1 --2- 172.21.15.29:0/1208339742 >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 0x7fb1d00ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:50:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.290+0000 7fb1cf7fe640 1 --2- 172.21.15.29:0/1208339742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d006b4c0 0x7fb1d00cd8f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80097a0 tx=0x7fb1c802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70bdcee937c0c63d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.290+0000 7fb1ce7fc640 1 -- 172.21.15.29:0/1208339742 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c802d040 con 0x7fb1d006b4c0 2023-10-16T15:50:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1ce7fc640 1 -- 172.21.15.29:0/1208339742 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c8031810 con 0x7fb1d006b4c0 2023-10-16T15:50:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1ce7fc640 1 -- 172.21.15.29:0/1208339742 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8031ba0 con 0x7fb1d006b4c0 2023-10-16T15:50:41.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 -- 172.21.15.29:0/1208339742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d006b4c0 msgr2=0x7fb1d00cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 --2- 172.21.15.29:0/1208339742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d006b4c0 0x7fb1d00cd8f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80097a0 tx=0x7fb1c802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 -- 172.21.15.29:0/1208339742 shutdown_connections 2023-10-16T15:50:41.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 --2- 172.21.15.29:0/1208339742 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d006b4c0 0x7fb1d00cd8f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 --2- 172.21.15.29:0/1208339742 >> v2:172.21.15.170:3300/0 conn(0x7fb1d0160420 0x7fb1d00ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 -- 172.21.15.29:0/1208339742 >> 172.21.15.29:0/1208339742 conn(0x7fb1d00b0d30 msgr2=0x7fb1d00b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:41.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 -- 172.21.15.29:0/1208339742 shutdown_connections 2023-10-16T15:50:41.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.291+0000 7fb1d5410640 1 -- 172.21.15.29:0/1208339742 wait complete. 2023-10-16T15:50:41.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.293+0000 7fb1d5410640 1 Processor -- start 2023-10-16T15:50:41.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.293+0000 7fb1d5410640 1 -- start start 2023-10-16T15:50:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.294+0000 7fb1d5410640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1d006b4c0 0x7fb1d015af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.294+0000 7fb1d5410640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 0x7fb1d009d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.294+0000 7fb1d5410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d0160800 0x7fb1d00c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.294+0000 7fb1d5410640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00c76d0 con 0x7fb1d0160420 2023-10-16T15:50:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.294+0000 7fb1d5410640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d009e0f0 con 0x7fb1d006b4c0 2023-10-16T15:50:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.294+0000 7fb1d5410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d009e3d0 con 0x7fb1d0160800 2023-10-16T15:50:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 0x7fb1d009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 0x7fb1d009d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41104/0 (socket says 172.21.15.29:41104) 2023-10-16T15:50:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 -- 172.21.15.29:0/3882988371 learned_addr learned my addr 172.21.15.29:0/3882988371 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1ceffd640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1d006b4c0 0x7fb1d015af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1cf7fe640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d0160800 0x7fb1d00c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 -- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d0160800 msgr2=0x7fb1d00c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d0160800 0x7fb1d00c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 -- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1d006b4c0 msgr2=0x7fb1d015af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1d006b4c0 0x7fb1d015af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.295+0000 7fb1c67fc640 1 -- 172.21.15.29:0/3882988371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c00094d0 con 0x7fb1d0160420 2023-10-16T15:50:41.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.296+0000 7fb1c67fc640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 0x7fb1d009d630 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c730 tx=0x7fb1b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:41.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.296+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8003b70 con 0x7fb1d0160420 2023-10-16T15:50:41.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.296+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8003cf0 con 0x7fb1d0160420 2023-10-16T15:50:41.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.296+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c8009400 con 0x7fb1d0160420 2023-10-16T15:50:41.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.296+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8003ea0 con 0x7fb1d0160420 2023-10-16T15:50:41.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.296+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d00c7360 con 0x7fb1d0160420 2023-10-16T15:50:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.297+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d009e8f0 con 0x7fb1d0160420 2023-10-16T15:50:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.298+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb1b800aa90 con 0x7fb1d0160420 2023-10-16T15:50:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.299+0000 7fb1c7fff640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb1b006af90 0x7fb1b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.299+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fb1b8089290 con 0x7fb1d0160420 2023-10-16T15:50:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.300+0000 7fb1ceffd640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb1b006af90 0x7fb1b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.301+0000 7fb1ceffd640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb1b006af90 0x7fb1b006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009cf0 tx=0x7fb1c0008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.303+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb1b8054560 con 0x7fb1d0160420 2023-10-16T15:50:41.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.501+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb1d00c6120 con 0x7fb1d0160420 2023-10-16T15:50:41.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.502+0000 7fb1c7fff640 1 -- 172.21.15.29:0/3882988371 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v205) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb1b8057b80 con 0x7fb1d0160420 2023-10-16T15:50:41.505 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:41.505 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:50:41.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:41.505 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:50:41.505 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:50:41.506 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:41.506 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:50:41.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb1b006af90 msgr2=0x7fb1b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb1b006af90 0x7fb1b006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009cf0 tx=0x7fb1c0008040 comp rx=0 tx=0).stop 2023-10-16T15:50:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 msgr2=0x7fb1d009d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 0x7fb1d009d630 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800c730 tx=0x7fb1b800cbf0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 shutdown_connections 2023-10-16T15:50:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb1b006af90 0x7fb1b006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1d0160420 0x7fb1d009d630 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb1d0160800 0x7fb1d00c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 --2- 172.21.15.29:0/3882988371 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1d006b4c0 0x7fb1d015af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.505+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 >> 172.21.15.29:0/3882988371 conn(0x7fb1d00b0d30 msgr2=0x7fb1d00cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.506+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 shutdown_connections 2023-10-16T15:50:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.506+0000 7fb1d5410640 1 -- 172.21.15.29:0/3882988371 wait complete. 2023-10-16T15:50:41.527 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T15:50:41.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T15:50:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.755+0000 7f4c5f59e640 1 Processor -- start 2023-10-16T15:50:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.755+0000 7f4c5f59e640 1 -- start start 2023-10-16T15:50:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.756+0000 7f4c5f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4c600ac890 0x7f4c600c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.757+0000 7f4c5f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600cc910 0x7f4c600ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.757+0000 7f4c5f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600cc480 con 0x7f4c600acc70 2023-10-16T15:50:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.757+0000 7f4c5f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600adfe0 con 0x7f4c600ac890 2023-10-16T15:50:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.757+0000 7f4c5f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6015d8c0 con 0x7f4c600cc910 2023-10-16T15:50:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5e59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4c600ac890 0x7f4c600c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5ed9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600cc910 0x7f4c600ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5e59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4c600ac890 0x7f4c600c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:47740/0 (socket says 172.21.15.29:47740) 2023-10-16T15:50:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5e59c640 1 -- 172.21.15.29:0/576480164 learned_addr learned my addr 172.21.15.29:0/576480164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5ed9d640 1 -- 172.21.15.29:0/576480164 >> v2:172.21.15.170:3300/0 conn(0x7f4c600ac890 msgr2=0x7f4c600c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5ed9d640 1 --2- 172.21.15.29:0/576480164 >> v2:172.21.15.170:3300/0 conn(0x7f4c600ac890 0x7f4c600c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5ed9d640 1 -- 172.21.15.29:0/576480164 >> v1:172.21.15.170:6789/0 conn(0x7f4c600acc70 legacy=0x7f4c6015d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.758+0000 7f4c5ed9d640 1 -- 172.21.15.29:0/576480164 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6015dba0 con 0x7f4c600cc910 2023-10-16T15:50:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.760+0000 7f4c5ed9d640 1 --2- 172.21.15.29:0/576480164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600cc910 0x7f4c600ccda0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009d90 tx=0x7f4c5802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69d2fa930d375c49 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.760+0000 7f4c5dd9b640 1 -- 172.21.15.29:0/576480164 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5802cc60 con 0x7f4c600cc910 2023-10-16T15:50:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.760+0000 7f4c5dd9b640 1 -- 172.21.15.29:0/576480164 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5802cdc0 con 0x7f4c600cc910 2023-10-16T15:50:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.760+0000 7f4c5dd9b640 1 -- 172.21.15.29:0/576480164 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58034660 con 0x7f4c600cc910 2023-10-16T15:50:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 -- 172.21.15.29:0/576480164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600cc910 msgr2=0x7f4c600ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/576480164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600cc910 0x7f4c600ccda0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009d90 tx=0x7f4c5802beb0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 -- 172.21.15.29:0/576480164 shutdown_connections 2023-10-16T15:50:41.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/576480164 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600cc910 0x7f4c600ccda0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/576480164 >> v2:172.21.15.170:3300/0 conn(0x7f4c600ac890 0x7f4c600c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 -- 172.21.15.29:0/576480164 >> 172.21.15.29:0/576480164 conn(0x7f4c600ce820 msgr2=0x7f4c600cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:41.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 -- 172.21.15.29:0/576480164 shutdown_connections 2023-10-16T15:50:41.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.761+0000 7f4c5f59e640 1 -- 172.21.15.29:0/576480164 wait complete. 2023-10-16T15:50:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.763+0000 7f4c5f59e640 1 Processor -- start 2023-10-16T15:50:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.763+0000 7f4c5f59e640 1 -- start start 2023-10-16T15:50:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.764+0000 7f4c5f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 0x7f4c6015f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.764+0000 7f4c5f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600acc70 0x7f4c60157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.764+0000 7f4c5f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c600cc910 0x7f4c601e66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.764+0000 7f4c5f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60158fa0 con 0x7f4c600ac890 2023-10-16T15:50:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.764+0000 7f4c5f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60158520 con 0x7f4c600cc910 2023-10-16T15:50:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.764+0000 7f4c5f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600c6d80 con 0x7f4c600acc70 2023-10-16T15:50:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 0x7f4c6015f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 0x7f4c6015f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41112/0 (socket says 172.21.15.29:41112) 2023-10-16T15:50:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 -- 172.21.15.29:0/601808637 learned_addr learned my addr 172.21.15.29:0/601808637 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5ed9d640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c600cc910 0x7f4c601e66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c57fff640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600acc70 0x7f4c60157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 -- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600acc70 msgr2=0x7f4c60157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600acc70 0x7f4c60157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 -- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c600cc910 msgr2=0x7f4c601e66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c600cc910 0x7f4c601e66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:41.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.765+0000 7f4c5e59c640 1 -- 172.21.15.29:0/601808637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58009a70 con 0x7f4c600ac890 2023-10-16T15:50:41.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.766+0000 7f4c5e59c640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 0x7f4c6015f750 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009fd0 tx=0x7f4c5000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:41.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.766+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000c940 con 0x7f4c600ac890 2023-10-16T15:50:41.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.766+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5000caa0 con 0x7f4c600ac890 2023-10-16T15:50:41.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.766+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000ce30 con 0x7f4c600ac890 2023-10-16T15:50:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.766+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c600c7060 con 0x7f4c600ac890 2023-10-16T15:50:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.766+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c600c7620 con 0x7f4c600ac890 2023-10-16T15:50:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.767+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6004e9b0 con 0x7f4c600ac890 2023-10-16T15:50:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.768+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4c50013470 con 0x7f4c600ac890 2023-10-16T15:50:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.769+0000 7f4c56ffd640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4c4006af90 0x7f4c4006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.769+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f4c50089560 con 0x7f4c600ac890 2023-10-16T15:50:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.770+0000 7f4c57fff640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4c4006af90 0x7f4c4006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.771+0000 7f4c57fff640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4c4006af90 0x7f4c4006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4c601f6380 tx=0x7f4c48009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.774+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4c500548c0 con 0x7f4c600ac890 2023-10-16T15:50:41.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:41.965+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4c600c8ab0 con 0x7f4c600ac890 2023-10-16T15:50:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.600+0000 7f4c56ffd640 1 -- 172.21.15.29:0/601808637 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v206) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4c50057ee0 con 0x7f4c600ac890 2023-10-16T15:50:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T15:50:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4c4006af90 msgr2=0x7f4c4006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4c4006af90 0x7f4c4006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4c601f6380 tx=0x7f4c48009040 comp rx=0 tx=0).stop 2023-10-16T15:50:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 msgr2=0x7f4c6015f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 0x7f4c6015f750 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009fd0 tx=0x7f4c5000ed40 comp rx=0 tx=0).stop 2023-10-16T15:50:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 shutdown_connections 2023-10-16T15:50:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f4c4006af90 0x7f4c4006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c600cc910 0x7f4c601e66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4c600acc70 0x7f4c60157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 --2- 172.21.15.29:0/601808637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c600ac890 0x7f4c6015f750 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 >> 172.21.15.29:0/601808637 conn(0x7f4c600ce820 msgr2=0x7f4c600ccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 shutdown_connections 2023-10-16T15:50:42.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.604+0000 7f4c5f59e640 1 -- 172.21.15.29:0/601808637 wait complete. 2023-10-16T15:50:42.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:50:42.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.844+0000 7fd55a193640 1 Processor -- start 2023-10-16T15:50:42.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.844+0000 7fd55a193640 1 -- start start 2023-10-16T15:50:42.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.846+0000 7fd55a193640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5540ca1a0 0x7fd5540cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:42.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.846+0000 7fd55a193640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd55406f940 0x7fd554069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:42.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.846+0000 7fd55a193640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd554161760 con 0x7fd5540ccb90 2023-10-16T15:50:42.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.846+0000 7fd55a193640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5540cd010 con 0x7fd55406f940 2023-10-16T15:50:42.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.846+0000 7fd55a193640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540cd2f0 con 0x7fd5540ca1a0 2023-10-16T15:50:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd559992640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd55406f940 0x7fd554069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd558990640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fd5540ccb90 0x7fd55406f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:56484/0 (socket says 172.21.15.29:56484) 2023-10-16T15:50:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd558990640 1 -- 172.21.15.29:0/280101358 learned_addr learned my addr 172.21.15.29:0/280101358 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd559191640 1 --2- 172.21.15.29:0/280101358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5540ca1a0 0x7fd5540cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd559992640 1 -- 172.21.15.29:0/280101358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5540ca1a0 msgr2=0x7fd5540cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd559992640 1 --2- 172.21.15.29:0/280101358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5540ca1a0 0x7fd5540cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd559992640 1 -- 172.21.15.29:0/280101358 >> v1:172.21.15.170:6789/0 conn(0x7fd5540ccb90 legacy=0x7fd55406f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.847+0000 7fd559992640 1 -- 172.21.15.29:0/280101358 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5540cd5d0 con 0x7fd55406f940 2023-10-16T15:50:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.849+0000 7fd559992640 1 --2- 172.21.15.29:0/280101358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd55406f940 0x7fd554069da0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009d90 tx=0x7fd54c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16b51ba93dac22d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.849+0000 7fd54bfff640 1 -- 172.21.15.29:0/280101358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c02cce0 con 0x7fd55406f940 2023-10-16T15:50:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.849+0000 7fd54bfff640 1 -- 172.21.15.29:0/280101358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c02ce40 con 0x7fd55406f940 2023-10-16T15:50:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.849+0000 7fd54bfff640 1 -- 172.21.15.29:0/280101358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c034660 con 0x7fd55406f940 2023-10-16T15:50:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 -- 172.21.15.29:0/280101358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd55406f940 msgr2=0x7fd554069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 --2- 172.21.15.29:0/280101358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd55406f940 0x7fd554069da0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009d90 tx=0x7fd54c02bf30 comp rx=0 tx=0).stop 2023-10-16T15:50:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 -- 172.21.15.29:0/280101358 shutdown_connections 2023-10-16T15:50:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 --2- 172.21.15.29:0/280101358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd55406f940 0x7fd554069da0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 --2- 172.21.15.29:0/280101358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5540ca1a0 0x7fd5540cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 -- 172.21.15.29:0/280101358 >> 172.21.15.29:0/280101358 conn(0x7fd5540ce6a0 msgr2=0x7fd5540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 -- 172.21.15.29:0/280101358 shutdown_connections 2023-10-16T15:50:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.850+0000 7fd55a193640 1 -- 172.21.15.29:0/280101358 wait complete. 2023-10-16T15:50:42.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.852+0000 7fd55a193640 1 Processor -- start 2023-10-16T15:50:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.852+0000 7fd55a193640 1 -- start start 2023-10-16T15:50:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.853+0000 7fd55a193640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd55406f940 0x7fd554160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.853+0000 7fd55a193640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5540ca1a0 0x7fd55406b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.853+0000 7fd55a193640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 0x7fd5541ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.853+0000 7fd55a193640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd554161210 con 0x7fd5540ccb90 2023-10-16T15:50:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.853+0000 7fd55a193640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5541f6600 con 0x7fd5540ca1a0 2023-10-16T15:50:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.853+0000 7fd55a193640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540c4080 con 0x7fd55406f940 2023-10-16T15:50:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 0x7fd5541ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 0x7fd5541ea6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41134/0 (socket says 172.21.15.29:41134) 2023-10-16T15:50:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 -- 172.21.15.29:0/221729902 learned_addr learned my addr 172.21.15.29:0/221729902 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559191640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd55406f940 0x7fd554160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd558990640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5540ca1a0 0x7fd55406b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 -- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd55406f940 msgr2=0x7fd554160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd55406f940 0x7fd554160cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 -- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5540ca1a0 msgr2=0x7fd55406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5540ca1a0 0x7fd55406b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.854+0000 7fd559992640 1 -- 172.21.15.29:0/221729902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c009a70 con 0x7fd5540ccb90 2023-10-16T15:50:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.855+0000 7fd559992640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 0x7fd5541ea6c0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009d90 tx=0x7fd54c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.855+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c03d600 con 0x7fd5540ccb90 2023-10-16T15:50:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.855+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c03d780 con 0x7fd5540ccb90 2023-10-16T15:50:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.855+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c03db30 con 0x7fd5540ccb90 2023-10-16T15:50:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.855+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5540c4360 con 0x7fd5540ccb90 2023-10-16T15:50:42.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.855+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5540c4920 con 0x7fd5540ccb90 2023-10-16T15:50:42.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.856+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5541f6790 con 0x7fd5540ccb90 2023-10-16T15:50:42.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.858+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd54c03b040 con 0x7fd5540ccb90 2023-10-16T15:50:42.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.858+0000 7fd549ffb640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd53806af90 0x7fd53806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:42.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.859+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fd54c0abe90 con 0x7fd5540ccb90 2023-10-16T15:50:42.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.859+0000 7fd559191640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd53806af90 0x7fd53806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:42.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.861+0000 7fd559191640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd53806af90 0x7fd53806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd550009ad0 tx=0x7fd550008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:42.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:42.863+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd54c0771f0 con 0x7fd5540ccb90 2023-10-16T15:50:43.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.049+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd55404e9b0 con 0x7fd5540ccb90 2023-10-16T15:50:43.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.607+0000 7fd549ffb640 1 -- 172.21.15.29:0/221729902 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v207) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd54c0390d0 con 0x7fd5540ccb90 2023-10-16T15:50:43.610 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T15:50:43.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.610+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd53806af90 msgr2=0x7fd53806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:43.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.610+0000 7fd55a193640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd53806af90 0x7fd53806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd550009ad0 tx=0x7fd550008040 comp rx=0 tx=0).stop 2023-10-16T15:50:43.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 msgr2=0x7fd5541ea6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:43.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 0x7fd5541ea6c0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009d90 tx=0x7fd54c004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:43.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 shutdown_connections 2023-10-16T15:50:43.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fd53806af90 0x7fd53806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5540ccb90 0x7fd5541ea6c0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5540ca1a0 0x7fd55406b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 --2- 172.21.15.29:0/221729902 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd55406f940 0x7fd554160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 >> 172.21.15.29:0/221729902 conn(0x7fd5540ce6a0 msgr2=0x7fd5540cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:43.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 shutdown_connections 2023-10-16T15:50:43.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.611+0000 7fd55a193640 1 -- 172.21.15.29:0/221729902 wait complete. 2023-10-16T15:50:43.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T15:50:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.838+0000 7fb80998a640 1 Processor -- start 2023-10-16T15:50:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.838+0000 7fb80998a640 1 -- start start 2023-10-16T15:50:43.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb80998a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb804159ab0 0x7fb8040ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:43.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb80998a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb80406c140 0x7fb80406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:43.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb80998a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb804070360 con 0x7fb804159e90 2023-10-16T15:50:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb80998a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80406cb10 con 0x7fb80406c140 2023-10-16T15:50:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb80998a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80415d890 con 0x7fb804159ab0 2023-10-16T15:50:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb8037fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb80406c140 0x7fb80406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb8037fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb80406c140 0x7fb80406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41136/0 (socket says 172.21.15.29:41136) 2023-10-16T15:50:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb8037fe640 1 -- 172.21.15.29:0/2785421523 learned_addr learned my addr 172.21.15.29:0/2785421523 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.840+0000 7fb802ffd640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb804159ab0 0x7fb8040ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.841+0000 7fb8037fe640 1 -- 172.21.15.29:0/2785421523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb804159ab0 msgr2=0x7fb8040ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.841+0000 7fb8037fe640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb804159ab0 0x7fb8040ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.841+0000 7fb8037fe640 1 -- 172.21.15.29:0/2785421523 >> v1:172.21.15.29:6789/0 conn(0x7fb804159e90 legacy=0x7fb80406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.841+0000 7fb8037fe640 1 -- 172.21.15.29:0/2785421523 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8040cd1f0 con 0x7fb80406c140 2023-10-16T15:50:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.841+0000 7fb802ffd640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb804159ab0 0x7fb8040ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.842+0000 7fb8037fe640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb80406c140 0x7fb80406c5d0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e80 tx=0x7fb7fc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7caf62b34a362a87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:43.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.842+0000 7fb8027fc640 1 -- 172.21.15.29:0/2785421523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc02c9d0 con 0x7fb80406c140 2023-10-16T15:50:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.842+0000 7fb8027fc640 1 -- 172.21.15.29:0/2785421523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc02cb30 con 0x7fb80406c140 2023-10-16T15:50:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.842+0000 7fb8027fc640 1 -- 172.21.15.29:0/2785421523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0343f0 con 0x7fb80406c140 2023-10-16T15:50:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 -- 172.21.15.29:0/2785421523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb80406c140 msgr2=0x7fb80406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb80406c140 0x7fb80406c5d0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e80 tx=0x7fb7fc02be30 comp rx=0 tx=0).stop 2023-10-16T15:50:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 -- 172.21.15.29:0/2785421523 shutdown_connections 2023-10-16T15:50:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb80406c140 0x7fb80406c5d0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 --2- 172.21.15.29:0/2785421523 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb804159ab0 0x7fb8040ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 -- 172.21.15.29:0/2785421523 >> 172.21.15.29:0/2785421523 conn(0x7fb8040ce9b0 msgr2=0x7fb8040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 -- 172.21.15.29:0/2785421523 shutdown_connections 2023-10-16T15:50:43.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.843+0000 7fb80998a640 1 -- 172.21.15.29:0/2785421523 wait complete. 2023-10-16T15:50:43.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.845+0000 7fb80998a640 1 Processor -- start 2023-10-16T15:50:43.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.845+0000 7fb80998a640 1 -- start start 2023-10-16T15:50:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb80998a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb80406c140 0x7fb804070760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb80998a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 0x7fb80414f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb80998a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb804159e90 0x7fb80414aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb80998a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb804153230 con 0x7fb804159ab0 2023-10-16T15:50:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb80998a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8041534a0 con 0x7fb804159e90 2023-10-16T15:50:43.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb80998a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb804153780 con 0x7fb80406c140 2023-10-16T15:50:43.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 0x7fb80414f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:43.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 0x7fb80414f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41146/0 (socket says 172.21.15.29:41146) 2023-10-16T15:50:43.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.846+0000 7fb7fbfff640 1 -- 172.21.15.29:0/527658207 learned_addr learned my addr 172.21.15.29:0/527658207 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:43.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb8037fe640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb804159e90 0x7fb80414aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:43.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb7fbfff640 1 -- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb80406c140 msgr2=0x7fb804070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:43.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb7fbfff640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb80406c140 0x7fb804070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb7fbfff640 1 -- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb804159e90 msgr2=0x7fb80414aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:43.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb7fbfff640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb804159e90 0x7fb80414aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb7fbfff640 1 -- 172.21.15.29:0/527658207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc009b30 con 0x7fb804159ab0 2023-10-16T15:50:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb7fbfff640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 0x7fb80414f730 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c450 tx=0x7fb7ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0052d0 con 0x7fb804159ab0 2023-10-16T15:50:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec005430 con 0x7fb804159ab0 2023-10-16T15:50:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec012870 con 0x7fb804159ab0 2023-10-16T15:50:43.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.847+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb804153a60 con 0x7fb804159ab0 2023-10-16T15:50:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.848+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb804153d40 con 0x7fb804159ab0 2023-10-16T15:50:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.849+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8041507d0 con 0x7fb804159ab0 2023-10-16T15:50:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.850+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb7ec0129d0 con 0x7fb804159ab0 2023-10-16T15:50:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.852+0000 7fb800ff9640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb7e406afe0 0x7fb7e406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.852+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fb7ec089160 con 0x7fb804159ab0 2023-10-16T15:50:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.852+0000 7fb802ffd640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb7e406afe0 0x7fb7e406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.854+0000 7fb802ffd640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb7e406afe0 0x7fb7e406d4a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4006fd0 tx=0x7fb7f4008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:43.857+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb8041507d0 con 0x7fb804159ab0 2023-10-16T15:50:44.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.086+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb8041507d0 con 0x7fb804159ab0 2023-10-16T15:50:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.635+0000 7fb800ff9640 1 -- 172.21.15.29:0/527658207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v96) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb8041507d0 con 0x7fb804159ab0 2023-10-16T15:50:44.637 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 24 and data pool 25 2023-10-16T15:50:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb7e406afe0 msgr2=0x7fb7e406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb7e406afe0 0x7fb7e406d4a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4006fd0 tx=0x7fb7f4008040 comp rx=0 tx=0).stop 2023-10-16T15:50:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 msgr2=0x7fb80414f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:44.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 0x7fb80414f730 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c450 tx=0x7fb7ec00c910 comp rx=0 tx=0).stop 2023-10-16T15:50:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 shutdown_connections 2023-10-16T15:50:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fb7e406afe0 0x7fb7e406d4a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb804159ab0 0x7fb80414f730 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb804159e90 0x7fb80414aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 --2- 172.21.15.29:0/527658207 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb80406c140 0x7fb804070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.639+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 >> 172.21.15.29:0/527658207 conn(0x7fb8040ce9b0 msgr2=0x7fb8040b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:44.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.642+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 shutdown_connections 2023-10-16T15:50:44.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.642+0000 7fb80998a640 1 -- 172.21.15.29:0/527658207 wait complete. 2023-10-16T15:50:44.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.866+0000 7fc8ee675640 1 Processor -- start 2023-10-16T15:50:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.866+0000 7fc8ee675640 1 -- start start 2023-10-16T15:50:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8ee675640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8e815aaa0 0x7fc8e815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8ee675640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e80ce390 0x7fc8e8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8ee675640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e815ed60 con 0x7fc8e815ae80 2023-10-16T15:50:44.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8ee675640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e806b810 con 0x7fc8e815aaa0 2023-10-16T15:50:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8ee675640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e80707e0 con 0x7fc8e80ce390 2023-10-16T15:50:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8e815aaa0 0x7fc8e815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8e815aaa0 0x7fc8e815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52202/0 (socket says 172.21.15.29:52202) 2023-10-16T15:50:44.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.868+0000 7fc8e7fff640 1 -- 172.21.15.29:0/2636114721 learned_addr learned my addr 172.21.15.29:0/2636114721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.869+0000 7fc8ecc12640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e80ce390 0x7fc8e8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.869+0000 7fc8e7fff640 1 -- 172.21.15.29:0/2636114721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e80ce390 msgr2=0x7fc8e8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.869+0000 7fc8e7fff640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e80ce390 0x7fc8e8070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.869+0000 7fc8e7fff640 1 -- 172.21.15.29:0/2636114721 >> v1:172.21.15.29:6789/0 conn(0x7fc8e815ae80 legacy=0x7fc8e80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.869+0000 7fc8e7fff640 1 -- 172.21.15.29:0/2636114721 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e815ef90 con 0x7fc8e815aaa0 2023-10-16T15:50:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.869+0000 7fc8ecc12640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e80ce390 0x7fc8e8070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.870+0000 7fc8e7fff640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e815e820 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009870 tx=0x7fc8d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b649671c445359 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.870+0000 7fc8e77fe640 1 -- 172.21.15.29:0/2636114721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8030840 con 0x7fc8e815aaa0 2023-10-16T15:50:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.870+0000 7fc8e77fe640 1 -- 172.21.15.29:0/2636114721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d80309a0 con 0x7fc8e815aaa0 2023-10-16T15:50:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.870+0000 7fc8e77fe640 1 -- 172.21.15.29:0/2636114721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8030d30 con 0x7fc8e815aaa0 2023-10-16T15:50:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 -- 172.21.15.29:0/2636114721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 msgr2=0x7fc8e815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e815e820 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009870 tx=0x7fc8d802b980 comp rx=0 tx=0).stop 2023-10-16T15:50:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 -- 172.21.15.29:0/2636114721 shutdown_connections 2023-10-16T15:50:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e80ce390 0x7fc8e8070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 --2- 172.21.15.29:0/2636114721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e815e820 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 -- 172.21.15.29:0/2636114721 >> 172.21.15.29:0/2636114721 conn(0x7fc8e80cb410 msgr2=0x7fc8e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 -- 172.21.15.29:0/2636114721 shutdown_connections 2023-10-16T15:50:44.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.871+0000 7fc8ee675640 1 -- 172.21.15.29:0/2636114721 wait complete. 2023-10-16T15:50:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.873+0000 7fc8ee675640 1 Processor -- start 2023-10-16T15:50:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.873+0000 7fc8ee675640 1 -- start start 2023-10-16T15:50:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8ee675640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8e80ce390 0x7fc8e80b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8ee675640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e80c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8ee675640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e815ae80 0x7fc8e80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8ee675640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e815d5a0 con 0x7fc8e815aaa0 2023-10-16T15:50:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8ee675640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e80c5180 con 0x7fc8e80ce390 2023-10-16T15:50:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8ee675640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e815d7f0 con 0x7fc8e815ae80 2023-10-16T15:50:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e80c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e80c4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52206/0 (socket says 172.21.15.29:52206) 2023-10-16T15:50:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.874+0000 7fc8dffff640 1 -- 172.21.15.29:0/1273596272 learned_addr learned my addr 172.21.15.29:0/1273596272 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8ecc12640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e815ae80 0x7fc8e80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8dffff640 1 -- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e815ae80 msgr2=0x7fc8e80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8dffff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e815ae80 0x7fc8e80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8e7fff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8e80ce390 0x7fc8e80b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8dffff640 1 -- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8e80ce390 msgr2=0x7fc8e80b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8dffff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8e80ce390 0x7fc8e80b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8dffff640 1 -- 172.21.15.29:0/1273596272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d80094d0 con 0x7fc8e815aaa0 2023-10-16T15:50:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8e7fff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8e80ce390 0x7fc8e80b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8dffff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e80c4c40 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c3c0 tx=0x7fc8d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:44.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d00052d0 con 0x7fc8e815aaa0 2023-10-16T15:50:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d0005430 con 0x7fc8e815aaa0 2023-10-16T15:50:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0003850 con 0x7fc8e815aaa0 2023-10-16T15:50:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.875+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e815dad0 con 0x7fc8e815aaa0 2023-10-16T15:50:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.876+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e815e090 con 0x7fc8e815aaa0 2023-10-16T15:50:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.877+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e804e160 con 0x7fc8e815aaa0 2023-10-16T15:50:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.879+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc8d00039b0 con 0x7fc8e815aaa0 2023-10-16T15:50:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.879+0000 7fc8e57fa640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc8c806af90 0x7fc8c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.879+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc8d0088d10 con 0x7fc8e815aaa0 2023-10-16T15:50:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.880+0000 7fc8e7fff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc8c806af90 0x7fc8c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.881+0000 7fc8e7fff640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc8c806af90 0x7fc8c806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8036000 tx=0x7fc8d803b040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:44.885+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8d008e020 con 0x7fc8e815aaa0 2023-10-16T15:50:45.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.073+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8e80cdc60 con 0x7fc8e815aaa0 2023-10-16T15:50:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.077+0000 7fc8e57fa640 1 -- 172.21.15.29:0/1273596272 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16636 (secure 0 0 0) 0x7fc8d0017090 con 0x7fc8e815aaa0 2023-10-16T15:50:45.079 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:45.079 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":208,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:44.088018+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc8c806af90 msgr2=0x7fc8c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc8c806af90 0x7fc8c806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8036000 tx=0x7fc8d803b040 comp rx=0 tx=0).stop 2023-10-16T15:50:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 msgr2=0x7fc8e80c4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e80c4c40 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c3c0 tx=0x7fc8d000c880 comp rx=0 tx=0).stop 2023-10-16T15:50:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 shutdown_connections 2023-10-16T15:50:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc8c806af90 0x7fc8c806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8e815aaa0 0x7fc8e80c4c40 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8e815ae80 0x7fc8e80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 --2- 172.21.15.29:0/1273596272 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8e80ce390 0x7fc8e80b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.079+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 >> 172.21.15.29:0/1273596272 conn(0x7fc8e80cb410 msgr2=0x7fc8e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.080+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 shutdown_connections 2023-10-16T15:50:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.080+0000 7fc8ee675640 1 -- 172.21.15.29:0/1273596272 wait complete. 2023-10-16T15:50:45.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.317+0000 7fc99ea40640 1 Processor -- start 2023-10-16T15:50:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.317+0000 7fc99ea40640 1 -- start start 2023-10-16T15:50:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99ea40640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc998159950 0x7fc99815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99ea40640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99815aef0 con 0x7fc998070450 2023-10-16T15:50:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99ea40640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99815d910 con 0x7fc99806c810 2023-10-16T15:50:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99ea40640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99815e490 con 0x7fc998159950 2023-10-16T15:50:45.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99da3e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc998159950 0x7fc99815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99da3e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc998159950 0x7fc99815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52210/0 (socket says 172.21.15.29:52210) 2023-10-16T15:50:45.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.318+0000 7fc99da3e640 1 -- 172.21.15.29:0/1350384662 learned_addr learned my addr 172.21.15.29:0/1350384662 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.319+0000 7fc99da3e640 1 -- 172.21.15.29:0/1350384662 >> v1:172.21.15.170:6789/0 conn(0x7fc99806c810 legacy=0x7fc9980c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.319+0000 7fc99da3e640 1 -- 172.21.15.29:0/1350384662 >> v1:172.21.15.29:6789/0 conn(0x7fc998070450 legacy=0x7fc99806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.319+0000 7fc99da3e640 1 -- 172.21.15.29:0/1350384662 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99815dbf0 con 0x7fc998159950 2023-10-16T15:50:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.320+0000 7fc99da3e640 1 --2- 172.21.15.29:0/1350384662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998159950 0x7fc99815a9b0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fc994009870 tx=0x7fc99402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af52be3b854630ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.320+0000 7fc99ca3c640 1 -- 172.21.15.29:0/1350384662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994031870 con 0x7fc998159950 2023-10-16T15:50:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.320+0000 7fc99ca3c640 1 -- 172.21.15.29:0/1350384662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9940319d0 con 0x7fc998159950 2023-10-16T15:50:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.320+0000 7fc99ca3c640 1 -- 172.21.15.29:0/1350384662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994031d60 con 0x7fc998159950 2023-10-16T15:50:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 -- 172.21.15.29:0/1350384662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998159950 msgr2=0x7fc99815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 --2- 172.21.15.29:0/1350384662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998159950 0x7fc99815a9b0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fc994009870 tx=0x7fc99402b910 comp rx=0 tx=0).stop 2023-10-16T15:50:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 -- 172.21.15.29:0/1350384662 shutdown_connections 2023-10-16T15:50:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 --2- 172.21.15.29:0/1350384662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998159950 0x7fc99815a9b0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 -- 172.21.15.29:0/1350384662 >> 172.21.15.29:0/1350384662 conn(0x7fc9980ce6a0 msgr2=0x7fc9980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 -- 172.21.15.29:0/1350384662 shutdown_connections 2023-10-16T15:50:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.321+0000 7fc99ea40640 1 -- 172.21.15.29:0/1350384662 wait complete. 2023-10-16T15:50:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.323+0000 7fc99ea40640 1 Processor -- start 2023-10-16T15:50:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.323+0000 7fc99ea40640 1 -- start start 2023-10-16T15:50:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.323+0000 7fc99ea40640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc99806c810 0x7fc9980c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99ea40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 0x7fc9980a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99ea40640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc998159950 0x7fc998147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99ea40640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980a89c0 con 0x7fc998070450 2023-10-16T15:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99ea40640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980c3ed0 con 0x7fc99806c810 2023-10-16T15:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99ea40640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980c32c0 con 0x7fc998159950 2023-10-16T15:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 0x7fc9980a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 0x7fc9980a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52226/0 (socket says 172.21.15.29:52226) 2023-10-16T15:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 -- 172.21.15.29:0/4202767828 learned_addr learned my addr 172.21.15.29:0/4202767828 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99da3e640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc99806c810 0x7fc9980c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 -- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc998159950 msgr2=0x7fc998147ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc998159950 0x7fc998147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 -- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc99806c810 msgr2=0x7fc9980c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc99806c810 0x7fc9980c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99d23d640 1 -- 172.21.15.29:0/4202767828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940094d0 con 0x7fc998070450 2023-10-16T15:50:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.324+0000 7fc99da3e640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc99806c810 0x7fc9980c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.325+0000 7fc99d23d640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 0x7fc9980a8480 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fc98800c450 tx=0x7fc98800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.325+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9880052d0 con 0x7fc998070450 2023-10-16T15:50:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.325+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc988005430 con 0x7fc998070450 2023-10-16T15:50:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.325+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988012870 con 0x7fc998070450 2023-10-16T15:50:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.325+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9980c35a0 con 0x7fc998070450 2023-10-16T15:50:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.325+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9980c3b60 con 0x7fc998070450 2023-10-16T15:50:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.326+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9980c4080 con 0x7fc998070450 2023-10-16T15:50:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.327+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9880129d0 con 0x7fc998070450 2023-10-16T15:50:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.328+0000 7fc98e7fc640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc97c06af90 0x7fc97c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.328+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc988089160 con 0x7fc998070450 2023-10-16T15:50:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.329+0000 7fc99da3e640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc97c06af90 0x7fc97c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.330+0000 7fc99da3e640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc97c06af90 0x7fc97c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc994036000 tx=0x7fc99403c040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.332+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9980c4080 con 0x7fc998070450 2023-10-16T15:50:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.518+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc99815aef0 con 0x7fc998070450 2023-10-16T15:50:45.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.522+0000 7fc98e7fc640 1 -- 172.21.15.29:0/4202767828 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16636 (secure 0 0 0) 0x7fc99815aef0 con 0x7fc998070450 2023-10-16T15:50:45.524 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:45.524 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":208,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:44.088018+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc97c06af90 msgr2=0x7fc97c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc97c06af90 0x7fc97c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc994036000 tx=0x7fc99403c040 comp rx=0 tx=0).stop 2023-10-16T15:50:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 msgr2=0x7fc9980a8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 0x7fc9980a8480 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fc98800c450 tx=0x7fc98800c910 comp rx=0 tx=0).stop 2023-10-16T15:50:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 shutdown_connections 2023-10-16T15:50:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fc97c06af90 0x7fc97c06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc998070450 0x7fc9980a8480 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc998159950 0x7fc998147ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 --2- 172.21.15.29:0/4202767828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc99806c810 0x7fc9980c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 >> 172.21.15.29:0/4202767828 conn(0x7fc9980ce6a0 msgr2=0x7fc99815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:45.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 shutdown_connections 2023-10-16T15:50:45.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.525+0000 7fc99ea40640 1 -- 172.21.15.29:0/4202767828 wait complete. 2023-10-16T15:50:45.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T15:50:45.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.770+0000 7f5925fb8640 1 Processor -- start 2023-10-16T15:50:45.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.770+0000 7f5925fb8640 1 -- start start 2023-10-16T15:50:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.772+0000 7f5925fb8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59200ad970 0x7f59200cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.772+0000 7f5925fb8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59200ccfb0 0x7f59200cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.772+0000 7f5925fb8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592015b9d0 con 0x7f592006c040 2023-10-16T15:50:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.772+0000 7f5925fb8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592015bcb0 con 0x7f59200ad970 2023-10-16T15:50:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.772+0000 7f5925fb8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592015bf90 con 0x7f59200ccfb0 2023-10-16T15:50:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.772+0000 7f5917fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59200ad970 0x7f59200cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f5917fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59200ad970 0x7f59200cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52240/0 (socket says 172.21.15.29:52240) 2023-10-16T15:50:45.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f5917fff640 1 -- 172.21.15.29:0/1751091579 learned_addr learned my addr 172.21.15.29:0/1751091579 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:45.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f59257b7640 1 --2- 172.21.15.29:0/1751091579 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59200ccfb0 0x7f59200cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f5917fff640 1 -- 172.21.15.29:0/1751091579 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59200ccfb0 msgr2=0x7f59200cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f5917fff640 1 --2- 172.21.15.29:0/1751091579 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59200ccfb0 0x7f59200cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f5917fff640 1 -- 172.21.15.29:0/1751091579 >> v1:172.21.15.170:6789/0 conn(0x7f592006c040 legacy=0x7f592006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.773+0000 7f5917fff640 1 -- 172.21.15.29:0/1751091579 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200cc270 con 0x7f59200ad970 2023-10-16T15:50:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.774+0000 7f5917fff640 1 --2- 172.21.15.29:0/1751091579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200cca70 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f5910009870 tx=0x7f591002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627568718a63f28c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:45.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.774+0000 7f59177fe640 1 -- 172.21.15.29:0/1751091579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910031640 con 0x7f59200ad970 2023-10-16T15:50:45.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.774+0000 7f59177fe640 1 -- 172.21.15.29:0/1751091579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59100317a0 con 0x7f59200ad970 2023-10-16T15:50:45.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.774+0000 7f59177fe640 1 -- 172.21.15.29:0/1751091579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910031b30 con 0x7f59200ad970 2023-10-16T15:50:45.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 -- 172.21.15.29:0/1751091579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 msgr2=0x7f59200cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 --2- 172.21.15.29:0/1751091579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200cca70 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f5910009870 tx=0x7f591002b8f0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 -- 172.21.15.29:0/1751091579 shutdown_connections 2023-10-16T15:50:45.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 --2- 172.21.15.29:0/1751091579 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59200ccfb0 0x7f59200cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 --2- 172.21.15.29:0/1751091579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200cca70 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 -- 172.21.15.29:0/1751091579 >> 172.21.15.29:0/1751091579 conn(0x7f59200ce6a0 msgr2=0x7f59200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:45.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 -- 172.21.15.29:0/1751091579 shutdown_connections 2023-10-16T15:50:45.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.775+0000 7f5925fb8640 1 -- 172.21.15.29:0/1751091579 wait complete. 2023-10-16T15:50:45.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.777+0000 7f5925fb8640 1 Processor -- start 2023-10-16T15:50:45.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.777+0000 7f5925fb8640 1 -- start start 2023-10-16T15:50:45.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5925fb8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f592006c040 0x7f59200b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5925fb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5925fb8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59200ccfb0 0x7f592006f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5925fb8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200b1060 con 0x7f59200ad970 2023-10-16T15:50:45.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5925fb8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200c4280 con 0x7f59200ccfb0 2023-10-16T15:50:45.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5925fb8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200afa80 con 0x7f592006c040 2023-10-16T15:50:45.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5917fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5917fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200c3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52246/0 (socket says 172.21.15.29:52246) 2023-10-16T15:50:45.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.778+0000 7f5917fff640 1 -- 172.21.15.29:0/2501453325 learned_addr learned my addr 172.21.15.29:0/2501453325 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f59257b7640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59200ccfb0 0x7f592006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5924fb6640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f592006c040 0x7f59200b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5917fff640 1 -- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f592006c040 msgr2=0x7f59200b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5917fff640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f592006c040 0x7f59200b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5917fff640 1 -- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59200ccfb0 msgr2=0x7f592006f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5917fff640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59200ccfb0 0x7f592006f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5917fff640 1 -- 172.21.15.29:0/2501453325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100094d0 con 0x7f59200ad970 2023-10-16T15:50:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f59257b7640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59200ccfb0 0x7f592006f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.779+0000 7f5917fff640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200c3d40 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f59100097c0 tx=0x7f59100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:45.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.780+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591003ad20 con 0x7f59200ad970 2023-10-16T15:50:45.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.780+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591003ae80 con 0x7f59200ad970 2023-10-16T15:50:45.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.780+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910044630 con 0x7f59200ad970 2023-10-16T15:50:45.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.780+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59200afd60 con 0x7f59200ad970 2023-10-16T15:50:45.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.780+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59200b0320 con 0x7f59200ad970 2023-10-16T15:50:45.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.781+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592006b330 con 0x7f59200ad970 2023-10-16T15:50:45.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.782+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f591002f050 con 0x7f59200ad970 2023-10-16T15:50:45.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.783+0000 7f59157fa640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f590c06af90 0x7f590c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:45.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.784+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f59100abac0 con 0x7f59200ad970 2023-10-16T15:50:45.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.784+0000 7f5924fb6640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f590c06af90 0x7f590c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:45.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.785+0000 7f5924fb6640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f590c06af90 0x7f590c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f591c003d90 tx=0x7f591c009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:45.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:45.788+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5910076800 con 0x7f59200ad970 2023-10-16T15:50:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.032+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f59200cc520 con 0x7f59200ad970 2023-10-16T15:50:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.677+0000 7f59157fa640 1 -- 172.21.15.29:0/2501453325 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v99) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5910035070 con 0x7f59200ad970 2023-10-16T15:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f590c06af90 msgr2=0x7f590c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f590c06af90 0x7f590c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f591c003d90 tx=0x7f591c009040 comp rx=0 tx=0).stop 2023-10-16T15:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 msgr2=0x7f59200c3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200c3d40 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f59100097c0 tx=0x7f59100046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 shutdown_connections 2023-10-16T15:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f590c06af90 0x7f590c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59200ad970 0x7f59200c3d40 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59200ccfb0 0x7f592006f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 --2- 172.21.15.29:0/2501453325 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f592006c040 0x7f59200b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 >> 172.21.15.29:0/2501453325 conn(0x7f59200ce6a0 msgr2=0x7f592006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 shutdown_connections 2023-10-16T15:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.682+0000 7f5925fb8640 1 -- 172.21.15.29:0/2501453325 wait complete. 2023-10-16T15:50:46.706 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:50:46.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T15:50:46.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:50:46.870 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:50:46.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.932+0000 7f2a53410640 1 Processor -- start 2023-10-16T15:50:46.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.932+0000 7f2a53410640 1 -- start start 2023-10-16T15:50:46.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.933+0000 7f2a53410640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2a4c15ad70 0x7f2a4c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:46.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.933+0000 7f2a53410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 0x7f2a4c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:46.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.933+0000 7f2a53410640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c0cb030 con 0x7f2a4c0ca3b0 2023-10-16T15:50:46.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.933+0000 7f2a53410640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c91c0 con 0x7f2a4c15ad70 2023-10-16T15:50:46.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.933+0000 7f2a53410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c94a0 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.934+0000 7f2a509ab640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 0x7f2a4c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:46.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.934+0000 7f2a511ac640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2a4c15ad70 0x7f2a4c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:46.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.934+0000 7f2a511ac640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2a4c15ad70 0x7f2a4c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:34532/0 (socket says 172.21.15.29:34532) 2023-10-16T15:50:46.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.934+0000 7f2a509ab640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 0x7f2a4c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:43770/0 (socket says 172.21.15.29:43770) 2023-10-16T15:50:46.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.934+0000 7f2a511ac640 1 -- 172.21.15.29:0/1260650924 learned_addr learned my addr 172.21.15.29:0/1260650924 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:46.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.934+0000 7f2a519ad640 1 --1- 172.21.15.29:0/1260650924 >> v1:172.21.15.170:6789/0 conn(0x7f2a4c0ca3b0 0x7f2a4c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:45394/0 (socket says 172.21.15.29:45394) 2023-10-16T15:50:46.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.935+0000 7f2a511ac640 1 -- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 msgr2=0x7f2a4c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:46.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.935+0000 7f2a511ac640 1 --2- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 0x7f2a4c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.935+0000 7f2a511ac640 1 -- 172.21.15.29:0/1260650924 >> v1:172.21.15.170:6789/0 conn(0x7f2a4c0ca3b0 legacy=0x7f2a4c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:46.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.935+0000 7f2a43fff640 1 -- 172.21.15.29:0/1260650924 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695078506 0 0) 0x7f2a4c0cb030 con 0x7f2a4c0ca3b0 2023-10-16T15:50:46.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.935+0000 7f2a511ac640 1 -- 172.21.15.29:0/1260650924 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c0c9780 con 0x7f2a4c15ad70 2023-10-16T15:50:46.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.935+0000 7f2a509ab640 1 --2- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 0x7f2a4c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:50:46.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.936+0000 7f2a511ac640 1 --2- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c15ad70 0x7f2a4c069da0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009870 tx=0x7f2a4802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd667716ae92011e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:46.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.936+0000 7f2a43fff640 1 -- 172.21.15.29:0/1260650924 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48030840 con 0x7f2a4c15ad70 2023-10-16T15:50:46.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.936+0000 7f2a43fff640 1 -- 172.21.15.29:0/1260650924 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a480309a0 con 0x7f2a4c15ad70 2023-10-16T15:50:46.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.936+0000 7f2a43fff640 1 -- 172.21.15.29:0/1260650924 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48030d30 con 0x7f2a4c15ad70 2023-10-16T15:50:46.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 -- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c15ad70 msgr2=0x7f2a4c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:46.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 --2- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c15ad70 0x7f2a4c069da0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009870 tx=0x7f2a4802b980 comp rx=0 tx=0).stop 2023-10-16T15:50:46.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 -- 172.21.15.29:0/1260650924 shutdown_connections 2023-10-16T15:50:46.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 --2- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c0c9ca0 0x7f2a4c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 --2- 172.21.15.29:0/1260650924 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c15ad70 0x7f2a4c069da0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 -- 172.21.15.29:0/1260650924 >> 172.21.15.29:0/1260650924 conn(0x7f2a4c0ce9b0 msgr2=0x7f2a4c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:46.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 -- 172.21.15.29:0/1260650924 shutdown_connections 2023-10-16T15:50:46.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.937+0000 7f2a53410640 1 -- 172.21.15.29:0/1260650924 wait complete. 2023-10-16T15:50:46.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.939+0000 7f2a53410640 1 Processor -- start 2023-10-16T15:50:46.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.939+0000 7f2a53410640 1 -- start start 2023-10-16T15:50:46.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.940+0000 7f2a53410640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 0x7f2a4c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:46.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.940+0000 7f2a53410640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0ca3b0 0x7f2a4c0c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:46.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.940+0000 7f2a53410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c15ad70 0x7f2a4c0c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.940+0000 7f2a53410640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c8430 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.940+0000 7f2a53410640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c78b0 con 0x7f2a4c0ca3b0 2023-10-16T15:50:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.940+0000 7f2a53410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c7b90 con 0x7f2a4c15ad70 2023-10-16T15:50:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 0x7f2a4c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 0x7f2a4c1591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52262/0 (socket says 172.21.15.29:52262) 2023-10-16T15:50:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 -- 172.21.15.29:0/200308057 learned_addr learned my addr 172.21.15.29:0/200308057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a509ab640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0ca3b0 0x7f2a4c0c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a519ad640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c15ad70 0x7f2a4c0c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 -- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c15ad70 msgr2=0x7f2a4c0c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c15ad70 0x7f2a4c0c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 -- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0ca3b0 msgr2=0x7f2a4c0c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0ca3b0 0x7f2a4c0c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.941+0000 7f2a511ac640 1 -- 172.21.15.29:0/200308057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c009400 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.942+0000 7f2a511ac640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 0x7f2a4c1591a0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f2a48000c00 tx=0x7f2a480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.942+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4803e070 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.942+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a48004030 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.942+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a480043e0 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.942+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a480094d0 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.942+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c0c8150 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.944+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4c0c2220 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.945+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a48042070 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.946+0000 7f2a41ffb640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3006af90 0x7f2a3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:46.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.946+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2a480ab4d0 con 0x7f2a4c0c9ca0 2023-10-16T15:50:46.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.947+0000 7f2a509ab640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3006af90 0x7f2a3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:46.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.948+0000 7f2a509ab640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3006af90 0x7f2a3006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0c3fb0 tx=0x7f2a3c009190 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:46.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:46.951+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2a4c0c2220 con 0x7f2a4c0c9ca0 2023-10-16T15:50:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:47.186+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2a4c0c2220 con 0x7f2a4c0c9ca0 2023-10-16T15:50:48.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.153+0000 7f2a41ffb640 1 -- 172.21.15.29:0/200308057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v100) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2a4c0c2220 con 0x7f2a4c0c9ca0 2023-10-16T15:50:48.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3006af90 msgr2=0x7f2a3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3006af90 0x7f2a3006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0c3fb0 tx=0x7f2a3c009190 comp rx=0 tx=0).stop 2023-10-16T15:50:48.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 msgr2=0x7f2a4c1591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 0x7f2a4c1591a0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f2a48000c00 tx=0x7f2a480046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 shutdown_connections 2023-10-16T15:50:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2a3006af90 0x7f2a3006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a4c15ad70 0x7f2a4c0c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a4c0ca3b0 0x7f2a4c0c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 --2- 172.21.15.29:0/200308057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c0c9ca0 0x7f2a4c1591a0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 >> 172.21.15.29:0/200308057 conn(0x7f2a4c0ce9b0 msgr2=0x7f2a4c0cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:48.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 shutdown_connections 2023-10-16T15:50:48.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.157+0000 7f2a53410640 1 -- 172.21.15.29:0/200308057 wait complete. 2023-10-16T15:50:48.182 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T15:50:48.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.412+0000 7f398b8ec640 1 Processor -- start 2023-10-16T15:50:48.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.412+0000 7f398b8ec640 1 -- start start 2023-10-16T15:50:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f398b8ec640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f398415ae80 0x7f398415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f398b8ec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39840ce390 0x7f3984070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f398b8ec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398415ed60 con 0x7f398415aaa0 2023-10-16T15:50:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f398b8ec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398406b810 con 0x7f39840ce390 2023-10-16T15:50:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f398b8ec640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840707e0 con 0x7f398415ae80 2023-10-16T15:50:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f3989e89640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39840ce390 0x7f3984070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f3989e89640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f39840ce390 0x7f3984070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52272/0 (socket says 172.21.15.29:52272) 2023-10-16T15:50:48.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.414+0000 7f3989e89640 1 -- 172.21.15.29:0/1359415505 learned_addr learned my addr 172.21.15.29:0/1359415505 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:48.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.415+0000 7f3988e87640 1 --2- 172.21.15.29:0/1359415505 >> v2:172.21.15.170:3300/0 conn(0x7f398415ae80 0x7f398415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.415+0000 7f3989e89640 1 -- 172.21.15.29:0/1359415505 >> v2:172.21.15.170:3300/0 conn(0x7f398415ae80 msgr2=0x7f398415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:48.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.415+0000 7f3989e89640 1 --2- 172.21.15.29:0/1359415505 >> v2:172.21.15.170:3300/0 conn(0x7f398415ae80 0x7f398415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.415+0000 7f3989e89640 1 -- 172.21.15.29:0/1359415505 >> v1:172.21.15.29:6789/0 conn(0x7f398415aaa0 legacy=0x7f39840cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.415+0000 7f3989e89640 1 -- 172.21.15.29:0/1359415505 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398415ef90 con 0x7f39840ce390 2023-10-16T15:50:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.416+0000 7f3989e89640 1 --2- 172.21.15.29:0/1359415505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39840ce390 0x7f3984070f50 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f39840c59c0 tx=0x7f397c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0e69e9085d86ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.416+0000 7f397bfff640 1 -- 172.21.15.29:0/1359415505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c02cb20 con 0x7f39840ce390 2023-10-16T15:50:48.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f397bfff640 1 -- 172.21.15.29:0/1359415505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c02cc80 con 0x7f39840ce390 2023-10-16T15:50:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f397bfff640 1 -- 172.21.15.29:0/1359415505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c0343f0 con 0x7f39840ce390 2023-10-16T15:50:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 -- 172.21.15.29:0/1359415505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39840ce390 msgr2=0x7f3984070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 --2- 172.21.15.29:0/1359415505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39840ce390 0x7f3984070f50 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f39840c59c0 tx=0x7f397c02bf80 comp rx=0 tx=0).stop 2023-10-16T15:50:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 -- 172.21.15.29:0/1359415505 shutdown_connections 2023-10-16T15:50:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 --2- 172.21.15.29:0/1359415505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39840ce390 0x7f3984070f50 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 --2- 172.21.15.29:0/1359415505 >> v2:172.21.15.170:3300/0 conn(0x7f398415ae80 0x7f398415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 -- 172.21.15.29:0/1359415505 >> 172.21.15.29:0/1359415505 conn(0x7f39840cb410 msgr2=0x7f39840cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 -- 172.21.15.29:0/1359415505 shutdown_connections 2023-10-16T15:50:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.417+0000 7f398b8ec640 1 -- 172.21.15.29:0/1359415505 wait complete. 2023-10-16T15:50:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.419+0000 7f398b8ec640 1 Processor -- start 2023-10-16T15:50:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.419+0000 7f398b8ec640 1 -- start start 2023-10-16T15:50:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.420+0000 7f398b8ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39840ce390 0x7f39840b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.420+0000 7f398b8ec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398415aaa0 0x7f398414a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.420+0000 7f398b8ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 0x7f3984151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.420+0000 7f398b8ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840c30a0 con 0x7f398415ae80 2023-10-16T15:50:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.420+0000 7f398b8ec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398406d560 con 0x7f398415aaa0 2023-10-16T15:50:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.420+0000 7f398b8ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39840c2490 con 0x7f39840ce390 2023-10-16T15:50:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 0x7f3984151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 0x7f3984151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52274/0 (socket says 172.21.15.29:52274) 2023-10-16T15:50:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 -- 172.21.15.29:0/779990872 learned_addr learned my addr 172.21.15.29:0/779990872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3988e87640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398415aaa0 0x7f398414a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989688640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39840ce390 0x7f39840b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 -- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39840ce390 msgr2=0x7f39840b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39840ce390 0x7f39840b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 -- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398415aaa0 msgr2=0x7f398414a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398415aaa0 0x7f398414a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 -- 172.21.15.29:0/779990872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c009b30 con 0x7f398415ae80 2023-10-16T15:50:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.421+0000 7f3989e89640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 0x7f3984151e10 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f397c036040 tx=0x7f397c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.422+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c009510 con 0x7f398415ae80 2023-10-16T15:50:48.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.422+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c02c920 con 0x7f398415ae80 2023-10-16T15:50:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.422+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39840c2770 con 0x7f398415ae80 2023-10-16T15:50:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.422+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c03e690 con 0x7f398415ae80 2023-10-16T15:50:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.422+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39840c2d30 con 0x7f398415ae80 2023-10-16T15:50:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.423+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398404e160 con 0x7f398415ae80 2023-10-16T15:50:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.424+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f397c03e7f0 con 0x7f398415ae80 2023-10-16T15:50:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.425+0000 7f3979ffb640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f396806af90 0x7f396806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.425+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f397c0ac2d0 con 0x7f398415ae80 2023-10-16T15:50:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.426+0000 7f3989688640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f396806af90 0x7f396806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.427+0000 7f3989688640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f396806af90 0x7f396806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3984152970 tx=0x7f3980009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:48.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.429+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f397c077260 con 0x7f398415ae80 2023-10-16T15:50:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.660+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39840cb000 con 0x7f398415ae80 2023-10-16T15:50:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.661+0000 7f3979ffb640 1 -- 172.21.15.29:0/779990872 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f397c07a880 con 0x7f398415ae80 2023-10-16T15:50:48.663 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:48.663 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:50:44.088346+0000","modified":"2023-10-16T15:50:47.188096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8407":{"gid":8407,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/2149952636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2149952636},{"type":"v1","addr":"172.21.15.29:6833","nonce":2149952636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T15:50:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-16T15:50:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f396806af90 msgr2=0x7f396806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f396806af90 0x7f396806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3984152970 tx=0x7f3980009040 comp rx=0 tx=0).stop 2023-10-16T15:50:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 msgr2=0x7f3984151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 0x7f3984151e10 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f397c036040 tx=0x7f397c004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 shutdown_connections 2023-10-16T15:50:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f396806af90 0x7f396806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f398415ae80 0x7f3984151e10 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f398415aaa0 0x7f398414a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 --2- 172.21.15.29:0/779990872 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39840ce390 0x7f39840b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 >> 172.21.15.29:0/779990872 conn(0x7f39840cb410 msgr2=0x7f398414e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 shutdown_connections 2023-10-16T15:50:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.664+0000 7f398b8ec640 1 -- 172.21.15.29:0/779990872 wait complete. 2023-10-16T15:50:48.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:50:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.913+0000 7f1463111640 1 Processor -- start 2023-10-16T15:50:48.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.913+0000 7f1463111640 1 -- start start 2023-10-16T15:50:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.914+0000 7f1463111640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f145c06c040 0x7f145c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.914+0000 7f1463111640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f145c0cda80 0x7f145c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.914+0000 7f1463111640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c06c960 con 0x7f145c0ad970 2023-10-16T15:50:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.914+0000 7f1463111640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145c15b9d0 con 0x7f145c0cda80 2023-10-16T15:50:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.914+0000 7f1463111640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145c15bcb0 con 0x7f145c06c040 2023-10-16T15:50:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f145c0cda80 0x7f145c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f145c0cda80 0x7f145c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52284/0 (socket says 172.21.15.29:52284) 2023-10-16T15:50:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 -- 172.21.15.29:0/2944375773 learned_addr learned my addr 172.21.15.29:0/2944375773 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f146210f640 1 --2- 172.21.15.29:0/2944375773 >> v2:172.21.15.170:3300/0 conn(0x7f145c06c040 0x7f145c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 -- 172.21.15.29:0/2944375773 >> v2:172.21.15.170:3300/0 conn(0x7f145c06c040 msgr2=0x7f145c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 --2- 172.21.15.29:0/2944375773 >> v2:172.21.15.170:3300/0 conn(0x7f145c06c040 0x7f145c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 -- 172.21.15.29:0/2944375773 >> v1:172.21.15.170:6789/0 conn(0x7f145c0ad970 legacy=0x7f145c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:48.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.915+0000 7f1462910640 1 -- 172.21.15.29:0/2944375773 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c15bf90 con 0x7f145c0cda80 2023-10-16T15:50:48.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.917+0000 7f1462910640 1 --2- 172.21.15.29:0/2944375773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c0cda80 0x7f145c15ed90 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f145c0cf350 tx=0x7f145402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1328c663674e9891 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:48.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.917+0000 7f146110d640 1 -- 172.21.15.29:0/2944375773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145402ca80 con 0x7f145c0cda80 2023-10-16T15:50:48.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.917+0000 7f146110d640 1 -- 172.21.15.29:0/2944375773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145402cbe0 con 0x7f145c0cda80 2023-10-16T15:50:48.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.917+0000 7f146110d640 1 -- 172.21.15.29:0/2944375773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14540343f0 con 0x7f145c0cda80 2023-10-16T15:50:48.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 -- 172.21.15.29:0/2944375773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c0cda80 msgr2=0x7f145c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 --2- 172.21.15.29:0/2944375773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c0cda80 0x7f145c15ed90 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f145c0cf350 tx=0x7f145402bee0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 -- 172.21.15.29:0/2944375773 shutdown_connections 2023-10-16T15:50:48.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 --2- 172.21.15.29:0/2944375773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c0cda80 0x7f145c15ed90 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 --2- 172.21.15.29:0/2944375773 >> v2:172.21.15.170:3300/0 conn(0x7f145c06c040 0x7f145c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 -- 172.21.15.29:0/2944375773 >> 172.21.15.29:0/2944375773 conn(0x7f145c0ce6a0 msgr2=0x7f145c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:48.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 -- 172.21.15.29:0/2944375773 shutdown_connections 2023-10-16T15:50:48.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.918+0000 7f1463111640 1 -- 172.21.15.29:0/2944375773 wait complete. 2023-10-16T15:50:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.920+0000 7f1463111640 1 Processor -- start 2023-10-16T15:50:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.920+0000 7f1463111640 1 -- start start 2023-10-16T15:50:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f1463111640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 0x7f145c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f1463111640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f145c0ad970 0x7f145c150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f1463111640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f145c0cda80 0x7f145c0c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f1463111640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0c8cf0 con 0x7f145c06c040 2023-10-16T15:50:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f1463111640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0c9870 con 0x7f145c0cda80 2023-10-16T15:50:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f1463111640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0c8f40 con 0x7f145c0ad970 2023-10-16T15:50:48.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f146210f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 0x7f145c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f146210f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 0x7f145c0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52288/0 (socket says 172.21.15.29:52288) 2023-10-16T15:50:48.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.921+0000 7f146210f640 1 -- 172.21.15.29:0/2811606868 learned_addr learned my addr 172.21.15.29:0/2811606868 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:48.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f146210f640 1 -- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f145c0ad970 msgr2=0x7f145c150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f146210f640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f145c0ad970 0x7f145c150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f146210f640 1 -- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f145c0cda80 msgr2=0x7f145c0c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f1462910640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f145c0cda80 0x7f145c0c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f146210f640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f145c0cda80 0x7f145c0c6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f146210f640 1 -- 172.21.15.29:0/2811606868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1454009b30 con 0x7f145c06c040 2023-10-16T15:50:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f146210f640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 0x7f145c0c74a0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f145800d620 tx=0x7f145800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800b620 con 0x7f145c06c040 2023-10-16T15:50:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800b780 con 0x7f145c06c040 2023-10-16T15:50:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800bb30 con 0x7f145c06c040 2023-10-16T15:50:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.922+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c0c9220 con 0x7f145c06c040 2023-10-16T15:50:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.923+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c150570 con 0x7f145c06c040 2023-10-16T15:50:48.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.924+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145c06b330 con 0x7f145c06c040 2023-10-16T15:50:48.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.925+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f145800bc90 con 0x7f145c06c040 2023-10-16T15:50:48.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.926+0000 7f1452ffd640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f144006af90 0x7f144006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:48.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.926+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1458089ce0 con 0x7f145c06c040 2023-10-16T15:50:48.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.926+0000 7f146190e640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f144006af90 0x7f144006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:48.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.927+0000 7f146190e640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f144006af90 0x7f144006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f145c0c48d0 tx=0x7f144c0079b0 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:48.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:48.931+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1458054c50 con 0x7f145c06c040 2023-10-16T15:50:49.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.123+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f145c04e9b0 con 0x7f145c06c040 2023-10-16T15:50:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.127+0000 7f1452ffd640 1 -- 172.21.15.29:0/2811606868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16623 (secure 0 0 0) 0x7f1458058270 con 0x7f145c06c040 2023-10-16T15:50:49.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:49.128 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":209,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:50:45.655430+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:50:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f144006af90 msgr2=0x7f144006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f144006af90 0x7f144006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f145c0c48d0 tx=0x7f144c0079b0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 msgr2=0x7f145c0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 0x7f145c0c74a0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f145800d620 tx=0x7f145800dae0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 shutdown_connections 2023-10-16T15:50:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f144006af90 0x7f144006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f145c0cda80 0x7f145c0c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f145c0ad970 0x7f145c150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 --2- 172.21.15.29:0/2811606868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f145c06c040 0x7f145c0c74a0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.130+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 >> 172.21.15.29:0/2811606868 conn(0x7f145c0ce6a0 msgr2=0x7f145c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.131+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 shutdown_connections 2023-10-16T15:50:49.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.131+0000 7f1463111640 1 -- 172.21.15.29:0/2811606868 wait complete. 2023-10-16T15:50:49.153 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:50:49.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:50:49.166 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-16T15:50:49.165+0000 7f4a5e447640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:50:49.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.382+0000 7f2faf483640 1 Processor -- start 2023-10-16T15:50:49.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.382+0000 7f2faf483640 1 -- start start 2023-10-16T15:50:49.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2faf483640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fa806c040 0x7f2fa806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2faf483640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2fa80ad970 0x7f2fa815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2faf483640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa806c960 con 0x7f2fa815ebc0 2023-10-16T15:50:49.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2faf483640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa815b9d0 con 0x7f2fa806c040 2023-10-16T15:50:49.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2faf483640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa815bcb0 con 0x7f2fa80ad970 2023-10-16T15:50:49.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2fae481640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fa806c040 0x7f2fa806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2fae481640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fa806c040 0x7f2fa806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52292/0 (socket says 172.21.15.29:52292) 2023-10-16T15:50:49.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.384+0000 7f2fae481640 1 -- 172.21.15.29:0/1034921009 learned_addr learned my addr 172.21.15.29:0/1034921009 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:49.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.385+0000 7f2fadc80640 1 --2- 172.21.15.29:0/1034921009 >> v2:172.21.15.170:3300/0 conn(0x7f2fa80ad970 0x7f2fa815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.385+0000 7f2fae481640 1 -- 172.21.15.29:0/1034921009 >> v2:172.21.15.170:3300/0 conn(0x7f2fa80ad970 msgr2=0x7f2fa815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:49.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.385+0000 7f2fae481640 1 --2- 172.21.15.29:0/1034921009 >> v2:172.21.15.170:3300/0 conn(0x7f2fa80ad970 0x7f2fa815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.385+0000 7f2fae481640 1 -- 172.21.15.29:0/1034921009 >> v1:172.21.15.29:6789/0 conn(0x7f2fa815ebc0 legacy=0x7f2fa80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.385+0000 7f2fae481640 1 -- 172.21.15.29:0/1034921009 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa815bf90 con 0x7f2fa806c040 2023-10-16T15:50:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.386+0000 7f2fae481640 1 --2- 172.21.15.29:0/1034921009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa806c040 0x7f2fa806c420 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80cb060 tx=0x7f2fa402b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a14155bb0d218 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.386+0000 7f2fad47f640 1 -- 172.21.15.29:0/1034921009 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4030880 con 0x7f2fa806c040 2023-10-16T15:50:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.386+0000 7f2fad47f640 1 -- 172.21.15.29:0/1034921009 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa40309e0 con 0x7f2fa806c040 2023-10-16T15:50:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.386+0000 7f2fad47f640 1 -- 172.21.15.29:0/1034921009 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4030d70 con 0x7f2fa806c040 2023-10-16T15:50:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 -- 172.21.15.29:0/1034921009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa806c040 msgr2=0x7f2fa806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 --2- 172.21.15.29:0/1034921009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa806c040 0x7f2fa806c420 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80cb060 tx=0x7f2fa402b9c0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 -- 172.21.15.29:0/1034921009 shutdown_connections 2023-10-16T15:50:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 --2- 172.21.15.29:0/1034921009 >> v2:172.21.15.170:3300/0 conn(0x7f2fa80ad970 0x7f2fa815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 --2- 172.21.15.29:0/1034921009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa806c040 0x7f2fa806c420 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 -- 172.21.15.29:0/1034921009 >> 172.21.15.29:0/1034921009 conn(0x7f2fa80ce6a0 msgr2=0x7f2fa80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 -- 172.21.15.29:0/1034921009 shutdown_connections 2023-10-16T15:50:49.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.387+0000 7f2faf483640 1 -- 172.21.15.29:0/1034921009 wait complete. 2023-10-16T15:50:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.389+0000 7f2faf483640 1 Processor -- start 2023-10-16T15:50:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.389+0000 7f2faf483640 1 -- start start 2023-10-16T15:50:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faf483640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fa806c040 0x7f2fa8146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faf483640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fa80ad970 0x7f2fa81490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faf483640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 0x7f2fa81495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faf483640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa81e5ea0 con 0x7f2fa815ebc0 2023-10-16T15:50:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faf483640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa81e4e40 con 0x7f2fa80ad970 2023-10-16T15:50:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faf483640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa81e50e0 con 0x7f2fa806c040 2023-10-16T15:50:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faec82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 0x7f2fa81495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faec82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 0x7f2fa81495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52306/0 (socket says 172.21.15.29:52306) 2023-10-16T15:50:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.390+0000 7f2faec82640 1 -- 172.21.15.29:0/1495182399 learned_addr learned my addr 172.21.15.29:0/1495182399 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2fae481640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fa806c040 0x7f2fa8146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faec82640 1 -- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fa806c040 msgr2=0x7f2fa8146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faec82640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fa806c040 0x7f2fa8146700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2fadc80640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fa80ad970 0x7f2fa81490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faec82640 1 -- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fa80ad970 msgr2=0x7f2fa81490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faec82640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fa80ad970 0x7f2fa81490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faec82640 1 -- 172.21.15.29:0/1495182399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40094d0 con 0x7f2fa815ebc0 2023-10-16T15:50:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faec82640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 0x7f2fa81495f0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c7a0 tx=0x7f2fa000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0003b70 con 0x7f2fa815ebc0 2023-10-16T15:50:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa0003cf0 con 0x7f2fa815ebc0 2023-10-16T15:50:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa000aa90 con 0x7f2fa815ebc0 2023-10-16T15:50:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.391+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa81e53c0 con 0x7f2fa815ebc0 2023-10-16T15:50:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.392+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa81e5980 con 0x7f2fa815ebc0 2023-10-16T15:50:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.393+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa806b330 con 0x7f2fa815ebc0 2023-10-16T15:50:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.394+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2fa000abf0 con 0x7f2fa815ebc0 2023-10-16T15:50:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.395+0000 7f2f9effd640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2f8c06af90 0x7f2f8c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.395+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2fa00896a0 con 0x7f2fa815ebc0 2023-10-16T15:50:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.395+0000 7f2fae481640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2f8c06af90 0x7f2f8c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.397+0000 7f2fae481640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2f8c06af90 0x7f2f8c06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4036000 tx=0x7f2fa403b040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:49.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.399+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2fa0054580 con 0x7f2fa815ebc0 2023-10-16T15:50:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.592+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 --> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fa80cc550 con 0x7f2f8c06af90 2023-10-16T15:50:49.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.621+0000 7f2f9effd640 1 -- 172.21.15.29:0/1495182399 <== mgr.7406 v2:172.21.15.170:6832/780156504 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286893 (secure 0 0 0) 0x7f2fa4030bd0 con 0x7f2f8c06af90 2023-10-16T15:50:49.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:49.631 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:50:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2f8c06af90 msgr2=0x7f2f8c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2f8c06af90 0x7f2f8c06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4036000 tx=0x7f2fa403b040 comp rx=0 tx=0).stop 2023-10-16T15:50:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 msgr2=0x7f2fa81495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 0x7f2fa81495f0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c7a0 tx=0x7f2fa000cc60 comp rx=0 tx=0).stop 2023-10-16T15:50:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 shutdown_connections 2023-10-16T15:50:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f2f8c06af90 0x7f2f8c06d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fa815ebc0 0x7f2fa81495f0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2fa80ad970 0x7f2fa81490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 --2- 172.21.15.29:0/1495182399 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2fa806c040 0x7f2fa8146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 >> 172.21.15.29:0/1495182399 conn(0x7f2fa80ce6a0 msgr2=0x7f2fa80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 shutdown_connections 2023-10-16T15:50:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.629+0000 7f2faf483640 1 -- 172.21.15.29:0/1495182399 wait complete. 2023-10-16T15:50:49.653 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T15:50:49.481386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":587,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":387688,"kb_used_data":21612,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754587032,"statfs":{"total":773094113280,"available":772697120768,"internally_reserved":0,"allocated":22130688,"data_stored":11411732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":374722105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002198"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674885+0000","last_change":"2023-10-16T15:50:44.660288+0000","last_active":"2023-10-16T15:50:45.674885+0000","last_peered":"2023-10-16T15:50:45.674885+0000","last_clean":"2023-10-16T15:50:45.674885+0000","last_became_active":"2023-10-16T15:50:44.659892+0000","last_became_peered":"2023-10-16T15:50:44.659892+0000","last_unstale":"2023-10-16T15:50:45.674885+0000","last_undegraded":"2023-10-16T15:50:45.674885+0000","last_fullsized":"2023-10-16T15:50:45.674885+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:48.194990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.670249+0000","last_change":"2023-10-16T15:50:44.666779+0000","last_active":"2023-10-16T15:50:44.670249+0000","last_peered":"2023-10-16T15:50:44.670249+0000","last_clean":"2023-10-16T15:50:44.670249+0000","last_became_active":"2023-10-16T15:50:44.666425+0000","last_became_peered":"2023-10-16T15:50:44.666425+0000","last_unstale":"2023-10-16T15:50:44.670249+0000","last_undegraded":"2023-10-16T15:50:44.670249+0000","last_fullsized":"2023-10-16T15:50:44.670249+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:33.048475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.677174+0000","last_change":"2023-10-16T15:50:44.674837+0000","last_active":"2023-10-16T15:50:44.677174+0000","last_peered":"2023-10-16T15:50:44.677174+0000","last_clean":"2023-10-16T15:50:44.677174+0000","last_became_active":"2023-10-16T15:50:44.674442+0000","last_became_peered":"2023-10-16T15:50:44.674442+0000","last_unstale":"2023-10-16T15:50:44.677174+0000","last_undegraded":"2023-10-16T15:50:44.677174+0000","last_fullsized":"2023-10-16T15:50:44.677174+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:02.605604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681564+0000","last_change":"2023-10-16T15:50:44.677894+0000","last_active":"2023-10-16T15:50:44.681564+0000","last_peered":"2023-10-16T15:50:44.681564+0000","last_clean":"2023-10-16T15:50:44.681564+0000","last_became_active":"2023-10-16T15:50:44.674402+0000","last_became_peered":"2023-10-16T15:50:44.674402+0000","last_unstale":"2023-10-16T15:50:44.681564+0000","last_undegraded":"2023-10-16T15:50:44.681564+0000","last_fullsized":"2023-10-16T15:50:44.681564+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:36.308210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.682108+0000","last_change":"2023-10-16T15:50:44.677119+0000","last_active":"2023-10-16T15:50:44.682108+0000","last_peered":"2023-10-16T15:50:44.682108+0000","last_clean":"2023-10-16T15:50:44.682108+0000","last_became_active":"2023-10-16T15:50:44.676859+0000","last_became_peered":"2023-10-16T15:50:44.676859+0000","last_unstale":"2023-10-16T15:50:44.682108+0000","last_undegraded":"2023-10-16T15:50:44.682108+0000","last_fullsized":"2023-10-16T15:50:44.682108+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:59.710850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.670801+0000","last_change":"2023-10-16T15:50:44.666098+0000","last_active":"2023-10-16T15:50:44.670801+0000","last_peered":"2023-10-16T15:50:44.670801+0000","last_clean":"2023-10-16T15:50:44.670801+0000","last_became_active":"2023-10-16T15:50:44.665803+0000","last_became_peered":"2023-10-16T15:50:44.665803+0000","last_unstale":"2023-10-16T15:50:44.670801+0000","last_undegraded":"2023-10-16T15:50:44.670801+0000","last_fullsized":"2023-10-16T15:50:44.670801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:56.322845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674204+0000","last_change":"2023-10-16T15:50:44.678699+0000","last_active":"2023-10-16T15:50:45.674204+0000","last_peered":"2023-10-16T15:50:45.674204+0000","last_clean":"2023-10-16T15:50:45.674204+0000","last_became_active":"2023-10-16T15:50:44.675445+0000","last_became_peered":"2023-10-16T15:50:44.675445+0000","last_unstale":"2023-10-16T15:50:45.674204+0000","last_undegraded":"2023-10-16T15:50:45.674204+0000","last_fullsized":"2023-10-16T15:50:45.674204+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:45.077152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668779+0000","last_change":"2023-10-16T15:50:44.673655+0000","last_active":"2023-10-16T15:50:45.668779+0000","last_peered":"2023-10-16T15:50:45.668779+0000","last_clean":"2023-10-16T15:50:45.668779+0000","last_became_active":"2023-10-16T15:50:44.673305+0000","last_became_peered":"2023-10-16T15:50:44.673305+0000","last_unstale":"2023-10-16T15:50:45.668779+0000","last_undegraded":"2023-10-16T15:50:45.668779+0000","last_fullsized":"2023-10-16T15:50:45.668779+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:05.676537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681284+0000","last_change":"2023-10-16T15:50:44.676168+0000","last_active":"2023-10-16T15:50:44.681284+0000","last_peered":"2023-10-16T15:50:44.681284+0000","last_clean":"2023-10-16T15:50:44.681284+0000","last_became_active":"2023-10-16T15:50:44.675859+0000","last_became_peered":"2023-10-16T15:50:44.675859+0000","last_unstale":"2023-10-16T15:50:44.681284+0000","last_undegraded":"2023-10-16T15:50:44.681284+0000","last_fullsized":"2023-10-16T15:50:44.681284+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:04.368424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477057+0000","last_change":"2023-10-16T15:50:44.659114+0000","last_active":"2023-10-16T15:50:46.477057+0000","last_peered":"2023-10-16T15:50:46.477057+0000","last_clean":"2023-10-16T15:50:46.477057+0000","last_became_active":"2023-10-16T15:50:44.658823+0000","last_became_peered":"2023-10-16T15:50:44.658823+0000","last_unstale":"2023-10-16T15:50:46.477057+0000","last_undegraded":"2023-10-16T15:50:46.477057+0000","last_fullsized":"2023-10-16T15:50:46.477057+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:48.955244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675563+0000","last_change":"2023-10-16T15:50:44.677048+0000","last_active":"2023-10-16T15:50:45.675563+0000","last_peered":"2023-10-16T15:50:45.675563+0000","last_clean":"2023-10-16T15:50:45.675563+0000","last_became_active":"2023-10-16T15:50:44.676774+0000","last_became_peered":"2023-10-16T15:50:44.676774+0000","last_unstale":"2023-10-16T15:50:45.675563+0000","last_undegraded":"2023-10-16T15:50:45.675563+0000","last_fullsized":"2023-10-16T15:50:45.675563+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:27.820962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681544+0000","last_change":"2023-10-16T15:50:44.675010+0000","last_active":"2023-10-16T15:50:44.681544+0000","last_peered":"2023-10-16T15:50:44.681544+0000","last_clean":"2023-10-16T15:50:44.681544+0000","last_became_active":"2023-10-16T15:50:44.673988+0000","last_became_peered":"2023-10-16T15:50:44.673988+0000","last_unstale":"2023-10-16T15:50:44.681544+0000","last_undegraded":"2023-10-16T15:50:44.681544+0000","last_fullsized":"2023-10-16T15:50:44.681544+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:42.567049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684087+0000","last_change":"2023-10-16T15:50:44.683501+0000","last_active":"2023-10-16T15:50:44.684087+0000","last_peered":"2023-10-16T15:50:44.684087+0000","last_clean":"2023-10-16T15:50:44.684087+0000","last_became_active":"2023-10-16T15:50:44.682120+0000","last_became_peered":"2023-10-16T15:50:44.682120+0000","last_unstale":"2023-10-16T15:50:44.684087+0000","last_undegraded":"2023-10-16T15:50:44.684087+0000","last_fullsized":"2023-10-16T15:50:44.684087+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:41.682891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.669744+0000","last_change":"2023-10-16T15:50:44.663126+0000","last_active":"2023-10-16T15:50:44.669744+0000","last_peered":"2023-10-16T15:50:44.669744+0000","last_clean":"2023-10-16T15:50:44.669744+0000","last_became_active":"2023-10-16T15:50:44.662838+0000","last_became_peered":"2023-10-16T15:50:44.662838+0000","last_unstale":"2023-10-16T15:50:44.669744+0000","last_undegraded":"2023-10-16T15:50:44.669744+0000","last_fullsized":"2023-10-16T15:50:44.669744+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:05.143314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680107+0000","last_change":"2023-10-16T15:50:44.671958+0000","last_active":"2023-10-16T15:50:44.680107+0000","last_peered":"2023-10-16T15:50:44.680107+0000","last_clean":"2023-10-16T15:50:44.680107+0000","last_became_active":"2023-10-16T15:50:44.671578+0000","last_became_peered":"2023-10-16T15:50:44.671578+0000","last_unstale":"2023-10-16T15:50:44.680107+0000","last_undegraded":"2023-10-16T15:50:44.680107+0000","last_fullsized":"2023-10-16T15:50:44.680107+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:28.836451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668779+0000","last_change":"2023-10-16T15:50:44.667019+0000","last_active":"2023-10-16T15:50:45.668779+0000","last_peered":"2023-10-16T15:50:45.668779+0000","last_clean":"2023-10-16T15:50:45.668779+0000","last_became_active":"2023-10-16T15:50:44.666753+0000","last_became_peered":"2023-10-16T15:50:44.666753+0000","last_unstale":"2023-10-16T15:50:45.668779+0000","last_undegraded":"2023-10-16T15:50:45.668779+0000","last_fullsized":"2023-10-16T15:50:45.668779+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:09.444429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685232+0000","last_change":"2023-10-16T15:50:44.674962+0000","last_active":"2023-10-16T15:50:44.685232+0000","last_peered":"2023-10-16T15:50:44.685232+0000","last_clean":"2023-10-16T15:50:44.685232+0000","last_became_active":"2023-10-16T15:50:44.674562+0000","last_became_peered":"2023-10-16T15:50:44.674562+0000","last_unstale":"2023-10-16T15:50:44.685232+0000","last_undegraded":"2023-10-16T15:50:44.685232+0000","last_fullsized":"2023-10-16T15:50:44.685232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:02.605604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675297+0000","last_change":"2023-10-16T15:50:44.666379+0000","last_active":"2023-10-16T15:50:45.675297+0000","last_peered":"2023-10-16T15:50:45.675297+0000","last_clean":"2023-10-16T15:50:45.675297+0000","last_became_active":"2023-10-16T15:50:44.665361+0000","last_became_peered":"2023-10-16T15:50:44.665361+0000","last_unstale":"2023-10-16T15:50:45.675297+0000","last_undegraded":"2023-10-16T15:50:45.675297+0000","last_fullsized":"2023-10-16T15:50:45.675297+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:13.733047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476428+0000","last_change":"2023-10-16T15:50:44.669691+0000","last_active":"2023-10-16T15:50:46.476428+0000","last_peered":"2023-10-16T15:50:46.476428+0000","last_clean":"2023-10-16T15:50:46.476428+0000","last_became_active":"2023-10-16T15:50:44.669408+0000","last_became_peered":"2023-10-16T15:50:44.669408+0000","last_unstale":"2023-10-16T15:50:46.476428+0000","last_undegraded":"2023-10-16T15:50:46.476428+0000","last_fullsized":"2023-10-16T15:50:46.476428+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:07.291215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685415+0000","last_change":"2023-10-16T15:50:44.684698+0000","last_active":"2023-10-16T15:50:44.685415+0000","last_peered":"2023-10-16T15:50:44.685415+0000","last_clean":"2023-10-16T15:50:44.685415+0000","last_became_active":"2023-10-16T15:50:44.683606+0000","last_became_peered":"2023-10-16T15:50:44.683606+0000","last_unstale":"2023-10-16T15:50:44.685415+0000","last_undegraded":"2023-10-16T15:50:44.685415+0000","last_fullsized":"2023-10-16T15:50:44.685415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:56.322845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.682088+0000","last_change":"2023-10-16T15:50:44.675425+0000","last_active":"2023-10-16T15:50:44.682088+0000","last_peered":"2023-10-16T15:50:44.682088+0000","last_clean":"2023-10-16T15:50:44.682088+0000","last_became_active":"2023-10-16T15:50:44.675138+0000","last_became_peered":"2023-10-16T15:50:44.675138+0000","last_unstale":"2023-10-16T15:50:44.682088+0000","last_undegraded":"2023-10-16T15:50:44.682088+0000","last_fullsized":"2023-10-16T15:50:44.682088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:02.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476231+0000","last_change":"2023-10-16T15:50:44.660710+0000","last_active":"2023-10-16T15:50:46.476231+0000","last_peered":"2023-10-16T15:50:46.476231+0000","last_clean":"2023-10-16T15:50:46.476231+0000","last_became_active":"2023-10-16T15:50:44.660424+0000","last_became_peered":"2023-10-16T15:50:44.660424+0000","last_unstale":"2023-10-16T15:50:46.476231+0000","last_undegraded":"2023-10-16T15:50:46.476231+0000","last_fullsized":"2023-10-16T15:50:46.476231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:51.317016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.666593+0000","last_change":"2023-10-16T15:50:44.659302+0000","last_active":"2023-10-16T15:50:44.666593+0000","last_peered":"2023-10-16T15:50:44.666593+0000","last_clean":"2023-10-16T15:50:44.666593+0000","last_became_active":"2023-10-16T15:50:44.659003+0000","last_became_peered":"2023-10-16T15:50:44.659003+0000","last_unstale":"2023-10-16T15:50:44.666593+0000","last_undegraded":"2023-10-16T15:50:44.666593+0000","last_fullsized":"2023-10-16T15:50:44.666593+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:49.488183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668944+0000","last_change":"2023-10-16T15:50:44.656189+0000","last_active":"2023-10-16T15:50:45.668944+0000","last_peered":"2023-10-16T15:50:45.668944+0000","last_clean":"2023-10-16T15:50:45.668944+0000","last_became_active":"2023-10-16T15:50:44.655904+0000","last_became_peered":"2023-10-16T15:50:44.655904+0000","last_unstale":"2023-10-16T15:50:45.668944+0000","last_undegraded":"2023-10-16T15:50:45.668944+0000","last_fullsized":"2023-10-16T15:50:45.668944+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:32.326238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.683725+0000","last_change":"2023-10-16T15:50:44.682004+0000","last_active":"2023-10-16T15:50:44.683725+0000","last_peered":"2023-10-16T15:50:44.683725+0000","last_clean":"2023-10-16T15:50:44.683725+0000","last_became_active":"2023-10-16T15:50:44.681649+0000","last_became_peered":"2023-10-16T15:50:44.681649+0000","last_unstale":"2023-10-16T15:50:44.683725+0000","last_undegraded":"2023-10-16T15:50:44.683725+0000","last_fullsized":"2023-10-16T15:50:44.683725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:38.868853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.673916+0000","last_change":"2023-10-16T15:50:44.670319+0000","last_active":"2023-10-16T15:50:45.673916+0000","last_peered":"2023-10-16T15:50:45.673916+0000","last_clean":"2023-10-16T15:50:45.673916+0000","last_became_active":"2023-10-16T15:50:44.669950+0000","last_became_peered":"2023-10-16T15:50:44.669950+0000","last_unstale":"2023-10-16T15:50:45.673916+0000","last_undegraded":"2023-10-16T15:50:45.673916+0000","last_fullsized":"2023-10-16T15:50:45.673916+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:02.605604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.668160+0000","last_change":"2023-10-16T15:50:44.664547+0000","last_active":"2023-10-16T15:50:44.668160+0000","last_peered":"2023-10-16T15:50:44.668160+0000","last_clean":"2023-10-16T15:50:44.668160+0000","last_became_active":"2023-10-16T15:50:44.664254+0000","last_became_peered":"2023-10-16T15:50:44.664254+0000","last_unstale":"2023-10-16T15:50:44.668160+0000","last_undegraded":"2023-10-16T15:50:44.668160+0000","last_fullsized":"2023-10-16T15:50:44.668160+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:07.426576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.679498+0000","last_change":"2023-10-16T15:50:44.667825+0000","last_active":"2023-10-16T15:50:44.679498+0000","last_peered":"2023-10-16T15:50:44.679498+0000","last_clean":"2023-10-16T15:50:44.679498+0000","last_became_active":"2023-10-16T15:50:44.667512+0000","last_became_peered":"2023-10-16T15:50:44.667512+0000","last_unstale":"2023-10-16T15:50:44.679498+0000","last_undegraded":"2023-10-16T15:50:44.679498+0000","last_fullsized":"2023-10-16T15:50:44.679498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:38.915700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.673320+0000","last_change":"2023-10-16T15:50:44.668024+0000","last_active":"2023-10-16T15:50:44.673320+0000","last_peered":"2023-10-16T15:50:44.673320+0000","last_clean":"2023-10-16T15:50:44.673320+0000","last_became_active":"2023-10-16T15:50:44.667724+0000","last_became_peered":"2023-10-16T15:50:44.667724+0000","last_unstale":"2023-10-16T15:50:44.673320+0000","last_undegraded":"2023-10-16T15:50:44.673320+0000","last_fullsized":"2023-10-16T15:50:44.673320+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:36.097780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.672923+0000","last_change":"2023-10-16T15:50:44.670185+0000","last_active":"2023-10-16T15:50:44.672923+0000","last_peered":"2023-10-16T15:50:44.672923+0000","last_clean":"2023-10-16T15:50:44.672923+0000","last_became_active":"2023-10-16T15:50:44.667024+0000","last_became_peered":"2023-10-16T15:50:44.667024+0000","last_unstale":"2023-10-16T15:50:44.672923+0000","last_undegraded":"2023-10-16T15:50:44.672923+0000","last_fullsized":"2023-10-16T15:50:44.672923+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:10.952350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.676607+0000","last_change":"2023-10-16T15:50:44.674047+0000","last_active":"2023-10-16T15:50:44.676607+0000","last_peered":"2023-10-16T15:50:44.676607+0000","last_clean":"2023-10-16T15:50:44.676607+0000","last_became_active":"2023-10-16T15:50:44.673740+0000","last_became_peered":"2023-10-16T15:50:44.673740+0000","last_unstale":"2023-10-16T15:50:44.676607+0000","last_undegraded":"2023-10-16T15:50:44.676607+0000","last_fullsized":"2023-10-16T15:50:44.676607+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:02.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.678198+0000","last_change":"2023-10-16T15:50:44.666444+0000","last_active":"2023-10-16T15:50:44.678198+0000","last_peered":"2023-10-16T15:50:44.678198+0000","last_clean":"2023-10-16T15:50:44.678198+0000","last_became_active":"2023-10-16T15:50:44.666173+0000","last_became_peered":"2023-10-16T15:50:44.666173+0000","last_unstale":"2023-10-16T15:50:44.678198+0000","last_undegraded":"2023-10-16T15:50:44.678198+0000","last_fullsized":"2023-10-16T15:50:44.678198+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:07.003930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669718+0000","last_change":"2023-10-16T15:50:44.669259+0000","last_active":"2023-10-16T15:50:45.669718+0000","last_peered":"2023-10-16T15:50:45.669718+0000","last_clean":"2023-10-16T15:50:45.669718+0000","last_became_active":"2023-10-16T15:50:44.668971+0000","last_became_peered":"2023-10-16T15:50:44.668971+0000","last_unstale":"2023-10-16T15:50:45.669718+0000","last_undegraded":"2023-10-16T15:50:45.669718+0000","last_fullsized":"2023-10-16T15:50:45.669718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:59.675199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"208'4","reported_seq":15,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685159+0000","last_change":"2023-10-16T15:50:43.645418+0000","last_active":"2023-10-16T15:50:44.685159+0000","last_peered":"2023-10-16T15:50:44.685159+0000","last_clean":"2023-10-16T15:50:44.685159+0000","last_became_active":"2023-10-16T15:50:43.644948+0000","last_became_peered":"2023-10-16T15:50:43.644948+0000","last_unstale":"2023-10-16T15:50:44.685159+0000","last_undegraded":"2023-10-16T15:50:44.685159+0000","last_fullsized":"2023-10-16T15:50:44.685159+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:37.156252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477692+0000","last_change":"2023-10-16T15:50:44.676056+0000","last_active":"2023-10-16T15:50:46.477692+0000","last_peered":"2023-10-16T15:50:46.477692+0000","last_clean":"2023-10-16T15:50:46.477692+0000","last_became_active":"2023-10-16T15:50:44.675788+0000","last_became_peered":"2023-10-16T15:50:44.675788+0000","last_unstale":"2023-10-16T15:50:46.477692+0000","last_undegraded":"2023-10-16T15:50:46.477692+0000","last_fullsized":"2023-10-16T15:50:46.477692+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:41:33.255162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684934+0000","last_change":"2023-10-16T15:50:43.631670+0000","last_active":"2023-10-16T15:50:44.684934+0000","last_peered":"2023-10-16T15:50:44.684934+0000","last_clean":"2023-10-16T15:50:44.684934+0000","last_became_active":"2023-10-16T15:50:43.631226+0000","last_became_peered":"2023-10-16T15:50:43.631226+0000","last_unstale":"2023-10-16T15:50:44.684934+0000","last_undegraded":"2023-10-16T15:50:44.684934+0000","last_fullsized":"2023-10-16T15:50:44.684934+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:02.797454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675361+0000","last_change":"2023-10-16T15:50:44.680661+0000","last_active":"2023-10-16T15:50:45.675361+0000","last_peered":"2023-10-16T15:50:45.675361+0000","last_clean":"2023-10-16T15:50:45.675361+0000","last_became_active":"2023-10-16T15:50:44.679585+0000","last_became_peered":"2023-10-16T15:50:44.679585+0000","last_unstale":"2023-10-16T15:50:45.675361+0000","last_undegraded":"2023-10-16T15:50:45.675361+0000","last_fullsized":"2023-10-16T15:50:45.675361+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:10.938622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"208'2","reported_seq":13,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.676395+0000","last_change":"2023-10-16T15:50:43.639994+0000","last_active":"2023-10-16T15:50:44.676395+0000","last_peered":"2023-10-16T15:50:44.676395+0000","last_clean":"2023-10-16T15:50:44.676395+0000","last_became_active":"2023-10-16T15:50:43.639664+0000","last_became_peered":"2023-10-16T15:50:43.639664+0000","last_unstale":"2023-10-16T15:50:44.676395+0000","last_undegraded":"2023-10-16T15:50:44.676395+0000","last_fullsized":"2023-10-16T15:50:44.676395+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:09.927764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.672347+0000","last_change":"2023-10-16T15:50:44.669625+0000","last_active":"2023-10-16T15:50:44.672347+0000","last_peered":"2023-10-16T15:50:44.672347+0000","last_clean":"2023-10-16T15:50:44.672347+0000","last_became_active":"2023-10-16T15:50:44.669314+0000","last_became_peered":"2023-10-16T15:50:44.669314+0000","last_unstale":"2023-10-16T15:50:44.672347+0000","last_undegraded":"2023-10-16T15:50:44.672347+0000","last_fullsized":"2023-10-16T15:50:44.672347+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:18:06.007151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"208'2","reported_seq":13,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684330+0000","last_change":"2023-10-16T15:50:43.639184+0000","last_active":"2023-10-16T15:50:44.684330+0000","last_peered":"2023-10-16T15:50:44.684330+0000","last_clean":"2023-10-16T15:50:44.684330+0000","last_became_active":"2023-10-16T15:50:43.637717+0000","last_became_peered":"2023-10-16T15:50:43.637717+0000","last_unstale":"2023-10-16T15:50:44.684330+0000","last_undegraded":"2023-10-16T15:50:44.684330+0000","last_fullsized":"2023-10-16T15:50:44.684330+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:18.944252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476599+0000","last_change":"2023-10-16T15:50:44.657933+0000","last_active":"2023-10-16T15:50:46.476599+0000","last_peered":"2023-10-16T15:50:46.476599+0000","last_clean":"2023-10-16T15:50:46.476599+0000","last_became_active":"2023-10-16T15:50:44.657527+0000","last_became_peered":"2023-10-16T15:50:44.657527+0000","last_unstale":"2023-10-16T15:50:46.476599+0000","last_undegraded":"2023-10-16T15:50:46.476599+0000","last_fullsized":"2023-10-16T15:50:46.476599+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:32.355617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681117+0000","last_change":"2023-10-16T15:50:43.645704+0000","last_active":"2023-10-16T15:50:44.681117+0000","last_peered":"2023-10-16T15:50:44.681117+0000","last_clean":"2023-10-16T15:50:44.681117+0000","last_became_active":"2023-10-16T15:50:43.643170+0000","last_became_peered":"2023-10-16T15:50:43.643170+0000","last_unstale":"2023-10-16T15:50:44.681117+0000","last_undegraded":"2023-10-16T15:50:44.681117+0000","last_fullsized":"2023-10-16T15:50:44.681117+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:06.415718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669627+0000","last_change":"2023-10-16T15:50:44.669139+0000","last_active":"2023-10-16T15:50:45.669627+0000","last_peered":"2023-10-16T15:50:45.669627+0000","last_clean":"2023-10-16T15:50:45.669627+0000","last_became_active":"2023-10-16T15:50:44.668868+0000","last_became_peered":"2023-10-16T15:50:44.668868+0000","last_unstale":"2023-10-16T15:50:45.669627+0000","last_undegraded":"2023-10-16T15:50:45.669627+0000","last_fullsized":"2023-10-16T15:50:45.669627+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:24.042552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"208'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669239+0000","last_change":"2023-10-16T15:50:43.641041+0000","last_active":"2023-10-16T15:50:45.669239+0000","last_peered":"2023-10-16T15:50:45.669239+0000","last_clean":"2023-10-16T15:50:45.669239+0000","last_became_active":"2023-10-16T15:50:43.640738+0000","last_became_peered":"2023-10-16T15:50:43.640738+0000","last_unstale":"2023-10-16T15:50:45.669239+0000","last_undegraded":"2023-10-16T15:50:45.669239+0000","last_fullsized":"2023-10-16T15:50:45.669239+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:07.245737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674411+0000","last_change":"2023-10-16T15:50:44.675275+0000","last_active":"2023-10-16T15:50:45.674411+0000","last_peered":"2023-10-16T15:50:45.674411+0000","last_clean":"2023-10-16T15:50:45.674411+0000","last_became_active":"2023-10-16T15:50:44.675011+0000","last_became_peered":"2023-10-16T15:50:44.675011+0000","last_unstale":"2023-10-16T15:50:45.674411+0000","last_undegraded":"2023-10-16T15:50:45.674411+0000","last_fullsized":"2023-10-16T15:50:45.674411+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:15.786243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.672628+0000","last_change":"2023-10-16T15:50:43.644935+0000","last_active":"2023-10-16T15:50:45.672628+0000","last_peered":"2023-10-16T15:50:45.672628+0000","last_clean":"2023-10-16T15:50:45.672628+0000","last_became_active":"2023-10-16T15:50:43.644495+0000","last_became_peered":"2023-10-16T15:50:43.644495+0000","last_unstale":"2023-10-16T15:50:45.672628+0000","last_undegraded":"2023-10-16T15:50:45.672628+0000","last_fullsized":"2023-10-16T15:50:45.672628+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:07.245737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.668149+0000","last_change":"2023-10-16T15:50:44.665018+0000","last_active":"2023-10-16T15:50:44.668149+0000","last_peered":"2023-10-16T15:50:44.668149+0000","last_clean":"2023-10-16T15:50:44.668149+0000","last_became_active":"2023-10-16T15:50:44.663070+0000","last_became_peered":"2023-10-16T15:50:44.663070+0000","last_unstale":"2023-10-16T15:50:44.668149+0000","last_undegraded":"2023-10-16T15:50:44.668149+0000","last_fullsized":"2023-10-16T15:50:44.668149+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:59.710850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675994+0000","last_change":"2023-10-16T15:50:43.640236+0000","last_active":"2023-10-16T15:50:45.675994+0000","last_peered":"2023-10-16T15:50:45.675994+0000","last_clean":"2023-10-16T15:50:45.675994+0000","last_became_active":"2023-10-16T15:50:43.639800+0000","last_became_peered":"2023-10-16T15:50:43.639800+0000","last_unstale":"2023-10-16T15:50:45.675994+0000","last_undegraded":"2023-10-16T15:50:45.675994+0000","last_fullsized":"2023-10-16T15:50:45.675994+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:55.311987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669998+0000","last_change":"2023-10-16T15:50:44.663216+0000","last_active":"2023-10-16T15:50:45.669998+0000","last_peered":"2023-10-16T15:50:45.669998+0000","last_clean":"2023-10-16T15:50:45.669998+0000","last_became_active":"2023-10-16T15:50:44.662950+0000","last_became_peered":"2023-10-16T15:50:44.662950+0000","last_unstale":"2023-10-16T15:50:45.669998+0000","last_undegraded":"2023-10-16T15:50:45.669998+0000","last_fullsized":"2023-10-16T15:50:45.669998+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:41:13.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.657846+0000","last_change":"2023-10-16T15:50:43.649647+0000","last_active":"2023-10-16T15:50:44.657846+0000","last_peered":"2023-10-16T15:50:44.657846+0000","last_clean":"2023-10-16T15:50:44.657846+0000","last_became_active":"2023-10-16T15:50:43.647770+0000","last_became_peered":"2023-10-16T15:50:43.647770+0000","last_unstale":"2023-10-16T15:50:44.657846+0000","last_undegraded":"2023-10-16T15:50:44.657846+0000","last_fullsized":"2023-10-16T15:50:44.657846+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:14.775386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675730+0000","last_change":"2023-10-16T15:50:44.678626+0000","last_active":"2023-10-16T15:50:45.675730+0000","last_peered":"2023-10-16T15:50:45.675730+0000","last_clean":"2023-10-16T15:50:45.675730+0000","last_became_active":"2023-10-16T15:50:44.678351+0000","last_became_peered":"2023-10-16T15:50:44.678351+0000","last_unstale":"2023-10-16T15:50:45.675730+0000","last_undegraded":"2023-10-16T15:50:45.675730+0000","last_fullsized":"2023-10-16T15:50:45.675730+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:25:17.782187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"208'3","reported_seq":14,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684385+0000","last_change":"2023-10-16T15:50:43.644264+0000","last_active":"2023-10-16T15:50:44.684385+0000","last_peered":"2023-10-16T15:50:44.684385+0000","last_clean":"2023-10-16T15:50:44.684385+0000","last_became_active":"2023-10-16T15:50:43.643988+0000","last_became_peered":"2023-10-16T15:50:43.643988+0000","last_unstale":"2023-10-16T15:50:44.684385+0000","last_undegraded":"2023-10-16T15:50:44.684385+0000","last_fullsized":"2023-10-16T15:50:44.684385+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:48.477325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477006+0000","last_change":"2023-10-16T15:50:44.663034+0000","last_active":"2023-10-16T15:50:46.477006+0000","last_peered":"2023-10-16T15:50:46.477006+0000","last_clean":"2023-10-16T15:50:46.477006+0000","last_became_active":"2023-10-16T15:50:44.661134+0000","last_became_peered":"2023-10-16T15:50:44.661134+0000","last_unstale":"2023-10-16T15:50:46.477006+0000","last_undegraded":"2023-10-16T15:50:46.477006+0000","last_fullsized":"2023-10-16T15:50:46.477006+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:19.162796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"208'6","reported_seq":17,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685774+0000","last_change":"2023-10-16T15:50:43.645522+0000","last_active":"2023-10-16T15:50:44.685774+0000","last_peered":"2023-10-16T15:50:44.685774+0000","last_clean":"2023-10-16T15:50:44.685774+0000","last_became_active":"2023-10-16T15:50:43.644816+0000","last_became_peered":"2023-10-16T15:50:43.644816+0000","last_unstale":"2023-10-16T15:50:44.685774+0000","last_undegraded":"2023-10-16T15:50:44.685774+0000","last_fullsized":"2023-10-16T15:50:44.685774+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:37.156252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.668239+0000","last_change":"2023-10-16T15:50:44.659452+0000","last_active":"2023-10-16T15:50:44.668239+0000","last_peered":"2023-10-16T15:50:44.668239+0000","last_clean":"2023-10-16T15:50:44.668239+0000","last_became_active":"2023-10-16T15:50:44.659146+0000","last_became_peered":"2023-10-16T15:50:44.659146+0000","last_unstale":"2023-10-16T15:50:44.668239+0000","last_undegraded":"2023-10-16T15:50:44.668239+0000","last_fullsized":"2023-10-16T15:50:44.668239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:38.915700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"208'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675231+0000","last_change":"2023-10-16T15:50:43.643008+0000","last_active":"2023-10-16T15:50:45.675231+0000","last_peered":"2023-10-16T15:50:45.675231+0000","last_clean":"2023-10-16T15:50:45.675231+0000","last_became_active":"2023-10-16T15:50:43.642704+0000","last_became_peered":"2023-10-16T15:50:43.642704+0000","last_unstale":"2023-10-16T15:50:45.675231+0000","last_undegraded":"2023-10-16T15:50:45.675231+0000","last_fullsized":"2023-10-16T15:50:45.675231+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:13.902662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674571+0000","last_change":"2023-10-16T15:50:44.678222+0000","last_active":"2023-10-16T15:50:45.674571+0000","last_peered":"2023-10-16T15:50:45.674571+0000","last_clean":"2023-10-16T15:50:45.674571+0000","last_became_active":"2023-10-16T15:50:44.677856+0000","last_became_peered":"2023-10-16T15:50:44.677856+0000","last_unstale":"2023-10-16T15:50:45.674571+0000","last_undegraded":"2023-10-16T15:50:45.674571+0000","last_fullsized":"2023-10-16T15:50:45.674571+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:38.167110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476850+0000","last_change":"2023-10-16T15:50:43.641577+0000","last_active":"2023-10-16T15:50:46.476850+0000","last_peered":"2023-10-16T15:50:46.476850+0000","last_clean":"2023-10-16T15:50:46.476850+0000","last_became_active":"2023-10-16T15:50:43.641280+0000","last_became_peered":"2023-10-16T15:50:43.641280+0000","last_unstale":"2023-10-16T15:50:46.476850+0000","last_undegraded":"2023-10-16T15:50:46.476850+0000","last_fullsized":"2023-10-16T15:50:46.476850+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:35.086923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.670175+0000","last_change":"2023-10-16T15:50:44.660076+0000","last_active":"2023-10-16T15:50:44.670175+0000","last_peered":"2023-10-16T15:50:44.670175+0000","last_clean":"2023-10-16T15:50:44.670175+0000","last_became_active":"2023-10-16T15:50:44.658027+0000","last_became_peered":"2023-10-16T15:50:44.658027+0000","last_unstale":"2023-10-16T15:50:44.670175+0000","last_undegraded":"2023-10-16T15:50:44.670175+0000","last_fullsized":"2023-10-16T15:50:44.670175+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:40.247155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668690+0000","last_change":"2023-10-16T15:50:43.647240+0000","last_active":"2023-10-16T15:50:45.668690+0000","last_peered":"2023-10-16T15:50:45.668690+0000","last_clean":"2023-10-16T15:50:45.668690+0000","last_became_active":"2023-10-16T15:50:43.645514+0000","last_became_peered":"2023-10-16T15:50:43.645514+0000","last_unstale":"2023-10-16T15:50:45.668690+0000","last_undegraded":"2023-10-16T15:50:45.668690+0000","last_fullsized":"2023-10-16T15:50:45.668690+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:38.342357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476048+0000","last_change":"2023-10-16T15:50:44.670849+0000","last_active":"2023-10-16T15:50:46.476048+0000","last_peered":"2023-10-16T15:50:46.476048+0000","last_clean":"2023-10-16T15:50:46.476048+0000","last_became_active":"2023-10-16T15:50:44.670571+0000","last_became_peered":"2023-10-16T15:50:44.670571+0000","last_unstale":"2023-10-16T15:50:46.476048+0000","last_undegraded":"2023-10-16T15:50:46.476048+0000","last_fullsized":"2023-10-16T15:50:46.476048+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:50.435932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"208'8","reported_seq":20,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.672793+0000","last_change":"2023-10-16T15:50:43.645287+0000","last_active":"2023-10-16T15:50:45.672793+0000","last_peered":"2023-10-16T15:50:45.672793+0000","last_clean":"2023-10-16T15:50:45.672793+0000","last_became_active":"2023-10-16T15:50:43.644908+0000","last_became_peered":"2023-10-16T15:50:43.644908+0000","last_unstale":"2023-10-16T15:50:45.672793+0000","last_undegraded":"2023-10-16T15:50:45.672793+0000","last_fullsized":"2023-10-16T15:50:45.672793+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:13.937593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477300+0000","last_change":"2023-10-16T15:50:44.672936+0000","last_active":"2023-10-16T15:50:46.477300+0000","last_peered":"2023-10-16T15:50:46.477300+0000","last_clean":"2023-10-16T15:50:46.477300+0000","last_became_active":"2023-10-16T15:50:44.672648+0000","last_became_peered":"2023-10-16T15:50:44.672648+0000","last_unstale":"2023-10-16T15:50:46.477300+0000","last_undegraded":"2023-10-16T15:50:46.477300+0000","last_fullsized":"2023-10-16T15:50:46.477300+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:38.753648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.683784+0000","last_change":"2023-10-16T15:50:43.643170+0000","last_active":"2023-10-16T15:50:44.683784+0000","last_peered":"2023-10-16T15:50:44.683784+0000","last_clean":"2023-10-16T15:50:44.683784+0000","last_became_active":"2023-10-16T15:50:43.642896+0000","last_became_peered":"2023-10-16T15:50:43.642896+0000","last_unstale":"2023-10-16T15:50:44.683784+0000","last_undegraded":"2023-10-16T15:50:44.683784+0000","last_fullsized":"2023-10-16T15:50:44.683784+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:49.420248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680556+0000","last_change":"2023-10-16T15:50:44.672955+0000","last_active":"2023-10-16T15:50:44.680556+0000","last_peered":"2023-10-16T15:50:44.680556+0000","last_clean":"2023-10-16T15:50:44.680556+0000","last_became_active":"2023-10-16T15:50:44.672661+0000","last_became_peered":"2023-10-16T15:50:44.672661+0000","last_unstale":"2023-10-16T15:50:44.680556+0000","last_undegraded":"2023-10-16T15:50:44.680556+0000","last_fullsized":"2023-10-16T15:50:44.680556+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:34.004737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"208'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669758+0000","last_change":"2023-10-16T15:50:43.642282+0000","last_active":"2023-10-16T15:50:45.669758+0000","last_peered":"2023-10-16T15:50:45.669758+0000","last_clean":"2023-10-16T15:50:45.669758+0000","last_became_active":"2023-10-16T15:50:43.641965+0000","last_became_peered":"2023-10-16T15:50:43.641965+0000","last_unstale":"2023-10-16T15:50:45.669758+0000","last_undegraded":"2023-10-16T15:50:45.669758+0000","last_fullsized":"2023-10-16T15:50:45.669758+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:43.125218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.675041+0000","last_change":"2023-10-16T15:50:44.668998+0000","last_active":"2023-10-16T15:50:44.675041+0000","last_peered":"2023-10-16T15:50:44.675041+0000","last_clean":"2023-10-16T15:50:44.675041+0000","last_became_active":"2023-10-16T15:50:44.668158+0000","last_became_peered":"2023-10-16T15:50:44.668158+0000","last_unstale":"2023-10-16T15:50:44.675041+0000","last_undegraded":"2023-10-16T15:50:44.675041+0000","last_fullsized":"2023-10-16T15:50:44.675041+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:32.111306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"208'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674762+0000","last_change":"2023-10-16T15:50:43.645363+0000","last_active":"2023-10-16T15:50:45.674762+0000","last_peered":"2023-10-16T15:50:45.674762+0000","last_clean":"2023-10-16T15:50:45.674762+0000","last_became_active":"2023-10-16T15:50:43.645069+0000","last_became_peered":"2023-10-16T15:50:43.645069+0000","last_unstale":"2023-10-16T15:50:45.674762+0000","last_undegraded":"2023-10-16T15:50:45.674762+0000","last_fullsized":"2023-10-16T15:50:45.674762+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.480540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.682134+0000","last_change":"2023-10-16T15:50:44.674754+0000","last_active":"2023-10-16T15:50:44.682134+0000","last_peered":"2023-10-16T15:50:44.682134+0000","last_clean":"2023-10-16T15:50:44.682134+0000","last_became_active":"2023-10-16T15:50:44.674485+0000","last_became_peered":"2023-10-16T15:50:44.674485+0000","last_unstale":"2023-10-16T15:50:44.682134+0000","last_undegraded":"2023-10-16T15:50:44.682134+0000","last_fullsized":"2023-10-16T15:50:44.682134+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:40.868217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675739+0000","last_change":"2023-10-16T15:50:43.648761+0000","last_active":"2023-10-16T15:50:45.675739+0000","last_peered":"2023-10-16T15:50:45.675739+0000","last_clean":"2023-10-16T15:50:45.675739+0000","last_became_active":"2023-10-16T15:50:43.648425+0000","last_became_peered":"2023-10-16T15:50:43.648425+0000","last_unstale":"2023-10-16T15:50:45.675739+0000","last_undegraded":"2023-10-16T15:50:45.675739+0000","last_fullsized":"2023-10-16T15:50:45.675739+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:37.904842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476631+0000","last_change":"2023-10-16T15:50:44.664122+0000","last_active":"2023-10-16T15:50:46.476631+0000","last_peered":"2023-10-16T15:50:46.476631+0000","last_clean":"2023-10-16T15:50:46.476631+0000","last_became_active":"2023-10-16T15:50:44.663798+0000","last_became_peered":"2023-10-16T15:50:44.663798+0000","last_unstale":"2023-10-16T15:50:46.476631+0000","last_undegraded":"2023-10-16T15:50:46.476631+0000","last_fullsized":"2023-10-16T15:50:46.476631+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:51.960978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"208'2","reported_seq":13,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680820+0000","last_change":"2023-10-16T15:50:43.642605+0000","last_active":"2023-10-16T15:50:44.680820+0000","last_peered":"2023-10-16T15:50:44.680820+0000","last_clean":"2023-10-16T15:50:44.680820+0000","last_became_active":"2023-10-16T15:50:43.642305+0000","last_became_peered":"2023-10-16T15:50:43.642305+0000","last_unstale":"2023-10-16T15:50:44.680820+0000","last_undegraded":"2023-10-16T15:50:44.680820+0000","last_fullsized":"2023-10-16T15:50:44.680820+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:44.066295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674925+0000","last_change":"2023-10-16T15:50:44.678526+0000","last_active":"2023-10-16T15:50:45.674925+0000","last_peered":"2023-10-16T15:50:45.674925+0000","last_clean":"2023-10-16T15:50:45.674925+0000","last_became_active":"2023-10-16T15:50:44.675146+0000","last_became_peered":"2023-10-16T15:50:44.675146+0000","last_unstale":"2023-10-16T15:50:45.674925+0000","last_undegraded":"2023-10-16T15:50:45.674925+0000","last_fullsized":"2023-10-16T15:50:45.674925+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:50.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.654222+0000","last_change":"2023-10-16T15:50:43.644608+0000","last_active":"2023-10-16T15:50:44.654222+0000","last_peered":"2023-10-16T15:50:44.654222+0000","last_clean":"2023-10-16T15:50:44.654222+0000","last_became_active":"2023-10-16T15:50:43.644262+0000","last_became_peered":"2023-10-16T15:50:43.644262+0000","last_unstale":"2023-10-16T15:50:44.654222+0000","last_undegraded":"2023-10-16T15:50:44.654222+0000","last_fullsized":"2023-10-16T15:50:44.654222+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:03.469421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680076+0000","last_change":"2023-10-16T15:50:44.673958+0000","last_active":"2023-10-16T15:50:44.680076+0000","last_peered":"2023-10-16T15:50:44.680076+0000","last_clean":"2023-10-16T15:50:44.680076+0000","last_became_active":"2023-10-16T15:50:44.673264+0000","last_became_peered":"2023-10-16T15:50:44.673264+0000","last_unstale":"2023-10-16T15:50:44.680076+0000","last_undegraded":"2023-10-16T15:50:44.680076+0000","last_fullsized":"2023-10-16T15:50:44.680076+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:50.435932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.653823+0000","last_change":"2023-10-16T15:50:43.647628+0000","last_active":"2023-10-16T15:50:44.653823+0000","last_peered":"2023-10-16T15:50:44.653823+0000","last_clean":"2023-10-16T15:50:44.653823+0000","last_became_active":"2023-10-16T15:50:43.647316+0000","last_became_peered":"2023-10-16T15:50:43.647316+0000","last_unstale":"2023-10-16T15:50:44.653823+0000","last_undegraded":"2023-10-16T15:50:44.653823+0000","last_fullsized":"2023-10-16T15:50:44.653823+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:49.420248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680320+0000","last_change":"2023-10-16T15:50:44.670215+0000","last_active":"2023-10-16T15:50:44.680320+0000","last_peered":"2023-10-16T15:50:44.680320+0000","last_clean":"2023-10-16T15:50:44.680320+0000","last_became_active":"2023-10-16T15:50:44.669936+0000","last_became_peered":"2023-10-16T15:50:44.669936+0000","last_unstale":"2023-10-16T15:50:44.680320+0000","last_undegraded":"2023-10-16T15:50:44.680320+0000","last_fullsized":"2023-10-16T15:50:44.680320+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:42:50.522444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.653140+0000","last_change":"2023-10-16T15:50:43.646567+0000","last_active":"2023-10-16T15:50:44.653140+0000","last_peered":"2023-10-16T15:50:44.653140+0000","last_clean":"2023-10-16T15:50:44.653140+0000","last_became_active":"2023-10-16T15:50:43.646077+0000","last_became_peered":"2023-10-16T15:50:43.646077+0000","last_unstale":"2023-10-16T15:50:44.653140+0000","last_undegraded":"2023-10-16T15:50:44.653140+0000","last_fullsized":"2023-10-16T15:50:44.653140+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:11.793925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681907+0000","last_change":"2023-10-16T15:50:44.672039+0000","last_active":"2023-10-16T15:50:44.681907+0000","last_peered":"2023-10-16T15:50:44.681907+0000","last_clean":"2023-10-16T15:50:44.681907+0000","last_became_active":"2023-10-16T15:50:44.670144+0000","last_became_peered":"2023-10-16T15:50:44.670144+0000","last_unstale":"2023-10-16T15:50:44.681907+0000","last_undegraded":"2023-10-16T15:50:44.681907+0000","last_fullsized":"2023-10-16T15:50:44.681907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:45.077152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"208'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675752+0000","last_change":"2023-10-16T15:50:43.644414+0000","last_active":"2023-10-16T15:50:45.675752+0000","last_peered":"2023-10-16T15:50:45.675752+0000","last_clean":"2023-10-16T15:50:45.675752+0000","last_became_active":"2023-10-16T15:50:43.644107+0000","last_became_peered":"2023-10-16T15:50:43.644107+0000","last_unstale":"2023-10-16T15:50:45.675752+0000","last_undegraded":"2023-10-16T15:50:45.675752+0000","last_fullsized":"2023-10-16T15:50:45.675752+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:38.787908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477771+0000","last_change":"2023-10-16T15:50:44.665939+0000","last_active":"2023-10-16T15:50:46.477771+0000","last_peered":"2023-10-16T15:50:46.477771+0000","last_clean":"2023-10-16T15:50:46.477771+0000","last_became_active":"2023-10-16T15:50:44.665247+0000","last_became_peered":"2023-10-16T15:50:44.665247+0000","last_unstale":"2023-10-16T15:50:46.477771+0000","last_undegraded":"2023-10-16T15:50:46.477771+0000","last_fullsized":"2023-10-16T15:50:46.477771+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:10.952350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477234+0000","last_change":"2023-10-16T15:50:43.646596+0000","last_active":"2023-10-16T15:50:46.477234+0000","last_peered":"2023-10-16T15:50:46.477234+0000","last_clean":"2023-10-16T15:50:46.477234+0000","last_became_active":"2023-10-16T15:50:43.646147+0000","last_became_peered":"2023-10-16T15:50:43.646147+0000","last_unstale":"2023-10-16T15:50:46.477234+0000","last_undegraded":"2023-10-16T15:50:46.477234+0000","last_fullsized":"2023-10-16T15:50:46.477234+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:40.672033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669087+0000","last_change":"2023-10-16T15:50:44.659744+0000","last_active":"2023-10-16T15:50:45.669087+0000","last_peered":"2023-10-16T15:50:45.669087+0000","last_clean":"2023-10-16T15:50:45.669087+0000","last_became_active":"2023-10-16T15:50:44.659375+0000","last_became_peered":"2023-10-16T15:50:44.659375+0000","last_unstale":"2023-10-16T15:50:45.669087+0000","last_undegraded":"2023-10-16T15:50:45.669087+0000","last_fullsized":"2023-10-16T15:50:45.669087+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:48.194990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476850+0000","last_change":"2023-10-16T15:50:43.643102+0000","last_active":"2023-10-16T15:50:46.476850+0000","last_peered":"2023-10-16T15:50:46.476850+0000","last_clean":"2023-10-16T15:50:46.476850+0000","last_became_active":"2023-10-16T15:50:43.642753+0000","last_became_peered":"2023-10-16T15:50:43.642753+0000","last_unstale":"2023-10-16T15:50:46.476850+0000","last_undegraded":"2023-10-16T15:50:46.476850+0000","last_fullsized":"2023-10-16T15:50:46.476850+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:41.556192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669725+0000","last_change":"2023-10-16T15:50:44.673317+0000","last_active":"2023-10-16T15:50:45.669725+0000","last_peered":"2023-10-16T15:50:45.669725+0000","last_clean":"2023-10-16T15:50:45.669725+0000","last_became_active":"2023-10-16T15:50:44.672945+0000","last_became_peered":"2023-10-16T15:50:44.672945+0000","last_unstale":"2023-10-16T15:50:45.669725+0000","last_undegraded":"2023-10-16T15:50:45.669725+0000","last_fullsized":"2023-10-16T15:50:45.669725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:13.733047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669224+0000","last_change":"2023-10-16T15:50:43.643080+0000","last_active":"2023-10-16T15:50:45.669224+0000","last_peered":"2023-10-16T15:50:45.669224+0000","last_clean":"2023-10-16T15:50:45.669224+0000","last_became_active":"2023-10-16T15:50:43.642790+0000","last_became_peered":"2023-10-16T15:50:43.642790+0000","last_unstale":"2023-10-16T15:50:45.669224+0000","last_undegraded":"2023-10-16T15:50:45.669224+0000","last_fullsized":"2023-10-16T15:50:45.669224+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:44.087812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674684+0000","last_change":"2023-10-16T15:50:44.650396+0000","last_active":"2023-10-16T15:50:45.674684+0000","last_peered":"2023-10-16T15:50:45.674684+0000","last_clean":"2023-10-16T15:50:45.674684+0000","last_became_active":"2023-10-16T15:50:44.649422+0000","last_became_peered":"2023-10-16T15:50:44.649422+0000","last_unstale":"2023-10-16T15:50:45.674684+0000","last_undegraded":"2023-10-16T15:50:45.674684+0000","last_fullsized":"2023-10-16T15:50:45.674684+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:32.326238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.652870+0000","last_change":"2023-10-16T15:50:43.635996+0000","last_active":"2023-10-16T15:50:44.652870+0000","last_peered":"2023-10-16T15:50:44.652870+0000","last_clean":"2023-10-16T15:50:44.652870+0000","last_became_active":"2023-10-16T15:50:43.635619+0000","last_became_peered":"2023-10-16T15:50:43.635619+0000","last_unstale":"2023-10-16T15:50:44.652870+0000","last_undegraded":"2023-10-16T15:50:44.652870+0000","last_fullsized":"2023-10-16T15:50:44.652870+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:52.907475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674989+0000","last_change":"2023-10-16T15:50:44.670966+0000","last_active":"2023-10-16T15:50:45.674989+0000","last_peered":"2023-10-16T15:50:45.674989+0000","last_clean":"2023-10-16T15:50:45.674989+0000","last_became_active":"2023-10-16T15:50:44.670174+0000","last_became_peered":"2023-10-16T15:50:44.670174+0000","last_unstale":"2023-10-16T15:50:45.674989+0000","last_undegraded":"2023-10-16T15:50:45.674989+0000","last_fullsized":"2023-10-16T15:50:45.674989+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:24.042552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.652639+0000","last_change":"2023-10-16T15:50:43.635679+0000","last_active":"2023-10-16T15:50:44.652639+0000","last_peered":"2023-10-16T15:50:44.652639+0000","last_clean":"2023-10-16T15:50:44.652639+0000","last_became_active":"2023-10-16T15:50:43.635264+0000","last_became_peered":"2023-10-16T15:50:43.635264+0000","last_unstale":"2023-10-16T15:50:44.652639+0000","last_undegraded":"2023-10-16T15:50:44.652639+0000","last_fullsized":"2023-10-16T15:50:44.652639+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:02.580282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681819+0000","last_change":"2023-10-16T15:50:44.674153+0000","last_active":"2023-10-16T15:50:44.681819+0000","last_peered":"2023-10-16T15:50:44.681819+0000","last_clean":"2023-10-16T15:50:44.681819+0000","last_became_active":"2023-10-16T15:50:44.673667+0000","last_became_peered":"2023-10-16T15:50:44.673667+0000","last_unstale":"2023-10-16T15:50:44.681819+0000","last_undegraded":"2023-10-16T15:50:44.681819+0000","last_fullsized":"2023-10-16T15:50:44.681819+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:26.730085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.659651+0000","last_change":"2023-10-16T15:50:43.644466+0000","last_active":"2023-10-16T15:50:44.659651+0000","last_peered":"2023-10-16T15:50:44.659651+0000","last_clean":"2023-10-16T15:50:44.659651+0000","last_became_active":"2023-10-16T15:50:43.642815+0000","last_became_peered":"2023-10-16T15:50:43.642815+0000","last_unstale":"2023-10-16T15:50:44.659651+0000","last_undegraded":"2023-10-16T15:50:44.659651+0000","last_fullsized":"2023-10-16T15:50:44.659651+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:59.573511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680298+0000","last_change":"2023-10-16T15:50:44.668739+0000","last_active":"2023-10-16T15:50:44.680298+0000","last_peered":"2023-10-16T15:50:44.680298+0000","last_clean":"2023-10-16T15:50:44.680298+0000","last_became_active":"2023-10-16T15:50:44.668437+0000","last_became_peered":"2023-10-16T15:50:44.668437+0000","last_unstale":"2023-10-16T15:50:44.680298+0000","last_undegraded":"2023-10-16T15:50:44.680298+0000","last_fullsized":"2023-10-16T15:50:44.680298+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:14.507460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.653097+0000","last_change":"2023-10-16T15:50:43.645512+0000","last_active":"2023-10-16T15:50:44.653097+0000","last_peered":"2023-10-16T15:50:44.653097+0000","last_clean":"2023-10-16T15:50:44.653097+0000","last_became_active":"2023-10-16T15:50:43.644654+0000","last_became_peered":"2023-10-16T15:50:43.644654+0000","last_unstale":"2023-10-16T15:50:44.653097+0000","last_undegraded":"2023-10-16T15:50:44.653097+0000","last_fullsized":"2023-10-16T15:50:44.653097+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:14.775386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.475604+0000","last_change":"2023-10-16T15:50:44.673743+0000","last_active":"2023-10-16T15:50:46.475604+0000","last_peered":"2023-10-16T15:50:46.475604+0000","last_clean":"2023-10-16T15:50:46.475604+0000","last_became_active":"2023-10-16T15:50:44.673504+0000","last_became_peered":"2023-10-16T15:50:44.673504+0000","last_unstale":"2023-10-16T15:50:46.475604+0000","last_undegraded":"2023-10-16T15:50:46.475604+0000","last_fullsized":"2023-10-16T15:50:46.475604+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:19.955110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.673699+0000","last_change":"2023-10-16T15:50:43.639488+0000","last_active":"2023-10-16T15:50:45.673699+0000","last_peered":"2023-10-16T15:50:45.673699+0000","last_clean":"2023-10-16T15:50:45.673699+0000","last_became_active":"2023-10-16T15:50:43.639062+0000","last_became_peered":"2023-10-16T15:50:43.639062+0000","last_unstale":"2023-10-16T15:50:45.673699+0000","last_undegraded":"2023-10-16T15:50:45.673699+0000","last_fullsized":"2023-10-16T15:50:45.673699+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:20.592600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"197'329","reported_seq":1151,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.645382+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:50:44.645382+0000","last_peered":"2023-10-16T15:50:44.645382+0000","last_clean":"2023-10-16T15:50:44.645382+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:50:44.645382+0000","last_undegraded":"2023-10-16T15:50:44.645382+0000","last_fullsized":"2023-10-16T15:50:44.645382+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":329,"ondisk_log_size":329,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575972,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60692,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311148,"statfs":{"total":96636764160,"available":96574615552,"internally_reserved":0,"allocated":3219456,"data_stored":1872699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.194,"15min":1.1799999999999999},"min":{"1min":0.76200000000000001,"5min":0.73599999999999999,"15min":0.70699999999999996},"max":{"1min":1.7330000000000001,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.155,"15min":1.1679999999999999},"min":{"1min":0.91600000000000004,"5min":0.86099999999999999,"15min":0.64200000000000002},"max":{"1min":1.4970000000000001,"5min":1.6910000000000001,"15min":1.748},"last":1.395}]},{"osd":1,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1839999999999999,"15min":1.194},"min":{"1min":0.92400000000000004,"5min":0.84399999999999997,"15min":0.67400000000000004},"max":{"1min":1.55,"5min":1.802,"15min":1.889},"last":1.22},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4019999999999999,"15min":1.389},"min":{"1min":1.1359999999999999,"5min":1.004,"15min":0.88400000000000001},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":0.76800000000000002}]},{"osd":2,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.359,"15min":1.3520000000000001},"min":{"1min":0.88100000000000001,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.468},{"interface":"front","average":{"1min":1.391,"5min":1.3380000000000001,"15min":1.3169999999999999},"min":{"1min":1.0760000000000001,"5min":0.94999999999999996,"15min":0.77400000000000002},"max":{"1min":1.7150000000000001,"5min":1.95,"15min":1.95},"last":1.0980000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.3939999999999999,"15min":1.3600000000000001},"min":{"1min":1.089,"5min":0.95999999999999996,"15min":0.878},"max":{"1min":1.702,"5min":1.869,"15min":1.9750000000000001},"last":1.351},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4690000000000001,"15min":1.4610000000000001},"min":{"1min":1.0409999999999999,"5min":0.93100000000000005,"15min":0.85199999999999998},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.6879999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.385,"15min":1.3759999999999999},"min":{"1min":0.95699999999999996,"5min":0.91800000000000004,"15min":0.80600000000000005},"max":{"1min":1.819,"5min":1.819,"15min":1.978},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.3660000000000001,"15min":1.3500000000000001},"min":{"1min":0.81999999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6419999999999999,"5min":2.0129999999999999,"15min":2.0409999999999999},"last":1}]},{"osd":5,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"min":{"1min":1.1240000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.1469999999999998},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5680000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.7609999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":1.0109999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.845,"5min":2.0840000000000001,"15min":2.2149999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.5860000000000001,"15min":1.595},"min":{"1min":1.21,"5min":1.0649999999999999,"15min":0.89500000000000002},"max":{"1min":2.2599999999999998,"5min":2.2599999999999998,"15min":2.3250000000000002},"last":1.8120000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575971,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29568,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342272,"statfs":{"total":96636764160,"available":96606486528,"internally_reserved":0,"allocated":2605056,"data_stored":1275899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.23,"15min":1.2010000000000001},"min":{"1min":0.91200000000000003,"5min":0.79600000000000004,"15min":0.73199999999999998},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.264,"15min":1.2509999999999999},"min":{"1min":0.88400000000000001,"5min":0.80800000000000005,"15min":0.60699999999999998},"max":{"1min":1.738,"5min":1.847,"15min":1.847},"last":1.2170000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.274,"15min":1.282},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.72199999999999998},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.802},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3240000000000001,"15min":1.319},"min":{"1min":0.94999999999999996,"5min":0.90000000000000002,"15min":0.66600000000000004},"max":{"1min":1.714,"5min":1.8740000000000001,"15min":1.9419999999999999},"last":1.0369999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.238,"15min":1.244},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.71899999999999997},"max":{"1min":1.5089999999999999,"5min":1.7090000000000001,"15min":1.8380000000000001},"last":1.083},{"interface":"front","average":{"1min":1.427,"5min":1.4139999999999999,"15min":1.3899999999999999},"min":{"1min":1.0820000000000001,"5min":0.97299999999999998,"15min":0.84399999999999997},"max":{"1min":1.764,"5min":2.0339999999999998,"15min":2.0419999999999998},"last":1.171}]},{"osd":3,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.4059999999999999,"15min":1.399},"min":{"1min":1.0740000000000001,"5min":0.84099999999999997,"15min":0.77700000000000002},"max":{"1min":1.7649999999999999,"5min":1.8360000000000001,"15min":2.0459999999999998},"last":1.119},{"interface":"front","average":{"1min":1.391,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":1.206,"5min":1.0349999999999999,"15min":0.81000000000000005},"max":{"1min":1.6890000000000001,"5min":1.7569999999999999,"15min":2.0699999999999998},"last":1.268}]},{"osd":4,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.3340000000000001,"15min":1.325},"min":{"1min":0.95599999999999996,"5min":0.81200000000000006,"15min":0.78300000000000003},"max":{"1min":1.917,"5min":1.917,"15min":2.028},"last":1.472},{"interface":"front","average":{"1min":1.49,"5min":1.3959999999999999,"15min":1.4079999999999999},"min":{"1min":1.115,"5min":0.92500000000000004,"15min":0.77600000000000002},"max":{"1min":1.819,"5min":1.956,"15min":2.0419999999999998},"last":0.93000000000000005}]},{"osd":5,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3799999999999999,"15min":1.353},"min":{"1min":1.0309999999999999,"5min":0.90700000000000003,"15min":0.67300000000000004},"max":{"1min":1.663,"5min":1.9419999999999999,"15min":2.016},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.514,"15min":1.5229999999999999},"min":{"1min":1.234,"5min":0.98499999999999999,"15min":0.88},"max":{"1min":1.9139999999999999,"5min":2.129,"15min":2.129},"last":1.371}]},{"osd":7,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.595,"15min":1.595},"min":{"1min":1.2529999999999999,"5min":1.0720000000000001,"15min":0.94599999999999995},"max":{"1min":1.8779999999999999,"5min":2.0510000000000002,"15min":2.4940000000000002},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.595,"15min":1.5900000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.97499999999999998},"max":{"1min":1.863,"5min":2.1179999999999999,"15min":2.169},"last":1.4490000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575971,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29568,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342272,"statfs":{"total":96636764160,"available":96606486528,"internally_reserved":0,"allocated":2605056,"data_stored":1275899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1419999999999999,"15min":1.1759999999999999},"min":{"1min":0.79500000000000004,"5min":0.77200000000000002,"15min":0.65600000000000003},"max":{"1min":1.55,"5min":1.613,"15min":1.821},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.262,"15min":1.2230000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.673,"5min":1.732,"15min":1.8120000000000001},"last":1.3660000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.224,"15min":1.2310000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.498,"5min":1.8140000000000001,"15min":1.871},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.306,"5min":1.29,"15min":1.3120000000000001},"min":{"1min":0.93799999999999994,"5min":0.87,"15min":0.87},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.1759999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.228,"15min":1.2609999999999999},"min":{"1min":0.96799999999999997,"5min":0.92000000000000004,"15min":0.86699999999999999},"max":{"1min":1.577,"5min":1.6890000000000001,"15min":1.9219999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4039999999999999,"15min":1.4079999999999999},"min":{"1min":1.2010000000000001,"5min":1.0349999999999999,"15min":0.94899999999999995},"max":{"1min":1.748,"5min":1.8460000000000001,"15min":2.2160000000000002},"last":1.04}]},{"osd":3,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.359,"15min":1.3440000000000001},"min":{"1min":0.998,"5min":0.97699999999999998,"15min":0.80000000000000004},"max":{"1min":1.829,"5min":1.8759999999999999,"15min":1.962},"last":1.282},{"interface":"front","average":{"1min":1.46,"5min":1.3819999999999999,"15min":1.3979999999999999},"min":{"1min":1.24,"5min":0.93899999999999995,"15min":0.93000000000000005},"max":{"1min":1.7150000000000001,"5min":1.748,"15min":2.1589999999999998},"last":1.212}]},{"osd":4,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.365,"15min":1.3400000000000001},"min":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.78100000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.9730000000000001},"last":1.466},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.5249999999999999},"min":{"1min":0.98899999999999999,"5min":0.90800000000000003,"15min":0.77000000000000002},"max":{"1min":1.8580000000000001,"5min":1.859,"15min":2.3210000000000002},"last":1.339}]},{"osd":6,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.3779999999999999,"15min":1.3660000000000001},"min":{"1min":0.92500000000000004,"5min":0.83599999999999997,"15min":0.80600000000000005},"max":{"1min":1.95,"5min":1.95,"15min":2.3809999999999998},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.4470000000000001,"15min":1.4390000000000001},"min":{"1min":1.129,"5min":0.88800000000000001,"15min":0.70599999999999996},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.643}]},{"osd":7,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5469999999999999,"15min":1.548},"min":{"1min":1.1140000000000001,"5min":1.069,"15min":0.98599999999999999},"max":{"1min":1.964,"5min":1.964,"15min":2.5129999999999999},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.6259999999999999,"15min":1.6339999999999999},"min":{"1min":1.032,"5min":1.032,"15min":0.70499999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.544},"last":1.5149999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51800,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320040,"statfs":{"total":96636764160,"available":96583720960,"internally_reserved":0,"allocated":2633728,"data_stored":1284421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.232,"15min":1.224},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.52700000000000002},"max":{"1min":1.6180000000000001,"5min":1.752,"15min":1.823},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2010000000000001,"15min":1.218},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.54,"5min":1.8049999999999999,"15min":1.998},"last":0.91200000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.2549999999999999,"15min":1.2330000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.629},"max":{"1min":1.4810000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.264,"5min":1.3440000000000001,"15min":1.319},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.667,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.21}]},{"osd":2,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.4550000000000001,"15min":1.4670000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8100000000000001,"5min":2.0489999999999999,"15min":2.8929999999999998},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4870000000000001,"15min":1.49},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.835,"5min":2.1829999999999998,"15min":2.4550000000000001},"last":1.3049999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3859999999999999,"15min":1.377},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7270000000000001,"5min":1.9199999999999999,"15min":2.3650000000000002},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.395,"15min":1.389},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":1.5980000000000001,"5min":2.0870000000000002,"15min":2.2970000000000002},"last":1.167}]},{"osd":5,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3420000000000001,"15min":1.343},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.7070000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.4370000000000001,"15min":1.4139999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7509999999999999,"5min":2.1139999999999999,"15min":2.4489999999999998},"last":1.52}]},{"osd":6,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.375,"15min":1.3979999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":1.675,"5min":2.2010000000000001,"15min":2.2389999999999999},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.5189999999999999,"15min":1.522},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":1.7729999999999999,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":1.722}]},{"osd":7,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5840000000000001,"15min":1.571},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.9350000000000001,"5min":2.1659999999999999,"15min":2.3460000000000001},"last":1.583},{"interface":"front","average":{"1min":1.514,"5min":1.609,"15min":1.587},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.1120000000000001,"5min":2.27,"15min":2.347},"last":1.76}]}]},{"osd":3,"up_from":13,"seq":55834575971,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51804,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320036,"statfs":{"total":96636764160,"available":96583716864,"internally_reserved":0,"allocated":2629632,"data_stored":1282421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.94999999999999996,"15min":0.96499999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2230000000000001,"5min":1.4950000000000001,"15min":2.073},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.042,"5min":1.0580000000000001,"15min":1.087},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":1.081}]},{"osd":1,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0600000000000001,"15min":1.056},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6679999999999999,"5min":2.0049999999999999,"15min":2.589},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1879999999999999,"15min":1.2270000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.722,"5min":1.722,"15min":2.5569999999999999},"last":1.1779999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4510000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.885,"5min":1.897,"15min":2.6600000000000001},"last":1.103},{"interface":"front","average":{"1min":1.429,"5min":1.4970000000000001,"15min":1.528},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.8839999999999999,"5min":1.9530000000000001,"15min":2.754},"last":0.98699999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.377,"15min":1.3660000000000001},"min":{"1min":1.036,"5min":0.94099999999999995,"15min":0.93400000000000005},"max":{"1min":1.6299999999999999,"5min":1.95,"15min":2.5390000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.4410000000000001,"15min":1.4399999999999999},"min":{"1min":1.1419999999999999,"5min":1.024,"15min":0.96999999999999997},"max":{"1min":1.8089999999999999,"5min":2.286,"15min":2.5179999999999998},"last":1.677}]},{"osd":5,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5780000000000001,"15min":1.587},"min":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.103},"max":{"1min":1.9930000000000001,"5min":2.2160000000000002,"15min":2.6320000000000001},"last":1.52},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.554,"15min":1.5760000000000001},"min":{"1min":1.141,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":1.8029999999999999,"5min":2.2770000000000001,"15min":2.8170000000000002},"last":1.2989999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.51,"15min":1.526},"min":{"1min":1.2010000000000001,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1099999999999999,"15min":2.2090000000000001},"last":1.363},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.605,"15min":1.619},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":2.0680000000000001,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.7689999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.6950000000000001,"15min":1.702},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.9790000000000001,"5min":2.3919999999999999,"15min":2.8780000000000001},"last":1.415},{"interface":"front","average":{"1min":1.55,"5min":1.645,"15min":1.6599999999999999},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":1.9079999999999999,"5min":2.2250000000000001,"15min":2.6699999999999999},"last":1.716}]}]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51792,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320048,"statfs":{"total":96636764160,"available":96583729152,"internally_reserved":0,"allocated":2617344,"data_stored":1276593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.94999999999999996,"15min":0.97299999999999998},"min":{"1min":0.749,"5min":0.65800000000000003,"15min":0.60099999999999998},"max":{"1min":1.716,"5min":1.716,"15min":2.165},"last":0.754},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.117,"15min":1.1140000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.45,"5min":1.762,"15min":2.0790000000000002},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0269999999999999,"15min":1.0620000000000001},"min":{"1min":0.83499999999999996,"5min":0.76600000000000001,"15min":0.72399999999999998},"max":{"1min":1.405,"5min":1.635,"15min":2.3300000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2470000000000001,"15min":1.2689999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":2.0779999999999998},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3029999999999999,"15min":1.327},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6739999999999999,"5min":1.9139999999999999,"15min":2.4780000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.363,"15min":1.387},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":2.8159999999999998},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.4970000000000001,"15min":1.5049999999999999},"min":{"1min":1.133,"5min":1.069,"15min":1.0589999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.4119999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.407,"5min":1.452,"15min":1.4470000000000001},"min":{"1min":1.032,"5min":1.0209999999999999,"15min":0.86299999999999999},"max":{"1min":1.915,"5min":2.0920000000000001,"15min":2.1589999999999998},"last":0.879}]},{"osd":5,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.5029999999999999,"15min":1.5149999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.657,"5min":2.657,"15min":2.657},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.653,"15min":1.6459999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.5569999999999999},"last":1.0089999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5800000000000001,"15min":1.581},"min":{"1min":1.0489999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":1.9610000000000001,"5min":2.1960000000000002,"15min":2.3849999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.649,"15min":1.641},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.2709999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.6870000000000001,"15min":1.6830000000000001},"min":{"1min":1.133,"5min":1.0920000000000001,"15min":1.0920000000000001},"max":{"1min":1.982,"5min":2.1080000000000001,"15min":2.4750000000000001},"last":1.256},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.742,"15min":1.758},"min":{"1min":1.3660000000000001,"5min":1.2270000000000001,"15min":1.2210000000000001},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.754},"last":1.3240000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575972,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47624,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324216,"statfs":{"total":96636764160,"available":96587997184,"internally_reserved":0,"allocated":2617344,"data_stored":1277511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.96499999999999997,"15min":0.95199999999999996},"min":{"1min":0.73699999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.5800000000000001,"5min":1.8069999999999999,"15min":1.8420000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1759999999999999,"15min":1.1739999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.603,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.875}]},{"osd":2,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0549999999999999,"15min":1.054},"min":{"1min":0.84499999999999997,"5min":0.621,"15min":0.621},"max":{"1min":1.591,"5min":1.913,"15min":2.286},"last":1.607},{"interface":"front","average":{"1min":1.258,"5min":1.266,"15min":1.252},"min":{"1min":0.91500000000000004,"5min":0.83999999999999997,"15min":0.73899999999999999},"max":{"1min":1.7090000000000001,"5min":1.806,"15min":1.853},"last":1.036}]},{"osd":3,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.397,"15min":1.4019999999999999},"min":{"1min":1.0289999999999999,"5min":0.91700000000000004,"15min":0.86699999999999999},"max":{"1min":1.8700000000000001,"5min":2.0619999999999998,"15min":2.1920000000000002},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.345,"15min":1.3480000000000001},"min":{"1min":1.0249999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.8740000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.373,"15min":1.3839999999999999},"min":{"1min":1.1459999999999999,"5min":0.871,"15min":0.86799999999999999},"max":{"1min":2.282,"5min":2.282,"15min":2.3809999999999998},"last":1.502},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.5129999999999999,"15min":1.5229999999999999},"min":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.071},"max":{"1min":2.198,"5min":2.198,"15min":2.6259999999999999},"last":1.5609999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.5349999999999999,"15min":1.538},"min":{"1min":1.0669999999999999,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":1.95,"5min":2,"15min":2.3799999999999999},"last":1.367},{"interface":"front","average":{"1min":1.494,"5min":1.4850000000000001,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.8720000000000001,"5min":2.254,"15min":2.4689999999999999},"last":1.821}]},{"osd":6,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.6499999999999999,"15min":1.6559999999999999},"min":{"1min":1.2929999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.5009999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.6100000000000001,"15min":1.653},"min":{"1min":1.264,"5min":1.135,"15min":1.135},"max":{"1min":2.024,"5min":2.2610000000000001,"15min":3.0430000000000001},"last":1.647}]},{"osd":7,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.72,"15min":1.764},"min":{"1min":1.504,"5min":1.3340000000000001,"15min":1.272},"max":{"1min":2.1309999999999998,"5min":2.2869999999999999,"15min":3.2090000000000001},"last":1.746},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.6439999999999999,"15min":1.661},"min":{"1min":1.1559999999999999,"5min":1.036,"15min":0.98699999999999999},"max":{"1min":2.1739999999999999,"5min":2.331,"15min":3.1539999999999999},"last":2.1349999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64840,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307000,"statfs":{"total":96636764160,"available":96570368000,"internally_reserved":0,"allocated":3203072,"data_stored":1866289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.97899999999999998,"15min":0.97799999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.4419999999999999,"5min":1.585,"15min":1.659},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2669999999999999,"15min":1.254},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.609,"5min":1.669,"15min":1.8029999999999999},"last":1.609}]},{"osd":2,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.3089999999999999,"15min":1.294},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.859,"5min":1.859,"15min":2.0259999999999998},"last":1.365},{"interface":"front","average":{"1min":1.173,"5min":1.252,"15min":1.2669999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.863},"last":1.22}]},{"osd":3,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.0349999999999999,"15min":1.0509999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.501},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.29},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.57199999999999995},"max":{"1min":1.9850000000000001,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.7090000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3420000000000001,"15min":1.359},"min":{"1min":0.998,"5min":0.93999999999999995,"15min":0.88800000000000001},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.8520000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.5489999999999999,"15min":1.534},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.74299999999999999},"max":{"1min":1.9099999999999999,"5min":2.452,"15min":2.452},"last":1.9099999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4870000000000001,"15min":1.5049999999999999},"min":{"1min":1.0760000000000001,"5min":1.05,"15min":0.871},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.448,"15min":1.446},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.81699999999999995},"max":{"1min":2.0859999999999999,"5min":2.282,"15min":2.282},"last":1.5369999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.6850000000000001,"15min":1.671},"min":{"1min":1.262,"5min":1.262,"15min":0.84499999999999997},"max":{"1min":2.0009999999999999,"5min":2.3279999999999998,"15min":2.7010000000000001},"last":1.776},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.595,"15min":1.599},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.0339999999999998,"5min":2.1030000000000002,"15min":2.137},"last":1.966}]},{"osd":7,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.6839999999999999,"15min":1.712},"min":{"1min":1.163,"5min":1.1120000000000001,"15min":0.877},"max":{"1min":2.2130000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":2.105},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.6339999999999999,"15min":1.6259999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":2.0390000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:50:49.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:50:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.930+0000 7f41cd928640 1 Processor -- start 2023-10-16T15:50:49.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.930+0000 7f41cd928640 1 -- start start 2023-10-16T15:50:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.931+0000 7f41cd928640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41c80cd800 0x7f41c80cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.931+0000 7f41cd928640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f41c806aeb0 0x7f41c806b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.931+0000 7f41cd928640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c80ce180 con 0x7f41c815cba0 2023-10-16T15:50:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.931+0000 7f41cd928640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c80d0500 con 0x7f41c80cd800 2023-10-16T15:50:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.931+0000 7f41cd928640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c80d07c0 con 0x7f41c806aeb0 2023-10-16T15:50:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41c80cd800 0x7f41c80cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41c80cd800 0x7f41c80cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52310/0 (socket says 172.21.15.29:52310) 2023-10-16T15:50:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 -- 172.21.15.29:0/3737456360 learned_addr learned my addr 172.21.15.29:0/3737456360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41cd127640 1 --2- 172.21.15.29:0/3737456360 >> v2:172.21.15.170:3300/0 conn(0x7f41c806aeb0 0x7f41c806b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 -- 172.21.15.29:0/3737456360 >> v2:172.21.15.170:3300/0 conn(0x7f41c806aeb0 msgr2=0x7f41c806b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3737456360 >> v2:172.21.15.170:3300/0 conn(0x7f41c806aeb0 0x7f41c806b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 -- 172.21.15.29:0/3737456360 >> v1:172.21.15.29:6789/0 conn(0x7f41c815cba0 legacy=0x7f41c8071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.932+0000 7f41c7fff640 1 -- 172.21.15.29:0/3737456360 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c80d0aa0 con 0x7f41c80cd800 2023-10-16T15:50:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.933+0000 7f41c7fff640 1 --2- 172.21.15.29:0/3737456360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c80cd800 0x7f41c80cdc40 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f41c80cd310 tx=0x7f41bc02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d302e06f77c38cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41c77fe640 1 -- 172.21.15.29:0/3737456360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc02cb10 con 0x7f41c80cd800 2023-10-16T15:50:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41c77fe640 1 -- 172.21.15.29:0/3737456360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc02cc70 con 0x7f41c80cd800 2023-10-16T15:50:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41c77fe640 1 -- 172.21.15.29:0/3737456360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0343f0 con 0x7f41c80cd800 2023-10-16T15:50:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41cd928640 1 -- 172.21.15.29:0/3737456360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c80cd800 msgr2=0x7f41c80cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41cd928640 1 --2- 172.21.15.29:0/3737456360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c80cd800 0x7f41c80cdc40 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f41c80cd310 tx=0x7f41bc02bf70 comp rx=0 tx=0).stop 2023-10-16T15:50:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41cd928640 1 -- 172.21.15.29:0/3737456360 shutdown_connections 2023-10-16T15:50:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41cd928640 1 --2- 172.21.15.29:0/3737456360 >> v2:172.21.15.170:3300/0 conn(0x7f41c806aeb0 0x7f41c806b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41cd928640 1 --2- 172.21.15.29:0/3737456360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c80cd800 0x7f41c80cdc40 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.934+0000 7f41cd928640 1 -- 172.21.15.29:0/3737456360 >> 172.21.15.29:0/3737456360 conn(0x7f41c8065ed0 msgr2=0x7f41c80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.935+0000 7f41cd928640 1 -- 172.21.15.29:0/3737456360 shutdown_connections 2023-10-16T15:50:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.935+0000 7f41cd928640 1 -- 172.21.15.29:0/3737456360 wait complete. 2023-10-16T15:50:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.936+0000 7f41cd928640 1 Processor -- start 2023-10-16T15:50:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.936+0000 7f41cd928640 1 -- start start 2023-10-16T15:50:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.937+0000 7f41cd928640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c806aeb0 0x7f41c80cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.937+0000 7f41cd928640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80cd800 0x7f41c80b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.937+0000 7f41cd928640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 0x7f41c80c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.937+0000 7f41cd928640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80cc8a0 con 0x7f41c815cba0 2023-10-16T15:50:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.937+0000 7f41cd928640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80b1670 con 0x7f41c806aeb0 2023-10-16T15:50:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.937+0000 7f41cd928640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c815ff30 con 0x7f41c80cd800 2023-10-16T15:50:49.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 0x7f41c80c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 0x7f41c80c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52326/0 (socket says 172.21.15.29:52326) 2023-10-16T15:50:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 -- 172.21.15.29:0/4087034049 learned_addr learned my addr 172.21.15.29:0/4087034049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cc926640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c806aeb0 0x7f41c80cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41c7fff640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80cd800 0x7f41c80b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 -- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80cd800 msgr2=0x7f41c80b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80cd800 0x7f41c80b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 -- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c806aeb0 msgr2=0x7f41c80cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c806aeb0 0x7f41c80cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.938+0000 7f41cd127640 1 -- 172.21.15.29:0/4087034049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc009b30 con 0x7f41c815cba0 2023-10-16T15:50:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.939+0000 7f41cd127640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 0x7f41c80c79b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f41c000c820 tx=0x7f41c000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.939+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0004020 con 0x7f41c815cba0 2023-10-16T15:50:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.939+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c0004180 con 0x7f41c815cba0 2023-10-16T15:50:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.939+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0007830 con 0x7f41c815cba0 2023-10-16T15:50:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.939+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c81f6060 con 0x7f41c815cba0 2023-10-16T15:50:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.939+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c81f6620 con 0x7f41c815cba0 2023-10-16T15:50:49.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.940+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c804e9b0 con 0x7f41c815cba0 2023-10-16T15:50:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.941+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f41c0007990 con 0x7f41c815cba0 2023-10-16T15:50:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.942+0000 7f41c57fa640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41b406af90 0x7f41b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:49.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.942+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f41c00895e0 con 0x7f41c815cba0 2023-10-16T15:50:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.943+0000 7f41cc926640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41b406af90 0x7f41b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.944+0000 7f41cc926640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41b406af90 0x7f41b406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41b8000f60 tx=0x7f41b8009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:49.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:49.946+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41c0054440 con 0x7f41c815cba0 2023-10-16T15:50:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.138+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 --> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41c80d1060 con 0x7f41b406af90 2023-10-16T15:50:50.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.162+0000 7f41c57fa640 1 -- 172.21.15.29:0/4087034049 <== mgr.7406 v2:172.21.15.170:6832/780156504 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286893 (secure 0 0 0) 0x7f41b80050d0 con 0x7f41b406af90 2023-10-16T15:50:50.167 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:50.168 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:50:50.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.168+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41b406af90 msgr2=0x7f41b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.168+0000 7f41cd928640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41b406af90 0x7f41b406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41b8000f60 tx=0x7f41b8009040 comp rx=0 tx=0).stop 2023-10-16T15:50:50.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.168+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 msgr2=0x7f41c80c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.168+0000 7f41cd928640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 0x7f41c80c79b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f41c000c820 tx=0x7f41c000cce0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 shutdown_connections 2023-10-16T15:50:50.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f41b406af90 0x7f41b406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c815cba0 0x7f41c80c79b0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41c80cd800 0x7f41c80b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 --2- 172.21.15.29:0/4087034049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41c806aeb0 0x7f41c80cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 >> 172.21.15.29:0/4087034049 conn(0x7f41c8065ed0 msgr2=0x7f41c80b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 shutdown_connections 2023-10-16T15:50:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.169+0000 7f41cd928640 1 -- 172.21.15.29:0/4087034049 wait complete. 2023-10-16T15:50:50.188 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T15:50:49.481386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":587,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":387688,"kb_used_data":21612,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754587032,"statfs":{"total":773094113280,"available":772697120768,"internally_reserved":0,"allocated":22130688,"data_stored":11411732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":374722105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002198"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674885+0000","last_change":"2023-10-16T15:50:44.660288+0000","last_active":"2023-10-16T15:50:45.674885+0000","last_peered":"2023-10-16T15:50:45.674885+0000","last_clean":"2023-10-16T15:50:45.674885+0000","last_became_active":"2023-10-16T15:50:44.659892+0000","last_became_peered":"2023-10-16T15:50:44.659892+0000","last_unstale":"2023-10-16T15:50:45.674885+0000","last_undegraded":"2023-10-16T15:50:45.674885+0000","last_fullsized":"2023-10-16T15:50:45.674885+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:48.194990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.670249+0000","last_change":"2023-10-16T15:50:44.666779+0000","last_active":"2023-10-16T15:50:44.670249+0000","last_peered":"2023-10-16T15:50:44.670249+0000","last_clean":"2023-10-16T15:50:44.670249+0000","last_became_active":"2023-10-16T15:50:44.666425+0000","last_became_peered":"2023-10-16T15:50:44.666425+0000","last_unstale":"2023-10-16T15:50:44.670249+0000","last_undegraded":"2023-10-16T15:50:44.670249+0000","last_fullsized":"2023-10-16T15:50:44.670249+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:33.048475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.677174+0000","last_change":"2023-10-16T15:50:44.674837+0000","last_active":"2023-10-16T15:50:44.677174+0000","last_peered":"2023-10-16T15:50:44.677174+0000","last_clean":"2023-10-16T15:50:44.677174+0000","last_became_active":"2023-10-16T15:50:44.674442+0000","last_became_peered":"2023-10-16T15:50:44.674442+0000","last_unstale":"2023-10-16T15:50:44.677174+0000","last_undegraded":"2023-10-16T15:50:44.677174+0000","last_fullsized":"2023-10-16T15:50:44.677174+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:02.605604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681564+0000","last_change":"2023-10-16T15:50:44.677894+0000","last_active":"2023-10-16T15:50:44.681564+0000","last_peered":"2023-10-16T15:50:44.681564+0000","last_clean":"2023-10-16T15:50:44.681564+0000","last_became_active":"2023-10-16T15:50:44.674402+0000","last_became_peered":"2023-10-16T15:50:44.674402+0000","last_unstale":"2023-10-16T15:50:44.681564+0000","last_undegraded":"2023-10-16T15:50:44.681564+0000","last_fullsized":"2023-10-16T15:50:44.681564+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:36.308210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.682108+0000","last_change":"2023-10-16T15:50:44.677119+0000","last_active":"2023-10-16T15:50:44.682108+0000","last_peered":"2023-10-16T15:50:44.682108+0000","last_clean":"2023-10-16T15:50:44.682108+0000","last_became_active":"2023-10-16T15:50:44.676859+0000","last_became_peered":"2023-10-16T15:50:44.676859+0000","last_unstale":"2023-10-16T15:50:44.682108+0000","last_undegraded":"2023-10-16T15:50:44.682108+0000","last_fullsized":"2023-10-16T15:50:44.682108+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:59.710850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.670801+0000","last_change":"2023-10-16T15:50:44.666098+0000","last_active":"2023-10-16T15:50:44.670801+0000","last_peered":"2023-10-16T15:50:44.670801+0000","last_clean":"2023-10-16T15:50:44.670801+0000","last_became_active":"2023-10-16T15:50:44.665803+0000","last_became_peered":"2023-10-16T15:50:44.665803+0000","last_unstale":"2023-10-16T15:50:44.670801+0000","last_undegraded":"2023-10-16T15:50:44.670801+0000","last_fullsized":"2023-10-16T15:50:44.670801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:56.322845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674204+0000","last_change":"2023-10-16T15:50:44.678699+0000","last_active":"2023-10-16T15:50:45.674204+0000","last_peered":"2023-10-16T15:50:45.674204+0000","last_clean":"2023-10-16T15:50:45.674204+0000","last_became_active":"2023-10-16T15:50:44.675445+0000","last_became_peered":"2023-10-16T15:50:44.675445+0000","last_unstale":"2023-10-16T15:50:45.674204+0000","last_undegraded":"2023-10-16T15:50:45.674204+0000","last_fullsized":"2023-10-16T15:50:45.674204+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:45.077152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668779+0000","last_change":"2023-10-16T15:50:44.673655+0000","last_active":"2023-10-16T15:50:45.668779+0000","last_peered":"2023-10-16T15:50:45.668779+0000","last_clean":"2023-10-16T15:50:45.668779+0000","last_became_active":"2023-10-16T15:50:44.673305+0000","last_became_peered":"2023-10-16T15:50:44.673305+0000","last_unstale":"2023-10-16T15:50:45.668779+0000","last_undegraded":"2023-10-16T15:50:45.668779+0000","last_fullsized":"2023-10-16T15:50:45.668779+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:05.676537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681284+0000","last_change":"2023-10-16T15:50:44.676168+0000","last_active":"2023-10-16T15:50:44.681284+0000","last_peered":"2023-10-16T15:50:44.681284+0000","last_clean":"2023-10-16T15:50:44.681284+0000","last_became_active":"2023-10-16T15:50:44.675859+0000","last_became_peered":"2023-10-16T15:50:44.675859+0000","last_unstale":"2023-10-16T15:50:44.681284+0000","last_undegraded":"2023-10-16T15:50:44.681284+0000","last_fullsized":"2023-10-16T15:50:44.681284+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:04.368424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477057+0000","last_change":"2023-10-16T15:50:44.659114+0000","last_active":"2023-10-16T15:50:46.477057+0000","last_peered":"2023-10-16T15:50:46.477057+0000","last_clean":"2023-10-16T15:50:46.477057+0000","last_became_active":"2023-10-16T15:50:44.658823+0000","last_became_peered":"2023-10-16T15:50:44.658823+0000","last_unstale":"2023-10-16T15:50:46.477057+0000","last_undegraded":"2023-10-16T15:50:46.477057+0000","last_fullsized":"2023-10-16T15:50:46.477057+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:48.955244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675563+0000","last_change":"2023-10-16T15:50:44.677048+0000","last_active":"2023-10-16T15:50:45.675563+0000","last_peered":"2023-10-16T15:50:45.675563+0000","last_clean":"2023-10-16T15:50:45.675563+0000","last_became_active":"2023-10-16T15:50:44.676774+0000","last_became_peered":"2023-10-16T15:50:44.676774+0000","last_unstale":"2023-10-16T15:50:45.675563+0000","last_undegraded":"2023-10-16T15:50:45.675563+0000","last_fullsized":"2023-10-16T15:50:45.675563+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:27.820962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681544+0000","last_change":"2023-10-16T15:50:44.675010+0000","last_active":"2023-10-16T15:50:44.681544+0000","last_peered":"2023-10-16T15:50:44.681544+0000","last_clean":"2023-10-16T15:50:44.681544+0000","last_became_active":"2023-10-16T15:50:44.673988+0000","last_became_peered":"2023-10-16T15:50:44.673988+0000","last_unstale":"2023-10-16T15:50:44.681544+0000","last_undegraded":"2023-10-16T15:50:44.681544+0000","last_fullsized":"2023-10-16T15:50:44.681544+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:42.567049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684087+0000","last_change":"2023-10-16T15:50:44.683501+0000","last_active":"2023-10-16T15:50:44.684087+0000","last_peered":"2023-10-16T15:50:44.684087+0000","last_clean":"2023-10-16T15:50:44.684087+0000","last_became_active":"2023-10-16T15:50:44.682120+0000","last_became_peered":"2023-10-16T15:50:44.682120+0000","last_unstale":"2023-10-16T15:50:44.684087+0000","last_undegraded":"2023-10-16T15:50:44.684087+0000","last_fullsized":"2023-10-16T15:50:44.684087+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:41.682891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.669744+0000","last_change":"2023-10-16T15:50:44.663126+0000","last_active":"2023-10-16T15:50:44.669744+0000","last_peered":"2023-10-16T15:50:44.669744+0000","last_clean":"2023-10-16T15:50:44.669744+0000","last_became_active":"2023-10-16T15:50:44.662838+0000","last_became_peered":"2023-10-16T15:50:44.662838+0000","last_unstale":"2023-10-16T15:50:44.669744+0000","last_undegraded":"2023-10-16T15:50:44.669744+0000","last_fullsized":"2023-10-16T15:50:44.669744+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:05.143314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680107+0000","last_change":"2023-10-16T15:50:44.671958+0000","last_active":"2023-10-16T15:50:44.680107+0000","last_peered":"2023-10-16T15:50:44.680107+0000","last_clean":"2023-10-16T15:50:44.680107+0000","last_became_active":"2023-10-16T15:50:44.671578+0000","last_became_peered":"2023-10-16T15:50:44.671578+0000","last_unstale":"2023-10-16T15:50:44.680107+0000","last_undegraded":"2023-10-16T15:50:44.680107+0000","last_fullsized":"2023-10-16T15:50:44.680107+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:28.836451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668779+0000","last_change":"2023-10-16T15:50:44.667019+0000","last_active":"2023-10-16T15:50:45.668779+0000","last_peered":"2023-10-16T15:50:45.668779+0000","last_clean":"2023-10-16T15:50:45.668779+0000","last_became_active":"2023-10-16T15:50:44.666753+0000","last_became_peered":"2023-10-16T15:50:44.666753+0000","last_unstale":"2023-10-16T15:50:45.668779+0000","last_undegraded":"2023-10-16T15:50:45.668779+0000","last_fullsized":"2023-10-16T15:50:45.668779+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:09.444429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685232+0000","last_change":"2023-10-16T15:50:44.674962+0000","last_active":"2023-10-16T15:50:44.685232+0000","last_peered":"2023-10-16T15:50:44.685232+0000","last_clean":"2023-10-16T15:50:44.685232+0000","last_became_active":"2023-10-16T15:50:44.674562+0000","last_became_peered":"2023-10-16T15:50:44.674562+0000","last_unstale":"2023-10-16T15:50:44.685232+0000","last_undegraded":"2023-10-16T15:50:44.685232+0000","last_fullsized":"2023-10-16T15:50:44.685232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:02.605604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675297+0000","last_change":"2023-10-16T15:50:44.666379+0000","last_active":"2023-10-16T15:50:45.675297+0000","last_peered":"2023-10-16T15:50:45.675297+0000","last_clean":"2023-10-16T15:50:45.675297+0000","last_became_active":"2023-10-16T15:50:44.665361+0000","last_became_peered":"2023-10-16T15:50:44.665361+0000","last_unstale":"2023-10-16T15:50:45.675297+0000","last_undegraded":"2023-10-16T15:50:45.675297+0000","last_fullsized":"2023-10-16T15:50:45.675297+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:13.733047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476428+0000","last_change":"2023-10-16T15:50:44.669691+0000","last_active":"2023-10-16T15:50:46.476428+0000","last_peered":"2023-10-16T15:50:46.476428+0000","last_clean":"2023-10-16T15:50:46.476428+0000","last_became_active":"2023-10-16T15:50:44.669408+0000","last_became_peered":"2023-10-16T15:50:44.669408+0000","last_unstale":"2023-10-16T15:50:46.476428+0000","last_undegraded":"2023-10-16T15:50:46.476428+0000","last_fullsized":"2023-10-16T15:50:46.476428+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:07.291215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685415+0000","last_change":"2023-10-16T15:50:44.684698+0000","last_active":"2023-10-16T15:50:44.685415+0000","last_peered":"2023-10-16T15:50:44.685415+0000","last_clean":"2023-10-16T15:50:44.685415+0000","last_became_active":"2023-10-16T15:50:44.683606+0000","last_became_peered":"2023-10-16T15:50:44.683606+0000","last_unstale":"2023-10-16T15:50:44.685415+0000","last_undegraded":"2023-10-16T15:50:44.685415+0000","last_fullsized":"2023-10-16T15:50:44.685415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:56.322845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.682088+0000","last_change":"2023-10-16T15:50:44.675425+0000","last_active":"2023-10-16T15:50:44.682088+0000","last_peered":"2023-10-16T15:50:44.682088+0000","last_clean":"2023-10-16T15:50:44.682088+0000","last_became_active":"2023-10-16T15:50:44.675138+0000","last_became_peered":"2023-10-16T15:50:44.675138+0000","last_unstale":"2023-10-16T15:50:44.682088+0000","last_undegraded":"2023-10-16T15:50:44.682088+0000","last_fullsized":"2023-10-16T15:50:44.682088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:02.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476231+0000","last_change":"2023-10-16T15:50:44.660710+0000","last_active":"2023-10-16T15:50:46.476231+0000","last_peered":"2023-10-16T15:50:46.476231+0000","last_clean":"2023-10-16T15:50:46.476231+0000","last_became_active":"2023-10-16T15:50:44.660424+0000","last_became_peered":"2023-10-16T15:50:44.660424+0000","last_unstale":"2023-10-16T15:50:46.476231+0000","last_undegraded":"2023-10-16T15:50:46.476231+0000","last_fullsized":"2023-10-16T15:50:46.476231+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:51.317016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.666593+0000","last_change":"2023-10-16T15:50:44.659302+0000","last_active":"2023-10-16T15:50:44.666593+0000","last_peered":"2023-10-16T15:50:44.666593+0000","last_clean":"2023-10-16T15:50:44.666593+0000","last_became_active":"2023-10-16T15:50:44.659003+0000","last_became_peered":"2023-10-16T15:50:44.659003+0000","last_unstale":"2023-10-16T15:50:44.666593+0000","last_undegraded":"2023-10-16T15:50:44.666593+0000","last_fullsized":"2023-10-16T15:50:44.666593+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:49.488183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668944+0000","last_change":"2023-10-16T15:50:44.656189+0000","last_active":"2023-10-16T15:50:45.668944+0000","last_peered":"2023-10-16T15:50:45.668944+0000","last_clean":"2023-10-16T15:50:45.668944+0000","last_became_active":"2023-10-16T15:50:44.655904+0000","last_became_peered":"2023-10-16T15:50:44.655904+0000","last_unstale":"2023-10-16T15:50:45.668944+0000","last_undegraded":"2023-10-16T15:50:45.668944+0000","last_fullsized":"2023-10-16T15:50:45.668944+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:32.326238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.683725+0000","last_change":"2023-10-16T15:50:44.682004+0000","last_active":"2023-10-16T15:50:44.683725+0000","last_peered":"2023-10-16T15:50:44.683725+0000","last_clean":"2023-10-16T15:50:44.683725+0000","last_became_active":"2023-10-16T15:50:44.681649+0000","last_became_peered":"2023-10-16T15:50:44.681649+0000","last_unstale":"2023-10-16T15:50:44.683725+0000","last_undegraded":"2023-10-16T15:50:44.683725+0000","last_fullsized":"2023-10-16T15:50:44.683725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:38.868853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.673916+0000","last_change":"2023-10-16T15:50:44.670319+0000","last_active":"2023-10-16T15:50:45.673916+0000","last_peered":"2023-10-16T15:50:45.673916+0000","last_clean":"2023-10-16T15:50:45.673916+0000","last_became_active":"2023-10-16T15:50:44.669950+0000","last_became_peered":"2023-10-16T15:50:44.669950+0000","last_unstale":"2023-10-16T15:50:45.673916+0000","last_undegraded":"2023-10-16T15:50:45.673916+0000","last_fullsized":"2023-10-16T15:50:45.673916+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:02.605604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.668160+0000","last_change":"2023-10-16T15:50:44.664547+0000","last_active":"2023-10-16T15:50:44.668160+0000","last_peered":"2023-10-16T15:50:44.668160+0000","last_clean":"2023-10-16T15:50:44.668160+0000","last_became_active":"2023-10-16T15:50:44.664254+0000","last_became_peered":"2023-10-16T15:50:44.664254+0000","last_unstale":"2023-10-16T15:50:44.668160+0000","last_undegraded":"2023-10-16T15:50:44.668160+0000","last_fullsized":"2023-10-16T15:50:44.668160+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:07.426576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.679498+0000","last_change":"2023-10-16T15:50:44.667825+0000","last_active":"2023-10-16T15:50:44.679498+0000","last_peered":"2023-10-16T15:50:44.679498+0000","last_clean":"2023-10-16T15:50:44.679498+0000","last_became_active":"2023-10-16T15:50:44.667512+0000","last_became_peered":"2023-10-16T15:50:44.667512+0000","last_unstale":"2023-10-16T15:50:44.679498+0000","last_undegraded":"2023-10-16T15:50:44.679498+0000","last_fullsized":"2023-10-16T15:50:44.679498+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:38.915700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.673320+0000","last_change":"2023-10-16T15:50:44.668024+0000","last_active":"2023-10-16T15:50:44.673320+0000","last_peered":"2023-10-16T15:50:44.673320+0000","last_clean":"2023-10-16T15:50:44.673320+0000","last_became_active":"2023-10-16T15:50:44.667724+0000","last_became_peered":"2023-10-16T15:50:44.667724+0000","last_unstale":"2023-10-16T15:50:44.673320+0000","last_undegraded":"2023-10-16T15:50:44.673320+0000","last_fullsized":"2023-10-16T15:50:44.673320+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:36.097780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.672923+0000","last_change":"2023-10-16T15:50:44.670185+0000","last_active":"2023-10-16T15:50:44.672923+0000","last_peered":"2023-10-16T15:50:44.672923+0000","last_clean":"2023-10-16T15:50:44.672923+0000","last_became_active":"2023-10-16T15:50:44.667024+0000","last_became_peered":"2023-10-16T15:50:44.667024+0000","last_unstale":"2023-10-16T15:50:44.672923+0000","last_undegraded":"2023-10-16T15:50:44.672923+0000","last_fullsized":"2023-10-16T15:50:44.672923+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:10.952350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.676607+0000","last_change":"2023-10-16T15:50:44.674047+0000","last_active":"2023-10-16T15:50:44.676607+0000","last_peered":"2023-10-16T15:50:44.676607+0000","last_clean":"2023-10-16T15:50:44.676607+0000","last_became_active":"2023-10-16T15:50:44.673740+0000","last_became_peered":"2023-10-16T15:50:44.673740+0000","last_unstale":"2023-10-16T15:50:44.676607+0000","last_undegraded":"2023-10-16T15:50:44.676607+0000","last_fullsized":"2023-10-16T15:50:44.676607+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:02.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.678198+0000","last_change":"2023-10-16T15:50:44.666444+0000","last_active":"2023-10-16T15:50:44.678198+0000","last_peered":"2023-10-16T15:50:44.678198+0000","last_clean":"2023-10-16T15:50:44.678198+0000","last_became_active":"2023-10-16T15:50:44.666173+0000","last_became_peered":"2023-10-16T15:50:44.666173+0000","last_unstale":"2023-10-16T15:50:44.678198+0000","last_undegraded":"2023-10-16T15:50:44.678198+0000","last_fullsized":"2023-10-16T15:50:44.678198+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:07.003930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669718+0000","last_change":"2023-10-16T15:50:44.669259+0000","last_active":"2023-10-16T15:50:45.669718+0000","last_peered":"2023-10-16T15:50:45.669718+0000","last_clean":"2023-10-16T15:50:45.669718+0000","last_became_active":"2023-10-16T15:50:44.668971+0000","last_became_peered":"2023-10-16T15:50:44.668971+0000","last_unstale":"2023-10-16T15:50:45.669718+0000","last_undegraded":"2023-10-16T15:50:45.669718+0000","last_fullsized":"2023-10-16T15:50:45.669718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:59.675199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"208'4","reported_seq":15,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685159+0000","last_change":"2023-10-16T15:50:43.645418+0000","last_active":"2023-10-16T15:50:44.685159+0000","last_peered":"2023-10-16T15:50:44.685159+0000","last_clean":"2023-10-16T15:50:44.685159+0000","last_became_active":"2023-10-16T15:50:43.644948+0000","last_became_peered":"2023-10-16T15:50:43.644948+0000","last_unstale":"2023-10-16T15:50:44.685159+0000","last_undegraded":"2023-10-16T15:50:44.685159+0000","last_fullsized":"2023-10-16T15:50:44.685159+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:37.156252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477692+0000","last_change":"2023-10-16T15:50:44.676056+0000","last_active":"2023-10-16T15:50:46.477692+0000","last_peered":"2023-10-16T15:50:46.477692+0000","last_clean":"2023-10-16T15:50:46.477692+0000","last_became_active":"2023-10-16T15:50:44.675788+0000","last_became_peered":"2023-10-16T15:50:44.675788+0000","last_unstale":"2023-10-16T15:50:46.477692+0000","last_undegraded":"2023-10-16T15:50:46.477692+0000","last_fullsized":"2023-10-16T15:50:46.477692+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:41:33.255162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684934+0000","last_change":"2023-10-16T15:50:43.631670+0000","last_active":"2023-10-16T15:50:44.684934+0000","last_peered":"2023-10-16T15:50:44.684934+0000","last_clean":"2023-10-16T15:50:44.684934+0000","last_became_active":"2023-10-16T15:50:43.631226+0000","last_became_peered":"2023-10-16T15:50:43.631226+0000","last_unstale":"2023-10-16T15:50:44.684934+0000","last_undegraded":"2023-10-16T15:50:44.684934+0000","last_fullsized":"2023-10-16T15:50:44.684934+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:02.797454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675361+0000","last_change":"2023-10-16T15:50:44.680661+0000","last_active":"2023-10-16T15:50:45.675361+0000","last_peered":"2023-10-16T15:50:45.675361+0000","last_clean":"2023-10-16T15:50:45.675361+0000","last_became_active":"2023-10-16T15:50:44.679585+0000","last_became_peered":"2023-10-16T15:50:44.679585+0000","last_unstale":"2023-10-16T15:50:45.675361+0000","last_undegraded":"2023-10-16T15:50:45.675361+0000","last_fullsized":"2023-10-16T15:50:45.675361+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:10.938622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"208'2","reported_seq":13,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.676395+0000","last_change":"2023-10-16T15:50:43.639994+0000","last_active":"2023-10-16T15:50:44.676395+0000","last_peered":"2023-10-16T15:50:44.676395+0000","last_clean":"2023-10-16T15:50:44.676395+0000","last_became_active":"2023-10-16T15:50:43.639664+0000","last_became_peered":"2023-10-16T15:50:43.639664+0000","last_unstale":"2023-10-16T15:50:44.676395+0000","last_undegraded":"2023-10-16T15:50:44.676395+0000","last_fullsized":"2023-10-16T15:50:44.676395+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:09.927764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.672347+0000","last_change":"2023-10-16T15:50:44.669625+0000","last_active":"2023-10-16T15:50:44.672347+0000","last_peered":"2023-10-16T15:50:44.672347+0000","last_clean":"2023-10-16T15:50:44.672347+0000","last_became_active":"2023-10-16T15:50:44.669314+0000","last_became_peered":"2023-10-16T15:50:44.669314+0000","last_unstale":"2023-10-16T15:50:44.672347+0000","last_undegraded":"2023-10-16T15:50:44.672347+0000","last_fullsized":"2023-10-16T15:50:44.672347+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:18:06.007151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"208'2","reported_seq":13,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684330+0000","last_change":"2023-10-16T15:50:43.639184+0000","last_active":"2023-10-16T15:50:44.684330+0000","last_peered":"2023-10-16T15:50:44.684330+0000","last_clean":"2023-10-16T15:50:44.684330+0000","last_became_active":"2023-10-16T15:50:43.637717+0000","last_became_peered":"2023-10-16T15:50:43.637717+0000","last_unstale":"2023-10-16T15:50:44.684330+0000","last_undegraded":"2023-10-16T15:50:44.684330+0000","last_fullsized":"2023-10-16T15:50:44.684330+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:18.944252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476599+0000","last_change":"2023-10-16T15:50:44.657933+0000","last_active":"2023-10-16T15:50:46.476599+0000","last_peered":"2023-10-16T15:50:46.476599+0000","last_clean":"2023-10-16T15:50:46.476599+0000","last_became_active":"2023-10-16T15:50:44.657527+0000","last_became_peered":"2023-10-16T15:50:44.657527+0000","last_unstale":"2023-10-16T15:50:46.476599+0000","last_undegraded":"2023-10-16T15:50:46.476599+0000","last_fullsized":"2023-10-16T15:50:46.476599+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:32.355617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681117+0000","last_change":"2023-10-16T15:50:43.645704+0000","last_active":"2023-10-16T15:50:44.681117+0000","last_peered":"2023-10-16T15:50:44.681117+0000","last_clean":"2023-10-16T15:50:44.681117+0000","last_became_active":"2023-10-16T15:50:43.643170+0000","last_became_peered":"2023-10-16T15:50:43.643170+0000","last_unstale":"2023-10-16T15:50:44.681117+0000","last_undegraded":"2023-10-16T15:50:44.681117+0000","last_fullsized":"2023-10-16T15:50:44.681117+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:06.415718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669627+0000","last_change":"2023-10-16T15:50:44.669139+0000","last_active":"2023-10-16T15:50:45.669627+0000","last_peered":"2023-10-16T15:50:45.669627+0000","last_clean":"2023-10-16T15:50:45.669627+0000","last_became_active":"2023-10-16T15:50:44.668868+0000","last_became_peered":"2023-10-16T15:50:44.668868+0000","last_unstale":"2023-10-16T15:50:45.669627+0000","last_undegraded":"2023-10-16T15:50:45.669627+0000","last_fullsized":"2023-10-16T15:50:45.669627+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:24.042552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"208'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669239+0000","last_change":"2023-10-16T15:50:43.641041+0000","last_active":"2023-10-16T15:50:45.669239+0000","last_peered":"2023-10-16T15:50:45.669239+0000","last_clean":"2023-10-16T15:50:45.669239+0000","last_became_active":"2023-10-16T15:50:43.640738+0000","last_became_peered":"2023-10-16T15:50:43.640738+0000","last_unstale":"2023-10-16T15:50:45.669239+0000","last_undegraded":"2023-10-16T15:50:45.669239+0000","last_fullsized":"2023-10-16T15:50:45.669239+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:07.245737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674411+0000","last_change":"2023-10-16T15:50:44.675275+0000","last_active":"2023-10-16T15:50:45.674411+0000","last_peered":"2023-10-16T15:50:45.674411+0000","last_clean":"2023-10-16T15:50:45.674411+0000","last_became_active":"2023-10-16T15:50:44.675011+0000","last_became_peered":"2023-10-16T15:50:44.675011+0000","last_unstale":"2023-10-16T15:50:45.674411+0000","last_undegraded":"2023-10-16T15:50:45.674411+0000","last_fullsized":"2023-10-16T15:50:45.674411+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:15.786243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.672628+0000","last_change":"2023-10-16T15:50:43.644935+0000","last_active":"2023-10-16T15:50:45.672628+0000","last_peered":"2023-10-16T15:50:45.672628+0000","last_clean":"2023-10-16T15:50:45.672628+0000","last_became_active":"2023-10-16T15:50:43.644495+0000","last_became_peered":"2023-10-16T15:50:43.644495+0000","last_unstale":"2023-10-16T15:50:45.672628+0000","last_undegraded":"2023-10-16T15:50:45.672628+0000","last_fullsized":"2023-10-16T15:50:45.672628+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:07.245737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.668149+0000","last_change":"2023-10-16T15:50:44.665018+0000","last_active":"2023-10-16T15:50:44.668149+0000","last_peered":"2023-10-16T15:50:44.668149+0000","last_clean":"2023-10-16T15:50:44.668149+0000","last_became_active":"2023-10-16T15:50:44.663070+0000","last_became_peered":"2023-10-16T15:50:44.663070+0000","last_unstale":"2023-10-16T15:50:44.668149+0000","last_undegraded":"2023-10-16T15:50:44.668149+0000","last_fullsized":"2023-10-16T15:50:44.668149+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:59.710850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675994+0000","last_change":"2023-10-16T15:50:43.640236+0000","last_active":"2023-10-16T15:50:45.675994+0000","last_peered":"2023-10-16T15:50:45.675994+0000","last_clean":"2023-10-16T15:50:45.675994+0000","last_became_active":"2023-10-16T15:50:43.639800+0000","last_became_peered":"2023-10-16T15:50:43.639800+0000","last_unstale":"2023-10-16T15:50:45.675994+0000","last_undegraded":"2023-10-16T15:50:45.675994+0000","last_fullsized":"2023-10-16T15:50:45.675994+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:55.311987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669998+0000","last_change":"2023-10-16T15:50:44.663216+0000","last_active":"2023-10-16T15:50:45.669998+0000","last_peered":"2023-10-16T15:50:45.669998+0000","last_clean":"2023-10-16T15:50:45.669998+0000","last_became_active":"2023-10-16T15:50:44.662950+0000","last_became_peered":"2023-10-16T15:50:44.662950+0000","last_unstale":"2023-10-16T15:50:45.669998+0000","last_undegraded":"2023-10-16T15:50:45.669998+0000","last_fullsized":"2023-10-16T15:50:45.669998+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:41:13.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.657846+0000","last_change":"2023-10-16T15:50:43.649647+0000","last_active":"2023-10-16T15:50:44.657846+0000","last_peered":"2023-10-16T15:50:44.657846+0000","last_clean":"2023-10-16T15:50:44.657846+0000","last_became_active":"2023-10-16T15:50:43.647770+0000","last_became_peered":"2023-10-16T15:50:43.647770+0000","last_unstale":"2023-10-16T15:50:44.657846+0000","last_undegraded":"2023-10-16T15:50:44.657846+0000","last_fullsized":"2023-10-16T15:50:44.657846+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:14.775386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675730+0000","last_change":"2023-10-16T15:50:44.678626+0000","last_active":"2023-10-16T15:50:45.675730+0000","last_peered":"2023-10-16T15:50:45.675730+0000","last_clean":"2023-10-16T15:50:45.675730+0000","last_became_active":"2023-10-16T15:50:44.678351+0000","last_became_peered":"2023-10-16T15:50:44.678351+0000","last_unstale":"2023-10-16T15:50:45.675730+0000","last_undegraded":"2023-10-16T15:50:45.675730+0000","last_fullsized":"2023-10-16T15:50:45.675730+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:25:17.782187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"208'3","reported_seq":14,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.684385+0000","last_change":"2023-10-16T15:50:43.644264+0000","last_active":"2023-10-16T15:50:44.684385+0000","last_peered":"2023-10-16T15:50:44.684385+0000","last_clean":"2023-10-16T15:50:44.684385+0000","last_became_active":"2023-10-16T15:50:43.643988+0000","last_became_peered":"2023-10-16T15:50:43.643988+0000","last_unstale":"2023-10-16T15:50:44.684385+0000","last_undegraded":"2023-10-16T15:50:44.684385+0000","last_fullsized":"2023-10-16T15:50:44.684385+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:48.477325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477006+0000","last_change":"2023-10-16T15:50:44.663034+0000","last_active":"2023-10-16T15:50:46.477006+0000","last_peered":"2023-10-16T15:50:46.477006+0000","last_clean":"2023-10-16T15:50:46.477006+0000","last_became_active":"2023-10-16T15:50:44.661134+0000","last_became_peered":"2023-10-16T15:50:44.661134+0000","last_unstale":"2023-10-16T15:50:46.477006+0000","last_undegraded":"2023-10-16T15:50:46.477006+0000","last_fullsized":"2023-10-16T15:50:46.477006+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:19.162796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"208'6","reported_seq":17,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.685774+0000","last_change":"2023-10-16T15:50:43.645522+0000","last_active":"2023-10-16T15:50:44.685774+0000","last_peered":"2023-10-16T15:50:44.685774+0000","last_clean":"2023-10-16T15:50:44.685774+0000","last_became_active":"2023-10-16T15:50:43.644816+0000","last_became_peered":"2023-10-16T15:50:43.644816+0000","last_unstale":"2023-10-16T15:50:44.685774+0000","last_undegraded":"2023-10-16T15:50:44.685774+0000","last_fullsized":"2023-10-16T15:50:44.685774+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:37.156252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.668239+0000","last_change":"2023-10-16T15:50:44.659452+0000","last_active":"2023-10-16T15:50:44.668239+0000","last_peered":"2023-10-16T15:50:44.668239+0000","last_clean":"2023-10-16T15:50:44.668239+0000","last_became_active":"2023-10-16T15:50:44.659146+0000","last_became_peered":"2023-10-16T15:50:44.659146+0000","last_unstale":"2023-10-16T15:50:44.668239+0000","last_undegraded":"2023-10-16T15:50:44.668239+0000","last_fullsized":"2023-10-16T15:50:44.668239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:38.915700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"208'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675231+0000","last_change":"2023-10-16T15:50:43.643008+0000","last_active":"2023-10-16T15:50:45.675231+0000","last_peered":"2023-10-16T15:50:45.675231+0000","last_clean":"2023-10-16T15:50:45.675231+0000","last_became_active":"2023-10-16T15:50:43.642704+0000","last_became_peered":"2023-10-16T15:50:43.642704+0000","last_unstale":"2023-10-16T15:50:45.675231+0000","last_undegraded":"2023-10-16T15:50:45.675231+0000","last_fullsized":"2023-10-16T15:50:45.675231+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:13.902662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674571+0000","last_change":"2023-10-16T15:50:44.678222+0000","last_active":"2023-10-16T15:50:45.674571+0000","last_peered":"2023-10-16T15:50:45.674571+0000","last_clean":"2023-10-16T15:50:45.674571+0000","last_became_active":"2023-10-16T15:50:44.677856+0000","last_became_peered":"2023-10-16T15:50:44.677856+0000","last_unstale":"2023-10-16T15:50:45.674571+0000","last_undegraded":"2023-10-16T15:50:45.674571+0000","last_fullsized":"2023-10-16T15:50:45.674571+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:38.167110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476850+0000","last_change":"2023-10-16T15:50:43.641577+0000","last_active":"2023-10-16T15:50:46.476850+0000","last_peered":"2023-10-16T15:50:46.476850+0000","last_clean":"2023-10-16T15:50:46.476850+0000","last_became_active":"2023-10-16T15:50:43.641280+0000","last_became_peered":"2023-10-16T15:50:43.641280+0000","last_unstale":"2023-10-16T15:50:46.476850+0000","last_undegraded":"2023-10-16T15:50:46.476850+0000","last_fullsized":"2023-10-16T15:50:46.476850+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:35.086923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.670175+0000","last_change":"2023-10-16T15:50:44.660076+0000","last_active":"2023-10-16T15:50:44.670175+0000","last_peered":"2023-10-16T15:50:44.670175+0000","last_clean":"2023-10-16T15:50:44.670175+0000","last_became_active":"2023-10-16T15:50:44.658027+0000","last_became_peered":"2023-10-16T15:50:44.658027+0000","last_unstale":"2023-10-16T15:50:44.670175+0000","last_undegraded":"2023-10-16T15:50:44.670175+0000","last_fullsized":"2023-10-16T15:50:44.670175+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:40.247155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.668690+0000","last_change":"2023-10-16T15:50:43.647240+0000","last_active":"2023-10-16T15:50:45.668690+0000","last_peered":"2023-10-16T15:50:45.668690+0000","last_clean":"2023-10-16T15:50:45.668690+0000","last_became_active":"2023-10-16T15:50:43.645514+0000","last_became_peered":"2023-10-16T15:50:43.645514+0000","last_unstale":"2023-10-16T15:50:45.668690+0000","last_undegraded":"2023-10-16T15:50:45.668690+0000","last_fullsized":"2023-10-16T15:50:45.668690+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:38.342357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476048+0000","last_change":"2023-10-16T15:50:44.670849+0000","last_active":"2023-10-16T15:50:46.476048+0000","last_peered":"2023-10-16T15:50:46.476048+0000","last_clean":"2023-10-16T15:50:46.476048+0000","last_became_active":"2023-10-16T15:50:44.670571+0000","last_became_peered":"2023-10-16T15:50:44.670571+0000","last_unstale":"2023-10-16T15:50:46.476048+0000","last_undegraded":"2023-10-16T15:50:46.476048+0000","last_fullsized":"2023-10-16T15:50:46.476048+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:50.435932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"208'8","reported_seq":20,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.672793+0000","last_change":"2023-10-16T15:50:43.645287+0000","last_active":"2023-10-16T15:50:45.672793+0000","last_peered":"2023-10-16T15:50:45.672793+0000","last_clean":"2023-10-16T15:50:45.672793+0000","last_became_active":"2023-10-16T15:50:43.644908+0000","last_became_peered":"2023-10-16T15:50:43.644908+0000","last_unstale":"2023-10-16T15:50:45.672793+0000","last_undegraded":"2023-10-16T15:50:45.672793+0000","last_fullsized":"2023-10-16T15:50:45.672793+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:13.937593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477300+0000","last_change":"2023-10-16T15:50:44.672936+0000","last_active":"2023-10-16T15:50:46.477300+0000","last_peered":"2023-10-16T15:50:46.477300+0000","last_clean":"2023-10-16T15:50:46.477300+0000","last_became_active":"2023-10-16T15:50:44.672648+0000","last_became_peered":"2023-10-16T15:50:44.672648+0000","last_unstale":"2023-10-16T15:50:46.477300+0000","last_undegraded":"2023-10-16T15:50:46.477300+0000","last_fullsized":"2023-10-16T15:50:46.477300+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:38.753648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"208'1","reported_seq":12,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.683784+0000","last_change":"2023-10-16T15:50:43.643170+0000","last_active":"2023-10-16T15:50:44.683784+0000","last_peered":"2023-10-16T15:50:44.683784+0000","last_clean":"2023-10-16T15:50:44.683784+0000","last_became_active":"2023-10-16T15:50:43.642896+0000","last_became_peered":"2023-10-16T15:50:43.642896+0000","last_unstale":"2023-10-16T15:50:44.683784+0000","last_undegraded":"2023-10-16T15:50:44.683784+0000","last_fullsized":"2023-10-16T15:50:44.683784+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:49.420248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680556+0000","last_change":"2023-10-16T15:50:44.672955+0000","last_active":"2023-10-16T15:50:44.680556+0000","last_peered":"2023-10-16T15:50:44.680556+0000","last_clean":"2023-10-16T15:50:44.680556+0000","last_became_active":"2023-10-16T15:50:44.672661+0000","last_became_peered":"2023-10-16T15:50:44.672661+0000","last_unstale":"2023-10-16T15:50:44.680556+0000","last_undegraded":"2023-10-16T15:50:44.680556+0000","last_fullsized":"2023-10-16T15:50:44.680556+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:34.004737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"208'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669758+0000","last_change":"2023-10-16T15:50:43.642282+0000","last_active":"2023-10-16T15:50:45.669758+0000","last_peered":"2023-10-16T15:50:45.669758+0000","last_clean":"2023-10-16T15:50:45.669758+0000","last_became_active":"2023-10-16T15:50:43.641965+0000","last_became_peered":"2023-10-16T15:50:43.641965+0000","last_unstale":"2023-10-16T15:50:45.669758+0000","last_undegraded":"2023-10-16T15:50:45.669758+0000","last_fullsized":"2023-10-16T15:50:45.669758+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:43.125218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.675041+0000","last_change":"2023-10-16T15:50:44.668998+0000","last_active":"2023-10-16T15:50:44.675041+0000","last_peered":"2023-10-16T15:50:44.675041+0000","last_clean":"2023-10-16T15:50:44.675041+0000","last_became_active":"2023-10-16T15:50:44.668158+0000","last_became_peered":"2023-10-16T15:50:44.668158+0000","last_unstale":"2023-10-16T15:50:44.675041+0000","last_undegraded":"2023-10-16T15:50:44.675041+0000","last_fullsized":"2023-10-16T15:50:44.675041+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:32.111306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"208'1","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674762+0000","last_change":"2023-10-16T15:50:43.645363+0000","last_active":"2023-10-16T15:50:45.674762+0000","last_peered":"2023-10-16T15:50:45.674762+0000","last_clean":"2023-10-16T15:50:45.674762+0000","last_became_active":"2023-10-16T15:50:43.645069+0000","last_became_peered":"2023-10-16T15:50:43.645069+0000","last_unstale":"2023-10-16T15:50:45.674762+0000","last_undegraded":"2023-10-16T15:50:45.674762+0000","last_fullsized":"2023-10-16T15:50:45.674762+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:20.480540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.682134+0000","last_change":"2023-10-16T15:50:44.674754+0000","last_active":"2023-10-16T15:50:44.682134+0000","last_peered":"2023-10-16T15:50:44.682134+0000","last_clean":"2023-10-16T15:50:44.682134+0000","last_became_active":"2023-10-16T15:50:44.674485+0000","last_became_peered":"2023-10-16T15:50:44.674485+0000","last_unstale":"2023-10-16T15:50:44.682134+0000","last_undegraded":"2023-10-16T15:50:44.682134+0000","last_fullsized":"2023-10-16T15:50:44.682134+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:40.868217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675739+0000","last_change":"2023-10-16T15:50:43.648761+0000","last_active":"2023-10-16T15:50:45.675739+0000","last_peered":"2023-10-16T15:50:45.675739+0000","last_clean":"2023-10-16T15:50:45.675739+0000","last_became_active":"2023-10-16T15:50:43.648425+0000","last_became_peered":"2023-10-16T15:50:43.648425+0000","last_unstale":"2023-10-16T15:50:45.675739+0000","last_undegraded":"2023-10-16T15:50:45.675739+0000","last_fullsized":"2023-10-16T15:50:45.675739+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:37.904842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476631+0000","last_change":"2023-10-16T15:50:44.664122+0000","last_active":"2023-10-16T15:50:46.476631+0000","last_peered":"2023-10-16T15:50:46.476631+0000","last_clean":"2023-10-16T15:50:46.476631+0000","last_became_active":"2023-10-16T15:50:44.663798+0000","last_became_peered":"2023-10-16T15:50:44.663798+0000","last_unstale":"2023-10-16T15:50:46.476631+0000","last_undegraded":"2023-10-16T15:50:46.476631+0000","last_fullsized":"2023-10-16T15:50:46.476631+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:51.960978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"208'2","reported_seq":13,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680820+0000","last_change":"2023-10-16T15:50:43.642605+0000","last_active":"2023-10-16T15:50:44.680820+0000","last_peered":"2023-10-16T15:50:44.680820+0000","last_clean":"2023-10-16T15:50:44.680820+0000","last_became_active":"2023-10-16T15:50:43.642305+0000","last_became_peered":"2023-10-16T15:50:43.642305+0000","last_unstale":"2023-10-16T15:50:44.680820+0000","last_undegraded":"2023-10-16T15:50:44.680820+0000","last_fullsized":"2023-10-16T15:50:44.680820+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:44.066295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674925+0000","last_change":"2023-10-16T15:50:44.678526+0000","last_active":"2023-10-16T15:50:45.674925+0000","last_peered":"2023-10-16T15:50:45.674925+0000","last_clean":"2023-10-16T15:50:45.674925+0000","last_became_active":"2023-10-16T15:50:44.675146+0000","last_became_peered":"2023-10-16T15:50:44.675146+0000","last_unstale":"2023-10-16T15:50:45.674925+0000","last_undegraded":"2023-10-16T15:50:45.674925+0000","last_fullsized":"2023-10-16T15:50:45.674925+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:50.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.654222+0000","last_change":"2023-10-16T15:50:43.644608+0000","last_active":"2023-10-16T15:50:44.654222+0000","last_peered":"2023-10-16T15:50:44.654222+0000","last_clean":"2023-10-16T15:50:44.654222+0000","last_became_active":"2023-10-16T15:50:43.644262+0000","last_became_peered":"2023-10-16T15:50:43.644262+0000","last_unstale":"2023-10-16T15:50:44.654222+0000","last_undegraded":"2023-10-16T15:50:44.654222+0000","last_fullsized":"2023-10-16T15:50:44.654222+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:03.469421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680076+0000","last_change":"2023-10-16T15:50:44.673958+0000","last_active":"2023-10-16T15:50:44.680076+0000","last_peered":"2023-10-16T15:50:44.680076+0000","last_clean":"2023-10-16T15:50:44.680076+0000","last_became_active":"2023-10-16T15:50:44.673264+0000","last_became_peered":"2023-10-16T15:50:44.673264+0000","last_unstale":"2023-10-16T15:50:44.680076+0000","last_undegraded":"2023-10-16T15:50:44.680076+0000","last_fullsized":"2023-10-16T15:50:44.680076+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:50.435932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.653823+0000","last_change":"2023-10-16T15:50:43.647628+0000","last_active":"2023-10-16T15:50:44.653823+0000","last_peered":"2023-10-16T15:50:44.653823+0000","last_clean":"2023-10-16T15:50:44.653823+0000","last_became_active":"2023-10-16T15:50:43.647316+0000","last_became_peered":"2023-10-16T15:50:43.647316+0000","last_unstale":"2023-10-16T15:50:44.653823+0000","last_undegraded":"2023-10-16T15:50:44.653823+0000","last_fullsized":"2023-10-16T15:50:44.653823+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:49.420248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680320+0000","last_change":"2023-10-16T15:50:44.670215+0000","last_active":"2023-10-16T15:50:44.680320+0000","last_peered":"2023-10-16T15:50:44.680320+0000","last_clean":"2023-10-16T15:50:44.680320+0000","last_became_active":"2023-10-16T15:50:44.669936+0000","last_became_peered":"2023-10-16T15:50:44.669936+0000","last_unstale":"2023-10-16T15:50:44.680320+0000","last_undegraded":"2023-10-16T15:50:44.680320+0000","last_fullsized":"2023-10-16T15:50:44.680320+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:42:50.522444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.653140+0000","last_change":"2023-10-16T15:50:43.646567+0000","last_active":"2023-10-16T15:50:44.653140+0000","last_peered":"2023-10-16T15:50:44.653140+0000","last_clean":"2023-10-16T15:50:44.653140+0000","last_became_active":"2023-10-16T15:50:43.646077+0000","last_became_peered":"2023-10-16T15:50:43.646077+0000","last_unstale":"2023-10-16T15:50:44.653140+0000","last_undegraded":"2023-10-16T15:50:44.653140+0000","last_fullsized":"2023-10-16T15:50:44.653140+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:11.793925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681907+0000","last_change":"2023-10-16T15:50:44.672039+0000","last_active":"2023-10-16T15:50:44.681907+0000","last_peered":"2023-10-16T15:50:44.681907+0000","last_clean":"2023-10-16T15:50:44.681907+0000","last_became_active":"2023-10-16T15:50:44.670144+0000","last_became_peered":"2023-10-16T15:50:44.670144+0000","last_unstale":"2023-10-16T15:50:44.681907+0000","last_undegraded":"2023-10-16T15:50:44.681907+0000","last_fullsized":"2023-10-16T15:50:44.681907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:45.077152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"208'2","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.675752+0000","last_change":"2023-10-16T15:50:43.644414+0000","last_active":"2023-10-16T15:50:45.675752+0000","last_peered":"2023-10-16T15:50:45.675752+0000","last_clean":"2023-10-16T15:50:45.675752+0000","last_became_active":"2023-10-16T15:50:43.644107+0000","last_became_peered":"2023-10-16T15:50:43.644107+0000","last_unstale":"2023-10-16T15:50:45.675752+0000","last_undegraded":"2023-10-16T15:50:45.675752+0000","last_fullsized":"2023-10-16T15:50:45.675752+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:38.787908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477771+0000","last_change":"2023-10-16T15:50:44.665939+0000","last_active":"2023-10-16T15:50:46.477771+0000","last_peered":"2023-10-16T15:50:46.477771+0000","last_clean":"2023-10-16T15:50:46.477771+0000","last_became_active":"2023-10-16T15:50:44.665247+0000","last_became_peered":"2023-10-16T15:50:44.665247+0000","last_unstale":"2023-10-16T15:50:46.477771+0000","last_undegraded":"2023-10-16T15:50:46.477771+0000","last_fullsized":"2023-10-16T15:50:46.477771+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:10.952350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.477234+0000","last_change":"2023-10-16T15:50:43.646596+0000","last_active":"2023-10-16T15:50:46.477234+0000","last_peered":"2023-10-16T15:50:46.477234+0000","last_clean":"2023-10-16T15:50:46.477234+0000","last_became_active":"2023-10-16T15:50:43.646147+0000","last_became_peered":"2023-10-16T15:50:43.646147+0000","last_unstale":"2023-10-16T15:50:46.477234+0000","last_undegraded":"2023-10-16T15:50:46.477234+0000","last_fullsized":"2023-10-16T15:50:46.477234+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:40.672033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669087+0000","last_change":"2023-10-16T15:50:44.659744+0000","last_active":"2023-10-16T15:50:45.669087+0000","last_peered":"2023-10-16T15:50:45.669087+0000","last_clean":"2023-10-16T15:50:45.669087+0000","last_became_active":"2023-10-16T15:50:44.659375+0000","last_became_peered":"2023-10-16T15:50:44.659375+0000","last_unstale":"2023-10-16T15:50:45.669087+0000","last_undegraded":"2023-10-16T15:50:45.669087+0000","last_fullsized":"2023-10-16T15:50:45.669087+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:36:48.194990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.476850+0000","last_change":"2023-10-16T15:50:43.643102+0000","last_active":"2023-10-16T15:50:46.476850+0000","last_peered":"2023-10-16T15:50:46.476850+0000","last_clean":"2023-10-16T15:50:46.476850+0000","last_became_active":"2023-10-16T15:50:43.642753+0000","last_became_peered":"2023-10-16T15:50:43.642753+0000","last_unstale":"2023-10-16T15:50:46.476850+0000","last_undegraded":"2023-10-16T15:50:46.476850+0000","last_fullsized":"2023-10-16T15:50:46.476850+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:41.556192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669725+0000","last_change":"2023-10-16T15:50:44.673317+0000","last_active":"2023-10-16T15:50:45.669725+0000","last_peered":"2023-10-16T15:50:45.669725+0000","last_clean":"2023-10-16T15:50:45.669725+0000","last_became_active":"2023-10-16T15:50:44.672945+0000","last_became_peered":"2023-10-16T15:50:44.672945+0000","last_unstale":"2023-10-16T15:50:45.669725+0000","last_undegraded":"2023-10-16T15:50:45.669725+0000","last_fullsized":"2023-10-16T15:50:45.669725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:13.733047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.669224+0000","last_change":"2023-10-16T15:50:43.643080+0000","last_active":"2023-10-16T15:50:45.669224+0000","last_peered":"2023-10-16T15:50:45.669224+0000","last_clean":"2023-10-16T15:50:45.669224+0000","last_became_active":"2023-10-16T15:50:43.642790+0000","last_became_peered":"2023-10-16T15:50:43.642790+0000","last_unstale":"2023-10-16T15:50:45.669224+0000","last_undegraded":"2023-10-16T15:50:45.669224+0000","last_fullsized":"2023-10-16T15:50:45.669224+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:44.087812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674684+0000","last_change":"2023-10-16T15:50:44.650396+0000","last_active":"2023-10-16T15:50:45.674684+0000","last_peered":"2023-10-16T15:50:45.674684+0000","last_clean":"2023-10-16T15:50:45.674684+0000","last_became_active":"2023-10-16T15:50:44.649422+0000","last_became_peered":"2023-10-16T15:50:44.649422+0000","last_unstale":"2023-10-16T15:50:45.674684+0000","last_undegraded":"2023-10-16T15:50:45.674684+0000","last_fullsized":"2023-10-16T15:50:45.674684+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:32.326238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.652870+0000","last_change":"2023-10-16T15:50:43.635996+0000","last_active":"2023-10-16T15:50:44.652870+0000","last_peered":"2023-10-16T15:50:44.652870+0000","last_clean":"2023-10-16T15:50:44.652870+0000","last_became_active":"2023-10-16T15:50:43.635619+0000","last_became_peered":"2023-10-16T15:50:43.635619+0000","last_unstale":"2023-10-16T15:50:44.652870+0000","last_undegraded":"2023-10-16T15:50:44.652870+0000","last_fullsized":"2023-10-16T15:50:44.652870+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:52.907475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.674989+0000","last_change":"2023-10-16T15:50:44.670966+0000","last_active":"2023-10-16T15:50:45.674989+0000","last_peered":"2023-10-16T15:50:45.674989+0000","last_clean":"2023-10-16T15:50:45.674989+0000","last_became_active":"2023-10-16T15:50:44.670174+0000","last_became_peered":"2023-10-16T15:50:44.670174+0000","last_unstale":"2023-10-16T15:50:45.674989+0000","last_undegraded":"2023-10-16T15:50:45.674989+0000","last_fullsized":"2023-10-16T15:50:45.674989+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:24.042552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.652639+0000","last_change":"2023-10-16T15:50:43.635679+0000","last_active":"2023-10-16T15:50:44.652639+0000","last_peered":"2023-10-16T15:50:44.652639+0000","last_clean":"2023-10-16T15:50:44.652639+0000","last_became_active":"2023-10-16T15:50:43.635264+0000","last_became_peered":"2023-10-16T15:50:43.635264+0000","last_unstale":"2023-10-16T15:50:44.652639+0000","last_undegraded":"2023-10-16T15:50:44.652639+0000","last_fullsized":"2023-10-16T15:50:44.652639+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:02.580282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.681819+0000","last_change":"2023-10-16T15:50:44.674153+0000","last_active":"2023-10-16T15:50:44.681819+0000","last_peered":"2023-10-16T15:50:44.681819+0000","last_clean":"2023-10-16T15:50:44.681819+0000","last_became_active":"2023-10-16T15:50:44.673667+0000","last_became_peered":"2023-10-16T15:50:44.673667+0000","last_unstale":"2023-10-16T15:50:44.681819+0000","last_undegraded":"2023-10-16T15:50:44.681819+0000","last_fullsized":"2023-10-16T15:50:44.681819+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:26.730085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.659651+0000","last_change":"2023-10-16T15:50:43.644466+0000","last_active":"2023-10-16T15:50:44.659651+0000","last_peered":"2023-10-16T15:50:44.659651+0000","last_clean":"2023-10-16T15:50:44.659651+0000","last_became_active":"2023-10-16T15:50:43.642815+0000","last_became_peered":"2023-10-16T15:50:43.642815+0000","last_unstale":"2023-10-16T15:50:44.659651+0000","last_undegraded":"2023-10-16T15:50:44.659651+0000","last_fullsized":"2023-10-16T15:50:44.659651+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:59.573511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.680298+0000","last_change":"2023-10-16T15:50:44.668739+0000","last_active":"2023-10-16T15:50:44.680298+0000","last_peered":"2023-10-16T15:50:44.680298+0000","last_clean":"2023-10-16T15:50:44.680298+0000","last_became_active":"2023-10-16T15:50:44.668437+0000","last_became_peered":"2023-10-16T15:50:44.668437+0000","last_unstale":"2023-10-16T15:50:44.680298+0000","last_undegraded":"2023-10-16T15:50:44.680298+0000","last_fullsized":"2023-10-16T15:50:44.680298+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:14.507460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.653097+0000","last_change":"2023-10-16T15:50:43.645512+0000","last_active":"2023-10-16T15:50:44.653097+0000","last_peered":"2023-10-16T15:50:44.653097+0000","last_clean":"2023-10-16T15:50:44.653097+0000","last_became_active":"2023-10-16T15:50:43.644654+0000","last_became_peered":"2023-10-16T15:50:43.644654+0000","last_unstale":"2023-10-16T15:50:44.653097+0000","last_undegraded":"2023-10-16T15:50:44.653097+0000","last_fullsized":"2023-10-16T15:50:44.653097+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:14.775386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:46.475604+0000","last_change":"2023-10-16T15:50:44.673743+0000","last_active":"2023-10-16T15:50:46.475604+0000","last_peered":"2023-10-16T15:50:46.475604+0000","last_clean":"2023-10-16T15:50:46.475604+0000","last_became_active":"2023-10-16T15:50:44.673504+0000","last_became_peered":"2023-10-16T15:50:44.673504+0000","last_unstale":"2023-10-16T15:50:46.475604+0000","last_undegraded":"2023-10-16T15:50:46.475604+0000","last_fullsized":"2023-10-16T15:50:46.475604+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:43.601290+0000","last_clean_scrub_stamp":"2023-10-16T15:50:43.601290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:19.955110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-16T15:50:45.673699+0000","last_change":"2023-10-16T15:50:43.639488+0000","last_active":"2023-10-16T15:50:45.673699+0000","last_peered":"2023-10-16T15:50:45.673699+0000","last_clean":"2023-10-16T15:50:45.673699+0000","last_became_active":"2023-10-16T15:50:43.639062+0000","last_became_peered":"2023-10-16T15:50:43.639062+0000","last_unstale":"2023-10-16T15:50:45.673699+0000","last_undegraded":"2023-10-16T15:50:45.673699+0000","last_fullsized":"2023-10-16T15:50:45.673699+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:50:42.590432+0000","last_clean_scrub_stamp":"2023-10-16T15:50:42.590432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:20.592600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"197'329","reported_seq":1151,"reported_epoch":208,"state":"active+clean","last_fresh":"2023-10-16T15:50:44.645382+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:50:44.645382+0000","last_peered":"2023-10-16T15:50:44.645382+0000","last_clean":"2023-10-16T15:50:44.645382+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:50:44.645382+0000","last_undegraded":"2023-10-16T15:50:44.645382+0000","last_fullsized":"2023-10-16T15:50:44.645382+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":329,"ondisk_log_size":329,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575972,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60692,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311148,"statfs":{"total":96636764160,"available":96574615552,"internally_reserved":0,"allocated":3219456,"data_stored":1872699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.194,"15min":1.1799999999999999},"min":{"1min":0.76200000000000001,"5min":0.73599999999999999,"15min":0.70699999999999996},"max":{"1min":1.7330000000000001,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.155,"15min":1.1679999999999999},"min":{"1min":0.91600000000000004,"5min":0.86099999999999999,"15min":0.64200000000000002},"max":{"1min":1.4970000000000001,"5min":1.6910000000000001,"15min":1.748},"last":1.395}]},{"osd":1,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1839999999999999,"15min":1.194},"min":{"1min":0.92400000000000004,"5min":0.84399999999999997,"15min":0.67400000000000004},"max":{"1min":1.55,"5min":1.802,"15min":1.889},"last":1.22},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4019999999999999,"15min":1.389},"min":{"1min":1.1359999999999999,"5min":1.004,"15min":0.88400000000000001},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":0.76800000000000002}]},{"osd":2,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.359,"15min":1.3520000000000001},"min":{"1min":0.88100000000000001,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.468},{"interface":"front","average":{"1min":1.391,"5min":1.3380000000000001,"15min":1.3169999999999999},"min":{"1min":1.0760000000000001,"5min":0.94999999999999996,"15min":0.77400000000000002},"max":{"1min":1.7150000000000001,"5min":1.95,"15min":1.95},"last":1.0980000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.3939999999999999,"15min":1.3600000000000001},"min":{"1min":1.089,"5min":0.95999999999999996,"15min":0.878},"max":{"1min":1.702,"5min":1.869,"15min":1.9750000000000001},"last":1.351},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4690000000000001,"15min":1.4610000000000001},"min":{"1min":1.0409999999999999,"5min":0.93100000000000005,"15min":0.85199999999999998},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.6879999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.385,"15min":1.3759999999999999},"min":{"1min":0.95699999999999996,"5min":0.91800000000000004,"15min":0.80600000000000005},"max":{"1min":1.819,"5min":1.819,"15min":1.978},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.3660000000000001,"15min":1.3500000000000001},"min":{"1min":0.81999999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6419999999999999,"5min":2.0129999999999999,"15min":2.0409999999999999},"last":1}]},{"osd":5,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"min":{"1min":1.1240000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.1469999999999998},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5680000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.7609999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":1.0109999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.845,"5min":2.0840000000000001,"15min":2.2149999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.5860000000000001,"15min":1.595},"min":{"1min":1.21,"5min":1.0649999999999999,"15min":0.89500000000000002},"max":{"1min":2.2599999999999998,"5min":2.2599999999999998,"15min":2.3250000000000002},"last":1.8120000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575971,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29568,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342272,"statfs":{"total":96636764160,"available":96606486528,"internally_reserved":0,"allocated":2605056,"data_stored":1275899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.23,"15min":1.2010000000000001},"min":{"1min":0.91200000000000003,"5min":0.79600000000000004,"15min":0.73199999999999998},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.264,"15min":1.2509999999999999},"min":{"1min":0.88400000000000001,"5min":0.80800000000000005,"15min":0.60699999999999998},"max":{"1min":1.738,"5min":1.847,"15min":1.847},"last":1.2170000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.274,"15min":1.282},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.72199999999999998},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.802},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3240000000000001,"15min":1.319},"min":{"1min":0.94999999999999996,"5min":0.90000000000000002,"15min":0.66600000000000004},"max":{"1min":1.714,"5min":1.8740000000000001,"15min":1.9419999999999999},"last":1.0369999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.238,"15min":1.244},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.71899999999999997},"max":{"1min":1.5089999999999999,"5min":1.7090000000000001,"15min":1.8380000000000001},"last":1.083},{"interface":"front","average":{"1min":1.427,"5min":1.4139999999999999,"15min":1.3899999999999999},"min":{"1min":1.0820000000000001,"5min":0.97299999999999998,"15min":0.84399999999999997},"max":{"1min":1.764,"5min":2.0339999999999998,"15min":2.0419999999999998},"last":1.171}]},{"osd":3,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.4059999999999999,"15min":1.399},"min":{"1min":1.0740000000000001,"5min":0.84099999999999997,"15min":0.77700000000000002},"max":{"1min":1.7649999999999999,"5min":1.8360000000000001,"15min":2.0459999999999998},"last":1.119},{"interface":"front","average":{"1min":1.391,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":1.206,"5min":1.0349999999999999,"15min":0.81000000000000005},"max":{"1min":1.6890000000000001,"5min":1.7569999999999999,"15min":2.0699999999999998},"last":1.268}]},{"osd":4,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.3340000000000001,"15min":1.325},"min":{"1min":0.95599999999999996,"5min":0.81200000000000006,"15min":0.78300000000000003},"max":{"1min":1.917,"5min":1.917,"15min":2.028},"last":1.472},{"interface":"front","average":{"1min":1.49,"5min":1.3959999999999999,"15min":1.4079999999999999},"min":{"1min":1.115,"5min":0.92500000000000004,"15min":0.77600000000000002},"max":{"1min":1.819,"5min":1.956,"15min":2.0419999999999998},"last":0.93000000000000005}]},{"osd":5,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3799999999999999,"15min":1.353},"min":{"1min":1.0309999999999999,"5min":0.90700000000000003,"15min":0.67300000000000004},"max":{"1min":1.663,"5min":1.9419999999999999,"15min":2.016},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.514,"15min":1.5229999999999999},"min":{"1min":1.234,"5min":0.98499999999999999,"15min":0.88},"max":{"1min":1.9139999999999999,"5min":2.129,"15min":2.129},"last":1.371}]},{"osd":7,"last update":"Mon Oct 16 15:50:26 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.595,"15min":1.595},"min":{"1min":1.2529999999999999,"5min":1.0720000000000001,"15min":0.94599999999999995},"max":{"1min":1.8779999999999999,"5min":2.0510000000000002,"15min":2.4940000000000002},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.595,"15min":1.5900000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.97499999999999998},"max":{"1min":1.863,"5min":2.1179999999999999,"15min":2.169},"last":1.4490000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575971,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29568,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342272,"statfs":{"total":96636764160,"available":96606486528,"internally_reserved":0,"allocated":2605056,"data_stored":1275899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1419999999999999,"15min":1.1759999999999999},"min":{"1min":0.79500000000000004,"5min":0.77200000000000002,"15min":0.65600000000000003},"max":{"1min":1.55,"5min":1.613,"15min":1.821},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.262,"15min":1.2230000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.673,"5min":1.732,"15min":1.8120000000000001},"last":1.3660000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.224,"15min":1.2310000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.498,"5min":1.8140000000000001,"15min":1.871},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.306,"5min":1.29,"15min":1.3120000000000001},"min":{"1min":0.93799999999999994,"5min":0.87,"15min":0.87},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.1759999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.228,"15min":1.2609999999999999},"min":{"1min":0.96799999999999997,"5min":0.92000000000000004,"15min":0.86699999999999999},"max":{"1min":1.577,"5min":1.6890000000000001,"15min":1.9219999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4039999999999999,"15min":1.4079999999999999},"min":{"1min":1.2010000000000001,"5min":1.0349999999999999,"15min":0.94899999999999995},"max":{"1min":1.748,"5min":1.8460000000000001,"15min":2.2160000000000002},"last":1.04}]},{"osd":3,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.359,"15min":1.3440000000000001},"min":{"1min":0.998,"5min":0.97699999999999998,"15min":0.80000000000000004},"max":{"1min":1.829,"5min":1.8759999999999999,"15min":1.962},"last":1.282},{"interface":"front","average":{"1min":1.46,"5min":1.3819999999999999,"15min":1.3979999999999999},"min":{"1min":1.24,"5min":0.93899999999999995,"15min":0.93000000000000005},"max":{"1min":1.7150000000000001,"5min":1.748,"15min":2.1589999999999998},"last":1.212}]},{"osd":4,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.365,"15min":1.3400000000000001},"min":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.78100000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.9730000000000001},"last":1.466},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.5249999999999999},"min":{"1min":0.98899999999999999,"5min":0.90800000000000003,"15min":0.77000000000000002},"max":{"1min":1.8580000000000001,"5min":1.859,"15min":2.3210000000000002},"last":1.339}]},{"osd":6,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.3779999999999999,"15min":1.3660000000000001},"min":{"1min":0.92500000000000004,"5min":0.83599999999999997,"15min":0.80600000000000005},"max":{"1min":1.95,"5min":1.95,"15min":2.3809999999999998},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.4470000000000001,"15min":1.4390000000000001},"min":{"1min":1.129,"5min":0.88800000000000001,"15min":0.70599999999999996},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.643}]},{"osd":7,"last update":"Mon Oct 16 15:50:37 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5469999999999999,"15min":1.548},"min":{"1min":1.1140000000000001,"5min":1.069,"15min":0.98599999999999999},"max":{"1min":1.964,"5min":1.964,"15min":2.5129999999999999},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.6259999999999999,"15min":1.6339999999999999},"min":{"1min":1.032,"5min":1.032,"15min":0.70499999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.544},"last":1.5149999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51800,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320040,"statfs":{"total":96636764160,"available":96583720960,"internally_reserved":0,"allocated":2633728,"data_stored":1284421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.232,"15min":1.224},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.52700000000000002},"max":{"1min":1.6180000000000001,"5min":1.752,"15min":1.823},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2010000000000001,"15min":1.218},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.54,"5min":1.8049999999999999,"15min":1.998},"last":0.91200000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.2549999999999999,"15min":1.2330000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.629},"max":{"1min":1.4810000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.264,"5min":1.3440000000000001,"15min":1.319},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.667,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.21}]},{"osd":2,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.4550000000000001,"15min":1.4670000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8100000000000001,"5min":2.0489999999999999,"15min":2.8929999999999998},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4870000000000001,"15min":1.49},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.835,"5min":2.1829999999999998,"15min":2.4550000000000001},"last":1.3049999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3859999999999999,"15min":1.377},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.7270000000000001,"5min":1.9199999999999999,"15min":2.3650000000000002},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.395,"15min":1.389},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":1.5980000000000001,"5min":2.0870000000000002,"15min":2.2970000000000002},"last":1.167}]},{"osd":5,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3420000000000001,"15min":1.343},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.7070000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.4370000000000001,"15min":1.4139999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7509999999999999,"5min":2.1139999999999999,"15min":2.4489999999999998},"last":1.52}]},{"osd":6,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.375,"15min":1.3979999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":1.675,"5min":2.2010000000000001,"15min":2.2389999999999999},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.5189999999999999,"15min":1.522},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":1.7729999999999999,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":1.722}]},{"osd":7,"last update":"Mon Oct 16 15:50:38 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5840000000000001,"15min":1.571},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.9350000000000001,"5min":2.1659999999999999,"15min":2.3460000000000001},"last":1.583},{"interface":"front","average":{"1min":1.514,"5min":1.609,"15min":1.587},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.1120000000000001,"5min":2.27,"15min":2.347},"last":1.76}]}]},{"osd":3,"up_from":13,"seq":55834575971,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51804,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320036,"statfs":{"total":96636764160,"available":96583716864,"internally_reserved":0,"allocated":2629632,"data_stored":1282421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.94999999999999996,"15min":0.96499999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.2230000000000001,"5min":1.4950000000000001,"15min":2.073},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.042,"5min":1.0580000000000001,"15min":1.087},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":1.081}]},{"osd":1,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0600000000000001,"15min":1.056},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6679999999999999,"5min":2.0049999999999999,"15min":2.589},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1879999999999999,"15min":1.2270000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.722,"5min":1.722,"15min":2.5569999999999999},"last":1.1779999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4510000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.885,"5min":1.897,"15min":2.6600000000000001},"last":1.103},{"interface":"front","average":{"1min":1.429,"5min":1.4970000000000001,"15min":1.528},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.8839999999999999,"5min":1.9530000000000001,"15min":2.754},"last":0.98699999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.377,"15min":1.3660000000000001},"min":{"1min":1.036,"5min":0.94099999999999995,"15min":0.93400000000000005},"max":{"1min":1.6299999999999999,"5min":1.95,"15min":2.5390000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.4410000000000001,"15min":1.4399999999999999},"min":{"1min":1.1419999999999999,"5min":1.024,"15min":0.96999999999999997},"max":{"1min":1.8089999999999999,"5min":2.286,"15min":2.5179999999999998},"last":1.677}]},{"osd":5,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5780000000000001,"15min":1.587},"min":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.103},"max":{"1min":1.9930000000000001,"5min":2.2160000000000002,"15min":2.6320000000000001},"last":1.52},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.554,"15min":1.5760000000000001},"min":{"1min":1.141,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":1.8029999999999999,"5min":2.2770000000000001,"15min":2.8170000000000002},"last":1.2989999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.51,"15min":1.526},"min":{"1min":1.2010000000000001,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1099999999999999,"15min":2.2090000000000001},"last":1.363},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.605,"15min":1.619},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":2.0680000000000001,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.7689999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:50:43 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.6950000000000001,"15min":1.702},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.9790000000000001,"5min":2.3919999999999999,"15min":2.8780000000000001},"last":1.415},{"interface":"front","average":{"1min":1.55,"5min":1.645,"15min":1.6599999999999999},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":1.9079999999999999,"5min":2.2250000000000001,"15min":2.6699999999999999},"last":1.716}]}]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51792,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320048,"statfs":{"total":96636764160,"available":96583729152,"internally_reserved":0,"allocated":2617344,"data_stored":1276593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.94999999999999996,"15min":0.97299999999999998},"min":{"1min":0.749,"5min":0.65800000000000003,"15min":0.60099999999999998},"max":{"1min":1.716,"5min":1.716,"15min":2.165},"last":0.754},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.117,"15min":1.1140000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.45,"5min":1.762,"15min":2.0790000000000002},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0269999999999999,"15min":1.0620000000000001},"min":{"1min":0.83499999999999996,"5min":0.76600000000000001,"15min":0.72399999999999998},"max":{"1min":1.405,"5min":1.635,"15min":2.3300000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2470000000000001,"15min":1.2689999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":2.0779999999999998},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3029999999999999,"15min":1.327},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6739999999999999,"5min":1.9139999999999999,"15min":2.4780000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.363,"15min":1.387},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":2.8159999999999998},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.4970000000000001,"15min":1.5049999999999999},"min":{"1min":1.133,"5min":1.069,"15min":1.0589999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.4119999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.407,"5min":1.452,"15min":1.4470000000000001},"min":{"1min":1.032,"5min":1.0209999999999999,"15min":0.86299999999999999},"max":{"1min":1.915,"5min":2.0920000000000001,"15min":2.1589999999999998},"last":0.879}]},{"osd":5,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.5029999999999999,"15min":1.5149999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":2.657,"5min":2.657,"15min":2.657},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.653,"15min":1.6459999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.5569999999999999},"last":1.0089999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5800000000000001,"15min":1.581},"min":{"1min":1.0489999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":1.9610000000000001,"5min":2.1960000000000002,"15min":2.3849999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.649,"15min":1.641},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.2709999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:50:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.6870000000000001,"15min":1.6830000000000001},"min":{"1min":1.133,"5min":1.0920000000000001,"15min":1.0920000000000001},"max":{"1min":1.982,"5min":2.1080000000000001,"15min":2.4750000000000001},"last":1.256},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.742,"15min":1.758},"min":{"1min":1.3660000000000001,"5min":1.2270000000000001,"15min":1.2210000000000001},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.754},"last":1.3240000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575972,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47624,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324216,"statfs":{"total":96636764160,"available":96587997184,"internally_reserved":0,"allocated":2617344,"data_stored":1277511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.96499999999999997,"15min":0.95199999999999996},"min":{"1min":0.73699999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.5800000000000001,"5min":1.8069999999999999,"15min":1.8420000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1759999999999999,"15min":1.1739999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.603,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.875}]},{"osd":2,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0549999999999999,"15min":1.054},"min":{"1min":0.84499999999999997,"5min":0.621,"15min":0.621},"max":{"1min":1.591,"5min":1.913,"15min":2.286},"last":1.607},{"interface":"front","average":{"1min":1.258,"5min":1.266,"15min":1.252},"min":{"1min":0.91500000000000004,"5min":0.83999999999999997,"15min":0.73899999999999999},"max":{"1min":1.7090000000000001,"5min":1.806,"15min":1.853},"last":1.036}]},{"osd":3,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.397,"15min":1.4019999999999999},"min":{"1min":1.0289999999999999,"5min":0.91700000000000004,"15min":0.86699999999999999},"max":{"1min":1.8700000000000001,"5min":2.0619999999999998,"15min":2.1920000000000002},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.345,"15min":1.3480000000000001},"min":{"1min":1.0249999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.8740000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.373,"15min":1.3839999999999999},"min":{"1min":1.1459999999999999,"5min":0.871,"15min":0.86799999999999999},"max":{"1min":2.282,"5min":2.282,"15min":2.3809999999999998},"last":1.502},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.5129999999999999,"15min":1.5229999999999999},"min":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.071},"max":{"1min":2.198,"5min":2.198,"15min":2.6259999999999999},"last":1.5609999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.5349999999999999,"15min":1.538},"min":{"1min":1.0669999999999999,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":1.95,"5min":2,"15min":2.3799999999999999},"last":1.367},{"interface":"front","average":{"1min":1.494,"5min":1.4850000000000001,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.8720000000000001,"5min":2.254,"15min":2.4689999999999999},"last":1.821}]},{"osd":6,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.6499999999999999,"15min":1.6559999999999999},"min":{"1min":1.2929999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.5009999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.6100000000000001,"15min":1.653},"min":{"1min":1.264,"5min":1.135,"15min":1.135},"max":{"1min":2.024,"5min":2.2610000000000001,"15min":3.0430000000000001},"last":1.647}]},{"osd":7,"last update":"Mon Oct 16 15:50:15 2023","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.72,"15min":1.764},"min":{"1min":1.504,"5min":1.3340000000000001,"15min":1.272},"max":{"1min":2.1309999999999998,"5min":2.2869999999999999,"15min":3.2090000000000001},"last":1.746},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.6439999999999999,"15min":1.661},"min":{"1min":1.1559999999999999,"5min":1.036,"15min":0.98699999999999999},"max":{"1min":2.1739999999999999,"5min":2.331,"15min":3.1539999999999999},"last":2.1349999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64840,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307000,"statfs":{"total":96636764160,"available":96570368000,"internally_reserved":0,"allocated":3203072,"data_stored":1866289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.97899999999999998,"15min":0.97799999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.4419999999999999,"5min":1.585,"15min":1.659},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2669999999999999,"15min":1.254},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.609,"5min":1.669,"15min":1.8029999999999999},"last":1.609}]},{"osd":2,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.3089999999999999,"15min":1.294},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.859,"5min":1.859,"15min":2.0259999999999998},"last":1.365},{"interface":"front","average":{"1min":1.173,"5min":1.252,"15min":1.2669999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.863},"last":1.22}]},{"osd":3,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.0349999999999999,"15min":1.0509999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.501},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.29},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.57199999999999995},"max":{"1min":1.9850000000000001,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.7090000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3420000000000001,"15min":1.359},"min":{"1min":0.998,"5min":0.93999999999999995,"15min":0.88800000000000001},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.8520000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.5489999999999999,"15min":1.534},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.74299999999999999},"max":{"1min":1.9099999999999999,"5min":2.452,"15min":2.452},"last":1.9099999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4870000000000001,"15min":1.5049999999999999},"min":{"1min":1.0760000000000001,"5min":1.05,"15min":0.871},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.448,"15min":1.446},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.81699999999999995},"max":{"1min":2.0859999999999999,"5min":2.282,"15min":2.282},"last":1.5369999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.6850000000000001,"15min":1.671},"min":{"1min":1.262,"5min":1.262,"15min":0.84499999999999997},"max":{"1min":2.0009999999999999,"5min":2.3279999999999998,"15min":2.7010000000000001},"last":1.776},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.595,"15min":1.599},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.0339999999999998,"5min":2.1030000000000002,"15min":2.137},"last":1.966}]},{"osd":7,"last update":"Mon Oct 16 15:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.6839999999999999,"15min":1.712},"min":{"1min":1.163,"5min":1.1120000000000001,"15min":0.877},"max":{"1min":2.2130000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":2.105},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.6339999999999999,"15min":1.6259999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":2.0390000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:50:50.194 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:50:50.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T15:50:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.471+0000 7feba4ebb640 1 Processor -- start 2023-10-16T15:50:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.472+0000 7feba4ebb640 1 -- start start 2023-10-16T15:50:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.473+0000 7feba4ebb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7feba00cc270 0x7feba015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.473+0000 7feba4ebb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feba00704e0 0x7feba0070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.473+0000 7feba4ebb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba015e5c0 con 0x7feba0159ac0 2023-10-16T15:50:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.473+0000 7feba4ebb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba0070eb0 con 0x7feba00704e0 2023-10-16T15:50:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.473+0000 7feba4ebb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba006b810 con 0x7feba00cc270 2023-10-16T15:50:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feba0159ac0 0x7feba00c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52764/0 (socket says 172.21.15.29:52764) 2023-10-16T15:50:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9f7fe640 1 -- 172.21.15.29:0/2492296352 learned_addr learned my addr 172.21.15.29:0/2492296352 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9ffff640 1 --2- 172.21.15.29:0/2492296352 >> v2:172.21.15.29:3300/0 conn(0x7feba00704e0 0x7feba0070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb97fff640 1 --2- 172.21.15.29:0/2492296352 >> v2:172.21.15.170:3300/0 conn(0x7feba00cc270 0x7feba015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9ffff640 1 -- 172.21.15.29:0/2492296352 >> v2:172.21.15.170:3300/0 conn(0x7feba00cc270 msgr2=0x7feba015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9ffff640 1 --2- 172.21.15.29:0/2492296352 >> v2:172.21.15.170:3300/0 conn(0x7feba00cc270 0x7feba015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9ffff640 1 -- 172.21.15.29:0/2492296352 >> v1:172.21.15.29:6789/0 conn(0x7feba0159ac0 legacy=0x7feba00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.474+0000 7feb9ffff640 1 -- 172.21.15.29:0/2492296352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00cc810 con 0x7feba00704e0 2023-10-16T15:50:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.476+0000 7feb9ffff640 1 --2- 172.21.15.29:0/2492296352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00704e0 0x7feba0070970 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7feba00cac70 tx=0x7feb9802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fba216f0aeb31ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.476+0000 7feb9effd640 1 -- 172.21.15.29:0/2492296352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9802cc70 con 0x7feba00704e0 2023-10-16T15:50:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.476+0000 7feb9effd640 1 -- 172.21.15.29:0/2492296352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9802cdd0 con 0x7feba00704e0 2023-10-16T15:50:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.476+0000 7feb9effd640 1 -- 172.21.15.29:0/2492296352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98034660 con 0x7feba00704e0 2023-10-16T15:50:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 -- 172.21.15.29:0/2492296352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00704e0 msgr2=0x7feba0070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 --2- 172.21.15.29:0/2492296352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00704e0 0x7feba0070970 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7feba00cac70 tx=0x7feb9802bec0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 -- 172.21.15.29:0/2492296352 shutdown_connections 2023-10-16T15:50:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 --2- 172.21.15.29:0/2492296352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00704e0 0x7feba0070970 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 --2- 172.21.15.29:0/2492296352 >> v2:172.21.15.170:3300/0 conn(0x7feba00cc270 0x7feba015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 -- 172.21.15.29:0/2492296352 >> 172.21.15.29:0/2492296352 conn(0x7feba00ce6a0 msgr2=0x7feba00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 -- 172.21.15.29:0/2492296352 shutdown_connections 2023-10-16T15:50:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.477+0000 7feba4ebb640 1 -- 172.21.15.29:0/2492296352 wait complete. 2023-10-16T15:50:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.479+0000 7feba4ebb640 1 Processor -- start 2023-10-16T15:50:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.480+0000 7feba4ebb640 1 -- start start 2023-10-16T15:50:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.481+0000 7feba4ebb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba00704e0 0x7feba0157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.481+0000 7feba4ebb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 0x7feba00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.481+0000 7feba4ebb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba0159ac0 0x7feba00adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.481+0000 7feba4ebb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0158210 con 0x7feba00cc270 2023-10-16T15:50:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.481+0000 7feba4ebb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba01f7ae0 con 0x7feba0159ac0 2023-10-16T15:50:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.481+0000 7feba4ebb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba00c6130 con 0x7feba00704e0 2023-10-16T15:50:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 0x7feba00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 0x7feba00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52358/0 (socket says 172.21.15.29:52358) 2023-10-16T15:50:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 -- 172.21.15.29:0/3799389690 learned_addr learned my addr 172.21.15.29:0/3799389690 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb9ffff640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba0159ac0 0x7feba00adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb9f7fe640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba00704e0 0x7feba0157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 -- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba00704e0 msgr2=0x7feba0157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba00704e0 0x7feba0157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 -- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba0159ac0 msgr2=0x7feba00adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba0159ac0 0x7feba00adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.482+0000 7feb97fff640 1 -- 172.21.15.29:0/3799389690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98009a70 con 0x7feba00cc270 2023-10-16T15:50:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.483+0000 7feb97fff640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 0x7feba00c5770 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7feb8800d800 tx=0x7feb8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.483+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8800b6c0 con 0x7feba00cc270 2023-10-16T15:50:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.483+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8800b820 con 0x7feba00cc270 2023-10-16T15:50:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.483+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba01f7d80 con 0x7feba00cc270 2023-10-16T15:50:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.483+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8800bbb0 con 0x7feba00cc270 2023-10-16T15:50:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.483+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba00c6410 con 0x7feba00cc270 2023-10-16T15:50:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.484+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba004e9b0 con 0x7feba00cc270 2023-10-16T15:50:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.486+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb8800bd10 con 0x7feba00cc270 2023-10-16T15:50:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.487+0000 7feb9cff9640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7feb8006af90 0x7feb8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.487+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feb88089d80 con 0x7feba00cc270 2023-10-16T15:50:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.488+0000 7feb9f7fe640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7feb8006af90 0x7feb8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.489+0000 7feb9f7fe640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7feb8006af90 0x7feb8006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb90005880 tx=0x7feb90009040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:50.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.492+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb88054cf0 con 0x7feba00cc270 2023-10-16T15:50:50.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.754+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feba00b4620 con 0x7feba00cc270 2023-10-16T15:50:50.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.759+0000 7feb9cff9640 1 -- 172.21.15.29:0/3799389690 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v48) v1 ==== 190+0+147 (secure 0 0 0) 0x7feb88058310 con 0x7feba00cc270 2023-10-16T15:50:50.760 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T15:50:50.761 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T15:50:50.761 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T15:50:50.761 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T15:50:50.761 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T15:50:50.762 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T15:50:50.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7feb8006af90 msgr2=0x7feb8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7feb8006af90 0x7feb8006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb90005880 tx=0x7feb90009040 comp rx=0 tx=0).stop 2023-10-16T15:50:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 msgr2=0x7feba00c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 0x7feba00c5770 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7feb8800d800 tx=0x7feb8800dcc0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 shutdown_connections 2023-10-16T15:50:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7feb8006af90 0x7feb8006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feba00cc270 0x7feba00c5770 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feba0159ac0 0x7feba00adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 --2- 172.21.15.29:0/3799389690 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feba00704e0 0x7feba0157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 >> 172.21.15.29:0/3799389690 conn(0x7feba00ce6a0 msgr2=0x7feba006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 shutdown_connections 2023-10-16T15:50:50.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.762+0000 7feba4ebb640 1 -- 172.21.15.29:0/3799389690 wait complete. 2023-10-16T15:50:50.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:50:50.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.998+0000 7f9d6bfff640 1 Processor -- start 2023-10-16T15:50:50.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.998+0000 7f9d6bfff640 1 -- start start 2023-10-16T15:50:51.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.999+0000 7f9d6bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9d6c159950 0x7f9d6c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.999+0000 7f9d6bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d6c070450 0x7f9d6c0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.999+0000 7f9d6bfff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c15aef0 con 0x7f9d6c06c100 2023-10-16T15:50:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.999+0000 7f9d6bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6c070e20 con 0x7f9d6c070450 2023-10-16T15:50:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:50.999+0000 7f9d6bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6c15e490 con 0x7f9d6c159950 2023-10-16T15:50:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d6c070450 0x7f9d6c0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d6c070450 0x7f9d6c0708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52372/0 (socket says 172.21.15.29:52372) 2023-10-16T15:50:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/1716814202 learned_addr learned my addr 172.21.15.29:0/1716814202 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6affd640 1 --2- 172.21.15.29:0/1716814202 >> v2:172.21.15.170:3300/0 conn(0x7f9d6c159950 0x7f9d6c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/1716814202 >> v2:172.21.15.170:3300/0 conn(0x7f9d6c159950 msgr2=0x7f9d6c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 --2- 172.21.15.29:0/1716814202 >> v2:172.21.15.170:3300/0 conn(0x7f9d6c159950 0x7f9d6c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/1716814202 >> v1:172.21.15.170:6789/0 conn(0x7f9d6c06c100 legacy=0x7f9d6c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.000+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/1716814202 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c15d910 con 0x7f9d6c070450 2023-10-16T15:50:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.002+0000 7f9d6a7fc640 1 --2- 172.21.15.29:0/1716814202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c0708e0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0ca950 tx=0x7f9d6002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1bd8f76015c9b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.002+0000 7f9d69ffb640 1 -- 172.21.15.29:0/1716814202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d600317e0 con 0x7f9d6c070450 2023-10-16T15:50:51.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.002+0000 7f9d69ffb640 1 -- 172.21.15.29:0/1716814202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d60031940 con 0x7f9d6c070450 2023-10-16T15:50:51.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.002+0000 7f9d69ffb640 1 -- 172.21.15.29:0/1716814202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60031cd0 con 0x7f9d6c070450 2023-10-16T15:50:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 -- 172.21.15.29:0/1716814202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 msgr2=0x7f9d6c0708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/1716814202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c0708e0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0ca950 tx=0x7f9d6002b880 comp rx=0 tx=0).stop 2023-10-16T15:50:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 -- 172.21.15.29:0/1716814202 shutdown_connections 2023-10-16T15:50:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/1716814202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c0708e0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/1716814202 >> v2:172.21.15.170:3300/0 conn(0x7f9d6c159950 0x7f9d6c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 -- 172.21.15.29:0/1716814202 >> 172.21.15.29:0/1716814202 conn(0x7f9d6c0ce6a0 msgr2=0x7f9d6c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 -- 172.21.15.29:0/1716814202 shutdown_connections 2023-10-16T15:50:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.003+0000 7f9d6bfff640 1 -- 172.21.15.29:0/1716814202 wait complete. 2023-10-16T15:50:51.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.005+0000 7f9d6bfff640 1 Processor -- start 2023-10-16T15:50:51.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.005+0000 7f9d6bfff640 1 -- start start 2023-10-16T15:50:51.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d6c06c100 0x7f9d6c0c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d6c159950 0x7f9d6c0ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0c4630 con 0x7f9d6c070450 2023-10-16T15:50:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c155ec0 con 0x7f9d6c159950 2023-10-16T15:50:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0c39b0 con 0x7f9d6c06c100 2023-10-16T15:50:51.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52380/0 (socket says 172.21.15.29:52380) 2023-10-16T15:50:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/4165174065 learned_addr learned my addr 172.21.15.29:0/4165174065 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6b7fe640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d6c159950 0x7f9d6c0ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d6c06c100 msgr2=0x7f9d6c0c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6a7fc640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d6c06c100 0x7f9d6c0c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.006+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d6c159950 msgr2=0x7f9d6c0ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d6a7fc640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d6c159950 0x7f9d6c0ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d6a7fc640 1 -- 172.21.15.29:0/4165174065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600094d0 con 0x7f9d6c070450 2023-10-16T15:50:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d6b7fe640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d6c159950 0x7f9d6c0ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d6a7fc640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c155980 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f9d60000c00 tx=0x7f9d600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60032880 con 0x7f9d6c070450 2023-10-16T15:50:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d600329e0 con 0x7f9d6c070450 2023-10-16T15:50:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60032d90 con 0x7f9d6c070450 2023-10-16T15:50:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.007+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c0c3c90 con 0x7f9d6c070450 2023-10-16T15:50:51.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.008+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c0c4250 con 0x7f9d6c070450 2023-10-16T15:50:51.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.009+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c0668c0 con 0x7f9d6c070450 2023-10-16T15:50:51.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.010+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9d600453f0 con 0x7f9d6c070450 2023-10-16T15:50:51.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.011+0000 7f9d537fe640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9d5806af90 0x7f9d5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.011+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9d600ab810 con 0x7f9d6c070450 2023-10-16T15:50:51.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.012+0000 7f9d6affd640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9d5806af90 0x7f9d5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.013+0000 7f9d6affd640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9d5806af90 0x7f9d5806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.016+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d60076670 con 0x7f9d6c070450 2023-10-16T15:50:51.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.247+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d6c15aef0 con 0x7f9d6c070450 2023-10-16T15:50:51.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.248+0000 7f9d537fe640 1 -- 172.21.15.29:0/4165174065 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f9d6c15aef0 con 0x7f9d6c070450 2023-10-16T15:50:51.250 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:50:51.250 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:50:44.088346+0000","modified":"2023-10-16T15:50:47.188096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8407":{"gid":8407,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/2149952636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2149952636},{"type":"v1","addr":"172.21.15.29:6833","nonce":2149952636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T15:50:51.250 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-16T15:50:51.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9d5806af90 msgr2=0x7f9d5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9d5806af90 0x7f9d5806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c008040 comp rx=0 tx=0).stop 2023-10-16T15:50:51.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 msgr2=0x7f9d6c155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c155980 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f9d60000c00 tx=0x7f9d600046c0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 shutdown_connections 2023-10-16T15:50:51.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f9d5806af90 0x7f9d5806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d6c070450 0x7f9d6c155980 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d6c159950 0x7f9d6c0ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 --2- 172.21.15.29:0/4165174065 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d6c06c100 0x7f9d6c0c32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.251+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 >> 172.21.15.29:0/4165174065 conn(0x7f9d6c0ce6a0 msgr2=0x7f9d6c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.252+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 shutdown_connections 2023-10-16T15:50:51.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.252+0000 7f9d6bfff640 1 -- 172.21.15.29:0/4165174065 wait complete. 2023-10-16T15:50:51.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:50:44.088346+0000', 'modified': '2023-10-16T15:50:47.188096+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8407}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8407': {'gid': 8407, 'name': 'c', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6833/2149952636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 2149952636}, {'type': 'v1', 'addr': '172.21.15.29:6833', 'nonce': 2149952636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:50:51.675 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:50:51.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T15:50:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.915+0000 7fddae81a640 1 Processor -- start 2023-10-16T15:50:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.915+0000 7fddae81a640 1 -- start start 2023-10-16T15:50:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.916+0000 7fddae81a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdda8159950 0x7fdda815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae81a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdda806c810 0x7fdda80708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae81a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda815aef0 con 0x7fdda8070450 2023-10-16T15:50:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae81a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda8070e20 con 0x7fdda806c810 2023-10-16T15:50:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae81a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda815e490 con 0x7fdda8159950 2023-10-16T15:50:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdda806c810 0x7fdda80708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdda806c810 0x7fdda80708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52384/0 (socket says 172.21.15.29:52384) 2023-10-16T15:50:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 -- 172.21.15.29:0/3530842767 learned_addr learned my addr 172.21.15.29:0/3530842767 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddad818640 1 --2- 172.21.15.29:0/3530842767 >> v2:172.21.15.170:3300/0 conn(0x7fdda8159950 0x7fdda815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 -- 172.21.15.29:0/3530842767 >> v2:172.21.15.170:3300/0 conn(0x7fdda8159950 msgr2=0x7fdda815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 --2- 172.21.15.29:0/3530842767 >> v2:172.21.15.170:3300/0 conn(0x7fdda8159950 0x7fdda815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 -- 172.21.15.29:0/3530842767 >> v1:172.21.15.170:6789/0 conn(0x7fdda8070450 legacy=0x7fdda806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.917+0000 7fddae019640 1 -- 172.21.15.29:0/3530842767 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda815d910 con 0x7fdda806c810 2023-10-16T15:50:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.918+0000 7fddae019640 1 --2- 172.21.15.29:0/3530842767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda806c810 0x7fdda80708e0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fdda80ca950 tx=0x7fdda002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f911c500170d9cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.919+0000 7fddac816640 1 -- 172.21.15.29:0/3530842767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda002c9f0 con 0x7fdda806c810 2023-10-16T15:50:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.919+0000 7fddac816640 1 -- 172.21.15.29:0/3530842767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda002cb50 con 0x7fdda806c810 2023-10-16T15:50:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.919+0000 7fddac816640 1 -- 172.21.15.29:0/3530842767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda00343f0 con 0x7fdda806c810 2023-10-16T15:50:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.919+0000 7fddae81a640 1 -- 172.21.15.29:0/3530842767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda806c810 msgr2=0x7fdda80708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.919+0000 7fddae81a640 1 --2- 172.21.15.29:0/3530842767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda806c810 0x7fdda80708e0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fdda80ca950 tx=0x7fdda002be50 comp rx=0 tx=0).stop 2023-10-16T15:50:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.920+0000 7fddae81a640 1 -- 172.21.15.29:0/3530842767 shutdown_connections 2023-10-16T15:50:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.920+0000 7fddae81a640 1 --2- 172.21.15.29:0/3530842767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda806c810 0x7fdda80708e0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.920+0000 7fddae81a640 1 --2- 172.21.15.29:0/3530842767 >> v2:172.21.15.170:3300/0 conn(0x7fdda8159950 0x7fdda815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.920+0000 7fddae81a640 1 -- 172.21.15.29:0/3530842767 >> 172.21.15.29:0/3530842767 conn(0x7fdda80ce6a0 msgr2=0x7fdda80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.920+0000 7fddae81a640 1 -- 172.21.15.29:0/3530842767 shutdown_connections 2023-10-16T15:50:51.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.920+0000 7fddae81a640 1 -- 172.21.15.29:0/3530842767 wait complete. 2023-10-16T15:50:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.921+0000 7fddae81a640 1 Processor -- start 2023-10-16T15:50:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.922+0000 7fddae81a640 1 -- start start 2023-10-16T15:50:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddae81a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdda806c810 0x7fdda80c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddae81a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 0x7fdda814aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddae81a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdda8159950 0x7fdda814aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddae81a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda814b530 con 0x7fdda8070450 2023-10-16T15:50:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddae81a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80c5810 con 0x7fdda806c810 2023-10-16T15:50:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddae81a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80c5af0 con 0x7fdda8159950 2023-10-16T15:50:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddad017640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 0x7fdda814aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddad017640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 0x7fdda814aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52398/0 (socket says 172.21.15.29:52398) 2023-10-16T15:50:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddad017640 1 -- 172.21.15.29:0/2721675877 learned_addr learned my addr 172.21.15.29:0/2721675877 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddad818640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdda806c810 0x7fdda80c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.923+0000 7fddad017640 1 -- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdda8159950 msgr2=0x7fdda814aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddad017640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdda8159950 0x7fdda814aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddad017640 1 -- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdda806c810 msgr2=0x7fdda80c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddad017640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdda806c810 0x7fdda80c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddad017640 1 -- 172.21.15.29:0/2721675877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda0009b30 con 0x7fdda8070450 2023-10-16T15:50:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddad017640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 0x7fdda814aab0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800c410 tx=0x7fdd9800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fdd9e7fc640 1 -- 172.21.15.29:0/2721675877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd980035f0 con 0x7fdda8070450 2023-10-16T15:50:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fdd9e7fc640 1 -- 172.21.15.29:0/2721675877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd98003750 con 0x7fdda8070450 2023-10-16T15:50:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fdd9e7fc640 1 -- 172.21.15.29:0/2721675877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd98003b00 con 0x7fdda8070450 2023-10-16T15:50:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda80c5dd0 con 0x7fdda8070450 2023-10-16T15:50:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.924+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda80c9840 con 0x7fdda8070450 2023-10-16T15:50:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.927+0000 7fdd9e7fc640 1 -- 172.21.15.29:0/2721675877 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd98003c60 con 0x7fdda8070450 2023-10-16T15:50:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.928+0000 7fdd9e7fc640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdd8c06af90 0x7fdd8c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.928+0000 7fdd9e7fc640 1 -- 172.21.15.29:0/2721675877 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd98088a00 con 0x7fdda8070450 2023-10-16T15:50:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.928+0000 7fddb260c740 1 Processor -- start 2023-10-16T15:50:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.929+0000 7fddad818640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdd8c06af90 0x7fdd8c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.929+0000 7fddb260c740 1 -- start start 2023-10-16T15:50:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddb260c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863ed8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f07f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863ed85b0 0x564863f08440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddb260c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863df87a0 con 0x564863f05170 2023-10-16T15:50:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863e4a740 con 0x564863f05550 2023-10-16T15:50:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x564863e47e30 con 0x564863ed85b0 2023-10-16T15:50:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdd8c06af90 0x7fdd8c06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdda4006fd0 tx=0x7fdda4008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863ed8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863ed8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52408/0 (socket says 172.21.15.29:52408) 2023-10-16T15:50:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 -- 172.21.15.29:0/3653275768 learned_addr learned my addr 172.21.15.29:0/3653275768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad017640 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f07f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 -- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863ed85b0 msgr2=0x564863f08440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863ed85b0 0x564863f08440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 -- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 msgr2=0x564863f07f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f07f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.930+0000 7fddad818640 1 -- 172.21.15.29:0/3653275768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564863e07910 con 0x564863f05170 2023-10-16T15:50:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.931+0000 7fddad818640 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863ed8070 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdda4004300 tx=0x7fdda4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa38a3bad02bd7db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.931+0000 7fdd8b7fe640 1 -- 172.21.15.29:0/3653275768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda400ecb0 con 0x564863f05170 2023-10-16T15:50:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.931+0000 7fdd8b7fe640 1 -- 172.21.15.29:0/3653275768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda400c050 con 0x564863f05170 2023-10-16T15:50:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.931+0000 7fdd8b7fe640 1 -- 172.21.15.29:0/3653275768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4003b10 con 0x564863f05170 2023-10-16T15:50:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 -- 172.21.15.29:0/3653275768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 msgr2=0x564863ed8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863ed8070 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdda4004300 tx=0x7fdda4011000 comp rx=0 tx=0).stop 2023-10-16T15:50:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 -- 172.21.15.29:0/3653275768 shutdown_connections 2023-10-16T15:50:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863ed85b0 0x564863f08440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f07f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 --2- 172.21.15.29:0/3653275768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863ed8070 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 -- 172.21.15.29:0/3653275768 >> 172.21.15.29:0/3653275768 conn(0x564863eda380 msgr2=0x564863f22f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 -- 172.21.15.29:0/3653275768 shutdown_connections 2023-10-16T15:50:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.932+0000 7fddb260c740 1 -- 172.21.15.29:0/3653275768 wait complete. 2023-10-16T15:50:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.933+0000 7fddb260c740 1 -- start start 2023-10-16T15:50:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.933+0000 7fddb260c740 1 Processor -- start 2023-10-16T15:50:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f19a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f0a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 0x564863f0a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863e43720 con 0x564863ef30b0 2023-10-16T15:50:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863de1970 con 0x564863f05550 2023-10-16T15:50:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x564863de1610 con 0x564863f05170 2023-10-16T15:50:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddb260c740 10 client.0 _ll_register_callbacks cb 0x564863e039d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:50:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddae019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 0x564863f0a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddae019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 0x564863f0a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52420/0 (socket says 172.21.15.29:52420) 2023-10-16T15:50:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.934+0000 7fddae019640 1 -- 172.21.15.29:0/3353813755 learned_addr learned my addr 172.21.15.29:0/3353813755 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddad017640 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f0a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddae019640 1 -- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 msgr2=0x564863f19a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddae019640 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f19a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddae019640 1 -- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 msgr2=0x564863f0a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddae019640 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f0a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddae019640 1 -- 172.21.15.29:0/3353813755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda4004070 con 0x564863ef30b0 2023-10-16T15:50:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddae019640 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 0x564863f0a990 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009f40 tx=0x7fdda0034600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a482a1b0668282cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fdd88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:50:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fdd88ff9640 1 -- 172.21.15.29:0/3353813755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0034a40 con 0x564863ef30b0 2023-10-16T15:50:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.935+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564863e6a160 con 0x564863ef30b0 2023-10-16T15:50:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fdd88ff9640 1 -- 172.21.15.29:0/3353813755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdda0034ba0 con 0x564863ef30b0 2023-10-16T15:50:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fdd88ff9640 1 -- 172.21.15.29:0/3353813755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdda0034d60 con 0x564863ef30b0 2023-10-16T15:50:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 10 client.10452 fetch_fsmap learned FSMap version 100 2023-10-16T15:50:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x564863e41da0 con 0x564863ef30b0 2023-10-16T15:50:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fdd88ff9640 1 -- 172.21.15.29:0/3353813755 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564863e6a160 con 0x564863ef30b0 2023-10-16T15:50:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fdd88ff9640 1 -- 172.21.15.29:0/3353813755 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 100) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdda003c070 con 0x564863ef30b0 2023-10-16T15:50:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 10 client.10452 fetch_fsmap finished waiting for FSMap version 100 2023-10-16T15:50:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 10 client.10452 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:50:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 20 client.10452 populate_metadata read hostname 'smithi029' 2023-10-16T15:50:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863ef6300 0x564863ef4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 4 client.10452 mds_command: new command op to 8407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:50:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.936+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 --> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564863e15cc0 con 0x564863ef6300 2023-10-16T15:50:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.937+0000 7fddad818640 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863ef6300 0x564863ef4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.937+0000 7fddad818640 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863ef6300 0x564863ef4120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b52dd588d73bec server_cookie=dcb638060b5ab69e in_seq=0 out_seq=0 2023-10-16T15:50:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.937+0000 7fdd88ff9640 10 client.10452 ms_handle_connect on v2:172.21.15.29:6832/2149952636 2023-10-16T15:50:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.938+0000 7fdd88ff9640 1 -- 172.21.15.29:0/3353813755 <== mds.0 v2:172.21.15.29:6832/2149952636 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564863e15cc0 con 0x564863ef6300 2023-10-16T15:50:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.938+0000 7fdd88ff9640 10 client.10452 handle_command_reply: tid=0 2023-10-16T15:50:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.938+0000 7fddb260c740 1 client.10452 shutdown 2023-10-16T15:50:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.938+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 msgr2=0x564863f0a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.938+0000 7fddb260c740 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 0x564863f0a990 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009f40 tx=0x7fdda0034600 comp rx=0 tx=0).stop 2023-10-16T15:50:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 shutdown_connections 2023-10-16T15:50:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863ef6300 0x564863ef4120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863ef30b0 0x564863f0a990 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863f05550 0x564863f0a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 --2- 172.21.15.29:0/3353813755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f19a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fdd88ff9640 0 client.10452 ms_handle_reset on v2:172.21.15.29:6832/2149952636 2023-10-16T15:50:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 >> 172.21.15.29:0/3353813755 conn(0x564863ed85b0 msgr2=0x564863f07ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 shutdown_connections 2023-10-16T15:50:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 1 -- 172.21.15.29:0/3353813755 wait complete. 2023-10-16T15:50:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.939+0000 7fddb260c740 20 client.10452 trim_cache size 0 max 16384 2023-10-16T15:50:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.956+0000 7fddb260c740 1 Processor -- start 2023-10-16T15:50:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.956+0000 7fddb260c740 1 -- start start 2023-10-16T15:50:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863f0bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f07e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddb260c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 0x564863f08390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddb260c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863ba7f20 con 0x564863f05550 2023-10-16T15:50:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863de95c0 con 0x564863ef30b0 2023-10-16T15:50:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x564863f0c4e0 con 0x564863f05170 2023-10-16T15:50:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddae019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 0x564863f08390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddae019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 0x564863f08390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52432/0 (socket says 172.21.15.29:52432) 2023-10-16T15:50:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.957+0000 7fddae019640 1 -- 172.21.15.29:0/4004520933 learned_addr learned my addr 172.21.15.29:0/4004520933 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddad017640 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f07e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddad818640 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863f0bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddae019640 1 -- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 msgr2=0x564863f07e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddae019640 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f07e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddae019640 1 -- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 msgr2=0x564863f0bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddae019640 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863f0bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddae019640 1 -- 172.21.15.29:0/4004520933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564863de0b80 con 0x564863f05550 2023-10-16T15:50:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.958+0000 7fddae019640 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 0x564863f08390 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fdda0007100 tx=0x7fdda0039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defde7c3e301bfe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fdd8bfff640 1 -- 172.21.15.29:0/4004520933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0034c40 con 0x564863f05550 2023-10-16T15:50:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fdd8bfff640 1 -- 172.21.15.29:0/4004520933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda0034dc0 con 0x564863f05550 2023-10-16T15:50:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fdd8bfff640 1 -- 172.21.15.29:0/4004520933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda003e850 con 0x564863f05550 2023-10-16T15:50:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 -- 172.21.15.29:0/4004520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 msgr2=0x564863f08390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 0x564863f08390 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fdda0007100 tx=0x7fdda0039000 comp rx=0 tx=0).stop 2023-10-16T15:50:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 -- 172.21.15.29:0/4004520933 shutdown_connections 2023-10-16T15:50:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05550 0x564863f08390 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05170 0x564863f07e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 --2- 172.21.15.29:0/4004520933 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863f0bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 -- 172.21.15.29:0/4004520933 >> 172.21.15.29:0/4004520933 conn(0x564863ed85b0 msgr2=0x564863f239d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.959+0000 7fddb260c740 1 -- 172.21.15.29:0/4004520933 shutdown_connections 2023-10-16T15:50:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.960+0000 7fddb260c740 1 -- 172.21.15.29:0/4004520933 wait complete. 2023-10-16T15:50:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.960+0000 7fddb260c740 1 -- start start 2023-10-16T15:50:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.960+0000 7fddb260c740 1 Processor -- start 2023-10-16T15:50:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863edd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863f18db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05550 0x564863f192f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863edddd0 con 0x564863f05170 2023-10-16T15:50:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564863f264a0 con 0x564863ef30b0 2023-10-16T15:50:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x564863f266f0 con 0x564863f05550 2023-10-16T15:50:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.961+0000 7fddb260c740 10 client.0 _ll_register_callbacks cb 0x7fdda0009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:50:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863f18db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863f18db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52434/0 (socket says 172.21.15.29:52434) 2023-10-16T15:50:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 -- 172.21.15.29:0/510554708 learned_addr learned my addr 172.21.15.29:0/510554708 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 -- 172.21.15.29:0/510554708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05550 msgr2=0x564863f192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05550 0x564863f192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 -- 172.21.15.29:0/510554708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 msgr2=0x564863edd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863edd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 -- 172.21.15.29:0/510554708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564863e41da0 con 0x564863f05170 2023-10-16T15:50:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fddad017640 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863f18db0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fdd98057330 tx=0x7fdd98057e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd828d62161a1c26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.962+0000 7fdd88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:50:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fdd88ff9640 1 -- 172.21.15.29:0/510554708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9805f6a0 con 0x564863f05170 2023-10-16T15:50:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fdd88ff9640 1 -- 172.21.15.29:0/510554708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd9805b780 con 0x564863f05170 2023-10-16T15:50:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564863f0b750 con 0x564863f05170 2023-10-16T15:50:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fdd88ff9640 1 -- 172.21.15.29:0/510554708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd9805f930 con 0x564863f05170 2023-10-16T15:50:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 10 client.10458 fetch_fsmap learned FSMap version 100 2023-10-16T15:50:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x564863e06170 con 0x564863f05170 2023-10-16T15:50:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fdd88ff9640 1 -- 172.21.15.29:0/510554708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564863f0b750 con 0x564863f05170 2023-10-16T15:50:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fdd88ff9640 1 -- 172.21.15.29:0/510554708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 100) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdd98053420 con 0x564863f05170 2023-10-16T15:50:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 10 client.10458 fetch_fsmap finished waiting for FSMap version 100 2023-10-16T15:50:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 10 client.10458 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:50:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 20 client.10458 populate_metadata read hostname 'smithi029' 2023-10-16T15:50:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863f07f10 0x564863f082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 4 client.10458 mds_command: new command op to 8407 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:50:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.963+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 --> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fdda400c050 con 0x564863f07f10 2023-10-16T15:50:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fddad818640 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863f07f10 0x564863f082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fddad818640 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863f07f10 0x564863f082f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52fea58acc5356e9 server_cookie=3249da89542e3f21 in_seq=0 out_seq=0 2023-10-16T15:50:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fdd88ff9640 10 client.10458 ms_handle_connect on v2:172.21.15.29:6832/2149952636 2023-10-16T15:50:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fdd88ff9640 1 -- 172.21.15.29:0/510554708 <== mds.0 v2:172.21.15.29:6832/2149952636 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x7fdda400c050 con 0x564863f07f10 2023-10-16T15:50:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fdd88ff9640 10 client.10458 handle_command_reply: tid=0 2023-10-16T15:50:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fddb260c740 1 client.10458 shutdown 2023-10-16T15:50:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 msgr2=0x564863f18db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.964+0000 7fddb260c740 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863f18db0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fdd98057330 tx=0x7fdd98057e70 comp rx=0 tx=0).stop 2023-10-16T15:50:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 shutdown_connections 2023-10-16T15:50:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:6832/2149952636,v1:172.21.15.29:6833/2149952636] conn(0x564863f07f10 0x564863f082f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564863f05170 0x564863f18db0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x564863f05550 0x564863f192f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 --2- 172.21.15.29:0/510554708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564863ef30b0 0x564863edd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fdd88ff9640 0 client.10458 ms_handle_reset on v2:172.21.15.29:6832/2149952636 2023-10-16T15:50:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 >> 172.21.15.29:0/510554708 conn(0x564863ed85b0 msgr2=0x564863f0bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 shutdown_connections 2023-10-16T15:50:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 1 -- 172.21.15.29:0/510554708 wait complete. 2023-10-16T15:50:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.965+0000 7fddb260c740 20 client.10458 trim_cache size 0 max 16384 2023-10-16T15:50:51.978 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:50:51.978 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:50:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:50:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8407, 2023-10-16T15:50:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:50:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:50:51.979 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T15:50:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 7.3256899559999997, 2023-10-16T15:50:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 100, 2023-10-16T15:50:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 209, 2023-10-16T15:50:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 209, 2023-10-16T15:50:51.980 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.638670233999999 2023-10-16T15:50:51.981 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:50:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdd8c06af90 msgr2=0x7fdd8c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdd8c06af90 0x7fdd8c06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdda4006fd0 tx=0x7fdda4008040 comp rx=0 tx=0).stop 2023-10-16T15:50:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 msgr2=0x7fdda814aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 0x7fdda814aab0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800c410 tx=0x7fdd9800c8d0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 shutdown_connections 2023-10-16T15:50:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7fdd8c06af90 0x7fdd8c06d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdda8070450 0x7fdda814aab0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdda8159950 0x7fdda814aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 --2- 172.21.15.29:0/2721675877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdda806c810 0x7fdda80c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 >> 172.21.15.29:0/2721675877 conn(0x7fdda80ce6a0 msgr2=0x7fdda815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:51.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 shutdown_connections 2023-10-16T15:50:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:51.966+0000 7fddae81a640 1 -- 172.21.15.29:0/2721675877 wait complete. 2023-10-16T15:50:51.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T15:50:52.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.213+0000 7f126359e640 1 Processor -- start 2023-10-16T15:50:52.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.213+0000 7f126359e640 1 -- start start 2023-10-16T15:50:52.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f126359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f126415aad0 0x7f12640cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f126359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b4620 con 0x7f12640712c0 2023-10-16T15:50:52.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f126359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b4340 con 0x7f1264070d70 2023-10-16T15:50:52.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f126359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126406f520 con 0x7f126415aad0 2023-10-16T15:50:52.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f1262d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f12640712c0 0x7f1264070000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52778/0 (socket says 172.21.15.29:52778) 2023-10-16T15:50:52.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f1262d9d640 1 -- 172.21.15.29:0/3192527312 learned_addr learned my addr 172.21.15.29:0/3192527312 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:52.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.215+0000 7f126259c640 1 --2- 172.21.15.29:0/3192527312 >> v2:172.21.15.170:3300/0 conn(0x7f126415aad0 0x7f12640cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391039574 0 0) 0x7f12640b4620 con 0x7f12640712c0 2023-10-16T15:50:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c003520 con 0x7f12640712c0 2023-10-16T15:50:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027808951 0 0) 0x7f124c003520 con 0x7f12640712c0 2023-10-16T15:50:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 >> v2:172.21.15.170:3300/0 conn(0x7f126415aad0 msgr2=0x7f12640cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 --2- 172.21.15.29:0/3192527312 >> v2:172.21.15.170:3300/0 conn(0x7f126415aad0 0x7f12640cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 >> v1:172.21.15.170:6789/0 conn(0x7f1264070d70 legacy=0x7f126406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126406f800 con 0x7f12640712c0 2023-10-16T15:50:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f125c0019f0 con 0x7f12640712c0 2023-10-16T15:50:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 >> v1:172.21.15.29:6789/0 conn(0x7f12640712c0 legacy=0x7f1264070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f124c0039c0 0x7f124c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f124c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f124c005180 0x7f124c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126406f520 con 0x7f124c0047c0 2023-10-16T15:50:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c00c060 con 0x7f124c005180 2023-10-16T15:50:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.216+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124c00c220 con 0x7f124c0039c0 2023-10-16T15:50:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1261d9b640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f124c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1262d9d640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f124c005180 0x7f124c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f126259c640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f124c0039c0 0x7f124c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1261d9b640 1 -- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f124c0039c0 msgr2=0x7f124c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1261d9b640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f124c0039c0 0x7f124c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1261d9b640 1 -- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f124c005180 msgr2=0x7f124c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1261d9b640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f124c005180 0x7f124c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1261d9b640 1 -- 172.21.15.29:0/3192527312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c003520 con 0x7f124c0047c0 2023-10-16T15:50:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.217+0000 7f1262d9d640 1 -- 172.21.15.29:0/3192527312 reap_dead start 2023-10-16T15:50:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.218+0000 7f1261d9b640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f124c004c40 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f1258001750 tx=0x7f124c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79757e6a3baa591 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12580052d0 con 0x7f124c0047c0 2023-10-16T15:50:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125802e050 con 0x7f124c0047c0 2023-10-16T15:50:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126159a640 1 -- 172.21.15.29:0/3192527312 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258003830 con 0x7f124c0047c0 2023-10-16T15:50:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 -- 172.21.15.29:0/3192527312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 msgr2=0x7f124c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f124c004c40 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f1258001750 tx=0x7f124c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 -- 172.21.15.29:0/3192527312 shutdown_connections 2023-10-16T15:50:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 --2- 172.21.15.29:0/3192527312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f124c004c40 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 -- 172.21.15.29:0/3192527312 >> 172.21.15.29:0/3192527312 conn(0x7f12640b0a90 msgr2=0x7f12640ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 -- 172.21.15.29:0/3192527312 shutdown_connections 2023-10-16T15:50:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.219+0000 7f126359e640 1 -- 172.21.15.29:0/3192527312 wait complete. 2023-10-16T15:50:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.221+0000 7f126359e640 1 Processor -- start 2023-10-16T15:50:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.221+0000 7f126359e640 1 -- start start 2023-10-16T15:50:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.222+0000 7f126359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f126414aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.222+0000 7f126359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640c48b0 0x7f12640ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.222+0000 7f126359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640c4d30 0x7f12640cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.222+0000 7f126359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12640c2250 con 0x7f124c0047c0 2023-10-16T15:50:52.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.222+0000 7f126359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126414b020 con 0x7f12640c48b0 2023-10-16T15:50:52.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.222+0000 7f126359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126415d660 con 0x7f12640c4d30 2023-10-16T15:50:52.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f126414aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f126414aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52460/0 (socket says 172.21.15.29:52460) 2023-10-16T15:50:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 -- 172.21.15.29:0/2004434184 learned_addr learned my addr 172.21.15.29:0/2004434184 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f1261d9b640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640c48b0 0x7f12640ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 -- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640c4d30 msgr2=0x7f12640cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640c4d30 0x7f12640cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 -- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640c48b0 msgr2=0x7f12640ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640c48b0 0x7f12640ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.223+0000 7f126259c640 1 -- 172.21.15.29:0/2004434184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126415d940 con 0x7f124c0047c0 2023-10-16T15:50:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.224+0000 7f126259c640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f126414aae0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f1254011650 tx=0x7f1254011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.224+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125400f9c0 con 0x7f124c0047c0 2023-10-16T15:50:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.224+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125400fb20 con 0x7f124c0047c0 2023-10-16T15:50:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.224+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125400fe90 con 0x7f124c0047c0 2023-10-16T15:50:52.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.224+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126415dc20 con 0x7f124c0047c0 2023-10-16T15:50:52.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.224+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126415d1f0 con 0x7f124c0047c0 2023-10-16T15:50:52.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.225+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126406f270 con 0x7f124c0047c0 2023-10-16T15:50:52.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.226+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1254015460 con 0x7f124c0047c0 2023-10-16T15:50:52.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.227+0000 7f1252ffd640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f124406af90 0x7f124406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:52.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.227+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1254089d50 con 0x7f124c0047c0 2023-10-16T15:50:52.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.228+0000 7f1261d9b640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f124406af90 0x7f124406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:52.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.229+0000 7f1261d9b640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f124406af90 0x7f124406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1258033000 tx=0x7f1258009030 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:52.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.232+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1254054c30 con 0x7f124c0047c0 2023-10-16T15:50:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.469+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f12640c2390 con 0x7f124c0047c0 2023-10-16T15:50:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.470+0000 7f1252ffd640 1 -- 172.21.15.29:0/2004434184 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v100) v1 ==== 53+0+73 (secure 0 0 0) 0x7f12640c2390 con 0x7f124c0047c0 2023-10-16T15:50:52.472 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T15:50:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.473+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f124406af90 msgr2=0x7f124406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.473+0000 7f126359e640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f124406af90 0x7f124406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1258033000 tx=0x7f1258009030 comp rx=0 tx=0).stop 2023-10-16T15:50:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.473+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 msgr2=0x7f126414aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.473+0000 7f126359e640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f126414aae0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f1254011650 tx=0x7f1254011b10 comp rx=0 tx=0).stop 2023-10-16T15:50:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 shutdown_connections 2023-10-16T15:50:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f124406af90 0x7f124406d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12640c4d30 0x7f12640cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12640c48b0 0x7f12640ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 --2- 172.21.15.29:0/2004434184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f124c0047c0 0x7f126414aae0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 >> 172.21.15.29:0/2004434184 conn(0x7f12640b0a90 msgr2=0x7f12640ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 shutdown_connections 2023-10-16T15:50:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:52.474+0000 7f126359e640 1 -- 172.21.15.29:0/2004434184 wait complete. 2023-10-16T15:50:52.495 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T15:50:52.495 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T15:50:52.496 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T15:50:52.496 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T15:50:52.496 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T15:50:52.496 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T15:50:52.497 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T15:50:52.497 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T15:50:52.498 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:50:52.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:52.498 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T15:50:52.555 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T15:50:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T15:50:52.555 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T15:50:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T15:50:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.556 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:50:52.556 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:50:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T15:50:52.557 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:50:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:50:52.557 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T15:50:52.557 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T15:50:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T15:50:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T15:50:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T15:50:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36672sec preferred_lft 36672sec 2023-10-16T15:50:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T15:50:52.558 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.559 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:50:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T15:50:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T15:50:52.559 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T15:50:52.559 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:50:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T15:50:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T15:50:52.560 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.560 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:50:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:50:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T15:50:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.561 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:50:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:50:52.561 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T15:50:52.562 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:50:52.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:52.563 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T15:50:52.659 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T15:50:52.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:52.665 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T15:50:52.722 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T15:50:52.723 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T15:50:52.723 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T15:50:52.723 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T15:50:52.724 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T15:50:52.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:52.725 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T15:50:52.726 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T15:50:52.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T15:50:52.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T15:50:52.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T15:50:52.728 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T15:50:52.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:52.929 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:50:52.981 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:50:52.981 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:50:52.982 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:50:52.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:52.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T15:50:53.087 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:50:53.088 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T15:50:53.150 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:50:53.150 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:50:53.151 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:50:53.207 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T15:50:53.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:53.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:50:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:50:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:50:53.364 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:50:53.388 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:50:53.388 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:50:53.389 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:50:53.389 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:50:53.389 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:50:53.417 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:50:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:50:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:50:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:50:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:50:53.418 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T15:50:53.419 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T15:50:53.419 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T15:50:53.420 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T15:50:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T15:50:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T15:50:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T15:50:53.421 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T15:50:53.423 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T15:50:53.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:53.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:50:53.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:53.520 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T15:50:53.592 INFO:teuthology.orchestra.run.smithi029.stdout:client7601 2023-10-16T15:50:53.593 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:50:53.593 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:50:53.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T15:50:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.877+0000 7f065a6af640 1 Processor -- start 2023-10-16T15:50:53.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.877+0000 7f065a6af640 1 -- start start 2023-10-16T15:50:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f065a6af640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 0x7f0654069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f065a6af640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f06540caac0 0x7f065406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f065a6af640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06540cb030 con 0x7f06540c9ca0 2023-10-16T15:50:53.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f065a6af640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06540c91c0 con 0x7f06540caac0 2023-10-16T15:50:53.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f065a6af640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c94a0 con 0x7f065415ad70 2023-10-16T15:50:53.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f064b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f06540c9ca0 0x7f06540ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52784/0 (socket says 172.21.15.29:52784) 2023-10-16T15:50:53.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.879+0000 7f064b7fe640 1 -- 172.21.15.29:0/1756949901 learned_addr learned my addr 172.21.15.29:0/1756949901 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:53.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f0653fff640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 0x7f0654069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:53.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f0658c4c640 1 --2- 172.21.15.29:0/1756949901 >> v2:172.21.15.170:3300/0 conn(0x7f06540caac0 0x7f065406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:53.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144779207 0 0) 0x7f06540cb030 con 0x7f06540c9ca0 2023-10-16T15:50:53.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0640003520 con 0x7f06540c9ca0 2023-10-16T15:50:53.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2493312972 0 0) 0x7f0640003520 con 0x7f06540c9ca0 2023-10-16T15:50:53.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 msgr2=0x7f0654069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:53.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 0x7f0654069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 >> v2:172.21.15.170:3300/0 conn(0x7f06540caac0 msgr2=0x7f065406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:53.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 --2- 172.21.15.29:0/1756949901 >> v2:172.21.15.170:3300/0 conn(0x7f06540caac0 0x7f065406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540c9780 con 0x7f06540c9ca0 2023-10-16T15:50:53.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f063c001670 con 0x7f06540c9ca0 2023-10-16T15:50:53.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 >> v1:172.21.15.29:6789/0 conn(0x7f06540c9ca0 legacy=0x7f06540ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:53.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0640003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.880+0000 7f06537fe640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 0x7f06400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f06537fe640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0640004d00 0x7f0640005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c91c0 con 0x7f0640003970 2023-10-16T15:50:53.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540c94a0 con 0x7f0640004d00 2023-10-16T15:50:53.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064000b980 con 0x7f065415ad70 2023-10-16T15:50:53.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f0653fff640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0640003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:53.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f0653fff640 1 -- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 msgr2=0x7f06400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:53.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f0653fff640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415ad70 0x7f06400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f0653fff640 1 -- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0640004d00 msgr2=0x7f0640005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:53.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f0653fff640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0640004d00 0x7f0640005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.881+0000 7f0653fff640 1 -- 172.21.15.29:0/1756949901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640003520 con 0x7f0640003970 2023-10-16T15:50:53.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.882+0000 7f0653fff640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0640003d90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f0644009fc0 tx=0x7f06400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8e5016c01fab03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:53.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064402cc60 con 0x7f0640003970 2023-10-16T15:50:53.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064400f040 con 0x7f0640003970 2023-10-16T15:50:53.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f06537fe640 1 -- 172.21.15.29:0/1756949901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644034440 con 0x7f0640003970 2023-10-16T15:50:53.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f065a6af640 1 -- 172.21.15.29:0/1756949901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 msgr2=0x7f0640003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f065a6af640 1 --2- 172.21.15.29:0/1756949901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0640003d90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f0644009fc0 tx=0x7f06400057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f0658c4c640 1 -- 172.21.15.29:0/1756949901 reap_dead start 2023-10-16T15:50:53.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f065a6af640 1 -- 172.21.15.29:0/1756949901 shutdown_connections 2023-10-16T15:50:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f065a6af640 1 -- 172.21.15.29:0/1756949901 >> 172.21.15.29:0/1756949901 conn(0x7f06540ce9b0 msgr2=0x7f06540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f065a6af640 1 -- 172.21.15.29:0/1756949901 shutdown_connections 2023-10-16T15:50:53.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.883+0000 7f065a6af640 1 -- 172.21.15.29:0/1756949901 wait complete. 2023-10-16T15:50:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.885+0000 7f065a6af640 1 Processor -- start 2023-10-16T15:50:53.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.886+0000 7f065a6af640 1 -- start start 2023-10-16T15:50:53.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f065a6af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0654145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f065a6af640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06540c27e0 0x7f0654159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f065a6af640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415a130 0x7f06540c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f065a6af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06541f2700 con 0x7f0640003970 2023-10-16T15:50:53.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f065a6af640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065415a5b0 con 0x7f06540c27e0 2023-10-16T15:50:53.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f065a6af640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06541464e0 con 0x7f065415a130 2023-10-16T15:50:53.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0654145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:53.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0654145c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52474/0 (socket says 172.21.15.29:52474) 2023-10-16T15:50:53.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 -- 172.21.15.29:0/1008422917 learned_addr learned my addr 172.21.15.29:0/1008422917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:53.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 -- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415a130 msgr2=0x7f06540c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:53.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415a130 0x7f06540c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 -- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06540c27e0 msgr2=0x7f0654159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:53.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06540c27e0 0x7f0654159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:53.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.887+0000 7f0653fff640 1 -- 172.21.15.29:0/1008422917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06541f1830 con 0x7f0640003970 2023-10-16T15:50:53.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.888+0000 7f0653fff640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0654145c90 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f064400e230 tx=0x7f0644009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:53.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.888+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644034b50 con 0x7f0640003970 2023-10-16T15:50:53.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.888+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644034cb0 con 0x7f0640003970 2023-10-16T15:50:53.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.888+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644003b90 con 0x7f0640003970 2023-10-16T15:50:53.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.888+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06541461d0 con 0x7f0640003970 2023-10-16T15:50:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.888+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06541f1d60 con 0x7f0640003970 2023-10-16T15:50:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.889+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065404e160 con 0x7f0640003970 2023-10-16T15:50:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.890+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0644040450 con 0x7f0640003970 2023-10-16T15:50:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.891+0000 7f06517fa640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f063406af90 0x7f063406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.891+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f06440ac000 con 0x7f0640003970 2023-10-16T15:50:53.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.892+0000 7f064b7fe640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f063406af90 0x7f063406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:53.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.893+0000 7f064b7fe640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f063406af90 0x7f063406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06540b34c0 tx=0x7f063c0014f0 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:53.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:53.896+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0644077130 con 0x7f0640003970 2023-10-16T15:50:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.114+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 --> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0654069da0 con 0x7f063406af90 2023-10-16T15:50:54.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.118+0000 7f06517fa640 1 -- 172.21.15.29:0/1008422917 <== mgr.7406 v2:172.21.15.170:6832/780156504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f063c001bb0 con 0x7f063406af90 2023-10-16T15:50:54.119 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:50:54.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:50:54.119 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T15:50:54.120 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:50:54.120 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:50:54.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f063406af90 msgr2=0x7f063406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:54.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f063406af90 0x7f063406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06540b34c0 tx=0x7f063c0014f0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 msgr2=0x7f0654145c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:54.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0654145c90 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f064400e230 tx=0x7f0644009750 comp rx=0 tx=0).stop 2023-10-16T15:50:54.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 shutdown_connections 2023-10-16T15:50:54.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f063406af90 0x7f063406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f065415a130 0x7f06540c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06540c27e0 0x7f0654159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 --2- 172.21.15.29:0/1008422917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0640003970 0x7f0654145c90 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 >> 172.21.15.29:0/1008422917 conn(0x7f06540ce9b0 msgr2=0x7f0654159170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:54.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 shutdown_connections 2023-10-16T15:50:54.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.120+0000 7f065a6af640 1 -- 172.21.15.29:0/1008422917 wait complete. 2023-10-16T15:50:54.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T15:50:54.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.369+0000 7f043ec9e640 1 Processor -- start 2023-10-16T15:50:54.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.369+0000 7f043ec9e640 1 -- start start 2023-10-16T15:50:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043ec9e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f043806fb50 0x7f04380cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043ec9e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f043806c9e0 0x7f04380ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043ec9e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04380ca560 con 0x7f04380c9060 2023-10-16T15:50:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043ec9e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04380c9950 con 0x7f043806c9e0 2023-10-16T15:50:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043ec9e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04380c9c30 con 0x7f043806fb50 2023-10-16T15:50:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043dc9c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f04380c9060 0x7f043806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52780/0 (socket says 172.21.15.29:52780) 2023-10-16T15:50:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043dc9c640 1 -- 172.21.15.29:0/3044818792 learned_addr learned my addr 172.21.15.29:0/3044818792 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043e49d640 1 --2- 172.21.15.29:0/3044818792 >> v2:172.21.15.29:3300/0 conn(0x7f043806c9e0 0x7f04380ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.371+0000 7f043d49b640 1 --2- 172.21.15.29:0/3044818792 >> v2:172.21.15.170:3300/0 conn(0x7f043806fb50 0x7f04380cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.372+0000 7f043e49d640 1 -- 172.21.15.29:0/3044818792 >> v2:172.21.15.170:3300/0 conn(0x7f043806fb50 msgr2=0x7f04380cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.372+0000 7f043e49d640 1 --2- 172.21.15.29:0/3044818792 >> v2:172.21.15.170:3300/0 conn(0x7f043806fb50 0x7f04380cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.372+0000 7f043e49d640 1 -- 172.21.15.29:0/3044818792 >> v1:172.21.15.29:6789/0 conn(0x7f04380c9060 legacy=0x7f043806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.372+0000 7f043e49d640 1 -- 172.21.15.29:0/3044818792 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04380c9f10 con 0x7f043806c9e0 2023-10-16T15:50:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.373+0000 7f043e49d640 1 --2- 172.21.15.29:0/3044818792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04380ccde0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f0438164cb0 tx=0x7f043002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fe93d85542048a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.373+0000 7f043cc9a640 1 -- 172.21.15.29:0/3044818792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430031830 con 0x7f043806c9e0 2023-10-16T15:50:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.373+0000 7f043cc9a640 1 -- 172.21.15.29:0/3044818792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0430031990 con 0x7f043806c9e0 2023-10-16T15:50:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.373+0000 7f043cc9a640 1 -- 172.21.15.29:0/3044818792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430031d20 con 0x7f043806c9e0 2023-10-16T15:50:54.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 -- 172.21.15.29:0/3044818792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 msgr2=0x7f04380ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:54.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 --2- 172.21.15.29:0/3044818792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04380ccde0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f0438164cb0 tx=0x7f043002b8d0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 -- 172.21.15.29:0/3044818792 shutdown_connections 2023-10-16T15:50:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 --2- 172.21.15.29:0/3044818792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04380ccde0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 --2- 172.21.15.29:0/3044818792 >> v2:172.21.15.170:3300/0 conn(0x7f043806fb50 0x7f04380cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 -- 172.21.15.29:0/3044818792 >> 172.21.15.29:0/3044818792 conn(0x7f04380ce6a0 msgr2=0x7f04380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 -- 172.21.15.29:0/3044818792 shutdown_connections 2023-10-16T15:50:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.374+0000 7f043ec9e640 1 -- 172.21.15.29:0/3044818792 wait complete. 2023-10-16T15:50:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.376+0000 7f043ec9e640 1 Processor -- start 2023-10-16T15:50:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.376+0000 7f043ec9e640 1 -- start start 2023-10-16T15:50:54.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.377+0000 7f043ec9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04381f9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:54.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.377+0000 7f043ec9e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043806fb50 0x7f04380c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:54.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.377+0000 7f043ec9e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04380c9060 0x7f04380c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:54.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.377+0000 7f043ec9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04381648a0 con 0x7f043806c9e0 2023-10-16T15:50:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.377+0000 7f043ec9e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04381f9bb0 con 0x7f04380c9060 2023-10-16T15:50:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.377+0000 7f043ec9e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04380b3110 con 0x7f043806fb50 2023-10-16T15:50:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04381f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04381f9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39234/0 (socket says 172.21.15.29:39234) 2023-10-16T15:50:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 -- 172.21.15.29:0/713459728 learned_addr learned my addr 172.21.15.29:0/713459728 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043d49b640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043806fb50 0x7f04380c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043e49d640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04380c9060 0x7f04380c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 -- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043806fb50 msgr2=0x7f04380c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043806fb50 0x7f04380c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 -- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04380c9060 msgr2=0x7f04380c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04380c9060 0x7f04380c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.378+0000 7f043dc9c640 1 -- 172.21.15.29:0/713459728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0430009400 con 0x7f043806c9e0 2023-10-16T15:50:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.379+0000 7f043dc9c640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04381f9670 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f043400cbc0 tx=0x7f043400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.379+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043400aca0 con 0x7f043806c9e0 2023-10-16T15:50:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.379+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043400ae00 con 0x7f043806c9e0 2023-10-16T15:50:54.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.379+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434012600 con 0x7f043806c9e0 2023-10-16T15:50:54.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.379+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04380b33f0 con 0x7f043806c9e0 2023-10-16T15:50:54.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.379+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04380b39b0 con 0x7f043806c9e0 2023-10-16T15:50:54.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.380+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043804e9b0 con 0x7f043806c9e0 2023-10-16T15:50:54.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.381+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 116) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0434007070 con 0x7f043806c9e0 2023-10-16T15:50:54.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.382+0000 7f042e7fc640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f041c06af90 0x7f041c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.382+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0434089690 con 0x7f043806c9e0 2023-10-16T15:50:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.383+0000 7f043d49b640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f041c06af90 0x7f041c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.384+0000 7f043d49b640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f041c06af90 0x7f041c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04380c4110 tx=0x7f0428008040 comp rx=0 tx=0).ready entity=mgr.7406 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.386+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0434054620 con 0x7f043806c9e0 2023-10-16T15:50:54.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:54.627+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f04380b4620 con 0x7f043806c9e0 2023-10-16T15:50:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.191+0000 7f042e7fc640 1 -- 172.21.15.29:0/713459728 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v117) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0434057c40 con 0x7f043806c9e0 2023-10-16T15:50:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.194+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f041c06af90 msgr2=0x7f041c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.194+0000 7f043ec9e640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f041c06af90 0x7f041c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04380c4110 tx=0x7f0428008040 comp rx=0 tx=0).stop 2023-10-16T15:50:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.194+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 msgr2=0x7f04381f9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.194+0000 7f043ec9e640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04381f9670 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f043400cbc0 tx=0x7f043400a060 comp rx=0 tx=0).stop 2023-10-16T15:50:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 shutdown_connections 2023-10-16T15:50:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f041c06af90 0x7f041c06d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f04380c9060 0x7f04380c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f043806fb50 0x7f04380c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 --2- 172.21.15.29:0/713459728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f043806c9e0 0x7f04381f9670 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 >> 172.21.15.29:0/713459728 conn(0x7f04380ce6a0 msgr2=0x7f04380c14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 shutdown_connections 2023-10-16T15:50:55.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.195+0000 7f043ec9e640 1 -- 172.21.15.29:0/713459728 wait complete. 2023-10-16T15:50:55.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T15:50:55.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.425+0000 7f258d328640 1 Processor -- start 2023-10-16T15:50:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.425+0000 7f258d328640 1 -- start start 2023-10-16T15:50:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f258d328640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f258815cd70 0x7f25880d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f258d328640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258815d150 con 0x7f25880cded0 2023-10-16T15:50:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f258d328640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25880d1180 con 0x7f25880b8480 2023-10-16T15:50:55.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f258d328640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25880d1460 con 0x7f258815cd70 2023-10-16T15:50:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f258cb27640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25880cded0 0x7f25880b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52798/0 (socket says 172.21.15.29:52798) 2023-10-16T15:50:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f258cb27640 1 -- 172.21.15.29:0/1463457612 learned_addr learned my addr 172.21.15.29:0/1463457612 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.427+0000 7f2587fff640 1 --2- 172.21.15.29:0/1463457612 >> v2:172.21.15.170:3300/0 conn(0x7f258815cd70 0x7f25880d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366735393 0 0) 0x7f258815d150 con 0x7f25880cded0 2023-10-16T15:50:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2570003520 con 0x7f25880cded0 2023-10-16T15:50:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 586017943 0 0) 0x7f2570003520 con 0x7f25880cded0 2023-10-16T15:50:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 >> v2:172.21.15.170:3300/0 conn(0x7f258815cd70 msgr2=0x7f25880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 --2- 172.21.15.29:0/1463457612 >> v2:172.21.15.170:3300/0 conn(0x7f258815cd70 0x7f25880d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 >> v1:172.21.15.170:6789/0 conn(0x7f25880b8480 legacy=0x7f25880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880d1740 con 0x7f25880cded0 2023-10-16T15:50:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2580001670 con 0x7f25880cded0 2023-10-16T15:50:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 >> v1:172.21.15.29:6789/0 conn(0x7f25880cded0 legacy=0x7f25880b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570003970 0x7f2570003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25700047c0 0x7f2570004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2570005180 0x7f257000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25880d1460 con 0x7f2570005180 2023-10-16T15:50:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257000c020 con 0x7f25700047c0 2023-10-16T15:50:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.428+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257000c1e0 con 0x7f2570003970 2023-10-16T15:50:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f258cb27640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2570005180 0x7f257000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f25877fe640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25700047c0 0x7f2570004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f2587fff640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570003970 0x7f2570003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f258cb27640 1 -- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570003970 msgr2=0x7f2570003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f258cb27640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570003970 0x7f2570003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f258cb27640 1 -- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25700047c0 msgr2=0x7f2570004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f258cb27640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f25700047c0 0x7f2570004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.429+0000 7f258cb27640 1 -- 172.21.15.29:0/1463457612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570003520 con 0x7f2570005180 2023-10-16T15:50:55.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.430+0000 7f258cb27640 1 -- 172.21.15.29:0/1463457612 reap_dead start 2023-10-16T15:50:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.430+0000 7f258cb27640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2570005180 0x7f257000b980 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f258000c240 tx=0x7f25700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f89c084db5679eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.430+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580005ad0 con 0x7f2570005180 2023-10-16T15:50:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.430+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580005c70 con 0x7f2570005180 2023-10-16T15:50:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.430+0000 7f2586ffd640 1 -- 172.21.15.29:0/1463457612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258002c8b0 con 0x7f2570005180 2023-10-16T15:50:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 -- 172.21.15.29:0/1463457612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2570005180 msgr2=0x7f257000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2570005180 0x7f257000b980 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f258000c240 tx=0x7f25700057f0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 -- 172.21.15.29:0/1463457612 shutdown_connections 2023-10-16T15:50:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 --2- 172.21.15.29:0/1463457612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2570005180 0x7f257000b980 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 -- 172.21.15.29:0/1463457612 >> 172.21.15.29:0/1463457612 conn(0x7f25880cf570 msgr2=0x7f25880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 -- 172.21.15.29:0/1463457612 shutdown_connections 2023-10-16T15:50:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.431+0000 7f258d328640 1 -- 172.21.15.29:0/1463457612 wait complete. 2023-10-16T15:50:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.434+0000 7f258d328640 1 Processor -- start 2023-10-16T15:50:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.435+0000 7f258d328640 1 -- start start 2023-10-16T15:50:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.436+0000 7f258d328640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570005180 0x7f25880a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.436+0000 7f258d328640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f258815e390 0x7f258814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.436+0000 7f258d328640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 0x7f258814f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.436+0000 7f258d328640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258814f9d0 con 0x7f258815e810 2023-10-16T15:50:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.436+0000 7f258d328640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258815ed70 con 0x7f258815e390 2023-10-16T15:50:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.436+0000 7f258d328640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258815f050 con 0x7f2570005180 2023-10-16T15:50:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 0x7f258814f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 0x7f258814f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39278/0 (socket says 172.21.15.29:39278) 2023-10-16T15:50:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 -- 172.21.15.29:0/2410282790 learned_addr learned my addr 172.21.15.29:0/2410282790 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f2587fff640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570005180 0x7f25880a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f25877fe640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f258815e390 0x7f258814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 -- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570005180 msgr2=0x7f25880a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570005180 0x7f25880a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 -- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f258815e390 msgr2=0x7f258814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f258815e390 0x7f258814ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 -- 172.21.15.29:0/2410282790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258815f330 con 0x7f258815e810 2023-10-16T15:50:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.437+0000 7f258cb27640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 0x7f258814f490 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f258000c360 tx=0x7f2580001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.438+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25800049e0 con 0x7f258815e810 2023-10-16T15:50:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.438+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580004b40 con 0x7f258815e810 2023-10-16T15:50:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.438+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258815f610 con 0x7f258815e810 2023-10-16T15:50:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.438+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258002de80 con 0x7f258815e810 2023-10-16T15:50:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.438+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25880a9ca0 con 0x7f258815e810 2023-10-16T15:50:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.439+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25880d35e0 con 0x7f258815e810 2023-10-16T15:50:55.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.440+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2580032070 con 0x7f258815e810 2023-10-16T15:50:55.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.441+0000 7f2584ff9640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 0x7f257406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.441+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f258003c070 con 0x7f258815e810 2023-10-16T15:50:55.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.441+0000 7f2587fff640 1 -- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 msgr2=0x7f257406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:55.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.441+0000 7f2587fff640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 0x7f257406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.445+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25800759c0 con 0x7f258815e810 2023-10-16T15:50:55.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.639+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f258804e9b0 con 0x7f258815e810 2023-10-16T15:50:55.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.642+0000 7f2587fff640 1 -- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 msgr2=0x7f257406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:55.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.642+0000 7f2587fff640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 0x7f257406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:55.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.644+0000 7f2584ff9640 1 -- 172.21.15.29:0/2410282790 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2580045020 con 0x7f258815e810 2023-10-16T15:50:55.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 msgr2=0x7f257406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:55.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 0x7f257406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 msgr2=0x7f258814f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 0x7f258814f490 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f258000c360 tx=0x7f2580001a00 comp rx=0 tx=0).stop 2023-10-16T15:50:55.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 shutdown_connections 2023-10-16T15:50:55.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f257406b010 0x7f257406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f258815e810 0x7f258814f490 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f258815e390 0x7f258814ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 --2- 172.21.15.29:0/2410282790 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2570005180 0x7f25880a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.647+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 >> 172.21.15.29:0/2410282790 conn(0x7f25880cf570 msgr2=0x7f258815be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:55.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.648+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 shutdown_connections 2023-10-16T15:50:55.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.648+0000 7f258d328640 1 -- 172.21.15.29:0/2410282790 wait complete. 2023-10-16T15:50:55.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T15:50:55.717 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:55.714+0000 7fa23ac3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:50:55.759 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:55.758+0000 7f761320d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:50:55.768 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:55.767+0000 7fa23ac3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:50:55.817 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:55.815+0000 7f761320d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:50:55.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.877+0000 7f6eb7475640 1 Processor -- start 2023-10-16T15:50:55.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.877+0000 7f6eb7475640 1 -- start start 2023-10-16T15:50:55.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb7475640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6eb00cf710 0x7f6eb00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb7475640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6eb00b6cd0 0x7f6eb00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb7475640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb00d3980 con 0x7f6eb00cfaf0 2023-10-16T15:50:55.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb7475640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb00d3c60 con 0x7f6eb00cf710 2023-10-16T15:50:55.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb7475640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb00ce630 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb4a10640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6eb00cfaf0 0x7f6eb00d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52804/0 (socket says 172.21.15.29:52804) 2023-10-16T15:50:55.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb5211640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6eb00cf710 0x7f6eb00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb4a10640 1 -- 172.21.15.29:0/2682717984 learned_addr learned my addr 172.21.15.29:0/2682717984 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:50:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.879+0000 7f6eb5a12640 1 --2- 172.21.15.29:0/2682717984 >> v2:172.21.15.170:3300/0 conn(0x7f6eb00b6cd0 0x7f6eb00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.880+0000 7f6eb5211640 1 -- 172.21.15.29:0/2682717984 >> v2:172.21.15.170:3300/0 conn(0x7f6eb00b6cd0 msgr2=0x7f6eb00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:55.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.880+0000 7f6eb5211640 1 --2- 172.21.15.29:0/2682717984 >> v2:172.21.15.170:3300/0 conn(0x7f6eb00b6cd0 0x7f6eb00b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.880+0000 7f6eb5211640 1 -- 172.21.15.29:0/2682717984 >> v1:172.21.15.29:6789/0 conn(0x7f6eb00cfaf0 legacy=0x7f6eb00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.880+0000 7f6eb5211640 1 -- 172.21.15.29:0/2682717984 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb00ce910 con 0x7f6eb00cf710 2023-10-16T15:50:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.881+0000 7f6eb5211640 1 --2- 172.21.15.29:0/2682717984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00cf710 0x7f6eb00b2d10 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00b5890 tx=0x7f6eac02be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b769bffa222553 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.881+0000 7f6ea7fff640 1 -- 172.21.15.29:0/2682717984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac02cc20 con 0x7f6eb00cf710 2023-10-16T15:50:55.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.881+0000 7f6ea7fff640 1 -- 172.21.15.29:0/2682717984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac02cd80 con 0x7f6eb00cf710 2023-10-16T15:50:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.881+0000 7f6ea7fff640 1 -- 172.21.15.29:0/2682717984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac034660 con 0x7f6eb00cf710 2023-10-16T15:50:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 -- 172.21.15.29:0/2682717984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00cf710 msgr2=0x7f6eb00b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2682717984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00cf710 0x7f6eb00b2d10 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00b5890 tx=0x7f6eac02be70 comp rx=0 tx=0).stop 2023-10-16T15:50:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 -- 172.21.15.29:0/2682717984 shutdown_connections 2023-10-16T15:50:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2682717984 >> v2:172.21.15.170:3300/0 conn(0x7f6eb00b6cd0 0x7f6eb00b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2682717984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00cf710 0x7f6eb00b2d10 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 -- 172.21.15.29:0/2682717984 >> 172.21.15.29:0/2682717984 conn(0x7f6eb00d2cc0 msgr2=0x7f6eb00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 -- 172.21.15.29:0/2682717984 shutdown_connections 2023-10-16T15:50:55.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.882+0000 7f6eb7475640 1 -- 172.21.15.29:0/2682717984 wait complete. 2023-10-16T15:50:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.884+0000 7f6eb7475640 1 Processor -- start 2023-10-16T15:50:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.884+0000 7f6eb7475640 1 -- start start 2023-10-16T15:50:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb7475640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 0x7f6eb0154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb7475640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb00cf710 0x7f6eb009d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb7475640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb00cfaf0 0x7f6eb009d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb7475640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00caaf0 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb7475640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00ca7a0 con 0x7f6eb00cf710 2023-10-16T15:50:55.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb7475640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb00a0580 con 0x7f6eb00cfaf0 2023-10-16T15:50:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb5211640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 0x7f6eb0154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb5211640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 0x7f6eb0154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39286/0 (socket says 172.21.15.29:39286) 2023-10-16T15:50:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb5211640 1 -- 172.21.15.29:0/2844664573 learned_addr learned my addr 172.21.15.29:0/2844664573 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb4a10640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb00cf710 0x7f6eb009d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:55.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb5211640 1 -- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb00cfaf0 msgr2=0x7f6eb009d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb5211640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb00cfaf0 0x7f6eb009d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.885+0000 7f6eb5211640 1 -- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb00cf710 msgr2=0x7f6eb009d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6eb5211640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb00cf710 0x7f6eb009d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6eb5211640 1 -- 172.21.15.29:0/2844664573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac009a70 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6eb4a10640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb00cf710 0x7f6eb009d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:50:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6eb5211640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 0x7f6eb0154040 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ed0 tx=0x7f6eac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac040650 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac0407b0 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb00a0860 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac040b60 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.886+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb00d17d0 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.888+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb008cd30 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.888+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6eac040cc0 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.889+0000 7f6ea5ffb640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 0x7f6e9406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.890+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6eac0ac6a0 con 0x7f6eb00b6cd0 2023-10-16T15:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.890+0000 7f6eb4a10640 1 -- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 msgr2=0x7f6e9406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.890+0000 7f6eb4a10640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 0x7f6e9406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:55.893+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6eac077610 con 0x7f6eb00b6cd0 2023-10-16T15:50:56.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.085+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f6eb004e160 con 0x7f6eb00b6cd0 2023-10-16T15:50:56.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.090+0000 7f6ea5ffb640 1 -- 172.21.15.29:0/2844664573 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6eac07ac30 con 0x7f6eb00b6cd0 2023-10-16T15:50:56.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.091+0000 7f6eb4a10640 1 -- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 msgr2=0x7f6e9406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:56.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.091+0000 7f6eb4a10640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 0x7f6e9406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:56.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 msgr2=0x7f6e9406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:56.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 0x7f6e9406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 msgr2=0x7f6eb0154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:56.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 0x7f6eb0154040 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ed0 tx=0x7f6eac004a40 comp rx=0 tx=0).stop 2023-10-16T15:50:56.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 shutdown_connections 2023-10-16T15:50:56.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f6e9406afe0 0x7f6e9406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6eb00cfaf0 0x7f6eb009d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6eb00cf710 0x7f6eb009d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 --2- 172.21.15.29:0/2844664573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eb00b6cd0 0x7f6eb0154040 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 >> 172.21.15.29:0/2844664573 conn(0x7f6eb00d2cc0 msgr2=0x7f6eb00b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 shutdown_connections 2023-10-16T15:50:56.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.094+0000 7f6eb7475640 1 -- 172.21.15.29:0/2844664573 wait complete. 2023-10-16T15:50:56.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:50:56.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T15:50:56.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T15:50:56.220 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.219+0000 7fa23ac3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:50:56.272 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.271+0000 7fa23ac3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:50:56.311 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.310+0000 7f761320d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:50:56.322 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.320+0000 7fa23ac3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:50:56.367 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.366+0000 7f761320d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:50:56.382 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.380+0000 7fa23ac3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:50:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.397+0000 7f43ba559640 1 Processor -- start 2023-10-16T15:50:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.397+0000 7f43ba559640 1 -- start start 2023-10-16T15:50:56.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.398+0000 7f43ba559640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f43b415cd70 0x7f43b40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.398+0000 7f43ba559640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43b40b8480 0x7f43b40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.398+0000 7f43ba559640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b40d0be0 0x7f43b40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.398+0000 7f43ba559640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b415d150 con 0x7f43b40b8480 2023-10-16T15:50:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.398+0000 7f43ba559640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40d1180 con 0x7f43b415cd70 2023-10-16T15:50:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.398+0000 7f43ba559640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40d1460 con 0x7f43b40d0be0 2023-10-16T15:50:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43b40b8480 0x7f43b40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43b40b8480 0x7f43b40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39294/0 (socket says 172.21.15.29:39294) 2023-10-16T15:50:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 -- 172.21.15.29:0/1129747583 learned_addr learned my addr 172.21.15.29:0/1129747583 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b9557640 1 --2- 172.21.15.29:0/1129747583 >> v2:172.21.15.170:3300/0 conn(0x7f43b415cd70 0x7f43b40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b9d58640 1 --2- 172.21.15.29:0/1129747583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b40d0be0 0x7f43b40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 -- 172.21.15.29:0/1129747583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b40d0be0 msgr2=0x7f43b40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 --2- 172.21.15.29:0/1129747583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b40d0be0 0x7f43b40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 -- 172.21.15.29:0/1129747583 >> v2:172.21.15.170:3300/0 conn(0x7f43b415cd70 msgr2=0x7f43b40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:50:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 --2- 172.21.15.29:0/1129747583 >> v2:172.21.15.170:3300/0 conn(0x7f43b415cd70 0x7f43b40d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.399+0000 7f43b8d56640 1 -- 172.21.15.29:0/1129747583 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40d1740 con 0x7f43b40b8480 2023-10-16T15:50:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.400+0000 7f43b8d56640 1 --2- 172.21.15.29:0/1129747583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40b8480 0x7f43b40cd7c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f43b40b56a0 tx=0x7f43a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe818c6bdc8d728 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.401+0000 7f43abfff640 1 -- 172.21.15.29:0/1129747583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4031870 con 0x7f43b40b8480 2023-10-16T15:50:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.401+0000 7f43abfff640 1 -- 172.21.15.29:0/1129747583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40319d0 con 0x7f43b40b8480 2023-10-16T15:50:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.401+0000 7f43abfff640 1 -- 172.21.15.29:0/1129747583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4031d60 con 0x7f43b40b8480 2023-10-16T15:50:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.401+0000 7f43ba559640 1 -- 172.21.15.29:0/1129747583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40b8480 msgr2=0x7f43b40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.401+0000 7f43ba559640 1 --2- 172.21.15.29:0/1129747583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40b8480 0x7f43b40cd7c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f43b40b56a0 tx=0x7f43a402b910 comp rx=0 tx=0).stop 2023-10-16T15:50:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 -- 172.21.15.29:0/1129747583 shutdown_connections 2023-10-16T15:50:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 --2- 172.21.15.29:0/1129747583 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b40d0be0 0x7f43b40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 --2- 172.21.15.29:0/1129747583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40b8480 0x7f43b40cd7c0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 --2- 172.21.15.29:0/1129747583 >> v2:172.21.15.170:3300/0 conn(0x7f43b415cd70 0x7f43b40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 -- 172.21.15.29:0/1129747583 >> 172.21.15.29:0/1129747583 conn(0x7f43b40cf570 msgr2=0x7f43b40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:50:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 -- 172.21.15.29:0/1129747583 shutdown_connections 2023-10-16T15:50:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.402+0000 7f43ba559640 1 -- 172.21.15.29:0/1129747583 wait complete. 2023-10-16T15:50:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.403+0000 7f43ba559640 1 Processor -- start 2023-10-16T15:50:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.403+0000 7f43ba559640 1 -- start start 2023-10-16T15:50:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.404+0000 7f43ba559640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43b40b8480 0x7f43b4159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.404+0000 7f43ba559640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40d0be0 0x7f43b40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.404+0000 7f43ba559640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b415cd70 0x7f43b40b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.404+0000 7f43ba559640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41f31d0 con 0x7f43b40d0be0 2023-10-16T15:50:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.404+0000 7f43ba559640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4158860 con 0x7f43b40b8480 2023-10-16T15:50:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.404+0000 7f43ba559640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4158b40 con 0x7f43b415cd70 2023-10-16T15:50:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40d0be0 0x7f43b40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40d0be0 0x7f43b40b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39302/0 (socket says 172.21.15.29:39302) 2023-10-16T15:50:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 -- 172.21.15.29:0/2496530894 learned_addr learned my addr 172.21.15.29:0/2496530894 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:50:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b9d58640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b415cd70 0x7f43b40b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43b40b8480 0x7f43b4159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:50:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b415cd70 msgr2=0x7f43b40b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43b415cd70 0x7f43b40b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43b40b8480 msgr2=0x7f43b4159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:50:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f43b40b8480 0x7f43b4159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.405+0000 7f43b8d56640 1 -- 172.21.15.29:0/2496530894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a40094d0 con 0x7f43b40d0be0 2023-10-16T15:50:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.406+0000 7f43b8d56640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40d0be0 0x7f43b40b42d0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f43a402b950 tx=0x7f43a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:50:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.406+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a403ad30 con 0x7f43b40d0be0 2023-10-16T15:50:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.406+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40443f0 con 0x7f43b40d0be0 2023-10-16T15:50:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.406+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40447a0 con 0x7f43b40d0be0 2023-10-16T15:50:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.406+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b4158e20 con 0x7f43b40d0be0 2023-10-16T15:50:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.406+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b4159880 con 0x7f43b40d0be0 2023-10-16T15:50:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.407+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404e9b0 con 0x7f43b40d0be0 2023-10-16T15:50:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.408+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f43a404d450 con 0x7f43b40d0be0 2023-10-16T15:50:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.410+0000 7f43a9ffb640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 0x7f439806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:50:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.410+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f43a402d040 con 0x7f43b40d0be0 2023-10-16T15:50:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.410+0000 7f43b9557640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 msgr2=0x7f439806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.410+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 0x7f439806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:50:56.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.414+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f43a4076da0 con 0x7f43b40d0be0 2023-10-16T15:50:56.422 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.422+0000 7f761320d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:50:56.431 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.430+0000 7fa23ac3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:50:56.488 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.487+0000 7f761320d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:50:56.540 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.539+0000 7f761320d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:50:56.542 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.540+0000 7fa23ac3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:50:56.604 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.602+0000 7fa23ac3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:56.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.611+0000 7f43b9557640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 msgr2=0x7f439806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:56.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.611+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 0x7f439806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:50:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:56.625+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 --> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f43b40d1d00 con 0x7f439806afe0 2023-10-16T15:50:56.653 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.651+0000 7fa23ac3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:50:56.657 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.656+0000 7f761320d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:50:56.710 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.708+0000 7fa23ac3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:50:56.724 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.723+0000 7f761320d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:56.760 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:56.758+0000 7fa23ac3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:50:56.776 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.775+0000 7f761320d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:50:56.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.835+0000 7f761320d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:50:56.889 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:56.887+0000 7f761320d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:50:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:57.012+0000 7f43b9557640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 msgr2=0x7f439806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:57.012+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 0x7f439806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T15:50:57.139 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.137+0000 7fa23ac3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:50:57.296 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:57.295+0000 7f761320d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:50:57.299 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.298+0000 7fa23ac3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:50:57.408 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.406+0000 7fa23ac3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:50:57.457 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.456+0000 7fa23ac3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:50:57.465 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:57.464+0000 7f761320d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:50:57.565 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.563+0000 7fa23ac3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:57.580 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:57.579+0000 7f761320d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:50:57.633 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:57.632+0000 7f761320d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:50:57.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.669+0000 7fa23ac3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:50:57.747 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:57.746+0000 7f761320d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:57.813+0000 7f43b9557640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 msgr2=0x7f439806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/780156504 2023-10-16T15:50:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:57.813+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 0x7f439806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T15:50:57.826 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.824+0000 7fa23ac3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:57.860 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:57.859+0000 7f761320d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:50:57.874 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.872+0000 7fa23ac3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:57.994 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:57.992+0000 7fa23ac3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:50:58.027 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:58.026+0000 7f761320d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:50:58.080 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:58.079+0000 7f761320d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:50:58.206 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:58.205+0000 7f761320d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:50:58.830 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:58.828+0000 7fa23ac3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:50:59.103 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:59.101+0000 7f761320d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:50:59.123 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.121+0000 7fa23ac3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:50:59.180 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.179+0000 7fa23ac3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:50:59.293 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.291+0000 7fa23ac3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:50:59.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:59.315+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 118) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f43a4077170 con 0x7f43b40d0be0 2023-10-16T15:50:59.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:59.316+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 msgr2=0x7f439806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:50:59.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:50:59.316+0000 7f43a9ffb640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/780156504,v1:172.21.15.170:6833/780156504] conn(0x7f439806afe0 0x7f439806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:50:59.438 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:59.437+0000 7f761320d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:50:59.497 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1ba775640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.498 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1ba775640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.498 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1ba775640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.498 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1ba775640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.498 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1ba775640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.498 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1b772f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.499 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1b772f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.499 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1b772f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.499 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1b772f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.499 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:50:59.495+0000 7fa1b772f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:50:59.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:59.505+0000 7f761320d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:50:59.626 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:50:59.625+0000 7f761320d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:51:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.325+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 119) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f43a40aa3a0 con 0x7f43b40d0be0 2023-10-16T15:51:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.326+0000 7f43a9ffb640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f43980358e0 0x7f4398074f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:00.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.326+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 --> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4398035cc0 con 0x7f43980358e0 2023-10-16T15:51:00.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.337+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f43980358e0 0x7f4398074f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:00.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.344+0000 7f43b9557640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f43980358e0 0x7f4398074f00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f43b0007010 tx=0x7f43b0008040 comp rx=0 tx=0).ready entity=mgr.7610 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:00.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.511+0000 7f43a9ffb640 1 -- 172.21.15.29:0/2496530894 <== mgr.7610 v2:172.21.15.170:6832/928368069 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4398035cc0 con 0x7f43980358e0 2023-10-16T15:51:00.513 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-16T15:51:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.514+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f43980358e0 msgr2=0x7f4398074f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.514+0000 7f43ba559640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f43980358e0 0x7f4398074f00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f43b0007010 tx=0x7f43b0008040 comp rx=0 tx=0).stop 2023-10-16T15:51:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.514+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40d0be0 msgr2=0x7f43b40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.514+0000 7f43ba559640 1 --2- 172.21.15.29:0/2496530894 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b40d0be0 0x7f43b40b42d0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f43a402b950 tx=0x7f43a40046c0 comp rx=0 tx=0).stop 2023-10-16T15:51:00.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.515+0000 7f43b9d58640 1 -- 172.21.15.29:0/2496530894 reap_dead start 2023-10-16T15:51:00.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.515+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 shutdown_connections 2023-10-16T15:51:00.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.515+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 >> 172.21.15.29:0/2496530894 conn(0x7f43b40cf570 msgr2=0x7f43b415bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:51:00.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.515+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 shutdown_connections 2023-10-16T15:51:00.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.515+0000 7f43ba559640 1 -- 172.21.15.29:0/2496530894 wait complete. 2023-10-16T15:51:00.538 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T15:51:00.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T15:51:00.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.769+0000 7f78cd5b6640 1 Processor -- start 2023-10-16T15:51:00.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.769+0000 7f78cd5b6640 1 -- start start 2023-10-16T15:51:00.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78cd5b6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78c8159da0 0x7f78c80c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:00.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78cd5b6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c815dfb0 con 0x7f78c81599c0 2023-10-16T15:51:00.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78cd5b6640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c815e290 con 0x7f78c80cce20 2023-10-16T15:51:00.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78cd5b6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c815e570 con 0x7f78c8159da0 2023-10-16T15:51:00.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78be7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78c8159da0 0x7f78c80c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:00.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78c6ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f78c81599c0 0x7f78c815d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52816/0 (socket says 172.21.15.29:52816) 2023-10-16T15:51:00.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.770+0000 7f78c6ffd640 1 -- 172.21.15.29:0/3227277588 learned_addr learned my addr 172.21.15.29:0/3227277588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:51:00.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.771+0000 7f78be7fc640 1 -- 172.21.15.29:0/3227277588 >> v1:172.21.15.170:6789/0 conn(0x7f78c80cce20 legacy=0x7f78c806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:51:00.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.771+0000 7f78be7fc640 1 -- 172.21.15.29:0/3227277588 >> v1:172.21.15.29:6789/0 conn(0x7f78c81599c0 legacy=0x7f78c815d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:00.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.771+0000 7f78be7fc640 1 -- 172.21.15.29:0/3227277588 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c806fda0 con 0x7f78c8159da0 2023-10-16T15:51:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.772+0000 7f78be7fc640 1 --2- 172.21.15.29:0/3227277588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c80c9300 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f78b00097a0 tx=0x7f78b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1058f5423f383f25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.772+0000 7f78c67fc640 1 -- 172.21.15.29:0/3227277588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0031830 con 0x7f78c8159da0 2023-10-16T15:51:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78c67fc640 1 -- 172.21.15.29:0/3227277588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b0031990 con 0x7f78c8159da0 2023-10-16T15:51:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78c67fc640 1 -- 172.21.15.29:0/3227277588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0031d20 con 0x7f78c8159da0 2023-10-16T15:51:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3227277588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 msgr2=0x7f78c80c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3227277588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c80c9300 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f78b00097a0 tx=0x7f78b002b8d0 comp rx=0 tx=0).stop 2023-10-16T15:51:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3227277588 shutdown_connections 2023-10-16T15:51:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3227277588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c80c9300 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3227277588 >> 172.21.15.29:0/3227277588 conn(0x7f78c80cb360 msgr2=0x7f78c80cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:51:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3227277588 shutdown_connections 2023-10-16T15:51:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.773+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3227277588 wait complete. 2023-10-16T15:51:00.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.775+0000 7f78cd5b6640 1 Processor -- start 2023-10-16T15:51:00.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.775+0000 7f78cd5b6640 1 -- start start 2023-10-16T15:51:00.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.776+0000 7f78cd5b6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c80cce20 0x7f78c80ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:00.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.776+0000 7f78cd5b6640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c81599c0 0x7f78c8155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:00.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.776+0000 7f78cd5b6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c8147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:00.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.776+0000 7f78cd5b6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80ad680 con 0x7f78c8159da0 2023-10-16T15:51:00.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.776+0000 7f78cd5b6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80c6010 con 0x7f78c80cce20 2023-10-16T15:51:00.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.776+0000 7f78cd5b6640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80c62d0 con 0x7f78c81599c0 2023-10-16T15:51:00.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c8147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:00.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c8147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39326/0 (socket says 172.21.15.29:39326) 2023-10-16T15:51:00.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 -- 172.21.15.29:0/3063887367 learned_addr learned my addr 172.21.15.29:0/3063887367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:51:00.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c6ffd640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c80cce20 0x7f78c80ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:00.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78be7fc640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c81599c0 0x7f78c8155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:00.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 -- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c81599c0 msgr2=0x7f78c8155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:00.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c81599c0 0x7f78c8155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:00.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 -- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c80cce20 msgr2=0x7f78c80ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:00.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c80cce20 0x7f78c80ad140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:00.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78c77fe640 1 -- 172.21.15.29:0/3063887367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b0009400 con 0x7f78c8159da0 2023-10-16T15:51:00.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.777+0000 7f78be7fc640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c81599c0 0x7f78c8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:51:00.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.778+0000 7f78c77fe640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c8147800 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c410 tx=0x7f78c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:00.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.778+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0011790 con 0x7f78c8159da0 2023-10-16T15:51:00.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.778+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c00118f0 con 0x7f78c8159da0 2023-10-16T15:51:00.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.778+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c80c65b0 con 0x7f78c8159da0 2023-10-16T15:51:00.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.778+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0011ce0 con 0x7f78c8159da0 2023-10-16T15:51:00.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.778+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c80c6b70 con 0x7f78c8159da0 2023-10-16T15:51:00.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.779+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c81f3810 con 0x7f78c8159da0 2023-10-16T15:51:00.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.780+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 119) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f78c001c460 con 0x7f78c8159da0 2023-10-16T15:51:00.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.781+0000 7f78bffff640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f78a806ad10 0x7f78a806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:00.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.781+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f78c0089180 con 0x7f78c8159da0 2023-10-16T15:51:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.782+0000 7f78c6ffd640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f78a806ad10 0x7f78a806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.783+0000 7f78c6ffd640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f78a806ad10 0x7f78a806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78b8005c00 tx=0x7f78b8009040 comp rx=0 tx=0).ready entity=mgr.7610 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:00.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:00.786+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78c0054320 con 0x7f78c8159da0 2023-10-16T15:51:01.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.006+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 --> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f78c806fa50 con 0x7f78a806ad10 2023-10-16T15:51:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.154+0000 7f78bffff640 1 -- 172.21.15.29:0/3063887367 <== mgr.7610 v2:172.21.15.170:6832/928368069 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f78b800ed30 con 0x7f78a806ad10 2023-10-16T15:51:01.156 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path snap_test_dir1 2023-10-16T15:51:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.157+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f78a806ad10 msgr2=0x7f78a806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f78a806ad10 0x7f78a806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78b8005c00 tx=0x7f78b8009040 comp rx=0 tx=0).stop 2023-10-16T15:51:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 msgr2=0x7f78c8147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c8147800 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c410 tx=0x7f78c000c8d0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 shutdown_connections 2023-10-16T15:51:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f78a806ad10 0x7f78a806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c8159da0 0x7f78c8147800 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f78c81599c0 0x7f78c8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 --2- 172.21.15.29:0/3063887367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f78c80cce20 0x7f78c80ad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 >> 172.21.15.29:0/3063887367 conn(0x7f78c80cb360 msgr2=0x7f78c80cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:51:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 shutdown_connections 2023-10-16T15:51:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.158+0000 7f78cd5b6640 1 -- 172.21.15.29:0/3063887367 wait complete. 2023-10-16T15:51:01.180 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T15:51:01.180 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-15_52_00 in ~59s... 2023-10-16T15:51:01.180 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T15:51:01.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T15:51:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.419+0000 7f3aa0ff5640 1 Processor -- start 2023-10-16T15:51:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.419+0000 7f3aa0ff5640 1 -- start start 2023-10-16T15:51:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.420+0000 7f3aa0ff5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 0x7f3a9c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.420+0000 7f3aa0ff5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3a9c06c140 0x7f3a9c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.420+0000 7f3aa0ff5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c070360 con 0x7f3a9c159ab0 2023-10-16T15:51:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.420+0000 7f3aa0ff5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a9c06cb10 con 0x7f3a9c06c140 2023-10-16T15:51:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.420+0000 7f3aa0ff5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c15d890 con 0x7f3a9c159e90 2023-10-16T15:51:01.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.421+0000 7f3a93fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 0x7f3a9c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:01.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.421+0000 7f3a93fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 0x7f3a9c0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:44946/0 (socket says 172.21.15.29:44946) 2023-10-16T15:51:01.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.421+0000 7f3a9a59c640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f3a9c159ab0 0x7f3a9c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:39222/0 (socket says 172.21.15.29:39222) 2023-10-16T15:51:01.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.421+0000 7f3a93fff640 1 -- 172.21.15.29:0/4159388857 learned_addr learned my addr 172.21.15.29:0/4159388857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:51:01.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.421+0000 7f3a9ad9d640 1 --2- 172.21.15.29:0/4159388857 >> v2:172.21.15.170:3300/0 conn(0x7f3a9c06c140 0x7f3a9c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:01.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.422+0000 7f3a93fff640 1 -- 172.21.15.29:0/4159388857 >> v2:172.21.15.170:3300/0 conn(0x7f3a9c06c140 msgr2=0x7f3a9c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:51:01.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.422+0000 7f3a93fff640 1 --2- 172.21.15.29:0/4159388857 >> v2:172.21.15.170:3300/0 conn(0x7f3a9c06c140 0x7f3a9c06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.422+0000 7f3a93fff640 1 -- 172.21.15.29:0/4159388857 >> v1:172.21.15.170:6789/0 conn(0x7f3a9c159ab0 legacy=0x7f3a9c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.422+0000 7f3a93fff640 1 -- 172.21.15.29:0/4159388857 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c0cd1f0 con 0x7f3a9c159e90 2023-10-16T15:51:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.423+0000 7f3a93fff640 1 --2- 172.21.15.29:0/4159388857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 0x7f3a9c0ce370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3a840098a0 tx=0x7f3a8402da70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9115c388135dcc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.424+0000 7f3a99d9b640 1 -- 172.21.15.29:0/4159388857 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8402f830 con 0x7f3a9c159e90 2023-10-16T15:51:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.424+0000 7f3a99d9b640 1 -- 172.21.15.29:0/4159388857 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8402f990 con 0x7f3a9c159e90 2023-10-16T15:51:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.424+0000 7f3a99d9b640 1 -- 172.21.15.29:0/4159388857 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8402fd20 con 0x7f3a9c159e90 2023-10-16T15:51:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/4159388857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 msgr2=0x7f3a9c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/4159388857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 0x7f3a9c0ce370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3a840098a0 tx=0x7f3a8402da70 comp rx=0 tx=0).stop 2023-10-16T15:51:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/4159388857 shutdown_connections 2023-10-16T15:51:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/4159388857 >> v2:172.21.15.170:3300/0 conn(0x7f3a9c06c140 0x7f3a9c06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/4159388857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c159e90 0x7f3a9c0ce370 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/4159388857 >> 172.21.15.29:0/4159388857 conn(0x7f3a9c0ce9b0 msgr2=0x7f3a9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:51:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/4159388857 shutdown_connections 2023-10-16T15:51:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.425+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/4159388857 wait complete. 2023-10-16T15:51:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.427+0000 7f3aa0ff5640 1 Processor -- start 2023-10-16T15:51:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.428+0000 7f3aa0ff5640 1 -- start start 2023-10-16T15:51:01.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3aa0ff5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c06c140 0x7f3a9c15ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3aa0ff5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a9c159ab0 0x7f3a9c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3aa0ff5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 0x7f3a9c15c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3aa0ff5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1576a0 con 0x7f3a9c159e90 2023-10-16T15:51:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3aa0ff5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c15c910 con 0x7f3a9c159ab0 2023-10-16T15:51:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3aa0ff5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c0c2be0 con 0x7f3a9c06c140 2023-10-16T15:51:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.429+0000 7f3a9ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 0x7f3a9c15c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a93fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a9c159ab0 0x7f3a9c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 0x7f3a9c15c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39334/0 (socket says 172.21.15.29:39334) 2023-10-16T15:51:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9a59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c06c140 0x7f3a9c15ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 -- 172.21.15.29:0/201511613 learned_addr learned my addr 172.21.15.29:0/201511613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:51:01.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 -- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c06c140 msgr2=0x7f3a9c15ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c06c140 0x7f3a9c15ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 -- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a9c159ab0 msgr2=0x7f3a9c157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a9c159ab0 0x7f3a9c157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a9ad9d640 1 -- 172.21.15.29:0/201511613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a840094d0 con 0x7f3a9c159e90 2023-10-16T15:51:01.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.430+0000 7f3a93fff640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a9c159ab0 0x7f3a9c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:51:01.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.431+0000 7f3a9ad9d640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 0x7f3a9c15c3d0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400efc0 tx=0x7f3a9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:01.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.431+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a94009ba0 con 0x7f3a9c159e90 2023-10-16T15:51:01.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.431+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9400f040 con 0x7f3a9c159e90 2023-10-16T15:51:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.431+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a940146c0 con 0x7f3a9c159e90 2023-10-16T15:51:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.431+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c15b420 con 0x7f3a9c159e90 2023-10-16T15:51:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.431+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9c15b9e0 con 0x7f3a9c159e90 2023-10-16T15:51:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.432+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9c0c2da0 con 0x7f3a9c159e90 2023-10-16T15:51:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.433+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 120) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3a94014820 con 0x7f3a9c159e90 2023-10-16T15:51:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.434+0000 7f3a92ffd640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f3a7c06afe0 0x7f3a7c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:51:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.435+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3a9408a790 con 0x7f3a9c159e90 2023-10-16T15:51:01.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.435+0000 7f3a9a59c640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f3a7c06afe0 0x7f3a7c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:51:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.436+0000 7f3a9a59c640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f3a7c06afe0 0x7f3a7c06d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c000f60 tx=0x7f3a8c000ed0 comp rx=0 tx=0).ready entity=mgr.7610 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:51:01.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.438+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a94055540 con 0x7f3a9c159e90 2023-10-16T15:51:01.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.653+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 --> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a9c0cd7b0 con 0x7f3a7c06afe0 2023-10-16T15:51:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.668+0000 7f3a92ffd640 1 -- 172.21.15.29:0/201511613 <== mgr.7610 v2:172.21.15.170:6832/928368069 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f3a8c004c70 con 0x7f3a7c06afe0 2023-10-16T15:51:01.669 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:51:01.670 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T15:51:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.670+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f3a7c06afe0 msgr2=0x7f3a7c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.670+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f3a7c06afe0 0x7f3a7c06d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c000f60 tx=0x7f3a8c000ed0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.670+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 msgr2=0x7f3a9c15c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:51:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.670+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 0x7f3a9c15c3d0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400efc0 tx=0x7f3a9400c490 comp rx=0 tx=0).stop 2023-10-16T15:51:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 shutdown_connections 2023-10-16T15:51:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f3a7c06afe0 0x7f3a7c06d4a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a9c159e90 0x7f3a9c15c3d0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a9c159ab0 0x7f3a9c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 --2- 172.21.15.29:0/201511613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a9c06c140 0x7f3a9c15ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:51:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 >> 172.21.15.29:0/201511613 conn(0x7f3a9c0ce9b0 msgr2=0x7f3a9c158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:51:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 shutdown_connections 2023-10-16T15:51:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:51:01.671+0000 7f3aa0ff5640 1 -- 172.21.15.29:0/201511613 wait complete. 2023-10-16T15:51:01.690 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T15:51:01.691 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T15:51:01.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:01.692 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:01.751 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:01.752 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:01.752 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:02.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:02.754 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:02.822 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:02.823 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:02.824 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:03.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:03.826 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:03.895 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:03.896 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:03.896 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:04.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:04.909 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:04.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:04.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:04.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:05.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:05.981 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:06.047 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:06.047 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:06.047 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:07.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:07.049 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:07.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:07.116 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:07.117 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:08.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:08.119 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:08.184 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:08.184 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:08.185 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:09.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:09.187 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:09.251 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:09.251 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:09.251 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:10.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:10.253 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:10.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:10.322 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:10.322 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:11.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:11.324 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:11.393 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:11.393 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:11.394 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:12.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:12.396 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:12.464 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:12.464 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:12.465 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:13.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:13.467 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:13.534 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:13.534 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:13.534 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:14.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:14.536 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:14.603 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:14.603 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:14.604 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:15.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:15.605 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:15.671 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:15.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:15.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:16.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:16.674 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:16.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:16.741 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:16.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:17.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:51:17.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:51:17.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:17.743 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:17.812 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:17.812 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:17.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:18.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:18.815 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:18.882 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:18.882 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:18.882 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:19.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:19.884 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:19.952 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:19.952 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:19.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:20.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:20.954 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:21.021 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:21.022 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:21.022 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:22.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:22.024 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:22.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:22.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:22.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:23.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:23.096 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:23.163 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:23.163 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:23.164 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:24.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:24.165 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:24.233 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:24.233 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:24.234 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:25.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:25.235 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:25.303 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:25.304 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:25.304 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:26.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:26.306 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:26.373 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:26.373 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:26.374 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:27.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:27.376 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:27.442 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:27.442 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:27.443 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:28.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:28.446 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:28.512 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:28.513 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:28.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:29.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:29.515 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:29.582 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:29.582 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:29.583 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:30.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:30.584 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:30.650 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:30.651 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:30.651 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:31.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:31.653 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:31.718 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:31.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:31.719 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:32.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:32.720 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:32.786 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:32.787 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:32.787 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:33.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:33.789 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:33.856 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:33.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:33.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:34.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:34.858 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:34.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:34.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:34.927 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:35.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:35.929 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:35.996 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:35.996 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:35.996 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:36.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:36.998 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:37.061 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:37.061 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:37.062 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:38.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:38.064 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:38.132 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:38.132 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:38.133 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:39.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:39.135 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:39.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:39.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:39.203 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:40.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:40.205 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:40.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:40.275 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:40.275 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:41.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:41.277 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:41.347 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:41.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:41.347 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:42.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:42.349 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:42.417 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:42.417 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:42.417 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:43.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:43.419 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:43.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:43.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:43.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:44.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:44.489 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:44.556 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:44.556 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:44.556 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:45.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:45.558 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:45.627 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:45.627 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:45.627 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:46.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:46.629 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:46.696 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:46.696 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:46.696 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:47.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:51:47.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:51:47.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:47.698 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:47.764 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:47.764 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:47.764 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:48.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:48.766 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:48.834 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:48.835 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:48.835 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:49.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:49.836 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:49.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:49.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:49.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:50.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:50.905 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:50.973 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:50.974 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:50.974 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:51.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:51.976 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:52.045 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:52.045 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:52.045 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:53.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:53.048 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:53.115 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:53.116 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:53.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:54.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:54.118 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:54.184 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:54.185 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:54.185 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:55.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:55.187 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:55.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:55.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:55.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:56.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:56.257 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:56.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:56.326 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:56.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:57.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:57.328 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:57.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:57.396 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:57.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:58.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:58.398 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:58.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:58.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:58.467 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:51:59.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:51:59.468 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:51:59.531 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T15:51:59.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:51:59.532 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:00.180 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:52:00.178+0000 7fa1a9713640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:52:00.180 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:52:00.178+0000 7fa1a9713640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:52:00.181 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:52:00.178+0000 7fa1a9713640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:52:00.181 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:52:00.178+0000 7fa1a9713640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:52:00.181 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:52:00.178+0000 7fa1a9713640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:52:00.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:00.534 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:00.600 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:00.601 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:00.601 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:00.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:00.602 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-15_52_00_UTC'] 2023-10-16T15:52:00.602 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697471461.1804695, now = 1697471520.6027749, timo = 59 2023-10-16T15:52:01.603 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-16-15_52_00 in ~119s... 2023-10-16T15:52:01.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:01.604 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:01.670 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:01.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:01.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:01.673 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:02.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:02.675 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:02.740 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:02.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:02.742 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:02.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:03.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:03.744 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:03.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:03.812 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:03.812 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:03.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:04.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:04.815 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:04.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:04.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:04.884 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:04.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:05.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:05.886 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:05.951 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:05.951 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:05.951 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:06.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:06.954 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:07.019 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:07.020 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:07.021 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:07.021 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:08.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:08.023 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:08.088 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:08.090 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:08.090 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:08.091 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:09.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:09.092 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:09.157 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:09.159 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:09.159 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:09.160 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:10.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:10.162 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:10.230 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:10.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:10.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:10.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:11.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:11.233 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:11.299 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:11.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:11.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:11.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:12.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:12.303 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:12.369 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:12.370 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:12.371 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:12.371 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:13.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:13.373 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:13.439 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:13.440 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:13.440 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:13.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:14.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:14.443 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:14.509 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:14.510 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:14.513 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:14.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:15.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:15.516 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:15.580 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:15.582 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:15.582 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:15.582 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:16.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:16.584 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:16.650 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:16.652 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:16.652 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:16.652 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:17.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:52:17.449 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:52:17.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:17.654 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:17.721 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:17.723 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:17.723 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:17.724 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:18.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:18.725 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:18.793 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:18.794 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:18.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:18.795 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:19.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:19.797 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:19.863 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:19.864 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:19.864 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:19.865 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:20.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:20.866 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:20.930 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:20.932 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:20.932 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:20.932 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:21.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:21.934 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:21.997 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:21.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:21.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:22.000 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:23.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:23.001 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:23.067 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:23.068 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:23.069 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:23.069 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:24.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:24.071 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:24.137 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:24.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:24.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:24.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:25.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:25.141 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:25.208 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:25.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:25.210 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:25.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:26.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:26.212 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:26.279 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:26.280 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:26.281 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:26.281 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:27.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:27.283 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:27.349 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:27.351 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:27.351 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:27.352 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:28.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:28.353 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:28.419 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:28.421 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:28.421 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:28.421 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:29.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:29.423 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:29.487 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:29.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:29.489 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:29.489 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:30.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:30.491 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:30.557 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:30.558 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:30.558 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:30.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:31.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:31.560 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:31.624 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:31.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:31.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:31.627 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:32.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:32.628 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:32.693 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:32.694 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:32.695 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:32.695 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:33.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:33.697 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:33.762 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:33.764 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:33.764 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:33.765 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:34.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:34.766 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:34.833 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:34.834 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:34.835 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:34.835 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:35.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:35.837 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:35.904 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:35.905 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:35.905 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:36.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:36.907 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:36.973 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:36.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:36.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:36.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:37.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:37.978 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:38.049 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:38.050 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:38.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:38.051 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:39.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:39.052 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:39.118 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:39.119 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:39.120 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:39.120 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:40.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:40.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:40.187 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:40.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:40.189 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:40.189 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:41.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:41.191 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:41.257 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:41.258 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:41.258 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:41.259 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:42.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:42.260 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:42.329 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:42.330 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:42.330 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:42.331 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:43.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:43.332 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:43.398 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:43.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:43.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:43.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:44.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:44.402 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:44.467 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:44.468 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:44.469 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:44.469 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:45.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:45.471 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:45.537 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:45.538 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:45.539 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:45.539 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:46.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:46.541 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:46.605 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:46.607 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:46.607 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:46.607 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:47.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:47.609 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:47.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:52:47.626 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:52:47.670 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:47.671 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:47.671 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:47.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:48.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:48.675 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:48.740 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:48.742 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:48.742 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:48.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:49.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:49.744 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:49.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:49.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:49.812 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:49.812 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:50.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:50.814 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:50.880 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:50.882 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:50.882 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:50.882 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:51.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:51.884 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:51.952 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:51.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:51.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:51.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:52.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:52.955 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:53.021 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:53.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:53.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:53.023 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:54.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:54.026 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:54.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:54.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:54.094 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:55.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:55.096 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:55.162 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:55.164 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:55.164 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:55.164 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:56.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:56.166 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:56.231 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:56.232 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:56.232 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:56.233 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:57.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:57.234 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:57.298 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:57.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:57.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:57.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:58.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:58.303 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:58.369 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:58.370 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:58.371 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:58.371 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:52:59.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:52:59.373 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:52:59.436 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:52:59.437 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:52:59.438 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:52:59.438 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:53:00.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:00.440 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:53:00.506 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_52_00_UTC 2023-10-16T15:53:00.506 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_53_00_UTC 2023-10-16T15:53:00.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_53_00_UTC', 'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:53:00.507 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-15_53_00_UTC'} 2023-10-16T15:53:00.508 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T15:53:01.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:01.510 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:53:01.576 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_53_00_UTC 2023-10-16T15:53:01.578 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-15_53_00_UTC'} 2023-10-16T15:53:01.578 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T15:53:01.579 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-16-15_52_00_UTC'} 2023-10-16T15:53:01.579 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-16-15_52_00_UTC'] 2023-10-16T15:53:01.579 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697471461.1804695, now = 1697471581.579799, timo = 119 2023-10-16T15:53:02.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T15:53:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.822+0000 7fa8e1ce5640 1 Processor -- start 2023-10-16T15:53:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.822+0000 7fa8e1ce5640 1 -- start start 2023-10-16T15:53:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.823+0000 7fa8e1ce5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8dc15af80 0x7fa8dc0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.823+0000 7fa8e1ce5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc15f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.823+0000 7fa8e1ce5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa8dc065ab0 0x7fa8dc065f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.823+0000 7fa8e1ce5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8dc0d0930 con 0x7fa8dc15af80 2023-10-16T15:53:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.823+0000 7fa8e1ce5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8dc066450 con 0x7fa8dc065ab0 2023-10-16T15:53:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.823+0000 7fa8e1ce5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc15eff0 con 0x7fa8dc06faf0 2023-10-16T15:53:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8dc15af80 0x7fa8dc0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa8dc15af80 0x7fa8dc0cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34992/0 (socket says 172.21.15.29:34992) 2023-10-16T15:53:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 -- 172.21.15.29:0/732348630 learned_addr learned my addr 172.21.15.29:0/732348630 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8daffd640 1 --2- 172.21.15.29:0/732348630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc15f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8dbfff640 1 --2- 172.21.15.29:0/732348630 >> v2:172.21.15.170:3300/0 conn(0x7fa8dc065ab0 0x7fa8dc065f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 -- 172.21.15.29:0/732348630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 msgr2=0x7fa8dc15f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 --2- 172.21.15.29:0/732348630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc15f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 -- 172.21.15.29:0/732348630 >> v2:172.21.15.170:3300/0 conn(0x7fa8dc065ab0 msgr2=0x7fa8dc065f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 --2- 172.21.15.29:0/732348630 >> v2:172.21.15.170:3300/0 conn(0x7fa8dc065ab0 0x7fa8dc065f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8db7fe640 1 -- 172.21.15.29:0/732348630 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc15ecc0 con 0x7fa8dc15af80 2023-10-16T15:53:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.824+0000 7fa8dbfff640 1 --2- 172.21.15.29:0/732348630 >> v2:172.21.15.170:3300/0 conn(0x7fa8dc065ab0 0x7fa8dc065f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.825+0000 7fa8db7fe640 1 --2- 172.21.15.29:0/732348630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc15af80 0x7fa8dc0cb1c0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0737d0 tx=0x7fa8cc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b557b87a4ecab01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:02.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.826+0000 7fa8da7fc640 1 -- 172.21.15.29:0/732348630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc02ca60 con 0x7fa8dc15af80 2023-10-16T15:53:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.826+0000 7fa8da7fc640 1 -- 172.21.15.29:0/732348630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc02cbc0 con 0x7fa8dc15af80 2023-10-16T15:53:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.826+0000 7fa8da7fc640 1 -- 172.21.15.29:0/732348630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0343f0 con 0x7fa8dc15af80 2023-10-16T15:53:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.826+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/732348630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc15af80 msgr2=0x7fa8dc0cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.826+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/732348630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc15af80 0x7fa8dc0cb1c0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0737d0 tx=0x7fa8cc02bec0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/732348630 shutdown_connections 2023-10-16T15:53:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/732348630 >> v2:172.21.15.170:3300/0 conn(0x7fa8dc065ab0 0x7fa8dc065f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/732348630 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc15f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/732348630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc15af80 0x7fa8dc0cb1c0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/732348630 >> 172.21.15.29:0/732348630 conn(0x7fa8dc0d0ab0 msgr2=0x7fa8dc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:02.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/732348630 shutdown_connections 2023-10-16T15:53:02.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.827+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/732348630 wait complete. 2023-10-16T15:53:02.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.828+0000 7fa8e1ce5640 1 Processor -- start 2023-10-16T15:53:02.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.829+0000 7fa8e1ce5640 1 -- start start 2023-10-16T15:53:02.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.829+0000 7fa8e1ce5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 0x7fa8dc142e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.829+0000 7fa8e1ce5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.829+0000 7fa8e1ce5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc15af80 0x7fa8dc158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.829+0000 7fa8e1ce5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc143390 con 0x7fa8dc065ab0 2023-10-16T15:53:02.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8e1ce5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0af720 con 0x7fa8dc15af80 2023-10-16T15:53:02.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8e1ce5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b5e50 con 0x7fa8dc06faf0 2023-10-16T15:53:02.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 0x7fa8dc142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 0x7fa8dc142e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35004/0 (socket says 172.21.15.29:35004) 2023-10-16T15:53:02.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 -- 172.21.15.29:0/2042873294 learned_addr learned my addr 172.21.15.29:0/2042873294 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:02.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8dbfff640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc15af80 0x7fa8dc158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8daffd640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 -- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 msgr2=0x7fa8dc0b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:02.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc0b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 -- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc15af80 msgr2=0x7fa8dc158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:02.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc15af80 0x7fa8dc158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:02.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8db7fe640 1 -- 172.21.15.29:0/2042873294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc009b30 con 0x7fa8dc065ab0 2023-10-16T15:53:02.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.830+0000 7fa8dbfff640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc15af80 0x7fa8dc158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:02.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.831+0000 7fa8db7fe640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 0x7fa8dc142e50 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc000f80 tx=0x7fa8cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:02.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.831+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc03e450 con 0x7fa8dc065ab0 2023-10-16T15:53:02.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.831+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc03e5b0 con 0x7fa8dc065ab0 2023-10-16T15:53:02.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.831+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc03e960 con 0x7fa8dc065ab0 2023-10-16T15:53:02.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.831+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8dc0b6130 con 0x7fa8dc065ab0 2023-10-16T15:53:02.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.831+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8dc0b66f0 con 0x7fa8dc065ab0 2023-10-16T15:53:02.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.832+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8dc04e160 con 0x7fa8dc065ab0 2023-10-16T15:53:02.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.833+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 123) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fa8cc03eac0 con 0x7fa8dc065ab0 2023-10-16T15:53:02.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.835+0000 7fa8c3fff640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa8c806b010 0x7fa8c806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:02.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.835+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7fa8cc0ac760 con 0x7fa8dc065ab0 2023-10-16T15:53:02.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.835+0000 7fa8daffd640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa8c806b010 0x7fa8c806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:02.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.837+0000 7fa8daffd640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa8c806b010 0x7fa8c806d4d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc159ed0 tx=0x7fa8d0008040 comp rx=0 tx=0).ready entity=mgr.7610 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:02.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:02.840+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8cc0774b0 con 0x7fa8dc065ab0 2023-10-16T15:53:03.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.055+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 --> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8dc0b6880 con 0x7fa8c806b010 2023-10-16T15:53:03.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.229+0000 7fa8c3fff640 1 -- 172.21.15.29:0/2042873294 <== mgr.7610 v2:172.21.15.170:6832/928368069 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa8dc0b6880 con 0x7fa8c806b010 2023-10-16T15:53:03.230 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-16T15:53:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.232+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa8c806b010 msgr2=0x7fa8c806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.232+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa8c806b010 0x7fa8c806d4d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc159ed0 tx=0x7fa8d0008040 comp rx=0 tx=0).stop 2023-10-16T15:53:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 msgr2=0x7fa8dc142e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 0x7fa8dc142e50 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc000f80 tx=0x7fa8cc004a40 comp rx=0 tx=0).stop 2023-10-16T15:53:03.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 shutdown_connections 2023-10-16T15:53:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa8c806b010 0x7fa8c806d4d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8dc15af80 0x7fa8dc158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8dc06faf0 0x7fa8dc0b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 --2- 172.21.15.29:0/2042873294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8dc065ab0 0x7fa8dc142e50 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 >> 172.21.15.29:0/2042873294 conn(0x7fa8dc0d0ab0 msgr2=0x7fa8dc0c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:03.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 shutdown_connections 2023-10-16T15:53:03.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.233+0000 7fa8e1ce5640 1 -- 172.21.15.29:0/2042873294 wait complete. 2023-10-16T15:53:03.253 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T15:53:03.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:03.254 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:53:03.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-16-15_53_00_UTC 2023-10-16T15:53:03.311 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-15_53_00_UTC 2023-10-16T15:53:03.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:03.312 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-15_53_00_UTC) 2023-10-16T15:53:03.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:03.376 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T15:53:03.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:53:03.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.669+0000 7f325ffff640 1 Processor -- start 2023-10-16T15:53:03.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.669+0000 7f325ffff640 1 -- start start 2023-10-16T15:53:03.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.671+0000 7f325ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f326006c040 0x7f326006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:03.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.671+0000 7f325ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f32600cda80 0x7f326015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:03.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.671+0000 7f325ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326006c960 con 0x7f32600ad970 2023-10-16T15:53:03.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.671+0000 7f325ffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326015b9d0 con 0x7f326006c040 2023-10-16T15:53:03.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.671+0000 7f325ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326015bcb0 con 0x7f32600cda80 2023-10-16T15:53:03.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.671+0000 7f325effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f326006c040 0x7f326006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:03.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f326006c040 0x7f326006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35018/0 (socket says 172.21.15.29:35018) 2023-10-16T15:53:03.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325effd640 1 -- 172.21.15.29:0/2288532445 learned_addr learned my addr 172.21.15.29:0/2288532445 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:03.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325f7fe640 1 --2- 172.21.15.29:0/2288532445 >> v2:172.21.15.170:3300/0 conn(0x7f32600cda80 0x7f326015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:03.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325effd640 1 -- 172.21.15.29:0/2288532445 >> v2:172.21.15.170:3300/0 conn(0x7f32600cda80 msgr2=0x7f326015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:03.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325effd640 1 --2- 172.21.15.29:0/2288532445 >> v2:172.21.15.170:3300/0 conn(0x7f32600cda80 0x7f326015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325effd640 1 -- 172.21.15.29:0/2288532445 >> v1:172.21.15.170:6789/0 conn(0x7f32600ad970 legacy=0x7f326015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:03.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.672+0000 7f325effd640 1 -- 172.21.15.29:0/2288532445 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326015bf90 con 0x7f326006c040 2023-10-16T15:53:03.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.674+0000 7f325effd640 1 --2- 172.21.15.29:0/2288532445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f326006c420 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f32600cf300 tx=0x7f325002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04dbb4afd52ead6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:03.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.674+0000 7f325e7fc640 1 -- 172.21.15.29:0/2288532445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325002cb20 con 0x7f326006c040 2023-10-16T15:53:03.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.674+0000 7f325e7fc640 1 -- 172.21.15.29:0/2288532445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325002cc80 con 0x7f326006c040 2023-10-16T15:53:03.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.674+0000 7f325e7fc640 1 -- 172.21.15.29:0/2288532445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500343f0 con 0x7f326006c040 2023-10-16T15:53:03.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 -- 172.21.15.29:0/2288532445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 msgr2=0x7f326006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:03.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 --2- 172.21.15.29:0/2288532445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f326006c420 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f32600cf300 tx=0x7f325002bf80 comp rx=0 tx=0).stop 2023-10-16T15:53:03.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 -- 172.21.15.29:0/2288532445 shutdown_connections 2023-10-16T15:53:03.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 --2- 172.21.15.29:0/2288532445 >> v2:172.21.15.170:3300/0 conn(0x7f32600cda80 0x7f326015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 --2- 172.21.15.29:0/2288532445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f326006c420 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 -- 172.21.15.29:0/2288532445 >> 172.21.15.29:0/2288532445 conn(0x7f32600ce6a0 msgr2=0x7f32600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 -- 172.21.15.29:0/2288532445 shutdown_connections 2023-10-16T15:53:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.675+0000 7f325ffff640 1 -- 172.21.15.29:0/2288532445 wait complete. 2023-10-16T15:53:03.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.677+0000 7f325ffff640 1 Processor -- start 2023-10-16T15:53:03.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.677+0000 7f325ffff640 1 -- start start 2023-10-16T15:53:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f3260149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32600ad970 0x7f32600afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32600cda80 0x7f32600ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:03.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600aebc0 con 0x7f326006c040 2023-10-16T15:53:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600aeea0 con 0x7f32600cda80 2023-10-16T15:53:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600af180 con 0x7f32600ad970 2023-10-16T15:53:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f3260149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f3260149a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35020/0 (socket says 172.21.15.29:35020) 2023-10-16T15:53:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.678+0000 7f325effd640 1 -- 172.21.15.29:0/314291998 learned_addr learned my addr 172.21.15.29:0/314291998 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:03.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325effd640 1 -- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32600ad970 msgr2=0x7f32600afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:03.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f32567fc640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32600ad970 0x7f32600afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:03.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325effd640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32600ad970 0x7f32600afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325effd640 1 -- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32600cda80 msgr2=0x7f32600ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:03.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325effd640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32600cda80 0x7f32600ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:03.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325effd640 1 -- 172.21.15.29:0/314291998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250009b30 con 0x7f326006c040 2023-10-16T15:53:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325effd640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f3260149a60 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f3250000f80 tx=0x7f3250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.679+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250009510 con 0x7f326006c040 2023-10-16T15:53:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.680+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3250007e60 con 0x7f326006c040 2023-10-16T15:53:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.680+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325003e690 con 0x7f326006c040 2023-10-16T15:53:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.680+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32600af460 con 0x7f326006c040 2023-10-16T15:53:03.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.680+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3260149fa0 con 0x7f326006c040 2023-10-16T15:53:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.681+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326006b330 con 0x7f326006c040 2023-10-16T15:53:03.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.682+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 123) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f325003e7f0 con 0x7f326006c040 2023-10-16T15:53:03.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.683+0000 7f325cff9640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f324006afe0 0x7f324006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:03.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.683+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7f32500ac650 con 0x7f326006c040 2023-10-16T15:53:03.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.684+0000 7f32567fc640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f324006afe0 0x7f324006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.685+0000 7f32567fc640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f324006afe0 0x7f324006d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f326015ae80 tx=0x7f3248009040 comp rx=0 tx=0).ready entity=mgr.7610 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.687+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3250077480 con 0x7f326006c040 2023-10-16T15:53:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:03.930+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f326004e9b0 con 0x7f326006c040 2023-10-16T15:53:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.292+0000 7f325cff9640 1 -- 172.21.15.29:0/314291998 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v124) v1 ==== 93+0+0 (secure 0 0 0) 0x7f325007aaa0 con 0x7f326006c040 2023-10-16T15:53:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f324006afe0 msgr2=0x7f324006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f324006afe0 0x7f324006d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f326015ae80 tx=0x7f3248009040 comp rx=0 tx=0).stop 2023-10-16T15:53:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 msgr2=0x7f3260149a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f3260149a60 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f3250000f80 tx=0x7f3250004a40 comp rx=0 tx=0).stop 2023-10-16T15:53:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 shutdown_connections 2023-10-16T15:53:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f324006afe0 0x7f324006d4a0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f326015ae80 tx=0x7f3248009040 comp rx=0 tx=0).stop 2023-10-16T15:53:04.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f32600cda80 0x7f32600ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32600ad970 0x7f32600afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 --2- 172.21.15.29:0/314291998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f326006c040 0x7f3260149a60 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.296+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 >> 172.21.15.29:0/314291998 conn(0x7f32600ce6a0 msgr2=0x7f32600c89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.297+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 shutdown_connections 2023-10-16T15:53:04.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.297+0000 7f325ffff640 1 -- 172.21.15.29:0/314291998 wait complete. 2023-10-16T15:53:04.326 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:53:04.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:53:04.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:53:04.534 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:53:04.535 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:53:04.536 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:04.590 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:53:04.591 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:04.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:04.823 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:04.821+0000 7f6b00a14040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:53:04.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:04.864+0000 7f4d18e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:53:04.876 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:04.874+0000 7f6b00a14040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:53:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.897+0000 7f879993c640 1 Processor -- start 2023-10-16T15:53:04.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.897+0000 7f879993c640 1 -- start start 2023-10-16T15:53:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.898+0000 7f879993c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87940cdf30 0x7f8794071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.898+0000 7f879993c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8794071fa0 0x7f879406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.898+0000 7f879993c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87940d12f0 con 0x7f879415cc80 2023-10-16T15:53:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.898+0000 7f879993c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87940d0500 con 0x7f8794071fa0 2023-10-16T15:53:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.898+0000 7f879993c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87940723f0 con 0x7f87940cdf30 2023-10-16T15:53:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879893a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f879415cc80 0x7f87940cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41054/0 (socket says 172.21.15.29:41054) 2023-10-16T15:53:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879893a640 1 -- 172.21.15.29:0/412211067 learned_addr learned my addr 172.21.15.29:0/412211067 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879913b640 1 --2- 172.21.15.29:0/412211067 >> v2:172.21.15.29:3300/0 conn(0x7f8794071fa0 0x7f879406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f8793fff640 1 --2- 172.21.15.29:0/412211067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87940cdf30 0x7f8794071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f87937fe640 1 -- 172.21.15.29:0/412211067 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824454160 0 0) 0x7f87940d12f0 con 0x7f879415cc80 2023-10-16T15:53:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879913b640 1 -- 172.21.15.29:0/412211067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87940cdf30 msgr2=0x7f8794071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879913b640 1 --2- 172.21.15.29:0/412211067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87940cdf30 0x7f8794071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879913b640 1 -- 172.21.15.29:0/412211067 >> v1:172.21.15.29:6789/0 conn(0x7f879415cc80 legacy=0x7f87940cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.899+0000 7f879913b640 1 -- 172.21.15.29:0/412211067 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87940d06e0 con 0x7f8794071fa0 2023-10-16T15:53:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.900+0000 7f879913b640 1 --2- 172.21.15.29:0/412211067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8794071fa0 0x7f879406aeb0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f878c009d90 tx=0x7f878c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b931eafa14b645 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f87937fe640 1 -- 172.21.15.29:0/412211067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c02cc50 con 0x7f8794071fa0 2023-10-16T15:53:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f87937fe640 1 -- 172.21.15.29:0/412211067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c02cdb0 con 0x7f8794071fa0 2023-10-16T15:53:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f87937fe640 1 -- 172.21.15.29:0/412211067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c034660 con 0x7f8794071fa0 2023-10-16T15:53:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f879993c640 1 -- 172.21.15.29:0/412211067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8794071fa0 msgr2=0x7f879406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f879993c640 1 --2- 172.21.15.29:0/412211067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8794071fa0 0x7f879406aeb0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f878c009d90 tx=0x7f878c02bea0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f879993c640 1 -- 172.21.15.29:0/412211067 shutdown_connections 2023-10-16T15:53:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f879993c640 1 --2- 172.21.15.29:0/412211067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8794071fa0 0x7f879406aeb0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f879993c640 1 --2- 172.21.15.29:0/412211067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87940cdf30 0x7f8794071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.901+0000 7f879993c640 1 -- 172.21.15.29:0/412211067 >> 172.21.15.29:0/412211067 conn(0x7f8794065ed0 msgr2=0x7f87940662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:04.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.902+0000 7f879993c640 1 -- 172.21.15.29:0/412211067 shutdown_connections 2023-10-16T15:53:04.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.902+0000 7f879993c640 1 -- 172.21.15.29:0/412211067 wait complete. 2023-10-16T15:53:04.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.903+0000 7f879993c640 1 Processor -- start 2023-10-16T15:53:04.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.903+0000 7f879993c640 1 -- start start 2023-10-16T15:53:04.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.904+0000 7f879993c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8794071fa0 0x7f87940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:04.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.904+0000 7f879993c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 0x7f879415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:04.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.904+0000 7f879993c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f879415cc80 0x7f87940c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:04.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.904+0000 7f879993c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940c6e10 con 0x7f87940cdf30 2023-10-16T15:53:04.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.904+0000 7f879993c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940c6040 con 0x7f8794071fa0 2023-10-16T15:53:04.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.904+0000 7f879993c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87940c6320 con 0x7f879415cc80 2023-10-16T15:53:04.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 0x7f879415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:04.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 0x7f879415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33546/0 (socket says 172.21.15.29:33546) 2023-10-16T15:53:04.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 -- 172.21.15.29:0/1152855568 learned_addr learned my addr 172.21.15.29:0/1152855568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:04.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f879893a640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8794071fa0 0x7f87940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:04.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f879913b640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f879415cc80 0x7f87940c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 -- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f879415cc80 msgr2=0x7f87940c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f879415cc80 0x7f87940c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 -- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8794071fa0 msgr2=0x7f87940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8794071fa0 0x7f87940ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 -- 172.21.15.29:0/1152855568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c009a70 con 0x7f87940cdf30 2023-10-16T15:53:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f879893a640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8794071fa0 0x7f87940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.905+0000 7f8793fff640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 0x7f879415a7f0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f878000d800 tx=0x7f878000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.906+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878000b6c0 con 0x7f87940cdf30 2023-10-16T15:53:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.906+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878000b820 con 0x7f87940cdf30 2023-10-16T15:53:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.906+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878000bbb0 con 0x7f87940cdf30 2023-10-16T15:53:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.906+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87940c6600 con 0x7f87940cdf30 2023-10-16T15:53:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.906+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87940ae3d0 con 0x7f87940cdf30 2023-10-16T15:53:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.907+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87940ae560 con 0x7f87940cdf30 2023-10-16T15:53:04.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.908+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f878000bd10 con 0x7f87940cdf30 2023-10-16T15:53:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.909+0000 7f87917fa640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 0x7f877806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.909+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7f8780089f80 con 0x7f87940cdf30 2023-10-16T15:53:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.909+0000 7f879893a640 1 -- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 msgr2=0x7f877806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.909+0000 7f879893a640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 0x7f877806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:04.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:04.917+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8780054cd0 con 0x7f87940cdf30 2023-10-16T15:53:04.926 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:04.925+0000 7f4d18e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:53:05.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.110+0000 7f879893a640 1 -- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 msgr2=0x7f877806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:05.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.110+0000 7f879893a640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 0x7f877806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:05.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.111+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f879404e9b0 con 0x7f87940cdf30 2023-10-16T15:53:05.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.114+0000 7f87917fa640 1 -- 172.21.15.29:0/1152855568 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17170 (secure 0 0 0) 0x7f87800582f0 con 0x7f87940cdf30 2023-10-16T15:53:05.115 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:05.116 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":212,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:52:00.390006+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.24","mappings":[{"from":0,"to":1}]},{"pgid":"25.2a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/423324657":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/3913323747":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2674320589":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/275392394":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6833/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6832/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2170359614":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/1066951432":"2023-10-17T15:50:59.297259+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:05.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 msgr2=0x7f877806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:05.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 0x7f877806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 msgr2=0x7f879415a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 0x7f879415a7f0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f878000d800 tx=0x7f878000dcc0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 shutdown_connections 2023-10-16T15:53:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f877806af90 0x7f877806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87940cdf30 0x7f879415a7f0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f879415cc80 0x7f87940c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 --2- 172.21.15.29:0/1152855568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8794071fa0 0x7f87940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 >> 172.21.15.29:0/1152855568 conn(0x7f8794065ed0 msgr2=0x7f879406b740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:05.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 shutdown_connections 2023-10-16T15:53:05.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.117+0000 7f879993c640 1 -- 172.21.15.29:0/1152855568 wait complete. 2023-10-16T15:53:05.137 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:50:42.031547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:50:43.111119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:53:05.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:53:05.322 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.319+0000 7f6b00a14040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:53:05.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.352+0000 7fa392c61640 1 Processor -- start 2023-10-16T15:53:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.353+0000 7fa392c61640 1 -- start start 2023-10-16T15:53:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.354+0000 7fa392c61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.354+0000 7fa392c61640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa38c0cd260 0x7fa38c0cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.354+0000 7fa392c61640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c0b5d90 con 0x7fa38c15d030 2023-10-16T15:53:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.354+0000 7fa392c61640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c0d2b50 con 0x7fa38c0cd260 2023-10-16T15:53:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.354+0000 7fa392c61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c160fc0 con 0x7fa38c15d410 2023-10-16T15:53:05.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.355+0000 7fa3911fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa38c0cd260 0x7fa38c0cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.355+0000 7fa3911fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa38c0cd260 0x7fa38c0cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:51392/0 (socket says 172.21.15.29:51392) 2023-10-16T15:53:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.355+0000 7fa3911fe640 1 -- 172.21.15.29:0/1031596332 learned_addr learned my addr 172.21.15.29:0/1031596332 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.355+0000 7fa38bfff640 1 --2- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.356+0000 7fa3911fe640 1 -- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 msgr2=0x7fa38c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.356+0000 7fa3911fe640 1 --2- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.356+0000 7fa3911fe640 1 -- 172.21.15.29:0/1031596332 >> v1:172.21.15.170:6789/0 conn(0x7fa38c15d030 legacy=0x7fa38c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.356+0000 7fa3911fe640 1 -- 172.21.15.29:0/1031596332 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0d25d0 con 0x7fa38c0cd260 2023-10-16T15:53:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.357+0000 7fa3911fe640 1 --2- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c0cd6f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa384009e50 tx=0x7fa38402bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cfec84802248545 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.357+0000 7fa38b7fe640 1 -- 172.21.15.29:0/1031596332 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38402cb90 con 0x7fa38c0cd260 2023-10-16T15:53:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.357+0000 7fa38b7fe640 1 -- 172.21.15.29:0/1031596332 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38402ccf0 con 0x7fa38c0cd260 2023-10-16T15:53:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.357+0000 7fa38b7fe640 1 -- 172.21.15.29:0/1031596332 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384034610 con 0x7fa38c0cd260 2023-10-16T15:53:05.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.358+0000 7fa392c61640 1 -- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 msgr2=0x7fa38c0cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.358+0000 7fa392c61640 1 --2- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c0cd6f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa384009e50 tx=0x7fa38402bff0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.358+0000 7fa392c61640 1 -- 172.21.15.29:0/1031596332 shutdown_connections 2023-10-16T15:53:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.358+0000 7fa392c61640 1 --2- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c0cd6f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.358+0000 7fa392c61640 1 --2- 172.21.15.29:0/1031596332 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c0d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.358+0000 7fa392c61640 1 -- 172.21.15.29:0/1031596332 >> 172.21.15.29:0/1031596332 conn(0x7fa38c0cf720 msgr2=0x7fa38c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:05.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.359+0000 7fa392c61640 1 -- 172.21.15.29:0/1031596332 shutdown_connections 2023-10-16T15:53:05.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.359+0000 7fa392c61640 1 -- 172.21.15.29:0/1031596332 wait complete. 2023-10-16T15:53:05.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.361+0000 7fa392c61640 1 Processor -- start 2023-10-16T15:53:05.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.361+0000 7fa392c61640 1 -- start start 2023-10-16T15:53:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.362+0000 7fa392c61640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.362+0000 7fa392c61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 0x7fa38c15f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.362+0000 7fa392c61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.362+0000 7fa392c61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c142f50 con 0x7fa38c15d030 2023-10-16T15:53:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.362+0000 7fa392c61640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c15fd70 con 0x7fa38c0cd260 2023-10-16T15:53:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.362+0000 7fa392c61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c160050 con 0x7fa38c15d410 2023-10-16T15:53:05.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 0x7fa38c15f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 0x7fa38c15f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33556/0 (socket says 172.21.15.29:33556) 2023-10-16T15:53:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 -- 172.21.15.29:0/1179523647 learned_addr learned my addr 172.21.15.29:0/1179523647 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa3909fd640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa3911fe640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 -- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 msgr2=0x7fa38c144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 -- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 msgr2=0x7fa38c15a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c15a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 -- 172.21.15.29:0/1179523647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa384009b30 con 0x7fa38c15d030 2023-10-16T15:53:05.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa3911fe640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:05.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.363+0000 7fa38bfff640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 0x7fa38c15f540 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fa37800e9b0 tx=0x7fa37800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.364+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378009670 con 0x7fa38c15d030 2023-10-16T15:53:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.364+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37800c9a0 con 0x7fa38c15d030 2023-10-16T15:53:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.364+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c142be0 con 0x7fa38c15d030 2023-10-16T15:53:05.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.364+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37800cd50 con 0x7fa38c15d030 2023-10-16T15:53:05.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.364+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c145290 con 0x7fa38c15d030 2023-10-16T15:53:05.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.366+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c0d1ef0 con 0x7fa38c15d030 2023-10-16T15:53:05.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.366+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa378013470 con 0x7fa38c15d030 2023-10-16T15:53:05.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.367+0000 7fa3897fa640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 0x7fa37006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.367+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7fa378089af0 con 0x7fa38c15d030 2023-10-16T15:53:05.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.367+0000 7fa3909fd640 1 -- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 msgr2=0x7fa37006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:05.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.367+0000 7fa3909fd640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 0x7fa37006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:05.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.371+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa378054980 con 0x7fa38c15d030 2023-10-16T15:53:05.374 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.372+0000 7f6b00a14040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:53:05.405 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.404+0000 7f4d18e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:53:05.424 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.422+0000 7f6b00a14040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:53:05.460 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.459+0000 7f4d18e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:53:05.483 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.482+0000 7f6b00a14040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:53:05.514 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.513+0000 7f4d18e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:53:05.532 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.531+0000 7f6b00a14040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:53:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.568+0000 7fa3909fd640 1 -- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 msgr2=0x7fa37006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:05.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.568+0000 7fa3909fd640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 0x7fa37006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.568+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa38c04e160 con 0x7fa38c15d030 2023-10-16T15:53:05.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.569+0000 7fa3897fa640 1 -- 172.21.15.29:0/1179523647 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v212) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3780170a0 con 0x7fa38c15d030 2023-10-16T15:53:05.570 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:53:05.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 msgr2=0x7fa37006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 0x7fa37006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 msgr2=0x7fa38c15f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 0x7fa38c15f540 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fa37800e9b0 tx=0x7fa37800ee70 comp rx=0 tx=0).stop 2023-10-16T15:53:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 shutdown_connections 2023-10-16T15:53:05.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fa37006af90 0x7fa37006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa38c15d030 0x7fa38c15f540 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa38c15d410 0x7fa38c144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 --2- 172.21.15.29:0/1179523647 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa38c0cd260 0x7fa38c15a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.571+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 >> 172.21.15.29:0/1179523647 conn(0x7fa38c0cf720 msgr2=0x7fa38c0b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.572+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 shutdown_connections 2023-10-16T15:53:05.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.572+0000 7fa392c61640 1 -- 172.21.15.29:0/1179523647 wait complete. 2023-10-16T15:53:05.578 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.577+0000 7f4d18e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:53:05.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:53:05.631 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.630+0000 7f4d18e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:53:05.643 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.641+0000 7f6b00a14040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:53:05.705 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.703+0000 7f6b00a14040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:05.750 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.749+0000 7f4d18e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:53:05.754 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.752+0000 7f6b00a14040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:53:05.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.799+0000 7f7c8b59e640 1 Processor -- start 2023-10-16T15:53:05.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.799+0000 7f7c8b59e640 1 -- start start 2023-10-16T15:53:05.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.801+0000 7f7c8b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7c8c15cd70 0x7f7c8c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.801+0000 7f7c8b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8c15d150 con 0x7f7c8c0cded0 2023-10-16T15:53:05.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.801+0000 7f7c8b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8c0d1180 con 0x7f7c8c0b8480 2023-10-16T15:53:05.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.801+0000 7f7c8b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8c0d1460 con 0x7f7c8c15cd70 2023-10-16T15:53:05.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.801+0000 7f7c8ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7c8c0cded0 0x7f7c8c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41064/0 (socket says 172.21.15.29:41064) 2023-10-16T15:53:05.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.801+0000 7f7c8ad9d640 1 -- 172.21.15.29:0/1579352834 learned_addr learned my addr 172.21.15.29:0/1579352834 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c8a59c640 1 --2- 172.21.15.29:0/1579352834 >> v2:172.21.15.170:3300/0 conn(0x7f7c8c15cd70 0x7f7c8c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395101416 0 0) 0x7f7c8c15d150 con 0x7f7c8c0cded0 2023-10-16T15:53:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c78003520 con 0x7f7c8c0cded0 2023-10-16T15:53:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245057268 0 0) 0x7f7c78003520 con 0x7f7c8c0cded0 2023-10-16T15:53:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 >> v2:172.21.15.170:3300/0 conn(0x7f7c8c15cd70 msgr2=0x7f7c8c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 --2- 172.21.15.29:0/1579352834 >> v2:172.21.15.170:3300/0 conn(0x7f7c8c15cd70 0x7f7c8c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 >> v1:172.21.15.170:6789/0 conn(0x7f7c8c0b8480 legacy=0x7f7c8c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c0d1740 con 0x7f7c8c0cded0 2023-10-16T15:53:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7c840019f0 con 0x7f7c8c0cded0 2023-10-16T15:53:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 >> v1:172.21.15.29:6789/0 conn(0x7f7c8c0cded0 legacy=0x7f7c8c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c780039c0 0x7f7c78003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c780047c0 0x7f7c78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c78005180 0x7f7c7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0d1460 con 0x7f7c78005180 2023-10-16T15:53:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7800c080 con 0x7f7c780047c0 2023-10-16T15:53:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.802+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7800c240 con 0x7f7c780039c0 2023-10-16T15:53:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c8ad9d640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c78005180 0x7f7c7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c83fff640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c780047c0 0x7f7c78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c8ad9d640 1 -- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c780039c0 msgr2=0x7f7c78003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c8ad9d640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c780039c0 0x7f7c78003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c8ad9d640 1 -- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c780047c0 msgr2=0x7f7c78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c8ad9d640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c780047c0 0x7f7c78004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.803+0000 7f7c8ad9d640 1 -- 172.21.15.29:0/1579352834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c78003520 con 0x7f7c78005180 2023-10-16T15:53:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.804+0000 7f7c8ad9d640 1 -- 172.21.15.29:0/1579352834 reap_dead start 2023-10-16T15:53:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.804+0000 7f7c8ad9d640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c78005180 0x7f7c7800b980 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f7c840013d0 tx=0x7f7c780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e2493bb4e5f824 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.804+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84034630 con 0x7f7c78005180 2023-10-16T15:53:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.804+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8402f070 con 0x7f7c78005180 2023-10-16T15:53:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.805+0000 7f7c89d9b640 1 -- 172.21.15.29:0/1579352834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84034a10 con 0x7f7c78005180 2023-10-16T15:53:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.805+0000 7f7c8b59e640 1 -- 172.21.15.29:0/1579352834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c78005180 msgr2=0x7f7c7800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.805+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c78005180 0x7f7c7800b980 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f7c840013d0 tx=0x7f7c780057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.805+0000 7f7c8b59e640 1 -- 172.21.15.29:0/1579352834 shutdown_connections 2023-10-16T15:53:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.805+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/1579352834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c78005180 0x7f7c7800b980 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.805+0000 7f7c8b59e640 1 -- 172.21.15.29:0/1579352834 >> 172.21.15.29:0/1579352834 conn(0x7f7c8c0cf570 msgr2=0x7f7c8c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.806+0000 7f7c8b59e640 1 -- 172.21.15.29:0/1579352834 shutdown_connections 2023-10-16T15:53:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.806+0000 7f7c8b59e640 1 -- 172.21.15.29:0/1579352834 wait complete. 2023-10-16T15:53:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.808+0000 7f7c8b59e640 1 Processor -- start 2023-10-16T15:53:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.809+0000 7f7c8b59e640 1 -- start start 2023-10-16T15:53:05.813 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.810+0000 7f6b00a14040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:53:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c8b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c78005180 0x7f7c8c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c8b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 0x7f7c8c0b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c8b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c8c0c8940 0x7f7c8c0b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c8b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0cbb20 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c8b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c1f22d0 con 0x7f7c8c0c8940 2023-10-16T15:53:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c8b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c1f25b0 con 0x7f7c78005180 2023-10-16T15:53:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 0x7f7c8c0b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 0x7f7c8c0b2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33568/0 (socket says 172.21.15.29:33568) 2023-10-16T15:53:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.810+0000 7f7c83fff640 1 -- 172.21.15.29:0/3184714363 learned_addr learned my addr 172.21.15.29:0/3184714363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c8a59c640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c78005180 0x7f7c8c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c8ad9d640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c8c0c8940 0x7f7c8c0b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c83fff640 1 -- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c78005180 msgr2=0x7f7c8c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c83fff640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c78005180 0x7f7c8c0cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c83fff640 1 -- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c8c0c8940 msgr2=0x7f7c8c0b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c83fff640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c8c0c8940 0x7f7c8c0b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c83fff640 1 -- 172.21.15.29:0/3184714363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c1f2890 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c8ad9d640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c8c0c8940 0x7f7c8c0b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c83fff640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 0x7f7c8c0b2510 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f7c74001eb0 tx=0x7f7c7400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74005480 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c74003610 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74003980 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.811+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8c1f2b70 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.812+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c1f1f50 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.813+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c04e9b0 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.814+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7c74003ae0 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.814+0000 7f7c82ffd640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6c06af90 0x7f7c6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:05.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.814+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7f7c74089850 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.815+0000 7f7c8a59c640 1 -- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6c06af90 msgr2=0x7f7c6c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:05.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.815+0000 7f7c8a59c640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6c06af90 0x7f7c6c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:05.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:05.819+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7c740545a0 con 0x7f7c8c0c84c0 2023-10-16T15:53:05.822 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.821+0000 7f4d18e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:05.861 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:05.860+0000 7f6b00a14040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:53:05.874 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.873+0000 7f4d18e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:53:05.935 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.934+0000 7f4d18e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:53:05.988 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:05.987+0000 7f4d18e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:53:06.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.011+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c8c0cd810 con 0x7f7c8c0c84c0 2023-10-16T15:53:06.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.012+0000 7f7c82ffd640 1 -- 172.21.15.29:0/3184714363 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v212) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c74057bc0 con 0x7f7c8c0c84c0 2023-10-16T15:53:06.013 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:53:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6c06af90 msgr2=0x7f7c6c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6c06af90 0x7f7c6c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 msgr2=0x7f7c8c0b2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 0x7f7c8c0b2510 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f7c74001eb0 tx=0x7f7c7400d9d0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 shutdown_connections 2023-10-16T15:53:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6c06af90 0x7f7c6c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c0c84c0 0x7f7c8c0b2510 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c8c0c8940 0x7f7c8c0b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 --2- 172.21.15.29:0/3184714363 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c78005180 0x7f7c8c0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 >> 172.21.15.29:0/3184714363 conn(0x7f7c8c0cf570 msgr2=0x7f7c8c0b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:06.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 shutdown_connections 2023-10-16T15:53:06.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.015+0000 7f7c8b59e640 1 -- 172.21.15.29:0/3184714363 wait complete. 2023-10-16T15:53:06.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:53:06.242 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.240+0000 7f6b00a14040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:53:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.293+0000 7fb22304b640 1 Processor -- start 2023-10-16T15:53:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.293+0000 7fb22304b640 1 -- start start 2023-10-16T15:53:06.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb22304b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb21c0cf710 0x7fb21c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb22304b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb21c0cfaf0 0x7fb21c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb22304b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0b38e0 0x7fb21c0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb22304b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c0d3c60 con 0x7fb21c0cfaf0 2023-10-16T15:53:06.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb22304b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c0d3980 con 0x7fb21c0cf710 2023-10-16T15:53:06.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb22304b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0ce630 con 0x7fb21c0b38e0 2023-10-16T15:53:06.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb21c0cfaf0 0x7fb21c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb21c0cfaf0 0x7fb21c0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33578/0 (socket says 172.21.15.29:33578) 2023-10-16T15:53:06.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 -- 172.21.15.29:0/2087354611 learned_addr learned my addr 172.21.15.29:0/2087354611 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:06.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb220de7640 1 --2- 172.21.15.29:0/2087354611 >> v2:172.21.15.170:3300/0 conn(0x7fb21c0cf710 0x7fb21c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb2215e8640 1 --2- 172.21.15.29:0/2087354611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0b38e0 0x7fb21c0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 -- 172.21.15.29:0/2087354611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0b38e0 msgr2=0x7fb21c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 --2- 172.21.15.29:0/2087354611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0b38e0 0x7fb21c0d0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 -- 172.21.15.29:0/2087354611 >> v2:172.21.15.170:3300/0 conn(0x7fb21c0cf710 msgr2=0x7fb21c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:06.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.295+0000 7fb21bfff640 1 --2- 172.21.15.29:0/2087354611 >> v2:172.21.15.170:3300/0 conn(0x7fb21c0cf710 0x7fb21c0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.296+0000 7fb21bfff640 1 -- 172.21.15.29:0/2087354611 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0ce910 con 0x7fb21c0cfaf0 2023-10-16T15:53:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.297+0000 7fb21bfff640 1 --2- 172.21.15.29:0/2087354611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cfaf0 0x7fb21c0b3250 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0d2c20 tx=0x7fb20802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47a4151fac74224 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.297+0000 7fb21b7fe640 1 -- 172.21.15.29:0/2087354611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080317e0 con 0x7fb21c0cfaf0 2023-10-16T15:53:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.297+0000 7fb21b7fe640 1 -- 172.21.15.29:0/2087354611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208031940 con 0x7fb21c0cfaf0 2023-10-16T15:53:06.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.297+0000 7fb21b7fe640 1 -- 172.21.15.29:0/2087354611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208031cd0 con 0x7fb21c0cfaf0 2023-10-16T15:53:06.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 -- 172.21.15.29:0/2087354611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cfaf0 msgr2=0x7fb21c0b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 --2- 172.21.15.29:0/2087354611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cfaf0 0x7fb21c0b3250 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0d2c20 tx=0x7fb20802b880 comp rx=0 tx=0).stop 2023-10-16T15:53:06.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 -- 172.21.15.29:0/2087354611 shutdown_connections 2023-10-16T15:53:06.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 --2- 172.21.15.29:0/2087354611 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0b38e0 0x7fb21c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 --2- 172.21.15.29:0/2087354611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cfaf0 0x7fb21c0b3250 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 --2- 172.21.15.29:0/2087354611 >> v2:172.21.15.170:3300/0 conn(0x7fb21c0cf710 0x7fb21c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 -- 172.21.15.29:0/2087354611 >> 172.21.15.29:0/2087354611 conn(0x7fb21c0d2cc0 msgr2=0x7fb21c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 -- 172.21.15.29:0/2087354611 shutdown_connections 2023-10-16T15:53:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.298+0000 7fb22304b640 1 -- 172.21.15.29:0/2087354611 wait complete. 2023-10-16T15:53:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.300+0000 7fb22304b640 1 Processor -- start 2023-10-16T15:53:06.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.300+0000 7fb22304b640 1 -- start start 2023-10-16T15:53:06.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb22304b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0b38e0 0x7fb21c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb22304b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 0x7fb21c156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb22304b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0cfaf0 0x7fb21c0d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb22304b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c1f6c60 con 0x7fb21c0cf710 2023-10-16T15:53:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb22304b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0cc7c0 con 0x7fb21c0b38e0 2023-10-16T15:53:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb22304b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0cbed0 con 0x7fb21c0cfaf0 2023-10-16T15:53:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 0x7fb21c156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 0x7fb21c156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33592/0 (socket says 172.21.15.29:33592) 2023-10-16T15:53:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.301+0000 7fb21bfff640 1 -- 172.21.15.29:0/194789649 learned_addr learned my addr 172.21.15.29:0/194789649 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb220de7640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0b38e0 0x7fb21c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb2215e8640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0cfaf0 0x7fb21c0d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb21bfff640 1 -- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0cfaf0 msgr2=0x7fb21c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb21bfff640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0cfaf0 0x7fb21c0d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb21bfff640 1 -- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0b38e0 msgr2=0x7fb21c0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb21bfff640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0b38e0 0x7fb21c0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb21bfff640 1 -- 172.21.15.29:0/194789649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080094d0 con 0x7fb21c0cf710 2023-10-16T15:53:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb2215e8640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0cfaf0 0x7fb21c0d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.302+0000 7fb21bfff640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 0x7fb21c156270 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fb20802b8c0 tx=0x7fb2080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.303+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208032980 con 0x7fb21c0cf710 2023-10-16T15:53:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.303+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208032b00 con 0x7fb21c0cf710 2023-10-16T15:53:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.303+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080453f0 con 0x7fb21c0cf710 2023-10-16T15:53:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.303+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c1f6980 con 0x7fb21c0cf710 2023-10-16T15:53:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.303+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c0cc450 con 0x7fb21c0cf710 2023-10-16T15:53:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.304+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c04e160 con 0x7fb21c0cf710 2023-10-16T15:53:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.305+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb20804d3f0 con 0x7fb21c0cf710 2023-10-16T15:53:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.306+0000 7fb2197fa640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 0x7fb20006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.306+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7fb2080abab0 con 0x7fb21c0cf710 2023-10-16T15:53:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.306+0000 7fb220de7640 1 -- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 msgr2=0x7fb20006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.306+0000 7fb220de7640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 0x7fb20006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.314+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb2080767a0 con 0x7fb21c0cf710 2023-10-16T15:53:06.404 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.402+0000 7f6b00a14040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:53:06.408 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:06.407+0000 7f4d18e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:53:06.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.506+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb21c0d2890 con 0x7fb21c0cf710 2023-10-16T15:53:06.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.507+0000 7fb220de7640 1 -- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 msgr2=0x7fb20006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:06.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.507+0000 7fb220de7640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 0x7fb20006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:06.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.507+0000 7fb2197fa640 1 -- 172.21.15.29:0/194789649 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v212) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb208079dc0 con 0x7fb21c0cf710 2023-10-16T15:53:06.509 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:53:06.510 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.508+0000 7f6b00a14040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:53:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.509+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 msgr2=0x7fb20006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.509+0000 7fb22304b640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 0x7fb20006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.509+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 msgr2=0x7fb21c156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.509+0000 7fb22304b640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 0x7fb21c156270 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fb20802b8c0 tx=0x7fb2080046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 shutdown_connections 2023-10-16T15:53:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7fb20006af90 0x7fb20006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb21c0cf710 0x7fb21c156270 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb21c0cfaf0 0x7fb21c0d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 --2- 172.21.15.29:0/194789649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb21c0b38e0 0x7fb21c0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 >> 172.21.15.29:0/194789649 conn(0x7fb21c0d2cc0 msgr2=0x7fb21c0b6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 shutdown_connections 2023-10-16T15:53:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.510+0000 7fb22304b640 1 -- 172.21.15.29:0/194789649 wait complete. 2023-10-16T15:53:06.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:53:06.559 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.557+0000 7f6b00a14040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:53:06.577 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:06.577+0000 7f4d18e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:53:06.666 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.665+0000 7f6b00a14040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:06.694 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:06.693+0000 7f4d18e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:53:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.739+0000 7f7c832da640 1 Processor -- start 2023-10-16T15:53:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.740+0000 7f7c832da640 1 -- start start 2023-10-16T15:53:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.741+0000 7f7c832da640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c7c160c80 0x7f7c7c161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.741+0000 7f7c832da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.741+0000 7f7c832da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0ae1a0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.741+0000 7f7c832da640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7c0ae480 con 0x7f7c7c160c80 2023-10-16T15:53:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.741+0000 7f7c832da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0ae760 con 0x7f7c7c070950 2023-10-16T15:53:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c7c160c80 0x7f7c7c161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7c7c160c80 0x7f7c7c161060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33604/0 (socket says 172.21.15.29:33604) 2023-10-16T15:53:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 -- 172.21.15.29:0/669621171 learned_addr learned my addr 172.21.15.29:0/669621171 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c82ad9640 1 --2- 172.21.15.29:0/669621171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 -- 172.21.15.29:0/669621171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 msgr2=0x7f7c7c140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 --2- 172.21.15.29:0/669621171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 -- 172.21.15.29:0/669621171 >> v1:172.21.15.29:6789/0 conn(0x7f7c7c0ad7a0 legacy=0x7f7c7c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c822d8640 1 -- 172.21.15.29:0/669621171 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0aea40 con 0x7f7c7c160c80 2023-10-16T15:53:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.742+0000 7f7c812d6640 1 -- 172.21.15.29:0/669621171 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272237909 0 0) 0x7f7c7c0ae1a0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.743+0000 7f7c822d8640 1 --2- 172.21.15.29:0/669621171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c160c80 0x7f7c7c161060 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009870 tx=0x7f7c7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9424ca6726379a46 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c812d6640 1 -- 172.21.15.29:0/669621171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78030840 con 0x7f7c7c160c80 2023-10-16T15:53:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c812d6640 1 -- 172.21.15.29:0/669621171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c780309a0 con 0x7f7c7c160c80 2023-10-16T15:53:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c812d6640 1 -- 172.21.15.29:0/669621171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78030d30 con 0x7f7c7c160c80 2023-10-16T15:53:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c832da640 1 -- 172.21.15.29:0/669621171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c160c80 msgr2=0x7f7c7c161060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c832da640 1 --2- 172.21.15.29:0/669621171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c160c80 0x7f7c7c161060 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009870 tx=0x7f7c7802b980 comp rx=0 tx=0).stop 2023-10-16T15:53:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c832da640 1 -- 172.21.15.29:0/669621171 shutdown_connections 2023-10-16T15:53:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c832da640 1 --2- 172.21.15.29:0/669621171 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.744+0000 7f7c832da640 1 --2- 172.21.15.29:0/669621171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c160c80 0x7f7c7c161060 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.745+0000 7f7c832da640 1 -- 172.21.15.29:0/669621171 >> 172.21.15.29:0/669621171 conn(0x7f7c7c0cc230 msgr2=0x7f7c7c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.745+0000 7f7c832da640 1 -- 172.21.15.29:0/669621171 shutdown_connections 2023-10-16T15:53:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.745+0000 7f7c832da640 1 -- 172.21.15.29:0/669621171 wait complete. 2023-10-16T15:53:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.746+0000 7f7c832da640 1 Processor -- start 2023-10-16T15:53:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.746+0000 7f7c832da640 1 -- start start 2023-10-16T15:53:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f7c832da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c15c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f7c832da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 0x7f7c7c06cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f7c832da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c160c80 0x7f7c7c15eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f7c832da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f5ec0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.752 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f4d18e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:53:06.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f7c832da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f61a0 con 0x7f7c7c160c80 2023-10-16T15:53:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.747+0000 7f7c832da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f6480 con 0x7f7c7c070950 2023-10-16T15:53:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 0x7f7c7c06cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 0x7f7c7c06cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33614/0 (socket says 172.21.15.29:33614) 2023-10-16T15:53:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 -- 172.21.15.29:0/2948136828 learned_addr learned my addr 172.21.15.29:0/2948136828 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c822d8640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c15c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c82ad9640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c160c80 0x7f7c7c15eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 -- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 msgr2=0x7f7c7c15c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c15c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 -- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c160c80 msgr2=0x7f7c7c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c160c80 0x7f7c7c15eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 -- 172.21.15.29:0/2948136828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780094d0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c822d8640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c82ad9640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c160c80 0x7f7c7c15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.748+0000 7f7c81ad7640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 0x7f7c7c06cb20 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00c4b0 tx=0x7f7c6c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.749+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c005480 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.749+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c0035f0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.749+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0039e0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.749+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c1f6760 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.749+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c15c730 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.750+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c0668c0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.752+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7c6c003b40 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.753+0000 7f7c72ffd640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 0x7f7c6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.753+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7f7c6c088ea0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.753+0000 7f7c822d8640 1 -- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 msgr2=0x7f7c6006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:06.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.753+0000 7f7c822d8640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 0x7f7c6006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:06.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.757+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7c6c053d20 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.774 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.773+0000 7f6b00a14040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:53:06.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:06.864+0000 7f4d18e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:06.929 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.927+0000 7f6b00a14040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:06.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.954+0000 7f7c822d8640 1 -- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 msgr2=0x7f7c6006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:06.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.954+0000 7f7c822d8640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 0x7f7c6006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:06.978 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:06.976+0000 7f6b00a14040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:06.979 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:06.978+0000 7f4d18e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:53:06.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.996+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7c7c04e9b0 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.997+0000 7f7c72ffd640 1 -- 172.21.15.29:0/2948136828 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v212) v1 ==== 80+0+474 (secure 0 0 0) 0x7f7c6c017070 con 0x7f7c7c0ad7a0 2023-10-16T15:53:06.998 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/423324657 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:06.998 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3913323747 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:06.999 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2674320589 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:06.999 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/275392394 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:06.999 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/780156504 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:06.999 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/780156504 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:06.999 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2170359614 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:07.000 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1066951432 2023-10-17T15:50:59.297259+0000 2023-10-16T15:53:07.002 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T15:53:07.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.999+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 msgr2=0x7f7c6006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.999+0000 7f7c832da640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 0x7f7c6006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.999+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 msgr2=0x7f7c7c06cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:06.999+0000 7f7c832da640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 0x7f7c7c06cb20 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00c4b0 tx=0x7f7c6c00c970 comp rx=0 tx=0).stop 2023-10-16T15:53:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 shutdown_connections 2023-10-16T15:53:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f7c6006af90 0x7f7c6006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c7c0ad7a0 0x7f7c7c06cb20 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7c7c160c80 0x7f7c7c15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 --2- 172.21.15.29:0/2948136828 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7c7c070950 0x7f7c7c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 >> 172.21.15.29:0/2948136828 conn(0x7f7c7c0cc230 msgr2=0x7f7c7c15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 shutdown_connections 2023-10-16T15:53:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.000+0000 7f7c832da640 1 -- 172.21.15.29:0/2948136828 wait complete. 2023-10-16T15:53:07.018 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:53:07.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:07.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:53:07.097 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:07.095+0000 7f6b00a14040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:53:07.126 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:53:07.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:07.127 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:53:07.150 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:07.150+0000 7f4d18e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:07.175 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:53:07.175 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:07.203 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:07.203+0000 7f4d18e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:07.231 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:53:07.231 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:53:07.232 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:07.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:53:07.288 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:53:07.288 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:07.333 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:07.331+0000 7f4d18e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:53:07.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:53:07.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.557+0000 7f2abe46f640 1 Processor -- start 2023-10-16T15:53:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.557+0000 7f2abe46f640 1 -- start start 2023-10-16T15:53:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2abe46f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2ab815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2abe46f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab815e3e0 con 0x7f2ab80cd0c0 2023-10-16T15:53:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2abe46f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80cd4a0 con 0x7f2ab80b0b20 2023-10-16T15:53:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2abe46f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab815e100 con 0x7f2ab81615f0 2023-10-16T15:53:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2ab7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2ab80cd0c0 0x7f2ab80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41080/0 (socket says 172.21.15.29:41080) 2023-10-16T15:53:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2ab7fff640 1 -- 172.21.15.29:0/2650198375 learned_addr learned my addr 172.21.15.29:0/2650198375 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2abca0c640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2ab815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.559+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952610404 0 0) 0x7f2ab815e3e0 con 0x7f2ab80cd0c0 2023-10-16T15:53:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa4003520 con 0x7f2ab80cd0c0 2023-10-16T15:53:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539828243 0 0) 0x7f2aa4003520 con 0x7f2ab80cd0c0 2023-10-16T15:53:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 msgr2=0x7f2ab815db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2ab815db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 >> v1:172.21.15.170:6789/0 conn(0x7f2ab80b0b20 legacy=0x7f2ab815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab8160b10 con 0x7f2ab80cd0c0 2023-10-16T15:53:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2aa8001670 con 0x7f2ab80cd0c0 2023-10-16T15:53:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 >> v1:172.21.15.29:6789/0 conn(0x7f2ab80cd0c0 legacy=0x7f2ab80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40039a0 0x7f2aa4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2aa40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa4004d00 0x7f2aa4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab815e100 con 0x7f2aa40039a0 2023-10-16T15:53:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa400bb10 con 0x7f2aa4004d00 2023-10-16T15:53:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa400bcd0 con 0x7f2ab81615f0 2023-10-16T15:53:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.560+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40039a0 0x7f2aa4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2abca0c640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa4004d00 0x7f2aa4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2ab7fff640 1 -- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 msgr2=0x7f2aa40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2aaffff640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2aa40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2aa40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2ab7fff640 1 -- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa4004d00 msgr2=0x7f2aa4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa4004d00 0x7f2aa4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2ab7fff640 1 -- 172.21.15.29:0/2650198375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa4003520 con 0x7f2aa40039a0 2023-10-16T15:53:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.561+0000 7f2aaffff640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab81615f0 0x7f2aa40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.562+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40039a0 0x7f2aa4003dc0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800c0e0 tx=0x7f2aa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252485e8ed1599c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.562+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8005ad0 con 0x7f2aa40039a0 2023-10-16T15:53:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.562+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa8005c30 con 0x7f2aa40039a0 2023-10-16T15:53:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.562+0000 7f2ab77fe640 1 -- 172.21.15.29:0/2650198375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa802c7f0 con 0x7f2aa40039a0 2023-10-16T15:53:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abe46f640 1 -- 172.21.15.29:0/2650198375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40039a0 msgr2=0x7f2aa4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2650198375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2aa40039a0 0x7f2aa4003dc0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800c0e0 tx=0x7f2aa40057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abca0c640 1 -- 172.21.15.29:0/2650198375 reap_dead start 2023-10-16T15:53:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abe46f640 1 -- 172.21.15.29:0/2650198375 shutdown_connections 2023-10-16T15:53:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abe46f640 1 -- 172.21.15.29:0/2650198375 >> 172.21.15.29:0/2650198375 conn(0x7f2ab80cb410 msgr2=0x7f2ab80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abe46f640 1 -- 172.21.15.29:0/2650198375 shutdown_connections 2023-10-16T15:53:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.563+0000 7f2abe46f640 1 -- 172.21.15.29:0/2650198375 wait complete. 2023-10-16T15:53:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.565+0000 7f2abe46f640 1 Processor -- start 2023-10-16T15:53:07.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.565+0000 7f2abe46f640 1 -- start start 2023-10-16T15:53:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abe46f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40039a0 0x7f2ab80c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abe46f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab80c5a60 0x7f2ab80ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abe46f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 0x7f2ab81f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abe46f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80c6ba0 con 0x7f2ab81f6030 2023-10-16T15:53:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abe46f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80c81a0 con 0x7f2aa40039a0 2023-10-16T15:53:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abe46f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80c6020 con 0x7f2ab80c5a60 2023-10-16T15:53:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abca0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 0x7f2ab81f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abca0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 0x7f2ab81f64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33640/0 (socket says 172.21.15.29:33640) 2023-10-16T15:53:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2abca0c640 1 -- 172.21.15.29:0/2972608619 learned_addr learned my addr 172.21.15.29:0/2972608619 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.566+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40039a0 0x7f2ab80c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abca0c640 1 -- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab80c5a60 msgr2=0x7f2ab80ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abca0c640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab80c5a60 0x7f2ab80ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abca0c640 1 -- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40039a0 msgr2=0x7f2ab80c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abca0c640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40039a0 0x7f2ab80c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abca0c640 1 -- 172.21.15.29:0/2972608619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab80c6300 con 0x7f2ab81f6030 2023-10-16T15:53:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abca0c640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 0x7f2ab81f64b0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80cc190 tx=0x7f2ab00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000e070 con 0x7f2ab81f6030 2023-10-16T15:53:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab0011e10 con 0x7f2ab81f6030 2023-10-16T15:53:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0003990 con 0x7f2ab81f6030 2023-10-16T15:53:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab80c65e0 con 0x7f2ab81f6030 2023-10-16T15:53:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.567+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab80cac90 con 0x7f2ab81f6030 2023-10-16T15:53:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.568+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab804e160 con 0x7f2ab81f6030 2023-10-16T15:53:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.569+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2ab0003af0 con 0x7f2ab81f6030 2023-10-16T15:53:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.570+0000 7f2ab57fa640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 0x7f2a9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.570+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7f2ab008a110 con 0x7f2ab81f6030 2023-10-16T15:53:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.570+0000 7f2ab7fff640 1 -- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 msgr2=0x7f2a9806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:07.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.570+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 0x7f2a9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:07.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.575+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ab0054ea0 con 0x7f2ab81f6030 2023-10-16T15:53:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.771+0000 7f2ab7fff640 1 -- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 msgr2=0x7f2a9806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.771+0000 7f2ab7fff640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 0x7f2a9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:07.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.807+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ab80c9a50 con 0x7f2ab81f6030 2023-10-16T15:53:07.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.808+0000 7f2ab57fa640 1 -- 172.21.15.29:0/2972608619 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f2ab00584c0 con 0x7f2ab81f6030 2023-10-16T15:53:07.809 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:07.809 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:50:44.088346+0000","modified":"2023-10-16T15:50:47.188096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8407":{"gid":8407,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/2149952636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2149952636},{"type":"v1","addr":"172.21.15.29:6833","nonce":2149952636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T15:53:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-16T15:53:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 msgr2=0x7f2a9806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 0x7f2a9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 msgr2=0x7f2ab81f64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 0x7f2ab81f64b0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80cc190 tx=0x7f2ab00115a0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 shutdown_connections 2023-10-16T15:53:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f2a9806af90 0x7f2a9806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ab81f6030 0x7f2ab81f64b0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ab80c5a60 0x7f2ab80ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 --2- 172.21.15.29:0/2972608619 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2aa40039a0 0x7f2ab80c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 >> 172.21.15.29:0/2972608619 conn(0x7f2ab80cb410 msgr2=0x7f2ab806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 shutdown_connections 2023-10-16T15:53:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:07.810+0000 7f2abe46f640 1 -- 172.21.15.29:0/2972608619 wait complete. 2023-10-16T15:53:07.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:07.940 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:07.938+0000 7f6b00a14040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:53:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.024+0000 7f05ce311640 1 Processor -- start 2023-10-16T15:53:08.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.024+0000 7f05ce311640 1 -- start start 2023-10-16T15:53:08.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05ce311640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05c80b8480 0x7f05c80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05ce311640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c80d0be0 0x7f05c80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05ce311640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c815d150 con 0x7f05c815cd70 2023-10-16T15:53:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05ce311640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c80d1180 con 0x7f05c80b8480 2023-10-16T15:53:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05ce311640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80d1460 con 0x7f05c80d0be0 2023-10-16T15:53:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05cd30f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f05c815cd70 0x7f05c80d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41084/0 (socket says 172.21.15.29:41084) 2023-10-16T15:53:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05cd30f640 1 -- 172.21.15.29:0/4076854950 learned_addr learned my addr 172.21.15.29:0/4076854950 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.026+0000 7f05ccb0e640 1 --2- 172.21.15.29:0/4076854950 >> v2:172.21.15.29:3300/0 conn(0x7f05c80b8480 0x7f05c80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.027+0000 7f05cdb10640 1 --2- 172.21.15.29:0/4076854950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c80d0be0 0x7f05c80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.027+0000 7f05bffff640 1 -- 172.21.15.29:0/4076854950 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282687985 0 0) 0x7f05c815d150 con 0x7f05c815cd70 2023-10-16T15:53:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.027+0000 7f05ccb0e640 1 -- 172.21.15.29:0/4076854950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c80d0be0 msgr2=0x7f05c80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.027+0000 7f05ccb0e640 1 --2- 172.21.15.29:0/4076854950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c80d0be0 0x7f05c80cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.027+0000 7f05ccb0e640 1 -- 172.21.15.29:0/4076854950 >> v1:172.21.15.29:6789/0 conn(0x7f05c815cd70 legacy=0x7f05c80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.027+0000 7f05ccb0e640 1 -- 172.21.15.29:0/4076854950 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80d1740 con 0x7f05c80b8480 2023-10-16T15:53:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.028+0000 7f05ccb0e640 1 --2- 172.21.15.29:0/4076854950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80b8480 0x7f05c80cd7c0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097a0 tx=0x7f05b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b469ee475b903d6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.028+0000 7f05bffff640 1 -- 172.21.15.29:0/4076854950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8031830 con 0x7f05c80b8480 2023-10-16T15:53:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.028+0000 7f05bffff640 1 -- 172.21.15.29:0/4076854950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8031990 con 0x7f05c80b8480 2023-10-16T15:53:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.028+0000 7f05bffff640 1 -- 172.21.15.29:0/4076854950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8031d20 con 0x7f05c80b8480 2023-10-16T15:53:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 -- 172.21.15.29:0/4076854950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80b8480 msgr2=0x7f05c80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 --2- 172.21.15.29:0/4076854950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80b8480 0x7f05c80cd7c0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097a0 tx=0x7f05b802b8d0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 -- 172.21.15.29:0/4076854950 shutdown_connections 2023-10-16T15:53:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 --2- 172.21.15.29:0/4076854950 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c80d0be0 0x7f05c80cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 --2- 172.21.15.29:0/4076854950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80b8480 0x7f05c80cd7c0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 -- 172.21.15.29:0/4076854950 >> 172.21.15.29:0/4076854950 conn(0x7f05c80cf570 msgr2=0x7f05c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 -- 172.21.15.29:0/4076854950 shutdown_connections 2023-10-16T15:53:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.029+0000 7f05ce311640 1 -- 172.21.15.29:0/4076854950 wait complete. 2023-10-16T15:53:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.031+0000 7f05ce311640 1 Processor -- start 2023-10-16T15:53:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.031+0000 7f05ce311640 1 -- start start 2023-10-16T15:53:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ce311640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05c80b8480 0x7f05c815fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ce311640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 0x7f05c8156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ce311640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c815cd70 0x7f05c815d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ce311640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8156960 con 0x7f05c80d0be0 2023-10-16T15:53:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ce311640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c814fe30 con 0x7f05c80b8480 2023-10-16T15:53:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ce311640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8160310 con 0x7f05c815cd70 2023-10-16T15:53:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ccb0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 0x7f05c8156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ccb0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 0x7f05c8156420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33664/0 (socket says 172.21.15.29:33664) 2023-10-16T15:53:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.032+0000 7f05ccb0e640 1 -- 172.21.15.29:0/3705499219 learned_addr learned my addr 172.21.15.29:0/3705499219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05cd30f640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05c80b8480 0x7f05c815fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05cdb10640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c815cd70 0x7f05c815d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05ccb0e640 1 -- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c815cd70 msgr2=0x7f05c815d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05ccb0e640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c815cd70 0x7f05c815d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05ccb0e640 1 -- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05c80b8480 msgr2=0x7f05c815fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05ccb0e640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05c80b8480 0x7f05c815fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05ccb0e640 1 -- 172.21.15.29:0/3705499219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8009400 con 0x7f05c80d0be0 2023-10-16T15:53:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05cd30f640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05c80b8480 0x7f05c815fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.033+0000 7f05ccb0e640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 0x7f05c8156420 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097a0 tx=0x7f05b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.034+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8004230 con 0x7f05c80d0be0 2023-10-16T15:53:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.034+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8004390 con 0x7f05c80d0be0 2023-10-16T15:53:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.034+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b803a870 con 0x7f05c80d0be0 2023-10-16T15:53:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.034+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c80cacb0 con 0x7f05c80d0be0 2023-10-16T15:53:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.034+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c80cb270 con 0x7f05c80d0be0 2023-10-16T15:53:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.035+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c804e9b0 con 0x7f05c80d0be0 2023-10-16T15:53:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.036+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 124) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f05b803a9d0 con 0x7f05c80d0be0 2023-10-16T15:53:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.037+0000 7f05bdffb640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f05ac06afc0 0x7f05ac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.037+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6139+0+0 (secure 0 0 0) 0x7f05b80abc60 con 0x7f05c80d0be0 2023-10-16T15:53:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.037+0000 7f05cd30f640 1 -- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f05ac06afc0 msgr2=0x7f05ac06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/928368069 2023-10-16T15:53:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.037+0000 7f05cd30f640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f05ac06afc0 0x7f05ac06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.042+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05b8076930 con 0x7f05c80d0be0 2023-10-16T15:53:08.193 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:08.191+0000 7f4d18e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:53:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.229+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05c80d2470 con 0x7f05c80d0be0 2023-10-16T15:53:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.232+0000 7f05bdffb640 1 -- 172.21.15.29:0/3705499219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17170 (secure 0 0 0) 0x7f05b8035020 con 0x7f05c80d0be0 2023-10-16T15:53:08.233 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:08.233 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":212,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:52:00.390006+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.24","mappings":[{"from":0,"to":1}]},{"pgid":"25.2a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/423324657":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/3913323747":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2674320589":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/275392394":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6833/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6832/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2170359614":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/1066951432":"2023-10-17T15:50:59.297259+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:08.234 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.232+0000 7f6b00a14040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:53:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f05ac06afc0 msgr2=0x7f05ac06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f05ac06afc0 0x7f05ac06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 msgr2=0x7f05c8156420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 0x7f05c8156420 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097a0 tx=0x7f05b80046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 shutdown_connections 2023-10-16T15:53:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:6832/928368069,v1:172.21.15.170:6833/928368069] conn(0x7f05ac06afc0 0x7f05ac06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05c80d0be0 0x7f05c8156420 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05c815cd70 0x7f05c815d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.234+0000 7f05ce311640 1 --2- 172.21.15.29:0/3705499219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05c80b8480 0x7f05c815fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.235+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 >> 172.21.15.29:0/3705499219 conn(0x7f05c80cf570 msgr2=0x7f05c80cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.235+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 shutdown_connections 2023-10-16T15:53:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.235+0000 7f05ce311640 1 -- 172.21.15.29:0/3705499219 wait complete. 2023-10-16T15:53:08.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:50:42.031547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:50:43.111119+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:53:08.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:53:08.289 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.287+0000 7f6b00a14040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:53:08.405 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.404+0000 7f6b00a14040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:53:08.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.454+0000 7ff2c5fb0640 1 Processor -- start 2023-10-16T15:53:08.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.454+0000 7ff2c5fb0640 1 -- start start 2023-10-16T15:53:08.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2c5fb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c015ad70 0x7ff2c0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2c5fb0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c00cb030 con 0x7ff2c00c9ca0 2023-10-16T15:53:08.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2c5fb0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c00c91c0 con 0x7ff2c00caac0 2023-10-16T15:53:08.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2c5fb0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00c94a0 con 0x7ff2c015ad70 2023-10-16T15:53:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2bf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c015ad70 0x7ff2c0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2beffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff2c00c9ca0 0x7ff2c00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41098/0 (socket says 172.21.15.29:41098) 2023-10-16T15:53:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2beffd640 1 -- 172.21.15.29:0/819607767 learned_addr learned my addr 172.21.15.29:0/819607767 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2bf7fe640 1 -- 172.21.15.29:0/819607767 >> v1:172.21.15.170:6789/0 conn(0x7ff2c00caac0 legacy=0x7ff2c00ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.456+0000 7ff2be7fc640 1 -- 172.21.15.29:0/819607767 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658383471 0 0) 0x7ff2c00cb030 con 0x7ff2c00c9ca0 2023-10-16T15:53:08.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.457+0000 7ff2bf7fe640 1 -- 172.21.15.29:0/819607767 >> v1:172.21.15.29:6789/0 conn(0x7ff2c00c9ca0 legacy=0x7ff2c00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.457+0000 7ff2bf7fe640 1 -- 172.21.15.29:0/819607767 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c00c9780 con 0x7ff2c015ad70 2023-10-16T15:53:08.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.458+0000 7ff2bf7fe640 1 --2- 172.21.15.29:0/819607767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c015ad70 0x7ff2c0069da0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009750 tx=0x7ff2b002b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ebe98802b229a4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:08.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.458+0000 7ff2be7fc640 1 -- 172.21.15.29:0/819607767 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0030710 con 0x7ff2c015ad70 2023-10-16T15:53:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.458+0000 7ff2be7fc640 1 -- 172.21.15.29:0/819607767 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b002e050 con 0x7ff2c015ad70 2023-10-16T15:53:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.458+0000 7ff2be7fc640 1 -- 172.21.15.29:0/819607767 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0030a90 con 0x7ff2c015ad70 2023-10-16T15:53:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/819607767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c015ad70 msgr2=0x7ff2c0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 --2- 172.21.15.29:0/819607767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c015ad70 0x7ff2c0069da0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009750 tx=0x7ff2b002b730 comp rx=0 tx=0).stop 2023-10-16T15:53:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/819607767 shutdown_connections 2023-10-16T15:53:08.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 --2- 172.21.15.29:0/819607767 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c015ad70 0x7ff2c0069da0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/819607767 >> 172.21.15.29:0/819607767 conn(0x7ff2c00ce9b0 msgr2=0x7ff2c00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:08.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/819607767 shutdown_connections 2023-10-16T15:53:08.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.459+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/819607767 wait complete. 2023-10-16T15:53:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.461+0000 7ff2c5fb0640 1 Processor -- start 2023-10-16T15:53:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.461+0000 7ff2c5fb0640 1 -- start start 2023-10-16T15:53:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2c5fb0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c00c9ca0 0x7ff2c00c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2c5fb0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 0x7ff2c00b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2c5fb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c015ad70 0x7ff2c0150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2c5fb0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00b3ac0 con 0x7ff2c015ad70 2023-10-16T15:53:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2c5fb0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c015d650 con 0x7ff2c00c9ca0 2023-10-16T15:53:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2c5fb0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c015d930 con 0x7ff2c00caac0 2023-10-16T15:53:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2beffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 0x7ff2c00b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c015ad70 0x7ff2c0150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2bf7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c00c9ca0 0x7ff2c00c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2beffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 0x7ff2c00b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:60334/0 (socket says 172.21.15.29:60334) 2023-10-16T15:53:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.462+0000 7ff2beffd640 1 -- 172.21.15.29:0/1847207009 learned_addr learned my addr 172.21.15.29:0/1847207009 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.463+0000 7ff2beffd640 1 -- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c00c9ca0 msgr2=0x7ff2c00c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.463+0000 7ff2beffd640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c00c9ca0 0x7ff2c00c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.463+0000 7ff2beffd640 1 -- 172.21.15.29:0/1847207009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c015ad70 msgr2=0x7ff2c0150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.463+0000 7ff2beffd640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c015ad70 0x7ff2c0150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.463+0000 7ff2beffd640 1 -- 172.21.15.29:0/1847207009 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b0009400 con 0x7ff2c00caac0 2023-10-16T15:53:08.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.463+0000 7ff2beffd640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 0x7ff2c00b3580 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400c4a0 tx=0x7ff2b400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:08.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.464+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40074e0 con 0x7ff2c00caac0 2023-10-16T15:53:08.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.464+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b4007640 con 0x7ff2c00caac0 2023-10-16T15:53:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.464+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4004a00 con 0x7ff2c00caac0 2023-10-16T15:53:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.464+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c015dc10 con 0x7ff2c00caac0 2023-10-16T15:53:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.464+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c0150fb0 con 0x7ff2c00caac0 2023-10-16T15:53:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.465+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c015e130 con 0x7ff2c00caac0 2023-10-16T15:53:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.466+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 4 ==== mgrmap(e 125) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff2b4004b60 con 0x7ff2c00caac0 2023-10-16T15:53:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.467+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7ff2b40885a0 con 0x7ff2c00caac0 2023-10-16T15:53:08.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.472+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2c015e130 con 0x7ff2c00caac0 2023-10-16T15:53:08.506 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.504+0000 7f6a85a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.506 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.504+0000 7f6a85a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.506 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.504+0000 7f6a85a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.507 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.504+0000 7f6a85a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.507 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.504+0000 7f6a85a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.507 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.505+0000 7f6a82a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.507 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.505+0000 7f6a82a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.508 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.505+0000 7f6a82a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.508 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.505+0000 7f6a82a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.508 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:08.505+0000 7f6a82a69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:08.539 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:08.538+0000 7f4d18e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:53:08.599 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:08.598+0000 7f4d18e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:53:08.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.662+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2c015e130 con 0x7ff2c00caac0 2023-10-16T15:53:08.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.663+0000 7ff2a7fff640 1 -- 172.21.15.29:0/1847207009 <== mon.2 v2:172.21.15.170:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v213) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff2c015e130 con 0x7ff2c00caac0 2023-10-16T15:53:08.664 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:53:08.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 msgr2=0x7ff2c00b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 0x7ff2c00b3580 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400c4a0 tx=0x7ff2b400c960 comp rx=0 tx=0).stop 2023-10-16T15:53:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 shutdown_connections 2023-10-16T15:53:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c015ad70 0x7ff2c0150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2c00caac0 0x7ff2c00b3580 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 --2- 172.21.15.29:0/1847207009 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2c00c9ca0 0x7ff2c00c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 >> 172.21.15.29:0/1847207009 conn(0x7ff2c00ce9b0 msgr2=0x7ff2c006b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 shutdown_connections 2023-10-16T15:53:08.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.666+0000 7ff2c5fb0640 1 -- 172.21.15.29:0/1847207009 wait complete. 2023-10-16T15:53:08.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:53:08.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:08.728+0000 7f4d18e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:53:08.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.908+0000 7fde20a4b640 1 Processor -- start 2023-10-16T15:53:08.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.908+0000 7fde20a4b640 1 -- start start 2023-10-16T15:53:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.909+0000 7fde20a4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c15d330 0x7fde1c1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.909+0000 7fde20a4b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde1c06c7d0 0x7fde1c065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.909+0000 7fde20a4b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c0d0890 con 0x7fde1c06cc80 2023-10-16T15:53:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.909+0000 7fde20a4b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c06b1e0 con 0x7fde1c06c7d0 2023-10-16T15:53:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.909+0000 7fde20a4b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0d05b0 con 0x7fde1c15d330 2023-10-16T15:53:08.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde1c06c7d0 0x7fde1c065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde1c06c7d0 0x7fde1c065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33690/0 (socket says 172.21.15.29:33690) 2023-10-16T15:53:08.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 -- 172.21.15.29:0/1443668216 learned_addr learned my addr 172.21.15.29:0/1443668216 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:08.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 -- 172.21.15.29:0/1443668216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c15d330 msgr2=0x7fde1c1618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:08.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 --2- 172.21.15.29:0/1443668216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c15d330 0x7fde1c1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 -- 172.21.15.29:0/1443668216 >> v1:172.21.15.170:6789/0 conn(0x7fde1c06cc80 legacy=0x7fde1c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:08.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.910+0000 7fde19d9b640 1 -- 172.21.15.29:0/1443668216 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c160dd0 con 0x7fde1c06c7d0 2023-10-16T15:53:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.911+0000 7fde19d9b640 1 --2- 172.21.15.29:0/1443668216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06c7d0 0x7fde1c065ab0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fde04009870 tx=0x7fde0402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7b4d8548511031 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.911+0000 7fde1959a640 1 -- 172.21.15.29:0/1443668216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde040317c0 con 0x7fde1c06c7d0 2023-10-16T15:53:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.911+0000 7fde1959a640 1 -- 172.21.15.29:0/1443668216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04031920 con 0x7fde1c06c7d0 2023-10-16T15:53:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.911+0000 7fde1959a640 1 -- 172.21.15.29:0/1443668216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04031cb0 con 0x7fde1c06c7d0 2023-10-16T15:53:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 -- 172.21.15.29:0/1443668216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06c7d0 msgr2=0x7fde1c065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 --2- 172.21.15.29:0/1443668216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06c7d0 0x7fde1c065ab0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fde04009870 tx=0x7fde0402b860 comp rx=0 tx=0).stop 2023-10-16T15:53:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 -- 172.21.15.29:0/1443668216 shutdown_connections 2023-10-16T15:53:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 --2- 172.21.15.29:0/1443668216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06c7d0 0x7fde1c065ab0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 --2- 172.21.15.29:0/1443668216 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c15d330 0x7fde1c1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 -- 172.21.15.29:0/1443668216 >> 172.21.15.29:0/1443668216 conn(0x7fde1c0d0ab0 msgr2=0x7fde1c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 -- 172.21.15.29:0/1443668216 shutdown_connections 2023-10-16T15:53:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.912+0000 7fde20a4b640 1 -- 172.21.15.29:0/1443668216 wait complete. 2023-10-16T15:53:08.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.914+0000 7fde20a4b640 1 Processor -- start 2023-10-16T15:53:08.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.914+0000 7fde20a4b640 1 -- start start 2023-10-16T15:53:08.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde20a4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c06c7d0 0x7fde1c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde20a4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 0x7fde1c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde20a4b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde1c15d330 0x7fde1c0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde20a4b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c15e660 con 0x7fde1c06cc80 2023-10-16T15:53:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde20a4b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0c8220 con 0x7fde1c15d330 2023-10-16T15:53:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde20a4b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c15f0f0 con 0x7fde1c06c7d0 2023-10-16T15:53:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 0x7fde1c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 0x7fde1c0ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33704/0 (socket says 172.21.15.29:33704) 2023-10-16T15:53:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 -- 172.21.15.29:0/46467274 learned_addr learned my addr 172.21.15.29:0/46467274 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 -- 172.21.15.29:0/46467274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c06c7d0 msgr2=0x7fde1c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c06c7d0 0x7fde1c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 -- 172.21.15.29:0/46467274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde1c15d330 msgr2=0x7fde1c0ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde1c15d330 0x7fde1c0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.915+0000 7fde19d9b640 1 -- 172.21.15.29:0/46467274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde040094d0 con 0x7fde1c06cc80 2023-10-16T15:53:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.916+0000 7fde19d9b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 0x7fde1c0ca270 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fde04000c00 tx=0x7fde040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.916+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04032860 con 0x7fde1c06cc80 2023-10-16T15:53:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.916+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde040329e0 con 0x7fde1c06cc80 2023-10-16T15:53:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.916+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c15e850 con 0x7fde1c06cc80 2023-10-16T15:53:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.916+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04032d90 con 0x7fde1c06cc80 2023-10-16T15:53:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.916+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c15ee10 con 0x7fde1c06cc80 2023-10-16T15:53:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.917+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c04e160 con 0x7fde1c06cc80 2023-10-16T15:53:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.918+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 125) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fde0404e450 con 0x7fde1c06cc80 2023-10-16T15:53:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.919+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7fde040ab770 con 0x7fde1c06cc80 2023-10-16T15:53:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:08.924+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde04076840 con 0x7fde1c06cc80 2023-10-16T15:53:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.113+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fde1c06f3a0 con 0x7fde1c06cc80 2023-10-16T15:53:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.114+0000 7fde12ffd640 1 -- 172.21.15.29:0/46467274 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v213) v1 ==== 104+0+11 (secure 0 0 0) 0x7fde04079e60 con 0x7fde1c06cc80 2023-10-16T15:53:09.116 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:53:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.116+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 msgr2=0x7fde1c0ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.116+0000 7fde20a4b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 0x7fde1c0ca270 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fde04000c00 tx=0x7fde040046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 shutdown_connections 2023-10-16T15:53:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde1c06cc80 0x7fde1c0ca270 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde1c15d330 0x7fde1c0ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 --2- 172.21.15.29:0/46467274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde1c06c7d0 0x7fde1c15e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 >> 172.21.15.29:0/46467274 conn(0x7fde1c0d0ab0 msgr2=0x7fde1c15bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 shutdown_connections 2023-10-16T15:53:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.117+0000 7fde20a4b640 1 -- 172.21.15.29:0/46467274 wait complete. 2023-10-16T15:53:09.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:53:09.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.411+0000 7f30388cf640 1 Processor -- start 2023-10-16T15:53:09.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.412+0000 7f30388cf640 1 -- start start 2023-10-16T15:53:09.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.413+0000 7f30388cf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3034070830 0x7f303406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.413+0000 7f30388cf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3034161740 con 0x7f30340cf5f0 2023-10-16T15:53:09.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.413+0000 7f30388cf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303406b0f0 con 0x7f3034070c10 2023-10-16T15:53:09.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.413+0000 7f30388cf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30341619b0 con 0x7f3034070830 2023-10-16T15:53:09.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.414+0000 7f302bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30340cf5f0 0x7f303406c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41112/0 (socket says 172.21.15.29:41112) 2023-10-16T15:53:09.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.414+0000 7f302bfff640 1 -- 172.21.15.29:0/3979847381 learned_addr learned my addr 172.21.15.29:0/3979847381 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.414+0000 7f303259c640 1 --2- 172.21.15.29:0/3979847381 >> v2:172.21.15.170:3300/0 conn(0x7f3034070830 0x7f303406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.414+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532720540 0 0) 0x7f3034161740 con 0x7f30340cf5f0 2023-10-16T15:53:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.414+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020003520 con 0x7f30340cf5f0 2023-10-16T15:53:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3822144554 0 0) 0x7f3020003520 con 0x7f30340cf5f0 2023-10-16T15:53:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 >> v2:172.21.15.170:3300/0 conn(0x7f3034070830 msgr2=0x7f303406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 --2- 172.21.15.29:0/3979847381 >> v2:172.21.15.170:3300/0 conn(0x7f3034070830 0x7f303406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 >> v1:172.21.15.170:6789/0 conn(0x7f3034070c10 legacy=0x7f30340cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303406b3d0 con 0x7f30340cf5f0 2023-10-16T15:53:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f301c001670 con 0x7f30340cf5f0 2023-10-16T15:53:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 >> v1:172.21.15.29:6789/0 conn(0x7f30340cf5f0 legacy=0x7f303406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3020003970 0x7f3020003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200047c0 0x7f3020004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3020005380 0x7f302000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30341619b0 con 0x7f30200047c0 2023-10-16T15:53:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302000c210 con 0x7f3020003970 2023-10-16T15:53:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302000c3d0 con 0x7f3020005380 2023-10-16T15:53:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f302bfff640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200047c0 0x7f3020004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f303259c640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3020003970 0x7f3020003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f3032d9d640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3020005380 0x7f302000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.415+0000 7f302bfff640 1 -- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3020005380 msgr2=0x7f302000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.416+0000 7f302bfff640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3020005380 0x7f302000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.416+0000 7f302bfff640 1 -- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3020003970 msgr2=0x7f3020003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.416+0000 7f302bfff640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3020003970 0x7f3020003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.416+0000 7f302bfff640 1 -- 172.21.15.29:0/3979847381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3020003520 con 0x7f30200047c0 2023-10-16T15:53:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.416+0000 7f303259c640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3020003970 0x7f3020003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.416+0000 7f3032d9d640 1 -- 172.21.15.29:0/3979847381 reap_dead start 2023-10-16T15:53:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.418+0000 7f302bfff640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200047c0 0x7f3020004c40 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f303406c850 tx=0x7f30200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fad7a700dce6f2c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.418+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c005ad0 con 0x7f30200047c0 2023-10-16T15:53:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.418+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c005c30 con 0x7f30200047c0 2023-10-16T15:53:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.418+0000 7f3031d9b640 1 -- 172.21.15.29:0/3979847381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c02ca90 con 0x7f30200047c0 2023-10-16T15:53:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 -- 172.21.15.29:0/3979847381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200047c0 msgr2=0x7f3020004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200047c0 0x7f3020004c40 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f303406c850 tx=0x7f30200057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 -- 172.21.15.29:0/3979847381 shutdown_connections 2023-10-16T15:53:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 --2- 172.21.15.29:0/3979847381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200047c0 0x7f3020004c40 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 -- 172.21.15.29:0/3979847381 >> 172.21.15.29:0/3979847381 conn(0x7f30340cfac0 msgr2=0x7f30340cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 -- 172.21.15.29:0/3979847381 shutdown_connections 2023-10-16T15:53:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.419+0000 7f30388cf640 1 -- 172.21.15.29:0/3979847381 wait complete. 2023-10-16T15:53:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.421+0000 7f30388cf640 1 Processor -- start 2023-10-16T15:53:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.421+0000 7f30388cf640 1 -- start start 2023-10-16T15:53:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f30388cf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30200047c0 0x7f30341f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f30388cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 0x7f30340ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f30388cf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30340aceb0 0x7f30341f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f30388cf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30340ad330 con 0x7f303415f960 2023-10-16T15:53:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f30388cf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30340c9dd0 con 0x7f30200047c0 2023-10-16T15:53:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f30388cf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30340c8310 con 0x7f30340aceb0 2023-10-16T15:53:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f302bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 0x7f30340ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f302bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 0x7f30340ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33720/0 (socket says 172.21.15.29:33720) 2023-10-16T15:53:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.422+0000 7f302bfff640 1 -- 172.21.15.29:0/2782936134 learned_addr learned my addr 172.21.15.29:0/2782936134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.423+0000 7f302bfff640 1 -- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30340aceb0 msgr2=0x7f30341f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.423+0000 7f302bfff640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30340aceb0 0x7f30341f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.423+0000 7f302bfff640 1 -- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30200047c0 msgr2=0x7f30341f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.423+0000 7f302bfff640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30200047c0 0x7f30341f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.423+0000 7f302bfff640 1 -- 172.21.15.29:0/2782936134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30340c85f0 con 0x7f303415f960 2023-10-16T15:53:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.423+0000 7f302bfff640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 0x7f30340ac970 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f301c02c3f0 tx=0x7f301c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.429+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c03c040 con 0x7f303415f960 2023-10-16T15:53:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.429+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c02cc60 con 0x7f303415f960 2023-10-16T15:53:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.429+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c007070 con 0x7f303415f960 2023-10-16T15:53:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.429+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30340c88d0 con 0x7f303415f960 2023-10-16T15:53:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.429+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30341f2810 con 0x7f303415f960 2023-10-16T15:53:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.430+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30341f2c80 con 0x7f303415f960 2023-10-16T15:53:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.432+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f301c032070 con 0x7f303415f960 2023-10-16T15:53:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.433+0000 7f302affd640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f301406acc0 0x7f301406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.433+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f301c0aa9f0 con 0x7f303415f960 2023-10-16T15:53:09.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.436+0000 7f303259c640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f301406acc0 0x7f301406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.439+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f301c0759d0 con 0x7f303415f960 2023-10-16T15:53:09.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.446+0000 7f303259c640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f301406acc0 0x7f301406d180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3024006a90 tx=0x7f3024006a20 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:09.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.636+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f303404e160 con 0x7f303415f960 2023-10-16T15:53:09.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.637+0000 7f302affd640 1 -- 172.21.15.29:0/2782936134 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v213) v1 ==== 100+0+11 (secure 0 0 0) 0x7f301c078ff0 con 0x7f303415f960 2023-10-16T15:53:09.638 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:53:09.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.639+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f301406acc0 msgr2=0x7f301406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f301406acc0 0x7f301406d180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3024006a90 tx=0x7f3024006a20 comp rx=0 tx=0).stop 2023-10-16T15:53:09.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 msgr2=0x7f30340ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 0x7f30340ac970 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f301c02c3f0 tx=0x7f301c02e070 comp rx=0 tx=0).stop 2023-10-16T15:53:09.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 shutdown_connections 2023-10-16T15:53:09.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f301406acc0 0x7f301406d180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f303415f960 0x7f30340ac970 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30340aceb0 0x7f30341f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 --2- 172.21.15.29:0/2782936134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30200047c0 0x7f30341f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 >> 172.21.15.29:0/2782936134 conn(0x7f30340cfac0 msgr2=0x7f30340ad640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 shutdown_connections 2023-10-16T15:53:09.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.640+0000 7f30388cf640 1 -- 172.21.15.29:0/2782936134 wait complete. 2023-10-16T15:53:09.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.887+0000 7f7e42eb4640 1 Processor -- start 2023-10-16T15:53:09.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.888+0000 7f7e42eb4640 1 -- start start 2023-10-16T15:53:09.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e42eb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c15cbe0 0x7f7e3c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e42eb4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e3c0b4c80 0x7f7e3c0b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e42eb4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c0d15a0 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e42eb4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c0d1880 con 0x7f7e3c0b4c80 2023-10-16T15:53:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e42eb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0d1b60 con 0x7f7e3c15cbe0 2023-10-16T15:53:09.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e426b3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e3c0b4c80 0x7f7e3c0b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e416b1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7e3c15cfc0 0x7f7e3c0b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41128/0 (socket says 172.21.15.29:41128) 2023-10-16T15:53:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.889+0000 7f7e416b1640 1 -- 172.21.15.29:0/362660759 learned_addr learned my addr 172.21.15.29:0/362660759 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e41eb2640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c15cbe0 0x7f7e3c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e426b3640 1 -- 172.21.15.29:0/362660759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c15cbe0 msgr2=0x7f7e3c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:09.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e426b3640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c15cbe0 0x7f7e3c0d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e426b3640 1 -- 172.21.15.29:0/362660759 >> v1:172.21.15.29:6789/0 conn(0x7f7e3c15cfc0 legacy=0x7f7e3c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e426b3640 1 -- 172.21.15.29:0/362660759 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c0d1e40 con 0x7f7e3c0b4c80 2023-10-16T15:53:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e40eb0640 1 -- 172.21.15.29:0/362660759 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739425216 0 0) 0x7f7e3c0d15a0 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.890+0000 7f7e41eb2640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c15cbe0 0x7f7e3c0d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.891+0000 7f7e426b3640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0b4c80 0x7f7e3c0b5c20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009d90 tx=0x7f7e3402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e0d9850a4e4ba4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.891+0000 7f7e40eb0640 1 -- 172.21.15.29:0/362660759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3402cce0 con 0x7f7e3c0b4c80 2023-10-16T15:53:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.891+0000 7f7e40eb0640 1 -- 172.21.15.29:0/362660759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3402ce40 con 0x7f7e3c0b4c80 2023-10-16T15:53:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.891+0000 7f7e40eb0640 1 -- 172.21.15.29:0/362660759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34034660 con 0x7f7e3c0b4c80 2023-10-16T15:53:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 -- 172.21.15.29:0/362660759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0b4c80 msgr2=0x7f7e3c0b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0b4c80 0x7f7e3c0b5c20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009d90 tx=0x7f7e3402bf30 comp rx=0 tx=0).stop 2023-10-16T15:53:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 -- 172.21.15.29:0/362660759 shutdown_connections 2023-10-16T15:53:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c0b4c80 0x7f7e3c0b5c20 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/362660759 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c15cbe0 0x7f7e3c0d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 -- 172.21.15.29:0/362660759 >> 172.21.15.29:0/362660759 conn(0x7f7e3c0cf570 msgr2=0x7f7e3c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 -- 172.21.15.29:0/362660759 shutdown_connections 2023-10-16T15:53:09.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.892+0000 7f7e42eb4640 1 -- 172.21.15.29:0/362660759 wait complete. 2023-10-16T15:53:09.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.894+0000 7f7e42eb4640 1 Processor -- start 2023-10-16T15:53:09.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.894+0000 7f7e42eb4640 1 -- start start 2023-10-16T15:53:09.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e42eb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c0b4c80 0x7f7e3c155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e42eb4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c15cbe0 0x7f7e3c0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e42eb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 0x7f7e3c0ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e42eb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0c8070 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e42eb4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c156730 con 0x7f7e3c15cbe0 2023-10-16T15:53:09.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e42eb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c0c9210 con 0x7f7e3c0b4c80 2023-10-16T15:53:09.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e426b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 0x7f7e3c0ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e426b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 0x7f7e3c0ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33742/0 (socket says 172.21.15.29:33742) 2023-10-16T15:53:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e426b3640 1 -- 172.21.15.29:0/4071811490 learned_addr learned my addr 172.21.15.29:0/4071811490 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e416b1640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c15cbe0 0x7f7e3c0c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e41eb2640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c0b4c80 0x7f7e3c155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e426b3640 1 -- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c0b4c80 msgr2=0x7f7e3c155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.895+0000 7f7e426b3640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c0b4c80 0x7f7e3c155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e426b3640 1 -- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c15cbe0 msgr2=0x7f7e3c0c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e426b3640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c15cbe0 0x7f7e3c0c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e426b3640 1 -- 172.21.15.29:0/4071811490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34009a70 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e41eb2640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c0b4c80 0x7f7e3c155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:09.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e416b1640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c15cbe0 0x7f7e3c0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e426b3640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 0x7f7e3c0ca9a0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f7e34037040 tx=0x7f7e34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3403d600 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3403d760 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3403db10 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c0c8490 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.896+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c0c76d0 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.898+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c0c81b0 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.899+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7e3403b040 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.900+0000 7f7e327fc640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e2006acc0 0x7f7e2006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.900+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f7e340ab8c0 con 0x7f7e3c15cfc0 2023-10-16T15:53:09.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.901+0000 7f7e41eb2640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e2006acc0 0x7f7e2006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.902+0000 7f7e41eb2640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e2006acc0 0x7f7e2006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0b7d30 tx=0x7f7e38008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:09.904+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e3c0c81b0 con 0x7f7e3c15cfc0 2023-10-16T15:53:10.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.135+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e3c0c81b0 con 0x7f7e3c15cfc0 2023-10-16T15:53:10.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.136+0000 7f7e327fc640 1 -- 172.21.15.29:0/4071811490 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f7e3c0c81b0 con 0x7f7e3c15cfc0 2023-10-16T15:53:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:10.137 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:50:44.088346+0000","modified":"2023-10-16T15:50:47.188096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8407":{"gid":8407,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/2149952636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2149952636},{"type":"v1","addr":"172.21.15.29:6833","nonce":2149952636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T15:53:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-16T15:53:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e2006acc0 msgr2=0x7f7e2006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e2006acc0 0x7f7e2006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0b7d30 tx=0x7f7e38008040 comp rx=0 tx=0).stop 2023-10-16T15:53:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 msgr2=0x7f7e3c0ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 0x7f7e3c0ca9a0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f7e34037040 tx=0x7f7e34004a40 comp rx=0 tx=0).stop 2023-10-16T15:53:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 shutdown_connections 2023-10-16T15:53:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e2006acc0 0x7f7e2006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e3c15cfc0 0x7f7e3c0ca9a0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e3c15cbe0 0x7f7e3c0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 --2- 172.21.15.29:0/4071811490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e3c0b4c80 0x7f7e3c155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.138+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 >> 172.21.15.29:0/4071811490 conn(0x7f7e3c0cf570 msgr2=0x7f7e3c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.139+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 shutdown_connections 2023-10-16T15:53:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.139+0000 7f7e42eb4640 1 -- 172.21.15.29:0/4071811490 wait complete. 2023-10-16T15:53:10.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.386+0000 7f30d295e640 1 Processor -- start 2023-10-16T15:53:10.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.387+0000 7f30d295e640 1 -- start start 2023-10-16T15:53:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30d295e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30cc15aaa0 0x7f30cc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30d295e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f30cc0ce390 0x7f30cc070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30d295e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc15ed60 con 0x7f30cc15ae80 2023-10-16T15:53:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30d295e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc06b810 con 0x7f30cc15aaa0 2023-10-16T15:53:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30d295e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc0707e0 con 0x7f30cc0ce390 2023-10-16T15:53:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30cbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30cc15aaa0 0x7f30cc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30cb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30cc15ae80 0x7f30cc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41134/0 (socket says 172.21.15.29:41134) 2023-10-16T15:53:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.388+0000 7f30cb7fe640 1 -- 172.21.15.29:0/2804514148 learned_addr learned my addr 172.21.15.29:0/2804514148 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.389+0000 7f30d0efb640 1 --2- 172.21.15.29:0/2804514148 >> v2:172.21.15.170:3300/0 conn(0x7f30cc0ce390 0x7f30cc070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.389+0000 7f30caffd640 1 -- 172.21.15.29:0/2804514148 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718881212 0 0) 0x7f30cc15ed60 con 0x7f30cc15ae80 2023-10-16T15:53:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.389+0000 7f30cbfff640 1 -- 172.21.15.29:0/2804514148 >> v2:172.21.15.170:3300/0 conn(0x7f30cc0ce390 msgr2=0x7f30cc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:10.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.389+0000 7f30cbfff640 1 --2- 172.21.15.29:0/2804514148 >> v2:172.21.15.170:3300/0 conn(0x7f30cc0ce390 0x7f30cc070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.389+0000 7f30cbfff640 1 -- 172.21.15.29:0/2804514148 >> v1:172.21.15.29:6789/0 conn(0x7f30cc15ae80 legacy=0x7f30cc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.389+0000 7f30cbfff640 1 -- 172.21.15.29:0/2804514148 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc15ef90 con 0x7f30cc15aaa0 2023-10-16T15:53:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.391+0000 7f30cbfff640 1 --2- 172.21.15.29:0/2804514148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc15aaa0 0x7f30cc15e820 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f30cc0c2490 tx=0x7f30bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2217de4991a68404 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.391+0000 7f30caffd640 1 -- 172.21.15.29:0/2804514148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc031830 con 0x7f30cc15aaa0 2023-10-16T15:53:10.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.391+0000 7f30caffd640 1 -- 172.21.15.29:0/2804514148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc031990 con 0x7f30cc15aaa0 2023-10-16T15:53:10.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.391+0000 7f30caffd640 1 -- 172.21.15.29:0/2804514148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc031d20 con 0x7f30cc15aaa0 2023-10-16T15:53:10.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.391+0000 7f30d295e640 1 -- 172.21.15.29:0/2804514148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc15aaa0 msgr2=0x7f30cc15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.391+0000 7f30d295e640 1 --2- 172.21.15.29:0/2804514148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc15aaa0 0x7f30cc15e820 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f30cc0c2490 tx=0x7f30bc02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.392+0000 7f30d295e640 1 -- 172.21.15.29:0/2804514148 shutdown_connections 2023-10-16T15:53:10.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.392+0000 7f30d295e640 1 --2- 172.21.15.29:0/2804514148 >> v2:172.21.15.170:3300/0 conn(0x7f30cc0ce390 0x7f30cc070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.392+0000 7f30d295e640 1 --2- 172.21.15.29:0/2804514148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc15aaa0 0x7f30cc15e820 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.392+0000 7f30d295e640 1 -- 172.21.15.29:0/2804514148 >> 172.21.15.29:0/2804514148 conn(0x7f30cc0cb410 msgr2=0x7f30cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:10.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.392+0000 7f30d295e640 1 -- 172.21.15.29:0/2804514148 shutdown_connections 2023-10-16T15:53:10.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.392+0000 7f30d295e640 1 -- 172.21.15.29:0/2804514148 wait complete. 2023-10-16T15:53:10.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.393+0000 7f30d295e640 1 Processor -- start 2023-10-16T15:53:10.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.394+0000 7f30d295e640 1 -- start start 2023-10-16T15:53:10.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30d295e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 0x7f30cc0a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30d295e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc15aaa0 0x7f30cc0aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30d295e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc15ae80 0x7f30cc0a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30d295e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0ab2e0 con 0x7f30cc0ce390 2023-10-16T15:53:10.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30d295e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0ab5c0 con 0x7f30cc15ae80 2023-10-16T15:53:10.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30d295e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0a8620 con 0x7f30cc15aaa0 2023-10-16T15:53:10.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 0x7f30cc0a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 0x7f30cc0a9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33772/0 (socket says 172.21.15.29:33772) 2023-10-16T15:53:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 -- 172.21.15.29:0/1908809003 learned_addr learned my addr 172.21.15.29:0/1908809003 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 -- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc15aaa0 msgr2=0x7f30cc0aaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc15aaa0 0x7f30cc0aaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 -- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc15ae80 msgr2=0x7f30cc0a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc15ae80 0x7f30cc0a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.395+0000 7f30cbfff640 1 -- 172.21.15.29:0/1908809003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc009400 con 0x7f30cc0ce390 2023-10-16T15:53:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.396+0000 7f30cbfff640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 0x7f30cc0a9db0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009890 tx=0x7f30bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.396+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0317d0 con 0x7f30cc0ce390 2023-10-16T15:53:10.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.396+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc03ba70 con 0x7f30cc0ce390 2023-10-16T15:53:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.396+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc03be20 con 0x7f30cc0ce390 2023-10-16T15:53:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.396+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc0a8900 con 0x7f30cc0ce390 2023-10-16T15:53:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.396+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc0aafd0 con 0x7f30cc0ce390 2023-10-16T15:53:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.397+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc0a8480 con 0x7f30cc0ce390 2023-10-16T15:53:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.398+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f30bc0453f0 con 0x7f30cc0ce390 2023-10-16T15:53:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.399+0000 7f30c8ff9640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f30b806acc0 0x7f30b806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.399+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f30bc0ab8c0 con 0x7f30cc0ce390 2023-10-16T15:53:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.400+0000 7f30cb7fe640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f30b806acc0 0x7f30b806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.401+0000 7f30cb7fe640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f30b806acc0 0x7f30b806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f30c0007fd0 tx=0x7f30c0007960 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.403+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30cc0a8480 con 0x7f30cc0ce390 2023-10-16T15:53:10.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.441+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f30bc076d90 con 0x7f30cc0ce390 2023-10-16T15:53:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.605+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30cc0a8e20 con 0x7f30cc0ce390 2023-10-16T15:53:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.608+0000 7f30c8ff9640 1 -- 172.21.15.29:0/1908809003 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+17789 (secure 0 0 0) 0x7f30cc0a8e20 con 0x7f30cc0ce390 2023-10-16T15:53:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:10.610 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":213,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:08.409916+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":213,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.24","mappings":[{"from":0,"to":1}]},{"pgid":"25.2a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/928368069":"2023-10-17T15:53:08.409834+0000","172.21.15.170:6832/928368069":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/1186413715":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/3227854992":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/568891934":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/2703119246":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/1539386380":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/3252182417":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/423324657":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/3913323747":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2674320589":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/275392394":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6833/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6832/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2170359614":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/1066951432":"2023-10-17T15:50:59.297259+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:10.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f30b806acc0 msgr2=0x7f30b806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f30b806acc0 0x7f30b806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f30c0007fd0 tx=0x7f30c0007960 comp rx=0 tx=0).stop 2023-10-16T15:53:10.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 msgr2=0x7f30cc0a9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 0x7f30cc0a9db0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009890 tx=0x7f30bc0046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 shutdown_connections 2023-10-16T15:53:10.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f30b806acc0 0x7f30b806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30cc15ae80 0x7f30cc0a9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30cc15aaa0 0x7f30cc0aaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 --2- 172.21.15.29:0/1908809003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc0ce390 0x7f30cc0a9db0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 >> 172.21.15.29:0/1908809003 conn(0x7f30cc0cb410 msgr2=0x7f30cc06f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 shutdown_connections 2023-10-16T15:53:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.611+0000 7f30d295e640 1 -- 172.21.15.29:0/1908809003 wait complete. 2023-10-16T15:53:10.632 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:53:10.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.850+0000 7fb677fff640 1 Processor -- start 2023-10-16T15:53:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.850+0000 7fb677fff640 1 -- start start 2023-10-16T15:53:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb677fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb678159950 0x7fb67815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb677fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb67806c810 0x7fb6780708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb677fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67815aef0 con 0x7fb678070450 2023-10-16T15:53:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb677fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb678070e20 con 0x7fb67806c810 2023-10-16T15:53:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb677fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67815e490 con 0x7fb678159950 2023-10-16T15:53:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb6767fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb678070450 0x7fb67806c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41146/0 (socket says 172.21.15.29:41146) 2023-10-16T15:53:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb6767fc640 1 -- 172.21.15.29:0/2548280098 learned_addr learned my addr 172.21.15.29:0/2548280098 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.852+0000 7fb6777fe640 1 --2- 172.21.15.29:0/2548280098 >> v2:172.21.15.29:3300/0 conn(0x7fb67806c810 0x7fb6780708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb676ffd640 1 --2- 172.21.15.29:0/2548280098 >> v2:172.21.15.170:3300/0 conn(0x7fb678159950 0x7fb67815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb675ffb640 1 -- 172.21.15.29:0/2548280098 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232519536 0 0) 0x7fb66c001110 con 0x7fb678070450 2023-10-16T15:53:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb675ffb640 1 -- 172.21.15.29:0/2548280098 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb660003520 con 0x7fb678070450 2023-10-16T15:53:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb6777fe640 1 -- 172.21.15.29:0/2548280098 >> v2:172.21.15.170:3300/0 conn(0x7fb678159950 msgr2=0x7fb67815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb6777fe640 1 --2- 172.21.15.29:0/2548280098 >> v2:172.21.15.170:3300/0 conn(0x7fb678159950 0x7fb67815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb6777fe640 1 -- 172.21.15.29:0/2548280098 >> v1:172.21.15.29:6789/0 conn(0x7fb678070450 legacy=0x7fb67806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.853+0000 7fb6777fe640 1 -- 172.21.15.29:0/2548280098 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67815d910 con 0x7fb67806c810 2023-10-16T15:53:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.854+0000 7fb6777fe640 1 --2- 172.21.15.29:0/2548280098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67806c810 0x7fb6780708e0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb6780ca950 tx=0x7fb67002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dcdb15d5f5fba0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb675ffb640 1 -- 172.21.15.29:0/2548280098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67002cbf0 con 0x7fb67806c810 2023-10-16T15:53:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb675ffb640 1 -- 172.21.15.29:0/2548280098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67002cd50 con 0x7fb67806c810 2023-10-16T15:53:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb675ffb640 1 -- 172.21.15.29:0/2548280098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670034640 con 0x7fb67806c810 2023-10-16T15:53:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb677fff640 1 -- 172.21.15.29:0/2548280098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67806c810 msgr2=0x7fb6780708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb677fff640 1 --2- 172.21.15.29:0/2548280098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67806c810 0x7fb6780708e0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb6780ca950 tx=0x7fb67002be40 comp rx=0 tx=0).stop 2023-10-16T15:53:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb677fff640 1 -- 172.21.15.29:0/2548280098 shutdown_connections 2023-10-16T15:53:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb677fff640 1 --2- 172.21.15.29:0/2548280098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67806c810 0x7fb6780708e0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb677fff640 1 --2- 172.21.15.29:0/2548280098 >> v2:172.21.15.170:3300/0 conn(0x7fb678159950 0x7fb67815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.855+0000 7fb677fff640 1 -- 172.21.15.29:0/2548280098 >> 172.21.15.29:0/2548280098 conn(0x7fb6780ce6a0 msgr2=0x7fb6780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.856+0000 7fb677fff640 1 -- 172.21.15.29:0/2548280098 shutdown_connections 2023-10-16T15:53:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.856+0000 7fb677fff640 1 -- 172.21.15.29:0/2548280098 wait complete. 2023-10-16T15:53:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.857+0000 7fb677fff640 1 Processor -- start 2023-10-16T15:53:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.857+0000 7fb677fff640 1 -- start start 2023-10-16T15:53:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.858+0000 7fb677fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67806c810 0x7fb6780c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.858+0000 7fb677fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 0x7fb6781502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.858+0000 7fb677fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb678159950 0x7fb6780c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb677fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678151710 con 0x7fb678070450 2023-10-16T15:53:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb677fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678150b00 con 0x7fb678159950 2023-10-16T15:53:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb677fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb678150de0 con 0x7fb67806c810 2023-10-16T15:53:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 0x7fb6781502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 0x7fb6781502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33782/0 (socket says 172.21.15.29:33782) 2023-10-16T15:53:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 -- 172.21.15.29:0/4158446663 learned_addr learned my addr 172.21.15.29:0/4158446663 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 -- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67806c810 msgr2=0x7fb6780c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67806c810 0x7fb6780c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 -- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb678159950 msgr2=0x7fb6780c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb678159950 0x7fb6780c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.859+0000 7fb6767fc640 1 -- 172.21.15.29:0/4158446663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670009a70 con 0x7fb678070450 2023-10-16T15:53:10.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.860+0000 7fb6767fc640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 0x7fb6781502d0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c700 tx=0x7fb66c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:10.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.860+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c005ab0 con 0x7fb678070450 2023-10-16T15:53:10.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.860+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c005c10 con 0x7fb678070450 2023-10-16T15:53:10.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.860+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c001d50 con 0x7fb678070450 2023-10-16T15:53:10.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.860+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6781510c0 con 0x7fb678070450 2023-10-16T15:53:10.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.860+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb678150810 con 0x7fb678070450 2023-10-16T15:53:10.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.861+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67806dab0 con 0x7fb678070450 2023-10-16T15:53:10.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.862+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb66c007070 con 0x7fb678070450 2023-10-16T15:53:10.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.864+0000 7fb65f7fe640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb66406af30 0x7fb66406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:10.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.864+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7fb66c088ec0 con 0x7fb678070450 2023-10-16T15:53:10.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.865+0000 7fb676ffd640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb66406af30 0x7fb66406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:10.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.866+0000 7fb676ffd640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb66406af30 0x7fb66406d3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb668006a90 tx=0x7fb668006a20 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:10.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:10.869+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb66c053ab0 con 0x7fb678070450 2023-10-16T15:53:11.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.100+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67804e9b0 con 0x7fb678070450 2023-10-16T15:53:11.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.101+0000 7fb65f7fe640 1 -- 172.21.15.29:0/4158446663 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4925 (secure 0 0 0) 0x7fb66c017030 con 0x7fb678070450 2023-10-16T15:53:11.102 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:11.102 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:50:44.088346+0000","modified":"2023-10-16T15:50:47.188096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8407":{"gid":8407,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/2149952636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2149952636},{"type":"v1","addr":"172.21.15.29:6833","nonce":2149952636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T15:53:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-16T15:53:11.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb66406af30 msgr2=0x7fb66406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb66406af30 0x7fb66406d3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb668006a90 tx=0x7fb668006a20 comp rx=0 tx=0).stop 2023-10-16T15:53:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 msgr2=0x7fb6781502d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 0x7fb6781502d0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c700 tx=0x7fb66c00cbc0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 shutdown_connections 2023-10-16T15:53:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb66406af30 0x7fb66406d3f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb678070450 0x7fb6781502d0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb678159950 0x7fb6780c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 --2- 172.21.15.29:0/4158446663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67806c810 0x7fb6780c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 >> 172.21.15.29:0/4158446663 conn(0x7fb6780ce6a0 msgr2=0x7fb67806b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 shutdown_connections 2023-10-16T15:53:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.104+0000 7fb677fff640 1 -- 172.21.15.29:0/4158446663 wait complete. 2023-10-16T15:53:11.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.352+0000 7f8f05002640 1 Processor -- start 2023-10-16T15:53:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.352+0000 7f8f05002640 1 -- start start 2023-10-16T15:53:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.353+0000 7f8f05002640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8f0015cfc0 0x7f8f000b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.353+0000 7f8f05002640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 0x7f8f000b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.353+0000 7f8f05002640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f000d15a0 con 0x7f8f0015cbe0 2023-10-16T15:53:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.353+0000 7f8f05002640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f000d1880 con 0x7f8f0015cfc0 2023-10-16T15:53:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.353+0000 7f8f05002640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f000d1b60 con 0x7f8f000d2640 2023-10-16T15:53:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.354+0000 7f8efffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8f0015cbe0 0x7f8f000d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41154/0 (socket says 172.21.15.29:41154) 2023-10-16T15:53:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.354+0000 7f8efffff640 1 -- 172.21.15.29:0/761884275 learned_addr learned my addr 172.21.15.29:0/761884275 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.354+0000 7f8f04801640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 0x7f8f000b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.354+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799708542 0 0) 0x7f8f000d15a0 con 0x7f8f0015cbe0 2023-10-16T15:53:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.354+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eec003520 con 0x7f8f0015cbe0 2023-10-16T15:53:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.354+0000 7f8ef77fe640 1 --2- 172.21.15.29:0/761884275 >> v2:172.21.15.170:3300/0 conn(0x7f8f0015cfc0 0x7f8f000b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107255052 0 0) 0x7f8eec003520 con 0x7f8f0015cbe0 2023-10-16T15:53:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 msgr2=0x7f8f000b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 0x7f8f000b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 >> v2:172.21.15.170:3300/0 conn(0x7f8f0015cfc0 msgr2=0x7f8f000b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 --2- 172.21.15.29:0/761884275 >> v2:172.21.15.170:3300/0 conn(0x7f8f0015cfc0 0x7f8f000b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f000d1e40 con 0x7f8f0015cbe0 2023-10-16T15:53:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8ef0001670 con 0x7f8f0015cbe0 2023-10-16T15:53:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 >> v1:172.21.15.29:6789/0 conn(0x7f8f0015cbe0 legacy=0x7f8f000d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 0x7f8eec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eec003ef0 0x7f8eec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eec004d00 0x7f8eec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f000d1880 con 0x7f8eec004d00 2023-10-16T15:53:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f000d1b60 con 0x7f8eec003ef0 2023-10-16T15:53:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec00bab0 con 0x7f8f000d2640 2023-10-16T15:53:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8ef77fe640 1 --2- 172.21.15.29:0/761884275 >> v2:172.21.15.170:3300/0 conn(0x7f8f0015cfc0 0x7f8f000b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.355+0000 7f8f04801640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eec004d00 0x7f8eec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.356+0000 7f8efffff640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 0x7f8eec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.356+0000 7f8f04801640 1 -- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 msgr2=0x7f8eec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.356+0000 7f8f04801640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8f000d2640 0x7f8eec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.356+0000 7f8f04801640 1 -- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eec003ef0 msgr2=0x7f8eec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:11.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.356+0000 7f8f04801640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eec003ef0 0x7f8eec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.356+0000 7f8f04801640 1 -- 172.21.15.29:0/761884275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec003520 con 0x7f8eec004d00 2023-10-16T15:53:11.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.357+0000 7f8f04801640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eec004d00 0x7f8eec005180 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009fc0 tx=0x7f8eec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e058e92c5caa3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:11.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.357+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef802cc20 con 0x7f8eec004d00 2023-10-16T15:53:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.357+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef800f040 con 0x7f8eec004d00 2023-10-16T15:53:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.357+0000 7f8eff7fe640 1 -- 172.21.15.29:0/761884275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8034440 con 0x7f8eec004d00 2023-10-16T15:53:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f05002640 1 -- 172.21.15.29:0/761884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eec004d00 msgr2=0x7f8eec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f05002640 1 --2- 172.21.15.29:0/761884275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eec004d00 0x7f8eec005180 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009fc0 tx=0x7f8eec0057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f04801640 1 -- 172.21.15.29:0/761884275 reap_dead start 2023-10-16T15:53:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f05002640 1 -- 172.21.15.29:0/761884275 shutdown_connections 2023-10-16T15:53:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f05002640 1 -- 172.21.15.29:0/761884275 >> 172.21.15.29:0/761884275 conn(0x7f8f000cf570 msgr2=0x7f8f000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f05002640 1 -- 172.21.15.29:0/761884275 shutdown_connections 2023-10-16T15:53:11.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.358+0000 7f8f05002640 1 -- 172.21.15.29:0/761884275 wait complete. 2023-10-16T15:53:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.360+0000 7f8f05002640 1 Processor -- start 2023-10-16T15:53:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.360+0000 7f8f05002640 1 -- start start 2023-10-16T15:53:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.361+0000 7f8f05002640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eec004d00 0x7f8f0014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.361+0000 7f8f05002640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f0015e8f0 0x7f8f0015ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.361+0000 7f8f05002640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 0x7f8f001e69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.361+0000 7f8f05002640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0015f2b0 con 0x7f8f001e6530 2023-10-16T15:53:11.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.361+0000 7f8f05002640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001f6350 con 0x7f8f0015e8f0 2023-10-16T15:53:11.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.361+0000 7f8f05002640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001f5780 con 0x7f8eec004d00 2023-10-16T15:53:11.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 0x7f8f001e69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 0x7f8f001e69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33804/0 (socket says 172.21.15.29:33804) 2023-10-16T15:53:11.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 -- 172.21.15.29:0/3383346862 learned_addr learned my addr 172.21.15.29:0/3383346862 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:11.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8efffff640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eec004d00 0x7f8f0014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8ef77fe640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f0015e8f0 0x7f8f0015ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 -- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eec004d00 msgr2=0x7f8f0014fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eec004d00 0x7f8f0014fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 -- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f0015e8f0 msgr2=0x7f8f0015ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f0015e8f0 0x7f8f0015ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 -- 172.21.15.29:0/3383346862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f001f5a20 con 0x7f8f001e6530 2023-10-16T15:53:11.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8ef77fe640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f0015e8f0 0x7f8f0015ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:11.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.362+0000 7f8f04801640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 0x7f8f001e69b0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8037040 tx=0x7f8ef8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:11.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.363+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef802cea0 con 0x7f8f001e6530 2023-10-16T15:53:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.363+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef8034ad0 con 0x7f8f001e6530 2023-10-16T15:53:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.363+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8034e80 con 0x7f8f001e6530 2023-10-16T15:53:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.363+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f001f5d00 con 0x7f8f001e6530 2023-10-16T15:53:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.363+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f00150370 con 0x7f8f001e6530 2023-10-16T15:53:11.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.364+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f000b3350 con 0x7f8f001e6530 2023-10-16T15:53:11.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.365+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8ef8047460 con 0x7f8f001e6530 2023-10-16T15:53:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.366+0000 7f8efd7fa640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8ee006af50 0x7f8ee006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.366+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f8ef80aafe0 con 0x7f8f001e6530 2023-10-16T15:53:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.367+0000 7f8efffff640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8ee006af50 0x7f8ee006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.368+0000 7f8efffff640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8ee006af50 0x7f8ee006d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0001cd0 tx=0x7f8ef0009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.371+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ef8075df0 con 0x7f8f001e6530 2023-10-16T15:53:11.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.605+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f0004e9b0 con 0x7f8f001e6530 2023-10-16T15:53:11.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.606+0000 7f8efd7fa640 1 -- 172.21.15.29:0/3383346862 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f8ef8079410 con 0x7f8f001e6530 2023-10-16T15:53:11.607 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:11.607 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:50:44.088346+0000","modified":"2023-10-16T15:50:47.188096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8407":{"gid":8407,"name":"c","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6833/2149952636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2149952636},{"type":"v1","addr":"172.21.15.29:6833","nonce":2149952636}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T15:53:11.607 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-16T15:53:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.608+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8ee006af50 msgr2=0x7f8ee006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.608+0000 7f8f05002640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8ee006af50 0x7f8ee006d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0001cd0 tx=0x7f8ef0009040 comp rx=0 tx=0).stop 2023-10-16T15:53:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.608+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 msgr2=0x7f8f001e69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.608+0000 7f8f05002640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 0x7f8f001e69b0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8037040 tx=0x7f8ef8009750 comp rx=0 tx=0).stop 2023-10-16T15:53:11.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.608+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 shutdown_connections 2023-10-16T15:53:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8ee006af50 0x7f8ee006d410 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f001e6530 0x7f8f001e69b0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f0015e8f0 0x7f8f0015ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 --2- 172.21.15.29:0/3383346862 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eec004d00 0x7f8f0014fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 >> 172.21.15.29:0/3383346862 conn(0x7f8f000cf570 msgr2=0x7f8f001520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:11.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 shutdown_connections 2023-10-16T15:53:11.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.609+0000 7f8f05002640 1 -- 172.21.15.29:0/3383346862 wait complete. 2023-10-16T15:53:11.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.853+0000 7f76b0f4e640 1 Processor -- start 2023-10-16T15:53:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.853+0000 7f76b0f4e640 1 -- start start 2023-10-16T15:53:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.855+0000 7f76b0f4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.855+0000 7f76b0f4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f76ac070f20 0x7f76ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.855+0000 7f76b0f4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ac160a90 con 0x7f76ac15c9f0 2023-10-16T15:53:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.855+0000 7f76b0f4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac0cdec0 con 0x7f76ac070f20 2023-10-16T15:53:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.855+0000 7f76b0f4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac0cd900 con 0x7f76ac15dde0 2023-10-16T15:53:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.856+0000 7f76ab7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76ac15c9f0 0x7f76ac0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41170/0 (socket says 172.21.15.29:41170) 2023-10-16T15:53:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.856+0000 7f76ab7fe640 1 -- 172.21.15.29:0/1349884053 learned_addr learned my addr 172.21.15.29:0/1349884053 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.856+0000 7f76abfff640 1 --2- 172.21.15.29:0/1349884053 >> v2:172.21.15.170:3300/0 conn(0x7f76ac070f20 0x7f76ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.856+0000 7f76a3fff640 1 --2- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.856+0000 7f76aaffd640 1 -- 172.21.15.29:0/1349884053 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102074211 0 0) 0x7f76ac160a90 con 0x7f76ac15c9f0 2023-10-16T15:53:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.856+0000 7f76aaffd640 1 -- 172.21.15.29:0/1349884053 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698003520 con 0x7f76ac15c9f0 2023-10-16T15:53:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.857+0000 7f76aaffd640 1 -- 172.21.15.29:0/1349884053 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106306647 0 0) 0x7f7698003520 con 0x7f76ac15c9f0 2023-10-16T15:53:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.857+0000 7f76abfff640 1 -- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 msgr2=0x7f76ac0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.857+0000 7f76abfff640 1 --2- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.857+0000 7f76abfff640 1 -- 172.21.15.29:0/1349884053 >> v1:172.21.15.29:6789/0 conn(0x7f76ac15c9f0 legacy=0x7f76ac0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.857+0000 7f76abfff640 1 -- 172.21.15.29:0/1349884053 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac0cdbe0 con 0x7f76ac070f20 2023-10-16T15:53:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.858+0000 7f76abfff640 1 --2- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac0ac890 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009d90 tx=0x7f76a402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4128944499d2553 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76aaffd640 1 -- 172.21.15.29:0/1349884053 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c001670 con 0x7f76ac070f20 2023-10-16T15:53:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76aaffd640 1 -- 172.21.15.29:0/1349884053 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a402c8a0 con 0x7f76ac070f20 2023-10-16T15:53:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76aaffd640 1 -- 172.21.15.29:0/1349884053 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a402cc30 con 0x7f76ac070f20 2023-10-16T15:53:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 -- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 msgr2=0x7f76ac0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac0ac890 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009d90 tx=0x7f76a402bd70 comp rx=0 tx=0).stop 2023-10-16T15:53:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 -- 172.21.15.29:0/1349884053 shutdown_connections 2023-10-16T15:53:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac0ac890 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/1349884053 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 -- 172.21.15.29:0/1349884053 >> 172.21.15.29:0/1349884053 conn(0x7f76ac0ce6a0 msgr2=0x7f76ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:11.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 -- 172.21.15.29:0/1349884053 shutdown_connections 2023-10-16T15:53:11.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.859+0000 7f76b0f4e640 1 -- 172.21.15.29:0/1349884053 wait complete. 2023-10-16T15:53:11.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.861+0000 7f76b0f4e640 1 Processor -- start 2023-10-16T15:53:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.861+0000 7f76b0f4e640 1 -- start start 2023-10-16T15:53:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.862+0000 7f76b0f4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.862+0000 7f76b0f4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 0x7f76ac0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.862+0000 7f76b0f4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac15f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.862+0000 7f76b0f4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac1435f0 con 0x7f76ac15c9f0 2023-10-16T15:53:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.862+0000 7f76b0f4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac145820 con 0x7f76ac070f20 2023-10-16T15:53:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.862+0000 7f76b0f4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac145b00 con 0x7f76ac15dde0 2023-10-16T15:53:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 0x7f76ac0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 0x7f76ac0ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33820/0 (socket says 172.21.15.29:33820) 2023-10-16T15:53:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 -- 172.21.15.29:0/460694952 learned_addr learned my addr 172.21.15.29:0/460694952 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76abfff640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac15f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76ab7fe640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 -- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 msgr2=0x7f76ac15f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac15f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 -- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 msgr2=0x7f76ac144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76a3fff640 1 -- 172.21.15.29:0/460694952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4009a70 con 0x7f76ac15c9f0 2023-10-16T15:53:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76abfff640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac15f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.863+0000 7f76ab7fe640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:11.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.864+0000 7f76a3fff640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 0x7f76ac0ccfd0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f769400e980 tx=0x7f769400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:11.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.864+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694009670 con 0x7f76ac15c9f0 2023-10-16T15:53:11.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.864+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769400ca00 con 0x7f76ac15c9f0 2023-10-16T15:53:11.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.864+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769400cd90 con 0x7f76ac15c9f0 2023-10-16T15:53:11.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.864+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac145de0 con 0x7f76ac15c9f0 2023-10-16T15:53:11.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.864+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac1463a0 con 0x7f76ac15c9f0 2023-10-16T15:53:11.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.865+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac04e9b0 con 0x7f76ac15c9f0 2023-10-16T15:53:11.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.866+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f7694013470 con 0x7f76ac15c9f0 2023-10-16T15:53:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.868+0000 7f76a8ff9640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f768c06af00 0x7f768c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.868+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f7694089c60 con 0x7f76ac15c9f0 2023-10-16T15:53:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.868+0000 7f76ab7fe640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f768c06af00 0x7f768c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:11.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.869+0000 7f76ab7fe640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f768c06af00 0x7f768c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f769c0011e0 tx=0x7f769c009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:11.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:11.872+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f76940548e0 con 0x7f76ac15c9f0 2023-10-16T15:53:12.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.066+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76ac0701b0 con 0x7f76ac15c9f0 2023-10-16T15:53:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.069+0000 7f76a8ff9640 1 -- 172.21.15.29:0/460694952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+17789 (secure 0 0 0) 0x7f7694057f00 con 0x7f76ac15c9f0 2023-10-16T15:53:12.070 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:12.071 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":213,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:08.409916+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:50:42.031547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T15:50:43.111119+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":213,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.24","mappings":[{"from":0,"to":1}]},{"pgid":"25.2a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6833/928368069":"2023-10-17T15:53:08.409834+0000","172.21.15.170:6832/928368069":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/1186413715":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/3227854992":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/568891934":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/2703119246":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/1539386380":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/3252182417":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/423324657":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/3913323747":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2674320589":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/275392394":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6833/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6832/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2170359614":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/1066951432":"2023-10-17T15:50:59.297259+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:12.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.071+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f768c06af00 msgr2=0x7f768c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f768c06af00 0x7f768c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f769c0011e0 tx=0x7f769c009040 comp rx=0 tx=0).stop 2023-10-16T15:53:12.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 msgr2=0x7f76ac0ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 0x7f76ac0ccfd0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f769400e980 tx=0x7f769400ee40 comp rx=0 tx=0).stop 2023-10-16T15:53:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 shutdown_connections 2023-10-16T15:53:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f768c06af00 0x7f768c06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76ac15c9f0 0x7f76ac0ccfd0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f76ac15dde0 0x7f76ac15f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 --2- 172.21.15.29:0/460694952 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f76ac070f20 0x7f76ac144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 >> 172.21.15.29:0/460694952 conn(0x7f76ac0ce6a0 msgr2=0x7f76ac15b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 shutdown_connections 2023-10-16T15:53:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.072+0000 7f76b0f4e640 1 -- 172.21.15.29:0/460694952 wait complete. 2023-10-16T15:53:12.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:53:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.326+0000 7f41a2413640 1 Processor -- start 2023-10-16T15:53:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.326+0000 7f41a2413640 1 -- start start 2023-10-16T15:53:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.327+0000 7f41a2413640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c06c040 0x7f419c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a2413640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f419c0ad970 0x7f419c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a2413640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f419c15ebc0 0x7f419c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a2413640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c06c960 con 0x7f419c15ebc0 2023-10-16T15:53:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a2413640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c15b9d0 con 0x7f419c0ad970 2023-10-16T15:53:12.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a2413640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c15bcb0 con 0x7f419c06c040 2023-10-16T15:53:12.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a1c12640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f419c15ebc0 0x7f419c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a1c12640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f419c15ebc0 0x7f419c15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33832/0 (socket says 172.21.15.29:33832) 2023-10-16T15:53:12.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a1c12640 1 -- 172.21.15.29:0/3983658074 learned_addr learned my addr 172.21.15.29:0/3983658074 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a1411640 1 --2- 172.21.15.29:0/3983658074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c06c040 0x7f419c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.328+0000 7f41a0c10640 1 --2- 172.21.15.29:0/3983658074 >> v2:172.21.15.170:3300/0 conn(0x7f419c0ad970 0x7f419c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.329+0000 7f41a1c12640 1 -- 172.21.15.29:0/3983658074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c06c040 msgr2=0x7f419c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.329+0000 7f41a1c12640 1 --2- 172.21.15.29:0/3983658074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c06c040 0x7f419c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.329+0000 7f41a1c12640 1 -- 172.21.15.29:0/3983658074 >> v2:172.21.15.170:3300/0 conn(0x7f419c0ad970 msgr2=0x7f419c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.329+0000 7f41a1c12640 1 --2- 172.21.15.29:0/3983658074 >> v2:172.21.15.170:3300/0 conn(0x7f419c0ad970 0x7f419c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.329+0000 7f41a1c12640 1 -- 172.21.15.29:0/3983658074 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c15bf90 con 0x7f419c15ebc0 2023-10-16T15:53:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.329+0000 7f41a0c10640 1 --2- 172.21.15.29:0/3983658074 >> v2:172.21.15.170:3300/0 conn(0x7f419c0ad970 0x7f419c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.330+0000 7f41a1c12640 1 --2- 172.21.15.29:0/3983658074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c15ebc0 0x7f419c15f050 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f4194009e50 tx=0x7f419402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdeeecae2a318c8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.330+0000 7f4193fff640 1 -- 172.21.15.29:0/3983658074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419402ca80 con 0x7f419c15ebc0 2023-10-16T15:53:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.330+0000 7f4193fff640 1 -- 172.21.15.29:0/3983658074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419402cbe0 con 0x7f419c15ebc0 2023-10-16T15:53:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.330+0000 7f4193fff640 1 -- 172.21.15.29:0/3983658074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41940343f0 con 0x7f419c15ebc0 2023-10-16T15:53:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 -- 172.21.15.29:0/3983658074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c15ebc0 msgr2=0x7f419c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 --2- 172.21.15.29:0/3983658074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c15ebc0 0x7f419c15f050 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f4194009e50 tx=0x7f419402bee0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 -- 172.21.15.29:0/3983658074 shutdown_connections 2023-10-16T15:53:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 --2- 172.21.15.29:0/3983658074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c15ebc0 0x7f419c15f050 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 --2- 172.21.15.29:0/3983658074 >> v2:172.21.15.170:3300/0 conn(0x7f419c0ad970 0x7f419c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 --2- 172.21.15.29:0/3983658074 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c06c040 0x7f419c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 -- 172.21.15.29:0/3983658074 >> 172.21.15.29:0/3983658074 conn(0x7f419c0ce6a0 msgr2=0x7f419c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 -- 172.21.15.29:0/3983658074 shutdown_connections 2023-10-16T15:53:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.331+0000 7f41a2413640 1 -- 172.21.15.29:0/3983658074 wait complete. 2023-10-16T15:53:12.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.333+0000 7f41a2413640 1 Processor -- start 2023-10-16T15:53:12.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.333+0000 7f41a2413640 1 -- start start 2023-10-16T15:53:12.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.334+0000 7f41a2413640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 0x7f419c156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.334+0000 7f41a2413640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c0ad970 0x7f419c0c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.334+0000 7f41a2413640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f419c15ebc0 0x7f419c157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.334+0000 7f41a2413640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0ae470 con 0x7f419c06c040 2023-10-16T15:53:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.334+0000 7f41a2413640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0ae750 con 0x7f419c15ebc0 2023-10-16T15:53:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.334+0000 7f41a2413640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0aea30 con 0x7f419c0ad970 2023-10-16T15:53:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 0x7f419c156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 0x7f419c156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33840/0 (socket says 172.21.15.29:33840) 2023-10-16T15:53:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 -- 172.21.15.29:0/322749878 learned_addr learned my addr 172.21.15.29:0/322749878 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 -- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c0ad970 msgr2=0x7f419c0c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c0ad970 0x7f419c0c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 -- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f419c15ebc0 msgr2=0x7f419c157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f419c15ebc0 0x7f419c157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 -- 172.21.15.29:0/322749878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4194009b30 con 0x7f419c06c040 2023-10-16T15:53:12.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.335+0000 7f41a1411640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 0x7f419c156ec0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f419800d800 tx=0x7f419800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:12.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.336+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41980096a0 con 0x7f419c06c040 2023-10-16T15:53:12.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.336+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4198009800 con 0x7f419c06c040 2023-10-16T15:53:12.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.336+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c0aed10 con 0x7f419c06c040 2023-10-16T15:53:12.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.336+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198009990 con 0x7f419c06c040 2023-10-16T15:53:12.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.336+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c15d980 con 0x7f419c06c040 2023-10-16T15:53:12.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.337+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c06b330 con 0x7f419c06c040 2023-10-16T15:53:12.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.338+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4198009af0 con 0x7f419c06c040 2023-10-16T15:53:12.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.340+0000 7f4191ffb640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f418006af00 0x7f418006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.340+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f4198089f30 con 0x7f419c06c040 2023-10-16T15:53:12.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.340+0000 7f41a0c10640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f418006af00 0x7f418006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.342+0000 7f41a0c10640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f418006af00 0x7f418006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f418c006fd0 tx=0x7f418c008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:12.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.345+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4198054bd0 con 0x7f419c06c040 2023-10-16T15:53:12.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.584+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f419c04e9b0 con 0x7f419c06c040 2023-10-16T15:53:12.586 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:53:12.586+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:53:12.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.590+0000 7f4191ffb640 1 -- 172.21.15.29:0/322749878 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v101) v1 ==== 161+0+0 (secure 0 0 0) 0x7f41980581f0 con 0x7f419c06c040 2023-10-16T15:53:12.591 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:53:12.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.594+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f418006af00 msgr2=0x7f418006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.594+0000 7f41a2413640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f418006af00 0x7f418006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f418c006fd0 tx=0x7f418c008040 comp rx=0 tx=0).stop 2023-10-16T15:53:12.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.594+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 msgr2=0x7f419c156ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.594+0000 7f41a2413640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 0x7f419c156ec0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f419800d800 tx=0x7f419800dcc0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.595+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 shutdown_connections 2023-10-16T15:53:12.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.595+0000 7f41a2413640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f418006af00 0x7f418006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.595+0000 7f41a2413640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f419c15ebc0 0x7f419c157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.595+0000 7f41a2413640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f419c0ad970 0x7f419c0c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.595+0000 7f41a2413640 1 --2- 172.21.15.29:0/322749878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f419c06c040 0x7f419c156ec0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.595+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 >> 172.21.15.29:0/322749878 conn(0x7f419c0ce6a0 msgr2=0x7f419c0c89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:12.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.598+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 shutdown_connections 2023-10-16T15:53:12.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.598+0000 7f41a2413640 1 -- 172.21.15.29:0/322749878 wait complete. 2023-10-16T15:53:12.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:53:12.666 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:53:12.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.811+0000 7f5d261d0640 1 Processor -- start 2023-10-16T15:53:12.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.812+0000 7f5d261d0640 1 -- start start 2023-10-16T15:53:12.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.813+0000 7f5d261d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 0x7f5d200b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.813+0000 7f5d261d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d200b1720 con 0x7f5d20159890 2023-10-16T15:53:12.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.813+0000 7f5d261d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d200b1a00 con 0x7f5d20071280 2023-10-16T15:53:12.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.813+0000 7f5d261d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200b1ce0 con 0x7f5d200ccac0 2023-10-16T15:53:12.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d251ce640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5d20159890 0x7f5d20070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41184/0 (socket says 172.21.15.29:41184) 2023-10-16T15:53:12.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d251ce640 1 -- 172.21.15.29:0/308257243 learned_addr learned my addr 172.21.15.29:0/308257243 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:12.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227530413 0 0) 0x7f5d200b1720 con 0x7f5d20159890 2023-10-16T15:53:12.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d259cf640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 0x7f5d200b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d0c005730 con 0x7f5d20159890 2023-10-16T15:53:12.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559402762 0 0) 0x7f5d0c005730 con 0x7f5d20159890 2023-10-16T15:53:12.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 msgr2=0x7f5d200b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:12.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 0x7f5d200b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 >> v1:172.21.15.170:6789/0 conn(0x7f5d20071280 legacy=0x7f5d200afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d200b1fc0 con 0x7f5d20159890 2023-10-16T15:53:12.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.814+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5d1c001670 con 0x7f5d20159890 2023-10-16T15:53:12.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 >> v1:172.21.15.29:6789/0 conn(0x7f5d20159890 legacy=0x7f5d20070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 0x7f5d0c005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d0c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d0c006f10 0x7f5d0c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200b1ce0 con 0x7f5d0c006230 2023-10-16T15:53:12.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c00ddd0 con 0x7f5d0c006f10 2023-10-16T15:53:12.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c00df90 con 0x7f5d200ccac0 2023-10-16T15:53:12.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d249cd640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d0c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d251ce640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 0x7f5d0c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d249cd640 1 -- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 msgr2=0x7f5d0c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:12.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d249cd640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d200ccac0 0x7f5d0c005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d259cf640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d0c006f10 0x7f5d0c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d249cd640 1 -- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d0c006f10 msgr2=0x7f5d0c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d249cd640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d0c006f10 0x7f5d0c007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.815+0000 7f5d249cd640 1 -- 172.21.15.29:0/308257243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c005730 con 0x7f5d0c006230 2023-10-16T15:53:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d249cd640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d0c0069d0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f5d10001720 tx=0x7f5d0c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6670b67fc9f9b0ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d100052d0 con 0x7f5d0c006230 2023-10-16T15:53:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1002e050 con 0x7f5d0c006230 2023-10-16T15:53:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d17fff640 1 -- 172.21.15.29:0/308257243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10003830 con 0x7f5d0c006230 2023-10-16T15:53:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d261d0640 1 -- 172.21.15.29:0/308257243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 msgr2=0x7f5d0c0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d261d0640 1 --2- 172.21.15.29:0/308257243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d0c0069d0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f5d10001720 tx=0x7f5d0c007a00 comp rx=0 tx=0).stop 2023-10-16T15:53:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.817+0000 7f5d259cf640 1 -- 172.21.15.29:0/308257243 reap_dead start 2023-10-16T15:53:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.818+0000 7f5d261d0640 1 -- 172.21.15.29:0/308257243 shutdown_connections 2023-10-16T15:53:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.818+0000 7f5d261d0640 1 -- 172.21.15.29:0/308257243 >> 172.21.15.29:0/308257243 conn(0x7f5d20069da0 msgr2=0x7f5d2006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.818+0000 7f5d261d0640 1 -- 172.21.15.29:0/308257243 shutdown_connections 2023-10-16T15:53:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.818+0000 7f5d261d0640 1 -- 172.21.15.29:0/308257243 wait complete. 2023-10-16T15:53:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.819+0000 7f5d261d0640 1 Processor -- start 2023-10-16T15:53:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.820+0000 7f5d261d0640 1 -- start start 2023-10-16T15:53:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.820+0000 7f5d261d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d200a8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d261d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d20151930 0x7f5d200a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d261d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d20143fc0 0x7f5d20153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d261d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d20151db0 con 0x7f5d0c006230 2023-10-16T15:53:12.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d261d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d20150b70 con 0x7f5d20143fc0 2023-10-16T15:53:12.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d261d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d20150e50 con 0x7f5d20151930 2023-10-16T15:53:12.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d200a8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d200a8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33862/0 (socket says 172.21.15.29:33862) 2023-10-16T15:53:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 -- 172.21.15.29:0/4090434527 learned_addr learned my addr 172.21.15.29:0/4090434527 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d249cd640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d20151930 0x7f5d200a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 -- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d20151930 msgr2=0x7f5d200a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d259cf640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d20143fc0 0x7f5d20153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d20151930 0x7f5d200a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 -- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d20143fc0 msgr2=0x7f5d20153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d20143fc0 0x7f5d20153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d251ce640 1 -- 172.21.15.29:0/4090434527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d20151130 con 0x7f5d0c006230 2023-10-16T15:53:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.821+0000 7f5d249cd640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d20151930 0x7f5d200a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.822+0000 7f5d251ce640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d200a8500 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00e730 tx=0x7f5d1c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.822+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00ee40 con 0x7f5d0c006230 2023-10-16T15:53:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.822+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c00c850 con 0x7f5d0c006230 2023-10-16T15:53:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.822+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00cbe0 con 0x7f5d0c006230 2023-10-16T15:53:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.822+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d20151410 con 0x7f5d0c006230 2023-10-16T15:53:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.822+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d200a9610 con 0x7f5d0c006230 2023-10-16T15:53:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.823+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d20150cb0 con 0x7f5d0c006230 2023-10-16T15:53:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.824+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5d1c00cd40 con 0x7f5d0c006230 2023-10-16T15:53:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.825+0000 7f5d15ffb640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5d0406af00 0x7f5d0406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.825+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f5d1c0899d0 con 0x7f5d0c006230 2023-10-16T15:53:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.826+0000 7f5d249cd640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5d0406af00 0x7f5d0406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.827+0000 7f5d249cd640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5d0406af00 0x7f5d0406d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5d10033000 tx=0x7f5d10009030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:12.830+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5d20150cb0 con 0x7f5d0c006230 2023-10-16T15:53:13.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.068+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5d20150cb0 con 0x7f5d0c006230 2023-10-16T15:53:13.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.242+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d1c054a90 con 0x7f5d0c006230 2023-10-16T15:53:13.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.593+0000 7f5d15ffb640 1 -- 172.21.15.29:0/4090434527 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v102) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5d1c087830 con 0x7f5d0c006230 2023-10-16T15:53:13.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5d0406af00 msgr2=0x7f5d0406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:13.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5d0406af00 0x7f5d0406d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5d10033000 tx=0x7f5d10009030 comp rx=0 tx=0).stop 2023-10-16T15:53:13.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 msgr2=0x7f5d200a8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:13.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d200a8500 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00e730 tx=0x7f5d1c00ebf0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 shutdown_connections 2023-10-16T15:53:13.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5d0406af00 0x7f5d0406d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5d20143fc0 0x7f5d20153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5d20151930 0x7f5d200a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.597+0000 7f5d261d0640 1 --2- 172.21.15.29:0/4090434527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0c006230 0x7f5d200a8500 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.598+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 >> 172.21.15.29:0/4090434527 conn(0x7f5d20069da0 msgr2=0x7f5d2006c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:13.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.598+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 shutdown_connections 2023-10-16T15:53:13.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.598+0000 7f5d261d0640 1 -- 172.21.15.29:0/4090434527 wait complete. 2023-10-16T15:53:13.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:53:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.849+0000 7ff95ebbd640 1 Processor -- start 2023-10-16T15:53:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.850+0000 7ff95ebbd640 1 -- start start 2023-10-16T15:53:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.851+0000 7ff95ebbd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 0x7ff95806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.851+0000 7ff95ebbd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff9580cda80 0x7ff95815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.851+0000 7ff95ebbd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95806c960 con 0x7ff9580ad970 2023-10-16T15:53:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.851+0000 7ff95ebbd640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95815b9d0 con 0x7ff9580cda80 2023-10-16T15:53:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.851+0000 7ff95ebbd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95815bcb0 con 0x7ff95806c040 2023-10-16T15:53:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.852+0000 7ff95d3ba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff9580ad970 0x7ff95815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41194/0 (socket says 172.21.15.29:41194) 2023-10-16T15:53:13.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.852+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3933139884 learned_addr learned my addr 172.21.15.29:0/3933139884 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:13.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.852+0000 7ff95e3bc640 1 --2- 172.21.15.29:0/3933139884 >> v2:172.21.15.170:3300/0 conn(0x7ff9580cda80 0x7ff95815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.852+0000 7ff95dbbb640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 0x7ff95806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.852+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014012245 0 0) 0x7ff95806c960 con 0x7ff9580ad970 2023-10-16T15:53:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.852+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff944003520 con 0x7ff9580ad970 2023-10-16T15:53:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2247657298 0 0) 0x7ff944003520 con 0x7ff9580ad970 2023-10-16T15:53:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 msgr2=0x7ff95806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 0x7ff95806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 >> v2:172.21.15.170:3300/0 conn(0x7ff9580cda80 msgr2=0x7ff95815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 --2- 172.21.15.29:0/3933139884 >> v2:172.21.15.170:3300/0 conn(0x7ff9580cda80 0x7ff95815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95815bf90 con 0x7ff9580ad970 2023-10-16T15:53:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7ff948001670 con 0x7ff9580ad970 2023-10-16T15:53:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 >> v1:172.21.15.29:6789/0 conn(0x7ff9580ad970 legacy=0x7ff95815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 0x7ff944003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff944003f40 0x7ff9440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff944004d00 0x7ff944005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95815b9d0 con 0x7ff944003f40 2023-10-16T15:53:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95815bcb0 con 0x7ff944004d00 2023-10-16T15:53:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400ba30 con 0x7ff95806c040 2023-10-16T15:53:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.853+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff944003f40 0x7ff9440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95dbbb640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 0x7ff944003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95e3bc640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff944004d00 0x7ff944005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 msgr2=0x7ff944003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff95806c040 0x7ff944003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff944004d00 msgr2=0x7ff944005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff944004d00 0x7ff944005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.854+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3933139884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944003520 con 0x7ff944003f40 2023-10-16T15:53:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.855+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff944003f40 0x7ff9440047c0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7ff94800c110 tx=0x7ff9440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82a0db3c12ee2b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.855+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948005ad0 con 0x7ff944003f40 2023-10-16T15:53:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.855+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff948005c50 con 0x7ff944003f40 2023-10-16T15:53:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.855+0000 7ff95cbb9640 1 -- 172.21.15.29:0/3933139884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94802c830 con 0x7ff944003f40 2023-10-16T15:53:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3933139884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff944003f40 msgr2=0x7ff9440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:13.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3933139884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff944003f40 0x7ff9440047c0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7ff94800c110 tx=0x7ff9440057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95e3bc640 1 -- 172.21.15.29:0/3933139884 reap_dead start 2023-10-16T15:53:13.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3933139884 shutdown_connections 2023-10-16T15:53:13.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3933139884 >> 172.21.15.29:0/3933139884 conn(0x7ff9580ce6a0 msgr2=0x7ff9580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3933139884 shutdown_connections 2023-10-16T15:53:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.856+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3933139884 wait complete. 2023-10-16T15:53:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.858+0000 7ff95ebbd640 1 Processor -- start 2023-10-16T15:53:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.858+0000 7ff95ebbd640 1 -- start start 2023-10-16T15:53:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95ebbd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff944003f40 0x7ff958150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95ebbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 0x7ff9580b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95ebbd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9580b2ba0 0x7ff958155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95ebbd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c58a0 con 0x7ff9580ae3f0 2023-10-16T15:53:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95ebbd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c5b40 con 0x7ff9580b2ba0 2023-10-16T15:53:13.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95ebbd640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c5e20 con 0x7ff944003f40 2023-10-16T15:53:13.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95d3ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 0x7ff9580b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95d3ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 0x7ff9580b2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33872/0 (socket says 172.21.15.29:33872) 2023-10-16T15:53:13.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.859+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3770976997 learned_addr learned my addr 172.21.15.29:0/3770976997 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:13.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95dbbb640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff944003f40 0x7ff958150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95e3bc640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9580b2ba0 0x7ff958155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff944003f40 msgr2=0x7ff958150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:13.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff944003f40 0x7ff958150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9580b2ba0 msgr2=0x7ff958155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:13.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9580b2ba0 0x7ff958155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:13.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95d3ba640 1 -- 172.21.15.29:0/3770976997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580c6100 con 0x7ff9580ae3f0 2023-10-16T15:53:13.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95e3bc640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9580b2ba0 0x7ff958155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:13.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.860+0000 7ff95d3ba640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 0x7ff9580b2660 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7ff94802c3d0 tx=0x7ff948001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:13.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.861+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948039070 con 0x7ff9580ae3f0 2023-10-16T15:53:13.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.861+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94802dcd0 con 0x7ff9580ae3f0 2023-10-16T15:53:13.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.861+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948007270 con 0x7ff9580ae3f0 2023-10-16T15:53:13.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.861+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9580c63e0 con 0x7ff9580ae3f0 2023-10-16T15:53:13.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.861+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9580b37e0 con 0x7ff9580ae3f0 2023-10-16T15:53:13.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.862+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95806b330 con 0x7ff9580ae3f0 2023-10-16T15:53:13.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.863+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff948032070 con 0x7ff9580ae3f0 2023-10-16T15:53:13.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.864+0000 7ff94e7fc640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff93c06afc0 0x7ff93c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:13.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.864+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6569+0+0 (secure 0 0 0) 0x7ff94803c040 con 0x7ff9580ae3f0 2023-10-16T15:53:13.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.865+0000 7ff95dbbb640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff93c06afc0 0x7ff93c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:13.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.866+0000 7ff95dbbb640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff93c06afc0 0x7ff93c06d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff954006fd0 tx=0x7ff954008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:13.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:13.869+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff948075ac0 con 0x7ff9580ae3f0 2023-10-16T15:53:14.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.067+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff9580b39d0 con 0x7ff9580ae3f0 2023-10-16T15:53:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.603+0000 7ff94e7fc640 1 -- 172.21.15.29:0/3770976997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v216) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff9580b39d0 con 0x7ff9580ae3f0 2023-10-16T15:53:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:53:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.609+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff93c06afc0 msgr2=0x7ff93c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.609+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff93c06afc0 0x7ff93c06d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff954006fd0 tx=0x7ff954008040 comp rx=0 tx=0).stop 2023-10-16T15:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.609+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 msgr2=0x7ff9580b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.609+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 0x7ff9580b2660 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7ff94802c3d0 tx=0x7ff948001930 comp rx=0 tx=0).stop 2023-10-16T15:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 shutdown_connections 2023-10-16T15:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff93c06afc0 0x7ff93c06d480 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9580ae3f0 0x7ff9580b2660 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff9580b2ba0 0x7ff958155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 --2- 172.21.15.29:0/3770976997 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff944003f40 0x7ff958150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 >> 172.21.15.29:0/3770976997 conn(0x7ff9580ce6a0 msgr2=0x7ff958159c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 shutdown_connections 2023-10-16T15:53:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.610+0000 7ff95ebbd640 1 -- 172.21.15.29:0/3770976997 wait complete. 2023-10-16T15:53:14.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:53:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.857+0000 7f86892d4640 1 Processor -- start 2023-10-16T15:53:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.857+0000 7f86892d4640 1 -- start start 2023-10-16T15:53:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.858+0000 7f86892d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 0x7f868406f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.858+0000 7f86892d4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f86840b0360 0x7f86840b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.858+0000 7f86892d4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86840c9620 con 0x7f868406f9d0 2023-10-16T15:53:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.858+0000 7f86892d4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868415d340 con 0x7f86840b0360 2023-10-16T15:53:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.858+0000 7f86892d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840cc860 con 0x7f868415cf60 2023-10-16T15:53:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.859+0000 7f8682ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 0x7f868406f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.859+0000 7f8682ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 0x7f868406f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:54834/0 (socket says 172.21.15.29:54834) 2023-10-16T15:53:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.859+0000 7f8682ffd640 1 -- 172.21.15.29:0/446987980 learned_addr learned my addr 172.21.15.29:0/446987980 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.859+0000 7f86837fe640 1 --2- 172.21.15.29:0/446987980 >> v2:172.21.15.170:3300/0 conn(0x7f86840b0360 0x7f86840b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.860+0000 7f8682ffd640 1 -- 172.21.15.29:0/446987980 >> v2:172.21.15.170:3300/0 conn(0x7f86840b0360 msgr2=0x7f86840b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.860+0000 7f8682ffd640 1 --2- 172.21.15.29:0/446987980 >> v2:172.21.15.170:3300/0 conn(0x7f86840b0360 0x7f86840b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.860+0000 7f8682ffd640 1 -- 172.21.15.29:0/446987980 >> v1:172.21.15.170:6789/0 conn(0x7f868406f9d0 legacy=0x7f86840c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.860+0000 7f8682ffd640 1 -- 172.21.15.29:0/446987980 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86840ad4a0 con 0x7f868415cf60 2023-10-16T15:53:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.861+0000 7f8682ffd640 1 --2- 172.21.15.29:0/446987980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 0x7f868406f490 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8674009870 tx=0x7f867402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d600e3885209e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.862+0000 7f8681ffb640 1 -- 172.21.15.29:0/446987980 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674030840 con 0x7f868415cf60 2023-10-16T15:53:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.862+0000 7f8681ffb640 1 -- 172.21.15.29:0/446987980 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86740309a0 con 0x7f868415cf60 2023-10-16T15:53:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.862+0000 7f8681ffb640 1 -- 172.21.15.29:0/446987980 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674030d30 con 0x7f868415cf60 2023-10-16T15:53:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 -- 172.21.15.29:0/446987980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 msgr2=0x7f868406f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 --2- 172.21.15.29:0/446987980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 0x7f868406f490 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8674009870 tx=0x7f867402b980 comp rx=0 tx=0).stop 2023-10-16T15:53:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 -- 172.21.15.29:0/446987980 shutdown_connections 2023-10-16T15:53:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 --2- 172.21.15.29:0/446987980 >> v2:172.21.15.170:3300/0 conn(0x7f86840b0360 0x7f86840b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 --2- 172.21.15.29:0/446987980 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f868415cf60 0x7f868406f490 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 -- 172.21.15.29:0/446987980 >> 172.21.15.29:0/446987980 conn(0x7f86840cb360 msgr2=0x7f86840cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 -- 172.21.15.29:0/446987980 shutdown_connections 2023-10-16T15:53:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.863+0000 7f86892d4640 1 -- 172.21.15.29:0/446987980 wait complete. 2023-10-16T15:53:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.865+0000 7f86892d4640 1 Processor -- start 2023-10-16T15:53:14.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.865+0000 7f86892d4640 1 -- start start 2023-10-16T15:53:14.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.866+0000 7f86892d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f868406f9d0 0x7f86840c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:14.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.866+0000 7f86892d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86840b0360 0x7f86841f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:14.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.866+0000 7f86892d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 0x7f86841f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:14.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.866+0000 7f86892d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8684159900 con 0x7f868415cf60 2023-10-16T15:53:14.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.866+0000 7f86892d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86841f3810 con 0x7f868406f9d0 2023-10-16T15:53:14.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.866+0000 7f86892d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86841f3af0 con 0x7f86840b0360 2023-10-16T15:53:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 0x7f86841f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 0x7f86841f6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53588/0 (socket says 172.21.15.29:53588) 2023-10-16T15:53:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 -- 172.21.15.29:0/2290790725 learned_addr learned my addr 172.21.15.29:0/2290790725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86827fc640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86840b0360 0x7f86841f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f8682ffd640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f868406f9d0 0x7f86840c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 -- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86840b0360 msgr2=0x7f86841f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86840b0360 0x7f86841f5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 -- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f868406f9d0 msgr2=0x7f86840c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f868406f9d0 0x7f86840c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.867+0000 7f86837fe640 1 -- 172.21.15.29:0/2290790725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86740094d0 con 0x7f868415cf60 2023-10-16T15:53:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.868+0000 7f86837fe640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 0x7f86841f6180 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f867c00efc0 tx=0x7f867c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.868+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c009ba0 con 0x7f868415cf60 2023-10-16T15:53:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.868+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00f040 con 0x7f868415cf60 2023-10-16T15:53:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.868+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c0146c0 con 0x7f868415cf60 2023-10-16T15:53:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.868+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86840c6e60 con 0x7f868415cf60 2023-10-16T15:53:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.868+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86840c7420 con 0x7f868415cf60 2023-10-16T15:53:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.869+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86841f3c60 con 0x7f868415cf60 2023-10-16T15:53:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.870+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f867c014820 con 0x7f868415cf60 2023-10-16T15:53:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.871+0000 7f867b7fe640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f866406af90 0x7f866406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:14.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.872+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6131+0+0 (secure 0 0 0) 0x7f867c08a5d0 con 0x7f868415cf60 2023-10-16T15:53:14.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.872+0000 7f8682ffd640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f866406af90 0x7f866406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:14.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.873+0000 7f8682ffd640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f866406af90 0x7f866406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8674036000 tx=0x7f867403b040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:14.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:14.876+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f867c055210 con 0x7f868415cf60 2023-10-16T15:53:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.071+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f868404e160 con 0x7f868415cf60 2023-10-16T15:53:15.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.616+0000 7f867b7fe640 1 -- 172.21.15.29:0/2290790725 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v217) v1 ==== 169+0+0 (secure 0 0 0) 0x7f867c058830 con 0x7f868415cf60 2023-10-16T15:53:15.617 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:53:15.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.621+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f866406af90 msgr2=0x7f866406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:15.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.621+0000 7f86892d4640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f866406af90 0x7f866406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8674036000 tx=0x7f867403b040 comp rx=0 tx=0).stop 2023-10-16T15:53:15.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.621+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 msgr2=0x7f86841f6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:15.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.621+0000 7f86892d4640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 0x7f86841f6180 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f867c00efc0 tx=0x7f867c00c490 comp rx=0 tx=0).stop 2023-10-16T15:53:15.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 shutdown_connections 2023-10-16T15:53:15.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f866406af90 0x7f866406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f868415cf60 0x7f86841f6180 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f86840b0360 0x7f86841f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 --2- 172.21.15.29:0/2290790725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f868406f9d0 0x7f86840c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 >> 172.21.15.29:0/2290790725 conn(0x7f86840cb360 msgr2=0x7f86840ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:15.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 shutdown_connections 2023-10-16T15:53:15.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.622+0000 7f86892d4640 1 -- 172.21.15.29:0/2290790725 wait complete. 2023-10-16T15:53:15.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:53:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.869+0000 7fc8822ad640 1 Processor -- start 2023-10-16T15:53:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.869+0000 7fc8822ad640 1 -- start start 2023-10-16T15:53:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8822ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc87c0ad970 0x7fc87c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8822ad640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc87c0ccfb0 0x7fc87c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8822ad640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c15b9d0 con 0x7fc87c06c040 2023-10-16T15:53:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8822ad640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c15bcb0 con 0x7fc87c0ad970 2023-10-16T15:53:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8822ad640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c15bf90 con 0x7fc87c0ccfb0 2023-10-16T15:53:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8812ab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc87c06c040 0x7fc87c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52240/0 (socket says 172.21.15.29:52240) 2023-10-16T15:53:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc8812ab640 1 -- 172.21.15.29:0/3247658711 learned_addr learned my addr 172.21.15.29:0/3247658711 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.871+0000 7fc880aaa640 1 --2- 172.21.15.29:0/3247658711 >> v2:172.21.15.29:3300/0 conn(0x7fc87c0ad970 0x7fc87c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc881aac640 1 --2- 172.21.15.29:0/3247658711 >> v2:172.21.15.170:3300/0 conn(0x7fc87c0ccfb0 0x7fc87c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc873fff640 1 -- 172.21.15.29:0/3247658711 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412604223 0 0) 0x7fc87c15b9d0 con 0x7fc87c06c040 2023-10-16T15:53:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc873fff640 1 -- 172.21.15.29:0/3247658711 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc868003520 con 0x7fc87c06c040 2023-10-16T15:53:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc880aaa640 1 -- 172.21.15.29:0/3247658711 >> v2:172.21.15.170:3300/0 conn(0x7fc87c0ccfb0 msgr2=0x7fc87c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc880aaa640 1 --2- 172.21.15.29:0/3247658711 >> v2:172.21.15.170:3300/0 conn(0x7fc87c0ccfb0 0x7fc87c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc880aaa640 1 -- 172.21.15.29:0/3247658711 >> v1:172.21.15.29:6789/0 conn(0x7fc87c06c040 legacy=0x7fc87c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc880aaa640 1 -- 172.21.15.29:0/3247658711 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0cc270 con 0x7fc87c0ad970 2023-10-16T15:53:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.872+0000 7fc881aac640 1 --2- 172.21.15.29:0/3247658711 >> v2:172.21.15.170:3300/0 conn(0x7fc87c0ccfb0 0x7fc87c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.874+0000 7fc880aaa640 1 --2- 172.21.15.29:0/3247658711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0cca70 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0097a0 tx=0x7fc86c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d04ccbd7829df59 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.874+0000 7fc873fff640 1 -- 172.21.15.29:0/3247658711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c031830 con 0x7fc87c0ad970 2023-10-16T15:53:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.874+0000 7fc873fff640 1 -- 172.21.15.29:0/3247658711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c031990 con 0x7fc87c0ad970 2023-10-16T15:53:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.874+0000 7fc873fff640 1 -- 172.21.15.29:0/3247658711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c031d20 con 0x7fc87c0ad970 2023-10-16T15:53:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 -- 172.21.15.29:0/3247658711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 msgr2=0x7fc87c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 --2- 172.21.15.29:0/3247658711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0cca70 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0097a0 tx=0x7fc86c02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 -- 172.21.15.29:0/3247658711 shutdown_connections 2023-10-16T15:53:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 --2- 172.21.15.29:0/3247658711 >> v2:172.21.15.170:3300/0 conn(0x7fc87c0ccfb0 0x7fc87c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 --2- 172.21.15.29:0/3247658711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0cca70 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 -- 172.21.15.29:0/3247658711 >> 172.21.15.29:0/3247658711 conn(0x7fc87c0ce6a0 msgr2=0x7fc87c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 -- 172.21.15.29:0/3247658711 shutdown_connections 2023-10-16T15:53:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.875+0000 7fc8822ad640 1 -- 172.21.15.29:0/3247658711 wait complete. 2023-10-16T15:53:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.877+0000 7fc8822ad640 1 Processor -- start 2023-10-16T15:53:15.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.877+0000 7fc8822ad640 1 -- start start 2023-10-16T15:53:15.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc8822ad640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c06c040 0x7fc87c1f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc8822ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc8822ad640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c0ccfb0 0x7fc87c0b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc8822ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0afab0 con 0x7fc87c0ad970 2023-10-16T15:53:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc8822ad640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0afd90 con 0x7fc87c0ccfb0 2023-10-16T15:53:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc8822ad640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0b0070 con 0x7fc87c06c040 2023-10-16T15:53:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc880aaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc880aaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53610/0 (socket says 172.21.15.29:53610) 2023-10-16T15:53:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc880aaa640 1 -- 172.21.15.29:0/2562915771 learned_addr learned my addr 172.21.15.29:0/2562915771 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc881aac640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c0ccfb0 0x7fc87c0b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc880aaa640 1 -- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c06c040 msgr2=0x7fc87c1f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc880aaa640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c06c040 0x7fc87c1f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.878+0000 7fc880aaa640 1 -- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c0ccfb0 msgr2=0x7fc87c0b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc880aaa640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c0ccfb0 0x7fc87c0b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc880aaa640 1 -- 172.21.15.29:0/2562915771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c009400 con 0x7fc87c0ad970 2023-10-16T15:53:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc881aac640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c0ccfb0 0x7fc87c0b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc880aaa640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0b1a60 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc86c02bee0 tx=0x7fc86c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c03a6f0 con 0x7fc87c0ad970 2023-10-16T15:53:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c03a870 con 0x7fc87c0ad970 2023-10-16T15:53:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c03ac20 con 0x7fc87c0ad970 2023-10-16T15:53:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c0b0350 con 0x7fc87c0ad970 2023-10-16T15:53:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.879+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c0b0b50 con 0x7fc87c0ad970 2023-10-16T15:53:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.880+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c06b330 con 0x7fc87c0ad970 2023-10-16T15:53:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.881+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc86c030380 con 0x7fc87c0ad970 2023-10-16T15:53:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.883+0000 7fc871ffb640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc86006af90 0x7fc86006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.883+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fc86c0ab1f0 con 0x7fc87c0ad970 2023-10-16T15:53:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.884+0000 7fc8812ab640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc86006af90 0x7fc86006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.885+0000 7fc8812ab640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc86006af90 0x7fc86006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc878001610 tx=0x7fc878007930 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:15.887+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc86c0760a0 con 0x7fc87c0ad970 2023-10-16T15:53:16.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.079+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc87c0b0870 con 0x7fc87c0ad970 2023-10-16T15:53:16.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.085+0000 7fc871ffb640 1 -- 172.21.15.29:0/2562915771 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc87c0b0870 con 0x7fc87c0ad970 2023-10-16T15:53:16.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc86006af90 msgr2=0x7fc86006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc86006af90 0x7fc86006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc878001610 tx=0x7fc878007930 comp rx=0 tx=0).stop 2023-10-16T15:53:16.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 msgr2=0x7fc87c0b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0b1a60 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc86c02bee0 tx=0x7fc86c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 shutdown_connections 2023-10-16T15:53:16.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc86006af90 0x7fc86006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc87c0ad970 0x7fc87c0b1a60 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc87c0ccfb0 0x7fc87c0b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 --2- 172.21.15.29:0/2562915771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc87c06c040 0x7fc87c1f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 >> 172.21.15.29:0/2562915771 conn(0x7fc87c0ce6a0 msgr2=0x7fc87c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:16.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 shutdown_connections 2023-10-16T15:53:16.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.088+0000 7fc8822ad640 1 -- 172.21.15.29:0/2562915771 wait complete. 2023-10-16T15:53:16.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:53:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.328+0000 7f607e267640 1 Processor -- start 2023-10-16T15:53:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.328+0000 7f607e267640 1 -- start start 2023-10-16T15:53:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607e267640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6078070f30 0x7f607806aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607e267640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f607806b3e0 0x7f60780c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607e267640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60780cd260 con 0x7f6078159ad0 2023-10-16T15:53:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607e267640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607815de30 con 0x7f6078070f30 2023-10-16T15:53:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607e267640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607815e110 con 0x7f607806b3e0 2023-10-16T15:53:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607d265640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6078159ad0 0x7f607815d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52252/0 (socket says 172.21.15.29:52252) 2023-10-16T15:53:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607d265640 1 -- 172.21.15.29:0/1973251075 learned_addr learned my addr 172.21.15.29:0/1973251075 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f607ca64640 1 --2- 172.21.15.29:0/1973251075 >> v2:172.21.15.29:3300/0 conn(0x7f6078070f30 0x7f607806aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.330+0000 7f606ffff640 1 -- 172.21.15.29:0/1973251075 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055734798 0 0) 0x7f60780cd260 con 0x7f6078159ad0 2023-10-16T15:53:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.331+0000 7f607da66640 1 --2- 172.21.15.29:0/1973251075 >> v2:172.21.15.170:3300/0 conn(0x7f607806b3e0 0x7f60780c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.331+0000 7f606ffff640 1 -- 172.21.15.29:0/1973251075 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6064003520 con 0x7f6078159ad0 2023-10-16T15:53:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.331+0000 7f607ca64640 1 -- 172.21.15.29:0/1973251075 >> v2:172.21.15.170:3300/0 conn(0x7f607806b3e0 msgr2=0x7f60780c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.331+0000 7f607ca64640 1 --2- 172.21.15.29:0/1973251075 >> v2:172.21.15.170:3300/0 conn(0x7f607806b3e0 0x7f60780c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.331+0000 7f607ca64640 1 -- 172.21.15.29:0/1973251075 >> v1:172.21.15.29:6789/0 conn(0x7f6078159ad0 legacy=0x7f607815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.331+0000 7f607ca64640 1 -- 172.21.15.29:0/1973251075 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607815e3f0 con 0x7f6078070f30 2023-10-16T15:53:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.332+0000 7f607ca64640 1 --2- 172.21.15.29:0/1973251075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f607806aea0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f607806d720 tx=0x7f606802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd8f7bf538169bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.332+0000 7f606ffff640 1 -- 172.21.15.29:0/1973251075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068031830 con 0x7f6078070f30 2023-10-16T15:53:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.332+0000 7f606ffff640 1 -- 172.21.15.29:0/1973251075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6068031990 con 0x7f6078070f30 2023-10-16T15:53:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.332+0000 7f606ffff640 1 -- 172.21.15.29:0/1973251075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068031d20 con 0x7f6078070f30 2023-10-16T15:53:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 -- 172.21.15.29:0/1973251075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 msgr2=0x7f607806aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 --2- 172.21.15.29:0/1973251075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f607806aea0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f607806d720 tx=0x7f606802b8d0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 -- 172.21.15.29:0/1973251075 shutdown_connections 2023-10-16T15:53:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 --2- 172.21.15.29:0/1973251075 >> v2:172.21.15.170:3300/0 conn(0x7f607806b3e0 0x7f60780c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 --2- 172.21.15.29:0/1973251075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f607806aea0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 -- 172.21.15.29:0/1973251075 >> 172.21.15.29:0/1973251075 conn(0x7f60780b0170 msgr2=0x7f60780b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 -- 172.21.15.29:0/1973251075 shutdown_connections 2023-10-16T15:53:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.333+0000 7f607e267640 1 -- 172.21.15.29:0/1973251075 wait complete. 2023-10-16T15:53:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.335+0000 7f607e267640 1 Processor -- start 2023-10-16T15:53:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.335+0000 7f607e267640 1 -- start start 2023-10-16T15:53:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607e267640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f607806b3e0 0x7f607814e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607e267640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f60780b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607e267640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6078159ad0 0x7f60780c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607e267640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607815b370 con 0x7f6078070f30 2023-10-16T15:53:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607e267640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607815c1d0 con 0x7f607806b3e0 2023-10-16T15:53:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607e267640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607815b5c0 con 0x7f6078159ad0 2023-10-16T15:53:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607ca64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f60780b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607ca64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f60780b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53638/0 (socket says 172.21.15.29:53638) 2023-10-16T15:53:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.336+0000 7f607ca64640 1 -- 172.21.15.29:0/1639472223 learned_addr learned my addr 172.21.15.29:0/1639472223 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607da66640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6078159ad0 0x7f60780c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607d265640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f607806b3e0 0x7f607814e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607ca64640 1 -- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6078159ad0 msgr2=0x7f60780c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607ca64640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6078159ad0 0x7f60780c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607ca64640 1 -- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f607806b3e0 msgr2=0x7f607814e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607ca64640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f607806b3e0 0x7f607814e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607ca64640 1 -- 172.21.15.29:0/1639472223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6068009400 con 0x7f6078070f30 2023-10-16T15:53:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607da66640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6078159ad0 0x7f60780c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607d265640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f607806b3e0 0x7f607814e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607ca64640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f60780b0550 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f6068009890 tx=0x7f60680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606803a6f0 con 0x7f6078070f30 2023-10-16T15:53:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606803a870 con 0x7f6078070f30 2023-10-16T15:53:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606803ac20 con 0x7f6078070f30 2023-10-16T15:53:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.337+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607815b8a0 con 0x7f6078070f30 2023-10-16T15:53:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.338+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607815be60 con 0x7f6078070f30 2023-10-16T15:53:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.339+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607804e9b0 con 0x7f6078070f30 2023-10-16T15:53:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.340+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6068043410 con 0x7f6078070f30 2023-10-16T15:53:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.341+0000 7f606dffb640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f605c06afe0 0x7f605c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.341+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f60680ab350 con 0x7f6078070f30 2023-10-16T15:53:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.341+0000 7f607d265640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f605c06afe0 0x7f605c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.342+0000 7f607d265640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f605c06afe0 0x7f605c06d4a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60740011e0 tx=0x7f6074009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.345+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f60680762f0 con 0x7f6078070f30 2023-10-16T15:53:16.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.536+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f60780b4600 con 0x7f6078070f30 2023-10-16T15:53:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.543+0000 7f606dffb640 1 -- 172.21.15.29:0/1639472223 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v74) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6068079910 con 0x7f6078070f30 2023-10-16T15:53:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.545+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f605c06afe0 msgr2=0x7f605c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.545+0000 7f607e267640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f605c06afe0 0x7f605c06d4a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60740011e0 tx=0x7f6074009040 comp rx=0 tx=0).stop 2023-10-16T15:53:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.545+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 msgr2=0x7f60780b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.545+0000 7f607e267640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f60780b0550 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f6068009890 tx=0x7f60680046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 shutdown_connections 2023-10-16T15:53:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f605c06afe0 0x7f605c06d4a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6078070f30 0x7f60780b0550 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6078159ad0 0x7f60780c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 --2- 172.21.15.29:0/1639472223 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f607806b3e0 0x7f607814e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 >> 172.21.15.29:0/1639472223 conn(0x7f60780b0170 msgr2=0x7f6078158e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 shutdown_connections 2023-10-16T15:53:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.546+0000 7f607e267640 1 -- 172.21.15.29:0/1639472223 wait complete. 2023-10-16T15:53:16.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:53:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.790+0000 7f8727fff640 1 Processor -- start 2023-10-16T15:53:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.790+0000 7f8727fff640 1 -- start start 2023-10-16T15:53:16.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8727fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 0x7f87280ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8727fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87280cd4a0 con 0x7f8728153740 2023-10-16T15:53:16.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8727fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872815ac60 con 0x7f87280ad4d0 2023-10-16T15:53:16.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8727fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8728070030 con 0x7f87280ccb00 2023-10-16T15:53:16.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8726ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8728153740 0x7f872815a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52262/0 (socket says 172.21.15.29:52262) 2023-10-16T15:53:16.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8726ffd640 1 -- 172.21.15.29:0/1456128692 learned_addr learned my addr 172.21.15.29:0/1456128692 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339817217 0 0) 0x7f87280cd4a0 con 0x7f8728153740 2023-10-16T15:53:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f87267fc640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 0x7f87280ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.792+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8710004630 con 0x7f8728153740 2023-10-16T15:53:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971043925 0 0) 0x7f8710004630 con 0x7f8728153740 2023-10-16T15:53:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 msgr2=0x7f87280ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 0x7f87280ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 >> v1:172.21.15.170:6789/0 conn(0x7f87280ad4d0 legacy=0x7f87280ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728070310 con 0x7f8728153740 2023-10-16T15:53:16.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8718001670 con 0x7f8728153740 2023-10-16T15:53:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 >> v1:172.21.15.29:6789/0 conn(0x7f8728153740 legacy=0x7f872815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 0x7f8710004ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8710005080 0x7f87100058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710005e10 0x7f8710006290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8728070030 con 0x7f8710005e10 2023-10-16T15:53:16.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871000cc80 con 0x7f8710005080 2023-10-16T15:53:16.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87100034c0 con 0x7f87280ccb00 2023-10-16T15:53:16.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.793+0000 7f87277fe640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710005e10 0x7f8710006290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f8726ffd640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 0x7f8710004ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f87267fc640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8710005080 0x7f87100058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f87277fe640 1 -- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 msgr2=0x7f8710004ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f87277fe640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87280ccb00 0x7f8710004ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f87277fe640 1 -- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8710005080 msgr2=0x7f87100058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f87277fe640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8710005080 0x7f87100058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.794+0000 7f87277fe640 1 -- 172.21.15.29:0/1456128692 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710004630 con 0x7f8710005e10 2023-10-16T15:53:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.795+0000 7f87277fe640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710005e10 0x7f8710006290 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f8720001e50 tx=0x7f8710006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c038c31e9c5816f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.795+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720003b70 con 0x7f8710005e10 2023-10-16T15:53:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.795+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8720003cd0 con 0x7f8710005e10 2023-10-16T15:53:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.795+0000 7f8725ffb640 1 -- 172.21.15.29:0/1456128692 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720005310 con 0x7f8710005e10 2023-10-16T15:53:16.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f8727fff640 1 -- 172.21.15.29:0/1456128692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710005e10 msgr2=0x7f8710006290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f8727fff640 1 --2- 172.21.15.29:0/1456128692 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8710005e10 0x7f8710006290 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f8720001e50 tx=0x7f8710006900 comp rx=0 tx=0).stop 2023-10-16T15:53:16.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f87277fe640 1 -- 172.21.15.29:0/1456128692 reap_dead start 2023-10-16T15:53:16.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f8727fff640 1 -- 172.21.15.29:0/1456128692 shutdown_connections 2023-10-16T15:53:16.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f8727fff640 1 -- 172.21.15.29:0/1456128692 >> 172.21.15.29:0/1456128692 conn(0x7f87280ce6a0 msgr2=0x7f87280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:16.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f8727fff640 1 -- 172.21.15.29:0/1456128692 shutdown_connections 2023-10-16T15:53:16.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.796+0000 7f8727fff640 1 -- 172.21.15.29:0/1456128692 wait complete. 2023-10-16T15:53:16.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.798+0000 7f8727fff640 1 Processor -- start 2023-10-16T15:53:16.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.798+0000 7f8727fff640 1 -- start start 2023-10-16T15:53:16.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f8727fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8710005e10 0x7f8728151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f8727fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f872815c140 0x7f872815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f8727fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 0x7f87280ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f8727fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872815cb00 con 0x7f87280ab500 2023-10-16T15:53:16.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f8727fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280b2ab0 con 0x7f872815c140 2023-10-16T15:53:16.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f8727fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87280acb10 con 0x7f8710005e10 2023-10-16T15:53:16.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.799+0000 7f87277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 0x7f87280ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 0x7f87280ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53660/0 (socket says 172.21.15.29:53660) 2023-10-16T15:53:16.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 -- 172.21.15.29:0/4123252537 learned_addr learned my addr 172.21.15.29:0/4123252537 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:16.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87267fc640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f872815c140 0x7f872815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f8726ffd640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8710005e10 0x7f8728151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 -- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8710005e10 msgr2=0x7f8728151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8710005e10 0x7f8728151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 -- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f872815c140 msgr2=0x7f872815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:16.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f872815c140 0x7f872815c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:16.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 -- 172.21.15.29:0/4123252537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87280ac270 con 0x7f87280ab500 2023-10-16T15:53:16.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.800+0000 7f87277fe640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 0x7f87280ab980 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8720001f70 tx=0x7f87200014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:16.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.801+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872003e070 con 0x7f87280ab500 2023-10-16T15:53:16.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.801+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8720003e40 con 0x7f87280ab500 2023-10-16T15:53:16.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.801+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87280ac550 con 0x7f87280ab500 2023-10-16T15:53:16.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.801+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720006d50 con 0x7f87280ab500 2023-10-16T15:53:16.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.801+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87280a8830 con 0x7f87280ab500 2023-10-16T15:53:16.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.802+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87280a8690 con 0x7f87280ab500 2023-10-16T15:53:16.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.803+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8720007030 con 0x7f87280ab500 2023-10-16T15:53:16.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.804+0000 7f870f7fe640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f871406afc0 0x7f871406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:16.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.804+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f87200aab80 con 0x7f87280ab500 2023-10-16T15:53:16.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.805+0000 7f8726ffd640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f871406afc0 0x7f871406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:16.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.806+0000 7f8726ffd640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f871406afc0 0x7f871406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8718001050 tx=0x7f8718009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:16.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:16.809+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87280a8690 con 0x7f87280ab500 2023-10-16T15:53:17.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.001+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f87280a8690 con 0x7f87280ab500 2023-10-16T15:53:17.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.011+0000 7f870f7fe640 1 -- 172.21.15.29:0/4123252537 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f87280a8690 con 0x7f87280ab500 2023-10-16T15:53:17.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.014+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f871406afc0 msgr2=0x7f871406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.014+0000 7f8727fff640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f871406afc0 0x7f871406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8718001050 tx=0x7f8718009040 comp rx=0 tx=0).stop 2023-10-16T15:53:17.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.014+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 msgr2=0x7f87280ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.014+0000 7f8727fff640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 0x7f87280ab980 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8720001f70 tx=0x7f87200014a0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 shutdown_connections 2023-10-16T15:53:17.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f871406afc0 0x7f871406d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87280ab500 0x7f87280ab980 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f872815c140 0x7f872815c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 --2- 172.21.15.29:0/4123252537 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8710005e10 0x7f8728151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 >> 172.21.15.29:0/4123252537 conn(0x7f87280ce6a0 msgr2=0x7f8728152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.015+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 shutdown_connections 2023-10-16T15:53:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.016+0000 7f8727fff640 1 -- 172.21.15.29:0/4123252537 wait complete. 2023-10-16T15:53:17.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T15:53:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.246+0000 7f2efb210640 1 Processor -- start 2023-10-16T15:53:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.246+0000 7f2efb210640 1 -- start start 2023-10-16T15:53:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.247+0000 7f2efb210640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2ef415aaa0 0x7f2ef415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.247+0000 7f2efb210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef415ae80 0x7f2ef40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.247+0000 7f2efb210640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ef40ccaa0 0x7f2ef4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.247+0000 7f2efb210640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef406b810 con 0x7f2ef40ccaa0 2023-10-16T15:53:17.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.247+0000 7f2efb210640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef415ed60 con 0x7f2ef415aaa0 2023-10-16T15:53:17.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.247+0000 7f2efb210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40707e0 con 0x7f2ef415ae80 2023-10-16T15:53:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.248+0000 7f2ef97ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ef40ccaa0 0x7f2ef4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.248+0000 7f2ef97ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ef40ccaa0 0x7f2ef4070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53662/0 (socket says 172.21.15.29:53662) 2023-10-16T15:53:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.248+0000 7f2ef8fac640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2ef415aaa0 0x7f2ef415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.248+0000 7f2ef97ad640 1 -- 172.21.15.29:0/2087407200 learned_addr learned my addr 172.21.15.29:0/2087407200 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.248+0000 7f2ef3fff640 1 --2- 172.21.15.29:0/2087407200 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef415ae80 0x7f2ef40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.249+0000 7f2ef97ad640 1 -- 172.21.15.29:0/2087407200 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef415ae80 msgr2=0x7f2ef40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:17.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.249+0000 7f2ef97ad640 1 --2- 172.21.15.29:0/2087407200 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef415ae80 0x7f2ef40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.249+0000 7f2ef97ad640 1 -- 172.21.15.29:0/2087407200 >> v2:172.21.15.170:3300/0 conn(0x7f2ef415aaa0 msgr2=0x7f2ef415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.249+0000 7f2ef97ad640 1 --2- 172.21.15.29:0/2087407200 >> v2:172.21.15.170:3300/0 conn(0x7f2ef415aaa0 0x7f2ef415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.249+0000 7f2ef97ad640 1 -- 172.21.15.29:0/2087407200 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef415ef90 con 0x7f2ef40ccaa0 2023-10-16T15:53:17.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.249+0000 7f2ef8fac640 1 --2- 172.21.15.29:0/2087407200 >> v2:172.21.15.170:3300/0 conn(0x7f2ef415aaa0 0x7f2ef415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.250+0000 7f2ef97ad640 1 --2- 172.21.15.29:0/2087407200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40ccaa0 0x7f2ef4070f50 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40cd990 tx=0x7f2eec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c36dc217ab595f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.250+0000 7f2ef37fe640 1 -- 172.21.15.29:0/2087407200 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec02ca80 con 0x7f2ef40ccaa0 2023-10-16T15:53:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.250+0000 7f2ef37fe640 1 -- 172.21.15.29:0/2087407200 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec02cbe0 con 0x7f2ef40ccaa0 2023-10-16T15:53:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.250+0000 7f2ef37fe640 1 -- 172.21.15.29:0/2087407200 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0343f0 con 0x7f2ef40ccaa0 2023-10-16T15:53:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 -- 172.21.15.29:0/2087407200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40ccaa0 msgr2=0x7f2ef4070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 --2- 172.21.15.29:0/2087407200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40ccaa0 0x7f2ef4070f50 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40cd990 tx=0x7f2eec02bee0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 -- 172.21.15.29:0/2087407200 shutdown_connections 2023-10-16T15:53:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 --2- 172.21.15.29:0/2087407200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef40ccaa0 0x7f2ef4070f50 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 --2- 172.21.15.29:0/2087407200 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef415ae80 0x7f2ef40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 --2- 172.21.15.29:0/2087407200 >> v2:172.21.15.170:3300/0 conn(0x7f2ef415aaa0 0x7f2ef415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 -- 172.21.15.29:0/2087407200 >> 172.21.15.29:0/2087407200 conn(0x7f2ef40cb410 msgr2=0x7f2ef40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 -- 172.21.15.29:0/2087407200 shutdown_connections 2023-10-16T15:53:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.251+0000 7f2efb210640 1 -- 172.21.15.29:0/2087407200 wait complete. 2023-10-16T15:53:17.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.253+0000 7f2efb210640 1 Processor -- start 2023-10-16T15:53:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.254+0000 7f2efb210640 1 -- start start 2023-10-16T15:53:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2efb210640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40ccaa0 0x7f2ef4155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2efb210640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 0x7f2ef40afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2efb210640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ef415ae80 0x7f2ef40b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2efb210640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40c24c0 con 0x7f2ef415aaa0 2023-10-16T15:53:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2efb210640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4155d60 con 0x7f2ef415ae80 2023-10-16T15:53:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2efb210640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40c68e0 con 0x7f2ef40ccaa0 2023-10-16T15:53:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 0x7f2ef40afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 0x7f2ef40afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53674/0 (socket says 172.21.15.29:53674) 2023-10-16T15:53:17.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.255+0000 7f2ef3fff640 1 -- 172.21.15.29:0/756298527 learned_addr learned my addr 172.21.15.29:0/756298527 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef97ad640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ef415ae80 0x7f2ef40b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef8fac640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40ccaa0 0x7f2ef4155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef3fff640 1 -- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40ccaa0 msgr2=0x7f2ef4155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef3fff640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40ccaa0 0x7f2ef4155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef3fff640 1 -- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ef415ae80 msgr2=0x7f2ef40b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef3fff640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ef415ae80 0x7f2ef40b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef3fff640 1 -- 172.21.15.29:0/756298527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec009b30 con 0x7f2ef415aaa0 2023-10-16T15:53:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef97ad640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ef415ae80 0x7f2ef40b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef3fff640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 0x7f2ef40afb90 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000d5f0 tx=0x7f2ee000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee00125f0 con 0x7f2ef415aaa0 2023-10-16T15:53:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee0012750 con 0x7f2ef415aaa0 2023-10-16T15:53:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0012b00 con 0x7f2ef415aaa0 2023-10-16T15:53:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.256+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef40c2710 con 0x7f2ef415aaa0 2023-10-16T15:53:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.257+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef40c2cd0 con 0x7f2ef415aaa0 2023-10-16T15:53:17.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.258+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef40c2e90 con 0x7f2ef415aaa0 2023-10-16T15:53:17.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.259+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ee0012c60 con 0x7f2ef415aaa0 2023-10-16T15:53:17.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.260+0000 7f2ef17fa640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f2ed806afe0 0x7f2ed806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.260+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2ee0089bc0 con 0x7f2ef415aaa0 2023-10-16T15:53:17.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.261+0000 7f2ef8fac640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f2ed806afe0 0x7f2ed806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.262+0000 7f2ef8fac640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f2ed806afe0 0x7f2ed806d4a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8006fd0 tx=0x7f2ee8008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:17.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.265+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ee0054c00 con 0x7f2ef415aaa0 2023-10-16T15:53:17.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.529+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f2ef404e160 con 0x7f2ef415aaa0 2023-10-16T15:53:17.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.621+0000 7f2ef17fa640 1 -- 172.21.15.29:0/756298527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3160) v1 ==== 163+0+0 (secure 0 0 0) 0x7f2ee0058220 con 0x7f2ef415aaa0 2023-10-16T15:53:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f2ed806afe0 msgr2=0x7f2ed806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f2ed806afe0 0x7f2ed806d4a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8006fd0 tx=0x7f2ee8008040 comp rx=0 tx=0).stop 2023-10-16T15:53:17.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 msgr2=0x7f2ef40afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 0x7f2ef40afb90 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000d5f0 tx=0x7f2ee000dab0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 shutdown_connections 2023-10-16T15:53:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f2ed806afe0 0x7f2ed806d4a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef415aaa0 0x7f2ef40afb90 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ef415ae80 0x7f2ef40b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 --2- 172.21.15.29:0/756298527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2ef40ccaa0 0x7f2ef4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 >> 172.21.15.29:0/756298527 conn(0x7f2ef40cb410 msgr2=0x7f2ef40b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 shutdown_connections 2023-10-16T15:53:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.624+0000 7f2efb210640 1 -- 172.21.15.29:0/756298527 wait complete. 2023-10-16T15:53:17.645 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-16T15:53:17.645 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2023-10-16T15:53:17.646 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T15:53:17.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:17.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:53:17.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:53:17.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.867+0000 7f3c8770f640 1 Processor -- start 2023-10-16T15:53:17.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.867+0000 7f3c8770f640 1 -- start start 2023-10-16T15:53:17.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c8770f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c800b3070 0x7f3c800cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c8770f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800d1b10 0x7f3c800cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c8770f640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800b3940 con 0x7f3c800b3e70 2023-10-16T15:53:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c8770f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c800b45e0 con 0x7f3c800b3070 2023-10-16T15:53:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c8770f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800b3b90 con 0x7f3c800d1b10 2023-10-16T15:53:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c854ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c800b3070 0x7f3c800cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c854ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c800b3070 0x7f3c800cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53676/0 (socket says 172.21.15.29:53676) 2023-10-16T15:53:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.868+0000 7f3c854ab640 1 -- 172.21.15.29:0/1205567992 learned_addr learned my addr 172.21.15.29:0/1205567992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.869+0000 7f3c85cac640 1 --2- 172.21.15.29:0/1205567992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800d1b10 0x7f3c800cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.869+0000 7f3c854ab640 1 -- 172.21.15.29:0/1205567992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800d1b10 msgr2=0x7f3c800cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.869+0000 7f3c854ab640 1 --2- 172.21.15.29:0/1205567992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800d1b10 0x7f3c800cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.869+0000 7f3c854ab640 1 -- 172.21.15.29:0/1205567992 >> v1:172.21.15.170:6789/0 conn(0x7f3c800b3e70 legacy=0x7f3c800c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:17.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.869+0000 7f3c854ab640 1 -- 172.21.15.29:0/1205567992 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c800d3840 con 0x7f3c800b3070 2023-10-16T15:53:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.870+0000 7f3c854ab640 1 --2- 172.21.15.29:0/1205567992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800b3070 0x7f3c800cf500 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009850 tx=0x7f3c7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583974406719fd3d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.870+0000 7f3c77fff640 1 -- 172.21.15.29:0/1205567992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c004230 con 0x7f3c800b3070 2023-10-16T15:53:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.870+0000 7f3c77fff640 1 -- 172.21.15.29:0/1205567992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c004390 con 0x7f3c800b3070 2023-10-16T15:53:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.870+0000 7f3c77fff640 1 -- 172.21.15.29:0/1205567992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c031830 con 0x7f3c800b3070 2023-10-16T15:53:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 -- 172.21.15.29:0/1205567992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800b3070 msgr2=0x7f3c800cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1205567992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800b3070 0x7f3c800cf500 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009850 tx=0x7f3c7c02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 -- 172.21.15.29:0/1205567992 shutdown_connections 2023-10-16T15:53:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1205567992 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800d1b10 0x7f3c800cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1205567992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800b3070 0x7f3c800cf500 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 -- 172.21.15.29:0/1205567992 >> 172.21.15.29:0/1205567992 conn(0x7f3c800d2d20 msgr2=0x7f3c800d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 -- 172.21.15.29:0/1205567992 shutdown_connections 2023-10-16T15:53:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.871+0000 7f3c8770f640 1 -- 172.21.15.29:0/1205567992 wait complete. 2023-10-16T15:53:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.873+0000 7f3c8770f640 1 Processor -- start 2023-10-16T15:53:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.873+0000 7f3c8770f640 1 -- start start 2023-10-16T15:53:17.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c8770f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c800b3070 0x7f3c800b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c8770f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800b3e70 0x7f3c8015e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c8770f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 0x7f3c8006aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c8770f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800b0d70 con 0x7f3c800d1b10 2023-10-16T15:53:17.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c8770f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8015ecc0 con 0x7f3c800b3070 2023-10-16T15:53:17.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c8770f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800afcd0 con 0x7f3c800b3e70 2023-10-16T15:53:17.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c85cac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 0x7f3c8006aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c85cac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 0x7f3c8006aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53688/0 (socket says 172.21.15.29:53688) 2023-10-16T15:53:17.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.874+0000 7f3c85cac640 1 -- 172.21.15.29:0/1051294841 learned_addr learned my addr 172.21.15.29:0/1051294841 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:17.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c84caa640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800b3e70 0x7f3c8015e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c854ab640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c800b3070 0x7f3c800b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c85cac640 1 -- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800b3e70 msgr2=0x7f3c8015e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c85cac640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800b3e70 0x7f3c8015e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c85cac640 1 -- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c800b3070 msgr2=0x7f3c800b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:17.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c85cac640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c800b3070 0x7f3c800b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c85cac640 1 -- 172.21.15.29:0/1051294841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c009460 con 0x7f3c800d1b10 2023-10-16T15:53:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c85cac640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 0x7f3c8006aad0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f3c78007e60 tx=0x7f3c7800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78014420 con 0x7f3c800d1b10 2023-10-16T15:53:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c780145a0 con 0x7f3c800d1b10 2023-10-16T15:53:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c800affb0 con 0x7f3c800d1b10 2023-10-16T15:53:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78014950 con 0x7f3c800d1b10 2023-10-16T15:53:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.875+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c800b0570 con 0x7f3c800d1b10 2023-10-16T15:53:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.877+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8004e160 con 0x7f3c800d1b10 2023-10-16T15:53:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.878+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3c78014ab0 con 0x7f3c800d1b10 2023-10-16T15:53:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.879+0000 7f3c75ffb640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3c6406af90 0x7f3c6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.879+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3c7808a150 con 0x7f3c800d1b10 2023-10-16T15:53:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.879+0000 7f3c854ab640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3c6406af90 0x7f3c6406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.880+0000 7f3c854ab640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3c6406af90 0x7f3c6406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c035000 tx=0x7f3c7c008030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:17.883+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c78055080 con 0x7f3c800d1b10 2023-10-16T15:53:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.079+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c800ce7a0 con 0x7f3c800d1b10 2023-10-16T15:53:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.082+0000 7f3c75ffb640 1 -- 172.21.15.29:0/1051294841 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+14053 (secure 0 0 0) 0x7f3c780586a0 con 0x7f3c800d1b10 2023-10-16T15:53:18.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:18.084 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":217,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:15.604335+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6833/2149952636":"2023-10-17T15:53:12.586145+0000","172.21.15.29:6832/2149952636":"2023-10-17T15:53:12.586145+0000","172.21.15.170:6833/928368069":"2023-10-17T15:53:08.409834+0000","172.21.15.170:6832/928368069":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/1186413715":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/3227854992":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/568891934":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/2703119246":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/1539386380":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/3252182417":"2023-10-17T15:53:08.409834+0000","172.21.15.170:0/423324657":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/3913323747":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2674320589":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/275392394":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6833/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:6832/780156504":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/2170359614":"2023-10-17T15:50:59.297259+0000","172.21.15.170:0/1066951432":"2023-10-17T15:50:59.297259+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3c6406af90 msgr2=0x7f3c6406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3c6406af90 0x7f3c6406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c035000 tx=0x7f3c7c008030 comp rx=0 tx=0).stop 2023-10-16T15:53:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 msgr2=0x7f3c8006aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 0x7f3c8006aad0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f3c78007e60 tx=0x7f3c7800e4b0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 shutdown_connections 2023-10-16T15:53:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3c6406af90 0x7f3c6406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c800d1b10 0x7f3c8006aad0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c800b3e70 0x7f3c8015e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 --2- 172.21.15.29:0/1051294841 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c800b3070 0x7f3c800b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 >> 172.21.15.29:0/1051294841 conn(0x7f3c800d2d20 msgr2=0x7f3c801606d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.084+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 shutdown_connections 2023-10-16T15:53:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.085+0000 7f3c8770f640 1 -- 172.21.15.29:0/1051294841 wait complete. 2023-10-16T15:53:18.106 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:53:18.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:53:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.337+0000 7f3214bc5640 1 Processor -- start 2023-10-16T15:53:18.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.337+0000 7f3214bc5640 1 -- start start 2023-10-16T15:53:18.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.339+0000 7f3214bc5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3210159e90 0x7f32100ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.339+0000 7f3214bc5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3210070360 con 0x7f321006c140 2023-10-16T15:53:18.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.339+0000 7f3214bc5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32100cd1f0 con 0x7f3210159ab0 2023-10-16T15:53:18.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.339+0000 7f3214bc5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321015d890 con 0x7f3210159e90 2023-10-16T15:53:18.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f321006c140 0x7f321015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52274/0 (socket says 172.21.15.29:52274) 2023-10-16T15:53:18.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320ed9d640 1 -- 172.21.15.29:0/1744929485 learned_addr learned my addr 172.21.15.29:0/1744929485 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:18.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320dd9b640 1 --2- 172.21.15.29:0/1744929485 >> v2:172.21.15.170:3300/0 conn(0x7f3210159e90 0x7f32100ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474740653 0 0) 0x7f3210070360 con 0x7f321006c140 2023-10-16T15:53:18.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8003520 con 0x7f321006c140 2023-10-16T15:53:18.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 896953627 0 0) 0x7f31f8003520 con 0x7f321006c140 2023-10-16T15:53:18.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 >> v2:172.21.15.170:3300/0 conn(0x7f3210159e90 msgr2=0x7f32100ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:18.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 --2- 172.21.15.29:0/1744929485 >> v2:172.21.15.170:3300/0 conn(0x7f3210159e90 0x7f32100ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 >> v1:172.21.15.170:6789/0 conn(0x7f3210159ab0 legacy=0x7f321006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.340+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32100cd4d0 con 0x7f321006c140 2023-10-16T15:53:18.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3208001670 con 0x7f321006c140 2023-10-16T15:53:18.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 >> v1:172.21.15.29:6789/0 conn(0x7f321006c140 legacy=0x7f321015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f31f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31f80047c0 0x7f31f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31f80052a0 0x7f31f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321015d890 con 0x7f31f80039c0 2023-10-16T15:53:18.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800c1a0 con 0x7f31f80047c0 2023-10-16T15:53:18.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800c360 con 0x7f31f80052a0 2023-10-16T15:53:18.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320e59c640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f31f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320e59c640 1 -- 172.21.15.29:0/1744929485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31f80052a0 msgr2=0x7f31f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:18.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320e59c640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f31f80052a0 0x7f31f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320e59c640 1 -- 172.21.15.29:0/1744929485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31f80047c0 msgr2=0x7f31f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320e59c640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f31f80047c0 0x7f31f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320e59c640 1 -- 172.21.15.29:0/1744929485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f8003520 con 0x7f31f80039c0 2023-10-16T15:53:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.341+0000 7f320ed9d640 1 -- 172.21.15.29:0/1744929485 reap_dead start 2023-10-16T15:53:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f320e59c640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f31f8003de0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f3200001e50 tx=0x7f31f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3502eea1d2f69a81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200003b70 con 0x7f31f80039c0 2023-10-16T15:53:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3200003cd0 con 0x7f31f80039c0 2023-10-16T15:53:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f320d59a640 1 -- 172.21.15.29:0/1744929485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200005310 con 0x7f31f80039c0 2023-10-16T15:53:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f3214bc5640 1 -- 172.21.15.29:0/1744929485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 msgr2=0x7f31f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f3214bc5640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f31f8003de0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f3200001e50 tx=0x7f31f80057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f3214bc5640 1 -- 172.21.15.29:0/1744929485 shutdown_connections 2023-10-16T15:53:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f3214bc5640 1 --2- 172.21.15.29:0/1744929485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f31f8003de0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.343+0000 7f3214bc5640 1 -- 172.21.15.29:0/1744929485 >> 172.21.15.29:0/1744929485 conn(0x7f32100ce9b0 msgr2=0x7f32100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.344+0000 7f3214bc5640 1 -- 172.21.15.29:0/1744929485 shutdown_connections 2023-10-16T15:53:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.344+0000 7f3214bc5640 1 -- 172.21.15.29:0/1744929485 wait complete. 2023-10-16T15:53:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.345+0000 7f3214bc5640 1 Processor -- start 2023-10-16T15:53:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.346+0000 7f3214bc5640 1 -- start start 2023-10-16T15:53:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.346+0000 7f3214bc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f32101e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.346+0000 7f3214bc5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3210159220 0x7f32101e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.346+0000 7f3214bc5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32101e94b0 0x7f321014a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f3214bc5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3210159080 con 0x7f31f80039c0 2023-10-16T15:53:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f3214bc5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100c6e10 con 0x7f3210159220 2023-10-16T15:53:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f3214bc5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32100c7990 con 0x7f32101e94b0 2023-10-16T15:53:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f32101e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f32101e5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53714/0 (socket says 172.21.15.29:53714) 2023-10-16T15:53:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 -- 172.21.15.29:0/450601684 learned_addr learned my addr 172.21.15.29:0/450601684 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320dd9b640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3210159220 0x7f32101e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320ed9d640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32101e94b0 0x7f321014a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 -- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32101e94b0 msgr2=0x7f321014a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32101e94b0 0x7f321014a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 -- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3210159220 msgr2=0x7f32101e54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3210159220 0x7f32101e54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320e59c640 1 -- 172.21.15.29:0/450601684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32101e59f0 con 0x7f31f80039c0 2023-10-16T15:53:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.347+0000 7f320ed9d640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32101e94b0 0x7f321014a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.348+0000 7f320e59c640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f32101e5d80 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f3200001f70 tx=0x7f32000014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.348+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320003e070 con 0x7f31f80039c0 2023-10-16T15:53:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.348+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3200006b50 con 0x7f31f80039c0 2023-10-16T15:53:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.348+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200007020 con 0x7f31f80039c0 2023-10-16T15:53:18.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.348+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32100c7060 con 0x7f31f80039c0 2023-10-16T15:53:18.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.348+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32100c7620 con 0x7f31f80039c0 2023-10-16T15:53:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.350+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321004e160 con 0x7f31f80039c0 2023-10-16T15:53:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.351+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3200007180 con 0x7f31f80039c0 2023-10-16T15:53:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.352+0000 7f31feffd640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f31f006af80 0x7f31f006d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.352+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f32000aafb0 con 0x7f31f80039c0 2023-10-16T15:53:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.353+0000 7f320dd9b640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f31f006af80 0x7f31f006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.354+0000 7f320dd9b640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f31f006af80 0x7f31f006d440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f32101ea020 tx=0x7f3204008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.357+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3200075f80 con 0x7f31f80039c0 2023-10-16T15:53:18.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.550+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f321006bf20 con 0x7f31f80039c0 2023-10-16T15:53:18.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.551+0000 7f31feffd640 1 -- 172.21.15.29:0/450601684 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32000795a0 con 0x7f31f80039c0 2023-10-16T15:53:18.552 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:53:18.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f31f006af80 msgr2=0x7f31f006d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f31f006af80 0x7f31f006d440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f32101ea020 tx=0x7f3204008040 comp rx=0 tx=0).stop 2023-10-16T15:53:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 msgr2=0x7f32101e5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f32101e5d80 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f3200001f70 tx=0x7f32000014a0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 shutdown_connections 2023-10-16T15:53:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f31f006af80 0x7f31f006d440 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f32101e94b0 0x7f321014a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3210159220 0x7f32101e54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 --2- 172.21.15.29:0/450601684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31f80039c0 0x7f32101e5d80 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.553+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 >> 172.21.15.29:0/450601684 conn(0x7f32100ce9b0 msgr2=0x7f321006b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:18.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.554+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 shutdown_connections 2023-10-16T15:53:18.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.554+0000 7f3214bc5640 1 -- 172.21.15.29:0/450601684 wait complete. 2023-10-16T15:53:18.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T15:53:18.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.805+0000 7f5cc852c640 1 Processor -- start 2023-10-16T15:53:18.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.805+0000 7f5cc852c640 1 -- start start 2023-10-16T15:53:18.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.807+0000 7f5cc852c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5cc015aaa0 0x7f5cc015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.807+0000 7f5cc852c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 0x7f5cc00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.807+0000 7f5cc852c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc015ed60 con 0x7f5cc00ccaa0 2023-10-16T15:53:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.807+0000 7f5cc852c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc006b810 con 0x7f5cc015aaa0 2023-10-16T15:53:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.807+0000 7f5cc852c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00707e0 con 0x7f5cc015ae80 2023-10-16T15:53:18.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 0x7f5cc00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc62c8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5cc015aaa0 0x7f5cc015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 0x7f5cc00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:54944/0 (socket says 172.21.15.29:54944) 2023-10-16T15:53:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 -- 172.21.15.29:0/2015431821 learned_addr learned my addr 172.21.15.29:0/2015431821 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 -- 172.21.15.29:0/2015431821 >> v2:172.21.15.170:3300/0 conn(0x7f5cc015aaa0 msgr2=0x7f5cc015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 --2- 172.21.15.29:0/2015431821 >> v2:172.21.15.170:3300/0 conn(0x7f5cc015aaa0 0x7f5cc015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 -- 172.21.15.29:0/2015431821 >> v1:172.21.15.170:6789/0 conn(0x7f5cc00ccaa0 legacy=0x7f5cc0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.808+0000 7f5cc5ac7640 1 -- 172.21.15.29:0/2015431821 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc015ef90 con 0x7f5cc015ae80 2023-10-16T15:53:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.810+0000 7f5cc5ac7640 1 --2- 172.21.15.29:0/2015431821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 0x7f5cc00cc560 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009870 tx=0x7f5cb002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f47bf12c1a7782 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:18.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.810+0000 7f5cc52c6640 1 -- 172.21.15.29:0/2015431821 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0030840 con 0x7f5cc015ae80 2023-10-16T15:53:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.810+0000 7f5cc52c6640 1 -- 172.21.15.29:0/2015431821 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb00309a0 con 0x7f5cc015ae80 2023-10-16T15:53:18.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.810+0000 7f5cc52c6640 1 -- 172.21.15.29:0/2015431821 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0030d30 con 0x7f5cc015ae80 2023-10-16T15:53:18.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 -- 172.21.15.29:0/2015431821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 msgr2=0x7f5cc00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 --2- 172.21.15.29:0/2015431821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 0x7f5cc00cc560 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009870 tx=0x7f5cb002b980 comp rx=0 tx=0).stop 2023-10-16T15:53:18.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 -- 172.21.15.29:0/2015431821 shutdown_connections 2023-10-16T15:53:18.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 --2- 172.21.15.29:0/2015431821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015ae80 0x7f5cc00cc560 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 --2- 172.21.15.29:0/2015431821 >> v2:172.21.15.170:3300/0 conn(0x7f5cc015aaa0 0x7f5cc015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 -- 172.21.15.29:0/2015431821 >> 172.21.15.29:0/2015431821 conn(0x7f5cc00cb410 msgr2=0x7f5cc00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:18.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 -- 172.21.15.29:0/2015431821 shutdown_connections 2023-10-16T15:53:18.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.811+0000 7f5cc852c640 1 -- 172.21.15.29:0/2015431821 wait complete. 2023-10-16T15:53:18.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.813+0000 7f5cc852c640 1 Processor -- start 2023-10-16T15:53:18.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.814+0000 7f5cc852c640 1 -- start start 2023-10-16T15:53:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc852c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc00ccaa0 0x7f5cc0149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc852c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015aaa0 0x7f5cc00c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc852c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 0x7f5cc015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc852c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc0154270 con 0x7f5cc015ae80 2023-10-16T15:53:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc852c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc015c7c0 con 0x7f5cc00ccaa0 2023-10-16T15:53:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc852c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00c29b0 con 0x7f5cc015aaa0 2023-10-16T15:53:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc6ac9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 0x7f5cc015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc6ac9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 0x7f5cc015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53720/0 (socket says 172.21.15.29:53720) 2023-10-16T15:53:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.815+0000 7f5cc6ac9640 1 -- 172.21.15.29:0/4155800918 learned_addr learned my addr 172.21.15.29:0/4155800918 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc62c8640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc00ccaa0 0x7f5cc0149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc5ac7640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015aaa0 0x7f5cc00c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc6ac9640 1 -- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015aaa0 msgr2=0x7f5cc00c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc6ac9640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015aaa0 0x7f5cc00c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc6ac9640 1 -- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc00ccaa0 msgr2=0x7f5cc0149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc6ac9640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc00ccaa0 0x7f5cc0149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc6ac9640 1 -- 172.21.15.29:0/4155800918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb00094d0 con 0x7f5cc015ae80 2023-10-16T15:53:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.816+0000 7f5cc5ac7640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015aaa0 0x7f5cc00c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.817+0000 7f5cc6ac9640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 0x7f5cc015cef0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800c6b0 tx=0x7f5cb800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.817+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8003b70 con 0x7f5cc015ae80 2023-10-16T15:53:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.817+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb8003cf0 con 0x7f5cc015ae80 2023-10-16T15:53:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.817+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8003ea0 con 0x7f5cc015ae80 2023-10-16T15:53:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.817+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc0155720 con 0x7f5cc015ae80 2023-10-16T15:53:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.817+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc015c200 con 0x7f5cc015ae80 2023-10-16T15:53:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.818+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc004e160 con 0x7f5cc015ae80 2023-10-16T15:53:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.819+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5cb8005310 con 0x7f5cc015ae80 2023-10-16T15:53:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.820+0000 7f5cb6ffd640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5ca406af90 0x7f5ca406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.820+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f5cb8089510 con 0x7f5cc015ae80 2023-10-16T15:53:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.821+0000 7f5cc62c8640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5ca406af90 0x7f5ca406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.822+0000 7f5cc62c8640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5ca406af90 0x7f5ca406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009b40 tx=0x7f5cbc009190 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:18.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:18.825+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5cb8054440 con 0x7f5cc015ae80 2023-10-16T15:53:19.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.090+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f5cc00c2af0 con 0x7f5cc015ae80 2023-10-16T15:53:19.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.633+0000 7f5cb6ffd640 1 -- 172.21.15.29:0/4155800918 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3162) v1 ==== 189+0+0 (secure 0 0 0) 0x7f5cb8057a60 con 0x7f5cc015ae80 2023-10-16T15:53:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.636+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5ca406af90 msgr2=0x7f5ca406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.636+0000 7f5cc852c640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5ca406af90 0x7f5ca406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009b40 tx=0x7f5cbc009190 comp rx=0 tx=0).stop 2023-10-16T15:53:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.636+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 msgr2=0x7f5cc015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.636+0000 7f5cc852c640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 0x7f5cc015cef0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800c6b0 tx=0x7f5cb800cb70 comp rx=0 tx=0).stop 2023-10-16T15:53:19.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 shutdown_connections 2023-10-16T15:53:19.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f5ca406af90 0x7f5ca406d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cc015ae80 0x7f5cc015cef0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cc015aaa0 0x7f5cc00c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 --2- 172.21.15.29:0/4155800918 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cc00ccaa0 0x7f5cc0149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 >> 172.21.15.29:0/4155800918 conn(0x7f5cc00cb410 msgr2=0x7f5cc00b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:19.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 shutdown_connections 2023-10-16T15:53:19.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.637+0000 7f5cc852c640 1 -- 172.21.15.29:0/4155800918 wait complete. 2023-10-16T15:53:19.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T15:53:19.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.897+0000 7f49fbea9640 1 Processor -- start 2023-10-16T15:53:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.897+0000 7f49fbea9640 1 -- start start 2023-10-16T15:53:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fbea9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159ab0 0x7f49f40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fbea9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f49f4159e90 0x7f49f406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fbea9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49f406c140 0x7f49f406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fbea9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f406cb10 con 0x7f49f406c140 2023-10-16T15:53:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fbea9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f415d890 con 0x7f49f4159e90 2023-10-16T15:53:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fbea9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40cd1f0 con 0x7f49f4159ab0 2023-10-16T15:53:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.899+0000 7f49fa446640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49f406c140 0x7f49f406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49f406c140 0x7f49f406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53724/0 (socket says 172.21.15.29:53724) 2023-10-16T15:53:19.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 -- 172.21.15.29:0/2636235140 learned_addr learned my addr 172.21.15.29:0/2636235140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:19.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49f9c45640 1 --2- 172.21.15.29:0/2636235140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159ab0 0x7f49f40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 -- 172.21.15.29:0/2636235140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159ab0 msgr2=0x7f49f40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 --2- 172.21.15.29:0/2636235140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159ab0 0x7f49f40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 -- 172.21.15.29:0/2636235140 >> v2:172.21.15.170:3300/0 conn(0x7f49f4159e90 msgr2=0x7f49f406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 --2- 172.21.15.29:0/2636235140 >> v2:172.21.15.170:3300/0 conn(0x7f49f4159e90 0x7f49f406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.900+0000 7f49fa446640 1 -- 172.21.15.29:0/2636235140 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40cd4d0 con 0x7f49f406c140 2023-10-16T15:53:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.901+0000 7f49fa446640 1 --2- 172.21.15.29:0/2636235140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f406c5d0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009e50 tx=0x7f49ec02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fd6b4b952c6252 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.901+0000 7f49f8c43640 1 -- 172.21.15.29:0/2636235140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec02c950 con 0x7f49f406c140 2023-10-16T15:53:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.901+0000 7f49f8c43640 1 -- 172.21.15.29:0/2636235140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec02cab0 con 0x7f49f406c140 2023-10-16T15:53:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.901+0000 7f49f8c43640 1 -- 172.21.15.29:0/2636235140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec02ce40 con 0x7f49f406c140 2023-10-16T15:53:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 -- 172.21.15.29:0/2636235140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 msgr2=0x7f49f406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:19.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2636235140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f406c5d0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009e50 tx=0x7f49ec02bdb0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 -- 172.21.15.29:0/2636235140 shutdown_connections 2023-10-16T15:53:19.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2636235140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f406c5d0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2636235140 >> v2:172.21.15.170:3300/0 conn(0x7f49f4159e90 0x7f49f406fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2636235140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159ab0 0x7f49f40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 -- 172.21.15.29:0/2636235140 >> 172.21.15.29:0/2636235140 conn(0x7f49f40ce9b0 msgr2=0x7f49f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:19.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 -- 172.21.15.29:0/2636235140 shutdown_connections 2023-10-16T15:53:19.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.902+0000 7f49fbea9640 1 -- 172.21.15.29:0/2636235140 wait complete. 2023-10-16T15:53:19.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.904+0000 7f49fbea9640 1 Processor -- start 2023-10-16T15:53:19.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.904+0000 7f49fbea9640 1 -- start start 2023-10-16T15:53:19.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49fbea9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f414ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49fbea9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49f4159ab0 0x7f49f40c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49fbea9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159e90 0x7f49f40c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49fbea9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f41e2bc0 con 0x7f49f406c140 2023-10-16T15:53:19.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49fbea9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f41e2dd0 con 0x7f49f4159ab0 2023-10-16T15:53:19.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49fbea9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f41e30b0 con 0x7f49f4159e90 2023-10-16T15:53:19.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49f9c45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f414ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:19.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49f9c45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f414ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53736/0 (socket says 172.21.15.29:53736) 2023-10-16T15:53:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.905+0000 7f49f9c45640 1 -- 172.21.15.29:0/2688399079 learned_addr learned my addr 172.21.15.29:0/2688399079 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49fa446640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159e90 0x7f49f40c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9444640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49f4159ab0 0x7f49f40c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9c45640 1 -- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159e90 msgr2=0x7f49f40c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9c45640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159e90 0x7f49f40c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9c45640 1 -- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49f4159ab0 msgr2=0x7f49f40c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:19.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9c45640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49f4159ab0 0x7f49f40c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:19.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9c45640 1 -- 172.21.15.29:0/2688399079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec009b30 con 0x7f49f406c140 2023-10-16T15:53:19.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49f9c45640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f414ad80 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f49f000d800 tx=0x7f49f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:19.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.906+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f000b6c0 con 0x7f49f406c140 2023-10-16T15:53:19.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.907+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f000b820 con 0x7f49f406c140 2023-10-16T15:53:19.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.907+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f41e3390 con 0x7f49f406c140 2023-10-16T15:53:19.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.907+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f000bbb0 con 0x7f49f406c140 2023-10-16T15:53:19.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.907+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f414b2c0 con 0x7f49f406c140 2023-10-16T15:53:19.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.908+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f406cf00 con 0x7f49f406c140 2023-10-16T15:53:19.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.909+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f49f000bd10 con 0x7f49f406c140 2023-10-16T15:53:19.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.910+0000 7f49ea7fc640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49d806af90 0x7f49d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:19.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.910+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f49f0089c70 con 0x7f49f406c140 2023-10-16T15:53:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.911+0000 7f49f9444640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49d806af90 0x7f49d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.912+0000 7f49f9444640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49d806af90 0x7f49d806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49f415d820 tx=0x7f49e4008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:19.914+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49f0054cd0 con 0x7f49f406c140 2023-10-16T15:53:20.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.110+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f49f404e160 con 0x7f49f406c140 2023-10-16T15:53:20.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.116+0000 7f49ea7fc640 1 -- 172.21.15.29:0/2688399079 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f49f00582f0 con 0x7f49f406c140 2023-10-16T15:53:20.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49d806af90 msgr2=0x7f49d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49d806af90 0x7f49d806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49f415d820 tx=0x7f49e4008040 comp rx=0 tx=0).stop 2023-10-16T15:53:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 msgr2=0x7f49f414ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f414ad80 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f49f000d800 tx=0x7f49f000dcc0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 shutdown_connections 2023-10-16T15:53:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49d806af90 0x7f49d806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49f4159e90 0x7f49f40c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49f4159ab0 0x7f49f40c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.118+0000 7f49fbea9640 1 --2- 172.21.15.29:0/2688399079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49f406c140 0x7f49f414ad80 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.119+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 >> 172.21.15.29:0/2688399079 conn(0x7f49f40ce9b0 msgr2=0x7f49f40b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.119+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 shutdown_connections 2023-10-16T15:53:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.119+0000 7f49fbea9640 1 -- 172.21.15.29:0/2688399079 wait complete. 2023-10-16T15:53:20.140 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:53:20.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:53:20.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:53:20.391 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:53:20.392 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:53:20.392 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:20.455 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:53:20.456 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:53:20.456 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:20.517 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:53:20.517 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:53:20.518 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:20.578 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:53:20.578 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:53:20.579 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:20.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:53:20.639 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T15:53:20.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:20.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.872+0000 7f0bb9b5a640 1 Processor -- start 2023-10-16T15:53:20.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.872+0000 7f0bb9b5a640 1 -- start start 2023-10-16T15:53:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.873+0000 7f0bb9b5a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0bb40b3070 0x7f0bb40cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.873+0000 7f0bb9b5a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0bb40d1b10 0x7f0bb40cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.873+0000 7f0bb9b5a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb40b3940 con 0x7f0bb40b3e70 2023-10-16T15:53:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.873+0000 7f0bb9b5a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40b45e0 con 0x7f0bb40b3070 2023-10-16T15:53:20.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.873+0000 7f0bb9b5a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40b3b90 con 0x7f0bb40d1b10 2023-10-16T15:53:20.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0bb40b3070 0x7f0bb40cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:20.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0bb40b3070 0x7f0bb40cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53748/0 (socket says 172.21.15.29:53748) 2023-10-16T15:53:20.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 -- 172.21.15.29:0/2473053007 learned_addr learned my addr 172.21.15.29:0/2473053007 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 -- 172.21.15.29:0/2473053007 >> v2:172.21.15.170:3300/0 conn(0x7f0bb40d1b10 msgr2=0x7f0bb40cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 --2- 172.21.15.29:0/2473053007 >> v2:172.21.15.170:3300/0 conn(0x7f0bb40d1b10 0x7f0bb40cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 -- 172.21.15.29:0/2473053007 >> v1:172.21.15.170:6789/0 conn(0x7f0bb40b3e70 legacy=0x7f0bb40c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:20.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.874+0000 7f0bb37fe640 1 -- 172.21.15.29:0/2473053007 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb40d3840 con 0x7f0bb40b3070 2023-10-16T15:53:20.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.875+0000 7f0bb37fe640 1 --2- 172.21.15.29:0/2473053007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40b3070 0x7f0bb40cf500 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f0bb408c1c0 tx=0x7f0ba402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e025ddfe2d38d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:20.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.876+0000 7f0bb2ffd640 1 -- 172.21.15.29:0/2473053007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4004230 con 0x7f0bb40b3070 2023-10-16T15:53:20.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.876+0000 7f0bb2ffd640 1 -- 172.21.15.29:0/2473053007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba4004390 con 0x7f0bb40b3070 2023-10-16T15:53:20.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.876+0000 7f0bb2ffd640 1 -- 172.21.15.29:0/2473053007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4003810 con 0x7f0bb40b3070 2023-10-16T15:53:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.876+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/2473053007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40b3070 msgr2=0x7f0bb40cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.876+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/2473053007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40b3070 0x7f0bb40cf500 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f0bb408c1c0 tx=0x7f0ba402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.877+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/2473053007 shutdown_connections 2023-10-16T15:53:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.877+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/2473053007 >> v2:172.21.15.170:3300/0 conn(0x7f0bb40d1b10 0x7f0bb40cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.877+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/2473053007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40b3070 0x7f0bb40cf500 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.877+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/2473053007 >> 172.21.15.29:0/2473053007 conn(0x7f0bb40d2d20 msgr2=0x7f0bb40d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.877+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/2473053007 shutdown_connections 2023-10-16T15:53:20.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.877+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/2473053007 wait complete. 2023-10-16T15:53:20.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.879+0000 7f0bb9b5a640 1 Processor -- start 2023-10-16T15:53:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.879+0000 7f0bb9b5a640 1 -- start start 2023-10-16T15:53:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb9b5a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bb40b3070 0x7f0bb40b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb9b5a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bb40b3e70 0x7f0bb415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb9b5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 0x7f0bb40cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb9b5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb415c720 con 0x7f0bb40d1b10 2023-10-16T15:53:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb9b5a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40b5ff0 con 0x7f0bb40b3e70 2023-10-16T15:53:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb9b5a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40b62d0 con 0x7f0bb40b3070 2023-10-16T15:53:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 0x7f0bb40cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 0x7f0bb40cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53764/0 (socket says 172.21.15.29:53764) 2023-10-16T15:53:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb3fff640 1 -- 172.21.15.29:0/1401895330 learned_addr learned my addr 172.21.15.29:0/1401895330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:20.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.880+0000 7f0bb37fe640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bb40b3070 0x7f0bb40b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0babfff640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bb40b3e70 0x7f0bb415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb3fff640 1 -- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bb40b3070 msgr2=0x7f0bb40b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb3fff640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bb40b3070 0x7f0bb40b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb3fff640 1 -- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bb40b3e70 msgr2=0x7f0bb415c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:20.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb3fff640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bb40b3e70 0x7f0bb415c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb3fff640 1 -- 172.21.15.29:0/1401895330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4009460 con 0x7f0bb40d1b10 2023-10-16T15:53:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0babfff640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bb40b3e70 0x7f0bb415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb3fff640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 0x7f0bb40cab50 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00b970 tx=0x7f0bac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00cb40 con 0x7f0bb40d1b10 2023-10-16T15:53:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00ccc0 con 0x7f0bb40d1b10 2023-10-16T15:53:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb40b65b0 con 0x7f0bb40d1b10 2023-10-16T15:53:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.881+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac014680 con 0x7f0bb40d1b10 2023-10-16T15:53:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.882+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb40b6b70 con 0x7f0bb40d1b10 2023-10-16T15:53:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.883+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb40b6d90 con 0x7f0bb40d1b10 2023-10-16T15:53:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.884+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0bac01e440 con 0x7f0bb40d1b10 2023-10-16T15:53:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.884+0000 7f0bb0ff9640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0b9406af90 0x7f0b9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.885+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f0bac089750 con 0x7f0bb40d1b10 2023-10-16T15:53:20.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.885+0000 7f0bb37fe640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0b9406af90 0x7f0b9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:20.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.886+0000 7f0bb37fe640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0b9406af90 0x7f0b9406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4035000 tx=0x7f0ba4008030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:20.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:20.889+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0bac054680 con 0x7f0bb40d1b10 2023-10-16T15:53:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.119+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bb404e160 con 0x7f0bb40d1b10 2023-10-16T15:53:21.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.121+0000 7f0bb0ff9640 1 -- 172.21.15.29:0/1401895330 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3708 (secure 0 0 0) 0x7f0bac019090 con 0x7f0bb40d1b10 2023-10-16T15:53:21.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:21.123 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":102,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/3363552284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":3363552284},{"type":"v1","addr":"172.21.15.170:6837","nonce":3363552284}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":7544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2880748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2880748068},{"type":"v1","addr":"172.21.15.29:6835","nonce":2880748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":8401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/2635272232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2635272232},{"type":"v1","addr":"172.21.15.170:6836","nonce":2635272232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95},{"gid":10728,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/2190963296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2190963296},{"type":"v1","addr":"172.21.15.29:6833","nonce":2190963296}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[]} 2023-10-16T15:53:21.123 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 102 2023-10-16T15:53:21.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0b9406af90 msgr2=0x7f0b9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:21.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0b9406af90 0x7f0b9406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4035000 tx=0x7f0ba4008030 comp rx=0 tx=0).stop 2023-10-16T15:53:21.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 msgr2=0x7f0bb40cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:21.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 0x7f0bb40cab50 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00b970 tx=0x7f0bac00be30 comp rx=0 tx=0).stop 2023-10-16T15:53:21.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 shutdown_connections 2023-10-16T15:53:21.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0b9406af90 0x7f0b9406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb40d1b10 0x7f0bb40cab50 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0bb40b3e70 0x7f0bb415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 --2- 172.21.15.29:0/1401895330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0bb40b3070 0x7f0bb40b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 >> 172.21.15.29:0/1401895330 conn(0x7f0bb40d2d20 msgr2=0x7f0bb406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:21.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 shutdown_connections 2023-10-16T15:53:21.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.124+0000 7f0bb9b5a640 1 -- 172.21.15.29:0/1401895330 wait complete. 2023-10-16T15:53:21.146 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T15:53:21.147 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T15:53:21.147 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:53:21.147 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:53:21.147 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T15:53:21.148 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T15:53:21.148 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:53:21.148 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:53:21.148 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T15:53:21.149 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T15:53:21.149 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:53:21.149 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:53:21.149 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T15:53:21.150 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T15:53:21.150 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:53:21.150 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:53:21.170 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:53:21.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T15:53:21.171 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:53:21.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T15:53:21.173 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:53:21.173 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T15:53:21.174 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:53:21.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T15:53:21.219 INFO:tasks.ceph.mds.a:Started 2023-10-16T15:53:21.220 DEBUG:teuthology.parallel:result is None 2023-10-16T15:53:21.222 INFO:tasks.ceph.mds.d:Started 2023-10-16T15:53:21.223 DEBUG:teuthology.parallel:result is None 2023-10-16T15:53:21.226 INFO:tasks.ceph.mds.c:Started 2023-10-16T15:53:21.227 DEBUG:teuthology.parallel:result is None 2023-10-16T15:53:21.229 INFO:tasks.ceph.mds.b:Started 2023-10-16T15:53:21.230 DEBUG:teuthology.parallel:result is None 2023-10-16T15:53:21.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T15:53:21.383 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:53:21.388 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:53:21.389 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T15:53:21.390 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:53:21.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.426+0000 7fe00954e640 1 Processor -- start 2023-10-16T15:53:21.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.426+0000 7fe00954e640 1 -- start start 2023-10-16T15:53:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fe00954e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00406b8c0 0x7fe00406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fe00954e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe00415d130 0x7fe0040cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fe00954e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0040cd390 con 0x7fe00406b1a0 2023-10-16T15:53:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fe00954e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00406c1e0 con 0x7fe00415d130 2023-10-16T15:53:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fe00954e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040cc2f0 con 0x7fe00406b8c0 2023-10-16T15:53:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fdffbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe00415d130 0x7fe0040cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fdffbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe00415d130 0x7fe0040cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53800/0 (socket says 172.21.15.29:53800) 2023-10-16T15:53:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fdffbfff640 1 -- 172.21.15.29:0/874809707 learned_addr learned my addr 172.21.15.29:0/874809707 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.428+0000 7fe002ffd640 1 --2- 172.21.15.29:0/874809707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00406b8c0 0x7fe00406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.429+0000 7fdffbfff640 1 -- 172.21.15.29:0/874809707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00406b8c0 msgr2=0x7fe00406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.429+0000 7fdffbfff640 1 --2- 172.21.15.29:0/874809707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00406b8c0 0x7fe00406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.429+0000 7fdffbfff640 1 -- 172.21.15.29:0/874809707 >> v1:172.21.15.170:6789/0 conn(0x7fe00406b1a0 legacy=0x7fe0040cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.429+0000 7fdffbfff640 1 -- 172.21.15.29:0/874809707 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0040cc5d0 con 0x7fe00415d130 2023-10-16T15:53:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.430+0000 7fdffbfff640 1 --2- 172.21.15.29:0/874809707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00415d130 0x7fe0040cbaa0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009870 tx=0x7fdfec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d895811c9c4c5663 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.430+0000 7fe0027fc640 1 -- 172.21.15.29:0/874809707 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec030840 con 0x7fe00415d130 2023-10-16T15:53:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.430+0000 7fe0027fc640 1 -- 172.21.15.29:0/874809707 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec0309a0 con 0x7fe00415d130 2023-10-16T15:53:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.430+0000 7fe0027fc640 1 -- 172.21.15.29:0/874809707 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec030d30 con 0x7fe00415d130 2023-10-16T15:53:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 -- 172.21.15.29:0/874809707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00415d130 msgr2=0x7fe0040cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 --2- 172.21.15.29:0/874809707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00415d130 0x7fe0040cbaa0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009870 tx=0x7fdfec02b980 comp rx=0 tx=0).stop 2023-10-16T15:53:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 -- 172.21.15.29:0/874809707 shutdown_connections 2023-10-16T15:53:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 --2- 172.21.15.29:0/874809707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00415d130 0x7fe0040cbaa0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 --2- 172.21.15.29:0/874809707 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00406b8c0 0x7fe00406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 -- 172.21.15.29:0/874809707 >> 172.21.15.29:0/874809707 conn(0x7fe0040d0ad0 msgr2=0x7fe0040d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 -- 172.21.15.29:0/874809707 shutdown_connections 2023-10-16T15:53:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.431+0000 7fe00954e640 1 -- 172.21.15.29:0/874809707 wait complete. 2023-10-16T15:53:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.433+0000 7fe00954e640 1 Processor -- start 2023-10-16T15:53:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.433+0000 7fe00954e640 1 -- start start 2023-10-16T15:53:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fe00954e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe00406b1a0 0x7fe00409b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fe00954e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 0x7fe004158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fe00954e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00415d130 0x7fe00406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fe00954e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00409e5c0 con 0x7fe00406b8c0 2023-10-16T15:53:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fe00954e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0041f6780 con 0x7fe00406b1a0 2023-10-16T15:53:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fe00954e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0041f64a0 con 0x7fe00415d130 2023-10-16T15:53:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 0x7fe004158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 0x7fe004158ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53816/0 (socket says 172.21.15.29:53816) 2023-10-16T15:53:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 -- 172.21.15.29:0/2160082884 learned_addr learned my addr 172.21.15.29:0/2160082884 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 -- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00415d130 msgr2=0x7fe00406e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00415d130 0x7fe00406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 -- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe00406b1a0 msgr2=0x7fe00409b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe00406b1a0 0x7fe00409b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.434+0000 7fdffbfff640 1 -- 172.21.15.29:0/2160082884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec0094d0 con 0x7fe00406b8c0 2023-10-16T15:53:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.435+0000 7fdffbfff640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 0x7fe004158ec0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009870 tx=0x7fdfec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.435+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec039a90 con 0x7fe00406b8c0 2023-10-16T15:53:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.435+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec039c10 con 0x7fe00406b8c0 2023-10-16T15:53:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.435+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0433f0 con 0x7fe00406b8c0 2023-10-16T15:53:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.435+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00409a870 con 0x7fe00406b8c0 2023-10-16T15:53:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.435+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00409ae30 con 0x7fe00406b8c0 2023-10-16T15:53:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.437+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00404e160 con 0x7fe00406b8c0 2023-10-16T15:53:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.437+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdfec042390 con 0x7fe00406b8c0 2023-10-16T15:53:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.438+0000 7fdffb7fe640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdfe406af90 0x7fdfe406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.438+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fdfec0abaa0 con 0x7fe00406b8c0 2023-10-16T15:53:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.439+0000 7fe002ffd640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdfe406af90 0x7fdfe406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.439+0000 7fe002ffd640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdfe406af90 0x7fdfe406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdff4006a30 tx=0x7fdff40069c0 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.442+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdfec0768c0 con 0x7fe00406b8c0 2023-10-16T15:53:21.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:21.650+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe00406b6a0 con 0x7fe00406b8c0 2023-10-16T15:53:22.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.141+0000 7fdffb7fe640 1 -- 172.21.15.29:0/2160082884 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v219) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdfec034080 con 0x7fe00406b8c0 2023-10-16T15:53:22.144 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T15:53:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.145+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdfe406af90 msgr2=0x7fdfe406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdfe406af90 0x7fdfe406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdff4006a30 tx=0x7fdff40069c0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 msgr2=0x7fe004158ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 0x7fe004158ec0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009870 tx=0x7fdfec004730 comp rx=0 tx=0).stop 2023-10-16T15:53:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 shutdown_connections 2023-10-16T15:53:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdfe406af90 0x7fdfe406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe00406b8c0 0x7fe004158ec0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe00415d130 0x7fe00406e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 --2- 172.21.15.29:0/2160082884 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe00406b1a0 0x7fe00409b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 >> 172.21.15.29:0/2160082884 conn(0x7fe0040d0ad0 msgr2=0x7fe0040ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:22.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 shutdown_connections 2023-10-16T15:53:22.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.146+0000 7fe00954e640 1 -- 172.21.15.29:0/2160082884 wait complete. 2023-10-16T15:53:22.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T15:53:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.397+0000 7fb67776c640 1 Processor -- start 2023-10-16T15:53:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.397+0000 7fb67776c640 1 -- start start 2023-10-16T15:53:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb67776c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb67015aaa0 0x7fb67015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb67776c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6700ce390 0x7fb670070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb67776c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67015ed60 con 0x7fb67015ae80 2023-10-16T15:53:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb67776c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67006b810 con 0x7fb67015aaa0 2023-10-16T15:53:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb67776c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700707e0 con 0x7fb6700ce390 2023-10-16T15:53:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb675508640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb67015aaa0 0x7fb67015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb675508640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb67015aaa0 0x7fb67015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53832/0 (socket says 172.21.15.29:53832) 2023-10-16T15:53:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.399+0000 7fb675508640 1 -- 172.21.15.29:0/4111275495 learned_addr learned my addr 172.21.15.29:0/4111275495 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.400+0000 7fb675508640 1 -- 172.21.15.29:0/4111275495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6700ce390 msgr2=0x7fb670070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:22.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.400+0000 7fb675508640 1 --2- 172.21.15.29:0/4111275495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6700ce390 0x7fb670070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.400+0000 7fb675508640 1 -- 172.21.15.29:0/4111275495 >> v1:172.21.15.170:6789/0 conn(0x7fb67015ae80 legacy=0x7fb6700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.400+0000 7fb675508640 1 -- 172.21.15.29:0/4111275495 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67015ef90 con 0x7fb67015aaa0 2023-10-16T15:53:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.401+0000 7fb675508640 1 --2- 172.21.15.29:0/4111275495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67015aaa0 0x7fb67015e820 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fb6700cd990 tx=0x7fb66c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924c1a9ebba17092 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.401+0000 7fb667fff640 1 -- 172.21.15.29:0/4111275495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c031640 con 0x7fb67015aaa0 2023-10-16T15:53:22.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.401+0000 7fb667fff640 1 -- 172.21.15.29:0/4111275495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c0317a0 con 0x7fb67015aaa0 2023-10-16T15:53:22.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.401+0000 7fb667fff640 1 -- 172.21.15.29:0/4111275495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c031b30 con 0x7fb67015aaa0 2023-10-16T15:53:22.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 -- 172.21.15.29:0/4111275495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67015aaa0 msgr2=0x7fb67015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 --2- 172.21.15.29:0/4111275495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67015aaa0 0x7fb67015e820 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fb6700cd990 tx=0x7fb66c02b8f0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 -- 172.21.15.29:0/4111275495 shutdown_connections 2023-10-16T15:53:22.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 --2- 172.21.15.29:0/4111275495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb6700ce390 0x7fb670070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 --2- 172.21.15.29:0/4111275495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb67015aaa0 0x7fb67015e820 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 -- 172.21.15.29:0/4111275495 >> 172.21.15.29:0/4111275495 conn(0x7fb6700cb410 msgr2=0x7fb6700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 -- 172.21.15.29:0/4111275495 shutdown_connections 2023-10-16T15:53:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.402+0000 7fb67776c640 1 -- 172.21.15.29:0/4111275495 wait complete. 2023-10-16T15:53:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.404+0000 7fb67776c640 1 Processor -- start 2023-10-16T15:53:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.404+0000 7fb67776c640 1 -- start start 2023-10-16T15:53:22.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb67776c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 0x7fb6700ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb67776c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67015aaa0 0x7fb6700b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb67776c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67015ae80 0x7fb6700b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb67776c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700c7b50 con 0x7fb6700ce390 2023-10-16T15:53:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb67776c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701f2eb0 con 0x7fb67015aaa0 2023-10-16T15:53:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb67776c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700c7d60 con 0x7fb67015ae80 2023-10-16T15:53:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb675508640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 0x7fb6700ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb675508640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 0x7fb6700ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53840/0 (socket says 172.21.15.29:53840) 2023-10-16T15:53:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.405+0000 7fb675508640 1 -- 172.21.15.29:0/402057758 learned_addr learned my addr 172.21.15.29:0/402057758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675d09640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67015ae80 0x7fb6700b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb674d07640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67015aaa0 0x7fb6700b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675508640 1 -- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67015ae80 msgr2=0x7fb6700b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675508640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67015ae80 0x7fb6700b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675508640 1 -- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67015aaa0 msgr2=0x7fb6700b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675508640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67015aaa0 0x7fb6700b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675508640 1 -- 172.21.15.29:0/402057758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0094d0 con 0x7fb6700ce390 2023-10-16T15:53:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb675508640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 0x7fb6700ca9e0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fb66c02b930 tx=0x7fb66c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.406+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c03ac40 con 0x7fb6700ce390 2023-10-16T15:53:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.407+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c03ada0 con 0x7fb6700ce390 2023-10-16T15:53:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.407+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c044630 con 0x7fb6700ce390 2023-10-16T15:53:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.407+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6700c8040 con 0x7fb6700ce390 2023-10-16T15:53:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.407+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6700c8600 con 0x7fb6700ce390 2023-10-16T15:53:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.408+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6700c87f0 con 0x7fb6700ce390 2023-10-16T15:53:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.409+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb66c02f050 con 0x7fb6700ce390 2023-10-16T15:53:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.410+0000 7fb665ffb640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb65406af90 0x7fb65406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.410+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fb66c0ab6a0 con 0x7fb6700ce390 2023-10-16T15:53:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.411+0000 7fb674d07640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb65406af90 0x7fb65406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.412+0000 7fb674d07640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb65406af90 0x7fb65406d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb660006fd0 tx=0x7fb660008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.415+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb66c0767c0 con 0x7fb6700ce390 2023-10-16T15:53:22.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.675+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb67004e160 con 0x7fb6700ce390 2023-10-16T15:53:22.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.676+0000 7fb665ffb640 1 -- 172.21.15.29:0/402057758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb66c031e60 con 0x7fb6700ce390 2023-10-16T15:53:22.678 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:22.678 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:53:22.679 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T15:53:22.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.679 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T15:53:22.679 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T15:53:22.679 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:53:22.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:53:22.680 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.680 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.681 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T15:53:22.681 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T15:53:22.682 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:53:22.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:53:22.682 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.683 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.683 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T15:53:22.683 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T15:53:22.684 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:53:22.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:53:22.684 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.685 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.685 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T15:53:22.685 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T15:53:22.685 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.686 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.686 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:53:22.686 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:53:22.686 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.687 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.687 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T15:53:22.687 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T15:53:22.688 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.688 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.688 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.688 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.688 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.689 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T15:53:22.689 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T15:53:22.689 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.689 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.690 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.690 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.691 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T15:53:22.691 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T15:53:22.691 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.691 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.691 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.691 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.692 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T15:53:22.692 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T15:53:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.693 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T15:53:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T15:53:22.694 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.696 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T15:53:22.696 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T15:53:22.696 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.696 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.696 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.697 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.697 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.697 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T15:53:22.698 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T15:53:22.698 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.698 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.698 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.698 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.699 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.699 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T15:53:22.699 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T15:53:22.699 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:53:22.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:53:22.700 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.700 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.701 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T15:53:22.701 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T15:53:22.701 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.701 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.701 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T15:53:22.702 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T15:53:22.702 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.702 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T15:53:22.703 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T15:53:22.703 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:53:22.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:53:22.703 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:53:22.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.704 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T15:53:22.704 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T15:53:22.705 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:53:22.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:53:22.705 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:53:22.706 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.706 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T15:53:22.706 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T15:53:22.707 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.707 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:53:22.707 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:53:22.707 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:53:22.707 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:53:22.707 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.708 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T15:53:22.708 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T15:53:22.708 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.709 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:53:22.709 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:53:22.709 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:53:22.709 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.709 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.710 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T15:53:22.710 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T15:53:22.710 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.711 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:53:22.711 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:53:22.711 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.711 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T15:53:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T15:53:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:53:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:53:22.712 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:53:22.713 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:53:22.713 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.713 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:22.713 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T15:53:22.713 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:53:22.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb65406af90 msgr2=0x7fb65406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb65406af90 0x7fb65406d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb660006fd0 tx=0x7fb660008040 comp rx=0 tx=0).stop 2023-10-16T15:53:22.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 msgr2=0x7fb6700ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 0x7fb6700ca9e0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fb66c02b930 tx=0x7fb66c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 shutdown_connections 2023-10-16T15:53:22.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fb65406af90 0x7fb65406d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb67015ae80 0x7fb6700b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67015aaa0 0x7fb6700b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 --2- 172.21.15.29:0/402057758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6700ce390 0x7fb6700ca9e0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.678+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 >> 172.21.15.29:0/402057758 conn(0x7fb6700cb410 msgr2=0x7fb6700b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.679+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 shutdown_connections 2023-10-16T15:53:22.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.679+0000 7fb67776c640 1 -- 172.21.15.29:0/402057758 wait complete. 2023-10-16T15:53:22.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:22.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.942+0000 7f34ad547640 1 Processor -- start 2023-10-16T15:53:22.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.942+0000 7f34ad547640 1 -- start start 2023-10-16T15:53:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34ad547640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8159950 0x7f34a815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34ad547640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34a8070450 0x7f34a80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34ad547640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f34a806c100 0x7f34a806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34ad547640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a815e490 con 0x7f34a8070450 2023-10-16T15:53:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34ad547640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a8070df0 con 0x7f34a806c100 2023-10-16T15:53:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34ad547640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a815aef0 con 0x7f34a8159950 2023-10-16T15:53:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.944+0000 7f34a77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34a8070450 0x7f34a80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34a8070450 0x7f34a80708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53850/0 (socket says 172.21.15.29:53850) 2023-10-16T15:53:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 -- 172.21.15.29:0/2872435274 learned_addr learned my addr 172.21.15.29:0/2872435274 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34acd46640 1 --2- 172.21.15.29:0/2872435274 >> v2:172.21.15.170:3300/0 conn(0x7f34a806c100 0x7f34a806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a7fff640 1 --2- 172.21.15.29:0/2872435274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8159950 0x7f34a815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 -- 172.21.15.29:0/2872435274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8159950 msgr2=0x7f34a815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 --2- 172.21.15.29:0/2872435274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8159950 0x7f34a815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 -- 172.21.15.29:0/2872435274 >> v2:172.21.15.170:3300/0 conn(0x7f34a806c100 msgr2=0x7f34a806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 --2- 172.21.15.29:0/2872435274 >> v2:172.21.15.170:3300/0 conn(0x7f34a806c100 0x7f34a806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.945+0000 7f34a77fe640 1 -- 172.21.15.29:0/2872435274 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a815d910 con 0x7f34a8070450 2023-10-16T15:53:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.946+0000 7f34a77fe640 1 --2- 172.21.15.29:0/2872435274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a8070450 0x7f34a80708b0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f34a80b0120 tx=0x7f349c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93167729837b10ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.947+0000 7f34a6ffd640 1 -- 172.21.15.29:0/2872435274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0317c0 con 0x7f34a8070450 2023-10-16T15:53:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.947+0000 7f34a6ffd640 1 -- 172.21.15.29:0/2872435274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c031920 con 0x7f34a8070450 2023-10-16T15:53:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.947+0000 7f34a6ffd640 1 -- 172.21.15.29:0/2872435274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c031cb0 con 0x7f34a8070450 2023-10-16T15:53:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.947+0000 7f34ad547640 1 -- 172.21.15.29:0/2872435274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a8070450 msgr2=0x7f34a80708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.947+0000 7f34ad547640 1 --2- 172.21.15.29:0/2872435274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a8070450 0x7f34a80708b0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f34a80b0120 tx=0x7f349c02b860 comp rx=0 tx=0).stop 2023-10-16T15:53:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 -- 172.21.15.29:0/2872435274 shutdown_connections 2023-10-16T15:53:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 --2- 172.21.15.29:0/2872435274 >> v2:172.21.15.170:3300/0 conn(0x7f34a806c100 0x7f34a806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 --2- 172.21.15.29:0/2872435274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a8070450 0x7f34a80708b0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 --2- 172.21.15.29:0/2872435274 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8159950 0x7f34a815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 -- 172.21.15.29:0/2872435274 >> 172.21.15.29:0/2872435274 conn(0x7f34a80ce6a0 msgr2=0x7f34a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 -- 172.21.15.29:0/2872435274 shutdown_connections 2023-10-16T15:53:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.948+0000 7f34ad547640 1 -- 172.21.15.29:0/2872435274 wait complete. 2023-10-16T15:53:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.950+0000 7f34ad547640 1 Processor -- start 2023-10-16T15:53:22.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.950+0000 7f34ad547640 1 -- start start 2023-10-16T15:53:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.951+0000 7f34ad547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 0x7f34a806b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.951+0000 7f34ad547640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8070450 0x7f34a80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.951+0000 7f34ad547640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34a8159950 0x7f34a81486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.951+0000 7f34ad547640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a80c7010 con 0x7f34a806c100 2023-10-16T15:53:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.951+0000 7f34ad547640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a80c6130 con 0x7f34a8159950 2023-10-16T15:53:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.951+0000 7f34ad547640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a80c6390 con 0x7f34a8070450 2023-10-16T15:53:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 0x7f34a806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 0x7f34a806b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53858/0 (socket says 172.21.15.29:53858) 2023-10-16T15:53:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 -- 172.21.15.29:0/823538521 learned_addr learned my addr 172.21.15.29:0/823538521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a77fe640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8070450 0x7f34a80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34acd46640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34a8159950 0x7f34a81486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 -- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8070450 msgr2=0x7f34a80acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8070450 0x7f34a80acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 -- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34a8159950 msgr2=0x7f34a81486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34a8159950 0x7f34a81486b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34a7fff640 1 -- 172.21.15.29:0/823538521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0094d0 con 0x7f34a806c100 2023-10-16T15:53:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.952+0000 7f34acd46640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34a8159950 0x7f34a81486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.953+0000 7f34a7fff640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 0x7f34a806b900 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f349800d800 tx=0x7f349800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.953+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349800b6c0 con 0x7f34a806c100 2023-10-16T15:53:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.953+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349800b820 con 0x7f34a806c100 2023-10-16T15:53:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.953+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349800bbb0 con 0x7f34a806c100 2023-10-16T15:53:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.953+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a80c6670 con 0x7f34a806c100 2023-10-16T15:53:22.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.953+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a80c6c30 con 0x7f34a806c100 2023-10-16T15:53:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.955+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a804e9b0 con 0x7f34a806c100 2023-10-16T15:53:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.955+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f349800bd10 con 0x7f34a806c100 2023-10-16T15:53:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.956+0000 7f34a4ff9640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f349406af90 0x7f349406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:22.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.956+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f34980899b0 con 0x7f34a806c100 2023-10-16T15:53:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.957+0000 7f34a77fe640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f349406af90 0x7f349406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.958+0000 7f34a77fe640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f349406af90 0x7f349406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f349c036000 tx=0x7f349c03d040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:22.961+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3498054c70 con 0x7f34a806c100 2023-10-16T15:53:23.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.158+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34a80ce020 con 0x7f34a806c100 2023-10-16T15:53:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.160+0000 7f34a4ff9640 1 -- 172.21.15.29:0/823538521 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f3498058290 con 0x7f34a806c100 2023-10-16T15:53:23.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:23.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":221,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:22.156847+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f349406af90 msgr2=0x7f349406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f349406af90 0x7f349406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f349c036000 tx=0x7f349c03d040 comp rx=0 tx=0).stop 2023-10-16T15:53:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 msgr2=0x7f34a806b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 0x7f34a806b900 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f349800d800 tx=0x7f349800dcc0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 shutdown_connections 2023-10-16T15:53:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f349406af90 0x7f349406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f34a8159950 0x7f34a81486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f34a8070450 0x7f34a80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 --2- 172.21.15.29:0/823538521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a806c100 0x7f34a806b900 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 >> 172.21.15.29:0/823538521 conn(0x7f34a80ce6a0 msgr2=0x7f34a80ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 shutdown_connections 2023-10-16T15:53:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.163+0000 7f34ad547640 1 -- 172.21.15.29:0/823538521 wait complete. 2023-10-16T15:53:23.185 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:53:23.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:53:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.398+0000 7f7e54e21640 1 Processor -- start 2023-10-16T15:53:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.399+0000 7f7e54e21640 1 -- start start 2023-10-16T15:53:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e54e21640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e500b0ea0 0x7f7e5015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e54e21640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7e500cd970 0x7f7e5015a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e54e21640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5006be30 con 0x7f7e500b3ec0 2023-10-16T15:53:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e54e21640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5006ca00 con 0x7f7e500b0ea0 2023-10-16T15:53:23.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e54e21640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500ada10 con 0x7f7e500cd970 2023-10-16T15:53:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e4effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e500b0ea0 0x7f7e5015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e4effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e500b0ea0 0x7f7e5015a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53872/0 (socket says 172.21.15.29:53872) 2023-10-16T15:53:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.400+0000 7f7e4effd640 1 -- 172.21.15.29:0/3022112427 learned_addr learned my addr 172.21.15.29:0/3022112427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.401+0000 7f7e4ffff640 1 --2- 172.21.15.29:0/3022112427 >> v2:172.21.15.170:3300/0 conn(0x7f7e500cd970 0x7f7e5015a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.401+0000 7f7e4effd640 1 -- 172.21.15.29:0/3022112427 >> v2:172.21.15.170:3300/0 conn(0x7f7e500cd970 msgr2=0x7f7e5015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.401+0000 7f7e4effd640 1 --2- 172.21.15.29:0/3022112427 >> v2:172.21.15.170:3300/0 conn(0x7f7e500cd970 0x7f7e5015a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.401+0000 7f7e4effd640 1 -- 172.21.15.29:0/3022112427 >> v1:172.21.15.170:6789/0 conn(0x7f7e500b3ec0 legacy=0x7f7e500cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:23.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.401+0000 7f7e4effd640 1 -- 172.21.15.29:0/3022112427 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5006c060 con 0x7f7e500b0ea0 2023-10-16T15:53:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.401+0000 7f7e4ffff640 1 --2- 172.21.15.29:0/3022112427 >> v2:172.21.15.170:3300/0 conn(0x7f7e500cd970 0x7f7e5015a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.402+0000 7f7e4effd640 1 --2- 172.21.15.29:0/3022112427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500b0ea0 0x7f7e5015a2f0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f7e50071b60 tx=0x7f7e3802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe4fcc7258f00ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.402+0000 7f7e4e7fc640 1 -- 172.21.15.29:0/3022112427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e380317e0 con 0x7f7e500b0ea0 2023-10-16T15:53:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.402+0000 7f7e4e7fc640 1 -- 172.21.15.29:0/3022112427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e38031940 con 0x7f7e500b0ea0 2023-10-16T15:53:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.402+0000 7f7e4e7fc640 1 -- 172.21.15.29:0/3022112427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38031cd0 con 0x7f7e500b0ea0 2023-10-16T15:53:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 -- 172.21.15.29:0/3022112427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500b0ea0 msgr2=0x7f7e5015a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3022112427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500b0ea0 0x7f7e5015a2f0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f7e50071b60 tx=0x7f7e3802b880 comp rx=0 tx=0).stop 2023-10-16T15:53:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 -- 172.21.15.29:0/3022112427 shutdown_connections 2023-10-16T15:53:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3022112427 >> v2:172.21.15.170:3300/0 conn(0x7f7e500cd970 0x7f7e5015a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3022112427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500b0ea0 0x7f7e5015a2f0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 -- 172.21.15.29:0/3022112427 >> 172.21.15.29:0/3022112427 conn(0x7f7e500b0170 msgr2=0x7f7e500b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 -- 172.21.15.29:0/3022112427 shutdown_connections 2023-10-16T15:53:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.403+0000 7f7e54e21640 1 -- 172.21.15.29:0/3022112427 wait complete. 2023-10-16T15:53:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.405+0000 7f7e54e21640 1 Processor -- start 2023-10-16T15:53:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.405+0000 7f7e54e21640 1 -- start start 2023-10-16T15:53:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e54e21640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e500b0ea0 0x7f7e500b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e54e21640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e500b3ec0 0x7f7e500b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e54e21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 0x7f7e5014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e54e21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014c660 con 0x7f7e500cd970 2023-10-16T15:53:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e54e21640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014e310 con 0x7f7e500b0ea0 2023-10-16T15:53:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e54e21640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014d700 con 0x7f7e500b3ec0 2023-10-16T15:53:23.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 0x7f7e5014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 0x7f7e5014c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53874/0 (socket says 172.21.15.29:53874) 2023-10-16T15:53:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.406+0000 7f7e4ffff640 1 -- 172.21.15.29:0/3343407868 learned_addr learned my addr 172.21.15.29:0/3343407868 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4f7fe640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e500b0ea0 0x7f7e500b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4effd640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e500b3ec0 0x7f7e500b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4ffff640 1 -- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e500b3ec0 msgr2=0x7f7e500b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4ffff640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e500b3ec0 0x7f7e500b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4ffff640 1 -- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e500b0ea0 msgr2=0x7f7e500b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4ffff640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e500b0ea0 0x7f7e500b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4ffff640 1 -- 172.21.15.29:0/3343407868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e380094d0 con 0x7f7e500cd970 2023-10-16T15:53:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4f7fe640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e500b0ea0 0x7f7e500b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:23.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e4ffff640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 0x7f7e5014c980 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800c7f0 tx=0x7f7e4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:23.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.407+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48007820 con 0x7f7e500cd970 2023-10-16T15:53:23.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.408+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48007980 con 0x7f7e500cd970 2023-10-16T15:53:23.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.408+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48007d70 con 0x7f7e500cd970 2023-10-16T15:53:23.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.408+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5014d9e0 con 0x7f7e500cd970 2023-10-16T15:53:23.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.408+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5014dfa0 con 0x7f7e500cd970 2023-10-16T15:53:23.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.409+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5014e4a0 con 0x7f7e500cd970 2023-10-16T15:53:23.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.410+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e480113f0 con 0x7f7e500cd970 2023-10-16T15:53:23.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.411+0000 7f7e47fff640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e3006afc0 0x7f7e3006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.411+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f7e48089940 con 0x7f7e500cd970 2023-10-16T15:53:23.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.412+0000 7f7e4f7fe640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e3006afc0 0x7f7e3006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.413+0000 7f7e4f7fe640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e3006afc0 0x7f7e3006d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e40007e90 tx=0x7f7e400073a0 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.416+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e48054a60 con 0x7f7e500cd970 2023-10-16T15:53:23.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.606+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e5006a890 con 0x7f7e500cd970 2023-10-16T15:53:23.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.607+0000 7f7e47fff640 1 -- 172.21.15.29:0/3343407868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v221) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e5006a890 con 0x7f7e500cd970 2023-10-16T15:53:23.609 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:53:23.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e3006afc0 msgr2=0x7f7e3006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e3006afc0 0x7f7e3006d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e40007e90 tx=0x7f7e400073a0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 msgr2=0x7f7e5014c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 0x7f7e5014c980 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800c7f0 tx=0x7f7e4800ccb0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 shutdown_connections 2023-10-16T15:53:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7e3006afc0 0x7f7e3006d480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e500cd970 0x7f7e5014c980 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e500b3ec0 0x7f7e500b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 --2- 172.21.15.29:0/3343407868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e500b0ea0 0x7f7e500b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.609+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 >> 172.21.15.29:0/3343407868 conn(0x7f7e500b0170 msgr2=0x7f7e500ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.610+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 shutdown_connections 2023-10-16T15:53:23.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.610+0000 7f7e54e21640 1 -- 172.21.15.29:0/3343407868 wait complete. 2023-10-16T15:53:23.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:53:23.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.856+0000 7f3a46ebb640 1 Processor -- start 2023-10-16T15:53:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.856+0000 7f3a46ebb640 1 -- start start 2023-10-16T15:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a46ebb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a4015c100 0x7f3a400ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a46ebb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a400cd4a0 con 0x7f3a40153740 2023-10-16T15:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a46ebb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4015ac60 con 0x7f3a400ccb00 2023-10-16T15:53:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a46ebb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a40070030 con 0x7f3a4015c100 2023-10-16T15:53:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a45eb9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3a40153740 0x7f3a4015a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52308/0 (socket says 172.21.15.29:52308) 2023-10-16T15:53:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a45eb9640 1 -- 172.21.15.29:0/2544873102 learned_addr learned my addr 172.21.15.29:0/2544873102 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.858+0000 7f3a466ba640 1 --2- 172.21.15.29:0/2544873102 >> v2:172.21.15.29:3300/0 conn(0x7f3a4015c100 0x7f3a400ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.859+0000 7f3a44eb7640 1 -- 172.21.15.29:0/2544873102 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320487457 0 0) 0x7f3a400cd4a0 con 0x7f3a40153740 2023-10-16T15:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.859+0000 7f3a44eb7640 1 -- 172.21.15.29:0/2544873102 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c003520 con 0x7f3a40153740 2023-10-16T15:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.859+0000 7f3a466ba640 1 -- 172.21.15.29:0/2544873102 >> v1:172.21.15.170:6789/0 conn(0x7f3a400ccb00 legacy=0x7f3a400ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.859+0000 7f3a466ba640 1 -- 172.21.15.29:0/2544873102 >> v1:172.21.15.29:6789/0 conn(0x7f3a40153740 legacy=0x7f3a4015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.859+0000 7f3a466ba640 1 -- 172.21.15.29:0/2544873102 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40070310 con 0x7f3a4015c100 2023-10-16T15:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.860+0000 7f3a466ba640 1 --2- 172.21.15.29:0/2544873102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a4015c100 0x7f3a400ccf60 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009d90 tx=0x7f3a3802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64132186b7b13346 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.860+0000 7f3a44eb7640 1 -- 172.21.15.29:0/2544873102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3802cd00 con 0x7f3a4015c100 2023-10-16T15:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.860+0000 7f3a44eb7640 1 -- 172.21.15.29:0/2544873102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3802ce60 con 0x7f3a4015c100 2023-10-16T15:53:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.860+0000 7f3a44eb7640 1 -- 172.21.15.29:0/2544873102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38034660 con 0x7f3a4015c100 2023-10-16T15:53:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 -- 172.21.15.29:0/2544873102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a4015c100 msgr2=0x7f3a400ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/2544873102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a4015c100 0x7f3a400ccf60 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009d90 tx=0x7f3a3802bf50 comp rx=0 tx=0).stop 2023-10-16T15:53:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 -- 172.21.15.29:0/2544873102 shutdown_connections 2023-10-16T15:53:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/2544873102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a4015c100 0x7f3a400ccf60 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 -- 172.21.15.29:0/2544873102 >> 172.21.15.29:0/2544873102 conn(0x7f3a400ce6a0 msgr2=0x7f3a400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:23.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 -- 172.21.15.29:0/2544873102 shutdown_connections 2023-10-16T15:53:23.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.861+0000 7f3a46ebb640 1 -- 172.21.15.29:0/2544873102 wait complete. 2023-10-16T15:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.863+0000 7f3a46ebb640 1 Processor -- start 2023-10-16T15:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.863+0000 7f3a46ebb640 1 -- start start 2023-10-16T15:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a46ebb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 0x7f3a4014d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a46ebb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a40153740 0x7f3a400c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a46ebb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a4015c100 0x7f3a400c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a46ebb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401570d0 con 0x7f3a400ccb00 2023-10-16T15:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a46ebb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401f2220 con 0x7f3a40153740 2023-10-16T15:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a46ebb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401f2500 con 0x7f3a4015c100 2023-10-16T15:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a45eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 0x7f3a4014d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a45eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 0x7f3a4014d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53898/0 (socket says 172.21.15.29:53898) 2023-10-16T15:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.864+0000 7f3a45eb9640 1 -- 172.21.15.29:0/1344604875 learned_addr learned my addr 172.21.15.29:0/1344604875 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a45eb9640 1 -- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a4015c100 msgr2=0x7f3a400c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a45eb9640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a4015c100 0x7f3a400c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a45eb9640 1 -- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a40153740 msgr2=0x7f3a400c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a45eb9640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a40153740 0x7f3a400c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a45eb9640 1 -- 172.21.15.29:0/1344604875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009a70 con 0x7f3a400ccb00 2023-10-16T15:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a45eb9640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 0x7f3a4014d000 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c500 tx=0x7f3a3c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c005ce0 con 0x7f3a400ccb00 2023-10-16T15:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c005e40 con 0x7f3a400ccb00 2023-10-16T15:53:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c007330 con 0x7f3a400ccb00 2023-10-16T15:53:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.865+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a401f27e0 con 0x7f3a400ccb00 2023-10-16T15:53:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.866+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a401f2fe0 con 0x7f3a400ccb00 2023-10-16T15:53:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.867+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a401f2d00 con 0x7f3a400ccb00 2023-10-16T15:53:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.868+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3a3c007550 con 0x7f3a400ccb00 2023-10-16T15:53:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.869+0000 7f3a367fc640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3a2406af90 0x7f3a2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:23.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.869+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f3a3c088a70 con 0x7f3a400ccb00 2023-10-16T15:53:23.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.869+0000 7f3a456b8640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3a2406af90 0x7f3a2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:23.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.871+0000 7f3a456b8640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3a2406af90 0x7f3a2406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000f60 tx=0x7f3a300079b0 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:23.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:23.873+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a401f2d00 con 0x7f3a400ccb00 2023-10-16T15:53:24.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.068+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3a401f2d00 con 0x7f3a400ccb00 2023-10-16T15:53:24.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.069+0000 7f3a367fc640 1 -- 172.21.15.29:0/1344604875 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v221) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3a3c017020 con 0x7f3a400ccb00 2023-10-16T15:53:24.072 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:24.072 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:53:24.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:24.072 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:53:24.073 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:53:24.073 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:24.073 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:53:24.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3a2406af90 msgr2=0x7f3a2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:24.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3a2406af90 0x7f3a2406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000f60 tx=0x7f3a300079b0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 msgr2=0x7f3a4014d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:24.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 0x7f3a4014d000 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c500 tx=0x7f3a3c00c9c0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 shutdown_connections 2023-10-16T15:53:24.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3a2406af90 0x7f3a2406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3a4015c100 0x7f3a400c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3a40153740 0x7f3a400c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 --2- 172.21.15.29:0/1344604875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a400ccb00 0x7f3a4014d000 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.072+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 >> 172.21.15.29:0/1344604875 conn(0x7f3a400ce6a0 msgr2=0x7f3a4006ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:24.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.073+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 shutdown_connections 2023-10-16T15:53:24.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.073+0000 7f3a46ebb640 1 -- 172.21.15.29:0/1344604875 wait complete. 2023-10-16T15:53:24.095 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T15:53:24.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T15:53:24.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.332+0000 7f03a759e640 1 Processor -- start 2023-10-16T15:53:24.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.333+0000 7f03a759e640 1 -- start start 2023-10-16T15:53:24.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815d100 0x7f03a80b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:24.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03a80b1810 0x7f03a80b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:24.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a80b8430 con 0x7f03a815cd20 2023-10-16T15:53:24.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a80d2160 con 0x7f03a80b1810 2023-10-16T15:53:24.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80b8680 con 0x7f03a815d100 2023-10-16T15:53:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f03a815cd20 0x7f03a80cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57726/0 (socket says 172.21.15.29:57726) 2023-10-16T15:53:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a659c640 1 -- 172.21.15.29:0/2081599465 learned_addr learned my addr 172.21.15.29:0/2081599465 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:24.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.334+0000 7f03a6d9d640 1 --2- 172.21.15.29:0/2081599465 >> v2:172.21.15.29:3300/0 conn(0x7f03a80b1810 0x7f03a80b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f039ffff640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815d100 0x7f03a80b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f03a5d9b640 1 -- 172.21.15.29:0/2081599465 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273898293 0 0) 0x7f03a80b8430 con 0x7f03a815cd20 2023-10-16T15:53:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f03a5d9b640 1 -- 172.21.15.29:0/2081599465 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0394003520 con 0x7f03a815cd20 2023-10-16T15:53:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f03a6d9d640 1 -- 172.21.15.29:0/2081599465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815d100 msgr2=0x7f03a80b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f03a6d9d640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815d100 0x7f03a80b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f03a6d9d640 1 -- 172.21.15.29:0/2081599465 >> v1:172.21.15.29:6789/0 conn(0x7f03a815cd20 legacy=0x7f03a80cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f03a6d9d640 1 -- 172.21.15.29:0/2081599465 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a80b8960 con 0x7f03a80b1810 2023-10-16T15:53:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.335+0000 7f039ffff640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815d100 0x7f03a80b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:24.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.336+0000 7f03a6d9d640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a80b4ad0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009d90 tx=0x7f03a002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6bbeb863bf53ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.336+0000 7f03a5d9b640 1 -- 172.21.15.29:0/2081599465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a002cce0 con 0x7f03a80b1810 2023-10-16T15:53:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a5d9b640 1 -- 172.21.15.29:0/2081599465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a002ce40 con 0x7f03a80b1810 2023-10-16T15:53:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a5d9b640 1 -- 172.21.15.29:0/2081599465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0034660 con 0x7f03a80b1810 2023-10-16T15:53:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 -- 172.21.15.29:0/2081599465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 msgr2=0x7f03a80b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:24.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a80b4ad0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009d90 tx=0x7f03a002bf30 comp rx=0 tx=0).stop 2023-10-16T15:53:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 -- 172.21.15.29:0/2081599465 shutdown_connections 2023-10-16T15:53:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a80b4ad0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 --2- 172.21.15.29:0/2081599465 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815d100 0x7f03a80b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 -- 172.21.15.29:0/2081599465 >> 172.21.15.29:0/2081599465 conn(0x7f03a80cf590 msgr2=0x7f03a80cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:24.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.337+0000 7f03a759e640 1 -- 172.21.15.29:0/2081599465 shutdown_connections 2023-10-16T15:53:24.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.338+0000 7f03a759e640 1 -- 172.21.15.29:0/2081599465 wait complete. 2023-10-16T15:53:24.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.339+0000 7f03a759e640 1 Processor -- start 2023-10-16T15:53:24.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.339+0000 7f03a759e640 1 -- start start 2023-10-16T15:53:24.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.340+0000 7f03a759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:24.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.340+0000 7f03a759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815cd20 0x7f03a80cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:24.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.340+0000 7f03a759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03a815d100 0x7f03a80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:24.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.340+0000 7f03a759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80cc090 con 0x7f03a80b1810 2023-10-16T15:53:24.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.340+0000 7f03a759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80cc370 con 0x7f03a815d100 2023-10-16T15:53:24.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.340+0000 7f03a759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a815df70 con 0x7f03a815cd20 2023-10-16T15:53:24.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:24.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a814e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45328/0 (socket says 172.21.15.29:45328) 2023-10-16T15:53:24.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 -- 172.21.15.29:0/4029281843 learned_addr learned my addr 172.21.15.29:0/4029281843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:24.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f039ffff640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815cd20 0x7f03a80cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:24.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 -- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815cd20 msgr2=0x7f03a80cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:24.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815cd20 0x7f03a80cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 -- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03a815d100 msgr2=0x7f03a80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:24.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a6d9d640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03a815d100 0x7f03a80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:24.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03a815d100 0x7f03a80c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:24.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.341+0000 7f03a659c640 1 -- 172.21.15.29:0/4029281843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0009a70 con 0x7f03a80b1810 2023-10-16T15:53:24.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.342+0000 7f03a659c640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a814e890 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f039800c500 tx=0x7f039800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:24.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.342+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398005ce0 con 0x7f03a80b1810 2023-10-16T15:53:24.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.342+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0398005e40 con 0x7f03a80b1810 2023-10-16T15:53:24.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.342+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398007330 con 0x7f03a80b1810 2023-10-16T15:53:24.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.342+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a815e250 con 0x7f03a80b1810 2023-10-16T15:53:24.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.342+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a814f2c0 con 0x7f03a80b1810 2023-10-16T15:53:24.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.343+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a815e490 con 0x7f03a80b1810 2023-10-16T15:53:24.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.344+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0398007550 con 0x7f03a80b1810 2023-10-16T15:53:24.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.345+0000 7f039effd640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f038806af90 0x7f038806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:24.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.345+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f0398088a70 con 0x7f03a80b1810 2023-10-16T15:53:24.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.346+0000 7f039ffff640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f038806af90 0x7f038806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:24.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.347+0000 7f039ffff640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f038806af90 0x7f038806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f03a80c5ce0 tx=0x7f0390008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:24.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.350+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03a815e490 con 0x7f03a80b1810 2023-10-16T15:53:24.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:24.546+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f03a815e490 con 0x7f03a80b1810 2023-10-16T15:53:25.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.161+0000 7f039effd640 1 -- 172.21.15.29:0/4029281843 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v222) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0398017020 con 0x7f03a80b1810 2023-10-16T15:53:25.164 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T15:53:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f038806af90 msgr2=0x7f038806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f038806af90 0x7f038806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f03a80c5ce0 tx=0x7f0390008040 comp rx=0 tx=0).stop 2023-10-16T15:53:25.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 msgr2=0x7f03a814e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:25.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a814e890 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f039800c500 tx=0x7f039800c9c0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 shutdown_connections 2023-10-16T15:53:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f038806af90 0x7f038806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03a815d100 0x7f03a80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03a815cd20 0x7f03a80cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 --2- 172.21.15.29:0/4029281843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a80b1810 0x7f03a814e890 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.165+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 >> 172.21.15.29:0/4029281843 conn(0x7f03a80cf590 msgr2=0x7f03a80b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:25.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.166+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 shutdown_connections 2023-10-16T15:53:25.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.166+0000 7f03a759e640 1 -- 172.21.15.29:0/4029281843 wait complete. 2023-10-16T15:53:25.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:53:25.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.417+0000 7f0cfd721640 1 Processor -- start 2023-10-16T15:53:25.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.418+0000 7f0cfd721640 1 -- start start 2023-10-16T15:53:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0cfd721640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf815ac70 0x7f0cf80cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0cfd721640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cf80b4160 0x7f0cf80cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0cfd721640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f0cf8069da0 0x7f0cf806a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0cfd721640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf806a770 con 0x7f0cf80b4160 2023-10-16T15:53:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0cfd721640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf80cc850 con 0x7f0cf8069da0 2023-10-16T15:53:25.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0cfd721640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80ccb30 con 0x7f0cf815ac70 2023-10-16T15:53:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.419+0000 7f0ceffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cf80b4160 0x7f0cf80cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cf80b4160 0x7f0cf80cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45336/0 (socket says 172.21.15.29:45336) 2023-10-16T15:53:25.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 -- 172.21.15.29:0/1178069419 learned_addr learned my addr 172.21.15.29:0/1178069419 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0cfcf20640 1 --2- 172.21.15.29:0/1178069419 >> v2:172.21.15.170:3300/0 conn(0x7f0cf8069da0 0x7f0cf806a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0cf7fff640 1 --2- 172.21.15.29:0/1178069419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf815ac70 0x7f0cf80cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 -- 172.21.15.29:0/1178069419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf815ac70 msgr2=0x7f0cf80cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 --2- 172.21.15.29:0/1178069419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf815ac70 0x7f0cf80cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 -- 172.21.15.29:0/1178069419 >> v2:172.21.15.170:3300/0 conn(0x7f0cf8069da0 msgr2=0x7f0cf806a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 --2- 172.21.15.29:0/1178069419 >> v2:172.21.15.170:3300/0 conn(0x7f0cf8069da0 0x7f0cf806a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.420+0000 7f0ceffff640 1 -- 172.21.15.29:0/1178069419 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80cce10 con 0x7f0cf80b4160 2023-10-16T15:53:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.421+0000 7f0ceffff640 1 --2- 172.21.15.29:0/1178069419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf80b4160 0x7f0cf80cd8f0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80adb20 tx=0x7f0ce002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48665e69501c234 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.422+0000 7f0cf77fe640 1 -- 172.21.15.29:0/1178069419 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0031870 con 0x7f0cf80b4160 2023-10-16T15:53:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.422+0000 7f0cf77fe640 1 -- 172.21.15.29:0/1178069419 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce00319d0 con 0x7f0cf80b4160 2023-10-16T15:53:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.422+0000 7f0cf77fe640 1 -- 172.21.15.29:0/1178069419 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0031d60 con 0x7f0cf80b4160 2023-10-16T15:53:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 -- 172.21.15.29:0/1178069419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf80b4160 msgr2=0x7f0cf80cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 --2- 172.21.15.29:0/1178069419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf80b4160 0x7f0cf80cd8f0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80adb20 tx=0x7f0ce002b910 comp rx=0 tx=0).stop 2023-10-16T15:53:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 -- 172.21.15.29:0/1178069419 shutdown_connections 2023-10-16T15:53:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 --2- 172.21.15.29:0/1178069419 >> v2:172.21.15.170:3300/0 conn(0x7f0cf8069da0 0x7f0cf806a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 --2- 172.21.15.29:0/1178069419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf80b4160 0x7f0cf80cd8f0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 --2- 172.21.15.29:0/1178069419 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf815ac70 0x7f0cf80cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 -- 172.21.15.29:0/1178069419 >> 172.21.15.29:0/1178069419 conn(0x7f0cf80ce6a0 msgr2=0x7f0cf80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 -- 172.21.15.29:0/1178069419 shutdown_connections 2023-10-16T15:53:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.423+0000 7f0cfd721640 1 -- 172.21.15.29:0/1178069419 wait complete. 2023-10-16T15:53:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.425+0000 7f0cfd721640 1 Processor -- start 2023-10-16T15:53:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.425+0000 7f0cfd721640 1 -- start start 2023-10-16T15:53:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfd721640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 0x7f0cf81579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfd721640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf80b4160 0x7f0cf8155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfd721640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cf815ac70 0x7f0cf8150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfd721640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81511d0 con 0x7f0cf8069da0 2023-10-16T15:53:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfd721640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8155880 con 0x7f0cf815ac70 2023-10-16T15:53:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfd721640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8155b60 con 0x7f0cf80b4160 2023-10-16T15:53:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 0x7f0cf81579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 0x7f0cf81579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45348/0 (socket says 172.21.15.29:45348) 2023-10-16T15:53:25.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cf7fff640 1 -- 172.21.15.29:0/763553659 learned_addr learned my addr 172.21.15.29:0/763553659 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:25.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0cfcf20640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cf815ac70 0x7f0cf8150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.426+0000 7f0ceffff640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf80b4160 0x7f0cf8155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf7fff640 1 -- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf80b4160 msgr2=0x7f0cf8155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:25.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf7fff640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf80b4160 0x7f0cf8155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf7fff640 1 -- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cf815ac70 msgr2=0x7f0cf8150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:25.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf7fff640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cf815ac70 0x7f0cf8150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:25.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf7fff640 1 -- 172.21.15.29:0/763553659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00094d0 con 0x7f0cf8069da0 2023-10-16T15:53:25.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf7fff640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 0x7f0cf81579e0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800d5f0 tx=0x7f0ce800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:25.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80125f0 con 0x7f0cf8069da0 2023-10-16T15:53:25.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce8012750 con 0x7f0cf8069da0 2023-10-16T15:53:25.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8012b00 con 0x7f0cf8069da0 2023-10-16T15:53:25.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.427+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf8155e40 con 0x7f0cf8069da0 2023-10-16T15:53:25.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.428+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf8156400 con 0x7f0cf8069da0 2023-10-16T15:53:25.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.429+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf8156590 con 0x7f0cf8069da0 2023-10-16T15:53:25.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.430+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0ce8012c60 con 0x7f0cf8069da0 2023-10-16T15:53:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.431+0000 7f0cf57fa640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0cd806af90 0x7f0cd806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.431+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f0ce808a5f0 con 0x7f0cf8069da0 2023-10-16T15:53:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.432+0000 7f0ceffff640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0cd806af90 0x7f0cd806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.433+0000 7f0ceffff640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0cd806af90 0x7f0cd806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0036000 tx=0x7f0ce003c040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.436+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ce8055730 con 0x7f0cf8069da0 2023-10-16T15:53:25.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:25.630+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0cf80cd3a0 con 0x7f0cf8069da0 2023-10-16T15:53:26.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.169+0000 7f0cf57fa640 1 -- 172.21.15.29:0/763553659 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v223) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0ce8058d50 con 0x7f0cf8069da0 2023-10-16T15:53:26.171 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T15:53:26.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0cd806af90 msgr2=0x7f0cd806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:26.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0cd806af90 0x7f0cd806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0036000 tx=0x7f0ce003c040 comp rx=0 tx=0).stop 2023-10-16T15:53:26.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 msgr2=0x7f0cf81579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:26.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 0x7f0cf81579e0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800d5f0 tx=0x7f0ce800dab0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 shutdown_connections 2023-10-16T15:53:26.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f0cd806af90 0x7f0cd806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0cf815ac70 0x7f0cf8150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0cf80b4160 0x7f0cf8155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 --2- 172.21.15.29:0/763553659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cf8069da0 0x7f0cf81579e0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 >> 172.21.15.29:0/763553659 conn(0x7f0cf80ce6a0 msgr2=0x7f0cf80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:26.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.173+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 shutdown_connections 2023-10-16T15:53:26.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.174+0000 7f0cfd721640 1 -- 172.21.15.29:0/763553659 wait complete. 2023-10-16T15:53:26.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T15:53:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.403+0000 7fc59fcc2640 1 Processor -- start 2023-10-16T15:53:26.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.403+0000 7fc59fcc2640 1 -- start start 2023-10-16T15:53:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.404+0000 7fc59fcc2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc598159ab0 0x7fc5980ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.404+0000 7fc59fcc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159e90 0x7fc59806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59fcc2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5980cd1f0 con 0x7fc59806c140 2023-10-16T15:53:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59fcc2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59815d890 con 0x7fc598159ab0 2023-10-16T15:53:26.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59fcc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5980cd4d0 con 0x7fc598159e90 2023-10-16T15:53:26.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59da5e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc598159ab0 0x7fc5980ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:26.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59da5e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc598159ab0 0x7fc5980ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45356/0 (socket says 172.21.15.29:45356) 2023-10-16T15:53:26.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59da5e640 1 -- 172.21.15.29:0/3047847896 learned_addr learned my addr 172.21.15.29:0/3047847896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:26.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.405+0000 7fc59d25d640 1 --2- 172.21.15.29:0/3047847896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159e90 0x7fc59806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:26.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.406+0000 7fc59da5e640 1 -- 172.21.15.29:0/3047847896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159e90 msgr2=0x7fc59806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:26.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.406+0000 7fc59da5e640 1 --2- 172.21.15.29:0/3047847896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159e90 0x7fc59806fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.406+0000 7fc59da5e640 1 -- 172.21.15.29:0/3047847896 >> v1:172.21.15.29:6789/0 conn(0x7fc59806c140 legacy=0x7fc59815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:26.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.406+0000 7fc59ca5c640 1 -- 172.21.15.29:0/3047847896 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322381972 0 0) 0x7fc5980cd1f0 con 0x7fc59806c140 2023-10-16T15:53:26.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.406+0000 7fc59da5e640 1 -- 172.21.15.29:0/3047847896 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980cd7b0 con 0x7fc598159ab0 2023-10-16T15:53:26.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.407+0000 7fc59da5e640 1 --2- 172.21.15.29:0/3047847896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc598159ab0 0x7fc5980ce370 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fc59400bb40 tx=0x7fc59402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ee2c00255e0133 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:26.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.407+0000 7fc59ca5c640 1 -- 172.21.15.29:0/3047847896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59402ede0 con 0x7fc598159ab0 2023-10-16T15:53:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.407+0000 7fc59ca5c640 1 -- 172.21.15.29:0/3047847896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594036440 con 0x7fc598159ab0 2023-10-16T15:53:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.407+0000 7fc59ca5c640 1 -- 172.21.15.29:0/3047847896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5940367d0 con 0x7fc598159ab0 2023-10-16T15:53:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 -- 172.21.15.29:0/3047847896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc598159ab0 msgr2=0x7fc5980ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/3047847896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc598159ab0 0x7fc5980ce370 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fc59400bb40 tx=0x7fc59402e030 comp rx=0 tx=0).stop 2023-10-16T15:53:26.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 -- 172.21.15.29:0/3047847896 shutdown_connections 2023-10-16T15:53:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/3047847896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159e90 0x7fc59806fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/3047847896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc598159ab0 0x7fc5980ce370 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 -- 172.21.15.29:0/3047847896 >> 172.21.15.29:0/3047847896 conn(0x7fc5980ce9b0 msgr2=0x7fc5980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 -- 172.21.15.29:0/3047847896 shutdown_connections 2023-10-16T15:53:26.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.408+0000 7fc59fcc2640 1 -- 172.21.15.29:0/3047847896 wait complete. 2023-10-16T15:53:26.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.410+0000 7fc59fcc2640 1 Processor -- start 2023-10-16T15:53:26.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.410+0000 7fc59fcc2640 1 -- start start 2023-10-16T15:53:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59fcc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 0x7fc59806b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59fcc2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159ab0 0x7fc59814f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59fcc2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc598159e90 0x7fc59814ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:26.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59fcc2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981f5ba0 con 0x7fc59806c140 2023-10-16T15:53:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59fcc2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981f5db0 con 0x7fc598159e90 2023-10-16T15:53:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59fcc2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981f6090 con 0x7fc598159ab0 2023-10-16T15:53:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 0x7fc59806b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 0x7fc59806b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45362/0 (socket says 172.21.15.29:45362) 2023-10-16T15:53:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 -- 172.21.15.29:0/2716956224 learned_addr learned my addr 172.21.15.29:0/2716956224 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59d25d640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159ab0 0x7fc59814f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 -- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159ab0 msgr2=0x7fc59814f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159ab0 0x7fc59814f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 -- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc598159e90 msgr2=0x7fc59814ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:26.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc598159e90 0x7fc59814ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.411+0000 7fc59da5e640 1 -- 172.21.15.29:0/2716956224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59400b770 con 0x7fc59806c140 2023-10-16T15:53:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.412+0000 7fc59da5e640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 0x7fc59806b610 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fc59402e070 tx=0x7fc5940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.412+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594034750 con 0x7fc59806c140 2023-10-16T15:53:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.412+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5940348b0 con 0x7fc59806c140 2023-10-16T15:53:26.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.412+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594034c60 con 0x7fc59806c140 2023-10-16T15:53:26.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.412+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5981f6370 con 0x7fc59806c140 2023-10-16T15:53:26.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.412+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59806bb50 con 0x7fc59806c140 2023-10-16T15:53:26.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.413+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59806bd40 con 0x7fc59806c140 2023-10-16T15:53:26.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.415+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc59404a3b0 con 0x7fc59806c140 2023-10-16T15:53:26.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.416+0000 7fc58e7fc640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc57c06afe0 0x7fc57c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:26.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.416+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fc5940abec0 con 0x7fc59806c140 2023-10-16T15:53:26.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.417+0000 7fc59d25d640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc57c06afe0 0x7fc57c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.418+0000 7fc59d25d640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc57c06afe0 0x7fc57c06d4a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5980c1e90 tx=0x7fc588008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:26.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.420+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc594076f10 con 0x7fc59806c140 2023-10-16T15:53:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:26.652+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc59804e160 con 0x7fc59806c140 2023-10-16T15:53:27.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.227+0000 7fc58e7fc640 1 -- 172.21.15.29:0/2716956224 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v105) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc59403b0c0 con 0x7fc59806c140 2023-10-16T15:53:27.233 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 26 and data pool 27 2023-10-16T15:53:27.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc57c06afe0 msgr2=0x7fc57c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc57c06afe0 0x7fc57c06d4a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5980c1e90 tx=0x7fc588008040 comp rx=0 tx=0).stop 2023-10-16T15:53:27.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 msgr2=0x7fc59806b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 0x7fc59806b610 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fc59402e070 tx=0x7fc5940046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 shutdown_connections 2023-10-16T15:53:27.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fc57c06afe0 0x7fc57c06d4a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc598159e90 0x7fc59814ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc598159ab0 0x7fc59814f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 --2- 172.21.15.29:0/2716956224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc59806c140 0x7fc59806b610 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.235+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 >> 172.21.15.29:0/2716956224 conn(0x7fc5980ce9b0 msgr2=0x7fc5980b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:27.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.236+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 shutdown_connections 2023-10-16T15:53:27.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.237+0000 7fc59fcc2640 1 -- 172.21.15.29:0/2716956224 wait complete. 2023-10-16T15:53:27.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.454+0000 7f3efaae5640 1 Processor -- start 2023-10-16T15:53:27.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.454+0000 7f3efaae5640 1 -- start start 2023-10-16T15:53:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.455+0000 7f3efaae5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ef40b0b20 0x7f3ef41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.455+0000 7f3efaae5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3ef415d3f0 0x7f3ef415d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.455+0000 7f3efaae5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef415e3e0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.455+0000 7f3efaae5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef40cd4a0 con 0x7f3ef40b0b20 2023-10-16T15:53:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.455+0000 7f3efaae5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef415e100 con 0x7f3ef415d3f0 2023-10-16T15:53:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ef40b0b20 0x7f3ef41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef8881640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3ef40cd0c0 0x7f3ef40b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57732/0 (socket says 172.21.15.29:57732) 2023-10-16T15:53:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ef40b0b20 0x7f3ef41615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45376/0 (socket says 172.21.15.29:45376) 2023-10-16T15:53:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef8881640 1 -- 172.21.15.29:0/765687515 learned_addr learned my addr 172.21.15.29:0/765687515 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:27.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef9082640 1 --2- 172.21.15.29:0/765687515 >> v2:172.21.15.170:3300/0 conn(0x7f3ef415d3f0 0x7f3ef415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef3fff640 1 -- 172.21.15.29:0/765687515 >> v2:172.21.15.170:3300/0 conn(0x7f3ef415d3f0 msgr2=0x7f3ef415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:27.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef3fff640 1 --2- 172.21.15.29:0/765687515 >> v2:172.21.15.170:3300/0 conn(0x7f3ef415d3f0 0x7f3ef415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef3fff640 1 -- 172.21.15.29:0/765687515 >> v1:172.21.15.29:6789/0 conn(0x7f3ef40cd0c0 legacy=0x7f3ef40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.456+0000 7f3ef37fe640 1 -- 172.21.15.29:0/765687515 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690356319 0 0) 0x7f3ef415e3e0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.457+0000 7f3ef3fff640 1 -- 172.21.15.29:0/765687515 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef4160b10 con 0x7f3ef40b0b20 2023-10-16T15:53:27.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.458+0000 7f3ef3fff640 1 --2- 172.21.15.29:0/765687515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40b0b20 0x7f3ef41615f0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f3ef406da60 tx=0x7f3ee002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5318a7d5a3359e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:27.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.458+0000 7f3ef37fe640 1 -- 172.21.15.29:0/765687515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee002e070 con 0x7f3ef40b0b20 2023-10-16T15:53:27.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.458+0000 7f3ef37fe640 1 -- 172.21.15.29:0/765687515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee00315a0 con 0x7f3ef40b0b20 2023-10-16T15:53:27.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.458+0000 7f3ef37fe640 1 -- 172.21.15.29:0/765687515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0031930 con 0x7f3ef40b0b20 2023-10-16T15:53:27.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 -- 172.21.15.29:0/765687515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40b0b20 msgr2=0x7f3ef41615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 --2- 172.21.15.29:0/765687515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40b0b20 0x7f3ef41615f0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f3ef406da60 tx=0x7f3ee002b840 comp rx=0 tx=0).stop 2023-10-16T15:53:27.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 -- 172.21.15.29:0/765687515 shutdown_connections 2023-10-16T15:53:27.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 --2- 172.21.15.29:0/765687515 >> v2:172.21.15.170:3300/0 conn(0x7f3ef415d3f0 0x7f3ef415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 --2- 172.21.15.29:0/765687515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40b0b20 0x7f3ef41615f0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 -- 172.21.15.29:0/765687515 >> 172.21.15.29:0/765687515 conn(0x7f3ef40cb410 msgr2=0x7f3ef40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:27.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 -- 172.21.15.29:0/765687515 shutdown_connections 2023-10-16T15:53:27.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.459+0000 7f3efaae5640 1 -- 172.21.15.29:0/765687515 wait complete. 2023-10-16T15:53:27.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.461+0000 7f3efaae5640 1 Processor -- start 2023-10-16T15:53:27.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.462+0000 7f3efaae5640 1 -- start start 2023-10-16T15:53:27.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3efaae5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ef40b0b20 0x7f3ef41f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3efaae5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 0x7f3ef40b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3efaae5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ef415d3f0 0x7f3ef40b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3efaae5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef40b1fd0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3efaae5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef40b2210 con 0x7f3ef415d3f0 2023-10-16T15:53:27.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3efaae5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415f760 con 0x7f3ef40b0b20 2023-10-16T15:53:27.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3ef3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 0x7f3ef40b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3ef3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 0x7f3ef40b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45388/0 (socket says 172.21.15.29:45388) 2023-10-16T15:53:27.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3ef3fff640 1 -- 172.21.15.29:0/1341691868 learned_addr learned my addr 172.21.15.29:0/1341691868 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:27.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.463+0000 7f3ef8881640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ef40b0b20 0x7f3ef41f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef9082640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ef415d3f0 0x7f3ef40b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef3fff640 1 -- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ef40b0b20 msgr2=0x7f3ef41f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef3fff640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ef40b0b20 0x7f3ef41f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef3fff640 1 -- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ef415d3f0 msgr2=0x7f3ef40b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef3fff640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ef415d3f0 0x7f3ef40b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef3fff640 1 -- 172.21.15.29:0/1341691868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee0009400 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef3fff640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 0x7f3ef40b1550 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0034040 tx=0x7f3ee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:27.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0004230 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee0031f00 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0032640 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.464+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef415fa40 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.465+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef4160000 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.466+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef40708e0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.467+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3ee00327a0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.468+0000 7f3ef17fa640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ed806af90 0x7f3ed806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.468+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3ee00ab9f0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.469+0000 7f3ef8881640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ed806af90 0x7f3ed806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.470+0000 7f3ef8881640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ed806af90 0x7f3ed806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000f60 tx=0x7f3ee8007c00 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:27.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.473+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ee00767c0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.671+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ef404e160 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.673+0000 7f3ef17fa640 1 -- 172.21.15.29:0/1341691868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v224) v1 ==== 74+0+16640 (secure 0 0 0) 0x7f3ee0079de0 con 0x7f3ef40cd0c0 2023-10-16T15:53:27.675 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:27.675 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":224,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:26.655072+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ed806af90 msgr2=0x7f3ed806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ed806af90 0x7f3ed806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000f60 tx=0x7f3ee8007c00 comp rx=0 tx=0).stop 2023-10-16T15:53:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 msgr2=0x7f3ef40b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 0x7f3ef40b1550 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0034040 tx=0x7f3ee00046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 shutdown_connections 2023-10-16T15:53:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ed806af90 0x7f3ed806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ef40cd0c0 0x7f3ef40b1550 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ef415d3f0 0x7f3ef40b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 --2- 172.21.15.29:0/1341691868 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ef40b0b20 0x7f3ef41f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.675+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 >> 172.21.15.29:0/1341691868 conn(0x7f3ef40cb410 msgr2=0x7f3ef415c670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:27.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.676+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 shutdown_connections 2023-10-16T15:53:27.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.676+0000 7f3efaae5640 1 -- 172.21.15.29:0/1341691868 wait complete. 2023-10-16T15:53:27.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:27.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.920+0000 7fe93284b640 1 Processor -- start 2023-10-16T15:53:27.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.920+0000 7fe93284b640 1 -- start start 2023-10-16T15:53:27.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe93284b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe92c15ae80 0x7fe92c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe93284b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe92c0ce390 0x7fe92c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe93284b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c15ed60 con 0x7fe92c15aaa0 2023-10-16T15:53:27.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe93284b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c06b810 con 0x7fe92c0ce390 2023-10-16T15:53:27.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe93284b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0707e0 con 0x7fe92c15ae80 2023-10-16T15:53:27.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe930de8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe92c0ce390 0x7fe92c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe92bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe92c15aaa0 0x7fe92c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57738/0 (socket says 172.21.15.29:57738) 2023-10-16T15:53:27.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.922+0000 7fe92bfff640 1 -- 172.21.15.29:0/1403833247 learned_addr learned my addr 172.21.15.29:0/1403833247 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:27.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.923+0000 7fe923fff640 1 --2- 172.21.15.29:0/1403833247 >> v2:172.21.15.170:3300/0 conn(0x7fe92c15ae80 0x7fe92c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.923+0000 7fe930de8640 1 -- 172.21.15.29:0/1403833247 >> v2:172.21.15.170:3300/0 conn(0x7fe92c15ae80 msgr2=0x7fe92c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.923+0000 7fe930de8640 1 --2- 172.21.15.29:0/1403833247 >> v2:172.21.15.170:3300/0 conn(0x7fe92c15ae80 0x7fe92c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.923+0000 7fe930de8640 1 -- 172.21.15.29:0/1403833247 >> v1:172.21.15.29:6789/0 conn(0x7fe92c15aaa0 legacy=0x7fe92c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.923+0000 7fe930de8640 1 -- 172.21.15.29:0/1403833247 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c15ef90 con 0x7fe92c0ce390 2023-10-16T15:53:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.924+0000 7fe930de8640 1 --2- 172.21.15.29:0/1403833247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c0ce390 0x7fe92c070f50 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0c59c0 tx=0x7fe92402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d25d2fcc7ed3cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe92b7fe640 1 -- 172.21.15.29:0/1403833247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92402cc70 con 0x7fe92c0ce390 2023-10-16T15:53:27.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe92b7fe640 1 -- 172.21.15.29:0/1403833247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92402cdd0 con 0x7fe92c0ce390 2023-10-16T15:53:27.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe92b7fe640 1 -- 172.21.15.29:0/1403833247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924034660 con 0x7fe92c0ce390 2023-10-16T15:53:27.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe93284b640 1 -- 172.21.15.29:0/1403833247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c0ce390 msgr2=0x7fe92c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe93284b640 1 --2- 172.21.15.29:0/1403833247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c0ce390 0x7fe92c070f50 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0c59c0 tx=0x7fe92402bec0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe93284b640 1 -- 172.21.15.29:0/1403833247 shutdown_connections 2023-10-16T15:53:27.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe93284b640 1 --2- 172.21.15.29:0/1403833247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c0ce390 0x7fe92c070f50 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe93284b640 1 --2- 172.21.15.29:0/1403833247 >> v2:172.21.15.170:3300/0 conn(0x7fe92c15ae80 0x7fe92c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.925+0000 7fe93284b640 1 -- 172.21.15.29:0/1403833247 >> 172.21.15.29:0/1403833247 conn(0x7fe92c0cb410 msgr2=0x7fe92c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.926+0000 7fe93284b640 1 -- 172.21.15.29:0/1403833247 shutdown_connections 2023-10-16T15:53:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.926+0000 7fe93284b640 1 -- 172.21.15.29:0/1403833247 wait complete. 2023-10-16T15:53:27.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.927+0000 7fe93284b640 1 Processor -- start 2023-10-16T15:53:27.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.927+0000 7fe93284b640 1 -- start start 2023-10-16T15:53:27.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.928+0000 7fe93284b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe92c0ce390 0x7fe92c14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.928+0000 7fe93284b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c15aaa0 0x7fe92c154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.928+0000 7fe93284b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 0x7fe92c155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.928+0000 7fe93284b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c15dd20 con 0x7fe92c15ae80 2023-10-16T15:53:27.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.928+0000 7fe93284b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c150410 con 0x7fe92c15aaa0 2023-10-16T15:53:27.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.928+0000 7fe93284b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c15d110 con 0x7fe92c0ce390 2023-10-16T15:53:27.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 0x7fe92c155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 0x7fe92c155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45408/0 (socket says 172.21.15.29:45408) 2023-10-16T15:53:27.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 -- 172.21.15.29:0/3475267221 learned_addr learned my addr 172.21.15.29:0/3475267221 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:27.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe923fff640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c15aaa0 0x7fe92c154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe92bfff640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe92c0ce390 0x7fe92c14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 -- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe92c0ce390 msgr2=0x7fe92c14fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe92c0ce390 0x7fe92c14fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 -- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c15aaa0 msgr2=0x7fe92c154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c15aaa0 0x7fe92c154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe930de8640 1 -- 172.21.15.29:0/3475267221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe924009a70 con 0x7fe92c15ae80 2023-10-16T15:53:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe923fff640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c15aaa0 0x7fe92c154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.929+0000 7fe92bfff640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe92c0ce390 0x7fe92c14fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.930+0000 7fe930de8640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 0x7fe92c155100 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fe924009e80 tx=0x7fe924004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:27.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.930+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924040650 con 0x7fe92c15ae80 2023-10-16T15:53:27.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.930+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9240407d0 con 0x7fe92c15ae80 2023-10-16T15:53:27.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.930+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924040b80 con 0x7fe92c15ae80 2023-10-16T15:53:27.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.930+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c15d3f0 con 0x7fe92c15ae80 2023-10-16T15:53:27.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.930+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c15d9b0 con 0x7fe92c15ae80 2023-10-16T15:53:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.931+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c15ded0 con 0x7fe92c15ae80 2023-10-16T15:53:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.932+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe924040ce0 con 0x7fe92c15ae80 2023-10-16T15:53:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.933+0000 7fe9297fa640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe90c06af90 0x7fe90c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.933+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe9240ac580 con 0x7fe92c15ae80 2023-10-16T15:53:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.934+0000 7fe92bfff640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe90c06af90 0x7fe90c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.935+0000 7fe92bfff640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe90c06af90 0x7fe90c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe91c005880 tx=0x7fe91c009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:27.937+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe92c15ded0 con 0x7fe92c15ae80 2023-10-16T15:53:28.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.129+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe92c04e160 con 0x7fe92c15ae80 2023-10-16T15:53:28.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.131+0000 7fe9297fa640 1 -- 172.21.15.29:0/3475267221 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v224) v1 ==== 74+0+16640 (secure 0 0 0) 0x7fe924077500 con 0x7fe92c15ae80 2023-10-16T15:53:28.133 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:28.134 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":224,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:26.655072+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:28.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.133+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe90c06af90 msgr2=0x7fe90c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:28.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.133+0000 7fe93284b640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe90c06af90 0x7fe90c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe91c005880 tx=0x7fe91c009040 comp rx=0 tx=0).stop 2023-10-16T15:53:28.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 msgr2=0x7fe92c155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:28.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 0x7fe92c155100 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fe924009e80 tx=0x7fe924004a40 comp rx=0 tx=0).stop 2023-10-16T15:53:28.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 shutdown_connections 2023-10-16T15:53:28.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe90c06af90 0x7fe90c06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe92c15ae80 0x7fe92c155100 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe92c15aaa0 0x7fe92c154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 --2- 172.21.15.29:0/3475267221 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe92c0ce390 0x7fe92c14fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 >> 172.21.15.29:0/3475267221 conn(0x7fe92c0cb410 msgr2=0x7fe92c06cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:28.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 shutdown_connections 2023-10-16T15:53:28.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.134+0000 7fe93284b640 1 -- 172.21.15.29:0/3475267221 wait complete. 2023-10-16T15:53:28.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T15:53:28.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.377+0000 7fdd37c0e640 1 Processor -- start 2023-10-16T15:53:28.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.377+0000 7fdd37c0e640 1 -- start start 2023-10-16T15:53:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd37c0e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdd3015d030 0x7fdd300d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd37c0e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd3015d410 0x7fdd30161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd37c0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd300b5d90 con 0x7fdd30161990 2023-10-16T15:53:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd37c0e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd300d2b50 con 0x7fdd3015d410 2023-10-16T15:53:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd37c0e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd30160fc0 con 0x7fdd3015d030 2023-10-16T15:53:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd351a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd3015d410 0x7fdd30161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd361ab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdd30161990 0x7fdd300cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57754/0 (socket says 172.21.15.29:57754) 2023-10-16T15:53:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.379+0000 7fdd361ab640 1 -- 172.21.15.29:0/2874272804 learned_addr learned my addr 172.21.15.29:0/2874272804 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.380+0000 7fdd359aa640 1 --2- 172.21.15.29:0/2874272804 >> v2:172.21.15.170:3300/0 conn(0x7fdd3015d030 0x7fdd300d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.380+0000 7fdd351a9640 1 -- 172.21.15.29:0/2874272804 >> v2:172.21.15.170:3300/0 conn(0x7fdd3015d030 msgr2=0x7fdd300d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.380+0000 7fdd351a9640 1 --2- 172.21.15.29:0/2874272804 >> v2:172.21.15.170:3300/0 conn(0x7fdd3015d030 0x7fdd300d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.380+0000 7fdd351a9640 1 -- 172.21.15.29:0/2874272804 >> v1:172.21.15.29:6789/0 conn(0x7fdd30161990 legacy=0x7fdd300cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.380+0000 7fdd349a8640 1 -- 172.21.15.29:0/2874272804 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350663629 0 0) 0x7fdd300b5d90 con 0x7fdd30161990 2023-10-16T15:53:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.380+0000 7fdd351a9640 1 -- 172.21.15.29:0/2874272804 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd300d25d0 con 0x7fdd3015d410 2023-10-16T15:53:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.381+0000 7fdd351a9640 1 --2- 172.21.15.29:0/2874272804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3015d410 0x7fdd30161450 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fdd3006cd10 tx=0x7fdd2002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e88237f1dbb504 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.381+0000 7fdd349a8640 1 -- 172.21.15.29:0/2874272804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20030610 con 0x7fdd3015d410 2023-10-16T15:53:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.381+0000 7fdd349a8640 1 -- 172.21.15.29:0/2874272804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd20030770 con 0x7fdd3015d410 2023-10-16T15:53:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.381+0000 7fdd349a8640 1 -- 172.21.15.29:0/2874272804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20030b00 con 0x7fdd3015d410 2023-10-16T15:53:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 -- 172.21.15.29:0/2874272804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3015d410 msgr2=0x7fdd30161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/2874272804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3015d410 0x7fdd30161450 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fdd3006cd10 tx=0x7fdd2002b960 comp rx=0 tx=0).stop 2023-10-16T15:53:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 -- 172.21.15.29:0/2874272804 shutdown_connections 2023-10-16T15:53:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/2874272804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3015d410 0x7fdd30161450 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/2874272804 >> v2:172.21.15.170:3300/0 conn(0x7fdd3015d030 0x7fdd300d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 -- 172.21.15.29:0/2874272804 >> 172.21.15.29:0/2874272804 conn(0x7fdd300cf720 msgr2=0x7fdd300cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 -- 172.21.15.29:0/2874272804 shutdown_connections 2023-10-16T15:53:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.382+0000 7fdd37c0e640 1 -- 172.21.15.29:0/2874272804 wait complete. 2023-10-16T15:53:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.384+0000 7fdd37c0e640 1 Processor -- start 2023-10-16T15:53:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.384+0000 7fdd37c0e640 1 -- start start 2023-10-16T15:53:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.385+0000 7fdd37c0e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd3015d030 0x7fdd3014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.385+0000 7fdd37c0e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd3015d410 0x7fdd300c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.385+0000 7fdd37c0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 0x7fdd300c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.385+0000 7fdd37c0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30160770 con 0x7fdd30161990 2023-10-16T15:53:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.385+0000 7fdd37c0e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3015fed0 con 0x7fdd3015d030 2023-10-16T15:53:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.385+0000 7fdd37c0e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd301601b0 con 0x7fdd3015d410 2023-10-16T15:53:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 0x7fdd300c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 0x7fdd300c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45426/0 (socket says 172.21.15.29:45426) 2023-10-16T15:53:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 -- 172.21.15.29:0/3884286300 learned_addr learned my addr 172.21.15.29:0/3884286300 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd351a9640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd3015d410 0x7fdd300c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd359aa640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd3015d030 0x7fdd3014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 -- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd3015d410 msgr2=0x7fdd300c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd3015d410 0x7fdd300c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 -- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd3015d030 msgr2=0x7fdd3014e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd3015d030 0x7fdd3014e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 -- 172.21.15.29:0/3884286300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20009400 con 0x7fdd30161990 2023-10-16T15:53:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd351a9640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd3015d410 0x7fdd300c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd359aa640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd3015d030 0x7fdd3014e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.386+0000 7fdd361ab640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 0x7fdd300c8ca0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fdd28001eb0 tx=0x7fdd2800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.387+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd280074e0 con 0x7fdd30161990 2023-10-16T15:53:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.387+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd28007640 con 0x7fdd30161990 2023-10-16T15:53:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.387+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd30160490 con 0x7fdd30161990 2023-10-16T15:53:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.387+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28004a00 con 0x7fdd30161990 2023-10-16T15:53:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.387+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3014f340 con 0x7fdd30161990 2023-10-16T15:53:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.388+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd300cb0a0 con 0x7fdd30161990 2023-10-16T15:53:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.389+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd28004b60 con 0x7fdd30161990 2023-10-16T15:53:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.390+0000 7fdd267fc640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdd1406afc0 0x7fdd1406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.390+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd28089580 con 0x7fdd30161990 2023-10-16T15:53:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.391+0000 7fdd359aa640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdd1406afc0 0x7fdd1406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.392+0000 7fdd359aa640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdd1406afc0 0x7fdd1406d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c007330 tx=0x7fdd2c00b040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.394+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd300cb0a0 con 0x7fdd30161990 2023-10-16T15:53:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:28.633+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdd300cb0a0 con 0x7fdd30161990 2023-10-16T15:53:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.260+0000 7fdd267fc640 1 -- 172.21.15.29:0/3884286300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v108) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdd300cb0a0 con 0x7fdd30161990 2023-10-16T15:53:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdd1406afc0 msgr2=0x7fdd1406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdd1406afc0 0x7fdd1406d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c007330 tx=0x7fdd2c00b040 comp rx=0 tx=0).stop 2023-10-16T15:53:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 msgr2=0x7fdd300c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 0x7fdd300c8ca0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fdd28001eb0 tx=0x7fdd2800c870 comp rx=0 tx=0).stop 2023-10-16T15:53:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 shutdown_connections 2023-10-16T15:53:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdd1406afc0 0x7fdd1406d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd30161990 0x7fdd300c8ca0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd3015d410 0x7fdd300c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 --2- 172.21.15.29:0/3884286300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd3015d030 0x7fdd3014e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 >> 172.21.15.29:0/3884286300 conn(0x7fdd300cf720 msgr2=0x7fdd300b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 shutdown_connections 2023-10-16T15:53:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.264+0000 7fdd37c0e640 1 -- 172.21.15.29:0/3884286300 wait complete. 2023-10-16T15:53:29.287 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:53:29.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T15:53:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.521+0000 7f08112a3640 1 Processor -- start 2023-10-16T15:53:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.521+0000 7f08112a3640 1 -- start start 2023-10-16T15:53:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.523+0000 7f08112a3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f080c070b60 0x7f080c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.523+0000 7f08112a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c15aef0 con 0x7f080c06c100 2023-10-16T15:53:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.523+0000 7f08112a3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c15d910 con 0x7f080c159950 2023-10-16T15:53:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.523+0000 7f08112a3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c15e490 con 0x7f080c070b60 2023-10-16T15:53:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.523+0000 7f0810aa2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f080c06c100 0x7f080c06c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57758/0 (socket says 172.21.15.29:57758) 2023-10-16T15:53:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.523+0000 7f0810aa2640 1 -- 172.21.15.29:0/2337598945 learned_addr learned my addr 172.21.15.29:0/2337598945 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770569286 0 0) 0x7f0804001490 con 0x7f080c06c100 2023-10-16T15:53:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080b7fe640 1 --2- 172.21.15.29:0/2337598945 >> v2:172.21.15.170:3300/0 conn(0x7f080c070b60 0x7f080c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f4003520 con 0x7f080c06c100 2023-10-16T15:53:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460119001 0 0) 0x7f07f4003520 con 0x7f080c06c100 2023-10-16T15:53:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 >> v2:172.21.15.170:3300/0 conn(0x7f080c070b60 msgr2=0x7f080c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 --2- 172.21.15.29:0/2337598945 >> v2:172.21.15.170:3300/0 conn(0x7f080c070b60 0x7f080c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 >> v1:172.21.15.170:6789/0 conn(0x7f080c159950 legacy=0x7f080c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c15dbf0 con 0x7f080c06c100 2023-10-16T15:53:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0804001b80 con 0x7f080c06c100 2023-10-16T15:53:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 >> v1:172.21.15.29:6789/0 conn(0x7f080c06c100 legacy=0x7f080c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f40039c0 0x7f07f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40047c0 0x7f07f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07f40051d0 0x7f07f400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c15e490 con 0x7f07f40039c0 2023-10-16T15:53:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f400e330 con 0x7f07f40051d0 2023-10-16T15:53:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.524+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f400e4f0 con 0x7f07f40047c0 2023-10-16T15:53:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080bfff640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f40039c0 0x7f07f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f0810aa2640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07f40051d0 0x7f07f400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080b7fe640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40047c0 0x7f07f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080bfff640 1 -- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40047c0 msgr2=0x7f07f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080bfff640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40047c0 0x7f07f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080bfff640 1 -- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07f40051d0 msgr2=0x7f07f400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080bfff640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f07f40051d0 0x7f07f400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080bfff640 1 -- 172.21.15.29:0/2337598945 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f4003520 con 0x7f07f40039c0 2023-10-16T15:53:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f0810aa2640 1 -- 172.21.15.29:0/2337598945 reap_dead start 2023-10-16T15:53:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.525+0000 7f080b7fe640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40047c0 0x7f07f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.527+0000 7f080bfff640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f40039c0 0x7f07f4003de0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f07fc001720 tx=0x7f07f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02db02c2e5e2236 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:29.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.527+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc0052d0 con 0x7f07f40039c0 2023-10-16T15:53:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.527+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc02e050 con 0x7f07f40039c0 2023-10-16T15:53:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.527+0000 7f080affd640 1 -- 172.21.15.29:0/2337598945 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc003830 con 0x7f07f40039c0 2023-10-16T15:53:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 -- 172.21.15.29:0/2337598945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f40039c0 msgr2=0x7f07f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f40039c0 0x7f07f4003de0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f07fc001720 tx=0x7f07f40057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 -- 172.21.15.29:0/2337598945 shutdown_connections 2023-10-16T15:53:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 --2- 172.21.15.29:0/2337598945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07f40039c0 0x7f07f4003de0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 -- 172.21.15.29:0/2337598945 >> 172.21.15.29:0/2337598945 conn(0x7f080c0ce6a0 msgr2=0x7f080c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 -- 172.21.15.29:0/2337598945 shutdown_connections 2023-10-16T15:53:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.528+0000 7f08112a3640 1 -- 172.21.15.29:0/2337598945 wait complete. 2023-10-16T15:53:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.530+0000 7f08112a3640 1 Processor -- start 2023-10-16T15:53:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.530+0000 7f08112a3640 1 -- start start 2023-10-16T15:53:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f08112a3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40039c0 0x7f080c0c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f08112a3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c158d70 0x7f080c1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f08112a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 0x7f080c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f08112a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c1591f0 con 0x7f080c0c5770 2023-10-16T15:53:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f08112a3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c1f3130 con 0x7f080c158d70 2023-10-16T15:53:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f08112a3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0b2440 con 0x7f07f40039c0 2023-10-16T15:53:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f0810aa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 0x7f080c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f0810aa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 0x7f080c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45448/0 (socket says 172.21.15.29:45448) 2023-10-16T15:53:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.531+0000 7f0810aa2640 1 -- 172.21.15.29:0/4098790116 learned_addr learned my addr 172.21.15.29:0/4098790116 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f080bfff640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40039c0 0x7f080c0c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f080b7fe640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c158d70 0x7f080c1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0810aa2640 1 -- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40039c0 msgr2=0x7f080c0c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0810aa2640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40039c0 0x7f080c0c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0810aa2640 1 -- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c158d70 msgr2=0x7f080c1f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0810aa2640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c158d70 0x7f080c1f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0810aa2640 1 -- 172.21.15.29:0/4098790116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c0b2720 con 0x7f080c0c5770 2023-10-16T15:53:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0810aa2640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 0x7f080c0ac890 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f080c0ccaa0 tx=0x7f0804007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.532+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080400ac40 con 0x7f080c0c5770 2023-10-16T15:53:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.533+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080400ada0 con 0x7f080c0c5770 2023-10-16T15:53:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.533+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804013610 con 0x7f080c0c5770 2023-10-16T15:53:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.533+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c0b2a00 con 0x7f080c0c5770 2023-10-16T15:53:29.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.533+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c0b2fc0 con 0x7f080c0c5770 2023-10-16T15:53:29.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.534+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c04e9b0 con 0x7f080c0c5770 2023-10-16T15:53:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.535+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0804013770 con 0x7f080c0c5770 2023-10-16T15:53:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.536+0000 7f0808ff9640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f07f806afc0 0x7f07f806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.536+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0804089fc0 con 0x7f080c0c5770 2023-10-16T15:53:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.537+0000 7f080bfff640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f07f806afc0 0x7f07f806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:29.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.538+0000 7f080bfff640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f07f806afc0 0x7f07f806d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07fc033000 tx=0x7f07fc009030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:29.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.540+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0804054f30 con 0x7f080c0c5770 2023-10-16T15:53:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:29.774+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f080c15aef0 con 0x7f080c0c5770 2023-10-16T15:53:30.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.263+0000 7f0808ff9640 1 -- 172.21.15.29:0/4098790116 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v109) v1 ==== 122+0+0 (secure 0 0 0) 0x7f080c15aef0 con 0x7f080c0c5770 2023-10-16T15:53:30.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f07f806afc0 msgr2=0x7f07f806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f07f806afc0 0x7f07f806d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07fc033000 tx=0x7f07fc009030 comp rx=0 tx=0).stop 2023-10-16T15:53:30.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 msgr2=0x7f080c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 0x7f080c0ac890 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f080c0ccaa0 tx=0x7f0804007b00 comp rx=0 tx=0).stop 2023-10-16T15:53:30.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 shutdown_connections 2023-10-16T15:53:30.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f07f806afc0 0x7f07f806d480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f080c0c5770 0x7f080c0ac890 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f080c158d70 0x7f080c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 --2- 172.21.15.29:0/4098790116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f07f40039c0 0x7f080c0c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 >> 172.21.15.29:0/4098790116 conn(0x7f080c0ce6a0 msgr2=0x7f080c15d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:30.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 shutdown_connections 2023-10-16T15:53:30.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.267+0000 7f08112a3640 1 -- 172.21.15.29:0/4098790116 wait complete. 2023-10-16T15:53:30.289 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T15:53:30.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:30.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.513+0000 7f26fffff640 1 Processor -- start 2023-10-16T15:53:30.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.513+0000 7f26fffff640 1 -- start start 2023-10-16T15:53:30.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.514+0000 7f26fffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2700159950 0x7f270015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.514+0000 7f26fffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2700070450 0x7f27000708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.515+0000 7f26fffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700070df0 con 0x7f270006c100 2023-10-16T15:53:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.515+0000 7f26fffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270015e490 con 0x7f2700070450 2023-10-16T15:53:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.515+0000 7f26fffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270015aef0 con 0x7f2700159950 2023-10-16T15:53:30.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.515+0000 7f26ff7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f270006c100 0x7f270006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:46606/0 (socket says 172.21.15.29:46606) 2023-10-16T15:53:30.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.515+0000 7f26ff7fe640 1 -- 172.21.15.29:0/568425490 learned_addr learned my addr 172.21.15.29:0/568425490 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.515+0000 7f26feffd640 1 --2- 172.21.15.29:0/568425490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2700159950 0x7f270015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.516+0000 7f26fe7fc640 1 --2- 172.21.15.29:0/568425490 >> v2:172.21.15.170:3300/0 conn(0x7f2700070450 0x7f27000708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.516+0000 7f26feffd640 1 -- 172.21.15.29:0/568425490 >> v2:172.21.15.170:3300/0 conn(0x7f2700070450 msgr2=0x7f27000708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.516+0000 7f26feffd640 1 --2- 172.21.15.29:0/568425490 >> v2:172.21.15.170:3300/0 conn(0x7f2700070450 0x7f27000708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.516+0000 7f26feffd640 1 -- 172.21.15.29:0/568425490 >> v1:172.21.15.170:6789/0 conn(0x7f270006c100 legacy=0x7f270006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.516+0000 7f26feffd640 1 -- 172.21.15.29:0/568425490 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270006cc80 con 0x7f2700159950 2023-10-16T15:53:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.518+0000 7f26feffd640 1 --2- 172.21.15.29:0/568425490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2700159950 0x7f270015a9b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009d90 tx=0x7f26f002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aed37df2c29e27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.518+0000 7f26fdffb640 1 -- 172.21.15.29:0/568425490 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f002cbc0 con 0x7f2700159950 2023-10-16T15:53:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.518+0000 7f26fdffb640 1 -- 172.21.15.29:0/568425490 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f002cd20 con 0x7f2700159950 2023-10-16T15:53:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.518+0000 7f26fdffb640 1 -- 172.21.15.29:0/568425490 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0034660 con 0x7f2700159950 2023-10-16T15:53:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 -- 172.21.15.29:0/568425490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2700159950 msgr2=0x7f270015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 --2- 172.21.15.29:0/568425490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2700159950 0x7f270015a9b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009d90 tx=0x7f26f002be10 comp rx=0 tx=0).stop 2023-10-16T15:53:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 -- 172.21.15.29:0/568425490 shutdown_connections 2023-10-16T15:53:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 --2- 172.21.15.29:0/568425490 >> v2:172.21.15.170:3300/0 conn(0x7f2700070450 0x7f27000708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 --2- 172.21.15.29:0/568425490 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2700159950 0x7f270015a9b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 -- 172.21.15.29:0/568425490 >> 172.21.15.29:0/568425490 conn(0x7f27000ce6a0 msgr2=0x7f27000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 -- 172.21.15.29:0/568425490 shutdown_connections 2023-10-16T15:53:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.519+0000 7f26fffff640 1 -- 172.21.15.29:0/568425490 wait complete. 2023-10-16T15:53:30.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.521+0000 7f26fffff640 1 Processor -- start 2023-10-16T15:53:30.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.521+0000 7f26fffff640 1 -- start start 2023-10-16T15:53:30.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26fffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f270006c100 0x7f2700146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:30.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26fffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2700070450 0x7f27000c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:30.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26fffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 0x7f27000c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:30.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26fffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000c61f0 con 0x7f2700159950 2023-10-16T15:53:30.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26fffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270014a670 con 0x7f2700070450 2023-10-16T15:53:30.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26fffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270014a390 con 0x7f270006c100 2023-10-16T15:53:30.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 0x7f27000c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:30.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 0x7f27000c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45454/0 (socket says 172.21.15.29:45454) 2023-10-16T15:53:30.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26ff7fe640 1 -- 172.21.15.29:0/3730808380 learned_addr learned my addr 172.21.15.29:0/3730808380 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:30.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.522+0000 7f26feffd640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f270006c100 0x7f2700146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:30.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26fe7fc640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2700070450 0x7f27000c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:30.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26ff7fe640 1 -- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f270006c100 msgr2=0x7f2700146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26ff7fe640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f270006c100 0x7f2700146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26ff7fe640 1 -- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2700070450 msgr2=0x7f27000c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26ff7fe640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2700070450 0x7f27000c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26ff7fe640 1 -- 172.21.15.29:0/3730808380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f0009a70 con 0x7f2700159950 2023-10-16T15:53:30.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26feffd640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f270006c100 0x7f2700146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:30.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26ff7fe640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 0x7f27000c5cb0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c370 tx=0x7f26f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:30.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.523+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8005ce0 con 0x7f2700159950 2023-10-16T15:53:30.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.524+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f8005e40 con 0x7f2700159950 2023-10-16T15:53:30.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.524+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270015b210 con 0x7f2700159950 2023-10-16T15:53:30.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.524+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f80072d0 con 0x7f2700159950 2023-10-16T15:53:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.524+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270015b7d0 con 0x7f2700159950 2023-10-16T15:53:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.525+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270004e9b0 con 0x7f2700159950 2023-10-16T15:53:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.526+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f26f80074f0 con 0x7f2700159950 2023-10-16T15:53:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.527+0000 7f26e77fe640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f26ec06af90 0x7f26ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.527+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f26f8088d00 con 0x7f2700159950 2023-10-16T15:53:30.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.528+0000 7f26feffd640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f26ec06af90 0x7f26ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:30.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.529+0000 7f26feffd640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f26ec06af90 0x7f26ec06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26f0036000 tx=0x7f26f0030000 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:30.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.531+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f26f8053c10 con 0x7f2700159950 2023-10-16T15:53:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.763+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27000b4620 con 0x7f2700159950 2023-10-16T15:53:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.764+0000 7f26e77fe640 1 -- 172.21.15.29:0/3730808380 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4932 (secure 0 0 0) 0x7f26f8057230 con 0x7f2700159950 2023-10-16T15:53:30.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:30.767 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":109,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104},{"gid":8761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T15:53:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 109 2023-10-16T15:53:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f26ec06af90 msgr2=0x7f26ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f26ec06af90 0x7f26ec06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26f0036000 tx=0x7f26f0030000 comp rx=0 tx=0).stop 2023-10-16T15:53:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 msgr2=0x7f27000c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 0x7f27000c5cb0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c370 tx=0x7f26f800c830 comp rx=0 tx=0).stop 2023-10-16T15:53:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 shutdown_connections 2023-10-16T15:53:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f26ec06af90 0x7f26ec06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2700159950 0x7f27000c5cb0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2700070450 0x7f27000c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 --2- 172.21.15.29:0/3730808380 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f270006c100 0x7f2700146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 >> 172.21.15.29:0/3730808380 conn(0x7f27000ce6a0 msgr2=0x7f2700159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 shutdown_connections 2023-10-16T15:53:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:30.767+0000 7f26fffff640 1 -- 172.21.15.29:0/3730808380 wait complete. 2023-10-16T15:53:30.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:31.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.013+0000 7f422bfff640 1 Processor -- start 2023-10-16T15:53:31.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.014+0000 7f422bfff640 1 -- start start 2023-10-16T15:53:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.015+0000 7f422bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c06c040 0x7f422c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.015+0000 7f422bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f422c0ad970 0x7f422c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.015+0000 7f422bfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f422c15ebc0 0x7f422c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.015+0000 7f422bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c06c960 con 0x7f422c0ad970 2023-10-16T15:53:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.015+0000 7f422bfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c15b9d0 con 0x7f422c15ebc0 2023-10-16T15:53:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f422bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c15bcb0 con 0x7f422c06c040 2023-10-16T15:53:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f422c0ad970 0x7f422c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f422c0ad970 0x7f422c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45462/0 (socket says 172.21.15.29:45462) 2023-10-16T15:53:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 -- 172.21.15.29:0/296252356 learned_addr learned my addr 172.21.15.29:0/296252356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:31.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f422affd640 1 --2- 172.21.15.29:0/296252356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c06c040 0x7f422c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f422b7fe640 1 --2- 172.21.15.29:0/296252356 >> v2:172.21.15.170:3300/0 conn(0x7f422c15ebc0 0x7f422c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 -- 172.21.15.29:0/296252356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c06c040 msgr2=0x7f422c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 --2- 172.21.15.29:0/296252356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c06c040 0x7f422c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 -- 172.21.15.29:0/296252356 >> v2:172.21.15.170:3300/0 conn(0x7f422c15ebc0 msgr2=0x7f422c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 --2- 172.21.15.29:0/296252356 >> v2:172.21.15.170:3300/0 conn(0x7f422c15ebc0 0x7f422c15f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.016+0000 7f4223fff640 1 -- 172.21.15.29:0/296252356 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c15bf90 con 0x7f422c0ad970 2023-10-16T15:53:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.018+0000 7f4223fff640 1 --2- 172.21.15.29:0/296252356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c0ad970 0x7f422c15e680 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f4214009870 tx=0x7f421402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd9fa7c20334189 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.018+0000 7f422a7fc640 1 -- 172.21.15.29:0/296252356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214031640 con 0x7f422c0ad970 2023-10-16T15:53:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.018+0000 7f422a7fc640 1 -- 172.21.15.29:0/296252356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42140317a0 con 0x7f422c0ad970 2023-10-16T15:53:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.018+0000 7f422a7fc640 1 -- 172.21.15.29:0/296252356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214031b30 con 0x7f422c0ad970 2023-10-16T15:53:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 -- 172.21.15.29:0/296252356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c0ad970 msgr2=0x7f422c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 --2- 172.21.15.29:0/296252356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c0ad970 0x7f422c15e680 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f4214009870 tx=0x7f421402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 -- 172.21.15.29:0/296252356 shutdown_connections 2023-10-16T15:53:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 --2- 172.21.15.29:0/296252356 >> v2:172.21.15.170:3300/0 conn(0x7f422c15ebc0 0x7f422c15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 --2- 172.21.15.29:0/296252356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c0ad970 0x7f422c15e680 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 --2- 172.21.15.29:0/296252356 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c06c040 0x7f422c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 -- 172.21.15.29:0/296252356 >> 172.21.15.29:0/296252356 conn(0x7f422c0ce6a0 msgr2=0x7f422c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 -- 172.21.15.29:0/296252356 shutdown_connections 2023-10-16T15:53:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.019+0000 7f422bfff640 1 -- 172.21.15.29:0/296252356 wait complete. 2023-10-16T15:53:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.021+0000 7f422bfff640 1 Processor -- start 2023-10-16T15:53:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.021+0000 7f422bfff640 1 -- start start 2023-10-16T15:53:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 0x7f422c14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422bfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c0ad970 0x7f422c15a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422bfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422c15ebc0 0x7f422c15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c158d70 con 0x7f422c06c040 2023-10-16T15:53:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422bfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0aecd0 con 0x7f422c15ebc0 2023-10-16T15:53:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422bfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0aefb0 con 0x7f422c0ad970 2023-10-16T15:53:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 0x7f422c14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 0x7f422c14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45474/0 (socket says 172.21.15.29:45474) 2023-10-16T15:53:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.022+0000 7f422affd640 1 -- 172.21.15.29:0/2899885613 learned_addr learned my addr 172.21.15.29:0/2899885613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f4223fff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c0ad970 0x7f422c15a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422affd640 1 -- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c0ad970 msgr2=0x7f422c15a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422affd640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c0ad970 0x7f422c15a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422b7fe640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422c15ebc0 0x7f422c15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422affd640 1 -- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422c15ebc0 msgr2=0x7f422c15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422affd640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422c15ebc0 0x7f422c15a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422affd640 1 -- 172.21.15.29:0/2899885613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140094d0 con 0x7f422c06c040 2023-10-16T15:53:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f422affd640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 0x7f422c14c960 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d800 tx=0x7f421c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c0096a0 con 0x7f422c06c040 2023-10-16T15:53:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c009800 con 0x7f422c06c040 2023-10-16T15:53:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.023+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c009990 con 0x7f422c06c040 2023-10-16T15:53:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.024+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c0af290 con 0x7f422c06c040 2023-10-16T15:53:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.024+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c0af850 con 0x7f422c06c040 2023-10-16T15:53:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.025+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c06b330 con 0x7f422c06c040 2023-10-16T15:53:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.026+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f421c009af0 con 0x7f422c06c040 2023-10-16T15:53:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.027+0000 7f42237fe640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f420c06af90 0x7f420c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.027+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f421c089b90 con 0x7f422c06c040 2023-10-16T15:53:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.028+0000 7f4223fff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f420c06af90 0x7f420c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.029+0000 7f4223fff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f420c06af90 0x7f420c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4214036000 tx=0x7f421403c040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.031+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f421c054ba0 con 0x7f422c06c040 2023-10-16T15:53:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.224+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f422c0cc520 con 0x7f422c06c040 2023-10-16T15:53:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.228+0000 7f42237fe640 1 -- 172.21.15.29:0/2899885613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+16627 (secure 0 0 0) 0x7f421c00b3a0 con 0x7f422c06c040 2023-10-16T15:53:31.231 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:31.231 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":225,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:28.224740+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f420c06af90 msgr2=0x7f420c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f420c06af90 0x7f420c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4214036000 tx=0x7f421403c040 comp rx=0 tx=0).stop 2023-10-16T15:53:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 msgr2=0x7f422c14c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 0x7f422c14c960 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d800 tx=0x7f421c00dcc0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 shutdown_connections 2023-10-16T15:53:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f420c06af90 0x7f420c06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f422c15ebc0 0x7f422c15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f422c0ad970 0x7f422c15a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 --2- 172.21.15.29:0/2899885613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f422c06c040 0x7f422c14c960 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 >> 172.21.15.29:0/2899885613 conn(0x7f422c0ce6a0 msgr2=0x7f422c0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 shutdown_connections 2023-10-16T15:53:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.231+0000 7f422bfff640 1 -- 172.21.15.29:0/2899885613 wait complete. 2023-10-16T15:53:31.254 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:53:31.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:31.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.482+0000 7ff023b4c640 1 Processor -- start 2023-10-16T15:53:31.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.483+0000 7ff023b4c640 1 -- start start 2023-10-16T15:53:31.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff023b4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff01c15c850 0x7ff01c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff023b4c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff01c0cdc40 0x7ff01c06adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff023b4c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c0ce690 0x7ff01c06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff023b4c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c06ffe0 con 0x7ff01c15c850 2023-10-16T15:53:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff023b4c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c0cb9e0 con 0x7ff01c0cdc40 2023-10-16T15:53:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff023b4c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0ccf40 con 0x7ff01c0ce690 2023-10-16T15:53:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff0218e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff01c15c850 0x7ff01c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff0218e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff01c15c850 0x7ff01c0cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45490/0 (socket says 172.21.15.29:45490) 2023-10-16T15:53:31.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.484+0000 7ff0218e8640 1 -- 172.21.15.29:0/865760878 learned_addr learned my addr 172.21.15.29:0/865760878 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:31.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0210e7640 1 --2- 172.21.15.29:0/865760878 >> v2:172.21.15.170:3300/0 conn(0x7ff01c0cdc40 0x7ff01c06adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0220e9640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c0ce690 0x7ff01c06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0218e8640 1 -- 172.21.15.29:0/865760878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c0ce690 msgr2=0x7ff01c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:31.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0218e8640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c0ce690 0x7ff01c06b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0218e8640 1 -- 172.21.15.29:0/865760878 >> v2:172.21.15.170:3300/0 conn(0x7ff01c0cdc40 msgr2=0x7ff01c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:31.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0218e8640 1 --2- 172.21.15.29:0/865760878 >> v2:172.21.15.170:3300/0 conn(0x7ff01c0cdc40 0x7ff01c06adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0218e8640 1 -- 172.21.15.29:0/865760878 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c0cb420 con 0x7ff01c15c850 2023-10-16T15:53:31.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0220e9640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c0ce690 0x7ff01c06b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:31.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.485+0000 7ff0210e7640 1 --2- 172.21.15.29:0/865760878 >> v2:172.21.15.170:3300/0 conn(0x7ff01c0cdc40 0x7ff01c06adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:31.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.486+0000 7ff0218e8640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c15c850 0x7ff01c0cd700 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7ff01c06f4a0 tx=0x7ff01802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7cfd0c05a220a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:31.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.486+0000 7ff0208e6640 1 -- 172.21.15.29:0/865760878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018031750 con 0x7ff01c15c850 2023-10-16T15:53:31.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.486+0000 7ff0208e6640 1 -- 172.21.15.29:0/865760878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0180318b0 con 0x7ff01c15c850 2023-10-16T15:53:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.486+0000 7ff0208e6640 1 -- 172.21.15.29:0/865760878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018031c40 con 0x7ff01c15c850 2023-10-16T15:53:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 -- 172.21.15.29:0/865760878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c15c850 msgr2=0x7ff01c0cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c15c850 0x7ff01c0cd700 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7ff01c06f4a0 tx=0x7ff01802b7f0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 -- 172.21.15.29:0/865760878 shutdown_connections 2023-10-16T15:53:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c0ce690 0x7ff01c06b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 --2- 172.21.15.29:0/865760878 >> v2:172.21.15.170:3300/0 conn(0x7ff01c0cdc40 0x7ff01c06adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 --2- 172.21.15.29:0/865760878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c15c850 0x7ff01c0cd700 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 -- 172.21.15.29:0/865760878 >> 172.21.15.29:0/865760878 conn(0x7ff01c0b0d30 msgr2=0x7ff01c0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:31.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 -- 172.21.15.29:0/865760878 shutdown_connections 2023-10-16T15:53:31.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.487+0000 7ff023b4c640 1 -- 172.21.15.29:0/865760878 wait complete. 2023-10-16T15:53:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.489+0000 7ff023b4c640 1 Processor -- start 2023-10-16T15:53:31.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.489+0000 7ff023b4c640 1 -- start start 2023-10-16T15:53:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff023b4c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff01c0cdc40 0x7ff01c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff023b4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 0x7ff01c0c3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff023b4c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c15c850 0x7ff01c0c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff023b4c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0b20b0 con 0x7ff01c0ce690 2023-10-16T15:53:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff023b4c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0c4500 con 0x7ff01c0cdc40 2023-10-16T15:53:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff023b4c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0b19f0 con 0x7ff01c15c850 2023-10-16T15:53:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff0210e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 0x7ff01c0c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff0210e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 0x7ff01c0c3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45496/0 (socket says 172.21.15.29:45496) 2023-10-16T15:53:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.490+0000 7ff0210e7640 1 -- 172.21.15.29:0/3503432584 learned_addr learned my addr 172.21.15.29:0/3503432584 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:31.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0218e8640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff01c0cdc40 0x7ff01c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0220e9640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c15c850 0x7ff01c0c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0210e7640 1 -- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c15c850 msgr2=0x7ff01c0c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0210e7640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c15c850 0x7ff01c0c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0210e7640 1 -- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff01c0cdc40 msgr2=0x7ff01c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0210e7640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff01c0cdc40 0x7ff01c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0210e7640 1 -- 172.21.15.29:0/3503432584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180094d0 con 0x7ff01c0ce690 2023-10-16T15:53:31.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0220e9640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c15c850 0x7ff01c0c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:31.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0210e7640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 0x7ff01c0c3790 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c7f0 tx=0x7ff00c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:31.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c007820 con 0x7ff01c0ce690 2023-10-16T15:53:31.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.491+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c007980 con 0x7ff01c0ce690 2023-10-16T15:53:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.492+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01c0b1cd0 con 0x7ff01c0ce690 2023-10-16T15:53:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.492+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c007d30 con 0x7ff01c0ce690 2023-10-16T15:53:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.492+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c0b4840 con 0x7ff01c0ce690 2023-10-16T15:53:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.493+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c0b1850 con 0x7ff01c0ce690 2023-10-16T15:53:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.495+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff00c0113f0 con 0x7ff01c0ce690 2023-10-16T15:53:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.495+0000 7ff0127fc640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff00006af90 0x7ff00006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:31.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.496+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff00c089be0 con 0x7ff01c0ce690 2023-10-16T15:53:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.496+0000 7ff0218e8640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff00006af90 0x7ff00006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.498+0000 7ff0218e8640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff00006af90 0x7ff00006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff018035000 tx=0x7ff018008030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.500+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff01c0b1850 con 0x7ff01c0ce690 2023-10-16T15:53:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.688+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff01c0cd1e0 con 0x7ff00006af90 2023-10-16T15:53:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.718+0000 7ff0127fc640 1 -- 172.21.15.29:0/3503432584 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287017 (secure 0 0 0) 0x7ff018031710 con 0x7ff00006af90 2023-10-16T15:53:31.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff00006af90 msgr2=0x7ff00006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff00006af90 0x7ff00006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff018035000 tx=0x7ff018008030 comp rx=0 tx=0).stop 2023-10-16T15:53:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 msgr2=0x7ff01c0c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 0x7ff01c0c3790 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c7f0 tx=0x7ff00c00ccb0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 shutdown_connections 2023-10-16T15:53:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff00006af90 0x7ff00006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff01c0ce690 0x7ff01c0c3790 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff01c15c850 0x7ff01c0c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 --2- 172.21.15.29:0/3503432584 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff01c0cdc40 0x7ff01c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 >> 172.21.15.29:0/3503432584 conn(0x7ff01c0b0d30 msgr2=0x7ff01c09ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 shutdown_connections 2023-10-16T15:53:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:31.725+0000 7ff023b4c640 1 -- 172.21.15.29:0/3503432584 wait complete. 2023-10-16T15:53:31.748 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T15:53:30.457614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":627,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":376852,"kb_used_data":23272,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754597868,"statfs":{"total":773094113280,"available":772708216832,"internally_reserved":0,"allocated":23830528,"data_stored":12250332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":361942585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.292337"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612164+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:28.612164+0000","last_peered":"2023-10-16T15:53:28.612164+0000","last_clean":"2023-10-16T15:53:28.612164+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:28.612164+0000","last_undegraded":"2023-10-16T15:53:28.612164+0000","last_fullsized":"2023-10-16T15:53:28.612164+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235156+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:28.235156+0000","last_peered":"2023-10-16T15:53:28.235156+0000","last_clean":"2023-10-16T15:53:28.235156+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:28.235156+0000","last_undegraded":"2023-10-16T15:53:28.235156+0000","last_fullsized":"2023-10-16T15:53:28.235156+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609117+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:28.609117+0000","last_peered":"2023-10-16T15:53:28.609117+0000","last_clean":"2023-10-16T15:53:28.609117+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:28.609117+0000","last_undegraded":"2023-10-16T15:53:28.609117+0000","last_fullsized":"2023-10-16T15:53:28.609117+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236949+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:28.236949+0000","last_peered":"2023-10-16T15:53:28.236949+0000","last_clean":"2023-10-16T15:53:28.236949+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:28.236949+0000","last_undegraded":"2023-10-16T15:53:28.236949+0000","last_fullsized":"2023-10-16T15:53:28.236949+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.233209+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:28.233209+0000","last_peered":"2023-10-16T15:53:28.233209+0000","last_clean":"2023-10-16T15:53:28.233209+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:28.233209+0000","last_undegraded":"2023-10-16T15:53:28.233209+0000","last_fullsized":"2023-10-16T15:53:28.233209+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231690+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:28.231690+0000","last_peered":"2023-10-16T15:53:28.231690+0000","last_clean":"2023-10-16T15:53:28.231690+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:28.231690+0000","last_undegraded":"2023-10-16T15:53:28.231690+0000","last_fullsized":"2023-10-16T15:53:28.231690+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609729+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:28.609729+0000","last_peered":"2023-10-16T15:53:28.609729+0000","last_clean":"2023-10-16T15:53:28.609729+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:28.609729+0000","last_undegraded":"2023-10-16T15:53:28.609729+0000","last_fullsized":"2023-10-16T15:53:28.609729+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231173+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:28.231173+0000","last_peered":"2023-10-16T15:53:28.231173+0000","last_clean":"2023-10-16T15:53:28.231173+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:28.231173+0000","last_undegraded":"2023-10-16T15:53:28.231173+0000","last_fullsized":"2023-10-16T15:53:28.231173+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235995+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:28.235995+0000","last_peered":"2023-10-16T15:53:28.235995+0000","last_clean":"2023-10-16T15:53:28.235995+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:28.235995+0000","last_undegraded":"2023-10-16T15:53:28.235995+0000","last_fullsized":"2023-10-16T15:53:28.235995+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611854+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:28.611854+0000","last_peered":"2023-10-16T15:53:28.611854+0000","last_clean":"2023-10-16T15:53:28.611854+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:28.611854+0000","last_undegraded":"2023-10-16T15:53:28.611854+0000","last_fullsized":"2023-10-16T15:53:28.611854+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609394+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:28.609394+0000","last_peered":"2023-10-16T15:53:28.609394+0000","last_clean":"2023-10-16T15:53:28.609394+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:28.609394+0000","last_undegraded":"2023-10-16T15:53:28.609394+0000","last_fullsized":"2023-10-16T15:53:28.609394+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610529+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:28.610529+0000","last_peered":"2023-10-16T15:53:28.610529+0000","last_clean":"2023-10-16T15:53:28.610529+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:28.610529+0000","last_undegraded":"2023-10-16T15:53:28.610529+0000","last_fullsized":"2023-10-16T15:53:28.610529+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610794+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:28.610794+0000","last_peered":"2023-10-16T15:53:28.610794+0000","last_clean":"2023-10-16T15:53:28.610794+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:28.610794+0000","last_undegraded":"2023-10-16T15:53:28.610794+0000","last_fullsized":"2023-10-16T15:53:28.610794+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610587+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:28.610587+0000","last_peered":"2023-10-16T15:53:28.610587+0000","last_clean":"2023-10-16T15:53:28.610587+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:28.610587+0000","last_undegraded":"2023-10-16T15:53:28.610587+0000","last_fullsized":"2023-10-16T15:53:28.610587+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609875+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:28.609875+0000","last_peered":"2023-10-16T15:53:28.609875+0000","last_clean":"2023-10-16T15:53:28.609875+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:28.609875+0000","last_undegraded":"2023-10-16T15:53:28.609875+0000","last_fullsized":"2023-10-16T15:53:28.609875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610601+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:28.610601+0000","last_peered":"2023-10-16T15:53:28.610601+0000","last_clean":"2023-10-16T15:53:28.610601+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:28.610601+0000","last_undegraded":"2023-10-16T15:53:28.610601+0000","last_fullsized":"2023-10-16T15:53:28.610601+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612493+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:28.612493+0000","last_peered":"2023-10-16T15:53:28.612493+0000","last_clean":"2023-10-16T15:53:28.612493+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:28.612493+0000","last_undegraded":"2023-10-16T15:53:28.612493+0000","last_fullsized":"2023-10-16T15:53:28.612493+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611993+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:28.611993+0000","last_peered":"2023-10-16T15:53:28.611993+0000","last_clean":"2023-10-16T15:53:28.611993+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:28.611993+0000","last_undegraded":"2023-10-16T15:53:28.611993+0000","last_fullsized":"2023-10-16T15:53:28.611993+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612888+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:28.612888+0000","last_peered":"2023-10-16T15:53:28.612888+0000","last_clean":"2023-10-16T15:53:28.612888+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:28.612888+0000","last_undegraded":"2023-10-16T15:53:28.612888+0000","last_fullsized":"2023-10-16T15:53:28.612888+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.233138+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:28.233138+0000","last_peered":"2023-10-16T15:53:28.233138+0000","last_clean":"2023-10-16T15:53:28.233138+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:28.233138+0000","last_undegraded":"2023-10-16T15:53:28.233138+0000","last_fullsized":"2023-10-16T15:53:28.233138+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236848+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:28.236848+0000","last_peered":"2023-10-16T15:53:28.236848+0000","last_clean":"2023-10-16T15:53:28.236848+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:28.236848+0000","last_undegraded":"2023-10-16T15:53:28.236848+0000","last_fullsized":"2023-10-16T15:53:28.236848+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406627+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:28.406627+0000","last_peered":"2023-10-16T15:53:28.406627+0000","last_clean":"2023-10-16T15:53:28.406627+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:28.406627+0000","last_undegraded":"2023-10-16T15:53:28.406627+0000","last_fullsized":"2023-10-16T15:53:28.406627+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406745+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:28.406745+0000","last_peered":"2023-10-16T15:53:28.406745+0000","last_clean":"2023-10-16T15:53:28.406745+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:28.406745+0000","last_undegraded":"2023-10-16T15:53:28.406745+0000","last_fullsized":"2023-10-16T15:53:28.406745+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234887+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:28.234887+0000","last_peered":"2023-10-16T15:53:28.234887+0000","last_clean":"2023-10-16T15:53:28.234887+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:28.234887+0000","last_undegraded":"2023-10-16T15:53:28.234887+0000","last_fullsized":"2023-10-16T15:53:28.234887+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235499+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:28.235499+0000","last_peered":"2023-10-16T15:53:28.235499+0000","last_clean":"2023-10-16T15:53:28.235499+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:28.235499+0000","last_undegraded":"2023-10-16T15:53:28.235499+0000","last_fullsized":"2023-10-16T15:53:28.235499+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.613187+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:28.613187+0000","last_peered":"2023-10-16T15:53:28.613187+0000","last_clean":"2023-10-16T15:53:28.613187+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:28.613187+0000","last_undegraded":"2023-10-16T15:53:28.613187+0000","last_fullsized":"2023-10-16T15:53:28.613187+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610848+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:28.610848+0000","last_peered":"2023-10-16T15:53:28.610848+0000","last_clean":"2023-10-16T15:53:28.610848+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:28.610848+0000","last_undegraded":"2023-10-16T15:53:28.610848+0000","last_fullsized":"2023-10-16T15:53:28.610848+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608340+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:28.608340+0000","last_peered":"2023-10-16T15:53:28.608340+0000","last_clean":"2023-10-16T15:53:28.608340+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:28.608340+0000","last_undegraded":"2023-10-16T15:53:28.608340+0000","last_fullsized":"2023-10-16T15:53:28.608340+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610516+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:28.610516+0000","last_peered":"2023-10-16T15:53:28.610516+0000","last_clean":"2023-10-16T15:53:28.610516+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:28.610516+0000","last_undegraded":"2023-10-16T15:53:28.610516+0000","last_fullsized":"2023-10-16T15:53:28.610516+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234710+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:28.234710+0000","last_peered":"2023-10-16T15:53:28.234710+0000","last_clean":"2023-10-16T15:53:28.234710+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:28.234710+0000","last_undegraded":"2023-10-16T15:53:28.234710+0000","last_fullsized":"2023-10-16T15:53:28.234710+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610198+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:28.610198+0000","last_peered":"2023-10-16T15:53:28.610198+0000","last_clean":"2023-10-16T15:53:28.610198+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:28.610198+0000","last_undegraded":"2023-10-16T15:53:28.610198+0000","last_fullsized":"2023-10-16T15:53:28.610198+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609901+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:28.609901+0000","last_peered":"2023-10-16T15:53:28.609901+0000","last_clean":"2023-10-16T15:53:28.609901+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:28.609901+0000","last_undegraded":"2023-10-16T15:53:28.609901+0000","last_fullsized":"2023-10-16T15:53:28.609901+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235459+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:28.235459+0000","last_peered":"2023-10-16T15:53:28.235459+0000","last_clean":"2023-10-16T15:53:28.235459+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:28.235459+0000","last_undegraded":"2023-10-16T15:53:28.235459+0000","last_fullsized":"2023-10-16T15:53:28.235459+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611017+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:28.611017+0000","last_peered":"2023-10-16T15:53:28.611017+0000","last_clean":"2023-10-16T15:53:28.611017+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:28.611017+0000","last_undegraded":"2023-10-16T15:53:28.611017+0000","last_fullsized":"2023-10-16T15:53:28.611017+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.407107+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:28.407107+0000","last_peered":"2023-10-16T15:53:28.407107+0000","last_clean":"2023-10-16T15:53:28.407107+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:28.407107+0000","last_undegraded":"2023-10-16T15:53:28.407107+0000","last_fullsized":"2023-10-16T15:53:28.407107+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235727+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:28.235727+0000","last_peered":"2023-10-16T15:53:28.235727+0000","last_clean":"2023-10-16T15:53:28.235727+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:28.235727+0000","last_undegraded":"2023-10-16T15:53:28.235727+0000","last_fullsized":"2023-10-16T15:53:28.235727+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611077+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:28.611077+0000","last_peered":"2023-10-16T15:53:28.611077+0000","last_clean":"2023-10-16T15:53:28.611077+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:28.611077+0000","last_undegraded":"2023-10-16T15:53:28.611077+0000","last_fullsized":"2023-10-16T15:53:28.611077+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609358+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:28.609358+0000","last_peered":"2023-10-16T15:53:28.609358+0000","last_clean":"2023-10-16T15:53:28.609358+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:28.609358+0000","last_undegraded":"2023-10-16T15:53:28.609358+0000","last_fullsized":"2023-10-16T15:53:28.609358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612023+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:28.612023+0000","last_peered":"2023-10-16T15:53:28.612023+0000","last_clean":"2023-10-16T15:53:28.612023+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:28.612023+0000","last_undegraded":"2023-10-16T15:53:28.612023+0000","last_fullsized":"2023-10-16T15:53:28.612023+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237011+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:28.237011+0000","last_peered":"2023-10-16T15:53:28.237011+0000","last_clean":"2023-10-16T15:53:28.237011+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:28.237011+0000","last_undegraded":"2023-10-16T15:53:28.237011+0000","last_fullsized":"2023-10-16T15:53:28.237011+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236777+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:28.236777+0000","last_peered":"2023-10-16T15:53:28.236777+0000","last_clean":"2023-10-16T15:53:28.236777+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:28.236777+0000","last_undegraded":"2023-10-16T15:53:28.236777+0000","last_fullsized":"2023-10-16T15:53:28.236777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231877+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:28.231877+0000","last_peered":"2023-10-16T15:53:28.231877+0000","last_clean":"2023-10-16T15:53:28.231877+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:28.231877+0000","last_undegraded":"2023-10-16T15:53:28.231877+0000","last_fullsized":"2023-10-16T15:53:28.231877+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231658+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:28.231658+0000","last_peered":"2023-10-16T15:53:28.231658+0000","last_clean":"2023-10-16T15:53:28.231658+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:28.231658+0000","last_undegraded":"2023-10-16T15:53:28.231658+0000","last_fullsized":"2023-10-16T15:53:28.231658+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.230704+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:28.230704+0000","last_peered":"2023-10-16T15:53:28.230704+0000","last_clean":"2023-10-16T15:53:28.230704+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:28.230704+0000","last_undegraded":"2023-10-16T15:53:28.230704+0000","last_fullsized":"2023-10-16T15:53:28.230704+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236361+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:28.236361+0000","last_peered":"2023-10-16T15:53:28.236361+0000","last_clean":"2023-10-16T15:53:28.236361+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:28.236361+0000","last_undegraded":"2023-10-16T15:53:28.236361+0000","last_fullsized":"2023-10-16T15:53:28.236361+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406045+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:28.406045+0000","last_peered":"2023-10-16T15:53:28.406045+0000","last_clean":"2023-10-16T15:53:28.406045+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:28.406045+0000","last_undegraded":"2023-10-16T15:53:28.406045+0000","last_fullsized":"2023-10-16T15:53:28.406045+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237377+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:28.237377+0000","last_peered":"2023-10-16T15:53:28.237377+0000","last_clean":"2023-10-16T15:53:28.237377+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:28.237377+0000","last_undegraded":"2023-10-16T15:53:28.237377+0000","last_fullsized":"2023-10-16T15:53:28.237377+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.232932+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:28.232932+0000","last_peered":"2023-10-16T15:53:28.232932+0000","last_clean":"2023-10-16T15:53:28.232932+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:28.232932+0000","last_undegraded":"2023-10-16T15:53:28.232932+0000","last_fullsized":"2023-10-16T15:53:28.232932+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608618+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:28.608618+0000","last_peered":"2023-10-16T15:53:28.608618+0000","last_clean":"2023-10-16T15:53:28.608618+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:28.608618+0000","last_undegraded":"2023-10-16T15:53:28.608618+0000","last_fullsized":"2023-10-16T15:53:28.608618+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406908+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:28.406908+0000","last_peered":"2023-10-16T15:53:28.406908+0000","last_clean":"2023-10-16T15:53:28.406908+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:28.406908+0000","last_undegraded":"2023-10-16T15:53:28.406908+0000","last_fullsized":"2023-10-16T15:53:28.406908+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236756+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:28.236756+0000","last_peered":"2023-10-16T15:53:28.236756+0000","last_clean":"2023-10-16T15:53:28.236756+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:28.236756+0000","last_undegraded":"2023-10-16T15:53:28.236756+0000","last_fullsized":"2023-10-16T15:53:28.236756+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612899+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:28.612899+0000","last_peered":"2023-10-16T15:53:28.612899+0000","last_clean":"2023-10-16T15:53:28.612899+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:28.612899+0000","last_undegraded":"2023-10-16T15:53:28.612899+0000","last_fullsized":"2023-10-16T15:53:28.612899+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611488+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:28.611488+0000","last_peered":"2023-10-16T15:53:28.611488+0000","last_clean":"2023-10-16T15:53:28.611488+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:28.611488+0000","last_undegraded":"2023-10-16T15:53:28.611488+0000","last_fullsized":"2023-10-16T15:53:28.611488+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"224'6","reported_seq":18,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237201+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:28.237201+0000","last_peered":"2023-10-16T15:53:28.237201+0000","last_clean":"2023-10-16T15:53:28.237201+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:28.237201+0000","last_undegraded":"2023-10-16T15:53:28.237201+0000","last_fullsized":"2023-10-16T15:53:28.237201+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231602+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:28.231602+0000","last_peered":"2023-10-16T15:53:28.231602+0000","last_clean":"2023-10-16T15:53:28.231602+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:28.231602+0000","last_undegraded":"2023-10-16T15:53:28.231602+0000","last_fullsized":"2023-10-16T15:53:28.231602+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.407124+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:28.407124+0000","last_peered":"2023-10-16T15:53:28.407124+0000","last_clean":"2023-10-16T15:53:28.407124+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:28.407124+0000","last_undegraded":"2023-10-16T15:53:28.407124+0000","last_fullsized":"2023-10-16T15:53:28.407124+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611510+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:28.611510+0000","last_peered":"2023-10-16T15:53:28.611510+0000","last_clean":"2023-10-16T15:53:28.611510+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:28.611510+0000","last_undegraded":"2023-10-16T15:53:28.611510+0000","last_fullsized":"2023-10-16T15:53:28.611510+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406837+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:28.406837+0000","last_peered":"2023-10-16T15:53:28.406837+0000","last_clean":"2023-10-16T15:53:28.406837+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:28.406837+0000","last_undegraded":"2023-10-16T15:53:28.406837+0000","last_fullsized":"2023-10-16T15:53:28.406837+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608102+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:28.608102+0000","last_peered":"2023-10-16T15:53:28.608102+0000","last_clean":"2023-10-16T15:53:28.608102+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:28.608102+0000","last_undegraded":"2023-10-16T15:53:28.608102+0000","last_fullsized":"2023-10-16T15:53:28.608102+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610295+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:28.610295+0000","last_peered":"2023-10-16T15:53:28.610295+0000","last_clean":"2023-10-16T15:53:28.610295+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:28.610295+0000","last_undegraded":"2023-10-16T15:53:28.610295+0000","last_fullsized":"2023-10-16T15:53:28.610295+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610016+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:28.610016+0000","last_peered":"2023-10-16T15:53:28.610016+0000","last_clean":"2023-10-16T15:53:28.610016+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:28.610016+0000","last_undegraded":"2023-10-16T15:53:28.610016+0000","last_fullsized":"2023-10-16T15:53:28.610016+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":20,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608624+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:28.608624+0000","last_peered":"2023-10-16T15:53:28.608624+0000","last_clean":"2023-10-16T15:53:28.608624+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:28.608624+0000","last_undegraded":"2023-10-16T15:53:28.608624+0000","last_fullsized":"2023-10-16T15:53:28.608624+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235542+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:28.235542+0000","last_peered":"2023-10-16T15:53:28.235542+0000","last_clean":"2023-10-16T15:53:28.235542+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:28.235542+0000","last_undegraded":"2023-10-16T15:53:28.235542+0000","last_fullsized":"2023-10-16T15:53:28.235542+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.232944+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:28.232944+0000","last_peered":"2023-10-16T15:53:28.232944+0000","last_clean":"2023-10-16T15:53:28.232944+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:28.232944+0000","last_undegraded":"2023-10-16T15:53:28.232944+0000","last_fullsized":"2023-10-16T15:53:28.232944+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235719+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:28.235719+0000","last_peered":"2023-10-16T15:53:28.235719+0000","last_clean":"2023-10-16T15:53:28.235719+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:28.235719+0000","last_undegraded":"2023-10-16T15:53:28.235719+0000","last_fullsized":"2023-10-16T15:53:28.235719+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235874+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:28.235874+0000","last_peered":"2023-10-16T15:53:28.235874+0000","last_clean":"2023-10-16T15:53:28.235874+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:28.235874+0000","last_undegraded":"2023-10-16T15:53:28.235874+0000","last_fullsized":"2023-10-16T15:53:28.235874+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608487+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:28.608487+0000","last_peered":"2023-10-16T15:53:28.608487+0000","last_clean":"2023-10-16T15:53:28.608487+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:28.608487+0000","last_undegraded":"2023-10-16T15:53:28.608487+0000","last_fullsized":"2023-10-16T15:53:28.608487+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612641+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:28.612641+0000","last_peered":"2023-10-16T15:53:28.612641+0000","last_clean":"2023-10-16T15:53:28.612641+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:28.612641+0000","last_undegraded":"2023-10-16T15:53:28.612641+0000","last_fullsized":"2023-10-16T15:53:28.612641+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612379+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:28.612379+0000","last_peered":"2023-10-16T15:53:28.612379+0000","last_clean":"2023-10-16T15:53:28.612379+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:28.612379+0000","last_undegraded":"2023-10-16T15:53:28.612379+0000","last_fullsized":"2023-10-16T15:53:28.612379+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608613+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:28.608613+0000","last_peered":"2023-10-16T15:53:28.608613+0000","last_clean":"2023-10-16T15:53:28.608613+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:28.608613+0000","last_undegraded":"2023-10-16T15:53:28.608613+0000","last_fullsized":"2023-10-16T15:53:28.608613+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611103+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:28.611103+0000","last_peered":"2023-10-16T15:53:28.611103+0000","last_clean":"2023-10-16T15:53:28.611103+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:28.611103+0000","last_undegraded":"2023-10-16T15:53:28.611103+0000","last_fullsized":"2023-10-16T15:53:28.611103+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611617+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:28.611617+0000","last_peered":"2023-10-16T15:53:28.611617+0000","last_clean":"2023-10-16T15:53:28.611617+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:28.611617+0000","last_undegraded":"2023-10-16T15:53:28.611617+0000","last_fullsized":"2023-10-16T15:53:28.611617+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608748+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:28.608748+0000","last_peered":"2023-10-16T15:53:28.608748+0000","last_clean":"2023-10-16T15:53:28.608748+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:28.608748+0000","last_undegraded":"2023-10-16T15:53:28.608748+0000","last_fullsized":"2023-10-16T15:53:28.608748+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611738+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:28.611738+0000","last_peered":"2023-10-16T15:53:28.611738+0000","last_clean":"2023-10-16T15:53:28.611738+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:28.611738+0000","last_undegraded":"2023-10-16T15:53:28.611738+0000","last_fullsized":"2023-10-16T15:53:28.611738+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234335+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:28.234335+0000","last_peered":"2023-10-16T15:53:28.234335+0000","last_clean":"2023-10-16T15:53:28.234335+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:28.234335+0000","last_undegraded":"2023-10-16T15:53:28.234335+0000","last_fullsized":"2023-10-16T15:53:28.234335+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610168+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:28.610168+0000","last_peered":"2023-10-16T15:53:28.610168+0000","last_clean":"2023-10-16T15:53:28.610168+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:28.610168+0000","last_undegraded":"2023-10-16T15:53:28.610168+0000","last_fullsized":"2023-10-16T15:53:28.610168+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406595+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:28.406595+0000","last_peered":"2023-10-16T15:53:28.406595+0000","last_clean":"2023-10-16T15:53:28.406595+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:28.406595+0000","last_undegraded":"2023-10-16T15:53:28.406595+0000","last_fullsized":"2023-10-16T15:53:28.406595+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231411+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:28.231411+0000","last_peered":"2023-10-16T15:53:28.231411+0000","last_clean":"2023-10-16T15:53:28.231411+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:28.231411+0000","last_undegraded":"2023-10-16T15:53:28.231411+0000","last_fullsized":"2023-10-16T15:53:28.231411+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610350+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:28.610350+0000","last_peered":"2023-10-16T15:53:28.610350+0000","last_clean":"2023-10-16T15:53:28.610350+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:28.610350+0000","last_undegraded":"2023-10-16T15:53:28.610350+0000","last_fullsized":"2023-10-16T15:53:28.610350+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608892+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:28.608892+0000","last_peered":"2023-10-16T15:53:28.608892+0000","last_clean":"2023-10-16T15:53:28.608892+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:28.608892+0000","last_undegraded":"2023-10-16T15:53:28.608892+0000","last_fullsized":"2023-10-16T15:53:28.608892+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612883+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:28.612883+0000","last_peered":"2023-10-16T15:53:28.612883+0000","last_clean":"2023-10-16T15:53:28.612883+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:28.612883+0000","last_undegraded":"2023-10-16T15:53:28.612883+0000","last_fullsized":"2023-10-16T15:53:28.612883+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237288+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:28.237288+0000","last_peered":"2023-10-16T15:53:28.237288+0000","last_clean":"2023-10-16T15:53:28.237288+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:28.237288+0000","last_undegraded":"2023-10-16T15:53:28.237288+0000","last_fullsized":"2023-10-16T15:53:28.237288+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235512+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:28.235512+0000","last_peered":"2023-10-16T15:53:28.235512+0000","last_clean":"2023-10-16T15:53:28.235512+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:28.235512+0000","last_undegraded":"2023-10-16T15:53:28.235512+0000","last_fullsized":"2023-10-16T15:53:28.235512+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.232785+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:28.232785+0000","last_peered":"2023-10-16T15:53:28.232785+0000","last_clean":"2023-10-16T15:53:28.232785+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:28.232785+0000","last_undegraded":"2023-10-16T15:53:28.232785+0000","last_fullsized":"2023-10-16T15:53:28.232785+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.407039+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:28.407039+0000","last_peered":"2023-10-16T15:53:28.407039+0000","last_clean":"2023-10-16T15:53:28.407039+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:28.407039+0000","last_undegraded":"2023-10-16T15:53:28.407039+0000","last_fullsized":"2023-10-16T15:53:28.407039+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608715+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:28.608715+0000","last_peered":"2023-10-16T15:53:28.608715+0000","last_clean":"2023-10-16T15:53:28.608715+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:28.608715+0000","last_undegraded":"2023-10-16T15:53:28.608715+0000","last_fullsized":"2023-10-16T15:53:28.608715+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611547+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:28.611547+0000","last_peered":"2023-10-16T15:53:28.611547+0000","last_clean":"2023-10-16T15:53:28.611547+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:28.611547+0000","last_undegraded":"2023-10-16T15:53:28.611547+0000","last_fullsized":"2023-10-16T15:53:28.611547+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237011+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:28.237011+0000","last_peered":"2023-10-16T15:53:28.237011+0000","last_clean":"2023-10-16T15:53:28.237011+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:28.237011+0000","last_undegraded":"2023-10-16T15:53:28.237011+0000","last_fullsized":"2023-10-16T15:53:28.237011+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610434+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:28.610434+0000","last_peered":"2023-10-16T15:53:28.610434+0000","last_clean":"2023-10-16T15:53:28.610434+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:28.610434+0000","last_undegraded":"2023-10-16T15:53:28.610434+0000","last_fullsized":"2023-10-16T15:53:28.610434+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231920+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:28.231920+0000","last_peered":"2023-10-16T15:53:28.231920+0000","last_clean":"2023-10-16T15:53:28.231920+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:28.231920+0000","last_undegraded":"2023-10-16T15:53:28.231920+0000","last_fullsized":"2023-10-16T15:53:28.231920+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.607957+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:28.607957+0000","last_peered":"2023-10-16T15:53:28.607957+0000","last_clean":"2023-10-16T15:53:28.607957+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:28.607957+0000","last_undegraded":"2023-10-16T15:53:28.607957+0000","last_fullsized":"2023-10-16T15:53:28.607957+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406966+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:28.406966+0000","last_peered":"2023-10-16T15:53:28.406966+0000","last_clean":"2023-10-16T15:53:28.406966+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:28.406966+0000","last_undegraded":"2023-10-16T15:53:28.406966+0000","last_fullsized":"2023-10-16T15:53:28.406966+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234909+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:28.234909+0000","last_peered":"2023-10-16T15:53:28.234909+0000","last_clean":"2023-10-16T15:53:28.234909+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:28.234909+0000","last_undegraded":"2023-10-16T15:53:28.234909+0000","last_fullsized":"2023-10-16T15:53:28.234909+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610352+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:28.610352+0000","last_peered":"2023-10-16T15:53:28.610352+0000","last_clean":"2023-10-16T15:53:28.610352+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:28.610352+0000","last_undegraded":"2023-10-16T15:53:28.610352+0000","last_fullsized":"2023-10-16T15:53:28.610352+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611133+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:28.611133+0000","last_peered":"2023-10-16T15:53:28.611133+0000","last_clean":"2023-10-16T15:53:28.611133+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:28.611133+0000","last_undegraded":"2023-10-16T15:53:28.611133+0000","last_fullsized":"2023-10-16T15:53:28.611133+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610712+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:28.610712+0000","last_peered":"2023-10-16T15:53:28.610712+0000","last_clean":"2023-10-16T15:53:28.610712+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:28.610712+0000","last_undegraded":"2023-10-16T15:53:28.610712+0000","last_fullsized":"2023-10-16T15:53:28.610712+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1247,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T15:53:27.214436+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:27.214436+0000","last_peered":"2023-10-16T15:53:27.214436+0000","last_clean":"2023-10-16T15:53:27.214436+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:27.214436+0000","last_undegraded":"2023-10-16T15:53:27.214436+0000","last_fullsized":"2023-10-16T15:53:27.214436+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60896,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310944,"statfs":{"total":96636764160,"available":96574406656,"internally_reserved":0,"allocated":3428352,"data_stored":1975077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":1.071},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":0.89000000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":0.996},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":0.94199999999999995}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.484},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":1.2210000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.379}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.319},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.163}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.45},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.839}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":1.613},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.7230000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29780,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342060,"statfs":{"total":96636764160,"available":96606269440,"internally_reserved":0,"allocated":2826240,"data_stored":1383679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.258,"15min":1.2190000000000001},"min":{"1min":0.79100000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.99,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.2869999999999999,"15min":1.262},"min":{"1min":0.91500000000000004,"5min":0.78300000000000003,"15min":0.60699999999999998},"max":{"1min":1.722,"5min":1.847,"15min":1.847},"last":1.268}]},{"osd":1,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2809999999999999,"15min":1.2829999999999999},"min":{"1min":0.96799999999999997,"5min":0.79800000000000004,"15min":0.72199999999999998},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.3460000000000001,"15min":1.331},"min":{"1min":1.0640000000000001,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.8580000000000001,"5min":1.8740000000000001,"15min":1.9419999999999999},"last":1.4330000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.256,"15min":1.248},"min":{"1min":0.96299999999999997,"5min":0.85699999999999998,"15min":0.71899999999999997},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.429,"15min":1.3939999999999999},"min":{"1min":1.052,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.468}]},{"osd":3,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.381,"15min":1.399},"min":{"1min":1.048,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.339,"15min":1.3420000000000001},"min":{"1min":1.0589999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.5680000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.631}]},{"osd":4,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3380000000000001,"15min":1.327},"min":{"1min":0.93300000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.698,"5min":1.917,"15min":2.028},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.391,"5min":1.4119999999999999,"15min":1.407},"min":{"1min":1.0349999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7450000000000001,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.1299999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3600000000000001,"15min":1.347},"min":{"1min":0.98599999999999999,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.6970000000000001,"5min":1.9419999999999999,"15min":2.016},"last":1.327},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.556,"15min":1.526},"min":{"1min":1.1499999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.964,"5min":2.169,"15min":2.169},"last":1.0469999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.609,"15min":1.5940000000000001},"min":{"1min":1.3100000000000001,"5min":0.627,"15min":0.627},"max":{"1min":1.9279999999999999,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.536},{"interface":"front","average":{"1min":1.625,"5min":1.6020000000000001,"15min":1.5900000000000001},"min":{"1min":1.383,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.843,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.575}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29788,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342052,"statfs":{"total":96636764160,"available":96606261248,"internally_reserved":0,"allocated":2826240,"data_stored":1383679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.80200000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.821},"last":1.006},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3,"15min":1.2470000000000001},"min":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.639,"5min":2.738,"15min":2.738},"last":0.95299999999999996}]},{"osd":1,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.232,"15min":1.236},"min":{"1min":0.91300000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7569999999999999,"5min":1.8140000000000001,"15min":1.8680000000000001},"last":1.075},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.284,"15min":1.3109999999999999},"min":{"1min":0.89300000000000002,"5min":0.87,"15min":0.87},"max":{"1min":1.675,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.514}]},{"osd":2,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2450000000000001,"15min":1.2629999999999999},"min":{"1min":0.85199999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.784,"5min":1.784,"15min":1.9219999999999999},"last":1.329},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.6990000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.667}]},{"osd":3,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3660000000000001,"15min":1.347},"min":{"1min":0.93100000000000005,"5min":0.86199999999999999,"15min":0.80000000000000004},"max":{"1min":1.7350000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.383,"15min":1.3939999999999999},"min":{"1min":1.103,"5min":0.752,"15min":0.752},"max":{"1min":1.7250000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.4399999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3759999999999999,"15min":1.3460000000000001},"min":{"1min":0.92300000000000004,"5min":0.85699999999999998,"15min":0.78100000000000003},"max":{"1min":1.8160000000000001,"5min":1.972,"15min":1.9730000000000001},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5389999999999999,"15min":1.532},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.77000000000000002},"max":{"1min":1.869,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.724}]},{"osd":6,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4359999999999999,"15min":1.385},"min":{"1min":1.04,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.768,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.458,"15min":1.4390000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.70599999999999996},"max":{"1min":1.7390000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":2.0230000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.556,"15min":1.544},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.923,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.847},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6499999999999999,"15min":1.643},"min":{"1min":1.1719999999999999,"5min":1.032,"15min":0.70499999999999996},"max":{"1min":1.845,"5min":3.109,"15min":3.109},"last":1.962}]}]},{"osd":4,"up_from":13,"seq":55834576003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52004,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319836,"statfs":{"total":96636764160,"available":96583512064,"internally_reserved":0,"allocated":2842624,"data_stored":1384813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2330000000000001,"15min":1.212},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.52700000000000002},"max":{"1min":1.794,"5min":1.883,"15min":1.883},"last":0.874},{"interface":"front","average":{"1min":1.302,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.79700000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.95799999999999996,"5min":0.83499999999999996,"15min":0.629},"max":{"1min":1.675,"5min":1.929,"15min":2.2160000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.3160000000000001},"min":{"1min":0.92000000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0179999999999998,"5min":2.157,"15min":2.2989999999999999},"last":1.448}]},{"osd":2,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4490000000000001,"15min":1.4670000000000001},"min":{"1min":1.1160000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.498,"15min":1.4970000000000001},"min":{"1min":1.1870000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.222}]},{"osd":3,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":1.081,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.23,"5min":2.23,"15min":2.3650000000000002},"last":1.117},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4099999999999999,"15min":1.401},"min":{"1min":1.0069999999999999,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.1479999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.375,"15min":1.3420000000000001},"min":{"1min":1.03,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.911,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.46,"15min":1.4370000000000001},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.161,"5min":2.161,"15min":2.4489999999999998},"last":1.7849999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":0.97499999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.554,"15min":1.5269999999999999},"min":{"1min":1.0089999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.2789999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":2.0049999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.617,"15min":1.5860000000000001},"min":{"1min":1.2669999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.946},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.6339999999999999,"15min":1.6040000000000001},"min":{"1min":1.3999999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.242,"5min":2.2599999999999998,"15min":2.347},"last":1.8380000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56156,"kb_used_data":2768,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315684,"statfs":{"total":96636764160,"available":96579260416,"internally_reserved":0,"allocated":2834432,"data_stored":1383279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.876}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.96799999999999997}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":1.621},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":1.732}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":1.212}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.3460000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.5409999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":1.47},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.647}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51984,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319856,"statfs":{"total":96636764160,"available":96583532544,"internally_reserved":0,"allocated":2822144,"data_stored":1383097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.94299999999999995,"15min":0.96499999999999997},"min":{"1min":0.68300000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.5740000000000001,"5min":1.726,"15min":1.8340000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.107},"min":{"1min":0.54200000000000004,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.8999999999999999},"last":1.228}]},{"osd":1,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0669999999999999,"15min":1.0680000000000001},"min":{"1min":0.622,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7410000000000001,"5min":1.974,"15min":1.974},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.2050000000000001,"15min":1.248},"min":{"1min":0.76600000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7090000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.579}]},{"osd":3,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.284,"15min":1.3109999999999999},"min":{"1min":0.69199999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.704,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.3560000000000001,"15min":1.3839999999999999},"min":{"1min":0.89300000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9570000000000001,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.6499999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.4910000000000001},"min":{"1min":1.147,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0800000000000001,"5min":2.335,"15min":2.335},"last":1.381},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3899999999999999,"15min":1.429},"min":{"1min":0.92300000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7549999999999999,"5min":2.0920000000000001,"15min":2.1589999999999998},"last":1.5289999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.486,"15min":1.5089999999999999},"min":{"1min":1.2390000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8620000000000001,"5min":2.657,"15min":2.657},"last":1.056},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.6060000000000001,"15min":1.631},"min":{"1min":1.0580000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2389999999999999,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.516,"15min":1.5720000000000001},"min":{"1min":1.087,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9019999999999999,"5min":1.9850000000000001,"15min":2.3849999999999998},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.629,"15min":1.6399999999999999},"min":{"1min":1.28,"5min":0.753,"15min":0.753},"max":{"1min":1.992,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.198}]},{"osd":7,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.28,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.0449999999999999,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.893},{"interface":"front","average":{"1min":1.756,"5min":1.7110000000000001,"15min":1.7529999999999999},"min":{"1min":1.526,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.113,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.3400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":1383153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.206}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.3660000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.8859999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":2.2959999999999998},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.9810000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":2.0539999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.7549999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":2.1739999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":2.5059999999999998}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":2.621},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":2.452}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48408,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323432,"statfs":{"total":96636764160,"available":96587194368,"internally_reserved":0,"allocated":3420160,"data_stored":1973555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.31},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":0.749},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":0.68400000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:31.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.018+0000 7f7fa9eea640 1 Processor -- start 2023-10-16T15:53:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.018+0000 7f7fa9eea640 1 -- start start 2023-10-16T15:53:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.020+0000 7f7fa9eea640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7fa406c040 0x7f7fa406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.020+0000 7f7fa9eea640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa415f4a0 con 0x7f7fa40ad970 2023-10-16T15:53:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.020+0000 7f7fa9eea640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa406c960 con 0x7f7fa40cda80 2023-10-16T15:53:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.020+0000 7f7fa9eea640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa415b9d0 con 0x7f7fa406c040 2023-10-16T15:53:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.020+0000 7f7fa3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7fa40ad970 0x7f7fa415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57774/0 (socket says 172.21.15.29:57774) 2023-10-16T15:53:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.020+0000 7f7fa3fff640 1 -- 172.21.15.29:0/4157801898 learned_addr learned my addr 172.21.15.29:0/4157801898 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa8ee8640 1 --2- 172.21.15.29:0/4157801898 >> v2:172.21.15.170:3300/0 conn(0x7f7fa406c040 0x7f7fa406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540738648 0 0) 0x7f7fa415f4a0 con 0x7f7fa40ad970 2023-10-16T15:53:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f94003520 con 0x7f7fa40ad970 2023-10-16T15:53:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 39484970 0 0) 0x7f7f94003520 con 0x7f7fa40ad970 2023-10-16T15:53:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 >> v2:172.21.15.170:3300/0 conn(0x7f7fa406c040 msgr2=0x7f7fa406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 --2- 172.21.15.29:0/4157801898 >> v2:172.21.15.170:3300/0 conn(0x7f7fa406c040 0x7f7fa406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 >> v1:172.21.15.170:6789/0 conn(0x7f7fa40cda80 legacy=0x7f7fa415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa415bcb0 con 0x7f7fa40ad970 2023-10-16T15:53:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7f90001670 con 0x7f7fa40ad970 2023-10-16T15:53:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 >> v1:172.21.15.29:6789/0 conn(0x7f7fa40ad970 legacy=0x7f7fa415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.021+0000 7f7fa37fe640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f94003970 0x7f7f94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa37fe640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f940047c0 0x7f7f94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa37fe640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f94005180 0x7f7f9400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa415b9d0 con 0x7f7f940047c0 2023-10-16T15:53:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9400c020 con 0x7f7f94005180 2023-10-16T15:53:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9400c1e0 con 0x7f7f94003970 2023-10-16T15:53:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f940047c0 0x7f7f94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa8ee8640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f94003970 0x7f7f94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa96e9640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f94005180 0x7f7f9400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa3fff640 1 -- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f94003970 msgr2=0x7f7f94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f94003970 0x7f7f94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa3fff640 1 -- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f94005180 msgr2=0x7f7f9400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f94005180 0x7f7f9400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa3fff640 1 -- 172.21.15.29:0/4157801898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f94003520 con 0x7f7f940047c0 2023-10-16T15:53:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa96e9640 1 -- 172.21.15.29:0/4157801898 reap_dead start 2023-10-16T15:53:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.022+0000 7f7fa8ee8640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f94003970 0x7f7f94003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:32.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f940047c0 0x7f7f94004c40 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c210 tx=0x7f7f940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd55a373cd690f0e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:32.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90005ad0 con 0x7f7f940047c0 2023-10-16T15:53:32.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f90005c30 con 0x7f7f940047c0 2023-10-16T15:53:32.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa37fe640 1 -- 172.21.15.29:0/4157801898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9002cad0 con 0x7f7f940047c0 2023-10-16T15:53:32.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa9eea640 1 -- 172.21.15.29:0/4157801898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f940047c0 msgr2=0x7f7f94004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f940047c0 0x7f7f94004c40 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c210 tx=0x7f7f940057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa9eea640 1 -- 172.21.15.29:0/4157801898 shutdown_connections 2023-10-16T15:53:32.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/4157801898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f940047c0 0x7f7f94004c40 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.024+0000 7f7fa9eea640 1 -- 172.21.15.29:0/4157801898 >> 172.21.15.29:0/4157801898 conn(0x7f7fa40ce6a0 msgr2=0x7f7fa40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:32.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.025+0000 7f7fa9eea640 1 -- 172.21.15.29:0/4157801898 shutdown_connections 2023-10-16T15:53:32.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.025+0000 7f7fa9eea640 1 -- 172.21.15.29:0/4157801898 wait complete. 2023-10-16T15:53:32.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.026+0000 7f7fa9eea640 1 Processor -- start 2023-10-16T15:53:32.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.027+0000 7f7fa9eea640 1 -- start start 2023-10-16T15:53:32.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.027+0000 7f7fa9eea640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f940047c0 0x7f7fa40ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa9eea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 0x7f7fa414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa9eea640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fa414f970 0x7f7fa41f9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa9eea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa41f7920 con 0x7f7fa414dc30 2023-10-16T15:53:32.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa9eea640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa40c6bd0 con 0x7f7fa414f970 2023-10-16T15:53:32.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa9eea640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa414fdf0 con 0x7f7f940047c0 2023-10-16T15:53:32.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 0x7f7fa414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 0x7f7fa414f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45506/0 (socket says 172.21.15.29:45506) 2023-10-16T15:53:32.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 -- 172.21.15.29:0/3209617714 learned_addr learned my addr 172.21.15.29:0/3209617714 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:32.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa8ee8640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f940047c0 0x7f7fa40ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 -- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f940047c0 msgr2=0x7f7fa40ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f940047c0 0x7f7fa40ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 -- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fa414f970 msgr2=0x7f7fa41f9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa96e9640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fa414f970 0x7f7fa41f9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fa414f970 0x7f7fa41f9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.028+0000 7f7fa3fff640 1 -- 172.21.15.29:0/3209617714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa41f6a00 con 0x7f7fa414dc30 2023-10-16T15:53:32.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.029+0000 7f7fa3fff640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 0x7f7fa414f430 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c210 tx=0x7f7f9002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:32.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.029+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9003c040 con 0x7f7fa414dc30 2023-10-16T15:53:32.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.029+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9002cd60 con 0x7f7fa414dc30 2023-10-16T15:53:32.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.029+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa41f6ca0 con 0x7f7fa414dc30 2023-10-16T15:53:32.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.029+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90007290 con 0x7f7fa414dc30 2023-10-16T15:53:32.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.029+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa41f7260 con 0x7f7fa414dc30 2023-10-16T15:53:32.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.030+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa406b330 con 0x7f7fa414dc30 2023-10-16T15:53:32.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.031+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f90032070 con 0x7f7fa414dc30 2023-10-16T15:53:32.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.032+0000 7f7fa17fa640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7f8806afc0 0x7f7f8806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.032+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7f900aad00 con 0x7f7fa414dc30 2023-10-16T15:53:32.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.033+0000 7f7fa8ee8640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7f8806afc0 0x7f7f8806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.034+0000 7f7fa8ee8640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7f8806afc0 0x7f7f8806d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4150110 tx=0x7f7f98008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:32.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.036+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f90075da0 con 0x7f7fa414dc30 2023-10-16T15:53:32.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.226+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fa40cc270 con 0x7f7f8806afc0 2023-10-16T15:53:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.249+0000 7f7fa17fa640 1 -- 172.21.15.29:0/3209617714 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287017 (secure 0 0 0) 0x7f7f98009640 con 0x7f7f8806afc0 2023-10-16T15:53:32.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.255+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7f8806afc0 msgr2=0x7f7f8806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.255+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7f8806afc0 0x7f7f8806d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4150110 tx=0x7f7f98008040 comp rx=0 tx=0).stop 2023-10-16T15:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.255+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 msgr2=0x7f7fa414f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.255+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 0x7f7fa414f430 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c210 tx=0x7f7f9002e070 comp rx=0 tx=0).stop 2023-10-16T15:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 shutdown_connections 2023-10-16T15:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7f8806afc0 0x7f7f8806d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa414dc30 0x7f7fa414f430 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7fa414f970 0x7f7fa41f9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 --2- 172.21.15.29:0/3209617714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f940047c0 0x7f7fa40ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 >> 172.21.15.29:0/3209617714 conn(0x7f7fa40ce6a0 msgr2=0x7f7fa40cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 shutdown_connections 2023-10-16T15:53:32.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.256+0000 7f7fa9eea640 1 -- 172.21.15.29:0/3209617714 wait complete. 2023-10-16T15:53:32.279 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T15:53:30.457614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":627,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":376852,"kb_used_data":23272,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754597868,"statfs":{"total":773094113280,"available":772708216832,"internally_reserved":0,"allocated":23830528,"data_stored":12250332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":361942585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.292337"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612164+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:28.612164+0000","last_peered":"2023-10-16T15:53:28.612164+0000","last_clean":"2023-10-16T15:53:28.612164+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:28.612164+0000","last_undegraded":"2023-10-16T15:53:28.612164+0000","last_fullsized":"2023-10-16T15:53:28.612164+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235156+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:28.235156+0000","last_peered":"2023-10-16T15:53:28.235156+0000","last_clean":"2023-10-16T15:53:28.235156+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:28.235156+0000","last_undegraded":"2023-10-16T15:53:28.235156+0000","last_fullsized":"2023-10-16T15:53:28.235156+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609117+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:28.609117+0000","last_peered":"2023-10-16T15:53:28.609117+0000","last_clean":"2023-10-16T15:53:28.609117+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:28.609117+0000","last_undegraded":"2023-10-16T15:53:28.609117+0000","last_fullsized":"2023-10-16T15:53:28.609117+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236949+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:28.236949+0000","last_peered":"2023-10-16T15:53:28.236949+0000","last_clean":"2023-10-16T15:53:28.236949+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:28.236949+0000","last_undegraded":"2023-10-16T15:53:28.236949+0000","last_fullsized":"2023-10-16T15:53:28.236949+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.233209+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:28.233209+0000","last_peered":"2023-10-16T15:53:28.233209+0000","last_clean":"2023-10-16T15:53:28.233209+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:28.233209+0000","last_undegraded":"2023-10-16T15:53:28.233209+0000","last_fullsized":"2023-10-16T15:53:28.233209+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231690+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:28.231690+0000","last_peered":"2023-10-16T15:53:28.231690+0000","last_clean":"2023-10-16T15:53:28.231690+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:28.231690+0000","last_undegraded":"2023-10-16T15:53:28.231690+0000","last_fullsized":"2023-10-16T15:53:28.231690+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609729+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:28.609729+0000","last_peered":"2023-10-16T15:53:28.609729+0000","last_clean":"2023-10-16T15:53:28.609729+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:28.609729+0000","last_undegraded":"2023-10-16T15:53:28.609729+0000","last_fullsized":"2023-10-16T15:53:28.609729+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231173+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:28.231173+0000","last_peered":"2023-10-16T15:53:28.231173+0000","last_clean":"2023-10-16T15:53:28.231173+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:28.231173+0000","last_undegraded":"2023-10-16T15:53:28.231173+0000","last_fullsized":"2023-10-16T15:53:28.231173+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235995+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:28.235995+0000","last_peered":"2023-10-16T15:53:28.235995+0000","last_clean":"2023-10-16T15:53:28.235995+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:28.235995+0000","last_undegraded":"2023-10-16T15:53:28.235995+0000","last_fullsized":"2023-10-16T15:53:28.235995+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611854+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:28.611854+0000","last_peered":"2023-10-16T15:53:28.611854+0000","last_clean":"2023-10-16T15:53:28.611854+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:28.611854+0000","last_undegraded":"2023-10-16T15:53:28.611854+0000","last_fullsized":"2023-10-16T15:53:28.611854+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609394+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:28.609394+0000","last_peered":"2023-10-16T15:53:28.609394+0000","last_clean":"2023-10-16T15:53:28.609394+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:28.609394+0000","last_undegraded":"2023-10-16T15:53:28.609394+0000","last_fullsized":"2023-10-16T15:53:28.609394+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610529+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:28.610529+0000","last_peered":"2023-10-16T15:53:28.610529+0000","last_clean":"2023-10-16T15:53:28.610529+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:28.610529+0000","last_undegraded":"2023-10-16T15:53:28.610529+0000","last_fullsized":"2023-10-16T15:53:28.610529+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610794+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:28.610794+0000","last_peered":"2023-10-16T15:53:28.610794+0000","last_clean":"2023-10-16T15:53:28.610794+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:28.610794+0000","last_undegraded":"2023-10-16T15:53:28.610794+0000","last_fullsized":"2023-10-16T15:53:28.610794+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610587+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:28.610587+0000","last_peered":"2023-10-16T15:53:28.610587+0000","last_clean":"2023-10-16T15:53:28.610587+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:28.610587+0000","last_undegraded":"2023-10-16T15:53:28.610587+0000","last_fullsized":"2023-10-16T15:53:28.610587+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609875+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:28.609875+0000","last_peered":"2023-10-16T15:53:28.609875+0000","last_clean":"2023-10-16T15:53:28.609875+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:28.609875+0000","last_undegraded":"2023-10-16T15:53:28.609875+0000","last_fullsized":"2023-10-16T15:53:28.609875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610601+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:28.610601+0000","last_peered":"2023-10-16T15:53:28.610601+0000","last_clean":"2023-10-16T15:53:28.610601+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:28.610601+0000","last_undegraded":"2023-10-16T15:53:28.610601+0000","last_fullsized":"2023-10-16T15:53:28.610601+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612493+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:28.612493+0000","last_peered":"2023-10-16T15:53:28.612493+0000","last_clean":"2023-10-16T15:53:28.612493+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:28.612493+0000","last_undegraded":"2023-10-16T15:53:28.612493+0000","last_fullsized":"2023-10-16T15:53:28.612493+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611993+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:28.611993+0000","last_peered":"2023-10-16T15:53:28.611993+0000","last_clean":"2023-10-16T15:53:28.611993+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:28.611993+0000","last_undegraded":"2023-10-16T15:53:28.611993+0000","last_fullsized":"2023-10-16T15:53:28.611993+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612888+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:28.612888+0000","last_peered":"2023-10-16T15:53:28.612888+0000","last_clean":"2023-10-16T15:53:28.612888+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:28.612888+0000","last_undegraded":"2023-10-16T15:53:28.612888+0000","last_fullsized":"2023-10-16T15:53:28.612888+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.233138+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:28.233138+0000","last_peered":"2023-10-16T15:53:28.233138+0000","last_clean":"2023-10-16T15:53:28.233138+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:28.233138+0000","last_undegraded":"2023-10-16T15:53:28.233138+0000","last_fullsized":"2023-10-16T15:53:28.233138+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236848+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:28.236848+0000","last_peered":"2023-10-16T15:53:28.236848+0000","last_clean":"2023-10-16T15:53:28.236848+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:28.236848+0000","last_undegraded":"2023-10-16T15:53:28.236848+0000","last_fullsized":"2023-10-16T15:53:28.236848+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406627+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:28.406627+0000","last_peered":"2023-10-16T15:53:28.406627+0000","last_clean":"2023-10-16T15:53:28.406627+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:28.406627+0000","last_undegraded":"2023-10-16T15:53:28.406627+0000","last_fullsized":"2023-10-16T15:53:28.406627+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406745+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:28.406745+0000","last_peered":"2023-10-16T15:53:28.406745+0000","last_clean":"2023-10-16T15:53:28.406745+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:28.406745+0000","last_undegraded":"2023-10-16T15:53:28.406745+0000","last_fullsized":"2023-10-16T15:53:28.406745+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234887+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:28.234887+0000","last_peered":"2023-10-16T15:53:28.234887+0000","last_clean":"2023-10-16T15:53:28.234887+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:28.234887+0000","last_undegraded":"2023-10-16T15:53:28.234887+0000","last_fullsized":"2023-10-16T15:53:28.234887+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235499+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:28.235499+0000","last_peered":"2023-10-16T15:53:28.235499+0000","last_clean":"2023-10-16T15:53:28.235499+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:28.235499+0000","last_undegraded":"2023-10-16T15:53:28.235499+0000","last_fullsized":"2023-10-16T15:53:28.235499+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.613187+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:28.613187+0000","last_peered":"2023-10-16T15:53:28.613187+0000","last_clean":"2023-10-16T15:53:28.613187+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:28.613187+0000","last_undegraded":"2023-10-16T15:53:28.613187+0000","last_fullsized":"2023-10-16T15:53:28.613187+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610848+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:28.610848+0000","last_peered":"2023-10-16T15:53:28.610848+0000","last_clean":"2023-10-16T15:53:28.610848+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:28.610848+0000","last_undegraded":"2023-10-16T15:53:28.610848+0000","last_fullsized":"2023-10-16T15:53:28.610848+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608340+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:28.608340+0000","last_peered":"2023-10-16T15:53:28.608340+0000","last_clean":"2023-10-16T15:53:28.608340+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:28.608340+0000","last_undegraded":"2023-10-16T15:53:28.608340+0000","last_fullsized":"2023-10-16T15:53:28.608340+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610516+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:28.610516+0000","last_peered":"2023-10-16T15:53:28.610516+0000","last_clean":"2023-10-16T15:53:28.610516+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:28.610516+0000","last_undegraded":"2023-10-16T15:53:28.610516+0000","last_fullsized":"2023-10-16T15:53:28.610516+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234710+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:28.234710+0000","last_peered":"2023-10-16T15:53:28.234710+0000","last_clean":"2023-10-16T15:53:28.234710+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:28.234710+0000","last_undegraded":"2023-10-16T15:53:28.234710+0000","last_fullsized":"2023-10-16T15:53:28.234710+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610198+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:28.610198+0000","last_peered":"2023-10-16T15:53:28.610198+0000","last_clean":"2023-10-16T15:53:28.610198+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:28.610198+0000","last_undegraded":"2023-10-16T15:53:28.610198+0000","last_fullsized":"2023-10-16T15:53:28.610198+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609901+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:28.609901+0000","last_peered":"2023-10-16T15:53:28.609901+0000","last_clean":"2023-10-16T15:53:28.609901+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:28.609901+0000","last_undegraded":"2023-10-16T15:53:28.609901+0000","last_fullsized":"2023-10-16T15:53:28.609901+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235459+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:28.235459+0000","last_peered":"2023-10-16T15:53:28.235459+0000","last_clean":"2023-10-16T15:53:28.235459+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:28.235459+0000","last_undegraded":"2023-10-16T15:53:28.235459+0000","last_fullsized":"2023-10-16T15:53:28.235459+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611017+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:28.611017+0000","last_peered":"2023-10-16T15:53:28.611017+0000","last_clean":"2023-10-16T15:53:28.611017+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:28.611017+0000","last_undegraded":"2023-10-16T15:53:28.611017+0000","last_fullsized":"2023-10-16T15:53:28.611017+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.407107+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:28.407107+0000","last_peered":"2023-10-16T15:53:28.407107+0000","last_clean":"2023-10-16T15:53:28.407107+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:28.407107+0000","last_undegraded":"2023-10-16T15:53:28.407107+0000","last_fullsized":"2023-10-16T15:53:28.407107+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235727+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:28.235727+0000","last_peered":"2023-10-16T15:53:28.235727+0000","last_clean":"2023-10-16T15:53:28.235727+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:28.235727+0000","last_undegraded":"2023-10-16T15:53:28.235727+0000","last_fullsized":"2023-10-16T15:53:28.235727+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611077+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:28.611077+0000","last_peered":"2023-10-16T15:53:28.611077+0000","last_clean":"2023-10-16T15:53:28.611077+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:28.611077+0000","last_undegraded":"2023-10-16T15:53:28.611077+0000","last_fullsized":"2023-10-16T15:53:28.611077+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.609358+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:28.609358+0000","last_peered":"2023-10-16T15:53:28.609358+0000","last_clean":"2023-10-16T15:53:28.609358+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:28.609358+0000","last_undegraded":"2023-10-16T15:53:28.609358+0000","last_fullsized":"2023-10-16T15:53:28.609358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612023+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:28.612023+0000","last_peered":"2023-10-16T15:53:28.612023+0000","last_clean":"2023-10-16T15:53:28.612023+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:28.612023+0000","last_undegraded":"2023-10-16T15:53:28.612023+0000","last_fullsized":"2023-10-16T15:53:28.612023+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237011+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:28.237011+0000","last_peered":"2023-10-16T15:53:28.237011+0000","last_clean":"2023-10-16T15:53:28.237011+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:28.237011+0000","last_undegraded":"2023-10-16T15:53:28.237011+0000","last_fullsized":"2023-10-16T15:53:28.237011+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236777+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:28.236777+0000","last_peered":"2023-10-16T15:53:28.236777+0000","last_clean":"2023-10-16T15:53:28.236777+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:28.236777+0000","last_undegraded":"2023-10-16T15:53:28.236777+0000","last_fullsized":"2023-10-16T15:53:28.236777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231877+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:28.231877+0000","last_peered":"2023-10-16T15:53:28.231877+0000","last_clean":"2023-10-16T15:53:28.231877+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:28.231877+0000","last_undegraded":"2023-10-16T15:53:28.231877+0000","last_fullsized":"2023-10-16T15:53:28.231877+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231658+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:28.231658+0000","last_peered":"2023-10-16T15:53:28.231658+0000","last_clean":"2023-10-16T15:53:28.231658+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:28.231658+0000","last_undegraded":"2023-10-16T15:53:28.231658+0000","last_fullsized":"2023-10-16T15:53:28.231658+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.230704+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:28.230704+0000","last_peered":"2023-10-16T15:53:28.230704+0000","last_clean":"2023-10-16T15:53:28.230704+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:28.230704+0000","last_undegraded":"2023-10-16T15:53:28.230704+0000","last_fullsized":"2023-10-16T15:53:28.230704+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236361+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:28.236361+0000","last_peered":"2023-10-16T15:53:28.236361+0000","last_clean":"2023-10-16T15:53:28.236361+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:28.236361+0000","last_undegraded":"2023-10-16T15:53:28.236361+0000","last_fullsized":"2023-10-16T15:53:28.236361+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406045+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:28.406045+0000","last_peered":"2023-10-16T15:53:28.406045+0000","last_clean":"2023-10-16T15:53:28.406045+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:28.406045+0000","last_undegraded":"2023-10-16T15:53:28.406045+0000","last_fullsized":"2023-10-16T15:53:28.406045+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237377+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:28.237377+0000","last_peered":"2023-10-16T15:53:28.237377+0000","last_clean":"2023-10-16T15:53:28.237377+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:28.237377+0000","last_undegraded":"2023-10-16T15:53:28.237377+0000","last_fullsized":"2023-10-16T15:53:28.237377+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.232932+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:28.232932+0000","last_peered":"2023-10-16T15:53:28.232932+0000","last_clean":"2023-10-16T15:53:28.232932+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:28.232932+0000","last_undegraded":"2023-10-16T15:53:28.232932+0000","last_fullsized":"2023-10-16T15:53:28.232932+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608618+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:28.608618+0000","last_peered":"2023-10-16T15:53:28.608618+0000","last_clean":"2023-10-16T15:53:28.608618+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:28.608618+0000","last_undegraded":"2023-10-16T15:53:28.608618+0000","last_fullsized":"2023-10-16T15:53:28.608618+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406908+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:28.406908+0000","last_peered":"2023-10-16T15:53:28.406908+0000","last_clean":"2023-10-16T15:53:28.406908+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:28.406908+0000","last_undegraded":"2023-10-16T15:53:28.406908+0000","last_fullsized":"2023-10-16T15:53:28.406908+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.236756+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:28.236756+0000","last_peered":"2023-10-16T15:53:28.236756+0000","last_clean":"2023-10-16T15:53:28.236756+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:28.236756+0000","last_undegraded":"2023-10-16T15:53:28.236756+0000","last_fullsized":"2023-10-16T15:53:28.236756+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612899+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:28.612899+0000","last_peered":"2023-10-16T15:53:28.612899+0000","last_clean":"2023-10-16T15:53:28.612899+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:28.612899+0000","last_undegraded":"2023-10-16T15:53:28.612899+0000","last_fullsized":"2023-10-16T15:53:28.612899+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611488+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:28.611488+0000","last_peered":"2023-10-16T15:53:28.611488+0000","last_clean":"2023-10-16T15:53:28.611488+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:28.611488+0000","last_undegraded":"2023-10-16T15:53:28.611488+0000","last_fullsized":"2023-10-16T15:53:28.611488+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"224'6","reported_seq":18,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237201+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:28.237201+0000","last_peered":"2023-10-16T15:53:28.237201+0000","last_clean":"2023-10-16T15:53:28.237201+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:28.237201+0000","last_undegraded":"2023-10-16T15:53:28.237201+0000","last_fullsized":"2023-10-16T15:53:28.237201+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231602+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:28.231602+0000","last_peered":"2023-10-16T15:53:28.231602+0000","last_clean":"2023-10-16T15:53:28.231602+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:28.231602+0000","last_undegraded":"2023-10-16T15:53:28.231602+0000","last_fullsized":"2023-10-16T15:53:28.231602+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.407124+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:28.407124+0000","last_peered":"2023-10-16T15:53:28.407124+0000","last_clean":"2023-10-16T15:53:28.407124+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:28.407124+0000","last_undegraded":"2023-10-16T15:53:28.407124+0000","last_fullsized":"2023-10-16T15:53:28.407124+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611510+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:28.611510+0000","last_peered":"2023-10-16T15:53:28.611510+0000","last_clean":"2023-10-16T15:53:28.611510+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:28.611510+0000","last_undegraded":"2023-10-16T15:53:28.611510+0000","last_fullsized":"2023-10-16T15:53:28.611510+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406837+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:28.406837+0000","last_peered":"2023-10-16T15:53:28.406837+0000","last_clean":"2023-10-16T15:53:28.406837+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:28.406837+0000","last_undegraded":"2023-10-16T15:53:28.406837+0000","last_fullsized":"2023-10-16T15:53:28.406837+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608102+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:28.608102+0000","last_peered":"2023-10-16T15:53:28.608102+0000","last_clean":"2023-10-16T15:53:28.608102+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:28.608102+0000","last_undegraded":"2023-10-16T15:53:28.608102+0000","last_fullsized":"2023-10-16T15:53:28.608102+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610295+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:28.610295+0000","last_peered":"2023-10-16T15:53:28.610295+0000","last_clean":"2023-10-16T15:53:28.610295+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:28.610295+0000","last_undegraded":"2023-10-16T15:53:28.610295+0000","last_fullsized":"2023-10-16T15:53:28.610295+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610016+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:28.610016+0000","last_peered":"2023-10-16T15:53:28.610016+0000","last_clean":"2023-10-16T15:53:28.610016+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:28.610016+0000","last_undegraded":"2023-10-16T15:53:28.610016+0000","last_fullsized":"2023-10-16T15:53:28.610016+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":20,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608624+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:28.608624+0000","last_peered":"2023-10-16T15:53:28.608624+0000","last_clean":"2023-10-16T15:53:28.608624+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:28.608624+0000","last_undegraded":"2023-10-16T15:53:28.608624+0000","last_fullsized":"2023-10-16T15:53:28.608624+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235542+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:28.235542+0000","last_peered":"2023-10-16T15:53:28.235542+0000","last_clean":"2023-10-16T15:53:28.235542+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:28.235542+0000","last_undegraded":"2023-10-16T15:53:28.235542+0000","last_fullsized":"2023-10-16T15:53:28.235542+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.232944+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:28.232944+0000","last_peered":"2023-10-16T15:53:28.232944+0000","last_clean":"2023-10-16T15:53:28.232944+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:28.232944+0000","last_undegraded":"2023-10-16T15:53:28.232944+0000","last_fullsized":"2023-10-16T15:53:28.232944+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235719+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:28.235719+0000","last_peered":"2023-10-16T15:53:28.235719+0000","last_clean":"2023-10-16T15:53:28.235719+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:28.235719+0000","last_undegraded":"2023-10-16T15:53:28.235719+0000","last_fullsized":"2023-10-16T15:53:28.235719+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235874+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:28.235874+0000","last_peered":"2023-10-16T15:53:28.235874+0000","last_clean":"2023-10-16T15:53:28.235874+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:28.235874+0000","last_undegraded":"2023-10-16T15:53:28.235874+0000","last_fullsized":"2023-10-16T15:53:28.235874+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608487+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:28.608487+0000","last_peered":"2023-10-16T15:53:28.608487+0000","last_clean":"2023-10-16T15:53:28.608487+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:28.608487+0000","last_undegraded":"2023-10-16T15:53:28.608487+0000","last_fullsized":"2023-10-16T15:53:28.608487+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612641+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:28.612641+0000","last_peered":"2023-10-16T15:53:28.612641+0000","last_clean":"2023-10-16T15:53:28.612641+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:28.612641+0000","last_undegraded":"2023-10-16T15:53:28.612641+0000","last_fullsized":"2023-10-16T15:53:28.612641+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612379+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:28.612379+0000","last_peered":"2023-10-16T15:53:28.612379+0000","last_clean":"2023-10-16T15:53:28.612379+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:28.612379+0000","last_undegraded":"2023-10-16T15:53:28.612379+0000","last_fullsized":"2023-10-16T15:53:28.612379+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608613+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:28.608613+0000","last_peered":"2023-10-16T15:53:28.608613+0000","last_clean":"2023-10-16T15:53:28.608613+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:28.608613+0000","last_undegraded":"2023-10-16T15:53:28.608613+0000","last_fullsized":"2023-10-16T15:53:28.608613+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611103+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:28.611103+0000","last_peered":"2023-10-16T15:53:28.611103+0000","last_clean":"2023-10-16T15:53:28.611103+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:28.611103+0000","last_undegraded":"2023-10-16T15:53:28.611103+0000","last_fullsized":"2023-10-16T15:53:28.611103+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611617+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:28.611617+0000","last_peered":"2023-10-16T15:53:28.611617+0000","last_clean":"2023-10-16T15:53:28.611617+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:28.611617+0000","last_undegraded":"2023-10-16T15:53:28.611617+0000","last_fullsized":"2023-10-16T15:53:28.611617+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608748+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:28.608748+0000","last_peered":"2023-10-16T15:53:28.608748+0000","last_clean":"2023-10-16T15:53:28.608748+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:28.608748+0000","last_undegraded":"2023-10-16T15:53:28.608748+0000","last_fullsized":"2023-10-16T15:53:28.608748+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611738+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:28.611738+0000","last_peered":"2023-10-16T15:53:28.611738+0000","last_clean":"2023-10-16T15:53:28.611738+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:28.611738+0000","last_undegraded":"2023-10-16T15:53:28.611738+0000","last_fullsized":"2023-10-16T15:53:28.611738+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234335+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:28.234335+0000","last_peered":"2023-10-16T15:53:28.234335+0000","last_clean":"2023-10-16T15:53:28.234335+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:28.234335+0000","last_undegraded":"2023-10-16T15:53:28.234335+0000","last_fullsized":"2023-10-16T15:53:28.234335+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610168+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:28.610168+0000","last_peered":"2023-10-16T15:53:28.610168+0000","last_clean":"2023-10-16T15:53:28.610168+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:28.610168+0000","last_undegraded":"2023-10-16T15:53:28.610168+0000","last_fullsized":"2023-10-16T15:53:28.610168+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406595+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:28.406595+0000","last_peered":"2023-10-16T15:53:28.406595+0000","last_clean":"2023-10-16T15:53:28.406595+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:28.406595+0000","last_undegraded":"2023-10-16T15:53:28.406595+0000","last_fullsized":"2023-10-16T15:53:28.406595+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231411+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:28.231411+0000","last_peered":"2023-10-16T15:53:28.231411+0000","last_clean":"2023-10-16T15:53:28.231411+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:28.231411+0000","last_undegraded":"2023-10-16T15:53:28.231411+0000","last_fullsized":"2023-10-16T15:53:28.231411+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610350+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:28.610350+0000","last_peered":"2023-10-16T15:53:28.610350+0000","last_clean":"2023-10-16T15:53:28.610350+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:28.610350+0000","last_undegraded":"2023-10-16T15:53:28.610350+0000","last_fullsized":"2023-10-16T15:53:28.610350+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608892+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:28.608892+0000","last_peered":"2023-10-16T15:53:28.608892+0000","last_clean":"2023-10-16T15:53:28.608892+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:28.608892+0000","last_undegraded":"2023-10-16T15:53:28.608892+0000","last_fullsized":"2023-10-16T15:53:28.608892+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.612883+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:28.612883+0000","last_peered":"2023-10-16T15:53:28.612883+0000","last_clean":"2023-10-16T15:53:28.612883+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:28.612883+0000","last_undegraded":"2023-10-16T15:53:28.612883+0000","last_fullsized":"2023-10-16T15:53:28.612883+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237288+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:28.237288+0000","last_peered":"2023-10-16T15:53:28.237288+0000","last_clean":"2023-10-16T15:53:28.237288+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:28.237288+0000","last_undegraded":"2023-10-16T15:53:28.237288+0000","last_fullsized":"2023-10-16T15:53:28.237288+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235512+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:28.235512+0000","last_peered":"2023-10-16T15:53:28.235512+0000","last_clean":"2023-10-16T15:53:28.235512+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:28.235512+0000","last_undegraded":"2023-10-16T15:53:28.235512+0000","last_fullsized":"2023-10-16T15:53:28.235512+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.232785+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:28.232785+0000","last_peered":"2023-10-16T15:53:28.232785+0000","last_clean":"2023-10-16T15:53:28.232785+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:28.232785+0000","last_undegraded":"2023-10-16T15:53:28.232785+0000","last_fullsized":"2023-10-16T15:53:28.232785+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.407039+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:28.407039+0000","last_peered":"2023-10-16T15:53:28.407039+0000","last_clean":"2023-10-16T15:53:28.407039+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:28.407039+0000","last_undegraded":"2023-10-16T15:53:28.407039+0000","last_fullsized":"2023-10-16T15:53:28.407039+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.608715+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:28.608715+0000","last_peered":"2023-10-16T15:53:28.608715+0000","last_clean":"2023-10-16T15:53:28.608715+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:28.608715+0000","last_undegraded":"2023-10-16T15:53:28.608715+0000","last_fullsized":"2023-10-16T15:53:28.608715+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611547+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:28.611547+0000","last_peered":"2023-10-16T15:53:28.611547+0000","last_clean":"2023-10-16T15:53:28.611547+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:28.611547+0000","last_undegraded":"2023-10-16T15:53:28.611547+0000","last_fullsized":"2023-10-16T15:53:28.611547+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.237011+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:28.237011+0000","last_peered":"2023-10-16T15:53:28.237011+0000","last_clean":"2023-10-16T15:53:28.237011+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:28.237011+0000","last_undegraded":"2023-10-16T15:53:28.237011+0000","last_fullsized":"2023-10-16T15:53:28.237011+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610434+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:28.610434+0000","last_peered":"2023-10-16T15:53:28.610434+0000","last_clean":"2023-10-16T15:53:28.610434+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:28.610434+0000","last_undegraded":"2023-10-16T15:53:28.610434+0000","last_fullsized":"2023-10-16T15:53:28.610434+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.231920+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:28.231920+0000","last_peered":"2023-10-16T15:53:28.231920+0000","last_clean":"2023-10-16T15:53:28.231920+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:28.231920+0000","last_undegraded":"2023-10-16T15:53:28.231920+0000","last_fullsized":"2023-10-16T15:53:28.231920+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.607957+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:28.607957+0000","last_peered":"2023-10-16T15:53:28.607957+0000","last_clean":"2023-10-16T15:53:28.607957+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:28.607957+0000","last_undegraded":"2023-10-16T15:53:28.607957+0000","last_fullsized":"2023-10-16T15:53:28.607957+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.406966+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:28.406966+0000","last_peered":"2023-10-16T15:53:28.406966+0000","last_clean":"2023-10-16T15:53:28.406966+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:28.406966+0000","last_undegraded":"2023-10-16T15:53:28.406966+0000","last_fullsized":"2023-10-16T15:53:28.406966+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.234909+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:28.234909+0000","last_peered":"2023-10-16T15:53:28.234909+0000","last_clean":"2023-10-16T15:53:28.234909+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:28.234909+0000","last_undegraded":"2023-10-16T15:53:28.234909+0000","last_fullsized":"2023-10-16T15:53:28.234909+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610352+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:28.610352+0000","last_peered":"2023-10-16T15:53:28.610352+0000","last_clean":"2023-10-16T15:53:28.610352+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:28.610352+0000","last_undegraded":"2023-10-16T15:53:28.610352+0000","last_fullsized":"2023-10-16T15:53:28.610352+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.611133+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:28.611133+0000","last_peered":"2023-10-16T15:53:28.611133+0000","last_clean":"2023-10-16T15:53:28.611133+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:28.611133+0000","last_undegraded":"2023-10-16T15:53:28.611133+0000","last_fullsized":"2023-10-16T15:53:28.611133+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.610712+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:28.610712+0000","last_peered":"2023-10-16T15:53:28.610712+0000","last_clean":"2023-10-16T15:53:28.610712+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:28.610712+0000","last_undegraded":"2023-10-16T15:53:28.610712+0000","last_fullsized":"2023-10-16T15:53:28.610712+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1247,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T15:53:27.214436+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:27.214436+0000","last_peered":"2023-10-16T15:53:27.214436+0000","last_clean":"2023-10-16T15:53:27.214436+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:27.214436+0000","last_undegraded":"2023-10-16T15:53:27.214436+0000","last_fullsized":"2023-10-16T15:53:27.214436+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60896,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310944,"statfs":{"total":96636764160,"available":96574406656,"internally_reserved":0,"allocated":3428352,"data_stored":1975077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":1.071},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":0.89000000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":0.996},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":0.94199999999999995}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.484},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":1.2210000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.379}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.319},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.163}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.45},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.839}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":1.613},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.7230000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29780,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342060,"statfs":{"total":96636764160,"available":96606269440,"internally_reserved":0,"allocated":2826240,"data_stored":1383679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.258,"15min":1.2190000000000001},"min":{"1min":0.79100000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.99,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.2869999999999999,"15min":1.262},"min":{"1min":0.91500000000000004,"5min":0.78300000000000003,"15min":0.60699999999999998},"max":{"1min":1.722,"5min":1.847,"15min":1.847},"last":1.268}]},{"osd":1,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2809999999999999,"15min":1.2829999999999999},"min":{"1min":0.96799999999999997,"5min":0.79800000000000004,"15min":0.72199999999999998},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.3460000000000001,"15min":1.331},"min":{"1min":1.0640000000000001,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.8580000000000001,"5min":1.8740000000000001,"15min":1.9419999999999999},"last":1.4330000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.256,"15min":1.248},"min":{"1min":0.96299999999999997,"5min":0.85699999999999998,"15min":0.71899999999999997},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.429,"15min":1.3939999999999999},"min":{"1min":1.052,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.468}]},{"osd":3,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.381,"15min":1.399},"min":{"1min":1.048,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.339,"15min":1.3420000000000001},"min":{"1min":1.0589999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.5680000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.631}]},{"osd":4,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3380000000000001,"15min":1.327},"min":{"1min":0.93300000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.698,"5min":1.917,"15min":2.028},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.391,"5min":1.4119999999999999,"15min":1.407},"min":{"1min":1.0349999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7450000000000001,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.1299999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3600000000000001,"15min":1.347},"min":{"1min":0.98599999999999999,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.6970000000000001,"5min":1.9419999999999999,"15min":2.016},"last":1.327},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.556,"15min":1.526},"min":{"1min":1.1499999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.964,"5min":2.169,"15min":2.169},"last":1.0469999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.609,"15min":1.5940000000000001},"min":{"1min":1.3100000000000001,"5min":0.627,"15min":0.627},"max":{"1min":1.9279999999999999,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.536},{"interface":"front","average":{"1min":1.625,"5min":1.6020000000000001,"15min":1.5900000000000001},"min":{"1min":1.383,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.843,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.575}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29788,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342052,"statfs":{"total":96636764160,"available":96606261248,"internally_reserved":0,"allocated":2826240,"data_stored":1383679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.80200000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.821},"last":1.006},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3,"15min":1.2470000000000001},"min":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.639,"5min":2.738,"15min":2.738},"last":0.95299999999999996}]},{"osd":1,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.232,"15min":1.236},"min":{"1min":0.91300000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7569999999999999,"5min":1.8140000000000001,"15min":1.8680000000000001},"last":1.075},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.284,"15min":1.3109999999999999},"min":{"1min":0.89300000000000002,"5min":0.87,"15min":0.87},"max":{"1min":1.675,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.514}]},{"osd":2,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2450000000000001,"15min":1.2629999999999999},"min":{"1min":0.85199999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.784,"5min":1.784,"15min":1.9219999999999999},"last":1.329},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.6990000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.667}]},{"osd":3,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3660000000000001,"15min":1.347},"min":{"1min":0.93100000000000005,"5min":0.86199999999999999,"15min":0.80000000000000004},"max":{"1min":1.7350000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.383,"15min":1.3939999999999999},"min":{"1min":1.103,"5min":0.752,"15min":0.752},"max":{"1min":1.7250000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.4399999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3759999999999999,"15min":1.3460000000000001},"min":{"1min":0.92300000000000004,"5min":0.85699999999999998,"15min":0.78100000000000003},"max":{"1min":1.8160000000000001,"5min":1.972,"15min":1.9730000000000001},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5389999999999999,"15min":1.532},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.77000000000000002},"max":{"1min":1.869,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.724}]},{"osd":6,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4359999999999999,"15min":1.385},"min":{"1min":1.04,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.768,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.458,"15min":1.4390000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.70599999999999996},"max":{"1min":1.7390000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":2.0230000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.556,"15min":1.544},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.923,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.847},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6499999999999999,"15min":1.643},"min":{"1min":1.1719999999999999,"5min":1.032,"15min":0.70499999999999996},"max":{"1min":1.845,"5min":3.109,"15min":3.109},"last":1.962}]}]},{"osd":4,"up_from":13,"seq":55834576003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52004,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319836,"statfs":{"total":96636764160,"available":96583512064,"internally_reserved":0,"allocated":2842624,"data_stored":1384813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2330000000000001,"15min":1.212},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.52700000000000002},"max":{"1min":1.794,"5min":1.883,"15min":1.883},"last":0.874},{"interface":"front","average":{"1min":1.302,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.79700000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.95799999999999996,"5min":0.83499999999999996,"15min":0.629},"max":{"1min":1.675,"5min":1.929,"15min":2.2160000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.3160000000000001},"min":{"1min":0.92000000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0179999999999998,"5min":2.157,"15min":2.2989999999999999},"last":1.448}]},{"osd":2,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4490000000000001,"15min":1.4670000000000001},"min":{"1min":1.1160000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.498,"15min":1.4970000000000001},"min":{"1min":1.1870000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.222}]},{"osd":3,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":1.081,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.23,"5min":2.23,"15min":2.3650000000000002},"last":1.117},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4099999999999999,"15min":1.401},"min":{"1min":1.0069999999999999,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.1479999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.375,"15min":1.3420000000000001},"min":{"1min":1.03,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.911,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.46,"15min":1.4370000000000001},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.161,"5min":2.161,"15min":2.4489999999999998},"last":1.7849999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":0.97499999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.554,"15min":1.5269999999999999},"min":{"1min":1.0089999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.2789999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":2.0049999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.617,"15min":1.5860000000000001},"min":{"1min":1.2669999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.946},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.6339999999999999,"15min":1.6040000000000001},"min":{"1min":1.3999999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.242,"5min":2.2599999999999998,"15min":2.347},"last":1.8380000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56156,"kb_used_data":2768,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315684,"statfs":{"total":96636764160,"available":96579260416,"internally_reserved":0,"allocated":2834432,"data_stored":1383279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.876}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.96799999999999997}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":1.621},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":1.732}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":1.212}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.3460000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.5409999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":1.47},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.647}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51984,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319856,"statfs":{"total":96636764160,"available":96583532544,"internally_reserved":0,"allocated":2822144,"data_stored":1383097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.94299999999999995,"15min":0.96499999999999997},"min":{"1min":0.68300000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.5740000000000001,"5min":1.726,"15min":1.8340000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.107},"min":{"1min":0.54200000000000004,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.8999999999999999},"last":1.228}]},{"osd":1,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0669999999999999,"15min":1.0680000000000001},"min":{"1min":0.622,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7410000000000001,"5min":1.974,"15min":1.974},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.2050000000000001,"15min":1.248},"min":{"1min":0.76600000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7090000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.579}]},{"osd":3,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.284,"15min":1.3109999999999999},"min":{"1min":0.69199999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.704,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.3560000000000001,"15min":1.3839999999999999},"min":{"1min":0.89300000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9570000000000001,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.6499999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.4910000000000001},"min":{"1min":1.147,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0800000000000001,"5min":2.335,"15min":2.335},"last":1.381},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3899999999999999,"15min":1.429},"min":{"1min":0.92300000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7549999999999999,"5min":2.0920000000000001,"15min":2.1589999999999998},"last":1.5289999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.486,"15min":1.5089999999999999},"min":{"1min":1.2390000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8620000000000001,"5min":2.657,"15min":2.657},"last":1.056},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.6060000000000001,"15min":1.631},"min":{"1min":1.0580000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2389999999999999,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.516,"15min":1.5720000000000001},"min":{"1min":1.087,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9019999999999999,"5min":1.9850000000000001,"15min":2.3849999999999998},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.629,"15min":1.6399999999999999},"min":{"1min":1.28,"5min":0.753,"15min":0.753},"max":{"1min":1.992,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.198}]},{"osd":7,"last update":"Mon Oct 16 15:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.28,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.0449999999999999,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.893},{"interface":"front","average":{"1min":1.756,"5min":1.7110000000000001,"15min":1.7529999999999999},"min":{"1min":1.526,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.113,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.3400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576003,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":1383153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.206}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.3660000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.8859999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":2.2959999999999998},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.9810000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":2.0539999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.7549999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":2.1739999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":2.5059999999999998}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":2.621},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":2.452}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48408,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323432,"statfs":{"total":96636764160,"available":96587194368,"internally_reserved":0,"allocated":3420160,"data_stored":1973555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.31},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":0.749},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":0.68400000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:32.284 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:53:32.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T15:53:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.563+0000 7f7a3efae640 1 Processor -- start 2023-10-16T15:53:32.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.563+0000 7f7a3efae640 1 -- start start 2023-10-16T15:53:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3efae640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7a38070bf0 0x7f7a38071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3efae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a380acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3efae640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a38070760 con 0x7f7a3815dc90 2023-10-16T15:53:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3efae640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a380715c0 con 0x7f7a38070bf0 2023-10-16T15:53:32.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3efae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a380ad240 con 0x7f7a380ac890 2023-10-16T15:53:32.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3d7ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7a38070bf0 0x7f7a38071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3d7ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7a38070bf0 0x7f7a38071080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45520/0 (socket says 172.21.15.29:45520) 2023-10-16T15:53:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.565+0000 7f7a3d7ab640 1 -- 172.21.15.29:0/2507668369 learned_addr learned my addr 172.21.15.29:0/2507668369 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.566+0000 7f7a3e7ad640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a380acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.566+0000 7f7a3d7ab640 1 -- 172.21.15.29:0/2507668369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 msgr2=0x7f7a380acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:32.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.566+0000 7f7a3d7ab640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a380acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.566+0000 7f7a3d7ab640 1 -- 172.21.15.29:0/2507668369 >> v1:172.21.15.170:6789/0 conn(0x7f7a3815dc90 legacy=0x7f7a380c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.566+0000 7f7a3d7ab640 1 -- 172.21.15.29:0/2507668369 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380cdac0 con 0x7f7a38070bf0 2023-10-16T15:53:32.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.566+0000 7f7a3e7ad640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a380acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:32.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.567+0000 7f7a3d7ab640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a38070bf0 0x7f7a38071080 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009870 tx=0x7f7a2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7333b8d6c801b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.567+0000 7f7a3cfaa640 1 -- 172.21.15.29:0/2507668369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28031640 con 0x7f7a38070bf0 2023-10-16T15:53:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.567+0000 7f7a3cfaa640 1 -- 172.21.15.29:0/2507668369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a280317a0 con 0x7f7a38070bf0 2023-10-16T15:53:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.567+0000 7f7a3cfaa640 1 -- 172.21.15.29:0/2507668369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28031b30 con 0x7f7a38070bf0 2023-10-16T15:53:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 -- 172.21.15.29:0/2507668369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a38070bf0 msgr2=0x7f7a38071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a38070bf0 0x7f7a38071080 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009870 tx=0x7f7a2802b8f0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 -- 172.21.15.29:0/2507668369 shutdown_connections 2023-10-16T15:53:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a380acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 --2- 172.21.15.29:0/2507668369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a38070bf0 0x7f7a38071080 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 -- 172.21.15.29:0/2507668369 >> 172.21.15.29:0/2507668369 conn(0x7f7a380ce6a0 msgr2=0x7f7a380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 -- 172.21.15.29:0/2507668369 shutdown_connections 2023-10-16T15:53:32.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.568+0000 7f7a3efae640 1 -- 172.21.15.29:0/2507668369 wait complete. 2023-10-16T15:53:32.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.570+0000 7f7a3efae640 1 Processor -- start 2023-10-16T15:53:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.570+0000 7f7a3efae640 1 -- start start 2023-10-16T15:53:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.571+0000 7f7a3efae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a38070bf0 0x7f7a3806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.571+0000 7f7a3efae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a38069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.571+0000 7f7a3efae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 0x7f7a380c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.571+0000 7f7a3efae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a380b2340 con 0x7f7a3815dc90 2023-10-16T15:53:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.571+0000 7f7a3efae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3806a680 con 0x7f7a38070bf0 2023-10-16T15:53:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.571+0000 7f7a3efae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3806a960 con 0x7f7a380ac890 2023-10-16T15:53:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 0x7f7a380c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 0x7f7a380c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45524/0 (socket says 172.21.15.29:45524) 2023-10-16T15:53:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 -- 172.21.15.29:0/725986843 learned_addr learned my addr 172.21.15.29:0/725986843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3d7ab640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a38069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 -- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 msgr2=0x7f7a38069e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a38069e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 -- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a38070bf0 msgr2=0x7f7a3806b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a38070bf0 0x7f7a3806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.572+0000 7f7a3e7ad640 1 -- 172.21.15.29:0/725986843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a280094d0 con 0x7f7a3815dc90 2023-10-16T15:53:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.573+0000 7f7a3e7ad640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 0x7f7a380c9cd0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000cef0 tx=0x7f7a3000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.573+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3000d840 con 0x7f7a3815dc90 2023-10-16T15:53:32.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.573+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3000d9c0 con 0x7f7a3815dc90 2023-10-16T15:53:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.573+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3000dd70 con 0x7f7a3815dc90 2023-10-16T15:53:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.573+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3806ac40 con 0x7f7a3815dc90 2023-10-16T15:53:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.573+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3806a370 con 0x7f7a3815dc90 2023-10-16T15:53:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.574+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a380b2480 con 0x7f7a3815dc90 2023-10-16T15:53:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.575+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a3001d360 con 0x7f7a3815dc90 2023-10-16T15:53:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.576+0000 7f7a2e7fc640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7a1c06af90 0x7f7a1c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.576+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7a3008a270 con 0x7f7a3815dc90 2023-10-16T15:53:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.577+0000 7f7a3dfac640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7a1c06af90 0x7f7a1c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.578+0000 7f7a3dfac640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7a1c06af90 0x7f7a1c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a34006fd0 tx=0x7f7a34006bc0 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:32.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.583+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a380b2480 con 0x7f7a3815dc90 2023-10-16T15:53:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.843+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7a380c8ab0 con 0x7f7a3815dc90 2023-10-16T15:53:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.848+0000 7f7a2e7fc640 1 -- 172.21.15.29:0/725986843 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7a300552a0 con 0x7f7a3815dc90 2023-10-16T15:53:32.851 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T15:53:32.851 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T15:53:32.851 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T15:53:32.851 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T15:53:32.851 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T15:53:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T15:53:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7a1c06af90 msgr2=0x7f7a1c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7a1c06af90 0x7f7a1c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a34006fd0 tx=0x7f7a34006bc0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 msgr2=0x7f7a380c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 0x7f7a380c9cd0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000cef0 tx=0x7f7a3000a710 comp rx=0 tx=0).stop 2023-10-16T15:53:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 shutdown_connections 2023-10-16T15:53:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7a1c06af90 0x7f7a1c06d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a3815dc90 0x7f7a380c9cd0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7a380ac890 0x7f7a38069e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 --2- 172.21.15.29:0/725986843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7a38070bf0 0x7f7a3806b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 >> 172.21.15.29:0/725986843 conn(0x7f7a380ce6a0 msgr2=0x7f7a38160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 shutdown_connections 2023-10-16T15:53:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:32.851+0000 7f7a3efae640 1 -- 172.21.15.29:0/725986843 wait complete. 2023-10-16T15:53:32.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.091+0000 7fe06e32e640 1 Processor -- start 2023-10-16T15:53:33.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.091+0000 7fe06e32e640 1 -- start start 2023-10-16T15:53:33.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe06e32e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe0680b4c80 0x7fe068167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe06e32e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680d15a0 con 0x7fe06815cfc0 2023-10-16T15:53:33.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe06e32e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680d1880 con 0x7fe06815cbe0 2023-10-16T15:53:33.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe06e32e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0680d1b60 con 0x7fe0680b4c80 2023-10-16T15:53:33.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe06cb2b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe06815cfc0 0x7fe0680b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57786/0 (socket says 172.21.15.29:57786) 2023-10-16T15:53:33.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe06cb2b640 1 -- 172.21.15.29:0/2461050788 learned_addr learned my addr 172.21.15.29:0/2461050788 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.093+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570267659 0 0) 0x7fe0680d15a0 con 0x7fe06815cfc0 2023-10-16T15:53:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe054003520 con 0x7fe06815cfc0 2023-10-16T15:53:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe06db2d640 1 --2- 172.21.15.29:0/2461050788 >> v2:172.21.15.170:3300/0 conn(0x7fe0680b4c80 0x7fe068167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750462229 0 0) 0x7fe054003520 con 0x7fe06815cfc0 2023-10-16T15:53:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 >> v2:172.21.15.170:3300/0 conn(0x7fe0680b4c80 msgr2=0x7fe068167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 --2- 172.21.15.29:0/2461050788 >> v2:172.21.15.170:3300/0 conn(0x7fe0680b4c80 0x7fe068167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 >> v1:172.21.15.170:6789/0 conn(0x7fe06815cbe0 legacy=0x7fe0680d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680d1e40 con 0x7fe06815cfc0 2023-10-16T15:53:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe058001670 con 0x7fe06815cfc0 2023-10-16T15:53:33.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 >> v1:172.21.15.29:6789/0 conn(0x7fe06815cfc0 legacy=0x7fe0680b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe054003970 0x7fe054003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe05400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680d1b60 con 0x7fe054005180 2023-10-16T15:53:33.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05400c0d0 con 0x7fe054003970 2023-10-16T15:53:33.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.094+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05400c290 con 0x7fe0540047c0 2023-10-16T15:53:33.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06db2d640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe05400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06db2d640 1 -- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0540047c0 msgr2=0x7fe054004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:33.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06cb2b640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06db2d640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0540047c0 0x7fe054004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06db2d640 1 -- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe054003970 msgr2=0x7fe054003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06d32c640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe054003970 0x7fe054003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06db2d640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe054003970 0x7fe054003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06db2d640 1 -- 172.21.15.29:0/2461050788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054003520 con 0x7fe054005180 2023-10-16T15:53:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.095+0000 7fe06d32c640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe054003970 0x7fe054003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:33.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.096+0000 7fe06db2d640 1 -- 172.21.15.29:0/2461050788 reap_dead start 2023-10-16T15:53:33.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.096+0000 7fe06db2d640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe05400b980 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fe060007360 tx=0x7fe0540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570338554cc36911 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06002ccf0 con 0x7fe054005180 2023-10-16T15:53:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06002ce90 con 0x7fe054005180 2023-10-16T15:53:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe05ffff640 1 -- 172.21.15.29:0/2461050788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060003bd0 con 0x7fe054005180 2023-10-16T15:53:33.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 -- 172.21.15.29:0/2461050788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 msgr2=0x7fe05400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe05400b980 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fe060007360 tx=0x7fe0540057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 -- 172.21.15.29:0/2461050788 shutdown_connections 2023-10-16T15:53:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 --2- 172.21.15.29:0/2461050788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe05400b980 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 -- 172.21.15.29:0/2461050788 >> 172.21.15.29:0/2461050788 conn(0x7fe0680cf570 msgr2=0x7fe0680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:33.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 -- 172.21.15.29:0/2461050788 shutdown_connections 2023-10-16T15:53:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.097+0000 7fe06e32e640 1 -- 172.21.15.29:0/2461050788 wait complete. 2023-10-16T15:53:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.099+0000 7fe06e32e640 1 Processor -- start 2023-10-16T15:53:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.099+0000 7fe06e32e640 1 -- start start 2023-10-16T15:53:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06e32e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe06815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06e32e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0681f6610 0x7fe0680b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06e32e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0681e64a0 0x7fe0681e6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06e32e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681e6e70 con 0x7fe054005180 2023-10-16T15:53:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06e32e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06815f0a0 con 0x7fe0681f6610 2023-10-16T15:53:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06e32e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680b26d0 con 0x7fe0681e64a0 2023-10-16T15:53:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06d32c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe06815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06d32c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe06815eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45544/0 (socket says 172.21.15.29:45544) 2023-10-16T15:53:33.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.100+0000 7fe06d32c640 1 -- 172.21.15.29:0/201583795 learned_addr learned my addr 172.21.15.29:0/201583795 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06db2d640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0681e64a0 0x7fe0681e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06cb2b640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0681f6610 0x7fe0680b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06d32c640 1 -- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0681e64a0 msgr2=0x7fe0681e6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06d32c640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0681e64a0 0x7fe0681e6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06d32c640 1 -- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0681f6610 msgr2=0x7fe0680b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06d32c640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0681f6610 0x7fe0680b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06d32c640 1 -- 172.21.15.29:0/201583795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680b29b0 con 0x7fe054005180 2023-10-16T15:53:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06d32c640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe06815eb60 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fe06400d7a0 tx=0x7fe06400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064005270 con 0x7fe054005180 2023-10-16T15:53:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0640053f0 con 0x7fe054005180 2023-10-16T15:53:33.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06400b8f0 con 0x7fe054005180 2023-10-16T15:53:33.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.101+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0680b2c90 con 0x7fe054005180 2023-10-16T15:53:33.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.102+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0680b3250 con 0x7fe054005180 2023-10-16T15:53:33.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.103+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0680b33e0 con 0x7fe054005180 2023-10-16T15:53:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.104+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe06400ba50 con 0x7fe054005180 2023-10-16T15:53:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.105+0000 7fe05dffb640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe04c06afc0 0x7fe04c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.105+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe064089890 con 0x7fe054005180 2023-10-16T15:53:33.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.105+0000 7fe06cb2b640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe04c06afc0 0x7fe04c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:33.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.107+0000 7fe06cb2b640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe04c06afc0 0x7fe04c06d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0580017a0 tx=0x7fe058009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:33.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.109+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe064054660 con 0x7fe054005180 2023-10-16T15:53:33.157 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-16T15:53:33.155+0000 7f3679aea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:53:33.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.338+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe06804e9b0 con 0x7fe054005180 2023-10-16T15:53:33.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.339+0000 7fe05dffb640 1 -- 172.21.15.29:0/201583795 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4932 (secure 0 0 0) 0x7fe064019310 con 0x7fe054005180 2023-10-16T15:53:33.341 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:33.341 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":109,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104},{"gid":8761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T15:53:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 109 2023-10-16T15:53:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe04c06afc0 msgr2=0x7fe04c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe04c06afc0 0x7fe04c06d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0580017a0 tx=0x7fe058009040 comp rx=0 tx=0).stop 2023-10-16T15:53:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 msgr2=0x7fe06815eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe06815eb60 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fe06400d7a0 tx=0x7fe06400dc60 comp rx=0 tx=0).stop 2023-10-16T15:53:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 shutdown_connections 2023-10-16T15:53:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe04c06afc0 0x7fe04c06d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.341+0000 7fe06e32e640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0681e64a0 0x7fe0681e6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.342+0000 7fe06e32e640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0681f6610 0x7fe0680b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.342+0000 7fe06e32e640 1 --2- 172.21.15.29:0/201583795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe054005180 0x7fe06815eb60 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.342+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 >> 172.21.15.29:0/201583795 conn(0x7fe0680cf570 msgr2=0x7fe0680b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.342+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 shutdown_connections 2023-10-16T15:53:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:33.342+0000 7fe06e32e640 1 -- 172.21.15.29:0/201583795 wait complete. 2023-10-16T15:53:33.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:53:26.655925+0000', 'modified': '2023-10-16T15:53:29.777218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10821}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10821': {'gid': 10821, 'name': 'c', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/3313937659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 3313937659}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 3313937659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:53:33.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:53:33.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T15:53:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.000+0000 7f99dca14640 1 Processor -- start 2023-10-16T15:53:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.000+0000 7f99dca14640 1 -- start start 2023-10-16T15:53:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.001+0000 7f99dca14640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 0x7f99d806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.001+0000 7f99dca14640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d815d910 con 0x7f99d8159950 2023-10-16T15:53:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.001+0000 7f99dca14640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d815dbf0 con 0x7f99d8070b60 2023-10-16T15:53:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.001+0000 7f99dca14640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d815e490 con 0x7f99d80ca0a0 2023-10-16T15:53:34.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.002+0000 7f99d6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f99d8159950 0x7f99d8070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38350/0 (socket says 172.21.15.29:38350) 2023-10-16T15:53:34.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.002+0000 7f99d6d9d640 1 -- 172.21.15.29:0/450154664 learned_addr learned my addr 172.21.15.29:0/450154664 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:34.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.002+0000 7f99d759e640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 0x7f99d806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.002+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739590168 0 0) 0x7f99d815d910 con 0x7f99d8159950 2023-10-16T15:53:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.002+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c4003520 con 0x7f99d8159950 2023-10-16T15:53:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197377479 0 0) 0x7f99c4003520 con 0x7f99d8159950 2023-10-16T15:53:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 msgr2=0x7f99d806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 0x7f99d806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 >> v1:172.21.15.170:6789/0 conn(0x7f99d8070b60 legacy=0x7f99d806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d815ded0 con 0x7f99d8159950 2023-10-16T15:53:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f99c8001670 con 0x7f99d8159950 2023-10-16T15:53:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 >> v1:172.21.15.29:6789/0 conn(0x7f99d8159950 legacy=0x7f99d8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 0x7f99c40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99c4004d00 0x7f99c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d815e490 con 0x7f99c4003f20 2023-10-16T15:53:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c400bb10 con 0x7f99c4004d00 2023-10-16T15:53:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c400dee0 con 0x7f99d80ca0a0 2023-10-16T15:53:34.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.003+0000 7f99d659c640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 0x7f99c40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d759e640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99c4004d00 0x7f99c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d659c640 1 -- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 msgr2=0x7f99c40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d659c640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80ca0a0 0x7f99c40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d659c640 1 -- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99c4004d00 msgr2=0x7f99c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d659c640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99c4004d00 0x7f99c4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.004+0000 7f99d659c640 1 -- 172.21.15.29:0/450154664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4003520 con 0x7f99c4003f20 2023-10-16T15:53:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.005+0000 7f99d659c640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99c40047c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001750 tx=0x7f99c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d36ad69ac8d2753 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.005+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c00052d0 con 0x7f99c4003f20 2023-10-16T15:53:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.005+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c002e050 con 0x7f99c4003f20 2023-10-16T15:53:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.005+0000 7f99d5d9b640 1 -- 172.21.15.29:0/450154664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c00035f0 con 0x7f99c4003f20 2023-10-16T15:53:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99dca14640 1 -- 172.21.15.29:0/450154664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 msgr2=0x7f99c40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99dca14640 1 --2- 172.21.15.29:0/450154664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99c40047c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001750 tx=0x7f99c40057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99d759e640 1 -- 172.21.15.29:0/450154664 reap_dead start 2023-10-16T15:53:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99dca14640 1 -- 172.21.15.29:0/450154664 shutdown_connections 2023-10-16T15:53:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99dca14640 1 -- 172.21.15.29:0/450154664 >> 172.21.15.29:0/450154664 conn(0x7f99d80ce6a0 msgr2=0x7f99d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99dca14640 1 -- 172.21.15.29:0/450154664 shutdown_connections 2023-10-16T15:53:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.006+0000 7f99dca14640 1 -- 172.21.15.29:0/450154664 wait complete. 2023-10-16T15:53:34.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.008+0000 7f99dca14640 1 Processor -- start 2023-10-16T15:53:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.008+0000 7f99dca14640 1 -- start start 2023-10-16T15:53:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99dca14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99d81571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99dca14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99d814baa0 0x7f99d80b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99dca14640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80b0d90 0x7f99d815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99dca14640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80af7b0 con 0x7f99c4003f20 2023-10-16T15:53:34.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99dca14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80afa90 con 0x7f99d814baa0 2023-10-16T15:53:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99dca14640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80afd70 con 0x7f99d80b0d90 2023-10-16T15:53:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99d81571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99d81571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47050/0 (socket says 172.21.15.29:47050) 2023-10-16T15:53:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.009+0000 7f99d6d9d640 1 -- 172.21.15.29:0/753428361 learned_addr learned my addr 172.21.15.29:0/753428361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99d6d9d640 1 -- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80b0d90 msgr2=0x7f99d815b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80b0d90 0x7f99d815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99d6d9d640 1 -- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99d814baa0 msgr2=0x7f99d80b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99d814baa0 0x7f99d80b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99d6d9d640 1 -- 172.21.15.29:0/753428361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d80b0050 con 0x7f99c4003f20 2023-10-16T15:53:34.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99d81571b0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f99c800e770 tx=0x7f99c800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.010+0000 7f99cf7fe640 1 -- 172.21.15.29:0/753428361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c800ee20 con 0x7f99c4003f20 2023-10-16T15:53:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.011+0000 7f99cf7fe640 1 -- 172.21.15.29:0/753428361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800c870 con 0x7f99c4003f20 2023-10-16T15:53:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.011+0000 7f99cf7fe640 1 -- 172.21.15.29:0/753428361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c800cc00 con 0x7f99c4003f20 2023-10-16T15:53:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.011+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d80b0330 con 0x7f99c4003f20 2023-10-16T15:53:34.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.011+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d80c58c0 con 0x7f99c4003f20 2023-10-16T15:53:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.013+0000 7f99cf7fe640 1 -- 172.21.15.29:0/753428361 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99c801d460 con 0x7f99c4003f20 2023-10-16T15:53:34.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.015+0000 7f99cf7fe640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f99b806af90 0x7f99b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.015+0000 7f99cf7fe640 1 -- 172.21.15.29:0/753428361 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99c80897b0 con 0x7f99c4003f20 2023-10-16T15:53:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.015+0000 7f99dfda8740 1 Processor -- start 2023-10-16T15:53:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.015+0000 7f99d659c640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f99b806af90 0x7f99b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.016+0000 7f99dfda8740 1 -- start start 2023-10-16T15:53:34.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.016+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254950e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.016+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254953df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254950e5b0 0x56254953e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99dfda8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56254942e7a0 con 0x56254953b550 2023-10-16T15:53:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562549480740 con 0x56254953b170 2023-10-16T15:53:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x56254947de30 con 0x56254950e5b0 2023-10-16T15:53:34.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254953df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254953df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47066/0 (socket says 172.21.15.29:47066) 2023-10-16T15:53:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 -- 172.21.15.29:0/1765615109 learned_addr learned my addr 172.21.15.29:0/1765615109 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f99b806af90 0x7f99b806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001750 tx=0x7f99c002f000 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254950e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 -- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254950e5b0 msgr2=0x56254953e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d759e640 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254950e5b0 0x56254953e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254950e5b0 0x56254953e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 -- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 msgr2=0x56254950e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254950e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d659c640 1 -- 172.21.15.29:0/1765615109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56254943d910 con 0x56254953b550 2023-10-16T15:53:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.017+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254950e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.018+0000 7f99d659c640 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254953df00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001840 tx=0x7f99c0009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5b8c41ab3fd403 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.018+0000 7f99ccff9640 1 -- 172.21.15.29:0/1765615109 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0005490 con 0x56254953b550 2023-10-16T15:53:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.018+0000 7f99ccff9640 1 -- 172.21.15.29:0/1765615109 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c00075c0 con 0x56254953b550 2023-10-16T15:53:34.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.018+0000 7f99ccff9640 1 -- 172.21.15.29:0/1765615109 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c00069d0 con 0x56254953b550 2023-10-16T15:53:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 -- 172.21.15.29:0/1765615109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 msgr2=0x56254953df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254953df00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001840 tx=0x7f99c0009030 comp rx=0 tx=0).stop 2023-10-16T15:53:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 -- 172.21.15.29:0/1765615109 shutdown_connections 2023-10-16T15:53:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254953df00 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254950e5b0 0x56254953e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1765615109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254950e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 -- 172.21.15.29:0/1765615109 >> 172.21.15.29:0/1765615109 conn(0x562549510380 msgr2=0x562549558f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 -- 172.21.15.29:0/1765615109 shutdown_connections 2023-10-16T15:53:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.019+0000 7f99dfda8740 1 -- 172.21.15.29:0/1765615109 wait complete. 2023-10-16T15:53:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.020+0000 7f99dfda8740 1 -- start start 2023-10-16T15:53:34.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.020+0000 7f99dfda8740 1 Processor -- start 2023-10-16T15:53:34.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 0x56254952f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254956c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x56254956c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562549479720 con 0x56254953b170 2023-10-16T15:53:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562549417970 con 0x56254953b550 2023-10-16T15:53:34.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x562549417610 con 0x56254953a260 2023-10-16T15:53:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99dfda8740 10 client.0 _ll_register_callbacks cb 0x5625494399d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:53:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 0x56254952f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 0x56254952f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47076/0 (socket says 172.21.15.29:47076) 2023-10-16T15:53:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.021+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2632268150 learned_addr learned my addr 172.21.15.29:0/2632268150 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 msgr2=0x56254956c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x56254956c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 msgr2=0x56254956c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d659c640 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254956c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254956c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2632268150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c4003520 con 0x56254953b170 2023-10-16T15:53:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 0x56254952f330 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f99c800c4b0 tx=0x7f99c8058410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ad44c60c49fddb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:53:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99b5ffb640 1 -- 172.21.15.29:0/2632268150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c805c150 con 0x56254953b170 2023-10-16T15:53:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99b5ffb640 1 -- 172.21.15.29:0/2632268150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99c8087c10 con 0x56254953b170 2023-10-16T15:53:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625494a0160 con 0x56254953b170 2023-10-16T15:53:34.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99b5ffb640 1 -- 172.21.15.29:0/2632268150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99c8087dd0 con 0x56254953b170 2023-10-16T15:53:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99dfda8740 10 client.10962 fetch_fsmap learned FSMap version 109 2023-10-16T15:53:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.022+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=2+}) v3 -- 0x562549477da0 con 0x56254953b170 2023-10-16T15:53:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99b5ffb640 1 -- 172.21.15.29:0/2632268150 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c8073020 con 0x56254953b170 2023-10-16T15:53:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99b5ffb640 1 -- 172.21.15.29:0/2632268150 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 109) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f99c8054a60 con 0x56254953b170 2023-10-16T15:53:34.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99dfda8740 10 client.10962 fetch_fsmap finished waiting for FSMap version 109 2023-10-16T15:53:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99dfda8740 10 client.10962 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:53:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99dfda8740 20 client.10962 populate_metadata read hostname 'smithi029' 2023-10-16T15:53:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x562549540860 0x5625495763c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99dfda8740 4 client.10962 mds_command: new command op to 10821 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:53:34.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 --> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56254944bcc0 con 0x562549540860 2023-10-16T15:53:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.023+0000 7f99d659c640 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x562549540860 0x5625495763c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.024+0000 7f99d659c640 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x562549540860 0x5625495763c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75326f702f9c887b server_cookie=5eeda14d0a3cb0e0 in_seq=0 out_seq=0 2023-10-16T15:53:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.024+0000 7f99b5ffb640 10 client.10962 ms_handle_connect on v2:172.21.15.29:6834/3313937659 2023-10-16T15:53:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99b5ffb640 1 -- 172.21.15.29:0/2632268150 <== mds.0 v2:172.21.15.29:6834/3313937659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56254944bcc0 con 0x562549540860 2023-10-16T15:53:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99b5ffb640 10 client.10962 handle_command_reply: tid=0 2023-10-16T15:53:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 client.10962 shutdown 2023-10-16T15:53:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 msgr2=0x56254952f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 0x56254952f330 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f99c800c4b0 tx=0x7f99c8058410 comp rx=0 tx=0).stop 2023-10-16T15:53:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 shutdown_connections 2023-10-16T15:53:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x56254956c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254956c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x562549540860 0x5625495763c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2632268150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b170 0x56254952f330 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 >> 172.21.15.29:0/2632268150 conn(0x56254950e5b0 msgr2=0x56254953def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99b5ffb640 0 client.10962 ms_handle_reset on v2:172.21.15.29:6834/3313937659 2023-10-16T15:53:34.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 shutdown_connections 2023-10-16T15:53:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 1 -- 172.21.15.29:0/2632268150 wait complete. 2023-10-16T15:53:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.025+0000 7f99dfda8740 20 client.10962 trim_cache size 0 max 16384 2023-10-16T15:53:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.042+0000 7f99dfda8740 1 Processor -- start 2023-10-16T15:53:34.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.042+0000 7f99dfda8740 1 -- start start 2023-10-16T15:53:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.043+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 0x56254950d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.043+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953b170 0x56254957a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.043+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254957a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.043+0000 7f99dfda8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625491ddf20 con 0x56254953a260 2023-10-16T15:53:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.043+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56254941f520 con 0x56254953b550 2023-10-16T15:53:34.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.043+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625495098b0 con 0x56254953b170 2023-10-16T15:53:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 0x56254950d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 0x56254950d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47078/0 (socket says 172.21.15.29:47078) 2023-10-16T15:53:34.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2829649198 learned_addr learned my addr 172.21.15.29:0/2829649198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d659c640 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953b170 0x56254957a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d759e640 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254957a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953b170 msgr2=0x56254957a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953b170 0x56254957a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 msgr2=0x56254957a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254957a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 -- 172.21.15.29:0/2829649198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562549416b80 con 0x56254953a260 2023-10-16T15:53:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d759e640 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254957a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.044+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 0x56254950d840 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f99c80582e0 tx=0x7f99c8015050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711d8c9fd1824286 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99cd7fa640 1 -- 172.21.15.29:0/2829649198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c805c150 con 0x56254953a260 2023-10-16T15:53:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99cd7fa640 1 -- 172.21.15.29:0/2829649198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c8073040 con 0x56254953a260 2023-10-16T15:53:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99cd7fa640 1 -- 172.21.15.29:0/2829649198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8054a60 con 0x56254953a260 2023-10-16T15:53:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 -- 172.21.15.29:0/2829649198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 msgr2=0x56254950d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 0x56254950d840 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f99c80582e0 tx=0x7f99c8015050 comp rx=0 tx=0).stop 2023-10-16T15:53:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 -- 172.21.15.29:0/2829649198 shutdown_connections 2023-10-16T15:53:34.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b550 0x56254957a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953b170 0x56254957a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 --2- 172.21.15.29:0/2829649198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953a260 0x56254950d840 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.045+0000 7f99dfda8740 1 -- 172.21.15.29:0/2829649198 >> 172.21.15.29:0/2829649198 conn(0x56254950e5b0 msgr2=0x56254953a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.046+0000 7f99dfda8740 1 -- 172.21.15.29:0/2829649198 shutdown_connections 2023-10-16T15:53:34.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.046+0000 7f99dfda8740 1 -- 172.21.15.29:0/2829649198 wait complete. 2023-10-16T15:53:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.046+0000 7f99dfda8740 1 -- start start 2023-10-16T15:53:34.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.046+0000 7f99dfda8740 1 Processor -- start 2023-10-16T15:53:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.047+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x5625495a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.047+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254957a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.047+0000 7f99dfda8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254957a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.047+0000 7f99dfda8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x562549556ce0 con 0x56254953b550 2023-10-16T15:53:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.047+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x562549513960 con 0x56254953b170 2023-10-16T15:53:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.047+0000 7f99dfda8740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x56254953f470 con 0x56254953a260 2023-10-16T15:53:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99dfda8740 10 client.0 _ll_register_callbacks cb 0x56254950e500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:53:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254957a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254957a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47090/0 (socket says 172.21.15.29:47090) 2023-10-16T15:53:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 -- 172.21.15.29:0/1426938708 learned_addr learned my addr 172.21.15.29:0/1426938708 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x5625495a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d659c640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254957a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 -- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 msgr2=0x5625495a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x5625495a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 -- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 msgr2=0x56254957a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254957a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d759e640 1 -- 172.21.15.29:0/1426938708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c4003520 con 0x56254953b550 2023-10-16T15:53:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.048+0000 7f99d659c640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254957a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99d759e640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254957a920 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f99d806d350 tx=0x7f99d0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ca7db228ddc9e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:53:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99b5ffb640 1 -- 172.21.15.29:0/1426938708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000e040 con 0x56254953b550 2023-10-16T15:53:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99b5ffb640 1 -- 172.21.15.29:0/1426938708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99d0007c10 con 0x56254953b550 2023-10-16T15:53:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99c00075c0 con 0x56254953b550 2023-10-16T15:53:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99b5ffb640 1 -- 172.21.15.29:0/1426938708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99d0007dd0 con 0x56254953b550 2023-10-16T15:53:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99dfda8740 10 client.10968 fetch_fsmap learned FSMap version 109 2023-10-16T15:53:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=2+}) v3 -- 0x56254943c170 con 0x56254953b550 2023-10-16T15:53:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99b5ffb640 1 -- 172.21.15.29:0/1426938708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c00075c0 con 0x56254953b550 2023-10-16T15:53:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99b5ffb640 1 -- 172.21.15.29:0/1426938708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 109) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f99d0014040 con 0x56254953b550 2023-10-16T15:53:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.049+0000 7f99dfda8740 10 client.10968 fetch_fsmap finished waiting for FSMap version 109 2023-10-16T15:53:34.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99dfda8740 10 client.10968 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:53:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99dfda8740 20 client.10968 populate_metadata read hostname 'smithi029' 2023-10-16T15:53:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x56254955a880 0x56254952d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99dfda8740 4 client.10968 mds_command: new command op to 10821 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:53:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 --> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562549472700 con 0x56254955a880 2023-10-16T15:53:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x56254955a880 0x56254952d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99d6d9d640 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x56254955a880 0x56254952d1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1e7b631df38b212 server_cookie=475e174a471ee931 in_seq=0 out_seq=0 2023-10-16T15:53:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.050+0000 7f99b5ffb640 10 client.10968 ms_handle_connect on v2:172.21.15.29:6834/3313937659 2023-10-16T15:53:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99b5ffb640 1 -- 172.21.15.29:0/1426938708 <== mds.0 v2:172.21.15.29:6834/3313937659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x562549472700 con 0x56254955a880 2023-10-16T15:53:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99b5ffb640 10 client.10968 handle_command_reply: tid=0 2023-10-16T15:53:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 client.10968 shutdown 2023-10-16T15:53:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 msgr2=0x56254957a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254957a920 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f99d806d350 tx=0x7f99d0003f90 comp rx=0 tx=0).stop 2023-10-16T15:53:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 shutdown_connections 2023-10-16T15:53:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:6834/3313937659,v1:172.21.15.29:6835/3313937659] conn(0x56254955a880 0x56254952d1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56254953b550 0x56254957a920 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56254953b170 0x56254957a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 --2- 172.21.15.29:0/1426938708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x56254953a260 0x5625495a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99b5ffb640 0 client.10968 ms_handle_reset on v2:172.21.15.29:6834/3313937659 2023-10-16T15:53:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 >> 172.21.15.29:0/1426938708 conn(0x56254950e5b0 msgr2=0x56254955bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 shutdown_connections 2023-10-16T15:53:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 1 -- 172.21.15.29:0/1426938708 wait complete. 2023-10-16T15:53:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.051+0000 7f99dfda8740 20 client.10968 trim_cache size 0 max 16384 2023-10-16T15:53:34.066 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:53:34.066 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:53:34.066 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:53:34.066 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 10821, 2023-10-16T15:53:34.066 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:53:34.067 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:53:34.067 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T15:53:34.067 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8241027049999996, 2023-10-16T15:53:34.067 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 109, 2023-10-16T15:53:34.067 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 225, 2023-10-16T15:53:34.067 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 225, 2023-10-16T15:53:34.068 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.659057043000001 2023-10-16T15:53:34.068 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:53:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f99b806af90 msgr2=0x7f99b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f99b806af90 0x7f99b806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001750 tx=0x7f99c002f000 comp rx=0 tx=0).stop 2023-10-16T15:53:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 msgr2=0x7f99d81571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99d81571b0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f99c800e770 tx=0x7f99c800ec30 comp rx=0 tx=0).stop 2023-10-16T15:53:34.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 shutdown_connections 2023-10-16T15:53:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f99b806af90 0x7f99b806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99d80b0d90 0x7f99d815b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99d814baa0 0x7f99d80b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 --2- 172.21.15.29:0/753428361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c4003f20 0x7f99d81571b0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 >> 172.21.15.29:0/753428361 conn(0x7f99d80ce6a0 msgr2=0x7f99d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 shutdown_connections 2023-10-16T15:53:34.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.052+0000 7f99dca14640 1 -- 172.21.15.29:0/753428361 wait complete. 2023-10-16T15:53:34.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T15:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.293+0000 7fd69e939640 1 Processor -- start 2023-10-16T15:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.293+0000 7fd69e939640 1 -- start start 2023-10-16T15:53:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.294+0000 7fd69e939640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 0x7fd69806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.294+0000 7fd69e939640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd6980b0b20 0x7fd69806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.294+0000 7fd69e939640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69806f7f0 con 0x7fd698159f00 2023-10-16T15:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.294+0000 7fd69e939640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69806c5e0 con 0x7fd6980b0b20 2023-10-16T15:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.294+0000 7fd69e939640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69806c8c0 con 0x7fd698159b20 2023-10-16T15:53:34.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.295+0000 7fd697fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 0x7fd69806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd69ced6640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd6980b0b20 0x7fd69806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd697fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 0x7fd69806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:40064/0 (socket says 172.21.15.29:40064) 2023-10-16T15:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd697fff640 1 -- 172.21.15.29:0/2259236815 learned_addr learned my addr 172.21.15.29:0/2259236815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd697fff640 1 -- 172.21.15.29:0/2259236815 >> v2:172.21.15.170:3300/0 conn(0x7fd6980b0b20 msgr2=0x7fd69806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd697fff640 1 --2- 172.21.15.29:0/2259236815 >> v2:172.21.15.170:3300/0 conn(0x7fd6980b0b20 0x7fd69806fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd697fff640 1 -- 172.21.15.29:0/2259236815 >> v1:172.21.15.170:6789/0 conn(0x7fd698159f00 legacy=0x7fd6980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.296+0000 7fd697fff640 1 -- 172.21.15.29:0/2259236815 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69806cba0 con 0x7fd698159b20 2023-10-16T15:53:34.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.298+0000 7fd697fff640 1 --2- 172.21.15.29:0/2259236815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 0x7fd69806bbe0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd688009e50 tx=0x7fd68802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee04d49dc635d44f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.298+0000 7fd696ffd640 1 -- 172.21.15.29:0/2259236815 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68802cb00 con 0x7fd698159b20 2023-10-16T15:53:34.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.298+0000 7fd696ffd640 1 -- 172.21.15.29:0/2259236815 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68802cc60 con 0x7fd698159b20 2023-10-16T15:53:34.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.298+0000 7fd696ffd640 1 -- 172.21.15.29:0/2259236815 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6880343f0 con 0x7fd698159b20 2023-10-16T15:53:34.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 -- 172.21.15.29:0/2259236815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 msgr2=0x7fd69806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 --2- 172.21.15.29:0/2259236815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 0x7fd69806bbe0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd688009e50 tx=0x7fd68802bf60 comp rx=0 tx=0).stop 2023-10-16T15:53:34.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 -- 172.21.15.29:0/2259236815 shutdown_connections 2023-10-16T15:53:34.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 --2- 172.21.15.29:0/2259236815 >> v2:172.21.15.170:3300/0 conn(0x7fd6980b0b20 0x7fd69806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 --2- 172.21.15.29:0/2259236815 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd698159b20 0x7fd69806bbe0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 -- 172.21.15.29:0/2259236815 >> 172.21.15.29:0/2259236815 conn(0x7fd6980cb410 msgr2=0x7fd6980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 -- 172.21.15.29:0/2259236815 shutdown_connections 2023-10-16T15:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.299+0000 7fd69e939640 1 -- 172.21.15.29:0/2259236815 wait complete. 2023-10-16T15:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.301+0000 7fd69e939640 1 Processor -- start 2023-10-16T15:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.301+0000 7fd69e939640 1 -- start start 2023-10-16T15:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.302+0000 7fd69e939640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980b0b20 0x7fd6981509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.302+0000 7fd69e939640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 0x7fd6980c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.302+0000 7fd69e939640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd698159f00 0x7fd6980c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.302+0000 7fd69e939640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980c8990 con 0x7fd698159b20 2023-10-16T15:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.302+0000 7fd69e939640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980c6730 con 0x7fd698159f00 2023-10-16T15:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.302+0000 7fd69e939640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980b3d50 con 0x7fd6980b0b20 2023-10-16T15:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 0x7fd6980c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 0x7fd6980c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47092/0 (socket says 172.21.15.29:47092) 2023-10-16T15:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 -- 172.21.15.29:0/4154266677 learned_addr learned my addr 172.21.15.29:0/4154266677 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd69ced6640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd698159f00 0x7fd6980c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd697fff640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980b0b20 0x7fd6981509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 -- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980b0b20 msgr2=0x7fd6981509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980b0b20 0x7fd6981509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 -- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd698159f00 msgr2=0x7fd6980c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd698159f00 0x7fd6980c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 -- 172.21.15.29:0/4154266677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009b30 con 0x7fd698159b20 2023-10-16T15:53:34.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd69ced6640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd698159f00 0x7fd6980c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:34.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.303+0000 7fd6977fe640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 0x7fd6980c60a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00c400 tx=0x7fd68c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.304+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0052d0 con 0x7fd698159b20 2023-10-16T15:53:34.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.304+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c005430 con 0x7fd698159b20 2023-10-16T15:53:34.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.304+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c003830 con 0x7fd698159b20 2023-10-16T15:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.304+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6980b2f90 con 0x7fd698159b20 2023-10-16T15:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.304+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6980b3550 con 0x7fd698159b20 2023-10-16T15:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.305+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69804e160 con 0x7fd698159b20 2023-10-16T15:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.306+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd68c003990 con 0x7fd698159b20 2023-10-16T15:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.307+0000 7fd694ff9640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fd68406afc0 0x7fd68406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.307+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd68c088d70 con 0x7fd698159b20 2023-10-16T15:53:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.308+0000 7fd697fff640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fd68406afc0 0x7fd68406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:34.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.309+0000 7fd697fff640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fd68406afc0 0x7fd68406d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd688035000 tx=0x7fd688030000 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:34.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.312+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd68c053ce0 con 0x7fd698159b20 2023-10-16T15:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.545+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd6980c9610 con 0x7fd698159b20 2023-10-16T15:53:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.546+0000 7fd694ff9640 1 -- 172.21.15.29:0/4154266677 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v109) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd68c017060 con 0x7fd698159b20 2023-10-16T15:53:34.548 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T15:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fd68406afc0 msgr2=0x7fd68406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fd68406afc0 0x7fd68406d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd688035000 tx=0x7fd688030000 comp rx=0 tx=0).stop 2023-10-16T15:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 msgr2=0x7fd6980c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 0x7fd6980c60a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00c400 tx=0x7fd68c00c8c0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 shutdown_connections 2023-10-16T15:53:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fd68406afc0 0x7fd68406d480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd698159b20 0x7fd6980c60a0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd698159f00 0x7fd6980c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.548+0000 7fd69e939640 1 --2- 172.21.15.29:0/4154266677 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd6980b0b20 0x7fd6981509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.549+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 >> 172.21.15.29:0/4154266677 conn(0x7fd6980cb410 msgr2=0x7fd6980b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.549+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 shutdown_connections 2023-10-16T15:53:34.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:34.549+0000 7fd69e939640 1 -- 172.21.15.29:0/4154266677 wait complete. 2023-10-16T15:53:34.569 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T15:53:34.570 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T15:53:34.570 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T15:53:34.570 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T15:53:34.571 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T15:53:34.571 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T15:53:34.571 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T15:53:34.572 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T15:53:34.572 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:53:34.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:34.573 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T15:53:34.630 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T15:53:34.630 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T15:53:34.631 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T15:53:34.631 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.631 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T15:53:34.631 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.631 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:53:34.632 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:53:34.632 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T15:53:34.632 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:53:34.632 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:53:34.632 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T15:53:34.633 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T15:53:34.633 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T15:53:34.633 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T15:53:34.633 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T15:53:34.633 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36510sec preferred_lft 36510sec 2023-10-16T15:53:34.634 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T15:53:34.634 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.634 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:53:34.634 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T15:53:34.634 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T15:53:34.635 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T15:53:34.635 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:53:34.635 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T15:53:34.635 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.635 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T15:53:34.636 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.636 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:53:34.636 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:53:34.636 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T15:53:34.636 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.637 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:53:34.637 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:53:34.637 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T15:53:34.637 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:53:34.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:34.639 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T15:53:34.737 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T15:53:34.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:34.742 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T15:53:34.802 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T15:53:34.803 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T15:53:34.803 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T15:53:34.803 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T15:53:34.803 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T15:53:34.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:34.804 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T15:53:34.805 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T15:53:34.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T15:53:34.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T15:53:34.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T15:53:34.806 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T15:53:35.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:35.012 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:53:35.063 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:53:35.064 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:53:35.065 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:53:35.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:35.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T15:53:35.164 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:53:35.164 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T15:53:35.226 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:53:35.226 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:53:35.227 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:53:35.285 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T15:53:35.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:35.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:53:35.433 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:53:35.434 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:53:35.434 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:53:35.457 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:53:35.457 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:53:35.457 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:53:35.458 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:53:35.458 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:53:35.488 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:53:35.488 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:53:35.488 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:53:35.488 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:53:35.488 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:53:35.489 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T15:53:35.489 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T15:53:35.489 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T15:53:35.490 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T15:53:35.490 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T15:53:35.490 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T15:53:35.490 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T15:53:35.491 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T15:53:35.493 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T15:53:35.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:35.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:53:35.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:35.584 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T15:53:35.647 INFO:teuthology.orchestra.run.smithi029.stdout:client7877 2023-10-16T15:53:35.647 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:53:35.648 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:53:35.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T15:53:35.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.978+0000 7f42faa26640 1 Processor -- start 2023-10-16T15:53:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.979+0000 7f42faa26640 1 -- start start 2023-10-16T15:53:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.980+0000 7f42faa26640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42f40ae220 0x7f42f406e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.980+0000 7f42faa26640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f406adb0 0x7f42f406b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.980+0000 7f42faa26640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f406e6b0 con 0x7f42f40ce690 2023-10-16T15:53:35.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.980+0000 7f42faa26640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f406b780 con 0x7f42f40ae220 2023-10-16T15:53:35.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.980+0000 7f42faa26640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40cae30 con 0x7f42f406adb0 2023-10-16T15:53:35.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42f40ce690 0x7f42f40c9820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38358/0 (socket says 172.21.15.29:38358) 2023-10-16T15:53:35.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f3fff640 1 -- 172.21.15.29:0/786937150 learned_addr learned my addr 172.21.15.29:0/786937150 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768712480 0 0) 0x7f42f406e6b0 con 0x7f42f40ce690 2023-10-16T15:53:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e0003520 con 0x7f42f40ce690 2023-10-16T15:53:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f37fe640 1 --2- 172.21.15.29:0/786937150 >> v2:172.21.15.170:3300/0 conn(0x7f42f40ae220 0x7f42f406e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f406adb0 0x7f42f406b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1060168116 0 0) 0x7f42e0003520 con 0x7f42f40ce690 2023-10-16T15:53:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f406adb0 msgr2=0x7f42f406b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f406adb0 0x7f42f406b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 >> v2:172.21.15.170:3300/0 conn(0x7f42f40ae220 msgr2=0x7f42f406e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 --2- 172.21.15.29:0/786937150 >> v2:172.21.15.170:3300/0 conn(0x7f42f40ae220 0x7f42f406e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40cb110 con 0x7f42f40ce690 2023-10-16T15:53:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.981+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f42e40019f0 con 0x7f42f40ce690 2023-10-16T15:53:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 >> v1:172.21.15.29:6789/0 conn(0x7f42f40ce690 legacy=0x7f42f40c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42e00039e0 0x7f42e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e00047c0 0x7f42e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e0005180 0x7f42e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f406b780 con 0x7f42e0005180 2023-10-16T15:53:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40cae30 con 0x7f42e00047c0 2023-10-16T15:53:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e000c100 con 0x7f42e00039e0 2023-10-16T15:53:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e0005180 0x7f42e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f37fe640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e00047c0 0x7f42e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f3fff640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42e00039e0 0x7f42e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f8fc3640 1 -- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42e00039e0 msgr2=0x7f42e0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42e00039e0 0x7f42e0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f8fc3640 1 -- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e00047c0 msgr2=0x7f42e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e00047c0 0x7f42e0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.982+0000 7f42f8fc3640 1 -- 172.21.15.29:0/786937150 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0003520 con 0x7f42e0005180 2023-10-16T15:53:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e0005180 0x7f42e000b980 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f42f406b190 tx=0x7f42e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db6444e9ba4ee70 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec034610 con 0x7f42e0005180 2023-10-16T15:53:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec02e070 con 0x7f42e0005180 2023-10-16T15:53:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42f2ffd640 1 -- 172.21.15.29:0/786937150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec034990 con 0x7f42e0005180 2023-10-16T15:53:35.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42faa26640 1 -- 172.21.15.29:0/786937150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e0005180 msgr2=0x7f42e000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:35.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42faa26640 1 --2- 172.21.15.29:0/786937150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e0005180 0x7f42e000b980 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f42f406b190 tx=0x7f42e00057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.984+0000 7f42f8fc3640 1 -- 172.21.15.29:0/786937150 reap_dead start 2023-10-16T15:53:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.985+0000 7f42faa26640 1 -- 172.21.15.29:0/786937150 shutdown_connections 2023-10-16T15:53:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.985+0000 7f42faa26640 1 -- 172.21.15.29:0/786937150 >> 172.21.15.29:0/786937150 conn(0x7f42f40b0cc0 msgr2=0x7f42f40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.985+0000 7f42faa26640 1 -- 172.21.15.29:0/786937150 shutdown_connections 2023-10-16T15:53:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.985+0000 7f42faa26640 1 -- 172.21.15.29:0/786937150 wait complete. 2023-10-16T15:53:35.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.987+0000 7f42faa26640 1 Processor -- start 2023-10-16T15:53:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.987+0000 7f42faa26640 1 -- start start 2023-10-16T15:53:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42faa26640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e0005180 0x7f42f415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42faa26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 0x7f42f409ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42faa26640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f40b29a0 0x7f42f40b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42faa26640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40b3370 con 0x7f42f415fd60 2023-10-16T15:53:35.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42faa26640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f409e730 con 0x7f42e0005180 2023-10-16T15:53:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42faa26640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f409ea10 con 0x7f42f40b29a0 2023-10-16T15:53:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 0x7f42f409ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 0x7f42f409ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47112/0 (socket says 172.21.15.29:47112) 2023-10-16T15:53:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 -- 172.21.15.29:0/417721394 learned_addr learned my addr 172.21.15.29:0/417721394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:35.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f40b29a0 0x7f42f40b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f3fff640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e0005180 0x7f42f415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 -- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f40b29a0 msgr2=0x7f42f40b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f40b29a0 0x7f42f40b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 -- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e0005180 msgr2=0x7f42f415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:35.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e0005180 0x7f42f415c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f37fe640 1 -- 172.21.15.29:0/417721394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f409ecf0 con 0x7f42f415fd60 2023-10-16T15:53:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f8fc3640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f40b29a0 0x7f42f40b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.988+0000 7f42f3fff640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e0005180 0x7f42f415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.989+0000 7f42f37fe640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 0x7f42f409ddc0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00f5d0 tx=0x7f42dc00fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.989+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc014870 con 0x7f42f415fd60 2023-10-16T15:53:35.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.989+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc0149d0 con 0x7f42f415fd60 2023-10-16T15:53:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.989+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f409efd0 con 0x7f42f415fd60 2023-10-16T15:53:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.989+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc014d40 con 0x7f42f415fd60 2023-10-16T15:53:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.989+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f409e300 con 0x7f42f415fd60 2023-10-16T15:53:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.990+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f404e160 con 0x7f42f415fd60 2023-10-16T15:53:36.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.992+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42dc00d460 con 0x7f42f415fd60 2023-10-16T15:53:36.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.993+0000 7f42f0ff9640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f42d406af90 0x7f42d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.993+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f42dc08a230 con 0x7f42f415fd60 2023-10-16T15:53:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.994+0000 7f42f3fff640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f42d406af90 0x7f42d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.995+0000 7f42f3fff640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f42d406af90 0x7f42d406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f42e4000f80 tx=0x7f42e4001310 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:36.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:35.998+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42dc0551b0 con 0x7f42f415fd60 2023-10-16T15:53:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.233+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f42f4099a80 con 0x7f42f415fd60 2023-10-16T15:53:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.301+0000 7f42f0ff9640 1 -- 172.21.15.29:0/417721394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v110) v1 ==== 136+0+0 (secure 0 0 0) 0x7f42dc0587d0 con 0x7f42f415fd60 2023-10-16T15:53:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.305+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f42d406af90 msgr2=0x7f42d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.305+0000 7f42faa26640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f42d406af90 0x7f42d406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f42e4000f80 tx=0x7f42e4001310 comp rx=0 tx=0).stop 2023-10-16T15:53:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.305+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 msgr2=0x7f42f409ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.305+0000 7f42faa26640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 0x7f42f409ddc0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00f5d0 tx=0x7f42dc00fa90 comp rx=0 tx=0).stop 2023-10-16T15:53:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 shutdown_connections 2023-10-16T15:53:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f42d406af90 0x7f42d406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f415fd60 0x7f42f409ddc0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f40b29a0 0x7f42f40b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 --2- 172.21.15.29:0/417721394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42e0005180 0x7f42f415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 >> 172.21.15.29:0/417721394 conn(0x7f42f40b0cc0 msgr2=0x7f42f40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 shutdown_connections 2023-10-16T15:53:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.306+0000 7f42faa26640 1 -- 172.21.15.29:0/417721394 wait complete. 2023-10-16T15:53:36.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.546+0000 7fe47dc4d640 1 Processor -- start 2023-10-16T15:53:36.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.546+0000 7fe47dc4d640 1 -- start start 2023-10-16T15:53:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.547+0000 7fe47dc4d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.547+0000 7fe47dc4d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe478070450 0x7fe4780708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.547+0000 7fe47dc4d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe478070df0 con 0x7fe47806c100 2023-10-16T15:53:36.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.547+0000 7fe47dc4d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47815e490 con 0x7fe478070450 2023-10-16T15:53:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.547+0000 7fe47dc4d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47815aef0 con 0x7fe478159950 2023-10-16T15:53:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe478070450 0x7fe4780708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe478070450 0x7fe4780708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47128/0 (socket says 172.21.15.29:47128) 2023-10-16T15:53:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 -- 172.21.15.29:0/2329962415 learned_addr learned my addr 172.21.15.29:0/2329962415 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:36.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 -- 172.21.15.29:0/2329962415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478159950 msgr2=0x7fe47815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe47cc4b640 1 --2- 172.21.15.29:0/2329962415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 --2- 172.21.15.29:0/2329962415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 -- 172.21.15.29:0/2329962415 >> v1:172.21.15.170:6789/0 conn(0x7fe47806c100 legacy=0x7fe47806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:36.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.548+0000 7fe477fff640 1 -- 172.21.15.29:0/2329962415 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47806cc80 con 0x7fe478070450 2023-10-16T15:53:36.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.549+0000 7fe477fff640 1 --2- 172.21.15.29:0/2329962415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478070450 0x7fe4780708b0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fe464009870 tx=0x7fe46402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e41d70c7014f665 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe4777fe640 1 -- 172.21.15.29:0/2329962415 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640317c0 con 0x7fe478070450 2023-10-16T15:53:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe4777fe640 1 -- 172.21.15.29:0/2329962415 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464031920 con 0x7fe478070450 2023-10-16T15:53:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe4777fe640 1 -- 172.21.15.29:0/2329962415 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464031cb0 con 0x7fe478070450 2023-10-16T15:53:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe47dc4d640 1 -- 172.21.15.29:0/2329962415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478070450 msgr2=0x7fe4780708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/2329962415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478070450 0x7fe4780708b0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fe464009870 tx=0x7fe46402b860 comp rx=0 tx=0).stop 2023-10-16T15:53:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe47dc4d640 1 -- 172.21.15.29:0/2329962415 shutdown_connections 2023-10-16T15:53:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/2329962415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478070450 0x7fe4780708b0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/2329962415 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.550+0000 7fe47dc4d640 1 -- 172.21.15.29:0/2329962415 >> 172.21.15.29:0/2329962415 conn(0x7fe4780ce6a0 msgr2=0x7fe4780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:36.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.551+0000 7fe47dc4d640 1 -- 172.21.15.29:0/2329962415 shutdown_connections 2023-10-16T15:53:36.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.551+0000 7fe47dc4d640 1 -- 172.21.15.29:0/2329962415 wait complete. 2023-10-16T15:53:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.552+0000 7fe47dc4d640 1 Processor -- start 2023-10-16T15:53:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.553+0000 7fe47dc4d640 1 -- start start 2023-10-16T15:53:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47dc4d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe47806c100 0x7fe4781f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47dc4d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478070450 0x7fe4781f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47dc4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe4780c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47dc4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781f3380 con 0x7fe478159950 2023-10-16T15:53:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47dc4d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47815b1c0 con 0x7fe47806c100 2023-10-16T15:53:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47dc4d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781f5f50 con 0x7fe478070450 2023-10-16T15:53:36.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47d44c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe4780c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47d44c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe4780c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47144/0 (socket says 172.21.15.29:47144) 2023-10-16T15:53:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47d44c640 1 -- 172.21.15.29:0/3816469453 learned_addr learned my addr 172.21.15.29:0/3816469453 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47cc4b640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe47806c100 0x7fe4781f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe477fff640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478070450 0x7fe4781f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47d44c640 1 -- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478070450 msgr2=0x7fe4781f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47d44c640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478070450 0x7fe4781f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.554+0000 7fe47d44c640 1 -- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe47806c100 msgr2=0x7fe4781f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe47d44c640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe47806c100 0x7fe4781f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe47d44c640 1 -- 172.21.15.29:0/3816469453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640094d0 con 0x7fe478159950 2023-10-16T15:53:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe477fff640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478070450 0x7fe4781f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:36.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe47d44c640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe4780c21b0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fe47000b9a0 tx=0x7fe47000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47000cba0 con 0x7fe478159950 2023-10-16T15:53:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47000cd00 con 0x7fe478159950 2023-10-16T15:53:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470014680 con 0x7fe478159950 2023-10-16T15:53:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4781f6230 con 0x7fe478159950 2023-10-16T15:53:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.555+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4781f21f0 con 0x7fe478159950 2023-10-16T15:53:36.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.557+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47804e9b0 con 0x7fe478159950 2023-10-16T15:53:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.557+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe47001d3b0 con 0x7fe478159950 2023-10-16T15:53:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.558+0000 7fe4757fa640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe45c06afc0 0x7fe45c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.559+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe470089550 con 0x7fe478159950 2023-10-16T15:53:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.559+0000 7fe47cc4b640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe45c06afc0 0x7fe45c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.560+0000 7fe47cc4b640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe45c06afc0 0x7fe45c06d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4781f58f0 tx=0x7fe46c008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.564+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4700544d0 con 0x7fe478159950 2023-10-16T15:53:36.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.760+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4780b4620 con 0x7fe478159950 2023-10-16T15:53:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.763+0000 7fe4757fa640 1 -- 172.21.15.29:0/3816469453 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+16627 (secure 0 0 0) 0x7fe470057af0 con 0x7fe478159950 2023-10-16T15:53:36.765 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:36.765 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":225,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:28.224740+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe45c06afc0 msgr2=0x7fe45c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe45c06afc0 0x7fe45c06d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4781f58f0 tx=0x7fe46c008040 comp rx=0 tx=0).stop 2023-10-16T15:53:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 msgr2=0x7fe4780c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:36.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe4780c21b0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fe47000b9a0 tx=0x7fe47000be60 comp rx=0 tx=0).stop 2023-10-16T15:53:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 shutdown_connections 2023-10-16T15:53:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe45c06afc0 0x7fe45c06d480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe4780c21b0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe478070450 0x7fe4781f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 --2- 172.21.15.29:0/3816469453 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe47806c100 0x7fe4781f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 >> 172.21.15.29:0/3816469453 conn(0x7fe4780ce6a0 msgr2=0x7fe478072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 shutdown_connections 2023-10-16T15:53:36.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:36.765+0000 7fe47dc4d640 1 -- 172.21.15.29:0/3816469453 wait complete. 2023-10-16T15:53:36.787 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:53:24.608039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:53:25.694889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:53:36.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:53:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.019+0000 7f59a5e1a640 1 Processor -- start 2023-10-16T15:53:37.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.019+0000 7f59a5e1a640 1 -- start start 2023-10-16T15:53:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.020+0000 7f59a5e1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f59a0159ad0 0x7f59a0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.020+0000 7f59a5e1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a00cd260 con 0x7f59a00c9de0 2023-10-16T15:53:37.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.020+0000 7f59a5e1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a015de30 con 0x7f59a015d650 2023-10-16T15:53:37.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.020+0000 7f59a5e1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a0071470 con 0x7f59a0159ad0 2023-10-16T15:53:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.021+0000 7f59a5619640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f59a00c9de0 0x7f59a00cd9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38374/0 (socket says 172.21.15.29:38374) 2023-10-16T15:53:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.021+0000 7f59a5619640 1 -- 172.21.15.29:0/3291133515 learned_addr learned my addr 172.21.15.29:0/3291133515 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.021+0000 7f59a4e18640 1 --2- 172.21.15.29:0/3291133515 >> v2:172.21.15.170:3300/0 conn(0x7f59a0159ad0 0x7f59a0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.021+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337349126 0 0) 0x7f59a00cd260 con 0x7f59a00c9de0 2023-10-16T15:53:37.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.021+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f598c003520 con 0x7f59a00c9de0 2023-10-16T15:53:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512022964 0 0) 0x7f598c003520 con 0x7f59a00c9de0 2023-10-16T15:53:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 >> v2:172.21.15.170:3300/0 conn(0x7f59a0159ad0 msgr2=0x7f59a0070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 --2- 172.21.15.29:0/3291133515 >> v2:172.21.15.170:3300/0 conn(0x7f59a0159ad0 0x7f59a0070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 >> v1:172.21.15.170:6789/0 conn(0x7f59a015d650 legacy=0x7f59a006aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a015e110 con 0x7f59a00c9de0 2023-10-16T15:53:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5998001670 con 0x7f59a00c9de0 2023-10-16T15:53:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 >> v1:172.21.15.29:6789/0 conn(0x7f59a00c9de0 legacy=0x7f59a00cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f598c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f598c0047c0 0x7f598c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598c005180 0x7f598c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0071470 con 0x7f598c003970 2023-10-16T15:53:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00c000 con 0x7f598c005180 2023-10-16T15:53:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00e3d0 con 0x7f598c0047c0 2023-10-16T15:53:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.022+0000 7f59a4e18640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f598c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f599ffff640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f598c0047c0 0x7f598c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f59a4e18640 1 -- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f598c0047c0 msgr2=0x7f598c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f59a4e18640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f598c0047c0 0x7f598c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f59a4e18640 1 -- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598c005180 msgr2=0x7f598c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f59a4e18640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f598c005180 0x7f598c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f59a4e18640 1 -- 172.21.15.29:0/3291133515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c003520 con 0x7f598c003970 2023-10-16T15:53:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.023+0000 7f59a5619640 1 -- 172.21.15.29:0/3291133515 reap_dead start 2023-10-16T15:53:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.024+0000 7f59a4e18640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f598c003d70 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f5990009fc0 tx=0x7f598c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff022916d1aa7d5c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.024+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599002cc50 con 0x7f598c003970 2023-10-16T15:53:37.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.024+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599000f040 con 0x7f598c003970 2023-10-16T15:53:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.024+0000 7f599f7fe640 1 -- 172.21.15.29:0/3291133515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990034440 con 0x7f598c003970 2023-10-16T15:53:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 -- 172.21.15.29:0/3291133515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 msgr2=0x7f598c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f598c003d70 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f5990009fc0 tx=0x7f598c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 -- 172.21.15.29:0/3291133515 shutdown_connections 2023-10-16T15:53:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/3291133515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f598c003d70 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 -- 172.21.15.29:0/3291133515 >> 172.21.15.29:0/3291133515 conn(0x7f59a00b0170 msgr2=0x7f59a00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 -- 172.21.15.29:0/3291133515 shutdown_connections 2023-10-16T15:53:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.025+0000 7f59a5e1a640 1 -- 172.21.15.29:0/3291133515 wait complete. 2023-10-16T15:53:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.027+0000 7f59a5e1a640 1 Processor -- start 2023-10-16T15:53:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.027+0000 7f59a5e1a640 1 -- start start 2023-10-16T15:53:37.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a5e1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f59a00af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a5e1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59a015c720 0x7f59a0152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a5e1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0151b50 0x7f59a015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a5e1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00c7250 con 0x7f598c003970 2023-10-16T15:53:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a5e1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00c6960 con 0x7f59a0151b50 2023-10-16T15:53:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a5e1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a015cc00 con 0x7f59a015c720 2023-10-16T15:53:37.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a4e18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f59a00af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a4e18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f59a00af230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47154/0 (socket says 172.21.15.29:47154) 2023-10-16T15:53:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.028+0000 7f59a4e18640 1 -- 172.21.15.29:0/63819967 learned_addr learned my addr 172.21.15.29:0/63819967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f599ffff640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59a015c720 0x7f59a0152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a5619640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0151b50 0x7f59a015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a4e18640 1 -- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59a015c720 msgr2=0x7f59a0152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a4e18640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59a015c720 0x7f59a0152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a4e18640 1 -- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0151b50 msgr2=0x7f59a015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a4e18640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0151b50 0x7f59a015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a4e18640 1 -- 172.21.15.29:0/63819967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a015cee0 con 0x7f598c003970 2023-10-16T15:53:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a5619640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0151b50 0x7f59a015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:37.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.029+0000 7f59a4e18640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f59a00af230 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f599000e230 tx=0x7f5990009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.030+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990034c50 con 0x7f598c003970 2023-10-16T15:53:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.030+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5990034db0 con 0x7f598c003970 2023-10-16T15:53:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.030+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990003b90 con 0x7f598c003970 2023-10-16T15:53:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.030+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a00c6c00 con 0x7f598c003970 2023-10-16T15:53:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.030+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a015b330 con 0x7f598c003970 2023-10-16T15:53:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.031+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a004e9b0 con 0x7f598c003970 2023-10-16T15:53:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.032+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5990040450 con 0x7f598c003970 2023-10-16T15:53:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.033+0000 7f599d7fa640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f598006afc0 0x7f598006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.033+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f59900abef0 con 0x7f598c003970 2023-10-16T15:53:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.034+0000 7f599ffff640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f598006afc0 0x7f598006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.035+0000 7f599ffff640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f598006afc0 0x7f598006d480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59a0156080 tx=0x7f5988009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.037+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5990076dd0 con 0x7f598c003970 2023-10-16T15:53:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.229+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59a00b4600 con 0x7f598c003970 2023-10-16T15:53:37.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.230+0000 7f599d7fa640 1 -- 172.21.15.29:0/63819967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v225) v1 ==== 93+0+10 (secure 0 0 0) 0x7f599007a3f0 con 0x7f598c003970 2023-10-16T15:53:37.233 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:53:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f598006afc0 msgr2=0x7f598006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f598006afc0 0x7f598006d480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59a0156080 tx=0x7f5988009040 comp rx=0 tx=0).stop 2023-10-16T15:53:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 msgr2=0x7f59a00af230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f59a00af230 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f599000e230 tx=0x7f5990009750 comp rx=0 tx=0).stop 2023-10-16T15:53:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 shutdown_connections 2023-10-16T15:53:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f598006afc0 0x7f598006d480 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0151b50 0x7f59a015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f59a015c720 0x7f59a0152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 --2- 172.21.15.29:0/63819967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f598c003970 0x7f59a00af230 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 >> 172.21.15.29:0/63819967 conn(0x7f59a00b0170 msgr2=0x7f59a00cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.233+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 shutdown_connections 2023-10-16T15:53:37.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.234+0000 7f59a5e1a640 1 -- 172.21.15.29:0/63819967 wait complete. 2023-10-16T15:53:37.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:53:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.485+0000 7f240d58e640 1 Processor -- start 2023-10-16T15:53:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.485+0000 7f240d58e640 1 -- start start 2023-10-16T15:53:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.487+0000 7f240d58e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.487+0000 7f240d58e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f24080c9230 0x7f24080709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.487+0000 7f240d58e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2408160a90 con 0x7f2408070f20 2023-10-16T15:53:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.487+0000 7f240d58e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24080cdec0 con 0x7f24080c9230 2023-10-16T15:53:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.487+0000 7f240d58e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24080cd900 con 0x7f240815c9f0 2023-10-16T15:53:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.488+0000 7f24077fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f24080c9230 0x7f24080709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.488+0000 7f2407fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.488+0000 7f24077fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f24080c9230 0x7f24080709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:33984/0 (socket says 172.21.15.29:33984) 2023-10-16T15:53:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.488+0000 7f24077fe640 1 -- 172.21.15.29:0/1833673430 learned_addr learned my addr 172.21.15.29:0/1833673430 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.489+0000 7f2407fff640 1 -- 172.21.15.29:0/1833673430 >> v2:172.21.15.170:3300/0 conn(0x7f24080c9230 msgr2=0x7f24080709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.489+0000 7f2407fff640 1 --2- 172.21.15.29:0/1833673430 >> v2:172.21.15.170:3300/0 conn(0x7f24080c9230 0x7f24080709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.489+0000 7f2407fff640 1 -- 172.21.15.29:0/1833673430 >> v1:172.21.15.170:6789/0 conn(0x7f2408070f20 legacy=0x7f24080ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.489+0000 7f2407fff640 1 -- 172.21.15.29:0/1833673430 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24080cdbe0 con 0x7f240815c9f0 2023-10-16T15:53:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.490+0000 7f2407fff640 1 --2- 172.21.15.29:0/1833673430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c8cf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009d90 tx=0x7f23f802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fa474d62d54a026 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.491+0000 7f2406ffd640 1 -- 172.21.15.29:0/1833673430 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f802cc50 con 0x7f240815c9f0 2023-10-16T15:53:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.491+0000 7f2406ffd640 1 -- 172.21.15.29:0/1833673430 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f802cdb0 con 0x7f240815c9f0 2023-10-16T15:53:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.491+0000 7f2406ffd640 1 -- 172.21.15.29:0/1833673430 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8034660 con 0x7f240815c9f0 2023-10-16T15:53:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 -- 172.21.15.29:0/1833673430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 msgr2=0x7f24080c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 --2- 172.21.15.29:0/1833673430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c8cf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009d90 tx=0x7f23f802bea0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 -- 172.21.15.29:0/1833673430 shutdown_connections 2023-10-16T15:53:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 --2- 172.21.15.29:0/1833673430 >> v2:172.21.15.170:3300/0 conn(0x7f24080c9230 0x7f24080709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 --2- 172.21.15.29:0/1833673430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c8cf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 -- 172.21.15.29:0/1833673430 >> 172.21.15.29:0/1833673430 conn(0x7f24080ce6a0 msgr2=0x7f24080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 -- 172.21.15.29:0/1833673430 shutdown_connections 2023-10-16T15:53:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.492+0000 7f240d58e640 1 -- 172.21.15.29:0/1833673430 wait complete. 2023-10-16T15:53:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.494+0000 7f240d58e640 1 Processor -- start 2023-10-16T15:53:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.494+0000 7f240d58e640 1 -- start start 2023-10-16T15:53:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.495+0000 7f240d58e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 0x7f24080c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.495+0000 7f240d58e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24080c9230 0x7f2408159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.495+0000 7f240d58e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.495+0000 7f240d58e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24080c5180 con 0x7f2408070f20 2023-10-16T15:53:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.495+0000 7f240d58e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2408159ab0 con 0x7f24080c9230 2023-10-16T15:53:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.495+0000 7f240d58e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240815eac0 con 0x7f240815c9f0 2023-10-16T15:53:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 0x7f24080c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 0x7f24080c4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47168/0 (socket says 172.21.15.29:47168) 2023-10-16T15:53:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 -- 172.21.15.29:0/3406734674 learned_addr learned my addr 172.21.15.29:0/3406734674 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f24077fe640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24080c9230 0x7f2408159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f240cd8d640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 -- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 msgr2=0x7f24080c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 -- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24080c9230 msgr2=0x7f2408159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24080c9230 0x7f2408159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.496+0000 7f2407fff640 1 -- 172.21.15.29:0/3406734674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009a70 con 0x7f2408070f20 2023-10-16T15:53:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.497+0000 7f2407fff640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 0x7f24080c4c40 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009d90 tx=0x7f23f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.497+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f803f420 con 0x7f2408070f20 2023-10-16T15:53:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.497+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f803f580 con 0x7f2408070f20 2023-10-16T15:53:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.497+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240815eda0 con 0x7f2408070f20 2023-10-16T15:53:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.497+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f803f8f0 con 0x7f2408070f20 2023-10-16T15:53:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.497+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240815f360 con 0x7f2408070f20 2023-10-16T15:53:37.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.498+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240804e9b0 con 0x7f2408070f20 2023-10-16T15:53:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.499+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f23f803fa50 con 0x7f2408070f20 2023-10-16T15:53:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.500+0000 7f2404ff9640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f23f406af90 0x7f23f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.500+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f23f80ac350 con 0x7f2408070f20 2023-10-16T15:53:37.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.501+0000 7f24077fe640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f23f406af90 0x7f23f406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.502+0000 7f24077fe640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f23f406af90 0x7f23f406d450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f240806ce30 tx=0x7f23fc008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.506+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23f8077360 con 0x7f2408070f20 2023-10-16T15:53:37.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.703+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24080c8a60 con 0x7f2408070f20 2023-10-16T15:53:37.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.705+0000 7f2404ff9640 1 -- 172.21.15.29:0/3406734674 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v225) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23f804d020 con 0x7f2408070f20 2023-10-16T15:53:37.707 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:53:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.707+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f23f406af90 msgr2=0x7f23f406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.707+0000 7f240d58e640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f23f406af90 0x7f23f406d450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f240806ce30 tx=0x7f23fc008040 comp rx=0 tx=0).stop 2023-10-16T15:53:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 msgr2=0x7f24080c4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 0x7f24080c4c40 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009d90 tx=0x7f23f8004a40 comp rx=0 tx=0).stop 2023-10-16T15:53:37.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 shutdown_connections 2023-10-16T15:53:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f23f406af90 0x7f23f406d450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f240815c9f0 0x7f24080c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f24080c9230 0x7f2408159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 --2- 172.21.15.29:0/3406734674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2408070f20 0x7f24080c4c40 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 >> 172.21.15.29:0/3406734674 conn(0x7f24080ce6a0 msgr2=0x7f24080ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:37.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 shutdown_connections 2023-10-16T15:53:37.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.708+0000 7f240d58e640 1 -- 172.21.15.29:0/3406734674 wait complete. 2023-10-16T15:53:37.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:53:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.940+0000 7f1d5a0d7640 1 Processor -- start 2023-10-16T15:53:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.940+0000 7f1d5a0d7640 1 -- start start 2023-10-16T15:53:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d5a0d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 0x7f1d5415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d5a0d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1d54070450 0x7f1d540708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d5a0d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d54070df0 con 0x7f1d5406c100 2023-10-16T15:53:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d5a0d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d5415e490 con 0x7f1d54070450 2023-10-16T15:53:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d5a0d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5415aef0 con 0x7f1d54159950 2023-10-16T15:53:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d598d6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d5406c100 0x7f1d5406c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38380/0 (socket says 172.21.15.29:38380) 2023-10-16T15:53:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.942+0000 7f1d598d6640 1 -- 172.21.15.29:0/559962620 learned_addr learned my addr 172.21.15.29:0/559962620 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d590d5640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 0x7f1d5415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d588d4640 1 --2- 172.21.15.29:0/559962620 >> v2:172.21.15.170:3300/0 conn(0x7f1d54070450 0x7f1d540708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703280426 0 0) 0x7f1d54070df0 con 0x7f1d5406c100 2023-10-16T15:53:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40005730 con 0x7f1d5406c100 2023-10-16T15:53:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082326873 0 0) 0x7f1d40005730 con 0x7f1d5406c100 2023-10-16T15:53:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 msgr2=0x7f1d5415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 0x7f1d5415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 >> v2:172.21.15.170:3300/0 conn(0x7f1d54070450 msgr2=0x7f1d540708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 --2- 172.21.15.29:0/559962620 >> v2:172.21.15.170:3300/0 conn(0x7f1d54070450 0x7f1d540708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5406cc80 con 0x7f1d5406c100 2023-10-16T15:53:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1d4c0019f0 con 0x7f1d5406c100 2023-10-16T15:53:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 >> v1:172.21.15.29:6789/0 conn(0x7f1d5406c100 legacy=0x7f1d5406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.943+0000 7f1d47fff640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d40005b80 0x7f1d40005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d47fff640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d400069d0 0x7f1d40006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d47fff640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 0x7f1d40007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5415e490 con 0x7f1d400069d0 2023-10-16T15:53:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5415aef0 con 0x7f1d40005b80 2023-10-16T15:53:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4000dc90 con 0x7f1d54159950 2023-10-16T15:53:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d588d4640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d400069d0 0x7f1d40006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d598d6640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 0x7f1d40007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d590d5640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d40005b80 0x7f1d40005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d588d4640 1 -- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 msgr2=0x7f1d40007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d588d4640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d54159950 0x7f1d40007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d588d4640 1 -- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d40005b80 msgr2=0x7f1d40005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d588d4640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d40005b80 0x7f1d40005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.944+0000 7f1d588d4640 1 -- 172.21.15.29:0/559962620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40005730 con 0x7f1d400069d0 2023-10-16T15:53:37.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.945+0000 7f1d588d4640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d400069d0 0x7f1d40006e50 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800ba30 tx=0x7f1d40007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53943f9805668644 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48032620 con 0x7f1d400069d0 2023-10-16T15:53:37.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d48032780 con 0x7f1d400069d0 2023-10-16T15:53:37.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d47fff640 1 -- 172.21.15.29:0/559962620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48032b10 con 0x7f1d400069d0 2023-10-16T15:53:37.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/559962620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d400069d0 msgr2=0x7f1d40006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/559962620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d400069d0 0x7f1d40006e50 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800ba30 tx=0x7f1d40007a00 comp rx=0 tx=0).stop 2023-10-16T15:53:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d598d6640 1 -- 172.21.15.29:0/559962620 reap_dead start 2023-10-16T15:53:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/559962620 shutdown_connections 2023-10-16T15:53:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.946+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/559962620 >> 172.21.15.29:0/559962620 conn(0x7f1d540ce6a0 msgr2=0x7f1d540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.947+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/559962620 shutdown_connections 2023-10-16T15:53:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.947+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/559962620 wait complete. 2023-10-16T15:53:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.948+0000 7f1d5a0d7640 1 Processor -- start 2023-10-16T15:53:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.949+0000 7f1d5a0d7640 1 -- start start 2023-10-16T15:53:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d5a0d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d400069d0 0x7f1d541f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d5a0d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 0x7f1d541577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d5a0d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d541f1710 0x7f1d541f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d5a0d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5415c0e0 con 0x7f1d54155b10 2023-10-16T15:53:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d5a0d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d540b13d0 con 0x7f1d541f1710 2023-10-16T15:53:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d5a0d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5415c8b0 con 0x7f1d400069d0 2023-10-16T15:53:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d588d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 0x7f1d541577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d588d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 0x7f1d541577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47186/0 (socket says 172.21.15.29:47186) 2023-10-16T15:53:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d588d4640 1 -- 172.21.15.29:0/2842839821 learned_addr learned my addr 172.21.15.29:0/2842839821 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d590d5640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d400069d0 0x7f1d541f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.950+0000 7f1d598d6640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d541f1710 0x7f1d541f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d588d4640 1 -- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d400069d0 msgr2=0x7f1d541f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d588d4640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d400069d0 0x7f1d541f6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d588d4640 1 -- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d541f1710 msgr2=0x7f1d541f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d588d4640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d541f1710 0x7f1d541f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d588d4640 1 -- 172.21.15.29:0/2842839821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d541f67b0 con 0x7f1d54155b10 2023-10-16T15:53:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d590d5640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d400069d0 0x7f1d541f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d598d6640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d541f1710 0x7f1d541f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d588d4640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 0x7f1d541577c0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f1d4802bea0 tx=0x7f1d48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48004020 con 0x7f1d54155b10 2023-10-16T15:53:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d480041a0 con 0x7f1d54155b10 2023-10-16T15:53:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d541f2100 con 0x7f1d54155b10 2023-10-16T15:53:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48003610 con 0x7f1d54155b10 2023-10-16T15:53:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.951+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5415c5d0 con 0x7f1d54155b10 2023-10-16T15:53:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.952+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5404e9b0 con 0x7f1d54155b10 2023-10-16T15:53:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.953+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1d48003770 con 0x7f1d54155b10 2023-10-16T15:53:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.954+0000 7f1d45ffb640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f1d3806af90 0x7f1d3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.954+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1d480ab870 con 0x7f1d54155b10 2023-10-16T15:53:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.955+0000 7f1d590d5640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f1d3806af90 0x7f1d3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.956+0000 7f1d590d5640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f1d3806af90 0x7f1d3806d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009910 tx=0x7f1d50008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:37.959+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1d48076760 con 0x7f1d54155b10 2023-10-16T15:53:38.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.150+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d540b4620 con 0x7f1d54155b10 2023-10-16T15:53:38.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.151+0000 7f1d45ffb640 1 -- 172.21.15.29:0/2842839821 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v225) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d48079d80 con 0x7f1d54155b10 2023-10-16T15:53:38.153 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:53:38.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f1d3806af90 msgr2=0x7f1d3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f1d3806af90 0x7f1d3806d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009910 tx=0x7f1d50008040 comp rx=0 tx=0).stop 2023-10-16T15:53:38.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 msgr2=0x7f1d541577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 0x7f1d541577c0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f1d4802bea0 tx=0x7f1d48004730 comp rx=0 tx=0).stop 2023-10-16T15:53:38.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 shutdown_connections 2023-10-16T15:53:38.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f1d3806af90 0x7f1d3806d450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d54155b10 0x7f1d541577c0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1d541f1710 0x7f1d541f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 --2- 172.21.15.29:0/2842839821 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1d400069d0 0x7f1d541f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.153+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 >> 172.21.15.29:0/2842839821 conn(0x7f1d540ce6a0 msgr2=0x7f1d540c9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:38.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.154+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 shutdown_connections 2023-10-16T15:53:38.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.154+0000 7f1d5a0d7640 1 -- 172.21.15.29:0/2842839821 wait complete. 2023-10-16T15:53:38.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:53:38.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.400+0000 7fdd11d4e640 1 Processor -- start 2023-10-16T15:53:38.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.400+0000 7fdd11d4e640 1 -- start start 2023-10-16T15:53:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd11d4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd0c0ad970 0x7fdd0c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd11d4e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdd0c0ccfb0 0x7fdd0c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd11d4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0c15b9d0 con 0x7fdd0c06c040 2023-10-16T15:53:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd11d4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0c15bcb0 con 0x7fdd0c0ad970 2023-10-16T15:53:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd11d4e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0c15bf90 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd0bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd0c0ad970 0x7fdd0c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd0bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd0c0ad970 0x7fdd0c0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47188/0 (socket says 172.21.15.29:47188) 2023-10-16T15:53:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.402+0000 7fdd0bfff640 1 -- 172.21.15.29:0/2855515875 learned_addr learned my addr 172.21.15.29:0/2855515875 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.403+0000 7fdd1154d640 1 --2- 172.21.15.29:0/2855515875 >> v2:172.21.15.170:3300/0 conn(0x7fdd0c0ccfb0 0x7fdd0c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.403+0000 7fdd0bfff640 1 -- 172.21.15.29:0/2855515875 >> v2:172.21.15.170:3300/0 conn(0x7fdd0c0ccfb0 msgr2=0x7fdd0c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.403+0000 7fdd0bfff640 1 --2- 172.21.15.29:0/2855515875 >> v2:172.21.15.170:3300/0 conn(0x7fdd0c0ccfb0 0x7fdd0c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.403+0000 7fdd0bfff640 1 -- 172.21.15.29:0/2855515875 >> v1:172.21.15.170:6789/0 conn(0x7fdd0c06c040 legacy=0x7fdd0c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.403+0000 7fdd0bfff640 1 -- 172.21.15.29:0/2855515875 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c0cc270 con 0x7fdd0c0ad970 2023-10-16T15:53:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.403+0000 7fdd1154d640 1 --2- 172.21.15.29:0/2855515875 >> v2:172.21.15.170:3300/0 conn(0x7fdd0c0ccfb0 0x7fdd0c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.404+0000 7fdd0bfff640 1 --2- 172.21.15.29:0/2855515875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0cca70 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009870 tx=0x7fdcf802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca46396267c951a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.404+0000 7fdd0b7fe640 1 -- 172.21.15.29:0/2855515875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8031860 con 0x7fdd0c0ad970 2023-10-16T15:53:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.405+0000 7fdd0b7fe640 1 -- 172.21.15.29:0/2855515875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf80319c0 con 0x7fdd0c0ad970 2023-10-16T15:53:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.405+0000 7fdd0b7fe640 1 -- 172.21.15.29:0/2855515875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8031d50 con 0x7fdd0c0ad970 2023-10-16T15:53:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.405+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2855515875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ad970 msgr2=0x7fdd0c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.405+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2855515875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0cca70 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009870 tx=0x7fdcf802b900 comp rx=0 tx=0).stop 2023-10-16T15:53:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.406+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2855515875 shutdown_connections 2023-10-16T15:53:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.406+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2855515875 >> v2:172.21.15.170:3300/0 conn(0x7fdd0c0ccfb0 0x7fdd0c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.406+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2855515875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0cca70 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.406+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2855515875 >> 172.21.15.29:0/2855515875 conn(0x7fdd0c0ce6a0 msgr2=0x7fdd0c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.406+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2855515875 shutdown_connections 2023-10-16T15:53:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.406+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2855515875 wait complete. 2023-10-16T15:53:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.408+0000 7fdd11d4e640 1 Processor -- start 2023-10-16T15:53:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.408+0000 7fdd11d4e640 1 -- start start 2023-10-16T15:53:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd11d4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd0c06c040 0x7fdd0c152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd11d4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd11d4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 0x7fdd0c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd11d4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0af9f0 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd11d4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0afcd0 con 0x7fdd0c0ad970 2023-10-16T15:53:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd11d4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c0affb0 con 0x7fdd0c06c040 2023-10-16T15:53:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd1154d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 0x7fdd0c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd1154d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 0x7fdd0c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47204/0 (socket says 172.21.15.29:47204) 2023-10-16T15:53:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.409+0000 7fdd1154d640 1 -- 172.21.15.29:0/2542476149 learned_addr learned my addr 172.21.15.29:0/2542476149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd0bfff640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd10d4c640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd0c06c040 0x7fdd0c152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd1154d640 1 -- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd0c06c040 msgr2=0x7fdd0c152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd1154d640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd0c06c040 0x7fdd0c152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd1154d640 1 -- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd0c0ad970 msgr2=0x7fdd0c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd1154d640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd1154d640 1 -- 172.21.15.29:0/2542476149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80094d0 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd1154d640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 0x7fdd0c0ac890 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400cd50 tx=0x7fdd0400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:38.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.410+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0400f680 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.411+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0400f800 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.411+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0400fbb0 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.411+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c0b0290 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.411+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c0af4c0 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.412+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0c06b330 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.413+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd0400b040 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.414+0000 7fdd097fa640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdcf006af90 0x7fdcf006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.414+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd0408a360 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.414+0000 7fdd10d4c640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdcf006af90 0x7fdcf006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.416+0000 7fdd10d4c640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdcf006af90 0x7fdcf006d450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd00007fd0 tx=0x7fdd00009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:38.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.418+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd04055140 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.611+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdd0c0b04d0 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.613+0000 7fdd097fa640 1 -- 172.21.15.29:0/2542476149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v225) v1 ==== 84+0+209 (secure 0 0 0) 0x7fdd0c0b04d0 con 0x7fdd0c0ccfb0 2023-10-16T15:53:38.615 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:38.615 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:53:38.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:38.616 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:53:38.616 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:53:38.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:38.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:38.616 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 26, 2023-10-16T15:53:38.616 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_metadata" 2023-10-16T15:53:38.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:38.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:38.617 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 27, 2023-10-16T15:53:38.617 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_data" 2023-10-16T15:53:38.617 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:38.618 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:53:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdcf006af90 msgr2=0x7fdcf006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdcf006af90 0x7fdcf006d450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd00007fd0 tx=0x7fdd00009040 comp rx=0 tx=0).stop 2023-10-16T15:53:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 msgr2=0x7fdd0c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 0x7fdd0c0ac890 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400cd50 tx=0x7fdd0400a650 comp rx=0 tx=0).stop 2023-10-16T15:53:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 shutdown_connections 2023-10-16T15:53:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdcf006af90 0x7fdcf006d450 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0ccfb0 0x7fdd0c0ac890 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd0c0ad970 0x7fdd0c0aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 --2- 172.21.15.29:0/2542476149 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdd0c06c040 0x7fdd0c152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 >> 172.21.15.29:0/2542476149 conn(0x7fdd0c0ce6a0 msgr2=0x7fdd0c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 shutdown_connections 2023-10-16T15:53:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.615+0000 7fdd11d4e640 1 -- 172.21.15.29:0/2542476149 wait complete. 2023-10-16T15:53:38.637 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T15:53:38.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T15:53:38.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.866+0000 7f490f59e640 1 Processor -- start 2023-10-16T15:53:38.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.866+0000 7f490f59e640 1 -- start start 2023-10-16T15:53:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f49100cd800 0x7f49100cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f491006aeb0 0x7f491006b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49100ce180 con 0x7f491015cba0 2023-10-16T15:53:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100d0500 con 0x7f491006aeb0 2023-10-16T15:53:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100d07c0 con 0x7f49100cd800 2023-10-16T15:53:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f491006aeb0 0x7f491006b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f491006aeb0 0x7f491006b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47208/0 (socket says 172.21.15.29:47208) 2023-10-16T15:53:38.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.868+0000 7f490ed9d640 1 -- 172.21.15.29:0/1066493007 learned_addr learned my addr 172.21.15.29:0/1066493007 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.869+0000 7f4905d9b640 1 --2- 172.21.15.29:0/1066493007 >> v2:172.21.15.170:3300/0 conn(0x7f49100cd800 0x7f49100cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.869+0000 7f490ed9d640 1 -- 172.21.15.29:0/1066493007 >> v2:172.21.15.170:3300/0 conn(0x7f49100cd800 msgr2=0x7f49100cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.869+0000 7f490ed9d640 1 --2- 172.21.15.29:0/1066493007 >> v2:172.21.15.170:3300/0 conn(0x7f49100cd800 0x7f49100cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.869+0000 7f490ed9d640 1 -- 172.21.15.29:0/1066493007 >> v1:172.21.15.170:6789/0 conn(0x7f491015cba0 legacy=0x7f4910071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.869+0000 7f490ed9d640 1 -- 172.21.15.29:0/1066493007 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100d0aa0 con 0x7f491006aeb0 2023-10-16T15:53:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.869+0000 7f4905d9b640 1 --2- 172.21.15.29:0/1066493007 >> v2:172.21.15.170:3300/0 conn(0x7f49100cd800 0x7f49100cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.870+0000 7f490ed9d640 1 --2- 172.21.15.29:0/1066493007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491006aeb0 0x7f491006b320 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f49100ce3f0 tx=0x7f490802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7a132d67202af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:38.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.870+0000 7f490dd9b640 1 -- 172.21.15.29:0/1066493007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908031870 con 0x7f491006aeb0 2023-10-16T15:53:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.870+0000 7f490dd9b640 1 -- 172.21.15.29:0/1066493007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49080319d0 con 0x7f491006aeb0 2023-10-16T15:53:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.870+0000 7f490dd9b640 1 -- 172.21.15.29:0/1066493007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908031d60 con 0x7f491006aeb0 2023-10-16T15:53:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 -- 172.21.15.29:0/1066493007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491006aeb0 msgr2=0x7f491006b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 --2- 172.21.15.29:0/1066493007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491006aeb0 0x7f491006b320 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f49100ce3f0 tx=0x7f490802b910 comp rx=0 tx=0).stop 2023-10-16T15:53:38.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 -- 172.21.15.29:0/1066493007 shutdown_connections 2023-10-16T15:53:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 --2- 172.21.15.29:0/1066493007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491006aeb0 0x7f491006b320 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 --2- 172.21.15.29:0/1066493007 >> v2:172.21.15.170:3300/0 conn(0x7f49100cd800 0x7f49100cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 -- 172.21.15.29:0/1066493007 >> 172.21.15.29:0/1066493007 conn(0x7f4910065ed0 msgr2=0x7f49100662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 -- 172.21.15.29:0/1066493007 shutdown_connections 2023-10-16T15:53:38.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.871+0000 7f490f59e640 1 -- 172.21.15.29:0/1066493007 wait complete. 2023-10-16T15:53:38.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.873+0000 7f490f59e640 1 Processor -- start 2023-10-16T15:53:38.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.873+0000 7f490f59e640 1 -- start start 2023-10-16T15:53:38.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.874+0000 7f490f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f491006aeb0 0x7f4910156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.874+0000 7f490f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49100cd800 0x7f491015de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.874+0000 7f490f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 0x7f491015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.874+0000 7f490f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100c5530 con 0x7f491015cba0 2023-10-16T15:53:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.874+0000 7f490f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491015e780 con 0x7f49100cd800 2023-10-16T15:53:38.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.874+0000 7f490f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491015ea60 con 0x7f491006aeb0 2023-10-16T15:53:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f4905d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49100cd800 0x7f491015de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 0x7f491015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f4905d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49100cd800 0x7f491015de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:34080/0 (socket says 172.21.15.29:34080) 2023-10-16T15:53:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f4905d9b640 1 -- 172.21.15.29:0/1570057247 learned_addr learned my addr 172.21.15.29:0/1570057247 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:38.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 0x7f491015f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47210/0 (socket says 172.21.15.29:47210) 2023-10-16T15:53:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490e59c640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f491006aeb0 0x7f4910156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 -- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f491006aeb0 msgr2=0x7f4910156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f491006aeb0 0x7f4910156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 -- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49100cd800 msgr2=0x7f491015de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49100cd800 0x7f491015de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.875+0000 7f490ed9d640 1 -- 172.21.15.29:0/1570057247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49080094d0 con 0x7f491015cba0 2023-10-16T15:53:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.876+0000 7f490ed9d640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 0x7f491015f7b0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f4908033000 tx=0x7f49080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.876+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490803adf0 con 0x7f491015cba0 2023-10-16T15:53:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.876+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49080443f0 con 0x7f491015cba0 2023-10-16T15:53:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.876+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49100c51c0 con 0x7f491015cba0 2023-10-16T15:53:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.876+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49080447a0 con 0x7f491015cba0 2023-10-16T15:53:38.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.876+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491015e3c0 con 0x7f491015cba0 2023-10-16T15:53:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.878+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491015ec70 con 0x7f491015cba0 2023-10-16T15:53:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.879+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f490804d450 con 0x7f491015cba0 2023-10-16T15:53:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.880+0000 7f49077fe640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f48f006af90 0x7f48f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.880+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f490802d040 con 0x7f491015cba0 2023-10-16T15:53:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.880+0000 7f490e59c640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f48f006af90 0x7f48f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.882+0000 7f490e59c640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f48f006af90 0x7f48f006d450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4900009fd0 tx=0x7f490000d040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:38.884+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4908076dc0 con 0x7f491015cba0 2023-10-16T15:53:39.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.078+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f491004e9b0 con 0x7f491015cba0 2023-10-16T15:53:39.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.336+0000 7f49077fe640 1 -- 172.21.15.29:0/1570057247 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v226) v1 ==== 144+0+0 (secure 0 0 0) 0x7f4908035020 con 0x7f491015cba0 2023-10-16T15:53:39.339 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' created 2023-10-16T15:53:39.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f48f006af90 msgr2=0x7f48f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:39.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f48f006af90 0x7f48f006d450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4900009fd0 tx=0x7f490000d040 comp rx=0 tx=0).stop 2023-10-16T15:53:39.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 msgr2=0x7f491015f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 0x7f491015f7b0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f4908033000 tx=0x7f49080046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 shutdown_connections 2023-10-16T15:53:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f48f006af90 0x7f48f006d450 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f491015cba0 0x7f491015f7b0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49100cd800 0x7f491015de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 --2- 172.21.15.29:0/1570057247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f491006aeb0 0x7f4910156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 >> 172.21.15.29:0/1570057247 conn(0x7f4910065ed0 msgr2=0x7f491006b290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 shutdown_connections 2023-10-16T15:53:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.341+0000 7f490f59e640 1 -- 172.21.15.29:0/1570057247 wait complete. 2023-10-16T15:53:39.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:53:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.580+0000 7fdf6cead640 1 Processor -- start 2023-10-16T15:53:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.581+0000 7fdf6cead640 1 -- start start 2023-10-16T15:53:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.582+0000 7fdf6cead640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6806a4b0 0x7fdf680cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.582+0000 7fdf6cead640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf680b4160 0x7fdf680cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.582+0000 7fdf6cead640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680cc850 con 0x7fdf6815ac70 2023-10-16T15:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.582+0000 7fdf6cead640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf680ccb30 con 0x7fdf680b4160 2023-10-16T15:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.582+0000 7fdf6cead640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680cce10 con 0x7fdf6806a4b0 2023-10-16T15:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf680b4160 0x7fdf680cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf680b4160 0x7fdf680cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47214/0 (socket says 172.21.15.29:47214) 2023-10-16T15:53:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 -- 172.21.15.29:0/3510061797 learned_addr learned my addr 172.21.15.29:0/3510061797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 -- 172.21.15.29:0/3510061797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6806a4b0 msgr2=0x7fdf680cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 --2- 172.21.15.29:0/3510061797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6806a4b0 0x7fdf680cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 -- 172.21.15.29:0/3510061797 >> v1:172.21.15.170:6789/0 conn(0x7fdf6815ac70 legacy=0x7fdf68069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.583+0000 7fdf67fff640 1 -- 172.21.15.29:0/3510061797 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680cd0f0 con 0x7fdf680b4160 2023-10-16T15:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.584+0000 7fdf67fff640 1 --2- 172.21.15.29:0/3510061797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680cd8f0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009e50 tx=0x7fdf6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997c9c1021e3f741 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.585+0000 7fdf66ffd640 1 -- 172.21.15.29:0/3510061797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6002ca70 con 0x7fdf680b4160 2023-10-16T15:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.585+0000 7fdf66ffd640 1 -- 172.21.15.29:0/3510061797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6002cbd0 con 0x7fdf680b4160 2023-10-16T15:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.585+0000 7fdf66ffd640 1 -- 172.21.15.29:0/3510061797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf600343f0 con 0x7fdf680b4160 2023-10-16T15:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.585+0000 7fdf6cead640 1 -- 172.21.15.29:0/3510061797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 msgr2=0x7fdf680cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.585+0000 7fdf6cead640 1 --2- 172.21.15.29:0/3510061797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680cd8f0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009e50 tx=0x7fdf6002bed0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.586+0000 7fdf6cead640 1 -- 172.21.15.29:0/3510061797 shutdown_connections 2023-10-16T15:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.586+0000 7fdf6cead640 1 --2- 172.21.15.29:0/3510061797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680cd8f0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.586+0000 7fdf6cead640 1 --2- 172.21.15.29:0/3510061797 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6806a4b0 0x7fdf680cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.586+0000 7fdf6cead640 1 -- 172.21.15.29:0/3510061797 >> 172.21.15.29:0/3510061797 conn(0x7fdf680ce6a0 msgr2=0x7fdf680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.586+0000 7fdf6cead640 1 -- 172.21.15.29:0/3510061797 shutdown_connections 2023-10-16T15:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.586+0000 7fdf6cead640 1 -- 172.21.15.29:0/3510061797 wait complete. 2023-10-16T15:53:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.588+0000 7fdf6cead640 1 Processor -- start 2023-10-16T15:53:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.588+0000 7fdf6cead640 1 -- start start 2023-10-16T15:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf6cead640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf6806a4b0 0x7fdf680c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf6cead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf6cead640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6815ac70 0x7fdf680adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf6cead640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680c6180 con 0x7fdf680b4160 2023-10-16T15:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf6cead640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680c63d0 con 0x7fdf6806a4b0 2023-10-16T15:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf6cead640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680c66b0 con 0x7fdf6815ac70 2023-10-16T15:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf5ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf5ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47226/0 (socket says 172.21.15.29:47226) 2023-10-16T15:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.589+0000 7fdf5ffff640 1 -- 172.21.15.29:0/1578408586 learned_addr learned my addr 172.21.15.29:0/1578408586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf67fff640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6815ac70 0x7fdf680adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf677fe640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf6806a4b0 0x7fdf680c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf5ffff640 1 -- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6815ac70 msgr2=0x7fdf680adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf5ffff640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6815ac70 0x7fdf680adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf5ffff640 1 -- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf6806a4b0 msgr2=0x7fdf680c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf5ffff640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf6806a4b0 0x7fdf680c72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf5ffff640 1 -- 172.21.15.29:0/1578408586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf60009b30 con 0x7fdf680b4160 2023-10-16T15:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf67fff640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6815ac70 0x7fdf680adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf5ffff640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680c77f0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000b4a0 tx=0x7fdf5000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50010610 con 0x7fdf680b4160 2023-10-16T15:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50010770 con 0x7fdf680b4160 2023-10-16T15:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50010b20 con 0x7fdf680b4160 2023-10-16T15:53:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.590+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf680c6990 con 0x7fdf680b4160 2023-10-16T15:53:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.591+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf68157dd0 con 0x7fdf680b4160 2023-10-16T15:53:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.592+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf680adb70 con 0x7fdf680b4160 2023-10-16T15:53:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.593+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf50010c80 con 0x7fdf680b4160 2023-10-16T15:53:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.594+0000 7fdf64ff9640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdf4806af90 0x7fdf4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.594+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6110+0+0 (secure 0 0 0) 0x7fdf50089920 con 0x7fdf680b4160 2023-10-16T15:53:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.595+0000 7fdf677fe640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdf4806af90 0x7fdf4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.596+0000 7fdf677fe640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdf4806af90 0x7fdf4806d450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdf58006fd0 tx=0x7fdf58008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:39.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.598+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf50054810 con 0x7fdf680b4160 2023-10-16T15:53:39.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:39.802+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdf6804e9b0 con 0x7fdf680b4160 2023-10-16T15:53:40.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.345+0000 7fdf64ff9640 1 -- 172.21.15.29:0/1578408586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v227) v1 ==== 176+0+0 (secure 0 0 0) 0x7fdf500140d0 con 0x7fdf680b4160 2023-10-16T15:53:40.348 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' created 2023-10-16T15:53:40.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.349+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdf4806af90 msgr2=0x7fdf4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.349+0000 7fdf6cead640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdf4806af90 0x7fdf4806d450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdf58006fd0 tx=0x7fdf58008040 comp rx=0 tx=0).stop 2023-10-16T15:53:40.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.349+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 msgr2=0x7fdf680c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.349+0000 7fdf6cead640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680c77f0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000b4a0 tx=0x7fdf5000b960 comp rx=0 tx=0).stop 2023-10-16T15:53:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.350+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 shutdown_connections 2023-10-16T15:53:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.350+0000 7fdf6cead640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fdf4806af90 0x7fdf4806d450 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.350+0000 7fdf6cead640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf680b4160 0x7fdf680c77f0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.350+0000 7fdf6cead640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf6815ac70 0x7fdf680adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.350+0000 7fdf6cead640 1 --2- 172.21.15.29:0/1578408586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf6806a4b0 0x7fdf680c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.350+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 >> 172.21.15.29:0/1578408586 conn(0x7fdf680ce6a0 msgr2=0x7fdf6806f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.351+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 shutdown_connections 2023-10-16T15:53:40.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.351+0000 7fdf6cead640 1 -- 172.21.15.29:0/1578408586 wait complete. 2023-10-16T15:53:40.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T15:53:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.577+0000 7f9064768640 1 Processor -- start 2023-10-16T15:53:40.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.577+0000 7f9064768640 1 -- start start 2023-10-16T15:53:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.578+0000 7f9064768640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f905c15ad70 0x7f905c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.578+0000 7f9064768640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f905c0caac0 0x7f905c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9064768640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c0cb030 con 0x7f905c0c9ca0 2023-10-16T15:53:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9064768640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905c0c91c0 con 0x7f905c15ad70 2023-10-16T15:53:40.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9064768640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905c0c94a0 con 0x7f905c0caac0 2023-10-16T15:53:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f905c15ad70 0x7f905c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f905c15ad70 0x7f905c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47228/0 (socket says 172.21.15.29:47228) 2023-10-16T15:53:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 -- 172.21.15.29:0/677096276 learned_addr learned my addr 172.21.15.29:0/677096276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:40.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 -- 172.21.15.29:0/677096276 >> v2:172.21.15.170:3300/0 conn(0x7f905c0caac0 msgr2=0x7f905c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:40.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 --2- 172.21.15.29:0/677096276 >> v2:172.21.15.170:3300/0 conn(0x7f905c0caac0 0x7f905c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 -- 172.21.15.29:0/677096276 >> v1:172.21.15.170:6789/0 conn(0x7f905c0c9ca0 legacy=0x7f905c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.579+0000 7f9062504640 1 -- 172.21.15.29:0/677096276 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c0c9780 con 0x7f905c15ad70 2023-10-16T15:53:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.581+0000 7f9062504640 1 --2- 172.21.15.29:0/677096276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c15ad70 0x7f905c069da0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f905c06d090 tx=0x7f905802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860d407587d31783 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.581+0000 7f9061502640 1 -- 172.21.15.29:0/677096276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058031870 con 0x7f905c15ad70 2023-10-16T15:53:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.581+0000 7f9061502640 1 -- 172.21.15.29:0/677096276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90580319d0 con 0x7f905c15ad70 2023-10-16T15:53:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.581+0000 7f9061502640 1 -- 172.21.15.29:0/677096276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058031d60 con 0x7f905c15ad70 2023-10-16T15:53:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 -- 172.21.15.29:0/677096276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c15ad70 msgr2=0x7f905c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 --2- 172.21.15.29:0/677096276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c15ad70 0x7f905c069da0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f905c06d090 tx=0x7f905802b910 comp rx=0 tx=0).stop 2023-10-16T15:53:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 -- 172.21.15.29:0/677096276 shutdown_connections 2023-10-16T15:53:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 --2- 172.21.15.29:0/677096276 >> v2:172.21.15.170:3300/0 conn(0x7f905c0caac0 0x7f905c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 --2- 172.21.15.29:0/677096276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c15ad70 0x7f905c069da0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 -- 172.21.15.29:0/677096276 >> 172.21.15.29:0/677096276 conn(0x7f905c0ce9b0 msgr2=0x7f905c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 -- 172.21.15.29:0/677096276 shutdown_connections 2023-10-16T15:53:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.582+0000 7f9064768640 1 -- 172.21.15.29:0/677096276 wait complete. 2023-10-16T15:53:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.584+0000 7f9064768640 1 Processor -- start 2023-10-16T15:53:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.584+0000 7f9064768640 1 -- start start 2023-10-16T15:53:40.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9064768640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 0x7f905c0ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:40.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9064768640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905c0caac0 0x7f905c0a8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9064768640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f905c15ad70 0x7f905c14ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9064768640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c14f320 con 0x7f905c0c9ca0 2023-10-16T15:53:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9064768640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c0ac710 con 0x7f905c15ad70 2023-10-16T15:53:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9064768640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905c0abb00 con 0x7f905c0caac0 2023-10-16T15:53:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9062504640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 0x7f905c0ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9062504640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 0x7f905c0ab420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47230/0 (socket says 172.21.15.29:47230) 2023-10-16T15:53:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.585+0000 7f9062504640 1 -- 172.21.15.29:0/3222543110 learned_addr learned my addr 172.21.15.29:0/3222543110 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9061d03640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905c0caac0 0x7f905c0a8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062504640 1 -- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905c0caac0 msgr2=0x7f905c0a8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062504640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905c0caac0 0x7f905c0a8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062d05640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f905c15ad70 0x7f905c14ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062504640 1 -- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f905c15ad70 msgr2=0x7f905c14ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062504640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f905c15ad70 0x7f905c14ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062504640 1 -- 172.21.15.29:0/3222543110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580094d0 con 0x7f905c0c9ca0 2023-10-16T15:53:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9061d03640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905c0caac0 0x7f905c0a8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.586+0000 7f9062504640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 0x7f905c0ab420 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f9058000c00 tx=0x7f90580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.587+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058031840 con 0x7f905c0c9ca0 2023-10-16T15:53:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.587+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90580443f0 con 0x7f905c0c9ca0 2023-10-16T15:53:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.587+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580447a0 con 0x7f905c0c9ca0 2023-10-16T15:53:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.587+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905c0abde0 con 0x7f905c0c9ca0 2023-10-16T15:53:40.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.587+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905c0ac3a0 con 0x7f905c0c9ca0 2023-10-16T15:53:40.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.588+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905c04e160 con 0x7f905c0c9ca0 2023-10-16T15:53:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.589+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f905804d450 con 0x7f905c0c9ca0 2023-10-16T15:53:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.590+0000 7f9052ffd640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f904006af90 0x7f904006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.591+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6495+0+0 (secure 0 0 0) 0x7f905802d040 con 0x7f905c0c9ca0 2023-10-16T15:53:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.591+0000 7f9061d03640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f904006af90 0x7f904006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.592+0000 7f9061d03640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f904006af90 0x7f904006d450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f905c06b610 tx=0x7f904c009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:40.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.595+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9058076ec0 con 0x7f905c0c9ca0 2023-10-16T15:53:40.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:40.828+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f905c0cefa0 con 0x7f905c0c9ca0 2023-10-16T15:53:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.367+0000 7f9052ffd640 1 -- 172.21.15.29:0/3222543110 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v111) v1 ==== 183+0+0 (secure 0 0 0) 0x7f905807a4e0 con 0x7f905c0c9ca0 2023-10-16T15:53:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 28 and data pool 29 2023-10-16T15:53:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.370+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f904006af90 msgr2=0x7f904006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.370+0000 7f9064768640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f904006af90 0x7f904006d450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f905c06b610 tx=0x7f904c009040 comp rx=0 tx=0).stop 2023-10-16T15:53:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.370+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 msgr2=0x7f905c0ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.370+0000 7f9064768640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 0x7f905c0ab420 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f9058000c00 tx=0x7f90580046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 shutdown_connections 2023-10-16T15:53:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f904006af90 0x7f904006d450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f905c15ad70 0x7f905c14ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f905c0caac0 0x7f905c0a8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 --2- 172.21.15.29:0/3222543110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f905c0c9ca0 0x7f905c0ab420 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 >> 172.21.15.29:0/3222543110 conn(0x7f905c0ce9b0 msgr2=0x7f905c0ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 shutdown_connections 2023-10-16T15:53:41.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.371+0000 7f9064768640 1 -- 172.21.15.29:0/3222543110 wait complete. 2023-10-16T15:53:41.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:41.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.598+0000 7fee4aa64640 1 Processor -- start 2023-10-16T15:53:41.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.598+0000 7fee4aa64640 1 -- start start 2023-10-16T15:53:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee4aa64640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fee440cd0c0 0x7fee441615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee4aa64640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4415e3e0 con 0x7fee440b0410 2023-10-16T15:53:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee4aa64640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee440cd4a0 con 0x7fee4415db00 2023-10-16T15:53:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee4aa64640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4415e100 con 0x7fee440cd0c0 2023-10-16T15:53:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee3bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fee440b0410 0x7fee4415d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38396/0 (socket says 172.21.15.29:38396) 2023-10-16T15:53:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee3bfff640 1 -- 172.21.15.29:0/2113755132 learned_addr learned my addr 172.21.15.29:0/2113755132 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:41.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.600+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328465491 0 0) 0x7fee4415e3e0 con 0x7fee440b0410 2023-10-16T15:53:41.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee2c003520 con 0x7fee440b0410 2023-10-16T15:53:41.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee48800640 1 --2- 172.21.15.29:0/2113755132 >> v2:172.21.15.170:3300/0 conn(0x7fee440cd0c0 0x7fee441615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682169794 0 0) 0x7fee2c003520 con 0x7fee440b0410 2023-10-16T15:53:41.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 >> v2:172.21.15.170:3300/0 conn(0x7fee440cd0c0 msgr2=0x7fee441615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 --2- 172.21.15.29:0/2113755132 >> v2:172.21.15.170:3300/0 conn(0x7fee440cd0c0 0x7fee441615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 >> v1:172.21.15.170:6789/0 conn(0x7fee4415db00 legacy=0x7fee4406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44160b10 con 0x7fee440b0410 2023-10-16T15:53:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fee34001670 con 0x7fee440b0410 2023-10-16T15:53:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 >> v1:172.21.15.29:6789/0 conn(0x7fee440b0410 legacy=0x7fee4415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee2c003970 0x7fee2c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c0047c0 0x7fee2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee2c0052a0 0x7fee2c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4415e100 con 0x7fee2c003970 2023-10-16T15:53:41.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00c150 con 0x7fee2c0047c0 2023-10-16T15:53:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.601+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00c310 con 0x7fee2c0052a0 2023-10-16T15:53:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee48800640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee2c003970 0x7fee2c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee3bfff640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c0047c0 0x7fee2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee49001640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee2c0052a0 0x7fee2c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee48800640 1 -- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee2c0052a0 msgr2=0x7fee2c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:41.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee48800640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee2c0052a0 0x7fee2c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee48800640 1 -- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c0047c0 msgr2=0x7fee2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:41.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee48800640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c0047c0 0x7fee2c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee49001640 1 -- 172.21.15.29:0/2113755132 reap_dead start 2023-10-16T15:53:41.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.602+0000 7fee48800640 1 -- 172.21.15.29:0/2113755132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c003520 con 0x7fee2c003970 2023-10-16T15:53:41.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.604+0000 7fee48800640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee2c003970 0x7fee2c003d70 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fee4000ba00 tx=0x7fee2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207266ae6b9d5ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.604+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee400325f0 con 0x7fee2c003970 2023-10-16T15:53:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.604+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee40032750 con 0x7fee2c003970 2023-10-16T15:53:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.604+0000 7fee3b7fe640 1 -- 172.21.15.29:0/2113755132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40032ae0 con 0x7fee2c003970 2023-10-16T15:53:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.604+0000 7fee4aa64640 1 -- 172.21.15.29:0/2113755132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee2c003970 msgr2=0x7fee2c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.604+0000 7fee4aa64640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee2c003970 0x7fee2c003d70 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fee4000ba00 tx=0x7fee2c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.605+0000 7fee4aa64640 1 -- 172.21.15.29:0/2113755132 shutdown_connections 2023-10-16T15:53:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.605+0000 7fee4aa64640 1 --2- 172.21.15.29:0/2113755132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee2c003970 0x7fee2c003d70 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.605+0000 7fee4aa64640 1 -- 172.21.15.29:0/2113755132 >> 172.21.15.29:0/2113755132 conn(0x7fee440cb410 msgr2=0x7fee440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.605+0000 7fee4aa64640 1 -- 172.21.15.29:0/2113755132 shutdown_connections 2023-10-16T15:53:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.605+0000 7fee4aa64640 1 -- 172.21.15.29:0/2113755132 wait complete. 2023-10-16T15:53:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.607+0000 7fee4aa64640 1 Processor -- start 2023-10-16T15:53:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.607+0000 7fee4aa64640 1 -- start start 2023-10-16T15:53:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee4aa64640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c003970 0x7fee4414ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee4aa64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 0x7fee440af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee4aa64640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee44149e50 0x7fee440af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee4aa64640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4414c360 con 0x7fee441499d0 2023-10-16T15:53:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee4aa64640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4414c640 con 0x7fee2c003970 2023-10-16T15:53:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee4aa64640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4414c920 con 0x7fee44149e50 2023-10-16T15:53:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee3bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 0x7fee440af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee3bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 0x7fee440af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47244/0 (socket says 172.21.15.29:47244) 2023-10-16T15:53:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.608+0000 7fee3bfff640 1 -- 172.21.15.29:0/621587056 learned_addr learned my addr 172.21.15.29:0/621587056 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee3bfff640 1 -- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee44149e50 msgr2=0x7fee440af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee48800640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c003970 0x7fee4414ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee49001640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee44149e50 0x7fee440af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee3bfff640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee44149e50 0x7fee440af740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee3bfff640 1 -- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c003970 msgr2=0x7fee4414ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee3bfff640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c003970 0x7fee4414ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee3bfff640 1 -- 172.21.15.29:0/621587056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee440b3460 con 0x7fee441499d0 2023-10-16T15:53:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee48800640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c003970 0x7fee4414ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee3bfff640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 0x7fee440af200 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fee3400e780 tx=0x7fee3400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400ee90 con 0x7fee441499d0 2023-10-16T15:53:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3400c8d0 con 0x7fee441499d0 2023-10-16T15:53:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400cc80 con 0x7fee441499d0 2023-10-16T15:53:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee440b3740 con 0x7fee441499d0 2023-10-16T15:53:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.609+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee441f6360 con 0x7fee441499d0 2023-10-16T15:53:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.611+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee440b3930 con 0x7fee441499d0 2023-10-16T15:53:41.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.612+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee3401d460 con 0x7fee441499d0 2023-10-16T15:53:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.613+0000 7fee397fa640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fee3006af90 0x7fee3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.613+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fee34089b60 con 0x7fee441499d0 2023-10-16T15:53:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.613+0000 7fee48800640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fee3006af90 0x7fee3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.615+0000 7fee48800640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fee3006af90 0x7fee3006d450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0057a0 tx=0x7fee4003d040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.617+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee340546e0 con 0x7fee441499d0 2023-10-16T15:53:41.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.813+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee4404e160 con 0x7fee441499d0 2023-10-16T15:53:41.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.818+0000 7fee397fa640 1 -- 172.21.15.29:0/621587056 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+20364 (secure 0 0 0) 0x7fee34057d00 con 0x7fee441499d0 2023-10-16T15:53:41.820 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:41.820 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":228,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:40.830507+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.820+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fee3006af90 msgr2=0x7fee3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.820+0000 7fee4aa64640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fee3006af90 0x7fee3006d450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0057a0 tx=0x7fee4003d040 comp rx=0 tx=0).stop 2023-10-16T15:53:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.820+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 msgr2=0x7fee440af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.820+0000 7fee4aa64640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 0x7fee440af200 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fee3400e780 tx=0x7fee3400ec40 comp rx=0 tx=0).stop 2023-10-16T15:53:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 shutdown_connections 2023-10-16T15:53:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fee3006af90 0x7fee3006d450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee441499d0 0x7fee440af200 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fee44149e50 0x7fee440af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 --2- 172.21.15.29:0/621587056 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee2c003970 0x7fee4414ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 >> 172.21.15.29:0/621587056 conn(0x7fee440cb410 msgr2=0x7fee4415d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 shutdown_connections 2023-10-16T15:53:41.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:41.821+0000 7fee4aa64640 1 -- 172.21.15.29:0/621587056 wait complete. 2023-10-16T15:53:41.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.061+0000 7f7d37fff640 1 Processor -- start 2023-10-16T15:53:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.062+0000 7f7d37fff640 1 -- start start 2023-10-16T15:53:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d37fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 0x7f7d3806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d37fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3815d910 con 0x7f7d38159950 2023-10-16T15:53:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d37fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3815dbf0 con 0x7f7d38070b60 2023-10-16T15:53:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d37fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815e490 con 0x7f7d380ca0a0 2023-10-16T15:53:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d36ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7d38159950 0x7f7d38070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38400/0 (socket says 172.21.15.29:38400) 2023-10-16T15:53:42.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d36ffd640 1 -- 172.21.15.29:0/506398736 learned_addr learned my addr 172.21.15.29:0/506398736 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.064+0000 7f7d377fe640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 0x7f7d3806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798164541 0 0) 0x7f7d3815d910 con 0x7f7d38159950 2023-10-16T15:53:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d24005730 con 0x7f7d38159950 2023-10-16T15:53:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878067681 0 0) 0x7f7d24005730 con 0x7f7d38159950 2023-10-16T15:53:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 msgr2=0x7f7d3806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 0x7f7d3806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 >> v1:172.21.15.170:6789/0 conn(0x7f7d38070b60 legacy=0x7f7d3806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3815ded0 con 0x7f7d38159950 2023-10-16T15:53:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7d28001670 con 0x7f7d38159950 2023-10-16T15:53:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 >> v1:172.21.15.29:6789/0 conn(0x7f7d38159950 legacy=0x7f7d38070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 0x7f7d24005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006130 0x7f7d240069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d24006f10 0x7f7d24007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815e490 con 0x7f7d24006f10 2023-10-16T15:53:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400dd20 con 0x7f7d24006130 2023-10-16T15:53:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400dee0 con 0x7f7d380ca0a0 2023-10-16T15:53:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.065+0000 7f7d377fe640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d24006f10 0x7f7d24007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d36ffd640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 0x7f7d24005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d2ffff640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006130 0x7f7d240069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d377fe640 1 -- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 msgr2=0x7f7d24005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d377fe640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380ca0a0 0x7f7d24005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d377fe640 1 -- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006130 msgr2=0x7f7d240069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d377fe640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006130 0x7f7d240069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d377fe640 1 -- 172.21.15.29:0/506398736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24005730 con 0x7f7d24006f10 2023-10-16T15:53:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.066+0000 7f7d2ffff640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006130 0x7f7d240069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.067+0000 7f7d377fe640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d24006f10 0x7f7d24007390 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009fc0 tx=0x7f7d24007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16edc71ecda0c522 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.067+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3002cc20 con 0x7f7d24006f10 2023-10-16T15:53:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.067+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3000f040 con 0x7f7d24006f10 2023-10-16T15:53:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.067+0000 7f7d367fc640 1 -- 172.21.15.29:0/506398736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30034440 con 0x7f7d24006f10 2023-10-16T15:53:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d37fff640 1 -- 172.21.15.29:0/506398736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d24006f10 msgr2=0x7f7d24007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d37fff640 1 --2- 172.21.15.29:0/506398736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d24006f10 0x7f7d24007390 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009fc0 tx=0x7f7d24007a00 comp rx=0 tx=0).stop 2023-10-16T15:53:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d377fe640 1 -- 172.21.15.29:0/506398736 reap_dead start 2023-10-16T15:53:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d37fff640 1 -- 172.21.15.29:0/506398736 shutdown_connections 2023-10-16T15:53:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d37fff640 1 -- 172.21.15.29:0/506398736 >> 172.21.15.29:0/506398736 conn(0x7f7d380ce6a0 msgr2=0x7f7d380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d37fff640 1 -- 172.21.15.29:0/506398736 shutdown_connections 2023-10-16T15:53:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.068+0000 7f7d37fff640 1 -- 172.21.15.29:0/506398736 wait complete. 2023-10-16T15:53:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.070+0000 7f7d37fff640 1 Processor -- start 2023-10-16T15:53:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.070+0000 7f7d37fff640 1 -- start start 2023-10-16T15:53:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d37fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006f10 0x7f7d38158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 0x7f7d38151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d37fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380b28b0 0x7f7d380b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d37fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380b3270 con 0x7f7d3815b930 2023-10-16T15:53:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d37fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815be80 con 0x7f7d24006f10 2023-10-16T15:53:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d37fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815c160 con 0x7f7d380b28b0 2023-10-16T15:53:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 0x7f7d38151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 0x7f7d38151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47260/0 (socket says 172.21.15.29:47260) 2023-10-16T15:53:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.071+0000 7f7d2ffff640 1 -- 172.21.15.29:0/684687047 learned_addr learned my addr 172.21.15.29:0/684687047 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d36ffd640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006f10 0x7f7d38158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2ffff640 1 -- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380b28b0 msgr2=0x7f7d380b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2ffff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380b28b0 0x7f7d380b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2ffff640 1 -- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006f10 msgr2=0x7f7d38158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2ffff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006f10 0x7f7d38158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2ffff640 1 -- 172.21.15.29:0/684687047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3815c440 con 0x7f7d3815b930 2023-10-16T15:53:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2ffff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 0x7f7d38151800 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d770 tx=0x7f7d2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20005270 con 0x7f7d3815b930 2023-10-16T15:53:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d200053d0 con 0x7f7d3815b930 2023-10-16T15:53:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000b870 con 0x7f7d3815b930 2023-10-16T15:53:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3815c720 con 0x7f7d3815b930 2023-10-16T15:53:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.072+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d38152060 con 0x7f7d3815b930 2023-10-16T15:53:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.074+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3804e9b0 con 0x7f7d3815b930 2023-10-16T15:53:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.075+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7d2000b9d0 con 0x7f7d3815b930 2023-10-16T15:53:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.075+0000 7f7d2f7fe640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7d1806af90 0x7f7d1806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.076+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f7d20089af0 con 0x7f7d3815b930 2023-10-16T15:53:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.076+0000 7f7d36ffd640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7d1806af90 0x7f7d1806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.077+0000 7f7d36ffd640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7d1806af90 0x7f7d1806d450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d28001200 tx=0x7f7d28001150 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.080+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7d20054600 con 0x7f7d3815b930 2023-10-16T15:53:42.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.279+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d380b4620 con 0x7f7d3815b930 2023-10-16T15:53:42.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.281+0000 7f7d2f7fe640 1 -- 172.21.15.29:0/684687047 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+20364 (secure 0 0 0) 0x7f7d20019310 con 0x7f7d3815b930 2023-10-16T15:53:42.283 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:42.284 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":228,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:40.830507+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7d1806af90 msgr2=0x7f7d1806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7d1806af90 0x7f7d1806d450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d28001200 tx=0x7f7d28001150 comp rx=0 tx=0).stop 2023-10-16T15:53:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 msgr2=0x7f7d38151800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 0x7f7d38151800 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d770 tx=0x7f7d2000dc30 comp rx=0 tx=0).stop 2023-10-16T15:53:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 shutdown_connections 2023-10-16T15:53:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f7d1806af90 0x7f7d1806d450 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d3815b930 0x7f7d38151800 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7d380b28b0 0x7f7d380b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 --2- 172.21.15.29:0/684687047 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7d24006f10 0x7f7d38158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 >> 172.21.15.29:0/684687047 conn(0x7f7d380ce6a0 msgr2=0x7f7d380c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 shutdown_connections 2023-10-16T15:53:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.284+0000 7f7d37fff640 1 -- 172.21.15.29:0/684687047 wait complete. 2023-10-16T15:53:42.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T15:53:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.534+0000 7fa1faa10640 1 Processor -- start 2023-10-16T15:53:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.535+0000 7fa1faa10640 1 -- start start 2023-10-16T15:53:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.536+0000 7fa1faa10640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa1f406a4b0 0x7fa1f40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.536+0000 7fa1faa10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 0x7fa1f40ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.536+0000 7fa1faa10640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40cb030 con 0x7fa1f415ad70 2023-10-16T15:53:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.536+0000 7fa1faa10640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f40cae30 con 0x7fa1f406a4b0 2023-10-16T15:53:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.536+0000 7fa1faa10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40c91c0 con 0x7fa1f40c9ca0 2023-10-16T15:53:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa1f415ad70 0x7fa1f4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38414/0 (socket says 172.21.15.29:38414) 2023-10-16T15:53:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f3fff640 1 -- 172.21.15.29:0/3815140636 learned_addr learned my addr 172.21.15.29:0/3815140636 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f37fe640 1 --2- 172.21.15.29:0/3815140636 >> v2:172.21.15.170:3300/0 conn(0x7fa1f406a4b0 0x7fa1f40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991588700 0 0) 0x7fa1f40cb030 con 0x7fa1f415ad70 2023-10-16T15:53:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f8fad640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 0x7fa1f40ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e0003520 con 0x7fa1f415ad70 2023-10-16T15:53:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2908019195 0 0) 0x7fa1e0003520 con 0x7fa1f415ad70 2023-10-16T15:53:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 msgr2=0x7fa1f40ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 0x7fa1f40ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 >> v2:172.21.15.170:3300/0 conn(0x7fa1f406a4b0 msgr2=0x7fa1f40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 --2- 172.21.15.29:0/3815140636 >> v2:172.21.15.170:3300/0 conn(0x7fa1f406a4b0 0x7fa1f40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40c94a0 con 0x7fa1f415ad70 2023-10-16T15:53:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.537+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa1e40019f0 con 0x7fa1f415ad70 2023-10-16T15:53:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 >> v1:172.21.15.29:6789/0 conn(0x7fa1f415ad70 legacy=0x7fa1f4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0003970 0x7fa1e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 0x7fa1e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e0004d00 0x7fa1e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40cae30 con 0x7fa1e0004d00 2023-10-16T15:53:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40c91c0 con 0x7fa1e0003970 2023-10-16T15:53:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e000b980 con 0x7fa1f40c9ca0 2023-10-16T15:53:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f8fad640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e0004d00 0x7fa1e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f3fff640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0003970 0x7fa1e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f37fe640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 0x7fa1e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f8fad640 1 -- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 msgr2=0x7fa1e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f8fad640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40c9ca0 0x7fa1e00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f8fad640 1 -- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0003970 msgr2=0x7fa1e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f8fad640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0003970 0x7fa1e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.538+0000 7fa1f8fad640 1 -- 172.21.15.29:0/3815140636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0003520 con 0x7fa1e0004d00 2023-10-16T15:53:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.539+0000 7fa1f8fad640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e0004d00 0x7fa1e0005180 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009ac0 tx=0x7fa1e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c7f425a5bd159e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec02c6b0 con 0x7fa1e0004d00 2023-10-16T15:53:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec02c810 con 0x7fa1e0004d00 2023-10-16T15:53:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1f2ffd640 1 -- 172.21.15.29:0/3815140636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec02cba0 con 0x7fa1e0004d00 2023-10-16T15:53:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1faa10640 1 -- 172.21.15.29:0/3815140636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e0004d00 msgr2=0x7fa1e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1faa10640 1 --2- 172.21.15.29:0/3815140636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e0004d00 0x7fa1e0005180 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009ac0 tx=0x7fa1e00057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1f8fad640 1 -- 172.21.15.29:0/3815140636 reap_dead start 2023-10-16T15:53:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1faa10640 1 -- 172.21.15.29:0/3815140636 shutdown_connections 2023-10-16T15:53:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1faa10640 1 -- 172.21.15.29:0/3815140636 >> 172.21.15.29:0/3815140636 conn(0x7fa1f40ce9b0 msgr2=0x7fa1f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.540+0000 7fa1faa10640 1 -- 172.21.15.29:0/3815140636 shutdown_connections 2023-10-16T15:53:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.541+0000 7fa1faa10640 1 -- 172.21.15.29:0/3815140636 wait complete. 2023-10-16T15:53:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.542+0000 7fa1faa10640 1 Processor -- start 2023-10-16T15:53:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.542+0000 7fa1faa10640 1 -- start start 2023-10-16T15:53:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.543+0000 7fa1faa10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0004d00 0x7fa1f414df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.543+0000 7fa1faa10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 0x7fa1f40c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.543+0000 7fa1faa10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40cd000 0x7fa1f40c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.543+0000 7fa1faa10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f415c500 con 0x7fa1f40ccb80 2023-10-16T15:53:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.543+0000 7fa1faa10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f415c7e0 con 0x7fa1e0004d00 2023-10-16T15:53:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.543+0000 7fa1faa10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f414e480 con 0x7fa1f40cd000 2023-10-16T15:53:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 0x7fa1f40c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 0x7fa1f40c7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47276/0 (socket says 172.21.15.29:47276) 2023-10-16T15:53:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 -- 172.21.15.29:0/466708367 learned_addr learned my addr 172.21.15.29:0/466708367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f3fff640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0004d00 0x7fa1f414df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 -- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40cd000 msgr2=0x7fa1f40c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f8fad640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40cd000 0x7fa1f40c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40cd000 0x7fa1f40c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 -- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0004d00 msgr2=0x7fa1f414df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0004d00 0x7fa1f414df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 -- 172.21.15.29:0/466708367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40ada60 con 0x7fa1f40ccb80 2023-10-16T15:53:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.544+0000 7fa1f37fe640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 0x7fa1f40c7c10 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0108e0 tx=0x7fa1dc010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.545+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00eb60 con 0x7fa1f40ccb80 2023-10-16T15:53:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.545+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc00ecc0 con 0x7fa1f40ccb80 2023-10-16T15:53:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.545+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc01c400 con 0x7fa1f40ccb80 2023-10-16T15:53:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.545+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f40add40 con 0x7fa1f40ccb80 2023-10-16T15:53:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.545+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f40ad680 con 0x7fa1f40ccb80 2023-10-16T15:53:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.546+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f404e160 con 0x7fa1f40ccb80 2023-10-16T15:53:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.547+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa1dc01c560 con 0x7fa1f40ccb80 2023-10-16T15:53:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.548+0000 7fa1f0ff9640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa1d406af90 0x7fa1d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.548+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fa1dc08a270 con 0x7fa1f40ccb80 2023-10-16T15:53:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.549+0000 7fa1f3fff640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa1d406af90 0x7fa1d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.550+0000 7fa1f3fff640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa1d406af90 0x7fa1d406d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40013d0 tx=0x7fa1e4009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.553+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa1dc054ea0 con 0x7fa1f40ccb80 2023-10-16T15:53:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:42.787+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa1f40cee60 con 0x7fa1f40ccb80 2023-10-16T15:53:43.158 INFO:tasks.ceph.mds.b.smithi170.stderr:2023-10-16T15:53:43.156+0000 7f1de6f29640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:53:43.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.409+0000 7fa1f0ff9640 1 -- 172.21.15.29:0/466708367 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v113) v1 ==== 121+0+0 (secure 0 0 0) 0x7fa1dc0584c0 con 0x7fa1f40ccb80 2023-10-16T15:53:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa1d406af90 msgr2=0x7fa1d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa1d406af90 0x7fa1d406d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40013d0 tx=0x7fa1e4009040 comp rx=0 tx=0).stop 2023-10-16T15:53:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 msgr2=0x7fa1f40c7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 0x7fa1f40c7c10 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0108e0 tx=0x7fa1dc010da0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 shutdown_connections 2023-10-16T15:53:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa1d406af90 0x7fa1d406d450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1f40ccb80 0x7fa1f40c7c10 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1f40cd000 0x7fa1f40c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 --2- 172.21.15.29:0/466708367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e0004d00 0x7fa1f414df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.413+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 >> 172.21.15.29:0/466708367 conn(0x7fa1f40ce9b0 msgr2=0x7fa1f406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:43.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.414+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 shutdown_connections 2023-10-16T15:53:43.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.414+0000 7fa1faa10640 1 -- 172.21.15.29:0/466708367 wait complete. 2023-10-16T15:53:43.438 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:53:43.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T15:53:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.665+0000 7f6c66c10640 1 Processor -- start 2023-10-16T15:53:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.665+0000 7f6c66c10640 1 -- start start 2023-10-16T15:53:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c66c10640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c6015ad70 0x7f6c60069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c66c10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c600c9ca0 0x7f6c6006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c66c10640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c600cb030 con 0x7f6c600ca3b0 2023-10-16T15:53:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c66c10640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c600c91c0 con 0x7f6c6015ad70 2023-10-16T15:53:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c66c10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600c94a0 con 0x7f6c600c9ca0 2023-10-16T15:53:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c649ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c6015ad70 0x7f6c60069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c649ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c6015ad70 0x7f6c60069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47286/0 (socket says 172.21.15.29:47286) 2023-10-16T15:53:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.667+0000 7f6c649ac640 1 -- 172.21.15.29:0/3427949168 learned_addr learned my addr 172.21.15.29:0/3427949168 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.668+0000 7f6c5ffff640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c600c9ca0 0x7f6c6006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.668+0000 7f6c649ac640 1 -- 172.21.15.29:0/3427949168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c600c9ca0 msgr2=0x7f6c6006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.668+0000 7f6c649ac640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c600c9ca0 0x7f6c6006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.668+0000 7f6c649ac640 1 -- 172.21.15.29:0/3427949168 >> v1:172.21.15.170:6789/0 conn(0x7f6c600ca3b0 legacy=0x7f6c600ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.668+0000 7f6c649ac640 1 -- 172.21.15.29:0/3427949168 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c600c9780 con 0x7f6c6015ad70 2023-10-16T15:53:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.668+0000 7f6c5ffff640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c600c9ca0 0x7f6c6006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.669+0000 7f6c649ac640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6015ad70 0x7f6c60069da0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009870 tx=0x7f6c5402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7cc0219e64f43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.669+0000 7f6c5f7fe640 1 -- 172.21.15.29:0/3427949168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54031640 con 0x7f6c6015ad70 2023-10-16T15:53:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.669+0000 7f6c5f7fe640 1 -- 172.21.15.29:0/3427949168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c540317a0 con 0x7f6c6015ad70 2023-10-16T15:53:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.669+0000 7f6c5f7fe640 1 -- 172.21.15.29:0/3427949168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54031b30 con 0x7f6c6015ad70 2023-10-16T15:53:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 -- 172.21.15.29:0/3427949168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6015ad70 msgr2=0x7f6c60069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6015ad70 0x7f6c60069da0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009870 tx=0x7f6c5402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 -- 172.21.15.29:0/3427949168 shutdown_connections 2023-10-16T15:53:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c600c9ca0 0x7f6c6006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3427949168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6015ad70 0x7f6c60069da0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 -- 172.21.15.29:0/3427949168 >> 172.21.15.29:0/3427949168 conn(0x7f6c600ce9b0 msgr2=0x7f6c600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 -- 172.21.15.29:0/3427949168 shutdown_connections 2023-10-16T15:53:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.670+0000 7f6c66c10640 1 -- 172.21.15.29:0/3427949168 wait complete. 2023-10-16T15:53:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.672+0000 7f6c66c10640 1 Processor -- start 2023-10-16T15:53:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.672+0000 7f6c66c10640 1 -- start start 2023-10-16T15:53:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c66c10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 0x7f6c600b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c66c10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c600ca3b0 0x7f6c601f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c66c10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c6015ad70 0x7f6c600acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c66c10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600b4270 con 0x7f6c600c9ca0 2023-10-16T15:53:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c66c10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600b37e0 con 0x7f6c600ca3b0 2023-10-16T15:53:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c66c10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c601f13d0 con 0x7f6c6015ad70 2023-10-16T15:53:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c649ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 0x7f6c600b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c649ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 0x7f6c600b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47302/0 (socket says 172.21.15.29:47302) 2023-10-16T15:53:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.673+0000 7f6c649ac640 1 -- 172.21.15.29:0/3290835729 learned_addr learned my addr 172.21.15.29:0/3290835729 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c651ad640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c6015ad70 0x7f6c600acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c5ffff640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c600ca3b0 0x7f6c601f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c649ac640 1 -- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c6015ad70 msgr2=0x7f6c600acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c649ac640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c6015ad70 0x7f6c600acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c649ac640 1 -- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c600ca3b0 msgr2=0x7f6c601f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c649ac640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c600ca3b0 0x7f6c601f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c649ac640 1 -- 172.21.15.29:0/3290835729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540094d0 con 0x7f6c600c9ca0 2023-10-16T15:53:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c651ad640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c6015ad70 0x7f6c600acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c5ffff640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c600ca3b0 0x7f6c601f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.674+0000 7f6c649ac640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 0x7f6c600b32a0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6c5402b930 tx=0x7f6c540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.675+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5403ac60 con 0x7f6c600c9ca0 2023-10-16T15:53:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.675+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5403ade0 con 0x7f6c600c9ca0 2023-10-16T15:53:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.675+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54042630 con 0x7f6c600c9ca0 2023-10-16T15:53:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.675+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c600b39d0 con 0x7f6c600c9ca0 2023-10-16T15:53:43.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.675+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c600b3f90 con 0x7f6c600c9ca0 2023-10-16T15:53:43.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.676+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6004e160 con 0x7f6c600c9ca0 2023-10-16T15:53:43.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.677+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c5402f050 con 0x7f6c600c9ca0 2023-10-16T15:53:43.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.678+0000 7f6c5d7fa640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f6c4406af90 0x7f6c4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:43.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.678+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f6c540ab720 con 0x7f6c600c9ca0 2023-10-16T15:53:43.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.679+0000 7f6c5ffff640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f6c4406af90 0x7f6c4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:43.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.680+0000 7f6c5ffff640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f6c4406af90 0x7f6c4406d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c006fd0 tx=0x7f6c4c008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:43.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.683+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c54076220 con 0x7f6c600c9ca0 2023-10-16T15:53:43.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:43.915+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6c600addf0 con 0x7f6c600c9ca0 2023-10-16T15:53:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.413+0000 7f6c5d7fa640 1 -- 172.21.15.29:0/3290835729 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v114) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6c54079840 con 0x7f6c600c9ca0 2023-10-16T15:53:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f6c4406af90 msgr2=0x7f6c4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f6c4406af90 0x7f6c4406d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c006fd0 tx=0x7f6c4c008040 comp rx=0 tx=0).stop 2023-10-16T15:53:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 msgr2=0x7f6c600b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 0x7f6c600b32a0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6c5402b930 tx=0x7f6c540046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 shutdown_connections 2023-10-16T15:53:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f6c4406af90 0x7f6c4406d450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6c6015ad70 0x7f6c600acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6c600ca3b0 0x7f6c601f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 --2- 172.21.15.29:0/3290835729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c600c9ca0 0x7f6c600b32a0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 >> 172.21.15.29:0/3290835729 conn(0x7f6c600ce9b0 msgr2=0x7f6c6015b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 shutdown_connections 2023-10-16T15:53:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.416+0000 7f6c66c10640 1 -- 172.21.15.29:0/3290835729 wait complete. 2023-10-16T15:53:44.442 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T15:53:44.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:44.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.667+0000 7f49f14e5640 1 Processor -- start 2023-10-16T15:53:44.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.667+0000 7f49f14e5640 1 -- start start 2023-10-16T15:53:44.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.668+0000 7f49f14e5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f49ec0b3e70 0x7f49ec0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.668+0000 7f49f14e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec0b3940 con 0x7f49ec0b3070 2023-10-16T15:53:44.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.668+0000 7f49f14e5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec0b45e0 con 0x7f49ec0cfc10 2023-10-16T15:53:44.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.668+0000 7f49f14e5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ec0c7880 con 0x7f49ec0b3e70 2023-10-16T15:53:44.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.669+0000 7f49eaffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f49ec0b3070 0x7f49ec0cf500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55978/0 (socket says 172.21.15.29:55978) 2023-10-16T15:53:44.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.669+0000 7f49eaffd640 1 -- 172.21.15.29:0/2294124455 learned_addr learned my addr 172.21.15.29:0/2294124455 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:44.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.669+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889335429 0 0) 0x7f49ec0b3940 con 0x7f49ec0b3070 2023-10-16T15:53:44.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.669+0000 7f49eb7fe640 1 --2- 172.21.15.29:0/2294124455 >> v2:172.21.15.170:3300/0 conn(0x7f49ec0b3e70 0x7f49ec0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.669+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d8003520 con 0x7f49ec0b3070 2023-10-16T15:53:44.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3995942997 0 0) 0x7f49d8003520 con 0x7f49ec0b3070 2023-10-16T15:53:44.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 >> v2:172.21.15.170:3300/0 conn(0x7f49ec0b3e70 msgr2=0x7f49ec0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:44.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 --2- 172.21.15.29:0/2294124455 >> v2:172.21.15.170:3300/0 conn(0x7f49ec0b3e70 0x7f49ec0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 >> v1:172.21.15.170:6789/0 conn(0x7f49ec0cfc10 legacy=0x7f49ec0d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:44.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec0b3b90 con 0x7f49ec0b3070 2023-10-16T15:53:44.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f49dc001b70 con 0x7f49ec0b3070 2023-10-16T15:53:44.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 >> v1:172.21.15.29:6789/0 conn(0x7f49ec0b3070 legacy=0x7f49ec0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d8003970 0x7f49d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d80047c0 0x7f49d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d8005180 0x7f49d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0c7880 con 0x7f49d8003970 2023-10-16T15:53:44.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d800c020 con 0x7f49d80047c0 2023-10-16T15:53:44.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d800c1e0 con 0x7f49d8005180 2023-10-16T15:53:44.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.670+0000 7f49eaffd640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d8003970 0x7f49d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49ea7fc640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d80047c0 0x7f49d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49eaffd640 1 -- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d8005180 msgr2=0x7f49d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:44.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49eaffd640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49d8005180 0x7f49d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49eaffd640 1 -- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d80047c0 msgr2=0x7f49d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49eaffd640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d80047c0 0x7f49d8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49eaffd640 1 -- 172.21.15.29:0/2294124455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d8003520 con 0x7f49d8003970 2023-10-16T15:53:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49ea7fc640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d80047c0 0x7f49d8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.671+0000 7f49eb7fe640 1 -- 172.21.15.29:0/2294124455 reap_dead start 2023-10-16T15:53:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.672+0000 7f49eaffd640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d8003970 0x7f49d8003d90 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c520 tx=0x7f49d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c421a947eb60b10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.672+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc02ce10 con 0x7f49d8003970 2023-10-16T15:53:44.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.672+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49dc005b20 con 0x7f49d8003970 2023-10-16T15:53:44.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.672+0000 7f49e9ffb640 1 -- 172.21.15.29:0/2294124455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc007070 con 0x7f49d8003970 2023-10-16T15:53:44.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.672+0000 7f49f14e5640 1 -- 172.21.15.29:0/2294124455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d8003970 msgr2=0x7f49d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.672+0000 7f49f14e5640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d8003970 0x7f49d8003d90 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c520 tx=0x7f49d80057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.673+0000 7f49f14e5640 1 -- 172.21.15.29:0/2294124455 shutdown_connections 2023-10-16T15:53:44.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.673+0000 7f49f14e5640 1 --2- 172.21.15.29:0/2294124455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49d8003970 0x7f49d8003d90 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.673+0000 7f49f14e5640 1 -- 172.21.15.29:0/2294124455 >> 172.21.15.29:0/2294124455 conn(0x7f49ec0d2d20 msgr2=0x7f49ec0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:44.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.673+0000 7f49f14e5640 1 -- 172.21.15.29:0/2294124455 shutdown_connections 2023-10-16T15:53:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.673+0000 7f49f14e5640 1 -- 172.21.15.29:0/2294124455 wait complete. 2023-10-16T15:53:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.675+0000 7f49f14e5640 1 Processor -- start 2023-10-16T15:53:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.675+0000 7f49f14e5640 1 -- start start 2023-10-16T15:53:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49f14e5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d8003970 0x7f49ec0ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49f14e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 0x7f49ec1535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49f14e5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ec15b930 0x7f49ec15bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49f14e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec155090 con 0x7f49ec155220 2023-10-16T15:53:44.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49f14e5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0ca5e0 con 0x7f49d8003970 2023-10-16T15:53:44.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49f14e5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec15c2f0 con 0x7f49ec15b930 2023-10-16T15:53:44.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49ea7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 0x7f49ec1535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49ea7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 0x7f49ec1535e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45780/0 (socket says 172.21.15.29:45780) 2023-10-16T15:53:44.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.676+0000 7f49ea7fc640 1 -- 172.21.15.29:0/3525524282 learned_addr learned my addr 172.21.15.29:0/3525524282 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:44.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49eb7fe640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ec15b930 0x7f49ec15bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49eaffd640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d8003970 0x7f49ec0ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49ea7fc640 1 -- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ec15b930 msgr2=0x7f49ec15bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49ea7fc640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ec15b930 0x7f49ec15bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49ea7fc640 1 -- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d8003970 msgr2=0x7f49ec0ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49ea7fc640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d8003970 0x7f49ec0ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49ea7fc640 1 -- 172.21.15.29:0/3525524282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec0b04f0 con 0x7f49ec155220 2023-10-16T15:53:44.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49ea7fc640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 0x7f49ec1535e0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f49d400d400 tx=0x7f49d400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:44.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d400b070 con 0x7f49ec155220 2023-10-16T15:53:44.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d4003610 con 0x7f49ec155220 2023-10-16T15:53:44.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d40039c0 con 0x7f49ec155220 2023-10-16T15:53:44.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.677+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec15ce70 con 0x7f49ec155220 2023-10-16T15:53:44.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.678+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec0ca7f0 con 0x7f49ec155220 2023-10-16T15:53:44.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.679+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec04e160 con 0x7f49ec155220 2023-10-16T15:53:44.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.680+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f49d4003b20 con 0x7f49ec155220 2023-10-16T15:53:44.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.681+0000 7f49e37fe640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49cc06af90 0x7f49cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:44.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.681+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f49d40893d0 con 0x7f49ec155220 2023-10-16T15:53:44.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.681+0000 7f49eaffd640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49cc06af90 0x7f49cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:44.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.683+0000 7f49eaffd640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49cc06af90 0x7f49cc06d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c4f0 tx=0x7f49dc009030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:44.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.685+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49d4053fc0 con 0x7f49ec155220 2023-10-16T15:53:44.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.918+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49ec06c120 con 0x7f49ec155220 2023-10-16T15:53:44.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.919+0000 7f49e37fe640 1 -- 172.21.15.29:0/3525524282 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f49d40575e0 con 0x7f49ec155220 2023-10-16T15:53:44.921 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:44.921 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:53:44.922 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 114 2023-10-16T15:53:44.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49cc06af90 msgr2=0x7f49cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49cc06af90 0x7f49cc06d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c4f0 tx=0x7f49dc009030 comp rx=0 tx=0).stop 2023-10-16T15:53:44.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 msgr2=0x7f49ec1535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:44.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 0x7f49ec1535e0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f49d400d400 tx=0x7f49d400d8c0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 shutdown_connections 2023-10-16T15:53:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f49cc06af90 0x7f49cc06d450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49ec155220 0x7f49ec1535e0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49ec15b930 0x7f49ec15bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 --2- 172.21.15.29:0/3525524282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49d8003970 0x7f49ec0ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.921+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 >> 172.21.15.29:0/3525524282 conn(0x7f49ec0d2d20 msgr2=0x7f49ec0cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.922+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 shutdown_connections 2023-10-16T15:53:44.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:44.922+0000 7f49f14e5640 1 -- 172.21.15.29:0/3525524282 wait complete. 2023-10-16T15:53:44.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:53:45.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.170+0000 7ff62c9ff640 1 Processor -- start 2023-10-16T15:53:45.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.170+0000 7ff62c9ff640 1 -- start start 2023-10-16T15:53:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff62c9ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62806c040 0x7ff62806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff62c9ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6280cda80 0x7ff62815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff62c9ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62806c960 con 0x7ff6280ad970 2023-10-16T15:53:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff62c9ff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62815b9d0 con 0x7ff6280cda80 2023-10-16T15:53:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff62c9ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62815bcb0 con 0x7ff62806c040 2023-10-16T15:53:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff627fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6280cda80 0x7ff62815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff626ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6280ad970 0x7ff62815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55992/0 (socket says 172.21.15.29:55992) 2023-10-16T15:53:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff626ffd640 1 -- 172.21.15.29:0/303387259 learned_addr learned my addr 172.21.15.29:0/303387259 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.172+0000 7ff6277fe640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62806c040 0x7ff62806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff6267fc640 1 -- 172.21.15.29:0/303387259 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891439612 0 0) 0x7ff62806c960 con 0x7ff6280ad970 2023-10-16T15:53:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff6267fc640 1 -- 172.21.15.29:0/303387259 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff610003520 con 0x7ff6280ad970 2023-10-16T15:53:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff627fff640 1 -- 172.21.15.29:0/303387259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62806c040 msgr2=0x7ff62806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff627fff640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62806c040 0x7ff62806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff627fff640 1 -- 172.21.15.29:0/303387259 >> v1:172.21.15.29:6789/0 conn(0x7ff6280ad970 legacy=0x7ff62815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff627fff640 1 -- 172.21.15.29:0/303387259 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62815bf90 con 0x7ff6280cda80 2023-10-16T15:53:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.173+0000 7ff6277fe640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62806c040 0x7ff62806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.174+0000 7ff627fff640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff62815ed90 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7ff620009d90 tx=0x7ff62002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760abb941d18b61b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.174+0000 7ff6267fc640 1 -- 172.21.15.29:0/303387259 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62002cc60 con 0x7ff6280cda80 2023-10-16T15:53:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.174+0000 7ff6267fc640 1 -- 172.21.15.29:0/303387259 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62002cdc0 con 0x7ff6280cda80 2023-10-16T15:53:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.174+0000 7ff6267fc640 1 -- 172.21.15.29:0/303387259 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620034660 con 0x7ff6280cda80 2023-10-16T15:53:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 -- 172.21.15.29:0/303387259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 msgr2=0x7ff62815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff62815ed90 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7ff620009d90 tx=0x7ff62002beb0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 -- 172.21.15.29:0/303387259 shutdown_connections 2023-10-16T15:53:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff62815ed90 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/303387259 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62806c040 0x7ff62806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 -- 172.21.15.29:0/303387259 >> 172.21.15.29:0/303387259 conn(0x7ff6280ce6a0 msgr2=0x7ff6280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 -- 172.21.15.29:0/303387259 shutdown_connections 2023-10-16T15:53:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.175+0000 7ff62c9ff640 1 -- 172.21.15.29:0/303387259 wait complete. 2023-10-16T15:53:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.177+0000 7ff62c9ff640 1 Processor -- start 2023-10-16T15:53:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.177+0000 7ff62c9ff640 1 -- start start 2023-10-16T15:53:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff62c9ff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62806c040 0x7ff628157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff62c9ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6280ad970 0x7ff628158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff62c9ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff6281f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff62c9ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280aede0 con 0x7ff6280cda80 2023-10-16T15:53:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff62c9ff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff628158990 con 0x7ff62806c040 2023-10-16T15:53:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff62c9ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff628155570 con 0x7ff6280ad970 2023-10-16T15:53:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff627fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff6281f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff627fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff6281f5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45798/0 (socket says 172.21.15.29:45798) 2023-10-16T15:53:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.178+0000 7ff627fff640 1 -- 172.21.15.29:0/3883802319 learned_addr learned my addr 172.21.15.29:0/3883802319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff627fff640 1 -- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6280ad970 msgr2=0x7ff628158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff627fff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6280ad970 0x7ff628158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff627fff640 1 -- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62806c040 msgr2=0x7ff628157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff6277fe640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62806c040 0x7ff628157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff627fff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62806c040 0x7ff628157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff627fff640 1 -- 172.21.15.29:0/3883802319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620009a70 con 0x7ff6280cda80 2023-10-16T15:53:45.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff627fff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff6281f5c30 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7ff620009d90 tx=0x7ff620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:45.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62003e610 con 0x7ff6280cda80 2023-10-16T15:53:45.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62003e770 con 0x7ff6280cda80 2023-10-16T15:53:45.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62003eb60 con 0x7ff6280cda80 2023-10-16T15:53:45.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.179+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6281f2b90 con 0x7ff6280cda80 2023-10-16T15:53:45.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.180+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6281f3150 con 0x7ff6280cda80 2023-10-16T15:53:45.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.181+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62806b330 con 0x7ff6280cda80 2023-10-16T15:53:45.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.182+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff62003b050 con 0x7ff6280cda80 2023-10-16T15:53:45.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.183+0000 7ff60ffff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff61406af90 0x7ff61406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.183+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff6200ac4f0 con 0x7ff6280cda80 2023-10-16T15:53:45.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.184+0000 7ff6277fe640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff61406af90 0x7ff61406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.185+0000 7ff6277fe640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff61406af90 0x7ff61406d450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff61800d8b0 tx=0x7ff618008040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:45.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.187+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff620077190 con 0x7ff6280cda80 2023-10-16T15:53:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.385+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6281556e0 con 0x7ff6280cda80 2023-10-16T15:53:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.387+0000 7ff60ffff640 1 -- 172.21.15.29:0/3883802319 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+20351 (secure 0 0 0) 0x7ff6281556e0 con 0x7ff6280cda80 2023-10-16T15:53:45.390 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:45.390 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":229,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:42.386701+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:53:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff61406af90 msgr2=0x7ff61406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff61406af90 0x7ff61406d450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff61800d8b0 tx=0x7ff618008040 comp rx=0 tx=0).stop 2023-10-16T15:53:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 msgr2=0x7ff6281f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff6281f5c30 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7ff620009d90 tx=0x7ff620004a40 comp rx=0 tx=0).stop 2023-10-16T15:53:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 shutdown_connections 2023-10-16T15:53:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff61406af90 0x7ff61406d450 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6280cda80 0x7ff6281f5c30 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6280ad970 0x7ff628158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 --2- 172.21.15.29:0/3883802319 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62806c040 0x7ff628157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 >> 172.21.15.29:0/3883802319 conn(0x7ff6280ce6a0 msgr2=0x7ff6280ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 shutdown_connections 2023-10-16T15:53:45.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.390+0000 7ff62c9ff640 1 -- 172.21.15.29:0/3883802319 wait complete. 2023-10-16T15:53:45.413 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:53:45.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:45.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.640+0000 7ff5256a9640 1 Processor -- start 2023-10-16T15:53:45.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.640+0000 7ff5256a9640 1 -- start start 2023-10-16T15:53:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.641+0000 7ff5256a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5200cd0c0 0x7ff5201615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.641+0000 7ff5256a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5200b0410 0x7ff5200b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.641+0000 7ff5256a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5200b0db0 con 0x7ff52015d3f0 2023-10-16T15:53:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.641+0000 7ff5256a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52015e3e0 con 0x7ff5200cd0c0 2023-10-16T15:53:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.641+0000 7ff5256a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200cd4a0 con 0x7ff5200b0410 2023-10-16T15:53:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5200cd0c0 0x7ff5201615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff52015d3f0 0x7ff52015d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56004/0 (socket says 172.21.15.29:56004) 2023-10-16T15:53:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51f7fe640 1 -- 172.21.15.29:0/1020359786 learned_addr learned my addr 172.21.15.29:0/1020359786 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51e7fc640 1 --2- 172.21.15.29:0/1020359786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5200b0410 0x7ff5200b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51effd640 1 -- 172.21.15.29:0/1020359786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5200b0410 msgr2=0x7ff5200b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51effd640 1 --2- 172.21.15.29:0/1020359786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5200b0410 0x7ff5200b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51dffb640 1 -- 172.21.15.29:0/1020359786 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425887936 0 0) 0x7ff5200b0db0 con 0x7ff52015d3f0 2023-10-16T15:53:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51effd640 1 -- 172.21.15.29:0/1020359786 >> v1:172.21.15.29:6789/0 conn(0x7ff52015d3f0 legacy=0x7ff52015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.642+0000 7ff51effd640 1 -- 172.21.15.29:0/1020359786 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52015e100 con 0x7ff5200cd0c0 2023-10-16T15:53:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.643+0000 7ff51effd640 1 --2- 172.21.15.29:0/1020359786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5201615f0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7ff510009d90 tx=0x7ff51002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d0be3dc7b427d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff51dffb640 1 -- 172.21.15.29:0/1020359786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51002cbc0 con 0x7ff5200cd0c0 2023-10-16T15:53:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff51dffb640 1 -- 172.21.15.29:0/1020359786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51002cd20 con 0x7ff5200cd0c0 2023-10-16T15:53:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff51dffb640 1 -- 172.21.15.29:0/1020359786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510034660 con 0x7ff5200cd0c0 2023-10-16T15:53:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff5256a9640 1 -- 172.21.15.29:0/1020359786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 msgr2=0x7ff5201615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff5256a9640 1 --2- 172.21.15.29:0/1020359786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5201615f0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7ff510009d90 tx=0x7ff51002be10 comp rx=0 tx=0).stop 2023-10-16T15:53:45.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff5256a9640 1 -- 172.21.15.29:0/1020359786 shutdown_connections 2023-10-16T15:53:45.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff5256a9640 1 --2- 172.21.15.29:0/1020359786 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff5200b0410 0x7ff5200b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff5256a9640 1 --2- 172.21.15.29:0/1020359786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5201615f0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.644+0000 7ff5256a9640 1 -- 172.21.15.29:0/1020359786 >> 172.21.15.29:0/1020359786 conn(0x7ff5200cb410 msgr2=0x7ff5200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:45.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.645+0000 7ff5256a9640 1 -- 172.21.15.29:0/1020359786 shutdown_connections 2023-10-16T15:53:45.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.645+0000 7ff5256a9640 1 -- 172.21.15.29:0/1020359786 wait complete. 2023-10-16T15:53:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.646+0000 7ff5256a9640 1 Processor -- start 2023-10-16T15:53:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.646+0000 7ff5256a9640 1 -- start start 2023-10-16T15:53:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.647+0000 7ff5256a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5200b0410 0x7ff5200b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.647+0000 7ff5256a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5200b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.647+0000 7ff5256a9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff52015d3f0 0x7ff5201e6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff5256a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200c4090 con 0x7ff5200cd0c0 2023-10-16T15:53:45.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff5256a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff520146650 con 0x7ff5200b0410 2023-10-16T15:53:45.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff5256a9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200c3480 con 0x7ff52015d3f0 2023-10-16T15:53:45.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5200b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5200b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45804/0 (socket says 172.21.15.29:45804) 2023-10-16T15:53:45.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 -- 172.21.15.29:0/2886592785 learned_addr learned my addr 172.21.15.29:0/2886592785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:45.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51effd640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5200b0410 0x7ff5200b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51f7fe640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff52015d3f0 0x7ff5201e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 -- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff52015d3f0 msgr2=0x7ff5201e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff52015d3f0 0x7ff5201e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 -- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5200b0410 msgr2=0x7ff5200b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5200b0410 0x7ff5200b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.648+0000 7ff51e7fc640 1 -- 172.21.15.29:0/2886592785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff510009a70 con 0x7ff5200cd0c0 2023-10-16T15:53:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.649+0000 7ff51e7fc640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5200b40d0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7ff50800d5f0 tx=0x7ff50800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.649+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5080125f0 con 0x7ff5200cd0c0 2023-10-16T15:53:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.649+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff508012750 con 0x7ff5200cd0c0 2023-10-16T15:53:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.649+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508012b00 con 0x7ff5200cd0c0 2023-10-16T15:53:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.649+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5200c3760 con 0x7ff5200cd0c0 2023-10-16T15:53:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.649+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5200c3d20 con 0x7ff5200cd0c0 2023-10-16T15:53:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.650+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5200c4200 con 0x7ff5200cd0c0 2023-10-16T15:53:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.652+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff508012c60 con 0x7ff5200cd0c0 2023-10-16T15:53:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.652+0000 7ff5177fe640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff50006afc0 0x7ff50006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.653+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff508089f40 con 0x7ff5200cd0c0 2023-10-16T15:53:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.653+0000 7ff51effd640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff50006afc0 0x7ff50006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.654+0000 7ff51effd640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff50006afc0 0x7ff50006d480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff510036000 tx=0x7ff510030000 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:45.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.657+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff508054b60 con 0x7ff5200cd0c0 2023-10-16T15:53:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.845+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff520160b10 con 0x7ff50006afc0 2023-10-16T15:53:45.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.889+0000 7ff5177fe640 1 -- 172.21.15.29:0/2886592785 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+521543 (secure 0 0 0) 0x7ff51002ce90 con 0x7ff50006afc0 2023-10-16T15:53:45.898 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:45.901 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:45.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.897+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff50006afc0 msgr2=0x7ff50006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.897+0000 7ff5256a9640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff50006afc0 0x7ff50006d480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff510036000 tx=0x7ff510030000 comp rx=0 tx=0).stop 2023-10-16T15:53:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.897+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 msgr2=0x7ff5200b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.897+0000 7ff5256a9640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5200b40d0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7ff50800d5f0 tx=0x7ff50800dab0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 shutdown_connections 2023-10-16T15:53:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff50006afc0 0x7ff50006d480 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5200cd0c0 0x7ff5200b40d0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff52015d3f0 0x7ff5201e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 --2- 172.21.15.29:0/2886592785 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff5200b0410 0x7ff5200b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 >> 172.21.15.29:0/2886592785 conn(0x7ff5200cb410 msgr2=0x7ff5200b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 shutdown_connections 2023-10-16T15:53:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:45.898+0000 7ff5256a9640 1 -- 172.21.15.29:0/2886592785 wait complete. 2023-10-16T15:53:45.923 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-16T15:53:44.462540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596363,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":655,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381312,"kb_used_data":23568,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593408,"statfs":{"total":773094113280,"available":772703649792,"internally_reserved":0,"allocated":24133632,"data_stored":12400665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366202425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3585,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.121643"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1248,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235155+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:28.235155+0000","last_peered":"2023-10-16T15:53:28.235155+0000","last_clean":"2023-10-16T15:53:28.235155+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:28.235155+0000","last_undegraded":"2023-10-16T15:53:28.235155+0000","last_fullsized":"2023-10-16T15:53:28.235155+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396662+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:41.396662+0000","last_peered":"2023-10-16T15:53:41.396662+0000","last_clean":"2023-10-16T15:53:41.396662+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:41.396662+0000","last_undegraded":"2023-10-16T15:53:41.396662+0000","last_fullsized":"2023-10-16T15:53:41.396662+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375906+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:41.375906+0000","last_peered":"2023-10-16T15:53:41.375906+0000","last_clean":"2023-10-16T15:53:41.375906+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:41.375906+0000","last_undegraded":"2023-10-16T15:53:41.375906+0000","last_fullsized":"2023-10-16T15:53:41.375906+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.355503+0000","last_change":"2023-10-16T15:53:39.355503+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.355503+0000","last_undegraded":"2023-10-16T15:53:39.355503+0000","last_fullsized":"2023-10-16T15:53:39.355503+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411355+0000","last_change":"2023-10-16T15:53:41.413779+0000","last_active":"2023-10-16T15:53:42.411355+0000","last_peered":"2023-10-16T15:53:42.411355+0000","last_clean":"2023-10-16T15:53:42.411355+0000","last_became_active":"2023-10-16T15:53:41.413379+0000","last_became_peered":"2023-10-16T15:53:41.413379+0000","last_unstale":"2023-10-16T15:53:42.411355+0000","last_undegraded":"2023-10-16T15:53:42.411355+0000","last_fullsized":"2023-10-16T15:53:42.411355+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384903+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:41.384903+0000","last_peered":"2023-10-16T15:53:41.384903+0000","last_clean":"2023-10-16T15:53:41.384903+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:41.384903+0000","last_undegraded":"2023-10-16T15:53:41.384903+0000","last_fullsized":"2023-10-16T15:53:41.384903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462231+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:29.462231+0000","last_peered":"2023-10-16T15:53:29.462231+0000","last_clean":"2023-10-16T15:53:29.462231+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:29.462231+0000","last_undegraded":"2023-10-16T15:53:29.462231+0000","last_fullsized":"2023-10-16T15:53:29.462231+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345783+0000","last_change":"2023-10-16T15:53:39.345783+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345783+0000","last_undegraded":"2023-10-16T15:53:39.345783+0000","last_fullsized":"2023-10-16T15:53:39.345783+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014672+0000","last_change":"2023-10-16T15:53:41.418345+0000","last_active":"2023-10-16T15:53:43.014672+0000","last_peered":"2023-10-16T15:53:43.014672+0000","last_clean":"2023-10-16T15:53:43.014672+0000","last_became_active":"2023-10-16T15:53:41.418078+0000","last_became_peered":"2023-10-16T15:53:41.418078+0000","last_unstale":"2023-10-16T15:53:43.014672+0000","last_undegraded":"2023-10-16T15:53:43.014672+0000","last_fullsized":"2023-10-16T15:53:43.014672+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:25.632327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461694+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:29.461694+0000","last_peered":"2023-10-16T15:53:29.461694+0000","last_clean":"2023-10-16T15:53:29.461694+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:29.461694+0000","last_undegraded":"2023-10-16T15:53:29.461694+0000","last_fullsized":"2023-10-16T15:53:29.461694+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380422+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:41.380422+0000","last_peered":"2023-10-16T15:53:41.380422+0000","last_clean":"2023-10-16T15:53:41.380422+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:41.380422+0000","last_undegraded":"2023-10-16T15:53:41.380422+0000","last_fullsized":"2023-10-16T15:53:41.380422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.405443+0000","last_change":"2023-10-16T15:53:40.384327+0000","last_active":"2023-10-16T15:53:41.405443+0000","last_peered":"2023-10-16T15:53:41.405443+0000","last_clean":"2023-10-16T15:53:41.405443+0000","last_became_active":"2023-10-16T15:53:40.384027+0000","last_became_peered":"2023-10-16T15:53:40.384027+0000","last_unstale":"2023-10-16T15:53:41.405443+0000","last_undegraded":"2023-10-16T15:53:41.405443+0000","last_fullsized":"2023-10-16T15:53:41.405443+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:09.776405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014626+0000","last_change":"2023-10-16T15:53:41.393836+0000","last_active":"2023-10-16T15:53:43.014626+0000","last_peered":"2023-10-16T15:53:43.014626+0000","last_clean":"2023-10-16T15:53:43.014626+0000","last_became_active":"2023-10-16T15:53:41.392583+0000","last_became_peered":"2023-10-16T15:53:41.392583+0000","last_unstale":"2023-10-16T15:53:43.014626+0000","last_undegraded":"2023-10-16T15:53:43.014626+0000","last_fullsized":"2023-10-16T15:53:43.014626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:22.777642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384099+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:41.384099+0000","last_peered":"2023-10-16T15:53:41.384099+0000","last_clean":"2023-10-16T15:53:41.384099+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:41.384099+0000","last_undegraded":"2023-10-16T15:53:41.384099+0000","last_fullsized":"2023-10-16T15:53:41.384099+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396904+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:41.396904+0000","last_peered":"2023-10-16T15:53:41.396904+0000","last_clean":"2023-10-16T15:53:41.396904+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:41.396904+0000","last_undegraded":"2023-10-16T15:53:41.396904+0000","last_fullsized":"2023-10-16T15:53:41.396904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395708+0000","last_change":"2023-10-16T15:53:40.378111+0000","last_active":"2023-10-16T15:53:41.395708+0000","last_peered":"2023-10-16T15:53:41.395708+0000","last_clean":"2023-10-16T15:53:41.395708+0000","last_became_active":"2023-10-16T15:53:40.377470+0000","last_became_peered":"2023-10-16T15:53:40.377470+0000","last_unstale":"2023-10-16T15:53:41.395708+0000","last_undegraded":"2023-10-16T15:53:41.395708+0000","last_fullsized":"2023-10-16T15:53:41.395708+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408902+0000","last_change":"2023-10-16T15:53:41.409618+0000","last_active":"2023-10-16T15:53:42.408902+0000","last_peered":"2023-10-16T15:53:42.408902+0000","last_clean":"2023-10-16T15:53:42.408902+0000","last_became_active":"2023-10-16T15:53:41.409365+0000","last_became_peered":"2023-10-16T15:53:41.409365+0000","last_unstale":"2023-10-16T15:53:42.408902+0000","last_undegraded":"2023-10-16T15:53:42.408902+0000","last_fullsized":"2023-10-16T15:53:42.408902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463159+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:29.463159+0000","last_peered":"2023-10-16T15:53:29.463159+0000","last_clean":"2023-10-16T15:53:29.463159+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:29.463159+0000","last_undegraded":"2023-10-16T15:53:29.463159+0000","last_fullsized":"2023-10-16T15:53:29.463159+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.346616+0000","last_change":"2023-10-16T15:53:39.346616+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.346616+0000","last_undegraded":"2023-10-16T15:53:39.346616+0000","last_fullsized":"2023-10-16T15:53:39.346616+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410745+0000","last_change":"2023-10-16T15:53:41.400067+0000","last_active":"2023-10-16T15:53:42.410745+0000","last_peered":"2023-10-16T15:53:42.410745+0000","last_clean":"2023-10-16T15:53:42.410745+0000","last_became_active":"2023-10-16T15:53:41.399778+0000","last_became_peered":"2023-10-16T15:53:41.399778+0000","last_unstale":"2023-10-16T15:53:42.410745+0000","last_undegraded":"2023-10-16T15:53:42.410745+0000","last_fullsized":"2023-10-16T15:53:42.410745+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:18.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384461+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:41.384461+0000","last_peered":"2023-10-16T15:53:41.384461+0000","last_clean":"2023-10-16T15:53:41.384461+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:41.384461+0000","last_undegraded":"2023-10-16T15:53:41.384461+0000","last_fullsized":"2023-10-16T15:53:41.384461+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461873+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:29.461873+0000","last_peered":"2023-10-16T15:53:29.461873+0000","last_clean":"2023-10-16T15:53:29.461873+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:29.461873+0000","last_undegraded":"2023-10-16T15:53:29.461873+0000","last_fullsized":"2023-10-16T15:53:29.461873+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374462+0000","last_change":"2023-10-16T15:53:40.375759+0000","last_active":"2023-10-16T15:53:41.374462+0000","last_peered":"2023-10-16T15:53:41.374462+0000","last_clean":"2023-10-16T15:53:41.374462+0000","last_became_active":"2023-10-16T15:53:40.375464+0000","last_became_peered":"2023-10-16T15:53:40.375464+0000","last_unstale":"2023-10-16T15:53:41.374462+0000","last_undegraded":"2023-10-16T15:53:41.374462+0000","last_fullsized":"2023-10-16T15:53:41.374462+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385405+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:41.385405+0000","last_peered":"2023-10-16T15:53:41.385405+0000","last_clean":"2023-10-16T15:53:41.385405+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:41.385405+0000","last_undegraded":"2023-10-16T15:53:41.385405+0000","last_fullsized":"2023-10-16T15:53:41.385405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462365+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:29.462365+0000","last_peered":"2023-10-16T15:53:29.462365+0000","last_clean":"2023-10-16T15:53:29.462365+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:29.462365+0000","last_undegraded":"2023-10-16T15:53:29.462365+0000","last_fullsized":"2023-10-16T15:53:29.462365+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.355490+0000","last_change":"2023-10-16T15:53:39.355490+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.355490+0000","last_undegraded":"2023-10-16T15:53:39.355490+0000","last_fullsized":"2023-10-16T15:53:39.355490+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.6","version":"225'1","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:31.400212+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:31.400212+0000","last_peered":"2023-10-16T15:53:31.400212+0000","last_clean":"2023-10-16T15:53:31.400212+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:31.400212+0000","last_undegraded":"2023-10-16T15:53:31.400212+0000","last_fullsized":"2023-10-16T15:53:31.400212+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376019+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:41.376019+0000","last_peered":"2023-10-16T15:53:41.376019+0000","last_clean":"2023-10-16T15:53:41.376019+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:41.376019+0000","last_undegraded":"2023-10-16T15:53:41.376019+0000","last_fullsized":"2023-10-16T15:53:41.376019+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384628+0000","last_change":"2023-10-16T15:53:40.375284+0000","last_active":"2023-10-16T15:53:41.384628+0000","last_peered":"2023-10-16T15:53:41.384628+0000","last_clean":"2023-10-16T15:53:41.384628+0000","last_became_active":"2023-10-16T15:53:40.374509+0000","last_became_peered":"2023-10-16T15:53:40.374509+0000","last_unstale":"2023-10-16T15:53:41.384628+0000","last_undegraded":"2023-10-16T15:53:41.384628+0000","last_fullsized":"2023-10-16T15:53:41.384628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:59.597435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407164+0000","last_change":"2023-10-16T15:53:41.414757+0000","last_active":"2023-10-16T15:53:42.407164+0000","last_peered":"2023-10-16T15:53:42.407164+0000","last_clean":"2023-10-16T15:53:42.407164+0000","last_became_active":"2023-10-16T15:53:41.414480+0000","last_became_peered":"2023-10-16T15:53:41.414480+0000","last_unstale":"2023-10-16T15:53:42.407164+0000","last_undegraded":"2023-10-16T15:53:42.407164+0000","last_fullsized":"2023-10-16T15:53:42.407164+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387180+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:41.387180+0000","last_peered":"2023-10-16T15:53:41.387180+0000","last_clean":"2023-10-16T15:53:41.387180+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:41.387180+0000","last_undegraded":"2023-10-16T15:53:41.387180+0000","last_fullsized":"2023-10-16T15:53:41.387180+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395410+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:41.395410+0000","last_peered":"2023-10-16T15:53:41.395410+0000","last_clean":"2023-10-16T15:53:41.395410+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:41.395410+0000","last_undegraded":"2023-10-16T15:53:41.395410+0000","last_fullsized":"2023-10-16T15:53:41.395410+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382520+0000","last_change":"2023-10-16T15:53:40.381870+0000","last_active":"2023-10-16T15:53:41.382520+0000","last_peered":"2023-10-16T15:53:41.382520+0000","last_clean":"2023-10-16T15:53:41.382520+0000","last_became_active":"2023-10-16T15:53:40.381577+0000","last_became_peered":"2023-10-16T15:53:40.381577+0000","last_unstale":"2023-10-16T15:53:41.382520+0000","last_undegraded":"2023-10-16T15:53:41.382520+0000","last_fullsized":"2023-10-16T15:53:41.382520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:33.030599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404650+0000","last_change":"2023-10-16T15:53:41.408405+0000","last_active":"2023-10-16T15:53:42.404650+0000","last_peered":"2023-10-16T15:53:42.404650+0000","last_clean":"2023-10-16T15:53:42.404650+0000","last_became_active":"2023-10-16T15:53:41.407251+0000","last_became_peered":"2023-10-16T15:53:41.407251+0000","last_unstale":"2023-10-16T15:53:42.404650+0000","last_undegraded":"2023-10-16T15:53:42.404650+0000","last_fullsized":"2023-10-16T15:53:42.404650+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.381328+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:41.381328+0000","last_peered":"2023-10-16T15:53:41.381328+0000","last_clean":"2023-10-16T15:53:41.381328+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:41.381328+0000","last_undegraded":"2023-10-16T15:53:41.381328+0000","last_fullsized":"2023-10-16T15:53:41.381328+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461650+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:29.461650+0000","last_peered":"2023-10-16T15:53:29.461650+0000","last_clean":"2023-10-16T15:53:29.461650+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:29.461650+0000","last_undegraded":"2023-10-16T15:53:29.461650+0000","last_fullsized":"2023-10-16T15:53:29.461650+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414970+0000","last_change":"2023-10-16T15:53:40.373067+0000","last_active":"2023-10-16T15:53:41.414970+0000","last_peered":"2023-10-16T15:53:41.414970+0000","last_clean":"2023-10-16T15:53:41.414970+0000","last_became_active":"2023-10-16T15:53:40.372307+0000","last_became_peered":"2023-10-16T15:53:40.372307+0000","last_unstale":"2023-10-16T15:53:41.414970+0000","last_undegraded":"2023-10-16T15:53:41.414970+0000","last_fullsized":"2023-10-16T15:53:41.414970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330512+0000","last_change":"2023-10-16T15:53:41.400515+0000","last_active":"2023-10-16T15:53:43.330512+0000","last_peered":"2023-10-16T15:53:43.330512+0000","last_clean":"2023-10-16T15:53:43.330512+0000","last_became_active":"2023-10-16T15:53:41.399835+0000","last_became_peered":"2023-10-16T15:53:41.399835+0000","last_unstale":"2023-10-16T15:53:43.330512+0000","last_undegraded":"2023-10-16T15:53:43.330512+0000","last_fullsized":"2023-10-16T15:53:43.330512+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:40.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375870+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:41.375870+0000","last_peered":"2023-10-16T15:53:41.375870+0000","last_clean":"2023-10-16T15:53:41.375870+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:41.375870+0000","last_undegraded":"2023-10-16T15:53:41.375870+0000","last_fullsized":"2023-10-16T15:53:41.375870+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462177+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:29.462177+0000","last_peered":"2023-10-16T15:53:29.462177+0000","last_clean":"2023-10-16T15:53:29.462177+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:29.462177+0000","last_undegraded":"2023-10-16T15:53:29.462177+0000","last_fullsized":"2023-10-16T15:53:29.462177+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.407785+0000","last_change":"2023-10-16T15:53:40.378428+0000","last_active":"2023-10-16T15:53:41.407785+0000","last_peered":"2023-10-16T15:53:41.407785+0000","last_clean":"2023-10-16T15:53:41.407785+0000","last_became_active":"2023-10-16T15:53:40.378077+0000","last_became_peered":"2023-10-16T15:53:40.378077+0000","last_unstale":"2023-10-16T15:53:41.407785+0000","last_undegraded":"2023-10-16T15:53:41.407785+0000","last_fullsized":"2023-10-16T15:53:41.407785+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410289+0000","last_change":"2023-10-16T15:53:41.399412+0000","last_active":"2023-10-16T15:53:42.410289+0000","last_peered":"2023-10-16T15:53:42.410289+0000","last_clean":"2023-10-16T15:53:42.410289+0000","last_became_active":"2023-10-16T15:53:41.399125+0000","last_became_peered":"2023-10-16T15:53:41.399125+0000","last_unstale":"2023-10-16T15:53:42.410289+0000","last_undegraded":"2023-10-16T15:53:42.410289+0000","last_fullsized":"2023-10-16T15:53:42.410289+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376676+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:41.376676+0000","last_peered":"2023-10-16T15:53:41.376676+0000","last_clean":"2023-10-16T15:53:41.376676+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:41.376676+0000","last_undegraded":"2023-10-16T15:53:41.376676+0000","last_fullsized":"2023-10-16T15:53:41.376676+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394140+0000","last_change":"2023-10-16T15:53:40.377426+0000","last_active":"2023-10-16T15:53:41.394140+0000","last_peered":"2023-10-16T15:53:41.394140+0000","last_clean":"2023-10-16T15:53:41.394140+0000","last_became_active":"2023-10-16T15:53:40.377160+0000","last_became_peered":"2023-10-16T15:53:40.377160+0000","last_unstale":"2023-10-16T15:53:41.394140+0000","last_undegraded":"2023-10-16T15:53:41.394140+0000","last_fullsized":"2023-10-16T15:53:41.394140+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:38.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408560+0000","last_change":"2023-10-16T15:53:41.400288+0000","last_active":"2023-10-16T15:53:42.408560+0000","last_peered":"2023-10-16T15:53:42.408560+0000","last_clean":"2023-10-16T15:53:42.408560+0000","last_became_active":"2023-10-16T15:53:41.400031+0000","last_became_peered":"2023-10-16T15:53:41.400031+0000","last_unstale":"2023-10-16T15:53:42.408560+0000","last_undegraded":"2023-10-16T15:53:42.408560+0000","last_fullsized":"2023-10-16T15:53:42.408560+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393978+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:41.393978+0000","last_peered":"2023-10-16T15:53:41.393978+0000","last_clean":"2023-10-16T15:53:41.393978+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:41.393978+0000","last_undegraded":"2023-10-16T15:53:41.393978+0000","last_fullsized":"2023-10-16T15:53:41.393978+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392991+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:41.392991+0000","last_peered":"2023-10-16T15:53:41.392991+0000","last_clean":"2023-10-16T15:53:41.392991+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:41.392991+0000","last_undegraded":"2023-10-16T15:53:41.392991+0000","last_fullsized":"2023-10-16T15:53:41.392991+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.347704+0000","last_change":"2023-10-16T15:53:39.347704+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.347704+0000","last_undegraded":"2023-10-16T15:53:39.347704+0000","last_fullsized":"2023-10-16T15:53:39.347704+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409916+0000","last_change":"2023-10-16T15:53:41.413217+0000","last_active":"2023-10-16T15:53:42.409916+0000","last_peered":"2023-10-16T15:53:42.409916+0000","last_clean":"2023-10-16T15:53:42.409916+0000","last_became_active":"2023-10-16T15:53:41.411878+0000","last_became_peered":"2023-10-16T15:53:41.411878+0000","last_unstale":"2023-10-16T15:53:42.409916+0000","last_undegraded":"2023-10-16T15:53:42.409916+0000","last_fullsized":"2023-10-16T15:53:42.409916+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384277+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:41.384277+0000","last_peered":"2023-10-16T15:53:41.384277+0000","last_clean":"2023-10-16T15:53:41.384277+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:41.384277+0000","last_undegraded":"2023-10-16T15:53:41.384277+0000","last_fullsized":"2023-10-16T15:53:41.384277+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394436+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:41.394436+0000","last_peered":"2023-10-16T15:53:41.394436+0000","last_clean":"2023-10-16T15:53:41.394436+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:41.394436+0000","last_undegraded":"2023-10-16T15:53:41.394436+0000","last_fullsized":"2023-10-16T15:53:41.394436+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.356230+0000","last_change":"2023-10-16T15:53:39.356230+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.356230+0000","last_undegraded":"2023-10-16T15:53:39.356230+0000","last_fullsized":"2023-10-16T15:53:39.356230+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013025+0000","last_change":"2023-10-16T15:53:41.397581+0000","last_active":"2023-10-16T15:53:43.013025+0000","last_peered":"2023-10-16T15:53:43.013025+0000","last_clean":"2023-10-16T15:53:43.013025+0000","last_became_active":"2023-10-16T15:53:41.397301+0000","last_became_peered":"2023-10-16T15:53:41.397301+0000","last_unstale":"2023-10-16T15:53:43.013025+0000","last_undegraded":"2023-10-16T15:53:43.013025+0000","last_fullsized":"2023-10-16T15:53:43.013025+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:20.816418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394523+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:41.394523+0000","last_peered":"2023-10-16T15:53:41.394523+0000","last_clean":"2023-10-16T15:53:41.394523+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:41.394523+0000","last_undegraded":"2023-10-16T15:53:41.394523+0000","last_fullsized":"2023-10-16T15:53:41.394523+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.411952+0000","last_change":"2023-10-16T15:53:40.376976+0000","last_active":"2023-10-16T15:53:41.411952+0000","last_peered":"2023-10-16T15:53:41.411952+0000","last_clean":"2023-10-16T15:53:41.411952+0000","last_became_active":"2023-10-16T15:53:40.376700+0000","last_became_peered":"2023-10-16T15:53:40.376700+0000","last_unstale":"2023-10-16T15:53:41.411952+0000","last_undegraded":"2023-10-16T15:53:41.411952+0000","last_fullsized":"2023-10-16T15:53:41.411952+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013467+0000","last_change":"2023-10-16T15:53:41.410633+0000","last_active":"2023-10-16T15:53:43.013467+0000","last_peered":"2023-10-16T15:53:43.013467+0000","last_clean":"2023-10-16T15:53:43.013467+0000","last_became_active":"2023-10-16T15:53:41.409266+0000","last_became_peered":"2023-10-16T15:53:41.409266+0000","last_unstale":"2023-10-16T15:53:43.013467+0000","last_undegraded":"2023-10-16T15:53:43.013467+0000","last_fullsized":"2023-10-16T15:53:43.013467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:52.749663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462595+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:29.462595+0000","last_peered":"2023-10-16T15:53:29.462595+0000","last_clean":"2023-10-16T15:53:29.462595+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:29.462595+0000","last_undegraded":"2023-10-16T15:53:29.462595+0000","last_fullsized":"2023-10-16T15:53:29.462595+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462687+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:29.462687+0000","last_peered":"2023-10-16T15:53:29.462687+0000","last_clean":"2023-10-16T15:53:29.462687+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:29.462687+0000","last_undegraded":"2023-10-16T15:53:29.462687+0000","last_fullsized":"2023-10-16T15:53:29.462687+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384497+0000","last_change":"2023-10-16T15:53:40.375609+0000","last_active":"2023-10-16T15:53:41.384497+0000","last_peered":"2023-10-16T15:53:41.384497+0000","last_clean":"2023-10-16T15:53:41.384497+0000","last_became_active":"2023-10-16T15:53:40.375039+0000","last_became_peered":"2023-10-16T15:53:40.375039+0000","last_unstale":"2023-10-16T15:53:41.384497+0000","last_undegraded":"2023-10-16T15:53:41.384497+0000","last_fullsized":"2023-10-16T15:53:41.384497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:02.147563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.328614+0000","last_change":"2023-10-16T15:53:41.399266+0000","last_active":"2023-10-16T15:53:43.328614+0000","last_peered":"2023-10-16T15:53:43.328614+0000","last_clean":"2023-10-16T15:53:43.328614+0000","last_became_active":"2023-10-16T15:53:41.398177+0000","last_became_peered":"2023-10-16T15:53:41.398177+0000","last_unstale":"2023-10-16T15:53:43.328614+0000","last_undegraded":"2023-10-16T15:53:43.328614+0000","last_fullsized":"2023-10-16T15:53:43.328614+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.144341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385470+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:41.385470+0000","last_peered":"2023-10-16T15:53:41.385470+0000","last_clean":"2023-10-16T15:53:41.385470+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:41.385470+0000","last_undegraded":"2023-10-16T15:53:41.385470+0000","last_fullsized":"2023-10-16T15:53:41.385470+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462492+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:29.462492+0000","last_peered":"2023-10-16T15:53:29.462492+0000","last_clean":"2023-10-16T15:53:29.462492+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:29.462492+0000","last_undegraded":"2023-10-16T15:53:29.462492+0000","last_fullsized":"2023-10-16T15:53:29.462492+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.355830+0000","last_change":"2023-10-16T15:53:39.355830+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.355830+0000","last_undegraded":"2023-10-16T15:53:39.355830+0000","last_fullsized":"2023-10-16T15:53:39.355830+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380707+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:41.380707+0000","last_peered":"2023-10-16T15:53:41.380707+0000","last_clean":"2023-10-16T15:53:41.380707+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:41.380707+0000","last_undegraded":"2023-10-16T15:53:41.380707+0000","last_fullsized":"2023-10-16T15:53:41.380707+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374760+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:41.374760+0000","last_peered":"2023-10-16T15:53:41.374760+0000","last_clean":"2023-10-16T15:53:41.374760+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:41.374760+0000","last_undegraded":"2023-10-16T15:53:41.374760+0000","last_fullsized":"2023-10-16T15:53:41.374760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375756+0000","last_change":"2023-10-16T15:53:40.377615+0000","last_active":"2023-10-16T15:53:41.375756+0000","last_peered":"2023-10-16T15:53:41.375756+0000","last_clean":"2023-10-16T15:53:41.375756+0000","last_became_active":"2023-10-16T15:53:40.377342+0000","last_became_peered":"2023-10-16T15:53:40.377342+0000","last_unstale":"2023-10-16T15:53:41.375756+0000","last_undegraded":"2023-10-16T15:53:41.375756+0000","last_fullsized":"2023-10-16T15:53:41.375756+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388735+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:41.388735+0000","last_peered":"2023-10-16T15:53:41.388735+0000","last_clean":"2023-10-16T15:53:41.388735+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:41.388735+0000","last_undegraded":"2023-10-16T15:53:41.388735+0000","last_fullsized":"2023-10-16T15:53:41.388735+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380271+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:41.380271+0000","last_peered":"2023-10-16T15:53:41.380271+0000","last_clean":"2023-10-16T15:53:41.380271+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:41.380271+0000","last_undegraded":"2023-10-16T15:53:41.380271+0000","last_fullsized":"2023-10-16T15:53:41.380271+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.347002+0000","last_change":"2023-10-16T15:53:39.347002+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.347002+0000","last_undegraded":"2023-10-16T15:53:39.347002+0000","last_fullsized":"2023-10-16T15:53:39.347002+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461781+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:29.461781+0000","last_peered":"2023-10-16T15:53:29.461781+0000","last_clean":"2023-10-16T15:53:29.461781+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:29.461781+0000","last_undegraded":"2023-10-16T15:53:29.461781+0000","last_fullsized":"2023-10-16T15:53:29.461781+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393875+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:41.393875+0000","last_peered":"2023-10-16T15:53:41.393875+0000","last_clean":"2023-10-16T15:53:41.393875+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:41.393875+0000","last_undegraded":"2023-10-16T15:53:41.393875+0000","last_fullsized":"2023-10-16T15:53:41.393875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"228'6","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414139+0000","last_change":"2023-10-16T15:53:40.376773+0000","last_active":"2023-10-16T15:53:41.414139+0000","last_peered":"2023-10-16T15:53:41.414139+0000","last_clean":"2023-10-16T15:53:41.414139+0000","last_became_active":"2023-10-16T15:53:40.376123+0000","last_became_peered":"2023-10-16T15:53:40.376123+0000","last_unstale":"2023-10-16T15:53:41.414139+0000","last_undegraded":"2023-10-16T15:53:41.414139+0000","last_fullsized":"2023-10-16T15:53:41.414139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:19.218849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461826+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:29.461826+0000","last_peered":"2023-10-16T15:53:29.461826+0000","last_clean":"2023-10-16T15:53:29.461826+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:29.461826+0000","last_undegraded":"2023-10-16T15:53:29.461826+0000","last_fullsized":"2023-10-16T15:53:29.461826+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383904+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:41.383904+0000","last_peered":"2023-10-16T15:53:41.383904+0000","last_clean":"2023-10-16T15:53:41.383904+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:41.383904+0000","last_undegraded":"2023-10-16T15:53:41.383904+0000","last_fullsized":"2023-10-16T15:53:41.383904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"228'3","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402070+0000","last_change":"2023-10-16T15:53:40.365148+0000","last_active":"2023-10-16T15:53:41.402070+0000","last_peered":"2023-10-16T15:53:41.402070+0000","last_clean":"2023-10-16T15:53:41.402070+0000","last_became_active":"2023-10-16T15:53:40.364874+0000","last_became_peered":"2023-10-16T15:53:40.364874+0000","last_unstale":"2023-10-16T15:53:41.402070+0000","last_undegraded":"2023-10-16T15:53:41.402070+0000","last_fullsized":"2023-10-16T15:53:41.402070+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:39.387182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411499+0000","last_change":"2023-10-16T15:53:41.405376+0000","last_active":"2023-10-16T15:53:42.411499+0000","last_peered":"2023-10-16T15:53:42.411499+0000","last_clean":"2023-10-16T15:53:42.411499+0000","last_became_active":"2023-10-16T15:53:41.404697+0000","last_became_peered":"2023-10-16T15:53:41.404697+0000","last_unstale":"2023-10-16T15:53:42.411499+0000","last_undegraded":"2023-10-16T15:53:42.411499+0000","last_fullsized":"2023-10-16T15:53:42.411499+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"225'8","reported_seq":21,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:36.400354+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:36.400354+0000","last_peered":"2023-10-16T15:53:36.400354+0000","last_clean":"2023-10-16T15:53:36.400354+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:36.400354+0000","last_undegraded":"2023-10-16T15:53:36.400354+0000","last_fullsized":"2023-10-16T15:53:36.400354+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376419+0000","last_change":"2023-10-16T15:53:40.387700+0000","last_active":"2023-10-16T15:53:41.376419+0000","last_peered":"2023-10-16T15:53:41.376419+0000","last_clean":"2023-10-16T15:53:41.376419+0000","last_became_active":"2023-10-16T15:53:40.387431+0000","last_became_peered":"2023-10-16T15:53:40.387431+0000","last_unstale":"2023-10-16T15:53:41.376419+0000","last_undegraded":"2023-10-16T15:53:41.376419+0000","last_fullsized":"2023-10-16T15:53:41.376419+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:19.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418316+0000","last_change":"2023-10-16T15:53:41.385042+0000","last_active":"2023-10-16T15:53:43.418316+0000","last_peered":"2023-10-16T15:53:43.418316+0000","last_clean":"2023-10-16T15:53:43.418316+0000","last_became_active":"2023-10-16T15:53:41.384431+0000","last_became_peered":"2023-10-16T15:53:41.384431+0000","last_unstale":"2023-10-16T15:53:43.418316+0000","last_undegraded":"2023-10-16T15:53:43.418316+0000","last_fullsized":"2023-10-16T15:53:43.418316+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394713+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:41.394713+0000","last_peered":"2023-10-16T15:53:41.394713+0000","last_clean":"2023-10-16T15:53:41.394713+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:41.394713+0000","last_undegraded":"2023-10-16T15:53:41.394713+0000","last_fullsized":"2023-10-16T15:53:41.394713+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463205+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:29.463205+0000","last_peered":"2023-10-16T15:53:29.463205+0000","last_clean":"2023-10-16T15:53:29.463205+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:29.463205+0000","last_undegraded":"2023-10-16T15:53:29.463205+0000","last_fullsized":"2023-10-16T15:53:29.463205+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.408329+0000","last_change":"2023-10-16T15:53:40.386183+0000","last_active":"2023-10-16T15:53:41.408329+0000","last_peered":"2023-10-16T15:53:41.408329+0000","last_clean":"2023-10-16T15:53:41.408329+0000","last_became_active":"2023-10-16T15:53:40.385897+0000","last_became_peered":"2023-10-16T15:53:40.385897+0000","last_unstale":"2023-10-16T15:53:41.408329+0000","last_undegraded":"2023-10-16T15:53:41.408329+0000","last_fullsized":"2023-10-16T15:53:41.408329+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:27.631953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461915+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:29.461915+0000","last_peered":"2023-10-16T15:53:29.461915+0000","last_clean":"2023-10-16T15:53:29.461915+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:29.461915+0000","last_undegraded":"2023-10-16T15:53:29.461915+0000","last_fullsized":"2023-10-16T15:53:29.461915+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387083+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:41.387083+0000","last_peered":"2023-10-16T15:53:41.387083+0000","last_clean":"2023-10-16T15:53:41.387083+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:41.387083+0000","last_undegraded":"2023-10-16T15:53:41.387083+0000","last_fullsized":"2023-10-16T15:53:41.387083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345091+0000","last_change":"2023-10-16T15:53:39.345091+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345091+0000","last_undegraded":"2023-10-16T15:53:39.345091+0000","last_fullsized":"2023-10-16T15:53:39.345091+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407893+0000","last_change":"2023-10-16T15:53:41.414765+0000","last_active":"2023-10-16T15:53:42.407893+0000","last_peered":"2023-10-16T15:53:42.407893+0000","last_clean":"2023-10-16T15:53:42.407893+0000","last_became_active":"2023-10-16T15:53:41.414484+0000","last_became_peered":"2023-10-16T15:53:41.414484+0000","last_unstale":"2023-10-16T15:53:42.407893+0000","last_undegraded":"2023-10-16T15:53:42.407893+0000","last_fullsized":"2023-10-16T15:53:42.407893+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385369+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:41.385369+0000","last_peered":"2023-10-16T15:53:41.385369+0000","last_clean":"2023-10-16T15:53:41.385369+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:41.385369+0000","last_undegraded":"2023-10-16T15:53:41.385369+0000","last_fullsized":"2023-10-16T15:53:41.385369+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463287+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:29.463287+0000","last_peered":"2023-10-16T15:53:29.463287+0000","last_clean":"2023-10-16T15:53:29.463287+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:29.463287+0000","last_undegraded":"2023-10-16T15:53:29.463287+0000","last_fullsized":"2023-10-16T15:53:29.463287+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382349+0000","last_change":"2023-10-16T15:53:40.376076+0000","last_active":"2023-10-16T15:53:41.382349+0000","last_peered":"2023-10-16T15:53:41.382349+0000","last_clean":"2023-10-16T15:53:41.382349+0000","last_became_active":"2023-10-16T15:53:40.375798+0000","last_became_peered":"2023-10-16T15:53:40.375798+0000","last_unstale":"2023-10-16T15:53:41.382349+0000","last_undegraded":"2023-10-16T15:53:41.382349+0000","last_fullsized":"2023-10-16T15:53:41.382349+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330851+0000","last_change":"2023-10-16T15:53:41.409878+0000","last_active":"2023-10-16T15:53:43.330851+0000","last_peered":"2023-10-16T15:53:43.330851+0000","last_clean":"2023-10-16T15:53:43.330851+0000","last_became_active":"2023-10-16T15:53:41.409594+0000","last_became_peered":"2023-10-16T15:53:41.409594+0000","last_unstale":"2023-10-16T15:53:43.330851+0000","last_undegraded":"2023-10-16T15:53:43.330851+0000","last_fullsized":"2023-10-16T15:53:43.330851+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:10.790473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345082+0000","last_change":"2023-10-16T15:53:39.345082+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345082+0000","last_undegraded":"2023-10-16T15:53:39.345082+0000","last_fullsized":"2023-10-16T15:53:39.345082+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461553+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:29.461553+0000","last_peered":"2023-10-16T15:53:29.461553+0000","last_clean":"2023-10-16T15:53:29.461553+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:29.461553+0000","last_undegraded":"2023-10-16T15:53:29.461553+0000","last_fullsized":"2023-10-16T15:53:29.461553+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463088+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:29.463088+0000","last_peered":"2023-10-16T15:53:29.463088+0000","last_clean":"2023-10-16T15:53:29.463088+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:29.463088+0000","last_undegraded":"2023-10-16T15:53:29.463088+0000","last_fullsized":"2023-10-16T15:53:29.463088+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410670+0000","last_change":"2023-10-16T15:53:41.411697+0000","last_active":"2023-10-16T15:53:42.410670+0000","last_peered":"2023-10-16T15:53:42.410670+0000","last_clean":"2023-10-16T15:53:42.410670+0000","last_became_active":"2023-10-16T15:53:41.410477+0000","last_became_peered":"2023-10-16T15:53:41.410477+0000","last_unstale":"2023-10-16T15:53:42.410670+0000","last_undegraded":"2023-10-16T15:53:42.410670+0000","last_fullsized":"2023-10-16T15:53:42.410670+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:52.524535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"228'4","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.418437+0000","last_change":"2023-10-16T15:53:40.380112+0000","last_active":"2023-10-16T15:53:41.418437+0000","last_peered":"2023-10-16T15:53:41.418437+0000","last_clean":"2023-10-16T15:53:41.418437+0000","last_became_active":"2023-10-16T15:53:40.379085+0000","last_became_peered":"2023-10-16T15:53:40.379085+0000","last_unstale":"2023-10-16T15:53:41.418437+0000","last_undegraded":"2023-10-16T15:53:41.418437+0000","last_fullsized":"2023-10-16T15:53:41.418437+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:48.756275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382648+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:41.382648+0000","last_peered":"2023-10-16T15:53:41.382648+0000","last_clean":"2023-10-16T15:53:41.382648+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:41.382648+0000","last_undegraded":"2023-10-16T15:53:41.382648+0000","last_fullsized":"2023-10-16T15:53:41.382648+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379944+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:41.379944+0000","last_peered":"2023-10-16T15:53:41.379944+0000","last_clean":"2023-10-16T15:53:41.379944+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:41.379944+0000","last_undegraded":"2023-10-16T15:53:41.379944+0000","last_fullsized":"2023-10-16T15:53:41.379944+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384001+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:41.384001+0000","last_peered":"2023-10-16T15:53:41.384001+0000","last_clean":"2023-10-16T15:53:41.384001+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:41.384001+0000","last_undegraded":"2023-10-16T15:53:41.384001+0000","last_fullsized":"2023-10-16T15:53:41.384001+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463135+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:29.463135+0000","last_peered":"2023-10-16T15:53:29.463135+0000","last_clean":"2023-10-16T15:53:29.463135+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:29.463135+0000","last_undegraded":"2023-10-16T15:53:29.463135+0000","last_fullsized":"2023-10-16T15:53:29.463135+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345766+0000","last_change":"2023-10-16T15:53:39.345766+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345766+0000","last_undegraded":"2023-10-16T15:53:39.345766+0000","last_fullsized":"2023-10-16T15:53:39.345766+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013915+0000","last_change":"2023-10-16T15:53:41.405768+0000","last_active":"2023-10-16T15:53:43.013915+0000","last_peered":"2023-10-16T15:53:43.013915+0000","last_clean":"2023-10-16T15:53:43.013915+0000","last_became_active":"2023-10-16T15:53:41.405230+0000","last_became_peered":"2023-10-16T15:53:41.405230+0000","last_unstale":"2023-10-16T15:53:43.013915+0000","last_undegraded":"2023-10-16T15:53:43.013915+0000","last_fullsized":"2023-10-16T15:53:43.013915+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:43.313902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463229+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:29.463229+0000","last_peered":"2023-10-16T15:53:29.463229+0000","last_clean":"2023-10-16T15:53:29.463229+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:29.463229+0000","last_undegraded":"2023-10-16T15:53:29.463229+0000","last_fullsized":"2023-10-16T15:53:29.463229+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394353+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:41.394353+0000","last_peered":"2023-10-16T15:53:41.394353+0000","last_clean":"2023-10-16T15:53:41.394353+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:41.394353+0000","last_undegraded":"2023-10-16T15:53:41.394353+0000","last_fullsized":"2023-10-16T15:53:41.394353+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.404498+0000","last_change":"2023-10-16T15:53:40.381325+0000","last_active":"2023-10-16T15:53:41.404498+0000","last_peered":"2023-10-16T15:53:41.404498+0000","last_clean":"2023-10-16T15:53:41.404498+0000","last_became_active":"2023-10-16T15:53:40.380822+0000","last_became_peered":"2023-10-16T15:53:40.380822+0000","last_unstale":"2023-10-16T15:53:41.404498+0000","last_undegraded":"2023-10-16T15:53:41.404498+0000","last_fullsized":"2023-10-16T15:53:41.404498+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385254+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:41.385254+0000","last_peered":"2023-10-16T15:53:41.385254+0000","last_clean":"2023-10-16T15:53:41.385254+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:41.385254+0000","last_undegraded":"2023-10-16T15:53:41.385254+0000","last_fullsized":"2023-10-16T15:53:41.385254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374210+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:41.374210+0000","last_peered":"2023-10-16T15:53:41.374210+0000","last_clean":"2023-10-16T15:53:41.374210+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:41.374210+0000","last_undegraded":"2023-10-16T15:53:41.374210+0000","last_fullsized":"2023-10-16T15:53:41.374210+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.409948+0000","last_change":"2023-10-16T15:53:40.377357+0000","last_active":"2023-10-16T15:53:41.409948+0000","last_peered":"2023-10-16T15:53:41.409948+0000","last_clean":"2023-10-16T15:53:41.409948+0000","last_became_active":"2023-10-16T15:53:40.377070+0000","last_became_peered":"2023-10-16T15:53:40.377070+0000","last_unstale":"2023-10-16T15:53:41.409948+0000","last_undegraded":"2023-10-16T15:53:41.409948+0000","last_fullsized":"2023-10-16T15:53:41.409948+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462545+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:29.462545+0000","last_peered":"2023-10-16T15:53:29.462545+0000","last_clean":"2023-10-16T15:53:29.462545+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:29.462545+0000","last_undegraded":"2023-10-16T15:53:29.462545+0000","last_fullsized":"2023-10-16T15:53:29.462545+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461956+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:29.461956+0000","last_peered":"2023-10-16T15:53:29.461956+0000","last_clean":"2023-10-16T15:53:29.461956+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:29.461956+0000","last_undegraded":"2023-10-16T15:53:29.461956+0000","last_fullsized":"2023-10-16T15:53:29.461956+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391139+0000","last_change":"2023-10-16T15:53:40.380015+0000","last_active":"2023-10-16T15:53:41.391139+0000","last_peered":"2023-10-16T15:53:41.391139+0000","last_clean":"2023-10-16T15:53:41.391139+0000","last_became_active":"2023-10-16T15:53:40.379716+0000","last_became_peered":"2023-10-16T15:53:40.379716+0000","last_unstale":"2023-10-16T15:53:41.391139+0000","last_undegraded":"2023-10-16T15:53:41.391139+0000","last_fullsized":"2023-10-16T15:53:41.391139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402124+0000","last_change":"2023-10-16T15:53:41.403677+0000","last_active":"2023-10-16T15:53:42.402124+0000","last_peered":"2023-10-16T15:53:42.402124+0000","last_clean":"2023-10-16T15:53:42.402124+0000","last_became_active":"2023-10-16T15:53:41.403372+0000","last_became_peered":"2023-10-16T15:53:41.403372+0000","last_unstale":"2023-10-16T15:53:42.402124+0000","last_undegraded":"2023-10-16T15:53:42.402124+0000","last_fullsized":"2023-10-16T15:53:42.402124+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396765+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:41.396765+0000","last_peered":"2023-10-16T15:53:41.396765+0000","last_clean":"2023-10-16T15:53:41.396765+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.396765+0000","last_undegraded":"2023-10-16T15:53:41.396765+0000","last_fullsized":"2023-10-16T15:53:41.396765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462449+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:29.462449+0000","last_peered":"2023-10-16T15:53:29.462449+0000","last_clean":"2023-10-16T15:53:29.462449+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:29.462449+0000","last_undegraded":"2023-10-16T15:53:29.462449+0000","last_fullsized":"2023-10-16T15:53:29.462449+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.356911+0000","last_change":"2023-10-16T15:53:39.356911+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.356911+0000","last_undegraded":"2023-10-16T15:53:39.356911+0000","last_fullsized":"2023-10-16T15:53:39.356911+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330261+0000","last_change":"2023-10-16T15:53:41.397528+0000","last_active":"2023-10-16T15:53:43.330261+0000","last_peered":"2023-10-16T15:53:43.330261+0000","last_clean":"2023-10-16T15:53:43.330261+0000","last_became_active":"2023-10-16T15:53:41.397247+0000","last_became_peered":"2023-10-16T15:53:41.397247+0000","last_unstale":"2023-10-16T15:53:43.330261+0000","last_undegraded":"2023-10-16T15:53:43.330261+0000","last_fullsized":"2023-10-16T15:53:43.330261+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:21.337037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.373989+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:41.373989+0000","last_peered":"2023-10-16T15:53:41.373989+0000","last_clean":"2023-10-16T15:53:41.373989+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:41.373989+0000","last_undegraded":"2023-10-16T15:53:41.373989+0000","last_fullsized":"2023-10-16T15:53:41.373989+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388628+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:41.388628+0000","last_peered":"2023-10-16T15:53:41.388628+0000","last_clean":"2023-10-16T15:53:41.388628+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:41.388628+0000","last_undegraded":"2023-10-16T15:53:41.388628+0000","last_fullsized":"2023-10-16T15:53:41.388628+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402941+0000","last_change":"2023-10-16T15:53:41.398094+0000","last_active":"2023-10-16T15:53:42.402941+0000","last_peered":"2023-10-16T15:53:42.402941+0000","last_clean":"2023-10-16T15:53:42.402941+0000","last_became_active":"2023-10-16T15:53:41.396310+0000","last_became_peered":"2023-10-16T15:53:41.396310+0000","last_unstale":"2023-10-16T15:53:42.402941+0000","last_undegraded":"2023-10-16T15:53:42.402941+0000","last_fullsized":"2023-10-16T15:53:42.402941+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462282+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:29.462282+0000","last_peered":"2023-10-16T15:53:29.462282+0000","last_clean":"2023-10-16T15:53:29.462282+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:29.462282+0000","last_undegraded":"2023-10-16T15:53:29.462282+0000","last_fullsized":"2023-10-16T15:53:29.462282+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395859+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:41.395859+0000","last_peered":"2023-10-16T15:53:41.395859+0000","last_clean":"2023-10-16T15:53:41.395859+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:41.395859+0000","last_undegraded":"2023-10-16T15:53:41.395859+0000","last_fullsized":"2023-10-16T15:53:41.395859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393894+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:41.393894+0000","last_peered":"2023-10-16T15:53:41.393894+0000","last_clean":"2023-10-16T15:53:41.393894+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:41.393894+0000","last_undegraded":"2023-10-16T15:53:41.393894+0000","last_fullsized":"2023-10-16T15:53:41.393894+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418360+0000","last_change":"2023-10-16T15:53:41.400386+0000","last_active":"2023-10-16T15:53:43.418360+0000","last_peered":"2023-10-16T15:53:43.418360+0000","last_clean":"2023-10-16T15:53:43.418360+0000","last_became_active":"2023-10-16T15:53:41.400083+0000","last_became_peered":"2023-10-16T15:53:41.400083+0000","last_unstale":"2023-10-16T15:53:43.418360+0000","last_undegraded":"2023-10-16T15:53:43.418360+0000","last_fullsized":"2023-10-16T15:53:43.418360+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375718+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:41.375718+0000","last_peered":"2023-10-16T15:53:41.375718+0000","last_clean":"2023-10-16T15:53:41.375718+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:41.375718+0000","last_undegraded":"2023-10-16T15:53:41.375718+0000","last_fullsized":"2023-10-16T15:53:41.375718+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012816+0000","last_change":"2023-10-16T15:53:41.399893+0000","last_active":"2023-10-16T15:53:43.012816+0000","last_peered":"2023-10-16T15:53:43.012816+0000","last_clean":"2023-10-16T15:53:43.012816+0000","last_became_active":"2023-10-16T15:53:41.398053+0000","last_became_peered":"2023-10-16T15:53:41.398053+0000","last_unstale":"2023-10-16T15:53:43.012816+0000","last_undegraded":"2023-10-16T15:53:43.012816+0000","last_fullsized":"2023-10-16T15:53:43.012816+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:18.848509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462645+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:29.462645+0000","last_peered":"2023-10-16T15:53:29.462645+0000","last_clean":"2023-10-16T15:53:29.462645+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:29.462645+0000","last_undegraded":"2023-10-16T15:53:29.462645+0000","last_fullsized":"2023-10-16T15:53:29.462645+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404083+0000","last_change":"2023-10-16T15:53:41.406721+0000","last_active":"2023-10-16T15:53:42.404083+0000","last_peered":"2023-10-16T15:53:42.404083+0000","last_clean":"2023-10-16T15:53:42.404083+0000","last_became_active":"2023-10-16T15:53:41.404619+0000","last_became_peered":"2023-10-16T15:53:41.404619+0000","last_unstale":"2023-10-16T15:53:42.404083+0000","last_undegraded":"2023-10-16T15:53:42.404083+0000","last_fullsized":"2023-10-16T15:53:42.404083+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463041+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:29.463041+0000","last_peered":"2023-10-16T15:53:29.463041+0000","last_clean":"2023-10-16T15:53:29.463041+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:29.463041+0000","last_undegraded":"2023-10-16T15:53:29.463041+0000","last_fullsized":"2023-10-16T15:53:29.463041+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461607+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:29.461607+0000","last_peered":"2023-10-16T15:53:29.461607+0000","last_clean":"2023-10-16T15:53:29.461607+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:29.461607+0000","last_undegraded":"2023-10-16T15:53:29.461607+0000","last_fullsized":"2023-10-16T15:53:29.461607+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402772+0000","last_change":"2023-10-16T15:53:41.399691+0000","last_active":"2023-10-16T15:53:42.402772+0000","last_peered":"2023-10-16T15:53:42.402772+0000","last_clean":"2023-10-16T15:53:42.402772+0000","last_became_active":"2023-10-16T15:53:41.397849+0000","last_became_peered":"2023-10-16T15:53:41.397849+0000","last_unstale":"2023-10-16T15:53:42.402772+0000","last_undegraded":"2023-10-16T15:53:42.402772+0000","last_fullsized":"2023-10-16T15:53:42.402772+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461742+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:29.461742+0000","last_peered":"2023-10-16T15:53:29.461742+0000","last_clean":"2023-10-16T15:53:29.461742+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:29.461742+0000","last_undegraded":"2023-10-16T15:53:29.461742+0000","last_fullsized":"2023-10-16T15:53:29.461742+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014730+0000","last_change":"2023-10-16T15:53:41.410559+0000","last_active":"2023-10-16T15:53:43.014730+0000","last_peered":"2023-10-16T15:53:43.014730+0000","last_clean":"2023-10-16T15:53:43.014730+0000","last_became_active":"2023-10-16T15:53:41.410258+0000","last_became_peered":"2023-10-16T15:53:41.410258+0000","last_unstale":"2023-10-16T15:53:43.014730+0000","last_undegraded":"2023-10-16T15:53:43.014730+0000","last_fullsized":"2023-10-16T15:53:43.014730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:25.813239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463112+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:29.463112+0000","last_peered":"2023-10-16T15:53:29.463112+0000","last_clean":"2023-10-16T15:53:29.463112+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:29.463112+0000","last_undegraded":"2023-10-16T15:53:29.463112+0000","last_fullsized":"2023-10-16T15:53:29.463112+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418981+0000","last_change":"2023-10-16T15:53:41.416192+0000","last_active":"2023-10-16T15:53:43.418981+0000","last_peered":"2023-10-16T15:53:43.418981+0000","last_clean":"2023-10-16T15:53:43.418981+0000","last_became_active":"2023-10-16T15:53:41.415862+0000","last_became_peered":"2023-10-16T15:53:41.415862+0000","last_unstale":"2023-10-16T15:53:43.418981+0000","last_undegraded":"2023-10-16T15:53:43.418981+0000","last_fullsized":"2023-10-16T15:53:43.418981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383281+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:41.383281+0000","last_peered":"2023-10-16T15:53:41.383281+0000","last_clean":"2023-10-16T15:53:41.383281+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:41.383281+0000","last_undegraded":"2023-10-16T15:53:41.383281+0000","last_fullsized":"2023-10-16T15:53:41.383281+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013226+0000","last_change":"2023-10-16T15:53:41.409431+0000","last_active":"2023-10-16T15:53:43.013226+0000","last_peered":"2023-10-16T15:53:43.013226+0000","last_clean":"2023-10-16T15:53:43.013226+0000","last_became_active":"2023-10-16T15:53:41.409023+0000","last_became_peered":"2023-10-16T15:53:41.409023+0000","last_unstale":"2023-10-16T15:53:43.013226+0000","last_undegraded":"2023-10-16T15:53:43.013226+0000","last_fullsized":"2023-10-16T15:53:43.013226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:31.127234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394621+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:41.394621+0000","last_peered":"2023-10-16T15:53:41.394621+0000","last_clean":"2023-10-16T15:53:41.394621+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:41.394621+0000","last_undegraded":"2023-10-16T15:53:41.394621+0000","last_fullsized":"2023-10-16T15:53:41.394621+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012572+0000","last_change":"2023-10-16T15:53:41.397505+0000","last_active":"2023-10-16T15:53:43.012572+0000","last_peered":"2023-10-16T15:53:43.012572+0000","last_clean":"2023-10-16T15:53:43.012572+0000","last_became_active":"2023-10-16T15:53:41.397198+0000","last_became_peered":"2023-10-16T15:53:41.397198+0000","last_unstale":"2023-10-16T15:53:43.012572+0000","last_undegraded":"2023-10-16T15:53:43.012572+0000","last_fullsized":"2023-10-16T15:53:43.012572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:07.665494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394898+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:41.394898+0000","last_peered":"2023-10-16T15:53:41.394898+0000","last_clean":"2023-10-16T15:53:41.394898+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:41.394898+0000","last_undegraded":"2023-10-16T15:53:41.394898+0000","last_fullsized":"2023-10-16T15:53:41.394898+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374811+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:41.374811+0000","last_peered":"2023-10-16T15:53:41.374811+0000","last_clean":"2023-10-16T15:53:41.374811+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:41.374811+0000","last_undegraded":"2023-10-16T15:53:41.374811+0000","last_fullsized":"2023-10-16T15:53:41.374811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404049+0000","last_change":"2023-10-16T15:53:41.397057+0000","last_active":"2023-10-16T15:53:42.404049+0000","last_peered":"2023-10-16T15:53:42.404049+0000","last_clean":"2023-10-16T15:53:42.404049+0000","last_became_active":"2023-10-16T15:53:41.396163+0000","last_became_peered":"2023-10-16T15:53:41.396163+0000","last_unstale":"2023-10-16T15:53:42.404049+0000","last_undegraded":"2023-10-16T15:53:42.404049+0000","last_fullsized":"2023-10-16T15:53:42.404049+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391109+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:41.391109+0000","last_peered":"2023-10-16T15:53:41.391109+0000","last_clean":"2023-10-16T15:53:41.391109+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:41.391109+0000","last_undegraded":"2023-10-16T15:53:41.391109+0000","last_fullsized":"2023-10-16T15:53:41.391109+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410984+0000","last_change":"2023-10-16T15:53:41.410981+0000","last_active":"2023-10-16T15:53:42.410984+0000","last_peered":"2023-10-16T15:53:42.410984+0000","last_clean":"2023-10-16T15:53:42.410984+0000","last_became_active":"2023-10-16T15:53:41.410713+0000","last_became_peered":"2023-10-16T15:53:41.410713+0000","last_unstale":"2023-10-16T15:53:42.410984+0000","last_undegraded":"2023-10-16T15:53:42.410984+0000","last_fullsized":"2023-10-16T15:53:42.410984+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.539110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374479+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:41.374479+0000","last_peered":"2023-10-16T15:53:41.374479+0000","last_clean":"2023-10-16T15:53:41.374479+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:41.374479+0000","last_undegraded":"2023-10-16T15:53:41.374479+0000","last_fullsized":"2023-10-16T15:53:41.374479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374715+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:41.374715+0000","last_peered":"2023-10-16T15:53:41.374715+0000","last_clean":"2023-10-16T15:53:41.374715+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:41.374715+0000","last_undegraded":"2023-10-16T15:53:41.374715+0000","last_fullsized":"2023-10-16T15:53:41.374715+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396559+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:41.396559+0000","last_peered":"2023-10-16T15:53:41.396559+0000","last_clean":"2023-10-16T15:53:41.396559+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:41.396559+0000","last_undegraded":"2023-10-16T15:53:41.396559+0000","last_fullsized":"2023-10-16T15:53:41.396559+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.400191+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:41.400191+0000","last_peered":"2023-10-16T15:53:41.400191+0000","last_clean":"2023-10-16T15:53:41.400191+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:41.400191+0000","last_undegraded":"2023-10-16T15:53:41.400191+0000","last_fullsized":"2023-10-16T15:53:41.400191+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385167+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:41.385167+0000","last_peered":"2023-10-16T15:53:41.385167+0000","last_clean":"2023-10-16T15:53:41.385167+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:41.385167+0000","last_undegraded":"2023-10-16T15:53:41.385167+0000","last_fullsized":"2023-10-16T15:53:41.385167+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394811+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:41.394811+0000","last_peered":"2023-10-16T15:53:41.394811+0000","last_clean":"2023-10-16T15:53:41.394811+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:41.394811+0000","last_undegraded":"2023-10-16T15:53:41.394811+0000","last_fullsized":"2023-10-16T15:53:41.394811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418006+0000","last_change":"2023-10-16T15:53:41.406425+0000","last_active":"2023-10-16T15:53:43.418006+0000","last_peered":"2023-10-16T15:53:43.418006+0000","last_clean":"2023-10-16T15:53:43.418006+0000","last_became_active":"2023-10-16T15:53:41.405266+0000","last_became_peered":"2023-10-16T15:53:41.405266+0000","last_unstale":"2023-10-16T15:53:43.418006+0000","last_undegraded":"2023-10-16T15:53:43.418006+0000","last_fullsized":"2023-10-16T15:53:43.418006+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:36.594777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462730+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:29.462730+0000","last_peered":"2023-10-16T15:53:29.462730+0000","last_clean":"2023-10-16T15:53:29.462730+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:29.462730+0000","last_undegraded":"2023-10-16T15:53:29.462730+0000","last_fullsized":"2023-10-16T15:53:29.462730+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379861+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:41.379861+0000","last_peered":"2023-10-16T15:53:41.379861+0000","last_clean":"2023-10-16T15:53:41.379861+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:41.379861+0000","last_undegraded":"2023-10-16T15:53:41.379861+0000","last_fullsized":"2023-10-16T15:53:41.379861+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374213+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:41.374213+0000","last_peered":"2023-10-16T15:53:41.374213+0000","last_clean":"2023-10-16T15:53:41.374213+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:41.374213+0000","last_undegraded":"2023-10-16T15:53:41.374213+0000","last_fullsized":"2023-10-16T15:53:41.374213+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013669+0000","last_change":"2023-10-16T15:53:41.411866+0000","last_active":"2023-10-16T15:53:43.013669+0000","last_peered":"2023-10-16T15:53:43.013669+0000","last_clean":"2023-10-16T15:53:43.013669+0000","last_became_active":"2023-10-16T15:53:41.411569+0000","last_became_peered":"2023-10-16T15:53:41.411569+0000","last_unstale":"2023-10-16T15:53:43.013669+0000","last_undegraded":"2023-10-16T15:53:43.013669+0000","last_fullsized":"2023-10-16T15:53:43.013669+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:01.576399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383777+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:41.383777+0000","last_peered":"2023-10-16T15:53:41.383777+0000","last_clean":"2023-10-16T15:53:41.383777+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:41.383777+0000","last_undegraded":"2023-10-16T15:53:41.383777+0000","last_fullsized":"2023-10-16T15:53:41.383777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409957+0000","last_change":"2023-10-16T15:53:41.399860+0000","last_active":"2023-10-16T15:53:42.409957+0000","last_peered":"2023-10-16T15:53:42.409957+0000","last_clean":"2023-10-16T15:53:42.409957+0000","last_became_active":"2023-10-16T15:53:41.399589+0000","last_became_peered":"2023-10-16T15:53:41.399589+0000","last_unstale":"2023-10-16T15:53:42.409957+0000","last_undegraded":"2023-10-16T15:53:42.409957+0000","last_fullsized":"2023-10-16T15:53:42.409957+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385445+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:41.385445+0000","last_peered":"2023-10-16T15:53:41.385445+0000","last_clean":"2023-10-16T15:53:41.385445+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:41.385445+0000","last_undegraded":"2023-10-16T15:53:41.385445+0000","last_fullsized":"2023-10-16T15:53:41.385445+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402749+0000","last_change":"2023-10-16T15:53:41.399919+0000","last_active":"2023-10-16T15:53:42.402749+0000","last_peered":"2023-10-16T15:53:42.402749+0000","last_clean":"2023-10-16T15:53:42.402749+0000","last_became_active":"2023-10-16T15:53:41.397770+0000","last_became_peered":"2023-10-16T15:53:41.397770+0000","last_unstale":"2023-10-16T15:53:42.402749+0000","last_undegraded":"2023-10-16T15:53:42.402749+0000","last_fullsized":"2023-10-16T15:53:42.402749+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463182+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:29.463182+0000","last_peered":"2023-10-16T15:53:29.463182+0000","last_clean":"2023-10-16T15:53:29.463182+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:29.463182+0000","last_undegraded":"2023-10-16T15:53:29.463182+0000","last_fullsized":"2023-10-16T15:53:29.463182+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014330+0000","last_change":"2023-10-16T15:53:41.401459+0000","last_active":"2023-10-16T15:53:43.014330+0000","last_peered":"2023-10-16T15:53:43.014330+0000","last_clean":"2023-10-16T15:53:43.014330+0000","last_became_active":"2023-10-16T15:53:41.401147+0000","last_became_peered":"2023-10-16T15:53:41.401147+0000","last_unstale":"2023-10-16T15:53:43.014330+0000","last_undegraded":"2023-10-16T15:53:43.014330+0000","last_fullsized":"2023-10-16T15:53:43.014330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:31.244497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.329048+0000","last_change":"2023-10-16T15:53:41.404413+0000","last_active":"2023-10-16T15:53:43.329048+0000","last_peered":"2023-10-16T15:53:43.329048+0000","last_clean":"2023-10-16T15:53:43.329048+0000","last_became_active":"2023-10-16T15:53:41.403862+0000","last_became_peered":"2023-10-16T15:53:41.403862+0000","last_unstale":"2023-10-16T15:53:43.329048+0000","last_undegraded":"2023-10-16T15:53:43.329048+0000","last_fullsized":"2023-10-16T15:53:43.329048+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:34.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462408+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:29.462408+0000","last_peered":"2023-10-16T15:53:29.462408+0000","last_clean":"2023-10-16T15:53:29.462408+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:29.462408+0000","last_undegraded":"2023-10-16T15:53:29.462408+0000","last_fullsized":"2023-10-16T15:53:29.462408+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407480+0000","last_change":"2023-10-16T15:53:41.406217+0000","last_active":"2023-10-16T15:53:42.407480+0000","last_peered":"2023-10-16T15:53:42.407480+0000","last_clean":"2023-10-16T15:53:42.407480+0000","last_became_active":"2023-10-16T15:53:41.405870+0000","last_became_peered":"2023-10-16T15:53:41.405870+0000","last_unstale":"2023-10-16T15:53:42.407480+0000","last_undegraded":"2023-10-16T15:53:42.407480+0000","last_fullsized":"2023-10-16T15:53:42.407480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374717+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:41.374717+0000","last_peered":"2023-10-16T15:53:41.374717+0000","last_clean":"2023-10-16T15:53:41.374717+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:41.374717+0000","last_undegraded":"2023-10-16T15:53:41.374717+0000","last_fullsized":"2023-10-16T15:53:41.374717+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":80,"acting":80,"num_store_stats":5},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60892,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310948,"statfs":{"total":96636764160,"available":96574410752,"internally_reserved":0,"allocated":3428352,"data_stored":1976322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":1.1830000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.323},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":1.2769999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":1.141},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.4099999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.9370000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.6419999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.502}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":1.722},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.5960000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1412470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.216},"min":{"1min":0.73699999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7709999999999999,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.117},{"interface":"front","average":{"1min":1.22,"5min":1.276,"15min":1.2609999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.5489999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2949999999999999,"15min":1.2929999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.72199999999999998},"max":{"1min":1.8740000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3420000000000001,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.579,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.353}]},{"osd":2,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.244,"15min":1.2450000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.752},"max":{"1min":1.554,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.431,"15min":1.401},"min":{"1min":1.0680000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.1990000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.88100000000000001,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.46},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.331,"15min":1.339},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.631,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3560000000000001,"15min":1.331},"min":{"1min":0.871,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.325},{"interface":"front","average":{"1min":1.444,"5min":1.4430000000000001,"15min":1.407},"min":{"1min":0.90100000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.6319999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":0.88300000000000001,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.792,"5min":1.9419999999999999,"15min":2.016},"last":1.022},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5640000000000001,"15min":1.5289999999999999},"min":{"1min":1.0309999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.393}]},{"osd":7,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.633,"15min":1.6060000000000001},"min":{"1min":1.1950000000000001,"5min":0.627,"15min":0.627},"max":{"1min":2.149,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.6220000000000001,"15min":1.5980000000000001},"min":{"1min":1.264,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.2849999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29788,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342052,"statfs":{"total":96636764160,"available":96606261248,"internally_reserved":0,"allocated":2826240,"data_stored":1383679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.80200000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.821},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3,"15min":1.2470000000000001},"min":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.639,"5min":2.738,"15min":2.738},"last":0.98399999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.232,"15min":1.236},"min":{"1min":0.91300000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7569999999999999,"5min":1.8140000000000001,"15min":1.8680000000000001},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.284,"15min":1.3109999999999999},"min":{"1min":0.89300000000000002,"5min":0.87,"15min":0.87},"max":{"1min":1.675,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.5600000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2450000000000001,"15min":1.2629999999999999},"min":{"1min":0.85199999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.784,"5min":1.784,"15min":1.9219999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.6990000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.7190000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3660000000000001,"15min":1.347},"min":{"1min":0.93100000000000005,"5min":0.86199999999999999,"15min":0.80000000000000004},"max":{"1min":1.7350000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.605},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.383,"15min":1.3939999999999999},"min":{"1min":1.103,"5min":0.752,"15min":0.752},"max":{"1min":1.7250000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.323}]},{"osd":4,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3759999999999999,"15min":1.3460000000000001},"min":{"1min":0.92300000000000004,"5min":0.85699999999999998,"15min":0.78100000000000003},"max":{"1min":1.8160000000000001,"5min":1.972,"15min":1.9730000000000001},"last":1.9079999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5389999999999999,"15min":1.532},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.77000000000000002},"max":{"1min":1.869,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.4159999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4359999999999999,"15min":1.385},"min":{"1min":1.04,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.768,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.8029999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.458,"15min":1.4390000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.70599999999999996},"max":{"1min":1.7390000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.8460000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.556,"15min":1.544},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.923,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.681},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6499999999999999,"15min":1.643},"min":{"1min":1.1719999999999999,"5min":1.032,"15min":0.70499999999999996},"max":{"1min":1.845,"5min":3.109,"15min":3.109},"last":1.877}]}]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52000,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319840,"statfs":{"total":96636764160,"available":96583516160,"internally_reserved":0,"allocated":2842624,"data_stored":1386058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2330000000000001,"15min":1.212},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.52700000000000002},"max":{"1min":1.794,"5min":1.883,"15min":1.883},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.95799999999999996,"5min":0.83499999999999996,"15min":0.629},"max":{"1min":1.675,"5min":1.929,"15min":2.2160000000000002},"last":1.04},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.3160000000000001},"min":{"1min":0.92000000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0179999999999998,"5min":2.157,"15min":2.2989999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4490000000000001,"15min":1.4670000000000001},"min":{"1min":1.1160000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.399},{"interface":"front","average":{"1min":1.591,"5min":1.498,"15min":1.4970000000000001},"min":{"1min":1.1870000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.119}]},{"osd":3,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":1.081,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.23,"5min":2.23,"15min":2.3650000000000002},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4099999999999999,"15min":1.401},"min":{"1min":1.0069999999999999,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.171}]},{"osd":5,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.375,"15min":1.3420000000000001},"min":{"1min":1.03,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.911,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.552},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.46,"15min":1.4370000000000001},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.161,"5min":2.161,"15min":2.4489999999999998},"last":1.375}]},{"osd":6,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":0.97499999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.554,"15min":1.5269999999999999},"min":{"1min":1.0089999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.2789999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.694}]},{"osd":7,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.617,"15min":1.5860000000000001},"min":{"1min":1.2669999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.6339999999999999,"15min":1.6040000000000001},"min":{"1min":1.3999999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.242,"5min":2.2599999999999998,"15min":2.347},"last":1.9339999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56228,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315612,"statfs":{"total":96636764160,"available":96579186688,"internally_reserved":0,"allocated":2899968,"data_stored":1413696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.44}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":0.92700000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":0.94399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1413500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":0.621}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.772}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":1.335},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.407}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.8049999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":2.093}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.8500000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.9930000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576006,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47884,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323956,"statfs":{"total":96636764160,"available":96587730944,"internally_reserved":0,"allocated":2883584,"data_stored":1411978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.236}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.4470000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":1.679},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.502}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.919}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":1.9810000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":1.867}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48468,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323372,"statfs":{"total":96636764160,"available":96587132928,"internally_reserved":0,"allocated":3477504,"data_stored":2002962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.85199999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":1.1850000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":1.498}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.288},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4650000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.363},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":1.552}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:45.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.195+0000 7f37ad660640 1 Processor -- start 2023-10-16T15:53:46.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.196+0000 7f37ad660640 1 -- start start 2023-10-16T15:53:46.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.197+0000 7f37ad660640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a806c040 0x7f37a806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.197+0000 7f37ad660640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f37a80ad970 0x7f37a815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.197+0000 7f37ad660640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a806c960 con 0x7f37a815ebc0 2023-10-16T15:53:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.197+0000 7f37ad660640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a815b9d0 con 0x7f37a80ad970 2023-10-16T15:53:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.197+0000 7f37ad660640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815bcb0 con 0x7f37a806c040 2023-10-16T15:53:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.198+0000 7f37ace5f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f37a815ebc0 0x7f37a80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56020/0 (socket says 172.21.15.29:56020) 2023-10-16T15:53:46.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.198+0000 7f37ace5f640 1 -- 172.21.15.29:0/3500636378 learned_addr learned my addr 172.21.15.29:0/3500636378 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.198+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223390698 0 0) 0x7f37a806c960 con 0x7f37a815ebc0 2023-10-16T15:53:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.198+0000 7f379ffff640 1 --2- 172.21.15.29:0/3500636378 >> v2:172.21.15.170:3300/0 conn(0x7f37a80ad970 0x7f37a815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.198+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003520 con 0x7f37a815ebc0 2023-10-16T15:53:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.198+0000 7f37a7fff640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a806c040 0x7f37a806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3541884277 0 0) 0x7f3794003520 con 0x7f37a815ebc0 2023-10-16T15:53:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a806c040 msgr2=0x7f37a806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:46.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a806c040 0x7f37a806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 >> v2:172.21.15.170:3300/0 conn(0x7f37a80ad970 msgr2=0x7f37a815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 --2- 172.21.15.29:0/3500636378 >> v2:172.21.15.170:3300/0 conn(0x7f37a80ad970 0x7f37a815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a815bf90 con 0x7f37a815ebc0 2023-10-16T15:53:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f37a00019f0 con 0x7f37a815ebc0 2023-10-16T15:53:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 >> v1:172.21.15.29:6789/0 conn(0x7f37a815ebc0 legacy=0x7f37a80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3794003970 0x7f3794003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f3794004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3794004f60 0x7f379400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815b9d0 con 0x7f3794004360 2023-10-16T15:53:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815bcb0 con 0x7f3794004f60 2023-10-16T15:53:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379400bef0 con 0x7f3794003970 2023-10-16T15:53:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37a7fff640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a806c040 0x7f37a806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f379ffff640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f3794004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.199+0000 7f37ace5f640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3794004f60 0x7f379400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.200+0000 7f379ffff640 1 -- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3794003970 msgr2=0x7f3794003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.200+0000 7f379ffff640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3794003970 0x7f3794003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.200+0000 7f379ffff640 1 -- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3794004f60 msgr2=0x7f379400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.200+0000 7f379ffff640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3794004f60 0x7f379400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.200+0000 7f379ffff640 1 -- 172.21.15.29:0/3500636378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794003520 con 0x7f3794004360 2023-10-16T15:53:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.200+0000 7f37ace5f640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3794004f60 0x7f379400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.201+0000 7f379ffff640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f3794004840 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f3790009ac0 tx=0x7f37940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b73fdbbaa66cce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.201+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379002c710 con 0x7f3794004360 2023-10-16T15:53:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.201+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379002c870 con 0x7f3794004360 2023-10-16T15:53:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.201+0000 7f37a77fe640 1 -- 172.21.15.29:0/3500636378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379002cc00 con 0x7f3794004360 2023-10-16T15:53:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ad660640 1 -- 172.21.15.29:0/3500636378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 msgr2=0x7f3794004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ad660640 1 --2- 172.21.15.29:0/3500636378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f3794004840 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f3790009ac0 tx=0x7f37940057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ace5f640 1 -- 172.21.15.29:0/3500636378 reap_dead start 2023-10-16T15:53:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ad660640 1 -- 172.21.15.29:0/3500636378 shutdown_connections 2023-10-16T15:53:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ad660640 1 -- 172.21.15.29:0/3500636378 >> 172.21.15.29:0/3500636378 conn(0x7f37a80ce6a0 msgr2=0x7f37a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ad660640 1 -- 172.21.15.29:0/3500636378 shutdown_connections 2023-10-16T15:53:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.202+0000 7f37ad660640 1 -- 172.21.15.29:0/3500636378 wait complete. 2023-10-16T15:53:46.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.204+0000 7f37ad660640 1 Processor -- start 2023-10-16T15:53:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.204+0000 7f37ad660640 1 -- start start 2023-10-16T15:53:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37ad660640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f37a80c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37ad660640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37a815a970 0x7f37a81532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37ad660640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a8158c40 0x7f37a81590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37ad660640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815a080 con 0x7f3794004360 2023-10-16T15:53:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37ad660640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815adf0 con 0x7f37a815a970 2023-10-16T15:53:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37ad660640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a8159ac0 con 0x7f37a8158c40 2023-10-16T15:53:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f37a80c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f37a80c8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45812/0 (socket says 172.21.15.29:45812) 2023-10-16T15:53:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.205+0000 7f37a7fff640 1 -- 172.21.15.29:0/2189420064 learned_addr learned my addr 172.21.15.29:0/2189420064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:46.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37ace5f640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a8158c40 0x7f37a81590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f379ffff640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37a815a970 0x7f37a81532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a7fff640 1 -- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a8158c40 msgr2=0x7f37a81590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a7fff640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a8158c40 0x7f37a81590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a7fff640 1 -- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37a815a970 msgr2=0x7f37a81532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a7fff640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37a815a970 0x7f37a81532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a7fff640 1 -- 172.21.15.29:0/2189420064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a8159da0 con 0x7f3794004360 2023-10-16T15:53:46.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a7fff640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f37a80c8200 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f3798006d40 tx=0x7f379800d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:46.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798004030 con 0x7f3794004360 2023-10-16T15:53:46.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37980041b0 con 0x7f3794004360 2023-10-16T15:53:46.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798003610 con 0x7f3794004360 2023-10-16T15:53:46.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.206+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a80ae240 con 0x7f3794004360 2023-10-16T15:53:46.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.207+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a8153830 con 0x7f3794004360 2023-10-16T15:53:46.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.208+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a80ae0a0 con 0x7f3794004360 2023-10-16T15:53:46.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.209+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3798003770 con 0x7f3794004360 2023-10-16T15:53:46.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.210+0000 7f37a57fa640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f378806af90 0x7f378806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.210+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f3798089b10 con 0x7f3794004360 2023-10-16T15:53:46.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.211+0000 7f379ffff640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f378806af90 0x7f378806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.212+0000 7f379ffff640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f378806af90 0x7f378806d450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3790036000 tx=0x7f3790030000 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:46.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.215+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f37a80ae0a0 con 0x7f3794004360 2023-10-16T15:53:46.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.407+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37a80cc550 con 0x7f378806af90 2023-10-16T15:53:46.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.450+0000 7f37a57fa640 1 -- 172.21.15.29:0/2189420064 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+521543 (secure 0 0 0) 0x7f379002cae0 con 0x7f378806af90 2023-10-16T15:53:46.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:46.463 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:46.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f378806af90 msgr2=0x7f378806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f378806af90 0x7f378806d450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3790036000 tx=0x7f3790030000 comp rx=0 tx=0).stop 2023-10-16T15:53:46.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 msgr2=0x7f37a80c8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f37a80c8200 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f3798006d40 tx=0x7f379800d910 comp rx=0 tx=0).stop 2023-10-16T15:53:46.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 shutdown_connections 2023-10-16T15:53:46.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f378806af90 0x7f378806d450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f37a8158c40 0x7f37a81590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37a815a970 0x7f37a81532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 --2- 172.21.15.29:0/2189420064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3794004360 0x7f37a80c8200 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.458+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 >> 172.21.15.29:0/2189420064 conn(0x7f37a80ce6a0 msgr2=0x7f37a80ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:46.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.459+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 shutdown_connections 2023-10-16T15:53:46.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.459+0000 7f37ad660640 1 -- 172.21.15.29:0/2189420064 wait complete. 2023-10-16T15:53:46.483 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-16T15:53:44.462540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596363,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":655,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381312,"kb_used_data":23568,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593408,"statfs":{"total":773094113280,"available":772703649792,"internally_reserved":0,"allocated":24133632,"data_stored":12400665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366202425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3585,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.121643"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1248,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:28.235155+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:28.235155+0000","last_peered":"2023-10-16T15:53:28.235155+0000","last_clean":"2023-10-16T15:53:28.235155+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:28.235155+0000","last_undegraded":"2023-10-16T15:53:28.235155+0000","last_fullsized":"2023-10-16T15:53:28.235155+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396662+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:41.396662+0000","last_peered":"2023-10-16T15:53:41.396662+0000","last_clean":"2023-10-16T15:53:41.396662+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:41.396662+0000","last_undegraded":"2023-10-16T15:53:41.396662+0000","last_fullsized":"2023-10-16T15:53:41.396662+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375906+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:41.375906+0000","last_peered":"2023-10-16T15:53:41.375906+0000","last_clean":"2023-10-16T15:53:41.375906+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:41.375906+0000","last_undegraded":"2023-10-16T15:53:41.375906+0000","last_fullsized":"2023-10-16T15:53:41.375906+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.355503+0000","last_change":"2023-10-16T15:53:39.355503+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.355503+0000","last_undegraded":"2023-10-16T15:53:39.355503+0000","last_fullsized":"2023-10-16T15:53:39.355503+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411355+0000","last_change":"2023-10-16T15:53:41.413779+0000","last_active":"2023-10-16T15:53:42.411355+0000","last_peered":"2023-10-16T15:53:42.411355+0000","last_clean":"2023-10-16T15:53:42.411355+0000","last_became_active":"2023-10-16T15:53:41.413379+0000","last_became_peered":"2023-10-16T15:53:41.413379+0000","last_unstale":"2023-10-16T15:53:42.411355+0000","last_undegraded":"2023-10-16T15:53:42.411355+0000","last_fullsized":"2023-10-16T15:53:42.411355+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384903+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:41.384903+0000","last_peered":"2023-10-16T15:53:41.384903+0000","last_clean":"2023-10-16T15:53:41.384903+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:41.384903+0000","last_undegraded":"2023-10-16T15:53:41.384903+0000","last_fullsized":"2023-10-16T15:53:41.384903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462231+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:29.462231+0000","last_peered":"2023-10-16T15:53:29.462231+0000","last_clean":"2023-10-16T15:53:29.462231+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:29.462231+0000","last_undegraded":"2023-10-16T15:53:29.462231+0000","last_fullsized":"2023-10-16T15:53:29.462231+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345783+0000","last_change":"2023-10-16T15:53:39.345783+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345783+0000","last_undegraded":"2023-10-16T15:53:39.345783+0000","last_fullsized":"2023-10-16T15:53:39.345783+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014672+0000","last_change":"2023-10-16T15:53:41.418345+0000","last_active":"2023-10-16T15:53:43.014672+0000","last_peered":"2023-10-16T15:53:43.014672+0000","last_clean":"2023-10-16T15:53:43.014672+0000","last_became_active":"2023-10-16T15:53:41.418078+0000","last_became_peered":"2023-10-16T15:53:41.418078+0000","last_unstale":"2023-10-16T15:53:43.014672+0000","last_undegraded":"2023-10-16T15:53:43.014672+0000","last_fullsized":"2023-10-16T15:53:43.014672+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:25.632327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461694+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:29.461694+0000","last_peered":"2023-10-16T15:53:29.461694+0000","last_clean":"2023-10-16T15:53:29.461694+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:29.461694+0000","last_undegraded":"2023-10-16T15:53:29.461694+0000","last_fullsized":"2023-10-16T15:53:29.461694+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380422+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:41.380422+0000","last_peered":"2023-10-16T15:53:41.380422+0000","last_clean":"2023-10-16T15:53:41.380422+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:41.380422+0000","last_undegraded":"2023-10-16T15:53:41.380422+0000","last_fullsized":"2023-10-16T15:53:41.380422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.405443+0000","last_change":"2023-10-16T15:53:40.384327+0000","last_active":"2023-10-16T15:53:41.405443+0000","last_peered":"2023-10-16T15:53:41.405443+0000","last_clean":"2023-10-16T15:53:41.405443+0000","last_became_active":"2023-10-16T15:53:40.384027+0000","last_became_peered":"2023-10-16T15:53:40.384027+0000","last_unstale":"2023-10-16T15:53:41.405443+0000","last_undegraded":"2023-10-16T15:53:41.405443+0000","last_fullsized":"2023-10-16T15:53:41.405443+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:09.776405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014626+0000","last_change":"2023-10-16T15:53:41.393836+0000","last_active":"2023-10-16T15:53:43.014626+0000","last_peered":"2023-10-16T15:53:43.014626+0000","last_clean":"2023-10-16T15:53:43.014626+0000","last_became_active":"2023-10-16T15:53:41.392583+0000","last_became_peered":"2023-10-16T15:53:41.392583+0000","last_unstale":"2023-10-16T15:53:43.014626+0000","last_undegraded":"2023-10-16T15:53:43.014626+0000","last_fullsized":"2023-10-16T15:53:43.014626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:22.777642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384099+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:41.384099+0000","last_peered":"2023-10-16T15:53:41.384099+0000","last_clean":"2023-10-16T15:53:41.384099+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:41.384099+0000","last_undegraded":"2023-10-16T15:53:41.384099+0000","last_fullsized":"2023-10-16T15:53:41.384099+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396904+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:41.396904+0000","last_peered":"2023-10-16T15:53:41.396904+0000","last_clean":"2023-10-16T15:53:41.396904+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:41.396904+0000","last_undegraded":"2023-10-16T15:53:41.396904+0000","last_fullsized":"2023-10-16T15:53:41.396904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395708+0000","last_change":"2023-10-16T15:53:40.378111+0000","last_active":"2023-10-16T15:53:41.395708+0000","last_peered":"2023-10-16T15:53:41.395708+0000","last_clean":"2023-10-16T15:53:41.395708+0000","last_became_active":"2023-10-16T15:53:40.377470+0000","last_became_peered":"2023-10-16T15:53:40.377470+0000","last_unstale":"2023-10-16T15:53:41.395708+0000","last_undegraded":"2023-10-16T15:53:41.395708+0000","last_fullsized":"2023-10-16T15:53:41.395708+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408902+0000","last_change":"2023-10-16T15:53:41.409618+0000","last_active":"2023-10-16T15:53:42.408902+0000","last_peered":"2023-10-16T15:53:42.408902+0000","last_clean":"2023-10-16T15:53:42.408902+0000","last_became_active":"2023-10-16T15:53:41.409365+0000","last_became_peered":"2023-10-16T15:53:41.409365+0000","last_unstale":"2023-10-16T15:53:42.408902+0000","last_undegraded":"2023-10-16T15:53:42.408902+0000","last_fullsized":"2023-10-16T15:53:42.408902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463159+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:29.463159+0000","last_peered":"2023-10-16T15:53:29.463159+0000","last_clean":"2023-10-16T15:53:29.463159+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:29.463159+0000","last_undegraded":"2023-10-16T15:53:29.463159+0000","last_fullsized":"2023-10-16T15:53:29.463159+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.346616+0000","last_change":"2023-10-16T15:53:39.346616+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.346616+0000","last_undegraded":"2023-10-16T15:53:39.346616+0000","last_fullsized":"2023-10-16T15:53:39.346616+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410745+0000","last_change":"2023-10-16T15:53:41.400067+0000","last_active":"2023-10-16T15:53:42.410745+0000","last_peered":"2023-10-16T15:53:42.410745+0000","last_clean":"2023-10-16T15:53:42.410745+0000","last_became_active":"2023-10-16T15:53:41.399778+0000","last_became_peered":"2023-10-16T15:53:41.399778+0000","last_unstale":"2023-10-16T15:53:42.410745+0000","last_undegraded":"2023-10-16T15:53:42.410745+0000","last_fullsized":"2023-10-16T15:53:42.410745+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:18.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384461+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:41.384461+0000","last_peered":"2023-10-16T15:53:41.384461+0000","last_clean":"2023-10-16T15:53:41.384461+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:41.384461+0000","last_undegraded":"2023-10-16T15:53:41.384461+0000","last_fullsized":"2023-10-16T15:53:41.384461+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461873+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:29.461873+0000","last_peered":"2023-10-16T15:53:29.461873+0000","last_clean":"2023-10-16T15:53:29.461873+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:29.461873+0000","last_undegraded":"2023-10-16T15:53:29.461873+0000","last_fullsized":"2023-10-16T15:53:29.461873+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374462+0000","last_change":"2023-10-16T15:53:40.375759+0000","last_active":"2023-10-16T15:53:41.374462+0000","last_peered":"2023-10-16T15:53:41.374462+0000","last_clean":"2023-10-16T15:53:41.374462+0000","last_became_active":"2023-10-16T15:53:40.375464+0000","last_became_peered":"2023-10-16T15:53:40.375464+0000","last_unstale":"2023-10-16T15:53:41.374462+0000","last_undegraded":"2023-10-16T15:53:41.374462+0000","last_fullsized":"2023-10-16T15:53:41.374462+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385405+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:41.385405+0000","last_peered":"2023-10-16T15:53:41.385405+0000","last_clean":"2023-10-16T15:53:41.385405+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:41.385405+0000","last_undegraded":"2023-10-16T15:53:41.385405+0000","last_fullsized":"2023-10-16T15:53:41.385405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462365+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:29.462365+0000","last_peered":"2023-10-16T15:53:29.462365+0000","last_clean":"2023-10-16T15:53:29.462365+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:29.462365+0000","last_undegraded":"2023-10-16T15:53:29.462365+0000","last_fullsized":"2023-10-16T15:53:29.462365+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.355490+0000","last_change":"2023-10-16T15:53:39.355490+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.355490+0000","last_undegraded":"2023-10-16T15:53:39.355490+0000","last_fullsized":"2023-10-16T15:53:39.355490+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.6","version":"225'1","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:31.400212+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:31.400212+0000","last_peered":"2023-10-16T15:53:31.400212+0000","last_clean":"2023-10-16T15:53:31.400212+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:31.400212+0000","last_undegraded":"2023-10-16T15:53:31.400212+0000","last_fullsized":"2023-10-16T15:53:31.400212+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376019+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:41.376019+0000","last_peered":"2023-10-16T15:53:41.376019+0000","last_clean":"2023-10-16T15:53:41.376019+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:41.376019+0000","last_undegraded":"2023-10-16T15:53:41.376019+0000","last_fullsized":"2023-10-16T15:53:41.376019+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384628+0000","last_change":"2023-10-16T15:53:40.375284+0000","last_active":"2023-10-16T15:53:41.384628+0000","last_peered":"2023-10-16T15:53:41.384628+0000","last_clean":"2023-10-16T15:53:41.384628+0000","last_became_active":"2023-10-16T15:53:40.374509+0000","last_became_peered":"2023-10-16T15:53:40.374509+0000","last_unstale":"2023-10-16T15:53:41.384628+0000","last_undegraded":"2023-10-16T15:53:41.384628+0000","last_fullsized":"2023-10-16T15:53:41.384628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:59.597435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407164+0000","last_change":"2023-10-16T15:53:41.414757+0000","last_active":"2023-10-16T15:53:42.407164+0000","last_peered":"2023-10-16T15:53:42.407164+0000","last_clean":"2023-10-16T15:53:42.407164+0000","last_became_active":"2023-10-16T15:53:41.414480+0000","last_became_peered":"2023-10-16T15:53:41.414480+0000","last_unstale":"2023-10-16T15:53:42.407164+0000","last_undegraded":"2023-10-16T15:53:42.407164+0000","last_fullsized":"2023-10-16T15:53:42.407164+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387180+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:41.387180+0000","last_peered":"2023-10-16T15:53:41.387180+0000","last_clean":"2023-10-16T15:53:41.387180+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:41.387180+0000","last_undegraded":"2023-10-16T15:53:41.387180+0000","last_fullsized":"2023-10-16T15:53:41.387180+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395410+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:41.395410+0000","last_peered":"2023-10-16T15:53:41.395410+0000","last_clean":"2023-10-16T15:53:41.395410+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:41.395410+0000","last_undegraded":"2023-10-16T15:53:41.395410+0000","last_fullsized":"2023-10-16T15:53:41.395410+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382520+0000","last_change":"2023-10-16T15:53:40.381870+0000","last_active":"2023-10-16T15:53:41.382520+0000","last_peered":"2023-10-16T15:53:41.382520+0000","last_clean":"2023-10-16T15:53:41.382520+0000","last_became_active":"2023-10-16T15:53:40.381577+0000","last_became_peered":"2023-10-16T15:53:40.381577+0000","last_unstale":"2023-10-16T15:53:41.382520+0000","last_undegraded":"2023-10-16T15:53:41.382520+0000","last_fullsized":"2023-10-16T15:53:41.382520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:33.030599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404650+0000","last_change":"2023-10-16T15:53:41.408405+0000","last_active":"2023-10-16T15:53:42.404650+0000","last_peered":"2023-10-16T15:53:42.404650+0000","last_clean":"2023-10-16T15:53:42.404650+0000","last_became_active":"2023-10-16T15:53:41.407251+0000","last_became_peered":"2023-10-16T15:53:41.407251+0000","last_unstale":"2023-10-16T15:53:42.404650+0000","last_undegraded":"2023-10-16T15:53:42.404650+0000","last_fullsized":"2023-10-16T15:53:42.404650+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.381328+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:41.381328+0000","last_peered":"2023-10-16T15:53:41.381328+0000","last_clean":"2023-10-16T15:53:41.381328+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:41.381328+0000","last_undegraded":"2023-10-16T15:53:41.381328+0000","last_fullsized":"2023-10-16T15:53:41.381328+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461650+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:29.461650+0000","last_peered":"2023-10-16T15:53:29.461650+0000","last_clean":"2023-10-16T15:53:29.461650+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:29.461650+0000","last_undegraded":"2023-10-16T15:53:29.461650+0000","last_fullsized":"2023-10-16T15:53:29.461650+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414970+0000","last_change":"2023-10-16T15:53:40.373067+0000","last_active":"2023-10-16T15:53:41.414970+0000","last_peered":"2023-10-16T15:53:41.414970+0000","last_clean":"2023-10-16T15:53:41.414970+0000","last_became_active":"2023-10-16T15:53:40.372307+0000","last_became_peered":"2023-10-16T15:53:40.372307+0000","last_unstale":"2023-10-16T15:53:41.414970+0000","last_undegraded":"2023-10-16T15:53:41.414970+0000","last_fullsized":"2023-10-16T15:53:41.414970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330512+0000","last_change":"2023-10-16T15:53:41.400515+0000","last_active":"2023-10-16T15:53:43.330512+0000","last_peered":"2023-10-16T15:53:43.330512+0000","last_clean":"2023-10-16T15:53:43.330512+0000","last_became_active":"2023-10-16T15:53:41.399835+0000","last_became_peered":"2023-10-16T15:53:41.399835+0000","last_unstale":"2023-10-16T15:53:43.330512+0000","last_undegraded":"2023-10-16T15:53:43.330512+0000","last_fullsized":"2023-10-16T15:53:43.330512+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:40.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375870+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:41.375870+0000","last_peered":"2023-10-16T15:53:41.375870+0000","last_clean":"2023-10-16T15:53:41.375870+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:41.375870+0000","last_undegraded":"2023-10-16T15:53:41.375870+0000","last_fullsized":"2023-10-16T15:53:41.375870+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462177+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:29.462177+0000","last_peered":"2023-10-16T15:53:29.462177+0000","last_clean":"2023-10-16T15:53:29.462177+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:29.462177+0000","last_undegraded":"2023-10-16T15:53:29.462177+0000","last_fullsized":"2023-10-16T15:53:29.462177+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.407785+0000","last_change":"2023-10-16T15:53:40.378428+0000","last_active":"2023-10-16T15:53:41.407785+0000","last_peered":"2023-10-16T15:53:41.407785+0000","last_clean":"2023-10-16T15:53:41.407785+0000","last_became_active":"2023-10-16T15:53:40.378077+0000","last_became_peered":"2023-10-16T15:53:40.378077+0000","last_unstale":"2023-10-16T15:53:41.407785+0000","last_undegraded":"2023-10-16T15:53:41.407785+0000","last_fullsized":"2023-10-16T15:53:41.407785+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410289+0000","last_change":"2023-10-16T15:53:41.399412+0000","last_active":"2023-10-16T15:53:42.410289+0000","last_peered":"2023-10-16T15:53:42.410289+0000","last_clean":"2023-10-16T15:53:42.410289+0000","last_became_active":"2023-10-16T15:53:41.399125+0000","last_became_peered":"2023-10-16T15:53:41.399125+0000","last_unstale":"2023-10-16T15:53:42.410289+0000","last_undegraded":"2023-10-16T15:53:42.410289+0000","last_fullsized":"2023-10-16T15:53:42.410289+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376676+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:41.376676+0000","last_peered":"2023-10-16T15:53:41.376676+0000","last_clean":"2023-10-16T15:53:41.376676+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:41.376676+0000","last_undegraded":"2023-10-16T15:53:41.376676+0000","last_fullsized":"2023-10-16T15:53:41.376676+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394140+0000","last_change":"2023-10-16T15:53:40.377426+0000","last_active":"2023-10-16T15:53:41.394140+0000","last_peered":"2023-10-16T15:53:41.394140+0000","last_clean":"2023-10-16T15:53:41.394140+0000","last_became_active":"2023-10-16T15:53:40.377160+0000","last_became_peered":"2023-10-16T15:53:40.377160+0000","last_unstale":"2023-10-16T15:53:41.394140+0000","last_undegraded":"2023-10-16T15:53:41.394140+0000","last_fullsized":"2023-10-16T15:53:41.394140+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:38.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408560+0000","last_change":"2023-10-16T15:53:41.400288+0000","last_active":"2023-10-16T15:53:42.408560+0000","last_peered":"2023-10-16T15:53:42.408560+0000","last_clean":"2023-10-16T15:53:42.408560+0000","last_became_active":"2023-10-16T15:53:41.400031+0000","last_became_peered":"2023-10-16T15:53:41.400031+0000","last_unstale":"2023-10-16T15:53:42.408560+0000","last_undegraded":"2023-10-16T15:53:42.408560+0000","last_fullsized":"2023-10-16T15:53:42.408560+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393978+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:41.393978+0000","last_peered":"2023-10-16T15:53:41.393978+0000","last_clean":"2023-10-16T15:53:41.393978+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:41.393978+0000","last_undegraded":"2023-10-16T15:53:41.393978+0000","last_fullsized":"2023-10-16T15:53:41.393978+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392991+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:41.392991+0000","last_peered":"2023-10-16T15:53:41.392991+0000","last_clean":"2023-10-16T15:53:41.392991+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:41.392991+0000","last_undegraded":"2023-10-16T15:53:41.392991+0000","last_fullsized":"2023-10-16T15:53:41.392991+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.347704+0000","last_change":"2023-10-16T15:53:39.347704+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.347704+0000","last_undegraded":"2023-10-16T15:53:39.347704+0000","last_fullsized":"2023-10-16T15:53:39.347704+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409916+0000","last_change":"2023-10-16T15:53:41.413217+0000","last_active":"2023-10-16T15:53:42.409916+0000","last_peered":"2023-10-16T15:53:42.409916+0000","last_clean":"2023-10-16T15:53:42.409916+0000","last_became_active":"2023-10-16T15:53:41.411878+0000","last_became_peered":"2023-10-16T15:53:41.411878+0000","last_unstale":"2023-10-16T15:53:42.409916+0000","last_undegraded":"2023-10-16T15:53:42.409916+0000","last_fullsized":"2023-10-16T15:53:42.409916+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384277+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:41.384277+0000","last_peered":"2023-10-16T15:53:41.384277+0000","last_clean":"2023-10-16T15:53:41.384277+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:41.384277+0000","last_undegraded":"2023-10-16T15:53:41.384277+0000","last_fullsized":"2023-10-16T15:53:41.384277+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394436+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:41.394436+0000","last_peered":"2023-10-16T15:53:41.394436+0000","last_clean":"2023-10-16T15:53:41.394436+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:41.394436+0000","last_undegraded":"2023-10-16T15:53:41.394436+0000","last_fullsized":"2023-10-16T15:53:41.394436+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.356230+0000","last_change":"2023-10-16T15:53:39.356230+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.356230+0000","last_undegraded":"2023-10-16T15:53:39.356230+0000","last_fullsized":"2023-10-16T15:53:39.356230+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013025+0000","last_change":"2023-10-16T15:53:41.397581+0000","last_active":"2023-10-16T15:53:43.013025+0000","last_peered":"2023-10-16T15:53:43.013025+0000","last_clean":"2023-10-16T15:53:43.013025+0000","last_became_active":"2023-10-16T15:53:41.397301+0000","last_became_peered":"2023-10-16T15:53:41.397301+0000","last_unstale":"2023-10-16T15:53:43.013025+0000","last_undegraded":"2023-10-16T15:53:43.013025+0000","last_fullsized":"2023-10-16T15:53:43.013025+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:20.816418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394523+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:41.394523+0000","last_peered":"2023-10-16T15:53:41.394523+0000","last_clean":"2023-10-16T15:53:41.394523+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:41.394523+0000","last_undegraded":"2023-10-16T15:53:41.394523+0000","last_fullsized":"2023-10-16T15:53:41.394523+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.411952+0000","last_change":"2023-10-16T15:53:40.376976+0000","last_active":"2023-10-16T15:53:41.411952+0000","last_peered":"2023-10-16T15:53:41.411952+0000","last_clean":"2023-10-16T15:53:41.411952+0000","last_became_active":"2023-10-16T15:53:40.376700+0000","last_became_peered":"2023-10-16T15:53:40.376700+0000","last_unstale":"2023-10-16T15:53:41.411952+0000","last_undegraded":"2023-10-16T15:53:41.411952+0000","last_fullsized":"2023-10-16T15:53:41.411952+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013467+0000","last_change":"2023-10-16T15:53:41.410633+0000","last_active":"2023-10-16T15:53:43.013467+0000","last_peered":"2023-10-16T15:53:43.013467+0000","last_clean":"2023-10-16T15:53:43.013467+0000","last_became_active":"2023-10-16T15:53:41.409266+0000","last_became_peered":"2023-10-16T15:53:41.409266+0000","last_unstale":"2023-10-16T15:53:43.013467+0000","last_undegraded":"2023-10-16T15:53:43.013467+0000","last_fullsized":"2023-10-16T15:53:43.013467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:52.749663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462595+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:29.462595+0000","last_peered":"2023-10-16T15:53:29.462595+0000","last_clean":"2023-10-16T15:53:29.462595+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:29.462595+0000","last_undegraded":"2023-10-16T15:53:29.462595+0000","last_fullsized":"2023-10-16T15:53:29.462595+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462687+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:29.462687+0000","last_peered":"2023-10-16T15:53:29.462687+0000","last_clean":"2023-10-16T15:53:29.462687+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:29.462687+0000","last_undegraded":"2023-10-16T15:53:29.462687+0000","last_fullsized":"2023-10-16T15:53:29.462687+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384497+0000","last_change":"2023-10-16T15:53:40.375609+0000","last_active":"2023-10-16T15:53:41.384497+0000","last_peered":"2023-10-16T15:53:41.384497+0000","last_clean":"2023-10-16T15:53:41.384497+0000","last_became_active":"2023-10-16T15:53:40.375039+0000","last_became_peered":"2023-10-16T15:53:40.375039+0000","last_unstale":"2023-10-16T15:53:41.384497+0000","last_undegraded":"2023-10-16T15:53:41.384497+0000","last_fullsized":"2023-10-16T15:53:41.384497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:02.147563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.328614+0000","last_change":"2023-10-16T15:53:41.399266+0000","last_active":"2023-10-16T15:53:43.328614+0000","last_peered":"2023-10-16T15:53:43.328614+0000","last_clean":"2023-10-16T15:53:43.328614+0000","last_became_active":"2023-10-16T15:53:41.398177+0000","last_became_peered":"2023-10-16T15:53:41.398177+0000","last_unstale":"2023-10-16T15:53:43.328614+0000","last_undegraded":"2023-10-16T15:53:43.328614+0000","last_fullsized":"2023-10-16T15:53:43.328614+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.144341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385470+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:41.385470+0000","last_peered":"2023-10-16T15:53:41.385470+0000","last_clean":"2023-10-16T15:53:41.385470+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:41.385470+0000","last_undegraded":"2023-10-16T15:53:41.385470+0000","last_fullsized":"2023-10-16T15:53:41.385470+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462492+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:29.462492+0000","last_peered":"2023-10-16T15:53:29.462492+0000","last_clean":"2023-10-16T15:53:29.462492+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:29.462492+0000","last_undegraded":"2023-10-16T15:53:29.462492+0000","last_fullsized":"2023-10-16T15:53:29.462492+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.355830+0000","last_change":"2023-10-16T15:53:39.355830+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.355830+0000","last_undegraded":"2023-10-16T15:53:39.355830+0000","last_fullsized":"2023-10-16T15:53:39.355830+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380707+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:41.380707+0000","last_peered":"2023-10-16T15:53:41.380707+0000","last_clean":"2023-10-16T15:53:41.380707+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:41.380707+0000","last_undegraded":"2023-10-16T15:53:41.380707+0000","last_fullsized":"2023-10-16T15:53:41.380707+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374760+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:41.374760+0000","last_peered":"2023-10-16T15:53:41.374760+0000","last_clean":"2023-10-16T15:53:41.374760+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:41.374760+0000","last_undegraded":"2023-10-16T15:53:41.374760+0000","last_fullsized":"2023-10-16T15:53:41.374760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375756+0000","last_change":"2023-10-16T15:53:40.377615+0000","last_active":"2023-10-16T15:53:41.375756+0000","last_peered":"2023-10-16T15:53:41.375756+0000","last_clean":"2023-10-16T15:53:41.375756+0000","last_became_active":"2023-10-16T15:53:40.377342+0000","last_became_peered":"2023-10-16T15:53:40.377342+0000","last_unstale":"2023-10-16T15:53:41.375756+0000","last_undegraded":"2023-10-16T15:53:41.375756+0000","last_fullsized":"2023-10-16T15:53:41.375756+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388735+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:41.388735+0000","last_peered":"2023-10-16T15:53:41.388735+0000","last_clean":"2023-10-16T15:53:41.388735+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:41.388735+0000","last_undegraded":"2023-10-16T15:53:41.388735+0000","last_fullsized":"2023-10-16T15:53:41.388735+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380271+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:41.380271+0000","last_peered":"2023-10-16T15:53:41.380271+0000","last_clean":"2023-10-16T15:53:41.380271+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:41.380271+0000","last_undegraded":"2023-10-16T15:53:41.380271+0000","last_fullsized":"2023-10-16T15:53:41.380271+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.347002+0000","last_change":"2023-10-16T15:53:39.347002+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.347002+0000","last_undegraded":"2023-10-16T15:53:39.347002+0000","last_fullsized":"2023-10-16T15:53:39.347002+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461781+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:29.461781+0000","last_peered":"2023-10-16T15:53:29.461781+0000","last_clean":"2023-10-16T15:53:29.461781+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:29.461781+0000","last_undegraded":"2023-10-16T15:53:29.461781+0000","last_fullsized":"2023-10-16T15:53:29.461781+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393875+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:41.393875+0000","last_peered":"2023-10-16T15:53:41.393875+0000","last_clean":"2023-10-16T15:53:41.393875+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:41.393875+0000","last_undegraded":"2023-10-16T15:53:41.393875+0000","last_fullsized":"2023-10-16T15:53:41.393875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"228'6","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414139+0000","last_change":"2023-10-16T15:53:40.376773+0000","last_active":"2023-10-16T15:53:41.414139+0000","last_peered":"2023-10-16T15:53:41.414139+0000","last_clean":"2023-10-16T15:53:41.414139+0000","last_became_active":"2023-10-16T15:53:40.376123+0000","last_became_peered":"2023-10-16T15:53:40.376123+0000","last_unstale":"2023-10-16T15:53:41.414139+0000","last_undegraded":"2023-10-16T15:53:41.414139+0000","last_fullsized":"2023-10-16T15:53:41.414139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:19.218849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461826+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:29.461826+0000","last_peered":"2023-10-16T15:53:29.461826+0000","last_clean":"2023-10-16T15:53:29.461826+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:29.461826+0000","last_undegraded":"2023-10-16T15:53:29.461826+0000","last_fullsized":"2023-10-16T15:53:29.461826+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383904+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:41.383904+0000","last_peered":"2023-10-16T15:53:41.383904+0000","last_clean":"2023-10-16T15:53:41.383904+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:41.383904+0000","last_undegraded":"2023-10-16T15:53:41.383904+0000","last_fullsized":"2023-10-16T15:53:41.383904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"228'3","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402070+0000","last_change":"2023-10-16T15:53:40.365148+0000","last_active":"2023-10-16T15:53:41.402070+0000","last_peered":"2023-10-16T15:53:41.402070+0000","last_clean":"2023-10-16T15:53:41.402070+0000","last_became_active":"2023-10-16T15:53:40.364874+0000","last_became_peered":"2023-10-16T15:53:40.364874+0000","last_unstale":"2023-10-16T15:53:41.402070+0000","last_undegraded":"2023-10-16T15:53:41.402070+0000","last_fullsized":"2023-10-16T15:53:41.402070+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:39.387182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411499+0000","last_change":"2023-10-16T15:53:41.405376+0000","last_active":"2023-10-16T15:53:42.411499+0000","last_peered":"2023-10-16T15:53:42.411499+0000","last_clean":"2023-10-16T15:53:42.411499+0000","last_became_active":"2023-10-16T15:53:41.404697+0000","last_became_peered":"2023-10-16T15:53:41.404697+0000","last_unstale":"2023-10-16T15:53:42.411499+0000","last_undegraded":"2023-10-16T15:53:42.411499+0000","last_fullsized":"2023-10-16T15:53:42.411499+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"225'8","reported_seq":21,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:36.400354+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:36.400354+0000","last_peered":"2023-10-16T15:53:36.400354+0000","last_clean":"2023-10-16T15:53:36.400354+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:36.400354+0000","last_undegraded":"2023-10-16T15:53:36.400354+0000","last_fullsized":"2023-10-16T15:53:36.400354+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376419+0000","last_change":"2023-10-16T15:53:40.387700+0000","last_active":"2023-10-16T15:53:41.376419+0000","last_peered":"2023-10-16T15:53:41.376419+0000","last_clean":"2023-10-16T15:53:41.376419+0000","last_became_active":"2023-10-16T15:53:40.387431+0000","last_became_peered":"2023-10-16T15:53:40.387431+0000","last_unstale":"2023-10-16T15:53:41.376419+0000","last_undegraded":"2023-10-16T15:53:41.376419+0000","last_fullsized":"2023-10-16T15:53:41.376419+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:19.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418316+0000","last_change":"2023-10-16T15:53:41.385042+0000","last_active":"2023-10-16T15:53:43.418316+0000","last_peered":"2023-10-16T15:53:43.418316+0000","last_clean":"2023-10-16T15:53:43.418316+0000","last_became_active":"2023-10-16T15:53:41.384431+0000","last_became_peered":"2023-10-16T15:53:41.384431+0000","last_unstale":"2023-10-16T15:53:43.418316+0000","last_undegraded":"2023-10-16T15:53:43.418316+0000","last_fullsized":"2023-10-16T15:53:43.418316+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394713+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:41.394713+0000","last_peered":"2023-10-16T15:53:41.394713+0000","last_clean":"2023-10-16T15:53:41.394713+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:41.394713+0000","last_undegraded":"2023-10-16T15:53:41.394713+0000","last_fullsized":"2023-10-16T15:53:41.394713+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463205+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:29.463205+0000","last_peered":"2023-10-16T15:53:29.463205+0000","last_clean":"2023-10-16T15:53:29.463205+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:29.463205+0000","last_undegraded":"2023-10-16T15:53:29.463205+0000","last_fullsized":"2023-10-16T15:53:29.463205+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.408329+0000","last_change":"2023-10-16T15:53:40.386183+0000","last_active":"2023-10-16T15:53:41.408329+0000","last_peered":"2023-10-16T15:53:41.408329+0000","last_clean":"2023-10-16T15:53:41.408329+0000","last_became_active":"2023-10-16T15:53:40.385897+0000","last_became_peered":"2023-10-16T15:53:40.385897+0000","last_unstale":"2023-10-16T15:53:41.408329+0000","last_undegraded":"2023-10-16T15:53:41.408329+0000","last_fullsized":"2023-10-16T15:53:41.408329+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:27.631953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461915+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:29.461915+0000","last_peered":"2023-10-16T15:53:29.461915+0000","last_clean":"2023-10-16T15:53:29.461915+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:29.461915+0000","last_undegraded":"2023-10-16T15:53:29.461915+0000","last_fullsized":"2023-10-16T15:53:29.461915+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387083+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:41.387083+0000","last_peered":"2023-10-16T15:53:41.387083+0000","last_clean":"2023-10-16T15:53:41.387083+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:41.387083+0000","last_undegraded":"2023-10-16T15:53:41.387083+0000","last_fullsized":"2023-10-16T15:53:41.387083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345091+0000","last_change":"2023-10-16T15:53:39.345091+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345091+0000","last_undegraded":"2023-10-16T15:53:39.345091+0000","last_fullsized":"2023-10-16T15:53:39.345091+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407893+0000","last_change":"2023-10-16T15:53:41.414765+0000","last_active":"2023-10-16T15:53:42.407893+0000","last_peered":"2023-10-16T15:53:42.407893+0000","last_clean":"2023-10-16T15:53:42.407893+0000","last_became_active":"2023-10-16T15:53:41.414484+0000","last_became_peered":"2023-10-16T15:53:41.414484+0000","last_unstale":"2023-10-16T15:53:42.407893+0000","last_undegraded":"2023-10-16T15:53:42.407893+0000","last_fullsized":"2023-10-16T15:53:42.407893+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385369+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:41.385369+0000","last_peered":"2023-10-16T15:53:41.385369+0000","last_clean":"2023-10-16T15:53:41.385369+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:41.385369+0000","last_undegraded":"2023-10-16T15:53:41.385369+0000","last_fullsized":"2023-10-16T15:53:41.385369+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463287+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:29.463287+0000","last_peered":"2023-10-16T15:53:29.463287+0000","last_clean":"2023-10-16T15:53:29.463287+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:29.463287+0000","last_undegraded":"2023-10-16T15:53:29.463287+0000","last_fullsized":"2023-10-16T15:53:29.463287+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382349+0000","last_change":"2023-10-16T15:53:40.376076+0000","last_active":"2023-10-16T15:53:41.382349+0000","last_peered":"2023-10-16T15:53:41.382349+0000","last_clean":"2023-10-16T15:53:41.382349+0000","last_became_active":"2023-10-16T15:53:40.375798+0000","last_became_peered":"2023-10-16T15:53:40.375798+0000","last_unstale":"2023-10-16T15:53:41.382349+0000","last_undegraded":"2023-10-16T15:53:41.382349+0000","last_fullsized":"2023-10-16T15:53:41.382349+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330851+0000","last_change":"2023-10-16T15:53:41.409878+0000","last_active":"2023-10-16T15:53:43.330851+0000","last_peered":"2023-10-16T15:53:43.330851+0000","last_clean":"2023-10-16T15:53:43.330851+0000","last_became_active":"2023-10-16T15:53:41.409594+0000","last_became_peered":"2023-10-16T15:53:41.409594+0000","last_unstale":"2023-10-16T15:53:43.330851+0000","last_undegraded":"2023-10-16T15:53:43.330851+0000","last_fullsized":"2023-10-16T15:53:43.330851+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:10.790473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345082+0000","last_change":"2023-10-16T15:53:39.345082+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345082+0000","last_undegraded":"2023-10-16T15:53:39.345082+0000","last_fullsized":"2023-10-16T15:53:39.345082+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461553+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:29.461553+0000","last_peered":"2023-10-16T15:53:29.461553+0000","last_clean":"2023-10-16T15:53:29.461553+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:29.461553+0000","last_undegraded":"2023-10-16T15:53:29.461553+0000","last_fullsized":"2023-10-16T15:53:29.461553+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463088+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:29.463088+0000","last_peered":"2023-10-16T15:53:29.463088+0000","last_clean":"2023-10-16T15:53:29.463088+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:29.463088+0000","last_undegraded":"2023-10-16T15:53:29.463088+0000","last_fullsized":"2023-10-16T15:53:29.463088+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410670+0000","last_change":"2023-10-16T15:53:41.411697+0000","last_active":"2023-10-16T15:53:42.410670+0000","last_peered":"2023-10-16T15:53:42.410670+0000","last_clean":"2023-10-16T15:53:42.410670+0000","last_became_active":"2023-10-16T15:53:41.410477+0000","last_became_peered":"2023-10-16T15:53:41.410477+0000","last_unstale":"2023-10-16T15:53:42.410670+0000","last_undegraded":"2023-10-16T15:53:42.410670+0000","last_fullsized":"2023-10-16T15:53:42.410670+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:52.524535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"228'4","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.418437+0000","last_change":"2023-10-16T15:53:40.380112+0000","last_active":"2023-10-16T15:53:41.418437+0000","last_peered":"2023-10-16T15:53:41.418437+0000","last_clean":"2023-10-16T15:53:41.418437+0000","last_became_active":"2023-10-16T15:53:40.379085+0000","last_became_peered":"2023-10-16T15:53:40.379085+0000","last_unstale":"2023-10-16T15:53:41.418437+0000","last_undegraded":"2023-10-16T15:53:41.418437+0000","last_fullsized":"2023-10-16T15:53:41.418437+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:48.756275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382648+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:41.382648+0000","last_peered":"2023-10-16T15:53:41.382648+0000","last_clean":"2023-10-16T15:53:41.382648+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:41.382648+0000","last_undegraded":"2023-10-16T15:53:41.382648+0000","last_fullsized":"2023-10-16T15:53:41.382648+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379944+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:41.379944+0000","last_peered":"2023-10-16T15:53:41.379944+0000","last_clean":"2023-10-16T15:53:41.379944+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:41.379944+0000","last_undegraded":"2023-10-16T15:53:41.379944+0000","last_fullsized":"2023-10-16T15:53:41.379944+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384001+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:41.384001+0000","last_peered":"2023-10-16T15:53:41.384001+0000","last_clean":"2023-10-16T15:53:41.384001+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:41.384001+0000","last_undegraded":"2023-10-16T15:53:41.384001+0000","last_fullsized":"2023-10-16T15:53:41.384001+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463135+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:29.463135+0000","last_peered":"2023-10-16T15:53:29.463135+0000","last_clean":"2023-10-16T15:53:29.463135+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:29.463135+0000","last_undegraded":"2023-10-16T15:53:29.463135+0000","last_fullsized":"2023-10-16T15:53:29.463135+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.345766+0000","last_change":"2023-10-16T15:53:39.345766+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.345766+0000","last_undegraded":"2023-10-16T15:53:39.345766+0000","last_fullsized":"2023-10-16T15:53:39.345766+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013915+0000","last_change":"2023-10-16T15:53:41.405768+0000","last_active":"2023-10-16T15:53:43.013915+0000","last_peered":"2023-10-16T15:53:43.013915+0000","last_clean":"2023-10-16T15:53:43.013915+0000","last_became_active":"2023-10-16T15:53:41.405230+0000","last_became_peered":"2023-10-16T15:53:41.405230+0000","last_unstale":"2023-10-16T15:53:43.013915+0000","last_undegraded":"2023-10-16T15:53:43.013915+0000","last_fullsized":"2023-10-16T15:53:43.013915+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:43.313902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463229+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:29.463229+0000","last_peered":"2023-10-16T15:53:29.463229+0000","last_clean":"2023-10-16T15:53:29.463229+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:29.463229+0000","last_undegraded":"2023-10-16T15:53:29.463229+0000","last_fullsized":"2023-10-16T15:53:29.463229+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394353+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:41.394353+0000","last_peered":"2023-10-16T15:53:41.394353+0000","last_clean":"2023-10-16T15:53:41.394353+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:41.394353+0000","last_undegraded":"2023-10-16T15:53:41.394353+0000","last_fullsized":"2023-10-16T15:53:41.394353+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.404498+0000","last_change":"2023-10-16T15:53:40.381325+0000","last_active":"2023-10-16T15:53:41.404498+0000","last_peered":"2023-10-16T15:53:41.404498+0000","last_clean":"2023-10-16T15:53:41.404498+0000","last_became_active":"2023-10-16T15:53:40.380822+0000","last_became_peered":"2023-10-16T15:53:40.380822+0000","last_unstale":"2023-10-16T15:53:41.404498+0000","last_undegraded":"2023-10-16T15:53:41.404498+0000","last_fullsized":"2023-10-16T15:53:41.404498+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385254+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:41.385254+0000","last_peered":"2023-10-16T15:53:41.385254+0000","last_clean":"2023-10-16T15:53:41.385254+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:41.385254+0000","last_undegraded":"2023-10-16T15:53:41.385254+0000","last_fullsized":"2023-10-16T15:53:41.385254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374210+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:41.374210+0000","last_peered":"2023-10-16T15:53:41.374210+0000","last_clean":"2023-10-16T15:53:41.374210+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:41.374210+0000","last_undegraded":"2023-10-16T15:53:41.374210+0000","last_fullsized":"2023-10-16T15:53:41.374210+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.409948+0000","last_change":"2023-10-16T15:53:40.377357+0000","last_active":"2023-10-16T15:53:41.409948+0000","last_peered":"2023-10-16T15:53:41.409948+0000","last_clean":"2023-10-16T15:53:41.409948+0000","last_became_active":"2023-10-16T15:53:40.377070+0000","last_became_peered":"2023-10-16T15:53:40.377070+0000","last_unstale":"2023-10-16T15:53:41.409948+0000","last_undegraded":"2023-10-16T15:53:41.409948+0000","last_fullsized":"2023-10-16T15:53:41.409948+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462545+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:29.462545+0000","last_peered":"2023-10-16T15:53:29.462545+0000","last_clean":"2023-10-16T15:53:29.462545+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:29.462545+0000","last_undegraded":"2023-10-16T15:53:29.462545+0000","last_fullsized":"2023-10-16T15:53:29.462545+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461956+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:29.461956+0000","last_peered":"2023-10-16T15:53:29.461956+0000","last_clean":"2023-10-16T15:53:29.461956+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:29.461956+0000","last_undegraded":"2023-10-16T15:53:29.461956+0000","last_fullsized":"2023-10-16T15:53:29.461956+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391139+0000","last_change":"2023-10-16T15:53:40.380015+0000","last_active":"2023-10-16T15:53:41.391139+0000","last_peered":"2023-10-16T15:53:41.391139+0000","last_clean":"2023-10-16T15:53:41.391139+0000","last_became_active":"2023-10-16T15:53:40.379716+0000","last_became_peered":"2023-10-16T15:53:40.379716+0000","last_unstale":"2023-10-16T15:53:41.391139+0000","last_undegraded":"2023-10-16T15:53:41.391139+0000","last_fullsized":"2023-10-16T15:53:41.391139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402124+0000","last_change":"2023-10-16T15:53:41.403677+0000","last_active":"2023-10-16T15:53:42.402124+0000","last_peered":"2023-10-16T15:53:42.402124+0000","last_clean":"2023-10-16T15:53:42.402124+0000","last_became_active":"2023-10-16T15:53:41.403372+0000","last_became_peered":"2023-10-16T15:53:41.403372+0000","last_unstale":"2023-10-16T15:53:42.402124+0000","last_undegraded":"2023-10-16T15:53:42.402124+0000","last_fullsized":"2023-10-16T15:53:42.402124+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396765+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:41.396765+0000","last_peered":"2023-10-16T15:53:41.396765+0000","last_clean":"2023-10-16T15:53:41.396765+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.396765+0000","last_undegraded":"2023-10-16T15:53:41.396765+0000","last_fullsized":"2023-10-16T15:53:41.396765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462449+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:29.462449+0000","last_peered":"2023-10-16T15:53:29.462449+0000","last_clean":"2023-10-16T15:53:29.462449+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:29.462449+0000","last_undegraded":"2023-10-16T15:53:29.462449+0000","last_fullsized":"2023-10-16T15:53:29.462449+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2023-10-16T15:53:39.356911+0000","last_change":"2023-10-16T15:53:39.356911+0000","last_active":"2023-10-16T15:53:39.323800+0000","last_peered":"2023-10-16T15:53:39.323800+0000","last_clean":"2023-10-16T15:53:39.323800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:39.356911+0000","last_undegraded":"2023-10-16T15:53:39.356911+0000","last_fullsized":"2023-10-16T15:53:39.356911+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330261+0000","last_change":"2023-10-16T15:53:41.397528+0000","last_active":"2023-10-16T15:53:43.330261+0000","last_peered":"2023-10-16T15:53:43.330261+0000","last_clean":"2023-10-16T15:53:43.330261+0000","last_became_active":"2023-10-16T15:53:41.397247+0000","last_became_peered":"2023-10-16T15:53:41.397247+0000","last_unstale":"2023-10-16T15:53:43.330261+0000","last_undegraded":"2023-10-16T15:53:43.330261+0000","last_fullsized":"2023-10-16T15:53:43.330261+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:21.337037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.373989+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:41.373989+0000","last_peered":"2023-10-16T15:53:41.373989+0000","last_clean":"2023-10-16T15:53:41.373989+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:41.373989+0000","last_undegraded":"2023-10-16T15:53:41.373989+0000","last_fullsized":"2023-10-16T15:53:41.373989+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388628+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:41.388628+0000","last_peered":"2023-10-16T15:53:41.388628+0000","last_clean":"2023-10-16T15:53:41.388628+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:41.388628+0000","last_undegraded":"2023-10-16T15:53:41.388628+0000","last_fullsized":"2023-10-16T15:53:41.388628+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402941+0000","last_change":"2023-10-16T15:53:41.398094+0000","last_active":"2023-10-16T15:53:42.402941+0000","last_peered":"2023-10-16T15:53:42.402941+0000","last_clean":"2023-10-16T15:53:42.402941+0000","last_became_active":"2023-10-16T15:53:41.396310+0000","last_became_peered":"2023-10-16T15:53:41.396310+0000","last_unstale":"2023-10-16T15:53:42.402941+0000","last_undegraded":"2023-10-16T15:53:42.402941+0000","last_fullsized":"2023-10-16T15:53:42.402941+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462282+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:29.462282+0000","last_peered":"2023-10-16T15:53:29.462282+0000","last_clean":"2023-10-16T15:53:29.462282+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:29.462282+0000","last_undegraded":"2023-10-16T15:53:29.462282+0000","last_fullsized":"2023-10-16T15:53:29.462282+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395859+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:41.395859+0000","last_peered":"2023-10-16T15:53:41.395859+0000","last_clean":"2023-10-16T15:53:41.395859+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:41.395859+0000","last_undegraded":"2023-10-16T15:53:41.395859+0000","last_fullsized":"2023-10-16T15:53:41.395859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393894+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:41.393894+0000","last_peered":"2023-10-16T15:53:41.393894+0000","last_clean":"2023-10-16T15:53:41.393894+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:41.393894+0000","last_undegraded":"2023-10-16T15:53:41.393894+0000","last_fullsized":"2023-10-16T15:53:41.393894+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418360+0000","last_change":"2023-10-16T15:53:41.400386+0000","last_active":"2023-10-16T15:53:43.418360+0000","last_peered":"2023-10-16T15:53:43.418360+0000","last_clean":"2023-10-16T15:53:43.418360+0000","last_became_active":"2023-10-16T15:53:41.400083+0000","last_became_peered":"2023-10-16T15:53:41.400083+0000","last_unstale":"2023-10-16T15:53:43.418360+0000","last_undegraded":"2023-10-16T15:53:43.418360+0000","last_fullsized":"2023-10-16T15:53:43.418360+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375718+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:41.375718+0000","last_peered":"2023-10-16T15:53:41.375718+0000","last_clean":"2023-10-16T15:53:41.375718+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:41.375718+0000","last_undegraded":"2023-10-16T15:53:41.375718+0000","last_fullsized":"2023-10-16T15:53:41.375718+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012816+0000","last_change":"2023-10-16T15:53:41.399893+0000","last_active":"2023-10-16T15:53:43.012816+0000","last_peered":"2023-10-16T15:53:43.012816+0000","last_clean":"2023-10-16T15:53:43.012816+0000","last_became_active":"2023-10-16T15:53:41.398053+0000","last_became_peered":"2023-10-16T15:53:41.398053+0000","last_unstale":"2023-10-16T15:53:43.012816+0000","last_undegraded":"2023-10-16T15:53:43.012816+0000","last_fullsized":"2023-10-16T15:53:43.012816+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:18.848509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462645+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:29.462645+0000","last_peered":"2023-10-16T15:53:29.462645+0000","last_clean":"2023-10-16T15:53:29.462645+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:29.462645+0000","last_undegraded":"2023-10-16T15:53:29.462645+0000","last_fullsized":"2023-10-16T15:53:29.462645+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404083+0000","last_change":"2023-10-16T15:53:41.406721+0000","last_active":"2023-10-16T15:53:42.404083+0000","last_peered":"2023-10-16T15:53:42.404083+0000","last_clean":"2023-10-16T15:53:42.404083+0000","last_became_active":"2023-10-16T15:53:41.404619+0000","last_became_peered":"2023-10-16T15:53:41.404619+0000","last_unstale":"2023-10-16T15:53:42.404083+0000","last_undegraded":"2023-10-16T15:53:42.404083+0000","last_fullsized":"2023-10-16T15:53:42.404083+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463041+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:29.463041+0000","last_peered":"2023-10-16T15:53:29.463041+0000","last_clean":"2023-10-16T15:53:29.463041+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:29.463041+0000","last_undegraded":"2023-10-16T15:53:29.463041+0000","last_fullsized":"2023-10-16T15:53:29.463041+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461607+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:29.461607+0000","last_peered":"2023-10-16T15:53:29.461607+0000","last_clean":"2023-10-16T15:53:29.461607+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:29.461607+0000","last_undegraded":"2023-10-16T15:53:29.461607+0000","last_fullsized":"2023-10-16T15:53:29.461607+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402772+0000","last_change":"2023-10-16T15:53:41.399691+0000","last_active":"2023-10-16T15:53:42.402772+0000","last_peered":"2023-10-16T15:53:42.402772+0000","last_clean":"2023-10-16T15:53:42.402772+0000","last_became_active":"2023-10-16T15:53:41.397849+0000","last_became_peered":"2023-10-16T15:53:41.397849+0000","last_unstale":"2023-10-16T15:53:42.402772+0000","last_undegraded":"2023-10-16T15:53:42.402772+0000","last_fullsized":"2023-10-16T15:53:42.402772+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.461742+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:29.461742+0000","last_peered":"2023-10-16T15:53:29.461742+0000","last_clean":"2023-10-16T15:53:29.461742+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:29.461742+0000","last_undegraded":"2023-10-16T15:53:29.461742+0000","last_fullsized":"2023-10-16T15:53:29.461742+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014730+0000","last_change":"2023-10-16T15:53:41.410559+0000","last_active":"2023-10-16T15:53:43.014730+0000","last_peered":"2023-10-16T15:53:43.014730+0000","last_clean":"2023-10-16T15:53:43.014730+0000","last_became_active":"2023-10-16T15:53:41.410258+0000","last_became_peered":"2023-10-16T15:53:41.410258+0000","last_unstale":"2023-10-16T15:53:43.014730+0000","last_undegraded":"2023-10-16T15:53:43.014730+0000","last_fullsized":"2023-10-16T15:53:43.014730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:25.813239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463112+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:29.463112+0000","last_peered":"2023-10-16T15:53:29.463112+0000","last_clean":"2023-10-16T15:53:29.463112+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:29.463112+0000","last_undegraded":"2023-10-16T15:53:29.463112+0000","last_fullsized":"2023-10-16T15:53:29.463112+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418981+0000","last_change":"2023-10-16T15:53:41.416192+0000","last_active":"2023-10-16T15:53:43.418981+0000","last_peered":"2023-10-16T15:53:43.418981+0000","last_clean":"2023-10-16T15:53:43.418981+0000","last_became_active":"2023-10-16T15:53:41.415862+0000","last_became_peered":"2023-10-16T15:53:41.415862+0000","last_unstale":"2023-10-16T15:53:43.418981+0000","last_undegraded":"2023-10-16T15:53:43.418981+0000","last_fullsized":"2023-10-16T15:53:43.418981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383281+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:41.383281+0000","last_peered":"2023-10-16T15:53:41.383281+0000","last_clean":"2023-10-16T15:53:41.383281+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:41.383281+0000","last_undegraded":"2023-10-16T15:53:41.383281+0000","last_fullsized":"2023-10-16T15:53:41.383281+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013226+0000","last_change":"2023-10-16T15:53:41.409431+0000","last_active":"2023-10-16T15:53:43.013226+0000","last_peered":"2023-10-16T15:53:43.013226+0000","last_clean":"2023-10-16T15:53:43.013226+0000","last_became_active":"2023-10-16T15:53:41.409023+0000","last_became_peered":"2023-10-16T15:53:41.409023+0000","last_unstale":"2023-10-16T15:53:43.013226+0000","last_undegraded":"2023-10-16T15:53:43.013226+0000","last_fullsized":"2023-10-16T15:53:43.013226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:31.127234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394621+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:41.394621+0000","last_peered":"2023-10-16T15:53:41.394621+0000","last_clean":"2023-10-16T15:53:41.394621+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:41.394621+0000","last_undegraded":"2023-10-16T15:53:41.394621+0000","last_fullsized":"2023-10-16T15:53:41.394621+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012572+0000","last_change":"2023-10-16T15:53:41.397505+0000","last_active":"2023-10-16T15:53:43.012572+0000","last_peered":"2023-10-16T15:53:43.012572+0000","last_clean":"2023-10-16T15:53:43.012572+0000","last_became_active":"2023-10-16T15:53:41.397198+0000","last_became_peered":"2023-10-16T15:53:41.397198+0000","last_unstale":"2023-10-16T15:53:43.012572+0000","last_undegraded":"2023-10-16T15:53:43.012572+0000","last_fullsized":"2023-10-16T15:53:43.012572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:07.665494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394898+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:41.394898+0000","last_peered":"2023-10-16T15:53:41.394898+0000","last_clean":"2023-10-16T15:53:41.394898+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:41.394898+0000","last_undegraded":"2023-10-16T15:53:41.394898+0000","last_fullsized":"2023-10-16T15:53:41.394898+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374811+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:41.374811+0000","last_peered":"2023-10-16T15:53:41.374811+0000","last_clean":"2023-10-16T15:53:41.374811+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:41.374811+0000","last_undegraded":"2023-10-16T15:53:41.374811+0000","last_fullsized":"2023-10-16T15:53:41.374811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404049+0000","last_change":"2023-10-16T15:53:41.397057+0000","last_active":"2023-10-16T15:53:42.404049+0000","last_peered":"2023-10-16T15:53:42.404049+0000","last_clean":"2023-10-16T15:53:42.404049+0000","last_became_active":"2023-10-16T15:53:41.396163+0000","last_became_peered":"2023-10-16T15:53:41.396163+0000","last_unstale":"2023-10-16T15:53:42.404049+0000","last_undegraded":"2023-10-16T15:53:42.404049+0000","last_fullsized":"2023-10-16T15:53:42.404049+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391109+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:41.391109+0000","last_peered":"2023-10-16T15:53:41.391109+0000","last_clean":"2023-10-16T15:53:41.391109+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:41.391109+0000","last_undegraded":"2023-10-16T15:53:41.391109+0000","last_fullsized":"2023-10-16T15:53:41.391109+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410984+0000","last_change":"2023-10-16T15:53:41.410981+0000","last_active":"2023-10-16T15:53:42.410984+0000","last_peered":"2023-10-16T15:53:42.410984+0000","last_clean":"2023-10-16T15:53:42.410984+0000","last_became_active":"2023-10-16T15:53:41.410713+0000","last_became_peered":"2023-10-16T15:53:41.410713+0000","last_unstale":"2023-10-16T15:53:42.410984+0000","last_undegraded":"2023-10-16T15:53:42.410984+0000","last_fullsized":"2023-10-16T15:53:42.410984+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.539110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374479+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:41.374479+0000","last_peered":"2023-10-16T15:53:41.374479+0000","last_clean":"2023-10-16T15:53:41.374479+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:41.374479+0000","last_undegraded":"2023-10-16T15:53:41.374479+0000","last_fullsized":"2023-10-16T15:53:41.374479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374715+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:41.374715+0000","last_peered":"2023-10-16T15:53:41.374715+0000","last_clean":"2023-10-16T15:53:41.374715+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:41.374715+0000","last_undegraded":"2023-10-16T15:53:41.374715+0000","last_fullsized":"2023-10-16T15:53:41.374715+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396559+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:41.396559+0000","last_peered":"2023-10-16T15:53:41.396559+0000","last_clean":"2023-10-16T15:53:41.396559+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:41.396559+0000","last_undegraded":"2023-10-16T15:53:41.396559+0000","last_fullsized":"2023-10-16T15:53:41.396559+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.400191+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:41.400191+0000","last_peered":"2023-10-16T15:53:41.400191+0000","last_clean":"2023-10-16T15:53:41.400191+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:41.400191+0000","last_undegraded":"2023-10-16T15:53:41.400191+0000","last_fullsized":"2023-10-16T15:53:41.400191+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385167+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:41.385167+0000","last_peered":"2023-10-16T15:53:41.385167+0000","last_clean":"2023-10-16T15:53:41.385167+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:41.385167+0000","last_undegraded":"2023-10-16T15:53:41.385167+0000","last_fullsized":"2023-10-16T15:53:41.385167+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394811+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:41.394811+0000","last_peered":"2023-10-16T15:53:41.394811+0000","last_clean":"2023-10-16T15:53:41.394811+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:41.394811+0000","last_undegraded":"2023-10-16T15:53:41.394811+0000","last_fullsized":"2023-10-16T15:53:41.394811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418006+0000","last_change":"2023-10-16T15:53:41.406425+0000","last_active":"2023-10-16T15:53:43.418006+0000","last_peered":"2023-10-16T15:53:43.418006+0000","last_clean":"2023-10-16T15:53:43.418006+0000","last_became_active":"2023-10-16T15:53:41.405266+0000","last_became_peered":"2023-10-16T15:53:41.405266+0000","last_unstale":"2023-10-16T15:53:43.418006+0000","last_undegraded":"2023-10-16T15:53:43.418006+0000","last_fullsized":"2023-10-16T15:53:43.418006+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:36.594777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462730+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:29.462730+0000","last_peered":"2023-10-16T15:53:29.462730+0000","last_clean":"2023-10-16T15:53:29.462730+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:29.462730+0000","last_undegraded":"2023-10-16T15:53:29.462730+0000","last_fullsized":"2023-10-16T15:53:29.462730+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379861+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:41.379861+0000","last_peered":"2023-10-16T15:53:41.379861+0000","last_clean":"2023-10-16T15:53:41.379861+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:41.379861+0000","last_undegraded":"2023-10-16T15:53:41.379861+0000","last_fullsized":"2023-10-16T15:53:41.379861+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T15:53:40.337868+0000","last_change":"2023-10-16T15:53:40.337868+0000","last_active":"2023-10-16T15:53:40.337868+0000","last_peered":"2023-10-16T15:53:40.337868+0000","last_clean":"2023-10-16T15:53:40.337868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T15:53:40.337868+0000","last_undegraded":"2023-10-16T15:53:40.337868+0000","last_fullsized":"2023-10-16T15:53:40.337868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374213+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:41.374213+0000","last_peered":"2023-10-16T15:53:41.374213+0000","last_clean":"2023-10-16T15:53:41.374213+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:41.374213+0000","last_undegraded":"2023-10-16T15:53:41.374213+0000","last_fullsized":"2023-10-16T15:53:41.374213+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013669+0000","last_change":"2023-10-16T15:53:41.411866+0000","last_active":"2023-10-16T15:53:43.013669+0000","last_peered":"2023-10-16T15:53:43.013669+0000","last_clean":"2023-10-16T15:53:43.013669+0000","last_became_active":"2023-10-16T15:53:41.411569+0000","last_became_peered":"2023-10-16T15:53:41.411569+0000","last_unstale":"2023-10-16T15:53:43.013669+0000","last_undegraded":"2023-10-16T15:53:43.013669+0000","last_fullsized":"2023-10-16T15:53:43.013669+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:01.576399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383777+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:41.383777+0000","last_peered":"2023-10-16T15:53:41.383777+0000","last_clean":"2023-10-16T15:53:41.383777+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:41.383777+0000","last_undegraded":"2023-10-16T15:53:41.383777+0000","last_fullsized":"2023-10-16T15:53:41.383777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409957+0000","last_change":"2023-10-16T15:53:41.399860+0000","last_active":"2023-10-16T15:53:42.409957+0000","last_peered":"2023-10-16T15:53:42.409957+0000","last_clean":"2023-10-16T15:53:42.409957+0000","last_became_active":"2023-10-16T15:53:41.399589+0000","last_became_peered":"2023-10-16T15:53:41.399589+0000","last_unstale":"2023-10-16T15:53:42.409957+0000","last_undegraded":"2023-10-16T15:53:42.409957+0000","last_fullsized":"2023-10-16T15:53:42.409957+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385445+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:41.385445+0000","last_peered":"2023-10-16T15:53:41.385445+0000","last_clean":"2023-10-16T15:53:41.385445+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:41.385445+0000","last_undegraded":"2023-10-16T15:53:41.385445+0000","last_fullsized":"2023-10-16T15:53:41.385445+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402749+0000","last_change":"2023-10-16T15:53:41.399919+0000","last_active":"2023-10-16T15:53:42.402749+0000","last_peered":"2023-10-16T15:53:42.402749+0000","last_clean":"2023-10-16T15:53:42.402749+0000","last_became_active":"2023-10-16T15:53:41.397770+0000","last_became_peered":"2023-10-16T15:53:41.397770+0000","last_unstale":"2023-10-16T15:53:42.402749+0000","last_undegraded":"2023-10-16T15:53:42.402749+0000","last_fullsized":"2023-10-16T15:53:42.402749+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.463182+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:29.463182+0000","last_peered":"2023-10-16T15:53:29.463182+0000","last_clean":"2023-10-16T15:53:29.463182+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:29.463182+0000","last_undegraded":"2023-10-16T15:53:29.463182+0000","last_fullsized":"2023-10-16T15:53:29.463182+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014330+0000","last_change":"2023-10-16T15:53:41.401459+0000","last_active":"2023-10-16T15:53:43.014330+0000","last_peered":"2023-10-16T15:53:43.014330+0000","last_clean":"2023-10-16T15:53:43.014330+0000","last_became_active":"2023-10-16T15:53:41.401147+0000","last_became_peered":"2023-10-16T15:53:41.401147+0000","last_unstale":"2023-10-16T15:53:43.014330+0000","last_undegraded":"2023-10-16T15:53:43.014330+0000","last_fullsized":"2023-10-16T15:53:43.014330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:31.244497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.329048+0000","last_change":"2023-10-16T15:53:41.404413+0000","last_active":"2023-10-16T15:53:43.329048+0000","last_peered":"2023-10-16T15:53:43.329048+0000","last_clean":"2023-10-16T15:53:43.329048+0000","last_became_active":"2023-10-16T15:53:41.403862+0000","last_became_peered":"2023-10-16T15:53:41.403862+0000","last_unstale":"2023-10-16T15:53:43.329048+0000","last_undegraded":"2023-10-16T15:53:43.329048+0000","last_fullsized":"2023-10-16T15:53:43.329048+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:34.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":12,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-16T15:53:29.462408+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:29.462408+0000","last_peered":"2023-10-16T15:53:29.462408+0000","last_clean":"2023-10-16T15:53:29.462408+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:29.462408+0000","last_undegraded":"2023-10-16T15:53:29.462408+0000","last_fullsized":"2023-10-16T15:53:29.462408+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407480+0000","last_change":"2023-10-16T15:53:41.406217+0000","last_active":"2023-10-16T15:53:42.407480+0000","last_peered":"2023-10-16T15:53:42.407480+0000","last_clean":"2023-10-16T15:53:42.407480+0000","last_became_active":"2023-10-16T15:53:41.405870+0000","last_became_peered":"2023-10-16T15:53:41.405870+0000","last_unstale":"2023-10-16T15:53:42.407480+0000","last_undegraded":"2023-10-16T15:53:42.407480+0000","last_fullsized":"2023-10-16T15:53:42.407480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374717+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:41.374717+0000","last_peered":"2023-10-16T15:53:41.374717+0000","last_clean":"2023-10-16T15:53:41.374717+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:41.374717+0000","last_undegraded":"2023-10-16T15:53:41.374717+0000","last_fullsized":"2023-10-16T15:53:41.374717+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":80,"acting":80,"num_store_stats":5},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60892,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310948,"statfs":{"total":96636764160,"available":96574410752,"internally_reserved":0,"allocated":3428352,"data_stored":1976322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":1.1830000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.323},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":1.2769999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":1.141},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.4099999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.9370000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.6419999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.502}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":1.722},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.5960000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1412470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.216},"min":{"1min":0.73699999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7709999999999999,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.117},{"interface":"front","average":{"1min":1.22,"5min":1.276,"15min":1.2609999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.5489999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2949999999999999,"15min":1.2929999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.72199999999999998},"max":{"1min":1.8740000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3420000000000001,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.579,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.353}]},{"osd":2,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.244,"15min":1.2450000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.752},"max":{"1min":1.554,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.431,"15min":1.401},"min":{"1min":1.0680000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.1990000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.88100000000000001,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.46},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.331,"15min":1.339},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.631,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3560000000000001,"15min":1.331},"min":{"1min":0.871,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.325},{"interface":"front","average":{"1min":1.444,"5min":1.4430000000000001,"15min":1.407},"min":{"1min":0.90100000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.6319999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":0.88300000000000001,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.792,"5min":1.9419999999999999,"15min":2.016},"last":1.022},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5640000000000001,"15min":1.5289999999999999},"min":{"1min":1.0309999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.393}]},{"osd":7,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.633,"15min":1.6060000000000001},"min":{"1min":1.1950000000000001,"5min":0.627,"15min":0.627},"max":{"1min":2.149,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.6220000000000001,"15min":1.5980000000000001},"min":{"1min":1.264,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.2849999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29788,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342052,"statfs":{"total":96636764160,"available":96606261248,"internally_reserved":0,"allocated":2826240,"data_stored":1383679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.80200000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.821},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3,"15min":1.2470000000000001},"min":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.639,"5min":2.738,"15min":2.738},"last":0.98399999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.232,"15min":1.236},"min":{"1min":0.91300000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7569999999999999,"5min":1.8140000000000001,"15min":1.8680000000000001},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.284,"15min":1.3109999999999999},"min":{"1min":0.89300000000000002,"5min":0.87,"15min":0.87},"max":{"1min":1.675,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.5600000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2450000000000001,"15min":1.2629999999999999},"min":{"1min":0.85199999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.784,"5min":1.784,"15min":1.9219999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.6990000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.7190000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3660000000000001,"15min":1.347},"min":{"1min":0.93100000000000005,"5min":0.86199999999999999,"15min":0.80000000000000004},"max":{"1min":1.7350000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.605},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.383,"15min":1.3939999999999999},"min":{"1min":1.103,"5min":0.752,"15min":0.752},"max":{"1min":1.7250000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.323}]},{"osd":4,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3759999999999999,"15min":1.3460000000000001},"min":{"1min":0.92300000000000004,"5min":0.85699999999999998,"15min":0.78100000000000003},"max":{"1min":1.8160000000000001,"5min":1.972,"15min":1.9730000000000001},"last":1.9079999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5389999999999999,"15min":1.532},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.77000000000000002},"max":{"1min":1.869,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.4159999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4359999999999999,"15min":1.385},"min":{"1min":1.04,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.768,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.8029999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.458,"15min":1.4390000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.70599999999999996},"max":{"1min":1.7390000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.8460000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.556,"15min":1.544},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.923,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.681},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6499999999999999,"15min":1.643},"min":{"1min":1.1719999999999999,"5min":1.032,"15min":0.70499999999999996},"max":{"1min":1.845,"5min":3.109,"15min":3.109},"last":1.877}]}]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52000,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319840,"statfs":{"total":96636764160,"available":96583516160,"internally_reserved":0,"allocated":2842624,"data_stored":1386058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2330000000000001,"15min":1.212},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.52700000000000002},"max":{"1min":1.794,"5min":1.883,"15min":1.883},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.95799999999999996,"5min":0.83499999999999996,"15min":0.629},"max":{"1min":1.675,"5min":1.929,"15min":2.2160000000000002},"last":1.04},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.3160000000000001},"min":{"1min":0.92000000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0179999999999998,"5min":2.157,"15min":2.2989999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4490000000000001,"15min":1.4670000000000001},"min":{"1min":1.1160000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.399},{"interface":"front","average":{"1min":1.591,"5min":1.498,"15min":1.4970000000000001},"min":{"1min":1.1870000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.119}]},{"osd":3,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":1.081,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.23,"5min":2.23,"15min":2.3650000000000002},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4099999999999999,"15min":1.401},"min":{"1min":1.0069999999999999,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.171}]},{"osd":5,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.375,"15min":1.3420000000000001},"min":{"1min":1.03,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.911,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.552},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.46,"15min":1.4370000000000001},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.161,"5min":2.161,"15min":2.4489999999999998},"last":1.375}]},{"osd":6,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":0.97499999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.554,"15min":1.5269999999999999},"min":{"1min":1.0089999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.2789999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.694}]},{"osd":7,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.617,"15min":1.5860000000000001},"min":{"1min":1.2669999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.6339999999999999,"15min":1.6040000000000001},"min":{"1min":1.3999999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.242,"5min":2.2599999999999998,"15min":2.347},"last":1.9339999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56228,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315612,"statfs":{"total":96636764160,"available":96579186688,"internally_reserved":0,"allocated":2899968,"data_stored":1413696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.44}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":0.92700000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":0.94399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1413500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":0.621}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.772}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":1.335},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.407}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.8049999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":2.093}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.8500000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.9930000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576006,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47884,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323956,"statfs":{"total":96636764160,"available":96587730944,"internally_reserved":0,"allocated":2883584,"data_stored":1411978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.236}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.4470000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":1.679},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.502}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.919}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":1.9810000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":1.867}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48468,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323372,"statfs":{"total":96636764160,"available":96587132928,"internally_reserved":0,"allocated":3477504,"data_stored":2002962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.85199999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":1.1850000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":1.498}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.288},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4650000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.363},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":1.552}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:46.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.755+0000 7f4d30ed7640 1 Processor -- start 2023-10-16T15:53:46.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.755+0000 7f4d30ed7640 1 -- start start 2023-10-16T15:53:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.756+0000 7f4d30ed7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cb730 0x7f4d2c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d30ed7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d2c0ce7f0 0x7f4d2c0cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d30ed7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c0cc940 con 0x7f4d2c0cd510 2023-10-16T15:53:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d30ed7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2c0d1560 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d30ed7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0cde30 con 0x7f4d2c0cb730 2023-10-16T15:53:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d2ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d2c0ce7f0 0x7f4d2c0cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d2ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d2c0ce7f0 0x7f4d2c0cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45824/0 (socket says 172.21.15.29:45824) 2023-10-16T15:53:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/2623278116 learned_addr learned my addr 172.21.15.29:0/2623278116 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.757+0000 7f4d29d9b640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cb730 0x7f4d2c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.758+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/2623278116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cb730 msgr2=0x7f4d2c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.758+0000 7f4d2ad9d640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cb730 0x7f4d2c0cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.758+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/2623278116 >> v1:172.21.15.170:6789/0 conn(0x7f4d2c0cd510 legacy=0x7f4d2c15f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.758+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/2623278116 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0cbd30 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.758+0000 7f4d29d9b640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cb730 0x7f4d2c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.759+0000 7f4d2ad9d640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c0cec80 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009870 tx=0x7f4d2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf1d8f795f37442 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.759+0000 7f4d2959a640 1 -- 172.21.15.29:0/2623278116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24031640 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.759+0000 7f4d2959a640 1 -- 172.21.15.29:0/2623278116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d240317a0 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.759+0000 7f4d2959a640 1 -- 172.21.15.29:0/2623278116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24031b30 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 -- 172.21.15.29:0/2623278116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 msgr2=0x7f4d2c0cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c0cec80 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009870 tx=0x7f4d2402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 -- 172.21.15.29:0/2623278116 shutdown_connections 2023-10-16T15:53:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c0cec80 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/2623278116 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cb730 0x7f4d2c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 -- 172.21.15.29:0/2623278116 >> 172.21.15.29:0/2623278116 conn(0x7f4d2c099ea0 msgr2=0x7f4d2c0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 -- 172.21.15.29:0/2623278116 shutdown_connections 2023-10-16T15:53:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.760+0000 7f4d30ed7640 1 -- 172.21.15.29:0/2623278116 wait complete. 2023-10-16T15:53:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.762+0000 7f4d30ed7640 1 Processor -- start 2023-10-16T15:53:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.762+0000 7f4d30ed7640 1 -- start start 2023-10-16T15:53:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d30ed7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d2c0cb730 0x7f4d2c0c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d30ed7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cd510 0x7f4d2c0b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d30ed7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d30ed7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c09d870 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d30ed7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0c6340 con 0x7f4d2c0cb730 2023-10-16T15:53:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d30ed7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c09b820 con 0x7f4d2c0cd510 2023-10-16T15:53:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d2ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d2ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45840/0 (socket says 172.21.15.29:45840) 2023-10-16T15:53:46.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.763+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/3727389960 learned_addr learned my addr 172.21.15.29:0/3727389960 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:46.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2a59c640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d2c0cb730 0x7f4d2c0c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d29d9b640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cd510 0x7f4d2c0b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cd510 msgr2=0x7f4d2c0b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2ad9d640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cd510 0x7f4d2c0b1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d2c0cb730 msgr2=0x7f4d2c0c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2ad9d640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d2c0cb730 0x7f4d2c0c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2ad9d640 1 -- 172.21.15.29:0/3727389960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d240094d0 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d2ad9d640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c153700 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f4d2402b930 tx=0x7f4d240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.764+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2403ac60 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.765+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2403ade0 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.765+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24044630 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.765+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c09bb00 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.765+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c0c4b90 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.766+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c09d640 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.767+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4d2402f050 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.768+0000 7f4d22ffd640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f4d0c06af90 0x7f4d0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.768+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f4d240abe50 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.768+0000 7f4d2a59c640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f4d0c06af90 0x7f4d0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.770+0000 7f4d2a59c640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f4d0c06af90 0x7f4d0c06d450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c005550 tx=0x7f4d1c009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.772+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4d2c09d640 con 0x7f4d2c0ce7f0 2023-10-16T15:53:46.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:46.966+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d2c0cc010 con 0x7f4d0c06af90 2023-10-16T15:53:47.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.007+0000 7f4d22ffd640 1 -- 172.21.15.29:0/3727389960 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526043 (secure 0 0 0) 0x7f4d1c0051f0 con 0x7f4d0c06af90 2023-10-16T15:53:47.013 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:47.020 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:47.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f4d0c06af90 msgr2=0x7f4d0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f4d0c06af90 0x7f4d0c06d450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c005550 tx=0x7f4d1c009040 comp rx=0 tx=0).stop 2023-10-16T15:53:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 msgr2=0x7f4d2c153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c153700 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f4d2402b930 tx=0x7f4d240046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 shutdown_connections 2023-10-16T15:53:47.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f4d0c06af90 0x7f4d0c06d450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d2c0ce7f0 0x7f4d2c153700 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4d2c0cd510 0x7f4d2c0b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 --2- 172.21.15.29:0/3727389960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4d2c0cb730 0x7f4d2c0c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 >> 172.21.15.29:0/3727389960 conn(0x7f4d2c099ea0 msgr2=0x7f4d2c159480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:47.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 shutdown_connections 2023-10-16T15:53:47.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.018+0000 7f4d30ed7640 1 -- 172.21.15.29:0/3727389960 wait complete. 2023-10-16T15:53:47.047 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-16T15:53:46.463642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":674,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381484,"kb_used_data":23720,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593236,"statfs":{"total":773094113280,"available":772703473664,"internally_reserved":0,"allocated":24289280,"data_stored":12487522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366202425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.114433"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411190+0000","last_change":"2023-10-16T15:53:41.399707+0000","last_active":"2023-10-16T15:53:42.411190+0000","last_peered":"2023-10-16T15:53:42.411190+0000","last_clean":"2023-10-16T15:53:42.411190+0000","last_became_active":"2023-10-16T15:53:41.399411+0000","last_became_peered":"2023-10-16T15:53:41.399411+0000","last_unstale":"2023-10-16T15:53:42.411190+0000","last_undegraded":"2023-10-16T15:53:42.411190+0000","last_fullsized":"2023-10-16T15:53:42.411190+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:45.368014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1251,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376480+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:41.376480+0000","last_peered":"2023-10-16T15:53:41.376480+0000","last_clean":"2023-10-16T15:53:41.376480+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:41.376480+0000","last_undegraded":"2023-10-16T15:53:41.376480+0000","last_fullsized":"2023-10-16T15:53:41.376480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396662+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:41.396662+0000","last_peered":"2023-10-16T15:53:41.396662+0000","last_clean":"2023-10-16T15:53:41.396662+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:41.396662+0000","last_undegraded":"2023-10-16T15:53:41.396662+0000","last_fullsized":"2023-10-16T15:53:41.396662+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375906+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:41.375906+0000","last_peered":"2023-10-16T15:53:41.375906+0000","last_clean":"2023-10-16T15:53:41.375906+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:41.375906+0000","last_undegraded":"2023-10-16T15:53:41.375906+0000","last_fullsized":"2023-10-16T15:53:41.375906+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.410846+0000","last_change":"2023-10-16T15:53:40.387170+0000","last_active":"2023-10-16T15:53:41.410846+0000","last_peered":"2023-10-16T15:53:41.410846+0000","last_clean":"2023-10-16T15:53:41.410846+0000","last_became_active":"2023-10-16T15:53:40.386872+0000","last_became_peered":"2023-10-16T15:53:40.386872+0000","last_unstale":"2023-10-16T15:53:41.410846+0000","last_undegraded":"2023-10-16T15:53:41.410846+0000","last_fullsized":"2023-10-16T15:53:41.410846+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411355+0000","last_change":"2023-10-16T15:53:41.413779+0000","last_active":"2023-10-16T15:53:42.411355+0000","last_peered":"2023-10-16T15:53:42.411355+0000","last_clean":"2023-10-16T15:53:42.411355+0000","last_became_active":"2023-10-16T15:53:41.413379+0000","last_became_peered":"2023-10-16T15:53:41.413379+0000","last_unstale":"2023-10-16T15:53:42.411355+0000","last_undegraded":"2023-10-16T15:53:42.411355+0000","last_fullsized":"2023-10-16T15:53:42.411355+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384903+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:41.384903+0000","last_peered":"2023-10-16T15:53:41.384903+0000","last_clean":"2023-10-16T15:53:41.384903+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:41.384903+0000","last_undegraded":"2023-10-16T15:53:41.384903+0000","last_fullsized":"2023-10-16T15:53:41.384903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386164+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:41.386164+0000","last_peered":"2023-10-16T15:53:41.386164+0000","last_clean":"2023-10-16T15:53:41.386164+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:41.386164+0000","last_undegraded":"2023-10-16T15:53:41.386164+0000","last_fullsized":"2023-10-16T15:53:41.386164+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378909+0000","last_change":"2023-10-16T15:53:40.378024+0000","last_active":"2023-10-16T15:53:41.378909+0000","last_peered":"2023-10-16T15:53:41.378909+0000","last_clean":"2023-10-16T15:53:41.378909+0000","last_became_active":"2023-10-16T15:53:40.377757+0000","last_became_peered":"2023-10-16T15:53:40.377757+0000","last_unstale":"2023-10-16T15:53:41.378909+0000","last_undegraded":"2023-10-16T15:53:41.378909+0000","last_fullsized":"2023-10-16T15:53:41.378909+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:48.714124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014672+0000","last_change":"2023-10-16T15:53:41.418345+0000","last_active":"2023-10-16T15:53:43.014672+0000","last_peered":"2023-10-16T15:53:43.014672+0000","last_clean":"2023-10-16T15:53:43.014672+0000","last_became_active":"2023-10-16T15:53:41.418078+0000","last_became_peered":"2023-10-16T15:53:41.418078+0000","last_unstale":"2023-10-16T15:53:43.014672+0000","last_undegraded":"2023-10-16T15:53:43.014672+0000","last_fullsized":"2023-10-16T15:53:43.014672+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:25.632327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389543+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:41.389543+0000","last_peered":"2023-10-16T15:53:41.389543+0000","last_clean":"2023-10-16T15:53:41.389543+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:41.389543+0000","last_undegraded":"2023-10-16T15:53:41.389543+0000","last_fullsized":"2023-10-16T15:53:41.389543+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380422+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:41.380422+0000","last_peered":"2023-10-16T15:53:41.380422+0000","last_clean":"2023-10-16T15:53:41.380422+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:41.380422+0000","last_undegraded":"2023-10-16T15:53:41.380422+0000","last_fullsized":"2023-10-16T15:53:41.380422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.405443+0000","last_change":"2023-10-16T15:53:40.384327+0000","last_active":"2023-10-16T15:53:41.405443+0000","last_peered":"2023-10-16T15:53:41.405443+0000","last_clean":"2023-10-16T15:53:41.405443+0000","last_became_active":"2023-10-16T15:53:40.384027+0000","last_became_peered":"2023-10-16T15:53:40.384027+0000","last_unstale":"2023-10-16T15:53:41.405443+0000","last_undegraded":"2023-10-16T15:53:41.405443+0000","last_fullsized":"2023-10-16T15:53:41.405443+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:09.776405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014626+0000","last_change":"2023-10-16T15:53:41.393836+0000","last_active":"2023-10-16T15:53:43.014626+0000","last_peered":"2023-10-16T15:53:43.014626+0000","last_clean":"2023-10-16T15:53:43.014626+0000","last_became_active":"2023-10-16T15:53:41.392583+0000","last_became_peered":"2023-10-16T15:53:41.392583+0000","last_unstale":"2023-10-16T15:53:43.014626+0000","last_undegraded":"2023-10-16T15:53:43.014626+0000","last_fullsized":"2023-10-16T15:53:43.014626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:22.777642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384099+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:41.384099+0000","last_peered":"2023-10-16T15:53:41.384099+0000","last_clean":"2023-10-16T15:53:41.384099+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:41.384099+0000","last_undegraded":"2023-10-16T15:53:41.384099+0000","last_fullsized":"2023-10-16T15:53:41.384099+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396904+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:41.396904+0000","last_peered":"2023-10-16T15:53:41.396904+0000","last_clean":"2023-10-16T15:53:41.396904+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:41.396904+0000","last_undegraded":"2023-10-16T15:53:41.396904+0000","last_fullsized":"2023-10-16T15:53:41.396904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395708+0000","last_change":"2023-10-16T15:53:40.378111+0000","last_active":"2023-10-16T15:53:41.395708+0000","last_peered":"2023-10-16T15:53:41.395708+0000","last_clean":"2023-10-16T15:53:41.395708+0000","last_became_active":"2023-10-16T15:53:40.377470+0000","last_became_peered":"2023-10-16T15:53:40.377470+0000","last_unstale":"2023-10-16T15:53:41.395708+0000","last_undegraded":"2023-10-16T15:53:41.395708+0000","last_fullsized":"2023-10-16T15:53:41.395708+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408902+0000","last_change":"2023-10-16T15:53:41.409618+0000","last_active":"2023-10-16T15:53:42.408902+0000","last_peered":"2023-10-16T15:53:42.408902+0000","last_clean":"2023-10-16T15:53:42.408902+0000","last_became_active":"2023-10-16T15:53:41.409365+0000","last_became_peered":"2023-10-16T15:53:41.409365+0000","last_unstale":"2023-10-16T15:53:42.408902+0000","last_undegraded":"2023-10-16T15:53:42.408902+0000","last_fullsized":"2023-10-16T15:53:42.408902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378147+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:41.378147+0000","last_peered":"2023-10-16T15:53:41.378147+0000","last_clean":"2023-10-16T15:53:41.378147+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:41.378147+0000","last_undegraded":"2023-10-16T15:53:41.378147+0000","last_fullsized":"2023-10-16T15:53:41.378147+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383981+0000","last_change":"2023-10-16T15:53:40.381668+0000","last_active":"2023-10-16T15:53:41.383981+0000","last_peered":"2023-10-16T15:53:41.383981+0000","last_clean":"2023-10-16T15:53:41.383981+0000","last_became_active":"2023-10-16T15:53:40.381369+0000","last_became_peered":"2023-10-16T15:53:40.381369+0000","last_unstale":"2023-10-16T15:53:41.383981+0000","last_undegraded":"2023-10-16T15:53:41.383981+0000","last_fullsized":"2023-10-16T15:53:41.383981+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:32.114344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410745+0000","last_change":"2023-10-16T15:53:41.400067+0000","last_active":"2023-10-16T15:53:42.410745+0000","last_peered":"2023-10-16T15:53:42.410745+0000","last_clean":"2023-10-16T15:53:42.410745+0000","last_became_active":"2023-10-16T15:53:41.399778+0000","last_became_peered":"2023-10-16T15:53:41.399778+0000","last_unstale":"2023-10-16T15:53:42.410745+0000","last_undegraded":"2023-10-16T15:53:42.410745+0000","last_fullsized":"2023-10-16T15:53:42.410745+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:18.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384461+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:41.384461+0000","last_peered":"2023-10-16T15:53:41.384461+0000","last_clean":"2023-10-16T15:53:41.384461+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:41.384461+0000","last_undegraded":"2023-10-16T15:53:41.384461+0000","last_fullsized":"2023-10-16T15:53:41.384461+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.397959+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:41.397959+0000","last_peered":"2023-10-16T15:53:41.397959+0000","last_clean":"2023-10-16T15:53:41.397959+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:41.397959+0000","last_undegraded":"2023-10-16T15:53:41.397959+0000","last_fullsized":"2023-10-16T15:53:41.397959+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374462+0000","last_change":"2023-10-16T15:53:40.375759+0000","last_active":"2023-10-16T15:53:41.374462+0000","last_peered":"2023-10-16T15:53:41.374462+0000","last_clean":"2023-10-16T15:53:41.374462+0000","last_became_active":"2023-10-16T15:53:40.375464+0000","last_became_peered":"2023-10-16T15:53:40.375464+0000","last_unstale":"2023-10-16T15:53:41.374462+0000","last_undegraded":"2023-10-16T15:53:41.374462+0000","last_fullsized":"2023-10-16T15:53:41.374462+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405804+0000","last_change":"2023-10-16T15:53:41.407972+0000","last_active":"2023-10-16T15:53:42.405804+0000","last_peered":"2023-10-16T15:53:42.405804+0000","last_clean":"2023-10-16T15:53:42.405804+0000","last_became_active":"2023-10-16T15:53:41.407725+0000","last_became_peered":"2023-10-16T15:53:41.407725+0000","last_unstale":"2023-10-16T15:53:42.405804+0000","last_undegraded":"2023-10-16T15:53:42.405804+0000","last_fullsized":"2023-10-16T15:53:42.405804+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:45.480076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385405+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:41.385405+0000","last_peered":"2023-10-16T15:53:41.385405+0000","last_clean":"2023-10-16T15:53:41.385405+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:41.385405+0000","last_undegraded":"2023-10-16T15:53:41.385405+0000","last_fullsized":"2023-10-16T15:53:41.385405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386135+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:41.386135+0000","last_peered":"2023-10-16T15:53:41.386135+0000","last_clean":"2023-10-16T15:53:41.386135+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:41.386135+0000","last_undegraded":"2023-10-16T15:53:41.386135+0000","last_fullsized":"2023-10-16T15:53:41.386135+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385992+0000","last_change":"2023-10-16T15:53:40.389163+0000","last_active":"2023-10-16T15:53:41.385992+0000","last_peered":"2023-10-16T15:53:41.385992+0000","last_clean":"2023-10-16T15:53:41.385992+0000","last_became_active":"2023-10-16T15:53:40.388144+0000","last_became_peered":"2023-10-16T15:53:40.388144+0000","last_unstale":"2023-10-16T15:53:41.385992+0000","last_undegraded":"2023-10-16T15:53:41.385992+0000","last_fullsized":"2023-10-16T15:53:41.385992+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406367+0000","last_change":"2023-10-16T15:53:41.408615+0000","last_active":"2023-10-16T15:53:42.406367+0000","last_peered":"2023-10-16T15:53:42.406367+0000","last_clean":"2023-10-16T15:53:42.406367+0000","last_became_active":"2023-10-16T15:53:41.408337+0000","last_became_peered":"2023-10-16T15:53:41.408337+0000","last_unstale":"2023-10-16T15:53:42.406367+0000","last_undegraded":"2023-10-16T15:53:42.406367+0000","last_fullsized":"2023-10-16T15:53:42.406367+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:20.232917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"225'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379246+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:41.379246+0000","last_peered":"2023-10-16T15:53:41.379246+0000","last_clean":"2023-10-16T15:53:41.379246+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:41.379246+0000","last_undegraded":"2023-10-16T15:53:41.379246+0000","last_fullsized":"2023-10-16T15:53:41.379246+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376019+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:41.376019+0000","last_peered":"2023-10-16T15:53:41.376019+0000","last_clean":"2023-10-16T15:53:41.376019+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:41.376019+0000","last_undegraded":"2023-10-16T15:53:41.376019+0000","last_fullsized":"2023-10-16T15:53:41.376019+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384628+0000","last_change":"2023-10-16T15:53:40.375284+0000","last_active":"2023-10-16T15:53:41.384628+0000","last_peered":"2023-10-16T15:53:41.384628+0000","last_clean":"2023-10-16T15:53:41.384628+0000","last_became_active":"2023-10-16T15:53:40.374509+0000","last_became_peered":"2023-10-16T15:53:40.374509+0000","last_unstale":"2023-10-16T15:53:41.384628+0000","last_undegraded":"2023-10-16T15:53:41.384628+0000","last_fullsized":"2023-10-16T15:53:41.384628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:59.597435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407164+0000","last_change":"2023-10-16T15:53:41.414757+0000","last_active":"2023-10-16T15:53:42.407164+0000","last_peered":"2023-10-16T15:53:42.407164+0000","last_clean":"2023-10-16T15:53:42.407164+0000","last_became_active":"2023-10-16T15:53:41.414480+0000","last_became_peered":"2023-10-16T15:53:41.414480+0000","last_unstale":"2023-10-16T15:53:42.407164+0000","last_undegraded":"2023-10-16T15:53:42.407164+0000","last_fullsized":"2023-10-16T15:53:42.407164+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387180+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:41.387180+0000","last_peered":"2023-10-16T15:53:41.387180+0000","last_clean":"2023-10-16T15:53:41.387180+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:41.387180+0000","last_undegraded":"2023-10-16T15:53:41.387180+0000","last_fullsized":"2023-10-16T15:53:41.387180+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395410+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:41.395410+0000","last_peered":"2023-10-16T15:53:41.395410+0000","last_clean":"2023-10-16T15:53:41.395410+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:41.395410+0000","last_undegraded":"2023-10-16T15:53:41.395410+0000","last_fullsized":"2023-10-16T15:53:41.395410+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382520+0000","last_change":"2023-10-16T15:53:40.381870+0000","last_active":"2023-10-16T15:53:41.382520+0000","last_peered":"2023-10-16T15:53:41.382520+0000","last_clean":"2023-10-16T15:53:41.382520+0000","last_became_active":"2023-10-16T15:53:40.381577+0000","last_became_peered":"2023-10-16T15:53:40.381577+0000","last_unstale":"2023-10-16T15:53:41.382520+0000","last_undegraded":"2023-10-16T15:53:41.382520+0000","last_fullsized":"2023-10-16T15:53:41.382520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:33.030599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404650+0000","last_change":"2023-10-16T15:53:41.408405+0000","last_active":"2023-10-16T15:53:42.404650+0000","last_peered":"2023-10-16T15:53:42.404650+0000","last_clean":"2023-10-16T15:53:42.404650+0000","last_became_active":"2023-10-16T15:53:41.407251+0000","last_became_peered":"2023-10-16T15:53:41.407251+0000","last_unstale":"2023-10-16T15:53:42.404650+0000","last_undegraded":"2023-10-16T15:53:42.404650+0000","last_fullsized":"2023-10-16T15:53:42.404650+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.381328+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:41.381328+0000","last_peered":"2023-10-16T15:53:41.381328+0000","last_clean":"2023-10-16T15:53:41.381328+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:41.381328+0000","last_undegraded":"2023-10-16T15:53:41.381328+0000","last_fullsized":"2023-10-16T15:53:41.381328+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389408+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:41.389408+0000","last_peered":"2023-10-16T15:53:41.389408+0000","last_clean":"2023-10-16T15:53:41.389408+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:41.389408+0000","last_undegraded":"2023-10-16T15:53:41.389408+0000","last_fullsized":"2023-10-16T15:53:41.389408+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414970+0000","last_change":"2023-10-16T15:53:40.373067+0000","last_active":"2023-10-16T15:53:41.414970+0000","last_peered":"2023-10-16T15:53:41.414970+0000","last_clean":"2023-10-16T15:53:41.414970+0000","last_became_active":"2023-10-16T15:53:40.372307+0000","last_became_peered":"2023-10-16T15:53:40.372307+0000","last_unstale":"2023-10-16T15:53:41.414970+0000","last_undegraded":"2023-10-16T15:53:41.414970+0000","last_fullsized":"2023-10-16T15:53:41.414970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330512+0000","last_change":"2023-10-16T15:53:41.400515+0000","last_active":"2023-10-16T15:53:43.330512+0000","last_peered":"2023-10-16T15:53:43.330512+0000","last_clean":"2023-10-16T15:53:43.330512+0000","last_became_active":"2023-10-16T15:53:41.399835+0000","last_became_peered":"2023-10-16T15:53:41.399835+0000","last_unstale":"2023-10-16T15:53:43.330512+0000","last_undegraded":"2023-10-16T15:53:43.330512+0000","last_fullsized":"2023-10-16T15:53:43.330512+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:40.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375870+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:41.375870+0000","last_peered":"2023-10-16T15:53:41.375870+0000","last_clean":"2023-10-16T15:53:41.375870+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:41.375870+0000","last_undegraded":"2023-10-16T15:53:41.375870+0000","last_fullsized":"2023-10-16T15:53:41.375870+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384670+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:41.384670+0000","last_peered":"2023-10-16T15:53:41.384670+0000","last_clean":"2023-10-16T15:53:41.384670+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:41.384670+0000","last_undegraded":"2023-10-16T15:53:41.384670+0000","last_fullsized":"2023-10-16T15:53:41.384670+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.407785+0000","last_change":"2023-10-16T15:53:40.378428+0000","last_active":"2023-10-16T15:53:41.407785+0000","last_peered":"2023-10-16T15:53:41.407785+0000","last_clean":"2023-10-16T15:53:41.407785+0000","last_became_active":"2023-10-16T15:53:40.378077+0000","last_became_peered":"2023-10-16T15:53:40.378077+0000","last_unstale":"2023-10-16T15:53:41.407785+0000","last_undegraded":"2023-10-16T15:53:41.407785+0000","last_fullsized":"2023-10-16T15:53:41.407785+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410289+0000","last_change":"2023-10-16T15:53:41.399412+0000","last_active":"2023-10-16T15:53:42.410289+0000","last_peered":"2023-10-16T15:53:42.410289+0000","last_clean":"2023-10-16T15:53:42.410289+0000","last_became_active":"2023-10-16T15:53:41.399125+0000","last_became_peered":"2023-10-16T15:53:41.399125+0000","last_unstale":"2023-10-16T15:53:42.410289+0000","last_undegraded":"2023-10-16T15:53:42.410289+0000","last_fullsized":"2023-10-16T15:53:42.410289+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376676+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:41.376676+0000","last_peered":"2023-10-16T15:53:41.376676+0000","last_clean":"2023-10-16T15:53:41.376676+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:41.376676+0000","last_undegraded":"2023-10-16T15:53:41.376676+0000","last_fullsized":"2023-10-16T15:53:41.376676+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394140+0000","last_change":"2023-10-16T15:53:40.377426+0000","last_active":"2023-10-16T15:53:41.394140+0000","last_peered":"2023-10-16T15:53:41.394140+0000","last_clean":"2023-10-16T15:53:41.394140+0000","last_became_active":"2023-10-16T15:53:40.377160+0000","last_became_peered":"2023-10-16T15:53:40.377160+0000","last_unstale":"2023-10-16T15:53:41.394140+0000","last_undegraded":"2023-10-16T15:53:41.394140+0000","last_fullsized":"2023-10-16T15:53:41.394140+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:38.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408560+0000","last_change":"2023-10-16T15:53:41.400288+0000","last_active":"2023-10-16T15:53:42.408560+0000","last_peered":"2023-10-16T15:53:42.408560+0000","last_clean":"2023-10-16T15:53:42.408560+0000","last_became_active":"2023-10-16T15:53:41.400031+0000","last_became_peered":"2023-10-16T15:53:41.400031+0000","last_unstale":"2023-10-16T15:53:42.408560+0000","last_undegraded":"2023-10-16T15:53:42.408560+0000","last_fullsized":"2023-10-16T15:53:42.408560+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393978+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:41.393978+0000","last_peered":"2023-10-16T15:53:41.393978+0000","last_clean":"2023-10-16T15:53:41.393978+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:41.393978+0000","last_undegraded":"2023-10-16T15:53:41.393978+0000","last_fullsized":"2023-10-16T15:53:41.393978+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392991+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:41.392991+0000","last_peered":"2023-10-16T15:53:41.392991+0000","last_clean":"2023-10-16T15:53:41.392991+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:41.392991+0000","last_undegraded":"2023-10-16T15:53:41.392991+0000","last_fullsized":"2023-10-16T15:53:41.392991+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414694+0000","last_change":"2023-10-16T15:53:40.379991+0000","last_active":"2023-10-16T15:53:41.414694+0000","last_peered":"2023-10-16T15:53:41.414694+0000","last_clean":"2023-10-16T15:53:41.414694+0000","last_became_active":"2023-10-16T15:53:40.379687+0000","last_became_peered":"2023-10-16T15:53:40.379687+0000","last_unstale":"2023-10-16T15:53:41.414694+0000","last_undegraded":"2023-10-16T15:53:41.414694+0000","last_fullsized":"2023-10-16T15:53:41.414694+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:39.813921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409916+0000","last_change":"2023-10-16T15:53:41.413217+0000","last_active":"2023-10-16T15:53:42.409916+0000","last_peered":"2023-10-16T15:53:42.409916+0000","last_clean":"2023-10-16T15:53:42.409916+0000","last_became_active":"2023-10-16T15:53:41.411878+0000","last_became_peered":"2023-10-16T15:53:41.411878+0000","last_unstale":"2023-10-16T15:53:42.409916+0000","last_undegraded":"2023-10-16T15:53:42.409916+0000","last_fullsized":"2023-10-16T15:53:42.409916+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384277+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:41.384277+0000","last_peered":"2023-10-16T15:53:41.384277+0000","last_clean":"2023-10-16T15:53:41.384277+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:41.384277+0000","last_undegraded":"2023-10-16T15:53:41.384277+0000","last_fullsized":"2023-10-16T15:53:41.384277+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394436+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:41.394436+0000","last_peered":"2023-10-16T15:53:41.394436+0000","last_clean":"2023-10-16T15:53:41.394436+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:41.394436+0000","last_undegraded":"2023-10-16T15:53:41.394436+0000","last_fullsized":"2023-10-16T15:53:41.394436+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389135+0000","last_change":"2023-10-16T15:53:40.384061+0000","last_active":"2023-10-16T15:53:41.389135+0000","last_peered":"2023-10-16T15:53:41.389135+0000","last_clean":"2023-10-16T15:53:41.389135+0000","last_became_active":"2023-10-16T15:53:40.383223+0000","last_became_peered":"2023-10-16T15:53:40.383223+0000","last_unstale":"2023-10-16T15:53:41.389135+0000","last_undegraded":"2023-10-16T15:53:41.389135+0000","last_fullsized":"2023-10-16T15:53:41.389135+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013025+0000","last_change":"2023-10-16T15:53:41.397581+0000","last_active":"2023-10-16T15:53:43.013025+0000","last_peered":"2023-10-16T15:53:43.013025+0000","last_clean":"2023-10-16T15:53:43.013025+0000","last_became_active":"2023-10-16T15:53:41.397301+0000","last_became_peered":"2023-10-16T15:53:41.397301+0000","last_unstale":"2023-10-16T15:53:43.013025+0000","last_undegraded":"2023-10-16T15:53:43.013025+0000","last_fullsized":"2023-10-16T15:53:43.013025+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:20.816418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394523+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:41.394523+0000","last_peered":"2023-10-16T15:53:41.394523+0000","last_clean":"2023-10-16T15:53:41.394523+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:41.394523+0000","last_undegraded":"2023-10-16T15:53:41.394523+0000","last_fullsized":"2023-10-16T15:53:41.394523+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.411952+0000","last_change":"2023-10-16T15:53:40.376976+0000","last_active":"2023-10-16T15:53:41.411952+0000","last_peered":"2023-10-16T15:53:41.411952+0000","last_clean":"2023-10-16T15:53:41.411952+0000","last_became_active":"2023-10-16T15:53:40.376700+0000","last_became_peered":"2023-10-16T15:53:40.376700+0000","last_unstale":"2023-10-16T15:53:41.411952+0000","last_undegraded":"2023-10-16T15:53:41.411952+0000","last_fullsized":"2023-10-16T15:53:41.411952+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013467+0000","last_change":"2023-10-16T15:53:41.410633+0000","last_active":"2023-10-16T15:53:43.013467+0000","last_peered":"2023-10-16T15:53:43.013467+0000","last_clean":"2023-10-16T15:53:43.013467+0000","last_became_active":"2023-10-16T15:53:41.409266+0000","last_became_peered":"2023-10-16T15:53:41.409266+0000","last_unstale":"2023-10-16T15:53:43.013467+0000","last_undegraded":"2023-10-16T15:53:43.013467+0000","last_fullsized":"2023-10-16T15:53:43.013467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:52.749663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392803+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:41.392803+0000","last_peered":"2023-10-16T15:53:41.392803+0000","last_clean":"2023-10-16T15:53:41.392803+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:41.392803+0000","last_undegraded":"2023-10-16T15:53:41.392803+0000","last_fullsized":"2023-10-16T15:53:41.392803+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393648+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:41.393648+0000","last_peered":"2023-10-16T15:53:41.393648+0000","last_clean":"2023-10-16T15:53:41.393648+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:41.393648+0000","last_undegraded":"2023-10-16T15:53:41.393648+0000","last_fullsized":"2023-10-16T15:53:41.393648+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384497+0000","last_change":"2023-10-16T15:53:40.375609+0000","last_active":"2023-10-16T15:53:41.384497+0000","last_peered":"2023-10-16T15:53:41.384497+0000","last_clean":"2023-10-16T15:53:41.384497+0000","last_became_active":"2023-10-16T15:53:40.375039+0000","last_became_peered":"2023-10-16T15:53:40.375039+0000","last_unstale":"2023-10-16T15:53:41.384497+0000","last_undegraded":"2023-10-16T15:53:41.384497+0000","last_fullsized":"2023-10-16T15:53:41.384497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:02.147563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.328614+0000","last_change":"2023-10-16T15:53:41.399266+0000","last_active":"2023-10-16T15:53:43.328614+0000","last_peered":"2023-10-16T15:53:43.328614+0000","last_clean":"2023-10-16T15:53:43.328614+0000","last_became_active":"2023-10-16T15:53:41.398177+0000","last_became_peered":"2023-10-16T15:53:41.398177+0000","last_unstale":"2023-10-16T15:53:43.328614+0000","last_undegraded":"2023-10-16T15:53:43.328614+0000","last_fullsized":"2023-10-16T15:53:43.328614+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.144341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385470+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:41.385470+0000","last_peered":"2023-10-16T15:53:41.385470+0000","last_clean":"2023-10-16T15:53:41.385470+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:41.385470+0000","last_undegraded":"2023-10-16T15:53:41.385470+0000","last_fullsized":"2023-10-16T15:53:41.385470+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392538+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:41.392538+0000","last_peered":"2023-10-16T15:53:41.392538+0000","last_clean":"2023-10-16T15:53:41.392538+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:41.392538+0000","last_undegraded":"2023-10-16T15:53:41.392538+0000","last_fullsized":"2023-10-16T15:53:41.392538+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386003+0000","last_change":"2023-10-16T15:53:40.388836+0000","last_active":"2023-10-16T15:53:41.386003+0000","last_peered":"2023-10-16T15:53:41.386003+0000","last_clean":"2023-10-16T15:53:41.386003+0000","last_became_active":"2023-10-16T15:53:40.387121+0000","last_became_peered":"2023-10-16T15:53:40.387121+0000","last_unstale":"2023-10-16T15:53:41.386003+0000","last_undegraded":"2023-10-16T15:53:41.386003+0000","last_fullsized":"2023-10-16T15:53:41.386003+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410790+0000","last_change":"2023-10-16T15:53:41.413661+0000","last_active":"2023-10-16T15:53:42.410790+0000","last_peered":"2023-10-16T15:53:42.410790+0000","last_clean":"2023-10-16T15:53:42.410790+0000","last_became_active":"2023-10-16T15:53:41.413219+0000","last_became_peered":"2023-10-16T15:53:41.413219+0000","last_unstale":"2023-10-16T15:53:42.410790+0000","last_undegraded":"2023-10-16T15:53:42.410790+0000","last_fullsized":"2023-10-16T15:53:42.410790+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:51.757315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380707+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:41.380707+0000","last_peered":"2023-10-16T15:53:41.380707+0000","last_clean":"2023-10-16T15:53:41.380707+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:41.380707+0000","last_undegraded":"2023-10-16T15:53:41.380707+0000","last_fullsized":"2023-10-16T15:53:41.380707+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374760+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:41.374760+0000","last_peered":"2023-10-16T15:53:41.374760+0000","last_clean":"2023-10-16T15:53:41.374760+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:41.374760+0000","last_undegraded":"2023-10-16T15:53:41.374760+0000","last_fullsized":"2023-10-16T15:53:41.374760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375756+0000","last_change":"2023-10-16T15:53:40.377615+0000","last_active":"2023-10-16T15:53:41.375756+0000","last_peered":"2023-10-16T15:53:41.375756+0000","last_clean":"2023-10-16T15:53:41.375756+0000","last_became_active":"2023-10-16T15:53:40.377342+0000","last_became_peered":"2023-10-16T15:53:40.377342+0000","last_unstale":"2023-10-16T15:53:41.375756+0000","last_undegraded":"2023-10-16T15:53:41.375756+0000","last_fullsized":"2023-10-16T15:53:41.375756+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.412874+0000","last_change":"2023-10-16T15:53:41.410293+0000","last_active":"2023-10-16T15:53:42.412874+0000","last_peered":"2023-10-16T15:53:42.412874+0000","last_clean":"2023-10-16T15:53:42.412874+0000","last_became_active":"2023-10-16T15:53:41.409897+0000","last_became_peered":"2023-10-16T15:53:41.409897+0000","last_unstale":"2023-10-16T15:53:42.412874+0000","last_undegraded":"2023-10-16T15:53:42.412874+0000","last_fullsized":"2023-10-16T15:53:42.412874+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:31.143645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388735+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:41.388735+0000","last_peered":"2023-10-16T15:53:41.388735+0000","last_clean":"2023-10-16T15:53:41.388735+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:41.388735+0000","last_undegraded":"2023-10-16T15:53:41.388735+0000","last_fullsized":"2023-10-16T15:53:41.388735+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380271+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:41.380271+0000","last_peered":"2023-10-16T15:53:41.380271+0000","last_clean":"2023-10-16T15:53:41.380271+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:41.380271+0000","last_undegraded":"2023-10-16T15:53:41.380271+0000","last_fullsized":"2023-10-16T15:53:41.380271+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.397754+0000","last_change":"2023-10-16T15:53:40.387403+0000","last_active":"2023-10-16T15:53:41.397754+0000","last_peered":"2023-10-16T15:53:41.397754+0000","last_clean":"2023-10-16T15:53:41.397754+0000","last_became_active":"2023-10-16T15:53:40.387110+0000","last_became_peered":"2023-10-16T15:53:40.387110+0000","last_unstale":"2023-10-16T15:53:41.397754+0000","last_undegraded":"2023-10-16T15:53:41.397754+0000","last_fullsized":"2023-10-16T15:53:41.397754+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407976+0000","last_change":"2023-10-16T15:53:41.414132+0000","last_active":"2023-10-16T15:53:42.407976+0000","last_peered":"2023-10-16T15:53:42.407976+0000","last_clean":"2023-10-16T15:53:42.407976+0000","last_became_active":"2023-10-16T15:53:41.413151+0000","last_became_peered":"2023-10-16T15:53:41.413151+0000","last_unstale":"2023-10-16T15:53:42.407976+0000","last_undegraded":"2023-10-16T15:53:42.407976+0000","last_fullsized":"2023-10-16T15:53:42.407976+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393257+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:41.393257+0000","last_peered":"2023-10-16T15:53:41.393257+0000","last_clean":"2023-10-16T15:53:41.393257+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.393257+0000","last_undegraded":"2023-10-16T15:53:41.393257+0000","last_fullsized":"2023-10-16T15:53:41.393257+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393875+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:41.393875+0000","last_peered":"2023-10-16T15:53:41.393875+0000","last_clean":"2023-10-16T15:53:41.393875+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:41.393875+0000","last_undegraded":"2023-10-16T15:53:41.393875+0000","last_fullsized":"2023-10-16T15:53:41.393875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"228'6","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414139+0000","last_change":"2023-10-16T15:53:40.376773+0000","last_active":"2023-10-16T15:53:41.414139+0000","last_peered":"2023-10-16T15:53:41.414139+0000","last_clean":"2023-10-16T15:53:41.414139+0000","last_became_active":"2023-10-16T15:53:40.376123+0000","last_became_peered":"2023-10-16T15:53:40.376123+0000","last_unstale":"2023-10-16T15:53:41.414139+0000","last_undegraded":"2023-10-16T15:53:41.414139+0000","last_fullsized":"2023-10-16T15:53:41.414139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:19.218849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.422217+0000","last_change":"2023-10-16T15:53:41.410543+0000","last_active":"2023-10-16T15:53:43.422217+0000","last_peered":"2023-10-16T15:53:43.422217+0000","last_clean":"2023-10-16T15:53:43.422217+0000","last_became_active":"2023-10-16T15:53:41.410213+0000","last_became_peered":"2023-10-16T15:53:41.410213+0000","last_unstale":"2023-10-16T15:53:43.422217+0000","last_undegraded":"2023-10-16T15:53:43.422217+0000","last_fullsized":"2023-10-16T15:53:43.422217+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387904+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:41.387904+0000","last_peered":"2023-10-16T15:53:41.387904+0000","last_clean":"2023-10-16T15:53:41.387904+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:41.387904+0000","last_undegraded":"2023-10-16T15:53:41.387904+0000","last_fullsized":"2023-10-16T15:53:41.387904+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383904+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:41.383904+0000","last_peered":"2023-10-16T15:53:41.383904+0000","last_clean":"2023-10-16T15:53:41.383904+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:41.383904+0000","last_undegraded":"2023-10-16T15:53:41.383904+0000","last_fullsized":"2023-10-16T15:53:41.383904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"228'3","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402070+0000","last_change":"2023-10-16T15:53:40.365148+0000","last_active":"2023-10-16T15:53:41.402070+0000","last_peered":"2023-10-16T15:53:41.402070+0000","last_clean":"2023-10-16T15:53:41.402070+0000","last_became_active":"2023-10-16T15:53:40.364874+0000","last_became_peered":"2023-10-16T15:53:40.364874+0000","last_unstale":"2023-10-16T15:53:41.402070+0000","last_undegraded":"2023-10-16T15:53:41.402070+0000","last_fullsized":"2023-10-16T15:53:41.402070+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:39.387182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411499+0000","last_change":"2023-10-16T15:53:41.405376+0000","last_active":"2023-10-16T15:53:42.411499+0000","last_peered":"2023-10-16T15:53:42.411499+0000","last_clean":"2023-10-16T15:53:42.411499+0000","last_became_active":"2023-10-16T15:53:41.404697+0000","last_became_peered":"2023-10-16T15:53:41.404697+0000","last_unstale":"2023-10-16T15:53:42.411499+0000","last_undegraded":"2023-10-16T15:53:42.411499+0000","last_fullsized":"2023-10-16T15:53:42.411499+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"225'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379765+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:41.379765+0000","last_peered":"2023-10-16T15:53:41.379765+0000","last_clean":"2023-10-16T15:53:41.379765+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:41.379765+0000","last_undegraded":"2023-10-16T15:53:41.379765+0000","last_fullsized":"2023-10-16T15:53:41.379765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376419+0000","last_change":"2023-10-16T15:53:40.387700+0000","last_active":"2023-10-16T15:53:41.376419+0000","last_peered":"2023-10-16T15:53:41.376419+0000","last_clean":"2023-10-16T15:53:41.376419+0000","last_became_active":"2023-10-16T15:53:40.387431+0000","last_became_peered":"2023-10-16T15:53:40.387431+0000","last_unstale":"2023-10-16T15:53:41.376419+0000","last_undegraded":"2023-10-16T15:53:41.376419+0000","last_fullsized":"2023-10-16T15:53:41.376419+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:19.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418316+0000","last_change":"2023-10-16T15:53:41.385042+0000","last_active":"2023-10-16T15:53:43.418316+0000","last_peered":"2023-10-16T15:53:43.418316+0000","last_clean":"2023-10-16T15:53:43.418316+0000","last_became_active":"2023-10-16T15:53:41.384431+0000","last_became_peered":"2023-10-16T15:53:41.384431+0000","last_unstale":"2023-10-16T15:53:43.418316+0000","last_undegraded":"2023-10-16T15:53:43.418316+0000","last_fullsized":"2023-10-16T15:53:43.418316+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394713+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:41.394713+0000","last_peered":"2023-10-16T15:53:41.394713+0000","last_clean":"2023-10-16T15:53:41.394713+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:41.394713+0000","last_undegraded":"2023-10-16T15:53:41.394713+0000","last_fullsized":"2023-10-16T15:53:41.394713+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377982+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:41.377982+0000","last_peered":"2023-10-16T15:53:41.377982+0000","last_clean":"2023-10-16T15:53:41.377982+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:41.377982+0000","last_undegraded":"2023-10-16T15:53:41.377982+0000","last_fullsized":"2023-10-16T15:53:41.377982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.408329+0000","last_change":"2023-10-16T15:53:40.386183+0000","last_active":"2023-10-16T15:53:41.408329+0000","last_peered":"2023-10-16T15:53:41.408329+0000","last_clean":"2023-10-16T15:53:41.408329+0000","last_became_active":"2023-10-16T15:53:40.385897+0000","last_became_peered":"2023-10-16T15:53:40.385897+0000","last_unstale":"2023-10-16T15:53:41.408329+0000","last_undegraded":"2023-10-16T15:53:41.408329+0000","last_fullsized":"2023-10-16T15:53:41.408329+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:27.631953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.420342+0000","last_change":"2023-10-16T15:53:41.400842+0000","last_active":"2023-10-16T15:53:43.420342+0000","last_peered":"2023-10-16T15:53:43.420342+0000","last_clean":"2023-10-16T15:53:43.420342+0000","last_became_active":"2023-10-16T15:53:41.400579+0000","last_became_peered":"2023-10-16T15:53:41.400579+0000","last_unstale":"2023-10-16T15:53:43.420342+0000","last_undegraded":"2023-10-16T15:53:43.420342+0000","last_fullsized":"2023-10-16T15:53:43.420342+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402696+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:41.402696+0000","last_peered":"2023-10-16T15:53:41.402696+0000","last_clean":"2023-10-16T15:53:41.402696+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:41.402696+0000","last_undegraded":"2023-10-16T15:53:41.402696+0000","last_fullsized":"2023-10-16T15:53:41.402696+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387083+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:41.387083+0000","last_peered":"2023-10-16T15:53:41.387083+0000","last_clean":"2023-10-16T15:53:41.387083+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:41.387083+0000","last_undegraded":"2023-10-16T15:53:41.387083+0000","last_fullsized":"2023-10-16T15:53:41.387083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"228'8","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.422402+0000","last_change":"2023-10-16T15:53:40.375924+0000","last_active":"2023-10-16T15:53:41.422402+0000","last_peered":"2023-10-16T15:53:41.422402+0000","last_clean":"2023-10-16T15:53:41.422402+0000","last_became_active":"2023-10-16T15:53:40.375638+0000","last_became_peered":"2023-10-16T15:53:40.375638+0000","last_unstale":"2023-10-16T15:53:41.422402+0000","last_undegraded":"2023-10-16T15:53:41.422402+0000","last_fullsized":"2023-10-16T15:53:41.422402+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:47.381835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407893+0000","last_change":"2023-10-16T15:53:41.414765+0000","last_active":"2023-10-16T15:53:42.407893+0000","last_peered":"2023-10-16T15:53:42.407893+0000","last_clean":"2023-10-16T15:53:42.407893+0000","last_became_active":"2023-10-16T15:53:41.414484+0000","last_became_peered":"2023-10-16T15:53:41.414484+0000","last_unstale":"2023-10-16T15:53:42.407893+0000","last_undegraded":"2023-10-16T15:53:42.407893+0000","last_fullsized":"2023-10-16T15:53:42.407893+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385369+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:41.385369+0000","last_peered":"2023-10-16T15:53:41.385369+0000","last_clean":"2023-10-16T15:53:41.385369+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:41.385369+0000","last_undegraded":"2023-10-16T15:53:41.385369+0000","last_fullsized":"2023-10-16T15:53:41.385369+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379140+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:41.379140+0000","last_peered":"2023-10-16T15:53:41.379140+0000","last_clean":"2023-10-16T15:53:41.379140+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:41.379140+0000","last_undegraded":"2023-10-16T15:53:41.379140+0000","last_fullsized":"2023-10-16T15:53:41.379140+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382349+0000","last_change":"2023-10-16T15:53:40.376076+0000","last_active":"2023-10-16T15:53:41.382349+0000","last_peered":"2023-10-16T15:53:41.382349+0000","last_clean":"2023-10-16T15:53:41.382349+0000","last_became_active":"2023-10-16T15:53:40.375798+0000","last_became_peered":"2023-10-16T15:53:40.375798+0000","last_unstale":"2023-10-16T15:53:41.382349+0000","last_undegraded":"2023-10-16T15:53:41.382349+0000","last_fullsized":"2023-10-16T15:53:41.382349+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330851+0000","last_change":"2023-10-16T15:53:41.409878+0000","last_active":"2023-10-16T15:53:43.330851+0000","last_peered":"2023-10-16T15:53:43.330851+0000","last_clean":"2023-10-16T15:53:43.330851+0000","last_became_active":"2023-10-16T15:53:41.409594+0000","last_became_peered":"2023-10-16T15:53:41.409594+0000","last_unstale":"2023-10-16T15:53:43.330851+0000","last_undegraded":"2023-10-16T15:53:43.330851+0000","last_fullsized":"2023-10-16T15:53:43.330851+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:10.790473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.419545+0000","last_change":"2023-10-16T15:53:40.385839+0000","last_active":"2023-10-16T15:53:41.419545+0000","last_peered":"2023-10-16T15:53:41.419545+0000","last_clean":"2023-10-16T15:53:41.419545+0000","last_became_active":"2023-10-16T15:53:40.384731+0000","last_became_peered":"2023-10-16T15:53:40.384731+0000","last_unstale":"2023-10-16T15:53:41.419545+0000","last_undegraded":"2023-10-16T15:53:41.419545+0000","last_fullsized":"2023-10-16T15:53:41.419545+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:17.818654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386473+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:41.386473+0000","last_peered":"2023-10-16T15:53:41.386473+0000","last_clean":"2023-10-16T15:53:41.386473+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:41.386473+0000","last_undegraded":"2023-10-16T15:53:41.386473+0000","last_fullsized":"2023-10-16T15:53:41.386473+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378524+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:41.378524+0000","last_peered":"2023-10-16T15:53:41.378524+0000","last_clean":"2023-10-16T15:53:41.378524+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:41.378524+0000","last_undegraded":"2023-10-16T15:53:41.378524+0000","last_fullsized":"2023-10-16T15:53:41.378524+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410670+0000","last_change":"2023-10-16T15:53:41.411697+0000","last_active":"2023-10-16T15:53:42.410670+0000","last_peered":"2023-10-16T15:53:42.410670+0000","last_clean":"2023-10-16T15:53:42.410670+0000","last_became_active":"2023-10-16T15:53:41.410477+0000","last_became_peered":"2023-10-16T15:53:41.410477+0000","last_unstale":"2023-10-16T15:53:42.410670+0000","last_undegraded":"2023-10-16T15:53:42.410670+0000","last_fullsized":"2023-10-16T15:53:42.410670+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:52.524535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"228'4","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.418437+0000","last_change":"2023-10-16T15:53:40.380112+0000","last_active":"2023-10-16T15:53:41.418437+0000","last_peered":"2023-10-16T15:53:41.418437+0000","last_clean":"2023-10-16T15:53:41.418437+0000","last_became_active":"2023-10-16T15:53:40.379085+0000","last_became_peered":"2023-10-16T15:53:40.379085+0000","last_unstale":"2023-10-16T15:53:41.418437+0000","last_undegraded":"2023-10-16T15:53:41.418437+0000","last_fullsized":"2023-10-16T15:53:41.418437+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:48.756275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382648+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:41.382648+0000","last_peered":"2023-10-16T15:53:41.382648+0000","last_clean":"2023-10-16T15:53:41.382648+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:41.382648+0000","last_undegraded":"2023-10-16T15:53:41.382648+0000","last_fullsized":"2023-10-16T15:53:41.382648+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379944+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:41.379944+0000","last_peered":"2023-10-16T15:53:41.379944+0000","last_clean":"2023-10-16T15:53:41.379944+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:41.379944+0000","last_undegraded":"2023-10-16T15:53:41.379944+0000","last_fullsized":"2023-10-16T15:53:41.379944+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421686+0000","last_change":"2023-10-16T15:53:41.405016+0000","last_active":"2023-10-16T15:53:43.421686+0000","last_peered":"2023-10-16T15:53:43.421686+0000","last_clean":"2023-10-16T15:53:43.421686+0000","last_became_active":"2023-10-16T15:53:41.404775+0000","last_became_peered":"2023-10-16T15:53:41.404775+0000","last_unstale":"2023-10-16T15:53:43.421686+0000","last_undegraded":"2023-10-16T15:53:43.421686+0000","last_fullsized":"2023-10-16T15:53:43.421686+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384001+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:41.384001+0000","last_peered":"2023-10-16T15:53:41.384001+0000","last_clean":"2023-10-16T15:53:41.384001+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:41.384001+0000","last_undegraded":"2023-10-16T15:53:41.384001+0000","last_fullsized":"2023-10-16T15:53:41.384001+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378030+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:41.378030+0000","last_peered":"2023-10-16T15:53:41.378030+0000","last_clean":"2023-10-16T15:53:41.378030+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:41.378030+0000","last_undegraded":"2023-10-16T15:53:41.378030+0000","last_fullsized":"2023-10-16T15:53:41.378030+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.420446+0000","last_change":"2023-10-16T15:53:40.382867+0000","last_active":"2023-10-16T15:53:41.420446+0000","last_peered":"2023-10-16T15:53:41.420446+0000","last_clean":"2023-10-16T15:53:41.420446+0000","last_became_active":"2023-10-16T15:53:40.378850+0000","last_became_peered":"2023-10-16T15:53:40.378850+0000","last_unstale":"2023-10-16T15:53:41.420446+0000","last_undegraded":"2023-10-16T15:53:41.420446+0000","last_fullsized":"2023-10-16T15:53:41.420446+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:00.525042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013915+0000","last_change":"2023-10-16T15:53:41.405768+0000","last_active":"2023-10-16T15:53:43.013915+0000","last_peered":"2023-10-16T15:53:43.013915+0000","last_clean":"2023-10-16T15:53:43.013915+0000","last_became_active":"2023-10-16T15:53:41.405230+0000","last_became_peered":"2023-10-16T15:53:41.405230+0000","last_unstale":"2023-10-16T15:53:43.013915+0000","last_undegraded":"2023-10-16T15:53:43.013915+0000","last_fullsized":"2023-10-16T15:53:43.013915+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:43.313902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378254+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:41.378254+0000","last_peered":"2023-10-16T15:53:41.378254+0000","last_clean":"2023-10-16T15:53:41.378254+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:41.378254+0000","last_undegraded":"2023-10-16T15:53:41.378254+0000","last_fullsized":"2023-10-16T15:53:41.378254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394353+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:41.394353+0000","last_peered":"2023-10-16T15:53:41.394353+0000","last_clean":"2023-10-16T15:53:41.394353+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:41.394353+0000","last_undegraded":"2023-10-16T15:53:41.394353+0000","last_fullsized":"2023-10-16T15:53:41.394353+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.404498+0000","last_change":"2023-10-16T15:53:40.381325+0000","last_active":"2023-10-16T15:53:41.404498+0000","last_peered":"2023-10-16T15:53:41.404498+0000","last_clean":"2023-10-16T15:53:41.404498+0000","last_became_active":"2023-10-16T15:53:40.380822+0000","last_became_peered":"2023-10-16T15:53:40.380822+0000","last_unstale":"2023-10-16T15:53:41.404498+0000","last_undegraded":"2023-10-16T15:53:41.404498+0000","last_fullsized":"2023-10-16T15:53:41.404498+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406853+0000","last_change":"2023-10-16T15:53:41.405116+0000","last_active":"2023-10-16T15:53:42.406853+0000","last_peered":"2023-10-16T15:53:42.406853+0000","last_clean":"2023-10-16T15:53:42.406853+0000","last_became_active":"2023-10-16T15:53:41.403778+0000","last_became_peered":"2023-10-16T15:53:41.403778+0000","last_unstale":"2023-10-16T15:53:42.406853+0000","last_undegraded":"2023-10-16T15:53:42.406853+0000","last_fullsized":"2023-10-16T15:53:42.406853+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385254+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:41.385254+0000","last_peered":"2023-10-16T15:53:41.385254+0000","last_clean":"2023-10-16T15:53:41.385254+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:41.385254+0000","last_undegraded":"2023-10-16T15:53:41.385254+0000","last_fullsized":"2023-10-16T15:53:41.385254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374210+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:41.374210+0000","last_peered":"2023-10-16T15:53:41.374210+0000","last_clean":"2023-10-16T15:53:41.374210+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:41.374210+0000","last_undegraded":"2023-10-16T15:53:41.374210+0000","last_fullsized":"2023-10-16T15:53:41.374210+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.409948+0000","last_change":"2023-10-16T15:53:40.377357+0000","last_active":"2023-10-16T15:53:41.409948+0000","last_peered":"2023-10-16T15:53:41.409948+0000","last_clean":"2023-10-16T15:53:41.409948+0000","last_became_active":"2023-10-16T15:53:40.377070+0000","last_became_peered":"2023-10-16T15:53:40.377070+0000","last_unstale":"2023-10-16T15:53:41.409948+0000","last_undegraded":"2023-10-16T15:53:41.409948+0000","last_fullsized":"2023-10-16T15:53:41.409948+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405599+0000","last_change":"2023-10-16T15:53:41.407410+0000","last_active":"2023-10-16T15:53:42.405599+0000","last_peered":"2023-10-16T15:53:42.405599+0000","last_clean":"2023-10-16T15:53:42.405599+0000","last_became_active":"2023-10-16T15:53:41.407034+0000","last_became_peered":"2023-10-16T15:53:41.407034+0000","last_unstale":"2023-10-16T15:53:42.405599+0000","last_undegraded":"2023-10-16T15:53:42.405599+0000","last_fullsized":"2023-10-16T15:53:42.405599+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:31.173108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392695+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:41.392695+0000","last_peered":"2023-10-16T15:53:41.392695+0000","last_clean":"2023-10-16T15:53:41.392695+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:41.392695+0000","last_undegraded":"2023-10-16T15:53:41.392695+0000","last_fullsized":"2023-10-16T15:53:41.392695+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.398663+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:41.398663+0000","last_peered":"2023-10-16T15:53:41.398663+0000","last_clean":"2023-10-16T15:53:41.398663+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:41.398663+0000","last_undegraded":"2023-10-16T15:53:41.398663+0000","last_fullsized":"2023-10-16T15:53:41.398663+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391139+0000","last_change":"2023-10-16T15:53:40.380015+0000","last_active":"2023-10-16T15:53:41.391139+0000","last_peered":"2023-10-16T15:53:41.391139+0000","last_clean":"2023-10-16T15:53:41.391139+0000","last_became_active":"2023-10-16T15:53:40.379716+0000","last_became_peered":"2023-10-16T15:53:40.379716+0000","last_unstale":"2023-10-16T15:53:41.391139+0000","last_undegraded":"2023-10-16T15:53:41.391139+0000","last_fullsized":"2023-10-16T15:53:41.391139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402124+0000","last_change":"2023-10-16T15:53:41.403677+0000","last_active":"2023-10-16T15:53:42.402124+0000","last_peered":"2023-10-16T15:53:42.402124+0000","last_clean":"2023-10-16T15:53:42.402124+0000","last_became_active":"2023-10-16T15:53:41.403372+0000","last_became_peered":"2023-10-16T15:53:41.403372+0000","last_unstale":"2023-10-16T15:53:42.402124+0000","last_undegraded":"2023-10-16T15:53:42.402124+0000","last_fullsized":"2023-10-16T15:53:42.402124+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396765+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:41.396765+0000","last_peered":"2023-10-16T15:53:41.396765+0000","last_clean":"2023-10-16T15:53:41.396765+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.396765+0000","last_undegraded":"2023-10-16T15:53:41.396765+0000","last_fullsized":"2023-10-16T15:53:41.396765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392453+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:41.392453+0000","last_peered":"2023-10-16T15:53:41.392453+0000","last_clean":"2023-10-16T15:53:41.392453+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:41.392453+0000","last_undegraded":"2023-10-16T15:53:41.392453+0000","last_fullsized":"2023-10-16T15:53:41.392453+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385863+0000","last_change":"2023-10-16T15:53:40.385668+0000","last_active":"2023-10-16T15:53:41.385863+0000","last_peered":"2023-10-16T15:53:41.385863+0000","last_clean":"2023-10-16T15:53:41.385863+0000","last_became_active":"2023-10-16T15:53:40.385360+0000","last_became_peered":"2023-10-16T15:53:40.385360+0000","last_unstale":"2023-10-16T15:53:41.385863+0000","last_undegraded":"2023-10-16T15:53:41.385863+0000","last_fullsized":"2023-10-16T15:53:41.385863+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:54.336040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330261+0000","last_change":"2023-10-16T15:53:41.397528+0000","last_active":"2023-10-16T15:53:43.330261+0000","last_peered":"2023-10-16T15:53:43.330261+0000","last_clean":"2023-10-16T15:53:43.330261+0000","last_became_active":"2023-10-16T15:53:41.397247+0000","last_became_peered":"2023-10-16T15:53:41.397247+0000","last_unstale":"2023-10-16T15:53:43.330261+0000","last_undegraded":"2023-10-16T15:53:43.330261+0000","last_fullsized":"2023-10-16T15:53:43.330261+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:21.337037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.373989+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:41.373989+0000","last_peered":"2023-10-16T15:53:41.373989+0000","last_clean":"2023-10-16T15:53:41.373989+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:41.373989+0000","last_undegraded":"2023-10-16T15:53:41.373989+0000","last_fullsized":"2023-10-16T15:53:41.373989+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410135+0000","last_change":"2023-10-16T15:53:41.408679+0000","last_active":"2023-10-16T15:53:42.410135+0000","last_peered":"2023-10-16T15:53:42.410135+0000","last_clean":"2023-10-16T15:53:42.410135+0000","last_became_active":"2023-10-16T15:53:41.407808+0000","last_became_peered":"2023-10-16T15:53:41.407808+0000","last_unstale":"2023-10-16T15:53:42.410135+0000","last_undegraded":"2023-10-16T15:53:42.410135+0000","last_fullsized":"2023-10-16T15:53:42.410135+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388628+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:41.388628+0000","last_peered":"2023-10-16T15:53:41.388628+0000","last_clean":"2023-10-16T15:53:41.388628+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:41.388628+0000","last_undegraded":"2023-10-16T15:53:41.388628+0000","last_fullsized":"2023-10-16T15:53:41.388628+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402941+0000","last_change":"2023-10-16T15:53:41.398094+0000","last_active":"2023-10-16T15:53:42.402941+0000","last_peered":"2023-10-16T15:53:42.402941+0000","last_clean":"2023-10-16T15:53:42.402941+0000","last_became_active":"2023-10-16T15:53:41.396310+0000","last_became_peered":"2023-10-16T15:53:41.396310+0000","last_unstale":"2023-10-16T15:53:42.402941+0000","last_undegraded":"2023-10-16T15:53:42.402941+0000","last_fullsized":"2023-10-16T15:53:42.402941+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389288+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:41.389288+0000","last_peered":"2023-10-16T15:53:41.389288+0000","last_clean":"2023-10-16T15:53:41.389288+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:41.389288+0000","last_undegraded":"2023-10-16T15:53:41.389288+0000","last_fullsized":"2023-10-16T15:53:41.389288+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408346+0000","last_change":"2023-10-16T15:53:41.409765+0000","last_active":"2023-10-16T15:53:42.408346+0000","last_peered":"2023-10-16T15:53:42.408346+0000","last_clean":"2023-10-16T15:53:42.408346+0000","last_became_active":"2023-10-16T15:53:41.409150+0000","last_became_peered":"2023-10-16T15:53:41.409150+0000","last_unstale":"2023-10-16T15:53:42.408346+0000","last_undegraded":"2023-10-16T15:53:42.408346+0000","last_fullsized":"2023-10-16T15:53:42.408346+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395859+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:41.395859+0000","last_peered":"2023-10-16T15:53:41.395859+0000","last_clean":"2023-10-16T15:53:41.395859+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:41.395859+0000","last_undegraded":"2023-10-16T15:53:41.395859+0000","last_fullsized":"2023-10-16T15:53:41.395859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406478+0000","last_change":"2023-10-16T15:53:41.405809+0000","last_active":"2023-10-16T15:53:42.406478+0000","last_peered":"2023-10-16T15:53:42.406478+0000","last_clean":"2023-10-16T15:53:42.406478+0000","last_became_active":"2023-10-16T15:53:41.405541+0000","last_became_peered":"2023-10-16T15:53:41.405541+0000","last_unstale":"2023-10-16T15:53:42.406478+0000","last_undegraded":"2023-10-16T15:53:42.406478+0000","last_fullsized":"2023-10-16T15:53:42.406478+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:39.653448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421402+0000","last_change":"2023-10-16T15:53:41.411322+0000","last_active":"2023-10-16T15:53:43.421402+0000","last_peered":"2023-10-16T15:53:43.421402+0000","last_clean":"2023-10-16T15:53:43.421402+0000","last_became_active":"2023-10-16T15:53:41.410176+0000","last_became_peered":"2023-10-16T15:53:41.410176+0000","last_unstale":"2023-10-16T15:53:43.421402+0000","last_undegraded":"2023-10-16T15:53:43.421402+0000","last_fullsized":"2023-10-16T15:53:43.421402+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393894+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:41.393894+0000","last_peered":"2023-10-16T15:53:41.393894+0000","last_clean":"2023-10-16T15:53:41.393894+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:41.393894+0000","last_undegraded":"2023-10-16T15:53:41.393894+0000","last_fullsized":"2023-10-16T15:53:41.393894+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418360+0000","last_change":"2023-10-16T15:53:41.400386+0000","last_active":"2023-10-16T15:53:43.418360+0000","last_peered":"2023-10-16T15:53:43.418360+0000","last_clean":"2023-10-16T15:53:43.418360+0000","last_became_active":"2023-10-16T15:53:41.400083+0000","last_became_peered":"2023-10-16T15:53:41.400083+0000","last_unstale":"2023-10-16T15:53:43.418360+0000","last_undegraded":"2023-10-16T15:53:43.418360+0000","last_fullsized":"2023-10-16T15:53:43.418360+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375718+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:41.375718+0000","last_peered":"2023-10-16T15:53:41.375718+0000","last_clean":"2023-10-16T15:53:41.375718+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:41.375718+0000","last_undegraded":"2023-10-16T15:53:41.375718+0000","last_fullsized":"2023-10-16T15:53:41.375718+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012816+0000","last_change":"2023-10-16T15:53:41.399893+0000","last_active":"2023-10-16T15:53:43.012816+0000","last_peered":"2023-10-16T15:53:43.012816+0000","last_clean":"2023-10-16T15:53:43.012816+0000","last_became_active":"2023-10-16T15:53:41.398053+0000","last_became_peered":"2023-10-16T15:53:41.398053+0000","last_unstale":"2023-10-16T15:53:43.012816+0000","last_undegraded":"2023-10-16T15:53:43.012816+0000","last_fullsized":"2023-10-16T15:53:43.012816+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:18.848509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393550+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:41.393550+0000","last_peered":"2023-10-16T15:53:41.393550+0000","last_clean":"2023-10-16T15:53:41.393550+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.393550+0000","last_undegraded":"2023-10-16T15:53:41.393550+0000","last_fullsized":"2023-10-16T15:53:41.393550+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404083+0000","last_change":"2023-10-16T15:53:41.406721+0000","last_active":"2023-10-16T15:53:42.404083+0000","last_peered":"2023-10-16T15:53:42.404083+0000","last_clean":"2023-10-16T15:53:42.404083+0000","last_became_active":"2023-10-16T15:53:41.404619+0000","last_became_peered":"2023-10-16T15:53:41.404619+0000","last_unstale":"2023-10-16T15:53:42.404083+0000","last_undegraded":"2023-10-16T15:53:42.404083+0000","last_fullsized":"2023-10-16T15:53:42.404083+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376431+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:41.376431+0000","last_peered":"2023-10-16T15:53:41.376431+0000","last_clean":"2023-10-16T15:53:41.376431+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:41.376431+0000","last_undegraded":"2023-10-16T15:53:41.376431+0000","last_fullsized":"2023-10-16T15:53:41.376431+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421928+0000","last_change":"2023-10-16T15:53:41.402971+0000","last_active":"2023-10-16T15:53:43.421928+0000","last_peered":"2023-10-16T15:53:43.421928+0000","last_clean":"2023-10-16T15:53:43.421928+0000","last_became_active":"2023-10-16T15:53:41.402710+0000","last_became_peered":"2023-10-16T15:53:41.402710+0000","last_unstale":"2023-10-16T15:53:43.421928+0000","last_undegraded":"2023-10-16T15:53:43.421928+0000","last_fullsized":"2023-10-16T15:53:43.421928+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389236+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:41.389236+0000","last_peered":"2023-10-16T15:53:41.389236+0000","last_clean":"2023-10-16T15:53:41.389236+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:41.389236+0000","last_undegraded":"2023-10-16T15:53:41.389236+0000","last_fullsized":"2023-10-16T15:53:41.389236+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402772+0000","last_change":"2023-10-16T15:53:41.399691+0000","last_active":"2023-10-16T15:53:42.402772+0000","last_peered":"2023-10-16T15:53:42.402772+0000","last_clean":"2023-10-16T15:53:42.402772+0000","last_became_active":"2023-10-16T15:53:41.397849+0000","last_became_peered":"2023-10-16T15:53:41.397849+0000","last_unstale":"2023-10-16T15:53:42.402772+0000","last_undegraded":"2023-10-16T15:53:42.402772+0000","last_fullsized":"2023-10-16T15:53:42.402772+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393175+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:41.393175+0000","last_peered":"2023-10-16T15:53:41.393175+0000","last_clean":"2023-10-16T15:53:41.393175+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:41.393175+0000","last_undegraded":"2023-10-16T15:53:41.393175+0000","last_fullsized":"2023-10-16T15:53:41.393175+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014730+0000","last_change":"2023-10-16T15:53:41.410559+0000","last_active":"2023-10-16T15:53:43.014730+0000","last_peered":"2023-10-16T15:53:43.014730+0000","last_clean":"2023-10-16T15:53:43.014730+0000","last_became_active":"2023-10-16T15:53:41.410258+0000","last_became_peered":"2023-10-16T15:53:41.410258+0000","last_unstale":"2023-10-16T15:53:43.014730+0000","last_undegraded":"2023-10-16T15:53:43.014730+0000","last_fullsized":"2023-10-16T15:53:43.014730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:25.813239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377897+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:41.377897+0000","last_peered":"2023-10-16T15:53:41.377897+0000","last_clean":"2023-10-16T15:53:41.377897+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:41.377897+0000","last_undegraded":"2023-10-16T15:53:41.377897+0000","last_fullsized":"2023-10-16T15:53:41.377897+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418981+0000","last_change":"2023-10-16T15:53:41.416192+0000","last_active":"2023-10-16T15:53:43.418981+0000","last_peered":"2023-10-16T15:53:43.418981+0000","last_clean":"2023-10-16T15:53:43.418981+0000","last_became_active":"2023-10-16T15:53:41.415862+0000","last_became_peered":"2023-10-16T15:53:41.415862+0000","last_unstale":"2023-10-16T15:53:43.418981+0000","last_undegraded":"2023-10-16T15:53:43.418981+0000","last_fullsized":"2023-10-16T15:53:43.418981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383281+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:41.383281+0000","last_peered":"2023-10-16T15:53:41.383281+0000","last_clean":"2023-10-16T15:53:41.383281+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:41.383281+0000","last_undegraded":"2023-10-16T15:53:41.383281+0000","last_fullsized":"2023-10-16T15:53:41.383281+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013226+0000","last_change":"2023-10-16T15:53:41.409431+0000","last_active":"2023-10-16T15:53:43.013226+0000","last_peered":"2023-10-16T15:53:43.013226+0000","last_clean":"2023-10-16T15:53:43.013226+0000","last_became_active":"2023-10-16T15:53:41.409023+0000","last_became_peered":"2023-10-16T15:53:41.409023+0000","last_unstale":"2023-10-16T15:53:43.013226+0000","last_undegraded":"2023-10-16T15:53:43.013226+0000","last_fullsized":"2023-10-16T15:53:43.013226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:31.127234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394621+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:41.394621+0000","last_peered":"2023-10-16T15:53:41.394621+0000","last_clean":"2023-10-16T15:53:41.394621+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:41.394621+0000","last_undegraded":"2023-10-16T15:53:41.394621+0000","last_fullsized":"2023-10-16T15:53:41.394621+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012572+0000","last_change":"2023-10-16T15:53:41.397505+0000","last_active":"2023-10-16T15:53:43.012572+0000","last_peered":"2023-10-16T15:53:43.012572+0000","last_clean":"2023-10-16T15:53:43.012572+0000","last_became_active":"2023-10-16T15:53:41.397198+0000","last_became_peered":"2023-10-16T15:53:41.397198+0000","last_unstale":"2023-10-16T15:53:43.012572+0000","last_undegraded":"2023-10-16T15:53:43.012572+0000","last_fullsized":"2023-10-16T15:53:43.012572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:07.665494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394898+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:41.394898+0000","last_peered":"2023-10-16T15:53:41.394898+0000","last_clean":"2023-10-16T15:53:41.394898+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:41.394898+0000","last_undegraded":"2023-10-16T15:53:41.394898+0000","last_fullsized":"2023-10-16T15:53:41.394898+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406566+0000","last_change":"2023-10-16T15:53:41.388087+0000","last_active":"2023-10-16T15:53:42.406566+0000","last_peered":"2023-10-16T15:53:42.406566+0000","last_clean":"2023-10-16T15:53:42.406566+0000","last_became_active":"2023-10-16T15:53:41.387820+0000","last_became_peered":"2023-10-16T15:53:41.387820+0000","last_unstale":"2023-10-16T15:53:42.406566+0000","last_undegraded":"2023-10-16T15:53:42.406566+0000","last_fullsized":"2023-10-16T15:53:42.406566+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374811+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:41.374811+0000","last_peered":"2023-10-16T15:53:41.374811+0000","last_clean":"2023-10-16T15:53:41.374811+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:41.374811+0000","last_undegraded":"2023-10-16T15:53:41.374811+0000","last_fullsized":"2023-10-16T15:53:41.374811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404049+0000","last_change":"2023-10-16T15:53:41.397057+0000","last_active":"2023-10-16T15:53:42.404049+0000","last_peered":"2023-10-16T15:53:42.404049+0000","last_clean":"2023-10-16T15:53:42.404049+0000","last_became_active":"2023-10-16T15:53:41.396163+0000","last_became_peered":"2023-10-16T15:53:41.396163+0000","last_unstale":"2023-10-16T15:53:42.404049+0000","last_undegraded":"2023-10-16T15:53:42.404049+0000","last_fullsized":"2023-10-16T15:53:42.404049+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391109+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:41.391109+0000","last_peered":"2023-10-16T15:53:41.391109+0000","last_clean":"2023-10-16T15:53:41.391109+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:41.391109+0000","last_undegraded":"2023-10-16T15:53:41.391109+0000","last_fullsized":"2023-10-16T15:53:41.391109+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410984+0000","last_change":"2023-10-16T15:53:41.410981+0000","last_active":"2023-10-16T15:53:42.410984+0000","last_peered":"2023-10-16T15:53:42.410984+0000","last_clean":"2023-10-16T15:53:42.410984+0000","last_became_active":"2023-10-16T15:53:41.410713+0000","last_became_peered":"2023-10-16T15:53:41.410713+0000","last_unstale":"2023-10-16T15:53:42.410984+0000","last_undegraded":"2023-10-16T15:53:42.410984+0000","last_fullsized":"2023-10-16T15:53:42.410984+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.539110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374479+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:41.374479+0000","last_peered":"2023-10-16T15:53:41.374479+0000","last_clean":"2023-10-16T15:53:41.374479+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:41.374479+0000","last_undegraded":"2023-10-16T15:53:41.374479+0000","last_fullsized":"2023-10-16T15:53:41.374479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405376+0000","last_change":"2023-10-16T15:53:41.413215+0000","last_active":"2023-10-16T15:53:42.405376+0000","last_peered":"2023-10-16T15:53:42.405376+0000","last_clean":"2023-10-16T15:53:42.405376+0000","last_became_active":"2023-10-16T15:53:41.412398+0000","last_became_peered":"2023-10-16T15:53:41.412398+0000","last_unstale":"2023-10-16T15:53:42.405376+0000","last_undegraded":"2023-10-16T15:53:42.405376+0000","last_fullsized":"2023-10-16T15:53:42.405376+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374715+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:41.374715+0000","last_peered":"2023-10-16T15:53:41.374715+0000","last_clean":"2023-10-16T15:53:41.374715+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:41.374715+0000","last_undegraded":"2023-10-16T15:53:41.374715+0000","last_fullsized":"2023-10-16T15:53:41.374715+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407630+0000","last_change":"2023-10-16T15:53:41.409487+0000","last_active":"2023-10-16T15:53:42.407630+0000","last_peered":"2023-10-16T15:53:42.407630+0000","last_clean":"2023-10-16T15:53:42.407630+0000","last_became_active":"2023-10-16T15:53:41.409160+0000","last_became_peered":"2023-10-16T15:53:41.409160+0000","last_unstale":"2023-10-16T15:53:42.407630+0000","last_undegraded":"2023-10-16T15:53:42.407630+0000","last_fullsized":"2023-10-16T15:53:42.407630+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396559+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:41.396559+0000","last_peered":"2023-10-16T15:53:41.396559+0000","last_clean":"2023-10-16T15:53:41.396559+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:41.396559+0000","last_undegraded":"2023-10-16T15:53:41.396559+0000","last_fullsized":"2023-10-16T15:53:41.396559+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.420943+0000","last_change":"2023-10-16T15:53:41.393762+0000","last_active":"2023-10-16T15:53:43.420943+0000","last_peered":"2023-10-16T15:53:43.420943+0000","last_clean":"2023-10-16T15:53:43.420943+0000","last_became_active":"2023-10-16T15:53:41.393255+0000","last_became_peered":"2023-10-16T15:53:41.393255+0000","last_unstale":"2023-10-16T15:53:43.420943+0000","last_undegraded":"2023-10-16T15:53:43.420943+0000","last_fullsized":"2023-10-16T15:53:43.420943+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.400191+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:41.400191+0000","last_peered":"2023-10-16T15:53:41.400191+0000","last_clean":"2023-10-16T15:53:41.400191+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:41.400191+0000","last_undegraded":"2023-10-16T15:53:41.400191+0000","last_fullsized":"2023-10-16T15:53:41.400191+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405352+0000","last_change":"2023-10-16T15:53:41.416464+0000","last_active":"2023-10-16T15:53:42.405352+0000","last_peered":"2023-10-16T15:53:42.405352+0000","last_clean":"2023-10-16T15:53:42.405352+0000","last_became_active":"2023-10-16T15:53:41.409828+0000","last_became_peered":"2023-10-16T15:53:41.409828+0000","last_unstale":"2023-10-16T15:53:42.405352+0000","last_undegraded":"2023-10-16T15:53:42.405352+0000","last_fullsized":"2023-10-16T15:53:42.405352+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385167+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:41.385167+0000","last_peered":"2023-10-16T15:53:41.385167+0000","last_clean":"2023-10-16T15:53:41.385167+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:41.385167+0000","last_undegraded":"2023-10-16T15:53:41.385167+0000","last_fullsized":"2023-10-16T15:53:41.385167+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405989+0000","last_change":"2023-10-16T15:53:41.400670+0000","last_active":"2023-10-16T15:53:42.405989+0000","last_peered":"2023-10-16T15:53:42.405989+0000","last_clean":"2023-10-16T15:53:42.405989+0000","last_became_active":"2023-10-16T15:53:41.400310+0000","last_became_peered":"2023-10-16T15:53:41.400310+0000","last_unstale":"2023-10-16T15:53:42.405989+0000","last_undegraded":"2023-10-16T15:53:42.405989+0000","last_fullsized":"2023-10-16T15:53:42.405989+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:55.180266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394811+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:41.394811+0000","last_peered":"2023-10-16T15:53:41.394811+0000","last_clean":"2023-10-16T15:53:41.394811+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:41.394811+0000","last_undegraded":"2023-10-16T15:53:41.394811+0000","last_fullsized":"2023-10-16T15:53:41.394811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418006+0000","last_change":"2023-10-16T15:53:41.406425+0000","last_active":"2023-10-16T15:53:43.418006+0000","last_peered":"2023-10-16T15:53:43.418006+0000","last_clean":"2023-10-16T15:53:43.418006+0000","last_became_active":"2023-10-16T15:53:41.405266+0000","last_became_peered":"2023-10-16T15:53:41.405266+0000","last_unstale":"2023-10-16T15:53:43.418006+0000","last_undegraded":"2023-10-16T15:53:43.418006+0000","last_fullsized":"2023-10-16T15:53:43.418006+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:36.594777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393796+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:41.393796+0000","last_peered":"2023-10-16T15:53:41.393796+0000","last_clean":"2023-10-16T15:53:41.393796+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:41.393796+0000","last_undegraded":"2023-10-16T15:53:41.393796+0000","last_fullsized":"2023-10-16T15:53:41.393796+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.413065+0000","last_change":"2023-10-16T15:53:41.410135+0000","last_active":"2023-10-16T15:53:42.413065+0000","last_peered":"2023-10-16T15:53:42.413065+0000","last_clean":"2023-10-16T15:53:42.413065+0000","last_became_active":"2023-10-16T15:53:41.409626+0000","last_became_peered":"2023-10-16T15:53:41.409626+0000","last_unstale":"2023-10-16T15:53:42.413065+0000","last_undegraded":"2023-10-16T15:53:42.413065+0000","last_fullsized":"2023-10-16T15:53:42.413065+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.484536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379861+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:41.379861+0000","last_peered":"2023-10-16T15:53:41.379861+0000","last_clean":"2023-10-16T15:53:41.379861+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:41.379861+0000","last_undegraded":"2023-10-16T15:53:41.379861+0000","last_fullsized":"2023-10-16T15:53:41.379861+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409800+0000","last_change":"2023-10-16T15:53:41.408039+0000","last_active":"2023-10-16T15:53:42.409800+0000","last_peered":"2023-10-16T15:53:42.409800+0000","last_clean":"2023-10-16T15:53:42.409800+0000","last_became_active":"2023-10-16T15:53:41.407756+0000","last_became_peered":"2023-10-16T15:53:41.407756+0000","last_unstale":"2023-10-16T15:53:42.409800+0000","last_undegraded":"2023-10-16T15:53:42.409800+0000","last_fullsized":"2023-10-16T15:53:42.409800+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374213+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:41.374213+0000","last_peered":"2023-10-16T15:53:41.374213+0000","last_clean":"2023-10-16T15:53:41.374213+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:41.374213+0000","last_undegraded":"2023-10-16T15:53:41.374213+0000","last_fullsized":"2023-10-16T15:53:41.374213+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013669+0000","last_change":"2023-10-16T15:53:41.411866+0000","last_active":"2023-10-16T15:53:43.013669+0000","last_peered":"2023-10-16T15:53:43.013669+0000","last_clean":"2023-10-16T15:53:43.013669+0000","last_became_active":"2023-10-16T15:53:41.411569+0000","last_became_peered":"2023-10-16T15:53:41.411569+0000","last_unstale":"2023-10-16T15:53:43.013669+0000","last_undegraded":"2023-10-16T15:53:43.013669+0000","last_fullsized":"2023-10-16T15:53:43.013669+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:01.576399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383777+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:41.383777+0000","last_peered":"2023-10-16T15:53:41.383777+0000","last_clean":"2023-10-16T15:53:41.383777+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:41.383777+0000","last_undegraded":"2023-10-16T15:53:41.383777+0000","last_fullsized":"2023-10-16T15:53:41.383777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409957+0000","last_change":"2023-10-16T15:53:41.399860+0000","last_active":"2023-10-16T15:53:42.409957+0000","last_peered":"2023-10-16T15:53:42.409957+0000","last_clean":"2023-10-16T15:53:42.409957+0000","last_became_active":"2023-10-16T15:53:41.399589+0000","last_became_peered":"2023-10-16T15:53:41.399589+0000","last_unstale":"2023-10-16T15:53:42.409957+0000","last_undegraded":"2023-10-16T15:53:42.409957+0000","last_fullsized":"2023-10-16T15:53:42.409957+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385445+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:41.385445+0000","last_peered":"2023-10-16T15:53:41.385445+0000","last_clean":"2023-10-16T15:53:41.385445+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:41.385445+0000","last_undegraded":"2023-10-16T15:53:41.385445+0000","last_fullsized":"2023-10-16T15:53:41.385445+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402749+0000","last_change":"2023-10-16T15:53:41.399919+0000","last_active":"2023-10-16T15:53:42.402749+0000","last_peered":"2023-10-16T15:53:42.402749+0000","last_clean":"2023-10-16T15:53:42.402749+0000","last_became_active":"2023-10-16T15:53:41.397770+0000","last_became_peered":"2023-10-16T15:53:41.397770+0000","last_unstale":"2023-10-16T15:53:42.402749+0000","last_undegraded":"2023-10-16T15:53:42.402749+0000","last_fullsized":"2023-10-16T15:53:42.402749+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377733+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:41.377733+0000","last_peered":"2023-10-16T15:53:41.377733+0000","last_clean":"2023-10-16T15:53:41.377733+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:41.377733+0000","last_undegraded":"2023-10-16T15:53:41.377733+0000","last_fullsized":"2023-10-16T15:53:41.377733+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014330+0000","last_change":"2023-10-16T15:53:41.401459+0000","last_active":"2023-10-16T15:53:43.014330+0000","last_peered":"2023-10-16T15:53:43.014330+0000","last_clean":"2023-10-16T15:53:43.014330+0000","last_became_active":"2023-10-16T15:53:41.401147+0000","last_became_peered":"2023-10-16T15:53:41.401147+0000","last_unstale":"2023-10-16T15:53:43.014330+0000","last_undegraded":"2023-10-16T15:53:43.014330+0000","last_fullsized":"2023-10-16T15:53:43.014330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:31.244497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.329048+0000","last_change":"2023-10-16T15:53:41.404413+0000","last_active":"2023-10-16T15:53:43.329048+0000","last_peered":"2023-10-16T15:53:43.329048+0000","last_clean":"2023-10-16T15:53:43.329048+0000","last_became_active":"2023-10-16T15:53:41.403862+0000","last_became_peered":"2023-10-16T15:53:41.403862+0000","last_unstale":"2023-10-16T15:53:43.329048+0000","last_undegraded":"2023-10-16T15:53:43.329048+0000","last_fullsized":"2023-10-16T15:53:43.329048+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:34.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392361+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:41.392361+0000","last_peered":"2023-10-16T15:53:41.392361+0000","last_clean":"2023-10-16T15:53:41.392361+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:41.392361+0000","last_undegraded":"2023-10-16T15:53:41.392361+0000","last_fullsized":"2023-10-16T15:53:41.392361+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407480+0000","last_change":"2023-10-16T15:53:41.406217+0000","last_active":"2023-10-16T15:53:42.407480+0000","last_peered":"2023-10-16T15:53:42.407480+0000","last_clean":"2023-10-16T15:53:42.407480+0000","last_became_active":"2023-10-16T15:53:41.405870+0000","last_became_peered":"2023-10-16T15:53:41.405870+0000","last_unstale":"2023-10-16T15:53:42.407480+0000","last_undegraded":"2023-10-16T15:53:42.407480+0000","last_fullsized":"2023-10-16T15:53:42.407480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374717+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:41.374717+0000","last_peered":"2023-10-16T15:53:41.374717+0000","last_clean":"2023-10-16T15:53:41.374717+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:41.374717+0000","last_undegraded":"2023-10-16T15:53:41.374717+0000","last_fullsized":"2023-10-16T15:53:41.374717+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576006,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":3400,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":3481600,"data_stored":2005127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":0.755},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":0.78000000000000003}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":0.80200000000000005}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.86699999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":0.82299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1412470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.216},"min":{"1min":0.73699999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7709999999999999,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.117},{"interface":"front","average":{"1min":1.22,"5min":1.276,"15min":1.2609999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.5489999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2949999999999999,"15min":1.2929999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.72199999999999998},"max":{"1min":1.8740000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3420000000000001,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.579,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.353}]},{"osd":2,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.244,"15min":1.2450000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.752},"max":{"1min":1.554,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.431,"15min":1.401},"min":{"1min":1.0680000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.1990000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.88100000000000001,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.46},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.331,"15min":1.339},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.631,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3560000000000001,"15min":1.331},"min":{"1min":0.871,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.325},{"interface":"front","average":{"1min":1.444,"5min":1.4430000000000001,"15min":1.407},"min":{"1min":0.90100000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.6319999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":0.88300000000000001,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.792,"5min":1.9419999999999999,"15min":2.016},"last":1.022},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5640000000000001,"15min":1.5289999999999999},"min":{"1min":1.0309999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.393}]},{"osd":7,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.633,"15min":1.6060000000000001},"min":{"1min":1.1950000000000001,"5min":0.627,"15min":0.627},"max":{"1min":2.149,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.6220000000000001,"15min":1.5980000000000001},"min":{"1min":1.264,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.2849999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29832,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342008,"statfs":{"total":96636764160,"available":96606216192,"internally_reserved":0,"allocated":2875392,"data_stored":1412414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1240000000000001,"15min":1.163},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6850000000000001,"5min":1.8069999999999999,"15min":1.821},"last":1.256},{"interface":"front","average":{"1min":1.026,"5min":1.2689999999999999,"15min":1.2430000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.53,"5min":2.738,"15min":2.738},"last":1.53}]},{"osd":1,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7669999999999999,"5min":1.7849999999999999,"15min":1.8680000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2629999999999999,"15min":1.302},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7270000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":0.86599999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.238,"15min":1.2529999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.577,"5min":1.784,"15min":1.9219999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.399,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7190000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.3260000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.343,"15min":1.3400000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9039999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.409},{"interface":"front","average":{"1min":1.169,"5min":1.3400000000000001,"15min":1.3799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.7130000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.1559999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.367,"15min":1.347},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.706},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.5069999999999999,"15min":1.5189999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.804,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.4810000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.385},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8660000000000001,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.669},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":1.431},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0230000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.585}]},{"osd":7,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.538,"15min":1.5369999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.847,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.639,"15min":1.633},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.962,"5min":3.109,"15min":3.109},"last":1.7430000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576006,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2895872,"data_stored":1415375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2330000000000001,"15min":1.212},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.52700000000000002},"max":{"1min":1.794,"5min":1.883,"15min":1.883},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.034}]},{"osd":1,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.95799999999999996,"5min":0.83499999999999996,"15min":0.629},"max":{"1min":1.675,"5min":1.929,"15min":2.2160000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.3160000000000001},"min":{"1min":0.92000000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0179999999999998,"5min":2.157,"15min":2.2989999999999999},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4490000000000001,"15min":1.4670000000000001},"min":{"1min":1.1160000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.498,"15min":1.4970000000000001},"min":{"1min":1.1870000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.3160000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":1.081,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.23,"5min":2.23,"15min":2.3650000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4099999999999999,"15min":1.401},"min":{"1min":1.0069999999999999,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.2310000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.375,"15min":1.3420000000000001},"min":{"1min":1.03,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.911,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.556},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.46,"15min":1.4370000000000001},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.161,"5min":2.161,"15min":2.4489999999999998},"last":1.177}]},{"osd":6,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":0.97499999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.554,"15min":1.5269999999999999},"min":{"1min":1.0089999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.2789999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.6850000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.617,"15min":1.5860000000000001},"min":{"1min":1.2669999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.415},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.6339999999999999,"15min":1.6040000000000001},"min":{"1min":1.3999999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.242,"5min":2.2599999999999998,"15min":2.347},"last":1.589}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56228,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315612,"statfs":{"total":96636764160,"available":96579186688,"internally_reserved":0,"allocated":2899968,"data_stored":1413696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.44}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":0.92700000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":0.94399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1413500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":0.621}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.772}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":1.335},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.407}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.8049999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":2.093}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.8500000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.9930000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576006,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47884,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323956,"statfs":{"total":96636764160,"available":96587730944,"internally_reserved":0,"allocated":2883584,"data_stored":1411978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.236}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.4470000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":1.679},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.502}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.919}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":1.9810000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":1.867}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48468,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323372,"statfs":{"total":96636764160,"available":96587132928,"internally_reserved":0,"allocated":3477504,"data_stored":2002962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.85199999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":1.1850000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":1.498}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.288},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4650000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.363},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":1.552}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:47.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-16T15:53:47.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.327+0000 7f87da0d7640 1 Processor -- start 2023-10-16T15:53:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.327+0000 7f87da0d7640 1 -- start start 2023-10-16T15:53:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.328+0000 7f87da0d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87d406f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.328+0000 7f87da0d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d40cb1d0 con 0x7f87d415b1c0 2023-10-16T15:53:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.328+0000 7f87da0d7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d40d06c0 con 0x7f87d40cd9a0 2023-10-16T15:53:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.328+0000 7f87da0d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d40ce4b0 con 0x7f87d40af9b0 2023-10-16T15:53:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.328+0000 7f87d90d5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f87d415b1c0 0x7f87d40cd290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56024/0 (socket says 172.21.15.29:56024) 2023-10-16T15:53:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.328+0000 7f87d90d5640 1 -- 172.21.15.29:0/3993738857 learned_addr learned my addr 172.21.15.29:0/3993738857 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87d98d6640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87d406f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389529037 0 0) 0x7f87d0001110 con 0x7f87d415b1c0 2023-10-16T15:53:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c0003520 con 0x7f87d415b1c0 2023-10-16T15:53:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 400878841 0 0) 0x7f87c0003520 con 0x7f87d415b1c0 2023-10-16T15:53:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 msgr2=0x7f87d406f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87d406f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 >> v1:172.21.15.170:6789/0 conn(0x7f87d40cd9a0 legacy=0x7f87d415eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d40b6850 con 0x7f87d415b1c0 2023-10-16T15:53:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f87d0001800 con 0x7f87d415b1c0 2023-10-16T15:53:47.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 >> v1:172.21.15.29:6789/0 conn(0x7f87d415b1c0 legacy=0x7f87d40cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87c00039a0 0x7f87c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.329+0000 7f87cbfff640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87c0004d00 0x7f87c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c000ba80 con 0x7f87c0004d00 2023-10-16T15:53:47.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c000bbf0 con 0x7f87c00039a0 2023-10-16T15:53:47.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c000bdb0 con 0x7f87d40af9b0 2023-10-16T15:53:47.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d98d6640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87c0004d00 0x7f87c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d90d5640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87c00039a0 0x7f87c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d98d6640 1 -- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 msgr2=0x7f87c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d88d4640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d98d6640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d98d6640 1 -- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87c00039a0 msgr2=0x7f87c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d98d6640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87c00039a0 0x7f87c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d98d6640 1 -- 172.21.15.29:0/3993738857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0003520 con 0x7f87c0004d00 2023-10-16T15:53:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.330+0000 7f87d88d4640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d40af9b0 0x7f87c00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:47.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.331+0000 7f87d98d6640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87c0004d00 0x7f87c0005180 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009fc0 tx=0x7f87c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8c4dd8aa6656ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:47.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.332+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc02cc00 con 0x7f87c0004d00 2023-10-16T15:53:47.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.332+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc00f040 con 0x7f87c0004d00 2023-10-16T15:53:47.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.332+0000 7f87cbfff640 1 -- 172.21.15.29:0/3993738857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc034440 con 0x7f87c0004d00 2023-10-16T15:53:47.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.332+0000 7f87da0d7640 1 -- 172.21.15.29:0/3993738857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87c0004d00 msgr2=0x7f87c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.332+0000 7f87da0d7640 1 --2- 172.21.15.29:0/3993738857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87c0004d00 0x7f87c0005180 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009fc0 tx=0x7f87c00057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.332+0000 7f87d98d6640 1 -- 172.21.15.29:0/3993738857 reap_dead start 2023-10-16T15:53:47.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.333+0000 7f87da0d7640 1 -- 172.21.15.29:0/3993738857 shutdown_connections 2023-10-16T15:53:47.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.333+0000 7f87da0d7640 1 -- 172.21.15.29:0/3993738857 >> 172.21.15.29:0/3993738857 conn(0x7f87d40b23d0 msgr2=0x7f87d40b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:47.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.333+0000 7f87da0d7640 1 -- 172.21.15.29:0/3993738857 shutdown_connections 2023-10-16T15:53:47.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.333+0000 7f87da0d7640 1 -- 172.21.15.29:0/3993738857 wait complete. 2023-10-16T15:53:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.335+0000 7f87da0d7640 1 Processor -- start 2023-10-16T15:53:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.335+0000 7f87da0d7640 1 -- start start 2023-10-16T15:53:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87da0d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d4157cd0 0x7f87d415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87da0d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d41580b0 0x7f87d40b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87da0d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 0x7f87d41f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87da0d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d40b5270 con 0x7f87d40b4df0 2023-10-16T15:53:47.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87da0d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d415bc70 con 0x7f87d41580b0 2023-10-16T15:53:47.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87da0d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d41f35e0 con 0x7f87d4157cd0 2023-10-16T15:53:47.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87d98d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 0x7f87d41f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87d98d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 0x7f87d41f2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45872/0 (socket says 172.21.15.29:45872) 2023-10-16T15:53:47.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87d98d6640 1 -- 172.21.15.29:0/1062253195 learned_addr learned my addr 172.21.15.29:0/1062253195 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:47.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87d90d5640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d4157cd0 0x7f87d415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.336+0000 7f87d88d4640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d41580b0 0x7f87d40b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d98d6640 1 -- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d4157cd0 msgr2=0x7f87d415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d98d6640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d4157cd0 0x7f87d415c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d98d6640 1 -- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d41580b0 msgr2=0x7f87d40b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d98d6640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d41580b0 0x7f87d40b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d98d6640 1 -- 172.21.15.29:0/1062253195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d415bec0 con 0x7f87d40b4df0 2023-10-16T15:53:47.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d88d4640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d41580b0 0x7f87d40b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:47.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87d98d6640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 0x7f87d41f2870 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f87cc002230 tx=0x7f87cc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:47.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc02de00 con 0x7f87d40b4df0 2023-10-16T15:53:47.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc034ad0 con 0x7f87d40b4df0 2023-10-16T15:53:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc034e80 con 0x7f87d40b4df0 2023-10-16T15:53:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d415c1a0 con 0x7f87d40b4df0 2023-10-16T15:53:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.337+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d415cee0 con 0x7f87d40b4df0 2023-10-16T15:53:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.339+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d415e4b0 con 0x7f87d40b4df0 2023-10-16T15:53:47.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.340+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87cc03e440 con 0x7f87d40b4df0 2023-10-16T15:53:47.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.340+0000 7f87c9ffb640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f87b806af90 0x7f87b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.341+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f87cc0ab7a0 con 0x7f87d40b4df0 2023-10-16T15:53:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.341+0000 7f87d90d5640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f87b806af90 0x7f87b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.342+0000 7f87d90d5640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f87b806af90 0x7f87b806d450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87d0001270 tx=0x7f87d0009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.345+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87cc076500 con 0x7f87d40b4df0 2023-10-16T15:53:47.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.611+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f87d40b0070 con 0x7f87d40b4df0 2023-10-16T15:53:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.612+0000 7f87c9ffb640 1 -- 172.21.15.29:0/1062253195 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1124 (secure 0 0 0) 0x7f87cc079b20 con 0x7f87d40b4df0 2023-10-16T15:53:47.614 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:47.614 INFO:teuthology.orchestra.run.smithi029.stdout:{"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5862,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":229,"num_osds":8,"num_up_osds":8,"osd_up_since":1697465777,"num_in_osds":8,"osd_in_since":1697465768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":45,"data_bytes":596433,"bytes_used":390639616,"bytes_avail":772703473664,"bytes_total":773094113280,"write_bytes_sec":2344,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":114,"id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"b","status":"up:active","gid":8761},{"filesystem_id":13,"rank":0,"name":"c","status":"up:active","gid":10821}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2023-10-16T15:52:01.358085+0000","services":{}},"progress_events":{}} 2023-10-16T15:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.614+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f87b806af90 msgr2=0x7f87b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.614+0000 7f87da0d7640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f87b806af90 0x7f87b806d450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87d0001270 tx=0x7f87d0009040 comp rx=0 tx=0).stop 2023-10-16T15:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.614+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 msgr2=0x7f87d41f2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.614+0000 7f87da0d7640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 0x7f87d41f2870 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f87cc002230 tx=0x7f87cc009750 comp rx=0 tx=0).stop 2023-10-16T15:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 shutdown_connections 2023-10-16T15:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f87b806af90 0x7f87b806d450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d40b4df0 0x7f87d41f2870 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d41580b0 0x7f87d40b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 --2- 172.21.15.29:0/1062253195 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d4157cd0 0x7f87d415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 >> 172.21.15.29:0/1062253195 conn(0x7f87d40b23d0 msgr2=0x7f87d40b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:47.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 shutdown_connections 2023-10-16T15:53:47.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.615+0000 7f87da0d7640 1 -- 172.21.15.29:0/1062253195 wait complete. 2023-10-16T15:53:47.637 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-16T15:53:47.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:47.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.867+0000 7f72f759e640 1 Processor -- start 2023-10-16T15:53:47.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.867+0000 7f72f759e640 1 -- start start 2023-10-16T15:53:47.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.868+0000 7f72f759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72f815aaa0 0x7f72f80b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80cad00 con 0x7f72f80ca8a0 2023-10-16T15:53:47.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80b2d00 con 0x7f72f8070240 2023-10-16T15:53:47.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80b2fe0 con 0x7f72f815aaa0 2023-10-16T15:53:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72f815aaa0 0x7f72f80b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72f815aaa0 0x7f72f80b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45878/0 (socket says 172.21.15.29:45878) 2023-10-16T15:53:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f5d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f72f80ca8a0 0x7f72f80b3da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56038/0 (socket says 172.21.15.29:56038) 2023-10-16T15:53:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.869+0000 7f72f659c640 1 -- 172.21.15.29:0/4175685389 learned_addr learned my addr 172.21.15.29:0/4175685389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:47.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.870+0000 7f72f659c640 1 -- 172.21.15.29:0/4175685389 >> v1:172.21.15.170:6789/0 conn(0x7f72f8070240 legacy=0x7f72f80c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:47.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.870+0000 7f72f659c640 1 -- 172.21.15.29:0/4175685389 >> v1:172.21.15.29:6789/0 conn(0x7f72f80ca8a0 legacy=0x7f72f80b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.870+0000 7f72f659c640 1 -- 172.21.15.29:0/4175685389 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80b32c0 con 0x7f72f815aaa0 2023-10-16T15:53:47.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.870+0000 7f72f559a640 1 -- 172.21.15.29:0/4175685389 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152531653 0 0) 0x7f72f80cad00 con 0x7f72f80ca8a0 2023-10-16T15:53:47.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.871+0000 7f72f659c640 1 --2- 172.21.15.29:0/4175685389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80b2600 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009870 tx=0x7f72e802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c44e6b9cdfcf2f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:47.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f559a640 1 -- 172.21.15.29:0/4175685389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8031860 con 0x7f72f815aaa0 2023-10-16T15:53:47.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f559a640 1 -- 172.21.15.29:0/4175685389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e80319c0 con 0x7f72f815aaa0 2023-10-16T15:53:47.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f559a640 1 -- 172.21.15.29:0/4175685389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8031d50 con 0x7f72f815aaa0 2023-10-16T15:53:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f759e640 1 -- 172.21.15.29:0/4175685389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 msgr2=0x7f72f80b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f759e640 1 --2- 172.21.15.29:0/4175685389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80b2600 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009870 tx=0x7f72e802b900 comp rx=0 tx=0).stop 2023-10-16T15:53:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f759e640 1 -- 172.21.15.29:0/4175685389 shutdown_connections 2023-10-16T15:53:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f759e640 1 --2- 172.21.15.29:0/4175685389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80b2600 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.872+0000 7f72f759e640 1 -- 172.21.15.29:0/4175685389 >> 172.21.15.29:0/4175685389 conn(0x7f72f80cc230 msgr2=0x7f72f80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:47.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.873+0000 7f72f759e640 1 -- 172.21.15.29:0/4175685389 shutdown_connections 2023-10-16T15:53:47.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.873+0000 7f72f759e640 1 -- 172.21.15.29:0/4175685389 wait complete. 2023-10-16T15:53:47.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.875+0000 7f72f759e640 1 Processor -- start 2023-10-16T15:53:47.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.875+0000 7f72f759e640 1 -- start start 2023-10-16T15:53:47.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72f8070240 0x7f72f8158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72f80ca8a0 0x7f72f80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c92f0 con 0x7f72f815aaa0 2023-10-16T15:53:47.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c95d0 con 0x7f72f8070240 2023-10-16T15:53:47.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c98b0 con 0x7f72f80ca8a0 2023-10-16T15:53:47.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80c8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45892/0 (socket says 172.21.15.29:45892) 2023-10-16T15:53:47.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.876+0000 7f72f6d9d640 1 -- 172.21.15.29:0/1397480366 learned_addr learned my addr 172.21.15.29:0/1397480366 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:47.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f5d9b640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72f80ca8a0 0x7f72f80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f659c640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72f8070240 0x7f72f8158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f6d9d640 1 -- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72f80ca8a0 msgr2=0x7f72f80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f6d9d640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72f80ca8a0 0x7f72f80c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f6d9d640 1 -- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72f8070240 msgr2=0x7f72f8158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:47.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f6d9d640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72f8070240 0x7f72f8158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:47.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f6d9d640 1 -- 172.21.15.29:0/1397480366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80094d0 con 0x7f72f815aaa0 2023-10-16T15:53:47.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f659c640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72f8070240 0x7f72f8158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:47.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f5d9b640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72f80ca8a0 0x7f72f80c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:47.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f6d9d640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80c8a80 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f72f000b970 tx=0x7f72f000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:47.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f000cb40 con 0x7f72f815aaa0 2023-10-16T15:53:47.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f000cca0 con 0x7f72f815aaa0 2023-10-16T15:53:47.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0014680 con 0x7f72f815aaa0 2023-10-16T15:53:47.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.877+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f81f2690 con 0x7f72f815aaa0 2023-10-16T15:53:47.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.878+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f80c8fc0 con 0x7f72f815aaa0 2023-10-16T15:53:47.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.879+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f804e9b0 con 0x7f72f815aaa0 2023-10-16T15:53:47.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.880+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f72f001d3b0 con 0x7f72f815aaa0 2023-10-16T15:53:47.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.881+0000 7f72eeffd640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f72d806af90 0x7f72d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:47.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.881+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f72f0089950 con 0x7f72f815aaa0 2023-10-16T15:53:47.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.881+0000 7f72f659c640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f72d806af90 0x7f72d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:47.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.882+0000 7f72f659c640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f72d806af90 0x7f72d806d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72e8036000 tx=0x7f72e8008030 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:47.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:47.885+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f72f0054560 con 0x7f72f815aaa0 2023-10-16T15:53:48.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:53:48.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:53:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.082+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72f80b3880 con 0x7f72d806af90 2023-10-16T15:53:48.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.101+0000 7f72eeffd640 1 -- 172.21.15.29:0/1397480366 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526043 (secure 0 0 0) 0x7f72e8031bb0 con 0x7f72d806af90 2023-10-16T15:53:48.110 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:48.114 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:48.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f72d806af90 msgr2=0x7f72d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:48.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f72d806af90 0x7f72d806d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72e8036000 tx=0x7f72e8008030 comp rx=0 tx=0).stop 2023-10-16T15:53:48.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 msgr2=0x7f72f80c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:48.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80c8a80 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f72f000b970 tx=0x7f72f000be30 comp rx=0 tx=0).stop 2023-10-16T15:53:48.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 shutdown_connections 2023-10-16T15:53:48.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f72d806af90 0x7f72d806d450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:48.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f815aaa0 0x7f72f80c8a80 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:48.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f72f80ca8a0 0x7f72f80c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:48.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 --2- 172.21.15.29:0/1397480366 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72f8070240 0x7f72f8158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:48.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 >> 172.21.15.29:0/1397480366 conn(0x7f72f80cc230 msgr2=0x7f72f80b18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:48.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 shutdown_connections 2023-10-16T15:53:48.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:48.109+0000 7f72f759e640 1 -- 172.21.15.29:0/1397480366 wait complete. 2023-10-16T15:53:48.131 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-16T15:53:46.463642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":674,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381484,"kb_used_data":23720,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593236,"statfs":{"total":773094113280,"available":772703473664,"internally_reserved":0,"allocated":24289280,"data_stored":12487522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366202425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.114433"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411190+0000","last_change":"2023-10-16T15:53:41.399707+0000","last_active":"2023-10-16T15:53:42.411190+0000","last_peered":"2023-10-16T15:53:42.411190+0000","last_clean":"2023-10-16T15:53:42.411190+0000","last_became_active":"2023-10-16T15:53:41.399411+0000","last_became_peered":"2023-10-16T15:53:41.399411+0000","last_unstale":"2023-10-16T15:53:42.411190+0000","last_undegraded":"2023-10-16T15:53:42.411190+0000","last_fullsized":"2023-10-16T15:53:42.411190+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:45.368014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1251,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376480+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:41.376480+0000","last_peered":"2023-10-16T15:53:41.376480+0000","last_clean":"2023-10-16T15:53:41.376480+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:41.376480+0000","last_undegraded":"2023-10-16T15:53:41.376480+0000","last_fullsized":"2023-10-16T15:53:41.376480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396662+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:41.396662+0000","last_peered":"2023-10-16T15:53:41.396662+0000","last_clean":"2023-10-16T15:53:41.396662+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:41.396662+0000","last_undegraded":"2023-10-16T15:53:41.396662+0000","last_fullsized":"2023-10-16T15:53:41.396662+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375906+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:41.375906+0000","last_peered":"2023-10-16T15:53:41.375906+0000","last_clean":"2023-10-16T15:53:41.375906+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:41.375906+0000","last_undegraded":"2023-10-16T15:53:41.375906+0000","last_fullsized":"2023-10-16T15:53:41.375906+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.410846+0000","last_change":"2023-10-16T15:53:40.387170+0000","last_active":"2023-10-16T15:53:41.410846+0000","last_peered":"2023-10-16T15:53:41.410846+0000","last_clean":"2023-10-16T15:53:41.410846+0000","last_became_active":"2023-10-16T15:53:40.386872+0000","last_became_peered":"2023-10-16T15:53:40.386872+0000","last_unstale":"2023-10-16T15:53:41.410846+0000","last_undegraded":"2023-10-16T15:53:41.410846+0000","last_fullsized":"2023-10-16T15:53:41.410846+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411355+0000","last_change":"2023-10-16T15:53:41.413779+0000","last_active":"2023-10-16T15:53:42.411355+0000","last_peered":"2023-10-16T15:53:42.411355+0000","last_clean":"2023-10-16T15:53:42.411355+0000","last_became_active":"2023-10-16T15:53:41.413379+0000","last_became_peered":"2023-10-16T15:53:41.413379+0000","last_unstale":"2023-10-16T15:53:42.411355+0000","last_undegraded":"2023-10-16T15:53:42.411355+0000","last_fullsized":"2023-10-16T15:53:42.411355+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384903+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:41.384903+0000","last_peered":"2023-10-16T15:53:41.384903+0000","last_clean":"2023-10-16T15:53:41.384903+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:41.384903+0000","last_undegraded":"2023-10-16T15:53:41.384903+0000","last_fullsized":"2023-10-16T15:53:41.384903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386164+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:41.386164+0000","last_peered":"2023-10-16T15:53:41.386164+0000","last_clean":"2023-10-16T15:53:41.386164+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:41.386164+0000","last_undegraded":"2023-10-16T15:53:41.386164+0000","last_fullsized":"2023-10-16T15:53:41.386164+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378909+0000","last_change":"2023-10-16T15:53:40.378024+0000","last_active":"2023-10-16T15:53:41.378909+0000","last_peered":"2023-10-16T15:53:41.378909+0000","last_clean":"2023-10-16T15:53:41.378909+0000","last_became_active":"2023-10-16T15:53:40.377757+0000","last_became_peered":"2023-10-16T15:53:40.377757+0000","last_unstale":"2023-10-16T15:53:41.378909+0000","last_undegraded":"2023-10-16T15:53:41.378909+0000","last_fullsized":"2023-10-16T15:53:41.378909+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:48.714124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014672+0000","last_change":"2023-10-16T15:53:41.418345+0000","last_active":"2023-10-16T15:53:43.014672+0000","last_peered":"2023-10-16T15:53:43.014672+0000","last_clean":"2023-10-16T15:53:43.014672+0000","last_became_active":"2023-10-16T15:53:41.418078+0000","last_became_peered":"2023-10-16T15:53:41.418078+0000","last_unstale":"2023-10-16T15:53:43.014672+0000","last_undegraded":"2023-10-16T15:53:43.014672+0000","last_fullsized":"2023-10-16T15:53:43.014672+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:25.632327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389543+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:41.389543+0000","last_peered":"2023-10-16T15:53:41.389543+0000","last_clean":"2023-10-16T15:53:41.389543+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:41.389543+0000","last_undegraded":"2023-10-16T15:53:41.389543+0000","last_fullsized":"2023-10-16T15:53:41.389543+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380422+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:41.380422+0000","last_peered":"2023-10-16T15:53:41.380422+0000","last_clean":"2023-10-16T15:53:41.380422+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:41.380422+0000","last_undegraded":"2023-10-16T15:53:41.380422+0000","last_fullsized":"2023-10-16T15:53:41.380422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.405443+0000","last_change":"2023-10-16T15:53:40.384327+0000","last_active":"2023-10-16T15:53:41.405443+0000","last_peered":"2023-10-16T15:53:41.405443+0000","last_clean":"2023-10-16T15:53:41.405443+0000","last_became_active":"2023-10-16T15:53:40.384027+0000","last_became_peered":"2023-10-16T15:53:40.384027+0000","last_unstale":"2023-10-16T15:53:41.405443+0000","last_undegraded":"2023-10-16T15:53:41.405443+0000","last_fullsized":"2023-10-16T15:53:41.405443+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:09.776405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014626+0000","last_change":"2023-10-16T15:53:41.393836+0000","last_active":"2023-10-16T15:53:43.014626+0000","last_peered":"2023-10-16T15:53:43.014626+0000","last_clean":"2023-10-16T15:53:43.014626+0000","last_became_active":"2023-10-16T15:53:41.392583+0000","last_became_peered":"2023-10-16T15:53:41.392583+0000","last_unstale":"2023-10-16T15:53:43.014626+0000","last_undegraded":"2023-10-16T15:53:43.014626+0000","last_fullsized":"2023-10-16T15:53:43.014626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:22.777642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384099+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:41.384099+0000","last_peered":"2023-10-16T15:53:41.384099+0000","last_clean":"2023-10-16T15:53:41.384099+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:41.384099+0000","last_undegraded":"2023-10-16T15:53:41.384099+0000","last_fullsized":"2023-10-16T15:53:41.384099+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396904+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:41.396904+0000","last_peered":"2023-10-16T15:53:41.396904+0000","last_clean":"2023-10-16T15:53:41.396904+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:41.396904+0000","last_undegraded":"2023-10-16T15:53:41.396904+0000","last_fullsized":"2023-10-16T15:53:41.396904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395708+0000","last_change":"2023-10-16T15:53:40.378111+0000","last_active":"2023-10-16T15:53:41.395708+0000","last_peered":"2023-10-16T15:53:41.395708+0000","last_clean":"2023-10-16T15:53:41.395708+0000","last_became_active":"2023-10-16T15:53:40.377470+0000","last_became_peered":"2023-10-16T15:53:40.377470+0000","last_unstale":"2023-10-16T15:53:41.395708+0000","last_undegraded":"2023-10-16T15:53:41.395708+0000","last_fullsized":"2023-10-16T15:53:41.395708+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408902+0000","last_change":"2023-10-16T15:53:41.409618+0000","last_active":"2023-10-16T15:53:42.408902+0000","last_peered":"2023-10-16T15:53:42.408902+0000","last_clean":"2023-10-16T15:53:42.408902+0000","last_became_active":"2023-10-16T15:53:41.409365+0000","last_became_peered":"2023-10-16T15:53:41.409365+0000","last_unstale":"2023-10-16T15:53:42.408902+0000","last_undegraded":"2023-10-16T15:53:42.408902+0000","last_fullsized":"2023-10-16T15:53:42.408902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378147+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:41.378147+0000","last_peered":"2023-10-16T15:53:41.378147+0000","last_clean":"2023-10-16T15:53:41.378147+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:41.378147+0000","last_undegraded":"2023-10-16T15:53:41.378147+0000","last_fullsized":"2023-10-16T15:53:41.378147+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383981+0000","last_change":"2023-10-16T15:53:40.381668+0000","last_active":"2023-10-16T15:53:41.383981+0000","last_peered":"2023-10-16T15:53:41.383981+0000","last_clean":"2023-10-16T15:53:41.383981+0000","last_became_active":"2023-10-16T15:53:40.381369+0000","last_became_peered":"2023-10-16T15:53:40.381369+0000","last_unstale":"2023-10-16T15:53:41.383981+0000","last_undegraded":"2023-10-16T15:53:41.383981+0000","last_fullsized":"2023-10-16T15:53:41.383981+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:32.114344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410745+0000","last_change":"2023-10-16T15:53:41.400067+0000","last_active":"2023-10-16T15:53:42.410745+0000","last_peered":"2023-10-16T15:53:42.410745+0000","last_clean":"2023-10-16T15:53:42.410745+0000","last_became_active":"2023-10-16T15:53:41.399778+0000","last_became_peered":"2023-10-16T15:53:41.399778+0000","last_unstale":"2023-10-16T15:53:42.410745+0000","last_undegraded":"2023-10-16T15:53:42.410745+0000","last_fullsized":"2023-10-16T15:53:42.410745+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:18.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384461+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:41.384461+0000","last_peered":"2023-10-16T15:53:41.384461+0000","last_clean":"2023-10-16T15:53:41.384461+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:41.384461+0000","last_undegraded":"2023-10-16T15:53:41.384461+0000","last_fullsized":"2023-10-16T15:53:41.384461+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.397959+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:41.397959+0000","last_peered":"2023-10-16T15:53:41.397959+0000","last_clean":"2023-10-16T15:53:41.397959+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:41.397959+0000","last_undegraded":"2023-10-16T15:53:41.397959+0000","last_fullsized":"2023-10-16T15:53:41.397959+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374462+0000","last_change":"2023-10-16T15:53:40.375759+0000","last_active":"2023-10-16T15:53:41.374462+0000","last_peered":"2023-10-16T15:53:41.374462+0000","last_clean":"2023-10-16T15:53:41.374462+0000","last_became_active":"2023-10-16T15:53:40.375464+0000","last_became_peered":"2023-10-16T15:53:40.375464+0000","last_unstale":"2023-10-16T15:53:41.374462+0000","last_undegraded":"2023-10-16T15:53:41.374462+0000","last_fullsized":"2023-10-16T15:53:41.374462+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405804+0000","last_change":"2023-10-16T15:53:41.407972+0000","last_active":"2023-10-16T15:53:42.405804+0000","last_peered":"2023-10-16T15:53:42.405804+0000","last_clean":"2023-10-16T15:53:42.405804+0000","last_became_active":"2023-10-16T15:53:41.407725+0000","last_became_peered":"2023-10-16T15:53:41.407725+0000","last_unstale":"2023-10-16T15:53:42.405804+0000","last_undegraded":"2023-10-16T15:53:42.405804+0000","last_fullsized":"2023-10-16T15:53:42.405804+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:45.480076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385405+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:41.385405+0000","last_peered":"2023-10-16T15:53:41.385405+0000","last_clean":"2023-10-16T15:53:41.385405+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:41.385405+0000","last_undegraded":"2023-10-16T15:53:41.385405+0000","last_fullsized":"2023-10-16T15:53:41.385405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386135+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:41.386135+0000","last_peered":"2023-10-16T15:53:41.386135+0000","last_clean":"2023-10-16T15:53:41.386135+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:41.386135+0000","last_undegraded":"2023-10-16T15:53:41.386135+0000","last_fullsized":"2023-10-16T15:53:41.386135+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385992+0000","last_change":"2023-10-16T15:53:40.389163+0000","last_active":"2023-10-16T15:53:41.385992+0000","last_peered":"2023-10-16T15:53:41.385992+0000","last_clean":"2023-10-16T15:53:41.385992+0000","last_became_active":"2023-10-16T15:53:40.388144+0000","last_became_peered":"2023-10-16T15:53:40.388144+0000","last_unstale":"2023-10-16T15:53:41.385992+0000","last_undegraded":"2023-10-16T15:53:41.385992+0000","last_fullsized":"2023-10-16T15:53:41.385992+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406367+0000","last_change":"2023-10-16T15:53:41.408615+0000","last_active":"2023-10-16T15:53:42.406367+0000","last_peered":"2023-10-16T15:53:42.406367+0000","last_clean":"2023-10-16T15:53:42.406367+0000","last_became_active":"2023-10-16T15:53:41.408337+0000","last_became_peered":"2023-10-16T15:53:41.408337+0000","last_unstale":"2023-10-16T15:53:42.406367+0000","last_undegraded":"2023-10-16T15:53:42.406367+0000","last_fullsized":"2023-10-16T15:53:42.406367+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:20.232917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"225'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379246+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:41.379246+0000","last_peered":"2023-10-16T15:53:41.379246+0000","last_clean":"2023-10-16T15:53:41.379246+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:41.379246+0000","last_undegraded":"2023-10-16T15:53:41.379246+0000","last_fullsized":"2023-10-16T15:53:41.379246+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376019+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:41.376019+0000","last_peered":"2023-10-16T15:53:41.376019+0000","last_clean":"2023-10-16T15:53:41.376019+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:41.376019+0000","last_undegraded":"2023-10-16T15:53:41.376019+0000","last_fullsized":"2023-10-16T15:53:41.376019+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384628+0000","last_change":"2023-10-16T15:53:40.375284+0000","last_active":"2023-10-16T15:53:41.384628+0000","last_peered":"2023-10-16T15:53:41.384628+0000","last_clean":"2023-10-16T15:53:41.384628+0000","last_became_active":"2023-10-16T15:53:40.374509+0000","last_became_peered":"2023-10-16T15:53:40.374509+0000","last_unstale":"2023-10-16T15:53:41.384628+0000","last_undegraded":"2023-10-16T15:53:41.384628+0000","last_fullsized":"2023-10-16T15:53:41.384628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:59.597435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407164+0000","last_change":"2023-10-16T15:53:41.414757+0000","last_active":"2023-10-16T15:53:42.407164+0000","last_peered":"2023-10-16T15:53:42.407164+0000","last_clean":"2023-10-16T15:53:42.407164+0000","last_became_active":"2023-10-16T15:53:41.414480+0000","last_became_peered":"2023-10-16T15:53:41.414480+0000","last_unstale":"2023-10-16T15:53:42.407164+0000","last_undegraded":"2023-10-16T15:53:42.407164+0000","last_fullsized":"2023-10-16T15:53:42.407164+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387180+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:41.387180+0000","last_peered":"2023-10-16T15:53:41.387180+0000","last_clean":"2023-10-16T15:53:41.387180+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:41.387180+0000","last_undegraded":"2023-10-16T15:53:41.387180+0000","last_fullsized":"2023-10-16T15:53:41.387180+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395410+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:41.395410+0000","last_peered":"2023-10-16T15:53:41.395410+0000","last_clean":"2023-10-16T15:53:41.395410+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:41.395410+0000","last_undegraded":"2023-10-16T15:53:41.395410+0000","last_fullsized":"2023-10-16T15:53:41.395410+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382520+0000","last_change":"2023-10-16T15:53:40.381870+0000","last_active":"2023-10-16T15:53:41.382520+0000","last_peered":"2023-10-16T15:53:41.382520+0000","last_clean":"2023-10-16T15:53:41.382520+0000","last_became_active":"2023-10-16T15:53:40.381577+0000","last_became_peered":"2023-10-16T15:53:40.381577+0000","last_unstale":"2023-10-16T15:53:41.382520+0000","last_undegraded":"2023-10-16T15:53:41.382520+0000","last_fullsized":"2023-10-16T15:53:41.382520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:33.030599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404650+0000","last_change":"2023-10-16T15:53:41.408405+0000","last_active":"2023-10-16T15:53:42.404650+0000","last_peered":"2023-10-16T15:53:42.404650+0000","last_clean":"2023-10-16T15:53:42.404650+0000","last_became_active":"2023-10-16T15:53:41.407251+0000","last_became_peered":"2023-10-16T15:53:41.407251+0000","last_unstale":"2023-10-16T15:53:42.404650+0000","last_undegraded":"2023-10-16T15:53:42.404650+0000","last_fullsized":"2023-10-16T15:53:42.404650+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.381328+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:41.381328+0000","last_peered":"2023-10-16T15:53:41.381328+0000","last_clean":"2023-10-16T15:53:41.381328+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:41.381328+0000","last_undegraded":"2023-10-16T15:53:41.381328+0000","last_fullsized":"2023-10-16T15:53:41.381328+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389408+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:41.389408+0000","last_peered":"2023-10-16T15:53:41.389408+0000","last_clean":"2023-10-16T15:53:41.389408+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:41.389408+0000","last_undegraded":"2023-10-16T15:53:41.389408+0000","last_fullsized":"2023-10-16T15:53:41.389408+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414970+0000","last_change":"2023-10-16T15:53:40.373067+0000","last_active":"2023-10-16T15:53:41.414970+0000","last_peered":"2023-10-16T15:53:41.414970+0000","last_clean":"2023-10-16T15:53:41.414970+0000","last_became_active":"2023-10-16T15:53:40.372307+0000","last_became_peered":"2023-10-16T15:53:40.372307+0000","last_unstale":"2023-10-16T15:53:41.414970+0000","last_undegraded":"2023-10-16T15:53:41.414970+0000","last_fullsized":"2023-10-16T15:53:41.414970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330512+0000","last_change":"2023-10-16T15:53:41.400515+0000","last_active":"2023-10-16T15:53:43.330512+0000","last_peered":"2023-10-16T15:53:43.330512+0000","last_clean":"2023-10-16T15:53:43.330512+0000","last_became_active":"2023-10-16T15:53:41.399835+0000","last_became_peered":"2023-10-16T15:53:41.399835+0000","last_unstale":"2023-10-16T15:53:43.330512+0000","last_undegraded":"2023-10-16T15:53:43.330512+0000","last_fullsized":"2023-10-16T15:53:43.330512+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:40.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375870+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:41.375870+0000","last_peered":"2023-10-16T15:53:41.375870+0000","last_clean":"2023-10-16T15:53:41.375870+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:41.375870+0000","last_undegraded":"2023-10-16T15:53:41.375870+0000","last_fullsized":"2023-10-16T15:53:41.375870+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384670+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:41.384670+0000","last_peered":"2023-10-16T15:53:41.384670+0000","last_clean":"2023-10-16T15:53:41.384670+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:41.384670+0000","last_undegraded":"2023-10-16T15:53:41.384670+0000","last_fullsized":"2023-10-16T15:53:41.384670+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.407785+0000","last_change":"2023-10-16T15:53:40.378428+0000","last_active":"2023-10-16T15:53:41.407785+0000","last_peered":"2023-10-16T15:53:41.407785+0000","last_clean":"2023-10-16T15:53:41.407785+0000","last_became_active":"2023-10-16T15:53:40.378077+0000","last_became_peered":"2023-10-16T15:53:40.378077+0000","last_unstale":"2023-10-16T15:53:41.407785+0000","last_undegraded":"2023-10-16T15:53:41.407785+0000","last_fullsized":"2023-10-16T15:53:41.407785+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410289+0000","last_change":"2023-10-16T15:53:41.399412+0000","last_active":"2023-10-16T15:53:42.410289+0000","last_peered":"2023-10-16T15:53:42.410289+0000","last_clean":"2023-10-16T15:53:42.410289+0000","last_became_active":"2023-10-16T15:53:41.399125+0000","last_became_peered":"2023-10-16T15:53:41.399125+0000","last_unstale":"2023-10-16T15:53:42.410289+0000","last_undegraded":"2023-10-16T15:53:42.410289+0000","last_fullsized":"2023-10-16T15:53:42.410289+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376676+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:41.376676+0000","last_peered":"2023-10-16T15:53:41.376676+0000","last_clean":"2023-10-16T15:53:41.376676+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:41.376676+0000","last_undegraded":"2023-10-16T15:53:41.376676+0000","last_fullsized":"2023-10-16T15:53:41.376676+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394140+0000","last_change":"2023-10-16T15:53:40.377426+0000","last_active":"2023-10-16T15:53:41.394140+0000","last_peered":"2023-10-16T15:53:41.394140+0000","last_clean":"2023-10-16T15:53:41.394140+0000","last_became_active":"2023-10-16T15:53:40.377160+0000","last_became_peered":"2023-10-16T15:53:40.377160+0000","last_unstale":"2023-10-16T15:53:41.394140+0000","last_undegraded":"2023-10-16T15:53:41.394140+0000","last_fullsized":"2023-10-16T15:53:41.394140+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:38.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408560+0000","last_change":"2023-10-16T15:53:41.400288+0000","last_active":"2023-10-16T15:53:42.408560+0000","last_peered":"2023-10-16T15:53:42.408560+0000","last_clean":"2023-10-16T15:53:42.408560+0000","last_became_active":"2023-10-16T15:53:41.400031+0000","last_became_peered":"2023-10-16T15:53:41.400031+0000","last_unstale":"2023-10-16T15:53:42.408560+0000","last_undegraded":"2023-10-16T15:53:42.408560+0000","last_fullsized":"2023-10-16T15:53:42.408560+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393978+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:41.393978+0000","last_peered":"2023-10-16T15:53:41.393978+0000","last_clean":"2023-10-16T15:53:41.393978+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:41.393978+0000","last_undegraded":"2023-10-16T15:53:41.393978+0000","last_fullsized":"2023-10-16T15:53:41.393978+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392991+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:41.392991+0000","last_peered":"2023-10-16T15:53:41.392991+0000","last_clean":"2023-10-16T15:53:41.392991+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:41.392991+0000","last_undegraded":"2023-10-16T15:53:41.392991+0000","last_fullsized":"2023-10-16T15:53:41.392991+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414694+0000","last_change":"2023-10-16T15:53:40.379991+0000","last_active":"2023-10-16T15:53:41.414694+0000","last_peered":"2023-10-16T15:53:41.414694+0000","last_clean":"2023-10-16T15:53:41.414694+0000","last_became_active":"2023-10-16T15:53:40.379687+0000","last_became_peered":"2023-10-16T15:53:40.379687+0000","last_unstale":"2023-10-16T15:53:41.414694+0000","last_undegraded":"2023-10-16T15:53:41.414694+0000","last_fullsized":"2023-10-16T15:53:41.414694+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:39.813921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409916+0000","last_change":"2023-10-16T15:53:41.413217+0000","last_active":"2023-10-16T15:53:42.409916+0000","last_peered":"2023-10-16T15:53:42.409916+0000","last_clean":"2023-10-16T15:53:42.409916+0000","last_became_active":"2023-10-16T15:53:41.411878+0000","last_became_peered":"2023-10-16T15:53:41.411878+0000","last_unstale":"2023-10-16T15:53:42.409916+0000","last_undegraded":"2023-10-16T15:53:42.409916+0000","last_fullsized":"2023-10-16T15:53:42.409916+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384277+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:41.384277+0000","last_peered":"2023-10-16T15:53:41.384277+0000","last_clean":"2023-10-16T15:53:41.384277+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:41.384277+0000","last_undegraded":"2023-10-16T15:53:41.384277+0000","last_fullsized":"2023-10-16T15:53:41.384277+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394436+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:41.394436+0000","last_peered":"2023-10-16T15:53:41.394436+0000","last_clean":"2023-10-16T15:53:41.394436+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:41.394436+0000","last_undegraded":"2023-10-16T15:53:41.394436+0000","last_fullsized":"2023-10-16T15:53:41.394436+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389135+0000","last_change":"2023-10-16T15:53:40.384061+0000","last_active":"2023-10-16T15:53:41.389135+0000","last_peered":"2023-10-16T15:53:41.389135+0000","last_clean":"2023-10-16T15:53:41.389135+0000","last_became_active":"2023-10-16T15:53:40.383223+0000","last_became_peered":"2023-10-16T15:53:40.383223+0000","last_unstale":"2023-10-16T15:53:41.389135+0000","last_undegraded":"2023-10-16T15:53:41.389135+0000","last_fullsized":"2023-10-16T15:53:41.389135+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013025+0000","last_change":"2023-10-16T15:53:41.397581+0000","last_active":"2023-10-16T15:53:43.013025+0000","last_peered":"2023-10-16T15:53:43.013025+0000","last_clean":"2023-10-16T15:53:43.013025+0000","last_became_active":"2023-10-16T15:53:41.397301+0000","last_became_peered":"2023-10-16T15:53:41.397301+0000","last_unstale":"2023-10-16T15:53:43.013025+0000","last_undegraded":"2023-10-16T15:53:43.013025+0000","last_fullsized":"2023-10-16T15:53:43.013025+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:20.816418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394523+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:41.394523+0000","last_peered":"2023-10-16T15:53:41.394523+0000","last_clean":"2023-10-16T15:53:41.394523+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:41.394523+0000","last_undegraded":"2023-10-16T15:53:41.394523+0000","last_fullsized":"2023-10-16T15:53:41.394523+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.411952+0000","last_change":"2023-10-16T15:53:40.376976+0000","last_active":"2023-10-16T15:53:41.411952+0000","last_peered":"2023-10-16T15:53:41.411952+0000","last_clean":"2023-10-16T15:53:41.411952+0000","last_became_active":"2023-10-16T15:53:40.376700+0000","last_became_peered":"2023-10-16T15:53:40.376700+0000","last_unstale":"2023-10-16T15:53:41.411952+0000","last_undegraded":"2023-10-16T15:53:41.411952+0000","last_fullsized":"2023-10-16T15:53:41.411952+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013467+0000","last_change":"2023-10-16T15:53:41.410633+0000","last_active":"2023-10-16T15:53:43.013467+0000","last_peered":"2023-10-16T15:53:43.013467+0000","last_clean":"2023-10-16T15:53:43.013467+0000","last_became_active":"2023-10-16T15:53:41.409266+0000","last_became_peered":"2023-10-16T15:53:41.409266+0000","last_unstale":"2023-10-16T15:53:43.013467+0000","last_undegraded":"2023-10-16T15:53:43.013467+0000","last_fullsized":"2023-10-16T15:53:43.013467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:52.749663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392803+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:41.392803+0000","last_peered":"2023-10-16T15:53:41.392803+0000","last_clean":"2023-10-16T15:53:41.392803+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:41.392803+0000","last_undegraded":"2023-10-16T15:53:41.392803+0000","last_fullsized":"2023-10-16T15:53:41.392803+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393648+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:41.393648+0000","last_peered":"2023-10-16T15:53:41.393648+0000","last_clean":"2023-10-16T15:53:41.393648+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:41.393648+0000","last_undegraded":"2023-10-16T15:53:41.393648+0000","last_fullsized":"2023-10-16T15:53:41.393648+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384497+0000","last_change":"2023-10-16T15:53:40.375609+0000","last_active":"2023-10-16T15:53:41.384497+0000","last_peered":"2023-10-16T15:53:41.384497+0000","last_clean":"2023-10-16T15:53:41.384497+0000","last_became_active":"2023-10-16T15:53:40.375039+0000","last_became_peered":"2023-10-16T15:53:40.375039+0000","last_unstale":"2023-10-16T15:53:41.384497+0000","last_undegraded":"2023-10-16T15:53:41.384497+0000","last_fullsized":"2023-10-16T15:53:41.384497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:02.147563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.328614+0000","last_change":"2023-10-16T15:53:41.399266+0000","last_active":"2023-10-16T15:53:43.328614+0000","last_peered":"2023-10-16T15:53:43.328614+0000","last_clean":"2023-10-16T15:53:43.328614+0000","last_became_active":"2023-10-16T15:53:41.398177+0000","last_became_peered":"2023-10-16T15:53:41.398177+0000","last_unstale":"2023-10-16T15:53:43.328614+0000","last_undegraded":"2023-10-16T15:53:43.328614+0000","last_fullsized":"2023-10-16T15:53:43.328614+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.144341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385470+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:41.385470+0000","last_peered":"2023-10-16T15:53:41.385470+0000","last_clean":"2023-10-16T15:53:41.385470+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:41.385470+0000","last_undegraded":"2023-10-16T15:53:41.385470+0000","last_fullsized":"2023-10-16T15:53:41.385470+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392538+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:41.392538+0000","last_peered":"2023-10-16T15:53:41.392538+0000","last_clean":"2023-10-16T15:53:41.392538+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:41.392538+0000","last_undegraded":"2023-10-16T15:53:41.392538+0000","last_fullsized":"2023-10-16T15:53:41.392538+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386003+0000","last_change":"2023-10-16T15:53:40.388836+0000","last_active":"2023-10-16T15:53:41.386003+0000","last_peered":"2023-10-16T15:53:41.386003+0000","last_clean":"2023-10-16T15:53:41.386003+0000","last_became_active":"2023-10-16T15:53:40.387121+0000","last_became_peered":"2023-10-16T15:53:40.387121+0000","last_unstale":"2023-10-16T15:53:41.386003+0000","last_undegraded":"2023-10-16T15:53:41.386003+0000","last_fullsized":"2023-10-16T15:53:41.386003+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410790+0000","last_change":"2023-10-16T15:53:41.413661+0000","last_active":"2023-10-16T15:53:42.410790+0000","last_peered":"2023-10-16T15:53:42.410790+0000","last_clean":"2023-10-16T15:53:42.410790+0000","last_became_active":"2023-10-16T15:53:41.413219+0000","last_became_peered":"2023-10-16T15:53:41.413219+0000","last_unstale":"2023-10-16T15:53:42.410790+0000","last_undegraded":"2023-10-16T15:53:42.410790+0000","last_fullsized":"2023-10-16T15:53:42.410790+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:51.757315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380707+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:41.380707+0000","last_peered":"2023-10-16T15:53:41.380707+0000","last_clean":"2023-10-16T15:53:41.380707+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:41.380707+0000","last_undegraded":"2023-10-16T15:53:41.380707+0000","last_fullsized":"2023-10-16T15:53:41.380707+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374760+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:41.374760+0000","last_peered":"2023-10-16T15:53:41.374760+0000","last_clean":"2023-10-16T15:53:41.374760+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:41.374760+0000","last_undegraded":"2023-10-16T15:53:41.374760+0000","last_fullsized":"2023-10-16T15:53:41.374760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375756+0000","last_change":"2023-10-16T15:53:40.377615+0000","last_active":"2023-10-16T15:53:41.375756+0000","last_peered":"2023-10-16T15:53:41.375756+0000","last_clean":"2023-10-16T15:53:41.375756+0000","last_became_active":"2023-10-16T15:53:40.377342+0000","last_became_peered":"2023-10-16T15:53:40.377342+0000","last_unstale":"2023-10-16T15:53:41.375756+0000","last_undegraded":"2023-10-16T15:53:41.375756+0000","last_fullsized":"2023-10-16T15:53:41.375756+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.412874+0000","last_change":"2023-10-16T15:53:41.410293+0000","last_active":"2023-10-16T15:53:42.412874+0000","last_peered":"2023-10-16T15:53:42.412874+0000","last_clean":"2023-10-16T15:53:42.412874+0000","last_became_active":"2023-10-16T15:53:41.409897+0000","last_became_peered":"2023-10-16T15:53:41.409897+0000","last_unstale":"2023-10-16T15:53:42.412874+0000","last_undegraded":"2023-10-16T15:53:42.412874+0000","last_fullsized":"2023-10-16T15:53:42.412874+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:31.143645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388735+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:41.388735+0000","last_peered":"2023-10-16T15:53:41.388735+0000","last_clean":"2023-10-16T15:53:41.388735+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:41.388735+0000","last_undegraded":"2023-10-16T15:53:41.388735+0000","last_fullsized":"2023-10-16T15:53:41.388735+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380271+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:41.380271+0000","last_peered":"2023-10-16T15:53:41.380271+0000","last_clean":"2023-10-16T15:53:41.380271+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:41.380271+0000","last_undegraded":"2023-10-16T15:53:41.380271+0000","last_fullsized":"2023-10-16T15:53:41.380271+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.397754+0000","last_change":"2023-10-16T15:53:40.387403+0000","last_active":"2023-10-16T15:53:41.397754+0000","last_peered":"2023-10-16T15:53:41.397754+0000","last_clean":"2023-10-16T15:53:41.397754+0000","last_became_active":"2023-10-16T15:53:40.387110+0000","last_became_peered":"2023-10-16T15:53:40.387110+0000","last_unstale":"2023-10-16T15:53:41.397754+0000","last_undegraded":"2023-10-16T15:53:41.397754+0000","last_fullsized":"2023-10-16T15:53:41.397754+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407976+0000","last_change":"2023-10-16T15:53:41.414132+0000","last_active":"2023-10-16T15:53:42.407976+0000","last_peered":"2023-10-16T15:53:42.407976+0000","last_clean":"2023-10-16T15:53:42.407976+0000","last_became_active":"2023-10-16T15:53:41.413151+0000","last_became_peered":"2023-10-16T15:53:41.413151+0000","last_unstale":"2023-10-16T15:53:42.407976+0000","last_undegraded":"2023-10-16T15:53:42.407976+0000","last_fullsized":"2023-10-16T15:53:42.407976+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393257+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:41.393257+0000","last_peered":"2023-10-16T15:53:41.393257+0000","last_clean":"2023-10-16T15:53:41.393257+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.393257+0000","last_undegraded":"2023-10-16T15:53:41.393257+0000","last_fullsized":"2023-10-16T15:53:41.393257+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393875+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:41.393875+0000","last_peered":"2023-10-16T15:53:41.393875+0000","last_clean":"2023-10-16T15:53:41.393875+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:41.393875+0000","last_undegraded":"2023-10-16T15:53:41.393875+0000","last_fullsized":"2023-10-16T15:53:41.393875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"228'6","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414139+0000","last_change":"2023-10-16T15:53:40.376773+0000","last_active":"2023-10-16T15:53:41.414139+0000","last_peered":"2023-10-16T15:53:41.414139+0000","last_clean":"2023-10-16T15:53:41.414139+0000","last_became_active":"2023-10-16T15:53:40.376123+0000","last_became_peered":"2023-10-16T15:53:40.376123+0000","last_unstale":"2023-10-16T15:53:41.414139+0000","last_undegraded":"2023-10-16T15:53:41.414139+0000","last_fullsized":"2023-10-16T15:53:41.414139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:19.218849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.422217+0000","last_change":"2023-10-16T15:53:41.410543+0000","last_active":"2023-10-16T15:53:43.422217+0000","last_peered":"2023-10-16T15:53:43.422217+0000","last_clean":"2023-10-16T15:53:43.422217+0000","last_became_active":"2023-10-16T15:53:41.410213+0000","last_became_peered":"2023-10-16T15:53:41.410213+0000","last_unstale":"2023-10-16T15:53:43.422217+0000","last_undegraded":"2023-10-16T15:53:43.422217+0000","last_fullsized":"2023-10-16T15:53:43.422217+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387904+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:41.387904+0000","last_peered":"2023-10-16T15:53:41.387904+0000","last_clean":"2023-10-16T15:53:41.387904+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:41.387904+0000","last_undegraded":"2023-10-16T15:53:41.387904+0000","last_fullsized":"2023-10-16T15:53:41.387904+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383904+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:41.383904+0000","last_peered":"2023-10-16T15:53:41.383904+0000","last_clean":"2023-10-16T15:53:41.383904+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:41.383904+0000","last_undegraded":"2023-10-16T15:53:41.383904+0000","last_fullsized":"2023-10-16T15:53:41.383904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"228'3","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402070+0000","last_change":"2023-10-16T15:53:40.365148+0000","last_active":"2023-10-16T15:53:41.402070+0000","last_peered":"2023-10-16T15:53:41.402070+0000","last_clean":"2023-10-16T15:53:41.402070+0000","last_became_active":"2023-10-16T15:53:40.364874+0000","last_became_peered":"2023-10-16T15:53:40.364874+0000","last_unstale":"2023-10-16T15:53:41.402070+0000","last_undegraded":"2023-10-16T15:53:41.402070+0000","last_fullsized":"2023-10-16T15:53:41.402070+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:39.387182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411499+0000","last_change":"2023-10-16T15:53:41.405376+0000","last_active":"2023-10-16T15:53:42.411499+0000","last_peered":"2023-10-16T15:53:42.411499+0000","last_clean":"2023-10-16T15:53:42.411499+0000","last_became_active":"2023-10-16T15:53:41.404697+0000","last_became_peered":"2023-10-16T15:53:41.404697+0000","last_unstale":"2023-10-16T15:53:42.411499+0000","last_undegraded":"2023-10-16T15:53:42.411499+0000","last_fullsized":"2023-10-16T15:53:42.411499+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"225'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379765+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:41.379765+0000","last_peered":"2023-10-16T15:53:41.379765+0000","last_clean":"2023-10-16T15:53:41.379765+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:41.379765+0000","last_undegraded":"2023-10-16T15:53:41.379765+0000","last_fullsized":"2023-10-16T15:53:41.379765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376419+0000","last_change":"2023-10-16T15:53:40.387700+0000","last_active":"2023-10-16T15:53:41.376419+0000","last_peered":"2023-10-16T15:53:41.376419+0000","last_clean":"2023-10-16T15:53:41.376419+0000","last_became_active":"2023-10-16T15:53:40.387431+0000","last_became_peered":"2023-10-16T15:53:40.387431+0000","last_unstale":"2023-10-16T15:53:41.376419+0000","last_undegraded":"2023-10-16T15:53:41.376419+0000","last_fullsized":"2023-10-16T15:53:41.376419+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:19.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418316+0000","last_change":"2023-10-16T15:53:41.385042+0000","last_active":"2023-10-16T15:53:43.418316+0000","last_peered":"2023-10-16T15:53:43.418316+0000","last_clean":"2023-10-16T15:53:43.418316+0000","last_became_active":"2023-10-16T15:53:41.384431+0000","last_became_peered":"2023-10-16T15:53:41.384431+0000","last_unstale":"2023-10-16T15:53:43.418316+0000","last_undegraded":"2023-10-16T15:53:43.418316+0000","last_fullsized":"2023-10-16T15:53:43.418316+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394713+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:41.394713+0000","last_peered":"2023-10-16T15:53:41.394713+0000","last_clean":"2023-10-16T15:53:41.394713+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:41.394713+0000","last_undegraded":"2023-10-16T15:53:41.394713+0000","last_fullsized":"2023-10-16T15:53:41.394713+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377982+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:41.377982+0000","last_peered":"2023-10-16T15:53:41.377982+0000","last_clean":"2023-10-16T15:53:41.377982+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:41.377982+0000","last_undegraded":"2023-10-16T15:53:41.377982+0000","last_fullsized":"2023-10-16T15:53:41.377982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.408329+0000","last_change":"2023-10-16T15:53:40.386183+0000","last_active":"2023-10-16T15:53:41.408329+0000","last_peered":"2023-10-16T15:53:41.408329+0000","last_clean":"2023-10-16T15:53:41.408329+0000","last_became_active":"2023-10-16T15:53:40.385897+0000","last_became_peered":"2023-10-16T15:53:40.385897+0000","last_unstale":"2023-10-16T15:53:41.408329+0000","last_undegraded":"2023-10-16T15:53:41.408329+0000","last_fullsized":"2023-10-16T15:53:41.408329+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:27.631953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.420342+0000","last_change":"2023-10-16T15:53:41.400842+0000","last_active":"2023-10-16T15:53:43.420342+0000","last_peered":"2023-10-16T15:53:43.420342+0000","last_clean":"2023-10-16T15:53:43.420342+0000","last_became_active":"2023-10-16T15:53:41.400579+0000","last_became_peered":"2023-10-16T15:53:41.400579+0000","last_unstale":"2023-10-16T15:53:43.420342+0000","last_undegraded":"2023-10-16T15:53:43.420342+0000","last_fullsized":"2023-10-16T15:53:43.420342+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402696+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:41.402696+0000","last_peered":"2023-10-16T15:53:41.402696+0000","last_clean":"2023-10-16T15:53:41.402696+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:41.402696+0000","last_undegraded":"2023-10-16T15:53:41.402696+0000","last_fullsized":"2023-10-16T15:53:41.402696+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387083+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:41.387083+0000","last_peered":"2023-10-16T15:53:41.387083+0000","last_clean":"2023-10-16T15:53:41.387083+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:41.387083+0000","last_undegraded":"2023-10-16T15:53:41.387083+0000","last_fullsized":"2023-10-16T15:53:41.387083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"228'8","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.422402+0000","last_change":"2023-10-16T15:53:40.375924+0000","last_active":"2023-10-16T15:53:41.422402+0000","last_peered":"2023-10-16T15:53:41.422402+0000","last_clean":"2023-10-16T15:53:41.422402+0000","last_became_active":"2023-10-16T15:53:40.375638+0000","last_became_peered":"2023-10-16T15:53:40.375638+0000","last_unstale":"2023-10-16T15:53:41.422402+0000","last_undegraded":"2023-10-16T15:53:41.422402+0000","last_fullsized":"2023-10-16T15:53:41.422402+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:47.381835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407893+0000","last_change":"2023-10-16T15:53:41.414765+0000","last_active":"2023-10-16T15:53:42.407893+0000","last_peered":"2023-10-16T15:53:42.407893+0000","last_clean":"2023-10-16T15:53:42.407893+0000","last_became_active":"2023-10-16T15:53:41.414484+0000","last_became_peered":"2023-10-16T15:53:41.414484+0000","last_unstale":"2023-10-16T15:53:42.407893+0000","last_undegraded":"2023-10-16T15:53:42.407893+0000","last_fullsized":"2023-10-16T15:53:42.407893+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385369+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:41.385369+0000","last_peered":"2023-10-16T15:53:41.385369+0000","last_clean":"2023-10-16T15:53:41.385369+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:41.385369+0000","last_undegraded":"2023-10-16T15:53:41.385369+0000","last_fullsized":"2023-10-16T15:53:41.385369+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379140+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:41.379140+0000","last_peered":"2023-10-16T15:53:41.379140+0000","last_clean":"2023-10-16T15:53:41.379140+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:41.379140+0000","last_undegraded":"2023-10-16T15:53:41.379140+0000","last_fullsized":"2023-10-16T15:53:41.379140+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382349+0000","last_change":"2023-10-16T15:53:40.376076+0000","last_active":"2023-10-16T15:53:41.382349+0000","last_peered":"2023-10-16T15:53:41.382349+0000","last_clean":"2023-10-16T15:53:41.382349+0000","last_became_active":"2023-10-16T15:53:40.375798+0000","last_became_peered":"2023-10-16T15:53:40.375798+0000","last_unstale":"2023-10-16T15:53:41.382349+0000","last_undegraded":"2023-10-16T15:53:41.382349+0000","last_fullsized":"2023-10-16T15:53:41.382349+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330851+0000","last_change":"2023-10-16T15:53:41.409878+0000","last_active":"2023-10-16T15:53:43.330851+0000","last_peered":"2023-10-16T15:53:43.330851+0000","last_clean":"2023-10-16T15:53:43.330851+0000","last_became_active":"2023-10-16T15:53:41.409594+0000","last_became_peered":"2023-10-16T15:53:41.409594+0000","last_unstale":"2023-10-16T15:53:43.330851+0000","last_undegraded":"2023-10-16T15:53:43.330851+0000","last_fullsized":"2023-10-16T15:53:43.330851+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:10.790473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.419545+0000","last_change":"2023-10-16T15:53:40.385839+0000","last_active":"2023-10-16T15:53:41.419545+0000","last_peered":"2023-10-16T15:53:41.419545+0000","last_clean":"2023-10-16T15:53:41.419545+0000","last_became_active":"2023-10-16T15:53:40.384731+0000","last_became_peered":"2023-10-16T15:53:40.384731+0000","last_unstale":"2023-10-16T15:53:41.419545+0000","last_undegraded":"2023-10-16T15:53:41.419545+0000","last_fullsized":"2023-10-16T15:53:41.419545+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:17.818654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386473+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:41.386473+0000","last_peered":"2023-10-16T15:53:41.386473+0000","last_clean":"2023-10-16T15:53:41.386473+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:41.386473+0000","last_undegraded":"2023-10-16T15:53:41.386473+0000","last_fullsized":"2023-10-16T15:53:41.386473+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378524+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:41.378524+0000","last_peered":"2023-10-16T15:53:41.378524+0000","last_clean":"2023-10-16T15:53:41.378524+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:41.378524+0000","last_undegraded":"2023-10-16T15:53:41.378524+0000","last_fullsized":"2023-10-16T15:53:41.378524+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410670+0000","last_change":"2023-10-16T15:53:41.411697+0000","last_active":"2023-10-16T15:53:42.410670+0000","last_peered":"2023-10-16T15:53:42.410670+0000","last_clean":"2023-10-16T15:53:42.410670+0000","last_became_active":"2023-10-16T15:53:41.410477+0000","last_became_peered":"2023-10-16T15:53:41.410477+0000","last_unstale":"2023-10-16T15:53:42.410670+0000","last_undegraded":"2023-10-16T15:53:42.410670+0000","last_fullsized":"2023-10-16T15:53:42.410670+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:52.524535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"228'4","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.418437+0000","last_change":"2023-10-16T15:53:40.380112+0000","last_active":"2023-10-16T15:53:41.418437+0000","last_peered":"2023-10-16T15:53:41.418437+0000","last_clean":"2023-10-16T15:53:41.418437+0000","last_became_active":"2023-10-16T15:53:40.379085+0000","last_became_peered":"2023-10-16T15:53:40.379085+0000","last_unstale":"2023-10-16T15:53:41.418437+0000","last_undegraded":"2023-10-16T15:53:41.418437+0000","last_fullsized":"2023-10-16T15:53:41.418437+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:48.756275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382648+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:41.382648+0000","last_peered":"2023-10-16T15:53:41.382648+0000","last_clean":"2023-10-16T15:53:41.382648+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:41.382648+0000","last_undegraded":"2023-10-16T15:53:41.382648+0000","last_fullsized":"2023-10-16T15:53:41.382648+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379944+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:41.379944+0000","last_peered":"2023-10-16T15:53:41.379944+0000","last_clean":"2023-10-16T15:53:41.379944+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:41.379944+0000","last_undegraded":"2023-10-16T15:53:41.379944+0000","last_fullsized":"2023-10-16T15:53:41.379944+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421686+0000","last_change":"2023-10-16T15:53:41.405016+0000","last_active":"2023-10-16T15:53:43.421686+0000","last_peered":"2023-10-16T15:53:43.421686+0000","last_clean":"2023-10-16T15:53:43.421686+0000","last_became_active":"2023-10-16T15:53:41.404775+0000","last_became_peered":"2023-10-16T15:53:41.404775+0000","last_unstale":"2023-10-16T15:53:43.421686+0000","last_undegraded":"2023-10-16T15:53:43.421686+0000","last_fullsized":"2023-10-16T15:53:43.421686+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384001+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:41.384001+0000","last_peered":"2023-10-16T15:53:41.384001+0000","last_clean":"2023-10-16T15:53:41.384001+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:41.384001+0000","last_undegraded":"2023-10-16T15:53:41.384001+0000","last_fullsized":"2023-10-16T15:53:41.384001+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378030+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:41.378030+0000","last_peered":"2023-10-16T15:53:41.378030+0000","last_clean":"2023-10-16T15:53:41.378030+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:41.378030+0000","last_undegraded":"2023-10-16T15:53:41.378030+0000","last_fullsized":"2023-10-16T15:53:41.378030+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.420446+0000","last_change":"2023-10-16T15:53:40.382867+0000","last_active":"2023-10-16T15:53:41.420446+0000","last_peered":"2023-10-16T15:53:41.420446+0000","last_clean":"2023-10-16T15:53:41.420446+0000","last_became_active":"2023-10-16T15:53:40.378850+0000","last_became_peered":"2023-10-16T15:53:40.378850+0000","last_unstale":"2023-10-16T15:53:41.420446+0000","last_undegraded":"2023-10-16T15:53:41.420446+0000","last_fullsized":"2023-10-16T15:53:41.420446+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:00.525042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013915+0000","last_change":"2023-10-16T15:53:41.405768+0000","last_active":"2023-10-16T15:53:43.013915+0000","last_peered":"2023-10-16T15:53:43.013915+0000","last_clean":"2023-10-16T15:53:43.013915+0000","last_became_active":"2023-10-16T15:53:41.405230+0000","last_became_peered":"2023-10-16T15:53:41.405230+0000","last_unstale":"2023-10-16T15:53:43.013915+0000","last_undegraded":"2023-10-16T15:53:43.013915+0000","last_fullsized":"2023-10-16T15:53:43.013915+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:43.313902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378254+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:41.378254+0000","last_peered":"2023-10-16T15:53:41.378254+0000","last_clean":"2023-10-16T15:53:41.378254+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:41.378254+0000","last_undegraded":"2023-10-16T15:53:41.378254+0000","last_fullsized":"2023-10-16T15:53:41.378254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394353+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:41.394353+0000","last_peered":"2023-10-16T15:53:41.394353+0000","last_clean":"2023-10-16T15:53:41.394353+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:41.394353+0000","last_undegraded":"2023-10-16T15:53:41.394353+0000","last_fullsized":"2023-10-16T15:53:41.394353+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.404498+0000","last_change":"2023-10-16T15:53:40.381325+0000","last_active":"2023-10-16T15:53:41.404498+0000","last_peered":"2023-10-16T15:53:41.404498+0000","last_clean":"2023-10-16T15:53:41.404498+0000","last_became_active":"2023-10-16T15:53:40.380822+0000","last_became_peered":"2023-10-16T15:53:40.380822+0000","last_unstale":"2023-10-16T15:53:41.404498+0000","last_undegraded":"2023-10-16T15:53:41.404498+0000","last_fullsized":"2023-10-16T15:53:41.404498+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406853+0000","last_change":"2023-10-16T15:53:41.405116+0000","last_active":"2023-10-16T15:53:42.406853+0000","last_peered":"2023-10-16T15:53:42.406853+0000","last_clean":"2023-10-16T15:53:42.406853+0000","last_became_active":"2023-10-16T15:53:41.403778+0000","last_became_peered":"2023-10-16T15:53:41.403778+0000","last_unstale":"2023-10-16T15:53:42.406853+0000","last_undegraded":"2023-10-16T15:53:42.406853+0000","last_fullsized":"2023-10-16T15:53:42.406853+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385254+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:41.385254+0000","last_peered":"2023-10-16T15:53:41.385254+0000","last_clean":"2023-10-16T15:53:41.385254+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:41.385254+0000","last_undegraded":"2023-10-16T15:53:41.385254+0000","last_fullsized":"2023-10-16T15:53:41.385254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374210+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:41.374210+0000","last_peered":"2023-10-16T15:53:41.374210+0000","last_clean":"2023-10-16T15:53:41.374210+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:41.374210+0000","last_undegraded":"2023-10-16T15:53:41.374210+0000","last_fullsized":"2023-10-16T15:53:41.374210+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.409948+0000","last_change":"2023-10-16T15:53:40.377357+0000","last_active":"2023-10-16T15:53:41.409948+0000","last_peered":"2023-10-16T15:53:41.409948+0000","last_clean":"2023-10-16T15:53:41.409948+0000","last_became_active":"2023-10-16T15:53:40.377070+0000","last_became_peered":"2023-10-16T15:53:40.377070+0000","last_unstale":"2023-10-16T15:53:41.409948+0000","last_undegraded":"2023-10-16T15:53:41.409948+0000","last_fullsized":"2023-10-16T15:53:41.409948+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405599+0000","last_change":"2023-10-16T15:53:41.407410+0000","last_active":"2023-10-16T15:53:42.405599+0000","last_peered":"2023-10-16T15:53:42.405599+0000","last_clean":"2023-10-16T15:53:42.405599+0000","last_became_active":"2023-10-16T15:53:41.407034+0000","last_became_peered":"2023-10-16T15:53:41.407034+0000","last_unstale":"2023-10-16T15:53:42.405599+0000","last_undegraded":"2023-10-16T15:53:42.405599+0000","last_fullsized":"2023-10-16T15:53:42.405599+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:31.173108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392695+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:41.392695+0000","last_peered":"2023-10-16T15:53:41.392695+0000","last_clean":"2023-10-16T15:53:41.392695+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:41.392695+0000","last_undegraded":"2023-10-16T15:53:41.392695+0000","last_fullsized":"2023-10-16T15:53:41.392695+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.398663+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:41.398663+0000","last_peered":"2023-10-16T15:53:41.398663+0000","last_clean":"2023-10-16T15:53:41.398663+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:41.398663+0000","last_undegraded":"2023-10-16T15:53:41.398663+0000","last_fullsized":"2023-10-16T15:53:41.398663+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391139+0000","last_change":"2023-10-16T15:53:40.380015+0000","last_active":"2023-10-16T15:53:41.391139+0000","last_peered":"2023-10-16T15:53:41.391139+0000","last_clean":"2023-10-16T15:53:41.391139+0000","last_became_active":"2023-10-16T15:53:40.379716+0000","last_became_peered":"2023-10-16T15:53:40.379716+0000","last_unstale":"2023-10-16T15:53:41.391139+0000","last_undegraded":"2023-10-16T15:53:41.391139+0000","last_fullsized":"2023-10-16T15:53:41.391139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402124+0000","last_change":"2023-10-16T15:53:41.403677+0000","last_active":"2023-10-16T15:53:42.402124+0000","last_peered":"2023-10-16T15:53:42.402124+0000","last_clean":"2023-10-16T15:53:42.402124+0000","last_became_active":"2023-10-16T15:53:41.403372+0000","last_became_peered":"2023-10-16T15:53:41.403372+0000","last_unstale":"2023-10-16T15:53:42.402124+0000","last_undegraded":"2023-10-16T15:53:42.402124+0000","last_fullsized":"2023-10-16T15:53:42.402124+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396765+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:41.396765+0000","last_peered":"2023-10-16T15:53:41.396765+0000","last_clean":"2023-10-16T15:53:41.396765+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.396765+0000","last_undegraded":"2023-10-16T15:53:41.396765+0000","last_fullsized":"2023-10-16T15:53:41.396765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392453+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:41.392453+0000","last_peered":"2023-10-16T15:53:41.392453+0000","last_clean":"2023-10-16T15:53:41.392453+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:41.392453+0000","last_undegraded":"2023-10-16T15:53:41.392453+0000","last_fullsized":"2023-10-16T15:53:41.392453+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385863+0000","last_change":"2023-10-16T15:53:40.385668+0000","last_active":"2023-10-16T15:53:41.385863+0000","last_peered":"2023-10-16T15:53:41.385863+0000","last_clean":"2023-10-16T15:53:41.385863+0000","last_became_active":"2023-10-16T15:53:40.385360+0000","last_became_peered":"2023-10-16T15:53:40.385360+0000","last_unstale":"2023-10-16T15:53:41.385863+0000","last_undegraded":"2023-10-16T15:53:41.385863+0000","last_fullsized":"2023-10-16T15:53:41.385863+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:54.336040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330261+0000","last_change":"2023-10-16T15:53:41.397528+0000","last_active":"2023-10-16T15:53:43.330261+0000","last_peered":"2023-10-16T15:53:43.330261+0000","last_clean":"2023-10-16T15:53:43.330261+0000","last_became_active":"2023-10-16T15:53:41.397247+0000","last_became_peered":"2023-10-16T15:53:41.397247+0000","last_unstale":"2023-10-16T15:53:43.330261+0000","last_undegraded":"2023-10-16T15:53:43.330261+0000","last_fullsized":"2023-10-16T15:53:43.330261+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:21.337037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.373989+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:41.373989+0000","last_peered":"2023-10-16T15:53:41.373989+0000","last_clean":"2023-10-16T15:53:41.373989+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:41.373989+0000","last_undegraded":"2023-10-16T15:53:41.373989+0000","last_fullsized":"2023-10-16T15:53:41.373989+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410135+0000","last_change":"2023-10-16T15:53:41.408679+0000","last_active":"2023-10-16T15:53:42.410135+0000","last_peered":"2023-10-16T15:53:42.410135+0000","last_clean":"2023-10-16T15:53:42.410135+0000","last_became_active":"2023-10-16T15:53:41.407808+0000","last_became_peered":"2023-10-16T15:53:41.407808+0000","last_unstale":"2023-10-16T15:53:42.410135+0000","last_undegraded":"2023-10-16T15:53:42.410135+0000","last_fullsized":"2023-10-16T15:53:42.410135+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388628+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:41.388628+0000","last_peered":"2023-10-16T15:53:41.388628+0000","last_clean":"2023-10-16T15:53:41.388628+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:41.388628+0000","last_undegraded":"2023-10-16T15:53:41.388628+0000","last_fullsized":"2023-10-16T15:53:41.388628+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402941+0000","last_change":"2023-10-16T15:53:41.398094+0000","last_active":"2023-10-16T15:53:42.402941+0000","last_peered":"2023-10-16T15:53:42.402941+0000","last_clean":"2023-10-16T15:53:42.402941+0000","last_became_active":"2023-10-16T15:53:41.396310+0000","last_became_peered":"2023-10-16T15:53:41.396310+0000","last_unstale":"2023-10-16T15:53:42.402941+0000","last_undegraded":"2023-10-16T15:53:42.402941+0000","last_fullsized":"2023-10-16T15:53:42.402941+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389288+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:41.389288+0000","last_peered":"2023-10-16T15:53:41.389288+0000","last_clean":"2023-10-16T15:53:41.389288+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:41.389288+0000","last_undegraded":"2023-10-16T15:53:41.389288+0000","last_fullsized":"2023-10-16T15:53:41.389288+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408346+0000","last_change":"2023-10-16T15:53:41.409765+0000","last_active":"2023-10-16T15:53:42.408346+0000","last_peered":"2023-10-16T15:53:42.408346+0000","last_clean":"2023-10-16T15:53:42.408346+0000","last_became_active":"2023-10-16T15:53:41.409150+0000","last_became_peered":"2023-10-16T15:53:41.409150+0000","last_unstale":"2023-10-16T15:53:42.408346+0000","last_undegraded":"2023-10-16T15:53:42.408346+0000","last_fullsized":"2023-10-16T15:53:42.408346+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395859+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:41.395859+0000","last_peered":"2023-10-16T15:53:41.395859+0000","last_clean":"2023-10-16T15:53:41.395859+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:41.395859+0000","last_undegraded":"2023-10-16T15:53:41.395859+0000","last_fullsized":"2023-10-16T15:53:41.395859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406478+0000","last_change":"2023-10-16T15:53:41.405809+0000","last_active":"2023-10-16T15:53:42.406478+0000","last_peered":"2023-10-16T15:53:42.406478+0000","last_clean":"2023-10-16T15:53:42.406478+0000","last_became_active":"2023-10-16T15:53:41.405541+0000","last_became_peered":"2023-10-16T15:53:41.405541+0000","last_unstale":"2023-10-16T15:53:42.406478+0000","last_undegraded":"2023-10-16T15:53:42.406478+0000","last_fullsized":"2023-10-16T15:53:42.406478+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:39.653448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421402+0000","last_change":"2023-10-16T15:53:41.411322+0000","last_active":"2023-10-16T15:53:43.421402+0000","last_peered":"2023-10-16T15:53:43.421402+0000","last_clean":"2023-10-16T15:53:43.421402+0000","last_became_active":"2023-10-16T15:53:41.410176+0000","last_became_peered":"2023-10-16T15:53:41.410176+0000","last_unstale":"2023-10-16T15:53:43.421402+0000","last_undegraded":"2023-10-16T15:53:43.421402+0000","last_fullsized":"2023-10-16T15:53:43.421402+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393894+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:41.393894+0000","last_peered":"2023-10-16T15:53:41.393894+0000","last_clean":"2023-10-16T15:53:41.393894+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:41.393894+0000","last_undegraded":"2023-10-16T15:53:41.393894+0000","last_fullsized":"2023-10-16T15:53:41.393894+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418360+0000","last_change":"2023-10-16T15:53:41.400386+0000","last_active":"2023-10-16T15:53:43.418360+0000","last_peered":"2023-10-16T15:53:43.418360+0000","last_clean":"2023-10-16T15:53:43.418360+0000","last_became_active":"2023-10-16T15:53:41.400083+0000","last_became_peered":"2023-10-16T15:53:41.400083+0000","last_unstale":"2023-10-16T15:53:43.418360+0000","last_undegraded":"2023-10-16T15:53:43.418360+0000","last_fullsized":"2023-10-16T15:53:43.418360+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375718+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:41.375718+0000","last_peered":"2023-10-16T15:53:41.375718+0000","last_clean":"2023-10-16T15:53:41.375718+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:41.375718+0000","last_undegraded":"2023-10-16T15:53:41.375718+0000","last_fullsized":"2023-10-16T15:53:41.375718+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012816+0000","last_change":"2023-10-16T15:53:41.399893+0000","last_active":"2023-10-16T15:53:43.012816+0000","last_peered":"2023-10-16T15:53:43.012816+0000","last_clean":"2023-10-16T15:53:43.012816+0000","last_became_active":"2023-10-16T15:53:41.398053+0000","last_became_peered":"2023-10-16T15:53:41.398053+0000","last_unstale":"2023-10-16T15:53:43.012816+0000","last_undegraded":"2023-10-16T15:53:43.012816+0000","last_fullsized":"2023-10-16T15:53:43.012816+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:18.848509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393550+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:41.393550+0000","last_peered":"2023-10-16T15:53:41.393550+0000","last_clean":"2023-10-16T15:53:41.393550+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.393550+0000","last_undegraded":"2023-10-16T15:53:41.393550+0000","last_fullsized":"2023-10-16T15:53:41.393550+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404083+0000","last_change":"2023-10-16T15:53:41.406721+0000","last_active":"2023-10-16T15:53:42.404083+0000","last_peered":"2023-10-16T15:53:42.404083+0000","last_clean":"2023-10-16T15:53:42.404083+0000","last_became_active":"2023-10-16T15:53:41.404619+0000","last_became_peered":"2023-10-16T15:53:41.404619+0000","last_unstale":"2023-10-16T15:53:42.404083+0000","last_undegraded":"2023-10-16T15:53:42.404083+0000","last_fullsized":"2023-10-16T15:53:42.404083+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376431+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:41.376431+0000","last_peered":"2023-10-16T15:53:41.376431+0000","last_clean":"2023-10-16T15:53:41.376431+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:41.376431+0000","last_undegraded":"2023-10-16T15:53:41.376431+0000","last_fullsized":"2023-10-16T15:53:41.376431+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421928+0000","last_change":"2023-10-16T15:53:41.402971+0000","last_active":"2023-10-16T15:53:43.421928+0000","last_peered":"2023-10-16T15:53:43.421928+0000","last_clean":"2023-10-16T15:53:43.421928+0000","last_became_active":"2023-10-16T15:53:41.402710+0000","last_became_peered":"2023-10-16T15:53:41.402710+0000","last_unstale":"2023-10-16T15:53:43.421928+0000","last_undegraded":"2023-10-16T15:53:43.421928+0000","last_fullsized":"2023-10-16T15:53:43.421928+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389236+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:41.389236+0000","last_peered":"2023-10-16T15:53:41.389236+0000","last_clean":"2023-10-16T15:53:41.389236+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:41.389236+0000","last_undegraded":"2023-10-16T15:53:41.389236+0000","last_fullsized":"2023-10-16T15:53:41.389236+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402772+0000","last_change":"2023-10-16T15:53:41.399691+0000","last_active":"2023-10-16T15:53:42.402772+0000","last_peered":"2023-10-16T15:53:42.402772+0000","last_clean":"2023-10-16T15:53:42.402772+0000","last_became_active":"2023-10-16T15:53:41.397849+0000","last_became_peered":"2023-10-16T15:53:41.397849+0000","last_unstale":"2023-10-16T15:53:42.402772+0000","last_undegraded":"2023-10-16T15:53:42.402772+0000","last_fullsized":"2023-10-16T15:53:42.402772+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393175+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:41.393175+0000","last_peered":"2023-10-16T15:53:41.393175+0000","last_clean":"2023-10-16T15:53:41.393175+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:41.393175+0000","last_undegraded":"2023-10-16T15:53:41.393175+0000","last_fullsized":"2023-10-16T15:53:41.393175+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014730+0000","last_change":"2023-10-16T15:53:41.410559+0000","last_active":"2023-10-16T15:53:43.014730+0000","last_peered":"2023-10-16T15:53:43.014730+0000","last_clean":"2023-10-16T15:53:43.014730+0000","last_became_active":"2023-10-16T15:53:41.410258+0000","last_became_peered":"2023-10-16T15:53:41.410258+0000","last_unstale":"2023-10-16T15:53:43.014730+0000","last_undegraded":"2023-10-16T15:53:43.014730+0000","last_fullsized":"2023-10-16T15:53:43.014730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:25.813239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377897+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:41.377897+0000","last_peered":"2023-10-16T15:53:41.377897+0000","last_clean":"2023-10-16T15:53:41.377897+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:41.377897+0000","last_undegraded":"2023-10-16T15:53:41.377897+0000","last_fullsized":"2023-10-16T15:53:41.377897+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418981+0000","last_change":"2023-10-16T15:53:41.416192+0000","last_active":"2023-10-16T15:53:43.418981+0000","last_peered":"2023-10-16T15:53:43.418981+0000","last_clean":"2023-10-16T15:53:43.418981+0000","last_became_active":"2023-10-16T15:53:41.415862+0000","last_became_peered":"2023-10-16T15:53:41.415862+0000","last_unstale":"2023-10-16T15:53:43.418981+0000","last_undegraded":"2023-10-16T15:53:43.418981+0000","last_fullsized":"2023-10-16T15:53:43.418981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383281+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:41.383281+0000","last_peered":"2023-10-16T15:53:41.383281+0000","last_clean":"2023-10-16T15:53:41.383281+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:41.383281+0000","last_undegraded":"2023-10-16T15:53:41.383281+0000","last_fullsized":"2023-10-16T15:53:41.383281+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013226+0000","last_change":"2023-10-16T15:53:41.409431+0000","last_active":"2023-10-16T15:53:43.013226+0000","last_peered":"2023-10-16T15:53:43.013226+0000","last_clean":"2023-10-16T15:53:43.013226+0000","last_became_active":"2023-10-16T15:53:41.409023+0000","last_became_peered":"2023-10-16T15:53:41.409023+0000","last_unstale":"2023-10-16T15:53:43.013226+0000","last_undegraded":"2023-10-16T15:53:43.013226+0000","last_fullsized":"2023-10-16T15:53:43.013226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:31.127234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394621+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:41.394621+0000","last_peered":"2023-10-16T15:53:41.394621+0000","last_clean":"2023-10-16T15:53:41.394621+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:41.394621+0000","last_undegraded":"2023-10-16T15:53:41.394621+0000","last_fullsized":"2023-10-16T15:53:41.394621+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012572+0000","last_change":"2023-10-16T15:53:41.397505+0000","last_active":"2023-10-16T15:53:43.012572+0000","last_peered":"2023-10-16T15:53:43.012572+0000","last_clean":"2023-10-16T15:53:43.012572+0000","last_became_active":"2023-10-16T15:53:41.397198+0000","last_became_peered":"2023-10-16T15:53:41.397198+0000","last_unstale":"2023-10-16T15:53:43.012572+0000","last_undegraded":"2023-10-16T15:53:43.012572+0000","last_fullsized":"2023-10-16T15:53:43.012572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:07.665494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394898+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:41.394898+0000","last_peered":"2023-10-16T15:53:41.394898+0000","last_clean":"2023-10-16T15:53:41.394898+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:41.394898+0000","last_undegraded":"2023-10-16T15:53:41.394898+0000","last_fullsized":"2023-10-16T15:53:41.394898+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406566+0000","last_change":"2023-10-16T15:53:41.388087+0000","last_active":"2023-10-16T15:53:42.406566+0000","last_peered":"2023-10-16T15:53:42.406566+0000","last_clean":"2023-10-16T15:53:42.406566+0000","last_became_active":"2023-10-16T15:53:41.387820+0000","last_became_peered":"2023-10-16T15:53:41.387820+0000","last_unstale":"2023-10-16T15:53:42.406566+0000","last_undegraded":"2023-10-16T15:53:42.406566+0000","last_fullsized":"2023-10-16T15:53:42.406566+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374811+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:41.374811+0000","last_peered":"2023-10-16T15:53:41.374811+0000","last_clean":"2023-10-16T15:53:41.374811+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:41.374811+0000","last_undegraded":"2023-10-16T15:53:41.374811+0000","last_fullsized":"2023-10-16T15:53:41.374811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404049+0000","last_change":"2023-10-16T15:53:41.397057+0000","last_active":"2023-10-16T15:53:42.404049+0000","last_peered":"2023-10-16T15:53:42.404049+0000","last_clean":"2023-10-16T15:53:42.404049+0000","last_became_active":"2023-10-16T15:53:41.396163+0000","last_became_peered":"2023-10-16T15:53:41.396163+0000","last_unstale":"2023-10-16T15:53:42.404049+0000","last_undegraded":"2023-10-16T15:53:42.404049+0000","last_fullsized":"2023-10-16T15:53:42.404049+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391109+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:41.391109+0000","last_peered":"2023-10-16T15:53:41.391109+0000","last_clean":"2023-10-16T15:53:41.391109+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:41.391109+0000","last_undegraded":"2023-10-16T15:53:41.391109+0000","last_fullsized":"2023-10-16T15:53:41.391109+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410984+0000","last_change":"2023-10-16T15:53:41.410981+0000","last_active":"2023-10-16T15:53:42.410984+0000","last_peered":"2023-10-16T15:53:42.410984+0000","last_clean":"2023-10-16T15:53:42.410984+0000","last_became_active":"2023-10-16T15:53:41.410713+0000","last_became_peered":"2023-10-16T15:53:41.410713+0000","last_unstale":"2023-10-16T15:53:42.410984+0000","last_undegraded":"2023-10-16T15:53:42.410984+0000","last_fullsized":"2023-10-16T15:53:42.410984+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.539110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374479+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:41.374479+0000","last_peered":"2023-10-16T15:53:41.374479+0000","last_clean":"2023-10-16T15:53:41.374479+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:41.374479+0000","last_undegraded":"2023-10-16T15:53:41.374479+0000","last_fullsized":"2023-10-16T15:53:41.374479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405376+0000","last_change":"2023-10-16T15:53:41.413215+0000","last_active":"2023-10-16T15:53:42.405376+0000","last_peered":"2023-10-16T15:53:42.405376+0000","last_clean":"2023-10-16T15:53:42.405376+0000","last_became_active":"2023-10-16T15:53:41.412398+0000","last_became_peered":"2023-10-16T15:53:41.412398+0000","last_unstale":"2023-10-16T15:53:42.405376+0000","last_undegraded":"2023-10-16T15:53:42.405376+0000","last_fullsized":"2023-10-16T15:53:42.405376+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374715+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:41.374715+0000","last_peered":"2023-10-16T15:53:41.374715+0000","last_clean":"2023-10-16T15:53:41.374715+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:41.374715+0000","last_undegraded":"2023-10-16T15:53:41.374715+0000","last_fullsized":"2023-10-16T15:53:41.374715+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407630+0000","last_change":"2023-10-16T15:53:41.409487+0000","last_active":"2023-10-16T15:53:42.407630+0000","last_peered":"2023-10-16T15:53:42.407630+0000","last_clean":"2023-10-16T15:53:42.407630+0000","last_became_active":"2023-10-16T15:53:41.409160+0000","last_became_peered":"2023-10-16T15:53:41.409160+0000","last_unstale":"2023-10-16T15:53:42.407630+0000","last_undegraded":"2023-10-16T15:53:42.407630+0000","last_fullsized":"2023-10-16T15:53:42.407630+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396559+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:41.396559+0000","last_peered":"2023-10-16T15:53:41.396559+0000","last_clean":"2023-10-16T15:53:41.396559+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:41.396559+0000","last_undegraded":"2023-10-16T15:53:41.396559+0000","last_fullsized":"2023-10-16T15:53:41.396559+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.420943+0000","last_change":"2023-10-16T15:53:41.393762+0000","last_active":"2023-10-16T15:53:43.420943+0000","last_peered":"2023-10-16T15:53:43.420943+0000","last_clean":"2023-10-16T15:53:43.420943+0000","last_became_active":"2023-10-16T15:53:41.393255+0000","last_became_peered":"2023-10-16T15:53:41.393255+0000","last_unstale":"2023-10-16T15:53:43.420943+0000","last_undegraded":"2023-10-16T15:53:43.420943+0000","last_fullsized":"2023-10-16T15:53:43.420943+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.400191+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:41.400191+0000","last_peered":"2023-10-16T15:53:41.400191+0000","last_clean":"2023-10-16T15:53:41.400191+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:41.400191+0000","last_undegraded":"2023-10-16T15:53:41.400191+0000","last_fullsized":"2023-10-16T15:53:41.400191+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405352+0000","last_change":"2023-10-16T15:53:41.416464+0000","last_active":"2023-10-16T15:53:42.405352+0000","last_peered":"2023-10-16T15:53:42.405352+0000","last_clean":"2023-10-16T15:53:42.405352+0000","last_became_active":"2023-10-16T15:53:41.409828+0000","last_became_peered":"2023-10-16T15:53:41.409828+0000","last_unstale":"2023-10-16T15:53:42.405352+0000","last_undegraded":"2023-10-16T15:53:42.405352+0000","last_fullsized":"2023-10-16T15:53:42.405352+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385167+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:41.385167+0000","last_peered":"2023-10-16T15:53:41.385167+0000","last_clean":"2023-10-16T15:53:41.385167+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:41.385167+0000","last_undegraded":"2023-10-16T15:53:41.385167+0000","last_fullsized":"2023-10-16T15:53:41.385167+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405989+0000","last_change":"2023-10-16T15:53:41.400670+0000","last_active":"2023-10-16T15:53:42.405989+0000","last_peered":"2023-10-16T15:53:42.405989+0000","last_clean":"2023-10-16T15:53:42.405989+0000","last_became_active":"2023-10-16T15:53:41.400310+0000","last_became_peered":"2023-10-16T15:53:41.400310+0000","last_unstale":"2023-10-16T15:53:42.405989+0000","last_undegraded":"2023-10-16T15:53:42.405989+0000","last_fullsized":"2023-10-16T15:53:42.405989+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:55.180266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394811+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:41.394811+0000","last_peered":"2023-10-16T15:53:41.394811+0000","last_clean":"2023-10-16T15:53:41.394811+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:41.394811+0000","last_undegraded":"2023-10-16T15:53:41.394811+0000","last_fullsized":"2023-10-16T15:53:41.394811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418006+0000","last_change":"2023-10-16T15:53:41.406425+0000","last_active":"2023-10-16T15:53:43.418006+0000","last_peered":"2023-10-16T15:53:43.418006+0000","last_clean":"2023-10-16T15:53:43.418006+0000","last_became_active":"2023-10-16T15:53:41.405266+0000","last_became_peered":"2023-10-16T15:53:41.405266+0000","last_unstale":"2023-10-16T15:53:43.418006+0000","last_undegraded":"2023-10-16T15:53:43.418006+0000","last_fullsized":"2023-10-16T15:53:43.418006+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:36.594777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393796+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:41.393796+0000","last_peered":"2023-10-16T15:53:41.393796+0000","last_clean":"2023-10-16T15:53:41.393796+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:41.393796+0000","last_undegraded":"2023-10-16T15:53:41.393796+0000","last_fullsized":"2023-10-16T15:53:41.393796+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.413065+0000","last_change":"2023-10-16T15:53:41.410135+0000","last_active":"2023-10-16T15:53:42.413065+0000","last_peered":"2023-10-16T15:53:42.413065+0000","last_clean":"2023-10-16T15:53:42.413065+0000","last_became_active":"2023-10-16T15:53:41.409626+0000","last_became_peered":"2023-10-16T15:53:41.409626+0000","last_unstale":"2023-10-16T15:53:42.413065+0000","last_undegraded":"2023-10-16T15:53:42.413065+0000","last_fullsized":"2023-10-16T15:53:42.413065+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.484536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379861+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:41.379861+0000","last_peered":"2023-10-16T15:53:41.379861+0000","last_clean":"2023-10-16T15:53:41.379861+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:41.379861+0000","last_undegraded":"2023-10-16T15:53:41.379861+0000","last_fullsized":"2023-10-16T15:53:41.379861+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409800+0000","last_change":"2023-10-16T15:53:41.408039+0000","last_active":"2023-10-16T15:53:42.409800+0000","last_peered":"2023-10-16T15:53:42.409800+0000","last_clean":"2023-10-16T15:53:42.409800+0000","last_became_active":"2023-10-16T15:53:41.407756+0000","last_became_peered":"2023-10-16T15:53:41.407756+0000","last_unstale":"2023-10-16T15:53:42.409800+0000","last_undegraded":"2023-10-16T15:53:42.409800+0000","last_fullsized":"2023-10-16T15:53:42.409800+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374213+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:41.374213+0000","last_peered":"2023-10-16T15:53:41.374213+0000","last_clean":"2023-10-16T15:53:41.374213+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:41.374213+0000","last_undegraded":"2023-10-16T15:53:41.374213+0000","last_fullsized":"2023-10-16T15:53:41.374213+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013669+0000","last_change":"2023-10-16T15:53:41.411866+0000","last_active":"2023-10-16T15:53:43.013669+0000","last_peered":"2023-10-16T15:53:43.013669+0000","last_clean":"2023-10-16T15:53:43.013669+0000","last_became_active":"2023-10-16T15:53:41.411569+0000","last_became_peered":"2023-10-16T15:53:41.411569+0000","last_unstale":"2023-10-16T15:53:43.013669+0000","last_undegraded":"2023-10-16T15:53:43.013669+0000","last_fullsized":"2023-10-16T15:53:43.013669+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:01.576399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383777+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:41.383777+0000","last_peered":"2023-10-16T15:53:41.383777+0000","last_clean":"2023-10-16T15:53:41.383777+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:41.383777+0000","last_undegraded":"2023-10-16T15:53:41.383777+0000","last_fullsized":"2023-10-16T15:53:41.383777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409957+0000","last_change":"2023-10-16T15:53:41.399860+0000","last_active":"2023-10-16T15:53:42.409957+0000","last_peered":"2023-10-16T15:53:42.409957+0000","last_clean":"2023-10-16T15:53:42.409957+0000","last_became_active":"2023-10-16T15:53:41.399589+0000","last_became_peered":"2023-10-16T15:53:41.399589+0000","last_unstale":"2023-10-16T15:53:42.409957+0000","last_undegraded":"2023-10-16T15:53:42.409957+0000","last_fullsized":"2023-10-16T15:53:42.409957+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385445+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:41.385445+0000","last_peered":"2023-10-16T15:53:41.385445+0000","last_clean":"2023-10-16T15:53:41.385445+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:41.385445+0000","last_undegraded":"2023-10-16T15:53:41.385445+0000","last_fullsized":"2023-10-16T15:53:41.385445+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402749+0000","last_change":"2023-10-16T15:53:41.399919+0000","last_active":"2023-10-16T15:53:42.402749+0000","last_peered":"2023-10-16T15:53:42.402749+0000","last_clean":"2023-10-16T15:53:42.402749+0000","last_became_active":"2023-10-16T15:53:41.397770+0000","last_became_peered":"2023-10-16T15:53:41.397770+0000","last_unstale":"2023-10-16T15:53:42.402749+0000","last_undegraded":"2023-10-16T15:53:42.402749+0000","last_fullsized":"2023-10-16T15:53:42.402749+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377733+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:41.377733+0000","last_peered":"2023-10-16T15:53:41.377733+0000","last_clean":"2023-10-16T15:53:41.377733+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:41.377733+0000","last_undegraded":"2023-10-16T15:53:41.377733+0000","last_fullsized":"2023-10-16T15:53:41.377733+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014330+0000","last_change":"2023-10-16T15:53:41.401459+0000","last_active":"2023-10-16T15:53:43.014330+0000","last_peered":"2023-10-16T15:53:43.014330+0000","last_clean":"2023-10-16T15:53:43.014330+0000","last_became_active":"2023-10-16T15:53:41.401147+0000","last_became_peered":"2023-10-16T15:53:41.401147+0000","last_unstale":"2023-10-16T15:53:43.014330+0000","last_undegraded":"2023-10-16T15:53:43.014330+0000","last_fullsized":"2023-10-16T15:53:43.014330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:31.244497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.329048+0000","last_change":"2023-10-16T15:53:41.404413+0000","last_active":"2023-10-16T15:53:43.329048+0000","last_peered":"2023-10-16T15:53:43.329048+0000","last_clean":"2023-10-16T15:53:43.329048+0000","last_became_active":"2023-10-16T15:53:41.403862+0000","last_became_peered":"2023-10-16T15:53:41.403862+0000","last_unstale":"2023-10-16T15:53:43.329048+0000","last_undegraded":"2023-10-16T15:53:43.329048+0000","last_fullsized":"2023-10-16T15:53:43.329048+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:34.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392361+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:41.392361+0000","last_peered":"2023-10-16T15:53:41.392361+0000","last_clean":"2023-10-16T15:53:41.392361+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:41.392361+0000","last_undegraded":"2023-10-16T15:53:41.392361+0000","last_fullsized":"2023-10-16T15:53:41.392361+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407480+0000","last_change":"2023-10-16T15:53:41.406217+0000","last_active":"2023-10-16T15:53:42.407480+0000","last_peered":"2023-10-16T15:53:42.407480+0000","last_clean":"2023-10-16T15:53:42.407480+0000","last_became_active":"2023-10-16T15:53:41.405870+0000","last_became_peered":"2023-10-16T15:53:41.405870+0000","last_unstale":"2023-10-16T15:53:42.407480+0000","last_undegraded":"2023-10-16T15:53:42.407480+0000","last_fullsized":"2023-10-16T15:53:42.407480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374717+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:41.374717+0000","last_peered":"2023-10-16T15:53:41.374717+0000","last_clean":"2023-10-16T15:53:41.374717+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:41.374717+0000","last_undegraded":"2023-10-16T15:53:41.374717+0000","last_fullsized":"2023-10-16T15:53:41.374717+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576006,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":3400,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":3481600,"data_stored":2005127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":0.755},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":0.78000000000000003}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":0.80200000000000005}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.86699999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":0.82299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1412470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.216},"min":{"1min":0.73699999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7709999999999999,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.117},{"interface":"front","average":{"1min":1.22,"5min":1.276,"15min":1.2609999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.5489999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2949999999999999,"15min":1.2929999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.72199999999999998},"max":{"1min":1.8740000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3420000000000001,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.579,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":1.353}]},{"osd":2,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.244,"15min":1.2450000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.752},"max":{"1min":1.554,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.431,"15min":1.401},"min":{"1min":1.0680000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.1990000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.88100000000000001,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.46},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.331,"15min":1.339},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.631,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3560000000000001,"15min":1.331},"min":{"1min":0.871,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.325},{"interface":"front","average":{"1min":1.444,"5min":1.4430000000000001,"15min":1.407},"min":{"1min":0.90100000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.6319999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":0.88300000000000001,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.792,"5min":1.9419999999999999,"15min":2.016},"last":1.022},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5640000000000001,"15min":1.5289999999999999},"min":{"1min":1.0309999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.393}]},{"osd":7,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.633,"15min":1.6060000000000001},"min":{"1min":1.1950000000000001,"5min":0.627,"15min":0.627},"max":{"1min":2.149,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.6220000000000001,"15min":1.5980000000000001},"min":{"1min":1.264,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.2849999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29832,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342008,"statfs":{"total":96636764160,"available":96606216192,"internally_reserved":0,"allocated":2875392,"data_stored":1412414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1240000000000001,"15min":1.163},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6850000000000001,"5min":1.8069999999999999,"15min":1.821},"last":1.256},{"interface":"front","average":{"1min":1.026,"5min":1.2689999999999999,"15min":1.2430000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.53,"5min":2.738,"15min":2.738},"last":1.53}]},{"osd":1,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7669999999999999,"5min":1.7849999999999999,"15min":1.8680000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2629999999999999,"15min":1.302},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7270000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":0.86599999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.238,"15min":1.2529999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.577,"5min":1.784,"15min":1.9219999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.399,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7190000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.3260000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.343,"15min":1.3400000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9039999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.409},{"interface":"front","average":{"1min":1.169,"5min":1.3400000000000001,"15min":1.3799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.7130000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.1559999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.367,"15min":1.347},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.706},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.5069999999999999,"15min":1.5189999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.804,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.4810000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.385},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8660000000000001,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.669},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":1.431},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0230000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.585}]},{"osd":7,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.538,"15min":1.5369999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.847,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.639,"15min":1.633},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.962,"5min":3.109,"15min":3.109},"last":1.7430000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576006,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2895872,"data_stored":1415375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2330000000000001,"15min":1.212},"min":{"1min":0.873,"5min":0.77000000000000002,"15min":0.52700000000000002},"max":{"1min":1.794,"5min":1.883,"15min":1.883},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.034}]},{"osd":1,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2549999999999999,"15min":1.2430000000000001},"min":{"1min":0.95799999999999996,"5min":0.83499999999999996,"15min":0.629},"max":{"1min":1.675,"5min":1.929,"15min":2.2160000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.3160000000000001},"min":{"1min":0.92000000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0179999999999998,"5min":2.157,"15min":2.2989999999999999},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4490000000000001,"15min":1.4670000000000001},"min":{"1min":1.1160000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.498,"15min":1.4970000000000001},"min":{"1min":1.1870000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.3160000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":1.081,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.23,"5min":2.23,"15min":2.3650000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4099999999999999,"15min":1.401},"min":{"1min":1.0069999999999999,"5min":0.89200000000000002,"15min":0.69699999999999995},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.2310000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.375,"15min":1.3420000000000001},"min":{"1min":1.03,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.911,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.556},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.46,"15min":1.4370000000000001},"min":{"1min":0.91000000000000003,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.161,"5min":2.161,"15min":2.4489999999999998},"last":1.177}]},{"osd":6,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":0.97499999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.554,"15min":1.5269999999999999},"min":{"1min":1.0089999999999999,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.2789999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.6850000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.617,"15min":1.5860000000000001},"min":{"1min":1.2669999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.415},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.6339999999999999,"15min":1.6040000000000001},"min":{"1min":1.3999999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.242,"5min":2.2599999999999998,"15min":2.347},"last":1.589}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56228,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315612,"statfs":{"total":96636764160,"available":96579186688,"internally_reserved":0,"allocated":2899968,"data_stored":1413696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.44}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":0.92700000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":0.94399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1413500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":0.621}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.772}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":1.335},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.407}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.8049999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":2.093}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.8500000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.9930000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576006,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47884,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323956,"statfs":{"total":96636764160,"available":96587730944,"internally_reserved":0,"allocated":2883584,"data_stored":1411978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.236}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.4470000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":1.679},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.502}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.919}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":1.9810000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":1.867}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48468,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323372,"statfs":{"total":96636764160,"available":96587132928,"internally_reserved":0,"allocated":3477504,"data_stored":2002962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.85199999999999998}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":1.1850000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":1.498}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.288},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4650000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.363},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":1.552}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:51.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:53:51.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.381+0000 7f3e8dcc3640 1 Processor -- start 2023-10-16T15:53:51.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.381+0000 7f3e8dcc3640 1 -- start start 2023-10-16T15:53:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e8dcc3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e8815dde0 0x7f3e880709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e8dcc3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e88160a90 con 0x7f3e8815c9f0 2023-10-16T15:53:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e8dcc3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880cd900 con 0x7f3e88070f20 2023-10-16T15:53:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e8dcc3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e88071350 con 0x7f3e8815dde0 2023-10-16T15:53:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e7ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e8815dde0 0x7f3e880709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e7ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e8815dde0 0x7f3e880709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45896/0 (socket says 172.21.15.29:45896) 2023-10-16T15:53:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.383+0000 7f3e7ffff640 1 -- 172.21.15.29:0/1149525070 learned_addr learned my addr 172.21.15.29:0/1149525070 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.384+0000 7f3e7ffff640 1 -- 172.21.15.29:0/1149525070 >> v1:172.21.15.170:6789/0 conn(0x7f3e88070f20 legacy=0x7f3e880ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.384+0000 7f3e7ffff640 1 -- 172.21.15.29:0/1149525070 >> v1:172.21.15.29:6789/0 conn(0x7f3e8815c9f0 legacy=0x7f3e880c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.384+0000 7f3e7ffff640 1 -- 172.21.15.29:0/1149525070 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880cdbe0 con 0x7f3e8815dde0 2023-10-16T15:53:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.384+0000 7f3e7f7fe640 1 -- 172.21.15.29:0/1149525070 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440812931 0 0) 0x7f3e88160a90 con 0x7f3e8815c9f0 2023-10-16T15:53:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.385+0000 7f3e7ffff640 1 --2- 172.21.15.29:0/1149525070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880709e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009870 tx=0x7f3e7802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2668df154478e5c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.385+0000 7f3e7f7fe640 1 -- 172.21.15.29:0/1149525070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78030860 con 0x7f3e8815dde0 2023-10-16T15:53:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e7f7fe640 1 -- 172.21.15.29:0/1149525070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e780309c0 con 0x7f3e8815dde0 2023-10-16T15:53:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e7f7fe640 1 -- 172.21.15.29:0/1149525070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78030d50 con 0x7f3e8815dde0 2023-10-16T15:53:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/1149525070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 msgr2=0x7f3e880709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/1149525070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880709e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009870 tx=0x7f3e7802b9a0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/1149525070 shutdown_connections 2023-10-16T15:53:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/1149525070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880709e0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/1149525070 >> 172.21.15.29:0/1149525070 conn(0x7f3e880ce6a0 msgr2=0x7f3e880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/1149525070 shutdown_connections 2023-10-16T15:53:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.386+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/1149525070 wait complete. 2023-10-16T15:53:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.388+0000 7f3e8dcc3640 1 Processor -- start 2023-10-16T15:53:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.388+0000 7f3e8dcc3640 1 -- start start 2023-10-16T15:53:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8dcc3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e88070f20 0x7f3e88150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8dcc3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8815c9f0 0x7f3e880c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8dcc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8dcc3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c29c0 con 0x7f3e8815dde0 2023-10-16T15:53:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8dcc3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c3260 con 0x7f3e88070f20 2023-10-16T15:53:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8dcc3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8806ac60 con 0x7f3e8815c9f0 2023-10-16T15:53:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8d4c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8d4c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45902/0 (socket says 172.21.15.29:45902) 2023-10-16T15:53:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.389+0000 7f3e8d4c2640 1 -- 172.21.15.29:0/2384977123 learned_addr learned my addr 172.21.15.29:0/2384977123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e7ffff640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8815c9f0 0x7f3e880c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8ccc1640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e88070f20 0x7f3e88150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8d4c2640 1 -- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8815c9f0 msgr2=0x7f3e880c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8d4c2640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8815c9f0 0x7f3e880c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8d4c2640 1 -- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e88070f20 msgr2=0x7f3e88150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8d4c2640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e88070f20 0x7f3e88150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8d4c2640 1 -- 172.21.15.29:0/2384977123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e780094d0 con 0x7f3e8815dde0 2023-10-16T15:53:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8ccc1640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e88070f20 0x7f3e88150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.390+0000 7f3e8d4c2640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880cc4a0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000b410 tx=0x7f3e8000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.391+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80004020 con 0x7f3e8815dde0 2023-10-16T15:53:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.391+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e80004180 con 0x7f3e8815dde0 2023-10-16T15:53:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.391+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80011630 con 0x7f3e8815dde0 2023-10-16T15:53:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.391+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e880c2c10 con 0x7f3e8815dde0 2023-10-16T15:53:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.391+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e880cc9e0 con 0x7f3e8815dde0 2023-10-16T15:53:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.392+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8804e9b0 con 0x7f3e8815dde0 2023-10-16T15:53:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.393+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e80011790 con 0x7f3e8815dde0 2023-10-16T15:53:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.394+0000 7f3e7d7fa640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3e7406af90 0x7f3e7406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.394+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f3e800898c0 con 0x7f3e8815dde0 2023-10-16T15:53:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.395+0000 7f3e8ccc1640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3e7406af90 0x7f3e7406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.396+0000 7f3e8ccc1640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3e7406af90 0x7f3e7406d450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e840012e0 tx=0x7f3e84009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.398+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e80054440 con 0x7f3e8815dde0 2023-10-16T15:53:51.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.588+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e880704c0 con 0x7f3e7406af90 2023-10-16T15:53:51.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.628+0000 7f3e7d7fa640 1 -- 172.21.15.29:0/2384977123 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525873 (secure 0 0 0) 0x7f3e84003990 con 0x7f3e7406af90 2023-10-16T15:53:51.633 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:51.641 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:53:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.640+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3e7406af90 msgr2=0x7f3e7406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.640+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3e7406af90 0x7f3e7406d450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e840012e0 tx=0x7f3e84009040 comp rx=0 tx=0).stop 2023-10-16T15:53:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.640+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 msgr2=0x7f3e880cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.640+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880cc4a0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000b410 tx=0x7f3e8000b8d0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.640+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 shutdown_connections 2023-10-16T15:53:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.640+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3e7406af90 0x7f3e7406d450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.641+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e8815dde0 0x7f3e880cc4a0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.641+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3e8815c9f0 0x7f3e880c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.641+0000 7f3e8dcc3640 1 --2- 172.21.15.29:0/2384977123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3e88070f20 0x7f3e88150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.641+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 >> 172.21.15.29:0/2384977123 conn(0x7f3e880ce6a0 msgr2=0x7f3e8815ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.641+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 shutdown_connections 2023-10-16T15:53:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.641+0000 7f3e8dcc3640 1 -- 172.21.15.29:0/2384977123 wait complete. 2023-10-16T15:53:51.669 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-10-16T15:53:50.465263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":675,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":431,"ondisk_log_size":431,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381484,"kb_used_data":23720,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593236,"statfs":{"total":773094113280,"available":772703473664,"internally_reserved":0,"allocated":24289280,"data_stored":12487522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366202425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.092772"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411190+0000","last_change":"2023-10-16T15:53:41.399707+0000","last_active":"2023-10-16T15:53:42.411190+0000","last_peered":"2023-10-16T15:53:42.411190+0000","last_clean":"2023-10-16T15:53:42.411190+0000","last_became_active":"2023-10-16T15:53:41.399411+0000","last_became_peered":"2023-10-16T15:53:41.399411+0000","last_unstale":"2023-10-16T15:53:42.411190+0000","last_undegraded":"2023-10-16T15:53:42.411190+0000","last_fullsized":"2023-10-16T15:53:42.411190+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:45.368014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"213'351","reported_seq":1251,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376480+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:53:41.376480+0000","last_peered":"2023-10-16T15:53:41.376480+0000","last_clean":"2023-10-16T15:53:41.376480+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:53:41.376480+0000","last_undegraded":"2023-10-16T15:53:41.376480+0000","last_fullsized":"2023-10-16T15:53:41.376480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396662+0000","last_change":"2023-10-16T15:53:26.207851+0000","last_active":"2023-10-16T15:53:41.396662+0000","last_peered":"2023-10-16T15:53:41.396662+0000","last_clean":"2023-10-16T15:53:41.396662+0000","last_became_active":"2023-10-16T15:53:26.207545+0000","last_became_peered":"2023-10-16T15:53:26.207545+0000","last_unstale":"2023-10-16T15:53:41.396662+0000","last_undegraded":"2023-10-16T15:53:41.396662+0000","last_fullsized":"2023-10-16T15:53:41.396662+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:20.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375906+0000","last_change":"2023-10-16T15:53:27.254575+0000","last_active":"2023-10-16T15:53:41.375906+0000","last_peered":"2023-10-16T15:53:41.375906+0000","last_clean":"2023-10-16T15:53:41.375906+0000","last_became_active":"2023-10-16T15:53:27.254277+0000","last_became_peered":"2023-10-16T15:53:27.254277+0000","last_unstale":"2023-10-16T15:53:41.375906+0000","last_undegraded":"2023-10-16T15:53:41.375906+0000","last_fullsized":"2023-10-16T15:53:41.375906+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.410846+0000","last_change":"2023-10-16T15:53:40.387170+0000","last_active":"2023-10-16T15:53:41.410846+0000","last_peered":"2023-10-16T15:53:41.410846+0000","last_clean":"2023-10-16T15:53:41.410846+0000","last_became_active":"2023-10-16T15:53:40.386872+0000","last_became_peered":"2023-10-16T15:53:40.386872+0000","last_unstale":"2023-10-16T15:53:41.410846+0000","last_undegraded":"2023-10-16T15:53:41.410846+0000","last_fullsized":"2023-10-16T15:53:41.410846+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411355+0000","last_change":"2023-10-16T15:53:41.413779+0000","last_active":"2023-10-16T15:53:42.411355+0000","last_peered":"2023-10-16T15:53:42.411355+0000","last_clean":"2023-10-16T15:53:42.411355+0000","last_became_active":"2023-10-16T15:53:41.413379+0000","last_became_peered":"2023-10-16T15:53:41.413379+0000","last_unstale":"2023-10-16T15:53:42.411355+0000","last_undegraded":"2023-10-16T15:53:42.411355+0000","last_fullsized":"2023-10-16T15:53:42.411355+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384903+0000","last_change":"2023-10-16T15:53:26.204180+0000","last_active":"2023-10-16T15:53:41.384903+0000","last_peered":"2023-10-16T15:53:41.384903+0000","last_clean":"2023-10-16T15:53:41.384903+0000","last_became_active":"2023-10-16T15:53:26.203898+0000","last_became_peered":"2023-10-16T15:53:26.203898+0000","last_unstale":"2023-10-16T15:53:41.384903+0000","last_undegraded":"2023-10-16T15:53:41.384903+0000","last_fullsized":"2023-10-16T15:53:41.384903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.232392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386164+0000","last_change":"2023-10-16T15:53:27.241017+0000","last_active":"2023-10-16T15:53:41.386164+0000","last_peered":"2023-10-16T15:53:41.386164+0000","last_clean":"2023-10-16T15:53:41.386164+0000","last_became_active":"2023-10-16T15:53:27.240723+0000","last_became_peered":"2023-10-16T15:53:27.240723+0000","last_unstale":"2023-10-16T15:53:41.386164+0000","last_undegraded":"2023-10-16T15:53:41.386164+0000","last_fullsized":"2023-10-16T15:53:41.386164+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:04.052813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409347+0000","last_change":"2023-10-16T15:53:40.378024+0000","last_active":"2023-10-16T15:53:42.409347+0000","last_peered":"2023-10-16T15:53:42.409347+0000","last_clean":"2023-10-16T15:53:42.409347+0000","last_became_active":"2023-10-16T15:53:40.377757+0000","last_became_peered":"2023-10-16T15:53:40.377757+0000","last_unstale":"2023-10-16T15:53:42.409347+0000","last_undegraded":"2023-10-16T15:53:42.409347+0000","last_fullsized":"2023-10-16T15:53:42.409347+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:48.714124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014672+0000","last_change":"2023-10-16T15:53:41.418345+0000","last_active":"2023-10-16T15:53:43.014672+0000","last_peered":"2023-10-16T15:53:43.014672+0000","last_clean":"2023-10-16T15:53:43.014672+0000","last_became_active":"2023-10-16T15:53:41.418078+0000","last_became_peered":"2023-10-16T15:53:41.418078+0000","last_unstale":"2023-10-16T15:53:43.014672+0000","last_undegraded":"2023-10-16T15:53:43.014672+0000","last_fullsized":"2023-10-16T15:53:43.014672+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:25.632327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389543+0000","last_change":"2023-10-16T15:53:26.204117+0000","last_active":"2023-10-16T15:53:41.389543+0000","last_peered":"2023-10-16T15:53:41.389543+0000","last_clean":"2023-10-16T15:53:41.389543+0000","last_became_active":"2023-10-16T15:53:26.203768+0000","last_became_peered":"2023-10-16T15:53:26.203768+0000","last_unstale":"2023-10-16T15:53:41.389543+0000","last_undegraded":"2023-10-16T15:53:41.389543+0000","last_fullsized":"2023-10-16T15:53:41.389543+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:47.971932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380422+0000","last_change":"2023-10-16T15:53:27.247350+0000","last_active":"2023-10-16T15:53:41.380422+0000","last_peered":"2023-10-16T15:53:41.380422+0000","last_clean":"2023-10-16T15:53:41.380422+0000","last_became_active":"2023-10-16T15:53:27.244211+0000","last_became_peered":"2023-10-16T15:53:27.244211+0000","last_unstale":"2023-10-16T15:53:41.380422+0000","last_undegraded":"2023-10-16T15:53:41.380422+0000","last_fullsized":"2023-10-16T15:53:41.380422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.405443+0000","last_change":"2023-10-16T15:53:40.384327+0000","last_active":"2023-10-16T15:53:41.405443+0000","last_peered":"2023-10-16T15:53:41.405443+0000","last_clean":"2023-10-16T15:53:41.405443+0000","last_became_active":"2023-10-16T15:53:40.384027+0000","last_became_peered":"2023-10-16T15:53:40.384027+0000","last_unstale":"2023-10-16T15:53:41.405443+0000","last_undegraded":"2023-10-16T15:53:41.405443+0000","last_fullsized":"2023-10-16T15:53:41.405443+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:09.776405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014626+0000","last_change":"2023-10-16T15:53:41.393836+0000","last_active":"2023-10-16T15:53:43.014626+0000","last_peered":"2023-10-16T15:53:43.014626+0000","last_clean":"2023-10-16T15:53:43.014626+0000","last_became_active":"2023-10-16T15:53:41.392583+0000","last_became_peered":"2023-10-16T15:53:41.392583+0000","last_unstale":"2023-10-16T15:53:43.014626+0000","last_undegraded":"2023-10-16T15:53:43.014626+0000","last_fullsized":"2023-10-16T15:53:43.014626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:22.777642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384099+0000","last_change":"2023-10-16T15:53:26.210857+0000","last_active":"2023-10-16T15:53:41.384099+0000","last_peered":"2023-10-16T15:53:41.384099+0000","last_clean":"2023-10-16T15:53:41.384099+0000","last_became_active":"2023-10-16T15:53:26.210301+0000","last_became_peered":"2023-10-16T15:53:26.210301+0000","last_unstale":"2023-10-16T15:53:41.384099+0000","last_undegraded":"2023-10-16T15:53:41.384099+0000","last_fullsized":"2023-10-16T15:53:41.384099+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396904+0000","last_change":"2023-10-16T15:53:27.240210+0000","last_active":"2023-10-16T15:53:41.396904+0000","last_peered":"2023-10-16T15:53:41.396904+0000","last_clean":"2023-10-16T15:53:41.396904+0000","last_became_active":"2023-10-16T15:53:27.239935+0000","last_became_peered":"2023-10-16T15:53:27.239935+0000","last_unstale":"2023-10-16T15:53:41.396904+0000","last_undegraded":"2023-10-16T15:53:41.396904+0000","last_fullsized":"2023-10-16T15:53:41.396904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:34.984992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395708+0000","last_change":"2023-10-16T15:53:40.378111+0000","last_active":"2023-10-16T15:53:41.395708+0000","last_peered":"2023-10-16T15:53:41.395708+0000","last_clean":"2023-10-16T15:53:41.395708+0000","last_became_active":"2023-10-16T15:53:40.377470+0000","last_became_peered":"2023-10-16T15:53:40.377470+0000","last_unstale":"2023-10-16T15:53:41.395708+0000","last_undegraded":"2023-10-16T15:53:41.395708+0000","last_fullsized":"2023-10-16T15:53:41.395708+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408902+0000","last_change":"2023-10-16T15:53:41.409618+0000","last_active":"2023-10-16T15:53:42.408902+0000","last_peered":"2023-10-16T15:53:42.408902+0000","last_clean":"2023-10-16T15:53:42.408902+0000","last_became_active":"2023-10-16T15:53:41.409365+0000","last_became_peered":"2023-10-16T15:53:41.409365+0000","last_unstale":"2023-10-16T15:53:42.408902+0000","last_undegraded":"2023-10-16T15:53:42.408902+0000","last_fullsized":"2023-10-16T15:53:42.408902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378147+0000","last_change":"2023-10-16T15:53:26.207100+0000","last_active":"2023-10-16T15:53:41.378147+0000","last_peered":"2023-10-16T15:53:41.378147+0000","last_clean":"2023-10-16T15:53:41.378147+0000","last_became_active":"2023-10-16T15:53:26.206792+0000","last_became_peered":"2023-10-16T15:53:26.206792+0000","last_unstale":"2023-10-16T15:53:41.378147+0000","last_undegraded":"2023-10-16T15:53:41.378147+0000","last_fullsized":"2023-10-16T15:53:41.378147+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:58.423183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.243193+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.236156+0000","last_became_peered":"2023-10-16T15:53:27.236156+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383981+0000","last_change":"2023-10-16T15:53:40.381668+0000","last_active":"2023-10-16T15:53:41.383981+0000","last_peered":"2023-10-16T15:53:41.383981+0000","last_clean":"2023-10-16T15:53:41.383981+0000","last_became_active":"2023-10-16T15:53:40.381369+0000","last_became_peered":"2023-10-16T15:53:40.381369+0000","last_unstale":"2023-10-16T15:53:41.383981+0000","last_undegraded":"2023-10-16T15:53:41.383981+0000","last_fullsized":"2023-10-16T15:53:41.383981+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:32.114344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410745+0000","last_change":"2023-10-16T15:53:41.400067+0000","last_active":"2023-10-16T15:53:42.410745+0000","last_peered":"2023-10-16T15:53:42.410745+0000","last_clean":"2023-10-16T15:53:42.410745+0000","last_became_active":"2023-10-16T15:53:41.399778+0000","last_became_peered":"2023-10-16T15:53:41.399778+0000","last_unstale":"2023-10-16T15:53:42.410745+0000","last_undegraded":"2023-10-16T15:53:42.410745+0000","last_fullsized":"2023-10-16T15:53:42.410745+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:18.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384461+0000","last_change":"2023-10-16T15:53:26.207900+0000","last_active":"2023-10-16T15:53:41.384461+0000","last_peered":"2023-10-16T15:53:41.384461+0000","last_clean":"2023-10-16T15:53:41.384461+0000","last_became_active":"2023-10-16T15:53:26.207420+0000","last_became_peered":"2023-10-16T15:53:26.207420+0000","last_unstale":"2023-10-16T15:53:41.384461+0000","last_undegraded":"2023-10-16T15:53:41.384461+0000","last_fullsized":"2023-10-16T15:53:41.384461+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:05.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.397959+0000","last_change":"2023-10-16T15:53:27.247231+0000","last_active":"2023-10-16T15:53:41.397959+0000","last_peered":"2023-10-16T15:53:41.397959+0000","last_clean":"2023-10-16T15:53:41.397959+0000","last_became_active":"2023-10-16T15:53:27.246355+0000","last_became_peered":"2023-10-16T15:53:27.246355+0000","last_unstale":"2023-10-16T15:53:41.397959+0000","last_undegraded":"2023-10-16T15:53:41.397959+0000","last_fullsized":"2023-10-16T15:53:41.397959+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:40.269921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374462+0000","last_change":"2023-10-16T15:53:40.375759+0000","last_active":"2023-10-16T15:53:41.374462+0000","last_peered":"2023-10-16T15:53:41.374462+0000","last_clean":"2023-10-16T15:53:41.374462+0000","last_became_active":"2023-10-16T15:53:40.375464+0000","last_became_peered":"2023-10-16T15:53:40.375464+0000","last_unstale":"2023-10-16T15:53:41.374462+0000","last_undegraded":"2023-10-16T15:53:41.374462+0000","last_fullsized":"2023-10-16T15:53:41.374462+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405804+0000","last_change":"2023-10-16T15:53:41.407972+0000","last_active":"2023-10-16T15:53:42.405804+0000","last_peered":"2023-10-16T15:53:42.405804+0000","last_clean":"2023-10-16T15:53:42.405804+0000","last_became_active":"2023-10-16T15:53:41.407725+0000","last_became_peered":"2023-10-16T15:53:41.407725+0000","last_unstale":"2023-10-16T15:53:42.405804+0000","last_undegraded":"2023-10-16T15:53:42.405804+0000","last_fullsized":"2023-10-16T15:53:42.405804+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:45.480076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385405+0000","last_change":"2023-10-16T15:53:26.214561+0000","last_active":"2023-10-16T15:53:41.385405+0000","last_peered":"2023-10-16T15:53:41.385405+0000","last_clean":"2023-10-16T15:53:41.385405+0000","last_became_active":"2023-10-16T15:53:26.213607+0000","last_became_peered":"2023-10-16T15:53:26.213607+0000","last_unstale":"2023-10-16T15:53:41.385405+0000","last_undegraded":"2023-10-16T15:53:41.385405+0000","last_fullsized":"2023-10-16T15:53:41.385405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:35.420368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386135+0000","last_change":"2023-10-16T15:53:27.243427+0000","last_active":"2023-10-16T15:53:41.386135+0000","last_peered":"2023-10-16T15:53:41.386135+0000","last_clean":"2023-10-16T15:53:41.386135+0000","last_became_active":"2023-10-16T15:53:27.243085+0000","last_became_peered":"2023-10-16T15:53:27.243085+0000","last_unstale":"2023-10-16T15:53:41.386135+0000","last_undegraded":"2023-10-16T15:53:41.386135+0000","last_fullsized":"2023-10-16T15:53:41.386135+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385992+0000","last_change":"2023-10-16T15:53:40.389163+0000","last_active":"2023-10-16T15:53:41.385992+0000","last_peered":"2023-10-16T15:53:41.385992+0000","last_clean":"2023-10-16T15:53:41.385992+0000","last_became_active":"2023-10-16T15:53:40.388144+0000","last_became_peered":"2023-10-16T15:53:40.388144+0000","last_unstale":"2023-10-16T15:53:41.385992+0000","last_undegraded":"2023-10-16T15:53:41.385992+0000","last_fullsized":"2023-10-16T15:53:41.385992+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406367+0000","last_change":"2023-10-16T15:53:41.408615+0000","last_active":"2023-10-16T15:53:42.406367+0000","last_peered":"2023-10-16T15:53:42.406367+0000","last_clean":"2023-10-16T15:53:42.406367+0000","last_became_active":"2023-10-16T15:53:41.408337+0000","last_became_peered":"2023-10-16T15:53:41.408337+0000","last_unstale":"2023-10-16T15:53:42.406367+0000","last_undegraded":"2023-10-16T15:53:42.406367+0000","last_fullsized":"2023-10-16T15:53:42.406367+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:20.232917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"225'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379246+0000","last_change":"2023-10-16T15:53:26.203420+0000","last_active":"2023-10-16T15:53:41.379246+0000","last_peered":"2023-10-16T15:53:41.379246+0000","last_clean":"2023-10-16T15:53:41.379246+0000","last_became_active":"2023-10-16T15:53:26.203112+0000","last_became_peered":"2023-10-16T15:53:26.203112+0000","last_unstale":"2023-10-16T15:53:41.379246+0000","last_undegraded":"2023-10-16T15:53:41.379246+0000","last_fullsized":"2023-10-16T15:53:41.379246+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:07.587942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376019+0000","last_change":"2023-10-16T15:53:27.246485+0000","last_active":"2023-10-16T15:53:41.376019+0000","last_peered":"2023-10-16T15:53:41.376019+0000","last_clean":"2023-10-16T15:53:41.376019+0000","last_became_active":"2023-10-16T15:53:27.244692+0000","last_became_peered":"2023-10-16T15:53:27.244692+0000","last_unstale":"2023-10-16T15:53:41.376019+0000","last_undegraded":"2023-10-16T15:53:41.376019+0000","last_fullsized":"2023-10-16T15:53:41.376019+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384628+0000","last_change":"2023-10-16T15:53:40.375284+0000","last_active":"2023-10-16T15:53:41.384628+0000","last_peered":"2023-10-16T15:53:41.384628+0000","last_clean":"2023-10-16T15:53:41.384628+0000","last_became_active":"2023-10-16T15:53:40.374509+0000","last_became_peered":"2023-10-16T15:53:40.374509+0000","last_unstale":"2023-10-16T15:53:41.384628+0000","last_undegraded":"2023-10-16T15:53:41.384628+0000","last_fullsized":"2023-10-16T15:53:41.384628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:59.597435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407164+0000","last_change":"2023-10-16T15:53:41.414757+0000","last_active":"2023-10-16T15:53:42.407164+0000","last_peered":"2023-10-16T15:53:42.407164+0000","last_clean":"2023-10-16T15:53:42.407164+0000","last_became_active":"2023-10-16T15:53:41.414480+0000","last_became_peered":"2023-10-16T15:53:41.414480+0000","last_unstale":"2023-10-16T15:53:42.407164+0000","last_undegraded":"2023-10-16T15:53:42.407164+0000","last_fullsized":"2023-10-16T15:53:42.407164+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387180+0000","last_change":"2023-10-16T15:53:26.212583+0000","last_active":"2023-10-16T15:53:41.387180+0000","last_peered":"2023-10-16T15:53:41.387180+0000","last_clean":"2023-10-16T15:53:41.387180+0000","last_became_active":"2023-10-16T15:53:26.211394+0000","last_became_peered":"2023-10-16T15:53:26.211394+0000","last_unstale":"2023-10-16T15:53:41.387180+0000","last_undegraded":"2023-10-16T15:53:41.387180+0000","last_fullsized":"2023-10-16T15:53:41.387180+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:48.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395410+0000","last_change":"2023-10-16T15:53:27.227878+0000","last_active":"2023-10-16T15:53:41.395410+0000","last_peered":"2023-10-16T15:53:41.395410+0000","last_clean":"2023-10-16T15:53:41.395410+0000","last_became_active":"2023-10-16T15:53:27.227343+0000","last_became_peered":"2023-10-16T15:53:27.227343+0000","last_unstale":"2023-10-16T15:53:41.395410+0000","last_undegraded":"2023-10-16T15:53:41.395410+0000","last_fullsized":"2023-10-16T15:53:41.395410+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:27.660085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382520+0000","last_change":"2023-10-16T15:53:40.381870+0000","last_active":"2023-10-16T15:53:41.382520+0000","last_peered":"2023-10-16T15:53:41.382520+0000","last_clean":"2023-10-16T15:53:41.382520+0000","last_became_active":"2023-10-16T15:53:40.381577+0000","last_became_peered":"2023-10-16T15:53:40.381577+0000","last_unstale":"2023-10-16T15:53:41.382520+0000","last_undegraded":"2023-10-16T15:53:41.382520+0000","last_fullsized":"2023-10-16T15:53:41.382520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:33.030599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404650+0000","last_change":"2023-10-16T15:53:41.408405+0000","last_active":"2023-10-16T15:53:42.404650+0000","last_peered":"2023-10-16T15:53:42.404650+0000","last_clean":"2023-10-16T15:53:42.404650+0000","last_became_active":"2023-10-16T15:53:41.407251+0000","last_became_peered":"2023-10-16T15:53:41.407251+0000","last_unstale":"2023-10-16T15:53:42.404650+0000","last_undegraded":"2023-10-16T15:53:42.404650+0000","last_fullsized":"2023-10-16T15:53:42.404650+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.381328+0000","last_change":"2023-10-16T15:53:26.206541+0000","last_active":"2023-10-16T15:53:41.381328+0000","last_peered":"2023-10-16T15:53:41.381328+0000","last_clean":"2023-10-16T15:53:41.381328+0000","last_became_active":"2023-10-16T15:53:26.205820+0000","last_became_peered":"2023-10-16T15:53:26.205820+0000","last_unstale":"2023-10-16T15:53:41.381328+0000","last_undegraded":"2023-10-16T15:53:41.381328+0000","last_fullsized":"2023-10-16T15:53:41.381328+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:40.752744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389408+0000","last_change":"2023-10-16T15:53:27.238532+0000","last_active":"2023-10-16T15:53:41.389408+0000","last_peered":"2023-10-16T15:53:41.389408+0000","last_clean":"2023-10-16T15:53:41.389408+0000","last_became_active":"2023-10-16T15:53:27.237372+0000","last_became_peered":"2023-10-16T15:53:27.237372+0000","last_unstale":"2023-10-16T15:53:41.389408+0000","last_undegraded":"2023-10-16T15:53:41.389408+0000","last_fullsized":"2023-10-16T15:53:41.389408+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:56.043594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414970+0000","last_change":"2023-10-16T15:53:40.373067+0000","last_active":"2023-10-16T15:53:41.414970+0000","last_peered":"2023-10-16T15:53:41.414970+0000","last_clean":"2023-10-16T15:53:41.414970+0000","last_became_active":"2023-10-16T15:53:40.372307+0000","last_became_peered":"2023-10-16T15:53:40.372307+0000","last_unstale":"2023-10-16T15:53:41.414970+0000","last_undegraded":"2023-10-16T15:53:41.414970+0000","last_fullsized":"2023-10-16T15:53:41.414970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:54.928376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330512+0000","last_change":"2023-10-16T15:53:41.400515+0000","last_active":"2023-10-16T15:53:43.330512+0000","last_peered":"2023-10-16T15:53:43.330512+0000","last_clean":"2023-10-16T15:53:43.330512+0000","last_became_active":"2023-10-16T15:53:41.399835+0000","last_became_peered":"2023-10-16T15:53:41.399835+0000","last_unstale":"2023-10-16T15:53:43.330512+0000","last_undegraded":"2023-10-16T15:53:43.330512+0000","last_fullsized":"2023-10-16T15:53:43.330512+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:40.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375870+0000","last_change":"2023-10-16T15:53:26.204095+0000","last_active":"2023-10-16T15:53:41.375870+0000","last_peered":"2023-10-16T15:53:41.375870+0000","last_clean":"2023-10-16T15:53:41.375870+0000","last_became_active":"2023-10-16T15:53:26.203770+0000","last_became_peered":"2023-10-16T15:53:26.203770+0000","last_unstale":"2023-10-16T15:53:41.375870+0000","last_undegraded":"2023-10-16T15:53:41.375870+0000","last_fullsized":"2023-10-16T15:53:41.375870+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:15.534574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384670+0000","last_change":"2023-10-16T15:53:27.240236+0000","last_active":"2023-10-16T15:53:41.384670+0000","last_peered":"2023-10-16T15:53:41.384670+0000","last_clean":"2023-10-16T15:53:41.384670+0000","last_became_active":"2023-10-16T15:53:27.238489+0000","last_became_peered":"2023-10-16T15:53:27.238489+0000","last_unstale":"2023-10-16T15:53:41.384670+0000","last_undegraded":"2023-10-16T15:53:41.384670+0000","last_fullsized":"2023-10-16T15:53:41.384670+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.407785+0000","last_change":"2023-10-16T15:53:40.378428+0000","last_active":"2023-10-16T15:53:41.407785+0000","last_peered":"2023-10-16T15:53:41.407785+0000","last_clean":"2023-10-16T15:53:41.407785+0000","last_became_active":"2023-10-16T15:53:40.378077+0000","last_became_peered":"2023-10-16T15:53:40.378077+0000","last_unstale":"2023-10-16T15:53:41.407785+0000","last_undegraded":"2023-10-16T15:53:41.407785+0000","last_fullsized":"2023-10-16T15:53:41.407785+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410289+0000","last_change":"2023-10-16T15:53:41.399412+0000","last_active":"2023-10-16T15:53:42.410289+0000","last_peered":"2023-10-16T15:53:42.410289+0000","last_clean":"2023-10-16T15:53:42.410289+0000","last_became_active":"2023-10-16T15:53:41.399125+0000","last_became_peered":"2023-10-16T15:53:41.399125+0000","last_unstale":"2023-10-16T15:53:42.410289+0000","last_undegraded":"2023-10-16T15:53:42.410289+0000","last_fullsized":"2023-10-16T15:53:42.410289+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376676+0000","last_change":"2023-10-16T15:53:26.205880+0000","last_active":"2023-10-16T15:53:41.376676+0000","last_peered":"2023-10-16T15:53:41.376676+0000","last_clean":"2023-10-16T15:53:41.376676+0000","last_became_active":"2023-10-16T15:53:26.205098+0000","last_became_peered":"2023-10-16T15:53:26.205098+0000","last_unstale":"2023-10-16T15:53:41.376676+0000","last_undegraded":"2023-10-16T15:53:41.376676+0000","last_fullsized":"2023-10-16T15:53:41.376676+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:12:33.970455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.247908+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.247675+0000","last_became_peered":"2023-10-16T15:53:27.247675+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:08.481130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394140+0000","last_change":"2023-10-16T15:53:40.377426+0000","last_active":"2023-10-16T15:53:41.394140+0000","last_peered":"2023-10-16T15:53:41.394140+0000","last_clean":"2023-10-16T15:53:41.394140+0000","last_became_active":"2023-10-16T15:53:40.377160+0000","last_became_peered":"2023-10-16T15:53:40.377160+0000","last_unstale":"2023-10-16T15:53:41.394140+0000","last_undegraded":"2023-10-16T15:53:41.394140+0000","last_fullsized":"2023-10-16T15:53:41.394140+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:38.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408560+0000","last_change":"2023-10-16T15:53:41.400288+0000","last_active":"2023-10-16T15:53:42.408560+0000","last_peered":"2023-10-16T15:53:42.408560+0000","last_clean":"2023-10-16T15:53:42.408560+0000","last_became_active":"2023-10-16T15:53:41.400031+0000","last_became_peered":"2023-10-16T15:53:41.400031+0000","last_unstale":"2023-10-16T15:53:42.408560+0000","last_undegraded":"2023-10-16T15:53:42.408560+0000","last_fullsized":"2023-10-16T15:53:42.408560+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393978+0000","last_change":"2023-10-16T15:53:26.206408+0000","last_active":"2023-10-16T15:53:41.393978+0000","last_peered":"2023-10-16T15:53:41.393978+0000","last_clean":"2023-10-16T15:53:41.393978+0000","last_became_active":"2023-10-16T15:53:26.205792+0000","last_became_peered":"2023-10-16T15:53:26.205792+0000","last_unstale":"2023-10-16T15:53:41.393978+0000","last_undegraded":"2023-10-16T15:53:41.393978+0000","last_fullsized":"2023-10-16T15:53:41.393978+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:33.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392991+0000","last_change":"2023-10-16T15:53:27.233581+0000","last_active":"2023-10-16T15:53:41.392991+0000","last_peered":"2023-10-16T15:53:41.392991+0000","last_clean":"2023-10-16T15:53:41.392991+0000","last_became_active":"2023-10-16T15:53:27.233278+0000","last_became_peered":"2023-10-16T15:53:27.233278+0000","last_unstale":"2023-10-16T15:53:41.392991+0000","last_undegraded":"2023-10-16T15:53:41.392991+0000","last_fullsized":"2023-10-16T15:53:41.392991+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:28.578095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414694+0000","last_change":"2023-10-16T15:53:40.379991+0000","last_active":"2023-10-16T15:53:41.414694+0000","last_peered":"2023-10-16T15:53:41.414694+0000","last_clean":"2023-10-16T15:53:41.414694+0000","last_became_active":"2023-10-16T15:53:40.379687+0000","last_became_peered":"2023-10-16T15:53:40.379687+0000","last_unstale":"2023-10-16T15:53:41.414694+0000","last_undegraded":"2023-10-16T15:53:41.414694+0000","last_fullsized":"2023-10-16T15:53:41.414694+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:39.813921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409916+0000","last_change":"2023-10-16T15:53:41.413217+0000","last_active":"2023-10-16T15:53:42.409916+0000","last_peered":"2023-10-16T15:53:42.409916+0000","last_clean":"2023-10-16T15:53:42.409916+0000","last_became_active":"2023-10-16T15:53:41.411878+0000","last_became_peered":"2023-10-16T15:53:41.411878+0000","last_unstale":"2023-10-16T15:53:42.409916+0000","last_undegraded":"2023-10-16T15:53:42.409916+0000","last_fullsized":"2023-10-16T15:53:42.409916+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384277+0000","last_change":"2023-10-16T15:53:26.211025+0000","last_active":"2023-10-16T15:53:41.384277+0000","last_peered":"2023-10-16T15:53:41.384277+0000","last_clean":"2023-10-16T15:53:41.384277+0000","last_became_active":"2023-10-16T15:53:26.210567+0000","last_became_peered":"2023-10-16T15:53:26.210567+0000","last_unstale":"2023-10-16T15:53:41.384277+0000","last_undegraded":"2023-10-16T15:53:41.384277+0000","last_fullsized":"2023-10-16T15:53:41.384277+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:01.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394436+0000","last_change":"2023-10-16T15:53:27.242413+0000","last_active":"2023-10-16T15:53:41.394436+0000","last_peered":"2023-10-16T15:53:41.394436+0000","last_clean":"2023-10-16T15:53:41.394436+0000","last_became_active":"2023-10-16T15:53:27.242150+0000","last_became_peered":"2023-10-16T15:53:27.242150+0000","last_unstale":"2023-10-16T15:53:41.394436+0000","last_undegraded":"2023-10-16T15:53:41.394436+0000","last_fullsized":"2023-10-16T15:53:41.394436+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:41.787649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389135+0000","last_change":"2023-10-16T15:53:40.384061+0000","last_active":"2023-10-16T15:53:41.389135+0000","last_peered":"2023-10-16T15:53:41.389135+0000","last_clean":"2023-10-16T15:53:41.389135+0000","last_became_active":"2023-10-16T15:53:40.383223+0000","last_became_peered":"2023-10-16T15:53:40.383223+0000","last_unstale":"2023-10-16T15:53:41.389135+0000","last_undegraded":"2023-10-16T15:53:41.389135+0000","last_fullsized":"2023-10-16T15:53:41.389135+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:35.580709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013025+0000","last_change":"2023-10-16T15:53:41.397581+0000","last_active":"2023-10-16T15:53:43.013025+0000","last_peered":"2023-10-16T15:53:43.013025+0000","last_clean":"2023-10-16T15:53:43.013025+0000","last_became_active":"2023-10-16T15:53:41.397301+0000","last_became_peered":"2023-10-16T15:53:41.397301+0000","last_unstale":"2023-10-16T15:53:43.013025+0000","last_undegraded":"2023-10-16T15:53:43.013025+0000","last_fullsized":"2023-10-16T15:53:43.013025+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:20.816418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394523+0000","last_change":"2023-10-16T15:53:26.197852+0000","last_active":"2023-10-16T15:53:41.394523+0000","last_peered":"2023-10-16T15:53:41.394523+0000","last_clean":"2023-10-16T15:53:41.394523+0000","last_became_active":"2023-10-16T15:53:26.197553+0000","last_became_peered":"2023-10-16T15:53:26.197553+0000","last_unstale":"2023-10-16T15:53:41.394523+0000","last_undegraded":"2023-10-16T15:53:41.394523+0000","last_fullsized":"2023-10-16T15:53:41.394523+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:32.893774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.246851+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.657559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.411952+0000","last_change":"2023-10-16T15:53:40.376976+0000","last_active":"2023-10-16T15:53:41.411952+0000","last_peered":"2023-10-16T15:53:41.411952+0000","last_clean":"2023-10-16T15:53:41.411952+0000","last_became_active":"2023-10-16T15:53:40.376700+0000","last_became_peered":"2023-10-16T15:53:40.376700+0000","last_unstale":"2023-10-16T15:53:41.411952+0000","last_undegraded":"2023-10-16T15:53:41.411952+0000","last_fullsized":"2023-10-16T15:53:41.411952+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:44.722446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013467+0000","last_change":"2023-10-16T15:53:41.410633+0000","last_active":"2023-10-16T15:53:43.013467+0000","last_peered":"2023-10-16T15:53:43.013467+0000","last_clean":"2023-10-16T15:53:43.013467+0000","last_became_active":"2023-10-16T15:53:41.409266+0000","last_became_peered":"2023-10-16T15:53:41.409266+0000","last_unstale":"2023-10-16T15:53:43.013467+0000","last_undegraded":"2023-10-16T15:53:43.013467+0000","last_fullsized":"2023-10-16T15:53:43.013467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:52.749663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392803+0000","last_change":"2023-10-16T15:53:26.209469+0000","last_active":"2023-10-16T15:53:41.392803+0000","last_peered":"2023-10-16T15:53:41.392803+0000","last_clean":"2023-10-16T15:53:41.392803+0000","last_became_active":"2023-10-16T15:53:26.209137+0000","last_became_peered":"2023-10-16T15:53:26.209137+0000","last_unstale":"2023-10-16T15:53:41.392803+0000","last_undegraded":"2023-10-16T15:53:41.392803+0000","last_fullsized":"2023-10-16T15:53:41.392803+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:25.682954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393648+0000","last_change":"2023-10-16T15:53:27.245243+0000","last_active":"2023-10-16T15:53:41.393648+0000","last_peered":"2023-10-16T15:53:41.393648+0000","last_clean":"2023-10-16T15:53:41.393648+0000","last_became_active":"2023-10-16T15:53:27.244467+0000","last_became_peered":"2023-10-16T15:53:27.244467+0000","last_unstale":"2023-10-16T15:53:41.393648+0000","last_undegraded":"2023-10-16T15:53:41.393648+0000","last_fullsized":"2023-10-16T15:53:41.393648+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384497+0000","last_change":"2023-10-16T15:53:40.375609+0000","last_active":"2023-10-16T15:53:41.384497+0000","last_peered":"2023-10-16T15:53:41.384497+0000","last_clean":"2023-10-16T15:53:41.384497+0000","last_became_active":"2023-10-16T15:53:40.375039+0000","last_became_peered":"2023-10-16T15:53:40.375039+0000","last_unstale":"2023-10-16T15:53:41.384497+0000","last_undegraded":"2023-10-16T15:53:41.384497+0000","last_fullsized":"2023-10-16T15:53:41.384497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:02.147563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.328614+0000","last_change":"2023-10-16T15:53:41.399266+0000","last_active":"2023-10-16T15:53:43.328614+0000","last_peered":"2023-10-16T15:53:43.328614+0000","last_clean":"2023-10-16T15:53:43.328614+0000","last_became_active":"2023-10-16T15:53:41.398177+0000","last_became_peered":"2023-10-16T15:53:41.398177+0000","last_unstale":"2023-10-16T15:53:43.328614+0000","last_undegraded":"2023-10-16T15:53:43.328614+0000","last_fullsized":"2023-10-16T15:53:43.328614+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.144341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385470+0000","last_change":"2023-10-16T15:53:26.209913+0000","last_active":"2023-10-16T15:53:41.385470+0000","last_peered":"2023-10-16T15:53:41.385470+0000","last_clean":"2023-10-16T15:53:41.385470+0000","last_became_active":"2023-10-16T15:53:26.209359+0000","last_became_peered":"2023-10-16T15:53:26.209359+0000","last_unstale":"2023-10-16T15:53:41.385470+0000","last_undegraded":"2023-10-16T15:53:41.385470+0000","last_fullsized":"2023-10-16T15:53:41.385470+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.408029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392538+0000","last_change":"2023-10-16T15:53:27.244974+0000","last_active":"2023-10-16T15:53:41.392538+0000","last_peered":"2023-10-16T15:53:41.392538+0000","last_clean":"2023-10-16T15:53:41.392538+0000","last_became_active":"2023-10-16T15:53:27.244666+0000","last_became_peered":"2023-10-16T15:53:27.244666+0000","last_unstale":"2023-10-16T15:53:41.392538+0000","last_undegraded":"2023-10-16T15:53:41.392538+0000","last_fullsized":"2023-10-16T15:53:41.392538+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:30.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386003+0000","last_change":"2023-10-16T15:53:40.388836+0000","last_active":"2023-10-16T15:53:41.386003+0000","last_peered":"2023-10-16T15:53:41.386003+0000","last_clean":"2023-10-16T15:53:41.386003+0000","last_became_active":"2023-10-16T15:53:40.387121+0000","last_became_peered":"2023-10-16T15:53:40.387121+0000","last_unstale":"2023-10-16T15:53:41.386003+0000","last_undegraded":"2023-10-16T15:53:41.386003+0000","last_fullsized":"2023-10-16T15:53:41.386003+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410790+0000","last_change":"2023-10-16T15:53:41.413661+0000","last_active":"2023-10-16T15:53:42.410790+0000","last_peered":"2023-10-16T15:53:42.410790+0000","last_clean":"2023-10-16T15:53:42.410790+0000","last_became_active":"2023-10-16T15:53:41.413219+0000","last_became_peered":"2023-10-16T15:53:41.413219+0000","last_unstale":"2023-10-16T15:53:42.410790+0000","last_undegraded":"2023-10-16T15:53:42.410790+0000","last_fullsized":"2023-10-16T15:53:42.410790+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:51.757315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380707+0000","last_change":"2023-10-16T15:53:26.203521+0000","last_active":"2023-10-16T15:53:41.380707+0000","last_peered":"2023-10-16T15:53:41.380707+0000","last_clean":"2023-10-16T15:53:41.380707+0000","last_became_active":"2023-10-16T15:53:26.203234+0000","last_became_peered":"2023-10-16T15:53:26.203234+0000","last_unstale":"2023-10-16T15:53:41.380707+0000","last_undegraded":"2023-10-16T15:53:41.380707+0000","last_fullsized":"2023-10-16T15:53:41.380707+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:43.832903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374760+0000","last_change":"2023-10-16T15:53:27.245475+0000","last_active":"2023-10-16T15:53:41.374760+0000","last_peered":"2023-10-16T15:53:41.374760+0000","last_clean":"2023-10-16T15:53:41.374760+0000","last_became_active":"2023-10-16T15:53:27.245179+0000","last_became_peered":"2023-10-16T15:53:27.245179+0000","last_unstale":"2023-10-16T15:53:41.374760+0000","last_undegraded":"2023-10-16T15:53:41.374760+0000","last_fullsized":"2023-10-16T15:53:41.374760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:26.697491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375756+0000","last_change":"2023-10-16T15:53:40.377615+0000","last_active":"2023-10-16T15:53:41.375756+0000","last_peered":"2023-10-16T15:53:41.375756+0000","last_clean":"2023-10-16T15:53:41.375756+0000","last_became_active":"2023-10-16T15:53:40.377342+0000","last_became_peered":"2023-10-16T15:53:40.377342+0000","last_unstale":"2023-10-16T15:53:41.375756+0000","last_undegraded":"2023-10-16T15:53:41.375756+0000","last_fullsized":"2023-10-16T15:53:41.375756+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:57.578541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.412874+0000","last_change":"2023-10-16T15:53:41.410293+0000","last_active":"2023-10-16T15:53:42.412874+0000","last_peered":"2023-10-16T15:53:42.412874+0000","last_clean":"2023-10-16T15:53:42.412874+0000","last_became_active":"2023-10-16T15:53:41.409897+0000","last_became_peered":"2023-10-16T15:53:41.409897+0000","last_unstale":"2023-10-16T15:53:42.412874+0000","last_undegraded":"2023-10-16T15:53:42.412874+0000","last_fullsized":"2023-10-16T15:53:42.412874+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:31.143645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388735+0000","last_change":"2023-10-16T15:53:26.202162+0000","last_active":"2023-10-16T15:53:41.388735+0000","last_peered":"2023-10-16T15:53:41.388735+0000","last_clean":"2023-10-16T15:53:41.388735+0000","last_became_active":"2023-10-16T15:53:26.200864+0000","last_became_peered":"2023-10-16T15:53:26.200864+0000","last_unstale":"2023-10-16T15:53:41.388735+0000","last_undegraded":"2023-10-16T15:53:41.388735+0000","last_fullsized":"2023-10-16T15:53:41.388735+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.380271+0000","last_change":"2023-10-16T15:53:27.251267+0000","last_active":"2023-10-16T15:53:41.380271+0000","last_peered":"2023-10-16T15:53:41.380271+0000","last_clean":"2023-10-16T15:53:41.380271+0000","last_became_active":"2023-10-16T15:53:27.250974+0000","last_became_peered":"2023-10-16T15:53:27.250974+0000","last_unstale":"2023-10-16T15:53:41.380271+0000","last_undegraded":"2023-10-16T15:53:41.380271+0000","last_fullsized":"2023-10-16T15:53:41.380271+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:06.922706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.397754+0000","last_change":"2023-10-16T15:53:40.387403+0000","last_active":"2023-10-16T15:53:41.397754+0000","last_peered":"2023-10-16T15:53:41.397754+0000","last_clean":"2023-10-16T15:53:41.397754+0000","last_became_active":"2023-10-16T15:53:40.387110+0000","last_became_peered":"2023-10-16T15:53:40.387110+0000","last_unstale":"2023-10-16T15:53:41.397754+0000","last_undegraded":"2023-10-16T15:53:41.397754+0000","last_fullsized":"2023-10-16T15:53:41.397754+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407976+0000","last_change":"2023-10-16T15:53:41.414132+0000","last_active":"2023-10-16T15:53:42.407976+0000","last_peered":"2023-10-16T15:53:42.407976+0000","last_clean":"2023-10-16T15:53:42.407976+0000","last_became_active":"2023-10-16T15:53:41.413151+0000","last_became_peered":"2023-10-16T15:53:41.413151+0000","last_unstale":"2023-10-16T15:53:42.407976+0000","last_undegraded":"2023-10-16T15:53:42.407976+0000","last_fullsized":"2023-10-16T15:53:42.407976+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393257+0000","last_change":"2023-10-16T15:53:26.210988+0000","last_active":"2023-10-16T15:53:41.393257+0000","last_peered":"2023-10-16T15:53:41.393257+0000","last_clean":"2023-10-16T15:53:41.393257+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.393257+0000","last_undegraded":"2023-10-16T15:53:41.393257+0000","last_fullsized":"2023-10-16T15:53:41.393257+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:28:25.118706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393875+0000","last_change":"2023-10-16T15:53:27.244493+0000","last_active":"2023-10-16T15:53:41.393875+0000","last_peered":"2023-10-16T15:53:41.393875+0000","last_clean":"2023-10-16T15:53:41.393875+0000","last_became_active":"2023-10-16T15:53:27.244169+0000","last_became_peered":"2023-10-16T15:53:27.244169+0000","last_unstale":"2023-10-16T15:53:41.393875+0000","last_undegraded":"2023-10-16T15:53:41.393875+0000","last_fullsized":"2023-10-16T15:53:41.393875+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:15.646770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"228'6","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.414139+0000","last_change":"2023-10-16T15:53:40.376773+0000","last_active":"2023-10-16T15:53:41.414139+0000","last_peered":"2023-10-16T15:53:41.414139+0000","last_clean":"2023-10-16T15:53:41.414139+0000","last_became_active":"2023-10-16T15:53:40.376123+0000","last_became_peered":"2023-10-16T15:53:40.376123+0000","last_unstale":"2023-10-16T15:53:41.414139+0000","last_undegraded":"2023-10-16T15:53:41.414139+0000","last_fullsized":"2023-10-16T15:53:41.414139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:19.218849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.422217+0000","last_change":"2023-10-16T15:53:41.410543+0000","last_active":"2023-10-16T15:53:43.422217+0000","last_peered":"2023-10-16T15:53:43.422217+0000","last_clean":"2023-10-16T15:53:43.422217+0000","last_became_active":"2023-10-16T15:53:41.410213+0000","last_became_peered":"2023-10-16T15:53:41.410213+0000","last_unstale":"2023-10-16T15:53:43.422217+0000","last_undegraded":"2023-10-16T15:53:43.422217+0000","last_fullsized":"2023-10-16T15:53:43.422217+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:21.097869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387904+0000","last_change":"2023-10-16T15:53:26.207091+0000","last_active":"2023-10-16T15:53:41.387904+0000","last_peered":"2023-10-16T15:53:41.387904+0000","last_clean":"2023-10-16T15:53:41.387904+0000","last_became_active":"2023-10-16T15:53:26.206769+0000","last_became_peered":"2023-10-16T15:53:26.206769+0000","last_unstale":"2023-10-16T15:53:41.387904+0000","last_undegraded":"2023-10-16T15:53:41.387904+0000","last_fullsized":"2023-10-16T15:53:41.387904+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:21.152648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383904+0000","last_change":"2023-10-16T15:53:27.233377+0000","last_active":"2023-10-16T15:53:41.383904+0000","last_peered":"2023-10-16T15:53:41.383904+0000","last_clean":"2023-10-16T15:53:41.383904+0000","last_became_active":"2023-10-16T15:53:27.233028+0000","last_became_peered":"2023-10-16T15:53:27.233028+0000","last_unstale":"2023-10-16T15:53:41.383904+0000","last_undegraded":"2023-10-16T15:53:41.383904+0000","last_fullsized":"2023-10-16T15:53:41.383904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:53.426970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"228'3","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402070+0000","last_change":"2023-10-16T15:53:40.365148+0000","last_active":"2023-10-16T15:53:41.402070+0000","last_peered":"2023-10-16T15:53:41.402070+0000","last_clean":"2023-10-16T15:53:41.402070+0000","last_became_active":"2023-10-16T15:53:40.364874+0000","last_became_peered":"2023-10-16T15:53:40.364874+0000","last_unstale":"2023-10-16T15:53:41.402070+0000","last_undegraded":"2023-10-16T15:53:41.402070+0000","last_fullsized":"2023-10-16T15:53:41.402070+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:39.387182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.411499+0000","last_change":"2023-10-16T15:53:41.405376+0000","last_active":"2023-10-16T15:53:42.411499+0000","last_peered":"2023-10-16T15:53:42.411499+0000","last_clean":"2023-10-16T15:53:42.411499+0000","last_became_active":"2023-10-16T15:53:41.404697+0000","last_became_peered":"2023-10-16T15:53:41.404697+0000","last_unstale":"2023-10-16T15:53:42.411499+0000","last_undegraded":"2023-10-16T15:53:42.411499+0000","last_fullsized":"2023-10-16T15:53:42.411499+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"225'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379765+0000","last_change":"2023-10-16T15:53:26.199623+0000","last_active":"2023-10-16T15:53:41.379765+0000","last_peered":"2023-10-16T15:53:41.379765+0000","last_clean":"2023-10-16T15:53:41.379765+0000","last_became_active":"2023-10-16T15:53:26.199216+0000","last_became_peered":"2023-10-16T15:53:26.199216+0000","last_unstale":"2023-10-16T15:53:41.379765+0000","last_undegraded":"2023-10-16T15:53:41.379765+0000","last_fullsized":"2023-10-16T15:53:41.379765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:47.904643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375992+0000","last_change":"2023-10-16T15:53:27.250217+0000","last_active":"2023-10-16T15:53:41.375992+0000","last_peered":"2023-10-16T15:53:41.375992+0000","last_clean":"2023-10-16T15:53:41.375992+0000","last_became_active":"2023-10-16T15:53:27.248674+0000","last_became_peered":"2023-10-16T15:53:27.248674+0000","last_unstale":"2023-10-16T15:53:41.375992+0000","last_undegraded":"2023-10-16T15:53:41.375992+0000","last_fullsized":"2023-10-16T15:53:41.375992+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376419+0000","last_change":"2023-10-16T15:53:40.387700+0000","last_active":"2023-10-16T15:53:41.376419+0000","last_peered":"2023-10-16T15:53:41.376419+0000","last_clean":"2023-10-16T15:53:41.376419+0000","last_became_active":"2023-10-16T15:53:40.387431+0000","last_became_peered":"2023-10-16T15:53:40.387431+0000","last_unstale":"2023-10-16T15:53:41.376419+0000","last_undegraded":"2023-10-16T15:53:41.376419+0000","last_fullsized":"2023-10-16T15:53:41.376419+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:19.583661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418316+0000","last_change":"2023-10-16T15:53:41.385042+0000","last_active":"2023-10-16T15:53:43.418316+0000","last_peered":"2023-10-16T15:53:43.418316+0000","last_clean":"2023-10-16T15:53:43.418316+0000","last_became_active":"2023-10-16T15:53:41.384431+0000","last_became_peered":"2023-10-16T15:53:41.384431+0000","last_unstale":"2023-10-16T15:53:43.418316+0000","last_undegraded":"2023-10-16T15:53:43.418316+0000","last_fullsized":"2023-10-16T15:53:43.418316+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:26:17.083674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394713+0000","last_change":"2023-10-16T15:53:26.198374+0000","last_active":"2023-10-16T15:53:41.394713+0000","last_peered":"2023-10-16T15:53:41.394713+0000","last_clean":"2023-10-16T15:53:41.394713+0000","last_became_active":"2023-10-16T15:53:26.198100+0000","last_became_peered":"2023-10-16T15:53:26.198100+0000","last_unstale":"2023-10-16T15:53:41.394713+0000","last_undegraded":"2023-10-16T15:53:41.394713+0000","last_fullsized":"2023-10-16T15:53:41.394713+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:09.575630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377982+0000","last_change":"2023-10-16T15:53:27.223512+0000","last_active":"2023-10-16T15:53:41.377982+0000","last_peered":"2023-10-16T15:53:41.377982+0000","last_clean":"2023-10-16T15:53:41.377982+0000","last_became_active":"2023-10-16T15:53:27.221539+0000","last_became_peered":"2023-10-16T15:53:27.221539+0000","last_unstale":"2023-10-16T15:53:41.377982+0000","last_undegraded":"2023-10-16T15:53:41.377982+0000","last_fullsized":"2023-10-16T15:53:41.377982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:46.563143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.408329+0000","last_change":"2023-10-16T15:53:40.386183+0000","last_active":"2023-10-16T15:53:41.408329+0000","last_peered":"2023-10-16T15:53:41.408329+0000","last_clean":"2023-10-16T15:53:41.408329+0000","last_became_active":"2023-10-16T15:53:40.385897+0000","last_became_peered":"2023-10-16T15:53:40.385897+0000","last_unstale":"2023-10-16T15:53:41.408329+0000","last_undegraded":"2023-10-16T15:53:41.408329+0000","last_fullsized":"2023-10-16T15:53:41.408329+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:27.631953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.420342+0000","last_change":"2023-10-16T15:53:41.400842+0000","last_active":"2023-10-16T15:53:43.420342+0000","last_peered":"2023-10-16T15:53:43.420342+0000","last_clean":"2023-10-16T15:53:43.420342+0000","last_became_active":"2023-10-16T15:53:41.400579+0000","last_became_peered":"2023-10-16T15:53:41.400579+0000","last_unstale":"2023-10-16T15:53:43.420342+0000","last_undegraded":"2023-10-16T15:53:43.420342+0000","last_fullsized":"2023-10-16T15:53:43.420342+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.402696+0000","last_change":"2023-10-16T15:53:26.211203+0000","last_active":"2023-10-16T15:53:41.402696+0000","last_peered":"2023-10-16T15:53:41.402696+0000","last_clean":"2023-10-16T15:53:41.402696+0000","last_became_active":"2023-10-16T15:53:26.210902+0000","last_became_peered":"2023-10-16T15:53:26.210902+0000","last_unstale":"2023-10-16T15:53:41.402696+0000","last_undegraded":"2023-10-16T15:53:41.402696+0000","last_fullsized":"2023-10-16T15:53:41.402696+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:32.744495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.387083+0000","last_change":"2023-10-16T15:53:27.252774+0000","last_active":"2023-10-16T15:53:41.387083+0000","last_peered":"2023-10-16T15:53:41.387083+0000","last_clean":"2023-10-16T15:53:41.387083+0000","last_became_active":"2023-10-16T15:53:27.252497+0000","last_became_peered":"2023-10-16T15:53:27.252497+0000","last_unstale":"2023-10-16T15:53:41.387083+0000","last_undegraded":"2023-10-16T15:53:41.387083+0000","last_fullsized":"2023-10-16T15:53:41.387083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:47.363948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"228'8","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.422402+0000","last_change":"2023-10-16T15:53:40.375924+0000","last_active":"2023-10-16T15:53:41.422402+0000","last_peered":"2023-10-16T15:53:41.422402+0000","last_clean":"2023-10-16T15:53:41.422402+0000","last_became_active":"2023-10-16T15:53:40.375638+0000","last_became_peered":"2023-10-16T15:53:40.375638+0000","last_unstale":"2023-10-16T15:53:41.422402+0000","last_undegraded":"2023-10-16T15:53:41.422402+0000","last_fullsized":"2023-10-16T15:53:41.422402+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:47.381835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407893+0000","last_change":"2023-10-16T15:53:41.414765+0000","last_active":"2023-10-16T15:53:42.407893+0000","last_peered":"2023-10-16T15:53:42.407893+0000","last_clean":"2023-10-16T15:53:42.407893+0000","last_became_active":"2023-10-16T15:53:41.414484+0000","last_became_peered":"2023-10-16T15:53:41.414484+0000","last_unstale":"2023-10-16T15:53:42.407893+0000","last_undegraded":"2023-10-16T15:53:42.407893+0000","last_fullsized":"2023-10-16T15:53:42.407893+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385369+0000","last_change":"2023-10-16T15:53:26.212261+0000","last_active":"2023-10-16T15:53:41.385369+0000","last_peered":"2023-10-16T15:53:41.385369+0000","last_clean":"2023-10-16T15:53:41.385369+0000","last_became_active":"2023-10-16T15:53:26.211989+0000","last_became_peered":"2023-10-16T15:53:26.211989+0000","last_unstale":"2023-10-16T15:53:41.385369+0000","last_undegraded":"2023-10-16T15:53:41.385369+0000","last_fullsized":"2023-10-16T15:53:41.385369+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:30.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379140+0000","last_change":"2023-10-16T15:53:27.244750+0000","last_active":"2023-10-16T15:53:41.379140+0000","last_peered":"2023-10-16T15:53:41.379140+0000","last_clean":"2023-10-16T15:53:41.379140+0000","last_became_active":"2023-10-16T15:53:27.244481+0000","last_became_peered":"2023-10-16T15:53:27.244481+0000","last_unstale":"2023-10-16T15:53:41.379140+0000","last_undegraded":"2023-10-16T15:53:41.379140+0000","last_fullsized":"2023-10-16T15:53:41.379140+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382349+0000","last_change":"2023-10-16T15:53:40.376076+0000","last_active":"2023-10-16T15:53:41.382349+0000","last_peered":"2023-10-16T15:53:41.382349+0000","last_clean":"2023-10-16T15:53:41.382349+0000","last_became_active":"2023-10-16T15:53:40.375798+0000","last_became_peered":"2023-10-16T15:53:40.375798+0000","last_unstale":"2023-10-16T15:53:41.382349+0000","last_undegraded":"2023-10-16T15:53:41.382349+0000","last_fullsized":"2023-10-16T15:53:41.382349+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330851+0000","last_change":"2023-10-16T15:53:41.409878+0000","last_active":"2023-10-16T15:53:43.330851+0000","last_peered":"2023-10-16T15:53:43.330851+0000","last_clean":"2023-10-16T15:53:43.330851+0000","last_became_active":"2023-10-16T15:53:41.409594+0000","last_became_peered":"2023-10-16T15:53:41.409594+0000","last_unstale":"2023-10-16T15:53:43.330851+0000","last_undegraded":"2023-10-16T15:53:43.330851+0000","last_fullsized":"2023-10-16T15:53:43.330851+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:10.790473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.419545+0000","last_change":"2023-10-16T15:53:40.385839+0000","last_active":"2023-10-16T15:53:41.419545+0000","last_peered":"2023-10-16T15:53:41.419545+0000","last_clean":"2023-10-16T15:53:41.419545+0000","last_became_active":"2023-10-16T15:53:40.384731+0000","last_became_peered":"2023-10-16T15:53:40.384731+0000","last_unstale":"2023-10-16T15:53:41.419545+0000","last_undegraded":"2023-10-16T15:53:41.419545+0000","last_fullsized":"2023-10-16T15:53:41.419545+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:17.818654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.386473+0000","last_change":"2023-10-16T15:53:26.200391+0000","last_active":"2023-10-16T15:53:41.386473+0000","last_peered":"2023-10-16T15:53:41.386473+0000","last_clean":"2023-10-16T15:53:41.386473+0000","last_became_active":"2023-10-16T15:53:26.200099+0000","last_became_peered":"2023-10-16T15:53:26.200099+0000","last_unstale":"2023-10-16T15:53:41.386473+0000","last_undegraded":"2023-10-16T15:53:41.386473+0000","last_fullsized":"2023-10-16T15:53:41.386473+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:45.421804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378524+0000","last_change":"2023-10-16T15:53:27.237850+0000","last_active":"2023-10-16T15:53:41.378524+0000","last_peered":"2023-10-16T15:53:41.378524+0000","last_clean":"2023-10-16T15:53:41.378524+0000","last_became_active":"2023-10-16T15:53:27.237572+0000","last_became_peered":"2023-10-16T15:53:27.237572+0000","last_unstale":"2023-10-16T15:53:41.378524+0000","last_undegraded":"2023-10-16T15:53:41.378524+0000","last_fullsized":"2023-10-16T15:53:41.378524+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:25.038456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410670+0000","last_change":"2023-10-16T15:53:41.411697+0000","last_active":"2023-10-16T15:53:42.410670+0000","last_peered":"2023-10-16T15:53:42.410670+0000","last_clean":"2023-10-16T15:53:42.410670+0000","last_became_active":"2023-10-16T15:53:41.410477+0000","last_became_peered":"2023-10-16T15:53:41.410477+0000","last_unstale":"2023-10-16T15:53:42.410670+0000","last_undegraded":"2023-10-16T15:53:42.410670+0000","last_fullsized":"2023-10-16T15:53:42.410670+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:52.524535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"228'4","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.418437+0000","last_change":"2023-10-16T15:53:40.380112+0000","last_active":"2023-10-16T15:53:41.418437+0000","last_peered":"2023-10-16T15:53:41.418437+0000","last_clean":"2023-10-16T15:53:41.418437+0000","last_became_active":"2023-10-16T15:53:40.379085+0000","last_became_peered":"2023-10-16T15:53:40.379085+0000","last_unstale":"2023-10-16T15:53:41.418437+0000","last_undegraded":"2023-10-16T15:53:41.418437+0000","last_fullsized":"2023-10-16T15:53:41.418437+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:48.756275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.382648+0000","last_change":"2023-10-16T15:53:26.213438+0000","last_active":"2023-10-16T15:53:41.382648+0000","last_peered":"2023-10-16T15:53:41.382648+0000","last_clean":"2023-10-16T15:53:41.382648+0000","last_became_active":"2023-10-16T15:53:26.213140+0000","last_became_peered":"2023-10-16T15:53:26.213140+0000","last_unstale":"2023-10-16T15:53:41.382648+0000","last_undegraded":"2023-10-16T15:53:41.382648+0000","last_fullsized":"2023-10-16T15:53:41.382648+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:55:57.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379944+0000","last_change":"2023-10-16T15:53:27.222268+0000","last_active":"2023-10-16T15:53:41.379944+0000","last_peered":"2023-10-16T15:53:41.379944+0000","last_clean":"2023-10-16T15:53:41.379944+0000","last_became_active":"2023-10-16T15:53:27.221435+0000","last_became_peered":"2023-10-16T15:53:27.221435+0000","last_unstale":"2023-10-16T15:53:41.379944+0000","last_undegraded":"2023-10-16T15:53:41.379944+0000","last_fullsized":"2023-10-16T15:53:41.379944+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.220741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421686+0000","last_change":"2023-10-16T15:53:41.405016+0000","last_active":"2023-10-16T15:53:43.421686+0000","last_peered":"2023-10-16T15:53:43.421686+0000","last_clean":"2023-10-16T15:53:43.421686+0000","last_became_active":"2023-10-16T15:53:41.404775+0000","last_became_peered":"2023-10-16T15:53:41.404775+0000","last_unstale":"2023-10-16T15:53:43.421686+0000","last_undegraded":"2023-10-16T15:53:43.421686+0000","last_fullsized":"2023-10-16T15:53:43.421686+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:03.799587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.384001+0000","last_change":"2023-10-16T15:53:26.208330+0000","last_active":"2023-10-16T15:53:41.384001+0000","last_peered":"2023-10-16T15:53:41.384001+0000","last_clean":"2023-10-16T15:53:41.384001+0000","last_became_active":"2023-10-16T15:53:26.208056+0000","last_became_peered":"2023-10-16T15:53:26.208056+0000","last_unstale":"2023-10-16T15:53:41.384001+0000","last_undegraded":"2023-10-16T15:53:41.384001+0000","last_fullsized":"2023-10-16T15:53:41.384001+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:34.580644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378030+0000","last_change":"2023-10-16T15:53:27.239876+0000","last_active":"2023-10-16T15:53:41.378030+0000","last_peered":"2023-10-16T15:53:41.378030+0000","last_clean":"2023-10-16T15:53:41.378030+0000","last_became_active":"2023-10-16T15:53:27.239564+0000","last_became_peered":"2023-10-16T15:53:27.239564+0000","last_unstale":"2023-10-16T15:53:41.378030+0000","last_undegraded":"2023-10-16T15:53:41.378030+0000","last_fullsized":"2023-10-16T15:53:41.378030+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:44:33.908311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"228'2","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.420446+0000","last_change":"2023-10-16T15:53:40.382867+0000","last_active":"2023-10-16T15:53:41.420446+0000","last_peered":"2023-10-16T15:53:41.420446+0000","last_clean":"2023-10-16T15:53:41.420446+0000","last_became_active":"2023-10-16T15:53:40.378850+0000","last_became_peered":"2023-10-16T15:53:40.378850+0000","last_unstale":"2023-10-16T15:53:41.420446+0000","last_undegraded":"2023-10-16T15:53:41.420446+0000","last_fullsized":"2023-10-16T15:53:41.420446+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:00.525042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013915+0000","last_change":"2023-10-16T15:53:41.405768+0000","last_active":"2023-10-16T15:53:43.013915+0000","last_peered":"2023-10-16T15:53:43.013915+0000","last_clean":"2023-10-16T15:53:43.013915+0000","last_became_active":"2023-10-16T15:53:41.405230+0000","last_became_peered":"2023-10-16T15:53:41.405230+0000","last_unstale":"2023-10-16T15:53:43.013915+0000","last_undegraded":"2023-10-16T15:53:43.013915+0000","last_fullsized":"2023-10-16T15:53:43.013915+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:14:43.313902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.378254+0000","last_change":"2023-10-16T15:53:26.202054+0000","last_active":"2023-10-16T15:53:41.378254+0000","last_peered":"2023-10-16T15:53:41.378254+0000","last_clean":"2023-10-16T15:53:41.378254+0000","last_became_active":"2023-10-16T15:53:26.201702+0000","last_became_peered":"2023-10-16T15:53:26.201702+0000","last_unstale":"2023-10-16T15:53:41.378254+0000","last_undegraded":"2023-10-16T15:53:41.378254+0000","last_fullsized":"2023-10-16T15:53:41.378254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:16.731484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394353+0000","last_change":"2023-10-16T15:53:27.247693+0000","last_active":"2023-10-16T15:53:41.394353+0000","last_peered":"2023-10-16T15:53:41.394353+0000","last_clean":"2023-10-16T15:53:41.394353+0000","last_became_active":"2023-10-16T15:53:27.247149+0000","last_became_peered":"2023-10-16T15:53:27.247149+0000","last_unstale":"2023-10-16T15:53:41.394353+0000","last_undegraded":"2023-10-16T15:53:41.394353+0000","last_fullsized":"2023-10-16T15:53:41.394353+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:37.440586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.404498+0000","last_change":"2023-10-16T15:53:40.381325+0000","last_active":"2023-10-16T15:53:41.404498+0000","last_peered":"2023-10-16T15:53:41.404498+0000","last_clean":"2023-10-16T15:53:41.404498+0000","last_became_active":"2023-10-16T15:53:40.380822+0000","last_became_peered":"2023-10-16T15:53:40.380822+0000","last_unstale":"2023-10-16T15:53:41.404498+0000","last_undegraded":"2023-10-16T15:53:41.404498+0000","last_fullsized":"2023-10-16T15:53:41.404498+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:49.596000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406853+0000","last_change":"2023-10-16T15:53:41.405116+0000","last_active":"2023-10-16T15:53:42.406853+0000","last_peered":"2023-10-16T15:53:42.406853+0000","last_clean":"2023-10-16T15:53:42.406853+0000","last_became_active":"2023-10-16T15:53:41.403778+0000","last_became_peered":"2023-10-16T15:53:41.403778+0000","last_unstale":"2023-10-16T15:53:42.406853+0000","last_undegraded":"2023-10-16T15:53:42.406853+0000","last_fullsized":"2023-10-16T15:53:42.406853+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385254+0000","last_change":"2023-10-16T15:53:26.208770+0000","last_active":"2023-10-16T15:53:41.385254+0000","last_peered":"2023-10-16T15:53:41.385254+0000","last_clean":"2023-10-16T15:53:41.385254+0000","last_became_active":"2023-10-16T15:53:26.206633+0000","last_became_peered":"2023-10-16T15:53:26.206633+0000","last_unstale":"2023-10-16T15:53:41.385254+0000","last_undegraded":"2023-10-16T15:53:41.385254+0000","last_fullsized":"2023-10-16T15:53:41.385254+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:52.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374210+0000","last_change":"2023-10-16T15:53:27.242460+0000","last_active":"2023-10-16T15:53:41.374210+0000","last_peered":"2023-10-16T15:53:41.374210+0000","last_clean":"2023-10-16T15:53:41.374210+0000","last_became_active":"2023-10-16T15:53:27.242142+0000","last_became_peered":"2023-10-16T15:53:27.242142+0000","last_unstale":"2023-10-16T15:53:41.374210+0000","last_undegraded":"2023-10-16T15:53:41.374210+0000","last_fullsized":"2023-10-16T15:53:41.374210+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"228'1","reported_seq":13,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.409948+0000","last_change":"2023-10-16T15:53:40.377357+0000","last_active":"2023-10-16T15:53:41.409948+0000","last_peered":"2023-10-16T15:53:41.409948+0000","last_clean":"2023-10-16T15:53:41.409948+0000","last_became_active":"2023-10-16T15:53:40.377070+0000","last_became_peered":"2023-10-16T15:53:40.377070+0000","last_unstale":"2023-10-16T15:53:41.409948+0000","last_undegraded":"2023-10-16T15:53:41.409948+0000","last_fullsized":"2023-10-16T15:53:41.409948+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:40.063109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405599+0000","last_change":"2023-10-16T15:53:41.407410+0000","last_active":"2023-10-16T15:53:42.405599+0000","last_peered":"2023-10-16T15:53:42.405599+0000","last_clean":"2023-10-16T15:53:42.405599+0000","last_became_active":"2023-10-16T15:53:41.407034+0000","last_became_peered":"2023-10-16T15:53:41.407034+0000","last_unstale":"2023-10-16T15:53:42.405599+0000","last_undegraded":"2023-10-16T15:53:42.405599+0000","last_fullsized":"2023-10-16T15:53:42.405599+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:31.173108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392695+0000","last_change":"2023-10-16T15:53:26.205566+0000","last_active":"2023-10-16T15:53:41.392695+0000","last_peered":"2023-10-16T15:53:41.392695+0000","last_clean":"2023-10-16T15:53:41.392695+0000","last_became_active":"2023-10-16T15:53:26.205259+0000","last_became_peered":"2023-10-16T15:53:26.205259+0000","last_unstale":"2023-10-16T15:53:41.392695+0000","last_undegraded":"2023-10-16T15:53:41.392695+0000","last_fullsized":"2023-10-16T15:53:41.392695+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:56.495329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.398663+0000","last_change":"2023-10-16T15:53:27.238019+0000","last_active":"2023-10-16T15:53:41.398663+0000","last_peered":"2023-10-16T15:53:41.398663+0000","last_clean":"2023-10-16T15:53:41.398663+0000","last_became_active":"2023-10-16T15:53:27.237746+0000","last_became_peered":"2023-10-16T15:53:27.237746+0000","last_unstale":"2023-10-16T15:53:41.398663+0000","last_undegraded":"2023-10-16T15:53:41.398663+0000","last_fullsized":"2023-10-16T15:53:41.398663+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:35.595181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391139+0000","last_change":"2023-10-16T15:53:40.380015+0000","last_active":"2023-10-16T15:53:41.391139+0000","last_peered":"2023-10-16T15:53:41.391139+0000","last_clean":"2023-10-16T15:53:41.391139+0000","last_became_active":"2023-10-16T15:53:40.379716+0000","last_became_peered":"2023-10-16T15:53:40.379716+0000","last_unstale":"2023-10-16T15:53:41.391139+0000","last_undegraded":"2023-10-16T15:53:41.391139+0000","last_fullsized":"2023-10-16T15:53:41.391139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:02.570396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402124+0000","last_change":"2023-10-16T15:53:41.403677+0000","last_active":"2023-10-16T15:53:42.402124+0000","last_peered":"2023-10-16T15:53:42.402124+0000","last_clean":"2023-10-16T15:53:42.402124+0000","last_became_active":"2023-10-16T15:53:41.403372+0000","last_became_peered":"2023-10-16T15:53:41.403372+0000","last_unstale":"2023-10-16T15:53:42.402124+0000","last_undegraded":"2023-10-16T15:53:42.402124+0000","last_fullsized":"2023-10-16T15:53:42.402124+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396765+0000","last_change":"2023-10-16T15:53:26.209709+0000","last_active":"2023-10-16T15:53:41.396765+0000","last_peered":"2023-10-16T15:53:41.396765+0000","last_clean":"2023-10-16T15:53:41.396765+0000","last_became_active":"2023-10-16T15:53:26.209395+0000","last_became_peered":"2023-10-16T15:53:26.209395+0000","last_unstale":"2023-10-16T15:53:41.396765+0000","last_undegraded":"2023-10-16T15:53:41.396765+0000","last_fullsized":"2023-10-16T15:53:41.396765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:25.148168+0000","last_clean_scrub_stamp":"2023-10-16T15:53:25.148168+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:03.202413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392453+0000","last_change":"2023-10-16T15:53:27.256892+0000","last_active":"2023-10-16T15:53:41.392453+0000","last_peered":"2023-10-16T15:53:41.392453+0000","last_clean":"2023-10-16T15:53:41.392453+0000","last_became_active":"2023-10-16T15:53:27.256602+0000","last_became_peered":"2023-10-16T15:53:27.256602+0000","last_unstale":"2023-10-16T15:53:41.392453+0000","last_undegraded":"2023-10-16T15:53:41.392453+0000","last_fullsized":"2023-10-16T15:53:41.392453+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:52.005844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385863+0000","last_change":"2023-10-16T15:53:40.385668+0000","last_active":"2023-10-16T15:53:41.385863+0000","last_peered":"2023-10-16T15:53:41.385863+0000","last_clean":"2023-10-16T15:53:41.385863+0000","last_became_active":"2023-10-16T15:53:40.385360+0000","last_became_peered":"2023-10-16T15:53:40.385360+0000","last_unstale":"2023-10-16T15:53:41.385863+0000","last_undegraded":"2023-10-16T15:53:41.385863+0000","last_fullsized":"2023-10-16T15:53:41.385863+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:39.323800+0000","last_clean_scrub_stamp":"2023-10-16T15:53:39.323800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:54.336040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.330261+0000","last_change":"2023-10-16T15:53:41.397528+0000","last_active":"2023-10-16T15:53:43.330261+0000","last_peered":"2023-10-16T15:53:43.330261+0000","last_clean":"2023-10-16T15:53:43.330261+0000","last_became_active":"2023-10-16T15:53:41.397247+0000","last_became_peered":"2023-10-16T15:53:41.397247+0000","last_unstale":"2023-10-16T15:53:43.330261+0000","last_undegraded":"2023-10-16T15:53:43.330261+0000","last_fullsized":"2023-10-16T15:53:43.330261+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:21.337037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.373989+0000","last_change":"2023-10-16T15:53:27.243191+0000","last_active":"2023-10-16T15:53:41.373989+0000","last_peered":"2023-10-16T15:53:41.373989+0000","last_clean":"2023-10-16T15:53:41.373989+0000","last_became_active":"2023-10-16T15:53:27.238317+0000","last_became_peered":"2023-10-16T15:53:27.238317+0000","last_unstale":"2023-10-16T15:53:41.373989+0000","last_undegraded":"2023-10-16T15:53:41.373989+0000","last_fullsized":"2023-10-16T15:53:41.373989+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410135+0000","last_change":"2023-10-16T15:53:41.408679+0000","last_active":"2023-10-16T15:53:42.410135+0000","last_peered":"2023-10-16T15:53:42.410135+0000","last_clean":"2023-10-16T15:53:42.410135+0000","last_became_active":"2023-10-16T15:53:41.407808+0000","last_became_peered":"2023-10-16T15:53:41.407808+0000","last_unstale":"2023-10-16T15:53:42.410135+0000","last_undegraded":"2023-10-16T15:53:42.410135+0000","last_fullsized":"2023-10-16T15:53:42.410135+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.121612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.388628+0000","last_change":"2023-10-16T15:53:27.243762+0000","last_active":"2023-10-16T15:53:41.388628+0000","last_peered":"2023-10-16T15:53:41.388628+0000","last_clean":"2023-10-16T15:53:41.388628+0000","last_became_active":"2023-10-16T15:53:27.243478+0000","last_became_peered":"2023-10-16T15:53:27.243478+0000","last_unstale":"2023-10-16T15:53:41.388628+0000","last_undegraded":"2023-10-16T15:53:41.388628+0000","last_fullsized":"2023-10-16T15:53:41.388628+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.647600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402941+0000","last_change":"2023-10-16T15:53:41.398094+0000","last_active":"2023-10-16T15:53:42.402941+0000","last_peered":"2023-10-16T15:53:42.402941+0000","last_clean":"2023-10-16T15:53:42.402941+0000","last_became_active":"2023-10-16T15:53:41.396310+0000","last_became_peered":"2023-10-16T15:53:41.396310+0000","last_unstale":"2023-10-16T15:53:42.402941+0000","last_undegraded":"2023-10-16T15:53:42.402941+0000","last_fullsized":"2023-10-16T15:53:42.402941+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.887677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389288+0000","last_change":"2023-10-16T15:53:27.244183+0000","last_active":"2023-10-16T15:53:41.389288+0000","last_peered":"2023-10-16T15:53:41.389288+0000","last_clean":"2023-10-16T15:53:41.389288+0000","last_became_active":"2023-10-16T15:53:27.243908+0000","last_became_peered":"2023-10-16T15:53:27.243908+0000","last_unstale":"2023-10-16T15:53:41.389288+0000","last_undegraded":"2023-10-16T15:53:41.389288+0000","last_fullsized":"2023-10-16T15:53:41.389288+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:07:57.474935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.408346+0000","last_change":"2023-10-16T15:53:41.409765+0000","last_active":"2023-10-16T15:53:42.408346+0000","last_peered":"2023-10-16T15:53:42.408346+0000","last_clean":"2023-10-16T15:53:42.408346+0000","last_became_active":"2023-10-16T15:53:41.409150+0000","last_became_peered":"2023-10-16T15:53:41.409150+0000","last_unstale":"2023-10-16T15:53:42.408346+0000","last_undegraded":"2023-10-16T15:53:42.408346+0000","last_fullsized":"2023-10-16T15:53:42.408346+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.395859+0000","last_change":"2023-10-16T15:53:27.232764+0000","last_active":"2023-10-16T15:53:41.395859+0000","last_peered":"2023-10-16T15:53:41.395859+0000","last_clean":"2023-10-16T15:53:41.395859+0000","last_became_active":"2023-10-16T15:53:27.232274+0000","last_became_peered":"2023-10-16T15:53:27.232274+0000","last_unstale":"2023-10-16T15:53:41.395859+0000","last_undegraded":"2023-10-16T15:53:41.395859+0000","last_fullsized":"2023-10-16T15:53:41.395859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:16.549111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406478+0000","last_change":"2023-10-16T15:53:41.405809+0000","last_active":"2023-10-16T15:53:42.406478+0000","last_peered":"2023-10-16T15:53:42.406478+0000","last_clean":"2023-10-16T15:53:42.406478+0000","last_became_active":"2023-10-16T15:53:41.405541+0000","last_became_peered":"2023-10-16T15:53:41.405541+0000","last_unstale":"2023-10-16T15:53:42.406478+0000","last_undegraded":"2023-10-16T15:53:42.406478+0000","last_fullsized":"2023-10-16T15:53:42.406478+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:39.653448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383859+0000","last_change":"2023-10-16T15:53:27.241821+0000","last_active":"2023-10-16T15:53:41.383859+0000","last_peered":"2023-10-16T15:53:41.383859+0000","last_clean":"2023-10-16T15:53:41.383859+0000","last_became_active":"2023-10-16T15:53:27.238011+0000","last_became_peered":"2023-10-16T15:53:27.238011+0000","last_unstale":"2023-10-16T15:53:41.383859+0000","last_undegraded":"2023-10-16T15:53:41.383859+0000","last_fullsized":"2023-10-16T15:53:41.383859+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:38.349373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421402+0000","last_change":"2023-10-16T15:53:41.411322+0000","last_active":"2023-10-16T15:53:43.421402+0000","last_peered":"2023-10-16T15:53:43.421402+0000","last_clean":"2023-10-16T15:53:43.421402+0000","last_became_active":"2023-10-16T15:53:41.410176+0000","last_became_peered":"2023-10-16T15:53:41.410176+0000","last_unstale":"2023-10-16T15:53:43.421402+0000","last_undegraded":"2023-10-16T15:53:43.421402+0000","last_fullsized":"2023-10-16T15:53:43.421402+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:59.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393894+0000","last_change":"2023-10-16T15:53:27.227619+0000","last_active":"2023-10-16T15:53:41.393894+0000","last_peered":"2023-10-16T15:53:41.393894+0000","last_clean":"2023-10-16T15:53:41.393894+0000","last_became_active":"2023-10-16T15:53:27.227089+0000","last_became_peered":"2023-10-16T15:53:27.227089+0000","last_unstale":"2023-10-16T15:53:41.393894+0000","last_undegraded":"2023-10-16T15:53:41.393894+0000","last_fullsized":"2023-10-16T15:53:41.393894+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:57.509866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418360+0000","last_change":"2023-10-16T15:53:41.400386+0000","last_active":"2023-10-16T15:53:43.418360+0000","last_peered":"2023-10-16T15:53:43.418360+0000","last_clean":"2023-10-16T15:53:43.418360+0000","last_became_active":"2023-10-16T15:53:41.400083+0000","last_became_peered":"2023-10-16T15:53:41.400083+0000","last_unstale":"2023-10-16T15:53:43.418360+0000","last_undegraded":"2023-10-16T15:53:43.418360+0000","last_fullsized":"2023-10-16T15:53:43.418360+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.375718+0000","last_change":"2023-10-16T15:53:27.241435+0000","last_active":"2023-10-16T15:53:41.375718+0000","last_peered":"2023-10-16T15:53:41.375718+0000","last_clean":"2023-10-16T15:53:41.375718+0000","last_became_active":"2023-10-16T15:53:27.240390+0000","last_became_peered":"2023-10-16T15:53:27.240390+0000","last_unstale":"2023-10-16T15:53:41.375718+0000","last_undegraded":"2023-10-16T15:53:41.375718+0000","last_fullsized":"2023-10-16T15:53:41.375718+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:06.603967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012816+0000","last_change":"2023-10-16T15:53:41.399893+0000","last_active":"2023-10-16T15:53:43.012816+0000","last_peered":"2023-10-16T15:53:43.012816+0000","last_clean":"2023-10-16T15:53:43.012816+0000","last_became_active":"2023-10-16T15:53:41.398053+0000","last_became_peered":"2023-10-16T15:53:41.398053+0000","last_unstale":"2023-10-16T15:53:43.012816+0000","last_undegraded":"2023-10-16T15:53:43.012816+0000","last_fullsized":"2023-10-16T15:53:43.012816+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:18.848509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393550+0000","last_change":"2023-10-16T15:53:27.243646+0000","last_active":"2023-10-16T15:53:41.393550+0000","last_peered":"2023-10-16T15:53:41.393550+0000","last_clean":"2023-10-16T15:53:41.393550+0000","last_became_active":"2023-10-16T15:53:27.243373+0000","last_became_peered":"2023-10-16T15:53:27.243373+0000","last_unstale":"2023-10-16T15:53:41.393550+0000","last_undegraded":"2023-10-16T15:53:41.393550+0000","last_fullsized":"2023-10-16T15:53:41.393550+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404083+0000","last_change":"2023-10-16T15:53:41.406721+0000","last_active":"2023-10-16T15:53:42.404083+0000","last_peered":"2023-10-16T15:53:42.404083+0000","last_clean":"2023-10-16T15:53:42.404083+0000","last_became_active":"2023-10-16T15:53:41.404619+0000","last_became_peered":"2023-10-16T15:53:41.404619+0000","last_unstale":"2023-10-16T15:53:42.404083+0000","last_undegraded":"2023-10-16T15:53:42.404083+0000","last_fullsized":"2023-10-16T15:53:42.404083+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.376431+0000","last_change":"2023-10-16T15:53:27.241214+0000","last_active":"2023-10-16T15:53:41.376431+0000","last_peered":"2023-10-16T15:53:41.376431+0000","last_clean":"2023-10-16T15:53:41.376431+0000","last_became_active":"2023-10-16T15:53:27.240173+0000","last_became_peered":"2023-10-16T15:53:27.240173+0000","last_unstale":"2023-10-16T15:53:41.376431+0000","last_undegraded":"2023-10-16T15:53:41.376431+0000","last_fullsized":"2023-10-16T15:53:41.376431+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:48.773558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.421928+0000","last_change":"2023-10-16T15:53:41.402971+0000","last_active":"2023-10-16T15:53:43.421928+0000","last_peered":"2023-10-16T15:53:43.421928+0000","last_clean":"2023-10-16T15:53:43.421928+0000","last_became_active":"2023-10-16T15:53:41.402710+0000","last_became_peered":"2023-10-16T15:53:41.402710+0000","last_unstale":"2023-10-16T15:53:43.421928+0000","last_undegraded":"2023-10-16T15:53:43.421928+0000","last_fullsized":"2023-10-16T15:53:43.421928+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.389236+0000","last_change":"2023-10-16T15:53:27.242855+0000","last_active":"2023-10-16T15:53:41.389236+0000","last_peered":"2023-10-16T15:53:41.389236+0000","last_clean":"2023-10-16T15:53:41.389236+0000","last_became_active":"2023-10-16T15:53:27.242589+0000","last_became_peered":"2023-10-16T15:53:27.242589+0000","last_unstale":"2023-10-16T15:53:41.389236+0000","last_undegraded":"2023-10-16T15:53:41.389236+0000","last_fullsized":"2023-10-16T15:53:41.389236+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:36.434905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402772+0000","last_change":"2023-10-16T15:53:41.399691+0000","last_active":"2023-10-16T15:53:42.402772+0000","last_peered":"2023-10-16T15:53:42.402772+0000","last_clean":"2023-10-16T15:53:42.402772+0000","last_became_active":"2023-10-16T15:53:41.397849+0000","last_became_peered":"2023-10-16T15:53:41.397849+0000","last_unstale":"2023-10-16T15:53:42.402772+0000","last_undegraded":"2023-10-16T15:53:42.402772+0000","last_fullsized":"2023-10-16T15:53:42.402772+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393175+0000","last_change":"2023-10-16T15:53:27.236957+0000","last_active":"2023-10-16T15:53:41.393175+0000","last_peered":"2023-10-16T15:53:41.393175+0000","last_clean":"2023-10-16T15:53:41.393175+0000","last_became_active":"2023-10-16T15:53:27.236453+0000","last_became_peered":"2023-10-16T15:53:27.236453+0000","last_unstale":"2023-10-16T15:53:41.393175+0000","last_undegraded":"2023-10-16T15:53:41.393175+0000","last_fullsized":"2023-10-16T15:53:41.393175+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:39.843256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014730+0000","last_change":"2023-10-16T15:53:41.410559+0000","last_active":"2023-10-16T15:53:43.014730+0000","last_peered":"2023-10-16T15:53:43.014730+0000","last_clean":"2023-10-16T15:53:43.014730+0000","last_became_active":"2023-10-16T15:53:41.410258+0000","last_became_peered":"2023-10-16T15:53:41.410258+0000","last_unstale":"2023-10-16T15:53:43.014730+0000","last_undegraded":"2023-10-16T15:53:43.014730+0000","last_fullsized":"2023-10-16T15:53:43.014730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:25.813239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377897+0000","last_change":"2023-10-16T15:53:27.242086+0000","last_active":"2023-10-16T15:53:41.377897+0000","last_peered":"2023-10-16T15:53:41.377897+0000","last_clean":"2023-10-16T15:53:41.377897+0000","last_became_active":"2023-10-16T15:53:27.240904+0000","last_became_peered":"2023-10-16T15:53:27.240904+0000","last_unstale":"2023-10-16T15:53:41.377897+0000","last_undegraded":"2023-10-16T15:53:41.377897+0000","last_fullsized":"2023-10-16T15:53:41.377897+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:10.590167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418981+0000","last_change":"2023-10-16T15:53:41.416192+0000","last_active":"2023-10-16T15:53:43.418981+0000","last_peered":"2023-10-16T15:53:43.418981+0000","last_clean":"2023-10-16T15:53:43.418981+0000","last_became_active":"2023-10-16T15:53:41.415862+0000","last_became_peered":"2023-10-16T15:53:41.415862+0000","last_unstale":"2023-10-16T15:53:43.418981+0000","last_undegraded":"2023-10-16T15:53:43.418981+0000","last_fullsized":"2023-10-16T15:53:43.418981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:07.602132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383281+0000","last_change":"2023-10-16T15:53:27.235313+0000","last_active":"2023-10-16T15:53:41.383281+0000","last_peered":"2023-10-16T15:53:41.383281+0000","last_clean":"2023-10-16T15:53:41.383281+0000","last_became_active":"2023-10-16T15:53:27.233574+0000","last_became_peered":"2023-10-16T15:53:27.233574+0000","last_unstale":"2023-10-16T15:53:41.383281+0000","last_undegraded":"2023-10-16T15:53:41.383281+0000","last_fullsized":"2023-10-16T15:53:41.383281+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:49.624425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013226+0000","last_change":"2023-10-16T15:53:41.409431+0000","last_active":"2023-10-16T15:53:43.013226+0000","last_peered":"2023-10-16T15:53:43.013226+0000","last_clean":"2023-10-16T15:53:43.013226+0000","last_became_active":"2023-10-16T15:53:41.409023+0000","last_became_peered":"2023-10-16T15:53:41.409023+0000","last_unstale":"2023-10-16T15:53:43.013226+0000","last_undegraded":"2023-10-16T15:53:43.013226+0000","last_fullsized":"2023-10-16T15:53:43.013226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:31.127234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394621+0000","last_change":"2023-10-16T15:53:27.247857+0000","last_active":"2023-10-16T15:53:41.394621+0000","last_peered":"2023-10-16T15:53:41.394621+0000","last_clean":"2023-10-16T15:53:41.394621+0000","last_became_active":"2023-10-16T15:53:27.247570+0000","last_became_peered":"2023-10-16T15:53:27.247570+0000","last_unstale":"2023-10-16T15:53:41.394621+0000","last_undegraded":"2023-10-16T15:53:41.394621+0000","last_fullsized":"2023-10-16T15:53:41.394621+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:22:37.541321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.012572+0000","last_change":"2023-10-16T15:53:41.397505+0000","last_active":"2023-10-16T15:53:43.012572+0000","last_peered":"2023-10-16T15:53:43.012572+0000","last_clean":"2023-10-16T15:53:43.012572+0000","last_became_active":"2023-10-16T15:53:41.397198+0000","last_became_peered":"2023-10-16T15:53:41.397198+0000","last_unstale":"2023-10-16T15:53:43.012572+0000","last_undegraded":"2023-10-16T15:53:43.012572+0000","last_fullsized":"2023-10-16T15:53:43.012572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:07.665494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394898+0000","last_change":"2023-10-16T15:53:27.258953+0000","last_active":"2023-10-16T15:53:41.394898+0000","last_peered":"2023-10-16T15:53:41.394898+0000","last_clean":"2023-10-16T15:53:41.394898+0000","last_became_active":"2023-10-16T15:53:27.258159+0000","last_became_peered":"2023-10-16T15:53:27.258159+0000","last_unstale":"2023-10-16T15:53:41.394898+0000","last_undegraded":"2023-10-16T15:53:41.394898+0000","last_fullsized":"2023-10-16T15:53:41.394898+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.030680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.406566+0000","last_change":"2023-10-16T15:53:41.388087+0000","last_active":"2023-10-16T15:53:42.406566+0000","last_peered":"2023-10-16T15:53:42.406566+0000","last_clean":"2023-10-16T15:53:42.406566+0000","last_became_active":"2023-10-16T15:53:41.387820+0000","last_became_peered":"2023-10-16T15:53:41.387820+0000","last_unstale":"2023-10-16T15:53:42.406566+0000","last_undegraded":"2023-10-16T15:53:42.406566+0000","last_fullsized":"2023-10-16T15:53:42.406566+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:02.460195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374811+0000","last_change":"2023-10-16T15:53:27.241074+0000","last_active":"2023-10-16T15:53:41.374811+0000","last_peered":"2023-10-16T15:53:41.374811+0000","last_clean":"2023-10-16T15:53:41.374811+0000","last_became_active":"2023-10-16T15:53:27.240743+0000","last_became_peered":"2023-10-16T15:53:27.240743+0000","last_unstale":"2023-10-16T15:53:41.374811+0000","last_undegraded":"2023-10-16T15:53:41.374811+0000","last_fullsized":"2023-10-16T15:53:41.374811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:14.887653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.404049+0000","last_change":"2023-10-16T15:53:41.397057+0000","last_active":"2023-10-16T15:53:42.404049+0000","last_peered":"2023-10-16T15:53:42.404049+0000","last_clean":"2023-10-16T15:53:42.404049+0000","last_became_active":"2023-10-16T15:53:41.396163+0000","last_became_peered":"2023-10-16T15:53:41.396163+0000","last_unstale":"2023-10-16T15:53:42.404049+0000","last_undegraded":"2023-10-16T15:53:42.404049+0000","last_fullsized":"2023-10-16T15:53:42.404049+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:49.341925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.391109+0000","last_change":"2023-10-16T15:53:27.241178+0000","last_active":"2023-10-16T15:53:41.391109+0000","last_peered":"2023-10-16T15:53:41.391109+0000","last_clean":"2023-10-16T15:53:41.391109+0000","last_became_active":"2023-10-16T15:53:27.240901+0000","last_became_peered":"2023-10-16T15:53:27.240901+0000","last_unstale":"2023-10-16T15:53:41.391109+0000","last_undegraded":"2023-10-16T15:53:41.391109+0000","last_fullsized":"2023-10-16T15:53:41.391109+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:14.396661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.410984+0000","last_change":"2023-10-16T15:53:41.410981+0000","last_active":"2023-10-16T15:53:42.410984+0000","last_peered":"2023-10-16T15:53:42.410984+0000","last_clean":"2023-10-16T15:53:42.410984+0000","last_became_active":"2023-10-16T15:53:41.410713+0000","last_became_peered":"2023-10-16T15:53:41.410713+0000","last_unstale":"2023-10-16T15:53:42.410984+0000","last_undegraded":"2023-10-16T15:53:42.410984+0000","last_fullsized":"2023-10-16T15:53:42.410984+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:01.539110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374479+0000","last_change":"2023-10-16T15:53:27.249042+0000","last_active":"2023-10-16T15:53:41.374479+0000","last_peered":"2023-10-16T15:53:41.374479+0000","last_clean":"2023-10-16T15:53:41.374479+0000","last_became_active":"2023-10-16T15:53:27.248787+0000","last_became_peered":"2023-10-16T15:53:27.248787+0000","last_unstale":"2023-10-16T15:53:41.374479+0000","last_undegraded":"2023-10-16T15:53:41.374479+0000","last_fullsized":"2023-10-16T15:53:41.374479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405376+0000","last_change":"2023-10-16T15:53:41.413215+0000","last_active":"2023-10-16T15:53:42.405376+0000","last_peered":"2023-10-16T15:53:42.405376+0000","last_clean":"2023-10-16T15:53:42.405376+0000","last_became_active":"2023-10-16T15:53:41.412398+0000","last_became_peered":"2023-10-16T15:53:41.412398+0000","last_unstale":"2023-10-16T15:53:42.405376+0000","last_undegraded":"2023-10-16T15:53:42.405376+0000","last_fullsized":"2023-10-16T15:53:42.405376+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:34.906610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374715+0000","last_change":"2023-10-16T15:53:27.253011+0000","last_active":"2023-10-16T15:53:41.374715+0000","last_peered":"2023-10-16T15:53:41.374715+0000","last_clean":"2023-10-16T15:53:41.374715+0000","last_became_active":"2023-10-16T15:53:27.248119+0000","last_became_peered":"2023-10-16T15:53:27.248119+0000","last_unstale":"2023-10-16T15:53:41.374715+0000","last_undegraded":"2023-10-16T15:53:41.374715+0000","last_fullsized":"2023-10-16T15:53:41.374715+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:57.170060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407630+0000","last_change":"2023-10-16T15:53:41.409487+0000","last_active":"2023-10-16T15:53:42.407630+0000","last_peered":"2023-10-16T15:53:42.407630+0000","last_clean":"2023-10-16T15:53:42.407630+0000","last_became_active":"2023-10-16T15:53:41.409160+0000","last_became_peered":"2023-10-16T15:53:41.409160+0000","last_unstale":"2023-10-16T15:53:42.407630+0000","last_undegraded":"2023-10-16T15:53:42.407630+0000","last_fullsized":"2023-10-16T15:53:42.407630+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:47.399054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.396559+0000","last_change":"2023-10-16T15:53:27.246993+0000","last_active":"2023-10-16T15:53:41.396559+0000","last_peered":"2023-10-16T15:53:41.396559+0000","last_clean":"2023-10-16T15:53:41.396559+0000","last_became_active":"2023-10-16T15:53:27.246685+0000","last_became_peered":"2023-10-16T15:53:27.246685+0000","last_unstale":"2023-10-16T15:53:41.396559+0000","last_undegraded":"2023-10-16T15:53:41.396559+0000","last_fullsized":"2023-10-16T15:53:41.396559+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:04.164873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.420943+0000","last_change":"2023-10-16T15:53:41.393762+0000","last_active":"2023-10-16T15:53:43.420943+0000","last_peered":"2023-10-16T15:53:43.420943+0000","last_clean":"2023-10-16T15:53:43.420943+0000","last_became_active":"2023-10-16T15:53:41.393255+0000","last_became_peered":"2023-10-16T15:53:41.393255+0000","last_unstale":"2023-10-16T15:53:43.420943+0000","last_undegraded":"2023-10-16T15:53:43.420943+0000","last_fullsized":"2023-10-16T15:53:43.420943+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:48.395903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.400191+0000","last_change":"2023-10-16T15:53:27.225825+0000","last_active":"2023-10-16T15:53:41.400191+0000","last_peered":"2023-10-16T15:53:41.400191+0000","last_clean":"2023-10-16T15:53:41.400191+0000","last_became_active":"2023-10-16T15:53:27.225122+0000","last_became_peered":"2023-10-16T15:53:27.225122+0000","last_unstale":"2023-10-16T15:53:41.400191+0000","last_undegraded":"2023-10-16T15:53:41.400191+0000","last_fullsized":"2023-10-16T15:53:41.400191+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405352+0000","last_change":"2023-10-16T15:53:41.416464+0000","last_active":"2023-10-16T15:53:42.405352+0000","last_peered":"2023-10-16T15:53:42.405352+0000","last_clean":"2023-10-16T15:53:42.405352+0000","last_became_active":"2023-10-16T15:53:41.409828+0000","last_became_peered":"2023-10-16T15:53:41.409828+0000","last_unstale":"2023-10-16T15:53:42.405352+0000","last_undegraded":"2023-10-16T15:53:42.405352+0000","last_fullsized":"2023-10-16T15:53:42.405352+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:22.656293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385167+0000","last_change":"2023-10-16T15:53:27.245551+0000","last_active":"2023-10-16T15:53:41.385167+0000","last_peered":"2023-10-16T15:53:41.385167+0000","last_clean":"2023-10-16T15:53:41.385167+0000","last_became_active":"2023-10-16T15:53:27.245268+0000","last_became_peered":"2023-10-16T15:53:27.245268+0000","last_unstale":"2023-10-16T15:53:41.385167+0000","last_undegraded":"2023-10-16T15:53:41.385167+0000","last_fullsized":"2023-10-16T15:53:41.385167+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:16.997946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.405989+0000","last_change":"2023-10-16T15:53:41.400670+0000","last_active":"2023-10-16T15:53:42.405989+0000","last_peered":"2023-10-16T15:53:42.405989+0000","last_clean":"2023-10-16T15:53:42.405989+0000","last_became_active":"2023-10-16T15:53:41.400310+0000","last_became_peered":"2023-10-16T15:53:41.400310+0000","last_unstale":"2023-10-16T15:53:42.405989+0000","last_undegraded":"2023-10-16T15:53:42.405989+0000","last_fullsized":"2023-10-16T15:53:42.405989+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:55.180266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.394811+0000","last_change":"2023-10-16T15:53:27.247621+0000","last_active":"2023-10-16T15:53:41.394811+0000","last_peered":"2023-10-16T15:53:41.394811+0000","last_clean":"2023-10-16T15:53:41.394811+0000","last_became_active":"2023-10-16T15:53:27.247280+0000","last_became_peered":"2023-10-16T15:53:27.247280+0000","last_unstale":"2023-10-16T15:53:41.394811+0000","last_undegraded":"2023-10-16T15:53:41.394811+0000","last_fullsized":"2023-10-16T15:53:41.394811+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:14.460023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.418006+0000","last_change":"2023-10-16T15:53:41.406425+0000","last_active":"2023-10-16T15:53:43.418006+0000","last_peered":"2023-10-16T15:53:43.418006+0000","last_clean":"2023-10-16T15:53:43.418006+0000","last_became_active":"2023-10-16T15:53:41.405266+0000","last_became_peered":"2023-10-16T15:53:41.405266+0000","last_unstale":"2023-10-16T15:53:43.418006+0000","last_undegraded":"2023-10-16T15:53:43.418006+0000","last_fullsized":"2023-10-16T15:53:43.418006+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:36.594777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.393796+0000","last_change":"2023-10-16T15:53:27.245452+0000","last_active":"2023-10-16T15:53:41.393796+0000","last_peered":"2023-10-16T15:53:41.393796+0000","last_clean":"2023-10-16T15:53:41.393796+0000","last_became_active":"2023-10-16T15:53:27.244961+0000","last_became_peered":"2023-10-16T15:53:27.244961+0000","last_unstale":"2023-10-16T15:53:41.393796+0000","last_undegraded":"2023-10-16T15:53:41.393796+0000","last_fullsized":"2023-10-16T15:53:41.393796+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:56.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.413065+0000","last_change":"2023-10-16T15:53:41.410135+0000","last_active":"2023-10-16T15:53:42.413065+0000","last_peered":"2023-10-16T15:53:42.413065+0000","last_clean":"2023-10-16T15:53:42.413065+0000","last_became_active":"2023-10-16T15:53:41.409626+0000","last_became_peered":"2023-10-16T15:53:41.409626+0000","last_unstale":"2023-10-16T15:53:42.413065+0000","last_undegraded":"2023-10-16T15:53:42.413065+0000","last_fullsized":"2023-10-16T15:53:42.413065+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.484536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.379861+0000","last_change":"2023-10-16T15:53:27.237786+0000","last_active":"2023-10-16T15:53:41.379861+0000","last_peered":"2023-10-16T15:53:41.379861+0000","last_clean":"2023-10-16T15:53:41.379861+0000","last_became_active":"2023-10-16T15:53:27.237432+0000","last_became_peered":"2023-10-16T15:53:27.237432+0000","last_unstale":"2023-10-16T15:53:41.379861+0000","last_undegraded":"2023-10-16T15:53:41.379861+0000","last_fullsized":"2023-10-16T15:53:41.379861+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:06.957352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409800+0000","last_change":"2023-10-16T15:53:41.408039+0000","last_active":"2023-10-16T15:53:42.409800+0000","last_peered":"2023-10-16T15:53:42.409800+0000","last_clean":"2023-10-16T15:53:42.409800+0000","last_became_active":"2023-10-16T15:53:41.407756+0000","last_became_peered":"2023-10-16T15:53:41.407756+0000","last_unstale":"2023-10-16T15:53:42.409800+0000","last_undegraded":"2023-10-16T15:53:42.409800+0000","last_fullsized":"2023-10-16T15:53:42.409800+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:55.883929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374213+0000","last_change":"2023-10-16T15:53:27.244165+0000","last_active":"2023-10-16T15:53:41.374213+0000","last_peered":"2023-10-16T15:53:41.374213+0000","last_clean":"2023-10-16T15:53:41.374213+0000","last_became_active":"2023-10-16T15:53:27.240768+0000","last_became_peered":"2023-10-16T15:53:27.240768+0000","last_unstale":"2023-10-16T15:53:41.374213+0000","last_undegraded":"2023-10-16T15:53:41.374213+0000","last_fullsized":"2023-10-16T15:53:41.374213+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:50.818010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.013669+0000","last_change":"2023-10-16T15:53:41.411866+0000","last_active":"2023-10-16T15:53:43.013669+0000","last_peered":"2023-10-16T15:53:43.013669+0000","last_clean":"2023-10-16T15:53:43.013669+0000","last_became_active":"2023-10-16T15:53:41.411569+0000","last_became_peered":"2023-10-16T15:53:41.411569+0000","last_unstale":"2023-10-16T15:53:43.013669+0000","last_undegraded":"2023-10-16T15:53:43.013669+0000","last_fullsized":"2023-10-16T15:53:43.013669+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:01.576399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.383777+0000","last_change":"2023-10-16T15:53:27.249218+0000","last_active":"2023-10-16T15:53:41.383777+0000","last_peered":"2023-10-16T15:53:41.383777+0000","last_clean":"2023-10-16T15:53:41.383777+0000","last_became_active":"2023-10-16T15:53:27.248920+0000","last_became_peered":"2023-10-16T15:53:27.248920+0000","last_unstale":"2023-10-16T15:53:41.383777+0000","last_undegraded":"2023-10-16T15:53:41.383777+0000","last_fullsized":"2023-10-16T15:53:41.383777+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:44.847440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.409957+0000","last_change":"2023-10-16T15:53:41.399860+0000","last_active":"2023-10-16T15:53:42.409957+0000","last_peered":"2023-10-16T15:53:42.409957+0000","last_clean":"2023-10-16T15:53:42.409957+0000","last_became_active":"2023-10-16T15:53:41.399589+0000","last_became_peered":"2023-10-16T15:53:41.399589+0000","last_unstale":"2023-10-16T15:53:42.409957+0000","last_undegraded":"2023-10-16T15:53:42.409957+0000","last_fullsized":"2023-10-16T15:53:42.409957+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:14.239042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385445+0000","last_change":"2023-10-16T15:53:27.242621+0000","last_active":"2023-10-16T15:53:41.385445+0000","last_peered":"2023-10-16T15:53:41.385445+0000","last_clean":"2023-10-16T15:53:41.385445+0000","last_became_active":"2023-10-16T15:53:27.242324+0000","last_became_peered":"2023-10-16T15:53:27.242324+0000","last_unstale":"2023-10-16T15:53:41.385445+0000","last_undegraded":"2023-10-16T15:53:41.385445+0000","last_fullsized":"2023-10-16T15:53:41.385445+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:09:00.063880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.402749+0000","last_change":"2023-10-16T15:53:41.399919+0000","last_active":"2023-10-16T15:53:42.402749+0000","last_peered":"2023-10-16T15:53:42.402749+0000","last_clean":"2023-10-16T15:53:42.402749+0000","last_became_active":"2023-10-16T15:53:41.397770+0000","last_became_peered":"2023-10-16T15:53:41.397770+0000","last_unstale":"2023-10-16T15:53:42.402749+0000","last_undegraded":"2023-10-16T15:53:42.402749+0000","last_fullsized":"2023-10-16T15:53:42.402749+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:40.633355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.377733+0000","last_change":"2023-10-16T15:53:27.237581+0000","last_active":"2023-10-16T15:53:41.377733+0000","last_peered":"2023-10-16T15:53:41.377733+0000","last_clean":"2023-10-16T15:53:41.377733+0000","last_became_active":"2023-10-16T15:53:27.237248+0000","last_became_peered":"2023-10-16T15:53:27.237248+0000","last_unstale":"2023-10-16T15:53:41.377733+0000","last_undegraded":"2023-10-16T15:53:41.377733+0000","last_fullsized":"2023-10-16T15:53:41.377733+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:34.871296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.014330+0000","last_change":"2023-10-16T15:53:41.401459+0000","last_active":"2023-10-16T15:53:43.014330+0000","last_peered":"2023-10-16T15:53:43.014330+0000","last_clean":"2023-10-16T15:53:43.014330+0000","last_became_active":"2023-10-16T15:53:41.401147+0000","last_became_peered":"2023-10-16T15:53:41.401147+0000","last_unstale":"2023-10-16T15:53:43.014330+0000","last_undegraded":"2023-10-16T15:53:43.014330+0000","last_fullsized":"2023-10-16T15:53:43.014330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:31.244497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.385078+0000","last_change":"2023-10-16T15:53:27.251169+0000","last_active":"2023-10-16T15:53:41.385078+0000","last_peered":"2023-10-16T15:53:41.385078+0000","last_clean":"2023-10-16T15:53:41.385078+0000","last_became_active":"2023-10-16T15:53:27.249827+0000","last_became_peered":"2023-10-16T15:53:27.249827+0000","last_unstale":"2023-10-16T15:53:41.385078+0000","last_undegraded":"2023-10-16T15:53:41.385078+0000","last_fullsized":"2023-10-16T15:53:41.385078+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:20.731448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:43.329048+0000","last_change":"2023-10-16T15:53:41.404413+0000","last_active":"2023-10-16T15:53:43.329048+0000","last_peered":"2023-10-16T15:53:43.329048+0000","last_clean":"2023-10-16T15:53:43.329048+0000","last_became_active":"2023-10-16T15:53:41.403862+0000","last_became_peered":"2023-10-16T15:53:41.403862+0000","last_unstale":"2023-10-16T15:53:43.329048+0000","last_undegraded":"2023-10-16T15:53:43.329048+0000","last_fullsized":"2023-10-16T15:53:43.329048+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:34.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.392361+0000","last_change":"2023-10-16T15:53:27.241783+0000","last_active":"2023-10-16T15:53:41.392361+0000","last_peered":"2023-10-16T15:53:41.392361+0000","last_clean":"2023-10-16T15:53:41.392361+0000","last_became_active":"2023-10-16T15:53:27.241197+0000","last_became_peered":"2023-10-16T15:53:27.241197+0000","last_unstale":"2023-10-16T15:53:41.392361+0000","last_undegraded":"2023-10-16T15:53:41.392361+0000","last_fullsized":"2023-10-16T15:53:41.392361+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:22.360181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T15:53:42.407480+0000","last_change":"2023-10-16T15:53:41.406217+0000","last_active":"2023-10-16T15:53:42.407480+0000","last_peered":"2023-10-16T15:53:42.407480+0000","last_clean":"2023-10-16T15:53:42.407480+0000","last_became_active":"2023-10-16T15:53:41.405870+0000","last_became_peered":"2023-10-16T15:53:41.405870+0000","last_unstale":"2023-10-16T15:53:42.407480+0000","last_undegraded":"2023-10-16T15:53:42.407480+0000","last_fullsized":"2023-10-16T15:53:42.407480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:40.337868+0000","last_clean_scrub_stamp":"2023-10-16T15:53:40.337868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:21.132514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T15:53:41.374717+0000","last_change":"2023-10-16T15:53:27.246185+0000","last_active":"2023-10-16T15:53:41.374717+0000","last_peered":"2023-10-16T15:53:41.374717+0000","last_clean":"2023-10-16T15:53:41.374717+0000","last_became_active":"2023-10-16T15:53:27.244324+0000","last_became_peered":"2023-10-16T15:53:27.244324+0000","last_unstale":"2023-10-16T15:53:41.374717+0000","last_undegraded":"2023-10-16T15:53:41.374717+0000","last_fullsized":"2023-10-16T15:53:41.374717+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:53:26.162705+0000","last_clean_scrub_stamp":"2023-10-16T15:53:26.162705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:48.237952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576007,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":3400,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":3481600,"data_stored":2005127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.194,"15min":1.1990000000000001},"min":{"1min":0.92600000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.794,"5min":1.8260000000000001,"15min":1.8879999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.139,"15min":1.167},"min":{"1min":0.84599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.748},"last":1.2450000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.151,"15min":1.1839999999999999},"min":{"1min":0.78700000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.76,"5min":1.76,"15min":1.889},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.329,"15min":1.3660000000000001},"min":{"1min":0.93799999999999994,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6060000000000001,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":1.5289999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.347,"15min":1.341},"min":{"1min":1.0780000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.893,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.325,"15min":1.321},"min":{"1min":0.91100000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.909,"5min":1.909,"15min":1.95},"last":1.5900000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3720000000000001,"15min":1.3660000000000001},"min":{"1min":1.145,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.869},"last":0.876},{"interface":"front","average":{"1min":1.415,"5min":1.446,"15min":1.45},"min":{"1min":1.143,"5min":1.0309999999999999,"15min":0.85199999999999998},"max":{"1min":1.8420000000000001,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.2969999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3839999999999999,"15min":1.379},"min":{"1min":0.91400000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7729999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3520000000000001,"15min":1.351},"min":{"1min":0.77200000000000002,"5min":0.71199999999999997,"15min":0.68300000000000005},"max":{"1min":1.8740000000000001,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.655}]},{"osd":5,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.486,"15min":1.468},"min":{"1min":1.125,"5min":0.91000000000000003,"15min":0.65200000000000002},"max":{"1min":1.796,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.347},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.958,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.6990000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:52:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.472,"15min":1.4770000000000001},"min":{"1min":1.137,"5min":0.88400000000000001,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":1.9510000000000001,"15min":2.2149999999999999},"last":1.482},{"interface":"front","average":{"1min":1.575,"5min":1.5760000000000001,"15min":1.5840000000000001},"min":{"1min":1.1919999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.905,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.782}]}]},{"osd":6,"up_from":13,"seq":55834576006,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1412470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.216},"min":{"1min":0.73699999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7709999999999999,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.276,"15min":1.2609999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.2909999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2949999999999999,"15min":1.2929999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.72199999999999998},"max":{"1min":1.8740000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3420000000000001,"15min":1.329},"min":{"1min":0.84699999999999998,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.579,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.98399999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.244,"15min":1.2450000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.752},"max":{"1min":1.554,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.877},{"interface":"front","average":{"1min":1.395,"5min":1.431,"15min":1.401},"min":{"1min":1.0680000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7,"5min":1.8620000000000001,"15min":2.0339999999999998},"last":1.5049999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.395},"min":{"1min":0.88100000000000001,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.331,"15min":1.339},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.631,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.135}]},{"osd":4,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3560000000000001,"15min":1.331},"min":{"1min":0.871,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.925},{"interface":"front","average":{"1min":1.444,"5min":1.4430000000000001,"15min":1.407},"min":{"1min":0.90100000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.5680000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":0.88300000000000001,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.792,"5min":1.9419999999999999,"15min":2.016},"last":1.619},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5640000000000001,"15min":1.5289999999999999},"min":{"1min":1.0309999999999999,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.853}]},{"osd":7,"last update":"Mon Oct 16 15:53:33 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.633,"15min":1.6060000000000001},"min":{"1min":1.1950000000000001,"5min":0.627,"15min":0.627},"max":{"1min":2.149,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.673,"5min":1.6220000000000001,"15min":1.5980000000000001},"min":{"1min":1.264,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.7410000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576006,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29832,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342008,"statfs":{"total":96636764160,"available":96606216192,"internally_reserved":0,"allocated":2875392,"data_stored":1412414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1240000000000001,"15min":1.163},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6850000000000001,"5min":1.8069999999999999,"15min":1.821},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.2689999999999999,"15min":1.2430000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.53,"5min":2.738,"15min":2.738},"last":1.1140000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7669999999999999,"5min":1.7849999999999999,"15min":1.8680000000000001},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2629999999999999,"15min":1.302},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7270000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.579}]},{"osd":2,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.238,"15min":1.2529999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.577,"5min":1.784,"15min":1.9219999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.399,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7190000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.4399999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.343,"15min":1.3400000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9039999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.3400000000000001,"15min":1.3799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.7130000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.286}]},{"osd":4,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.367,"15min":1.347},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.724},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.5069999999999999,"15min":1.5189999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.804,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.405}]},{"osd":6,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.385},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8660000000000001,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.371},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":1.431},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0230000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.484}]},{"osd":7,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.538,"15min":1.5369999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.847,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.536},{"interface":"front","average":{"1min":1.516,"5min":1.639,"15min":1.633},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.962,"5min":3.109,"15min":3.109},"last":1.6799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2895872,"data_stored":1415375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1919999999999999,"15min":1.202},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.52700000000000002},"max":{"1min":1.861,"5min":1.883,"15min":1.883},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1890000000000001,"15min":1.198},"min":{"1min":0.78800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.875,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.387}]},{"osd":1,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.2370000000000001},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.629},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.88100000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.952,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.0609999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.413,"15min":1.4570000000000001},"min":{"1min":0.99399999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.6240000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4910000000000001,"15min":1.4990000000000001},"min":{"1min":1.0900000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9099999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.6679999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.425,"15min":1.391},"min":{"1min":0.84699999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.1099999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.391,"15min":1.3919999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.6220000000000001,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.254}]},{"osd":5,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.377,"15min":1.347},"min":{"1min":1.046,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.8899999999999999,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.454},{"interface":"front","average":{"1min":1.522,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":1.177,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.004,"5min":2.161,"15min":2.4489999999999998},"last":1.601}]},{"osd":6,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3999999999999999,"15min":1.4039999999999999},"min":{"1min":0.94099999999999995,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":1.8300000000000001,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.639},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5529999999999999,"15min":1.528},"min":{"1min":1.198,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.0049999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.5640000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.625,"15min":1.5880000000000001},"min":{"1min":1.139,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.2200000000000002,"5min":2.399,"15min":2.399},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.6180000000000001,"15min":1.6040000000000001},"min":{"1min":1.1299999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0449999999999999,"5min":2.2599999999999998,"15min":2.347},"last":1.706}]}]},{"osd":3,"up_from":13,"seq":55834576006,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56228,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315612,"statfs":{"total":96636764160,"available":96579186688,"internally_reserved":0,"allocated":2899968,"data_stored":1413696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.80600000000000005,"5min":0.622,"15min":0.622},"max":{"1min":1.2909999999999999,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0700000000000001,"15min":1.079},"min":{"1min":0.80800000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.5860000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.98899999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.84699999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.544,"5min":1.6679999999999999,"15min":2.589},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.173,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.92500000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3140000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.131}]},{"osd":2,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4119999999999999,"15min":1.4399999999999999},"min":{"1min":1.0900000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8149999999999999,"5min":1.885,"15min":2.302},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":1.3089999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.76,"5min":1.9530000000000001,"15min":2.754},"last":0.90000000000000002}]},{"osd":4,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.361,"15min":1.365},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.988,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":1.256},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0249999999999999,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.1930000000000001,"5min":2.2530000000000001,"15min":2.5179999999999998},"last":1.355}]},{"osd":5,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5669999999999999,"15min":1.581},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.6320000000000001},"last":1.466},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.5640000000000001,"15min":1.5840000000000001},"min":{"1min":1.3320000000000001,"5min":1.0920000000000001,"15min":1.008},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.51}]},{"osd":6,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.5189999999999999},"min":{"1min":1.125,"5min":0.998,"15min":0.97899999999999998},"max":{"1min":2.173,"5min":2.173,"15min":2.2090000000000001},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6120000000000001,"15min":1.6200000000000001},"min":{"1min":1.2969999999999999,"5min":1.02,"15min":1.02},"max":{"1min":2.23,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.5720000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:52:45 2023","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.6850000000000001,"15min":1.698},"min":{"1min":1.216,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2679999999999998,"15min":2.8780000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.764,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":1.298,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.4159999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576007,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1413500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":1.7789999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":1.0489999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":0.996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.341}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.23},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.486}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.147},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.536}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.641},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.581}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.857},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.6990000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576007,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47884,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323956,"statfs":{"total":96636764160,"available":96587730944,"internally_reserved":0,"allocated":2883584,"data_stored":1411978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95499999999999996,"15min":0.94899999999999995},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.49,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.016,"5min":1.0880000000000001,"15min":1.151},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.0109999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.016,"15min":1.042},"min":{"1min":0.53800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.6299999999999999,"5min":1.833,"15min":2.286},"last":1.087},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.238},"min":{"1min":0.70399999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6819999999999999,"5min":1.806,"15min":1.853},"last":1.163}]},{"osd":3,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.385,"15min":1.3839999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.78,"5min":2.0619999999999998,"15min":2.137},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.1400000000000001,"5min":2.1629999999999998,"15min":2.9910000000000001},"last":1.5820000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.327,"5min":2.327,"15min":2.3809999999999998},"last":1.369},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.91200000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.526}]},{"osd":5,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.51,"15min":1.54},"min":{"1min":1,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.468,"15min":1.504},"min":{"1min":1.0660000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0640000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.4930000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6779999999999999,"15min":1.665},"min":{"1min":0.97899999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.234,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.647},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.621,"15min":1.653},"min":{"1min":0.83999999999999997,"5min":0.749,"15min":0.749},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.7609999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:53:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.7170000000000001,"15min":1.754},"min":{"1min":0.89600000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.734,"5min":2.734,"15min":3.2090000000000001},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6479999999999999,"15min":1.6599999999999999},"min":{"1min":1.0980000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.2029999999999998,"5min":2.395,"15min":3.1539999999999999},"last":1.72}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48468,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323372,"statfs":{"total":96636764160,"available":96587132928,"internally_reserved":0,"allocated":3477504,"data_stored":2002962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.96499999999999997,"15min":0.96399999999999997},"min":{"1min":0.61499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":1.3700000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.256,"5min":1.2490000000000001,"15min":1.246},"min":{"1min":0.71499999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.76,"5min":2.036,"15min":2.036},"last":0.498}]},{"osd":2,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.294},"min":{"1min":0.95999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.768,"5min":1.913,"15min":2.0259999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2170000000000001,"15min":1.242},"min":{"1min":0.72199999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7849999999999999,"15min":1.843},"last":0.92300000000000004}]},{"osd":3,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.024,"15min":1.0469999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2729999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.173,"5min":1.2869999999999999,"15min":1.2829999999999999},"min":{"1min":0.83999999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4750000000000001,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.351},"min":{"1min":0.93500000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6890000000000001,"5min":1.8520000000000001,"15min":2.137},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.494,"15min":1.522},"min":{"1min":1.0840000000000001,"5min":0.77900000000000003,"15min":0.74299999999999999},"max":{"1min":1.6379999999999999,"5min":1.99,"15min":2.452},"last":0.83799999999999997}]},{"osd":5,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":0.94099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0640000000000001,"5min":2.137,"15min":2.137},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.995,"5min":0.87,"15min":0.81699999999999995},"max":{"1min":1.55,"5min":2.282,"15min":2.282},"last":0.98299999999999998}]},{"osd":6,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.6419999999999999,"15min":1.6539999999999999},"min":{"1min":1.1619999999999999,"5min":0.99399999999999999,"15min":0.84499999999999997},"max":{"1min":1.887,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.349},{"interface":"front","average":{"1min":1.583,"5min":1.6060000000000001,"15min":1.5880000000000001},"min":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.78200000000000003},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.0169999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:52:51 2023","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.6850000000000001,"15min":1.7010000000000001},"min":{"1min":1.1200000000000001,"5min":0.97899999999999998,"15min":0.877},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.51},{"interface":"front","average":{"1min":1.643,"5min":1.637,"15min":1.6220000000000001},"min":{"1min":1.1599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.976,"5min":2.1739999999999999,"15min":2.3380000000000001},"last":1.8540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:53:51.679 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:53:51.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:53:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.944+0000 7f3eeeaf3640 1 Processor -- start 2023-10-16T15:53:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.944+0000 7f3eeeaf3640 1 -- start start 2023-10-16T15:53:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eeeaf3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ee815b320 0x7f3ee80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eeeaf3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee815b9d0 con 0x7f3ee806c040 2023-10-16T15:53:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eeeaf3640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee815bcb0 con 0x7f3ee80ad970 2023-10-16T15:53:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eeeaf3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee815b7d0 con 0x7f3ee815b320 2023-10-16T15:53:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eee2f2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ee815b320 0x7f3ee80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eee2f2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ee815b320 0x7f3ee80cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45904/0 (socket says 172.21.15.29:45904) 2023-10-16T15:53:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eedaf1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3ee806c040 0x7f3ee806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56048/0 (socket says 172.21.15.29:56048) 2023-10-16T15:53:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.946+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2639518245 learned_addr learned my addr 172.21.15.29:0/2639518245 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.947+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2639518245 >> v1:172.21.15.170:6789/0 conn(0x7f3ee80ad970 legacy=0x7f3ee80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.947+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2639518245 >> v1:172.21.15.29:6789/0 conn(0x7f3ee806c040 legacy=0x7f3ee806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.947+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2639518245 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee815bf90 con 0x7f3ee815b320 2023-10-16T15:53:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.947+0000 7f3eecaef640 1 -- 172.21.15.29:0/2639518245 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405618067 0 0) 0x7f3ee815b9d0 con 0x7f3ee806c040 2023-10-16T15:53:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.948+0000 7f3eee2f2640 1 --2- 172.21.15.29:0/2639518245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80cd180 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00098a0 tx=0x7f3ee002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e80bd687258d0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.948+0000 7f3eecaef640 1 -- 172.21.15.29:0/2639518245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee002f830 con 0x7f3ee815b320 2023-10-16T15:53:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.948+0000 7f3eecaef640 1 -- 172.21.15.29:0/2639518245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee002f990 con 0x7f3ee815b320 2023-10-16T15:53:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.948+0000 7f3eecaef640 1 -- 172.21.15.29:0/2639518245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee002fd20 con 0x7f3ee815b320 2023-10-16T15:53:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2639518245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 msgr2=0x7f3ee80cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2639518245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80cd180 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00098a0 tx=0x7f3ee002ba10 comp rx=0 tx=0).stop 2023-10-16T15:53:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2639518245 shutdown_connections 2023-10-16T15:53:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2639518245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80cd180 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2639518245 >> 172.21.15.29:0/2639518245 conn(0x7f3ee80ce6a0 msgr2=0x7f3ee80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2639518245 shutdown_connections 2023-10-16T15:53:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.949+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2639518245 wait complete. 2023-10-16T15:53:51.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.951+0000 7f3eeeaf3640 1 Processor -- start 2023-10-16T15:53:51.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.951+0000 7f3eeeaf3640 1 -- start start 2023-10-16T15:53:51.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eeeaf3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee806c040 0x7f3ee8155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eeeaf3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ee80ad970 0x7f3ee80b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eeeaf3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eeeaf3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80af730 con 0x7f3ee815b320 2023-10-16T15:53:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eeeaf3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80aace0 con 0x7f3ee806c040 2023-10-16T15:53:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eeeaf3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80aee20 con 0x7f3ee80ad970 2023-10-16T15:53:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.952+0000 7f3eee2f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80a8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45918/0 (socket says 172.21.15.29:45918) 2023-10-16T15:53:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2350335801 learned_addr learned my addr 172.21.15.29:0/2350335801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eedaf1640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee806c040 0x7f3ee8155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ee80ad970 msgr2=0x7f3ee80b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ee80ad970 0x7f3ee80b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee806c040 msgr2=0x7f3ee8155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:51.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee806c040 0x7f3ee8155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:51.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 -- 172.21.15.29:0/2350335801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00094d0 con 0x7f3ee815b320 2023-10-16T15:53:51.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.953+0000 7f3eee2f2640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80a8a70 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00097f0 tx=0x7f3ee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.954+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee003b040 con 0x7f3ee815b320 2023-10-16T15:53:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.954+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee0037c70 con 0x7f3ee815b320 2023-10-16T15:53:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.954+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee81f0170 con 0x7f3ee815b320 2023-10-16T15:53:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.954+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0004030 con 0x7f3ee815b320 2023-10-16T15:53:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.954+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee80af070 con 0x7f3ee815b320 2023-10-16T15:53:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.955+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee806b330 con 0x7f3ee815b320 2023-10-16T15:53:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.956+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3ee0004190 con 0x7f3ee815b320 2023-10-16T15:53:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.957+0000 7f3ede7fc640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ecc06af90 0x7f3ecc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.957+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f3ee00acb30 con 0x7f3ee815b320 2023-10-16T15:53:51.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.958+0000 7f3eedaf1640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ecc06af90 0x7f3ecc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:51.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.959+0000 7f3eedaf1640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ecc06af90 0x7f3ecc06d450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40012e0 tx=0x7f3ee4005bf0 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:51.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:51.961+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ee00776d0 con 0x7f3ee815b320 2023-10-16T15:53:52.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.190+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ee80af870 con 0x7f3ee815b320 2023-10-16T15:53:52.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.192+0000 7f3ede7fc640 1 -- 172.21.15.29:0/2350335801 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f3ee80af870 con 0x7f3ee815b320 2023-10-16T15:53:52.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:53:52.194 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:53:52.194 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 114 2023-10-16T15:53:52.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.194+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ecc06af90 msgr2=0x7f3ecc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.194+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ecc06af90 0x7f3ecc06d450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40012e0 tx=0x7f3ee4005bf0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.194+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 msgr2=0x7f3ee80a8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.194+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80a8a70 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00097f0 tx=0x7f3ee00046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 shutdown_connections 2023-10-16T15:53:52.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f3ecc06af90 0x7f3ecc06d450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ee815b320 0x7f3ee80a8a70 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3ee80ad970 0x7f3ee80b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 --2- 172.21.15.29:0/2350335801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3ee806c040 0x7f3ee8155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 >> 172.21.15.29:0/2350335801 conn(0x7f3ee80ce6a0 msgr2=0x7f3ee80c8020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 shutdown_connections 2023-10-16T15:53:52.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.195+0000 7f3eeeaf3640 1 -- 172.21.15.29:0/2350335801 wait complete. 2023-10-16T15:53:52.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:53:40.830877+0000', 'modified': '2023-10-16T15:53:43.918016+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8761}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8761': {'gid': 8761, 'name': 'b', 'rank': 0, 'incarnation': 111, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.170:6837/1674117611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6835', 'nonce': 1674117611}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 1674117611}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:53:52.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:53:52.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T15:53:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.858+0000 7ff47f440640 1 Processor -- start 2023-10-16T15:53:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.859+0000 7ff47f440640 1 -- start start 2023-10-16T15:53:52.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.860+0000 7ff47f440640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff47806c040 0x7ff47806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.860+0000 7ff47f440640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff4780cda80 0x7ff47815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.860+0000 7ff47f440640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff47806c960 con 0x7ff4780ad970 2023-10-16T15:53:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.860+0000 7ff47f440640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47815b9d0 con 0x7ff47806c040 2023-10-16T15:53:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.860+0000 7ff47f440640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47815bcb0 con 0x7ff4780cda80 2023-10-16T15:53:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff47806c040 0x7ff47806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff47806c040 0x7ff47806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45932/0 (socket says 172.21.15.29:45932) 2023-10-16T15:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 -- 172.21.15.29:0/895985783 learned_addr learned my addr 172.21.15.29:0/895985783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47ec3f640 1 --2- 172.21.15.29:0/895985783 >> v2:172.21.15.170:3300/0 conn(0x7ff4780cda80 0x7ff47815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 -- 172.21.15.29:0/895985783 >> v2:172.21.15.170:3300/0 conn(0x7ff4780cda80 msgr2=0x7ff47815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 --2- 172.21.15.29:0/895985783 >> v2:172.21.15.170:3300/0 conn(0x7ff4780cda80 0x7ff47815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 -- 172.21.15.29:0/895985783 >> v1:172.21.15.29:6789/0 conn(0x7ff4780ad970 legacy=0x7ff47815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.861+0000 7ff47e43e640 1 -- 172.21.15.29:0/895985783 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47815bf90 con 0x7ff47806c040 2023-10-16T15:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.862+0000 7ff47e43e640 1 --2- 172.21.15.29:0/895985783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff47806c040 0x7ff47806c420 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7ff4780cf300 tx=0x7ff47402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbe3f76297b6eac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47d43c640 1 -- 172.21.15.29:0/895985783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474030860 con 0x7ff47806c040 2023-10-16T15:53:52.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47d43c640 1 -- 172.21.15.29:0/895985783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4740309c0 con 0x7ff47806c040 2023-10-16T15:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47d43c640 1 -- 172.21.15.29:0/895985783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474030d50 con 0x7ff47806c040 2023-10-16T15:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 -- 172.21.15.29:0/895985783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff47806c040 msgr2=0x7ff47806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 --2- 172.21.15.29:0/895985783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff47806c040 0x7ff47806c420 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7ff4780cf300 tx=0x7ff47402b9a0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 -- 172.21.15.29:0/895985783 shutdown_connections 2023-10-16T15:53:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 --2- 172.21.15.29:0/895985783 >> v2:172.21.15.170:3300/0 conn(0x7ff4780cda80 0x7ff47815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 --2- 172.21.15.29:0/895985783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff47806c040 0x7ff47806c420 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 -- 172.21.15.29:0/895985783 >> 172.21.15.29:0/895985783 conn(0x7ff4780ce6a0 msgr2=0x7ff4780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.863+0000 7ff47f440640 1 -- 172.21.15.29:0/895985783 shutdown_connections 2023-10-16T15:53:52.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.864+0000 7ff47f440640 1 -- 172.21.15.29:0/895985783 wait complete. 2023-10-16T15:53:52.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.865+0000 7ff47f440640 1 Processor -- start 2023-10-16T15:53:52.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.865+0000 7ff47f440640 1 -- start start 2023-10-16T15:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.866+0000 7ff47f440640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff47806c040 0x7ff4780ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.866+0000 7ff47f440640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 0x7ff4780aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.866+0000 7ff47f440640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4780cda80 0x7ff4780ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.866+0000 7ff47f440640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780c91d0 con 0x7ff4780ad970 2023-10-16T15:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.866+0000 7ff47f440640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780c94b0 con 0x7ff47806c040 2023-10-16T15:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.866+0000 7ff47f440640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780c9790 con 0x7ff4780cda80 2023-10-16T15:53:52.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 0x7ff4780aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 0x7ff4780aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45938/0 (socket says 172.21.15.29:45938) 2023-10-16T15:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 -- 172.21.15.29:0/3677033592 learned_addr learned my addr 172.21.15.29:0/3677033592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 -- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4780cda80 msgr2=0x7ff4780ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4780cda80 0x7ff4780ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 -- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff47806c040 msgr2=0x7ff4780ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47e43e640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff47806c040 0x7ff4780ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff47806c040 0x7ff4780ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 -- 172.21.15.29:0/3677033592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4740094d0 con 0x7ff4780ad970 2023-10-16T15:53:52.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.867+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 0x7ff4780aed80 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ff46800c400 tx=0x7ff46800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.868+0000 7ff46effd640 1 -- 172.21.15.29:0/3677033592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4680052d0 con 0x7ff4780ad970 2023-10-16T15:53:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.868+0000 7ff46effd640 1 -- 172.21.15.29:0/3677033592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff468005430 con 0x7ff4780ad970 2023-10-16T15:53:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.868+0000 7ff46effd640 1 -- 172.21.15.29:0/3677033592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468003830 con 0x7ff4780ad970 2023-10-16T15:53:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.868+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4781f26e0 con 0x7ff4780ad970 2023-10-16T15:53:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.868+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4780ca3d0 con 0x7ff4780ad970 2023-10-16T15:53:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.870+0000 7ff46effd640 1 -- 172.21.15.29:0/3677033592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff468003990 con 0x7ff4780ad970 2023-10-16T15:53:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.871+0000 7ff46effd640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff45c06afc0 0x7ff45c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.871+0000 7ff46effd640 1 -- 172.21.15.29:0/3677033592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff4680890d0 con 0x7ff4780ad970 2023-10-16T15:53:52.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.872+0000 7ff47e43e640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff45c06afc0 0x7ff45c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.872+0000 7ff483233740 1 Processor -- start 2023-10-16T15:53:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.872+0000 7ff483233740 1 -- start start 2023-10-16T15:53:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff47e43e640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff45c06afc0 0x7ff45c06d480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff474036000 tx=0x7ff47403b040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8820d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff483233740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e8823cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8820d5b0 0x558e8823d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff483233740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8812d7a0 con 0x558e8823a550 2023-10-16T15:53:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8817f740 con 0x558e8823a170 2023-10-16T15:53:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e8817ce30 con 0x558e8820d5b0 2023-10-16T15:53:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff47dc3d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e8823cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff47dc3d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e8823cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45942/0 (socket says 172.21.15.29:45942) 2023-10-16T15:53:52.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff47dc3d640 1 -- 172.21.15.29:0/1527176370 learned_addr learned my addr 172.21.15.29:0/1527176370 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:52.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff47e43e640 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8820d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.873+0000 7ff47ec3f640 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8820d5b0 0x558e8823d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47dc3d640 1 -- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8820d5b0 msgr2=0x558e8823d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8820d5b0 0x558e8823d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47dc3d640 1 -- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 msgr2=0x558e8820d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8820d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47dc3d640 1 -- 172.21.15.29:0/1527176370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e8813c910 con 0x558e8823a550 2023-10-16T15:53:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47ec3f640 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8820d5b0 0x558e8823d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e8823cf00 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7ff46800cfd0 tx=0x7ff4680581e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca3e8ac0774a81e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff45bfff640 1 -- 172.21.15.29:0/1527176370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46805fca0 con 0x558e8823a550 2023-10-16T15:53:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff45bfff640 1 -- 172.21.15.29:0/1527176370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46806f170 con 0x558e8823a550 2023-10-16T15:53:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.874+0000 7ff45bfff640 1 -- 172.21.15.29:0/1527176370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46806f2e0 con 0x558e8823a550 2023-10-16T15:53:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 -- 172.21.15.29:0/1527176370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 msgr2=0x558e8823cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e8823cf00 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7ff46800cfd0 tx=0x7ff4680581e0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 -- 172.21.15.29:0/1527176370 shutdown_connections 2023-10-16T15:53:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e8823cf00 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8820d5b0 0x558e8823d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 --2- 172.21.15.29:0/1527176370 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8820d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 -- 172.21.15.29:0/1527176370 >> 172.21.15.29:0/1527176370 conn(0x558e8820f380 msgr2=0x558e88257f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 -- 172.21.15.29:0/1527176370 shutdown_connections 2023-10-16T15:53:52.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.875+0000 7ff483233740 1 -- 172.21.15.29:0/1527176370 wait complete. 2023-10-16T15:53:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.876+0000 7ff483233740 1 -- start start 2023-10-16T15:53:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.876+0000 7ff483233740 1 Processor -- start 2023-10-16T15:53:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e88239030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8823a550 0x558e88252d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 0x558e88258640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e88178720 con 0x558e882581c0 2023-10-16T15:53:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e88116970 con 0x558e8823a170 2023-10-16T15:53:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e88116610 con 0x558e8823a550 2023-10-16T15:53:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff483233740 10 client.0 _ll_register_callbacks cb 0x558e881389d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:53:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff47ec3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 0x558e88258640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff47ec3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 0x558e88258640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45944/0 (socket says 172.21.15.29:45944) 2023-10-16T15:53:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.877+0000 7ff47ec3f640 1 -- 172.21.15.29:0/1269765500 learned_addr learned my addr 172.21.15.29:0/1269765500 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff47ec3f640 1 -- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8823a550 msgr2=0x558e88252d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff47ec3f640 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8823a550 0x558e88252d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff47ec3f640 1 -- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 msgr2=0x558e88239030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff47ec3f640 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e88239030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff47ec3f640 1 -- 172.21.15.29:0/1269765500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4780ca3d0 con 0x558e882581c0 2023-10-16T15:53:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff47ec3f640 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 0x558e88258640 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7ff4700079a0 tx=0x7ff470007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b038550143cb18e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff4597fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:53:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff4597fa640 1 -- 172.21.15.29:0/1269765500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47000fba0 con 0x558e882581c0 2023-10-16T15:53:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff4597fa640 1 -- 172.21.15.29:0/1269765500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff47000fd00 con 0x558e882581c0 2023-10-16T15:53:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e8819f160 con 0x558e882581c0 2023-10-16T15:53:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff4597fa640 1 -- 172.21.15.29:0/1269765500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff47000fec0 con 0x558e882581c0 2023-10-16T15:53:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff483233740 10 client.11130 fetch_fsmap learned FSMap version 114 2023-10-16T15:53:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.878+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x558e88176da0 con 0x558e882581c0 2023-10-16T15:53:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff4597fa640 1 -- 172.21.15.29:0/1269765500 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e8819f160 con 0x558e882581c0 2023-10-16T15:53:52.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff4597fa640 1 -- 172.21.15.29:0/1269765500 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 114) v1 ==== 3891+0+0 (secure 0 0 0) 0x7ff47001d3f0 con 0x558e882581c0 2023-10-16T15:53:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff483233740 10 client.11130 fetch_fsmap finished waiting for FSMap version 114 2023-10-16T15:53:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff483233740 10 client.11130 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T15:53:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff483233740 20 client.11130 populate_metadata read hostname 'smithi029' 2023-10-16T15:53:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff483233740 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8826bba0 0x558e88275390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff483233740 4 client.11130 mds_command: new command op to 8761 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:53:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.879+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 --> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e8814acc0 con 0x558e8826bba0 2023-10-16T15:53:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.880+0000 7ff47e43e640 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8826bba0 0x558e88275390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.880+0000 7ff47e43e640 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8826bba0 0x558e88275390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47febb5fa985588d server_cookie=dcab4c1f717b9272 in_seq=0 out_seq=0 2023-10-16T15:53:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.880+0000 7ff4597fa640 10 client.11130 ms_handle_connect on v2:172.21.15.170:6835/1674117611 2023-10-16T15:53:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.882+0000 7ff4597fa640 1 -- 172.21.15.29:0/1269765500 <== mds.0 v2:172.21.15.170:6835/1674117611 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558e8814acc0 con 0x558e8826bba0 2023-10-16T15:53:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.882+0000 7ff4597fa640 10 client.11130 handle_command_reply: tid=0 2023-10-16T15:53:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 client.11130 shutdown 2023-10-16T15:53:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 msgr2=0x558e88258640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 0x558e88258640 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7ff4700079a0 tx=0x7ff470007e60 comp rx=0 tx=0).stop 2023-10-16T15:53:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 shutdown_connections 2023-10-16T15:53:52.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8826bba0 0x558e88275390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e882581c0 0x558e88258640 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e8823a550 0x558e88252d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 --2- 172.21.15.29:0/1269765500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e88239030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff4597fa640 0 client.11130 ms_handle_reset on v2:172.21.15.170:6835/1674117611 2023-10-16T15:53:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 >> 172.21.15.29:0/1269765500 conn(0x558e8820d5b0 msgr2=0x558e88211130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 shutdown_connections 2023-10-16T15:53:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 1 -- 172.21.15.29:0/1269765500 wait complete. 2023-10-16T15:53:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.883+0000 7ff483233740 20 client.11130 trim_cache size 0 max 16384 2023-10-16T15:53:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.900+0000 7ff483233740 1 Processor -- start 2023-10-16T15:53:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.900+0000 7ff483233740 1 -- start start 2023-10-16T15:53:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8825b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff483233740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e88252760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 0x558e88252ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff483233740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e87edcf20 con 0x558e8823a550 2023-10-16T15:53:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8824f490 con 0x558e8823a170 2023-10-16T15:53:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e8820d400 con 0x558e882581c0 2023-10-16T15:53:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.901+0000 7ff47dc3d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e88252760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e88252760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45946/0 (socket says 172.21.15.29:45946) 2023-10-16T15:53:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 -- 172.21.15.29:0/4116228532 learned_addr learned my addr 172.21.15.29:0/4116228532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 -- 172.21.15.29:0/4116228532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 msgr2=0x558e88252ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 0x558e88252ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 -- 172.21.15.29:0/4116228532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 msgr2=0x558e8825b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8825b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 -- 172.21.15.29:0/4116228532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e88115b80 con 0x558e8823a550 2023-10-16T15:53:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e88252760 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7ff468055cf0 tx=0x7ff46801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349a5c73b1eececa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff46cff9640 1 -- 172.21.15.29:0/4116228532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46806f2e0 con 0x558e8823a550 2023-10-16T15:53:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff46cff9640 1 -- 172.21.15.29:0/4116228532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4680574e0 con 0x558e8823a550 2023-10-16T15:53:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.902+0000 7ff46cff9640 1 -- 172.21.15.29:0/4116228532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46805fca0 con 0x558e8823a550 2023-10-16T15:53:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 -- 172.21.15.29:0/4116228532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 msgr2=0x558e88252760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e88252760 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7ff468055cf0 tx=0x7ff46801e000 comp rx=0 tx=0).stop 2023-10-16T15:53:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 -- 172.21.15.29:0/4116228532 shutdown_connections 2023-10-16T15:53:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a550 0x558e88252760 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 0x558e88252ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 --2- 172.21.15.29:0/4116228532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a170 0x558e8825b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 -- 172.21.15.29:0/4116228532 >> 172.21.15.29:0/4116228532 conn(0x558e8820d5b0 msgr2=0x558e88238e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 -- 172.21.15.29:0/4116228532 shutdown_connections 2023-10-16T15:53:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.903+0000 7ff483233740 1 -- 172.21.15.29:0/4116228532 wait complete. 2023-10-16T15:53:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.904+0000 7ff483233740 1 -- start start 2023-10-16T15:53:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.904+0000 7ff483233740 1 Processor -- start 2023-10-16T15:53:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 0x558e882bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a550 0x558e88252530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 0x558e88252a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e88210c40 con 0x558e8823a170 2023-10-16T15:53:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e88241290 con 0x558e8823a550 2023-10-16T15:53:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e882c0280 con 0x558e882581c0 2023-10-16T15:53:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff483233740 10 client.0 _ll_register_callbacks cb 0x7ff4700097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:53:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff47e43e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 0x558e882bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff47e43e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 0x558e882bf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45948/0 (socket says 172.21.15.29:45948) 2023-10-16T15:53:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff47e43e640 1 -- 172.21.15.29:0/1849007876 learned_addr learned my addr 172.21.15.29:0/1849007876 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff47e43e640 1 -- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 msgr2=0x558e88252a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.905+0000 7ff47e43e640 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 0x558e88252a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff47e43e640 1 -- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a550 msgr2=0x558e88252530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff47e43e640 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a550 0x558e88252530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff47e43e640 1 -- 172.21.15.29:0/1849007876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e8813c910 con 0x558e8823a170 2023-10-16T15:53:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff47e43e640 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 0x558e882bf7b0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7ff47402bfb0 tx=0x7ff4740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489fff25ddd18414 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff4597fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:53:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff4597fa640 1 -- 172.21.15.29:0/1849007876 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474053260 con 0x558e8823a170 2023-10-16T15:53:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff4597fa640 1 -- 172.21.15.29:0/1849007876 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff47403e040 con 0x558e8823a170 2023-10-16T15:53:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff46806f170 con 0x558e8823a170 2023-10-16T15:53:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff4597fa640 1 -- 172.21.15.29:0/1849007876 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4740533c0 con 0x558e8823a170 2023-10-16T15:53:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff483233740 10 client.11136 fetch_fsmap learned FSMap version 114 2023-10-16T15:53:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.906+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x558e8813b170 con 0x558e8823a170 2023-10-16T15:53:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff4597fa640 1 -- 172.21.15.29:0/1849007876 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46806f170 con 0x558e8823a170 2023-10-16T15:53:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff4597fa640 1 -- 172.21.15.29:0/1849007876 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 114) v1 ==== 3891+0+0 (secure 0 0 0) 0x7ff474042050 con 0x558e8823a170 2023-10-16T15:53:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff483233740 10 client.11136 fetch_fsmap finished waiting for FSMap version 114 2023-10-16T15:53:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff483233740 10 client.11136 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T15:53:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff483233740 20 client.11136 populate_metadata read hostname 'smithi029' 2023-10-16T15:53:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff483233740 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8823e770 0x558e8823eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff483233740 4 client.11136 mds_command: new command op to 8761 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:53:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.907+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 --> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558e88171700 con 0x558e8823e770 2023-10-16T15:53:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.908+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8823e770 0x558e8823eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.908+0000 7ff47dc3d640 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8823e770 0x558e8823eb50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac593c387b9f0fcf server_cookie=6930b0ed529e1fa6 in_seq=0 out_seq=0 2023-10-16T15:53:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.908+0000 7ff4597fa640 10 client.11136 ms_handle_connect on v2:172.21.15.170:6835/1674117611 2023-10-16T15:53:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff4597fa640 1 -- 172.21.15.29:0/1849007876 <== mds.0 v2:172.21.15.170:6835/1674117611 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x558e88171700 con 0x558e8823e770 2023-10-16T15:53:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff4597fa640 10 client.11136 handle_command_reply: tid=0 2023-10-16T15:53:52.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 client.11136 shutdown 2023-10-16T15:53:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 msgr2=0x558e882bf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 0x558e882bf7b0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7ff47402bfb0 tx=0x7ff4740046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 shutdown_connections 2023-10-16T15:53:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:6835/1674117611,v1:172.21.15.170:6837/1674117611] conn(0x558e8823e770 0x558e8823eb50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x558e882581c0 0x558e88252a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558e8823a550 0x558e88252530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 --2- 172.21.15.29:0/1849007876 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558e8823a170 0x558e882bf7b0 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff4597fa640 0 client.11136 ms_handle_reset on v2:172.21.15.170:6835/1674117611 2023-10-16T15:53:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 >> 172.21.15.29:0/1849007876 conn(0x558e8820d5b0 msgr2=0x558e88238e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 shutdown_connections 2023-10-16T15:53:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.909+0000 7ff483233740 1 -- 172.21.15.29:0/1849007876 wait complete. 2023-10-16T15:53:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff483233740 20 client.11136 trim_cache size 0 max 16384 2023-10-16T15:53:52.933 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:53:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:53:52.933 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:53:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8761, 2023-10-16T15:53:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:53:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:53:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "backup_fs", 2023-10-16T15:53:52.934 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 11.537702778, 2023-10-16T15:53:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 114, 2023-10-16T15:53:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 229, 2023-10-16T15:53:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 229, 2023-10-16T15:53:52.935 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 31.514453477 2023-10-16T15:53:52.935 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:53:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff45c06afc0 msgr2=0x7ff45c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff45c06afc0 0x7ff45c06d480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff474036000 tx=0x7ff47403b040 comp rx=0 tx=0).stop 2023-10-16T15:53:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 msgr2=0x7ff4780aed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 0x7ff4780aed80 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ff46800c400 tx=0x7ff46800c8c0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 shutdown_connections 2023-10-16T15:53:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7ff45c06afc0 0x7ff45c06d480 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4780ad970 0x7ff4780aed80 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff4780cda80 0x7ff4780ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 --2- 172.21.15.29:0/3677033592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff47806c040 0x7ff4780ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 >> 172.21.15.29:0/3677033592 conn(0x7ff4780ce6a0 msgr2=0x7ff47815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 shutdown_connections 2023-10-16T15:53:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:52.910+0000 7ff47f440640 1 -- 172.21.15.29:0/3677033592 wait complete. 2023-10-16T15:53:52.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T15:53:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.168+0000 7fa77f3e9640 1 Processor -- start 2023-10-16T15:53:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.168+0000 7fa77f3e9640 1 -- start start 2023-10-16T15:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.170+0000 7fa77f3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa778069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.170+0000 7fa77f3e9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa7780caac0 0x7fa77806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.170+0000 7fa77f3e9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7780cb030 con 0x7fa7780c9ca0 2023-10-16T15:53:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.170+0000 7fa77f3e9640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7780c91c0 con 0x7fa7780caac0 2023-10-16T15:53:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.170+0000 7fa77f3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7780c94a0 con 0x7fa77815ad70 2023-10-16T15:53:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.171+0000 7fa77d185640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa778069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.171+0000 7fa77d185640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa778069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:50668/0 (socket says 172.21.15.29:50668) 2023-10-16T15:53:53.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.171+0000 7fa77d185640 1 -- 172.21.15.29:0/1455553401 learned_addr learned my addr 172.21.15.29:0/1455553401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.171+0000 7fa77d986640 1 --2- 172.21.15.29:0/1455553401 >> v2:172.21.15.170:3300/0 conn(0x7fa7780caac0 0x7fa77806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.172+0000 7fa77d185640 1 -- 172.21.15.29:0/1455553401 >> v2:172.21.15.170:3300/0 conn(0x7fa7780caac0 msgr2=0x7fa77806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.172+0000 7fa77d185640 1 --2- 172.21.15.29:0/1455553401 >> v2:172.21.15.170:3300/0 conn(0x7fa7780caac0 0x7fa77806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.172+0000 7fa77d185640 1 -- 172.21.15.29:0/1455553401 >> v1:172.21.15.170:6789/0 conn(0x7fa7780c9ca0 legacy=0x7fa7780ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.172+0000 7fa77d185640 1 -- 172.21.15.29:0/1455553401 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7780c9780 con 0x7fa77815ad70 2023-10-16T15:53:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.172+0000 7fa77d986640 1 --2- 172.21.15.29:0/1455553401 >> v2:172.21.15.170:3300/0 conn(0x7fa7780caac0 0x7fa77806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.173+0000 7fa77d185640 1 --2- 172.21.15.29:0/1455553401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa778069da0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa774009870 tx=0x7fa77402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dd1803bd33032c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.174+0000 7fa76ffff640 1 -- 172.21.15.29:0/1455553401 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774030840 con 0x7fa77815ad70 2023-10-16T15:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.174+0000 7fa76ffff640 1 -- 172.21.15.29:0/1455553401 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7740309a0 con 0x7fa77815ad70 2023-10-16T15:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.174+0000 7fa76ffff640 1 -- 172.21.15.29:0/1455553401 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774030d30 con 0x7fa77815ad70 2023-10-16T15:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 -- 172.21.15.29:0/1455553401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 msgr2=0x7fa778069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/1455553401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa778069da0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa774009870 tx=0x7fa77402b980 comp rx=0 tx=0).stop 2023-10-16T15:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 -- 172.21.15.29:0/1455553401 shutdown_connections 2023-10-16T15:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/1455553401 >> v2:172.21.15.170:3300/0 conn(0x7fa7780caac0 0x7fa77806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/1455553401 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa778069da0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 -- 172.21.15.29:0/1455553401 >> 172.21.15.29:0/1455553401 conn(0x7fa7780ce9b0 msgr2=0x7fa7780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 -- 172.21.15.29:0/1455553401 shutdown_connections 2023-10-16T15:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.175+0000 7fa77f3e9640 1 -- 172.21.15.29:0/1455553401 wait complete. 2023-10-16T15:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.177+0000 7fa77f3e9640 1 Processor -- start 2023-10-16T15:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.178+0000 7fa77f3e9640 1 -- start start 2023-10-16T15:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77f3e9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7780c9ca0 0x7fa77814de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77f3e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 0x7fa7780ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77f3e9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa7780acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77f3e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7780b2370 con 0x7fa7780caac0 2023-10-16T15:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77f3e9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7780b2650 con 0x7fa7780c9ca0 2023-10-16T15:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77f3e9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7780b2930 con 0x7fa77815ad70 2023-10-16T15:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77c984640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 0x7fa7780ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77c984640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 0x7fa7780ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45960/0 (socket says 172.21.15.29:45960) 2023-10-16T15:53:53.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.179+0000 7fa77c984640 1 -- 172.21.15.29:0/2578272244 learned_addr learned my addr 172.21.15.29:0/2578272244 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:53.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77d986640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa7780acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77d185640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7780c9ca0 0x7fa77814de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77c984640 1 -- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 msgr2=0x7fa7780acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77c984640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa7780acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77c984640 1 -- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7780c9ca0 msgr2=0x7fa77814de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77c984640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7780c9ca0 0x7fa77814de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77c984640 1 -- 172.21.15.29:0/2578272244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7740094d0 con 0x7fa7780caac0 2023-10-16T15:53:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77d185640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7780c9ca0 0x7fa77814de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:53.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.180+0000 7fa77c984640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 0x7fa7780ac890 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fa76800c400 tx=0x7fa76800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:53.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.181+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7680052d0 con 0x7fa7780caac0 2023-10-16T15:53:53.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.181+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa768005450 con 0x7fa7780caac0 2023-10-16T15:53:53.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.181+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768007260 con 0x7fa7780caac0 2023-10-16T15:53:53.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.181+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7780b2c10 con 0x7fa7780caac0 2023-10-16T15:53:53.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.181+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77814e380 con 0x7fa7780caac0 2023-10-16T15:53:53.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.182+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7780ad490 con 0x7fa7780caac0 2023-10-16T15:53:53.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.183+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa7680073c0 con 0x7fa7780caac0 2023-10-16T15:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.184+0000 7fa76dffb640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa75c06afc0 0x7fa75c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.185+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fa768088fa0 con 0x7fa7780caac0 2023-10-16T15:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.185+0000 7fa77d185640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa75c06afc0 0x7fa75c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.186+0000 7fa77d185640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa75c06afc0 0x7fa75c06d480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa774036000 tx=0x7fa77403b040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.190+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa768053b20 con 0x7fa7780caac0 2023-10-16T15:53:53.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.414+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7780b27c0 con 0x7fa75c06afc0 2023-10-16T15:53:53.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.418+0000 7fa76dffb640 1 -- 172.21.15.29:0/2578272244 <== mgr.8608 v2:172.21.15.170:6832/3980026359 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fa7780b27c0 con 0x7fa75c06afc0 2023-10-16T15:53:53.419 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:53:53.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T15:53:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:53:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:53:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "backup_fs" 2023-10-16T15:53:53.420 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:53:53.421 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:53:53.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa75c06afc0 msgr2=0x7fa75c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa75c06afc0 0x7fa75c06d480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa774036000 tx=0x7fa77403b040 comp rx=0 tx=0).stop 2023-10-16T15:53:53.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 msgr2=0x7fa7780ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 0x7fa7780ac890 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fa76800c400 tx=0x7fa76800c8c0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 shutdown_connections 2023-10-16T15:53:53.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fa75c06afc0 0x7fa75c06d480 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7780caac0 0x7fa7780ac890 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa77815ad70 0x7fa7780acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 --2- 172.21.15.29:0/2578272244 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7780c9ca0 0x7fa77814de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 >> 172.21.15.29:0/2578272244 conn(0x7fa7780ce9b0 msgr2=0x7fa7780ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:53.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 shutdown_connections 2023-10-16T15:53:53.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.421+0000 7fa77f3e9640 1 -- 172.21.15.29:0/2578272244 wait complete. 2023-10-16T15:53:53.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T15:53:53.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.671+0000 7fe8b096b640 1 Processor -- start 2023-10-16T15:53:53.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.671+0000 7fe8b096b640 1 -- start start 2023-10-16T15:53:53.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8b096b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe8ac0c9060 0x7fe8ac06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8b096b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 0x7fe8ac06f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8b096b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac0ca560 con 0x7fe8ac06c9e0 2023-10-16T15:53:53.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8b096b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c9950 con 0x7fe8ac0c9060 2023-10-16T15:53:53.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8b096b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c9c30 con 0x7fe8ac0c9440 2023-10-16T15:53:53.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8a3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe8ac06c9e0 0x7fe8ac164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56078/0 (socket says 172.21.15.29:56078) 2023-10-16T15:53:53.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.673+0000 7fe8a3fff640 1 -- 172.21.15.29:0/422366662 learned_addr learned my addr 172.21.15.29:0/422366662 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:53.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aad9d640 1 --2- 172.21.15.29:0/422366662 >> v2:172.21.15.170:3300/0 conn(0x7fe8ac0c9060 0x7fe8ac06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8ab59e640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 0x7fe8ac06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524296395 0 0) 0x7fe8ac0ca560 con 0x7fe8ac06c9e0 2023-10-16T15:53:53.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898003520 con 0x7fe8ac06c9e0 2023-10-16T15:53:53.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047839625 0 0) 0x7fe898003520 con 0x7fe8ac06c9e0 2023-10-16T15:53:53.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 msgr2=0x7fe8ac06f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:53.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 0x7fe8ac06f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 >> v2:172.21.15.170:3300/0 conn(0x7fe8ac0c9060 msgr2=0x7fe8ac06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 --2- 172.21.15.29:0/422366662 >> v2:172.21.15.170:3300/0 conn(0x7fe8ac0c9060 0x7fe8ac06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ac0c9f10 con 0x7fe8ac06c9e0 2023-10-16T15:53:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe894001670 con 0x7fe8ac06c9e0 2023-10-16T15:53:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 >> v1:172.21.15.29:6789/0 conn(0x7fe8ac06c9e0 legacy=0x7fe8ac164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 0x7fe898003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898003f40 0x7fe8980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe898005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c9950 con 0x7fe898004d00 2023-10-16T15:53:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c9c30 con 0x7fe898003f40 2023-10-16T15:53:53.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.674+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89800b980 con 0x7fe8ac0c9440 2023-10-16T15:53:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8ab59e640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe898005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8aad9d640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 0x7fe898003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898003f40 0x7fe8980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8ab59e640 1 -- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 msgr2=0x7fe898003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:53.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8ab59e640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac0c9440 0x7fe898003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8ab59e640 1 -- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898003f40 msgr2=0x7fe8980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:53.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8ab59e640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898003f40 0x7fe8980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8ab59e640 1 -- 172.21.15.29:0/422366662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898003520 con 0x7fe898004d00 2023-10-16T15:53:53.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.675+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe898003f40 0x7fe8980047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:53:53.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.677+0000 7fe8ab59e640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe898005180 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009fc0 tx=0x7fe8980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c9f0d87a9fc293 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:53.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.677+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a402cbd0 con 0x7fe898004d00 2023-10-16T15:53:53.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.677+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a400f040 con 0x7fe898004d00 2023-10-16T15:53:53.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.677+0000 7fe8aa59c640 1 -- 172.21.15.29:0/422366662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4034440 con 0x7fe898004d00 2023-10-16T15:53:53.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8b096b640 1 -- 172.21.15.29:0/422366662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 msgr2=0x7fe898005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8b096b640 1 --2- 172.21.15.29:0/422366662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe898005180 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009fc0 tx=0x7fe8980057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8ab59e640 1 -- 172.21.15.29:0/422366662 reap_dead start 2023-10-16T15:53:53.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8b096b640 1 -- 172.21.15.29:0/422366662 shutdown_connections 2023-10-16T15:53:53.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8b096b640 1 -- 172.21.15.29:0/422366662 >> 172.21.15.29:0/422366662 conn(0x7fe8ac0ce6a0 msgr2=0x7fe8ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:53.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8b096b640 1 -- 172.21.15.29:0/422366662 shutdown_connections 2023-10-16T15:53:53.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.678+0000 7fe8b096b640 1 -- 172.21.15.29:0/422366662 wait complete. 2023-10-16T15:53:53.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.680+0000 7fe8b096b640 1 Processor -- start 2023-10-16T15:53:53.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.680+0000 7fe8b096b640 1 -- start start 2023-10-16T15:53:53.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8b096b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe8ac0b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8b096b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac162050 0x7fe8ac15f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8b096b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8ac0c4220 0x7fe8ac0c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8b096b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c3180 con 0x7fe898004d00 2023-10-16T15:53:53.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8b096b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c3460 con 0x7fe8ac0c4220 2023-10-16T15:53:53.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8b096b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac0c3740 con 0x7fe8ac162050 2023-10-16T15:53:53.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe8ac0b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe8ac0b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45978/0 (socket says 172.21.15.29:45978) 2023-10-16T15:53:53.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.681+0000 7fe8aad9d640 1 -- 172.21.15.29:0/194213533 learned_addr learned my addr 172.21.15.29:0/194213533 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:53.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac162050 0x7fe8ac15f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8aad9d640 1 -- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac162050 msgr2=0x7fe8ac15f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8aad9d640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac162050 0x7fe8ac15f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8aad9d640 1 -- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8ac0c4220 msgr2=0x7fe8ac0c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:53.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8aad9d640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8ac0c4220 0x7fe8ac0c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:53.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8aad9d640 1 -- 172.21.15.29:0/194213533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ac0c3a20 con 0x7fe898004d00 2023-10-16T15:53:53.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8aad9d640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe8ac0b38d0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0116d0 tx=0x7fe89c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:53.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c00f9c0 con 0x7fe898004d00 2023-10-16T15:53:53.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c00fb20 con 0x7fe898004d00 2023-10-16T15:53:53.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c00fe90 con 0x7fe898004d00 2023-10-16T15:53:53.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.682+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ac0c3d00 con 0x7fe898004d00 2023-10-16T15:53:53.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.683+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ac1624d0 con 0x7fe898004d00 2023-10-16T15:53:53.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.684+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ac0c3e90 con 0x7fe898004d00 2023-10-16T15:53:53.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.685+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe89c015460 con 0x7fe898004d00 2023-10-16T15:53:53.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.685+0000 7fe8a37fe640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe88c06afc0 0x7fe88c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:53.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.686+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fe89c089f50 con 0x7fe898004d00 2023-10-16T15:53:53.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.686+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe88c06afc0 0x7fe88c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:53.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.687+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe88c06afc0 0x7fe88c06d480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac06cde0 tx=0x7fe894009040 comp rx=0 tx=0).ready entity=mgr.8608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:53.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.690+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe89c054cb0 con 0x7fe898004d00 2023-10-16T15:53:53.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:53.926+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe8ac04e9b0 con 0x7fe898004d00 2023-10-16T15:53:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.279+0000 7fe8a37fe640 1 -- 172.21.15.29:0/194213533 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v129) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe89c0220a0 con 0x7fe898004d00 2023-10-16T15:53:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe88c06afc0 msgr2=0x7fe88c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe88c06afc0 0x7fe88c06d480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac06cde0 tx=0x7fe894009040 comp rx=0 tx=0).stop 2023-10-16T15:53:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 msgr2=0x7fe8ac0b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe8ac0b38d0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0116d0 tx=0x7fe89c011b90 comp rx=0 tx=0).stop 2023-10-16T15:53:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 shutdown_connections 2023-10-16T15:53:54.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7fe88c06afc0 0x7fe88c06d480 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe8ac0c4220 0x7fe8ac0c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8ac162050 0x7fe8ac15f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 --2- 172.21.15.29:0/194213533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898004d00 0x7fe8ac0b38d0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.284+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 >> 172.21.15.29:0/194213533 conn(0x7fe8ac0ce6a0 msgr2=0x7fe8ac160b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.286+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 shutdown_connections 2023-10-16T15:53:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.286+0000 7fe8b096b640 1 -- 172.21.15.29:0/194213533 wait complete. 2023-10-16T15:53:54.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T15:53:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.502+0000 7f82fb9b0640 1 Processor -- start 2023-10-16T15:53:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.502+0000 7f82fb9b0640 1 -- start start 2023-10-16T15:53:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.503+0000 7f82fb9b0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 0x7f82f40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.503+0000 7f82fb9b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 0x7f82f4161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.503+0000 7f82fb9b0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40b5d90 con 0x7f82f4161990 2023-10-16T15:53:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.503+0000 7f82fb9b0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40d2b50 con 0x7f82f415d030 2023-10-16T15:53:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.503+0000 7f82fb9b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4160fc0 con 0x7f82f415d410 2023-10-16T15:53:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.504+0000 7f82f974c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 0x7f82f40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.504+0000 7f82f8f4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 0x7f82f4161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.504+0000 7f82f8f4b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 0x7f82f4161450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:44182/0 (socket says 172.21.15.29:44182) 2023-10-16T15:53:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.504+0000 7f82f974c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 0x7f82f40d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:45962/0 (socket says 172.21.15.29:45962) 2023-10-16T15:53:54.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.504+0000 7f82f8f4b640 1 -- 172.21.15.29:0/1922660891 learned_addr learned my addr 172.21.15.29:0/1922660891 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:54.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.505+0000 7f82f8f4b640 1 -- 172.21.15.29:0/1922660891 >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 msgr2=0x7f82f40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:54.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.505+0000 7f82f8f4b640 1 --2- 172.21.15.29:0/1922660891 >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 0x7f82f40d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.505+0000 7f82f8f4b640 1 -- 172.21.15.29:0/1922660891 >> v1:172.21.15.170:6789/0 conn(0x7f82f4161990 legacy=0x7f82f40cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.505+0000 7f82f8f4b640 1 -- 172.21.15.29:0/1922660891 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40d25d0 con 0x7f82f415d410 2023-10-16T15:53:54.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.505+0000 7f82f974c640 1 --2- 172.21.15.29:0/1922660891 >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 0x7f82f40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:53:54.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.507+0000 7f82f8f4b640 1 --2- 172.21.15.29:0/1922660891 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 0x7f82f4161450 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009870 tx=0x7f82e402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde84adec379980b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:54.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82ebfff640 1 -- 172.21.15.29:0/1922660891 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e402f830 con 0x7f82f415d410 2023-10-16T15:53:54.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82ebfff640 1 -- 172.21.15.29:0/1922660891 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e402f990 con 0x7f82f415d410 2023-10-16T15:53:54.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82ebfff640 1 -- 172.21.15.29:0/1922660891 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e402fd20 con 0x7f82f415d410 2023-10-16T15:53:54.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82fb9b0640 1 -- 172.21.15.29:0/1922660891 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 msgr2=0x7f82f4161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/1922660891 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 0x7f82f4161450 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009870 tx=0x7f82e402ba10 comp rx=0 tx=0).stop 2023-10-16T15:53:54.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82fb9b0640 1 -- 172.21.15.29:0/1922660891 shutdown_connections 2023-10-16T15:53:54.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/1922660891 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f415d410 0x7f82f4161450 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/1922660891 >> v2:172.21.15.170:3300/0 conn(0x7f82f415d030 0x7f82f40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.508+0000 7f82fb9b0640 1 -- 172.21.15.29:0/1922660891 >> 172.21.15.29:0/1922660891 conn(0x7f82f40cf720 msgr2=0x7f82f40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:54.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.509+0000 7f82fb9b0640 1 -- 172.21.15.29:0/1922660891 shutdown_connections 2023-10-16T15:53:54.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.509+0000 7f82fb9b0640 1 -- 172.21.15.29:0/1922660891 wait complete. 2023-10-16T15:53:54.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.510+0000 7f82fb9b0640 1 Processor -- start 2023-10-16T15:53:54.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.511+0000 7f82fb9b0640 1 -- start start 2023-10-16T15:53:54.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.511+0000 7f82fb9b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f415d030 0x7f82f40cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.511+0000 7f82fb9b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 0x7f82f41f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.511+0000 7f82fb9b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f4161990 0x7f82f41f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82fb9b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40cb830 con 0x7f82f415d410 2023-10-16T15:53:54.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82fb9b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40cbb10 con 0x7f82f415d030 2023-10-16T15:53:54.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82fb9b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40cbdf0 con 0x7f82f4161990 2023-10-16T15:53:54.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 0x7f82f41f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 0x7f82f41f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48616/0 (socket says 172.21.15.29:48616) 2023-10-16T15:53:54.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 -- 172.21.15.29:0/390598039 learned_addr learned my addr 172.21.15.29:0/390598039 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:54.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f974c640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f415d030 0x7f82f40cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f9f4d640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f4161990 0x7f82f41f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 -- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f4161990 msgr2=0x7f82f41f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f4161990 0x7f82f41f32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 -- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f415d030 msgr2=0x7f82f40cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f415d030 0x7f82f40cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.512+0000 7f82f8f4b640 1 -- 172.21.15.29:0/390598039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0009400 con 0x7f82f415d410 2023-10-16T15:53:54.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.513+0000 7f82f8f4b640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 0x7f82f41f2db0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000c00 tx=0x7f82e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:54.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.513+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4037c40 con 0x7f82f415d410 2023-10-16T15:53:54.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.513+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e4037da0 con 0x7f82f415d410 2023-10-16T15:53:54.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.513+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4041640 con 0x7f82f415d410 2023-10-16T15:53:54.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.513+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e40094d0 con 0x7f82f415d410 2023-10-16T15:53:54.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.513+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40cc3b0 con 0x7f82f415d410 2023-10-16T15:53:54.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.515+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f404e160 con 0x7f82f415d410 2023-10-16T15:53:54.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.516+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 129) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f82e40417a0 con 0x7f82f415d410 2023-10-16T15:53:54.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.518+0000 7f82e9ffb640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 0x7f82d806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.518+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f82e403c070 con 0x7f82f415d410 2023-10-16T15:53:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.518+0000 7f82f974c640 1 -- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 msgr2=0x7f82d806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.518+0000 7f82f974c640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 0x7f82d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:54.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.523+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f82e4077440 con 0x7f82f415d410 2023-10-16T15:53:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.719+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f82f40c9aa0 con 0x7f82f415d410 2023-10-16T15:53:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.719+0000 7f82f974c640 1 -- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 msgr2=0x7f82d806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.719+0000 7f82f974c640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 0x7f82d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.723+0000 7f82e9ffb640 1 -- 172.21.15.29:0/390598039 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f82e407aa60 con 0x7f82f415d410 2023-10-16T15:53:54.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 msgr2=0x7f82d806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:54.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 0x7f82d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 msgr2=0x7f82f41f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 0x7f82f41f2db0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000c00 tx=0x7f82e4004730 comp rx=0 tx=0).stop 2023-10-16T15:53:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 shutdown_connections 2023-10-16T15:53:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f82d806afe0 0x7f82d806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82f415d410 0x7f82f41f2db0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f82f4161990 0x7f82f41f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 --2- 172.21.15.29:0/390598039 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f82f415d030 0x7f82f40cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 >> 172.21.15.29:0/390598039 conn(0x7f82f40cf720 msgr2=0x7f82f406bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 shutdown_connections 2023-10-16T15:53:54.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.726+0000 7f82fb9b0640 1 -- 172.21.15.29:0/390598039 wait complete. 2023-10-16T15:53:54.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T15:53:54.813 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:54.811+0000 7fcf5b40e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:53:54.830 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:54.829+0000 7f8aa062b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:53:54.866 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:54.864+0000 7fcf5b40e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:53:54.889 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:54.888+0000 7f8aa062b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:53:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.961+0000 7f666013c640 1 Processor -- start 2023-10-16T15:53:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.961+0000 7f666013c640 1 -- start start 2023-10-16T15:53:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.963+0000 7f666013c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66580ac890 0x7f66580c9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.963+0000 7f666013c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f66580c9d40 0x7f6658160a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.963+0000 7f666013c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665806ad30 con 0x7f66580cb100 2023-10-16T15:53:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.963+0000 7f666013c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665806a760 con 0x7f66580ac890 2023-10-16T15:53:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.963+0000 7f666013c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66580ae400 con 0x7f66580c9d40 2023-10-16T15:53:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.964+0000 7f665d6d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66580ac890 0x7f66580c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.964+0000 7f665d6d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66580ac890 0x7f66580c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48622/0 (socket says 172.21.15.29:48622) 2023-10-16T15:53:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.964+0000 7f665e6d9640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f66580c9d40 0x7f6658160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.964+0000 7f665ded8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f66580cb100 0x7f665806b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36808/0 (socket says 172.21.15.29:36808) 2023-10-16T15:53:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.964+0000 7f665d6d7640 1 -- 172.21.15.29:0/25939747 learned_addr learned my addr 172.21.15.29:0/25939747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.964+0000 7f665d6d7640 1 -- 172.21.15.29:0/25939747 >> v2:172.21.15.170:3300/0 conn(0x7f66580c9d40 msgr2=0x7f6658160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.965+0000 7f665d6d7640 1 --2- 172.21.15.29:0/25939747 >> v2:172.21.15.170:3300/0 conn(0x7f66580c9d40 0x7f6658160a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.965+0000 7f665d6d7640 1 -- 172.21.15.29:0/25939747 >> v1:172.21.15.29:6789/0 conn(0x7f66580cb100 legacy=0x7f665806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.965+0000 7f665d6d7640 1 -- 172.21.15.29:0/25939747 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66580ae6e0 con 0x7f66580ac890 2023-10-16T15:53:54.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.966+0000 7f665d6d7640 1 --2- 172.21.15.29:0/25939747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580ac890 0x7f66580c9800 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f665806e8b0 tx=0x7f664802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ff05fd39c9d5aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.966+0000 7f665ced6640 1 -- 172.21.15.29:0/25939747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648030880 con 0x7f66580ac890 2023-10-16T15:53:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.966+0000 7f665ced6640 1 -- 172.21.15.29:0/25939747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66480309e0 con 0x7f66580ac890 2023-10-16T15:53:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.966+0000 7f665ced6640 1 -- 172.21.15.29:0/25939747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648030d70 con 0x7f66580ac890 2023-10-16T15:53:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.966+0000 7f666013c640 1 -- 172.21.15.29:0/25939747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580ac890 msgr2=0x7f66580c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 --2- 172.21.15.29:0/25939747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580ac890 0x7f66580c9800 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f665806e8b0 tx=0x7f664802b9c0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 -- 172.21.15.29:0/25939747 shutdown_connections 2023-10-16T15:53:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 --2- 172.21.15.29:0/25939747 >> v2:172.21.15.170:3300/0 conn(0x7f66580c9d40 0x7f6658160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 --2- 172.21.15.29:0/25939747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580ac890 0x7f66580c9800 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 -- 172.21.15.29:0/25939747 >> 172.21.15.29:0/25939747 conn(0x7f665806bc10 msgr2=0x7f665806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 -- 172.21.15.29:0/25939747 shutdown_connections 2023-10-16T15:53:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.967+0000 7f666013c640 1 -- 172.21.15.29:0/25939747 wait complete. 2023-10-16T15:53:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.968+0000 7f666013c640 1 Processor -- start 2023-10-16T15:53:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.969+0000 7f666013c640 1 -- start start 2023-10-16T15:53:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.969+0000 7f666013c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66580ac890 0x7f665815d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.969+0000 7f666013c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 0x7f66580c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.969+0000 7f666013c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66580cb100 0x7f66580c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f666013c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665815b900 con 0x7f66580c9d40 2023-10-16T15:53:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f666013c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66580c8b00 con 0x7f66580ac890 2023-10-16T15:53:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f666013c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66580c8de0 con 0x7f66580cb100 2023-10-16T15:53:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 0x7f66580c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 0x7f66580c3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48630/0 (socket says 172.21.15.29:48630) 2023-10-16T15:53:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 -- 172.21.15.29:0/191436899 learned_addr learned my addr 172.21.15.29:0/191436899 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665ded8640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66580ac890 0x7f665815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665e6d9640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66580cb100 0x7f66580c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 -- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66580cb100 msgr2=0x7f66580c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66580cb100 0x7f66580c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 -- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66580ac890 msgr2=0x7f665815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66580ac890 0x7f665815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.970+0000 7f665d6d7640 1 -- 172.21.15.29:0/191436899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66480094d0 con 0x7f66580c9d40 2023-10-16T15:53:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.971+0000 7f665d6d7640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 0x7f66580c3cc0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f664802ba00 tx=0x7f6648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.971+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648039d60 con 0x7f66580c9d40 2023-10-16T15:53:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.971+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648043400 con 0x7f66580c9d40 2023-10-16T15:53:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.971+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66580c90c0 con 0x7f66580c9d40 2023-10-16T15:53:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.971+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480437b0 con 0x7f66580c9d40 2023-10-16T15:53:54.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.971+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665815a810 con 0x7f66580c9d40 2023-10-16T15:53:54.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.973+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66580c8c40 con 0x7f66580c9d40 2023-10-16T15:53:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.973+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 129) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6648043910 con 0x7f66580c9d40 2023-10-16T15:53:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.975+0000 7f664e7fc640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 0x7f663c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.975+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f664803e070 con 0x7f66580c9d40 2023-10-16T15:53:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.975+0000 7f665ded8640 1 -- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 msgr2=0x7f663c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.975+0000 7f665ded8640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 0x7f663c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:54.979+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66580c8c40 con 0x7f66580c9d40 2023-10-16T15:53:55.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.171+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f665815ba40 con 0x7f66580c9d40 2023-10-16T15:53:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.176+0000 7f664e7fc640 1 -- 172.21.15.29:0/191436899 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f665815ba40 con 0x7f66580c9d40 2023-10-16T15:53:55.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.176+0000 7f665ded8640 1 -- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 msgr2=0x7f663c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:55.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.176+0000 7f665ded8640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 0x7f663c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 msgr2=0x7f663c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 0x7f663c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 msgr2=0x7f66580c3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 0x7f66580c3cc0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f664802ba00 tx=0x7f6648004730 comp rx=0 tx=0).stop 2023-10-16T15:53:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 shutdown_connections 2023-10-16T15:53:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f663c06afe0 0x7f663c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66580c9d40 0x7f66580c3cc0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66580cb100 0x7f66580c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 --2- 172.21.15.29:0/191436899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66580ac890 0x7f665815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 >> 172.21.15.29:0/191436899 conn(0x7f665806bc10 msgr2=0x7f6658069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:55.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 shutdown_connections 2023-10-16T15:53:55.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.178+0000 7f666013c640 1 -- 172.21.15.29:0/191436899 wait complete. 2023-10-16T15:53:55.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:53:55.198 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T15:53:55.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T15:53:55.315 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.314+0000 7fcf5b40e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:53:55.367 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.366+0000 7f8aa062b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:53:55.368 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.366+0000 7fcf5b40e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:53:55.418 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.416+0000 7fcf5b40e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:53:55.422 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.421+0000 7f8aa062b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:53:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.450+0000 7f8b7eaf8640 1 Processor -- start 2023-10-16T15:53:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.450+0000 7f8b7eaf8640 1 -- start start 2023-10-16T15:53:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7eaf8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b78159950 0x7f8b7815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7eaf8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8b78070450 0x7f8b780708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7eaf8640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b7815aef0 con 0x7f8b7806c100 2023-10-16T15:53:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7eaf8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b78070e20 con 0x7f8b78159950 2023-10-16T15:53:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7eaf8640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7815e490 con 0x7f8b78070450 2023-10-16T15:53:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7daf6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b78159950 0x7f8b7815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7daf6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b78159950 0x7f8b7815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48640/0 (socket says 172.21.15.29:48640) 2023-10-16T15:53:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.452+0000 7f8b7daf6640 1 -- 172.21.15.29:0/3939035436 learned_addr learned my addr 172.21.15.29:0/3939035436 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:55.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.453+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/3939035436 >> v2:172.21.15.170:3300/0 conn(0x7f8b78070450 0x7f8b780708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:55.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.453+0000 7f8b7daf6640 1 -- 172.21.15.29:0/3939035436 >> v2:172.21.15.170:3300/0 conn(0x7f8b78070450 msgr2=0x7f8b780708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.453+0000 7f8b7daf6640 1 --2- 172.21.15.29:0/3939035436 >> v2:172.21.15.170:3300/0 conn(0x7f8b78070450 0x7f8b780708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.453+0000 7f8b7daf6640 1 -- 172.21.15.29:0/3939035436 >> v1:172.21.15.170:6789/0 conn(0x7f8b7806c100 legacy=0x7f8b7806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.453+0000 7f8b7daf6640 1 -- 172.21.15.29:0/3939035436 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7815d910 con 0x7f8b78159950 2023-10-16T15:53:55.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.454+0000 7f8b7daf6640 1 --2- 172.21.15.29:0/3939035436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b78159950 0x7f8b7815a9b0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f8b780cab70 tx=0x7f8b7402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d28cd72af0c0f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.454+0000 7f8b7caf4640 1 -- 172.21.15.29:0/3939035436 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7402ee10 con 0x7f8b78159950 2023-10-16T15:53:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.454+0000 7f8b7caf4640 1 -- 172.21.15.29:0/3939035436 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74036440 con 0x7f8b78159950 2023-10-16T15:53:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.454+0000 7f8b7caf4640 1 -- 172.21.15.29:0/3939035436 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b740367d0 con 0x7f8b78159950 2023-10-16T15:53:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/3939035436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b78159950 msgr2=0x7f8b7815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 --2- 172.21.15.29:0/3939035436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b78159950 0x7f8b7815a9b0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f8b780cab70 tx=0x7f8b7402e060 comp rx=0 tx=0).stop 2023-10-16T15:53:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/3939035436 shutdown_connections 2023-10-16T15:53:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 --2- 172.21.15.29:0/3939035436 >> v2:172.21.15.170:3300/0 conn(0x7f8b78070450 0x7f8b780708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 --2- 172.21.15.29:0/3939035436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b78159950 0x7f8b7815a9b0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/3939035436 >> 172.21.15.29:0/3939035436 conn(0x7f8b780ce6a0 msgr2=0x7f8b780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/3939035436 shutdown_connections 2023-10-16T15:53:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.455+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/3939035436 wait complete. 2023-10-16T15:53:55.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.457+0000 7f8b7eaf8640 1 Processor -- start 2023-10-16T15:53:55.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.457+0000 7f8b7eaf8640 1 -- start start 2023-10-16T15:53:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7eaf8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7806c100 0x7f8b780c3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7eaf8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b78070450 0x7f8b78151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7eaf8640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b78159950 0x7f8b780c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7eaf8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b78157880 con 0x7f8b7806c100 2023-10-16T15:53:55.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7eaf8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c29d0 con 0x7f8b78070450 2023-10-16T15:53:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7eaf8640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c2cb0 con 0x7f8b78159950 2023-10-16T15:53:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7806c100 0x7f8b780c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7806c100 0x7f8b780c3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48654/0 (socket says 172.21.15.29:48654) 2023-10-16T15:53:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 -- 172.21.15.29:0/2577796344 learned_addr learned my addr 172.21.15.29:0/2577796344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:55.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b78070450 0x7f8b78151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b78159950 msgr2=0x7f8b780c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b78159950 0x7f8b780c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b78070450 msgr2=0x7f8b78151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b78070450 0x7f8b78151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:55.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.458+0000 7f8b7daf6640 1 -- 172.21.15.29:0/2577796344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7400b770 con 0x7f8b7806c100 2023-10-16T15:53:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.459+0000 7f8b7daf6640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7806c100 0x7f8b780c3790 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f8b7402e0a0 tx=0x7f8b740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.459+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74034b80 con 0x7f8b7806c100 2023-10-16T15:53:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.459+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74034d00 con 0x7f8b7806c100 2023-10-16T15:53:55.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.459+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b780c2f90 con 0x7f8b7806c100 2023-10-16T15:53:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.459+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74041680 con 0x7f8b7806c100 2023-10-16T15:53:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.459+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b781520b0 con 0x7f8b7806c100 2023-10-16T15:53:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.460+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b78152240 con 0x7f8b7806c100 2023-10-16T15:53:55.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.461+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 129) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f8b740403e0 con 0x7f8b7806c100 2023-10-16T15:53:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.462+0000 7f8b6e7fc640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 0x7f8b5c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:55.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.462+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f8b740ac130 con 0x7f8b7806c100 2023-10-16T15:53:55.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.462+0000 7f8b7d2f5640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 msgr2=0x7f8b5c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:55.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.463+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 0x7f8b5c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:53:55.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.467+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b74076d50 con 0x7f8b7806c100 2023-10-16T15:53:55.478 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.477+0000 7f8aa062b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:53:55.479 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.478+0000 7fcf5b40e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:53:55.528 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.527+0000 7fcf5b40e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:53:55.541 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.541+0000 7f8aa062b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:53:55.593 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.592+0000 7f8aa062b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:53:55.640 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.638+0000 7fcf5b40e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:53:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.664+0000 7f8b7d2f5640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 msgr2=0x7f8b5c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.664+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 0x7f8b5c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:53:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:55.682+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 --> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b7815ded0 con 0x7f8b5c06afe0 2023-10-16T15:53:55.702 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.700+0000 7fcf5b40e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:55.710 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.709+0000 7f8aa062b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:53:55.751 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.749+0000 7fcf5b40e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:53:55.776 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.775+0000 7f8aa062b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:55.809 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.806+0000 7fcf5b40e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:53:55.828 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.827+0000 7f8aa062b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:53:55.859 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:55.857+0000 7fcf5b40e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:53:55.887 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.886+0000 7f8aa062b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:53:55.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:55.938+0000 7f8aa062b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:53:56.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:56.065+0000 7f8b7d2f5640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 msgr2=0x7f8b5c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:56.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:56.065+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 0x7f8b5c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T15:53:56.240 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.238+0000 7fcf5b40e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:53:56.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:56.335+0000 7f8aa062b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:53:56.402 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.400+0000 7fcf5b40e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:53:56.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:56.506+0000 7f8aa062b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:53:56.510 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.508+0000 7fcf5b40e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:53:56.560 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.558+0000 7fcf5b40e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:53:56.619 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:56.618+0000 7f8aa062b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:53:56.667 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.666+0000 7fcf5b40e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:56.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:56.670+0000 7f8aa062b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:53:56.773 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.772+0000 7fcf5b40e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:53:56.785 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:56.784+0000 7f8aa062b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:56.866+0000 7f8b7d2f5640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 msgr2=0x7f8b5c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3980026359 2023-10-16T15:53:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:56.866+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 0x7f8b5c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T15:53:56.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:56.895+0000 7f8aa062b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:53:56.930 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.928+0000 7fcf5b40e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:56.979 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:56.977+0000 7fcf5b40e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:57.060 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:57.058+0000 7f8aa062b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:53:57.098 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:57.096+0000 7fcf5b40e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:53:57.111 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:57.110+0000 7f8aa062b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:53:57.235 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:57.234+0000 7f8aa062b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:53:57.916 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:57.913+0000 7fcf5b40e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:53:58.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:58.094+0000 7f8aa062b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:53:58.213 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.211+0000 7fcf5b40e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:53:58.271 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.269+0000 7fcf5b40e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:53:58.382 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.381+0000 7fcf5b40e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:53:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:58.403+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 130) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f8b740aa2d0 con 0x7f8b7806c100 2023-10-16T15:53:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:58.404+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 msgr2=0x7f8b5c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:53:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:58.404+0000 7f8b6e7fc640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/3980026359,v1:172.21.15.170:6833/3980026359] conn(0x7f8b5c06afe0 0x7f8b5c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:58.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:58.413+0000 7f8aa062b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:53:58.480 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:58.478+0000 7f8aa062b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:53:58.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:53:58.604+0000 7f8aa062b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:53:58.653 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.652+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.652+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.652+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.652+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.652+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.655 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.653+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.655 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.653+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.655 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.653+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.655 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.653+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.656 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.653+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.670 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.670 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.670 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.670 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fceda346640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.671 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:58.672 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:53:58.668+0000 7fced7340640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:53:59.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.412+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f8b74077120 con 0x7f8b7806c100 2023-10-16T15:53:59.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.413+0000 7f8b6e7fc640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8b5c035880 0x7f8b5c077530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.413+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b5c035c60 con 0x7f8b5c035880 2023-10-16T15:53:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.424+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8b5c035880 0x7f8b5c077530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.433+0000 7f8b7d2f5640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8b5c035880 0x7f8b5c077530 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8b780af380 tx=0x7f8b68008040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:59.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.590+0000 7f8b6e7fc640 1 -- 172.21.15.29:0/2577796344 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7f8b5c035c60 con 0x7f8b5c035880 2023-10-16T15:53:59.591 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T15:53:59.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.593+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8b5c035880 msgr2=0x7f8b5c077530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.593+0000 7f8b7eaf8640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8b5c035880 0x7f8b5c077530 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8b780af380 tx=0x7f8b68008040 comp rx=0 tx=0).stop 2023-10-16T15:53:59.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.593+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7806c100 msgr2=0x7f8b780c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.593+0000 7f8b7eaf8640 1 --2- 172.21.15.29:0/2577796344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b7806c100 0x7f8b780c3790 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f8b7402e0a0 tx=0x7f8b740046c0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.593+0000 7f8b7e2f7640 1 -- 172.21.15.29:0/2577796344 reap_dead start 2023-10-16T15:53:59.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.594+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 shutdown_connections 2023-10-16T15:53:59.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.594+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 >> 172.21.15.29:0/2577796344 conn(0x7f8b780ce6a0 msgr2=0x7f8b780cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:59.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.594+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 shutdown_connections 2023-10-16T15:53:59.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.594+0000 7f8b7eaf8640 1 -- 172.21.15.29:0/2577796344 wait complete. 2023-10-16T15:53:59.616 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T15:53:59.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T15:53:59.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.842+0000 7f9c5991a640 1 Processor -- start 2023-10-16T15:53:59.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.842+0000 7f9c5991a640 1 -- start start 2023-10-16T15:53:59.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.844+0000 7f9c5991a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9c54159950 0x7f9c5415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.844+0000 7f9c5991a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5415aef0 con 0x7f9c54070450 2023-10-16T15:53:59.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.844+0000 7f9c5991a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5415d910 con 0x7f9c5406c810 2023-10-16T15:53:59.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.844+0000 7f9c5991a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5415e490 con 0x7f9c54159950 2023-10-16T15:53:59.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.845+0000 7f9c53fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9c54070450 0x7f9c5406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36814/0 (socket says 172.21.15.29:36814) 2023-10-16T15:53:59.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.845+0000 7f9c53fff640 1 -- 172.21.15.29:0/1384423300 learned_addr learned my addr 172.21.15.29:0/1384423300 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:53:59.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.845+0000 7f9c58918640 1 --2- 172.21.15.29:0/1384423300 >> v2:172.21.15.170:3300/0 conn(0x7f9c54159950 0x7f9c5415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.845+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276727974 0 0) 0x7f9c40001110 con 0x7f9c54070450 2023-10-16T15:53:59.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.845+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c44003520 con 0x7f9c54070450 2023-10-16T15:53:59.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210428087 0 0) 0x7f9c44003520 con 0x7f9c54070450 2023-10-16T15:53:59.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 >> v2:172.21.15.170:3300/0 conn(0x7f9c54159950 msgr2=0x7f9c5415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:59.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 --2- 172.21.15.29:0/1384423300 >> v2:172.21.15.170:3300/0 conn(0x7f9c54159950 0x7f9c5415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 >> v1:172.21.15.170:6789/0 conn(0x7f9c5406c810 legacy=0x7f9c540c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5415dbf0 con 0x7f9c54070450 2023-10-16T15:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9c40001800 con 0x7f9c54070450 2023-10-16T15:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 >> v1:172.21.15.29:6789/0 conn(0x7f9c54070450 legacy=0x7f9c5406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440039c0 0x7f9c44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c440047c0 0x7f9c44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c440051d0 0x7f9c4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5415e490 con 0x7f9c440047c0 2023-10-16T15:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4400c120 con 0x7f9c440039c0 2023-10-16T15:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.846+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4400c2e0 con 0x7f9c440051d0 2023-10-16T15:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c440047c0 0x7f9c44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c58918640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440039c0 0x7f9c44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c59119640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c440051d0 0x7f9c4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c53fff640 1 -- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c440051d0 msgr2=0x7f9c4400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c440051d0 0x7f9c4400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c53fff640 1 -- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440039c0 msgr2=0x7f9c44003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440039c0 0x7f9c44003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c53fff640 1 -- 172.21.15.29:0/1384423300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44003520 con 0x7f9c440047c0 2023-10-16T15:53:59.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.847+0000 7f9c59119640 1 -- 172.21.15.29:0/1384423300 reap_dead start 2023-10-16T15:53:59.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.848+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c440047c0 0x7f9c44004c40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f9c40001eb0 tx=0x7f9c440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe112508372c85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40005ad0 con 0x7f9c440047c0 2023-10-16T15:53:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40005c50 con 0x7f9c440047c0 2023-10-16T15:53:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c537fe640 1 -- 172.21.15.29:0/1384423300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4002c7c0 con 0x7f9c440047c0 2023-10-16T15:53:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c5991a640 1 -- 172.21.15.29:0/1384423300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c440047c0 msgr2=0x7f9c44004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c440047c0 0x7f9c44004c40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f9c40001eb0 tx=0x7f9c440057f0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c5991a640 1 -- 172.21.15.29:0/1384423300 shutdown_connections 2023-10-16T15:53:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1384423300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c440047c0 0x7f9c44004c40 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.849+0000 7f9c5991a640 1 -- 172.21.15.29:0/1384423300 >> 172.21.15.29:0/1384423300 conn(0x7f9c540ce6a0 msgr2=0x7f9c540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:53:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.850+0000 7f9c5991a640 1 -- 172.21.15.29:0/1384423300 shutdown_connections 2023-10-16T15:53:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.850+0000 7f9c5991a640 1 -- 172.21.15.29:0/1384423300 wait complete. 2023-10-16T15:53:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.851+0000 7f9c5991a640 1 Processor -- start 2023-10-16T15:53:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.852+0000 7f9c5991a640 1 -- start start 2023-10-16T15:53:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c5991a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440047c0 0x7f9c5415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c5991a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 0x7f9c541552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c5991a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c54155ba0 0x7f9c540ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c5991a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541f3420 con 0x7f9c541f2eb0 2023-10-16T15:53:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c5991a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5415baf0 con 0x7f9c440047c0 2023-10-16T15:53:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c5991a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5415b1c0 con 0x7f9c54155ba0 2023-10-16T15:53:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 0x7f9c541552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 0x7f9c541552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48684/0 (socket says 172.21.15.29:48684) 2023-10-16T15:53:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 -- 172.21.15.29:0/1777991825 learned_addr learned my addr 172.21.15.29:0/1777991825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:53:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c58918640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440047c0 0x7f9c5415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c59119640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c54155ba0 0x7f9c540ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 -- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c54155ba0 msgr2=0x7f9c540ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c54155ba0 0x7f9c540ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 -- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440047c0 msgr2=0x7f9c5415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:53:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440047c0 0x7f9c5415bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:53:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.853+0000 7f9c53fff640 1 -- 172.21.15.29:0/1777991825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5415b4a0 con 0x7f9c541f2eb0 2023-10-16T15:53:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.854+0000 7f9c53fff640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 0x7f9c541552c0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f9c4002c360 tx=0x7f9c4002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.854+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4003c040 con 0x7f9c541f2eb0 2023-10-16T15:53:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.854+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4002cbd0 con 0x7f9c541f2eb0 2023-10-16T15:53:59.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.854+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40007070 con 0x7f9c541f2eb0 2023-10-16T15:53:59.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.854+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5415b780 con 0x7f9c541f2eb0 2023-10-16T15:53:59.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.854+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c54155800 con 0x7f9c541f2eb0 2023-10-16T15:53:59.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.856+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5404e9b0 con 0x7f9c541f2eb0 2023-10-16T15:53:59.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.856+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9c40032070 con 0x7f9c541f2eb0 2023-10-16T15:53:59.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.857+0000 7f9c517fa640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9c3806ad10 0x7f9c3806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:53:59.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.857+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f9c400aa2c0 con 0x7f9c541f2eb0 2023-10-16T15:53:59.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.858+0000 7f9c58918640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9c3806ad10 0x7f9c3806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:53:59.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.859+0000 7f9c58918640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9c3806ad10 0x7f9c3806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c54063ab0 tx=0x7f9c48008040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:53:59.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:53:59.862+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c400751d0 con 0x7f9c541f2eb0 2023-10-16T15:54:00.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.078+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c5415e1b0 con 0x7f9c3806ad10 2023-10-16T15:54:00.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.189+0000 7f9c517fa640 1 -- 172.21.15.29:0/1777991825 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9c480138b0 con 0x7f9c3806ad10 2023-10-16T15:54:00.190 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T15:54:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.192+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9c3806ad10 msgr2=0x7f9c3806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.192+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9c3806ad10 0x7f9c3806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c54063ab0 tx=0x7f9c48008040 comp rx=0 tx=0).stop 2023-10-16T15:54:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 msgr2=0x7f9c541552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 0x7f9c541552c0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f9c4002c360 tx=0x7f9c4002e070 comp rx=0 tx=0).stop 2023-10-16T15:54:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 shutdown_connections 2023-10-16T15:54:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c541f2eb0 0x7f9c541552c0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9c3806ad10 0x7f9c3806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c54155ba0 0x7f9c540ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 --2- 172.21.15.29:0/1777991825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c440047c0 0x7f9c5415bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 >> 172.21.15.29:0/1777991825 conn(0x7f9c540ce6a0 msgr2=0x7f9c540ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 shutdown_connections 2023-10-16T15:54:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.193+0000 7f9c5991a640 1 -- 172.21.15.29:0/1777991825 wait complete. 2023-10-16T15:54:00.215 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T15:54:00.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T15:54:00.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.449+0000 7f8e4c155640 1 Processor -- start 2023-10-16T15:54:00.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.449+0000 7f8e4c155640 1 -- start start 2023-10-16T15:54:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e4c155640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 0x7f8e440caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e4c155640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8e4406a4b0 0x7f8e441650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e4c155640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440cb140 con 0x7f8e440c9000 2023-10-16T15:54:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e4c155640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e44161a50 con 0x7f8e4406a4b0 2023-10-16T15:54:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e4c155640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4406b030 con 0x7f8e440addb0 2023-10-16T15:54:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e496f0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8e440c9000 0x7f8e44069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36828/0 (socket says 172.21.15.29:36828) 2023-10-16T15:54:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.451+0000 7f8e496f0640 1 -- 172.21.15.29:0/1029909451 learned_addr learned my addr 172.21.15.29:0/1029909451 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:00.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e4a6f2640 1 --2- 172.21.15.29:0/1029909451 >> v2:172.21.15.170:3300/0 conn(0x7f8e4406a4b0 0x7f8e441650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:00.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402282736 0 0) 0x7f8e440cb140 con 0x7f8e440c9000 2023-10-16T15:54:00.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e30003520 con 0x7f8e440c9000 2023-10-16T15:54:00.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e49ef1640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 0x7f8e440caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:00.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 719396281 0 0) 0x7f8e30003520 con 0x7f8e440c9000 2023-10-16T15:54:00.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 msgr2=0x7f8e440caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:00.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 0x7f8e440caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 >> v2:172.21.15.170:3300/0 conn(0x7f8e4406a4b0 msgr2=0x7f8e441650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 --2- 172.21.15.29:0/1029909451 >> v2:172.21.15.170:3300/0 conn(0x7f8e4406a4b0 0x7f8e441650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4406b310 con 0x7f8e440c9000 2023-10-16T15:54:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8e34001670 con 0x7f8e440c9000 2023-10-16T15:54:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 >> v1:172.21.15.29:6789/0 conn(0x7f8e440c9000 legacy=0x7f8e44069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 0x7f8e30003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e30003ff0 0x7f8e300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.452+0000 7f8e48eef640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44161a50 con 0x7f8e30004d00 2023-10-16T15:54:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4406b030 con 0x7f8e30003ff0 2023-10-16T15:54:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3000bae0 con 0x7f8e440addb0 2023-10-16T15:54:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e4a6f2640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e4a6f2640 1 -- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 msgr2=0x7f8e30003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e4a6f2640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e440addb0 0x7f8e30003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e4a6f2640 1 -- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e30003ff0 msgr2=0x7f8e300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e4a6f2640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e30003ff0 0x7f8e300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.453+0000 7f8e4a6f2640 1 -- 172.21.15.29:0/1029909451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30003520 con 0x7f8e30004d00 2023-10-16T15:54:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.454+0000 7f8e4a6f2640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f8e4406d830 tx=0x7f8e300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804b130d007577e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.455+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c02cca0 con 0x7f8e30004d00 2023-10-16T15:54:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.455+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c02ce00 con 0x7f8e30004d00 2023-10-16T15:54:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.455+0000 7f8e48eef640 1 -- 172.21.15.29:0/1029909451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c034680 con 0x7f8e30004d00 2023-10-16T15:54:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4c155640 1 -- 172.21.15.29:0/1029909451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e30004d00 msgr2=0x7f8e30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4c155640 1 --2- 172.21.15.29:0/1029909451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f8e4406d830 tx=0x7f8e300057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4a6f2640 1 -- 172.21.15.29:0/1029909451 reap_dead start 2023-10-16T15:54:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4c155640 1 -- 172.21.15.29:0/1029909451 shutdown_connections 2023-10-16T15:54:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4c155640 1 -- 172.21.15.29:0/1029909451 >> 172.21.15.29:0/1029909451 conn(0x7f8e440ce9b0 msgr2=0x7f8e440cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4c155640 1 -- 172.21.15.29:0/1029909451 shutdown_connections 2023-10-16T15:54:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.456+0000 7f8e4c155640 1 -- 172.21.15.29:0/1029909451 wait complete. 2023-10-16T15:54:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.458+0000 7f8e4c155640 1 Processor -- start 2023-10-16T15:54:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.458+0000 7f8e4c155640 1 -- start start 2023-10-16T15:54:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.459+0000 7f8e4c155640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e30004d00 0x7f8e440b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.459+0000 7f8e4c155640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e440b1e30 0x7f8e4415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.459+0000 7f8e4c155640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e440c7c50 0x7f8e440c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.459+0000 7f8e4c155640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e440c8610 con 0x7f8e440c7c50 2023-10-16T15:54:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.459+0000 7f8e4c155640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4415b130 con 0x7f8e440b1e30 2023-10-16T15:54:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.459+0000 7f8e4c155640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4415d040 con 0x7f8e30004d00 2023-10-16T15:54:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.460+0000 7f8e49ef1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e30004d00 0x7f8e440b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.460+0000 7f8e49ef1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e30004d00 0x7f8e440b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:44250/0 (socket says 172.21.15.29:44250) 2023-10-16T15:54:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.460+0000 7f8e49ef1640 1 -- 172.21.15.29:0/4291770905 learned_addr learned my addr 172.21.15.29:0/4291770905 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.460+0000 7f8e496f0640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e440b1e30 0x7f8e4415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.461+0000 7f8e496f0640 1 -- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e30004d00 msgr2=0x7f8e440b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.461+0000 7f8e496f0640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e30004d00 0x7f8e440b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.461+0000 7f8e496f0640 1 -- 172.21.15.29:0/4291770905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e440c7c50 msgr2=0x7f8e440c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.461+0000 7f8e496f0640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e440c7c50 0x7f8e440c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.461+0000 7f8e496f0640 1 -- 172.21.15.29:0/4291770905 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440b1b10 con 0x7f8e440b1e30 2023-10-16T15:54:00.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.462+0000 7f8e496f0640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e440b1e30 0x7f8e4415a530 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c690 tx=0x7f8e3400cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.473+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e340072d0 con 0x7f8e440b1e30 2023-10-16T15:54:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.473+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e440b23a0 con 0x7f8e440b1e30 2023-10-16T15:54:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.473+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e440b2960 con 0x7f8e440b1e30 2023-10-16T15:54:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.473+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e34007430 con 0x7f8e440b1e30 2023-10-16T15:54:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.473+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400a5e0 con 0x7f8e440b1e30 2023-10-16T15:54:00.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.474+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c005250 con 0x7f8e440b1e30 2023-10-16T15:54:00.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.478+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f8e3400a740 con 0x7f8e440b1e30 2023-10-16T15:54:00.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.479+0000 7f8e3a7fc640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8e2806b130 0x7f8e2806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:00.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.479+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f8e34089880 con 0x7f8e440b1e30 2023-10-16T15:54:00.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.480+0000 7f8e49ef1640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8e2806b130 0x7f8e2806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:00.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.480+0000 7f8e49ef1640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8e2806b130 0x7f8e2806d5f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e40006a90 tx=0x7f8e40008040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:00.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.492+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e34054100 con 0x7f8e440b1e30 2023-10-16T15:54:00.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.714+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e2c000c70 con 0x7f8e2806b130 2023-10-16T15:54:00.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.866+0000 7f8e3a7fc640 1 -- 172.21.15.29:0/4291770905 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f8e2c000c70 con 0x7f8e2806b130 2023-10-16T15:54:00.867 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path mandatory_fs_argument_test_dir 2023-10-16T15:54:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.869+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8e2806b130 msgr2=0x7f8e2806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.869+0000 7f8e4c155640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8e2806b130 0x7f8e2806d5f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e40006a90 tx=0x7f8e40008040 comp rx=0 tx=0).stop 2023-10-16T15:54:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.869+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e440b1e30 msgr2=0x7f8e4415a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.869+0000 7f8e4c155640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e440b1e30 0x7f8e4415a530 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c690 tx=0x7f8e3400cb50 comp rx=0 tx=0).stop 2023-10-16T15:54:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 shutdown_connections 2023-10-16T15:54:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e440c7c50 0x7f8e440c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f8e2806b130 0x7f8e2806d5f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8e440b1e30 0x7f8e4415a530 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 --2- 172.21.15.29:0/4291770905 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8e30004d00 0x7f8e440b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 >> 172.21.15.29:0/4291770905 conn(0x7f8e440ce9b0 msgr2=0x7f8e44069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 shutdown_connections 2023-10-16T15:54:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:00.870+0000 7f8e4c155640 1 -- 172.21.15.29:0/4291770905 wait complete. 2023-10-16T15:54:00.891 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2023-10-16T15:54:00.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T15:54:01.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.117+0000 7f3ad1dbf640 1 Processor -- start 2023-10-16T15:54:01.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.117+0000 7f3ad1dbf640 1 -- start start 2023-10-16T15:54:01.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.119+0000 7f3ad1dbf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3acc0ad970 0x7f3acc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.119+0000 7f3ad1dbf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3acc15b9d0 con 0x7f3acc06c040 2023-10-16T15:54:01.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.119+0000 7f3ad1dbf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3acc15bcb0 con 0x7f3acc0ccfb0 2023-10-16T15:54:01.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.119+0000 7f3ad1dbf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3acc15bf90 con 0x7f3acc0ad970 2023-10-16T15:54:01.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.119+0000 7f3ad0dbd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3acc06c040 0x7f3acc06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36840/0 (socket says 172.21.15.29:36840) 2023-10-16T15:54:01.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.119+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/2588943634 learned_addr learned my addr 172.21.15.29:0/2588943634 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:01.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acbfff640 1 --2- 172.21.15.29:0/2588943634 >> v2:172.21.15.170:3300/0 conn(0x7f3acc0ad970 0x7f3acc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258565353 0 0) 0x7f3acc15b9d0 con 0x7f3acc06c040 2023-10-16T15:54:01.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc003520 con 0x7f3acc06c040 2023-10-16T15:54:01.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2008179058 0 0) 0x7f3abc003520 con 0x7f3acc06c040 2023-10-16T15:54:01.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 >> v2:172.21.15.170:3300/0 conn(0x7f3acc0ad970 msgr2=0x7f3acc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:01.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/2588943634 >> v2:172.21.15.170:3300/0 conn(0x7f3acc0ad970 0x7f3acc0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 >> v1:172.21.15.170:6789/0 conn(0x7f3acc0ccfb0 legacy=0x7f3acc0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0cc270 con 0x7f3acc06c040 2023-10-16T15:54:01.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f3ac0001670 con 0x7f3acc06c040 2023-10-16T15:54:01.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 >> v1:172.21.15.29:6789/0 conn(0x7f3acc06c040 legacy=0x7f3acc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3abc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.120+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3abc0047c0 0x7f3abc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc005180 0x7f3abc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc15bf90 con 0x7f3abc0039c0 2023-10-16T15:54:01.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc00c080 con 0x7f3abc005180 2023-10-16T15:54:01.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc00c240 con 0x7f3abc0047c0 2023-10-16T15:54:01.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3abc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/2588943634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3abc0047c0 msgr2=0x7f3abc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:01.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3abc0047c0 0x7f3abc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/2588943634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc005180 msgr2=0x7f3abc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:01.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc005180 0x7f3abc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/2588943634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc003520 con 0x7f3abc0039c0 2023-10-16T15:54:01.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.121+0000 7f3ad15be640 1 -- 172.21.15.29:0/2588943634 reap_dead start 2023-10-16T15:54:01.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.122+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3abc003de0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c0e0 tx=0x7f3abc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8271ad07e09dffa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:01.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.123+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0005ad0 con 0x7f3abc0039c0 2023-10-16T15:54:01.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.123+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac0005c30 con 0x7f3abc0039c0 2023-10-16T15:54:01.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.123+0000 7f3acb7fe640 1 -- 172.21.15.29:0/2588943634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac002ca50 con 0x7f3abc0039c0 2023-10-16T15:54:01.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.123+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/2588943634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 msgr2=0x7f3abc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.123+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3abc003de0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c0e0 tx=0x7f3abc0057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.124+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/2588943634 shutdown_connections 2023-10-16T15:54:01.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.124+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/2588943634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3abc003de0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.124+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/2588943634 >> 172.21.15.29:0/2588943634 conn(0x7f3acc0ce6a0 msgr2=0x7f3acc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:01.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.124+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/2588943634 shutdown_connections 2023-10-16T15:54:01.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.124+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/2588943634 wait complete. 2023-10-16T15:54:01.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.125+0000 7f3ad1dbf640 1 Processor -- start 2023-10-16T15:54:01.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.126+0000 7f3ad1dbf640 1 -- start start 2023-10-16T15:54:01.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad1dbf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3acc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad1dbf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3acc15f330 0x7f3acc15d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad1dbf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc15dcf0 0x7f3acc0b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad1dbf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc15e290 con 0x7f3abc0039c0 2023-10-16T15:54:01.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad1dbf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc15e570 con 0x7f3acc15dcf0 2023-10-16T15:54:01.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad1dbf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc15e850 con 0x7f3acc15f330 2023-10-16T15:54:01.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad0dbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3acc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad0dbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3acc157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48722/0 (socket says 172.21.15.29:48722) 2023-10-16T15:54:01.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.127+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/385942005 learned_addr learned my addr 172.21.15.29:0/385942005 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:01.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3acbfff640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3acc15f330 0x7f3acc15d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad15be640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc15dcf0 0x7f3acc0b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3acc15f330 msgr2=0x7f3acc15d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3acc15f330 0x7f3acc15d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc15dcf0 msgr2=0x7f3acc0b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc15dcf0 0x7f3acc0b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad0dbd640 1 -- 172.21.15.29:0/385942005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc15eb30 con 0x7f3abc0039c0 2023-10-16T15:54:01.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad0dbd640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3acc157c40 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f3ac002c3b0 tx=0x7f3ac0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:01.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0039070 con 0x7f3abc0039c0 2023-10-16T15:54:01.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac002dcd0 con 0x7f3abc0039c0 2023-10-16T15:54:01.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac003c040 con 0x7f3abc0039c0 2023-10-16T15:54:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3acc15ee10 con 0x7f3abc0039c0 2023-10-16T15:54:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.128+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3acc15aba0 con 0x7f3abc0039c0 2023-10-16T15:54:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.130+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3acc06b330 con 0x7f3abc0039c0 2023-10-16T15:54:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.131+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f3ac0032070 con 0x7f3abc0039c0 2023-10-16T15:54:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.131+0000 7f3ac97fa640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f3ab006b130 0x7f3ab006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.132+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f3ac00ab1e0 con 0x7f3abc0039c0 2023-10-16T15:54:01.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.132+0000 7f3acbfff640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f3ab006b130 0x7f3ab006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.133+0000 7f3acbfff640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f3ab006b130 0x7f3ab006d5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3acc1f5f70 tx=0x7f3ab8008040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:01.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.136+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ac0075a40 con 0x7f3abc0039c0 2023-10-16T15:54:01.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.350+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3acc15e6e0 con 0x7f3ab006b130 2023-10-16T15:54:01.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.366+0000 7f3ac97fa640 1 -- 172.21.15.29:0/385942005 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f3acc15e6e0 con 0x7f3ab006b130 2023-10-16T15:54:01.367 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:01.367 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T15:54:01.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.368+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f3ab006b130 msgr2=0x7f3ab006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.368+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f3ab006b130 0x7f3ab006d5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3acc1f5f70 tx=0x7f3ab8008040 comp rx=0 tx=0).stop 2023-10-16T15:54:01.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.368+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 msgr2=0x7f3acc157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.368+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3acc157c40 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f3ac002c3b0 tx=0x7f3ac0001930 comp rx=0 tx=0).stop 2023-10-16T15:54:01.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 shutdown_connections 2023-10-16T15:54:01.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f3ab006b130 0x7f3ab006d5f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3acc15dcf0 0x7f3acc0b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3acc15f330 0x7f3acc15d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 --2- 172.21.15.29:0/385942005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3abc0039c0 0x7f3acc157c40 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 >> 172.21.15.29:0/385942005 conn(0x7f3acc0ce6a0 msgr2=0x7f3acc0c78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:01.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 shutdown_connections 2023-10-16T15:54:01.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.369+0000 7f3ad1dbf640 1 -- 172.21.15.29:0/385942005 wait complete. 2023-10-16T15:54:01.386 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T15:54:01.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T15:54:01.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.605+0000 7fc1b9202640 1 Processor -- start 2023-10-16T15:54:01.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.605+0000 7fc1b9202640 1 -- start start 2023-10-16T15:54:01.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b9202640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b40b8480 0x7fc1b40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b9202640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc1b40d0be0 0x7fc1b40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b9202640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b415d150 con 0x7fc1b415cd70 2023-10-16T15:54:01.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b9202640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40d1180 con 0x7fc1b40d0be0 2023-10-16T15:54:01.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b9202640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40d1460 con 0x7fc1b40b8480 2023-10-16T15:54:01.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc1b415cd70 0x7fc1b40d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36852/0 (socket says 172.21.15.29:36852) 2023-10-16T15:54:01.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b3fff640 1 -- 172.21.15.29:0/3735767532 learned_addr learned my addr 172.21.15.29:0/3735767532 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:01.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.607+0000 7fc1b8a01640 1 --2- 172.21.15.29:0/3735767532 >> v2:172.21.15.29:3300/0 conn(0x7fc1b40d0be0 0x7fc1b40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b2ffd640 1 -- 172.21.15.29:0/3735767532 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536963268 0 0) 0x7fc1b415d150 con 0x7fc1b415cd70 2023-10-16T15:54:01.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b37fe640 1 --2- 172.21.15.29:0/3735767532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b40b8480 0x7fc1b40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b2ffd640 1 -- 172.21.15.29:0/3735767532 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c003520 con 0x7fc1b415cd70 2023-10-16T15:54:01.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b8a01640 1 -- 172.21.15.29:0/3735767532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b40b8480 msgr2=0x7fc1b40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:01.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b8a01640 1 --2- 172.21.15.29:0/3735767532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b40b8480 0x7fc1b40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b8a01640 1 -- 172.21.15.29:0/3735767532 >> v1:172.21.15.29:6789/0 conn(0x7fc1b415cd70 legacy=0x7fc1b40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.608+0000 7fc1b8a01640 1 -- 172.21.15.29:0/3735767532 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40d1740 con 0x7fc1b40d0be0 2023-10-16T15:54:01.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.609+0000 7fc1b8a01640 1 --2- 172.21.15.29:0/3735767532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b40cdd00 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009d90 tx=0x7fc1ac02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b321f22026dd090 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b2ffd640 1 -- 172.21.15.29:0/3735767532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac02cc50 con 0x7fc1b40d0be0 2023-10-16T15:54:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b2ffd640 1 -- 172.21.15.29:0/3735767532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac02cdb0 con 0x7fc1b40d0be0 2023-10-16T15:54:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b2ffd640 1 -- 172.21.15.29:0/3735767532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac034640 con 0x7fc1b40d0be0 2023-10-16T15:54:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 -- 172.21.15.29:0/3735767532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 msgr2=0x7fc1b40cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3735767532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b40cdd00 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009d90 tx=0x7fc1ac02bea0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 -- 172.21.15.29:0/3735767532 shutdown_connections 2023-10-16T15:54:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3735767532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b40cdd00 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3735767532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b40b8480 0x7fc1b40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 -- 172.21.15.29:0/3735767532 >> 172.21.15.29:0/3735767532 conn(0x7fc1b40cf570 msgr2=0x7fc1b40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.610+0000 7fc1b9202640 1 -- 172.21.15.29:0/3735767532 shutdown_connections 2023-10-16T15:54:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.611+0000 7fc1b9202640 1 -- 172.21.15.29:0/3735767532 wait complete. 2023-10-16T15:54:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.613+0000 7fc1b9202640 1 Processor -- start 2023-10-16T15:54:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.613+0000 7fc1b9202640 1 -- start start 2023-10-16T15:54:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.614+0000 7fc1b9202640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b40b8480 0x7fc1b415fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.614+0000 7fc1b9202640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b4157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.614+0000 7fc1b9202640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b415cd70 0x7fc1b40cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.614+0000 7fc1b9202640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4158a20 con 0x7fc1b40d0be0 2023-10-16T15:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.614+0000 7fc1b9202640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40cacf0 con 0x7fc1b40b8480 2023-10-16T15:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.614+0000 7fc1b9202640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4158460 con 0x7fc1b415cd70 2023-10-16T15:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b4157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b4157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48736/0 (socket says 172.21.15.29:48736) 2023-10-16T15:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 -- 172.21.15.29:0/3474116428 learned_addr learned my addr 172.21.15.29:0/3474116428 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b3fff640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b40b8480 0x7fc1b415fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b8a01640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b415cd70 0x7fc1b40cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 -- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b415cd70 msgr2=0x7fc1b40cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b415cd70 0x7fc1b40cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 -- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b40b8480 msgr2=0x7fc1b415fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b40b8480 0x7fc1b415fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.615+0000 7fc1b37fe640 1 -- 172.21.15.29:0/3474116428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac009a70 con 0x7fc1b40d0be0 2023-10-16T15:54:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.616+0000 7fc1b37fe640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b4157b30 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800d5f0 tx=0x7fc1a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:01.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.616+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a80125f0 con 0x7fc1b40d0be0 2023-10-16T15:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.616+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8012750 con 0x7fc1b40d0be0 2023-10-16T15:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.616+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b4158740 con 0x7fc1b40d0be0 2023-10-16T15:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.616+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8012b00 con 0x7fc1b40d0be0 2023-10-16T15:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.616+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b415ff60 con 0x7fc1b40d0be0 2023-10-16T15:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.617+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b40cae60 con 0x7fc1b40d0be0 2023-10-16T15:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.618+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7fc1a8012c60 con 0x7fc1b40d0be0 2023-10-16T15:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.620+0000 7fc1b0ff9640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fc1a006b100 0x7fc1a006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.620+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fc1a808a4c0 con 0x7fc1b40d0be0 2023-10-16T15:54:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.621+0000 7fc1b3fff640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fc1a006b100 0x7fc1a006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.622+0000 7fc1b3fff640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fc1a006b100 0x7fc1a006d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4005880 tx=0x7fc1a4009040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.626+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc1a8054d20 con 0x7fc1b40d0be0 2023-10-16T15:54:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.838+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1b40d1d00 con 0x7fc1a006b100 2023-10-16T15:54:01.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.853+0000 7fc1b0ff9640 1 -- 172.21.15.29:0/3474116428 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+338 (secure 0 0 0) 0x7fc1a40013f0 con 0x7fc1a006b100 2023-10-16T15:54:01.855 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:01.855 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T15:53:59", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T15:54:01.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fc1a006b100 msgr2=0x7fc1a006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fc1a006b100 0x7fc1a006d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4005880 tx=0x7fc1a4009040 comp rx=0 tx=0).stop 2023-10-16T15:54:01.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 msgr2=0x7fc1b4157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:01.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b4157b30 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800d5f0 tx=0x7fc1a800dab0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 shutdown_connections 2023-10-16T15:54:01.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc1b40d0be0 0x7fc1b4157b30 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fc1a006b100 0x7fc1a006d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc1b415cd70 0x7fc1b40cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 --2- 172.21.15.29:0/3474116428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc1b40b8480 0x7fc1b415fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:01.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 >> 172.21.15.29:0/3474116428 conn(0x7fc1b40cf570 msgr2=0x7fc1b40b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:01.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 shutdown_connections 2023-10-16T15:54:01.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:01.856+0000 7fc1b9202640 1 -- 172.21.15.29:0/3474116428 wait complete. 2023-10-16T15:54:01.875 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T15:53:59", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T15:54:01.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T15:54:02.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.076+0000 7f19a759e640 1 Processor -- start 2023-10-16T15:54:02.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.077+0000 7f19a759e640 1 -- start start 2023-10-16T15:54:02.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.078+0000 7f19a759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f19a80b3ec0 0x7f19a80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.078+0000 7f19a759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19a80b43c0 0x7f19a80cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.078+0000 7f19a759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a806be30 con 0x7f19a80b0ea0 2023-10-16T15:54:02.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.078+0000 7f19a759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a806ca00 con 0x7f19a80b43c0 2023-10-16T15:54:02.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.078+0000 7f19a759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a80ada10 con 0x7f19a80b3ec0 2023-10-16T15:54:02.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19a80b43c0 0x7f19a80cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19a80b43c0 0x7f19a80cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48746/0 (socket says 172.21.15.29:48746) 2023-10-16T15:54:02.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 -- 172.21.15.29:0/1921231413 learned_addr learned my addr 172.21.15.29:0/1921231413 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:02.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a5d9b640 1 --1- 172.21.15.29:0/1921231413 >> v1:172.21.15.29:6789/0 conn(0x7f19a80b0ea0 0x7f19a815a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36854/0 (socket says 172.21.15.29:36854) 2023-10-16T15:54:02.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a659c640 1 --2- 172.21.15.29:0/1921231413 >> v2:172.21.15.170:3300/0 conn(0x7f19a80b3ec0 0x7f19a80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 -- 172.21.15.29:0/1921231413 >> v2:172.21.15.170:3300/0 conn(0x7f19a80b3ec0 msgr2=0x7f19a80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:02.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 --2- 172.21.15.29:0/1921231413 >> v2:172.21.15.170:3300/0 conn(0x7f19a80b3ec0 0x7f19a80cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 -- 172.21.15.29:0/1921231413 >> v1:172.21.15.29:6789/0 conn(0x7f19a80b0ea0 legacy=0x7f19a815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.079+0000 7f19a6d9d640 1 -- 172.21.15.29:0/1921231413 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a806c060 con 0x7f19a80b43c0 2023-10-16T15:54:02.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.080+0000 7f19a6d9d640 1 --2- 172.21.15.29:0/1921231413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80cd7a0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f19a80742a0 tx=0x7f19a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05b174db744bab3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:02.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a559a640 1 -- 172.21.15.29:0/1921231413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0031870 con 0x7f19a80b43c0 2023-10-16T15:54:02.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a559a640 1 -- 172.21.15.29:0/1921231413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a00319d0 con 0x7f19a80b43c0 2023-10-16T15:54:02.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a559a640 1 -- 172.21.15.29:0/1921231413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0031d60 con 0x7f19a80b43c0 2023-10-16T15:54:02.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a759e640 1 -- 172.21.15.29:0/1921231413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 msgr2=0x7f19a80cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a759e640 1 --2- 172.21.15.29:0/1921231413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80cd7a0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f19a80742a0 tx=0x7f19a002b910 comp rx=0 tx=0).stop 2023-10-16T15:54:02.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a759e640 1 -- 172.21.15.29:0/1921231413 shutdown_connections 2023-10-16T15:54:02.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a759e640 1 --2- 172.21.15.29:0/1921231413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80cd7a0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a759e640 1 --2- 172.21.15.29:0/1921231413 >> v2:172.21.15.170:3300/0 conn(0x7f19a80b3ec0 0x7f19a80cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.081+0000 7f19a759e640 1 -- 172.21.15.29:0/1921231413 >> 172.21.15.29:0/1921231413 conn(0x7f19a80b0170 msgr2=0x7f19a80b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:02.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.082+0000 7f19a759e640 1 -- 172.21.15.29:0/1921231413 shutdown_connections 2023-10-16T15:54:02.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.082+0000 7f19a759e640 1 -- 172.21.15.29:0/1921231413 wait complete. 2023-10-16T15:54:02.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.083+0000 7f19a759e640 1 Processor -- start 2023-10-16T15:54:02.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.083+0000 7f19a759e640 1 -- start start 2023-10-16T15:54:02.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.084+0000 7f19a759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19a80b0ea0 0x7f19a81f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.084+0000 7f19a759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19a80b3ec0 0x7f19a81f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.084+0000 7f19a759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.084+0000 7f19a759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a81e3fb0 con 0x7f19a80b43c0 2023-10-16T15:54:02.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.084+0000 7f19a759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80c6fd0 con 0x7f19a80b3ec0 2023-10-16T15:54:02.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.084+0000 7f19a759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80c72b0 con 0x7f19a80b0ea0 2023-10-16T15:54:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80c67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48760/0 (socket says 172.21.15.29:48760) 2023-10-16T15:54:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 -- 172.21.15.29:0/2213274064 learned_addr learned my addr 172.21.15.29:0/2213274064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a659c640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19a80b0ea0 0x7f19a81f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a5d9b640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19a80b3ec0 0x7f19a81f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 -- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19a80b0ea0 msgr2=0x7f19a81f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19a80b0ea0 0x7f19a81f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 -- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19a80b3ec0 msgr2=0x7f19a81f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19a80b3ec0 0x7f19a81f1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 -- 172.21.15.29:0/2213274064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00094d0 con 0x7f19a80b43c0 2023-10-16T15:54:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.085+0000 7f19a6d9d640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80c67a0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f19a00090b0 tx=0x7f19a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.086+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a003ac60 con 0x7f19a80b43c0 2023-10-16T15:54:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.086+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a003adc0 con 0x7f19a80b43c0 2023-10-16T15:54:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.086+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0044630 con 0x7f19a80b43c0 2023-10-16T15:54:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.086+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a80c7590 con 0x7f19a80b43c0 2023-10-16T15:54:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.086+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a80c7b50 con 0x7f19a80b43c0 2023-10-16T15:54:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.087+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a80674f0 con 0x7f19a80b43c0 2023-10-16T15:54:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.088+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f19a004d450 con 0x7f19a80b43c0 2023-10-16T15:54:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.089+0000 7f199effd640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f198806b100 0x7f198806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.089+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f19a002d040 con 0x7f19a80b43c0 2023-10-16T15:54:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.090+0000 7f19a659c640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f198806b100 0x7f198806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.091+0000 7f19a659c640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f198806b100 0x7f198806d5c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19a80c1af0 tx=0x7f19980039b0 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.093+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f19a0076de0 con 0x7f19a80b43c0 2023-10-16T15:54:02.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.312+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f19a806c620 con 0x7f198806b100 2023-10-16T15:54:02.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.445+0000 7f199effd640 1 -- 172.21.15.29:0/2213274064 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1998009650 con 0x7f198806b100 2023-10-16T15:54:02.446 INFO:teuthology.orchestra.run.smithi029.stdout:Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T15:54:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f198806b100 msgr2=0x7f198806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f198806b100 0x7f198806d5c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19a80c1af0 tx=0x7f19980039b0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 msgr2=0x7f19a80c67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80c67a0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f19a00090b0 tx=0x7f19a00046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 shutdown_connections 2023-10-16T15:54:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19a80b43c0 0x7f19a80c67a0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f198806b100 0x7f198806d5c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19a80b3ec0 0x7f19a81f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 --2- 172.21.15.29:0/2213274064 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19a80b0ea0 0x7f19a81f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 >> 172.21.15.29:0/2213274064 conn(0x7f19a80b0170 msgr2=0x7f19a80ae0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:02.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 shutdown_connections 2023-10-16T15:54:02.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.449+0000 7f19a759e640 1 -- 172.21.15.29:0/2213274064 wait complete. 2023-10-16T15:54:02.471 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T15:54:02.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T15:54:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.704+0000 7f4f8759e640 1 Processor -- start 2023-10-16T15:54:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.704+0000 7f4f8759e640 1 -- start start 2023-10-16T15:54:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.706+0000 7f4f8759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 0x7f4f88071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.706+0000 7f4f8759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f4f88071fa0 0x7f4f8806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.706+0000 7f4f8759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f880d12f0 con 0x7f4f8815cc80 2023-10-16T15:54:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.706+0000 7f4f8759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f880d0500 con 0x7f4f88071fa0 2023-10-16T15:54:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.706+0000 7f4f8759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880723f0 con 0x7f4f880cdf30 2023-10-16T15:54:02.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.707+0000 7f4f8659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f8815cc80 0x7f4f880cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36870/0 (socket says 172.21.15.29:36870) 2023-10-16T15:54:02.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.707+0000 7f4f8659c640 1 -- 172.21.15.29:0/2787743073 learned_addr learned my addr 172.21.15.29:0/2787743073 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:02.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.707+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/2787743073 >> v2:172.21.15.170:3300/0 conn(0x7f4f88071fa0 0x7f4f8806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.707+0000 7f4f7ffff640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 0x7f4f88071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.707+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786644159 0 0) 0x7f4f880d12f0 con 0x7f4f8815cc80 2023-10-16T15:54:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.707+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f74003520 con 0x7f4f8815cc80 2023-10-16T15:54:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885980453 0 0) 0x7f4f74003520 con 0x7f4f8815cc80 2023-10-16T15:54:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 msgr2=0x7f4f88071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 0x7f4f88071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 >> v2:172.21.15.170:3300/0 conn(0x7f4f88071fa0 msgr2=0x7f4f8806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 --2- 172.21.15.29:0/2787743073 >> v2:172.21.15.170:3300/0 conn(0x7f4f88071fa0 0x7f4f8806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880d06e0 con 0x7f4f8815cc80 2023-10-16T15:54:02.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4f78001670 con 0x7f4f8815cc80 2023-10-16T15:54:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/2787743073 >> v2:172.21.15.170:3300/0 conn(0x7f4f88071fa0 0x7f4f8806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 >> v1:172.21.15.29:6789/0 conn(0x7f4f8815cc80 legacy=0x7f4f880cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74003970 0x7f4f74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 0x7f4f740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f74004d00 0x7f4f74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880d0500 con 0x7f4f74004d00 2023-10-16T15:54:02.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880723f0 con 0x7f4f74003970 2023-10-16T15:54:02.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7400b980 con 0x7f4f880cdf30 2023-10-16T15:54:02.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.708+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f74004d00 0x7f4f74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f8659c640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74003970 0x7f4f74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f7ffff640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 0x7f4f740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f86d9d640 1 -- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 msgr2=0x7f4f740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880cdf30 0x7f4f740047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f86d9d640 1 -- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74003970 msgr2=0x7f4f74003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:02.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74003970 0x7f4f74003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.709+0000 7f4f86d9d640 1 -- 172.21.15.29:0/2787743073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74003520 con 0x7f4f74004d00 2023-10-16T15:54:02.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.710+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f74004d00 0x7f4f74005180 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e350 tx=0x7f4f740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa5fd98b6c318f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.710+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8002ce00 con 0x7f4f74004d00 2023-10-16T15:54:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.710+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8000f040 con 0x7f4f74004d00 2023-10-16T15:54:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.710+0000 7f4f85d9b640 1 -- 172.21.15.29:0/2787743073 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80034640 con 0x7f4f74004d00 2023-10-16T15:54:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f8759e640 1 -- 172.21.15.29:0/2787743073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f74004d00 msgr2=0x7f4f74005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f8759e640 1 --2- 172.21.15.29:0/2787743073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f74004d00 0x7f4f74005180 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e350 tx=0x7f4f740057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f86d9d640 1 -- 172.21.15.29:0/2787743073 reap_dead start 2023-10-16T15:54:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f8759e640 1 -- 172.21.15.29:0/2787743073 shutdown_connections 2023-10-16T15:54:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f8759e640 1 -- 172.21.15.29:0/2787743073 >> 172.21.15.29:0/2787743073 conn(0x7f4f88065ed0 msgr2=0x7f4f880662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f8759e640 1 -- 172.21.15.29:0/2787743073 shutdown_connections 2023-10-16T15:54:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.711+0000 7f4f8759e640 1 -- 172.21.15.29:0/2787743073 wait complete. 2023-10-16T15:54:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.713+0000 7f4f8759e640 1 Processor -- start 2023-10-16T15:54:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.713+0000 7f4f8759e640 1 -- start start 2023-10-16T15:54:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.714+0000 7f4f8759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74004d00 0x7f4f880af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.714+0000 7f4f8759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880b5fa0 0x7f4f880ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.714+0000 7f4f8759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 0x7f4f8815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.714+0000 7f4f8759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88148820 con 0x7f4f880acdd0 2023-10-16T15:54:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.714+0000 7f4f8759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880ad250 con 0x7f4f74004d00 2023-10-16T15:54:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.714+0000 7f4f8759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880af7b0 con 0x7f4f880b5fa0 2023-10-16T15:54:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 0x7f4f8815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 0x7f4f8815ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48768/0 (socket says 172.21.15.29:48768) 2023-10-16T15:54:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 -- 172.21.15.29:0/3825638782 learned_addr learned my addr 172.21.15.29:0/3825638782 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f8659c640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74004d00 0x7f4f880af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f7ffff640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880b5fa0 0x7f4f880ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 -- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880b5fa0 msgr2=0x7f4f880ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880b5fa0 0x7f4f880ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 -- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74004d00 msgr2=0x7f4f880af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74004d00 0x7f4f880af270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 -- 172.21.15.29:0/3825638782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8814a580 con 0x7f4f880acdd0 2023-10-16T15:54:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f8659c640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74004d00 0x7f4f880af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.715+0000 7f4f86d9d640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 0x7f4f8815ee40 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e350 tx=0x7f4f80009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.716+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80044070 con 0x7f4f880acdd0 2023-10-16T15:54:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.716+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f80003990 con 0x7f4f880acdd0 2023-10-16T15:54:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.716+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80003d40 con 0x7f4f880acdd0 2023-10-16T15:54:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.716+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f88147ba0 con 0x7f4f880acdd0 2023-10-16T15:54:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.716+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f88148160 con 0x7f4f880acdd0 2023-10-16T15:54:02.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.717+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f88148960 con 0x7f4f880acdd0 2023-10-16T15:54:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.719+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f4f8003e610 con 0x7f4f880acdd0 2023-10-16T15:54:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.719+0000 7f4f7effd640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f4f6806b100 0x7f4f6806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.720+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f4f800abe30 con 0x7f4f880acdd0 2023-10-16T15:54:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.720+0000 7f4f8659c640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f4f6806b100 0x7f4f6806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.721+0000 7f4f8659c640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f4f6806b100 0x7f4f6806d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f780058b0 tx=0x7f4f78000fd0 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:02.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.724+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f88148960 con 0x7f4f880acdd0 2023-10-16T15:54:02.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:02.943+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f880d0ca0 con 0x7f4f6806b100 2023-10-16T15:54:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.076+0000 7f4f7effd640 1 -- 172.21.15.29:0/3825638782 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f4f78005e30 con 0x7f4f6806b100 2023-10-16T15:54:03.078 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T15:54:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.079+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f4f6806b100 msgr2=0x7f4f6806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.079+0000 7f4f8759e640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f4f6806b100 0x7f4f6806d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f780058b0 tx=0x7f4f78000fd0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.079+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 msgr2=0x7f4f8815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.079+0000 7f4f8759e640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 0x7f4f8815ee40 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e350 tx=0x7f4f80009750 comp rx=0 tx=0).stop 2023-10-16T15:54:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 shutdown_connections 2023-10-16T15:54:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f880acdd0 0x7f4f8815ee40 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f4f6806b100 0x7f4f6806d5c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4f880b5fa0 0x7f4f880ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 --2- 172.21.15.29:0/3825638782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f74004d00 0x7f4f880af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 >> 172.21.15.29:0/3825638782 conn(0x7f4f88065ed0 msgr2=0x7f4f8815d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 shutdown_connections 2023-10-16T15:54:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.080+0000 7f4f8759e640 1 -- 172.21.15.29:0/3825638782 wait complete. 2023-10-16T15:54:03.102 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T15:54:03.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T15:54:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.335+0000 7f759f475640 1 Processor -- start 2023-10-16T15:54:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.336+0000 7f759f475640 1 -- start start 2023-10-16T15:54:03.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.337+0000 7f759f475640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f759806f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:03.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.337+0000 7f759f475640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75980b0360 0x7f75980b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:03.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.337+0000 7f759f475640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75980c9620 con 0x7f759806f9d0 2023-10-16T15:54:03.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.337+0000 7f759f475640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759815d340 con 0x7f75980b0360 2023-10-16T15:54:03.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.337+0000 7f759f475640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980cc860 con 0x7f759815cf60 2023-10-16T15:54:03.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.338+0000 7f759da12640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75980b0360 0x7f75980b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:03.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.338+0000 7f759d211640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f759806f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:03.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.338+0000 7f759da12640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75980b0360 0x7f75980b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48778/0 (socket says 172.21.15.29:48778) 2023-10-16T15:54:03.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.338+0000 7f759da12640 1 -- 172.21.15.29:0/1328306290 learned_addr learned my addr 172.21.15.29:0/1328306290 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:03.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.339+0000 7f759da12640 1 -- 172.21.15.29:0/1328306290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 msgr2=0x7f759806f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.339+0000 7f759da12640 1 --2- 172.21.15.29:0/1328306290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f759806f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.339+0000 7f759da12640 1 -- 172.21.15.29:0/1328306290 >> v1:172.21.15.29:6789/0 conn(0x7f759806f9d0 legacy=0x7f75980c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.339+0000 7f759da12640 1 -- 172.21.15.29:0/1328306290 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980ad4a0 con 0x7f75980b0360 2023-10-16T15:54:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.339+0000 7f758ffff640 1 -- 172.21.15.29:0/1328306290 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000201313 0 0) 0x7f75980c9620 con 0x7f759806f9d0 2023-10-16T15:54:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.340+0000 7f759da12640 1 --2- 172.21.15.29:0/1328306290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75980b0360 0x7f75980b07f0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f7590009e50 tx=0x7f759002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f950338b9b935c57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.340+0000 7f758ffff640 1 -- 172.21.15.29:0/1328306290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759002cb00 con 0x7f75980b0360 2023-10-16T15:54:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.340+0000 7f758ffff640 1 -- 172.21.15.29:0/1328306290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759002cc60 con 0x7f75980b0360 2023-10-16T15:54:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.340+0000 7f758ffff640 1 -- 172.21.15.29:0/1328306290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900343f0 con 0x7f75980b0360 2023-10-16T15:54:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 -- 172.21.15.29:0/1328306290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75980b0360 msgr2=0x7f75980b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 --2- 172.21.15.29:0/1328306290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75980b0360 0x7f75980b07f0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f7590009e50 tx=0x7f759002bf60 comp rx=0 tx=0).stop 2023-10-16T15:54:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 -- 172.21.15.29:0/1328306290 shutdown_connections 2023-10-16T15:54:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 --2- 172.21.15.29:0/1328306290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75980b0360 0x7f75980b07f0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 --2- 172.21.15.29:0/1328306290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f759806f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 -- 172.21.15.29:0/1328306290 >> 172.21.15.29:0/1328306290 conn(0x7f75980cb360 msgr2=0x7f75980cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 -- 172.21.15.29:0/1328306290 shutdown_connections 2023-10-16T15:54:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.341+0000 7f759f475640 1 -- 172.21.15.29:0/1328306290 wait complete. 2023-10-16T15:54:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.343+0000 7f759f475640 1 Processor -- start 2023-10-16T15:54:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.343+0000 7f759f475640 1 -- start start 2023-10-16T15:54:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759f475640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 0x7f75980c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759f475640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75980b0360 0x7f7598157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759f475640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f75980c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759f475640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980c53d0 con 0x7f759806f9d0 2023-10-16T15:54:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759f475640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598158450 con 0x7f75980b0360 2023-10-16T15:54:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759f475640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980b15a0 con 0x7f759815cf60 2023-10-16T15:54:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759d211640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 0x7f75980c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759d211640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 0x7f75980c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48784/0 (socket says 172.21.15.29:48784) 2023-10-16T15:54:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.344+0000 7f759d211640 1 -- 172.21.15.29:0/1970274810 learned_addr learned my addr 172.21.15.29:0/1970274810 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759da12640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f75980c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759ca10640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75980b0360 0x7f7598157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759d211640 1 -- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 msgr2=0x7f75980c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759d211640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f75980c4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759d211640 1 -- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75980b0360 msgr2=0x7f7598157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759d211640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75980b0360 0x7f7598157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759d211640 1 -- 172.21.15.29:0/1970274810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7590009b30 con 0x7f759806f9d0 2023-10-16T15:54:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759ca10640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75980b0360 0x7f7598157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:03.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.345+0000 7f759d211640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 0x7f75980c6f70 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f759400e8a0 tx=0x7f759400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:03.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.346+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940094f0 con 0x7f759806f9d0 2023-10-16T15:54:03.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.346+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759400c940 con 0x7f759806f9d0 2023-10-16T15:54:03.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.346+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400ccf0 con 0x7f759806f9d0 2023-10-16T15:54:03.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.346+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75980c4890 con 0x7f759806f9d0 2023-10-16T15:54:03.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.346+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75980c4e10 con 0x7f759806f9d0 2023-10-16T15:54:03.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.347+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75980b1370 con 0x7f759806f9d0 2023-10-16T15:54:03.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.348+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 133) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f7594013470 con 0x7f759806f9d0 2023-10-16T15:54:03.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.349+0000 7f758dffb640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f757c06b1c0 0x7f757c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:03.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.349+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f759408a080 con 0x7f759806f9d0 2023-10-16T15:54:03.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.350+0000 7f759ca10640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f757c06b1c0 0x7f757c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:03.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.351+0000 7f759ca10640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f757c06b1c0 0x7f757c06d680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f75980c3ea0 tx=0x7f7588009040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:03.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.353+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75980b1370 con 0x7f759806f9d0 2023-10-16T15:54:03.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.573+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 --> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f75980c92b0 con 0x7f757c06b1c0 2023-10-16T15:54:03.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.727+0000 7f758dffb640 1 -- 172.21.15.29:0/1970274810 <== mgr.8917 v2:172.21.15.170:6832/1824188539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f7588001320 con 0x7f757c06b1c0 2023-10-16T15:54:03.728 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T15:54:03.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.730+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f757c06b1c0 msgr2=0x7f757c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.730+0000 7f759f475640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f757c06b1c0 0x7f757c06d680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f75980c3ea0 tx=0x7f7588009040 comp rx=0 tx=0).stop 2023-10-16T15:54:03.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.730+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 msgr2=0x7f75980c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:03.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.730+0000 7f759f475640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 0x7f75980c6f70 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f759400e8a0 tx=0x7f759400ed60 comp rx=0 tx=0).stop 2023-10-16T15:54:03.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 shutdown_connections 2023-10-16T15:54:03.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f757c06b1c0 0x7f757c06d680 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f759815cf60 0x7f75980c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75980b0360 0x7f7598157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 --2- 172.21.15.29:0/1970274810 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f759806f9d0 0x7f75980c6f70 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:03.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 >> 172.21.15.29:0/1970274810 conn(0x7f75980cb360 msgr2=0x7f75980ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:03.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 shutdown_connections 2023-10-16T15:54:03.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:03.731+0000 7f759f475640 1 -- 172.21.15.29:0/1970274810 wait complete. 2023-10-16T15:54:03.753 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T15:54:03.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:03.754 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T15:54:03.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:54:04.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.047+0000 7f9acc0da640 1 Processor -- start 2023-10-16T15:54:04.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.047+0000 7f9acc0da640 1 -- start start 2023-10-16T15:54:04.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9acc0da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac40cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:04.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9acc0da640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ac40ca780 0x7f9ac406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:04.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9acc0da640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac40c3610 con 0x7f9ac406b3f0 2023-10-16T15:54:04.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9acc0da640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac406b880 con 0x7f9ac40ca780 2023-10-16T15:54:04.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9acc0da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac415a220 con 0x7f9ac4159e40 2023-10-16T15:54:04.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9ac9675640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ac40ca780 0x7f9ac406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:04.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9ac9675640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ac40ca780 0x7f9ac406aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44264/0 (socket says 172.21.15.29:44264) 2023-10-16T15:54:04.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9ac9675640 1 -- 172.21.15.29:0/2897671043 learned_addr learned my addr 172.21.15.29:0/2897671043 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9ac9675640 1 -- 172.21.15.29:0/2897671043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 msgr2=0x7f9ac40cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.049+0000 7f9ac9e76640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac40cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:04.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.050+0000 7f9ac9675640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac40cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:04.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.050+0000 7f9ac9675640 1 -- 172.21.15.29:0/2897671043 >> v1:172.21.15.170:6789/0 conn(0x7f9ac406b3f0 legacy=0x7f9ac40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:04.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.050+0000 7f9ac9675640 1 -- 172.21.15.29:0/2897671043 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac40cafd0 con 0x7f9ac40ca780 2023-10-16T15:54:04.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.050+0000 7f9ac9e76640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac40cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:04.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.051+0000 7f9ac9675640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac406aeb0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009870 tx=0x7f9ab402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f0825b123a9b90 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:04.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.051+0000 7f9ac8e74640 1 -- 172.21.15.29:0/2897671043 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4031640 con 0x7f9ac40ca780 2023-10-16T15:54:04.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.051+0000 7f9ac8e74640 1 -- 172.21.15.29:0/2897671043 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab40317a0 con 0x7f9ac40ca780 2023-10-16T15:54:04.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.051+0000 7f9ac8e74640 1 -- 172.21.15.29:0/2897671043 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4031b30 con 0x7f9ac40ca780 2023-10-16T15:54:04.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 -- 172.21.15.29:0/2897671043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 msgr2=0x7f9ac406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:04.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac406aeb0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009870 tx=0x7f9ab402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:54:04.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 -- 172.21.15.29:0/2897671043 shutdown_connections 2023-10-16T15:54:04.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac406aeb0 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:04.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 --2- 172.21.15.29:0/2897671043 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac40cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:04.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 -- 172.21.15.29:0/2897671043 >> 172.21.15.29:0/2897671043 conn(0x7f9ac40cfac0 msgr2=0x7f9ac40cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:04.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 -- 172.21.15.29:0/2897671043 shutdown_connections 2023-10-16T15:54:04.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.052+0000 7f9acc0da640 1 -- 172.21.15.29:0/2897671043 wait complete. 2023-10-16T15:54:04.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.054+0000 7f9acc0da640 1 Processor -- start 2023-10-16T15:54:04.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.054+0000 7f9acc0da640 1 -- start start 2023-10-16T15:54:04.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.055+0000 7f9acc0da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ac406b3f0 0x7f9ac4146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:04.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.055+0000 7f9acc0da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac4156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:04.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.055+0000 7f9acc0da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac4162680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:04.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.055+0000 7f9acc0da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41657f0 con 0x7f9ac40ca780 2023-10-16T15:54:04.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.055+0000 7f9acc0da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41473a0 con 0x7f9ac406b3f0 2023-10-16T15:54:04.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.055+0000 7f9acc0da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4155a80 con 0x7f9ac4159e40 2023-10-16T15:54:04.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac4156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:04.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac4156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44278/0 (socket says 172.21.15.29:44278) 2023-10-16T15:54:04.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 -- 172.21.15.29:0/3757182500 learned_addr learned my addr 172.21.15.29:0/3757182500 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:04.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9aca677640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac4162680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:04.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9e76640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ac406b3f0 0x7f9ac4146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:04.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 -- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 msgr2=0x7f9ac4162680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:04.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac4162680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:04.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 -- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ac406b3f0 msgr2=0x7f9ac4146e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:04.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ac406b3f0 0x7f9ac4146e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:04.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 -- 172.21.15.29:0/3757182500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab40094d0 con 0x7f9ac40ca780 2023-10-16T15:54:04.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9e76640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ac406b3f0 0x7f9ac4146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:04.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.056+0000 7f9ac9675640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac4156b20 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4000c00 tx=0x7f9ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:04.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.057+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab403ac60 con 0x7f9ac40ca780 2023-10-16T15:54:04.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.057+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab403ade0 con 0x7f9ac40ca780 2023-10-16T15:54:04.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.057+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4044630 con 0x7f9ac40ca780 2023-10-16T15:54:04.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.057+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac4155d60 con 0x7f9ac40ca780 2023-10-16T15:54:04.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.057+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac4156320 con 0x7f9ac40ca780 2023-10-16T15:54:04.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.058+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac404e160 con 0x7f9ac40ca780 2023-10-16T15:54:04.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.059+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 133) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f9ab402f050 con 0x7f9ac40ca780 2023-10-16T15:54:04.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.060+0000 7f9aba7fc640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b190 0x7f9aa806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:04.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.060+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f9ab40ac1d0 con 0x7f9ac40ca780 2023-10-16T15:54:04.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.061+0000 7f9ac9e76640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b190 0x7f9aa806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:04.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.062+0000 7f9ac9e76640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b190 0x7f9aa806d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006fd0 tx=0x7f9ac0008040 comp rx=0 tx=0).ready entity=mgr.8917 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:04.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.064+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ab4076900 con 0x7f9ac40ca780 2023-10-16T15:54:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:04.307+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f9ac4155770 con 0x7f9ac40ca780 2023-10-16T15:54:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.267+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v134) v1 ==== 93+0+0 (secure 0 0 0) 0x7f9ab4079f20 con 0x7f9ac40ca780 2023-10-16T15:54:05.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.274+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b190 msgr2=0x7f9aa806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:05.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.274+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b190 0x7f9aa806d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006fd0 tx=0x7f9ac0008040 comp rx=0 tx=0).stop 2023-10-16T15:54:05.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.274+0000 7f9aba7fc640 1 -- 172.21.15.29:0/3757182500 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f9ab4076cd0 con 0x7f9ac40ca780 2023-10-16T15:54:05.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.274+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 msgr2=0x7f9ac4156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:05.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.274+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac4156b20 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4000c00 tx=0x7f9ab40046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.275+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 shutdown_connections 2023-10-16T15:54:05.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.275+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac40ca780 0x7f9ac4156b20 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.275+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b190 0x7f9aa806d650 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.275+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ac4159e40 0x7f9ac4162680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.275+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ac406b3f0 0x7f9ac4146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.275+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 >> 172.21.15.29:0/3757182500 conn(0x7f9ac40cfac0 msgr2=0x7f9ac40c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:05.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.276+0000 7f9aba7fc640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b640 0x7f9aa8075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.276+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 shutdown_connections 2023-10-16T15:54:05.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.276+0000 7f9acc0da640 1 --2- 172.21.15.29:0/3757182500 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f9aa806b640 0x7f9aa8075c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.276+0000 7f9acc0da640 1 -- 172.21.15.29:0/3757182500 wait complete. 2023-10-16T15:54:05.301 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:54:05.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:54:05.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:54:05.504 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:54:05.505 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:54:05.505 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:05.558 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:54:05.559 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:05.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:05.818 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:05.817+0000 7f188fa38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:54:05.849 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:05.848+0000 7f2630c43040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:54:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.859+0000 7fb01359e640 1 Processor -- start 2023-10-16T15:54:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.859+0000 7fb01359e640 1 -- start start 2023-10-16T15:54:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.860+0000 7fb01359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0140cdf30 0x7fb014071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.860+0000 7fb01359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 0x7fb01406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.860+0000 7fb01359e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0140d12f0 con 0x7fb01415cc80 2023-10-16T15:54:05.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.860+0000 7fb01359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0140d0500 con 0x7fb0140cdf30 2023-10-16T15:54:05.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.860+0000 7fb01359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140723d0 con 0x7fb014071fa0 2023-10-16T15:54:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.861+0000 7fb011d9b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0140cdf30 0x7fb014071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.861+0000 7fb012d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 0x7fb01406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.861+0000 7fb012d9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 0x7fb01406aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:53768/0 (socket says 172.21.15.29:53768) 2023-10-16T15:54:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.861+0000 7fb012d9d640 1 -- 172.21.15.29:0/3923740975 learned_addr learned my addr 172.21.15.29:0/3923740975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.861+0000 7fb01259c640 1 --1- 172.21.15.29:0/3923740975 >> v1:172.21.15.170:6789/0 conn(0x7fb01415cc80 0x7fb0140cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:37906/0 (socket says 172.21.15.29:37906) 2023-10-16T15:54:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.862+0000 7fb012d9d640 1 -- 172.21.15.29:0/3923740975 >> v2:172.21.15.170:3300/0 conn(0x7fb0140cdf30 msgr2=0x7fb014071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.862+0000 7fb012d9d640 1 --2- 172.21.15.29:0/3923740975 >> v2:172.21.15.170:3300/0 conn(0x7fb0140cdf30 0x7fb014071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.862+0000 7fb012d9d640 1 -- 172.21.15.29:0/3923740975 >> v1:172.21.15.170:6789/0 conn(0x7fb01415cc80 legacy=0x7fb0140cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:05.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.862+0000 7fb01159a640 1 -- 172.21.15.29:0/3923740975 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375374653 0 0) 0x7fb0140d12f0 con 0x7fb01415cc80 2023-10-16T15:54:05.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.862+0000 7fb012d9d640 1 -- 172.21.15.29:0/3923740975 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0140d06e0 con 0x7fb014071fa0 2023-10-16T15:54:05.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.864+0000 7fb012d9d640 1 --2- 172.21.15.29:0/3923740975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 0x7fb01406aeb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009e50 tx=0x7fb00c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f429d8cdfe0de222 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:05.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.864+0000 7fb01159a640 1 -- 172.21.15.29:0/3923740975 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c02cb00 con 0x7fb014071fa0 2023-10-16T15:54:05.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.864+0000 7fb01159a640 1 -- 172.21.15.29:0/3923740975 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c02cc60 con 0x7fb014071fa0 2023-10-16T15:54:05.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.864+0000 7fb01159a640 1 -- 172.21.15.29:0/3923740975 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0343f0 con 0x7fb014071fa0 2023-10-16T15:54:05.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 -- 172.21.15.29:0/3923740975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 msgr2=0x7fb01406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:05.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 --2- 172.21.15.29:0/3923740975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 0x7fb01406aeb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009e50 tx=0x7fb00c02bf60 comp rx=0 tx=0).stop 2023-10-16T15:54:05.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 -- 172.21.15.29:0/3923740975 shutdown_connections 2023-10-16T15:54:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 --2- 172.21.15.29:0/3923740975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb014071fa0 0x7fb01406aeb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 --2- 172.21.15.29:0/3923740975 >> v2:172.21.15.170:3300/0 conn(0x7fb0140cdf30 0x7fb014071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 -- 172.21.15.29:0/3923740975 >> 172.21.15.29:0/3923740975 conn(0x7fb014065ed0 msgr2=0x7fb0140662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 -- 172.21.15.29:0/3923740975 shutdown_connections 2023-10-16T15:54:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.865+0000 7fb01359e640 1 -- 172.21.15.29:0/3923740975 wait complete. 2023-10-16T15:54:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.867+0000 7fb01359e640 1 Processor -- start 2023-10-16T15:54:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.867+0000 7fb01359e640 1 -- start start 2023-10-16T15:54:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb014071fa0 0x7fb0141f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0140cdf30 0x7fb0140b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 0x7fb0140b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140b3f10 con 0x7fb01415cc80 2023-10-16T15:54:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140b41b0 con 0x7fb014071fa0 2023-10-16T15:54:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140b4490 con 0x7fb0140cdf30 2023-10-16T15:54:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 0x7fb0140b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 0x7fb0140b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44302/0 (socket says 172.21.15.29:44302) 2023-10-16T15:54:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 -- 172.21.15.29:0/383271041 learned_addr learned my addr 172.21.15.29:0/383271041 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01259c640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb014071fa0 0x7fb0141f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb011d9b640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0140cdf30 0x7fb0140b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 -- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0140cdf30 msgr2=0x7fb0140b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0140cdf30 0x7fb0140b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 -- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb014071fa0 msgr2=0x7fb0141f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb014071fa0 0x7fb0141f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb012d9d640 1 -- 172.21.15.29:0/383271041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c009b30 con 0x7fb01415cc80 2023-10-16T15:54:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.868+0000 7fb01259c640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb014071fa0 0x7fb0141f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.869+0000 7fb012d9d640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 0x7fb0140b1200 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009e50 tx=0x7fb00c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.869+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c009510 con 0x7fb01415cc80 2023-10-16T15:54:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.869+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c03e450 con 0x7fb01415cc80 2023-10-16T15:54:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.869+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c03e800 con 0x7fb01415cc80 2023-10-16T15:54:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.869+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0140b4770 con 0x7fb01415cc80 2023-10-16T15:54:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.869+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0141f22b0 con 0x7fb01415cc80 2023-10-16T15:54:05.876 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:05.871+0000 7f188fa38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:54:05.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.870+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0140742a0 con 0x7fb01415cc80 2023-10-16T15:54:05.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.871+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fb00c03e960 con 0x7fb01415cc80 2023-10-16T15:54:05.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.872+0000 7fb002ffd640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 0x7faff406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:05.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.872+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fb00c0acc00 con 0x7fb01415cc80 2023-10-16T15:54:05.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.873+0000 7fb01259c640 1 -- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 msgr2=0x7faff406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:05.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.873+0000 7fb01259c640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 0x7faff406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:05.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:05.877+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb00c0b3080 con 0x7fb01415cc80 2023-10-16T15:54:05.908 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:05.907+0000 7f2630c43040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:54:06.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.070+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0140d0c70 con 0x7fb01415cc80 2023-10-16T15:54:06.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.073+0000 7fb002ffd640 1 -- 172.21.15.29:0/383271041 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+20859 (secure 0 0 0) 0x7fb00c0774d0 con 0x7fb01415cc80 2023-10-16T15:54:06.075 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:06.076 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":230,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:53:58.385975+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:06.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.074+0000 7fb01259c640 1 -- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 msgr2=0x7faff406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:06.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.074+0000 7fb01259c640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 0x7faff406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:54:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 msgr2=0x7faff406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 0x7faff406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 msgr2=0x7fb0140b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 0x7fb0140b1200 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009e50 tx=0x7fb00c004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 shutdown_connections 2023-10-16T15:54:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb01415cc80 0x7fb0140b1200 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7faff406b140 0x7faff406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb0140cdf30 0x7fb0140b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 --2- 172.21.15.29:0/383271041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb014071fa0 0x7fb0141f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 >> 172.21.15.29:0/383271041 conn(0x7fb014065ed0 msgr2=0x7fb0140afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 shutdown_connections 2023-10-16T15:54:06.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.076+0000 7fb01359e640 1 -- 172.21.15.29:0/383271041 wait complete. 2023-10-16T15:54:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:53:24.608039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:53:25.694889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T15:53:39.139862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T15:53:39.864914+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:54:06.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:54:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.311+0000 7fa2076e7640 1 Processor -- start 2023-10-16T15:54:06.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.311+0000 7fa2076e7640 1 -- start start 2023-10-16T15:54:06.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.312+0000 7fa2076e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.312+0000 7fa2076e7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa200070c10 0x7fa2000cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.312+0000 7fa2076e7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa200161740 con 0x7fa2000cf5f0 2023-10-16T15:54:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.312+0000 7fa2076e7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2000cdbd0 con 0x7fa200070c10 2023-10-16T15:54:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.312+0000 7fa2076e7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20006b0f0 con 0x7fa200070830 2023-10-16T15:54:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.313+0000 7fa205c84640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa200070c10 0x7fa2000cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.313+0000 7fa205c84640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa200070c10 0x7fa2000cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:46694/0 (socket says 172.21.15.29:46694) 2023-10-16T15:54:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.313+0000 7fa205c84640 1 -- 172.21.15.29:0/2811684127 learned_addr learned my addr 172.21.15.29:0/2811684127 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.313+0000 7fa205483640 1 --2- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.314+0000 7fa205c84640 1 -- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 msgr2=0x7fa20006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.314+0000 7fa205c84640 1 --2- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.314+0000 7fa205c84640 1 -- 172.21.15.29:0/2811684127 >> v1:172.21.15.170:6789/0 conn(0x7fa2000cf5f0 legacy=0x7fa20006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.314+0000 7fa205c84640 1 -- 172.21.15.29:0/2811684127 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2001619b0 con 0x7fa200070c10 2023-10-16T15:54:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.315+0000 7fa205c84640 1 --2- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa200070c10 0x7fa2000cd690 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009e50 tx=0x7fa1f802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8831ac48c9873db3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.316+0000 7fa1f7fff640 1 -- 172.21.15.29:0/2811684127 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f802c9e0 con 0x7fa200070c10 2023-10-16T15:54:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.316+0000 7fa1f7fff640 1 -- 172.21.15.29:0/2811684127 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f802cb40 con 0x7fa200070c10 2023-10-16T15:54:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.316+0000 7fa1f7fff640 1 -- 172.21.15.29:0/2811684127 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80343f0 con 0x7fa200070c10 2023-10-16T15:54:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 -- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa200070c10 msgr2=0x7fa2000cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 --2- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa200070c10 0x7fa2000cd690 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009e50 tx=0x7fa1f802be40 comp rx=0 tx=0).stop 2023-10-16T15:54:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 -- 172.21.15.29:0/2811684127 shutdown_connections 2023-10-16T15:54:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 --2- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa200070c10 0x7fa2000cd690 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 --2- 172.21.15.29:0/2811684127 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 -- 172.21.15.29:0/2811684127 >> 172.21.15.29:0/2811684127 conn(0x7fa2000cfac0 msgr2=0x7fa2000cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 -- 172.21.15.29:0/2811684127 shutdown_connections 2023-10-16T15:54:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.317+0000 7fa2076e7640 1 -- 172.21.15.29:0/2811684127 wait complete. 2023-10-16T15:54:06.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.319+0000 7fa2076e7640 1 Processor -- start 2023-10-16T15:54:06.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.319+0000 7fa2076e7640 1 -- start start 2023-10-16T15:54:06.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa2076e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20015bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa2076e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 0x7fa20014aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa2076e7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2000cf5f0 0x7fa2001e8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa2076e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa200158a20 con 0x7fa200070c10 2023-10-16T15:54:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa2076e7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa200159560 con 0x7fa2000cf5f0 2023-10-16T15:54:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa2076e7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2000c81f0 con 0x7fa200070830 2023-10-16T15:54:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 0x7fa20014aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 0x7fa20014aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44306/0 (socket says 172.21.15.29:44306) 2023-10-16T15:54:06.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 -- 172.21.15.29:0/389306752 learned_addr learned my addr 172.21.15.29:0/389306752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa205c84640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2000cf5f0 0x7fa2001e8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa205483640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20015bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 -- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 msgr2=0x7fa20015bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20015bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 -- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2000cf5f0 msgr2=0x7fa2001e8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2000cf5f0 0x7fa2001e8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.320+0000 7fa204c82640 1 -- 172.21.15.29:0/389306752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f8009b30 con 0x7fa200070c10 2023-10-16T15:54:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa205c84640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2000cf5f0 0x7fa2001e8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa204c82640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 0x7fa20014aea0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000c3f0 tx=0x7fa1f000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:06.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00052d0 con 0x7fa200070c10 2023-10-16T15:54:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f0005430 con 0x7fa200070c10 2023-10-16T15:54:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2001597b0 con 0x7fa200070c10 2023-10-16T15:54:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0003830 con 0x7fa200070c10 2023-10-16T15:54:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.321+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2000c7e80 con 0x7fa200070c10 2023-10-16T15:54:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.322+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2000cd670 con 0x7fa200070c10 2023-10-16T15:54:06.327 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.322+0000 7f188fa38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:54:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.323+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fa1f0003990 con 0x7fa200070c10 2023-10-16T15:54:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.324+0000 7fa1f5ffb640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fa1e406b140 0x7fa1e406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.324+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fa1f0089630 con 0x7fa200070c10 2023-10-16T15:54:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.325+0000 7fa205483640 1 -- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fa1e406b140 msgr2=0x7fa1e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:06.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.325+0000 7fa205483640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fa1e406b140 0x7fa1e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:06.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.328+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa1f0053c80 con 0x7fa200070c10 2023-10-16T15:54:06.376 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.375+0000 7f188fa38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:54:06.395 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.394+0000 7f2630c43040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:54:06.427 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.426+0000 7f188fa38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:54:06.450 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.449+0000 7f2630c43040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:54:06.488 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.486+0000 7f188fa38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:54:06.503 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.502+0000 7f2630c43040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:54:06.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.521+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa20004e160 con 0x7fa200070c10 2023-10-16T15:54:06.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.522+0000 7fa1f5ffb640 1 -- 172.21.15.29:0/389306752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v230) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1f00572a0 con 0x7fa200070c10 2023-10-16T15:54:06.523 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:54:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.524+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fa1e406b140 msgr2=0x7fa1e406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.524+0000 7fa2076e7640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fa1e406b140 0x7fa1e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 msgr2=0x7fa20014aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 0x7fa20014aea0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000c3f0 tx=0x7fa1f000c8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 shutdown_connections 2023-10-16T15:54:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa200070c10 0x7fa20014aea0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fa1e406b140 0x7fa1e406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa2000cf5f0 0x7fa2001e8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 --2- 172.21.15.29:0/389306752 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa200070830 0x7fa20015bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 >> 172.21.15.29:0/389306752 conn(0x7fa2000cfac0 msgr2=0x7fa2000cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 shutdown_connections 2023-10-16T15:54:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.525+0000 7fa2076e7640 1 -- 172.21.15.29:0/389306752 wait complete. 2023-10-16T15:54:06.536 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.535+0000 7f188fa38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:54:06.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:54:06.569 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.568+0000 7f2630c43040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:54:06.622 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.621+0000 7f2630c43040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:54:06.646 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.644+0000 7f188fa38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:54:06.708 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.706+0000 7f188fa38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:54:06.739 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.739+0000 7f2630c43040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:54:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.753+0000 7fefc3a87640 1 Processor -- start 2023-10-16T15:54:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.754+0000 7fefc3a87640 1 -- start start 2023-10-16T15:54:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.755+0000 7fefc3a87640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fefbc15e2f0 0x7fefbc0b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.755+0000 7fefc3a87640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc09cc50 0x7fefbc0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.755+0000 7fefc3a87640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefbc06ae20 0x7fefbc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.755+0000 7fefc3a87640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefbc065c50 con 0x7fefbc06ae20 2023-10-16T15:54:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.755+0000 7fefc3a87640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefbc065f30 con 0x7fefbc15e2f0 2023-10-16T15:54:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.755+0000 7fefc3a87640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc06bf50 con 0x7fefbc09cc50 2023-10-16T15:54:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefbc06ae20 0x7fefbc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefbc06ae20 0x7fefbc0b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44314/0 (socket says 172.21.15.29:44314) 2023-10-16T15:54:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 -- 172.21.15.29:0/1769912447 learned_addr learned my addr 172.21.15.29:0/1769912447 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:06.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc1823640 1 --2- 172.21.15.29:0/1769912447 >> v2:172.21.15.170:3300/0 conn(0x7fefbc15e2f0 0x7fefbc0b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc1022640 1 --2- 172.21.15.29:0/1769912447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc09cc50 0x7fefbc0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.759 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.756+0000 7f188fa38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:54:06.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 -- 172.21.15.29:0/1769912447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc09cc50 msgr2=0x7fefbc0b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:06.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 --2- 172.21.15.29:0/1769912447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc09cc50 0x7fefbc0b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 -- 172.21.15.29:0/1769912447 >> v2:172.21.15.170:3300/0 conn(0x7fefbc15e2f0 msgr2=0x7fefbc0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:06.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 --2- 172.21.15.29:0/1769912447 >> v2:172.21.15.170:3300/0 conn(0x7fefbc15e2f0 0x7fefbc0b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.756+0000 7fefc2024640 1 -- 172.21.15.29:0/1769912447 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc15e6d0 con 0x7fefbc06ae20 2023-10-16T15:54:06.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.758+0000 7fefc2024640 1 --2- 172.21.15.29:0/1769912447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc0b2a40 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0b1960 tx=0x7fefb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add156c45dec61c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:06.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.758+0000 7fefc0821640 1 -- 172.21.15.29:0/1769912447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb402ca70 con 0x7fefbc06ae20 2023-10-16T15:54:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.758+0000 7fefc0821640 1 -- 172.21.15.29:0/1769912447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb402cbd0 con 0x7fefbc06ae20 2023-10-16T15:54:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.758+0000 7fefc0821640 1 -- 172.21.15.29:0/1769912447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40343f0 con 0x7fefbc06ae20 2023-10-16T15:54:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 -- 172.21.15.29:0/1769912447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 msgr2=0x7fefbc0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1769912447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc0b2a40 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0b1960 tx=0x7fefb402bed0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 -- 172.21.15.29:0/1769912447 shutdown_connections 2023-10-16T15:54:06.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1769912447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc0b2a40 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1769912447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc09cc50 0x7fefbc0b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1769912447 >> v2:172.21.15.170:3300/0 conn(0x7fefbc15e2f0 0x7fefbc0b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 -- 172.21.15.29:0/1769912447 >> 172.21.15.29:0/1769912447 conn(0x7fefbc0cbc60 msgr2=0x7fefbc0cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:06.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 -- 172.21.15.29:0/1769912447 shutdown_connections 2023-10-16T15:54:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.759+0000 7fefc3a87640 1 -- 172.21.15.29:0/1769912447 wait complete. 2023-10-16T15:54:06.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.761+0000 7fefc3a87640 1 Processor -- start 2023-10-16T15:54:06.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.761+0000 7fefc3a87640 1 -- start start 2023-10-16T15:54:06.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc3a87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc3a87640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc15e2f0 0x7fefbc0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc3a87640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0cea90 0x7fefbc15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc3a87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0b0090 con 0x7fefbc06ae20 2023-10-16T15:54:06.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc3a87640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0aeb40 con 0x7fefbc15e2f0 2023-10-16T15:54:06.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc3a87640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0aed90 con 0x7fefbc0cea90 2023-10-16T15:54:06.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc1823640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc1823640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44324/0 (socket says 172.21.15.29:44324) 2023-10-16T15:54:06.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.762+0000 7fefc1823640 1 -- 172.21.15.29:0/1037658550 learned_addr learned my addr 172.21.15.29:0/1037658550 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:06.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1022640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc15e2f0 0x7fefbc0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:06.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1823640 1 -- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0cea90 msgr2=0x7fefbc15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1823640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0cea90 0x7fefbc15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1823640 1 -- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc15e2f0 msgr2=0x7fefbc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1823640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc15e2f0 0x7fefbc0afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1823640 1 -- 172.21.15.29:0/1037658550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4009b30 con 0x7fefbc06ae20 2023-10-16T15:54:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc1823640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc150820 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fefb800e870 tx=0x7fefb800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80094f0 con 0x7fefbc06ae20 2023-10-16T15:54:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb800c940 con 0x7fefbc06ae20 2023-10-16T15:54:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb800ccf0 con 0x7fefbc06ae20 2023-10-16T15:54:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.763+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc0af070 con 0x7fefbc06ae20 2023-10-16T15:54:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.764+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc0af630 con 0x7fefbc06ae20 2023-10-16T15:54:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.765+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc04e160 con 0x7fefbc06ae20 2023-10-16T15:54:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.766+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fefb8013470 con 0x7fefbc06ae20 2023-10-16T15:54:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.767+0000 7fefb27fc640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 0x7fefa006d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.767+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fefb808a110 con 0x7fefbc06ae20 2023-10-16T15:54:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.767+0000 7fefc1022640 1 -- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 msgr2=0x7fefa006d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:06.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.767+0000 7fefc1022640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 0x7fefa006d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.771+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefb80548b0 con 0x7fefbc06ae20 2023-10-16T15:54:06.809 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.808+0000 7f2630c43040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:54:06.815 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.813+0000 7f188fa38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:54:06.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.861+0000 7f2630c43040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:54:06.866 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:06.864+0000 7f188fa38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:54:06.921 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.920+0000 7f2630c43040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:54:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.965+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fefbc06dc10 con 0x7fefbc06ae20 2023-10-16T15:54:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.965+0000 7fefb27fc640 1 -- 172.21.15.29:0/1037658550 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v230) v1 ==== 104+0+11 (secure 0 0 0) 0x7fefb8057ed0 con 0x7fefbc06ae20 2023-10-16T15:54:06.967 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:54:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc1022640 1 -- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 msgr2=0x7fefa006d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:06.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc1022640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 0x7fefa006d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:54:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 msgr2=0x7fefa006d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 0x7fefa006d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 msgr2=0x7fefbc150820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc150820 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fefb800e870 tx=0x7fefb800ed30 comp rx=0 tx=0).stop 2023-10-16T15:54:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 shutdown_connections 2023-10-16T15:54:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fefa006b170 0x7fefa006d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0cea90 0x7fefbc15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefbc15e2f0 0x7fefbc0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 --2- 172.21.15.29:0/1037658550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc06ae20 0x7fefbc150820 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 >> 172.21.15.29:0/1037658550 conn(0x7fefbc0cbc60 msgr2=0x7fefbc15d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 shutdown_connections 2023-10-16T15:54:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:06.968+0000 7fefc3a87640 1 -- 172.21.15.29:0/1037658550 wait complete. 2023-10-16T15:54:06.974 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:06.974+0000 7f2630c43040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:54:06.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:54:07.250 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.249+0000 7f188fa38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:54:07.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.251+0000 7fcc0caa9640 1 Processor -- start 2023-10-16T15:54:07.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.251+0000 7fcc0caa9640 1 -- start start 2023-10-16T15:54:07.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.252+0000 7fcc0caa9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0815ad70 0x7fcc08069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.252+0000 7fcc0caa9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc080cb030 con 0x7fcc080c9ca0 2023-10-16T15:54:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.252+0000 7fcc0caa9640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc080c91c0 con 0x7fcc080caac0 2023-10-16T15:54:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.252+0000 7fcc0caa9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc080c94a0 con 0x7fcc0815ad70 2023-10-16T15:54:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc0659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0815ad70 0x7fcc08069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc0659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0815ad70 0x7fcc08069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44336/0 (socket says 172.21.15.29:44336) 2023-10-16T15:54:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc0659c640 1 -- 172.21.15.29:0/4192635888 learned_addr learned my addr 172.21.15.29:0/4192635888 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc0659c640 1 -- 172.21.15.29:0/4192635888 >> v1:172.21.15.170:6789/0 conn(0x7fcc080caac0 legacy=0x7fcc080ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:07.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc0659c640 1 -- 172.21.15.29:0/4192635888 >> v1:172.21.15.29:6789/0 conn(0x7fcc080c9ca0 legacy=0x7fcc080ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc0659c640 1 -- 172.21.15.29:0/4192635888 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc080c9780 con 0x7fcc0815ad70 2023-10-16T15:54:07.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.253+0000 7fcc05d9b640 1 -- 172.21.15.29:0/4192635888 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391825202 0 0) 0x7fcc080cb030 con 0x7fcc080c9ca0 2023-10-16T15:54:07.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.254+0000 7fcc0659c640 1 --2- 172.21.15.29:0/4192635888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0815ad70 0x7fcc08069da0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009870 tx=0x7fcbf802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33bbfa6225ac125 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:07.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.255+0000 7fcc05d9b640 1 -- 172.21.15.29:0/4192635888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8030860 con 0x7fcc0815ad70 2023-10-16T15:54:07.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.255+0000 7fcc05d9b640 1 -- 172.21.15.29:0/4192635888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf80309c0 con 0x7fcc0815ad70 2023-10-16T15:54:07.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.255+0000 7fcc05d9b640 1 -- 172.21.15.29:0/4192635888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8030d50 con 0x7fcc0815ad70 2023-10-16T15:54:07.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.255+0000 7fcc0caa9640 1 -- 172.21.15.29:0/4192635888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0815ad70 msgr2=0x7fcc08069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.255+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/4192635888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0815ad70 0x7fcc08069da0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009870 tx=0x7fcbf802b9a0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.256+0000 7fcc0caa9640 1 -- 172.21.15.29:0/4192635888 shutdown_connections 2023-10-16T15:54:07.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.256+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/4192635888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0815ad70 0x7fcc08069da0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.256+0000 7fcc0caa9640 1 -- 172.21.15.29:0/4192635888 >> 172.21.15.29:0/4192635888 conn(0x7fcc080ce9b0 msgr2=0x7fcc080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:07.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.256+0000 7fcc0caa9640 1 -- 172.21.15.29:0/4192635888 shutdown_connections 2023-10-16T15:54:07.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.256+0000 7fcc0caa9640 1 -- 172.21.15.29:0/4192635888 wait complete. 2023-10-16T15:54:07.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.257+0000 7fcc0caa9640 1 Processor -- start 2023-10-16T15:54:07.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.258+0000 7fcc0caa9640 1 -- start start 2023-10-16T15:54:07.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0caa9640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc080c9ca0 0x7fcc0815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0caa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 0x7fcc080c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0caa9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc0815ad70 0x7fcc080c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0caa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0815e120 con 0x7fcc080caac0 2023-10-16T15:54:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0caa9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0815c540 con 0x7fcc0815ad70 2023-10-16T15:54:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0caa9640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0815c820 con 0x7fcc080c9ca0 2023-10-16T15:54:07.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 0x7fcc080c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 0x7fcc080c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44346/0 (socket says 172.21.15.29:44346) 2023-10-16T15:54:07.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 -- 172.21.15.29:0/1523558028 learned_addr learned my addr 172.21.15.29:0/1523558028 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:07.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc0659c640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc080c9ca0 0x7fcc0815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcc06d9d640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc0815ad70 0x7fcc080c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 -- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc080c9ca0 msgr2=0x7fcc0815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc080c9ca0 0x7fcc0815dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 -- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc0815ad70 msgr2=0x7fcc080c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc0815ad70 0x7fcc080c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.259+0000 7fcbfffff640 1 -- 172.21.15.29:0/1523558028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf80094d0 con 0x7fcc080caac0 2023-10-16T15:54:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.260+0000 7fcbfffff640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 0x7fcc080c7550 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000c4b0 tx=0x7fcbf000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.260+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0005480 con 0x7fcc080caac0 2023-10-16T15:54:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.260+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf00035f0 con 0x7fcc080caac0 2023-10-16T15:54:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.260+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0815cb00 con 0x7fcc080caac0 2023-10-16T15:54:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.260+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf00039e0 con 0x7fcc080caac0 2023-10-16T15:54:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.260+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0815d0c0 con 0x7fcc080caac0 2023-10-16T15:54:07.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.261+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0815d2e0 con 0x7fcc080caac0 2023-10-16T15:54:07.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.262+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fcbf0003b40 con 0x7fcc080caac0 2023-10-16T15:54:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.264+0000 7fcbfeffd640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fcbe806b140 0x7fcbe806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.264+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fcbf0089680 con 0x7fcc080caac0 2023-10-16T15:54:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.264+0000 7fcc0659c640 1 -- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fcbe806b140 msgr2=0x7fcbe806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.264+0000 7fcc0659c640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fcbe806b140 0x7fcbe806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.269+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcbf0053db0 con 0x7fcc080caac0 2023-10-16T15:54:07.386 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:07.385+0000 7f2630c43040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:54:07.414 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.412+0000 7f188fa38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:54:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.461+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc0804e160 con 0x7fcc080caac0 2023-10-16T15:54:07.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.462+0000 7fcbfeffd640 1 -- 172.21.15.29:0/1523558028 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v230) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbf00170a0 con 0x7fcc080caac0 2023-10-16T15:54:07.463 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:54:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fcbe806b140 msgr2=0x7fcbe806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fcbe806b140 0x7fcbe806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 msgr2=0x7fcc080c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 0x7fcc080c7550 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000c4b0 tx=0x7fcbf000c970 comp rx=0 tx=0).stop 2023-10-16T15:54:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 shutdown_connections 2023-10-16T15:54:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc080caac0 0x7fcc080c7550 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fcbe806b140 0x7fcbe806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcc0815ad70 0x7fcc080c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 --2- 172.21.15.29:0/1523558028 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fcc080c9ca0 0x7fcc0815dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.464+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 >> 172.21.15.29:0/1523558028 conn(0x7fcc080ce9b0 msgr2=0x7fcc0806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.465+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 shutdown_connections 2023-10-16T15:54:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.465+0000 7fcc0caa9640 1 -- 172.21.15.29:0/1523558028 wait complete. 2023-10-16T15:54:07.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T15:54:07.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.520+0000 7f188fa38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:54:07.558 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:07.557+0000 7f2630c43040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:54:07.572 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.570+0000 7f188fa38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:54:07.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:07.672+0000 7f2630c43040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:54:07.678 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.677+0000 7f188fa38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:54:07.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.704+0000 7fe83410f640 1 Processor -- start 2023-10-16T15:54:07.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.704+0000 7fe83410f640 1 -- start start 2023-10-16T15:54:07.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.705+0000 7fe83410f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe82c1428c0 0x7fe82c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.705+0000 7fe83410f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 0x7fe82c0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.705+0000 7fe83410f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c0c9af0 con 0x7fe82c06b660 2023-10-16T15:54:07.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.705+0000 7fe83410f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0cc6b0 con 0x7fe82c1428c0 2023-10-16T15:54:07.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.705+0000 7fe83410f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c1599b0 con 0x7fe82c0c9150 2023-10-16T15:54:07.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.706+0000 7fe8326ac640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe82c06b660 0x7fe82c15d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60134/0 (socket says 172.21.15.29:60134) 2023-10-16T15:54:07.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.706+0000 7fe8326ac640 1 -- 172.21.15.29:0/516439406 learned_addr learned my addr 172.21.15.29:0/516439406 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:07.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.706+0000 7fe8316aa640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 0x7fe82c0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.706+0000 7fe831eab640 1 --2- 172.21.15.29:0/516439406 >> v2:172.21.15.170:3300/0 conn(0x7fe82c1428c0 0x7fe82c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.706+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166519304 0 0) 0x7fe82c0c9af0 con 0x7fe82c06b660 2023-10-16T15:54:07.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818003520 con 0x7fe82c06b660 2023-10-16T15:54:07.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155461510 0 0) 0x7fe818003520 con 0x7fe82c06b660 2023-10-16T15:54:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 msgr2=0x7fe82c0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 0x7fe82c0c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 >> v2:172.21.15.170:3300/0 conn(0x7fe82c1428c0 msgr2=0x7fe82c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 --2- 172.21.15.29:0/516439406 >> v2:172.21.15.170:3300/0 conn(0x7fe82c1428c0 0x7fe82c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c06c290 con 0x7fe82c06b660 2023-10-16T15:54:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe824001670 con 0x7fe82c06b660 2023-10-16T15:54:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 >> v1:172.21.15.29:6789/0 conn(0x7fe82c06b660 legacy=0x7fe82c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818003970 0x7fe818003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8180047c0 0x7fe818004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 0x7fe818005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0cc6b0 con 0x7fe8180047c0 2023-10-16T15:54:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c1599b0 con 0x7fe818003970 2023-10-16T15:54:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81800b980 con 0x7fe82c0c9150 2023-10-16T15:54:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.707+0000 7fe8316aa640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8180047c0 0x7fe818004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.708+0000 7fe831eab640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818003970 0x7fe818003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.708+0000 7fe8316aa640 1 -- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 msgr2=0x7fe818005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.708+0000 7fe8316aa640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe82c0c9150 0x7fe818005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.708+0000 7fe8316aa640 1 -- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818003970 msgr2=0x7fe818003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.708+0000 7fe8316aa640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe818003970 0x7fe818003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.708+0000 7fe8316aa640 1 -- 172.21.15.29:0/516439406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe818003520 con 0x7fe8180047c0 2023-10-16T15:54:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.709+0000 7fe8316aa640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8180047c0 0x7fe818004c40 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009ac0 tx=0x7fe8180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64c94498207d5a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:07.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.709+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02c6d0 con 0x7fe8180047c0 2023-10-16T15:54:07.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.709+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c02c830 con 0x7fe8180047c0 2023-10-16T15:54:07.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.709+0000 7fe830ea9640 1 -- 172.21.15.29:0/516439406 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02cbc0 con 0x7fe8180047c0 2023-10-16T15:54:07.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe83410f640 1 -- 172.21.15.29:0/516439406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8180047c0 msgr2=0x7fe818004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe83410f640 1 --2- 172.21.15.29:0/516439406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8180047c0 0x7fe818004c40 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009ac0 tx=0x7fe8180057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe8326ac640 1 -- 172.21.15.29:0/516439406 reap_dead start 2023-10-16T15:54:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe83410f640 1 -- 172.21.15.29:0/516439406 shutdown_connections 2023-10-16T15:54:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe83410f640 1 -- 172.21.15.29:0/516439406 >> 172.21.15.29:0/516439406 conn(0x7fe82c0ce9b0 msgr2=0x7fe82c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe83410f640 1 -- 172.21.15.29:0/516439406 shutdown_connections 2023-10-16T15:54:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.710+0000 7fe83410f640 1 -- 172.21.15.29:0/516439406 wait complete. 2023-10-16T15:54:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.712+0000 7fe83410f640 1 Processor -- start 2023-10-16T15:54:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.712+0000 7fe83410f640 1 -- start start 2023-10-16T15:54:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe83410f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180047c0 0x7fe82c1ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe83410f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe82c0c7cc0 0x7fe82c0c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe83410f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 0x7fe82c1564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe83410f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0c8680 con 0x7fe82c156010 2023-10-16T15:54:07.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe83410f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c156db0 con 0x7fe82c0c7cc0 2023-10-16T15:54:07.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe83410f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c157090 con 0x7fe8180047c0 2023-10-16T15:54:07.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe8326ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 0x7fe82c1564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe8326ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 0x7fe82c1564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44364/0 (socket says 172.21.15.29:44364) 2023-10-16T15:54:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe8326ac640 1 -- 172.21.15.29:0/2071019660 learned_addr learned my addr 172.21.15.29:0/2071019660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe831eab640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180047c0 0x7fe82c1ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.713+0000 7fe8316aa640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe82c0c7cc0 0x7fe82c0c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8326ac640 1 -- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180047c0 msgr2=0x7fe82c1ee0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8326ac640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180047c0 0x7fe82c1ee0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8326ac640 1 -- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe82c0c7cc0 msgr2=0x7fe82c0c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8326ac640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe82c0c7cc0 0x7fe82c0c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8326ac640 1 -- 172.21.15.29:0/2071019660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c157370 con 0x7fe82c156010 2023-10-16T15:54:07.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe831eab640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180047c0 0x7fe82c1ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:07.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8316aa640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe82c0c7cc0 0x7fe82c0c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:07.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8326ac640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 0x7fe82c1564c0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fe82c071960 tx=0x7fe82400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824007440 con 0x7fe82c156010 2023-10-16T15:54:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8240075e0 con 0x7fe82c156010 2023-10-16T15:54:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824004a20 con 0x7fe82c156010 2023-10-16T15:54:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c157650 con 0x7fe82c156010 2023-10-16T15:54:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.714+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c1ee5e0 con 0x7fe82c156010 2023-10-16T15:54:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.715+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c04e160 con 0x7fe82c156010 2023-10-16T15:54:07.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.716+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fe824004b80 con 0x7fe82c156010 2023-10-16T15:54:07.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.717+0000 7fe8227fc640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fe81006b1c0 0x7fe81006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:07.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.717+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fe8240898b0 con 0x7fe82c156010 2023-10-16T15:54:07.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.717+0000 7fe831eab640 1 -- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fe81006b1c0 msgr2=0x7fe81006d680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:07.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.717+0000 7fe831eab640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fe81006b1c0 0x7fe81006d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:07.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.721+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe824054090 con 0x7fe82c156010 2023-10-16T15:54:07.728 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:07.727+0000 7f2630c43040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:54:07.786 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.784+0000 7f188fa38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:54:07.848 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:07.847+0000 7f2630c43040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:54:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.913+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe82c06ac60 con 0x7fe82c156010 2023-10-16T15:54:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.913+0000 7fe8227fc640 1 -- 172.21.15.29:0/2071019660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v230) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe8240576b0 con 0x7fe82c156010 2023-10-16T15:54:07.914 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:54:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fe81006b1c0 msgr2=0x7fe81006d680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fe81006b1c0 0x7fe81006d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 msgr2=0x7fe82c1564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 0x7fe82c1564c0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fe82c071960 tx=0x7fe82400c960 comp rx=0 tx=0).stop 2023-10-16T15:54:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 shutdown_connections 2023-10-16T15:54:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe82c156010 0x7fe82c1564c0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fe81006b1c0 0x7fe81006d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe82c0c7cc0 0x7fe82c0c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 --2- 172.21.15.29:0/2071019660 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe8180047c0 0x7fe82c1ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 >> 172.21.15.29:0/2071019660 conn(0x7fe82c0ce9b0 msgr2=0x7fe82c1554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.916+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 shutdown_connections 2023-10-16T15:54:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:07.917+0000 7fe83410f640 1 -- 172.21.15.29:0/2071019660 wait complete. 2023-10-16T15:54:07.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T15:54:07.942 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.940+0000 7f188fa38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:54:07.963 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:07.962+0000 7f2630c43040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:54:07.991 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:07.989+0000 7f188fa38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:54:08.110 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:08.108+0000 7f188fa38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:54:08.134 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:08.133+0000 7f2630c43040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:54:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.157+0000 7fbe09410640 1 Processor -- start 2023-10-16T15:54:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.157+0000 7fbe09410640 1 -- start start 2023-10-16T15:54:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe09410640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fbe040cf710 0x7fbe040b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe09410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbe040b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe09410640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe040d3980 con 0x7fbe040b38e0 2023-10-16T15:54:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe09410640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe040d3c60 con 0x7fbe040cf710 2023-10-16T15:54:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe09410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040ce630 con 0x7fbe040cfaf0 2023-10-16T15:54:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe037fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbe040b38e0 0x7fbe040d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60144/0 (socket says 172.21.15.29:60144) 2023-10-16T15:54:08.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.159+0000 7fbe037fe640 1 -- 172.21.15.29:0/3116163304 learned_addr learned my addr 172.21.15.29:0/3116163304 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/3116163304 >> v2:172.21.15.170:3300/0 conn(0x7fbe040cf710 0x7fbe040b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799564352 0 0) 0x7fbe040d3980 con 0x7fbe040b38e0 2023-10-16T15:54:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe027fc640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbe040b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec003520 con 0x7fbe040b38e0 2023-10-16T15:54:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1519586443 0 0) 0x7fbdec003520 con 0x7fbe040b38e0 2023-10-16T15:54:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 msgr2=0x7fbe040b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbe040b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 >> v2:172.21.15.170:3300/0 conn(0x7fbe040cf710 msgr2=0x7fbe040b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 --2- 172.21.15.29:0/3116163304 >> v2:172.21.15.170:3300/0 conn(0x7fbe040cf710 0x7fbe040b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe040ce910 con 0x7fbe040b38e0 2023-10-16T15:54:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbdfc0019f0 con 0x7fbe040b38e0 2023-10-16T15:54:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 >> v1:172.21.15.29:6789/0 conn(0x7fbe040b38e0 legacy=0x7fbe040d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec003970 0x7fbdec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbdec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdec004d00 0x7fbdec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040d3c60 con 0x7fbdec003970 2023-10-16T15:54:08.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040ce630 con 0x7fbdec004d00 2023-10-16T15:54:08.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec00bae0 con 0x7fbe040cfaf0 2023-10-16T15:54:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.160+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec003970 0x7fbdec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe027fc640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbdec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe037fe640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdec004d00 0x7fbdec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe02ffd640 1 -- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 msgr2=0x7fbdec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbdec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe02ffd640 1 -- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdec004d00 msgr2=0x7fbdec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:08.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdec004d00 0x7fbdec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe02ffd640 1 -- 172.21.15.29:0/3116163304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec003520 con 0x7fbdec003970 2023-10-16T15:54:08.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe027fc640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbe040cfaf0 0x7fbdec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:08.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.161+0000 7fbe037fe640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbdec004d00 0x7fbdec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:08.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.162+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec003970 0x7fbdec003d90 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400ba10 tx=0x7fbdec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ede9adc35d0d82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:08.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.162+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4005b90 con 0x7fbdec003970 2023-10-16T15:54:08.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.162+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4005cf0 con 0x7fbdec003970 2023-10-16T15:54:08.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.162+0000 7fbe01ffb640 1 -- 172.21.15.29:0/3116163304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4006270 con 0x7fbdec003970 2023-10-16T15:54:08.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe09410640 1 -- 172.21.15.29:0/3116163304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec003970 msgr2=0x7fbdec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe09410640 1 --2- 172.21.15.29:0/3116163304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdec003970 0x7fbdec003d90 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400ba10 tx=0x7fbdec0057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe037fe640 1 -- 172.21.15.29:0/3116163304 reap_dead start 2023-10-16T15:54:08.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe09410640 1 -- 172.21.15.29:0/3116163304 shutdown_connections 2023-10-16T15:54:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe09410640 1 -- 172.21.15.29:0/3116163304 >> 172.21.15.29:0/3116163304 conn(0x7fbe040d2cc0 msgr2=0x7fbe040d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe09410640 1 -- 172.21.15.29:0/3116163304 shutdown_connections 2023-10-16T15:54:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.163+0000 7fbe09410640 1 -- 172.21.15.29:0/3116163304 wait complete. 2023-10-16T15:54:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.165+0000 7fbe09410640 1 Processor -- start 2023-10-16T15:54:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.165+0000 7fbe09410640 1 -- start start 2023-10-16T15:54:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe09410640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdec003970 0x7fbe0415c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe09410640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe041f2fd0 0x7fbe040b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe09410640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 0x7fbe040c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe09410640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040b0060 con 0x7fbe040b0a30 2023-10-16T15:54:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe09410640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040afa80 con 0x7fbe041f2fd0 2023-10-16T15:54:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe09410640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040b1940 con 0x7fbdec003970 2023-10-16T15:54:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 0x7fbe040c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 0x7fbe040c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44382/0 (socket says 172.21.15.29:44382) 2023-10-16T15:54:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.166+0000 7fbe037fe640 1 -- 172.21.15.29:0/1616592979 learned_addr learned my addr 172.21.15.29:0/1616592979 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe027fc640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe041f2fd0 0x7fbe040b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdec003970 0x7fbe0415c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe037fe640 1 -- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdec003970 msgr2=0x7fbe0415c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe037fe640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdec003970 0x7fbe0415c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe037fe640 1 -- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe041f2fd0 msgr2=0x7fbe040b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe037fe640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe041f2fd0 0x7fbe040b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe037fe640 1 -- 172.21.15.29:0/1616592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe040afcf0 con 0x7fbe040b0a30 2023-10-16T15:54:08.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdec003970 0x7fbe0415c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:08.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe037fe640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 0x7fbe040c9d90 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000f80 tx=0x7fbdfc00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:08.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc00b8d0 con 0x7fbe040b0a30 2023-10-16T15:54:08.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdfc00ba30 con 0x7fbe040b0a30 2023-10-16T15:54:08.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe040cbe20 con 0x7fbe040b0a30 2023-10-16T15:54:08.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.167+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc00bdc0 con 0x7fbe040b0a30 2023-10-16T15:54:08.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.168+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe040b15a0 con 0x7fbe040b0a30 2023-10-16T15:54:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.169+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe040b01a0 con 0x7fbe040b0a30 2023-10-16T15:54:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.170+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fbdfc007110 con 0x7fbe040b0a30 2023-10-16T15:54:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.171+0000 7fbdeb7fe640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 0x7fbdf006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.171+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fbdfc08a190 con 0x7fbe040b0a30 2023-10-16T15:54:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.171+0000 7fbe02ffd640 1 -- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 msgr2=0x7fbdf006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.171+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 0x7fbdf006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.176+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbdfc054990 con 0x7fbe040b0a30 2023-10-16T15:54:08.189 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:08.188+0000 7f2630c43040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:54:08.319 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:08.318+0000 7f2630c43040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:54:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.372+0000 7fbe02ffd640 1 -- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 msgr2=0x7fbdf006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.372+0000 7fbe02ffd640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 0x7fbdf006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:54:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.375+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe04073100 con 0x7fbe040b0a30 2023-10-16T15:54:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.376+0000 7fbdeb7fe640 1 -- 172.21.15.29:0/1616592979 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v230) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbdfc057fb0 con 0x7fbe040b0a30 2023-10-16T15:54:08.377 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:54:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.379+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 msgr2=0x7fbdf006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.379+0000 7fbe09410640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 0x7fbdf006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 msgr2=0x7fbe040c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 0x7fbe040c9d90 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000f80 tx=0x7fbdfc00b0d0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 shutdown_connections 2023-10-16T15:54:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe040b0a30 0x7fbe040c9d90 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7fbdf006b140 0x7fbdf006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe041f2fd0 0x7fbe040b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 --2- 172.21.15.29:0/1616592979 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbdec003970 0x7fbe0415c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 >> 172.21.15.29:0/1616592979 conn(0x7fbe040d2cc0 msgr2=0x7fbe040d1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 shutdown_connections 2023-10-16T15:54:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.380+0000 7fbe09410640 1 -- 172.21.15.29:0/1616592979 wait complete. 2023-10-16T15:54:08.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:54:08.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.666+0000 7f77ca864640 1 Processor -- start 2023-10-16T15:54:08.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.666+0000 7f77ca864640 1 -- start start 2023-10-16T15:54:08.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.668+0000 7f77ca864640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f77c415ad70 0x7f77c4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.668+0000 7f77ca864640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.668+0000 7f77ca864640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40cb030 con 0x7f77c40ca3b0 2023-10-16T15:54:08.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.668+0000 7f77ca864640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c40c91c0 con 0x7f77c415ad70 2023-10-16T15:54:08.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.668+0000 7f77ca864640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40c94a0 con 0x7f77c40c9ca0 2023-10-16T15:54:08.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f77c415ad70 0x7f77c4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:53902/0 (socket says 172.21.15.29:53902) 2023-10-16T15:54:08.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c37fe640 1 -- 172.21.15.29:0/1191639324 learned_addr learned my addr 172.21.15.29:0/1191639324 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:08.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c3fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f77c415ad70 0x7f77c4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:46772/0 (socket says 172.21.15.29:46772) 2023-10-16T15:54:08.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c3fff640 1 -- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 msgr2=0x7f77c406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:08.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c3fff640 1 --2- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c3fff640 1 -- 172.21.15.29:0/1191639324 >> v1:172.21.15.170:6789/0 conn(0x7f77c40ca3b0 legacy=0x7f77c40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c3fff640 1 -- 172.21.15.29:0/1191639324 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40c9780 con 0x7f77c415ad70 2023-10-16T15:54:08.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.669+0000 7f77c37fe640 1 --2- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:08.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.670+0000 7f77c3fff640 1 --2- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c4069da0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f77b40097a0 tx=0x7f77b402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e148fc597a64ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77c2ffd640 1 -- 172.21.15.29:0/1191639324 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b402d040 con 0x7f77c415ad70 2023-10-16T15:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77c2ffd640 1 -- 172.21.15.29:0/1191639324 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b4031810 con 0x7f77c415ad70 2023-10-16T15:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77c2ffd640 1 -- 172.21.15.29:0/1191639324 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b4031ba0 con 0x7f77c415ad70 2023-10-16T15:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77ca864640 1 -- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 msgr2=0x7f77c4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77ca864640 1 --2- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c4069da0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f77b40097a0 tx=0x7f77b402b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77ca864640 1 -- 172.21.15.29:0/1191639324 shutdown_connections 2023-10-16T15:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77ca864640 1 --2- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77ca864640 1 --2- 172.21.15.29:0/1191639324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c4069da0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.671+0000 7f77ca864640 1 -- 172.21.15.29:0/1191639324 >> 172.21.15.29:0/1191639324 conn(0x7f77c40ce9b0 msgr2=0x7f77c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.672+0000 7f77ca864640 1 -- 172.21.15.29:0/1191639324 shutdown_connections 2023-10-16T15:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.672+0000 7f77ca864640 1 -- 172.21.15.29:0/1191639324 wait complete. 2023-10-16T15:54:08.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.673+0000 7f77ca864640 1 Processor -- start 2023-10-16T15:54:08.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.673+0000 7f77ca864640 1 -- start start 2023-10-16T15:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.674+0000 7f77ca864640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c40b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.674+0000 7f77ca864640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 0x7f77c4149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.674+0000 7f77ca864640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c414c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.674+0000 7f77ca864640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c4157440 con 0x7f77c40ca3b0 2023-10-16T15:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.674+0000 7f77ca864640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40b27c0 con 0x7f77c415ad70 2023-10-16T15:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.674+0000 7f77ca864640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40b2a60 con 0x7f77c40c9ca0 2023-10-16T15:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 0x7f77c4149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 0x7f77c4149740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44388/0 (socket says 172.21.15.29:44388) 2023-10-16T15:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 -- 172.21.15.29:0/1134778477 learned_addr learned my addr 172.21.15.29:0/1134778477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c3fff640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c40b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c8e01640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c414c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 -- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 msgr2=0x7f77c40b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c40b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 -- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 msgr2=0x7f77c414c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c414c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 -- 172.21.15.29:0/1134778477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b80094d0 con 0x7f77c40ca3b0 2023-10-16T15:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.675+0000 7f77c37fe640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 0x7f77c4149740 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f77b800e9d0 tx=0x7f77b800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.676+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b800ca60 con 0x7f77c40ca3b0 2023-10-16T15:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.676+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b800cbc0 con 0x7f77c40ca3b0 2023-10-16T15:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.676+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b8013470 con 0x7f77c40ca3b0 2023-10-16T15:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.676+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b4009400 con 0x7f77c40ca3b0 2023-10-16T15:54:08.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.676+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c40b3300 con 0x7f77c40ca3b0 2023-10-16T15:54:08.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.677+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c40b2e30 con 0x7f77c40ca3b0 2023-10-16T15:54:08.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.678+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f77b80135d0 con 0x7f77c40ca3b0 2023-10-16T15:54:08.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.679+0000 7f77c0ff9640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 0x7f77b006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:08.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.679+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f77b808a270 con 0x7f77c40ca3b0 2023-10-16T15:54:08.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.679+0000 7f77c3fff640 1 -- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 msgr2=0x7f77b006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:08.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.679+0000 7f77c3fff640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 0x7f77b006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:08.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.687+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f77b8054ae0 con 0x7f77c40ca3b0 2023-10-16T15:54:08.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.880+0000 7f77c3fff640 1 -- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 msgr2=0x7f77b006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:08.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.880+0000 7f77c3fff640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 0x7f77b006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:54:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.885+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f77c404e1b0 con 0x7f77c40ca3b0 2023-10-16T15:54:08.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.886+0000 7f77c0ff9640 1 -- 172.21.15.29:0/1134778477 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v230) v1 ==== 80+0+477 (secure 0 0 0) 0x7f77b8058100 con 0x7f77c40ca3b0 2023-10-16T15:54:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/3980026359 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2963466728 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/606018791 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/3980026359 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2707368093 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.888 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3876776495 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.888 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1834694957 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.888 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1551430148 2023-10-17T15:53:58.385901+0000 2023-10-16T15:54:08.888 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-16T15:54:08.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 msgr2=0x7f77b006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:08.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 0x7f77b006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 msgr2=0x7f77c4149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 0x7f77c4149740 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f77b800e9d0 tx=0x7f77b800ee90 comp rx=0 tx=0).stop 2023-10-16T15:54:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 shutdown_connections 2023-10-16T15:54:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77c40ca3b0 0x7f77c4149740 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f77b006b140 0x7f77b006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77c415ad70 0x7f77c414c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 --2- 172.21.15.29:0/1134778477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77c40c9ca0 0x7f77c40b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 >> 172.21.15.29:0/1134778477 conn(0x7f77c40ce9b0 msgr2=0x7f77c40ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 shutdown_connections 2023-10-16T15:54:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:08.888+0000 7f77ca864640 1 -- 172.21.15.29:0/1134778477 wait complete. 2023-10-16T15:54:08.906 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:54:08.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:08.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:54:08.940 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:08.939+0000 7f188fa38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:54:08.991 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:54:08.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:08.992 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:54:09.035 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:54:09.035 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:09.091 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:54:09.091 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:54:09.091 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:09.147 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:54:09.148 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:54:09.148 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:09.190 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:09.189+0000 7f2630c43040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:54:09.205 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:54:09.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:09.245 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.244+0000 7f188fa38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:54:09.307 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.305+0000 7f188fa38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:54:09.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.417+0000 7f411d640640 1 Processor -- start 2023-10-16T15:54:09.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.417+0000 7f411d640640 1 -- start start 2023-10-16T15:54:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f411d640640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f411806b5c0 0x7f41180cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f411d640640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180d0180 con 0x7f41180afef0 2023-10-16T15:54:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f411d640640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180cfea0 con 0x7f41180b24a0 2023-10-16T15:54:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f411d640640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41180b04d0 con 0x7f411806b5c0 2023-10-16T15:54:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f4116ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41180afef0 0x7f411806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60160/0 (socket says 172.21.15.29:60160) 2023-10-16T15:54:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f4116ffd640 1 -- 172.21.15.29:0/3986365446 learned_addr learned my addr 172.21.15.29:0/3986365446 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.419+0000 7f41167fc640 1 --2- 172.21.15.29:0/3986365446 >> v2:172.21.15.29:3300/0 conn(0x7f411806b5c0 0x7f41180cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.420+0000 7f4115ffb640 1 -- 172.21.15.29:0/3986365446 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762134041 0 0) 0x7f41180d0180 con 0x7f41180afef0 2023-10-16T15:54:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.420+0000 7f4115ffb640 1 -- 172.21.15.29:0/3986365446 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4100003520 con 0x7f41180afef0 2023-10-16T15:54:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.420+0000 7f41167fc640 1 -- 172.21.15.29:0/3986365446 >> v1:172.21.15.170:6789/0 conn(0x7f41180b24a0 legacy=0x7f41180b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.420+0000 7f41167fc640 1 -- 172.21.15.29:0/3986365446 >> v1:172.21.15.29:6789/0 conn(0x7f41180afef0 legacy=0x7f411806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.420+0000 7f41167fc640 1 -- 172.21.15.29:0/3986365446 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180b07b0 con 0x7f411806b5c0 2023-10-16T15:54:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.421+0000 7f41167fc640 1 --2- 172.21.15.29:0/3986365446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f411806b5c0 0x7f41180cf8c0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f410c0097a0 tx=0x7f410c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e8f8bd82a4aca1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.421+0000 7f4115ffb640 1 -- 172.21.15.29:0/3986365446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c02e070 con 0x7f411806b5c0 2023-10-16T15:54:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.421+0000 7f4115ffb640 1 -- 172.21.15.29:0/3986365446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c0315a0 con 0x7f411806b5c0 2023-10-16T15:54:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.421+0000 7f4115ffb640 1 -- 172.21.15.29:0/3986365446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c031930 con 0x7f411806b5c0 2023-10-16T15:54:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 -- 172.21.15.29:0/3986365446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f411806b5c0 msgr2=0x7f41180cf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 --2- 172.21.15.29:0/3986365446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f411806b5c0 0x7f41180cf8c0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f410c0097a0 tx=0x7f410c02b840 comp rx=0 tx=0).stop 2023-10-16T15:54:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 -- 172.21.15.29:0/3986365446 shutdown_connections 2023-10-16T15:54:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 --2- 172.21.15.29:0/3986365446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f411806b5c0 0x7f41180cf8c0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 -- 172.21.15.29:0/3986365446 >> 172.21.15.29:0/3986365446 conn(0x7f41180d0ad0 msgr2=0x7f41180d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 -- 172.21.15.29:0/3986365446 shutdown_connections 2023-10-16T15:54:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.422+0000 7f411d640640 1 -- 172.21.15.29:0/3986365446 wait complete. 2023-10-16T15:54:09.426 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.420+0000 7f188fa38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:54:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.424+0000 7f411d640640 1 Processor -- start 2023-10-16T15:54:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.424+0000 7f411d640640 1 -- start start 2023-10-16T15:54:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.424+0000 7f411d640640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41180afef0 0x7f41180c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f411d640640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41180b24a0 0x7f41180c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f411d640640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 0x7f41180cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f411d640640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41180c8410 con 0x7f41180ca8d0 2023-10-16T15:54:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f411d640640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41180c86f0 con 0x7f41180afef0 2023-10-16T15:54:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f411d640640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41180c89d0 con 0x7f41180b24a0 2023-10-16T15:54:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 0x7f41180cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 0x7f41180cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44414/0 (socket says 172.21.15.29:44414) 2023-10-16T15:54:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 -- 172.21.15.29:0/4080903519 learned_addr learned my addr 172.21.15.29:0/4080903519 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41167fc640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41180b24a0 0x7f41180c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 -- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41180b24a0 msgr2=0x7f41180c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f4116ffd640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41180afef0 0x7f41180c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41180b24a0 0x7f41180c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 -- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41180afef0 msgr2=0x7f41180c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41180afef0 0x7f41180c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.425+0000 7f41177fe640 1 -- 172.21.15.29:0/4080903519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c009400 con 0x7f41180ca8d0 2023-10-16T15:54:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.426+0000 7f41177fe640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 0x7f41180cc0c0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f411000c7a0 tx=0x7f411000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.427+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4110003b70 con 0x7f41180ca8d0 2023-10-16T15:54:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.427+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41180c8cb0 con 0x7f41180ca8d0 2023-10-16T15:54:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.427+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41180c8050 con 0x7f41180ca8d0 2023-10-16T15:54:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.427+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4110001830 con 0x7f41180ca8d0 2023-10-16T15:54:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.427+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000a850 con 0x7f41180ca8d0 2023-10-16T15:54:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.428+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41180c91d0 con 0x7f41180ca8d0 2023-10-16T15:54:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.430+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f411000aa70 con 0x7f41180ca8d0 2023-10-16T15:54:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.431+0000 7f40ff7fe640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f410406b140 0x7f410406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.431+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f4110089e40 con 0x7f41180ca8d0 2023-10-16T15:54:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.431+0000 7f4116ffd640 1 -- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f410406b140 msgr2=0x7f410406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1824188539 2023-10-16T15:54:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.431+0000 7f4116ffd640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f410406b140 0x7f410406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:54:09.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.436+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 135) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f411000ad20 con 0x7f41180ca8d0 2023-10-16T15:54:09.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.436+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f410406b140 msgr2=0x7f410406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:54:09.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.436+0000 7f40ff7fe640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f410406b140 0x7f410406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.443+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4110081bf0 con 0x7f41180ca8d0 2023-10-16T15:54:09.516 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:09.515+0000 7f2630c43040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:54:09.521 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.519+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.521 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.519+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.519+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.519+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.519+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.520+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.522 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.520+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.523 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.520+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.523 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.520+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.523 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.520+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.535 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.533+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.535 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.533+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.535 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.533+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.535 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.533+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.536 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.533+0000 7f1813487640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.536 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.534+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.536 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.534+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.536 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.534+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.537 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.534+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.537 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:54:09.534+0000 7f1811483640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:54:09.578 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:09.577+0000 7f2630c43040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:54:09.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.672+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41180c81e0 con 0x7f41180ca8d0 2023-10-16T15:54:09.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.673+0000 7f40ff7fe640 1 -- 172.21.15.29:0/4080903519 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f4110057e00 con 0x7f41180ca8d0 2023-10-16T15:54:09.674 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:09.674 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:09.674 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 114 2023-10-16T15:54:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 msgr2=0x7f41180cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 0x7f41180cc0c0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f411000c7a0 tx=0x7f411000cc60 comp rx=0 tx=0).stop 2023-10-16T15:54:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 shutdown_connections 2023-10-16T15:54:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41180ca8d0 0x7f41180cc0c0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:6832/1824188539,v1:172.21.15.170:6833/1824188539] conn(0x7f410406b140 0x7f410406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41180b24a0 0x7f41180c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 --2- 172.21.15.29:0/4080903519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41180afef0 0x7f41180c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.675+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 >> 172.21.15.29:0/4080903519 conn(0x7f41180d0ad0 msgr2=0x7f41180aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.676+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 shutdown_connections 2023-10-16T15:54:09.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.676+0000 7f411d640640 1 -- 172.21.15.29:0/4080903519 wait complete. 2023-10-16T15:54:09.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:09.703 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:54:09.697+0000 7f2630c43040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:54:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.905+0000 7ffb1ffff640 1 Processor -- start 2023-10-16T15:54:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.905+0000 7ffb1ffff640 1 -- start start 2023-10-16T15:54:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.907+0000 7ffb1ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb200ad970 0x7ffb200cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.907+0000 7ffb1ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffb200ccfb0 0x7ffb200cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.907+0000 7ffb1ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2015b9d0 con 0x7ffb2006c040 2023-10-16T15:54:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.907+0000 7ffb1ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2015bcb0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.907+0000 7ffb1ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015bf90 con 0x7ffb200ad970 2023-10-16T15:54:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.908+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb200ad970 0x7ffb200cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.908+0000 7ffb1f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffb200ccfb0 0x7ffb200cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.908+0000 7ffb1f7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffb200ccfb0 0x7ffb200cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:46784/0 (socket says 172.21.15.29:46784) 2023-10-16T15:54:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.908+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3028356798 learned_addr learned my addr 172.21.15.29:0/3028356798 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.908+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb200ad970 0x7ffb200cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:53922/0 (socket says 172.21.15.29:53922) 2023-10-16T15:54:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.909+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb200ad970 msgr2=0x7ffb200cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.909+0000 7ffb1f7fe640 1 --2- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb200ad970 0x7ffb200cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.909+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3028356798 >> v1:172.21.15.170:6789/0 conn(0x7ffb2006c040 legacy=0x7ffb2006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.909+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3028356798 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200cc270 con 0x7ffb200ccfb0 2023-10-16T15:54:09.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.910+0000 7ffb1f7fe640 1 --2- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ccfb0 0x7ffb200cd440 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009e80 tx=0x7ffb1802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef375180f4b659cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.911+0000 7ffb1e7fc640 1 -- 172.21.15.29:0/3028356798 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1802caf0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.911+0000 7ffb1e7fc640 1 -- 172.21.15.29:0/3028356798 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1802cc50 con 0x7ffb200ccfb0 2023-10-16T15:54:09.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.911+0000 7ffb1e7fc640 1 -- 172.21.15.29:0/3028356798 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb180343f0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.911+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ccfb0 msgr2=0x7ffb200cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.911+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ccfb0 0x7ffb200cd440 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009e80 tx=0x7ffb1802bf50 comp rx=0 tx=0).stop 2023-10-16T15:54:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.912+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3028356798 shutdown_connections 2023-10-16T15:54:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.912+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ccfb0 0x7ffb200cd440 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.912+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3028356798 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb200ad970 0x7ffb200cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.912+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3028356798 >> 172.21.15.29:0/3028356798 conn(0x7ffb200ce6a0 msgr2=0x7ffb200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.912+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3028356798 shutdown_connections 2023-10-16T15:54:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.912+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3028356798 wait complete. 2023-10-16T15:54:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.913+0000 7ffb1ffff640 1 Processor -- start 2023-10-16T15:54:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 -- start start 2023-10-16T15:54:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb2006c040 0x7ffb20156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ad970 0x7ffb201596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 0x7ffb200b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb201574d0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015a010 con 0x7ffb200ad970 2023-10-16T15:54:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.914+0000 7ffb1ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015a2f0 con 0x7ffb2006c040 2023-10-16T15:54:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 0x7ffb200b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 0x7ffb200b0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44426/0 (socket says 172.21.15.29:44426) 2023-10-16T15:54:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3356728498 learned_addr learned my addr 172.21.15.29:0/3356728498 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1effd640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb2006c040 0x7ffb20156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb17fff640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ad970 0x7ffb201596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb2006c040 msgr2=0x7ffb20156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb2006c040 0x7ffb20156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ad970 msgr2=0x7ffb201596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ad970 0x7ffb201596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.915+0000 7ffb1f7fe640 1 -- 172.21.15.29:0/3356728498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18009b30 con 0x7ffb200ccfb0 2023-10-16T15:54:09.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.916+0000 7ffb1f7fe640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 0x7ffb200b0680 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7ffb18036040 tx=0x7ffb18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:09.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.916+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18040670 con 0x7ffb200ccfb0 2023-10-16T15:54:09.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.916+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb180407f0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.916+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18040ba0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.916+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2015a5d0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.916+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb20159be0 con 0x7ffb200ccfb0 2023-10-16T15:54:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.917+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2006b330 con 0x7ffb200ccfb0 2023-10-16T15:54:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.919+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 135) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ffb18040d00 con 0x7ffb200ccfb0 2023-10-16T15:54:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.919+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7ffb180ac670 con 0x7ffb200ccfb0 2023-10-16T15:54:09.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:09.924+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffb180774c0 con 0x7ffb200ccfb0 2023-10-16T15:54:10.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.117+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb200ae0c0 con 0x7ffb200ccfb0 2023-10-16T15:54:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.122+0000 7ffb177fe640 1 -- 172.21.15.29:0/3356728498 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+21557 (secure 0 0 0) 0x7ffb200ae0c0 con 0x7ffb200ccfb0 2023-10-16T15:54:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:10.124 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":231,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:09.426111+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 msgr2=0x7ffb200b0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 0x7ffb200b0680 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7ffb18036040 tx=0x7ffb18004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 shutdown_connections 2023-10-16T15:54:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb200ccfb0 0x7ffb200b0680 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffb200ad970 0x7ffb201596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 --2- 172.21.15.29:0/3356728498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffb2006c040 0x7ffb20156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 >> 172.21.15.29:0/3356728498 conn(0x7ffb200ce6a0 msgr2=0x7ffb200c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:10.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 shutdown_connections 2023-10-16T15:54:10.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.126+0000 7ffb1ffff640 1 -- 172.21.15.29:0/3356728498 wait complete. 2023-10-16T15:54:10.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:53:24.608039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:53:25.694889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T15:53:39.139862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T15:53:39.864914+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:54:10.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:54:10.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.371+0000 7f413d19e640 1 Processor -- start 2023-10-16T15:54:10.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.371+0000 7f413d19e640 1 -- start start 2023-10-16T15:54:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.373+0000 7f413d19e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f41380b3ec0 0x7f41380cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.373+0000 7f413d19e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b0ea0 0x7f41380cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.373+0000 7f413d19e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413806be30 con 0x7f41380b43c0 2023-10-16T15:54:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.373+0000 7f413d19e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413806ca00 con 0x7f41380b3ec0 2023-10-16T15:54:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.373+0000 7f413d19e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41380ada10 con 0x7f41380b0ea0 2023-10-16T15:54:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f41380b3ec0 0x7f41380cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f41380b3ec0 0x7f41380cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:46788/0 (socket says 172.21.15.29:46788) 2023-10-16T15:54:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 -- 172.21.15.29:0/918339096 learned_addr learned my addr 172.21.15.29:0/918339096 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f412ffff640 1 --2- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b0ea0 0x7f41380cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 -- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b0ea0 msgr2=0x7f41380cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 --2- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b0ea0 0x7f41380cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 -- 172.21.15.29:0/918339096 >> v1:172.21.15.170:6789/0 conn(0x7f41380b43c0 legacy=0x7f413815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.374+0000 7f4137fff640 1 -- 172.21.15.29:0/918339096 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413806c060 con 0x7f41380b3ec0 2023-10-16T15:54:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.376+0000 7f4137fff640 1 --2- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b3ec0 0x7f41380cd260 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4128009870 tx=0x7f412802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=996266dd31177d68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.376+0000 7f41377fe640 1 -- 172.21.15.29:0/918339096 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128031640 con 0x7f41380b3ec0 2023-10-16T15:54:10.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.376+0000 7f41377fe640 1 -- 172.21.15.29:0/918339096 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41280317a0 con 0x7f41380b3ec0 2023-10-16T15:54:10.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.376+0000 7f41377fe640 1 -- 172.21.15.29:0/918339096 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128031b30 con 0x7f41380b3ec0 2023-10-16T15:54:10.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 -- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b3ec0 msgr2=0x7f41380cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 --2- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b3ec0 0x7f41380cd260 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4128009870 tx=0x7f412802b8f0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 -- 172.21.15.29:0/918339096 shutdown_connections 2023-10-16T15:54:10.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 --2- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b0ea0 0x7f41380cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 --2- 172.21.15.29:0/918339096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b3ec0 0x7f41380cd260 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 -- 172.21.15.29:0/918339096 >> 172.21.15.29:0/918339096 conn(0x7f41380b0170 msgr2=0x7f41380b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:10.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 -- 172.21.15.29:0/918339096 shutdown_connections 2023-10-16T15:54:10.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.377+0000 7f413d19e640 1 -- 172.21.15.29:0/918339096 wait complete. 2023-10-16T15:54:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.379+0000 7f413d19e640 1 Processor -- start 2023-10-16T15:54:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.379+0000 7f413d19e640 1 -- start start 2023-10-16T15:54:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413d19e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b0ea0 0x7f41381f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413d19e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b3ec0 0x7f41380c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413d19e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 0x7f41380c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413d19e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41381568b0 con 0x7f41380b43c0 2023-10-16T15:54:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413d19e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41381f7ad0 con 0x7f41380b0ea0 2023-10-16T15:54:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413d19e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4138158720 con 0x7f41380b3ec0 2023-10-16T15:54:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413c99d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 0x7f41380c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413c99d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 0x7f41380c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44436/0 (socket says 172.21.15.29:44436) 2023-10-16T15:54:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.380+0000 7f413c99d640 1 -- 172.21.15.29:0/538072620 learned_addr learned my addr 172.21.15.29:0/538072620 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f412ffff640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b3ec0 0x7f41380c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f4137fff640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b0ea0 0x7f41381f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413c99d640 1 -- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b3ec0 msgr2=0x7f41380c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413c99d640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b3ec0 0x7f41380c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413c99d640 1 -- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b0ea0 msgr2=0x7f41381f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413c99d640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b0ea0 0x7f41381f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413c99d640 1 -- 172.21.15.29:0/538072620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41280094d0 con 0x7f41380b43c0 2023-10-16T15:54:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413c99d640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 0x7f41380c7ec0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f413000c730 tx=0x7f413000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130006d50 con 0x7f41380b43c0 2023-10-16T15:54:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130003970 con 0x7f41380b43c0 2023-10-16T15:54:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4138158a00 con 0x7f41380b43c0 2023-10-16T15:54:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130003d60 con 0x7f41380b43c0 2023-10-16T15:54:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.381+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4138158fc0 con 0x7f41380b43c0 2023-10-16T15:54:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.383+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413804e9b0 con 0x7f41380b43c0 2023-10-16T15:54:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.383+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 135) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f41300052e0 con 0x7f41380b43c0 2023-10-16T15:54:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.384+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f4130089610 con 0x7f41380b43c0 2023-10-16T15:54:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.388+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4130054370 con 0x7f41380b43c0 2023-10-16T15:54:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.448+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 136) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4130053e20 con 0x7f41380b43c0 2023-10-16T15:54:10.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.449+0000 7f41357fa640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4118072d90 0x7f4118075180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.455+0000 7f4137fff640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4118072d90 0x7f4118075180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.462+0000 7f4137fff640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4118072d90 0x7f4118075180 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4128036000 tx=0x7f412803c040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.594+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f413806abd0 con 0x7f41380b43c0 2023-10-16T15:54:10.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.595+0000 7f41357fa640 1 -- 172.21.15.29:0/538072620 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v231) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41300725a0 con 0x7f41380b43c0 2023-10-16T15:54:10.596 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:54:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.597+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4118072d90 msgr2=0x7f4118075180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4118072d90 0x7f4118075180 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4128036000 tx=0x7f412803c040 comp rx=0 tx=0).stop 2023-10-16T15:54:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 msgr2=0x7f41380c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 0x7f41380c7ec0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f413000c730 tx=0x7f413000cbf0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 shutdown_connections 2023-10-16T15:54:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4118072d90 0x7f4118075180 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41380b43c0 0x7f41380c7ec0 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f41380b3ec0 0x7f41380c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 --2- 172.21.15.29:0/538072620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f41380b0ea0 0x7f41381f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 >> 172.21.15.29:0/538072620 conn(0x7f41380b0170 msgr2=0x7f41380af750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:10.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 shutdown_connections 2023-10-16T15:54:10.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.598+0000 7f413d19e640 1 -- 172.21.15.29:0/538072620 wait complete. 2023-10-16T15:54:10.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:54:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.846+0000 7f62d8125640 1 Processor -- start 2023-10-16T15:54:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.846+0000 7f62d8125640 1 -- start start 2023-10-16T15:54:10.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.847+0000 7f62d8125640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62d00ad350 0x7f62d0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d8125640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f62d006a2e0 0x7f62d00c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d8125640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d006a730 con 0x7f62d015d150 2023-10-16T15:54:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d8125640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d015d530 con 0x7f62d00ad350 2023-10-16T15:54:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d8125640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d00ac890 con 0x7f62d006a2e0 2023-10-16T15:54:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62d00ad350 0x7f62d0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62d00ad350 0x7f62d0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44448/0 (socket says 172.21.15.29:44448) 2023-10-16T15:54:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 -- 172.21.15.29:0/2620947571 learned_addr learned my addr 172.21.15.29:0/2620947571 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 -- 172.21.15.29:0/2620947571 >> v2:172.21.15.170:3300/0 conn(0x7f62d006a2e0 msgr2=0x7f62d00c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 --2- 172.21.15.29:0/2620947571 >> v2:172.21.15.170:3300/0 conn(0x7f62d006a2e0 0x7f62d00c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 -- 172.21.15.29:0/2620947571 >> v1:172.21.15.170:6789/0 conn(0x7f62d015d150 legacy=0x7f62d0160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.848+0000 7f62d56c0640 1 -- 172.21.15.29:0/2620947571 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d00ad7a0 con 0x7f62d00ad350 2023-10-16T15:54:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.850+0000 7f62d56c0640 1 --2- 172.21.15.29:0/2620947571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d00ad350 0x7f62d0069da0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f62d00adde0 tx=0x7f62c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450be12a09a8e190 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.850+0000 7f62d4ebf640 1 -- 172.21.15.29:0/2620947571 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0031870 con 0x7f62d00ad350 2023-10-16T15:54:10.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.850+0000 7f62d4ebf640 1 -- 172.21.15.29:0/2620947571 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c00319d0 con 0x7f62d00ad350 2023-10-16T15:54:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.850+0000 7f62d4ebf640 1 -- 172.21.15.29:0/2620947571 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0031d60 con 0x7f62d00ad350 2023-10-16T15:54:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.850+0000 7f62d8125640 1 -- 172.21.15.29:0/2620947571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d00ad350 msgr2=0x7f62d0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.850+0000 7f62d8125640 1 --2- 172.21.15.29:0/2620947571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d00ad350 0x7f62d0069da0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f62d00adde0 tx=0x7f62c002b910 comp rx=0 tx=0).stop 2023-10-16T15:54:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.851+0000 7f62d8125640 1 -- 172.21.15.29:0/2620947571 shutdown_connections 2023-10-16T15:54:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.851+0000 7f62d8125640 1 --2- 172.21.15.29:0/2620947571 >> v2:172.21.15.170:3300/0 conn(0x7f62d006a2e0 0x7f62d00c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.851+0000 7f62d8125640 1 --2- 172.21.15.29:0/2620947571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d00ad350 0x7f62d0069da0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.851+0000 7f62d8125640 1 -- 172.21.15.29:0/2620947571 >> 172.21.15.29:0/2620947571 conn(0x7f62d00ce9b0 msgr2=0x7f62d00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.851+0000 7f62d8125640 1 -- 172.21.15.29:0/2620947571 shutdown_connections 2023-10-16T15:54:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.851+0000 7f62d8125640 1 -- 172.21.15.29:0/2620947571 wait complete. 2023-10-16T15:54:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.852+0000 7f62d8125640 1 Processor -- start 2023-10-16T15:54:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.853+0000 7f62d8125640 1 -- start start 2023-10-16T15:54:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.853+0000 7f62d8125640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 0x7f62d00c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d8125640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d015d150 0x7f62d015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d8125640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01449a0 0x7f62d0144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d8125640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0145390 con 0x7f62d006a2e0 2023-10-16T15:54:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d8125640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01f4600 con 0x7f62d01449a0 2023-10-16T15:54:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d8125640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d00c4080 con 0x7f62d015d150 2023-10-16T15:54:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d5ec1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 0x7f62d00c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d5ec1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 0x7f62d00c4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44464/0 (socket says 172.21.15.29:44464) 2023-10-16T15:54:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d5ec1640 1 -- 172.21.15.29:0/2012965286 learned_addr learned my addr 172.21.15.29:0/2012965286 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d56c0640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d015d150 0x7f62d015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.854+0000 7f62d66c2640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01449a0 0x7f62d0144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d5ec1640 1 -- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d015d150 msgr2=0x7f62d015ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d5ec1640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d015d150 0x7f62d015ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d5ec1640 1 -- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01449a0 msgr2=0x7f62d0144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d5ec1640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01449a0 0x7f62d0144e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d5ec1640 1 -- 172.21.15.29:0/2012965286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00094d0 con 0x7f62d006a2e0 2023-10-16T15:54:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d5ec1640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 0x7f62d00c4e40 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b410 tx=0x7f62cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc011830 con 0x7f62d006a2e0 2023-10-16T15:54:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc011990 con 0x7f62d006a2e0 2023-10-16T15:54:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc011d40 con 0x7f62d006a2e0 2023-10-16T15:54:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d00c4360 con 0x7f62d006a2e0 2023-10-16T15:54:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.855+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d00c4920 con 0x7f62d006a2e0 2023-10-16T15:54:10.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.857+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0066de0 con 0x7f62d006a2e0 2023-10-16T15:54:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.858+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f62cc01a400 con 0x7f62d006a2e0 2023-10-16T15:54:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.859+0000 7f62c67fc640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f62b406acc0 0x7f62b406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.859+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f62cc089b00 con 0x7f62d006a2e0 2023-10-16T15:54:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.859+0000 7f62d56c0640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f62b406acc0 0x7f62b406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.860+0000 7f62d56c0640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f62b406acc0 0x7f62b406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f62c0036000 tx=0x7f62c003c040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:10.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:10.863+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f62cc08be60 con 0x7f62d006a2e0 2023-10-16T15:54:11.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.059+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62d00b4750 con 0x7f62d006a2e0 2023-10-16T15:54:11.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.060+0000 7f62c67fc640 1 -- 172.21.15.29:0/2012965286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v231) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62cc015020 con 0x7f62d006a2e0 2023-10-16T15:54:11.061 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:54:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f62b406acc0 msgr2=0x7f62b406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f62b406acc0 0x7f62b406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f62c0036000 tx=0x7f62c003c040 comp rx=0 tx=0).stop 2023-10-16T15:54:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 msgr2=0x7f62d00c4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 0x7f62d00c4e40 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b410 tx=0x7f62cc00b8d0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 shutdown_connections 2023-10-16T15:54:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f62b406acc0 0x7f62b406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01449a0 0x7f62d0144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62d015d150 0x7f62d015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 --2- 172.21.15.29:0/2012965286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62d006a2e0 0x7f62d00c4e40 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 >> 172.21.15.29:0/2012965286 conn(0x7f62d00ce9b0 msgr2=0x7f62d0070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:11.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 shutdown_connections 2023-10-16T15:54:11.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.063+0000 7f62d8125640 1 -- 172.21.15.29:0/2012965286 wait complete. 2023-10-16T15:54:11.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:54:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.317+0000 7fe3df2e7640 1 Processor -- start 2023-10-16T15:54:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.317+0000 7fe3df2e7640 1 -- start start 2023-10-16T15:54:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3df2e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d815ad70 0x7fe3d8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3df2e7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3d80caac0 0x7fe3d806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3df2e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d80cb030 con 0x7fe3d80c9ca0 2023-10-16T15:54:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3df2e7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d80c91c0 con 0x7fe3d80caac0 2023-10-16T15:54:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3df2e7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d80c94a0 con 0x7fe3d815ad70 2023-10-16T15:54:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3dc882640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe3d80c9ca0 0x7fe3d80ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60168/0 (socket says 172.21.15.29:60168) 2023-10-16T15:54:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3dc882640 1 -- 172.21.15.29:0/777296409 learned_addr learned my addr 172.21.15.29:0/777296409 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3dd884640 1 --2- 172.21.15.29:0/777296409 >> v2:172.21.15.29:3300/0 conn(0x7fe3d80caac0 0x7fe3d806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.319+0000 7fe3dd083640 1 --2- 172.21.15.29:0/777296409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d815ad70 0x7fe3d8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.320+0000 7fe3d7fff640 1 -- 172.21.15.29:0/777296409 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118757317 0 0) 0x7fe3d80cb030 con 0x7fe3d80c9ca0 2023-10-16T15:54:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.320+0000 7fe3d7fff640 1 -- 172.21.15.29:0/777296409 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c0003520 con 0x7fe3d80c9ca0 2023-10-16T15:54:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.320+0000 7fe3dd884640 1 -- 172.21.15.29:0/777296409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d815ad70 msgr2=0x7fe3d8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.320+0000 7fe3dd884640 1 --2- 172.21.15.29:0/777296409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d815ad70 0x7fe3d8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.320+0000 7fe3dd884640 1 -- 172.21.15.29:0/777296409 >> v1:172.21.15.29:6789/0 conn(0x7fe3d80c9ca0 legacy=0x7fe3d80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.320+0000 7fe3dd884640 1 -- 172.21.15.29:0/777296409 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d80c9780 con 0x7fe3d80caac0 2023-10-16T15:54:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.321+0000 7fe3dd884640 1 --2- 172.21.15.29:0/777296409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d80caac0 0x7fe3d806a2e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009d90 tx=0x7fe3d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcd2c70fb100f41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.321+0000 7fe3d7fff640 1 -- 172.21.15.29:0/777296409 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d002cc60 con 0x7fe3d80caac0 2023-10-16T15:54:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.321+0000 7fe3d7fff640 1 -- 172.21.15.29:0/777296409 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d002cdc0 con 0x7fe3d80caac0 2023-10-16T15:54:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.321+0000 7fe3d7fff640 1 -- 172.21.15.29:0/777296409 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0034660 con 0x7fe3d80caac0 2023-10-16T15:54:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 -- 172.21.15.29:0/777296409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d80caac0 msgr2=0x7fe3d806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/777296409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d80caac0 0x7fe3d806a2e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009d90 tx=0x7fe3d002beb0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 -- 172.21.15.29:0/777296409 shutdown_connections 2023-10-16T15:54:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/777296409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d80caac0 0x7fe3d806a2e0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/777296409 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d815ad70 0x7fe3d8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 -- 172.21.15.29:0/777296409 >> 172.21.15.29:0/777296409 conn(0x7fe3d80ce9b0 msgr2=0x7fe3d80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 -- 172.21.15.29:0/777296409 shutdown_connections 2023-10-16T15:54:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.322+0000 7fe3df2e7640 1 -- 172.21.15.29:0/777296409 wait complete. 2023-10-16T15:54:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.324+0000 7fe3df2e7640 1 Processor -- start 2023-10-16T15:54:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.324+0000 7fe3df2e7640 1 -- start start 2023-10-16T15:54:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3df2e7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3d80c9ca0 0x7fe3d8152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3df2e7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d80caac0 0x7fe3d815c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3df2e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 0x7fe3d80ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3df2e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d815c9e0 con 0x7fe3d815ad70 2023-10-16T15:54:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3df2e7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d815cbd0 con 0x7fe3d80c9ca0 2023-10-16T15:54:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3df2e7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d815ceb0 con 0x7fe3d80caac0 2023-10-16T15:54:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.325+0000 7fe3dd884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 0x7fe3d80ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 0x7fe3d80ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44492/0 (socket says 172.21.15.29:44492) 2023-10-16T15:54:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 -- 172.21.15.29:0/1597639638 learned_addr learned my addr 172.21.15.29:0/1597639638 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dc882640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d80caac0 0x7fe3d815c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd083640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3d80c9ca0 0x7fe3d8152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 -- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d80caac0 msgr2=0x7fe3d815c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d80caac0 0x7fe3d815c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 -- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3d80c9ca0 msgr2=0x7fe3d8152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3d80c9ca0 0x7fe3d8152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 -- 172.21.15.29:0/1597639638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0009a70 con 0x7fe3d815ad70 2023-10-16T15:54:11.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd083640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3d80c9ca0 0x7fe3d8152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:11.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.326+0000 7fe3dd884640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 0x7fe3d80ccc30 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009d90 tx=0x7fe3d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:11.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.327+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0040650 con 0x7fe3d815ad70 2023-10-16T15:54:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.327+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d00407b0 con 0x7fe3d815ad70 2023-10-16T15:54:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.327+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0040b60 con 0x7fe3d815ad70 2023-10-16T15:54:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.327+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d815d190 con 0x7fe3d815ad70 2023-10-16T15:54:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.327+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d80b0480 con 0x7fe3d815ad70 2023-10-16T15:54:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.328+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d806cdd0 con 0x7fe3d815ad70 2023-10-16T15:54:11.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.329+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe3d0040cc0 con 0x7fe3d815ad70 2023-10-16T15:54:11.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.330+0000 7fe3d5ffb640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe3c406acc0 0x7fe3c406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.330+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7fe3d00ac5c0 con 0x7fe3d815ad70 2023-10-16T15:54:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.331+0000 7fe3dd083640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe3c406acc0 0x7fe3c406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.332+0000 7fe3dd083640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe3c406acc0 0x7fe3c406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009a20 tx=0x7fe3c8008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.334+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3d0077300 con 0x7fe3d815ad70 2023-10-16T15:54:11.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.461+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 137) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fe3d00776d0 con 0x7fe3d815ad70 2023-10-16T15:54:11.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.535+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3d806aa70 con 0x7fe3d815ad70 2023-10-16T15:54:11.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.536+0000 7fe3d5ffb640 1 -- 172.21.15.29:0/1597639638 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v231) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe3d008eee0 con 0x7fe3d815ad70 2023-10-16T15:54:11.537 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:54:11.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe3c406acc0 msgr2=0x7fe3c406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe3c406acc0 0x7fe3c406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009a20 tx=0x7fe3c8008040 comp rx=0 tx=0).stop 2023-10-16T15:54:11.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 msgr2=0x7fe3d80ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 0x7fe3d80ccc30 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009d90 tx=0x7fe3d0004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:11.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 shutdown_connections 2023-10-16T15:54:11.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe3c406acc0 0x7fe3c406d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3d815ad70 0x7fe3d80ccc30 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe3d80caac0 0x7fe3d815c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 --2- 172.21.15.29:0/1597639638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe3d80c9ca0 0x7fe3d8152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 >> 172.21.15.29:0/1597639638 conn(0x7fe3d80ce9b0 msgr2=0x7fe3d80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:11.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 shutdown_connections 2023-10-16T15:54:11.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.539+0000 7fe3df2e7640 1 -- 172.21.15.29:0/1597639638 wait complete. 2023-10-16T15:54:11.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T15:54:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.786+0000 7f5e62487640 1 Processor -- start 2023-10-16T15:54:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.786+0000 7f5e62487640 1 -- start start 2023-10-16T15:54:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e62487640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5e5c159ab0 0x7f5e5c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e62487640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e5c06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e62487640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0cd1f0 con 0x7f5e5c06c140 2023-10-16T15:54:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e62487640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c15d890 con 0x7f5e5c159ab0 2023-10-16T15:54:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e62487640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cd4d0 con 0x7f5e5c159e90 2023-10-16T15:54:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e60a24640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5e5c06c140 0x7f5e5c15dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60180/0 (socket says 172.21.15.29:60180) 2023-10-16T15:54:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e60a24640 1 -- 172.21.15.29:0/1453265101 learned_addr learned my addr 172.21.15.29:0/1453265101 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:11.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/1453265101 >> v2:172.21.15.170:3300/0 conn(0x7f5e5c159ab0 0x7f5e5c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e5b7fe640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e5c06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.788+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722455409 0 0) 0x7f5e5c0cd1f0 con 0x7f5e5c06c140 2023-10-16T15:54:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e44003520 con 0x7f5e5c06c140 2023-10-16T15:54:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106616125 0 0) 0x7f5e44003520 con 0x7f5e5c06c140 2023-10-16T15:54:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 msgr2=0x7f5e5c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e5c06fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 >> v2:172.21.15.170:3300/0 conn(0x7f5e5c159ab0 msgr2=0x7f5e5c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:11.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 --2- 172.21.15.29:0/1453265101 >> v2:172.21.15.170:3300/0 conn(0x7f5e5c159ab0 0x7f5e5c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0cd7b0 con 0x7f5e5c06c140 2023-10-16T15:54:11.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5e540019f0 con 0x7f5e5c06c140 2023-10-16T15:54:11.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 >> v1:172.21.15.29:6789/0 conn(0x7f5e5c06c140 legacy=0x7f5e5c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e44003970 0x7f5e44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44004d00 0x7f5e44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c15d890 con 0x7f5e44003970 2023-10-16T15:54:11.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cd4d0 con 0x7f5e44004d00 2023-10-16T15:54:11.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4400ba00 con 0x7f5e5c159e90 2023-10-16T15:54:11.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.789+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e44003970 0x7f5e44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5b7fe640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e60a24640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44004d00 0x7f5e44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5bfff640 1 -- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 msgr2=0x7f5e440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:11.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5bfff640 1 -- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44004d00 msgr2=0x7f5e44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44004d00 0x7f5e44005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5bfff640 1 -- 172.21.15.29:0/1453265101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44003520 con 0x7f5e44003970 2023-10-16T15:54:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e5b7fe640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c159e90 0x7f5e440047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:11.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.790+0000 7f5e60a24640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44004d00 0x7f5e44005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.791+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e44003970 0x7f5e44003d90 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009ac0 tx=0x7f5e440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d625524700b8cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.791+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02c710 con 0x7f5e44003970 2023-10-16T15:54:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.791+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c02c870 con 0x7f5e44003970 2023-10-16T15:54:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.791+0000 7f5e5affd640 1 -- 172.21.15.29:0/1453265101 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02cc00 con 0x7f5e44003970 2023-10-16T15:54:11.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e62487640 1 -- 172.21.15.29:0/1453265101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e44003970 msgr2=0x7f5e44003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e62487640 1 --2- 172.21.15.29:0/1453265101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e44003970 0x7f5e44003d90 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009ac0 tx=0x7f5e440057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e60a24640 1 -- 172.21.15.29:0/1453265101 reap_dead start 2023-10-16T15:54:11.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e62487640 1 -- 172.21.15.29:0/1453265101 shutdown_connections 2023-10-16T15:54:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e62487640 1 -- 172.21.15.29:0/1453265101 >> 172.21.15.29:0/1453265101 conn(0x7f5e5c0ce9b0 msgr2=0x7f5e5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e62487640 1 -- 172.21.15.29:0/1453265101 shutdown_connections 2023-10-16T15:54:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.792+0000 7f5e62487640 1 -- 172.21.15.29:0/1453265101 wait complete. 2023-10-16T15:54:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.794+0000 7f5e62487640 1 Processor -- start 2023-10-16T15:54:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.794+0000 7f5e62487640 1 -- start start 2023-10-16T15:54:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e62487640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44003970 0x7f5e5c15baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e62487640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c06ba40 0x7f5e5c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e62487640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 0x7f5e5c0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e62487640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06a640 con 0x7f5e5c0c4920 2023-10-16T15:54:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e62487640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0c52e0 con 0x7f5e44003970 2023-10-16T15:54:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e62487640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06bec0 con 0x7f5e5c06ba40 2023-10-16T15:54:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e60a24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 0x7f5e5c0c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e60a24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 0x7f5e5c0c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44506/0 (socket says 172.21.15.29:44506) 2023-10-16T15:54:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.795+0000 7f5e60a24640 1 -- 172.21.15.29:0/2261371836 learned_addr learned my addr 172.21.15.29:0/2261371836 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e5b7fe640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c06ba40 0x7f5e5c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44003970 0x7f5e5c15baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e60a24640 1 -- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c06ba40 msgr2=0x7f5e5c151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e60a24640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c06ba40 0x7f5e5c151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e60a24640 1 -- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44003970 msgr2=0x7f5e5c15baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e60a24640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44003970 0x7f5e5c15baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e60a24640 1 -- 172.21.15.29:0/2261371836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c15c0f0 con 0x7f5e5c0c4920 2023-10-16T15:54:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44003970 0x7f5e5c15baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e5b7fe640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c06ba40 0x7f5e5c151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e60a24640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 0x7f5e5c0c4da0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0c24b0 tx=0x7f5e5400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54004d60 con 0x7f5e5c0c4920 2023-10-16T15:54:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5400bb20 con 0x7f5e5c0c4920 2023-10-16T15:54:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5400bed0 con 0x7f5e5c0c4920 2023-10-16T15:54:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.796+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c15c3d0 con 0x7f5e5c0c4920 2023-10-16T15:54:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.797+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c15c990 con 0x7f5e5c0c4920 2023-10-16T15:54:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.798+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c04e160 con 0x7f5e5c0c4920 2023-10-16T15:54:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.799+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f5e54007070 con 0x7f5e5c0c4920 2023-10-16T15:54:11.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.799+0000 7f5e58ff9640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e4806b010 0x7f5e4806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:11.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.800+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f5e5408a4a0 con 0x7f5e5c0c4920 2023-10-16T15:54:11.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.800+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e4806b010 0x7f5e4806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:11.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.801+0000 7f5e5bfff640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e4806b010 0x7f5e4806d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c036000 tx=0x7f5e4c030000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:11.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.804+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e54054a70 con 0x7f5e5c0c4920 2023-10-16T15:54:11.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.998+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e5c0cefd0 con 0x7f5e5c0c4920 2023-10-16T15:54:12.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:11.999+0000 7f5e58ff9640 1 -- 172.21.15.29:0/2261371836 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v231) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5e54058090 con 0x7f5e5c0c4920 2023-10-16T15:54:12.000 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:54:12.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.001+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e4806b010 msgr2=0x7f5e4806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.001+0000 7f5e62487640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e4806b010 0x7f5e4806d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c036000 tx=0x7f5e4c030000 comp rx=0 tx=0).stop 2023-10-16T15:54:12.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.001+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 msgr2=0x7f5e5c0c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.001+0000 7f5e62487640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 0x7f5e5c0c4da0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0c24b0 tx=0x7f5e5400b4a0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 shutdown_connections 2023-10-16T15:54:12.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e4806b010 0x7f5e4806d4d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e5c0c4920 0x7f5e5c0c4da0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e5c06ba40 0x7f5e5c151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 --2- 172.21.15.29:0/2261371836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e44003970 0x7f5e5c15baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 >> 172.21.15.29:0/2261371836 conn(0x7f5e5c0ce9b0 msgr2=0x7f5e5c0c1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 shutdown_connections 2023-10-16T15:54:12.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.002+0000 7f5e62487640 1 -- 172.21.15.29:0/2261371836 wait complete. 2023-10-16T15:54:12.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T15:54:12.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.262+0000 7f7e18516640 1 Processor -- start 2023-10-16T15:54:12.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.262+0000 7f7e18516640 1 -- start start 2023-10-16T15:54:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.264+0000 7f7e18516640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e100ad350 0x7f7e10069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.264+0000 7f7e18516640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1006a730 con 0x7f7e1006a2e0 2023-10-16T15:54:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.264+0000 7f7e18516640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100ac890 con 0x7f7e1015d150 2023-10-16T15:54:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.264+0000 7f7e18516640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1015d530 con 0x7f7e100ad350 2023-10-16T15:54:12.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e15ab1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e100ad350 0x7f7e10069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e15ab1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e100ad350 0x7f7e10069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44514/0 (socket says 172.21.15.29:44514) 2023-10-16T15:54:12.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e15ab1640 1 -- 172.21.15.29:0/2861570157 learned_addr learned my addr 172.21.15.29:0/2861570157 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:12.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e15ab1640 1 -- 172.21.15.29:0/2861570157 >> v1:172.21.15.170:6789/0 conn(0x7f7e1015d150 legacy=0x7f7e10160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e15ab1640 1 -- 172.21.15.29:0/2861570157 >> v1:172.21.15.29:6789/0 conn(0x7f7e1006a2e0 legacy=0x7f7e100c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e15ab1640 1 -- 172.21.15.29:0/2861570157 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100ad7a0 con 0x7f7e100ad350 2023-10-16T15:54:12.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.265+0000 7f7e152b0640 1 -- 172.21.15.29:0/2861570157 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369199693 0 0) 0x7f7e08001490 con 0x7f7e1006a2e0 2023-10-16T15:54:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.266+0000 7f7e15ab1640 1 --2- 172.21.15.29:0/2861570157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e10069da0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e0002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b074358c791c92b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.267+0000 7f7e152b0640 1 -- 172.21.15.29:0/2861570157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002f840 con 0x7f7e100ad350 2023-10-16T15:54:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.267+0000 7f7e152b0640 1 -- 172.21.15.29:0/2861570157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0002f9a0 con 0x7f7e100ad350 2023-10-16T15:54:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.267+0000 7f7e152b0640 1 -- 172.21.15.29:0/2861570157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002fd30 con 0x7f7e100ad350 2023-10-16T15:54:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.267+0000 7f7e18516640 1 -- 172.21.15.29:0/2861570157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 msgr2=0x7f7e10069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.267+0000 7f7e18516640 1 --2- 172.21.15.29:0/2861570157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e10069da0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e0002ba20 comp rx=0 tx=0).stop 2023-10-16T15:54:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.268+0000 7f7e18516640 1 -- 172.21.15.29:0/2861570157 shutdown_connections 2023-10-16T15:54:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.268+0000 7f7e18516640 1 --2- 172.21.15.29:0/2861570157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e10069da0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.268+0000 7f7e18516640 1 -- 172.21.15.29:0/2861570157 >> 172.21.15.29:0/2861570157 conn(0x7f7e100ce9b0 msgr2=0x7f7e100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.268+0000 7f7e18516640 1 -- 172.21.15.29:0/2861570157 shutdown_connections 2023-10-16T15:54:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.268+0000 7f7e18516640 1 -- 172.21.15.29:0/2861570157 wait complete. 2023-10-16T15:54:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.270+0000 7f7e18516640 1 Processor -- start 2023-10-16T15:54:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.270+0000 7f7e18516640 1 -- start start 2023-10-16T15:54:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e18516640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e1006a2e0 0x7f7e1015e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e18516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e100c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e18516640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e1015d150 0x7f7e1015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e18516640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1015ec60 con 0x7f7e100ad350 2023-10-16T15:54:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e18516640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1015c5a0 con 0x7f7e1006a2e0 2023-10-16T15:54:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e18516640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1015ee70 con 0x7f7e1015d150 2023-10-16T15:54:12.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e15ab1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e100c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e15ab1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e100c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44518/0 (socket says 172.21.15.29:44518) 2023-10-16T15:54:12.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e15ab1640 1 -- 172.21.15.29:0/4048441134 learned_addr learned my addr 172.21.15.29:0/4048441134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:12.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e15ab1640 1 -- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e1015d150 msgr2=0x7f7e1015b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e15ab1640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e1015d150 0x7f7e1015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e15ab1640 1 -- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e1006a2e0 msgr2=0x7f7e1015e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.271+0000 7f7e162b2640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e1006a2e0 0x7f7e1015e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e15ab1640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e1006a2e0 0x7f7e1015e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e15ab1640 1 -- 172.21.15.29:0/4048441134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000094d0 con 0x7f7e100ad350 2023-10-16T15:54:12.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e162b2640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e1006a2e0 0x7f7e1015e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:12.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e15ab1640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e100c24e0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002f7f0 con 0x7f7e100ad350 2023-10-16T15:54:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e00041400 con 0x7f7e100ad350 2023-10-16T15:54:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e000417b0 con 0x7f7e100ad350 2023-10-16T15:54:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e100c2a20 con 0x7f7e100ad350 2023-10-16T15:54:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.272+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e100b2d40 con 0x7f7e100ad350 2023-10-16T15:54:12.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.274+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1004e160 con 0x7f7e100ad350 2023-10-16T15:54:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.274+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f7e00041910 con 0x7f7e100ad350 2023-10-16T15:54:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.276+0000 7f7e06ffd640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7df406b020 0x7f7df406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.276+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f7e0003c070 con 0x7f7e100ad350 2023-10-16T15:54:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.277+0000 7f7e162b2640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7df406b020 0x7f7df406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.278+0000 7f7e162b2640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7df406b020 0x7f7df406d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e100cb370 tx=0x7f7e0c009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:12.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.281+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e000776f0 con 0x7f7e100ad350 2023-10-16T15:54:12.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.474+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e100ace20 con 0x7f7e100ad350 2023-10-16T15:54:12.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.475+0000 7f7e06ffd640 1 -- 172.21.15.29:0/4048441134 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v231) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7e0007ad10 con 0x7f7e100ad350 2023-10-16T15:54:12.476 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:54:12.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7df406b020 msgr2=0x7f7df406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7df406b020 0x7f7df406d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e100cb370 tx=0x7f7e0c009040 comp rx=0 tx=0).stop 2023-10-16T15:54:12.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 msgr2=0x7f7e100c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e100c24e0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009870 tx=0x7f7e00004730 comp rx=0 tx=0).stop 2023-10-16T15:54:12.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 shutdown_connections 2023-10-16T15:54:12.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7df406b020 0x7f7df406d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e100ad350 0x7f7e100c24e0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e1015d150 0x7f7e1015b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 --2- 172.21.15.29:0/4048441134 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e1006a2e0 0x7f7e1015e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 >> 172.21.15.29:0/4048441134 conn(0x7f7e100ce9b0 msgr2=0x7f7e1006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:12.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 shutdown_connections 2023-10-16T15:54:12.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.478+0000 7f7e18516640 1 -- 172.21.15.29:0/4048441134 wait complete. 2023-10-16T15:54:12.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.707+0000 7ff61f59e640 1 Processor -- start 2023-10-16T15:54:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.708+0000 7ff61f59e640 1 -- start start 2023-10-16T15:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.709+0000 7ff61f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff62015cbe0 0x7ff6200d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.709+0000 7ff61f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff62015cfc0 0x7ff6200b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.709+0000 7ff61f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200d1090 0x7ff6200b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.709+0000 7ff61f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200d15a0 con 0x7ff62015cfc0 2023-10-16T15:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.709+0000 7ff61f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200d1880 con 0x7ff62015cbe0 2023-10-16T15:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.709+0000 7ff61f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200d1b60 con 0x7ff6200d1090 2023-10-16T15:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff62015cfc0 0x7ff6200b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff62015cfc0 0x7ff6200b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44526/0 (socket says 172.21.15.29:44526) 2023-10-16T15:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 -- 172.21.15.29:0/2171406908 learned_addr learned my addr 172.21.15.29:0/2171406908 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff61e59c640 1 --2- 172.21.15.29:0/2171406908 >> v2:172.21.15.170:3300/0 conn(0x7ff62015cbe0 0x7ff6200d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff61ed9d640 1 --2- 172.21.15.29:0/2171406908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200d1090 0x7ff6200b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 -- 172.21.15.29:0/2171406908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200d1090 msgr2=0x7ff6200b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 --2- 172.21.15.29:0/2171406908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200d1090 0x7ff6200b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 -- 172.21.15.29:0/2171406908 >> v2:172.21.15.170:3300/0 conn(0x7ff62015cbe0 msgr2=0x7ff6200d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 --2- 172.21.15.29:0/2171406908 >> v2:172.21.15.170:3300/0 conn(0x7ff62015cbe0 0x7ff6200d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.710+0000 7ff617fff640 1 -- 172.21.15.29:0/2171406908 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200d1e40 con 0x7ff62015cfc0 2023-10-16T15:54:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.711+0000 7ff617fff640 1 --2- 172.21.15.29:0/2171406908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015cfc0 0x7ff6200b4570 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7ff6200b53f0 tx=0x7ff60802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdafdcd652f40f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.712+0000 7ff61dd9b640 1 -- 172.21.15.29:0/2171406908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608030860 con 0x7ff62015cfc0 2023-10-16T15:54:12.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.712+0000 7ff61dd9b640 1 -- 172.21.15.29:0/2171406908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6080309c0 con 0x7ff62015cfc0 2023-10-16T15:54:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.712+0000 7ff61dd9b640 1 -- 172.21.15.29:0/2171406908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608030d50 con 0x7ff62015cfc0 2023-10-16T15:54:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.712+0000 7ff61f59e640 1 -- 172.21.15.29:0/2171406908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015cfc0 msgr2=0x7ff6200b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.712+0000 7ff61f59e640 1 --2- 172.21.15.29:0/2171406908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015cfc0 0x7ff6200b4570 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7ff6200b53f0 tx=0x7ff60802b9a0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 -- 172.21.15.29:0/2171406908 shutdown_connections 2023-10-16T15:54:12.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 --2- 172.21.15.29:0/2171406908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200d1090 0x7ff6200b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 --2- 172.21.15.29:0/2171406908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015cfc0 0x7ff6200b4570 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 --2- 172.21.15.29:0/2171406908 >> v2:172.21.15.170:3300/0 conn(0x7ff62015cbe0 0x7ff6200d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 -- 172.21.15.29:0/2171406908 >> 172.21.15.29:0/2171406908 conn(0x7ff6200cf570 msgr2=0x7ff6200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 -- 172.21.15.29:0/2171406908 shutdown_connections 2023-10-16T15:54:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.713+0000 7ff61f59e640 1 -- 172.21.15.29:0/2171406908 wait complete. 2023-10-16T15:54:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.715+0000 7ff61f59e640 1 Processor -- start 2023-10-16T15:54:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.715+0000 7ff61f59e640 1 -- start start 2023-10-16T15:54:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.716+0000 7ff61f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 0x7ff6200cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.716+0000 7ff61f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62015cbe0 0x7ff6201f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.716+0000 7ff61f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015cfc0 0x7ff6200c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.716+0000 7ff61f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6201f68a0 con 0x7ff6200d1090 2023-10-16T15:54:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.716+0000 7ff61f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200cc8c0 con 0x7ff62015cbe0 2023-10-16T15:54:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.716+0000 7ff61f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200c83f0 con 0x7ff62015cfc0 2023-10-16T15:54:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 0x7ff6200cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 0x7ff6200cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44530/0 (socket says 172.21.15.29:44530) 2023-10-16T15:54:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 -- 172.21.15.29:0/3773958486 learned_addr learned my addr 172.21.15.29:0/3773958486 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 -- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015cfc0 msgr2=0x7ff6200c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015cfc0 0x7ff6200c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 -- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62015cbe0 msgr2=0x7ff6201f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62015cbe0 0x7ff6201f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 -- 172.21.15.29:0/3773958486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080094d0 con 0x7ff6200d1090 2023-10-16T15:54:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.717+0000 7ff61e59c640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 0x7ff6200cc1c0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7ff61000d800 tx=0x7ff61000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.718+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61000b6c0 con 0x7ff6200d1090 2023-10-16T15:54:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.718+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61000b820 con 0x7ff6200d1090 2023-10-16T15:54:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.718+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61000bb90 con 0x7ff6200d1090 2023-10-16T15:54:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.718+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6200c86d0 con 0x7ff6200d1090 2023-10-16T15:54:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.718+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6200c9410 con 0x7ff6200d1090 2023-10-16T15:54:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.719+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62004e9b0 con 0x7ff6200d1090 2023-10-16T15:54:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.720+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89377+0+0 (secure 0 0 0) 0x7ff61000bcf0 con 0x7ff6200d1090 2023-10-16T15:54:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.721+0000 7ff616ffd640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff60006b050 0x7ff60006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.721+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7ff61008a560 con 0x7ff6200d1090 2023-10-16T15:54:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.721+0000 7ff617fff640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff60006b050 0x7ff60006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:12.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.723+0000 7ff617fff640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff60006b050 0x7ff60006d510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff608036000 tx=0x7ff60803b040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.725+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff610054c40 con 0x7ff6200d1090 2023-10-16T15:54:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.950+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff620159970 con 0x7ff6200d1090 2023-10-16T15:54:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.952+0000 7ff616ffd640 1 -- 172.21.15.29:0/3773958486 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+6155 (secure 0 0 0) 0x7ff610058260 con 0x7ff6200d1090 2023-10-16T15:54:12.953 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:12.953 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 114 2023-10-16T15:54:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff60006b050 msgr2=0x7ff60006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff60006b050 0x7ff60006d510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff608036000 tx=0x7ff60803b040 comp rx=0 tx=0).stop 2023-10-16T15:54:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 msgr2=0x7ff6200cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 0x7ff6200cc1c0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7ff61000d800 tx=0x7ff61000dcc0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 shutdown_connections 2023-10-16T15:54:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff60006b050 0x7ff60006d510 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015cfc0 0x7ff6200c8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62015cbe0 0x7ff6201f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 --2- 172.21.15.29:0/3773958486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6200d1090 0x7ff6200cc1c0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 >> 172.21.15.29:0/3773958486 conn(0x7ff6200cf570 msgr2=0x7ff6200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 shutdown_connections 2023-10-16T15:54:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:12.954+0000 7ff61f59e640 1 -- 172.21.15.29:0/3773958486 wait complete. 2023-10-16T15:54:12.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.187+0000 7fe02359e640 1 Processor -- start 2023-10-16T15:54:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.188+0000 7fe02359e640 1 -- start start 2023-10-16T15:54:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.189+0000 7fe02359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe02415cbe0 0x7fe0240d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.189+0000 7fe02359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe02415cfc0 0x7fe0240b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.189+0000 7fe02359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0240d1090 0x7fe0240b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.189+0000 7fe02359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240d15a0 con 0x7fe02415cbe0 2023-10-16T15:54:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.189+0000 7fe02359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240d1880 con 0x7fe02415cfc0 2023-10-16T15:54:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.189+0000 7fe02359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240d1b60 con 0x7fe0240d1090 2023-10-16T15:54:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe02415cbe0 0x7fe0240d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe02415cbe0 0x7fe0240d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44544/0 (socket says 172.21.15.29:44544) 2023-10-16T15:54:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 -- 172.21.15.29:0/2307589664 learned_addr learned my addr 172.21.15.29:0/2307589664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe022d9d640 1 --2- 172.21.15.29:0/2307589664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0240d1090 0x7fe0240b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe021d9b640 1 --2- 172.21.15.29:0/2307589664 >> v2:172.21.15.170:3300/0 conn(0x7fe02415cfc0 0x7fe0240b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 -- 172.21.15.29:0/2307589664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0240d1090 msgr2=0x7fe0240b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 --2- 172.21.15.29:0/2307589664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0240d1090 0x7fe0240b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 -- 172.21.15.29:0/2307589664 >> v2:172.21.15.170:3300/0 conn(0x7fe02415cfc0 msgr2=0x7fe0240b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 --2- 172.21.15.29:0/2307589664 >> v2:172.21.15.170:3300/0 conn(0x7fe02415cfc0 0x7fe0240b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.190+0000 7fe02259c640 1 -- 172.21.15.29:0/2307589664 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240d1e40 con 0x7fe02415cbe0 2023-10-16T15:54:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.191+0000 7fe02259c640 1 --2- 172.21.15.29:0/2307589664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0240d0b50 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7fe0240cf4b0 tx=0x7fe01402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638c3e920a907a8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.192+0000 7fe02159a640 1 -- 172.21.15.29:0/2307589664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01402cb00 con 0x7fe02415cbe0 2023-10-16T15:54:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.192+0000 7fe02159a640 1 -- 172.21.15.29:0/2307589664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01402cc60 con 0x7fe02415cbe0 2023-10-16T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.192+0000 7fe02159a640 1 -- 172.21.15.29:0/2307589664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0140343f0 con 0x7fe02415cbe0 2023-10-16T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.192+0000 7fe02359e640 1 -- 172.21.15.29:0/2307589664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 msgr2=0x7fe0240d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.192+0000 7fe02359e640 1 --2- 172.21.15.29:0/2307589664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0240d0b50 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7fe0240cf4b0 tx=0x7fe01402bf60 comp rx=0 tx=0).stop 2023-10-16T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 -- 172.21.15.29:0/2307589664 shutdown_connections 2023-10-16T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 --2- 172.21.15.29:0/2307589664 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe0240d1090 0x7fe0240b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 --2- 172.21.15.29:0/2307589664 >> v2:172.21.15.170:3300/0 conn(0x7fe02415cfc0 0x7fe0240b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 --2- 172.21.15.29:0/2307589664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0240d0b50 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 -- 172.21.15.29:0/2307589664 >> 172.21.15.29:0/2307589664 conn(0x7fe0240cf570 msgr2=0x7fe0240cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 -- 172.21.15.29:0/2307589664 shutdown_connections 2023-10-16T15:54:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.193+0000 7fe02359e640 1 -- 172.21.15.29:0/2307589664 wait complete. 2023-10-16T15:54:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.195+0000 7fe02359e640 1 Processor -- start 2023-10-16T15:54:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.195+0000 7fe02359e640 1 -- start start 2023-10-16T15:54:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.196+0000 7fe02359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0240d1090 0x7fe0241f1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.196+0000 7fe02359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0241f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.196+0000 7fe02359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe02415cfc0 0x7fe0240c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe02359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241f61c0 con 0x7fe02415cbe0 2023-10-16T15:54:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe02359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241f64a0 con 0x7fe0240d1090 2023-10-16T15:54:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe02359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241f6780 con 0x7fe02415cfc0 2023-10-16T15:54:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0241f58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0241f58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44556/0 (socket says 172.21.15.29:44556) 2023-10-16T15:54:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 -- 172.21.15.29:0/2520184548 learned_addr learned my addr 172.21.15.29:0/2520184548 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe02259c640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0240d1090 0x7fe0241f1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 -- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe02415cfc0 msgr2=0x7fe0240c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe02415cfc0 0x7fe0240c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 -- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0240d1090 msgr2=0x7fe0241f1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0240d1090 0x7fe0241f1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe021d9b640 1 -- 172.21.15.29:0/2520184548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014009b30 con 0x7fe02415cbe0 2023-10-16T15:54:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.197+0000 7fe02259c640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0240d1090 0x7fe0241f1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.198+0000 7fe021d9b640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0241f58c0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00d5c0 tx=0x7fe00c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.198+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c0125f0 con 0x7fe02415cbe0 2023-10-16T15:54:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.198+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c012750 con 0x7fe02415cbe0 2023-10-16T15:54:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.198+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c012b00 con 0x7fe02415cbe0 2023-10-16T15:54:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.198+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0240c7090 con 0x7fe02415cbe0 2023-10-16T15:54:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.198+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0241f20f0 con 0x7fe02415cbe0 2023-10-16T15:54:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.199+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0240c6ee0 con 0x7fe02415cbe0 2023-10-16T15:54:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.200+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fe00c012c60 con 0x7fe02415cbe0 2023-10-16T15:54:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.201+0000 7fe01affd640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe00406b020 0x7fe00406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.201+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7fe00c08a4e0 con 0x7fe02415cbe0 2023-10-16T15:54:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.202+0000 7fe02259c640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe00406b020 0x7fe00406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.203+0000 7fe02259c640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe00406b020 0x7fe00406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe014035000 tx=0x7fe014030000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.205+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe00c054bc0 con 0x7fe02415cbe0 2023-10-16T15:54:13.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.391+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe02404e9b0 con 0x7fe02415cbe0 2023-10-16T15:54:13.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.396+0000 7fe01affd640 1 -- 172.21.15.29:0/2520184548 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+21557 (secure 0 0 0) 0x7fe00c0581e0 con 0x7fe02415cbe0 2023-10-16T15:54:13.398 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:13.398 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":231,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:09.426111+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe00406b020 msgr2=0x7fe00406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe00406b020 0x7fe00406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe014035000 tx=0x7fe014030000 comp rx=0 tx=0).stop 2023-10-16T15:54:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 msgr2=0x7fe0241f58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0241f58c0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00d5c0 tx=0x7fe00c00da80 comp rx=0 tx=0).stop 2023-10-16T15:54:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 shutdown_connections 2023-10-16T15:54:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe00406b020 0x7fe00406d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe02415cbe0 0x7fe0241f58c0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe02415cfc0 0x7fe0240c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 --2- 172.21.15.29:0/2520184548 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0240d1090 0x7fe0241f1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 >> 172.21.15.29:0/2520184548 conn(0x7fe0240cf570 msgr2=0x7fe0240b7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 shutdown_connections 2023-10-16T15:54:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.400+0000 7fe02359e640 1 -- 172.21.15.29:0/2520184548 wait complete. 2023-10-16T15:54:13.420 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:54:13.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:13.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.635+0000 7f0fcd993640 1 Processor -- start 2023-10-16T15:54:13.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.635+0000 7f0fcd993640 1 -- start start 2023-10-16T15:54:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcd993640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0fc8070b60 0x7f0fc815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcd993640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc806c100 0x7f0fc806c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcd993640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc815aef0 con 0x7f0fc8159950 2023-10-16T15:54:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcd993640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc815e490 con 0x7f0fc8070b60 2023-10-16T15:54:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcd993640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc815d910 con 0x7f0fc806c100 2023-10-16T15:54:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcc991640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0fc8159950 0x7f0fc8070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60196/0 (socket says 172.21.15.29:60196) 2023-10-16T15:54:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fcc991640 1 -- 172.21.15.29:0/2589131247 learned_addr learned my addr 172.21.15.29:0/2589131247 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:13.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.637+0000 7f0fc7fff640 1 --2- 172.21.15.29:0/2589131247 >> v2:172.21.15.29:3300/0 conn(0x7f0fc8070b60 0x7f0fc815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.638+0000 7f0fcd192640 1 --2- 172.21.15.29:0/2589131247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc806c100 0x7f0fc806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.638+0000 7f0fc7fff640 1 -- 172.21.15.29:0/2589131247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc806c100 msgr2=0x7f0fc806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:13.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.638+0000 7f0fc7fff640 1 --2- 172.21.15.29:0/2589131247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc806c100 0x7f0fc806c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.638+0000 7f0fc7fff640 1 -- 172.21.15.29:0/2589131247 >> v1:172.21.15.29:6789/0 conn(0x7f0fc8159950 legacy=0x7f0fc8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.638+0000 7f0fc7fff640 1 -- 172.21.15.29:0/2589131247 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc815dbf0 con 0x7f0fc8070b60 2023-10-16T15:54:13.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.639+0000 7f0fc7fff640 1 --2- 172.21.15.29:0/2589131247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc8070b60 0x7f0fc815a9b0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40097a0 tx=0x7f0fb402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20713defd36e1a4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:13.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.639+0000 7f0fc77fe640 1 -- 172.21.15.29:0/2589131247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4031780 con 0x7f0fc8070b60 2023-10-16T15:54:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.639+0000 7f0fc77fe640 1 -- 172.21.15.29:0/2589131247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb40318e0 con 0x7f0fc8070b60 2023-10-16T15:54:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.639+0000 7f0fc77fe640 1 -- 172.21.15.29:0/2589131247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4031c70 con 0x7f0fc8070b60 2023-10-16T15:54:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 -- 172.21.15.29:0/2589131247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc8070b60 msgr2=0x7f0fc815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 --2- 172.21.15.29:0/2589131247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc8070b60 0x7f0fc815a9b0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40097a0 tx=0x7f0fb402b820 comp rx=0 tx=0).stop 2023-10-16T15:54:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 -- 172.21.15.29:0/2589131247 shutdown_connections 2023-10-16T15:54:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 --2- 172.21.15.29:0/2589131247 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc806c100 0x7f0fc806c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 --2- 172.21.15.29:0/2589131247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc8070b60 0x7f0fc815a9b0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 -- 172.21.15.29:0/2589131247 >> 172.21.15.29:0/2589131247 conn(0x7f0fc80ce6a0 msgr2=0x7f0fc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 -- 172.21.15.29:0/2589131247 shutdown_connections 2023-10-16T15:54:13.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.640+0000 7f0fcd993640 1 -- 172.21.15.29:0/2589131247 wait complete. 2023-10-16T15:54:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.642+0000 7f0fcd993640 1 Processor -- start 2023-10-16T15:54:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.642+0000 7f0fcd993640 1 -- start start 2023-10-16T15:54:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcd993640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 0x7f0fc81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcd993640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fc8070b60 0x7f0fc815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcd993640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc8159950 0x7f0fc80c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcd993640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81f21e0 con 0x7f0fc806c100 2023-10-16T15:54:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcd993640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc815cda0 con 0x7f0fc8070b60 2023-10-16T15:54:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcd993640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81f2480 con 0x7f0fc8159950 2023-10-16T15:54:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcc991640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 0x7f0fc81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcc991640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 0x7f0fc81570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44566/0 (socket says 172.21.15.29:44566) 2023-10-16T15:54:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.643+0000 7f0fcc991640 1 -- 172.21.15.29:0/1681489607 learned_addr learned my addr 172.21.15.29:0/1681489607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fc7fff640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fc8070b60 0x7f0fc815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcd192640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc8159950 0x7f0fc80c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcc991640 1 -- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc8159950 msgr2=0x7f0fc80c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcc991640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc8159950 0x7f0fc80c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcc991640 1 -- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fc8070b60 msgr2=0x7f0fc815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcc991640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fc8070b60 0x7f0fc815be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcc991640 1 -- 172.21.15.29:0/1681489607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4009400 con 0x7f0fc806c100 2023-10-16T15:54:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fcc991640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 0x7f0fc81570a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c480 tx=0x7f0fbc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.644+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc005ce0 con 0x7f0fc806c100 2023-10-16T15:54:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.645+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc005e40 con 0x7f0fc806c100 2023-10-16T15:54:13.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.645+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0072f0 con 0x7f0fc806c100 2023-10-16T15:54:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.645+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc80c7b70 con 0x7f0fc806c100 2023-10-16T15:54:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.645+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc815c7e0 con 0x7f0fc806c100 2023-10-16T15:54:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.646+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc804e9b0 con 0x7f0fc806c100 2023-10-16T15:54:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.647+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0fbc007510 con 0x7f0fc806c100 2023-10-16T15:54:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.648+0000 7f0fc57fa640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0fac06b0b0 0x7f0fac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:13.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.648+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f0fbc089480 con 0x7f0fc806c100 2023-10-16T15:54:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.649+0000 7f0fc7fff640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0fac06b0b0 0x7f0fac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.650+0000 7f0fc7fff640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0fac06b0b0 0x7f0fac06d570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4036000 tx=0x7f0fb403d040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:13.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.652+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0fbc053bc0 con 0x7f0fc806c100 2023-10-16T15:54:13.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.875+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fc815aef0 con 0x7f0fc806c100 2023-10-16T15:54:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.876+0000 7f0fc57fa640 1 -- 172.21.15.29:0/1681489607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f0fc815aef0 con 0x7f0fc806c100 2023-10-16T15:54:13.877 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:13.878 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 114 2023-10-16T15:54:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.878+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0fac06b0b0 msgr2=0x7f0fac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.878+0000 7f0fcd993640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0fac06b0b0 0x7f0fac06d570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4036000 tx=0x7f0fb403d040 comp rx=0 tx=0).stop 2023-10-16T15:54:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.878+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 msgr2=0x7f0fc81570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 0x7f0fc81570a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c480 tx=0x7f0fbc00c940 comp rx=0 tx=0).stop 2023-10-16T15:54:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 shutdown_connections 2023-10-16T15:54:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0fac06b0b0 0x7f0fac06d570 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0fc8159950 0x7f0fc80c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0fc8070b60 0x7f0fc815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 --2- 172.21.15.29:0/1681489607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc806c100 0x7f0fc81570a0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 >> 172.21.15.29:0/1681489607 conn(0x7f0fc80ce6a0 msgr2=0x7f0fc8070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 shutdown_connections 2023-10-16T15:54:13.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:13.879+0000 7f0fcd993640 1 -- 172.21.15.29:0/1681489607 wait complete. 2023-10-16T15:54:13.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:14.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.103+0000 7ff8afff0640 1 Processor -- start 2023-10-16T15:54:14.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.103+0000 7ff8afff0640 1 -- start start 2023-10-16T15:54:14.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8afff0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a80cd0c0 0x7ff8a81615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8afff0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8a80b0410 0x7ff8a80b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8afff0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a80b0db0 con 0x7ff8a815d3f0 2023-10-16T15:54:14.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8afff0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a815e3e0 con 0x7ff8a80b0410 2023-10-16T15:54:14.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8afff0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80cd4a0 con 0x7ff8a80cd0c0 2023-10-16T15:54:14.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8ad58b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8a80b0410 0x7ff8a80b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8ad58b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8a80b0410 0x7ff8a80b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59986/0 (socket says 172.21.15.29:59986) 2023-10-16T15:54:14.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.105+0000 7ff8ad58b640 1 -- 172.21.15.29:0/897878507 learned_addr learned my addr 172.21.15.29:0/897878507 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:14.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.106+0000 7ff8add8c640 1 --2- 172.21.15.29:0/897878507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a80cd0c0 0x7ff8a81615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.106+0000 7ff8ad58b640 1 -- 172.21.15.29:0/897878507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a80cd0c0 msgr2=0x7ff8a81615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:14.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.106+0000 7ff8ad58b640 1 --2- 172.21.15.29:0/897878507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a80cd0c0 0x7ff8a81615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.106+0000 7ff8ad58b640 1 -- 172.21.15.29:0/897878507 >> v1:172.21.15.170:6789/0 conn(0x7ff8a815d3f0 legacy=0x7ff8a815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:14.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.106+0000 7ff8ad58b640 1 -- 172.21.15.29:0/897878507 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a815e100 con 0x7ff8a80b0410 2023-10-16T15:54:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.107+0000 7ff8ad58b640 1 --2- 172.21.15.29:0/897878507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80b0870 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7ff898009870 tx=0x7ff89802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a889de5544bdac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.107+0000 7ff8acd8a640 1 -- 172.21.15.29:0/897878507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8980317c0 con 0x7ff8a80b0410 2023-10-16T15:54:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.107+0000 7ff8acd8a640 1 -- 172.21.15.29:0/897878507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898031920 con 0x7ff8a80b0410 2023-10-16T15:54:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.107+0000 7ff8acd8a640 1 -- 172.21.15.29:0/897878507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898031cb0 con 0x7ff8a80b0410 2023-10-16T15:54:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 -- 172.21.15.29:0/897878507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 msgr2=0x7ff8a80b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 --2- 172.21.15.29:0/897878507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80b0870 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7ff898009870 tx=0x7ff89802b860 comp rx=0 tx=0).stop 2023-10-16T15:54:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 -- 172.21.15.29:0/897878507 shutdown_connections 2023-10-16T15:54:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 --2- 172.21.15.29:0/897878507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80b0870 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 --2- 172.21.15.29:0/897878507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a80cd0c0 0x7ff8a81615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 -- 172.21.15.29:0/897878507 >> 172.21.15.29:0/897878507 conn(0x7ff8a80cb410 msgr2=0x7ff8a80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:14.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 -- 172.21.15.29:0/897878507 shutdown_connections 2023-10-16T15:54:14.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.108+0000 7ff8afff0640 1 -- 172.21.15.29:0/897878507 wait complete. 2023-10-16T15:54:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.110+0000 7ff8afff0640 1 Processor -- start 2023-10-16T15:54:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.110+0000 7ff8afff0640 1 -- start start 2023-10-16T15:54:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8afff0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8afff0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8a80cd0c0 0x7ff8a8159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8afff0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a815d3f0 0x7ff8a8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8afff0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80c3570 con 0x7ff8a80b0410 2023-10-16T15:54:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8afff0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80c3850 con 0x7ff8a80cd0c0 2023-10-16T15:54:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8afff0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80c3b30 con 0x7ff8a815d3f0 2023-10-16T15:54:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8add8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8add8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80c4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59988/0 (socket says 172.21.15.29:59988) 2023-10-16T15:54:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8add8c640 1 -- 172.21.15.29:0/3904393880 learned_addr learned my addr 172.21.15.29:0/3904393880 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8add8c640 1 -- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a815d3f0 msgr2=0x7ff8a8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.111+0000 7ff8ae58d640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a815d3f0 0x7ff8a8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8ad58b640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8a80cd0c0 0x7ff8a8159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8add8c640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a815d3f0 0x7ff8a8155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8add8c640 1 -- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8a80cd0c0 msgr2=0x7ff8a8159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8add8c640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8a80cd0c0 0x7ff8a8159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8add8c640 1 -- 172.21.15.29:0/3904393880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980094d0 con 0x7ff8a80b0410 2023-10-16T15:54:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8ad58b640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8a80cd0c0 0x7ff8a8159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8add8c640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80c4be0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400d800 tx=0x7ff8a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a400b6c0 con 0x7ff8a80b0410 2023-10-16T15:54:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a400b820 con 0x7ff8a80b0410 2023-10-16T15:54:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a400bbb0 con 0x7ff8a80b0410 2023-10-16T15:54:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a80c3e10 con 0x7ff8a80b0410 2023-10-16T15:54:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.112+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a81599c0 con 0x7ff8a80b0410 2023-10-16T15:54:14.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.113+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a80c4000 con 0x7ff8a80b0410 2023-10-16T15:54:14.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.114+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff8a400bd10 con 0x7ff8a80b0410 2023-10-16T15:54:14.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.115+0000 7ff89e7fc640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff88c06b0b0 0x7ff88c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.115+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7ff8a408a730 con 0x7ff8a80b0410 2023-10-16T15:54:14.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.116+0000 7ff8ad58b640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff88c06b0b0 0x7ff88c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.117+0000 7ff8ad58b640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff88c06b0b0 0x7ff88c06d570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff898036000 tx=0x7ff89803d040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:14.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.119+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff8a4054d30 con 0x7ff8a80b0410 2023-10-16T15:54:14.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.346+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8a804e160 con 0x7ff8a80b0410 2023-10-16T15:54:14.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.348+0000 7ff89e7fc640 1 -- 172.21.15.29:0/3904393880 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+6155 (secure 0 0 0) 0x7ff8a4058350 con 0x7ff8a80b0410 2023-10-16T15:54:14.349 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:14.349 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:26.655925+0000","modified":"2023-10-16T15:53:29.777218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10821":{"gid":10821,"name":"c","rank":0,"incarnation":105,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3313937659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3313937659},{"type":"v1","addr":"172.21.15.29:6835","nonce":3313937659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 114 2023-10-16T15:54:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff88c06b0b0 msgr2=0x7ff88c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff88c06b0b0 0x7ff88c06d570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff898036000 tx=0x7ff89803d040 comp rx=0 tx=0).stop 2023-10-16T15:54:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 msgr2=0x7ff8a80c4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80c4be0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400d800 tx=0x7ff8a400dcc0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 shutdown_connections 2023-10-16T15:54:14.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff88c06b0b0 0x7ff88c06d570 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff8a815d3f0 0x7ff8a8155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8a80cd0c0 0x7ff8a8159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 --2- 172.21.15.29:0/3904393880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a80b0410 0x7ff8a80c4be0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 >> 172.21.15.29:0/3904393880 conn(0x7ff8a80cb410 msgr2=0x7ff8a80ca3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:14.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 shutdown_connections 2023-10-16T15:54:14.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.350+0000 7ff8afff0640 1 -- 172.21.15.29:0/3904393880 wait complete. 2023-10-16T15:54:14.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:14.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.589+0000 7f7f3ffff640 1 Processor -- start 2023-10-16T15:54:14.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.589+0000 7f7f3ffff640 1 -- start start 2023-10-16T15:54:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 0x7f7f4006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7f400ad970 0x7f7f4015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4006c960 con 0x7f7f4015ebc0 2023-10-16T15:54:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4015b9d0 con 0x7f7f400ad970 2023-10-16T15:54:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4015bcb0 con 0x7f7f4006c040 2023-10-16T15:54:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7f4015ebc0 0x7f7f400cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55708/0 (socket says 172.21.15.29:55708) 2023-10-16T15:54:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3f7fe640 1 -- 172.21.15.29:0/3667746975 learned_addr learned my addr 172.21.15.29:0/3667746975 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:14.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3effd640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 0x7f7f4006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f37fff640 1 --2- 172.21.15.29:0/3667746975 >> v2:172.21.15.170:3300/0 conn(0x7f7f400ad970 0x7f7f4015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.591+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338230817 0 0) 0x7f7f4006c960 con 0x7f7f4015ebc0 2023-10-16T15:54:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c003520 con 0x7f7f4015ebc0 2023-10-16T15:54:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707915209 0 0) 0x7f7f2c003520 con 0x7f7f4015ebc0 2023-10-16T15:54:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 msgr2=0x7f7f4006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 0x7f7f4006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 >> v2:172.21.15.170:3300/0 conn(0x7f7f400ad970 msgr2=0x7f7f4015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 --2- 172.21.15.29:0/3667746975 >> v2:172.21.15.170:3300/0 conn(0x7f7f400ad970 0x7f7f4015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4015bf90 con 0x7f7f4015ebc0 2023-10-16T15:54:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7f380019f0 con 0x7f7f4015ebc0 2023-10-16T15:54:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 >> v1:172.21.15.29:6789/0 conn(0x7f7f4015ebc0 legacy=0x7f7f400cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 0x7f7f2c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c003f40 0x7f7f2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4015b9d0 con 0x7f7f2c004d00 2023-10-16T15:54:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4015bcb0 con 0x7f7f2c003f40 2023-10-16T15:54:14.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c00b980 con 0x7f7f4006c040 2023-10-16T15:54:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.592+0000 7f7f3f7fe640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f3effd640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 0x7f7f2c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f37fff640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c003f40 0x7f7f2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f3f7fe640 1 -- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 msgr2=0x7f7f2c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f3f7fe640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f4006c040 0x7f7f2c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f3f7fe640 1 -- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c003f40 msgr2=0x7f7f2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f3f7fe640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f2c003f40 0x7f7f2c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.593+0000 7f7f3f7fe640 1 -- 172.21.15.29:0/3667746975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c003520 con 0x7f7f2c004d00 2023-10-16T15:54:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.594+0000 7f7f3f7fe640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f2c005180 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800a790 tx=0x7f7f2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440c8cc017a83be6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.594+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38034630 con 0x7f7f2c004d00 2023-10-16T15:54:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.594+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3802f070 con 0x7f7f2c004d00 2023-10-16T15:54:14.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.594+0000 7f7f3e7fc640 1 -- 172.21.15.29:0/3667746975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f380349f0 con 0x7f7f2c004d00 2023-10-16T15:54:14.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3ffff640 1 -- 172.21.15.29:0/3667746975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 msgr2=0x7f7f2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/3667746975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f2c005180 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800a790 tx=0x7f7f2c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3f7fe640 1 -- 172.21.15.29:0/3667746975 reap_dead start 2023-10-16T15:54:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3ffff640 1 -- 172.21.15.29:0/3667746975 shutdown_connections 2023-10-16T15:54:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3ffff640 1 -- 172.21.15.29:0/3667746975 >> 172.21.15.29:0/3667746975 conn(0x7f7f400ce6a0 msgr2=0x7f7f400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3ffff640 1 -- 172.21.15.29:0/3667746975 shutdown_connections 2023-10-16T15:54:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.595+0000 7f7f3ffff640 1 -- 172.21.15.29:0/3667746975 wait complete. 2023-10-16T15:54:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.597+0000 7f7f3ffff640 1 Processor -- start 2023-10-16T15:54:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.597+0000 7f7f3ffff640 1 -- start start 2023-10-16T15:54:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f401f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f401f2cd0 0x7f7f401e4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f401f1ab0 0x7f7f400ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f401f39e0 con 0x7f7f2c004d00 2023-10-16T15:54:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f401f3cc0 con 0x7f7f401f2cd0 2023-10-16T15:54:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f401f3fa0 con 0x7f7f401f1ab0 2023-10-16T15:54:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f401f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f401f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59998/0 (socket says 172.21.15.29:59998) 2023-10-16T15:54:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.598+0000 7f7f3effd640 1 -- 172.21.15.29:0/1471130083 learned_addr learned my addr 172.21.15.29:0/1471130083 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3effd640 1 -- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f401f1ab0 msgr2=0x7f7f400ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3effd640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f401f1ab0 0x7f7f400ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3effd640 1 -- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f401f2cd0 msgr2=0x7f7f401e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3effd640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f401f2cd0 0x7f7f401e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3effd640 1 -- 172.21.15.29:0/1471130083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f401f4280 con 0x7f7f2c004d00 2023-10-16T15:54:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3effd640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f401f33d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f7f300119a0 tx=0x7f7f30011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000fa70 con 0x7f7f2c004d00 2023-10-16T15:54:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3000fbd0 con 0x7f7f2c004d00 2023-10-16T15:54:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30014440 con 0x7f7f2c004d00 2023-10-16T15:54:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.599+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f401f4560 con 0x7f7f2c004d00 2023-10-16T15:54:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.600+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f401e49a0 con 0x7f7f2c004d00 2023-10-16T15:54:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.601+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4006b310 con 0x7f7f2c004d00 2023-10-16T15:54:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.602+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7f300145a0 con 0x7f7f2c004d00 2023-10-16T15:54:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.602+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7f2006b0b0 0x7f7f2006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.603+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f7f3008a9a0 con 0x7f7f2c004d00 2023-10-16T15:54:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.603+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7f2006b0b0 0x7f7f2006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.604+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7f2006b0b0 0x7f7f2006d570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f40159ac0 tx=0x7f7f28008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.606+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f30054fa0 con 0x7f7f2c004d00 2023-10-16T15:54:14.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.803+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f4004e9b0 con 0x7f7f2c004d00 2023-10-16T15:54:14.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.806+0000 7f7f377fe640 1 -- 172.21.15.29:0/1471130083 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+21557 (secure 0 0 0) 0x7f7f30022090 con 0x7f7f2c004d00 2023-10-16T15:54:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:14.807 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":231,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:09.426111+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:53:24.608039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T15:53:25.694889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:14.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.808+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7f2006b0b0 msgr2=0x7f7f2006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.808+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7f2006b0b0 0x7f7f2006d570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f40159ac0 tx=0x7f7f28008040 comp rx=0 tx=0).stop 2023-10-16T15:54:14.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.808+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 msgr2=0x7f7f401f33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:14.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.808+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f401f33d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f7f300119a0 tx=0x7f7f30011e60 comp rx=0 tx=0).stop 2023-10-16T15:54:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 shutdown_connections 2023-10-16T15:54:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7f2006b0b0 0x7f7f2006d570 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f401f1ab0 0x7f7f400ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f401f2cd0 0x7f7f401e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 --2- 172.21.15.29:0/1471130083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f2c004d00 0x7f7f401f33d0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 >> 172.21.15.29:0/1471130083 conn(0x7f7f400ce6a0 msgr2=0x7f7f4015b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 shutdown_connections 2023-10-16T15:54:14.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:14.809+0000 7f7f3ffff640 1 -- 172.21.15.29:0/1471130083 wait complete. 2023-10-16T15:54:14.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:54:15.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.055+0000 7f5e94f9e640 1 Processor -- start 2023-10-16T15:54:15.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.056+0000 7f5e94f9e640 1 -- start start 2023-10-16T15:54:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.057+0000 7f5e94f9e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5e900caa80 0x7f5e90069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.057+0000 7f5e94f9e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e9006a2e0 0x7f5e901650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.057+0000 7f5e94f9e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9006a730 con 0x7f5e900addb0 2023-10-16T15:54:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.057+0000 7f5e94f9e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e90161a50 con 0x7f5e900caa80 2023-10-16T15:54:15.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.057+0000 7f5e94f9e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900caef0 con 0x7f5e9006a2e0 2023-10-16T15:54:15.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5e900caa80 0x7f5e90069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5e900caa80 0x7f5e90069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60010/0 (socket says 172.21.15.29:60010) 2023-10-16T15:54:15.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/2084045397 learned_addr learned my addr 172.21.15.29:0/2084045397 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8ed9d640 1 --2- 172.21.15.29:0/2084045397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e9006a2e0 0x7f5e901650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/2084045397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e9006a2e0 msgr2=0x7f5e901650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 --2- 172.21.15.29:0/2084045397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e9006a2e0 0x7f5e901650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/2084045397 >> v1:172.21.15.29:6789/0 conn(0x7f5e900addb0 legacy=0x7f5e900c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.058+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/2084045397 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900cb140 con 0x7f5e900caa80 2023-10-16T15:54:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.059+0000 7f5e8dd9b640 1 --2- 172.21.15.29:0/2084045397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900caa80 0x7f5e90069da0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009870 tx=0x7f5e7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f8275fd3e50135 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.060+0000 7f5e8d59a640 1 -- 172.21.15.29:0/2084045397 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78030840 con 0x7f5e900caa80 2023-10-16T15:54:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.060+0000 7f5e8d59a640 1 -- 172.21.15.29:0/2084045397 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e780309a0 con 0x7f5e900caa80 2023-10-16T15:54:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.060+0000 7f5e8d59a640 1 -- 172.21.15.29:0/2084045397 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78030d30 con 0x7f5e900caa80 2023-10-16T15:54:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.060+0000 7f5e94f9e640 1 -- 172.21.15.29:0/2084045397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900caa80 msgr2=0x7f5e90069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.060+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/2084045397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900caa80 0x7f5e90069da0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009870 tx=0x7f5e7802b980 comp rx=0 tx=0).stop 2023-10-16T15:54:15.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.061+0000 7f5e94f9e640 1 -- 172.21.15.29:0/2084045397 shutdown_connections 2023-10-16T15:54:15.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.061+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/2084045397 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e9006a2e0 0x7f5e901650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.061+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/2084045397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900caa80 0x7f5e90069da0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.061+0000 7f5e94f9e640 1 -- 172.21.15.29:0/2084045397 >> 172.21.15.29:0/2084045397 conn(0x7f5e900ce9b0 msgr2=0x7f5e900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:15.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.061+0000 7f5e94f9e640 1 -- 172.21.15.29:0/2084045397 shutdown_connections 2023-10-16T15:54:15.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.061+0000 7f5e94f9e640 1 -- 172.21.15.29:0/2084045397 wait complete. 2023-10-16T15:54:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.062+0000 7f5e94f9e640 1 Processor -- start 2023-10-16T15:54:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 -- start start 2023-10-16T15:54:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e9006a2e0 0x7f5e901f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 0x7f5e90157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e900caa80 0x7f5e900c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900c7ad0 con 0x7f5e900addb0 2023-10-16T15:54:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900b2350 con 0x7f5e9006a2e0 2023-10-16T15:54:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.063+0000 7f5e94f9e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900b2630 con 0x7f5e900caa80 2023-10-16T15:54:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 0x7f5e90157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 0x7f5e90157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60018/0 (socket says 172.21.15.29:60018) 2023-10-16T15:54:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/816094635 learned_addr learned my addr 172.21.15.29:0/816094635 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8e59c640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e9006a2e0 0x7f5e901f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e900caa80 msgr2=0x7f5e900c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8ed9d640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e900caa80 0x7f5e900c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e900caa80 0x7f5e900c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e9006a2e0 msgr2=0x7f5e901f6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e9006a2e0 0x7f5e901f6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8dd9b640 1 -- 172.21.15.29:0/816094635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e780094d0 con 0x7f5e900addb0 2023-10-16T15:54:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.064+0000 7f5e8e59c640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e9006a2e0 0x7f5e901f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.065+0000 7f5e8dd9b640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 0x7f5e90157f50 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f5e7802b9c0 tx=0x7f5e780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.065+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7803e070 con 0x7f5e900addb0 2023-10-16T15:54:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.065+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e78030810 con 0x7f5e900addb0 2023-10-16T15:54:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.065+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780423f0 con 0x7f5e900addb0 2023-10-16T15:54:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.065+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900b2910 con 0x7f5e900addb0 2023-10-16T15:54:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.065+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e90158490 con 0x7f5e900addb0 2023-10-16T15:54:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.066+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e900c7c10 con 0x7f5e900addb0 2023-10-16T15:54:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.067+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5e78042550 con 0x7f5e900addb0 2023-10-16T15:54:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.068+0000 7f5e86ffd640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e7006b0b0 0x7f5e7006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.068+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7418+0+0 (secure 0 0 0) 0x7f5e780ac180 con 0x7f5e900addb0 2023-10-16T15:54:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.069+0000 7f5e8e59c640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e7006b0b0 0x7f5e7006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.070+0000 7f5e8e59c640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e7006b0b0 0x7f5e7006d570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e800012e0 tx=0x7f5e80009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.072+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e780765e0 con 0x7f5e900addb0 2023-10-16T15:54:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.310+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5e9004e160 con 0x7f5e900addb0 2023-10-16T15:54:15.313 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:54:15.312+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:54:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.316+0000 7f5e86ffd640 1 -- 172.21.15.29:0/816094635 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v115) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5e78034090 con 0x7f5e900addb0 2023-10-16T15:54:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:54:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e7006b0b0 msgr2=0x7f5e7006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e7006b0b0 0x7f5e7006d570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e800012e0 tx=0x7f5e80009040 comp rx=0 tx=0).stop 2023-10-16T15:54:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 msgr2=0x7f5e90157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 0x7f5e90157f50 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f5e7802b9c0 tx=0x7f5e780046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 shutdown_connections 2023-10-16T15:54:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5e7006b0b0 0x7f5e7006d570 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900addb0 0x7f5e90157f50 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5e900caa80 0x7f5e900c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 --2- 172.21.15.29:0/816094635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e9006a2e0 0x7f5e901f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 >> 172.21.15.29:0/816094635 conn(0x7f5e900ce9b0 msgr2=0x7f5e900ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.319+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 shutdown_connections 2023-10-16T15:54:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.320+0000 7f5e94f9e640 1 -- 172.21.15.29:0/816094635 wait complete. 2023-10-16T15:54:15.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:54:15.350 INFO:tasks.ceph.mds.c.smithi029.stderr: -4777> 2023-10-16T15:53:33.155+0000 7f3679aea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:54:15.425 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:54:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.543+0000 7f5ce0e10640 1 Processor -- start 2023-10-16T15:54:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.543+0000 7f5ce0e10640 1 -- start start 2023-10-16T15:54:15.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5ce0e10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5ce0e10640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5cdc06c140 0x7f5cdc06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5ce0e10640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc070360 con 0x7f5cdc159e90 2023-10-16T15:54:15.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5ce0e10640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc06cb10 con 0x7f5cdc06c140 2023-10-16T15:54:15.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5ce0e10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc15d890 con 0x7f5cdc159ab0 2023-10-16T15:54:15.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5cd9d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5cdc159e90 0x7f5cdc06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55746/0 (socket says 172.21.15.29:55746) 2023-10-16T15:54:15.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5cd9d9b640 1 -- 172.21.15.29:0/4024346298 learned_addr learned my addr 172.21.15.29:0/4024346298 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:15.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.545+0000 7f5cdad9d640 1 --2- 172.21.15.29:0/4024346298 >> v2:172.21.15.29:3300/0 conn(0x7f5cdc06c140 0x7f5cdc06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.546+0000 7f5cda59c640 1 --2- 172.21.15.29:0/4024346298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.546+0000 7f5cdad9d640 1 -- 172.21.15.29:0/4024346298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 msgr2=0x7f5cdc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:15.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.546+0000 7f5cd959a640 1 -- 172.21.15.29:0/4024346298 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203810651 0 0) 0x7f5cdc070360 con 0x7f5cdc159e90 2023-10-16T15:54:15.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.546+0000 7f5cdad9d640 1 --2- 172.21.15.29:0/4024346298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.546+0000 7f5cdad9d640 1 -- 172.21.15.29:0/4024346298 >> v1:172.21.15.29:6789/0 conn(0x7f5cdc159e90 legacy=0x7f5cdc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.546+0000 7f5cdad9d640 1 -- 172.21.15.29:0/4024346298 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0cd1f0 con 0x7f5cdc06c140 2023-10-16T15:54:15.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.547+0000 7f5cdad9d640 1 --2- 172.21.15.29:0/4024346298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc06c5d0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009dc0 tx=0x7f5cd402bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf40f6d8c7533f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:15.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.547+0000 7f5cd959a640 1 -- 172.21.15.29:0/4024346298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd402cb20 con 0x7f5cdc06c140 2023-10-16T15:54:15.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.547+0000 7f5cd959a640 1 -- 172.21.15.29:0/4024346298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd402cc80 con 0x7f5cdc06c140 2023-10-16T15:54:15.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.547+0000 7f5cd959a640 1 -- 172.21.15.29:0/4024346298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4034440 con 0x7f5cdc06c140 2023-10-16T15:54:15.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 -- 172.21.15.29:0/4024346298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 msgr2=0x7f5cdc06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/4024346298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc06c5d0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009dc0 tx=0x7f5cd402bd70 comp rx=0 tx=0).stop 2023-10-16T15:54:15.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 -- 172.21.15.29:0/4024346298 shutdown_connections 2023-10-16T15:54:15.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/4024346298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc06c5d0 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/4024346298 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 -- 172.21.15.29:0/4024346298 >> 172.21.15.29:0/4024346298 conn(0x7f5cdc0ce9b0 msgr2=0x7f5cdc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:15.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 -- 172.21.15.29:0/4024346298 shutdown_connections 2023-10-16T15:54:15.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.548+0000 7f5ce0e10640 1 -- 172.21.15.29:0/4024346298 wait complete. 2023-10-16T15:54:15.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.550+0000 7f5ce0e10640 1 Processor -- start 2023-10-16T15:54:15.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.550+0000 7f5ce0e10640 1 -- start start 2023-10-16T15:54:15.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5ce0e10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5ce0e10640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc1f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5ce0e10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cdc159e90 0x7f5cdc0c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5ce0e10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0c6700 con 0x7f5cdc06c140 2023-10-16T15:54:15.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5ce0e10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc1f1fb0 con 0x7f5cdc159e90 2023-10-16T15:54:15.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5ce0e10640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0c5a80 con 0x7f5cdc159ab0 2023-10-16T15:54:15.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60060/0 (socket says 172.21.15.29:60060) 2023-10-16T15:54:15.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.551+0000 7f5cda59c640 1 -- 172.21.15.29:0/2222632801 learned_addr learned my addr 172.21.15.29:0/2222632801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:15.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cda59c640 1 -- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 msgr2=0x7f5cdc1f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cda59c640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc1f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cdad9d640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cdc159e90 0x7f5cdc0c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cda59c640 1 -- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cdc159e90 msgr2=0x7f5cdc0c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:15.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cda59c640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cdc159e90 0x7f5cdc0c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:15.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cda59c640 1 -- 172.21.15.29:0/2222632801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009a70 con 0x7f5cdc06c140 2023-10-16T15:54:15.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cdad9d640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cdc159e90 0x7f5cdc0c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:15.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cda59c640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc14fff0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00d620 tx=0x7f5ccc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00b620 con 0x7f5cdc06c140 2023-10-16T15:54:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc00b780 con 0x7f5cdc06c140 2023-10-16T15:54:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00bb30 con 0x7f5cdc06c140 2023-10-16T15:54:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.552+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc0c5d60 con 0x7f5cdc06c140 2023-10-16T15:54:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.553+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc0c6320 con 0x7f5cdc06c140 2023-10-16T15:54:15.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.553+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc0c64e0 con 0x7f5cdc06c140 2023-10-16T15:54:15.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.555+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5ccc00bc90 con 0x7f5cdc06c140 2023-10-16T15:54:15.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.556+0000 7f5cd2ffd640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5cbc06b0b0 0x7f5cbc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:15.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.556+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7504+0+0 (secure 0 0 0) 0x7f5ccc08a690 con 0x7f5cdc06c140 2023-10-16T15:54:15.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.556+0000 7f5cd9d9b640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5cbc06b0b0 0x7f5cbc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:15.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.557+0000 7f5cd9d9b640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5cbc06b0b0 0x7f5cbc06d570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007fd0 tx=0x7f5cc4007920 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:15.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.560+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5ccc054c60 con 0x7f5cdc06c140 2023-10-16T15:54:15.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:15.791+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5cdc06c520 con 0x7f5cdc06c140 2023-10-16T15:54:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.327+0000 7f5cd2ffd640 1 -- 172.21.15.29:0/2222632801 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v116) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5ccc058280 con 0x7f5cdc06c140 2023-10-16T15:54:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5cbc06b0b0 msgr2=0x7f5cbc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5cbc06b0b0 0x7f5cbc06d570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007fd0 tx=0x7f5cc4007920 comp rx=0 tx=0).stop 2023-10-16T15:54:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 msgr2=0x7f5cdc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc14fff0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00d620 tx=0x7f5ccc00dae0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 shutdown_connections 2023-10-16T15:54:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5cbc06b0b0 0x7f5cbc06d570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5cdc159e90 0x7f5cdc0c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5cdc159ab0 0x7f5cdc1f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 --2- 172.21.15.29:0/2222632801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cdc06c140 0x7f5cdc14fff0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 >> 172.21.15.29:0/2222632801 conn(0x7f5cdc0ce9b0 msgr2=0x7f5cdc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 shutdown_connections 2023-10-16T15:54:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.331+0000 7f5ce0e10640 1 -- 172.21.15.29:0/2222632801 wait complete. 2023-10-16T15:54:16.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:54:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.590+0000 7ff348f8c640 1 Processor -- start 2023-10-16T15:54:16.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.591+0000 7ff348f8c640 1 -- start start 2023-10-16T15:54:16.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.592+0000 7ff348f8c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff34415cd70 0x7ff3440d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:16.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.592+0000 7ff348f8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:16.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.592+0000 7ff348f8c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34415d150 con 0x7ff3440b8480 2023-10-16T15:54:16.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.592+0000 7ff348f8c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3440d1180 con 0x7ff34415cd70 2023-10-16T15:54:16.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.592+0000 7ff348f8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440d1460 con 0x7ff3440cded0 2023-10-16T15:54:16.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff34415cd70 0x7ff3440d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:16.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff34415cd70 0x7ff3440d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60062/0 (socket says 172.21.15.29:60062) 2023-10-16T15:54:16.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 -- 172.21.15.29:0/1200983063 learned_addr learned my addr 172.21.15.29:0/1200983063 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:16.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 -- 172.21.15.29:0/1200983063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 msgr2=0x7ff3440b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:16.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 --2- 172.21.15.29:0/1200983063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 -- 172.21.15.29:0/1200983063 >> v1:172.21.15.170:6789/0 conn(0x7ff3440b8480 legacy=0x7ff3440cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:16.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.593+0000 7ff3437fe640 1 -- 172.21.15.29:0/1200983063 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440d1740 con 0x7ff34415cd70 2023-10-16T15:54:16.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.594+0000 7ff3437fe640 1 --2- 172.21.15.29:0/1200983063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff34415cd70 0x7ff3440d2220 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7ff3440b56a0 tx=0x7ff33402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71df3af025e6abb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:16.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.595+0000 7ff3427fc640 1 -- 172.21.15.29:0/1200983063 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334031830 con 0x7ff34415cd70 2023-10-16T15:54:16.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.595+0000 7ff3427fc640 1 -- 172.21.15.29:0/1200983063 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334031990 con 0x7ff34415cd70 2023-10-16T15:54:16.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.595+0000 7ff3427fc640 1 -- 172.21.15.29:0/1200983063 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334031d20 con 0x7ff34415cd70 2023-10-16T15:54:16.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.595+0000 7ff348f8c640 1 -- 172.21.15.29:0/1200983063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff34415cd70 msgr2=0x7ff3440d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.595+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1200983063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff34415cd70 0x7ff3440d2220 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7ff3440b56a0 tx=0x7ff33402b8d0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.596+0000 7ff348f8c640 1 -- 172.21.15.29:0/1200983063 shutdown_connections 2023-10-16T15:54:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.596+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1200983063 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.596+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1200983063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff34415cd70 0x7ff3440d2220 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.596+0000 7ff348f8c640 1 -- 172.21.15.29:0/1200983063 >> 172.21.15.29:0/1200983063 conn(0x7ff3440cf570 msgr2=0x7ff3440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:16.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.596+0000 7ff348f8c640 1 -- 172.21.15.29:0/1200983063 shutdown_connections 2023-10-16T15:54:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.596+0000 7ff348f8c640 1 -- 172.21.15.29:0/1200983063 wait complete. 2023-10-16T15:54:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.598+0000 7ff348f8c640 1 Processor -- start 2023-10-16T15:54:16.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.598+0000 7ff348f8c640 1 -- start start 2023-10-16T15:54:16.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff348f8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 0x7ff344146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:16.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff348f8c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:16.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff348f8c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff34415cd70 0x7ff3441570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:16.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff348f8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3441f5c30 con 0x7ff3440b8480 2023-10-16T15:54:16.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff348f8c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3441f5f10 con 0x7ff34415cd70 2023-10-16T15:54:16.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff348f8c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3441f61f0 con 0x7ff3440cded0 2023-10-16T15:54:16.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff3437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 0x7ff344146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:16.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff3437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 0x7ff344146910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60064/0 (socket says 172.21.15.29:60064) 2023-10-16T15:54:16.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff3437fe640 1 -- 172.21.15.29:0/1627110231 learned_addr learned my addr 172.21.15.29:0/1627110231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:16.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.599+0000 7ff342ffd640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff3437fe640 1 -- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 msgr2=0x7ff3440cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff343fff640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff34415cd70 0x7ff3441570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff3437fe640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff3437fe640 1 -- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff34415cd70 msgr2=0x7ff3441570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:16.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff3437fe640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff34415cd70 0x7ff3441570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff3437fe640 1 -- 172.21.15.29:0/1627110231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340094d0 con 0x7ff3440b8480 2023-10-16T15:54:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff342ffd640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff3437fe640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 0x7ff344146910 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7ff3340090b0 tx=0x7ff3340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33403aa60 con 0x7ff3440b8480 2023-10-16T15:54:16.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33403abe0 con 0x7ff3440b8480 2023-10-16T15:54:16.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3340443f0 con 0x7ff3440b8480 2023-10-16T15:54:16.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.600+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3441f64d0 con 0x7ff3440b8480 2023-10-16T15:54:16.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.601+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3441f58c0 con 0x7ff3440b8480 2023-10-16T15:54:16.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.602+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34404e9b0 con 0x7ff3440b8480 2023-10-16T15:54:16.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.603+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff33404d450 con 0x7ff3440b8480 2023-10-16T15:54:16.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.603+0000 7ff32bfff640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff33006b0b0 0x7ff33006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:16.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.604+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7504+0+0 (secure 0 0 0) 0x7ff3340ac3a0 con 0x7ff3440b8480 2023-10-16T15:54:16.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.604+0000 7ff342ffd640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff33006b0b0 0x7ff33006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:16.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.605+0000 7ff342ffd640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff33006b0b0 0x7ff33006d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3441f5820 tx=0x7ff338009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:16.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.608+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff334076970 con 0x7ff3440b8480 2023-10-16T15:54:16.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:16.803+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3440d0ea0 con 0x7ff3440b8480 2023-10-16T15:54:17.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.511+0000 7ff32bfff640 1 -- 172.21.15.29:0/1627110231 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v233) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff334079f90 con 0x7ff3440b8480 2023-10-16T15:54:17.512 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:54:17.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.514+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff33006b0b0 msgr2=0x7ff33006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:17.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.514+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff33006b0b0 0x7ff33006d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3441f5820 tx=0x7ff338009040 comp rx=0 tx=0).stop 2023-10-16T15:54:17.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.514+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 msgr2=0x7ff344146910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:17.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.514+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 0x7ff344146910 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7ff3340090b0 tx=0x7ff3340046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 shutdown_connections 2023-10-16T15:54:17.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff33006b0b0 0x7ff33006d570 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff34415cd70 0x7ff3441570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff3440cded0 0x7ff3440cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 --2- 172.21.15.29:0/1627110231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3440b8480 0x7ff344146910 secure :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7ff3340090b0 tx=0x7ff3340046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 >> 172.21.15.29:0/1627110231 conn(0x7ff3440cf570 msgr2=0x7ff3440b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:17.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.516+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 shutdown_connections 2023-10-16T15:54:17.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.517+0000 7ff348f8c640 1 -- 172.21.15.29:0/1627110231 wait complete. 2023-10-16T15:54:17.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:54:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.762+0000 7fed3f890640 1 Processor -- start 2023-10-16T15:54:17.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.762+0000 7fed3f890640 1 -- start start 2023-10-16T15:54:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.763+0000 7fed3f890640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fed3815ac10 0x7fed3806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.763+0000 7fed3f890640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed380cac20 0x7fed380b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.763+0000 7fed3f890640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3815b310 con 0x7fed380b0360 2023-10-16T15:54:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed3f890640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3815f800 con 0x7fed380cac20 2023-10-16T15:54:17.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed3f890640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed380b0d30 con 0x7fed3815ac10 2023-10-16T15:54:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed3ce2b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed380b0360 0x7fed380ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55760/0 (socket says 172.21.15.29:55760) 2023-10-16T15:54:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed3ce2b640 1 -- 172.21.15.29:0/3703312661 learned_addr learned my addr 172.21.15.29:0/3703312661 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed3de2d640 1 --2- 172.21.15.29:0/3703312661 >> v2:172.21.15.29:3300/0 conn(0x7fed380cac20 0x7fed380b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed3d62c640 1 --2- 172.21.15.29:0/3703312661 >> v2:172.21.15.170:3300/0 conn(0x7fed3815ac10 0x7fed3806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:17.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed2bfff640 1 -- 172.21.15.29:0/3703312661 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070677549 0 0) 0x7fed3815b310 con 0x7fed380b0360 2023-10-16T15:54:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.764+0000 7fed2bfff640 1 -- 172.21.15.29:0/3703312661 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24003520 con 0x7fed380b0360 2023-10-16T15:54:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.765+0000 7fed3de2d640 1 -- 172.21.15.29:0/3703312661 >> v2:172.21.15.170:3300/0 conn(0x7fed3815ac10 msgr2=0x7fed3806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.765+0000 7fed3de2d640 1 --2- 172.21.15.29:0/3703312661 >> v2:172.21.15.170:3300/0 conn(0x7fed3815ac10 0x7fed3806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.765+0000 7fed3de2d640 1 -- 172.21.15.29:0/3703312661 >> v1:172.21.15.29:6789/0 conn(0x7fed380b0360 legacy=0x7fed380ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.765+0000 7fed3de2d640 1 -- 172.21.15.29:0/3703312661 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3806f640 con 0x7fed380cac20 2023-10-16T15:54:17.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.766+0000 7fed3de2d640 1 --2- 172.21.15.29:0/3703312661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380cac20 0x7fed380b07f0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fed3806d540 tx=0x7fed3002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ed7ff33d21f8b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:17.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed2bfff640 1 -- 172.21.15.29:0/3703312661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3002cbe0 con 0x7fed380cac20 2023-10-16T15:54:17.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed2bfff640 1 -- 172.21.15.29:0/3703312661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3002cd40 con 0x7fed380cac20 2023-10-16T15:54:17.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed2bfff640 1 -- 172.21.15.29:0/3703312661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30034660 con 0x7fed380cac20 2023-10-16T15:54:17.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 -- 172.21.15.29:0/3703312661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380cac20 msgr2=0x7fed380b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:17.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 --2- 172.21.15.29:0/3703312661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380cac20 0x7fed380b07f0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fed3806d540 tx=0x7fed3002be30 comp rx=0 tx=0).stop 2023-10-16T15:54:17.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 -- 172.21.15.29:0/3703312661 shutdown_connections 2023-10-16T15:54:17.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 --2- 172.21.15.29:0/3703312661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed380cac20 0x7fed380b07f0 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 --2- 172.21.15.29:0/3703312661 >> v2:172.21.15.170:3300/0 conn(0x7fed3815ac10 0x7fed3806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 -- 172.21.15.29:0/3703312661 >> 172.21.15.29:0/3703312661 conn(0x7fed380cb360 msgr2=0x7fed380cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:17.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 -- 172.21.15.29:0/3703312661 shutdown_connections 2023-10-16T15:54:17.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.767+0000 7fed3f890640 1 -- 172.21.15.29:0/3703312661 wait complete. 2023-10-16T15:54:17.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.769+0000 7fed3f890640 1 Processor -- start 2023-10-16T15:54:17.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.769+0000 7fed3f890640 1 -- start start 2023-10-16T15:54:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.770+0000 7fed3f890640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed380b0360 0x7fed380ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.770+0000 7fed3f890640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed380cac20 0x7fed3814ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.770+0000 7fed3f890640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 0x7fed380cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.770+0000 7fed3f890640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3814f230 con 0x7fed3815ac10 2023-10-16T15:54:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.770+0000 7fed3f890640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed380cd0b0 con 0x7fed380b0360 2023-10-16T15:54:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.770+0000 7fed3f890640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed380cd390 con 0x7fed380cac20 2023-10-16T15:54:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 0x7fed380cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 0x7fed380cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60070/0 (socket says 172.21.15.29:60070) 2023-10-16T15:54:17.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 -- 172.21.15.29:0/4064413919 learned_addr learned my addr 172.21.15.29:0/4064413919 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3d62c640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed380b0360 0x7fed380ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3ce2b640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed380cac20 0x7fed3814ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 -- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed380cac20 msgr2=0x7fed3814ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:17.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed380cac20 0x7fed3814ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 -- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed380b0360 msgr2=0x7fed380ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed380b0360 0x7fed380ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 -- 172.21.15.29:0/4064413919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed30009a70 con 0x7fed3815ac10 2023-10-16T15:54:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3d62c640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed380b0360 0x7fed380ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:17.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.771+0000 7fed3de2d640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 0x7fed380cc560 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fed30009e80 tx=0x7fed30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.772+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30040650 con 0x7fed3815ac10 2023-10-16T15:54:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.772+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed300407b0 con 0x7fed3815ac10 2023-10-16T15:54:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.772+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30040b60 con 0x7fed3815ac10 2023-10-16T15:54:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.772+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed380cd670 con 0x7fed3815ac10 2023-10-16T15:54:17.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.772+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed380cdc30 con 0x7fed3815ac10 2023-10-16T15:54:17.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.773+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3804e160 con 0x7fed3815ac10 2023-10-16T15:54:17.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.774+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fed30040cc0 con 0x7fed3815ac10 2023-10-16T15:54:17.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.775+0000 7fed29ffb640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed1c06b0b0 0x7fed1c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:17.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.775+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7066+0+0 (secure 0 0 0) 0x7fed300acde0 con 0x7fed3815ac10 2023-10-16T15:54:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.776+0000 7fed3d62c640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed1c06b0b0 0x7fed1c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:17.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.777+0000 7fed3d62c640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed1c06b0b0 0x7fed1c06d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fed34006fd0 tx=0x7fed34008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:17.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.780+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fed30077540 con 0x7fed3815ac10 2023-10-16T15:54:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:17.983+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fed3806c2e0 con 0x7fed3815ac10 2023-10-16T15:54:18.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:54:18.189 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:54:18.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.514+0000 7fed29ffb640 1 -- 172.21.15.29:0/4064413919 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v234) v1 ==== 169+0+0 (secure 0 0 0) 0x7fed30009450 con 0x7fed3815ac10 2023-10-16T15:54:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:54:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed1c06b0b0 msgr2=0x7fed1c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed1c06b0b0 0x7fed1c06d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fed34006fd0 tx=0x7fed34008040 comp rx=0 tx=0).stop 2023-10-16T15:54:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 msgr2=0x7fed380cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 0x7fed380cc560 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fed30009e80 tx=0x7fed30004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 shutdown_connections 2023-10-16T15:54:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed1c06b0b0 0x7fed1c06d570 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed3815ac10 0x7fed380cc560 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed380cac20 0x7fed3814ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 --2- 172.21.15.29:0/4064413919 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed380b0360 0x7fed380ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 >> 172.21.15.29:0/4064413919 conn(0x7fed380cb360 msgr2=0x7fed3815e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.518+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 shutdown_connections 2023-10-16T15:54:18.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.519+0000 7fed3f890640 1 -- 172.21.15.29:0/4064413919 wait complete. 2023-10-16T15:54:18.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.757+0000 7f6d88725640 1 Processor -- start 2023-10-16T15:54:18.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.757+0000 7f6d88725640 1 -- start start 2023-10-16T15:54:18.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.758+0000 7f6d88725640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6d800b0b20 0x7f6d801615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:18.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.758+0000 7f6d88725640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6d8015d3f0 0x7f6d8015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.758+0000 7f6d88725640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8015e3e0 con 0x7f6d800cd0c0 2023-10-16T15:54:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d88725640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d800cd4a0 con 0x7f6d8015d3f0 2023-10-16T15:54:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d88725640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8015e100 con 0x7f6d800b0b20 2023-10-16T15:54:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d864c1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6d800cd0c0 0x7f6d800b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55764/0 (socket says 172.21.15.29:55764) 2023-10-16T15:54:18.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d864c1640 1 -- 172.21.15.29:0/4259946750 learned_addr learned my addr 172.21.15.29:0/4259946750 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d86cc2640 1 --2- 172.21.15.29:0/4259946750 >> v2:172.21.15.29:3300/0 conn(0x7f6d8015d3f0 0x7f6d8015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d854bf640 1 -- 172.21.15.29:0/4259946750 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958461926 0 0) 0x7f6d8015e3e0 con 0x7f6d800cd0c0 2023-10-16T15:54:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d854bf640 1 -- 172.21.15.29:0/4259946750 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6c003520 con 0x7f6d800cd0c0 2023-10-16T15:54:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d85cc0640 1 --2- 172.21.15.29:0/4259946750 >> v2:172.21.15.170:3300/0 conn(0x7f6d800b0b20 0x7f6d801615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d86cc2640 1 -- 172.21.15.29:0/4259946750 >> v2:172.21.15.170:3300/0 conn(0x7f6d800b0b20 msgr2=0x7f6d801615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:18.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d86cc2640 1 --2- 172.21.15.29:0/4259946750 >> v2:172.21.15.170:3300/0 conn(0x7f6d800b0b20 0x7f6d801615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d86cc2640 1 -- 172.21.15.29:0/4259946750 >> v1:172.21.15.29:6789/0 conn(0x7f6d800cd0c0 legacy=0x7f6d800b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.759+0000 7f6d86cc2640 1 -- 172.21.15.29:0/4259946750 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d80160b10 con 0x7f6d8015d3f0 2023-10-16T15:54:18.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.761+0000 7f6d86cc2640 1 --2- 172.21.15.29:0/4259946750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d8015d820 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f6d8015afc0 tx=0x7f6d7802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8371f1c1509483 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:18.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.761+0000 7f6d854bf640 1 -- 172.21.15.29:0/4259946750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7802e070 con 0x7f6d8015d3f0 2023-10-16T15:54:18.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.761+0000 7f6d854bf640 1 -- 172.21.15.29:0/4259946750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d780315a0 con 0x7f6d8015d3f0 2023-10-16T15:54:18.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.761+0000 7f6d854bf640 1 -- 172.21.15.29:0/4259946750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78031910 con 0x7f6d8015d3f0 2023-10-16T15:54:18.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.761+0000 7f6d88725640 1 -- 172.21.15.29:0/4259946750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 msgr2=0x7f6d8015d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.761+0000 7f6d88725640 1 --2- 172.21.15.29:0/4259946750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d8015d820 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f6d8015afc0 tx=0x7f6d7802b840 comp rx=0 tx=0).stop 2023-10-16T15:54:18.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.762+0000 7f6d88725640 1 -- 172.21.15.29:0/4259946750 shutdown_connections 2023-10-16T15:54:18.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.762+0000 7f6d88725640 1 --2- 172.21.15.29:0/4259946750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d8015d820 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.762+0000 7f6d88725640 1 --2- 172.21.15.29:0/4259946750 >> v2:172.21.15.170:3300/0 conn(0x7f6d800b0b20 0x7f6d801615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.762+0000 7f6d88725640 1 -- 172.21.15.29:0/4259946750 >> 172.21.15.29:0/4259946750 conn(0x7f6d800cb410 msgr2=0x7f6d800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.762+0000 7f6d88725640 1 -- 172.21.15.29:0/4259946750 shutdown_connections 2023-10-16T15:54:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.762+0000 7f6d88725640 1 -- 172.21.15.29:0/4259946750 wait complete. 2023-10-16T15:54:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.763+0000 7f6d88725640 1 Processor -- start 2023-10-16T15:54:18.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.764+0000 7f6d88725640 1 -- start start 2023-10-16T15:54:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d88725640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d800b0b20 0x7f6d801f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d88725640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d800cd0c0 0x7f6d800c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d88725640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d800b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d88725640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b23e0 con 0x7f6d8015d3f0 2023-10-16T15:54:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d88725640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b1550 con 0x7f6d800cd0c0 2023-10-16T15:54:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d88725640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b2800 con 0x7f6d800b0b20 2023-10-16T15:54:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d86cc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d800b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d86cc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d800b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60098/0 (socket says 172.21.15.29:60098) 2023-10-16T15:54:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d86cc2640 1 -- 172.21.15.29:0/2696758689 learned_addr learned my addr 172.21.15.29:0/2696758689 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d864c1640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d800b0b20 0x7f6d801f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.765+0000 7f6d85cc0640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d800cd0c0 0x7f6d800c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d86cc2640 1 -- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d800b0b20 msgr2=0x7f6d801f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d86cc2640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d800b0b20 0x7f6d801f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d86cc2640 1 -- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d800cd0c0 msgr2=0x7f6d800c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d86cc2640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d800cd0c0 0x7f6d800c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d86cc2640 1 -- 172.21.15.29:0/2696758689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78009400 con 0x7f6d8015d3f0 2023-10-16T15:54:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d864c1640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d800b0b20 0x7f6d801f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d85cc0640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d800cd0c0 0x7f6d800c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:18.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d86cc2640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d800b2a80 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f6d78034040 tx=0x7f6d780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78032630 con 0x7f6d8015d3f0 2023-10-16T15:54:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78032790 con 0x7f6d8015d3f0 2023-10-16T15:54:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78032b20 con 0x7f6d8015d3f0 2023-10-16T15:54:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.766+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d800b1760 con 0x7f6d8015d3f0 2023-10-16T15:54:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.767+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d800b1d20 con 0x7f6d8015d3f0 2023-10-16T15:54:18.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.768+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d800b2520 con 0x7f6d8015d3f0 2023-10-16T15:54:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.769+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6d78032c80 con 0x7f6d8015d3f0 2023-10-16T15:54:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.770+0000 7f6d76ffd640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d6406b0b0 0x7f6d6406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.770+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f6d780abff0 con 0x7f6d8015d3f0 2023-10-16T15:54:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.770+0000 7f6d864c1640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d6406b0b0 0x7f6d6406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.772+0000 7f6d864c1640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d6406b0b0 0x7f6d6406d570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c005c00 tx=0x7f6d7c009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.774+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d780deca0 con 0x7f6d8015d3f0 2023-10-16T15:54:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.972+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d8004e160 con 0x7f6d8015d3f0 2023-10-16T15:54:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.974+0000 7f6d76ffd640 1 -- 172.21.15.29:0/2696758689 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+17971 (secure 0 0 0) 0x7f6d800b2520 con 0x7f6d8015d3f0 2023-10-16T15:54:18.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:18.975 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":234,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:18.510582+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.29:6834/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:18.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d6406b0b0 msgr2=0x7f6d6406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d6406b0b0 0x7f6d6406d570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c005c00 tx=0x7f6d7c009040 comp rx=0 tx=0).stop 2023-10-16T15:54:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 msgr2=0x7f6d800b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d800b2a80 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f6d78034040 tx=0x7f6d780046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 shutdown_connections 2023-10-16T15:54:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d6406b0b0 0x7f6d6406d570 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d8015d3f0 0x7f6d800b2a80 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d800cd0c0 0x7f6d800c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 --2- 172.21.15.29:0/2696758689 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d800b0b20 0x7f6d801f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 >> 172.21.15.29:0/2696758689 conn(0x7f6d800cb410 msgr2=0x7f6d8006cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:18.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 shutdown_connections 2023-10-16T15:54:18.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:18.977+0000 7f6d88725640 1 -- 172.21.15.29:0/2696758689 wait complete. 2023-10-16T15:54:18.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T15:53:39.139862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T15:53:39.864914+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:54:18.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:54:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.207+0000 7f6b45555640 1 Processor -- start 2023-10-16T15:54:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.207+0000 7f6b45555640 1 -- start start 2023-10-16T15:54:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b45555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b4015ad70 0x7f6b40069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b45555640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6b400c9ca0 0x7f6b4006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b45555640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b400ca3b0 0x7f6b400ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b45555640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b400cad80 con 0x7f6b400ca3b0 2023-10-16T15:54:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b45555640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b400cb030 con 0x7f6b400c9ca0 2023-10-16T15:54:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b45555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b400c91c0 con 0x7f6b4015ad70 2023-10-16T15:54:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b3f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b400ca3b0 0x7f6b400ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b3f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b400ca3b0 0x7f6b400ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60108/0 (socket says 172.21.15.29:60108) 2023-10-16T15:54:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.209+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/2798970587 learned_addr learned my addr 172.21.15.29:0/2798970587 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b3effd640 1 --2- 172.21.15.29:0/2798970587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b4015ad70 0x7f6b40069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b37fff640 1 --2- 172.21.15.29:0/2798970587 >> v2:172.21.15.170:3300/0 conn(0x7f6b400c9ca0 0x7f6b4006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/2798970587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b4015ad70 msgr2=0x7f6b40069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b3f7fe640 1 --2- 172.21.15.29:0/2798970587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b4015ad70 0x7f6b40069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/2798970587 >> v2:172.21.15.170:3300/0 conn(0x7f6b400c9ca0 msgr2=0x7f6b4006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b3f7fe640 1 --2- 172.21.15.29:0/2798970587 >> v2:172.21.15.170:3300/0 conn(0x7f6b400c9ca0 0x7f6b4006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.210+0000 7f6b3f7fe640 1 -- 172.21.15.29:0/2798970587 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b400c94a0 con 0x7f6b400ca3b0 2023-10-16T15:54:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.211+0000 7f6b3f7fe640 1 --2- 172.21.15.29:0/2798970587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ca3b0 0x7f6b400ca840 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f6b4006af00 tx=0x7f6b3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e533a265ad0e363 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.211+0000 7f6b3e7fc640 1 -- 172.21.15.29:0/2798970587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3802ca70 con 0x7f6b400ca3b0 2023-10-16T15:54:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.211+0000 7f6b3e7fc640 1 -- 172.21.15.29:0/2798970587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3802cbd0 con 0x7f6b400ca3b0 2023-10-16T15:54:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.211+0000 7f6b3e7fc640 1 -- 172.21.15.29:0/2798970587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b380343f0 con 0x7f6b400ca3b0 2023-10-16T15:54:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 -- 172.21.15.29:0/2798970587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ca3b0 msgr2=0x7f6b400ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 --2- 172.21.15.29:0/2798970587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ca3b0 0x7f6b400ca840 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f6b4006af00 tx=0x7f6b3802bed0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 -- 172.21.15.29:0/2798970587 shutdown_connections 2023-10-16T15:54:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 --2- 172.21.15.29:0/2798970587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400ca3b0 0x7f6b400ca840 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 --2- 172.21.15.29:0/2798970587 >> v2:172.21.15.170:3300/0 conn(0x7f6b400c9ca0 0x7f6b4006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 --2- 172.21.15.29:0/2798970587 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b4015ad70 0x7f6b40069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 -- 172.21.15.29:0/2798970587 >> 172.21.15.29:0/2798970587 conn(0x7f6b400ce9b0 msgr2=0x7f6b400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 -- 172.21.15.29:0/2798970587 shutdown_connections 2023-10-16T15:54:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.212+0000 7f6b45555640 1 -- 172.21.15.29:0/2798970587 wait complete. 2023-10-16T15:54:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.214+0000 7f6b45555640 1 Processor -- start 2023-10-16T15:54:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.214+0000 7f6b45555640 1 -- start start 2023-10-16T15:54:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b45555640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 0x7f6b401f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b45555640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b400ca3b0 0x7f6b400c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b45555640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b4015ad70 0x7f6b400ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b45555640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b401f6a10 con 0x7f6b400c9ca0 2023-10-16T15:54:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b45555640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b400cc610 con 0x7f6b4015ad70 2023-10-16T15:54:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b45555640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b401f2a20 con 0x7f6b400ca3b0 2023-10-16T15:54:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.215+0000 7f6b3effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 0x7f6b401f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 0x7f6b401f64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60110/0 (socket says 172.21.15.29:60110) 2023-10-16T15:54:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 -- 172.21.15.29:0/4161265651 learned_addr learned my addr 172.21.15.29:0/4161265651 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 -- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b400ca3b0 msgr2=0x7f6b400c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b400ca3b0 0x7f6b400c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 -- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b4015ad70 msgr2=0x7f6b400ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b4015ad70 0x7f6b400ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 -- 172.21.15.29:0/4161265651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b38009b30 con 0x7f6b400c9ca0 2023-10-16T15:54:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b3effd640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 0x7f6b401f64d0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c7f0 tx=0x7f6b3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.216+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30004220 con 0x7f6b400c9ca0 2023-10-16T15:54:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.217+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b30004380 con 0x7f6b400c9ca0 2023-10-16T15:54:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.217+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30007840 con 0x7f6b400c9ca0 2023-10-16T15:54:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.217+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b401f2d00 con 0x7f6b400c9ca0 2023-10-16T15:54:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.217+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b401f32c0 con 0x7f6b400c9ca0 2023-10-16T15:54:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.218+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4004e160 con 0x7f6b400c9ca0 2023-10-16T15:54:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.220+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6b30007bb0 con 0x7f6b400c9ca0 2023-10-16T15:54:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.221+0000 7f6b377fe640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6b2006b0b0 0x7f6b2006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.221+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f6b30089eb0 con 0x7f6b400c9ca0 2023-10-16T15:54:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.222+0000 7f6b37fff640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6b2006b0b0 0x7f6b2006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.223+0000 7f6b37fff640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6b2006b0b0 0x7f6b2006d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006fd0 tx=0x7f6b28008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.228+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6b300546a0 con 0x7f6b400c9ca0 2023-10-16T15:54:19.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.420+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b400b44d0 con 0x7f6b400c9ca0 2023-10-16T15:54:19.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.421+0000 7f6b377fe640 1 -- 172.21.15.29:0/4161265651 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v234) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b3001e020 con 0x7f6b400c9ca0 2023-10-16T15:54:19.422 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:54:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6b2006b0b0 msgr2=0x7f6b2006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6b2006b0b0 0x7f6b2006d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006fd0 tx=0x7f6b28008040 comp rx=0 tx=0).stop 2023-10-16T15:54:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 msgr2=0x7f6b401f64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 0x7f6b401f64d0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c7f0 tx=0x7f6b3000ccb0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 shutdown_connections 2023-10-16T15:54:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6b2006b0b0 0x7f6b2006d570 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6b4015ad70 0x7f6b400ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6b400ca3b0 0x7f6b400c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 --2- 172.21.15.29:0/4161265651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b400c9ca0 0x7f6b401f64d0 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.424+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 >> 172.21.15.29:0/4161265651 conn(0x7f6b400ce9b0 msgr2=0x7f6b400b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.425+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 shutdown_connections 2023-10-16T15:54:19.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.425+0000 7f6b45555640 1 -- 172.21.15.29:0/4161265651 wait complete. 2023-10-16T15:54:19.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T15:54:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.679+0000 7f026b49e640 1 Processor -- start 2023-10-16T15:54:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.680+0000 7f026b49e640 1 -- start start 2023-10-16T15:54:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026b49e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02640cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026b49e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f02640d0be0 0x7f02640cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026b49e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026415d150 con 0x7f026415cd70 2023-10-16T15:54:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026b49e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02640d1180 con 0x7f02640d0be0 2023-10-16T15:54:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026b49e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02640d1460 con 0x7f02640b8480 2023-10-16T15:54:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026a49c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f026415cd70 0x7f02640d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55772/0 (socket says 172.21.15.29:55772) 2023-10-16T15:54:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.681+0000 7f026a49c640 1 -- 172.21.15.29:0/3679256665 learned_addr learned my addr 172.21.15.29:0/3679256665 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f0269c9b640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02640cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026ac9d640 1 --2- 172.21.15.29:0/3679256665 >> v2:172.21.15.170:3300/0 conn(0x7f02640d0be0 0x7f02640cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751035300 0 0) 0x7f026415d150 con 0x7f026415cd70 2023-10-16T15:54:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250003520 con 0x7f026415cd70 2023-10-16T15:54:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160507107 0 0) 0x7f0250003520 con 0x7f026415cd70 2023-10-16T15:54:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 msgr2=0x7f02640cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02640cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 >> v2:172.21.15.170:3300/0 conn(0x7f02640d0be0 msgr2=0x7f02640cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 --2- 172.21.15.29:0/3679256665 >> v2:172.21.15.170:3300/0 conn(0x7f02640d0be0 0x7f02640cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02640d1740 con 0x7f026415cd70 2023-10-16T15:54:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.682+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f0260001670 con 0x7f026415cd70 2023-10-16T15:54:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 >> v1:172.21.15.29:6789/0 conn(0x7f026415cd70 legacy=0x7f02640d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0250003970 0x7f0250003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250004d00 0x7f0250005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02640d1180 con 0x7f0250003970 2023-10-16T15:54:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02640d1460 con 0x7f0250004d00 2023-10-16T15:54:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025000b980 con 0x7f02640b8480 2023-10-16T15:54:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026a49c640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0250003970 0x7f0250003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f0269c9b640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.683+0000 7f026ac9d640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250004d00 0x7f0250005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.684+0000 7f026a49c640 1 -- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 msgr2=0x7f02500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.684+0000 7f026a49c640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.684+0000 7f026a49c640 1 -- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250004d00 msgr2=0x7f0250005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.684+0000 7f026a49c640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250004d00 0x7f0250005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.684+0000 7f026a49c640 1 -- 172.21.15.29:0/3679256665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0250003520 con 0x7f0250003970 2023-10-16T15:54:19.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.684+0000 7f0269c9b640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b8480 0x7f02500047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.685+0000 7f026a49c640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0250003970 0x7f0250003d90 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f026000c290 tx=0x7f02500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679bb2509f426f4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.685+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260005ad0 con 0x7f0250003970 2023-10-16T15:54:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.685+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260005c30 con 0x7f0250003970 2023-10-16T15:54:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.685+0000 7f026949a640 1 -- 172.21.15.29:0/3679256665 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026002c8d0 con 0x7f0250003970 2023-10-16T15:54:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026b49e640 1 -- 172.21.15.29:0/3679256665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0250003970 msgr2=0x7f0250003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026b49e640 1 --2- 172.21.15.29:0/3679256665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0250003970 0x7f0250003d90 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f026000c290 tx=0x7f02500057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026ac9d640 1 -- 172.21.15.29:0/3679256665 reap_dead start 2023-10-16T15:54:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026b49e640 1 -- 172.21.15.29:0/3679256665 shutdown_connections 2023-10-16T15:54:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026b49e640 1 -- 172.21.15.29:0/3679256665 >> 172.21.15.29:0/3679256665 conn(0x7f02640cf570 msgr2=0x7f02640cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026b49e640 1 -- 172.21.15.29:0/3679256665 shutdown_connections 2023-10-16T15:54:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.686+0000 7f026b49e640 1 -- 172.21.15.29:0/3679256665 wait complete. 2023-10-16T15:54:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.688+0000 7f026b49e640 1 Processor -- start 2023-10-16T15:54:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.688+0000 7f026b49e640 1 -- start start 2023-10-16T15:54:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.689+0000 7f026b49e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250003970 0x7f02640cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.689+0000 7f026b49e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 0x7f02640b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.689+0000 7f026b49e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b0b00 0x7f02640b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.689+0000 7f026b49e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02640b14c0 con 0x7f02640ca5d0 2023-10-16T15:54:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.689+0000 7f026b49e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02640ccd40 con 0x7f0250003970 2023-10-16T15:54:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.689+0000 7f026b49e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02640cb9c0 con 0x7f02640b0b00 2023-10-16T15:54:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 0x7f02640b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 0x7f02640b7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60130/0 (socket says 172.21.15.29:60130) 2023-10-16T15:54:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 -- 172.21.15.29:0/2030998837 learned_addr learned my addr 172.21.15.29:0/2030998837 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f026ac9d640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b0b00 0x7f02640b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f026a49c640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250003970 0x7f02640cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 -- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b0b00 msgr2=0x7f02640b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b0b00 0x7f02640b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 -- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250003970 msgr2=0x7f02640cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250003970 0x7f02640cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f0269c9b640 1 -- 172.21.15.29:0/2030998837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02640cadb0 con 0x7f02640ca5d0 2023-10-16T15:54:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.690+0000 7f026ac9d640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b0b00 0x7f02640b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.691+0000 7f0269c9b640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 0x7f02640b7300 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f0254011540 tx=0x7f0254011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.691+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254015870 con 0x7f02640ca5d0 2023-10-16T15:54:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.691+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02540159d0 con 0x7f02640ca5d0 2023-10-16T15:54:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.691+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254015d40 con 0x7f02640ca5d0 2023-10-16T15:54:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.691+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02640cb090 con 0x7f02640ca5d0 2023-10-16T15:54:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.691+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02640cb650 con 0x7f02640ca5d0 2023-10-16T15:54:19.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.692+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02640cbb50 con 0x7f02640ca5d0 2023-10-16T15:54:19.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.693+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f025400f460 con 0x7f02640ca5d0 2023-10-16T15:54:19.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.694+0000 7f025affd640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f024806b0b0 0x7f024806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:19.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.694+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f025408a6a0 con 0x7f02640ca5d0 2023-10-16T15:54:19.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.695+0000 7f026a49c640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f024806b0b0 0x7f024806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:19.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.696+0000 7f026a49c640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f024806b0b0 0x7f024806d570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0260034000 tx=0x7f0260007000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:19.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.698+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0254054fa0 con 0x7f02640ca5d0 2023-10-16T15:54:19.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.893+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f026404e9b0 con 0x7f02640ca5d0 2023-10-16T15:54:19.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.895+0000 7f025affd640 1 -- 172.21.15.29:0/2030998837 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v234) v1 ==== 107+0+11 (secure 0 0 0) 0x7f02540585c0 con 0x7f02640ca5d0 2023-10-16T15:54:19.896 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:54:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.897+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f024806b0b0 msgr2=0x7f024806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.897+0000 7f026b49e640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f024806b0b0 0x7f024806d570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0260034000 tx=0x7f0260007000 comp rx=0 tx=0).stop 2023-10-16T15:54:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 msgr2=0x7f02640b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 0x7f02640b7300 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f0254011540 tx=0x7f0254011a00 comp rx=0 tx=0).stop 2023-10-16T15:54:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 shutdown_connections 2023-10-16T15:54:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f024806b0b0 0x7f024806d570 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02640ca5d0 0x7f02640b7300 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f02640b0b00 0x7f02640b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 --2- 172.21.15.29:0/2030998837 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0250003970 0x7f02640cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 >> 172.21.15.29:0/2030998837 conn(0x7f02640cf570 msgr2=0x7f02640d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 shutdown_connections 2023-10-16T15:54:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:19.898+0000 7f026b49e640 1 -- 172.21.15.29:0/2030998837 wait complete. 2023-10-16T15:54:19.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T15:54:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.150+0000 7f9c41602640 1 Processor -- start 2023-10-16T15:54:20.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.150+0000 7f9c41602640 1 -- start start 2023-10-16T15:54:20.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.151+0000 7f9c41602640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c3c159ab0 0x7f9c3c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.151+0000 7f9c41602640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9c3c06c140 0x7f9c3c06c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.151+0000 7f9c41602640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c06cad0 con 0x7f9c3c159e90 2023-10-16T15:54:20.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.151+0000 7f9c41602640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c15d890 con 0x7f9c3c159ab0 2023-10-16T15:54:20.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.151+0000 7f9c41602640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c070360 con 0x7f9c3c06c140 2023-10-16T15:54:20.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c3c159ab0 0x7f9c3c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c3c159ab0 0x7f9c3c0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60146/0 (socket says 172.21.15.29:60146) 2023-10-16T15:54:20.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 -- 172.21.15.29:0/1239097367 learned_addr learned my addr 172.21.15.29:0/1239097367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 -- 172.21.15.29:0/1239097367 >> v2:172.21.15.170:3300/0 conn(0x7f9c3c06c140 msgr2=0x7f9c3c06c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 --2- 172.21.15.29:0/1239097367 >> v2:172.21.15.170:3300/0 conn(0x7f9c3c06c140 0x7f9c3c06c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 -- 172.21.15.29:0/1239097367 >> v1:172.21.15.170:6789/0 conn(0x7f9c3c159e90 legacy=0x7f9c3c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.152+0000 7f9c3affd640 1 -- 172.21.15.29:0/1239097367 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c0cd1f0 con 0x7f9c3c159ab0 2023-10-16T15:54:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.153+0000 7f9c3affd640 1 --2- 172.21.15.29:0/1239097367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c159ab0 0x7f9c3c0ce370 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0adcd0 tx=0x7f9c2c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe06ac54d82b269 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.154+0000 7f9c39ffb640 1 -- 172.21.15.29:0/1239097367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c02c960 con 0x7f9c3c159ab0 2023-10-16T15:54:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.154+0000 7f9c39ffb640 1 -- 172.21.15.29:0/1239097367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c02cac0 con 0x7f9c3c159ab0 2023-10-16T15:54:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.154+0000 7f9c39ffb640 1 -- 172.21.15.29:0/1239097367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c02ce50 con 0x7f9c3c159ab0 2023-10-16T15:54:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.154+0000 7f9c41602640 1 -- 172.21.15.29:0/1239097367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c159ab0 msgr2=0x7f9c3c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.154+0000 7f9c41602640 1 --2- 172.21.15.29:0/1239097367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c159ab0 0x7f9c3c0ce370 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0adcd0 tx=0x7f9c2c02bdc0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.155+0000 7f9c41602640 1 -- 172.21.15.29:0/1239097367 shutdown_connections 2023-10-16T15:54:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.155+0000 7f9c41602640 1 --2- 172.21.15.29:0/1239097367 >> v2:172.21.15.170:3300/0 conn(0x7f9c3c06c140 0x7f9c3c06c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.155+0000 7f9c41602640 1 --2- 172.21.15.29:0/1239097367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c159ab0 0x7f9c3c0ce370 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.155+0000 7f9c41602640 1 -- 172.21.15.29:0/1239097367 >> 172.21.15.29:0/1239097367 conn(0x7f9c3c0ce9b0 msgr2=0x7f9c3c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.155+0000 7f9c41602640 1 -- 172.21.15.29:0/1239097367 shutdown_connections 2023-10-16T15:54:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.155+0000 7f9c41602640 1 -- 172.21.15.29:0/1239097367 wait complete. 2023-10-16T15:54:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.157+0000 7f9c41602640 1 Processor -- start 2023-10-16T15:54:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.157+0000 7f9c41602640 1 -- start start 2023-10-16T15:54:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c41602640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 0x7f9c3c154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c41602640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c3c159ab0 0x7f9c3c14ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c41602640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c3c159e90 0x7f9c3c1674a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c41602640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c154f80 con 0x7f9c3c06c140 2023-10-16T15:54:20.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c41602640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0b2de0 con 0x7f9c3c159ab0 2023-10-16T15:54:20.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c41602640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c166a50 con 0x7f9c3c159e90 2023-10-16T15:54:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 0x7f9c3c154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 0x7f9c3c154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60154/0 (socket says 172.21.15.29:60154) 2023-10-16T15:54:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 -- 172.21.15.29:0/4152836269 learned_addr learned my addr 172.21.15.29:0/4152836269 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 -- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c3c159e90 msgr2=0x7f9c3c1674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c3c159e90 0x7f9c3c1674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 -- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c3c159ab0 msgr2=0x7f9c3c14ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c3c159ab0 0x7f9c3c14ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.158+0000 7f9c3affd640 1 -- 172.21.15.29:0/4152836269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c009b30 con 0x7f9c3c06c140 2023-10-16T15:54:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.159+0000 7f9c3affd640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 0x7f9c3c154a40 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c000f80 tx=0x7f9c2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.159+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c03e450 con 0x7f9c3c06c140 2023-10-16T15:54:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.159+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c03e5f0 con 0x7f9c3c06c140 2023-10-16T15:54:20.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.159+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c03e9a0 con 0x7f9c3c06c140 2023-10-16T15:54:20.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.159+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c14b1b0 con 0x7f9c3c06c140 2023-10-16T15:54:20.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.159+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c0b3310 con 0x7f9c3c06c140 2023-10-16T15:54:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.160+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c04e160 con 0x7f9c3c06c140 2023-10-16T15:54:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.161+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9c2c03eb00 con 0x7f9c3c06c140 2023-10-16T15:54:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.162+0000 7f9c337fe640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9c1c06b0b0 0x7f9c1c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.162+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f9c2c0acbc0 con 0x7f9c3c06c140 2023-10-16T15:54:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.163+0000 7f9c3a7fc640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9c1c06b0b0 0x7f9c1c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.164+0000 7f9c3a7fc640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9c1c06b0b0 0x7f9c1c06d570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c166810 tx=0x7f9c24007990 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:20.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.166+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c2c0774c0 con 0x7f9c3c06c140 2023-10-16T15:54:20.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.359+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c3c06cad0 con 0x7f9c3c06c140 2023-10-16T15:54:20.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.359+0000 7f9c337fe640 1 -- 172.21.15.29:0/4152836269 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v234) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9c3c06cad0 con 0x7f9c3c06c140 2023-10-16T15:54:20.361 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:54:20.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9c1c06b0b0 msgr2=0x7f9c1c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9c1c06b0b0 0x7f9c1c06d570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c166810 tx=0x7f9c24007990 comp rx=0 tx=0).stop 2023-10-16T15:54:20.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 msgr2=0x7f9c3c154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 0x7f9c3c154a40 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c000f80 tx=0x7f9c2c004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 shutdown_connections 2023-10-16T15:54:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9c1c06b0b0 0x7f9c1c06d570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9c3c159e90 0x7f9c3c1674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c3c159ab0 0x7f9c3c14ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 --2- 172.21.15.29:0/4152836269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c3c06c140 0x7f9c3c154a40 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 >> 172.21.15.29:0/4152836269 conn(0x7f9c3c0ce9b0 msgr2=0x7f9c3c0ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:20.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 shutdown_connections 2023-10-16T15:54:20.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.362+0000 7f9c41602640 1 -- 172.21.15.29:0/4152836269 wait complete. 2023-10-16T15:54:20.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:20.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.604+0000 7f601759e640 1 Processor -- start 2023-10-16T15:54:20.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.605+0000 7f601759e640 1 -- start start 2023-10-16T15:54:20.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.606+0000 7f601759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f601815c9f0 0x7f60180c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.606+0000 7f601759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 0x7f60180709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.606+0000 7f601759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6018160a90 con 0x7f6018070f20 2023-10-16T15:54:20.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.606+0000 7f601759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60180cdec0 con 0x7f601815c9f0 2023-10-16T15:54:20.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.606+0000 7f601759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60180cd900 con 0x7f60180c9230 2023-10-16T15:54:20.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.607+0000 7f6016d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6018070f20 0x7f60180ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55778/0 (socket says 172.21.15.29:55778) 2023-10-16T15:54:20.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.607+0000 7f6016d9d640 1 -- 172.21.15.29:0/1020633372 learned_addr learned my addr 172.21.15.29:0/1020633372 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:20.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.607+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585806208 0 0) 0x7f6018160a90 con 0x7f6018070f20 2023-10-16T15:54:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.607+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6004003520 con 0x7f6018070f20 2023-10-16T15:54:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.607+0000 7f601659c640 1 --2- 172.21.15.29:0/1020633372 >> v2:172.21.15.170:3300/0 conn(0x7f601815c9f0 0x7f60180c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.607+0000 7f6015d9b640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 0x7f60180709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716128127 0 0) 0x7f6004003520 con 0x7f6018070f20 2023-10-16T15:54:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 msgr2=0x7f60180709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 0x7f60180709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 >> v2:172.21.15.170:3300/0 conn(0x7f601815c9f0 msgr2=0x7f60180c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:20.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 --2- 172.21.15.29:0/1020633372 >> v2:172.21.15.170:3300/0 conn(0x7f601815c9f0 0x7f60180c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60180cdbe0 con 0x7f6018070f20 2023-10-16T15:54:20.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6010001670 con 0x7f6018070f20 2023-10-16T15:54:20.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 >> v1:172.21.15.29:6789/0 conn(0x7f6018070f20 legacy=0x7f60180ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6004003970 0x7f6004003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 0x7f60040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6004004d00 0x7f6004005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180cd900 con 0x7f6004003970 2023-10-16T15:54:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600400ba30 con 0x7f6004004d00 2023-10-16T15:54:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600400bbf0 con 0x7f60180c9230 2023-10-16T15:54:20.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.608+0000 7f601659c640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6004003970 0x7f6004003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f6015d9b640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 0x7f60040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f6016d9d640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6004004d00 0x7f6004005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f601659c640 1 -- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 msgr2=0x7f60040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:20.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f601659c640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60180c9230 0x7f60040047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f601659c640 1 -- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6004004d00 msgr2=0x7f6004005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:20.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f601659c640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6004004d00 0x7f6004005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.609+0000 7f601659c640 1 -- 172.21.15.29:0/1020633372 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6004003520 con 0x7f6004003970 2023-10-16T15:54:20.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.610+0000 7f601659c640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6004003970 0x7f6004003d90 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f6008007360 tx=0x7f60040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e0bfbaa915e044 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:20.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.610+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600802ccf0 con 0x7f6004003970 2023-10-16T15:54:20.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.610+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600802ce50 con 0x7f6004003970 2023-10-16T15:54:20.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.610+0000 7f601559a640 1 -- 172.21.15.29:0/1020633372 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008034660 con 0x7f6004003970 2023-10-16T15:54:20.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f601759e640 1 -- 172.21.15.29:0/1020633372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6004003970 msgr2=0x7f6004003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f601759e640 1 --2- 172.21.15.29:0/1020633372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6004003970 0x7f6004003d90 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f6008007360 tx=0x7f60040057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f6016d9d640 1 -- 172.21.15.29:0/1020633372 reap_dead start 2023-10-16T15:54:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f601759e640 1 -- 172.21.15.29:0/1020633372 shutdown_connections 2023-10-16T15:54:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f601759e640 1 -- 172.21.15.29:0/1020633372 >> 172.21.15.29:0/1020633372 conn(0x7f60180ce6a0 msgr2=0x7f60180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f601759e640 1 -- 172.21.15.29:0/1020633372 shutdown_connections 2023-10-16T15:54:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.611+0000 7f601759e640 1 -- 172.21.15.29:0/1020633372 wait complete. 2023-10-16T15:54:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.613+0000 7f601759e640 1 Processor -- start 2023-10-16T15:54:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.613+0000 7f601759e640 1 -- start start 2023-10-16T15:54:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f601759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6004003970 0x7f601815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f601759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 0x7f601815f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f601759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60180c4180 0x7f60180c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f601759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180cc7c0 con 0x7f6018149c80 2023-10-16T15:54:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f601759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180c4e60 con 0x7f60180c4180 2023-10-16T15:54:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f601759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60180c5140 con 0x7f6004003970 2023-10-16T15:54:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f6015d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 0x7f601815f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f6015d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 0x7f601815f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60168/0 (socket says 172.21.15.29:60168) 2023-10-16T15:54:20.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.614+0000 7f6015d9b640 1 -- 172.21.15.29:0/2442672208 learned_addr learned my addr 172.21.15.29:0/2442672208 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:20.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f601659c640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6004003970 0x7f601815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6016d9d640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60180c4180 0x7f60180c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6015d9b640 1 -- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6004003970 msgr2=0x7f601815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6015d9b640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6004003970 0x7f601815efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6015d9b640 1 -- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60180c4180 msgr2=0x7f60180c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6015d9b640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60180c4180 0x7f60180c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6015d9b640 1 -- 172.21.15.29:0/2442672208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60180c5420 con 0x7f6018149c80 2023-10-16T15:54:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f6015d9b640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 0x7f601815f4e0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f600000e7e0 tx=0x7f600000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600000c6d0 con 0x7f6018149c80 2023-10-16T15:54:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600000c830 con 0x7f6018149c80 2023-10-16T15:54:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600000cba0 con 0x7f6018149c80 2023-10-16T15:54:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.615+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601814a180 con 0x7f6018149c80 2023-10-16T15:54:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.616+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60180c4b70 con 0x7f6018149c80 2023-10-16T15:54:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.617+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018161670 con 0x7f6018149c80 2023-10-16T15:54:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.618+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f600000cd00 con 0x7f6018149c80 2023-10-16T15:54:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.619+0000 7f600effd640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5ff806b0e0 0x7f5ff806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.619+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f6000089d20 con 0x7f6018149c80 2023-10-16T15:54:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.620+0000 7f601659c640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5ff806b0e0 0x7f5ff806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.621+0000 7f601659c640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5ff806b0e0 0x7f5ff806d5a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6008036000 tx=0x7f6008030000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.623+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f60000545e0 con 0x7f6018149c80 2023-10-16T15:54:20.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.853+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f601804e9b0 con 0x7f6018149c80 2023-10-16T15:54:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.855+0000 7f600effd640 1 -- 172.21.15.29:0/2442672208 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4930 (secure 0 0 0) 0x7f6000021090 con 0x7f6018149c80 2023-10-16T15:54:20.856 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:20.856 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":116,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104},{"gid":11418,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/248379662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":248379662},{"type":"v1","addr":"172.21.15.29:6835","nonce":248379662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:20.857 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 116 2023-10-16T15:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.857+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5ff806b0e0 msgr2=0x7f5ff806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.857+0000 7f601759e640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5ff806b0e0 0x7f5ff806d5a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6008036000 tx=0x7f6008030000 comp rx=0 tx=0).stop 2023-10-16T15:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.857+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 msgr2=0x7f601815f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.857+0000 7f601759e640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 0x7f601815f4e0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f600000e7e0 tx=0x7f600000eca0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 shutdown_connections 2023-10-16T15:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5ff806b0e0 0x7f5ff806d5a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6018149c80 0x7f601815f4e0 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60180c4180 0x7f60180c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 --2- 172.21.15.29:0/2442672208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6004003970 0x7f601815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 >> 172.21.15.29:0/2442672208 conn(0x7f60180ce6a0 msgr2=0x7f60180ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 shutdown_connections 2023-10-16T15:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:20.858+0000 7f601759e640 1 -- 172.21.15.29:0/2442672208 wait complete. 2023-10-16T15:54:20.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:21.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.103+0000 7f22f4802640 1 Processor -- start 2023-10-16T15:54:21.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.103+0000 7f22f4802640 1 -- start start 2023-10-16T15:54:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.104+0000 7f22f4802640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22f0159950 0x7f22f015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.104+0000 7f22f4802640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f015aef0 con 0x7f22f0070450 2023-10-16T15:54:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.104+0000 7f22f4802640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f015d910 con 0x7f22f006c810 2023-10-16T15:54:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22f4802640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f015e490 con 0x7f22f0159950 2023-10-16T15:54:21.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22ef7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22f0159950 0x7f22f015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22e7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f22f0070450 0x7f22f006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55788/0 (socket says 172.21.15.29:55788) 2023-10-16T15:54:21.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22e7fff640 1 -- 172.21.15.29:0/4059958190 learned_addr learned my addr 172.21.15.29:0/4059958190 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4059958190 >> v1:172.21.15.170:6789/0 conn(0x7f22f006c810 legacy=0x7f22f00c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4059958190 >> v1:172.21.15.29:6789/0 conn(0x7f22f0070450 legacy=0x7f22f006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4059958190 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f015dbf0 con 0x7f22f0159950 2023-10-16T15:54:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.105+0000 7f22eeffd640 1 -- 172.21.15.29:0/4059958190 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197134895 0 0) 0x7f22d8001110 con 0x7f22f0070450 2023-10-16T15:54:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.107+0000 7f22ef7fe640 1 --2- 172.21.15.29:0/4059958190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f015a9b0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f22e00097a0 tx=0x7f22e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141cd0620ef6b89c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.107+0000 7f22eeffd640 1 -- 172.21.15.29:0/4059958190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e002e070 con 0x7f22f0159950 2023-10-16T15:54:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.107+0000 7f22eeffd640 1 -- 172.21.15.29:0/4059958190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e00315a0 con 0x7f22f0159950 2023-10-16T15:54:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.107+0000 7f22eeffd640 1 -- 172.21.15.29:0/4059958190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0031930 con 0x7f22f0159950 2023-10-16T15:54:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.107+0000 7f22f4802640 1 -- 172.21.15.29:0/4059958190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 msgr2=0x7f22f015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.107+0000 7f22f4802640 1 --2- 172.21.15.29:0/4059958190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f015a9b0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f22e00097a0 tx=0x7f22e002b840 comp rx=0 tx=0).stop 2023-10-16T15:54:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.108+0000 7f22f4802640 1 -- 172.21.15.29:0/4059958190 shutdown_connections 2023-10-16T15:54:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.108+0000 7f22f4802640 1 --2- 172.21.15.29:0/4059958190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f015a9b0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.108+0000 7f22f4802640 1 -- 172.21.15.29:0/4059958190 >> 172.21.15.29:0/4059958190 conn(0x7f22f00ce6a0 msgr2=0x7f22f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.108+0000 7f22f4802640 1 -- 172.21.15.29:0/4059958190 shutdown_connections 2023-10-16T15:54:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.108+0000 7f22f4802640 1 -- 172.21.15.29:0/4059958190 wait complete. 2023-10-16T15:54:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.109+0000 7f22f4802640 1 Processor -- start 2023-10-16T15:54:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.110+0000 7f22f4802640 1 -- start start 2023-10-16T15:54:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22f4802640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22f006c810 0x7f22f00a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22f4802640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0070450 0x7f22f00c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22f4802640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f01f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22f4802640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00a8cf0 con 0x7f22f0159950 2023-10-16T15:54:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22f4802640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00c4920 con 0x7f22f0070450 2023-10-16T15:54:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22f4802640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00c4c00 con 0x7f22f006c810 2023-10-16T15:54:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f01f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f01f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60172/0 (socket says 172.21.15.29:60172) 2023-10-16T15:54:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22effff640 1 -- 172.21.15.29:0/447688830 learned_addr learned my addr 172.21.15.29:0/447688830 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22ef7fe640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22f006c810 0x7f22f00a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22e7fff640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0070450 0x7f22f00c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22effff640 1 -- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22f006c810 msgr2=0x7f22f00a87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.111+0000 7f22effff640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22f006c810 0x7f22f00a87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22effff640 1 -- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0070450 msgr2=0x7f22f00c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22effff640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0070450 0x7f22f00c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22effff640 1 -- 172.21.15.29:0/447688830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0009400 con 0x7f22f0159950 2023-10-16T15:54:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22e7fff640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0070450 0x7f22f00c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22effff640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f01f18f0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f22e800c780 tx=0x7f22e800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8004b60 con 0x7f22f0159950 2023-10-16T15:54:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800ce90 con 0x7f22f0159950 2023-10-16T15:54:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800a850 con 0x7f22f0159950 2023-10-16T15:54:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f00c4ee0 con 0x7f22f0159950 2023-10-16T15:54:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.112+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f00c4550 con 0x7f22f0159950 2023-10-16T15:54:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.113+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0144f80 con 0x7f22f0159950 2023-10-16T15:54:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.114+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f22e800a9b0 con 0x7f22f0159950 2023-10-16T15:54:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.115+0000 7f22ecff9640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f22d006b0b0 0x7f22d006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.116+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f22e8089ff0 con 0x7f22f0159950 2023-10-16T15:54:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.116+0000 7f22ef7fe640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f22d006b0b0 0x7f22d006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.117+0000 7f22ef7fe640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f22d006b0b0 0x7f22d006d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f22e0036000 tx=0x7f22e0008030 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.120+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f22f0144f80 con 0x7f22f0159950 2023-10-16T15:54:21.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.312+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22f0144f80 con 0x7f22f0159950 2023-10-16T15:54:21.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.314+0000 7f22ecff9640 1 -- 172.21.15.29:0/447688830 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+17971 (secure 0 0 0) 0x7f22f0144f80 con 0x7f22f0159950 2023-10-16T15:54:21.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:21.316 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":234,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:18.510582+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.29:6834/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.317+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f22d006b0b0 msgr2=0x7f22d006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.317+0000 7f22f4802640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f22d006b0b0 0x7f22d006d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f22e0036000 tx=0x7f22e0008030 comp rx=0 tx=0).stop 2023-10-16T15:54:21.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.317+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 msgr2=0x7f22f01f18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.317+0000 7f22f4802640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f01f18f0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f22e800c780 tx=0x7f22e800cc40 comp rx=0 tx=0).stop 2023-10-16T15:54:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 shutdown_connections 2023-10-16T15:54:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f22d006b0b0 0x7f22d006d570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f0159950 0x7f22f01f18f0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f22f0070450 0x7f22f00c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 --2- 172.21.15.29:0/447688830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f22f006c810 0x7f22f00a87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 >> 172.21.15.29:0/447688830 conn(0x7f22f00ce6a0 msgr2=0x7f22f015d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 shutdown_connections 2023-10-16T15:54:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.318+0000 7f22f4802640 1 -- 172.21.15.29:0/447688830 wait complete. 2023-10-16T15:54:21.340 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T15:54:21.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.550+0000 7f63f2e83640 1 Processor -- start 2023-10-16T15:54:21.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.550+0000 7f63f2e83640 1 -- start start 2023-10-16T15:54:21.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.552+0000 7f63f2e83640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f63ec15dde0 0x7f63ec0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.552+0000 7f63f2e83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63ec070f20 0x7f63ec0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.552+0000 7f63f2e83640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ec160a90 con 0x7f63ec15c9f0 2023-10-16T15:54:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.552+0000 7f63f2e83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec071350 con 0x7f63ec070f20 2023-10-16T15:54:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.552+0000 7f63f2e83640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec0cdec0 con 0x7f63ec15dde0 2023-10-16T15:54:21.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63ec070f20 0x7f63ec0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63ec070f20 0x7f63ec0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60184/0 (socket says 172.21.15.29:60184) 2023-10-16T15:54:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 -- 172.21.15.29:0/3522399337 learned_addr learned my addr 172.21.15.29:0/3522399337 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:21.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 -- 172.21.15.29:0/3522399337 >> v2:172.21.15.170:3300/0 conn(0x7f63ec15dde0 msgr2=0x7f63ec0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:21.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 --2- 172.21.15.29:0/3522399337 >> v2:172.21.15.170:3300/0 conn(0x7f63ec15dde0 0x7f63ec0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 -- 172.21.15.29:0/3522399337 >> v1:172.21.15.170:6789/0 conn(0x7f63ec15c9f0 legacy=0x7f63ec0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:21.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.553+0000 7f63f2682640 1 -- 172.21.15.29:0/3522399337 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec0cd900 con 0x7f63ec070f20 2023-10-16T15:54:21.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.554+0000 7f63f2682640 1 --2- 172.21.15.29:0/3522399337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0ac890 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0b1260 tx=0x7f63e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35417feee9670203 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f0e7f640 1 -- 172.21.15.29:0/3522399337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e402ca80 con 0x7f63ec070f20 2023-10-16T15:54:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f0e7f640 1 -- 172.21.15.29:0/3522399337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e402cbe0 con 0x7f63ec070f20 2023-10-16T15:54:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f0e7f640 1 -- 172.21.15.29:0/3522399337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e40343f0 con 0x7f63ec070f20 2023-10-16T15:54:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f2e83640 1 -- 172.21.15.29:0/3522399337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 msgr2=0x7f63ec0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f2e83640 1 --2- 172.21.15.29:0/3522399337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0ac890 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0b1260 tx=0x7f63e402bee0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f2e83640 1 -- 172.21.15.29:0/3522399337 shutdown_connections 2023-10-16T15:54:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.555+0000 7f63f2e83640 1 --2- 172.21.15.29:0/3522399337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0ac890 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.556+0000 7f63f2e83640 1 --2- 172.21.15.29:0/3522399337 >> v2:172.21.15.170:3300/0 conn(0x7f63ec15dde0 0x7f63ec0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.556+0000 7f63f2e83640 1 -- 172.21.15.29:0/3522399337 >> 172.21.15.29:0/3522399337 conn(0x7f63ec0ce6a0 msgr2=0x7f63ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.556+0000 7f63f2e83640 1 -- 172.21.15.29:0/3522399337 shutdown_connections 2023-10-16T15:54:21.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.556+0000 7f63f2e83640 1 -- 172.21.15.29:0/3522399337 wait complete. 2023-10-16T15:54:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.558+0000 7f63f2e83640 1 Processor -- start 2023-10-16T15:54:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.558+0000 7f63f2e83640 1 -- start start 2023-10-16T15:54:21.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f2e83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f2e83640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63ec15c9f0 0x7f63ec154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f2e83640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63ec15dde0 0x7f63ec154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f2e83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0c3350 con 0x7f63ec070f20 2023-10-16T15:54:21.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f2e83640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0c2740 con 0x7f63ec15c9f0 2023-10-16T15:54:21.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f2e83640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0c2a20 con 0x7f63ec15dde0 2023-10-16T15:54:21.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f1e81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f1e81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0c37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60196/0 (socket says 172.21.15.29:60196) 2023-10-16T15:54:21.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.559+0000 7f63f1e81640 1 -- 172.21.15.29:0/2993181586 learned_addr learned my addr 172.21.15.29:0/2993181586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:21.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f2682640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63ec15dde0 0x7f63ec154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1680640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63ec15c9f0 0x7f63ec154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1e81640 1 -- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63ec15dde0 msgr2=0x7f63ec154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1e81640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63ec15dde0 0x7f63ec154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1e81640 1 -- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63ec15c9f0 msgr2=0x7f63ec154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1e81640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63ec15c9f0 0x7f63ec154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1e81640 1 -- 172.21.15.29:0/2993181586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e4009b30 con 0x7f63ec070f20 2023-10-16T15:54:21.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f1e81640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0c37e0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c430 tx=0x7f63e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8011790 con 0x7f63ec070f20 2023-10-16T15:54:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e80118f0 con 0x7f63ec070f20 2023-10-16T15:54:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec0c2d00 con 0x7f63ec070f20 2023-10-16T15:54:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.560+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8011ce0 con 0x7f63ec070f20 2023-10-16T15:54:21.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.561+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ec0c3d20 con 0x7f63ec070f20 2023-10-16T15:54:21.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.562+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ec04e9b0 con 0x7f63ec070f20 2023-10-16T15:54:21.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.563+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f63e801c460 con 0x7f63ec070f20 2023-10-16T15:54:21.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.563+0000 7f63e27fc640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f63d006b0e0 0x7f63d006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:21.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.564+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f63e8089d50 con 0x7f63ec070f20 2023-10-16T15:54:21.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.564+0000 7f63f1680640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f63d006b0e0 0x7f63d006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:21.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.565+0000 7f63f1680640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f63d006b0e0 0x7f63d006d5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0cf480 tx=0x7f63dc008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:21.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.567+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63e8054720 con 0x7f63ec070f20 2023-10-16T15:54:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.804+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63ec06f130 con 0x7f63ec070f20 2023-10-16T15:54:21.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.805+0000 7f63e27fc640 1 -- 172.21.15.29:0/2993181586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4930 (secure 0 0 0) 0x7f63e8057d40 con 0x7f63ec070f20 2023-10-16T15:54:21.806 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:21.807 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":116,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104},{"gid":11418,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/248379662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":248379662},{"type":"v1","addr":"172.21.15.29:6835","nonce":248379662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:21.807 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 116 2023-10-16T15:54:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f63d006b0e0 msgr2=0x7f63d006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f63d006b0e0 0x7f63d006d5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0cf480 tx=0x7f63dc008040 comp rx=0 tx=0).stop 2023-10-16T15:54:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 msgr2=0x7f63ec0c37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0c37e0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c430 tx=0x7f63e800c8f0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 shutdown_connections 2023-10-16T15:54:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f63d006b0e0 0x7f63d006d5a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63ec15dde0 0x7f63ec154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63ec15c9f0 0x7f63ec154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 --2- 172.21.15.29:0/2993181586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63ec070f20 0x7f63ec0c37e0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 >> 172.21.15.29:0/2993181586 conn(0x7f63ec0ce6a0 msgr2=0x7f63ec15a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 shutdown_connections 2023-10-16T15:54:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:21.808+0000 7f63f2e83640 1 -- 172.21.15.29:0/2993181586 wait complete. 2023-10-16T15:54:21.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.053+0000 7f9ab5ec3640 1 Processor -- start 2023-10-16T15:54:22.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.054+0000 7f9ab5ec3640 1 -- start start 2023-10-16T15:54:22.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.055+0000 7f9ab5ec3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9ab015cbe0 0x7f9ab00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.055+0000 7f9ab5ec3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.055+0000 7f9ab5ec3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ab00d1090 0x7f9ab00b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab5ec3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00d15a0 con 0x7f9ab00d1090 2023-10-16T15:54:22.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab5ec3640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00d1880 con 0x7f9ab015cbe0 2023-10-16T15:54:22.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab5ec3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00d1b60 con 0x7f9ab015cfc0 2023-10-16T15:54:22.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab56c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ab00d1090 0x7f9ab00b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab56c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ab00d1090 0x7f9ab00b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60206/0 (socket says 172.21.15.29:60206) 2023-10-16T15:54:22.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab56c2640 1 -- 172.21.15.29:0/1630739341 learned_addr learned my addr 172.21.15.29:0/1630739341 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:22.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9ab4ec1640 1 --2- 172.21.15.29:0/1630739341 >> v2:172.21.15.170:3300/0 conn(0x7f9ab015cbe0 0x7f9ab00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.056+0000 7f9aaffff640 1 --2- 172.21.15.29:0/1630739341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.057+0000 7f9ab56c2640 1 -- 172.21.15.29:0/1630739341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 msgr2=0x7f9ab00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.057+0000 7f9ab56c2640 1 --2- 172.21.15.29:0/1630739341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab00b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.057+0000 7f9ab56c2640 1 -- 172.21.15.29:0/1630739341 >> v2:172.21.15.170:3300/0 conn(0x7f9ab015cbe0 msgr2=0x7f9ab00d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.057+0000 7f9ab56c2640 1 --2- 172.21.15.29:0/1630739341 >> v2:172.21.15.170:3300/0 conn(0x7f9ab015cbe0 0x7f9ab00d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.057+0000 7f9ab56c2640 1 -- 172.21.15.29:0/1630739341 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00d1e40 con 0x7f9ab00d1090 2023-10-16T15:54:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.057+0000 7f9ab4ec1640 1 --2- 172.21.15.29:0/1630739341 >> v2:172.21.15.170:3300/0 conn(0x7f9ab015cbe0 0x7f9ab00d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.058+0000 7f9ab56c2640 1 --2- 172.21.15.29:0/1630739341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab00d1090 0x7f9ab00b4ab0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00b53f0 tx=0x7f9aa802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50e7fd5219eafa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.059+0000 7f9aaf7fe640 1 -- 172.21.15.29:0/1630739341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa802cb00 con 0x7f9ab00d1090 2023-10-16T15:54:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.059+0000 7f9aaf7fe640 1 -- 172.21.15.29:0/1630739341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa802cc60 con 0x7f9ab00d1090 2023-10-16T15:54:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.059+0000 7f9aaf7fe640 1 -- 172.21.15.29:0/1630739341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80343f0 con 0x7f9ab00d1090 2023-10-16T15:54:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.059+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1630739341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab00d1090 msgr2=0x7f9ab00b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.059+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1630739341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab00d1090 0x7f9ab00b4ab0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00b53f0 tx=0x7f9aa802bf60 comp rx=0 tx=0).stop 2023-10-16T15:54:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1630739341 shutdown_connections 2023-10-16T15:54:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1630739341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab00d1090 0x7f9ab00b4ab0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1630739341 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1630739341 >> v2:172.21.15.170:3300/0 conn(0x7f9ab015cbe0 0x7f9ab00d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1630739341 >> 172.21.15.29:0/1630739341 conn(0x7f9ab00cf570 msgr2=0x7f9ab00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1630739341 shutdown_connections 2023-10-16T15:54:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.060+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1630739341 wait complete. 2023-10-16T15:54:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.061+0000 7f9ab5ec3640 1 Processor -- start 2023-10-16T15:54:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.061+0000 7f9ab5ec3640 1 -- start start 2023-10-16T15:54:22.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.062+0000 7f9ab5ec3640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ab00d1090 0x7f9ab0151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.062+0000 7f9ab5ec3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 0x7f9ab00c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.062+0000 7f9ab5ec3640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab0152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.062+0000 7f9ab5ec3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0151e40 con 0x7f9ab015cbe0 2023-10-16T15:54:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.062+0000 7f9ab5ec3640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0152360 con 0x7f9ab00d1090 2023-10-16T15:54:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.062+0000 7f9ab5ec3640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00c7ac0 con 0x7f9ab015cfc0 2023-10-16T15:54:22.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 0x7f9ab00c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 0x7f9ab00c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60222/0 (socket says 172.21.15.29:60222) 2023-10-16T15:54:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 -- 172.21.15.29:0/1371789430 learned_addr learned my addr 172.21.15.29:0/1371789430 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9ab4ec1640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ab00d1090 0x7f9ab0151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9ab56c2640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab0152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 -- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 msgr2=0x7f9ab0152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab0152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 -- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ab00d1090 msgr2=0x7f9ab0151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ab00d1090 0x7f9ab0151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 -- 172.21.15.29:0/1371789430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8009b30 con 0x7f9ab015cbe0 2023-10-16T15:54:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9ab56c2640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab0152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.063+0000 7f9aaffff640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 0x7f9ab00c8e10 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00e8a0 tx=0x7f9a9c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.064+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0094f0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.064+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c00c940 con 0x7f9ab015cbe0 2023-10-16T15:54:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.064+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab00c7da0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.064+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c00ccf0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.064+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab00c8360 con 0x7f9ab015cbe0 2023-10-16T15:54:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.065+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab015e5d0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.066+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9a9c013470 con 0x7f9ab015cbe0 2023-10-16T15:54:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.067+0000 7f9aad7fa640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9a9406b0b0 0x7f9a9406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.067+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f9a9c089e80 con 0x7f9ab015cbe0 2023-10-16T15:54:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.067+0000 7f9ab4ec1640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9a9406b0b0 0x7f9a9406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.069+0000 7f9ab4ec1640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9a9406b0b0 0x7f9a9406d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006fd0 tx=0x7f9aa4008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.071+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ab015e5d0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.293+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ab015e5d0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.294+0000 7f9aad7fa640 1 -- 172.21.15.29:0/1371789430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4930 (secure 0 0 0) 0x7f9ab015e5d0 con 0x7f9ab015cbe0 2023-10-16T15:54:22.295 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:22.296 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":116,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104},{"gid":11418,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/248379662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":248379662},{"type":"v1","addr":"172.21.15.29:6835","nonce":248379662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:53:40.830877+0000","modified":"2023-10-16T15:53:43.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8761":{"gid":8761,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":7,"addr":"172.21.15.170:6837/1674117611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":1674117611},{"type":"v1","addr":"172.21.15.170:6837","nonce":1674117611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T15:54:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 116 2023-10-16T15:54:22.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9a9406b0b0 msgr2=0x7f9a9406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9a9406b0b0 0x7f9a9406d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006fd0 tx=0x7f9aa4008040 comp rx=0 tx=0).stop 2023-10-16T15:54:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 msgr2=0x7f9ab00c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 0x7f9ab00c8e10 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00e8a0 tx=0x7f9a9c00ed60 comp rx=0 tx=0).stop 2023-10-16T15:54:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 shutdown_connections 2023-10-16T15:54:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9a9406b0b0 0x7f9a9406d570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab015cbe0 0x7f9ab00c8e10 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9ab015cfc0 0x7f9ab0152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 --2- 172.21.15.29:0/1371789430 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ab00d1090 0x7f9ab0151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 >> 172.21.15.29:0/1371789430 conn(0x7f9ab00cf570 msgr2=0x7f9ab00b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.297+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 shutdown_connections 2023-10-16T15:54:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.298+0000 7f9ab5ec3640 1 -- 172.21.15.29:0/1371789430 wait complete. 2023-10-16T15:54:22.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:22.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.530+0000 7f0eef59e640 1 Processor -- start 2023-10-16T15:54:22.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.530+0000 7f0eef59e640 1 -- start start 2023-10-16T15:54:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eef59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ef015b320 0x7f0ef00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eef59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef015b9d0 con 0x7f0ef00ad970 2023-10-16T15:54:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eef59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef015bcb0 con 0x7f0ef006c040 2023-10-16T15:54:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eef59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef015b7d0 con 0x7f0ef015b320 2023-10-16T15:54:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eeed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ef015b320 0x7f0ef00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eeed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0ef015b320 0x7f0ef00cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60228/0 (socket says 172.21.15.29:60228) 2023-10-16T15:54:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eeed9d640 1 -- 172.21.15.29:0/2528314791 learned_addr learned my addr 172.21.15.29:0/2528314791 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.532+0000 7f0eedd9b640 1 --1- 172.21.15.29:0/2528314791 >> v1:172.21.15.29:6789/0 conn(0x7f0ef00ad970 0x7f0ef00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55794/0 (socket says 172.21.15.29:55794) 2023-10-16T15:54:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.533+0000 7f0eeed9d640 1 -- 172.21.15.29:0/2528314791 >> v1:172.21.15.170:6789/0 conn(0x7f0ef006c040 legacy=0x7f0ef006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.533+0000 7f0eeed9d640 1 -- 172.21.15.29:0/2528314791 >> v1:172.21.15.29:6789/0 conn(0x7f0ef00ad970 legacy=0x7f0ef00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.533+0000 7f0eeed9d640 1 -- 172.21.15.29:0/2528314791 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef015bf90 con 0x7f0ef015b320 2023-10-16T15:54:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.534+0000 7f0eeed9d640 1 --2- 172.21.15.29:0/2528314791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef00cd180 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009e80 tx=0x7f0ee802bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982e00070751feaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.534+0000 7f0eed59a640 1 -- 172.21.15.29:0/2528314791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee802c8c0 con 0x7f0ef015b320 2023-10-16T15:54:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.534+0000 7f0eed59a640 1 -- 172.21.15.29:0/2528314791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee802ca20 con 0x7f0ef015b320 2023-10-16T15:54:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.534+0000 7f0eed59a640 1 -- 172.21.15.29:0/2528314791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee802cdb0 con 0x7f0ef015b320 2023-10-16T15:54:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 -- 172.21.15.29:0/2528314791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 msgr2=0x7f0ef00cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 --2- 172.21.15.29:0/2528314791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef00cd180 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009e80 tx=0x7f0ee802bda0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 -- 172.21.15.29:0/2528314791 shutdown_connections 2023-10-16T15:54:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 --2- 172.21.15.29:0/2528314791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef00cd180 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 -- 172.21.15.29:0/2528314791 >> 172.21.15.29:0/2528314791 conn(0x7f0ef00ce6a0 msgr2=0x7f0ef00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 -- 172.21.15.29:0/2528314791 shutdown_connections 2023-10-16T15:54:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.535+0000 7f0eef59e640 1 -- 172.21.15.29:0/2528314791 wait complete. 2023-10-16T15:54:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.537+0000 7f0eef59e640 1 Processor -- start 2023-10-16T15:54:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.537+0000 7f0eef59e640 1 -- start start 2023-10-16T15:54:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eef59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ef006c040 0x7f0ef0151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eef59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ef00ad970 0x7f0ef00b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eef59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eef59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00b18f0 con 0x7f0ef015b320 2023-10-16T15:54:22.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eef59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00b42a0 con 0x7f0ef00ad970 2023-10-16T15:54:22.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eef59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00b12c0 con 0x7f0ef006c040 2023-10-16T15:54:22.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eeed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eeed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef015c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60238/0 (socket says 172.21.15.29:60238) 2023-10-16T15:54:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.538+0000 7f0eeed9d640 1 -- 172.21.15.29:0/1616652429 learned_addr learned my addr 172.21.15.29:0/1616652429 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eee59c640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ef006c040 0x7f0ef0151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eedd9b640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ef00ad970 0x7f0ef00b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eeed9d640 1 -- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ef006c040 msgr2=0x7f0ef0151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eeed9d640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ef006c040 0x7f0ef0151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eeed9d640 1 -- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ef00ad970 msgr2=0x7f0ef00b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eeed9d640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ef00ad970 0x7f0ef00b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eeed9d640 1 -- 172.21.15.29:0/1616652429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8009b30 con 0x7f0ef015b320 2023-10-16T15:54:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.539+0000 7f0eeed9d640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef015c1d0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f0ee802c3b0 tx=0x7f0ee8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.540+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8034bb0 con 0x7f0ef015b320 2023-10-16T15:54:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.540+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee8034d30 con 0x7f0ef015b320 2023-10-16T15:54:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.540+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef00b0940 con 0x7f0ef015b320 2023-10-16T15:54:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.540+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8034ea0 con 0x7f0ef015b320 2023-10-16T15:54:22.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.540+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef00b0ee0 con 0x7f0ef015b320 2023-10-16T15:54:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.541+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef00b1a30 con 0x7f0ef015b320 2023-10-16T15:54:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.542+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0ee80406b0 con 0x7f0ef015b320 2023-10-16T15:54:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.543+0000 7f0edeffd640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0ed006b0e0 0x7f0ed006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.543+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f0ee80ace10 con 0x7f0ef015b320 2023-10-16T15:54:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.544+0000 7f0eee59c640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0ed006b0e0 0x7f0ed006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.545+0000 7f0eee59c640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0ed006b0e0 0x7f0ed006d5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0006fd0 tx=0x7f0ee0008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.548+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ef00b1a30 con 0x7f0ef015b320 2023-10-16T15:54:22.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.742+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ef00b1a30 con 0x7f0ef015b320 2023-10-16T15:54:22.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.745+0000 7f0edeffd640 1 -- 172.21.15.29:0/1616652429 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+17971 (secure 0 0 0) 0x7f0ef00b1a30 con 0x7f0ef015b320 2023-10-16T15:54:22.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:22.746 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":234,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:18.510582+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:53:39.139862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:53:39.864914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.29:6834/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.747+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0ed006b0e0 msgr2=0x7f0ed006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.747+0000 7f0eef59e640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0ed006b0e0 0x7f0ed006d5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0006fd0 tx=0x7f0ee0008040 comp rx=0 tx=0).stop 2023-10-16T15:54:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.747+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 msgr2=0x7f0ef015c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.747+0000 7f0eef59e640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef015c1d0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f0ee802c3b0 tx=0x7f0ee8004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 shutdown_connections 2023-10-16T15:54:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f0ed006b0e0 0x7f0ed006d5a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef015b320 0x7f0ef015c1d0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ef00ad970 0x7f0ef00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 --2- 172.21.15.29:0/1616652429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0ef006c040 0x7f0ef0151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 >> 172.21.15.29:0/1616652429 conn(0x7f0ef00ce6a0 msgr2=0x7f0ef00ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 shutdown_connections 2023-10-16T15:54:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.748+0000 7f0eef59e640 1 -- 172.21.15.29:0/1616652429 wait complete. 2023-10-16T15:54:22.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T15:54:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.986+0000 7f196ffff640 1 Processor -- start 2023-10-16T15:54:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.986+0000 7f196ffff640 1 -- start start 2023-10-16T15:54:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.987+0000 7f196ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f197015cd70 0x7f19700d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.987+0000 7f196ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700b8480 0x7f19700cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.987+0000 7f196ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197015d150 con 0x7f19700d0be0 2023-10-16T15:54:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.987+0000 7f196ffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19700d1180 con 0x7f197015cd70 2023-10-16T15:54:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.987+0000 7f196ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19700d1460 con 0x7f19700b8480 2023-10-16T15:54:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f197015cd70 0x7f19700d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f197015cd70 0x7f19700d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60242/0 (socket says 172.21.15.29:60242) 2023-10-16T15:54:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 -- 172.21.15.29:0/386889658 learned_addr learned my addr 172.21.15.29:0/386889658 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 -- 172.21.15.29:0/386889658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700b8480 msgr2=0x7f19700cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 --2- 172.21.15.29:0/386889658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700b8480 0x7f19700cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 -- 172.21.15.29:0/386889658 >> v1:172.21.15.170:6789/0 conn(0x7f19700d0be0 legacy=0x7f19700b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.988+0000 7f196effd640 1 -- 172.21.15.29:0/386889658 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700d1740 con 0x7f197015cd70 2023-10-16T15:54:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.989+0000 7f196effd640 1 --2- 172.21.15.29:0/386889658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f197015cd70 0x7f19700d2220 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f1960009e50 tx=0x7f196002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5786ce274e001 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196dffb640 1 -- 172.21.15.29:0/386889658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196002ca80 con 0x7f197015cd70 2023-10-16T15:54:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196dffb640 1 -- 172.21.15.29:0/386889658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196002cbe0 con 0x7f197015cd70 2023-10-16T15:54:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196dffb640 1 -- 172.21.15.29:0/386889658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600343f0 con 0x7f197015cd70 2023-10-16T15:54:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196ffff640 1 -- 172.21.15.29:0/386889658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f197015cd70 msgr2=0x7f19700d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196ffff640 1 --2- 172.21.15.29:0/386889658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f197015cd70 0x7f19700d2220 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f1960009e50 tx=0x7f196002bee0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196ffff640 1 -- 172.21.15.29:0/386889658 shutdown_connections 2023-10-16T15:54:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196ffff640 1 --2- 172.21.15.29:0/386889658 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700b8480 0x7f19700cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196ffff640 1 --2- 172.21.15.29:0/386889658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f197015cd70 0x7f19700d2220 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.990+0000 7f196ffff640 1 -- 172.21.15.29:0/386889658 >> 172.21.15.29:0/386889658 conn(0x7f19700cf570 msgr2=0x7f19700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.991+0000 7f196ffff640 1 -- 172.21.15.29:0/386889658 shutdown_connections 2023-10-16T15:54:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.991+0000 7f196ffff640 1 -- 172.21.15.29:0/386889658 wait complete. 2023-10-16T15:54:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.992+0000 7f196ffff640 1 Processor -- start 2023-10-16T15:54:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.993+0000 7f196ffff640 1 -- start start 2023-10-16T15:54:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 0x7f1970154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700d0be0 0x7f19701f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f197015cd70 0x7f19700cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970155110 con 0x7f19700b8480 2023-10-16T15:54:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701553f0 con 0x7f197015cd70 2023-10-16T15:54:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19701556d0 con 0x7f19700d0be0 2023-10-16T15:54:22.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 0x7f1970154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 0x7f1970154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60252/0 (socket says 172.21.15.29:60252) 2023-10-16T15:54:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 -- 172.21.15.29:0/2000189674 learned_addr learned my addr 172.21.15.29:0/2000189674 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 -- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700d0be0 msgr2=0x7f19701f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700d0be0 0x7f19701f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 -- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f197015cd70 msgr2=0x7f19700cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f197015cd70 0x7f19700cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.994+0000 7f196effd640 1 -- 172.21.15.29:0/2000189674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960009b30 con 0x7f19700b8480 2023-10-16T15:54:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.995+0000 7f196effd640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 0x7f1970154a70 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f1960009e50 tx=0x7f1960004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.995+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196003e450 con 0x7f19700b8480 2023-10-16T15:54:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.995+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196003e5f0 con 0x7f19700b8480 2023-10-16T15:54:22.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.995+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196003e9a0 con 0x7f19700b8480 2023-10-16T15:54:23.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.995+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19701559b0 con 0x7f19700b8480 2023-10-16T15:54:23.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.995+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19701f6b00 con 0x7f19700b8480 2023-10-16T15:54:23.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.997+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970155810 con 0x7f19700b8480 2023-10-16T15:54:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.997+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f196002cd50 con 0x7f19700b8480 2023-10-16T15:54:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.999+0000 7f19677fe640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f195006b0b0 0x7f195006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:22.999+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6652+0+0 (secure 0 0 0) 0x7f19600aca90 con 0x7f19700b8480 2023-10-16T15:54:23.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.000+0000 7f196e7fc640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f195006b0b0 0x7f195006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.001+0000 7f196e7fc640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f195006b0b0 0x7f195006d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1970145b40 tx=0x7f1958006bf0 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.005+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1970155810 con 0x7f19700b8480 2023-10-16T15:54:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.236+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f1970155810 con 0x7f19700b8480 2023-10-16T15:54:23.239 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:54:23.238+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:54:23.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.242+0000 7f19677fe640 1 -- 172.21.15.29:0/2000189674 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v117) v1 ==== 167+0+0 (secure 0 0 0) 0x7f1970155810 con 0x7f19700b8480 2023-10-16T15:54:23.243 INFO:teuthology.orchestra.run.smithi029.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:54:23.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.244+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f195006b0b0 msgr2=0x7f195006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.244+0000 7f196ffff640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f195006b0b0 0x7f195006d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1970145b40 tx=0x7f1958006bf0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.244+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 msgr2=0x7f1970154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.244+0000 7f196ffff640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 0x7f1970154a70 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f1960009e50 tx=0x7f1960004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 shutdown_connections 2023-10-16T15:54:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f195006b0b0 0x7f195006d570 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f197015cd70 0x7f19700cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f19700d0be0 0x7f19701f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 --2- 172.21.15.29:0/2000189674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19700b8480 0x7f1970154a70 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 >> 172.21.15.29:0/2000189674 conn(0x7f19700cf570 msgr2=0x7f197006bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 shutdown_connections 2023-10-16T15:54:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.245+0000 7f196ffff640 1 -- 172.21.15.29:0/2000189674 wait complete. 2023-10-16T15:54:23.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T15:54:23.269 INFO:tasks.ceph.mds.b.smithi170.stderr: -2185> 2023-10-16T15:53:43.156+0000 7f1de6f29640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:54:23.343 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:54:23.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.487+0000 7f5dca34e640 1 Processor -- start 2023-10-16T15:54:23.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.487+0000 7f5dca34e640 1 -- start start 2023-10-16T15:54:23.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dca34e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5dc406c040 0x7f5dc406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dca34e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc40ad970 0x7f5dc415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dca34e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5dc415ebc0 0x7f5dc415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dca34e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc406c960 con 0x7f5dc415ebc0 2023-10-16T15:54:23.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dca34e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc415b9d0 con 0x7f5dc406c040 2023-10-16T15:54:23.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dca34e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc415bcb0 con 0x7f5dc40ad970 2023-10-16T15:54:23.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dc9b4d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5dc415ebc0 0x7f5dc415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dc9b4d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5dc415ebc0 0x7f5dc415f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60266/0 (socket says 172.21.15.29:60266) 2023-10-16T15:54:23.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dc9b4d640 1 -- 172.21.15.29:0/3664285694 learned_addr learned my addr 172.21.15.29:0/3664285694 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:23.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.489+0000 7f5dc8b4b640 1 --2- 172.21.15.29:0/3664285694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc40ad970 0x7f5dc415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.490+0000 7f5dc934c640 1 --2- 172.21.15.29:0/3664285694 >> v2:172.21.15.170:3300/0 conn(0x7f5dc406c040 0x7f5dc406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.490+0000 7f5dc9b4d640 1 -- 172.21.15.29:0/3664285694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc40ad970 msgr2=0x7f5dc415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:23.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.490+0000 7f5dc9b4d640 1 --2- 172.21.15.29:0/3664285694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc40ad970 0x7f5dc415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.490+0000 7f5dc9b4d640 1 -- 172.21.15.29:0/3664285694 >> v2:172.21.15.170:3300/0 conn(0x7f5dc406c040 msgr2=0x7f5dc406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:23.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.490+0000 7f5dc9b4d640 1 --2- 172.21.15.29:0/3664285694 >> v2:172.21.15.170:3300/0 conn(0x7f5dc406c040 0x7f5dc406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.490+0000 7f5dc9b4d640 1 -- 172.21.15.29:0/3664285694 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc415bf90 con 0x7f5dc415ebc0 2023-10-16T15:54:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.491+0000 7f5dc9b4d640 1 --2- 172.21.15.29:0/3664285694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc415ebc0 0x7f5dc415f050 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009e50 tx=0x7f5dbc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a46b1dfa59a446 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.491+0000 7f5dbbfff640 1 -- 172.21.15.29:0/3664285694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc02ca80 con 0x7f5dc415ebc0 2023-10-16T15:54:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.491+0000 7f5dbbfff640 1 -- 172.21.15.29:0/3664285694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc02cbe0 con 0x7f5dc415ebc0 2023-10-16T15:54:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.491+0000 7f5dbbfff640 1 -- 172.21.15.29:0/3664285694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0343f0 con 0x7f5dc415ebc0 2023-10-16T15:54:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 -- 172.21.15.29:0/3664285694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc415ebc0 msgr2=0x7f5dc415f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 --2- 172.21.15.29:0/3664285694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc415ebc0 0x7f5dc415f050 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009e50 tx=0x7f5dbc02bee0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 -- 172.21.15.29:0/3664285694 shutdown_connections 2023-10-16T15:54:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 --2- 172.21.15.29:0/3664285694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc415ebc0 0x7f5dc415f050 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 --2- 172.21.15.29:0/3664285694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc40ad970 0x7f5dc415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 --2- 172.21.15.29:0/3664285694 >> v2:172.21.15.170:3300/0 conn(0x7f5dc406c040 0x7f5dc406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 -- 172.21.15.29:0/3664285694 >> 172.21.15.29:0/3664285694 conn(0x7f5dc40ce6a0 msgr2=0x7f5dc40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 -- 172.21.15.29:0/3664285694 shutdown_connections 2023-10-16T15:54:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.492+0000 7f5dca34e640 1 -- 172.21.15.29:0/3664285694 wait complete. 2023-10-16T15:54:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.494+0000 7f5dca34e640 1 Processor -- start 2023-10-16T15:54:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.494+0000 7f5dca34e640 1 -- start start 2023-10-16T15:54:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dca34e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 0x7f5dc40c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dca34e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dc40ad970 0x7f5dc4150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dca34e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc415ebc0 0x7f5dc40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dca34e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc40c67c0 con 0x7f5dc406c040 2023-10-16T15:54:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dca34e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc40c8570 con 0x7f5dc40ad970 2023-10-16T15:54:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dca34e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc40c69d0 con 0x7f5dc415ebc0 2023-10-16T15:54:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 0x7f5dc40c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 0x7f5dc40c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60274/0 (socket says 172.21.15.29:60274) 2023-10-16T15:54:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 -- 172.21.15.29:0/2925424778 learned_addr learned my addr 172.21.15.29:0/2925424778 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc9b4d640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc415ebc0 0x7f5dc40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc8b4b640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dc40ad970 0x7f5dc4150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 -- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc415ebc0 msgr2=0x7f5dc40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc415ebc0 0x7f5dc40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 -- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dc40ad970 msgr2=0x7f5dc4150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dc40ad970 0x7f5dc4150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc934c640 1 -- 172.21.15.29:0/2925424778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc009b30 con 0x7f5dc406c040 2023-10-16T15:54:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.495+0000 7f5dc8b4b640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dc40ad970 0x7f5dc4150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.496+0000 7f5dc934c640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 0x7f5dc40c8030 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c570 tx=0x7f5dc000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.496+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0011800 con 0x7f5dc406c040 2023-10-16T15:54:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.496+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0011960 con 0x7f5dc406c040 2023-10-16T15:54:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.496+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0011d10 con 0x7f5dc406c040 2023-10-16T15:54:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.496+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc40c6cb0 con 0x7f5dc406c040 2023-10-16T15:54:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.496+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc40c7270 con 0x7f5dc406c040 2023-10-16T15:54:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.497+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc406b330 con 0x7f5dc406c040 2023-10-16T15:54:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.498+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5dc0018360 con 0x7f5dc406c040 2023-10-16T15:54:23.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.499+0000 7f5db9ffb640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5da806b0b0 0x7f5da806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:23.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.499+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6738+0+0 (secure 0 0 0) 0x7f5dc0089fe0 con 0x7f5dc406c040 2023-10-16T15:54:23.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.500+0000 7f5dc8b4b640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5da806b0b0 0x7f5da806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.501+0000 7f5dc8b4b640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5da806b0b0 0x7f5da806d570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5db4006fd0 tx=0x7f5db4008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.503+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5dc00547b0 con 0x7f5dc406c040 2023-10-16T15:54:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:23.730+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5dc404e9b0 con 0x7f5dc406c040 2023-10-16T15:54:24.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.252+0000 7f5db9ffb640 1 -- 172.21.15.29:0/2925424778 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v118) v1 ==== 107+0+0 (secure 0 0 0) 0x7f5dc0057dd0 con 0x7f5dc406c040 2023-10-16T15:54:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5da806b0b0 msgr2=0x7f5da806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5da806b0b0 0x7f5da806d570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5db4006fd0 tx=0x7f5db4008040 comp rx=0 tx=0).stop 2023-10-16T15:54:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 msgr2=0x7f5dc40c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 0x7f5dc40c8030 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f5dc000c570 tx=0x7f5dc000ca30 comp rx=0 tx=0).stop 2023-10-16T15:54:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 shutdown_connections 2023-10-16T15:54:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f5da806b0b0 0x7f5da806d570 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5dc415ebc0 0x7f5dc40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5dc40ad970 0x7f5dc4150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 --2- 172.21.15.29:0/2925424778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5dc406c040 0x7f5dc40c8030 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.256+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 >> 172.21.15.29:0/2925424778 conn(0x7f5dc40ce6a0 msgr2=0x7f5dc40c9240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.257+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 shutdown_connections 2023-10-16T15:54:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.257+0000 7f5dca34e640 1 -- 172.21.15.29:0/2925424778 wait complete. 2023-10-16T15:54:24.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T15:54:24.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.515+0000 7f661b59e640 1 Processor -- start 2023-10-16T15:54:24.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.515+0000 7f661b59e640 1 -- start start 2023-10-16T15:54:24.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.516+0000 7f661b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f661c0ad7a0 0x7f661c070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.516+0000 7f661b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0ae1a0 con 0x7f661c160c80 2023-10-16T15:54:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.516+0000 7f661b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0ae480 con 0x7f661c0adc30 2023-10-16T15:54:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.516+0000 7f661b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c0ae760 con 0x7f661c0ad7a0 2023-10-16T15:54:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f661a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f661c160c80 0x7f661c161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50248/0 (socket says 172.21.15.29:50248) 2023-10-16T15:54:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f661a59c640 1 -- 172.21.15.29:0/1603422600 learned_addr learned my addr 172.21.15.29:0/1603422600 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:24.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f6619d9b640 1 --2- 172.21.15.29:0/1603422600 >> v2:172.21.15.29:3300/0 conn(0x7f661c0ad7a0 0x7f661c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:24.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f661959a640 1 -- 172.21.15.29:0/1603422600 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173503017 0 0) 0x7f661c0ae1a0 con 0x7f661c160c80 2023-10-16T15:54:24.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f661959a640 1 -- 172.21.15.29:0/1603422600 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6608003520 con 0x7f661c160c80 2023-10-16T15:54:24.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f6619d9b640 1 -- 172.21.15.29:0/1603422600 >> v1:172.21.15.170:6789/0 conn(0x7f661c0adc30 legacy=0x7f661c140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:24.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f6619d9b640 1 -- 172.21.15.29:0/1603422600 >> v1:172.21.15.29:6789/0 conn(0x7f661c160c80 legacy=0x7f661c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.517+0000 7f6619d9b640 1 -- 172.21.15.29:0/1603422600 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0aea40 con 0x7f661c0ad7a0 2023-10-16T15:54:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.518+0000 7f6619d9b640 1 --2- 172.21.15.29:0/1603422600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c0ad7a0 0x7f661c070240 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f6604009d90 tx=0x7f660402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fbc5f167132a14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.519+0000 7f661959a640 1 -- 172.21.15.29:0/1603422600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660402cd00 con 0x7f661c0ad7a0 2023-10-16T15:54:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.519+0000 7f661959a640 1 -- 172.21.15.29:0/1603422600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660402ce60 con 0x7f661c0ad7a0 2023-10-16T15:54:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.519+0000 7f661959a640 1 -- 172.21.15.29:0/1603422600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604034660 con 0x7f661c0ad7a0 2023-10-16T15:54:24.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.519+0000 7f661b59e640 1 -- 172.21.15.29:0/1603422600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c0ad7a0 msgr2=0x7f661c070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:24.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.519+0000 7f661b59e640 1 --2- 172.21.15.29:0/1603422600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c0ad7a0 0x7f661c070240 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f6604009d90 tx=0x7f660402bf50 comp rx=0 tx=0).stop 2023-10-16T15:54:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.520+0000 7f661b59e640 1 -- 172.21.15.29:0/1603422600 shutdown_connections 2023-10-16T15:54:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.520+0000 7f661b59e640 1 --2- 172.21.15.29:0/1603422600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c0ad7a0 0x7f661c070240 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.520+0000 7f661b59e640 1 -- 172.21.15.29:0/1603422600 >> 172.21.15.29:0/1603422600 conn(0x7f661c0cc230 msgr2=0x7f661c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.520+0000 7f661b59e640 1 -- 172.21.15.29:0/1603422600 shutdown_connections 2023-10-16T15:54:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.520+0000 7f661b59e640 1 -- 172.21.15.29:0/1603422600 wait complete. 2023-10-16T15:54:24.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.521+0000 7f661b59e640 1 Processor -- start 2023-10-16T15:54:24.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.522+0000 7f661b59e640 1 -- start start 2023-10-16T15:54:24.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f661c0ad7a0 0x7f661c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:24.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f661c0adc30 0x7f661c144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:24.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 0x7f661c0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:24.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c06dc40 con 0x7f661c160c80 2023-10-16T15:54:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c06d310 con 0x7f661c0adc30 2023-10-16T15:54:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c06d5f0 con 0x7f661c0ad7a0 2023-10-16T15:54:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 0x7f661c0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 0x7f661c0ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49174/0 (socket says 172.21.15.29:49174) 2023-10-16T15:54:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661ad9d640 1 -- 172.21.15.29:0/2192541986 learned_addr learned my addr 172.21.15.29:0/2192541986 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f661a59c640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f661c0ad7a0 0x7f661c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.523+0000 7f6619d9b640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f661c0adc30 0x7f661c144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661ad9d640 1 -- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f661c0ad7a0 msgr2=0x7f661c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661ad9d640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f661c0ad7a0 0x7f661c159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661ad9d640 1 -- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f661c0adc30 msgr2=0x7f661c144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661ad9d640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f661c0adc30 0x7f661c144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661ad9d640 1 -- 172.21.15.29:0/2192541986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604009a70 con 0x7f661c160c80 2023-10-16T15:54:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661ad9d640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 0x7f661c0ce7e0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f6614001eb0 tx=0x7f661400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66140052d0 con 0x7f661c160c80 2023-10-16T15:54:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614005430 con 0x7f661c160c80 2023-10-16T15:54:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614003810 con 0x7f661c160c80 2023-10-16T15:54:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.524+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c06d8d0 con 0x7f661c160c80 2023-10-16T15:54:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.525+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c15a3e0 con 0x7f661c160c80 2023-10-16T15:54:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.526+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c06ddb0 con 0x7f661c160c80 2023-10-16T15:54:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.527+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6614003970 con 0x7f661c160c80 2023-10-16T15:54:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.528+0000 7f6612ffd640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f65fc06b0b0 0x7f65fc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.528+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6738+0+0 (secure 0 0 0) 0x7f66140893f0 con 0x7f661c160c80 2023-10-16T15:54:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.528+0000 7f661a59c640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f65fc06b0b0 0x7f65fc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.529+0000 7f661a59c640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f65fc06b0b0 0x7f65fc06d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f660c0011e0 tx=0x7f660c009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.532+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6614053d60 con 0x7f661c160c80 2023-10-16T15:54:24.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:24.723+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f661c04e9b0 con 0x7f661c160c80 2023-10-16T15:54:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.570+0000 7f6612ffd640 1 -- 172.21.15.29:0/2192541986 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v236) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6614057380 con 0x7f661c160c80 2023-10-16T15:54:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' removed 2023-10-16T15:54:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f65fc06b0b0 msgr2=0x7f65fc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f65fc06b0b0 0x7f65fc06d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f660c0011e0 tx=0x7f660c009040 comp rx=0 tx=0).stop 2023-10-16T15:54:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 msgr2=0x7f661c0ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 0x7f661c0ce7e0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f6614001eb0 tx=0x7f661400c890 comp rx=0 tx=0).stop 2023-10-16T15:54:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 shutdown_connections 2023-10-16T15:54:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f65fc06b0b0 0x7f65fc06d570 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f661c160c80 0x7f661c0ce7e0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f661c0adc30 0x7f661c144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 --2- 172.21.15.29:0/2192541986 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f661c0ad7a0 0x7f661c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.573+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 >> 172.21.15.29:0/2192541986 conn(0x7f661c0cc230 msgr2=0x7f661c15af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:25.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.574+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 shutdown_connections 2023-10-16T15:54:25.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.574+0000 7f661b59e640 1 -- 172.21.15.29:0/2192541986 wait complete. 2023-10-16T15:54:25.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T15:54:25.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.828+0000 7ffbc5905640 1 Processor -- start 2023-10-16T15:54:25.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.828+0000 7ffbc5905640 1 -- start start 2023-10-16T15:54:25.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.829+0000 7ffbc5905640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc00b0360 0x7ffbc00b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:25.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.829+0000 7ffbc5905640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffbc00ca510 0x7ffbc00ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:25.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.829+0000 7ffbc5905640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc00b0d00 con 0x7ffbc015ac10 2023-10-16T15:54:25.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.829+0000 7ffbc5905640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc015f800 con 0x7ffbc00ca510 2023-10-16T15:54:25.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.829+0000 7ffbc5905640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc006ce40 con 0x7ffbc00b0360 2023-10-16T15:54:25.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.830+0000 7ffbbf7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffbc00ca510 0x7ffbc00ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:25.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.830+0000 7ffbbe7fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc00b0360 0x7ffbc00b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:25.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.830+0000 7ffbbf7fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ffbc00ca510 0x7ffbc00ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:57332/0 (socket says 172.21.15.29:57332) 2023-10-16T15:54:25.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.830+0000 7ffbbf7fe640 1 -- 172.21.15.29:0/2878117008 learned_addr learned my addr 172.21.15.29:0/2878117008 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:25.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.831+0000 7ffbbf7fe640 1 -- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc00b0360 msgr2=0x7ffbc00b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:25.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.831+0000 7ffbbf7fe640 1 --2- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc00b0360 0x7ffbc00b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.831+0000 7ffbbf7fe640 1 -- 172.21.15.29:0/2878117008 >> v1:172.21.15.170:6789/0 conn(0x7ffbc015ac10 legacy=0x7ffbc006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:25.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.831+0000 7ffbbf7fe640 1 -- 172.21.15.29:0/2878117008 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc015b310 con 0x7ffbc00ca510 2023-10-16T15:54:25.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.833+0000 7ffbbf7fe640 1 --2- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00ca940 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009e50 tx=0x7ffbb802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f149c78700718ba9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:25.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.833+0000 7ffbbdffb640 1 -- 172.21.15.29:0/2878117008 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb802ca70 con 0x7ffbc00ca510 2023-10-16T15:54:25.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbbdffb640 1 -- 172.21.15.29:0/2878117008 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb802cbd0 con 0x7ffbc00ca510 2023-10-16T15:54:25.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbbdffb640 1 -- 172.21.15.29:0/2878117008 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80343f0 con 0x7ffbc00ca510 2023-10-16T15:54:25.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 -- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 msgr2=0x7ffbc00ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:25.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00ca940 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009e50 tx=0x7ffbb802bed0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 -- 172.21.15.29:0/2878117008 shutdown_connections 2023-10-16T15:54:25.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00ca940 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2878117008 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc00b0360 0x7ffbc00b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 -- 172.21.15.29:0/2878117008 >> 172.21.15.29:0/2878117008 conn(0x7ffbc00cb360 msgr2=0x7ffbc00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:25.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 -- 172.21.15.29:0/2878117008 shutdown_connections 2023-10-16T15:54:25.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.834+0000 7ffbc5905640 1 -- 172.21.15.29:0/2878117008 wait complete. 2023-10-16T15:54:25.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.836+0000 7ffbc5905640 1 Processor -- start 2023-10-16T15:54:25.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.836+0000 7ffbc5905640 1 -- start start 2023-10-16T15:54:25.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbc5905640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 0x7ffbc00c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:25.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbc5905640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:25.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbc5905640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc015ac10 0x7ffbc00c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:25.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbc5905640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc014d8b0 con 0x7ffbc00b0360 2023-10-16T15:54:25.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbc5905640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0158a30 con 0x7ffbc00ca510 2023-10-16T15:54:25.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbc5905640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0158d10 con 0x7ffbc015ac10 2023-10-16T15:54:25.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbbeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 0x7ffbc00c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:25.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbbeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 0x7ffbc00c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49188/0 (socket says 172.21.15.29:49188) 2023-10-16T15:54:25.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.837+0000 7ffbbeffd640 1 -- 172.21.15.29:0/2265938571 learned_addr learned my addr 172.21.15.29:0/2265938571 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:25.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbf7fe640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc015ac10 0x7ffbc00c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:25.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbe7fc640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:25.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbeffd640 1 -- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc015ac10 msgr2=0x7ffbc00c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbeffd640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc015ac10 0x7ffbc00c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbeffd640 1 -- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 msgr2=0x7ffbc00c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbeffd640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbeffd640 1 -- 172.21.15.29:0/2265938571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8009b30 con 0x7ffbc00b0360 2023-10-16T15:54:25.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.838+0000 7ffbbeffd640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 0x7ffbc00c7520 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c400 tx=0x7ffbb000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.839+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00052d0 con 0x7ffbc00b0360 2023-10-16T15:54:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.839+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb0005450 con 0x7ffbc00b0360 2023-10-16T15:54:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.839+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0007260 con 0x7ffbc00b0360 2023-10-16T15:54:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.839+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc0158ff0 con 0x7ffbc00b0360 2023-10-16T15:54:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.839+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc01595b0 con 0x7ffbc00b0360 2023-10-16T15:54:25.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.840+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc01f02d0 con 0x7ffbc00b0360 2023-10-16T15:54:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.841+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ffbb00073c0 con 0x7ffbc00b0360 2023-10-16T15:54:25.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.842+0000 7ffba7fff640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ffbac06b0b0 0x7ffbac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.842+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6294+0+0 (secure 0 0 0) 0x7ffbb0089060 con 0x7ffbc00b0360 2023-10-16T15:54:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.843+0000 7ffbbe7fc640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ffbac06b0b0 0x7ffbac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:25.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.844+0000 7ffbbe7fc640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ffbac06b0b0 0x7ffbac06d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00afa70 tx=0x7ffbb4008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:25.846+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffbb00bbca0 con 0x7ffbc00b0360 2023-10-16T15:54:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.038+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffbc006f610 con 0x7ffbc00b0360 2023-10-16T15:54:26.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.581+0000 7ffba7fff640 1 -- 172.21.15.29:0/2265938571 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v237) v1 ==== 178+0+0 (secure 0 0 0) 0x7ffbb0057620 con 0x7ffbc00b0360 2023-10-16T15:54:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' removed 2023-10-16T15:54:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.587+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ffbac06b0b0 msgr2=0x7ffbac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.587+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ffbac06b0b0 0x7ffbac06d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00afa70 tx=0x7ffbb4008040 comp rx=0 tx=0).stop 2023-10-16T15:54:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.587+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 msgr2=0x7ffbc00c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.587+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 0x7ffbc00c7520 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c400 tx=0x7ffbb000c8c0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.588+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 shutdown_connections 2023-10-16T15:54:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.588+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ffbac06b0b0 0x7ffbac06d570 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00afa70 tx=0x7ffbb4008040 comp rx=0 tx=0).stop 2023-10-16T15:54:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.588+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ffbc015ac10 0x7ffbc00c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.588+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffbc00ca510 0x7ffbc00c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.588+0000 7ffbc5905640 1 --2- 172.21.15.29:0/2265938571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbc00b0360 0x7ffbc00c7520 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.588+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 >> 172.21.15.29:0/2265938571 conn(0x7ffbc00cb360 msgr2=0x7ffbc00ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.589+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 shutdown_connections 2023-10-16T15:54:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.589+0000 7ffbc5905640 1 -- 172.21.15.29:0/2265938571 wait complete. 2023-10-16T15:54:26.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:54:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.834+0000 7f7bc501a640 1 Processor -- start 2023-10-16T15:54:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.834+0000 7f7bc501a640 1 -- start start 2023-10-16T15:54:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.836+0000 7f7bc501a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 0x7f7bc00acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.836+0000 7f7bc501a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc0160a90 con 0x7f7bc015c9f0 2023-10-16T15:54:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.836+0000 7f7bc501a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc00cd900 con 0x7f7bc015dde0 2023-10-16T15:54:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.836+0000 7f7bc501a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00cdec0 con 0x7f7bc00ac890 2023-10-16T15:54:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.836+0000 7f7bbffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7bc015c9f0 0x7f7bc00c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50254/0 (socket says 172.21.15.29:50254) 2023-10-16T15:54:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.836+0000 7f7bbffff640 1 -- 172.21.15.29:0/1022788638 learned_addr learned my addr 172.21.15.29:0/1022788638 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278924529 0 0) 0x7f7bc0160a90 con 0x7f7bc015c9f0 2023-10-16T15:54:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba8003520 con 0x7f7bc015c9f0 2023-10-16T15:54:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bc4819640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 0x7f7bc00acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390852695 0 0) 0x7f7ba8003520 con 0x7f7bc015c9f0 2023-10-16T15:54:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 msgr2=0x7f7bc00acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 0x7f7bc00acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 >> v1:172.21.15.170:6789/0 conn(0x7f7bc015dde0 legacy=0x7f7bc00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc00cdbe0 con 0x7f7bc015c9f0 2023-10-16T15:54:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7bb0001670 con 0x7f7bc015c9f0 2023-10-16T15:54:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 >> v1:172.21.15.29:6789/0 conn(0x7f7bc015c9f0 legacy=0x7f7bc00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ba80039a0 0x7f7ba8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ba80047c0 0x7f7ba8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 0x7f7ba80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba800ba80 con 0x7f7ba80047c0 2023-10-16T15:54:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba800bbf0 con 0x7f7ba80039a0 2023-10-16T15:54:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba800bdb0 con 0x7f7bc00ac890 2023-10-16T15:54:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.837+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ba80047c0 0x7f7ba8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbffff640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ba80039a0 0x7f7ba8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 msgr2=0x7f7ba80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7bc00ac890 0x7f7ba80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ba80039a0 msgr2=0x7f7ba8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ba80039a0 0x7f7ba8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1022788638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8003520 con 0x7f7ba80047c0 2023-10-16T15:54:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.838+0000 7f7bbffff640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ba80039a0 0x7f7ba8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:26.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.839+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ba80047c0 0x7f7ba8004c40 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4001750 tx=0x7f7ba80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0762f923c84dc85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:26.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.839+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40052d0 con 0x7f7ba80047c0 2023-10-16T15:54:26.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.839+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb402e050 con 0x7f7ba80047c0 2023-10-16T15:54:26.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.839+0000 7f7bbeffd640 1 -- 172.21.15.29:0/1022788638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40035f0 con 0x7f7ba80047c0 2023-10-16T15:54:26.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc501a640 1 -- 172.21.15.29:0/1022788638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ba80047c0 msgr2=0x7f7ba8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1022788638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ba80047c0 0x7f7ba8004c40 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4001750 tx=0x7f7ba80057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc4819640 1 -- 172.21.15.29:0/1022788638 reap_dead start 2023-10-16T15:54:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc501a640 1 -- 172.21.15.29:0/1022788638 shutdown_connections 2023-10-16T15:54:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc501a640 1 -- 172.21.15.29:0/1022788638 >> 172.21.15.29:0/1022788638 conn(0x7f7bc00ce6a0 msgr2=0x7f7bc00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc501a640 1 -- 172.21.15.29:0/1022788638 shutdown_connections 2023-10-16T15:54:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.840+0000 7f7bc501a640 1 -- 172.21.15.29:0/1022788638 wait complete. 2023-10-16T15:54:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.842+0000 7f7bc501a640 1 Processor -- start 2023-10-16T15:54:26.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.842+0000 7f7bc501a640 1 -- start start 2023-10-16T15:54:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bc501a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ba80047c0 0x7f7bc01f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bc501a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 0x7f7bc00c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bc501a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bc006b9a0 0x7f7bc00c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bc501a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01f6510 con 0x7f7bc006b520 2023-10-16T15:54:26.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bc501a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01f5be0 con 0x7f7bc006b9a0 2023-10-16T15:54:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bc501a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01f5ec0 con 0x7f7ba80047c0 2023-10-16T15:54:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 0x7f7bc00c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bbf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 0x7f7bc00c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49194/0 (socket says 172.21.15.29:49194) 2023-10-16T15:54:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.843+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1139326024 learned_addr learned my addr 172.21.15.29:0/1139326024 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bc4819640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bc006b9a0 0x7f7bc00c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ba80047c0 msgr2=0x7f7bc01f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbffff640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ba80047c0 0x7f7bc01f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ba80047c0 0x7f7bc01f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bc006b9a0 msgr2=0x7f7bc00c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bc006b9a0 0x7f7bc00c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbf7fe640 1 -- 172.21.15.29:0/1139326024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc01f61a0 con 0x7f7bc006b520 2023-10-16T15:54:26.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bc4819640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bc006b9a0 0x7f7bc00c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbf7fe640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 0x7f7bc00c9490 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4008fd0 tx=0x7f7bb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40076c0 con 0x7f7bc006b520 2023-10-16T15:54:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb4007a30 con 0x7f7bc006b520 2023-10-16T15:54:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4006c00 con 0x7f7bc006b520 2023-10-16T15:54:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.844+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc01f6510 con 0x7f7bc006b520 2023-10-16T15:54:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.845+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc01f69a0 con 0x7f7bc006b520 2023-10-16T15:54:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.846+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc004e9b0 con 0x7f7bc006b520 2023-10-16T15:54:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.847+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7bb40035f0 con 0x7f7bc006b520 2023-10-16T15:54:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.848+0000 7f7bbcff9640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7bac06b100 0x7f7bac06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.848+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f7bb40aba90 con 0x7f7bc006b520 2023-10-16T15:54:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.848+0000 7f7bbffff640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7bac06b100 0x7f7bac06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.850+0000 7f7bbffff640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7bac06b100 0x7f7bac06d5c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0001eb0 tx=0x7f7bb0009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:26.852+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7bb40767d0 con 0x7f7bc006b520 2023-10-16T15:54:27.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.046+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7bc01f5d20 con 0x7f7bc006b520 2023-10-16T15:54:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.053+0000 7f7bbcff9640 1 -- 172.21.15.29:0/1139326024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7bc01f5d20 con 0x7f7bc006b520 2023-10-16T15:54:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.055+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7bac06b100 msgr2=0x7f7bac06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.055+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7bac06b100 0x7f7bac06d5c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0001eb0 tx=0x7f7bb0009040 comp rx=0 tx=0).stop 2023-10-16T15:54:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.055+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 msgr2=0x7f7bc00c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.055+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 0x7f7bc00c9490 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4008fd0 tx=0x7f7bb4001120 comp rx=0 tx=0).stop 2023-10-16T15:54:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 shutdown_connections 2023-10-16T15:54:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f7bac06b100 0x7f7bac06d5c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc006b520 0x7f7bc00c9490 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7bc006b9a0 0x7f7bc00c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 --2- 172.21.15.29:0/1139326024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7ba80047c0 0x7f7bc01f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 >> 172.21.15.29:0/1139326024 conn(0x7f7bc00ce6a0 msgr2=0x7f7bc00cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 shutdown_connections 2023-10-16T15:54:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.056+0000 7f7bc501a640 1 -- 172.21.15.29:0/1139326024 wait complete. 2023-10-16T15:54:27.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:54:27.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.300+0000 7f96c3180640 1 Processor -- start 2023-10-16T15:54:27.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.300+0000 7f96c3180640 1 -- start start 2023-10-16T15:54:27.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.301+0000 7f96c3180640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f96bc0b2080 0x7f96bc0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.301+0000 7f96c3180640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0cfc40 0x7f96bc0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.301+0000 7f96c3180640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc0d0970 con 0x7f96bc0b1ca0 2023-10-16T15:54:27.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.301+0000 7f96c3180640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96bc0d0ea0 con 0x7f96bc0b2080 2023-10-16T15:54:27.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.301+0000 7f96c3180640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0d0bc0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c217e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f96bc0b1ca0 0x7f96bc0ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50266/0 (socket says 172.21.15.29:50266) 2023-10-16T15:54:27.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c217e640 1 -- 172.21.15.29:0/944391582 learned_addr learned my addr 172.21.15.29:0/944391582 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:27.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c197d640 1 --2- 172.21.15.29:0/944391582 >> v2:172.21.15.29:3300/0 conn(0x7f96bc0b2080 0x7f96bc0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c117c640 1 -- 172.21.15.29:0/944391582 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540135091 0 0) 0x7f96bc0d0970 con 0x7f96bc0b1ca0 2023-10-16T15:54:27.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c117c640 1 -- 172.21.15.29:0/944391582 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a8003520 con 0x7f96bc0b1ca0 2023-10-16T15:54:27.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c297f640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0cfc40 0x7f96bc0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c197d640 1 -- 172.21.15.29:0/944391582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0cfc40 msgr2=0x7f96bc0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:27.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c197d640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0cfc40 0x7f96bc0b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c197d640 1 -- 172.21.15.29:0/944391582 >> v1:172.21.15.29:6789/0 conn(0x7f96bc0b1ca0 legacy=0x7f96bc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c197d640 1 -- 172.21.15.29:0/944391582 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0cd940 con 0x7f96bc0b2080 2023-10-16T15:54:27.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.302+0000 7f96c297f640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0cfc40 0x7f96bc0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:27.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.304+0000 7f96c197d640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0b2080 0x7f96bc0d36b0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0097a0 tx=0x7f96ac02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a2e776138a09fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:27.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.304+0000 7f96c117c640 1 -- 172.21.15.29:0/944391582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac030800 con 0x7f96bc0b2080 2023-10-16T15:54:27.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.304+0000 7f96c117c640 1 -- 172.21.15.29:0/944391582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac030960 con 0x7f96bc0b2080 2023-10-16T15:54:27.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.304+0000 7f96c117c640 1 -- 172.21.15.29:0/944391582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac030cf0 con 0x7f96bc0b2080 2023-10-16T15:54:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.304+0000 7f96c3180640 1 -- 172.21.15.29:0/944391582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0b2080 msgr2=0x7f96bc0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.304+0000 7f96c3180640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0b2080 0x7f96bc0d36b0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0097a0 tx=0x7f96ac02b940 comp rx=0 tx=0).stop 2023-10-16T15:54:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.305+0000 7f96c3180640 1 -- 172.21.15.29:0/944391582 shutdown_connections 2023-10-16T15:54:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.305+0000 7f96c3180640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0cfc40 0x7f96bc0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.305+0000 7f96c3180640 1 --2- 172.21.15.29:0/944391582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0b2080 0x7f96bc0d36b0 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.305+0000 7f96c3180640 1 -- 172.21.15.29:0/944391582 >> 172.21.15.29:0/944391582 conn(0x7f96bc0d2b90 msgr2=0x7f96bc0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.305+0000 7f96c3180640 1 -- 172.21.15.29:0/944391582 shutdown_connections 2023-10-16T15:54:27.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.305+0000 7f96c3180640 1 -- 172.21.15.29:0/944391582 wait complete. 2023-10-16T15:54:27.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.306+0000 7f96c3180640 1 Processor -- start 2023-10-16T15:54:27.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.307+0000 7f96c3180640 1 -- start start 2023-10-16T15:54:27.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.307+0000 7f96c3180640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0b1ca0 0x7f96bc15cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.307+0000 7f96c3180640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96bc0b2080 0x7f96bc09ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c3180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 0x7f96bc09d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c3180640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1f5930 con 0x7f96bc0cfc40 2023-10-16T15:54:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c3180640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1f68d0 con 0x7f96bc0b2080 2023-10-16T15:54:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c3180640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc15d2e0 con 0x7f96bc0b1ca0 2023-10-16T15:54:27.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 0x7f96bc09d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 0x7f96bc09d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49222/0 (socket says 172.21.15.29:49222) 2023-10-16T15:54:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 -- 172.21.15.29:0/36319612 learned_addr learned my addr 172.21.15.29:0/36319612 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c197d640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96bc0b2080 0x7f96bc09ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c217e640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0b1ca0 0x7f96bc15cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 -- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0b1ca0 msgr2=0x7f96bc15cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0b1ca0 0x7f96bc15cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 -- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96bc0b2080 msgr2=0x7f96bc09ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96bc0b2080 0x7f96bc09ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c297f640 1 -- 172.21.15.29:0/36319612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac009400 con 0x7f96bc0cfc40 2023-10-16T15:54:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.308+0000 7f96c197d640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96bc0b2080 0x7f96bc09ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.309+0000 7f96c297f640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 0x7f96bc09d7c0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f96b400cd50 tx=0x7f96b400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.309+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b400f680 con 0x7f96bc0cfc40 2023-10-16T15:54:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.309+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b400f800 con 0x7f96bc0cfc40 2023-10-16T15:54:27.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.309+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b400fbb0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.309+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc1f5b10 con 0x7f96bc0cfc40 2023-10-16T15:54:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.309+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc1f60d0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.310+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc04e9b0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.311+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f96b400b040 con 0x7f96bc0cfc40 2023-10-16T15:54:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.312+0000 7f96b2ffd640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f96a006b0b0 0x7f96a006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.312+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f96b408a6b0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.313+0000 7f96c217e640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f96a006b0b0 0x7f96a006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.314+0000 7f96c217e640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f96a006b0b0 0x7f96a006d570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f96b80011e0 tx=0x7f96b8009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:27.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.317+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f96b40551a0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.511+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f96bc073a40 con 0x7f96bc0cfc40 2023-10-16T15:54:27.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.518+0000 7f96b2ffd640 1 -- 172.21.15.29:0/36319612 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v80) v1 ==== 115+0+0 (secure 0 0 0) 0x7f96b40587c0 con 0x7f96bc0cfc40 2023-10-16T15:54:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.520+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f96a006b0b0 msgr2=0x7f96a006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.520+0000 7f96c3180640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f96a006b0b0 0x7f96a006d570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f96b80011e0 tx=0x7f96b8009040 comp rx=0 tx=0).stop 2023-10-16T15:54:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 msgr2=0x7f96bc09d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 0x7f96bc09d7c0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f96b400cd50 tx=0x7f96b400a650 comp rx=0 tx=0).stop 2023-10-16T15:54:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 shutdown_connections 2023-10-16T15:54:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f96a006b0b0 0x7f96a006d570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96bc0cfc40 0x7f96bc09d7c0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96bc0b2080 0x7f96bc09ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 --2- 172.21.15.29:0/36319612 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f96bc0b1ca0 0x7f96bc15cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 >> 172.21.15.29:0/36319612 conn(0x7f96bc0d2b90 msgr2=0x7f96bc06b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 shutdown_connections 2023-10-16T15:54:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.521+0000 7f96c3180640 1 -- 172.21.15.29:0/36319612 wait complete. 2023-10-16T15:54:27.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:54:27.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.774+0000 7faf14f63640 1 Processor -- start 2023-10-16T15:54:27.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.774+0000 7faf14f63640 1 -- start start 2023-10-16T15:54:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.775+0000 7faf14f63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 0x7faf100c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf14f63640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7faf1015dde0 0x7faf100ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf14f63640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf10160a90 con 0x7faf100c9230 2023-10-16T15:54:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf14f63640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf100cdec0 con 0x7faf1015dde0 2023-10-16T15:54:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf14f63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf100cd900 con 0x7faf1015c9f0 2023-10-16T15:54:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf0effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faf100c9230 0x7faf100709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50268/0 (socket says 172.21.15.29:50268) 2023-10-16T15:54:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf0effd640 1 -- 172.21.15.29:0/603844016 learned_addr learned my addr 172.21.15.29:0/603844016 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf0ffff640 1 --2- 172.21.15.29:0/603844016 >> v2:172.21.15.170:3300/0 conn(0x7faf1015dde0 0x7faf100ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.776+0000 7faf0f7fe640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 0x7faf100c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847517582 0 0) 0x7faf10160a90 con 0x7faf100c9230 2023-10-16T15:54:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef8003520 con 0x7faf100c9230 2023-10-16T15:54:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085341429 0 0) 0x7faef8003520 con 0x7faf100c9230 2023-10-16T15:54:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 msgr2=0x7faf100c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 0x7faf100c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 >> v2:172.21.15.170:3300/0 conn(0x7faf1015dde0 msgr2=0x7faf100ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 --2- 172.21.15.29:0/603844016 >> v2:172.21.15.170:3300/0 conn(0x7faf1015dde0 0x7faf100ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf100cdbe0 con 0x7faf100c9230 2023-10-16T15:54:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7faf04001670 con 0x7faf100c9230 2023-10-16T15:54:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 >> v1:172.21.15.29:6789/0 conn(0x7faf100c9230 legacy=0x7faf100709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef8003970 0x7faef8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef80047c0 0x7faef8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 0x7faef8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf100cd900 con 0x7faef80047c0 2023-10-16T15:54:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef800b980 con 0x7faef8003970 2023-10-16T15:54:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.777+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef800bac0 con 0x7faf1015c9f0 2023-10-16T15:54:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0effd640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef80047c0 0x7faef8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0f7fe640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef8003970 0x7faef8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0ffff640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 0x7faef8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0effd640 1 -- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 msgr2=0x7faef8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0effd640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf1015c9f0 0x7faef8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0effd640 1 -- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef8003970 msgr2=0x7faef8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0effd640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef8003970 0x7faef8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.778+0000 7faf0effd640 1 -- 172.21.15.29:0/603844016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8003520 con 0x7faef80047c0 2023-10-16T15:54:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.779+0000 7faf0effd640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef80047c0 0x7faef8004c40 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7faf0400c160 tx=0x7faef80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88033b2478351d1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.780+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04005ad0 con 0x7faef80047c0 2023-10-16T15:54:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.780+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf04005c30 con 0x7faef80047c0 2023-10-16T15:54:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.780+0000 7faf0e7fc640 1 -- 172.21.15.29:0/603844016 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0402c810 con 0x7faef80047c0 2023-10-16T15:54:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.780+0000 7faf14f63640 1 -- 172.21.15.29:0/603844016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef80047c0 msgr2=0x7faef8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.780+0000 7faf14f63640 1 --2- 172.21.15.29:0/603844016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faef80047c0 0x7faef8004c40 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7faf0400c160 tx=0x7faef80057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.780+0000 7faf0ffff640 1 -- 172.21.15.29:0/603844016 reap_dead start 2023-10-16T15:54:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.781+0000 7faf14f63640 1 -- 172.21.15.29:0/603844016 shutdown_connections 2023-10-16T15:54:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.781+0000 7faf14f63640 1 -- 172.21.15.29:0/603844016 >> 172.21.15.29:0/603844016 conn(0x7faf100ce6a0 msgr2=0x7faf100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.781+0000 7faf14f63640 1 -- 172.21.15.29:0/603844016 shutdown_connections 2023-10-16T15:54:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.781+0000 7faf14f63640 1 -- 172.21.15.29:0/603844016 wait complete. 2023-10-16T15:54:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.783+0000 7faf14f63640 1 Processor -- start 2023-10-16T15:54:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.783+0000 7faf14f63640 1 -- start start 2023-10-16T15:54:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf14f63640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef80047c0 0x7faf1006b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf14f63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 0x7faf10158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf14f63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf100c5770 0x7faf100c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf14f63640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101f26f0 con 0x7faf101f5b40 2023-10-16T15:54:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf14f63640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1006b930 con 0x7faef80047c0 2023-10-16T15:54:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf14f63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf101f60e0 con 0x7faf100c5770 2023-10-16T15:54:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 0x7faf10158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 0x7faf10158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49250/0 (socket says 172.21.15.29:49250) 2023-10-16T15:54:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 -- 172.21.15.29:0/3193729508 learned_addr learned my addr 172.21.15.29:0/3193729508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0f7fe640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef80047c0 0x7faf1006b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 -- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf100c5770 msgr2=0x7faf100c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf100c5770 0x7faf100c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 -- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef80047c0 msgr2=0x7faf1006b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.784+0000 7faf0effd640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef80047c0 0x7faf1006b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faf0effd640 1 -- 172.21.15.29:0/3193729508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf101f63c0 con 0x7faf101f5b40 2023-10-16T15:54:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faf0f7fe640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef80047c0 0x7faf1006b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faf0effd640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 0x7faf10158ce0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7faf04000c00 tx=0x7faf04000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0403c040 con 0x7faf101f5b40 2023-10-16T15:54:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0402cc20 con 0x7faf101f5b40 2023-10-16T15:54:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf101f66a0 con 0x7faf101f5b40 2023-10-16T15:54:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04007070 con 0x7faf101f5b40 2023-10-16T15:54:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.785+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf101f6980 con 0x7faf101f5b40 2023-10-16T15:54:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.786+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1004e9b0 con 0x7faf101f5b40 2023-10-16T15:54:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.787+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faf04032070 con 0x7faf101f5b40 2023-10-16T15:54:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.789+0000 7faef7fff640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faefc06b0e0 0x7faefc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.789+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7faf040aaec0 con 0x7faf101f5b40 2023-10-16T15:54:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.790+0000 7faf0f7fe640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faefc06b0e0 0x7faefc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.791+0000 7faf0f7fe640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faefc06b0e0 0x7faefc06d5a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faf100ca5f0 tx=0x7faf00008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.794+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faf04075b50 con 0x7faf101f5b40 2023-10-16T15:54:27.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.987+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7faf100adac0 con 0x7faf101f5b40 2023-10-16T15:54:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.993+0000 7faef7fff640 1 -- 172.21.15.29:0/3193729508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) v1 ==== 120+0+0 (secure 0 0 0) 0x7faf04079170 con 0x7faf101f5b40 2023-10-16T15:54:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faefc06b0e0 msgr2=0x7faefc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faefc06b0e0 0x7faefc06d5a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faf100ca5f0 tx=0x7faf00008040 comp rx=0 tx=0).stop 2023-10-16T15:54:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 msgr2=0x7faf10158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 0x7faf10158ce0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7faf04000c00 tx=0x7faf04000fe0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 shutdown_connections 2023-10-16T15:54:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faefc06b0e0 0x7faefc06d5a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf101f5b40 0x7faf10158ce0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faf100c5770 0x7faf100c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 --2- 172.21.15.29:0/3193729508 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faef80047c0 0x7faf1006b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 >> 172.21.15.29:0/3193729508 conn(0x7faf100ce6a0 msgr2=0x7faf1015b910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 shutdown_connections 2023-10-16T15:54:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:27.996+0000 7faf14f63640 1 -- 172.21.15.29:0/3193729508 wait complete. 2023-10-16T15:54:28.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T15:54:28.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.239+0000 7faa4fd05640 1 Processor -- start 2023-10-16T15:54:28.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.239+0000 7faa4fd05640 1 -- start start 2023-10-16T15:54:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4fd05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa48159f00 0x7faa480b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4fd05640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa4815d880 0x7faa480b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4fd05640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4806f7f0 con 0x7faa48159b20 2023-10-16T15:54:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4fd05640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4806c5e0 con 0x7faa4815d880 2023-10-16T15:54:28.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4fd05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4806c8c0 con 0x7faa48159f00 2023-10-16T15:54:28.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4e2a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa4815d880 0x7faa480b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4e2a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa4815d880 0x7faa480b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49256/0 (socket says 172.21.15.29:49256) 2023-10-16T15:54:28.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.241+0000 7faa4e2a2640 1 -- 172.21.15.29:0/1848678290 learned_addr learned my addr 172.21.15.29:0/1848678290 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.242+0000 7faa4d2a0640 1 --2- 172.21.15.29:0/1848678290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa48159f00 0x7faa480b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.242+0000 7faa4e2a2640 1 -- 172.21.15.29:0/1848678290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa48159f00 msgr2=0x7faa480b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.242+0000 7faa4e2a2640 1 --2- 172.21.15.29:0/1848678290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa48159f00 0x7faa480b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.242+0000 7faa4e2a2640 1 -- 172.21.15.29:0/1848678290 >> v1:172.21.15.170:6789/0 conn(0x7faa48159b20 legacy=0x7faa4806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.242+0000 7faa4e2a2640 1 -- 172.21.15.29:0/1848678290 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4806cba0 con 0x7faa4815d880 2023-10-16T15:54:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.243+0000 7faa4e2a2640 1 --2- 172.21.15.29:0/1848678290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa4815d880 0x7faa480b0950 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7faa4000d150 tx=0x7faa4002e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b492bdf9c9a49eae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.243+0000 7faa4ca9f640 1 -- 172.21.15.29:0/1848678290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000b0d0 con 0x7faa4815d880 2023-10-16T15:54:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.243+0000 7faa4ca9f640 1 -- 172.21.15.29:0/1848678290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4000b230 con 0x7faa4815d880 2023-10-16T15:54:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.243+0000 7faa4ca9f640 1 -- 172.21.15.29:0/1848678290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40009600 con 0x7faa4815d880 2023-10-16T15:54:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 -- 172.21.15.29:0/1848678290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa4815d880 msgr2=0x7faa480b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 --2- 172.21.15.29:0/1848678290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa4815d880 0x7faa480b0950 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7faa4000d150 tx=0x7faa4002e160 comp rx=0 tx=0).stop 2023-10-16T15:54:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 -- 172.21.15.29:0/1848678290 shutdown_connections 2023-10-16T15:54:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 --2- 172.21.15.29:0/1848678290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa4815d880 0x7faa480b0950 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 --2- 172.21.15.29:0/1848678290 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa48159f00 0x7faa480b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 -- 172.21.15.29:0/1848678290 >> 172.21.15.29:0/1848678290 conn(0x7faa480cb410 msgr2=0x7faa480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 -- 172.21.15.29:0/1848678290 shutdown_connections 2023-10-16T15:54:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.244+0000 7faa4fd05640 1 -- 172.21.15.29:0/1848678290 wait complete. 2023-10-16T15:54:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.246+0000 7faa4fd05640 1 Processor -- start 2023-10-16T15:54:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.247+0000 7faa4fd05640 1 -- start start 2023-10-16T15:54:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.247+0000 7faa4fd05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 0x7faa4814ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.247+0000 7faa4fd05640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa48159f00 0x7faa4815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4fd05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa4815d880 0x7faa4815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4fd05640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa481535d0 con 0x7faa48159b20 2023-10-16T15:54:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4fd05640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa481538b0 con 0x7faa48159f00 2023-10-16T15:54:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4fd05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa48153b90 con 0x7faa4815d880 2023-10-16T15:54:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 0x7faa4814ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 0x7faa4814ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49258/0 (socket says 172.21.15.29:49258) 2023-10-16T15:54:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 -- 172.21.15.29:0/846881334 learned_addr learned my addr 172.21.15.29:0/846881334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 -- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa4815d880 msgr2=0x7faa4815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa4815d880 0x7faa4815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 -- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa48159f00 msgr2=0x7faa4815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa48159f00 0x7faa4815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.248+0000 7faa4daa1640 1 -- 172.21.15.29:0/846881334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4000bd40 con 0x7faa48159b20 2023-10-16T15:54:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.249+0000 7faa4daa1640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 0x7faa4814ef30 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c400 tx=0x7faa4400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.249+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa440052d0 con 0x7faa48159b20 2023-10-16T15:54:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.249+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44005430 con 0x7faa48159b20 2023-10-16T15:54:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.249+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44003830 con 0x7faa48159b20 2023-10-16T15:54:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.249+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa48153e70 con 0x7faa48159b20 2023-10-16T15:54:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.249+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4814f470 con 0x7faa48159b20 2023-10-16T15:54:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.250+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa48154390 con 0x7faa48159b20 2023-10-16T15:54:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.251+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faa44003990 con 0x7faa48159b20 2023-10-16T15:54:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.252+0000 7faa3e7fc640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faa2c06b0e0 0x7faa2c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.252+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7faa44088eb0 con 0x7faa48159b20 2023-10-16T15:54:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.253+0000 7faa4d2a0640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faa2c06b0e0 0x7faa2c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.254+0000 7faa4d2a0640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faa2c06b0e0 0x7faa2c06d5a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faa4815a2e0 tx=0x7faa38008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.257+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa48154390 con 0x7faa48159b20 2023-10-16T15:54:28.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.526+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7faa4814f600 con 0x7faa48159b20 2023-10-16T15:54:28.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.587+0000 7faa3e7fc640 1 -- 172.21.15.29:0/846881334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3225) v1 ==== 186+0+0 (secure 0 0 0) 0x7faa44017100 con 0x7faa48159b20 2023-10-16T15:54:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faa2c06b0e0 msgr2=0x7faa2c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faa2c06b0e0 0x7faa2c06d5a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faa4815a2e0 tx=0x7faa38008040 comp rx=0 tx=0).stop 2023-10-16T15:54:28.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 msgr2=0x7faa4814ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 0x7faa4814ef30 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c400 tx=0x7faa4400c8c0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 shutdown_connections 2023-10-16T15:54:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7faa2c06b0e0 0x7faa2c06d5a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faa4815d880 0x7faa4815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faa48159f00 0x7faa4815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 --2- 172.21.15.29:0/846881334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa48159b20 0x7faa4814ef30 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 >> 172.21.15.29:0/846881334 conn(0x7faa480cb410 msgr2=0x7faa480adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 shutdown_connections 2023-10-16T15:54:28.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.590+0000 7faa4fd05640 1 -- 172.21.15.29:0/846881334 wait complete. 2023-10-16T15:54:28.611 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2023-10-16T15:54:28.611 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2023-10-16T15:54:28.612 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T15:54:28.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:28.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.829+0000 7f4314b80640 1 Processor -- start 2023-10-16T15:54:28.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.829+0000 7f4314b80640 1 -- start start 2023-10-16T15:54:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.830+0000 7f4314b80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43100cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.830+0000 7f4314b80640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f43100ccfb0 0x7f43100cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f4314b80640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431015b9d0 con 0x7f431006c040 2023-10-16T15:54:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f4314b80640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431015bcb0 con 0x7f43100ccfb0 2023-10-16T15:54:28.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f4314b80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431015bf90 con 0x7f43100ad970 2023-10-16T15:54:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f4307fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43100cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f430ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f43100ccfb0 0x7f43100cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f430f7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f431006c040 0x7f431006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:39242/0 (socket says 172.21.15.29:39242) 2023-10-16T15:54:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.831+0000 7f430f7fe640 1 -- 172.21.15.29:0/400717369 learned_addr learned my addr 172.21.15.29:0/400717369 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.832+0000 7f4307fff640 1 -- 172.21.15.29:0/400717369 >> v2:172.21.15.170:3300/0 conn(0x7f43100ccfb0 msgr2=0x7f43100cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.832+0000 7f4307fff640 1 --2- 172.21.15.29:0/400717369 >> v2:172.21.15.170:3300/0 conn(0x7f43100ccfb0 0x7f43100cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.832+0000 7f4307fff640 1 -- 172.21.15.29:0/400717369 >> v1:172.21.15.170:6789/0 conn(0x7f431006c040 legacy=0x7f431006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.832+0000 7f4307fff640 1 -- 172.21.15.29:0/400717369 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100cc270 con 0x7f43100ad970 2023-10-16T15:54:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.832+0000 7f430ffff640 1 --2- 172.21.15.29:0/400717369 >> v2:172.21.15.170:3300/0 conn(0x7f43100ccfb0 0x7f43100cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.834+0000 7f4307fff640 1 --2- 172.21.15.29:0/400717369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43100cca70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097d0 tx=0x7f42f802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47732a1795d5812e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.835+0000 7f430effd640 1 -- 172.21.15.29:0/400717369 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f802d040 con 0x7f43100ad970 2023-10-16T15:54:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.835+0000 7f430effd640 1 -- 172.21.15.29:0/400717369 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f8031810 con 0x7f43100ad970 2023-10-16T15:54:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.835+0000 7f430effd640 1 -- 172.21.15.29:0/400717369 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8031ba0 con 0x7f43100ad970 2023-10-16T15:54:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.835+0000 7f4314b80640 1 -- 172.21.15.29:0/400717369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 msgr2=0x7f43100cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.835+0000 7f4314b80640 1 --2- 172.21.15.29:0/400717369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43100cca70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097d0 tx=0x7f42f802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.836+0000 7f4314b80640 1 -- 172.21.15.29:0/400717369 shutdown_connections 2023-10-16T15:54:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.836+0000 7f4314b80640 1 --2- 172.21.15.29:0/400717369 >> v2:172.21.15.170:3300/0 conn(0x7f43100ccfb0 0x7f43100cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.836+0000 7f4314b80640 1 --2- 172.21.15.29:0/400717369 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43100cca70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.836+0000 7f4314b80640 1 -- 172.21.15.29:0/400717369 >> 172.21.15.29:0/400717369 conn(0x7f43100ce6a0 msgr2=0x7f43100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.836+0000 7f4314b80640 1 -- 172.21.15.29:0/400717369 shutdown_connections 2023-10-16T15:54:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.836+0000 7f4314b80640 1 -- 172.21.15.29:0/400717369 wait complete. 2023-10-16T15:54:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.838+0000 7f4314b80640 1 Processor -- start 2023-10-16T15:54:28.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.838+0000 7f4314b80640 1 -- start start 2023-10-16T15:54:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.839+0000 7f4314b80640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431006c040 0x7f43100c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.839+0000 7f4314b80640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43101f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f4314b80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 0x7f43100aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f4314b80640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43101f4b10 con 0x7f43100ccfb0 2023-10-16T15:54:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f4314b80640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43100c5770 con 0x7f431006c040 2023-10-16T15:54:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f4314b80640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43100c59c0 con 0x7f43100ad970 2023-10-16T15:54:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 0x7f43100aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 0x7f43100aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49268/0 (socket says 172.21.15.29:49268) 2023-10-16T15:54:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 -- 172.21.15.29:0/2489815451 learned_addr learned my addr 172.21.15.29:0/2489815451 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430f7fe640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431006c040 0x7f43100c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f4307fff640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43101f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 -- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 msgr2=0x7f43101f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43101f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 -- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431006c040 msgr2=0x7f43100c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431006c040 0x7f43100c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.840+0000 7f430ffff640 1 -- 172.21.15.29:0/2489815451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308009a70 con 0x7f43100ccfb0 2023-10-16T15:54:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.841+0000 7f430ffff640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 0x7f43100aeae0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f430800efc0 tx=0x7f430800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.841+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308009cd0 con 0x7f43100ccfb0 2023-10-16T15:54:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.841+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430800f040 con 0x7f43100ccfb0 2023-10-16T15:54:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.841+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308003990 con 0x7f43100ccfb0 2023-10-16T15:54:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.841+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f8009400 con 0x7f43100ccfb0 2023-10-16T15:54:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.841+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43100c5f80 con 0x7f43100ccfb0 2023-10-16T15:54:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.842+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431006b330 con 0x7f43100ccfb0 2023-10-16T15:54:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.843+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4308014480 con 0x7f43100ccfb0 2023-10-16T15:54:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.844+0000 7f430cff9640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f42f006b0e0 0x7f42f006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.844+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f430808a850 con 0x7f43100ccfb0 2023-10-16T15:54:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.845+0000 7f430f7fe640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f42f006b0e0 0x7f42f006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.846+0000 7f430f7fe640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f42f006b0e0 0x7f42f006d5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4300005880 tx=0x7f4300009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:28.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:28.849+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4308055340 con 0x7f43100ccfb0 2023-10-16T15:54:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.035+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f431004e9b0 con 0x7f43100ccfb0 2023-10-16T15:54:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.037+0000 7f430cff9640 1 -- 172.21.15.29:0/2489815451 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+14379 (secure 0 0 0) 0x7f4308058960 con 0x7f43100ccfb0 2023-10-16T15:54:29.039 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:29.039 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":237,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:26.571911+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:6835/1674117611":"2023-10-17T15:54:23.238166+0000","172.21.15.170:0/533263605":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1480733109":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/2011924145":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/1824188539":"2023-10-17T15:54:09.426049+0000","172.21.15.29:6834/3313937659":"2023-10-17T15:54:15.311880+0000","172.21.15.170:0/3705788759":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1184531267":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6837/1674117611":"2023-10-17T15:54:23.238166+0000","172.21.15.170:0/1484905351":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/3617680928":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6832/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2963466728":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/606018791":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2373276542":"2023-10-17T15:54:09.426049+0000","172.21.15.170:6833/3980026359":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/2707368093":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/3876776495":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1834694957":"2023-10-17T15:53:58.385901+0000","172.21.15.170:0/1658522756":"2023-10-17T15:54:09.426049+0000","172.21.15.170:0/1551430148":"2023-10-17T15:53:58.385901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f42f006b0e0 msgr2=0x7f42f006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f42f006b0e0 0x7f42f006d5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4300005880 tx=0x7f4300009040 comp rx=0 tx=0).stop 2023-10-16T15:54:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 msgr2=0x7f43100aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 0x7f43100aeae0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f430800efc0 tx=0x7f430800c490 comp rx=0 tx=0).stop 2023-10-16T15:54:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 shutdown_connections 2023-10-16T15:54:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f42f006b0e0 0x7f42f006d5a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43100ccfb0 0x7f43100aeae0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43100ad970 0x7f43101f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 --2- 172.21.15.29:0/2489815451 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431006c040 0x7f43100c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 >> 172.21.15.29:0/2489815451 conn(0x7f43100ce6a0 msgr2=0x7f431015a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.039+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 shutdown_connections 2023-10-16T15:54:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.040+0000 7f4314b80640 1 -- 172.21.15.29:0/2489815451 wait complete. 2023-10-16T15:54:29.060 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:54:29.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:54:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.286+0000 7f8d10d73640 1 Processor -- start 2023-10-16T15:54:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.287+0000 7f8d10d73640 1 -- start start 2023-10-16T15:54:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.288+0000 7f8d10d73640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8d0c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.288+0000 7f8d10d73640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8d0c06c140 0x7f8d0c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.288+0000 7f8d10d73640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c070360 con 0x7f8d0c159e90 2023-10-16T15:54:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.288+0000 7f8d10d73640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c06cb10 con 0x7f8d0c06c140 2023-10-16T15:54:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.288+0000 7f8d10d73640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c15d890 con 0x7f8d0c159ab0 2023-10-16T15:54:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d03fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8d0c159e90 0x7f8d0c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50276/0 (socket says 172.21.15.29:50276) 2023-10-16T15:54:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d03fff640 1 -- 172.21.15.29:0/1544463309 learned_addr learned my addr 172.21.15.29:0/1544463309 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d0a59c640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8d0c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/1544463309 >> v2:172.21.15.170:3300/0 conn(0x7f8d0c06c140 0x7f8d0c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37551400 0 0) 0x7f8d0c070360 con 0x7f8d0c159e90 2023-10-16T15:54:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf8003520 con 0x7f8d0c159e90 2023-10-16T15:54:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.289+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722358478 0 0) 0x7f8cf8003520 con 0x7f8d0c159e90 2023-10-16T15:54:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 msgr2=0x7f8d0c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8d0c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 >> v2:172.21.15.170:3300/0 conn(0x7f8d0c06c140 msgr2=0x7f8d0c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 --2- 172.21.15.29:0/1544463309 >> v2:172.21.15.170:3300/0 conn(0x7f8d0c06c140 0x7f8d0c06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0cd1f0 con 0x7f8d0c159e90 2023-10-16T15:54:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8cf4001670 con 0x7f8d0c159e90 2023-10-16T15:54:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 >> v1:172.21.15.29:6789/0 conn(0x7f8d0c159e90 legacy=0x7f8d0c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cf8003970 0x7f8cf8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8cf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cf8004d00 0x7f8cf8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c06cb10 con 0x7f8cf8004d00 2023-10-16T15:54:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c15d890 con 0x7f8cf8003970 2023-10-16T15:54:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf800ba00 con 0x7f8d0c159ab0 2023-10-16T15:54:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cf8004d00 0x7f8cf8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d0a59c640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cf8003970 0x7f8cf8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d03fff640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8cf80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 msgr2=0x7f8cf80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8cf80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.290+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cf8003970 msgr2=0x7f8cf8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:29.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.291+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cf8003970 0x7f8cf8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.291+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/1544463309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf8003520 con 0x7f8cf8004d00 2023-10-16T15:54:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.291+0000 7f8d03fff640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8d0c159ab0 0x7f8cf80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.291+0000 7f8d0a59c640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cf8003970 0x7f8cf8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cf8004d00 0x7f8cf8005180 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009fc0 tx=0x7f8cf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdbef91c12c5f21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:29.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0402cc60 con 0x7f8cf8004d00 2023-10-16T15:54:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0400f040 con 0x7f8cf8004d00 2023-10-16T15:54:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d09d9b640 1 -- 172.21.15.29:0/1544463309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04034440 con 0x7f8cf8004d00 2023-10-16T15:54:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d10d73640 1 -- 172.21.15.29:0/1544463309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cf8004d00 msgr2=0x7f8cf8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d10d73640 1 --2- 172.21.15.29:0/1544463309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cf8004d00 0x7f8cf8005180 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009fc0 tx=0x7f8cf80057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.292+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/1544463309 reap_dead start 2023-10-16T15:54:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.293+0000 7f8d10d73640 1 -- 172.21.15.29:0/1544463309 shutdown_connections 2023-10-16T15:54:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.293+0000 7f8d10d73640 1 -- 172.21.15.29:0/1544463309 >> 172.21.15.29:0/1544463309 conn(0x7f8d0c0ce9b0 msgr2=0x7f8d0c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.293+0000 7f8d10d73640 1 -- 172.21.15.29:0/1544463309 shutdown_connections 2023-10-16T15:54:29.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.293+0000 7f8d10d73640 1 -- 172.21.15.29:0/1544463309 wait complete. 2023-10-16T15:54:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.295+0000 7f8d10d73640 1 Processor -- start 2023-10-16T15:54:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.295+0000 7f8d10d73640 1 -- start start 2023-10-16T15:54:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d10d73640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cf8004d00 0x7f8d0c0c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d10d73640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c0c5a80 0x7f8d0c0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d10d73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 0x7f8d0c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d10d73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0c9ea0 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d10d73640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0c78e0 con 0x7f8d0c0c5a80 2023-10-16T15:54:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d10d73640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0c7bc0 con 0x7f8cf8004d00 2023-10-16T15:54:29.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 0x7f8d0c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 0x7f8d0c150980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49288/0 (socket says 172.21.15.29:49288) 2023-10-16T15:54:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/2346587814 learned_addr learned my addr 172.21.15.29:0/2346587814 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d0a59c640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cf8004d00 0x7f8d0c0c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.296+0000 7f8d03fff640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c0c5a80 0x7f8d0c0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cf8004d00 msgr2=0x7f8d0c0c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cf8004d00 0x7f8d0c0c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c0c5a80 msgr2=0x7f8d0c0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c0c5a80 0x7f8d0c0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d0ad9d640 1 -- 172.21.15.29:0/2346587814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0c7ea0 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d03fff640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c0c5a80 0x7f8d0c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d0ad9d640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 0x7f8d0c150980 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f8d04037040 tx=0x7f8d04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04034d50 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04003990 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c0c8180 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04003d40 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.297+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c0c5500 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.298+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c0c86a0 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.300+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8d04004030 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.300+0000 7f8d02ffd640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8cec06b0e0 0x7f8cec06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.301+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f8d040ab3a0 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.301+0000 7f8d0a59c640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8cec06b0e0 0x7f8cec06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.302+0000 7f8d0a59c640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8cec06b0e0 0x7f8cec06d5a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009990 tx=0x7f8cfc008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.305+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d0c0c86a0 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.491+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d0c0c86a0 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.492+0000 7f8d02ffd640 1 -- 172.21.15.29:0/2346587814 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v237) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d04076050 con 0x7f8d0c0c9a20 2023-10-16T15:54:29.494 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:54:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8cec06b0e0 msgr2=0x7f8cec06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8cec06b0e0 0x7f8cec06d5a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009990 tx=0x7f8cfc008040 comp rx=0 tx=0).stop 2023-10-16T15:54:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 msgr2=0x7f8d0c150980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 0x7f8d0c150980 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f8d04037040 tx=0x7f8d04009750 comp rx=0 tx=0).stop 2023-10-16T15:54:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 shutdown_connections 2023-10-16T15:54:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8cec06b0e0 0x7f8cec06d5a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d0c0c9a20 0x7f8d0c150980 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d0c0c5a80 0x7f8d0c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 --2- 172.21.15.29:0/2346587814 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cf8004d00 0x7f8d0c0c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 >> 172.21.15.29:0/2346587814 conn(0x7f8d0c0ce9b0 msgr2=0x7f8d0c1522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:29.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 shutdown_connections 2023-10-16T15:54:29.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.495+0000 7f8d10d73640 1 -- 172.21.15.29:0/2346587814 wait complete. 2023-10-16T15:54:29.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T15:54:29.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.737+0000 7f80bce60640 1 Processor -- start 2023-10-16T15:54:29.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.738+0000 7f80bce60640 1 -- start start 2023-10-16T15:54:29.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.739+0000 7f80bce60640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80b806c040 0x7f80b806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.739+0000 7f80bce60640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f80b80cda80 0x7f80b815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.739+0000 7f80bce60640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b806c960 con 0x7f80b80ad970 2023-10-16T15:54:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.739+0000 7f80bce60640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b815b9d0 con 0x7f80b806c040 2023-10-16T15:54:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.739+0000 7f80bce60640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b815bcb0 con 0x7f80b80cda80 2023-10-16T15:54:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80affff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f80b80ad970 0x7f80b815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50288/0 (socket says 172.21.15.29:50288) 2023-10-16T15:54:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80affff640 1 -- 172.21.15.29:0/2955370340 learned_addr learned my addr 172.21.15.29:0/2955370340 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b77fe640 1 --2- 172.21.15.29:0/2955370340 >> v2:172.21.15.29:3300/0 conn(0x7f80b806c040 0x7f80b806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b7fff640 1 --2- 172.21.15.29:0/2955370340 >> v2:172.21.15.170:3300/0 conn(0x7f80b80cda80 0x7f80b815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b6ffd640 1 -- 172.21.15.29:0/2955370340 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795666551 0 0) 0x7f80b806c960 con 0x7f80b80ad970 2023-10-16T15:54:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b6ffd640 1 -- 172.21.15.29:0/2955370340 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a4003520 con 0x7f80b80ad970 2023-10-16T15:54:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b77fe640 1 -- 172.21.15.29:0/2955370340 >> v2:172.21.15.170:3300/0 conn(0x7f80b80cda80 msgr2=0x7f80b815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b77fe640 1 --2- 172.21.15.29:0/2955370340 >> v2:172.21.15.170:3300/0 conn(0x7f80b80cda80 0x7f80b815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b77fe640 1 -- 172.21.15.29:0/2955370340 >> v1:172.21.15.29:6789/0 conn(0x7f80b80ad970 legacy=0x7f80b815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b6ffd640 1 -- 172.21.15.29:0/2955370340 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906276612 0 0) 0x7f80a4003520 con 0x7f80b80ad970 2023-10-16T15:54:29.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.740+0000 7f80b77fe640 1 -- 172.21.15.29:0/2955370340 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b815bf90 con 0x7f80b806c040 2023-10-16T15:54:29.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.741+0000 7f80b77fe640 1 --2- 172.21.15.29:0/2955370340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b806c420 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f80b80cf300 tx=0x7f80a802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a05d327e0f80a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:29.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80b6ffd640 1 -- 172.21.15.29:0/2955370340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0001670 con 0x7f80b806c040 2023-10-16T15:54:29.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80b6ffd640 1 -- 172.21.15.29:0/2955370340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a8033440 con 0x7f80b806c040 2023-10-16T15:54:29.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80b6ffd640 1 -- 172.21.15.29:0/2955370340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a80337d0 con 0x7f80b806c040 2023-10-16T15:54:29.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 -- 172.21.15.29:0/2955370340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 msgr2=0x7f80b806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 --2- 172.21.15.29:0/2955370340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b806c420 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f80b80cf300 tx=0x7f80a802b7a0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 -- 172.21.15.29:0/2955370340 shutdown_connections 2023-10-16T15:54:29.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 --2- 172.21.15.29:0/2955370340 >> v2:172.21.15.170:3300/0 conn(0x7f80b80cda80 0x7f80b815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 --2- 172.21.15.29:0/2955370340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b806c420 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 -- 172.21.15.29:0/2955370340 >> 172.21.15.29:0/2955370340 conn(0x7f80b80ce6a0 msgr2=0x7f80b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:29.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 -- 172.21.15.29:0/2955370340 shutdown_connections 2023-10-16T15:54:29.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.742+0000 7f80bce60640 1 -- 172.21.15.29:0/2955370340 wait complete. 2023-10-16T15:54:29.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.744+0000 7f80bce60640 1 Processor -- start 2023-10-16T15:54:29.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.744+0000 7f80bce60640 1 -- start start 2023-10-16T15:54:29.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80bce60640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80bce60640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80b80ad970 0x7f80b80c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80bce60640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80b80cda80 0x7f80b8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80bce60640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b80acdd0 con 0x7f80b806c040 2023-10-16T15:54:29.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80bce60640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b80c9b60 con 0x7f80b80cda80 2023-10-16T15:54:29.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80bce60640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b8156f10 con 0x7f80b80ad970 2023-10-16T15:54:29.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.745+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49304/0 (socket says 172.21.15.29:49304) 2023-10-16T15:54:29.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 -- 172.21.15.29:0/3542674471 learned_addr learned my addr 172.21.15.29:0/3542674471 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:29.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b7fff640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80b80cda80 0x7f80b8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80affff640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80b80ad970 0x7f80b80c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 -- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80b80ad970 msgr2=0x7f80b80c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80b80ad970 0x7f80b80c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 -- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80b80cda80 msgr2=0x7f80b8155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:29.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80b80cda80 0x7f80b8155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:29.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 -- 172.21.15.29:0/3542674471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8009400 con 0x7f80b806c040 2023-10-16T15:54:29.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b7fff640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80b80cda80 0x7f80b8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:29.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.746+0000 7f80b77fe640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b80ac890 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009890 tx=0x7f80a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:29.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.747+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a803d940 con 0x7f80b806c040 2023-10-16T15:54:29.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.747+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a803dac0 con 0x7f80b806c040 2023-10-16T15:54:29.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.747+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a803de70 con 0x7f80b806c040 2023-10-16T15:54:29.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.747+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b80c9190 con 0x7f80b806c040 2023-10-16T15:54:29.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.747+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b815a480 con 0x7f80b806c040 2023-10-16T15:54:29.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.748+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b806b330 con 0x7f80b806c040 2023-10-16T15:54:29.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.749+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f80a803a040 con 0x7f80b806c040 2023-10-16T15:54:29.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.750+0000 7f80b4ff9640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f809806b0b0 0x7f809806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:29.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.750+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f80a80ab960 con 0x7f80b806c040 2023-10-16T15:54:29.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.751+0000 7f80affff640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f809806b0b0 0x7f809806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:29.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.752+0000 7f80affff640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f809806b0b0 0x7f809806d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f80a0001380 tx=0x7f80a0009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:29.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:29.754+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f80a80767b0 con 0x7f80b806c040 2023-10-16T15:54:30.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.014+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f80b804e9b0 con 0x7f80b806c040 2023-10-16T15:54:30.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.599+0000 7f80b4ff9640 1 -- 172.21.15.29:0/3542674471 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3227) v1 ==== 190+0+0 (secure 0 0 0) 0x7f80a8038100 con 0x7f80b806c040 2023-10-16T15:54:30.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f809806b0b0 msgr2=0x7f809806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:30.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f809806b0b0 0x7f809806d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f80a0001380 tx=0x7f80a0009040 comp rx=0 tx=0).stop 2023-10-16T15:54:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 msgr2=0x7f80b80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b80ac890 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009890 tx=0x7f80a80046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 shutdown_connections 2023-10-16T15:54:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f809806b0b0 0x7f809806d570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80b80cda80 0x7f80b8155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80b80ad970 0x7f80b80c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 --2- 172.21.15.29:0/3542674471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80b806c040 0x7f80b80ac890 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 >> 172.21.15.29:0/3542674471 conn(0x7f80b80ce6a0 msgr2=0x7f80b80b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:30.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 shutdown_connections 2023-10-16T15:54:30.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.603+0000 7f80bce60640 1 -- 172.21.15.29:0/3542674471 wait complete. 2023-10-16T15:54:30.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T15:54:30.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.857+0000 7f6271cec640 1 Processor -- start 2023-10-16T15:54:30.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.857+0000 7f6271cec640 1 -- start start 2023-10-16T15:54:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6271cec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f626c15c9f0 0x7f626c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6271cec640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f626c0c9230 0x7f626c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6271cec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c070f20 0x7f626c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6271cec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c0cdec0 con 0x7f626c15c9f0 2023-10-16T15:54:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6271cec640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c160a90 con 0x7f626c0c9230 2023-10-16T15:54:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6271cec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0cd900 con 0x7f626c070f20 2023-10-16T15:54:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6270cea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f626c15c9f0 0x7f626c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6270cea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f626c15c9f0 0x7f626c0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49316/0 (socket says 172.21.15.29:49316) 2023-10-16T15:54:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.859+0000 7f6270cea640 1 -- 172.21.15.29:0/540832714 learned_addr learned my addr 172.21.15.29:0/540832714 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f62714eb640 1 --2- 172.21.15.29:0/540832714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c070f20 0x7f626c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f6263fff640 1 --2- 172.21.15.29:0/540832714 >> v2:172.21.15.170:3300/0 conn(0x7f626c0c9230 0x7f626c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f6270cea640 1 -- 172.21.15.29:0/540832714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c070f20 msgr2=0x7f626c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f6270cea640 1 --2- 172.21.15.29:0/540832714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c070f20 0x7f626c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f6270cea640 1 -- 172.21.15.29:0/540832714 >> v2:172.21.15.170:3300/0 conn(0x7f626c0c9230 msgr2=0x7f626c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f6270cea640 1 --2- 172.21.15.29:0/540832714 >> v2:172.21.15.170:3300/0 conn(0x7f626c0c9230 0x7f626c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.860+0000 7f6270cea640 1 -- 172.21.15.29:0/540832714 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0cdbe0 con 0x7f626c15c9f0 2023-10-16T15:54:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.862+0000 7f6270cea640 1 --2- 172.21.15.29:0/540832714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c15c9f0 0x7f626c0c8cf0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f626c0b1260 tx=0x7f626802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8ce323acf8eba6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f62637fe640 1 -- 172.21.15.29:0/540832714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626802ca70 con 0x7f626c15c9f0 2023-10-16T15:54:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f62637fe640 1 -- 172.21.15.29:0/540832714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626802cbd0 con 0x7f626c15c9f0 2023-10-16T15:54:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f62637fe640 1 -- 172.21.15.29:0/540832714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62680343f0 con 0x7f626c15c9f0 2023-10-16T15:54:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 -- 172.21.15.29:0/540832714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c15c9f0 msgr2=0x7f626c0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 --2- 172.21.15.29:0/540832714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c15c9f0 0x7f626c0c8cf0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f626c0b1260 tx=0x7f626802bed0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 -- 172.21.15.29:0/540832714 shutdown_connections 2023-10-16T15:54:30.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 --2- 172.21.15.29:0/540832714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c070f20 0x7f626c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 --2- 172.21.15.29:0/540832714 >> v2:172.21.15.170:3300/0 conn(0x7f626c0c9230 0x7f626c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 --2- 172.21.15.29:0/540832714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c15c9f0 0x7f626c0c8cf0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 -- 172.21.15.29:0/540832714 >> 172.21.15.29:0/540832714 conn(0x7f626c0ce6a0 msgr2=0x7f626c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 -- 172.21.15.29:0/540832714 shutdown_connections 2023-10-16T15:54:30.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.863+0000 7f6271cec640 1 -- 172.21.15.29:0/540832714 wait complete. 2023-10-16T15:54:30.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.865+0000 7f6271cec640 1 Processor -- start 2023-10-16T15:54:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.865+0000 7f6271cec640 1 -- start start 2023-10-16T15:54:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.866+0000 7f6271cec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 0x7f626c159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.866+0000 7f6271cec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f626c0c9230 0x7f626c0c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.866+0000 7f6271cec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c15c9f0 0x7f626c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.866+0000 7f6271cec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c6490 con 0x7f626c070f20 2023-10-16T15:54:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.866+0000 7f6271cec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c6770 con 0x7f626c0c9230 2023-10-16T15:54:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.866+0000 7f6271cec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c6a50 con 0x7f626c15c9f0 2023-10-16T15:54:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 0x7f626c159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 0x7f626c159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49322/0 (socket says 172.21.15.29:49322) 2023-10-16T15:54:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 -- 172.21.15.29:0/1297882462 learned_addr learned my addr 172.21.15.29:0/1297882462 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:30.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6263fff640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f626c0c9230 0x7f626c0c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f62714eb640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c15c9f0 0x7f626c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 -- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c15c9f0 msgr2=0x7f626c0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:30.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c15c9f0 0x7f626c0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 -- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f626c0c9230 msgr2=0x7f626c0c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:30.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f626c0c9230 0x7f626c0c5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:30.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.867+0000 7f6270cea640 1 -- 172.21.15.29:0/1297882462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268009b30 con 0x7f626c070f20 2023-10-16T15:54:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.868+0000 7f6270cea640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 0x7f626c159b40 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f6268036040 tx=0x7f6268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.868+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268009510 con 0x7f626c070f20 2023-10-16T15:54:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.868+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626803e450 con 0x7f626c070f20 2023-10-16T15:54:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.868+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626803e800 con 0x7f626c070f20 2023-10-16T15:54:30.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.868+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c0c6d30 con 0x7f626c070f20 2023-10-16T15:54:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.868+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c0c33e0 con 0x7f626c070f20 2023-10-16T15:54:30.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.869+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c0c3570 con 0x7f626c070f20 2023-10-16T15:54:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.870+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f626803e960 con 0x7f626c070f20 2023-10-16T15:54:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.871+0000 7f62617fa640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f625806b0b0 0x7f625806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.871+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f62680ac750 con 0x7f626c070f20 2023-10-16T15:54:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.872+0000 7f6263fff640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f625806b0b0 0x7f625806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.873+0000 7f6263fff640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f625806b0b0 0x7f625806d570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f626c0cd1d0 tx=0x7f625c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:30.875+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6268077480 con 0x7f626c070f20 2023-10-16T15:54:31.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.067+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f626c04e9b0 con 0x7f626c070f20 2023-10-16T15:54:31.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.075+0000 7f62617fa640 1 -- 172.21.15.29:0/1297882462 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f626807aaa0 con 0x7f626c070f20 2023-10-16T15:54:31.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f625806b0b0 msgr2=0x7f625806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:31.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f625806b0b0 0x7f625806d570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f626c0cd1d0 tx=0x7f625c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:31.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 msgr2=0x7f626c159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:31.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 0x7f626c159b40 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f6268036040 tx=0x7f6268004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:31.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 shutdown_connections 2023-10-16T15:54:31.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f625806b0b0 0x7f625806d570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f626c15c9f0 0x7f626c0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f626c0c9230 0x7f626c0c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 --2- 172.21.15.29:0/1297882462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f626c070f20 0x7f626c159b40 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 >> 172.21.15.29:0/1297882462 conn(0x7f626c0ce6a0 msgr2=0x7f626c06bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:31.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 shutdown_connections 2023-10-16T15:54:31.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.078+0000 7f6271cec640 1 -- 172.21.15.29:0/1297882462 wait complete. 2023-10-16T15:54:31.098 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:54:31.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:54:31.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:54:31.343 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:54:31.343 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:54:31.344 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:31.408 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:54:31.408 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:54:31.408 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:31.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:54:31.468 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:54:31.468 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:31.534 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:54:31.534 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:54:31.534 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:31.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:54:31.594 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T15:54:31.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:31.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.829+0000 7f6d2ce8a640 1 Processor -- start 2023-10-16T15:54:31.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.829+0000 7f6d2ce8a640 1 -- start start 2023-10-16T15:54:31.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.830+0000 7f6d2ce8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 0x7f6d280d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.830+0000 7f6d2ce8a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6d280b4c80 0x7f6d280b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.830+0000 7f6d2ce8a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d280d15a0 con 0x7f6d2815cfc0 2023-10-16T15:54:31.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.830+0000 7f6d2ce8a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d280d1880 con 0x7f6d280b4c80 2023-10-16T15:54:31.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.830+0000 7f6d2ce8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280d1b60 con 0x7f6d2815cbe0 2023-10-16T15:54:31.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d1ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6d2815cfc0 0x7f6d280b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50296/0 (socket says 172.21.15.29:50296) 2023-10-16T15:54:31.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d1ffff640 1 -- 172.21.15.29:0/4158737004 learned_addr learned my addr 172.21.15.29:0/4158737004 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:31.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d277fe640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 0x7f6d280d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d27fff640 1 --2- 172.21.15.29:0/4158737004 >> v2:172.21.15.170:3300/0 conn(0x7f6d280b4c80 0x7f6d280b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104949185 0 0) 0x7f6d280d15a0 con 0x7f6d2815cfc0 2023-10-16T15:54:31.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d14003520 con 0x7f6d2815cfc0 2023-10-16T15:54:31.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384671570 0 0) 0x7f6d14003520 con 0x7f6d2815cfc0 2023-10-16T15:54:31.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 msgr2=0x7f6d280d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:31.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 0x7f6d280d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 >> v2:172.21.15.170:3300/0 conn(0x7f6d280b4c80 msgr2=0x7f6d280b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:31.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.831+0000 7f6d26ffd640 1 --2- 172.21.15.29:0/4158737004 >> v2:172.21.15.170:3300/0 conn(0x7f6d280b4c80 0x7f6d280b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d280d1e40 con 0x7f6d2815cfc0 2023-10-16T15:54:31.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6d10001670 con 0x7f6d2815cfc0 2023-10-16T15:54:31.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 >> v1:172.21.15.29:6789/0 conn(0x7f6d2815cfc0 legacy=0x7f6d280b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:31.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 0x7f6d14003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d14003f40 0x7f6d140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d14004d00 0x7f6d14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280d1880 con 0x7f6d14004d00 2023-10-16T15:54:31.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280d1b60 con 0x7f6d14003f40 2023-10-16T15:54:31.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1400bac0 con 0x7f6d2815cbe0 2023-10-16T15:54:31.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d27fff640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d14004d00 0x7f6d14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d277fe640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 0x7f6d14003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.832+0000 7f6d1ffff640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d14003f40 0x7f6d140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.833+0000 7f6d27fff640 1 -- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 msgr2=0x7f6d14003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:31.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.833+0000 7f6d27fff640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d2815cbe0 0x7f6d14003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.833+0000 7f6d27fff640 1 -- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d14003f40 msgr2=0x7f6d140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:31.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.833+0000 7f6d27fff640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d14003f40 0x7f6d140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.833+0000 7f6d27fff640 1 -- 172.21.15.29:0/4158737004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d14003520 con 0x7f6d14004d00 2023-10-16T15:54:31.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.833+0000 7f6d1ffff640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d14003f40 0x7f6d140047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:31.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.834+0000 7f6d27fff640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d14004d00 0x7f6d14005180 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009f90 tx=0x7f6d140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c5d2693ddbd76a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:31.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.834+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002cc20 con 0x7f6d14004d00 2023-10-16T15:54:31.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2000f040 con 0x7f6d14004d00 2023-10-16T15:54:31.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d26ffd640 1 -- 172.21.15.29:0/4158737004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20034440 con 0x7f6d14004d00 2023-10-16T15:54:31.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/4158737004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d14004d00 msgr2=0x7f6d14005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:31.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/4158737004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d14004d00 0x7f6d14005180 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009f90 tx=0x7f6d140057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d27fff640 1 -- 172.21.15.29:0/4158737004 reap_dead start 2023-10-16T15:54:31.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/4158737004 shutdown_connections 2023-10-16T15:54:31.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/4158737004 >> 172.21.15.29:0/4158737004 conn(0x7f6d280cf570 msgr2=0x7f6d280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/4158737004 shutdown_connections 2023-10-16T15:54:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.835+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/4158737004 wait complete. 2023-10-16T15:54:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.837+0000 7f6d2ce8a640 1 Processor -- start 2023-10-16T15:54:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.837+0000 7f6d2ce8a640 1 -- start start 2023-10-16T15:54:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.838+0000 7f6d2ce8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 0x7f6d280c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.838+0000 7f6d2ce8a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d281f3000 0x7f6d280d2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.838+0000 7f6d2ce8a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d280d2b80 0x7f6d2815e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.838+0000 7f6d2ce8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280c94b0 con 0x7f6d28155350 2023-10-16T15:54:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.838+0000 7f6d2ce8a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280d3000 con 0x7f6d281f3000 2023-10-16T15:54:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.838+0000 7f6d2ce8a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280c8930 con 0x7f6d280d2b80 2023-10-16T15:54:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 0x7f6d280c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 0x7f6d280c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49346/0 (socket says 172.21.15.29:49346) 2023-10-16T15:54:31.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 -- 172.21.15.29:0/1649649991 learned_addr learned my addr 172.21.15.29:0/1649649991 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:31.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d1ffff640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d281f3000 0x7f6d280d2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d27fff640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d280d2b80 0x7f6d2815e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 -- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d280d2b80 msgr2=0x7f6d2815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:31.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d280d2b80 0x7f6d2815e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 -- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d281f3000 msgr2=0x7f6d280d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d281f3000 0x7f6d280d2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 -- 172.21.15.29:0/1649649991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d280c8c10 con 0x7f6d28155350 2023-10-16T15:54:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d1ffff640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d281f3000 0x7f6d280d2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.839+0000 7f6d277fe640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 0x7f6d280c8290 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800b7f0 tx=0x7f6d1800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.840+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18010830 con 0x7f6d28155350 2023-10-16T15:54:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.840+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d18010990 con 0x7f6d28155350 2023-10-16T15:54:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.840+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18010d20 con 0x7f6d28155350 2023-10-16T15:54:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.840+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d280c8ef0 con 0x7f6d28155350 2023-10-16T15:54:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.840+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d281f3460 con 0x7f6d28155350 2023-10-16T15:54:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.841+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2804e9b0 con 0x7f6d28155350 2023-10-16T15:54:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.842+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6d18014430 con 0x7f6d28155350 2023-10-16T15:54:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.843+0000 7f6d24ff9640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d0806b0b0 0x7f6d0806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.843+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f6d1808a310 con 0x7f6d28155350 2023-10-16T15:54:31.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.844+0000 7f6d1ffff640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d0806b0b0 0x7f6d0806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.845+0000 7f6d1ffff640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d0806b0b0 0x7f6d0806d570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000c00 tx=0x7f6d10001ad0 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:31.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:31.847+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d18054fa0 con 0x7f6d28155350 2023-10-16T15:54:32.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.078+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d281f3740 con 0x7f6d28155350 2023-10-16T15:54:32.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.079+0000 7f6d24ff9640 1 -- 172.21.15.29:0/1649649991 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3705 (secure 0 0 0) 0x7f6d18022e90 con 0x7f6d28155350 2023-10-16T15:54:32.081 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:32.081 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":118,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6833/1954049131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1954049131},{"type":"v1","addr":"172.21.15.29:6833","nonce":1954049131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103},{"gid":7817,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/1122068174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":1122068174},{"type":"v1","addr":"172.21.15.170:6836","nonce":1122068174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104},{"gid":9187,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/994997589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":994997589},{"type":"v1","addr":"172.21.15.170:6837","nonce":994997589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":11418,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/248379662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":248379662},{"type":"v1","addr":"172.21.15.29:6835","nonce":248379662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[]} 2023-10-16T15:54:32.081 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 118 2023-10-16T15:54:32.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d0806b0b0 msgr2=0x7f6d0806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:32.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d0806b0b0 0x7f6d0806d570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000c00 tx=0x7f6d10001ad0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 msgr2=0x7f6d280c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:32.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 0x7f6d280c8290 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800b7f0 tx=0x7f6d1800bcb0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 shutdown_connections 2023-10-16T15:54:32.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6d0806b0b0 0x7f6d0806d570 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d280d2b80 0x7f6d2815e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d281f3000 0x7f6d280d2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 --2- 172.21.15.29:0/1649649991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d28155350 0x7f6d280c8290 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 >> 172.21.15.29:0/1649649991 conn(0x7f6d280cf570 msgr2=0x7f6d280b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:32.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.081+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 shutdown_connections 2023-10-16T15:54:32.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.082+0000 7f6d2ce8a640 1 -- 172.21.15.29:0/1649649991 wait complete. 2023-10-16T15:54:32.102 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T15:54:32.103 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T15:54:32.103 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:54:32.103 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:54:32.104 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T15:54:32.104 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T15:54:32.104 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:54:32.104 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:54:32.105 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T15:54:32.105 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T15:54:32.105 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:54:32.106 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:54:32.106 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T15:54:32.106 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T15:54:32.106 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:54:32.107 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:54:32.120 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:54:32.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T15:54:32.122 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:54:32.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T15:54:32.129 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:54:32.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T15:54:32.131 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:54:32.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T15:54:32.169 INFO:tasks.ceph.mds.c:Started 2023-10-16T15:54:32.170 DEBUG:teuthology.parallel:result is None 2023-10-16T15:54:32.177 INFO:tasks.ceph.mds.a:Started 2023-10-16T15:54:32.178 DEBUG:teuthology.parallel:result is None 2023-10-16T15:54:32.178 INFO:tasks.ceph.mds.b:Started 2023-10-16T15:54:32.179 DEBUG:teuthology.parallel:result is None 2023-10-16T15:54:32.186 INFO:tasks.ceph.mds.d:Started 2023-10-16T15:54:32.187 DEBUG:teuthology.parallel:result is None 2023-10-16T15:54:32.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T15:54:32.337 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:54:32.340 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:54:32.343 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:54:32.344 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T15:54:32.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.383+0000 7f2e00f05640 1 Processor -- start 2023-10-16T15:54:32.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.383+0000 7f2e00f05640 1 -- start start 2023-10-16T15:54:32.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2e00f05640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2dfc15ad70 0x7f2dfc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:32.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2e00f05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:32.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2e00f05640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc0cb030 con 0x7f2dfc0ca3b0 2023-10-16T15:54:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2e00f05640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0c91c0 con 0x7f2dfc15ad70 2023-10-16T15:54:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2e00f05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0c94a0 con 0x7f2dfc0c9ca0 2023-10-16T15:54:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2dfa59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2dfc15ad70 0x7f2dfc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2dfa59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2dfc15ad70 0x7f2dfc069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49382/0 (socket says 172.21.15.29:49382) 2023-10-16T15:54:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2dfa59c640 1 -- 172.21.15.29:0/1061226060 learned_addr learned my addr 172.21.15.29:0/1061226060 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2dfad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2dfc0ca3b0 0x7f2dfc0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50310/0 (socket says 172.21.15.29:50310) 2023-10-16T15:54:32.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.384+0000 7f2df3fff640 1 --2- 172.21.15.29:0/1061226060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:32.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.385+0000 7f2dfa59c640 1 -- 172.21.15.29:0/1061226060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 msgr2=0x7f2dfc06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:32.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.385+0000 7f2dfa59c640 1 --2- 172.21.15.29:0/1061226060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.385+0000 7f2dfa59c640 1 -- 172.21.15.29:0/1061226060 >> v1:172.21.15.29:6789/0 conn(0x7f2dfc0ca3b0 legacy=0x7f2dfc0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:32.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.385+0000 7f2dfa59c640 1 -- 172.21.15.29:0/1061226060 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0c9780 con 0x7f2dfc15ad70 2023-10-16T15:54:32.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.386+0000 7f2dfa59c640 1 --2- 172.21.15.29:0/1061226060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc069da0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009870 tx=0x7f2dec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86776bdd7ab9dc00 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:32.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.386+0000 7f2df9d9b640 1 -- 172.21.15.29:0/1061226060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec030840 con 0x7f2dfc15ad70 2023-10-16T15:54:32.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.386+0000 7f2df9d9b640 1 -- 172.21.15.29:0/1061226060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec0309a0 con 0x7f2dfc15ad70 2023-10-16T15:54:32.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.386+0000 7f2df9d9b640 1 -- 172.21.15.29:0/1061226060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec030d30 con 0x7f2dfc15ad70 2023-10-16T15:54:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 -- 172.21.15.29:0/1061226060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 msgr2=0x7f2dfc069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 --2- 172.21.15.29:0/1061226060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc069da0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009870 tx=0x7f2dec02b980 comp rx=0 tx=0).stop 2023-10-16T15:54:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 -- 172.21.15.29:0/1061226060 shutdown_connections 2023-10-16T15:54:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 --2- 172.21.15.29:0/1061226060 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 --2- 172.21.15.29:0/1061226060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc069da0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 -- 172.21.15.29:0/1061226060 >> 172.21.15.29:0/1061226060 conn(0x7f2dfc0ce9b0 msgr2=0x7f2dfc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 -- 172.21.15.29:0/1061226060 shutdown_connections 2023-10-16T15:54:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.387+0000 7f2e00f05640 1 -- 172.21.15.29:0/1061226060 wait complete. 2023-10-16T15:54:32.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.389+0000 7f2e00f05640 1 Processor -- start 2023-10-16T15:54:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.389+0000 7f2e00f05640 1 -- start start 2023-10-16T15:54:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2e00f05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2e00f05640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2dfc0ca3b0 0x7f2dfc0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:32.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2e00f05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc15b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2e00f05640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0b3260 con 0x7f2dfc15ad70 2023-10-16T15:54:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2e00f05640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0b3540 con 0x7f2dfc0ca3b0 2023-10-16T15:54:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2e00f05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0b3820 con 0x7f2dfc0c9ca0 2023-10-16T15:54:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2dfad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc15b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2dfad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc15b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49384/0 (socket says 172.21.15.29:49384) 2023-10-16T15:54:32.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2dfad9d640 1 -- 172.21.15.29:0/2715476717 learned_addr learned my addr 172.21.15.29:0/2715476717 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:32.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2dfa59c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:32.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.390+0000 7f2df3fff640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2dfc0ca3b0 0x7f2dfc0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:32.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfad9d640 1 -- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 msgr2=0x7f2dfc153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:32.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfad9d640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfad9d640 1 -- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2dfc0ca3b0 msgr2=0x7f2dfc0b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:32.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfad9d640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2dfc0ca3b0 0x7f2dfc0b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfad9d640 1 -- 172.21.15.29:0/2715476717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0094d0 con 0x7f2dfc15ad70 2023-10-16T15:54:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfa59c640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2dfad9d640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc15b4e0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f2df400c780 tx=0x7f2df400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df400ce90 con 0x7f2dfc15ad70 2023-10-16T15:54:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400a640 con 0x7f2dfc15ad70 2023-10-16T15:54:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df400a9f0 con 0x7f2dfc15ad70 2023-10-16T15:54:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc0b3b00 con 0x7f2dfc15ad70 2023-10-16T15:54:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.391+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc154470 con 0x7f2dfc15ad70 2023-10-16T15:54:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.392+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc0b4020 con 0x7f2dfc15ad70 2023-10-16T15:54:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.394+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2df400ab50 con 0x7f2dfc15ad70 2023-10-16T15:54:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.394+0000 7f2df2ffd640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2ddc06b0b0 0x7f2ddc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.395+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5874+0+0 (secure 0 0 0) 0x7f2df4089280 con 0x7f2dfc15ad70 2023-10-16T15:54:32.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.395+0000 7f2dfa59c640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2ddc06b0b0 0x7f2ddc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:32.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.396+0000 7f2dfa59c640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2ddc06b0b0 0x7f2ddc06d570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2dec036000 tx=0x7f2dec008030 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.399+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2dfc0b4020 con 0x7f2dfc15ad70 2023-10-16T15:54:32.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:32.607+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2dfc0b4020 con 0x7f2dfc15ad70 2023-10-16T15:54:34.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.111+0000 7f2df2ffd640 1 -- 172.21.15.29:0/2715476717 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v242) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2dfc0b4020 con 0x7f2dfc15ad70 2023-10-16T15:54:34.113 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T15:54:34.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2ddc06b0b0 msgr2=0x7f2ddc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2ddc06b0b0 0x7f2ddc06d570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2dec036000 tx=0x7f2dec008030 comp rx=0 tx=0).stop 2023-10-16T15:54:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 msgr2=0x7f2dfc15b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc15b4e0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f2df400c780 tx=0x7f2df400cc40 comp rx=0 tx=0).stop 2023-10-16T15:54:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 shutdown_connections 2023-10-16T15:54:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2ddc06b0b0 0x7f2ddc06d570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc15ad70 0x7f2dfc15b4e0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2dfc0ca3b0 0x7f2dfc0b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 --2- 172.21.15.29:0/2715476717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2dfc0c9ca0 0x7f2dfc153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 >> 172.21.15.29:0/2715476717 conn(0x7f2dfc0ce9b0 msgr2=0x7f2dfc15dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:34.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.115+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 shutdown_connections 2023-10-16T15:54:34.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.116+0000 7f2e00f05640 1 -- 172.21.15.29:0/2715476717 wait complete. 2023-10-16T15:54:34.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T15:54:34.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.357+0000 7f35572bb640 1 Processor -- start 2023-10-16T15:54:34.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.357+0000 7f35572bb640 1 -- start start 2023-10-16T15:54:34.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35572bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3550159950 0x7f355015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35572bb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3550070450 0x7f35500708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35572bb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355015aef0 con 0x7f355006c100 2023-10-16T15:54:34.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35572bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3550070e20 con 0x7f3550159950 2023-10-16T15:54:34.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35572bb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355015e490 con 0x7f3550070450 2023-10-16T15:54:34.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35562b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3550159950 0x7f355015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35562b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3550159950 0x7f355015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56032/0 (socket says 172.21.15.29:56032) 2023-10-16T15:54:34.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.359+0000 7f35562b9640 1 -- 172.21.15.29:0/3554834088 learned_addr learned my addr 172.21.15.29:0/3554834088 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:34.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.360+0000 7f35562b9640 1 -- 172.21.15.29:0/3554834088 >> v2:172.21.15.170:3300/0 conn(0x7f3550070450 msgr2=0x7f35500708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:34.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.360+0000 7f35562b9640 1 --2- 172.21.15.29:0/3554834088 >> v2:172.21.15.170:3300/0 conn(0x7f3550070450 0x7f35500708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.360+0000 7f35562b9640 1 -- 172.21.15.29:0/3554834088 >> v1:172.21.15.170:6789/0 conn(0x7f355006c100 legacy=0x7f355006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:34.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.360+0000 7f35562b9640 1 -- 172.21.15.29:0/3554834088 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355015d910 con 0x7f3550159950 2023-10-16T15:54:34.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.361+0000 7f35562b9640 1 --2- 172.21.15.29:0/3554834088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159950 0x7f355015a9b0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f35500cab70 tx=0x7f354c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab40a20b93be8b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:34.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.361+0000 7f35552b7640 1 -- 172.21.15.29:0/3554834088 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c02c9f0 con 0x7f3550159950 2023-10-16T15:54:34.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.361+0000 7f35552b7640 1 -- 172.21.15.29:0/3554834088 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c02cb50 con 0x7f3550159950 2023-10-16T15:54:34.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.361+0000 7f35552b7640 1 -- 172.21.15.29:0/3554834088 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0343f0 con 0x7f3550159950 2023-10-16T15:54:34.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 -- 172.21.15.29:0/3554834088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159950 msgr2=0x7f355015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 --2- 172.21.15.29:0/3554834088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159950 0x7f355015a9b0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f35500cab70 tx=0x7f354c02be50 comp rx=0 tx=0).stop 2023-10-16T15:54:34.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 -- 172.21.15.29:0/3554834088 shutdown_connections 2023-10-16T15:54:34.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 --2- 172.21.15.29:0/3554834088 >> v2:172.21.15.170:3300/0 conn(0x7f3550070450 0x7f35500708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 --2- 172.21.15.29:0/3554834088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159950 0x7f355015a9b0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 -- 172.21.15.29:0/3554834088 >> 172.21.15.29:0/3554834088 conn(0x7f35500ce6a0 msgr2=0x7f35500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:34.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 -- 172.21.15.29:0/3554834088 shutdown_connections 2023-10-16T15:54:34.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.362+0000 7f35572bb640 1 -- 172.21.15.29:0/3554834088 wait complete. 2023-10-16T15:54:34.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.364+0000 7f35572bb640 1 Processor -- start 2023-10-16T15:54:34.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.364+0000 7f35572bb640 1 -- start start 2023-10-16T15:54:34.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.365+0000 7f35572bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 0x7f35500c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35572bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3550070450 0x7f3550158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35572bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3550159950 0x7f35500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35572bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c7200 con 0x7f355006c100 2023-10-16T15:54:34.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35572bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c65f0 con 0x7f3550070450 2023-10-16T15:54:34.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35572bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c68d0 con 0x7f3550159950 2023-10-16T15:54:34.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 0x7f35500c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 0x7f35500c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56044/0 (socket says 172.21.15.29:56044) 2023-10-16T15:54:34.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 -- 172.21.15.29:0/2413535073 learned_addr learned my addr 172.21.15.29:0/2413535073 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:34.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 -- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3550159950 msgr2=0x7f35500ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3550159950 0x7f35500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 -- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3550070450 msgr2=0x7f3550158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3550070450 0x7f3550158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.366+0000 7f35562b9640 1 -- 172.21.15.29:0/2413535073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c009b30 con 0x7f355006c100 2023-10-16T15:54:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.367+0000 7f35562b9640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 0x7f35500c7690 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f354c0093e0 tx=0x7f354c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.367+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c03e450 con 0x7f355006c100 2023-10-16T15:54:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.367+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c03e5b0 con 0x7f355006c100 2023-10-16T15:54:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.367+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c03e960 con 0x7f355006c100 2023-10-16T15:54:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.367+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35500c6bb0 con 0x7f355006c100 2023-10-16T15:54:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.367+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3550158cb0 con 0x7f355006c100 2023-10-16T15:54:34.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.369+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35500c6a10 con 0x7f355006c100 2023-10-16T15:54:34.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.369+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f354c03eac0 con 0x7f355006c100 2023-10-16T15:54:34.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.370+0000 7f3546ffd640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f353406b0b0 0x7f353406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.371+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f354c0ac730 con 0x7f355006c100 2023-10-16T15:54:34.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.371+0000 7f3555ab8640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f353406b0b0 0x7f353406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.372+0000 7f3555ab8640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f353406b0b0 0x7f353406d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f355015d0c0 tx=0x7f3540006c90 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:34.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.375+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f35500c6a10 con 0x7f355006c100 2023-10-16T15:54:34.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.644+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f35500c6a10 con 0x7f355006c100 2023-10-16T15:54:34.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.645+0000 7f3546ffd640 1 -- 172.21.15.29:0/2413535073 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f35500c6a10 con 0x7f355006c100 2023-10-16T15:54:34.648 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:34.648 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:54:34.648 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T15:54:34.648 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.648 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T15:54:34.649 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T15:54:34.649 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.649 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.649 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:54:34.649 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:54:34.650 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.650 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.650 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.650 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.651 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T15:54:34.651 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T15:54:34.651 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.651 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.651 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:54:34.652 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:54:34.652 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.652 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.652 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.652 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T15:54:34.653 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T15:54:34.653 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.653 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.653 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:54:34.653 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:54:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T15:54:34.655 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T15:54:34.655 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.655 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.655 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:54:34.655 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:54:34.656 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.656 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.656 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.656 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T15:54:34.657 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T15:54:34.657 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.657 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.657 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.657 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.658 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.658 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.658 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.658 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T15:54:34.658 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T15:54:34.659 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.659 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.659 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.659 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.659 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.660 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T15:54:34.660 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T15:54:34.660 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.660 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.661 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.661 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.661 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.661 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.662 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T15:54:34.662 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T15:54:34.662 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.662 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.662 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.663 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.663 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.663 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T15:54:34.664 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T15:54:34.664 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.664 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.664 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.664 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.665 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T15:54:34.665 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T15:54:34.665 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.666 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.666 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.666 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.667 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T15:54:34.667 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T15:54:34.667 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.667 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.668 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.668 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.668 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.668 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T15:54:34.669 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T15:54:34.669 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.669 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:54:34.669 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:54:34.669 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.670 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.670 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.670 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T15:54:34.670 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T15:54:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T15:54:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T15:54:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T15:54:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T15:54:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.673 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:54:34.673 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:54:34.673 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:54:34.673 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.674 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.674 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T15:54:34.674 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T15:54:34.674 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.674 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.675 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:54:34.675 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:54:34.675 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:54:34.675 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.676 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T15:54:34.676 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T15:54:34.676 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.676 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:54:34.677 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:54:34.677 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:54:34.677 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:54:34.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.678 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T15:54:34.678 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T15:54:34.678 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.678 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:54:34.678 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:54:34.679 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:54:34.679 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.679 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.680 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T15:54:34.680 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T15:54:34.680 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:54:34.680 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:54:34.680 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.681 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:34.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:34.681 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T15:54:34.681 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T15:54:34.682 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:54:34.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:54:34.682 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:54:34.682 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:54:34.682 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.683 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:34.683 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T15:54:34.683 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:54:34.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f353406b0b0 msgr2=0x7f353406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f353406b0b0 0x7f353406d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f355015d0c0 tx=0x7f3540006c90 comp rx=0 tx=0).stop 2023-10-16T15:54:34.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 msgr2=0x7f35500c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 0x7f35500c7690 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f354c0093e0 tx=0x7f354c004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:34.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 shutdown_connections 2023-10-16T15:54:34.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f353406b0b0 0x7f353406d570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3550159950 0x7f35500ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3550070450 0x7f3550158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 --2- 172.21.15.29:0/2413535073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f355006c100 0x7f35500c7690 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.648+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 >> 172.21.15.29:0/2413535073 conn(0x7f35500ce6a0 msgr2=0x7f355015a690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:34.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.649+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 shutdown_connections 2023-10-16T15:54:34.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.649+0000 7f35572bb640 1 -- 172.21.15.29:0/2413535073 wait complete. 2023-10-16T15:54:34.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:34.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.916+0000 7fa84359e640 1 Processor -- start 2023-10-16T15:54:34.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.917+0000 7fa84359e640 1 -- start start 2023-10-16T15:54:34.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.918+0000 7fa84359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 0x7fa8440d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.918+0000 7fa84359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa8440b4c80 0x7fa8440b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.918+0000 7fa84359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8440d15a0 con 0x7fa84415cfc0 2023-10-16T15:54:34.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.918+0000 7fa84359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8440d1880 con 0x7fa8440b4c80 2023-10-16T15:54:34.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.918+0000 7fa84359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440d1b60 con 0x7fa84415cbe0 2023-10-16T15:54:34.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.919+0000 7fa841d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa84415cfc0 0x7fa8440b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53176/0 (socket says 172.21.15.29:53176) 2023-10-16T15:54:34.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.919+0000 7fa841d9b640 1 -- 172.21.15.29:0/627065734 learned_addr learned my addr 172.21.15.29:0/627065734 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:34.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.919+0000 7fa84259c640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 0x7fa8440d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.919+0000 7fa842d9d640 1 --2- 172.21.15.29:0/627065734 >> v2:172.21.15.170:3300/0 conn(0x7fa8440b4c80 0x7fa8440b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.919+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056046021 0 0) 0x7fa8440d15a0 con 0x7fa84415cfc0 2023-10-16T15:54:34.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.919+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c003520 con 0x7fa84415cfc0 2023-10-16T15:54:34.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4273536667 0 0) 0x7fa82c003520 con 0x7fa84415cfc0 2023-10-16T15:54:34.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 msgr2=0x7fa8440d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:34.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 0x7fa8440d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 >> v2:172.21.15.170:3300/0 conn(0x7fa8440b4c80 msgr2=0x7fa8440b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:34.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 --2- 172.21.15.29:0/627065734 >> v2:172.21.15.170:3300/0 conn(0x7fa8440b4c80 0x7fa8440b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8440d1e40 con 0x7fa84415cfc0 2023-10-16T15:54:34.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa838001670 con 0x7fa84415cfc0 2023-10-16T15:54:34.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 >> v1:172.21.15.29:6789/0 conn(0x7fa84415cfc0 legacy=0x7fa8440b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c003970 0x7fa82c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa82c0047c0 0x7fa82c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 0x7fa82c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440d1880 con 0x7fa82c0047c0 2023-10-16T15:54:34.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440d1b60 con 0x7fa82c003970 2023-10-16T15:54:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c00b9b0 con 0x7fa84415cbe0 2023-10-16T15:54:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.920+0000 7fa841d9b640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa82c0047c0 0x7fa82c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa841d9b640 1 -- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 msgr2=0x7fa82c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa841d9b640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa84415cbe0 0x7fa82c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa841d9b640 1 -- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c003970 msgr2=0x7fa82c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa84259c640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c003970 0x7fa82c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa841d9b640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c003970 0x7fa82c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa841d9b640 1 -- 172.21.15.29:0/627065734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c003520 con 0x7fa82c0047c0 2023-10-16T15:54:34.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.921+0000 7fa84259c640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c003970 0x7fa82c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.922+0000 7fa841d9b640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa82c0047c0 0x7fa82c004c40 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fa83800c0e0 tx=0x7fa82c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8331af8bd93372f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.922+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838005ad0 con 0x7fa82c0047c0 2023-10-16T15:54:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.922+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838005c50 con 0x7fa82c0047c0 2023-10-16T15:54:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.922+0000 7fa84159a640 1 -- 172.21.15.29:0/627065734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83802c810 con 0x7fa82c0047c0 2023-10-16T15:54:34.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa84359e640 1 -- 172.21.15.29:0/627065734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa82c0047c0 msgr2=0x7fa82c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa84359e640 1 --2- 172.21.15.29:0/627065734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa82c0047c0 0x7fa82c004c40 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fa83800c0e0 tx=0x7fa82c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa842d9d640 1 -- 172.21.15.29:0/627065734 reap_dead start 2023-10-16T15:54:34.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa84359e640 1 -- 172.21.15.29:0/627065734 shutdown_connections 2023-10-16T15:54:34.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa84359e640 1 -- 172.21.15.29:0/627065734 >> 172.21.15.29:0/627065734 conn(0x7fa8440cf570 msgr2=0x7fa8440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:34.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa84359e640 1 -- 172.21.15.29:0/627065734 shutdown_connections 2023-10-16T15:54:34.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.923+0000 7fa84359e640 1 -- 172.21.15.29:0/627065734 wait complete. 2023-10-16T15:54:34.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.925+0000 7fa84359e640 1 Processor -- start 2023-10-16T15:54:34.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.925+0000 7fa84359e640 1 -- start start 2023-10-16T15:54:34.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa84359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c0047c0 0x7fa8440cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa84359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8440c7510 0x7fa8440c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa84359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 0x7fa8441e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa84359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8441e72d0 con 0x7fa8441e6910 2023-10-16T15:54:34.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa84359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84415fcf0 con 0x7fa82c0047c0 2023-10-16T15:54:34.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa84359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa844160180 con 0x7fa8440c7510 2023-10-16T15:54:34.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa842d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 0x7fa8441e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa842d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 0x7fa8441e6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56062/0 (socket says 172.21.15.29:56062) 2023-10-16T15:54:34.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.926+0000 7fa842d9d640 1 -- 172.21.15.29:0/1524110024 learned_addr learned my addr 172.21.15.29:0/1524110024 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:34.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa84259c640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c0047c0 0x7fa8440cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa841d9b640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8440c7510 0x7fa8440c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa842d9d640 1 -- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8440c7510 msgr2=0x7fa8440c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa842d9d640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8440c7510 0x7fa8440c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa842d9d640 1 -- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c0047c0 msgr2=0x7fa8440cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:34.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa842d9d640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c0047c0 0x7fa8440cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:34.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa842d9d640 1 -- 172.21.15.29:0/1524110024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84415f0e0 con 0x7fa8441e6910 2023-10-16T15:54:34.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa841d9b640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8440c7510 0x7fa8440c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:34.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa842d9d640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 0x7fa8441e6d90 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fa8440b5b70 tx=0x7fa83c00e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:34.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c013680 con 0x7fa8441e6910 2023-10-16T15:54:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c0137e0 con 0x7fa8441e6910 2023-10-16T15:54:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c013b90 con 0x7fa8441e6910 2023-10-16T15:54:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.927+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84415f3c0 con 0x7fa8441e6910 2023-10-16T15:54:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.928+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84415f980 con 0x7fa8441e6910 2023-10-16T15:54:34.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.929+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84404e9b0 con 0x7fa8441e6910 2023-10-16T15:54:34.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.930+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa83c013cf0 con 0x7fa8441e6910 2023-10-16T15:54:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.931+0000 7fa832ffd640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa82406b0b0 0x7fa82406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.931+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fa83c089470 con 0x7fa8441e6910 2023-10-16T15:54:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.931+0000 7fa84259c640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa82406b0b0 0x7fa82406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.933+0000 7fa84259c640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa82406b0b0 0x7fa82406d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa834003b60 tx=0x7fa83400b040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:34.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:34.935+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa83c054450 con 0x7fa8441e6910 2023-10-16T15:54:35.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.135+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8440b8070 con 0x7fa8441e6910 2023-10-16T15:54:35.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.136+0000 7fa832ffd640 1 -- 172.21.15.29:0/1524110024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+12911 (secure 0 0 0) 0x7fa83c057a70 con 0x7fa8441e6910 2023-10-16T15:54:35.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:35.139 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":242,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:34.108867+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:35.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa82406b0b0 msgr2=0x7fa82406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa82406b0b0 0x7fa82406d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa834003b60 tx=0x7fa83400b040 comp rx=0 tx=0).stop 2023-10-16T15:54:35.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 msgr2=0x7fa8441e6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 0x7fa8441e6d90 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fa8440b5b70 tx=0x7fa83c00e9d0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 shutdown_connections 2023-10-16T15:54:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa82406b0b0 0x7fa82406d570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8441e6910 0x7fa8441e6d90 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa8440c7510 0x7fa8440c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 --2- 172.21.15.29:0/1524110024 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa82c0047c0 0x7fa8440cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 >> 172.21.15.29:0/1524110024 conn(0x7fa8440cf570 msgr2=0x7fa84415ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:35.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 shutdown_connections 2023-10-16T15:54:35.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.139+0000 7fa84359e640 1 -- 172.21.15.29:0/1524110024 wait complete. 2023-10-16T15:54:35.162 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:54:35.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:54:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.379+0000 7fc8cffff640 1 Processor -- start 2023-10-16T15:54:35.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.379+0000 7fc8cffff640 1 -- start start 2023-10-16T15:54:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.380+0000 7fc8cffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc8d00b3ec0 0x7fc8d00cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.380+0000 7fc8cffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 0x7fc8d00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.381+0000 7fc8cffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d006be30 con 0x7fc8d00b0ea0 2023-10-16T15:54:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.381+0000 7fc8cffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d006ca00 con 0x7fc8d00b3ec0 2023-10-16T15:54:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.381+0000 7fc8cffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00ada10 con 0x7fc8d00b43c0 2023-10-16T15:54:35.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.381+0000 7fc8cf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 0x7fc8d00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.382+0000 7fc8cf7fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 0x7fc8d00cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:60804/0 (socket says 172.21.15.29:60804) 2023-10-16T15:54:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.382+0000 7fc8ce7fc640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fc8d00b0ea0 0x7fc8d015a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:37920/0 (socket says 172.21.15.29:37920) 2023-10-16T15:54:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.382+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/4009462935 learned_addr learned my addr 172.21.15.29:0/4009462935 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.382+0000 7fc8ceffd640 1 --2- 172.21.15.29:0/4009462935 >> v2:172.21.15.170:3300/0 conn(0x7fc8d00b3ec0 0x7fc8d00cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.383+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/4009462935 >> v2:172.21.15.170:3300/0 conn(0x7fc8d00b3ec0 msgr2=0x7fc8d00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.383+0000 7fc8cf7fe640 1 --2- 172.21.15.29:0/4009462935 >> v2:172.21.15.170:3300/0 conn(0x7fc8d00b3ec0 0x7fc8d00cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.383+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/4009462935 >> v1:172.21.15.170:6789/0 conn(0x7fc8d00b0ea0 legacy=0x7fc8d015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.383+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/4009462935 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d006c060 con 0x7fc8d00b43c0 2023-10-16T15:54:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.383+0000 7fc8ceffd640 1 --2- 172.21.15.29:0/4009462935 >> v2:172.21.15.170:3300/0 conn(0x7fc8d00b3ec0 0x7fc8d00cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.384+0000 7fc8cf7fe640 1 --2- 172.21.15.29:0/4009462935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 0x7fc8d00cd7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009e50 tx=0x7fc8c802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80cca75874ccb718 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.384+0000 7fc8cdffb640 1 -- 172.21.15.29:0/4009462935 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c802ca80 con 0x7fc8d00b43c0 2023-10-16T15:54:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.384+0000 7fc8cdffb640 1 -- 172.21.15.29:0/4009462935 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c802cbe0 con 0x7fc8d00b43c0 2023-10-16T15:54:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.384+0000 7fc8cdffb640 1 -- 172.21.15.29:0/4009462935 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80343f0 con 0x7fc8d00b43c0 2023-10-16T15:54:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 -- 172.21.15.29:0/4009462935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 msgr2=0x7fc8d00cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 --2- 172.21.15.29:0/4009462935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 0x7fc8d00cd7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009e50 tx=0x7fc8c802bee0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 -- 172.21.15.29:0/4009462935 shutdown_connections 2023-10-16T15:54:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 --2- 172.21.15.29:0/4009462935 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b43c0 0x7fc8d00cd7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 --2- 172.21.15.29:0/4009462935 >> v2:172.21.15.170:3300/0 conn(0x7fc8d00b3ec0 0x7fc8d00cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 -- 172.21.15.29:0/4009462935 >> 172.21.15.29:0/4009462935 conn(0x7fc8d00b0170 msgr2=0x7fc8d00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 -- 172.21.15.29:0/4009462935 shutdown_connections 2023-10-16T15:54:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.385+0000 7fc8cffff640 1 -- 172.21.15.29:0/4009462935 wait complete. 2023-10-16T15:54:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.387+0000 7fc8cffff640 1 Processor -- start 2023-10-16T15:54:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.387+0000 7fc8cffff640 1 -- start start 2023-10-16T15:54:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b0ea0 0x7fc8d00c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d00b3ec0 0x7fc8d0070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 0x7fc8d0155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00c4b10 con 0x7fc8d00b43c0 2023-10-16T15:54:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00c26f0 con 0x7fc8d00b3ec0 2023-10-16T15:54:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00c3e90 con 0x7fc8d00b0ea0 2023-10-16T15:54:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.388+0000 7fc8cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 0x7fc8d0155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 0x7fc8d0155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56078/0 (socket says 172.21.15.29:56078) 2023-10-16T15:54:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/1411040965 learned_addr learned my addr 172.21.15.29:0/1411040965 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8ceffd640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b0ea0 0x7fc8d00c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8ce7fc640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d00b3ec0 0x7fc8d0070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b0ea0 msgr2=0x7fc8d00c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b0ea0 0x7fc8d00c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d00b3ec0 msgr2=0x7fc8d0070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d00b3ec0 0x7fc8d0070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 -- 172.21.15.29:0/1411040965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0009400 con 0x7fc8d00b43c0 2023-10-16T15:54:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.389+0000 7fc8cf7fe640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 0x7fc8d0155760 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009e50 tx=0x7fc8c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.390+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8009510 con 0x7fc8d00b43c0 2023-10-16T15:54:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.390+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c803e450 con 0x7fc8d00b43c0 2023-10-16T15:54:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.390+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c8009b30 con 0x7fc8d00b43c0 2023-10-16T15:54:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.390+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c803e800 con 0x7fc8d00b43c0 2023-10-16T15:54:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.390+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d00c4450 con 0x7fc8d00b43c0 2023-10-16T15:54:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.391+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d00c4c50 con 0x7fc8d00b43c0 2023-10-16T15:54:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.392+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc8c802cd50 con 0x7fc8d00b43c0 2023-10-16T15:54:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.393+0000 7fc8b77fe640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc8bc06b0b0 0x7fc8bc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.393+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fc8c80ac120 con 0x7fc8d00b43c0 2023-10-16T15:54:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.394+0000 7fc8ceffd640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc8bc06b0b0 0x7fc8bc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.395+0000 7fc8ceffd640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc8bc06b0b0 0x7fc8bc06d570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00c2120 tx=0x7fc8c0008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:35.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.397+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8d00c4c50 con 0x7fc8d00b43c0 2023-10-16T15:54:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.587+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8d00c4c50 con 0x7fc8d00b43c0 2023-10-16T15:54:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.588+0000 7fc8b77fe640 1 -- 172.21.15.29:0/1411040965 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v242) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8d00c4c50 con 0x7fc8d00b43c0 2023-10-16T15:54:35.590 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:54:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc8bc06b0b0 msgr2=0x7fc8bc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc8bc06b0b0 0x7fc8bc06d570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00c2120 tx=0x7fc8c0008040 comp rx=0 tx=0).stop 2023-10-16T15:54:35.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 msgr2=0x7fc8d0155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 0x7fc8d0155760 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009e50 tx=0x7fc8c8004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 shutdown_connections 2023-10-16T15:54:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc8bc06b0b0 0x7fc8bc06d570 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8d00b43c0 0x7fc8d0155760 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc8d00b3ec0 0x7fc8d0070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 --2- 172.21.15.29:0/1411040965 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc8d00b0ea0 0x7fc8d00c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 >> 172.21.15.29:0/1411040965 conn(0x7fc8d00b0170 msgr2=0x7fc8d00b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 shutdown_connections 2023-10-16T15:54:35.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.590+0000 7fc8cffff640 1 -- 172.21.15.29:0/1411040965 wait complete. 2023-10-16T15:54:35.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:54:35.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.824+0000 7f8ed5f63640 1 Processor -- start 2023-10-16T15:54:35.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.824+0000 7f8ed5f63640 1 -- start start 2023-10-16T15:54:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.826+0000 7f8ed5f63640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8ed00ad970 0x7f8ed00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.826+0000 7f8ed5f63640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed015b9d0 con 0x7f8ed006c040 2023-10-16T15:54:35.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.826+0000 7f8ed5f63640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed015bcb0 con 0x7f8ed00ccfb0 2023-10-16T15:54:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.826+0000 7f8ed5f63640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed015bf90 con 0x7f8ed00ad970 2023-10-16T15:54:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.826+0000 7f8ed4f61640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8ed006c040 0x7f8ed006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53190/0 (socket says 172.21.15.29:53190) 2023-10-16T15:54:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.826+0000 7f8ed4f61640 1 -- 172.21.15.29:0/1451696653 learned_addr learned my addr 172.21.15.29:0/1451696653 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:35.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576184061 0 0) 0x7f8ed015b9d0 con 0x7f8ed006c040 2023-10-16T15:54:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1451696653 >> v2:172.21.15.170:3300/0 conn(0x7f8ed00ad970 0x7f8ed00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec0003520 con 0x7f8ed006c040 2023-10-16T15:54:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079377956 0 0) 0x7f8ec0003520 con 0x7f8ed006c040 2023-10-16T15:54:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 >> v2:172.21.15.170:3300/0 conn(0x7f8ed00ad970 msgr2=0x7f8ed00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/1451696653 >> v2:172.21.15.170:3300/0 conn(0x7f8ed00ad970 0x7f8ed00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 >> v1:172.21.15.170:6789/0 conn(0x7f8ed00ccfb0 legacy=0x7f8ed00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00cc270 con 0x7f8ed006c040 2023-10-16T15:54:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8ec4001670 con 0x7f8ed006c040 2023-10-16T15:54:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 >> v1:172.21.15.29:6789/0 conn(0x7f8ed006c040 legacy=0x7f8ed006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ec00039c0 0x7f8ec0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ec00047c0 0x7f8ec0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec0005180 0x7f8ec000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed015bf90 con 0x7f8ec00047c0 2023-10-16T15:54:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec000c080 con 0x7f8ec0005180 2023-10-16T15:54:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.827+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec000c240 con 0x7f8ec00039c0 2023-10-16T15:54:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ec00047c0 0x7f8ec0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ed4f61640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ec00039c0 0x7f8ec0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ecffff640 1 -- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ec00039c0 msgr2=0x7f8ec0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ec00039c0 0x7f8ec0003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ecffff640 1 -- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec0005180 msgr2=0x7f8ec000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:35.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ed5762640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec0005180 0x7f8ec000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec0005180 0x7f8ec000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ecffff640 1 -- 172.21.15.29:0/1451696653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec0003520 con 0x7f8ec00047c0 2023-10-16T15:54:35.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ed5762640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec0005180 0x7f8ec000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:35.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.828+0000 7f8ed5762640 1 -- 172.21.15.29:0/1451696653 reap_dead start 2023-10-16T15:54:35.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.829+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ec00047c0 0x7f8ec0004c40 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009a90 tx=0x7f8ec00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e7fa15420a6075 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:35.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.829+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc02c680 con 0x7f8ec00047c0 2023-10-16T15:54:35.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc02c7e0 con 0x7f8ec00047c0 2023-10-16T15:54:35.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/1451696653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc02cb90 con 0x7f8ec00047c0 2023-10-16T15:54:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1451696653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ec00047c0 msgr2=0x7f8ec0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ec00047c0 0x7f8ec0004c40 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009a90 tx=0x7f8ec00057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1451696653 shutdown_connections 2023-10-16T15:54:35.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1451696653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ec00047c0 0x7f8ec0004c40 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1451696653 >> 172.21.15.29:0/1451696653 conn(0x7f8ed00ce6a0 msgr2=0x7f8ed00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:35.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1451696653 shutdown_connections 2023-10-16T15:54:35.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.830+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1451696653 wait complete. 2023-10-16T15:54:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.832+0000 7f8ed5f63640 1 Processor -- start 2023-10-16T15:54:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.832+0000 7f8ed5f63640 1 -- start start 2023-10-16T15:54:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.833+0000 7f8ed5f63640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec00047c0 0x7f8ed00b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.833+0000 7f8ed5f63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ed00af220 0x7f8ed0154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.833+0000 7f8ed5f63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 0x7f8ed00b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.833+0000 7f8ed5f63640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00b0c20 con 0x7f8ed00b0260 2023-10-16T15:54:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.833+0000 7f8ed5f63640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00af6a0 con 0x7f8ec00047c0 2023-10-16T15:54:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.833+0000 7f8ed5f63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00b3420 con 0x7f8ed00af220 2023-10-16T15:54:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 0x7f8ed00b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 0x7f8ed00b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56094/0 (socket says 172.21.15.29:56094) 2023-10-16T15:54:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 -- 172.21.15.29:0/1521074928 learned_addr learned my addr 172.21.15.29:0/1521074928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed4f61640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec00047c0 0x7f8ed00b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ed00af220 0x7f8ed0154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 -- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ed00af220 msgr2=0x7f8ed0154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ed00af220 0x7f8ed0154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 -- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec00047c0 msgr2=0x7f8ed00b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec00047c0 0x7f8ed00b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 -- 172.21.15.29:0/1521074928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00b3700 con 0x7f8ed00b0260 2023-10-16T15:54:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ecffff640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ed00af220 0x7f8ed0154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed4f61640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec00047c0 0x7f8ed00b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:35.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.834+0000 7f8ed5762640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 0x7f8ed00b06e0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00cd390 tx=0x7f8ec800d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:35.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.835+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800b070 con 0x7f8ed00b0260 2023-10-16T15:54:35.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.835+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec8003e60 con 0x7f8ed00b0260 2023-10-16T15:54:35.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.835+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8006d90 con 0x7f8ed00b0260 2023-10-16T15:54:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.835+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed00b39e0 con 0x7f8ed00b0260 2023-10-16T15:54:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.835+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed01551e0 con 0x7f8ed00b0260 2023-10-16T15:54:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.836+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed006b330 con 0x7f8ed00b0260 2023-10-16T15:54:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.837+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8ec8005070 con 0x7f8ed00b0260 2023-10-16T15:54:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.838+0000 7f8ecd7fa640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8eb406b0b0 0x7f8eb406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:35.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.838+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f8ec8088920 con 0x7f8ed00b0260 2023-10-16T15:54:35.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.839+0000 7f8ed4f61640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8eb406b0b0 0x7f8eb406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:35.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.840+0000 7f8ed4f61640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8eb406b0b0 0x7f8eb406d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40017a0 tx=0x7f8ec4009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:35.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:35.842+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ec8053890 con 0x7f8ed00b0260 2023-10-16T15:54:36.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.040+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8ed004e9b0 con 0x7f8ed00b0260 2023-10-16T15:54:36.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.041+0000 7f8ecd7fa640 1 -- 172.21.15.29:0/1521074928 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v242) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8ec8018070 con 0x7f8ed00b0260 2023-10-16T15:54:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:36.043 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:54:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:54:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:54:36.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:36.044 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:54:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8eb406b0b0 msgr2=0x7f8eb406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8eb406b0b0 0x7f8eb406d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40017a0 tx=0x7f8ec4009040 comp rx=0 tx=0).stop 2023-10-16T15:54:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 msgr2=0x7f8ed00b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 0x7f8ed00b06e0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00cd390 tx=0x7f8ec800d900 comp rx=0 tx=0).stop 2023-10-16T15:54:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 shutdown_connections 2023-10-16T15:54:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f8eb406b0b0 0x7f8eb406d570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00b0260 0x7f8ed00b06e0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8ed00af220 0x7f8ed0154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 --2- 172.21.15.29:0/1521074928 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8ec00047c0 0x7f8ed00b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.043+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 >> 172.21.15.29:0/1521074928 conn(0x7f8ed00ce6a0 msgr2=0x7f8ed00c2010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.044+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 shutdown_connections 2023-10-16T15:54:36.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.044+0000 7f8ed5f63640 1 -- 172.21.15.29:0/1521074928 wait complete. 2023-10-16T15:54:36.065 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T15:54:36.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T15:54:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.296+0000 7fdb92267640 1 Processor -- start 2023-10-16T15:54:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.296+0000 7fdb92267640 1 -- start start 2023-10-16T15:54:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.297+0000 7fdb92267640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c06d280 0x7fdb8c0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.297+0000 7fdb92267640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdb8c0cae40 0x7fdb8c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb92267640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c0c8e70 con 0x7fdb8c15ad80 2023-10-16T15:54:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb92267640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c0b4170 con 0x7fdb8c0cae40 2023-10-16T15:54:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb92267640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0adde0 con 0x7fdb8c06d280 2023-10-16T15:54:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb91265640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdb8c15ad80 0x7fdb8c06cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53202/0 (socket says 172.21.15.29:53202) 2023-10-16T15:54:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb91265640 1 -- 172.21.15.29:0/920455628 learned_addr learned my addr 172.21.15.29:0/920455628 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb91a66640 1 --2- 172.21.15.29:0/920455628 >> v2:172.21.15.29:3300/0 conn(0x7fdb8c0cae40 0x7fdb8c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb90a64640 1 --2- 172.21.15.29:0/920455628 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c06d280 0x7fdb8c0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb83fff640 1 -- 172.21.15.29:0/920455628 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265424851 0 0) 0x7fdb8c0c8e70 con 0x7fdb8c15ad80 2023-10-16T15:54:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.298+0000 7fdb83fff640 1 -- 172.21.15.29:0/920455628 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78003520 con 0x7fdb8c15ad80 2023-10-16T15:54:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.299+0000 7fdb91a66640 1 -- 172.21.15.29:0/920455628 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c06d280 msgr2=0x7fdb8c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.299+0000 7fdb91a66640 1 --2- 172.21.15.29:0/920455628 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c06d280 0x7fdb8c0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.299+0000 7fdb91a66640 1 -- 172.21.15.29:0/920455628 >> v1:172.21.15.29:6789/0 conn(0x7fdb8c15ad80 legacy=0x7fdb8c06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.299+0000 7fdb91a66640 1 -- 172.21.15.29:0/920455628 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c0b4410 con 0x7fdb8c0cae40 2023-10-16T15:54:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.300+0000 7fdb91a66640 1 --2- 172.21.15.29:0/920455628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c9840 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009d90 tx=0x7fdb8402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fda823005ee469 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.300+0000 7fdb83fff640 1 -- 172.21.15.29:0/920455628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402cc50 con 0x7fdb8c0cae40 2023-10-16T15:54:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.300+0000 7fdb83fff640 1 -- 172.21.15.29:0/920455628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8402cdb0 con 0x7fdb8c0cae40 2023-10-16T15:54:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.300+0000 7fdb83fff640 1 -- 172.21.15.29:0/920455628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84034660 con 0x7fdb8c0cae40 2023-10-16T15:54:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 -- 172.21.15.29:0/920455628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c0cae40 msgr2=0x7fdb8c0c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 --2- 172.21.15.29:0/920455628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c9840 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009d90 tx=0x7fdb8402bea0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 -- 172.21.15.29:0/920455628 shutdown_connections 2023-10-16T15:54:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 --2- 172.21.15.29:0/920455628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c9840 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 --2- 172.21.15.29:0/920455628 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c06d280 0x7fdb8c0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 -- 172.21.15.29:0/920455628 >> 172.21.15.29:0/920455628 conn(0x7fdb8c0ce820 msgr2=0x7fdb8c0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 -- 172.21.15.29:0/920455628 shutdown_connections 2023-10-16T15:54:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.301+0000 7fdb92267640 1 -- 172.21.15.29:0/920455628 wait complete. 2023-10-16T15:54:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.303+0000 7fdb92267640 1 Processor -- start 2023-10-16T15:54:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.303+0000 7fdb92267640 1 -- start start 2023-10-16T15:54:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb92267640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 0x7fdb8c14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb92267640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb92267640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c15ad80 0x7fdb8c0c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb92267640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0c7050 con 0x7fdb8c06d280 2023-10-16T15:54:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb92267640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c14ca80 con 0x7fdb8c0cae40 2023-10-16T15:54:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb92267640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0c66b0 con 0x7fdb8c15ad80 2023-10-16T15:54:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb91265640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 0x7fdb8c14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb91265640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 0x7fdb8c14c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56100/0 (socket says 172.21.15.29:56100) 2023-10-16T15:54:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.304+0000 7fdb91265640 1 -- 172.21.15.29:0/3924611662 learned_addr learned my addr 172.21.15.29:0/3924611662 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:36.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91a66640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c15ad80 0x7fdb8c0c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:36.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb90a64640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:36.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91265640 1 -- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c15ad80 msgr2=0x7fdb8c0c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:36.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91265640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c15ad80 0x7fdb8c0c79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91265640 1 -- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb8c0cae40 msgr2=0x7fdb8c0c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:36.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91265640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:36.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91265640 1 -- 172.21.15.29:0/3924611662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb84009a70 con 0x7fdb8c06d280 2023-10-16T15:54:36.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91a66640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c15ad80 0x7fdb8c0c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:36.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb90a64640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:36.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb91265640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 0x7fdb8c14c540 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800c440 tx=0x7fdb8800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88005ce0 con 0x7fdb8c06d280 2023-10-16T15:54:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb88005e40 con 0x7fdb8c06d280 2023-10-16T15:54:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.305+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb880072b0 con 0x7fdb8c06d280 2023-10-16T15:54:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.306+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c0c6990 con 0x7fdb8c06d280 2023-10-16T15:54:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.306+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c149a00 con 0x7fdb8c06d280 2023-10-16T15:54:36.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.307+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8c0c72c0 con 0x7fdb8c06d280 2023-10-16T15:54:36.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.308+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdb88007410 con 0x7fdb8c06d280 2023-10-16T15:54:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.309+0000 7fdb81ffb640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fdb7006b0b0 0x7fdb7006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.309+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fdb88088c50 con 0x7fdb8c06d280 2023-10-16T15:54:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.310+0000 7fdb90a64640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fdb7006b0b0 0x7fdb7006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:36.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.311+0000 7fdb90a64640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fdb7006b0b0 0x7fdb7006d570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0c5190 tx=0x7fdb7c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:36.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.313+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdb8808e020 con 0x7fdb8c06d280 2023-10-16T15:54:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:36.507+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdb8c149c50 con 0x7fdb8c06d280 2023-10-16T15:54:37.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.142+0000 7fdb81ffb640 1 -- 172.21.15.29:0/3924611662 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v243) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb88017020 con 0x7fdb8c06d280 2023-10-16T15:54:37.145 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T15:54:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fdb7006b0b0 msgr2=0x7fdb7006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fdb7006b0b0 0x7fdb7006d570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0c5190 tx=0x7fdb7c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 msgr2=0x7fdb8c14c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 0x7fdb8c14c540 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800c440 tx=0x7fdb8800c900 comp rx=0 tx=0).stop 2023-10-16T15:54:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 shutdown_connections 2023-10-16T15:54:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fdb7006b0b0 0x7fdb7006d570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdb8c15ad80 0x7fdb8c0c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb8c0cae40 0x7fdb8c0c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 --2- 172.21.15.29:0/3924611662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb8c06d280 0x7fdb8c14c540 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 >> 172.21.15.29:0/3924611662 conn(0x7fdb8c0ce820 msgr2=0x7fdb8c0ae240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 shutdown_connections 2023-10-16T15:54:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.147+0000 7fdb92267640 1 -- 172.21.15.29:0/3924611662 wait complete. 2023-10-16T15:54:37.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:54:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.412+0000 7ff08df93640 1 Processor -- start 2023-10-16T15:54:37.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.413+0000 7ff08df93640 1 -- start start 2023-10-16T15:54:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.414+0000 7ff08df93640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff08815cba0 0x7ff088071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:37.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.414+0000 7ff08df93640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0880720d0 0x7ff0880cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.414+0000 7ff08df93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880d0500 con 0x7ff0880cdd40 2023-10-16T15:54:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.414+0000 7ff08df93640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880ce1b0 con 0x7ff0880720d0 2023-10-16T15:54:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.414+0000 7ff08df93640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880d07c0 con 0x7ff08815cba0 2023-10-16T15:54:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff087fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0880720d0 0x7ff0880cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff087fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0880720d0 0x7ff0880cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56102/0 (socket says 172.21.15.29:56102) 2023-10-16T15:54:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff087fff640 1 -- 172.21.15.29:0/2260587190 learned_addr learned my addr 172.21.15.29:0/2260587190 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff08d792640 1 --1- 172.21.15.29:0/2260587190 >> v1:172.21.15.29:6789/0 conn(0x7ff0880cdd40 0x7ff0881607e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53218/0 (socket says 172.21.15.29:53218) 2023-10-16T15:54:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff08cf91640 1 --2- 172.21.15.29:0/2260587190 >> v2:172.21.15.170:3300/0 conn(0x7ff08815cba0 0x7ff088071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff087fff640 1 -- 172.21.15.29:0/2260587190 >> v2:172.21.15.170:3300/0 conn(0x7ff08815cba0 msgr2=0x7ff088071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff087fff640 1 --2- 172.21.15.29:0/2260587190 >> v2:172.21.15.170:3300/0 conn(0x7ff08815cba0 0x7ff088071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff087fff640 1 -- 172.21.15.29:0/2260587190 >> v1:172.21.15.29:6789/0 conn(0x7ff0880cdd40 legacy=0x7ff0881607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.415+0000 7ff0877fe640 1 -- 172.21.15.29:0/2260587190 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237230673 0 0) 0x7ff0880d0500 con 0x7ff0880cdd40 2023-10-16T15:54:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.416+0000 7ff087fff640 1 -- 172.21.15.29:0/2260587190 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880d0aa0 con 0x7ff0880720d0 2023-10-16T15:54:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.417+0000 7ff087fff640 1 --2- 172.21.15.29:0/2260587190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880720d0 0x7ff0880cd800 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7ff08806e5b0 tx=0x7ff07c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6e861c7a972072 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.417+0000 7ff0877fe640 1 -- 172.21.15.29:0/2260587190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c030860 con 0x7ff0880720d0 2023-10-16T15:54:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.417+0000 7ff0877fe640 1 -- 172.21.15.29:0/2260587190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c0309c0 con 0x7ff0880720d0 2023-10-16T15:54:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.417+0000 7ff0877fe640 1 -- 172.21.15.29:0/2260587190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c030d50 con 0x7ff0880720d0 2023-10-16T15:54:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.417+0000 7ff08df93640 1 -- 172.21.15.29:0/2260587190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880720d0 msgr2=0x7ff0880cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.417+0000 7ff08df93640 1 --2- 172.21.15.29:0/2260587190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880720d0 0x7ff0880cd800 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7ff08806e5b0 tx=0x7ff07c02b9a0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.418+0000 7ff08df93640 1 -- 172.21.15.29:0/2260587190 shutdown_connections 2023-10-16T15:54:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.418+0000 7ff08df93640 1 --2- 172.21.15.29:0/2260587190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0880720d0 0x7ff0880cd800 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.418+0000 7ff08df93640 1 --2- 172.21.15.29:0/2260587190 >> v2:172.21.15.170:3300/0 conn(0x7ff08815cba0 0x7ff088071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.418+0000 7ff08df93640 1 -- 172.21.15.29:0/2260587190 >> 172.21.15.29:0/2260587190 conn(0x7ff088065ed0 msgr2=0x7ff0880662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.418+0000 7ff08df93640 1 -- 172.21.15.29:0/2260587190 shutdown_connections 2023-10-16T15:54:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.418+0000 7ff08df93640 1 -- 172.21.15.29:0/2260587190 wait complete. 2023-10-16T15:54:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.419+0000 7ff08df93640 1 Processor -- start 2023-10-16T15:54:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.420+0000 7ff08df93640 1 -- start start 2023-10-16T15:54:37.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08df93640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0880720d0 0x7ff0881f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:37.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08df93640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0880cdd40 0x7ff0881f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:37.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08df93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 0x7ff0880c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:37.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08df93640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881f3e00 con 0x7ff08815cba0 2023-10-16T15:54:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08df93640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881f40e0 con 0x7ff0880cdd40 2023-10-16T15:54:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08df93640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881f43c0 con 0x7ff0880720d0 2023-10-16T15:54:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08d792640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 0x7ff0880c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08d792640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 0x7ff0880c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56116/0 (socket says 172.21.15.29:56116) 2023-10-16T15:54:37.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08d792640 1 -- 172.21.15.29:0/3904052513 learned_addr learned my addr 172.21.15.29:0/3904052513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08cf91640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0880720d0 0x7ff0881f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff087fff640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0880cdd40 0x7ff0881f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.421+0000 7ff08d792640 1 -- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0880720d0 msgr2=0x7ff0881f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08d792640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0880720d0 0x7ff0881f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08d792640 1 -- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0880cdd40 msgr2=0x7ff0881f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:37.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08d792640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0880cdd40 0x7ff0881f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08d792640 1 -- 172.21.15.29:0/3904052513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c0094d0 con 0x7ff08815cba0 2023-10-16T15:54:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff087fff640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0880cdd40 0x7ff0881f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08d792640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 0x7ff0880c5fa0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7ff08000c450 tx=0x7ff08000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800074e0 con 0x7ff08815cba0 2023-10-16T15:54:37.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080007660 con 0x7ff08815cba0 2023-10-16T15:54:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0881f46a0 con 0x7ff08815cba0 2023-10-16T15:54:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080004a20 con 0x7ff08815cba0 2023-10-16T15:54:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.422+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0881601c0 con 0x7ff08815cba0 2023-10-16T15:54:37.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.423+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08804e9b0 con 0x7ff08815cba0 2023-10-16T15:54:37.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.425+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff080004b80 con 0x7ff08815cba0 2023-10-16T15:54:37.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.426+0000 7ff0857fa640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff07406b0b0 0x7ff07406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:37.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.426+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5255+0+0 (secure 0 0 0) 0x7ff080088d80 con 0x7ff08815cba0 2023-10-16T15:54:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.427+0000 7ff08cf91640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff07406b0b0 0x7ff07406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.428+0000 7ff08cf91640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff07406b0b0 0x7ff07406d570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff088160c20 tx=0x7ff0780039b0 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:37.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.430+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff080053d50 con 0x7ff08815cba0 2023-10-16T15:54:37.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:37.624+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff088159470 con 0x7ff08815cba0 2023-10-16T15:54:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.162+0000 7ff0857fa640 1 -- 172.21.15.29:0/3904052513 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v244) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff080018020 con 0x7ff08815cba0 2023-10-16T15:54:38.164 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T15:54:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.165+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff07406b0b0 msgr2=0x7ff07406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.165+0000 7ff08df93640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff07406b0b0 0x7ff07406d570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff088160c20 tx=0x7ff0780039b0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.165+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 msgr2=0x7ff0880c5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.165+0000 7ff08df93640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 0x7ff0880c5fa0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7ff08000c450 tx=0x7ff08000c910 comp rx=0 tx=0).stop 2023-10-16T15:54:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 shutdown_connections 2023-10-16T15:54:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff07406b0b0 0x7ff07406d570 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff08815cba0 0x7ff0880c5fa0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff0880cdd40 0x7ff0881f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 --2- 172.21.15.29:0/3904052513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff0880720d0 0x7ff0881f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 >> 172.21.15.29:0/3904052513 conn(0x7ff088065ed0 msgr2=0x7ff0880b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 shutdown_connections 2023-10-16T15:54:38.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.166+0000 7ff08df93640 1 -- 172.21.15.29:0/3904052513 wait complete. 2023-10-16T15:54:38.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T15:54:38.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.392+0000 7f2ba10b0640 1 Processor -- start 2023-10-16T15:54:38.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.392+0000 7f2ba10b0640 1 -- start start 2023-10-16T15:54:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.394+0000 7f2ba10b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b9c06a4b0 0x7f2b9c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.394+0000 7f2ba10b0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f2b9c0c9ca0 0x7f2b9c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:38.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.394+0000 7f2ba10b0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c0cb030 con 0x7f2b9c15ad70 2023-10-16T15:54:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.394+0000 7f2ba10b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9c0cae30 con 0x7f2b9c06a4b0 2023-10-16T15:54:38.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.394+0000 7f2ba10b0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9c0c91c0 con 0x7f2b9c0c9ca0 2023-10-16T15:54:38.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b9c06a4b0 0x7f2b9c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:38.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b9c06a4b0 0x7f2b9c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56126/0 (socket says 172.21.15.29:56126) 2023-10-16T15:54:38.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 -- 172.21.15.29:0/2344853650 learned_addr learned my addr 172.21.15.29:0/2344853650 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:38.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9b59e640 1 --2- 172.21.15.29:0/2344853650 >> v2:172.21.15.170:3300/0 conn(0x7f2b9c0c9ca0 0x7f2b9c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:38.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 -- 172.21.15.29:0/2344853650 >> v2:172.21.15.170:3300/0 conn(0x7f2b9c0c9ca0 msgr2=0x7f2b9c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:38.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 --2- 172.21.15.29:0/2344853650 >> v2:172.21.15.170:3300/0 conn(0x7f2b9c0c9ca0 0x7f2b9c0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 -- 172.21.15.29:0/2344853650 >> v1:172.21.15.170:6789/0 conn(0x7f2b9c15ad70 legacy=0x7f2b9c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:38.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.395+0000 7f2b9a59c640 1 -- 172.21.15.29:0/2344853650 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c0c94a0 con 0x7f2b9c06a4b0 2023-10-16T15:54:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.396+0000 7f2b9a59c640 1 --2- 172.21.15.29:0/2344853650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c06a4b0 0x7f2b9c0ca3b0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c06f270 tx=0x7f2b9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90733ac538642ba8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2b99d9b640 1 -- 172.21.15.29:0/2344853650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90031870 con 0x7f2b9c06a4b0 2023-10-16T15:54:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2b99d9b640 1 -- 172.21.15.29:0/2344853650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b900319d0 con 0x7f2b9c06a4b0 2023-10-16T15:54:38.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2b99d9b640 1 -- 172.21.15.29:0/2344853650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90031d60 con 0x7f2b9c06a4b0 2023-10-16T15:54:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 -- 172.21.15.29:0/2344853650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c06a4b0 msgr2=0x7f2b9c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/2344853650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c06a4b0 0x7f2b9c0ca3b0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c06f270 tx=0x7f2b9002b910 comp rx=0 tx=0).stop 2023-10-16T15:54:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 -- 172.21.15.29:0/2344853650 shutdown_connections 2023-10-16T15:54:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/2344853650 >> v2:172.21.15.170:3300/0 conn(0x7f2b9c0c9ca0 0x7f2b9c0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/2344853650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c06a4b0 0x7f2b9c0ca3b0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 -- 172.21.15.29:0/2344853650 >> 172.21.15.29:0/2344853650 conn(0x7f2b9c0ce9b0 msgr2=0x7f2b9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 -- 172.21.15.29:0/2344853650 shutdown_connections 2023-10-16T15:54:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.397+0000 7f2ba10b0640 1 -- 172.21.15.29:0/2344853650 wait complete. 2023-10-16T15:54:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.399+0000 7f2ba10b0640 1 Processor -- start 2023-10-16T15:54:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.399+0000 7f2ba10b0640 1 -- start start 2023-10-16T15:54:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.400+0000 7f2ba10b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9c06a4b0 0x7f2b9c0c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.400+0000 7f2ba10b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b9c0c9ca0 0x7f2b9c151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.400+0000 7f2ba10b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 0x7f2b9c0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:38.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.400+0000 7f2ba10b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0ac890 con 0x7f2b9c15ad70 2023-10-16T15:54:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.400+0000 7f2ba10b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c152000 con 0x7f2b9c0c9ca0 2023-10-16T15:54:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.400+0000 7f2ba10b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0acae0 con 0x7f2b9c06a4b0 2023-10-16T15:54:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 0x7f2b9c0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:38.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 0x7f2b9c0cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56138/0 (socket says 172.21.15.29:56138) 2023-10-16T15:54:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 -- 172.21.15.29:0/803953126 learned_addr learned my addr 172.21.15.29:0/803953126 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9a59c640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b9c0c9ca0 0x7f2b9c151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:38.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9ad9d640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9c06a4b0 0x7f2b9c0c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 -- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9c06a4b0 msgr2=0x7f2b9c0c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9c06a4b0 0x7f2b9c0c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 -- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b9c0c9ca0 msgr2=0x7f2b9c151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b9c0c9ca0 0x7f2b9c151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:38.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 -- 172.21.15.29:0/803953126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b900094d0 con 0x7f2b9c15ad70 2023-10-16T15:54:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9ad9d640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9c06a4b0 0x7f2b9c0c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.401+0000 7f2b9b59e640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 0x7f2b9c0cd470 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f2b94007e40 tx=0x7f2b9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.402+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94014620 con 0x7f2b9c15ad70 2023-10-16T15:54:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.402+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9400ee90 con 0x7f2b9c15ad70 2023-10-16T15:54:38.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.402+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c0acdc0 con 0x7f2b9c15ad70 2023-10-16T15:54:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.402+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b940149e0 con 0x7f2b9c15ad70 2023-10-16T15:54:38.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.402+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c0ad380 con 0x7f2b9c15ad70 2023-10-16T15:54:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.403+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c04e160 con 0x7f2b9c15ad70 2023-10-16T15:54:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.405+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2b94014b40 con 0x7f2b9c15ad70 2023-10-16T15:54:38.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.406+0000 7f2b837fe640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2b8806b0b0 0x7f2b8806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.406+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f2b9408a2c0 con 0x7f2b9c15ad70 2023-10-16T15:54:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.407+0000 7f2b9ad9d640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2b8806b0b0 0x7f2b8806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.408+0000 7f2b9ad9d640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2b8806b0b0 0x7f2b8806d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00a830 tx=0x7f2b8c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:38.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.411+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b940551b0 con 0x7f2b9c15ad70 2023-10-16T15:54:38.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:38.638+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2b9c06c040 con 0x7f2b9c15ad70 2023-10-16T15:54:39.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.257+0000 7f2b837fe640 1 -- 172.21.15.29:0/803953126 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v120) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2b940587d0 con 0x7f2b9c15ad70 2023-10-16T15:54:39.259 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 30 and data pool 31 2023-10-16T15:54:39.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.261+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2b8806b0b0 msgr2=0x7f2b8806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.261+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2b8806b0b0 0x7f2b8806d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00a830 tx=0x7f2b8c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:39.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.261+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 msgr2=0x7f2b9c0cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.261+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 0x7f2b9c0cd470 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f2b94007e40 tx=0x7f2b9400e4b0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 shutdown_connections 2023-10-16T15:54:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2b8806b0b0 0x7f2b8806d570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b9c15ad70 0x7f2b9c0cd470 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b9c0c9ca0 0x7f2b9c151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 --2- 172.21.15.29:0/803953126 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b9c06a4b0 0x7f2b9c0c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 >> 172.21.15.29:0/803953126 conn(0x7f2b9c0ce9b0 msgr2=0x7f2b9c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 shutdown_connections 2023-10-16T15:54:39.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.262+0000 7f2ba10b0640 1 -- 172.21.15.29:0/803953126 wait complete. 2023-10-16T15:54:39.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:39.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.478+0000 7f66cd1c4640 1 Processor -- start 2023-10-16T15:54:39.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.478+0000 7f66cd1c4640 1 -- start start 2023-10-16T15:54:39.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66cd1c4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66c80b8480 0x7f66c80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66cd1c4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c815d150 con 0x7f66c815cd70 2023-10-16T15:54:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66cd1c4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c80d1180 con 0x7f66c80d0be0 2023-10-16T15:54:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66cd1c4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c80d1460 con 0x7f66c80b8480 2023-10-16T15:54:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66c77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66c80b8480 0x7f66c80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66c77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66c80b8480 0x7f66c80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56146/0 (socket says 172.21.15.29:56146) 2023-10-16T15:54:39.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.480+0000 7f66c77fe640 1 -- 172.21.15.29:0/406327616 learned_addr learned my addr 172.21.15.29:0/406327616 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:39.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.481+0000 7f66c77fe640 1 -- 172.21.15.29:0/406327616 >> v1:172.21.15.170:6789/0 conn(0x7f66c80d0be0 legacy=0x7f66c80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:39.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.481+0000 7f66c77fe640 1 -- 172.21.15.29:0/406327616 >> v1:172.21.15.29:6789/0 conn(0x7f66c815cd70 legacy=0x7f66c80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.481+0000 7f66c77fe640 1 -- 172.21.15.29:0/406327616 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80d1740 con 0x7f66c80b8480 2023-10-16T15:54:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.482+0000 7f66c77fe640 1 --2- 172.21.15.29:0/406327616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80b8480 0x7f66c80cd7c0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009870 tx=0x7f66bc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152c843be68944b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.482+0000 7f66c6ffd640 1 -- 172.21.15.29:0/406327616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc031860 con 0x7f66c80b8480 2023-10-16T15:54:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.482+0000 7f66c6ffd640 1 -- 172.21.15.29:0/406327616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc0319c0 con 0x7f66c80b8480 2023-10-16T15:54:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.482+0000 7f66c6ffd640 1 -- 172.21.15.29:0/406327616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc031d50 con 0x7f66c80b8480 2023-10-16T15:54:39.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 -- 172.21.15.29:0/406327616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80b8480 msgr2=0x7f66c80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/406327616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80b8480 0x7f66c80cd7c0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009870 tx=0x7f66bc02b900 comp rx=0 tx=0).stop 2023-10-16T15:54:39.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 -- 172.21.15.29:0/406327616 shutdown_connections 2023-10-16T15:54:39.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/406327616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80b8480 0x7f66c80cd7c0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 -- 172.21.15.29:0/406327616 >> 172.21.15.29:0/406327616 conn(0x7f66c80cf570 msgr2=0x7f66c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:39.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 -- 172.21.15.29:0/406327616 shutdown_connections 2023-10-16T15:54:39.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.483+0000 7f66cd1c4640 1 -- 172.21.15.29:0/406327616 wait complete. 2023-10-16T15:54:39.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.485+0000 7f66cd1c4640 1 Processor -- start 2023-10-16T15:54:39.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.485+0000 7f66cd1c4640 1 -- start start 2023-10-16T15:54:39.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66cd1c4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66c80b8480 0x7f66c8148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66cd1c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 0x7f66c8156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66cd1c4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66c815cd70 0x7f66c8156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66cd1c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c8159670 con 0x7f66c80d0be0 2023-10-16T15:54:39.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66cd1c4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c8159950 con 0x7f66c80b8480 2023-10-16T15:54:39.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66cd1c4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80b3020 con 0x7f66c815cd70 2023-10-16T15:54:39.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 0x7f66c8156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 0x7f66c8156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56154/0 (socket says 172.21.15.29:56154) 2023-10-16T15:54:39.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.486+0000 7f66c77fe640 1 -- 172.21.15.29:0/1344207146 learned_addr learned my addr 172.21.15.29:0/1344207146 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:39.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c77fe640 1 -- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66c815cd70 msgr2=0x7f66c8156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c77fe640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66c815cd70 0x7f66c8156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c77fe640 1 -- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66c80b8480 msgr2=0x7f66c8148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c77fe640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66c80b8480 0x7f66c8148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c77fe640 1 -- 172.21.15.29:0/1344207146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc0094d0 con 0x7f66c80d0be0 2023-10-16T15:54:39.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c77fe640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 0x7f66c8156800 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009870 tx=0x7f66bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:39.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc031810 con 0x7f66c80d0be0 2023-10-16T15:54:39.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc0443f0 con 0x7f66c80d0be0 2023-10-16T15:54:39.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.487+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0447a0 con 0x7f66c80d0be0 2023-10-16T15:54:39.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.488+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c80b3300 con 0x7f66c80d0be0 2023-10-16T15:54:39.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.488+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c81492d0 con 0x7f66c80d0be0 2023-10-16T15:54:39.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.489+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c80b2e80 con 0x7f66c80d0be0 2023-10-16T15:54:39.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.490+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f66bc04d450 con 0x7f66c80d0be0 2023-10-16T15:54:39.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.491+0000 7f66c4ff9640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f66b406b0b0 0x7f66b406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.491+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f66bc0ac300 con 0x7f66c80d0be0 2023-10-16T15:54:39.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.492+0000 7f66c7fff640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f66b406b0b0 0x7f66b406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.493+0000 7f66c7fff640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f66b406b0b0 0x7f66b406d570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66b80012e0 tx=0x7f66b8007d30 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:39.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.495+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66c80b2e80 con 0x7f66c80d0be0 2023-10-16T15:54:39.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.687+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66c80b3540 con 0x7f66c80d0be0 2023-10-16T15:54:39.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.689+0000 7f66c4ff9640 1 -- 172.21.15.29:0/1344207146 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) v1 ==== 74+0+16576 (secure 0 0 0) 0x7f66c80b3540 con 0x7f66c80d0be0 2023-10-16T15:54:39.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:39.692 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":245,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:38.640623+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:39.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f66b406b0b0 msgr2=0x7f66b406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f66b406b0b0 0x7f66b406d570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66b80012e0 tx=0x7f66b8007d30 comp rx=0 tx=0).stop 2023-10-16T15:54:39.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 msgr2=0x7f66c8156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 0x7f66c8156800 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009870 tx=0x7f66bc0046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 shutdown_connections 2023-10-16T15:54:39.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f66b406b0b0 0x7f66b406d570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66c80d0be0 0x7f66c8156800 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f66c815cd70 0x7f66c8156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 --2- 172.21.15.29:0/1344207146 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66c80b8480 0x7f66c8148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.692+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 >> 172.21.15.29:0/1344207146 conn(0x7f66c80cf570 msgr2=0x7f66c80b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:39.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.693+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 shutdown_connections 2023-10-16T15:54:39.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.693+0000 7f66cd1c4640 1 -- 172.21.15.29:0/1344207146 wait complete. 2023-10-16T15:54:39.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.926+0000 7ff1e8f39640 1 Processor -- start 2023-10-16T15:54:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.926+0000 7ff1e8f39640 1 -- start start 2023-10-16T15:54:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.927+0000 7ff1e8f39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e40cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.927+0000 7ff1e8f39640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff1e40b3e70 0x7ff1e40cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.927+0000 7ff1e8f39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1e40b3450 0x7ff1e40c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.927+0000 7ff1e8f39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e40b3940 con 0x7ff1e40b3450 2023-10-16T15:54:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.927+0000 7ff1e8f39640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e40b45e0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.927+0000 7ff1e8f39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40b3b90 con 0x7ff1e40b3070 2023-10-16T15:54:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1e40b3450 0x7ff1e40c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1e40b3450 0x7ff1e40c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56156/0 (socket says 172.21.15.29:56156) 2023-10-16T15:54:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/558933198 learned_addr learned my addr 172.21.15.29:0/558933198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e259c640 1 --2- 172.21.15.29:0/558933198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e40cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e1d9b640 1 --2- 172.21.15.29:0/558933198 >> v2:172.21.15.170:3300/0 conn(0x7ff1e40b3e70 0x7ff1e40cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/558933198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 msgr2=0x7ff1e40cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 --2- 172.21.15.29:0/558933198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e40cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/558933198 >> v2:172.21.15.170:3300/0 conn(0x7ff1e40b3e70 msgr2=0x7ff1e40cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 --2- 172.21.15.29:0/558933198 >> v2:172.21.15.170:3300/0 conn(0x7ff1e40b3e70 0x7ff1e40cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/558933198 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40d3840 con 0x7ff1e40b3450 2023-10-16T15:54:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.928+0000 7ff1e1d9b640 1 --2- 172.21.15.29:0/558933198 >> v2:172.21.15.170:3300/0 conn(0x7ff1e40b3e70 0x7ff1e40cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.929+0000 7ff1e2d9d640 1 --2- 172.21.15.29:0/558933198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3450 0x7ff1e40c6870 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7ff1e408bf20 tx=0x7ff1dc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f5e85b3cfe5d30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e159a640 1 -- 172.21.15.29:0/558933198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc031860 con 0x7ff1e40b3450 2023-10-16T15:54:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e159a640 1 -- 172.21.15.29:0/558933198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0319c0 con 0x7ff1e40b3450 2023-10-16T15:54:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e159a640 1 -- 172.21.15.29:0/558933198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc031d50 con 0x7ff1e40b3450 2023-10-16T15:54:39.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 -- 172.21.15.29:0/558933198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3450 msgr2=0x7ff1e40c6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/558933198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3450 0x7ff1e40c6870 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7ff1e408bf20 tx=0x7ff1dc02b900 comp rx=0 tx=0).stop 2023-10-16T15:54:39.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 -- 172.21.15.29:0/558933198 shutdown_connections 2023-10-16T15:54:39.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/558933198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3450 0x7ff1e40c6870 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/558933198 >> v2:172.21.15.170:3300/0 conn(0x7ff1e40b3e70 0x7ff1e40cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/558933198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e40cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 -- 172.21.15.29:0/558933198 >> 172.21.15.29:0/558933198 conn(0x7ff1e40d2d20 msgr2=0x7ff1e40d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:39.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 -- 172.21.15.29:0/558933198 shutdown_connections 2023-10-16T15:54:39.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.930+0000 7ff1e8f39640 1 -- 172.21.15.29:0/558933198 wait complete. 2023-10-16T15:54:39.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.932+0000 7ff1e8f39640 1 Processor -- start 2023-10-16T15:54:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.932+0000 7ff1e8f39640 1 -- start start 2023-10-16T15:54:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e8f39640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e41e6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e8f39640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1e40b3450 0x7ff1e406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e8f39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 0x7ff1e4151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e8f39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40caf70 con 0x7ff1e40b3e70 2023-10-16T15:54:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e8f39640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40ca360 con 0x7ff1e40b3450 2023-10-16T15:54:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e8f39640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40ca640 con 0x7ff1e40b3070 2023-10-16T15:54:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.933+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 0x7ff1e4151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 0x7ff1e4151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56166/0 (socket says 172.21.15.29:56166) 2023-10-16T15:54:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/1393887964 learned_addr learned my addr 172.21.15.29:0/1393887964 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e259c640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e41e6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e1d9b640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1e40b3450 0x7ff1e406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 msgr2=0x7ff1e41e6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e41e6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1e40b3450 msgr2=0x7ff1e406a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1e40b3450 0x7ff1e406a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 -- 172.21.15.29:0/1393887964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc0094d0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e1d9b640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1e40b3450 0x7ff1e406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.934+0000 7ff1e2d9d640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 0x7ff1e4151c80 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc02b940 tx=0x7ff1dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.935+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0317f0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.935+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0443f0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.935+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0447a0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.935+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e40ca920 con 0x7ff1e40b3e70 2023-10-16T15:54:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.935+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e41e71a0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.936+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e404e160 con 0x7ff1e40b3e70 2023-10-16T15:54:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.937+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff1dc044900 con 0x7ff1e40b3e70 2023-10-16T15:54:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.938+0000 7ff1daffd640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff1c406b0e0 0x7ff1c406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.938+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff1dc0ac2c0 con 0x7ff1e40b3e70 2023-10-16T15:54:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.939+0000 7ff1e259c640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff1c406b0e0 0x7ff1c406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.940+0000 7ff1e259c640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff1c406b0e0 0x7ff1c406d5a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4008ec0 tx=0x7ff1d4008410 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:39.942+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1dc076ff0 con 0x7ff1e40b3e70 2023-10-16T15:54:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.134+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1e406b4a0 con 0x7ff1e40b3e70 2023-10-16T15:54:40.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.136+0000 7ff1daffd640 1 -- 172.21.15.29:0/1393887964 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) v1 ==== 74+0+16576 (secure 0 0 0) 0x7ff1dc07a610 con 0x7ff1e40b3e70 2023-10-16T15:54:40.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:40.139 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":245,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:38.640623+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:40.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff1c406b0e0 msgr2=0x7ff1c406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff1c406b0e0 0x7ff1c406d5a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4008ec0 tx=0x7ff1d4008410 comp rx=0 tx=0).stop 2023-10-16T15:54:40.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 msgr2=0x7ff1e4151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 0x7ff1e4151c80 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc02b940 tx=0x7ff1dc0046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 shutdown_connections 2023-10-16T15:54:40.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff1c406b0e0 0x7ff1c406d5a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1e40b3e70 0x7ff1e4151c80 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff1e40b3450 0x7ff1e406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 --2- 172.21.15.29:0/1393887964 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1e40b3070 0x7ff1e41e6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.139+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 >> 172.21.15.29:0/1393887964 conn(0x7ff1e40d2d20 msgr2=0x7ff1e406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:40.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.140+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 shutdown_connections 2023-10-16T15:54:40.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.140+0000 7ff1e8f39640 1 -- 172.21.15.29:0/1393887964 wait complete. 2023-10-16T15:54:40.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T15:54:40.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.383+0000 7f58723ed640 1 Processor -- start 2023-10-16T15:54:40.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.383+0000 7f58723ed640 1 -- start start 2023-10-16T15:54:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.384+0000 7f58723ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 0x7f586c0cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.384+0000 7f58723ed640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c0b3730 con 0x7f586c0b4740 2023-10-16T15:54:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.384+0000 7f58723ed640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c0b3980 con 0x7f586c15aa60 2023-10-16T15:54:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.384+0000 7f58723ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c0b3c60 con 0x7f586c0cc2e0 2023-10-16T15:54:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f5871bec640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f586c0b4740 0x7f586c0b4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53236/0 (socket says 172.21.15.29:53236) 2023-10-16T15:54:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f5871bec640 1 -- 172.21.15.29:0/856321663 learned_addr learned my addr 172.21.15.29:0/856321663 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f5870bea640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 0x7f586c0cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412413250 0 0) 0x7f586c0b3730 con 0x7f586c0b4740 2023-10-16T15:54:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5854003520 con 0x7f586c0b4740 2023-10-16T15:54:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919618568 0 0) 0x7f5854003520 con 0x7f586c0b4740 2023-10-16T15:54:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 msgr2=0x7f586c0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 0x7f586c0cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 >> v1:172.21.15.170:6789/0 conn(0x7f586c15aa60 legacy=0x7f586c0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.385+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c0b3f40 con 0x7f586c0b4740 2023-10-16T15:54:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5864001670 con 0x7f586c0b4740 2023-10-16T15:54:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 >> v1:172.21.15.29:6789/0 conn(0x7f586c0b4740 legacy=0x7f586c0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 0x7f58540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5854003ee0 0x7f58540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854004d00 0x7f5854005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c0b3c60 con 0x7f5854003ee0 2023-10-16T15:54:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585400b9e0 con 0x7f5854004d00 2023-10-16T15:54:40.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585400bba0 con 0x7f586c0cc2e0 2023-10-16T15:54:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5870bea640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5854003ee0 0x7f58540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5871bec640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854004d00 0x7f5854005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f58713eb640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 0x7f58540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5870bea640 1 -- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 msgr2=0x7f58540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:40.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5870bea640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0cc2e0 0x7f58540039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5870bea640 1 -- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854004d00 msgr2=0x7f5854005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5870bea640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854004d00 0x7f5854005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.386+0000 7f5870bea640 1 -- 172.21.15.29:0/856321663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5854003520 con 0x7f5854003ee0 2023-10-16T15:54:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f5870bea640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5854003ee0 0x7f58540047c0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f5860009ac0 tx=0x7f58540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64e777d59d79190 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586002c600 con 0x7f5854003ee0 2023-10-16T15:54:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586002c760 con 0x7f5854003ee0 2023-10-16T15:54:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f586bfff640 1 -- 172.21.15.29:0/856321663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586002caf0 con 0x7f5854003ee0 2023-10-16T15:54:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f58723ed640 1 -- 172.21.15.29:0/856321663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5854003ee0 msgr2=0x7f58540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f58723ed640 1 --2- 172.21.15.29:0/856321663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5854003ee0 0x7f58540047c0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f5860009ac0 tx=0x7f58540057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f5871bec640 1 -- 172.21.15.29:0/856321663 reap_dead start 2023-10-16T15:54:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f58723ed640 1 -- 172.21.15.29:0/856321663 shutdown_connections 2023-10-16T15:54:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f58723ed640 1 -- 172.21.15.29:0/856321663 >> 172.21.15.29:0/856321663 conn(0x7f586c065ed0 msgr2=0x7f586c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.388+0000 7f58723ed640 1 -- 172.21.15.29:0/856321663 shutdown_connections 2023-10-16T15:54:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.389+0000 7f58723ed640 1 -- 172.21.15.29:0/856321663 wait complete. 2023-10-16T15:54:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.390+0000 7f58723ed640 1 Processor -- start 2023-10-16T15:54:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.390+0000 7f58723ed640 1 -- start start 2023-10-16T15:54:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.391+0000 7f58723ed640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854003ee0 0x7f586c1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.391+0000 7f58723ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 0x7f586c0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.391+0000 7f58723ed640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0ca0c0 0x7f586c0c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.391+0000 7f58723ed640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c155720 con 0x7f586c1edcd0 2023-10-16T15:54:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.391+0000 7f58723ed640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c0ca540 con 0x7f5854003ee0 2023-10-16T15:54:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.391+0000 7f58723ed640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c15ce90 con 0x7f586c0ca0c0 2023-10-16T15:54:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 0x7f586c0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 0x7f586c0c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56192/0 (socket says 172.21.15.29:56192) 2023-10-16T15:54:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 -- 172.21.15.29:0/1732440513 learned_addr learned my addr 172.21.15.29:0/1732440513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f58713eb640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854003ee0 0x7f586c1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 -- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0ca0c0 msgr2=0x7f586c0c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0ca0c0 0x7f586c0c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 -- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854003ee0 msgr2=0x7f586c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854003ee0 0x7f586c1551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f5870bea640 1 -- 172.21.15.29:0/1732440513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c15d170 con 0x7f586c1edcd0 2023-10-16T15:54:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.392+0000 7f58713eb640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854003ee0 0x7f586c1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.393+0000 7f5870bea640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 0x7f586c0c9b80 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f586002c1a0 tx=0x7f5860004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.393+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860003810 con 0x7f586c1edcd0 2023-10-16T15:54:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.393+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5860003970 con 0x7f586c1edcd0 2023-10-16T15:54:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.393+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860003ce0 con 0x7f586c1edcd0 2023-10-16T15:54:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.393+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c15d450 con 0x7f586c1edcd0 2023-10-16T15:54:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.393+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c15da10 con 0x7f586c1edcd0 2023-10-16T15:54:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.394+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586c15dba0 con 0x7f586c1edcd0 2023-10-16T15:54:40.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.395+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5860034440 con 0x7f586c1edcd0 2023-10-16T15:54:40.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.396+0000 7f5869ffb640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f585806b0b0 0x7f585806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:40.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.396+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58600abb10 con 0x7f586c1edcd0 2023-10-16T15:54:40.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.397+0000 7f58713eb640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f585806b0b0 0x7f585806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:40.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.398+0000 7f58713eb640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f585806b0b0 0x7f585806d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f585c0012e0 tx=0x7f585c009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:40.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.400+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58600767a0 con 0x7f586c1edcd0 2023-10-16T15:54:40.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:40.625+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f586c0aff00 con 0x7f586c1edcd0 2023-10-16T15:54:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.272+0000 7f5869ffb640 1 -- 172.21.15.29:0/1732440513 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v123) v1 ==== 118+0+0 (secure 0 0 0) 0x7f586004c0b0 con 0x7f586c1edcd0 2023-10-16T15:54:41.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f585806b0b0 msgr2=0x7f585806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:41.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f585806b0b0 0x7f585806d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f585c0012e0 tx=0x7f585c009040 comp rx=0 tx=0).stop 2023-10-16T15:54:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 msgr2=0x7f586c0c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 0x7f586c0c9b80 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f586002c1a0 tx=0x7f5860004730 comp rx=0 tx=0).stop 2023-10-16T15:54:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 shutdown_connections 2023-10-16T15:54:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f585806b0b0 0x7f585806d570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f586c1edcd0 0x7f586c0c9b80 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f586c0ca0c0 0x7f586c0c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 --2- 172.21.15.29:0/1732440513 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5854003ee0 0x7f586c1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 >> 172.21.15.29:0/1732440513 conn(0x7f586c065ed0 msgr2=0x7f586c071380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 shutdown_connections 2023-10-16T15:54:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.276+0000 7f58723ed640 1 -- 172.21.15.29:0/1732440513 wait complete. 2023-10-16T15:54:41.298 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:54:41.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T15:54:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.529+0000 7f47cb59e640 1 Processor -- start 2023-10-16T15:54:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.529+0000 7f47cb59e640 1 -- start start 2023-10-16T15:54:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.530+0000 7f47cb59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f47cc15cd70 0x7f47cc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.530+0000 7f47cb59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 0x7f47cc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.530+0000 7f47cb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc15d150 con 0x7f47cc0b8480 2023-10-16T15:54:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.530+0000 7f47cb59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc0d1180 con 0x7f47cc15cd70 2023-10-16T15:54:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.530+0000 7f47cb59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0d1460 con 0x7f47cc0cded0 2023-10-16T15:54:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.531+0000 7f47c9d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f47cc0b8480 0x7f47cc0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53240/0 (socket says 172.21.15.29:53240) 2023-10-16T15:54:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.531+0000 7f47c9d9b640 1 -- 172.21.15.29:0/747519839 learned_addr learned my addr 172.21.15.29:0/747519839 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.531+0000 7f47ca59c640 1 --2- 172.21.15.29:0/747519839 >> v2:172.21.15.170:3300/0 conn(0x7f47cc15cd70 0x7f47cc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.531+0000 7f47cad9d640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 0x7f47cc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.531+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878185720 0 0) 0x7f47cc15d150 con 0x7f47cc0b8480 2023-10-16T15:54:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.531+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b8003520 con 0x7f47cc0b8480 2023-10-16T15:54:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4022058582 0 0) 0x7f47b8003520 con 0x7f47cc0b8480 2023-10-16T15:54:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 msgr2=0x7f47cc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 0x7f47cc0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 >> v2:172.21.15.170:3300/0 conn(0x7f47cc15cd70 msgr2=0x7f47cc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 --2- 172.21.15.29:0/747519839 >> v2:172.21.15.170:3300/0 conn(0x7f47cc15cd70 0x7f47cc0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0d1740 con 0x7f47cc0b8480 2023-10-16T15:54:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f47b4001690 con 0x7f47cc0b8480 2023-10-16T15:54:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 >> v1:172.21.15.29:6789/0 conn(0x7f47cc0b8480 legacy=0x7f47cc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b80047c0 0x7f47b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 0x7f47b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0d1180 con 0x7f47b8003970 2023-10-16T15:54:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0d1460 con 0x7f47b80047c0 2023-10-16T15:54:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b800b980 con 0x7f47cc0cded0 2023-10-16T15:54:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.532+0000 7f47ca59c640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47c9d9b640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b80047c0 0x7f47b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47ca59c640 1 -- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 msgr2=0x7f47b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47ca59c640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc0cded0 0x7f47b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47ca59c640 1 -- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b80047c0 msgr2=0x7f47b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47ca59c640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b80047c0 0x7f47b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47ca59c640 1 -- 172.21.15.29:0/747519839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b8003520 con 0x7f47b8003970 2023-10-16T15:54:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.533+0000 7f47c9d9b640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47b80047c0 0x7f47b8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.534+0000 7f47ca59c640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47b8003d90 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f47bc009af0 tx=0x7f47b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd983bf5b4b40d5f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.534+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc02c710 con 0x7f47b8003970 2023-10-16T15:54:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.534+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc02c870 con 0x7f47b8003970 2023-10-16T15:54:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.534+0000 7f47c959a640 1 -- 172.21.15.29:0/747519839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc02cc00 con 0x7f47b8003970 2023-10-16T15:54:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cb59e640 1 -- 172.21.15.29:0/747519839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 msgr2=0x7f47b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cb59e640 1 --2- 172.21.15.29:0/747519839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47b8003d90 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f47bc009af0 tx=0x7f47b80057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cad9d640 1 -- 172.21.15.29:0/747519839 reap_dead start 2023-10-16T15:54:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cb59e640 1 -- 172.21.15.29:0/747519839 shutdown_connections 2023-10-16T15:54:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cb59e640 1 -- 172.21.15.29:0/747519839 >> 172.21.15.29:0/747519839 conn(0x7f47cc0cf570 msgr2=0x7f47cc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cb59e640 1 -- 172.21.15.29:0/747519839 shutdown_connections 2023-10-16T15:54:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.535+0000 7f47cb59e640 1 -- 172.21.15.29:0/747519839 wait complete. 2023-10-16T15:54:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.537+0000 7f47cb59e640 1 Processor -- start 2023-10-16T15:54:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.537+0000 7f47cb59e640 1 -- start start 2023-10-16T15:54:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47cb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47cc15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47cb59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc15f1d0 0x7f47cc15a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47cb59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0cbd90 0x7f47cc0cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47cb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0cc750 con 0x7f47b8003970 2023-10-16T15:54:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47cb59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc15e5a0 con 0x7f47cc0cbd90 2023-10-16T15:54:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47cb59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0a9f50 con 0x7f47cc15f1d0 2023-10-16T15:54:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47ca59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47cc15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47ca59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47cc15dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56216/0 (socket says 172.21.15.29:56216) 2023-10-16T15:54:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.538+0000 7f47ca59c640 1 -- 172.21.15.29:0/4282783473 learned_addr learned my addr 172.21.15.29:0/4282783473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47cad9d640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0cbd90 0x7f47cc0cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47c9d9b640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc15f1d0 0x7f47cc15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47ca59c640 1 -- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc15f1d0 msgr2=0x7f47cc15a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47ca59c640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc15f1d0 0x7f47cc15a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47ca59c640 1 -- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0cbd90 msgr2=0x7f47cc0cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47ca59c640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0cbd90 0x7f47cc0cc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47ca59c640 1 -- 172.21.15.29:0/4282783473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0a9620 con 0x7f47b8003970 2023-10-16T15:54:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.539+0000 7f47ca59c640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47cc15dd50 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f47bc037040 tx=0x7f47bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.540+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc003810 con 0x7f47b8003970 2023-10-16T15:54:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.540+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc02de90 con 0x7f47b8003970 2023-10-16T15:54:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.540+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc003b70 con 0x7f47b8003970 2023-10-16T15:54:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.540+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc0a9900 con 0x7f47b8003970 2023-10-16T15:54:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.540+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc15e290 con 0x7f47b8003970 2023-10-16T15:54:41.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.541+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc04e9b0 con 0x7f47b8003970 2023-10-16T15:54:41.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.542+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f47bc003cd0 con 0x7f47b8003970 2023-10-16T15:54:41.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.543+0000 7f47c2ffd640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f47ac06b0e0 0x7f47ac06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:41.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.543+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f47bc0ab840 con 0x7f47b8003970 2023-10-16T15:54:41.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.544+0000 7f47c9d9b640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f47ac06b0e0 0x7f47ac06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:41.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.545+0000 7f47c9d9b640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f47ac06b0e0 0x7f47ac06d5a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47b4001eb0 tx=0x7f47b4009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.548+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f47bc076610 con 0x7f47b8003970 2023-10-16T15:54:41.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:41.781+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f47cc0cc750 con 0x7f47b8003970 2023-10-16T15:54:42.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.282+0000 7f47c2ffd640 1 -- 172.21.15.29:0/4282783473 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v124) v1 ==== 122+0+0 (secure 0 0 0) 0x7f47bc079c30 con 0x7f47b8003970 2023-10-16T15:54:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f47ac06b0e0 msgr2=0x7f47ac06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f47ac06b0e0 0x7f47ac06d5a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f47b4001eb0 tx=0x7f47b4009040 comp rx=0 tx=0).stop 2023-10-16T15:54:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 msgr2=0x7f47cc15dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47cc15dd50 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f47bc037040 tx=0x7f47bc004730 comp rx=0 tx=0).stop 2023-10-16T15:54:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 shutdown_connections 2023-10-16T15:54:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f47ac06b0e0 0x7f47ac06d5a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f47cc0cbd90 0x7f47cc0cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f47cc15f1d0 0x7f47cc15a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 --2- 172.21.15.29:0/4282783473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47b8003970 0x7f47cc15dd50 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.285+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 >> 172.21.15.29:0/4282783473 conn(0x7f47cc0cf570 msgr2=0x7f47cc15c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.286+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 shutdown_connections 2023-10-16T15:54:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.286+0000 7f47cb59e640 1 -- 172.21.15.29:0/4282783473 wait complete. 2023-10-16T15:54:42.310 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T15:54:42.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.530+0000 7fd50e975640 1 Processor -- start 2023-10-16T15:54:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.530+0000 7fd50e975640 1 -- start start 2023-10-16T15:54:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50e975640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd5080cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50e975640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50815dc60 con 0x7fd5080cda20 2023-10-16T15:54:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50e975640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50815d370 con 0x7fd50806a4b0 2023-10-16T15:54:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50e975640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c66f0 con 0x7fd5080c6310 2023-10-16T15:54:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50d172640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd5080cda20 0x7fd508069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53246/0 (socket says 172.21.15.29:53246) 2023-10-16T15:54:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50d172640 1 -- 172.21.15.29:0/1127704507 learned_addr learned my addr 172.21.15.29:0/1127704507 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295278232 0 0) 0x7fd50815dc60 con 0x7fd5080cda20 2023-10-16T15:54:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4003520 con 0x7fd5080cda20 2023-10-16T15:54:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.532+0000 7fd50d973640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd5080cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3413032521 0 0) 0x7fd4f4003520 con 0x7fd5080cda20 2023-10-16T15:54:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 msgr2=0x7fd5080cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd5080cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 >> v1:172.21.15.170:6789/0 conn(0x7fd50806a4b0 legacy=0x7fd508145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50815e0f0 con 0x7fd5080cda20 2023-10-16T15:54:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fd4f8001670 con 0x7fd5080cda20 2023-10-16T15:54:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 >> v1:172.21.15.29:6789/0 conn(0x7fd5080cda20 legacy=0x7fd508069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd4f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4f4004020 0x7fd4f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c66f0 con 0x7fd4f4004d00 2023-10-16T15:54:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400bc60 con 0x7fd4f4004020 2023-10-16T15:54:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400be20 con 0x7fd5080c6310 2023-10-16T15:54:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.533+0000 7fd50e174640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50d172640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4f4004020 0x7fd4f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50d973640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd4f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50e174640 1 -- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 msgr2=0x7fd4f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50e174640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd4f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50e174640 1 -- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4f4004020 msgr2=0x7fd4f40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50e174640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4f4004020 0x7fd4f40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50e174640 1 -- 172.21.15.29:0/1127704507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4003520 con 0x7fd4f4004d00 2023-10-16T15:54:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50d172640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4f4004020 0x7fd4f40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.534+0000 7fd50d973640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd5080c6310 0x7fd4f40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.535+0000 7fd50e174640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fd500001720 tx=0x7fd4f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec654b48aad1a113 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.535+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5000052d0 con 0x7fd4f4004d00 2023-10-16T15:54:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.535+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50002e050 con 0x7fd4f4004d00 2023-10-16T15:54:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.535+0000 7fd50c971640 1 -- 172.21.15.29:0/1127704507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500003830 con 0x7fd4f4004d00 2023-10-16T15:54:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e975640 1 -- 172.21.15.29:0/1127704507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 msgr2=0x7fd4f4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e975640 1 --2- 172.21.15.29:0/1127704507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fd500001720 tx=0x7fd4f40057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e174640 1 -- 172.21.15.29:0/1127704507 reap_dead start 2023-10-16T15:54:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e975640 1 -- 172.21.15.29:0/1127704507 shutdown_connections 2023-10-16T15:54:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e975640 1 -- 172.21.15.29:0/1127704507 >> 172.21.15.29:0/1127704507 conn(0x7fd5080cf910 msgr2=0x7fd5080cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e975640 1 -- 172.21.15.29:0/1127704507 shutdown_connections 2023-10-16T15:54:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.536+0000 7fd50e975640 1 -- 172.21.15.29:0/1127704507 wait complete. 2023-10-16T15:54:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.538+0000 7fd50e975640 1 Processor -- start 2023-10-16T15:54:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.538+0000 7fd50e975640 1 -- start start 2023-10-16T15:54:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50e975640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4f4004d00 0x7fd508147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50e975640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 0x7fd5080b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50e975640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5080b2d60 0x7fd5080b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50e975640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081596b0 con 0x7fd5080b28e0 2023-10-16T15:54:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50e975640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508158d80 con 0x7fd5080b2d60 2023-10-16T15:54:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50e975640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd508148130 con 0x7fd4f4004d00 2023-10-16T15:54:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50d172640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 0x7fd5080b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50d172640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 0x7fd5080b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56230/0 (socket says 172.21.15.29:56230) 2023-10-16T15:54:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.539+0000 7fd50d172640 1 -- 172.21.15.29:0/2283395416 learned_addr learned my addr 172.21.15.29:0/2283395416 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50e174640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5080b2d60 0x7fd5080b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d973640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4f4004d00 0x7fd508147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d172640 1 -- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4f4004d00 msgr2=0x7fd508147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d172640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4f4004d00 0x7fd508147bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d172640 1 -- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5080b2d60 msgr2=0x7fd5080b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d172640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5080b2d60 0x7fd5080b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d172640 1 -- 172.21.15.29:0/2283395416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd508158ff0 con 0x7fd5080b28e0 2023-10-16T15:54:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50e174640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5080b2d60 0x7fd5080b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50d172640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 0x7fd5080b4bb0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800d910 tx=0x7fd4f800ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80074e0 con 0x7fd5080b28e0 2023-10-16T15:54:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8007640 con 0x7fd5080b28e0 2023-10-16T15:54:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5081592d0 con 0x7fd5080b28e0 2023-10-16T15:54:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.540+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80049e0 con 0x7fd5080b28e0 2023-10-16T15:54:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.541+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd508149d90 con 0x7fd5080b28e0 2023-10-16T15:54:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.542+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5081597f0 con 0x7fd5080b28e0 2023-10-16T15:54:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.543+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd4f8004b40 con 0x7fd5080b28e0 2023-10-16T15:54:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.544+0000 7fd4fe7fc640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fd4ec06b0b0 0x7fd4ec06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.544+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd4f8089650 con 0x7fd5080b28e0 2023-10-16T15:54:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.545+0000 7fd50d973640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fd4ec06b0b0 0x7fd4ec06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.546+0000 7fd50d973640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fd4ec06b0b0 0x7fd4ec06d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd504009fd0 tx=0x7fd50400b040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.549+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5081597f0 con 0x7fd5080b28e0 2023-10-16T15:54:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.778+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5081597f0 con 0x7fd5080b28e0 2023-10-16T15:54:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.779+0000 7fd4fe7fc640 1 -- 172.21.15.29:0/2283395416 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4930 (secure 0 0 0) 0x7fd5081597f0 con 0x7fd5080b28e0 2023-10-16T15:54:42.782 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:42.782 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":11613,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T15:54:42.782 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 124 2023-10-16T15:54:42.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fd4ec06b0b0 msgr2=0x7fd4ec06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fd4ec06b0b0 0x7fd4ec06d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd504009fd0 tx=0x7fd50400b040 comp rx=0 tx=0).stop 2023-10-16T15:54:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 msgr2=0x7fd5080b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 0x7fd5080b4bb0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800d910 tx=0x7fd4f800ddd0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 shutdown_connections 2023-10-16T15:54:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fd4ec06b0b0 0x7fd4ec06d570 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5080b28e0 0x7fd5080b4bb0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd5080b2d60 0x7fd5080b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 --2- 172.21.15.29:0/2283395416 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd4f4004d00 0x7fd508147bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 >> 172.21.15.29:0/2283395416 conn(0x7fd5080cf910 msgr2=0x7fd5080b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 shutdown_connections 2023-10-16T15:54:42.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:42.782+0000 7fd50e975640 1 -- 172.21.15.29:0/2283395416 wait complete. 2023-10-16T15:54:42.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:43.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.024+0000 7f71a27d7640 1 Processor -- start 2023-10-16T15:54:43.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.024+0000 7f71a27d7640 1 -- start start 2023-10-16T15:54:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a27d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f719c159950 0x7f719c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a27d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 0x7f719c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a27d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719c070df0 con 0x7f719c06c100 2023-10-16T15:54:43.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a27d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719c15e490 con 0x7f719c159950 2023-10-16T15:54:43.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a27d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719c15aef0 con 0x7f719c070450 2023-10-16T15:54:43.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a1fd6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f719c06c100 0x7f719c06c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53252/0 (socket says 172.21.15.29:53252) 2023-10-16T15:54:43.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.026+0000 7f71a1fd6640 1 -- 172.21.15.29:0/1716435743 learned_addr learned my addr 172.21.15.29:0/1716435743 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 0x7f719c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f71a17d5640 1 --2- 172.21.15.29:0/1716435743 >> v2:172.21.15.170:3300/0 conn(0x7f719c159950 0x7f719c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267351756 0 0) 0x7f719c070df0 con 0x7f719c06c100 2023-10-16T15:54:43.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7188003520 con 0x7f719c06c100 2023-10-16T15:54:43.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2937069272 0 0) 0x7f7188003520 con 0x7f719c06c100 2023-10-16T15:54:43.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 msgr2=0x7f719c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:43.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 0x7f719c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 >> v2:172.21.15.170:3300/0 conn(0x7f719c159950 msgr2=0x7f719c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:43.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 --2- 172.21.15.29:0/1716435743 >> v2:172.21.15.170:3300/0 conn(0x7f719c159950 0x7f719c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c06cc80 con 0x7f719c06c100 2023-10-16T15:54:43.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.027+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7194001670 con 0x7f719c06c100 2023-10-16T15:54:43.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 >> v1:172.21.15.29:6789/0 conn(0x7f719c06c100 legacy=0x7f719c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 0x7f7188003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7188003ef0 0x7f71880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7188004d00 0x7f7188005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c15e490 con 0x7f7188003ef0 2023-10-16T15:54:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c15aef0 con 0x7f7188004d00 2023-10-16T15:54:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718800b9d0 con 0x7f719c070450 2023-10-16T15:54:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7188003ef0 0x7f71880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a17d5640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 0x7f7188003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a1fd6640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7188004d00 0x7f7188005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a0fd4640 1 -- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 msgr2=0x7f7188003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f719c070450 0x7f7188003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a0fd4640 1 -- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7188004d00 msgr2=0x7f7188005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7188004d00 0x7f7188005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a0fd4640 1 -- 172.21.15.29:0/1716435743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7188003520 con 0x7f7188003ef0 2023-10-16T15:54:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.028+0000 7f71a1fd6640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7188004d00 0x7f7188005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7188003ef0 0x7f71880047c0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f718c009af0 tx=0x7f71880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f4dcfab1ddbeaf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:43.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c02c6b0 con 0x7f7188003ef0 2023-10-16T15:54:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c02c810 con 0x7f7188003ef0 2023-10-16T15:54:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f7193fff640 1 -- 172.21.15.29:0/1716435743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c02cba0 con 0x7f7188003ef0 2023-10-16T15:54:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f71a27d7640 1 -- 172.21.15.29:0/1716435743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7188003ef0 msgr2=0x7f71880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1716435743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7188003ef0 0x7f71880047c0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f718c009af0 tx=0x7f71880057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.030+0000 7f71a1fd6640 1 -- 172.21.15.29:0/1716435743 reap_dead start 2023-10-16T15:54:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.031+0000 7f71a27d7640 1 -- 172.21.15.29:0/1716435743 shutdown_connections 2023-10-16T15:54:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.031+0000 7f71a27d7640 1 -- 172.21.15.29:0/1716435743 >> 172.21.15.29:0/1716435743 conn(0x7f719c0ce6a0 msgr2=0x7f719c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.031+0000 7f71a27d7640 1 -- 172.21.15.29:0/1716435743 shutdown_connections 2023-10-16T15:54:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.031+0000 7f71a27d7640 1 -- 172.21.15.29:0/1716435743 wait complete. 2023-10-16T15:54:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.033+0000 7f71a27d7640 1 Processor -- start 2023-10-16T15:54:43.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.033+0000 7f71a27d7640 1 -- start start 2023-10-16T15:54:43.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a27d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7188003ef0 0x7f719c1f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a27d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c1f6540 0x7f719c0acf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a27d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 0x7f719c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a27d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c0ad940 con 0x7f719c0ad4b0 2023-10-16T15:54:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a27d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c1564d0 con 0x7f719c1f6540 2023-10-16T15:54:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a27d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719c156740 con 0x7f7188003ef0 2023-10-16T15:54:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 0x7f719c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 0x7f719c0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56250/0 (socket says 172.21.15.29:56250) 2023-10-16T15:54:43.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 -- 172.21.15.29:0/1752643763 learned_addr learned my addr 172.21.15.29:0/1752643763 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c1f6540 0x7f719c0acf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a17d5640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7188003ef0 0x7f719c1f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 -- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7188003ef0 msgr2=0x7f719c1f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7188003ef0 0x7f719c1f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 -- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c1f6540 msgr2=0x7f719c0acf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c1f6540 0x7f719c0acf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.034+0000 7f71a1fd6640 1 -- 172.21.15.29:0/1752643763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c156a20 con 0x7f719c0ad4b0 2023-10-16T15:54:43.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f71a0fd4640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c1f6540 0x7f719c0acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:43.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f71a1fd6640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 0x7f719c0c5770 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f719c06d350 tx=0x7f719400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:43.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194007440 con 0x7f719c0ad4b0 2023-10-16T15:54:43.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71940075a0 con 0x7f719c0ad4b0 2023-10-16T15:54:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940049e0 con 0x7f719c0ad4b0 2023-10-16T15:54:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c156d00 con 0x7f719c0ad4b0 2023-10-16T15:54:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.035+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c0c5cb0 con 0x7f719c0ad4b0 2023-10-16T15:54:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.036+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c157220 con 0x7f719c0ad4b0 2023-10-16T15:54:43.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.037+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7194004b40 con 0x7f719c0ad4b0 2023-10-16T15:54:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.039+0000 7f7191ffb640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f718006b0b0 0x7f718006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.039+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f71940898e0 con 0x7f719c0ad4b0 2023-10-16T15:54:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.040+0000 7f71a17d5640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f718006b0b0 0x7f718006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.041+0000 7f71a17d5640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f718006b0b0 0x7f718006d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7198007380 tx=0x7f719800b040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.043+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f719c157220 con 0x7f719c0ad4b0 2023-10-16T15:54:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.240+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f719c157220 con 0x7f719c0ad4b0 2023-10-16T15:54:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.243+0000 7f7191ffb640 1 -- 172.21.15.29:0/1752643763 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+16563 (secure 0 0 0) 0x7f719c157220 con 0x7f719c0ad4b0 2023-10-16T15:54:43.246 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:43.246 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":246,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:40.167118+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:43.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.246+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f718006b0b0 msgr2=0x7f718006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.246+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f718006b0b0 0x7f718006d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7198007380 tx=0x7f719800b040 comp rx=0 tx=0).stop 2023-10-16T15:54:43.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.246+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 msgr2=0x7f719c0c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.246+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 0x7f719c0c5770 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f719c06d350 tx=0x7f719400d9d0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 shutdown_connections 2023-10-16T15:54:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f718006b0b0 0x7f718006d570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f719c0ad4b0 0x7f719c0c5770 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f719c1f6540 0x7f719c0acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 --2- 172.21.15.29:0/1752643763 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7188003ef0 0x7f719c1f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 >> 172.21.15.29:0/1752643763 conn(0x7f719c0ce6a0 msgr2=0x7f719c06b410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:43.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 shutdown_connections 2023-10-16T15:54:43.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.247+0000 7f71a27d7640 1 -- 172.21.15.29:0/1752643763 wait complete. 2023-10-16T15:54:43.269 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:54:43.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:54:43.271 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T15:54:43.269+0000 7fd6bf6f5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:54:43.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.495+0000 7f9176b4e640 1 Processor -- start 2023-10-16T15:54:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.495+0000 7f9176b4e640 1 -- start start 2023-10-16T15:54:43.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.496+0000 7f9176b4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91700b4c80 0x7f9170167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.496+0000 7f9176b4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91700d15a0 con 0x7f917015cfc0 2023-10-16T15:54:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.496+0000 7f9176b4e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91700d1880 con 0x7f917015cbe0 2023-10-16T15:54:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.496+0000 7f9176b4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91700d1b60 con 0x7f91700b4c80 2023-10-16T15:54:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f917534b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f917015cfc0 0x7f91700b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53266/0 (socket says 172.21.15.29:53266) 2023-10-16T15:54:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f917534b640 1 -- 172.21.15.29:0/1844092558 learned_addr learned my addr 172.21.15.29:0/1844092558 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:43.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f917634d640 1 --2- 172.21.15.29:0/1844092558 >> v2:172.21.15.29:3300/0 conn(0x7f91700b4c80 0x7f9170167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f9174b4a640 1 -- 172.21.15.29:0/1844092558 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445320986 0 0) 0x7f91700d15a0 con 0x7f917015cfc0 2023-10-16T15:54:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f9174b4a640 1 -- 172.21.15.29:0/1844092558 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915c003520 con 0x7f917015cfc0 2023-10-16T15:54:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f917634d640 1 -- 172.21.15.29:0/1844092558 >> v1:172.21.15.170:6789/0 conn(0x7f917015cbe0 legacy=0x7f91700d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f917634d640 1 -- 172.21.15.29:0/1844092558 >> v1:172.21.15.29:6789/0 conn(0x7f917015cfc0 legacy=0x7f91700b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.497+0000 7f917634d640 1 -- 172.21.15.29:0/1844092558 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91700d1e40 con 0x7f91700b4c80 2023-10-16T15:54:43.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.499+0000 7f917634d640 1 --2- 172.21.15.29:0/1844092558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91700b4c80 0x7f9170167910 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f91680097a0 tx=0x7f916802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbc870b7d9c571d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:43.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.499+0000 7f9174b4a640 1 -- 172.21.15.29:0/1844092558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168030610 con 0x7f91700b4c80 2023-10-16T15:54:43.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.499+0000 7f9174b4a640 1 -- 172.21.15.29:0/1844092558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9168030770 con 0x7f91700b4c80 2023-10-16T15:54:43.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.499+0000 7f9174b4a640 1 -- 172.21.15.29:0/1844092558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168030b00 con 0x7f91700b4c80 2023-10-16T15:54:43.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.499+0000 7f9176b4e640 1 -- 172.21.15.29:0/1844092558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91700b4c80 msgr2=0x7f9170167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.499+0000 7f9176b4e640 1 --2- 172.21.15.29:0/1844092558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91700b4c80 0x7f9170167910 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f91680097a0 tx=0x7f916802b960 comp rx=0 tx=0).stop 2023-10-16T15:54:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.500+0000 7f9176b4e640 1 -- 172.21.15.29:0/1844092558 shutdown_connections 2023-10-16T15:54:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.500+0000 7f9176b4e640 1 --2- 172.21.15.29:0/1844092558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91700b4c80 0x7f9170167910 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.500+0000 7f9176b4e640 1 -- 172.21.15.29:0/1844092558 >> 172.21.15.29:0/1844092558 conn(0x7f91700cf570 msgr2=0x7f91700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.500+0000 7f9176b4e640 1 -- 172.21.15.29:0/1844092558 shutdown_connections 2023-10-16T15:54:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.500+0000 7f9176b4e640 1 -- 172.21.15.29:0/1844092558 wait complete. 2023-10-16T15:54:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.501+0000 7f9176b4e640 1 Processor -- start 2023-10-16T15:54:43.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.502+0000 7f9176b4e640 1 -- start start 2023-10-16T15:54:43.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9176b4e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91700b4c80 0x7f9170155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9176b4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 0x7f91700cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9176b4e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f917015cfc0 0x7f91700cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9176b4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9170156490 con 0x7f917015cbe0 2023-10-16T15:54:43.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9176b4e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91700ccdc0 con 0x7f917015cfc0 2023-10-16T15:54:43.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9176b4e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9170157ca0 con 0x7f91700b4c80 2023-10-16T15:54:43.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 0x7f91700cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 0x7f91700cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56276/0 (socket says 172.21.15.29:56276) 2023-10-16T15:54:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 -- 172.21.15.29:0/382761123 learned_addr learned my addr 172.21.15.29:0/382761123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9175b4c640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91700b4c80 0x7f9170155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 -- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91700b4c80 msgr2=0x7f9170155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917634d640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f917015cfc0 0x7f91700cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91700b4c80 0x7f9170155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 -- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f917015cfc0 msgr2=0x7f91700cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f917015cfc0 0x7f91700cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f917534b640 1 -- 172.21.15.29:0/382761123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168009400 con 0x7f917015cbe0 2023-10-16T15:54:43.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.503+0000 7f9175b4c640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91700b4c80 0x7f9170155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:43.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.504+0000 7f917534b640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 0x7f91700cc200 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f916000c3e0 tx=0x7f916000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:43.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.504+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160005ce0 con 0x7f917015cbe0 2023-10-16T15:54:43.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.504+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9160005e40 con 0x7f917015cbe0 2023-10-16T15:54:43.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.504+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91600072d0 con 0x7f917015cbe0 2023-10-16T15:54:43.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.504+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9170157f80 con 0x7f917015cbe0 2023-10-16T15:54:43.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.504+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91700cd0a0 con 0x7f917015cbe0 2023-10-16T15:54:43.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.506+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917004e9b0 con 0x7f917015cbe0 2023-10-16T15:54:43.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.506+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f91600074f0 con 0x7f917015cbe0 2023-10-16T15:54:43.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.507+0000 7f91667fc640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f915406b0b0 0x7f915406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:43.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.507+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9160088f20 con 0x7f917015cbe0 2023-10-16T15:54:43.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.508+0000 7f9175b4c640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f915406b0b0 0x7f915406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:43.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.509+0000 7f9175b4c640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f915406b0b0 0x7f915406d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f916c001d70 tx=0x7f916c009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:43.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.512+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9160053d00 con 0x7f917015cbe0 2023-10-16T15:54:43.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.703+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 --> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91700d1260 con 0x7f915406b0b0 2023-10-16T15:54:43.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.732+0000 7f91667fc640 1 -- 172.21.15.29:0/382761123 <== mgr.9022 v2:172.21.15.170:6832/58791103 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287134 (secure 0 0 0) 0x7f916c00d8c0 con 0x7f915406b0b0 2023-10-16T15:54:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:43.740 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:54:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.739+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f915406b0b0 msgr2=0x7f915406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.739+0000 7f9176b4e640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f915406b0b0 0x7f915406d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f916c001d70 tx=0x7f916c009040 comp rx=0 tx=0).stop 2023-10-16T15:54:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.739+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 msgr2=0x7f91700cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 0x7f91700cc200 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f916000c3e0 tx=0x7f916000c8a0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 shutdown_connections 2023-10-16T15:54:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f915406b0b0 0x7f915406d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917015cbe0 0x7f91700cc200 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f917015cfc0 0x7f91700cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 --2- 172.21.15.29:0/382761123 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f91700b4c80 0x7f9170155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 >> 172.21.15.29:0/382761123 conn(0x7f91700cf570 msgr2=0x7f91700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 shutdown_connections 2023-10-16T15:54:43.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:43.740+0000 7f9176b4e640 1 -- 172.21.15.29:0/382761123 wait complete. 2023-10-16T15:54:43.764 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T15:54:43.478207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":667,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":405336,"kb_used_data":25356,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754569384,"statfs":{"total":773094113280,"available":772679049216,"internally_reserved":0,"allocated":25964544,"data_stored":13362916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":388943417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002070"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173234+0000","last_change":"2023-10-16T15:54:39.213468+0000","last_active":"2023-10-16T15:54:40.173234+0000","last_peered":"2023-10-16T15:54:40.173234+0000","last_clean":"2023-10-16T15:54:40.173234+0000","last_became_active":"2023-10-16T15:54:39.213201+0000","last_became_peered":"2023-10-16T15:54:39.213201+0000","last_unstale":"2023-10-16T15:54:40.173234+0000","last_undegraded":"2023-10-16T15:54:40.173234+0000","last_fullsized":"2023-10-16T15:54:40.173234+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:54.453092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173154+0000","last_change":"2023-10-16T15:54:39.209387+0000","last_active":"2023-10-16T15:54:40.173154+0000","last_peered":"2023-10-16T15:54:40.173154+0000","last_clean":"2023-10-16T15:54:40.173154+0000","last_became_active":"2023-10-16T15:54:39.207929+0000","last_became_peered":"2023-10-16T15:54:39.207929+0000","last_unstale":"2023-10-16T15:54:40.173154+0000","last_undegraded":"2023-10-16T15:54:40.173154+0000","last_fullsized":"2023-10-16T15:54:40.173154+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:21.226148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355505+0000","last_change":"2023-10-16T15:54:39.205624+0000","last_active":"2023-10-16T15:54:40.355505+0000","last_peered":"2023-10-16T15:54:40.355505+0000","last_clean":"2023-10-16T15:54:40.355505+0000","last_became_active":"2023-10-16T15:54:39.205294+0000","last_became_peered":"2023-10-16T15:54:39.205294+0000","last_unstale":"2023-10-16T15:54:40.355505+0000","last_undegraded":"2023-10-16T15:54:40.355505+0000","last_fullsized":"2023-10-16T15:54:40.355505+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.276795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351329+0000","last_change":"2023-10-16T15:54:39.203634+0000","last_active":"2023-10-16T15:54:40.351329+0000","last_peered":"2023-10-16T15:54:40.351329+0000","last_clean":"2023-10-16T15:54:40.351329+0000","last_became_active":"2023-10-16T15:54:39.203333+0000","last_became_peered":"2023-10-16T15:54:39.203333+0000","last_unstale":"2023-10-16T15:54:40.351329+0000","last_undegraded":"2023-10-16T15:54:40.351329+0000","last_fullsized":"2023-10-16T15:54:40.351329+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.177094+0000","last_change":"2023-10-16T15:54:39.205871+0000","last_active":"2023-10-16T15:54:40.177094+0000","last_peered":"2023-10-16T15:54:40.177094+0000","last_clean":"2023-10-16T15:54:40.177094+0000","last_became_active":"2023-10-16T15:54:39.204811+0000","last_became_peered":"2023-10-16T15:54:39.204811+0000","last_unstale":"2023-10-16T15:54:40.177094+0000","last_undegraded":"2023-10-16T15:54:40.177094+0000","last_fullsized":"2023-10-16T15:54:40.177094+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:59.938902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352180+0000","last_change":"2023-10-16T15:54:39.198994+0000","last_active":"2023-10-16T15:54:40.352180+0000","last_peered":"2023-10-16T15:54:40.352180+0000","last_clean":"2023-10-16T15:54:40.352180+0000","last_became_active":"2023-10-16T15:54:39.198650+0000","last_became_peered":"2023-10-16T15:54:39.198650+0000","last_unstale":"2023-10-16T15:54:40.352180+0000","last_undegraded":"2023-10-16T15:54:40.352180+0000","last_fullsized":"2023-10-16T15:54:40.352180+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350554+0000","last_change":"2023-10-16T15:54:39.190398+0000","last_active":"2023-10-16T15:54:40.350554+0000","last_peered":"2023-10-16T15:54:40.350554+0000","last_clean":"2023-10-16T15:54:40.350554+0000","last_became_active":"2023-10-16T15:54:39.189834+0000","last_became_peered":"2023-10-16T15:54:39.189834+0000","last_unstale":"2023-10-16T15:54:40.350554+0000","last_undegraded":"2023-10-16T15:54:40.350554+0000","last_fullsized":"2023-10-16T15:54:40.350554+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351163+0000","last_change":"2023-10-16T15:54:39.203378+0000","last_active":"2023-10-16T15:54:40.351163+0000","last_peered":"2023-10-16T15:54:40.351163+0000","last_clean":"2023-10-16T15:54:40.351163+0000","last_became_active":"2023-10-16T15:54:39.202647+0000","last_became_peered":"2023-10-16T15:54:39.202647+0000","last_unstale":"2023-10-16T15:54:40.351163+0000","last_undegraded":"2023-10-16T15:54:40.351163+0000","last_fullsized":"2023-10-16T15:54:40.351163+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:49:25.249610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.177972+0000","last_change":"2023-10-16T15:54:39.190692+0000","last_active":"2023-10-16T15:54:40.177972+0000","last_peered":"2023-10-16T15:54:40.177972+0000","last_clean":"2023-10-16T15:54:40.177972+0000","last_became_active":"2023-10-16T15:54:39.189950+0000","last_became_peered":"2023-10-16T15:54:39.189950+0000","last_unstale":"2023-10-16T15:54:40.177972+0000","last_undegraded":"2023-10-16T15:54:40.177972+0000","last_fullsized":"2023-10-16T15:54:40.177972+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:05.334097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352055+0000","last_change":"2023-10-16T15:54:39.203147+0000","last_active":"2023-10-16T15:54:40.352055+0000","last_peered":"2023-10-16T15:54:40.352055+0000","last_clean":"2023-10-16T15:54:40.352055+0000","last_became_active":"2023-10-16T15:54:39.202529+0000","last_became_peered":"2023-10-16T15:54:39.202529+0000","last_unstale":"2023-10-16T15:54:40.352055+0000","last_undegraded":"2023-10-16T15:54:40.352055+0000","last_fullsized":"2023-10-16T15:54:40.352055+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354384+0000","last_change":"2023-10-16T15:54:39.200394+0000","last_active":"2023-10-16T15:54:40.354384+0000","last_peered":"2023-10-16T15:54:40.354384+0000","last_clean":"2023-10-16T15:54:40.354384+0000","last_became_active":"2023-10-16T15:54:39.199822+0000","last_became_peered":"2023-10-16T15:54:39.199822+0000","last_unstale":"2023-10-16T15:54:40.354384+0000","last_undegraded":"2023-10-16T15:54:40.354384+0000","last_fullsized":"2023-10-16T15:54:40.354384+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:38.462835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352956+0000","last_change":"2023-10-16T15:54:39.210009+0000","last_active":"2023-10-16T15:54:40.352956+0000","last_peered":"2023-10-16T15:54:40.352956+0000","last_clean":"2023-10-16T15:54:40.352956+0000","last_became_active":"2023-10-16T15:54:39.209277+0000","last_became_peered":"2023-10-16T15:54:39.209277+0000","last_unstale":"2023-10-16T15:54:40.352956+0000","last_undegraded":"2023-10-16T15:54:40.352956+0000","last_fullsized":"2023-10-16T15:54:40.352956+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352987+0000","last_change":"2023-10-16T15:54:39.194527+0000","last_active":"2023-10-16T15:54:40.352987+0000","last_peered":"2023-10-16T15:54:40.352987+0000","last_clean":"2023-10-16T15:54:40.352987+0000","last_became_active":"2023-10-16T15:54:39.194123+0000","last_became_peered":"2023-10-16T15:54:39.194123+0000","last_unstale":"2023-10-16T15:54:40.352987+0000","last_undegraded":"2023-10-16T15:54:40.352987+0000","last_fullsized":"2023-10-16T15:54:40.352987+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172658+0000","last_change":"2023-10-16T15:54:39.208938+0000","last_active":"2023-10-16T15:54:40.172658+0000","last_peered":"2023-10-16T15:54:40.172658+0000","last_clean":"2023-10-16T15:54:40.172658+0000","last_became_active":"2023-10-16T15:54:39.208579+0000","last_became_peered":"2023-10-16T15:54:39.208579+0000","last_unstale":"2023-10-16T15:54:40.172658+0000","last_undegraded":"2023-10-16T15:54:40.172658+0000","last_fullsized":"2023-10-16T15:54:40.172658+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:32.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178736+0000","last_change":"2023-10-16T15:54:39.187772+0000","last_active":"2023-10-16T15:54:40.178736+0000","last_peered":"2023-10-16T15:54:40.178736+0000","last_clean":"2023-10-16T15:54:40.178736+0000","last_became_active":"2023-10-16T15:54:39.187210+0000","last_became_peered":"2023-10-16T15:54:39.187210+0000","last_unstale":"2023-10-16T15:54:40.178736+0000","last_undegraded":"2023-10-16T15:54:40.178736+0000","last_fullsized":"2023-10-16T15:54:40.178736+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:31.089495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352161+0000","last_change":"2023-10-16T15:54:39.199187+0000","last_active":"2023-10-16T15:54:40.352161+0000","last_peered":"2023-10-16T15:54:40.352161+0000","last_clean":"2023-10-16T15:54:40.352161+0000","last_became_active":"2023-10-16T15:54:39.195076+0000","last_became_peered":"2023-10-16T15:54:39.195076+0000","last_unstale":"2023-10-16T15:54:40.352161+0000","last_undegraded":"2023-10-16T15:54:40.352161+0000","last_fullsized":"2023-10-16T15:54:40.352161+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:01.081391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352979+0000","last_change":"2023-10-16T15:54:39.212389+0000","last_active":"2023-10-16T15:54:40.352979+0000","last_peered":"2023-10-16T15:54:40.352979+0000","last_clean":"2023-10-16T15:54:40.352979+0000","last_became_active":"2023-10-16T15:54:39.212017+0000","last_became_peered":"2023-10-16T15:54:39.212017+0000","last_unstale":"2023-10-16T15:54:40.352979+0000","last_undegraded":"2023-10-16T15:54:40.352979+0000","last_fullsized":"2023-10-16T15:54:40.352979+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173250+0000","last_change":"2023-10-16T15:54:39.214749+0000","last_active":"2023-10-16T15:54:40.173250+0000","last_peered":"2023-10-16T15:54:40.173250+0000","last_clean":"2023-10-16T15:54:40.173250+0000","last_became_active":"2023-10-16T15:54:39.214456+0000","last_became_peered":"2023-10-16T15:54:39.214456+0000","last_unstale":"2023-10-16T15:54:40.173250+0000","last_undegraded":"2023-10-16T15:54:40.173250+0000","last_fullsized":"2023-10-16T15:54:40.173250+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:36.851092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352958+0000","last_change":"2023-10-16T15:54:39.202583+0000","last_active":"2023-10-16T15:54:40.352958+0000","last_peered":"2023-10-16T15:54:40.352958+0000","last_clean":"2023-10-16T15:54:40.352958+0000","last_became_active":"2023-10-16T15:54:39.202278+0000","last_became_peered":"2023-10-16T15:54:39.202278+0000","last_unstale":"2023-10-16T15:54:40.352958+0000","last_undegraded":"2023-10-16T15:54:40.352958+0000","last_fullsized":"2023-10-16T15:54:40.352958+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355046+0000","last_change":"2023-10-16T15:54:39.205053+0000","last_active":"2023-10-16T15:54:40.355046+0000","last_peered":"2023-10-16T15:54:40.355046+0000","last_clean":"2023-10-16T15:54:40.355046+0000","last_became_active":"2023-10-16T15:54:39.202502+0000","last_became_peered":"2023-10-16T15:54:39.202502+0000","last_unstale":"2023-10-16T15:54:40.355046+0000","last_undegraded":"2023-10-16T15:54:40.355046+0000","last_fullsized":"2023-10-16T15:54:40.355046+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:38.045649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351189+0000","last_change":"2023-10-16T15:54:39.202366+0000","last_active":"2023-10-16T15:54:40.351189+0000","last_peered":"2023-10-16T15:54:40.351189+0000","last_clean":"2023-10-16T15:54:40.351189+0000","last_became_active":"2023-10-16T15:54:39.201981+0000","last_became_peered":"2023-10-16T15:54:39.201981+0000","last_unstale":"2023-10-16T15:54:40.351189+0000","last_undegraded":"2023-10-16T15:54:40.351189+0000","last_fullsized":"2023-10-16T15:54:40.351189+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:00.755938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350758+0000","last_change":"2023-10-16T15:54:39.209910+0000","last_active":"2023-10-16T15:54:40.350758+0000","last_peered":"2023-10-16T15:54:40.350758+0000","last_clean":"2023-10-16T15:54:40.350758+0000","last_became_active":"2023-10-16T15:54:39.209504+0000","last_became_peered":"2023-10-16T15:54:39.209504+0000","last_unstale":"2023-10-16T15:54:40.350758+0000","last_undegraded":"2023-10-16T15:54:40.350758+0000","last_fullsized":"2023-10-16T15:54:40.350758+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351869+0000","last_change":"2023-10-16T15:54:39.207379+0000","last_active":"2023-10-16T15:54:40.351869+0000","last_peered":"2023-10-16T15:54:40.351869+0000","last_clean":"2023-10-16T15:54:40.351869+0000","last_became_active":"2023-10-16T15:54:39.207010+0000","last_became_peered":"2023-10-16T15:54:39.207010+0000","last_unstale":"2023-10-16T15:54:40.351869+0000","last_undegraded":"2023-10-16T15:54:40.351869+0000","last_fullsized":"2023-10-16T15:54:40.351869+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:45.890691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172589+0000","last_change":"2023-10-16T15:54:39.199635+0000","last_active":"2023-10-16T15:54:40.172589+0000","last_peered":"2023-10-16T15:54:40.172589+0000","last_clean":"2023-10-16T15:54:40.172589+0000","last_became_active":"2023-10-16T15:54:39.199152+0000","last_became_peered":"2023-10-16T15:54:39.199152+0000","last_unstale":"2023-10-16T15:54:40.172589+0000","last_undegraded":"2023-10-16T15:54:40.172589+0000","last_fullsized":"2023-10-16T15:54:40.172589+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:45.206271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352011+0000","last_change":"2023-10-16T15:54:39.205509+0000","last_active":"2023-10-16T15:54:40.352011+0000","last_peered":"2023-10-16T15:54:40.352011+0000","last_clean":"2023-10-16T15:54:40.352011+0000","last_became_active":"2023-10-16T15:54:39.204206+0000","last_became_peered":"2023-10-16T15:54:39.204206+0000","last_unstale":"2023-10-16T15:54:40.352011+0000","last_undegraded":"2023-10-16T15:54:40.352011+0000","last_fullsized":"2023-10-16T15:54:40.352011+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352395+0000","last_change":"2023-10-16T15:54:39.194931+0000","last_active":"2023-10-16T15:54:40.352395+0000","last_peered":"2023-10-16T15:54:40.352395+0000","last_clean":"2023-10-16T15:54:40.352395+0000","last_became_active":"2023-10-16T15:54:39.194605+0000","last_became_peered":"2023-10-16T15:54:39.194605+0000","last_unstale":"2023-10-16T15:54:40.352395+0000","last_undegraded":"2023-10-16T15:54:40.352395+0000","last_fullsized":"2023-10-16T15:54:40.352395+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354085+0000","last_change":"2023-10-16T15:54:39.204169+0000","last_active":"2023-10-16T15:54:40.354085+0000","last_peered":"2023-10-16T15:54:40.354085+0000","last_clean":"2023-10-16T15:54:40.354085+0000","last_became_active":"2023-10-16T15:54:39.200108+0000","last_became_peered":"2023-10-16T15:54:39.200108+0000","last_unstale":"2023-10-16T15:54:40.354085+0000","last_undegraded":"2023-10-16T15:54:40.354085+0000","last_fullsized":"2023-10-16T15:54:40.354085+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:23.620457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172893+0000","last_change":"2023-10-16T15:54:39.203124+0000","last_active":"2023-10-16T15:54:40.172893+0000","last_peered":"2023-10-16T15:54:40.172893+0000","last_clean":"2023-10-16T15:54:40.172893+0000","last_became_active":"2023-10-16T15:54:39.200245+0000","last_became_peered":"2023-10-16T15:54:39.200245+0000","last_unstale":"2023-10-16T15:54:40.172893+0000","last_undegraded":"2023-10-16T15:54:40.172893+0000","last_fullsized":"2023-10-16T15:54:40.172893+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:43.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352069+0000","last_change":"2023-10-16T15:54:39.203684+0000","last_active":"2023-10-16T15:54:40.352069+0000","last_peered":"2023-10-16T15:54:40.352069+0000","last_clean":"2023-10-16T15:54:40.352069+0000","last_became_active":"2023-10-16T15:54:39.203327+0000","last_became_peered":"2023-10-16T15:54:39.203327+0000","last_unstale":"2023-10-16T15:54:40.352069+0000","last_undegraded":"2023-10-16T15:54:40.352069+0000","last_fullsized":"2023-10-16T15:54:40.352069+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:29.051715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178220+0000","last_change":"2023-10-16T15:54:39.203576+0000","last_active":"2023-10-16T15:54:40.178220+0000","last_peered":"2023-10-16T15:54:40.178220+0000","last_clean":"2023-10-16T15:54:40.178220+0000","last_became_active":"2023-10-16T15:54:39.203072+0000","last_became_peered":"2023-10-16T15:54:39.203072+0000","last_unstale":"2023-10-16T15:54:40.178220+0000","last_undegraded":"2023-10-16T15:54:40.178220+0000","last_fullsized":"2023-10-16T15:54:40.178220+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:05.472712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351137+0000","last_change":"2023-10-16T15:54:39.210101+0000","last_active":"2023-10-16T15:54:40.351137+0000","last_peered":"2023-10-16T15:54:40.351137+0000","last_clean":"2023-10-16T15:54:40.351137+0000","last_became_active":"2023-10-16T15:54:39.209787+0000","last_became_peered":"2023-10-16T15:54:39.209787+0000","last_unstale":"2023-10-16T15:54:40.351137+0000","last_undegraded":"2023-10-16T15:54:40.351137+0000","last_fullsized":"2023-10-16T15:54:40.351137+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172635+0000","last_change":"2023-10-16T15:54:39.198844+0000","last_active":"2023-10-16T15:54:40.172635+0000","last_peered":"2023-10-16T15:54:40.172635+0000","last_clean":"2023-10-16T15:54:40.172635+0000","last_became_active":"2023-10-16T15:54:39.198444+0000","last_became_peered":"2023-10-16T15:54:39.198444+0000","last_unstale":"2023-10-16T15:54:40.172635+0000","last_undegraded":"2023-10-16T15:54:40.172635+0000","last_fullsized":"2023-10-16T15:54:40.172635+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:47.149143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356170+0000","last_change":"2023-10-16T15:54:39.213572+0000","last_active":"2023-10-16T15:54:40.356170+0000","last_peered":"2023-10-16T15:54:40.356170+0000","last_clean":"2023-10-16T15:54:40.356170+0000","last_became_active":"2023-10-16T15:54:39.213226+0000","last_became_peered":"2023-10-16T15:54:39.213226+0000","last_unstale":"2023-10-16T15:54:40.356170+0000","last_undegraded":"2023-10-16T15:54:40.356170+0000","last_fullsized":"2023-10-16T15:54:40.356170+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:23.249562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352934+0000","last_change":"2023-10-16T15:54:38.194041+0000","last_active":"2023-10-16T15:54:40.352934+0000","last_peered":"2023-10-16T15:54:40.352934+0000","last_clean":"2023-10-16T15:54:40.352934+0000","last_became_active":"2023-10-16T15:54:38.193770+0000","last_became_peered":"2023-10-16T15:54:38.193770+0000","last_unstale":"2023-10-16T15:54:40.352934+0000","last_undegraded":"2023-10-16T15:54:40.352934+0000","last_fullsized":"2023-10-16T15:54:40.352934+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352886+0000","last_change":"2023-10-16T15:54:39.209350+0000","last_active":"2023-10-16T15:54:40.352886+0000","last_peered":"2023-10-16T15:54:40.352886+0000","last_clean":"2023-10-16T15:54:40.352886+0000","last_became_active":"2023-10-16T15:54:39.208829+0000","last_became_peered":"2023-10-16T15:54:39.208829+0000","last_unstale":"2023-10-16T15:54:40.352886+0000","last_undegraded":"2023-10-16T15:54:40.352886+0000","last_fullsized":"2023-10-16T15:54:40.352886+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:20.584860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355522+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:40.355522+0000","last_peered":"2023-10-16T15:54:40.355522+0000","last_clean":"2023-10-16T15:54:40.355522+0000","last_became_active":"2023-10-16T15:54:38.195767+0000","last_became_peered":"2023-10-16T15:54:38.195767+0000","last_unstale":"2023-10-16T15:54:40.355522+0000","last_undegraded":"2023-10-16T15:54:40.355522+0000","last_fullsized":"2023-10-16T15:54:40.355522+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:59.473824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353354+0000","last_change":"2023-10-16T15:54:39.209739+0000","last_active":"2023-10-16T15:54:40.353354+0000","last_peered":"2023-10-16T15:54:40.353354+0000","last_clean":"2023-10-16T15:54:40.353354+0000","last_became_active":"2023-10-16T15:54:39.208616+0000","last_became_peered":"2023-10-16T15:54:39.208616+0000","last_unstale":"2023-10-16T15:54:40.353354+0000","last_undegraded":"2023-10-16T15:54:40.353354+0000","last_fullsized":"2023-10-16T15:54:40.353354+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356264+0000","last_change":"2023-10-16T15:54:38.192004+0000","last_active":"2023-10-16T15:54:40.356264+0000","last_peered":"2023-10-16T15:54:40.356264+0000","last_clean":"2023-10-16T15:54:40.356264+0000","last_became_active":"2023-10-16T15:54:38.191365+0000","last_became_peered":"2023-10-16T15:54:38.191365+0000","last_unstale":"2023-10-16T15:54:40.356264+0000","last_undegraded":"2023-10-16T15:54:40.356264+0000","last_fullsized":"2023-10-16T15:54:40.356264+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:00.022703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354115+0000","last_change":"2023-10-16T15:54:39.201055+0000","last_active":"2023-10-16T15:54:40.354115+0000","last_peered":"2023-10-16T15:54:40.354115+0000","last_clean":"2023-10-16T15:54:40.354115+0000","last_became_active":"2023-10-16T15:54:39.199846+0000","last_became_peered":"2023-10-16T15:54:39.199846+0000","last_unstale":"2023-10-16T15:54:40.354115+0000","last_undegraded":"2023-10-16T15:54:40.354115+0000","last_fullsized":"2023-10-16T15:54:40.354115+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:28.581753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350984+0000","last_change":"2023-10-16T15:54:38.188455+0000","last_active":"2023-10-16T15:54:40.350984+0000","last_peered":"2023-10-16T15:54:40.350984+0000","last_clean":"2023-10-16T15:54:40.350984+0000","last_became_active":"2023-10-16T15:54:38.188167+0000","last_became_peered":"2023-10-16T15:54:38.188167+0000","last_unstale":"2023-10-16T15:54:40.350984+0000","last_undegraded":"2023-10-16T15:54:40.350984+0000","last_fullsized":"2023-10-16T15:54:40.350984+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:21.554285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178984+0000","last_change":"2023-10-16T15:54:39.198491+0000","last_active":"2023-10-16T15:54:40.178984+0000","last_peered":"2023-10-16T15:54:40.178984+0000","last_clean":"2023-10-16T15:54:40.178984+0000","last_became_active":"2023-10-16T15:54:39.194488+0000","last_became_peered":"2023-10-16T15:54:39.194488+0000","last_unstale":"2023-10-16T15:54:40.178984+0000","last_undegraded":"2023-10-16T15:54:40.178984+0000","last_fullsized":"2023-10-16T15:54:40.178984+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.663424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172779+0000","last_change":"2023-10-16T15:54:38.190757+0000","last_active":"2023-10-16T15:54:40.172779+0000","last_peered":"2023-10-16T15:54:40.172779+0000","last_clean":"2023-10-16T15:54:40.172779+0000","last_became_active":"2023-10-16T15:54:38.190491+0000","last_became_peered":"2023-10-16T15:54:38.190491+0000","last_unstale":"2023-10-16T15:54:40.172779+0000","last_undegraded":"2023-10-16T15:54:40.172779+0000","last_fullsized":"2023-10-16T15:54:40.172779+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:59.249150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352516+0000","last_change":"2023-10-16T15:54:39.206722+0000","last_active":"2023-10-16T15:54:40.352516+0000","last_peered":"2023-10-16T15:54:40.352516+0000","last_clean":"2023-10-16T15:54:40.352516+0000","last_became_active":"2023-10-16T15:54:39.204447+0000","last_became_peered":"2023-10-16T15:54:39.204447+0000","last_unstale":"2023-10-16T15:54:40.352516+0000","last_undegraded":"2023-10-16T15:54:40.352516+0000","last_fullsized":"2023-10-16T15:54:40.352516+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:50.316780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350787+0000","last_change":"2023-10-16T15:54:38.189769+0000","last_active":"2023-10-16T15:54:40.350787+0000","last_peered":"2023-10-16T15:54:40.350787+0000","last_clean":"2023-10-16T15:54:40.350787+0000","last_became_active":"2023-10-16T15:54:38.189418+0000","last_became_peered":"2023-10-16T15:54:38.189418+0000","last_unstale":"2023-10-16T15:54:40.350787+0000","last_undegraded":"2023-10-16T15:54:40.350787+0000","last_fullsized":"2023-10-16T15:54:40.350787+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.444572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172625+0000","last_change":"2023-10-16T15:54:39.205679+0000","last_active":"2023-10-16T15:54:40.172625+0000","last_peered":"2023-10-16T15:54:40.172625+0000","last_clean":"2023-10-16T15:54:40.172625+0000","last_became_active":"2023-10-16T15:54:39.204821+0000","last_became_peered":"2023-10-16T15:54:39.204821+0000","last_unstale":"2023-10-16T15:54:40.172625+0000","last_undegraded":"2023-10-16T15:54:40.172625+0000","last_fullsized":"2023-10-16T15:54:40.172625+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:44.694895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352789+0000","last_change":"2023-10-16T15:54:38.191853+0000","last_active":"2023-10-16T15:54:40.352789+0000","last_peered":"2023-10-16T15:54:40.352789+0000","last_clean":"2023-10-16T15:54:40.352789+0000","last_became_active":"2023-10-16T15:54:38.190725+0000","last_became_peered":"2023-10-16T15:54:38.190725+0000","last_unstale":"2023-10-16T15:54:40.352789+0000","last_undegraded":"2023-10-16T15:54:40.352789+0000","last_fullsized":"2023-10-16T15:54:40.352789+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:51.409385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173318+0000","last_change":"2023-10-16T15:54:39.205421+0000","last_active":"2023-10-16T15:54:40.173318+0000","last_peered":"2023-10-16T15:54:40.173318+0000","last_clean":"2023-10-16T15:54:40.173318+0000","last_became_active":"2023-10-16T15:54:39.204865+0000","last_became_peered":"2023-10-16T15:54:39.204865+0000","last_unstale":"2023-10-16T15:54:40.173318+0000","last_undegraded":"2023-10-16T15:54:40.173318+0000","last_fullsized":"2023-10-16T15:54:40.173318+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:33.811103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353190+0000","last_change":"2023-10-16T15:54:38.196240+0000","last_active":"2023-10-16T15:54:40.353190+0000","last_peered":"2023-10-16T15:54:40.353190+0000","last_clean":"2023-10-16T15:54:40.353190+0000","last_became_active":"2023-10-16T15:54:38.195833+0000","last_became_peered":"2023-10-16T15:54:38.195833+0000","last_unstale":"2023-10-16T15:54:40.353190+0000","last_undegraded":"2023-10-16T15:54:40.353190+0000","last_fullsized":"2023-10-16T15:54:40.353190+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354077+0000","last_change":"2023-10-16T15:54:39.204677+0000","last_active":"2023-10-16T15:54:40.354077+0000","last_peered":"2023-10-16T15:54:40.354077+0000","last_clean":"2023-10-16T15:54:40.354077+0000","last_became_active":"2023-10-16T15:54:39.202794+0000","last_became_peered":"2023-10-16T15:54:39.202794+0000","last_unstale":"2023-10-16T15:54:40.354077+0000","last_undegraded":"2023-10-16T15:54:40.354077+0000","last_fullsized":"2023-10-16T15:54:40.354077+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:11.420100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355954+0000","last_change":"2023-10-16T15:54:38.195322+0000","last_active":"2023-10-16T15:54:40.355954+0000","last_peered":"2023-10-16T15:54:40.355954+0000","last_clean":"2023-10-16T15:54:40.355954+0000","last_became_active":"2023-10-16T15:54:38.192163+0000","last_became_peered":"2023-10-16T15:54:38.192163+0000","last_unstale":"2023-10-16T15:54:40.355954+0000","last_undegraded":"2023-10-16T15:54:40.355954+0000","last_fullsized":"2023-10-16T15:54:40.355954+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:45.552506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356311+0000","last_change":"2023-10-16T15:54:39.212753+0000","last_active":"2023-10-16T15:54:40.356311+0000","last_peered":"2023-10-16T15:54:40.356311+0000","last_clean":"2023-10-16T15:54:40.356311+0000","last_became_active":"2023-10-16T15:54:39.212450+0000","last_became_peered":"2023-10-16T15:54:39.212450+0000","last_unstale":"2023-10-16T15:54:40.356311+0000","last_undegraded":"2023-10-16T15:54:40.356311+0000","last_fullsized":"2023-10-16T15:54:40.356311+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:04.593631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"245'3","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173038+0000","last_change":"2023-10-16T15:54:38.188991+0000","last_active":"2023-10-16T15:54:40.173038+0000","last_peered":"2023-10-16T15:54:40.173038+0000","last_clean":"2023-10-16T15:54:40.173038+0000","last_became_active":"2023-10-16T15:54:38.187521+0000","last_became_peered":"2023-10-16T15:54:38.187521+0000","last_unstale":"2023-10-16T15:54:40.173038+0000","last_undegraded":"2023-10-16T15:54:40.173038+0000","last_fullsized":"2023-10-16T15:54:40.173038+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:37.422309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352055+0000","last_change":"2023-10-16T15:54:39.188004+0000","last_active":"2023-10-16T15:54:40.352055+0000","last_peered":"2023-10-16T15:54:40.352055+0000","last_clean":"2023-10-16T15:54:40.352055+0000","last_became_active":"2023-10-16T15:54:39.187443+0000","last_became_peered":"2023-10-16T15:54:39.187443+0000","last_unstale":"2023-10-16T15:54:40.352055+0000","last_undegraded":"2023-10-16T15:54:40.352055+0000","last_fullsized":"2023-10-16T15:54:40.352055+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"245'6","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172866+0000","last_change":"2023-10-16T15:54:38.192821+0000","last_active":"2023-10-16T15:54:40.172866+0000","last_peered":"2023-10-16T15:54:40.172866+0000","last_clean":"2023-10-16T15:54:40.172866+0000","last_became_active":"2023-10-16T15:54:38.191495+0000","last_became_peered":"2023-10-16T15:54:38.191495+0000","last_unstale":"2023-10-16T15:54:40.172866+0000","last_undegraded":"2023-10-16T15:54:40.172866+0000","last_fullsized":"2023-10-16T15:54:40.172866+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:36.442402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350812+0000","last_change":"2023-10-16T15:54:39.203188+0000","last_active":"2023-10-16T15:54:40.350812+0000","last_peered":"2023-10-16T15:54:40.350812+0000","last_clean":"2023-10-16T15:54:40.350812+0000","last_became_active":"2023-10-16T15:54:39.202646+0000","last_became_peered":"2023-10-16T15:54:39.202646+0000","last_unstale":"2023-10-16T15:54:40.350812+0000","last_undegraded":"2023-10-16T15:54:40.350812+0000","last_fullsized":"2023-10-16T15:54:40.350812+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172824+0000","last_change":"2023-10-16T15:54:38.190094+0000","last_active":"2023-10-16T15:54:40.172824+0000","last_peered":"2023-10-16T15:54:40.172824+0000","last_clean":"2023-10-16T15:54:40.172824+0000","last_became_active":"2023-10-16T15:54:38.189825+0000","last_became_peered":"2023-10-16T15:54:38.189825+0000","last_unstale":"2023-10-16T15:54:40.172824+0000","last_undegraded":"2023-10-16T15:54:40.172824+0000","last_fullsized":"2023-10-16T15:54:40.172824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:17.021872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.179335+0000","last_change":"2023-10-16T15:54:39.205965+0000","last_active":"2023-10-16T15:54:40.179335+0000","last_peered":"2023-10-16T15:54:40.179335+0000","last_clean":"2023-10-16T15:54:40.179335+0000","last_became_active":"2023-10-16T15:54:39.205686+0000","last_became_peered":"2023-10-16T15:54:39.205686+0000","last_unstale":"2023-10-16T15:54:40.179335+0000","last_undegraded":"2023-10-16T15:54:40.179335+0000","last_fullsized":"2023-10-16T15:54:40.179335+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:36.119641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353169+0000","last_change":"2023-10-16T15:54:38.195373+0000","last_active":"2023-10-16T15:54:40.353169+0000","last_peered":"2023-10-16T15:54:40.353169+0000","last_clean":"2023-10-16T15:54:40.353169+0000","last_became_active":"2023-10-16T15:54:38.194574+0000","last_became_peered":"2023-10-16T15:54:38.194574+0000","last_unstale":"2023-10-16T15:54:40.353169+0000","last_undegraded":"2023-10-16T15:54:40.353169+0000","last_fullsized":"2023-10-16T15:54:40.353169+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:35.101378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351086+0000","last_change":"2023-10-16T15:54:39.207613+0000","last_active":"2023-10-16T15:54:40.351086+0000","last_peered":"2023-10-16T15:54:40.351086+0000","last_clean":"2023-10-16T15:54:40.351086+0000","last_became_active":"2023-10-16T15:54:39.207320+0000","last_became_peered":"2023-10-16T15:54:39.207320+0000","last_unstale":"2023-10-16T15:54:40.351086+0000","last_undegraded":"2023-10-16T15:54:40.351086+0000","last_fullsized":"2023-10-16T15:54:40.351086+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351014+0000","last_change":"2023-10-16T15:54:38.196425+0000","last_active":"2023-10-16T15:54:40.351014+0000","last_peered":"2023-10-16T15:54:40.351014+0000","last_clean":"2023-10-16T15:54:40.351014+0000","last_became_active":"2023-10-16T15:54:38.195269+0000","last_became_peered":"2023-10-16T15:54:38.195269+0000","last_unstale":"2023-10-16T15:54:40.351014+0000","last_undegraded":"2023-10-16T15:54:40.351014+0000","last_fullsized":"2023-10-16T15:54:40.351014+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:01.207615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350986+0000","last_change":"2023-10-16T15:54:39.205462+0000","last_active":"2023-10-16T15:54:40.350986+0000","last_peered":"2023-10-16T15:54:40.350986+0000","last_clean":"2023-10-16T15:54:40.350986+0000","last_became_active":"2023-10-16T15:54:39.204956+0000","last_became_peered":"2023-10-16T15:54:39.204956+0000","last_unstale":"2023-10-16T15:54:40.350986+0000","last_undegraded":"2023-10-16T15:54:40.350986+0000","last_fullsized":"2023-10-16T15:54:40.350986+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"245'8","reported_seq":20,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350416+0000","last_change":"2023-10-16T15:54:38.194134+0000","last_active":"2023-10-16T15:54:40.350416+0000","last_peered":"2023-10-16T15:54:40.350416+0000","last_clean":"2023-10-16T15:54:40.350416+0000","last_became_active":"2023-10-16T15:54:38.193858+0000","last_became_peered":"2023-10-16T15:54:38.193858+0000","last_unstale":"2023-10-16T15:54:40.350416+0000","last_undegraded":"2023-10-16T15:54:40.350416+0000","last_fullsized":"2023-10-16T15:54:40.350416+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:37.866131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352402+0000","last_change":"2023-10-16T15:54:39.203343+0000","last_active":"2023-10-16T15:54:40.352402+0000","last_peered":"2023-10-16T15:54:40.352402+0000","last_clean":"2023-10-16T15:54:40.352402+0000","last_became_active":"2023-10-16T15:54:39.203044+0000","last_became_peered":"2023-10-16T15:54:39.203044+0000","last_unstale":"2023-10-16T15:54:40.352402+0000","last_undegraded":"2023-10-16T15:54:40.352402+0000","last_fullsized":"2023-10-16T15:54:40.352402+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:23:49.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353379+0000","last_change":"2023-10-16T15:54:38.193502+0000","last_active":"2023-10-16T15:54:40.353379+0000","last_peered":"2023-10-16T15:54:40.353379+0000","last_clean":"2023-10-16T15:54:40.353379+0000","last_became_active":"2023-10-16T15:54:38.193186+0000","last_became_peered":"2023-10-16T15:54:38.193186+0000","last_unstale":"2023-10-16T15:54:40.353379+0000","last_undegraded":"2023-10-16T15:54:40.353379+0000","last_fullsized":"2023-10-16T15:54:40.353379+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356122+0000","last_change":"2023-10-16T15:54:39.206218+0000","last_active":"2023-10-16T15:54:40.356122+0000","last_peered":"2023-10-16T15:54:40.356122+0000","last_clean":"2023-10-16T15:54:40.356122+0000","last_became_active":"2023-10-16T15:54:39.205815+0000","last_became_peered":"2023-10-16T15:54:39.205815+0000","last_unstale":"2023-10-16T15:54:40.356122+0000","last_undegraded":"2023-10-16T15:54:40.356122+0000","last_fullsized":"2023-10-16T15:54:40.356122+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:24.642539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355904+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:40.355904+0000","last_peered":"2023-10-16T15:54:40.355904+0000","last_clean":"2023-10-16T15:54:40.355904+0000","last_became_active":"2023-10-16T15:54:38.195752+0000","last_became_peered":"2023-10-16T15:54:38.195752+0000","last_unstale":"2023-10-16T15:54:40.355904+0000","last_undegraded":"2023-10-16T15:54:40.355904+0000","last_fullsized":"2023-10-16T15:54:40.355904+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:15.145108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.179269+0000","last_change":"2023-10-16T15:54:39.198603+0000","last_active":"2023-10-16T15:54:40.179269+0000","last_peered":"2023-10-16T15:54:40.179269+0000","last_clean":"2023-10-16T15:54:40.179269+0000","last_became_active":"2023-10-16T15:54:39.195793+0000","last_became_peered":"2023-10-16T15:54:39.195793+0000","last_unstale":"2023-10-16T15:54:40.179269+0000","last_undegraded":"2023-10-16T15:54:40.179269+0000","last_fullsized":"2023-10-16T15:54:40.179269+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:16.753544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.177915+0000","last_change":"2023-10-16T15:54:38.193258+0000","last_active":"2023-10-16T15:54:40.177915+0000","last_peered":"2023-10-16T15:54:40.177915+0000","last_clean":"2023-10-16T15:54:40.177915+0000","last_became_active":"2023-10-16T15:54:38.192538+0000","last_became_peered":"2023-10-16T15:54:38.192538+0000","last_unstale":"2023-10-16T15:54:40.177915+0000","last_undegraded":"2023-10-16T15:54:40.177915+0000","last_fullsized":"2023-10-16T15:54:40.177915+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:07.463985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353601+0000","last_change":"2023-10-16T15:54:39.206968+0000","last_active":"2023-10-16T15:54:40.353601+0000","last_peered":"2023-10-16T15:54:40.353601+0000","last_clean":"2023-10-16T15:54:40.353601+0000","last_became_active":"2023-10-16T15:54:39.206271+0000","last_became_peered":"2023-10-16T15:54:39.206271+0000","last_unstale":"2023-10-16T15:54:40.353601+0000","last_undegraded":"2023-10-16T15:54:40.353601+0000","last_fullsized":"2023-10-16T15:54:40.353601+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353462+0000","last_change":"2023-10-16T15:54:38.194296+0000","last_active":"2023-10-16T15:54:40.353462+0000","last_peered":"2023-10-16T15:54:40.353462+0000","last_clean":"2023-10-16T15:54:40.353462+0000","last_became_active":"2023-10-16T15:54:38.194015+0000","last_became_peered":"2023-10-16T15:54:38.194015+0000","last_unstale":"2023-10-16T15:54:40.353462+0000","last_undegraded":"2023-10-16T15:54:40.353462+0000","last_fullsized":"2023-10-16T15:54:40.353462+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:58.920639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352345+0000","last_change":"2023-10-16T15:54:39.206512+0000","last_active":"2023-10-16T15:54:40.352345+0000","last_peered":"2023-10-16T15:54:40.352345+0000","last_clean":"2023-10-16T15:54:40.352345+0000","last_became_active":"2023-10-16T15:54:39.206174+0000","last_became_peered":"2023-10-16T15:54:39.206174+0000","last_unstale":"2023-10-16T15:54:40.352345+0000","last_undegraded":"2023-10-16T15:54:40.352345+0000","last_fullsized":"2023-10-16T15:54:40.352345+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355215+0000","last_change":"2023-10-16T15:54:38.199965+0000","last_active":"2023-10-16T15:54:40.355215+0000","last_peered":"2023-10-16T15:54:40.355215+0000","last_clean":"2023-10-16T15:54:40.355215+0000","last_became_active":"2023-10-16T15:54:38.199417+0000","last_became_peered":"2023-10-16T15:54:38.199417+0000","last_unstale":"2023-10-16T15:54:40.355215+0000","last_undegraded":"2023-10-16T15:54:40.355215+0000","last_fullsized":"2023-10-16T15:54:40.355215+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:55.688720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172846+0000","last_change":"2023-10-16T15:54:39.199452+0000","last_active":"2023-10-16T15:54:40.172846+0000","last_peered":"2023-10-16T15:54:40.172846+0000","last_clean":"2023-10-16T15:54:40.172846+0000","last_became_active":"2023-10-16T15:54:39.199132+0000","last_became_peered":"2023-10-16T15:54:39.199132+0000","last_unstale":"2023-10-16T15:54:40.172846+0000","last_undegraded":"2023-10-16T15:54:40.172846+0000","last_fullsized":"2023-10-16T15:54:40.172846+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:49.564533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354346+0000","last_change":"2023-10-16T15:54:38.193942+0000","last_active":"2023-10-16T15:54:40.354346+0000","last_peered":"2023-10-16T15:54:40.354346+0000","last_clean":"2023-10-16T15:54:40.354346+0000","last_became_active":"2023-10-16T15:54:38.193397+0000","last_became_peered":"2023-10-16T15:54:38.193397+0000","last_unstale":"2023-10-16T15:54:40.354346+0000","last_undegraded":"2023-10-16T15:54:40.354346+0000","last_fullsized":"2023-10-16T15:54:40.354346+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:40.933295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350317+0000","last_change":"2023-10-16T15:54:39.188638+0000","last_active":"2023-10-16T15:54:40.350317+0000","last_peered":"2023-10-16T15:54:40.350317+0000","last_clean":"2023-10-16T15:54:40.350317+0000","last_became_active":"2023-10-16T15:54:39.187890+0000","last_became_peered":"2023-10-16T15:54:39.187890+0000","last_unstale":"2023-10-16T15:54:40.350317+0000","last_undegraded":"2023-10-16T15:54:40.350317+0000","last_fullsized":"2023-10-16T15:54:40.350317+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:15:41.121120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178593+0000","last_change":"2023-10-16T15:54:38.196671+0000","last_active":"2023-10-16T15:54:40.178593+0000","last_peered":"2023-10-16T15:54:40.178593+0000","last_clean":"2023-10-16T15:54:40.178593+0000","last_became_active":"2023-10-16T15:54:38.196392+0000","last_became_peered":"2023-10-16T15:54:38.196392+0000","last_unstale":"2023-10-16T15:54:40.178593+0000","last_undegraded":"2023-10-16T15:54:40.178593+0000","last_fullsized":"2023-10-16T15:54:40.178593+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:35.189299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351934+0000","last_change":"2023-10-16T15:54:39.203931+0000","last_active":"2023-10-16T15:54:40.351934+0000","last_peered":"2023-10-16T15:54:40.351934+0000","last_clean":"2023-10-16T15:54:40.351934+0000","last_became_active":"2023-10-16T15:54:39.203614+0000","last_became_peered":"2023-10-16T15:54:39.203614+0000","last_unstale":"2023-10-16T15:54:40.351934+0000","last_undegraded":"2023-10-16T15:54:40.351934+0000","last_fullsized":"2023-10-16T15:54:40.351934+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350465+0000","last_change":"2023-10-16T15:54:38.187905+0000","last_active":"2023-10-16T15:54:40.350465+0000","last_peered":"2023-10-16T15:54:40.350465+0000","last_clean":"2023-10-16T15:54:40.350465+0000","last_became_active":"2023-10-16T15:54:38.187553+0000","last_became_peered":"2023-10-16T15:54:38.187553+0000","last_unstale":"2023-10-16T15:54:40.350465+0000","last_undegraded":"2023-10-16T15:54:40.350465+0000","last_fullsized":"2023-10-16T15:54:40.350465+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:27.563490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352704+0000","last_change":"2023-10-16T15:54:39.201684+0000","last_active":"2023-10-16T15:54:40.352704+0000","last_peered":"2023-10-16T15:54:40.352704+0000","last_clean":"2023-10-16T15:54:40.352704+0000","last_became_active":"2023-10-16T15:54:39.201363+0000","last_became_peered":"2023-10-16T15:54:39.201363+0000","last_unstale":"2023-10-16T15:54:40.352704+0000","last_undegraded":"2023-10-16T15:54:40.352704+0000","last_fullsized":"2023-10-16T15:54:40.352704+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352107+0000","last_change":"2023-10-16T15:54:38.193490+0000","last_active":"2023-10-16T15:54:40.352107+0000","last_peered":"2023-10-16T15:54:40.352107+0000","last_clean":"2023-10-16T15:54:40.352107+0000","last_became_active":"2023-10-16T15:54:38.193206+0000","last_became_peered":"2023-10-16T15:54:38.193206+0000","last_unstale":"2023-10-16T15:54:40.352107+0000","last_undegraded":"2023-10-16T15:54:40.352107+0000","last_fullsized":"2023-10-16T15:54:40.352107+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:39.542212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173233+0000","last_change":"2023-10-16T15:54:39.206833+0000","last_active":"2023-10-16T15:54:40.173233+0000","last_peered":"2023-10-16T15:54:40.173233+0000","last_clean":"2023-10-16T15:54:40.173233+0000","last_became_active":"2023-10-16T15:54:39.206491+0000","last_became_peered":"2023-10-16T15:54:39.206491+0000","last_unstale":"2023-10-16T15:54:40.173233+0000","last_undegraded":"2023-10-16T15:54:40.173233+0000","last_fullsized":"2023-10-16T15:54:40.173233+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:22.432507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178415+0000","last_change":"2023-10-16T15:54:38.198292+0000","last_active":"2023-10-16T15:54:40.178415+0000","last_peered":"2023-10-16T15:54:40.178415+0000","last_clean":"2023-10-16T15:54:40.178415+0000","last_became_active":"2023-10-16T15:54:38.197965+0000","last_became_peered":"2023-10-16T15:54:38.197965+0000","last_unstale":"2023-10-16T15:54:40.178415+0000","last_undegraded":"2023-10-16T15:54:40.178415+0000","last_fullsized":"2023-10-16T15:54:40.178415+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:07.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353233+0000","last_change":"2023-10-16T15:54:39.198360+0000","last_active":"2023-10-16T15:54:40.353233+0000","last_peered":"2023-10-16T15:54:40.353233+0000","last_clean":"2023-10-16T15:54:40.353233+0000","last_became_active":"2023-10-16T15:54:39.196440+0000","last_became_peered":"2023-10-16T15:54:39.196440+0000","last_unstale":"2023-10-16T15:54:40.353233+0000","last_undegraded":"2023-10-16T15:54:40.353233+0000","last_fullsized":"2023-10-16T15:54:40.353233+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:23.439544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178145+0000","last_change":"2023-10-16T15:54:38.199930+0000","last_active":"2023-10-16T15:54:40.178145+0000","last_peered":"2023-10-16T15:54:40.178145+0000","last_clean":"2023-10-16T15:54:40.178145+0000","last_became_active":"2023-10-16T15:54:38.199386+0000","last_became_peered":"2023-10-16T15:54:38.199386+0000","last_unstale":"2023-10-16T15:54:40.178145+0000","last_undegraded":"2023-10-16T15:54:40.178145+0000","last_fullsized":"2023-10-16T15:54:40.178145+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:57.311627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354487+0000","last_change":"2023-10-16T15:54:39.205703+0000","last_active":"2023-10-16T15:54:40.354487+0000","last_peered":"2023-10-16T15:54:40.354487+0000","last_clean":"2023-10-16T15:54:40.354487+0000","last_became_active":"2023-10-16T15:54:39.205415+0000","last_became_peered":"2023-10-16T15:54:39.205415+0000","last_unstale":"2023-10-16T15:54:40.354487+0000","last_undegraded":"2023-10-16T15:54:40.354487+0000","last_fullsized":"2023-10-16T15:54:40.354487+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:02.225878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353717+0000","last_change":"2023-10-16T15:54:38.184497+0000","last_active":"2023-10-16T15:54:40.353717+0000","last_peered":"2023-10-16T15:54:40.353717+0000","last_clean":"2023-10-16T15:54:40.353717+0000","last_became_active":"2023-10-16T15:54:38.184043+0000","last_became_peered":"2023-10-16T15:54:38.184043+0000","last_unstale":"2023-10-16T15:54:40.353717+0000","last_undegraded":"2023-10-16T15:54:40.353717+0000","last_fullsized":"2023-10-16T15:54:40.353717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:04.454449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351389+0000","last_change":"2023-10-16T15:54:39.192680+0000","last_active":"2023-10-16T15:54:40.351389+0000","last_peered":"2023-10-16T15:54:40.351389+0000","last_clean":"2023-10-16T15:54:40.351389+0000","last_became_active":"2023-10-16T15:54:39.192254+0000","last_became_peered":"2023-10-16T15:54:39.192254+0000","last_unstale":"2023-10-16T15:54:40.351389+0000","last_undegraded":"2023-10-16T15:54:40.351389+0000","last_fullsized":"2023-10-16T15:54:40.351389+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353194+0000","last_change":"2023-10-16T15:54:38.192812+0000","last_active":"2023-10-16T15:54:40.353194+0000","last_peered":"2023-10-16T15:54:40.353194+0000","last_clean":"2023-10-16T15:54:40.353194+0000","last_became_active":"2023-10-16T15:54:38.192528+0000","last_became_peered":"2023-10-16T15:54:38.192528+0000","last_unstale":"2023-10-16T15:54:40.353194+0000","last_undegraded":"2023-10-16T15:54:40.353194+0000","last_fullsized":"2023-10-16T15:54:40.353194+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:36.002573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351573+0000","last_change":"2023-10-16T15:54:39.207142+0000","last_active":"2023-10-16T15:54:40.351573+0000","last_peered":"2023-10-16T15:54:40.351573+0000","last_clean":"2023-10-16T15:54:40.351573+0000","last_became_active":"2023-10-16T15:54:39.206593+0000","last_became_peered":"2023-10-16T15:54:39.206593+0000","last_unstale":"2023-10-16T15:54:40.351573+0000","last_undegraded":"2023-10-16T15:54:40.351573+0000","last_fullsized":"2023-10-16T15:54:40.351573+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:58.545523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352233+0000","last_change":"2023-10-16T15:54:38.179331+0000","last_active":"2023-10-16T15:54:40.352233+0000","last_peered":"2023-10-16T15:54:40.352233+0000","last_clean":"2023-10-16T15:54:40.352233+0000","last_became_active":"2023-10-16T15:54:38.178684+0000","last_became_peered":"2023-10-16T15:54:38.178684+0000","last_unstale":"2023-10-16T15:54:40.352233+0000","last_undegraded":"2023-10-16T15:54:40.352233+0000","last_fullsized":"2023-10-16T15:54:40.352233+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351371+0000","last_change":"2023-10-16T15:54:39.203357+0000","last_active":"2023-10-16T15:54:40.351371+0000","last_peered":"2023-10-16T15:54:40.351371+0000","last_clean":"2023-10-16T15:54:40.351371+0000","last_became_active":"2023-10-16T15:54:39.203058+0000","last_became_peered":"2023-10-16T15:54:39.203058+0000","last_unstale":"2023-10-16T15:54:40.351371+0000","last_undegraded":"2023-10-16T15:54:40.351371+0000","last_fullsized":"2023-10-16T15:54:40.351371+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:37.020836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353460+0000","last_change":"2023-10-16T15:54:38.195221+0000","last_active":"2023-10-16T15:54:40.353460+0000","last_peered":"2023-10-16T15:54:40.353460+0000","last_clean":"2023-10-16T15:54:40.353460+0000","last_became_active":"2023-10-16T15:54:38.194937+0000","last_became_peered":"2023-10-16T15:54:38.194937+0000","last_unstale":"2023-10-16T15:54:40.353460+0000","last_undegraded":"2023-10-16T15:54:40.353460+0000","last_fullsized":"2023-10-16T15:54:40.353460+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:41.726203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172653+0000","last_change":"2023-10-16T15:54:39.207259+0000","last_active":"2023-10-16T15:54:40.172653+0000","last_peered":"2023-10-16T15:54:40.172653+0000","last_clean":"2023-10-16T15:54:40.172653+0000","last_became_active":"2023-10-16T15:54:39.206912+0000","last_became_peered":"2023-10-16T15:54:39.206912+0000","last_unstale":"2023-10-16T15:54:40.172653+0000","last_undegraded":"2023-10-16T15:54:40.172653+0000","last_fullsized":"2023-10-16T15:54:40.172653+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:53.691146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352887+0000","last_change":"2023-10-16T15:54:38.200919+0000","last_active":"2023-10-16T15:54:40.352887+0000","last_peered":"2023-10-16T15:54:40.352887+0000","last_clean":"2023-10-16T15:54:40.352887+0000","last_became_active":"2023-10-16T15:54:38.200141+0000","last_became_peered":"2023-10-16T15:54:38.200141+0000","last_unstale":"2023-10-16T15:54:40.352887+0000","last_undegraded":"2023-10-16T15:54:40.352887+0000","last_fullsized":"2023-10-16T15:54:40.352887+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:40.432210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178560+0000","last_change":"2023-10-16T15:54:39.208421+0000","last_active":"2023-10-16T15:54:40.178560+0000","last_peered":"2023-10-16T15:54:40.178560+0000","last_clean":"2023-10-16T15:54:40.178560+0000","last_became_active":"2023-10-16T15:54:39.207372+0000","last_became_peered":"2023-10-16T15:54:39.207372+0000","last_unstale":"2023-10-16T15:54:40.178560+0000","last_undegraded":"2023-10-16T15:54:40.178560+0000","last_fullsized":"2023-10-16T15:54:40.178560+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:11:16.655726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"245'4","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356391+0000","last_change":"2023-10-16T15:54:38.200929+0000","last_active":"2023-10-16T15:54:40.356391+0000","last_peered":"2023-10-16T15:54:40.356391+0000","last_clean":"2023-10-16T15:54:40.356391+0000","last_became_active":"2023-10-16T15:54:38.200195+0000","last_became_peered":"2023-10-16T15:54:38.200195+0000","last_unstale":"2023-10-16T15:54:40.356391+0000","last_undegraded":"2023-10-16T15:54:40.356391+0000","last_fullsized":"2023-10-16T15:54:40.356391+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:20.533246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"231'373","reported_seq":1346,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T15:54:39.175655+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:54:39.175655+0000","last_peered":"2023-10-16T15:54:39.175655+0000","last_clean":"2023-10-16T15:54:39.175655+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:54:39.175655+0000","last_undegraded":"2023-10-16T15:54:39.175655+0000","last_fullsized":"2023-10-16T15:54:39.175655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576016,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65312,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306528,"statfs":{"total":96636764160,"available":96569884672,"internally_reserved":0,"allocated":3682304,"data_stored":2110235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.131,"15min":1.181},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.554,"5min":1.794,"15min":1.8879999999999999},"last":1.444},{"interface":"front","average":{"1min":1.056,"5min":1.1180000000000001,"15min":1.1519999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.149}]},{"osd":1,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.1140000000000001,"15min":1.1719999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6519999999999999,"5min":1.76,"15min":1.889},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.29,"15min":1.355},"min":{"1min":0.71399999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6919999999999999,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":1.2050000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3460000000000001,"15min":1.339},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.661,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.276,"15min":1.3080000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5900000000000001,"5min":1.909,"15min":1.95},"last":1.2509999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.3160000000000001,"15min":1.349},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.746,"5min":1.843,"15min":1.869},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.405,"15min":1.4430000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.7749999999999999,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.3680000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3879999999999999,"15min":1.383},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.9370000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3180000000000001,"15min":1.341},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.655,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.3220000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4690000000000001,"15min":1.464},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.929,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.506,"15min":1.5489999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.839,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.619}]},{"osd":6,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.454,"15min":1.472},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5640000000000001,"15min":1.5780000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.988,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.5549999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48092,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323748,"statfs":{"total":96636764160,"available":96587517952,"internally_reserved":0,"allocated":3092480,"data_stored":1520401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.25,"15min":1.214},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.258,"15min":1.2609999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70999999999999996},"max":{"1min":1.5669999999999999,"5min":1.905,"15min":1.905},"last":0.98499999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3140000000000001,"15min":1.296},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.72199999999999998},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.516},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.3049999999999999,"15min":1.3200000000000001},"min":{"1min":0.747,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.859,"15min":1.8740000000000001},"last":1.22}]},{"osd":2,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2330000000000001,"15min":1.238},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.831,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.86299999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.905,"5min":1.905,"15min":2.0339999999999998},"last":1.4550000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3680000000000001,"15min":1.3919999999999999},"min":{"1min":0.93000000000000005,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.107},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3149999999999999,"15min":1.3340000000000001},"min":{"1min":0.91000000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.0489999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.409,"15min":1.337},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3359999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.617},{"interface":"front","average":{"1min":1.518,"5min":1.454,"15min":1.4059999999999999},"min":{"1min":1.2,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.335}]},{"osd":5,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.369,"15min":1.3480000000000001},"min":{"1min":1.022,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":2.016},"last":1.163},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6020000000000001,"15min":1.534},"min":{"1min":1.194,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.681}]},{"osd":7,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.635,"15min":1.599},"min":{"1min":1.274,"5min":0.627,"15min":0.627},"max":{"1min":2.0110000000000001,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.667,"5min":1.6319999999999999,"15min":1.595},"min":{"1min":1.2849999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.028,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.3740000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30052,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341788,"statfs":{"total":96636764160,"available":96605990912,"internally_reserved":0,"allocated":3096576,"data_stored":1526125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1240000000000001,"15min":1.163},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6850000000000001,"5min":1.8069999999999999,"15min":1.821},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.2689999999999999,"15min":1.2430000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.53,"5min":2.738,"15min":2.738},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7669999999999999,"5min":1.7849999999999999,"15min":1.8680000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2629999999999999,"15min":1.302},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7270000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.3100000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.238,"15min":1.2529999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.577,"5min":1.784,"15min":1.9219999999999999},"last":1.754},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.399,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7190000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.5}]},{"osd":3,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.343,"15min":1.3400000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9039999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.641},{"interface":"front","average":{"1min":1.169,"5min":1.3400000000000001,"15min":1.3799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.7130000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.966}]},{"osd":4,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.367,"15min":1.347},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.5069999999999999,"15min":1.5189999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.804,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.1579999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.385},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8660000000000001,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.599},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":1.431},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0230000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.365}]},{"osd":7,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.538,"15min":1.5369999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.847,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.873},{"interface":"front","average":{"1min":1.516,"5min":1.639,"15min":1.633},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.962,"5min":3.109,"15min":3.109},"last":1.7}]}]},{"osd":4,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52268,"kb_used_data":3032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319572,"statfs":{"total":96636764160,"available":96583241728,"internally_reserved":0,"allocated":3104768,"data_stored":1521501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1919999999999999,"15min":1.202},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.52700000000000002},"max":{"1min":1.861,"5min":1.883,"15min":1.883},"last":1.319},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1890000000000001,"15min":1.198},"min":{"1min":0.78800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.875,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.0349999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.2370000000000001},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.629},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.88100000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.952,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.405}]},{"osd":2,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.413,"15min":1.4570000000000001},"min":{"1min":0.99399999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.6240000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.462},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4910000000000001,"15min":1.4990000000000001},"min":{"1min":1.0900000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9099999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.575}]},{"osd":3,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.425,"15min":1.391},"min":{"1min":0.84699999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.1099999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.391,"15min":1.3919999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.6220000000000001,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.532}]},{"osd":5,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.377,"15min":1.347},"min":{"1min":1.046,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.8899999999999999,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":1.177,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.004,"5min":2.161,"15min":2.4489999999999998},"last":1.4319999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3999999999999999,"15min":1.4039999999999999},"min":{"1min":0.94099999999999995,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":1.8300000000000001,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.256},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5529999999999999,"15min":1.528},"min":{"1min":1.198,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.0049999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.2110000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.625,"15min":1.5880000000000001},"min":{"1min":1.139,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.2200000000000002,"5min":2.399,"15min":2.399},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.6180000000000001,"15min":1.6040000000000001},"min":{"1min":1.1299999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0449999999999999,"5min":2.2599999999999998,"15min":2.347},"last":1.373}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56412,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315428,"statfs":{"total":96636764160,"available":96578998272,"internally_reserved":0,"allocated":3088384,"data_stored":1520379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.94199999999999995,"15min":0.97099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.268,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0389999999999999,"15min":1.0660000000000001},"min":{"1min":0.44,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6850000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":1.0049999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.97099999999999997,"15min":1.0269999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.504,"5min":1.6679999999999999,"15min":2.589},"last":1.105},{"interface":"front","average":{"1min":1.071,"5min":1.1719999999999999,"15min":1.218},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.7210000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.3580000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.3759999999999999,"15min":1.4239999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.829,"5min":1.885,"15min":2.302},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.4430000000000001,"15min":1.508},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.768,"5min":1.9530000000000001,"15min":2.754},"last":0.90600000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.3009999999999999,"15min":1.3480000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7150000000000001,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.3660000000000001,"15min":1.421},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.276}]},{"osd":5,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.542,"15min":1.5669999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0590000000000002,"5min":2.2909999999999999,"15min":2.5009999999999999},"last":1.413},{"interface":"front","average":{"1min":1.361,"5min":1.524,"15min":1.573},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.0030000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.4890000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.4930000000000001,"15min":1.504},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.931,"5min":2.173,"15min":2.2090000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.5920000000000001,"15min":1.6060000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.1539999999999999,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.573}]},{"osd":7,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.6419999999999999,"15min":1.6879999999999999},"min":{"1min":0.97499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0899999999999999,"5min":2.214,"15min":2.8780000000000001},"last":1.244},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.617,"15min":1.6379999999999999},"min":{"1min":0.94399999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.1120000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.4470000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576016,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56416,"kb_used_data":3028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315424,"statfs":{"total":96636764160,"available":96578994176,"internally_reserved":0,"allocated":3100672,"data_stored":1526215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":1.252},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.3089999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.3540000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.5800000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.385}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.788},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.7250000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3117056,"data_stored":1527861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94399999999999995,"15min":0.94899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.452,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":0.75},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0429999999999999,"15min":1.1299999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6619999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.4279999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98699999999999999,"15min":1.0349999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.4179999999999999,"5min":1.6299999999999999,"15min":2.286},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.109,"5min":1.1919999999999999,"15min":1.2270000000000001},"min":{"1min":0.59199999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.657,"5min":1.7929999999999999,"15min":1.853},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3220000000000001,"15min":1.3680000000000001},"min":{"1min":0.69699999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7290000000000001,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.2529999999999999,"15min":1.3129999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8859999999999999,"5min":2.1629999999999998,"15min":2.7839999999999998},"last":0.83299999999999996}]},{"osd":4,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3620000000000001,"15min":1.381},"min":{"1min":0.79300000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.2959999999999998,"5min":2.327,"15min":2.3809999999999998},"last":1.246},{"interface":"front","average":{"1min":1.409,"5min":1.512,"15min":1.524},"min":{"1min":0.92000000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.081,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.105}]},{"osd":5,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.49,"15min":1.5269999999999999},"min":{"1min":0.72499999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0539999999999998,"5min":2.456,"15min":2.456},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4930000000000001,"15min":1.502},"min":{"1min":1.0069999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.093,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.1859999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.647,"15min":1.655},"min":{"1min":0.97299999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.218,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.367},{"interface":"front","average":{"1min":1.542,"5min":1.6160000000000001,"15min":1.643},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.5059999999999998,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.7030000000000001,"15min":1.744},"min":{"1min":0.80300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.621,"5min":2.734,"15min":3.2090000000000001},"last":1.302},{"interface":"front","average":{"1min":1.577,"5min":1.647,"15min":1.655},"min":{"1min":0.90100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.452,"5min":2.452,"15min":3.1539999999999999},"last":1.028}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48668,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323172,"statfs":{"total":96636764160,"available":96586928128,"internally_reserved":0,"allocated":3682304,"data_stored":2110199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3129999999999999,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.1879999999999999,"15min":1.2270000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.5589999999999999,"5min":2.036,"15min":2.036},"last":1.127}]},{"osd":2,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.226,"15min":1.274},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.486,"5min":1.913,"15min":2.0259999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.046,"5min":1.169,"15min":1.226},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.6850000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.628}]},{"osd":3,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.006,"15min":1.0329999999999999},"min":{"1min":0.54700000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.4370000000000001,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.113,"5min":1.244,"15min":1.27},"min":{"1min":0.57599999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.677,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":0.99399999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.3200000000000001,"15min":1.341},"min":{"1min":0.749,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.575,"5min":1.8520000000000001,"15min":2.137},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4450000000000001,"15min":1.51},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.901,"5min":1.99,"15min":2.452},"last":1.24}]},{"osd":5,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.4690000000000001,"15min":1.4890000000000001},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.1059999999999999,"5min":2.137,"15min":2.137},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.365,"15min":1.419},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.6879999999999999,"5min":2.0859999999999999,"15min":2.282},"last":0.88500000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.5980000000000001,"15min":1.6399999999999999},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.84499999999999997},"max":{"1min":1.974,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5640000000000001,"15min":1.5780000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.085,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.6719999999999999,"15min":1.6910000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2200000000000002,"5min":2.339,"15min":2.339},"last":1.085},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6100000000000001,"15min":1.611},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.8540000000000001,"5min":2.1739999999999999,"15min":2.274},"last":0.96099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:54:43.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:54:44.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.035+0000 7f4455e13640 1 Processor -- start 2023-10-16T15:54:44.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.036+0000 7f4455e13640 1 -- start start 2023-10-16T15:54:44.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.037+0000 7f4455e13640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f445006c040 0x7f445006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.037+0000 7f4455e13640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44500ad970 0x7f445015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.037+0000 7f4455e13640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f445015ebc0 0x7f445015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.037+0000 7f4455e13640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445006c960 con 0x7f445015ebc0 2023-10-16T15:54:44.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.037+0000 7f4455e13640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445015b9d0 con 0x7f445006c040 2023-10-16T15:54:44.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.037+0000 7f4455e13640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445015bcb0 con 0x7f44500ad970 2023-10-16T15:54:44.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f445015ebc0 0x7f445015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f445015ebc0 0x7f445015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47554/0 (socket says 172.21.15.29:47554) 2023-10-16T15:54:44.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 -- 172.21.15.29:0/3327953532 learned_addr learned my addr 172.21.15.29:0/3327953532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:44.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4454e11640 1 --2- 172.21.15.29:0/3327953532 >> v2:172.21.15.170:3300/0 conn(0x7f445006c040 0x7f445006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f444ffff640 1 --2- 172.21.15.29:0/3327953532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44500ad970 0x7f445015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 -- 172.21.15.29:0/3327953532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44500ad970 msgr2=0x7f445015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:44.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 --2- 172.21.15.29:0/3327953532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44500ad970 0x7f445015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 -- 172.21.15.29:0/3327953532 >> v2:172.21.15.170:3300/0 conn(0x7f445006c040 msgr2=0x7f445006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:44.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 --2- 172.21.15.29:0/3327953532 >> v2:172.21.15.170:3300/0 conn(0x7f445006c040 0x7f445006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.038+0000 7f4455612640 1 -- 172.21.15.29:0/3327953532 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445015bf90 con 0x7f445015ebc0 2023-10-16T15:54:44.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.039+0000 7f4455612640 1 --2- 172.21.15.29:0/3327953532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445015ebc0 0x7f445015f050 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f4448009e50 tx=0x7f444802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1eda8c3ac122a6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:44.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.040+0000 7f444f7fe640 1 -- 172.21.15.29:0/3327953532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444802ca70 con 0x7f445015ebc0 2023-10-16T15:54:44.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.040+0000 7f444f7fe640 1 -- 172.21.15.29:0/3327953532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444802cbd0 con 0x7f445015ebc0 2023-10-16T15:54:44.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.040+0000 7f444f7fe640 1 -- 172.21.15.29:0/3327953532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44480343f0 con 0x7f445015ebc0 2023-10-16T15:54:44.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.040+0000 7f4455e13640 1 -- 172.21.15.29:0/3327953532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445015ebc0 msgr2=0x7f445015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.040+0000 7f4455e13640 1 --2- 172.21.15.29:0/3327953532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445015ebc0 0x7f445015f050 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f4448009e50 tx=0x7f444802bed0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 -- 172.21.15.29:0/3327953532 shutdown_connections 2023-10-16T15:54:44.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 --2- 172.21.15.29:0/3327953532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445015ebc0 0x7f445015f050 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 --2- 172.21.15.29:0/3327953532 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44500ad970 0x7f445015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 --2- 172.21.15.29:0/3327953532 >> v2:172.21.15.170:3300/0 conn(0x7f445006c040 0x7f445006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 -- 172.21.15.29:0/3327953532 >> 172.21.15.29:0/3327953532 conn(0x7f44500ce6a0 msgr2=0x7f44500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:44.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 -- 172.21.15.29:0/3327953532 shutdown_connections 2023-10-16T15:54:44.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.041+0000 7f4455e13640 1 -- 172.21.15.29:0/3327953532 wait complete. 2023-10-16T15:54:44.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.042+0000 7f4455e13640 1 Processor -- start 2023-10-16T15:54:44.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.043+0000 7f4455e13640 1 -- start start 2023-10-16T15:54:44.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.043+0000 7f4455e13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 0x7f44500cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.043+0000 7f4455e13640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500ad970 0x7f44500b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4455e13640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f445015ebc0 0x7f44500b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4455e13640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44500c5b80 con 0x7f445006c040 2023-10-16T15:54:44.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4455e13640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4450142040 con 0x7f44500ad970 2023-10-16T15:54:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4455e13640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4450142320 con 0x7f445015ebc0 2023-10-16T15:54:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 0x7f44500cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 0x7f44500cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47570/0 (socket says 172.21.15.29:47570) 2023-10-16T15:54:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 -- 172.21.15.29:0/1892812248 learned_addr learned my addr 172.21.15.29:0/1892812248 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4455612640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f445015ebc0 0x7f44500b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f444ffff640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500ad970 0x7f44500b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 -- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f445015ebc0 msgr2=0x7f44500b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f445015ebc0 0x7f44500b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 -- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500ad970 msgr2=0x7f44500b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500ad970 0x7f44500b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f4454e11640 1 -- 172.21.15.29:0/1892812248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448009b30 con 0x7f445006c040 2023-10-16T15:54:44.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.044+0000 7f444ffff640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500ad970 0x7f44500b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:44.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.045+0000 7f4454e11640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 0x7f44500cd230 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f444400d800 tx=0x7f444400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:44.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.045+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44440096a0 con 0x7f445006c040 2023-10-16T15:54:44.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.045+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4444009800 con 0x7f445006c040 2023-10-16T15:54:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.045+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4450142600 con 0x7f445006c040 2023-10-16T15:54:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.045+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444009990 con 0x7f445006c040 2023-10-16T15:54:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.045+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44500cd770 con 0x7f445006c040 2023-10-16T15:54:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.046+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445006b330 con 0x7f445006c040 2023-10-16T15:54:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.047+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4444009af0 con 0x7f445006c040 2023-10-16T15:54:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.048+0000 7f444d7fa640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f443406b0b0 0x7f443406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.048+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4444089eb0 con 0x7f445006c040 2023-10-16T15:54:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.049+0000 7f444ffff640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f443406b0b0 0x7f443406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.050+0000 7f444ffff640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f443406b0b0 0x7f443406d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f443c006fd0 tx=0x7f443c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.053+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4444054c10 con 0x7f445006c040 2023-10-16T15:54:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.244+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 --> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44500cc550 con 0x7f443406b0b0 2023-10-16T15:54:44.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.266+0000 7f444d7fa640 1 -- 172.21.15.29:0/1892812248 <== mgr.9022 v2:172.21.15.170:6832/58791103 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287134 (secure 0 0 0) 0x7f443c009530 con 0x7f443406b0b0 2023-10-16T15:54:44.271 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:44.273 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:54:44.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.270+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f443406b0b0 msgr2=0x7f443406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.270+0000 7f4455e13640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f443406b0b0 0x7f443406d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f443c006fd0 tx=0x7f443c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.270+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 msgr2=0x7f44500cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.270+0000 7f4455e13640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 0x7f44500cd230 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f444400d800 tx=0x7f444400dcc0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 shutdown_connections 2023-10-16T15:54:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f443406b0b0 0x7f443406d570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f445015ebc0 0x7f44500b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44500ad970 0x7f44500b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 --2- 172.21.15.29:0/1892812248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445006c040 0x7f44500cd230 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 >> 172.21.15.29:0/1892812248 conn(0x7f44500ce6a0 msgr2=0x7f445015a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 shutdown_connections 2023-10-16T15:54:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.271+0000 7f4455e13640 1 -- 172.21.15.29:0/1892812248 wait complete. 2023-10-16T15:54:44.290 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T15:54:43.478207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":667,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":405336,"kb_used_data":25356,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754569384,"statfs":{"total":773094113280,"available":772679049216,"internally_reserved":0,"allocated":25964544,"data_stored":13362916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":388943417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002070"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173234+0000","last_change":"2023-10-16T15:54:39.213468+0000","last_active":"2023-10-16T15:54:40.173234+0000","last_peered":"2023-10-16T15:54:40.173234+0000","last_clean":"2023-10-16T15:54:40.173234+0000","last_became_active":"2023-10-16T15:54:39.213201+0000","last_became_peered":"2023-10-16T15:54:39.213201+0000","last_unstale":"2023-10-16T15:54:40.173234+0000","last_undegraded":"2023-10-16T15:54:40.173234+0000","last_fullsized":"2023-10-16T15:54:40.173234+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:54.453092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173154+0000","last_change":"2023-10-16T15:54:39.209387+0000","last_active":"2023-10-16T15:54:40.173154+0000","last_peered":"2023-10-16T15:54:40.173154+0000","last_clean":"2023-10-16T15:54:40.173154+0000","last_became_active":"2023-10-16T15:54:39.207929+0000","last_became_peered":"2023-10-16T15:54:39.207929+0000","last_unstale":"2023-10-16T15:54:40.173154+0000","last_undegraded":"2023-10-16T15:54:40.173154+0000","last_fullsized":"2023-10-16T15:54:40.173154+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:21.226148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355505+0000","last_change":"2023-10-16T15:54:39.205624+0000","last_active":"2023-10-16T15:54:40.355505+0000","last_peered":"2023-10-16T15:54:40.355505+0000","last_clean":"2023-10-16T15:54:40.355505+0000","last_became_active":"2023-10-16T15:54:39.205294+0000","last_became_peered":"2023-10-16T15:54:39.205294+0000","last_unstale":"2023-10-16T15:54:40.355505+0000","last_undegraded":"2023-10-16T15:54:40.355505+0000","last_fullsized":"2023-10-16T15:54:40.355505+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.276795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351329+0000","last_change":"2023-10-16T15:54:39.203634+0000","last_active":"2023-10-16T15:54:40.351329+0000","last_peered":"2023-10-16T15:54:40.351329+0000","last_clean":"2023-10-16T15:54:40.351329+0000","last_became_active":"2023-10-16T15:54:39.203333+0000","last_became_peered":"2023-10-16T15:54:39.203333+0000","last_unstale":"2023-10-16T15:54:40.351329+0000","last_undegraded":"2023-10-16T15:54:40.351329+0000","last_fullsized":"2023-10-16T15:54:40.351329+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.177094+0000","last_change":"2023-10-16T15:54:39.205871+0000","last_active":"2023-10-16T15:54:40.177094+0000","last_peered":"2023-10-16T15:54:40.177094+0000","last_clean":"2023-10-16T15:54:40.177094+0000","last_became_active":"2023-10-16T15:54:39.204811+0000","last_became_peered":"2023-10-16T15:54:39.204811+0000","last_unstale":"2023-10-16T15:54:40.177094+0000","last_undegraded":"2023-10-16T15:54:40.177094+0000","last_fullsized":"2023-10-16T15:54:40.177094+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:59.938902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352180+0000","last_change":"2023-10-16T15:54:39.198994+0000","last_active":"2023-10-16T15:54:40.352180+0000","last_peered":"2023-10-16T15:54:40.352180+0000","last_clean":"2023-10-16T15:54:40.352180+0000","last_became_active":"2023-10-16T15:54:39.198650+0000","last_became_peered":"2023-10-16T15:54:39.198650+0000","last_unstale":"2023-10-16T15:54:40.352180+0000","last_undegraded":"2023-10-16T15:54:40.352180+0000","last_fullsized":"2023-10-16T15:54:40.352180+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350554+0000","last_change":"2023-10-16T15:54:39.190398+0000","last_active":"2023-10-16T15:54:40.350554+0000","last_peered":"2023-10-16T15:54:40.350554+0000","last_clean":"2023-10-16T15:54:40.350554+0000","last_became_active":"2023-10-16T15:54:39.189834+0000","last_became_peered":"2023-10-16T15:54:39.189834+0000","last_unstale":"2023-10-16T15:54:40.350554+0000","last_undegraded":"2023-10-16T15:54:40.350554+0000","last_fullsized":"2023-10-16T15:54:40.350554+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351163+0000","last_change":"2023-10-16T15:54:39.203378+0000","last_active":"2023-10-16T15:54:40.351163+0000","last_peered":"2023-10-16T15:54:40.351163+0000","last_clean":"2023-10-16T15:54:40.351163+0000","last_became_active":"2023-10-16T15:54:39.202647+0000","last_became_peered":"2023-10-16T15:54:39.202647+0000","last_unstale":"2023-10-16T15:54:40.351163+0000","last_undegraded":"2023-10-16T15:54:40.351163+0000","last_fullsized":"2023-10-16T15:54:40.351163+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:49:25.249610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.177972+0000","last_change":"2023-10-16T15:54:39.190692+0000","last_active":"2023-10-16T15:54:40.177972+0000","last_peered":"2023-10-16T15:54:40.177972+0000","last_clean":"2023-10-16T15:54:40.177972+0000","last_became_active":"2023-10-16T15:54:39.189950+0000","last_became_peered":"2023-10-16T15:54:39.189950+0000","last_unstale":"2023-10-16T15:54:40.177972+0000","last_undegraded":"2023-10-16T15:54:40.177972+0000","last_fullsized":"2023-10-16T15:54:40.177972+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:05.334097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352055+0000","last_change":"2023-10-16T15:54:39.203147+0000","last_active":"2023-10-16T15:54:40.352055+0000","last_peered":"2023-10-16T15:54:40.352055+0000","last_clean":"2023-10-16T15:54:40.352055+0000","last_became_active":"2023-10-16T15:54:39.202529+0000","last_became_peered":"2023-10-16T15:54:39.202529+0000","last_unstale":"2023-10-16T15:54:40.352055+0000","last_undegraded":"2023-10-16T15:54:40.352055+0000","last_fullsized":"2023-10-16T15:54:40.352055+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354384+0000","last_change":"2023-10-16T15:54:39.200394+0000","last_active":"2023-10-16T15:54:40.354384+0000","last_peered":"2023-10-16T15:54:40.354384+0000","last_clean":"2023-10-16T15:54:40.354384+0000","last_became_active":"2023-10-16T15:54:39.199822+0000","last_became_peered":"2023-10-16T15:54:39.199822+0000","last_unstale":"2023-10-16T15:54:40.354384+0000","last_undegraded":"2023-10-16T15:54:40.354384+0000","last_fullsized":"2023-10-16T15:54:40.354384+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:38.462835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352956+0000","last_change":"2023-10-16T15:54:39.210009+0000","last_active":"2023-10-16T15:54:40.352956+0000","last_peered":"2023-10-16T15:54:40.352956+0000","last_clean":"2023-10-16T15:54:40.352956+0000","last_became_active":"2023-10-16T15:54:39.209277+0000","last_became_peered":"2023-10-16T15:54:39.209277+0000","last_unstale":"2023-10-16T15:54:40.352956+0000","last_undegraded":"2023-10-16T15:54:40.352956+0000","last_fullsized":"2023-10-16T15:54:40.352956+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352987+0000","last_change":"2023-10-16T15:54:39.194527+0000","last_active":"2023-10-16T15:54:40.352987+0000","last_peered":"2023-10-16T15:54:40.352987+0000","last_clean":"2023-10-16T15:54:40.352987+0000","last_became_active":"2023-10-16T15:54:39.194123+0000","last_became_peered":"2023-10-16T15:54:39.194123+0000","last_unstale":"2023-10-16T15:54:40.352987+0000","last_undegraded":"2023-10-16T15:54:40.352987+0000","last_fullsized":"2023-10-16T15:54:40.352987+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172658+0000","last_change":"2023-10-16T15:54:39.208938+0000","last_active":"2023-10-16T15:54:40.172658+0000","last_peered":"2023-10-16T15:54:40.172658+0000","last_clean":"2023-10-16T15:54:40.172658+0000","last_became_active":"2023-10-16T15:54:39.208579+0000","last_became_peered":"2023-10-16T15:54:39.208579+0000","last_unstale":"2023-10-16T15:54:40.172658+0000","last_undegraded":"2023-10-16T15:54:40.172658+0000","last_fullsized":"2023-10-16T15:54:40.172658+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:32.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178736+0000","last_change":"2023-10-16T15:54:39.187772+0000","last_active":"2023-10-16T15:54:40.178736+0000","last_peered":"2023-10-16T15:54:40.178736+0000","last_clean":"2023-10-16T15:54:40.178736+0000","last_became_active":"2023-10-16T15:54:39.187210+0000","last_became_peered":"2023-10-16T15:54:39.187210+0000","last_unstale":"2023-10-16T15:54:40.178736+0000","last_undegraded":"2023-10-16T15:54:40.178736+0000","last_fullsized":"2023-10-16T15:54:40.178736+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:31.089495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352161+0000","last_change":"2023-10-16T15:54:39.199187+0000","last_active":"2023-10-16T15:54:40.352161+0000","last_peered":"2023-10-16T15:54:40.352161+0000","last_clean":"2023-10-16T15:54:40.352161+0000","last_became_active":"2023-10-16T15:54:39.195076+0000","last_became_peered":"2023-10-16T15:54:39.195076+0000","last_unstale":"2023-10-16T15:54:40.352161+0000","last_undegraded":"2023-10-16T15:54:40.352161+0000","last_fullsized":"2023-10-16T15:54:40.352161+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:01.081391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352979+0000","last_change":"2023-10-16T15:54:39.212389+0000","last_active":"2023-10-16T15:54:40.352979+0000","last_peered":"2023-10-16T15:54:40.352979+0000","last_clean":"2023-10-16T15:54:40.352979+0000","last_became_active":"2023-10-16T15:54:39.212017+0000","last_became_peered":"2023-10-16T15:54:39.212017+0000","last_unstale":"2023-10-16T15:54:40.352979+0000","last_undegraded":"2023-10-16T15:54:40.352979+0000","last_fullsized":"2023-10-16T15:54:40.352979+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173250+0000","last_change":"2023-10-16T15:54:39.214749+0000","last_active":"2023-10-16T15:54:40.173250+0000","last_peered":"2023-10-16T15:54:40.173250+0000","last_clean":"2023-10-16T15:54:40.173250+0000","last_became_active":"2023-10-16T15:54:39.214456+0000","last_became_peered":"2023-10-16T15:54:39.214456+0000","last_unstale":"2023-10-16T15:54:40.173250+0000","last_undegraded":"2023-10-16T15:54:40.173250+0000","last_fullsized":"2023-10-16T15:54:40.173250+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:36.851092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352958+0000","last_change":"2023-10-16T15:54:39.202583+0000","last_active":"2023-10-16T15:54:40.352958+0000","last_peered":"2023-10-16T15:54:40.352958+0000","last_clean":"2023-10-16T15:54:40.352958+0000","last_became_active":"2023-10-16T15:54:39.202278+0000","last_became_peered":"2023-10-16T15:54:39.202278+0000","last_unstale":"2023-10-16T15:54:40.352958+0000","last_undegraded":"2023-10-16T15:54:40.352958+0000","last_fullsized":"2023-10-16T15:54:40.352958+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355046+0000","last_change":"2023-10-16T15:54:39.205053+0000","last_active":"2023-10-16T15:54:40.355046+0000","last_peered":"2023-10-16T15:54:40.355046+0000","last_clean":"2023-10-16T15:54:40.355046+0000","last_became_active":"2023-10-16T15:54:39.202502+0000","last_became_peered":"2023-10-16T15:54:39.202502+0000","last_unstale":"2023-10-16T15:54:40.355046+0000","last_undegraded":"2023-10-16T15:54:40.355046+0000","last_fullsized":"2023-10-16T15:54:40.355046+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:38.045649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351189+0000","last_change":"2023-10-16T15:54:39.202366+0000","last_active":"2023-10-16T15:54:40.351189+0000","last_peered":"2023-10-16T15:54:40.351189+0000","last_clean":"2023-10-16T15:54:40.351189+0000","last_became_active":"2023-10-16T15:54:39.201981+0000","last_became_peered":"2023-10-16T15:54:39.201981+0000","last_unstale":"2023-10-16T15:54:40.351189+0000","last_undegraded":"2023-10-16T15:54:40.351189+0000","last_fullsized":"2023-10-16T15:54:40.351189+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:00.755938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350758+0000","last_change":"2023-10-16T15:54:39.209910+0000","last_active":"2023-10-16T15:54:40.350758+0000","last_peered":"2023-10-16T15:54:40.350758+0000","last_clean":"2023-10-16T15:54:40.350758+0000","last_became_active":"2023-10-16T15:54:39.209504+0000","last_became_peered":"2023-10-16T15:54:39.209504+0000","last_unstale":"2023-10-16T15:54:40.350758+0000","last_undegraded":"2023-10-16T15:54:40.350758+0000","last_fullsized":"2023-10-16T15:54:40.350758+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351869+0000","last_change":"2023-10-16T15:54:39.207379+0000","last_active":"2023-10-16T15:54:40.351869+0000","last_peered":"2023-10-16T15:54:40.351869+0000","last_clean":"2023-10-16T15:54:40.351869+0000","last_became_active":"2023-10-16T15:54:39.207010+0000","last_became_peered":"2023-10-16T15:54:39.207010+0000","last_unstale":"2023-10-16T15:54:40.351869+0000","last_undegraded":"2023-10-16T15:54:40.351869+0000","last_fullsized":"2023-10-16T15:54:40.351869+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:45.890691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172589+0000","last_change":"2023-10-16T15:54:39.199635+0000","last_active":"2023-10-16T15:54:40.172589+0000","last_peered":"2023-10-16T15:54:40.172589+0000","last_clean":"2023-10-16T15:54:40.172589+0000","last_became_active":"2023-10-16T15:54:39.199152+0000","last_became_peered":"2023-10-16T15:54:39.199152+0000","last_unstale":"2023-10-16T15:54:40.172589+0000","last_undegraded":"2023-10-16T15:54:40.172589+0000","last_fullsized":"2023-10-16T15:54:40.172589+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:45.206271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352011+0000","last_change":"2023-10-16T15:54:39.205509+0000","last_active":"2023-10-16T15:54:40.352011+0000","last_peered":"2023-10-16T15:54:40.352011+0000","last_clean":"2023-10-16T15:54:40.352011+0000","last_became_active":"2023-10-16T15:54:39.204206+0000","last_became_peered":"2023-10-16T15:54:39.204206+0000","last_unstale":"2023-10-16T15:54:40.352011+0000","last_undegraded":"2023-10-16T15:54:40.352011+0000","last_fullsized":"2023-10-16T15:54:40.352011+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352395+0000","last_change":"2023-10-16T15:54:39.194931+0000","last_active":"2023-10-16T15:54:40.352395+0000","last_peered":"2023-10-16T15:54:40.352395+0000","last_clean":"2023-10-16T15:54:40.352395+0000","last_became_active":"2023-10-16T15:54:39.194605+0000","last_became_peered":"2023-10-16T15:54:39.194605+0000","last_unstale":"2023-10-16T15:54:40.352395+0000","last_undegraded":"2023-10-16T15:54:40.352395+0000","last_fullsized":"2023-10-16T15:54:40.352395+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354085+0000","last_change":"2023-10-16T15:54:39.204169+0000","last_active":"2023-10-16T15:54:40.354085+0000","last_peered":"2023-10-16T15:54:40.354085+0000","last_clean":"2023-10-16T15:54:40.354085+0000","last_became_active":"2023-10-16T15:54:39.200108+0000","last_became_peered":"2023-10-16T15:54:39.200108+0000","last_unstale":"2023-10-16T15:54:40.354085+0000","last_undegraded":"2023-10-16T15:54:40.354085+0000","last_fullsized":"2023-10-16T15:54:40.354085+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:23.620457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172893+0000","last_change":"2023-10-16T15:54:39.203124+0000","last_active":"2023-10-16T15:54:40.172893+0000","last_peered":"2023-10-16T15:54:40.172893+0000","last_clean":"2023-10-16T15:54:40.172893+0000","last_became_active":"2023-10-16T15:54:39.200245+0000","last_became_peered":"2023-10-16T15:54:39.200245+0000","last_unstale":"2023-10-16T15:54:40.172893+0000","last_undegraded":"2023-10-16T15:54:40.172893+0000","last_fullsized":"2023-10-16T15:54:40.172893+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:43.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352069+0000","last_change":"2023-10-16T15:54:39.203684+0000","last_active":"2023-10-16T15:54:40.352069+0000","last_peered":"2023-10-16T15:54:40.352069+0000","last_clean":"2023-10-16T15:54:40.352069+0000","last_became_active":"2023-10-16T15:54:39.203327+0000","last_became_peered":"2023-10-16T15:54:39.203327+0000","last_unstale":"2023-10-16T15:54:40.352069+0000","last_undegraded":"2023-10-16T15:54:40.352069+0000","last_fullsized":"2023-10-16T15:54:40.352069+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:29.051715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178220+0000","last_change":"2023-10-16T15:54:39.203576+0000","last_active":"2023-10-16T15:54:40.178220+0000","last_peered":"2023-10-16T15:54:40.178220+0000","last_clean":"2023-10-16T15:54:40.178220+0000","last_became_active":"2023-10-16T15:54:39.203072+0000","last_became_peered":"2023-10-16T15:54:39.203072+0000","last_unstale":"2023-10-16T15:54:40.178220+0000","last_undegraded":"2023-10-16T15:54:40.178220+0000","last_fullsized":"2023-10-16T15:54:40.178220+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:05.472712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351137+0000","last_change":"2023-10-16T15:54:39.210101+0000","last_active":"2023-10-16T15:54:40.351137+0000","last_peered":"2023-10-16T15:54:40.351137+0000","last_clean":"2023-10-16T15:54:40.351137+0000","last_became_active":"2023-10-16T15:54:39.209787+0000","last_became_peered":"2023-10-16T15:54:39.209787+0000","last_unstale":"2023-10-16T15:54:40.351137+0000","last_undegraded":"2023-10-16T15:54:40.351137+0000","last_fullsized":"2023-10-16T15:54:40.351137+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172635+0000","last_change":"2023-10-16T15:54:39.198844+0000","last_active":"2023-10-16T15:54:40.172635+0000","last_peered":"2023-10-16T15:54:40.172635+0000","last_clean":"2023-10-16T15:54:40.172635+0000","last_became_active":"2023-10-16T15:54:39.198444+0000","last_became_peered":"2023-10-16T15:54:39.198444+0000","last_unstale":"2023-10-16T15:54:40.172635+0000","last_undegraded":"2023-10-16T15:54:40.172635+0000","last_fullsized":"2023-10-16T15:54:40.172635+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:47.149143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356170+0000","last_change":"2023-10-16T15:54:39.213572+0000","last_active":"2023-10-16T15:54:40.356170+0000","last_peered":"2023-10-16T15:54:40.356170+0000","last_clean":"2023-10-16T15:54:40.356170+0000","last_became_active":"2023-10-16T15:54:39.213226+0000","last_became_peered":"2023-10-16T15:54:39.213226+0000","last_unstale":"2023-10-16T15:54:40.356170+0000","last_undegraded":"2023-10-16T15:54:40.356170+0000","last_fullsized":"2023-10-16T15:54:40.356170+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:23.249562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352934+0000","last_change":"2023-10-16T15:54:38.194041+0000","last_active":"2023-10-16T15:54:40.352934+0000","last_peered":"2023-10-16T15:54:40.352934+0000","last_clean":"2023-10-16T15:54:40.352934+0000","last_became_active":"2023-10-16T15:54:38.193770+0000","last_became_peered":"2023-10-16T15:54:38.193770+0000","last_unstale":"2023-10-16T15:54:40.352934+0000","last_undegraded":"2023-10-16T15:54:40.352934+0000","last_fullsized":"2023-10-16T15:54:40.352934+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352886+0000","last_change":"2023-10-16T15:54:39.209350+0000","last_active":"2023-10-16T15:54:40.352886+0000","last_peered":"2023-10-16T15:54:40.352886+0000","last_clean":"2023-10-16T15:54:40.352886+0000","last_became_active":"2023-10-16T15:54:39.208829+0000","last_became_peered":"2023-10-16T15:54:39.208829+0000","last_unstale":"2023-10-16T15:54:40.352886+0000","last_undegraded":"2023-10-16T15:54:40.352886+0000","last_fullsized":"2023-10-16T15:54:40.352886+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:20.584860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355522+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:40.355522+0000","last_peered":"2023-10-16T15:54:40.355522+0000","last_clean":"2023-10-16T15:54:40.355522+0000","last_became_active":"2023-10-16T15:54:38.195767+0000","last_became_peered":"2023-10-16T15:54:38.195767+0000","last_unstale":"2023-10-16T15:54:40.355522+0000","last_undegraded":"2023-10-16T15:54:40.355522+0000","last_fullsized":"2023-10-16T15:54:40.355522+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:59.473824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353354+0000","last_change":"2023-10-16T15:54:39.209739+0000","last_active":"2023-10-16T15:54:40.353354+0000","last_peered":"2023-10-16T15:54:40.353354+0000","last_clean":"2023-10-16T15:54:40.353354+0000","last_became_active":"2023-10-16T15:54:39.208616+0000","last_became_peered":"2023-10-16T15:54:39.208616+0000","last_unstale":"2023-10-16T15:54:40.353354+0000","last_undegraded":"2023-10-16T15:54:40.353354+0000","last_fullsized":"2023-10-16T15:54:40.353354+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356264+0000","last_change":"2023-10-16T15:54:38.192004+0000","last_active":"2023-10-16T15:54:40.356264+0000","last_peered":"2023-10-16T15:54:40.356264+0000","last_clean":"2023-10-16T15:54:40.356264+0000","last_became_active":"2023-10-16T15:54:38.191365+0000","last_became_peered":"2023-10-16T15:54:38.191365+0000","last_unstale":"2023-10-16T15:54:40.356264+0000","last_undegraded":"2023-10-16T15:54:40.356264+0000","last_fullsized":"2023-10-16T15:54:40.356264+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:00.022703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354115+0000","last_change":"2023-10-16T15:54:39.201055+0000","last_active":"2023-10-16T15:54:40.354115+0000","last_peered":"2023-10-16T15:54:40.354115+0000","last_clean":"2023-10-16T15:54:40.354115+0000","last_became_active":"2023-10-16T15:54:39.199846+0000","last_became_peered":"2023-10-16T15:54:39.199846+0000","last_unstale":"2023-10-16T15:54:40.354115+0000","last_undegraded":"2023-10-16T15:54:40.354115+0000","last_fullsized":"2023-10-16T15:54:40.354115+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:28.581753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350984+0000","last_change":"2023-10-16T15:54:38.188455+0000","last_active":"2023-10-16T15:54:40.350984+0000","last_peered":"2023-10-16T15:54:40.350984+0000","last_clean":"2023-10-16T15:54:40.350984+0000","last_became_active":"2023-10-16T15:54:38.188167+0000","last_became_peered":"2023-10-16T15:54:38.188167+0000","last_unstale":"2023-10-16T15:54:40.350984+0000","last_undegraded":"2023-10-16T15:54:40.350984+0000","last_fullsized":"2023-10-16T15:54:40.350984+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:21.554285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178984+0000","last_change":"2023-10-16T15:54:39.198491+0000","last_active":"2023-10-16T15:54:40.178984+0000","last_peered":"2023-10-16T15:54:40.178984+0000","last_clean":"2023-10-16T15:54:40.178984+0000","last_became_active":"2023-10-16T15:54:39.194488+0000","last_became_peered":"2023-10-16T15:54:39.194488+0000","last_unstale":"2023-10-16T15:54:40.178984+0000","last_undegraded":"2023-10-16T15:54:40.178984+0000","last_fullsized":"2023-10-16T15:54:40.178984+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.663424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172779+0000","last_change":"2023-10-16T15:54:38.190757+0000","last_active":"2023-10-16T15:54:40.172779+0000","last_peered":"2023-10-16T15:54:40.172779+0000","last_clean":"2023-10-16T15:54:40.172779+0000","last_became_active":"2023-10-16T15:54:38.190491+0000","last_became_peered":"2023-10-16T15:54:38.190491+0000","last_unstale":"2023-10-16T15:54:40.172779+0000","last_undegraded":"2023-10-16T15:54:40.172779+0000","last_fullsized":"2023-10-16T15:54:40.172779+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:59.249150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352516+0000","last_change":"2023-10-16T15:54:39.206722+0000","last_active":"2023-10-16T15:54:40.352516+0000","last_peered":"2023-10-16T15:54:40.352516+0000","last_clean":"2023-10-16T15:54:40.352516+0000","last_became_active":"2023-10-16T15:54:39.204447+0000","last_became_peered":"2023-10-16T15:54:39.204447+0000","last_unstale":"2023-10-16T15:54:40.352516+0000","last_undegraded":"2023-10-16T15:54:40.352516+0000","last_fullsized":"2023-10-16T15:54:40.352516+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:50.316780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350787+0000","last_change":"2023-10-16T15:54:38.189769+0000","last_active":"2023-10-16T15:54:40.350787+0000","last_peered":"2023-10-16T15:54:40.350787+0000","last_clean":"2023-10-16T15:54:40.350787+0000","last_became_active":"2023-10-16T15:54:38.189418+0000","last_became_peered":"2023-10-16T15:54:38.189418+0000","last_unstale":"2023-10-16T15:54:40.350787+0000","last_undegraded":"2023-10-16T15:54:40.350787+0000","last_fullsized":"2023-10-16T15:54:40.350787+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.444572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172625+0000","last_change":"2023-10-16T15:54:39.205679+0000","last_active":"2023-10-16T15:54:40.172625+0000","last_peered":"2023-10-16T15:54:40.172625+0000","last_clean":"2023-10-16T15:54:40.172625+0000","last_became_active":"2023-10-16T15:54:39.204821+0000","last_became_peered":"2023-10-16T15:54:39.204821+0000","last_unstale":"2023-10-16T15:54:40.172625+0000","last_undegraded":"2023-10-16T15:54:40.172625+0000","last_fullsized":"2023-10-16T15:54:40.172625+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:44.694895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352789+0000","last_change":"2023-10-16T15:54:38.191853+0000","last_active":"2023-10-16T15:54:40.352789+0000","last_peered":"2023-10-16T15:54:40.352789+0000","last_clean":"2023-10-16T15:54:40.352789+0000","last_became_active":"2023-10-16T15:54:38.190725+0000","last_became_peered":"2023-10-16T15:54:38.190725+0000","last_unstale":"2023-10-16T15:54:40.352789+0000","last_undegraded":"2023-10-16T15:54:40.352789+0000","last_fullsized":"2023-10-16T15:54:40.352789+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:51.409385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173318+0000","last_change":"2023-10-16T15:54:39.205421+0000","last_active":"2023-10-16T15:54:40.173318+0000","last_peered":"2023-10-16T15:54:40.173318+0000","last_clean":"2023-10-16T15:54:40.173318+0000","last_became_active":"2023-10-16T15:54:39.204865+0000","last_became_peered":"2023-10-16T15:54:39.204865+0000","last_unstale":"2023-10-16T15:54:40.173318+0000","last_undegraded":"2023-10-16T15:54:40.173318+0000","last_fullsized":"2023-10-16T15:54:40.173318+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:33.811103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353190+0000","last_change":"2023-10-16T15:54:38.196240+0000","last_active":"2023-10-16T15:54:40.353190+0000","last_peered":"2023-10-16T15:54:40.353190+0000","last_clean":"2023-10-16T15:54:40.353190+0000","last_became_active":"2023-10-16T15:54:38.195833+0000","last_became_peered":"2023-10-16T15:54:38.195833+0000","last_unstale":"2023-10-16T15:54:40.353190+0000","last_undegraded":"2023-10-16T15:54:40.353190+0000","last_fullsized":"2023-10-16T15:54:40.353190+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354077+0000","last_change":"2023-10-16T15:54:39.204677+0000","last_active":"2023-10-16T15:54:40.354077+0000","last_peered":"2023-10-16T15:54:40.354077+0000","last_clean":"2023-10-16T15:54:40.354077+0000","last_became_active":"2023-10-16T15:54:39.202794+0000","last_became_peered":"2023-10-16T15:54:39.202794+0000","last_unstale":"2023-10-16T15:54:40.354077+0000","last_undegraded":"2023-10-16T15:54:40.354077+0000","last_fullsized":"2023-10-16T15:54:40.354077+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:11.420100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355954+0000","last_change":"2023-10-16T15:54:38.195322+0000","last_active":"2023-10-16T15:54:40.355954+0000","last_peered":"2023-10-16T15:54:40.355954+0000","last_clean":"2023-10-16T15:54:40.355954+0000","last_became_active":"2023-10-16T15:54:38.192163+0000","last_became_peered":"2023-10-16T15:54:38.192163+0000","last_unstale":"2023-10-16T15:54:40.355954+0000","last_undegraded":"2023-10-16T15:54:40.355954+0000","last_fullsized":"2023-10-16T15:54:40.355954+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:45.552506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356311+0000","last_change":"2023-10-16T15:54:39.212753+0000","last_active":"2023-10-16T15:54:40.356311+0000","last_peered":"2023-10-16T15:54:40.356311+0000","last_clean":"2023-10-16T15:54:40.356311+0000","last_became_active":"2023-10-16T15:54:39.212450+0000","last_became_peered":"2023-10-16T15:54:39.212450+0000","last_unstale":"2023-10-16T15:54:40.356311+0000","last_undegraded":"2023-10-16T15:54:40.356311+0000","last_fullsized":"2023-10-16T15:54:40.356311+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:04.593631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"245'3","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173038+0000","last_change":"2023-10-16T15:54:38.188991+0000","last_active":"2023-10-16T15:54:40.173038+0000","last_peered":"2023-10-16T15:54:40.173038+0000","last_clean":"2023-10-16T15:54:40.173038+0000","last_became_active":"2023-10-16T15:54:38.187521+0000","last_became_peered":"2023-10-16T15:54:38.187521+0000","last_unstale":"2023-10-16T15:54:40.173038+0000","last_undegraded":"2023-10-16T15:54:40.173038+0000","last_fullsized":"2023-10-16T15:54:40.173038+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:37.422309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352055+0000","last_change":"2023-10-16T15:54:39.188004+0000","last_active":"2023-10-16T15:54:40.352055+0000","last_peered":"2023-10-16T15:54:40.352055+0000","last_clean":"2023-10-16T15:54:40.352055+0000","last_became_active":"2023-10-16T15:54:39.187443+0000","last_became_peered":"2023-10-16T15:54:39.187443+0000","last_unstale":"2023-10-16T15:54:40.352055+0000","last_undegraded":"2023-10-16T15:54:40.352055+0000","last_fullsized":"2023-10-16T15:54:40.352055+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"245'6","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172866+0000","last_change":"2023-10-16T15:54:38.192821+0000","last_active":"2023-10-16T15:54:40.172866+0000","last_peered":"2023-10-16T15:54:40.172866+0000","last_clean":"2023-10-16T15:54:40.172866+0000","last_became_active":"2023-10-16T15:54:38.191495+0000","last_became_peered":"2023-10-16T15:54:38.191495+0000","last_unstale":"2023-10-16T15:54:40.172866+0000","last_undegraded":"2023-10-16T15:54:40.172866+0000","last_fullsized":"2023-10-16T15:54:40.172866+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:36.442402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350812+0000","last_change":"2023-10-16T15:54:39.203188+0000","last_active":"2023-10-16T15:54:40.350812+0000","last_peered":"2023-10-16T15:54:40.350812+0000","last_clean":"2023-10-16T15:54:40.350812+0000","last_became_active":"2023-10-16T15:54:39.202646+0000","last_became_peered":"2023-10-16T15:54:39.202646+0000","last_unstale":"2023-10-16T15:54:40.350812+0000","last_undegraded":"2023-10-16T15:54:40.350812+0000","last_fullsized":"2023-10-16T15:54:40.350812+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172824+0000","last_change":"2023-10-16T15:54:38.190094+0000","last_active":"2023-10-16T15:54:40.172824+0000","last_peered":"2023-10-16T15:54:40.172824+0000","last_clean":"2023-10-16T15:54:40.172824+0000","last_became_active":"2023-10-16T15:54:38.189825+0000","last_became_peered":"2023-10-16T15:54:38.189825+0000","last_unstale":"2023-10-16T15:54:40.172824+0000","last_undegraded":"2023-10-16T15:54:40.172824+0000","last_fullsized":"2023-10-16T15:54:40.172824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:17.021872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.179335+0000","last_change":"2023-10-16T15:54:39.205965+0000","last_active":"2023-10-16T15:54:40.179335+0000","last_peered":"2023-10-16T15:54:40.179335+0000","last_clean":"2023-10-16T15:54:40.179335+0000","last_became_active":"2023-10-16T15:54:39.205686+0000","last_became_peered":"2023-10-16T15:54:39.205686+0000","last_unstale":"2023-10-16T15:54:40.179335+0000","last_undegraded":"2023-10-16T15:54:40.179335+0000","last_fullsized":"2023-10-16T15:54:40.179335+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:36.119641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353169+0000","last_change":"2023-10-16T15:54:38.195373+0000","last_active":"2023-10-16T15:54:40.353169+0000","last_peered":"2023-10-16T15:54:40.353169+0000","last_clean":"2023-10-16T15:54:40.353169+0000","last_became_active":"2023-10-16T15:54:38.194574+0000","last_became_peered":"2023-10-16T15:54:38.194574+0000","last_unstale":"2023-10-16T15:54:40.353169+0000","last_undegraded":"2023-10-16T15:54:40.353169+0000","last_fullsized":"2023-10-16T15:54:40.353169+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:35.101378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351086+0000","last_change":"2023-10-16T15:54:39.207613+0000","last_active":"2023-10-16T15:54:40.351086+0000","last_peered":"2023-10-16T15:54:40.351086+0000","last_clean":"2023-10-16T15:54:40.351086+0000","last_became_active":"2023-10-16T15:54:39.207320+0000","last_became_peered":"2023-10-16T15:54:39.207320+0000","last_unstale":"2023-10-16T15:54:40.351086+0000","last_undegraded":"2023-10-16T15:54:40.351086+0000","last_fullsized":"2023-10-16T15:54:40.351086+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351014+0000","last_change":"2023-10-16T15:54:38.196425+0000","last_active":"2023-10-16T15:54:40.351014+0000","last_peered":"2023-10-16T15:54:40.351014+0000","last_clean":"2023-10-16T15:54:40.351014+0000","last_became_active":"2023-10-16T15:54:38.195269+0000","last_became_peered":"2023-10-16T15:54:38.195269+0000","last_unstale":"2023-10-16T15:54:40.351014+0000","last_undegraded":"2023-10-16T15:54:40.351014+0000","last_fullsized":"2023-10-16T15:54:40.351014+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:01.207615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350986+0000","last_change":"2023-10-16T15:54:39.205462+0000","last_active":"2023-10-16T15:54:40.350986+0000","last_peered":"2023-10-16T15:54:40.350986+0000","last_clean":"2023-10-16T15:54:40.350986+0000","last_became_active":"2023-10-16T15:54:39.204956+0000","last_became_peered":"2023-10-16T15:54:39.204956+0000","last_unstale":"2023-10-16T15:54:40.350986+0000","last_undegraded":"2023-10-16T15:54:40.350986+0000","last_fullsized":"2023-10-16T15:54:40.350986+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"245'8","reported_seq":20,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350416+0000","last_change":"2023-10-16T15:54:38.194134+0000","last_active":"2023-10-16T15:54:40.350416+0000","last_peered":"2023-10-16T15:54:40.350416+0000","last_clean":"2023-10-16T15:54:40.350416+0000","last_became_active":"2023-10-16T15:54:38.193858+0000","last_became_peered":"2023-10-16T15:54:38.193858+0000","last_unstale":"2023-10-16T15:54:40.350416+0000","last_undegraded":"2023-10-16T15:54:40.350416+0000","last_fullsized":"2023-10-16T15:54:40.350416+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:37.866131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352402+0000","last_change":"2023-10-16T15:54:39.203343+0000","last_active":"2023-10-16T15:54:40.352402+0000","last_peered":"2023-10-16T15:54:40.352402+0000","last_clean":"2023-10-16T15:54:40.352402+0000","last_became_active":"2023-10-16T15:54:39.203044+0000","last_became_peered":"2023-10-16T15:54:39.203044+0000","last_unstale":"2023-10-16T15:54:40.352402+0000","last_undegraded":"2023-10-16T15:54:40.352402+0000","last_fullsized":"2023-10-16T15:54:40.352402+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:23:49.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353379+0000","last_change":"2023-10-16T15:54:38.193502+0000","last_active":"2023-10-16T15:54:40.353379+0000","last_peered":"2023-10-16T15:54:40.353379+0000","last_clean":"2023-10-16T15:54:40.353379+0000","last_became_active":"2023-10-16T15:54:38.193186+0000","last_became_peered":"2023-10-16T15:54:38.193186+0000","last_unstale":"2023-10-16T15:54:40.353379+0000","last_undegraded":"2023-10-16T15:54:40.353379+0000","last_fullsized":"2023-10-16T15:54:40.353379+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356122+0000","last_change":"2023-10-16T15:54:39.206218+0000","last_active":"2023-10-16T15:54:40.356122+0000","last_peered":"2023-10-16T15:54:40.356122+0000","last_clean":"2023-10-16T15:54:40.356122+0000","last_became_active":"2023-10-16T15:54:39.205815+0000","last_became_peered":"2023-10-16T15:54:39.205815+0000","last_unstale":"2023-10-16T15:54:40.356122+0000","last_undegraded":"2023-10-16T15:54:40.356122+0000","last_fullsized":"2023-10-16T15:54:40.356122+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:24.642539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355904+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:40.355904+0000","last_peered":"2023-10-16T15:54:40.355904+0000","last_clean":"2023-10-16T15:54:40.355904+0000","last_became_active":"2023-10-16T15:54:38.195752+0000","last_became_peered":"2023-10-16T15:54:38.195752+0000","last_unstale":"2023-10-16T15:54:40.355904+0000","last_undegraded":"2023-10-16T15:54:40.355904+0000","last_fullsized":"2023-10-16T15:54:40.355904+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:15.145108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.179269+0000","last_change":"2023-10-16T15:54:39.198603+0000","last_active":"2023-10-16T15:54:40.179269+0000","last_peered":"2023-10-16T15:54:40.179269+0000","last_clean":"2023-10-16T15:54:40.179269+0000","last_became_active":"2023-10-16T15:54:39.195793+0000","last_became_peered":"2023-10-16T15:54:39.195793+0000","last_unstale":"2023-10-16T15:54:40.179269+0000","last_undegraded":"2023-10-16T15:54:40.179269+0000","last_fullsized":"2023-10-16T15:54:40.179269+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:16.753544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"245'1","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.177915+0000","last_change":"2023-10-16T15:54:38.193258+0000","last_active":"2023-10-16T15:54:40.177915+0000","last_peered":"2023-10-16T15:54:40.177915+0000","last_clean":"2023-10-16T15:54:40.177915+0000","last_became_active":"2023-10-16T15:54:38.192538+0000","last_became_peered":"2023-10-16T15:54:38.192538+0000","last_unstale":"2023-10-16T15:54:40.177915+0000","last_undegraded":"2023-10-16T15:54:40.177915+0000","last_fullsized":"2023-10-16T15:54:40.177915+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:07.463985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353601+0000","last_change":"2023-10-16T15:54:39.206968+0000","last_active":"2023-10-16T15:54:40.353601+0000","last_peered":"2023-10-16T15:54:40.353601+0000","last_clean":"2023-10-16T15:54:40.353601+0000","last_became_active":"2023-10-16T15:54:39.206271+0000","last_became_peered":"2023-10-16T15:54:39.206271+0000","last_unstale":"2023-10-16T15:54:40.353601+0000","last_undegraded":"2023-10-16T15:54:40.353601+0000","last_fullsized":"2023-10-16T15:54:40.353601+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353462+0000","last_change":"2023-10-16T15:54:38.194296+0000","last_active":"2023-10-16T15:54:40.353462+0000","last_peered":"2023-10-16T15:54:40.353462+0000","last_clean":"2023-10-16T15:54:40.353462+0000","last_became_active":"2023-10-16T15:54:38.194015+0000","last_became_peered":"2023-10-16T15:54:38.194015+0000","last_unstale":"2023-10-16T15:54:40.353462+0000","last_undegraded":"2023-10-16T15:54:40.353462+0000","last_fullsized":"2023-10-16T15:54:40.353462+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:58.920639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352345+0000","last_change":"2023-10-16T15:54:39.206512+0000","last_active":"2023-10-16T15:54:40.352345+0000","last_peered":"2023-10-16T15:54:40.352345+0000","last_clean":"2023-10-16T15:54:40.352345+0000","last_became_active":"2023-10-16T15:54:39.206174+0000","last_became_peered":"2023-10-16T15:54:39.206174+0000","last_unstale":"2023-10-16T15:54:40.352345+0000","last_undegraded":"2023-10-16T15:54:40.352345+0000","last_fullsized":"2023-10-16T15:54:40.352345+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.355215+0000","last_change":"2023-10-16T15:54:38.199965+0000","last_active":"2023-10-16T15:54:40.355215+0000","last_peered":"2023-10-16T15:54:40.355215+0000","last_clean":"2023-10-16T15:54:40.355215+0000","last_became_active":"2023-10-16T15:54:38.199417+0000","last_became_peered":"2023-10-16T15:54:38.199417+0000","last_unstale":"2023-10-16T15:54:40.355215+0000","last_undegraded":"2023-10-16T15:54:40.355215+0000","last_fullsized":"2023-10-16T15:54:40.355215+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:55.688720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172846+0000","last_change":"2023-10-16T15:54:39.199452+0000","last_active":"2023-10-16T15:54:40.172846+0000","last_peered":"2023-10-16T15:54:40.172846+0000","last_clean":"2023-10-16T15:54:40.172846+0000","last_became_active":"2023-10-16T15:54:39.199132+0000","last_became_peered":"2023-10-16T15:54:39.199132+0000","last_unstale":"2023-10-16T15:54:40.172846+0000","last_undegraded":"2023-10-16T15:54:40.172846+0000","last_fullsized":"2023-10-16T15:54:40.172846+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:49.564533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354346+0000","last_change":"2023-10-16T15:54:38.193942+0000","last_active":"2023-10-16T15:54:40.354346+0000","last_peered":"2023-10-16T15:54:40.354346+0000","last_clean":"2023-10-16T15:54:40.354346+0000","last_became_active":"2023-10-16T15:54:38.193397+0000","last_became_peered":"2023-10-16T15:54:38.193397+0000","last_unstale":"2023-10-16T15:54:40.354346+0000","last_undegraded":"2023-10-16T15:54:40.354346+0000","last_fullsized":"2023-10-16T15:54:40.354346+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:40.933295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350317+0000","last_change":"2023-10-16T15:54:39.188638+0000","last_active":"2023-10-16T15:54:40.350317+0000","last_peered":"2023-10-16T15:54:40.350317+0000","last_clean":"2023-10-16T15:54:40.350317+0000","last_became_active":"2023-10-16T15:54:39.187890+0000","last_became_peered":"2023-10-16T15:54:39.187890+0000","last_unstale":"2023-10-16T15:54:40.350317+0000","last_undegraded":"2023-10-16T15:54:40.350317+0000","last_fullsized":"2023-10-16T15:54:40.350317+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:15:41.121120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178593+0000","last_change":"2023-10-16T15:54:38.196671+0000","last_active":"2023-10-16T15:54:40.178593+0000","last_peered":"2023-10-16T15:54:40.178593+0000","last_clean":"2023-10-16T15:54:40.178593+0000","last_became_active":"2023-10-16T15:54:38.196392+0000","last_became_peered":"2023-10-16T15:54:38.196392+0000","last_unstale":"2023-10-16T15:54:40.178593+0000","last_undegraded":"2023-10-16T15:54:40.178593+0000","last_fullsized":"2023-10-16T15:54:40.178593+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:35.189299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351934+0000","last_change":"2023-10-16T15:54:39.203931+0000","last_active":"2023-10-16T15:54:40.351934+0000","last_peered":"2023-10-16T15:54:40.351934+0000","last_clean":"2023-10-16T15:54:40.351934+0000","last_became_active":"2023-10-16T15:54:39.203614+0000","last_became_peered":"2023-10-16T15:54:39.203614+0000","last_unstale":"2023-10-16T15:54:40.351934+0000","last_undegraded":"2023-10-16T15:54:40.351934+0000","last_fullsized":"2023-10-16T15:54:40.351934+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.350465+0000","last_change":"2023-10-16T15:54:38.187905+0000","last_active":"2023-10-16T15:54:40.350465+0000","last_peered":"2023-10-16T15:54:40.350465+0000","last_clean":"2023-10-16T15:54:40.350465+0000","last_became_active":"2023-10-16T15:54:38.187553+0000","last_became_peered":"2023-10-16T15:54:38.187553+0000","last_unstale":"2023-10-16T15:54:40.350465+0000","last_undegraded":"2023-10-16T15:54:40.350465+0000","last_fullsized":"2023-10-16T15:54:40.350465+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:27.563490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352704+0000","last_change":"2023-10-16T15:54:39.201684+0000","last_active":"2023-10-16T15:54:40.352704+0000","last_peered":"2023-10-16T15:54:40.352704+0000","last_clean":"2023-10-16T15:54:40.352704+0000","last_became_active":"2023-10-16T15:54:39.201363+0000","last_became_peered":"2023-10-16T15:54:39.201363+0000","last_unstale":"2023-10-16T15:54:40.352704+0000","last_undegraded":"2023-10-16T15:54:40.352704+0000","last_fullsized":"2023-10-16T15:54:40.352704+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"245'2","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352107+0000","last_change":"2023-10-16T15:54:38.193490+0000","last_active":"2023-10-16T15:54:40.352107+0000","last_peered":"2023-10-16T15:54:40.352107+0000","last_clean":"2023-10-16T15:54:40.352107+0000","last_became_active":"2023-10-16T15:54:38.193206+0000","last_became_peered":"2023-10-16T15:54:38.193206+0000","last_unstale":"2023-10-16T15:54:40.352107+0000","last_undegraded":"2023-10-16T15:54:40.352107+0000","last_fullsized":"2023-10-16T15:54:40.352107+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:39.542212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.173233+0000","last_change":"2023-10-16T15:54:39.206833+0000","last_active":"2023-10-16T15:54:40.173233+0000","last_peered":"2023-10-16T15:54:40.173233+0000","last_clean":"2023-10-16T15:54:40.173233+0000","last_became_active":"2023-10-16T15:54:39.206491+0000","last_became_peered":"2023-10-16T15:54:39.206491+0000","last_unstale":"2023-10-16T15:54:40.173233+0000","last_undegraded":"2023-10-16T15:54:40.173233+0000","last_fullsized":"2023-10-16T15:54:40.173233+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:22.432507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178415+0000","last_change":"2023-10-16T15:54:38.198292+0000","last_active":"2023-10-16T15:54:40.178415+0000","last_peered":"2023-10-16T15:54:40.178415+0000","last_clean":"2023-10-16T15:54:40.178415+0000","last_became_active":"2023-10-16T15:54:38.197965+0000","last_became_peered":"2023-10-16T15:54:38.197965+0000","last_unstale":"2023-10-16T15:54:40.178415+0000","last_undegraded":"2023-10-16T15:54:40.178415+0000","last_fullsized":"2023-10-16T15:54:40.178415+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:07.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353233+0000","last_change":"2023-10-16T15:54:39.198360+0000","last_active":"2023-10-16T15:54:40.353233+0000","last_peered":"2023-10-16T15:54:40.353233+0000","last_clean":"2023-10-16T15:54:40.353233+0000","last_became_active":"2023-10-16T15:54:39.196440+0000","last_became_peered":"2023-10-16T15:54:39.196440+0000","last_unstale":"2023-10-16T15:54:40.353233+0000","last_undegraded":"2023-10-16T15:54:40.353233+0000","last_fullsized":"2023-10-16T15:54:40.353233+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:23.439544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178145+0000","last_change":"2023-10-16T15:54:38.199930+0000","last_active":"2023-10-16T15:54:40.178145+0000","last_peered":"2023-10-16T15:54:40.178145+0000","last_clean":"2023-10-16T15:54:40.178145+0000","last_became_active":"2023-10-16T15:54:38.199386+0000","last_became_peered":"2023-10-16T15:54:38.199386+0000","last_unstale":"2023-10-16T15:54:40.178145+0000","last_undegraded":"2023-10-16T15:54:40.178145+0000","last_fullsized":"2023-10-16T15:54:40.178145+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:57.311627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.354487+0000","last_change":"2023-10-16T15:54:39.205703+0000","last_active":"2023-10-16T15:54:40.354487+0000","last_peered":"2023-10-16T15:54:40.354487+0000","last_clean":"2023-10-16T15:54:40.354487+0000","last_became_active":"2023-10-16T15:54:39.205415+0000","last_became_peered":"2023-10-16T15:54:39.205415+0000","last_unstale":"2023-10-16T15:54:40.354487+0000","last_undegraded":"2023-10-16T15:54:40.354487+0000","last_fullsized":"2023-10-16T15:54:40.354487+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:02.225878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353717+0000","last_change":"2023-10-16T15:54:38.184497+0000","last_active":"2023-10-16T15:54:40.353717+0000","last_peered":"2023-10-16T15:54:40.353717+0000","last_clean":"2023-10-16T15:54:40.353717+0000","last_became_active":"2023-10-16T15:54:38.184043+0000","last_became_peered":"2023-10-16T15:54:38.184043+0000","last_unstale":"2023-10-16T15:54:40.353717+0000","last_undegraded":"2023-10-16T15:54:40.353717+0000","last_fullsized":"2023-10-16T15:54:40.353717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:04.454449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351389+0000","last_change":"2023-10-16T15:54:39.192680+0000","last_active":"2023-10-16T15:54:40.351389+0000","last_peered":"2023-10-16T15:54:40.351389+0000","last_clean":"2023-10-16T15:54:40.351389+0000","last_became_active":"2023-10-16T15:54:39.192254+0000","last_became_peered":"2023-10-16T15:54:39.192254+0000","last_unstale":"2023-10-16T15:54:40.351389+0000","last_undegraded":"2023-10-16T15:54:40.351389+0000","last_fullsized":"2023-10-16T15:54:40.351389+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353194+0000","last_change":"2023-10-16T15:54:38.192812+0000","last_active":"2023-10-16T15:54:40.353194+0000","last_peered":"2023-10-16T15:54:40.353194+0000","last_clean":"2023-10-16T15:54:40.353194+0000","last_became_active":"2023-10-16T15:54:38.192528+0000","last_became_peered":"2023-10-16T15:54:38.192528+0000","last_unstale":"2023-10-16T15:54:40.353194+0000","last_undegraded":"2023-10-16T15:54:40.353194+0000","last_fullsized":"2023-10-16T15:54:40.353194+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:36.002573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351573+0000","last_change":"2023-10-16T15:54:39.207142+0000","last_active":"2023-10-16T15:54:40.351573+0000","last_peered":"2023-10-16T15:54:40.351573+0000","last_clean":"2023-10-16T15:54:40.351573+0000","last_became_active":"2023-10-16T15:54:39.206593+0000","last_became_peered":"2023-10-16T15:54:39.206593+0000","last_unstale":"2023-10-16T15:54:40.351573+0000","last_undegraded":"2023-10-16T15:54:40.351573+0000","last_fullsized":"2023-10-16T15:54:40.351573+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:58.545523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352233+0000","last_change":"2023-10-16T15:54:38.179331+0000","last_active":"2023-10-16T15:54:40.352233+0000","last_peered":"2023-10-16T15:54:40.352233+0000","last_clean":"2023-10-16T15:54:40.352233+0000","last_became_active":"2023-10-16T15:54:38.178684+0000","last_became_peered":"2023-10-16T15:54:38.178684+0000","last_unstale":"2023-10-16T15:54:40.352233+0000","last_undegraded":"2023-10-16T15:54:40.352233+0000","last_fullsized":"2023-10-16T15:54:40.352233+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.351371+0000","last_change":"2023-10-16T15:54:39.203357+0000","last_active":"2023-10-16T15:54:40.351371+0000","last_peered":"2023-10-16T15:54:40.351371+0000","last_clean":"2023-10-16T15:54:40.351371+0000","last_became_active":"2023-10-16T15:54:39.203058+0000","last_became_peered":"2023-10-16T15:54:39.203058+0000","last_unstale":"2023-10-16T15:54:40.351371+0000","last_undegraded":"2023-10-16T15:54:40.351371+0000","last_fullsized":"2023-10-16T15:54:40.351371+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:37.020836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.353460+0000","last_change":"2023-10-16T15:54:38.195221+0000","last_active":"2023-10-16T15:54:40.353460+0000","last_peered":"2023-10-16T15:54:40.353460+0000","last_clean":"2023-10-16T15:54:40.353460+0000","last_became_active":"2023-10-16T15:54:38.194937+0000","last_became_peered":"2023-10-16T15:54:38.194937+0000","last_unstale":"2023-10-16T15:54:40.353460+0000","last_undegraded":"2023-10-16T15:54:40.353460+0000","last_fullsized":"2023-10-16T15:54:40.353460+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:41.726203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.172653+0000","last_change":"2023-10-16T15:54:39.207259+0000","last_active":"2023-10-16T15:54:40.172653+0000","last_peered":"2023-10-16T15:54:40.172653+0000","last_clean":"2023-10-16T15:54:40.172653+0000","last_became_active":"2023-10-16T15:54:39.206912+0000","last_became_peered":"2023-10-16T15:54:39.206912+0000","last_unstale":"2023-10-16T15:54:40.172653+0000","last_undegraded":"2023-10-16T15:54:40.172653+0000","last_fullsized":"2023-10-16T15:54:40.172653+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:53.691146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.352887+0000","last_change":"2023-10-16T15:54:38.200919+0000","last_active":"2023-10-16T15:54:40.352887+0000","last_peered":"2023-10-16T15:54:40.352887+0000","last_clean":"2023-10-16T15:54:40.352887+0000","last_became_active":"2023-10-16T15:54:38.200141+0000","last_became_peered":"2023-10-16T15:54:38.200141+0000","last_unstale":"2023-10-16T15:54:40.352887+0000","last_undegraded":"2023-10-16T15:54:40.352887+0000","last_fullsized":"2023-10-16T15:54:40.352887+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:40.432210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.178560+0000","last_change":"2023-10-16T15:54:39.208421+0000","last_active":"2023-10-16T15:54:40.178560+0000","last_peered":"2023-10-16T15:54:40.178560+0000","last_clean":"2023-10-16T15:54:40.178560+0000","last_became_active":"2023-10-16T15:54:39.207372+0000","last_became_peered":"2023-10-16T15:54:39.207372+0000","last_unstale":"2023-10-16T15:54:40.178560+0000","last_undegraded":"2023-10-16T15:54:40.178560+0000","last_fullsized":"2023-10-16T15:54:40.178560+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:11:16.655726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"245'4","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-16T15:54:40.356391+0000","last_change":"2023-10-16T15:54:38.200929+0000","last_active":"2023-10-16T15:54:40.356391+0000","last_peered":"2023-10-16T15:54:40.356391+0000","last_clean":"2023-10-16T15:54:40.356391+0000","last_became_active":"2023-10-16T15:54:38.200195+0000","last_became_peered":"2023-10-16T15:54:38.200195+0000","last_unstale":"2023-10-16T15:54:40.356391+0000","last_undegraded":"2023-10-16T15:54:40.356391+0000","last_fullsized":"2023-10-16T15:54:40.356391+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:20.533246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"231'373","reported_seq":1346,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T15:54:39.175655+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:54:39.175655+0000","last_peered":"2023-10-16T15:54:39.175655+0000","last_clean":"2023-10-16T15:54:39.175655+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:54:39.175655+0000","last_undegraded":"2023-10-16T15:54:39.175655+0000","last_fullsized":"2023-10-16T15:54:39.175655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576016,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65312,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306528,"statfs":{"total":96636764160,"available":96569884672,"internally_reserved":0,"allocated":3682304,"data_stored":2110235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.131,"15min":1.181},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.554,"5min":1.794,"15min":1.8879999999999999},"last":1.444},{"interface":"front","average":{"1min":1.056,"5min":1.1180000000000001,"15min":1.1519999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":1.149}]},{"osd":1,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.1140000000000001,"15min":1.1719999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6519999999999999,"5min":1.76,"15min":1.889},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.29,"15min":1.355},"min":{"1min":0.71399999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6919999999999999,"5min":1.8919999999999999,"15min":2.0569999999999999},"last":1.2050000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3460000000000001,"15min":1.339},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.661,"5min":2.0470000000000002,"15min":2.1269999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.276,"15min":1.3080000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5900000000000001,"5min":1.909,"15min":1.95},"last":1.2509999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.3160000000000001,"15min":1.349},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.746,"5min":1.843,"15min":1.869},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.405,"15min":1.4430000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.7749999999999999,"5min":2.1680000000000001,"15min":2.1779999999999999},"last":1.3680000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3879999999999999,"15min":1.383},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.9370000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3180000000000001,"15min":1.341},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.655,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.3220000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.4690000000000001,"15min":1.464},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.929,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.506,"15min":1.5489999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.839,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.619}]},{"osd":6,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.454,"15min":1.472},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5640000000000001,"15min":1.5780000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.988,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.5549999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48092,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323748,"statfs":{"total":96636764160,"available":96587517952,"internally_reserved":0,"allocated":3092480,"data_stored":1520401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.25,"15min":1.214},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.258,"15min":1.2609999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70999999999999996},"max":{"1min":1.5669999999999999,"5min":1.905,"15min":1.905},"last":0.98499999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3140000000000001,"15min":1.296},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.72199999999999998},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.516},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.3049999999999999,"15min":1.3200000000000001},"min":{"1min":0.747,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.859,"15min":1.8740000000000001},"last":1.22}]},{"osd":2,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2330000000000001,"15min":1.238},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.831,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.86299999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.905,"5min":1.905,"15min":2.0339999999999998},"last":1.4550000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3680000000000001,"15min":1.3919999999999999},"min":{"1min":0.93000000000000005,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.107},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3149999999999999,"15min":1.3340000000000001},"min":{"1min":0.91000000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.0489999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.409,"15min":1.337},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3359999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.617},{"interface":"front","average":{"1min":1.518,"5min":1.454,"15min":1.4059999999999999},"min":{"1min":1.2,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.335}]},{"osd":5,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.369,"15min":1.3480000000000001},"min":{"1min":1.022,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":2.016},"last":1.163},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6020000000000001,"15min":1.534},"min":{"1min":1.194,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.681}]},{"osd":7,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.635,"15min":1.599},"min":{"1min":1.274,"5min":0.627,"15min":0.627},"max":{"1min":2.0110000000000001,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.667,"5min":1.6319999999999999,"15min":1.595},"min":{"1min":1.2849999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.028,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.3740000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30052,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341788,"statfs":{"total":96636764160,"available":96605990912,"internally_reserved":0,"allocated":3096576,"data_stored":1526125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1240000000000001,"15min":1.163},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6850000000000001,"5min":1.8069999999999999,"15min":1.821},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.2689999999999999,"15min":1.2430000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.53,"5min":2.738,"15min":2.738},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7669999999999999,"5min":1.7849999999999999,"15min":1.8680000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2629999999999999,"15min":1.302},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7270000000000001,"5min":1.8160000000000001,"15min":1.9339999999999999},"last":1.3100000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.238,"15min":1.2529999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.577,"5min":1.784,"15min":1.9219999999999999},"last":1.754},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.399,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7190000000000001,"5min":2.093,"15min":2.2160000000000002},"last":1.5}]},{"osd":3,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.343,"15min":1.3400000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9039999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.641},{"interface":"front","average":{"1min":1.169,"5min":1.3400000000000001,"15min":1.3799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.7130000000000001,"5min":1.75,"15min":2.1589999999999998},"last":1.966}]},{"osd":4,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.367,"15min":1.347},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.5069999999999999,"15min":1.5189999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.804,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.1579999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.4430000000000001,"15min":1.385},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8660000000000001,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.599},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":1.431},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0230000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.365}]},{"osd":7,"last update":"Mon Oct 16 15:53:42 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.538,"15min":1.5369999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.847,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.873},{"interface":"front","average":{"1min":1.516,"5min":1.639,"15min":1.633},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.962,"5min":3.109,"15min":3.109},"last":1.7}]}]},{"osd":4,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52268,"kb_used_data":3032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319572,"statfs":{"total":96636764160,"available":96583241728,"internally_reserved":0,"allocated":3104768,"data_stored":1521501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1919999999999999,"15min":1.202},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.52700000000000002},"max":{"1min":1.861,"5min":1.883,"15min":1.883},"last":1.319},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1890000000000001,"15min":1.198},"min":{"1min":0.78800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.875,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.0349999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.2370000000000001},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.629},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.88100000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.952,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.405}]},{"osd":2,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.413,"15min":1.4570000000000001},"min":{"1min":0.99399999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.6240000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.462},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4910000000000001,"15min":1.4990000000000001},"min":{"1min":1.0900000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9099999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.575}]},{"osd":3,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.425,"15min":1.391},"min":{"1min":0.84699999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.1099999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.391,"15min":1.3919999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.6220000000000001,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.532}]},{"osd":5,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.377,"15min":1.347},"min":{"1min":1.046,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.8899999999999999,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.4850000000000001,"15min":1.4430000000000001},"min":{"1min":1.177,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.004,"5min":2.161,"15min":2.4489999999999998},"last":1.4319999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3999999999999999,"15min":1.4039999999999999},"min":{"1min":0.94099999999999995,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":1.8300000000000001,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.256},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5529999999999999,"15min":1.528},"min":{"1min":1.198,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.0049999999999999,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.2110000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:53:46 2023","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.625,"15min":1.5880000000000001},"min":{"1min":1.139,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.2200000000000002,"5min":2.399,"15min":2.399},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.6180000000000001,"15min":1.6040000000000001},"min":{"1min":1.1299999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0449999999999999,"5min":2.2599999999999998,"15min":2.347},"last":1.373}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56412,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315428,"statfs":{"total":96636764160,"available":96578998272,"internally_reserved":0,"allocated":3088384,"data_stored":1520379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.94199999999999995,"15min":0.97099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.268,"5min":1.4950000000000001,"15min":2.0430000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0389999999999999,"15min":1.0660000000000001},"min":{"1min":0.44,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6850000000000001,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":1.0049999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.97099999999999997,"15min":1.0269999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.504,"5min":1.6679999999999999,"15min":2.589},"last":1.105},{"interface":"front","average":{"1min":1.071,"5min":1.1719999999999999,"15min":1.218},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.7210000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.3580000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.3759999999999999,"15min":1.4239999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.829,"5min":1.885,"15min":2.302},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.4430000000000001,"15min":1.508},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.768,"5min":1.9530000000000001,"15min":2.754},"last":0.90600000000000003}]},{"osd":4,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.3009999999999999,"15min":1.3480000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7150000000000001,"5min":2.0110000000000001,"15min":2.5390000000000001},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.3660000000000001,"15min":1.421},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.276}]},{"osd":5,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.542,"15min":1.5669999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0590000000000002,"5min":2.2909999999999999,"15min":2.5009999999999999},"last":1.413},{"interface":"front","average":{"1min":1.361,"5min":1.524,"15min":1.573},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.0030000000000001,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.4890000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.4930000000000001,"15min":1.504},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.931,"5min":2.173,"15min":2.2090000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.5920000000000001,"15min":1.6060000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.1539999999999999,"5min":2.5019999999999998,"15min":2.7669999999999999},"last":1.573}]},{"osd":7,"last update":"Mon Oct 16 15:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.6419999999999999,"15min":1.6879999999999999},"min":{"1min":0.97499999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0899999999999999,"5min":2.214,"15min":2.8780000000000001},"last":1.244},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.617,"15min":1.6379999999999999},"min":{"1min":0.94399999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.1120000000000001,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.4470000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576016,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56416,"kb_used_data":3028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315424,"statfs":{"total":96636764160,"available":96578994176,"internally_reserved":0,"allocated":3100672,"data_stored":1526215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.91100000000000003,"15min":0.94999999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.4970000000000001,"5min":1.726,"15min":1.76},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.103},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.804,"5min":1.804,"15min":1.8999999999999999},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.036,"15min":1.0529999999999999},"min":{"1min":0.55500000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.272,"5min":1.974,"15min":1.974},"last":1.252},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1930000000000001,"15min":1.2450000000000001},"min":{"1min":0.503,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.7010000000000001,"5min":1.879,"15min":2.0779999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.242,"15min":1.294},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.6819999999999999,"5min":1.9139999999999999,"15min":2.1509999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.361,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.9139999999999999,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.3089999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4890000000000001,"15min":1.488},"min":{"1min":0.88900000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.946,"5min":2.335,"15min":2.335},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.349,"15min":1.4179999999999999},"min":{"1min":0.74299999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.802,"5min":1.915,"15min":2.1589999999999998},"last":1.3540000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.431,"15min":1.4850000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0640000000000001,"5min":2.657,"15min":2.657},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.599,"15min":1.6259999999999999},"min":{"1min":0.95399999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.5800000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.476,"15min":1.5569999999999999},"min":{"1min":0.81200000000000006,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.9510000000000001,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.591,"15min":1.625},"min":{"1min":0.79800000000000004,"5min":0.753,"15min":0.753},"max":{"1min":2.1509999999999998,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.385}]},{"osd":7,"last update":"Mon Oct 16 15:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.601,"15min":1.6619999999999999},"min":{"1min":0.97799999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.788},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.677,"15min":1.7330000000000001},"min":{"1min":0.92800000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2069999999999999,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.7250000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":3117056,"data_stored":1527861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94399999999999995,"15min":0.94899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.452,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":0.75},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0429999999999999,"15min":1.1299999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6619999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.4279999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98699999999999999,"15min":1.0349999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.4179999999999999,"5min":1.6299999999999999,"15min":2.286},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.109,"5min":1.1919999999999999,"15min":1.2270000000000001},"min":{"1min":0.59199999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.657,"5min":1.7929999999999999,"15min":1.853},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3220000000000001,"15min":1.3680000000000001},"min":{"1min":0.69699999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7290000000000001,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.2529999999999999,"15min":1.3129999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8859999999999999,"5min":2.1629999999999998,"15min":2.7839999999999998},"last":0.83299999999999996}]},{"osd":4,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3620000000000001,"15min":1.381},"min":{"1min":0.79300000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.2959999999999998,"5min":2.327,"15min":2.3809999999999998},"last":1.246},{"interface":"front","average":{"1min":1.409,"5min":1.512,"15min":1.524},"min":{"1min":0.92000000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.081,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.105}]},{"osd":5,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.49,"15min":1.5269999999999999},"min":{"1min":0.72499999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0539999999999998,"5min":2.456,"15min":2.456},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4930000000000001,"15min":1.502},"min":{"1min":1.0069999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.093,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.1859999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.647,"15min":1.655},"min":{"1min":0.97299999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.218,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.367},{"interface":"front","average":{"1min":1.542,"5min":1.6160000000000001,"15min":1.643},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.5059999999999998,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.7030000000000001,"15min":1.744},"min":{"1min":0.80300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.621,"5min":2.734,"15min":3.2090000000000001},"last":1.302},{"interface":"front","average":{"1min":1.577,"5min":1.647,"15min":1.655},"min":{"1min":0.90100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.452,"5min":2.452,"15min":3.1539999999999999},"last":1.028}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48668,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323172,"statfs":{"total":96636764160,"available":96586928128,"internally_reserved":0,"allocated":3682304,"data_stored":2110199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3129999999999999,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.1879999999999999,"15min":1.2270000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.5589999999999999,"5min":2.036,"15min":2.036},"last":1.127}]},{"osd":2,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.226,"15min":1.274},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.486,"5min":1.913,"15min":2.0259999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.046,"5min":1.169,"15min":1.226},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.6850000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.628}]},{"osd":3,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.006,"15min":1.0329999999999999},"min":{"1min":0.54700000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.4370000000000001,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.113,"5min":1.244,"15min":1.27},"min":{"1min":0.57599999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.677,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":0.99399999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.3200000000000001,"15min":1.341},"min":{"1min":0.749,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.575,"5min":1.8520000000000001,"15min":2.137},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4450000000000001,"15min":1.51},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.901,"5min":1.99,"15min":2.452},"last":1.24}]},{"osd":5,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.4690000000000001,"15min":1.4890000000000001},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.1059999999999999,"5min":2.137,"15min":2.137},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.365,"15min":1.419},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.6879999999999999,"5min":2.0859999999999999,"15min":2.282},"last":0.88500000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.5980000000000001,"15min":1.6399999999999999},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.84499999999999997},"max":{"1min":1.974,"5min":2.2269999999999999,"15min":2.7010000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5640000000000001,"15min":1.5780000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.085,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 15:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.6719999999999999,"15min":1.6910000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2200000000000002,"5min":2.339,"15min":2.339},"last":1.085},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6100000000000001,"15min":1.611},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.8540000000000001,"5min":2.1739999999999999,"15min":2.274},"last":0.96099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:54:44.296 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:54:44.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T15:54:44.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.550+0000 7f51cd600640 1 Processor -- start 2023-10-16T15:54:44.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.550+0000 7f51cd600640 1 -- start start 2023-10-16T15:54:44.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.551+0000 7f51cd600640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f51c8159f00 0x7f51c80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.551+0000 7f51cd600640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 0x7f51c80b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.551+0000 7f51cd600640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c806f890 con 0x7f51c8159b20 2023-10-16T15:54:44.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.551+0000 7f51cd600640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c8070d30 con 0x7f51c8159f00 2023-10-16T15:54:44.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.551+0000 7f51cd600640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c806c650 con 0x7f51c815d8c0 2023-10-16T15:54:44.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.552+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 0x7f51c80b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.552+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 0x7f51c80b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:56304/0 (socket says 172.21.15.29:56304) 2023-10-16T15:54:44.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.552+0000 7f51c77fe640 1 -- 172.21.15.29:0/2741581480 learned_addr learned my addr 172.21.15.29:0/2741581480 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:44.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.552+0000 7f51c6ffd640 1 --1- 172.21.15.29:0/2741581480 >> v1:172.21.15.170:6789/0 conn(0x7f51c8159b20 0x7f51c806bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:35856/0 (socket says 172.21.15.29:35856) 2023-10-16T15:54:44.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.552+0000 7f51c67fc640 1 --2- 172.21.15.29:0/2741581480 >> v2:172.21.15.170:3300/0 conn(0x7f51c8159f00 0x7f51c80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.553+0000 7f51c67fc640 1 -- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 msgr2=0x7f51c80b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:44.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.553+0000 7f51c67fc640 1 --2- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 0x7f51c80b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.553+0000 7f51c67fc640 1 -- 172.21.15.29:0/2741581480 >> v1:172.21.15.170:6789/0 conn(0x7f51c8159b20 legacy=0x7f51c806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.553+0000 7f51c67fc640 1 -- 172.21.15.29:0/2741581480 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c806c930 con 0x7f51c8159f00 2023-10-16T15:54:44.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.553+0000 7f51c77fe640 1 --2- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 0x7f51c80b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:44.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.554+0000 7f51c67fc640 1 --2- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b0410 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0097a0 tx=0x7f51bc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4abbdd9fbcf2bb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:44.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.554+0000 7f51c5ffb640 1 -- 172.21.15.29:0/2741581480 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc02d040 con 0x7f51c8159f00 2023-10-16T15:54:44.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51c5ffb640 1 -- 172.21.15.29:0/2741581480 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc031810 con 0x7f51c8159f00 2023-10-16T15:54:44.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51c5ffb640 1 -- 172.21.15.29:0/2741581480 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc031ba0 con 0x7f51c8159f00 2023-10-16T15:54:44.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 -- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 msgr2=0x7f51c80b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 --2- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b0410 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0097a0 tx=0x7f51bc02b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 -- 172.21.15.29:0/2741581480 shutdown_connections 2023-10-16T15:54:44.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 --2- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c815d8c0 0x7f51c80b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 --2- 172.21.15.29:0/2741581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b0410 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 -- 172.21.15.29:0/2741581480 >> 172.21.15.29:0/2741581480 conn(0x7f51c80cb410 msgr2=0x7f51c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:44.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 -- 172.21.15.29:0/2741581480 shutdown_connections 2023-10-16T15:54:44.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.555+0000 7f51cd600640 1 -- 172.21.15.29:0/2741581480 wait complete. 2023-10-16T15:54:44.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.557+0000 7f51cd600640 1 Processor -- start 2023-10-16T15:54:44.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.557+0000 7f51cd600640 1 -- start start 2023-10-16T15:54:44.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.558+0000 7f51cd600640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c8159b20 0x7f51c8151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.558+0000 7f51cd600640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.558+0000 7f51cd600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 0x7f51c80b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.558+0000 7f51cd600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80b3bd0 con 0x7f51c815d8c0 2023-10-16T15:54:44.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.558+0000 7f51cd600640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80b26c0 con 0x7f51c8159f00 2023-10-16T15:54:44.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.558+0000 7f51cd600640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80b29a0 con 0x7f51c8159b20 2023-10-16T15:54:44.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 0x7f51c80b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 0x7f51c80b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47572/0 (socket says 172.21.15.29:47572) 2023-10-16T15:54:44.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 -- 172.21.15.29:0/2681042507 learned_addr learned my addr 172.21.15.29:0/2681042507 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:44.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c6ffd640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c8159b20 0x7f51c8151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c67fc640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 -- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c8159b20 msgr2=0x7f51c8151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c8159b20 0x7f51c8151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 -- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 msgr2=0x7f51c80b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 -- 172.21.15.29:0/2681042507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c00094d0 con 0x7f51c815d8c0 2023-10-16T15:54:44.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.559+0000 7f51c77fe640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 0x7f51c80b3690 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f51c0006e00 tx=0x7f51c000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:44.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.560+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0004030 con 0x7f51c815d8c0 2023-10-16T15:54:44.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.560+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c0004190 con 0x7f51c815d8c0 2023-10-16T15:54:44.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.560+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c000c910 con 0x7f51c815d8c0 2023-10-16T15:54:44.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.560+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51bc009400 con 0x7f51c815d8c0 2023-10-16T15:54:44.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.560+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c80b2f60 con 0x7f51c815d8c0 2023-10-16T15:54:44.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.561+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c80b3150 con 0x7f51c815d8c0 2023-10-16T15:54:44.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.562+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f51c000ca70 con 0x7f51c815d8c0 2023-10-16T15:54:44.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.563+0000 7f51af7fe640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f51b406b0e0 0x7f51b406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:44.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.563+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f51c0089a40 con 0x7f51c815d8c0 2023-10-16T15:54:44.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.564+0000 7f51c6ffd640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f51b406b0e0 0x7f51b406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:44.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.565+0000 7f51c6ffd640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f51b406b0e0 0x7f51b406d5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51b80012e0 tx=0x7f51b8005b70 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.568+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51c008be60 con 0x7f51c815d8c0 2023-10-16T15:54:44.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.834+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f51c804e160 con 0x7f51c815d8c0 2023-10-16T15:54:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.838+0000 7f51af7fe640 1 -- 172.21.15.29:0/2681042507 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+147 (secure 0 0 0) 0x7f51c0058200 con 0x7f51c815d8c0 2023-10-16T15:54:44.841 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T15:54:44.841 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T15:54:44.841 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T15:54:44.841 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T15:54:44.841 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T15:54:44.842 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T15:54:44.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f51b406b0e0 msgr2=0x7f51b406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f51b406b0e0 0x7f51b406d5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51b80012e0 tx=0x7f51b8005b70 comp rx=0 tx=0).stop 2023-10-16T15:54:44.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 msgr2=0x7f51c80b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:44.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 0x7f51c80b3690 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f51c0006e00 tx=0x7f51c000ed40 comp rx=0 tx=0).stop 2023-10-16T15:54:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 shutdown_connections 2023-10-16T15:54:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f51b406b0e0 0x7f51b406d5a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51c815d8c0 0x7f51c80b3690 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f51c8159f00 0x7f51c80b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 --2- 172.21.15.29:0/2681042507 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f51c8159b20 0x7f51c8151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 >> 172.21.15.29:0/2681042507 conn(0x7f51c80cb410 msgr2=0x7f51c80c9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 shutdown_connections 2023-10-16T15:54:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:44.841+0000 7f51cd600640 1 -- 172.21.15.29:0/2681042507 wait complete. 2023-10-16T15:54:44.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:45.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.078+0000 7f6d0affb640 1 Processor -- start 2023-10-16T15:54:45.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.078+0000 7f6d0affb640 1 -- start start 2023-10-16T15:54:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.079+0000 7f6d0affb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6d04153740 0x7f6d0415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.079+0000 7f6d0affb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6d040ccb00 0x7f6d040ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.079+0000 7f6d0affb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d040ad4d0 0x7f6d040ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.079+0000 7f6d0affb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d040cd4a0 con 0x7f6d04153740 2023-10-16T15:54:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.079+0000 7f6d0affb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0415ac60 con 0x7f6d040ccb00 2023-10-16T15:54:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.079+0000 7f6d0affb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04070030 con 0x7f6d040ad4d0 2023-10-16T15:54:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6d04153740 0x7f6d0415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6d04153740 0x7f6d0415a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47586/0 (socket says 172.21.15.29:47586) 2023-10-16T15:54:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 -- 172.21.15.29:0/2646824275 learned_addr learned my addr 172.21.15.29:0/2646824275 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d0a7fa640 1 --2- 172.21.15.29:0/2646824275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d040ad4d0 0x7f6d040ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d097f8640 1 --2- 172.21.15.29:0/2646824275 >> v2:172.21.15.170:3300/0 conn(0x7f6d040ccb00 0x7f6d040ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 -- 172.21.15.29:0/2646824275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d040ad4d0 msgr2=0x7f6d040ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/2646824275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d040ad4d0 0x7f6d040ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 -- 172.21.15.29:0/2646824275 >> v2:172.21.15.170:3300/0 conn(0x7f6d040ccb00 msgr2=0x7f6d040ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/2646824275 >> v2:172.21.15.170:3300/0 conn(0x7f6d040ccb00 0x7f6d040ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d09ff9640 1 -- 172.21.15.29:0/2646824275 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d04070310 con 0x7f6d04153740 2023-10-16T15:54:45.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.080+0000 7f6d097f8640 1 --2- 172.21.15.29:0/2646824275 >> v2:172.21.15.170:3300/0 conn(0x7f6d040ccb00 0x7f6d040ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:45.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.081+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/2646824275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04153740 0x7f6d0415a0e0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6d0406d6e0 tx=0x7f6d0002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a6ecd5d7740be0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.082+0000 7f6d08ff7640 1 -- 172.21.15.29:0/2646824275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0002ee90 con 0x7f6d04153740 2023-10-16T15:54:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.082+0000 7f6d08ff7640 1 -- 172.21.15.29:0/2646824275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d00036440 con 0x7f6d04153740 2023-10-16T15:54:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.082+0000 7f6d08ff7640 1 -- 172.21.15.29:0/2646824275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d000367d0 con 0x7f6d04153740 2023-10-16T15:54:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.082+0000 7f6d0affb640 1 -- 172.21.15.29:0/2646824275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04153740 msgr2=0x7f6d0415a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.082+0000 7f6d0affb640 1 --2- 172.21.15.29:0/2646824275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04153740 0x7f6d0415a0e0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6d0406d6e0 tx=0x7f6d0002e0e0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 -- 172.21.15.29:0/2646824275 shutdown_connections 2023-10-16T15:54:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 --2- 172.21.15.29:0/2646824275 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d040ad4d0 0x7f6d040ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 --2- 172.21.15.29:0/2646824275 >> v2:172.21.15.170:3300/0 conn(0x7f6d040ccb00 0x7f6d040ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 --2- 172.21.15.29:0/2646824275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04153740 0x7f6d0415a0e0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 -- 172.21.15.29:0/2646824275 >> 172.21.15.29:0/2646824275 conn(0x7f6d040ce6a0 msgr2=0x7f6d040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 -- 172.21.15.29:0/2646824275 shutdown_connections 2023-10-16T15:54:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.083+0000 7f6d0affb640 1 -- 172.21.15.29:0/2646824275 wait complete. 2023-10-16T15:54:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.084+0000 7f6d0affb640 1 Processor -- start 2023-10-16T15:54:45.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.085+0000 7f6d0affb640 1 -- start start 2023-10-16T15:54:45.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.085+0000 7f6d0affb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d040ad4d0 0x7f6d040b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.085+0000 7f6d0affb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 0x7f6d0414c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.085+0000 7f6d0affb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d04153740 0x7f6d0414c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d0affb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040b2d50 con 0x7f6d040ccb00 2023-10-16T15:54:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d0affb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040c2400 con 0x7f6d040ad4d0 2023-10-16T15:54:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d0affb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040b2210 con 0x7f6d04153740 2023-10-16T15:54:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 0x7f6d0414c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 0x7f6d0414c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47590/0 (socket says 172.21.15.29:47590) 2023-10-16T15:54:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 -- 172.21.15.29:0/1526032494 learned_addr learned my addr 172.21.15.29:0/1526032494 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d040ad4d0 0x7f6d040b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d0a7fa640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d04153740 0x7f6d0414c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 -- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d04153740 msgr2=0x7f6d0414c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d04153740 0x7f6d0414c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 -- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d040ad4d0 msgr2=0x7f6d040b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d040ad4d0 0x7f6d040b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d097f8640 1 -- 172.21.15.29:0/1526032494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0000b770 con 0x7f6d040ccb00 2023-10-16T15:54:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.086+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d040ad4d0 0x7f6d040b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:45.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.087+0000 7f6d097f8640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 0x7f6d0414c0a0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c7f0 tx=0x7f6cf400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.087+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4007a30 con 0x7f6d040ccb00 2023-10-16T15:54:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.087+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf4007b90 con 0x7f6d040ccb00 2023-10-16T15:54:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.087+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400ce40 con 0x7f6d040ccb00 2023-10-16T15:54:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.087+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d040b24b0 con 0x7f6d040ccb00 2023-10-16T15:54:45.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.087+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d040b2a70 con 0x7f6d040ccb00 2023-10-16T15:54:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.088+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d040c2570 con 0x7f6d040ccb00 2023-10-16T15:54:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.089+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6cf40113f0 con 0x7f6d040ccb00 2023-10-16T15:54:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.090+0000 7f6cfa7fc640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6ce806b0b0 0x7f6ce806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.090+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6cf4089d10 con 0x7f6d040ccb00 2023-10-16T15:54:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.091+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6ce806b0b0 0x7f6ce806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.092+0000 7f6d09ff9640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6ce806b0b0 0x7f6ce806d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d00038000 tx=0x7f6d00008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:45.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.095+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6cf4054930 con 0x7f6d040ccb00 2023-10-16T15:54:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.327+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d0404e9b0 con 0x7f6d040ccb00 2023-10-16T15:54:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.328+0000 7f6cfa7fc640 1 -- 172.21.15.29:0/1526032494 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4930 (secure 0 0 0) 0x7f6cf401f090 con 0x7f6d040ccb00 2023-10-16T15:54:45.331 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:45.331 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":11613,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T15:54:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 124 2023-10-16T15:54:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.330+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6ce806b0b0 msgr2=0x7f6ce806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.330+0000 7f6d0affb640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6ce806b0b0 0x7f6ce806d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d00038000 tx=0x7f6d00008040 comp rx=0 tx=0).stop 2023-10-16T15:54:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.330+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 msgr2=0x7f6d0414c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.330+0000 7f6d0affb640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 0x7f6d0414c0a0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c7f0 tx=0x7f6cf400cbf0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 shutdown_connections 2023-10-16T15:54:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6ce806b0b0 0x7f6ce806d570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d040ccb00 0x7f6d0414c0a0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6d04153740 0x7f6d0414c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 --2- 172.21.15.29:0/1526032494 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6d040ad4d0 0x7f6d040b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 >> 172.21.15.29:0/1526032494 conn(0x7f6d040ce6a0 msgr2=0x7f6d0414e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 shutdown_connections 2023-10-16T15:54:45.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.331+0000 7f6d0affb640 1 -- 172.21.15.29:0/1526032494 wait complete. 2023-10-16T15:54:45.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:54:38.641105+0000', 'modified': '2023-10-16T15:54:41.783513+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11616}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11616': {'gid': 11616, 'name': 'a', 'rank': 0, 'incarnation': 120, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/3076937560', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 3076937560}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 3076937560}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:54:45.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:54:45.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T15:54:45.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.997+0000 7f1483832640 1 Processor -- start 2023-10-16T15:54:45.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.997+0000 7f1483832640 1 -- start start 2023-10-16T15:54:46.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.998+0000 7f1483832640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f147c159e90 0x7f147c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.998+0000 7f1483832640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f147c06c140 0x7f147c06c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.998+0000 7f1483832640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c06cad0 con 0x7f147c159ab0 2023-10-16T15:54:46.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.998+0000 7f1483832640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147c15d890 con 0x7f147c06c140 2023-10-16T15:54:46.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.998+0000 7f1483832640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147c070360 con 0x7f147c159e90 2023-10-16T15:54:46.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.999+0000 7f1481dcf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f147c06c140 0x7f147c06c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.999+0000 7f1481dcf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f147c06c140 0x7f147c06c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47602/0 (socket says 172.21.15.29:47602) 2023-10-16T15:54:46.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.999+0000 7f1481dcf640 1 -- 172.21.15.29:0/3699270926 learned_addr learned my addr 172.21.15.29:0/3699270926 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:45.999+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3699270926 >> v2:172.21.15.170:3300/0 conn(0x7f147c159e90 0x7f147c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.000+0000 7f1481dcf640 1 -- 172.21.15.29:0/3699270926 >> v2:172.21.15.170:3300/0 conn(0x7f147c159e90 msgr2=0x7f147c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.000+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3699270926 >> v2:172.21.15.170:3300/0 conn(0x7f147c159e90 0x7f147c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.000+0000 7f1481dcf640 1 -- 172.21.15.29:0/3699270926 >> v1:172.21.15.170:6789/0 conn(0x7f147c159ab0 legacy=0x7f147c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.000+0000 7f1481dcf640 1 -- 172.21.15.29:0/3699270926 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0cd1f0 con 0x7f147c06c140 2023-10-16T15:54:46.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.001+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3699270926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c06c590 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f147c06d900 tx=0x7f147402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91bebdb8bc6ef23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.001+0000 7f1473fff640 1 -- 172.21.15.29:0/3699270926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14740317e0 con 0x7f147c06c140 2023-10-16T15:54:46.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.001+0000 7f1473fff640 1 -- 172.21.15.29:0/3699270926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474031940 con 0x7f147c06c140 2023-10-16T15:54:46.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.001+0000 7f1473fff640 1 -- 172.21.15.29:0/3699270926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474031cd0 con 0x7f147c06c140 2023-10-16T15:54:46.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 -- 172.21.15.29:0/3699270926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 msgr2=0x7f147c06c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 --2- 172.21.15.29:0/3699270926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c06c590 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f147c06d900 tx=0x7f147402b880 comp rx=0 tx=0).stop 2023-10-16T15:54:46.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 -- 172.21.15.29:0/3699270926 shutdown_connections 2023-10-16T15:54:46.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 --2- 172.21.15.29:0/3699270926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c06c590 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 --2- 172.21.15.29:0/3699270926 >> v2:172.21.15.170:3300/0 conn(0x7f147c159e90 0x7f147c0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 -- 172.21.15.29:0/3699270926 >> 172.21.15.29:0/3699270926 conn(0x7f147c0ce9b0 msgr2=0x7f147c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 -- 172.21.15.29:0/3699270926 shutdown_connections 2023-10-16T15:54:46.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.002+0000 7f1483832640 1 -- 172.21.15.29:0/3699270926 wait complete. 2023-10-16T15:54:46.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.004+0000 7f1483832640 1 Processor -- start 2023-10-16T15:54:46.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.004+0000 7f1483832640 1 -- start start 2023-10-16T15:54:46.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1483832640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1483832640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f147c159ab0 0x7f147c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1483832640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f147c159e90 0x7f147c0c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1483832640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c151bb0 con 0x7f147c06c140 2023-10-16T15:54:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1483832640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0c5010 con 0x7f147c159e90 2023-10-16T15:54:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1483832640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c1522b0 con 0x7f147c159ab0 2023-10-16T15:54:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f14815ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f14815ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47616/0 (socket says 172.21.15.29:47616) 2023-10-16T15:54:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f14815ce640 1 -- 172.21.15.29:0/3916446385 learned_addr learned my addr 172.21.15.29:0/3916446385 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f147c159e90 0x7f147c0c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f147c159ab0 0x7f147c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.005+0000 7f14815ce640 1 -- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f147c159ab0 msgr2=0x7f147c151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f14815ce640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f147c159ab0 0x7f147c151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f14815ce640 1 -- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f147c159e90 msgr2=0x7f147c0c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f14815ce640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f147c159e90 0x7f147c0c68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f14815ce640 1 -- 172.21.15.29:0/3916446385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14740094d0 con 0x7f147c06c140 2023-10-16T15:54:46.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f14815ce640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c151670 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f147800c870 tx=0x7f147800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f1471ffb640 1 -- 172.21.15.29:0/3916446385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478003b70 con 0x7f147c06c140 2023-10-16T15:54:46.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f1471ffb640 1 -- 172.21.15.29:0/3916446385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1478003cf0 con 0x7f147c06c140 2023-10-16T15:54:46.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c0c46e0 con 0x7f147c06c140 2023-10-16T15:54:46.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f1471ffb640 1 -- 172.21.15.29:0/3916446385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800aa90 con 0x7f147c06c140 2023-10-16T15:54:46.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.006+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c0c4ca0 con 0x7f147c06c140 2023-10-16T15:54:46.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.008+0000 7f1471ffb640 1 -- 172.21.15.29:0/3916446385 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f147800abf0 con 0x7f147c06c140 2023-10-16T15:54:46.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.009+0000 7f1471ffb640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f146006b0b0 0x7f146006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.010+0000 7f1471ffb640 1 -- 172.21.15.29:0/3916446385 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1478089850 con 0x7f147c06c140 2023-10-16T15:54:46.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.010+0000 7f14863c4740 1 Processor -- start 2023-10-16T15:54:46.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.010+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f146006b0b0 0x7f146006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.010+0000 7f14863c4740 1 -- start start 2023-10-16T15:54:46.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3ba72e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3baa0360 0x55ad3baa2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f14863c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 0x55ad3baa3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f14863c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b9937a0 con 0x55ad3ba733c0 2023-10-16T15:54:46.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b9e5740 con 0x55ad3ba9ff80 2023-10-16T15:54:46.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b9e2e30 con 0x55ad3baa0360 2023-10-16T15:54:46.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f146006b0b0 0x7f146006d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f147c0b4780 tx=0x7f146c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f1481dcf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 0x55ad3baa3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f1481dcf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 0x55ad3baa3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47618/0 (socket says 172.21.15.29:47618) 2023-10-16T15:54:46.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.011+0000 7f1481dcf640 1 -- 172.21.15.29:0/3333107678 learned_addr learned my addr 172.21.15.29:0/3333107678 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f14815ce640 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3ba72e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3baa0360 0x55ad3baa2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1481dcf640 1 -- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3baa0360 msgr2=0x55ad3baa2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3baa0360 0x55ad3baa2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1481dcf640 1 -- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 msgr2=0x55ad3ba72e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3ba72e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1481dcf640 1 -- 172.21.15.29:0/3333107678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad3b9a2910 con 0x55ad3ba733c0 2023-10-16T15:54:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.012+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 0x55ad3baa3250 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f147402b8c0 tx=0x7f1474008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97a305c0414d038 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f1466ffd640 1 -- 172.21.15.29:0/3333107678 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14740317a0 con 0x55ad3ba733c0 2023-10-16T15:54:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f1466ffd640 1 -- 172.21.15.29:0/3333107678 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474031ab0 con 0x55ad3ba733c0 2023-10-16T15:54:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f1466ffd640 1 -- 172.21.15.29:0/3333107678 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474031e40 con 0x55ad3ba733c0 2023-10-16T15:54:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 -- 172.21.15.29:0/3333107678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 msgr2=0x55ad3baa3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 0x55ad3baa3250 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f147402b8c0 tx=0x7f1474008030 comp rx=0 tx=0).stop 2023-10-16T15:54:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 -- 172.21.15.29:0/3333107678 shutdown_connections 2023-10-16T15:54:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba733c0 0x55ad3baa3250 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3baa0360 0x55ad3baa2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 --2- 172.21.15.29:0/3333107678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3ba72e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 -- 172.21.15.29:0/3333107678 >> 172.21.15.29:0/3333107678 conn(0x55ad3ba75190 msgr2=0x55ad3babdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 -- 172.21.15.29:0/3333107678 shutdown_connections 2023-10-16T15:54:46.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.013+0000 7f14863c4740 1 -- 172.21.15.29:0/3333107678 wait complete. 2023-10-16T15:54:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.014+0000 7f14863c4740 1 -- start start 2023-10-16T15:54:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.014+0000 7f14863c4740 1 Processor -- start 2023-10-16T15:54:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.015+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3baa6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.015+0000 7f14863c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f14863c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b9de720 con 0x55ad3baa0360 2023-10-16T15:54:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b97c970 con 0x55ad3ba9ff80 2023-10-16T15:54:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b97c610 con 0x55ad3ba9de00 2023-10-16T15:54:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f14863c4740 10 client.0 _ll_register_callbacks cb 0x55ad3b99e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:54:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47624/0 (socket says 172.21.15.29:47624) 2023-10-16T15:54:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 -- 172.21.15.29:0/3025260411 learned_addr learned my addr 172.21.15.29:0/3025260411 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1481dcf640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f14815ce640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3baa6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 -- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 msgr2=0x55ad3bab7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 -- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 msgr2=0x55ad3baa6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3baa6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.016+0000 7f1480dcd640 1 -- 172.21.15.29:0/3025260411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c0cd1f0 con 0x55ad3baa0360 2023-10-16T15:54:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6c20 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f146c00ef80 tx=0x7f146c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b68103dec7c823 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f145bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:54:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f145bfff640 1 -- 172.21.15.29:0/3025260411 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c015e90 con 0x55ad3baa0360 2023-10-16T15:54:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad3ba05160 con 0x55ad3baa0360 2023-10-16T15:54:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f145bfff640 1 -- 172.21.15.29:0/3025260411 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f146c020410 con 0x55ad3baa0360 2023-10-16T15:54:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f145bfff640 1 -- 172.21.15.29:0/3025260411 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146c0205d0 con 0x55ad3baa0360 2023-10-16T15:54:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f14863c4740 10 client.11751 fetch_fsmap learned FSMap version 124 2023-10-16T15:54:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.017+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55ad3b9dcda0 con 0x55ad3baa0360 2023-10-16T15:54:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f145bfff640 1 -- 172.21.15.29:0/3025260411 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ad3ba05160 con 0x55ad3baa0360 2023-10-16T15:54:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f145bfff640 1 -- 172.21.15.29:0/3025260411 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 124) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f146c020730 con 0x55ad3baa0360 2023-10-16T15:54:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14863c4740 10 client.11751 fetch_fsmap finished waiting for FSMap version 124 2023-10-16T15:54:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14863c4740 10 client.11751 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T15:54:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14863c4740 20 client.11751 populate_metadata read hostname 'smithi029' 2023-10-16T15:54:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14863c4740 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3bad0f40 0x55ad3baa43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14863c4740 4 client.11751 mds_command: new command op to 11616 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:54:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 --> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad3b9b0cc0 con 0x55ad3bad0f40 2023-10-16T15:54:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.018+0000 7f14815ce640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3bad0f40 0x55ad3baa43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.019+0000 7f14815ce640 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3bad0f40 0x55ad3baa43c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=622bc6a8c7515326 server_cookie=1d70a2c2a5d93538 in_seq=0 out_seq=0 2023-10-16T15:54:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.019+0000 7f145bfff640 10 client.11751 ms_handle_connect on v2:172.21.15.29:6833/3076937560 2023-10-16T15:54:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.019+0000 7f145bfff640 1 -- 172.21.15.29:0/3025260411 <== mds.0 v2:172.21.15.29:6833/3076937560 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ad3b9b0cc0 con 0x55ad3bad0f40 2023-10-16T15:54:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.019+0000 7f145bfff640 10 client.11751 handle_command_reply: tid=0 2023-10-16T15:54:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.019+0000 7f14863c4740 1 client.11751 shutdown 2023-10-16T15:54:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 msgr2=0x55ad3bab6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6c20 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f146c00ef80 tx=0x7f146c004730 comp rx=0 tx=0).stop 2023-10-16T15:54:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 shutdown_connections 2023-10-16T15:54:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3bad0f40 0x55ad3baa43c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6c20 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 --2- 172.21.15.29:0/3025260411 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3baa6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 >> 172.21.15.29:0/3025260411 conn(0x55ad3ba733c0 msgr2=0x55ad3ba76f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f145bfff640 0 client.11751 ms_handle_reset on v2:172.21.15.29:6833/3076937560 2023-10-16T15:54:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 shutdown_connections 2023-10-16T15:54:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 1 -- 172.21.15.29:0/3025260411 wait complete. 2023-10-16T15:54:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.020+0000 7f14863c4740 20 client.11751 trim_cache size 0 max 16384 2023-10-16T15:54:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.037+0000 7f14863c4740 1 Processor -- start 2023-10-16T15:54:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.037+0000 7f14863c4740 1 -- start start 2023-10-16T15:54:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3bab5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f14863c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f14863c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b742f20 con 0x55ad3baa0360 2023-10-16T15:54:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3ba74f20 con 0x55ad3ba9ff80 2023-10-16T15:54:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3ba72aa0 con 0x55ad3ba9de00 2023-10-16T15:54:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f1481dcf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f1481dcf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47626/0 (socket says 172.21.15.29:47626) 2023-10-16T15:54:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f1481dcf640 1 -- 172.21.15.29:0/2241761861 learned_addr learned my addr 172.21.15.29:0/2241761861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f1481dcf640 1 -- 172.21.15.29:0/2241761861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 msgr2=0x55ad3bab4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.038+0000 7f1481dcf640 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f1481dcf640 1 -- 172.21.15.29:0/2241761861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 msgr2=0x55ad3bab5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f1481dcf640 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3bab5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f1481dcf640 1 -- 172.21.15.29:0/2241761861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad3b97bb80 con 0x55ad3baa0360 2023-10-16T15:54:46.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f1481dcf640 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6400 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f1474009960 tx=0x7f1474030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74110e8947604f8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f14677fe640 1 -- 172.21.15.29:0/2241761861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474031c20 con 0x55ad3baa0360 2023-10-16T15:54:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f14677fe640 1 -- 172.21.15.29:0/2241761861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474031da0 con 0x55ad3baa0360 2023-10-16T15:54:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.039+0000 7f14677fe640 1 -- 172.21.15.29:0/2241761861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474031f10 con 0x55ad3baa0360 2023-10-16T15:54:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 -- 172.21.15.29:0/2241761861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 msgr2=0x55ad3bab6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6400 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f1474009960 tx=0x7f1474030000 comp rx=0 tx=0).stop 2023-10-16T15:54:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 -- 172.21.15.29:0/2241761861 shutdown_connections 2023-10-16T15:54:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3baa0360 0x55ad3bab6400 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3ba9ff80 0x55ad3bab5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 --2- 172.21.15.29:0/2241761861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9de00 0x55ad3bab4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 -- 172.21.15.29:0/2241761861 >> 172.21.15.29:0/2241761861 conn(0x55ad3ba733c0 msgr2=0x55ad3ba76a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 -- 172.21.15.29:0/2241761861 shutdown_connections 2023-10-16T15:54:46.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.040+0000 7f14863c4740 1 -- 172.21.15.29:0/2241761861 wait complete. 2023-10-16T15:54:46.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.041+0000 7f14863c4740 1 -- start start 2023-10-16T15:54:46.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.041+0000 7f14863c4740 1 Processor -- start 2023-10-16T15:54:46.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 0x55ad3bb20560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9ff80 0x55ad3bb263c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3baa0360 0x55ad3bb26900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3bb27260 con 0x55ad3ba9de00 2023-10-16T15:54:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3bacd780 con 0x55ad3baa0360 2023-10-16T15:54:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3bacd4a0 con 0x55ad3ba9ff80 2023-10-16T15:54:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14863c4740 10 client.0 _ll_register_callbacks cb 0x55ad3ba792e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:54:46.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 0x55ad3bb20560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 0x55ad3bb20560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47632/0 (socket says 172.21.15.29:47632) 2023-10-16T15:54:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 -- 172.21.15.29:0/3670885083 learned_addr learned my addr 172.21.15.29:0/3670885083 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 -- 172.21.15.29:0/3670885083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9ff80 msgr2=0x55ad3bb263c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9ff80 0x55ad3bb263c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 -- 172.21.15.29:0/3670885083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3baa0360 msgr2=0x55ad3bb26900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3baa0360 0x55ad3bb26900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.042+0000 7f14815ce640 1 -- 172.21.15.29:0/3670885083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad3b9a2910 con 0x55ad3ba9de00 2023-10-16T15:54:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f14815ce640 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 0x55ad3bb20560 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f147800a430 tx=0x7f147800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae1a8e55ea8f220 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f145bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:54:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f145bfff640 1 -- 172.21.15.29:0/3670885083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478086070 con 0x55ad3ba9de00 2023-10-16T15:54:46.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f145bfff640 1 -- 172.21.15.29:0/3670885083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f14780604a0 con 0x55ad3ba9de00 2023-10-16T15:54:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1474031ab0 con 0x55ad3ba9de00 2023-10-16T15:54:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f145bfff640 1 -- 172.21.15.29:0/3670885083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1478060010 con 0x55ad3ba9de00 2023-10-16T15:54:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f14863c4740 10 client.11757 fetch_fsmap learned FSMap version 124 2023-10-16T15:54:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55ad3b9a1170 con 0x55ad3ba9de00 2023-10-16T15:54:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.043+0000 7f145bfff640 1 -- 172.21.15.29:0/3670885083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474031ab0 con 0x55ad3ba9de00 2023-10-16T15:54:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f145bfff640 1 -- 172.21.15.29:0/3670885083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 124) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1478019020 con 0x55ad3ba9de00 2023-10-16T15:54:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f14863c4740 10 client.11757 fetch_fsmap finished waiting for FSMap version 124 2023-10-16T15:54:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f14863c4740 10 client.11757 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T15:54:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f14863c4740 20 client.11757 populate_metadata read hostname 'smithi029' 2023-10-16T15:54:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f14863c4740 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3ba77f60 0x55ad3ba96170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f14863c4740 4 client.11757 mds_command: new command op to 11616 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:54:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 --> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ad3b9d7700 con 0x55ad3ba77f60 2023-10-16T15:54:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3ba77f60 0x55ad3ba96170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f1480dcd640 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3ba77f60 0x55ad3ba96170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=120b19d3e56d4771 server_cookie=21d48a930a59ed7e in_seq=0 out_seq=0 2023-10-16T15:54:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.044+0000 7f145bfff640 10 client.11757 ms_handle_connect on v2:172.21.15.29:6833/3076937560 2023-10-16T15:54:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f145bfff640 1 -- 172.21.15.29:0/3670885083 <== mds.0 v2:172.21.15.29:6833/3076937560 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55ad3b9d7700 con 0x55ad3ba77f60 2023-10-16T15:54:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f145bfff640 10 client.11757 handle_command_reply: tid=0 2023-10-16T15:54:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 client.11757 shutdown 2023-10-16T15:54:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 msgr2=0x55ad3bb20560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 0x55ad3bb20560 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f147800a430 tx=0x7f147800a560 comp rx=0 tx=0).stop 2023-10-16T15:54:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 shutdown_connections 2023-10-16T15:54:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:6833/3076937560,v1:172.21.15.29:6835/3076937560] conn(0x55ad3ba77f60 0x55ad3ba96170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ad3baa0360 0x55ad3bb26900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55ad3ba9ff80 0x55ad3bb263c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 --2- 172.21.15.29:0/3670885083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ad3ba9de00 0x55ad3bb20560 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f145bfff640 0 client.11757 ms_handle_reset on v2:172.21.15.29:6833/3076937560 2023-10-16T15:54:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 >> 172.21.15.29:0/3670885083 conn(0x55ad3ba733c0 msgr2=0x55ad3baa6360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 shutdown_connections 2023-10-16T15:54:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 1 -- 172.21.15.29:0/3670885083 wait complete. 2023-10-16T15:54:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.045+0000 7f14863c4740 20 client.11757 trim_cache size 0 max 16384 2023-10-16T15:54:46.058 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:54:46.058 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:54:46.058 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:54:46.059 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 11616, 2023-10-16T15:54:46.059 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:54:46.059 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:54:46.059 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T15:54:46.059 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.7864583439999997, 2023-10-16T15:54:46.060 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 124, 2023-10-16T15:54:46.060 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 246, 2023-10-16T15:54:46.060 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 246, 2023-10-16T15:54:46.060 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.702679606 2023-10-16T15:54:46.060 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:54:46.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f146006b0b0 msgr2=0x7f146006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f146006b0b0 0x7f146006d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f147c0b4780 tx=0x7f146c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:46.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 msgr2=0x7f147c151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c151670 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f147800c870 tx=0x7f147800cd30 comp rx=0 tx=0).stop 2023-10-16T15:54:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 shutdown_connections 2023-10-16T15:54:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f146006b0b0 0x7f146006d570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f147c159e90 0x7f147c0c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f147c159ab0 0x7f147c151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 --2- 172.21.15.29:0/3916446385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f147c06c140 0x7f147c151670 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 >> 172.21.15.29:0/3916446385 conn(0x7f147c0ce9b0 msgr2=0x7f147c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 shutdown_connections 2023-10-16T15:54:46.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.046+0000 7f1483832640 1 -- 172.21.15.29:0/3916446385 wait complete. 2023-10-16T15:54:46.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T15:54:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.292+0000 7f83b3fff640 1 Processor -- start 2023-10-16T15:54:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.293+0000 7f83b3fff640 1 -- start start 2023-10-16T15:54:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.294+0000 7f83b3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b4070d70 0x7f83b406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.294+0000 7f83b3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83b40712c0 0x7f83b406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.294+0000 7f83b3fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b406c9c0 con 0x7f83b415aad0 2023-10-16T15:54:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.294+0000 7f83b3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b40b4620 con 0x7f83b40712c0 2023-10-16T15:54:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.294+0000 7f83b3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40b4340 con 0x7f83b4070d70 2023-10-16T15:54:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.295+0000 7f83b37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83b40712c0 0x7f83b406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.295+0000 7f83b37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83b40712c0 0x7f83b406c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47640/0 (socket says 172.21.15.29:47640) 2023-10-16T15:54:46.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.295+0000 7f83b37fe640 1 -- 172.21.15.29:0/2958443263 learned_addr learned my addr 172.21.15.29:0/2958443263 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.295+0000 7f83abfff640 1 --2- 172.21.15.29:0/2958443263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b4070d70 0x7f83b406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.296+0000 7f83b37fe640 1 -- 172.21.15.29:0/2958443263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b4070d70 msgr2=0x7f83b406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.296+0000 7f83b37fe640 1 --2- 172.21.15.29:0/2958443263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b4070d70 0x7f83b406bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.296+0000 7f83b37fe640 1 -- 172.21.15.29:0/2958443263 >> v1:172.21.15.170:6789/0 conn(0x7f83b415aad0 legacy=0x7f83b40cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.296+0000 7f83b37fe640 1 -- 172.21.15.29:0/2958443263 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b406f520 con 0x7f83b40712c0 2023-10-16T15:54:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.297+0000 7f83b37fe640 1 --2- 172.21.15.29:0/2958443263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b40712c0 0x7f83b406c480 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009e50 tx=0x7f83ac02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f0b95fbe72be57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.297+0000 7f83b27fc640 1 -- 172.21.15.29:0/2958443263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac02cb90 con 0x7f83b40712c0 2023-10-16T15:54:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.297+0000 7f83b27fc640 1 -- 172.21.15.29:0/2958443263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac02ccf0 con 0x7f83b40712c0 2023-10-16T15:54:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.297+0000 7f83b27fc640 1 -- 172.21.15.29:0/2958443263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac034610 con 0x7f83b40712c0 2023-10-16T15:54:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 -- 172.21.15.29:0/2958443263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b40712c0 msgr2=0x7f83b406c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 --2- 172.21.15.29:0/2958443263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b40712c0 0x7f83b406c480 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009e50 tx=0x7f83ac02bff0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 -- 172.21.15.29:0/2958443263 shutdown_connections 2023-10-16T15:54:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 --2- 172.21.15.29:0/2958443263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b40712c0 0x7f83b406c480 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 --2- 172.21.15.29:0/2958443263 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b4070d70 0x7f83b406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 -- 172.21.15.29:0/2958443263 >> 172.21.15.29:0/2958443263 conn(0x7f83b40b0a90 msgr2=0x7f83b40ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 -- 172.21.15.29:0/2958443263 shutdown_connections 2023-10-16T15:54:46.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.298+0000 7f83b3fff640 1 -- 172.21.15.29:0/2958443263 wait complete. 2023-10-16T15:54:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.300+0000 7f83b3fff640 1 Processor -- start 2023-10-16T15:54:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.300+0000 7f83b3fff640 1 -- start start 2023-10-16T15:54:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b3fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83b4070d70 0x7f83b40c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b3fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b40712c0 0x7f83b41f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 0x7f83b40c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41f5bd0 con 0x7f83b415aad0 2023-10-16T15:54:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b3fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40c2190 con 0x7f83b4070d70 2023-10-16T15:54:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b3fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40c65c0 con 0x7f83b40712c0 2023-10-16T15:54:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 0x7f83b40c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.301+0000 7f83b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 0x7f83b40c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47654/0 (socket says 172.21.15.29:47654) 2023-10-16T15:54:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 -- 172.21.15.29:0/1528983838 learned_addr learned my addr 172.21.15.29:0/1528983838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83abfff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b40712c0 0x7f83b41f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b2ffd640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83b4070d70 0x7f83b40c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 -- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b40712c0 msgr2=0x7f83b41f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b40712c0 0x7f83b41f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 -- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83b4070d70 msgr2=0x7f83b40c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83b4070d70 0x7f83b40c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 -- 172.21.15.29:0/1528983838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac009b30 con 0x7f83b415aad0 2023-10-16T15:54:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b2ffd640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83b4070d70 0x7f83b40c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.302+0000 7f83b37fe640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 0x7f83b40c6080 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f83ac036040 tx=0x7f83ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.303+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac009510 con 0x7f83b415aad0 2023-10-16T15:54:46.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.303+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac03e450 con 0x7f83b415aad0 2023-10-16T15:54:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.303+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac03e7e0 con 0x7f83b415aad0 2023-10-16T15:54:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.303+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b41f5e00 con 0x7f83b415aad0 2023-10-16T15:54:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.303+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b41f63c0 con 0x7f83b415aad0 2023-10-16T15:54:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.304+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b404e9b0 con 0x7f83b415aad0 2023-10-16T15:54:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.305+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f83ac03e940 con 0x7f83b415aad0 2023-10-16T15:54:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.306+0000 7f83ab7fe640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f839406b100 0x7f839406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.306+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f83ac0ac530 con 0x7f83b415aad0 2023-10-16T15:54:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.307+0000 7f83b2ffd640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f839406b100 0x7f839406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:46.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.308+0000 7f83b2ffd640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f839406b100 0x7f839406d5c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f83a4000f60 tx=0x7f83a4009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:46.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.310+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83ac0773d0 con 0x7f83b415aad0 2023-10-16T15:54:46.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.540+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f83b40cf520 con 0x7f83b415aad0 2023-10-16T15:54:46.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.541+0000 7f83ab7fe640 1 -- 172.21.15.29:0/1528983838 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v124) v1 ==== 53+0+73 (secure 0 0 0) 0x7f83ac02ce60 con 0x7f83b415aad0 2023-10-16T15:54:46.543 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T15:54:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.543+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f839406b100 msgr2=0x7f839406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.543+0000 7f83b3fff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f839406b100 0x7f839406d5c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f83a4000f60 tx=0x7f83a4009040 comp rx=0 tx=0).stop 2023-10-16T15:54:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.543+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 msgr2=0x7f83b40c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.543+0000 7f83b3fff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 0x7f83b40c6080 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f83ac036040 tx=0x7f83ac004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.543+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 shutdown_connections 2023-10-16T15:54:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.543+0000 7f83b3fff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f839406b100 0x7f839406d5c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.544+0000 7f83b3fff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83b415aad0 0x7f83b40c6080 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.544+0000 7f83b3fff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83b40712c0 0x7f83b41f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.544+0000 7f83b3fff640 1 --2- 172.21.15.29:0/1528983838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83b4070d70 0x7f83b40c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.544+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 >> 172.21.15.29:0/1528983838 conn(0x7f83b40b0a90 msgr2=0x7f83b4070000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.544+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 shutdown_connections 2023-10-16T15:54:46.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:46.544+0000 7f83b3fff640 1 -- 172.21.15.29:0/1528983838 wait complete. 2023-10-16T15:54:46.563 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T15:54:46.563 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T15:54:46.564 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T15:54:46.564 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T15:54:46.564 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T15:54:46.564 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T15:54:46.565 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T15:54:46.565 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T15:54:46.565 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:54:46.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:46.566 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T15:54:46.621 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T15:54:46.622 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T15:54:46.622 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T15:54:46.622 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.622 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T15:54:46.622 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.623 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:54:46.623 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:54:46.623 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T15:54:46.623 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:54:46.623 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:54:46.623 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T15:54:46.624 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T15:54:46.624 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T15:54:46.624 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T15:54:46.624 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T15:54:46.624 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36438sec preferred_lft 36438sec 2023-10-16T15:54:46.625 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T15:54:46.625 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.625 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:54:46.625 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T15:54:46.625 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T15:54:46.625 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T15:54:46.626 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:54:46.626 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T15:54:46.626 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.626 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T15:54:46.626 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.627 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:54:46.627 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:54:46.627 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T15:54:46.627 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.627 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:54:46.627 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:54:46.628 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T15:54:46.628 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:54:46.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:46.629 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T15:54:46.722 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T15:54:46.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:46.727 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T15:54:46.782 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T15:54:46.782 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T15:54:46.782 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T15:54:46.782 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T15:54:46.782 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T15:54:46.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:46.783 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T15:54:46.783 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T15:54:46.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T15:54:46.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T15:54:46.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T15:54:46.785 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T15:54:46.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:46.976 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:54:47.036 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:54:47.036 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:54:47.037 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:54:47.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:47.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T15:54:47.147 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:54:47.148 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T15:54:47.209 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:54:47.209 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:54:47.210 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:54:47.269 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T15:54:47.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:47.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:54:47.414 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:54:47.414 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:54:47.415 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:54:47.437 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:54:47.437 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:54:47.437 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:54:47.437 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:54:47.438 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:54:47.464 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:54:47.464 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:54:47.464 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:54:47.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:54:47.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:54:47.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T15:54:47.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T15:54:47.465 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T15:54:47.466 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T15:54:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T15:54:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T15:54:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T15:54:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T15:54:47.470 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T15:54:47.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:47.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:54:47.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:54:47.563 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T15:54:47.636 INFO:teuthology.orchestra.run.smithi029.stdout:client8309 2023-10-16T15:54:47.637 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:54:47.637 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:54:47.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T15:54:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.921+0000 7fce9e690640 1 Processor -- start 2023-10-16T15:54:47.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.921+0000 7fce9e690640 1 -- start start 2023-10-16T15:54:47.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9e690640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce980adec0 0x7fce98164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9e690640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9e690640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980cae70 con 0x7fce98071460 2023-10-16T15:54:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9e690640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce98165de0 con 0x7fce980adec0 2023-10-16T15:54:47.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9e690640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9806b860 con 0x7fce980c18e0 2023-10-16T15:54:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9d68e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce980adec0 0x7fce98164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9ce8d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fce98071460 0x7fce981653a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52016/0 (socket says 172.21.15.29:52016) 2023-10-16T15:54:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9ce8d640 1 -- 172.21.15.29:0/2046269714 learned_addr learned my addr 172.21.15.29:0/2046269714 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.923+0000 7fce9de8f640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.924+0000 7fce9d68e640 1 -- 172.21.15.29:0/2046269714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 msgr2=0x7fce980c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.924+0000 7fce9d68e640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.924+0000 7fce9d68e640 1 -- 172.21.15.29:0/2046269714 >> v1:172.21.15.29:6789/0 conn(0x7fce98071460 legacy=0x7fce981653a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.924+0000 7fce9d68e640 1 -- 172.21.15.29:0/2046269714 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9806b580 con 0x7fce980adec0 2023-10-16T15:54:47.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.924+0000 7fce9de8f640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.925+0000 7fce9d68e640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce980adec0 0x7fce98164e60 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fce940097a0 tx=0x7fce9402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b3b7b70af67527 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.925+0000 7fce8ffff640 1 -- 172.21.15.29:0/2046269714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94030800 con 0x7fce980adec0 2023-10-16T15:54:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.925+0000 7fce8ffff640 1 -- 172.21.15.29:0/2046269714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce94030960 con 0x7fce980adec0 2023-10-16T15:54:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.925+0000 7fce8ffff640 1 -- 172.21.15.29:0/2046269714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94030cf0 con 0x7fce980adec0 2023-10-16T15:54:47.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 -- 172.21.15.29:0/2046269714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce980adec0 msgr2=0x7fce98164e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce980adec0 0x7fce98164e60 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fce940097a0 tx=0x7fce9402b940 comp rx=0 tx=0).stop 2023-10-16T15:54:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 -- 172.21.15.29:0/2046269714 shutdown_connections 2023-10-16T15:54:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 --2- 172.21.15.29:0/2046269714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce980adec0 0x7fce98164e60 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 -- 172.21.15.29:0/2046269714 >> 172.21.15.29:0/2046269714 conn(0x7fce980ce820 msgr2=0x7fce980cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:47.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 -- 172.21.15.29:0/2046269714 shutdown_connections 2023-10-16T15:54:47.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.926+0000 7fce9e690640 1 -- 172.21.15.29:0/2046269714 wait complete. 2023-10-16T15:54:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.928+0000 7fce9e690640 1 Processor -- start 2023-10-16T15:54:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.928+0000 7fce9e690640 1 -- start start 2023-10-16T15:54:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9e690640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 0x7fce981f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9e690640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce980adec0 0x7fce98165360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:47.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9e690640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9e690640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981658a0 con 0x7fce98071460 2023-10-16T15:54:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9e690640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981642c0 con 0x7fce980adec0 2023-10-16T15:54:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9e690640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce981645a0 con 0x7fce980c18e0 2023-10-16T15:54:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9d68e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 0x7fce981f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9d68e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 0x7fce981f6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47662/0 (socket says 172.21.15.29:47662) 2023-10-16T15:54:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.929+0000 7fce9d68e640 1 -- 172.21.15.29:0/2744682816 learned_addr learned my addr 172.21.15.29:0/2744682816 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9d68e640 1 -- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 msgr2=0x7fce980c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9d68e640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9d68e640 1 -- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce980adec0 msgr2=0x7fce98165360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9ce8d640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce980adec0 0x7fce98165360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9d68e640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce980adec0 0x7fce98165360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9d68e640 1 -- 172.21.15.29:0/2744682816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94009400 con 0x7fce98071460 2023-10-16T15:54:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9ce8d640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce980adec0 0x7fce98165360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9d68e640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 0x7fce981f6690 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fce9402bf50 tx=0x7fce94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9403e040 con 0x7fce98071460 2023-10-16T15:54:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce94039d00 con 0x7fce98071460 2023-10-16T15:54:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce98164880 con 0x7fce98071460 2023-10-16T15:54:47.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.930+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94044630 con 0x7fce98071460 2023-10-16T15:54:47.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.931+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce98164e40 con 0x7fce98071460 2023-10-16T15:54:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.932+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9804e9b0 con 0x7fce98071460 2023-10-16T15:54:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.933+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fce9404d3b0 con 0x7fce98071460 2023-10-16T15:54:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.934+0000 7fce8dffb640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fce7c06b0b0 0x7fce7c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.934+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fce940ac2c0 con 0x7fce98071460 2023-10-16T15:54:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.934+0000 7fce9ce8d640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fce7c06b0b0 0x7fce7c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.935+0000 7fce9ce8d640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fce7c06b0b0 0x7fce7c06d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce980c1f70 tx=0x7fce88009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:47.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:47.938+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce94076ed0 con 0x7fce98071460 2023-10-16T15:54:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.169+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fce980cf730 con 0x7fce98071460 2023-10-16T15:54:48.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.314+0000 7fce8dffb640 1 -- 172.21.15.29:0/2744682816 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v125) v1 ==== 136+0+0 (secure 0 0 0) 0x7fce9407a4f0 con 0x7fce98071460 2023-10-16T15:54:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fce7c06b0b0 msgr2=0x7fce7c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fce7c06b0b0 0x7fce7c06d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce980c1f70 tx=0x7fce88009040 comp rx=0 tx=0).stop 2023-10-16T15:54:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 msgr2=0x7fce981f6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 0x7fce981f6690 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fce9402bf50 tx=0x7fce94004730 comp rx=0 tx=0).stop 2023-10-16T15:54:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 shutdown_connections 2023-10-16T15:54:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fce7c06b0b0 0x7fce7c06d570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce980c18e0 0x7fce980c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce980adec0 0x7fce98165360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 --2- 172.21.15.29:0/2744682816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce98071460 0x7fce981f6690 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.317+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 >> 172.21.15.29:0/2744682816 conn(0x7fce980ce820 msgr2=0x7fce980705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.318+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 shutdown_connections 2023-10-16T15:54:48.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.318+0000 7fce9e690640 1 -- 172.21.15.29:0/2744682816 wait complete. 2023-10-16T15:54:48.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:54:48.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:54:48.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.598+0000 7fb004938640 1 Processor -- start 2023-10-16T15:54:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.598+0000 7fb004938640 1 -- start start 2023-10-16T15:54:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.599+0000 7fb004938640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb00006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.599+0000 7fb004938640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0000cda80 0x7fb00015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.599+0000 7fb004938640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00006c960 con 0x7fb0000ad970 2023-10-16T15:54:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.599+0000 7fb004938640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00015b9d0 con 0x7fb0000cda80 2023-10-16T15:54:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.599+0000 7fb004938640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00015bcb0 con 0x7fb00006c040 2023-10-16T15:54:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.600+0000 7faffed9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb00006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.600+0000 7fafff59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb0000cda80 0x7fb00015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.600+0000 7faffed9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb00006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:56430/0 (socket says 172.21.15.29:56430) 2023-10-16T15:54:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.600+0000 7faffed9d640 1 -- 172.21.15.29:0/2271268695 learned_addr learned my addr 172.21.15.29:0/2271268695 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.600+0000 7faffe59c640 1 --1- 172.21.15.29:0/2271268695 >> v1:172.21.15.170:6789/0 conn(0x7fb0000ad970 0x7fb00015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:35890/0 (socket says 172.21.15.29:35890) 2023-10-16T15:54:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.601+0000 7faffed9d640 1 -- 172.21.15.29:0/2271268695 >> v2:172.21.15.170:3300/0 conn(0x7fb0000cda80 msgr2=0x7fb00015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.601+0000 7faffed9d640 1 --2- 172.21.15.29:0/2271268695 >> v2:172.21.15.170:3300/0 conn(0x7fb0000cda80 0x7fb00015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.601+0000 7faffed9d640 1 -- 172.21.15.29:0/2271268695 >> v1:172.21.15.170:6789/0 conn(0x7fb0000ad970 legacy=0x7fb00015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.601+0000 7faffed9d640 1 -- 172.21.15.29:0/2271268695 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00015bf90 con 0x7fb00006c040 2023-10-16T15:54:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.601+0000 7fafff59e640 1 --2- 172.21.15.29:0/2271268695 >> v2:172.21.15.170:3300/0 conn(0x7fb0000cda80 0x7fb00015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.602+0000 7faffed9d640 1 --2- 172.21.15.29:0/2271268695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb00006c420 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7faff0009870 tx=0x7faff002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28f46348e573bb69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.603+0000 7faffdd9b640 1 -- 172.21.15.29:0/2271268695 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0030840 con 0x7fb00006c040 2023-10-16T15:54:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.603+0000 7faffdd9b640 1 -- 172.21.15.29:0/2271268695 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff00309a0 con 0x7fb00006c040 2023-10-16T15:54:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7faffdd9b640 1 -- 172.21.15.29:0/2271268695 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0030d30 con 0x7fb00006c040 2023-10-16T15:54:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 -- 172.21.15.29:0/2271268695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 msgr2=0x7fb00006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 --2- 172.21.15.29:0/2271268695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb00006c420 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7faff0009870 tx=0x7faff002b980 comp rx=0 tx=0).stop 2023-10-16T15:54:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 -- 172.21.15.29:0/2271268695 shutdown_connections 2023-10-16T15:54:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 --2- 172.21.15.29:0/2271268695 >> v2:172.21.15.170:3300/0 conn(0x7fb0000cda80 0x7fb00015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 --2- 172.21.15.29:0/2271268695 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb00006c420 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 -- 172.21.15.29:0/2271268695 >> 172.21.15.29:0/2271268695 conn(0x7fb0000ce6a0 msgr2=0x7fb0000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 -- 172.21.15.29:0/2271268695 shutdown_connections 2023-10-16T15:54:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.604+0000 7fb004938640 1 -- 172.21.15.29:0/2271268695 wait complete. 2023-10-16T15:54:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.606+0000 7fb004938640 1 Processor -- start 2023-10-16T15:54:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.606+0000 7fb004938640 1 -- start start 2023-10-16T15:54:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fb004938640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb000157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fb004938640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0000ad970 0x7fb0000c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fb004938640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 0x7fb00006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fb004938640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000af160 con 0x7fb0000cda80 2023-10-16T15:54:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fb004938640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000ae500 con 0x7fb0000ad970 2023-10-16T15:54:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fb004938640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000ae7c0 con 0x7fb00006c040 2023-10-16T15:54:48.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.607+0000 7fafff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 0x7fb00006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:48.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 0x7fb00006fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47666/0 (socket says 172.21.15.29:47666) 2023-10-16T15:54:48.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 -- 172.21.15.29:0/3362437582 learned_addr learned my addr 172.21.15.29:0/3362437582 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:48.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7faffed9d640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb000157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:48.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7faffe59c640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0000ad970 0x7fb0000c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 -- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 msgr2=0x7fb000157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb000157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 -- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0000ad970 msgr2=0x7fb0000c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0000ad970 0x7fb0000c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 -- 172.21.15.29:0/3362437582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff8009a70 con 0x7fb0000cda80 2023-10-16T15:54:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.608+0000 7fafff59e640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 0x7fb00006fee0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7faff800efc0 tx=0x7faff800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.609+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff8009cd0 con 0x7fb0000cda80 2023-10-16T15:54:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.609+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff800f040 con 0x7fb0000cda80 2023-10-16T15:54:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.609+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff00094d0 con 0x7fb0000cda80 2023-10-16T15:54:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.609+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff8003990 con 0x7fb0000cda80 2023-10-16T15:54:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.609+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0000aed80 con 0x7fb0000cda80 2023-10-16T15:54:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.610+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00006b330 con 0x7fb0000cda80 2023-10-16T15:54:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.611+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faff8014480 con 0x7fb0000cda80 2023-10-16T15:54:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.612+0000 7fafe77fe640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fafec06b0b0 0x7fafec06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.612+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faff808a7f0 con 0x7fb0000cda80 2023-10-16T15:54:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.613+0000 7faffed9d640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fafec06b0b0 0x7fafec06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.614+0000 7faffed9d640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fafec06b0b0 0x7fafec06d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faff0036000 tx=0x7faff0008030 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:48.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.616+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faff8055380 con 0x7fb0000cda80 2023-10-16T15:54:48.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.811+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0000af2a0 con 0x7fb0000cda80 2023-10-16T15:54:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.812+0000 7fafe77fe640 1 -- 172.21.15.29:0/3362437582 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+16563 (secure 0 0 0) 0x7fb0000af2a0 con 0x7fb0000cda80 2023-10-16T15:54:48.815 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:48.815 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":246,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:40.167118+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fafec06b0b0 msgr2=0x7fafec06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fafec06b0b0 0x7fafec06d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faff0036000 tx=0x7faff0008030 comp rx=0 tx=0).stop 2023-10-16T15:54:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 msgr2=0x7fb00006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 0x7fb00006fee0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7faff800efc0 tx=0x7faff800c490 comp rx=0 tx=0).stop 2023-10-16T15:54:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 shutdown_connections 2023-10-16T15:54:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fafec06b0b0 0x7fafec06d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0000cda80 0x7fb00006fee0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0000ad970 0x7fb0000c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 --2- 172.21.15.29:0/3362437582 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb00006c040 0x7fb000157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 >> 172.21.15.29:0/3362437582 conn(0x7fb0000ce6a0 msgr2=0x7fb00006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 shutdown_connections 2023-10-16T15:54:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:48.815+0000 7fb004938640 1 -- 172.21.15.29:0/3362437582 wait complete. 2023-10-16T15:54:48.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:54:36.572242+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:54:37.685848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:54:48.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:54:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.064+0000 7f9dd359e640 1 Processor -- start 2023-10-16T15:54:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.064+0000 7f9dd359e640 1 -- start start 2023-10-16T15:54:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.065+0000 7f9dd359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9dd40cd260 0x7f9dd40ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 0x7f9dd40c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd40ce480 con 0x7f9dd415cc00 2023-10-16T15:54:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40cd7b0 con 0x7f9dd40cd260 2023-10-16T15:54:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40cda90 con 0x7f9dd40ad670 2023-10-16T15:54:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9dd415cc00 0x7f9dd40ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52028/0 (socket says 172.21.15.29:52028) 2023-10-16T15:54:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd259c640 1 -- 172.21.15.29:0/3809028512 learned_addr learned my addr 172.21.15.29:0/3809028512 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:49.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd1d9b640 1 --2- 172.21.15.29:0/3809028512 >> v2:172.21.15.170:3300/0 conn(0x7f9dd40cd260 0x7f9dd40ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.066+0000 7f9dd2d9d640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 0x7f9dd40c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510206699 0 0) 0x7f9dd40ce480 con 0x7f9dd415cc00 2023-10-16T15:54:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc0003520 con 0x7f9dd415cc00 2023-10-16T15:54:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759227349 0 0) 0x7f9dc0003520 con 0x7f9dd415cc00 2023-10-16T15:54:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 msgr2=0x7f9dd40c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 0x7f9dd40c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 >> v2:172.21.15.170:3300/0 conn(0x7f9dd40cd260 msgr2=0x7f9dd40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 --2- 172.21.15.29:0/3809028512 >> v2:172.21.15.170:3300/0 conn(0x7f9dd40cd260 0x7f9dd40ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40cdd70 con 0x7f9dd415cc00 2023-10-16T15:54:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9dc40019f0 con 0x7f9dd415cc00 2023-10-16T15:54:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd1d9b640 1 --2- 172.21.15.29:0/3809028512 >> v2:172.21.15.170:3300/0 conn(0x7f9dd40cd260 0x7f9dd40ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 >> v1:172.21.15.29:6789/0 conn(0x7f9dd415cc00 legacy=0x7f9dd40ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dc0003970 0x7f9dc0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 0x7f9dc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0004d00 0x7f9dc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40cd7b0 con 0x7f9dc0003970 2023-10-16T15:54:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40cda90 con 0x7f9dc0004d00 2023-10-16T15:54:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.067+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000b980 con 0x7f9dd40ad670 2023-10-16T15:54:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd259c640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dc0003970 0x7f9dc0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd1d9b640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 0x7f9dc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd2d9d640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0004d00 0x7f9dc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd259c640 1 -- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 msgr2=0x7f9dc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd259c640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd40ad670 0x7f9dc00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd259c640 1 -- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0004d00 msgr2=0x7f9dc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd259c640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0004d00 0x7f9dc0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.068+0000 7f9dd259c640 1 -- 172.21.15.29:0/3809028512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0003520 con 0x7f9dc0003970 2023-10-16T15:54:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.069+0000 7f9dd259c640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dc0003970 0x7f9dc0003d90 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400a830 tx=0x7f9dc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd5f3522f5befa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:49.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40072d0 con 0x7f9dc0003970 2023-10-16T15:54:49.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc402f070 con 0x7f9dc0003970 2023-10-16T15:54:49.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd159a640 1 -- 172.21.15.29:0/3809028512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40076b0 con 0x7f9dc0003970 2023-10-16T15:54:49.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd359e640 1 -- 172.21.15.29:0/3809028512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dc0003970 msgr2=0x7f9dc0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd359e640 1 --2- 172.21.15.29:0/3809028512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dc0003970 0x7f9dc0003d90 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400a830 tx=0x7f9dc00057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd2d9d640 1 -- 172.21.15.29:0/3809028512 reap_dead start 2023-10-16T15:54:49.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd359e640 1 -- 172.21.15.29:0/3809028512 shutdown_connections 2023-10-16T15:54:49.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.070+0000 7f9dd359e640 1 -- 172.21.15.29:0/3809028512 >> 172.21.15.29:0/3809028512 conn(0x7f9dd40b0170 msgr2=0x7f9dd40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.071+0000 7f9dd359e640 1 -- 172.21.15.29:0/3809028512 shutdown_connections 2023-10-16T15:54:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.071+0000 7f9dd359e640 1 -- 172.21.15.29:0/3809028512 wait complete. 2023-10-16T15:54:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.073+0000 7f9dd359e640 1 Processor -- start 2023-10-16T15:54:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.073+0000 7f9dd359e640 1 -- start start 2023-10-16T15:54:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0003970 0x7f9dd40c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd415f640 0x7f9dd40b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 0x7f9dd415e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd415ed70 con 0x7f9dd415e3b0 2023-10-16T15:54:49.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40b1440 con 0x7f9dc0003970 2023-10-16T15:54:49.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40c5750 con 0x7f9dd415f640 2023-10-16T15:54:49.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 0x7f9dd415e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 0x7f9dd415e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47690/0 (socket says 172.21.15.29:47690) 2023-10-16T15:54:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd2d9d640 1 -- 172.21.15.29:0/2083365870 learned_addr learned my addr 172.21.15.29:0/2083365870 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd259c640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0003970 0x7f9dd40c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.074+0000 7f9dd1d9b640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd415f640 0x7f9dd40b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd2d9d640 1 -- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd415f640 msgr2=0x7f9dd40b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd2d9d640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd415f640 0x7f9dd40b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd2d9d640 1 -- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0003970 msgr2=0x7f9dd40c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd2d9d640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0003970 0x7f9dd40c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd2d9d640 1 -- 172.21.15.29:0/2083365870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40b3fa0 con 0x7f9dd415e3b0 2023-10-16T15:54:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd259c640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0003970 0x7f9dd40c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd1d9b640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd415f640 0x7f9dd40b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd2d9d640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 0x7f9dd415e830 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc011490 tx=0x7f9dcc011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc004500 con 0x7f9dd415e3b0 2023-10-16T15:54:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc003610 con 0x7f9dd415e3b0 2023-10-16T15:54:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0039a0 con 0x7f9dd415e3b0 2023-10-16T15:54:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd40b0ea0 con 0x7f9dd415e3b0 2023-10-16T15:54:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.075+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd40b18e0 con 0x7f9dd415e3b0 2023-10-16T15:54:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.077+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd404e9b0 con 0x7f9dd415e3b0 2023-10-16T15:54:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.078+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9dcc003b00 con 0x7f9dd415e3b0 2023-10-16T15:54:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.079+0000 7f9dcaffd640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9db406b0b0 0x7f9db406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.079+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9dcc089e60 con 0x7f9dd415e3b0 2023-10-16T15:54:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.080+0000 7f9dd259c640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9db406b0b0 0x7f9db406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.081+0000 7f9dd259c640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9db406b0b0 0x7f9db406d570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4034000 tx=0x7f9dc4007000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.083+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9dcc054b80 con 0x7f9dd415e3b0 2023-10-16T15:54:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.282+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9dd406d320 con 0x7f9dd415e3b0 2023-10-16T15:54:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.283+0000 7f9dcaffd640 1 -- 172.21.15.29:0/2083365870 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v246) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9dcc019020 con 0x7f9dd415e3b0 2023-10-16T15:54:49.285 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:54:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.285+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9db406b0b0 msgr2=0x7f9db406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.285+0000 7f9dd359e640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9db406b0b0 0x7f9db406d570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4034000 tx=0x7f9dc4007000 comp rx=0 tx=0).stop 2023-10-16T15:54:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.285+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 msgr2=0x7f9dd415e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.285+0000 7f9dd359e640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 0x7f9dd415e830 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc011490 tx=0x7f9dcc011950 comp rx=0 tx=0).stop 2023-10-16T15:54:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 shutdown_connections 2023-10-16T15:54:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9db406b0b0 0x7f9db406d570 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dd415e3b0 0x7f9dd415e830 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9dd415f640 0x7f9dd40b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 --2- 172.21.15.29:0/2083365870 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dc0003970 0x7f9dd40c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 >> 172.21.15.29:0/2083365870 conn(0x7f9dd40b0170 msgr2=0x7f9dd40c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 shutdown_connections 2023-10-16T15:54:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.286+0000 7f9dd359e640 1 -- 172.21.15.29:0/2083365870 wait complete. 2023-10-16T15:54:49.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:54:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.531+0000 7fed7f44b640 1 Processor -- start 2023-10-16T15:54:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.532+0000 7fed7f44b640 1 -- start start 2023-10-16T15:54:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.533+0000 7fed7f44b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780addb0 0x7fed780caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7f44b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed7806a4b0 0x7fed781650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7f44b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed780cb140 con 0x7fed780c9000 2023-10-16T15:54:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7f44b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed78161a50 con 0x7fed7806a4b0 2023-10-16T15:54:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7f44b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7806b030 con 0x7fed780addb0 2023-10-16T15:54:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7d9e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed7806a4b0 0x7fed781650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7d9e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed7806a4b0 0x7fed781650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47698/0 (socket says 172.21.15.29:47698) 2023-10-16T15:54:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7d9e8640 1 -- 172.21.15.29:0/2794953863 learned_addr learned my addr 172.21.15.29:0/2794953863 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.534+0000 7fed7d1e7640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780addb0 0x7fed780caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.535+0000 7fed7d9e8640 1 -- 172.21.15.29:0/2794953863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780addb0 msgr2=0x7fed780caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.535+0000 7fed7d9e8640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780addb0 0x7fed780caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.535+0000 7fed7d9e8640 1 -- 172.21.15.29:0/2794953863 >> v1:172.21.15.170:6789/0 conn(0x7fed780c9000 legacy=0x7fed78069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.535+0000 7fed7d9e8640 1 -- 172.21.15.29:0/2794953863 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7806b310 con 0x7fed7806a4b0 2023-10-16T15:54:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.535+0000 7fed7d1e7640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780addb0 0x7fed780caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.536+0000 7fed7d9e8640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed781650d0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fed70009e50 tx=0x7fed7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926058281cf85540 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.536+0000 7fed6ffff640 1 -- 172.21.15.29:0/2794953863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7002ca70 con 0x7fed7806a4b0 2023-10-16T15:54:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.536+0000 7fed6ffff640 1 -- 172.21.15.29:0/2794953863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7002cbd0 con 0x7fed7806a4b0 2023-10-16T15:54:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.536+0000 7fed6ffff640 1 -- 172.21.15.29:0/2794953863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed700343f0 con 0x7fed7806a4b0 2023-10-16T15:54:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 -- 172.21.15.29:0/2794953863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 msgr2=0x7fed781650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed781650d0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fed70009e50 tx=0x7fed7002bed0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 -- 172.21.15.29:0/2794953863 shutdown_connections 2023-10-16T15:54:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed781650d0 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 --2- 172.21.15.29:0/2794953863 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780addb0 0x7fed780caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 -- 172.21.15.29:0/2794953863 >> 172.21.15.29:0/2794953863 conn(0x7fed780ce9b0 msgr2=0x7fed780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 -- 172.21.15.29:0/2794953863 shutdown_connections 2023-10-16T15:54:49.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.537+0000 7fed7f44b640 1 -- 172.21.15.29:0/2794953863 wait complete. 2023-10-16T15:54:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.539+0000 7fed7f44b640 1 Processor -- start 2023-10-16T15:54:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.540+0000 7fed7f44b640 1 -- start start 2023-10-16T15:54:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7f44b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed780c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7f44b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed780addb0 0x7fed78155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7f44b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780c9000 0x7fed780c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7f44b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed780c2f00 con 0x7fed7806a4b0 2023-10-16T15:54:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7f44b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed78156390 con 0x7fed780addb0 2023-10-16T15:54:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7f44b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed780c6a70 con 0x7fed780c9000 2023-10-16T15:54:49.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7d1e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed780c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7d1e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed780c6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47710/0 (socket says 172.21.15.29:47710) 2023-10-16T15:54:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.541+0000 7fed7d1e7640 1 -- 172.21.15.29:0/957076566 learned_addr learned my addr 172.21.15.29:0/957076566 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7c9e6640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed780addb0 0x7fed78155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d1e7640 1 -- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780c9000 msgr2=0x7fed780c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d9e8640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780c9000 0x7fed780c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d1e7640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780c9000 0x7fed780c84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d1e7640 1 -- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed780addb0 msgr2=0x7fed78155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d1e7640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed780addb0 0x7fed78155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d1e7640 1 -- 172.21.15.29:0/957076566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70009b30 con 0x7fed7806a4b0 2023-10-16T15:54:49.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7c9e6640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed780addb0 0x7fed78155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:49.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed7d1e7640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed780c6490 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fed7400c7f0 tx=0x7fed7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:49.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74007a30 con 0x7fed7806a4b0 2023-10-16T15:54:49.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed74007b90 con 0x7fed7806a4b0 2023-10-16T15:54:49.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.542+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7400ce40 con 0x7fed7806a4b0 2023-10-16T15:54:49.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.543+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed780c6d50 con 0x7fed7806a4b0 2023-10-16T15:54:49.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.543+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed780c7310 con 0x7fed7806a4b0 2023-10-16T15:54:49.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.544+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed780c3040 con 0x7fed7806a4b0 2023-10-16T15:54:49.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.545+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fed740113f0 con 0x7fed7806a4b0 2023-10-16T15:54:49.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.546+0000 7fed6dffb640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed5c06b0b0 0x7fed5c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:49.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.546+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fed74089e70 con 0x7fed7806a4b0 2023-10-16T15:54:49.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.547+0000 7fed7c9e6640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed5c06b0b0 0x7fed5c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:49.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.548+0000 7fed7c9e6640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed5c06b0b0 0x7fed5c06d570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fed7815fd80 tx=0x7fed68008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:49.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.550+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fed780c3040 con 0x7fed7806a4b0 2023-10-16T15:54:49.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.746+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed780c2cd0 con 0x7fed7806a4b0 2023-10-16T15:54:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.747+0000 7fed6dffb640 1 -- 172.21.15.29:0/957076566 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v246) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed7401f0a0 con 0x7fed7806a4b0 2023-10-16T15:54:49.749 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:54:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.749+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed5c06b0b0 msgr2=0x7fed5c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.749+0000 7fed7f44b640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed5c06b0b0 0x7fed5c06d570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fed7815fd80 tx=0x7fed68008040 comp rx=0 tx=0).stop 2023-10-16T15:54:49.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.749+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 msgr2=0x7fed780c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:49.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.749+0000 7fed7f44b640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed780c6490 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fed7400c7f0 tx=0x7fed7400cbf0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 shutdown_connections 2023-10-16T15:54:49.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fed5c06b0b0 0x7fed5c06d570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fed780c9000 0x7fed780c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed780addb0 0x7fed78155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 --2- 172.21.15.29:0/957076566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed7806a4b0 0x7fed780c6490 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:49.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 >> 172.21.15.29:0/957076566 conn(0x7fed780ce9b0 msgr2=0x7fed7806bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:49.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 shutdown_connections 2023-10-16T15:54:49.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:49.750+0000 7fed7f44b640 1 -- 172.21.15.29:0/957076566 wait complete. 2023-10-16T15:54:49.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:54:50.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.005+0000 7f761b59e640 1 Processor -- start 2023-10-16T15:54:50.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.005+0000 7f761b59e640 1 -- start start 2023-10-16T15:54:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.006+0000 7f761b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c0b8480 0x7f761c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.006+0000 7f761b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f761c0d0be0 0x7f761c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.006+0000 7f761b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c15d150 con 0x7f761c15cd70 2023-10-16T15:54:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.006+0000 7f761b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c0d1180 con 0x7f761c0d0be0 2023-10-16T15:54:50.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.006+0000 7f761b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c0d1460 con 0x7f761c0b8480 2023-10-16T15:54:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.007+0000 7f761ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f761c0d0be0 0x7f761c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.007+0000 7f761a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f761c15cd70 0x7f761c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52034/0 (socket says 172.21.15.29:52034) 2023-10-16T15:54:50.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.007+0000 7f761a59c640 1 -- 172.21.15.29:0/2228693433 learned_addr learned my addr 172.21.15.29:0/2228693433 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.007+0000 7f7619d9b640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c0b8480 0x7f761c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.007+0000 7f761959a640 1 -- 172.21.15.29:0/2228693433 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249784932 0 0) 0x7f761c15d150 con 0x7f761c15cd70 2023-10-16T15:54:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.007+0000 7f761959a640 1 -- 172.21.15.29:0/2228693433 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608003520 con 0x7f761c15cd70 2023-10-16T15:54:50.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.008+0000 7f761ad9d640 1 -- 172.21.15.29:0/2228693433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c0b8480 msgr2=0x7f761c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.008+0000 7f761ad9d640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c0b8480 0x7f761c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.008+0000 7f761ad9d640 1 -- 172.21.15.29:0/2228693433 >> v1:172.21.15.29:6789/0 conn(0x7f761c15cd70 legacy=0x7f761c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.008+0000 7f761ad9d640 1 -- 172.21.15.29:0/2228693433 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0d1740 con 0x7f761c0d0be0 2023-10-16T15:54:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.008+0000 7f7619d9b640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c0b8480 0x7f761c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:50.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.009+0000 7f761ad9d640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0d0be0 0x7f761c0cdd00 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f76140097a0 tx=0x7f761402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff993cd14c85b5c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.009+0000 7f761959a640 1 -- 172.21.15.29:0/2228693433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761402d040 con 0x7f761c0d0be0 2023-10-16T15:54:50.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.009+0000 7f761959a640 1 -- 172.21.15.29:0/2228693433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7614031810 con 0x7f761c0d0be0 2023-10-16T15:54:50.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.009+0000 7f761959a640 1 -- 172.21.15.29:0/2228693433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614031ba0 con 0x7f761c0d0be0 2023-10-16T15:54:50.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 -- 172.21.15.29:0/2228693433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0d0be0 msgr2=0x7f761c0cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0d0be0 0x7f761c0cdd00 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f76140097a0 tx=0x7f761402b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 -- 172.21.15.29:0/2228693433 shutdown_connections 2023-10-16T15:54:50.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0d0be0 0x7f761c0cdd00 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 --2- 172.21.15.29:0/2228693433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c0b8480 0x7f761c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 -- 172.21.15.29:0/2228693433 >> 172.21.15.29:0/2228693433 conn(0x7f761c0cf570 msgr2=0x7f761c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:50.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 -- 172.21.15.29:0/2228693433 shutdown_connections 2023-10-16T15:54:50.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.010+0000 7f761b59e640 1 -- 172.21.15.29:0/2228693433 wait complete. 2023-10-16T15:54:50.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.012+0000 7f761b59e640 1 Processor -- start 2023-10-16T15:54:50.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.012+0000 7f761b59e640 1 -- start start 2023-10-16T15:54:50.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 0x7f761c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f761c0d0be0 0x7f761c14f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c15cd70 0x7f761c14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c15a910 con 0x7f761c0b8480 2023-10-16T15:54:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c15d880 con 0x7f761c0d0be0 2023-10-16T15:54:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c0b3290 con 0x7f761c15cd70 2023-10-16T15:54:50.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 0x7f761c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 0x7f761c15d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47718/0 (socket says 172.21.15.29:47718) 2023-10-16T15:54:50.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.013+0000 7f761a59c640 1 -- 172.21.15.29:0/390498139 learned_addr learned my addr 172.21.15.29:0/390498139 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:50.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761ad9d640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c15cd70 0x7f761c14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761a59c640 1 -- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c15cd70 msgr2=0x7f761c14fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761a59c640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c15cd70 0x7f761c14fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761a59c640 1 -- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f761c0d0be0 msgr2=0x7f761c14f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f7619d9b640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f761c0d0be0 0x7f761c14f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761a59c640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f761c0d0be0 0x7f761c14f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761a59c640 1 -- 172.21.15.29:0/390498139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614009400 con 0x7f761c0b8480 2023-10-16T15:54:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761a59c640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 0x7f761c15d340 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f760c00c460 tx=0x7f760c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c005ce0 con 0x7f761c0b8480 2023-10-16T15:54:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760c005e40 con 0x7f761c0b8480 2023-10-16T15:54:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c007290 con 0x7f761c0b8480 2023-10-16T15:54:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.014+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c0b3570 con 0x7f761c0b8480 2023-10-16T15:54:50.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.015+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c0b3b30 con 0x7f761c0b8480 2023-10-16T15:54:50.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.016+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c0b3d50 con 0x7f761c0b8480 2023-10-16T15:54:50.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.017+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f760c0073f0 con 0x7f761c0b8480 2023-10-16T15:54:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.018+0000 7f7612ffd640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f75fc06b0b0 0x7f75fc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.018+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f760c0890c0 con 0x7f761c0b8480 2023-10-16T15:54:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.019+0000 7f7619d9b640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f75fc06b0b0 0x7f75fc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.019+0000 7f7619d9b640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f75fc06b0b0 0x7f75fc06d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f761c15add0 tx=0x7f760400b040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.022+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f760c053d50 con 0x7f761c0b8480 2023-10-16T15:54:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.215+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f761c04e9b0 con 0x7f761c0b8480 2023-10-16T15:54:50.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.216+0000 7f7612ffd640 1 -- 172.21.15.29:0/390498139 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v246) v1 ==== 100+0+11 (secure 0 0 0) 0x7f760c017020 con 0x7f761c0b8480 2023-10-16T15:54:50.219 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:54:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f75fc06b0b0 msgr2=0x7f75fc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f75fc06b0b0 0x7f75fc06d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f761c15add0 tx=0x7f760400b040 comp rx=0 tx=0).stop 2023-10-16T15:54:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 msgr2=0x7f761c15d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 0x7f761c15d340 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f760c00c460 tx=0x7f760c00c920 comp rx=0 tx=0).stop 2023-10-16T15:54:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 shutdown_connections 2023-10-16T15:54:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f75fc06b0b0 0x7f75fc06d570 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f761c15cd70 0x7f761c14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f761c0d0be0 0x7f761c14f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 --2- 172.21.15.29:0/390498139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0b8480 0x7f761c15d340 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 >> 172.21.15.29:0/390498139 conn(0x7f761c0cf570 msgr2=0x7f761c1e7930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 shutdown_connections 2023-10-16T15:54:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.219+0000 7f761b59e640 1 -- 172.21.15.29:0/390498139 wait complete. 2023-10-16T15:54:50.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:54:50.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.454+0000 7f99fd2cf640 1 Processor -- start 2023-10-16T15:54:50.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.454+0000 7f99fd2cf640 1 -- start start 2023-10-16T15:54:50.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.455+0000 7f99fd2cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f806c7d0 0x7f99f80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.455+0000 7f99fd2cf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f99f80b4c70 0x7f99f80b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.455+0000 7f99fd2cf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80cd630 con 0x7f99f806c3f0 2023-10-16T15:54:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.455+0000 7f99fd2cf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f815e7d0 con 0x7f99f806c7d0 2023-10-16T15:54:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.455+0000 7f99fd2cf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80ce1b0 con 0x7f99f80b4c70 2023-10-16T15:54:50.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f806c7d0 0x7f99f80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f806c7d0 0x7f99f80b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47732/0 (socket says 172.21.15.29:47732) 2023-10-16T15:54:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 -- 172.21.15.29:0/3163419035 learned_addr learned my addr 172.21.15.29:0/3163419035 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:50.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f77fe640 1 --2- 172.21.15.29:0/3163419035 >> v2:172.21.15.170:3300/0 conn(0x7f99f80b4c70 0x7f99f80b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 -- 172.21.15.29:0/3163419035 >> v2:172.21.15.170:3300/0 conn(0x7f99f80b4c70 msgr2=0x7f99f80b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:50.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 --2- 172.21.15.29:0/3163419035 >> v2:172.21.15.170:3300/0 conn(0x7f99f80b4c70 0x7f99f80b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 -- 172.21.15.29:0/3163419035 >> v1:172.21.15.170:6789/0 conn(0x7f99f806c3f0 legacy=0x7f99f815e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:50.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f67fc640 1 -- 172.21.15.29:0/3163419035 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80b50a0 con 0x7f99f806c7d0 2023-10-16T15:54:50.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.456+0000 7f99f77fe640 1 --2- 172.21.15.29:0/3163419035 >> v2:172.21.15.170:3300/0 conn(0x7f99f80b4c70 0x7f99f80b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:50.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.457+0000 7f99f67fc640 1 --2- 172.21.15.29:0/3163419035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80b4730 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f99f80d27d0 tx=0x7f99e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf2f031af2363cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99f5ffb640 1 -- 172.21.15.29:0/3163419035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8031870 con 0x7f99f806c7d0 2023-10-16T15:54:50.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99f5ffb640 1 -- 172.21.15.29:0/3163419035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e80319d0 con 0x7f99f806c7d0 2023-10-16T15:54:50.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99f5ffb640 1 -- 172.21.15.29:0/3163419035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8031d60 con 0x7f99f806c7d0 2023-10-16T15:54:50.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 -- 172.21.15.29:0/3163419035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 msgr2=0x7f99f80b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/3163419035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80b4730 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f99f80d27d0 tx=0x7f99e802b910 comp rx=0 tx=0).stop 2023-10-16T15:54:50.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 -- 172.21.15.29:0/3163419035 shutdown_connections 2023-10-16T15:54:50.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/3163419035 >> v2:172.21.15.170:3300/0 conn(0x7f99f80b4c70 0x7f99f80b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/3163419035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80b4730 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 -- 172.21.15.29:0/3163419035 >> 172.21.15.29:0/3163419035 conn(0x7f99f80d2cb0 msgr2=0x7f99f80d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:50.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 -- 172.21.15.29:0/3163419035 shutdown_connections 2023-10-16T15:54:50.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.458+0000 7f99fd2cf640 1 -- 172.21.15.29:0/3163419035 wait complete. 2023-10-16T15:54:50.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.460+0000 7f99fd2cf640 1 Processor -- start 2023-10-16T15:54:50.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.460+0000 7f99fd2cf640 1 -- start start 2023-10-16T15:54:50.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.461+0000 7f99fd2cf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99f806c3f0 0x7f99f8157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.461+0000 7f99fd2cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.461+0000 7f99fd2cf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99f80b4c70 0x7f99f815fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.461+0000 7f99fd2cf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80cc2d0 con 0x7f99f806c7d0 2023-10-16T15:54:50.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.461+0000 7f99fd2cf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80cb650 con 0x7f99f806c3f0 2023-10-16T15:54:50.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.461+0000 7f99fd2cf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80cb930 con 0x7f99f80b4c70 2023-10-16T15:54:50.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80cc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47736/0 (socket says 172.21.15.29:47736) 2023-10-16T15:54:50.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 -- 172.21.15.29:0/1814615662 learned_addr learned my addr 172.21.15.29:0/1814615662 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:50.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f77fe640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99f80b4c70 0x7f99f815fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 -- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99f80b4c70 msgr2=0x7f99f815fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f6ffd640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99f806c3f0 0x7f99f8157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99f80b4c70 0x7f99f815fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 -- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99f806c3f0 msgr2=0x7f99f8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99f806c3f0 0x7f99f8157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 -- 172.21.15.29:0/1814615662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80094d0 con 0x7f99f806c7d0 2023-10-16T15:54:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f77fe640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99f80b4c70 0x7f99f815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.462+0000 7f99f67fc640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80cc6f0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f99e8000c00 tx=0x7f99e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.463+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8031840 con 0x7f99f806c7d0 2023-10-16T15:54:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.463+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e80443f0 con 0x7f99f806c7d0 2023-10-16T15:54:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.463+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80447a0 con 0x7f99f806c7d0 2023-10-16T15:54:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.463+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f80cbc10 con 0x7f99f806c7d0 2023-10-16T15:54:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.463+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f8157640 con 0x7f99f806c7d0 2023-10-16T15:54:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.464+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f80cc410 con 0x7f99f806c7d0 2023-10-16T15:54:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.465+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f99e804d450 con 0x7f99f806c7d0 2023-10-16T15:54:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.466+0000 7f99ef7fe640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f99d806b0b0 0x7f99d806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.466+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99e802d040 con 0x7f99f806c7d0 2023-10-16T15:54:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.467+0000 7f99f6ffd640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f99d806b0b0 0x7f99d806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.468+0000 7f99f6ffd640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f99d806b0b0 0x7f99d806d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006fd0 tx=0x7f99e0008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.470+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f99f80cc410 con 0x7f99f806c7d0 2023-10-16T15:54:50.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.664+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f99f80cc410 con 0x7f99f806c7d0 2023-10-16T15:54:50.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.665+0000 7f99ef7fe640 1 -- 172.21.15.29:0/1814615662 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v246) v1 ==== 84+0+209 (secure 0 0 0) 0x7f99e80350f0 con 0x7f99f806c7d0 2023-10-16T15:54:50.667 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:50.668 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:54:50.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:50.668 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:54:50.668 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:54:50.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:50.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:50.669 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 30, 2023-10-16T15:54:50.669 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_metadata" 2023-10-16T15:54:50.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:54:50.670 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:54:50.670 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 31, 2023-10-16T15:54:50.670 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_data" 2023-10-16T15:54:50.670 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:54:50.671 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:54:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f99d806b0b0 msgr2=0x7f99d806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f99d806b0b0 0x7f99d806d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006fd0 tx=0x7f99e0008040 comp rx=0 tx=0).stop 2023-10-16T15:54:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 msgr2=0x7f99f80cc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80cc6f0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f99e8000c00 tx=0x7f99e80046c0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 shutdown_connections 2023-10-16T15:54:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f99d806b0b0 0x7f99d806d570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f806c7d0 0x7f99f80cc6f0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f99f80b4c70 0x7f99f815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 --2- 172.21.15.29:0/1814615662 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f99f806c3f0 0x7f99f8157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 >> 172.21.15.29:0/1814615662 conn(0x7f99f80d2cb0 msgr2=0x7f99f815e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 shutdown_connections 2023-10-16T15:54:50.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.668+0000 7f99fd2cf640 1 -- 172.21.15.29:0/1814615662 wait complete. 2023-10-16T15:54:50.688 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T15:54:50.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T15:54:50.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.894+0000 7f2a2ca26640 1 Processor -- start 2023-10-16T15:54:50.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.895+0000 7f2a2ca26640 1 -- start start 2023-10-16T15:54:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.896+0000 7f2a2ca26640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a2806cc60 0x7f2a280cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.896+0000 7f2a2ca26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a2806b630 0x7f2a280cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.896+0000 7f2a2ca26640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2806bbc0 con 0x7f2a280cca70 2023-10-16T15:54:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.896+0000 7f2a2ca26640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a280ce480 con 0x7f2a2806b630 2023-10-16T15:54:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.896+0000 7f2a2ca26640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2806bea0 con 0x7f2a2806cc60 2023-10-16T15:54:50.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a2659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a280cca70 0x7f2a280cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52050/0 (socket says 172.21.15.29:52050) 2023-10-16T15:54:50.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a2659c640 1 -- 172.21.15.29:0/1374225479 learned_addr learned my addr 172.21.15.29:0/1374225479 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:50.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a26d9d640 1 --2- 172.21.15.29:0/1374225479 >> v2:172.21.15.29:3300/0 conn(0x7f2a2806b630 0x7f2a280cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a25d9b640 1 --2- 172.21.15.29:0/1374225479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a2806cc60 0x7f2a280cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a26d9d640 1 -- 172.21.15.29:0/1374225479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a2806cc60 msgr2=0x7f2a280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:50.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a26d9d640 1 --2- 172.21.15.29:0/1374225479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a2806cc60 0x7f2a280cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a26d9d640 1 -- 172.21.15.29:0/1374225479 >> v1:172.21.15.29:6789/0 conn(0x7f2a280cca70 legacy=0x7f2a280cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a2559a640 1 -- 172.21.15.29:0/1374225479 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528737389 0 0) 0x7f2a2806bbc0 con 0x7f2a280cca70 2023-10-16T15:54:50.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.897+0000 7f2a26d9d640 1 -- 172.21.15.29:0/1374225479 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2806c180 con 0x7f2a2806b630 2023-10-16T15:54:50.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a26d9d640 1 --2- 172.21.15.29:0/1374225479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280cf830 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d90 tx=0x7f2a2002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612b6c454ef646dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2559a640 1 -- 172.21.15.29:0/1374225479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2002cc50 con 0x7f2a2806b630 2023-10-16T15:54:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2559a640 1 -- 172.21.15.29:0/1374225479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2002cdb0 con 0x7f2a2806b630 2023-10-16T15:54:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2559a640 1 -- 172.21.15.29:0/1374225479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20034660 con 0x7f2a2806b630 2023-10-16T15:54:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2ca26640 1 -- 172.21.15.29:0/1374225479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 msgr2=0x7f2a280cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/1374225479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280cf830 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d90 tx=0x7f2a2002bea0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2ca26640 1 -- 172.21.15.29:0/1374225479 shutdown_connections 2023-10-16T15:54:50.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/1374225479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280cf830 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/1374225479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a2806cc60 0x7f2a280cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.899+0000 7f2a2ca26640 1 -- 172.21.15.29:0/1374225479 >> 172.21.15.29:0/1374225479 conn(0x7f2a28099e30 msgr2=0x7f2a280d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:50.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.900+0000 7f2a2ca26640 1 -- 172.21.15.29:0/1374225479 shutdown_connections 2023-10-16T15:54:50.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.900+0000 7f2a2ca26640 1 -- 172.21.15.29:0/1374225479 wait complete. 2023-10-16T15:54:50.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.901+0000 7f2a2ca26640 1 Processor -- start 2023-10-16T15:54:50.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.901+0000 7f2a2ca26640 1 -- start start 2023-10-16T15:54:50.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.902+0000 7f2a2ca26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.902+0000 7f2a2ca26640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a2806cc60 0x7f2a2815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.902+0000 7f2a2ca26640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a280cca70 0x7f2a280b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.902+0000 7f2a2ca26640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280b4a00 con 0x7f2a2806b630 2023-10-16T15:54:50.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.902+0000 7f2a2ca26640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2815c5c0 con 0x7f2a2806cc60 2023-10-16T15:54:50.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.902+0000 7f2a2ca26640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280b4160 con 0x7f2a280cca70 2023-10-16T15:54:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47746/0 (socket says 172.21.15.29:47746) 2023-10-16T15:54:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 -- 172.21.15.29:0/2255700198 learned_addr learned my addr 172.21.15.29:0/2255700198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a25d9b640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a2806cc60 0x7f2a2815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 -- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a280cca70 msgr2=0x7f2a280b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a280cca70 0x7f2a280b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 -- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a2806cc60 msgr2=0x7f2a2815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:50.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a2806cc60 0x7f2a2815c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:50.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 -- 172.21.15.29:0/2255700198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20009a70 con 0x7f2a2806b630 2023-10-16T15:54:50.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a25d9b640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a2806cc60 0x7f2a2815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:50.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.903+0000 7f2a2659c640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280c8d80 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c430 tx=0x7f2a1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.904+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a180074e0 con 0x7f2a2806b630 2023-10-16T15:54:50.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.904+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18007660 con 0x7f2a2806b630 2023-10-16T15:54:50.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.904+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18004a20 con 0x7f2a2806b630 2023-10-16T15:54:50.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.904+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a280b4440 con 0x7f2a2806b630 2023-10-16T15:54:50.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.904+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a280b10a0 con 0x7f2a2806b630 2023-10-16T15:54:50.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.905+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a280b12c0 con 0x7f2a2806b630 2023-10-16T15:54:50.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.906+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2a18004b80 con 0x7f2a2806b630 2023-10-16T15:54:50.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.907+0000 7f2a16ffd640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2a0806b0b0 0x7f2a0806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:50.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.907+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2a180898e0 con 0x7f2a2806b630 2023-10-16T15:54:50.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.907+0000 7f2a25d9b640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2a0806b0b0 0x7f2a0806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:50.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.908+0000 7f2a25d9b640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2a0806b0b0 0x7f2a0806d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a281616f0 tx=0x7f2a1c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:50.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:50.911+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2a18054600 con 0x7f2a2806b630 2023-10-16T15:54:51.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.099+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2a2804e160 con 0x7f2a2806b630 2023-10-16T15:54:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.343+0000 7f2a16ffd640 1 -- 172.21.15.29:0/2255700198 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v247) v1 ==== 144+0+0 (secure 0 0 0) 0x7f2a18089030 con 0x7f2a2806b630 2023-10-16T15:54:51.346 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' created 2023-10-16T15:54:51.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.347+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2a0806b0b0 msgr2=0x7f2a0806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.347+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2a0806b0b0 0x7f2a0806d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a281616f0 tx=0x7f2a1c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:51.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.347+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 msgr2=0x7f2a280c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.347+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280c8d80 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c430 tx=0x7f2a1800c8f0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 shutdown_connections 2023-10-16T15:54:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2a0806b0b0 0x7f2a0806d570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2a280cca70 0x7f2a280b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2a2806cc60 0x7f2a2815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 --2- 172.21.15.29:0/2255700198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a2806b630 0x7f2a280c8d80 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 >> 172.21.15.29:0/2255700198 conn(0x7f2a28099e30 msgr2=0x7f2a280cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:51.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 shutdown_connections 2023-10-16T15:54:51.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.348+0000 7f2a2ca26640 1 -- 172.21.15.29:0/2255700198 wait complete. 2023-10-16T15:54:51.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:54:51.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.587+0000 7fc666df0640 1 Processor -- start 2023-10-16T15:54:51.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.587+0000 7fc666df0640 1 -- start start 2023-10-16T15:54:51.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.589+0000 7fc666df0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc660070bf0 0x7fc660071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.589+0000 7fc666df0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc660070760 con 0x7fc6600ac890 2023-10-16T15:54:51.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.589+0000 7fc666df0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6600cdac0 con 0x7fc66015dc90 2023-10-16T15:54:51.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.589+0000 7fc666df0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600715c0 con 0x7fc660070bf0 2023-10-16T15:54:51.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.590+0000 7fc6665ef640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc6600ac890 0x7fc6600accb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52064/0 (socket says 172.21.15.29:52064) 2023-10-16T15:54:51.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.590+0000 7fc6665ef640 1 -- 172.21.15.29:0/3908721829 learned_addr learned my addr 172.21.15.29:0/3908721829 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:51.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.590+0000 7fc6655ed640 1 --2- 172.21.15.29:0/3908721829 >> v2:172.21.15.170:3300/0 conn(0x7fc660070bf0 0x7fc660071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.590+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663319692 0 0) 0x7fc660070760 con 0x7fc6600ac890 2023-10-16T15:54:51.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.590+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc64c003520 con 0x7fc6600ac890 2023-10-16T15:54:51.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391630425 0 0) 0x7fc64c003520 con 0x7fc6600ac890 2023-10-16T15:54:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 >> v2:172.21.15.170:3300/0 conn(0x7fc660070bf0 msgr2=0x7fc660071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 --2- 172.21.15.29:0/3908721829 >> v2:172.21.15.170:3300/0 conn(0x7fc660070bf0 0x7fc660071080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 >> v1:172.21.15.170:6789/0 conn(0x7fc66015dc90 legacy=0x7fc6600c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc660070110 con 0x7fc6600ac890 2023-10-16T15:54:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fc6580019f0 con 0x7fc6600ac890 2023-10-16T15:54:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 >> v1:172.21.15.29:6789/0 conn(0x7fc6600ac890 legacy=0x7fc6600accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0039c0 0x7fc64c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc64c0047c0 0x7fc64c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc64c005180 0x7fc64c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c00c050 con 0x7fc64c0047c0 2023-10-16T15:54:51.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c00c1c0 con 0x7fc64c005180 2023-10-16T15:54:51.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c00c380 con 0x7fc64c0039c0 2023-10-16T15:54:51.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.591+0000 7fc6655ed640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc64c0047c0 0x7fc64c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc665dee640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0039c0 0x7fc64c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6665ef640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc64c005180 0x7fc64c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6655ed640 1 -- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0039c0 msgr2=0x7fc64c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:51.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6655ed640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0039c0 0x7fc64c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6655ed640 1 -- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc64c005180 msgr2=0x7fc64c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:51.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6655ed640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc64c005180 0x7fc64c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6655ed640 1 -- 172.21.15.29:0/3908721829 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c003520 con 0x7fc64c0047c0 2023-10-16T15:54:51.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6665ef640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc64c005180 0x7fc64c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:51.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.592+0000 7fc6665ef640 1 -- 172.21.15.29:0/3908721829 reap_dead start 2023-10-16T15:54:51.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.593+0000 7fc6655ed640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc64c0047c0 0x7fc64c004c40 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fc654009ac0 tx=0x7fc64c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3dd3afb935b13a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:51.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.593+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65402c6b0 con 0x7fc64c0047c0 2023-10-16T15:54:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.593+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65402c810 con 0x7fc64c0047c0 2023-10-16T15:54:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc664dec640 1 -- 172.21.15.29:0/3908721829 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65402cba0 con 0x7fc64c0047c0 2023-10-16T15:54:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 -- 172.21.15.29:0/3908721829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc64c0047c0 msgr2=0x7fc64c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc64c0047c0 0x7fc64c004c40 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fc654009ac0 tx=0x7fc64c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 -- 172.21.15.29:0/3908721829 shutdown_connections 2023-10-16T15:54:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 --2- 172.21.15.29:0/3908721829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc64c0047c0 0x7fc64c004c40 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 -- 172.21.15.29:0/3908721829 >> 172.21.15.29:0/3908721829 conn(0x7fc6600ce6a0 msgr2=0x7fc6600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:51.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 -- 172.21.15.29:0/3908721829 shutdown_connections 2023-10-16T15:54:51.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.594+0000 7fc666df0640 1 -- 172.21.15.29:0/3908721829 wait complete. 2023-10-16T15:54:51.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.596+0000 7fc666df0640 1 Processor -- start 2023-10-16T15:54:51.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.596+0000 7fc666df0640 1 -- start start 2023-10-16T15:54:51.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc666df0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0047c0 0x7fc660150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc666df0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6600ccdd0 0x7fc6600cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc666df0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 0x7fc6600b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc666df0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c4ca0 con 0x7fc6600b1f70 2023-10-16T15:54:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc666df0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600cd790 con 0x7fc6600ccdd0 2023-10-16T15:54:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc666df0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc660151030 con 0x7fc64c0047c0 2023-10-16T15:54:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc6665ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 0x7fc6600b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc6665ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 0x7fc6600b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47760/0 (socket says 172.21.15.29:47760) 2023-10-16T15:54:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.597+0000 7fc6665ef640 1 -- 172.21.15.29:0/4139193122 learned_addr learned my addr 172.21.15.29:0/4139193122 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:51.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc665dee640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0047c0 0x7fc660150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6655ed640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6600ccdd0 0x7fc6600cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6665ef640 1 -- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0047c0 msgr2=0x7fc660150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6665ef640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0047c0 0x7fc660150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6665ef640 1 -- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6600ccdd0 msgr2=0x7fc6600cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6665ef640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6600ccdd0 0x7fc6600cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6665ef640 1 -- 172.21.15.29:0/4139193122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600c4300 con 0x7fc6600b1f70 2023-10-16T15:54:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.598+0000 7fc6665ef640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 0x7fc6600b23f0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fc658000f80 tx=0x7fc65800df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.599+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800b800 con 0x7fc6600b1f70 2023-10-16T15:54:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.599+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800b980 con 0x7fc6600b1f70 2023-10-16T15:54:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.599+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800bd30 con 0x7fc6600b1f70 2023-10-16T15:54:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.599+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6600c45e0 con 0x7fc6600b1f70 2023-10-16T15:54:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.599+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6600b2930 con 0x7fc6600b1f70 2023-10-16T15:54:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.600+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6600c4de0 con 0x7fc6600b1f70 2023-10-16T15:54:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.601+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc658007110 con 0x7fc6600b1f70 2023-10-16T15:54:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.602+0000 7fc6527fc640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc64406b0b0 0x7fc64406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.602+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6110+0+0 (secure 0 0 0) 0x7fc65808a5c0 con 0x7fc6600b1f70 2023-10-16T15:54:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.603+0000 7fc665dee640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc64406b0b0 0x7fc64406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.604+0000 7fc665dee640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc64406b0b0 0x7fc64406d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc65c000c00 tx=0x7fc65c009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.606+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc658058b70 con 0x7fc6600b1f70 2023-10-16T15:54:51.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:51.800+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc6600c4de0 con 0x7fc6600b1f70 2023-10-16T15:54:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.359+0000 7fc6527fc640 1 -- 172.21.15.29:0/4139193122 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v248) v1 ==== 176+0+0 (secure 0 0 0) 0x7fc6600c4de0 con 0x7fc6600b1f70 2023-10-16T15:54:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' created 2023-10-16T15:54:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.364+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc64406b0b0 msgr2=0x7fc64406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.364+0000 7fc666df0640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc64406b0b0 0x7fc64406d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc65c000c00 tx=0x7fc65c009040 comp rx=0 tx=0).stop 2023-10-16T15:54:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.364+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 msgr2=0x7fc6600b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.364+0000 7fc666df0640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 0x7fc6600b23f0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fc658000f80 tx=0x7fc65800df30 comp rx=0 tx=0).stop 2023-10-16T15:54:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 shutdown_connections 2023-10-16T15:54:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fc64406b0b0 0x7fc64406d570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6600b1f70 0x7fc6600b23f0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc6600ccdd0 0x7fc6600cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 --2- 172.21.15.29:0/4139193122 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc64c0047c0 0x7fc660150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 >> 172.21.15.29:0/4139193122 conn(0x7fc6600ce6a0 msgr2=0x7fc66015f140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.365+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 shutdown_connections 2023-10-16T15:54:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.366+0000 7fc666df0640 1 -- 172.21.15.29:0/4139193122 wait complete. 2023-10-16T15:54:52.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T15:54:52.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.588+0000 7fe1adce5640 1 Processor -- start 2023-10-16T15:54:52.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.589+0000 7fe1adce5640 1 -- start start 2023-10-16T15:54:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.590+0000 7fe1adce5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a81618b0 0x7fe1a8065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.590+0000 7fe1adce5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a806cee0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.590+0000 7fe1adce5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80663d0 con 0x7fe1a815d330 2023-10-16T15:54:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.590+0000 7fe1adce5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a806b1e0 con 0x7fe1a81618b0 2023-10-16T15:54:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe1a8065ff0 0x7fe1a80ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52076/0 (socket says 172.21.15.29:52076) 2023-10-16T15:54:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a7fff640 1 -- 172.21.15.29:0/509603417 learned_addr learned my addr 172.21.15.29:0/509603417 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a6ffd640 1 --2- 172.21.15.29:0/509603417 >> v2:172.21.15.29:3300/0 conn(0x7fe1a81618b0 0x7fe1a8065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a67fc640 1 -- 172.21.15.29:0/509603417 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109240996 0 0) 0x7fe1a806cee0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a67fc640 1 -- 172.21.15.29:0/509603417 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe190003520 con 0x7fe1a8065ff0 2023-10-16T15:54:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a6ffd640 1 -- 172.21.15.29:0/509603417 >> v1:172.21.15.170:6789/0 conn(0x7fe1a815d330 legacy=0x7fe1a806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a6ffd640 1 -- 172.21.15.29:0/509603417 >> v1:172.21.15.29:6789/0 conn(0x7fe1a8065ff0 legacy=0x7fe1a80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.591+0000 7fe1a6ffd640 1 -- 172.21.15.29:0/509603417 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80d0890 con 0x7fe1a81618b0 2023-10-16T15:54:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1a6ffd640 1 --2- 172.21.15.29:0/509603417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a81618b0 0x7fe1a8065ab0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0097a0 tx=0x7fe19c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba6441112e6062b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1a67fc640 1 -- 172.21.15.29:0/509603417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c031830 con 0x7fe1a81618b0 2023-10-16T15:54:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1a67fc640 1 -- 172.21.15.29:0/509603417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c031990 con 0x7fe1a81618b0 2023-10-16T15:54:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1a67fc640 1 -- 172.21.15.29:0/509603417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c031d20 con 0x7fe1a81618b0 2023-10-16T15:54:52.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1adce5640 1 -- 172.21.15.29:0/509603417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a81618b0 msgr2=0x7fe1a8065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1adce5640 1 --2- 172.21.15.29:0/509603417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a81618b0 0x7fe1a8065ab0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0097a0 tx=0x7fe19c02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1adce5640 1 -- 172.21.15.29:0/509603417 shutdown_connections 2023-10-16T15:54:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1adce5640 1 --2- 172.21.15.29:0/509603417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a81618b0 0x7fe1a8065ab0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.593+0000 7fe1adce5640 1 -- 172.21.15.29:0/509603417 >> 172.21.15.29:0/509603417 conn(0x7fe1a80d0ab0 msgr2=0x7fe1a80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.594+0000 7fe1adce5640 1 -- 172.21.15.29:0/509603417 shutdown_connections 2023-10-16T15:54:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.594+0000 7fe1adce5640 1 -- 172.21.15.29:0/509603417 wait complete. 2023-10-16T15:54:52.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.595+0000 7fe1adce5640 1 Processor -- start 2023-10-16T15:54:52.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.595+0000 7fe1adce5640 1 -- start start 2023-10-16T15:54:52.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.596+0000 7fe1adce5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 0x7fe1a815db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:52.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.596+0000 7fe1adce5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1a815d330 0x7fe1a81e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.596+0000 7fe1adce5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1a81618b0 0x7fe1a81f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.596+0000 7fe1adce5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81e9030 con 0x7fe1a8065ff0 2023-10-16T15:54:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.596+0000 7fe1adce5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81e8420 con 0x7fe1a81618b0 2023-10-16T15:54:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.596+0000 7fe1adce5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81e8700 con 0x7fe1a815d330 2023-10-16T15:54:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 0x7fe1a815db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 0x7fe1a815db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47770/0 (socket says 172.21.15.29:47770) 2023-10-16T15:54:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 -- 172.21.15.29:0/582195026 learned_addr learned my addr 172.21.15.29:0/582195026 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a7fff640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1a81618b0 0x7fe1a81f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a6ffd640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1a815d330 0x7fe1a81e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:52.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 -- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1a815d330 msgr2=0x7fe1a81e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1a815d330 0x7fe1a81e7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 -- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1a81618b0 msgr2=0x7fe1a81f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:52.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1a81618b0 0x7fe1a81f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a77fe640 1 -- 172.21.15.29:0/582195026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c009400 con 0x7fe1a8065ff0 2023-10-16T15:54:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.597+0000 7fe1a6ffd640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1a815d330 0x7fe1a81e7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.598+0000 7fe1a77fe640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 0x7fe1a815db40 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fe19800c400 tx=0x7fe19800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.598+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980052d0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.598+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe198005430 con 0x7fe1a8065ff0 2023-10-16T15:54:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.598+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a81e89e0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.598+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198003830 con 0x7fe1a8065ff0 2023-10-16T15:54:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.598+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a815e080 con 0x7fe1a8065ff0 2023-10-16T15:54:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.599+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a80affe0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.601+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe198003990 con 0x7fe1a8065ff0 2023-10-16T15:54:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.601+0000 7fe18ffff640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe19406b0b0 0x7fe19406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.602+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6495+0+0 (secure 0 0 0) 0x7fe1980896d0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.602+0000 7fe1a6ffd640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe19406b0b0 0x7fe19406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.603+0000 7fe1a6ffd640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe19406b0b0 0x7fe19406d570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe19c036000 tx=0x7fe19c03c040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:52.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.605+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe1980541c0 con 0x7fe1a8065ff0 2023-10-16T15:54:52.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:52.834+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fe1a815e210 con 0x7fe1a8065ff0 2023-10-16T15:54:53.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.391+0000 7fe18ffff640 1 -- 172.21.15.29:0/582195026 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v126) v1 ==== 183+0+0 (secure 0 0 0) 0x7fe1980170d0 con 0x7fe1a8065ff0 2023-10-16T15:54:53.392 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 32 and data pool 33 2023-10-16T15:54:53.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.396+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe19406b0b0 msgr2=0x7fe19406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.396+0000 7fe1adce5640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe19406b0b0 0x7fe19406d570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe19c036000 tx=0x7fe19c03c040 comp rx=0 tx=0).stop 2023-10-16T15:54:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.396+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 msgr2=0x7fe1a815db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.396+0000 7fe1adce5640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 0x7fe1a815db40 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fe19800c400 tx=0x7fe19800c8c0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.398+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 shutdown_connections 2023-10-16T15:54:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.398+0000 7fe1adce5640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe19406b0b0 0x7fe19406d570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.398+0000 7fe1adce5640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1a81618b0 0x7fe1a81f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.398+0000 7fe1adce5640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1a815d330 0x7fe1a81e7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.398+0000 7fe1adce5640 1 --2- 172.21.15.29:0/582195026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a8065ff0 0x7fe1a815db40 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.398+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 >> 172.21.15.29:0/582195026 conn(0x7fe1a80d0ab0 msgr2=0x7fe1a8159110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:53.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.400+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 shutdown_connections 2023-10-16T15:54:53.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.400+0000 7fe1adce5640 1 -- 172.21.15.29:0/582195026 wait complete. 2023-10-16T15:54:53.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.613+0000 7ff6f4c7c640 1 Processor -- start 2023-10-16T15:54:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.613+0000 7ff6f4c7c640 1 -- start start 2023-10-16T15:54:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6f4c7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6f015ae80 0x7ff6f015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6f4c7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f015ed60 con 0x7ff6f00ce390 2023-10-16T15:54:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6f4c7c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00707e0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6f4c7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f006b810 con 0x7ff6f015ae80 2023-10-16T15:54:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6eed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6f00ce390 0x7ff6f0070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52084/0 (socket says 172.21.15.29:52084) 2023-10-16T15:54:53.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6eed9d640 1 -- 172.21.15.29:0/217761475 learned_addr learned my addr 172.21.15.29:0/217761475 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:53.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.615+0000 7ff6edd9b640 1 --2- 172.21.15.29:0/217761475 >> v2:172.21.15.29:3300/0 conn(0x7ff6f015ae80 0x7ff6f015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.616+0000 7ff6ed59a640 1 -- 172.21.15.29:0/217761475 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188837845 0 0) 0x7ff6f015ed60 con 0x7ff6f00ce390 2023-10-16T15:54:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.616+0000 7ff6edd9b640 1 -- 172.21.15.29:0/217761475 >> v1:172.21.15.170:6789/0 conn(0x7ff6f015aaa0 legacy=0x7ff6f00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.616+0000 7ff6edd9b640 1 -- 172.21.15.29:0/217761475 >> v1:172.21.15.29:6789/0 conn(0x7ff6f00ce390 legacy=0x7ff6f0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.616+0000 7ff6edd9b640 1 -- 172.21.15.29:0/217761475 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f015ef90 con 0x7ff6f015ae80 2023-10-16T15:54:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.617+0000 7ff6edd9b640 1 --2- 172.21.15.29:0/217761475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015ae80 0x7ff6f015e820 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009d90 tx=0x7ff6d802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d765f45d6fa252c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.617+0000 7ff6ed59a640 1 -- 172.21.15.29:0/217761475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d802cd00 con 0x7ff6f015ae80 2023-10-16T15:54:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.617+0000 7ff6ed59a640 1 -- 172.21.15.29:0/217761475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d802ce60 con 0x7ff6f015ae80 2023-10-16T15:54:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.617+0000 7ff6ed59a640 1 -- 172.21.15.29:0/217761475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8034660 con 0x7ff6f015ae80 2023-10-16T15:54:53.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/217761475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015ae80 msgr2=0x7ff6f015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/217761475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015ae80 0x7ff6f015e820 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009d90 tx=0x7ff6d802bf50 comp rx=0 tx=0).stop 2023-10-16T15:54:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/217761475 shutdown_connections 2023-10-16T15:54:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/217761475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015ae80 0x7ff6f015e820 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/217761475 >> 172.21.15.29:0/217761475 conn(0x7ff6f00cb410 msgr2=0x7ff6f00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/217761475 shutdown_connections 2023-10-16T15:54:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.618+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/217761475 wait complete. 2023-10-16T15:54:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.620+0000 7ff6f4c7c640 1 Processor -- start 2023-10-16T15:54:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.620+0000 7ff6f4c7c640 1 -- start start 2023-10-16T15:54:53.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6f4c7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6f00ce390 0x7ff6f014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:53.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6f4c7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 0x7ff6f01f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:53.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6f4c7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6f015ae80 0x7ff6f01f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6f4c7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00c5380 con 0x7ff6f015aaa0 2023-10-16T15:54:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6f4c7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f014ecf0 con 0x7ff6f015ae80 2023-10-16T15:54:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6f4c7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00c4ae0 con 0x7ff6f00ce390 2023-10-16T15:54:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 0x7ff6f01f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 0x7ff6f01f37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47792/0 (socket says 172.21.15.29:47792) 2023-10-16T15:54:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 -- 172.21.15.29:0/2078399167 learned_addr learned my addr 172.21.15.29:0/2078399167 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 -- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6f00ce390 msgr2=0x7ff6f014e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6f00ce390 0x7ff6f014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 -- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6f015ae80 msgr2=0x7ff6f01f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6f015ae80 0x7ff6f01f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.621+0000 7ff6edd9b640 1 -- 172.21.15.29:0/2078399167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d8009a70 con 0x7ff6f015aaa0 2023-10-16T15:54:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.622+0000 7ff6edd9b640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 0x7ff6f01f37b0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8000f80 tx=0x7ff6d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.622+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8003b90 con 0x7ff6f015aaa0 2023-10-16T15:54:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.622+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d8003cf0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.622+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8004230 con 0x7ff6f015aaa0 2023-10-16T15:54:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.622+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f00c4dc0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.622+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00c5c00 con 0x7ff6f015aaa0 2023-10-16T15:54:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.623+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f00c5a40 con 0x7ff6f015aaa0 2023-10-16T15:54:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.624+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff6d803b050 con 0x7ff6f015aaa0 2023-10-16T15:54:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.625+0000 7ff6e6ffd640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff6d006b0e0 0x7ff6d006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.625+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff6d80ac6f0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.626+0000 7ff6ee59c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff6d006b0e0 0x7ff6d006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.628+0000 7ff6ee59c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff6d006b0e0 0x7ff6d006d5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0000f60 tx=0x7ff6e00079b0 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.630+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6d80770c0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.816+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6f00c5df0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.818+0000 7ff6e6ffd640 1 -- 172.21.15.29:0/2078399167 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+20244 (secure 0 0 0) 0x7ff6d807a6e0 con 0x7ff6f015aaa0 2023-10-16T15:54:53.820 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:53.820 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":249,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:52.835510+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:53.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff6d006b0e0 msgr2=0x7ff6d006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff6d006b0e0 0x7ff6d006d5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0000f60 tx=0x7ff6e00079b0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 msgr2=0x7ff6f01f37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:53.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 0x7ff6f01f37b0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8000f80 tx=0x7ff6d8004a40 comp rx=0 tx=0).stop 2023-10-16T15:54:53.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 shutdown_connections 2023-10-16T15:54:53.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff6d006b0e0 0x7ff6d006d5a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f015aaa0 0x7ff6f01f37b0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6f015ae80 0x7ff6f01f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 --2- 172.21.15.29:0/2078399167 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6f00ce390 0x7ff6f014e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:53.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 >> 172.21.15.29:0/2078399167 conn(0x7ff6f00cb410 msgr2=0x7ff6f0159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:53.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 shutdown_connections 2023-10-16T15:54:53.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:53.821+0000 7ff6f4c7c640 1 -- 172.21.15.29:0/2078399167 wait complete. 2023-10-16T15:54:53.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.061+0000 7f6143fff640 1 Processor -- start 2023-10-16T15:54:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.061+0000 7f6143fff640 1 -- start start 2023-10-16T15:54:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f6143fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6144159950 0x7f614415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f6143fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6144070450 0x7f61440708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f6143fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614415aef0 con 0x7f614406c100 2023-10-16T15:54:54.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f6143fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6144070e20 con 0x7f6144070450 2023-10-16T15:54:54.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f6143fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614415e490 con 0x7f6144159950 2023-10-16T15:54:54.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f61427fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6144070450 0x7f61440708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f61437fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f614406c100 0x7f614406c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33232/0 (socket says 172.21.15.29:33232) 2023-10-16T15:54:54.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.063+0000 7f61437fe640 1 -- 172.21.15.29:0/1540768362 learned_addr learned my addr 172.21.15.29:0/1540768362 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f6142ffd640 1 --2- 172.21.15.29:0/1540768362 >> v2:172.21.15.170:3300/0 conn(0x7f6144159950 0x7f614415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f61427fc640 1 -- 172.21.15.29:0/1540768362 >> v2:172.21.15.170:3300/0 conn(0x7f6144159950 msgr2=0x7f614415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f61427fc640 1 --2- 172.21.15.29:0/1540768362 >> v2:172.21.15.170:3300/0 conn(0x7f6144159950 0x7f614415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f61427fc640 1 -- 172.21.15.29:0/1540768362 >> v1:172.21.15.29:6789/0 conn(0x7f614406c100 legacy=0x7f614406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f61427fc640 1 -- 172.21.15.29:0/1540768362 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614415d910 con 0x7f6144070450 2023-10-16T15:54:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f6142ffd640 1 --2- 172.21.15.29:0/1540768362 >> v2:172.21.15.170:3300/0 conn(0x7f6144159950 0x7f614415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.064+0000 7f6141ffb640 1 -- 172.21.15.29:0/1540768362 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457964774 0 0) 0x7f613c001490 con 0x7f614406c100 2023-10-16T15:54:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.065+0000 7f61427fc640 1 --2- 172.21.15.29:0/1540768362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6144070450 0x7f61440708e0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f61440ca950 tx=0x7f613802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971c02f2039d167f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.065+0000 7f6141ffb640 1 -- 172.21.15.29:0/1540768362 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613802e070 con 0x7f6144070450 2023-10-16T15:54:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.065+0000 7f6141ffb640 1 -- 172.21.15.29:0/1540768362 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61380315a0 con 0x7f6144070450 2023-10-16T15:54:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.065+0000 7f6141ffb640 1 -- 172.21.15.29:0/1540768362 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138031930 con 0x7f6144070450 2023-10-16T15:54:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 -- 172.21.15.29:0/1540768362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6144070450 msgr2=0x7f61440708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 --2- 172.21.15.29:0/1540768362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6144070450 0x7f61440708e0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f61440ca950 tx=0x7f613802b840 comp rx=0 tx=0).stop 2023-10-16T15:54:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 -- 172.21.15.29:0/1540768362 shutdown_connections 2023-10-16T15:54:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 --2- 172.21.15.29:0/1540768362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6144070450 0x7f61440708e0 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 --2- 172.21.15.29:0/1540768362 >> v2:172.21.15.170:3300/0 conn(0x7f6144159950 0x7f614415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 -- 172.21.15.29:0/1540768362 >> 172.21.15.29:0/1540768362 conn(0x7f61440ce6a0 msgr2=0x7f61440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 -- 172.21.15.29:0/1540768362 shutdown_connections 2023-10-16T15:54:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.066+0000 7f6143fff640 1 -- 172.21.15.29:0/1540768362 wait complete. 2023-10-16T15:54:54.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.068+0000 7f6143fff640 1 Processor -- start 2023-10-16T15:54:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.068+0000 7f6143fff640 1 -- start start 2023-10-16T15:54:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6143fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 0x7f61440c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6143fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6144070450 0x7f61440c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6143fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6144159950 0x7f61440c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6143fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614414b5e0 con 0x7f614406c100 2023-10-16T15:54:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6143fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440c2240 con 0x7f6144159950 2023-10-16T15:54:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6143fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440c6320 con 0x7f6144070450 2023-10-16T15:54:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6142ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 0x7f61440c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6142ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 0x7f61440c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36248/0 (socket says 172.21.15.29:36248) 2023-10-16T15:54:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.069+0000 7f6142ffd640 1 -- 172.21.15.29:0/368558023 learned_addr learned my addr 172.21.15.29:0/368558023 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f61437fe640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6144159950 0x7f61440c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6142ffd640 1 -- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6144070450 msgr2=0x7f61440c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6142ffd640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6144070450 0x7f61440c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6142ffd640 1 -- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6144159950 msgr2=0x7f61440c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6142ffd640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6144159950 0x7f61440c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6142ffd640 1 -- 172.21.15.29:0/368558023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138009400 con 0x7f614406c100 2023-10-16T15:54:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6142ffd640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 0x7f61440c4830 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f613400c7f0 tx=0x7f613400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61340077c0 con 0x7f614406c100 2023-10-16T15:54:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6134007920 con 0x7f614406c100 2023-10-16T15:54:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134007c90 con 0x7f614406c100 2023-10-16T15:54:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.070+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61440c65c0 con 0x7f614406c100 2023-10-16T15:54:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.071+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61440c6b80 con 0x7f614406c100 2023-10-16T15:54:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.072+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614404e9b0 con 0x7f614406c100 2023-10-16T15:54:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.072+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f61340113f0 con 0x7f614406c100 2023-10-16T15:54:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.073+0000 7f612b7fe640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f613006b0b0 0x7f613006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.074+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f613408a1c0 con 0x7f614406c100 2023-10-16T15:54:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.074+0000 7f61427fc640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f613006b0b0 0x7f613006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.076+0000 7f61427fc640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f613006b0b0 0x7f613006d570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f6138030000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.078+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6134054a70 con 0x7f614406c100 2023-10-16T15:54:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.278+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61440b4620 con 0x7f614406c100 2023-10-16T15:54:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.282+0000 7f612b7fe640 1 -- 172.21.15.29:0/368558023 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+20244 (secure 0 0 0) 0x7f6134058090 con 0x7f614406c100 2023-10-16T15:54:54.284 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:54.285 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":249,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:52.835510+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:54.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f613006b0b0 msgr2=0x7f613006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f613006b0b0 0x7f613006d570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f6138030000 comp rx=0 tx=0).stop 2023-10-16T15:54:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 msgr2=0x7f61440c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 0x7f61440c4830 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f613400c7f0 tx=0x7f613400ccb0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 shutdown_connections 2023-10-16T15:54:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f613006b0b0 0x7f613006d570 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6144159950 0x7f61440c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6144070450 0x7f61440c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 --2- 172.21.15.29:0/368558023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f614406c100 0x7f61440c4830 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 >> 172.21.15.29:0/368558023 conn(0x7f61440ce6a0 msgr2=0x7f61440ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.284+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 shutdown_connections 2023-10-16T15:54:54.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.285+0000 7f6143fff640 1 -- 172.21.15.29:0/368558023 wait complete. 2023-10-16T15:54:54.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T15:54:54.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.532+0000 7ff0809e1640 1 Processor -- start 2023-10-16T15:54:54.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.533+0000 7ff0809e1640 1 -- start start 2023-10-16T15:54:54.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff0809e1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff07c0cf1c0 0x7ff07c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff0809e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff0809e1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c0cdbc0 con 0x7ff07c1582f0 2023-10-16T15:54:54.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff0809e1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07c0cc5e0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff0809e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c0cc8c0 con 0x7ff07c06f440 2023-10-16T15:54:54.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff07ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff07c0cf1c0 0x7ff07c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff07b59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff07c1582f0 0x7ff07c158780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33240/0 (socket says 172.21.15.29:33240) 2023-10-16T15:54:54.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.534+0000 7ff07b59e640 1 -- 172.21.15.29:0/1100661182 learned_addr learned my addr 172.21.15.29:0/1100661182 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:54.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff073fff640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff07a59c640 1 -- 172.21.15.29:0/1100661182 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030901092 0 0) 0x7ff07c0cdbc0 con 0x7ff07c1582f0 2023-10-16T15:54:54.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff07a59c640 1 -- 172.21.15.29:0/1100661182 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff068003520 con 0x7ff07c1582f0 2023-10-16T15:54:54.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff07ad9d640 1 -- 172.21.15.29:0/1100661182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 msgr2=0x7ff07c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:54.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff07ad9d640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff07ad9d640 1 -- 172.21.15.29:0/1100661182 >> v1:172.21.15.29:6789/0 conn(0x7ff07c1582f0 legacy=0x7ff07c158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff07ad9d640 1 -- 172.21.15.29:0/1100661182 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c0ccba0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.535+0000 7ff073fff640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:54.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.536+0000 7ff07ad9d640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c06a910 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097a0 tx=0x7ff06c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa142ba89c2fec82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:54.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.537+0000 7ff07a59c640 1 -- 172.21.15.29:0/1100661182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c02d040 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.537+0000 7ff07a59c640 1 -- 172.21.15.29:0/1100661182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c031810 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.537+0000 7ff07a59c640 1 -- 172.21.15.29:0/1100661182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c031ba0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 -- 172.21.15.29:0/1100661182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 msgr2=0x7ff07c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c06a910 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097a0 tx=0x7ff06c02b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 -- 172.21.15.29:0/1100661182 shutdown_connections 2023-10-16T15:54:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1100661182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c06a910 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 -- 172.21.15.29:0/1100661182 >> 172.21.15.29:0/1100661182 conn(0x7ff07c0ce6a0 msgr2=0x7ff07c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 -- 172.21.15.29:0/1100661182 shutdown_connections 2023-10-16T15:54:54.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.538+0000 7ff0809e1640 1 -- 172.21.15.29:0/1100661182 wait complete. 2023-10-16T15:54:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.540+0000 7ff0809e1640 1 Processor -- start 2023-10-16T15:54:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.540+0000 7ff0809e1640 1 -- start start 2023-10-16T15:54:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff0809e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0c0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff0809e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c0c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff0809e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff07c1582f0 0x7ff07c06b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff0809e1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1f6680 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff0809e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1f5a00 con 0x7ff07c1582f0 2023-10-16T15:54:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff0809e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1f5ce0 con 0x7ff07c06f440 2023-10-16T15:54:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff073fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c0c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff073fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c0c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36276/0 (socket says 172.21.15.29:36276) 2023-10-16T15:54:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.541+0000 7ff073fff640 1 -- 172.21.15.29:0/1282298161 learned_addr learned my addr 172.21.15.29:0/1282298161 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff07ad9d640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0c0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff07b59e640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff07c1582f0 0x7ff07c06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff073fff640 1 -- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 msgr2=0x7ff07c0c0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff073fff640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0c0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff073fff640 1 -- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff07c1582f0 msgr2=0x7ff07c06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff073fff640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff07c1582f0 0x7ff07c06b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff073fff640 1 -- 172.21.15.29:0/1282298161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c009400 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff07b59e640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff07c1582f0 0x7ff07c06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff073fff640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c0c21d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7ff06400c7f0 tx=0x7ff06400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064007820 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064007980 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.542+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064007cf0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.543+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c1f5fc0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.543+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c0c1340 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.544+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c0c3e00 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.545+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff0640113f0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.546+0000 7ff0737fe640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff05c06b0b0 0x7ff05c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.546+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff06408a1f0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.547+0000 7ff07ad9d640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff05c06b0b0 0x7ff05c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.548+0000 7ff07ad9d640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff05c06b0b0 0x7ff05c06d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff06c036000 tx=0x7ff06c03c040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:54.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.550+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff064054aa0 con 0x7ff07c0cf1c0 2023-10-16T15:54:54.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:54.789+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff07c0cd160 con 0x7ff07c0cf1c0 2023-10-16T15:54:55.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.415+0000 7ff0737fe640 1 -- 172.21.15.29:0/1282298161 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v129) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff0640580c0 con 0x7ff07c0cf1c0 2023-10-16T15:54:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.419+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff05c06b0b0 msgr2=0x7ff05c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.419+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff05c06b0b0 0x7ff05c06d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff06c036000 tx=0x7ff06c03c040 comp rx=0 tx=0).stop 2023-10-16T15:54:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.419+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 msgr2=0x7ff07c0c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.419+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c0c21d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7ff06400c7f0 tx=0x7ff06400ccb0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.419+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 shutdown_connections 2023-10-16T15:54:55.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.419+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff05c06b0b0 0x7ff05c06d570 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.420+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff07c0cf1c0 0x7ff07c0c21d0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.420+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff07c1582f0 0x7ff07c06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.420+0000 7ff0809e1640 1 --2- 172.21.15.29:0/1282298161 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff07c06f440 0x7ff07c0c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.420+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 >> 172.21.15.29:0/1282298161 conn(0x7ff07c0ce6a0 msgr2=0x7ff07c15cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:55.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.420+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 shutdown_connections 2023-10-16T15:54:55.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.420+0000 7ff0809e1640 1 -- 172.21.15.29:0/1282298161 wait complete. 2023-10-16T15:54:55.441 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:54:55.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T15:54:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.678+0000 7f1f83fff640 1 Processor -- start 2023-10-16T15:54:55.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.678+0000 7f1f83fff640 1 -- start start 2023-10-16T15:54:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.679+0000 7f1f83fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 0x7f1f8406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.679+0000 7f1f83fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8415f4a0 con 0x7f1f840ad970 2023-10-16T15:54:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.679+0000 7f1f83fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8406c960 con 0x7f1f840cda80 2023-10-16T15:54:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.679+0000 7f1f83fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8415b9d0 con 0x7f1f8406c040 2023-10-16T15:54:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.679+0000 7f1f827fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1f840ad970 0x7f1f8415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33254/0 (socket says 172.21.15.29:33254) 2023-10-16T15:54:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.679+0000 7f1f827fc640 1 -- 172.21.15.29:0/543363771 learned_addr learned my addr 172.21.15.29:0/543363771 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f82ffd640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 0x7f1f8406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431061531 0 0) 0x7f1f8415f4a0 con 0x7f1f840ad970 2023-10-16T15:54:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c003520 con 0x7f1f840ad970 2023-10-16T15:54:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738820410 0 0) 0x7f1f6c003520 con 0x7f1f840ad970 2023-10-16T15:54:55.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 msgr2=0x7f1f8406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:55.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 0x7f1f8406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 >> v1:172.21.15.170:6789/0 conn(0x7f1f840cda80 legacy=0x7f1f8415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8415bcb0 con 0x7f1f840ad970 2023-10-16T15:54:55.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1f78001670 con 0x7f1f840ad970 2023-10-16T15:54:55.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 >> v1:172.21.15.29:6789/0 conn(0x7f1f840ad970 legacy=0x7f1f8415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.680+0000 7f1f81ffb640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f6c0039a0 0x7f1f6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f81ffb640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f81ffb640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 0x7f1f6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8415b9d0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c00ba30 con 0x7f1f6c0039a0 2023-10-16T15:54:55.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c00bbf0 con 0x7f1f8406c040 2023-10-16T15:54:55.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f827fc640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f82ffd640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f6c0039a0 0x7f1f6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f837fe640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 0x7f1f6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f827fc640 1 -- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 msgr2=0x7f1f6c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f827fc640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f8406c040 0x7f1f6c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f827fc640 1 -- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f6c0039a0 msgr2=0x7f1f6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f827fc640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f6c0039a0 0x7f1f6c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f827fc640 1 -- 172.21.15.29:0/543363771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c003520 con 0x7f1f6c0047c0 2023-10-16T15:54:55.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.681+0000 7f1f82ffd640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f6c0039a0 0x7f1f6c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:55.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.682+0000 7f1f827fc640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f6c004c40 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c210 tx=0x7f1f6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7a5a0cf573732b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:55.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78005ad0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f78005c50 con 0x7f1f6c0047c0 2023-10-16T15:54:55.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f81ffb640 1 -- 172.21.15.29:0/543363771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7802c870 con 0x7f1f6c0047c0 2023-10-16T15:54:55.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f83fff640 1 -- 172.21.15.29:0/543363771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 msgr2=0x7f1f6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f83fff640 1 --2- 172.21.15.29:0/543363771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f6c004c40 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c210 tx=0x7f1f6c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f837fe640 1 -- 172.21.15.29:0/543363771 reap_dead start 2023-10-16T15:54:55.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f83fff640 1 -- 172.21.15.29:0/543363771 shutdown_connections 2023-10-16T15:54:55.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.683+0000 7f1f83fff640 1 -- 172.21.15.29:0/543363771 >> 172.21.15.29:0/543363771 conn(0x7f1f840ce6a0 msgr2=0x7f1f840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:55.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.684+0000 7f1f83fff640 1 -- 172.21.15.29:0/543363771 shutdown_connections 2023-10-16T15:54:55.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.684+0000 7f1f83fff640 1 -- 172.21.15.29:0/543363771 wait complete. 2023-10-16T15:54:55.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.685+0000 7f1f83fff640 1 Processor -- start 2023-10-16T15:54:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.686+0000 7f1f83fff640 1 -- start start 2023-10-16T15:54:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.686+0000 7f1f83fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f841f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.686+0000 7f1f83fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f840b25c0 0x7f1f840b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.686+0000 7f1f83fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f840ac890 0x7f1f840acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f83fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840ad250 con 0x7f1f6c0047c0 2023-10-16T15:54:55.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f83fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840b3010 con 0x7f1f840ac890 2023-10-16T15:54:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f83fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f841f5900 con 0x7f1f840b25c0 2023-10-16T15:54:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f841f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f841f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36296/0 (socket says 172.21.15.29:36296) 2023-10-16T15:54:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 -- 172.21.15.29:0/3745968715 learned_addr learned my addr 172.21.15.29:0/3745968715 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:55.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f837fe640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f840ac890 0x7f1f840acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f827fc640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f840b25c0 0x7f1f840b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 -- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f840b25c0 msgr2=0x7f1f840b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f840b25c0 0x7f1f840b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 -- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f840ac890 msgr2=0x7f1f840acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:55.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f840ac890 0x7f1f840acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:55.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f82ffd640 1 -- 172.21.15.29:0/3745968715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f841f5be0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f837fe640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f840ac890 0x7f1f840acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:55.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.687+0000 7f1f827fc640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f840b25c0 0x7f1f840b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:55.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.688+0000 7f1f82ffd640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f841f1bf0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f1f740116d0 tx=0x7f1f74011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:55.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.688+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7400f9c0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.688+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7400fb20 con 0x7f1f6c0047c0 2023-10-16T15:54:55.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.688+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7400fed0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.688+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f841f5ec0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.688+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f841f2130 con 0x7f1f6c0047c0 2023-10-16T15:54:55.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.690+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8406b330 con 0x7f1f6c0047c0 2023-10-16T15:54:55.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.690+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1f74015460 con 0x7f1f6c0047c0 2023-10-16T15:54:55.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.692+0000 7f1f6b7fe640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f1f7006b0e0 0x7f1f7006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:55.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.692+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f1f7408a6b0 con 0x7f1f6c0047c0 2023-10-16T15:54:55.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.693+0000 7f1f827fc640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f1f7006b0e0 0x7f1f7006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:55.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.694+0000 7f1f827fc640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f1f7006b0e0 0x7f1f7006d5a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1f78034000 tx=0x7f1f78009030 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:55.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.696+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1f74055080 con 0x7f1f6c0047c0 2023-10-16T15:54:55.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:55.929+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1f8404e9b0 con 0x7f1f6c0047c0 2023-10-16T15:54:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.426+0000 7f1f6b7fe640 1 -- 172.21.15.29:0/3745968715 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v130) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1f740586a0 con 0x7f1f6c0047c0 2023-10-16T15:54:56.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f1f7006b0e0 msgr2=0x7f1f7006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f1f7006b0e0 0x7f1f7006d5a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1f78034000 tx=0x7f1f78009030 comp rx=0 tx=0).stop 2023-10-16T15:54:56.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 msgr2=0x7f1f841f1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f841f1bf0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f1f740116d0 tx=0x7f1f74011b90 comp rx=0 tx=0).stop 2023-10-16T15:54:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 shutdown_connections 2023-10-16T15:54:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f1f7006b0e0 0x7f1f7006d5a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1f840ac890 0x7f1f840acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1f840b25c0 0x7f1f840b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 --2- 172.21.15.29:0/3745968715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f6c0047c0 0x7f1f841f1bf0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 >> 172.21.15.29:0/3745968715 conn(0x7f1f840ce6a0 msgr2=0x7f1f840cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 shutdown_connections 2023-10-16T15:54:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.431+0000 7f1f83fff640 1 -- 172.21.15.29:0/3745968715 wait complete. 2023-10-16T15:54:56.455 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T15:54:56.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.685+0000 7f603dc52640 1 Processor -- start 2023-10-16T15:54:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.685+0000 7f603dc52640 1 -- start start 2023-10-16T15:54:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603dc52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f603806c040 0x7f603806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603dc52640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60380ad970 0x7f603815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603dc52640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603806c960 con 0x7f603815ebc0 2023-10-16T15:54:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603dc52640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603815b9d0 con 0x7f60380ad970 2023-10-16T15:54:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603dc52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603815bcb0 con 0x7f603806c040 2023-10-16T15:54:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f6037fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60380ad970 0x7f603815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603d451640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f603815ebc0 0x7f60380cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33264/0 (socket says 172.21.15.29:33264) 2023-10-16T15:54:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603d451640 1 -- 172.21.15.29:0/4074430495 learned_addr learned my addr 172.21.15.29:0/4074430495 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.687+0000 7f603cc50640 1 --2- 172.21.15.29:0/4074430495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f603806c040 0x7f603806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.688+0000 7f6037fff640 1 -- 172.21.15.29:0/4074430495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f603806c040 msgr2=0x7f603806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.688+0000 7f6037fff640 1 --2- 172.21.15.29:0/4074430495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f603806c040 0x7f603806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.688+0000 7f6037fff640 1 -- 172.21.15.29:0/4074430495 >> v1:172.21.15.29:6789/0 conn(0x7f603815ebc0 legacy=0x7f60380cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.688+0000 7f6037fff640 1 -- 172.21.15.29:0/4074430495 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603815bf90 con 0x7f60380ad970 2023-10-16T15:54:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.689+0000 7f6037fff640 1 --2- 172.21.15.29:0/4074430495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60380ad970 0x7f603815e680 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f60200097d0 tx=0x7f602002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf94f67b3b3ede8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.689+0000 7f60377fe640 1 -- 172.21.15.29:0/4074430495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602002d040 con 0x7f60380ad970 2023-10-16T15:54:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.689+0000 7f60377fe640 1 -- 172.21.15.29:0/4074430495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6020031810 con 0x7f60380ad970 2023-10-16T15:54:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.689+0000 7f60377fe640 1 -- 172.21.15.29:0/4074430495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020031ba0 con 0x7f60380ad970 2023-10-16T15:54:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 -- 172.21.15.29:0/4074430495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60380ad970 msgr2=0x7f603815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 --2- 172.21.15.29:0/4074430495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60380ad970 0x7f603815e680 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f60200097d0 tx=0x7f602002b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 -- 172.21.15.29:0/4074430495 shutdown_connections 2023-10-16T15:54:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 --2- 172.21.15.29:0/4074430495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60380ad970 0x7f603815e680 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 --2- 172.21.15.29:0/4074430495 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f603806c040 0x7f603806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 -- 172.21.15.29:0/4074430495 >> 172.21.15.29:0/4074430495 conn(0x7f60380ce6a0 msgr2=0x7f60380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:56.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 -- 172.21.15.29:0/4074430495 shutdown_connections 2023-10-16T15:54:56.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.690+0000 7f603dc52640 1 -- 172.21.15.29:0/4074430495 wait complete. 2023-10-16T15:54:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.692+0000 7f603dc52640 1 Processor -- start 2023-10-16T15:54:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.692+0000 7f603dc52640 1 -- start start 2023-10-16T15:54:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603dc52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 0x7f6038153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603dc52640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60380ad970 0x7f6038154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603dc52640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f603815ebc0 0x7f6038159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603dc52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380c7500 con 0x7f603806c040 2023-10-16T15:54:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603dc52640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380c68f0 con 0x7f603815ebc0 2023-10-16T15:54:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603dc52640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380c6bd0 con 0x7f60380ad970 2023-10-16T15:54:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603cc50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 0x7f6038153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603cc50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 0x7f6038153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36310/0 (socket says 172.21.15.29:36310) 2023-10-16T15:54:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.693+0000 7f603cc50640 1 -- 172.21.15.29:0/4012529934 learned_addr learned my addr 172.21.15.29:0/4012529934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603cc50640 1 -- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60380ad970 msgr2=0x7f6038154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603cc50640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60380ad970 0x7f6038154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603cc50640 1 -- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f603815ebc0 msgr2=0x7f6038159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603cc50640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f603815ebc0 0x7f6038159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603cc50640 1 -- 172.21.15.29:0/4012529934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020009400 con 0x7f603806c040 2023-10-16T15:54:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603cc50640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 0x7f6038153eb0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f6028007e40 tx=0x7f602800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028014620 con 0x7f603806c040 2023-10-16T15:54:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602800ee90 con 0x7f603806c040 2023-10-16T15:54:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60280149c0 con 0x7f603806c040 2023-10-16T15:54:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60380c6eb0 con 0x7f603806c040 2023-10-16T15:54:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.694+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6038155220 con 0x7f603806c040 2023-10-16T15:54:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.696+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603806b330 con 0x7f603806c040 2023-10-16T15:54:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.696+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6028014b20 con 0x7f603806c040 2023-10-16T15:54:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.697+0000 7f60357fa640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f601806b0b0 0x7f601806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.698+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f602808a680 con 0x7f603806c040 2023-10-16T15:54:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.698+0000 7f6037fff640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f601806b0b0 0x7f601806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.699+0000 7f6037fff640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f601806b0b0 0x7f601806d570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6020036000 tx=0x7f6020030000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:56.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.702+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6028055110 con 0x7f603806c040 2023-10-16T15:54:56.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.931+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603804e9b0 con 0x7f603806c040 2023-10-16T15:54:56.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.932+0000 7f60357fa640 1 -- 172.21.15.29:0/4012529934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f6028058730 con 0x7f603806c040 2023-10-16T15:54:56.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:56.934 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:54:56.934 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 130 2023-10-16T15:54:56.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f601806b0b0 msgr2=0x7f601806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f601806b0b0 0x7f601806d570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6020036000 tx=0x7f6020030000 comp rx=0 tx=0).stop 2023-10-16T15:54:56.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 msgr2=0x7f6038153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:56.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 0x7f6038153eb0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f6028007e40 tx=0x7f602800e4b0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 shutdown_connections 2023-10-16T15:54:56.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f601806b0b0 0x7f601806d570 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f603815ebc0 0x7f6038159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f60380ad970 0x7f6038154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 --2- 172.21.15.29:0/4012529934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f603806c040 0x7f6038153eb0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:56.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 >> 172.21.15.29:0/4012529934 conn(0x7f60380ce6a0 msgr2=0x7f6038069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:56.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 shutdown_connections 2023-10-16T15:54:56.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:56.935+0000 7f603dc52640 1 -- 172.21.15.29:0/4012529934 wait complete. 2023-10-16T15:54:56.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:54:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.182+0000 7f6afbfff640 1 Processor -- start 2023-10-16T15:54:57.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.182+0000 7f6afbfff640 1 -- start start 2023-10-16T15:54:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afbfff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6afc160c80 0x7f6afc161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6afc140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afbfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc0ae1a0 con 0x7f6afc0ad7a0 2023-10-16T15:54:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afbfff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6afc0ae480 con 0x7f6afc160c80 2023-10-16T15:54:57.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0ae760 con 0x7f6afc070950 2023-10-16T15:54:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afa7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6afc0ad7a0 0x7f6afc070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33270/0 (socket says 172.21.15.29:33270) 2023-10-16T15:54:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.184+0000 7f6afa7fc640 1 -- 172.21.15.29:0/3425539984 learned_addr learned my addr 172.21.15.29:0/3425539984 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6afaffd640 1 --2- 172.21.15.29:0/3425539984 >> v2:172.21.15.170:3300/0 conn(0x7f6afc160c80 0x7f6afc161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6afb7fe640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6afc140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615106764 0 0) 0x7f6afc0ae1a0 con 0x7f6afc0ad7a0 2023-10-16T15:54:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae8003520 con 0x7f6afc0ad7a0 2023-10-16T15:54:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816816794 0 0) 0x7f6ae8003520 con 0x7f6afc0ad7a0 2023-10-16T15:54:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 msgr2=0x7f6afc140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6afc140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 >> v2:172.21.15.170:3300/0 conn(0x7f6afc160c80 msgr2=0x7f6afc161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 --2- 172.21.15.29:0/3425539984 >> v2:172.21.15.170:3300/0 conn(0x7f6afc160c80 0x7f6afc161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc0aea40 con 0x7f6afc0ad7a0 2023-10-16T15:54:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f6ae4001670 con 0x7f6afc0ad7a0 2023-10-16T15:54:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 >> v1:172.21.15.29:6789/0 conn(0x7f6afc0ad7a0 legacy=0x7f6afc070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ae8003970 0x7f6ae8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6ae8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6ae8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0ae480 con 0x7f6ae80047c0 2023-10-16T15:54:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0ae760 con 0x7f6ae8003970 2023-10-16T15:54:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.185+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae800b980 con 0x7f6afc070950 2023-10-16T15:54:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6ae8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afb7fe640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6ae8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afaffd640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ae8003970 0x7f6ae8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afa7fc640 1 -- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 msgr2=0x7f6ae8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6ae8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afa7fc640 1 -- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ae8003970 msgr2=0x7f6ae8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ae8003970 0x7f6ae8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afa7fc640 1 -- 172.21.15.29:0/3425539984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae8003520 con 0x7f6ae80047c0 2023-10-16T15:54:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afaffd640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ae8003970 0x7f6ae8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.186+0000 7f6afb7fe640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc070950 0x7f6ae8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.187+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6ae8004c40 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400c240 tx=0x7f6ae80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffca8c2c964b6486 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.187+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae402cb00 con 0x7f6ae80047c0 2023-10-16T15:54:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae402cc60 con 0x7f6ae80047c0 2023-10-16T15:54:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6af9ffb640 1 -- 172.21.15.29:0/3425539984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40058d0 con 0x7f6ae80047c0 2023-10-16T15:54:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afbfff640 1 -- 172.21.15.29:0/3425539984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 msgr2=0x7f6ae8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afbfff640 1 --2- 172.21.15.29:0/3425539984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6ae8004c40 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400c240 tx=0x7f6ae80057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afb7fe640 1 -- 172.21.15.29:0/3425539984 reap_dead start 2023-10-16T15:54:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afbfff640 1 -- 172.21.15.29:0/3425539984 shutdown_connections 2023-10-16T15:54:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afbfff640 1 -- 172.21.15.29:0/3425539984 >> 172.21.15.29:0/3425539984 conn(0x7f6afc0cc230 msgr2=0x7f6afc0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afbfff640 1 -- 172.21.15.29:0/3425539984 shutdown_connections 2023-10-16T15:54:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.188+0000 7f6afbfff640 1 -- 172.21.15.29:0/3425539984 wait complete. 2023-10-16T15:54:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.190+0000 7f6afbfff640 1 Processor -- start 2023-10-16T15:54:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.190+0000 7f6afbfff640 1 -- start start 2023-10-16T15:54:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.191+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6afc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.191+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc1f35f0 0x7f6afc1f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.191+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6afc145a90 0x7f6afc153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.191+0000 7f6afbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc1525e0 con 0x7f6ae80047c0 2023-10-16T15:54:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.191+0000 7f6afbfff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc1f30e0 con 0x7f6afc145a90 2023-10-16T15:54:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.191+0000 7f6afbfff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc151cb0 con 0x7f6afc1f35f0 2023-10-16T15:54:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6afc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6afc06d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36338/0 (socket says 172.21.15.29:36338) 2023-10-16T15:54:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 -- 172.21.15.29:0/28028337 learned_addr learned my addr 172.21.15.29:0/28028337 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc1f35f0 0x7f6afc1f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afb7fe640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6afc145a90 0x7f6afc153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 -- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc1f35f0 msgr2=0x7f6afc1f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc1f35f0 0x7f6afc1f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 -- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6afc145a90 msgr2=0x7f6afc153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6afc145a90 0x7f6afc153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 -- 172.21.15.29:0/28028337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc151f90 con 0x7f6ae80047c0 2023-10-16T15:54:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afb7fe640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6afc145a90 0x7f6afc153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.192+0000 7f6afaffd640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6afc06d5a0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00f3d0 tx=0x7f6aec00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.193+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec01a070 con 0x7f6ae80047c0 2023-10-16T15:54:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.193+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec003990 con 0x7f6ae80047c0 2023-10-16T15:54:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.193+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec003d00 con 0x7f6ae80047c0 2023-10-16T15:54:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.193+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc152270 con 0x7f6ae80047c0 2023-10-16T15:54:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.193+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc145f10 con 0x7f6ae80047c0 2023-10-16T15:54:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.194+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc04e9b0 con 0x7f6ae80047c0 2023-10-16T15:54:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.195+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6aec004020 con 0x7f6ae80047c0 2023-10-16T15:54:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.196+0000 7f6af37fe640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6adc06b0e0 0x7f6adc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.196+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f6aec08aab0 con 0x7f6ae80047c0 2023-10-16T15:54:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.197+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6adc06b0e0 0x7f6adc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.198+0000 7f6afa7fc640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6adc06b0e0 0x7f6adc06d5a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4034000 tx=0x7f6ae4007000 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.201+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6aec055400 con 0x7f6ae80047c0 2023-10-16T15:54:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.397+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6afc0c5fa0 con 0x7f6ae80047c0 2023-10-16T15:54:57.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.399+0000 7f6af37fe640 1 -- 172.21.15.29:0/28028337 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+20231 (secure 0 0 0) 0x7f6aec003e60 con 0x7f6ae80047c0 2023-10-16T15:54:57.400 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:57.400 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":250,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:54:54.390804+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:54:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6adc06b0e0 msgr2=0x7f6adc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6adc06b0e0 0x7f6adc06d5a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4034000 tx=0x7f6ae4007000 comp rx=0 tx=0).stop 2023-10-16T15:54:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 msgr2=0x7f6afc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6afc06d5a0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00f3d0 tx=0x7f6aec00f890 comp rx=0 tx=0).stop 2023-10-16T15:54:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 shutdown_connections 2023-10-16T15:54:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f6adc06b0e0 0x7f6adc06d5a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6afc145a90 0x7f6afc153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6afc1f35f0 0x7f6afc1f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 --2- 172.21.15.29:0/28028337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae80047c0 0x7f6afc06d5a0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 >> 172.21.15.29:0/28028337 conn(0x7f6afc0cc230 msgr2=0x7f6afc0ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 shutdown_connections 2023-10-16T15:54:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.402+0000 7f6afbfff640 1 -- 172.21.15.29:0/28028337 wait complete. 2023-10-16T15:54:57.423 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:54:57.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:54:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.647+0000 7fe463fff640 1 Processor -- start 2023-10-16T15:54:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.647+0000 7fe463fff640 1 -- start start 2023-10-16T15:54:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.649+0000 7fe463fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4640d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.649+0000 7fe463fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe4640b4c80 0x7fe4640b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.649+0000 7fe463fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640d15a0 con 0x7fe46415cfc0 2023-10-16T15:54:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.649+0000 7fe463fff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640d1880 con 0x7fe4640b4c80 2023-10-16T15:54:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.649+0000 7fe463fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640d1b60 con 0x7fe46415cbe0 2023-10-16T15:54:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.649+0000 7fe462ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4640d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe462ffd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4640d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:53190/0 (socket says 172.21.15.29:53190) 2023-10-16T15:54:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe462ffd640 1 -- 172.21.15.29:0/1298829799 learned_addr learned my addr 172.21.15.29:0/1298829799 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe4637fe640 1 --2- 172.21.15.29:0/1298829799 >> v2:172.21.15.170:3300/0 conn(0x7fe4640b4c80 0x7fe4640b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe462ffd640 1 -- 172.21.15.29:0/1298829799 >> v2:172.21.15.170:3300/0 conn(0x7fe4640b4c80 msgr2=0x7fe4640b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe462ffd640 1 --2- 172.21.15.29:0/1298829799 >> v2:172.21.15.170:3300/0 conn(0x7fe4640b4c80 0x7fe4640b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe462ffd640 1 -- 172.21.15.29:0/1298829799 >> v1:172.21.15.170:6789/0 conn(0x7fe46415cfc0 legacy=0x7fe4640b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.650+0000 7fe462ffd640 1 -- 172.21.15.29:0/1298829799 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640d1e40 con 0x7fe46415cbe0 2023-10-16T15:54:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.652+0000 7fe462ffd640 1 --2- 172.21.15.29:0/1298829799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4640d0b50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009870 tx=0x7fe44c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e1f13485c1049b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.652+0000 7fe461ffb640 1 -- 172.21.15.29:0/1298829799 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c02f830 con 0x7fe46415cbe0 2023-10-16T15:54:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.652+0000 7fe461ffb640 1 -- 172.21.15.29:0/1298829799 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c02f990 con 0x7fe46415cbe0 2023-10-16T15:54:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.652+0000 7fe461ffb640 1 -- 172.21.15.29:0/1298829799 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c02fd20 con 0x7fe46415cbe0 2023-10-16T15:54:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.653+0000 7fe463fff640 1 -- 172.21.15.29:0/1298829799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 msgr2=0x7fe4640d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.653+0000 7fe463fff640 1 --2- 172.21.15.29:0/1298829799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4640d0b50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009870 tx=0x7fe44c02ba10 comp rx=0 tx=0).stop 2023-10-16T15:54:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.653+0000 7fe463fff640 1 -- 172.21.15.29:0/1298829799 shutdown_connections 2023-10-16T15:54:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.653+0000 7fe463fff640 1 --2- 172.21.15.29:0/1298829799 >> v2:172.21.15.170:3300/0 conn(0x7fe4640b4c80 0x7fe4640b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.653+0000 7fe463fff640 1 --2- 172.21.15.29:0/1298829799 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4640d0b50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.653+0000 7fe463fff640 1 -- 172.21.15.29:0/1298829799 >> 172.21.15.29:0/1298829799 conn(0x7fe4640cf570 msgr2=0x7fe4640cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.654+0000 7fe463fff640 1 -- 172.21.15.29:0/1298829799 shutdown_connections 2023-10-16T15:54:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.654+0000 7fe463fff640 1 -- 172.21.15.29:0/1298829799 wait complete. 2023-10-16T15:54:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.656+0000 7fe463fff640 1 Processor -- start 2023-10-16T15:54:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.656+0000 7fe463fff640 1 -- start start 2023-10-16T15:54:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.657+0000 7fe463fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4640b4c80 0x7fe46415fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.657+0000 7fe463fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4641f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.657+0000 7fe463fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 0x7fe46415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.657+0000 7fe463fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641f6890 con 0x7fe46415cfc0 2023-10-16T15:54:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.657+0000 7fe463fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe464154ee0 con 0x7fe4640b4c80 2023-10-16T15:54:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.657+0000 7fe463fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641551a0 con 0x7fe46415cbe0 2023-10-16T15:54:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 0x7fe46415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 0x7fe46415e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36342/0 (socket says 172.21.15.29:36342) 2023-10-16T15:54:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 -- 172.21.15.29:0/2611927051 learned_addr learned my addr 172.21.15.29:0/2611927051 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe462ffd640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4640b4c80 0x7fe46415fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4627fc640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4641f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 -- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 msgr2=0x7fe4641f5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4641f5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 -- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4640b4c80 msgr2=0x7fe46415fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4640b4c80 0x7fe46415fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4637fe640 1 -- 172.21.15.29:0/2611927051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c0094d0 con 0x7fe46415cfc0 2023-10-16T15:54:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.658+0000 7fe4627fc640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4641f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.659+0000 7fe4637fe640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 0x7fe46415e610 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00e7a0 tx=0x7fe45c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.659+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c00c730 con 0x7fe46415cfc0 2023-10-16T15:54:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.659+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c00c890 con 0x7fe46415cfc0 2023-10-16T15:54:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.659+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c00cc40 con 0x7fe46415cfc0 2023-10-16T15:54:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.659+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe464155480 con 0x7fe46415cfc0 2023-10-16T15:54:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.659+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe464155a40 con 0x7fe46415cfc0 2023-10-16T15:54:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.660+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46404e9b0 con 0x7fe46415cfc0 2023-10-16T15:54:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.661+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe45c013470 con 0x7fe46415cfc0 2023-10-16T15:54:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.662+0000 7fe45bfff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe44406b0b0 0x7fe44406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.663+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fe45c08a110 con 0x7fe46415cfc0 2023-10-16T15:54:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.663+0000 7fe462ffd640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe44406b0b0 0x7fe44406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.664+0000 7fe462ffd640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe44406b0b0 0x7fe44406d570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe44c036000 tx=0x7fe44c008030 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.667+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe45c054af0 con 0x7fe46415cfc0 2023-10-16T15:54:57.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.861+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 --> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4640d2120 con 0x7fe44406b0b0 2023-10-16T15:54:57.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.904+0000 7fe45bfff640 1 -- 172.21.15.29:0/2611927051 <== mgr.9022 v2:172.21.15.170:6832/58791103 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526154 (secure 0 0 0) 0x7fe44c02fb00 con 0x7fe44406b0b0 2023-10-16T15:54:57.908 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:54:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe44406b0b0 msgr2=0x7fe44406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe44406b0b0 0x7fe44406d570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe44c036000 tx=0x7fe44c008030 comp rx=0 tx=0).stop 2023-10-16T15:54:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 msgr2=0x7fe46415e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 0x7fe46415e610 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00e7a0 tx=0x7fe45c00ec60 comp rx=0 tx=0).stop 2023-10-16T15:54:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 shutdown_connections 2023-10-16T15:54:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe44406b0b0 0x7fe44406d570 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe46415cfc0 0x7fe46415e610 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe46415cbe0 0x7fe4641f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 --2- 172.21.15.29:0/2611927051 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe4640b4c80 0x7fe46415fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 >> 172.21.15.29:0/2611927051 conn(0x7fe4640cf570 msgr2=0x7fe4640cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 shutdown_connections 2023-10-16T15:54:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:57.916+0000 7fe463fff640 1 -- 172.21.15.29:0/2611927051 wait complete. 2023-10-16T15:54:57.944 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2023-10-16T15:54:57.483647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":714,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":313,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423892,"kb_used_data":25864,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754550828,"statfs":{"total":773094113280,"available":772660047872,"internally_reserved":0,"allocated":26484736,"data_stored":13631746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":407424569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002641"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531731+0000","last_change":"2023-10-16T15:54:53.408335+0000","last_active":"2023-10-16T15:54:54.531731+0000","last_peered":"2023-10-16T15:54:54.531731+0000","last_clean":"2023-10-16T15:54:54.531731+0000","last_became_active":"2023-10-16T15:54:53.407999+0000","last_became_peered":"2023-10-16T15:54:53.407999+0000","last_unstale":"2023-10-16T15:54:54.531731+0000","last_undegraded":"2023-10-16T15:54:54.531731+0000","last_fullsized":"2023-10-16T15:54:54.531731+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:00.091345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530314+0000","last_change":"2023-10-16T15:54:52.399702+0000","last_active":"2023-10-16T15:54:54.530314+0000","last_peered":"2023-10-16T15:54:54.530314+0000","last_clean":"2023-10-16T15:54:54.530314+0000","last_became_active":"2023-10-16T15:54:52.398980+0000","last_became_peered":"2023-10-16T15:54:52.398980+0000","last_unstale":"2023-10-16T15:54:54.530314+0000","last_undegraded":"2023-10-16T15:54:54.530314+0000","last_fullsized":"2023-10-16T15:54:54.530314+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:19:37.756838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.392325+0000","last_change":"2023-10-16T15:54:39.213468+0000","last_active":"2023-10-16T15:54:53.392325+0000","last_peered":"2023-10-16T15:54:53.392325+0000","last_clean":"2023-10-16T15:54:53.392325+0000","last_became_active":"2023-10-16T15:54:39.213201+0000","last_became_peered":"2023-10-16T15:54:39.213201+0000","last_unstale":"2023-10-16T15:54:53.392325+0000","last_undegraded":"2023-10-16T15:54:53.392325+0000","last_fullsized":"2023-10-16T15:54:53.392325+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:54.453092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530716+0000","last_change":"2023-10-16T15:54:53.412071+0000","last_active":"2023-10-16T15:54:54.530716+0000","last_peered":"2023-10-16T15:54:54.530716+0000","last_clean":"2023-10-16T15:54:54.530716+0000","last_became_active":"2023-10-16T15:54:53.411773+0000","last_became_peered":"2023-10-16T15:54:53.411773+0000","last_unstale":"2023-10-16T15:54:54.530716+0000","last_undegraded":"2023-10-16T15:54:54.530716+0000","last_fullsized":"2023-10-16T15:54:54.530716+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:36.633161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531701+0000","last_change":"2023-10-16T15:54:52.392349+0000","last_active":"2023-10-16T15:54:54.531701+0000","last_peered":"2023-10-16T15:54:54.531701+0000","last_clean":"2023-10-16T15:54:54.531701+0000","last_became_active":"2023-10-16T15:54:52.390500+0000","last_became_peered":"2023-10-16T15:54:52.390500+0000","last_unstale":"2023-10-16T15:54:54.531701+0000","last_undegraded":"2023-10-16T15:54:54.531701+0000","last_fullsized":"2023-10-16T15:54:54.531701+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:14.574852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391480+0000","last_change":"2023-10-16T15:54:39.209387+0000","last_active":"2023-10-16T15:54:53.391480+0000","last_peered":"2023-10-16T15:54:53.391480+0000","last_clean":"2023-10-16T15:54:53.391480+0000","last_became_active":"2023-10-16T15:54:39.207929+0000","last_became_peered":"2023-10-16T15:54:39.207929+0000","last_unstale":"2023-10-16T15:54:53.391480+0000","last_undegraded":"2023-10-16T15:54:53.391480+0000","last_fullsized":"2023-10-16T15:54:53.391480+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:21.226148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531576+0000","last_change":"2023-10-16T15:54:53.407832+0000","last_active":"2023-10-16T15:54:54.531576+0000","last_peered":"2023-10-16T15:54:54.531576+0000","last_clean":"2023-10-16T15:54:54.531576+0000","last_became_active":"2023-10-16T15:54:53.407351+0000","last_became_peered":"2023-10-16T15:54:53.407351+0000","last_unstale":"2023-10-16T15:54:54.531576+0000","last_undegraded":"2023-10-16T15:54:54.531576+0000","last_fullsized":"2023-10-16T15:54:54.531576+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:48.011756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405437+0000","last_change":"2023-10-16T15:54:52.400989+0000","last_active":"2023-10-16T15:54:54.405437+0000","last_peered":"2023-10-16T15:54:54.405437+0000","last_clean":"2023-10-16T15:54:54.405437+0000","last_became_active":"2023-10-16T15:54:52.399858+0000","last_became_peered":"2023-10-16T15:54:52.399858+0000","last_unstale":"2023-10-16T15:54:54.405437+0000","last_undegraded":"2023-10-16T15:54:54.405437+0000","last_fullsized":"2023-10-16T15:54:54.405437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:59.813150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391667+0000","last_change":"2023-10-16T15:54:39.205624+0000","last_active":"2023-10-16T15:54:53.391667+0000","last_peered":"2023-10-16T15:54:53.391667+0000","last_clean":"2023-10-16T15:54:53.391667+0000","last_became_active":"2023-10-16T15:54:39.205294+0000","last_became_peered":"2023-10-16T15:54:39.205294+0000","last_unstale":"2023-10-16T15:54:53.391667+0000","last_undegraded":"2023-10-16T15:54:53.391667+0000","last_fullsized":"2023-10-16T15:54:53.391667+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.276795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530730+0000","last_change":"2023-10-16T15:54:53.410389+0000","last_active":"2023-10-16T15:54:54.530730+0000","last_peered":"2023-10-16T15:54:54.530730+0000","last_clean":"2023-10-16T15:54:54.530730+0000","last_became_active":"2023-10-16T15:54:53.410085+0000","last_became_peered":"2023-10-16T15:54:53.410085+0000","last_unstale":"2023-10-16T15:54:54.530730+0000","last_undegraded":"2023-10-16T15:54:54.530730+0000","last_fullsized":"2023-10-16T15:54:54.530730+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:08.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405841+0000","last_change":"2023-10-16T15:54:52.394074+0000","last_active":"2023-10-16T15:54:54.405841+0000","last_peered":"2023-10-16T15:54:54.405841+0000","last_clean":"2023-10-16T15:54:54.405841+0000","last_became_active":"2023-10-16T15:54:52.393784+0000","last_became_peered":"2023-10-16T15:54:52.393784+0000","last_unstale":"2023-10-16T15:54:54.405841+0000","last_undegraded":"2023-10-16T15:54:54.405841+0000","last_fullsized":"2023-10-16T15:54:54.405841+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:40.812320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.401295+0000","last_change":"2023-10-16T15:54:39.203634+0000","last_active":"2023-10-16T15:54:53.401295+0000","last_peered":"2023-10-16T15:54:53.401295+0000","last_clean":"2023-10-16T15:54:53.401295+0000","last_became_active":"2023-10-16T15:54:39.203333+0000","last_became_peered":"2023-10-16T15:54:39.203333+0000","last_unstale":"2023-10-16T15:54:53.401295+0000","last_undegraded":"2023-10-16T15:54:53.401295+0000","last_fullsized":"2023-10-16T15:54:53.401295+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533691+0000","last_change":"2023-10-16T15:54:53.405288+0000","last_active":"2023-10-16T15:54:54.533691+0000","last_peered":"2023-10-16T15:54:54.533691+0000","last_clean":"2023-10-16T15:54:54.533691+0000","last_became_active":"2023-10-16T15:54:53.403686+0000","last_became_peered":"2023-10-16T15:54:53.403686+0000","last_unstale":"2023-10-16T15:54:54.533691+0000","last_undegraded":"2023-10-16T15:54:54.533691+0000","last_fullsized":"2023-10-16T15:54:54.533691+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:54.761128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532016+0000","last_change":"2023-10-16T15:54:52.396757+0000","last_active":"2023-10-16T15:54:54.532016+0000","last_peered":"2023-10-16T15:54:54.532016+0000","last_clean":"2023-10-16T15:54:54.532016+0000","last_became_active":"2023-10-16T15:54:52.396104+0000","last_became_peered":"2023-10-16T15:54:52.396104+0000","last_unstale":"2023-10-16T15:54:54.532016+0000","last_undegraded":"2023-10-16T15:54:54.532016+0000","last_fullsized":"2023-10-16T15:54:54.532016+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:01.735363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387719+0000","last_change":"2023-10-16T15:54:39.205871+0000","last_active":"2023-10-16T15:54:53.387719+0000","last_peered":"2023-10-16T15:54:53.387719+0000","last_clean":"2023-10-16T15:54:53.387719+0000","last_became_active":"2023-10-16T15:54:39.204811+0000","last_became_peered":"2023-10-16T15:54:39.204811+0000","last_unstale":"2023-10-16T15:54:53.387719+0000","last_undegraded":"2023-10-16T15:54:53.387719+0000","last_fullsized":"2023-10-16T15:54:53.387719+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:59.938902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533031+0000","last_change":"2023-10-16T15:54:53.405860+0000","last_active":"2023-10-16T15:54:54.533031+0000","last_peered":"2023-10-16T15:54:54.533031+0000","last_clean":"2023-10-16T15:54:54.533031+0000","last_became_active":"2023-10-16T15:54:53.405398+0000","last_became_peered":"2023-10-16T15:54:53.405398+0000","last_unstale":"2023-10-16T15:54:54.533031+0000","last_undegraded":"2023-10-16T15:54:54.533031+0000","last_fullsized":"2023-10-16T15:54:54.533031+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:00.830634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.534693+0000","last_change":"2023-10-16T15:54:52.396047+0000","last_active":"2023-10-16T15:54:54.534693+0000","last_peered":"2023-10-16T15:54:54.534693+0000","last_clean":"2023-10-16T15:54:54.534693+0000","last_became_active":"2023-10-16T15:54:52.395716+0000","last_became_peered":"2023-10-16T15:54:52.395716+0000","last_unstale":"2023-10-16T15:54:54.534693+0000","last_undegraded":"2023-10-16T15:54:54.534693+0000","last_fullsized":"2023-10-16T15:54:54.534693+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:58.389442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384037+0000","last_change":"2023-10-16T15:54:39.198994+0000","last_active":"2023-10-16T15:54:53.384037+0000","last_peered":"2023-10-16T15:54:53.384037+0000","last_clean":"2023-10-16T15:54:53.384037+0000","last_became_active":"2023-10-16T15:54:39.198650+0000","last_became_peered":"2023-10-16T15:54:39.198650+0000","last_unstale":"2023-10-16T15:54:53.384037+0000","last_undegraded":"2023-10-16T15:54:53.384037+0000","last_fullsized":"2023-10-16T15:54:53.384037+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530813+0000","last_change":"2023-10-16T15:54:53.410527+0000","last_active":"2023-10-16T15:54:54.530813+0000","last_peered":"2023-10-16T15:54:54.530813+0000","last_clean":"2023-10-16T15:54:54.530813+0000","last_became_active":"2023-10-16T15:54:53.410214+0000","last_became_peered":"2023-10-16T15:54:53.410214+0000","last_unstale":"2023-10-16T15:54:54.530813+0000","last_undegraded":"2023-10-16T15:54:54.530813+0000","last_fullsized":"2023-10-16T15:54:54.530813+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:55.476370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533913+0000","last_change":"2023-10-16T15:54:52.395832+0000","last_active":"2023-10-16T15:54:54.533913+0000","last_peered":"2023-10-16T15:54:54.533913+0000","last_clean":"2023-10-16T15:54:54.533913+0000","last_became_active":"2023-10-16T15:54:52.394271+0000","last_became_peered":"2023-10-16T15:54:52.394271+0000","last_unstale":"2023-10-16T15:54:54.533913+0000","last_undegraded":"2023-10-16T15:54:54.533913+0000","last_fullsized":"2023-10-16T15:54:54.533913+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:03:11.728693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381116+0000","last_change":"2023-10-16T15:54:39.190398+0000","last_active":"2023-10-16T15:54:53.381116+0000","last_peered":"2023-10-16T15:54:53.381116+0000","last_clean":"2023-10-16T15:54:53.381116+0000","last_became_active":"2023-10-16T15:54:39.189834+0000","last_became_peered":"2023-10-16T15:54:39.189834+0000","last_unstale":"2023-10-16T15:54:53.381116+0000","last_undegraded":"2023-10-16T15:54:53.381116+0000","last_fullsized":"2023-10-16T15:54:53.381116+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532463+0000","last_change":"2023-10-16T15:54:53.407326+0000","last_active":"2023-10-16T15:54:54.532463+0000","last_peered":"2023-10-16T15:54:54.532463+0000","last_clean":"2023-10-16T15:54:54.532463+0000","last_became_active":"2023-10-16T15:54:53.406995+0000","last_became_peered":"2023-10-16T15:54:53.406995+0000","last_unstale":"2023-10-16T15:54:54.532463+0000","last_undegraded":"2023-10-16T15:54:54.532463+0000","last_fullsized":"2023-10-16T15:54:54.532463+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:06.856360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533071+0000","last_change":"2023-10-16T15:54:52.398585+0000","last_active":"2023-10-16T15:54:54.533071+0000","last_peered":"2023-10-16T15:54:54.533071+0000","last_clean":"2023-10-16T15:54:54.533071+0000","last_became_active":"2023-10-16T15:54:52.397274+0000","last_became_peered":"2023-10-16T15:54:52.397274+0000","last_unstale":"2023-10-16T15:54:54.533071+0000","last_undegraded":"2023-10-16T15:54:54.533071+0000","last_fullsized":"2023-10-16T15:54:54.533071+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:02.747703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383616+0000","last_change":"2023-10-16T15:54:39.203378+0000","last_active":"2023-10-16T15:54:53.383616+0000","last_peered":"2023-10-16T15:54:53.383616+0000","last_clean":"2023-10-16T15:54:53.383616+0000","last_became_active":"2023-10-16T15:54:39.202647+0000","last_became_peered":"2023-10-16T15:54:39.202647+0000","last_unstale":"2023-10-16T15:54:53.383616+0000","last_undegraded":"2023-10-16T15:54:53.383616+0000","last_fullsized":"2023-10-16T15:54:53.383616+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:49:25.249610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405400+0000","last_change":"2023-10-16T15:54:53.407942+0000","last_active":"2023-10-16T15:54:54.405400+0000","last_peered":"2023-10-16T15:54:54.405400+0000","last_clean":"2023-10-16T15:54:54.405400+0000","last_became_active":"2023-10-16T15:54:53.407656+0000","last_became_peered":"2023-10-16T15:54:53.407656+0000","last_unstale":"2023-10-16T15:54:54.405400+0000","last_undegraded":"2023-10-16T15:54:54.405400+0000","last_fullsized":"2023-10-16T15:54:54.405400+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:03.724354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.408329+0000","last_change":"2023-10-16T15:54:52.392722+0000","last_active":"2023-10-16T15:54:54.408329+0000","last_peered":"2023-10-16T15:54:54.408329+0000","last_clean":"2023-10-16T15:54:54.408329+0000","last_became_active":"2023-10-16T15:54:52.389599+0000","last_became_peered":"2023-10-16T15:54:52.389599+0000","last_unstale":"2023-10-16T15:54:54.408329+0000","last_undegraded":"2023-10-16T15:54:54.408329+0000","last_fullsized":"2023-10-16T15:54:54.408329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:57:51.500627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.374482+0000","last_change":"2023-10-16T15:54:39.190692+0000","last_active":"2023-10-16T15:54:53.374482+0000","last_peered":"2023-10-16T15:54:53.374482+0000","last_clean":"2023-10-16T15:54:53.374482+0000","last_became_active":"2023-10-16T15:54:39.189950+0000","last_became_peered":"2023-10-16T15:54:39.189950+0000","last_unstale":"2023-10-16T15:54:53.374482+0000","last_undegraded":"2023-10-16T15:54:53.374482+0000","last_fullsized":"2023-10-16T15:54:53.374482+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:05.334097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530093+0000","last_change":"2023-10-16T15:54:53.394815+0000","last_active":"2023-10-16T15:54:54.530093+0000","last_peered":"2023-10-16T15:54:54.530093+0000","last_clean":"2023-10-16T15:54:54.530093+0000","last_became_active":"2023-10-16T15:54:53.394474+0000","last_became_peered":"2023-10-16T15:54:53.394474+0000","last_unstale":"2023-10-16T15:54:54.530093+0000","last_undegraded":"2023-10-16T15:54:54.530093+0000","last_fullsized":"2023-10-16T15:54:54.530093+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:58.006262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.409356+0000","last_change":"2023-10-16T15:54:52.388176+0000","last_active":"2023-10-16T15:54:54.409356+0000","last_peered":"2023-10-16T15:54:54.409356+0000","last_clean":"2023-10-16T15:54:54.409356+0000","last_became_active":"2023-10-16T15:54:52.387732+0000","last_became_peered":"2023-10-16T15:54:52.387732+0000","last_unstale":"2023-10-16T15:54:54.409356+0000","last_undegraded":"2023-10-16T15:54:54.409356+0000","last_fullsized":"2023-10-16T15:54:54.409356+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:34.734679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390036+0000","last_change":"2023-10-16T15:54:39.203147+0000","last_active":"2023-10-16T15:54:53.390036+0000","last_peered":"2023-10-16T15:54:53.390036+0000","last_clean":"2023-10-16T15:54:53.390036+0000","last_became_active":"2023-10-16T15:54:39.202529+0000","last_became_peered":"2023-10-16T15:54:39.202529+0000","last_unstale":"2023-10-16T15:54:53.390036+0000","last_undegraded":"2023-10-16T15:54:53.390036+0000","last_fullsized":"2023-10-16T15:54:53.390036+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529812+0000","last_change":"2023-10-16T15:54:53.413494+0000","last_active":"2023-10-16T15:54:54.529812+0000","last_peered":"2023-10-16T15:54:54.529812+0000","last_clean":"2023-10-16T15:54:54.529812+0000","last_became_active":"2023-10-16T15:54:53.413037+0000","last_became_peered":"2023-10-16T15:54:53.413037+0000","last_unstale":"2023-10-16T15:54:54.529812+0000","last_undegraded":"2023-10-16T15:54:54.529812+0000","last_fullsized":"2023-10-16T15:54:54.529812+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:06.628301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410566+0000","last_change":"2023-10-16T15:54:52.378294+0000","last_active":"2023-10-16T15:54:54.410566+0000","last_peered":"2023-10-16T15:54:54.410566+0000","last_clean":"2023-10-16T15:54:54.410566+0000","last_became_active":"2023-10-16T15:54:52.377745+0000","last_became_peered":"2023-10-16T15:54:52.377745+0000","last_unstale":"2023-10-16T15:54:54.410566+0000","last_undegraded":"2023-10-16T15:54:54.410566+0000","last_fullsized":"2023-10-16T15:54:54.410566+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:54:46.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370636+0000","last_change":"2023-10-16T15:54:39.200394+0000","last_active":"2023-10-16T15:54:53.370636+0000","last_peered":"2023-10-16T15:54:53.370636+0000","last_clean":"2023-10-16T15:54:53.370636+0000","last_became_active":"2023-10-16T15:54:39.199822+0000","last_became_peered":"2023-10-16T15:54:39.199822+0000","last_unstale":"2023-10-16T15:54:53.370636+0000","last_undegraded":"2023-10-16T15:54:53.370636+0000","last_fullsized":"2023-10-16T15:54:53.370636+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:38.462835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529626+0000","last_change":"2023-10-16T15:54:53.414402+0000","last_active":"2023-10-16T15:54:54.529626+0000","last_peered":"2023-10-16T15:54:54.529626+0000","last_clean":"2023-10-16T15:54:54.529626+0000","last_became_active":"2023-10-16T15:54:53.412369+0000","last_became_peered":"2023-10-16T15:54:53.412369+0000","last_unstale":"2023-10-16T15:54:54.529626+0000","last_undegraded":"2023-10-16T15:54:54.529626+0000","last_fullsized":"2023-10-16T15:54:54.529626+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:56.945120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532557+0000","last_change":"2023-10-16T15:54:52.390908+0000","last_active":"2023-10-16T15:54:54.532557+0000","last_peered":"2023-10-16T15:54:54.532557+0000","last_clean":"2023-10-16T15:54:54.532557+0000","last_became_active":"2023-10-16T15:54:52.390553+0000","last_became_peered":"2023-10-16T15:54:52.390553+0000","last_unstale":"2023-10-16T15:54:54.532557+0000","last_undegraded":"2023-10-16T15:54:54.532557+0000","last_fullsized":"2023-10-16T15:54:54.532557+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:24.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390136+0000","last_change":"2023-10-16T15:54:39.210009+0000","last_active":"2023-10-16T15:54:53.390136+0000","last_peered":"2023-10-16T15:54:53.390136+0000","last_clean":"2023-10-16T15:54:53.390136+0000","last_became_active":"2023-10-16T15:54:39.209277+0000","last_became_peered":"2023-10-16T15:54:39.209277+0000","last_unstale":"2023-10-16T15:54:53.390136+0000","last_undegraded":"2023-10-16T15:54:53.390136+0000","last_fullsized":"2023-10-16T15:54:53.390136+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533298+0000","last_change":"2023-10-16T15:54:53.403243+0000","last_active":"2023-10-16T15:54:54.533298+0000","last_peered":"2023-10-16T15:54:54.533298+0000","last_clean":"2023-10-16T15:54:54.533298+0000","last_became_active":"2023-10-16T15:54:53.402945+0000","last_became_peered":"2023-10-16T15:54:53.402945+0000","last_unstale":"2023-10-16T15:54:54.533298+0000","last_undegraded":"2023-10-16T15:54:54.533298+0000","last_fullsized":"2023-10-16T15:54:54.533298+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:40.579694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532139+0000","last_change":"2023-10-16T15:54:52.398247+0000","last_active":"2023-10-16T15:54:54.532139+0000","last_peered":"2023-10-16T15:54:54.532139+0000","last_clean":"2023-10-16T15:54:54.532139+0000","last_became_active":"2023-10-16T15:54:52.397968+0000","last_became_peered":"2023-10-16T15:54:52.397968+0000","last_unstale":"2023-10-16T15:54:54.532139+0000","last_undegraded":"2023-10-16T15:54:54.532139+0000","last_fullsized":"2023-10-16T15:54:54.532139+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:46.112000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386274+0000","last_change":"2023-10-16T15:54:39.194527+0000","last_active":"2023-10-16T15:54:53.386274+0000","last_peered":"2023-10-16T15:54:53.386274+0000","last_clean":"2023-10-16T15:54:53.386274+0000","last_became_active":"2023-10-16T15:54:39.194123+0000","last_became_peered":"2023-10-16T15:54:39.194123+0000","last_unstale":"2023-10-16T15:54:53.386274+0000","last_undegraded":"2023-10-16T15:54:53.386274+0000","last_fullsized":"2023-10-16T15:54:53.386274+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532030+0000","last_change":"2023-10-16T15:54:53.409519+0000","last_active":"2023-10-16T15:54:54.532030+0000","last_peered":"2023-10-16T15:54:54.532030+0000","last_clean":"2023-10-16T15:54:54.532030+0000","last_became_active":"2023-10-16T15:54:53.408181+0000","last_became_peered":"2023-10-16T15:54:53.408181+0000","last_unstale":"2023-10-16T15:54:54.532030+0000","last_undegraded":"2023-10-16T15:54:54.532030+0000","last_fullsized":"2023-10-16T15:54:54.532030+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:11.802127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410541+0000","last_change":"2023-10-16T15:54:52.387751+0000","last_active":"2023-10-16T15:54:54.410541+0000","last_peered":"2023-10-16T15:54:54.410541+0000","last_clean":"2023-10-16T15:54:54.410541+0000","last_became_active":"2023-10-16T15:54:52.387202+0000","last_became_peered":"2023-10-16T15:54:52.387202+0000","last_unstale":"2023-10-16T15:54:54.410541+0000","last_undegraded":"2023-10-16T15:54:54.410541+0000","last_fullsized":"2023-10-16T15:54:54.410541+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:03.499950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385166+0000","last_change":"2023-10-16T15:54:39.208938+0000","last_active":"2023-10-16T15:54:53.385166+0000","last_peered":"2023-10-16T15:54:53.385166+0000","last_clean":"2023-10-16T15:54:53.385166+0000","last_became_active":"2023-10-16T15:54:39.208579+0000","last_became_peered":"2023-10-16T15:54:39.208579+0000","last_unstale":"2023-10-16T15:54:53.385166+0000","last_undegraded":"2023-10-16T15:54:53.385166+0000","last_fullsized":"2023-10-16T15:54:53.385166+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:32.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.411597+0000","last_change":"2023-10-16T15:54:53.412100+0000","last_active":"2023-10-16T15:54:54.411597+0000","last_peered":"2023-10-16T15:54:54.411597+0000","last_clean":"2023-10-16T15:54:54.411597+0000","last_became_active":"2023-10-16T15:54:53.411747+0000","last_became_peered":"2023-10-16T15:54:53.411747+0000","last_unstale":"2023-10-16T15:54:54.411597+0000","last_undegraded":"2023-10-16T15:54:54.411597+0000","last_fullsized":"2023-10-16T15:54:54.411597+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:37.450215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533197+0000","last_change":"2023-10-16T15:54:52.395278+0000","last_active":"2023-10-16T15:54:54.533197+0000","last_peered":"2023-10-16T15:54:54.533197+0000","last_clean":"2023-10-16T15:54:54.533197+0000","last_became_active":"2023-10-16T15:54:52.394638+0000","last_became_peered":"2023-10-16T15:54:52.394638+0000","last_unstale":"2023-10-16T15:54:54.533197+0000","last_undegraded":"2023-10-16T15:54:54.533197+0000","last_fullsized":"2023-10-16T15:54:54.533197+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:44.272665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375358+0000","last_change":"2023-10-16T15:54:39.187772+0000","last_active":"2023-10-16T15:54:53.375358+0000","last_peered":"2023-10-16T15:54:53.375358+0000","last_clean":"2023-10-16T15:54:53.375358+0000","last_became_active":"2023-10-16T15:54:39.187210+0000","last_became_peered":"2023-10-16T15:54:39.187210+0000","last_unstale":"2023-10-16T15:54:53.375358+0000","last_undegraded":"2023-10-16T15:54:53.375358+0000","last_fullsized":"2023-10-16T15:54:53.375358+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:31.089495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532688+0000","last_change":"2023-10-16T15:54:53.408805+0000","last_active":"2023-10-16T15:54:54.532688+0000","last_peered":"2023-10-16T15:54:54.532688+0000","last_clean":"2023-10-16T15:54:54.532688+0000","last_became_active":"2023-10-16T15:54:53.406250+0000","last_became_peered":"2023-10-16T15:54:53.406250+0000","last_unstale":"2023-10-16T15:54:54.532688+0000","last_undegraded":"2023-10-16T15:54:54.532688+0000","last_fullsized":"2023-10-16T15:54:54.532688+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:26.499127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412445+0000","last_change":"2023-10-16T15:54:52.399176+0000","last_active":"2023-10-16T15:54:54.412445+0000","last_peered":"2023-10-16T15:54:54.412445+0000","last_clean":"2023-10-16T15:54:54.412445+0000","last_became_active":"2023-10-16T15:54:52.398362+0000","last_became_peered":"2023-10-16T15:54:52.398362+0000","last_unstale":"2023-10-16T15:54:54.412445+0000","last_undegraded":"2023-10-16T15:54:54.412445+0000","last_fullsized":"2023-10-16T15:54:54.412445+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:44.118800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385912+0000","last_change":"2023-10-16T15:54:39.199187+0000","last_active":"2023-10-16T15:54:53.385912+0000","last_peered":"2023-10-16T15:54:53.385912+0000","last_clean":"2023-10-16T15:54:53.385912+0000","last_became_active":"2023-10-16T15:54:39.195076+0000","last_became_peered":"2023-10-16T15:54:39.195076+0000","last_unstale":"2023-10-16T15:54:53.385912+0000","last_undegraded":"2023-10-16T15:54:53.385912+0000","last_fullsized":"2023-10-16T15:54:53.385912+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:01.081391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530386+0000","last_change":"2023-10-16T15:54:53.401211+0000","last_active":"2023-10-16T15:54:54.530386+0000","last_peered":"2023-10-16T15:54:54.530386+0000","last_clean":"2023-10-16T15:54:54.530386+0000","last_became_active":"2023-10-16T15:54:53.399693+0000","last_became_peered":"2023-10-16T15:54:53.399693+0000","last_unstale":"2023-10-16T15:54:54.530386+0000","last_undegraded":"2023-10-16T15:54:54.530386+0000","last_fullsized":"2023-10-16T15:54:54.530386+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:36.775050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"249'8","reported_seq":20,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533845+0000","last_change":"2023-10-16T15:54:52.390353+0000","last_active":"2023-10-16T15:54:54.533845+0000","last_peered":"2023-10-16T15:54:54.533845+0000","last_clean":"2023-10-16T15:54:54.533845+0000","last_became_active":"2023-10-16T15:54:52.388675+0000","last_became_peered":"2023-10-16T15:54:52.388675+0000","last_unstale":"2023-10-16T15:54:54.533845+0000","last_undegraded":"2023-10-16T15:54:54.533845+0000","last_fullsized":"2023-10-16T15:54:54.533845+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:49.390733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.402409+0000","last_change":"2023-10-16T15:54:39.212389+0000","last_active":"2023-10-16T15:54:53.402409+0000","last_peered":"2023-10-16T15:54:53.402409+0000","last_clean":"2023-10-16T15:54:53.402409+0000","last_became_active":"2023-10-16T15:54:39.212017+0000","last_became_peered":"2023-10-16T15:54:39.212017+0000","last_unstale":"2023-10-16T15:54:53.402409+0000","last_undegraded":"2023-10-16T15:54:53.402409+0000","last_fullsized":"2023-10-16T15:54:53.402409+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529935+0000","last_change":"2023-10-16T15:54:53.412238+0000","last_active":"2023-10-16T15:54:54.529935+0000","last_peered":"2023-10-16T15:54:54.529935+0000","last_clean":"2023-10-16T15:54:54.529935+0000","last_became_active":"2023-10-16T15:54:53.411974+0000","last_became_peered":"2023-10-16T15:54:53.411974+0000","last_unstale":"2023-10-16T15:54:54.529935+0000","last_undegraded":"2023-10-16T15:54:54.529935+0000","last_fullsized":"2023-10-16T15:54:54.529935+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:53.511292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410103+0000","last_change":"2023-10-16T15:54:52.394346+0000","last_active":"2023-10-16T15:54:54.410103+0000","last_peered":"2023-10-16T15:54:54.410103+0000","last_clean":"2023-10-16T15:54:54.410103+0000","last_became_active":"2023-10-16T15:54:52.393847+0000","last_became_peered":"2023-10-16T15:54:52.393847+0000","last_unstale":"2023-10-16T15:54:54.410103+0000","last_undegraded":"2023-10-16T15:54:54.410103+0000","last_fullsized":"2023-10-16T15:54:54.410103+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:51.228820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391631+0000","last_change":"2023-10-16T15:54:39.214749+0000","last_active":"2023-10-16T15:54:53.391631+0000","last_peered":"2023-10-16T15:54:53.391631+0000","last_clean":"2023-10-16T15:54:53.391631+0000","last_became_active":"2023-10-16T15:54:39.214456+0000","last_became_peered":"2023-10-16T15:54:39.214456+0000","last_unstale":"2023-10-16T15:54:53.391631+0000","last_undegraded":"2023-10-16T15:54:53.391631+0000","last_fullsized":"2023-10-16T15:54:53.391631+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:36.851092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529685+0000","last_change":"2023-10-16T15:54:53.395570+0000","last_active":"2023-10-16T15:54:54.529685+0000","last_peered":"2023-10-16T15:54:54.529685+0000","last_clean":"2023-10-16T15:54:54.529685+0000","last_became_active":"2023-10-16T15:54:53.395267+0000","last_became_peered":"2023-10-16T15:54:53.395267+0000","last_unstale":"2023-10-16T15:54:54.529685+0000","last_undegraded":"2023-10-16T15:54:54.529685+0000","last_fullsized":"2023-10-16T15:54:54.529685+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:15.241619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405678+0000","last_change":"2023-10-16T15:54:52.395718+0000","last_active":"2023-10-16T15:54:54.405678+0000","last_peered":"2023-10-16T15:54:54.405678+0000","last_clean":"2023-10-16T15:54:54.405678+0000","last_became_active":"2023-10-16T15:54:52.395435+0000","last_became_peered":"2023-10-16T15:54:52.395435+0000","last_unstale":"2023-10-16T15:54:54.405678+0000","last_undegraded":"2023-10-16T15:54:54.405678+0000","last_fullsized":"2023-10-16T15:54:54.405678+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:39.562211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396099+0000","last_change":"2023-10-16T15:54:39.202583+0000","last_active":"2023-10-16T15:54:53.396099+0000","last_peered":"2023-10-16T15:54:53.396099+0000","last_clean":"2023-10-16T15:54:53.396099+0000","last_became_active":"2023-10-16T15:54:39.202278+0000","last_became_peered":"2023-10-16T15:54:39.202278+0000","last_unstale":"2023-10-16T15:54:53.396099+0000","last_undegraded":"2023-10-16T15:54:53.396099+0000","last_fullsized":"2023-10-16T15:54:53.396099+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531055+0000","last_change":"2023-10-16T15:54:53.399831+0000","last_active":"2023-10-16T15:54:54.531055+0000","last_peered":"2023-10-16T15:54:54.531055+0000","last_clean":"2023-10-16T15:54:54.531055+0000","last_became_active":"2023-10-16T15:54:53.399486+0000","last_became_peered":"2023-10-16T15:54:53.399486+0000","last_unstale":"2023-10-16T15:54:54.531055+0000","last_undegraded":"2023-10-16T15:54:54.531055+0000","last_fullsized":"2023-10-16T15:54:54.531055+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:56.150842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.411593+0000","last_change":"2023-10-16T15:54:52.389878+0000","last_active":"2023-10-16T15:54:54.411593+0000","last_peered":"2023-10-16T15:54:54.411593+0000","last_clean":"2023-10-16T15:54:54.411593+0000","last_became_active":"2023-10-16T15:54:52.389586+0000","last_became_peered":"2023-10-16T15:54:52.389586+0000","last_unstale":"2023-10-16T15:54:54.411593+0000","last_undegraded":"2023-10-16T15:54:54.411593+0000","last_fullsized":"2023-10-16T15:54:54.411593+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:51.818377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385313+0000","last_change":"2023-10-16T15:54:39.205053+0000","last_active":"2023-10-16T15:54:53.385313+0000","last_peered":"2023-10-16T15:54:53.385313+0000","last_clean":"2023-10-16T15:54:53.385313+0000","last_became_active":"2023-10-16T15:54:39.202502+0000","last_became_peered":"2023-10-16T15:54:39.202502+0000","last_unstale":"2023-10-16T15:54:53.385313+0000","last_undegraded":"2023-10-16T15:54:53.385313+0000","last_fullsized":"2023-10-16T15:54:53.385313+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:38.045649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532817+0000","last_change":"2023-10-16T15:54:53.406861+0000","last_active":"2023-10-16T15:54:54.532817+0000","last_peered":"2023-10-16T15:54:54.532817+0000","last_clean":"2023-10-16T15:54:54.532817+0000","last_became_active":"2023-10-16T15:54:53.406612+0000","last_became_peered":"2023-10-16T15:54:53.406612+0000","last_unstale":"2023-10-16T15:54:54.532817+0000","last_undegraded":"2023-10-16T15:54:54.532817+0000","last_fullsized":"2023-10-16T15:54:54.532817+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:33:35.711439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"249'6","reported_seq":18,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531935+0000","last_change":"2023-10-16T15:54:52.384065+0000","last_active":"2023-10-16T15:54:54.531935+0000","last_peered":"2023-10-16T15:54:54.531935+0000","last_clean":"2023-10-16T15:54:54.531935+0000","last_became_active":"2023-10-16T15:54:52.383745+0000","last_became_peered":"2023-10-16T15:54:52.383745+0000","last_unstale":"2023-10-16T15:54:54.531935+0000","last_undegraded":"2023-10-16T15:54:54.531935+0000","last_fullsized":"2023-10-16T15:54:54.531935+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:23.741186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387849+0000","last_change":"2023-10-16T15:54:39.202366+0000","last_active":"2023-10-16T15:54:53.387849+0000","last_peered":"2023-10-16T15:54:53.387849+0000","last_clean":"2023-10-16T15:54:53.387849+0000","last_became_active":"2023-10-16T15:54:39.201981+0000","last_became_peered":"2023-10-16T15:54:39.201981+0000","last_unstale":"2023-10-16T15:54:53.387849+0000","last_undegraded":"2023-10-16T15:54:53.387849+0000","last_fullsized":"2023-10-16T15:54:53.387849+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:00.755938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531929+0000","last_change":"2023-10-16T15:54:53.409568+0000","last_active":"2023-10-16T15:54:54.531929+0000","last_peered":"2023-10-16T15:54:54.531929+0000","last_clean":"2023-10-16T15:54:54.531929+0000","last_became_active":"2023-10-16T15:54:53.407950+0000","last_became_peered":"2023-10-16T15:54:53.407950+0000","last_unstale":"2023-10-16T15:54:54.531929+0000","last_undegraded":"2023-10-16T15:54:54.531929+0000","last_fullsized":"2023-10-16T15:54:54.531929+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:57.466035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"249'3","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530078+0000","last_change":"2023-10-16T15:54:52.398323+0000","last_active":"2023-10-16T15:54:54.530078+0000","last_peered":"2023-10-16T15:54:54.530078+0000","last_clean":"2023-10-16T15:54:54.530078+0000","last_became_active":"2023-10-16T15:54:52.398021+0000","last_became_peered":"2023-10-16T15:54:52.398021+0000","last_unstale":"2023-10-16T15:54:54.530078+0000","last_undegraded":"2023-10-16T15:54:54.530078+0000","last_fullsized":"2023-10-16T15:54:54.530078+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:18.517267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381338+0000","last_change":"2023-10-16T15:54:39.209910+0000","last_active":"2023-10-16T15:54:53.381338+0000","last_peered":"2023-10-16T15:54:53.381338+0000","last_clean":"2023-10-16T15:54:53.381338+0000","last_became_active":"2023-10-16T15:54:39.209504+0000","last_became_peered":"2023-10-16T15:54:39.209504+0000","last_unstale":"2023-10-16T15:54:53.381338+0000","last_undegraded":"2023-10-16T15:54:53.381338+0000","last_fullsized":"2023-10-16T15:54:53.381338+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530629+0000","last_change":"2023-10-16T15:54:53.404840+0000","last_active":"2023-10-16T15:54:54.530629+0000","last_peered":"2023-10-16T15:54:54.530629+0000","last_clean":"2023-10-16T15:54:54.530629+0000","last_became_active":"2023-10-16T15:54:53.404549+0000","last_became_peered":"2023-10-16T15:54:53.404549+0000","last_unstale":"2023-10-16T15:54:54.530629+0000","last_undegraded":"2023-10-16T15:54:54.530629+0000","last_fullsized":"2023-10-16T15:54:54.530629+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:16.610186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532770+0000","last_change":"2023-10-16T15:54:52.396630+0000","last_active":"2023-10-16T15:54:54.532770+0000","last_peered":"2023-10-16T15:54:54.532770+0000","last_clean":"2023-10-16T15:54:54.532770+0000","last_became_active":"2023-10-16T15:54:52.395733+0000","last_became_peered":"2023-10-16T15:54:52.395733+0000","last_unstale":"2023-10-16T15:54:54.532770+0000","last_undegraded":"2023-10-16T15:54:54.532770+0000","last_fullsized":"2023-10-16T15:54:54.532770+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:18.062870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384380+0000","last_change":"2023-10-16T15:54:39.207379+0000","last_active":"2023-10-16T15:54:53.384380+0000","last_peered":"2023-10-16T15:54:53.384380+0000","last_clean":"2023-10-16T15:54:53.384380+0000","last_became_active":"2023-10-16T15:54:39.207010+0000","last_became_peered":"2023-10-16T15:54:39.207010+0000","last_unstale":"2023-10-16T15:54:53.384380+0000","last_undegraded":"2023-10-16T15:54:53.384380+0000","last_fullsized":"2023-10-16T15:54:53.384380+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:45.890691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533869+0000","last_change":"2023-10-16T15:54:53.395468+0000","last_active":"2023-10-16T15:54:54.533869+0000","last_peered":"2023-10-16T15:54:54.533869+0000","last_clean":"2023-10-16T15:54:54.533869+0000","last_became_active":"2023-10-16T15:54:53.395099+0000","last_became_peered":"2023-10-16T15:54:53.395099+0000","last_unstale":"2023-10-16T15:54:54.533869+0000","last_undegraded":"2023-10-16T15:54:54.533869+0000","last_fullsized":"2023-10-16T15:54:54.533869+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:51.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531178+0000","last_change":"2023-10-16T15:54:52.400699+0000","last_active":"2023-10-16T15:54:54.531178+0000","last_peered":"2023-10-16T15:54:54.531178+0000","last_clean":"2023-10-16T15:54:54.531178+0000","last_became_active":"2023-10-16T15:54:52.399180+0000","last_became_peered":"2023-10-16T15:54:52.399180+0000","last_unstale":"2023-10-16T15:54:54.531178+0000","last_undegraded":"2023-10-16T15:54:54.531178+0000","last_fullsized":"2023-10-16T15:54:54.531178+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:27.946798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383190+0000","last_change":"2023-10-16T15:54:39.199635+0000","last_active":"2023-10-16T15:54:53.383190+0000","last_peered":"2023-10-16T15:54:53.383190+0000","last_clean":"2023-10-16T15:54:53.383190+0000","last_became_active":"2023-10-16T15:54:39.199152+0000","last_became_peered":"2023-10-16T15:54:39.199152+0000","last_unstale":"2023-10-16T15:54:53.383190+0000","last_undegraded":"2023-10-16T15:54:53.383190+0000","last_fullsized":"2023-10-16T15:54:53.383190+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:45.206271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531419+0000","last_change":"2023-10-16T15:54:53.391417+0000","last_active":"2023-10-16T15:54:54.531419+0000","last_peered":"2023-10-16T15:54:54.531419+0000","last_clean":"2023-10-16T15:54:54.531419+0000","last_became_active":"2023-10-16T15:54:53.390886+0000","last_became_peered":"2023-10-16T15:54:53.390886+0000","last_unstale":"2023-10-16T15:54:54.531419+0000","last_undegraded":"2023-10-16T15:54:54.531419+0000","last_fullsized":"2023-10-16T15:54:54.531419+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:03.765186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531543+0000","last_change":"2023-10-16T15:54:52.396387+0000","last_active":"2023-10-16T15:54:54.531543+0000","last_peered":"2023-10-16T15:54:54.531543+0000","last_clean":"2023-10-16T15:54:54.531543+0000","last_became_active":"2023-10-16T15:54:52.396048+0000","last_became_peered":"2023-10-16T15:54:52.396048+0000","last_unstale":"2023-10-16T15:54:54.531543+0000","last_undegraded":"2023-10-16T15:54:54.531543+0000","last_fullsized":"2023-10-16T15:54:54.531543+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:31.223305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.403931+0000","last_change":"2023-10-16T15:54:39.205509+0000","last_active":"2023-10-16T15:54:53.403931+0000","last_peered":"2023-10-16T15:54:53.403931+0000","last_clean":"2023-10-16T15:54:53.403931+0000","last_became_active":"2023-10-16T15:54:39.204206+0000","last_became_peered":"2023-10-16T15:54:39.204206+0000","last_unstale":"2023-10-16T15:54:53.403931+0000","last_undegraded":"2023-10-16T15:54:53.403931+0000","last_fullsized":"2023-10-16T15:54:53.403931+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529484+0000","last_change":"2023-10-16T15:54:53.399798+0000","last_active":"2023-10-16T15:54:54.529484+0000","last_peered":"2023-10-16T15:54:54.529484+0000","last_clean":"2023-10-16T15:54:54.529484+0000","last_became_active":"2023-10-16T15:54:53.399501+0000","last_became_peered":"2023-10-16T15:54:53.399501+0000","last_unstale":"2023-10-16T15:54:54.529484+0000","last_undegraded":"2023-10-16T15:54:54.529484+0000","last_fullsized":"2023-10-16T15:54:54.529484+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:30.954197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531959+0000","last_change":"2023-10-16T15:54:52.395316+0000","last_active":"2023-10-16T15:54:54.531959+0000","last_peered":"2023-10-16T15:54:54.531959+0000","last_clean":"2023-10-16T15:54:54.531959+0000","last_became_active":"2023-10-16T15:54:52.395019+0000","last_became_peered":"2023-10-16T15:54:52.395019+0000","last_unstale":"2023-10-16T15:54:54.531959+0000","last_undegraded":"2023-10-16T15:54:54.531959+0000","last_fullsized":"2023-10-16T15:54:54.531959+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:51.646005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390171+0000","last_change":"2023-10-16T15:54:39.194931+0000","last_active":"2023-10-16T15:54:53.390171+0000","last_peered":"2023-10-16T15:54:53.390171+0000","last_clean":"2023-10-16T15:54:53.390171+0000","last_became_active":"2023-10-16T15:54:39.194605+0000","last_became_peered":"2023-10-16T15:54:39.194605+0000","last_unstale":"2023-10-16T15:54:53.390171+0000","last_undegraded":"2023-10-16T15:54:53.390171+0000","last_fullsized":"2023-10-16T15:54:53.390171+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529559+0000","last_change":"2023-10-16T15:54:53.394365+0000","last_active":"2023-10-16T15:54:54.529559+0000","last_peered":"2023-10-16T15:54:54.529559+0000","last_clean":"2023-10-16T15:54:54.529559+0000","last_became_active":"2023-10-16T15:54:53.394016+0000","last_became_peered":"2023-10-16T15:54:53.394016+0000","last_unstale":"2023-10-16T15:54:54.529559+0000","last_undegraded":"2023-10-16T15:54:54.529559+0000","last_fullsized":"2023-10-16T15:54:54.529559+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:50.320294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530731+0000","last_change":"2023-10-16T15:54:52.380695+0000","last_active":"2023-10-16T15:54:54.530731+0000","last_peered":"2023-10-16T15:54:54.530731+0000","last_clean":"2023-10-16T15:54:54.530731+0000","last_became_active":"2023-10-16T15:54:52.379787+0000","last_became_peered":"2023-10-16T15:54:52.379787+0000","last_unstale":"2023-10-16T15:54:54.530731+0000","last_undegraded":"2023-10-16T15:54:54.530731+0000","last_fullsized":"2023-10-16T15:54:54.530731+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:19.194940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370882+0000","last_change":"2023-10-16T15:54:39.204169+0000","last_active":"2023-10-16T15:54:53.370882+0000","last_peered":"2023-10-16T15:54:53.370882+0000","last_clean":"2023-10-16T15:54:53.370882+0000","last_became_active":"2023-10-16T15:54:39.200108+0000","last_became_peered":"2023-10-16T15:54:39.200108+0000","last_unstale":"2023-10-16T15:54:53.370882+0000","last_undegraded":"2023-10-16T15:54:53.370882+0000","last_fullsized":"2023-10-16T15:54:53.370882+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:23.620457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405952+0000","last_change":"2023-10-16T15:54:53.398381+0000","last_active":"2023-10-16T15:54:54.405952+0000","last_peered":"2023-10-16T15:54:54.405952+0000","last_clean":"2023-10-16T15:54:54.405952+0000","last_became_active":"2023-10-16T15:54:53.394042+0000","last_became_peered":"2023-10-16T15:54:53.394042+0000","last_unstale":"2023-10-16T15:54:54.405952+0000","last_undegraded":"2023-10-16T15:54:54.405952+0000","last_fullsized":"2023-10-16T15:54:54.405952+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:16.136129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532866+0000","last_change":"2023-10-16T15:54:52.396735+0000","last_active":"2023-10-16T15:54:54.532866+0000","last_peered":"2023-10-16T15:54:54.532866+0000","last_clean":"2023-10-16T15:54:54.532866+0000","last_became_active":"2023-10-16T15:54:52.396343+0000","last_became_peered":"2023-10-16T15:54:52.396343+0000","last_unstale":"2023-10-16T15:54:54.532866+0000","last_undegraded":"2023-10-16T15:54:54.532866+0000","last_fullsized":"2023-10-16T15:54:54.532866+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:51.623743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.389265+0000","last_change":"2023-10-16T15:54:39.203124+0000","last_active":"2023-10-16T15:54:53.389265+0000","last_peered":"2023-10-16T15:54:53.389265+0000","last_clean":"2023-10-16T15:54:53.389265+0000","last_became_active":"2023-10-16T15:54:39.200245+0000","last_became_peered":"2023-10-16T15:54:39.200245+0000","last_unstale":"2023-10-16T15:54:53.389265+0000","last_undegraded":"2023-10-16T15:54:53.389265+0000","last_fullsized":"2023-10-16T15:54:53.389265+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:43.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412113+0000","last_change":"2023-10-16T15:54:53.399367+0000","last_active":"2023-10-16T15:54:54.412113+0000","last_peered":"2023-10-16T15:54:54.412113+0000","last_clean":"2023-10-16T15:54:54.412113+0000","last_became_active":"2023-10-16T15:54:53.399017+0000","last_became_peered":"2023-10-16T15:54:53.399017+0000","last_unstale":"2023-10-16T15:54:54.412113+0000","last_undegraded":"2023-10-16T15:54:54.412113+0000","last_fullsized":"2023-10-16T15:54:54.412113+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:32.240788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530754+0000","last_change":"2023-10-16T15:54:52.396851+0000","last_active":"2023-10-16T15:54:54.530754+0000","last_peered":"2023-10-16T15:54:54.530754+0000","last_clean":"2023-10-16T15:54:54.530754+0000","last_became_active":"2023-10-16T15:54:52.396542+0000","last_became_peered":"2023-10-16T15:54:52.396542+0000","last_unstale":"2023-10-16T15:54:54.530754+0000","last_undegraded":"2023-10-16T15:54:54.530754+0000","last_fullsized":"2023-10-16T15:54:54.530754+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:09.513209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370742+0000","last_change":"2023-10-16T15:54:39.203684+0000","last_active":"2023-10-16T15:54:53.370742+0000","last_peered":"2023-10-16T15:54:53.370742+0000","last_clean":"2023-10-16T15:54:53.370742+0000","last_became_active":"2023-10-16T15:54:39.203327+0000","last_became_peered":"2023-10-16T15:54:39.203327+0000","last_unstale":"2023-10-16T15:54:53.370742+0000","last_undegraded":"2023-10-16T15:54:53.370742+0000","last_fullsized":"2023-10-16T15:54:53.370742+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:29.051715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532144+0000","last_change":"2023-10-16T15:54:53.395970+0000","last_active":"2023-10-16T15:54:54.532144+0000","last_peered":"2023-10-16T15:54:54.532144+0000","last_clean":"2023-10-16T15:54:54.532144+0000","last_became_active":"2023-10-16T15:54:53.395642+0000","last_became_peered":"2023-10-16T15:54:53.395642+0000","last_unstale":"2023-10-16T15:54:54.532144+0000","last_undegraded":"2023-10-16T15:54:54.532144+0000","last_fullsized":"2023-10-16T15:54:54.532144+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:57.375885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530070+0000","last_change":"2023-10-16T15:54:52.389455+0000","last_active":"2023-10-16T15:54:54.530070+0000","last_peered":"2023-10-16T15:54:54.530070+0000","last_clean":"2023-10-16T15:54:54.530070+0000","last_became_active":"2023-10-16T15:54:52.389180+0000","last_became_peered":"2023-10-16T15:54:52.389180+0000","last_unstale":"2023-10-16T15:54:54.530070+0000","last_undegraded":"2023-10-16T15:54:54.530070+0000","last_fullsized":"2023-10-16T15:54:54.530070+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:28.726584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396883+0000","last_change":"2023-10-16T15:54:39.203576+0000","last_active":"2023-10-16T15:54:53.396883+0000","last_peered":"2023-10-16T15:54:53.396883+0000","last_clean":"2023-10-16T15:54:53.396883+0000","last_became_active":"2023-10-16T15:54:39.203072+0000","last_became_peered":"2023-10-16T15:54:39.203072+0000","last_unstale":"2023-10-16T15:54:53.396883+0000","last_undegraded":"2023-10-16T15:54:53.396883+0000","last_fullsized":"2023-10-16T15:54:53.396883+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:05.472712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532864+0000","last_change":"2023-10-16T15:54:53.413070+0000","last_active":"2023-10-16T15:54:54.532864+0000","last_peered":"2023-10-16T15:54:54.532864+0000","last_clean":"2023-10-16T15:54:54.532864+0000","last_became_active":"2023-10-16T15:54:53.412519+0000","last_became_peered":"2023-10-16T15:54:53.412519+0000","last_unstale":"2023-10-16T15:54:54.532864+0000","last_undegraded":"2023-10-16T15:54:54.532864+0000","last_fullsized":"2023-10-16T15:54:54.532864+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:08.839899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532435+0000","last_change":"2023-10-16T15:54:52.392344+0000","last_active":"2023-10-16T15:54:54.532435+0000","last_peered":"2023-10-16T15:54:54.532435+0000","last_clean":"2023-10-16T15:54:54.532435+0000","last_became_active":"2023-10-16T15:54:52.388373+0000","last_became_peered":"2023-10-16T15:54:52.388373+0000","last_unstale":"2023-10-16T15:54:54.532435+0000","last_undegraded":"2023-10-16T15:54:54.532435+0000","last_fullsized":"2023-10-16T15:54:54.532435+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:57.878065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386595+0000","last_change":"2023-10-16T15:54:39.210101+0000","last_active":"2023-10-16T15:54:53.386595+0000","last_peered":"2023-10-16T15:54:53.386595+0000","last_clean":"2023-10-16T15:54:53.386595+0000","last_became_active":"2023-10-16T15:54:39.209787+0000","last_became_peered":"2023-10-16T15:54:39.209787+0000","last_unstale":"2023-10-16T15:54:53.386595+0000","last_undegraded":"2023-10-16T15:54:53.386595+0000","last_fullsized":"2023-10-16T15:54:53.386595+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533984+0000","last_change":"2023-10-16T15:54:53.403896+0000","last_active":"2023-10-16T15:54:54.533984+0000","last_peered":"2023-10-16T15:54:54.533984+0000","last_clean":"2023-10-16T15:54:54.533984+0000","last_became_active":"2023-10-16T15:54:53.403550+0000","last_became_peered":"2023-10-16T15:54:53.403550+0000","last_unstale":"2023-10-16T15:54:54.533984+0000","last_undegraded":"2023-10-16T15:54:54.533984+0000","last_fullsized":"2023-10-16T15:54:54.533984+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:48.011756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"249'4","reported_seq":16,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530936+0000","last_change":"2023-10-16T15:54:52.389234+0000","last_active":"2023-10-16T15:54:54.530936+0000","last_peered":"2023-10-16T15:54:54.530936+0000","last_clean":"2023-10-16T15:54:54.530936+0000","last_became_active":"2023-10-16T15:54:52.388899+0000","last_became_peered":"2023-10-16T15:54:52.388899+0000","last_unstale":"2023-10-16T15:54:54.530936+0000","last_undegraded":"2023-10-16T15:54:54.530936+0000","last_fullsized":"2023-10-16T15:54:54.530936+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.408543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383375+0000","last_change":"2023-10-16T15:54:39.198844+0000","last_active":"2023-10-16T15:54:53.383375+0000","last_peered":"2023-10-16T15:54:53.383375+0000","last_clean":"2023-10-16T15:54:53.383375+0000","last_became_active":"2023-10-16T15:54:39.198444+0000","last_became_peered":"2023-10-16T15:54:39.198444+0000","last_unstale":"2023-10-16T15:54:53.383375+0000","last_undegraded":"2023-10-16T15:54:53.383375+0000","last_fullsized":"2023-10-16T15:54:53.383375+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:47.149143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410474+0000","last_change":"2023-10-16T15:54:53.399239+0000","last_active":"2023-10-16T15:54:54.410474+0000","last_peered":"2023-10-16T15:54:54.410474+0000","last_clean":"2023-10-16T15:54:54.410474+0000","last_became_active":"2023-10-16T15:54:53.398895+0000","last_became_peered":"2023-10-16T15:54:53.398895+0000","last_unstale":"2023-10-16T15:54:54.410474+0000","last_undegraded":"2023-10-16T15:54:54.410474+0000","last_fullsized":"2023-10-16T15:54:54.410474+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:15.592335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387724+0000","last_change":"2023-10-16T15:54:39.213572+0000","last_active":"2023-10-16T15:54:53.387724+0000","last_peered":"2023-10-16T15:54:53.387724+0000","last_clean":"2023-10-16T15:54:53.387724+0000","last_became_active":"2023-10-16T15:54:39.213226+0000","last_became_peered":"2023-10-16T15:54:39.213226+0000","last_unstale":"2023-10-16T15:54:53.387724+0000","last_undegraded":"2023-10-16T15:54:53.387724+0000","last_fullsized":"2023-10-16T15:54:53.387724+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:23.249562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390476+0000","last_change":"2023-10-16T15:54:38.194041+0000","last_active":"2023-10-16T15:54:53.390476+0000","last_peered":"2023-10-16T15:54:53.390476+0000","last_clean":"2023-10-16T15:54:53.390476+0000","last_became_active":"2023-10-16T15:54:38.193770+0000","last_became_peered":"2023-10-16T15:54:38.193770+0000","last_unstale":"2023-10-16T15:54:53.390476+0000","last_undegraded":"2023-10-16T15:54:53.390476+0000","last_fullsized":"2023-10-16T15:54:53.390476+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530335+0000","last_change":"2023-10-16T15:54:53.409258+0000","last_active":"2023-10-16T15:54:54.530335+0000","last_peered":"2023-10-16T15:54:54.530335+0000","last_clean":"2023-10-16T15:54:54.530335+0000","last_became_active":"2023-10-16T15:54:53.408227+0000","last_became_peered":"2023-10-16T15:54:53.408227+0000","last_unstale":"2023-10-16T15:54:54.530335+0000","last_undegraded":"2023-10-16T15:54:54.530335+0000","last_fullsized":"2023-10-16T15:54:54.530335+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:35.426801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369598+0000","last_change":"2023-10-16T15:54:39.209350+0000","last_active":"2023-10-16T15:54:53.369598+0000","last_peered":"2023-10-16T15:54:53.369598+0000","last_clean":"2023-10-16T15:54:53.369598+0000","last_became_active":"2023-10-16T15:54:39.208829+0000","last_became_peered":"2023-10-16T15:54:39.208829+0000","last_unstale":"2023-10-16T15:54:53.369598+0000","last_undegraded":"2023-10-16T15:54:53.369598+0000","last_fullsized":"2023-10-16T15:54:53.369598+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:20.584860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391767+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:53.391767+0000","last_peered":"2023-10-16T15:54:53.391767+0000","last_clean":"2023-10-16T15:54:53.391767+0000","last_became_active":"2023-10-16T15:54:38.195767+0000","last_became_peered":"2023-10-16T15:54:38.195767+0000","last_unstale":"2023-10-16T15:54:53.391767+0000","last_undegraded":"2023-10-16T15:54:53.391767+0000","last_fullsized":"2023-10-16T15:54:53.391767+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:59.473824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531558+0000","last_change":"2023-10-16T15:54:53.405520+0000","last_active":"2023-10-16T15:54:54.531558+0000","last_peered":"2023-10-16T15:54:54.531558+0000","last_clean":"2023-10-16T15:54:54.531558+0000","last_became_active":"2023-10-16T15:54:53.405248+0000","last_became_peered":"2023-10-16T15:54:53.405248+0000","last_unstale":"2023-10-16T15:54:54.531558+0000","last_undegraded":"2023-10-16T15:54:54.531558+0000","last_fullsized":"2023-10-16T15:54:54.531558+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:23.283640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396200+0000","last_change":"2023-10-16T15:54:39.209739+0000","last_active":"2023-10-16T15:54:53.396200+0000","last_peered":"2023-10-16T15:54:53.396200+0000","last_clean":"2023-10-16T15:54:53.396200+0000","last_became_active":"2023-10-16T15:54:39.208616+0000","last_became_peered":"2023-10-16T15:54:39.208616+0000","last_unstale":"2023-10-16T15:54:53.396200+0000","last_undegraded":"2023-10-16T15:54:53.396200+0000","last_fullsized":"2023-10-16T15:54:53.396200+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385621+0000","last_change":"2023-10-16T15:54:38.192004+0000","last_active":"2023-10-16T15:54:53.385621+0000","last_peered":"2023-10-16T15:54:53.385621+0000","last_clean":"2023-10-16T15:54:53.385621+0000","last_became_active":"2023-10-16T15:54:38.191365+0000","last_became_peered":"2023-10-16T15:54:38.191365+0000","last_unstale":"2023-10-16T15:54:53.385621+0000","last_undegraded":"2023-10-16T15:54:53.385621+0000","last_fullsized":"2023-10-16T15:54:53.385621+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:00.022703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532106+0000","last_change":"2023-10-16T15:54:53.408294+0000","last_active":"2023-10-16T15:54:54.532106+0000","last_peered":"2023-10-16T15:54:54.532106+0000","last_clean":"2023-10-16T15:54:54.532106+0000","last_became_active":"2023-10-16T15:54:53.407567+0000","last_became_peered":"2023-10-16T15:54:53.407567+0000","last_unstale":"2023-10-16T15:54:54.532106+0000","last_undegraded":"2023-10-16T15:54:54.532106+0000","last_fullsized":"2023-10-16T15:54:54.532106+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:13.405161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370148+0000","last_change":"2023-10-16T15:54:39.201055+0000","last_active":"2023-10-16T15:54:53.370148+0000","last_peered":"2023-10-16T15:54:53.370148+0000","last_clean":"2023-10-16T15:54:53.370148+0000","last_became_active":"2023-10-16T15:54:39.199846+0000","last_became_peered":"2023-10-16T15:54:39.199846+0000","last_unstale":"2023-10-16T15:54:53.370148+0000","last_undegraded":"2023-10-16T15:54:53.370148+0000","last_fullsized":"2023-10-16T15:54:53.370148+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:28.581753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385713+0000","last_change":"2023-10-16T15:54:38.188455+0000","last_active":"2023-10-16T15:54:53.385713+0000","last_peered":"2023-10-16T15:54:53.385713+0000","last_clean":"2023-10-16T15:54:53.385713+0000","last_became_active":"2023-10-16T15:54:38.188167+0000","last_became_peered":"2023-10-16T15:54:38.188167+0000","last_unstale":"2023-10-16T15:54:53.385713+0000","last_undegraded":"2023-10-16T15:54:53.385713+0000","last_fullsized":"2023-10-16T15:54:53.385713+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:21.554285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.534298+0000","last_change":"2023-10-16T15:54:53.400521+0000","last_active":"2023-10-16T15:54:54.534298+0000","last_peered":"2023-10-16T15:54:54.534298+0000","last_clean":"2023-10-16T15:54:54.534298+0000","last_became_active":"2023-10-16T15:54:53.400227+0000","last_became_peered":"2023-10-16T15:54:53.400227+0000","last_unstale":"2023-10-16T15:54:54.534298+0000","last_undegraded":"2023-10-16T15:54:54.534298+0000","last_fullsized":"2023-10-16T15:54:54.534298+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:00.410178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.377860+0000","last_change":"2023-10-16T15:54:39.198491+0000","last_active":"2023-10-16T15:54:53.377860+0000","last_peered":"2023-10-16T15:54:53.377860+0000","last_clean":"2023-10-16T15:54:53.377860+0000","last_became_active":"2023-10-16T15:54:39.194488+0000","last_became_peered":"2023-10-16T15:54:39.194488+0000","last_unstale":"2023-10-16T15:54:53.377860+0000","last_undegraded":"2023-10-16T15:54:53.377860+0000","last_fullsized":"2023-10-16T15:54:53.377860+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.663424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385449+0000","last_change":"2023-10-16T15:54:38.190757+0000","last_active":"2023-10-16T15:54:53.385449+0000","last_peered":"2023-10-16T15:54:53.385449+0000","last_clean":"2023-10-16T15:54:53.385449+0000","last_became_active":"2023-10-16T15:54:38.190491+0000","last_became_peered":"2023-10-16T15:54:38.190491+0000","last_unstale":"2023-10-16T15:54:53.385449+0000","last_undegraded":"2023-10-16T15:54:53.385449+0000","last_fullsized":"2023-10-16T15:54:53.385449+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:59.249150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533699+0000","last_change":"2023-10-16T15:54:53.406145+0000","last_active":"2023-10-16T15:54:54.533699+0000","last_peered":"2023-10-16T15:54:54.533699+0000","last_clean":"2023-10-16T15:54:54.533699+0000","last_became_active":"2023-10-16T15:54:53.405801+0000","last_became_peered":"2023-10-16T15:54:53.405801+0000","last_unstale":"2023-10-16T15:54:54.533699+0000","last_undegraded":"2023-10-16T15:54:54.533699+0000","last_fullsized":"2023-10-16T15:54:54.533699+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:41.829804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386011+0000","last_change":"2023-10-16T15:54:39.206722+0000","last_active":"2023-10-16T15:54:53.386011+0000","last_peered":"2023-10-16T15:54:53.386011+0000","last_clean":"2023-10-16T15:54:53.386011+0000","last_became_active":"2023-10-16T15:54:39.204447+0000","last_became_peered":"2023-10-16T15:54:39.204447+0000","last_unstale":"2023-10-16T15:54:53.386011+0000","last_undegraded":"2023-10-16T15:54:53.386011+0000","last_fullsized":"2023-10-16T15:54:53.386011+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:50.316780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381531+0000","last_change":"2023-10-16T15:54:38.189769+0000","last_active":"2023-10-16T15:54:53.381531+0000","last_peered":"2023-10-16T15:54:53.381531+0000","last_clean":"2023-10-16T15:54:53.381531+0000","last_became_active":"2023-10-16T15:54:38.189418+0000","last_became_peered":"2023-10-16T15:54:38.189418+0000","last_unstale":"2023-10-16T15:54:53.381531+0000","last_undegraded":"2023-10-16T15:54:53.381531+0000","last_fullsized":"2023-10-16T15:54:53.381531+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.444572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533368+0000","last_change":"2023-10-16T15:54:53.412728+0000","last_active":"2023-10-16T15:54:54.533368+0000","last_peered":"2023-10-16T15:54:54.533368+0000","last_clean":"2023-10-16T15:54:54.533368+0000","last_became_active":"2023-10-16T15:54:53.411210+0000","last_became_peered":"2023-10-16T15:54:53.411210+0000","last_unstale":"2023-10-16T15:54:54.533368+0000","last_undegraded":"2023-10-16T15:54:54.533368+0000","last_fullsized":"2023-10-16T15:54:54.533368+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:01.541143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384245+0000","last_change":"2023-10-16T15:54:39.205679+0000","last_active":"2023-10-16T15:54:53.384245+0000","last_peered":"2023-10-16T15:54:53.384245+0000","last_clean":"2023-10-16T15:54:53.384245+0000","last_became_active":"2023-10-16T15:54:39.204821+0000","last_became_peered":"2023-10-16T15:54:39.204821+0000","last_unstale":"2023-10-16T15:54:53.384245+0000","last_undegraded":"2023-10-16T15:54:53.384245+0000","last_fullsized":"2023-10-16T15:54:53.384245+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:44.694895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386273+0000","last_change":"2023-10-16T15:54:38.191853+0000","last_active":"2023-10-16T15:54:53.386273+0000","last_peered":"2023-10-16T15:54:53.386273+0000","last_clean":"2023-10-16T15:54:53.386273+0000","last_became_active":"2023-10-16T15:54:38.190725+0000","last_became_peered":"2023-10-16T15:54:38.190725+0000","last_unstale":"2023-10-16T15:54:53.386273+0000","last_undegraded":"2023-10-16T15:54:53.386273+0000","last_fullsized":"2023-10-16T15:54:53.386273+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:51.409385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529773+0000","last_change":"2023-10-16T15:54:53.402476+0000","last_active":"2023-10-16T15:54:54.529773+0000","last_peered":"2023-10-16T15:54:54.529773+0000","last_clean":"2023-10-16T15:54:54.529773+0000","last_became_active":"2023-10-16T15:54:53.402138+0000","last_became_peered":"2023-10-16T15:54:53.402138+0000","last_unstale":"2023-10-16T15:54:54.529773+0000","last_undegraded":"2023-10-16T15:54:54.529773+0000","last_fullsized":"2023-10-16T15:54:54.529773+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:15.461715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391760+0000","last_change":"2023-10-16T15:54:39.205421+0000","last_active":"2023-10-16T15:54:53.391760+0000","last_peered":"2023-10-16T15:54:53.391760+0000","last_clean":"2023-10-16T15:54:53.391760+0000","last_became_active":"2023-10-16T15:54:39.204865+0000","last_became_peered":"2023-10-16T15:54:39.204865+0000","last_unstale":"2023-10-16T15:54:53.391760+0000","last_undegraded":"2023-10-16T15:54:53.391760+0000","last_fullsized":"2023-10-16T15:54:53.391760+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:33.811103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.402410+0000","last_change":"2023-10-16T15:54:38.196240+0000","last_active":"2023-10-16T15:54:53.402410+0000","last_peered":"2023-10-16T15:54:53.402410+0000","last_clean":"2023-10-16T15:54:53.402410+0000","last_became_active":"2023-10-16T15:54:38.195833+0000","last_became_peered":"2023-10-16T15:54:38.195833+0000","last_unstale":"2023-10-16T15:54:53.402410+0000","last_undegraded":"2023-10-16T15:54:53.402410+0000","last_fullsized":"2023-10-16T15:54:53.402410+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531020+0000","last_change":"2023-10-16T15:54:53.392476+0000","last_active":"2023-10-16T15:54:54.531020+0000","last_peered":"2023-10-16T15:54:54.531020+0000","last_clean":"2023-10-16T15:54:54.531020+0000","last_became_active":"2023-10-16T15:54:53.391954+0000","last_became_peered":"2023-10-16T15:54:53.391954+0000","last_unstale":"2023-10-16T15:54:54.531020+0000","last_undegraded":"2023-10-16T15:54:54.531020+0000","last_fullsized":"2023-10-16T15:54:54.531020+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:07.891800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.371969+0000","last_change":"2023-10-16T15:54:39.204677+0000","last_active":"2023-10-16T15:54:53.371969+0000","last_peered":"2023-10-16T15:54:53.371969+0000","last_clean":"2023-10-16T15:54:53.371969+0000","last_became_active":"2023-10-16T15:54:39.202794+0000","last_became_peered":"2023-10-16T15:54:39.202794+0000","last_unstale":"2023-10-16T15:54:53.371969+0000","last_undegraded":"2023-10-16T15:54:53.371969+0000","last_fullsized":"2023-10-16T15:54:53.371969+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:11.420100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.382524+0000","last_change":"2023-10-16T15:54:38.195322+0000","last_active":"2023-10-16T15:54:53.382524+0000","last_peered":"2023-10-16T15:54:53.382524+0000","last_clean":"2023-10-16T15:54:53.382524+0000","last_became_active":"2023-10-16T15:54:38.192163+0000","last_became_peered":"2023-10-16T15:54:38.192163+0000","last_unstale":"2023-10-16T15:54:53.382524+0000","last_undegraded":"2023-10-16T15:54:53.382524+0000","last_fullsized":"2023-10-16T15:54:53.382524+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:45.552506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405742+0000","last_change":"2023-10-16T15:54:53.404252+0000","last_active":"2023-10-16T15:54:54.405742+0000","last_peered":"2023-10-16T15:54:54.405742+0000","last_clean":"2023-10-16T15:54:54.405742+0000","last_became_active":"2023-10-16T15:54:53.403522+0000","last_became_peered":"2023-10-16T15:54:53.403522+0000","last_unstale":"2023-10-16T15:54:54.405742+0000","last_undegraded":"2023-10-16T15:54:54.405742+0000","last_fullsized":"2023-10-16T15:54:54.405742+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:44.102171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385490+0000","last_change":"2023-10-16T15:54:39.212753+0000","last_active":"2023-10-16T15:54:53.385490+0000","last_peered":"2023-10-16T15:54:53.385490+0000","last_clean":"2023-10-16T15:54:53.385490+0000","last_became_active":"2023-10-16T15:54:39.212450+0000","last_became_peered":"2023-10-16T15:54:39.212450+0000","last_unstale":"2023-10-16T15:54:53.385490+0000","last_undegraded":"2023-10-16T15:54:53.385490+0000","last_fullsized":"2023-10-16T15:54:53.385490+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:04.593631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"245'3","reported_seq":19,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391258+0000","last_change":"2023-10-16T15:54:38.188991+0000","last_active":"2023-10-16T15:54:53.391258+0000","last_peered":"2023-10-16T15:54:53.391258+0000","last_clean":"2023-10-16T15:54:53.391258+0000","last_became_active":"2023-10-16T15:54:38.187521+0000","last_became_peered":"2023-10-16T15:54:38.187521+0000","last_unstale":"2023-10-16T15:54:53.391258+0000","last_undegraded":"2023-10-16T15:54:53.391258+0000","last_fullsized":"2023-10-16T15:54:53.391258+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:37.422309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405892+0000","last_change":"2023-10-16T15:54:53.396365+0000","last_active":"2023-10-16T15:54:54.405892+0000","last_peered":"2023-10-16T15:54:54.405892+0000","last_clean":"2023-10-16T15:54:54.405892+0000","last_became_active":"2023-10-16T15:54:53.393308+0000","last_became_peered":"2023-10-16T15:54:53.393308+0000","last_unstale":"2023-10-16T15:54:54.405892+0000","last_undegraded":"2023-10-16T15:54:54.405892+0000","last_fullsized":"2023-10-16T15:54:54.405892+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:58.760049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386407+0000","last_change":"2023-10-16T15:54:39.188004+0000","last_active":"2023-10-16T15:54:53.386407+0000","last_peered":"2023-10-16T15:54:53.386407+0000","last_clean":"2023-10-16T15:54:53.386407+0000","last_became_active":"2023-10-16T15:54:39.187443+0000","last_became_peered":"2023-10-16T15:54:39.187443+0000","last_unstale":"2023-10-16T15:54:53.386407+0000","last_undegraded":"2023-10-16T15:54:53.386407+0000","last_fullsized":"2023-10-16T15:54:53.386407+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"247'8","reported_seq":24,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.389360+0000","last_change":"2023-10-16T15:54:38.192821+0000","last_active":"2023-10-16T15:54:53.389360+0000","last_peered":"2023-10-16T15:54:53.389360+0000","last_clean":"2023-10-16T15:54:53.389360+0000","last_became_active":"2023-10-16T15:54:38.191495+0000","last_became_peered":"2023-10-16T15:54:38.191495+0000","last_unstale":"2023-10-16T15:54:53.389360+0000","last_undegraded":"2023-10-16T15:54:53.389360+0000","last_fullsized":"2023-10-16T15:54:53.389360+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:36.442402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530637+0000","last_change":"2023-10-16T15:54:53.404553+0000","last_active":"2023-10-16T15:54:54.530637+0000","last_peered":"2023-10-16T15:54:54.530637+0000","last_clean":"2023-10-16T15:54:54.530637+0000","last_became_active":"2023-10-16T15:54:53.404272+0000","last_became_peered":"2023-10-16T15:54:53.404272+0000","last_unstale":"2023-10-16T15:54:54.530637+0000","last_undegraded":"2023-10-16T15:54:54.530637+0000","last_fullsized":"2023-10-16T15:54:54.530637+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:55.651126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384915+0000","last_change":"2023-10-16T15:54:39.203188+0000","last_active":"2023-10-16T15:54:53.384915+0000","last_peered":"2023-10-16T15:54:53.384915+0000","last_clean":"2023-10-16T15:54:53.384915+0000","last_became_active":"2023-10-16T15:54:39.202646+0000","last_became_peered":"2023-10-16T15:54:39.202646+0000","last_unstale":"2023-10-16T15:54:53.384915+0000","last_undegraded":"2023-10-16T15:54:53.384915+0000","last_fullsized":"2023-10-16T15:54:53.384915+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387656+0000","last_change":"2023-10-16T15:54:38.190094+0000","last_active":"2023-10-16T15:54:53.387656+0000","last_peered":"2023-10-16T15:54:53.387656+0000","last_clean":"2023-10-16T15:54:53.387656+0000","last_became_active":"2023-10-16T15:54:38.189825+0000","last_became_peered":"2023-10-16T15:54:38.189825+0000","last_unstale":"2023-10-16T15:54:53.387656+0000","last_undegraded":"2023-10-16T15:54:53.387656+0000","last_fullsized":"2023-10-16T15:54:53.387656+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:17.021872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533723+0000","last_change":"2023-10-16T15:54:53.403026+0000","last_active":"2023-10-16T15:54:54.533723+0000","last_peered":"2023-10-16T15:54:54.533723+0000","last_clean":"2023-10-16T15:54:54.533723+0000","last_became_active":"2023-10-16T15:54:53.402600+0000","last_became_peered":"2023-10-16T15:54:53.402600+0000","last_unstale":"2023-10-16T15:54:54.533723+0000","last_undegraded":"2023-10-16T15:54:54.533723+0000","last_fullsized":"2023-10-16T15:54:54.533723+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:08.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.378063+0000","last_change":"2023-10-16T15:54:39.205965+0000","last_active":"2023-10-16T15:54:53.378063+0000","last_peered":"2023-10-16T15:54:53.378063+0000","last_clean":"2023-10-16T15:54:53.378063+0000","last_became_active":"2023-10-16T15:54:39.205686+0000","last_became_peered":"2023-10-16T15:54:39.205686+0000","last_unstale":"2023-10-16T15:54:53.378063+0000","last_undegraded":"2023-10-16T15:54:53.378063+0000","last_fullsized":"2023-10-16T15:54:53.378063+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:36.119641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385348+0000","last_change":"2023-10-16T15:54:38.195373+0000","last_active":"2023-10-16T15:54:53.385348+0000","last_peered":"2023-10-16T15:54:53.385348+0000","last_clean":"2023-10-16T15:54:53.385348+0000","last_became_active":"2023-10-16T15:54:38.194574+0000","last_became_peered":"2023-10-16T15:54:38.194574+0000","last_unstale":"2023-10-16T15:54:53.385348+0000","last_undegraded":"2023-10-16T15:54:53.385348+0000","last_fullsized":"2023-10-16T15:54:53.385348+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:35.101378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.535386+0000","last_change":"2023-10-16T15:54:53.395661+0000","last_active":"2023-10-16T15:54:54.535386+0000","last_peered":"2023-10-16T15:54:54.535386+0000","last_clean":"2023-10-16T15:54:54.535386+0000","last_became_active":"2023-10-16T15:54:53.395114+0000","last_became_peered":"2023-10-16T15:54:53.395114+0000","last_unstale":"2023-10-16T15:54:54.535386+0000","last_undegraded":"2023-10-16T15:54:54.535386+0000","last_fullsized":"2023-10-16T15:54:54.535386+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:57.466035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.382788+0000","last_change":"2023-10-16T15:54:39.207613+0000","last_active":"2023-10-16T15:54:53.382788+0000","last_peered":"2023-10-16T15:54:53.382788+0000","last_clean":"2023-10-16T15:54:53.382788+0000","last_became_active":"2023-10-16T15:54:39.207320+0000","last_became_peered":"2023-10-16T15:54:39.207320+0000","last_unstale":"2023-10-16T15:54:53.382788+0000","last_undegraded":"2023-10-16T15:54:53.382788+0000","last_fullsized":"2023-10-16T15:54:53.382788+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383033+0000","last_change":"2023-10-16T15:54:38.196425+0000","last_active":"2023-10-16T15:54:53.383033+0000","last_peered":"2023-10-16T15:54:53.383033+0000","last_clean":"2023-10-16T15:54:53.383033+0000","last_became_active":"2023-10-16T15:54:38.195269+0000","last_became_peered":"2023-10-16T15:54:38.195269+0000","last_unstale":"2023-10-16T15:54:53.383033+0000","last_undegraded":"2023-10-16T15:54:53.383033+0000","last_fullsized":"2023-10-16T15:54:53.383033+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:01.207615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531467+0000","last_change":"2023-10-16T15:54:53.400454+0000","last_active":"2023-10-16T15:54:54.531467+0000","last_peered":"2023-10-16T15:54:54.531467+0000","last_clean":"2023-10-16T15:54:54.531467+0000","last_became_active":"2023-10-16T15:54:53.400141+0000","last_became_peered":"2023-10-16T15:54:53.400141+0000","last_unstale":"2023-10-16T15:54:54.531467+0000","last_undegraded":"2023-10-16T15:54:54.531467+0000","last_fullsized":"2023-10-16T15:54:54.531467+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:57.598566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381337+0000","last_change":"2023-10-16T15:54:39.205462+0000","last_active":"2023-10-16T15:54:53.381337+0000","last_peered":"2023-10-16T15:54:53.381337+0000","last_clean":"2023-10-16T15:54:53.381337+0000","last_became_active":"2023-10-16T15:54:39.204956+0000","last_became_peered":"2023-10-16T15:54:39.204956+0000","last_unstale":"2023-10-16T15:54:53.381337+0000","last_undegraded":"2023-10-16T15:54:53.381337+0000","last_fullsized":"2023-10-16T15:54:53.381337+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"245'8","reported_seq":24,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369047+0000","last_change":"2023-10-16T15:54:38.194134+0000","last_active":"2023-10-16T15:54:53.369047+0000","last_peered":"2023-10-16T15:54:53.369047+0000","last_clean":"2023-10-16T15:54:53.369047+0000","last_became_active":"2023-10-16T15:54:38.193858+0000","last_became_peered":"2023-10-16T15:54:38.193858+0000","last_unstale":"2023-10-16T15:54:53.369047+0000","last_undegraded":"2023-10-16T15:54:53.369047+0000","last_fullsized":"2023-10-16T15:54:53.369047+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:37.866131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531697+0000","last_change":"2023-10-16T15:54:53.416410+0000","last_active":"2023-10-16T15:54:54.531697+0000","last_peered":"2023-10-16T15:54:54.531697+0000","last_clean":"2023-10-16T15:54:54.531697+0000","last_became_active":"2023-10-16T15:54:53.413028+0000","last_became_peered":"2023-10-16T15:54:53.413028+0000","last_unstale":"2023-10-16T15:54:54.531697+0000","last_undegraded":"2023-10-16T15:54:54.531697+0000","last_fullsized":"2023-10-16T15:54:54.531697+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:10.907637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391617+0000","last_change":"2023-10-16T15:54:39.203343+0000","last_active":"2023-10-16T15:54:53.391617+0000","last_peered":"2023-10-16T15:54:53.391617+0000","last_clean":"2023-10-16T15:54:53.391617+0000","last_became_active":"2023-10-16T15:54:39.203044+0000","last_became_peered":"2023-10-16T15:54:39.203044+0000","last_unstale":"2023-10-16T15:54:53.391617+0000","last_undegraded":"2023-10-16T15:54:53.391617+0000","last_fullsized":"2023-10-16T15:54:53.391617+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:23:49.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.389407+0000","last_change":"2023-10-16T15:54:38.193502+0000","last_active":"2023-10-16T15:54:53.389407+0000","last_peered":"2023-10-16T15:54:53.389407+0000","last_clean":"2023-10-16T15:54:53.389407+0000","last_became_active":"2023-10-16T15:54:38.193186+0000","last_became_peered":"2023-10-16T15:54:38.193186+0000","last_unstale":"2023-10-16T15:54:53.389407+0000","last_undegraded":"2023-10-16T15:54:53.389407+0000","last_fullsized":"2023-10-16T15:54:53.389407+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531341+0000","last_change":"2023-10-16T15:54:53.406582+0000","last_active":"2023-10-16T15:54:54.531341+0000","last_peered":"2023-10-16T15:54:54.531341+0000","last_clean":"2023-10-16T15:54:54.531341+0000","last_became_active":"2023-10-16T15:54:53.406311+0000","last_became_peered":"2023-10-16T15:54:53.406311+0000","last_unstale":"2023-10-16T15:54:54.531341+0000","last_undegraded":"2023-10-16T15:54:54.531341+0000","last_fullsized":"2023-10-16T15:54:54.531341+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:51.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.388049+0000","last_change":"2023-10-16T15:54:39.206218+0000","last_active":"2023-10-16T15:54:53.388049+0000","last_peered":"2023-10-16T15:54:53.388049+0000","last_clean":"2023-10-16T15:54:53.388049+0000","last_became_active":"2023-10-16T15:54:39.205815+0000","last_became_peered":"2023-10-16T15:54:39.205815+0000","last_unstale":"2023-10-16T15:54:53.388049+0000","last_undegraded":"2023-10-16T15:54:53.388049+0000","last_fullsized":"2023-10-16T15:54:53.388049+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:24.642539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.382751+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:53.382751+0000","last_peered":"2023-10-16T15:54:53.382751+0000","last_clean":"2023-10-16T15:54:53.382751+0000","last_became_active":"2023-10-16T15:54:38.195752+0000","last_became_peered":"2023-10-16T15:54:38.195752+0000","last_unstale":"2023-10-16T15:54:53.382751+0000","last_undegraded":"2023-10-16T15:54:53.382751+0000","last_fullsized":"2023-10-16T15:54:53.382751+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:15.145108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530007+0000","last_change":"2023-10-16T15:54:53.416186+0000","last_active":"2023-10-16T15:54:54.530007+0000","last_peered":"2023-10-16T15:54:54.530007+0000","last_clean":"2023-10-16T15:54:54.530007+0000","last_became_active":"2023-10-16T15:54:53.415871+0000","last_became_peered":"2023-10-16T15:54:53.415871+0000","last_unstale":"2023-10-16T15:54:54.530007+0000","last_undegraded":"2023-10-16T15:54:54.530007+0000","last_fullsized":"2023-10-16T15:54:54.530007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:57.375885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.372975+0000","last_change":"2023-10-16T15:54:39.198603+0000","last_active":"2023-10-16T15:54:53.372975+0000","last_peered":"2023-10-16T15:54:53.372975+0000","last_clean":"2023-10-16T15:54:53.372975+0000","last_became_active":"2023-10-16T15:54:39.195793+0000","last_became_peered":"2023-10-16T15:54:39.195793+0000","last_unstale":"2023-10-16T15:54:53.372975+0000","last_undegraded":"2023-10-16T15:54:53.372975+0000","last_fullsized":"2023-10-16T15:54:53.372975+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:16.753544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375520+0000","last_change":"2023-10-16T15:54:38.193258+0000","last_active":"2023-10-16T15:54:53.375520+0000","last_peered":"2023-10-16T15:54:53.375520+0000","last_clean":"2023-10-16T15:54:53.375520+0000","last_became_active":"2023-10-16T15:54:38.192538+0000","last_became_peered":"2023-10-16T15:54:38.192538+0000","last_unstale":"2023-10-16T15:54:53.375520+0000","last_undegraded":"2023-10-16T15:54:53.375520+0000","last_fullsized":"2023-10-16T15:54:53.375520+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:07.463985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531738+0000","last_change":"2023-10-16T15:54:53.407645+0000","last_active":"2023-10-16T15:54:54.531738+0000","last_peered":"2023-10-16T15:54:54.531738+0000","last_clean":"2023-10-16T15:54:54.531738+0000","last_became_active":"2023-10-16T15:54:53.407018+0000","last_became_peered":"2023-10-16T15:54:53.407018+0000","last_unstale":"2023-10-16T15:54:54.531738+0000","last_undegraded":"2023-10-16T15:54:54.531738+0000","last_fullsized":"2023-10-16T15:54:54.531738+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:00.410178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396297+0000","last_change":"2023-10-16T15:54:39.206968+0000","last_active":"2023-10-16T15:54:53.396297+0000","last_peered":"2023-10-16T15:54:53.396297+0000","last_clean":"2023-10-16T15:54:53.396297+0000","last_became_active":"2023-10-16T15:54:39.206271+0000","last_became_peered":"2023-10-16T15:54:39.206271+0000","last_unstale":"2023-10-16T15:54:53.396297+0000","last_undegraded":"2023-10-16T15:54:53.396297+0000","last_fullsized":"2023-10-16T15:54:53.396297+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369863+0000","last_change":"2023-10-16T15:54:38.194296+0000","last_active":"2023-10-16T15:54:53.369863+0000","last_peered":"2023-10-16T15:54:53.369863+0000","last_clean":"2023-10-16T15:54:53.369863+0000","last_became_active":"2023-10-16T15:54:38.194015+0000","last_became_peered":"2023-10-16T15:54:38.194015+0000","last_unstale":"2023-10-16T15:54:53.369863+0000","last_undegraded":"2023-10-16T15:54:53.369863+0000","last_fullsized":"2023-10-16T15:54:53.369863+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:58.920639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412516+0000","last_change":"2023-10-16T15:54:53.411797+0000","last_active":"2023-10-16T15:54:54.412516+0000","last_peered":"2023-10-16T15:54:54.412516+0000","last_clean":"2023-10-16T15:54:54.412516+0000","last_became_active":"2023-10-16T15:54:53.411433+0000","last_became_peered":"2023-10-16T15:54:53.411433+0000","last_unstale":"2023-10-16T15:54:54.412516+0000","last_undegraded":"2023-10-16T15:54:54.412516+0000","last_fullsized":"2023-10-16T15:54:54.412516+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:51.661319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387054+0000","last_change":"2023-10-16T15:54:39.206512+0000","last_active":"2023-10-16T15:54:53.387054+0000","last_peered":"2023-10-16T15:54:53.387054+0000","last_clean":"2023-10-16T15:54:53.387054+0000","last_became_active":"2023-10-16T15:54:39.206174+0000","last_became_peered":"2023-10-16T15:54:39.206174+0000","last_unstale":"2023-10-16T15:54:53.387054+0000","last_undegraded":"2023-10-16T15:54:53.387054+0000","last_fullsized":"2023-10-16T15:54:53.387054+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.388052+0000","last_change":"2023-10-16T15:54:38.199965+0000","last_active":"2023-10-16T15:54:53.388052+0000","last_peered":"2023-10-16T15:54:53.388052+0000","last_clean":"2023-10-16T15:54:53.388052+0000","last_became_active":"2023-10-16T15:54:38.199417+0000","last_became_peered":"2023-10-16T15:54:38.199417+0000","last_unstale":"2023-10-16T15:54:53.388052+0000","last_undegraded":"2023-10-16T15:54:53.388052+0000","last_fullsized":"2023-10-16T15:54:53.388052+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:55.688720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533410+0000","last_change":"2023-10-16T15:54:53.396057+0000","last_active":"2023-10-16T15:54:54.533410+0000","last_peered":"2023-10-16T15:54:54.533410+0000","last_clean":"2023-10-16T15:54:54.533410+0000","last_became_active":"2023-10-16T15:54:53.395705+0000","last_became_peered":"2023-10-16T15:54:53.395705+0000","last_unstale":"2023-10-16T15:54:54.533410+0000","last_undegraded":"2023-10-16T15:54:54.533410+0000","last_fullsized":"2023-10-16T15:54:54.533410+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:35.426801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387556+0000","last_change":"2023-10-16T15:54:39.199452+0000","last_active":"2023-10-16T15:54:53.387556+0000","last_peered":"2023-10-16T15:54:53.387556+0000","last_clean":"2023-10-16T15:54:53.387556+0000","last_became_active":"2023-10-16T15:54:39.199132+0000","last_became_peered":"2023-10-16T15:54:39.199132+0000","last_unstale":"2023-10-16T15:54:53.387556+0000","last_undegraded":"2023-10-16T15:54:53.387556+0000","last_fullsized":"2023-10-16T15:54:53.387556+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:49.564533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.372618+0000","last_change":"2023-10-16T15:54:38.193942+0000","last_active":"2023-10-16T15:54:53.372618+0000","last_peered":"2023-10-16T15:54:53.372618+0000","last_clean":"2023-10-16T15:54:53.372618+0000","last_became_active":"2023-10-16T15:54:38.193397+0000","last_became_peered":"2023-10-16T15:54:38.193397+0000","last_unstale":"2023-10-16T15:54:53.372618+0000","last_undegraded":"2023-10-16T15:54:53.372618+0000","last_fullsized":"2023-10-16T15:54:53.372618+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:40.933295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533641+0000","last_change":"2023-10-16T15:54:53.404406+0000","last_active":"2023-10-16T15:54:54.533641+0000","last_peered":"2023-10-16T15:54:54.533641+0000","last_clean":"2023-10-16T15:54:54.533641+0000","last_became_active":"2023-10-16T15:54:53.403821+0000","last_became_peered":"2023-10-16T15:54:53.403821+0000","last_unstale":"2023-10-16T15:54:54.533641+0000","last_undegraded":"2023-10-16T15:54:54.533641+0000","last_fullsized":"2023-10-16T15:54:54.533641+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:36.633161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.368895+0000","last_change":"2023-10-16T15:54:39.188638+0000","last_active":"2023-10-16T15:54:53.368895+0000","last_peered":"2023-10-16T15:54:53.368895+0000","last_clean":"2023-10-16T15:54:53.368895+0000","last_became_active":"2023-10-16T15:54:39.187890+0000","last_became_peered":"2023-10-16T15:54:39.187890+0000","last_unstale":"2023-10-16T15:54:53.368895+0000","last_undegraded":"2023-10-16T15:54:53.368895+0000","last_fullsized":"2023-10-16T15:54:53.368895+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:15:41.121120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375961+0000","last_change":"2023-10-16T15:54:38.196671+0000","last_active":"2023-10-16T15:54:53.375961+0000","last_peered":"2023-10-16T15:54:53.375961+0000","last_clean":"2023-10-16T15:54:53.375961+0000","last_became_active":"2023-10-16T15:54:38.196392+0000","last_became_peered":"2023-10-16T15:54:38.196392+0000","last_unstale":"2023-10-16T15:54:53.375961+0000","last_undegraded":"2023-10-16T15:54:53.375961+0000","last_fullsized":"2023-10-16T15:54:53.375961+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:35.189299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.406794+0000","last_change":"2023-10-16T15:54:53.392163+0000","last_active":"2023-10-16T15:54:54.406794+0000","last_peered":"2023-10-16T15:54:54.406794+0000","last_clean":"2023-10-16T15:54:54.406794+0000","last_became_active":"2023-10-16T15:54:53.389078+0000","last_became_peered":"2023-10-16T15:54:53.389078+0000","last_unstale":"2023-10-16T15:54:54.406794+0000","last_undegraded":"2023-10-16T15:54:54.406794+0000","last_fullsized":"2023-10-16T15:54:54.406794+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:05.340702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384551+0000","last_change":"2023-10-16T15:54:39.203931+0000","last_active":"2023-10-16T15:54:53.384551+0000","last_peered":"2023-10-16T15:54:53.384551+0000","last_clean":"2023-10-16T15:54:53.384551+0000","last_became_active":"2023-10-16T15:54:39.203614+0000","last_became_peered":"2023-10-16T15:54:39.203614+0000","last_unstale":"2023-10-16T15:54:53.384551+0000","last_undegraded":"2023-10-16T15:54:53.384551+0000","last_fullsized":"2023-10-16T15:54:53.384551+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381805+0000","last_change":"2023-10-16T15:54:38.187905+0000","last_active":"2023-10-16T15:54:53.381805+0000","last_peered":"2023-10-16T15:54:53.381805+0000","last_clean":"2023-10-16T15:54:53.381805+0000","last_became_active":"2023-10-16T15:54:38.187553+0000","last_became_peered":"2023-10-16T15:54:38.187553+0000","last_unstale":"2023-10-16T15:54:53.381805+0000","last_undegraded":"2023-10-16T15:54:53.381805+0000","last_fullsized":"2023-10-16T15:54:53.381805+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:27.563490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.408240+0000","last_change":"2023-10-16T15:54:53.404941+0000","last_active":"2023-10-16T15:54:54.408240+0000","last_peered":"2023-10-16T15:54:54.408240+0000","last_clean":"2023-10-16T15:54:54.408240+0000","last_became_active":"2023-10-16T15:54:53.404667+0000","last_became_peered":"2023-10-16T15:54:53.404667+0000","last_unstale":"2023-10-16T15:54:54.408240+0000","last_undegraded":"2023-10-16T15:54:54.408240+0000","last_fullsized":"2023-10-16T15:54:54.408240+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:51:29.050486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390320+0000","last_change":"2023-10-16T15:54:39.201684+0000","last_active":"2023-10-16T15:54:53.390320+0000","last_peered":"2023-10-16T15:54:53.390320+0000","last_clean":"2023-10-16T15:54:53.390320+0000","last_became_active":"2023-10-16T15:54:39.201363+0000","last_became_peered":"2023-10-16T15:54:39.201363+0000","last_unstale":"2023-10-16T15:54:53.390320+0000","last_undegraded":"2023-10-16T15:54:53.390320+0000","last_fullsized":"2023-10-16T15:54:53.390320+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.404031+0000","last_change":"2023-10-16T15:54:38.193490+0000","last_active":"2023-10-16T15:54:53.404031+0000","last_peered":"2023-10-16T15:54:53.404031+0000","last_clean":"2023-10-16T15:54:53.404031+0000","last_became_active":"2023-10-16T15:54:38.193206+0000","last_became_peered":"2023-10-16T15:54:38.193206+0000","last_unstale":"2023-10-16T15:54:53.404031+0000","last_undegraded":"2023-10-16T15:54:53.404031+0000","last_fullsized":"2023-10-16T15:54:53.404031+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:39.542212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531029+0000","last_change":"2023-10-16T15:54:53.395414+0000","last_active":"2023-10-16T15:54:54.531029+0000","last_peered":"2023-10-16T15:54:54.531029+0000","last_clean":"2023-10-16T15:54:54.531029+0000","last_became_active":"2023-10-16T15:54:53.394997+0000","last_became_peered":"2023-10-16T15:54:53.394997+0000","last_unstale":"2023-10-16T15:54:54.531029+0000","last_undegraded":"2023-10-16T15:54:54.531029+0000","last_fullsized":"2023-10-16T15:54:54.531029+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:01.349796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.392507+0000","last_change":"2023-10-16T15:54:39.206833+0000","last_active":"2023-10-16T15:54:53.392507+0000","last_peered":"2023-10-16T15:54:53.392507+0000","last_clean":"2023-10-16T15:54:53.392507+0000","last_became_active":"2023-10-16T15:54:39.206491+0000","last_became_peered":"2023-10-16T15:54:39.206491+0000","last_unstale":"2023-10-16T15:54:53.392507+0000","last_undegraded":"2023-10-16T15:54:53.392507+0000","last_fullsized":"2023-10-16T15:54:53.392507+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:22.432507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"246'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.399788+0000","last_change":"2023-10-16T15:54:38.198292+0000","last_active":"2023-10-16T15:54:53.399788+0000","last_peered":"2023-10-16T15:54:53.399788+0000","last_clean":"2023-10-16T15:54:53.399788+0000","last_became_active":"2023-10-16T15:54:38.197965+0000","last_became_peered":"2023-10-16T15:54:38.197965+0000","last_unstale":"2023-10-16T15:54:53.399788+0000","last_undegraded":"2023-10-16T15:54:53.399788+0000","last_fullsized":"2023-10-16T15:54:53.399788+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:07.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529426+0000","last_change":"2023-10-16T15:54:53.403179+0000","last_active":"2023-10-16T15:54:54.529426+0000","last_peered":"2023-10-16T15:54:54.529426+0000","last_clean":"2023-10-16T15:54:54.529426+0000","last_became_active":"2023-10-16T15:54:53.402875+0000","last_became_peered":"2023-10-16T15:54:53.402875+0000","last_unstale":"2023-10-16T15:54:54.529426+0000","last_undegraded":"2023-10-16T15:54:54.529426+0000","last_fullsized":"2023-10-16T15:54:54.529426+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:13.864078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369757+0000","last_change":"2023-10-16T15:54:39.198360+0000","last_active":"2023-10-16T15:54:53.369757+0000","last_peered":"2023-10-16T15:54:53.369757+0000","last_clean":"2023-10-16T15:54:53.369757+0000","last_became_active":"2023-10-16T15:54:39.196440+0000","last_became_peered":"2023-10-16T15:54:39.196440+0000","last_unstale":"2023-10-16T15:54:53.369757+0000","last_undegraded":"2023-10-16T15:54:53.369757+0000","last_fullsized":"2023-10-16T15:54:53.369757+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:23.439544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375749+0000","last_change":"2023-10-16T15:54:38.199930+0000","last_active":"2023-10-16T15:54:53.375749+0000","last_peered":"2023-10-16T15:54:53.375749+0000","last_clean":"2023-10-16T15:54:53.375749+0000","last_became_active":"2023-10-16T15:54:38.199386+0000","last_became_peered":"2023-10-16T15:54:38.199386+0000","last_unstale":"2023-10-16T15:54:53.375749+0000","last_undegraded":"2023-10-16T15:54:53.375749+0000","last_fullsized":"2023-10-16T15:54:53.375749+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:57.311627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532738+0000","last_change":"2023-10-16T15:54:53.404203+0000","last_active":"2023-10-16T15:54:54.532738+0000","last_peered":"2023-10-16T15:54:54.532738+0000","last_clean":"2023-10-16T15:54:54.532738+0000","last_became_active":"2023-10-16T15:54:53.403894+0000","last_became_peered":"2023-10-16T15:54:53.403894+0000","last_unstale":"2023-10-16T15:54:54.532738+0000","last_undegraded":"2023-10-16T15:54:54.532738+0000","last_fullsized":"2023-10-16T15:54:54.532738+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:01.054329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.371113+0000","last_change":"2023-10-16T15:54:39.205703+0000","last_active":"2023-10-16T15:54:53.371113+0000","last_peered":"2023-10-16T15:54:53.371113+0000","last_clean":"2023-10-16T15:54:53.371113+0000","last_became_active":"2023-10-16T15:54:39.205415+0000","last_became_peered":"2023-10-16T15:54:39.205415+0000","last_unstale":"2023-10-16T15:54:53.371113+0000","last_undegraded":"2023-10-16T15:54:53.371113+0000","last_fullsized":"2023-10-16T15:54:53.371113+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:02.225878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370038+0000","last_change":"2023-10-16T15:54:38.184497+0000","last_active":"2023-10-16T15:54:53.370038+0000","last_peered":"2023-10-16T15:54:53.370038+0000","last_clean":"2023-10-16T15:54:53.370038+0000","last_became_active":"2023-10-16T15:54:38.184043+0000","last_became_peered":"2023-10-16T15:54:38.184043+0000","last_unstale":"2023-10-16T15:54:53.370038+0000","last_undegraded":"2023-10-16T15:54:53.370038+0000","last_fullsized":"2023-10-16T15:54:53.370038+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:04.454449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532675+0000","last_change":"2023-10-16T15:54:53.404681+0000","last_active":"2023-10-16T15:54:54.532675+0000","last_peered":"2023-10-16T15:54:54.532675+0000","last_clean":"2023-10-16T15:54:54.532675+0000","last_became_active":"2023-10-16T15:54:53.404185+0000","last_became_peered":"2023-10-16T15:54:53.404185+0000","last_unstale":"2023-10-16T15:54:54.532675+0000","last_undegraded":"2023-10-16T15:54:54.532675+0000","last_fullsized":"2023-10-16T15:54:54.532675+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:36.773201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385453+0000","last_change":"2023-10-16T15:54:39.192680+0000","last_active":"2023-10-16T15:54:53.385453+0000","last_peered":"2023-10-16T15:54:53.385453+0000","last_clean":"2023-10-16T15:54:53.385453+0000","last_became_active":"2023-10-16T15:54:39.192254+0000","last_became_peered":"2023-10-16T15:54:39.192254+0000","last_unstale":"2023-10-16T15:54:53.385453+0000","last_undegraded":"2023-10-16T15:54:53.385453+0000","last_fullsized":"2023-10-16T15:54:53.385453+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386828+0000","last_change":"2023-10-16T15:54:38.192812+0000","last_active":"2023-10-16T15:54:53.386828+0000","last_peered":"2023-10-16T15:54:53.386828+0000","last_clean":"2023-10-16T15:54:53.386828+0000","last_became_active":"2023-10-16T15:54:38.192528+0000","last_became_peered":"2023-10-16T15:54:38.192528+0000","last_unstale":"2023-10-16T15:54:53.386828+0000","last_undegraded":"2023-10-16T15:54:53.386828+0000","last_fullsized":"2023-10-16T15:54:53.386828+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:36.002573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.534498+0000","last_change":"2023-10-16T15:54:53.415073+0000","last_active":"2023-10-16T15:54:54.534498+0000","last_peered":"2023-10-16T15:54:54.534498+0000","last_clean":"2023-10-16T15:54:54.534498+0000","last_became_active":"2023-10-16T15:54:53.414773+0000","last_became_peered":"2023-10-16T15:54:53.414773+0000","last_unstale":"2023-10-16T15:54:54.534498+0000","last_undegraded":"2023-10-16T15:54:54.534498+0000","last_fullsized":"2023-10-16T15:54:54.534498+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:40.643056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387976+0000","last_change":"2023-10-16T15:54:39.207142+0000","last_active":"2023-10-16T15:54:53.387976+0000","last_peered":"2023-10-16T15:54:53.387976+0000","last_clean":"2023-10-16T15:54:53.387976+0000","last_became_active":"2023-10-16T15:54:39.206593+0000","last_became_peered":"2023-10-16T15:54:39.206593+0000","last_unstale":"2023-10-16T15:54:53.387976+0000","last_undegraded":"2023-10-16T15:54:53.387976+0000","last_fullsized":"2023-10-16T15:54:53.387976+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:58.545523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384227+0000","last_change":"2023-10-16T15:54:38.179331+0000","last_active":"2023-10-16T15:54:53.384227+0000","last_peered":"2023-10-16T15:54:53.384227+0000","last_clean":"2023-10-16T15:54:53.384227+0000","last_became_active":"2023-10-16T15:54:38.178684+0000","last_became_peered":"2023-10-16T15:54:38.178684+0000","last_unstale":"2023-10-16T15:54:53.384227+0000","last_undegraded":"2023-10-16T15:54:53.384227+0000","last_fullsized":"2023-10-16T15:54:53.384227+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.411175+0000","last_change":"2023-10-16T15:54:53.399801+0000","last_active":"2023-10-16T15:54:54.411175+0000","last_peered":"2023-10-16T15:54:54.411175+0000","last_clean":"2023-10-16T15:54:54.411175+0000","last_became_active":"2023-10-16T15:54:53.399400+0000","last_became_peered":"2023-10-16T15:54:53.399400+0000","last_unstale":"2023-10-16T15:54:54.411175+0000","last_undegraded":"2023-10-16T15:54:54.411175+0000","last_fullsized":"2023-10-16T15:54:54.411175+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:38.843193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383843+0000","last_change":"2023-10-16T15:54:39.203357+0000","last_active":"2023-10-16T15:54:53.383843+0000","last_peered":"2023-10-16T15:54:53.383843+0000","last_clean":"2023-10-16T15:54:53.383843+0000","last_became_active":"2023-10-16T15:54:39.203058+0000","last_became_peered":"2023-10-16T15:54:39.203058+0000","last_unstale":"2023-10-16T15:54:53.383843+0000","last_undegraded":"2023-10-16T15:54:53.383843+0000","last_fullsized":"2023-10-16T15:54:53.383843+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:37.020836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386249+0000","last_change":"2023-10-16T15:54:38.195221+0000","last_active":"2023-10-16T15:54:53.386249+0000","last_peered":"2023-10-16T15:54:53.386249+0000","last_clean":"2023-10-16T15:54:53.386249+0000","last_became_active":"2023-10-16T15:54:38.194937+0000","last_became_peered":"2023-10-16T15:54:38.194937+0000","last_unstale":"2023-10-16T15:54:53.386249+0000","last_undegraded":"2023-10-16T15:54:53.386249+0000","last_fullsized":"2023-10-16T15:54:53.386249+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:41.726203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412524+0000","last_change":"2023-10-16T15:54:53.402565+0000","last_active":"2023-10-16T15:54:54.412524+0000","last_peered":"2023-10-16T15:54:54.412524+0000","last_clean":"2023-10-16T15:54:54.412524+0000","last_became_active":"2023-10-16T15:54:53.400842+0000","last_became_peered":"2023-10-16T15:54:53.400842+0000","last_unstale":"2023-10-16T15:54:54.412524+0000","last_undegraded":"2023-10-16T15:54:54.412524+0000","last_fullsized":"2023-10-16T15:54:54.412524+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:14.468066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384473+0000","last_change":"2023-10-16T15:54:39.207259+0000","last_active":"2023-10-16T15:54:53.384473+0000","last_peered":"2023-10-16T15:54:53.384473+0000","last_clean":"2023-10-16T15:54:53.384473+0000","last_became_active":"2023-10-16T15:54:39.206912+0000","last_became_peered":"2023-10-16T15:54:39.206912+0000","last_unstale":"2023-10-16T15:54:53.384473+0000","last_undegraded":"2023-10-16T15:54:53.384473+0000","last_fullsized":"2023-10-16T15:54:53.384473+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:53.691146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386481+0000","last_change":"2023-10-16T15:54:38.200919+0000","last_active":"2023-10-16T15:54:53.386481+0000","last_peered":"2023-10-16T15:54:53.386481+0000","last_clean":"2023-10-16T15:54:53.386481+0000","last_became_active":"2023-10-16T15:54:38.200141+0000","last_became_peered":"2023-10-16T15:54:38.200141+0000","last_unstale":"2023-10-16T15:54:53.386481+0000","last_undegraded":"2023-10-16T15:54:53.386481+0000","last_fullsized":"2023-10-16T15:54:53.386481+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:40.432210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531052+0000","last_change":"2023-10-16T15:54:53.411598+0000","last_active":"2023-10-16T15:54:54.531052+0000","last_peered":"2023-10-16T15:54:54.531052+0000","last_clean":"2023-10-16T15:54:54.531052+0000","last_became_active":"2023-10-16T15:54:53.411267+0000","last_became_peered":"2023-10-16T15:54:53.411267+0000","last_unstale":"2023-10-16T15:54:54.531052+0000","last_undegraded":"2023-10-16T15:54:54.531052+0000","last_fullsized":"2023-10-16T15:54:54.531052+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:47.129484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.399705+0000","last_change":"2023-10-16T15:54:39.208421+0000","last_active":"2023-10-16T15:54:53.399705+0000","last_peered":"2023-10-16T15:54:53.399705+0000","last_clean":"2023-10-16T15:54:53.399705+0000","last_became_active":"2023-10-16T15:54:39.207372+0000","last_became_peered":"2023-10-16T15:54:39.207372+0000","last_unstale":"2023-10-16T15:54:53.399705+0000","last_undegraded":"2023-10-16T15:54:53.399705+0000","last_fullsized":"2023-10-16T15:54:53.399705+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:11:16.655726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"245'4","reported_seq":20,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.388161+0000","last_change":"2023-10-16T15:54:38.200929+0000","last_active":"2023-10-16T15:54:53.388161+0000","last_peered":"2023-10-16T15:54:53.388161+0000","last_clean":"2023-10-16T15:54:53.388161+0000","last_became_active":"2023-10-16T15:54:38.200195+0000","last_became_peered":"2023-10-16T15:54:38.200195+0000","last_unstale":"2023-10-16T15:54:53.388161+0000","last_undegraded":"2023-10-16T15:54:53.388161+0000","last_fullsized":"2023-10-16T15:54:53.388161+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:20.533246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"231'373","reported_seq":1350,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.372019+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:54:53.372019+0000","last_peered":"2023-10-16T15:54:53.372019+0000","last_clean":"2023-10-16T15:54:53.372019+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:54:53.372019+0000","last_undegraded":"2023-10-16T15:54:53.372019+0000","last_fullsized":"2023-10-16T15:54:53.372019+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576019,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65376,"kb_used_data":3660,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306464,"statfs":{"total":96636764160,"available":96569819136,"internally_reserved":0,"allocated":3747840,"data_stored":2145368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.095,"15min":1.171},"min":{"1min":0.61499999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.6779999999999999,"5min":1.794,"15min":1.8879999999999999},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.1040000000000001,"15min":1.149},"min":{"1min":0.71899999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6299999999999999,"5min":1.796,"15min":1.796},"last":1.0409999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1080000000000001,"15min":1.1679999999999999},"min":{"1min":0.82999999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4610000000000001,"5min":1.76,"15min":1.8580000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.244,"15min":1.3460000000000001},"min":{"1min":0.69199999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.365}]},{"osd":2,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3100000000000001,"15min":1.331},"min":{"1min":0.80400000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7410000000000001,"5min":1.893,"15min":2.1269999999999998},"last":1.49},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2310000000000001,"15min":1.296},"min":{"1min":0.877,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6020000000000001,"5min":1.909,"15min":1.95},"last":1.419}]},{"osd":3,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.2529999999999999,"15min":1.3260000000000001},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5149999999999999,"5min":1.843,"15min":1.869},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.3999999999999999,"15min":1.4450000000000001},"min":{"1min":1.089,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.7490000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3819999999999999,"15min":1.383},"min":{"1min":1.054,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6479999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3089999999999999,"15min":1.339},"min":{"1min":0.88100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6439999999999999,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.304}]},{"osd":5,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4570000000000001,"15min":1.46},"min":{"1min":0.87,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.7889999999999999,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.502,"15min":1.5489999999999999},"min":{"1min":0.98599999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.04,"5min":2.04,"15min":2.3540000000000001},"last":1.6379999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.4550000000000001,"15min":1.4750000000000001},"min":{"1min":1.2390000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8260000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.179},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5669999999999999,"15min":1.5800000000000001},"min":{"1min":1.048,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":1.988,"15min":2.2599999999999998},"last":1.6930000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576018,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48160,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323680,"statfs":{"total":96636764160,"available":96587448320,"internally_reserved":0,"allocated":3162112,"data_stored":1555576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.25,"15min":1.214},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.258,"15min":1.2609999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70999999999999996},"max":{"1min":1.5669999999999999,"5min":1.905,"15min":1.905},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3140000000000001,"15min":1.296},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.72199999999999998},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.3049999999999999,"15min":1.3200000000000001},"min":{"1min":0.747,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.859,"15min":1.8740000000000001},"last":1.093}]},{"osd":2,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2330000000000001,"15min":1.238},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.831,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.86299999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.905,"5min":1.905,"15min":2.0339999999999998},"last":1.365}]},{"osd":3,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3680000000000001,"15min":1.3919999999999999},"min":{"1min":0.93000000000000005,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.214},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3149999999999999,"15min":1.3340000000000001},"min":{"1min":0.91000000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.2609999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.409,"15min":1.337},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3359999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.488},{"interface":"front","average":{"1min":1.518,"5min":1.454,"15min":1.4059999999999999},"min":{"1min":1.2,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.573}]},{"osd":5,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.369,"15min":1.3480000000000001},"min":{"1min":1.022,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":2.016},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6020000000000001,"15min":1.534},"min":{"1min":1.194,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.6859999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.635,"15min":1.599},"min":{"1min":1.274,"5min":0.627,"15min":0.627},"max":{"1min":2.0110000000000001,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.667,"5min":1.6319999999999999,"15min":1.595},"min":{"1min":1.2849999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.028,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.544}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30096,"kb_used_data":3072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341744,"statfs":{"total":96636764160,"available":96605945856,"internally_reserved":0,"allocated":3145728,"data_stored":1554860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.111,"15min":1.1579999999999999},"min":{"1min":0.68899999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.802,"5min":1.8069999999999999,"15min":1.821},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.234,"15min":1.24},"min":{"1min":0.79300000000000004,"5min":0.622,"15min":0.622},"max":{"1min":1.766,"5min":2.738,"15min":2.738},"last":0.86699999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.196,"15min":1.228},"min":{"1min":0.66600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6639999999999999,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.2410000000000001,"15min":1.2889999999999999},"min":{"1min":0.65300000000000002,"5min":0.443,"15min":0.443},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.0580000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.248},"min":{"1min":0.61799999999999999,"5min":0.505,"15min":0.505},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.999},{"interface":"front","average":{"1min":1.325,"5min":1.375,"15min":1.3939999999999999},"min":{"1min":0.77500000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.2160000000000002},"last":1.466}]},{"osd":3,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3300000000000001,"15min":1.335},"min":{"1min":0.753,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8720000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.3260000000000001,"15min":1.3740000000000001},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.966,"5min":1.966,"15min":2.1589999999999998},"last":1.498}]},{"osd":4,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.347},"min":{"1min":0.79900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.8500000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5049999999999999,"15min":1.5149999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0800000000000001,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.554}]},{"osd":6,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4630000000000001,"15min":1.385},"min":{"1min":0.81000000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0369999999999999,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.302},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.468,"15min":1.4319999999999999},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9870000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.2390000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.5289999999999999,"15min":1.532},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.125,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.651,"15min":1.631},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.4340000000000002,"5min":3.109,"15min":3.109},"last":1.573}]}]},{"osd":4,"up_from":13,"seq":55834576019,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70396,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301444,"statfs":{"total":96636764160,"available":96564678656,"internally_reserved":0,"allocated":3186688,"data_stored":1559477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.1419999999999999,"15min":1.194},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.159,"15min":1.194},"min":{"1min":0.76300000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.4059999999999999,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1890000000000001,"15min":1.2310000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.629},"max":{"1min":1.6579999999999999,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.254,"5min":1.284,"15min":1.3089999999999999},"min":{"1min":0.75900000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9750000000000001,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.51}]},{"osd":2,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3939999999999999,"15min":1.4510000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7150000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":2.0899999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.456,"15min":1.492},"min":{"1min":0.82499999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6819999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.667}]},{"osd":3,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.391,"15min":1.385},"min":{"1min":1.02,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8129999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.341,"15min":1.383},"min":{"1min":0.92100000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.7509999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.369}]},{"osd":5,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.371,"15min":1.341},"min":{"1min":0.80200000000000005,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.849,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.944},{"interface":"front","average":{"1min":1.417,"5min":1.4730000000000001,"15min":1.4410000000000001},"min":{"1min":0.88900000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7949999999999999,"5min":2.161,"15min":2.4489999999999998},"last":1.4179999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.389,"15min":1.405},"min":{"1min":0.97699999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5629999999999999,"15min":1.526},"min":{"1min":1.1990000000000001,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.246,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":2.1920000000000002}]},{"osd":7,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6100000000000001,"15min":1.583},"min":{"1min":1.04,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.9730000000000001,"5min":2.399,"15min":2.399},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.6000000000000001,"15min":1.601},"min":{"1min":1.129,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.839}]}]},{"osd":3,"up_from":13,"seq":55834576018,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56488,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315352,"statfs":{"total":96636764160,"available":96578920448,"internally_reserved":0,"allocated":3166208,"data_stored":1557636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92100000000000004,"15min":0.96799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.419,"5min":1.48,"15min":2.0430000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0189999999999999,"15min":1.0580000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.722,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.79900000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94199999999999995,"15min":1.0189999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.776,"5min":1.776,"15min":2.589},"last":0.751},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.147,"15min":1.204},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.556,"5min":1.748,"15min":2.5569999999999999},"last":0.86299999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.304,"15min":1.403},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.302},"last":1.224},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3919999999999999,"15min":1.488},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.954,"5min":1.954,"15min":2.754},"last":1.409}]},{"osd":4,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2669999999999999,"15min":1.3440000000000001},"min":{"1min":0.79100000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9770000000000001,"5min":2.0110000000000001,"15min":2.512},"last":1.528},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3440000000000001,"15min":1.4119999999999999},"min":{"1min":0.76300000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.0249999999999999,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.1220000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.53,"15min":1.5620000000000001},"min":{"1min":0.94099999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2280000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.597},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4890000000000001,"15min":1.5629999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.2599999999999998,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.3080000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.46,"15min":1.5},"min":{"1min":0.97499999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.0680000000000001,"5min":2.173,"15min":2.2090000000000001},"last":1.78},{"interface":"front","average":{"1min":1.585,"5min":1.5649999999999999,"15min":1.6000000000000001},"min":{"1min":0.85299999999999998,"5min":0.749,"15min":0.749},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.7669999999999999},"last":1.389}]},{"osd":7,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.591,"15min":1.6779999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":1.357},{"interface":"front","average":{"1min":1.571,"5min":1.579,"15min":1.6339999999999999},"min":{"1min":1.002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.133,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.6839999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576019,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56472,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315368,"statfs":{"total":96636764160,"available":96578936832,"internally_reserved":0,"allocated":3149824,"data_stored":1554950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.89300000000000002,"15min":0.93700000000000006},"min":{"1min":0.443,"5min":0.378,"15min":0.378},"max":{"1min":1.2749999999999999,"5min":1.726,"15min":1.76},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.093},"min":{"1min":0.39300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7789999999999999,"5min":1.804,"15min":1.8999999999999999},"last":1.1599999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0209999999999999,"15min":1.046},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.504,"5min":1.974,"15min":1.974},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.1479999999999999,"15min":1.224},"min":{"1min":0.55000000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.772,"5min":1.879,"15min":2.0779999999999998},"last":1.4990000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.161,"15min":1.2709999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.5289999999999999,"5min":1.704,"15min":2.1509999999999998},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.3180000000000001,"15min":1.365},"min":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.661,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.3560000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.448,"15min":1.4730000000000001},"min":{"1min":0.84499999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.9299999999999999,"5min":2.335,"15min":2.335},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.345,"15min":1.411},"min":{"1min":0.65900000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.901,"5min":1.915,"15min":2.1469999999999998},"last":1.304}]},{"osd":5,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.403,"15min":1.4730000000000001},"min":{"1min":0.72899999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8080000000000001,"5min":2.657,"15min":2.657},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.5449999999999999,"15min":1.6100000000000001},"min":{"1min":0.81499999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.093,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.6020000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4570000000000001,"15min":1.5429999999999999},"min":{"1min":0.77900000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.861,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5589999999999999,"15min":1.6100000000000001},"min":{"1min":0.89300000000000002,"5min":0.753,"15min":0.753},"max":{"1min":1.8500000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.2569999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.5649999999999999,"15min":1.6499999999999999},"min":{"1min":0.85999999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.234,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.548},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.6499999999999999,"15min":1.722},"min":{"1min":1.0129999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.1890000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.4650000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576019,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48176,"kb_used_data":3108,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323664,"statfs":{"total":96636764160,"available":96587431936,"internally_reserved":0,"allocated":3182592,"data_stored":1558617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94399999999999995,"15min":0.94899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.452,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0429999999999999,"15min":1.1299999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6619999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98699999999999999,"15min":1.0349999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.4179999999999999,"5min":1.6299999999999999,"15min":2.286},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.109,"5min":1.1919999999999999,"15min":1.2270000000000001},"min":{"1min":0.59199999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.657,"5min":1.7929999999999999,"15min":1.853},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3220000000000001,"15min":1.3680000000000001},"min":{"1min":0.69699999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7290000000000001,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.101,"5min":1.2529999999999999,"15min":1.3129999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8859999999999999,"5min":2.1629999999999998,"15min":2.7839999999999998},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3620000000000001,"15min":1.381},"min":{"1min":0.79300000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.2959999999999998,"5min":2.327,"15min":2.3809999999999998},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.409,"5min":1.512,"15min":1.524},"min":{"1min":0.92000000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.081,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.1279999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.49,"15min":1.5269999999999999},"min":{"1min":0.72499999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0539999999999998,"5min":2.456,"15min":2.456},"last":1.042},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4930000000000001,"15min":1.502},"min":{"1min":1.0069999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.093,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.647,"15min":1.655},"min":{"1min":0.97299999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.218,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.542,"5min":1.6160000000000001,"15min":1.643},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.5059999999999998,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.7030000000000001,"15min":1.744},"min":{"1min":0.80300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.621,"5min":2.734,"15min":3.2090000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.577,"5min":1.647,"15min":1.655},"min":{"1min":0.90100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.452,"5min":2.452,"15min":3.1539999999999999},"last":1.214}]}]},{"osd":0,"up_from":13,"seq":55834576019,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48728,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323112,"statfs":{"total":96636764160,"available":96586866688,"internally_reserved":0,"allocated":3743744,"data_stored":2145262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91000000000000003,"15min":0.94999999999999996},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.2789999999999999,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1180000000000001,"15min":1.2070000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.5800000000000001,"5min":2.036,"15min":2.036},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.1739999999999999,"15min":1.26},"min":{"1min":0.56499999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.117,"15min":1.2170000000000001},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.462,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.107}]},{"osd":3,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.004,"15min":1.032},"min":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6339999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.032,"5min":1.175,"15min":1.2549999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.774,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.319,"15min":1.337},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":2.137},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.401,"15min":1.4970000000000001},"min":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.9390000000000001,"5min":1.99,"15min":2.452},"last":1.4990000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.4239999999999999,"15min":1.476},"min":{"1min":0.66200000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8779999999999999,"5min":2.137,"15min":2.137},"last":1.548},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.345,"15min":1.411},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":2.0859999999999999,"15min":2.282},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5349999999999999,"15min":1.6259999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.7010000000000001},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.526,"15min":1.569},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.1869999999999998,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4279999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.621,"15min":1.6779999999999999},"min":{"1min":0.82199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1480000000000001,"5min":2.339,"15min":2.339},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.5760000000000001,"15min":1.601},"min":{"1min":0.89200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.1259999999999999,"5min":2.1739999999999999,"15min":2.274},"last":1.76}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:54:57.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:54:58.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.222+0000 7f4a315f4640 1 Processor -- start 2023-10-16T15:54:58.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.223+0000 7f4a315f4640 1 -- start start 2023-10-16T15:54:58.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.224+0000 7f4a315f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 0x7f4a2c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.224+0000 7f4a315f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0acd70 con 0x7f4a2c0ce280 2023-10-16T15:54:58.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.224+0000 7f4a315f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0ac4a0 con 0x7f4a2c065ab0 2023-10-16T15:54:58.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.224+0000 7f4a315f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c065f40 con 0x7f4a2c06d220 2023-10-16T15:54:58.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.225+0000 7f4a30df3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4a2c0ce280 0x7f4a2c0ad200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33284/0 (socket says 172.21.15.29:33284) 2023-10-16T15:54:58.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.225+0000 7f4a30df3640 1 -- 172.21.15.29:0/3118688674 learned_addr learned my addr 172.21.15.29:0/3118688674 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.225+0000 7f4a2bfff640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 0x7f4a2c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.225+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680671332 0 0) 0x7f4a2c0acd70 con 0x7f4a2c0ce280 2023-10-16T15:54:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.225+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a18003520 con 0x7f4a2c0ce280 2023-10-16T15:54:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157428644 0 0) 0x7f4a18003520 con 0x7f4a2c0ce280 2023-10-16T15:54:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 msgr2=0x7f4a2c0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 0x7f4a2c0c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 >> v1:172.21.15.170:6789/0 conn(0x7f4a2c065ab0 legacy=0x7f4a2c0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0ac720 con 0x7f4a2c0ce280 2023-10-16T15:54:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4a240019f0 con 0x7f4a2c0ce280 2023-10-16T15:54:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 >> v1:172.21.15.29:6789/0 conn(0x7f4a2c0ce280 legacy=0x7f4a2c0ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 0x7f4a180039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a18003ee0 0x7f4a180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a18004d00 0x7f4a18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c065f40 con 0x7f4a18003ee0 2023-10-16T15:54:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1800ba30 con 0x7f4a18004d00 2023-10-16T15:54:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1800bbf0 con 0x7f4a2c06d220 2023-10-16T15:54:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.226+0000 7f4a23fff640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a18003ee0 0x7f4a180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a30df3640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a18004d00 0x7f4a18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a2bfff640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 0x7f4a180039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a23fff640 1 -- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 msgr2=0x7f4a180039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a23fff640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a2c06d220 0x7f4a180039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a23fff640 1 -- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a18004d00 msgr2=0x7f4a18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a23fff640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a18004d00 0x7f4a18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.227+0000 7f4a23fff640 1 -- 172.21.15.29:0/3118688674 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18003520 con 0x7f4a18003ee0 2023-10-16T15:54:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.228+0000 7f4a23fff640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a18003ee0 0x7f4a180047c0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f4a14001860 tx=0x7f4a180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0d15de41fa42d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.228+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14033410 con 0x7f4a18003ee0 2023-10-16T15:54:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.228+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14033590 con 0x7f4a18003ee0 2023-10-16T15:54:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.228+0000 7f4a2b7fe640 1 -- 172.21.15.29:0/3118688674 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14033940 con 0x7f4a18003ee0 2023-10-16T15:54:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a315f4640 1 -- 172.21.15.29:0/3118688674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a18003ee0 msgr2=0x7f4a180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a315f4640 1 --2- 172.21.15.29:0/3118688674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a18003ee0 0x7f4a180047c0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f4a14001860 tx=0x7f4a180057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a30df3640 1 -- 172.21.15.29:0/3118688674 reap_dead start 2023-10-16T15:54:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a315f4640 1 -- 172.21.15.29:0/3118688674 shutdown_connections 2023-10-16T15:54:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a315f4640 1 -- 172.21.15.29:0/3118688674 >> 172.21.15.29:0/3118688674 conn(0x7f4a2c0b0a80 msgr2=0x7f4a2c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a315f4640 1 -- 172.21.15.29:0/3118688674 shutdown_connections 2023-10-16T15:54:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.229+0000 7f4a315f4640 1 -- 172.21.15.29:0/3118688674 wait complete. 2023-10-16T15:54:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.231+0000 7f4a315f4640 1 Processor -- start 2023-10-16T15:54:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.231+0000 7f4a315f4640 1 -- start start 2023-10-16T15:54:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a315f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a18003ee0 0x7f4a2c1f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a315f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c09aa50 0x7f4a2c06c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a315f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 0x7f4a2c0b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a315f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0b07e0 con 0x7f4a2c0afe20 2023-10-16T15:54:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a315f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c09bf90 con 0x7f4a2c09aa50 2023-10-16T15:54:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a315f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c09bb00 con 0x7f4a18003ee0 2023-10-16T15:54:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a30df3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 0x7f4a2c0b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a30df3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 0x7f4a2c0b02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36366/0 (socket says 172.21.15.29:36366) 2023-10-16T15:54:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.232+0000 7f4a30df3640 1 -- 172.21.15.29:0/68960869 learned_addr learned my addr 172.21.15.29:0/68960869 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a2bfff640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a18003ee0 0x7f4a2c1f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a23fff640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c09aa50 0x7f4a2c06c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a30df3640 1 -- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a18003ee0 msgr2=0x7f4a2c1f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a30df3640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a18003ee0 0x7f4a2c1f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a30df3640 1 -- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c09aa50 msgr2=0x7f4a2c06c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a30df3640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c09aa50 0x7f4a2c06c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a30df3640 1 -- 172.21.15.29:0/68960869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c1f2400 con 0x7f4a2c0afe20 2023-10-16T15:54:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a30df3640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 0x7f4a2c0b02a0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c067210 tx=0x7f4a2400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.233+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24007ae0 con 0x7f4a2c0afe20 2023-10-16T15:54:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.234+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2400ee50 con 0x7f4a2c0afe20 2023-10-16T15:54:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.234+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24014670 con 0x7f4a2c0afe20 2023-10-16T15:54:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.234+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c09aef0 con 0x7f4a2c0afe20 2023-10-16T15:54:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.234+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c09b4b0 con 0x7f4a2c0afe20 2023-10-16T15:54:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.235+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c06d000 con 0x7f4a2c0afe20 2023-10-16T15:54:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.236+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4a240147d0 con 0x7f4a2c0afe20 2023-10-16T15:54:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.237+0000 7f4a297fa640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4a0c06b0e0 0x7f4a0c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.237+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f4a24089bf0 con 0x7f4a2c0afe20 2023-10-16T15:54:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.238+0000 7f4a2bfff640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4a0c06b0e0 0x7f4a0c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.239+0000 7f4a2bfff640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4a0c06b0e0 0x7f4a0c06d5a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009a10 tx=0x7f4a1c008040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:58.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.242+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a24054520 con 0x7f4a2c0afe20 2023-10-16T15:54:58.272 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-16T15:54:58.271+0000 7fa11ade4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:54:58.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.443+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 --> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a2c0aca00 con 0x7f4a0c06b0e0 2023-10-16T15:54:58.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.485+0000 7f4a297fa640 1 -- 172.21.15.29:0/68960869 <== mgr.9022 v2:172.21.15.170:6832/58791103 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526154 (secure 0 0 0) 0x7f4a1c0128f0 con 0x7f4a0c06b0e0 2023-10-16T15:54:58.493 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:58.496 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:54:58.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4a0c06b0e0 msgr2=0x7f4a0c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4a0c06b0e0 0x7f4a0c06d5a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009a10 tx=0x7f4a1c008040 comp rx=0 tx=0).stop 2023-10-16T15:54:58.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 msgr2=0x7f4a2c0b02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 0x7f4a2c0b02a0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c067210 tx=0x7f4a2400e530 comp rx=0 tx=0).stop 2023-10-16T15:54:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 shutdown_connections 2023-10-16T15:54:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f4a0c06b0e0 0x7f4a0c06d5a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2c0afe20 0x7f4a2c0b02a0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4a2c09aa50 0x7f4a2c06c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 --2- 172.21.15.29:0/68960869 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4a18003ee0 0x7f4a2c1f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 >> 172.21.15.29:0/68960869 conn(0x7f4a2c0b0a80 msgr2=0x7f4a2c0ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:58.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.492+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 shutdown_connections 2023-10-16T15:54:58.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.493+0000 7f4a315f4640 1 -- 172.21.15.29:0/68960869 wait complete. 2023-10-16T15:54:58.513 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2023-10-16T15:54:57.483647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":714,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":313,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423892,"kb_used_data":25864,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754550828,"statfs":{"total":773094113280,"available":772660047872,"internally_reserved":0,"allocated":26484736,"data_stored":13631746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":407424569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002641"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531731+0000","last_change":"2023-10-16T15:54:53.408335+0000","last_active":"2023-10-16T15:54:54.531731+0000","last_peered":"2023-10-16T15:54:54.531731+0000","last_clean":"2023-10-16T15:54:54.531731+0000","last_became_active":"2023-10-16T15:54:53.407999+0000","last_became_peered":"2023-10-16T15:54:53.407999+0000","last_unstale":"2023-10-16T15:54:54.531731+0000","last_undegraded":"2023-10-16T15:54:54.531731+0000","last_fullsized":"2023-10-16T15:54:54.531731+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:00.091345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530314+0000","last_change":"2023-10-16T15:54:52.399702+0000","last_active":"2023-10-16T15:54:54.530314+0000","last_peered":"2023-10-16T15:54:54.530314+0000","last_clean":"2023-10-16T15:54:54.530314+0000","last_became_active":"2023-10-16T15:54:52.398980+0000","last_became_peered":"2023-10-16T15:54:52.398980+0000","last_unstale":"2023-10-16T15:54:54.530314+0000","last_undegraded":"2023-10-16T15:54:54.530314+0000","last_fullsized":"2023-10-16T15:54:54.530314+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:19:37.756838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.392325+0000","last_change":"2023-10-16T15:54:39.213468+0000","last_active":"2023-10-16T15:54:53.392325+0000","last_peered":"2023-10-16T15:54:53.392325+0000","last_clean":"2023-10-16T15:54:53.392325+0000","last_became_active":"2023-10-16T15:54:39.213201+0000","last_became_peered":"2023-10-16T15:54:39.213201+0000","last_unstale":"2023-10-16T15:54:53.392325+0000","last_undegraded":"2023-10-16T15:54:53.392325+0000","last_fullsized":"2023-10-16T15:54:53.392325+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:54.453092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530716+0000","last_change":"2023-10-16T15:54:53.412071+0000","last_active":"2023-10-16T15:54:54.530716+0000","last_peered":"2023-10-16T15:54:54.530716+0000","last_clean":"2023-10-16T15:54:54.530716+0000","last_became_active":"2023-10-16T15:54:53.411773+0000","last_became_peered":"2023-10-16T15:54:53.411773+0000","last_unstale":"2023-10-16T15:54:54.530716+0000","last_undegraded":"2023-10-16T15:54:54.530716+0000","last_fullsized":"2023-10-16T15:54:54.530716+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:36.633161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531701+0000","last_change":"2023-10-16T15:54:52.392349+0000","last_active":"2023-10-16T15:54:54.531701+0000","last_peered":"2023-10-16T15:54:54.531701+0000","last_clean":"2023-10-16T15:54:54.531701+0000","last_became_active":"2023-10-16T15:54:52.390500+0000","last_became_peered":"2023-10-16T15:54:52.390500+0000","last_unstale":"2023-10-16T15:54:54.531701+0000","last_undegraded":"2023-10-16T15:54:54.531701+0000","last_fullsized":"2023-10-16T15:54:54.531701+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:14.574852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391480+0000","last_change":"2023-10-16T15:54:39.209387+0000","last_active":"2023-10-16T15:54:53.391480+0000","last_peered":"2023-10-16T15:54:53.391480+0000","last_clean":"2023-10-16T15:54:53.391480+0000","last_became_active":"2023-10-16T15:54:39.207929+0000","last_became_peered":"2023-10-16T15:54:39.207929+0000","last_unstale":"2023-10-16T15:54:53.391480+0000","last_undegraded":"2023-10-16T15:54:53.391480+0000","last_fullsized":"2023-10-16T15:54:53.391480+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:21.226148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531576+0000","last_change":"2023-10-16T15:54:53.407832+0000","last_active":"2023-10-16T15:54:54.531576+0000","last_peered":"2023-10-16T15:54:54.531576+0000","last_clean":"2023-10-16T15:54:54.531576+0000","last_became_active":"2023-10-16T15:54:53.407351+0000","last_became_peered":"2023-10-16T15:54:53.407351+0000","last_unstale":"2023-10-16T15:54:54.531576+0000","last_undegraded":"2023-10-16T15:54:54.531576+0000","last_fullsized":"2023-10-16T15:54:54.531576+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:48.011756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405437+0000","last_change":"2023-10-16T15:54:52.400989+0000","last_active":"2023-10-16T15:54:54.405437+0000","last_peered":"2023-10-16T15:54:54.405437+0000","last_clean":"2023-10-16T15:54:54.405437+0000","last_became_active":"2023-10-16T15:54:52.399858+0000","last_became_peered":"2023-10-16T15:54:52.399858+0000","last_unstale":"2023-10-16T15:54:54.405437+0000","last_undegraded":"2023-10-16T15:54:54.405437+0000","last_fullsized":"2023-10-16T15:54:54.405437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:59.813150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391667+0000","last_change":"2023-10-16T15:54:39.205624+0000","last_active":"2023-10-16T15:54:53.391667+0000","last_peered":"2023-10-16T15:54:53.391667+0000","last_clean":"2023-10-16T15:54:53.391667+0000","last_became_active":"2023-10-16T15:54:39.205294+0000","last_became_peered":"2023-10-16T15:54:39.205294+0000","last_unstale":"2023-10-16T15:54:53.391667+0000","last_undegraded":"2023-10-16T15:54:53.391667+0000","last_fullsized":"2023-10-16T15:54:53.391667+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:10.276795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530730+0000","last_change":"2023-10-16T15:54:53.410389+0000","last_active":"2023-10-16T15:54:54.530730+0000","last_peered":"2023-10-16T15:54:54.530730+0000","last_clean":"2023-10-16T15:54:54.530730+0000","last_became_active":"2023-10-16T15:54:53.410085+0000","last_became_peered":"2023-10-16T15:54:53.410085+0000","last_unstale":"2023-10-16T15:54:54.530730+0000","last_undegraded":"2023-10-16T15:54:54.530730+0000","last_fullsized":"2023-10-16T15:54:54.530730+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:08.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405841+0000","last_change":"2023-10-16T15:54:52.394074+0000","last_active":"2023-10-16T15:54:54.405841+0000","last_peered":"2023-10-16T15:54:54.405841+0000","last_clean":"2023-10-16T15:54:54.405841+0000","last_became_active":"2023-10-16T15:54:52.393784+0000","last_became_peered":"2023-10-16T15:54:52.393784+0000","last_unstale":"2023-10-16T15:54:54.405841+0000","last_undegraded":"2023-10-16T15:54:54.405841+0000","last_fullsized":"2023-10-16T15:54:54.405841+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:40.812320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.401295+0000","last_change":"2023-10-16T15:54:39.203634+0000","last_active":"2023-10-16T15:54:53.401295+0000","last_peered":"2023-10-16T15:54:53.401295+0000","last_clean":"2023-10-16T15:54:53.401295+0000","last_became_active":"2023-10-16T15:54:39.203333+0000","last_became_peered":"2023-10-16T15:54:39.203333+0000","last_unstale":"2023-10-16T15:54:53.401295+0000","last_undegraded":"2023-10-16T15:54:53.401295+0000","last_fullsized":"2023-10-16T15:54:53.401295+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533691+0000","last_change":"2023-10-16T15:54:53.405288+0000","last_active":"2023-10-16T15:54:54.533691+0000","last_peered":"2023-10-16T15:54:54.533691+0000","last_clean":"2023-10-16T15:54:54.533691+0000","last_became_active":"2023-10-16T15:54:53.403686+0000","last_became_peered":"2023-10-16T15:54:53.403686+0000","last_unstale":"2023-10-16T15:54:54.533691+0000","last_undegraded":"2023-10-16T15:54:54.533691+0000","last_fullsized":"2023-10-16T15:54:54.533691+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:54.761128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532016+0000","last_change":"2023-10-16T15:54:52.396757+0000","last_active":"2023-10-16T15:54:54.532016+0000","last_peered":"2023-10-16T15:54:54.532016+0000","last_clean":"2023-10-16T15:54:54.532016+0000","last_became_active":"2023-10-16T15:54:52.396104+0000","last_became_peered":"2023-10-16T15:54:52.396104+0000","last_unstale":"2023-10-16T15:54:54.532016+0000","last_undegraded":"2023-10-16T15:54:54.532016+0000","last_fullsized":"2023-10-16T15:54:54.532016+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:01.735363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387719+0000","last_change":"2023-10-16T15:54:39.205871+0000","last_active":"2023-10-16T15:54:53.387719+0000","last_peered":"2023-10-16T15:54:53.387719+0000","last_clean":"2023-10-16T15:54:53.387719+0000","last_became_active":"2023-10-16T15:54:39.204811+0000","last_became_peered":"2023-10-16T15:54:39.204811+0000","last_unstale":"2023-10-16T15:54:53.387719+0000","last_undegraded":"2023-10-16T15:54:53.387719+0000","last_fullsized":"2023-10-16T15:54:53.387719+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:59.938902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533031+0000","last_change":"2023-10-16T15:54:53.405860+0000","last_active":"2023-10-16T15:54:54.533031+0000","last_peered":"2023-10-16T15:54:54.533031+0000","last_clean":"2023-10-16T15:54:54.533031+0000","last_became_active":"2023-10-16T15:54:53.405398+0000","last_became_peered":"2023-10-16T15:54:53.405398+0000","last_unstale":"2023-10-16T15:54:54.533031+0000","last_undegraded":"2023-10-16T15:54:54.533031+0000","last_fullsized":"2023-10-16T15:54:54.533031+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:00.830634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.534693+0000","last_change":"2023-10-16T15:54:52.396047+0000","last_active":"2023-10-16T15:54:54.534693+0000","last_peered":"2023-10-16T15:54:54.534693+0000","last_clean":"2023-10-16T15:54:54.534693+0000","last_became_active":"2023-10-16T15:54:52.395716+0000","last_became_peered":"2023-10-16T15:54:52.395716+0000","last_unstale":"2023-10-16T15:54:54.534693+0000","last_undegraded":"2023-10-16T15:54:54.534693+0000","last_fullsized":"2023-10-16T15:54:54.534693+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:58.389442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384037+0000","last_change":"2023-10-16T15:54:39.198994+0000","last_active":"2023-10-16T15:54:53.384037+0000","last_peered":"2023-10-16T15:54:53.384037+0000","last_clean":"2023-10-16T15:54:53.384037+0000","last_became_active":"2023-10-16T15:54:39.198650+0000","last_became_peered":"2023-10-16T15:54:39.198650+0000","last_unstale":"2023-10-16T15:54:53.384037+0000","last_undegraded":"2023-10-16T15:54:53.384037+0000","last_fullsized":"2023-10-16T15:54:53.384037+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530813+0000","last_change":"2023-10-16T15:54:53.410527+0000","last_active":"2023-10-16T15:54:54.530813+0000","last_peered":"2023-10-16T15:54:54.530813+0000","last_clean":"2023-10-16T15:54:54.530813+0000","last_became_active":"2023-10-16T15:54:53.410214+0000","last_became_peered":"2023-10-16T15:54:53.410214+0000","last_unstale":"2023-10-16T15:54:54.530813+0000","last_undegraded":"2023-10-16T15:54:54.530813+0000","last_fullsized":"2023-10-16T15:54:54.530813+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:55.476370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533913+0000","last_change":"2023-10-16T15:54:52.395832+0000","last_active":"2023-10-16T15:54:54.533913+0000","last_peered":"2023-10-16T15:54:54.533913+0000","last_clean":"2023-10-16T15:54:54.533913+0000","last_became_active":"2023-10-16T15:54:52.394271+0000","last_became_peered":"2023-10-16T15:54:52.394271+0000","last_unstale":"2023-10-16T15:54:54.533913+0000","last_undegraded":"2023-10-16T15:54:54.533913+0000","last_fullsized":"2023-10-16T15:54:54.533913+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:03:11.728693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381116+0000","last_change":"2023-10-16T15:54:39.190398+0000","last_active":"2023-10-16T15:54:53.381116+0000","last_peered":"2023-10-16T15:54:53.381116+0000","last_clean":"2023-10-16T15:54:53.381116+0000","last_became_active":"2023-10-16T15:54:39.189834+0000","last_became_peered":"2023-10-16T15:54:39.189834+0000","last_unstale":"2023-10-16T15:54:53.381116+0000","last_undegraded":"2023-10-16T15:54:53.381116+0000","last_fullsized":"2023-10-16T15:54:53.381116+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532463+0000","last_change":"2023-10-16T15:54:53.407326+0000","last_active":"2023-10-16T15:54:54.532463+0000","last_peered":"2023-10-16T15:54:54.532463+0000","last_clean":"2023-10-16T15:54:54.532463+0000","last_became_active":"2023-10-16T15:54:53.406995+0000","last_became_peered":"2023-10-16T15:54:53.406995+0000","last_unstale":"2023-10-16T15:54:54.532463+0000","last_undegraded":"2023-10-16T15:54:54.532463+0000","last_fullsized":"2023-10-16T15:54:54.532463+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:06.856360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533071+0000","last_change":"2023-10-16T15:54:52.398585+0000","last_active":"2023-10-16T15:54:54.533071+0000","last_peered":"2023-10-16T15:54:54.533071+0000","last_clean":"2023-10-16T15:54:54.533071+0000","last_became_active":"2023-10-16T15:54:52.397274+0000","last_became_peered":"2023-10-16T15:54:52.397274+0000","last_unstale":"2023-10-16T15:54:54.533071+0000","last_undegraded":"2023-10-16T15:54:54.533071+0000","last_fullsized":"2023-10-16T15:54:54.533071+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:02.747703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383616+0000","last_change":"2023-10-16T15:54:39.203378+0000","last_active":"2023-10-16T15:54:53.383616+0000","last_peered":"2023-10-16T15:54:53.383616+0000","last_clean":"2023-10-16T15:54:53.383616+0000","last_became_active":"2023-10-16T15:54:39.202647+0000","last_became_peered":"2023-10-16T15:54:39.202647+0000","last_unstale":"2023-10-16T15:54:53.383616+0000","last_undegraded":"2023-10-16T15:54:53.383616+0000","last_fullsized":"2023-10-16T15:54:53.383616+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:49:25.249610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405400+0000","last_change":"2023-10-16T15:54:53.407942+0000","last_active":"2023-10-16T15:54:54.405400+0000","last_peered":"2023-10-16T15:54:54.405400+0000","last_clean":"2023-10-16T15:54:54.405400+0000","last_became_active":"2023-10-16T15:54:53.407656+0000","last_became_peered":"2023-10-16T15:54:53.407656+0000","last_unstale":"2023-10-16T15:54:54.405400+0000","last_undegraded":"2023-10-16T15:54:54.405400+0000","last_fullsized":"2023-10-16T15:54:54.405400+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:03.724354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.408329+0000","last_change":"2023-10-16T15:54:52.392722+0000","last_active":"2023-10-16T15:54:54.408329+0000","last_peered":"2023-10-16T15:54:54.408329+0000","last_clean":"2023-10-16T15:54:54.408329+0000","last_became_active":"2023-10-16T15:54:52.389599+0000","last_became_peered":"2023-10-16T15:54:52.389599+0000","last_unstale":"2023-10-16T15:54:54.408329+0000","last_undegraded":"2023-10-16T15:54:54.408329+0000","last_fullsized":"2023-10-16T15:54:54.408329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:57:51.500627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.374482+0000","last_change":"2023-10-16T15:54:39.190692+0000","last_active":"2023-10-16T15:54:53.374482+0000","last_peered":"2023-10-16T15:54:53.374482+0000","last_clean":"2023-10-16T15:54:53.374482+0000","last_became_active":"2023-10-16T15:54:39.189950+0000","last_became_peered":"2023-10-16T15:54:39.189950+0000","last_unstale":"2023-10-16T15:54:53.374482+0000","last_undegraded":"2023-10-16T15:54:53.374482+0000","last_fullsized":"2023-10-16T15:54:53.374482+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:05.334097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530093+0000","last_change":"2023-10-16T15:54:53.394815+0000","last_active":"2023-10-16T15:54:54.530093+0000","last_peered":"2023-10-16T15:54:54.530093+0000","last_clean":"2023-10-16T15:54:54.530093+0000","last_became_active":"2023-10-16T15:54:53.394474+0000","last_became_peered":"2023-10-16T15:54:53.394474+0000","last_unstale":"2023-10-16T15:54:54.530093+0000","last_undegraded":"2023-10-16T15:54:54.530093+0000","last_fullsized":"2023-10-16T15:54:54.530093+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:58.006262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.409356+0000","last_change":"2023-10-16T15:54:52.388176+0000","last_active":"2023-10-16T15:54:54.409356+0000","last_peered":"2023-10-16T15:54:54.409356+0000","last_clean":"2023-10-16T15:54:54.409356+0000","last_became_active":"2023-10-16T15:54:52.387732+0000","last_became_peered":"2023-10-16T15:54:52.387732+0000","last_unstale":"2023-10-16T15:54:54.409356+0000","last_undegraded":"2023-10-16T15:54:54.409356+0000","last_fullsized":"2023-10-16T15:54:54.409356+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:34.734679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390036+0000","last_change":"2023-10-16T15:54:39.203147+0000","last_active":"2023-10-16T15:54:53.390036+0000","last_peered":"2023-10-16T15:54:53.390036+0000","last_clean":"2023-10-16T15:54:53.390036+0000","last_became_active":"2023-10-16T15:54:39.202529+0000","last_became_peered":"2023-10-16T15:54:39.202529+0000","last_unstale":"2023-10-16T15:54:53.390036+0000","last_undegraded":"2023-10-16T15:54:53.390036+0000","last_fullsized":"2023-10-16T15:54:53.390036+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529812+0000","last_change":"2023-10-16T15:54:53.413494+0000","last_active":"2023-10-16T15:54:54.529812+0000","last_peered":"2023-10-16T15:54:54.529812+0000","last_clean":"2023-10-16T15:54:54.529812+0000","last_became_active":"2023-10-16T15:54:53.413037+0000","last_became_peered":"2023-10-16T15:54:53.413037+0000","last_unstale":"2023-10-16T15:54:54.529812+0000","last_undegraded":"2023-10-16T15:54:54.529812+0000","last_fullsized":"2023-10-16T15:54:54.529812+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:06.628301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410566+0000","last_change":"2023-10-16T15:54:52.378294+0000","last_active":"2023-10-16T15:54:54.410566+0000","last_peered":"2023-10-16T15:54:54.410566+0000","last_clean":"2023-10-16T15:54:54.410566+0000","last_became_active":"2023-10-16T15:54:52.377745+0000","last_became_peered":"2023-10-16T15:54:52.377745+0000","last_unstale":"2023-10-16T15:54:54.410566+0000","last_undegraded":"2023-10-16T15:54:54.410566+0000","last_fullsized":"2023-10-16T15:54:54.410566+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:54:46.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370636+0000","last_change":"2023-10-16T15:54:39.200394+0000","last_active":"2023-10-16T15:54:53.370636+0000","last_peered":"2023-10-16T15:54:53.370636+0000","last_clean":"2023-10-16T15:54:53.370636+0000","last_became_active":"2023-10-16T15:54:39.199822+0000","last_became_peered":"2023-10-16T15:54:39.199822+0000","last_unstale":"2023-10-16T15:54:53.370636+0000","last_undegraded":"2023-10-16T15:54:53.370636+0000","last_fullsized":"2023-10-16T15:54:53.370636+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:38.462835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529626+0000","last_change":"2023-10-16T15:54:53.414402+0000","last_active":"2023-10-16T15:54:54.529626+0000","last_peered":"2023-10-16T15:54:54.529626+0000","last_clean":"2023-10-16T15:54:54.529626+0000","last_became_active":"2023-10-16T15:54:53.412369+0000","last_became_peered":"2023-10-16T15:54:53.412369+0000","last_unstale":"2023-10-16T15:54:54.529626+0000","last_undegraded":"2023-10-16T15:54:54.529626+0000","last_fullsized":"2023-10-16T15:54:54.529626+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:56.945120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532557+0000","last_change":"2023-10-16T15:54:52.390908+0000","last_active":"2023-10-16T15:54:54.532557+0000","last_peered":"2023-10-16T15:54:54.532557+0000","last_clean":"2023-10-16T15:54:54.532557+0000","last_became_active":"2023-10-16T15:54:52.390553+0000","last_became_peered":"2023-10-16T15:54:52.390553+0000","last_unstale":"2023-10-16T15:54:54.532557+0000","last_undegraded":"2023-10-16T15:54:54.532557+0000","last_fullsized":"2023-10-16T15:54:54.532557+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:24.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390136+0000","last_change":"2023-10-16T15:54:39.210009+0000","last_active":"2023-10-16T15:54:53.390136+0000","last_peered":"2023-10-16T15:54:53.390136+0000","last_clean":"2023-10-16T15:54:53.390136+0000","last_became_active":"2023-10-16T15:54:39.209277+0000","last_became_peered":"2023-10-16T15:54:39.209277+0000","last_unstale":"2023-10-16T15:54:53.390136+0000","last_undegraded":"2023-10-16T15:54:53.390136+0000","last_fullsized":"2023-10-16T15:54:53.390136+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533298+0000","last_change":"2023-10-16T15:54:53.403243+0000","last_active":"2023-10-16T15:54:54.533298+0000","last_peered":"2023-10-16T15:54:54.533298+0000","last_clean":"2023-10-16T15:54:54.533298+0000","last_became_active":"2023-10-16T15:54:53.402945+0000","last_became_peered":"2023-10-16T15:54:53.402945+0000","last_unstale":"2023-10-16T15:54:54.533298+0000","last_undegraded":"2023-10-16T15:54:54.533298+0000","last_fullsized":"2023-10-16T15:54:54.533298+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:40.579694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532139+0000","last_change":"2023-10-16T15:54:52.398247+0000","last_active":"2023-10-16T15:54:54.532139+0000","last_peered":"2023-10-16T15:54:54.532139+0000","last_clean":"2023-10-16T15:54:54.532139+0000","last_became_active":"2023-10-16T15:54:52.397968+0000","last_became_peered":"2023-10-16T15:54:52.397968+0000","last_unstale":"2023-10-16T15:54:54.532139+0000","last_undegraded":"2023-10-16T15:54:54.532139+0000","last_fullsized":"2023-10-16T15:54:54.532139+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:46.112000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386274+0000","last_change":"2023-10-16T15:54:39.194527+0000","last_active":"2023-10-16T15:54:53.386274+0000","last_peered":"2023-10-16T15:54:53.386274+0000","last_clean":"2023-10-16T15:54:53.386274+0000","last_became_active":"2023-10-16T15:54:39.194123+0000","last_became_peered":"2023-10-16T15:54:39.194123+0000","last_unstale":"2023-10-16T15:54:53.386274+0000","last_undegraded":"2023-10-16T15:54:53.386274+0000","last_fullsized":"2023-10-16T15:54:53.386274+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:53.770029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532030+0000","last_change":"2023-10-16T15:54:53.409519+0000","last_active":"2023-10-16T15:54:54.532030+0000","last_peered":"2023-10-16T15:54:54.532030+0000","last_clean":"2023-10-16T15:54:54.532030+0000","last_became_active":"2023-10-16T15:54:53.408181+0000","last_became_peered":"2023-10-16T15:54:53.408181+0000","last_unstale":"2023-10-16T15:54:54.532030+0000","last_undegraded":"2023-10-16T15:54:54.532030+0000","last_fullsized":"2023-10-16T15:54:54.532030+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:11.802127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410541+0000","last_change":"2023-10-16T15:54:52.387751+0000","last_active":"2023-10-16T15:54:54.410541+0000","last_peered":"2023-10-16T15:54:54.410541+0000","last_clean":"2023-10-16T15:54:54.410541+0000","last_became_active":"2023-10-16T15:54:52.387202+0000","last_became_peered":"2023-10-16T15:54:52.387202+0000","last_unstale":"2023-10-16T15:54:54.410541+0000","last_undegraded":"2023-10-16T15:54:54.410541+0000","last_fullsized":"2023-10-16T15:54:54.410541+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:03.499950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385166+0000","last_change":"2023-10-16T15:54:39.208938+0000","last_active":"2023-10-16T15:54:53.385166+0000","last_peered":"2023-10-16T15:54:53.385166+0000","last_clean":"2023-10-16T15:54:53.385166+0000","last_became_active":"2023-10-16T15:54:39.208579+0000","last_became_peered":"2023-10-16T15:54:39.208579+0000","last_unstale":"2023-10-16T15:54:53.385166+0000","last_undegraded":"2023-10-16T15:54:53.385166+0000","last_fullsized":"2023-10-16T15:54:53.385166+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:32.928830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.411597+0000","last_change":"2023-10-16T15:54:53.412100+0000","last_active":"2023-10-16T15:54:54.411597+0000","last_peered":"2023-10-16T15:54:54.411597+0000","last_clean":"2023-10-16T15:54:54.411597+0000","last_became_active":"2023-10-16T15:54:53.411747+0000","last_became_peered":"2023-10-16T15:54:53.411747+0000","last_unstale":"2023-10-16T15:54:54.411597+0000","last_undegraded":"2023-10-16T15:54:54.411597+0000","last_fullsized":"2023-10-16T15:54:54.411597+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:37.450215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533197+0000","last_change":"2023-10-16T15:54:52.395278+0000","last_active":"2023-10-16T15:54:54.533197+0000","last_peered":"2023-10-16T15:54:54.533197+0000","last_clean":"2023-10-16T15:54:54.533197+0000","last_became_active":"2023-10-16T15:54:52.394638+0000","last_became_peered":"2023-10-16T15:54:52.394638+0000","last_unstale":"2023-10-16T15:54:54.533197+0000","last_undegraded":"2023-10-16T15:54:54.533197+0000","last_fullsized":"2023-10-16T15:54:54.533197+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:44.272665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375358+0000","last_change":"2023-10-16T15:54:39.187772+0000","last_active":"2023-10-16T15:54:53.375358+0000","last_peered":"2023-10-16T15:54:53.375358+0000","last_clean":"2023-10-16T15:54:53.375358+0000","last_became_active":"2023-10-16T15:54:39.187210+0000","last_became_peered":"2023-10-16T15:54:39.187210+0000","last_unstale":"2023-10-16T15:54:53.375358+0000","last_undegraded":"2023-10-16T15:54:53.375358+0000","last_fullsized":"2023-10-16T15:54:53.375358+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:31.089495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532688+0000","last_change":"2023-10-16T15:54:53.408805+0000","last_active":"2023-10-16T15:54:54.532688+0000","last_peered":"2023-10-16T15:54:54.532688+0000","last_clean":"2023-10-16T15:54:54.532688+0000","last_became_active":"2023-10-16T15:54:53.406250+0000","last_became_peered":"2023-10-16T15:54:53.406250+0000","last_unstale":"2023-10-16T15:54:54.532688+0000","last_undegraded":"2023-10-16T15:54:54.532688+0000","last_fullsized":"2023-10-16T15:54:54.532688+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:26.499127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412445+0000","last_change":"2023-10-16T15:54:52.399176+0000","last_active":"2023-10-16T15:54:54.412445+0000","last_peered":"2023-10-16T15:54:54.412445+0000","last_clean":"2023-10-16T15:54:54.412445+0000","last_became_active":"2023-10-16T15:54:52.398362+0000","last_became_peered":"2023-10-16T15:54:52.398362+0000","last_unstale":"2023-10-16T15:54:54.412445+0000","last_undegraded":"2023-10-16T15:54:54.412445+0000","last_fullsized":"2023-10-16T15:54:54.412445+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:44.118800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385912+0000","last_change":"2023-10-16T15:54:39.199187+0000","last_active":"2023-10-16T15:54:53.385912+0000","last_peered":"2023-10-16T15:54:53.385912+0000","last_clean":"2023-10-16T15:54:53.385912+0000","last_became_active":"2023-10-16T15:54:39.195076+0000","last_became_peered":"2023-10-16T15:54:39.195076+0000","last_unstale":"2023-10-16T15:54:53.385912+0000","last_undegraded":"2023-10-16T15:54:53.385912+0000","last_fullsized":"2023-10-16T15:54:53.385912+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:01.081391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530386+0000","last_change":"2023-10-16T15:54:53.401211+0000","last_active":"2023-10-16T15:54:54.530386+0000","last_peered":"2023-10-16T15:54:54.530386+0000","last_clean":"2023-10-16T15:54:54.530386+0000","last_became_active":"2023-10-16T15:54:53.399693+0000","last_became_peered":"2023-10-16T15:54:53.399693+0000","last_unstale":"2023-10-16T15:54:54.530386+0000","last_undegraded":"2023-10-16T15:54:54.530386+0000","last_fullsized":"2023-10-16T15:54:54.530386+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:24:36.775050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"249'8","reported_seq":20,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533845+0000","last_change":"2023-10-16T15:54:52.390353+0000","last_active":"2023-10-16T15:54:54.533845+0000","last_peered":"2023-10-16T15:54:54.533845+0000","last_clean":"2023-10-16T15:54:54.533845+0000","last_became_active":"2023-10-16T15:54:52.388675+0000","last_became_peered":"2023-10-16T15:54:52.388675+0000","last_unstale":"2023-10-16T15:54:54.533845+0000","last_undegraded":"2023-10-16T15:54:54.533845+0000","last_fullsized":"2023-10-16T15:54:54.533845+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:49.390733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.402409+0000","last_change":"2023-10-16T15:54:39.212389+0000","last_active":"2023-10-16T15:54:53.402409+0000","last_peered":"2023-10-16T15:54:53.402409+0000","last_clean":"2023-10-16T15:54:53.402409+0000","last_became_active":"2023-10-16T15:54:39.212017+0000","last_became_peered":"2023-10-16T15:54:39.212017+0000","last_unstale":"2023-10-16T15:54:53.402409+0000","last_undegraded":"2023-10-16T15:54:53.402409+0000","last_fullsized":"2023-10-16T15:54:53.402409+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529935+0000","last_change":"2023-10-16T15:54:53.412238+0000","last_active":"2023-10-16T15:54:54.529935+0000","last_peered":"2023-10-16T15:54:54.529935+0000","last_clean":"2023-10-16T15:54:54.529935+0000","last_became_active":"2023-10-16T15:54:53.411974+0000","last_became_peered":"2023-10-16T15:54:53.411974+0000","last_unstale":"2023-10-16T15:54:54.529935+0000","last_undegraded":"2023-10-16T15:54:54.529935+0000","last_fullsized":"2023-10-16T15:54:54.529935+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:53.511292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410103+0000","last_change":"2023-10-16T15:54:52.394346+0000","last_active":"2023-10-16T15:54:54.410103+0000","last_peered":"2023-10-16T15:54:54.410103+0000","last_clean":"2023-10-16T15:54:54.410103+0000","last_became_active":"2023-10-16T15:54:52.393847+0000","last_became_peered":"2023-10-16T15:54:52.393847+0000","last_unstale":"2023-10-16T15:54:54.410103+0000","last_undegraded":"2023-10-16T15:54:54.410103+0000","last_fullsized":"2023-10-16T15:54:54.410103+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:51.228820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391631+0000","last_change":"2023-10-16T15:54:39.214749+0000","last_active":"2023-10-16T15:54:53.391631+0000","last_peered":"2023-10-16T15:54:53.391631+0000","last_clean":"2023-10-16T15:54:53.391631+0000","last_became_active":"2023-10-16T15:54:39.214456+0000","last_became_peered":"2023-10-16T15:54:39.214456+0000","last_unstale":"2023-10-16T15:54:53.391631+0000","last_undegraded":"2023-10-16T15:54:53.391631+0000","last_fullsized":"2023-10-16T15:54:53.391631+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:36.851092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529685+0000","last_change":"2023-10-16T15:54:53.395570+0000","last_active":"2023-10-16T15:54:54.529685+0000","last_peered":"2023-10-16T15:54:54.529685+0000","last_clean":"2023-10-16T15:54:54.529685+0000","last_became_active":"2023-10-16T15:54:53.395267+0000","last_became_peered":"2023-10-16T15:54:53.395267+0000","last_unstale":"2023-10-16T15:54:54.529685+0000","last_undegraded":"2023-10-16T15:54:54.529685+0000","last_fullsized":"2023-10-16T15:54:54.529685+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:15.241619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405678+0000","last_change":"2023-10-16T15:54:52.395718+0000","last_active":"2023-10-16T15:54:54.405678+0000","last_peered":"2023-10-16T15:54:54.405678+0000","last_clean":"2023-10-16T15:54:54.405678+0000","last_became_active":"2023-10-16T15:54:52.395435+0000","last_became_peered":"2023-10-16T15:54:52.395435+0000","last_unstale":"2023-10-16T15:54:54.405678+0000","last_undegraded":"2023-10-16T15:54:54.405678+0000","last_fullsized":"2023-10-16T15:54:54.405678+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:39.562211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396099+0000","last_change":"2023-10-16T15:54:39.202583+0000","last_active":"2023-10-16T15:54:53.396099+0000","last_peered":"2023-10-16T15:54:53.396099+0000","last_clean":"2023-10-16T15:54:53.396099+0000","last_became_active":"2023-10-16T15:54:39.202278+0000","last_became_peered":"2023-10-16T15:54:39.202278+0000","last_unstale":"2023-10-16T15:54:53.396099+0000","last_undegraded":"2023-10-16T15:54:53.396099+0000","last_fullsized":"2023-10-16T15:54:53.396099+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531055+0000","last_change":"2023-10-16T15:54:53.399831+0000","last_active":"2023-10-16T15:54:54.531055+0000","last_peered":"2023-10-16T15:54:54.531055+0000","last_clean":"2023-10-16T15:54:54.531055+0000","last_became_active":"2023-10-16T15:54:53.399486+0000","last_became_peered":"2023-10-16T15:54:53.399486+0000","last_unstale":"2023-10-16T15:54:54.531055+0000","last_undegraded":"2023-10-16T15:54:54.531055+0000","last_fullsized":"2023-10-16T15:54:54.531055+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:56.150842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.411593+0000","last_change":"2023-10-16T15:54:52.389878+0000","last_active":"2023-10-16T15:54:54.411593+0000","last_peered":"2023-10-16T15:54:54.411593+0000","last_clean":"2023-10-16T15:54:54.411593+0000","last_became_active":"2023-10-16T15:54:52.389586+0000","last_became_peered":"2023-10-16T15:54:52.389586+0000","last_unstale":"2023-10-16T15:54:54.411593+0000","last_undegraded":"2023-10-16T15:54:54.411593+0000","last_fullsized":"2023-10-16T15:54:54.411593+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:51.818377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385313+0000","last_change":"2023-10-16T15:54:39.205053+0000","last_active":"2023-10-16T15:54:53.385313+0000","last_peered":"2023-10-16T15:54:53.385313+0000","last_clean":"2023-10-16T15:54:53.385313+0000","last_became_active":"2023-10-16T15:54:39.202502+0000","last_became_peered":"2023-10-16T15:54:39.202502+0000","last_unstale":"2023-10-16T15:54:53.385313+0000","last_undegraded":"2023-10-16T15:54:53.385313+0000","last_fullsized":"2023-10-16T15:54:53.385313+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:38.045649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532817+0000","last_change":"2023-10-16T15:54:53.406861+0000","last_active":"2023-10-16T15:54:54.532817+0000","last_peered":"2023-10-16T15:54:54.532817+0000","last_clean":"2023-10-16T15:54:54.532817+0000","last_became_active":"2023-10-16T15:54:53.406612+0000","last_became_peered":"2023-10-16T15:54:53.406612+0000","last_unstale":"2023-10-16T15:54:54.532817+0000","last_undegraded":"2023-10-16T15:54:54.532817+0000","last_fullsized":"2023-10-16T15:54:54.532817+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:33:35.711439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"249'6","reported_seq":18,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531935+0000","last_change":"2023-10-16T15:54:52.384065+0000","last_active":"2023-10-16T15:54:54.531935+0000","last_peered":"2023-10-16T15:54:54.531935+0000","last_clean":"2023-10-16T15:54:54.531935+0000","last_became_active":"2023-10-16T15:54:52.383745+0000","last_became_peered":"2023-10-16T15:54:52.383745+0000","last_unstale":"2023-10-16T15:54:54.531935+0000","last_undegraded":"2023-10-16T15:54:54.531935+0000","last_fullsized":"2023-10-16T15:54:54.531935+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:23.741186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387849+0000","last_change":"2023-10-16T15:54:39.202366+0000","last_active":"2023-10-16T15:54:53.387849+0000","last_peered":"2023-10-16T15:54:53.387849+0000","last_clean":"2023-10-16T15:54:53.387849+0000","last_became_active":"2023-10-16T15:54:39.201981+0000","last_became_peered":"2023-10-16T15:54:39.201981+0000","last_unstale":"2023-10-16T15:54:53.387849+0000","last_undegraded":"2023-10-16T15:54:53.387849+0000","last_fullsized":"2023-10-16T15:54:53.387849+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:00.755938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531929+0000","last_change":"2023-10-16T15:54:53.409568+0000","last_active":"2023-10-16T15:54:54.531929+0000","last_peered":"2023-10-16T15:54:54.531929+0000","last_clean":"2023-10-16T15:54:54.531929+0000","last_became_active":"2023-10-16T15:54:53.407950+0000","last_became_peered":"2023-10-16T15:54:53.407950+0000","last_unstale":"2023-10-16T15:54:54.531929+0000","last_undegraded":"2023-10-16T15:54:54.531929+0000","last_fullsized":"2023-10-16T15:54:54.531929+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:57.466035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"249'3","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530078+0000","last_change":"2023-10-16T15:54:52.398323+0000","last_active":"2023-10-16T15:54:54.530078+0000","last_peered":"2023-10-16T15:54:54.530078+0000","last_clean":"2023-10-16T15:54:54.530078+0000","last_became_active":"2023-10-16T15:54:52.398021+0000","last_became_peered":"2023-10-16T15:54:52.398021+0000","last_unstale":"2023-10-16T15:54:54.530078+0000","last_undegraded":"2023-10-16T15:54:54.530078+0000","last_fullsized":"2023-10-16T15:54:54.530078+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:18.517267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381338+0000","last_change":"2023-10-16T15:54:39.209910+0000","last_active":"2023-10-16T15:54:53.381338+0000","last_peered":"2023-10-16T15:54:53.381338+0000","last_clean":"2023-10-16T15:54:53.381338+0000","last_became_active":"2023-10-16T15:54:39.209504+0000","last_became_peered":"2023-10-16T15:54:39.209504+0000","last_unstale":"2023-10-16T15:54:53.381338+0000","last_undegraded":"2023-10-16T15:54:53.381338+0000","last_fullsized":"2023-10-16T15:54:53.381338+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530629+0000","last_change":"2023-10-16T15:54:53.404840+0000","last_active":"2023-10-16T15:54:54.530629+0000","last_peered":"2023-10-16T15:54:54.530629+0000","last_clean":"2023-10-16T15:54:54.530629+0000","last_became_active":"2023-10-16T15:54:53.404549+0000","last_became_peered":"2023-10-16T15:54:53.404549+0000","last_unstale":"2023-10-16T15:54:54.530629+0000","last_undegraded":"2023-10-16T15:54:54.530629+0000","last_fullsized":"2023-10-16T15:54:54.530629+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:16.610186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532770+0000","last_change":"2023-10-16T15:54:52.396630+0000","last_active":"2023-10-16T15:54:54.532770+0000","last_peered":"2023-10-16T15:54:54.532770+0000","last_clean":"2023-10-16T15:54:54.532770+0000","last_became_active":"2023-10-16T15:54:52.395733+0000","last_became_peered":"2023-10-16T15:54:52.395733+0000","last_unstale":"2023-10-16T15:54:54.532770+0000","last_undegraded":"2023-10-16T15:54:54.532770+0000","last_fullsized":"2023-10-16T15:54:54.532770+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:18.062870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384380+0000","last_change":"2023-10-16T15:54:39.207379+0000","last_active":"2023-10-16T15:54:53.384380+0000","last_peered":"2023-10-16T15:54:53.384380+0000","last_clean":"2023-10-16T15:54:53.384380+0000","last_became_active":"2023-10-16T15:54:39.207010+0000","last_became_peered":"2023-10-16T15:54:39.207010+0000","last_unstale":"2023-10-16T15:54:53.384380+0000","last_undegraded":"2023-10-16T15:54:53.384380+0000","last_fullsized":"2023-10-16T15:54:53.384380+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:45.890691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533869+0000","last_change":"2023-10-16T15:54:53.395468+0000","last_active":"2023-10-16T15:54:54.533869+0000","last_peered":"2023-10-16T15:54:54.533869+0000","last_clean":"2023-10-16T15:54:54.533869+0000","last_became_active":"2023-10-16T15:54:53.395099+0000","last_became_peered":"2023-10-16T15:54:53.395099+0000","last_unstale":"2023-10-16T15:54:54.533869+0000","last_undegraded":"2023-10-16T15:54:54.533869+0000","last_fullsized":"2023-10-16T15:54:54.533869+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:51.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531178+0000","last_change":"2023-10-16T15:54:52.400699+0000","last_active":"2023-10-16T15:54:54.531178+0000","last_peered":"2023-10-16T15:54:54.531178+0000","last_clean":"2023-10-16T15:54:54.531178+0000","last_became_active":"2023-10-16T15:54:52.399180+0000","last_became_peered":"2023-10-16T15:54:52.399180+0000","last_unstale":"2023-10-16T15:54:54.531178+0000","last_undegraded":"2023-10-16T15:54:54.531178+0000","last_fullsized":"2023-10-16T15:54:54.531178+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:27.946798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383190+0000","last_change":"2023-10-16T15:54:39.199635+0000","last_active":"2023-10-16T15:54:53.383190+0000","last_peered":"2023-10-16T15:54:53.383190+0000","last_clean":"2023-10-16T15:54:53.383190+0000","last_became_active":"2023-10-16T15:54:39.199152+0000","last_became_peered":"2023-10-16T15:54:39.199152+0000","last_unstale":"2023-10-16T15:54:53.383190+0000","last_undegraded":"2023-10-16T15:54:53.383190+0000","last_fullsized":"2023-10-16T15:54:53.383190+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:45.206271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531419+0000","last_change":"2023-10-16T15:54:53.391417+0000","last_active":"2023-10-16T15:54:54.531419+0000","last_peered":"2023-10-16T15:54:54.531419+0000","last_clean":"2023-10-16T15:54:54.531419+0000","last_became_active":"2023-10-16T15:54:53.390886+0000","last_became_peered":"2023-10-16T15:54:53.390886+0000","last_unstale":"2023-10-16T15:54:54.531419+0000","last_undegraded":"2023-10-16T15:54:54.531419+0000","last_fullsized":"2023-10-16T15:54:54.531419+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:03.765186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531543+0000","last_change":"2023-10-16T15:54:52.396387+0000","last_active":"2023-10-16T15:54:54.531543+0000","last_peered":"2023-10-16T15:54:54.531543+0000","last_clean":"2023-10-16T15:54:54.531543+0000","last_became_active":"2023-10-16T15:54:52.396048+0000","last_became_peered":"2023-10-16T15:54:52.396048+0000","last_unstale":"2023-10-16T15:54:54.531543+0000","last_undegraded":"2023-10-16T15:54:54.531543+0000","last_fullsized":"2023-10-16T15:54:54.531543+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:31.223305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.403931+0000","last_change":"2023-10-16T15:54:39.205509+0000","last_active":"2023-10-16T15:54:53.403931+0000","last_peered":"2023-10-16T15:54:53.403931+0000","last_clean":"2023-10-16T15:54:53.403931+0000","last_became_active":"2023-10-16T15:54:39.204206+0000","last_became_peered":"2023-10-16T15:54:39.204206+0000","last_unstale":"2023-10-16T15:54:53.403931+0000","last_undegraded":"2023-10-16T15:54:53.403931+0000","last_fullsized":"2023-10-16T15:54:53.403931+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529484+0000","last_change":"2023-10-16T15:54:53.399798+0000","last_active":"2023-10-16T15:54:54.529484+0000","last_peered":"2023-10-16T15:54:54.529484+0000","last_clean":"2023-10-16T15:54:54.529484+0000","last_became_active":"2023-10-16T15:54:53.399501+0000","last_became_peered":"2023-10-16T15:54:53.399501+0000","last_unstale":"2023-10-16T15:54:54.529484+0000","last_undegraded":"2023-10-16T15:54:54.529484+0000","last_fullsized":"2023-10-16T15:54:54.529484+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:30.954197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531959+0000","last_change":"2023-10-16T15:54:52.395316+0000","last_active":"2023-10-16T15:54:54.531959+0000","last_peered":"2023-10-16T15:54:54.531959+0000","last_clean":"2023-10-16T15:54:54.531959+0000","last_became_active":"2023-10-16T15:54:52.395019+0000","last_became_peered":"2023-10-16T15:54:52.395019+0000","last_unstale":"2023-10-16T15:54:54.531959+0000","last_undegraded":"2023-10-16T15:54:54.531959+0000","last_fullsized":"2023-10-16T15:54:54.531959+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:51.646005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390171+0000","last_change":"2023-10-16T15:54:39.194931+0000","last_active":"2023-10-16T15:54:53.390171+0000","last_peered":"2023-10-16T15:54:53.390171+0000","last_clean":"2023-10-16T15:54:53.390171+0000","last_became_active":"2023-10-16T15:54:39.194605+0000","last_became_peered":"2023-10-16T15:54:39.194605+0000","last_unstale":"2023-10-16T15:54:53.390171+0000","last_undegraded":"2023-10-16T15:54:53.390171+0000","last_fullsized":"2023-10-16T15:54:53.390171+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:11.013060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529559+0000","last_change":"2023-10-16T15:54:53.394365+0000","last_active":"2023-10-16T15:54:54.529559+0000","last_peered":"2023-10-16T15:54:54.529559+0000","last_clean":"2023-10-16T15:54:54.529559+0000","last_became_active":"2023-10-16T15:54:53.394016+0000","last_became_peered":"2023-10-16T15:54:53.394016+0000","last_unstale":"2023-10-16T15:54:54.529559+0000","last_undegraded":"2023-10-16T15:54:54.529559+0000","last_fullsized":"2023-10-16T15:54:54.529559+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:50.320294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530731+0000","last_change":"2023-10-16T15:54:52.380695+0000","last_active":"2023-10-16T15:54:54.530731+0000","last_peered":"2023-10-16T15:54:54.530731+0000","last_clean":"2023-10-16T15:54:54.530731+0000","last_became_active":"2023-10-16T15:54:52.379787+0000","last_became_peered":"2023-10-16T15:54:52.379787+0000","last_unstale":"2023-10-16T15:54:54.530731+0000","last_undegraded":"2023-10-16T15:54:54.530731+0000","last_fullsized":"2023-10-16T15:54:54.530731+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:19.194940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370882+0000","last_change":"2023-10-16T15:54:39.204169+0000","last_active":"2023-10-16T15:54:53.370882+0000","last_peered":"2023-10-16T15:54:53.370882+0000","last_clean":"2023-10-16T15:54:53.370882+0000","last_became_active":"2023-10-16T15:54:39.200108+0000","last_became_peered":"2023-10-16T15:54:39.200108+0000","last_unstale":"2023-10-16T15:54:53.370882+0000","last_undegraded":"2023-10-16T15:54:53.370882+0000","last_fullsized":"2023-10-16T15:54:53.370882+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:23.620457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405952+0000","last_change":"2023-10-16T15:54:53.398381+0000","last_active":"2023-10-16T15:54:54.405952+0000","last_peered":"2023-10-16T15:54:54.405952+0000","last_clean":"2023-10-16T15:54:54.405952+0000","last_became_active":"2023-10-16T15:54:53.394042+0000","last_became_peered":"2023-10-16T15:54:53.394042+0000","last_unstale":"2023-10-16T15:54:54.405952+0000","last_undegraded":"2023-10-16T15:54:54.405952+0000","last_fullsized":"2023-10-16T15:54:54.405952+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:16.136129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532866+0000","last_change":"2023-10-16T15:54:52.396735+0000","last_active":"2023-10-16T15:54:54.532866+0000","last_peered":"2023-10-16T15:54:54.532866+0000","last_clean":"2023-10-16T15:54:54.532866+0000","last_became_active":"2023-10-16T15:54:52.396343+0000","last_became_peered":"2023-10-16T15:54:52.396343+0000","last_unstale":"2023-10-16T15:54:54.532866+0000","last_undegraded":"2023-10-16T15:54:54.532866+0000","last_fullsized":"2023-10-16T15:54:54.532866+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:51.623743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.389265+0000","last_change":"2023-10-16T15:54:39.203124+0000","last_active":"2023-10-16T15:54:53.389265+0000","last_peered":"2023-10-16T15:54:53.389265+0000","last_clean":"2023-10-16T15:54:53.389265+0000","last_became_active":"2023-10-16T15:54:39.200245+0000","last_became_peered":"2023-10-16T15:54:39.200245+0000","last_unstale":"2023-10-16T15:54:53.389265+0000","last_undegraded":"2023-10-16T15:54:53.389265+0000","last_fullsized":"2023-10-16T15:54:53.389265+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:43.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412113+0000","last_change":"2023-10-16T15:54:53.399367+0000","last_active":"2023-10-16T15:54:54.412113+0000","last_peered":"2023-10-16T15:54:54.412113+0000","last_clean":"2023-10-16T15:54:54.412113+0000","last_became_active":"2023-10-16T15:54:53.399017+0000","last_became_peered":"2023-10-16T15:54:53.399017+0000","last_unstale":"2023-10-16T15:54:54.412113+0000","last_undegraded":"2023-10-16T15:54:54.412113+0000","last_fullsized":"2023-10-16T15:54:54.412113+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:32.240788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530754+0000","last_change":"2023-10-16T15:54:52.396851+0000","last_active":"2023-10-16T15:54:54.530754+0000","last_peered":"2023-10-16T15:54:54.530754+0000","last_clean":"2023-10-16T15:54:54.530754+0000","last_became_active":"2023-10-16T15:54:52.396542+0000","last_became_peered":"2023-10-16T15:54:52.396542+0000","last_unstale":"2023-10-16T15:54:54.530754+0000","last_undegraded":"2023-10-16T15:54:54.530754+0000","last_fullsized":"2023-10-16T15:54:54.530754+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:09.513209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370742+0000","last_change":"2023-10-16T15:54:39.203684+0000","last_active":"2023-10-16T15:54:53.370742+0000","last_peered":"2023-10-16T15:54:53.370742+0000","last_clean":"2023-10-16T15:54:53.370742+0000","last_became_active":"2023-10-16T15:54:39.203327+0000","last_became_peered":"2023-10-16T15:54:39.203327+0000","last_unstale":"2023-10-16T15:54:53.370742+0000","last_undegraded":"2023-10-16T15:54:53.370742+0000","last_fullsized":"2023-10-16T15:54:53.370742+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:29.051715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532144+0000","last_change":"2023-10-16T15:54:53.395970+0000","last_active":"2023-10-16T15:54:54.532144+0000","last_peered":"2023-10-16T15:54:54.532144+0000","last_clean":"2023-10-16T15:54:54.532144+0000","last_became_active":"2023-10-16T15:54:53.395642+0000","last_became_peered":"2023-10-16T15:54:53.395642+0000","last_unstale":"2023-10-16T15:54:54.532144+0000","last_undegraded":"2023-10-16T15:54:54.532144+0000","last_fullsized":"2023-10-16T15:54:54.532144+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:57.375885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530070+0000","last_change":"2023-10-16T15:54:52.389455+0000","last_active":"2023-10-16T15:54:54.530070+0000","last_peered":"2023-10-16T15:54:54.530070+0000","last_clean":"2023-10-16T15:54:54.530070+0000","last_became_active":"2023-10-16T15:54:52.389180+0000","last_became_peered":"2023-10-16T15:54:52.389180+0000","last_unstale":"2023-10-16T15:54:54.530070+0000","last_undegraded":"2023-10-16T15:54:54.530070+0000","last_fullsized":"2023-10-16T15:54:54.530070+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:28.726584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396883+0000","last_change":"2023-10-16T15:54:39.203576+0000","last_active":"2023-10-16T15:54:53.396883+0000","last_peered":"2023-10-16T15:54:53.396883+0000","last_clean":"2023-10-16T15:54:53.396883+0000","last_became_active":"2023-10-16T15:54:39.203072+0000","last_became_peered":"2023-10-16T15:54:39.203072+0000","last_unstale":"2023-10-16T15:54:53.396883+0000","last_undegraded":"2023-10-16T15:54:53.396883+0000","last_fullsized":"2023-10-16T15:54:53.396883+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:05.472712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532864+0000","last_change":"2023-10-16T15:54:53.413070+0000","last_active":"2023-10-16T15:54:54.532864+0000","last_peered":"2023-10-16T15:54:54.532864+0000","last_clean":"2023-10-16T15:54:54.532864+0000","last_became_active":"2023-10-16T15:54:53.412519+0000","last_became_peered":"2023-10-16T15:54:53.412519+0000","last_unstale":"2023-10-16T15:54:54.532864+0000","last_undegraded":"2023-10-16T15:54:54.532864+0000","last_fullsized":"2023-10-16T15:54:54.532864+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:08.839899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532435+0000","last_change":"2023-10-16T15:54:52.392344+0000","last_active":"2023-10-16T15:54:54.532435+0000","last_peered":"2023-10-16T15:54:54.532435+0000","last_clean":"2023-10-16T15:54:54.532435+0000","last_became_active":"2023-10-16T15:54:52.388373+0000","last_became_peered":"2023-10-16T15:54:52.388373+0000","last_unstale":"2023-10-16T15:54:54.532435+0000","last_undegraded":"2023-10-16T15:54:54.532435+0000","last_fullsized":"2023-10-16T15:54:54.532435+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:57.878065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386595+0000","last_change":"2023-10-16T15:54:39.210101+0000","last_active":"2023-10-16T15:54:53.386595+0000","last_peered":"2023-10-16T15:54:53.386595+0000","last_clean":"2023-10-16T15:54:53.386595+0000","last_became_active":"2023-10-16T15:54:39.209787+0000","last_became_peered":"2023-10-16T15:54:39.209787+0000","last_unstale":"2023-10-16T15:54:53.386595+0000","last_undegraded":"2023-10-16T15:54:53.386595+0000","last_fullsized":"2023-10-16T15:54:53.386595+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533984+0000","last_change":"2023-10-16T15:54:53.403896+0000","last_active":"2023-10-16T15:54:54.533984+0000","last_peered":"2023-10-16T15:54:54.533984+0000","last_clean":"2023-10-16T15:54:54.533984+0000","last_became_active":"2023-10-16T15:54:53.403550+0000","last_became_peered":"2023-10-16T15:54:53.403550+0000","last_unstale":"2023-10-16T15:54:54.533984+0000","last_undegraded":"2023-10-16T15:54:54.533984+0000","last_fullsized":"2023-10-16T15:54:54.533984+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:48.011756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"249'4","reported_seq":16,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530936+0000","last_change":"2023-10-16T15:54:52.389234+0000","last_active":"2023-10-16T15:54:54.530936+0000","last_peered":"2023-10-16T15:54:54.530936+0000","last_clean":"2023-10-16T15:54:54.530936+0000","last_became_active":"2023-10-16T15:54:52.388899+0000","last_became_peered":"2023-10-16T15:54:52.388899+0000","last_unstale":"2023-10-16T15:54:54.530936+0000","last_undegraded":"2023-10-16T15:54:54.530936+0000","last_fullsized":"2023-10-16T15:54:54.530936+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:51.328255+0000","last_clean_scrub_stamp":"2023-10-16T15:54:51.328255+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.408543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383375+0000","last_change":"2023-10-16T15:54:39.198844+0000","last_active":"2023-10-16T15:54:53.383375+0000","last_peered":"2023-10-16T15:54:53.383375+0000","last_clean":"2023-10-16T15:54:53.383375+0000","last_became_active":"2023-10-16T15:54:39.198444+0000","last_became_peered":"2023-10-16T15:54:39.198444+0000","last_unstale":"2023-10-16T15:54:53.383375+0000","last_undegraded":"2023-10-16T15:54:53.383375+0000","last_fullsized":"2023-10-16T15:54:53.383375+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:47.149143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.410474+0000","last_change":"2023-10-16T15:54:53.399239+0000","last_active":"2023-10-16T15:54:54.410474+0000","last_peered":"2023-10-16T15:54:54.410474+0000","last_clean":"2023-10-16T15:54:54.410474+0000","last_became_active":"2023-10-16T15:54:53.398895+0000","last_became_peered":"2023-10-16T15:54:53.398895+0000","last_unstale":"2023-10-16T15:54:54.410474+0000","last_undegraded":"2023-10-16T15:54:54.410474+0000","last_fullsized":"2023-10-16T15:54:54.410474+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:39:15.592335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387724+0000","last_change":"2023-10-16T15:54:39.213572+0000","last_active":"2023-10-16T15:54:53.387724+0000","last_peered":"2023-10-16T15:54:53.387724+0000","last_clean":"2023-10-16T15:54:53.387724+0000","last_became_active":"2023-10-16T15:54:39.213226+0000","last_became_peered":"2023-10-16T15:54:39.213226+0000","last_unstale":"2023-10-16T15:54:53.387724+0000","last_undegraded":"2023-10-16T15:54:53.387724+0000","last_fullsized":"2023-10-16T15:54:53.387724+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:23.249562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390476+0000","last_change":"2023-10-16T15:54:38.194041+0000","last_active":"2023-10-16T15:54:53.390476+0000","last_peered":"2023-10-16T15:54:53.390476+0000","last_clean":"2023-10-16T15:54:53.390476+0000","last_became_active":"2023-10-16T15:54:38.193770+0000","last_became_peered":"2023-10-16T15:54:38.193770+0000","last_unstale":"2023-10-16T15:54:53.390476+0000","last_undegraded":"2023-10-16T15:54:53.390476+0000","last_fullsized":"2023-10-16T15:54:53.390476+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530335+0000","last_change":"2023-10-16T15:54:53.409258+0000","last_active":"2023-10-16T15:54:54.530335+0000","last_peered":"2023-10-16T15:54:54.530335+0000","last_clean":"2023-10-16T15:54:54.530335+0000","last_became_active":"2023-10-16T15:54:53.408227+0000","last_became_peered":"2023-10-16T15:54:53.408227+0000","last_unstale":"2023-10-16T15:54:54.530335+0000","last_undegraded":"2023-10-16T15:54:54.530335+0000","last_fullsized":"2023-10-16T15:54:54.530335+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:35.426801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369598+0000","last_change":"2023-10-16T15:54:39.209350+0000","last_active":"2023-10-16T15:54:53.369598+0000","last_peered":"2023-10-16T15:54:53.369598+0000","last_clean":"2023-10-16T15:54:53.369598+0000","last_became_active":"2023-10-16T15:54:39.208829+0000","last_became_peered":"2023-10-16T15:54:39.208829+0000","last_unstale":"2023-10-16T15:54:53.369598+0000","last_undegraded":"2023-10-16T15:54:53.369598+0000","last_fullsized":"2023-10-16T15:54:53.369598+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:20.584860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391767+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:53.391767+0000","last_peered":"2023-10-16T15:54:53.391767+0000","last_clean":"2023-10-16T15:54:53.391767+0000","last_became_active":"2023-10-16T15:54:38.195767+0000","last_became_peered":"2023-10-16T15:54:38.195767+0000","last_unstale":"2023-10-16T15:54:53.391767+0000","last_undegraded":"2023-10-16T15:54:53.391767+0000","last_fullsized":"2023-10-16T15:54:53.391767+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:59.473824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531558+0000","last_change":"2023-10-16T15:54:53.405520+0000","last_active":"2023-10-16T15:54:54.531558+0000","last_peered":"2023-10-16T15:54:54.531558+0000","last_clean":"2023-10-16T15:54:54.531558+0000","last_became_active":"2023-10-16T15:54:53.405248+0000","last_became_peered":"2023-10-16T15:54:53.405248+0000","last_unstale":"2023-10-16T15:54:54.531558+0000","last_undegraded":"2023-10-16T15:54:54.531558+0000","last_fullsized":"2023-10-16T15:54:54.531558+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:23.283640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396200+0000","last_change":"2023-10-16T15:54:39.209739+0000","last_active":"2023-10-16T15:54:53.396200+0000","last_peered":"2023-10-16T15:54:53.396200+0000","last_clean":"2023-10-16T15:54:53.396200+0000","last_became_active":"2023-10-16T15:54:39.208616+0000","last_became_peered":"2023-10-16T15:54:39.208616+0000","last_unstale":"2023-10-16T15:54:53.396200+0000","last_undegraded":"2023-10-16T15:54:53.396200+0000","last_fullsized":"2023-10-16T15:54:53.396200+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385621+0000","last_change":"2023-10-16T15:54:38.192004+0000","last_active":"2023-10-16T15:54:53.385621+0000","last_peered":"2023-10-16T15:54:53.385621+0000","last_clean":"2023-10-16T15:54:53.385621+0000","last_became_active":"2023-10-16T15:54:38.191365+0000","last_became_peered":"2023-10-16T15:54:38.191365+0000","last_unstale":"2023-10-16T15:54:53.385621+0000","last_undegraded":"2023-10-16T15:54:53.385621+0000","last_fullsized":"2023-10-16T15:54:53.385621+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:00.022703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532106+0000","last_change":"2023-10-16T15:54:53.408294+0000","last_active":"2023-10-16T15:54:54.532106+0000","last_peered":"2023-10-16T15:54:54.532106+0000","last_clean":"2023-10-16T15:54:54.532106+0000","last_became_active":"2023-10-16T15:54:53.407567+0000","last_became_peered":"2023-10-16T15:54:53.407567+0000","last_unstale":"2023-10-16T15:54:54.532106+0000","last_undegraded":"2023-10-16T15:54:54.532106+0000","last_fullsized":"2023-10-16T15:54:54.532106+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:13.405161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370148+0000","last_change":"2023-10-16T15:54:39.201055+0000","last_active":"2023-10-16T15:54:53.370148+0000","last_peered":"2023-10-16T15:54:53.370148+0000","last_clean":"2023-10-16T15:54:53.370148+0000","last_became_active":"2023-10-16T15:54:39.199846+0000","last_became_peered":"2023-10-16T15:54:39.199846+0000","last_unstale":"2023-10-16T15:54:53.370148+0000","last_undegraded":"2023-10-16T15:54:53.370148+0000","last_fullsized":"2023-10-16T15:54:53.370148+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:28.581753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385713+0000","last_change":"2023-10-16T15:54:38.188455+0000","last_active":"2023-10-16T15:54:53.385713+0000","last_peered":"2023-10-16T15:54:53.385713+0000","last_clean":"2023-10-16T15:54:53.385713+0000","last_became_active":"2023-10-16T15:54:38.188167+0000","last_became_peered":"2023-10-16T15:54:38.188167+0000","last_unstale":"2023-10-16T15:54:53.385713+0000","last_undegraded":"2023-10-16T15:54:53.385713+0000","last_fullsized":"2023-10-16T15:54:53.385713+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:21.554285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.534298+0000","last_change":"2023-10-16T15:54:53.400521+0000","last_active":"2023-10-16T15:54:54.534298+0000","last_peered":"2023-10-16T15:54:54.534298+0000","last_clean":"2023-10-16T15:54:54.534298+0000","last_became_active":"2023-10-16T15:54:53.400227+0000","last_became_peered":"2023-10-16T15:54:53.400227+0000","last_unstale":"2023-10-16T15:54:54.534298+0000","last_undegraded":"2023-10-16T15:54:54.534298+0000","last_fullsized":"2023-10-16T15:54:54.534298+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:00.410178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.377860+0000","last_change":"2023-10-16T15:54:39.198491+0000","last_active":"2023-10-16T15:54:53.377860+0000","last_peered":"2023-10-16T15:54:53.377860+0000","last_clean":"2023-10-16T15:54:53.377860+0000","last_became_active":"2023-10-16T15:54:39.194488+0000","last_became_peered":"2023-10-16T15:54:39.194488+0000","last_unstale":"2023-10-16T15:54:53.377860+0000","last_undegraded":"2023-10-16T15:54:53.377860+0000","last_fullsized":"2023-10-16T15:54:53.377860+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:59.663424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385449+0000","last_change":"2023-10-16T15:54:38.190757+0000","last_active":"2023-10-16T15:54:53.385449+0000","last_peered":"2023-10-16T15:54:53.385449+0000","last_clean":"2023-10-16T15:54:53.385449+0000","last_became_active":"2023-10-16T15:54:38.190491+0000","last_became_peered":"2023-10-16T15:54:38.190491+0000","last_unstale":"2023-10-16T15:54:53.385449+0000","last_undegraded":"2023-10-16T15:54:53.385449+0000","last_fullsized":"2023-10-16T15:54:53.385449+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:59.249150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533699+0000","last_change":"2023-10-16T15:54:53.406145+0000","last_active":"2023-10-16T15:54:54.533699+0000","last_peered":"2023-10-16T15:54:54.533699+0000","last_clean":"2023-10-16T15:54:54.533699+0000","last_became_active":"2023-10-16T15:54:53.405801+0000","last_became_peered":"2023-10-16T15:54:53.405801+0000","last_unstale":"2023-10-16T15:54:54.533699+0000","last_undegraded":"2023-10-16T15:54:54.533699+0000","last_fullsized":"2023-10-16T15:54:54.533699+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:41.829804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386011+0000","last_change":"2023-10-16T15:54:39.206722+0000","last_active":"2023-10-16T15:54:53.386011+0000","last_peered":"2023-10-16T15:54:53.386011+0000","last_clean":"2023-10-16T15:54:53.386011+0000","last_became_active":"2023-10-16T15:54:39.204447+0000","last_became_peered":"2023-10-16T15:54:39.204447+0000","last_unstale":"2023-10-16T15:54:53.386011+0000","last_undegraded":"2023-10-16T15:54:53.386011+0000","last_fullsized":"2023-10-16T15:54:53.386011+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:50.316780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381531+0000","last_change":"2023-10-16T15:54:38.189769+0000","last_active":"2023-10-16T15:54:53.381531+0000","last_peered":"2023-10-16T15:54:53.381531+0000","last_clean":"2023-10-16T15:54:53.381531+0000","last_became_active":"2023-10-16T15:54:38.189418+0000","last_became_peered":"2023-10-16T15:54:38.189418+0000","last_unstale":"2023-10-16T15:54:53.381531+0000","last_undegraded":"2023-10-16T15:54:53.381531+0000","last_fullsized":"2023-10-16T15:54:53.381531+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:37.444572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533368+0000","last_change":"2023-10-16T15:54:53.412728+0000","last_active":"2023-10-16T15:54:54.533368+0000","last_peered":"2023-10-16T15:54:54.533368+0000","last_clean":"2023-10-16T15:54:54.533368+0000","last_became_active":"2023-10-16T15:54:53.411210+0000","last_became_peered":"2023-10-16T15:54:53.411210+0000","last_unstale":"2023-10-16T15:54:54.533368+0000","last_undegraded":"2023-10-16T15:54:54.533368+0000","last_fullsized":"2023-10-16T15:54:54.533368+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:01.541143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384245+0000","last_change":"2023-10-16T15:54:39.205679+0000","last_active":"2023-10-16T15:54:53.384245+0000","last_peered":"2023-10-16T15:54:53.384245+0000","last_clean":"2023-10-16T15:54:53.384245+0000","last_became_active":"2023-10-16T15:54:39.204821+0000","last_became_peered":"2023-10-16T15:54:39.204821+0000","last_unstale":"2023-10-16T15:54:53.384245+0000","last_undegraded":"2023-10-16T15:54:53.384245+0000","last_fullsized":"2023-10-16T15:54:53.384245+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:44.694895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386273+0000","last_change":"2023-10-16T15:54:38.191853+0000","last_active":"2023-10-16T15:54:53.386273+0000","last_peered":"2023-10-16T15:54:53.386273+0000","last_clean":"2023-10-16T15:54:53.386273+0000","last_became_active":"2023-10-16T15:54:38.190725+0000","last_became_peered":"2023-10-16T15:54:38.190725+0000","last_unstale":"2023-10-16T15:54:53.386273+0000","last_undegraded":"2023-10-16T15:54:53.386273+0000","last_fullsized":"2023-10-16T15:54:53.386273+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:51.409385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529773+0000","last_change":"2023-10-16T15:54:53.402476+0000","last_active":"2023-10-16T15:54:54.529773+0000","last_peered":"2023-10-16T15:54:54.529773+0000","last_clean":"2023-10-16T15:54:54.529773+0000","last_became_active":"2023-10-16T15:54:53.402138+0000","last_became_peered":"2023-10-16T15:54:53.402138+0000","last_unstale":"2023-10-16T15:54:54.529773+0000","last_undegraded":"2023-10-16T15:54:54.529773+0000","last_fullsized":"2023-10-16T15:54:54.529773+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:15.461715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391760+0000","last_change":"2023-10-16T15:54:39.205421+0000","last_active":"2023-10-16T15:54:53.391760+0000","last_peered":"2023-10-16T15:54:53.391760+0000","last_clean":"2023-10-16T15:54:53.391760+0000","last_became_active":"2023-10-16T15:54:39.204865+0000","last_became_peered":"2023-10-16T15:54:39.204865+0000","last_unstale":"2023-10-16T15:54:53.391760+0000","last_undegraded":"2023-10-16T15:54:53.391760+0000","last_fullsized":"2023-10-16T15:54:53.391760+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:33.811103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.402410+0000","last_change":"2023-10-16T15:54:38.196240+0000","last_active":"2023-10-16T15:54:53.402410+0000","last_peered":"2023-10-16T15:54:53.402410+0000","last_clean":"2023-10-16T15:54:53.402410+0000","last_became_active":"2023-10-16T15:54:38.195833+0000","last_became_peered":"2023-10-16T15:54:38.195833+0000","last_unstale":"2023-10-16T15:54:53.402410+0000","last_undegraded":"2023-10-16T15:54:53.402410+0000","last_fullsized":"2023-10-16T15:54:53.402410+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531020+0000","last_change":"2023-10-16T15:54:53.392476+0000","last_active":"2023-10-16T15:54:54.531020+0000","last_peered":"2023-10-16T15:54:54.531020+0000","last_clean":"2023-10-16T15:54:54.531020+0000","last_became_active":"2023-10-16T15:54:53.391954+0000","last_became_peered":"2023-10-16T15:54:53.391954+0000","last_unstale":"2023-10-16T15:54:54.531020+0000","last_undegraded":"2023-10-16T15:54:54.531020+0000","last_fullsized":"2023-10-16T15:54:54.531020+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:07.891800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.371969+0000","last_change":"2023-10-16T15:54:39.204677+0000","last_active":"2023-10-16T15:54:53.371969+0000","last_peered":"2023-10-16T15:54:53.371969+0000","last_clean":"2023-10-16T15:54:53.371969+0000","last_became_active":"2023-10-16T15:54:39.202794+0000","last_became_peered":"2023-10-16T15:54:39.202794+0000","last_unstale":"2023-10-16T15:54:53.371969+0000","last_undegraded":"2023-10-16T15:54:53.371969+0000","last_fullsized":"2023-10-16T15:54:53.371969+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:11.420100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.382524+0000","last_change":"2023-10-16T15:54:38.195322+0000","last_active":"2023-10-16T15:54:53.382524+0000","last_peered":"2023-10-16T15:54:53.382524+0000","last_clean":"2023-10-16T15:54:53.382524+0000","last_became_active":"2023-10-16T15:54:38.192163+0000","last_became_peered":"2023-10-16T15:54:38.192163+0000","last_unstale":"2023-10-16T15:54:53.382524+0000","last_undegraded":"2023-10-16T15:54:53.382524+0000","last_fullsized":"2023-10-16T15:54:53.382524+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:45.552506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405742+0000","last_change":"2023-10-16T15:54:53.404252+0000","last_active":"2023-10-16T15:54:54.405742+0000","last_peered":"2023-10-16T15:54:54.405742+0000","last_clean":"2023-10-16T15:54:54.405742+0000","last_became_active":"2023-10-16T15:54:53.403522+0000","last_became_peered":"2023-10-16T15:54:53.403522+0000","last_unstale":"2023-10-16T15:54:54.405742+0000","last_undegraded":"2023-10-16T15:54:54.405742+0000","last_fullsized":"2023-10-16T15:54:54.405742+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:44.102171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385490+0000","last_change":"2023-10-16T15:54:39.212753+0000","last_active":"2023-10-16T15:54:53.385490+0000","last_peered":"2023-10-16T15:54:53.385490+0000","last_clean":"2023-10-16T15:54:53.385490+0000","last_became_active":"2023-10-16T15:54:39.212450+0000","last_became_peered":"2023-10-16T15:54:39.212450+0000","last_unstale":"2023-10-16T15:54:53.385490+0000","last_undegraded":"2023-10-16T15:54:53.385490+0000","last_fullsized":"2023-10-16T15:54:53.385490+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:04.593631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"245'3","reported_seq":19,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391258+0000","last_change":"2023-10-16T15:54:38.188991+0000","last_active":"2023-10-16T15:54:53.391258+0000","last_peered":"2023-10-16T15:54:53.391258+0000","last_clean":"2023-10-16T15:54:53.391258+0000","last_became_active":"2023-10-16T15:54:38.187521+0000","last_became_peered":"2023-10-16T15:54:38.187521+0000","last_unstale":"2023-10-16T15:54:53.391258+0000","last_undegraded":"2023-10-16T15:54:53.391258+0000","last_fullsized":"2023-10-16T15:54:53.391258+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:37.422309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.405892+0000","last_change":"2023-10-16T15:54:53.396365+0000","last_active":"2023-10-16T15:54:54.405892+0000","last_peered":"2023-10-16T15:54:54.405892+0000","last_clean":"2023-10-16T15:54:54.405892+0000","last_became_active":"2023-10-16T15:54:53.393308+0000","last_became_peered":"2023-10-16T15:54:53.393308+0000","last_unstale":"2023-10-16T15:54:54.405892+0000","last_undegraded":"2023-10-16T15:54:54.405892+0000","last_fullsized":"2023-10-16T15:54:54.405892+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:58.760049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386407+0000","last_change":"2023-10-16T15:54:39.188004+0000","last_active":"2023-10-16T15:54:53.386407+0000","last_peered":"2023-10-16T15:54:53.386407+0000","last_clean":"2023-10-16T15:54:53.386407+0000","last_became_active":"2023-10-16T15:54:39.187443+0000","last_became_peered":"2023-10-16T15:54:39.187443+0000","last_unstale":"2023-10-16T15:54:53.386407+0000","last_undegraded":"2023-10-16T15:54:53.386407+0000","last_fullsized":"2023-10-16T15:54:53.386407+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"247'8","reported_seq":24,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.389360+0000","last_change":"2023-10-16T15:54:38.192821+0000","last_active":"2023-10-16T15:54:53.389360+0000","last_peered":"2023-10-16T15:54:53.389360+0000","last_clean":"2023-10-16T15:54:53.389360+0000","last_became_active":"2023-10-16T15:54:38.191495+0000","last_became_peered":"2023-10-16T15:54:38.191495+0000","last_unstale":"2023-10-16T15:54:53.389360+0000","last_undegraded":"2023-10-16T15:54:53.389360+0000","last_fullsized":"2023-10-16T15:54:53.389360+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:36.442402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530637+0000","last_change":"2023-10-16T15:54:53.404553+0000","last_active":"2023-10-16T15:54:54.530637+0000","last_peered":"2023-10-16T15:54:54.530637+0000","last_clean":"2023-10-16T15:54:54.530637+0000","last_became_active":"2023-10-16T15:54:53.404272+0000","last_became_peered":"2023-10-16T15:54:53.404272+0000","last_unstale":"2023-10-16T15:54:54.530637+0000","last_undegraded":"2023-10-16T15:54:54.530637+0000","last_fullsized":"2023-10-16T15:54:54.530637+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:55.651126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384915+0000","last_change":"2023-10-16T15:54:39.203188+0000","last_active":"2023-10-16T15:54:53.384915+0000","last_peered":"2023-10-16T15:54:53.384915+0000","last_clean":"2023-10-16T15:54:53.384915+0000","last_became_active":"2023-10-16T15:54:39.202646+0000","last_became_peered":"2023-10-16T15:54:39.202646+0000","last_unstale":"2023-10-16T15:54:53.384915+0000","last_undegraded":"2023-10-16T15:54:53.384915+0000","last_fullsized":"2023-10-16T15:54:53.384915+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:46.629980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387656+0000","last_change":"2023-10-16T15:54:38.190094+0000","last_active":"2023-10-16T15:54:53.387656+0000","last_peered":"2023-10-16T15:54:53.387656+0000","last_clean":"2023-10-16T15:54:53.387656+0000","last_became_active":"2023-10-16T15:54:38.189825+0000","last_became_peered":"2023-10-16T15:54:38.189825+0000","last_unstale":"2023-10-16T15:54:53.387656+0000","last_undegraded":"2023-10-16T15:54:53.387656+0000","last_fullsized":"2023-10-16T15:54:53.387656+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:17.021872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533723+0000","last_change":"2023-10-16T15:54:53.403026+0000","last_active":"2023-10-16T15:54:54.533723+0000","last_peered":"2023-10-16T15:54:54.533723+0000","last_clean":"2023-10-16T15:54:54.533723+0000","last_became_active":"2023-10-16T15:54:53.402600+0000","last_became_peered":"2023-10-16T15:54:53.402600+0000","last_unstale":"2023-10-16T15:54:54.533723+0000","last_undegraded":"2023-10-16T15:54:54.533723+0000","last_fullsized":"2023-10-16T15:54:54.533723+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:08.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.378063+0000","last_change":"2023-10-16T15:54:39.205965+0000","last_active":"2023-10-16T15:54:53.378063+0000","last_peered":"2023-10-16T15:54:53.378063+0000","last_clean":"2023-10-16T15:54:53.378063+0000","last_became_active":"2023-10-16T15:54:39.205686+0000","last_became_peered":"2023-10-16T15:54:39.205686+0000","last_unstale":"2023-10-16T15:54:53.378063+0000","last_undegraded":"2023-10-16T15:54:53.378063+0000","last_fullsized":"2023-10-16T15:54:53.378063+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:36.119641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385348+0000","last_change":"2023-10-16T15:54:38.195373+0000","last_active":"2023-10-16T15:54:53.385348+0000","last_peered":"2023-10-16T15:54:53.385348+0000","last_clean":"2023-10-16T15:54:53.385348+0000","last_became_active":"2023-10-16T15:54:38.194574+0000","last_became_peered":"2023-10-16T15:54:38.194574+0000","last_unstale":"2023-10-16T15:54:53.385348+0000","last_undegraded":"2023-10-16T15:54:53.385348+0000","last_fullsized":"2023-10-16T15:54:53.385348+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:35.101378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.535386+0000","last_change":"2023-10-16T15:54:53.395661+0000","last_active":"2023-10-16T15:54:54.535386+0000","last_peered":"2023-10-16T15:54:54.535386+0000","last_clean":"2023-10-16T15:54:54.535386+0000","last_became_active":"2023-10-16T15:54:53.395114+0000","last_became_peered":"2023-10-16T15:54:53.395114+0000","last_unstale":"2023-10-16T15:54:54.535386+0000","last_undegraded":"2023-10-16T15:54:54.535386+0000","last_fullsized":"2023-10-16T15:54:54.535386+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:57.466035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.382788+0000","last_change":"2023-10-16T15:54:39.207613+0000","last_active":"2023-10-16T15:54:53.382788+0000","last_peered":"2023-10-16T15:54:53.382788+0000","last_clean":"2023-10-16T15:54:53.382788+0000","last_became_active":"2023-10-16T15:54:39.207320+0000","last_became_peered":"2023-10-16T15:54:39.207320+0000","last_unstale":"2023-10-16T15:54:53.382788+0000","last_undegraded":"2023-10-16T15:54:53.382788+0000","last_fullsized":"2023-10-16T15:54:53.382788+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:27.629150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383033+0000","last_change":"2023-10-16T15:54:38.196425+0000","last_active":"2023-10-16T15:54:53.383033+0000","last_peered":"2023-10-16T15:54:53.383033+0000","last_clean":"2023-10-16T15:54:53.383033+0000","last_became_active":"2023-10-16T15:54:38.195269+0000","last_became_peered":"2023-10-16T15:54:38.195269+0000","last_unstale":"2023-10-16T15:54:53.383033+0000","last_undegraded":"2023-10-16T15:54:53.383033+0000","last_fullsized":"2023-10-16T15:54:53.383033+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:01.207615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531467+0000","last_change":"2023-10-16T15:54:53.400454+0000","last_active":"2023-10-16T15:54:54.531467+0000","last_peered":"2023-10-16T15:54:54.531467+0000","last_clean":"2023-10-16T15:54:54.531467+0000","last_became_active":"2023-10-16T15:54:53.400141+0000","last_became_peered":"2023-10-16T15:54:53.400141+0000","last_unstale":"2023-10-16T15:54:54.531467+0000","last_undegraded":"2023-10-16T15:54:54.531467+0000","last_fullsized":"2023-10-16T15:54:54.531467+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:57.598566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381337+0000","last_change":"2023-10-16T15:54:39.205462+0000","last_active":"2023-10-16T15:54:53.381337+0000","last_peered":"2023-10-16T15:54:53.381337+0000","last_clean":"2023-10-16T15:54:53.381337+0000","last_became_active":"2023-10-16T15:54:39.204956+0000","last_became_peered":"2023-10-16T15:54:39.204956+0000","last_unstale":"2023-10-16T15:54:53.381337+0000","last_undegraded":"2023-10-16T15:54:53.381337+0000","last_fullsized":"2023-10-16T15:54:53.381337+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:26.379041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"245'8","reported_seq":24,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369047+0000","last_change":"2023-10-16T15:54:38.194134+0000","last_active":"2023-10-16T15:54:53.369047+0000","last_peered":"2023-10-16T15:54:53.369047+0000","last_clean":"2023-10-16T15:54:53.369047+0000","last_became_active":"2023-10-16T15:54:38.193858+0000","last_became_peered":"2023-10-16T15:54:38.193858+0000","last_unstale":"2023-10-16T15:54:53.369047+0000","last_undegraded":"2023-10-16T15:54:53.369047+0000","last_fullsized":"2023-10-16T15:54:53.369047+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:37.866131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531697+0000","last_change":"2023-10-16T15:54:53.416410+0000","last_active":"2023-10-16T15:54:54.531697+0000","last_peered":"2023-10-16T15:54:54.531697+0000","last_clean":"2023-10-16T15:54:54.531697+0000","last_became_active":"2023-10-16T15:54:53.413028+0000","last_became_peered":"2023-10-16T15:54:53.413028+0000","last_unstale":"2023-10-16T15:54:54.531697+0000","last_undegraded":"2023-10-16T15:54:54.531697+0000","last_fullsized":"2023-10-16T15:54:54.531697+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:10.907637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.391617+0000","last_change":"2023-10-16T15:54:39.203343+0000","last_active":"2023-10-16T15:54:53.391617+0000","last_peered":"2023-10-16T15:54:53.391617+0000","last_clean":"2023-10-16T15:54:53.391617+0000","last_became_active":"2023-10-16T15:54:39.203044+0000","last_became_peered":"2023-10-16T15:54:39.203044+0000","last_unstale":"2023-10-16T15:54:53.391617+0000","last_undegraded":"2023-10-16T15:54:53.391617+0000","last_fullsized":"2023-10-16T15:54:53.391617+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:23:49.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.389407+0000","last_change":"2023-10-16T15:54:38.193502+0000","last_active":"2023-10-16T15:54:53.389407+0000","last_peered":"2023-10-16T15:54:53.389407+0000","last_clean":"2023-10-16T15:54:53.389407+0000","last_became_active":"2023-10-16T15:54:38.193186+0000","last_became_peered":"2023-10-16T15:54:38.193186+0000","last_unstale":"2023-10-16T15:54:53.389407+0000","last_undegraded":"2023-10-16T15:54:53.389407+0000","last_fullsized":"2023-10-16T15:54:53.389407+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:59.883297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531341+0000","last_change":"2023-10-16T15:54:53.406582+0000","last_active":"2023-10-16T15:54:54.531341+0000","last_peered":"2023-10-16T15:54:54.531341+0000","last_clean":"2023-10-16T15:54:54.531341+0000","last_became_active":"2023-10-16T15:54:53.406311+0000","last_became_peered":"2023-10-16T15:54:53.406311+0000","last_unstale":"2023-10-16T15:54:54.531341+0000","last_undegraded":"2023-10-16T15:54:54.531341+0000","last_fullsized":"2023-10-16T15:54:54.531341+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:51.051745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.388049+0000","last_change":"2023-10-16T15:54:39.206218+0000","last_active":"2023-10-16T15:54:53.388049+0000","last_peered":"2023-10-16T15:54:53.388049+0000","last_clean":"2023-10-16T15:54:53.388049+0000","last_became_active":"2023-10-16T15:54:39.205815+0000","last_became_peered":"2023-10-16T15:54:39.205815+0000","last_unstale":"2023-10-16T15:54:53.388049+0000","last_undegraded":"2023-10-16T15:54:53.388049+0000","last_fullsized":"2023-10-16T15:54:53.388049+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:24.642539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.382751+0000","last_change":"2023-10-16T15:54:38.196048+0000","last_active":"2023-10-16T15:54:53.382751+0000","last_peered":"2023-10-16T15:54:53.382751+0000","last_clean":"2023-10-16T15:54:53.382751+0000","last_became_active":"2023-10-16T15:54:38.195752+0000","last_became_peered":"2023-10-16T15:54:38.195752+0000","last_unstale":"2023-10-16T15:54:53.382751+0000","last_undegraded":"2023-10-16T15:54:53.382751+0000","last_fullsized":"2023-10-16T15:54:53.382751+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:15.145108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.530007+0000","last_change":"2023-10-16T15:54:53.416186+0000","last_active":"2023-10-16T15:54:54.530007+0000","last_peered":"2023-10-16T15:54:54.530007+0000","last_clean":"2023-10-16T15:54:54.530007+0000","last_became_active":"2023-10-16T15:54:53.415871+0000","last_became_peered":"2023-10-16T15:54:53.415871+0000","last_unstale":"2023-10-16T15:54:54.530007+0000","last_undegraded":"2023-10-16T15:54:54.530007+0000","last_fullsized":"2023-10-16T15:54:54.530007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:57.375885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.372975+0000","last_change":"2023-10-16T15:54:39.198603+0000","last_active":"2023-10-16T15:54:53.372975+0000","last_peered":"2023-10-16T15:54:53.372975+0000","last_clean":"2023-10-16T15:54:53.372975+0000","last_became_active":"2023-10-16T15:54:39.195793+0000","last_became_peered":"2023-10-16T15:54:39.195793+0000","last_unstale":"2023-10-16T15:54:53.372975+0000","last_undegraded":"2023-10-16T15:54:53.372975+0000","last_fullsized":"2023-10-16T15:54:53.372975+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:16.753544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"245'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375520+0000","last_change":"2023-10-16T15:54:38.193258+0000","last_active":"2023-10-16T15:54:53.375520+0000","last_peered":"2023-10-16T15:54:53.375520+0000","last_clean":"2023-10-16T15:54:53.375520+0000","last_became_active":"2023-10-16T15:54:38.192538+0000","last_became_peered":"2023-10-16T15:54:38.192538+0000","last_unstale":"2023-10-16T15:54:53.375520+0000","last_undegraded":"2023-10-16T15:54:53.375520+0000","last_fullsized":"2023-10-16T15:54:53.375520+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:07.463985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531738+0000","last_change":"2023-10-16T15:54:53.407645+0000","last_active":"2023-10-16T15:54:54.531738+0000","last_peered":"2023-10-16T15:54:54.531738+0000","last_clean":"2023-10-16T15:54:54.531738+0000","last_became_active":"2023-10-16T15:54:53.407018+0000","last_became_peered":"2023-10-16T15:54:53.407018+0000","last_unstale":"2023-10-16T15:54:54.531738+0000","last_undegraded":"2023-10-16T15:54:54.531738+0000","last_fullsized":"2023-10-16T15:54:54.531738+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:00.410178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.396297+0000","last_change":"2023-10-16T15:54:39.206968+0000","last_active":"2023-10-16T15:54:53.396297+0000","last_peered":"2023-10-16T15:54:53.396297+0000","last_clean":"2023-10-16T15:54:53.396297+0000","last_became_active":"2023-10-16T15:54:39.206271+0000","last_became_peered":"2023-10-16T15:54:39.206271+0000","last_unstale":"2023-10-16T15:54:53.396297+0000","last_undegraded":"2023-10-16T15:54:53.396297+0000","last_fullsized":"2023-10-16T15:54:53.396297+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:26.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369863+0000","last_change":"2023-10-16T15:54:38.194296+0000","last_active":"2023-10-16T15:54:53.369863+0000","last_peered":"2023-10-16T15:54:53.369863+0000","last_clean":"2023-10-16T15:54:53.369863+0000","last_became_active":"2023-10-16T15:54:38.194015+0000","last_became_peered":"2023-10-16T15:54:38.194015+0000","last_unstale":"2023-10-16T15:54:53.369863+0000","last_undegraded":"2023-10-16T15:54:53.369863+0000","last_fullsized":"2023-10-16T15:54:53.369863+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:58.920639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412516+0000","last_change":"2023-10-16T15:54:53.411797+0000","last_active":"2023-10-16T15:54:54.412516+0000","last_peered":"2023-10-16T15:54:54.412516+0000","last_clean":"2023-10-16T15:54:54.412516+0000","last_became_active":"2023-10-16T15:54:53.411433+0000","last_became_peered":"2023-10-16T15:54:53.411433+0000","last_unstale":"2023-10-16T15:54:54.412516+0000","last_undegraded":"2023-10-16T15:54:54.412516+0000","last_fullsized":"2023-10-16T15:54:54.412516+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:51.661319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387054+0000","last_change":"2023-10-16T15:54:39.206512+0000","last_active":"2023-10-16T15:54:53.387054+0000","last_peered":"2023-10-16T15:54:53.387054+0000","last_clean":"2023-10-16T15:54:53.387054+0000","last_became_active":"2023-10-16T15:54:39.206174+0000","last_became_peered":"2023-10-16T15:54:39.206174+0000","last_unstale":"2023-10-16T15:54:53.387054+0000","last_undegraded":"2023-10-16T15:54:53.387054+0000","last_fullsized":"2023-10-16T15:54:53.387054+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.853676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.388052+0000","last_change":"2023-10-16T15:54:38.199965+0000","last_active":"2023-10-16T15:54:53.388052+0000","last_peered":"2023-10-16T15:54:53.388052+0000","last_clean":"2023-10-16T15:54:53.388052+0000","last_became_active":"2023-10-16T15:54:38.199417+0000","last_became_peered":"2023-10-16T15:54:38.199417+0000","last_unstale":"2023-10-16T15:54:53.388052+0000","last_undegraded":"2023-10-16T15:54:53.388052+0000","last_fullsized":"2023-10-16T15:54:53.388052+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:55.688720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533410+0000","last_change":"2023-10-16T15:54:53.396057+0000","last_active":"2023-10-16T15:54:54.533410+0000","last_peered":"2023-10-16T15:54:54.533410+0000","last_clean":"2023-10-16T15:54:54.533410+0000","last_became_active":"2023-10-16T15:54:53.395705+0000","last_became_peered":"2023-10-16T15:54:53.395705+0000","last_unstale":"2023-10-16T15:54:54.533410+0000","last_undegraded":"2023-10-16T15:54:54.533410+0000","last_fullsized":"2023-10-16T15:54:54.533410+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:06:35.426801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387556+0000","last_change":"2023-10-16T15:54:39.199452+0000","last_active":"2023-10-16T15:54:53.387556+0000","last_peered":"2023-10-16T15:54:53.387556+0000","last_clean":"2023-10-16T15:54:53.387556+0000","last_became_active":"2023-10-16T15:54:39.199132+0000","last_became_peered":"2023-10-16T15:54:39.199132+0000","last_unstale":"2023-10-16T15:54:53.387556+0000","last_undegraded":"2023-10-16T15:54:53.387556+0000","last_fullsized":"2023-10-16T15:54:53.387556+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:49.564533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.372618+0000","last_change":"2023-10-16T15:54:38.193942+0000","last_active":"2023-10-16T15:54:53.372618+0000","last_peered":"2023-10-16T15:54:53.372618+0000","last_clean":"2023-10-16T15:54:53.372618+0000","last_became_active":"2023-10-16T15:54:38.193397+0000","last_became_peered":"2023-10-16T15:54:38.193397+0000","last_unstale":"2023-10-16T15:54:53.372618+0000","last_undegraded":"2023-10-16T15:54:53.372618+0000","last_fullsized":"2023-10-16T15:54:53.372618+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:40.933295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.533641+0000","last_change":"2023-10-16T15:54:53.404406+0000","last_active":"2023-10-16T15:54:54.533641+0000","last_peered":"2023-10-16T15:54:54.533641+0000","last_clean":"2023-10-16T15:54:54.533641+0000","last_became_active":"2023-10-16T15:54:53.403821+0000","last_became_peered":"2023-10-16T15:54:53.403821+0000","last_unstale":"2023-10-16T15:54:54.533641+0000","last_undegraded":"2023-10-16T15:54:54.533641+0000","last_fullsized":"2023-10-16T15:54:54.533641+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:36.633161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.368895+0000","last_change":"2023-10-16T15:54:39.188638+0000","last_active":"2023-10-16T15:54:53.368895+0000","last_peered":"2023-10-16T15:54:53.368895+0000","last_clean":"2023-10-16T15:54:53.368895+0000","last_became_active":"2023-10-16T15:54:39.187890+0000","last_became_peered":"2023-10-16T15:54:39.187890+0000","last_unstale":"2023-10-16T15:54:53.368895+0000","last_undegraded":"2023-10-16T15:54:53.368895+0000","last_fullsized":"2023-10-16T15:54:53.368895+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:15:41.121120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375961+0000","last_change":"2023-10-16T15:54:38.196671+0000","last_active":"2023-10-16T15:54:53.375961+0000","last_peered":"2023-10-16T15:54:53.375961+0000","last_clean":"2023-10-16T15:54:53.375961+0000","last_became_active":"2023-10-16T15:54:38.196392+0000","last_became_peered":"2023-10-16T15:54:38.196392+0000","last_unstale":"2023-10-16T15:54:53.375961+0000","last_undegraded":"2023-10-16T15:54:53.375961+0000","last_fullsized":"2023-10-16T15:54:53.375961+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:35.189299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.406794+0000","last_change":"2023-10-16T15:54:53.392163+0000","last_active":"2023-10-16T15:54:54.406794+0000","last_peered":"2023-10-16T15:54:54.406794+0000","last_clean":"2023-10-16T15:54:54.406794+0000","last_became_active":"2023-10-16T15:54:53.389078+0000","last_became_peered":"2023-10-16T15:54:53.389078+0000","last_unstale":"2023-10-16T15:54:54.406794+0000","last_undegraded":"2023-10-16T15:54:54.406794+0000","last_fullsized":"2023-10-16T15:54:54.406794+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:05.340702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384551+0000","last_change":"2023-10-16T15:54:39.203931+0000","last_active":"2023-10-16T15:54:53.384551+0000","last_peered":"2023-10-16T15:54:53.384551+0000","last_clean":"2023-10-16T15:54:53.384551+0000","last_became_active":"2023-10-16T15:54:39.203614+0000","last_became_peered":"2023-10-16T15:54:39.203614+0000","last_unstale":"2023-10-16T15:54:53.384551+0000","last_undegraded":"2023-10-16T15:54:53.384551+0000","last_fullsized":"2023-10-16T15:54:53.384551+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:22.572548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.381805+0000","last_change":"2023-10-16T15:54:38.187905+0000","last_active":"2023-10-16T15:54:53.381805+0000","last_peered":"2023-10-16T15:54:53.381805+0000","last_clean":"2023-10-16T15:54:53.381805+0000","last_became_active":"2023-10-16T15:54:38.187553+0000","last_became_peered":"2023-10-16T15:54:38.187553+0000","last_unstale":"2023-10-16T15:54:53.381805+0000","last_undegraded":"2023-10-16T15:54:53.381805+0000","last_fullsized":"2023-10-16T15:54:53.381805+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:27.563490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.408240+0000","last_change":"2023-10-16T15:54:53.404941+0000","last_active":"2023-10-16T15:54:54.408240+0000","last_peered":"2023-10-16T15:54:54.408240+0000","last_clean":"2023-10-16T15:54:54.408240+0000","last_became_active":"2023-10-16T15:54:53.404667+0000","last_became_peered":"2023-10-16T15:54:53.404667+0000","last_unstale":"2023-10-16T15:54:54.408240+0000","last_undegraded":"2023-10-16T15:54:54.408240+0000","last_fullsized":"2023-10-16T15:54:54.408240+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:51:29.050486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.390320+0000","last_change":"2023-10-16T15:54:39.201684+0000","last_active":"2023-10-16T15:54:53.390320+0000","last_peered":"2023-10-16T15:54:53.390320+0000","last_clean":"2023-10-16T15:54:53.390320+0000","last_became_active":"2023-10-16T15:54:39.201363+0000","last_became_peered":"2023-10-16T15:54:39.201363+0000","last_unstale":"2023-10-16T15:54:53.390320+0000","last_undegraded":"2023-10-16T15:54:53.390320+0000","last_fullsized":"2023-10-16T15:54:53.390320+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:49.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"245'2","reported_seq":18,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.404031+0000","last_change":"2023-10-16T15:54:38.193490+0000","last_active":"2023-10-16T15:54:53.404031+0000","last_peered":"2023-10-16T15:54:53.404031+0000","last_clean":"2023-10-16T15:54:53.404031+0000","last_became_active":"2023-10-16T15:54:38.193206+0000","last_became_peered":"2023-10-16T15:54:38.193206+0000","last_unstale":"2023-10-16T15:54:53.404031+0000","last_undegraded":"2023-10-16T15:54:53.404031+0000","last_fullsized":"2023-10-16T15:54:53.404031+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:39.542212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531029+0000","last_change":"2023-10-16T15:54:53.395414+0000","last_active":"2023-10-16T15:54:54.531029+0000","last_peered":"2023-10-16T15:54:54.531029+0000","last_clean":"2023-10-16T15:54:54.531029+0000","last_became_active":"2023-10-16T15:54:53.394997+0000","last_became_peered":"2023-10-16T15:54:53.394997+0000","last_unstale":"2023-10-16T15:54:54.531029+0000","last_undegraded":"2023-10-16T15:54:54.531029+0000","last_fullsized":"2023-10-16T15:54:54.531029+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:01.349796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.392507+0000","last_change":"2023-10-16T15:54:39.206833+0000","last_active":"2023-10-16T15:54:53.392507+0000","last_peered":"2023-10-16T15:54:53.392507+0000","last_clean":"2023-10-16T15:54:53.392507+0000","last_became_active":"2023-10-16T15:54:39.206491+0000","last_became_peered":"2023-10-16T15:54:39.206491+0000","last_unstale":"2023-10-16T15:54:53.392507+0000","last_undegraded":"2023-10-16T15:54:53.392507+0000","last_fullsized":"2023-10-16T15:54:53.392507+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:22.432507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"246'1","reported_seq":17,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.399788+0000","last_change":"2023-10-16T15:54:38.198292+0000","last_active":"2023-10-16T15:54:53.399788+0000","last_peered":"2023-10-16T15:54:53.399788+0000","last_clean":"2023-10-16T15:54:53.399788+0000","last_became_active":"2023-10-16T15:54:38.197965+0000","last_became_peered":"2023-10-16T15:54:38.197965+0000","last_unstale":"2023-10-16T15:54:53.399788+0000","last_undegraded":"2023-10-16T15:54:53.399788+0000","last_fullsized":"2023-10-16T15:54:53.399788+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:07.579428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.529426+0000","last_change":"2023-10-16T15:54:53.403179+0000","last_active":"2023-10-16T15:54:54.529426+0000","last_peered":"2023-10-16T15:54:54.529426+0000","last_clean":"2023-10-16T15:54:54.529426+0000","last_became_active":"2023-10-16T15:54:53.402875+0000","last_became_peered":"2023-10-16T15:54:53.402875+0000","last_unstale":"2023-10-16T15:54:54.529426+0000","last_undegraded":"2023-10-16T15:54:54.529426+0000","last_fullsized":"2023-10-16T15:54:54.529426+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:13.864078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.369757+0000","last_change":"2023-10-16T15:54:39.198360+0000","last_active":"2023-10-16T15:54:53.369757+0000","last_peered":"2023-10-16T15:54:53.369757+0000","last_clean":"2023-10-16T15:54:53.369757+0000","last_became_active":"2023-10-16T15:54:39.196440+0000","last_became_peered":"2023-10-16T15:54:39.196440+0000","last_unstale":"2023-10-16T15:54:53.369757+0000","last_undegraded":"2023-10-16T15:54:53.369757+0000","last_fullsized":"2023-10-16T15:54:53.369757+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:23.439544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.375749+0000","last_change":"2023-10-16T15:54:38.199930+0000","last_active":"2023-10-16T15:54:53.375749+0000","last_peered":"2023-10-16T15:54:53.375749+0000","last_clean":"2023-10-16T15:54:53.375749+0000","last_became_active":"2023-10-16T15:54:38.199386+0000","last_became_peered":"2023-10-16T15:54:38.199386+0000","last_unstale":"2023-10-16T15:54:53.375749+0000","last_undegraded":"2023-10-16T15:54:53.375749+0000","last_fullsized":"2023-10-16T15:54:53.375749+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:57.311627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532738+0000","last_change":"2023-10-16T15:54:53.404203+0000","last_active":"2023-10-16T15:54:54.532738+0000","last_peered":"2023-10-16T15:54:54.532738+0000","last_clean":"2023-10-16T15:54:54.532738+0000","last_became_active":"2023-10-16T15:54:53.403894+0000","last_became_peered":"2023-10-16T15:54:53.403894+0000","last_unstale":"2023-10-16T15:54:54.532738+0000","last_undegraded":"2023-10-16T15:54:54.532738+0000","last_fullsized":"2023-10-16T15:54:54.532738+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:01.054329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.371113+0000","last_change":"2023-10-16T15:54:39.205703+0000","last_active":"2023-10-16T15:54:53.371113+0000","last_peered":"2023-10-16T15:54:53.371113+0000","last_clean":"2023-10-16T15:54:53.371113+0000","last_became_active":"2023-10-16T15:54:39.205415+0000","last_became_peered":"2023-10-16T15:54:39.205415+0000","last_unstale":"2023-10-16T15:54:53.371113+0000","last_undegraded":"2023-10-16T15:54:53.371113+0000","last_fullsized":"2023-10-16T15:54:53.371113+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:02.225878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.370038+0000","last_change":"2023-10-16T15:54:38.184497+0000","last_active":"2023-10-16T15:54:53.370038+0000","last_peered":"2023-10-16T15:54:53.370038+0000","last_clean":"2023-10-16T15:54:53.370038+0000","last_became_active":"2023-10-16T15:54:38.184043+0000","last_became_peered":"2023-10-16T15:54:38.184043+0000","last_unstale":"2023-10-16T15:54:53.370038+0000","last_undegraded":"2023-10-16T15:54:53.370038+0000","last_fullsized":"2023-10-16T15:54:53.370038+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:04.454449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.532675+0000","last_change":"2023-10-16T15:54:53.404681+0000","last_active":"2023-10-16T15:54:54.532675+0000","last_peered":"2023-10-16T15:54:54.532675+0000","last_clean":"2023-10-16T15:54:54.532675+0000","last_became_active":"2023-10-16T15:54:53.404185+0000","last_became_peered":"2023-10-16T15:54:53.404185+0000","last_unstale":"2023-10-16T15:54:54.532675+0000","last_undegraded":"2023-10-16T15:54:54.532675+0000","last_fullsized":"2023-10-16T15:54:54.532675+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:36.773201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.385453+0000","last_change":"2023-10-16T15:54:39.192680+0000","last_active":"2023-10-16T15:54:53.385453+0000","last_peered":"2023-10-16T15:54:53.385453+0000","last_clean":"2023-10-16T15:54:53.385453+0000","last_became_active":"2023-10-16T15:54:39.192254+0000","last_became_peered":"2023-10-16T15:54:39.192254+0000","last_unstale":"2023-10-16T15:54:53.385453+0000","last_undegraded":"2023-10-16T15:54:53.385453+0000","last_fullsized":"2023-10-16T15:54:53.385453+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:40.560475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386828+0000","last_change":"2023-10-16T15:54:38.192812+0000","last_active":"2023-10-16T15:54:53.386828+0000","last_peered":"2023-10-16T15:54:53.386828+0000","last_clean":"2023-10-16T15:54:53.386828+0000","last_became_active":"2023-10-16T15:54:38.192528+0000","last_became_peered":"2023-10-16T15:54:38.192528+0000","last_unstale":"2023-10-16T15:54:53.386828+0000","last_undegraded":"2023-10-16T15:54:53.386828+0000","last_fullsized":"2023-10-16T15:54:53.386828+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:36.002573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.534498+0000","last_change":"2023-10-16T15:54:53.415073+0000","last_active":"2023-10-16T15:54:54.534498+0000","last_peered":"2023-10-16T15:54:54.534498+0000","last_clean":"2023-10-16T15:54:54.534498+0000","last_became_active":"2023-10-16T15:54:53.414773+0000","last_became_peered":"2023-10-16T15:54:53.414773+0000","last_unstale":"2023-10-16T15:54:54.534498+0000","last_undegraded":"2023-10-16T15:54:54.534498+0000","last_fullsized":"2023-10-16T15:54:54.534498+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:40.643056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.387976+0000","last_change":"2023-10-16T15:54:39.207142+0000","last_active":"2023-10-16T15:54:53.387976+0000","last_peered":"2023-10-16T15:54:53.387976+0000","last_clean":"2023-10-16T15:54:53.387976+0000","last_became_active":"2023-10-16T15:54:39.206593+0000","last_became_peered":"2023-10-16T15:54:39.206593+0000","last_unstale":"2023-10-16T15:54:53.387976+0000","last_undegraded":"2023-10-16T15:54:53.387976+0000","last_fullsized":"2023-10-16T15:54:53.387976+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:58.545523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384227+0000","last_change":"2023-10-16T15:54:38.179331+0000","last_active":"2023-10-16T15:54:53.384227+0000","last_peered":"2023-10-16T15:54:53.384227+0000","last_clean":"2023-10-16T15:54:53.384227+0000","last_became_active":"2023-10-16T15:54:38.178684+0000","last_became_peered":"2023-10-16T15:54:38.178684+0000","last_unstale":"2023-10-16T15:54:53.384227+0000","last_undegraded":"2023-10-16T15:54:53.384227+0000","last_fullsized":"2023-10-16T15:54:53.384227+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:45:44.872428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.411175+0000","last_change":"2023-10-16T15:54:53.399801+0000","last_active":"2023-10-16T15:54:54.411175+0000","last_peered":"2023-10-16T15:54:54.411175+0000","last_clean":"2023-10-16T15:54:54.411175+0000","last_became_active":"2023-10-16T15:54:53.399400+0000","last_became_peered":"2023-10-16T15:54:53.399400+0000","last_unstale":"2023-10-16T15:54:54.411175+0000","last_undegraded":"2023-10-16T15:54:54.411175+0000","last_fullsized":"2023-10-16T15:54:54.411175+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:38.843193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.383843+0000","last_change":"2023-10-16T15:54:39.203357+0000","last_active":"2023-10-16T15:54:53.383843+0000","last_peered":"2023-10-16T15:54:53.383843+0000","last_clean":"2023-10-16T15:54:53.383843+0000","last_became_active":"2023-10-16T15:54:39.203058+0000","last_became_peered":"2023-10-16T15:54:39.203058+0000","last_unstale":"2023-10-16T15:54:53.383843+0000","last_undegraded":"2023-10-16T15:54:53.383843+0000","last_fullsized":"2023-10-16T15:54:53.383843+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:37.020836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386249+0000","last_change":"2023-10-16T15:54:38.195221+0000","last_active":"2023-10-16T15:54:53.386249+0000","last_peered":"2023-10-16T15:54:53.386249+0000","last_clean":"2023-10-16T15:54:53.386249+0000","last_became_active":"2023-10-16T15:54:38.194937+0000","last_became_peered":"2023-10-16T15:54:38.194937+0000","last_unstale":"2023-10-16T15:54:53.386249+0000","last_undegraded":"2023-10-16T15:54:53.386249+0000","last_fullsized":"2023-10-16T15:54:53.386249+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:41.726203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.412524+0000","last_change":"2023-10-16T15:54:53.402565+0000","last_active":"2023-10-16T15:54:54.412524+0000","last_peered":"2023-10-16T15:54:54.412524+0000","last_clean":"2023-10-16T15:54:54.412524+0000","last_became_active":"2023-10-16T15:54:53.400842+0000","last_became_peered":"2023-10-16T15:54:53.400842+0000","last_unstale":"2023-10-16T15:54:54.412524+0000","last_undegraded":"2023-10-16T15:54:54.412524+0000","last_fullsized":"2023-10-16T15:54:54.412524+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:14.468066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.384473+0000","last_change":"2023-10-16T15:54:39.207259+0000","last_active":"2023-10-16T15:54:53.384473+0000","last_peered":"2023-10-16T15:54:53.384473+0000","last_clean":"2023-10-16T15:54:53.384473+0000","last_became_active":"2023-10-16T15:54:39.206912+0000","last_became_peered":"2023-10-16T15:54:39.206912+0000","last_unstale":"2023-10-16T15:54:53.384473+0000","last_undegraded":"2023-10-16T15:54:53.384473+0000","last_fullsized":"2023-10-16T15:54:53.384473+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:53.691146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.386481+0000","last_change":"2023-10-16T15:54:38.200919+0000","last_active":"2023-10-16T15:54:53.386481+0000","last_peered":"2023-10-16T15:54:53.386481+0000","last_clean":"2023-10-16T15:54:53.386481+0000","last_became_active":"2023-10-16T15:54:38.200141+0000","last_became_peered":"2023-10-16T15:54:38.200141+0000","last_unstale":"2023-10-16T15:54:53.386481+0000","last_undegraded":"2023-10-16T15:54:53.386481+0000","last_fullsized":"2023-10-16T15:54:53.386481+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:40.432210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T15:54:54.531052+0000","last_change":"2023-10-16T15:54:53.411598+0000","last_active":"2023-10-16T15:54:54.531052+0000","last_peered":"2023-10-16T15:54:54.531052+0000","last_clean":"2023-10-16T15:54:54.531052+0000","last_became_active":"2023-10-16T15:54:53.411267+0000","last_became_peered":"2023-10-16T15:54:53.411267+0000","last_unstale":"2023-10-16T15:54:54.531052+0000","last_undegraded":"2023-10-16T15:54:54.531052+0000","last_fullsized":"2023-10-16T15:54:54.531052+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:52.345739+0000","last_clean_scrub_stamp":"2023-10-16T15:54:52.345739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:47.129484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":15,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.399705+0000","last_change":"2023-10-16T15:54:39.208421+0000","last_active":"2023-10-16T15:54:53.399705+0000","last_peered":"2023-10-16T15:54:53.399705+0000","last_clean":"2023-10-16T15:54:53.399705+0000","last_became_active":"2023-10-16T15:54:39.207372+0000","last_became_peered":"2023-10-16T15:54:39.207372+0000","last_unstale":"2023-10-16T15:54:53.399705+0000","last_undegraded":"2023-10-16T15:54:53.399705+0000","last_fullsized":"2023-10-16T15:54:53.399705+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:38.145085+0000","last_clean_scrub_stamp":"2023-10-16T15:54:38.145085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:11:16.655726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"245'4","reported_seq":20,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.388161+0000","last_change":"2023-10-16T15:54:38.200929+0000","last_active":"2023-10-16T15:54:53.388161+0000","last_peered":"2023-10-16T15:54:53.388161+0000","last_clean":"2023-10-16T15:54:53.388161+0000","last_became_active":"2023-10-16T15:54:38.200195+0000","last_became_peered":"2023-10-16T15:54:38.200195+0000","last_unstale":"2023-10-16T15:54:53.388161+0000","last_undegraded":"2023-10-16T15:54:53.388161+0000","last_fullsized":"2023-10-16T15:54:53.388161+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:54:37.126822+0000","last_clean_scrub_stamp":"2023-10-16T15:54:37.126822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:20.533246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"231'373","reported_seq":1350,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-16T15:54:53.372019+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:54:53.372019+0000","last_peered":"2023-10-16T15:54:53.372019+0000","last_clean":"2023-10-16T15:54:53.372019+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:54:53.372019+0000","last_undegraded":"2023-10-16T15:54:53.372019+0000","last_fullsized":"2023-10-16T15:54:53.372019+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576019,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65376,"kb_used_data":3660,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306464,"statfs":{"total":96636764160,"available":96569819136,"internally_reserved":0,"allocated":3747840,"data_stored":2145368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.095,"15min":1.171},"min":{"1min":0.61499999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.6779999999999999,"5min":1.794,"15min":1.8879999999999999},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.1040000000000001,"15min":1.149},"min":{"1min":0.71899999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6299999999999999,"5min":1.796,"15min":1.796},"last":1.0409999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1080000000000001,"15min":1.1679999999999999},"min":{"1min":0.82999999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4610000000000001,"5min":1.76,"15min":1.8580000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.244,"15min":1.3460000000000001},"min":{"1min":0.69199999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.365}]},{"osd":2,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3100000000000001,"15min":1.331},"min":{"1min":0.80400000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7410000000000001,"5min":1.893,"15min":2.1269999999999998},"last":1.49},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2310000000000001,"15min":1.296},"min":{"1min":0.877,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6020000000000001,"5min":1.909,"15min":1.95},"last":1.419}]},{"osd":3,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.2529999999999999,"15min":1.3260000000000001},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5149999999999999,"5min":1.843,"15min":1.869},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.3999999999999999,"15min":1.4450000000000001},"min":{"1min":1.089,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.7490000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3819999999999999,"15min":1.383},"min":{"1min":1.054,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6479999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3089999999999999,"15min":1.339},"min":{"1min":0.88100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6439999999999999,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.304}]},{"osd":5,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4570000000000001,"15min":1.46},"min":{"1min":0.87,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.7889999999999999,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.502,"15min":1.5489999999999999},"min":{"1min":0.98599999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.04,"5min":2.04,"15min":2.3540000000000001},"last":1.6379999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.4550000000000001,"15min":1.4750000000000001},"min":{"1min":1.2390000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8260000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.179},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5669999999999999,"15min":1.5800000000000001},"min":{"1min":1.048,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":1.988,"15min":2.2599999999999998},"last":1.6930000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576018,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48160,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323680,"statfs":{"total":96636764160,"available":96587448320,"internally_reserved":0,"allocated":3162112,"data_stored":1555576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.25,"15min":1.214},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.258,"15min":1.2609999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70999999999999996},"max":{"1min":1.5669999999999999,"5min":1.905,"15min":1.905},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3140000000000001,"15min":1.296},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.72199999999999998},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.3049999999999999,"15min":1.3200000000000001},"min":{"1min":0.747,"5min":0.69599999999999995,"15min":0.66600000000000004},"max":{"1min":1.6240000000000001,"5min":1.859,"15min":1.8740000000000001},"last":1.093}]},{"osd":2,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2330000000000001,"15min":1.238},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.831,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.86299999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.905,"5min":1.905,"15min":2.0339999999999998},"last":1.365}]},{"osd":3,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3680000000000001,"15min":1.3919999999999999},"min":{"1min":0.93000000000000005,"5min":0.83899999999999997,"15min":0.77700000000000002},"max":{"1min":1.8320000000000001,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.214},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3149999999999999,"15min":1.3340000000000001},"min":{"1min":0.91000000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.2609999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.409,"15min":1.337},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3359999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.488},{"interface":"front","average":{"1min":1.518,"5min":1.454,"15min":1.4059999999999999},"min":{"1min":1.2,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.573}]},{"osd":5,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.369,"15min":1.3480000000000001},"min":{"1min":1.022,"5min":0.81699999999999995,"15min":0.67300000000000004},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":2.016},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6020000000000001,"15min":1.534},"min":{"1min":1.194,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.6859999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:54:35 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.635,"15min":1.599},"min":{"1min":1.274,"5min":0.627,"15min":0.627},"max":{"1min":2.0110000000000001,"5min":2.2040000000000002,"15min":2.4940000000000002},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.667,"5min":1.6319999999999999,"15min":1.595},"min":{"1min":1.2849999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.028,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.544}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30096,"kb_used_data":3072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341744,"statfs":{"total":96636764160,"available":96605945856,"internally_reserved":0,"allocated":3145728,"data_stored":1554860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.111,"15min":1.1579999999999999},"min":{"1min":0.68899999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.802,"5min":1.8069999999999999,"15min":1.821},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.234,"15min":1.24},"min":{"1min":0.79300000000000004,"5min":0.622,"15min":0.622},"max":{"1min":1.766,"5min":2.738,"15min":2.738},"last":0.86699999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.196,"15min":1.228},"min":{"1min":0.66600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6639999999999999,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.2410000000000001,"15min":1.2889999999999999},"min":{"1min":0.65300000000000002,"5min":0.443,"15min":0.443},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.0580000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.248},"min":{"1min":0.61799999999999999,"5min":0.505,"15min":0.505},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.999},{"interface":"front","average":{"1min":1.325,"5min":1.375,"15min":1.3939999999999999},"min":{"1min":0.77500000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.2160000000000002},"last":1.466}]},{"osd":3,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3300000000000001,"15min":1.335},"min":{"1min":0.753,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8720000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.3260000000000001,"15min":1.3740000000000001},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.966,"5min":1.966,"15min":2.1589999999999998},"last":1.498}]},{"osd":4,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.347},"min":{"1min":0.79900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.8500000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5049999999999999,"15min":1.5149999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0800000000000001,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.554}]},{"osd":6,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4630000000000001,"15min":1.385},"min":{"1min":0.81000000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0369999999999999,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.302},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.468,"15min":1.4319999999999999},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9870000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.2390000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.5289999999999999,"15min":1.532},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.125,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.651,"15min":1.631},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.4340000000000002,"5min":3.109,"15min":3.109},"last":1.573}]}]},{"osd":4,"up_from":13,"seq":55834576019,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70396,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301444,"statfs":{"total":96636764160,"available":96564678656,"internally_reserved":0,"allocated":3186688,"data_stored":1559477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.1419999999999999,"15min":1.194},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.159,"15min":1.194},"min":{"1min":0.76300000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.4059999999999999,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1890000000000001,"15min":1.2310000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.629},"max":{"1min":1.6579999999999999,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.254,"5min":1.284,"15min":1.3089999999999999},"min":{"1min":0.75900000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9750000000000001,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.51}]},{"osd":2,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3939999999999999,"15min":1.4510000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7150000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":2.0899999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.456,"15min":1.492},"min":{"1min":0.82499999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6819999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.667}]},{"osd":3,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.391,"15min":1.385},"min":{"1min":1.02,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8129999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.341,"15min":1.383},"min":{"1min":0.92100000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.7509999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.369}]},{"osd":5,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.371,"15min":1.341},"min":{"1min":0.80200000000000005,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.849,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.944},{"interface":"front","average":{"1min":1.417,"5min":1.4730000000000001,"15min":1.4410000000000001},"min":{"1min":0.88900000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7949999999999999,"5min":2.161,"15min":2.4489999999999998},"last":1.4179999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.389,"15min":1.405},"min":{"1min":0.97699999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5629999999999999,"15min":1.526},"min":{"1min":1.1990000000000001,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.246,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":2.1920000000000002}]},{"osd":7,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6100000000000001,"15min":1.583},"min":{"1min":1.04,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.9730000000000001,"5min":2.399,"15min":2.399},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.6000000000000001,"15min":1.601},"min":{"1min":1.129,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.839}]}]},{"osd":3,"up_from":13,"seq":55834576018,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56488,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315352,"statfs":{"total":96636764160,"available":96578920448,"internally_reserved":0,"allocated":3166208,"data_stored":1557636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92100000000000004,"15min":0.96799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.419,"5min":1.48,"15min":2.0430000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0189999999999999,"15min":1.0580000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.722,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.79900000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94199999999999995,"15min":1.0189999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.776,"5min":1.776,"15min":2.589},"last":0.751},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.147,"15min":1.204},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.556,"5min":1.748,"15min":2.5569999999999999},"last":0.86299999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.304,"15min":1.403},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.302},"last":1.224},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3919999999999999,"15min":1.488},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.954,"5min":1.954,"15min":2.754},"last":1.409}]},{"osd":4,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2669999999999999,"15min":1.3440000000000001},"min":{"1min":0.79100000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9770000000000001,"5min":2.0110000000000001,"15min":2.512},"last":1.528},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3440000000000001,"15min":1.4119999999999999},"min":{"1min":0.76300000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.0249999999999999,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.1220000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.53,"15min":1.5620000000000001},"min":{"1min":0.94099999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2280000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.597},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4890000000000001,"15min":1.5629999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.2599999999999998,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.3080000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.46,"15min":1.5},"min":{"1min":0.97499999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.0680000000000001,"5min":2.173,"15min":2.2090000000000001},"last":1.78},{"interface":"front","average":{"1min":1.585,"5min":1.5649999999999999,"15min":1.6000000000000001},"min":{"1min":0.85299999999999998,"5min":0.749,"15min":0.749},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.7669999999999999},"last":1.389}]},{"osd":7,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.591,"15min":1.6779999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":1.357},{"interface":"front","average":{"1min":1.571,"5min":1.579,"15min":1.6339999999999999},"min":{"1min":1.002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.133,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.6839999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576019,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56472,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315368,"statfs":{"total":96636764160,"available":96578936832,"internally_reserved":0,"allocated":3149824,"data_stored":1554950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.89300000000000002,"15min":0.93700000000000006},"min":{"1min":0.443,"5min":0.378,"15min":0.378},"max":{"1min":1.2749999999999999,"5min":1.726,"15min":1.76},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.093},"min":{"1min":0.39300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7789999999999999,"5min":1.804,"15min":1.8999999999999999},"last":1.1599999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0209999999999999,"15min":1.046},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.504,"5min":1.974,"15min":1.974},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.1479999999999999,"15min":1.224},"min":{"1min":0.55000000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.772,"5min":1.879,"15min":2.0779999999999998},"last":1.4990000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.161,"15min":1.2709999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.5289999999999999,"5min":1.704,"15min":2.1509999999999998},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.3180000000000001,"15min":1.365},"min":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.661,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.3560000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.448,"15min":1.4730000000000001},"min":{"1min":0.84499999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.9299999999999999,"5min":2.335,"15min":2.335},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.345,"15min":1.411},"min":{"1min":0.65900000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.901,"5min":1.915,"15min":2.1469999999999998},"last":1.304}]},{"osd":5,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.403,"15min":1.4730000000000001},"min":{"1min":0.72899999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8080000000000001,"5min":2.657,"15min":2.657},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.5449999999999999,"15min":1.6100000000000001},"min":{"1min":0.81499999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.093,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.6020000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4570000000000001,"15min":1.5429999999999999},"min":{"1min":0.77900000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.861,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5589999999999999,"15min":1.6100000000000001},"min":{"1min":0.89300000000000002,"5min":0.753,"15min":0.753},"max":{"1min":1.8500000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.2569999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.5649999999999999,"15min":1.6499999999999999},"min":{"1min":0.85999999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.234,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.548},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.6499999999999999,"15min":1.722},"min":{"1min":1.0129999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.1890000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.4650000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576019,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48176,"kb_used_data":3108,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323664,"statfs":{"total":96636764160,"available":96587431936,"internally_reserved":0,"allocated":3182592,"data_stored":1558617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94399999999999995,"15min":0.94899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.452,"5min":1.5800000000000001,"15min":1.8420000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0429999999999999,"15min":1.1299999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6619999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98699999999999999,"15min":1.0349999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.4179999999999999,"5min":1.6299999999999999,"15min":2.286},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.109,"5min":1.1919999999999999,"15min":1.2270000000000001},"min":{"1min":0.59199999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.657,"5min":1.7929999999999999,"15min":1.853},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3220000000000001,"15min":1.3680000000000001},"min":{"1min":0.69699999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7290000000000001,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.101,"5min":1.2529999999999999,"15min":1.3129999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8859999999999999,"5min":2.1629999999999998,"15min":2.7839999999999998},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3620000000000001,"15min":1.381},"min":{"1min":0.79300000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.2959999999999998,"5min":2.327,"15min":2.3809999999999998},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.409,"5min":1.512,"15min":1.524},"min":{"1min":0.92000000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.081,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.1279999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.49,"15min":1.5269999999999999},"min":{"1min":0.72499999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0539999999999998,"5min":2.456,"15min":2.456},"last":1.042},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4930000000000001,"15min":1.502},"min":{"1min":1.0069999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.093,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.647,"15min":1.655},"min":{"1min":0.97299999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.218,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.542,"5min":1.6160000000000001,"15min":1.643},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.5059999999999998,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon Oct 16 15:54:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.7030000000000001,"15min":1.744},"min":{"1min":0.80300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.621,"5min":2.734,"15min":3.2090000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.577,"5min":1.647,"15min":1.655},"min":{"1min":0.90100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.452,"5min":2.452,"15min":3.1539999999999999},"last":1.214}]}]},{"osd":0,"up_from":13,"seq":55834576019,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48728,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323112,"statfs":{"total":96636764160,"available":96586866688,"internally_reserved":0,"allocated":3743744,"data_stored":2145262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91000000000000003,"15min":0.94999999999999996},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.2789999999999999,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1180000000000001,"15min":1.2070000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.5800000000000001,"5min":2.036,"15min":2.036},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.1739999999999999,"15min":1.26},"min":{"1min":0.56499999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.117,"15min":1.2170000000000001},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.462,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.107}]},{"osd":3,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.004,"15min":1.032},"min":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6339999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.032,"5min":1.175,"15min":1.2549999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.774,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.319,"15min":1.337},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":2.137},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.401,"15min":1.4970000000000001},"min":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.9390000000000001,"5min":1.99,"15min":2.452},"last":1.4990000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.4239999999999999,"15min":1.476},"min":{"1min":0.66200000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8779999999999999,"5min":2.137,"15min":2.137},"last":1.548},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.345,"15min":1.411},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":2.0859999999999999,"15min":2.282},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5349999999999999,"15min":1.6259999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.7010000000000001},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.526,"15min":1.569},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.1869999999999998,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4279999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.621,"15min":1.6779999999999999},"min":{"1min":0.82199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1480000000000001,"5min":2.339,"15min":2.339},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.5760000000000001,"15min":1.601},"min":{"1min":0.89200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.1259999999999999,"5min":2.1739999999999999,"15min":2.274},"last":1.76}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:54:58.523 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:54:58.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:54:58.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.783+0000 7f20b5bc5640 1 Processor -- start 2023-10-16T15:54:58.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.783+0000 7f20b5bc5640 1 -- start start 2023-10-16T15:54:58.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.785+0000 7f20b5bc5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f20b0070fd0 0x7f20b00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.785+0000 7f20b5bc5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b00b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.785+0000 7f20b5bc5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b0071450 con 0x7f20b00adb50 2023-10-16T15:54:58.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.785+0000 7f20b5bc5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b00cc600 con 0x7f20b0070fd0 2023-10-16T15:54:58.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.785+0000 7f20b5bc5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00cc8e0 con 0x7f20b00adf30 2023-10-16T15:54:58.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20aeffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f20b0070fd0 0x7f20b00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.785+0000 7f20affff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b00b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20af7fe640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7f20b00adb50 0x7f20b006f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:47030/0 (socket says 172.21.15.29:47030) 2023-10-16T15:54:58.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20af7fe640 1 -- 172.21.15.29:0/3635894346 learned_addr learned my addr 172.21.15.29:0/3635894346 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:58.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20aeffd640 1 -- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 msgr2=0x7f20b00b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:58.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20aeffd640 1 --2- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b00b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20aeffd640 1 -- 172.21.15.29:0/3635894346 >> v1:172.21.15.170:6789/0 conn(0x7f20b00adb50 legacy=0x7f20b006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20aeffd640 1 -- 172.21.15.29:0/3635894346 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00ccbc0 con 0x7f20b0070fd0 2023-10-16T15:54:58.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.786+0000 7f20affff640 1 --2- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b00b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:54:58.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.788+0000 7f20aeffd640 1 --2- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b0070fd0 0x7f20b00b0410 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f20980097a0 tx=0x7f209802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d7b5bd7a5c3617 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:58.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.788+0000 7f20ae7fc640 1 -- 172.21.15.29:0/3635894346 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209802d040 con 0x7f20b0070fd0 2023-10-16T15:54:58.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.788+0000 7f20ae7fc640 1 -- 172.21.15.29:0/3635894346 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098031810 con 0x7f20b0070fd0 2023-10-16T15:54:58.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.788+0000 7f20ae7fc640 1 -- 172.21.15.29:0/3635894346 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098031ba0 con 0x7f20b0070fd0 2023-10-16T15:54:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 -- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b0070fd0 msgr2=0x7f20b00b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b0070fd0 0x7f20b00b0410 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f20980097a0 tx=0x7f209802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 -- 172.21.15.29:0/3635894346 shutdown_connections 2023-10-16T15:54:58.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b00b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/3635894346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b0070fd0 0x7f20b00b0410 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 -- 172.21.15.29:0/3635894346 >> 172.21.15.29:0/3635894346 conn(0x7f20b00cb410 msgr2=0x7f20b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:58.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.789+0000 7f20b5bc5640 1 -- 172.21.15.29:0/3635894346 shutdown_connections 2023-10-16T15:54:58.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.790+0000 7f20b5bc5640 1 -- 172.21.15.29:0/3635894346 wait complete. 2023-10-16T15:54:58.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.792+0000 7f20b5bc5640 1 Processor -- start 2023-10-16T15:54:58.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.792+0000 7f20b5bc5640 1 -- start start 2023-10-16T15:54:58.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20b5bc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 0x7f20b01f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20b5bc5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b00adb50 0x7f20b00c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20b5bc5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b015eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20b5bc5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00c3450 con 0x7f20b0070fd0 2023-10-16T15:54:58.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20b5bc5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00b3f40 con 0x7f20b00adb50 2023-10-16T15:54:58.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20b5bc5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b01f7f20 con 0x7f20b00adf30 2023-10-16T15:54:58.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20af7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 0x7f20b01f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20af7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 0x7f20b01f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36380/0 (socket says 172.21.15.29:36380) 2023-10-16T15:54:58.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20af7fe640 1 -- 172.21.15.29:0/2964614348 learned_addr learned my addr 172.21.15.29:0/2964614348 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:58.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.794+0000 7f20aeffd640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b00adb50 0x7f20b00c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20affff640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b015eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20af7fe640 1 -- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 msgr2=0x7f20b015eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20af7fe640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b015eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20af7fe640 1 -- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b00adb50 msgr2=0x7f20b00c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20af7fe640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b00adb50 0x7f20b00c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20af7fe640 1 -- 172.21.15.29:0/2964614348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8009400 con 0x7f20b0070fd0 2023-10-16T15:54:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20af7fe640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 0x7f20b01f79e0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f20a000cba0 tx=0x7f20a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.795+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a000ad40 con 0x7f20b0070fd0 2023-10-16T15:54:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.796+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0012400 con 0x7f20b0070fd0 2023-10-16T15:54:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.796+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2098009400 con 0x7f20b0070fd0 2023-10-16T15:54:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.796+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a000aea0 con 0x7f20b0070fd0 2023-10-16T15:54:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.796+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b00c2ab0 con 0x7f20b0070fd0 2023-10-16T15:54:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.797+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b00c3590 con 0x7f20b0070fd0 2023-10-16T15:54:58.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.798+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f20a0007070 con 0x7f20b0070fd0 2023-10-16T15:54:58.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.799+0000 7f20a7fff640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f209006b0b0 0x7f209006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:58.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.799+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f20a0089c80 con 0x7f20b0070fd0 2023-10-16T15:54:58.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.800+0000 7f20aeffd640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f209006b0b0 0x7f209006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:58.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.801+0000 7f20aeffd640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f209006b0b0 0x7f209006d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2098036000 tx=0x7f209803c040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:58.803+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f20b00c3590 con 0x7f20b0070fd0 2023-10-16T15:54:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.039+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20b00c3590 con 0x7f20b0070fd0 2023-10-16T15:54:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.040+0000 7f20a7fff640 1 -- 172.21.15.29:0/2964614348 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f20b00c3590 con 0x7f20b0070fd0 2023-10-16T15:54:59.042 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:54:59.042 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:54:59.042 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 130 2023-10-16T15:54:59.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f209006b0b0 msgr2=0x7f209006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f209006b0b0 0x7f209006d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2098036000 tx=0x7f209803c040 comp rx=0 tx=0).stop 2023-10-16T15:54:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 msgr2=0x7f20b01f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 0x7f20b01f79e0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f20a000cba0 tx=0x7f20a000a430 comp rx=0 tx=0).stop 2023-10-16T15:54:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 shutdown_connections 2023-10-16T15:54:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f209006b0b0 0x7f209006d570 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f20b00adf30 0x7f20b015eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20b00adb50 0x7f20b00c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 --2- 172.21.15.29:0/2964614348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20b0070fd0 0x7f20b01f79e0 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.043+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 >> 172.21.15.29:0/2964614348 conn(0x7f20b00cb410 msgr2=0x7f20b0160e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.044+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 shutdown_connections 2023-10-16T15:54:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.044+0000 7f20b5bc5640 1 -- 172.21.15.29:0/2964614348 wait complete. 2023-10-16T15:54:59.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:54:52.836357+0000', 'modified': '2023-10-16T15:54:55.930651+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11613}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11613': {'gid': 11613, 'name': 'c', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.29:6834/2428945430', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 2428945430}, {'type': 'v1', 'addr': '172.21.15.29:6834', 'nonce': 2428945430}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:54:59.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:54:59.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T15:54:59.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.704+0000 7f9d0eb0c640 1 Processor -- start 2023-10-16T15:54:59.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.704+0000 7f9d0eb0c640 1 -- start start 2023-10-16T15:54:59.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.705+0000 7f9d0eb0c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d0806c040 0x7f9d0806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.705+0000 7f9d0eb0c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9d080cda80 0x7f9d0815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.705+0000 7f9d0eb0c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0806c960 con 0x7f9d080ad970 2023-10-16T15:54:59.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.705+0000 7f9d0eb0c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0815b9d0 con 0x7f9d080cda80 2023-10-16T15:54:59.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.705+0000 7f9d0eb0c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815bcb0 con 0x7f9d0806c040 2023-10-16T15:54:59.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.706+0000 7f9d0d309640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9d080ad970 0x7f9d0815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33292/0 (socket says 172.21.15.29:33292) 2023-10-16T15:54:59.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.706+0000 7f9d0d309640 1 -- 172.21.15.29:0/2963099592 learned_addr learned my addr 172.21.15.29:0/2963099592 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:54:59.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.706+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d0806c040 0x7f9d0806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.706+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/2963099592 >> v2:172.21.15.170:3300/0 conn(0x7f9d080cda80 0x7f9d0815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.706+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174639407 0 0) 0x7f9d0806c960 con 0x7f9d080ad970 2023-10-16T15:54:59.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.706+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf4003520 con 0x7f9d080ad970 2023-10-16T15:54:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717679634 0 0) 0x7f9cf4003520 con 0x7f9d080ad970 2023-10-16T15:54:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d0806c040 msgr2=0x7f9d0806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d0806c040 0x7f9d0806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 >> v2:172.21.15.170:3300/0 conn(0x7f9d080cda80 msgr2=0x7f9d0815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 --2- 172.21.15.29:0/2963099592 >> v2:172.21.15.170:3300/0 conn(0x7f9d080cda80 0x7f9d0815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0815bf90 con 0x7f9d080ad970 2023-10-16T15:54:59.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f9cf8001670 con 0x7f9d080ad970 2023-10-16T15:54:59.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 >> v1:172.21.15.29:6789/0 conn(0x7f9d080ad970 legacy=0x7f9d0815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9cf4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cf40047c0 0x7f9cf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cf4005180 0x7f9cf400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815b9d0 con 0x7f9cf4003970 2023-10-16T15:54:59.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0815bcb0 con 0x7f9cf40047c0 2023-10-16T15:54:59.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf400bff0 con 0x7f9cf4005180 2023-10-16T15:54:59.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/2963099592 >> v2:172.21.15.170:3300/0 conn(0x7f9d080cda80 0x7f9d0815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:59.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d0806c040 0x7f9d0806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:59.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.707+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9cf4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0d309640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cf40047c0 0x7f9cf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cf4005180 0x7f9cf400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0db0a640 1 -- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cf4005180 msgr2=0x7f9cf400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9cf4005180 0x7f9cf400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0db0a640 1 -- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cf40047c0 msgr2=0x7f9cf4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cf40047c0 0x7f9cf4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0db0a640 1 -- 172.21.15.29:0/2963099592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4003520 con 0x7f9cf4003970 2023-10-16T15:54:59.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.708+0000 7f9d0d309640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cf40047c0 0x7f9cf4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:59.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.709+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9cf4003d90 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f9d04006fd0 tx=0x7f9cf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337207fab90c77a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.709+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04033600 con 0x7f9cf4003970 2023-10-16T15:54:59.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.709+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0402f050 con 0x7f9cf4003970 2023-10-16T15:54:59.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.709+0000 7f9d0cb08640 1 -- 172.21.15.29:0/2963099592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04033980 con 0x7f9cf4003970 2023-10-16T15:54:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/2963099592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 msgr2=0x7f9cf4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/2963099592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9cf4003d90 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f9d04006fd0 tx=0x7f9cf40057f0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0e30b640 1 -- 172.21.15.29:0/2963099592 reap_dead start 2023-10-16T15:54:59.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/2963099592 shutdown_connections 2023-10-16T15:54:59.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/2963099592 >> 172.21.15.29:0/2963099592 conn(0x7f9d080ce6a0 msgr2=0x7f9d080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/2963099592 shutdown_connections 2023-10-16T15:54:59.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.710+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/2963099592 wait complete. 2023-10-16T15:54:59.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.712+0000 7f9d0eb0c640 1 Processor -- start 2023-10-16T15:54:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.712+0000 7f9d0eb0c640 1 -- start start 2023-10-16T15:54:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0eb0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9d08156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0eb0c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d080b27d0 0x7f9d080b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0eb0c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d080c78a0 0x7f9d080c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0eb0c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c8290 con 0x7f9cf4003970 2023-10-16T15:54:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0eb0c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080b3190 con 0x7f9d080b27d0 2023-10-16T15:54:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0eb0c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081f68a0 con 0x7f9d080c78a0 2023-10-16T15:54:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0db0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9d08156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0db0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9d08156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36402/0 (socket says 172.21.15.29:36402) 2023-10-16T15:54:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.713+0000 7f9d0db0a640 1 -- 172.21.15.29:0/1234914061 learned_addr learned my addr 172.21.15.29:0/1234914061 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d080c78a0 0x7f9d080c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0d309640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d080b27d0 0x7f9d080b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0db0a640 1 -- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d080c78a0 msgr2=0x7f9d080c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d080c78a0 0x7f9d080c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0db0a640 1 -- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d080b27d0 msgr2=0x7f9d080b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d080b27d0 0x7f9d080b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0db0a640 1 -- 172.21.15.29:0/1234914061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d081f2460 con 0x7f9cf4003970 2023-10-16T15:54:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0d309640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d080b27d0 0x7f9d080b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9d08156bc0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f9d0402bdc0 tx=0x7f9d04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.714+0000 7f9cfe7fc640 1 -- 172.21.15.29:0/1234914061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04032830 con 0x7f9cf4003970 2023-10-16T15:54:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.715+0000 7f9cfe7fc640 1 -- 172.21.15.29:0/1234914061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04032990 con 0x7f9cf4003970 2023-10-16T15:54:59.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.715+0000 7f9cfe7fc640 1 -- 172.21.15.29:0/1234914061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04032d20 con 0x7f9cf4003970 2023-10-16T15:54:59.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.715+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d081f2740 con 0x7f9cf4003970 2023-10-16T15:54:59.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.715+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081f2d00 con 0x7f9cf4003970 2023-10-16T15:54:59.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.717+0000 7f9cfe7fc640 1 -- 172.21.15.29:0/1234914061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9d0403d410 con 0x7f9cf4003970 2023-10-16T15:54:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.718+0000 7f9cfe7fc640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9cec06b0e0 0x7f9cec06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.719+0000 7f9cfe7fc640 1 -- 172.21.15.29:0/1234914061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f9d040abe90 con 0x7f9cf4003970 2023-10-16T15:54:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.719+0000 7f9d129dd740 1 Processor -- start 2023-10-16T15:54:59.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.719+0000 7f9d0d309640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9cec06b0e0 0x7f9cec06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.719+0000 7f9d129dd740 1 -- start start 2023-10-16T15:54:59.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 0x55b6cdf53070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80550 0x55b6cdf82f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf535b0 0x55b6cdf83440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d129dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cde737a0 con 0x55b6cdf80170 2023-10-16T15:54:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdec5740 con 0x55b6cdf80550 2023-10-16T15:54:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdec2e30 con 0x55b6cdf535b0 2023-10-16T15:54:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.720+0000 7f9d0d309640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9cec06b0e0 0x7f9cec06d5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8001eb0 tx=0x7f9cf8009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 0x55b6cdf53070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 0x55b6cdf53070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36410/0 (socket says 172.21.15.29:36410) 2023-10-16T15:54:59.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 -- 172.21.15.29:0/460689179 learned_addr learned my addr 172.21.15.29:0/460689179 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0d309640 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80550 0x55b6cdf82f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf535b0 0x55b6cdf83440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 -- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf535b0 msgr2=0x55b6cdf83440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf535b0 0x55b6cdf83440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 -- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80550 msgr2=0x55b6cdf82f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80550 0x55b6cdf82f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 -- 172.21.15.29:0/460689179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6cde82910 con 0x55b6cdf80170 2023-10-16T15:54:59.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0d309640 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80550 0x55b6cdf82f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:54:59.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.721+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 0x55b6cdf53070 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f9d0402bfd0 tx=0x7f9d04079f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc5c99d5d5996e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.722+0000 7f9ceb7fe640 1 -- 172.21.15.29:0/460689179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04082740 con 0x55b6cdf80170 2023-10-16T15:54:59.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.722+0000 7f9ceb7fe640 1 -- 172.21.15.29:0/460689179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04076c00 con 0x55b6cdf80170 2023-10-16T15:54:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.722+0000 7f9ceb7fe640 1 -- 172.21.15.29:0/460689179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0407a990 con 0x55b6cdf80170 2023-10-16T15:54:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.722+0000 7f9d129dd740 1 -- 172.21.15.29:0/460689179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 msgr2=0x55b6cdf53070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.722+0000 7f9d129dd740 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 0x55b6cdf53070 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f9d0402bfd0 tx=0x7f9d04079f80 comp rx=0 tx=0).stop 2023-10-16T15:54:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 -- 172.21.15.29:0/460689179 shutdown_connections 2023-10-16T15:54:59.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf535b0 0x55b6cdf83440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80550 0x55b6cdf82f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 --2- 172.21.15.29:0/460689179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80170 0x55b6cdf53070 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 -- 172.21.15.29:0/460689179 >> 172.21.15.29:0/460689179 conn(0x55b6cdf55380 msgr2=0x55b6cdf9df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 -- 172.21.15.29:0/460689179 shutdown_connections 2023-10-16T15:54:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 -- 172.21.15.29:0/460689179 wait complete. 2023-10-16T15:54:59.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.723+0000 7f9d129dd740 1 -- start start 2023-10-16T15:54:59.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.724+0000 7f9d129dd740 1 Processor -- start 2023-10-16T15:54:59.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdf74430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80550 0x55b6cdfadae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 0x55b6cdfae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdebe720 con 0x55b6cdf98530 2023-10-16T15:54:59.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cde5c970 con 0x55b6cdf80170 2023-10-16T15:54:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6cde5c610 con 0x55b6cdf80550 2023-10-16T15:54:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d129dd740 10 client.0 _ll_register_callbacks cb 0x55b6cde7e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:54:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d0e30b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 0x55b6cdfae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d0e30b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 0x55b6cdfae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36420/0 (socket says 172.21.15.29:36420) 2023-10-16T15:54:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d0e30b640 1 -- 172.21.15.29:0/873178003 learned_addr learned my addr 172.21.15.29:0/873178003 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdf74430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.725+0000 7f9d0d309640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80550 0x55b6cdfadae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d0e30b640 1 -- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80550 msgr2=0x55b6cdfadae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80550 0x55b6cdfadae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d0e30b640 1 -- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 msgr2=0x55b6cdf74430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdf74430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d0e30b640 1 -- 172.21.15.29:0/873178003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081f2d00 con 0x55b6cdf98530 2023-10-16T15:54:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d0e30b640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 0x55b6cdfae020 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f9d00007790 tx=0x7f9d00007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de85c9fe490d13cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9ce8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:54:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/873178003 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00011730 con 0x55b6cdf98530 2023-10-16T15:54:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/873178003 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f9d00011890 con 0x55b6cdf98530 2023-10-16T15:54:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.726+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6cdee5160 con 0x55b6cdf98530 2023-10-16T15:54:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/873178003 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d00011a50 con 0x55b6cdf98530 2023-10-16T15:54:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 10 client.11898 fetch_fsmap learned FSMap version 130 2023-10-16T15:54:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55b6cdebcda0 con 0x55b6cdf98530 2023-10-16T15:54:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/873178003 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b6cdee5160 con 0x55b6cdf98530 2023-10-16T15:54:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/873178003 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 130) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f9d00014070 con 0x55b6cdf98530 2023-10-16T15:54:59.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 10 client.11898 fetch_fsmap finished waiting for FSMap version 130 2023-10-16T15:54:59.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 10 client.11898 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:54:59.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 20 client.11898 populate_metadata read hostname 'smithi029' 2023-10-16T15:54:59.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf86660 0x55b6cdfae560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 4 client.11898 mds_command: new command op to 11613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:54:59.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.727+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 --> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6cde90cc0 con 0x55b6cdf86660 2023-10-16T15:54:59.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.728+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf86660 0x55b6cdfae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.728+0000 7f9ce8ff9640 10 client.11898 ms_handle_connect on v2:172.21.15.29:6832/2428945430 2023-10-16T15:54:59.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.728+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf86660 0x55b6cdfae560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b0ee65eb232faef server_cookie=4f7ea0f746fa862f in_seq=0 out_seq=0 2023-10-16T15:54:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/873178003 <== mds.0 v2:172.21.15.29:6832/2428945430 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b6cde90cc0 con 0x55b6cdf86660 2023-10-16T15:54:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9ce8ff9640 10 client.11898 handle_command_reply: tid=0 2023-10-16T15:54:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 client.11898 shutdown 2023-10-16T15:54:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 msgr2=0x55b6cdfae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 0x55b6cdfae020 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f9d00007790 tx=0x7f9d00007c50 comp rx=0 tx=0).stop 2023-10-16T15:54:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 shutdown_connections 2023-10-16T15:54:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf98530 0x55b6cdfae020 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf86660 0x55b6cdfae560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80550 0x55b6cdfadae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 --2- 172.21.15.29:0/873178003 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdf74430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 >> 172.21.15.29:0/873178003 conn(0x55b6cdf535b0 msgr2=0x55b6cdf56f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.729+0000 7f9ce8ff9640 0 client.11898 ms_handle_reset on v2:172.21.15.29:6832/2428945430 2023-10-16T15:54:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.730+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 shutdown_connections 2023-10-16T15:54:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.730+0000 7f9d129dd740 1 -- 172.21.15.29:0/873178003 wait complete. 2023-10-16T15:54:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.730+0000 7f9d129dd740 20 client.11898 trim_cache size 0 max 16384 2023-10-16T15:54:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 Processor -- start 2023-10-16T15:54:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 -- start start 2023-10-16T15:54:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdfa0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdf543e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf98530 0x55b6cdf520f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdf57c70 con 0x55b6cdf80550 2023-10-16T15:54:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdc22f20 con 0x55b6cdf80170 2023-10-16T15:54:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.747+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdf59830 con 0x55b6cdf98530 2023-10-16T15:54:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdf543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdf543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36434/0 (socket says 172.21.15.29:36434) 2023-10-16T15:54:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 -- 172.21.15.29:0/2861301140 learned_addr learned my addr 172.21.15.29:0/2861301140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdfa0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 -- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf98530 msgr2=0x55b6cdf520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf98530 0x55b6cdf520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 -- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 msgr2=0x55b6cdfa0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdfa0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0d309640 1 -- 172.21.15.29:0/2861301140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6cde5bb80 con 0x55b6cdf80550 2023-10-16T15:54:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.748+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdfa0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:54:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.749+0000 7f9d0d309640 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdf543e0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800ea10 tx=0x7f9cf800c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d8d4b34d887446 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.749+0000 7f9cebfff640 1 -- 172.21.15.29:0/2861301140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800c070 con 0x55b6cdf80550 2023-10-16T15:54:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.749+0000 7f9cebfff640 1 -- 172.21.15.29:0/2861301140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf800d040 con 0x55b6cdf80550 2023-10-16T15:54:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.749+0000 7f9cebfff640 1 -- 172.21.15.29:0/2861301140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8007450 con 0x55b6cdf80550 2023-10-16T15:54:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 -- 172.21.15.29:0/2861301140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 msgr2=0x55b6cdf543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdf543e0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800ea10 tx=0x7f9cf800c000 comp rx=0 tx=0).stop 2023-10-16T15:54:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 -- 172.21.15.29:0/2861301140 shutdown_connections 2023-10-16T15:54:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdf543e0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf98530 0x55b6cdf520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 --2- 172.21.15.29:0/2861301140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf80170 0x55b6cdfa0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 -- 172.21.15.29:0/2861301140 >> 172.21.15.29:0/2861301140 conn(0x55b6cdf535b0 msgr2=0x55b6cdf94a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 -- 172.21.15.29:0/2861301140 shutdown_connections 2023-10-16T15:54:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 -- 172.21.15.29:0/2861301140 wait complete. 2023-10-16T15:54:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.750+0000 7f9d129dd740 1 -- start start 2023-10-16T15:54:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.751+0000 7f9d129dd740 1 Processor -- start 2023-10-16T15:54:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80170 0x55b6ce003640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdfa0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf98530 0x55b6cdfa09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdf58050 con 0x55b6cdf80550 2023-10-16T15:54:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ce003b80 con 0x55b6cdf98530 2023-10-16T15:54:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6cdf57e40 con 0x55b6cdf80170 2023-10-16T15:54:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d129dd740 10 client.0 _ll_register_callbacks cb 0x7f9d00009750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:54:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdfa0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdfa0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36440/0 (socket says 172.21.15.29:36440) 2023-10-16T15:54:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 -- 172.21.15.29:0/4104447711 learned_addr learned my addr 172.21.15.29:0/4104447711 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:54:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 -- 172.21.15.29:0/4104447711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80170 msgr2=0x55b6ce003640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80170 0x55b6ce003640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 -- 172.21.15.29:0/4104447711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf98530 msgr2=0x55b6cdfa09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:54:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf98530 0x55b6cdfa09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.752+0000 7f9d0d309640 1 -- 172.21.15.29:0/4104447711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0815bf90 con 0x55b6cdf80550 2023-10-16T15:54:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9d0d309640 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdfa0460 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8015040 tx=0x7f9cf80019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0d1d722fdc7c1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:54:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9ce8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:54:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/4104447711 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800c050 con 0x55b6cdf80550 2023-10-16T15:54:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/4104447711 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f9cf8005e30 con 0x55b6cdf80550 2023-10-16T15:54:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d04076c00 con 0x55b6cdf80550 2023-10-16T15:54:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/4104447711 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cf8013690 con 0x55b6cdf80550 2023-10-16T15:54:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9d129dd740 10 client.11904 fetch_fsmap learned FSMap version 130 2023-10-16T15:54:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.753+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55b6cde81170 con 0x55b6cdf80550 2023-10-16T15:54:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/4104447711 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04076c00 con 0x55b6cdf80550 2023-10-16T15:54:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/4104447711 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 130) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f9cf80137f0 con 0x55b6cdf80550 2023-10-16T15:54:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d129dd740 10 client.11904 fetch_fsmap finished waiting for FSMap version 130 2023-10-16T15:54:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d129dd740 10 client.11904 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T15:54:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d129dd740 20 client.11904 populate_metadata read hostname 'smithi029' 2023-10-16T15:54:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d129dd740 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf9f880 0x55b6cdfb6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:54:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d129dd740 4 client.11904 mds_command: new command op to 11613 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:54:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 --> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b6cdeb7700 con 0x55b6cdf9f880 2023-10-16T15:54:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.754+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf9f880 0x55b6cdfb6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:54:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d0db0a640 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf9f880 0x55b6cdfb6e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6479e684ec341f1b server_cookie=51bf1ab5ebea350f in_seq=0 out_seq=0 2023-10-16T15:54:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9ce8ff9640 10 client.11904 ms_handle_connect on v2:172.21.15.29:6832/2428945430 2023-10-16T15:54:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9ce8ff9640 1 -- 172.21.15.29:0/4104447711 <== mds.0 v2:172.21.15.29:6832/2428945430 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x55b6cdeb7700 con 0x55b6cdf9f880 2023-10-16T15:54:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9ce8ff9640 10 client.11904 handle_command_reply: tid=0 2023-10-16T15:54:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 client.11904 shutdown 2023-10-16T15:54:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 msgr2=0x55b6cdfa0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdfa0460 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8015040 tx=0x7f9cf80019a0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 shutdown_connections 2023-10-16T15:54:59.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b6cdf80550 0x55b6cdfa0460 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.29:6832/2428945430,v1:172.21.15.29:6834/2428945430] conn(0x55b6cdf9f880 0x55b6cdfb6e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b6cdf98530 0x55b6cdfa09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 --2- 172.21.15.29:0/4104447711 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55b6cdf80170 0x55b6ce003640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9ce8ff9640 0 client.11904 ms_handle_reset on v2:172.21.15.29:6832/2428945430 2023-10-16T15:54:59.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 >> 172.21.15.29:0/4104447711 conn(0x55b6cdf535b0 msgr2=0x55b6cdf4e810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 shutdown_connections 2023-10-16T15:54:59.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.755+0000 7f9d129dd740 1 -- 172.21.15.29:0/4104447711 wait complete. 2023-10-16T15:54:59.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d129dd740 20 client.11904 trim_cache size 0 max 16384 2023-10-16T15:54:59.767 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:54:59.768 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:54:59.768 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:54:59.768 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 11613, 2023-10-16T15:54:59.768 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:54:59.768 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:54:59.768 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "backup_fs", 2023-10-16T15:54:59.769 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.3638433609999998, 2023-10-16T15:54:59.769 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 130, 2023-10-16T15:54:59.769 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 250, 2023-10-16T15:54:59.769 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 250, 2023-10-16T15:54:59.769 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 27.412589691000001 2023-10-16T15:54:59.769 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:54:59.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9cec06b0e0 msgr2=0x7f9cec06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9cec06b0e0 0x7f9cec06d5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8001eb0 tx=0x7f9cf8009040 comp rx=0 tx=0).stop 2023-10-16T15:54:59.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 msgr2=0x7f9d08156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:54:59.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9d08156bc0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f9d0402bdc0 tx=0x7f9d04004730 comp rx=0 tx=0).stop 2023-10-16T15:54:59.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 shutdown_connections 2023-10-16T15:54:59.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f9cec06b0e0 0x7f9cec06d5a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d080c78a0 0x7f9d080c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d080b27d0 0x7f9d080b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 --2- 172.21.15.29:0/1234914061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cf4003970 0x7f9d08156bc0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:54:59.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 >> 172.21.15.29:0/1234914061 conn(0x7f9d080ce6a0 msgr2=0x7f9d080ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:54:59.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 shutdown_connections 2023-10-16T15:54:59.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:54:59.756+0000 7f9d0eb0c640 1 -- 172.21.15.29:0/1234914061 wait complete. 2023-10-16T15:54:59.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T15:55:00.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.000+0000 7fa874215640 1 Processor -- start 2023-10-16T15:55:00.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.000+0000 7fa874215640 1 -- start start 2023-10-16T15:55:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.001+0000 7fa874215640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa86c06b8c0 0x7fa86c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.001+0000 7fa874215640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa86c06b1a0 0x7fa86c0ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.001+0000 7fa874215640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c0cd390 con 0x7fa86c15d130 2023-10-16T15:55:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.001+0000 7fa874215640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c06c1e0 con 0x7fa86c06b1a0 2023-10-16T15:55:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.001+0000 7fa874215640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c0cc2f0 con 0x7fa86c06b8c0 2023-10-16T15:55:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8727b2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa86c06b1a0 0x7fa86c0ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8717b0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa86c15d130 0x7fa86c0cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33302/0 (socket says 172.21.15.29:33302) 2023-10-16T15:55:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8717b0640 1 -- 172.21.15.29:0/2960823758 learned_addr learned my addr 172.21.15.29:0/2960823758 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa871fb1640 1 --2- 172.21.15.29:0/2960823758 >> v2:172.21.15.170:3300/0 conn(0x7fa86c06b8c0 0x7fa86c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8727b2640 1 -- 172.21.15.29:0/2960823758 >> v2:172.21.15.170:3300/0 conn(0x7fa86c06b8c0 msgr2=0x7fa86c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8727b2640 1 --2- 172.21.15.29:0/2960823758 >> v2:172.21.15.170:3300/0 conn(0x7fa86c06b8c0 0x7fa86c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8727b2640 1 -- 172.21.15.29:0/2960823758 >> v1:172.21.15.29:6789/0 conn(0x7fa86c15d130 legacy=0x7fa86c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa8727b2640 1 -- 172.21.15.29:0/2960823758 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0cc5d0 con 0x7fa86c06b1a0 2023-10-16T15:55:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa870faf640 1 -- 172.21.15.29:0/2960823758 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133089076 0 0) 0x7fa86c0cd390 con 0x7fa86c15d130 2023-10-16T15:55:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.002+0000 7fa871fb1640 1 --2- 172.21.15.29:0/2960823758 >> v2:172.21.15.170:3300/0 conn(0x7fa86c06b8c0 0x7fa86c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.003+0000 7fa8727b2640 1 --2- 172.21.15.29:0/2960823758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c0ad6c0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fa86c06ad90 tx=0x7fa86402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2580f061d272ddc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.003+0000 7fa870faf640 1 -- 172.21.15.29:0/2960823758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864031830 con 0x7fa86c06b1a0 2023-10-16T15:55:00.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa870faf640 1 -- 172.21.15.29:0/2960823758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864031990 con 0x7fa86c06b1a0 2023-10-16T15:55:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa870faf640 1 -- 172.21.15.29:0/2960823758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864031d20 con 0x7fa86c06b1a0 2023-10-16T15:55:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 -- 172.21.15.29:0/2960823758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 msgr2=0x7fa86c0ad6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 --2- 172.21.15.29:0/2960823758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c0ad6c0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fa86c06ad90 tx=0x7fa86402b8d0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 -- 172.21.15.29:0/2960823758 shutdown_connections 2023-10-16T15:55:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 --2- 172.21.15.29:0/2960823758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c0ad6c0 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 --2- 172.21.15.29:0/2960823758 >> v2:172.21.15.170:3300/0 conn(0x7fa86c06b8c0 0x7fa86c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 -- 172.21.15.29:0/2960823758 >> 172.21.15.29:0/2960823758 conn(0x7fa86c0d0ad0 msgr2=0x7fa86c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:00.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 -- 172.21.15.29:0/2960823758 shutdown_connections 2023-10-16T15:55:00.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.004+0000 7fa874215640 1 -- 172.21.15.29:0/2960823758 wait complete. 2023-10-16T15:55:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.006+0000 7fa874215640 1 Processor -- start 2023-10-16T15:55:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.006+0000 7fa874215640 1 -- start start 2023-10-16T15:55:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa874215640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa874215640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c06b8c0 0x7fa86c15ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa874215640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa86c15d130 0x7fa86c06e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa874215640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1f1fc0 con 0x7fa86c06b1a0 2023-10-16T15:55:00.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa874215640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1f2240 con 0x7fa86c06b8c0 2023-10-16T15:55:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa874215640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1f2520 con 0x7fa86c15d130 2023-10-16T15:55:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.007+0000 7fa871fb1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c154080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36470/0 (socket says 172.21.15.29:36470) 2023-10-16T15:55:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 -- 172.21.15.29:0/771984140 learned_addr learned my addr 172.21.15.29:0/771984140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:00.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa8717b0640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c06b8c0 0x7fa86c15ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa8727b2640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa86c15d130 0x7fa86c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 -- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa86c15d130 msgr2=0x7fa86c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa86c15d130 0x7fa86c06e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 -- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c06b8c0 msgr2=0x7fa86c15ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c06b8c0 0x7fa86c15ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 -- 172.21.15.29:0/771984140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa864009400 con 0x7fa86c06b1a0 2023-10-16T15:55:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.008+0000 7fa871fb1640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c154080 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fa86800d100 tx=0x7fa86800d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.009+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868003b90 con 0x7fa86c06b1a0 2023-10-16T15:55:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.009+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86800f040 con 0x7fa86c06b1a0 2023-10-16T15:55:00.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.009+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868004030 con 0x7fa86c06b1a0 2023-10-16T15:55:00.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.009+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c1f2800 con 0x7fa86c06b1a0 2023-10-16T15:55:00.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.009+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c15f010 con 0x7fa86c06b1a0 2023-10-16T15:55:00.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.010+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c04e160 con 0x7fa86c06b1a0 2023-10-16T15:55:00.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.011+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa868004190 con 0x7fa86c06b1a0 2023-10-16T15:55:00.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.012+0000 7fa8627fc640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa85006b0b0 0x7fa85006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.012+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fa86808a2f0 con 0x7fa86c06b1a0 2023-10-16T15:55:00.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.013+0000 7fa8717b0640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa85006b0b0 0x7fa85006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.014+0000 7fa8717b0640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa85006b0b0 0x7fa85006d570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa86c15c210 tx=0x7fa85c007bd0 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:00.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.016+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa868054bc0 con 0x7fa86c06b1a0 2023-10-16T15:55:00.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.233+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 --> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa86c1f23b0 con 0x7fa85006b0b0 2023-10-16T15:55:00.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.236+0000 7fa8627fc640 1 -- 172.21.15.29:0/771984140 <== mgr.9022 v2:172.21.15.170:6832/58791103 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fa86c1f23b0 con 0x7fa85006b0b0 2023-10-16T15:55:00.237 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:55:00.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:00.238 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T15:55:00.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:00.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:00.239 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "backup_fs" 2023-10-16T15:55:00.239 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:00.239 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:55:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa85006b0b0 msgr2=0x7fa85006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa85006b0b0 0x7fa85006d570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa86c15c210 tx=0x7fa85c007bd0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 msgr2=0x7fa86c154080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c154080 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fa86800d100 tx=0x7fa86800d5c0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 shutdown_connections 2023-10-16T15:55:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fa85006b0b0 0x7fa85006d570 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa86c15d130 0x7fa86c06e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c06b8c0 0x7fa86c15ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 --2- 172.21.15.29:0/771984140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c06b1a0 0x7fa86c154080 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 >> 172.21.15.29:0/771984140 conn(0x7fa86c0d0ad0 msgr2=0x7fa86c15c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:00.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 shutdown_connections 2023-10-16T15:55:00.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.239+0000 7fa874215640 1 -- 172.21.15.29:0/771984140 wait complete. 2023-10-16T15:55:00.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T15:55:00.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.488+0000 7ff7fd773640 1 Processor -- start 2023-10-16T15:55:00.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.488+0000 7ff7fd773640 1 -- start start 2023-10-16T15:55:00.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.489+0000 7ff7fd773640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f806fb50 0x7ff7f80cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.489+0000 7ff7fd773640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff7f806c9e0 0x7ff7f80ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.489+0000 7ff7fd773640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80ca560 con 0x7ff7f80c9060 2023-10-16T15:55:00.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.489+0000 7ff7fd773640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80c9950 con 0x7ff7f806c9e0 2023-10-16T15:55:00.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.489+0000 7ff7fd773640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80c9c30 con 0x7ff7f806fb50 2023-10-16T15:55:00.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.490+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f806fb50 0x7ff7f80cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.490+0000 7ff7f7fff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7ff7f80c9060 0x7ff7f806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:47042/0 (socket says 172.21.15.29:47042) 2023-10-16T15:55:00.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.490+0000 7ff7f7fff640 1 -- 172.21.15.29:0/2217562433 learned_addr learned my addr 172.21.15.29:0/2217562433 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:00.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.490+0000 7ff7fcf72640 1 --2- 172.21.15.29:0/2217562433 >> v2:172.21.15.170:3300/0 conn(0x7ff7f806c9e0 0x7ff7f80ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.491+0000 7ff7f6ffd640 1 -- 172.21.15.29:0/2217562433 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516588216 0 0) 0x7ff7f80ca560 con 0x7ff7f80c9060 2023-10-16T15:55:00.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.491+0000 7ff7f6ffd640 1 -- 172.21.15.29:0/2217562433 --> v1:172.21.15.170:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0003520 con 0x7ff7f80c9060 2023-10-16T15:55:00.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.491+0000 7ff7f77fe640 1 -- 172.21.15.29:0/2217562433 >> v2:172.21.15.170:3300/0 conn(0x7ff7f806c9e0 msgr2=0x7ff7f80ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:00.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.491+0000 7ff7f77fe640 1 --2- 172.21.15.29:0/2217562433 >> v2:172.21.15.170:3300/0 conn(0x7ff7f806c9e0 0x7ff7f80ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.491+0000 7ff7f77fe640 1 -- 172.21.15.29:0/2217562433 >> v1:172.21.15.170:6789/0 conn(0x7ff7f80c9060 legacy=0x7ff7f806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.491+0000 7ff7f77fe640 1 -- 172.21.15.29:0/2217562433 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f80c9f10 con 0x7ff7f806fb50 2023-10-16T15:55:00.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.493+0000 7ff7f77fe640 1 --2- 172.21.15.29:0/2217562433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f806fb50 0x7ff7f80cc8a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0097a0 tx=0x7ff7ec02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc19c5664f79f563 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:00.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7f6ffd640 1 -- 172.21.15.29:0/2217562433 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec031830 con 0x7ff7f806fb50 2023-10-16T15:55:00.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7f6ffd640 1 -- 172.21.15.29:0/2217562433 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec031990 con 0x7ff7f806fb50 2023-10-16T15:55:00.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7f6ffd640 1 -- 172.21.15.29:0/2217562433 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec031d20 con 0x7ff7f806fb50 2023-10-16T15:55:00.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 -- 172.21.15.29:0/2217562433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f806fb50 msgr2=0x7ff7f80cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 --2- 172.21.15.29:0/2217562433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f806fb50 0x7ff7f80cc8a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0097a0 tx=0x7ff7ec02b8d0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 -- 172.21.15.29:0/2217562433 shutdown_connections 2023-10-16T15:55:00.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 --2- 172.21.15.29:0/2217562433 >> v2:172.21.15.170:3300/0 conn(0x7ff7f806c9e0 0x7ff7f80ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 --2- 172.21.15.29:0/2217562433 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f806fb50 0x7ff7f80cc8a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 -- 172.21.15.29:0/2217562433 >> 172.21.15.29:0/2217562433 conn(0x7ff7f80ce6a0 msgr2=0x7ff7f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:00.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 -- 172.21.15.29:0/2217562433 shutdown_connections 2023-10-16T15:55:00.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.494+0000 7ff7fd773640 1 -- 172.21.15.29:0/2217562433 wait complete. 2023-10-16T15:55:00.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.496+0000 7ff7fd773640 1 Processor -- start 2023-10-16T15:55:00.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.496+0000 7ff7fd773640 1 -- start start 2023-10-16T15:55:00.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.497+0000 7ff7fd773640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7f806c9e0 0x7ff7f8156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.497+0000 7ff7fd773640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 0x7ff7f80b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.497+0000 7ff7fd773640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f80c9060 0x7ff7f80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.497+0000 7ff7fd773640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b1fd0 con 0x7ff7f806fb50 2023-10-16T15:55:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.497+0000 7ff7fd773640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b3f70 con 0x7ff7f806c9e0 2023-10-16T15:55:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.497+0000 7ff7fd773640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80c2d50 con 0x7ff7f80c9060 2023-10-16T15:55:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 0x7ff7f80b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 0x7ff7f80b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36478/0 (socket says 172.21.15.29:36478) 2023-10-16T15:55:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 -- 172.21.15.29:0/768754032 learned_addr learned my addr 172.21.15.29:0/768754032 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:00.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f7fff640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7f806c9e0 0x7ff7f8156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7fcf72640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f80c9060 0x7ff7f80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 -- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f80c9060 msgr2=0x7ff7f80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f80c9060 0x7ff7f80b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 -- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7f806c9e0 msgr2=0x7ff7f8156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:00.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7f806c9e0 0x7ff7f8156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 -- 172.21.15.29:0/768754032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec009400 con 0x7ff7f806fb50 2023-10-16T15:55:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f7fff640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7f806c9e0 0x7ff7f8156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.498+0000 7ff7f77fe640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 0x7ff7f80b34f0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec02bee0 tx=0x7ff7ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.499+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec03a630 con 0x7ff7f806fb50 2023-10-16T15:55:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.499+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec03a7b0 con 0x7ff7f806fb50 2023-10-16T15:55:00.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.499+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec03ab60 con 0x7ff7f806fb50 2023-10-16T15:55:00.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.499+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f80c3030 con 0x7ff7f806fb50 2023-10-16T15:55:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.499+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f80c35f0 con 0x7ff7f806fb50 2023-10-16T15:55:00.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.500+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f8071170 con 0x7ff7f806fb50 2023-10-16T15:55:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.501+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 138) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff7ec03acc0 con 0x7ff7f806fb50 2023-10-16T15:55:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.502+0000 7ff7f4ff9640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff7e406b0b0 0x7ff7e406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:00.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.502+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff7ec0ab760 con 0x7ff7f806fb50 2023-10-16T15:55:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.503+0000 7ff7f7fff640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff7e406b0b0 0x7ff7e406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.504+0000 7ff7f7fff640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff7e406b0b0 0x7ff7e406d570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8007fd0 tx=0x7ff7e8009040 comp rx=0 tx=0).ready entity=mgr.9022 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:00.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.506+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7ec0760e0 con 0x7ff7f806fb50 2023-10-16T15:55:00.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:00.759+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff7f804e9b0 con 0x7ff7f806fb50 2023-10-16T15:55:01.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.469+0000 7ff7f4ff9640 1 -- 172.21.15.29:0/768754032 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v139) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff7ec079700 con 0x7ff7f806fb50 2023-10-16T15:55:01.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.474+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff7e406b0b0 msgr2=0x7ff7e406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.474+0000 7ff7fd773640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff7e406b0b0 0x7ff7e406d570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8007fd0 tx=0x7ff7e8009040 comp rx=0 tx=0).stop 2023-10-16T15:55:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.474+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 msgr2=0x7ff7f80b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.474+0000 7ff7fd773640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 0x7ff7f80b34f0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec02bee0 tx=0x7ff7ec0046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.476+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 shutdown_connections 2023-10-16T15:55:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.476+0000 7ff7fd773640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7ff7e406b0b0 0x7ff7e406d570 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.476+0000 7ff7fd773640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7f806fb50 0x7ff7f80b34f0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.476+0000 7ff7fd773640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7f80c9060 0x7ff7f80b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.476+0000 7ff7fd773640 1 --2- 172.21.15.29:0/768754032 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7f806c9e0 0x7ff7f8156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.476+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 >> 172.21.15.29:0/768754032 conn(0x7ff7f80ce6a0 msgr2=0x7ff7f80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.477+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 shutdown_connections 2023-10-16T15:55:01.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.477+0000 7ff7fd773640 1 -- 172.21.15.29:0/768754032 wait complete. 2023-10-16T15:55:01.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T15:55:01.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.715+0000 7f390873c640 1 Processor -- start 2023-10-16T15:55:01.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.716+0000 7f390873c640 1 -- start start 2023-10-16T15:55:01.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.717+0000 7f390873c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f390015ad70 0x7f3900069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:01.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.717+0000 7f390873c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f390006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:01.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.717+0000 7f390873c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39000cb030 con 0x7f39000c9ca0 2023-10-16T15:55:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.717+0000 7f390873c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39000c91c0 con 0x7f390015ad70 2023-10-16T15:55:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.717+0000 7f390873c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39000c94a0 con 0x7f39000caac0 2023-10-16T15:55:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.718+0000 7f39064d8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f390015ad70 0x7f3900069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.718+0000 7f39064d8640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f390015ad70 0x7f3900069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:33796/0 (socket says 172.21.15.29:33796) 2023-10-16T15:55:01.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.718+0000 7f39064d8640 1 -- 172.21.15.29:0/564903835 learned_addr learned my addr 172.21.15.29:0/564903835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.718+0000 7f3906cd9640 1 --2- 172.21.15.29:0/564903835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f390006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.719+0000 7f3906cd9640 1 -- 172.21.15.29:0/564903835 >> v2:172.21.15.170:3300/0 conn(0x7f390015ad70 msgr2=0x7f3900069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.719+0000 7f3906cd9640 1 --2- 172.21.15.29:0/564903835 >> v2:172.21.15.170:3300/0 conn(0x7f390015ad70 0x7f3900069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.719+0000 7f3906cd9640 1 -- 172.21.15.29:0/564903835 >> v1:172.21.15.170:6789/0 conn(0x7f39000c9ca0 legacy=0x7f39000ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.719+0000 7f3906cd9640 1 -- 172.21.15.29:0/564903835 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39000c9780 con 0x7f39000caac0 2023-10-16T15:55:01.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.720+0000 7f3906cd9640 1 --2- 172.21.15.29:0/564903835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f390006a2e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009d90 tx=0x7f38f802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad849b4f03f57b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:01.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f39054d6640 1 -- 172.21.15.29:0/564903835 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f802cc60 con 0x7f39000caac0 2023-10-16T15:55:01.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f39054d6640 1 -- 172.21.15.29:0/564903835 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f802cdc0 con 0x7f39000caac0 2023-10-16T15:55:01.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f39054d6640 1 -- 172.21.15.29:0/564903835 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8034660 con 0x7f39000caac0 2023-10-16T15:55:01.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f390873c640 1 -- 172.21.15.29:0/564903835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 msgr2=0x7f390006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f390873c640 1 --2- 172.21.15.29:0/564903835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f390006a2e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009d90 tx=0x7f38f802beb0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f390873c640 1 -- 172.21.15.29:0/564903835 shutdown_connections 2023-10-16T15:55:01.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f390873c640 1 --2- 172.21.15.29:0/564903835 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f390006a2e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f390873c640 1 --2- 172.21.15.29:0/564903835 >> v2:172.21.15.170:3300/0 conn(0x7f390015ad70 0x7f3900069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.721+0000 7f390873c640 1 -- 172.21.15.29:0/564903835 >> 172.21.15.29:0/564903835 conn(0x7f39000ce9b0 msgr2=0x7f39000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:01.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.722+0000 7f390873c640 1 -- 172.21.15.29:0/564903835 shutdown_connections 2023-10-16T15:55:01.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.722+0000 7f390873c640 1 -- 172.21.15.29:0/564903835 wait complete. 2023-10-16T15:55:01.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.723+0000 7f390873c640 1 Processor -- start 2023-10-16T15:55:01.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.724+0000 7f390873c640 1 -- start start 2023-10-16T15:55:01.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f390873c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39000c9ca0 0x7f39000ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:01.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f390873c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f3900150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:01.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f390873c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 0x7f390015be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:01.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f390873c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390015e350 con 0x7f390015ad70 2023-10-16T15:55:01.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f390873c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390015c9e0 con 0x7f39000c9ca0 2023-10-16T15:55:01.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f390873c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39000cd390 con 0x7f39000caac0 2023-10-16T15:55:01.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f3906cd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 0x7f390015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:01.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f3906cd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 0x7f390015be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36502/0 (socket says 172.21.15.29:36502) 2023-10-16T15:55:01.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f3906cd9640 1 -- 172.21.15.29:0/3695288834 learned_addr learned my addr 172.21.15.29:0/3695288834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:01.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f39064d8640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39000c9ca0 0x7f39000ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:01.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.725+0000 7f3905cd7640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f3900150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:01.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.726+0000 7f3906cd9640 1 -- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 msgr2=0x7f3900150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.726+0000 7f3906cd9640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f3900150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.726+0000 7f3906cd9640 1 -- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39000c9ca0 msgr2=0x7f39000ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.726+0000 7f3906cd9640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39000c9ca0 0x7f39000ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.726+0000 7f3906cd9640 1 -- 172.21.15.29:0/3695288834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8009a70 con 0x7f390015ad70 2023-10-16T15:55:01.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.726+0000 7f3906cd9640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 0x7f390015be60 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009eb0 tx=0x7f38f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:01.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.727+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8040650 con 0x7f390015ad70 2023-10-16T15:55:01.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.727+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f80407b0 con 0x7f390015ad70 2023-10-16T15:55:01.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.727+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8040b60 con 0x7f390015ad70 2023-10-16T15:55:01.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.727+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39000cd670 con 0x7f390015ad70 2023-10-16T15:55:01.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.727+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39000cdc30 con 0x7f390015ad70 2023-10-16T15:55:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.728+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390004e160 con 0x7f390015ad70 2023-10-16T15:55:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.729+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 139) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f38f8040cc0 con 0x7f390015ad70 2023-10-16T15:55:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.730+0000 7f38f6ffd640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 0x7f38e406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.730+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f38f80ac860 con 0x7f390015ad70 2023-10-16T15:55:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.730+0000 7f39064d8640 1 -- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 msgr2=0x7f38e406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.731+0000 7f39064d8640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 0x7f38e406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.735+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f38f8077560 con 0x7f390015ad70 2023-10-16T15:55:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.932+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f39000b1310 con 0x7f390015ad70 2023-10-16T15:55:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.932+0000 7f39064d8640 1 -- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 msgr2=0x7f38e406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.932+0000 7f39064d8640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 0x7f38e406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.938+0000 7f38f6ffd640 1 -- 172.21.15.29:0/3695288834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f38f8009450 con 0x7f390015ad70 2023-10-16T15:55:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 msgr2=0x7f38e406d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 0x7f38e406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 msgr2=0x7f390015be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 0x7f390015be60 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009eb0 tx=0x7f38f8004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 shutdown_connections 2023-10-16T15:55:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f38e406b100 0x7f38e406d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f390015ad70 0x7f390015be60 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f39000caac0 0x7f3900150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 --2- 172.21.15.29:0/3695288834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39000c9ca0 0x7f39000ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 >> 172.21.15.29:0/3695288834 conn(0x7f39000ce9b0 msgr2=0x7f39000b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 shutdown_connections 2023-10-16T15:55:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:01.941+0000 7f390873c640 1 -- 172.21.15.29:0/3695288834 wait complete. 2023-10-16T15:55:01.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T15:55:01.999 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:01.998+0000 7f1eea413040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:02.017 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.016+0000 7f6cece32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:02.051 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.050+0000 7f1eea413040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:02.073 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.072+0000 7f6cece32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:02.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.173+0000 7fe18db93640 1 Processor -- start 2023-10-16T15:55:02.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.174+0000 7fe18db93640 1 -- start start 2023-10-16T15:55:02.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18db93640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1880b8480 0x7fe1880cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18db93640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe1880d0be0 0x7fe1880cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18db93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18815d150 con 0x7fe18815cd70 2023-10-16T15:55:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18db93640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880d1180 con 0x7fe1880b8480 2023-10-16T15:55:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18db93640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880d1460 con 0x7fe1880d0be0 2023-10-16T15:55:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18cb91640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe18815cd70 0x7fe1880d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33304/0 (socket says 172.21.15.29:33304) 2023-10-16T15:55:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.175+0000 7fe18cb91640 1 -- 172.21.15.29:0/2857399968 learned_addr learned my addr 172.21.15.29:0/2857399968 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:02.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe187fff640 1 --2- 172.21.15.29:0/2857399968 >> v2:172.21.15.29:3300/0 conn(0x7fe1880b8480 0x7fe1880cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe18d392640 1 --2- 172.21.15.29:0/2857399968 >> v2:172.21.15.170:3300/0 conn(0x7fe1880d0be0 0x7fe1880cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe1877fe640 1 -- 172.21.15.29:0/2857399968 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010305646 0 0) 0x7fe18815d150 con 0x7fe18815cd70 2023-10-16T15:55:02.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe1877fe640 1 -- 172.21.15.29:0/2857399968 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe170003520 con 0x7fe18815cd70 2023-10-16T15:55:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe187fff640 1 -- 172.21.15.29:0/2857399968 >> v2:172.21.15.170:3300/0 conn(0x7fe1880d0be0 msgr2=0x7fe1880cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe187fff640 1 --2- 172.21.15.29:0/2857399968 >> v2:172.21.15.170:3300/0 conn(0x7fe1880d0be0 0x7fe1880cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe187fff640 1 -- 172.21.15.29:0/2857399968 >> v1:172.21.15.29:6789/0 conn(0x7fe18815cd70 legacy=0x7fe1880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.176+0000 7fe187fff640 1 -- 172.21.15.29:0/2857399968 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880d1740 con 0x7fe1880b8480 2023-10-16T15:55:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.177+0000 7fe187fff640 1 --2- 172.21.15.29:0/2857399968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe1880cd7c0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fe1880b8910 tx=0x7fe17802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1780f1e9c73ad22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe1877fe640 1 -- 172.21.15.29:0/2857399968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17802e070 con 0x7fe1880b8480 2023-10-16T15:55:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe1877fe640 1 -- 172.21.15.29:0/2857399968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1780315a0 con 0x7fe1880b8480 2023-10-16T15:55:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe1877fe640 1 -- 172.21.15.29:0/2857399968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178031930 con 0x7fe1880b8480 2023-10-16T15:55:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 -- 172.21.15.29:0/2857399968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 msgr2=0x7fe1880cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 --2- 172.21.15.29:0/2857399968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe1880cd7c0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fe1880b8910 tx=0x7fe17802b840 comp rx=0 tx=0).stop 2023-10-16T15:55:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 -- 172.21.15.29:0/2857399968 shutdown_connections 2023-10-16T15:55:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 --2- 172.21.15.29:0/2857399968 >> v2:172.21.15.170:3300/0 conn(0x7fe1880d0be0 0x7fe1880cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 --2- 172.21.15.29:0/2857399968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe1880cd7c0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 -- 172.21.15.29:0/2857399968 >> 172.21.15.29:0/2857399968 conn(0x7fe1880cf570 msgr2=0x7fe1880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 -- 172.21.15.29:0/2857399968 shutdown_connections 2023-10-16T15:55:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.178+0000 7fe18db93640 1 -- 172.21.15.29:0/2857399968 wait complete. 2023-10-16T15:55:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.180+0000 7fe18db93640 1 Processor -- start 2023-10-16T15:55:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.180+0000 7fe18db93640 1 -- start start 2023-10-16T15:55:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.181+0000 7fe18db93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe18815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.181+0000 7fe18db93640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1880d0be0 0x7fe188144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.181+0000 7fe18db93640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe18815cd70 0x7fe1881f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.181+0000 7fe18db93640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18815b920 con 0x7fe1880b8480 2023-10-16T15:55:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.181+0000 7fe18db93640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18815ad10 con 0x7fe18815cd70 2023-10-16T15:55:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.181+0000 7fe18db93640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18815aff0 con 0x7fe1880d0be0 2023-10-16T15:55:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe18815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe18815bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36526/0 (socket says 172.21.15.29:36526) 2023-10-16T15:55:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 -- 172.21.15.29:0/271952607 learned_addr learned my addr 172.21.15.29:0/271952607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe187fff640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1880d0be0 0x7fe188144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18d392640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe18815cd70 0x7fe1881f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 -- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1880d0be0 msgr2=0x7fe188144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1880d0be0 0x7fe188144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 -- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe18815cd70 msgr2=0x7fe1881f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe18815cd70 0x7fe1881f61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 -- 172.21.15.29:0/271952607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178009400 con 0x7fe1880b8480 2023-10-16T15:55:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18d392640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe18815cd70 0x7fe1881f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.182+0000 7fe18cb91640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe18815bdb0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c500 tx=0x7fe17c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.183+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c005ce0 con 0x7fe1880b8480 2023-10-16T15:55:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.183+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c005e40 con 0x7fe1880b8480 2023-10-16T15:55:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.183+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c007330 con 0x7fe1880b8480 2023-10-16T15:55:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.183+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18815b2d0 con 0x7fe1880b8480 2023-10-16T15:55:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.183+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18815aa00 con 0x7fe1880b8480 2023-10-16T15:55:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.184+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1880c9980 con 0x7fe1880b8480 2023-10-16T15:55:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.185+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 139) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fe17c007490 con 0x7fe1880b8480 2023-10-16T15:55:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.186+0000 7fe1857fa640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 0x7fe17406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.187+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fe17c0894c0 con 0x7fe1880b8480 2023-10-16T15:55:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.187+0000 7fe187fff640 1 -- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 msgr2=0x7fe17406d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.187+0000 7fe187fff640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 0x7fe17406d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.191+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe17c053f40 con 0x7fe1880b8480 2023-10-16T15:55:02.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.385+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe18804e9b0 con 0x7fe1880b8480 2023-10-16T15:55:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.388+0000 7fe187fff640 1 -- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 msgr2=0x7fe17406d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.388+0000 7fe187fff640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 0x7fe17406d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:02.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.390+0000 7fe1857fa640 1 -- 172.21.15.29:0/271952607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe17c057560 con 0x7fe1880b8480 2023-10-16T15:55:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 msgr2=0x7fe17406d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 0x7fe17406d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 msgr2=0x7fe18815bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe18815bdb0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c500 tx=0x7fe17c00c9c0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 shutdown_connections 2023-10-16T15:55:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7fe17406b130 0x7fe17406d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe18815cd70 0x7fe1881f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe1880d0be0 0x7fe188144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 --2- 172.21.15.29:0/271952607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1880b8480 0x7fe18815bdb0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.392+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 >> 172.21.15.29:0/271952607 conn(0x7fe1880cf570 msgr2=0x7fe1880b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.393+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 shutdown_connections 2023-10-16T15:55:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.393+0000 7fe18db93640 1 -- 172.21.15.29:0/271952607 wait complete. 2023-10-16T15:55:02.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:02.412 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T15:55:02.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2023-10-16T15:55:02.503 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.502+0000 7f1eea413040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:02.554 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.554+0000 7f1eea413040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:02.563 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.562+0000 7f6cece32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:02.604 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.603+0000 7f1eea413040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:02.618 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.618+0000 7f6cece32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:02.664 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.663+0000 7f1eea413040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:02.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.671+0000 7f6cece32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:02.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.677+0000 7f2d2f11d640 1 Processor -- start 2023-10-16T15:55:02.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.677+0000 7f2d2f11d640 1 -- start start 2023-10-16T15:55:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.678+0000 7f2d2f11d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d280b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.678+0000 7f2d2f11d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280d3980 con 0x7f2d280b3420 2023-10-16T15:55:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.678+0000 7f2d2f11d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280ce630 con 0x7f2d280cf710 2023-10-16T15:55:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.678+0000 7f2d2f11d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280d3c60 con 0x7f2d280cfaf0 2023-10-16T15:55:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.679+0000 7f2d27fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2d280b3420 0x7f2d280b5cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33308/0 (socket says 172.21.15.29:33308) 2023-10-16T15:55:02.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.679+0000 7f2d27fff640 1 -- 172.21.15.29:0/1816676916 learned_addr learned my addr 172.21.15.29:0/1816676916 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.679+0000 7f2d2d6ba640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d280b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.679+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863194444 0 0) 0x7f2d280d3980 con 0x7f2d280b3420 2023-10-16T15:55:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.679+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d18003520 con 0x7f2d280b3420 2023-10-16T15:55:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 658056269 0 0) 0x7f2d18003520 con 0x7f2d280b3420 2023-10-16T15:55:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 msgr2=0x7f2d280b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d280b63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 >> v1:172.21.15.170:6789/0 conn(0x7f2d280cf710 legacy=0x7f2d280b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280ce910 con 0x7f2d280b3420 2023-10-16T15:55:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f2d14001670 con 0x7f2d280b3420 2023-10-16T15:55:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 >> v1:172.21.15.29:6789/0 conn(0x7f2d280b3420 legacy=0x7f2d280b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d180039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d18003ee0 0x7f2d180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18004d00 0x7f2d18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280d3c60 con 0x7f2d18003ee0 2023-10-16T15:55:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800ba90 con 0x7f2d18004d00 2023-10-16T15:55:02.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800bc50 con 0x7f2d280cfaf0 2023-10-16T15:55:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d27fff640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d18003ee0 0x7f2d180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.680+0000 7f2d2d6ba640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18004d00 0x7f2d18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d180039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d27fff640 1 -- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 msgr2=0x7f2d180039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d27fff640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d180039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d27fff640 1 -- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18004d00 msgr2=0x7f2d18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:02.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d27fff640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d18004d00 0x7f2d18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d27fff640 1 -- 172.21.15.29:0/1816676916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18003520 con 0x7f2d18003ee0 2023-10-16T15:55:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.681+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d280cfaf0 0x7f2d180039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.683+0000 7f2d27fff640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d18003ee0 0x7f2d180047c0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400c240 tx=0x7f2d180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1371334e8f7849 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:02.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.683+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14005ad0 con 0x7f2d18003ee0 2023-10-16T15:55:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.683+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14005c30 con 0x7f2d18003ee0 2023-10-16T15:55:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.683+0000 7f2d277fe640 1 -- 172.21.15.29:0/1816676916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1402c8a0 con 0x7f2d18003ee0 2023-10-16T15:55:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2f11d640 1 -- 172.21.15.29:0/1816676916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d18003ee0 msgr2=0x7f2d180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2f11d640 1 --2- 172.21.15.29:0/1816676916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d18003ee0 0x7f2d180047c0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400c240 tx=0x7f2d180057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2d6ba640 1 -- 172.21.15.29:0/1816676916 reap_dead start 2023-10-16T15:55:02.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2f11d640 1 -- 172.21.15.29:0/1816676916 shutdown_connections 2023-10-16T15:55:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2f11d640 1 -- 172.21.15.29:0/1816676916 >> 172.21.15.29:0/1816676916 conn(0x7f2d280d2cc0 msgr2=0x7f2d280d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2f11d640 1 -- 172.21.15.29:0/1816676916 shutdown_connections 2023-10-16T15:55:02.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.684+0000 7f2d2f11d640 1 -- 172.21.15.29:0/1816676916 wait complete. 2023-10-16T15:55:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.687+0000 7f2d2f11d640 1 Processor -- start 2023-10-16T15:55:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.687+0000 7f2d2f11d640 1 -- start start 2023-10-16T15:55:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.688+0000 7f2d2f11d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d18003ee0 0x7f2d2815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.688+0000 7f2d2f11d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d28150a80 0x7f2d280c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.688+0000 7f2d2f11d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d280c9380 0x7f2d2815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.688+0000 7f2d2f11d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2815f400 con 0x7f2d28150a80 2023-10-16T15:55:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.688+0000 7f2d2f11d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2815dbd0 con 0x7f2d280c9380 2023-10-16T15:55:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.688+0000 7f2d2f11d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28151d50 con 0x7f2d18003ee0 2023-10-16T15:55:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d28150a80 0x7f2d280c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d28150a80 0x7f2d280c8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36536/0 (socket says 172.21.15.29:36536) 2023-10-16T15:55:02.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 -- 172.21.15.29:0/368601287 learned_addr learned my addr 172.21.15.29:0/368601287 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:02.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d2d6ba640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d280c9380 0x7f2d2815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d18003ee0 0x7f2d2815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d18003ee0 msgr2=0x7f2d2815eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2d18003ee0 0x7f2d2815eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d280c9380 msgr2=0x7f2d2815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2d280c9380 0x7f2d2815fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.689+0000 7f2d27fff640 1 -- 172.21.15.29:0/368601287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2815de20 con 0x7f2d28150a80 2023-10-16T15:55:02.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.690+0000 7f2d27fff640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d28150a80 0x7f2d280c8e40 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f2d1402c440 tx=0x7f2d2815d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.690+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1403c040 con 0x7f2d28150a80 2023-10-16T15:55:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.690+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1402ccb0 con 0x7f2d28150a80 2023-10-16T15:55:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.690+0000 7f2d2f11d640 1 -- 172.21.15.29:0/368601287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2815e100 con 0x7f2d28150a80 2023-10-16T15:55:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.690+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d140072b0 con 0x7f2d28150a80 2023-10-16T15:55:02.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.690+0000 7f2d2f11d640 1 -- 172.21.15.29:0/368601287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2815e6c0 con 0x7f2d28150a80 2023-10-16T15:55:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.691+0000 7f2d2f11d640 1 -- 172.21.15.29:0/368601287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d280c9760 con 0x7f2d28150a80 2023-10-16T15:55:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.693+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 139) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f2d14032070 con 0x7f2d28150a80 2023-10-16T15:55:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.694+0000 7f2d257fa640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 0x7f2d0c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.694+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f2d140aae40 con 0x7f2d28150a80 2023-10-16T15:55:02.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.694+0000 7f2d2ceb9640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 msgr2=0x7f2d0c06d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.694+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 0x7f2d0c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.699+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2d14075800 con 0x7f2d28150a80 2023-10-16T15:55:02.714 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.713+0000 7f1eea413040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:02.741 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.740+0000 7f6cece32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:02.793 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.792+0000 7f6cece32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:02.824 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.824+0000 7f1eea413040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:55:02.886 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.886+0000 7f1eea413040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:02.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.895+0000 7f2d2ceb9640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 msgr2=0x7f2d0c06d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:02.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.895+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 0x7f2d0c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:02.910 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.909+0000 7f6cece32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:55:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:02.912+0000 7f2d2f11d640 1 -- 172.21.15.29:0/368601287 --> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d280d1d50 con 0x7f2d0c06b100 2023-10-16T15:55:02.936 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.935+0000 7f1eea413040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:55:02.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:02.976+0000 7f6cece32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:02.993 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:02.992+0000 7f1eea413040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:55:03.028 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.027+0000 7f6cece32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:55:03.043 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.042+0000 7f1eea413040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:55:03.087 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.086+0000 7f6cece32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:55:03.140 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.138+0000 7f6cece32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:55:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:03.296+0000 7f2d2ceb9640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 msgr2=0x7f2d0c06d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:03.296+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 0x7f2d0c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T15:55:03.424 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.423+0000 7f1eea413040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:55:03.542 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.541+0000 7f6cece32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:55:03.579 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.579+0000 7f1eea413040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:55:03.686 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.685+0000 7f1eea413040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:55:03.709 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.708+0000 7f6cece32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:55:03.735 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.734+0000 7f1eea413040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:55:03.822 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.821+0000 7f6cece32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:55:03.841 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.840+0000 7f1eea413040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:03.873 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.873+0000 7f6cece32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:55:03.947 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:03.947+0000 7f1eea413040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:55:03.986 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:03.985+0000 7f6cece32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:04.097 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:04.096+0000 7f6cece32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:55:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:04.097+0000 7f2d2ceb9640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 msgr2=0x7f2d0c06d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/58791103 2023-10-16T15:55:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:04.097+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 0x7f2d0c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T15:55:04.102 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:04.101+0000 7f1eea413040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:04.151 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:04.150+0000 7f1eea413040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:04.260 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:04.259+0000 7f6cece32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:04.268 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:04.267+0000 7f1eea413040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:55:04.311 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:04.310+0000 7f6cece32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:04.436 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:04.435+0000 7f6cece32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:55:05.097 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.096+0000 7f1eea413040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:55:05.302 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:05.301+0000 7f6cece32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:55:05.389 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.388+0000 7f1eea413040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:55:05.445 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.444+0000 7f1eea413040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:55:05.557 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.556+0000 7f1eea413040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:55:05.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:05.581+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 140) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f2d14075bd0 con 0x7f2d28150a80 2023-10-16T15:55:05.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:05.582+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 msgr2=0x7f2d0c06d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:05.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:05.582+0000 7f2d257fa640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/58791103,v1:172.21.15.170:6833/58791103] conn(0x7f2d0c06b100 0x7f2d0c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:05.618 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:05.617+0000 7f6cece32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:55:05.683 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:05.682+0000 7f6cece32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:55:05.810 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:05.809+0000 7f6cece32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:55:05.811 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.811 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.813 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.813 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.813 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.811+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.827 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.828 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.828 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.828 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.828 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e6629a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.829 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.829 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.829 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.830 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:05.830 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:05.827+0000 7f1e692a0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.590+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2d14081330 con 0x7f2d28150a80 2023-10-16T15:55:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.591+0000 7f2d257fa640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f2d0c0356c0 0x7f2d0c035aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.591+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 --> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d0c035fe0 con 0x7f2d0c0356c0 2023-10-16T15:55:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.597+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f2d0c0356c0 0x7f2d0c035aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.606+0000 7f2d2ceb9640 1 --2- 172.21.15.29:0/368601287 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f2d0c0356c0 0x7f2d0c035aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c007c70 tx=0x7f2d1c009040 comp rx=0 tx=0).ready entity=mgr.8381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:06.611 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:06.609+0000 7f1e702ee640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2023-10-16T15:55:06.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.610+0000 7f2d257fa640 1 -- 172.21.15.29:0/368601287 <== mgr.8381 v2:172.21.15.170:6832/3738368341 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7f2d0c035fe0 con 0x7f2d0c0356c0 2023-10-16T15:55:06.612 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2023-10-16T15:55:06.628 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T15:55:06.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:06.629 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T15:55:06.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:55:06.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.903+0000 7f37997d0640 1 Processor -- start 2023-10-16T15:55:06.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.903+0000 7f37997d0640 1 -- start start 2023-10-16T15:55:06.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f37997d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406fb50 0x7f37940cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f37997d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f379406c9e0 0x7f37940ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f37997d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37940ca560 con 0x7f37940c9060 2023-10-16T15:55:06.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f37997d0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37940c9950 con 0x7f379406c9e0 2023-10-16T15:55:06.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f37997d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37940c9c30 con 0x7f379406fb50 2023-10-16T15:55:06.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f3793fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f37940c9060 0x7f379406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58466/0 (socket says 172.21.15.29:58466) 2023-10-16T15:55:06.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f3793fff640 1 -- 172.21.15.29:0/714133942 learned_addr learned my addr 172.21.15.29:0/714133942 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:06.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f3798fcf640 1 --2- 172.21.15.29:0/714133942 >> v2:172.21.15.29:3300/0 conn(0x7f379406c9e0 0x7f37940ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f37937fe640 1 --2- 172.21.15.29:0/714133942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406fb50 0x7f37940cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.905+0000 7f3792ffd640 1 -- 172.21.15.29:0/714133942 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594271621 0 0) 0x7f37940ca560 con 0x7f37940c9060 2023-10-16T15:55:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.906+0000 7f3792ffd640 1 -- 172.21.15.29:0/714133942 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c003520 con 0x7f37940c9060 2023-10-16T15:55:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.906+0000 7f3798fcf640 1 -- 172.21.15.29:0/714133942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406fb50 msgr2=0x7f37940cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.906+0000 7f3798fcf640 1 --2- 172.21.15.29:0/714133942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406fb50 0x7f37940cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.906+0000 7f3798fcf640 1 -- 172.21.15.29:0/714133942 >> v1:172.21.15.29:6789/0 conn(0x7f37940c9060 legacy=0x7f379406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.906+0000 7f3798fcf640 1 -- 172.21.15.29:0/714133942 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37940c9f10 con 0x7f379406c9e0 2023-10-16T15:55:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.907+0000 7f3798fcf640 1 --2- 172.21.15.29:0/714133942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f379406c9e0 0x7f37940ccde0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f378c0097a0 tx=0x7f378c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84eea7388c9093bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.907+0000 7f3792ffd640 1 -- 172.21.15.29:0/714133942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c02d040 con 0x7f379406c9e0 2023-10-16T15:55:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.907+0000 7f3792ffd640 1 -- 172.21.15.29:0/714133942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c031810 con 0x7f379406c9e0 2023-10-16T15:55:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.907+0000 7f3792ffd640 1 -- 172.21.15.29:0/714133942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c031ba0 con 0x7f379406c9e0 2023-10-16T15:55:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 -- 172.21.15.29:0/714133942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f379406c9e0 msgr2=0x7f37940ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 --2- 172.21.15.29:0/714133942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f379406c9e0 0x7f37940ccde0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f378c0097a0 tx=0x7f378c02b8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 -- 172.21.15.29:0/714133942 shutdown_connections 2023-10-16T15:55:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 --2- 172.21.15.29:0/714133942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f379406c9e0 0x7f37940ccde0 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 --2- 172.21.15.29:0/714133942 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406fb50 0x7f37940cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 -- 172.21.15.29:0/714133942 >> 172.21.15.29:0/714133942 conn(0x7f37940ce6a0 msgr2=0x7f37940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 -- 172.21.15.29:0/714133942 shutdown_connections 2023-10-16T15:55:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.908+0000 7f37997d0640 1 -- 172.21.15.29:0/714133942 wait complete. 2023-10-16T15:55:06.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.910+0000 7f37997d0640 1 Processor -- start 2023-10-16T15:55:06.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.910+0000 7f37997d0640 1 -- start start 2023-10-16T15:55:06.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f37997d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406c9e0 0x7f3794156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f37997d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f379406fb50 0x7f37940b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f37997d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 0x7f37940c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f37997d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940c5770 con 0x7f37940c9060 2023-10-16T15:55:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f37997d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3794156f60 con 0x7f379406fb50 2023-10-16T15:55:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f37997d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37940c5fa0 con 0x7f379406c9e0 2023-10-16T15:55:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 0x7f37940c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 0x7f37940c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60068/0 (socket says 172.21.15.29:60068) 2023-10-16T15:55:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 -- 172.21.15.29:0/3869093892 learned_addr learned my addr 172.21.15.29:0/3869093892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 -- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406c9e0 msgr2=0x7f3794156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406c9e0 0x7f3794156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 -- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f379406fb50 msgr2=0x7f37940b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f379406fb50 0x7f37940b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.911+0000 7f3798fcf640 1 -- 172.21.15.29:0/3869093892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c009400 con 0x7f37940c9060 2023-10-16T15:55:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.912+0000 7f3798fcf640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 0x7f37940c43d0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f378c02b8f0 tx=0x7f378c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.912+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c006e10 con 0x7f37940c9060 2023-10-16T15:55:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.912+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c03ace0 con 0x7f37940c9060 2023-10-16T15:55:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.912+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c044630 con 0x7f37940c9060 2023-10-16T15:55:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.912+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37940b3990 con 0x7f37940c9060 2023-10-16T15:55:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.912+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37940c5c30 con 0x7f37940c9060 2023-10-16T15:55:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.913+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379404e9b0 con 0x7f37940c9060 2023-10-16T15:55:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.914+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 141) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f378c044790 con 0x7f37940c9060 2023-10-16T15:55:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.915+0000 7f3790ff9640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f378006ad40 0x7f378006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.915+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f378c0aba20 con 0x7f37940c9060 2023-10-16T15:55:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.916+0000 7f3793fff640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f378006ad40 0x7f378006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.917+0000 7f3793fff640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f378006ad40 0x7f378006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3784001570 tx=0x7f3784007930 comp rx=0 tx=0).ready entity=mgr.8381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:06.920+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f378c0768e0 con 0x7f37940c9060 2023-10-16T15:55:07.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.153+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f37940c60e0 con 0x7f37940c9060 2023-10-16T15:55:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.633+0000 7f3790ff9640 1 -- 172.21.15.29:0/3869093892 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v142) v1 ==== 93+0+0 (secure 0 0 0) 0x7f378c079f00 con 0x7f37940c9060 2023-10-16T15:55:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.639+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f378006ad40 msgr2=0x7f378006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.639+0000 7f37997d0640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f378006ad40 0x7f378006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3784001570 tx=0x7f3784007930 comp rx=0 tx=0).stop 2023-10-16T15:55:07.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.639+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 msgr2=0x7f37940c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.639+0000 7f37997d0640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 0x7f37940c43d0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f378c02b8f0 tx=0x7f378c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 shutdown_connections 2023-10-16T15:55:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f378006ad40 0x7f378006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37940c9060 0x7f37940c43d0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:07.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f379406fb50 0x7f37940b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 --2- 172.21.15.29:0/3869093892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f379406c9e0 0x7f3794156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 >> 172.21.15.29:0/3869093892 conn(0x7f37940ce6a0 msgr2=0x7f37940cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 shutdown_connections 2023-10-16T15:55:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:07.641+0000 7f37997d0640 1 -- 172.21.15.29:0/3869093892 wait complete. 2023-10-16T15:55:07.663 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:55:07.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:55:07.761 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:55:07.857 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:55:07.858 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:07.858 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:07.913 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:07.913 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:07.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:08.160 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.158+0000 7f48b483a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:08.205 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.204+0000 7f42cd84c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:08.212 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.210+0000 7f48b483a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.219+0000 7fe7025f0640 1 Processor -- start 2023-10-16T15:55:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.220+0000 7fe7025f0640 1 -- start start 2023-10-16T15:55:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.221+0000 7fe7025f0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe6fc159b20 0x7fe6fc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.221+0000 7fe7025f0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc06f7f0 con 0x7fe6fc0b0b20 2023-10-16T15:55:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.221+0000 7fe7025f0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc06c5e0 con 0x7fe6fc159f00 2023-10-16T15:55:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.221+0000 7fe7025f0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc06c8c0 con 0x7fe6fc159b20 2023-10-16T15:55:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.222+0000 7fe700b8d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe6fc0b0b20 0x7fe6fc06fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58472/0 (socket says 172.21.15.29:58472) 2023-10-16T15:55:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.222+0000 7fe700b8d640 1 -- 172.21.15.29:0/3323647730 learned_addr learned my addr 172.21.15.29:0/3323647730 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.222+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3323647730 >> v2:172.21.15.170:3300/0 conn(0x7fe6fc159b20 0x7fe6fc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.222+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336961131 0 0) 0x7fe6fc06f7f0 con 0x7fe6fc0b0b20 2023-10-16T15:55:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.222+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e8003520 con 0x7fe6fc0b0b20 2023-10-16T15:55:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1880683518 0 0) 0x7fe6e8003520 con 0x7fe6fc0b0b20 2023-10-16T15:55:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 >> v2:172.21.15.170:3300/0 conn(0x7fe6fc159b20 msgr2=0x7fe6fc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 --2- 172.21.15.29:0/3323647730 >> v2:172.21.15.170:3300/0 conn(0x7fe6fc159b20 0x7fe6fc06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 >> v1:172.21.15.170:6789/0 conn(0x7fe6fc159f00 legacy=0x7fe6fc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc06cba0 con 0x7fe6fc0b0b20 2023-10-16T15:55:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe6f4001670 con 0x7fe6fc0b0b20 2023-10-16T15:55:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 >> v1:172.21.15.29:6789/0 conn(0x7fe6fc0b0b20 legacy=0x7fe6fc06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80039c0 0x7fe6e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6e80047c0 0x7fe6e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e8005180 0x7fe6e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc06c8c0 con 0x7fe6e80039c0 2023-10-16T15:55:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e800c080 con 0x7fe6e8005180 2023-10-16T15:55:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e800c240 con 0x7fe6e80047c0 2023-10-16T15:55:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.223+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80039c0 0x7fe6e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6f37fe640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6e80047c0 0x7fe6e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe700b8d640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e8005180 0x7fe6e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6fbfff640 1 -- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6e80047c0 msgr2=0x7fe6e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6e80047c0 0x7fe6e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6fbfff640 1 -- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e8005180 msgr2=0x7fe6e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:08.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e8005180 0x7fe6e800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6fbfff640 1 -- 172.21.15.29:0/3323647730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8003520 con 0x7fe6e80039c0 2023-10-16T15:55:08.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe700b8d640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e8005180 0x7fe6e800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:08.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe700b8d640 1 -- 172.21.15.29:0/3323647730 reap_dead start 2023-10-16T15:55:08.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.224+0000 7fe6f37fe640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6e80047c0 0x7fe6e8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.225+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80039c0 0x7fe6e8003de0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009650 tx=0x7fe6e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a6380cdebe4145 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.225+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec02e8c0 con 0x7fe6e80039c0 2023-10-16T15:55:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.225+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec02ea20 con 0x7fe6e80039c0 2023-10-16T15:55:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.225+0000 7fe6fb7fe640 1 -- 172.21.15.29:0/3323647730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec02edb0 con 0x7fe6e80039c0 2023-10-16T15:55:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 -- 172.21.15.29:0/3323647730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80039c0 msgr2=0x7fe6e8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80039c0 0x7fe6e8003de0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009650 tx=0x7fe6e80057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 -- 172.21.15.29:0/3323647730 shutdown_connections 2023-10-16T15:55:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3323647730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80039c0 0x7fe6e8003de0 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 -- 172.21.15.29:0/3323647730 >> 172.21.15.29:0/3323647730 conn(0x7fe6fc0cb410 msgr2=0x7fe6fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 -- 172.21.15.29:0/3323647730 shutdown_connections 2023-10-16T15:55:08.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.226+0000 7fe7025f0640 1 -- 172.21.15.29:0/3323647730 wait complete. 2023-10-16T15:55:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.228+0000 7fe7025f0640 1 Processor -- start 2023-10-16T15:55:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.228+0000 7fe7025f0640 1 -- start start 2023-10-16T15:55:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.229+0000 7fe7025f0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e80039c0 0x7fe6fc14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.229+0000 7fe7025f0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6fc155c80 0x7fe6fc1f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.229+0000 7fe7025f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 0x7fe6fc0acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.229+0000 7fe7025f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc1f7020 con 0x7fe6fc0ac890 2023-10-16T15:55:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.229+0000 7fe7025f0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0ad250 con 0x7fe6e80039c0 2023-10-16T15:55:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.229+0000 7fe7025f0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c8490 con 0x7fe6fc155c80 2023-10-16T15:55:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 0x7fe6fc0acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 0x7fe6fc0acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60090/0 (socket says 172.21.15.29:60090) 2023-10-16T15:55:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 -- 172.21.15.29:0/3436410960 learned_addr learned my addr 172.21.15.29:0/3436410960 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e80039c0 0x7fe6fc14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe6f37fe640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6fc155c80 0x7fe6fc1f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 -- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6fc155c80 msgr2=0x7fe6fc1f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6fc155c80 0x7fe6fc1f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 -- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e80039c0 msgr2=0x7fe6fc14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e80039c0 0x7fe6fc14eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 -- 172.21.15.29:0/3436410960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0c8770 con 0x7fe6fc0ac890 2023-10-16T15:55:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e80039c0 0x7fe6fc14eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.230+0000 7fe700b8d640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 0x7fe6fc0acd10 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc06d340 tx=0x7fe6f400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.231+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4007440 con 0x7fe6fc0ac890 2023-10-16T15:55:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.231+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f40075c0 con 0x7fe6fc0ac890 2023-10-16T15:55:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.231+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc0b2da0 con 0x7fe6fc0ac890 2023-10-16T15:55:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.231+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4004a20 con 0x7fe6fc0ac890 2023-10-16T15:55:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.231+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc0b2a90 con 0x7fe6fc0ac890 2023-10-16T15:55:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.232+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc04e160 con 0x7fe6fc0ac890 2023-10-16T15:55:08.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.234+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe6f4004b80 con 0x7fe6fc0ac890 2023-10-16T15:55:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.234+0000 7fe6f97fa640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 0x7fe6dc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.235+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7fe6f4089d90 con 0x7fe6fc0ac890 2023-10-16T15:55:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.235+0000 7fe6fbfff640 1 -- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 msgr2=0x7fe6dc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.235+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 0x7fe6dc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:08.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.239+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe6f4054400 con 0x7fe6fc0ac890 2023-10-16T15:55:08.266 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.265+0000 7f42cd84c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.436+0000 7fe6fbfff640 1 -- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 msgr2=0x7fe6dc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.436+0000 7fe6fbfff640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 0x7fe6dc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.439+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6fc0b10b0 con 0x7fe6fc0ac890 2023-10-16T15:55:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.440+0000 7fe6f97fa640 1 -- 172.21.15.29:0/3436410960 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+20862 (secure 0 0 0) 0x7fe6f4057a20 con 0x7fe6fc0ac890 2023-10-16T15:55:08.442 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:08.442 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":251,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:05.561201+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 msgr2=0x7fe6dc06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 0x7fe6dc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 msgr2=0x7fe6fc0acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 0x7fe6fc0acd10 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc06d340 tx=0x7fe6f400d950 comp rx=0 tx=0).stop 2023-10-16T15:55:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 shutdown_connections 2023-10-16T15:55:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fe6dc06b0b0 0x7fe6dc06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6fc0ac890 0x7fe6fc0acd10 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe6fc155c80 0x7fe6fc1f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 --2- 172.21.15.29:0/3436410960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6e80039c0 0x7fe6fc14eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 >> 172.21.15.29:0/3436410960 conn(0x7fe6fc0cb410 msgr2=0x7fe6fc06fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 shutdown_connections 2023-10-16T15:55:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.443+0000 7fe7025f0640 1 -- 172.21.15.29:0/3436410960 wait complete. 2023-10-16T15:55:08.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:54:36.572242+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:54:37.685848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T15:54:51.157450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T15:54:51.864591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:55:08.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:55:08.666 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.664+0000 7f48b483a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:08.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.681+0000 7f59230c2640 1 Processor -- start 2023-10-16T15:55:08.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.682+0000 7f59230c2640 1 -- start start 2023-10-16T15:55:08.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f59230c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f591c15ad70 0x7f591c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f59230c2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f591c0c9ca0 0x7f591c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f59230c2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c0cb030 con 0x7f591c0ca3b0 2023-10-16T15:55:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f59230c2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0c91c0 con 0x7f591c15ad70 2023-10-16T15:55:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f59230c2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0c94a0 con 0x7f591c0c9ca0 2023-10-16T15:55:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f5920e5e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f591c15ad70 0x7f591c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f5920e5e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f591c15ad70 0x7f591c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60102/0 (socket says 172.21.15.29:60102) 2023-10-16T15:55:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f5920e5e640 1 -- 172.21.15.29:0/2159056361 learned_addr learned my addr 172.21.15.29:0/2159056361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.683+0000 7f592165f640 1 --1- 172.21.15.29:0/2159056361 >> v1:172.21.15.29:6789/0 conn(0x7f591c0ca3b0 0x7f591c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58474/0 (socket says 172.21.15.29:58474) 2023-10-16T15:55:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.684+0000 7f591bfff640 1 --2- 172.21.15.29:0/2159056361 >> v2:172.21.15.170:3300/0 conn(0x7f591c0c9ca0 0x7f591c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.684+0000 7f5920e5e640 1 -- 172.21.15.29:0/2159056361 >> v2:172.21.15.170:3300/0 conn(0x7f591c0c9ca0 msgr2=0x7f591c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.684+0000 7f5920e5e640 1 --2- 172.21.15.29:0/2159056361 >> v2:172.21.15.170:3300/0 conn(0x7f591c0c9ca0 0x7f591c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.684+0000 7f5920e5e640 1 -- 172.21.15.29:0/2159056361 >> v1:172.21.15.29:6789/0 conn(0x7f591c0ca3b0 legacy=0x7f591c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.684+0000 7f5920e5e640 1 -- 172.21.15.29:0/2159056361 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0c9780 con 0x7f591c15ad70 2023-10-16T15:55:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.685+0000 7f5920e5e640 1 --2- 172.21.15.29:0/2159056361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c069da0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f591c0ce870 tx=0x7f591002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1528c5c7fb3b1e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.685+0000 7f591b7fe640 1 -- 172.21.15.29:0/2159056361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910030860 con 0x7f591c15ad70 2023-10-16T15:55:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.685+0000 7f591b7fe640 1 -- 172.21.15.29:0/2159056361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59100309c0 con 0x7f591c15ad70 2023-10-16T15:55:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.685+0000 7f591b7fe640 1 -- 172.21.15.29:0/2159056361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910030d50 con 0x7f591c15ad70 2023-10-16T15:55:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 -- 172.21.15.29:0/2159056361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 msgr2=0x7f591c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 --2- 172.21.15.29:0/2159056361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c069da0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f591c0ce870 tx=0x7f591002b9a0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 -- 172.21.15.29:0/2159056361 shutdown_connections 2023-10-16T15:55:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 --2- 172.21.15.29:0/2159056361 >> v2:172.21.15.170:3300/0 conn(0x7f591c0c9ca0 0x7f591c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 --2- 172.21.15.29:0/2159056361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c069da0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 -- 172.21.15.29:0/2159056361 >> 172.21.15.29:0/2159056361 conn(0x7f591c0ce9b0 msgr2=0x7f591c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 -- 172.21.15.29:0/2159056361 shutdown_connections 2023-10-16T15:55:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.686+0000 7f59230c2640 1 -- 172.21.15.29:0/2159056361 wait complete. 2023-10-16T15:55:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.688+0000 7f59230c2640 1 Processor -- start 2023-10-16T15:55:08.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.688+0000 7f59230c2640 1 -- start start 2023-10-16T15:55:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f59230c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f591c0c9ca0 0x7f591c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f59230c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f591c0ca3b0 0x7f591c15ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f59230c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c0a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f59230c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c15d790 con 0x7f591c15ad70 2023-10-16T15:55:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f59230c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c15da70 con 0x7f591c0ca3b0 2023-10-16T15:55:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f59230c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c15dd50 con 0x7f591c0c9ca0 2023-10-16T15:55:08.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f592165f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c0a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f592165f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c0a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60114/0 (socket says 172.21.15.29:60114) 2023-10-16T15:55:08.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f592165f640 1 -- 172.21.15.29:0/898171831 learned_addr learned my addr 172.21.15.29:0/898171831 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:08.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f591bfff640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f591c0ca3b0 0x7f591c15ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.689+0000 7f5920e5e640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f591c0c9ca0 0x7f591c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f592165f640 1 -- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f591c0c9ca0 msgr2=0x7f591c157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f592165f640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f591c0c9ca0 0x7f591c157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f592165f640 1 -- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f591c0ca3b0 msgr2=0x7f591c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f592165f640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f591c0ca3b0 0x7f591c15ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f592165f640 1 -- 172.21.15.29:0/898171831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100094d0 con 0x7f591c15ad70 2023-10-16T15:55:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f591bfff640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f591c0ca3b0 0x7f591c15ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f592165f640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c0a8480 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f591400c780 tx=0x7f591400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400ce90 con 0x7f591c15ad70 2023-10-16T15:55:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400a640 con 0x7f591c15ad70 2023-10-16T15:55:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c0c6470 con 0x7f591c15ad70 2023-10-16T15:55:08.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400a9f0 con 0x7f591c15ad70 2023-10-16T15:55:08.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.690+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c158500 con 0x7f591c15ad70 2023-10-16T15:55:08.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.691+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c0c6990 con 0x7f591c15ad70 2023-10-16T15:55:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.693+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f591400ab50 con 0x7f591c15ad70 2023-10-16T15:55:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.694+0000 7f59197fa640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f590806b0b0 0x7f590806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.694+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f59140896e0 con 0x7f591c15ad70 2023-10-16T15:55:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.695+0000 7f5920e5e640 1 -- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f590806b0b0 msgr2=0x7f590806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.695+0000 7f5920e5e640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f590806b0b0 0x7f590806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.699+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f591c0c6990 con 0x7f591c15ad70 2023-10-16T15:55:08.719 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.718+0000 7f48b483a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:08.752 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.751+0000 7f42cd84c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:08.769 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.767+0000 7f48b483a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:08.806 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.805+0000 7f42cd84c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:08.829 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.827+0000 7f48b483a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:08.859 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.858+0000 7f42cd84c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:08.877 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.876+0000 7f48b483a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:08.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.888+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f591c0c6990 con 0x7f591c15ad70 2023-10-16T15:55:08.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.889+0000 7f59197fa640 1 -- 172.21.15.29:0/898171831 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v251) v1 ==== 93+0+10 (secure 0 0 0) 0x7f591c0c6990 con 0x7f591c15ad70 2023-10-16T15:55:08.890 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:55:08.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f590806b0b0 msgr2=0x7f590806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f590806b0b0 0x7f590806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 msgr2=0x7f591c0a8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c0a8480 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f591400c780 tx=0x7f591400cc40 comp rx=0 tx=0).stop 2023-10-16T15:55:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 shutdown_connections 2023-10-16T15:55:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f590806b0b0 0x7f590806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f591c15ad70 0x7f591c0a8480 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f591c0ca3b0 0x7f591c15ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 --2- 172.21.15.29:0/898171831 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f591c0c9ca0 0x7f591c157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 >> 172.21.15.29:0/898171831 conn(0x7f591c0ce9b0 msgr2=0x7f591c0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.891+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 shutdown_connections 2023-10-16T15:55:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:08.892+0000 7f59230c2640 1 -- 172.21.15.29:0/898171831 wait complete. 2023-10-16T15:55:08.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:55:08.926 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.925+0000 7f42cd84c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:08.979 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:08.978+0000 7f42cd84c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:08.989 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:08.987+0000 7f48b483a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:55:09.051 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.049+0000 7f48b483a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:09.097 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.096+0000 7f42cd84c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:55:09.100 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.098+0000 7f48b483a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:55:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.125+0000 7f5406abb640 1 Processor -- start 2023-10-16T15:55:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.125+0000 7f5406abb640 1 -- start start 2023-10-16T15:55:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.126+0000 7f5406abb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f54000b3420 0x7f54000b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.126+0000 7f5406abb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54000cfaf0 0x7f54000b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.126+0000 7f5406abb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54000d3980 con 0x7f54000cf710 2023-10-16T15:55:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.126+0000 7f5406abb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54000d3c60 con 0x7f54000cfaf0 2023-10-16T15:55:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.126+0000 7f5406abb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54000ce630 con 0x7f54000b3420 2023-10-16T15:55:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5404857640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f54000cf710 0x7f54000b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58480/0 (socket says 172.21.15.29:58480) 2023-10-16T15:55:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5404857640 1 -- 172.21.15.29:0/1883061070 learned_addr learned my addr 172.21.15.29:0/1883061070 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5405058640 1 --2- 172.21.15.29:0/1883061070 >> v2:172.21.15.29:3300/0 conn(0x7f54000cfaf0 0x7f54000b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f53fffff640 1 --2- 172.21.15.29:0/1883061070 >> v2:172.21.15.170:3300/0 conn(0x7f54000b3420 0x7f54000b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f53ff7fe640 1 -- 172.21.15.29:0/1883061070 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368790307 0 0) 0x7f54000d3980 con 0x7f54000cf710 2023-10-16T15:55:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f53ff7fe640 1 -- 172.21.15.29:0/1883061070 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f0003520 con 0x7f54000cf710 2023-10-16T15:55:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5405058640 1 -- 172.21.15.29:0/1883061070 >> v2:172.21.15.170:3300/0 conn(0x7f54000b3420 msgr2=0x7f54000b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5405058640 1 --2- 172.21.15.29:0/1883061070 >> v2:172.21.15.170:3300/0 conn(0x7f54000b3420 0x7f54000b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5405058640 1 -- 172.21.15.29:0/1883061070 >> v1:172.21.15.29:6789/0 conn(0x7f54000cf710 legacy=0x7f54000b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f5405058640 1 -- 172.21.15.29:0/1883061070 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54000ce910 con 0x7f54000cfaf0 2023-10-16T15:55:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.127+0000 7f53fffff640 1 --2- 172.21.15.29:0/1883061070 >> v2:172.21.15.170:3300/0 conn(0x7f54000b3420 0x7f54000b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.129+0000 7f5405058640 1 --2- 172.21.15.29:0/1883061070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000cfaf0 0x7f54000b6200 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f540008d210 tx=0x7f53f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8800a235a6cc40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.129+0000 7f53ff7fe640 1 -- 172.21.15.29:0/1883061070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f802cbf0 con 0x7f54000cfaf0 2023-10-16T15:55:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.129+0000 7f53ff7fe640 1 -- 172.21.15.29:0/1883061070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f802cd50 con 0x7f54000cfaf0 2023-10-16T15:55:09.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.129+0000 7f53ff7fe640 1 -- 172.21.15.29:0/1883061070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8034640 con 0x7f54000cfaf0 2023-10-16T15:55:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 -- 172.21.15.29:0/1883061070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000cfaf0 msgr2=0x7f54000b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 --2- 172.21.15.29:0/1883061070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000cfaf0 0x7f54000b6200 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f540008d210 tx=0x7f53f802be40 comp rx=0 tx=0).stop 2023-10-16T15:55:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 -- 172.21.15.29:0/1883061070 shutdown_connections 2023-10-16T15:55:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 --2- 172.21.15.29:0/1883061070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000cfaf0 0x7f54000b6200 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 --2- 172.21.15.29:0/1883061070 >> v2:172.21.15.170:3300/0 conn(0x7f54000b3420 0x7f54000b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 -- 172.21.15.29:0/1883061070 >> 172.21.15.29:0/1883061070 conn(0x7f54000d2cc0 msgr2=0x7f54000d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 -- 172.21.15.29:0/1883061070 shutdown_connections 2023-10-16T15:55:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.130+0000 7f5406abb640 1 -- 172.21.15.29:0/1883061070 wait complete. 2023-10-16T15:55:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.132+0000 7f5406abb640 1 Processor -- start 2023-10-16T15:55:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.132+0000 7f5406abb640 1 -- start start 2023-10-16T15:55:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5406abb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 0x7f54000ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5406abb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f54000cf710 0x7f54000cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5406abb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f54000cfaf0 0x7f5400149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5406abb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54000c9110 con 0x7f54000b3420 2023-10-16T15:55:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5406abb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54000b05c0 con 0x7f54000cf710 2023-10-16T15:55:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5406abb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540014a4f0 con 0x7f54000cfaf0 2023-10-16T15:55:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5404857640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 0x7f54000ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5404857640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 0x7f54000ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60138/0 (socket says 172.21.15.29:60138) 2023-10-16T15:55:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.133+0000 7f5404857640 1 -- 172.21.15.29:0/2097322908 learned_addr learned my addr 172.21.15.29:0/2097322908 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f53fffff640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f54000cf710 0x7f54000cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5405058640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f54000cfaf0 0x7f5400149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5404857640 1 -- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f54000cfaf0 msgr2=0x7f5400149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5404857640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f54000cfaf0 0x7f5400149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5404857640 1 -- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f54000cf710 msgr2=0x7f54000cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5404857640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f54000cf710 0x7f54000cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5404857640 1 -- 172.21.15.29:0/2097322908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8009a70 con 0x7f54000b3420 2023-10-16T15:55:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5404857640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 0x7f54000ca7a0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c400 tx=0x7f53f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40074e0 con 0x7f54000b3420 2023-10-16T15:55:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f40057a0 con 0x7f54000b3420 2023-10-16T15:55:09.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4007660 con 0x7f54000b3420 2023-10-16T15:55:09.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54000cc700 con 0x7f54000b3420 2023-10-16T15:55:09.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.134+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54000cccc0 con 0x7f54000b3420 2023-10-16T15:55:09.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.136+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54000b0780 con 0x7f54000b3420 2023-10-16T15:55:09.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.137+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f53f4005b50 con 0x7f54000b3420 2023-10-16T15:55:09.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.138+0000 7f53fd7fa640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 0x7f53e406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.138+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f53f4089530 con 0x7f54000b3420 2023-10-16T15:55:09.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.138+0000 7f53fffff640 1 -- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 msgr2=0x7f53e406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:09.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.138+0000 7f53fffff640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 0x7f53e406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:09.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.142+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f53f4053d40 con 0x7f54000b3420 2023-10-16T15:55:09.157 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.156+0000 7f48b483a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:55:09.166 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.165+0000 7f42cd84c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:09.207 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.205+0000 7f48b483a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:55:09.219 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.218+0000 7f42cd84c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:55:09.279 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.278+0000 7f42cd84c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:55:09.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.335+0000 7f42cd84c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:55:09.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.339+0000 7f53fffff640 1 -- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 msgr2=0x7f53e406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:09.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.339+0000 7f53fffff640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 0x7f53e406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:09.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.339+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f540004e160 con 0x7f54000b3420 2023-10-16T15:55:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.340+0000 7f53fd7fa640 1 -- 172.21.15.29:0/2097322908 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v251) v1 ==== 104+0+11 (secure 0 0 0) 0x7f53f4057360 con 0x7f54000b3420 2023-10-16T15:55:09.341 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:55:09.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.342+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 msgr2=0x7f53e406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:09.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.342+0000 7f5406abb640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 0x7f53e406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.342+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 msgr2=0x7f54000ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.342+0000 7f5406abb640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 0x7f54000ca7a0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c400 tx=0x7f53f400c8c0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 shutdown_connections 2023-10-16T15:55:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f53e406b0e0 0x7f53e406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f54000cfaf0 0x7f5400149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f54000cf710 0x7f54000cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 --2- 172.21.15.29:0/2097322908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54000b3420 0x7f54000ca7a0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 >> 172.21.15.29:0/2097322908 conn(0x7f54000d2cc0 msgr2=0x7f540006a410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 shutdown_connections 2023-10-16T15:55:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.343+0000 7f5406abb640 1 -- 172.21.15.29:0/2097322908 wait complete. 2023-10-16T15:55:09.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:55:09.594 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.592+0000 7f48b483a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:55:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.617+0000 7f050b047640 1 Processor -- start 2023-10-16T15:55:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.617+0000 7f050b047640 1 -- start start 2023-10-16T15:55:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.619+0000 7f050b047640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05040b8480 0x7f05040cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.619+0000 7f050b047640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05040d0be0 0x7f05040cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.619+0000 7f050b047640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050415d150 con 0x7f050415cd70 2023-10-16T15:55:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.619+0000 7f050b047640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05040d1180 con 0x7f05040b8480 2023-10-16T15:55:09.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.619+0000 7f050b047640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05040d1460 con 0x7f05040d0be0 2023-10-16T15:55:09.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.619+0000 7f0509844640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05040b8480 0x7f05040cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f050a846640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05040d0be0 0x7f05040cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f0509844640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f05040b8480 0x7f05040cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:40148/0 (socket says 172.21.15.29:40148) 2023-10-16T15:55:09.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f0509844640 1 -- 172.21.15.29:0/544710678 learned_addr learned my addr 172.21.15.29:0/544710678 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f0509844640 1 -- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05040d0be0 msgr2=0x7f05040cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f0509844640 1 --2- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05040d0be0 0x7f05040cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f0509844640 1 -- 172.21.15.29:0/544710678 >> v1:172.21.15.170:6789/0 conn(0x7f050415cd70 legacy=0x7f05040d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.620+0000 7f0509844640 1 -- 172.21.15.29:0/544710678 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05040d1740 con 0x7f05040b8480 2023-10-16T15:55:09.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.622+0000 7f0509844640 1 --2- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040b8480 0x7f05040cd7c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009870 tx=0x7f04f402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4019895a18ac0a5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:09.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.622+0000 7f0509043640 1 -- 172.21.15.29:0/544710678 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4030840 con 0x7f05040b8480 2023-10-16T15:55:09.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.622+0000 7f0509043640 1 -- 172.21.15.29:0/544710678 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f40309a0 con 0x7f05040b8480 2023-10-16T15:55:09.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.622+0000 7f0509043640 1 -- 172.21.15.29:0/544710678 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4030d30 con 0x7f05040b8480 2023-10-16T15:55:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 -- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040b8480 msgr2=0x7f05040cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 --2- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040b8480 0x7f05040cd7c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009870 tx=0x7f04f402b980 comp rx=0 tx=0).stop 2023-10-16T15:55:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 -- 172.21.15.29:0/544710678 shutdown_connections 2023-10-16T15:55:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 --2- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f05040d0be0 0x7f05040cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 --2- 172.21.15.29:0/544710678 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040b8480 0x7f05040cd7c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 -- 172.21.15.29:0/544710678 >> 172.21.15.29:0/544710678 conn(0x7f05040cf570 msgr2=0x7f05040cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 -- 172.21.15.29:0/544710678 shutdown_connections 2023-10-16T15:55:09.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.623+0000 7f050b047640 1 -- 172.21.15.29:0/544710678 wait complete. 2023-10-16T15:55:09.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.626+0000 7f050b047640 1 Processor -- start 2023-10-16T15:55:09.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.626+0000 7f050b047640 1 -- start start 2023-10-16T15:55:09.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050b047640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 0x7f05041f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050b047640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040d0be0 0x7f05040cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050b047640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f050415cd70 0x7f05040cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050b047640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050415e430 con 0x7f05040b8480 2023-10-16T15:55:09.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050b047640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05040cb130 con 0x7f05040d0be0 2023-10-16T15:55:09.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050b047640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050415e6a0 con 0x7f050415cd70 2023-10-16T15:55:09.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050a045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 0x7f05041f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050a045640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 0x7f05041f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60148/0 (socket says 172.21.15.29:60148) 2023-10-16T15:55:09.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.627+0000 7f050a045640 1 -- 172.21.15.29:0/951621896 learned_addr learned my addr 172.21.15.29:0/951621896 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:09.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f0509844640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040d0be0 0x7f05040cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a846640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f050415cd70 0x7f05040cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:09.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a045640 1 -- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f050415cd70 msgr2=0x7f05040cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a045640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f050415cd70 0x7f05040cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a045640 1 -- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040d0be0 msgr2=0x7f05040cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a045640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040d0be0 0x7f05040cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a045640 1 -- 172.21.15.29:0/951621896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40094d0 con 0x7f05040b8480 2023-10-16T15:55:09.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050a045640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 0x7f05041f1bf0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f050000c450 tx=0x7f050000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:09.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05000052d0 con 0x7f05040b8480 2023-10-16T15:55:09.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0500005430 con 0x7f05040b8480 2023-10-16T15:55:09.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500012870 con 0x7f05040b8480 2023-10-16T15:55:09.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.628+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050415e980 con 0x7f05040b8480 2023-10-16T15:55:09.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.629+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05040ccd90 con 0x7f05040b8480 2023-10-16T15:55:09.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.630+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050415eb70 con 0x7f05040b8480 2023-10-16T15:55:09.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.631+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f05000129d0 con 0x7f05040b8480 2023-10-16T15:55:09.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.631+0000 7f04faffd640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f04e806b0e0 0x7f04e806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:09.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.632+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f0500089ba0 con 0x7f05040b8480 2023-10-16T15:55:09.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.632+0000 7f0509844640 1 -- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f04e806b0e0 msgr2=0x7f04e806d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:09.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.632+0000 7f0509844640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f04e806b0e0 0x7f04e806d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:09.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.636+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05000542a0 con 0x7f05040b8480 2023-10-16T15:55:09.755 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.754+0000 7f48b483a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:55:09.765 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.765+0000 7f42cd84c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:55:09.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.828+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f050404e9b0 con 0x7f05040b8480 2023-10-16T15:55:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.829+0000 7f04faffd640 1 -- 172.21.15.29:0/951621896 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v251) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05000578c0 con 0x7f05040b8480 2023-10-16T15:55:09.830 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:55:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.831+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f04e806b0e0 msgr2=0x7f04e806d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.831+0000 7f050b047640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f04e806b0e0 0x7f04e806d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.831+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 msgr2=0x7f05041f1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.831+0000 7f050b047640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 0x7f05041f1bf0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f050000c450 tx=0x7f050000c910 comp rx=0 tx=0).stop 2023-10-16T15:55:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 shutdown_connections 2023-10-16T15:55:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f04e806b0e0 0x7f04e806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f050415cd70 0x7f05040cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f05040d0be0 0x7f05040cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 --2- 172.21.15.29:0/951621896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05040b8480 0x7f05041f1bf0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:09.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 >> 172.21.15.29:0/951621896 conn(0x7f05040cf570 msgr2=0x7f05040b7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:09.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 shutdown_connections 2023-10-16T15:55:09.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:09.832+0000 7f050b047640 1 -- 172.21.15.29:0/951621896 wait complete. 2023-10-16T15:55:09.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T15:55:09.863 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.861+0000 7f48b483a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:55:09.913 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:09.911+0000 7f48b483a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:55:09.973 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:09.972+0000 7f42cd84c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:55:10.020 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:10.018+0000 7f48b483a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:10.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.072+0000 7f9d394ec640 1 Processor -- start 2023-10-16T15:55:10.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.072+0000 7f9d394ec640 1 -- start start 2023-10-16T15:55:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d394ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 0x7f9d340c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d394ec640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9d34065ab0 0x7f9d340cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d394ec640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d340acd70 con 0x7f9d3406d600 2023-10-16T15:55:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d394ec640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d340ac4a0 con 0x7f9d34065ab0 2023-10-16T15:55:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d394ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d34065f40 con 0x7f9d3406d220 2023-10-16T15:55:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d33fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 0x7f9d340c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d337fe640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9d34065ab0 0x7f9d340cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d33fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 0x7f9d340c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:56510/0 (socket says 172.21.15.29:56510) 2023-10-16T15:55:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.074+0000 7f9d33fff640 1 -- 172.21.15.29:0/246141715 learned_addr learned my addr 172.21.15.29:0/246141715 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.075+0000 7f9d33fff640 1 -- 172.21.15.29:0/246141715 >> v2:172.21.15.170:3300/0 conn(0x7f9d34065ab0 msgr2=0x7f9d340cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.075+0000 7f9d33fff640 1 --2- 172.21.15.29:0/246141715 >> v2:172.21.15.170:3300/0 conn(0x7f9d34065ab0 0x7f9d340cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.075+0000 7f9d33fff640 1 -- 172.21.15.29:0/246141715 >> v1:172.21.15.170:6789/0 conn(0x7f9d3406d600 legacy=0x7f9d340ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.075+0000 7f9d33fff640 1 -- 172.21.15.29:0/246141715 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d340ac720 con 0x7f9d3406d220 2023-10-16T15:55:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.075+0000 7f9d337fe640 1 --2- 172.21.15.29:0/246141715 >> v2:172.21.15.170:3300/0 conn(0x7f9d34065ab0 0x7f9d340cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:55:10.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.076+0000 7f9d33fff640 1 --2- 172.21.15.29:0/246141715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 0x7f9d340c95e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009870 tx=0x7f9d2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbafd0df58b82087 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:10.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.077+0000 7f9d32ffd640 1 -- 172.21.15.29:0/246141715 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24030840 con 0x7f9d3406d220 2023-10-16T15:55:10.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.077+0000 7f9d32ffd640 1 -- 172.21.15.29:0/246141715 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d240309a0 con 0x7f9d3406d220 2023-10-16T15:55:10.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.077+0000 7f9d32ffd640 1 -- 172.21.15.29:0/246141715 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24030d30 con 0x7f9d3406d220 2023-10-16T15:55:10.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 -- 172.21.15.29:0/246141715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 msgr2=0x7f9d340c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 --2- 172.21.15.29:0/246141715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 0x7f9d340c95e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009870 tx=0x7f9d2402b980 comp rx=0 tx=0).stop 2023-10-16T15:55:10.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 -- 172.21.15.29:0/246141715 shutdown_connections 2023-10-16T15:55:10.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 --2- 172.21.15.29:0/246141715 >> v2:172.21.15.170:3300/0 conn(0x7f9d34065ab0 0x7f9d340cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 --2- 172.21.15.29:0/246141715 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d3406d220 0x7f9d340c95e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 -- 172.21.15.29:0/246141715 >> 172.21.15.29:0/246141715 conn(0x7f9d340b0a80 msgr2=0x7f9d340b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 -- 172.21.15.29:0/246141715 shutdown_connections 2023-10-16T15:55:10.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.078+0000 7f9d394ec640 1 -- 172.21.15.29:0/246141715 wait complete. 2023-10-16T15:55:10.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.080+0000 7f9d394ec640 1 Processor -- start 2023-10-16T15:55:10.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.081+0000 7f9d394ec640 1 -- start start 2023-10-16T15:55:10.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d394ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 0x7f9d341f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d394ec640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d3406d600 0x7f9d341f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d394ec640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d34099a10 0x7f9d340cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d394ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3409bbb0 con 0x7f9d3406d220 2023-10-16T15:55:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d394ec640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d341f6b60 con 0x7f9d3406d600 2023-10-16T15:55:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d394ec640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3409b570 con 0x7f9d34099a10 2023-10-16T15:55:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 0x7f9d341f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d33fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 0x7f9d341f6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60162/0 (socket says 172.21.15.29:60162) 2023-10-16T15:55:10.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d33fff640 1 -- 172.21.15.29:0/2244858840 learned_addr learned my addr 172.21.15.29:0/2244858840 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:10.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d38ceb640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d34099a10 0x7f9d340cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.082+0000 7f9d337fe640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d3406d600 0x7f9d341f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d33fff640 1 -- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d34099a10 msgr2=0x7f9d340cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d33fff640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d34099a10 0x7f9d340cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d33fff640 1 -- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d3406d600 msgr2=0x7f9d341f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d33fff640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d3406d600 0x7f9d341f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d33fff640 1 -- 172.21.15.29:0/2244858840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28009400 con 0x7f9d3406d220 2023-10-16T15:55:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d337fe640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d3406d600 0x7f9d341f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.083+0000 7f9d33fff640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 0x7f9d341f6620 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f9d24000c00 tx=0x7f9d24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.084+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24039c90 con 0x7f9d3406d220 2023-10-16T15:55:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.084+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d24039df0 con 0x7f9d3406d220 2023-10-16T15:55:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.084+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24043640 con 0x7f9d3406d220 2023-10-16T15:55:10.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.084+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d240094d0 con 0x7f9d3406d220 2023-10-16T15:55:10.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.084+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d340b4480 con 0x7f9d3406d220 2023-10-16T15:55:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.085+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3404e9b0 con 0x7f9d3406d220 2023-10-16T15:55:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.086+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f9d240437a0 con 0x7f9d3406d220 2023-10-16T15:55:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.087+0000 7f9d30ff9640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 0x7f9d2006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.087+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f9d2403e070 con 0x7f9d3406d220 2023-10-16T15:55:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.087+0000 7f9d337fe640 1 -- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 msgr2=0x7f9d2006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.087+0000 7f9d337fe640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 0x7f9d2006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:10.093 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.090+0000 7f42cd84c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:55:10.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.091+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d240777e0 con 0x7f9d3406d220 2023-10-16T15:55:10.128 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:10.126+0000 7f48b483a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:55:10.144 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.143+0000 7f42cd84c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:55:10.261 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.261+0000 7f42cd84c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:10.286 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:10.284+0000 7f48b483a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:10.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.287+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d340b4f10 con 0x7f9d3406d220 2023-10-16T15:55:10.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.288+0000 7f9d30ff9640 1 -- 172.21.15.29:0/2244858840 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v251) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9d2407ae00 con 0x7f9d3406d220 2023-10-16T15:55:10.289 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:55:10.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.288+0000 7f9d337fe640 1 -- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 msgr2=0x7f9d2006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:10.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.288+0000 7f9d337fe640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 0x7f9d2006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:10.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 msgr2=0x7f9d2006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:10.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 0x7f9d2006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 msgr2=0x7f9d341f6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 0x7f9d341f6620 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f9d24000c00 tx=0x7f9d24004730 comp rx=0 tx=0).stop 2023-10-16T15:55:10.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 shutdown_connections 2023-10-16T15:55:10.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f9d2006b0b0 0x7f9d2006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9d34099a10 0x7f9d340cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9d3406d600 0x7f9d341f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 --2- 172.21.15.29:0/2244858840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d3406d220 0x7f9d341f6620 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 >> 172.21.15.29:0/2244858840 conn(0x7f9d340b0a80 msgr2=0x7f9d340c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:10.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.290+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 shutdown_connections 2023-10-16T15:55:10.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.291+0000 7f9d394ec640 1 -- 172.21.15.29:0/2244858840 wait complete. 2023-10-16T15:55:10.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T15:55:10.335 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:10.334+0000 7f48b483a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:10.377 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.377+0000 7f42cd84c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:55:10.454 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:10.452+0000 7f48b483a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:55:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.521+0000 7f6285de4640 1 Processor -- start 2023-10-16T15:55:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.521+0000 7f6285de4640 1 -- start start 2023-10-16T15:55:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.523+0000 7f6285de4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f628015cd70 0x7f62800d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.523+0000 7f6285de4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f62800cded0 0x7f62800b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.523+0000 7f6285de4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628015d150 con 0x7f62800b8480 2023-10-16T15:55:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.523+0000 7f6285de4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62800d1180 con 0x7f62800cded0 2023-10-16T15:55:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.523+0000 7f6285de4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62800d1460 con 0x7f628015cd70 2023-10-16T15:55:10.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f6284de2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f628015cd70 0x7f62800d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f627ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f62800b8480 0x7f62800cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58482/0 (socket says 172.21.15.29:58482) 2023-10-16T15:55:10.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f627ffff640 1 -- 172.21.15.29:0/750664671 learned_addr learned my addr 172.21.15.29:0/750664671 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:10.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f62855e3640 1 --2- 172.21.15.29:0/750664671 >> v2:172.21.15.170:3300/0 conn(0x7f62800cded0 0x7f62800b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f627f7fe640 1 -- 172.21.15.29:0/750664671 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758250533 0 0) 0x7f628015d150 con 0x7f62800b8480 2023-10-16T15:55:10.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f627f7fe640 1 -- 172.21.15.29:0/750664671 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270003520 con 0x7f62800b8480 2023-10-16T15:55:10.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f6284de2640 1 -- 172.21.15.29:0/750664671 >> v2:172.21.15.170:3300/0 conn(0x7f62800cded0 msgr2=0x7f62800b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:10.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f6284de2640 1 --2- 172.21.15.29:0/750664671 >> v2:172.21.15.170:3300/0 conn(0x7f62800cded0 0x7f62800b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f6284de2640 1 -- 172.21.15.29:0/750664671 >> v1:172.21.15.29:6789/0 conn(0x7f62800b8480 legacy=0x7f62800cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.524+0000 7f6284de2640 1 -- 172.21.15.29:0/750664671 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800d1740 con 0x7f628015cd70 2023-10-16T15:55:10.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.526+0000 7f6284de2640 1 --2- 172.21.15.29:0/750664671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f628015cd70 0x7f62800d2220 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f62740097a0 tx=0x7f627402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50b311cff5e1ef80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:10.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.526+0000 7f627f7fe640 1 -- 172.21.15.29:0/750664671 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627402d040 con 0x7f628015cd70 2023-10-16T15:55:10.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.526+0000 7f627f7fe640 1 -- 172.21.15.29:0/750664671 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6274031810 con 0x7f628015cd70 2023-10-16T15:55:10.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.526+0000 7f627f7fe640 1 -- 172.21.15.29:0/750664671 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274031b80 con 0x7f628015cd70 2023-10-16T15:55:10.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 -- 172.21.15.29:0/750664671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f628015cd70 msgr2=0x7f62800d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 --2- 172.21.15.29:0/750664671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f628015cd70 0x7f62800d2220 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f62740097a0 tx=0x7f627402b8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 -- 172.21.15.29:0/750664671 shutdown_connections 2023-10-16T15:55:10.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 --2- 172.21.15.29:0/750664671 >> v2:172.21.15.170:3300/0 conn(0x7f62800cded0 0x7f62800b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 --2- 172.21.15.29:0/750664671 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f628015cd70 0x7f62800d2220 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 -- 172.21.15.29:0/750664671 >> 172.21.15.29:0/750664671 conn(0x7f62800cf570 msgr2=0x7f62800cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:10.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 -- 172.21.15.29:0/750664671 shutdown_connections 2023-10-16T15:55:10.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.527+0000 7f6285de4640 1 -- 172.21.15.29:0/750664671 wait complete. 2023-10-16T15:55:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.529+0000 7f6285de4640 1 Processor -- start 2023-10-16T15:55:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.529+0000 7f6285de4640 1 -- start start 2023-10-16T15:55:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f6285de4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62800b8480 0x7f62800cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f6285de4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62800cded0 0x7f62800d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f6285de4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 0x7f62800b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f6285de4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800ccb20 con 0x7f628014bca0 2023-10-16T15:55:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f6285de4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800a9990 con 0x7f62800cded0 2023-10-16T15:55:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f6285de4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628014c130 con 0x7f62800b8480 2023-10-16T15:55:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f62855e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 0x7f62800b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f62855e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 0x7f62800b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60164/0 (socket says 172.21.15.29:60164) 2023-10-16T15:55:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.530+0000 7f62855e3640 1 -- 172.21.15.29:0/2804450177 learned_addr learned my addr 172.21.15.29:0/2804450177 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f6284de2640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62800b8480 0x7f62800cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f627ffff640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62800cded0 0x7f62800d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f62855e3640 1 -- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62800b8480 msgr2=0x7f62800cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f62855e3640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62800b8480 0x7f62800cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f62855e3640 1 -- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62800cded0 msgr2=0x7f62800d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f62855e3640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62800cded0 0x7f62800d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f62855e3640 1 -- 172.21.15.29:0/2804450177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274009400 con 0x7f628014bca0 2023-10-16T15:55:10.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f62855e3640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 0x7f62800b0810 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f627800ec10 tx=0x7f627800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:10.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627800cd20 con 0x7f628014bca0 2023-10-16T15:55:10.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627800f040 con 0x7f628014bca0 2023-10-16T15:55:10.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628014c410 con 0x7f628014bca0 2023-10-16T15:55:10.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.531+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780146c0 con 0x7f628014bca0 2023-10-16T15:55:10.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.532+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628014c9d0 con 0x7f628014bca0 2023-10-16T15:55:10.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.533+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628004e9b0 con 0x7f628014bca0 2023-10-16T15:55:10.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.534+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f6278014a30 con 0x7f628014bca0 2023-10-16T15:55:10.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.535+0000 7f627d7fa640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f626406b0b0 0x7f626406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.535+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f627808ade0 con 0x7f628014bca0 2023-10-16T15:55:10.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.535+0000 7f6284de2640 1 -- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f626406b0b0 msgr2=0x7f626406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:10.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.535+0000 7f6284de2640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f626406b0b0 0x7f626406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:10.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.539+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f62780554e0 con 0x7f628014bca0 2023-10-16T15:55:10.546 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.545+0000 7f42cd84c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:10.599 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.598+0000 7f42cd84c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:10.727 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:10.726+0000 7f42cd84c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:55:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.729+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f628015b610 con 0x7f628014bca0 2023-10-16T15:55:10.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.730+0000 7f627d7fa640 1 -- 172.21.15.29:0/2804450177 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v251) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6278018020 con 0x7f628014bca0 2023-10-16T15:55:10.731 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:55:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f626406b0b0 msgr2=0x7f626406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f626406b0b0 0x7f626406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 msgr2=0x7f62800b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 0x7f62800b0810 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f627800ec10 tx=0x7f627800c490 comp rx=0 tx=0).stop 2023-10-16T15:55:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 shutdown_connections 2023-10-16T15:55:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f626406b0b0 0x7f626406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628014bca0 0x7f62800b0810 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62800cded0 0x7f62800d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 --2- 172.21.15.29:0/2804450177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f62800b8480 0x7f62800cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 >> 172.21.15.29:0/2804450177 conn(0x7f62800cf570 msgr2=0x7f62800b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 shutdown_connections 2023-10-16T15:55:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.732+0000 7f6285de4640 1 -- 172.21.15.29:0/2804450177 wait complete. 2023-10-16T15:55:10.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:55:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.961+0000 7f0f49155640 1 Processor -- start 2023-10-16T15:55:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.962+0000 7f0f49155640 1 -- start start 2023-10-16T15:55:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.963+0000 7f0f49155640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.963+0000 7f0f49155640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f440cfaf0 0x7f0f440b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.963+0000 7f0f49155640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f440d3980 con 0x7f0f440b38e0 2023-10-16T15:55:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.963+0000 7f0f49155640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f440d3c60 con 0x7f0f440cfaf0 2023-10-16T15:55:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.963+0000 7f0f49155640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440ce630 con 0x7f0f440cf710 2023-10-16T15:55:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f440cfaf0 0x7f0f440b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f440cfaf0 0x7f0f440b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60178/0 (socket says 172.21.15.29:60178) 2023-10-16T15:55:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 -- 172.21.15.29:0/3826058681 learned_addr learned my addr 172.21.15.29:0/3826058681 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f42d9d640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 -- 172.21.15.29:0/3826058681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 msgr2=0x7f0f440b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 -- 172.21.15.29:0/3826058681 >> v1:172.21.15.170:6789/0 conn(0x7f0f440b38e0 legacy=0x7f0f440d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f3bfff640 1 -- 172.21.15.29:0/3826058681 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f440ce910 con 0x7f0f440cfaf0 2023-10-16T15:55:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.964+0000 7f0f42d9d640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.965+0000 7f0f3bfff640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f440b3250 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009870 tx=0x7f0f2c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893d9786b059f8b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.966+0000 7f0f4259c640 1 -- 172.21.15.29:0/3826058681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c031860 con 0x7f0f440cfaf0 2023-10-16T15:55:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.966+0000 7f0f4259c640 1 -- 172.21.15.29:0/3826058681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c0319c0 con 0x7f0f440cfaf0 2023-10-16T15:55:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.966+0000 7f0f4259c640 1 -- 172.21.15.29:0/3826058681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c031d50 con 0x7f0f440cfaf0 2023-10-16T15:55:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.966+0000 7f0f49155640 1 -- 172.21.15.29:0/3826058681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 msgr2=0x7f0f440b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.966+0000 7f0f49155640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f440b3250 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009870 tx=0x7f0f2c02b900 comp rx=0 tx=0).stop 2023-10-16T15:55:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.967+0000 7f0f49155640 1 -- 172.21.15.29:0/3826058681 shutdown_connections 2023-10-16T15:55:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.967+0000 7f0f49155640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f440b3250 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.967+0000 7f0f49155640 1 --2- 172.21.15.29:0/3826058681 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.967+0000 7f0f49155640 1 -- 172.21.15.29:0/3826058681 >> 172.21.15.29:0/3826058681 conn(0x7f0f440d2cc0 msgr2=0x7f0f440d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:10.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.967+0000 7f0f49155640 1 -- 172.21.15.29:0/3826058681 shutdown_connections 2023-10-16T15:55:10.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.967+0000 7f0f49155640 1 -- 172.21.15.29:0/3826058681 wait complete. 2023-10-16T15:55:10.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.968+0000 7f0f49155640 1 Processor -- start 2023-10-16T15:55:10.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.968+0000 7f0f49155640 1 -- start start 2023-10-16T15:55:10.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.969+0000 7f0f49155640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f440b38e0 0x7f0f4415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.969+0000 7f0f49155640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.969+0000 7f0f49155640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f4414b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.969+0000 7f0f49155640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4414ace0 con 0x7f0f440cfaf0 2023-10-16T15:55:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.969+0000 7f0f49155640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4414a0d0 con 0x7f0f440b38e0 2023-10-16T15:55:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.969+0000 7f0f49155640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4414a3b0 con 0x7f0f440cf710 2023-10-16T15:55:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f4414b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f4414b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60192/0 (socket says 172.21.15.29:60192) 2023-10-16T15:55:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 -- 172.21.15.29:0/1847860131 learned_addr learned my addr 172.21.15.29:0/1847860131 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f42d9d640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f440b38e0 0x7f0f4415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f3bfff640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 -- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 msgr2=0x7f0f440c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 -- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f440b38e0 msgr2=0x7f0f4415cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f440b38e0 0x7f0f4415cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f4359e640 1 -- 172.21.15.29:0/1847860131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0094d0 con 0x7f0f440cfaf0 2023-10-16T15:55:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f3bfff640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.970+0000 7f0f42d9d640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f440b38e0 0x7f0f4415cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:55:10.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.971+0000 7f0f4359e640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f4414b170 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00c6b0 tx=0x7f0f3c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:10.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.971+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c003b70 con 0x7f0f440cfaf0 2023-10-16T15:55:10.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.971+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c003cf0 con 0x7f0f440cfaf0 2023-10-16T15:55:10.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.971+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c003ea0 con 0x7f0f440cfaf0 2023-10-16T15:55:10.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.971+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4414a690 con 0x7f0f440cfaf0 2023-10-16T15:55:10.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.971+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4415d330 con 0x7f0f440cfaf0 2023-10-16T15:55:10.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.973+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f440cef30 con 0x7f0f440cfaf0 2023-10-16T15:55:10.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.973+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f0f3c005310 con 0x7f0f440cfaf0 2023-10-16T15:55:10.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.974+0000 7f0f3b7fe640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 0x7f0f2406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:10.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.974+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7f0f3c089d20 con 0x7f0f440cfaf0 2023-10-16T15:55:10.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.974+0000 7f0f42d9d640 1 -- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 msgr2=0x7f0f2406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:10.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.974+0000 7f0f42d9d640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 0x7f0f2406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:10.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:10.978+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0f3c0544a0 con 0x7f0f440cfaf0 2023-10-16T15:55:11.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.175+0000 7f0f42d9d640 1 -- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 msgr2=0x7f0f2406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:11.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.175+0000 7f0f42d9d640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 0x7f0f2406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:11.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.225+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0f4404e160 con 0x7f0f440cfaf0 2023-10-16T15:55:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.226+0000 7f0f3b7fe640 1 -- 172.21.15.29:0/1847860131 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v251) v1 ==== 81+0+592 (secure 0 0 0) 0x7f0f3c057ac0 con 0x7f0f440cfaf0 2023-10-16T15:55:11.227 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2581880557 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.227 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/58791103 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.227 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/58791103 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.227 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3658934206 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.228 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3878756216 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.228 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3202676598 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.228 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2572859596 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.228 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2502304360 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.228 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2231350835 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.228 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3329304075 2023-10-17T15:55:05.561118+0000 2023-10-16T15:55:11.229 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2023-10-16T15:55:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.229+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 msgr2=0x7f0f2406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.229+0000 7f0f49155640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 0x7f0f2406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.229+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 msgr2=0x7f0f4414b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.229+0000 7f0f49155640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f4414b170 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00c6b0 tx=0x7f0f3c00cb70 comp rx=0 tx=0).stop 2023-10-16T15:55:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.229+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 shutdown_connections 2023-10-16T15:55:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.229+0000 7f0f49155640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7f0f2406b0b0 0x7f0f2406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.230+0000 7f0f49155640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f440cfaf0 0x7f0f4414b170 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.230+0000 7f0f49155640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f0f440cf710 0x7f0f440c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.230+0000 7f0f49155640 1 --2- 172.21.15.29:0/1847860131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f440b38e0 0x7f0f4415cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.230+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 >> 172.21.15.29:0/1847860131 conn(0x7f0f440d2cc0 msgr2=0x7f0f440b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.230+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 shutdown_connections 2023-10-16T15:55:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.230+0000 7f0f49155640 1 -- 172.21.15.29:0/1847860131 wait complete. 2023-10-16T15:55:11.250 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:55:11.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:11.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:55:11.296 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.294+0000 7f48b483a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:55:11.338 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:55:11.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:11.338 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:55:11.380 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:55:11.381 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:11.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:55:11.437 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:11.437 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:11.493 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:55:11.494 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:55:11.494 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:11.551 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:55:11.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:11.601 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.600+0000 7f48b483a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:55:11.615 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:11.614+0000 7f42cd84c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:55:11.660 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.659+0000 7f48b483a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:55:11.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.770+0000 7fa1eca00640 1 Processor -- start 2023-10-16T15:55:11.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.770+0000 7fa1eca00640 1 -- start start 2023-10-16T15:55:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.771+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e806c3f0 0x7fa1e806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.771+0000 7fa1eca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa1e815e0c0 0x7fa1e80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.771+0000 7fa1eca00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80cd630 con 0x7fa1e815e540 2023-10-16T15:55:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1eca00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e80ce1b0 con 0x7fa1e815e0c0 2023-10-16T15:55:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1eca00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e806cd10 con 0x7fa1e806c3f0 2023-10-16T15:55:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e5d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa1e815e0c0 0x7fa1e80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa1e815e540 0x7fa1e80b1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58484/0 (socket says 172.21.15.29:58484) 2023-10-16T15:55:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e6d9d640 1 -- 172.21.15.29:0/3088774316 learned_addr learned my addr 172.21.15.29:0/3088774316 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e659c640 1 --2- 172.21.15.29:0/3088774316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e806c3f0 0x7fa1e806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:11.775 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.771+0000 7f48b483a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:55:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e5d9b640 1 -- 172.21.15.29:0/3088774316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e806c3f0 msgr2=0x7fa1e806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e5d9b640 1 --2- 172.21.15.29:0/3088774316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e806c3f0 0x7fa1e806c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e5d9b640 1 -- 172.21.15.29:0/3088774316 >> v1:172.21.15.29:6789/0 conn(0x7fa1e815e540 legacy=0x7fa1e80b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e5d9b640 1 -- 172.21.15.29:0/3088774316 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80d2170 con 0x7fa1e815e0c0 2023-10-16T15:55:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.772+0000 7fa1e559a640 1 -- 172.21.15.29:0/3088774316 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898004907 0 0) 0x7fa1e80cd630 con 0x7fa1e815e540 2023-10-16T15:55:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1e5d9b640 1 --2- 172.21.15.29:0/3088774316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e815e0c0 0x7fa1e80b4730 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0097a0 tx=0x7fa1dc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1d78cfff57bba4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1e559a640 1 -- 172.21.15.29:0/3088774316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc02d040 con 0x7fa1e815e0c0 2023-10-16T15:55:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1e559a640 1 -- 172.21.15.29:0/3088774316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc031810 con 0x7fa1e815e0c0 2023-10-16T15:55:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1e559a640 1 -- 172.21.15.29:0/3088774316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc031ba0 con 0x7fa1e815e0c0 2023-10-16T15:55:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1eca00640 1 -- 172.21.15.29:0/3088774316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e815e0c0 msgr2=0x7fa1e80b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3088774316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e815e0c0 0x7fa1e80b4730 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0097a0 tx=0x7fa1dc02b8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.774+0000 7fa1eca00640 1 -- 172.21.15.29:0/3088774316 shutdown_connections 2023-10-16T15:55:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.775+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3088774316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e815e0c0 0x7fa1e80b4730 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.775+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3088774316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e806c3f0 0x7fa1e806c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.775+0000 7fa1eca00640 1 -- 172.21.15.29:0/3088774316 >> 172.21.15.29:0/3088774316 conn(0x7fa1e80d2cb0 msgr2=0x7fa1e80d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.775+0000 7fa1eca00640 1 -- 172.21.15.29:0/3088774316 shutdown_connections 2023-10-16T15:55:11.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.775+0000 7fa1eca00640 1 -- 172.21.15.29:0/3088774316 wait complete. 2023-10-16T15:55:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.776+0000 7fa1eca00640 1 Processor -- start 2023-10-16T15:55:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.776+0000 7fa1eca00640 1 -- start start 2023-10-16T15:55:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.777+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 0x7fa1e814b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:11.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.777+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e815e0c0 0x7fa1e80b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e815e540 0x7fa1e80b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1eca00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814caa0 con 0x7fa1e806c3f0 2023-10-16T15:55:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1eca00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814be20 con 0x7fa1e815e0c0 2023-10-16T15:55:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1eca00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814c100 con 0x7fa1e815e540 2023-10-16T15:55:11.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 0x7fa1e814b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 0x7fa1e814b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60210/0 (socket says 172.21.15.29:60210) 2023-10-16T15:55:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 -- 172.21.15.29:0/3592461305 learned_addr learned my addr 172.21.15.29:0/3592461305 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e6d9d640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e815e540 0x7fa1e80b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 -- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e815e540 msgr2=0x7fa1e80b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e815e540 0x7fa1e80b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 -- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e815e0c0 msgr2=0x7fa1e80b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e815e0c0 0x7fa1e80b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.778+0000 7fa1e659c640 1 -- 172.21.15.29:0/3592461305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009400 con 0x7fa1e806c3f0 2023-10-16T15:55:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.779+0000 7fa1e659c640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 0x7fa1e814b520 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8007e40 tx=0x7fa1d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.779+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8014620 con 0x7fa1e806c3f0 2023-10-16T15:55:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.779+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d800ee90 con 0x7fa1e806c3f0 2023-10-16T15:55:11.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.779+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80149c0 con 0x7fa1e806c3f0 2023-10-16T15:55:11.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.779+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e814c3e0 con 0x7fa1e806c3f0 2023-10-16T15:55:11.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.779+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e814ba60 con 0x7fa1e806c3f0 2023-10-16T15:55:11.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.780+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e804e160 con 0x7fa1e806c3f0 2023-10-16T15:55:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.781+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fa1d8014b20 con 0x7fa1e806c3f0 2023-10-16T15:55:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.783+0000 7fa1d6ffd640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fa1c806b0b0 0x7fa1c806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.783+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7031+0+0 (secure 0 0 0) 0x7fa1d808a880 con 0x7fa1e806c3f0 2023-10-16T15:55:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.783+0000 7fa1e5d9b640 1 -- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fa1c806b0b0 msgr2=0x7fa1c806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3738368341 2023-10-16T15:55:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.783+0000 7fa1e5d9b640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fa1c806b0b0 0x7fa1c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.786+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 143) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa1d80554a0 con 0x7fa1e806c3f0 2023-10-16T15:55:11.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.786+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fa1c806b0b0 msgr2=0x7fa1c806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.786+0000 7fa1d6ffd640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fa1c806b0b0 0x7fa1c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:11.793+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa1d8087c20 con 0x7fa1e806c3f0 2023-10-16T15:55:11.868 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.866+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.868 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.866+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.869 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.866+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.869 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.866+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.869 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.866+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.870 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.868+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.870 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.868+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.870 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.868+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.870 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.868+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.870 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.868+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.882 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.880+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.882 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.880+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.882 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.880+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.882 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.880+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.883 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.880+0000 7f4836b4c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.883 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.881+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.883 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.881+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.884 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.881+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.884 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.881+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.884 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:11.881+0000 7f4839b52640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:11.945 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:11.944+0000 7f42cd84c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:55:12.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:12.004+0000 7f42cd84c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:55:12.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.032+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e80d0c80 con 0x7fa1e806c3f0 2023-10-16T15:55:12.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.033+0000 7fa1d6ffd640 1 -- 172.21.15.29:0/3592461305 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6155 (secure 0 0 0) 0x7fa1d80586f0 con 0x7fa1e806c3f0 2023-10-16T15:55:12.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:12.034 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:12.035 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 130 2023-10-16T15:55:12.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.036+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 msgr2=0x7fa1e814b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.036+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 0x7fa1e814b520 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8007e40 tx=0x7fa1d800e4b0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 shutdown_connections 2023-10-16T15:55:12.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:6832/3738368341,v1:172.21.15.170:6833/3738368341] conn(0x7fa1c806b0b0 0x7fa1c806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa1e815e540 0x7fa1e80b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa1e815e0c0 0x7fa1e80b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 --2- 172.21.15.29:0/3592461305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1e806c3f0 0x7fa1e814b520 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 >> 172.21.15.29:0/3592461305 conn(0x7fa1e80d2cb0 msgr2=0x7fa1e80b84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:12.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 shutdown_connections 2023-10-16T15:55:12.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.037+0000 7fa1eca00640 1 -- 172.21.15.29:0/3592461305 wait complete. 2023-10-16T15:55:12.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:12.134 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:12.134+0000 7f42cd84c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:55:12.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.290+0000 7ff627b05640 1 Processor -- start 2023-10-16T15:55:12.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.290+0000 7ff627b05640 1 -- start start 2023-10-16T15:55:12.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.291+0000 7ff627b05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 0x7ff62015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.291+0000 7ff627b05640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62015ed60 con 0x7ff62015aaa0 2023-10-16T15:55:12.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.291+0000 7ff627b05640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6200707e0 con 0x7ff6200ce390 2023-10-16T15:55:12.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.291+0000 7ff627b05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62006b810 con 0x7ff62015ae80 2023-10-16T15:55:12.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.292+0000 7ff6258a1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff62015aaa0 0x7ff6200cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58496/0 (socket says 172.21.15.29:58496) 2023-10-16T15:55:12.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.292+0000 7ff6258a1640 1 -- 172.21.15.29:0/4095565189 learned_addr learned my addr 172.21.15.29:0/4095565189 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:12.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.292+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279003505 0 0) 0x7ff62015ed60 con 0x7ff62015aaa0 2023-10-16T15:55:12.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.292+0000 7ff6250a0640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 0x7ff62015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.292+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60c003520 con 0x7ff62015aaa0 2023-10-16T15:55:12.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2300981763 0 0) 0x7ff60c003520 con 0x7ff62015aaa0 2023-10-16T15:55:12.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 msgr2=0x7ff62015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:12.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 0x7ff62015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 >> v1:172.21.15.170:6789/0 conn(0x7ff6200ce390 legacy=0x7ff620070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:12.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62015ef90 con 0x7ff62015aaa0 2023-10-16T15:55:12.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7ff61c0019f0 con 0x7ff62015aaa0 2023-10-16T15:55:12.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 >> v1:172.21.15.29:6789/0 conn(0x7ff62015aaa0 legacy=0x7ff6200cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0039a0 0x7ff60c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 0x7ff60c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62006b810 con 0x7ff60c0047c0 2023-10-16T15:55:12.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c00bb40 con 0x7ff60c0039a0 2023-10-16T15:55:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c00bd00 con 0x7ff62015ae80 2023-10-16T15:55:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.293+0000 7ff6250a0640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6258a1640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0039a0 0x7ff60c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6260a2640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 0x7ff60c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6250a0640 1 -- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 msgr2=0x7ff60c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6250a0640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff62015ae80 0x7ff60c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6250a0640 1 -- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0039a0 msgr2=0x7ff60c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6250a0640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0039a0 0x7ff60c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.294+0000 7ff6250a0640 1 -- 172.21.15.29:0/4095565189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c003520 con 0x7ff60c0047c0 2023-10-16T15:55:12.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.295+0000 7ff6250a0640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7ff61000ba30 tx=0x7ff60c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcdef1052e064cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:12.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.296+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610032640 con 0x7ff60c0047c0 2023-10-16T15:55:12.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.296+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6100327a0 con 0x7ff60c0047c0 2023-10-16T15:55:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.296+0000 7ff62489f640 1 -- 172.21.15.29:0/4095565189 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610032b30 con 0x7ff60c0047c0 2023-10-16T15:55:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.296+0000 7ff627b05640 1 -- 172.21.15.29:0/4095565189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c0047c0 msgr2=0x7ff60c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.296+0000 7ff627b05640 1 --2- 172.21.15.29:0/4095565189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7ff61000ba30 tx=0x7ff60c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.297+0000 7ff6260a2640 1 -- 172.21.15.29:0/4095565189 reap_dead start 2023-10-16T15:55:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.297+0000 7ff627b05640 1 -- 172.21.15.29:0/4095565189 shutdown_connections 2023-10-16T15:55:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.297+0000 7ff627b05640 1 -- 172.21.15.29:0/4095565189 >> 172.21.15.29:0/4095565189 conn(0x7ff6200cb410 msgr2=0x7ff6200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.297+0000 7ff627b05640 1 -- 172.21.15.29:0/4095565189 shutdown_connections 2023-10-16T15:55:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.297+0000 7ff627b05640 1 -- 172.21.15.29:0/4095565189 wait complete. 2023-10-16T15:55:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.299+0000 7ff627b05640 1 Processor -- start 2023-10-16T15:55:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.299+0000 7ff627b05640 1 -- start start 2023-10-16T15:55:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff627b05640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0047c0 0x7ff6201f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff627b05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 0x7ff62015d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff627b05640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200c3510 0x7ff6200c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff627b05640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200c4ec0 con 0x7ff62015d550 2023-10-16T15:55:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff627b05640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200c4240 con 0x7ff60c0047c0 2023-10-16T15:55:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff627b05640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200c4520 con 0x7ff6200c3510 2023-10-16T15:55:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff6250a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 0x7ff62015d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff6250a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 0x7ff62015d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60230/0 (socket says 172.21.15.29:60230) 2023-10-16T15:55:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff6250a0640 1 -- 172.21.15.29:0/1870516943 learned_addr learned my addr 172.21.15.29:0/1870516943 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff6258a1640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0047c0 0x7ff6201f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.300+0000 7ff6260a2640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200c3510 0x7ff6200c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.301+0000 7ff6250a0640 1 -- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200c3510 msgr2=0x7ff6200c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.301+0000 7ff6250a0640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200c3510 0x7ff6200c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.301+0000 7ff6250a0640 1 -- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0047c0 msgr2=0x7ff6201f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.301+0000 7ff6250a0640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0047c0 0x7ff6201f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.301+0000 7ff6250a0640 1 -- 172.21.15.29:0/1870516943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200c4800 con 0x7ff62015d550 2023-10-16T15:55:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.301+0000 7ff6250a0640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 0x7ff62015d9d0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7ff61002bec0 tx=0x7ff610004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.302+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61003ba70 con 0x7ff62015d550 2023-10-16T15:55:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.302+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61003bbd0 con 0x7ff62015d550 2023-10-16T15:55:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.302+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610030410 con 0x7ff62015d550 2023-10-16T15:55:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.302+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6200c4ae0 con 0x7ff62015d550 2023-10-16T15:55:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.302+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6200c3f00 con 0x7ff62015d550 2023-10-16T15:55:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.303+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62004e160 con 0x7ff62015d550 2023-10-16T15:55:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.304+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff610030570 con 0x7ff62015d550 2023-10-16T15:55:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.305+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7ff6100ac0c0 con 0x7ff62015d550 2023-10-16T15:55:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.310+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff610076c20 con 0x7ff62015d550 2023-10-16T15:55:12.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.508+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6200cdc60 con 0x7ff62015d550 2023-10-16T15:55:12.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.512+0000 7ff6167fc640 1 -- 172.21.15.29:0/1870516943 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+21497 (secure 0 0 0) 0x7ff610036070 con 0x7ff62015d550 2023-10-16T15:55:12.513 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:12.513 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":252,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:11.776929+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.514+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 msgr2=0x7ff62015d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.514+0000 7ff627b05640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 0x7ff62015d9d0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7ff61002bec0 tx=0x7ff610004730 comp rx=0 tx=0).stop 2023-10-16T15:55:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 shutdown_connections 2023-10-16T15:55:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff62015d550 0x7ff62015d9d0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff6200c3510 0x7ff6200c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 --2- 172.21.15.29:0/1870516943 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff60c0047c0 0x7ff6201f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 >> 172.21.15.29:0/1870516943 conn(0x7ff6200cb410 msgr2=0x7ff62006fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 shutdown_connections 2023-10-16T15:55:12.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.515+0000 7ff627b05640 1 -- 172.21.15.29:0/1870516943 wait complete. 2023-10-16T15:55:12.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:54:36.572242+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:54:37.685848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T15:54:51.157450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T15:54:51.864591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:55:12.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:55:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.754+0000 7faffe660640 1 Processor -- start 2023-10-16T15:55:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.754+0000 7faffe660640 1 -- start start 2023-10-16T15:55:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.755+0000 7faffe660640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faff80b3ec0 0x7faff80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.755+0000 7faffe660640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b43c0 0x7faff80cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffe660640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff806be30 con 0x7faff80b0ea0 2023-10-16T15:55:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffe660640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff806ca00 con 0x7faff80b3ec0 2023-10-16T15:55:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffe660640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff80ada10 con 0x7faff80b43c0 2023-10-16T15:55:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faff80b3ec0 0x7faff80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faff80b3ec0 0x7faff80cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60244/0 (socket says 172.21.15.29:60244) 2023-10-16T15:55:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 -- 172.21.15.29:0/3214934272 learned_addr learned my addr 172.21.15.29:0/3214934272 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffde5f640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b43c0 0x7faff80cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 -- 172.21.15.29:0/3214934272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b43c0 msgr2=0x7faff80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b43c0 0x7faff80cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 -- 172.21.15.29:0/3214934272 >> v1:172.21.15.170:6789/0 conn(0x7faff80b0ea0 legacy=0x7faff815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffd65e640 1 -- 172.21.15.29:0/3214934272 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff806c060 con 0x7faff80b3ec0 2023-10-16T15:55:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.756+0000 7faffde5f640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b43c0 0x7faff80cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.758+0000 7faffd65e640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b3ec0 0x7faff80cd260 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7faff4009870 tx=0x7faff402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e83fb05db9d0fe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.758+0000 7fafeffff640 1 -- 172.21.15.29:0/3214934272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40317c0 con 0x7faff80b3ec0 2023-10-16T15:55:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.758+0000 7fafeffff640 1 -- 172.21.15.29:0/3214934272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4031920 con 0x7faff80b3ec0 2023-10-16T15:55:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.758+0000 7fafeffff640 1 -- 172.21.15.29:0/3214934272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4031cb0 con 0x7faff80b3ec0 2023-10-16T15:55:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.758+0000 7faffe660640 1 -- 172.21.15.29:0/3214934272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b3ec0 msgr2=0x7faff80cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.758+0000 7faffe660640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b3ec0 0x7faff80cd260 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7faff4009870 tx=0x7faff402b860 comp rx=0 tx=0).stop 2023-10-16T15:55:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.759+0000 7faffe660640 1 -- 172.21.15.29:0/3214934272 shutdown_connections 2023-10-16T15:55:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.759+0000 7faffe660640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b43c0 0x7faff80cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.759+0000 7faffe660640 1 --2- 172.21.15.29:0/3214934272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b3ec0 0x7faff80cd260 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.759+0000 7faffe660640 1 -- 172.21.15.29:0/3214934272 >> 172.21.15.29:0/3214934272 conn(0x7faff80b0170 msgr2=0x7faff80b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.759+0000 7faffe660640 1 -- 172.21.15.29:0/3214934272 shutdown_connections 2023-10-16T15:55:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.759+0000 7faffe660640 1 -- 172.21.15.29:0/3214934272 wait complete. 2023-10-16T15:55:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.760+0000 7faffe660640 1 Processor -- start 2023-10-16T15:55:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.761+0000 7faffe660640 1 -- start start 2023-10-16T15:55:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.761+0000 7faffe660640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faff80b0ea0 0x7faff815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.761+0000 7faffe660640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b3ec0 0x7faff80c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffe660640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 0x7faff8143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffe660640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff81f28c0 con 0x7faff80b43c0 2023-10-16T15:55:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffe660640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff815b160 con 0x7faff80b0ea0 2023-10-16T15:55:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffe660640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff815b430 con 0x7faff80b3ec0 2023-10-16T15:55:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 0x7faff8143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 0x7faff8143920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60254/0 (socket says 172.21.15.29:60254) 2023-10-16T15:55:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 -- 172.21.15.29:0/4121459096 learned_addr learned my addr 172.21.15.29:0/4121459096 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffd65e640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faff80b0ea0 0x7faff815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffce5d640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b3ec0 0x7faff80c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 -- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b3ec0 msgr2=0x7faff80c2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b3ec0 0x7faff80c2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 -- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faff80b0ea0 msgr2=0x7faff815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faff80b0ea0 0x7faff815ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.762+0000 7faffde5f640 1 -- 172.21.15.29:0/4121459096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40094d0 con 0x7faff80b43c0 2023-10-16T15:55:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.763+0000 7faffde5f640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 0x7faff8143920 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7faff000cea0 tx=0x7faff000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.763+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0003b90 con 0x7faff80b43c0 2023-10-16T15:55:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.763+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0003d10 con 0x7faff80b43c0 2023-10-16T15:55:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.763+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff815b710 con 0x7faff80b43c0 2023-10-16T15:55:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.763+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0004260 con 0x7faff80b43c0 2023-10-16T15:55:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.763+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff815bcd0 con 0x7faff80b43c0 2023-10-16T15:55:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.764+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff804e9b0 con 0x7faff80b43c0 2023-10-16T15:55:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.765+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 88940+0+0 (secure 0 0 0) 0x7faff00043c0 con 0x7faff80b43c0 2023-10-16T15:55:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.766+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7faff008a080 con 0x7faff80b43c0 2023-10-16T15:55:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.770+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faff0054ea0 con 0x7faff80b43c0 2023-10-16T15:55:12.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.793+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 144) v1 ==== 89067+0+0 (secure 0 0 0) 0x7faff0054950 con 0x7faff80b43c0 2023-10-16T15:55:12.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.794+0000 7fafedffb640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fafdc072de0 0x7fafdc0751d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:12.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.795+0000 7faffd65e640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fafdc072de0 0x7fafdc0751d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:12.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.804+0000 7faffd65e640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fafdc072de0 0x7fafdc0751d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faff4036000 tx=0x7faff403d040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:12.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.970+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faff80b0800 con 0x7faff80b43c0 2023-10-16T15:55:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.971+0000 7fafedffb640 1 -- 172.21.15.29:0/4121459096 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v252) v1 ==== 93+0+10 (secure 0 0 0) 0x7faff00730d0 con 0x7faff80b43c0 2023-10-16T15:55:12.972 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:55:12.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fafdc072de0 msgr2=0x7fafdc0751d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fafdc072de0 0x7fafdc0751d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faff4036000 tx=0x7faff403d040 comp rx=0 tx=0).stop 2023-10-16T15:55:12.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 msgr2=0x7faff8143920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 0x7faff8143920 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7faff000cea0 tx=0x7faff000cf90 comp rx=0 tx=0).stop 2023-10-16T15:55:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 shutdown_connections 2023-10-16T15:55:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fafdc072de0 0x7fafdc0751d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff80b43c0 0x7faff8143920 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7faff80b3ec0 0x7faff80c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 --2- 172.21.15.29:0/4121459096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7faff80b0ea0 0x7faff815ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 >> 172.21.15.29:0/4121459096 conn(0x7faff80b0170 msgr2=0x7faff80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 shutdown_connections 2023-10-16T15:55:12.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:12.974+0000 7faffe660640 1 -- 172.21.15.29:0/4121459096 wait complete. 2023-10-16T15:55:12.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:55:13.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.210+0000 7f441f244640 1 Processor -- start 2023-10-16T15:55:13.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.211+0000 7f441f244640 1 -- start start 2023-10-16T15:55:13.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.212+0000 7f441f244640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f441815ad70 0x7f4418069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.212+0000 7f441f244640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f44180c9ca0 0x7f441806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.212+0000 7f441f244640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180cb030 con 0x7f44180ca3b0 2023-10-16T15:55:13.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.212+0000 7f441f244640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44180c91c0 con 0x7f441815ad70 2023-10-16T15:55:13.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.212+0000 7f441f244640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44180c94a0 con 0x7f44180c9ca0 2023-10-16T15:55:13.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f441815ad70 0x7f4418069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f441815ad70 0x7f4418069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60268/0 (socket says 172.21.15.29:60268) 2023-10-16T15:55:13.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 -- 172.21.15.29:0/689201929 learned_addr learned my addr 172.21.15.29:0/689201929 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f4417fff640 1 --2- 172.21.15.29:0/689201929 >> v2:172.21.15.170:3300/0 conn(0x7f44180c9ca0 0x7f441806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 -- 172.21.15.29:0/689201929 >> v2:172.21.15.170:3300/0 conn(0x7f44180c9ca0 msgr2=0x7f441806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 --2- 172.21.15.29:0/689201929 >> v2:172.21.15.170:3300/0 conn(0x7f44180c9ca0 0x7f441806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 -- 172.21.15.29:0/689201929 >> v1:172.21.15.170:6789/0 conn(0x7f44180ca3b0 legacy=0x7f44180ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.213+0000 7f441cfe0640 1 -- 172.21.15.29:0/689201929 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180c9780 con 0x7f441815ad70 2023-10-16T15:55:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.214+0000 7f441cfe0640 1 --2- 172.21.15.29:0/689201929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f441815ad70 0x7f4418069da0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f44180ce870 tx=0x7f440c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a405735a33b4b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.215+0000 7f44177fe640 1 -- 172.21.15.29:0/689201929 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c031870 con 0x7f441815ad70 2023-10-16T15:55:13.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.215+0000 7f44177fe640 1 -- 172.21.15.29:0/689201929 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c0319d0 con 0x7f441815ad70 2023-10-16T15:55:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.215+0000 7f44177fe640 1 -- 172.21.15.29:0/689201929 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c031d60 con 0x7f441815ad70 2023-10-16T15:55:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 -- 172.21.15.29:0/689201929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f441815ad70 msgr2=0x7f4418069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 --2- 172.21.15.29:0/689201929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f441815ad70 0x7f4418069da0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f44180ce870 tx=0x7f440c02b910 comp rx=0 tx=0).stop 2023-10-16T15:55:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 -- 172.21.15.29:0/689201929 shutdown_connections 2023-10-16T15:55:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 --2- 172.21.15.29:0/689201929 >> v2:172.21.15.170:3300/0 conn(0x7f44180c9ca0 0x7f441806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 --2- 172.21.15.29:0/689201929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f441815ad70 0x7f4418069da0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 -- 172.21.15.29:0/689201929 >> 172.21.15.29:0/689201929 conn(0x7f44180ce9b0 msgr2=0x7f44180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 -- 172.21.15.29:0/689201929 shutdown_connections 2023-10-16T15:55:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.216+0000 7f441f244640 1 -- 172.21.15.29:0/689201929 wait complete. 2023-10-16T15:55:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.218+0000 7f441f244640 1 Processor -- start 2023-10-16T15:55:13.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.218+0000 7f441f244640 1 -- start start 2023-10-16T15:55:13.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441f244640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 0x7f4418151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441f244640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44180ca3b0 0x7f44180cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441f244640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f441815ad70 0x7f44180cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441f244640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44181f24c0 con 0x7f44180c9ca0 2023-10-16T15:55:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441f244640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441815ce60 con 0x7f441815ad70 2023-10-16T15:55:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441f244640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441815c2e0 con 0x7f44180ca3b0 2023-10-16T15:55:13.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441cfe0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 0x7f4418151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441cfe0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 0x7f4418151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60280/0 (socket says 172.21.15.29:60280) 2023-10-16T15:55:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.219+0000 7f441cfe0640 1 -- 172.21.15.29:0/3906349036 learned_addr learned my addr 172.21.15.29:0/3906349036 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441d7e1640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f441815ad70 0x7f44180cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441cfe0640 1 -- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44180ca3b0 msgr2=0x7f44180cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441cfe0640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44180ca3b0 0x7f44180cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441cfe0640 1 -- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f441815ad70 msgr2=0x7f44180cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441cfe0640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f441815ad70 0x7f44180cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441cfe0640 1 -- 172.21.15.29:0/3906349036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c0094d0 con 0x7f44180c9ca0 2023-10-16T15:55:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441d7e1640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f441815ad70 0x7f44180cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:13.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441cfe0640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 0x7f4418151d50 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f440c0090b0 tx=0x7f440c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c03ad20 con 0x7f44180c9ca0 2023-10-16T15:55:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c0443f0 con 0x7f44180c9ca0 2023-10-16T15:55:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c03aea0 con 0x7f44180c9ca0 2023-10-16T15:55:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.220+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441815c5c0 con 0x7f44180c9ca0 2023-10-16T15:55:13.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.221+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441815cb80 con 0x7f44180c9ca0 2023-10-16T15:55:13.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.222+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44181f2630 con 0x7f44180c9ca0 2023-10-16T15:55:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.223+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 144) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f440c04d450 con 0x7f44180c9ca0 2023-10-16T15:55:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.224+0000 7f44157fa640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f43fc06acc0 0x7f43fc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.224+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f440c02d040 con 0x7f44180c9ca0 2023-10-16T15:55:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.225+0000 7f4417fff640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f43fc06acc0 0x7f43fc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.226+0000 7f4417fff640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f43fc06acc0 0x7f43fc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f44181f5990 tx=0x7f4404008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:13.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.229+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44181f2630 con 0x7f44180c9ca0 2023-10-16T15:55:13.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.424+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44181f2600 con 0x7f44180c9ca0 2023-10-16T15:55:13.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.424+0000 7f44157fa640 1 -- 172.21.15.29:0/3906349036 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v252) v1 ==== 104+0+11 (secure 0 0 0) 0x7f440c0ab070 con 0x7f44180c9ca0 2023-10-16T15:55:13.425 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:55:13.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f43fc06acc0 msgr2=0x7f43fc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f43fc06acc0 0x7f43fc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f44181f5990 tx=0x7f4404008040 comp rx=0 tx=0).stop 2023-10-16T15:55:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 msgr2=0x7f4418151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 0x7f4418151d50 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f440c0090b0 tx=0x7f440c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 shutdown_connections 2023-10-16T15:55:13.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f43fc06acc0 0x7f43fc06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f441815ad70 0x7f44180cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44180ca3b0 0x7f44180cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 --2- 172.21.15.29:0/3906349036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44180c9ca0 0x7f4418151d50 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.427+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 >> 172.21.15.29:0/3906349036 conn(0x7f44180ce9b0 msgr2=0x7f44180acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:13.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.428+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 shutdown_connections 2023-10-16T15:55:13.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.428+0000 7f441f244640 1 -- 172.21.15.29:0/3906349036 wait complete. 2023-10-16T15:55:13.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:55:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.678+0000 7fd83754e640 1 Processor -- start 2023-10-16T15:55:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.679+0000 7fd83754e640 1 -- start start 2023-10-16T15:55:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.680+0000 7fd83754e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd83015cd70 0x7fd8300d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.680+0000 7fd83754e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd8300b8480 0x7fd8300cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.680+0000 7fd83754e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83015d150 con 0x7fd8300d0be0 2023-10-16T15:55:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.680+0000 7fd83754e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300d1180 con 0x7fd8300b8480 2023-10-16T15:55:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.680+0000 7fd83754e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300d1460 con 0x7fd83015cd70 2023-10-16T15:55:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.681+0000 7fd83654c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd83015cd70 0x7fd8300d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.681+0000 7fd835d4b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd8300b8480 0x7fd8300cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.681+0000 7fd835d4b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd8300b8480 0x7fd8300cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:40282/0 (socket says 172.21.15.29:40282) 2023-10-16T15:55:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.681+0000 7fd835d4b640 1 -- 172.21.15.29:0/3044800361 learned_addr learned my addr 172.21.15.29:0/3044800361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.682+0000 7fd835d4b640 1 -- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd83015cd70 msgr2=0x7fd8300d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.682+0000 7fd835d4b640 1 --2- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd83015cd70 0x7fd8300d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.682+0000 7fd835d4b640 1 -- 172.21.15.29:0/3044800361 >> v1:172.21.15.170:6789/0 conn(0x7fd8300d0be0 legacy=0x7fd8300b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.682+0000 7fd835d4b640 1 -- 172.21.15.29:0/3044800361 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300d1740 con 0x7fd8300b8480 2023-10-16T15:55:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.682+0000 7fd83654c640 1 --2- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd83015cd70 0x7fd8300d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:55:13.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.683+0000 7fd835d4b640 1 --2- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300b8480 0x7fd8300cd7c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd820009870 tx=0x7fd82002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18487bf89c6459e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:13.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.684+0000 7fd83554a640 1 -- 172.21.15.29:0/3044800361 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820030840 con 0x7fd8300b8480 2023-10-16T15:55:13.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.684+0000 7fd83554a640 1 -- 172.21.15.29:0/3044800361 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8200309a0 con 0x7fd8300b8480 2023-10-16T15:55:13.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.684+0000 7fd83554a640 1 -- 172.21.15.29:0/3044800361 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820030d30 con 0x7fd8300b8480 2023-10-16T15:55:13.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 -- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300b8480 msgr2=0x7fd8300cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 --2- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300b8480 0x7fd8300cd7c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd820009870 tx=0x7fd82002b980 comp rx=0 tx=0).stop 2023-10-16T15:55:13.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 -- 172.21.15.29:0/3044800361 shutdown_connections 2023-10-16T15:55:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 --2- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300b8480 0x7fd8300cd7c0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 --2- 172.21.15.29:0/3044800361 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd83015cd70 0x7fd8300d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 -- 172.21.15.29:0/3044800361 >> 172.21.15.29:0/3044800361 conn(0x7fd8300cf570 msgr2=0x7fd8300cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 -- 172.21.15.29:0/3044800361 shutdown_connections 2023-10-16T15:55:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.685+0000 7fd83754e640 1 -- 172.21.15.29:0/3044800361 wait complete. 2023-10-16T15:55:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.687+0000 7fd83754e640 1 Processor -- start 2023-10-16T15:55:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.688+0000 7fd83754e640 1 -- start start 2023-10-16T15:55:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd83754e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8300b8480 0x7fd830151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd83754e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300d0be0 0x7fd8300b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd83754e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 0x7fd8300b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd83754e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8301f5900 con 0x7fd83015cd70 2023-10-16T15:55:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd83754e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8301f5be0 con 0x7fd8300d0be0 2023-10-16T15:55:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd83754e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8301f5ec0 con 0x7fd8300b8480 2023-10-16T15:55:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd836d4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 0x7fd8300b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd836d4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 0x7fd8300b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60294/0 (socket says 172.21.15.29:60294) 2023-10-16T15:55:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.689+0000 7fd836d4d640 1 -- 172.21.15.29:0/2608084481 learned_addr learned my addr 172.21.15.29:0/2608084481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd835d4b640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300d0be0 0x7fd8300b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd83654c640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8300b8480 0x7fd830151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd836d4d640 1 -- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8300b8480 msgr2=0x7fd830151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd836d4d640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8300b8480 0x7fd830151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd836d4d640 1 -- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300d0be0 msgr2=0x7fd8300b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd836d4d640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300d0be0 0x7fd8300b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd836d4d640 1 -- 172.21.15.29:0/2608084481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c009400 con 0x7fd83015cd70 2023-10-16T15:55:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.690+0000 7fd836d4d640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 0x7fd8300b4b00 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fd82800c730 tx=0x7fd82800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.691+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828003b70 con 0x7fd83015cd70 2023-10-16T15:55:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.691+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd828003cf0 con 0x7fd83015cd70 2023-10-16T15:55:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.691+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828003ea0 con 0x7fd83015cd70 2023-10-16T15:55:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.691+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8200094d0 con 0x7fd83015cd70 2023-10-16T15:55:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.691+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8300caee0 con 0x7fd83015cd70 2023-10-16T15:55:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.692+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8300c7ff0 con 0x7fd83015cd70 2023-10-16T15:55:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.693+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 144) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd828005520 con 0x7fd83015cd70 2023-10-16T15:55:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.694+0000 7fd826ffd640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd81406acf0 0x7fd81406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.694+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7fd8280899a0 con 0x7fd83015cd70 2023-10-16T15:55:13.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.695+0000 7fd83654c640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd81406acf0 0x7fd81406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:13.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.696+0000 7fd83654c640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd81406acf0 0x7fd81406d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009c60 tx=0x7fd82c008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:13.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.698+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd8300c7ff0 con 0x7fd83015cd70 2023-10-16T15:55:13.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.811+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fd8280549b0 con 0x7fd83015cd70 2023-10-16T15:55:13.847 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:13.844+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.897+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd83004e9b0 con 0x7fd83015cd70 2023-10-16T15:55:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.898+0000 7fd826ffd640 1 -- 172.21.15.29:0/2608084481 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v252) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd828072130 con 0x7fd83015cd70 2023-10-16T15:55:13.899 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:55:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.900+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd81406acf0 msgr2=0x7fd81406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.900+0000 7fd83754e640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd81406acf0 0x7fd81406d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009c60 tx=0x7fd82c008040 comp rx=0 tx=0).stop 2023-10-16T15:55:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.900+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 msgr2=0x7fd8300b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.900+0000 7fd83754e640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 0x7fd8300b4b00 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fd82800c730 tx=0x7fd82800cbf0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 shutdown_connections 2023-10-16T15:55:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd81406acf0 0x7fd81406d1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83015cd70 0x7fd8300b4b00 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd8300d0be0 0x7fd8300b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 --2- 172.21.15.29:0/2608084481 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd8300b8480 0x7fd830151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 >> 172.21.15.29:0/2608084481 conn(0x7fd8300cf570 msgr2=0x7fd8300b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 shutdown_connections 2023-10-16T15:55:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:13.901+0000 7fd83754e640 1 -- 172.21.15.29:0/2608084481 wait complete. 2023-10-16T15:55:13.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T15:55:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.141+0000 7f5611154640 1 Processor -- start 2023-10-16T15:55:14.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.142+0000 7f5611154640 1 -- start start 2023-10-16T15:55:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f5611154640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f560c0b8480 0x7f560c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f5611154640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f560c0d0be0 0x7f560c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f5611154640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c15d150 con 0x7f560c15cd70 2023-10-16T15:55:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f5611154640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c0d1180 con 0x7f560c0b8480 2023-10-16T15:55:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f5611154640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c0d1460 con 0x7f560c0d0be0 2023-10-16T15:55:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f560b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f560c0b8480 0x7f560c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f560b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f560c0b8480 0x7f560c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37656/0 (socket says 172.21.15.29:37656) 2023-10-16T15:55:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.143+0000 7f560b7fe640 1 -- 172.21.15.29:0/1500388142 learned_addr learned my addr 172.21.15.29:0/1500388142 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.144+0000 7f560b7fe640 1 -- 172.21.15.29:0/1500388142 >> v2:172.21.15.170:3300/0 conn(0x7f560c0d0be0 msgr2=0x7f560c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.144+0000 7f560b7fe640 1 --2- 172.21.15.29:0/1500388142 >> v2:172.21.15.170:3300/0 conn(0x7f560c0d0be0 0x7f560c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.144+0000 7f560b7fe640 1 -- 172.21.15.29:0/1500388142 >> v1:172.21.15.170:6789/0 conn(0x7f560c15cd70 legacy=0x7f560c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.144+0000 7f560b7fe640 1 -- 172.21.15.29:0/1500388142 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c0d1740 con 0x7f560c0b8480 2023-10-16T15:55:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.145+0000 7f560b7fe640 1 --2- 172.21.15.29:0/1500388142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c0cd7c0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f560c0b8910 tx=0x7f560002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a910939b5a55b669 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.146+0000 7f560affd640 1 -- 172.21.15.29:0/1500388142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000317e0 con 0x7f560c0b8480 2023-10-16T15:55:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.146+0000 7f560affd640 1 -- 172.21.15.29:0/1500388142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600031940 con 0x7f560c0b8480 2023-10-16T15:55:14.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.146+0000 7f560affd640 1 -- 172.21.15.29:0/1500388142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600031cd0 con 0x7f560c0b8480 2023-10-16T15:55:14.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.146+0000 7f5611154640 1 -- 172.21.15.29:0/1500388142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 msgr2=0x7f560c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.146+0000 7f5611154640 1 --2- 172.21.15.29:0/1500388142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c0cd7c0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f560c0b8910 tx=0x7f560002b880 comp rx=0 tx=0).stop 2023-10-16T15:55:14.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.147+0000 7f5611154640 1 -- 172.21.15.29:0/1500388142 shutdown_connections 2023-10-16T15:55:14.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.147+0000 7f5611154640 1 --2- 172.21.15.29:0/1500388142 >> v2:172.21.15.170:3300/0 conn(0x7f560c0d0be0 0x7f560c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.147+0000 7f5611154640 1 --2- 172.21.15.29:0/1500388142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c0cd7c0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.147+0000 7f5611154640 1 -- 172.21.15.29:0/1500388142 >> 172.21.15.29:0/1500388142 conn(0x7f560c0cf570 msgr2=0x7f560c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.147+0000 7f5611154640 1 -- 172.21.15.29:0/1500388142 shutdown_connections 2023-10-16T15:55:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.147+0000 7f5611154640 1 -- 172.21.15.29:0/1500388142 wait complete. 2023-10-16T15:55:14.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.148+0000 7f5611154640 1 Processor -- start 2023-10-16T15:55:14.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.149+0000 7f5611154640 1 -- start start 2023-10-16T15:55:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.149+0000 7f5611154640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c1406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.149+0000 7f5611154640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f560c0d0be0 0x7f560c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f5611154640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f560c15cd70 0x7f560c13f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f5611154640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1f5a50 con 0x7f560c0b8480 2023-10-16T15:55:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f5611154640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c15d880 con 0x7f560c0d0be0 2023-10-16T15:55:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f5611154640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c13dac0 con 0x7f560c15cd70 2023-10-16T15:55:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c1406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c1406b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37666/0 (socket says 172.21.15.29:37666) 2023-10-16T15:55:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 -- 172.21.15.29:0/4122922358 learned_addr learned my addr 172.21.15.29:0/4122922358 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 -- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f560c15cd70 msgr2=0x7f560c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f560c15cd70 0x7f560c13f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 -- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f560c0d0be0 msgr2=0x7f560c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f560c0d0be0 0x7f560c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.150+0000 7f560bfff640 1 -- 172.21.15.29:0/4122922358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56000094d0 con 0x7f560c0b8480 2023-10-16T15:55:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.151+0000 7f560bfff640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c1406b0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c7a0 tx=0x7f55fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.151+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc003d80 con 0x7f560c0b8480 2023-10-16T15:55:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.151+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc005070 con 0x7f560c0b8480 2023-10-16T15:55:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.151+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc005420 con 0x7f560c0b8480 2023-10-16T15:55:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.151+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c13f940 con 0x7f560c0b8480 2023-10-16T15:55:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.151+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c15ddc0 con 0x7f560c0b8480 2023-10-16T15:55:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.153+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c04e9b0 con 0x7f560c0b8480 2023-10-16T15:55:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.153+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f55fc005580 con 0x7f560c0b8480 2023-10-16T15:55:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.154+0000 7f5608ff9640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f55f806b020 0x7f55f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.154+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f55fc0893e0 con 0x7f560c0b8480 2023-10-16T15:55:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.155+0000 7f560b7fe640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f55f806b020 0x7f55f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.156+0000 7f560b7fe640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f55f806b020 0x7f55f806d4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5600036000 tx=0x7f5600030000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.159+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f55fc053bc0 con 0x7f560c0b8480 2023-10-16T15:55:14.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.355+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f560c0cec40 con 0x7f560c0b8480 2023-10-16T15:55:14.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.356+0000 7f5608ff9640 1 -- 172.21.15.29:0/4122922358 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v252) v1 ==== 107+0+11 (secure 0 0 0) 0x7f55fc0571e0 con 0x7f560c0b8480 2023-10-16T15:55:14.358 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:55:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f55f806b020 msgr2=0x7f55f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f55f806b020 0x7f55f806d4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5600036000 tx=0x7f5600030000 comp rx=0 tx=0).stop 2023-10-16T15:55:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 msgr2=0x7f560c1406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c1406b0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c7a0 tx=0x7f55fc00cc60 comp rx=0 tx=0).stop 2023-10-16T15:55:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 shutdown_connections 2023-10-16T15:55:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f55f806b020 0x7f55f806d4e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f560c15cd70 0x7f560c13f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f560c0d0be0 0x7f560c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 --2- 172.21.15.29:0/4122922358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f560c0b8480 0x7f560c1406b0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 >> 172.21.15.29:0/4122922358 conn(0x7f560c0cf570 msgr2=0x7f560c0b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 shutdown_connections 2023-10-16T15:55:14.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.360+0000 7f5611154640 1 -- 172.21.15.29:0/4122922358 wait complete. 2023-10-16T15:55:14.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T15:55:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.606+0000 7f589a64b640 1 Processor -- start 2023-10-16T15:55:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.606+0000 7f589a64b640 1 -- start start 2023-10-16T15:55:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.608+0000 7f589a64b640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58940caac0 0x7f58940ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.608+0000 7f589a64b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940cb030 con 0x7f589406a4b0 2023-10-16T15:55:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.608+0000 7f589a64b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940c91c0 con 0x7f589415ad70 2023-10-16T15:55:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.608+0000 7f589a64b640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58940c94a0 con 0x7f58940caac0 2023-10-16T15:55:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.608+0000 7f588bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f589406a4b0 0x7f58940ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37584/0 (socket says 172.21.15.29:37584) 2023-10-16T15:55:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.608+0000 7f588bfff640 1 -- 172.21.15.29:0/4016345089 learned_addr learned my addr 172.21.15.29:0/4016345089 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f5898be8640 1 --2- 172.21.15.29:0/4016345089 >> v2:172.21.15.170:3300/0 conn(0x7f58940caac0 0x7f58940ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170405308 0 0) 0x7f58940cb030 con 0x7f589406a4b0 2023-10-16T15:55:14.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880003520 con 0x7f589406a4b0 2023-10-16T15:55:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801924617 0 0) 0x7f5880003520 con 0x7f589406a4b0 2023-10-16T15:55:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 >> v2:172.21.15.170:3300/0 conn(0x7f58940caac0 msgr2=0x7f58940ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f58937fe640 1 --2- 172.21.15.29:0/4016345089 >> v2:172.21.15.170:3300/0 conn(0x7f58940caac0 0x7f58940ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.609+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 >> v1:172.21.15.170:6789/0 conn(0x7f589415ad70 legacy=0x7f5894069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940c9780 con 0x7f589406a4b0 2023-10-16T15:55:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f587c001670 con 0x7f589406a4b0 2023-10-16T15:55:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 >> v1:172.21.15.29:6789/0 conn(0x7f589406a4b0 legacy=0x7f58940ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58800039c0 0x7f5880003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58800047c0 0x7f5880004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5880005180 0x7f588000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940c94a0 con 0x7f58800047c0 2023-10-16T15:55:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588000c080 con 0x7f58800039c0 2023-10-16T15:55:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588000c240 con 0x7f5880005180 2023-10-16T15:55:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f588bfff640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58800047c0 0x7f5880004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f588bfff640 1 -- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5880005180 msgr2=0x7f588000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f588bfff640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5880005180 0x7f588000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f588bfff640 1 -- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58800039c0 msgr2=0x7f5880003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:14.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.610+0000 7f5893fff640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58800039c0 0x7f5880003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.611+0000 7f588bfff640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58800039c0 0x7f5880003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.611+0000 7f588bfff640 1 -- 172.21.15.29:0/4016345089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880003520 con 0x7f58800047c0 2023-10-16T15:55:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.611+0000 7f5898be8640 1 -- 172.21.15.29:0/4016345089 reap_dead start 2023-10-16T15:55:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.611+0000 7f5893fff640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58800039c0 0x7f5880003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.612+0000 7f588bfff640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58800047c0 0x7f5880004c40 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f587c00c0e0 tx=0x7f58800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5e8c3ef31a2fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.612+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c005ad0 con 0x7f58800047c0 2023-10-16T15:55:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.612+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587c005c30 con 0x7f58800047c0 2023-10-16T15:55:14.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.612+0000 7f58937fe640 1 -- 172.21.15.29:0/4016345089 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c02ca00 con 0x7f58800047c0 2023-10-16T15:55:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 -- 172.21.15.29:0/4016345089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58800047c0 msgr2=0x7f5880004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58800047c0 0x7f5880004c40 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f587c00c0e0 tx=0x7f58800057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 -- 172.21.15.29:0/4016345089 shutdown_connections 2023-10-16T15:55:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 --2- 172.21.15.29:0/4016345089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58800047c0 0x7f5880004c40 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 -- 172.21.15.29:0/4016345089 >> 172.21.15.29:0/4016345089 conn(0x7f58940ce9b0 msgr2=0x7f58940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 -- 172.21.15.29:0/4016345089 shutdown_connections 2023-10-16T15:55:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.613+0000 7f589a64b640 1 -- 172.21.15.29:0/4016345089 wait complete. 2023-10-16T15:55:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.615+0000 7f589a64b640 1 Processor -- start 2023-10-16T15:55:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.615+0000 7f589a64b640 1 -- start start 2023-10-16T15:55:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f589a64b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58800047c0 0x7f5894157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f589a64b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58940ae7d0 0x7f58940aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f589a64b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 0x7f58940c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f589a64b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940af190 con 0x7f58940c5cc0 2023-10-16T15:55:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f589a64b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940cd080 con 0x7f58940ae7d0 2023-10-16T15:55:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f589a64b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58940cd360 con 0x7f58800047c0 2023-10-16T15:55:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f5898be8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 0x7f58940c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f5898be8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 0x7f58940c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37676/0 (socket says 172.21.15.29:37676) 2023-10-16T15:55:14.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.616+0000 7f5898be8640 1 -- 172.21.15.29:0/4145930316 learned_addr learned my addr 172.21.15.29:0/4145930316 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f588bfff640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58940ae7d0 0x7f58940aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f5898be8640 1 -- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58800047c0 msgr2=0x7f5894157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f5898be8640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58800047c0 0x7f5894157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f5898be8640 1 -- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58940ae7d0 msgr2=0x7f58940aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f5898be8640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58940ae7d0 0x7f58940aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f5898be8640 1 -- 172.21.15.29:0/4145930316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940cd640 con 0x7f58940c5cc0 2023-10-16T15:55:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f5898be8640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 0x7f58940c6130 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f589406d860 tx=0x7f588c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c010040 con 0x7f58940c5cc0 2023-10-16T15:55:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588c00bc80 con 0x7f58940c5cc0 2023-10-16T15:55:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c014400 con 0x7f58940c5cc0 2023-10-16T15:55:14.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58940cd920 con 0x7f58940c5cc0 2023-10-16T15:55:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.617+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58940c6670 con 0x7f58940c5cc0 2023-10-16T15:55:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.619+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58940cde40 con 0x7f58940c5cc0 2023-10-16T15:55:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.620+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f588c014560 con 0x7f58940c5cc0 2023-10-16T15:55:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.621+0000 7f58917fa640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f587406b050 0x7f587406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:14.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.621+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f588c08a930 con 0x7f58940c5cc0 2023-10-16T15:55:14.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.621+0000 7f5893fff640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f587406b050 0x7f587406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:14.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.623+0000 7f5893fff640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f587406b050 0x7f587406d510 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f58940c3130 tx=0x7f5884009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:14.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.625+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58940cde40 con 0x7f58940c5cc0 2023-10-16T15:55:14.755 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:14.753+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:14.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.825+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f58940cde40 con 0x7f58940c5cc0 2023-10-16T15:55:14.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.826+0000 7f58917fa640 1 -- 172.21.15.29:0/4145930316 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v252) v1 ==== 103+0+11 (secure 0 0 0) 0x7f58940cde40 con 0x7f58940c5cc0 2023-10-16T15:55:14.827 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:55:14.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f587406b050 msgr2=0x7f587406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f587406b050 0x7f587406d510 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f58940c3130 tx=0x7f5884009040 comp rx=0 tx=0).stop 2023-10-16T15:55:14.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 msgr2=0x7f58940c6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 0x7f58940c6130 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f589406d860 tx=0x7f588c00b480 comp rx=0 tx=0).stop 2023-10-16T15:55:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 shutdown_connections 2023-10-16T15:55:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f587406b050 0x7f587406d510 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58940c5cc0 0x7f58940c6130 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58940ae7d0 0x7f58940aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 --2- 172.21.15.29:0/4145930316 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58800047c0 0x7f5894157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 >> 172.21.15.29:0/4145930316 conn(0x7f58940ce9b0 msgr2=0x7f5894159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 shutdown_connections 2023-10-16T15:55:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:14.829+0000 7f589a64b640 1 -- 172.21.15.29:0/4145930316 wait complete. 2023-10-16T15:55:14.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:15.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.055+0000 7f3b8e3e2640 1 Processor -- start 2023-10-16T15:55:15.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.055+0000 7f3b8e3e2640 1 -- start start 2023-10-16T15:55:15.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b8e3e2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3b88159b20 0x7f3b8806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b8e3e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b88159f00 0x7f3b880b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b8e3e2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3b8806c120 0x7f3b880b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b8e3e2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8806f7f0 con 0x7f3b88159b20 2023-10-16T15:55:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b8e3e2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8806c5e0 con 0x7f3b8806c120 2023-10-16T15:55:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b8e3e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8806c8c0 con 0x7f3b88159f00 2023-10-16T15:55:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b87fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3b88159b20 0x7f3b8806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b87fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3b88159b20 0x7f3b8806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37684/0 (socket says 172.21.15.29:37684) 2023-10-16T15:55:15.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.057+0000 7f3b87fff640 1 -- 172.21.15.29:0/1826171577 learned_addr learned my addr 172.21.15.29:0/1826171577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:15.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b8c97f640 1 --2- 172.21.15.29:0/1826171577 >> v2:172.21.15.170:3300/0 conn(0x7f3b8806c120 0x7f3b880b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b877fe640 1 --2- 172.21.15.29:0/1826171577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b88159f00 0x7f3b880b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b87fff640 1 -- 172.21.15.29:0/1826171577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b88159f00 msgr2=0x7f3b880b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:15.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1826171577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b88159f00 0x7f3b880b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b87fff640 1 -- 172.21.15.29:0/1826171577 >> v2:172.21.15.170:3300/0 conn(0x7f3b8806c120 msgr2=0x7f3b880b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1826171577 >> v2:172.21.15.170:3300/0 conn(0x7f3b8806c120 0x7f3b880b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.058+0000 7f3b87fff640 1 -- 172.21.15.29:0/1826171577 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8806cba0 con 0x7f3b88159b20 2023-10-16T15:55:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.059+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1826171577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159b20 0x7f3b8806bbe0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f3b880c9270 tx=0x7f3b7802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a772636ff1aedb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1826171577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78031860 con 0x7f3b88159b20 2023-10-16T15:55:15.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1826171577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b780319c0 con 0x7f3b88159b20 2023-10-16T15:55:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1826171577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78031d50 con 0x7f3b88159b20 2023-10-16T15:55:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/1826171577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159b20 msgr2=0x7f3b8806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/1826171577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159b20 0x7f3b8806bbe0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f3b880c9270 tx=0x7f3b7802b900 comp rx=0 tx=0).stop 2023-10-16T15:55:15.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/1826171577 shutdown_connections 2023-10-16T15:55:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/1826171577 >> v2:172.21.15.170:3300/0 conn(0x7f3b8806c120 0x7f3b880b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/1826171577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b88159f00 0x7f3b880b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/1826171577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159b20 0x7f3b8806bbe0 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/1826171577 >> 172.21.15.29:0/1826171577 conn(0x7f3b880cb410 msgr2=0x7f3b880cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/1826171577 shutdown_connections 2023-10-16T15:55:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.060+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/1826171577 wait complete. 2023-10-16T15:55:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.062+0000 7f3b8e3e2640 1 Processor -- start 2023-10-16T15:55:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.062+0000 7f3b8e3e2640 1 -- start start 2023-10-16T15:55:15.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.063+0000 7f3b8e3e2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b8806c120 0x7f3b88157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.063+0000 7f3b8e3e2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b88159b20 0x7f3b880b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.063+0000 7f3b8e3e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 0x7f3b880b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.063+0000 7f3b8e3e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b88158500 con 0x7f3b88159f00 2023-10-16T15:55:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.063+0000 7f3b8e3e2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b880c6960 con 0x7f3b88159b20 2023-10-16T15:55:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.063+0000 7f3b8e3e2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b880c6c40 con 0x7f3b8806c120 2023-10-16T15:55:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 0x7f3b880b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 0x7f3b880b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37700/0 (socket says 172.21.15.29:37700) 2023-10-16T15:55:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 -- 172.21.15.29:0/4294167883 learned_addr learned my addr 172.21.15.29:0/4294167883 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b877fe640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b88159b20 0x7f3b880b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b87fff640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b8806c120 0x7f3b88157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 -- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b8806c120 msgr2=0x7f3b88157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b8806c120 0x7f3b88157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 -- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b88159b20 msgr2=0x7f3b880b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b88159b20 0x7f3b880b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b8c97f640 1 -- 172.21.15.29:0/4294167883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b780094d0 con 0x7f3b88159f00 2023-10-16T15:55:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.064+0000 7f3b877fe640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b88159b20 0x7f3b880b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.065+0000 7f3b8c97f640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 0x7f3b880b3110 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f3b80007de0 tx=0x7f3b8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.065+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80014620 con 0x7f3b88159f00 2023-10-16T15:55:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.065+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8000ee90 con 0x7f3b88159f00 2023-10-16T15:55:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.065+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b880c6f20 con 0x7f3b88159f00 2023-10-16T15:55:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.065+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800149e0 con 0x7f3b88159f00 2023-10-16T15:55:15.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.065+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b880c5c70 con 0x7f3b88159f00 2023-10-16T15:55:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.067+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8804e160 con 0x7f3b88159f00 2023-10-16T15:55:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.067+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f3b80014b40 con 0x7f3b88159f00 2023-10-16T15:55:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.068+0000 7f3b84ff9640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3b7406b020 0x7f3b7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.068+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f3b8008aa50 con 0x7f3b88159f00 2023-10-16T15:55:15.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.069+0000 7f3b87fff640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3b7406b020 0x7f3b7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.070+0000 7f3b87fff640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3b7406b020 0x7f3b7406d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b78036000 tx=0x7f3b7803c040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:15.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.072+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3b800551a0 con 0x7f3b88159f00 2023-10-16T15:55:15.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.311+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b8806d490 con 0x7f3b88159f00 2023-10-16T15:55:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.312+0000 7f3b84ff9640 1 -- 172.21.15.29:0/4294167883 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f3b800587c0 con 0x7f3b88159f00 2023-10-16T15:55:15.313 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:15.313 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 130 2023-10-16T15:55:15.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3b7406b020 msgr2=0x7f3b7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3b7406b020 0x7f3b7406d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b78036000 tx=0x7f3b7803c040 comp rx=0 tx=0).stop 2023-10-16T15:55:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 msgr2=0x7f3b880b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 0x7f3b880b3110 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f3b80007de0 tx=0x7f3b8000e4b0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 shutdown_connections 2023-10-16T15:55:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3b7406b020 0x7f3b7406d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b88159f00 0x7f3b880b3110 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3b88159b20 0x7f3b880b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 --2- 172.21.15.29:0/4294167883 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3b8806c120 0x7f3b88157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 >> 172.21.15.29:0/4294167883 conn(0x7f3b880cb410 msgr2=0x7f3b880ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 shutdown_connections 2023-10-16T15:55:15.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.315+0000 7f3b8e3e2640 1 -- 172.21.15.29:0/4294167883 wait complete. 2023-10-16T15:55:15.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:15.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.561+0000 7f493e225640 1 Processor -- start 2023-10-16T15:55:15.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.561+0000 7f493e225640 1 -- start start 2023-10-16T15:55:15.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.563+0000 7f493e225640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49380b8480 0x7f49380cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.563+0000 7f493e225640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49380d0be0 0x7f49380cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493e225640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493815d150 con 0x7f493815cd70 2023-10-16T15:55:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493e225640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49380d1180 con 0x7f49380b8480 2023-10-16T15:55:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493e225640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49380d1460 con 0x7f49380d0be0 2023-10-16T15:55:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49380b8480 0x7f49380cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49380b8480 0x7f49380cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37716/0 (socket says 172.21.15.29:37716) 2023-10-16T15:55:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 -- 172.21.15.29:0/3276925595 learned_addr learned my addr 172.21.15.29:0/3276925595 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493d223640 1 --1- 172.21.15.29:0/3276925595 >> v1:172.21.15.29:6789/0 conn(0x7f493815cd70 0x7f49380d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37598/0 (socket says 172.21.15.29:37598) 2023-10-16T15:55:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 -- 172.21.15.29:0/3276925595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49380d0be0 msgr2=0x7f49380cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493da24640 1 --2- 172.21.15.29:0/3276925595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49380d0be0 0x7f49380cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 --2- 172.21.15.29:0/3276925595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49380d0be0 0x7f49380cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 -- 172.21.15.29:0/3276925595 >> v1:172.21.15.29:6789/0 conn(0x7f493815cd70 legacy=0x7f49380d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f492ffff640 1 -- 172.21.15.29:0/3276925595 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443899539 0 0) 0x7f493815d150 con 0x7f493815cd70 2023-10-16T15:55:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.564+0000 7f493ca22640 1 -- 172.21.15.29:0/3276925595 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49380d1740 con 0x7f49380b8480 2023-10-16T15:55:15.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.566+0000 7f493ca22640 1 --2- 172.21.15.29:0/3276925595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380b8480 0x7f49380cd7c0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f4928009870 tx=0x7f492802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236fdf597da20405 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:15.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.566+0000 7f492ffff640 1 -- 172.21.15.29:0/3276925595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928030840 con 0x7f49380b8480 2023-10-16T15:55:15.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.566+0000 7f492ffff640 1 -- 172.21.15.29:0/3276925595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49280309a0 con 0x7f49380b8480 2023-10-16T15:55:15.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.566+0000 7f492ffff640 1 -- 172.21.15.29:0/3276925595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928030d30 con 0x7f49380b8480 2023-10-16T15:55:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.566+0000 7f493e225640 1 -- 172.21.15.29:0/3276925595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380b8480 msgr2=0x7f49380cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.566+0000 7f493e225640 1 --2- 172.21.15.29:0/3276925595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380b8480 0x7f49380cd7c0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f4928009870 tx=0x7f492802b980 comp rx=0 tx=0).stop 2023-10-16T15:55:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.567+0000 7f493e225640 1 -- 172.21.15.29:0/3276925595 shutdown_connections 2023-10-16T15:55:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.567+0000 7f493e225640 1 --2- 172.21.15.29:0/3276925595 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f49380d0be0 0x7f49380cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.567+0000 7f493e225640 1 --2- 172.21.15.29:0/3276925595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380b8480 0x7f49380cd7c0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.567+0000 7f493e225640 1 -- 172.21.15.29:0/3276925595 >> 172.21.15.29:0/3276925595 conn(0x7f49380cf570 msgr2=0x7f49380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:15.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.567+0000 7f493e225640 1 -- 172.21.15.29:0/3276925595 shutdown_connections 2023-10-16T15:55:15.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.567+0000 7f493e225640 1 -- 172.21.15.29:0/3276925595 wait complete. 2023-10-16T15:55:15.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.568+0000 7f493e225640 1 Processor -- start 2023-10-16T15:55:15.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.569+0000 7f493e225640 1 -- start start 2023-10-16T15:55:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.569+0000 7f493e225640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49380b8480 0x7f493814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493e225640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 0x7f4938157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493e225640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f493815cd70 0x7f4938159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493e225640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49380cc2d0 con 0x7f49380d0be0 2023-10-16T15:55:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493e225640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493814d470 con 0x7f49380b8480 2023-10-16T15:55:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493e225640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49380cb6c0 con 0x7f493815cd70 2023-10-16T15:55:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 0x7f4938157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 0x7f4938157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37720/0 (socket says 172.21.15.29:37720) 2023-10-16T15:55:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 -- 172.21.15.29:0/2276702117 learned_addr learned my addr 172.21.15.29:0/2276702117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493d223640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49380b8480 0x7f493814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493da24640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f493815cd70 0x7f4938159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 -- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f493815cd70 msgr2=0x7f4938159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f493815cd70 0x7f4938159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 -- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49380b8480 msgr2=0x7f493814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49380b8480 0x7f493814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493ca22640 1 -- 172.21.15.29:0/2276702117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280094d0 con 0x7f49380d0be0 2023-10-16T15:55:15.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.570+0000 7f493d223640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49380b8480 0x7f493814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:15.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.571+0000 7f493ca22640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 0x7f4938157350 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f4928000c00 tx=0x7f4928004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:15.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.571+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928030810 con 0x7f49380d0be0 2023-10-16T15:55:15.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.571+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4928043400 con 0x7f49380d0be0 2023-10-16T15:55:15.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.571+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49280437d0 con 0x7f49380d0be0 2023-10-16T15:55:15.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.571+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49380cb9a0 con 0x7f49380d0be0 2023-10-16T15:55:15.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.572+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49380cbf60 con 0x7f49380d0be0 2023-10-16T15:55:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.573+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49380cc120 con 0x7f49380d0be0 2023-10-16T15:55:15.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.574+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f4928043930 con 0x7f49380d0be0 2023-10-16T15:55:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.575+0000 7f492dffb640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f491c06b020 0x7f491c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.575+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f492803e070 con 0x7f49380d0be0 2023-10-16T15:55:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.576+0000 7f493d223640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f491c06b020 0x7f491c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:15.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.577+0000 7f493d223640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f491c06b020 0x7f491c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4934000f60 tx=0x7f4934009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:15.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.580+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49280779b0 con 0x7f49380d0be0 2023-10-16T15:55:15.660 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:15.657+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:15.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.777+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49380742a0 con 0x7f49380d0be0 2023-10-16T15:55:15.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.779+0000 7f492dffb640 1 -- 172.21.15.29:0/2276702117 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+21497 (secure 0 0 0) 0x7f492807afd0 con 0x7f49380d0be0 2023-10-16T15:55:15.780 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:15.780 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":252,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:11.776929+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:15.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f491c06b020 msgr2=0x7f491c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f491c06b020 0x7f491c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4934000f60 tx=0x7f4934009040 comp rx=0 tx=0).stop 2023-10-16T15:55:15.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 msgr2=0x7f4938157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:15.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 0x7f4938157350 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f4928000c00 tx=0x7f4928004730 comp rx=0 tx=0).stop 2023-10-16T15:55:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 shutdown_connections 2023-10-16T15:55:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f491c06b020 0x7f491c06d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49380d0be0 0x7f4938157350 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f493815cd70 0x7f4938159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 --2- 172.21.15.29:0/2276702117 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f49380b8480 0x7f493814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 >> 172.21.15.29:0/2276702117 conn(0x7f49380cf570 msgr2=0x7f49380b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 shutdown_connections 2023-10-16T15:55:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:15.782+0000 7f493e225640 1 -- 172.21.15.29:0/2276702117 wait complete. 2023-10-16T15:55:15.803 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:55:15.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:16.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.032+0000 7fd708bd0640 1 Processor -- start 2023-10-16T15:55:16.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.032+0000 7fd708bd0640 1 -- start start 2023-10-16T15:55:16.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.033+0000 7fd708bd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7040ce390 0x7fd70415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.033+0000 7fd708bd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70415ed60 con 0x7fd70415ae80 2023-10-16T15:55:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd708bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7040707e0 con 0x7fd70415aaa0 2023-10-16T15:55:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd708bd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd70406b810 con 0x7fd7040ce390 2023-10-16T15:55:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd702d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7040ce390 0x7fd70415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd701d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd70415ae80 0x7fd704070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37606/0 (socket says 172.21.15.29:37606) 2023-10-16T15:55:16.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd701d9b640 1 -- 172.21.15.29:0/1689512948 learned_addr learned my addr 172.21.15.29:0/1689512948 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:16.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd702d9d640 1 -- 172.21.15.29:0/1689512948 >> v1:172.21.15.170:6789/0 conn(0x7fd70415aaa0 legacy=0x7fd7040cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.034+0000 7fd70159a640 1 -- 172.21.15.29:0/1689512948 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661483883 0 0) 0x7fd70415ed60 con 0x7fd70415ae80 2023-10-16T15:55:16.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.035+0000 7fd702d9d640 1 -- 172.21.15.29:0/1689512948 >> v1:172.21.15.29:6789/0 conn(0x7fd70415ae80 legacy=0x7fd704070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.035+0000 7fd702d9d640 1 -- 172.21.15.29:0/1689512948 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70415ef90 con 0x7fd7040ce390 2023-10-16T15:55:16.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.036+0000 7fd702d9d640 1 --2- 172.21.15.29:0/1689512948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7040ce390 0x7fd70415e820 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097a0 tx=0x7fd6fc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91daeaaeaf618318 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:16.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd70159a640 1 -- 172.21.15.29:0/1689512948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc030610 con 0x7fd7040ce390 2023-10-16T15:55:16.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd70159a640 1 -- 172.21.15.29:0/1689512948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc030770 con 0x7fd7040ce390 2023-10-16T15:55:16.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd70159a640 1 -- 172.21.15.29:0/1689512948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc030b00 con 0x7fd7040ce390 2023-10-16T15:55:16.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd708bd0640 1 -- 172.21.15.29:0/1689512948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7040ce390 msgr2=0x7fd70415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1689512948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7040ce390 0x7fd70415e820 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097a0 tx=0x7fd6fc02b960 comp rx=0 tx=0).stop 2023-10-16T15:55:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd708bd0640 1 -- 172.21.15.29:0/1689512948 shutdown_connections 2023-10-16T15:55:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1689512948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7040ce390 0x7fd70415e820 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd708bd0640 1 -- 172.21.15.29:0/1689512948 >> 172.21.15.29:0/1689512948 conn(0x7fd7040cb410 msgr2=0x7fd7040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.037+0000 7fd708bd0640 1 -- 172.21.15.29:0/1689512948 shutdown_connections 2023-10-16T15:55:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.038+0000 7fd708bd0640 1 -- 172.21.15.29:0/1689512948 wait complete. 2023-10-16T15:55:16.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.039+0000 7fd708bd0640 1 Processor -- start 2023-10-16T15:55:16.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.039+0000 7fd708bd0640 1 -- start start 2023-10-16T15:55:16.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.040+0000 7fd708bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7040ce390 0x7fd70415dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.040+0000 7fd708bd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 0x7fd7040a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.040+0000 7fd708bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70415ae80 0x7fd70414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.040+0000 7fd708bd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7041f1820 con 0x7fd70415aaa0 2023-10-16T15:55:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.040+0000 7fd708bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70414e460 con 0x7fd70415ae80 2023-10-16T15:55:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.040+0000 7fd708bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7041f0c60 con 0x7fd7040ce390 2023-10-16T15:55:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 0x7fd7040a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 0x7fd7040a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37738/0 (socket says 172.21.15.29:37738) 2023-10-16T15:55:16.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 -- 172.21.15.29:0/1964968533 learned_addr learned my addr 172.21.15.29:0/1964968533 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:16.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd702d9d640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70415ae80 0x7fd70414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd70259c640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7040ce390 0x7fd70415dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 -- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7040ce390 msgr2=0x7fd70415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7040ce390 0x7fd70415dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 -- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70415ae80 msgr2=0x7fd70414df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70415ae80 0x7fd70414df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd701d9b640 1 -- 172.21.15.29:0/1964968533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc009400 con 0x7fd70415aaa0 2023-10-16T15:55:16.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.041+0000 7fd702d9d640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70415ae80 0x7fd70414df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.042+0000 7fd701d9b640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 0x7fd7040a84f0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c3f0 tx=0x7fd6ec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.042+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec0074e0 con 0x7fd70415aaa0 2023-10-16T15:55:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.042+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec007660 con 0x7fd70415aaa0 2023-10-16T15:55:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.042+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec004a20 con 0x7fd70415aaa0 2023-10-16T15:55:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.042+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd704158200 con 0x7fd70415aaa0 2023-10-16T15:55:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.042+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70415e330 con 0x7fd70415aaa0 2023-10-16T15:55:16.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.043+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7041f1960 con 0x7fd70415aaa0 2023-10-16T15:55:16.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.044+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fd6ec004b80 con 0x7fd70415aaa0 2023-10-16T15:55:16.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.045+0000 7fd6faffd640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd6e406b070 0x7fd6e406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.045+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7fd6ec089ce0 con 0x7fd70415aaa0 2023-10-16T15:55:16.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.046+0000 7fd70259c640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd6e406b070 0x7fd6e406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.047+0000 7fd70259c640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd6e406b070 0x7fd6e406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4007fd0 tx=0x7fd6f4009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:16.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.049+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7041f1960 con 0x7fd70415aaa0 2023-10-16T15:55:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.285+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7041f1960 con 0x7fd70415aaa0 2023-10-16T15:55:16.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.286+0000 7fd6faffd640 1 -- 172.21.15.29:0/1964968533 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6155 (secure 0 0 0) 0x7fd7041f1960 con 0x7fd70415aaa0 2023-10-16T15:55:16.287 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:16.287 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 130 2023-10-16T15:55:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.288+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd6e406b070 msgr2=0x7fd6e406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.288+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd6e406b070 0x7fd6e406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4007fd0 tx=0x7fd6f4009040 comp rx=0 tx=0).stop 2023-10-16T15:55:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.288+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 msgr2=0x7fd7040a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.288+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 0x7fd7040a84f0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c3f0 tx=0x7fd6ec00c8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.288+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 shutdown_connections 2023-10-16T15:55:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd6e406b070 0x7fd6e406d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70415aaa0 0x7fd7040a84f0 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd70415ae80 0x7fd70414df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 --2- 172.21.15.29:0/1964968533 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7040ce390 0x7fd70415dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 >> 172.21.15.29:0/1964968533 conn(0x7fd7040cb410 msgr2=0x7fd7040700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 shutdown_connections 2023-10-16T15:55:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.289+0000 7fd708bd0640 1 -- 172.21.15.29:0/1964968533 wait complete. 2023-10-16T15:55:16.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:16.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.531+0000 7f65773d0640 1 Processor -- start 2023-10-16T15:55:16.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.531+0000 7f65773d0640 1 -- start start 2023-10-16T15:55:16.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.533+0000 7f65773d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f657015ae80 0x7f657015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.533+0000 7f65773d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657015ed60 con 0x7f65700ce390 2023-10-16T15:55:16.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.533+0000 7f65773d0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700707e0 con 0x7f657015aaa0 2023-10-16T15:55:16.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.533+0000 7f65773d0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657006b810 con 0x7f657015ae80 2023-10-16T15:55:16.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.533+0000 7f657496b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f657015ae80 0x7f657015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.534+0000 7f657596d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65700ce390 0x7f6570070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37612/0 (socket says 172.21.15.29:37612) 2023-10-16T15:55:16.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.534+0000 7f657596d640 1 -- 172.21.15.29:0/2217871814 learned_addr learned my addr 172.21.15.29:0/2217871814 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:16.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.534+0000 7f657496b640 1 -- 172.21.15.29:0/2217871814 >> v1:172.21.15.170:6789/0 conn(0x7f657015aaa0 legacy=0x7f65700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.534+0000 7f657496b640 1 -- 172.21.15.29:0/2217871814 >> v1:172.21.15.29:6789/0 conn(0x7f65700ce390 legacy=0x7f6570070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.534+0000 7f657496b640 1 -- 172.21.15.29:0/2217871814 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657015ef90 con 0x7f657015ae80 2023-10-16T15:55:16.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.535+0000 7f657496b640 1 --2- 172.21.15.29:0/2217871814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015ae80 0x7f657015e820 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f65600097a0 tx=0x7f656002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dda3a39cce1e9a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:16.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.536+0000 7f6567fff640 1 -- 172.21.15.29:0/2217871814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560030610 con 0x7f657015ae80 2023-10-16T15:55:16.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.536+0000 7f6567fff640 1 -- 172.21.15.29:0/2217871814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6560030770 con 0x7f657015ae80 2023-10-16T15:55:16.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.536+0000 7f6567fff640 1 -- 172.21.15.29:0/2217871814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560030b00 con 0x7f657015ae80 2023-10-16T15:55:16.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.536+0000 7f65773d0640 1 -- 172.21.15.29:0/2217871814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015ae80 msgr2=0x7f657015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.536+0000 7f65773d0640 1 --2- 172.21.15.29:0/2217871814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015ae80 0x7f657015e820 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f65600097a0 tx=0x7f656002b960 comp rx=0 tx=0).stop 2023-10-16T15:55:16.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.537+0000 7f65773d0640 1 -- 172.21.15.29:0/2217871814 shutdown_connections 2023-10-16T15:55:16.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.537+0000 7f65773d0640 1 --2- 172.21.15.29:0/2217871814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015ae80 0x7f657015e820 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.537+0000 7f65773d0640 1 -- 172.21.15.29:0/2217871814 >> 172.21.15.29:0/2217871814 conn(0x7f65700cb410 msgr2=0x7f65700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:16.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.537+0000 7f65773d0640 1 -- 172.21.15.29:0/2217871814 shutdown_connections 2023-10-16T15:55:16.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.537+0000 7f65773d0640 1 -- 172.21.15.29:0/2217871814 wait complete. 2023-10-16T15:55:16.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.538+0000 7f65773d0640 1 Processor -- start 2023-10-16T15:55:16.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.539+0000 7f65773d0640 1 -- start start 2023-10-16T15:55:16.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.539+0000 7f65773d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65700ce390 0x7f657015d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f65773d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 0x7f657014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f65773d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657015ae80 0x7f657015be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f65773d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657014d3c0 con 0x7f657015aaa0 2023-10-16T15:55:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f65773d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657015c760 con 0x7f657015ae80 2023-10-16T15:55:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f65773d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657015ca40 con 0x7f65700ce390 2023-10-16T15:55:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 0x7f657014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 0x7f657014ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37752/0 (socket says 172.21.15.29:37752) 2023-10-16T15:55:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 -- 172.21.15.29:0/2252065209 learned_addr learned my addr 172.21.15.29:0/2252065209 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:16.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657596d640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657015ae80 0x7f657015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657516c640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65700ce390 0x7f657015d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 -- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65700ce390 msgr2=0x7f657015d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65700ce390 0x7f657015d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 -- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657015ae80 msgr2=0x7f657015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657015ae80 0x7f657015be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657496b640 1 -- 172.21.15.29:0/2252065209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6560009400 con 0x7f657015aaa0 2023-10-16T15:55:16.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657516c640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65700ce390 0x7f657015d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:16.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.540+0000 7f657596d640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657015ae80 0x7f657015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:16.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.541+0000 7f657496b640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 0x7f657014ce80 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f656002bf70 tx=0x7f65600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.541+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656003e070 con 0x7f657015aaa0 2023-10-16T15:55:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.541+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6560039a10 con 0x7f657015aaa0 2023-10-16T15:55:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.541+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560039da0 con 0x7f657015aaa0 2023-10-16T15:55:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.541+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657015cd20 con 0x7f657015aaa0 2023-10-16T15:55:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.541+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657015d2e0 con 0x7f657015aaa0 2023-10-16T15:55:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.542+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65700ce200 con 0x7f657015aaa0 2023-10-16T15:55:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.543+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f656004b400 con 0x7f657015aaa0 2023-10-16T15:55:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.544+0000 7f6565ffb640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f655406b070 0x7f655406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.544+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f65600ac5e0 con 0x7f657015aaa0 2023-10-16T15:55:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.545+0000 7f657516c640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f655406b070 0x7f655406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.546+0000 7f657516c640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f655406b070 0x7f655406d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f656c000f60 tx=0x7f656c009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.549+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6560076c90 con 0x7f657015aaa0 2023-10-16T15:55:16.565 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:16.562+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.785+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f657004e160 con 0x7f657015aaa0 2023-10-16T15:55:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.785+0000 7f6565ffb640 1 -- 172.21.15.29:0/2252065209 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6155 (secure 0 0 0) 0x7f6560034090 con 0x7f657015aaa0 2023-10-16T15:55:16.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:16.787 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:38.641105+0000","modified":"2023-10-16T15:54:41.783513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11616},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11616":{"gid":11616,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/3076937560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3076937560},{"type":"v1","addr":"172.21.15.29:6835","nonce":3076937560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 130 2023-10-16T15:55:16.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f655406b070 msgr2=0x7f655406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f655406b070 0x7f655406d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f656c000f60 tx=0x7f656c009040 comp rx=0 tx=0).stop 2023-10-16T15:55:16.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 msgr2=0x7f657014ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:16.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 0x7f657014ce80 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f656002bf70 tx=0x7f65600046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 shutdown_connections 2023-10-16T15:55:16.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f655406b070 0x7f655406d530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f657015aaa0 0x7f657014ce80 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f657015ae80 0x7f657015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 --2- 172.21.15.29:0/2252065209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f65700ce390 0x7f657015d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.788+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 >> 172.21.15.29:0/2252065209 conn(0x7f65700cb410 msgr2=0x7f65700b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:16.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.789+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 shutdown_connections 2023-10-16T15:55:16.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:16.789+0000 7f65773d0640 1 -- 172.21.15.29:0/2252065209 wait complete. 2023-10-16T15:55:16.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:17.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.025+0000 7fe2e20d4640 1 Processor -- start 2023-10-16T15:55:17.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.025+0000 7fe2e20d4640 1 -- start start 2023-10-16T15:55:17.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.026+0000 7fe2e20d4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe2dc15d000 0x7fe2dc06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.026+0000 7fe2e20d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 0x7fe2dc06adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.026+0000 7fe2e20d4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0ae4f0 con 0x7fe2dc15d3e0 2023-10-16T15:55:17.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.026+0000 7fe2e20d4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc06e6f0 con 0x7fe2dc15d000 2023-10-16T15:55:17.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.026+0000 7fe2e20d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc070f80 con 0x7fe2dc071d40 2023-10-16T15:55:17.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.027+0000 7fe2dbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe2dc15d3e0 0x7fe2dc0ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37614/0 (socket says 172.21.15.29:37614) 2023-10-16T15:55:17.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.027+0000 7fe2dbfff640 1 -- 172.21.15.29:0/1379764327 learned_addr learned my addr 172.21.15.29:0/1379764327 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:17.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.027+0000 7fe2db7fe640 1 --2- 172.21.15.29:0/1379764327 >> v2:172.21.15.170:3300/0 conn(0x7fe2dc15d000 0x7fe2dc06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.027+0000 7fe2d3fff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 0x7fe2dc06adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.027+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767553217 0 0) 0x7fe2dc0ae4f0 con 0x7fe2dc15d3e0 2023-10-16T15:55:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.027+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c8003520 con 0x7fe2dc15d3e0 2023-10-16T15:55:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889000316 0 0) 0x7fe2c8003520 con 0x7fe2dc15d3e0 2023-10-16T15:55:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 msgr2=0x7fe2dc06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 0x7fe2dc06adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 >> v2:172.21.15.170:3300/0 conn(0x7fe2dc15d000 msgr2=0x7fe2dc06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 --2- 172.21.15.29:0/1379764327 >> v2:172.21.15.170:3300/0 conn(0x7fe2dc15d000 0x7fe2dc06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc071260 con 0x7fe2dc15d3e0 2023-10-16T15:55:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe2d40019f0 con 0x7fe2dc15d3e0 2023-10-16T15:55:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 >> v1:172.21.15.29:6789/0 conn(0x7fe2dc15d3e0 legacy=0x7fe2dc0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 0x7fe2c8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c8003f40 0x7fe2c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc06e6f0 con 0x7fe2c8004d00 2023-10-16T15:55:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc070f80 con 0x7fe2c8003f40 2023-10-16T15:55:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c800b980 con 0x7fe2dc071d40 2023-10-16T15:55:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.028+0000 7fe2dbfff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2d3fff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c8003f40 0x7fe2c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2dbfff640 1 -- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 msgr2=0x7fe2c8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2dbfff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc071d40 0x7fe2c8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2dbfff640 1 -- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c8003f40 msgr2=0x7fe2c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2dbfff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c8003f40 0x7fe2c80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2dbfff640 1 -- 172.21.15.29:0/1379764327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c8003520 con 0x7fe2c8004d00 2023-10-16T15:55:17.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.029+0000 7fe2d3fff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c8003f40 0x7fe2c80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:17.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.030+0000 7fe2dbfff640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2c8005180 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400a790 tx=0x7fe2c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e770d126a1a6e82a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.030+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4034630 con 0x7fe2c8004d00 2023-10-16T15:55:17.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.030+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d402f070 con 0x7fe2c8004d00 2023-10-16T15:55:17.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.030+0000 7fe2daffd640 1 -- 172.21.15.29:0/1379764327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4034a90 con 0x7fe2c8004d00 2023-10-16T15:55:17.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1379764327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 msgr2=0x7fe2c8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1379764327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2c8005180 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400a790 tx=0x7fe2c80057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2dbfff640 1 -- 172.21.15.29:0/1379764327 reap_dead start 2023-10-16T15:55:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1379764327 shutdown_connections 2023-10-16T15:55:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1379764327 >> 172.21.15.29:0/1379764327 conn(0x7fe2dc0b0c80 msgr2=0x7fe2dc0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1379764327 shutdown_connections 2023-10-16T15:55:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.031+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1379764327 wait complete. 2023-10-16T15:55:17.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.033+0000 7fe2e20d4640 1 Processor -- start 2023-10-16T15:55:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.033+0000 7fe2e20d4640 1 -- start start 2023-10-16T15:55:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2e20d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2dc0c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2e20d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc069dd0 0x7fe2dc06a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2e20d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2dc0b3890 0x7fe2dc0b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2e20d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc06a790 con 0x7fe2c8004d00 2023-10-16T15:55:17.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2e20d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0b4b40 con 0x7fe2dc0b3890 2023-10-16T15:55:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2e20d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0c2cb0 con 0x7fe2dc069dd0 2023-10-16T15:55:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2dc0c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2dc0c5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37778/0 (socket says 172.21.15.29:37778) 2023-10-16T15:55:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.034+0000 7fe2db7fe640 1 -- 172.21.15.29:0/1120918503 learned_addr learned my addr 172.21.15.29:0/1120918503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:17.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2d3fff640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc069dd0 0x7fe2dc06a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2db7fe640 1 -- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc069dd0 msgr2=0x7fe2dc06a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2db7fe640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc069dd0 0x7fe2dc06a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2db7fe640 1 -- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2dc0b3890 msgr2=0x7fe2dc0b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2db7fe640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2dc0b3890 0x7fe2dc0b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2db7fe640 1 -- 172.21.15.29:0/1120918503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0b4580 con 0x7fe2c8004d00 2023-10-16T15:55:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2db7fe640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2dc0c5780 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00e7c0 tx=0x7fe2cc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc00eed0 con 0x7fe2c8004d00 2023-10-16T15:55:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.035+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc00c8c0 con 0x7fe2c8004d00 2023-10-16T15:55:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.036+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc00cc70 con 0x7fe2c8004d00 2023-10-16T15:55:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.036+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc0b4860 con 0x7fe2c8004d00 2023-10-16T15:55:17.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.036+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc0b4280 con 0x7fe2c8004d00 2023-10-16T15:55:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.037+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc04e160 con 0x7fe2c8004d00 2023-10-16T15:55:17.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.038+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fe2cc013470 con 0x7fe2c8004d00 2023-10-16T15:55:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.039+0000 7fe2d8ff9640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2bc06b050 0x7fe2bc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.039+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7fe2cc089fa0 con 0x7fe2c8004d00 2023-10-16T15:55:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.040+0000 7fe2d3fff640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2bc06b050 0x7fe2bc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.041+0000 7fe2d3fff640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2bc06b050 0x7fe2bc06d510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0cb1b0 tx=0x7fe2c4008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.043+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2cc0bcca0 con 0x7fe2c8004d00 2023-10-16T15:55:17.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.245+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2dc0ae470 con 0x7fe2c8004d00 2023-10-16T15:55:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.249+0000 7fe2d8ff9640 1 -- 172.21.15.29:0/1120918503 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+21497 (secure 0 0 0) 0x7fe2cc017090 con 0x7fe2c8004d00 2023-10-16T15:55:17.251 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:17.251 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":252,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:11.776929+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:54:36.572242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T15:54:37.685848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2bc06b050 msgr2=0x7fe2bc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2bc06b050 0x7fe2bc06d510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0cb1b0 tx=0x7fe2c4008040 comp rx=0 tx=0).stop 2023-10-16T15:55:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 msgr2=0x7fe2dc0c5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2dc0c5780 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00e7c0 tx=0x7fe2cc00ec80 comp rx=0 tx=0).stop 2023-10-16T15:55:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 shutdown_connections 2023-10-16T15:55:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2bc06b050 0x7fe2bc06d510 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2dc0b3890 0x7fe2dc0b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2dc069dd0 0x7fe2dc06a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 --2- 172.21.15.29:0/1120918503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c8004d00 0x7fe2dc0c5780 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 >> 172.21.15.29:0/1120918503 conn(0x7fe2dc0b0c80 msgr2=0x7fe2dc15c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 shutdown_connections 2023-10-16T15:55:17.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.252+0000 7fe2e20d4640 1 -- 172.21.15.29:0/1120918503 wait complete. 2023-10-16T15:55:17.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:55:17.474 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:17.471+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:17.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.495+0000 7f6522ad7640 1 Processor -- start 2023-10-16T15:55:17.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.496+0000 7f6522ad7640 1 -- start start 2023-10-16T15:55:17.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.497+0000 7f6522ad7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f651c159ab0 0x7f651c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.497+0000 7f6522ad7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159e90 0x7f651c06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.497+0000 7f6522ad7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f651c06c140 0x7f651c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.497+0000 7f6522ad7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651c06cb10 con 0x7f651c159ab0 2023-10-16T15:55:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.497+0000 7f6522ad7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651c15d890 con 0x7f651c06c140 2023-10-16T15:55:17.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.497+0000 7f6522ad7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0cd1f0 con 0x7f651c159e90 2023-10-16T15:55:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f651c159ab0 0x7f651c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f651c159ab0 0x7f651c0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37792/0 (socket says 172.21.15.29:37792) 2023-10-16T15:55:17.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 -- 172.21.15.29:0/3227374165 learned_addr learned my addr 172.21.15.29:0/3227374165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6521074640 1 --2- 172.21.15.29:0/3227374165 >> v2:172.21.15.170:3300/0 conn(0x7f651c06c140 0x7f651c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 -- 172.21.15.29:0/3227374165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159e90 msgr2=0x7f651c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 --2- 172.21.15.29:0/3227374165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159e90 0x7f651c06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 -- 172.21.15.29:0/3227374165 >> v2:172.21.15.170:3300/0 conn(0x7f651c06c140 msgr2=0x7f651c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 --2- 172.21.15.29:0/3227374165 >> v2:172.21.15.170:3300/0 conn(0x7f651c06c140 0x7f651c06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.498+0000 7f6520873640 1 -- 172.21.15.29:0/3227374165 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c0cd4d0 con 0x7f651c159ab0 2023-10-16T15:55:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.499+0000 7f6520873640 1 --2- 172.21.15.29:0/3227374165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159ab0 0x7f651c0ce370 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f650c009e50 tx=0x7f650c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1602479107483b24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.500+0000 7f651b7fe640 1 -- 172.21.15.29:0/3227374165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c02c960 con 0x7f651c159ab0 2023-10-16T15:55:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.500+0000 7f651b7fe640 1 -- 172.21.15.29:0/3227374165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c02cac0 con 0x7f651c159ab0 2023-10-16T15:55:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.500+0000 7f651b7fe640 1 -- 172.21.15.29:0/3227374165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c02ce50 con 0x7f651c159ab0 2023-10-16T15:55:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.500+0000 7f6522ad7640 1 -- 172.21.15.29:0/3227374165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159ab0 msgr2=0x7f651c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.500+0000 7f6522ad7640 1 --2- 172.21.15.29:0/3227374165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159ab0 0x7f651c0ce370 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f650c009e50 tx=0x7f650c02bdc0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 -- 172.21.15.29:0/3227374165 shutdown_connections 2023-10-16T15:55:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 --2- 172.21.15.29:0/3227374165 >> v2:172.21.15.170:3300/0 conn(0x7f651c06c140 0x7f651c06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 --2- 172.21.15.29:0/3227374165 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159e90 0x7f651c06fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 --2- 172.21.15.29:0/3227374165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159ab0 0x7f651c0ce370 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 -- 172.21.15.29:0/3227374165 >> 172.21.15.29:0/3227374165 conn(0x7f651c0ce9b0 msgr2=0x7f651c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:17.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 -- 172.21.15.29:0/3227374165 shutdown_connections 2023-10-16T15:55:17.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.501+0000 7f6522ad7640 1 -- 172.21.15.29:0/3227374165 wait complete. 2023-10-16T15:55:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.502+0000 7f6522ad7640 1 Processor -- start 2023-10-16T15:55:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.503+0000 7f6522ad7640 1 -- start start 2023-10-16T15:55:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6522ad7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f651c06c140 0x7f651c0c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6522ad7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159ab0 0x7f651c0b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6522ad7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 0x7f651c0af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6522ad7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0b2eb0 con 0x7f651c159e90 2023-10-16T15:55:17.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6522ad7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0b22a0 con 0x7f651c06c140 2023-10-16T15:55:17.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6522ad7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0b2580 con 0x7f651c159ab0 2023-10-16T15:55:17.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 0x7f651c0af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 0x7f651c0af5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37802/0 (socket says 172.21.15.29:37802) 2023-10-16T15:55:17.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 -- 172.21.15.29:0/824965922 learned_addr learned my addr 172.21.15.29:0/824965922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:17.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6520873640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f651c06c140 0x7f651c0c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f651bfff640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159ab0 0x7f651c0b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 -- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159ab0 msgr2=0x7f651c0b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159ab0 0x7f651c0b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 -- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f651c06c140 msgr2=0x7f651c0c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f651c06c140 0x7f651c0c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f6521074640 1 -- 172.21.15.29:0/824965922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c009b30 con 0x7f651c159e90 2023-10-16T15:55:17.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.504+0000 7f651bfff640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159ab0 0x7f651c0b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:17.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.505+0000 7f6521074640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 0x7f651c0af5b0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f651400c550 tx=0x7f651400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.505+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514004220 con 0x7f651c159e90 2023-10-16T15:55:17.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.505+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6514004380 con 0x7f651c159e90 2023-10-16T15:55:17.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.505+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c0b2860 con 0x7f651c159e90 2023-10-16T15:55:17.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.505+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514003830 con 0x7f651c159e90 2023-10-16T15:55:17.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.505+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c0b3340 con 0x7f651c159e90 2023-10-16T15:55:17.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.507+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c0b3060 con 0x7f651c159e90 2023-10-16T15:55:17.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.508+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f6514003990 con 0x7f651c159e90 2023-10-16T15:55:17.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.509+0000 7f65197fa640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f650806b050 0x7f650806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.509+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7461+0+0 (secure 0 0 0) 0x7f6514089e40 con 0x7f651c159e90 2023-10-16T15:55:17.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.510+0000 7f6520873640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f650806b050 0x7f650806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.511+0000 7f6520873640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f650806b050 0x7f650806d510 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f650c035000 tx=0x7f650c030000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.514+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65140543e0 con 0x7f651c159e90 2023-10-16T15:55:17.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.747+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f651c0cd780 con 0x7f651c159e90 2023-10-16T15:55:17.749 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:55:17.748+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:55:17.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.752+0000 7f65197fa640 1 -- 172.21.15.29:0/824965922 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v131) v1 ==== 161+0+0 (secure 0 0 0) 0x7f651401e090 con 0x7f651c159e90 2023-10-16T15:55:17.753 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:55:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.755+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f650806b050 msgr2=0x7f650806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.755+0000 7f6522ad7640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f650806b050 0x7f650806d510 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f650c035000 tx=0x7f650c030000 comp rx=0 tx=0).stop 2023-10-16T15:55:17.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.755+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 msgr2=0x7f651c0af5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.755+0000 7f6522ad7640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 0x7f651c0af5b0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f651400c550 tx=0x7f651400ca10 comp rx=0 tx=0).stop 2023-10-16T15:55:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 shutdown_connections 2023-10-16T15:55:17.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f650806b050 0x7f650806d510 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f651c159e90 0x7f651c0af5b0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f651c159ab0 0x7f651c0b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 --2- 172.21.15.29:0/824965922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f651c06c140 0x7f651c0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 >> 172.21.15.29:0/824965922 conn(0x7f651c0ce9b0 msgr2=0x7f651c06fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 shutdown_connections 2023-10-16T15:55:17.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.756+0000 7f6522ad7640 1 -- 172.21.15.29:0/824965922 wait complete. 2023-10-16T15:55:17.767 INFO:tasks.ceph.mds.a.smithi029.stderr: -4311> 2023-10-16T15:54:43.269+0000 7fd6bf6f5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:55:17.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:55:17.838 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:55:17.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.977+0000 7fd2b1207640 1 Processor -- start 2023-10-16T15:55:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.977+0000 7fd2b1207640 1 -- start start 2023-10-16T15:55:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.979+0000 7fd2b1207640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2ac159e90 0x7fd2ac0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.979+0000 7fd2b1207640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd2ac06c140 0x7fd2ac06c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.979+0000 7fd2b1207640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac06cad0 con 0x7fd2ac159ab0 2023-10-16T15:55:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.979+0000 7fd2b1207640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac15d890 con 0x7fd2ac159e90 2023-10-16T15:55:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.979+0000 7fd2b1207640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac070360 con 0x7fd2ac06c140 2023-10-16T15:55:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.979+0000 7fd2aa59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2ac159e90 0x7fd2ac0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2aa59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2ac159e90 0x7fd2ac0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37808/0 (socket says 172.21.15.29:37808) 2023-10-16T15:55:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2aa59c640 1 -- 172.21.15.29:0/3194293739 learned_addr learned my addr 172.21.15.29:0/3194293739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2ab59e640 1 --2- 172.21.15.29:0/3194293739 >> v2:172.21.15.170:3300/0 conn(0x7fd2ac06c140 0x7fd2ac06c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2aa59c640 1 -- 172.21.15.29:0/3194293739 >> v2:172.21.15.170:3300/0 conn(0x7fd2ac06c140 msgr2=0x7fd2ac06c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2aa59c640 1 --2- 172.21.15.29:0/3194293739 >> v2:172.21.15.170:3300/0 conn(0x7fd2ac06c140 0x7fd2ac06c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2aa59c640 1 -- 172.21.15.29:0/3194293739 >> v1:172.21.15.170:6789/0 conn(0x7fd2ac159ab0 legacy=0x7fd2ac06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.980+0000 7fd2aa59c640 1 -- 172.21.15.29:0/3194293739 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac0cd1f0 con 0x7fd2ac159e90 2023-10-16T15:55:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.981+0000 7fd2aa59c640 1 --2- 172.21.15.29:0/3194293739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac159e90 0x7fd2ac0ce370 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0adcd0 tx=0x7fd2a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcef165635c1b70 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.982+0000 7fd2a9d9b640 1 -- 172.21.15.29:0/3194293739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00317e0 con 0x7fd2ac159e90 2023-10-16T15:55:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.982+0000 7fd2a9d9b640 1 -- 172.21.15.29:0/3194293739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a0031940 con 0x7fd2ac159e90 2023-10-16T15:55:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.982+0000 7fd2a9d9b640 1 -- 172.21.15.29:0/3194293739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0031cd0 con 0x7fd2ac159e90 2023-10-16T15:55:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.982+0000 7fd2b1207640 1 -- 172.21.15.29:0/3194293739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac159e90 msgr2=0x7fd2ac0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.982+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3194293739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac159e90 0x7fd2ac0ce370 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0adcd0 tx=0x7fd2a002b880 comp rx=0 tx=0).stop 2023-10-16T15:55:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.983+0000 7fd2b1207640 1 -- 172.21.15.29:0/3194293739 shutdown_connections 2023-10-16T15:55:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.983+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3194293739 >> v2:172.21.15.170:3300/0 conn(0x7fd2ac06c140 0x7fd2ac06c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.983+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3194293739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac159e90 0x7fd2ac0ce370 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.983+0000 7fd2b1207640 1 -- 172.21.15.29:0/3194293739 >> 172.21.15.29:0/3194293739 conn(0x7fd2ac0ce9b0 msgr2=0x7fd2ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.983+0000 7fd2b1207640 1 -- 172.21.15.29:0/3194293739 shutdown_connections 2023-10-16T15:55:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.983+0000 7fd2b1207640 1 -- 172.21.15.29:0/3194293739 wait complete. 2023-10-16T15:55:17.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.985+0000 7fd2b1207640 1 Processor -- start 2023-10-16T15:55:17.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.985+0000 7fd2b1207640 1 -- start start 2023-10-16T15:55:17.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2b1207640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 0x7fd2ac150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2b1207640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2ac159ab0 0x7fd2ac1f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2b1207640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2ac070c50 0x7fd2ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2b1207640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac1f7260 con 0x7fd2ac06c140 2023-10-16T15:55:17.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2b1207640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0c24e0 con 0x7fd2ac070c50 2023-10-16T15:55:17.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2b1207640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0c2780 con 0x7fd2ac159ab0 2023-10-16T15:55:17.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 0x7fd2ac150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 0x7fd2ac150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37814/0 (socket says 172.21.15.29:37814) 2023-10-16T15:55:17.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 -- 172.21.15.29:0/3908959452 learned_addr learned my addr 172.21.15.29:0/3908959452 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:17.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 -- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2ac159ab0 msgr2=0x7fd2ac1f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2ac159ab0 0x7fd2ac1f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 -- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2ac070c50 msgr2=0x7fd2ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:17.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.986+0000 7fd2aad9d640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2ac070c50 0x7fd2ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:17.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2aad9d640 1 -- 172.21.15.29:0/3908959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00094d0 con 0x7fd2ac06c140 2023-10-16T15:55:17.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2aad9d640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 0x7fd2ac150080 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c450 tx=0x7fd29c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0052d0 con 0x7fd2ac06c140 2023-10-16T15:55:17.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c005430 con 0x7fd2ac06c140 2023-10-16T15:55:17.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c012870 con 0x7fd2ac06c140 2023-10-16T15:55:17.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac0c2a60 con 0x7fd2ac06c140 2023-10-16T15:55:17.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.987+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac0c3020 con 0x7fd2ac06c140 2023-10-16T15:55:17.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.989+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac0b0250 con 0x7fd2ac06c140 2023-10-16T15:55:17.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.990+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fd29c0129d0 con 0x7fd2ac06c140 2023-10-16T15:55:17.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.990+0000 7fd2937fe640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd29806b050 0x7fd29806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:17.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.991+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7547+0+0 (secure 0 0 0) 0x7fd29c089930 con 0x7fd2ac06c140 2023-10-16T15:55:17.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.991+0000 7fd2aa59c640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd29806b050 0x7fd29806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:17.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.992+0000 7fd2aa59c640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd29806b050 0x7fd29806d510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0036000 tx=0x7fd2a0030000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:17.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:17.995+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd29c0540c0 con 0x7fd2ac06c140 2023-10-16T15:55:18.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.234+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd2ac06cad0 con 0x7fd2ac06c140 2023-10-16T15:55:18.420 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:18.417+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:18.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:55:18.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:55:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.828+0000 7fd2937fe640 1 -- 172.21.15.29:0/3908959452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v132) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd2ac06cad0 con 0x7fd2ac06c140 2023-10-16T15:55:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd29806b050 msgr2=0x7fd29806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd29806b050 0x7fd29806d510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0036000 tx=0x7fd2a0030000 comp rx=0 tx=0).stop 2023-10-16T15:55:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 msgr2=0x7fd2ac150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 0x7fd2ac150080 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c450 tx=0x7fd29c00c910 comp rx=0 tx=0).stop 2023-10-16T15:55:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 shutdown_connections 2023-10-16T15:55:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd29806b050 0x7fd29806d510 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd2ac070c50 0x7fd2ac0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd2ac159ab0 0x7fd2ac1f6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 --2- 172.21.15.29:0/3908959452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ac06c140 0x7fd2ac150080 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 >> 172.21.15.29:0/3908959452 conn(0x7fd2ac0ce9b0 msgr2=0x7fd2ac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 shutdown_connections 2023-10-16T15:55:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:18.832+0000 7fd2b1207640 1 -- 172.21.15.29:0/3908959452 wait complete. 2023-10-16T15:55:18.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:55:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.086+0000 7f1697fff640 1 Processor -- start 2023-10-16T15:55:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.086+0000 7f1697fff640 1 -- start start 2023-10-16T15:55:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1697fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1697fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f169806c9e0 0x7f16980ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1697fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980ca560 con 0x7f16980c9060 2023-10-16T15:55:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1697fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16980c9950 con 0x7f169806c9e0 2023-10-16T15:55:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1697fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c9c30 con 0x7f169806fb50 2023-10-16T15:55:19.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f16977fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f169806c9e0 0x7f16980ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:19.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1696ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16980c9060 0x7f169806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37622/0 (socket says 172.21.15.29:37622) 2023-10-16T15:55:19.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f1696ffd640 1 -- 172.21.15.29:0/2477837349 learned_addr learned my addr 172.21.15.29:0/2477837349 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:19.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f16967fc640 1 -- 172.21.15.29:0/2477837349 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341257855 0 0) 0x7f16980ca560 con 0x7f16980c9060 2023-10-16T15:55:19.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f16967fc640 1 -- 172.21.15.29:0/2477837349 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1684003520 con 0x7f16980c9060 2023-10-16T15:55:19.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.088+0000 7f168e7fc640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:19.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.089+0000 7f16977fe640 1 -- 172.21.15.29:0/2477837349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 msgr2=0x7f16980cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:19.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.089+0000 7f16977fe640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.089+0000 7f16977fe640 1 -- 172.21.15.29:0/2477837349 >> v1:172.21.15.29:6789/0 conn(0x7f16980c9060 legacy=0x7f169806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:19.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.089+0000 7f16977fe640 1 -- 172.21.15.29:0/2477837349 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980c9f10 con 0x7f169806c9e0 2023-10-16T15:55:19.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.089+0000 7f168e7fc640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:19.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.090+0000 7f16977fe640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169806c9e0 0x7f16980ccde0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f16900097a0 tx=0x7f169002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b054c1c8593916 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.090+0000 7f16967fc640 1 -- 172.21.15.29:0/2477837349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169002d040 con 0x7f169806c9e0 2023-10-16T15:55:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.090+0000 7f16967fc640 1 -- 172.21.15.29:0/2477837349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690031810 con 0x7f169806c9e0 2023-10-16T15:55:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.090+0000 7f16967fc640 1 -- 172.21.15.29:0/2477837349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690031ba0 con 0x7f169806c9e0 2023-10-16T15:55:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 -- 172.21.15.29:0/2477837349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169806c9e0 msgr2=0x7f16980ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:19.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169806c9e0 0x7f16980ccde0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f16900097a0 tx=0x7f169002b8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 -- 172.21.15.29:0/2477837349 shutdown_connections 2023-10-16T15:55:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f169806c9e0 0x7f16980ccde0 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 --2- 172.21.15.29:0/2477837349 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 -- 172.21.15.29:0/2477837349 >> 172.21.15.29:0/2477837349 conn(0x7f16980ce6a0 msgr2=0x7f16980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:19.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 -- 172.21.15.29:0/2477837349 shutdown_connections 2023-10-16T15:55:19.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.091+0000 7f1697fff640 1 -- 172.21.15.29:0/2477837349 wait complete. 2023-10-16T15:55:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.093+0000 7f1697fff640 1 Processor -- start 2023-10-16T15:55:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.093+0000 7f1697fff640 1 -- start start 2023-10-16T15:55:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f1697fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f169806c9e0 0x7f16981f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f1697fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f1697fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 0x7f169806b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:19.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f1697fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c6be0 con 0x7f16980c9060 2023-10-16T15:55:19.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f1697fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c6e30 con 0x7f169806c9e0 2023-10-16T15:55:19.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f1697fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c7110 con 0x7f169806fb50 2023-10-16T15:55:19.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f16977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 0x7f169806b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:19.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f16977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 0x7f169806b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37836/0 (socket says 172.21.15.29:37836) 2023-10-16T15:55:19.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.094+0000 7f16977fe640 1 -- 172.21.15.29:0/693601555 learned_addr learned my addr 172.21.15.29:0/693601555 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f168e7fc640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f1696ffd640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f169806c9e0 0x7f16981f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f16977fe640 1 -- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 msgr2=0x7f16980c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f16977fe640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f16977fe640 1 -- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f169806c9e0 msgr2=0x7f16981f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:19.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f16977fe640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f169806c9e0 0x7f16981f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f16977fe640 1 -- 172.21.15.29:0/693601555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1690009400 con 0x7f16980c9060 2023-10-16T15:55:19.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f1696ffd640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f169806c9e0 0x7f16981f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:19.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f16977fe640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 0x7f169806b330 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f169002b8f0 tx=0x7f16900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:19.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690006e10 con 0x7f16980c9060 2023-10-16T15:55:19.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169003ace0 con 0x7f16980c9060 2023-10-16T15:55:19.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690044610 con 0x7f16980c9060 2023-10-16T15:55:19.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.095+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16980c73f0 con 0x7f16980c9060 2023-10-16T15:55:19.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.096+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16980c79b0 con 0x7f16980c9060 2023-10-16T15:55:19.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.097+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16980c7ba0 con 0x7f16980c9060 2023-10-16T15:55:19.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.098+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f1690044770 con 0x7f16980c9060 2023-10-16T15:55:19.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.099+0000 7f168ffff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:19.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.099+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7547+0+0 (secure 0 0 0) 0x7f16900ac3e0 con 0x7f16980c9060 2023-10-16T15:55:19.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.100+0000 7f1696ffd640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:19.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.101+0000 7f1696ffd640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f167806b020 0x7f167806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16880011e0 tx=0x7f1688009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:19.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.103+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f16900769d0 con 0x7f16980c9060 2023-10-16T15:55:19.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.311+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f169804e9b0 con 0x7f16980c9060 2023-10-16T15:55:19.352 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:19.350+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.854+0000 7f168ffff640 1 -- 172.21.15.29:0/693601555 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v254) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1690079ff0 con 0x7f16980c9060 2023-10-16T15:55:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:55:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f167806b020 msgr2=0x7f167806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f167806b020 0x7f167806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16880011e0 tx=0x7f1688009040 comp rx=0 tx=0).stop 2023-10-16T15:55:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 msgr2=0x7f169806b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 0x7f169806b330 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f169002b8f0 tx=0x7f16900046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 shutdown_connections 2023-10-16T15:55:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16980c9060 0x7f169806b330 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f169806fb50 0x7f16980c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 --2- 172.21.15.29:0/693601555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f169806c9e0 0x7f16981f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.857+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 >> 172.21.15.29:0/693601555 conn(0x7f16980ce6a0 msgr2=0x7f1698069fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.858+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 shutdown_connections 2023-10-16T15:55:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:19.858+0000 7f1697fff640 1 -- 172.21.15.29:0/693601555 wait complete. 2023-10-16T15:55:19.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:55:20.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.100+0000 7f70aa4fc640 1 Processor -- start 2023-10-16T15:55:20.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.100+0000 7f70aa4fc640 1 -- start start 2023-10-16T15:55:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.101+0000 7f70aa4fc640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f70a406c040 0x7f70a406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:20.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.101+0000 7f70aa4fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:20.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.101+0000 7f70aa4fc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a406c960 con 0x7f70a40ad970 2023-10-16T15:55:20.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.101+0000 7f70aa4fc640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a415b9d0 con 0x7f70a406c040 2023-10-16T15:55:20.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.101+0000 7f70aa4fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a415bcb0 con 0x7f70a40cda80 2023-10-16T15:55:20.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.102+0000 7f70a8cf9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70a40ad970 0x7f70a415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37624/0 (socket says 172.21.15.29:37624) 2023-10-16T15:55:20.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.102+0000 7f70a8cf9640 1 -- 172.21.15.29:0/2700725172 learned_addr learned my addr 172.21.15.29:0/2700725172 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:20.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.102+0000 7f70a9cfb640 1 --2- 172.21.15.29:0/2700725172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:20.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.102+0000 7f70a94fa640 1 --2- 172.21.15.29:0/2700725172 >> v2:172.21.15.170:3300/0 conn(0x7f70a406c040 0x7f70a406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:20.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f709bfff640 1 -- 172.21.15.29:0/2700725172 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203796436 0 0) 0x7f70a406c960 con 0x7f70a40ad970 2023-10-16T15:55:20.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f709bfff640 1 -- 172.21.15.29:0/2700725172 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7090003520 con 0x7f70a40ad970 2023-10-16T15:55:20.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f70a9cfb640 1 -- 172.21.15.29:0/2700725172 >> v2:172.21.15.170:3300/0 conn(0x7f70a406c040 msgr2=0x7f70a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:20.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f70a9cfb640 1 --2- 172.21.15.29:0/2700725172 >> v2:172.21.15.170:3300/0 conn(0x7f70a406c040 0x7f70a406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f70a9cfb640 1 -- 172.21.15.29:0/2700725172 >> v1:172.21.15.29:6789/0 conn(0x7f70a40ad970 legacy=0x7f70a415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:20.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f70a9cfb640 1 -- 172.21.15.29:0/2700725172 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a415bf90 con 0x7f70a40cda80 2023-10-16T15:55:20.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.103+0000 7f70a94fa640 1 --2- 172.21.15.29:0/2700725172 >> v2:172.21.15.170:3300/0 conn(0x7f70a406c040 0x7f70a406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:55:20.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.104+0000 7f70a9cfb640 1 --2- 172.21.15.29:0/2700725172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a415ed90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f709c009d90 tx=0x7f709c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0923fc9c0ed3ea4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:20.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.105+0000 7f709bfff640 1 -- 172.21.15.29:0/2700725172 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c02cc50 con 0x7f70a40cda80 2023-10-16T15:55:20.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.105+0000 7f709bfff640 1 -- 172.21.15.29:0/2700725172 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c02cdb0 con 0x7f70a40cda80 2023-10-16T15:55:20.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.105+0000 7f709bfff640 1 -- 172.21.15.29:0/2700725172 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c034640 con 0x7f70a40cda80 2023-10-16T15:55:20.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 -- 172.21.15.29:0/2700725172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 msgr2=0x7f70a415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:20.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/2700725172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a415ed90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f709c009d90 tx=0x7f709c02bea0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 -- 172.21.15.29:0/2700725172 shutdown_connections 2023-10-16T15:55:20.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/2700725172 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a415ed90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/2700725172 >> v2:172.21.15.170:3300/0 conn(0x7f70a406c040 0x7f70a406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 -- 172.21.15.29:0/2700725172 >> 172.21.15.29:0/2700725172 conn(0x7f70a40ce6a0 msgr2=0x7f70a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:20.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 -- 172.21.15.29:0/2700725172 shutdown_connections 2023-10-16T15:55:20.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.106+0000 7f70aa4fc640 1 -- 172.21.15.29:0/2700725172 wait complete. 2023-10-16T15:55:20.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.108+0000 7f70aa4fc640 1 Processor -- start 2023-10-16T15:55:20.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.109+0000 7f70aa4fc640 1 -- start start 2023-10-16T15:55:20.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70aa4fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 0x7f70a40af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:20.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70aa4fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70a40ad970 0x7f70a40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:20.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70aa4fc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:20.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70aa4fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41f4810 con 0x7f70a406c040 2023-10-16T15:55:20.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70aa4fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41f01e0 con 0x7f70a40ad970 2023-10-16T15:55:20.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70aa4fc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41f4530 con 0x7f70a40cda80 2023-10-16T15:55:20.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70a94fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 0x7f70a40af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:20.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70a94fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 0x7f70a40af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37838/0 (socket says 172.21.15.29:37838) 2023-10-16T15:55:20.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70a94fa640 1 -- 172.21.15.29:0/3039154725 learned_addr learned my addr 172.21.15.29:0/3039154725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:20.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.110+0000 7f70a8cf9640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70a40ad970 0x7f70a40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:20.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a9cfb640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:20.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a94fa640 1 -- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 msgr2=0x7f70a40b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:20.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a94fa640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a40b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a94fa640 1 -- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70a40ad970 msgr2=0x7f70a40b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a94fa640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70a40ad970 0x7f70a40b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a94fa640 1 -- 172.21.15.29:0/3039154725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a0009400 con 0x7f70a406c040 2023-10-16T15:55:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f70a94fa640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 0x7f70a40af960 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f70a000e960 tx=0x7f70a000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.111+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a000ca00 con 0x7f70a406c040 2023-10-16T15:55:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.112+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a000cb60 con 0x7f70a406c040 2023-10-16T15:55:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.112+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709c009a70 con 0x7f70a406c040 2023-10-16T15:55:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.112+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a0013470 con 0x7f70a406c040 2023-10-16T15:55:20.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.112+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a41f0720 con 0x7f70a406c040 2023-10-16T15:55:20.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.113+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a406b330 con 0x7f70a406c040 2023-10-16T15:55:20.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.114+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f70a00135d0 con 0x7f70a406c040 2023-10-16T15:55:20.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.115+0000 7f7099ffb640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f708806b020 0x7f708806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:20.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.115+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7109+0+0 (secure 0 0 0) 0x7f70a008a1f0 con 0x7f70a406c040 2023-10-16T15:55:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.116+0000 7f70a8cf9640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f708806b020 0x7f708806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.117+0000 7f70a8cf9640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f708806b020 0x7f708806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f70a40b01c0 tx=0x7f7094009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.119+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f70a0054a00 con 0x7f70a406c040 2023-10-16T15:55:20.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.330+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f70a404e9b0 con 0x7f70a406c040 2023-10-16T15:55:20.332 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:20.329+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.857+0000 7f7099ffb640 1 -- 172.21.15.29:0/3039154725 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v255) v1 ==== 169+0+0 (secure 0 0 0) 0x7f70a0058020 con 0x7f70a406c040 2023-10-16T15:55:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:55:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.861+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f708806b020 msgr2=0x7f708806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.861+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f708806b020 0x7f708806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f70a40b01c0 tx=0x7f7094009040 comp rx=0 tx=0).stop 2023-10-16T15:55:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.861+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 msgr2=0x7f70a40af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:20.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.861+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 0x7f70a40af960 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f70a000e960 tx=0x7f70a000ee20 comp rx=0 tx=0).stop 2023-10-16T15:55:20.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.862+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 shutdown_connections 2023-10-16T15:55:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.862+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f708806b020 0x7f708806d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.862+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70a40cda80 0x7f70a40b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.862+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70a40ad970 0x7f70a40b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.862+0000 7f70aa4fc640 1 --2- 172.21.15.29:0/3039154725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a406c040 0x7f70a40af960 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.862+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 >> 172.21.15.29:0/3039154725 conn(0x7f70a40ce6a0 msgr2=0x7f70a415dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.863+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 shutdown_connections 2023-10-16T15:55:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:20.864+0000 7f70aa4fc640 1 -- 172.21.15.29:0/3039154725 wait complete. 2023-10-16T15:55:20.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:21.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.102+0000 7feffb59e640 1 Processor -- start 2023-10-16T15:55:21.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.102+0000 7feffb59e640 1 -- start start 2023-10-16T15:55:21.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feffc159290 0x7feffc0cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffb59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7feffc0c9cf0 0x7feffc0ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0ca580 con 0x7feffc06fa20 2023-10-16T15:55:21.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc0ad5e0 con 0x7feffc159290 2023-10-16T15:55:21.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffb59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc0ca2a0 con 0x7feffc0c9cf0 2023-10-16T15:55:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feffc159290 0x7feffc0cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feffc159290 0x7feffc0cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37852/0 (socket says 172.21.15.29:37852) 2023-10-16T15:55:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 -- 172.21.15.29:0/1852822120 learned_addr learned my addr 172.21.15.29:0/1852822120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:21.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feff9d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feffc06fa20 0x7feffc0c95e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37634/0 (socket says 172.21.15.29:37634) 2023-10-16T15:55:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 -- 172.21.15.29:0/1852822120 >> v2:172.21.15.170:3300/0 conn(0x7feffc0c9cf0 msgr2=0x7feffc0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 --2- 172.21.15.29:0/1852822120 >> v2:172.21.15.170:3300/0 conn(0x7feffc0c9cf0 0x7feffc0ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 -- 172.21.15.29:0/1852822120 >> v1:172.21.15.29:6789/0 conn(0x7feffc06fa20 legacy=0x7feffc0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.104+0000 7feffa59c640 1 -- 172.21.15.29:0/1852822120 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc0acf20 con 0x7feffc159290 2023-10-16T15:55:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.106+0000 7feffa59c640 1 --2- 172.21.15.29:0/1852822120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc159290 0x7feffc0cdb70 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fefec00af60 tx=0x7fefec02d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8f1fa6de92f067 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.106+0000 7feff959a640 1 -- 172.21.15.29:0/1852822120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec02da10 con 0x7feffc159290 2023-10-16T15:55:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.106+0000 7feff959a640 1 -- 172.21.15.29:0/1852822120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec02db70 con 0x7feffc159290 2023-10-16T15:55:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.106+0000 7feff959a640 1 -- 172.21.15.29:0/1852822120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec008500 con 0x7feffc159290 2023-10-16T15:55:21.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 -- 172.21.15.29:0/1852822120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc159290 msgr2=0x7feffc0cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 --2- 172.21.15.29:0/1852822120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc159290 0x7feffc0cdb70 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fefec00af60 tx=0x7fefec02d070 comp rx=0 tx=0).stop 2023-10-16T15:55:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 -- 172.21.15.29:0/1852822120 shutdown_connections 2023-10-16T15:55:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 --2- 172.21.15.29:0/1852822120 >> v2:172.21.15.170:3300/0 conn(0x7feffc0c9cf0 0x7feffc0ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 --2- 172.21.15.29:0/1852822120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc159290 0x7feffc0cdb70 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 -- 172.21.15.29:0/1852822120 >> 172.21.15.29:0/1852822120 conn(0x7feffc0b0a80 msgr2=0x7feffc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 -- 172.21.15.29:0/1852822120 shutdown_connections 2023-10-16T15:55:21.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.107+0000 7feffb59e640 1 -- 172.21.15.29:0/1852822120 wait complete. 2023-10-16T15:55:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.109+0000 7feffb59e640 1 Processor -- start 2023-10-16T15:55:21.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.110+0000 7feffb59e640 1 -- start start 2023-10-16T15:55:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 0x7feffc09aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffb59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feffc0c9cf0 0x7feffc0af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffb59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feffc159290 0x7feffc0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc09a600 con 0x7feffc06fa20 2023-10-16T15:55:21.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffb59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc099a10 con 0x7feffc159290 2023-10-16T15:55:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffb59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc099c60 con 0x7feffc0c9cf0 2023-10-16T15:55:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 0x7feffc09aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 0x7feffc09aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37860/0 (socket says 172.21.15.29:37860) 2023-10-16T15:55:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 -- 172.21.15.29:0/899908577 learned_addr learned my addr 172.21.15.29:0/899908577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:21.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 -- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feffc0c9cf0 msgr2=0x7feffc0af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feffc0c9cf0 0x7feffc0af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 -- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feffc159290 msgr2=0x7feffc0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feffc159290 0x7feffc0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.111+0000 7feffa59c640 1 -- 172.21.15.29:0/899908577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec00ac40 con 0x7feffc06fa20 2023-10-16T15:55:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.112+0000 7feffa59c640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 0x7feffc09aa20 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7fefec037040 tx=0x7fefec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.112+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec040670 con 0x7feffc06fa20 2023-10-16T15:55:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.112+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec040810 con 0x7feffc06fa20 2023-10-16T15:55:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.112+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec040bc0 con 0x7feffc06fa20 2023-10-16T15:55:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.112+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc099f40 con 0x7feffc06fa20 2023-10-16T15:55:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.112+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc158ca0 con 0x7feffc06fa20 2023-10-16T15:55:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.114+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc09a740 con 0x7feffc06fa20 2023-10-16T15:55:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.114+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fefec040d20 con 0x7feffc06fa20 2023-10-16T15:55:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.115+0000 7feff2ffd640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fefdc06b020 0x7fefdc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.115+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7fefec0ad260 con 0x7feffc06fa20 2023-10-16T15:55:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.116+0000 7feff9d9b640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fefdc06b020 0x7fefdc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.117+0000 7feff9d9b640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fefdc06b020 0x7fefdc06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feffc09cb80 tx=0x7fefe40079c0 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:21.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.120+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefec077df0 con 0x7feffc06fa20 2023-10-16T15:55:21.301 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:21.298+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.331+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feffc158e30 con 0x7feffc06fa20 2023-10-16T15:55:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.333+0000 7feff2ffd640 1 -- 172.21.15.29:0/899908577 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+17975 (secure 0 0 0) 0x7fefec07b410 con 0x7feffc06fa20 2023-10-16T15:55:21.335 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:21.335 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":255,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:20.854182+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.29:6835/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.29:6833/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fefdc06b020 msgr2=0x7fefdc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fefdc06b020 0x7fefdc06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feffc09cb80 tx=0x7fefe40079c0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 msgr2=0x7feffc09aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 0x7feffc09aa20 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7fefec037040 tx=0x7fefec004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 shutdown_connections 2023-10-16T15:55:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fefdc06b020 0x7fefdc06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feffc159290 0x7feffc0afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7feffc0c9cf0 0x7feffc0af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 --2- 172.21.15.29:0/899908577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feffc06fa20 0x7feffc09aa20 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 >> 172.21.15.29:0/899908577 conn(0x7feffc0b0a80 msgr2=0x7feffc0c93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.336+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 shutdown_connections 2023-10-16T15:55:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.337+0000 7feffb59e640 1 -- 172.21.15.29:0/899908577 wait complete. 2023-10-16T15:55:21.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T15:54:51.157450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T15:54:51.864591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:55:21.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:55:21.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.589+0000 7f2caffff640 1 Processor -- start 2023-10-16T15:55:21.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.590+0000 7f2caffff640 1 -- start start 2023-10-16T15:55:21.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.591+0000 7f2caffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2cb015cc00 0x7f2cb00ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00c8cd0 0x7f2cb00c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb00ce480 con 0x7f2cb00cd260 2023-10-16T15:55:21.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb00c96c0 con 0x7f2cb015cc00 2023-10-16T15:55:21.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00ca950 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2cb015cc00 0x7f2cb00ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2cae7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2cb00cd260 0x7f2cb00ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37648/0 (socket says 172.21.15.29:37648) 2023-10-16T15:55:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2cae7fc640 1 -- 172.21.15.29:0/1690170788 learned_addr learned my addr 172.21.15.29:0/1690170788 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caf7fe640 1 --2- 172.21.15.29:0/1690170788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00c8cd0 0x7f2cb00c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caeffd640 1 -- 172.21.15.29:0/1690170788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00c8cd0 msgr2=0x7f2cb00c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.592+0000 7f2caeffd640 1 --2- 172.21.15.29:0/1690170788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00c8cd0 0x7f2cb00c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.593+0000 7f2caeffd640 1 -- 172.21.15.29:0/1690170788 >> v1:172.21.15.29:6789/0 conn(0x7f2cb00cd260 legacy=0x7f2cb00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.593+0000 7f2caeffd640 1 -- 172.21.15.29:0/1690170788 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00cd7b0 con 0x7f2cb015cc00 2023-10-16T15:55:21.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.593+0000 7f2cadffb640 1 -- 172.21.15.29:0/1690170788 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907887682 0 0) 0x7f2cb00ce480 con 0x7f2cb00cd260 2023-10-16T15:55:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.594+0000 7f2caeffd640 1 --2- 172.21.15.29:0/1690170788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb015cc00 0x7f2cb00ca410 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00097d0 tx=0x7f2ca002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f18d0d750406db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.594+0000 7f2cadffb640 1 -- 172.21.15.29:0/1690170788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca002d040 con 0x7f2cb015cc00 2023-10-16T15:55:21.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.594+0000 7f2cadffb640 1 -- 172.21.15.29:0/1690170788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca0031810 con 0x7f2cb015cc00 2023-10-16T15:55:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.594+0000 7f2cadffb640 1 -- 172.21.15.29:0/1690170788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0031ba0 con 0x7f2cb015cc00 2023-10-16T15:55:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 -- 172.21.15.29:0/1690170788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb015cc00 msgr2=0x7f2cb00ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 --2- 172.21.15.29:0/1690170788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb015cc00 0x7f2cb00ca410 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00097d0 tx=0x7f2ca002b8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 -- 172.21.15.29:0/1690170788 shutdown_connections 2023-10-16T15:55:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 --2- 172.21.15.29:0/1690170788 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00c8cd0 0x7f2cb00c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 --2- 172.21.15.29:0/1690170788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb015cc00 0x7f2cb00ca410 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 -- 172.21.15.29:0/1690170788 >> 172.21.15.29:0/1690170788 conn(0x7f2cb00b0170 msgr2=0x7f2cb00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:21.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 -- 172.21.15.29:0/1690170788 shutdown_connections 2023-10-16T15:55:21.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.595+0000 7f2caffff640 1 -- 172.21.15.29:0/1690170788 wait complete. 2023-10-16T15:55:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.597+0000 7f2caffff640 1 Processor -- start 2023-10-16T15:55:21.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.597+0000 7f2caffff640 1 -- start start 2023-10-16T15:55:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.598+0000 7f2caffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 0x7f2cb006b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.598+0000 7f2caffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00cd260 0x7f2cb00b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.598+0000 7f2caffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cb015cc00 0x7f2cb00b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.598+0000 7f2caffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00c5750 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.598+0000 7f2caffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00b13a0 con 0x7f2cb015cc00 2023-10-16T15:55:21.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.598+0000 7f2caffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00c59a0 con 0x7f2cb00cd260 2023-10-16T15:55:21.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 0x7f2cb006b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 0x7f2cb006b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37884/0 (socket says 172.21.15.29:37884) 2023-10-16T15:55:21.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 -- 172.21.15.29:0/3263562381 learned_addr learned my addr 172.21.15.29:0/3263562381 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:21.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2cae7fc640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00cd260 0x7f2cb00b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caf7fe640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cb015cc00 0x7f2cb00b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 -- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00cd260 msgr2=0x7f2cb00b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00cd260 0x7f2cb00b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 -- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cb015cc00 msgr2=0x7f2cb00b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cb015cc00 0x7f2cb00b1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.599+0000 7f2caeffd640 1 -- 172.21.15.29:0/3263562381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0009400 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.600+0000 7f2caeffd640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 0x7f2cb006b060 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00098f0 tx=0x7f2ca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:21.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.600+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca003ad60 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.600+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca0044400 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.600+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00447b0 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.600+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb00c5c80 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.600+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb006b5a0 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.601+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb006b7f0 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.602+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f2ca004d440 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.603+0000 7f2c977fe640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2c9c06b020 0x7f2c9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:21.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.603+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7f2ca00ac150 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.604+0000 7f2cae7fc640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2c9c06b020 0x7f2c9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:21.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.605+0000 7f2cae7fc640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2c9c06b020 0x7f2c9c06d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00b38d0 tx=0x7f2ca4007b50 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:21.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.608+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ca0076b20 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.807+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2cb00c61a0 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.808+0000 7f2c977fe640 1 -- 172.21.15.29:0/3263562381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v255) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ca007a140 con 0x7f2cb00c8cd0 2023-10-16T15:55:21.809 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:55:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2c9c06b020 msgr2=0x7f2c9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2c9c06b020 0x7f2c9c06d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00b38d0 tx=0x7f2ca4007b50 comp rx=0 tx=0).stop 2023-10-16T15:55:21.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 msgr2=0x7f2cb006b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:21.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 0x7f2cb006b060 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00098f0 tx=0x7f2ca00046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 shutdown_connections 2023-10-16T15:55:21.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2c9c06b020 0x7f2c9c06d4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2cb015cc00 0x7f2cb00b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2cb00cd260 0x7f2cb00b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 --2- 172.21.15.29:0/3263562381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cb00c8cd0 0x7f2cb006b060 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:21.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 >> 172.21.15.29:0/3263562381 conn(0x7f2cb00b0170 msgr2=0x7f2cb00ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:21.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 shutdown_connections 2023-10-16T15:55:21.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:21.811+0000 7f2caffff640 1 -- 172.21.15.29:0/3263562381 wait complete. 2023-10-16T15:55:21.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T15:55:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.067+0000 7f42f4867640 1 Processor -- start 2023-10-16T15:55:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.067+0000 7f42f4867640 1 -- start start 2023-10-16T15:55:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42f4867640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f42f00c9060 0x7f42f006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42f4867640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f006c9e0 0x7f42f006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42f4867640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f00ca560 con 0x7f42f00c9440 2023-10-16T15:55:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42f4867640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f00c9950 con 0x7f42f006c9e0 2023-10-16T15:55:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42f4867640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f00c9c30 con 0x7f42f00c9060 2023-10-16T15:55:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42eeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f006c9e0 0x7f42f006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42eeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42f006c9e0 0x7f42f006f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37898/0 (socket says 172.21.15.29:37898) 2023-10-16T15:55:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.069+0000 7f42eeffd640 1 -- 172.21.15.29:0/4172377001 learned_addr learned my addr 172.21.15.29:0/4172377001 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.070+0000 7f42ef7fe640 1 --2- 172.21.15.29:0/4172377001 >> v2:172.21.15.170:3300/0 conn(0x7f42f00c9060 0x7f42f006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.070+0000 7f42eeffd640 1 -- 172.21.15.29:0/4172377001 >> v2:172.21.15.170:3300/0 conn(0x7f42f00c9060 msgr2=0x7f42f006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.070+0000 7f42eeffd640 1 --2- 172.21.15.29:0/4172377001 >> v2:172.21.15.170:3300/0 conn(0x7f42f00c9060 0x7f42f006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.070+0000 7f42eeffd640 1 -- 172.21.15.29:0/4172377001 >> v1:172.21.15.170:6789/0 conn(0x7f42f00c9440 legacy=0x7f42f0164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.070+0000 7f42eeffd640 1 -- 172.21.15.29:0/4172377001 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f00c9f10 con 0x7f42f006c9e0 2023-10-16T15:55:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.071+0000 7f42eeffd640 1 --2- 172.21.15.29:0/4172377001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f006c9e0 0x7f42f006f980 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f42f006d490 tx=0x7f42e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e9bd2d69616947 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.071+0000 7f42ee7fc640 1 -- 172.21.15.29:0/4172377001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4031870 con 0x7f42f006c9e0 2023-10-16T15:55:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.071+0000 7f42ee7fc640 1 -- 172.21.15.29:0/4172377001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e40319d0 con 0x7f42f006c9e0 2023-10-16T15:55:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.071+0000 7f42ee7fc640 1 -- 172.21.15.29:0/4172377001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4031d60 con 0x7f42f006c9e0 2023-10-16T15:55:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 -- 172.21.15.29:0/4172377001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f006c9e0 msgr2=0x7f42f006f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 --2- 172.21.15.29:0/4172377001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f006c9e0 0x7f42f006f980 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f42f006d490 tx=0x7f42e402b910 comp rx=0 tx=0).stop 2023-10-16T15:55:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 -- 172.21.15.29:0/4172377001 shutdown_connections 2023-10-16T15:55:22.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 --2- 172.21.15.29:0/4172377001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f006c9e0 0x7f42f006f980 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 --2- 172.21.15.29:0/4172377001 >> v2:172.21.15.170:3300/0 conn(0x7f42f00c9060 0x7f42f006f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 -- 172.21.15.29:0/4172377001 >> 172.21.15.29:0/4172377001 conn(0x7f42f00ce6a0 msgr2=0x7f42f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:22.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 -- 172.21.15.29:0/4172377001 shutdown_connections 2023-10-16T15:55:22.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.072+0000 7f42f4867640 1 -- 172.21.15.29:0/4172377001 wait complete. 2023-10-16T15:55:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.074+0000 7f42f4867640 1 Processor -- start 2023-10-16T15:55:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.074+0000 7f42f4867640 1 -- start start 2023-10-16T15:55:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.075+0000 7f42f4867640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f006c9e0 0x7f42f0161510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.075+0000 7f42f4867640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f00c9060 0x7f42f0159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.075+0000 7f42f4867640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 0x7f42f0162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.075+0000 7f42f4867640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f015ae40 con 0x7f42f00c9440 2023-10-16T15:55:22.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.075+0000 7f42f4867640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0161c10 con 0x7f42f006c9e0 2023-10-16T15:55:22.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.075+0000 7f42f4867640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0161ef0 con 0x7f42f00c9060 2023-10-16T15:55:22.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 0x7f42f0162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 0x7f42f0162cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37906/0 (socket says 172.21.15.29:37906) 2023-10-16T15:55:22.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 -- 172.21.15.29:0/246764984 learned_addr learned my addr 172.21.15.29:0/246764984 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:22.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42ef7fe640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f006c9e0 0x7f42f0161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42eeffd640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f00c9060 0x7f42f0159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 -- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f00c9060 msgr2=0x7f42f0159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f00c9060 0x7f42f0159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 -- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f006c9e0 msgr2=0x7f42f0161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f006c9e0 0x7f42f0161510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 -- 172.21.15.29:0/246764984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40094d0 con 0x7f42f00c9440 2023-10-16T15:55:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42eeffd640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f00c9060 0x7f42f0159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:22.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.076+0000 7f42effff640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 0x7f42f0162cb0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f42e800b9e0 tx=0x7f42e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:22.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.077+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8003b70 con 0x7f42f00c9440 2023-10-16T15:55:22.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.077+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e8003cd0 con 0x7f42f00c9440 2023-10-16T15:55:22.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.077+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e800cb60 con 0x7f42f00c9440 2023-10-16T15:55:22.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.077+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f01621d0 con 0x7f42f00c9440 2023-10-16T15:55:22.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.077+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f0162790 con 0x7f42f00c9440 2023-10-16T15:55:22.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.078+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f004e9b0 con 0x7f42f00c9440 2023-10-16T15:55:22.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.079+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f42e800ccc0 con 0x7f42f00c9440 2023-10-16T15:55:22.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.080+0000 7f42d7fff640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f42dc06b020 0x7f42dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.080+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7f42e8089b00 con 0x7f42f00c9440 2023-10-16T15:55:22.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.081+0000 7f42ef7fe640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f42dc06b020 0x7f42dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.082+0000 7f42ef7fe640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f42dc06b020 0x7f42dc06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f42e0006fd0 tx=0x7f42e0008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.085+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42e808be60 con 0x7f42f00c9440 2023-10-16T15:55:22.255 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:22.253+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.295+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42f00b4340 con 0x7f42f00c9440 2023-10-16T15:55:22.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.296+0000 7f42d7fff640 1 -- 172.21.15.29:0/246764984 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v255) v1 ==== 107+0+11 (secure 0 0 0) 0x7f42e80544b0 con 0x7f42f00c9440 2023-10-16T15:55:22.297 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:55:22.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f42dc06b020 msgr2=0x7f42dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f42dc06b020 0x7f42dc06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f42e0006fd0 tx=0x7f42e0008040 comp rx=0 tx=0).stop 2023-10-16T15:55:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 msgr2=0x7f42f0162cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 0x7f42f0162cb0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f42e800b9e0 tx=0x7f42e800bea0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 shutdown_connections 2023-10-16T15:55:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f42dc06b020 0x7f42dc06d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f00c9440 0x7f42f0162cb0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42f00c9060 0x7f42f0159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 --2- 172.21.15.29:0/246764984 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f006c9e0 0x7f42f0161510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.298+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 >> 172.21.15.29:0/246764984 conn(0x7f42f00ce6a0 msgr2=0x7f42f00af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.299+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 shutdown_connections 2023-10-16T15:55:22.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.299+0000 7f42f4867640 1 -- 172.21.15.29:0/246764984 wait complete. 2023-10-16T15:55:22.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T15:55:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.553+0000 7ff22ede4640 1 Processor -- start 2023-10-16T15:55:22.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.553+0000 7ff22ede4640 1 -- start start 2023-10-16T15:55:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.555+0000 7ff22ede4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 0x7ff2280cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.555+0000 7ff22ede4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2280ccfb0 0x7ff2280cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.555+0000 7ff22ede4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22815b9d0 con 0x7ff22806c040 2023-10-16T15:55:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.555+0000 7ff22ede4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22815bcb0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.555+0000 7ff22ede4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22815bf90 con 0x7ff2280ad970 2023-10-16T15:55:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.556+0000 7ff22d5e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 0x7ff2280cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.556+0000 7ff22e5e3640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff2280ccfb0 0x7ff2280cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.556+0000 7ff22d5e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 0x7ff2280cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:44190/0 (socket says 172.21.15.29:44190) 2023-10-16T15:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.556+0000 7ff22d5e1640 1 -- 172.21.15.29:0/2250138780 learned_addr learned my addr 172.21.15.29:0/2250138780 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.557+0000 7ff22d5e1640 1 -- 172.21.15.29:0/2250138780 >> v2:172.21.15.170:3300/0 conn(0x7ff2280ccfb0 msgr2=0x7ff2280cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.557+0000 7ff22d5e1640 1 --2- 172.21.15.29:0/2250138780 >> v2:172.21.15.170:3300/0 conn(0x7ff2280ccfb0 0x7ff2280cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.557+0000 7ff22d5e1640 1 -- 172.21.15.29:0/2250138780 >> v1:172.21.15.170:6789/0 conn(0x7ff22806c040 legacy=0x7ff22806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.557+0000 7ff22d5e1640 1 -- 172.21.15.29:0/2250138780 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280cc270 con 0x7ff2280ad970 2023-10-16T15:55:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.558+0000 7ff22d5e1640 1 --2- 172.21.15.29:0/2250138780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 0x7ff2280cca70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff2180098a0 tx=0x7ff21802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc60d15f3f23d6a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22cde0640 1 -- 172.21.15.29:0/2250138780 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218030840 con 0x7ff2280ad970 2023-10-16T15:55:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22cde0640 1 -- 172.21.15.29:0/2250138780 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2180309a0 con 0x7ff2280ad970 2023-10-16T15:55:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22cde0640 1 -- 172.21.15.29:0/2250138780 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218030d30 con 0x7ff2280ad970 2023-10-16T15:55:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22ede4640 1 -- 172.21.15.29:0/2250138780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 msgr2=0x7ff2280cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2250138780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 0x7ff2280cca70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff2180098a0 tx=0x7ff21802b980 comp rx=0 tx=0).stop 2023-10-16T15:55:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22ede4640 1 -- 172.21.15.29:0/2250138780 shutdown_connections 2023-10-16T15:55:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2250138780 >> v2:172.21.15.170:3300/0 conn(0x7ff2280ccfb0 0x7ff2280cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2250138780 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff2280ad970 0x7ff2280cca70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.559+0000 7ff22ede4640 1 -- 172.21.15.29:0/2250138780 >> 172.21.15.29:0/2250138780 conn(0x7ff2280ce6a0 msgr2=0x7ff2280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.560+0000 7ff22ede4640 1 -- 172.21.15.29:0/2250138780 shutdown_connections 2023-10-16T15:55:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.560+0000 7ff22ede4640 1 -- 172.21.15.29:0/2250138780 wait complete. 2023-10-16T15:55:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.561+0000 7ff22ede4640 1 Processor -- start 2023-10-16T15:55:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.562+0000 7ff22ede4640 1 -- start start 2023-10-16T15:55:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22ede4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff22806c040 0x7ff2280c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22ede4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2280ad970 0x7ff228153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22ede4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 0x7ff2280ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22ede4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228156f80 con 0x7ff2280ccfb0 2023-10-16T15:55:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22ede4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228156ca0 con 0x7ff2280ad970 2023-10-16T15:55:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22ede4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280c8e90 con 0x7ff22806c040 2023-10-16T15:55:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22e5e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 0x7ff2280ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22e5e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 0x7ff2280ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37912/0 (socket says 172.21.15.29:37912) 2023-10-16T15:55:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22e5e3640 1 -- 172.21.15.29:0/2768934861 learned_addr learned my addr 172.21.15.29:0/2768934861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22d5e1640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2280ad970 0x7ff228153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.563+0000 7ff22dde2640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff22806c040 0x7ff2280c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22e5e3640 1 -- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff22806c040 msgr2=0x7ff2280c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22e5e3640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff22806c040 0x7ff2280c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22e5e3640 1 -- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2280ad970 msgr2=0x7ff228153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22e5e3640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2280ad970 0x7ff228153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22e5e3640 1 -- 172.21.15.29:0/2768934861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2180094d0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22e5e3640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 0x7ff2280ac890 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7ff22000efc0 tx=0x7ff22000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220009ba0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22000f040 con 0x7ff2280ccfb0 2023-10-16T15:55:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2200146c0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2280c9170 con 0x7ff2280ccfb0 2023-10-16T15:55:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.564+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2281540b0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.566+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22806b330 con 0x7ff2280ccfb0 2023-10-16T15:55:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.567+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7ff220014820 con 0x7ff2280ccfb0 2023-10-16T15:55:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.568+0000 7ff21e7fc640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff20c06b020 0x7ff20c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.568+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7ff22008a950 con 0x7ff2280ccfb0 2023-10-16T15:55:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.569+0000 7ff22dde2640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff20c06b020 0x7ff20c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.570+0000 7ff22dde2640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff20c06b020 0x7ff20c06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff224001640 tx=0x7ff224009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.573+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff220055290 con 0x7ff2280ccfb0 2023-10-16T15:55:22.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.775+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2280c8cf0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.775+0000 7ff21e7fc640 1 -- 172.21.15.29:0/2768934861 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v255) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff2280c8cf0 con 0x7ff2280ccfb0 2023-10-16T15:55:22.776 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:55:22.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff20c06b020 msgr2=0x7ff20c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff20c06b020 0x7ff20c06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff224001640 tx=0x7ff224009040 comp rx=0 tx=0).stop 2023-10-16T15:55:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 msgr2=0x7ff2280ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 0x7ff2280ac890 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7ff22000efc0 tx=0x7ff22000c490 comp rx=0 tx=0).stop 2023-10-16T15:55:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 shutdown_connections 2023-10-16T15:55:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff20c06b020 0x7ff20c06d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2280ccfb0 0x7ff2280ac890 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff2280ad970 0x7ff228153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 --2- 172.21.15.29:0/2768934861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff22806c040 0x7ff2280c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:22.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 >> 172.21.15.29:0/2768934861 conn(0x7ff2280ce6a0 msgr2=0x7ff2280b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:22.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 shutdown_connections 2023-10-16T15:55:22.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:22.778+0000 7ff22ede4640 1 -- 172.21.15.29:0/2768934861 wait complete. 2023-10-16T15:55:22.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.030+0000 7efea1767640 1 Processor -- start 2023-10-16T15:55:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.030+0000 7efea1767640 1 -- start start 2023-10-16T15:55:23.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efea1767640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7efe9c06c040 0x7efe9c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efea1767640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c15f4a0 con 0x7efe9c0ad970 2023-10-16T15:55:23.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efea1767640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c06c960 con 0x7efe9c0cda80 2023-10-16T15:55:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efea1767640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9c15b9d0 con 0x7efe9c06c040 2023-10-16T15:55:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efe93fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efe9c0ad970 0x7efe9c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37658/0 (socket says 172.21.15.29:37658) 2023-10-16T15:55:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efe93fff640 1 -- 172.21.15.29:0/525149594 learned_addr learned my addr 172.21.15.29:0/525149594 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831287873 0 0) 0x7efe9c15f4a0 con 0x7efe9c0ad970 2023-10-16T15:55:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.032+0000 7efe9bfff640 1 --2- 172.21.15.29:0/525149594 >> v2:172.21.15.170:3300/0 conn(0x7efe9c06c040 0x7efe9c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88003520 con 0x7efe9c0ad970 2023-10-16T15:55:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892222048 0 0) 0x7efe88003520 con 0x7efe9c0ad970 2023-10-16T15:55:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 >> v2:172.21.15.170:3300/0 conn(0x7efe9c06c040 msgr2=0x7efe9c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 --2- 172.21.15.29:0/525149594 >> v2:172.21.15.170:3300/0 conn(0x7efe9c06c040 0x7efe9c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 >> v1:172.21.15.170:6789/0 conn(0x7efe9c0cda80 legacy=0x7efe9c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c15bcb0 con 0x7efe9c0ad970 2023-10-16T15:55:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7efe84001670 con 0x7efe9c0ad970 2023-10-16T15:55:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 >> v1:172.21.15.29:6789/0 conn(0x7efe9c0ad970 legacy=0x7efe9c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe880039c0 0x7efe88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe880047c0 0x7efe88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c15b9d0 con 0x7efe880051d0 2023-10-16T15:55:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8800c100 con 0x7efe880047c0 2023-10-16T15:55:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8800c2c0 con 0x7efe880039c0 2023-10-16T15:55:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.033+0000 7efea0f66640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efe93fff640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe880047c0 0x7efe88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efe9bfff640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe880039c0 0x7efe88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efea0f66640 1 -- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe880039c0 msgr2=0x7efe88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efea0f66640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe880039c0 0x7efe88003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efea0f66640 1 -- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe880047c0 msgr2=0x7efe88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efea0f66640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe880047c0 0x7efe88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efea0f66640 1 -- 172.21.15.29:0/525149594 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe88003520 con 0x7efe880051d0 2023-10-16T15:55:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efe9bfff640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe880039c0 0x7efe88003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.034+0000 7efe93fff640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe880047c0 0x7efe88004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.035+0000 7efea0f66640 1 -- 172.21.15.29:0/525149594 reap_dead start 2023-10-16T15:55:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.035+0000 7efea0f66640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe8800b980 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7efe94004480 tx=0x7efe880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88b58b526af4f74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94030400 con 0x7efe880051d0 2023-10-16T15:55:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe94030560 con 0x7efe880051d0 2023-10-16T15:55:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efe9b7fe640 1 -- 172.21.15.29:0/525149594 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe940308f0 con 0x7efe880051d0 2023-10-16T15:55:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 -- 172.21.15.29:0/525149594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 msgr2=0x7efe8800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe8800b980 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7efe94004480 tx=0x7efe880057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 -- 172.21.15.29:0/525149594 shutdown_connections 2023-10-16T15:55:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 --2- 172.21.15.29:0/525149594 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe8800b980 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 -- 172.21.15.29:0/525149594 >> 172.21.15.29:0/525149594 conn(0x7efe9c0ce6a0 msgr2=0x7efe9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 -- 172.21.15.29:0/525149594 shutdown_connections 2023-10-16T15:55:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.036+0000 7efea1767640 1 -- 172.21.15.29:0/525149594 wait complete. 2023-10-16T15:55:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.038+0000 7efea1767640 1 Processor -- start 2023-10-16T15:55:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.039+0000 7efea1767640 1 -- start start 2023-10-16T15:55:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea1767640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe9c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea1767640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe9c0c21b0 0x7efe9c0ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea1767640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe9c0c2630 0x7efe9c0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea1767640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c2b90 con 0x7efe880051d0 2023-10-16T15:55:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea1767640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c2e70 con 0x7efe9c0c2630 2023-10-16T15:55:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea1767640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c3150 con 0x7efe9c0c21b0 2023-10-16T15:55:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efe9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe9c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efe9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe9c15d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37926/0 (socket says 172.21.15.29:37926) 2023-10-16T15:55:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efe9bfff640 1 -- 172.21.15.29:0/3510012739 learned_addr learned my addr 172.21.15.29:0/3510012739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efea0f66640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe9c0c2630 0x7efe9c0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efe93fff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe9c0c21b0 0x7efe9c0ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.040+0000 7efe9bfff640 1 -- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe9c0c21b0 msgr2=0x7efe9c0ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe9bfff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe9c0c21b0 0x7efe9c0ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe9bfff640 1 -- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe9c0c2630 msgr2=0x7efe9c0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe9bfff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe9c0c2630 0x7efe9c0aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe9bfff640 1 -- 172.21.15.29:0/3510012739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c0adff0 con 0x7efe880051d0 2023-10-16T15:55:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efea0f66640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe9c0c2630 0x7efe9c0aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe93fff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe9c0c21b0 0x7efe9c0ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe9bfff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe9c15d740 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00f550 tx=0x7efe8c00fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c013810 con 0x7efe880051d0 2023-10-16T15:55:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c013970 con 0x7efe880051d0 2023-10-16T15:55:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c013ce0 con 0x7efe880051d0 2023-10-16T15:55:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c1f3b60 con 0x7efe880051d0 2023-10-16T15:55:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.041+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c15de60 con 0x7efe880051d0 2023-10-16T15:55:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.043+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c06b330 con 0x7efe880051d0 2023-10-16T15:55:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.044+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7efe8c01f460 con 0x7efe880051d0 2023-10-16T15:55:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.045+0000 7efe997fa640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efe7c06b050 0x7efe7c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.045+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7efe8c089eb0 con 0x7efe880051d0 2023-10-16T15:55:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.046+0000 7efe93fff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efe7c06b050 0x7efe7c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.047+0000 7efe93fff640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efe7c06b050 0x7efe7c06d510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efe84001eb0 tx=0x7efe84001de0 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.050+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efe8c054900 con 0x7efe880051d0 2023-10-16T15:55:23.221 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:23.218+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:23.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.287+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe9c0ade50 con 0x7efe880051d0 2023-10-16T15:55:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.289+0000 7efe997fa640 1 -- 172.21.15.29:0/3510012739 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+4930 (secure 0 0 0) 0x7efe9c0ade50 con 0x7efe880051d0 2023-10-16T15:55:23.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:23.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":132,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":12150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/356635286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":356635286},{"type":"v1","addr":"172.21.15.29:6835","nonce":356635286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 132 2023-10-16T15:55:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.292+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efe7c06b050 msgr2=0x7efe7c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efe7c06b050 0x7efe7c06d510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efe84001eb0 tx=0x7efe84001de0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 msgr2=0x7efe9c15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe9c15d740 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00f550 tx=0x7efe8c00fa10 comp rx=0 tx=0).stop 2023-10-16T15:55:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 shutdown_connections 2023-10-16T15:55:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efe7c06b050 0x7efe7c06d510 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efe9c0c2630 0x7efe9c0aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efe9c0c21b0 0x7efe9c0ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 --2- 172.21.15.29:0/3510012739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe880051d0 0x7efe9c15d740 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 >> 172.21.15.29:0/3510012739 conn(0x7efe9c0ce6a0 msgr2=0x7efe9c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 shutdown_connections 2023-10-16T15:55:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.293+0000 7efea1767640 1 -- 172.21.15.29:0/3510012739 wait complete. 2023-10-16T15:55:23.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.542+0000 7fa72929b640 1 Processor -- start 2023-10-16T15:55:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.542+0000 7fa72929b640 1 -- start start 2023-10-16T15:55:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa72929b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa724159b20 0x7fa72406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa72929b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72406f7f0 con 0x7fa724159f00 2023-10-16T15:55:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa72929b640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72406c5e0 con 0x7fa7240b0b20 2023-10-16T15:55:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa72929b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72406c8c0 con 0x7fa724159b20 2023-10-16T15:55:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa722ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa724159b20 0x7fa72406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa722ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa724159b20 0x7fa72406bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37940/0 (socket says 172.21.15.29:37940) 2023-10-16T15:55:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa722ffd640 1 -- 172.21.15.29:0/1270261567 learned_addr learned my addr 172.21.15.29:0/1270261567 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:23.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.544+0000 7fa7227fc640 1 --1- 172.21.15.29:0/1270261567 >> v1:172.21.15.29:6789/0 conn(0x7fa724159f00 0x7fa7240b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37674/0 (socket says 172.21.15.29:37674) 2023-10-16T15:55:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.545+0000 7fa722ffd640 1 -- 172.21.15.29:0/1270261567 >> v1:172.21.15.170:6789/0 conn(0x7fa7240b0b20 legacy=0x7fa72406fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.545+0000 7fa722ffd640 1 -- 172.21.15.29:0/1270261567 >> v1:172.21.15.29:6789/0 conn(0x7fa724159f00 legacy=0x7fa7240b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.545+0000 7fa722ffd640 1 -- 172.21.15.29:0/1270261567 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72406cba0 con 0x7fa724159b20 2023-10-16T15:55:23.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.546+0000 7fa722ffd640 1 --2- 172.21.15.29:0/1270261567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa724159b20 0x7fa72406bbe0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fa714009870 tx=0x7fa71402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237c612967acb88c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.546+0000 7fa721ffb640 1 -- 172.21.15.29:0/1270261567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714030860 con 0x7fa724159b20 2023-10-16T15:55:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.546+0000 7fa721ffb640 1 -- 172.21.15.29:0/1270261567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7140309c0 con 0x7fa724159b20 2023-10-16T15:55:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.546+0000 7fa721ffb640 1 -- 172.21.15.29:0/1270261567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714030d50 con 0x7fa724159b20 2023-10-16T15:55:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 -- 172.21.15.29:0/1270261567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa724159b20 msgr2=0x7fa72406bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 --2- 172.21.15.29:0/1270261567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa724159b20 0x7fa72406bbe0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fa714009870 tx=0x7fa71402b9a0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 -- 172.21.15.29:0/1270261567 shutdown_connections 2023-10-16T15:55:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 --2- 172.21.15.29:0/1270261567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa724159b20 0x7fa72406bbe0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 -- 172.21.15.29:0/1270261567 >> 172.21.15.29:0/1270261567 conn(0x7fa7240cb410 msgr2=0x7fa7240cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 -- 172.21.15.29:0/1270261567 shutdown_connections 2023-10-16T15:55:23.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.547+0000 7fa72929b640 1 -- 172.21.15.29:0/1270261567 wait complete. 2023-10-16T15:55:23.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.549+0000 7fa72929b640 1 Processor -- start 2023-10-16T15:55:23.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.549+0000 7fa72929b640 1 -- start start 2023-10-16T15:55:23.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa72929b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 0x7fa72414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa72929b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa724159b20 0x7fa7240b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa72929b640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa724159f00 0x7fa7241514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa72929b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240c7c00 con 0x7fa7240b0b20 2023-10-16T15:55:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa72929b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240c2a30 con 0x7fa724159b20 2023-10-16T15:55:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa72929b640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240c2d10 con 0x7fa724159f00 2023-10-16T15:55:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.550+0000 7fa722ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 0x7fa72414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 0x7fa72414f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37946/0 (socket says 172.21.15.29:37946) 2023-10-16T15:55:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 -- 172.21.15.29:0/629951978 learned_addr learned my addr 172.21.15.29:0/629951978 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa7227fc640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa724159b20 0x7fa7240b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa7237fe640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa724159f00 0x7fa7241514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 -- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa724159f00 msgr2=0x7fa7241514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa724159f00 0x7fa7241514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 -- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa724159b20 msgr2=0x7fa7240b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa724159b20 0x7fa7240b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 -- 172.21.15.29:0/629951978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140094d0 con 0x7fa7240b0b20 2023-10-16T15:55:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa7237fe640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa724159f00 0x7fa7241514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.551+0000 7fa722ffd640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 0x7fa72414f780 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fa714009870 tx=0x7fa714004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.552+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714030810 con 0x7fa7240b0b20 2023-10-16T15:55:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.552+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa714043400 con 0x7fa7240b0b20 2023-10-16T15:55:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.552+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7240c2ff0 con 0x7fa7240b0b20 2023-10-16T15:55:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.552+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7140437b0 con 0x7fa7240b0b20 2023-10-16T15:55:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.552+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7240c6430 con 0x7fa7240b0b20 2023-10-16T15:55:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.553+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7240c6290 con 0x7fa7240b0b20 2023-10-16T15:55:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.554+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa714043910 con 0x7fa7240b0b20 2023-10-16T15:55:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.555+0000 7fa70b7fe640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa71006b020 0x7fa71006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.555+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7fa71403e070 con 0x7fa7240b0b20 2023-10-16T15:55:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.556+0000 7fa7227fc640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa71006b020 0x7fa71006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:23.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.557+0000 7fa7227fc640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa71006b020 0x7fa71006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7240b3250 tx=0x7fa718007ca0 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:23.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.559+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7240c6290 con 0x7fa7240b0b20 2023-10-16T15:55:23.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.757+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7240c6290 con 0x7fa7240b0b20 2023-10-16T15:55:23.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.761+0000 7fa70b7fe640 1 -- 172.21.15.29:0/629951978 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+17975 (secure 0 0 0) 0x7fa7240c6290 con 0x7fa7240b0b20 2023-10-16T15:55:23.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:23.762 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":255,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:20.854182+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.29:6835/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.29:6833/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.763+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa71006b020 msgr2=0x7fa71006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.763+0000 7fa72929b640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa71006b020 0x7fa71006d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7240b3250 tx=0x7fa718007ca0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.763+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 msgr2=0x7fa72414f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:23.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.763+0000 7fa72929b640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 0x7fa72414f780 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fa714009870 tx=0x7fa714004730 comp rx=0 tx=0).stop 2023-10-16T15:55:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 shutdown_connections 2023-10-16T15:55:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa71006b020 0x7fa71006d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa724159f00 0x7fa7241514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa724159b20 0x7fa7240b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 --2- 172.21.15.29:0/629951978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7240b0b20 0x7fa72414f780 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 >> 172.21.15.29:0/629951978 conn(0x7fa7240cb410 msgr2=0x7fa7240b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 shutdown_connections 2023-10-16T15:55:23.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:23.764+0000 7fa72929b640 1 -- 172.21.15.29:0/629951978 wait complete. 2023-10-16T15:55:23.784 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T15:55:23.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.005+0000 7f5665097640 1 Processor -- start 2023-10-16T15:55:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.006+0000 7f5665097640 1 -- start start 2023-10-16T15:55:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f5665097640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f566006b4c0 0x7f56600ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f5665097640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f56600cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f5665097640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566006e170 con 0x7f5660160420 2023-10-16T15:55:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f5665097640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56600cde30 con 0x7f566006b4c0 2023-10-16T15:55:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f5665097640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56600cd460 con 0x7f5660160800 2023-10-16T15:55:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f565ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5660160420 0x7f566006adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54720/0 (socket says 172.21.15.29:54720) 2023-10-16T15:55:24.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.007+0000 7f565ed9d640 1 -- 172.21.15.29:0/2245471449 learned_addr learned my addr 172.21.15.29:0/2245471449 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947258446 0 0) 0x7f566006e170 con 0x7f5660160420 2023-10-16T15:55:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565f59e640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f56600cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f564c003520 con 0x7f5660160420 2023-10-16T15:55:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565659c640 1 --2- 172.21.15.29:0/2245471449 >> v2:172.21.15.170:3300/0 conn(0x7f566006b4c0 0x7f56600ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365100624 0 0) 0x7f564c003520 con 0x7f5660160420 2023-10-16T15:55:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 msgr2=0x7f56600cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f56600cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 >> v2:172.21.15.170:3300/0 conn(0x7f566006b4c0 msgr2=0x7f56600ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 --2- 172.21.15.29:0/2245471449 >> v2:172.21.15.170:3300/0 conn(0x7f566006b4c0 0x7f56600ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566006e3c0 con 0x7f5660160420 2023-10-16T15:55:24.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f5650001670 con 0x7f5660160420 2023-10-16T15:55:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 >> v1:172.21.15.29:6789/0 conn(0x7f5660160420 legacy=0x7f566006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f564c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c003ef0 0x7f564c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f564c004d00 0x7f564c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56600cde30 con 0x7f564c003ef0 2023-10-16T15:55:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56600cd460 con 0x7f564c004d00 2023-10-16T15:55:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.008+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c00b9d0 con 0x7f5660160800 2023-10-16T15:55:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565659c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c003ef0 0x7f564c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565f59e640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f564c004d00 0x7f564c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565ed9d640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f564c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565659c640 1 -- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 msgr2=0x7f564c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565659c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f564c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565659c640 1 -- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f564c004d00 msgr2=0x7f564c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565659c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f564c004d00 0x7f564c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565659c640 1 -- 172.21.15.29:0/2245471449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c003520 con 0x7f564c003ef0 2023-10-16T15:55:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565f59e640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f564c004d00 0x7f564c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.009+0000 7f565ed9d640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5660160800 0x7f564c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.010+0000 7f565659c640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c003ef0 0x7f564c0047c0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f564800ba30 tx=0x7f564c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeefb5453b0f7593 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648030660 con 0x7f564c003ef0 2023-10-16T15:55:24.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56480307c0 con 0x7f564c003ef0 2023-10-16T15:55:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f565e59c640 1 -- 172.21.15.29:0/2245471449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648030b50 con 0x7f564c003ef0 2023-10-16T15:55:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f5665097640 1 -- 172.21.15.29:0/2245471449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c003ef0 msgr2=0x7f564c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f5665097640 1 --2- 172.21.15.29:0/2245471449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c003ef0 0x7f564c0047c0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f564800ba30 tx=0x7f564c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f565f59e640 1 -- 172.21.15.29:0/2245471449 reap_dead start 2023-10-16T15:55:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f5665097640 1 -- 172.21.15.29:0/2245471449 shutdown_connections 2023-10-16T15:55:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f5665097640 1 -- 172.21.15.29:0/2245471449 >> 172.21.15.29:0/2245471449 conn(0x7f56600b0d30 msgr2=0x7f56600b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f5665097640 1 -- 172.21.15.29:0/2245471449 shutdown_connections 2023-10-16T15:55:24.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.011+0000 7f5665097640 1 -- 172.21.15.29:0/2245471449 wait complete. 2023-10-16T15:55:24.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.013+0000 7f5665097640 1 Processor -- start 2023-10-16T15:55:24.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.013+0000 7f5665097640 1 -- start start 2023-10-16T15:55:24.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.014+0000 7f5665097640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f564c003ef0 0x7f5660159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.014+0000 7f5665097640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566009fb50 0x7f566006a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.014+0000 7f5665097640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 0x7f566006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.014+0000 7f5665097640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566009e7a0 con 0x7f566009ffd0 2023-10-16T15:55:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.014+0000 7f5665097640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566009ea80 con 0x7f566009fb50 2023-10-16T15:55:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.014+0000 7f5665097640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566009ed60 con 0x7f564c003ef0 2023-10-16T15:55:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 0x7f566006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 0x7f566006ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54206/0 (socket says 172.21.15.29:54206) 2023-10-16T15:55:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 -- 172.21.15.29:0/2555897417 learned_addr learned my addr 172.21.15.29:0/2555897417 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565ed9d640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f564c003ef0 0x7f5660159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565659c640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566009fb50 0x7f566006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 -- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f564c003ef0 msgr2=0x7f5660159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f564c003ef0 0x7f5660159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 -- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566009fb50 msgr2=0x7f566006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566009fb50 0x7f566006a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 -- 172.21.15.29:0/2555897417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566009f040 con 0x7f566009ffd0 2023-10-16T15:55:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.015+0000 7f565f59e640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 0x7f566006ab80 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f5660160c50 tx=0x7f565800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.016+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658017400 con 0x7f566009ffd0 2023-10-16T15:55:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.016+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5658011040 con 0x7f566009ffd0 2023-10-16T15:55:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.016+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566009f320 con 0x7f566009ffd0 2023-10-16T15:55:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.016+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580177c0 con 0x7f566009ffd0 2023-10-16T15:55:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.016+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566015bf40 con 0x7f566009ffd0 2023-10-16T15:55:24.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.017+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5660073550 con 0x7f566009ffd0 2023-10-16T15:55:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.018+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f56580203b0 con 0x7f566009ffd0 2023-10-16T15:55:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.019+0000 7f5657fff640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f564006b050 0x7f564006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.019+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7f5658089f00 con 0x7f566009ffd0 2023-10-16T15:55:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.020+0000 7f565ed9d640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f564006b050 0x7f564006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.021+0000 7f565ed9d640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f564006b050 0x7f564006d510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5650001820 tx=0x7f5650009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:24.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.025+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5658054840 con 0x7f566009ffd0 2023-10-16T15:55:24.189 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:24.186+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:24.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.257+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f566004e160 con 0x7f566009ffd0 2023-10-16T15:55:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.258+0000 7f5657fff640 1 -- 172.21.15.29:0/2555897417 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+4930 (secure 0 0 0) 0x7f5658057e60 con 0x7f566009ffd0 2023-10-16T15:55:24.260 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:24.260 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":132,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":12150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/356635286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":356635286},{"type":"v1","addr":"172.21.15.29:6835","nonce":356635286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 132 2023-10-16T15:55:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f564006b050 msgr2=0x7f564006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f564006b050 0x7f564006d510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5650001820 tx=0x7f5650009040 comp rx=0 tx=0).stop 2023-10-16T15:55:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 msgr2=0x7f566006ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 0x7f566006ab80 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f5660160c50 tx=0x7f565800f4a0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 shutdown_connections 2023-10-16T15:55:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f564006b050 0x7f564006d510 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f566009ffd0 0x7f566006ab80 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f566009fb50 0x7f566006a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 --2- 172.21.15.29:0/2555897417 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f564c003ef0 0x7f5660159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 >> 172.21.15.29:0/2555897417 conn(0x7f56600b0d30 msgr2=0x7f56600afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 shutdown_connections 2023-10-16T15:55:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.261+0000 7f5665097640 1 -- 172.21.15.29:0/2555897417 wait complete. 2023-10-16T15:55:24.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:24.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.513+0000 7f7e7972e640 1 Processor -- start 2023-10-16T15:55:24.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.513+0000 7f7e7972e640 1 -- start start 2023-10-16T15:55:24.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e7972e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e7406c040 0x7f7e7406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e7972e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7e740ad970 0x7f7e7415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e7972e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7406c960 con 0x7f7e7415ebc0 2023-10-16T15:55:24.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e7972e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7415b9d0 con 0x7f7e7406c040 2023-10-16T15:55:24.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e7972e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7415bcb0 con 0x7f7e740ad970 2023-10-16T15:55:24.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e73fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e7406c040 0x7f7e7406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e73fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7e7406c040 0x7f7e7406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54208/0 (socket says 172.21.15.29:54208) 2023-10-16T15:55:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.515+0000 7f7e73fff640 1 -- 172.21.15.29:0/405377807 learned_addr learned my addr 172.21.15.29:0/405377807 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.516+0000 7f7e737fe640 1 --2- 172.21.15.29:0/405377807 >> v2:172.21.15.170:3300/0 conn(0x7f7e740ad970 0x7f7e7415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.516+0000 7f7e73fff640 1 -- 172.21.15.29:0/405377807 >> v2:172.21.15.170:3300/0 conn(0x7f7e740ad970 msgr2=0x7f7e7415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:24.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.516+0000 7f7e73fff640 1 --2- 172.21.15.29:0/405377807 >> v2:172.21.15.170:3300/0 conn(0x7f7e740ad970 0x7f7e7415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.516+0000 7f7e73fff640 1 -- 172.21.15.29:0/405377807 >> v1:172.21.15.170:6789/0 conn(0x7f7e7415ebc0 legacy=0x7f7e740cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:24.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.516+0000 7f7e73fff640 1 -- 172.21.15.29:0/405377807 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7415bf90 con 0x7f7e7406c040 2023-10-16T15:55:24.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.516+0000 7f7e737fe640 1 --2- 172.21.15.29:0/405377807 >> v2:172.21.15.170:3300/0 conn(0x7f7e740ad970 0x7f7e7415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.517+0000 7f7e73fff640 1 --2- 172.21.15.29:0/405377807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7406c040 0x7f7e7406c420 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f7e740cb060 tx=0x7f7e6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6142c7c549c0dfb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.517+0000 7f7e72ffd640 1 -- 172.21.15.29:0/405377807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64031870 con 0x7f7e7406c040 2023-10-16T15:55:24.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.517+0000 7f7e72ffd640 1 -- 172.21.15.29:0/405377807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e640319d0 con 0x7f7e7406c040 2023-10-16T15:55:24.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.517+0000 7f7e72ffd640 1 -- 172.21.15.29:0/405377807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64031d60 con 0x7f7e7406c040 2023-10-16T15:55:24.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 -- 172.21.15.29:0/405377807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7406c040 msgr2=0x7f7e7406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 --2- 172.21.15.29:0/405377807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7406c040 0x7f7e7406c420 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f7e740cb060 tx=0x7f7e6402b910 comp rx=0 tx=0).stop 2023-10-16T15:55:24.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 -- 172.21.15.29:0/405377807 shutdown_connections 2023-10-16T15:55:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 --2- 172.21.15.29:0/405377807 >> v2:172.21.15.170:3300/0 conn(0x7f7e740ad970 0x7f7e7415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 --2- 172.21.15.29:0/405377807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e7406c040 0x7f7e7406c420 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 -- 172.21.15.29:0/405377807 >> 172.21.15.29:0/405377807 conn(0x7f7e740ce6a0 msgr2=0x7f7e740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.518+0000 7f7e7972e640 1 -- 172.21.15.29:0/405377807 shutdown_connections 2023-10-16T15:55:24.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.519+0000 7f7e7972e640 1 -- 172.21.15.29:0/405377807 wait complete. 2023-10-16T15:55:24.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.520+0000 7f7e7972e640 1 Processor -- start 2023-10-16T15:55:24.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.521+0000 7f7e7972e640 1 -- start start 2023-10-16T15:55:24.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e7972e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e7406c040 0x7f7e741f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e7972e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 0x7f7e741f51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e7972e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e7415ebc0 0x7f7e741f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e7972e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7415a860 con 0x7f7e740ad970 2023-10-16T15:55:24.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e7972e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741f3350 con 0x7f7e7415ebc0 2023-10-16T15:55:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e7972e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741f5730 con 0x7f7e7406c040 2023-10-16T15:55:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 0x7f7e741f51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 0x7f7e741f51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54214/0 (socket says 172.21.15.29:54214) 2023-10-16T15:55:24.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 -- 172.21.15.29:0/3747985606 learned_addr learned my addr 172.21.15.29:0/3747985606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 -- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e7406c040 msgr2=0x7f7e741f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e7406c040 0x7f7e741f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 -- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e7415ebc0 msgr2=0x7f7e741f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e78f2d640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e7415ebc0 0x7f7e741f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.522+0000 7f7e737fe640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e7415ebc0 0x7f7e741f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e737fe640 1 -- 172.21.15.29:0/3747985606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e640094d0 con 0x7f7e740ad970 2023-10-16T15:55:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e737fe640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 0x7f7e741f51f0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800c7f0 tx=0x7f7e6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68007820 con 0x7f7e740ad970 2023-10-16T15:55:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68007980 con 0x7f7e740ad970 2023-10-16T15:55:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e740af2f0 con 0x7f7e740ad970 2023-10-16T15:55:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68007d30 con 0x7f7e740ad970 2023-10-16T15:55:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.523+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e740af8b0 con 0x7f7e740ad970 2023-10-16T15:55:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.525+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7406b330 con 0x7f7e740ad970 2023-10-16T15:55:24.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.526+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f7e680113f0 con 0x7f7e740ad970 2023-10-16T15:55:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.527+0000 7f7e70ff9640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f7e6006b020 0x7f7e6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:24.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.527+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7f7e6808a150 con 0x7f7e740ad970 2023-10-16T15:55:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.527+0000 7f7e73fff640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f7e6006b020 0x7f7e6006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.529+0000 7f7e73fff640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f7e6006b020 0x7f7e6006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e64036000 tx=0x7f7e64030000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:24.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.532+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e68054a90 con 0x7f7e740ad970 2023-10-16T15:55:24.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.770+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e7404e9b0 con 0x7f7e740ad970 2023-10-16T15:55:24.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.772+0000 7f7e70ff9640 1 -- 172.21.15.29:0/3747985606 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+4930 (secure 0 0 0) 0x7f7e680580b0 con 0x7f7e740ad970 2023-10-16T15:55:24.774 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:24.774 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":132,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":12150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/356635286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":356635286},{"type":"v1","addr":"172.21.15.29:6835","nonce":356635286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:54:52.836357+0000","modified":"2023-10-16T15:54:55.930651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11613":{"gid":11613,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6834/2428945430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2428945430},{"type":"v1","addr":"172.21.15.29:6834","nonce":2428945430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T15:55:24.775 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 132 2023-10-16T15:55:24.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f7e6006b020 msgr2=0x7f7e6006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f7e6006b020 0x7f7e6006d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e64036000 tx=0x7f7e64030000 comp rx=0 tx=0).stop 2023-10-16T15:55:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 msgr2=0x7f7e741f51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 0x7f7e741f51f0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800c7f0 tx=0x7f7e6800ccb0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 shutdown_connections 2023-10-16T15:55:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f7e6006b020 0x7f7e6006d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e740ad970 0x7f7e741f51f0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7e7415ebc0 0x7f7e741f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 --2- 172.21.15.29:0/3747985606 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7e7406c040 0x7f7e741f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:24.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 >> 172.21.15.29:0/3747985606 conn(0x7f7e740ce6a0 msgr2=0x7f7e740c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.774+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 shutdown_connections 2023-10-16T15:55:24.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:24.775+0000 7f7e7972e640 1 -- 172.21.15.29:0/3747985606 wait complete. 2023-10-16T15:55:24.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:25.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.027+0000 7ff195ccf640 1 Processor -- start 2023-10-16T15:55:25.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.027+0000 7ff195ccf640 1 -- start start 2023-10-16T15:55:25.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff195ccf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff19015aaa0 0x7ff19015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff195ccf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff1900ce390 0x7ff190070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff195ccf640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19015ed60 con 0x7ff19015ae80 2023-10-16T15:55:25.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff195ccf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19006b810 con 0x7ff19015aaa0 2023-10-16T15:55:25.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff195ccf640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900707e0 con 0x7ff1900ce390 2023-10-16T15:55:25.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff18f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff19015aaa0 0x7ff19015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff18f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff19015aaa0 0x7ff19015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54228/0 (socket says 172.21.15.29:54228) 2023-10-16T15:55:25.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.029+0000 7ff18f7fe640 1 -- 172.21.15.29:0/3175981905 learned_addr learned my addr 172.21.15.29:0/3175981905 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:25.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.030+0000 7ff18ffff640 1 --2- 172.21.15.29:0/3175981905 >> v2:172.21.15.170:3300/0 conn(0x7ff1900ce390 0x7ff190070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.030+0000 7ff18f7fe640 1 -- 172.21.15.29:0/3175981905 >> v2:172.21.15.170:3300/0 conn(0x7ff1900ce390 msgr2=0x7ff190070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.030+0000 7ff18f7fe640 1 --2- 172.21.15.29:0/3175981905 >> v2:172.21.15.170:3300/0 conn(0x7ff1900ce390 0x7ff190070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.030+0000 7ff18f7fe640 1 -- 172.21.15.29:0/3175981905 >> v1:172.21.15.170:6789/0 conn(0x7ff19015ae80 legacy=0x7ff1900cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.030+0000 7ff18f7fe640 1 -- 172.21.15.29:0/3175981905 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19015ef90 con 0x7ff19015aaa0 2023-10-16T15:55:25.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.030+0000 7ff18ffff640 1 --2- 172.21.15.29:0/3175981905 >> v2:172.21.15.170:3300/0 conn(0x7ff1900ce390 0x7ff190070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:25.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.031+0000 7ff18f7fe640 1 --2- 172.21.15.29:0/3175981905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff19015e820 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7ff1900c2490 tx=0x7ff18002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aabd2090226a4cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.031+0000 7ff18effd640 1 -- 172.21.15.29:0/3175981905 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180031870 con 0x7ff19015aaa0 2023-10-16T15:55:25.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.031+0000 7ff18effd640 1 -- 172.21.15.29:0/3175981905 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1800319d0 con 0x7ff19015aaa0 2023-10-16T15:55:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.031+0000 7ff18effd640 1 -- 172.21.15.29:0/3175981905 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180031d60 con 0x7ff19015aaa0 2023-10-16T15:55:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 -- 172.21.15.29:0/3175981905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 msgr2=0x7ff19015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 --2- 172.21.15.29:0/3175981905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff19015e820 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7ff1900c2490 tx=0x7ff18002b910 comp rx=0 tx=0).stop 2023-10-16T15:55:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 -- 172.21.15.29:0/3175981905 shutdown_connections 2023-10-16T15:55:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 --2- 172.21.15.29:0/3175981905 >> v2:172.21.15.170:3300/0 conn(0x7ff1900ce390 0x7ff190070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 --2- 172.21.15.29:0/3175981905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff19015e820 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 -- 172.21.15.29:0/3175981905 >> 172.21.15.29:0/3175981905 conn(0x7ff1900cb410 msgr2=0x7ff1900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 -- 172.21.15.29:0/3175981905 shutdown_connections 2023-10-16T15:55:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.032+0000 7ff195ccf640 1 -- 172.21.15.29:0/3175981905 wait complete. 2023-10-16T15:55:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.034+0000 7ff195ccf640 1 Processor -- start 2023-10-16T15:55:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.034+0000 7ff195ccf640 1 -- start start 2023-10-16T15:55:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.035+0000 7ff195ccf640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1900ce390 0x7ff1900c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.035+0000 7ff195ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff1900a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.035+0000 7ff195ccf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff19015ae80 0x7ff1900c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.035+0000 7ff195ccf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900c8db0 con 0x7ff19015aaa0 2023-10-16T15:55:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.035+0000 7ff195ccf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190158770 con 0x7ff19015ae80 2023-10-16T15:55:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.035+0000 7ff195ccf640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900c7310 con 0x7ff1900ce390 2023-10-16T15:55:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff1900a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff1900a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54230/0 (socket says 172.21.15.29:54230) 2023-10-16T15:55:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 -- 172.21.15.29:0/1172218017 learned_addr learned my addr 172.21.15.29:0/1172218017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff18f7fe640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1900ce390 0x7ff1900c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff18ffff640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff19015ae80 0x7ff1900c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 -- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1900ce390 msgr2=0x7ff1900c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1900ce390 0x7ff1900c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 -- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff19015ae80 msgr2=0x7ff1900c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff19015ae80 0x7ff1900c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 -- 172.21.15.29:0/1172218017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1800094d0 con 0x7ff19015aaa0 2023-10-16T15:55:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.036+0000 7ff187fff640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff1900a84f0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7ff17800b3c0 tx=0x7ff17800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.037+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178011830 con 0x7ff19015aaa0 2023-10-16T15:55:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.037+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff178011990 con 0x7ff19015aaa0 2023-10-16T15:55:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.037+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178011d60 con 0x7ff19015aaa0 2023-10-16T15:55:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.037+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1900c75f0 con 0x7ff19015aaa0 2023-10-16T15:55:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.037+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1900c5fa0 con 0x7ff19015aaa0 2023-10-16T15:55:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.038+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19004e160 con 0x7ff19015aaa0 2023-10-16T15:55:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.039+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7ff17801b460 con 0x7ff19015aaa0 2023-10-16T15:55:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.040+0000 7ff18cff9640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff17006b020 0x7ff17006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.040+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7ff178089ae0 con 0x7ff19015aaa0 2023-10-16T15:55:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.041+0000 7ff18f7fe640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff17006b020 0x7ff17006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.042+0000 7ff18f7fe640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff17006b020 0x7ff17006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff180036000 tx=0x7ff18003c040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.045+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1780544b0 con 0x7ff19015aaa0 2023-10-16T15:55:25.160 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:25.157+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.247+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1900cdc60 con 0x7ff19015aaa0 2023-10-16T15:55:25.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.250+0000 7ff18cff9640 1 -- 172.21.15.29:0/1172218017 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+17975 (secure 0 0 0) 0x7ff178057ad0 con 0x7ff19015aaa0 2023-10-16T15:55:25.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:25.253 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":255,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:20.854182+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T15:54:51.157450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T15:54:51.864591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.29:6835/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.29:6833/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:25.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.253+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff17006b020 msgr2=0x7ff17006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.253+0000 7ff195ccf640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff17006b020 0x7ff17006d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff180036000 tx=0x7ff18003c040 comp rx=0 tx=0).stop 2023-10-16T15:55:25.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.253+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 msgr2=0x7ff1900a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.253+0000 7ff195ccf640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff1900a84f0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7ff17800b3c0 tx=0x7ff17800b880 comp rx=0 tx=0).stop 2023-10-16T15:55:25.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 shutdown_connections 2023-10-16T15:55:25.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff17006b020 0x7ff17006d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19015aaa0 0x7ff1900a84f0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff19015ae80 0x7ff1900c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 --2- 172.21.15.29:0/1172218017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff1900ce390 0x7ff1900c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 >> 172.21.15.29:0/1172218017 conn(0x7ff1900cb410 msgr2=0x7ff1900b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 shutdown_connections 2023-10-16T15:55:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.254+0000 7ff195ccf640 1 -- 172.21.15.29:0/1172218017 wait complete. 2023-10-16T15:55:25.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T15:55:25.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.509+0000 7fe5f7c87640 1 Processor -- start 2023-10-16T15:55:25.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.510+0000 7fe5f7c87640 1 -- start start 2023-10-16T15:55:25.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.511+0000 7fe5f7c87640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe5f006d570 0x7fe5f006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.511+0000 7fe5f7c87640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f015b1f0 con 0x7fe5f00caca0 2023-10-16T15:55:25.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.511+0000 7fe5f7c87640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f006bb70 con 0x7fe5f015ae10 2023-10-16T15:55:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.511+0000 7fe5f7c87640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f006f650 con 0x7fe5f006d570 2023-10-16T15:55:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.511+0000 7fe5f6224640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe5f00caca0 0x7fe5f006c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54734/0 (socket says 172.21.15.29:54734) 2023-10-16T15:55:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.511+0000 7fe5f6224640 1 -- 172.21.15.29:0/571202317 learned_addr learned my addr 172.21.15.29:0/571202317 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f5222640 1 --2- 172.21.15.29:0/571202317 >> v2:172.21.15.170:3300/0 conn(0x7fe5f006d570 0x7fe5f006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745396762 0 0) 0x7fe5e8001490 con 0x7fe5f00caca0 2023-10-16T15:55:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5dc003520 con 0x7fe5f00caca0 2023-10-16T15:55:25.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2406247597 0 0) 0x7fe5dc003520 con 0x7fe5f00caca0 2023-10-16T15:55:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 >> v2:172.21.15.170:3300/0 conn(0x7fe5f006d570 msgr2=0x7fe5f006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 --2- 172.21.15.29:0/571202317 >> v2:172.21.15.170:3300/0 conn(0x7fe5f006d570 0x7fe5f006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 >> v1:172.21.15.170:6789/0 conn(0x7fe5f015ae10 legacy=0x7fe5f00ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f00b4190 con 0x7fe5f00caca0 2023-10-16T15:55:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.512+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fe5e8001b80 con 0x7fe5f00caca0 2023-10-16T15:55:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 >> v1:172.21.15.29:6789/0 conn(0x7fe5f00caca0 legacy=0x7fe5f006c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5dc003970 0x7fe5dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc0047c0 0x7fe5dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5dc005180 0x7fe5dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc00bff0 con 0x7fe5dc005180 2023-10-16T15:55:25.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc00c160 con 0x7fe5dc0047c0 2023-10-16T15:55:25.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc00c320 con 0x7fe5dc003970 2023-10-16T15:55:25.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f6224640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5dc005180 0x7fe5dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f5222640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc0047c0 0x7fe5dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f5a23640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5dc003970 0x7fe5dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.513+0000 7fe5f6224640 1 -- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5dc003970 msgr2=0x7fe5dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.514+0000 7fe5f6224640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5dc003970 0x7fe5dc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.514+0000 7fe5f6224640 1 -- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc0047c0 msgr2=0x7fe5dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.514+0000 7fe5f6224640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc0047c0 0x7fe5dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.514+0000 7fe5f6224640 1 -- 172.21.15.29:0/571202317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc003520 con 0x7fe5dc005180 2023-10-16T15:55:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.514+0000 7fe5f5222640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc0047c0 0x7fe5dc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.514+0000 7fe5f6224640 1 -- 172.21.15.29:0/571202317 reap_dead start 2023-10-16T15:55:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.515+0000 7fe5f6224640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5dc005180 0x7fe5dc00b980 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800a800 tx=0x7fe5dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01ae4e0eae8b1fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.515+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8034630 con 0x7fe5dc005180 2023-10-16T15:55:25.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.515+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e802f070 con 0x7fe5dc005180 2023-10-16T15:55:25.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.515+0000 7fe5f4a21640 1 -- 172.21.15.29:0/571202317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80349f0 con 0x7fe5dc005180 2023-10-16T15:55:25.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 -- 172.21.15.29:0/571202317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5dc005180 msgr2=0x7fe5dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5dc005180 0x7fe5dc00b980 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800a800 tx=0x7fe5dc0057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 -- 172.21.15.29:0/571202317 shutdown_connections 2023-10-16T15:55:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/571202317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5dc005180 0x7fe5dc00b980 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 -- 172.21.15.29:0/571202317 >> 172.21.15.29:0/571202317 conn(0x7fe5f00ce840 msgr2=0x7fe5f00cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:25.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 -- 172.21.15.29:0/571202317 shutdown_connections 2023-10-16T15:55:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.516+0000 7fe5f7c87640 1 -- 172.21.15.29:0/571202317 wait complete. 2023-10-16T15:55:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.518+0000 7fe5f7c87640 1 Processor -- start 2023-10-16T15:55:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.518+0000 7fe5f7c87640 1 -- start start 2023-10-16T15:55:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f7c87640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc005180 0x7fe5f00c5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f7c87640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5f00c44f0 0x7fe5f0158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f7c87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 0x7fe5f00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f7c87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0159450 con 0x7fe5f00cd860 2023-10-16T15:55:25.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f7c87640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f01f29b0 con 0x7fe5dc005180 2023-10-16T15:55:25.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f7c87640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f01f2c90 con 0x7fe5f00c44f0 2023-10-16T15:55:25.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f6224640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 0x7fe5f00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f6224640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 0x7fe5f00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54258/0 (socket says 172.21.15.29:54258) 2023-10-16T15:55:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.519+0000 7fe5f6224640 1 -- 172.21.15.29:0/4289186048 learned_addr learned my addr 172.21.15.29:0/4289186048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f5222640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5f00c44f0 0x7fe5f0158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f5a23640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc005180 0x7fe5f00c5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f6224640 1 -- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5f00c44f0 msgr2=0x7fe5f0158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f6224640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5f00c44f0 0x7fe5f0158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f6224640 1 -- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc005180 msgr2=0x7fe5f00c5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f6224640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc005180 0x7fe5f00c5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f6224640 1 -- 172.21.15.29:0/4289186048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f01f2f70 con 0x7fe5f00cd860 2023-10-16T15:55:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f5222640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5f00c44f0 0x7fe5f0158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f6224640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 0x7fe5f00ac890 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800a0b0 tx=0x7fe5e8001f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8004d60 con 0x7fe5f00cd860 2023-10-16T15:55:25.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e802ce90 con 0x7fe5f00cd860 2023-10-16T15:55:25.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.520+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f01f3250 con 0x7fe5f00cd860 2023-10-16T15:55:25.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.521+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8034670 con 0x7fe5f00cd860 2023-10-16T15:55:25.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.521+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f01f25f0 con 0x7fe5f00cd860 2023-10-16T15:55:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.522+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f00b2620 con 0x7fe5f00cd860 2023-10-16T15:55:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.523+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fe5e8031030 con 0x7fe5f00cd860 2023-10-16T15:55:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.524+0000 7fe5e67fc640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe5d406b050 0x7fe5d406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.524+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6695+0+0 (secure 0 0 0) 0x7fe5e80ab7f0 con 0x7fe5f00cd860 2023-10-16T15:55:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.524+0000 7fe5f5a23640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe5d406b050 0x7fe5d406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.526+0000 7fe5f5a23640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe5d406b050 0x7fe5d406d510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec007fd0 tx=0x7fe5ec009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.528+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe5e80761e0 con 0x7fe5f00cd860 2023-10-16T15:55:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.761+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fe5f015b1f0 con 0x7fe5f00cd860 2023-10-16T15:55:25.764 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:55:25.762+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:55:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.766+0000 7fe5e67fc640 1 -- 172.21.15.29:0/4289186048 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v133) v1 ==== 167+0+0 (secure 0 0 0) 0x7fe5f015b1f0 con 0x7fe5f00cd860 2023-10-16T15:55:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:55:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.769+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe5d406b050 msgr2=0x7fe5d406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.769+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe5d406b050 0x7fe5d406d510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec007fd0 tx=0x7fe5ec009040 comp rx=0 tx=0).stop 2023-10-16T15:55:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.769+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 msgr2=0x7fe5f00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.769+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 0x7fe5f00ac890 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800a0b0 tx=0x7fe5e8001f10 comp rx=0 tx=0).stop 2023-10-16T15:55:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 shutdown_connections 2023-10-16T15:55:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe5d406b050 0x7fe5d406d510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5f00cd860 0x7fe5f00ac890 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe5f00c44f0 0x7fe5f0158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 --2- 172.21.15.29:0/4289186048 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5dc005180 0x7fe5f00c5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 >> 172.21.15.29:0/4289186048 conn(0x7fe5f00ce840 msgr2=0x7fe5f015d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 shutdown_connections 2023-10-16T15:55:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.770+0000 7fe5f7c87640 1 -- 172.21.15.29:0/4289186048 wait complete. 2023-10-16T15:55:25.786 INFO:tasks.ceph.mds.c.smithi029.stderr: -1718> 2023-10-16T15:54:58.271+0000 7fa11ade4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:55:25.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T15:55:25.842 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:55:25.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.990+0000 7f037b4d7640 1 Processor -- start 2023-10-16T15:55:25.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.990+0000 7f037b4d7640 1 -- start start 2023-10-16T15:55:25.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.991+0000 7f037b4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f037415ad70 0x7f0374069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f037b4d7640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f03740c9ca0 0x7f037406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f037b4d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03740ca3b0 0x7f03740ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:25.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f037b4d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03740cad80 con 0x7f03740ca3b0 2023-10-16T15:55:25.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f037b4d7640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03740cb030 con 0x7f03740c9ca0 2023-10-16T15:55:25.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f037b4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03740c91c0 con 0x7f037415ad70 2023-10-16T15:55:25.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0379a74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03740ca3b0 0x7f03740ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0379a74640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03740ca3b0 0x7f03740ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54282/0 (socket says 172.21.15.29:54282) 2023-10-16T15:55:25.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0379a74640 1 -- 172.21.15.29:0/901567294 learned_addr learned my addr 172.21.15.29:0/901567294 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:25.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0378a72640 1 --2- 172.21.15.29:0/901567294 >> v2:172.21.15.170:3300/0 conn(0x7f03740c9ca0 0x7f037406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:25.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0379a74640 1 -- 172.21.15.29:0/901567294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f037415ad70 msgr2=0x7f0374069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0379a74640 1 --2- 172.21.15.29:0/901567294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f037415ad70 0x7f0374069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.992+0000 7f0379a74640 1 -- 172.21.15.29:0/901567294 >> v2:172.21.15.170:3300/0 conn(0x7f03740c9ca0 msgr2=0x7f037406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:25.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.993+0000 7f0379a74640 1 --2- 172.21.15.29:0/901567294 >> v2:172.21.15.170:3300/0 conn(0x7f03740c9ca0 0x7f037406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:25.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.993+0000 7f0379a74640 1 -- 172.21.15.29:0/901567294 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740c94a0 con 0x7f03740ca3b0 2023-10-16T15:55:25.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.993+0000 7f0378a72640 1 --2- 172.21.15.29:0/901567294 >> v2:172.21.15.170:3300/0 conn(0x7f03740c9ca0 0x7f037406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:25.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.994+0000 7f0379a74640 1 --2- 172.21.15.29:0/901567294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03740ca3b0 0x7f03740ca840 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f037406af00 tx=0x7f036c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d360df292c35e629 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:25.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.994+0000 7f036bfff640 1 -- 172.21.15.29:0/901567294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c02ca70 con 0x7f03740ca3b0 2023-10-16T15:55:25.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.994+0000 7f036bfff640 1 -- 172.21.15.29:0/901567294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c02cbd0 con 0x7f03740ca3b0 2023-10-16T15:55:25.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.994+0000 7f036bfff640 1 -- 172.21.15.29:0/901567294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c0343f0 con 0x7f03740ca3b0 2023-10-16T15:55:26.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 -- 172.21.15.29:0/901567294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03740ca3b0 msgr2=0x7f03740ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:26.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 --2- 172.21.15.29:0/901567294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03740ca3b0 0x7f03740ca840 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f037406af00 tx=0x7f036c02bed0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 -- 172.21.15.29:0/901567294 shutdown_connections 2023-10-16T15:55:26.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 --2- 172.21.15.29:0/901567294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03740ca3b0 0x7f03740ca840 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 --2- 172.21.15.29:0/901567294 >> v2:172.21.15.170:3300/0 conn(0x7f03740c9ca0 0x7f037406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 --2- 172.21.15.29:0/901567294 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f037415ad70 0x7f0374069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 -- 172.21.15.29:0/901567294 >> 172.21.15.29:0/901567294 conn(0x7f03740ce9b0 msgr2=0x7f03740cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 -- 172.21.15.29:0/901567294 shutdown_connections 2023-10-16T15:55:26.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.995+0000 7f037b4d7640 1 -- 172.21.15.29:0/901567294 wait complete. 2023-10-16T15:55:26.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.997+0000 7f037b4d7640 1 Processor -- start 2023-10-16T15:55:26.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.997+0000 7f037b4d7640 1 -- start start 2023-10-16T15:55:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f037b4d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03740c9ca0 0x7f03740b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f037b4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03740ca3b0 0x7f0374158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f037b4d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 0x7f03740adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f037b4d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740b3ab0 con 0x7f037415ad70 2023-10-16T15:55:26.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f037b4d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740b30e0 con 0x7f03740c9ca0 2023-10-16T15:55:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f037b4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03740b27d0 con 0x7f03740ca3b0 2023-10-16T15:55:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f0379a74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 0x7f03740adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f0379a74640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 0x7f03740adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54286/0 (socket says 172.21.15.29:54286) 2023-10-16T15:55:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.998+0000 7f0379a74640 1 -- 172.21.15.29:0/1804970879 learned_addr learned my addr 172.21.15.29:0/1804970879 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:26.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379273640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03740c9ca0 0x7f03740b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379a74640 1 -- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03740ca3b0 msgr2=0x7f0374158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379a74640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03740ca3b0 0x7f0374158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379a74640 1 -- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03740c9ca0 msgr2=0x7f03740b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379a74640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03740c9ca0 0x7f03740b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379a74640 1 -- 172.21.15.29:0/1804970879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c009b30 con 0x7f037415ad70 2023-10-16T15:55:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379273640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03740c9ca0 0x7f03740b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0379a74640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 0x7f03740adcc0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f036c036040 tx=0x7f036c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c03e450 con 0x7f037415ad70 2023-10-16T15:55:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c03e5d0 con 0x7f037415ad70 2023-10-16T15:55:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c03e980 con 0x7f037415ad70 2023-10-16T15:55:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:25.999+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740b2a90 con 0x7f037415ad70 2023-10-16T15:55:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.000+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03741593c0 con 0x7f037415ad70 2023-10-16T15:55:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.001+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037404e160 con 0x7f037415ad70 2023-10-16T15:55:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.002+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f036c03eae0 con 0x7f037415ad70 2023-10-16T15:55:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.003+0000 7f0369ffb640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f035806b020 0x7f035806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.003+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f036c0accf0 con 0x7f037415ad70 2023-10-16T15:55:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.003+0000 7f0379273640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f035806b020 0x7f035806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:26.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.004+0000 7f0379273640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f035806b020 0x7f035806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03740cac50 tx=0x7f0370008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:26.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.007+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f036c077650 con 0x7f037415ad70 2023-10-16T15:55:26.166 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:26.163+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:26.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.243+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f03740b3220 con 0x7f037415ad70 2023-10-16T15:55:26.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.769+0000 7f0369ffb640 1 -- 172.21.15.29:0/1804970879 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v134) v1 ==== 107+0+0 (secure 0 0 0) 0x7f036c0ac020 con 0x7f037415ad70 2023-10-16T15:55:26.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f035806b020 msgr2=0x7f035806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:26.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f035806b020 0x7f035806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03740cac50 tx=0x7f0370008040 comp rx=0 tx=0).stop 2023-10-16T15:55:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 msgr2=0x7f03740adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 0x7f03740adcc0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f036c036040 tx=0x7f036c004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 shutdown_connections 2023-10-16T15:55:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f035806b020 0x7f035806d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f037415ad70 0x7f03740adcc0 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f03740ca3b0 0x7f0374158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 --2- 172.21.15.29:0/1804970879 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f03740c9ca0 0x7f03740b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:26.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.774+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 >> 172.21.15.29:0/1804970879 conn(0x7f03740ce9b0 msgr2=0x7f037415b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:26.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.775+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 shutdown_connections 2023-10-16T15:55:26.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:26.775+0000 7f037b4d7640 1 -- 172.21.15.29:0/1804970879 wait complete. 2023-10-16T15:55:26.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T15:55:27.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.025+0000 7f1607bd0640 1 Processor -- start 2023-10-16T15:55:27.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.025+0000 7f1607bd0640 1 -- start start 2023-10-16T15:55:27.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f1607bd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16000cac20 0x7f16000b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:27.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f1607bd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160015b310 con 0x7f16000b0360 2023-10-16T15:55:27.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f1607bd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160006f640 con 0x7f160015ac10 2023-10-16T15:55:27.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f1607bd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160015f800 con 0x7f16000cac20 2023-10-16T15:55:27.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f160616d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16000cac20 0x7f16000b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:27.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f160616d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16000cac20 0x7f16000b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54298/0 (socket says 172.21.15.29:54298) 2023-10-16T15:55:27.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.027+0000 7f160616d640 1 -- 172.21.15.29:0/2146097626 learned_addr learned my addr 172.21.15.29:0/2146097626 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:27.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.028+0000 7f160616d640 1 -- 172.21.15.29:0/2146097626 >> v1:172.21.15.170:6789/0 conn(0x7f160015ac10 legacy=0x7f160006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.028+0000 7f160616d640 1 -- 172.21.15.29:0/2146097626 >> v1:172.21.15.29:6789/0 conn(0x7f16000b0360 legacy=0x7f16000ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.028+0000 7f160616d640 1 -- 172.21.15.29:0/2146097626 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160015b5b0 con 0x7f16000cac20 2023-10-16T15:55:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.029+0000 7f160616d640 1 --2- 172.21.15.29:0/2146097626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f16000b07f0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e50 tx=0x7f15f802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f79d5fcf1e6598 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.029+0000 7f160496a640 1 -- 172.21.15.29:0/2146097626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f802cb10 con 0x7f16000cac20 2023-10-16T15:55:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.029+0000 7f160496a640 1 -- 172.21.15.29:0/2146097626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f802cc70 con 0x7f16000cac20 2023-10-16T15:55:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.029+0000 7f160496a640 1 -- 172.21.15.29:0/2146097626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80343f0 con 0x7f16000cac20 2023-10-16T15:55:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 -- 172.21.15.29:0/2146097626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 msgr2=0x7f16000b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 --2- 172.21.15.29:0/2146097626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f16000b07f0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e50 tx=0x7f15f802bf70 comp rx=0 tx=0).stop 2023-10-16T15:55:27.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 -- 172.21.15.29:0/2146097626 shutdown_connections 2023-10-16T15:55:27.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 --2- 172.21.15.29:0/2146097626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f16000b07f0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 -- 172.21.15.29:0/2146097626 >> 172.21.15.29:0/2146097626 conn(0x7f16000cb360 msgr2=0x7f16000cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:27.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 -- 172.21.15.29:0/2146097626 shutdown_connections 2023-10-16T15:55:27.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.030+0000 7f1607bd0640 1 -- 172.21.15.29:0/2146097626 wait complete. 2023-10-16T15:55:27.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.033+0000 7f1607bd0640 1 Processor -- start 2023-10-16T15:55:27.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.033+0000 7f1607bd0640 1 -- start start 2023-10-16T15:55:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.034+0000 7f1607bd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16000b0360 0x7f16000cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.034+0000 7f1607bd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f1600158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:27.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.034+0000 7f1607bd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f160015ac10 0x7f16000aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.034+0000 7f1607bd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000c8070 con 0x7f16000cac20 2023-10-16T15:55:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.034+0000 7f1607bd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000cf1f0 con 0x7f160015ac10 2023-10-16T15:55:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.034+0000 7f1607bd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16001f2d90 con 0x7f16000b0360 2023-10-16T15:55:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f1600158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f1600158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54302/0 (socket says 172.21.15.29:54302) 2023-10-16T15:55:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 -- 172.21.15.29:0/1203439400 learned_addr learned my addr 172.21.15.29:0/1203439400 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160616d640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f160015ac10 0x7f16000aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160596c640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16000b0360 0x7f16000cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 -- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16000b0360 msgr2=0x7f16000cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16000b0360 0x7f16000cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 -- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f160015ac10 msgr2=0x7f16000aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f160015ac10 0x7f16000aee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 -- 172.21.15.29:0/1203439400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8009b30 con 0x7f16000cac20 2023-10-16T15:55:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160616d640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f160015ac10 0x7f16000aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:27.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160596c640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16000b0360 0x7f16000cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:27.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.035+0000 7f160516b640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f1600158b60 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f15f000c400 tx=0x7f15f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.036+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f00052d0 con 0x7f16000cac20 2023-10-16T15:55:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.036+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0005430 con 0x7f16000cac20 2023-10-16T15:55:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.036+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0003830 con 0x7f16000cac20 2023-10-16T15:55:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.036+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16001f3070 con 0x7f16000cac20 2023-10-16T15:55:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.036+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16001f3630 con 0x7f16000cac20 2023-10-16T15:55:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.037+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160004e160 con 0x7f16000cac20 2023-10-16T15:55:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.038+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f15f0003990 con 0x7f16000cac20 2023-10-16T15:55:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.039+0000 7f15f67fc640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f15e406b020 0x7f15e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.040+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f15f00892f0 con 0x7f16000cac20 2023-10-16T15:55:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.040+0000 7f160596c640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f15e406b020 0x7f15e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.041+0000 7f160596c640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f15e406b020 0x7f15e406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f15fc000f60 tx=0x7f15fc009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.044+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f15f0053ce0 con 0x7f16000cac20 2023-10-16T15:55:27.192 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:27.190+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.243+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f160006c9c0 con 0x7f16000cac20 2023-10-16T15:55:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.780+0000 7f15f67fc640 1 -- 172.21.15.29:0/1203439400 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v257) v1 ==== 190+0+0 (secure 0 0 0) 0x7f15f0057300 con 0x7f16000cac20 2023-10-16T15:55:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' removed 2023-10-16T15:55:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f15e406b020 msgr2=0x7f15e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f15e406b020 0x7f15e406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f15fc000f60 tx=0x7f15fc009040 comp rx=0 tx=0).stop 2023-10-16T15:55:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 msgr2=0x7f1600158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f1600158b60 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f15f000c400 tx=0x7f15f000c8c0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 shutdown_connections 2023-10-16T15:55:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f15e406b020 0x7f15e406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16000cac20 0x7f1600158b60 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f160015ac10 0x7f16000aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 --2- 172.21.15.29:0/1203439400 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f16000b0360 0x7f16000cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.784+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 >> 172.21.15.29:0/1203439400 conn(0x7f16000cb360 msgr2=0x7f160006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.785+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 shutdown_connections 2023-10-16T15:55:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:27.785+0000 7f1607bd0640 1 -- 172.21.15.29:0/1203439400 wait complete. 2023-10-16T15:55:27.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T15:55:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.050+0000 7fd7ccf93640 1 Processor -- start 2023-10-16T15:55:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.050+0000 7fd7ccf93640 1 -- start start 2023-10-16T15:55:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7ccf93640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd7c806c040 0x7fd7c806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7ccf93640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c80ad970 0x7fd7c815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7ccf93640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7c815ebc0 0x7fd7c815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7ccf93640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c806c960 con 0x7fd7c815ebc0 2023-10-16T15:55:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7ccf93640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c815b9d0 con 0x7fd7c806c040 2023-10-16T15:55:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7ccf93640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c815bcb0 con 0x7fd7c80ad970 2023-10-16T15:55:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.052+0000 7fd7c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7c815ebc0 0x7fd7c815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7c815ebc0 0x7fd7c815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54304/0 (socket says 172.21.15.29:54304) 2023-10-16T15:55:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 -- 172.21.15.29:0/3444486598 learned_addr learned my addr 172.21.15.29:0/3444486598 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c77fe640 1 --2- 172.21.15.29:0/3444486598 >> v2:172.21.15.170:3300/0 conn(0x7fd7c806c040 0x7fd7c806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c6ffd640 1 --2- 172.21.15.29:0/3444486598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c80ad970 0x7fd7c815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 -- 172.21.15.29:0/3444486598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c80ad970 msgr2=0x7fd7c815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 --2- 172.21.15.29:0/3444486598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c80ad970 0x7fd7c815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 -- 172.21.15.29:0/3444486598 >> v2:172.21.15.170:3300/0 conn(0x7fd7c806c040 msgr2=0x7fd7c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 --2- 172.21.15.29:0/3444486598 >> v2:172.21.15.170:3300/0 conn(0x7fd7c806c040 0x7fd7c806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.053+0000 7fd7c7fff640 1 -- 172.21.15.29:0/3444486598 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c815bf90 con 0x7fd7c815ebc0 2023-10-16T15:55:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.054+0000 7fd7c7fff640 1 --2- 172.21.15.29:0/3444486598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c815ebc0 0x7fd7c815f050 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009e50 tx=0x7fd7c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e939c16af8af6b4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7c67fc640 1 -- 172.21.15.29:0/3444486598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c002ca70 con 0x7fd7c815ebc0 2023-10-16T15:55:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7c67fc640 1 -- 172.21.15.29:0/3444486598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c002cbd0 con 0x7fd7c815ebc0 2023-10-16T15:55:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7c67fc640 1 -- 172.21.15.29:0/3444486598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00343f0 con 0x7fd7c815ebc0 2023-10-16T15:55:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3444486598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c815ebc0 msgr2=0x7fd7c815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3444486598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c815ebc0 0x7fd7c815f050 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009e50 tx=0x7fd7c002bed0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3444486598 shutdown_connections 2023-10-16T15:55:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3444486598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c815ebc0 0x7fd7c815f050 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3444486598 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c80ad970 0x7fd7c815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3444486598 >> v2:172.21.15.170:3300/0 conn(0x7fd7c806c040 0x7fd7c806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3444486598 >> 172.21.15.29:0/3444486598 conn(0x7fd7c80ce6a0 msgr2=0x7fd7c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3444486598 shutdown_connections 2023-10-16T15:55:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.055+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3444486598 wait complete. 2023-10-16T15:55:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.057+0000 7fd7ccf93640 1 Processor -- start 2023-10-16T15:55:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.057+0000 7fd7ccf93640 1 -- start start 2023-10-16T15:55:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.058+0000 7fd7ccf93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 0x7fd7c8156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.058+0000 7fd7ccf93640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7c80ad970 0x7fd7c80c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.058+0000 7fd7ccf93640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c815ebc0 0x7fd7c80afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.058+0000 7fd7ccf93640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80acfd0 con 0x7fd7c806c040 2023-10-16T15:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.058+0000 7fd7ccf93640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80ad270 con 0x7fd7c80ad970 2023-10-16T15:55:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.058+0000 7fd7ccf93640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80c7350 con 0x7fd7c815ebc0 2023-10-16T15:55:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 0x7fd7c8156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 0x7fd7c8156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54314/0 (socket says 172.21.15.29:54314) 2023-10-16T15:55:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 -- 172.21.15.29:0/3411446873 learned_addr learned my addr 172.21.15.29:0/3411446873 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c6ffd640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7c80ad970 0x7fd7c80c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c7fff640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c815ebc0 0x7fd7c80afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 -- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c815ebc0 msgr2=0x7fd7c80afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c815ebc0 0x7fd7c80afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 -- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7c80ad970 msgr2=0x7fd7c80c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7c80ad970 0x7fd7c80c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c77fe640 1 -- 172.21.15.29:0/3411446873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0009b30 con 0x7fd7c806c040 2023-10-16T15:55:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.059+0000 7fd7c7fff640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c815ebc0 0x7fd7c80afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.060+0000 7fd7c77fe640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 0x7fd7c8156170 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800d800 tx=0x7fd7b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.060+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80096a0 con 0x7fd7c806c040 2023-10-16T15:55:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.060+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b8009800 con 0x7fd7c806c040 2023-10-16T15:55:28.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.060+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8009990 con 0x7fd7c806c040 2023-10-16T15:55:28.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.060+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c80c7630 con 0x7fd7c806c040 2023-10-16T15:55:28.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.060+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c8157580 con 0x7fd7c806c040 2023-10-16T15:55:28.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.061+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c806b2f0 con 0x7fd7c806c040 2023-10-16T15:55:28.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.062+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fd7b8009af0 con 0x7fd7c806c040 2023-10-16T15:55:28.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.063+0000 7fd7affff640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd7b406b070 0x7fd7b406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:28.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.063+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fd7b808a0b0 con 0x7fd7c806c040 2023-10-16T15:55:28.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.064+0000 7fd7c6ffd640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd7b406b070 0x7fd7b406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:28.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.065+0000 7fd7c6ffd640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd7b406b070 0x7fd7b406d530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc006fd0 tx=0x7fd7bc008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:28.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.069+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7b8054c40 con 0x7fd7c806c040 2023-10-16T15:55:28.217 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:28.214+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:28.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.270+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd7c80c77c0 con 0x7fd7c806c040 2023-10-16T15:55:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.792+0000 7fd7affff640 1 -- 172.21.15.29:0/3411446873 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v258) v1 ==== 178+0+0 (secure 0 0 0) 0x7fd7b800b3a0 con 0x7fd7c806c040 2023-10-16T15:55:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' removed 2023-10-16T15:55:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd7b406b070 msgr2=0x7fd7b406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd7b406b070 0x7fd7b406d530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc006fd0 tx=0x7fd7bc008040 comp rx=0 tx=0).stop 2023-10-16T15:55:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 msgr2=0x7fd7c8156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:28.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 0x7fd7c8156170 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800d800 tx=0x7fd7b800dcc0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 shutdown_connections 2023-10-16T15:55:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd7b406b070 0x7fd7b406d530 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd7c815ebc0 0x7fd7c80afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7c80ad970 0x7fd7c80c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 --2- 172.21.15.29:0/3411446873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c806c040 0x7fd7c8156170 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:28.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.796+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 >> 172.21.15.29:0/3411446873 conn(0x7fd7c80ce6a0 msgr2=0x7fd7c80c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:28.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.797+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 shutdown_connections 2023-10-16T15:55:28.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:28.798+0000 7fd7ccf93640 1 -- 172.21.15.29:0/3411446873 wait complete. 2023-10-16T15:55:28.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:55:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.043+0000 7f87ed17c640 1 Processor -- start 2023-10-16T15:55:29.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.043+0000 7f87ed17c640 1 -- start start 2023-10-16T15:55:29.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.044+0000 7f87ed17c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f87e80caac0 0x7f87e80ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.044+0000 7f87ed17c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e80cb030 con 0x7f87e815ad70 2023-10-16T15:55:29.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.044+0000 7f87ed17c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e80c91c0 con 0x7f87e806a4b0 2023-10-16T15:55:29.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.044+0000 7f87ed17c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e80c94a0 con 0x7f87e80caac0 2023-10-16T15:55:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f87e815ad70 0x7f87e8069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54760/0 (socket says 172.21.15.29:54760) 2023-10-16T15:55:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e6d9d640 1 -- 172.21.15.29:0/4019770803 learned_addr learned my addr 172.21.15.29:0/4019770803 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e759e640 1 --2- 172.21.15.29:0/4019770803 >> v2:172.21.15.170:3300/0 conn(0x7f87e80caac0 0x7f87e80ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120501046 0 0) 0x7f87e80cb030 con 0x7f87e815ad70 2023-10-16T15:55:29.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d0003520 con 0x7f87e815ad70 2023-10-16T15:55:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1336680529 0 0) 0x7f87d0003520 con 0x7f87e815ad70 2023-10-16T15:55:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 >> v2:172.21.15.170:3300/0 conn(0x7f87e80caac0 msgr2=0x7f87e80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 --2- 172.21.15.29:0/4019770803 >> v2:172.21.15.170:3300/0 conn(0x7f87e80caac0 0x7f87e80ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 >> v1:172.21.15.170:6789/0 conn(0x7f87e806a4b0 legacy=0x7f87e80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80c9780 con 0x7f87e815ad70 2023-10-16T15:55:29.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.045+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f87d8001670 con 0x7f87e815ad70 2023-10-16T15:55:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 >> v1:172.21.15.29:6789/0 conn(0x7f87e815ad70 legacy=0x7f87e8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00039c0 0x7f87d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d00047c0 0x7f87d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80c94a0 con 0x7f87d00051d0 2023-10-16T15:55:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d000c100 con 0x7f87d00039c0 2023-10-16T15:55:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d000c2c0 con 0x7f87d00047c0 2023-10-16T15:55:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e759e640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e6d9d640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00039c0 0x7f87d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.046+0000 7f87e659c640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d00047c0 0x7f87d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.047+0000 7f87e759e640 1 -- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d00047c0 msgr2=0x7f87d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.047+0000 7f87e759e640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87d00047c0 0x7f87d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.047+0000 7f87e759e640 1 -- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00039c0 msgr2=0x7f87d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.047+0000 7f87e759e640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87d00039c0 0x7f87d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.047+0000 7f87e759e640 1 -- 172.21.15.29:0/4019770803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d0003520 con 0x7f87d00051d0 2023-10-16T15:55:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.048+0000 7f87e759e640 1 -- 172.21.15.29:0/4019770803 reap_dead start 2023-10-16T15:55:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.048+0000 7f87e759e640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87d000b980 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f87e0007330 tx=0x7f87d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296206791fa677b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.048+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e002ccd0 con 0x7f87d00051d0 2023-10-16T15:55:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.048+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e002ce30 con 0x7f87d00051d0 2023-10-16T15:55:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.048+0000 7f87e5d9b640 1 -- 172.21.15.29:0/4019770803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0003bd0 con 0x7f87d00051d0 2023-10-16T15:55:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 -- 172.21.15.29:0/4019770803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 msgr2=0x7f87d000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87d000b980 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f87e0007330 tx=0x7f87d00057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 -- 172.21.15.29:0/4019770803 shutdown_connections 2023-10-16T15:55:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 --2- 172.21.15.29:0/4019770803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87d000b980 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 -- 172.21.15.29:0/4019770803 >> 172.21.15.29:0/4019770803 conn(0x7f87e80ce9b0 msgr2=0x7f87e80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 -- 172.21.15.29:0/4019770803 shutdown_connections 2023-10-16T15:55:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.049+0000 7f87ed17c640 1 -- 172.21.15.29:0/4019770803 wait complete. 2023-10-16T15:55:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.051+0000 7f87ed17c640 1 Processor -- start 2023-10-16T15:55:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.051+0000 7f87ed17c640 1 -- start start 2023-10-16T15:55:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87ed17c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87e80ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87ed17c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87e80c88f0 0x7f87e8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87ed17c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87e80b2c20 0x7f87e815c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87ed17c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cdbd0 con 0x7f87d00051d0 2023-10-16T15:55:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87ed17c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cd030 con 0x7f87e80b2c20 2023-10-16T15:55:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87ed17c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cd2a0 con 0x7f87e80c88f0 2023-10-16T15:55:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87e80ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87e80ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54332/0 (socket says 172.21.15.29:54332) 2023-10-16T15:55:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.052+0000 7f87e6d9d640 1 -- 172.21.15.29:0/895334479 learned_addr learned my addr 172.21.15.29:0/895334479 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87e6d9d640 1 -- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87e80c88f0 msgr2=0x7f87e8159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87e6d9d640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87e80c88f0 0x7f87e8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87e6d9d640 1 -- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87e80b2c20 msgr2=0x7f87e815c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87e6d9d640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87e80b2c20 0x7f87e815c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87e6d9d640 1 -- 172.21.15.29:0/895334479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80cd580 con 0x7f87d00051d0 2023-10-16T15:55:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87e6d9d640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87e80ce060 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f87d800e7b0 tx=0x7f87d800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d800ee60 con 0x7f87d00051d0 2023-10-16T15:55:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d800c8d0 con 0x7f87d00051d0 2023-10-16T15:55:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e80cd860 con 0x7f87d00051d0 2023-10-16T15:55:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d800cc80 con 0x7f87d00051d0 2023-10-16T15:55:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.053+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e80b30b0 con 0x7f87d00051d0 2023-10-16T15:55:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.054+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e804e160 con 0x7f87d00051d0 2023-10-16T15:55:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.055+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f87d801d460 con 0x7f87d00051d0 2023-10-16T15:55:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.056+0000 7f87cf7fe640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f87d406b050 0x7f87d406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.057+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f87d8089a10 con 0x7f87d00051d0 2023-10-16T15:55:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.057+0000 7f87e659c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f87d406b050 0x7f87d406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.058+0000 7f87e659c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f87d406b050 0x7f87d406d510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87e80c8cd0 tx=0x7f87dc009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.061+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87d8054770 con 0x7f87d00051d0 2023-10-16T15:55:29.248 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:29.245+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.261+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f87e80708b0 con 0x7f87d00051d0 2023-10-16T15:55:29.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.267+0000 7f87cf7fe640 1 -- 172.21.15.29:0/895334479 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87d80014d0 con 0x7f87d00051d0 2023-10-16T15:55:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f87d406b050 msgr2=0x7f87d406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f87d406b050 0x7f87d406d510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87e80c8cd0 tx=0x7f87dc009040 comp rx=0 tx=0).stop 2023-10-16T15:55:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 msgr2=0x7f87e80ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87e80ce060 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f87d800e7b0 tx=0x7f87d800ec70 comp rx=0 tx=0).stop 2023-10-16T15:55:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 shutdown_connections 2023-10-16T15:55:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f87d406b050 0x7f87d406d510 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87e80b2c20 0x7f87e815c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87e80c88f0 0x7f87e8159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 --2- 172.21.15.29:0/895334479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87d00051d0 0x7f87e80ce060 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 >> 172.21.15.29:0/895334479 conn(0x7f87e80ce9b0 msgr2=0x7f87e8070da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 shutdown_connections 2023-10-16T15:55:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.270+0000 7f87ed17c640 1 -- 172.21.15.29:0/895334479 wait complete. 2023-10-16T15:55:29.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:55:29.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.512+0000 7fe2c66c2640 1 Processor -- start 2023-10-16T15:55:29.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.512+0000 7fe2c66c2640 1 -- start start 2023-10-16T15:55:29.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.513+0000 7fe2c66c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.513+0000 7fe2c66c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2c00ce390 0x7fe2c0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.513+0000 7fe2c66c2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00707e0 con 0x7fe2c015ae80 2023-10-16T15:55:29.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.513+0000 7fe2c66c2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c006b7a0 con 0x7fe2c00ce390 2023-10-16T15:55:29.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.513+0000 7fe2c66c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c015ef20 con 0x7fe2c015aaa0 2023-10-16T15:55:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2c00ce390 0x7fe2c0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe2c00ce390 0x7fe2c0070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54348/0 (socket says 172.21.15.29:54348) 2023-10-16T15:55:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/343678158 learned_addr learned my addr 172.21.15.29:0/343678158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2bffff640 1 --2- 172.21.15.29:0/343678158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/343678158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 msgr2=0x7fe2c015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 --2- 172.21.15.29:0/343678158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/343678158 >> v1:172.21.15.29:6789/0 conn(0x7fe2c015ae80 legacy=0x7fe2c00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/343678158 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c015f200 con 0x7fe2c00ce390 2023-10-16T15:55:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.514+0000 7fe2beffd640 1 -- 172.21.15.29:0/343678158 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036129601 0 0) 0x7fe2c00707e0 con 0x7fe2c015ae80 2023-10-16T15:55:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.515+0000 7fe2c4c5f640 1 --2- 172.21.15.29:0/343678158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c00ce390 0x7fe2c0070f50 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009e50 tx=0x7fe2b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d50a9633fff40fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2beffd640 1 -- 172.21.15.29:0/343678158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b802ca70 con 0x7fe2c00ce390 2023-10-16T15:55:29.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2beffd640 1 -- 172.21.15.29:0/343678158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b802cbd0 con 0x7fe2c00ce390 2023-10-16T15:55:29.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2beffd640 1 -- 172.21.15.29:0/343678158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80343f0 con 0x7fe2c00ce390 2023-10-16T15:55:29.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2c66c2640 1 -- 172.21.15.29:0/343678158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c00ce390 msgr2=0x7fe2c0070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/343678158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c00ce390 0x7fe2c0070f50 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009e50 tx=0x7fe2b802bed0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2c66c2640 1 -- 172.21.15.29:0/343678158 shutdown_connections 2023-10-16T15:55:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/343678158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c00ce390 0x7fe2c0070f50 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/343678158 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.516+0000 7fe2c66c2640 1 -- 172.21.15.29:0/343678158 >> 172.21.15.29:0/343678158 conn(0x7fe2c00cb410 msgr2=0x7fe2c00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:29.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.517+0000 7fe2c66c2640 1 -- 172.21.15.29:0/343678158 shutdown_connections 2023-10-16T15:55:29.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.517+0000 7fe2c66c2640 1 -- 172.21.15.29:0/343678158 wait complete. 2023-10-16T15:55:29.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.518+0000 7fe2c66c2640 1 Processor -- start 2023-10-16T15:55:29.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 -- start start 2023-10-16T15:55:29.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c00ce390 0x7fe2c014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 0x7fe2c00b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c014c110 con 0x7fe2c015ae80 2023-10-16T15:55:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c014c3f0 con 0x7fe2c00ce390 2023-10-16T15:55:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.519+0000 7fe2c66c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c014c6d0 con 0x7fe2c015aaa0 2023-10-16T15:55:29.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 0x7fe2c00b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 0x7fe2c00b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54358/0 (socket says 172.21.15.29:54358) 2023-10-16T15:55:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/1184079996 learned_addr learned my addr 172.21.15.29:0/1184079996 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2bffff640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c00ce390 0x7fe2c014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 msgr2=0x7fe2c014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c00ce390 msgr2=0x7fe2c014ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c00ce390 0x7fe2c014ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 -- 172.21.15.29:0/1184079996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b8009b30 con 0x7fe2c015ae80 2023-10-16T15:55:29.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.520+0000 7fe2c4c5f640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 0x7fe2c00b2150 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009e50 tx=0x7fe2b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.521+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8009510 con 0x7fe2c015ae80 2023-10-16T15:55:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.521+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b803e450 con 0x7fe2c015ae80 2023-10-16T15:55:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.521+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b803e800 con 0x7fe2c015ae80 2023-10-16T15:55:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.521+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c014c9b0 con 0x7fe2c015ae80 2023-10-16T15:55:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.521+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c00b37f0 con 0x7fe2c015ae80 2023-10-16T15:55:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.522+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c014ce70 con 0x7fe2c015ae80 2023-10-16T15:55:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.523+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fe2b803e960 con 0x7fe2c015ae80 2023-10-16T15:55:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.524+0000 7fe2bcff9640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2ac06b070 0x7fe2ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.524+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7fe2b80ac770 con 0x7fe2c015ae80 2023-10-16T15:55:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.525+0000 7fe2bffff640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2ac06b070 0x7fe2ac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.526+0000 7fe2bffff640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2ac06b070 0x7fe2ac06d530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0006fd0 tx=0x7fe2b0008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.528+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2b8077540 con 0x7fe2c015ae80 2023-10-16T15:55:29.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.732+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe2c004e160 con 0x7fe2c015ae80 2023-10-16T15:55:29.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.738+0000 7fe2bcff9640 1 -- 172.21.15.29:0/1184079996 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v86) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe2b807ab60 con 0x7fe2c015ae80 2023-10-16T15:55:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.740+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2ac06b070 msgr2=0x7fe2ac06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.740+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2ac06b070 0x7fe2ac06d530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0006fd0 tx=0x7fe2b0008040 comp rx=0 tx=0).stop 2023-10-16T15:55:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.740+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 msgr2=0x7fe2c00b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.740+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 0x7fe2c00b2150 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009e50 tx=0x7fe2b8004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 shutdown_connections 2023-10-16T15:55:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fe2ac06b070 0x7fe2ac06d530 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2c015ae80 0x7fe2c00b2150 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe2c015aaa0 0x7fe2c014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 --2- 172.21.15.29:0/1184079996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2c00ce390 0x7fe2c014ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 >> 172.21.15.29:0/1184079996 conn(0x7fe2c00cb410 msgr2=0x7fe2c00afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:29.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 shutdown_connections 2023-10-16T15:55:29.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.741+0000 7fe2c66c2640 1 -- 172.21.15.29:0/1184079996 wait complete. 2023-10-16T15:55:29.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:55:29.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.969+0000 7f3c93eb4640 1 Processor -- start 2023-10-16T15:55:29.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.970+0000 7f3c93eb4640 1 -- start start 2023-10-16T15:55:29.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.971+0000 7f3c93eb4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c8c06a4b0 0x7f3c8c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.971+0000 7f3c93eb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c0c9ca0 0x7f3c8c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.971+0000 7f3c93eb4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0cb030 con 0x7f3c8c15ad70 2023-10-16T15:55:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c93eb4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c0cae30 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c93eb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c0c91c0 con 0x7f3c8c0c9ca0 2023-10-16T15:55:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c9144f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c8c06a4b0 0x7f3c8c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c9144f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c8c06a4b0 0x7f3c8c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54368/0 (socket says 172.21.15.29:54368) 2023-10-16T15:55:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c9144f640 1 -- 172.21.15.29:0/2066698477 learned_addr learned my addr 172.21.15.29:0/2066698477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c92451640 1 --2- 172.21.15.29:0/2066698477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c0c9ca0 0x7f3c8c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c9144f640 1 -- 172.21.15.29:0/2066698477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c0c9ca0 msgr2=0x7f3c8c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.972+0000 7f3c9144f640 1 --2- 172.21.15.29:0/2066698477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c0c9ca0 0x7f3c8c0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.973+0000 7f3c9144f640 1 -- 172.21.15.29:0/2066698477 >> v1:172.21.15.170:6789/0 conn(0x7f3c8c15ad70 legacy=0x7f3c8c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.973+0000 7f3c9144f640 1 -- 172.21.15.29:0/2066698477 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c0c94a0 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.974+0000 7f3c9144f640 1 --2- 172.21.15.29:0/2066698477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0ca3b0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009870 tx=0x7f3c7c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ac9e635af525ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.974+0000 7f3c90c4e640 1 -- 172.21.15.29:0/2066698477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c031640 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.974+0000 7f3c90c4e640 1 -- 172.21.15.29:0/2066698477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c0317a0 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.974+0000 7f3c90c4e640 1 -- 172.21.15.29:0/2066698477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c031b30 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2066698477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 msgr2=0x7f3c8c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2066698477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0ca3b0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009870 tx=0x7f3c7c02b8f0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2066698477 shutdown_connections 2023-10-16T15:55:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2066698477 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c0c9ca0 0x7f3c8c0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2066698477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0ca3b0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2066698477 >> 172.21.15.29:0/2066698477 conn(0x7f3c8c0ce9b0 msgr2=0x7f3c8c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2066698477 shutdown_connections 2023-10-16T15:55:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.975+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2066698477 wait complete. 2023-10-16T15:55:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.977+0000 7f3c93eb4640 1 Processor -- start 2023-10-16T15:55:29.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.977+0000 7f3c93eb4640 1 -- start start 2023-10-16T15:55:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c93eb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c93eb4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c8c0c9ca0 0x7f3c8c159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c93eb4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c15ad70 0x7f3c8c157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c93eb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c1f1b90 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c93eb4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c159c60 con 0x7f3c8c0c9ca0 2023-10-16T15:55:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c93eb4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c1f2110 con 0x7f3c8c15ad70 2023-10-16T15:55:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c91c50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c91c50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54384/0 (socket says 172.21.15.29:54384) 2023-10-16T15:55:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.978+0000 7f3c91c50640 1 -- 172.21.15.29:0/2179899826 learned_addr learned my addr 172.21.15.29:0/2179899826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c91c50640 1 -- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c15ad70 msgr2=0x7f3c8c157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c91c50640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c15ad70 0x7f3c8c157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c91c50640 1 -- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c8c0c9ca0 msgr2=0x7f3c8c159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c91c50640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c8c0c9ca0 0x7f3c8c159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c91c50640 1 -- 172.21.15.29:0/2179899826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0094d0 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c91c50640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0c5a80 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b420 tx=0x7f3c8800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88011830 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c88011990 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.979+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88011d40 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.980+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c0aef80 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.980+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c1f1da0 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.981+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c04e160 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.982+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f3c8801b460 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.983+0000 7f3c827fc640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3c7006b020 0x7f3c7006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:29.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.983+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f3c880895d0 con 0x7f3c8c06a4b0 2023-10-16T15:55:29.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.983+0000 7f3c9144f640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3c7006b020 0x7f3c7006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:29.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.984+0000 7f3c9144f640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3c7006b020 0x7f3c7006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c036000 tx=0x7f3c7c030000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:29.987+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3c88054320 con 0x7f3c8c06a4b0 2023-10-16T15:55:30.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.191+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f3c8c0b2b10 con 0x7f3c8c06a4b0 2023-10-16T15:55:30.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.196+0000 7f3c827fc640 1 -- 172.21.15.29:0/2179899826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3c88057940 con 0x7f3c8c06a4b0 2023-10-16T15:55:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.199+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3c7006b020 msgr2=0x7f3c7006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.199+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3c7006b020 0x7f3c7006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c036000 tx=0x7f3c7c030000 comp rx=0 tx=0).stop 2023-10-16T15:55:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.199+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 msgr2=0x7f3c8c0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.199+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0c5a80 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b420 tx=0x7f3c8800b8e0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 shutdown_connections 2023-10-16T15:55:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f3c7006b020 0x7f3c7006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3c8c15ad70 0x7f3c8c157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3c8c0c9ca0 0x7f3c8c159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 --2- 172.21.15.29:0/2179899826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c06a4b0 0x7f3c8c0c5a80 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 >> 172.21.15.29:0/2179899826 conn(0x7f3c8c0ce9b0 msgr2=0x7f3c8c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:30.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 shutdown_connections 2023-10-16T15:55:30.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.200+0000 7f3c93eb4640 1 -- 172.21.15.29:0/2179899826 wait complete. 2023-10-16T15:55:30.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T15:55:30.253 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:30.250+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:30.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.446+0000 7f9880e00640 1 Processor -- start 2023-10-16T15:55:30.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.447+0000 7f9880e00640 1 -- start start 2023-10-16T15:55:30.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.448+0000 7f9880e00640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f987c15aaa0 0x7f987c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:30.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.448+0000 7f9880e00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c0ce390 0x7f987c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:30.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.448+0000 7f9880e00640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c15ed60 con 0x7f987c15ae80 2023-10-16T15:55:30.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.448+0000 7f9880e00640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c06b810 con 0x7f987c15aaa0 2023-10-16T15:55:30.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.448+0000 7f9880e00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0707e0 con 0x7f987c0ce390 2023-10-16T15:55:30.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.449+0000 7f987a59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f987c15aaa0 0x7f987c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:30.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.449+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c0ce390 0x7f987c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:30.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.449+0000 7f987a59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f987c15aaa0 0x7f987c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:59930/0 (socket says 172.21.15.29:59930) 2023-10-16T15:55:30.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.449+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c0ce390 0x7f987c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:44692/0 (socket says 172.21.15.29:44692) 2023-10-16T15:55:30.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.449+0000 7f987a59c640 1 -- 172.21.15.29:0/1764675113 learned_addr learned my addr 172.21.15.29:0/1764675113 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.450+0000 7f987a59c640 1 -- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c0ce390 msgr2=0x7f987c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.450+0000 7f987a59c640 1 --2- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c0ce390 0x7f987c070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.450+0000 7f987a59c640 1 -- 172.21.15.29:0/1764675113 >> v1:172.21.15.170:6789/0 conn(0x7f987c15ae80 legacy=0x7f987c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.450+0000 7f987a59c640 1 -- 172.21.15.29:0/1764675113 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c15ef90 con 0x7f987c15aaa0 2023-10-16T15:55:30.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.451+0000 7f987a59c640 1 --2- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c15aaa0 0x7f987c15e820 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f986c009870 tx=0x7f986c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5175ad3543a7cc1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.451+0000 7f987959a640 1 -- 172.21.15.29:0/1764675113 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c030840 con 0x7f987c15aaa0 2023-10-16T15:55:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f987959a640 1 -- 172.21.15.29:0/1764675113 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c0309a0 con 0x7f987c15aaa0 2023-10-16T15:55:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f987959a640 1 -- 172.21.15.29:0/1764675113 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c030d30 con 0x7f987c15aaa0 2023-10-16T15:55:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 -- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c15aaa0 msgr2=0x7f987c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 --2- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c15aaa0 0x7f987c15e820 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f986c009870 tx=0x7f986c02b980 comp rx=0 tx=0).stop 2023-10-16T15:55:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 -- 172.21.15.29:0/1764675113 shutdown_connections 2023-10-16T15:55:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 --2- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c0ce390 0x7f987c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 --2- 172.21.15.29:0/1764675113 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c15aaa0 0x7f987c15e820 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 -- 172.21.15.29:0/1764675113 >> 172.21.15.29:0/1764675113 conn(0x7f987c0cb410 msgr2=0x7f987c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 -- 172.21.15.29:0/1764675113 shutdown_connections 2023-10-16T15:55:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.452+0000 7f9880e00640 1 -- 172.21.15.29:0/1764675113 wait complete. 2023-10-16T15:55:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.454+0000 7f9880e00640 1 Processor -- start 2023-10-16T15:55:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.454+0000 7f9880e00640 1 -- start start 2023-10-16T15:55:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.455+0000 7f9880e00640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c0ce390 0x7f987c0c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.455+0000 7f9880e00640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c15aaa0 0x7f987c0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.455+0000 7f9880e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 0x7f987c06fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.455+0000 7f9880e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0c5a60 con 0x7f987c15ae80 2023-10-16T15:55:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.455+0000 7f9880e00640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0c4330 con 0x7f987c0ce390 2023-10-16T15:55:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.455+0000 7f9880e00640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0c5170 con 0x7f987c15aaa0 2023-10-16T15:55:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 0x7f987c06fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 0x7f987c06fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54386/0 (socket says 172.21.15.29:54386) 2023-10-16T15:55:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 -- 172.21.15.29:0/3805058516 learned_addr learned my addr 172.21.15.29:0/3805058516 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987a59c640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c0ce390 0x7f987c0c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f9879d9b640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c15aaa0 0x7f987c0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 -- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c15aaa0 msgr2=0x7f987c0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c15aaa0 0x7f987c0c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 -- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c0ce390 msgr2=0x7f987c0c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c0ce390 0x7f987c0c30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987ad9d640 1 -- 172.21.15.29:0/3805058516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0094d0 con 0x7f987c15ae80 2023-10-16T15:55:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.456+0000 7f987a59c640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c0ce390 0x7f987c0c30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.457+0000 7f987ad9d640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 0x7f987c06fc40 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f987400efc0 tx=0x7f987400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.457+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9874014480 con 0x7f987c15ae80 2023-10-16T15:55:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.457+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987400f040 con 0x7f987c15ae80 2023-10-16T15:55:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.457+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c0c4560 con 0x7f987c15ae80 2023-10-16T15:55:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.457+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9874014820 con 0x7f987c15ae80 2023-10-16T15:55:30.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.457+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c0c4b20 con 0x7f987c15ae80 2023-10-16T15:55:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.458+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c0c52e0 con 0x7f987c15ae80 2023-10-16T15:55:30.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.459+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f9874014980 con 0x7f987c15ae80 2023-10-16T15:55:30.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.460+0000 7f9872ffd640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f985c06b020 0x7f985c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:30.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.460+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f987408a7c0 con 0x7f987c15ae80 2023-10-16T15:55:30.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.461+0000 7f987a59c640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f985c06b020 0x7f985c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:30.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.463+0000 7f987a59c640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f985c06b020 0x7f985c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c03b040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:30.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.465+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9874055300 con 0x7f987c15ae80 2023-10-16T15:55:30.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.731+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f987c04e160 con 0x7f987c15ae80 2023-10-16T15:55:30.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.805+0000 7f9872ffd640 1 -- 172.21.15.29:0/3805058516 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3282) v1 ==== 187+0+0 (secure 0 0 0) 0x7f9874058920 con 0x7f987c15ae80 2023-10-16T15:55:30.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f985c06b020 msgr2=0x7f985c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f985c06b020 0x7f985c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c03b040 comp rx=0 tx=0).stop 2023-10-16T15:55:30.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 msgr2=0x7f987c06fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:30.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 0x7f987c06fc40 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f987400efc0 tx=0x7f987400c490 comp rx=0 tx=0).stop 2023-10-16T15:55:30.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 shutdown_connections 2023-10-16T15:55:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f985c06b020 0x7f985c06d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f987c15ae80 0x7f987c06fc40 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f987c15aaa0 0x7f987c0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 --2- 172.21.15.29:0/3805058516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f987c0ce390 0x7f987c0c30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.808+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 >> 172.21.15.29:0/3805058516 conn(0x7f987c0cb410 msgr2=0x7f987c0b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:30.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.809+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 shutdown_connections 2023-10-16T15:55:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:30.809+0000 7f9880e00640 1 -- 172.21.15.29:0/3805058516 wait complete. 2023-10-16T15:55:30.833 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2023-10-16T15:55:30.833 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-16T15:55:30.833 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-16T15:55:30.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.038+0000 7f791aedc640 1 Processor -- start 2023-10-16T15:55:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.038+0000 7f791aedc640 1 -- start start 2023-10-16T15:55:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f791aedc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79140ad970 0x7f79140cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f791aedc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 0x7f79140cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f791aedc640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791415b9d0 con 0x7f791406c040 2023-10-16T15:55:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f791aedc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791415bcb0 con 0x7f79140ad970 2023-10-16T15:55:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f791aedc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791415bf90 con 0x7f79140ccfb0 2023-10-16T15:55:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f79196d9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79140ad970 0x7f79140cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f79196d9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79140ad970 0x7f79140cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54402/0 (socket says 172.21.15.29:54402) 2023-10-16T15:55:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f79196d9640 1 -- 172.21.15.29:0/3883480052 learned_addr learned my addr 172.21.15.29:0/3883480052 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f79196d9640 1 -- 172.21.15.29:0/3883480052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 msgr2=0x7f79140cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.040+0000 7f79196d9640 1 --2- 172.21.15.29:0/3883480052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 0x7f79140cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.041+0000 7f79196d9640 1 -- 172.21.15.29:0/3883480052 >> v1:172.21.15.170:6789/0 conn(0x7f791406c040 legacy=0x7f791406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.041+0000 7f79196d9640 1 -- 172.21.15.29:0/3883480052 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79140cc270 con 0x7f79140ad970 2023-10-16T15:55:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.042+0000 7f79196d9640 1 --2- 172.21.15.29:0/3883480052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79140ad970 0x7f79140cca70 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f7908009870 tx=0x7f790802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324f92ff4339f343 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.042+0000 7f7918ed8640 1 -- 172.21.15.29:0/3883480052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908031640 con 0x7f79140ad970 2023-10-16T15:55:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.042+0000 7f7918ed8640 1 -- 172.21.15.29:0/3883480052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79080317a0 con 0x7f79140ad970 2023-10-16T15:55:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.042+0000 7f7918ed8640 1 -- 172.21.15.29:0/3883480052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908031b30 con 0x7f79140ad970 2023-10-16T15:55:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 -- 172.21.15.29:0/3883480052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79140ad970 msgr2=0x7f79140cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 --2- 172.21.15.29:0/3883480052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79140ad970 0x7f79140cca70 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f7908009870 tx=0x7f790802b8f0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 -- 172.21.15.29:0/3883480052 shutdown_connections 2023-10-16T15:55:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 --2- 172.21.15.29:0/3883480052 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 0x7f79140cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 --2- 172.21.15.29:0/3883480052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79140ad970 0x7f79140cca70 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 -- 172.21.15.29:0/3883480052 >> 172.21.15.29:0/3883480052 conn(0x7f79140ce6a0 msgr2=0x7f79140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 -- 172.21.15.29:0/3883480052 shutdown_connections 2023-10-16T15:55:31.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.043+0000 7f791aedc640 1 -- 172.21.15.29:0/3883480052 wait complete. 2023-10-16T15:55:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.045+0000 7f791aedc640 1 Processor -- start 2023-10-16T15:55:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.045+0000 7f791aedc640 1 -- start start 2023-10-16T15:55:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f791aedc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 0x7f79140c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f791aedc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140ad970 0x7f79140c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f791aedc640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 0x7f79140acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f791aedc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79140c3060 con 0x7f791406c040 2023-10-16T15:55:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f791aedc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79140c9c40 con 0x7f79140ad970 2023-10-16T15:55:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f791aedc640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79140c5b50 con 0x7f79140ccfb0 2023-10-16T15:55:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f7919eda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 0x7f79140c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f7919eda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 0x7f79140c2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54410/0 (socket says 172.21.15.29:54410) 2023-10-16T15:55:31.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f7919eda640 1 -- 172.21.15.29:0/3522318013 learned_addr learned my addr 172.21.15.29:0/3522318013 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.047+0000 7f79196d9640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140ad970 0x7f79140c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f7919eda640 1 -- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 msgr2=0x7f79140acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f7919eda640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 0x7f79140acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f7919eda640 1 -- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140ad970 msgr2=0x7f79140c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f7919eda640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140ad970 0x7f79140c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f7919eda640 1 -- 172.21.15.29:0/3522318013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79080094d0 con 0x7f791406c040 2023-10-16T15:55:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f79196d9640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140ad970 0x7f79140c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f7919eda640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 0x7f79140c2b20 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f790400c7c0 tx=0x7f790400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790400aa50 con 0x7f791406c040 2023-10-16T15:55:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790400abb0 con 0x7f791406c040 2023-10-16T15:55:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904012420 con 0x7f791406c040 2023-10-16T15:55:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79140c5e30 con 0x7f791406c040 2023-10-16T15:55:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.048+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79140c63f0 con 0x7f791406c040 2023-10-16T15:55:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.050+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791406b330 con 0x7f791406c040 2023-10-16T15:55:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.051+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f7904012580 con 0x7f791406c040 2023-10-16T15:55:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.052+0000 7f79127fc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f790006b050 0x7f790006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.052+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f7904089950 con 0x7f791406c040 2023-10-16T15:55:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.052+0000 7f79196d9640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f790006b050 0x7f790006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.054+0000 7f79196d9640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f790006b050 0x7f790006d510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7908036000 tx=0x7f790803c040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.056+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f79040546b0 con 0x7f791406c040 2023-10-16T15:55:31.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.253+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79140c6b20 con 0x7f791406c040 2023-10-16T15:55:31.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.254+0000 7f79127fc640 1 -- 172.21.15.29:0/3522318013 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+14437 (secure 0 0 0) 0x7f7904016090 con 0x7f791406c040 2023-10-16T15:55:31.257 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:31.257 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":258,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:28.782592+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6834/2428945430":"2023-10-17T15:55:25.763889+0000","172.21.15.170:0/3622898311":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6833/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:6832/3738368341":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3389497932":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2059078240":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/1624942667":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3780993554":"2023-10-17T15:55:11.776893+0000","172.21.15.29:6835/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2581880557":"2023-10-17T15:55:05.561118+0000","172.21.15.170:6833/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.29:6832/2428945430":"2023-10-17T15:55:25.763889+0000","172.21.15.170:6832/58791103":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3658934206":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3878756216":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3202676598":"2023-10-17T15:55:05.561118+0000","172.21.15.29:6833/3076937560":"2023-10-17T15:55:17.748639+0000","172.21.15.170:0/2546993573":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/3985010689":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2572859596":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/2502304360":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/750862606":"2023-10-17T15:55:11.776893+0000","172.21.15.170:0/2231350835":"2023-10-17T15:55:05.561118+0000","172.21.15.170:0/3329304075":"2023-10-17T15:55:05.561118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:31.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f790006b050 msgr2=0x7f790006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f790006b050 0x7f790006d510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7908036000 tx=0x7f790803c040 comp rx=0 tx=0).stop 2023-10-16T15:55:31.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 msgr2=0x7f79140c2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 0x7f79140c2b20 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f790400c7c0 tx=0x7f790400cc80 comp rx=0 tx=0).stop 2023-10-16T15:55:31.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 shutdown_connections 2023-10-16T15:55:31.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f790006b050 0x7f790006d510 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f79140ccfb0 0x7f79140acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f79140ad970 0x7f79140c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 --2- 172.21.15.29:0/3522318013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f791406c040 0x7f79140c2b20 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 >> 172.21.15.29:0/3522318013 conn(0x7f79140ce6a0 msgr2=0x7f791406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:31.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 shutdown_connections 2023-10-16T15:55:31.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.257+0000 7f791aedc640 1 -- 172.21.15.29:0/3522318013 wait complete. 2023-10-16T15:55:31.279 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:55:31.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:55:31.284 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:31.281+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.506+0000 7efe0214e640 1 Processor -- start 2023-10-16T15:55:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.506+0000 7efe0214e640 1 -- start start 2023-10-16T15:55:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0214e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc159ad0 0x7efdfc070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0214e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efdfc0c9de0 0x7efdfc0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0214e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdfc0cd260 con 0x7efdfc15d650 2023-10-16T15:55:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0214e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdfc071470 con 0x7efdfc0c9de0 2023-10-16T15:55:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0214e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0ca7b0 con 0x7efdfc159ad0 2023-10-16T15:55:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0194d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efdfc0c9de0 0x7efdfc0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0094b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efdfc15d650 0x7efdfc06aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54766/0 (socket says 172.21.15.29:54766) 2023-10-16T15:55:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0094b640 1 -- 172.21.15.29:0/3288031210 learned_addr learned my addr 172.21.15.29:0/3288031210 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efe0114c640 1 --2- 172.21.15.29:0/3288031210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc159ad0 0x7efdfc070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.508+0000 7efdf3fff640 1 -- 172.21.15.29:0/3288031210 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436332086 0 0) 0x7efdfc0cd260 con 0x7efdfc15d650 2023-10-16T15:55:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.509+0000 7efdf3fff640 1 -- 172.21.15.29:0/3288031210 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efde8003520 con 0x7efdfc15d650 2023-10-16T15:55:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.509+0000 7efe0194d640 1 -- 172.21.15.29:0/3288031210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc159ad0 msgr2=0x7efdfc070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.509+0000 7efe0194d640 1 --2- 172.21.15.29:0/3288031210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc159ad0 0x7efdfc070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.509+0000 7efe0194d640 1 -- 172.21.15.29:0/3288031210 >> v1:172.21.15.29:6789/0 conn(0x7efdfc15d650 legacy=0x7efdfc06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.509+0000 7efe0194d640 1 -- 172.21.15.29:0/3288031210 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc15de30 con 0x7efdfc0c9de0 2023-10-16T15:55:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.510+0000 7efe0194d640 1 --2- 172.21.15.29:0/3288031210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc0ca270 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009d90 tx=0x7efdf402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b856488ef014f09c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.510+0000 7efdf3fff640 1 -- 172.21.15.29:0/3288031210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf402cc50 con 0x7efdfc0c9de0 2023-10-16T15:55:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.510+0000 7efdf3fff640 1 -- 172.21.15.29:0/3288031210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf402cdb0 con 0x7efdfc0c9de0 2023-10-16T15:55:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.510+0000 7efdf3fff640 1 -- 172.21.15.29:0/3288031210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4034660 con 0x7efdfc0c9de0 2023-10-16T15:55:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 -- 172.21.15.29:0/3288031210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 msgr2=0x7efdfc0ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 --2- 172.21.15.29:0/3288031210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc0ca270 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009d90 tx=0x7efdf402bea0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 -- 172.21.15.29:0/3288031210 shutdown_connections 2023-10-16T15:55:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 --2- 172.21.15.29:0/3288031210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc0ca270 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 --2- 172.21.15.29:0/3288031210 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc159ad0 0x7efdfc070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 -- 172.21.15.29:0/3288031210 >> 172.21.15.29:0/3288031210 conn(0x7efdfc0b0170 msgr2=0x7efdfc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 -- 172.21.15.29:0/3288031210 shutdown_connections 2023-10-16T15:55:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.511+0000 7efe0214e640 1 -- 172.21.15.29:0/3288031210 wait complete. 2023-10-16T15:55:31.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.513+0000 7efe0214e640 1 Processor -- start 2023-10-16T15:55:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.513+0000 7efe0214e640 1 -- start start 2023-10-16T15:55:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.514+0000 7efe0214e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.514+0000 7efe0214e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdfc159ad0 0x7efdfc14fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.514+0000 7efe0214e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc15d650 0x7efdfc0c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.514+0000 7efe0214e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1510c0 con 0x7efdfc0c9de0 2023-10-16T15:55:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.514+0000 7efe0214e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc150040 con 0x7efdfc159ad0 2023-10-16T15:55:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.514+0000 7efe0214e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0c5a70 con 0x7efdfc15d650 2023-10-16T15:55:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54420/0 (socket says 172.21.15.29:54420) 2023-10-16T15:55:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 -- 172.21.15.29:0/1566996934 learned_addr learned my addr 172.21.15.29:0/1566996934 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0094b640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdfc159ad0 0x7efdfc14fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0194d640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc15d650 0x7efdfc0c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 -- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc15d650 msgr2=0x7efdfc0c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc15d650 0x7efdfc0c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 -- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdfc159ad0 msgr2=0x7efdfc14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdfc159ad0 0x7efdfc14fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.515+0000 7efe0114c640 1 -- 172.21.15.29:0/1566996934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4009a70 con 0x7efdfc0c9de0 2023-10-16T15:55:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.516+0000 7efe0114c640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc158fe0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7efdf800d830 tx=0x7efdf800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.516+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800b660 con 0x7efdfc0c9de0 2023-10-16T15:55:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.516+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf800b7c0 con 0x7efdfc0c9de0 2023-10-16T15:55:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.516+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc150de0 con 0x7efdfc0c9de0 2023-10-16T15:55:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.516+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800bb50 con 0x7efdfc0c9de0 2023-10-16T15:55:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.516+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc14f300 con 0x7efdfc0c9de0 2023-10-16T15:55:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.517+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc14ee80 con 0x7efdfc0c9de0 2023-10-16T15:55:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.518+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7efdf801e460 con 0x7efdfc0c9de0 2023-10-16T15:55:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.519+0000 7efdf1ffb640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efde006b020 0x7efde006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.520+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7efdf808a150 con 0x7efdfc0c9de0 2023-10-16T15:55:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.520+0000 7efe0094b640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efde006b020 0x7efde006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.521+0000 7efe0094b640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efde006b020 0x7efde006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efdfc1f5690 tx=0x7efdec009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.524+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efdfc14ee80 con 0x7efdfc0c9de0 2023-10-16T15:55:31.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.718+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efdfc14ee80 con 0x7efdfc0c9de0 2023-10-16T15:55:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.719+0000 7efdf1ffb640 1 -- 172.21.15.29:0/1566996934 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) v1 ==== 93+0+10 (secure 0 0 0) 0x7efdfc14ee80 con 0x7efdfc0c9de0 2023-10-16T15:55:31.721 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:55:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efde006b020 msgr2=0x7efde006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efde006b020 0x7efde006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efdfc1f5690 tx=0x7efdec009040 comp rx=0 tx=0).stop 2023-10-16T15:55:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 msgr2=0x7efdfc158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc158fe0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7efdf800d830 tx=0x7efdf800dcf0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 shutdown_connections 2023-10-16T15:55:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7efde006b020 0x7efde006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7efdfc15d650 0x7efdfc0c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efdfc159ad0 0x7efdfc14fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 --2- 172.21.15.29:0/1566996934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdfc0c9de0 0x7efdfc158fe0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 >> 172.21.15.29:0/1566996934 conn(0x7efdfc0b0170 msgr2=0x7efdfc1513f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 shutdown_connections 2023-10-16T15:55:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.722+0000 7efe0214e640 1 -- 172.21.15.29:0/1566996934 wait complete. 2023-10-16T15:55:31.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T15:55:31.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.973+0000 7f81014da640 1 Processor -- start 2023-10-16T15:55:31.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.973+0000 7f81014da640 1 -- start start 2023-10-16T15:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.974+0000 7f81014da640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80fc0cdef0 0x7f80fc1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f81014da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc0b84a0 0x7f80fc0d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f81014da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc0d0060 con 0x7f80fc15cdf0 2023-10-16T15:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f81014da640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80fc161970 con 0x7f80fc0cdef0 2023-10-16T15:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f81014da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0d0e60 con 0x7f80fc0b84a0 2023-10-16T15:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80fc0cdef0 0x7f80fc1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80fc0cdef0 0x7f80fc1609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54430/0 (socket says 172.21.15.29:54430) 2023-10-16T15:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80fbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f80fc15cdf0 0x7f80fc0cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54776/0 (socket says 172.21.15.29:54776) 2023-10-16T15:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 -- 172.21.15.29:0/1377177772 learned_addr learned my addr 172.21.15.29:0/1377177772 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 -- 172.21.15.29:0/1377177772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc0b84a0 msgr2=0x7f80fc0d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 --2- 172.21.15.29:0/1377177772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc0b84a0 0x7f80fc0d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 -- 172.21.15.29:0/1377177772 >> v1:172.21.15.29:6789/0 conn(0x7f80fc15cdf0 legacy=0x7f80fc0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.975+0000 7f80f37fe640 1 -- 172.21.15.29:0/1377177772 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc1610d0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.976+0000 7f80f37fe640 1 --2- 172.21.15.29:0/1377177772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc1609f0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009870 tx=0x7f80e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6f3b47f77ec4bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.977+0000 7f80fb7fe640 1 -- 172.21.15.29:0/1377177772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4030840 con 0x7f80fc0cdef0 2023-10-16T15:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.977+0000 7f80fb7fe640 1 -- 172.21.15.29:0/1377177772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e40309a0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.977+0000 7f80fb7fe640 1 -- 172.21.15.29:0/1377177772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4030d30 con 0x7f80fc0cdef0 2023-10-16T15:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 -- 172.21.15.29:0/1377177772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 msgr2=0x7f80fc1609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 --2- 172.21.15.29:0/1377177772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc1609f0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009870 tx=0x7f80e402b980 comp rx=0 tx=0).stop 2023-10-16T15:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 -- 172.21.15.29:0/1377177772 shutdown_connections 2023-10-16T15:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 --2- 172.21.15.29:0/1377177772 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc0b84a0 0x7f80fc0d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 --2- 172.21.15.29:0/1377177772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc1609f0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 -- 172.21.15.29:0/1377177772 >> 172.21.15.29:0/1377177772 conn(0x7f80fc06bc60 msgr2=0x7f80fc06c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 -- 172.21.15.29:0/1377177772 shutdown_connections 2023-10-16T15:55:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.978+0000 7f81014da640 1 -- 172.21.15.29:0/1377177772 wait complete. 2023-10-16T15:55:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.980+0000 7f81014da640 1 Processor -- start 2023-10-16T15:55:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.980+0000 7f81014da640 1 -- start start 2023-10-16T15:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f81014da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc0b84a0 0x7f80fc1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f81014da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f81014da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc15cdf0 0x7f80fc0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f81014da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0c9be0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f81014da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc15df10 con 0x7f80fc0b84a0 2023-10-16T15:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f81014da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc15e1f0 con 0x7f80fc15cdf0 2023-10-16T15:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f80f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f80f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc15d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54446/0 (socket says 172.21.15.29:54446) 2023-10-16T15:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.981+0000 7f80f37fe640 1 -- 172.21.15.29:0/3335951568 learned_addr learned my addr 172.21.15.29:0/3335951568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80fbfff640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc0b84a0 0x7f80fc1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f8100cd9640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc15cdf0 0x7f80fc0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f37fe640 1 -- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc15cdf0 msgr2=0x7f80fc0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f37fe640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc15cdf0 0x7f80fc0cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f37fe640 1 -- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc0b84a0 msgr2=0x7f80fc1508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f37fe640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc0b84a0 0x7f80fc1508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f37fe640 1 -- 172.21.15.29:0/3335951568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40094d0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80fbfff640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc0b84a0 0x7f80fc1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f8100cd9640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc15cdf0 0x7f80fc0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f37fe640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc15d4c0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009870 tx=0x7f80e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40307f0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e4043400 con 0x7f80fc0cdef0 2023-10-16T15:55:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40437b0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.982+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc15e4d0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.983+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc150df0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.984+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc04e9b0 con 0x7f80fc0cdef0 2023-10-16T15:55:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.985+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f80e4043910 con 0x7f80fc0cdef0 2023-10-16T15:55:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.986+0000 7f80f97fa640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f80dc06b020 0x7f80dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.986+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f80e403e070 con 0x7f80fc0cdef0 2023-10-16T15:55:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.987+0000 7f80fbfff640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f80dc06b020 0x7f80dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.988+0000 7f80fbfff640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f80dc06b020 0x7f80dc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80ec000f60 tx=0x7f80ec009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:31.990+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f80e4077b70 con 0x7f80fc0cdef0 2023-10-16T15:55:32.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.269+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f80fc0d07e0 con 0x7f80fc0cdef0 2023-10-16T15:55:32.314 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:32.311+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.818+0000 7f80f97fa640 1 -- 172.21.15.29:0/3335951568 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3284) v1 ==== 158+0+0 (secure 0 0 0) 0x7f80e40340e0 con 0x7f80fc0cdef0 2023-10-16T15:55:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.821+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f80dc06b020 msgr2=0x7f80dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.821+0000 7f81014da640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f80dc06b020 0x7f80dc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80ec000f60 tx=0x7f80ec009040 comp rx=0 tx=0).stop 2023-10-16T15:55:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.821+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 msgr2=0x7f80fc15d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.821+0000 7f81014da640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc15d4c0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009870 tx=0x7f80e4004730 comp rx=0 tx=0).stop 2023-10-16T15:55:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 shutdown_connections 2023-10-16T15:55:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f80dc06b020 0x7f80dc06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc0cdef0 0x7f80fc15d4c0 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f80fc15cdf0 0x7f80fc0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 --2- 172.21.15.29:0/3335951568 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f80fc0b84a0 0x7f80fc1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 >> 172.21.15.29:0/3335951568 conn(0x7f80fc06bc60 msgr2=0x7f80fc0c6640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 shutdown_connections 2023-10-16T15:55:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:32.822+0000 7f81014da640 1 -- 172.21.15.29:0/3335951568 wait complete. 2023-10-16T15:55:32.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T15:55:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.069+0000 7f6f1c4d4640 1 Processor -- start 2023-10-16T15:55:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.069+0000 7f6f1c4d4640 1 -- start start 2023-10-16T15:55:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.070+0000 7f6f1c4d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f14159ab0 0x7f6f140ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.070+0000 7f6f1c4d4640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6f14159e90 0x7f6f1406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.070+0000 7f6f1c4d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f1406c140 0x7f6f1406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.070+0000 7f6f1c4d4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1406cb10 con 0x7f6f14159ab0 2023-10-16T15:55:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.070+0000 7f6f1c4d4640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1415d890 con 0x7f6f14159e90 2023-10-16T15:55:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.070+0000 7f6f1c4d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f140cd1f0 con 0x7f6f1406c140 2023-10-16T15:55:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f14159ab0 0x7f6f140ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f14159ab0 0x7f6f140ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54452/0 (socket says 172.21.15.29:54452) 2023-10-16T15:55:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 -- 172.21.15.29:0/190049776 learned_addr learned my addr 172.21.15.29:0/190049776 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1aa71640 1 --2- 172.21.15.29:0/190049776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f1406c140 0x7f6f1406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f19a6f640 1 --2- 172.21.15.29:0/190049776 >> v2:172.21.15.170:3300/0 conn(0x7f6f14159e90 0x7f6f1406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 -- 172.21.15.29:0/190049776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f1406c140 msgr2=0x7f6f1406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 --2- 172.21.15.29:0/190049776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f1406c140 0x7f6f1406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 -- 172.21.15.29:0/190049776 >> v2:172.21.15.170:3300/0 conn(0x7f6f14159e90 msgr2=0x7f6f1406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 --2- 172.21.15.29:0/190049776 >> v2:172.21.15.170:3300/0 conn(0x7f6f14159e90 0x7f6f1406fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f1a270640 1 -- 172.21.15.29:0/190049776 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f140cd4d0 con 0x7f6f14159ab0 2023-10-16T15:55:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.071+0000 7f6f19a6f640 1 --2- 172.21.15.29:0/190049776 >> v2:172.21.15.170:3300/0 conn(0x7f6f14159e90 0x7f6f1406fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.072+0000 7f6f1a270640 1 --2- 172.21.15.29:0/190049776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f14159ab0 0x7f6f140ce370 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f6f1406d900 tx=0x7f6f1002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e063ae5de231a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1926e640 1 -- 172.21.15.29:0/190049776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10031740 con 0x7f6f14159ab0 2023-10-16T15:55:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1926e640 1 -- 172.21.15.29:0/190049776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f100318a0 con 0x7f6f14159ab0 2023-10-16T15:55:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1926e640 1 -- 172.21.15.29:0/190049776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10031c30 con 0x7f6f14159ab0 2023-10-16T15:55:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/190049776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f14159ab0 msgr2=0x7f6f140ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/190049776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f14159ab0 0x7f6f140ce370 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f6f1406d900 tx=0x7f6f1002b7e0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/190049776 shutdown_connections 2023-10-16T15:55:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/190049776 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f1406c140 0x7f6f1406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/190049776 >> v2:172.21.15.170:3300/0 conn(0x7f6f14159e90 0x7f6f1406fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/190049776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f14159ab0 0x7f6f140ce370 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/190049776 >> 172.21.15.29:0/190049776 conn(0x7f6f140ce9b0 msgr2=0x7f6f140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:33.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/190049776 shutdown_connections 2023-10-16T15:55:33.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.073+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/190049776 wait complete. 2023-10-16T15:55:33.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.075+0000 7f6f1c4d4640 1 Processor -- start 2023-10-16T15:55:33.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.075+0000 7f6f1c4d4640 1 -- start start 2023-10-16T15:55:33.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.076+0000 7f6f1c4d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 0x7f6f14158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.076+0000 7f6f1c4d4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f14159ab0 0x7f6f140aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.076+0000 7f6f1c4d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f14159e90 0x7f6f140af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.076+0000 7f6f1c4d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1415ae10 con 0x7f6f1406c140 2023-10-16T15:55:33.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.076+0000 7f6f1c4d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f141e3cc0 con 0x7f6f14159e90 2023-10-16T15:55:33.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.076+0000 7f6f1c4d4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1415b3b0 con 0x7f6f14159ab0 2023-10-16T15:55:33.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 0x7f6f14158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 0x7f6f14158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54454/0 (socket says 172.21.15.29:54454) 2023-10-16T15:55:33.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 -- 172.21.15.29:0/917246100 learned_addr learned my addr 172.21.15.29:0/917246100 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:33.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f19a6f640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f14159ab0 0x7f6f140aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1aa71640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f14159e90 0x7f6f140af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 -- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f14159ab0 msgr2=0x7f6f140aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:33.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f14159ab0 0x7f6f140aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 -- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f14159e90 msgr2=0x7f6f140af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:33.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f14159e90 0x7f6f140af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 -- 172.21.15.29:0/917246100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f100094d0 con 0x7f6f1406c140 2023-10-16T15:55:33.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.077+0000 7f6f1a270640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 0x7f6f14158aa0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f6f1002b820 tx=0x7f6f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:33.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.078+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10033c70 con 0x7f6f1406c140 2023-10-16T15:55:33.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.078+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f10033df0 con 0x7f6f1406c140 2023-10-16T15:55:33.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.078+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1003c640 con 0x7f6f1406c140 2023-10-16T15:55:33.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.078+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f141e3700 con 0x7f6f1406c140 2023-10-16T15:55:33.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.078+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1415b040 con 0x7f6f1406c140 2023-10-16T15:55:33.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.079+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1404e160 con 0x7f6f1406c140 2023-10-16T15:55:33.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.080+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f6f1003a040 con 0x7f6f1406c140 2023-10-16T15:55:33.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.081+0000 7f6f06ffd640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6ef806b050 0x7f6ef806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:33.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.081+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f6f100ab570 con 0x7f6f1406c140 2023-10-16T15:55:33.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.082+0000 7f6f19a6f640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6ef806b050 0x7f6ef806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:33.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.083+0000 7f6f19a6f640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6ef806b050 0x7f6ef806d510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f140afa90 tx=0x7f6f08008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:33.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.086+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6f100761b0 con 0x7f6f1406c140 2023-10-16T15:55:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.286+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6f1415b1d0 con 0x7f6f1406c140 2023-10-16T15:55:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.292+0000 7f6f06ffd640 1 -- 172.21.15.29:0/917246100 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6f100797d0 con 0x7f6f1406c140 2023-10-16T15:55:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6ef806b050 msgr2=0x7f6ef806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6ef806b050 0x7f6ef806d510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6f140afa90 tx=0x7f6f08008040 comp rx=0 tx=0).stop 2023-10-16T15:55:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 msgr2=0x7f6f14158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 0x7f6f14158aa0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f6f1002b820 tx=0x7f6f100046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 shutdown_connections 2023-10-16T15:55:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6ef806b050 0x7f6ef806d510 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f14159e90 0x7f6f140af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6f14159ab0 0x7f6f140aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 --2- 172.21.15.29:0/917246100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f1406c140 0x7f6f14158aa0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.295+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 >> 172.21.15.29:0/917246100 conn(0x7f6f140ce9b0 msgr2=0x7f6f14070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.296+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 shutdown_connections 2023-10-16T15:55:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:33.296+0000 7f6f1c4d4640 1 -- 172.21.15.29:0/917246100 wait complete. 2023-10-16T15:55:33.318 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:55:33.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:55:33.387 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:33.384+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:33.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:55:33.574 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:55:33.575 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:33.575 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:33.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:55:33.639 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:55:33.640 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:33.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:55:33.696 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:33.697 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:33.761 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:55:33.762 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:55:33.762 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:33.819 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:55:33.819 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T15:55:33.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.056+0000 7f58c6954640 1 Processor -- start 2023-10-16T15:55:34.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.056+0000 7f58c6954640 1 -- start start 2023-10-16T15:55:34.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.057+0000 7f58c6954640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58c015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.057+0000 7f58c6954640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f58c0070450 0x7f58c00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.057+0000 7f58c6954640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c0070df0 con 0x7f58c006c100 2023-10-16T15:55:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.057+0000 7f58c6954640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c015e490 con 0x7f58c0070450 2023-10-16T15:55:34.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.057+0000 7f58c6954640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c015aef0 con 0x7f58c0159950 2023-10-16T15:55:34.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.058+0000 7f58c6153640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f58c006c100 0x7f58c006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50324/0 (socket says 172.21.15.29:50324) 2023-10-16T15:55:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.058+0000 7f58c6153640 1 -- 172.21.15.29:0/3112832067 learned_addr learned my addr 172.21.15.29:0/3112832067 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.058+0000 7f58c5952640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58c015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.058+0000 7f58c5151640 1 --2- 172.21.15.29:0/3112832067 >> v2:172.21.15.170:3300/0 conn(0x7f58c0070450 0x7f58c00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473038156 0 0) 0x7f58c0070df0 con 0x7f58c006c100 2023-10-16T15:55:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac003520 con 0x7f58c006c100 2023-10-16T15:55:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885760177 0 0) 0x7f58ac003520 con 0x7f58c006c100 2023-10-16T15:55:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 msgr2=0x7f58c015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58c015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 >> v2:172.21.15.170:3300/0 conn(0x7f58c0070450 msgr2=0x7f58c00708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 --2- 172.21.15.29:0/3112832067 >> v2:172.21.15.170:3300/0 conn(0x7f58c0070450 0x7f58c00708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c006cc80 con 0x7f58c006c100 2023-10-16T15:55:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f58b80019f0 con 0x7f58c006c100 2023-10-16T15:55:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 >> v1:172.21.15.29:6789/0 conn(0x7f58c006c100 legacy=0x7f58c006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ac003970 0x7f58ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac0047c0 0x7f58ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c015e490 con 0x7f58ac003970 2023-10-16T15:55:34.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c015aef0 con 0x7f58ac0047c0 2023-10-16T15:55:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac00b980 con 0x7f58c0159950 2023-10-16T15:55:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.059+0000 7f58c5952640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ac003970 0x7f58ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c5952640 1 -- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 msgr2=0x7f58ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c6153640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c5151640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac0047c0 0x7f58ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c5952640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c5952640 1 -- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac0047c0 msgr2=0x7f58ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c5952640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac0047c0 0x7f58ac004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c5952640 1 -- 172.21.15.29:0/3112832067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac003520 con 0x7f58ac003970 2023-10-16T15:55:34.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.060+0000 7f58c6153640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c0159950 0x7f58ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.061+0000 7f58c5952640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ac003970 0x7f58ac003d90 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009af0 tx=0x7f58ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954f8f98df7d31ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.061+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc02c6b0 con 0x7f58ac003970 2023-10-16T15:55:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.061+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc02c810 con 0x7f58ac003970 2023-10-16T15:55:34.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.061+0000 7f58c4950640 1 -- 172.21.15.29:0/3112832067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc02cba0 con 0x7f58ac003970 2023-10-16T15:55:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6954640 1 -- 172.21.15.29:0/3112832067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ac003970 msgr2=0x7f58ac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6954640 1 --2- 172.21.15.29:0/3112832067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ac003970 0x7f58ac003d90 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009af0 tx=0x7f58ac0057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6153640 1 -- 172.21.15.29:0/3112832067 reap_dead start 2023-10-16T15:55:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6954640 1 -- 172.21.15.29:0/3112832067 shutdown_connections 2023-10-16T15:55:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6954640 1 -- 172.21.15.29:0/3112832067 >> 172.21.15.29:0/3112832067 conn(0x7f58c00ce6a0 msgr2=0x7f58c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:34.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6954640 1 -- 172.21.15.29:0/3112832067 shutdown_connections 2023-10-16T15:55:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.062+0000 7f58c6954640 1 -- 172.21.15.29:0/3112832067 wait complete. 2023-10-16T15:55:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.064+0000 7f58c6954640 1 Processor -- start 2023-10-16T15:55:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.064+0000 7f58c6954640 1 -- start start 2023-10-16T15:55:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.065+0000 7f58c6954640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac003970 0x7f58c0153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.065+0000 7f58c6954640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 0x7f58c00c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.065+0000 7f58c6954640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c00c27e0 0x7f58c00c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.065+0000 7f58c6954640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c01543c0 con 0x7f58c00c2360 2023-10-16T15:55:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.065+0000 7f58c6954640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c0157bd0 con 0x7f58ac003970 2023-10-16T15:55:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.065+0000 7f58c6954640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c0157eb0 con 0x7f58c00c27e0 2023-10-16T15:55:34.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 0x7f58c00c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 0x7f58c00c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38134/0 (socket says 172.21.15.29:38134) 2023-10-16T15:55:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 -- 172.21.15.29:0/1316209295 learned_addr learned my addr 172.21.15.29:0/1316209295 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5952640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac003970 0x7f58c0153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c6153640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c00c27e0 0x7f58c00c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 -- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c00c27e0 msgr2=0x7f58c00c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c00c27e0 0x7f58c00c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 -- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac003970 msgr2=0x7f58c0153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac003970 0x7f58c0153e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 -- 172.21.15.29:0/1316209295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c0158190 con 0x7f58c00c2360 2023-10-16T15:55:34.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c6153640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c00c27e0 0x7f58c00c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.066+0000 7f58c5151640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 0x7f58c00c7940 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f58b00108e0 tx=0x7f58b0010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.067+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b000eb60 con 0x7f58c00c2360 2023-10-16T15:55:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.067+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b000ecc0 con 0x7f58c00c2360 2023-10-16T15:55:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.067+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b001c400 con 0x7f58c00c2360 2023-10-16T15:55:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.067+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c0158470 con 0x7f58c00c2360 2023-10-16T15:55:34.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.067+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c00c1e80 con 0x7f58c00c2360 2023-10-16T15:55:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.068+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c00c2070 con 0x7f58c00c2360 2023-10-16T15:55:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.069+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f58b001c560 con 0x7f58c00c2360 2023-10-16T15:55:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.070+0000 7f58b67fc640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f58a406b050 0x7f58a406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.070+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f58b008a070 con 0x7f58c00c2360 2023-10-16T15:55:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.070+0000 7f58c5952640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f58a406b050 0x7f58a406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.072+0000 7f58c5952640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f58a406b050 0x7f58a406d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58bc036000 tx=0x7f58bc030000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:34.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.074+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58b00bcca0 con 0x7f58c00c2360 2023-10-16T15:55:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.316+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58c004e9b0 con 0x7f58c00c2360 2023-10-16T15:55:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.317+0000 7f58b67fc640 1 -- 172.21.15.29:0/1316209295 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+3706 (secure 0 0 0) 0x7f58b0021e90 con 0x7f58c00c2360 2023-10-16T15:55:34.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:34.320 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8249,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2114997655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6835","nonce":2114997655},{"type":"v1","addr":"172.21.15.170:6837","nonce":2114997655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":9241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6836/140566426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":140566426},{"type":"v1","addr":"172.21.15.170:6836","nonce":140566426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":12150,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/356635286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":356635286},{"type":"v1","addr":"172.21.15.29:6835","nonce":356635286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":12243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/1110446337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1110446337},{"type":"v1","addr":"172.21.15.29:6834","nonce":1110446337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[]} 2023-10-16T15:55:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 134 2023-10-16T15:55:34.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f58a406b050 msgr2=0x7f58a406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f58a406b050 0x7f58a406d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58bc036000 tx=0x7f58bc030000 comp rx=0 tx=0).stop 2023-10-16T15:55:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 msgr2=0x7f58c00c7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 0x7f58c00c7940 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f58b00108e0 tx=0x7f58b0010da0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 shutdown_connections 2023-10-16T15:55:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f58a406b050 0x7f58a406d510 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58c00c2360 0x7f58c00c7940 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f58c00c27e0 0x7f58c00c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 --2- 172.21.15.29:0/1316209295 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f58ac003970 0x7f58c0153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 >> 172.21.15.29:0/1316209295 conn(0x7f58c00ce6a0 msgr2=0x7f58c00b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 shutdown_connections 2023-10-16T15:55:34.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.320+0000 7f58c6954640 1 -- 172.21.15.29:0/1316209295 wait complete. 2023-10-16T15:55:34.343 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T15:55:34.344 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T15:55:34.344 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:55:34.344 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:55:34.345 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T15:55:34.345 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T15:55:34.345 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:55:34.345 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:55:34.346 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T15:55:34.346 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T15:55:34.346 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:55:34.347 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:55:34.347 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T15:55:34.347 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T15:55:34.347 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:55:34.348 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:55:34.364 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:55:34.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T15:55:34.366 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:55:34.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T15:55:34.374 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:55:34.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T15:55:34.375 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:55:34.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T15:55:34.414 INFO:tasks.ceph.mds.a:Started 2023-10-16T15:55:34.414 DEBUG:teuthology.parallel:result is None 2023-10-16T15:55:34.421 INFO:tasks.ceph.mds.c:Started 2023-10-16T15:55:34.421 DEBUG:teuthology.parallel:result is None 2023-10-16T15:55:34.423 INFO:tasks.ceph.mds.b:Started 2023-10-16T15:55:34.424 DEBUG:teuthology.parallel:result is None 2023-10-16T15:55:34.430 INFO:tasks.ceph.mds.d:Started 2023-10-16T15:55:34.431 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:34.424+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:34.431 DEBUG:teuthology.parallel:result is None 2023-10-16T15:55:34.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T15:55:34.588 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-16T15:55:34.589 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:55:34.589 INFO:tasks.ceph.mds.b.smithi170.stdout:starting mds.b at 2023-10-16T15:55:34.598 INFO:tasks.ceph.mds.d.smithi170.stdout:starting mds.d at 2023-10-16T15:55:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.629+0000 7f230d67c640 1 Processor -- start 2023-10-16T15:55:34.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.629+0000 7f230d67c640 1 -- start start 2023-10-16T15:55:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.630+0000 7f230d67c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f23080caa80 0x7f2308069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.630+0000 7f230d67c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080caef0 con 0x7f230806a2e0 2023-10-16T15:55:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f230d67c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230806a6f0 con 0x7f23080addb0 2023-10-16T15:55:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f230d67c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2308161a50 con 0x7f23080caa80 2023-10-16T15:55:34.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23067fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f23080caa80 0x7f2308069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23077fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f230806a2e0 0x7f23081650d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50336/0 (socket says 172.21.15.29:50336) 2023-10-16T15:55:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23067fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f23080caa80 0x7f2308069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38184/0 (socket says 172.21.15.29:38184) 2023-10-16T15:55:34.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23077fe640 1 -- 172.21.15.29:0/3868886585 learned_addr learned my addr 172.21.15.29:0/3868886585 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23067fc640 1 -- 172.21.15.29:0/3868886585 >> v1:172.21.15.170:6789/0 conn(0x7f23080addb0 legacy=0x7f23080c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23067fc640 1 -- 172.21.15.29:0/3868886585 >> v1:172.21.15.29:6789/0 conn(0x7f230806a2e0 legacy=0x7f23081650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f2305ffb640 1 -- 172.21.15.29:0/3868886585 <== mon.1 v1:172.21.15.170:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990368166 0 0) 0x7f230806a6f0 con 0x7f23080addb0 2023-10-16T15:55:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.631+0000 7f23067fc640 1 -- 172.21.15.29:0/3868886585 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080cb140 con 0x7f23080caa80 2023-10-16T15:55:34.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.632+0000 7f23067fc640 1 --2- 172.21.15.29:0/3868886585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23080caa80 0x7f2308069da0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0097a0 tx=0x7f22fc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b1ae1ff37b769 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:34.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.633+0000 7f2305ffb640 1 -- 172.21.15.29:0/3868886585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc030710 con 0x7f23080caa80 2023-10-16T15:55:34.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.633+0000 7f2305ffb640 1 -- 172.21.15.29:0/3868886585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc02e050 con 0x7f23080caa80 2023-10-16T15:55:34.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.633+0000 7f2305ffb640 1 -- 172.21.15.29:0/3868886585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc030a90 con 0x7f23080caa80 2023-10-16T15:55:34.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.633+0000 7f230d67c640 1 -- 172.21.15.29:0/3868886585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23080caa80 msgr2=0x7f2308069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.633+0000 7f230d67c640 1 --2- 172.21.15.29:0/3868886585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23080caa80 0x7f2308069da0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0097a0 tx=0x7f22fc02b790 comp rx=0 tx=0).stop 2023-10-16T15:55:34.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.634+0000 7f230d67c640 1 -- 172.21.15.29:0/3868886585 shutdown_connections 2023-10-16T15:55:34.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.634+0000 7f230d67c640 1 --2- 172.21.15.29:0/3868886585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23080caa80 0x7f2308069da0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.634+0000 7f230d67c640 1 -- 172.21.15.29:0/3868886585 >> 172.21.15.29:0/3868886585 conn(0x7f23080ce9b0 msgr2=0x7f23080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:34.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.634+0000 7f230d67c640 1 -- 172.21.15.29:0/3868886585 shutdown_connections 2023-10-16T15:55:34.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.634+0000 7f230d67c640 1 -- 172.21.15.29:0/3868886585 wait complete. 2023-10-16T15:55:34.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.635+0000 7f230d67c640 1 Processor -- start 2023-10-16T15:55:34.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.635+0000 7f230d67c640 1 -- start start 2023-10-16T15:55:34.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f230d67c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 0x7f23080b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f230d67c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23080addb0 0x7f23080b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f230d67c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23080caa80 0x7f230815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f230d67c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230815ba50 con 0x7f230806a2e0 2023-10-16T15:55:34.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f230d67c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230815e2c0 con 0x7f23080caa80 2023-10-16T15:55:34.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f230d67c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23080b1b80 con 0x7f23080addb0 2023-10-16T15:55:34.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 0x7f23080b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 0x7f23080b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38186/0 (socket says 172.21.15.29:38186) 2023-10-16T15:55:34.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 -- 172.21.15.29:0/2231049936 learned_addr learned my addr 172.21.15.29:0/2231049936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:34.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f23077fe640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23080caa80 0x7f230815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 -- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23080addb0 msgr2=0x7f23080b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23080addb0 0x7f23080b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 -- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23080caa80 msgr2=0x7f230815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:34.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23080caa80 0x7f230815b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:34.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.637+0000 7f2306ffd640 1 -- 172.21.15.29:0/2231049936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc009400 con 0x7f230806a2e0 2023-10-16T15:55:34.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.638+0000 7f2306ffd640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 0x7f23080b15a0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c4b0 tx=0x7f22f800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:34.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.638+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8005480 con 0x7f230806a2e0 2023-10-16T15:55:34.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.638+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f80035f0 con 0x7f230806a2e0 2023-10-16T15:55:34.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.638+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80039e0 con 0x7f230806a2e0 2023-10-16T15:55:34.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.638+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23080b1e60 con 0x7f230806a2e0 2023-10-16T15:55:34.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.638+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23080b2420 con 0x7f230806a2e0 2023-10-16T15:55:34.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.639+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23080b2610 con 0x7f230806a2e0 2023-10-16T15:55:34.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.641+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f22f8003b40 con 0x7f230806a2e0 2023-10-16T15:55:34.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.641+0000 7f22ef7fe640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f22f406b020 0x7f22f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:34.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.642+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f22f8088f40 con 0x7f230806a2e0 2023-10-16T15:55:34.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.642+0000 7f23067fc640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f22f406b020 0x7f22f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:34.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.643+0000 7f23067fc640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f22f406b020 0x7f22f406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22fc034000 tx=0x7f22fc02d000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:34.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.646+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f22f8053d50 con 0x7f230806a2e0 2023-10-16T15:55:34.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:34.864+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f230804e160 con 0x7f230806a2e0 2023-10-16T15:55:35.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.316+0000 7f22ef7fe640 1 -- 172.21.15.29:0/2231049936 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v260) v1 ==== 97+0+0 (secure 0 0 0) 0x7f22f8017070 con 0x7f230806a2e0 2023-10-16T15:55:35.318 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-16T15:55:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.319+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f22f406b020 msgr2=0x7f22f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.319+0000 7f230d67c640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f22f406b020 0x7f22f406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22fc034000 tx=0x7f22fc02d000 comp rx=0 tx=0).stop 2023-10-16T15:55:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.319+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 msgr2=0x7f23080b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.319+0000 7f230d67c640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 0x7f23080b15a0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c4b0 tx=0x7f22f800c970 comp rx=0 tx=0).stop 2023-10-16T15:55:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 shutdown_connections 2023-10-16T15:55:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f22f406b020 0x7f22f406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f23080caa80 0x7f230815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f23080addb0 0x7f23080b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 --2- 172.21.15.29:0/2231049936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f230806a2e0 0x7f23080b15a0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 >> 172.21.15.29:0/2231049936 conn(0x7f23080ce9b0 msgr2=0x7f2308072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 shutdown_connections 2023-10-16T15:55:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.320+0000 7f230d67c640 1 -- 172.21.15.29:0/2231049936 wait complete. 2023-10-16T15:55:35.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T15:55:35.477 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:35.474+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.568+0000 7f8ebe226640 1 Processor -- start 2023-10-16T15:55:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.568+0000 7f8ebe226640 1 -- start start 2023-10-16T15:55:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8ebe226640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8eb815aaa0 0x7f8eb815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8ebe226640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb815ae80 0x7f8eb80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8ebe226640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8eb80ccaa0 0x7f8eb8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8ebe226640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb806b810 con 0x7f8eb80ccaa0 2023-10-16T15:55:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8ebe226640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb815ed60 con 0x7f8eb815aaa0 2023-10-16T15:55:35.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8ebe226640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80707e0 con 0x7f8eb815ae80 2023-10-16T15:55:35.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.570+0000 7f8eb7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8eb80ccaa0 0x7f8eb8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8eb80ccaa0 0x7f8eb8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38198/0 (socket says 172.21.15.29:38198) 2023-10-16T15:55:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 -- 172.21.15.29:0/2580800088 learned_addr learned my addr 172.21.15.29:0/2580800088 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eaffff640 1 --2- 172.21.15.29:0/2580800088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb815ae80 0x7f8eb80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb77fe640 1 --2- 172.21.15.29:0/2580800088 >> v2:172.21.15.170:3300/0 conn(0x7f8eb815aaa0 0x7f8eb815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 -- 172.21.15.29:0/2580800088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb815ae80 msgr2=0x7f8eb80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:35.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 --2- 172.21.15.29:0/2580800088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb815ae80 0x7f8eb80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 -- 172.21.15.29:0/2580800088 >> v2:172.21.15.170:3300/0 conn(0x7f8eb815aaa0 msgr2=0x7f8eb815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 --2- 172.21.15.29:0/2580800088 >> v2:172.21.15.170:3300/0 conn(0x7f8eb815aaa0 0x7f8eb815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.571+0000 7f8eb7fff640 1 -- 172.21.15.29:0/2580800088 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb815ef90 con 0x7f8eb80ccaa0 2023-10-16T15:55:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.572+0000 7f8eb7fff640 1 --2- 172.21.15.29:0/2580800088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80ccaa0 0x7f8eb8070f50 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80cd990 tx=0x7f8eb002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02fa5472f5bcd06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.573+0000 7f8eb6ffd640 1 -- 172.21.15.29:0/2580800088 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb002ca80 con 0x7f8eb80ccaa0 2023-10-16T15:55:35.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.573+0000 7f8eb6ffd640 1 -- 172.21.15.29:0/2580800088 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb002cbe0 con 0x7f8eb80ccaa0 2023-10-16T15:55:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.573+0000 7f8eb6ffd640 1 -- 172.21.15.29:0/2580800088 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb00343f0 con 0x7f8eb80ccaa0 2023-10-16T15:55:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 -- 172.21.15.29:0/2580800088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80ccaa0 msgr2=0x7f8eb8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 --2- 172.21.15.29:0/2580800088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80ccaa0 0x7f8eb8070f50 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80cd990 tx=0x7f8eb002bee0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 -- 172.21.15.29:0/2580800088 shutdown_connections 2023-10-16T15:55:35.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 --2- 172.21.15.29:0/2580800088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb80ccaa0 0x7f8eb8070f50 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 --2- 172.21.15.29:0/2580800088 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb815ae80 0x7f8eb80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 --2- 172.21.15.29:0/2580800088 >> v2:172.21.15.170:3300/0 conn(0x7f8eb815aaa0 0x7f8eb815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 -- 172.21.15.29:0/2580800088 >> 172.21.15.29:0/2580800088 conn(0x7f8eb80cb410 msgr2=0x7f8eb80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 -- 172.21.15.29:0/2580800088 shutdown_connections 2023-10-16T15:55:35.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.574+0000 7f8ebe226640 1 -- 172.21.15.29:0/2580800088 wait complete. 2023-10-16T15:55:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.576+0000 7f8ebe226640 1 Processor -- start 2023-10-16T15:55:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.576+0000 7f8ebe226640 1 -- start start 2023-10-16T15:55:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8ebe226640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80ccaa0 0x7f8eb81ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8ebe226640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eb815aaa0 0x7f8eb81eedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8ebe226640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 0x7f8eb815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8ebe226640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb81ee370 con 0x7f8eb815ae80 2023-10-16T15:55:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8ebe226640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815c3f0 con 0x7f8eb815aaa0 2023-10-16T15:55:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8ebe226640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815c6d0 con 0x7f8eb80ccaa0 2023-10-16T15:55:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8eb7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 0x7f8eb815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8eb7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 0x7f8eb815b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38212/0 (socket says 172.21.15.29:38212) 2023-10-16T15:55:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8eb7fff640 1 -- 172.21.15.29:0/1003889670 learned_addr learned my addr 172.21.15.29:0/1003889670 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8eb77fe640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80ccaa0 0x7f8eb81ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.577+0000 7f8eaffff640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eb815aaa0 0x7f8eb81eedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb7fff640 1 -- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80ccaa0 msgr2=0x7f8eb81ede30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb7fff640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80ccaa0 0x7f8eb81ede30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb7fff640 1 -- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eb815aaa0 msgr2=0x7f8eb81eedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb7fff640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eb815aaa0 0x7f8eb81eedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb7fff640 1 -- 172.21.15.29:0/1003889670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb0009b30 con 0x7f8eb815ae80 2023-10-16T15:55:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eaffff640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eb815aaa0 0x7f8eb81eedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb7fff640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 0x7f8eb815b2e0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0036040 tx=0x7f8eb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0009510 con 0x7f8eb815ae80 2023-10-16T15:55:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb003e450 con 0x7f8eb815ae80 2023-10-16T15:55:35.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb003e800 con 0x7f8eb815ae80 2023-10-16T15:55:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.578+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb81ee5c0 con 0x7f8eb815ae80 2023-10-16T15:55:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.579+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb815c9b0 con 0x7f8eb815ae80 2023-10-16T15:55:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.580+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb80cbd20 con 0x7f8eb815ae80 2023-10-16T15:55:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.581+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f8eb002cd50 con 0x7f8eb815ae80 2023-10-16T15:55:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.581+0000 7f8eb4ff9640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8e9806b070 0x7f8e9806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.582+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f8eb00abf70 con 0x7f8eb815ae80 2023-10-16T15:55:35.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.582+0000 7f8eb77fe640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8e9806b070 0x7f8e9806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.583+0000 7f8eb77fe640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8e9806b070 0x7f8e9806d530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8006fd0 tx=0x7f8ea8008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.586+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8eb00deca0 con 0x7f8eb815ae80 2023-10-16T15:55:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.861+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8eb81eea80 con 0x7f8eb815ae80 2023-10-16T15:55:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.862+0000 7f8eb4ff9640 1 -- 172.21.15.29:0/1003889670 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8eb00771b0 con 0x7f8eb815ae80 2023-10-16T15:55:35.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:35.865 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:55:35.865 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-16T15:55:35.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.865 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-16T15:55:35.865 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1l+LurHRAAw3QsmC/SWuDuQkzQKqKIrg==", 2023-10-16T15:55:35.866 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.866 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.866 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:55:35.866 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:55:35.866 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.866 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.867 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-16T15:55:35.867 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lE99JBhAAM04DfKp3Jd8hROGJIpCtug==", 2023-10-16T15:55:35.867 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.868 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.868 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:55:35.868 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:55:35.868 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.868 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.869 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-16T15:55:35.869 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lVsZRMBAAykBCXHPJo3VmsH1m+53n+g==", 2023-10-16T15:55:35.869 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.870 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.870 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:55:35.870 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:55:35.870 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.870 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.871 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-16T15:55:35.871 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1lQ7S6FxAANi1pR3KwwOXKfmPBwzWJMQ==", 2023-10-16T15:55:35.871 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.872 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:55:35.872 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:55:35.872 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.873 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-16T15:55:35.873 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB4RS1laLqVAxAAe37Fc5kc73ntOF4GPX5IOA==", 2023-10-16T15:55:35.873 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.873 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.873 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.874 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.874 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.874 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-16T15:55:35.874 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB7RS1lAHisHRAAat3V5CiniY9VZoOreAjs7w==", 2023-10-16T15:55:35.875 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.875 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.875 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.876 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-16T15:55:35.876 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB+RS1lg59fNxAAVDz9mGMnVrXFd/EHlrrTrw==", 2023-10-16T15:55:35.876 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.876 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.877 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.877 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.877 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-16T15:55:35.878 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCCRS1lbseeFRAAMMDh5/zcTez7PMuLSdk9uw==", 2023-10-16T15:55:35.878 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.878 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.878 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.878 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.879 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.879 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-16T15:55:35.879 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCHRS1lX6A9JhAAIf+Us+sxNkb43jiUoxsy6g==", 2023-10-16T15:55:35.880 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.880 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.880 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.880 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.880 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.881 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-16T15:55:35.881 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCKRS1l9XSLLxAAI2qQmrtuZEn3IIqWxcynCw==", 2023-10-16T15:55:35.881 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.882 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.882 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.882 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.883 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-16T15:55:35.883 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCORS1lhZOTDBAAkRA74A/OfF8UFNKBghqfZw==", 2023-10-16T15:55:35.883 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.883 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.883 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.883 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-16T15:55:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQCRRS1lQNPrJBAAsYKICxZz92uChdYO1Ckg7g==", 2023-10-16T15:55:35.884 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.885 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-16T15:55:35.885 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-16T15:55:35.885 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.885 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.886 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-16T15:55:35.886 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA==", 2023-10-16T15:55:35.886 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.886 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.887 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-16T15:55:35.887 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T15:55:35.887 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.887 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-16T15:55:35.888 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lx2fKAhAAk7Z3D5f8aHm9AB6CXmwEIw==", 2023-10-16T15:55:35.888 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.888 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.888 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:55:35.888 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:55:35.888 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:55:35.889 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.889 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-16T15:55:35.889 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB1RS1l9qlKMhAAQVVvbNLnBQYjZTy5XU4rcQ==", 2023-10-16T15:55:35.890 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.890 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.890 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:55:35.890 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:55:35.890 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:55:35.890 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.891 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-16T15:55:35.891 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB2RS1lh8krDhAAyKKFf+d7fkcMx4ipBielWg==", 2023-10-16T15:55:35.891 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.892 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-16T15:55:35.892 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-16T15:55:35.892 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-16T15:55:35.892 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-16T15:55:35.892 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.893 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-16T15:55:35.893 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBzRS1lAZeUDBAAFuCZWXBxIu3XOUBBVFwdYA==", 2023-10-16T15:55:35.893 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.893 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:55:35.893 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-16T15:55:35.894 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-16T15:55:35.894 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.894 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.894 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-16T15:55:35.895 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lKGqyEBAAgta5aOSArYwE/WnyqeUYJw==", 2023-10-16T15:55:35.895 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.895 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:55:35.895 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:55:35.895 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.895 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-16T15:55:35.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:35.896 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-16T15:55:35.896 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQB0RS1lu7X1AxAA8YP/AL9JqMzQfq3SQ5zMYA==", 2023-10-16T15:55:35.896 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-16T15:55:35.897 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-16T15:55:35.897 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-16T15:55:35.897 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-16T15:55:35.897 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.897 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:35.897 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-16T15:55:35.898 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:55:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8e9806b070 msgr2=0x7f8e9806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8e9806b070 0x7f8e9806d530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8006fd0 tx=0x7f8ea8008040 comp rx=0 tx=0).stop 2023-10-16T15:55:35.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 msgr2=0x7f8eb815b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 0x7f8eb815b2e0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0036040 tx=0x7f8eb0004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 shutdown_connections 2023-10-16T15:55:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8e9806b070 0x7f8e9806d530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb815ae80 0x7f8eb815b2e0 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8eb815aaa0 0x7f8eb81eedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.865+0000 7f8ebe226640 1 --2- 172.21.15.29:0/1003889670 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8eb80ccaa0 0x7f8eb81ede30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.866+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 >> 172.21.15.29:0/1003889670 conn(0x7f8eb80cb410 msgr2=0x7f8eb8070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.866+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 shutdown_connections 2023-10-16T15:55:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:35.866+0000 7f8ebe226640 1 -- 172.21.15.29:0/1003889670 wait complete. 2023-10-16T15:55:35.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:36.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.135+0000 7f7351b63640 1 Processor -- start 2023-10-16T15:55:36.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.135+0000 7f7351b63640 1 -- start start 2023-10-16T15:55:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7351b63640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f734c0ad970 0x7f734c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7351b63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7351b63640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c15b9d0 con 0x7f734c06c040 2023-10-16T15:55:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7351b63640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734c15bcb0 con 0x7f734c0ad970 2023-10-16T15:55:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7351b63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c15bf90 con 0x7f734c0ccfb0 2023-10-16T15:55:36.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f734bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f734c0ad970 0x7f734c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7350b61640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f734c06c040 0x7f734c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50338/0 (socket says 172.21.15.29:50338) 2023-10-16T15:55:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f734bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f734c0ad970 0x7f734c0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38216/0 (socket says 172.21.15.29:38216) 2023-10-16T15:55:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.137+0000 7f7350b61640 1 -- 172.21.15.29:0/95437744 learned_addr learned my addr 172.21.15.29:0/95437744 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.138+0000 7f7351362640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.138+0000 7f734bfff640 1 -- 172.21.15.29:0/95437744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 msgr2=0x7f734c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.138+0000 7f734bfff640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.138+0000 7f734bfff640 1 -- 172.21.15.29:0/95437744 >> v1:172.21.15.29:6789/0 conn(0x7f734c06c040 legacy=0x7f734c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.138+0000 7f734bfff640 1 -- 172.21.15.29:0/95437744 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0cc270 con 0x7f734c0ad970 2023-10-16T15:55:36.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.138+0000 7f7351362640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:36.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.139+0000 7f734bfff640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0cca70 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f73380097a0 tx=0x7f733802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5ad7359a65417c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:36.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f734b7fe640 1 -- 172.21.15.29:0/95437744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733802d040 con 0x7f734c0ad970 2023-10-16T15:55:36.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f734b7fe640 1 -- 172.21.15.29:0/95437744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7338031810 con 0x7f734c0ad970 2023-10-16T15:55:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f734b7fe640 1 -- 172.21.15.29:0/95437744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338031ba0 con 0x7f734c0ad970 2023-10-16T15:55:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 -- 172.21.15.29:0/95437744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 msgr2=0x7f734c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0cca70 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f73380097a0 tx=0x7f733802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 -- 172.21.15.29:0/95437744 shutdown_connections 2023-10-16T15:55:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 --2- 172.21.15.29:0/95437744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0cca70 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 -- 172.21.15.29:0/95437744 >> 172.21.15.29:0/95437744 conn(0x7f734c0ce6a0 msgr2=0x7f734c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 -- 172.21.15.29:0/95437744 shutdown_connections 2023-10-16T15:55:36.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.140+0000 7f7351b63640 1 -- 172.21.15.29:0/95437744 wait complete. 2023-10-16T15:55:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.142+0000 7f7351b63640 1 Processor -- start 2023-10-16T15:55:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.142+0000 7f7351b63640 1 -- start start 2023-10-16T15:55:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.143+0000 7f7351b63640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c06c040 0x7f734c1e6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.143+0000 7f7351b63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.143+0000 7f7351b63640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.143+0000 7f7351b63640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0c66c0 con 0x7f734c0ad970 2023-10-16T15:55:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.143+0000 7f7351b63640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0c68f0 con 0x7f734c06c040 2023-10-16T15:55:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.143+0000 7f7351b63640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c0c6bd0 con 0x7f734c0ccfb0 2023-10-16T15:55:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38224/0 (socket says 172.21.15.29:38224) 2023-10-16T15:55:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 -- 172.21.15.29:0/2769729367 learned_addr learned my addr 172.21.15.29:0/2769729367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f7351362640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f7350b61640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c06c040 0x7f734c1e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 -- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 msgr2=0x7f734c153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 -- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c06c040 msgr2=0x7f734c1e6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c06c040 0x7f734c1e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.144+0000 7f734bfff640 1 -- 172.21.15.29:0/2769729367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7338009400 con 0x7f734c0ad970 2023-10-16T15:55:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.145+0000 7f734bfff640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0c6180 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f73380096f0 tx=0x7f73380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.145+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733803ad60 con 0x7f734c0ad970 2023-10-16T15:55:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.145+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7338044400 con 0x7f734c0ad970 2023-10-16T15:55:36.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.145+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73380447b0 con 0x7f734c0ad970 2023-10-16T15:55:36.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.145+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c0c6eb0 con 0x7f734c0ad970 2023-10-16T15:55:36.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.145+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c0c7470 con 0x7f734c0ad970 2023-10-16T15:55:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.146+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c06b330 con 0x7f734c0ad970 2023-10-16T15:55:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.147+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f733804d440 con 0x7f734c0ad970 2023-10-16T15:55:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.148+0000 7f73497fa640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f733006b020 0x7f733006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.148+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f73380abbb0 con 0x7f734c0ad970 2023-10-16T15:55:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.149+0000 7f7350b61640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f733006b020 0x7f733006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.150+0000 7f7350b61640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f733006b020 0x7f733006d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7340005880 tx=0x7f7340009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.153+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7338076bf0 con 0x7f734c0ad970 2023-10-16T15:55:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.356+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f734c06ad20 con 0x7f734c0ad970 2023-10-16T15:55:36.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.359+0000 7f73497fa640 1 -- 172.21.15.29:0/2769729367 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f733807a210 con 0x7f734c0ad970 2023-10-16T15:55:36.361 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:36.361 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":262,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:35.333898+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f733006b020 msgr2=0x7f733006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f733006b020 0x7f733006d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7340005880 tx=0x7f7340009040 comp rx=0 tx=0).stop 2023-10-16T15:55:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 msgr2=0x7f734c0c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0c6180 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f73380096f0 tx=0x7f73380046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 shutdown_connections 2023-10-16T15:55:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f733006b020 0x7f733006d4e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f734c0ad970 0x7f734c0c6180 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f734c0ccfb0 0x7f734c153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 --2- 172.21.15.29:0/2769729367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f734c06c040 0x7f734c1e6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.361+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 >> 172.21.15.29:0/2769729367 conn(0x7f734c0ce6a0 msgr2=0x7f734c159660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.362+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 shutdown_connections 2023-10-16T15:55:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.362+0000 7f7351b63640 1 -- 172.21.15.29:0/2769729367 wait complete. 2023-10-16T15:55:36.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:55:36.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:55:36.572 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:36.570+0000 7f484e3fb640 -1 client.9532: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8402:32c40b6c-2dc2-44a4-89e8-25173c4067bc:v1:172.21.15.170:0/3985010689] 2023-10-16T15:55:36.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.600+0000 7fa3625f4640 1 Processor -- start 2023-10-16T15:55:36.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.601+0000 7fa3625f4640 1 -- start start 2023-10-16T15:55:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.602+0000 7fa3625f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 0x7fa35c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.602+0000 7fa3625f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0d0970 con 0x7fa35c0b1ca0 2023-10-16T15:55:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.602+0000 7fa3625f4640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0d0ea0 con 0x7fa35c0b2080 2023-10-16T15:55:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.602+0000 7fa3625f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0d0bc0 con 0x7fa35c0cfc40 2023-10-16T15:55:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa3615f2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa35c0b1ca0 0x7fa35c0ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50354/0 (socket says 172.21.15.29:50354) 2023-10-16T15:55:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa3615f2640 1 -- 172.21.15.29:0/64473977 learned_addr learned my addr 172.21.15.29:0/64473977 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa361df3640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 0x7fa35c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585785718 0 0) 0x7fa35c0d0970 con 0x7fa35c0b1ca0 2023-10-16T15:55:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa348003520 con 0x7fa35c0b1ca0 2023-10-16T15:55:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260170802 0 0) 0x7fa348003520 con 0x7fa35c0b1ca0 2023-10-16T15:55:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 msgr2=0x7fa35c0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 0x7fa35c0d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 >> v1:172.21.15.170:6789/0 conn(0x7fa35c0b2080 legacy=0x7fa35c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.603+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0cd940 con 0x7fa35c0b1ca0 2023-10-16T15:55:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa3580019f0 con 0x7fa35c0b1ca0 2023-10-16T15:55:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 >> v1:172.21.15.29:6789/0 conn(0x7fa35c0b1ca0 legacy=0x7fa35c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa348003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3480047c0 0x7fa348004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 0x7fa348005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0d0bc0 con 0x7fa3480039a0 2023-10-16T15:55:36.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34800bb20 con 0x7fa3480047c0 2023-10-16T15:55:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34800bce0 con 0x7fa35c0cfc40 2023-10-16T15:55:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa3615f2640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa348003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa360df1640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3480047c0 0x7fa348004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa361df3640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 0x7fa348005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa3615f2640 1 -- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 msgr2=0x7fa348005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa3615f2640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cfc40 0x7fa348005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa3615f2640 1 -- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3480047c0 msgr2=0x7fa348004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa3615f2640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa3480047c0 0x7fa348004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.604+0000 7fa3615f2640 1 -- 172.21.15.29:0/64473977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa348003520 con 0x7fa3480039a0 2023-10-16T15:55:36.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa3615f2640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa348003dc0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fa358002230 tx=0x7fa3480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade2503b927c6340 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358034630 con 0x7fa3480039a0 2023-10-16T15:55:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35802f070 con 0x7fa3480039a0 2023-10-16T15:55:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa353fff640 1 -- 172.21.15.29:0/64473977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3580349b0 con 0x7fa3480039a0 2023-10-16T15:55:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa3625f4640 1 -- 172.21.15.29:0/64473977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 msgr2=0x7fa348003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa3625f4640 1 --2- 172.21.15.29:0/64473977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa348003dc0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fa358002230 tx=0x7fa3480057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.606+0000 7fa361df3640 1 -- 172.21.15.29:0/64473977 reap_dead start 2023-10-16T15:55:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.607+0000 7fa3625f4640 1 -- 172.21.15.29:0/64473977 shutdown_connections 2023-10-16T15:55:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.607+0000 7fa3625f4640 1 -- 172.21.15.29:0/64473977 >> 172.21.15.29:0/64473977 conn(0x7fa35c0d2b90 msgr2=0x7fa35c0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.607+0000 7fa3625f4640 1 -- 172.21.15.29:0/64473977 shutdown_connections 2023-10-16T15:55:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.607+0000 7fa3625f4640 1 -- 172.21.15.29:0/64473977 wait complete. 2023-10-16T15:55:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.608+0000 7fa3625f4640 1 Processor -- start 2023-10-16T15:55:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.609+0000 7fa3625f4640 1 -- start start 2023-10-16T15:55:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.609+0000 7fa3625f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa35c0c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3625f4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa35c0c97f0 0x7fa35c0afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3625f4640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cc9d0 0x7fa35c0b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3625f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0caa10 con 0x7fa3480039a0 2023-10-16T15:55:36.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3625f4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0c6b30 con 0x7fa35c0c97f0 2023-10-16T15:55:36.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3625f4640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0c6e10 con 0x7fa35c0cc9d0 2023-10-16T15:55:36.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa35c0c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa35c0c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38252/0 (socket says 172.21.15.29:38252) 2023-10-16T15:55:36.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 -- 172.21.15.29:0/877622541 learned_addr learned my addr 172.21.15.29:0/877622541 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:36.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 -- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cc9d0 msgr2=0x7fa35c0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cc9d0 0x7fa35c0b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 -- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa35c0c97f0 msgr2=0x7fa35c0afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa360df1640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa35c0c97f0 0x7fa35c0afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa35c0c97f0 0x7fa35c0afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.610+0000 7fa3615f2640 1 -- 172.21.15.29:0/877622541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0c70f0 con 0x7fa3480039a0 2023-10-16T15:55:36.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.611+0000 7fa3615f2640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa35c0c92b0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fa358002230 tx=0x7fa358001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.611+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358034640 con 0x7fa3480039a0 2023-10-16T15:55:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.611+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3580347a0 con 0x7fa3480039a0 2023-10-16T15:55:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.611+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35c0c73d0 con 0x7fa3480039a0 2023-10-16T15:55:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.611+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358034b50 con 0x7fa3480039a0 2023-10-16T15:55:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.611+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35c0c66c0 con 0x7fa3480039a0 2023-10-16T15:55:36.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.613+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35c158750 con 0x7fa3480039a0 2023-10-16T15:55:36.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.613+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa358031030 con 0x7fa3480039a0 2023-10-16T15:55:36.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.614+0000 7fa351ffb640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa34006b020 0x7fa34006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:36.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.614+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fa3580abd70 con 0x7fa3480039a0 2023-10-16T15:55:36.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.615+0000 7fa360df1640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa34006b020 0x7fa34006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:36.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.616+0000 7fa360df1640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa34006b020 0x7fa34006d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa34c007fd0 tx=0x7fa34c009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:36.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.619+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa35c158750 con 0x7fa3480039a0 2023-10-16T15:55:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.814+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa35c158750 con 0x7fa3480039a0 2023-10-16T15:55:36.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.815+0000 7fa351ffb640 1 -- 172.21.15.29:0/877622541 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v262) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa35c158750 con 0x7fa3480039a0 2023-10-16T15:55:36.817 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:55:36.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa34006b020 msgr2=0x7fa34006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa34006b020 0x7fa34006d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa34c007fd0 tx=0x7fa34c009040 comp rx=0 tx=0).stop 2023-10-16T15:55:36.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 msgr2=0x7fa35c0c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:36.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa35c0c92b0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fa358002230 tx=0x7fa358001cb0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 shutdown_connections 2023-10-16T15:55:36.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa34006b020 0x7fa34006d4e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa35c0cc9d0 0x7fa35c0b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa35c0c97f0 0x7fa35c0afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 --2- 172.21.15.29:0/877622541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3480039a0 0x7fa35c0c92b0 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.817+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 >> 172.21.15.29:0/877622541 conn(0x7fa35c0d2b90 msgr2=0x7fa35c0d1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.818+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 shutdown_connections 2023-10-16T15:55:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:36.818+0000 7fa3625f4640 1 -- 172.21.15.29:0/877622541 wait complete. 2023-10-16T15:55:36.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T15:55:37.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.072+0000 7fd59b59e640 1 Processor -- start 2023-10-16T15:55:37.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.072+0000 7fd59b59e640 1 -- start start 2023-10-16T15:55:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.073+0000 7fd59b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c160c80 0x7fd59c161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.073+0000 7fd59b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd59c0ad7a0 0x7fd59c070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd59b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c0ae1a0 con 0x7fd59c0adc00 2023-10-16T15:55:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd59b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c0ae480 con 0x7fd59c0ad7a0 2023-10-16T15:55:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd59b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0ae760 con 0x7fd59c160c80 2023-10-16T15:55:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd59c0ad7a0 0x7fd59c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd59c0ad7a0 0x7fd59c070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38256/0 (socket says 172.21.15.29:38256) 2023-10-16T15:55:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 -- 172.21.15.29:0/3486877528 learned_addr learned my addr 172.21.15.29:0/3486877528 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 -- 172.21.15.29:0/3486877528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c160c80 msgr2=0x7fd59c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 --2- 172.21.15.29:0/3486877528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c160c80 0x7fd59c161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 -- 172.21.15.29:0/3486877528 >> v1:172.21.15.170:6789/0 conn(0x7fd59c0adc00 legacy=0x7fd59c142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.074+0000 7fd599d9b640 1 -- 172.21.15.29:0/3486877528 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0aea40 con 0x7fd59c0ad7a0 2023-10-16T15:55:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.076+0000 7fd599d9b640 1 --2- 172.21.15.29:0/3486877528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0ad7a0 0x7fd59c070240 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fd584009e50 tx=0x7fd58402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccc5e53475a18f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.076+0000 7fd59959a640 1 -- 172.21.15.29:0/3486877528 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58402ca80 con 0x7fd59c0ad7a0 2023-10-16T15:55:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.076+0000 7fd59959a640 1 -- 172.21.15.29:0/3486877528 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58402cbe0 con 0x7fd59c0ad7a0 2023-10-16T15:55:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.076+0000 7fd59959a640 1 -- 172.21.15.29:0/3486877528 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5840343f0 con 0x7fd59c0ad7a0 2023-10-16T15:55:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 -- 172.21.15.29:0/3486877528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0ad7a0 msgr2=0x7fd59c070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 --2- 172.21.15.29:0/3486877528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0ad7a0 0x7fd59c070240 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fd584009e50 tx=0x7fd58402bee0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 -- 172.21.15.29:0/3486877528 shutdown_connections 2023-10-16T15:55:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 --2- 172.21.15.29:0/3486877528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0ad7a0 0x7fd59c070240 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 --2- 172.21.15.29:0/3486877528 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c160c80 0x7fd59c161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 -- 172.21.15.29:0/3486877528 >> 172.21.15.29:0/3486877528 conn(0x7fd59c0cc230 msgr2=0x7fd59c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 -- 172.21.15.29:0/3486877528 shutdown_connections 2023-10-16T15:55:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.077+0000 7fd59b59e640 1 -- 172.21.15.29:0/3486877528 wait complete. 2023-10-16T15:55:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.079+0000 7fd59b59e640 1 Processor -- start 2023-10-16T15:55:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.079+0000 7fd59b59e640 1 -- start start 2023-10-16T15:55:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd59b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c0ad7a0 0x7fd59c144830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd59b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 0x7fd59c0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd59b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd59c160c80 0x7fd59c0b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd59b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b1790 con 0x7fd59c0adc00 2023-10-16T15:55:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd59b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b2fd0 con 0x7fd59c160c80 2023-10-16T15:55:37.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd59b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b1a10 con 0x7fd59c0ad7a0 2023-10-16T15:55:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 0x7fd59c0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 0x7fd59c0b1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38258/0 (socket says 172.21.15.29:38258) 2023-10-16T15:55:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 -- 172.21.15.29:0/4196652104 learned_addr learned my addr 172.21.15.29:0/4196652104 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 -- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c0ad7a0 msgr2=0x7fd59c144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c0ad7a0 0x7fd59c144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 -- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd59c160c80 msgr2=0x7fd59c0b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd59c160c80 0x7fd59c0b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.080+0000 7fd599d9b640 1 -- 172.21.15.29:0/4196652104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584009b30 con 0x7fd59c0adc00 2023-10-16T15:55:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.081+0000 7fd599d9b640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 0x7fd59c0b1250 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fd584009e50 tx=0x7fd584004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.081+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58403e450 con 0x7fd59c0adc00 2023-10-16T15:55:37.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.081+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58403e5b0 con 0x7fd59c0adc00 2023-10-16T15:55:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.081+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58403e960 con 0x7fd59c0adc00 2023-10-16T15:55:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.081+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c0b1cf0 con 0x7fd59c0adc00 2023-10-16T15:55:37.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.081+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c0b22b0 con 0x7fd59c0adc00 2023-10-16T15:55:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.083+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c04e9b0 con 0x7fd59c0adc00 2023-10-16T15:55:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.084+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fd58402cd50 con 0x7fd59c0adc00 2023-10-16T15:55:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.085+0000 7fd592ffd640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd57c06b020 0x7fd57c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.085+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fd5840ac2a0 con 0x7fd59c0adc00 2023-10-16T15:55:37.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.085+0000 7fd59a59c640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd57c06b020 0x7fd57c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.087+0000 7fd59a59c640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd57c06b020 0x7fd57c06d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009670 tx=0x7fd58c006bf0 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:37.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.090+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd584077400 con 0x7fd59c0adc00 2023-10-16T15:55:37.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.294+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd59c0b5730 con 0x7fd59c0adc00 2023-10-16T15:55:37.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.295+0000 7fd592ffd640 1 -- 172.21.15.29:0/4196652104 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v262) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd58407aa20 con 0x7fd59c0adc00 2023-10-16T15:55:37.297 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:37.297 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:55:37.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:37.298 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-16T15:55:37.298 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-16T15:55:37.298 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:37.298 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:55:37.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd57c06b020 msgr2=0x7fd57c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd57c06b020 0x7fd57c06d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009670 tx=0x7fd58c006bf0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 msgr2=0x7fd59c0b1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 0x7fd59c0b1250 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fd584009e50 tx=0x7fd584004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:37.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 shutdown_connections 2023-10-16T15:55:37.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd57c06b020 0x7fd57c06d4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd59c0adc00 0x7fd59c0b1250 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd59c160c80 0x7fd59c0b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 --2- 172.21.15.29:0/4196652104 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd59c0ad7a0 0x7fd59c144830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 >> 172.21.15.29:0/4196652104 conn(0x7fd59c0cc230 msgr2=0x7fd59c15d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:37.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 shutdown_connections 2023-10-16T15:55:37.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.298+0000 7fd59b59e640 1 -- 172.21.15.29:0/4196652104 wait complete. 2023-10-16T15:55:37.322 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T15:55:37.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T15:55:37.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.559+0000 7fa56ffff640 1 Processor -- start 2023-10-16T15:55:37.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.559+0000 7fa56ffff640 1 -- start start 2023-10-16T15:55:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.561+0000 7fa56ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa5700ac890 0x7fa5700c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.561+0000 7fa56ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700cc480 con 0x7fa5700acc70 2023-10-16T15:55:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.561+0000 7fa56ffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700adfe0 con 0x7fa5700cc910 2023-10-16T15:55:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.561+0000 7fa56ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57015d8c0 con 0x7fa5700ac890 2023-10-16T15:55:37.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.561+0000 7fa567fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa5700acc70 0x7fa57015d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50368/0 (socket says 172.21.15.29:50368) 2023-10-16T15:55:37.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.561+0000 7fa567fff640 1 -- 172.21.15.29:0/1480148041 learned_addr learned my addr 172.21.15.29:0/1480148041 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:37.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56effd640 1 --2- 172.21.15.29:0/1480148041 >> v2:172.21.15.170:3300/0 conn(0x7fa5700ac890 0x7fa5700c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584579520 0 0) 0x7fa5700cc480 con 0x7fa5700acc70 2023-10-16T15:55:37.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c003520 con 0x7fa5700acc70 2023-10-16T15:55:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044210187 0 0) 0x7fa55c003520 con 0x7fa5700acc70 2023-10-16T15:55:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 >> v2:172.21.15.170:3300/0 conn(0x7fa5700ac890 msgr2=0x7fa5700c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 --2- 172.21.15.29:0/1480148041 >> v2:172.21.15.170:3300/0 conn(0x7fa5700ac890 0x7fa5700c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 >> v1:172.21.15.170:6789/0 conn(0x7fa5700cc910 legacy=0x7fa5700ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57015dba0 con 0x7fa5700acc70 2023-10-16T15:55:37.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa558001670 con 0x7fa5700acc70 2023-10-16T15:55:37.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 >> v1:172.21.15.29:6789/0 conn(0x7fa5700acc70 legacy=0x7fa57015d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa55c003a30 0x7fa55c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0047c0 0x7fa55c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa55c0052a0 0x7fa55c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57015d8c0 con 0x7fa55c0052a0 2023-10-16T15:55:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c00c170 con 0x7fa55c003a30 2023-10-16T15:55:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.562+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c00c330 con 0x7fa55c0047c0 2023-10-16T15:55:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa55c0052a0 0x7fa55c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56effd640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa55c003a30 0x7fa55c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa567fff640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0047c0 0x7fa55c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56f7fe640 1 -- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0047c0 msgr2=0x7fa55c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0047c0 0x7fa55c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56f7fe640 1 -- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa55c003a30 msgr2=0x7fa55c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa55c003a30 0x7fa55c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56f7fe640 1 -- 172.21.15.29:0/1480148041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c003520 con 0x7fa55c0052a0 2023-10-16T15:55:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.563+0000 7fa56effd640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa55c003a30 0x7fa55c003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.564+0000 7fa56f7fe640 1 -- 172.21.15.29:0/1480148041 reap_dead start 2023-10-16T15:55:37.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.564+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa55c0052a0 0x7fa55c00b980 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fa568004480 tx=0x7fa55c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae947f1273f5f31d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:37.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.565+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568030400 con 0x7fa55c0052a0 2023-10-16T15:55:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.565+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa568030560 con 0x7fa55c0052a0 2023-10-16T15:55:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.565+0000 7fa56e7fc640 1 -- 172.21.15.29:0/1480148041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5680308f0 con 0x7fa55c0052a0 2023-10-16T15:55:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.565+0000 7fa56ffff640 1 -- 172.21.15.29:0/1480148041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa55c0052a0 msgr2=0x7fa55c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.565+0000 7fa56ffff640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa55c0052a0 0x7fa55c00b980 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fa568004480 tx=0x7fa55c0057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.566+0000 7fa56ffff640 1 -- 172.21.15.29:0/1480148041 shutdown_connections 2023-10-16T15:55:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.566+0000 7fa56ffff640 1 --2- 172.21.15.29:0/1480148041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa55c0052a0 0x7fa55c00b980 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.566+0000 7fa56ffff640 1 -- 172.21.15.29:0/1480148041 >> 172.21.15.29:0/1480148041 conn(0x7fa5700ce820 msgr2=0x7fa5700cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.566+0000 7fa56ffff640 1 -- 172.21.15.29:0/1480148041 shutdown_connections 2023-10-16T15:55:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.566+0000 7fa56ffff640 1 -- 172.21.15.29:0/1480148041 wait complete. 2023-10-16T15:55:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.568+0000 7fa56ffff640 1 Processor -- start 2023-10-16T15:55:37.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.568+0000 7fa56ffff640 1 -- start start 2023-10-16T15:55:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0052a0 0x7fa5701570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5701603e0 0x7fa57006f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 0x7fa570159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57015a690 con 0x7fa57006fc40 2023-10-16T15:55:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700700c0 con 0x7fa5701603e0 2023-10-16T15:55:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa570157630 con 0x7fa55c0052a0 2023-10-16T15:55:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 0x7fa570159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 0x7fa570159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38268/0 (socket says 172.21.15.29:38268) 2023-10-16T15:55:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56f7fe640 1 -- 172.21.15.29:0/2363881269 learned_addr learned my addr 172.21.15.29:0/2363881269 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:37.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa567fff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5701603e0 0x7fa57006f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56effd640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0052a0 0x7fa5701570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56f7fe640 1 -- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0052a0 msgr2=0x7fa5701570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.569+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0052a0 0x7fa5701570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56f7fe640 1 -- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5701603e0 msgr2=0x7fa57006f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:37.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5701603e0 0x7fa57006f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56f7fe640 1 -- 172.21.15.29:0/2363881269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57015bd70 con 0x7fa57006fc40 2023-10-16T15:55:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa567fff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5701603e0 0x7fa57006f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56f7fe640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 0x7fa570159a40 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fa568000f80 tx=0x7fa568003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568006d50 con 0x7fa57006fc40 2023-10-16T15:55:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5680397a0 con 0x7fa57006fc40 2023-10-16T15:55:37.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568039b50 con 0x7fa57006fc40 2023-10-16T15:55:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57015c050 con 0x7fa57006fc40 2023-10-16T15:55:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.570+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57015c610 con 0x7fa57006fc40 2023-10-16T15:55:37.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.571+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57015c800 con 0x7fa57006fc40 2023-10-16T15:55:37.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.573+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa568039cb0 con 0x7fa57006fc40 2023-10-16T15:55:37.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.573+0000 7fa56cff9640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa55006b0a0 0x7fa55006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:37.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.574+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fa56803e070 con 0x7fa57006fc40 2023-10-16T15:55:37.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.574+0000 7fa56effd640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa55006b0a0 0x7fa55006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:37.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.575+0000 7fa56effd640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa55006b0a0 0x7fa55006d560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa560006fd0 tx=0x7fa560008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:37.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.579+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa568076610 con 0x7fa57006fc40 2023-10-16T15:55:37.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:37.775+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa57004e9b0 con 0x7fa57006fc40 2023-10-16T15:55:38.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.329+0000 7fa56cff9640 1 -- 172.21.15.29:0/2363881269 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v263) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa56802d030 con 0x7fa57006fc40 2023-10-16T15:55:38.332 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-16T15:55:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.334+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa55006b0a0 msgr2=0x7fa55006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.334+0000 7fa56ffff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa55006b0a0 0x7fa55006d560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa560006fd0 tx=0x7fa560008040 comp rx=0 tx=0).stop 2023-10-16T15:55:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.334+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 msgr2=0x7fa570159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.334+0000 7fa56ffff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 0x7fa570159a40 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fa568000f80 tx=0x7fa568003ae0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 shutdown_connections 2023-10-16T15:55:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa55006b0a0 0x7fa55006d560 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa57006fc40 0x7fa570159a40 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa5701603e0 0x7fa57006f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 --2- 172.21.15.29:0/2363881269 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa55c0052a0 0x7fa5701570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 >> 172.21.15.29:0/2363881269 conn(0x7fa5700ce820 msgr2=0x7fa5700ae320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:38.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 shutdown_connections 2023-10-16T15:55:38.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.335+0000 7fa56ffff640 1 -- 172.21.15.29:0/2363881269 wait complete. 2023-10-16T15:55:38.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T15:55:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.579+0000 7f9e213ae640 1 Processor -- start 2023-10-16T15:55:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.579+0000 7f9e213ae640 1 -- start start 2023-10-16T15:55:38.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e213ae640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e1c15ac70 0x7f9e1c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e213ae640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f9e1c0b4160 0x7f9e1c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e213ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c069da0 0x7f9e1c06a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e213ae640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c06a770 con 0x7f9e1c15ac70 2023-10-16T15:55:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e213ae640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0cc850 con 0x7f9e1c0b4160 2023-10-16T15:55:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e213ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0ccb30 con 0x7f9e1c069da0 2023-10-16T15:55:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e1c15ac70 0x7f9e1c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e1c15ac70 0x7f9e1c0cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38276/0 (socket says 172.21.15.29:38276) 2023-10-16T15:55:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.581+0000 7f9e1bfff640 1 -- 172.21.15.29:0/4024227743 learned_addr learned my addr 172.21.15.29:0/4024227743 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e20bad640 1 --2- 172.21.15.29:0/4024227743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c069da0 0x7f9e1c06a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e1b7fe640 1 --2- 172.21.15.29:0/4024227743 >> v2:172.21.15.170:3300/0 conn(0x7f9e1c0b4160 0x7f9e1c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e1bfff640 1 -- 172.21.15.29:0/4024227743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c069da0 msgr2=0x7f9e1c06a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e1bfff640 1 --2- 172.21.15.29:0/4024227743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c069da0 0x7f9e1c06a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e1bfff640 1 -- 172.21.15.29:0/4024227743 >> v2:172.21.15.170:3300/0 conn(0x7f9e1c0b4160 msgr2=0x7f9e1c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e1bfff640 1 --2- 172.21.15.29:0/4024227743 >> v2:172.21.15.170:3300/0 conn(0x7f9e1c0b4160 0x7f9e1c0cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.582+0000 7f9e1bfff640 1 -- 172.21.15.29:0/4024227743 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c0cce10 con 0x7f9e1c15ac70 2023-10-16T15:55:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.584+0000 7f9e1bfff640 1 --2- 172.21.15.29:0/4024227743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c15ac70 0x7f9e1c0cc270 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c073ca0 tx=0x7f9e0c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b7c58a9d38113 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.584+0000 7f9e1affd640 1 -- 172.21.15.29:0/4024227743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c031860 con 0x7f9e1c15ac70 2023-10-16T15:55:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.584+0000 7f9e1affd640 1 -- 172.21.15.29:0/4024227743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0319c0 con 0x7f9e1c15ac70 2023-10-16T15:55:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.584+0000 7f9e1affd640 1 -- 172.21.15.29:0/4024227743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c031d50 con 0x7f9e1c15ac70 2023-10-16T15:55:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 -- 172.21.15.29:0/4024227743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c15ac70 msgr2=0x7f9e1c0cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 --2- 172.21.15.29:0/4024227743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c15ac70 0x7f9e1c0cc270 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c073ca0 tx=0x7f9e0c02b900 comp rx=0 tx=0).stop 2023-10-16T15:55:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 -- 172.21.15.29:0/4024227743 shutdown_connections 2023-10-16T15:55:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 --2- 172.21.15.29:0/4024227743 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c069da0 0x7f9e1c06a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 --2- 172.21.15.29:0/4024227743 >> v2:172.21.15.170:3300/0 conn(0x7f9e1c0b4160 0x7f9e1c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 --2- 172.21.15.29:0/4024227743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c15ac70 0x7f9e1c0cc270 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 -- 172.21.15.29:0/4024227743 >> 172.21.15.29:0/4024227743 conn(0x7f9e1c0ce6a0 msgr2=0x7f9e1c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 -- 172.21.15.29:0/4024227743 shutdown_connections 2023-10-16T15:55:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.585+0000 7f9e213ae640 1 -- 172.21.15.29:0/4024227743 wait complete. 2023-10-16T15:55:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.587+0000 7f9e213ae640 1 Processor -- start 2023-10-16T15:55:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.587+0000 7f9e213ae640 1 -- start start 2023-10-16T15:55:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e213ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 0x7f9e1c070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e213ae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c0b4160 0x7f9e1c14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e213ae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e1c15ac70 0x7f9e1c0c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e213ae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0c5ac0 con 0x7f9e1c069da0 2023-10-16T15:55:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e213ae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0c5da0 con 0x7f9e1c15ac70 2023-10-16T15:55:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e213ae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0c6080 con 0x7f9e1c0b4160 2023-10-16T15:55:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 0x7f9e1c070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 0x7f9e1c070ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38282/0 (socket says 172.21.15.29:38282) 2023-10-16T15:55:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.588+0000 7f9e1bfff640 1 -- 172.21.15.29:0/672774593 learned_addr learned my addr 172.21.15.29:0/672774593 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e1bfff640 1 -- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c0b4160 msgr2=0x7f9e1c14ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e1bfff640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c0b4160 0x7f9e1c14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e1bfff640 1 -- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e1c15ac70 msgr2=0x7f9e1c0c91c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T15:55:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e1bfff640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e1c15ac70 0x7f9e1c0c91c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e1bfff640 1 -- 172.21.15.29:0/672774593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c0094d0 con 0x7f9e1c069da0 2023-10-16T15:55:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e1bfff640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 0x7f9e1c070ec0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c02b940 tx=0x7f9e0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.589+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c03ad30 con 0x7f9e1c069da0 2023-10-16T15:55:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.590+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0443f0 con 0x7f9e1c069da0 2023-10-16T15:55:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.590+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0447a0 con 0x7f9e1c069da0 2023-10-16T15:55:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.590+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c1f5ac0 con 0x7f9e1c069da0 2023-10-16T15:55:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.590+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c0c6360 con 0x7f9e1c069da0 2023-10-16T15:55:38.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.591+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c1f5d00 con 0x7f9e1c069da0 2023-10-16T15:55:38.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.592+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f9e0c04d450 con 0x7f9e1c069da0 2023-10-16T15:55:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.593+0000 7f9e18ff9640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f9e0806b020 0x7f9e0806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.593+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5298+0+0 (secure 0 0 0) 0x7f9e0c0ac110 con 0x7f9e1c069da0 2023-10-16T15:55:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.594+0000 7f9e1b7fe640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f9e0806b020 0x7f9e0806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.595+0000 7f9e1b7fe640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f9e0806b020 0x7f9e0806d4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e10006fd0 tx=0x7f9e10008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.597+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9e0c0770c0 con 0x7f9e1c069da0 2023-10-16T15:55:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:38.790+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9e1c1f5d00 con 0x7f9e1c069da0 2023-10-16T15:55:39.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.338+0000 7f9e18ff9640 1 -- 172.21.15.29:0/672774593 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v264) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9e1c1f5d00 con 0x7f9e1c069da0 2023-10-16T15:55:39.341 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-16T15:55:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.341+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f9e0806b020 msgr2=0x7f9e0806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.341+0000 7f9e213ae640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f9e0806b020 0x7f9e0806d4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e10006fd0 tx=0x7f9e10008040 comp rx=0 tx=0).stop 2023-10-16T15:55:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 msgr2=0x7f9e1c070ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 0x7f9e1c070ec0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c02b940 tx=0x7f9e0c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 shutdown_connections 2023-10-16T15:55:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f9e0806b020 0x7f9e0806d4e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9e1c15ac70 0x7f9e1c0c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f9e1c0b4160 0x7f9e1c14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 --2- 172.21.15.29:0/672774593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e1c069da0 0x7f9e1c070ec0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 >> 172.21.15.29:0/672774593 conn(0x7f9e1c0ce6a0 msgr2=0x7f9e1c0b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:39.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 shutdown_connections 2023-10-16T15:55:39.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.342+0000 7f9e213ae640 1 -- 172.21.15.29:0/672774593 wait complete. 2023-10-16T15:55:39.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T15:55:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.569+0000 7fefc0975640 1 Processor -- start 2023-10-16T15:55:39.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.569+0000 7fefc0975640 1 -- start start 2023-10-16T15:55:39.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.570+0000 7fefc0975640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fefbc0ca0a0 0x7fefbc06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.570+0000 7fefc0975640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefbc15d910 con 0x7fefbc159950 2023-10-16T15:55:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.570+0000 7fefc0975640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefbc15dbf0 con 0x7fefbc070b60 2023-10-16T15:55:39.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.570+0000 7fefc0975640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefbc15e490 con 0x7fefbc0ca0a0 2023-10-16T15:55:39.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.571+0000 7fefbb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fefbc159950 0x7fefbc070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50374/0 (socket says 172.21.15.29:50374) 2023-10-16T15:55:39.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.571+0000 7fefbb7fe640 1 -- 172.21.15.29:0/1944125324 learned_addr learned my addr 172.21.15.29:0/1944125324 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:39.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.571+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1944125324 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0ca0a0 0x7fefbc06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.571+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210487518 0 0) 0x7fefbc15d910 con 0x7fefbc159950 2023-10-16T15:55:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa8003520 con 0x7fefbc159950 2023-10-16T15:55:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803206281 0 0) 0x7fefa8003520 con 0x7fefbc159950 2023-10-16T15:55:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0ca0a0 msgr2=0x7fefbc06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 --2- 172.21.15.29:0/1944125324 >> v2:172.21.15.170:3300/0 conn(0x7fefbc0ca0a0 0x7fefbc06c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 >> v1:172.21.15.170:6789/0 conn(0x7fefbc070b60 legacy=0x7fefbc06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc15ded0 con 0x7fefbc159950 2023-10-16T15:55:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fefac001670 con 0x7fefbc159950 2023-10-16T15:55:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 >> v1:172.21.15.29:6789/0 conn(0x7fefbc159950 legacy=0x7fefbc070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefa8003970 0x7fefa8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa80047c0 0x7fefa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa8005180 0x7fefa800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc15e490 con 0x7fefa80047c0 2023-10-16T15:55:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa800e330 con 0x7fefa8005180 2023-10-16T15:55:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa800e4f0 con 0x7fefa8003970 2023-10-16T15:55:39.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.572+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa80047c0 0x7fefa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefa8003970 0x7fefa8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbaffd640 1 -- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefa8003970 msgr2=0x7fefa8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefa8003970 0x7fefa8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbaffd640 1 -- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa8005180 msgr2=0x7fefa800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:39.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa8005180 0x7fefa800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbaffd640 1 -- 172.21.15.29:0/1944125324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8003520 con 0x7fefa80047c0 2023-10-16T15:55:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.573+0000 7fefbbfff640 1 -- 172.21.15.29:0/1944125324 reap_dead start 2023-10-16T15:55:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.574+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa80047c0 0x7fefa8004c40 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fefa4001e50 tx=0x7fefa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de9703a79eabd85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.574+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4003b70 con 0x7fefa80047c0 2023-10-16T15:55:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.574+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa4003cd0 con 0x7fefa80047c0 2023-10-16T15:55:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.574+0000 7fefba7fc640 1 -- 172.21.15.29:0/1944125324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4005310 con 0x7fefa80047c0 2023-10-16T15:55:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 -- 172.21.15.29:0/1944125324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa80047c0 msgr2=0x7fefa8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa80047c0 0x7fefa8004c40 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fefa4001e50 tx=0x7fefa80057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 -- 172.21.15.29:0/1944125324 shutdown_connections 2023-10-16T15:55:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 --2- 172.21.15.29:0/1944125324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefa80047c0 0x7fefa8004c40 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 -- 172.21.15.29:0/1944125324 >> 172.21.15.29:0/1944125324 conn(0x7fefbc0ce6a0 msgr2=0x7fefbc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 -- 172.21.15.29:0/1944125324 shutdown_connections 2023-10-16T15:55:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.575+0000 7fefc0975640 1 -- 172.21.15.29:0/1944125324 wait complete. 2023-10-16T15:55:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.577+0000 7fefc0975640 1 Processor -- start 2023-10-16T15:55:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.577+0000 7fefc0975640 1 -- start start 2023-10-16T15:55:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefc0975640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa80047c0 0x7fefbc0ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefc0975640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b11f0 0x7fefbc0ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefc0975640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 0x7fefbc0ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefc0975640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0af020 con 0x7fefbc0aeba0 2023-10-16T15:55:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefc0975640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0b24e0 con 0x7fefa80047c0 2023-10-16T15:55:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefc0975640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0b1960 con 0x7fefbc0b11f0 2023-10-16T15:55:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 0x7fefbc0ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 0x7fefbc0ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38312/0 (socket says 172.21.15.29:38312) 2023-10-16T15:55:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.578+0000 7fefbbfff640 1 -- 172.21.15.29:0/1110001703 learned_addr learned my addr 172.21.15.29:0/1110001703 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa80047c0 0x7fefbc0ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbaffd640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b11f0 0x7fefbc0ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbbfff640 1 -- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b11f0 msgr2=0x7fefbc0ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b11f0 0x7fefbc0ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbbfff640 1 -- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa80047c0 msgr2=0x7fefbc0ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa80047c0 0x7fefbc0ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbbfff640 1 -- 172.21.15.29:0/1110001703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc0b1c40 con 0x7fefbc0aeba0 2023-10-16T15:55:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefbbfff640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 0x7fefbc0ccb00 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fefbc06d350 tx=0x7fefb400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.579+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4004030 con 0x7fefbc0aeba0 2023-10-16T15:55:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.580+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb4004190 con 0x7fefbc0aeba0 2023-10-16T15:55:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.580+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4003610 con 0x7fefbc0aeba0 2023-10-16T15:55:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.580+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc0b1f20 con 0x7fefbc0aeba0 2023-10-16T15:55:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.580+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc0b1650 con 0x7fefbc0aeba0 2023-10-16T15:55:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.581+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc04e9b0 con 0x7fefbc0aeba0 2023-10-16T15:55:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.584+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fefb4003770 con 0x7fefbc0aeba0 2023-10-16T15:55:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.584+0000 7fefb3fff640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fef9c06b020 0x7fef9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.585+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5680+0+0 (secure 0 0 0) 0x7fefb408a4d0 con 0x7fefbc0aeba0 2023-10-16T15:55:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.585+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fef9c06b020 0x7fef9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.587+0000 7fefbb7fe640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fef9c06b020 0x7fef9c06d4e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fefac001050 tx=0x7fefac009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.588+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefb4055290 con 0x7fefbc0aeba0 2023-10-16T15:55:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:39.823+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fefbc0b4620 con 0x7fefbc0aeba0 2023-10-16T15:55:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.359+0000 7fefb3fff640 1 -- 172.21.15.29:0/1110001703 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v137) v1 ==== 174+0+0 (secure 0 0 0) 0x7fefb40588b0 con 0x7fefbc0aeba0 2023-10-16T15:55:40.362 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 34 and data pool 35 2023-10-16T15:55:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fef9c06b020 msgr2=0x7fef9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fef9c06b020 0x7fef9c06d4e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fefac001050 tx=0x7fefac009040 comp rx=0 tx=0).stop 2023-10-16T15:55:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 msgr2=0x7fefbc0ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 0x7fefbc0ccb00 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fefbc06d350 tx=0x7fefb400ec00 comp rx=0 tx=0).stop 2023-10-16T15:55:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 shutdown_connections 2023-10-16T15:55:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fef9c06b020 0x7fef9c06d4e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefbc0aeba0 0x7fefbc0ccb00 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fefbc0b11f0 0x7fefbc0ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 --2- 172.21.15.29:0/1110001703 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fefa80047c0 0x7fefbc0ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.363+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 >> 172.21.15.29:0/1110001703 conn(0x7fefbc0ce6a0 msgr2=0x7fefbc1589f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.364+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 shutdown_connections 2023-10-16T15:55:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.364+0000 7fefc0975640 1 -- 172.21.15.29:0/1110001703 wait complete. 2023-10-16T15:55:40.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:40.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.596+0000 7f429fd7c640 1 Processor -- start 2023-10-16T15:55:40.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.596+0000 7f429fd7c640 1 -- start start 2023-10-16T15:55:40.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.597+0000 7f429fd7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42980adb50 0x7f429806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:40.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.597+0000 7f429fd7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4298070fd0 0x7f42980b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:40.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.597+0000 7f429fd7c640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4298071450 con 0x7f42980adf30 2023-10-16T15:55:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.597+0000 7f429fd7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42980cc600 con 0x7f42980adb50 2023-10-16T15:55:40.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.597+0000 7f429fd7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42980cc8e0 con 0x7f4298070fd0 2023-10-16T15:55:40.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42980adb50 0x7f429806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:40.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42980adb50 0x7f429806f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38322/0 (socket says 172.21.15.29:38322) 2023-10-16T15:55:40.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 -- 172.21.15.29:0/167299728 learned_addr learned my addr 172.21.15.29:0/167299728 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:40.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429d317640 1 --2- 172.21.15.29:0/167299728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4298070fd0 0x7f42980b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:40.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 -- 172.21.15.29:0/167299728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4298070fd0 msgr2=0x7f42980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:40.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 --2- 172.21.15.29:0/167299728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4298070fd0 0x7f42980b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 -- 172.21.15.29:0/167299728 >> v1:172.21.15.170:6789/0 conn(0x7f42980adf30 legacy=0x7f4298164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:40.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.598+0000 7f429db18640 1 -- 172.21.15.29:0/167299728 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42980ccbc0 con 0x7f42980adb50 2023-10-16T15:55:40.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.599+0000 7f429db18640 1 --2- 172.21.15.29:0/167299728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42980adb50 0x7f429806f550 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef6cdd5deadbbed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:40.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.600+0000 7f429cb16640 1 -- 172.21.15.29:0/167299728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294031640 con 0x7f42980adb50 2023-10-16T15:55:40.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.600+0000 7f429cb16640 1 -- 172.21.15.29:0/167299728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42940317a0 con 0x7f42980adb50 2023-10-16T15:55:40.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.600+0000 7f429cb16640 1 -- 172.21.15.29:0/167299728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294031b30 con 0x7f42980adb50 2023-10-16T15:55:40.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.600+0000 7f429fd7c640 1 -- 172.21.15.29:0/167299728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42980adb50 msgr2=0x7f429806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.600+0000 7f429fd7c640 1 --2- 172.21.15.29:0/167299728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42980adb50 0x7f429806f550 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b8f0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.601+0000 7f429fd7c640 1 -- 172.21.15.29:0/167299728 shutdown_connections 2023-10-16T15:55:40.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.601+0000 7f429fd7c640 1 --2- 172.21.15.29:0/167299728 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f4298070fd0 0x7f42980b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.601+0000 7f429fd7c640 1 --2- 172.21.15.29:0/167299728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42980adb50 0x7f429806f550 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.601+0000 7f429fd7c640 1 -- 172.21.15.29:0/167299728 >> 172.21.15.29:0/167299728 conn(0x7f42980cb410 msgr2=0x7f42980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:40.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.601+0000 7f429fd7c640 1 -- 172.21.15.29:0/167299728 shutdown_connections 2023-10-16T15:55:40.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.601+0000 7f429fd7c640 1 -- 172.21.15.29:0/167299728 wait complete. 2023-10-16T15:55:40.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.602+0000 7f429fd7c640 1 Processor -- start 2023-10-16T15:55:40.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.602+0000 7f429fd7c640 1 -- start start 2023-10-16T15:55:40.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.603+0000 7f429fd7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 0x7f4298161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:40.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.603+0000 7f429fd7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42980adb50 0x7f42980af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:40.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.603+0000 7f429fd7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42980adf30 0x7f42980afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:40.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.603+0000 7f429fd7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42980c0250 con 0x7f4298070fd0 2023-10-16T15:55:40.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.603+0000 7f429fd7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4298161570 con 0x7f42980adf30 2023-10-16T15:55:40.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.603+0000 7f429fd7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42980bfee0 con 0x7f42980adb50 2023-10-16T15:55:40.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 0x7f4298161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:40.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 0x7f4298161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38326/0 (socket says 172.21.15.29:38326) 2023-10-16T15:55:40.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 -- 172.21.15.29:0/644474699 learned_addr learned my addr 172.21.15.29:0/644474699 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:40.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429e319640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42980adf30 0x7f42980afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:40.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429d317640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42980adb50 0x7f42980af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 -- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42980adb50 msgr2=0x7f42980af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42980adb50 0x7f42980af7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 -- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42980adf30 msgr2=0x7f42980afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42980adf30 0x7f42980afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429db18640 1 -- 172.21.15.29:0/644474699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940094d0 con 0x7f4298070fd0 2023-10-16T15:55:40.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.604+0000 7f429e319640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42980adf30 0x7f42980afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:40.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.605+0000 7f429db18640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 0x7f4298161030 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f42940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:40.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.605+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429403ac60 con 0x7f4298070fd0 2023-10-16T15:55:40.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.605+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429403ade0 con 0x7f4298070fd0 2023-10-16T15:55:40.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.605+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294044630 con 0x7f4298070fd0 2023-10-16T15:55:40.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.605+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4298164f60 con 0x7f4298070fd0 2023-10-16T15:55:40.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.605+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4298165520 con 0x7f4298070fd0 2023-10-16T15:55:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.606+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429804e160 con 0x7f4298070fd0 2023-10-16T15:55:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.607+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f429402f050 con 0x7f4298070fd0 2023-10-16T15:55:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.608+0000 7f428e7fc640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f427c06b0e0 0x7f427c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.608+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f42940abca0 con 0x7f4298070fd0 2023-10-16T15:55:40.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.609+0000 7f429d317640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f427c06b0e0 0x7f427c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:40.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.610+0000 7f429d317640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f427c06b0e0 0x7f427c06d5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f42980b4140 tx=0x7f42880083a0 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.613+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42940768c0 con 0x7f4298070fd0 2023-10-16T15:55:40.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.800+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42980b4910 con 0x7f4298070fd0 2023-10-16T15:55:40.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.803+0000 7f428e7fc640 1 -- 172.21.15.29:0/644474699 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f4294035090 con 0x7f4298070fd0 2023-10-16T15:55:40.806 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:40.806 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":265,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:39.826037+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f427c06b0e0 msgr2=0x7f427c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f427c06b0e0 0x7f427c06d5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f42980b4140 tx=0x7f42880083a0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 msgr2=0x7f4298161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 0x7f4298161030 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f42940046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 shutdown_connections 2023-10-16T15:55:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f427c06b0e0 0x7f427c06d5a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42980adf30 0x7f42980afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42980adb50 0x7f42980af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 --2- 172.21.15.29:0/644474699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4298070fd0 0x7f4298161030 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 >> 172.21.15.29:0/644474699 conn(0x7f42980cb410 msgr2=0x7f429806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 shutdown_connections 2023-10-16T15:55:40.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:40.806+0000 7f429fd7c640 1 -- 172.21.15.29:0/644474699 wait complete. 2023-10-16T15:55:40.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.049+0000 7fba39813640 1 Processor -- start 2023-10-16T15:55:41.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.049+0000 7fba39813640 1 -- start start 2023-10-16T15:55:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba39813640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba340ac890 0x7fba340accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba39813640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba34160a90 con 0x7fba3415dde0 2023-10-16T15:55:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba39813640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba340ad200 con 0x7fba3415c9f0 2023-10-16T15:55:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba39813640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba340cdec0 con 0x7fba340ac890 2023-10-16T15:55:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba39012640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba340ac890 0x7fba340accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba33fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fba3415dde0 0x7fba340709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50384/0 (socket says 172.21.15.29:50384) 2023-10-16T15:55:41.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.051+0000 7fba33fff640 1 -- 172.21.15.29:0/1581071234 learned_addr learned my addr 172.21.15.29:0/1581071234 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:41.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.052+0000 7fba39012640 1 -- 172.21.15.29:0/1581071234 >> v1:172.21.15.170:6789/0 conn(0x7fba3415c9f0 legacy=0x7fba340c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:41.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.052+0000 7fba39012640 1 -- 172.21.15.29:0/1581071234 >> v1:172.21.15.29:6789/0 conn(0x7fba3415dde0 legacy=0x7fba340709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.052+0000 7fba39012640 1 -- 172.21.15.29:0/1581071234 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340cd900 con 0x7fba340ac890 2023-10-16T15:55:41.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.052+0000 7fba337fe640 1 -- 172.21.15.29:0/1581071234 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749840193 0 0) 0x7fba34160a90 con 0x7fba3415dde0 2023-10-16T15:55:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.053+0000 7fba39012640 1 --2- 172.21.15.29:0/1581071234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba340ac890 0x7fba340accc0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fba340ad390 tx=0x7fba2c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89e571fc45e0e59 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.053+0000 7fba337fe640 1 -- 172.21.15.29:0/1581071234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c02cc50 con 0x7fba340ac890 2023-10-16T15:55:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.053+0000 7fba337fe640 1 -- 172.21.15.29:0/1581071234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c02cdb0 con 0x7fba340ac890 2023-10-16T15:55:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.053+0000 7fba337fe640 1 -- 172.21.15.29:0/1581071234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c034660 con 0x7fba340ac890 2023-10-16T15:55:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 -- 172.21.15.29:0/1581071234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba340ac890 msgr2=0x7fba340accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 --2- 172.21.15.29:0/1581071234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba340ac890 0x7fba340accc0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fba340ad390 tx=0x7fba2c02bea0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 -- 172.21.15.29:0/1581071234 shutdown_connections 2023-10-16T15:55:41.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 --2- 172.21.15.29:0/1581071234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba340ac890 0x7fba340accc0 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 -- 172.21.15.29:0/1581071234 >> 172.21.15.29:0/1581071234 conn(0x7fba340ce6a0 msgr2=0x7fba340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:41.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 -- 172.21.15.29:0/1581071234 shutdown_connections 2023-10-16T15:55:41.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.054+0000 7fba39813640 1 -- 172.21.15.29:0/1581071234 wait complete. 2023-10-16T15:55:41.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.056+0000 7fba39813640 1 Processor -- start 2023-10-16T15:55:41.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.056+0000 7fba39813640 1 -- start start 2023-10-16T15:55:41.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39813640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba340ac890 0x7fba341f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39813640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba3415c9f0 0x7fba34069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39813640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 0x7fba3415a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39813640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba340c2500 con 0x7fba3415dde0 2023-10-16T15:55:41.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39813640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba340c3a00 con 0x7fba340ac890 2023-10-16T15:55:41.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39813640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba341f68b0 con 0x7fba3415c9f0 2023-10-16T15:55:41.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39012640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 0x7fba3415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39012640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 0x7fba3415a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38342/0 (socket says 172.21.15.29:38342) 2023-10-16T15:55:41.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.057+0000 7fba39012640 1 -- 172.21.15.29:0/4060100270 learned_addr learned my addr 172.21.15.29:0/4060100270 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:41.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba39012640 1 -- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba3415c9f0 msgr2=0x7fba34069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba39012640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba3415c9f0 0x7fba34069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba39012640 1 -- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba340ac890 msgr2=0x7fba341f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba39012640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba340ac890 0x7fba341f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba39012640 1 -- 172.21.15.29:0/4060100270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c009a70 con 0x7fba3415dde0 2023-10-16T15:55:41.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba39012640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 0x7fba3415a580 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009e80 tx=0x7fba2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:41.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c03e610 con 0x7fba3415dde0 2023-10-16T15:55:41.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c03e790 con 0x7fba3415dde0 2023-10-16T15:55:41.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.058+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c03eb40 con 0x7fba3415dde0 2023-10-16T15:55:41.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.059+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba341f65d0 con 0x7fba3415dde0 2023-10-16T15:55:41.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.059+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba340c3f10 con 0x7fba3415dde0 2023-10-16T15:55:41.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.060+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3404e9b0 con 0x7fba3415dde0 2023-10-16T15:55:41.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.061+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fba2c03b050 con 0x7fba3415dde0 2023-10-16T15:55:41.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.062+0000 7fba317fa640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fba1806b0b0 0x7fba1806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.062+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5780+0+0 (secure 0 0 0) 0x7fba2c0ac4a0 con 0x7fba3415dde0 2023-10-16T15:55:41.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.064+0000 7fba38811640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fba1806b0b0 0x7fba1806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.067+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fba2c077100 con 0x7fba3415dde0 2023-10-16T15:55:41.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.068+0000 7fba38811640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fba1806b0b0 0x7fba1806d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fba28003cc0 tx=0x7fba280079b0 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:41.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.260+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba340adac0 con 0x7fba3415dde0 2023-10-16T15:55:41.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.262+0000 7fba317fa640 1 -- 172.21.15.29:0/4060100270 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+16646 (secure 0 0 0) 0x7fba2c07a720 con 0x7fba3415dde0 2023-10-16T15:55:41.265 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:41.265 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":265,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:39.826037+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:41.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fba1806b0b0 msgr2=0x7fba1806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fba1806b0b0 0x7fba1806d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fba28003cc0 tx=0x7fba280079b0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 msgr2=0x7fba3415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 0x7fba3415a580 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009e80 tx=0x7fba2c004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:41.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 shutdown_connections 2023-10-16T15:55:41.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fba1806b0b0 0x7fba1806d570 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba3415dde0 0x7fba3415a580 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fba3415c9f0 0x7fba34069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 --2- 172.21.15.29:0/4060100270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fba340ac890 0x7fba341f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 >> 172.21.15.29:0/4060100270 conn(0x7fba340ce6a0 msgr2=0x7fba340cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 shutdown_connections 2023-10-16T15:55:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.265+0000 7fba39813640 1 -- 172.21.15.29:0/4060100270 wait complete. 2023-10-16T15:55:41.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T15:55:41.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.516+0000 7f677b1c5640 1 Processor -- start 2023-10-16T15:55:41.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.516+0000 7f677b1c5640 1 -- start start 2023-10-16T15:55:41.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.517+0000 7f677b1c5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f677415ad70 0x7f6774069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.517+0000 7f677b1c5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740c9ca0 0x7f677406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.517+0000 7f677b1c5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67740cb030 con 0x7f67740ca3b0 2023-10-16T15:55:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.518+0000 7f677b1c5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67740c91c0 con 0x7f677415ad70 2023-10-16T15:55:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.518+0000 7f677b1c5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740c94a0 con 0x7f67740c9ca0 2023-10-16T15:55:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.518+0000 7f6778f61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f677415ad70 0x7f6774069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.518+0000 7f6778f61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f677415ad70 0x7f6774069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38344/0 (socket says 172.21.15.29:38344) 2023-10-16T15:55:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.518+0000 7f6778f61640 1 -- 172.21.15.29:0/4043730795 learned_addr learned my addr 172.21.15.29:0/4043730795 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.518+0000 7f6773fff640 1 --2- 172.21.15.29:0/4043730795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740c9ca0 0x7f677406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.519+0000 7f6778f61640 1 -- 172.21.15.29:0/4043730795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740c9ca0 msgr2=0x7f677406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.519+0000 7f6778f61640 1 --2- 172.21.15.29:0/4043730795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740c9ca0 0x7f677406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.519+0000 7f6778f61640 1 -- 172.21.15.29:0/4043730795 >> v1:172.21.15.170:6789/0 conn(0x7f67740ca3b0 legacy=0x7f67740ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.519+0000 7f6778f61640 1 -- 172.21.15.29:0/4043730795 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740c9780 con 0x7f677415ad70 2023-10-16T15:55:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.520+0000 7f6778f61640 1 --2- 172.21.15.29:0/4043730795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f677415ad70 0x7f6774069da0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f6764009e50 tx=0x7f676402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d6ae3e1f3a6ecd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.520+0000 7f67737fe640 1 -- 172.21.15.29:0/4043730795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676402ca70 con 0x7f677415ad70 2023-10-16T15:55:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.520+0000 7f67737fe640 1 -- 172.21.15.29:0/4043730795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676402cbd0 con 0x7f677415ad70 2023-10-16T15:55:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.520+0000 7f67737fe640 1 -- 172.21.15.29:0/4043730795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67640343f0 con 0x7f677415ad70 2023-10-16T15:55:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 -- 172.21.15.29:0/4043730795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f677415ad70 msgr2=0x7f6774069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4043730795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f677415ad70 0x7f6774069da0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f6764009e50 tx=0x7f676402bed0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 -- 172.21.15.29:0/4043730795 shutdown_connections 2023-10-16T15:55:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4043730795 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740c9ca0 0x7f677406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4043730795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f677415ad70 0x7f6774069da0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 -- 172.21.15.29:0/4043730795 >> 172.21.15.29:0/4043730795 conn(0x7f67740ce9b0 msgr2=0x7f67740cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 -- 172.21.15.29:0/4043730795 shutdown_connections 2023-10-16T15:55:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.521+0000 7f677b1c5640 1 -- 172.21.15.29:0/4043730795 wait complete. 2023-10-16T15:55:41.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.523+0000 7f677b1c5640 1 Processor -- start 2023-10-16T15:55:41.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.523+0000 7f677b1c5640 1 -- start start 2023-10-16T15:55:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f677b1c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 0x7f677415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f677b1c5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740ca3b0 0x7f67741f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f677b1c5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f677415ad70 0x7f67740ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f677b1c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67741f38c0 con 0x7f67740c9ca0 2023-10-16T15:55:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f677b1c5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774150ed0 con 0x7f677415ad70 2023-10-16T15:55:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f677b1c5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677415ced0 con 0x7f67740ca3b0 2023-10-16T15:55:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.524+0000 7f6778f61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 0x7f677415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 0x7f677415c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38352/0 (socket says 172.21.15.29:38352) 2023-10-16T15:55:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 -- 172.21.15.29:0/4207214442 learned_addr learned my addr 172.21.15.29:0/4207214442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6773fff640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740ca3b0 0x7f67741f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6779762640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f677415ad70 0x7f67740ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 -- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740ca3b0 msgr2=0x7f67741f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740ca3b0 0x7f67741f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 -- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f677415ad70 msgr2=0x7f67740ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f677415ad70 0x7f67740ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 -- 172.21.15.29:0/4207214442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6764009b30 con 0x7f67740c9ca0 2023-10-16T15:55:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.525+0000 7f6778f61640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 0x7f677415c990 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f6764036040 tx=0x7f6764004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.526+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764009510 con 0x7f67740c9ca0 2023-10-16T15:55:41.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.526+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676403e450 con 0x7f67740c9ca0 2023-10-16T15:55:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.526+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67741f2db0 con 0x7f67740c9ca0 2023-10-16T15:55:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.526+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676403e800 con 0x7f67740c9ca0 2023-10-16T15:55:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.526+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67741f30d0 con 0x7f67740c9ca0 2023-10-16T15:55:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.527+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677404e160 con 0x7f67740c9ca0 2023-10-16T15:55:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.528+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f676403e960 con 0x7f67740c9ca0 2023-10-16T15:55:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.529+0000 7f67717fa640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f675406b0b0 0x7f675406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.529+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f67640ac8e0 con 0x7f67740c9ca0 2023-10-16T15:55:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.530+0000 7f6773fff640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f675406b0b0 0x7f675406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.532+0000 7f6773fff640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f675406b0b0 0x7f675406d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67741545e0 tx=0x7f675c008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.534+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6764077540 con 0x7f67740c9ca0 2023-10-16T15:55:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:41.765+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f67741f36a0 con 0x7f67740c9ca0 2023-10-16T15:55:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.403+0000 7f67717fa640 1 -- 172.21.15.29:0/4207214442 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v140) v1 ==== 118+0+0 (secure 0 0 0) 0x7f676407ab60 con 0x7f67740c9ca0 2023-10-16T15:55:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f675406b0b0 msgr2=0x7f675406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f675406b0b0 0x7f675406d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67741545e0 tx=0x7f675c008040 comp rx=0 tx=0).stop 2023-10-16T15:55:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 msgr2=0x7f677415c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 0x7f677415c990 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f6764036040 tx=0x7f6764004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 shutdown_connections 2023-10-16T15:55:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f675406b0b0 0x7f675406d570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f677415ad70 0x7f67740ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67740ca3b0 0x7f67741f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 --2- 172.21.15.29:0/4207214442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67740c9ca0 0x7f677415c990 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 >> 172.21.15.29:0/4207214442 conn(0x7f67740ce9b0 msgr2=0x7f67740ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 shutdown_connections 2023-10-16T15:55:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.407+0000 7f677b1c5640 1 -- 172.21.15.29:0/4207214442 wait complete. 2023-10-16T15:55:42.429 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T15:55:42.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T15:55:42.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.654+0000 7fa79e0da640 1 Processor -- start 2023-10-16T15:55:42.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.654+0000 7fa79e0da640 1 -- start start 2023-10-16T15:55:42.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.655+0000 7fa79e0da640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa79815d330 0x7fa7981618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.655+0000 7fa79e0da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 0x7fa7980ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa79e0da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7980d0890 con 0x7fa79806c7d0 2023-10-16T15:55:42.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa79e0da640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79806b1e0 con 0x7fa79815d330 2023-10-16T15:55:42.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa79e0da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980d05b0 con 0x7fa7980661c0 2023-10-16T15:55:42.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa796ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa79806c7d0 0x7fa798065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50388/0 (socket says 172.21.15.29:50388) 2023-10-16T15:55:42.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa796ffd640 1 -- 172.21.15.29:0/4140004386 learned_addr learned my addr 172.21.15.29:0/4140004386 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:42.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016427132 0 0) 0x7fa7980d0890 con 0x7fa79806c7d0 2023-10-16T15:55:42.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa7977fe640 1 --2- 172.21.15.29:0/4140004386 >> v2:172.21.15.170:3300/0 conn(0x7fa79815d330 0x7fa7981618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa784003520 con 0x7fa79806c7d0 2023-10-16T15:55:42.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.656+0000 7fa797fff640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 0x7fa7980ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280449935 0 0) 0x7fa784003520 con 0x7fa79806c7d0 2023-10-16T15:55:42.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 msgr2=0x7fa7980ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:42.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 0x7fa7980ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 >> v2:172.21.15.170:3300/0 conn(0x7fa79815d330 msgr2=0x7fa7981618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 --2- 172.21.15.29:0/4140004386 >> v2:172.21.15.170:3300/0 conn(0x7fa79815d330 0x7fa7981618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798160dd0 con 0x7fa79806c7d0 2023-10-16T15:55:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fa780001670 con 0x7fa79806c7d0 2023-10-16T15:55:42.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 >> v1:172.21.15.29:6789/0 conn(0x7fa79806c7d0 legacy=0x7fa798065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa784003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 0x7fa7840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa784004d00 0x7fa784005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79806b1e0 con 0x7fa784003970 2023-10-16T15:55:42.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980d05b0 con 0x7fa784004d00 2023-10-16T15:55:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78400bb40 con 0x7fa7980661c0 2023-10-16T15:55:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7977fe640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa784003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa797fff640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa784004d00 0x7fa784005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa796ffd640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 0x7fa7840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.657+0000 7fa7977fe640 1 -- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 msgr2=0x7fa7840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.658+0000 7fa7977fe640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980661c0 0x7fa7840047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.658+0000 7fa7977fe640 1 -- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa784004d00 msgr2=0x7fa784005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.658+0000 7fa7977fe640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa784004d00 0x7fa784005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.658+0000 7fa7977fe640 1 -- 172.21.15.29:0/4140004386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784003520 con 0x7fa784003970 2023-10-16T15:55:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.658+0000 7fa797fff640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa784004d00 0x7fa784005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.659+0000 7fa7977fe640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa784003d90 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fa788007360 tx=0x7fa7840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2751361eec2c2d97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:42.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.659+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78802ccf0 con 0x7fa784003970 2023-10-16T15:55:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.659+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78802ce50 con 0x7fa784003970 2023-10-16T15:55:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.659+0000 7fa7967fc640 1 -- 172.21.15.29:0/4140004386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788034660 con 0x7fa784003970 2023-10-16T15:55:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa79e0da640 1 -- 172.21.15.29:0/4140004386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 msgr2=0x7fa784003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa79e0da640 1 --2- 172.21.15.29:0/4140004386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa784003d90 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fa788007360 tx=0x7fa7840057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa797fff640 1 -- 172.21.15.29:0/4140004386 reap_dead start 2023-10-16T15:55:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa79e0da640 1 -- 172.21.15.29:0/4140004386 shutdown_connections 2023-10-16T15:55:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa79e0da640 1 -- 172.21.15.29:0/4140004386 >> 172.21.15.29:0/4140004386 conn(0x7fa7980d0ab0 msgr2=0x7fa7980d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa79e0da640 1 -- 172.21.15.29:0/4140004386 shutdown_connections 2023-10-16T15:55:42.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.660+0000 7fa79e0da640 1 -- 172.21.15.29:0/4140004386 wait complete. 2023-10-16T15:55:42.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.662+0000 7fa79e0da640 1 Processor -- start 2023-10-16T15:55:42.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.662+0000 7fa79e0da640 1 -- start start 2023-10-16T15:55:42.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa79e0da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa7980adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa79e0da640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980ca2a0 0x7fa7981e7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa79e0da640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7980ca720 0x7fa7981e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa79e0da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980af660 con 0x7fa784003970 2023-10-16T15:55:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa79e0da640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980ae210 con 0x7fa7980ca720 2023-10-16T15:55:42.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa79e0da640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980acc30 con 0x7fa7980ca2a0 2023-10-16T15:55:42.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa7980adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa7980adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38372/0 (socket says 172.21.15.29:38372) 2023-10-16T15:55:42.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 -- 172.21.15.29:0/733034881 learned_addr learned my addr 172.21.15.29:0/733034881 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:42.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 -- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980ca2a0 msgr2=0x7fa7981e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:42.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980ca2a0 0x7fa7981e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 -- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7980ca720 msgr2=0x7fa7981e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7980ca720 0x7fa7981e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.663+0000 7fa7977fe640 1 -- 172.21.15.29:0/733034881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7980acf10 con 0x7fa784003970 2023-10-16T15:55:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.664+0000 7fa7977fe640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa7980adcd0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fa788000f80 tx=0x7fa788004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.664+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788003b90 con 0x7fa784003970 2023-10-16T15:55:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.664+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa788003d30 con 0x7fa784003970 2023-10-16T15:55:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.664+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788003ea0 con 0x7fa784003970 2023-10-16T15:55:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.664+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7980ad1f0 con 0x7fa784003970 2023-10-16T15:55:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.664+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7980ad7b0 con 0x7fa784003970 2023-10-16T15:55:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.665+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79804e160 con 0x7fa784003970 2023-10-16T15:55:42.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.666+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa78803b040 con 0x7fa784003970 2023-10-16T15:55:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.668+0000 7fa78ffff640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa77806b0b0 0x7fa77806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.668+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7fa7880ab8a0 con 0x7fa784003970 2023-10-16T15:55:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.669+0000 7fa796ffd640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa77806b0b0 0x7fa77806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:42.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.670+0000 7fa796ffd640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa77806b0b0 0x7fa77806d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7981f5b50 tx=0x7fa780009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:42.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.673+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7880765b0 con 0x7fa784003970 2023-10-16T15:55:42.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:42.903+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa79806f3a0 con 0x7fa784003970 2023-10-16T15:55:43.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.409+0000 7fa78ffff640 1 -- 172.21.15.29:0/733034881 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v141) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7880390c0 con 0x7fa784003970 2023-10-16T15:55:43.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa77806b0b0 msgr2=0x7fa77806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa77806b0b0 0x7fa77806d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7981f5b50 tx=0x7fa780009040 comp rx=0 tx=0).stop 2023-10-16T15:55:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 msgr2=0x7fa7980adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa7980adcd0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fa788000f80 tx=0x7fa788004ab0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 shutdown_connections 2023-10-16T15:55:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fa77806b0b0 0x7fa77806d570 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa7980ca720 0x7fa7981e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa7980ca2a0 0x7fa7981e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 --2- 172.21.15.29:0/733034881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa784003970 0x7fa7980adcd0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.412+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 >> 172.21.15.29:0/733034881 conn(0x7fa7980d0ab0 msgr2=0x7fa7980cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:43.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.413+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 shutdown_connections 2023-10-16T15:55:43.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.413+0000 7fa79e0da640 1 -- 172.21.15.29:0/733034881 wait complete. 2023-10-16T15:55:43.436 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T15:55:43.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.662+0000 7f8cc359e640 1 Processor -- start 2023-10-16T15:55:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.662+0000 7f8cc359e640 1 -- start start 2023-10-16T15:55:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cc406c040 0x7f8cc406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:43.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc359e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8cc40ad970 0x7f8cc415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc406c960 con 0x7f8cc415ebc0 2023-10-16T15:55:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc415b9d0 con 0x7f8cc406c040 2023-10-16T15:55:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc359e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc415bcb0 con 0x7f8cc40ad970 2023-10-16T15:55:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cc406c040 0x7f8cc406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cc406c040 0x7f8cc406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38376/0 (socket says 172.21.15.29:38376) 2023-10-16T15:55:43.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.664+0000 7f8cc259c640 1 -- 172.21.15.29:0/3900693963 learned_addr learned my addr 172.21.15.29:0/3900693963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.665+0000 7f8cc259c640 1 -- 172.21.15.29:0/3900693963 >> v2:172.21.15.170:3300/0 conn(0x7f8cc40ad970 msgr2=0x7f8cc415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.665+0000 7f8cb9d9b640 1 --2- 172.21.15.29:0/3900693963 >> v2:172.21.15.170:3300/0 conn(0x7f8cc40ad970 0x7f8cc415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.665+0000 7f8cc259c640 1 --2- 172.21.15.29:0/3900693963 >> v2:172.21.15.170:3300/0 conn(0x7f8cc40ad970 0x7f8cc415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.665+0000 7f8cc259c640 1 -- 172.21.15.29:0/3900693963 >> v1:172.21.15.29:6789/0 conn(0x7f8cc415ebc0 legacy=0x7f8cc40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.665+0000 7f8cc259c640 1 -- 172.21.15.29:0/3900693963 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc415bf90 con 0x7f8cc406c040 2023-10-16T15:55:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.665+0000 7f8cb9d9b640 1 --2- 172.21.15.29:0/3900693963 >> v2:172.21.15.170:3300/0 conn(0x7f8cc40ad970 0x7f8cc415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.666+0000 7f8cc259c640 1 --2- 172.21.15.29:0/3900693963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c040 0x7f8cc406c420 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40cb060 tx=0x7f8cb402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f681a39398319220 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.666+0000 7f8cc1d9b640 1 -- 172.21.15.29:0/3900693963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4030870 con 0x7f8cc406c040 2023-10-16T15:55:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.666+0000 7f8cc1d9b640 1 -- 172.21.15.29:0/3900693963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb40309d0 con 0x7f8cc406c040 2023-10-16T15:55:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.666+0000 7f8cc1d9b640 1 -- 172.21.15.29:0/3900693963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4030d60 con 0x7f8cc406c040 2023-10-16T15:55:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 -- 172.21.15.29:0/3900693963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c040 msgr2=0x7f8cc406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 --2- 172.21.15.29:0/3900693963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c040 0x7f8cc406c420 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40cb060 tx=0x7f8cb402b9b0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 -- 172.21.15.29:0/3900693963 shutdown_connections 2023-10-16T15:55:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 --2- 172.21.15.29:0/3900693963 >> v2:172.21.15.170:3300/0 conn(0x7f8cc40ad970 0x7f8cc415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 --2- 172.21.15.29:0/3900693963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c040 0x7f8cc406c420 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 -- 172.21.15.29:0/3900693963 >> 172.21.15.29:0/3900693963 conn(0x7f8cc40ce6a0 msgr2=0x7f8cc40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 -- 172.21.15.29:0/3900693963 shutdown_connections 2023-10-16T15:55:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.667+0000 7f8cc359e640 1 -- 172.21.15.29:0/3900693963 wait complete. 2023-10-16T15:55:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.669+0000 7f8cc359e640 1 Processor -- start 2023-10-16T15:55:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.669+0000 7f8cc359e640 1 -- start start 2023-10-16T15:55:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cc359e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cc406c040 0x7f8cc414ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cc359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 0x7f8cc41f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cc359e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cc415ebc0 0x7f8cc41564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cc359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc41f3580 con 0x7f8cc40ad970 2023-10-16T15:55:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cc359e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40c8cf0 con 0x7f8cc406c040 2023-10-16T15:55:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cc359e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40c8f90 con 0x7f8cc415ebc0 2023-10-16T15:55:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cb9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 0x7f8cc41f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cb9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 0x7f8cc41f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38388/0 (socket says 172.21.15.29:38388) 2023-10-16T15:55:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.670+0000 7f8cb9d9b640 1 -- 172.21.15.29:0/709021472 learned_addr learned my addr 172.21.15.29:0/709021472 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cc259c640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cc406c040 0x7f8cc414ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cc2d9d640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cc415ebc0 0x7f8cc41564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cb9d9b640 1 -- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cc415ebc0 msgr2=0x7f8cc41564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cb9d9b640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cc415ebc0 0x7f8cc41564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cb9d9b640 1 -- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cc406c040 msgr2=0x7f8cc414ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cb9d9b640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cc406c040 0x7f8cc414ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cb9d9b640 1 -- 172.21.15.29:0/709021472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb40094d0 con 0x7f8cc40ad970 2023-10-16T15:55:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cb9d9b640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 0x7f8cc41f2de0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00cde0 tx=0x7f8cac00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00f680 con 0x7f8cc40ad970 2023-10-16T15:55:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac00f7e0 con 0x7f8cc40ad970 2023-10-16T15:55:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40c9270 con 0x7f8cc40ad970 2023-10-16T15:55:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.671+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00fb90 con 0x7f8cc40ad970 2023-10-16T15:55:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.672+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc40c9830 con 0x7f8cc40ad970 2023-10-16T15:55:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.673+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc406b330 con 0x7f8cc40ad970 2023-10-16T15:55:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.674+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8cac00b040 con 0x7f8cc40ad970 2023-10-16T15:55:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.674+0000 7f8cbb7fe640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8ca406b0e0 0x7f8ca406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.675+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f8cac08a630 con 0x7f8cc40ad970 2023-10-16T15:55:43.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.675+0000 7f8cc259c640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8ca406b0e0 0x7f8ca406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:43.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.676+0000 7f8cc259c640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8ca406b0e0 0x7f8ca406d5a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4036000 tx=0x7f8cb403b040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:43.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.679+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8cac058ce0 con 0x7f8cc40ad970 2023-10-16T15:55:43.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.904+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cc404e9b0 con 0x7f8cc40ad970 2023-10-16T15:55:43.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.905+0000 7f8cbb7fe640 1 -- 172.21.15.29:0/709021472 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f8cac058870 con 0x7f8cc40ad970 2023-10-16T15:55:43.908 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:43.908 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:55:39.826547+0000","modified":"2023-10-16T15:55:42.906111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12345},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12345":{"gid":12345,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2425185007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2425185007},{"type":"v1","addr":"172.21.15.29:6835","nonce":2425185007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T15:55:43.908 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-16T15:55:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8ca406b0e0 msgr2=0x7f8ca406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8ca406b0e0 0x7f8ca406d5a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4036000 tx=0x7f8cb403b040 comp rx=0 tx=0).stop 2023-10-16T15:55:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 msgr2=0x7f8cc41f2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 0x7f8cc41f2de0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00cde0 tx=0x7f8cac00a650 comp rx=0 tx=0).stop 2023-10-16T15:55:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 shutdown_connections 2023-10-16T15:55:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f8ca406b0e0 0x7f8ca406d5a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc40ad970 0x7f8cc41f2de0 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8cc415ebc0 0x7f8cc41564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 --2- 172.21.15.29:0/709021472 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8cc406c040 0x7f8cc414ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 >> 172.21.15.29:0/709021472 conn(0x7f8cc40ce6a0 msgr2=0x7f8cc40c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.908+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 shutdown_connections 2023-10-16T15:55:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:43.909+0000 7f8cc359e640 1 -- 172.21.15.29:0/709021472 wait complete. 2023-10-16T15:55:43.928 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:44.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.144+0000 7fdf8b260640 1 Processor -- start 2023-10-16T15:55:44.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.144+0000 7fdf8b260640 1 -- start start 2023-10-16T15:55:44.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.145+0000 7fdf8b260640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fdf8415cd70 0x7fdf840d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.145+0000 7fdf8b260640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf840cded0 0x7fdf840b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.145+0000 7fdf8b260640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8415d150 con 0x7fdf840b8480 2023-10-16T15:55:44.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.145+0000 7fdf8b260640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840d1180 con 0x7fdf840cded0 2023-10-16T15:55:44.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.145+0000 7fdf8b260640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840d1460 con 0x7fdf8415cd70 2023-10-16T15:55:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf840cded0 0x7fdf840b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf840cded0 0x7fdf840b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53960/0 (socket says 172.21.15.29:53960) 2023-10-16T15:55:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1310913804 learned_addr learned my addr 172.21.15.29:0/1310913804 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8a25e640 1 --2- 172.21.15.29:0/1310913804 >> v2:172.21.15.170:3300/0 conn(0x7fdf8415cd70 0x7fdf840d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1310913804 >> v2:172.21.15.170:3300/0 conn(0x7fdf8415cd70 msgr2=0x7fdf840d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 --2- 172.21.15.29:0/1310913804 >> v2:172.21.15.170:3300/0 conn(0x7fdf8415cd70 0x7fdf840d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1310913804 >> v1:172.21.15.170:6789/0 conn(0x7fdf840b8480 legacy=0x7fdf840cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.146+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1310913804 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840d1740 con 0x7fdf840cded0 2023-10-16T15:55:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.148+0000 7fdf8aa5f640 1 --2- 172.21.15.29:0/1310913804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840cded0 0x7fdf840b5c90 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fdf840b56a0 tx=0x7fdf7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8661e836f5cbdd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.148+0000 7fdf8925c640 1 -- 172.21.15.29:0/1310913804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c031870 con 0x7fdf840cded0 2023-10-16T15:55:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.148+0000 7fdf8925c640 1 -- 172.21.15.29:0/1310913804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c0319d0 con 0x7fdf840cded0 2023-10-16T15:55:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.148+0000 7fdf8925c640 1 -- 172.21.15.29:0/1310913804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c031d60 con 0x7fdf840cded0 2023-10-16T15:55:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 -- 172.21.15.29:0/1310913804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840cded0 msgr2=0x7fdf840b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1310913804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840cded0 0x7fdf840b5c90 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fdf840b56a0 tx=0x7fdf7c02b910 comp rx=0 tx=0).stop 2023-10-16T15:55:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 -- 172.21.15.29:0/1310913804 shutdown_connections 2023-10-16T15:55:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1310913804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840cded0 0x7fdf840b5c90 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1310913804 >> v2:172.21.15.170:3300/0 conn(0x7fdf8415cd70 0x7fdf840d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 -- 172.21.15.29:0/1310913804 >> 172.21.15.29:0/1310913804 conn(0x7fdf840cf570 msgr2=0x7fdf840cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 -- 172.21.15.29:0/1310913804 shutdown_connections 2023-10-16T15:55:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.149+0000 7fdf8b260640 1 -- 172.21.15.29:0/1310913804 wait complete. 2023-10-16T15:55:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.151+0000 7fdf8b260640 1 Processor -- start 2023-10-16T15:55:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.152+0000 7fdf8b260640 1 -- start start 2023-10-16T15:55:44.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8b260640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf840b8480 0x7fdf84151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8b260640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf840cded0 0x7fdf840cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8b260640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 0x7fdf840cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8b260640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840ab2c0 con 0x7fdf8415cd70 2023-10-16T15:55:44.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8b260640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840aae30 con 0x7fdf840b8480 2023-10-16T15:55:44.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8b260640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840aa220 con 0x7fdf840cded0 2023-10-16T15:55:44.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8aa5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 0x7fdf840cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.153+0000 7fdf8aa5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 0x7fdf840cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53974/0 (socket says 172.21.15.29:53974) 2023-10-16T15:55:44.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1387176621 learned_addr learned my addr 172.21.15.29:0/1387176621 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:44.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8a25e640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf840b8480 0x7fdf84151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf89a5d640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf840cded0 0x7fdf840cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf840cded0 msgr2=0x7fdf840cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf840cded0 0x7fdf840cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf840b8480 msgr2=0x7fdf84151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf840b8480 0x7fdf84151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 -- 172.21.15.29:0/1387176621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0094d0 con 0x7fdf8415cd70 2023-10-16T15:55:44.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.154+0000 7fdf8aa5f640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 0x7fdf840cc2d0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c02b950 tx=0x7fdf7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:44.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.155+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c03abe0 con 0x7fdf8415cd70 2023-10-16T15:55:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.155+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c03ad40 con 0x7fdf8415cd70 2023-10-16T15:55:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.155+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c044630 con 0x7fdf8415cd70 2023-10-16T15:55:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.155+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf840aa500 con 0x7fdf8415cd70 2023-10-16T15:55:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.155+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf840aaac0 con 0x7fdf8415cd70 2023-10-16T15:55:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.156+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf840aac50 con 0x7fdf8415cd70 2023-10-16T15:55:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.157+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdf7c044790 con 0x7fdf8415cd70 2023-10-16T15:55:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.158+0000 7fdf7affd640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fdf6806b0b0 0x7fdf6806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.158+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7fdf7c02d040 con 0x7fdf8415cd70 2023-10-16T15:55:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.159+0000 7fdf8a25e640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fdf6806b0b0 0x7fdf6806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.160+0000 7fdf8a25e640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fdf6806b0b0 0x7fdf6806d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf800099c0 tx=0x7fdf80003990 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:44.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.163+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf7c0deca0 con 0x7fdf8415cd70 2023-10-16T15:55:44.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.357+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf8404e9b0 con 0x7fdf8415cd70 2023-10-16T15:55:44.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.359+0000 7fdf7affd640 1 -- 172.21.15.29:0/1387176621 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+16633 (secure 0 0 0) 0x7fdf7c035090 con 0x7fdf8415cd70 2023-10-16T15:55:44.361 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:44.362 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":266,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:41.360602+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:44.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fdf6806b0b0 msgr2=0x7fdf6806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fdf6806b0b0 0x7fdf6806d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf800099c0 tx=0x7fdf80003990 comp rx=0 tx=0).stop 2023-10-16T15:55:44.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 msgr2=0x7fdf840cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 0x7fdf840cc2d0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c02b950 tx=0x7fdf7c0046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 shutdown_connections 2023-10-16T15:55:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fdf6806b0b0 0x7fdf6806d570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8415cd70 0x7fdf840cc2d0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fdf840cded0 0x7fdf840cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.361+0000 7fdf8b260640 1 --2- 172.21.15.29:0/1387176621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf840b8480 0x7fdf84151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.362+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 >> 172.21.15.29:0/1387176621 conn(0x7fdf840cf570 msgr2=0x7fdf840b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:44.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.362+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 shutdown_connections 2023-10-16T15:55:44.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.362+0000 7fdf8b260640 1 -- 172.21.15.29:0/1387176621 wait complete. 2023-10-16T15:55:44.384 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T15:55:44.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:55:44.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.606+0000 7f445eb1d640 1 Processor -- start 2023-10-16T15:55:44.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.607+0000 7f445eb1d640 1 -- start start 2023-10-16T15:55:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.608+0000 7f445eb1d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f445815b010 0x7f44580ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.608+0000 7f445eb1d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580afee0 0x7f44580b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445eb1d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f44580cf020 0x7f44580cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445eb1d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44580cf9f0 con 0x7f445815b010 2023-10-16T15:55:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445eb1d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44580cee00 con 0x7f44580cf020 2023-10-16T15:55:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445eb1d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580aeba0 con 0x7f44580afee0 2023-10-16T15:55:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f445815b010 0x7f44580ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f445815b010 0x7f44580ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53988/0 (socket says 172.21.15.29:53988) 2023-10-16T15:55:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 -- 172.21.15.29:0/1149376283 learned_addr learned my addr 172.21.15.29:0/1149376283 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 -- 172.21.15.29:0/1149376283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580afee0 msgr2=0x7f44580b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 --2- 172.21.15.29:0/1149376283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580afee0 0x7f44580b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 -- 172.21.15.29:0/1149376283 >> v2:172.21.15.170:3300/0 conn(0x7f44580cf020 msgr2=0x7f44580cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 --2- 172.21.15.29:0/1149376283 >> v2:172.21.15.170:3300/0 conn(0x7f44580cf020 0x7f44580cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.609+0000 7f445c8b9640 1 -- 172.21.15.29:0/1149376283 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580ae270 con 0x7f445815b010 2023-10-16T15:55:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.611+0000 7f445c8b9640 1 --2- 172.21.15.29:0/1149376283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f44580ce820 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f44580cb120 tx=0x7f445402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79525bdc36d227ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.611+0000 7f444f7fe640 1 -- 172.21.15.29:0/1149376283 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454004230 con 0x7f445815b010 2023-10-16T15:55:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.611+0000 7f444f7fe640 1 -- 172.21.15.29:0/1149376283 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4454004390 con 0x7f445815b010 2023-10-16T15:55:44.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.611+0000 7f444f7fe640 1 -- 172.21.15.29:0/1149376283 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454030840 con 0x7f445815b010 2023-10-16T15:55:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.611+0000 7f445eb1d640 1 -- 172.21.15.29:0/1149376283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 msgr2=0x7f44580ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.611+0000 7f445eb1d640 1 --2- 172.21.15.29:0/1149376283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f44580ce820 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f44580cb120 tx=0x7f445402b980 comp rx=0 tx=0).stop 2023-10-16T15:55:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 -- 172.21.15.29:0/1149376283 shutdown_connections 2023-10-16T15:55:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 --2- 172.21.15.29:0/1149376283 >> v2:172.21.15.170:3300/0 conn(0x7f44580cf020 0x7f44580cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 --2- 172.21.15.29:0/1149376283 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580afee0 0x7f44580b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 --2- 172.21.15.29:0/1149376283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f44580ce820 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 -- 172.21.15.29:0/1149376283 >> 172.21.15.29:0/1149376283 conn(0x7f44580d0ad0 msgr2=0x7f44580d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 -- 172.21.15.29:0/1149376283 shutdown_connections 2023-10-16T15:55:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.612+0000 7f445eb1d640 1 -- 172.21.15.29:0/1149376283 wait complete. 2023-10-16T15:55:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.613+0000 7f445eb1d640 1 Processor -- start 2023-10-16T15:55:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.614+0000 7f445eb1d640 1 -- start start 2023-10-16T15:55:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.614+0000 7f445eb1d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44580afee0 0x7f4458155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.614+0000 7f445eb1d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580cf020 0x7f44580b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.614+0000 7f445eb1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f445806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445eb1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580ca780 con 0x7f445815b010 2023-10-16T15:55:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445eb1d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580b0e80 con 0x7f44580afee0 2023-10-16T15:55:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445eb1d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580b2870 con 0x7f44580cf020 2023-10-16T15:55:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f445806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f445806e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53996/0 (socket says 172.21.15.29:53996) 2023-10-16T15:55:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 -- 172.21.15.29:0/2899763145 learned_addr learned my addr 172.21.15.29:0/2899763145 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445c8b9640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44580afee0 0x7f4458155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f444ffff640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580cf020 0x7f44580b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 -- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580cf020 msgr2=0x7f44580b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580cf020 0x7f44580b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 -- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44580afee0 msgr2=0x7f4458155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44580afee0 0x7f4458155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f445d0ba640 1 -- 172.21.15.29:0/2899763145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454009460 con 0x7f445815b010 2023-10-16T15:55:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.615+0000 7f444ffff640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580cf020 0x7f44580b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.616+0000 7f445d0ba640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f445806e6e0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f445000b970 tx=0x7f445000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.616+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000cb40 con 0x7f445815b010 2023-10-16T15:55:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.616+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445000ccc0 con 0x7f445815b010 2023-10-16T15:55:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.616+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44580b2b50 con 0x7f445815b010 2023-10-16T15:55:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.616+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450014680 con 0x7f445815b010 2023-10-16T15:55:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.616+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4458156870 con 0x7f445815b010 2023-10-16T15:55:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.617+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44580b3070 con 0x7f445815b010 2023-10-16T15:55:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.618+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f445001e440 con 0x7f445815b010 2023-10-16T15:55:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.619+0000 7f444d7fa640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f444406b0b0 0x7f444406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.619+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f4450089a80 con 0x7f445815b010 2023-10-16T15:55:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.620+0000 7f445c8b9640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f444406b0b0 0x7f444406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:44.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.621+0000 7f445c8b9640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f444406b0b0 0x7f444406d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4454035000 tx=0x7f4454008030 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:44.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.624+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44580b3070 con 0x7f445815b010 2023-10-16T15:55:44.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.813+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 --> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44580b29e0 con 0x7f444406b0b0 2023-10-16T15:55:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.843+0000 7f444d7fa640 1 -- 172.21.15.29:0/2899763145 <== mgr.9472 v2:172.21.15.170:6832/3469082932 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287113 (secure 0 0 0) 0x7f44580b29e0 con 0x7f444406b0b0 2023-10-16T15:55:44.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:44.851 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:55:44.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.849+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f444406b0b0 msgr2=0x7f444406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.849+0000 7f445eb1d640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f444406b0b0 0x7f444406d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4454035000 tx=0x7f4454008030 comp rx=0 tx=0).stop 2023-10-16T15:55:44.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.849+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 msgr2=0x7f445806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:44.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.849+0000 7f445eb1d640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f445806e6e0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f445000b970 tx=0x7f445000be30 comp rx=0 tx=0).stop 2023-10-16T15:55:44.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 shutdown_connections 2023-10-16T15:55:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f444406b0b0 0x7f444406d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f445815b010 0x7f445806e6e0 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f44580cf020 0x7f44580b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 --2- 172.21.15.29:0/2899763145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f44580afee0 0x7f4458155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:44.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 >> 172.21.15.29:0/2899763145 conn(0x7f44580d0ad0 msgr2=0x7f445815b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 shutdown_connections 2023-10-16T15:55:44.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:44.850+0000 7f445eb1d640 1 -- 172.21.15.29:0/2899763145 wait complete. 2023-10-16T15:55:44.874 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-16T15:55:43.818453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2487,"num_read_kb":5168,"num_write":689,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2825,"ondisk_log_size":2825,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437952,"kb_used_data":27464,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754536768,"statfs":{"total":773094113280,"available":772645650432,"internally_reserved":0,"allocated":28123136,"data_stored":14480148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":420204089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":201,"num_write":63,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.485222"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464368+0000","last_change":"2023-10-16T15:55:40.387281+0000","last_active":"2023-10-16T15:55:41.464368+0000","last_peered":"2023-10-16T15:55:41.464368+0000","last_clean":"2023-10-16T15:55:41.464368+0000","last_became_active":"2023-10-16T15:55:40.386970+0000","last_became_peered":"2023-10-16T15:55:40.386970+0000","last_unstale":"2023-10-16T15:55:41.464368+0000","last_undegraded":"2023-10-16T15:55:41.464368+0000","last_fullsized":"2023-10-16T15:55:41.464368+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:10.351916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463588+0000","last_change":"2023-10-16T15:55:40.389422+0000","last_active":"2023-10-16T15:55:41.463588+0000","last_peered":"2023-10-16T15:55:41.463588+0000","last_clean":"2023-10-16T15:55:41.463588+0000","last_became_active":"2023-10-16T15:55:40.388623+0000","last_became_peered":"2023-10-16T15:55:40.388623+0000","last_unstale":"2023-10-16T15:55:41.463588+0000","last_undegraded":"2023-10-16T15:55:41.463588+0000","last_fullsized":"2023-10-16T15:55:41.463588+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:50.360866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462704+0000","last_change":"2023-10-16T15:55:40.382115+0000","last_active":"2023-10-16T15:55:41.462704+0000","last_peered":"2023-10-16T15:55:41.462704+0000","last_clean":"2023-10-16T15:55:41.462704+0000","last_became_active":"2023-10-16T15:55:40.380673+0000","last_became_peered":"2023-10-16T15:55:40.380673+0000","last_unstale":"2023-10-16T15:55:41.462704+0000","last_undegraded":"2023-10-16T15:55:41.462704+0000","last_fullsized":"2023-10-16T15:55:41.462704+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:22.188972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373330+0000","last_change":"2023-10-16T15:55:40.394800+0000","last_active":"2023-10-16T15:55:41.373330+0000","last_peered":"2023-10-16T15:55:41.373330+0000","last_clean":"2023-10-16T15:55:41.373330+0000","last_became_active":"2023-10-16T15:55:40.394532+0000","last_became_peered":"2023-10-16T15:55:40.394532+0000","last_unstale":"2023-10-16T15:55:41.373330+0000","last_undegraded":"2023-10-16T15:55:41.373330+0000","last_fullsized":"2023-10-16T15:55:41.373330+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:43.929734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462700+0000","last_change":"2023-10-16T15:55:40.387329+0000","last_active":"2023-10-16T15:55:41.462700+0000","last_peered":"2023-10-16T15:55:41.462700+0000","last_clean":"2023-10-16T15:55:41.462700+0000","last_became_active":"2023-10-16T15:55:40.384947+0000","last_became_peered":"2023-10-16T15:55:40.384947+0000","last_unstale":"2023-10-16T15:55:41.462700+0000","last_undegraded":"2023-10-16T15:55:41.462700+0000","last_fullsized":"2023-10-16T15:55:41.462700+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:55.956352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461700+0000","last_change":"2023-10-16T15:55:40.391628+0000","last_active":"2023-10-16T15:55:41.461700+0000","last_peered":"2023-10-16T15:55:41.461700+0000","last_clean":"2023-10-16T15:55:41.461700+0000","last_became_active":"2023-10-16T15:55:40.390173+0000","last_became_peered":"2023-10-16T15:55:40.390173+0000","last_unstale":"2023-10-16T15:55:41.461700+0000","last_undegraded":"2023-10-16T15:55:41.461700+0000","last_fullsized":"2023-10-16T15:55:41.461700+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:32.301642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464028+0000","last_change":"2023-10-16T15:55:40.416742+0000","last_active":"2023-10-16T15:55:41.464028+0000","last_peered":"2023-10-16T15:55:41.464028+0000","last_clean":"2023-10-16T15:55:41.464028+0000","last_became_active":"2023-10-16T15:55:40.416470+0000","last_became_peered":"2023-10-16T15:55:40.416470+0000","last_unstale":"2023-10-16T15:55:41.464028+0000","last_undegraded":"2023-10-16T15:55:41.464028+0000","last_fullsized":"2023-10-16T15:55:41.464028+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:05.778899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461464+0000","last_change":"2023-10-16T15:55:40.415928+0000","last_active":"2023-10-16T15:55:41.461464+0000","last_peered":"2023-10-16T15:55:41.461464+0000","last_clean":"2023-10-16T15:55:41.461464+0000","last_became_active":"2023-10-16T15:55:40.414893+0000","last_became_peered":"2023-10-16T15:55:40.414893+0000","last_unstale":"2023-10-16T15:55:41.461464+0000","last_undegraded":"2023-10-16T15:55:41.461464+0000","last_fullsized":"2023-10-16T15:55:41.461464+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:17.348639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368475+0000","last_change":"2023-10-16T15:55:40.383222+0000","last_active":"2023-10-16T15:55:41.368475+0000","last_peered":"2023-10-16T15:55:41.368475+0000","last_clean":"2023-10-16T15:55:41.368475+0000","last_became_active":"2023-10-16T15:55:40.382883+0000","last_became_peered":"2023-10-16T15:55:40.382883+0000","last_unstale":"2023-10-16T15:55:41.368475+0000","last_undegraded":"2023-10-16T15:55:41.368475+0000","last_fullsized":"2023-10-16T15:55:41.368475+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:05.778899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373951+0000","last_change":"2023-10-16T15:55:40.387547+0000","last_active":"2023-10-16T15:55:41.373951+0000","last_peered":"2023-10-16T15:55:41.373951+0000","last_clean":"2023-10-16T15:55:41.373951+0000","last_became_active":"2023-10-16T15:55:40.387250+0000","last_became_peered":"2023-10-16T15:55:40.387250+0000","last_unstale":"2023-10-16T15:55:41.373951+0000","last_undegraded":"2023-10-16T15:55:41.373951+0000","last_fullsized":"2023-10-16T15:55:41.373951+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:53.612916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.371846+0000","last_change":"2023-10-16T15:55:40.385178+0000","last_active":"2023-10-16T15:55:41.371846+0000","last_peered":"2023-10-16T15:55:41.371846+0000","last_clean":"2023-10-16T15:55:41.371846+0000","last_became_active":"2023-10-16T15:55:40.383382+0000","last_became_peered":"2023-10-16T15:55:40.383382+0000","last_unstale":"2023-10-16T15:55:41.371846+0000","last_undegraded":"2023-10-16T15:55:41.371846+0000","last_fullsized":"2023-10-16T15:55:41.371846+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:44.990876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463813+0000","last_change":"2023-10-16T15:55:40.380601+0000","last_active":"2023-10-16T15:55:41.463813+0000","last_peered":"2023-10-16T15:55:41.463813+0000","last_clean":"2023-10-16T15:55:41.463813+0000","last_became_active":"2023-10-16T15:55:40.380282+0000","last_became_peered":"2023-10-16T15:55:40.380282+0000","last_unstale":"2023-10-16T15:55:41.463813+0000","last_undegraded":"2023-10-16T15:55:41.463813+0000","last_fullsized":"2023-10-16T15:55:41.463813+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.146234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464682+0000","last_change":"2023-10-16T15:55:40.383331+0000","last_active":"2023-10-16T15:55:41.464682+0000","last_peered":"2023-10-16T15:55:41.464682+0000","last_clean":"2023-10-16T15:55:41.464682+0000","last_became_active":"2023-10-16T15:55:40.382103+0000","last_became_peered":"2023-10-16T15:55:40.382103+0000","last_unstale":"2023-10-16T15:55:41.464682+0000","last_undegraded":"2023-10-16T15:55:41.464682+0000","last_fullsized":"2023-10-16T15:55:41.464682+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:57.219840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372604+0000","last_change":"2023-10-16T15:55:40.409428+0000","last_active":"2023-10-16T15:55:41.372604+0000","last_peered":"2023-10-16T15:55:41.372604+0000","last_clean":"2023-10-16T15:55:41.372604+0000","last_became_active":"2023-10-16T15:55:40.408749+0000","last_became_peered":"2023-10-16T15:55:40.408749+0000","last_unstale":"2023-10-16T15:55:41.372604+0000","last_undegraded":"2023-10-16T15:55:41.372604+0000","last_fullsized":"2023-10-16T15:55:41.372604+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:00.848692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464180+0000","last_change":"2023-10-16T15:55:40.414412+0000","last_active":"2023-10-16T15:55:41.464180+0000","last_peered":"2023-10-16T15:55:41.464180+0000","last_clean":"2023-10-16T15:55:41.464180+0000","last_became_active":"2023-10-16T15:55:40.414082+0000","last_became_peered":"2023-10-16T15:55:40.414082+0000","last_unstale":"2023-10-16T15:55:41.464180+0000","last_undegraded":"2023-10-16T15:55:41.464180+0000","last_fullsized":"2023-10-16T15:55:41.464180+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:02.429267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461475+0000","last_change":"2023-10-16T15:55:40.416890+0000","last_active":"2023-10-16T15:55:41.461475+0000","last_peered":"2023-10-16T15:55:41.461475+0000","last_clean":"2023-10-16T15:55:41.461475+0000","last_became_active":"2023-10-16T15:55:40.414569+0000","last_became_peered":"2023-10-16T15:55:40.414569+0000","last_unstale":"2023-10-16T15:55:41.461475+0000","last_undegraded":"2023-10-16T15:55:41.461475+0000","last_fullsized":"2023-10-16T15:55:41.461475+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:22.736777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368591+0000","last_change":"2023-10-16T15:55:40.400249+0000","last_active":"2023-10-16T15:55:41.368591+0000","last_peered":"2023-10-16T15:55:41.368591+0000","last_clean":"2023-10-16T15:55:41.368591+0000","last_became_active":"2023-10-16T15:55:40.399938+0000","last_became_peered":"2023-10-16T15:55:40.399938+0000","last_unstale":"2023-10-16T15:55:41.368591+0000","last_undegraded":"2023-10-16T15:55:41.368591+0000","last_fullsized":"2023-10-16T15:55:41.368591+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:00.643688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463440+0000","last_change":"2023-10-16T15:55:40.401905+0000","last_active":"2023-10-16T15:55:41.463440+0000","last_peered":"2023-10-16T15:55:41.463440+0000","last_clean":"2023-10-16T15:55:41.463440+0000","last_became_active":"2023-10-16T15:55:40.401443+0000","last_became_peered":"2023-10-16T15:55:40.401443+0000","last_unstale":"2023-10-16T15:55:41.463440+0000","last_undegraded":"2023-10-16T15:55:41.463440+0000","last_fullsized":"2023-10-16T15:55:41.463440+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:34.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463362+0000","last_change":"2023-10-16T15:55:40.418498+0000","last_active":"2023-10-16T15:55:41.463362+0000","last_peered":"2023-10-16T15:55:41.463362+0000","last_clean":"2023-10-16T15:55:41.463362+0000","last_became_active":"2023-10-16T15:55:40.418198+0000","last_became_peered":"2023-10-16T15:55:40.418198+0000","last_unstale":"2023-10-16T15:55:41.463362+0000","last_undegraded":"2023-10-16T15:55:41.463362+0000","last_fullsized":"2023-10-16T15:55:41.463362+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:25.827807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464064+0000","last_change":"2023-10-16T15:55:40.409785+0000","last_active":"2023-10-16T15:55:41.464064+0000","last_peered":"2023-10-16T15:55:41.464064+0000","last_clean":"2023-10-16T15:55:41.464064+0000","last_became_active":"2023-10-16T15:55:40.409511+0000","last_became_peered":"2023-10-16T15:55:40.409511+0000","last_unstale":"2023-10-16T15:55:41.464064+0000","last_undegraded":"2023-10-16T15:55:41.464064+0000","last_fullsized":"2023-10-16T15:55:41.464064+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:00.643688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464895+0000","last_change":"2023-10-16T15:55:40.381870+0000","last_active":"2023-10-16T15:55:41.464895+0000","last_peered":"2023-10-16T15:55:41.464895+0000","last_clean":"2023-10-16T15:55:41.464895+0000","last_became_active":"2023-10-16T15:55:40.381465+0000","last_became_peered":"2023-10-16T15:55:40.381465+0000","last_unstale":"2023-10-16T15:55:41.464895+0000","last_undegraded":"2023-10-16T15:55:41.464895+0000","last_fullsized":"2023-10-16T15:55:41.464895+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:59.515158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461838+0000","last_change":"2023-10-16T15:55:40.394214+0000","last_active":"2023-10-16T15:55:41.461838+0000","last_peered":"2023-10-16T15:55:41.461838+0000","last_clean":"2023-10-16T15:55:41.461838+0000","last_became_active":"2023-10-16T15:55:40.393912+0000","last_became_peered":"2023-10-16T15:55:40.393912+0000","last_unstale":"2023-10-16T15:55:41.461838+0000","last_undegraded":"2023-10-16T15:55:41.461838+0000","last_fullsized":"2023-10-16T15:55:41.461838+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:06.788057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460824+0000","last_change":"2023-10-16T15:55:40.417190+0000","last_active":"2023-10-16T15:55:41.460824+0000","last_peered":"2023-10-16T15:55:41.460824+0000","last_clean":"2023-10-16T15:55:41.460824+0000","last_became_active":"2023-10-16T15:55:40.416483+0000","last_became_peered":"2023-10-16T15:55:40.416483+0000","last_unstale":"2023-10-16T15:55:41.460824+0000","last_undegraded":"2023-10-16T15:55:41.460824+0000","last_fullsized":"2023-10-16T15:55:41.460824+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:53:11.743284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461296+0000","last_change":"2023-10-16T15:55:40.415757+0000","last_active":"2023-10-16T15:55:41.461296+0000","last_peered":"2023-10-16T15:55:41.461296+0000","last_clean":"2023-10-16T15:55:41.461296+0000","last_became_active":"2023-10-16T15:55:40.415415+0000","last_became_peered":"2023-10-16T15:55:40.415415+0000","last_unstale":"2023-10-16T15:55:41.461296+0000","last_undegraded":"2023-10-16T15:55:41.461296+0000","last_fullsized":"2023-10-16T15:55:41.461296+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:25.538277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462483+0000","last_change":"2023-10-16T15:55:40.382989+0000","last_active":"2023-10-16T15:55:41.462483+0000","last_peered":"2023-10-16T15:55:41.462483+0000","last_clean":"2023-10-16T15:55:41.462483+0000","last_became_active":"2023-10-16T15:55:40.382698+0000","last_became_peered":"2023-10-16T15:55:40.382698+0000","last_unstale":"2023-10-16T15:55:41.462483+0000","last_undegraded":"2023-10-16T15:55:41.462483+0000","last_fullsized":"2023-10-16T15:55:41.462483+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:49.737461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464688+0000","last_change":"2023-10-16T15:55:40.386135+0000","last_active":"2023-10-16T15:55:41.464688+0000","last_peered":"2023-10-16T15:55:41.464688+0000","last_clean":"2023-10-16T15:55:41.464688+0000","last_became_active":"2023-10-16T15:55:40.385751+0000","last_became_peered":"2023-10-16T15:55:40.385751+0000","last_unstale":"2023-10-16T15:55:41.464688+0000","last_undegraded":"2023-10-16T15:55:41.464688+0000","last_fullsized":"2023-10-16T15:55:41.464688+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:09.667516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463716+0000","last_change":"2023-10-16T15:55:40.414957+0000","last_active":"2023-10-16T15:55:41.463716+0000","last_peered":"2023-10-16T15:55:41.463716+0000","last_clean":"2023-10-16T15:55:41.463716+0000","last_became_active":"2023-10-16T15:55:40.414673+0000","last_became_peered":"2023-10-16T15:55:40.414673+0000","last_unstale":"2023-10-16T15:55:41.463716+0000","last_undegraded":"2023-10-16T15:55:41.463716+0000","last_fullsized":"2023-10-16T15:55:41.463716+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:24.434830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373088+0000","last_change":"2023-10-16T15:55:40.389402+0000","last_active":"2023-10-16T15:55:41.373088+0000","last_peered":"2023-10-16T15:55:41.373088+0000","last_clean":"2023-10-16T15:55:41.373088+0000","last_became_active":"2023-10-16T15:55:40.389059+0000","last_became_peered":"2023-10-16T15:55:40.389059+0000","last_unstale":"2023-10-16T15:55:41.373088+0000","last_undegraded":"2023-10-16T15:55:41.373088+0000","last_fullsized":"2023-10-16T15:55:41.373088+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:17.938812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462559+0000","last_change":"2023-10-16T15:55:40.414324+0000","last_active":"2023-10-16T15:55:41.462559+0000","last_peered":"2023-10-16T15:55:41.462559+0000","last_clean":"2023-10-16T15:55:41.462559+0000","last_became_active":"2023-10-16T15:55:40.413979+0000","last_became_peered":"2023-10-16T15:55:40.413979+0000","last_unstale":"2023-10-16T15:55:41.462559+0000","last_undegraded":"2023-10-16T15:55:41.462559+0000","last_fullsized":"2023-10-16T15:55:41.462559+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:51.502048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464425+0000","last_change":"2023-10-16T15:55:40.413964+0000","last_active":"2023-10-16T15:55:41.464425+0000","last_peered":"2023-10-16T15:55:41.464425+0000","last_clean":"2023-10-16T15:55:41.464425+0000","last_became_active":"2023-10-16T15:55:40.409602+0000","last_became_peered":"2023-10-16T15:55:40.409602+0000","last_unstale":"2023-10-16T15:55:41.464425+0000","last_undegraded":"2023-10-16T15:55:41.464425+0000","last_fullsized":"2023-10-16T15:55:41.464425+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:58.706145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367328+0000","last_change":"2023-10-16T15:55:40.398402+0000","last_active":"2023-10-16T15:55:41.367328+0000","last_peered":"2023-10-16T15:55:41.367328+0000","last_clean":"2023-10-16T15:55:41.367328+0000","last_became_active":"2023-10-16T15:55:40.395289+0000","last_became_peered":"2023-10-16T15:55:40.395289+0000","last_unstale":"2023-10-16T15:55:41.367328+0000","last_undegraded":"2023-10-16T15:55:41.367328+0000","last_fullsized":"2023-10-16T15:55:41.367328+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:39.230917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464618+0000","last_change":"2023-10-16T15:55:40.397194+0000","last_active":"2023-10-16T15:55:41.464618+0000","last_peered":"2023-10-16T15:55:41.464618+0000","last_clean":"2023-10-16T15:55:41.464618+0000","last_became_active":"2023-10-16T15:55:40.396924+0000","last_became_peered":"2023-10-16T15:55:40.396924+0000","last_unstale":"2023-10-16T15:55:41.464618+0000","last_undegraded":"2023-10-16T15:55:41.464618+0000","last_fullsized":"2023-10-16T15:55:41.464618+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:37.392830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"265'4","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464556+0000","last_change":"2023-10-16T15:55:39.377649+0000","last_active":"2023-10-16T15:55:41.464556+0000","last_peered":"2023-10-16T15:55:41.464556+0000","last_clean":"2023-10-16T15:55:41.464556+0000","last_became_active":"2023-10-16T15:55:39.377037+0000","last_became_peered":"2023-10-16T15:55:39.377037+0000","last_unstale":"2023-10-16T15:55:41.464556+0000","last_undegraded":"2023-10-16T15:55:41.464556+0000","last_fullsized":"2023-10-16T15:55:41.464556+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:23.520225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373584+0000","last_change":"2023-10-16T15:55:40.384026+0000","last_active":"2023-10-16T15:55:41.373584+0000","last_peered":"2023-10-16T15:55:41.373584+0000","last_clean":"2023-10-16T15:55:41.373584+0000","last_became_active":"2023-10-16T15:55:40.383658+0000","last_became_peered":"2023-10-16T15:55:40.383658+0000","last_unstale":"2023-10-16T15:55:41.373584+0000","last_undegraded":"2023-10-16T15:55:41.373584+0000","last_fullsized":"2023-10-16T15:55:41.373584+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:42.460984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368558+0000","last_change":"2023-10-16T15:55:39.365271+0000","last_active":"2023-10-16T15:55:41.368558+0000","last_peered":"2023-10-16T15:55:41.368558+0000","last_clean":"2023-10-16T15:55:41.368558+0000","last_became_active":"2023-10-16T15:55:39.363945+0000","last_became_peered":"2023-10-16T15:55:39.363945+0000","last_unstale":"2023-10-16T15:55:41.368558+0000","last_undegraded":"2023-10-16T15:55:41.368558+0000","last_fullsized":"2023-10-16T15:55:41.368558+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:56.878210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463369+0000","last_change":"2023-10-16T15:55:40.386790+0000","last_active":"2023-10-16T15:55:41.463369+0000","last_peered":"2023-10-16T15:55:41.463369+0000","last_clean":"2023-10-16T15:55:41.463369+0000","last_became_active":"2023-10-16T15:55:40.384638+0000","last_became_peered":"2023-10-16T15:55:40.384638+0000","last_unstale":"2023-10-16T15:55:41.463369+0000","last_undegraded":"2023-10-16T15:55:41.463369+0000","last_fullsized":"2023-10-16T15:55:41.463369+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:00.468527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368033+0000","last_change":"2023-10-16T15:55:39.376491+0000","last_active":"2023-10-16T15:55:41.368033+0000","last_peered":"2023-10-16T15:55:41.368033+0000","last_clean":"2023-10-16T15:55:41.368033+0000","last_became_active":"2023-10-16T15:55:39.376187+0000","last_became_peered":"2023-10-16T15:55:39.376187+0000","last_unstale":"2023-10-16T15:55:41.368033+0000","last_undegraded":"2023-10-16T15:55:41.368033+0000","last_fullsized":"2023-10-16T15:55:41.368033+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.212193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462904+0000","last_change":"2023-10-16T15:55:40.385279+0000","last_active":"2023-10-16T15:55:41.462904+0000","last_peered":"2023-10-16T15:55:41.462904+0000","last_clean":"2023-10-16T15:55:41.462904+0000","last_became_active":"2023-10-16T15:55:40.384995+0000","last_became_peered":"2023-10-16T15:55:40.384995+0000","last_unstale":"2023-10-16T15:55:41.462904+0000","last_undegraded":"2023-10-16T15:55:41.462904+0000","last_fullsized":"2023-10-16T15:55:41.462904+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:10.414031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464373+0000","last_change":"2023-10-16T15:55:39.375356+0000","last_active":"2023-10-16T15:55:41.464373+0000","last_peered":"2023-10-16T15:55:41.464373+0000","last_clean":"2023-10-16T15:55:41.464373+0000","last_became_active":"2023-10-16T15:55:39.374963+0000","last_became_peered":"2023-10-16T15:55:39.374963+0000","last_unstale":"2023-10-16T15:55:41.464373+0000","last_undegraded":"2023-10-16T15:55:41.464373+0000","last_fullsized":"2023-10-16T15:55:41.464373+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:15.714641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460706+0000","last_change":"2023-10-16T15:55:40.385508+0000","last_active":"2023-10-16T15:55:41.460706+0000","last_peered":"2023-10-16T15:55:41.460706+0000","last_clean":"2023-10-16T15:55:41.460706+0000","last_became_active":"2023-10-16T15:55:40.385185+0000","last_became_peered":"2023-10-16T15:55:40.385185+0000","last_unstale":"2023-10-16T15:55:41.460706+0000","last_undegraded":"2023-10-16T15:55:41.460706+0000","last_fullsized":"2023-10-16T15:55:41.460706+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:35.454918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462030+0000","last_change":"2023-10-16T15:55:39.378795+0000","last_active":"2023-10-16T15:55:41.462030+0000","last_peered":"2023-10-16T15:55:41.462030+0000","last_clean":"2023-10-16T15:55:41.462030+0000","last_became_active":"2023-10-16T15:55:39.378515+0000","last_became_peered":"2023-10-16T15:55:39.378515+0000","last_unstale":"2023-10-16T15:55:41.462030+0000","last_undegraded":"2023-10-16T15:55:41.462030+0000","last_fullsized":"2023-10-16T15:55:41.462030+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:30.014727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462918+0000","last_change":"2023-10-16T15:55:40.410500+0000","last_active":"2023-10-16T15:55:41.462918+0000","last_peered":"2023-10-16T15:55:41.462918+0000","last_clean":"2023-10-16T15:55:41.462918+0000","last_became_active":"2023-10-16T15:55:40.409313+0000","last_became_peered":"2023-10-16T15:55:40.409313+0000","last_unstale":"2023-10-16T15:55:41.462918+0000","last_undegraded":"2023-10-16T15:55:41.462918+0000","last_fullsized":"2023-10-16T15:55:41.462918+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:39.230917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462890+0000","last_change":"2023-10-16T15:55:39.375431+0000","last_active":"2023-10-16T15:55:41.462890+0000","last_peered":"2023-10-16T15:55:41.462890+0000","last_clean":"2023-10-16T15:55:41.462890+0000","last_became_active":"2023-10-16T15:55:39.374524+0000","last_became_peered":"2023-10-16T15:55:39.374524+0000","last_unstale":"2023-10-16T15:55:41.462890+0000","last_undegraded":"2023-10-16T15:55:41.462890+0000","last_fullsized":"2023-10-16T15:55:41.462890+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:38.611799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367406+0000","last_change":"2023-10-16T15:55:40.391186+0000","last_active":"2023-10-16T15:55:41.367406+0000","last_peered":"2023-10-16T15:55:41.367406+0000","last_clean":"2023-10-16T15:55:41.367406+0000","last_became_active":"2023-10-16T15:55:40.390887+0000","last_became_peered":"2023-10-16T15:55:40.390887+0000","last_unstale":"2023-10-16T15:55:41.367406+0000","last_undegraded":"2023-10-16T15:55:41.367406+0000","last_fullsized":"2023-10-16T15:55:41.367406+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:11.462063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462359+0000","last_change":"2023-10-16T15:55:39.373139+0000","last_active":"2023-10-16T15:55:41.462359+0000","last_peered":"2023-10-16T15:55:41.462359+0000","last_clean":"2023-10-16T15:55:41.462359+0000","last_became_active":"2023-10-16T15:55:39.372758+0000","last_became_peered":"2023-10-16T15:55:39.372758+0000","last_unstale":"2023-10-16T15:55:41.462359+0000","last_undegraded":"2023-10-16T15:55:41.462359+0000","last_fullsized":"2023-10-16T15:55:41.462359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:46.380751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462569+0000","last_change":"2023-10-16T15:55:40.397291+0000","last_active":"2023-10-16T15:55:41.462569+0000","last_peered":"2023-10-16T15:55:41.462569+0000","last_clean":"2023-10-16T15:55:41.462569+0000","last_became_active":"2023-10-16T15:55:40.397022+0000","last_became_peered":"2023-10-16T15:55:40.397022+0000","last_unstale":"2023-10-16T15:55:41.462569+0000","last_undegraded":"2023-10-16T15:55:41.462569+0000","last_fullsized":"2023-10-16T15:55:41.462569+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:50.749801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463774+0000","last_change":"2023-10-16T15:55:39.366393+0000","last_active":"2023-10-16T15:55:41.463774+0000","last_peered":"2023-10-16T15:55:41.463774+0000","last_clean":"2023-10-16T15:55:41.463774+0000","last_became_active":"2023-10-16T15:55:39.365879+0000","last_became_peered":"2023-10-16T15:55:39.365879+0000","last_unstale":"2023-10-16T15:55:41.463774+0000","last_undegraded":"2023-10-16T15:55:41.463774+0000","last_fullsized":"2023-10-16T15:55:41.463774+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:54.810472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463783+0000","last_change":"2023-10-16T15:55:40.401719+0000","last_active":"2023-10-16T15:55:41.463783+0000","last_peered":"2023-10-16T15:55:41.463783+0000","last_clean":"2023-10-16T15:55:41.463783+0000","last_became_active":"2023-10-16T15:55:40.401046+0000","last_became_peered":"2023-10-16T15:55:40.401046+0000","last_unstale":"2023-10-16T15:55:41.463783+0000","last_undegraded":"2023-10-16T15:55:41.463783+0000","last_fullsized":"2023-10-16T15:55:41.463783+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:44.450650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373476+0000","last_change":"2023-10-16T15:55:39.369873+0000","last_active":"2023-10-16T15:55:41.373476+0000","last_peered":"2023-10-16T15:55:41.373476+0000","last_clean":"2023-10-16T15:55:41.373476+0000","last_became_active":"2023-10-16T15:55:39.369512+0000","last_became_peered":"2023-10-16T15:55:39.369512+0000","last_unstale":"2023-10-16T15:55:41.373476+0000","last_undegraded":"2023-10-16T15:55:41.373476+0000","last_fullsized":"2023-10-16T15:55:41.373476+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:58.501117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373282+0000","last_change":"2023-10-16T15:55:40.411562+0000","last_active":"2023-10-16T15:55:41.373282+0000","last_peered":"2023-10-16T15:55:41.373282+0000","last_clean":"2023-10-16T15:55:41.373282+0000","last_became_active":"2023-10-16T15:55:40.411271+0000","last_became_peered":"2023-10-16T15:55:40.411271+0000","last_unstale":"2023-10-16T15:55:41.373282+0000","last_undegraded":"2023-10-16T15:55:41.373282+0000","last_fullsized":"2023-10-16T15:55:41.373282+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:37.304909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372885+0000","last_change":"2023-10-16T15:55:39.369380+0000","last_active":"2023-10-16T15:55:41.372885+0000","last_peered":"2023-10-16T15:55:41.372885+0000","last_clean":"2023-10-16T15:55:41.372885+0000","last_became_active":"2023-10-16T15:55:39.369039+0000","last_became_peered":"2023-10-16T15:55:39.369039+0000","last_unstale":"2023-10-16T15:55:41.372885+0000","last_undegraded":"2023-10-16T15:55:41.372885+0000","last_fullsized":"2023-10-16T15:55:41.372885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:54.476845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372857+0000","last_change":"2023-10-16T15:55:40.386505+0000","last_active":"2023-10-16T15:55:41.372857+0000","last_peered":"2023-10-16T15:55:41.372857+0000","last_clean":"2023-10-16T15:55:41.372857+0000","last_became_active":"2023-10-16T15:55:40.386164+0000","last_became_peered":"2023-10-16T15:55:40.386164+0000","last_unstale":"2023-10-16T15:55:41.372857+0000","last_undegraded":"2023-10-16T15:55:41.372857+0000","last_fullsized":"2023-10-16T15:55:41.372857+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:03.594800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"265'3","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368269+0000","last_change":"2023-10-16T15:55:39.376511+0000","last_active":"2023-10-16T15:55:41.368269+0000","last_peered":"2023-10-16T15:55:41.368269+0000","last_clean":"2023-10-16T15:55:41.368269+0000","last_became_active":"2023-10-16T15:55:39.376225+0000","last_became_peered":"2023-10-16T15:55:39.376225+0000","last_unstale":"2023-10-16T15:55:41.368269+0000","last_undegraded":"2023-10-16T15:55:41.368269+0000","last_fullsized":"2023-10-16T15:55:41.368269+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:52.598875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373617+0000","last_change":"2023-10-16T15:55:40.385691+0000","last_active":"2023-10-16T15:55:41.373617+0000","last_peered":"2023-10-16T15:55:41.373617+0000","last_clean":"2023-10-16T15:55:41.373617+0000","last_became_active":"2023-10-16T15:55:40.382892+0000","last_became_peered":"2023-10-16T15:55:40.382892+0000","last_unstale":"2023-10-16T15:55:41.373617+0000","last_undegraded":"2023-10-16T15:55:41.373617+0000","last_fullsized":"2023-10-16T15:55:41.373617+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:02.226234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"265'6","reported_seq":18,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463665+0000","last_change":"2023-10-16T15:55:39.375502+0000","last_active":"2023-10-16T15:55:41.463665+0000","last_peered":"2023-10-16T15:55:41.463665+0000","last_clean":"2023-10-16T15:55:41.463665+0000","last_became_active":"2023-10-16T15:55:39.375141+0000","last_became_peered":"2023-10-16T15:55:39.375141+0000","last_unstale":"2023-10-16T15:55:41.463665+0000","last_undegraded":"2023-10-16T15:55:41.463665+0000","last_fullsized":"2023-10-16T15:55:41.463665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:50.230725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462124+0000","last_change":"2023-10-16T15:55:40.410525+0000","last_active":"2023-10-16T15:55:41.462124+0000","last_peered":"2023-10-16T15:55:41.462124+0000","last_clean":"2023-10-16T15:55:41.462124+0000","last_became_active":"2023-10-16T15:55:40.409376+0000","last_became_peered":"2023-10-16T15:55:40.409376+0000","last_unstale":"2023-10-16T15:55:41.462124+0000","last_undegraded":"2023-10-16T15:55:41.462124+0000","last_fullsized":"2023-10-16T15:55:41.462124+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:02.266659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461359+0000","last_change":"2023-10-16T15:55:39.372116+0000","last_active":"2023-10-16T15:55:41.461359+0000","last_peered":"2023-10-16T15:55:41.461359+0000","last_clean":"2023-10-16T15:55:41.461359+0000","last_became_active":"2023-10-16T15:55:39.369639+0000","last_became_peered":"2023-10-16T15:55:39.369639+0000","last_unstale":"2023-10-16T15:55:41.461359+0000","last_undegraded":"2023-10-16T15:55:41.461359+0000","last_fullsized":"2023-10-16T15:55:41.461359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:44.611503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462690+0000","last_change":"2023-10-16T15:55:40.403239+0000","last_active":"2023-10-16T15:55:41.462690+0000","last_peered":"2023-10-16T15:55:41.462690+0000","last_clean":"2023-10-16T15:55:41.462690+0000","last_became_active":"2023-10-16T15:55:40.402967+0000","last_became_peered":"2023-10-16T15:55:40.402967+0000","last_unstale":"2023-10-16T15:55:41.462690+0000","last_undegraded":"2023-10-16T15:55:41.462690+0000","last_fullsized":"2023-10-16T15:55:41.462690+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:23.617775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460482+0000","last_change":"2023-10-16T15:55:39.370774+0000","last_active":"2023-10-16T15:55:41.460482+0000","last_peered":"2023-10-16T15:55:41.460482+0000","last_clean":"2023-10-16T15:55:41.460482+0000","last_became_active":"2023-10-16T15:55:39.370493+0000","last_became_peered":"2023-10-16T15:55:39.370493+0000","last_unstale":"2023-10-16T15:55:41.460482+0000","last_undegraded":"2023-10-16T15:55:41.460482+0000","last_fullsized":"2023-10-16T15:55:41.460482+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:51:06.129842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.370588+0000","last_change":"2023-10-16T15:55:40.402774+0000","last_active":"2023-10-16T15:55:41.370588+0000","last_peered":"2023-10-16T15:55:41.370588+0000","last_clean":"2023-10-16T15:55:41.370588+0000","last_became_active":"2023-10-16T15:55:40.402477+0000","last_became_peered":"2023-10-16T15:55:40.402477+0000","last_unstale":"2023-10-16T15:55:41.370588+0000","last_undegraded":"2023-10-16T15:55:41.370588+0000","last_fullsized":"2023-10-16T15:55:41.370588+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:40.495907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.371983+0000","last_change":"2023-10-16T15:55:39.369210+0000","last_active":"2023-10-16T15:55:41.371983+0000","last_peered":"2023-10-16T15:55:41.371983+0000","last_clean":"2023-10-16T15:55:41.371983+0000","last_became_active":"2023-10-16T15:55:39.368926+0000","last_became_peered":"2023-10-16T15:55:39.368926+0000","last_unstale":"2023-10-16T15:55:41.371983+0000","last_undegraded":"2023-10-16T15:55:41.371983+0000","last_fullsized":"2023-10-16T15:55:41.371983+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:56.205799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464540+0000","last_change":"2023-10-16T15:55:40.410297+0000","last_active":"2023-10-16T15:55:41.464540+0000","last_peered":"2023-10-16T15:55:41.464540+0000","last_clean":"2023-10-16T15:55:41.464540+0000","last_became_active":"2023-10-16T15:55:40.409046+0000","last_became_peered":"2023-10-16T15:55:40.409046+0000","last_unstale":"2023-10-16T15:55:41.464540+0000","last_undegraded":"2023-10-16T15:55:41.464540+0000","last_fullsized":"2023-10-16T15:55:41.464540+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:20.318733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"265'8","reported_seq":20,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460743+0000","last_change":"2023-10-16T15:55:39.374027+0000","last_active":"2023-10-16T15:55:41.460743+0000","last_peered":"2023-10-16T15:55:41.460743+0000","last_clean":"2023-10-16T15:55:41.460743+0000","last_became_active":"2023-10-16T15:55:39.373598+0000","last_became_peered":"2023-10-16T15:55:39.373598+0000","last_unstale":"2023-10-16T15:55:41.460743+0000","last_undegraded":"2023-10-16T15:55:41.460743+0000","last_fullsized":"2023-10-16T15:55:41.460743+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:36.466341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461297+0000","last_change":"2023-10-16T15:55:40.380158+0000","last_active":"2023-10-16T15:55:41.461297+0000","last_peered":"2023-10-16T15:55:41.461297+0000","last_clean":"2023-10-16T15:55:41.461297+0000","last_became_active":"2023-10-16T15:55:40.379861+0000","last_became_peered":"2023-10-16T15:55:40.379861+0000","last_unstale":"2023-10-16T15:55:41.461297+0000","last_undegraded":"2023-10-16T15:55:41.461297+0000","last_fullsized":"2023-10-16T15:55:41.461297+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:07:07.197038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373679+0000","last_change":"2023-10-16T15:55:39.368770+0000","last_active":"2023-10-16T15:55:41.373679+0000","last_peered":"2023-10-16T15:55:41.373679+0000","last_clean":"2023-10-16T15:55:41.373679+0000","last_became_active":"2023-10-16T15:55:39.368471+0000","last_became_peered":"2023-10-16T15:55:39.368471+0000","last_unstale":"2023-10-16T15:55:41.373679+0000","last_undegraded":"2023-10-16T15:55:41.373679+0000","last_fullsized":"2023-10-16T15:55:41.373679+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:56.501266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463160+0000","last_change":"2023-10-16T15:55:40.393143+0000","last_active":"2023-10-16T15:55:41.463160+0000","last_peered":"2023-10-16T15:55:41.463160+0000","last_clean":"2023-10-16T15:55:41.463160+0000","last_became_active":"2023-10-16T15:55:40.392886+0000","last_became_peered":"2023-10-16T15:55:40.392886+0000","last_unstale":"2023-10-16T15:55:41.463160+0000","last_undegraded":"2023-10-16T15:55:41.463160+0000","last_fullsized":"2023-10-16T15:55:41.463160+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:36.781554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464016+0000","last_change":"2023-10-16T15:55:39.372893+0000","last_active":"2023-10-16T15:55:41.464016+0000","last_peered":"2023-10-16T15:55:41.464016+0000","last_clean":"2023-10-16T15:55:41.464016+0000","last_became_active":"2023-10-16T15:55:39.372623+0000","last_became_peered":"2023-10-16T15:55:39.372623+0000","last_unstale":"2023-10-16T15:55:41.464016+0000","last_undegraded":"2023-10-16T15:55:41.464016+0000","last_fullsized":"2023-10-16T15:55:41.464016+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:13.813201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462605+0000","last_change":"2023-10-16T15:55:40.399956+0000","last_active":"2023-10-16T15:55:41.462605+0000","last_peered":"2023-10-16T15:55:41.462605+0000","last_clean":"2023-10-16T15:55:41.462605+0000","last_became_active":"2023-10-16T15:55:40.399669+0000","last_became_peered":"2023-10-16T15:55:40.399669+0000","last_unstale":"2023-10-16T15:55:41.462605+0000","last_undegraded":"2023-10-16T15:55:41.462605+0000","last_fullsized":"2023-10-16T15:55:41.462605+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:44.360499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373695+0000","last_change":"2023-10-16T15:55:39.371802+0000","last_active":"2023-10-16T15:55:41.373695+0000","last_peered":"2023-10-16T15:55:41.373695+0000","last_clean":"2023-10-16T15:55:41.373695+0000","last_became_active":"2023-10-16T15:55:39.371519+0000","last_became_peered":"2023-10-16T15:55:39.371519+0000","last_unstale":"2023-10-16T15:55:41.373695+0000","last_undegraded":"2023-10-16T15:55:41.373695+0000","last_fullsized":"2023-10-16T15:55:41.373695+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:15.714641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463325+0000","last_change":"2023-10-16T15:55:40.385206+0000","last_active":"2023-10-16T15:55:41.463325+0000","last_peered":"2023-10-16T15:55:41.463325+0000","last_clean":"2023-10-16T15:55:41.463325+0000","last_became_active":"2023-10-16T15:55:40.384921+0000","last_became_peered":"2023-10-16T15:55:40.384921+0000","last_unstale":"2023-10-16T15:55:41.463325+0000","last_undegraded":"2023-10-16T15:55:41.463325+0000","last_fullsized":"2023-10-16T15:55:41.463325+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:47.794581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373357+0000","last_change":"2023-10-16T15:55:39.361037+0000","last_active":"2023-10-16T15:55:41.373357+0000","last_peered":"2023-10-16T15:55:41.373357+0000","last_clean":"2023-10-16T15:55:41.373357+0000","last_became_active":"2023-10-16T15:55:39.359478+0000","last_became_peered":"2023-10-16T15:55:39.359478+0000","last_unstale":"2023-10-16T15:55:41.373357+0000","last_undegraded":"2023-10-16T15:55:41.373357+0000","last_fullsized":"2023-10-16T15:55:41.373357+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:17:05.050926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367589+0000","last_change":"2023-10-16T15:55:40.386722+0000","last_active":"2023-10-16T15:55:41.367589+0000","last_peered":"2023-10-16T15:55:41.367589+0000","last_clean":"2023-10-16T15:55:41.367589+0000","last_became_active":"2023-10-16T15:55:40.386406+0000","last_became_peered":"2023-10-16T15:55:40.386406+0000","last_unstale":"2023-10-16T15:55:41.367589+0000","last_undegraded":"2023-10-16T15:55:41.367589+0000","last_fullsized":"2023-10-16T15:55:41.367589+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:24.434830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.465085+0000","last_change":"2023-10-16T15:55:39.368103+0000","last_active":"2023-10-16T15:55:41.465085+0000","last_peered":"2023-10-16T15:55:41.465085+0000","last_clean":"2023-10-16T15:55:41.465085+0000","last_became_active":"2023-10-16T15:55:39.367824+0000","last_became_peered":"2023-10-16T15:55:39.367824+0000","last_unstale":"2023-10-16T15:55:41.465085+0000","last_undegraded":"2023-10-16T15:55:41.465085+0000","last_fullsized":"2023-10-16T15:55:41.465085+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:12.469700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463084+0000","last_change":"2023-10-16T15:55:40.403096+0000","last_active":"2023-10-16T15:55:41.463084+0000","last_peered":"2023-10-16T15:55:41.463084+0000","last_clean":"2023-10-16T15:55:41.463084+0000","last_became_active":"2023-10-16T15:55:40.402237+0000","last_became_peered":"2023-10-16T15:55:40.402237+0000","last_unstale":"2023-10-16T15:55:41.463084+0000","last_undegraded":"2023-10-16T15:55:41.463084+0000","last_fullsized":"2023-10-16T15:55:41.463084+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:55.638360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461927+0000","last_change":"2023-10-16T15:55:39.368966+0000","last_active":"2023-10-16T15:55:41.461927+0000","last_peered":"2023-10-16T15:55:41.461927+0000","last_clean":"2023-10-16T15:55:41.461927+0000","last_became_active":"2023-10-16T15:55:39.368663+0000","last_became_peered":"2023-10-16T15:55:39.368663+0000","last_unstale":"2023-10-16T15:55:41.461927+0000","last_undegraded":"2023-10-16T15:55:41.461927+0000","last_fullsized":"2023-10-16T15:55:41.461927+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:59.226241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462260+0000","last_change":"2023-10-16T15:55:40.402398+0000","last_active":"2023-10-16T15:55:41.462260+0000","last_peered":"2023-10-16T15:55:41.462260+0000","last_clean":"2023-10-16T15:55:41.462260+0000","last_became_active":"2023-10-16T15:55:40.401938+0000","last_became_peered":"2023-10-16T15:55:40.401938+0000","last_unstale":"2023-10-16T15:55:41.462260+0000","last_undegraded":"2023-10-16T15:55:41.462260+0000","last_fullsized":"2023-10-16T15:55:41.462260+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:34.114098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372032+0000","last_change":"2023-10-16T15:55:39.365241+0000","last_active":"2023-10-16T15:55:41.372032+0000","last_peered":"2023-10-16T15:55:41.372032+0000","last_clean":"2023-10-16T15:55:41.372032+0000","last_became_active":"2023-10-16T15:55:39.364546+0000","last_became_peered":"2023-10-16T15:55:39.364546+0000","last_unstale":"2023-10-16T15:55:41.372032+0000","last_undegraded":"2023-10-16T15:55:41.372032+0000","last_fullsized":"2023-10-16T15:55:41.372032+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:08.653475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464242+0000","last_change":"2023-10-16T15:55:40.384442+0000","last_active":"2023-10-16T15:55:41.464242+0000","last_peered":"2023-10-16T15:55:41.464242+0000","last_clean":"2023-10-16T15:55:41.464242+0000","last_became_active":"2023-10-16T15:55:40.384048+0000","last_became_peered":"2023-10-16T15:55:40.384048+0000","last_unstale":"2023-10-16T15:55:41.464242+0000","last_undegraded":"2023-10-16T15:55:41.464242+0000","last_fullsized":"2023-10-16T15:55:41.464242+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:17.324790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462067+0000","last_change":"2023-10-16T15:55:39.376672+0000","last_active":"2023-10-16T15:55:41.462067+0000","last_peered":"2023-10-16T15:55:41.462067+0000","last_clean":"2023-10-16T15:55:41.462067+0000","last_became_active":"2023-10-16T15:55:39.375874+0000","last_became_peered":"2023-10-16T15:55:39.375874+0000","last_unstale":"2023-10-16T15:55:41.462067+0000","last_undegraded":"2023-10-16T15:55:41.462067+0000","last_fullsized":"2023-10-16T15:55:41.462067+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:12.642092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463324+0000","last_change":"2023-10-16T15:55:40.414024+0000","last_active":"2023-10-16T15:55:41.463324+0000","last_peered":"2023-10-16T15:55:41.463324+0000","last_clean":"2023-10-16T15:55:41.463324+0000","last_became_active":"2023-10-16T15:55:40.409652+0000","last_became_peered":"2023-10-16T15:55:40.409652+0000","last_unstale":"2023-10-16T15:55:41.463324+0000","last_undegraded":"2023-10-16T15:55:41.463324+0000","last_fullsized":"2023-10-16T15:55:41.463324+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:11.462063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373492+0000","last_change":"2023-10-16T15:55:39.368876+0000","last_active":"2023-10-16T15:55:41.373492+0000","last_peered":"2023-10-16T15:55:41.373492+0000","last_clean":"2023-10-16T15:55:41.373492+0000","last_became_active":"2023-10-16T15:55:39.367570+0000","last_became_peered":"2023-10-16T15:55:39.367570+0000","last_unstale":"2023-10-16T15:55:41.373492+0000","last_undegraded":"2023-10-16T15:55:41.373492+0000","last_fullsized":"2023-10-16T15:55:41.373492+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:17.528310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463540+0000","last_change":"2023-10-16T15:55:40.411831+0000","last_active":"2023-10-16T15:55:41.463540+0000","last_peered":"2023-10-16T15:55:41.463540+0000","last_clean":"2023-10-16T15:55:41.463540+0000","last_became_active":"2023-10-16T15:55:40.410913+0000","last_became_peered":"2023-10-16T15:55:40.410913+0000","last_unstale":"2023-10-16T15:55:41.463540+0000","last_undegraded":"2023-10-16T15:55:41.463540+0000","last_fullsized":"2023-10-16T15:55:41.463540+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:47.394792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463746+0000","last_change":"2023-10-16T15:55:39.368288+0000","last_active":"2023-10-16T15:55:41.463746+0000","last_peered":"2023-10-16T15:55:41.463746+0000","last_clean":"2023-10-16T15:55:41.463746+0000","last_became_active":"2023-10-16T15:55:39.368003+0000","last_became_peered":"2023-10-16T15:55:39.368003+0000","last_unstale":"2023-10-16T15:55:41.463746+0000","last_undegraded":"2023-10-16T15:55:41.463746+0000","last_fullsized":"2023-10-16T15:55:41.463746+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:40:01.562908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464266+0000","last_change":"2023-10-16T15:55:40.402889+0000","last_active":"2023-10-16T15:55:41.464266+0000","last_peered":"2023-10-16T15:55:41.464266+0000","last_clean":"2023-10-16T15:55:41.464266+0000","last_became_active":"2023-10-16T15:55:40.402596+0000","last_became_peered":"2023-10-16T15:55:40.402596+0000","last_unstale":"2023-10-16T15:55:41.464266+0000","last_undegraded":"2023-10-16T15:55:41.464266+0000","last_fullsized":"2023-10-16T15:55:41.464266+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:36.800461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372098+0000","last_change":"2023-10-16T15:55:39.370080+0000","last_active":"2023-10-16T15:55:41.372098+0000","last_peered":"2023-10-16T15:55:41.372098+0000","last_clean":"2023-10-16T15:55:41.372098+0000","last_became_active":"2023-10-16T15:55:39.367029+0000","last_became_peered":"2023-10-16T15:55:39.367029+0000","last_unstale":"2023-10-16T15:55:41.372098+0000","last_undegraded":"2023-10-16T15:55:41.372098+0000","last_fullsized":"2023-10-16T15:55:41.372098+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:54.377409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463696+0000","last_change":"2023-10-16T15:55:40.383931+0000","last_active":"2023-10-16T15:55:41.463696+0000","last_peered":"2023-10-16T15:55:41.463696+0000","last_clean":"2023-10-16T15:55:41.463696+0000","last_became_active":"2023-10-16T15:55:40.383650+0000","last_became_peered":"2023-10-16T15:55:40.383650+0000","last_unstale":"2023-10-16T15:55:41.463696+0000","last_undegraded":"2023-10-16T15:55:41.463696+0000","last_fullsized":"2023-10-16T15:55:41.463696+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:57.544172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372952+0000","last_change":"2023-10-16T15:55:39.377996+0000","last_active":"2023-10-16T15:55:41.372952+0000","last_peered":"2023-10-16T15:55:41.372952+0000","last_clean":"2023-10-16T15:55:41.372952+0000","last_became_active":"2023-10-16T15:55:39.376673+0000","last_became_peered":"2023-10-16T15:55:39.376673+0000","last_unstale":"2023-10-16T15:55:41.372952+0000","last_undegraded":"2023-10-16T15:55:41.372952+0000","last_fullsized":"2023-10-16T15:55:41.372952+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:14.934855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367500+0000","last_change":"2023-10-16T15:55:40.387645+0000","last_active":"2023-10-16T15:55:41.367500+0000","last_peered":"2023-10-16T15:55:41.367500+0000","last_clean":"2023-10-16T15:55:41.367500+0000","last_became_active":"2023-10-16T15:55:40.386939+0000","last_became_peered":"2023-10-16T15:55:40.386939+0000","last_unstale":"2023-10-16T15:55:41.367500+0000","last_undegraded":"2023-10-16T15:55:41.367500+0000","last_fullsized":"2023-10-16T15:55:41.367500+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:25.827807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462928+0000","last_change":"2023-10-16T15:55:39.378661+0000","last_active":"2023-10-16T15:55:41.462928+0000","last_peered":"2023-10-16T15:55:41.462928+0000","last_clean":"2023-10-16T15:55:41.462928+0000","last_became_active":"2023-10-16T15:55:39.378375+0000","last_became_peered":"2023-10-16T15:55:39.378375+0000","last_unstale":"2023-10-16T15:55:41.462928+0000","last_undegraded":"2023-10-16T15:55:41.462928+0000","last_fullsized":"2023-10-16T15:55:41.462928+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:31.260722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463380+0000","last_change":"2023-10-16T15:55:40.398774+0000","last_active":"2023-10-16T15:55:41.463380+0000","last_peered":"2023-10-16T15:55:41.463380+0000","last_clean":"2023-10-16T15:55:41.463380+0000","last_became_active":"2023-10-16T15:55:40.398496+0000","last_became_peered":"2023-10-16T15:55:40.398496+0000","last_unstale":"2023-10-16T15:55:41.463380+0000","last_undegraded":"2023-10-16T15:55:41.463380+0000","last_fullsized":"2023-10-16T15:55:41.463380+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:38.036360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463199+0000","last_change":"2023-10-16T15:55:39.368481+0000","last_active":"2023-10-16T15:55:41.463199+0000","last_peered":"2023-10-16T15:55:41.463199+0000","last_clean":"2023-10-16T15:55:41.463199+0000","last_became_active":"2023-10-16T15:55:39.367089+0000","last_became_peered":"2023-10-16T15:55:39.367089+0000","last_unstale":"2023-10-16T15:55:41.463199+0000","last_undegraded":"2023-10-16T15:55:41.463199+0000","last_fullsized":"2023-10-16T15:55:41.463199+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:54.624319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462628+0000","last_change":"2023-10-16T15:55:40.403822+0000","last_active":"2023-10-16T15:55:41.462628+0000","last_peered":"2023-10-16T15:55:41.462628+0000","last_clean":"2023-10-16T15:55:41.462628+0000","last_became_active":"2023-10-16T15:55:40.402310+0000","last_became_peered":"2023-10-16T15:55:40.402310+0000","last_unstale":"2023-10-16T15:55:41.462628+0000","last_undegraded":"2023-10-16T15:55:41.462628+0000","last_fullsized":"2023-10-16T15:55:41.462628+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:22.411416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462615+0000","last_change":"2023-10-16T15:55:39.367291+0000","last_active":"2023-10-16T15:55:41.462615+0000","last_peered":"2023-10-16T15:55:41.462615+0000","last_clean":"2023-10-16T15:55:41.462615+0000","last_became_active":"2023-10-16T15:55:39.366958+0000","last_became_peered":"2023-10-16T15:55:39.366958+0000","last_unstale":"2023-10-16T15:55:41.462615+0000","last_undegraded":"2023-10-16T15:55:41.462615+0000","last_fullsized":"2023-10-16T15:55:41.462615+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:43.436609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.371795+0000","last_change":"2023-10-16T15:55:40.416692+0000","last_active":"2023-10-16T15:55:41.371795+0000","last_peered":"2023-10-16T15:55:41.371795+0000","last_clean":"2023-10-16T15:55:41.371795+0000","last_became_active":"2023-10-16T15:55:40.416407+0000","last_became_peered":"2023-10-16T15:55:40.416407+0000","last_unstale":"2023-10-16T15:55:41.371795+0000","last_undegraded":"2023-10-16T15:55:41.371795+0000","last_fullsized":"2023-10-16T15:55:41.371795+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:32.274763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462124+0000","last_change":"2023-10-16T15:55:39.366052+0000","last_active":"2023-10-16T15:55:41.462124+0000","last_peered":"2023-10-16T15:55:41.462124+0000","last_clean":"2023-10-16T15:55:41.462124+0000","last_became_active":"2023-10-16T15:55:39.365766+0000","last_became_peered":"2023-10-16T15:55:39.365766+0000","last_unstale":"2023-10-16T15:55:41.462124+0000","last_undegraded":"2023-10-16T15:55:41.462124+0000","last_fullsized":"2023-10-16T15:55:41.462124+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:24.152311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372122+0000","last_change":"2023-10-16T15:55:40.395448+0000","last_active":"2023-10-16T15:55:41.372122+0000","last_peered":"2023-10-16T15:55:41.372122+0000","last_clean":"2023-10-16T15:55:41.372122+0000","last_became_active":"2023-10-16T15:55:40.395178+0000","last_became_peered":"2023-10-16T15:55:40.395178+0000","last_unstale":"2023-10-16T15:55:41.372122+0000","last_undegraded":"2023-10-16T15:55:41.372122+0000","last_fullsized":"2023-10-16T15:55:41.372122+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:25:23.759665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"263'2787","reported_seq":3840,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T15:55:40.370189+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:55:40.370189+0000","last_peered":"2023-10-16T15:55:40.370189+0000","last_clean":"2023-10-16T15:55:40.370189+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:55:40.370189+0000","last_undegraded":"2023-10-16T15:55:40.370189+0000","last_fullsized":"2023-10-16T15:55:40.370189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2487,"num_read_kb":5168,"num_write":645,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2487,"num_read_kb":5168,"num_write":645,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576028,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57252,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314588,"statfs":{"total":96636764160,"available":96578138112,"internally_reserved":0,"allocated":3952640,"data_stored":2252192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.095,"15min":1.171},"min":{"1min":0.61499999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.6779999999999999,"5min":1.794,"15min":1.8879999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.1040000000000001,"15min":1.149},"min":{"1min":0.71899999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6299999999999999,"5min":1.796,"15min":1.796},"last":1.1439999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1080000000000001,"15min":1.1679999999999999},"min":{"1min":0.82999999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4610000000000001,"5min":1.76,"15min":1.8580000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.244,"15min":1.3460000000000001},"min":{"1min":0.69199999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.7809999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3100000000000001,"15min":1.331},"min":{"1min":0.80400000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7410000000000001,"5min":1.893,"15min":2.1269999999999998},"last":1.621},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2310000000000001,"15min":1.296},"min":{"1min":0.877,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6020000000000001,"5min":1.909,"15min":1.95},"last":1.319}]},{"osd":3,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.2529999999999999,"15min":1.3260000000000001},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5149999999999999,"5min":1.843,"15min":1.869},"last":1.397},{"interface":"front","average":{"1min":1.423,"5min":1.3999999999999999,"15min":1.4450000000000001},"min":{"1min":1.089,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.8500000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3819999999999999,"15min":1.383},"min":{"1min":1.054,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6479999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.573},{"interface":"front","average":{"1min":1.347,"5min":1.3089999999999999,"15min":1.339},"min":{"1min":0.88100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6439999999999999,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.3620000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4570000000000001,"15min":1.46},"min":{"1min":0.87,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.7889999999999999,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.502,"15min":1.5489999999999999},"min":{"1min":0.98599999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.04,"5min":2.04,"15min":2.3540000000000001},"last":1.9650000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.4550000000000001,"15min":1.4750000000000001},"min":{"1min":1.2390000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8260000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5669999999999999,"15min":1.5800000000000001},"min":{"1min":1.048,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":1.988,"15min":2.2599999999999998},"last":1.9159999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52512,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319328,"statfs":{"total":96636764160,"available":96582991872,"internally_reserved":0,"allocated":3362816,"data_stored":1661914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.208,"15min":1.206},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9099999999999999,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.2130000000000001,"15min":1.2430000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":1.5309999999999999,"5min":1.905,"15min":1.905},"last":1.3440000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.72199999999999998},"max":{"1min":1.675,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.107},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2889999999999999,"15min":1.3160000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1639999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2330000000000001,"15min":1.232},"min":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.611,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3859999999999999,"15min":1.3959999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8400000000000001,"5min":1.905,"15min":2.0339999999999998},"last":1.212}]},{"osd":3,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.361,"15min":1.391},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.9079999999999999,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.284,"15min":1.331},"min":{"1min":0.82799999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.554}]},{"osd":4,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.421,"15min":1.351},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.978,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.3},{"interface":"front","average":{"1min":1.512,"5min":1.458,"15min":1.4119999999999999},"min":{"1min":0.81100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0830000000000002,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.0189999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3759999999999999,"15min":1.357},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.67300000000000004},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.627},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5980000000000001,"15min":1.54},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1059999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.667}]},{"osd":7,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.6419999999999999,"15min":1.6040000000000001},"min":{"1min":0.86699999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.4940000000000002},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.599,"5min":1.6399999999999999,"15min":1.5980000000000001},"min":{"1min":0.98099999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.778}]}]},{"osd":5,"up_from":13,"seq":55834576027,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30320,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341520,"statfs":{"total":96636764160,"available":96605716480,"internally_reserved":0,"allocated":3379200,"data_stored":1664088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.111,"15min":1.1579999999999999},"min":{"1min":0.68899999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.802,"5min":1.8069999999999999,"15min":1.821},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.234,"15min":1.24},"min":{"1min":0.79300000000000004,"5min":0.622,"15min":0.622},"max":{"1min":1.766,"5min":2.738,"15min":2.738},"last":0.93000000000000005}]},{"osd":1,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.196,"15min":1.228},"min":{"1min":0.66600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6639999999999999,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.2410000000000001,"15min":1.2889999999999999},"min":{"1min":0.65300000000000002,"5min":0.443,"15min":0.443},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.5129999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.248},"min":{"1min":0.61799999999999999,"5min":0.505,"15min":0.505},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.339},{"interface":"front","average":{"1min":1.325,"5min":1.375,"15min":1.3939999999999999},"min":{"1min":0.77500000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.2160000000000002},"last":1.891}]},{"osd":3,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3300000000000001,"15min":1.335},"min":{"1min":0.753,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8720000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.3260000000000001,"15min":1.3740000000000001},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.966,"5min":1.966,"15min":2.1589999999999998},"last":1.232}]},{"osd":4,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.347},"min":{"1min":0.79900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.8500000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5049999999999999,"15min":1.5149999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0800000000000001,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.397}]},{"osd":6,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4630000000000001,"15min":1.385},"min":{"1min":0.81000000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0369999999999999,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":2.077},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.468,"15min":1.4319999999999999},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9870000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3}]},{"osd":7,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.5289999999999999,"15min":1.532},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.125,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.474},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.651,"15min":1.631},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.4340000000000002,"5min":3.109,"15min":3.109},"last":2.1890000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576028,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70568,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301272,"statfs":{"total":96636764160,"available":96564502528,"internally_reserved":0,"allocated":3366912,"data_stored":1661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.1419999999999999,"15min":1.194},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.512},{"interface":"front","average":{"1min":1.085,"5min":1.159,"15min":1.194},"min":{"1min":0.76300000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.4059999999999999,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1890000000000001,"15min":1.2310000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.629},"max":{"1min":1.6579999999999999,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.663},{"interface":"front","average":{"1min":1.254,"5min":1.284,"15min":1.3089999999999999},"min":{"1min":0.75900000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9750000000000001,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":0.83999999999999997}]},{"osd":2,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3939999999999999,"15min":1.4510000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7150000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.456,"15min":1.492},"min":{"1min":0.82499999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6819999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.8109999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.391,"15min":1.385},"min":{"1min":1.02,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8129999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.873},{"interface":"front","average":{"1min":1.232,"5min":1.341,"15min":1.383},"min":{"1min":0.92100000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.7509999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.744}]},{"osd":5,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.371,"15min":1.341},"min":{"1min":0.80200000000000005,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.849,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.617},{"interface":"front","average":{"1min":1.417,"5min":1.4730000000000001,"15min":1.4410000000000001},"min":{"1min":0.88900000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7949999999999999,"5min":2.161,"15min":2.4489999999999998},"last":1.196}]},{"osd":6,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.389,"15min":1.405},"min":{"1min":0.97699999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.29},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5629999999999999,"15min":1.526},"min":{"1min":1.1990000000000001,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.246,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.2350000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6100000000000001,"15min":1.583},"min":{"1min":1.04,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.9730000000000001,"5min":2.399,"15min":2.399},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.6000000000000001,"15min":1.601},"min":{"1min":1.129,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.4630000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56684,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315156,"statfs":{"total":96636764160,"available":96578719744,"internally_reserved":0,"allocated":3371008,"data_stored":1663436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92100000000000004,"15min":0.96799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.419,"5min":1.48,"15min":2.0430000000000001},"last":1.042},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0189999999999999,"15min":1.0580000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.722,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.995}]},{"osd":1,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94199999999999995,"15min":1.0189999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.776,"5min":1.776,"15min":2.589},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.147,"15min":1.204},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.556,"5min":1.748,"15min":2.5569999999999999},"last":1.264}]},{"osd":2,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.304,"15min":1.403},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.302},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3919999999999999,"15min":1.488},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.954,"5min":1.954,"15min":2.754},"last":1.7}]},{"osd":4,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2669999999999999,"15min":1.3440000000000001},"min":{"1min":0.79100000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9770000000000001,"5min":2.0110000000000001,"15min":2.512},"last":1.212},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3440000000000001,"15min":1.4119999999999999},"min":{"1min":0.76300000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.0249999999999999,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.3260000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.53,"15min":1.5620000000000001},"min":{"1min":0.94099999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2280000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4890000000000001,"15min":1.5629999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.2599999999999998,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.5589999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.46,"15min":1.5},"min":{"1min":0.97499999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.0680000000000001,"5min":2.173,"15min":2.2090000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.5649999999999999,"15min":1.6000000000000001},"min":{"1min":0.85299999999999998,"5min":0.749,"15min":0.749},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.7669999999999999},"last":1.5920000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.591,"15min":1.6779999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":1.7829999999999999},{"interface":"front","average":{"1min":1.571,"5min":1.579,"15min":1.6339999999999999},"min":{"1min":1.002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.133,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.637}]}]},{"osd":2,"up_from":13,"seq":55834576028,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56676,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315164,"statfs":{"total":96636764160,"available":96578727936,"internally_reserved":0,"allocated":3362816,"data_stored":1661914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.89300000000000002,"15min":0.93700000000000006},"min":{"1min":0.443,"5min":0.378,"15min":0.378},"max":{"1min":1.2749999999999999,"5min":1.726,"15min":1.76},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.093},"min":{"1min":0.39300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7789999999999999,"5min":1.804,"15min":1.8999999999999999},"last":1.6739999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0209999999999999,"15min":1.046},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.504,"5min":1.974,"15min":1.974},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.1479999999999999,"15min":1.224},"min":{"1min":0.55000000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.772,"5min":1.879,"15min":2.0779999999999998},"last":1.1879999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.161,"15min":1.2709999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.5289999999999999,"5min":1.704,"15min":2.1509999999999998},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.3180000000000001,"15min":1.365},"min":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.661,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.389}]},{"osd":4,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.448,"15min":1.4730000000000001},"min":{"1min":0.84499999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.9299999999999999,"5min":2.335,"15min":2.335},"last":2.0409999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.345,"15min":1.411},"min":{"1min":0.65900000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.901,"5min":1.915,"15min":2.1469999999999998},"last":1.28}]},{"osd":5,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.403,"15min":1.4730000000000001},"min":{"1min":0.72899999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8080000000000001,"5min":2.657,"15min":2.657},"last":2.008},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.5449999999999999,"15min":1.6100000000000001},"min":{"1min":0.81499999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.093,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.4790000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4570000000000001,"15min":1.5429999999999999},"min":{"1min":0.77900000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.861,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5589999999999999,"15min":1.6100000000000001},"min":{"1min":0.89300000000000002,"5min":0.753,"15min":0.753},"max":{"1min":1.8500000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.5}]},{"osd":7,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.5649999999999999,"15min":1.6499999999999999},"min":{"1min":0.85999999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.234,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":2.181},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.6499999999999999,"15min":1.722},"min":{"1min":1.0129999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.1890000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":2.0960000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576028,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52520,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319320,"statfs":{"total":96636764160,"available":96582983680,"internally_reserved":0,"allocated":3362816,"data_stored":1661858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.90900000000000003,"15min":0.93999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.351,"5min":1.49,"15min":1.8420000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98899999999999999,"15min":1.113},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6639999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.97999999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.94199999999999995,"15min":1.0209999999999999},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.764,"5min":1.764,"15min":2.286},"last":0.87},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.125,"15min":1.214},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7929999999999999,"15min":1.853},"last":0.74199999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.254,"15min":1.353},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.651,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1879999999999999,"15min":1.2929999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.532,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":0.85099999999999998}]},{"osd":4,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.319,"15min":1.365},"min":{"1min":0.89200000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.915,"5min":2.327,"15min":2.3809999999999998},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.47,"15min":1.5109999999999999},"min":{"1min":0.874,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.8759999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.4059999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.52},"min":{"1min":0.86499999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9830000000000001,"5min":2.456,"15min":2.456},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.4450000000000001,"15min":1.484},"min":{"1min":0.79200000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9490000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.0509999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.595,"15min":1.641},"min":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0489999999999999,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.635},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5880000000000001,"15min":1.637},"min":{"1min":0.84199999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.343,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.427}]},{"osd":7,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.6499999999999999,"15min":1.7310000000000001},"min":{"1min":0.79900000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1379999999999999,"5min":2.734,"15min":3.2090000000000001},"last":1.554},{"interface":"front","average":{"1min":1.381,"5min":1.5820000000000001,"15min":1.6399999999999999},"min":{"1min":0.89000000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1110000000000002,"5min":2.452,"15min":3.1539999999999999},"last":1.24}]}]},{"osd":0,"up_from":13,"seq":55834576028,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61420,"kb_used_data":3872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310420,"statfs":{"total":96636764160,"available":96573870080,"internally_reserved":0,"allocated":3964928,"data_stored":2252830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91000000000000003,"15min":0.94999999999999996},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.2789999999999999,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1180000000000001,"15min":1.2070000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.5800000000000001,"5min":2.036,"15min":2.036},"last":0.99299999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.1739999999999999,"15min":1.26},"min":{"1min":0.56499999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.117,"15min":1.2170000000000001},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.462,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.70699999999999996}]},{"osd":3,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.004,"15min":1.032},"min":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6339999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.032,"5min":1.175,"15min":1.2549999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.774,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.1180000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.319,"15min":1.337},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":2.137},"last":1.042},{"interface":"front","average":{"1min":1.327,"5min":1.401,"15min":1.4970000000000001},"min":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.9390000000000001,"5min":1.99,"15min":2.452},"last":1.073}]},{"osd":5,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.4239999999999999,"15min":1.476},"min":{"1min":0.66200000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8779999999999999,"5min":2.137,"15min":2.137},"last":1.026},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.345,"15min":1.411},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":2.0859999999999999,"15min":2.282},"last":1.155}]},{"osd":6,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5349999999999999,"15min":1.6259999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.7010000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.526,"15min":1.569},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.1869999999999998,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.2050000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.621,"15min":1.6779999999999999},"min":{"1min":0.82199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1480000000000001,"5min":2.339,"15min":2.339},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.5760000000000001,"15min":1.601},"min":{"1min":0.89200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.1259999999999999,"5min":2.1739999999999999,"15min":2.274},"last":1.341}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:55:44.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:55:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.148+0000 7f30a50d0640 1 Processor -- start 2023-10-16T15:55:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.148+0000 7f30a50d0640 1 -- start start 2023-10-16T15:55:45.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.149+0000 7f30a50d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 0x7f30a015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.149+0000 7f30a50d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f30a00ce390 0x7f30a0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.149+0000 7f30a50d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a015ed60 con 0x7f30a015aaa0 2023-10-16T15:55:45.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.149+0000 7f30a50d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a006b810 con 0x7f30a00ce390 2023-10-16T15:55:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.149+0000 7f30a50d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00707e0 con 0x7f30a015ae80 2023-10-16T15:55:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.150+0000 7f309ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30a015aaa0 0x7f30a00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60644/0 (socket says 172.21.15.29:60644) 2023-10-16T15:55:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.150+0000 7f309ed9d640 1 -- 172.21.15.29:0/2048263836 learned_addr learned my addr 172.21.15.29:0/2048263836 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.150+0000 7f309f59e640 1 --2- 172.21.15.29:0/2048263836 >> v2:172.21.15.170:3300/0 conn(0x7f30a00ce390 0x7f30a0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.150+0000 7f309e59c640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 0x7f30a015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.150+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104611159 0 0) 0x7f30a015ed60 con 0x7f30a015aaa0 2023-10-16T15:55:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.150+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3088003520 con 0x7f30a015aaa0 2023-10-16T15:55:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126586404 0 0) 0x7f3088003520 con 0x7f30a015aaa0 2023-10-16T15:55:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 msgr2=0x7f30a015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 0x7f30a015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 >> v2:172.21.15.170:3300/0 conn(0x7f30a00ce390 msgr2=0x7f30a0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 --2- 172.21.15.29:0/2048263836 >> v2:172.21.15.170:3300/0 conn(0x7f30a00ce390 0x7f30a0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a015ef90 con 0x7f30a015aaa0 2023-10-16T15:55:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f30900019f0 con 0x7f30a015aaa0 2023-10-16T15:55:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 >> v1:172.21.15.29:6789/0 conn(0x7f30a015aaa0 legacy=0x7f30a00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f3088003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30880047c0 0x7f3088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 0x7f3088005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a006b810 con 0x7f3088003970 2023-10-16T15:55:45.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00707e0 con 0x7f30880047c0 2023-10-16T15:55:45.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308800b980 con 0x7f30a015ae80 2023-10-16T15:55:45.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.151+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f3088003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.152+0000 7f309f59e640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 0x7f3088005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.152+0000 7f309ed9d640 1 -- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 msgr2=0x7f3088005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:45.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.152+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a015ae80 0x7f3088005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.152+0000 7f309ed9d640 1 -- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30880047c0 msgr2=0x7f3088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:45.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.152+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30880047c0 0x7f3088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.152+0000 7f309ed9d640 1 -- 172.21.15.29:0/2048263836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088003520 con 0x7f3088003970 2023-10-16T15:55:45.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.153+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f3088003d90 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f309000a860 tx=0x7f30880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaae412701b98d22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.153+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090034630 con 0x7f3088003970 2023-10-16T15:55:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.153+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309002f070 con 0x7f3088003970 2023-10-16T15:55:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.153+0000 7f309dd9b640 1 -- 172.21.15.29:0/2048263836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090034a70 con 0x7f3088003970 2023-10-16T15:55:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f30a50d0640 1 -- 172.21.15.29:0/2048263836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 msgr2=0x7f3088003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2048263836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f3088003d90 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f309000a860 tx=0x7f30880057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f309f59e640 1 -- 172.21.15.29:0/2048263836 reap_dead start 2023-10-16T15:55:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f30a50d0640 1 -- 172.21.15.29:0/2048263836 shutdown_connections 2023-10-16T15:55:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f30a50d0640 1 -- 172.21.15.29:0/2048263836 >> 172.21.15.29:0/2048263836 conn(0x7f30a00cb410 msgr2=0x7f30a00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f30a50d0640 1 -- 172.21.15.29:0/2048263836 shutdown_connections 2023-10-16T15:55:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.154+0000 7f30a50d0640 1 -- 172.21.15.29:0/2048263836 wait complete. 2023-10-16T15:55:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.156+0000 7f30a50d0640 1 Processor -- start 2023-10-16T15:55:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.156+0000 7f30a50d0640 1 -- start start 2023-10-16T15:55:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f30a50d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f30a00c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f30a50d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a014d120 0x7f30a0159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f30a50d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30a0159640 0x7f30a014aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f30a50d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0158060 con 0x7f3088003970 2023-10-16T15:55:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f30a50d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0158340 con 0x7f30a0159640 2023-10-16T15:55:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f30a50d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0158620 con 0x7f30a014d120 2023-10-16T15:55:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f309ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f30a00c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f309ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f30a00c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54006/0 (socket says 172.21.15.29:54006) 2023-10-16T15:55:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f309ed9d640 1 -- 172.21.15.29:0/2545114892 learned_addr learned my addr 172.21.15.29:0/2545114892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f309f59e640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30a0159640 0x7f30a014aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f309ed9d640 1 -- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a014d120 msgr2=0x7f30a0159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.157+0000 7f309e59c640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a014d120 0x7f30a0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a014d120 0x7f30a0159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f309ed9d640 1 -- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30a0159640 msgr2=0x7f30a014aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30a0159640 0x7f30a014aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f309ed9d640 1 -- 172.21.15.29:0/2545114892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a0158900 con 0x7f3088003970 2023-10-16T15:55:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f309f59e640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30a0159640 0x7f30a014aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f309ed9d640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f30a00c4ba0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f309000a980 tx=0x7f3090001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090034980 con 0x7f3088003970 2023-10-16T15:55:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3090034ae0 con 0x7f3088003970 2023-10-16T15:55:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090034e50 con 0x7f3088003970 2023-10-16T15:55:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a0158be0 con 0x7f3088003970 2023-10-16T15:55:45.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.158+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a00c50e0 con 0x7f3088003970 2023-10-16T15:55:45.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.160+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a00c5300 con 0x7f3088003970 2023-10-16T15:55:45.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.161+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3090031030 con 0x7f3088003970 2023-10-16T15:55:45.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.161+0000 7f30877fe640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f308c06b0b0 0x7f308c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.162+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f30900ac170 con 0x7f3088003970 2023-10-16T15:55:45.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.162+0000 7f309e59c640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f308c06b0b0 0x7f308c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.163+0000 7f309e59c640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f308c06b0b0 0x7f308c06d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f30a00cadc0 tx=0x7f3094008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:45.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.166+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30a00c5300 con 0x7f3088003970 2023-10-16T15:55:45.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.356+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 --> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30a01584b0 con 0x7f308c06b0b0 2023-10-16T15:55:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.379+0000 7f30877fe640 1 -- 172.21.15.29:0/2545114892 <== mgr.9472 v2:172.21.15.170:6832/3469082932 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287113 (secure 0 0 0) 0x7f30a01584b0 con 0x7f308c06b0b0 2023-10-16T15:55:45.386 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:45.387 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:55:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f308c06b0b0 msgr2=0x7f308c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f308c06b0b0 0x7f308c06d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f30a00cadc0 tx=0x7f3094008040 comp rx=0 tx=0).stop 2023-10-16T15:55:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 msgr2=0x7f30a00c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f30a00c4ba0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f309000a980 tx=0x7f3090001d80 comp rx=0 tx=0).stop 2023-10-16T15:55:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 shutdown_connections 2023-10-16T15:55:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f308c06b0b0 0x7f308c06d570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f30a0159640 0x7f30a014aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30a014d120 0x7f30a0159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 --2- 172.21.15.29:0/2545114892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3088003970 0x7f30a00c4ba0 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 >> 172.21.15.29:0/2545114892 conn(0x7f30a00cb410 msgr2=0x7f30a00b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 shutdown_connections 2023-10-16T15:55:45.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.386+0000 7f30a50d0640 1 -- 172.21.15.29:0/2545114892 wait complete. 2023-10-16T15:55:45.408 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-16T15:55:43.818453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2487,"num_read_kb":5168,"num_write":689,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2825,"ondisk_log_size":2825,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437952,"kb_used_data":27464,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754536768,"statfs":{"total":773094113280,"available":772645650432,"internally_reserved":0,"allocated":28123136,"data_stored":14480148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":420204089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":201,"num_write":63,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.485222"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464368+0000","last_change":"2023-10-16T15:55:40.387281+0000","last_active":"2023-10-16T15:55:41.464368+0000","last_peered":"2023-10-16T15:55:41.464368+0000","last_clean":"2023-10-16T15:55:41.464368+0000","last_became_active":"2023-10-16T15:55:40.386970+0000","last_became_peered":"2023-10-16T15:55:40.386970+0000","last_unstale":"2023-10-16T15:55:41.464368+0000","last_undegraded":"2023-10-16T15:55:41.464368+0000","last_fullsized":"2023-10-16T15:55:41.464368+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:10.351916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463588+0000","last_change":"2023-10-16T15:55:40.389422+0000","last_active":"2023-10-16T15:55:41.463588+0000","last_peered":"2023-10-16T15:55:41.463588+0000","last_clean":"2023-10-16T15:55:41.463588+0000","last_became_active":"2023-10-16T15:55:40.388623+0000","last_became_peered":"2023-10-16T15:55:40.388623+0000","last_unstale":"2023-10-16T15:55:41.463588+0000","last_undegraded":"2023-10-16T15:55:41.463588+0000","last_fullsized":"2023-10-16T15:55:41.463588+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:50.360866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462704+0000","last_change":"2023-10-16T15:55:40.382115+0000","last_active":"2023-10-16T15:55:41.462704+0000","last_peered":"2023-10-16T15:55:41.462704+0000","last_clean":"2023-10-16T15:55:41.462704+0000","last_became_active":"2023-10-16T15:55:40.380673+0000","last_became_peered":"2023-10-16T15:55:40.380673+0000","last_unstale":"2023-10-16T15:55:41.462704+0000","last_undegraded":"2023-10-16T15:55:41.462704+0000","last_fullsized":"2023-10-16T15:55:41.462704+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:22.188972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373330+0000","last_change":"2023-10-16T15:55:40.394800+0000","last_active":"2023-10-16T15:55:41.373330+0000","last_peered":"2023-10-16T15:55:41.373330+0000","last_clean":"2023-10-16T15:55:41.373330+0000","last_became_active":"2023-10-16T15:55:40.394532+0000","last_became_peered":"2023-10-16T15:55:40.394532+0000","last_unstale":"2023-10-16T15:55:41.373330+0000","last_undegraded":"2023-10-16T15:55:41.373330+0000","last_fullsized":"2023-10-16T15:55:41.373330+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:43.929734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462700+0000","last_change":"2023-10-16T15:55:40.387329+0000","last_active":"2023-10-16T15:55:41.462700+0000","last_peered":"2023-10-16T15:55:41.462700+0000","last_clean":"2023-10-16T15:55:41.462700+0000","last_became_active":"2023-10-16T15:55:40.384947+0000","last_became_peered":"2023-10-16T15:55:40.384947+0000","last_unstale":"2023-10-16T15:55:41.462700+0000","last_undegraded":"2023-10-16T15:55:41.462700+0000","last_fullsized":"2023-10-16T15:55:41.462700+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:55.956352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461700+0000","last_change":"2023-10-16T15:55:40.391628+0000","last_active":"2023-10-16T15:55:41.461700+0000","last_peered":"2023-10-16T15:55:41.461700+0000","last_clean":"2023-10-16T15:55:41.461700+0000","last_became_active":"2023-10-16T15:55:40.390173+0000","last_became_peered":"2023-10-16T15:55:40.390173+0000","last_unstale":"2023-10-16T15:55:41.461700+0000","last_undegraded":"2023-10-16T15:55:41.461700+0000","last_fullsized":"2023-10-16T15:55:41.461700+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:32.301642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464028+0000","last_change":"2023-10-16T15:55:40.416742+0000","last_active":"2023-10-16T15:55:41.464028+0000","last_peered":"2023-10-16T15:55:41.464028+0000","last_clean":"2023-10-16T15:55:41.464028+0000","last_became_active":"2023-10-16T15:55:40.416470+0000","last_became_peered":"2023-10-16T15:55:40.416470+0000","last_unstale":"2023-10-16T15:55:41.464028+0000","last_undegraded":"2023-10-16T15:55:41.464028+0000","last_fullsized":"2023-10-16T15:55:41.464028+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:05.778899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461464+0000","last_change":"2023-10-16T15:55:40.415928+0000","last_active":"2023-10-16T15:55:41.461464+0000","last_peered":"2023-10-16T15:55:41.461464+0000","last_clean":"2023-10-16T15:55:41.461464+0000","last_became_active":"2023-10-16T15:55:40.414893+0000","last_became_peered":"2023-10-16T15:55:40.414893+0000","last_unstale":"2023-10-16T15:55:41.461464+0000","last_undegraded":"2023-10-16T15:55:41.461464+0000","last_fullsized":"2023-10-16T15:55:41.461464+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:17.348639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368475+0000","last_change":"2023-10-16T15:55:40.383222+0000","last_active":"2023-10-16T15:55:41.368475+0000","last_peered":"2023-10-16T15:55:41.368475+0000","last_clean":"2023-10-16T15:55:41.368475+0000","last_became_active":"2023-10-16T15:55:40.382883+0000","last_became_peered":"2023-10-16T15:55:40.382883+0000","last_unstale":"2023-10-16T15:55:41.368475+0000","last_undegraded":"2023-10-16T15:55:41.368475+0000","last_fullsized":"2023-10-16T15:55:41.368475+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:05.778899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373951+0000","last_change":"2023-10-16T15:55:40.387547+0000","last_active":"2023-10-16T15:55:41.373951+0000","last_peered":"2023-10-16T15:55:41.373951+0000","last_clean":"2023-10-16T15:55:41.373951+0000","last_became_active":"2023-10-16T15:55:40.387250+0000","last_became_peered":"2023-10-16T15:55:40.387250+0000","last_unstale":"2023-10-16T15:55:41.373951+0000","last_undegraded":"2023-10-16T15:55:41.373951+0000","last_fullsized":"2023-10-16T15:55:41.373951+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:53.612916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.371846+0000","last_change":"2023-10-16T15:55:40.385178+0000","last_active":"2023-10-16T15:55:41.371846+0000","last_peered":"2023-10-16T15:55:41.371846+0000","last_clean":"2023-10-16T15:55:41.371846+0000","last_became_active":"2023-10-16T15:55:40.383382+0000","last_became_peered":"2023-10-16T15:55:40.383382+0000","last_unstale":"2023-10-16T15:55:41.371846+0000","last_undegraded":"2023-10-16T15:55:41.371846+0000","last_fullsized":"2023-10-16T15:55:41.371846+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:44.990876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463813+0000","last_change":"2023-10-16T15:55:40.380601+0000","last_active":"2023-10-16T15:55:41.463813+0000","last_peered":"2023-10-16T15:55:41.463813+0000","last_clean":"2023-10-16T15:55:41.463813+0000","last_became_active":"2023-10-16T15:55:40.380282+0000","last_became_peered":"2023-10-16T15:55:40.380282+0000","last_unstale":"2023-10-16T15:55:41.463813+0000","last_undegraded":"2023-10-16T15:55:41.463813+0000","last_fullsized":"2023-10-16T15:55:41.463813+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.146234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464682+0000","last_change":"2023-10-16T15:55:40.383331+0000","last_active":"2023-10-16T15:55:41.464682+0000","last_peered":"2023-10-16T15:55:41.464682+0000","last_clean":"2023-10-16T15:55:41.464682+0000","last_became_active":"2023-10-16T15:55:40.382103+0000","last_became_peered":"2023-10-16T15:55:40.382103+0000","last_unstale":"2023-10-16T15:55:41.464682+0000","last_undegraded":"2023-10-16T15:55:41.464682+0000","last_fullsized":"2023-10-16T15:55:41.464682+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:57.219840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372604+0000","last_change":"2023-10-16T15:55:40.409428+0000","last_active":"2023-10-16T15:55:41.372604+0000","last_peered":"2023-10-16T15:55:41.372604+0000","last_clean":"2023-10-16T15:55:41.372604+0000","last_became_active":"2023-10-16T15:55:40.408749+0000","last_became_peered":"2023-10-16T15:55:40.408749+0000","last_unstale":"2023-10-16T15:55:41.372604+0000","last_undegraded":"2023-10-16T15:55:41.372604+0000","last_fullsized":"2023-10-16T15:55:41.372604+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:00.848692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464180+0000","last_change":"2023-10-16T15:55:40.414412+0000","last_active":"2023-10-16T15:55:41.464180+0000","last_peered":"2023-10-16T15:55:41.464180+0000","last_clean":"2023-10-16T15:55:41.464180+0000","last_became_active":"2023-10-16T15:55:40.414082+0000","last_became_peered":"2023-10-16T15:55:40.414082+0000","last_unstale":"2023-10-16T15:55:41.464180+0000","last_undegraded":"2023-10-16T15:55:41.464180+0000","last_fullsized":"2023-10-16T15:55:41.464180+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:02.429267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461475+0000","last_change":"2023-10-16T15:55:40.416890+0000","last_active":"2023-10-16T15:55:41.461475+0000","last_peered":"2023-10-16T15:55:41.461475+0000","last_clean":"2023-10-16T15:55:41.461475+0000","last_became_active":"2023-10-16T15:55:40.414569+0000","last_became_peered":"2023-10-16T15:55:40.414569+0000","last_unstale":"2023-10-16T15:55:41.461475+0000","last_undegraded":"2023-10-16T15:55:41.461475+0000","last_fullsized":"2023-10-16T15:55:41.461475+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:22.736777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368591+0000","last_change":"2023-10-16T15:55:40.400249+0000","last_active":"2023-10-16T15:55:41.368591+0000","last_peered":"2023-10-16T15:55:41.368591+0000","last_clean":"2023-10-16T15:55:41.368591+0000","last_became_active":"2023-10-16T15:55:40.399938+0000","last_became_peered":"2023-10-16T15:55:40.399938+0000","last_unstale":"2023-10-16T15:55:41.368591+0000","last_undegraded":"2023-10-16T15:55:41.368591+0000","last_fullsized":"2023-10-16T15:55:41.368591+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:00.643688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463440+0000","last_change":"2023-10-16T15:55:40.401905+0000","last_active":"2023-10-16T15:55:41.463440+0000","last_peered":"2023-10-16T15:55:41.463440+0000","last_clean":"2023-10-16T15:55:41.463440+0000","last_became_active":"2023-10-16T15:55:40.401443+0000","last_became_peered":"2023-10-16T15:55:40.401443+0000","last_unstale":"2023-10-16T15:55:41.463440+0000","last_undegraded":"2023-10-16T15:55:41.463440+0000","last_fullsized":"2023-10-16T15:55:41.463440+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:34.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463362+0000","last_change":"2023-10-16T15:55:40.418498+0000","last_active":"2023-10-16T15:55:41.463362+0000","last_peered":"2023-10-16T15:55:41.463362+0000","last_clean":"2023-10-16T15:55:41.463362+0000","last_became_active":"2023-10-16T15:55:40.418198+0000","last_became_peered":"2023-10-16T15:55:40.418198+0000","last_unstale":"2023-10-16T15:55:41.463362+0000","last_undegraded":"2023-10-16T15:55:41.463362+0000","last_fullsized":"2023-10-16T15:55:41.463362+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:25.827807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464064+0000","last_change":"2023-10-16T15:55:40.409785+0000","last_active":"2023-10-16T15:55:41.464064+0000","last_peered":"2023-10-16T15:55:41.464064+0000","last_clean":"2023-10-16T15:55:41.464064+0000","last_became_active":"2023-10-16T15:55:40.409511+0000","last_became_peered":"2023-10-16T15:55:40.409511+0000","last_unstale":"2023-10-16T15:55:41.464064+0000","last_undegraded":"2023-10-16T15:55:41.464064+0000","last_fullsized":"2023-10-16T15:55:41.464064+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:00.643688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464895+0000","last_change":"2023-10-16T15:55:40.381870+0000","last_active":"2023-10-16T15:55:41.464895+0000","last_peered":"2023-10-16T15:55:41.464895+0000","last_clean":"2023-10-16T15:55:41.464895+0000","last_became_active":"2023-10-16T15:55:40.381465+0000","last_became_peered":"2023-10-16T15:55:40.381465+0000","last_unstale":"2023-10-16T15:55:41.464895+0000","last_undegraded":"2023-10-16T15:55:41.464895+0000","last_fullsized":"2023-10-16T15:55:41.464895+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:59.515158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461838+0000","last_change":"2023-10-16T15:55:40.394214+0000","last_active":"2023-10-16T15:55:41.461838+0000","last_peered":"2023-10-16T15:55:41.461838+0000","last_clean":"2023-10-16T15:55:41.461838+0000","last_became_active":"2023-10-16T15:55:40.393912+0000","last_became_peered":"2023-10-16T15:55:40.393912+0000","last_unstale":"2023-10-16T15:55:41.461838+0000","last_undegraded":"2023-10-16T15:55:41.461838+0000","last_fullsized":"2023-10-16T15:55:41.461838+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:06.788057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460824+0000","last_change":"2023-10-16T15:55:40.417190+0000","last_active":"2023-10-16T15:55:41.460824+0000","last_peered":"2023-10-16T15:55:41.460824+0000","last_clean":"2023-10-16T15:55:41.460824+0000","last_became_active":"2023-10-16T15:55:40.416483+0000","last_became_peered":"2023-10-16T15:55:40.416483+0000","last_unstale":"2023-10-16T15:55:41.460824+0000","last_undegraded":"2023-10-16T15:55:41.460824+0000","last_fullsized":"2023-10-16T15:55:41.460824+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:53:11.743284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461296+0000","last_change":"2023-10-16T15:55:40.415757+0000","last_active":"2023-10-16T15:55:41.461296+0000","last_peered":"2023-10-16T15:55:41.461296+0000","last_clean":"2023-10-16T15:55:41.461296+0000","last_became_active":"2023-10-16T15:55:40.415415+0000","last_became_peered":"2023-10-16T15:55:40.415415+0000","last_unstale":"2023-10-16T15:55:41.461296+0000","last_undegraded":"2023-10-16T15:55:41.461296+0000","last_fullsized":"2023-10-16T15:55:41.461296+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:25.538277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462483+0000","last_change":"2023-10-16T15:55:40.382989+0000","last_active":"2023-10-16T15:55:41.462483+0000","last_peered":"2023-10-16T15:55:41.462483+0000","last_clean":"2023-10-16T15:55:41.462483+0000","last_became_active":"2023-10-16T15:55:40.382698+0000","last_became_peered":"2023-10-16T15:55:40.382698+0000","last_unstale":"2023-10-16T15:55:41.462483+0000","last_undegraded":"2023-10-16T15:55:41.462483+0000","last_fullsized":"2023-10-16T15:55:41.462483+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:49.737461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464688+0000","last_change":"2023-10-16T15:55:40.386135+0000","last_active":"2023-10-16T15:55:41.464688+0000","last_peered":"2023-10-16T15:55:41.464688+0000","last_clean":"2023-10-16T15:55:41.464688+0000","last_became_active":"2023-10-16T15:55:40.385751+0000","last_became_peered":"2023-10-16T15:55:40.385751+0000","last_unstale":"2023-10-16T15:55:41.464688+0000","last_undegraded":"2023-10-16T15:55:41.464688+0000","last_fullsized":"2023-10-16T15:55:41.464688+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:09.667516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463716+0000","last_change":"2023-10-16T15:55:40.414957+0000","last_active":"2023-10-16T15:55:41.463716+0000","last_peered":"2023-10-16T15:55:41.463716+0000","last_clean":"2023-10-16T15:55:41.463716+0000","last_became_active":"2023-10-16T15:55:40.414673+0000","last_became_peered":"2023-10-16T15:55:40.414673+0000","last_unstale":"2023-10-16T15:55:41.463716+0000","last_undegraded":"2023-10-16T15:55:41.463716+0000","last_fullsized":"2023-10-16T15:55:41.463716+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:24.434830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373088+0000","last_change":"2023-10-16T15:55:40.389402+0000","last_active":"2023-10-16T15:55:41.373088+0000","last_peered":"2023-10-16T15:55:41.373088+0000","last_clean":"2023-10-16T15:55:41.373088+0000","last_became_active":"2023-10-16T15:55:40.389059+0000","last_became_peered":"2023-10-16T15:55:40.389059+0000","last_unstale":"2023-10-16T15:55:41.373088+0000","last_undegraded":"2023-10-16T15:55:41.373088+0000","last_fullsized":"2023-10-16T15:55:41.373088+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:17.938812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462559+0000","last_change":"2023-10-16T15:55:40.414324+0000","last_active":"2023-10-16T15:55:41.462559+0000","last_peered":"2023-10-16T15:55:41.462559+0000","last_clean":"2023-10-16T15:55:41.462559+0000","last_became_active":"2023-10-16T15:55:40.413979+0000","last_became_peered":"2023-10-16T15:55:40.413979+0000","last_unstale":"2023-10-16T15:55:41.462559+0000","last_undegraded":"2023-10-16T15:55:41.462559+0000","last_fullsized":"2023-10-16T15:55:41.462559+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:51.502048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464425+0000","last_change":"2023-10-16T15:55:40.413964+0000","last_active":"2023-10-16T15:55:41.464425+0000","last_peered":"2023-10-16T15:55:41.464425+0000","last_clean":"2023-10-16T15:55:41.464425+0000","last_became_active":"2023-10-16T15:55:40.409602+0000","last_became_peered":"2023-10-16T15:55:40.409602+0000","last_unstale":"2023-10-16T15:55:41.464425+0000","last_undegraded":"2023-10-16T15:55:41.464425+0000","last_fullsized":"2023-10-16T15:55:41.464425+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:58.706145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367328+0000","last_change":"2023-10-16T15:55:40.398402+0000","last_active":"2023-10-16T15:55:41.367328+0000","last_peered":"2023-10-16T15:55:41.367328+0000","last_clean":"2023-10-16T15:55:41.367328+0000","last_became_active":"2023-10-16T15:55:40.395289+0000","last_became_peered":"2023-10-16T15:55:40.395289+0000","last_unstale":"2023-10-16T15:55:41.367328+0000","last_undegraded":"2023-10-16T15:55:41.367328+0000","last_fullsized":"2023-10-16T15:55:41.367328+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:39.230917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464618+0000","last_change":"2023-10-16T15:55:40.397194+0000","last_active":"2023-10-16T15:55:41.464618+0000","last_peered":"2023-10-16T15:55:41.464618+0000","last_clean":"2023-10-16T15:55:41.464618+0000","last_became_active":"2023-10-16T15:55:40.396924+0000","last_became_peered":"2023-10-16T15:55:40.396924+0000","last_unstale":"2023-10-16T15:55:41.464618+0000","last_undegraded":"2023-10-16T15:55:41.464618+0000","last_fullsized":"2023-10-16T15:55:41.464618+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:37.392830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"265'4","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464556+0000","last_change":"2023-10-16T15:55:39.377649+0000","last_active":"2023-10-16T15:55:41.464556+0000","last_peered":"2023-10-16T15:55:41.464556+0000","last_clean":"2023-10-16T15:55:41.464556+0000","last_became_active":"2023-10-16T15:55:39.377037+0000","last_became_peered":"2023-10-16T15:55:39.377037+0000","last_unstale":"2023-10-16T15:55:41.464556+0000","last_undegraded":"2023-10-16T15:55:41.464556+0000","last_fullsized":"2023-10-16T15:55:41.464556+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:23.520225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373584+0000","last_change":"2023-10-16T15:55:40.384026+0000","last_active":"2023-10-16T15:55:41.373584+0000","last_peered":"2023-10-16T15:55:41.373584+0000","last_clean":"2023-10-16T15:55:41.373584+0000","last_became_active":"2023-10-16T15:55:40.383658+0000","last_became_peered":"2023-10-16T15:55:40.383658+0000","last_unstale":"2023-10-16T15:55:41.373584+0000","last_undegraded":"2023-10-16T15:55:41.373584+0000","last_fullsized":"2023-10-16T15:55:41.373584+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:42.460984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368558+0000","last_change":"2023-10-16T15:55:39.365271+0000","last_active":"2023-10-16T15:55:41.368558+0000","last_peered":"2023-10-16T15:55:41.368558+0000","last_clean":"2023-10-16T15:55:41.368558+0000","last_became_active":"2023-10-16T15:55:39.363945+0000","last_became_peered":"2023-10-16T15:55:39.363945+0000","last_unstale":"2023-10-16T15:55:41.368558+0000","last_undegraded":"2023-10-16T15:55:41.368558+0000","last_fullsized":"2023-10-16T15:55:41.368558+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:56.878210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463369+0000","last_change":"2023-10-16T15:55:40.386790+0000","last_active":"2023-10-16T15:55:41.463369+0000","last_peered":"2023-10-16T15:55:41.463369+0000","last_clean":"2023-10-16T15:55:41.463369+0000","last_became_active":"2023-10-16T15:55:40.384638+0000","last_became_peered":"2023-10-16T15:55:40.384638+0000","last_unstale":"2023-10-16T15:55:41.463369+0000","last_undegraded":"2023-10-16T15:55:41.463369+0000","last_fullsized":"2023-10-16T15:55:41.463369+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:00.468527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368033+0000","last_change":"2023-10-16T15:55:39.376491+0000","last_active":"2023-10-16T15:55:41.368033+0000","last_peered":"2023-10-16T15:55:41.368033+0000","last_clean":"2023-10-16T15:55:41.368033+0000","last_became_active":"2023-10-16T15:55:39.376187+0000","last_became_peered":"2023-10-16T15:55:39.376187+0000","last_unstale":"2023-10-16T15:55:41.368033+0000","last_undegraded":"2023-10-16T15:55:41.368033+0000","last_fullsized":"2023-10-16T15:55:41.368033+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.212193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462904+0000","last_change":"2023-10-16T15:55:40.385279+0000","last_active":"2023-10-16T15:55:41.462904+0000","last_peered":"2023-10-16T15:55:41.462904+0000","last_clean":"2023-10-16T15:55:41.462904+0000","last_became_active":"2023-10-16T15:55:40.384995+0000","last_became_peered":"2023-10-16T15:55:40.384995+0000","last_unstale":"2023-10-16T15:55:41.462904+0000","last_undegraded":"2023-10-16T15:55:41.462904+0000","last_fullsized":"2023-10-16T15:55:41.462904+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:10.414031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464373+0000","last_change":"2023-10-16T15:55:39.375356+0000","last_active":"2023-10-16T15:55:41.464373+0000","last_peered":"2023-10-16T15:55:41.464373+0000","last_clean":"2023-10-16T15:55:41.464373+0000","last_became_active":"2023-10-16T15:55:39.374963+0000","last_became_peered":"2023-10-16T15:55:39.374963+0000","last_unstale":"2023-10-16T15:55:41.464373+0000","last_undegraded":"2023-10-16T15:55:41.464373+0000","last_fullsized":"2023-10-16T15:55:41.464373+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:15.714641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460706+0000","last_change":"2023-10-16T15:55:40.385508+0000","last_active":"2023-10-16T15:55:41.460706+0000","last_peered":"2023-10-16T15:55:41.460706+0000","last_clean":"2023-10-16T15:55:41.460706+0000","last_became_active":"2023-10-16T15:55:40.385185+0000","last_became_peered":"2023-10-16T15:55:40.385185+0000","last_unstale":"2023-10-16T15:55:41.460706+0000","last_undegraded":"2023-10-16T15:55:41.460706+0000","last_fullsized":"2023-10-16T15:55:41.460706+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:35.454918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462030+0000","last_change":"2023-10-16T15:55:39.378795+0000","last_active":"2023-10-16T15:55:41.462030+0000","last_peered":"2023-10-16T15:55:41.462030+0000","last_clean":"2023-10-16T15:55:41.462030+0000","last_became_active":"2023-10-16T15:55:39.378515+0000","last_became_peered":"2023-10-16T15:55:39.378515+0000","last_unstale":"2023-10-16T15:55:41.462030+0000","last_undegraded":"2023-10-16T15:55:41.462030+0000","last_fullsized":"2023-10-16T15:55:41.462030+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:30.014727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462918+0000","last_change":"2023-10-16T15:55:40.410500+0000","last_active":"2023-10-16T15:55:41.462918+0000","last_peered":"2023-10-16T15:55:41.462918+0000","last_clean":"2023-10-16T15:55:41.462918+0000","last_became_active":"2023-10-16T15:55:40.409313+0000","last_became_peered":"2023-10-16T15:55:40.409313+0000","last_unstale":"2023-10-16T15:55:41.462918+0000","last_undegraded":"2023-10-16T15:55:41.462918+0000","last_fullsized":"2023-10-16T15:55:41.462918+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:39.230917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462890+0000","last_change":"2023-10-16T15:55:39.375431+0000","last_active":"2023-10-16T15:55:41.462890+0000","last_peered":"2023-10-16T15:55:41.462890+0000","last_clean":"2023-10-16T15:55:41.462890+0000","last_became_active":"2023-10-16T15:55:39.374524+0000","last_became_peered":"2023-10-16T15:55:39.374524+0000","last_unstale":"2023-10-16T15:55:41.462890+0000","last_undegraded":"2023-10-16T15:55:41.462890+0000","last_fullsized":"2023-10-16T15:55:41.462890+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:38.611799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367406+0000","last_change":"2023-10-16T15:55:40.391186+0000","last_active":"2023-10-16T15:55:41.367406+0000","last_peered":"2023-10-16T15:55:41.367406+0000","last_clean":"2023-10-16T15:55:41.367406+0000","last_became_active":"2023-10-16T15:55:40.390887+0000","last_became_peered":"2023-10-16T15:55:40.390887+0000","last_unstale":"2023-10-16T15:55:41.367406+0000","last_undegraded":"2023-10-16T15:55:41.367406+0000","last_fullsized":"2023-10-16T15:55:41.367406+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:11.462063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462359+0000","last_change":"2023-10-16T15:55:39.373139+0000","last_active":"2023-10-16T15:55:41.462359+0000","last_peered":"2023-10-16T15:55:41.462359+0000","last_clean":"2023-10-16T15:55:41.462359+0000","last_became_active":"2023-10-16T15:55:39.372758+0000","last_became_peered":"2023-10-16T15:55:39.372758+0000","last_unstale":"2023-10-16T15:55:41.462359+0000","last_undegraded":"2023-10-16T15:55:41.462359+0000","last_fullsized":"2023-10-16T15:55:41.462359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:46.380751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462569+0000","last_change":"2023-10-16T15:55:40.397291+0000","last_active":"2023-10-16T15:55:41.462569+0000","last_peered":"2023-10-16T15:55:41.462569+0000","last_clean":"2023-10-16T15:55:41.462569+0000","last_became_active":"2023-10-16T15:55:40.397022+0000","last_became_peered":"2023-10-16T15:55:40.397022+0000","last_unstale":"2023-10-16T15:55:41.462569+0000","last_undegraded":"2023-10-16T15:55:41.462569+0000","last_fullsized":"2023-10-16T15:55:41.462569+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:50.749801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463774+0000","last_change":"2023-10-16T15:55:39.366393+0000","last_active":"2023-10-16T15:55:41.463774+0000","last_peered":"2023-10-16T15:55:41.463774+0000","last_clean":"2023-10-16T15:55:41.463774+0000","last_became_active":"2023-10-16T15:55:39.365879+0000","last_became_peered":"2023-10-16T15:55:39.365879+0000","last_unstale":"2023-10-16T15:55:41.463774+0000","last_undegraded":"2023-10-16T15:55:41.463774+0000","last_fullsized":"2023-10-16T15:55:41.463774+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:54.810472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463783+0000","last_change":"2023-10-16T15:55:40.401719+0000","last_active":"2023-10-16T15:55:41.463783+0000","last_peered":"2023-10-16T15:55:41.463783+0000","last_clean":"2023-10-16T15:55:41.463783+0000","last_became_active":"2023-10-16T15:55:40.401046+0000","last_became_peered":"2023-10-16T15:55:40.401046+0000","last_unstale":"2023-10-16T15:55:41.463783+0000","last_undegraded":"2023-10-16T15:55:41.463783+0000","last_fullsized":"2023-10-16T15:55:41.463783+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:44.450650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373476+0000","last_change":"2023-10-16T15:55:39.369873+0000","last_active":"2023-10-16T15:55:41.373476+0000","last_peered":"2023-10-16T15:55:41.373476+0000","last_clean":"2023-10-16T15:55:41.373476+0000","last_became_active":"2023-10-16T15:55:39.369512+0000","last_became_peered":"2023-10-16T15:55:39.369512+0000","last_unstale":"2023-10-16T15:55:41.373476+0000","last_undegraded":"2023-10-16T15:55:41.373476+0000","last_fullsized":"2023-10-16T15:55:41.373476+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:58.501117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373282+0000","last_change":"2023-10-16T15:55:40.411562+0000","last_active":"2023-10-16T15:55:41.373282+0000","last_peered":"2023-10-16T15:55:41.373282+0000","last_clean":"2023-10-16T15:55:41.373282+0000","last_became_active":"2023-10-16T15:55:40.411271+0000","last_became_peered":"2023-10-16T15:55:40.411271+0000","last_unstale":"2023-10-16T15:55:41.373282+0000","last_undegraded":"2023-10-16T15:55:41.373282+0000","last_fullsized":"2023-10-16T15:55:41.373282+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:37.304909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372885+0000","last_change":"2023-10-16T15:55:39.369380+0000","last_active":"2023-10-16T15:55:41.372885+0000","last_peered":"2023-10-16T15:55:41.372885+0000","last_clean":"2023-10-16T15:55:41.372885+0000","last_became_active":"2023-10-16T15:55:39.369039+0000","last_became_peered":"2023-10-16T15:55:39.369039+0000","last_unstale":"2023-10-16T15:55:41.372885+0000","last_undegraded":"2023-10-16T15:55:41.372885+0000","last_fullsized":"2023-10-16T15:55:41.372885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:54.476845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372857+0000","last_change":"2023-10-16T15:55:40.386505+0000","last_active":"2023-10-16T15:55:41.372857+0000","last_peered":"2023-10-16T15:55:41.372857+0000","last_clean":"2023-10-16T15:55:41.372857+0000","last_became_active":"2023-10-16T15:55:40.386164+0000","last_became_peered":"2023-10-16T15:55:40.386164+0000","last_unstale":"2023-10-16T15:55:41.372857+0000","last_undegraded":"2023-10-16T15:55:41.372857+0000","last_fullsized":"2023-10-16T15:55:41.372857+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:03.594800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"265'3","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.368269+0000","last_change":"2023-10-16T15:55:39.376511+0000","last_active":"2023-10-16T15:55:41.368269+0000","last_peered":"2023-10-16T15:55:41.368269+0000","last_clean":"2023-10-16T15:55:41.368269+0000","last_became_active":"2023-10-16T15:55:39.376225+0000","last_became_peered":"2023-10-16T15:55:39.376225+0000","last_unstale":"2023-10-16T15:55:41.368269+0000","last_undegraded":"2023-10-16T15:55:41.368269+0000","last_fullsized":"2023-10-16T15:55:41.368269+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:52.598875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373617+0000","last_change":"2023-10-16T15:55:40.385691+0000","last_active":"2023-10-16T15:55:41.373617+0000","last_peered":"2023-10-16T15:55:41.373617+0000","last_clean":"2023-10-16T15:55:41.373617+0000","last_became_active":"2023-10-16T15:55:40.382892+0000","last_became_peered":"2023-10-16T15:55:40.382892+0000","last_unstale":"2023-10-16T15:55:41.373617+0000","last_undegraded":"2023-10-16T15:55:41.373617+0000","last_fullsized":"2023-10-16T15:55:41.373617+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:02.226234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"265'6","reported_seq":18,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463665+0000","last_change":"2023-10-16T15:55:39.375502+0000","last_active":"2023-10-16T15:55:41.463665+0000","last_peered":"2023-10-16T15:55:41.463665+0000","last_clean":"2023-10-16T15:55:41.463665+0000","last_became_active":"2023-10-16T15:55:39.375141+0000","last_became_peered":"2023-10-16T15:55:39.375141+0000","last_unstale":"2023-10-16T15:55:41.463665+0000","last_undegraded":"2023-10-16T15:55:41.463665+0000","last_fullsized":"2023-10-16T15:55:41.463665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:50.230725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462124+0000","last_change":"2023-10-16T15:55:40.410525+0000","last_active":"2023-10-16T15:55:41.462124+0000","last_peered":"2023-10-16T15:55:41.462124+0000","last_clean":"2023-10-16T15:55:41.462124+0000","last_became_active":"2023-10-16T15:55:40.409376+0000","last_became_peered":"2023-10-16T15:55:40.409376+0000","last_unstale":"2023-10-16T15:55:41.462124+0000","last_undegraded":"2023-10-16T15:55:41.462124+0000","last_fullsized":"2023-10-16T15:55:41.462124+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:02.266659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461359+0000","last_change":"2023-10-16T15:55:39.372116+0000","last_active":"2023-10-16T15:55:41.461359+0000","last_peered":"2023-10-16T15:55:41.461359+0000","last_clean":"2023-10-16T15:55:41.461359+0000","last_became_active":"2023-10-16T15:55:39.369639+0000","last_became_peered":"2023-10-16T15:55:39.369639+0000","last_unstale":"2023-10-16T15:55:41.461359+0000","last_undegraded":"2023-10-16T15:55:41.461359+0000","last_fullsized":"2023-10-16T15:55:41.461359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:00:44.611503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462690+0000","last_change":"2023-10-16T15:55:40.403239+0000","last_active":"2023-10-16T15:55:41.462690+0000","last_peered":"2023-10-16T15:55:41.462690+0000","last_clean":"2023-10-16T15:55:41.462690+0000","last_became_active":"2023-10-16T15:55:40.402967+0000","last_became_peered":"2023-10-16T15:55:40.402967+0000","last_unstale":"2023-10-16T15:55:41.462690+0000","last_undegraded":"2023-10-16T15:55:41.462690+0000","last_fullsized":"2023-10-16T15:55:41.462690+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:23.617775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460482+0000","last_change":"2023-10-16T15:55:39.370774+0000","last_active":"2023-10-16T15:55:41.460482+0000","last_peered":"2023-10-16T15:55:41.460482+0000","last_clean":"2023-10-16T15:55:41.460482+0000","last_became_active":"2023-10-16T15:55:39.370493+0000","last_became_peered":"2023-10-16T15:55:39.370493+0000","last_unstale":"2023-10-16T15:55:41.460482+0000","last_undegraded":"2023-10-16T15:55:41.460482+0000","last_fullsized":"2023-10-16T15:55:41.460482+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:51:06.129842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.370588+0000","last_change":"2023-10-16T15:55:40.402774+0000","last_active":"2023-10-16T15:55:41.370588+0000","last_peered":"2023-10-16T15:55:41.370588+0000","last_clean":"2023-10-16T15:55:41.370588+0000","last_became_active":"2023-10-16T15:55:40.402477+0000","last_became_peered":"2023-10-16T15:55:40.402477+0000","last_unstale":"2023-10-16T15:55:41.370588+0000","last_undegraded":"2023-10-16T15:55:41.370588+0000","last_fullsized":"2023-10-16T15:55:41.370588+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:40.495907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.371983+0000","last_change":"2023-10-16T15:55:39.369210+0000","last_active":"2023-10-16T15:55:41.371983+0000","last_peered":"2023-10-16T15:55:41.371983+0000","last_clean":"2023-10-16T15:55:41.371983+0000","last_became_active":"2023-10-16T15:55:39.368926+0000","last_became_peered":"2023-10-16T15:55:39.368926+0000","last_unstale":"2023-10-16T15:55:41.371983+0000","last_undegraded":"2023-10-16T15:55:41.371983+0000","last_fullsized":"2023-10-16T15:55:41.371983+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:56.205799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464540+0000","last_change":"2023-10-16T15:55:40.410297+0000","last_active":"2023-10-16T15:55:41.464540+0000","last_peered":"2023-10-16T15:55:41.464540+0000","last_clean":"2023-10-16T15:55:41.464540+0000","last_became_active":"2023-10-16T15:55:40.409046+0000","last_became_peered":"2023-10-16T15:55:40.409046+0000","last_unstale":"2023-10-16T15:55:41.464540+0000","last_undegraded":"2023-10-16T15:55:41.464540+0000","last_fullsized":"2023-10-16T15:55:41.464540+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:20.318733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"265'8","reported_seq":20,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.460743+0000","last_change":"2023-10-16T15:55:39.374027+0000","last_active":"2023-10-16T15:55:41.460743+0000","last_peered":"2023-10-16T15:55:41.460743+0000","last_clean":"2023-10-16T15:55:41.460743+0000","last_became_active":"2023-10-16T15:55:39.373598+0000","last_became_peered":"2023-10-16T15:55:39.373598+0000","last_unstale":"2023-10-16T15:55:41.460743+0000","last_undegraded":"2023-10-16T15:55:41.460743+0000","last_fullsized":"2023-10-16T15:55:41.460743+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:36.466341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461297+0000","last_change":"2023-10-16T15:55:40.380158+0000","last_active":"2023-10-16T15:55:41.461297+0000","last_peered":"2023-10-16T15:55:41.461297+0000","last_clean":"2023-10-16T15:55:41.461297+0000","last_became_active":"2023-10-16T15:55:40.379861+0000","last_became_peered":"2023-10-16T15:55:40.379861+0000","last_unstale":"2023-10-16T15:55:41.461297+0000","last_undegraded":"2023-10-16T15:55:41.461297+0000","last_fullsized":"2023-10-16T15:55:41.461297+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:07:07.197038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373679+0000","last_change":"2023-10-16T15:55:39.368770+0000","last_active":"2023-10-16T15:55:41.373679+0000","last_peered":"2023-10-16T15:55:41.373679+0000","last_clean":"2023-10-16T15:55:41.373679+0000","last_became_active":"2023-10-16T15:55:39.368471+0000","last_became_peered":"2023-10-16T15:55:39.368471+0000","last_unstale":"2023-10-16T15:55:41.373679+0000","last_undegraded":"2023-10-16T15:55:41.373679+0000","last_fullsized":"2023-10-16T15:55:41.373679+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:56.501266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463160+0000","last_change":"2023-10-16T15:55:40.393143+0000","last_active":"2023-10-16T15:55:41.463160+0000","last_peered":"2023-10-16T15:55:41.463160+0000","last_clean":"2023-10-16T15:55:41.463160+0000","last_became_active":"2023-10-16T15:55:40.392886+0000","last_became_peered":"2023-10-16T15:55:40.392886+0000","last_unstale":"2023-10-16T15:55:41.463160+0000","last_undegraded":"2023-10-16T15:55:41.463160+0000","last_fullsized":"2023-10-16T15:55:41.463160+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:36.781554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464016+0000","last_change":"2023-10-16T15:55:39.372893+0000","last_active":"2023-10-16T15:55:41.464016+0000","last_peered":"2023-10-16T15:55:41.464016+0000","last_clean":"2023-10-16T15:55:41.464016+0000","last_became_active":"2023-10-16T15:55:39.372623+0000","last_became_peered":"2023-10-16T15:55:39.372623+0000","last_unstale":"2023-10-16T15:55:41.464016+0000","last_undegraded":"2023-10-16T15:55:41.464016+0000","last_fullsized":"2023-10-16T15:55:41.464016+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:13.813201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462605+0000","last_change":"2023-10-16T15:55:40.399956+0000","last_active":"2023-10-16T15:55:41.462605+0000","last_peered":"2023-10-16T15:55:41.462605+0000","last_clean":"2023-10-16T15:55:41.462605+0000","last_became_active":"2023-10-16T15:55:40.399669+0000","last_became_peered":"2023-10-16T15:55:40.399669+0000","last_unstale":"2023-10-16T15:55:41.462605+0000","last_undegraded":"2023-10-16T15:55:41.462605+0000","last_fullsized":"2023-10-16T15:55:41.462605+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:44.360499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"265'1","reported_seq":13,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373695+0000","last_change":"2023-10-16T15:55:39.371802+0000","last_active":"2023-10-16T15:55:41.373695+0000","last_peered":"2023-10-16T15:55:41.373695+0000","last_clean":"2023-10-16T15:55:41.373695+0000","last_became_active":"2023-10-16T15:55:39.371519+0000","last_became_peered":"2023-10-16T15:55:39.371519+0000","last_unstale":"2023-10-16T15:55:41.373695+0000","last_undegraded":"2023-10-16T15:55:41.373695+0000","last_fullsized":"2023-10-16T15:55:41.373695+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:15.714641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463325+0000","last_change":"2023-10-16T15:55:40.385206+0000","last_active":"2023-10-16T15:55:41.463325+0000","last_peered":"2023-10-16T15:55:41.463325+0000","last_clean":"2023-10-16T15:55:41.463325+0000","last_became_active":"2023-10-16T15:55:40.384921+0000","last_became_peered":"2023-10-16T15:55:40.384921+0000","last_unstale":"2023-10-16T15:55:41.463325+0000","last_undegraded":"2023-10-16T15:55:41.463325+0000","last_fullsized":"2023-10-16T15:55:41.463325+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:47.794581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373357+0000","last_change":"2023-10-16T15:55:39.361037+0000","last_active":"2023-10-16T15:55:41.373357+0000","last_peered":"2023-10-16T15:55:41.373357+0000","last_clean":"2023-10-16T15:55:41.373357+0000","last_became_active":"2023-10-16T15:55:39.359478+0000","last_became_peered":"2023-10-16T15:55:39.359478+0000","last_unstale":"2023-10-16T15:55:41.373357+0000","last_undegraded":"2023-10-16T15:55:41.373357+0000","last_fullsized":"2023-10-16T15:55:41.373357+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:17:05.050926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367589+0000","last_change":"2023-10-16T15:55:40.386722+0000","last_active":"2023-10-16T15:55:41.367589+0000","last_peered":"2023-10-16T15:55:41.367589+0000","last_clean":"2023-10-16T15:55:41.367589+0000","last_became_active":"2023-10-16T15:55:40.386406+0000","last_became_peered":"2023-10-16T15:55:40.386406+0000","last_unstale":"2023-10-16T15:55:41.367589+0000","last_undegraded":"2023-10-16T15:55:41.367589+0000","last_fullsized":"2023-10-16T15:55:41.367589+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:24.434830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.465085+0000","last_change":"2023-10-16T15:55:39.368103+0000","last_active":"2023-10-16T15:55:41.465085+0000","last_peered":"2023-10-16T15:55:41.465085+0000","last_clean":"2023-10-16T15:55:41.465085+0000","last_became_active":"2023-10-16T15:55:39.367824+0000","last_became_peered":"2023-10-16T15:55:39.367824+0000","last_unstale":"2023-10-16T15:55:41.465085+0000","last_undegraded":"2023-10-16T15:55:41.465085+0000","last_fullsized":"2023-10-16T15:55:41.465085+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:12.469700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463084+0000","last_change":"2023-10-16T15:55:40.403096+0000","last_active":"2023-10-16T15:55:41.463084+0000","last_peered":"2023-10-16T15:55:41.463084+0000","last_clean":"2023-10-16T15:55:41.463084+0000","last_became_active":"2023-10-16T15:55:40.402237+0000","last_became_peered":"2023-10-16T15:55:40.402237+0000","last_unstale":"2023-10-16T15:55:41.463084+0000","last_undegraded":"2023-10-16T15:55:41.463084+0000","last_fullsized":"2023-10-16T15:55:41.463084+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:55.638360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.461927+0000","last_change":"2023-10-16T15:55:39.368966+0000","last_active":"2023-10-16T15:55:41.461927+0000","last_peered":"2023-10-16T15:55:41.461927+0000","last_clean":"2023-10-16T15:55:41.461927+0000","last_became_active":"2023-10-16T15:55:39.368663+0000","last_became_peered":"2023-10-16T15:55:39.368663+0000","last_unstale":"2023-10-16T15:55:41.461927+0000","last_undegraded":"2023-10-16T15:55:41.461927+0000","last_fullsized":"2023-10-16T15:55:41.461927+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:59.226241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462260+0000","last_change":"2023-10-16T15:55:40.402398+0000","last_active":"2023-10-16T15:55:41.462260+0000","last_peered":"2023-10-16T15:55:41.462260+0000","last_clean":"2023-10-16T15:55:41.462260+0000","last_became_active":"2023-10-16T15:55:40.401938+0000","last_became_peered":"2023-10-16T15:55:40.401938+0000","last_unstale":"2023-10-16T15:55:41.462260+0000","last_undegraded":"2023-10-16T15:55:41.462260+0000","last_fullsized":"2023-10-16T15:55:41.462260+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:34.114098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372032+0000","last_change":"2023-10-16T15:55:39.365241+0000","last_active":"2023-10-16T15:55:41.372032+0000","last_peered":"2023-10-16T15:55:41.372032+0000","last_clean":"2023-10-16T15:55:41.372032+0000","last_became_active":"2023-10-16T15:55:39.364546+0000","last_became_peered":"2023-10-16T15:55:39.364546+0000","last_unstale":"2023-10-16T15:55:41.372032+0000","last_undegraded":"2023-10-16T15:55:41.372032+0000","last_fullsized":"2023-10-16T15:55:41.372032+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:08.653475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464242+0000","last_change":"2023-10-16T15:55:40.384442+0000","last_active":"2023-10-16T15:55:41.464242+0000","last_peered":"2023-10-16T15:55:41.464242+0000","last_clean":"2023-10-16T15:55:41.464242+0000","last_became_active":"2023-10-16T15:55:40.384048+0000","last_became_peered":"2023-10-16T15:55:40.384048+0000","last_unstale":"2023-10-16T15:55:41.464242+0000","last_undegraded":"2023-10-16T15:55:41.464242+0000","last_fullsized":"2023-10-16T15:55:41.464242+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:17.324790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462067+0000","last_change":"2023-10-16T15:55:39.376672+0000","last_active":"2023-10-16T15:55:41.462067+0000","last_peered":"2023-10-16T15:55:41.462067+0000","last_clean":"2023-10-16T15:55:41.462067+0000","last_became_active":"2023-10-16T15:55:39.375874+0000","last_became_peered":"2023-10-16T15:55:39.375874+0000","last_unstale":"2023-10-16T15:55:41.462067+0000","last_undegraded":"2023-10-16T15:55:41.462067+0000","last_fullsized":"2023-10-16T15:55:41.462067+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:12.642092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463324+0000","last_change":"2023-10-16T15:55:40.414024+0000","last_active":"2023-10-16T15:55:41.463324+0000","last_peered":"2023-10-16T15:55:41.463324+0000","last_clean":"2023-10-16T15:55:41.463324+0000","last_became_active":"2023-10-16T15:55:40.409652+0000","last_became_peered":"2023-10-16T15:55:40.409652+0000","last_unstale":"2023-10-16T15:55:41.463324+0000","last_undegraded":"2023-10-16T15:55:41.463324+0000","last_fullsized":"2023-10-16T15:55:41.463324+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:11.462063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"265'2","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.373492+0000","last_change":"2023-10-16T15:55:39.368876+0000","last_active":"2023-10-16T15:55:41.373492+0000","last_peered":"2023-10-16T15:55:41.373492+0000","last_clean":"2023-10-16T15:55:41.373492+0000","last_became_active":"2023-10-16T15:55:39.367570+0000","last_became_peered":"2023-10-16T15:55:39.367570+0000","last_unstale":"2023-10-16T15:55:41.373492+0000","last_undegraded":"2023-10-16T15:55:41.373492+0000","last_fullsized":"2023-10-16T15:55:41.373492+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:17.528310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463540+0000","last_change":"2023-10-16T15:55:40.411831+0000","last_active":"2023-10-16T15:55:41.463540+0000","last_peered":"2023-10-16T15:55:41.463540+0000","last_clean":"2023-10-16T15:55:41.463540+0000","last_became_active":"2023-10-16T15:55:40.410913+0000","last_became_peered":"2023-10-16T15:55:40.410913+0000","last_unstale":"2023-10-16T15:55:41.463540+0000","last_undegraded":"2023-10-16T15:55:41.463540+0000","last_fullsized":"2023-10-16T15:55:41.463540+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:47.394792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463746+0000","last_change":"2023-10-16T15:55:39.368288+0000","last_active":"2023-10-16T15:55:41.463746+0000","last_peered":"2023-10-16T15:55:41.463746+0000","last_clean":"2023-10-16T15:55:41.463746+0000","last_became_active":"2023-10-16T15:55:39.368003+0000","last_became_peered":"2023-10-16T15:55:39.368003+0000","last_unstale":"2023-10-16T15:55:41.463746+0000","last_undegraded":"2023-10-16T15:55:41.463746+0000","last_fullsized":"2023-10-16T15:55:41.463746+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:40:01.562908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.464266+0000","last_change":"2023-10-16T15:55:40.402889+0000","last_active":"2023-10-16T15:55:41.464266+0000","last_peered":"2023-10-16T15:55:41.464266+0000","last_clean":"2023-10-16T15:55:41.464266+0000","last_became_active":"2023-10-16T15:55:40.402596+0000","last_became_peered":"2023-10-16T15:55:40.402596+0000","last_unstale":"2023-10-16T15:55:41.464266+0000","last_undegraded":"2023-10-16T15:55:41.464266+0000","last_fullsized":"2023-10-16T15:55:41.464266+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:36.800461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372098+0000","last_change":"2023-10-16T15:55:39.370080+0000","last_active":"2023-10-16T15:55:41.372098+0000","last_peered":"2023-10-16T15:55:41.372098+0000","last_clean":"2023-10-16T15:55:41.372098+0000","last_became_active":"2023-10-16T15:55:39.367029+0000","last_became_peered":"2023-10-16T15:55:39.367029+0000","last_unstale":"2023-10-16T15:55:41.372098+0000","last_undegraded":"2023-10-16T15:55:41.372098+0000","last_fullsized":"2023-10-16T15:55:41.372098+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:54.377409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463696+0000","last_change":"2023-10-16T15:55:40.383931+0000","last_active":"2023-10-16T15:55:41.463696+0000","last_peered":"2023-10-16T15:55:41.463696+0000","last_clean":"2023-10-16T15:55:41.463696+0000","last_became_active":"2023-10-16T15:55:40.383650+0000","last_became_peered":"2023-10-16T15:55:40.383650+0000","last_unstale":"2023-10-16T15:55:41.463696+0000","last_undegraded":"2023-10-16T15:55:41.463696+0000","last_fullsized":"2023-10-16T15:55:41.463696+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:57.544172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372952+0000","last_change":"2023-10-16T15:55:39.377996+0000","last_active":"2023-10-16T15:55:41.372952+0000","last_peered":"2023-10-16T15:55:41.372952+0000","last_clean":"2023-10-16T15:55:41.372952+0000","last_became_active":"2023-10-16T15:55:39.376673+0000","last_became_peered":"2023-10-16T15:55:39.376673+0000","last_unstale":"2023-10-16T15:55:41.372952+0000","last_undegraded":"2023-10-16T15:55:41.372952+0000","last_fullsized":"2023-10-16T15:55:41.372952+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:59:14.934855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.367500+0000","last_change":"2023-10-16T15:55:40.387645+0000","last_active":"2023-10-16T15:55:41.367500+0000","last_peered":"2023-10-16T15:55:41.367500+0000","last_clean":"2023-10-16T15:55:41.367500+0000","last_became_active":"2023-10-16T15:55:40.386939+0000","last_became_peered":"2023-10-16T15:55:40.386939+0000","last_unstale":"2023-10-16T15:55:41.367500+0000","last_undegraded":"2023-10-16T15:55:41.367500+0000","last_fullsized":"2023-10-16T15:55:41.367500+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:25.827807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462928+0000","last_change":"2023-10-16T15:55:39.378661+0000","last_active":"2023-10-16T15:55:41.462928+0000","last_peered":"2023-10-16T15:55:41.462928+0000","last_clean":"2023-10-16T15:55:41.462928+0000","last_became_active":"2023-10-16T15:55:39.378375+0000","last_became_peered":"2023-10-16T15:55:39.378375+0000","last_unstale":"2023-10-16T15:55:41.462928+0000","last_undegraded":"2023-10-16T15:55:41.462928+0000","last_fullsized":"2023-10-16T15:55:41.462928+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:31.260722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463380+0000","last_change":"2023-10-16T15:55:40.398774+0000","last_active":"2023-10-16T15:55:41.463380+0000","last_peered":"2023-10-16T15:55:41.463380+0000","last_clean":"2023-10-16T15:55:41.463380+0000","last_became_active":"2023-10-16T15:55:40.398496+0000","last_became_peered":"2023-10-16T15:55:40.398496+0000","last_unstale":"2023-10-16T15:55:41.463380+0000","last_undegraded":"2023-10-16T15:55:41.463380+0000","last_fullsized":"2023-10-16T15:55:41.463380+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:38.036360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.463199+0000","last_change":"2023-10-16T15:55:39.368481+0000","last_active":"2023-10-16T15:55:41.463199+0000","last_peered":"2023-10-16T15:55:41.463199+0000","last_clean":"2023-10-16T15:55:41.463199+0000","last_became_active":"2023-10-16T15:55:39.367089+0000","last_became_peered":"2023-10-16T15:55:39.367089+0000","last_unstale":"2023-10-16T15:55:41.463199+0000","last_undegraded":"2023-10-16T15:55:41.463199+0000","last_fullsized":"2023-10-16T15:55:41.463199+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:54.624319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462628+0000","last_change":"2023-10-16T15:55:40.403822+0000","last_active":"2023-10-16T15:55:41.462628+0000","last_peered":"2023-10-16T15:55:41.462628+0000","last_clean":"2023-10-16T15:55:41.462628+0000","last_became_active":"2023-10-16T15:55:40.402310+0000","last_became_peered":"2023-10-16T15:55:40.402310+0000","last_unstale":"2023-10-16T15:55:41.462628+0000","last_undegraded":"2023-10-16T15:55:41.462628+0000","last_fullsized":"2023-10-16T15:55:41.462628+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:22.411416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462615+0000","last_change":"2023-10-16T15:55:39.367291+0000","last_active":"2023-10-16T15:55:41.462615+0000","last_peered":"2023-10-16T15:55:41.462615+0000","last_clean":"2023-10-16T15:55:41.462615+0000","last_became_active":"2023-10-16T15:55:39.366958+0000","last_became_peered":"2023-10-16T15:55:39.366958+0000","last_unstale":"2023-10-16T15:55:41.462615+0000","last_undegraded":"2023-10-16T15:55:41.462615+0000","last_fullsized":"2023-10-16T15:55:41.462615+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:43.436609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.371795+0000","last_change":"2023-10-16T15:55:40.416692+0000","last_active":"2023-10-16T15:55:41.371795+0000","last_peered":"2023-10-16T15:55:41.371795+0000","last_clean":"2023-10-16T15:55:41.371795+0000","last_became_active":"2023-10-16T15:55:40.416407+0000","last_became_peered":"2023-10-16T15:55:40.416407+0000","last_unstale":"2023-10-16T15:55:41.371795+0000","last_undegraded":"2023-10-16T15:55:41.371795+0000","last_fullsized":"2023-10-16T15:55:41.371795+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:32.274763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.462124+0000","last_change":"2023-10-16T15:55:39.366052+0000","last_active":"2023-10-16T15:55:41.462124+0000","last_peered":"2023-10-16T15:55:41.462124+0000","last_clean":"2023-10-16T15:55:41.462124+0000","last_became_active":"2023-10-16T15:55:39.365766+0000","last_became_peered":"2023-10-16T15:55:39.365766+0000","last_unstale":"2023-10-16T15:55:41.462124+0000","last_undegraded":"2023-10-16T15:55:41.462124+0000","last_fullsized":"2023-10-16T15:55:41.462124+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:38.316312+0000","last_clean_scrub_stamp":"2023-10-16T15:55:38.316312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:24.152311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":11,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-16T15:55:41.372122+0000","last_change":"2023-10-16T15:55:40.395448+0000","last_active":"2023-10-16T15:55:41.372122+0000","last_peered":"2023-10-16T15:55:41.372122+0000","last_clean":"2023-10-16T15:55:41.372122+0000","last_became_active":"2023-10-16T15:55:40.395178+0000","last_became_peered":"2023-10-16T15:55:40.395178+0000","last_unstale":"2023-10-16T15:55:41.372122+0000","last_undegraded":"2023-10-16T15:55:41.372122+0000","last_fullsized":"2023-10-16T15:55:41.372122+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T15:55:39.330353+0000","last_clean_scrub_stamp":"2023-10-16T15:55:39.330353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:25:23.759665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"263'2787","reported_seq":3840,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T15:55:40.370189+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:55:40.370189+0000","last_peered":"2023-10-16T15:55:40.370189+0000","last_clean":"2023-10-16T15:55:40.370189+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:55:40.370189+0000","last_undegraded":"2023-10-16T15:55:40.370189+0000","last_fullsized":"2023-10-16T15:55:40.370189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2487,"num_read_kb":5168,"num_write":645,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2487,"num_read_kb":5168,"num_write":645,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576028,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57252,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314588,"statfs":{"total":96636764160,"available":96578138112,"internally_reserved":0,"allocated":3952640,"data_stored":2252192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.095,"15min":1.171},"min":{"1min":0.61499999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.6779999999999999,"5min":1.794,"15min":1.8879999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.1040000000000001,"15min":1.149},"min":{"1min":0.71899999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6299999999999999,"5min":1.796,"15min":1.796},"last":1.1439999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1080000000000001,"15min":1.1679999999999999},"min":{"1min":0.82999999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4610000000000001,"5min":1.76,"15min":1.8580000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.244,"15min":1.3460000000000001},"min":{"1min":0.69199999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.7809999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3100000000000001,"15min":1.331},"min":{"1min":0.80400000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7410000000000001,"5min":1.893,"15min":2.1269999999999998},"last":1.621},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2310000000000001,"15min":1.296},"min":{"1min":0.877,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6020000000000001,"5min":1.909,"15min":1.95},"last":1.319}]},{"osd":3,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.2529999999999999,"15min":1.3260000000000001},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5149999999999999,"5min":1.843,"15min":1.869},"last":1.397},{"interface":"front","average":{"1min":1.423,"5min":1.3999999999999999,"15min":1.4450000000000001},"min":{"1min":1.089,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.8500000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3819999999999999,"15min":1.383},"min":{"1min":1.054,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6479999999999999,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.573},{"interface":"front","average":{"1min":1.347,"5min":1.3089999999999999,"15min":1.339},"min":{"1min":0.88100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6439999999999999,"5min":2.0259999999999998,"15min":2.0409999999999999},"last":1.3620000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4570000000000001,"15min":1.46},"min":{"1min":0.87,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.7889999999999999,"5min":2.0939999999999999,"15min":2.1469999999999998},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.502,"15min":1.5489999999999999},"min":{"1min":0.98599999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.04,"5min":2.04,"15min":2.3540000000000001},"last":1.9650000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:51 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.4550000000000001,"15min":1.4750000000000001},"min":{"1min":1.2390000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.8260000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.5669999999999999,"15min":1.5800000000000001},"min":{"1min":1.048,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.9850000000000001,"5min":1.988,"15min":2.2599999999999998},"last":1.9159999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52512,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319328,"statfs":{"total":96636764160,"available":96582991872,"internally_reserved":0,"allocated":3362816,"data_stored":1661914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.208,"15min":1.206},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9099999999999999,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.2130000000000001,"15min":1.2430000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":1.5309999999999999,"5min":1.905,"15min":1.905},"last":1.3440000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.72199999999999998},"max":{"1min":1.675,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.107},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2889999999999999,"15min":1.3160000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1639999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2330000000000001,"15min":1.232},"min":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.611,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3859999999999999,"15min":1.3959999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8400000000000001,"5min":1.905,"15min":2.0339999999999998},"last":1.212}]},{"osd":3,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.361,"15min":1.391},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.9079999999999999,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.284,"15min":1.331},"min":{"1min":0.82799999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.554}]},{"osd":4,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.421,"15min":1.351},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.978,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.3},{"interface":"front","average":{"1min":1.512,"5min":1.458,"15min":1.4119999999999999},"min":{"1min":0.81100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0830000000000002,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.0189999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3759999999999999,"15min":1.357},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.67300000000000004},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.627},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5980000000000001,"15min":1.54},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1059999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.667}]},{"osd":7,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.6419999999999999,"15min":1.6040000000000001},"min":{"1min":0.86699999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.4940000000000002},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.599,"5min":1.6399999999999999,"15min":1.5980000000000001},"min":{"1min":0.98099999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.778}]}]},{"osd":5,"up_from":13,"seq":55834576027,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30320,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341520,"statfs":{"total":96636764160,"available":96605716480,"internally_reserved":0,"allocated":3379200,"data_stored":1664088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.111,"15min":1.1579999999999999},"min":{"1min":0.68899999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.802,"5min":1.8069999999999999,"15min":1.821},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.234,"15min":1.24},"min":{"1min":0.79300000000000004,"5min":0.622,"15min":0.622},"max":{"1min":1.766,"5min":2.738,"15min":2.738},"last":0.93000000000000005}]},{"osd":1,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.196,"15min":1.228},"min":{"1min":0.66600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6639999999999999,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.2410000000000001,"15min":1.2889999999999999},"min":{"1min":0.65300000000000002,"5min":0.443,"15min":0.443},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.5129999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.22,"15min":1.248},"min":{"1min":0.61799999999999999,"5min":0.505,"15min":0.505},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.339},{"interface":"front","average":{"1min":1.325,"5min":1.375,"15min":1.3939999999999999},"min":{"1min":0.77500000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.2160000000000002},"last":1.891}]},{"osd":3,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3300000000000001,"15min":1.335},"min":{"1min":0.753,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8720000000000001,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.3260000000000001,"15min":1.3740000000000001},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.966,"5min":1.966,"15min":2.1589999999999998},"last":1.232}]},{"osd":4,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3799999999999999,"15min":1.347},"min":{"1min":0.79900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.8500000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5049999999999999,"15min":1.5149999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0800000000000001,"5min":2.2829999999999999,"15min":2.3210000000000002},"last":1.397}]},{"osd":6,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4630000000000001,"15min":1.385},"min":{"1min":0.81000000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0369999999999999,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":2.077},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.468,"15min":1.4319999999999999},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9870000000000001,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3}]},{"osd":7,"last update":"Mon Oct 16 15:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.5289999999999999,"15min":1.532},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.125,"5min":2.4729999999999999,"15min":2.5129999999999999},"last":1.474},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.651,"15min":1.631},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.4340000000000002,"5min":3.109,"15min":3.109},"last":2.1890000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576028,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70568,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301272,"statfs":{"total":96636764160,"available":96564502528,"internally_reserved":0,"allocated":3366912,"data_stored":1661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.1419999999999999,"15min":1.194},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.512},{"interface":"front","average":{"1min":1.085,"5min":1.159,"15min":1.194},"min":{"1min":0.76300000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.4059999999999999,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1890000000000001,"15min":1.2310000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.629},"max":{"1min":1.6579999999999999,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.663},{"interface":"front","average":{"1min":1.254,"5min":1.284,"15min":1.3089999999999999},"min":{"1min":0.75900000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9750000000000001,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":0.83999999999999997}]},{"osd":2,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3939999999999999,"15min":1.4510000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.7150000000000001,"5min":2.1970000000000001,"15min":2.8929999999999998},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.456,"15min":1.492},"min":{"1min":0.82499999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6819999999999999,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.8109999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.391,"15min":1.385},"min":{"1min":1.02,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8129999999999999,"5min":2.23,"15min":2.3650000000000002},"last":1.873},{"interface":"front","average":{"1min":1.232,"5min":1.341,"15min":1.383},"min":{"1min":0.92100000000000004,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.7509999999999999,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.744}]},{"osd":5,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.371,"15min":1.341},"min":{"1min":0.80200000000000005,"5min":0.76800000000000002,"15min":0.754},"max":{"1min":1.849,"5min":2.3660000000000001,"15min":2.5089999999999999},"last":1.617},{"interface":"front","average":{"1min":1.417,"5min":1.4730000000000001,"15min":1.4410000000000001},"min":{"1min":0.88900000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.7949999999999999,"5min":2.161,"15min":2.4489999999999998},"last":1.196}]},{"osd":6,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.389,"15min":1.405},"min":{"1min":0.97699999999999998,"5min":0.77700000000000002,"15min":0.75900000000000001},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.29},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5629999999999999,"15min":1.526},"min":{"1min":1.1990000000000001,"5min":0.85899999999999999,"15min":0.82499999999999996},"max":{"1min":2.246,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.2350000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:48 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6100000000000001,"15min":1.583},"min":{"1min":1.04,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.9730000000000001,"5min":2.399,"15min":2.399},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.6000000000000001,"15min":1.601},"min":{"1min":1.129,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.4630000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56684,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315156,"statfs":{"total":96636764160,"available":96578719744,"internally_reserved":0,"allocated":3371008,"data_stored":1663436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.92100000000000004,"15min":0.96799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.419,"5min":1.48,"15min":2.0430000000000001},"last":1.042},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0189999999999999,"15min":1.0580000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.722,"5min":1.7889999999999999,"15min":2.5990000000000002},"last":0.995}]},{"osd":1,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94199999999999995,"15min":1.0189999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.776,"5min":1.776,"15min":2.589},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.147,"15min":1.204},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.556,"5min":1.748,"15min":2.5569999999999999},"last":1.264}]},{"osd":2,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.304,"15min":1.403},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.302},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3919999999999999,"15min":1.488},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.954,"5min":1.954,"15min":2.754},"last":1.7}]},{"osd":4,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2669999999999999,"15min":1.3440000000000001},"min":{"1min":0.79100000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9770000000000001,"5min":2.0110000000000001,"15min":2.512},"last":1.212},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3440000000000001,"15min":1.4119999999999999},"min":{"1min":0.76300000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.0249999999999999,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.3260000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.53,"15min":1.5620000000000001},"min":{"1min":0.94099999999999995,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2280000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4890000000000001,"15min":1.5629999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.2599999999999998,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.5589999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.46,"15min":1.5},"min":{"1min":0.97499999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.0680000000000001,"5min":2.173,"15min":2.2090000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.5649999999999999,"15min":1.6000000000000001},"min":{"1min":0.85299999999999998,"5min":0.749,"15min":0.749},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.7669999999999999},"last":1.5920000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.591,"15min":1.6779999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":1.7829999999999999},{"interface":"front","average":{"1min":1.571,"5min":1.579,"15min":1.6339999999999999},"min":{"1min":1.002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.133,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.637}]}]},{"osd":2,"up_from":13,"seq":55834576028,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56676,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315164,"statfs":{"total":96636764160,"available":96578727936,"internally_reserved":0,"allocated":3362816,"data_stored":1661914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.89300000000000002,"15min":0.93700000000000006},"min":{"1min":0.443,"5min":0.378,"15min":0.378},"max":{"1min":1.2749999999999999,"5min":1.726,"15min":1.76},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.093},"min":{"1min":0.39300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7789999999999999,"5min":1.804,"15min":1.8999999999999999},"last":1.6739999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0209999999999999,"15min":1.046},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.504,"5min":1.974,"15min":1.974},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.1479999999999999,"15min":1.224},"min":{"1min":0.55000000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.772,"5min":1.879,"15min":2.0779999999999998},"last":1.1879999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.161,"15min":1.2709999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.5289999999999999,"5min":1.704,"15min":2.1509999999999998},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.3180000000000001,"15min":1.365},"min":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.661,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.389}]},{"osd":4,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.448,"15min":1.4730000000000001},"min":{"1min":0.84499999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.9299999999999999,"5min":2.335,"15min":2.335},"last":2.0409999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.345,"15min":1.411},"min":{"1min":0.65900000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.901,"5min":1.915,"15min":2.1469999999999998},"last":1.28}]},{"osd":5,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.403,"15min":1.4730000000000001},"min":{"1min":0.72899999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.8080000000000001,"5min":2.657,"15min":2.657},"last":2.008},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.5449999999999999,"15min":1.6100000000000001},"min":{"1min":0.81499999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.093,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.4790000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4570000000000001,"15min":1.5429999999999999},"min":{"1min":0.77900000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.861,"5min":1.9610000000000001,"15min":2.3849999999999998},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5589999999999999,"15min":1.6100000000000001},"min":{"1min":0.89300000000000002,"5min":0.753,"15min":0.753},"max":{"1min":1.8500000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.5}]},{"osd":7,"last update":"Mon Oct 16 15:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.5649999999999999,"15min":1.6499999999999999},"min":{"1min":0.85999999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.234,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":2.181},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.6499999999999999,"15min":1.722},"min":{"1min":1.0129999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.1890000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":2.0960000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576028,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52520,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319320,"statfs":{"total":96636764160,"available":96582983680,"internally_reserved":0,"allocated":3362816,"data_stored":1661858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.90900000000000003,"15min":0.93999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.351,"5min":1.49,"15min":1.8420000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98899999999999999,"15min":1.113},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6639999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.97999999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.94199999999999995,"15min":1.0209999999999999},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.764,"5min":1.764,"15min":2.286},"last":0.87},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.125,"15min":1.214},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7929999999999999,"15min":1.853},"last":0.74199999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.254,"15min":1.353},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.651,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1879999999999999,"15min":1.2929999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.532,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":0.85099999999999998}]},{"osd":4,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.319,"15min":1.365},"min":{"1min":0.89200000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.915,"5min":2.327,"15min":2.3809999999999998},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.47,"15min":1.5109999999999999},"min":{"1min":0.874,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.8759999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.4059999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.52},"min":{"1min":0.86499999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9830000000000001,"5min":2.456,"15min":2.456},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.4450000000000001,"15min":1.484},"min":{"1min":0.79200000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9490000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.0509999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.595,"15min":1.641},"min":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0489999999999999,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.635},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5880000000000001,"15min":1.637},"min":{"1min":0.84199999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.343,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.427}]},{"osd":7,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.6499999999999999,"15min":1.7310000000000001},"min":{"1min":0.79900000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1379999999999999,"5min":2.734,"15min":3.2090000000000001},"last":1.554},{"interface":"front","average":{"1min":1.381,"5min":1.5820000000000001,"15min":1.6399999999999999},"min":{"1min":0.89000000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1110000000000002,"5min":2.452,"15min":3.1539999999999999},"last":1.24}]}]},{"osd":0,"up_from":13,"seq":55834576028,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61420,"kb_used_data":3872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310420,"statfs":{"total":96636764160,"available":96573870080,"internally_reserved":0,"allocated":3964928,"data_stored":2252830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91000000000000003,"15min":0.94999999999999996},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.2789999999999999,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.1180000000000001,"15min":1.2070000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.5800000000000001,"5min":2.036,"15min":2.036},"last":0.99299999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.1739999999999999,"15min":1.26},"min":{"1min":0.56499999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.117,"15min":1.2170000000000001},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.462,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.70699999999999996}]},{"osd":3,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.004,"15min":1.032},"min":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.6339999999999999,"5min":1.6799999999999999,"15min":1.7889999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.032,"5min":1.175,"15min":1.2549999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.774,"5min":1.9850000000000001,"15min":2.0899999999999999},"last":1.1180000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.319,"15min":1.337},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":2.137},"last":1.042},{"interface":"front","average":{"1min":1.327,"5min":1.401,"15min":1.4970000000000001},"min":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.9390000000000001,"5min":1.99,"15min":2.452},"last":1.073}]},{"osd":5,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.4239999999999999,"15min":1.476},"min":{"1min":0.66200000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8779999999999999,"5min":2.137,"15min":2.137},"last":1.026},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.345,"15min":1.411},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7609999999999999,"5min":2.0859999999999999,"15min":2.282},"last":1.155}]},{"osd":6,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.5349999999999999,"15min":1.6259999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.7010000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.526,"15min":1.569},"min":{"1min":0.73299999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.1869999999999998,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.2050000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:54:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.621,"15min":1.6779999999999999},"min":{"1min":0.82199999999999995,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1480000000000001,"5min":2.339,"15min":2.339},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.5760000000000001,"15min":1.601},"min":{"1min":0.89200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.1259999999999999,"5min":2.1739999999999999,"15min":2.274},"last":1.341}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:55:45.414 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T15:55:45.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T15:55:45.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.686+0000 7ff7dbce7640 1 Processor -- start 2023-10-16T15:55:45.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.686+0000 7ff7dbce7640 1 -- start start 2023-10-16T15:55:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.688+0000 7ff7dbce7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff7d40b0b20 0x7ff7d415d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.688+0000 7ff7dbce7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d41615d0 0x7ff7d415d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.688+0000 7ff7dbce7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d415e3c0 con 0x7ff7d40ccfc0 2023-10-16T15:55:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.688+0000 7ff7dbce7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d415de70 con 0x7ff7d40b0b20 2023-10-16T15:55:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.688+0000 7ff7dbce7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d415e0e0 con 0x7ff7d41615d0 2023-10-16T15:55:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.688+0000 7ff7d9a83640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff7d40ccfc0 0x7ff7d40b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60648/0 (socket says 172.21.15.29:60648) 2023-10-16T15:55:45.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d9a83640 1 -- 172.21.15.29:0/1271635124 learned_addr learned my addr 172.21.15.29:0/1271635124 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:45.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d9282640 1 --2- 172.21.15.29:0/1271635124 >> v2:172.21.15.29:3300/0 conn(0x7ff7d40b0b20 0x7ff7d415d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7da284640 1 --2- 172.21.15.29:0/1271635124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d41615d0 0x7ff7d415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d8a81640 1 -- 172.21.15.29:0/1271635124 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669248710 0 0) 0x7ff7d415e3c0 con 0x7ff7d40ccfc0 2023-10-16T15:55:45.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d8a81640 1 -- 172.21.15.29:0/1271635124 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c0003520 con 0x7ff7d40ccfc0 2023-10-16T15:55:45.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d9282640 1 -- 172.21.15.29:0/1271635124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d41615d0 msgr2=0x7ff7d415d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:45.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d9282640 1 --2- 172.21.15.29:0/1271635124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d41615d0 0x7ff7d415d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d9282640 1 -- 172.21.15.29:0/1271635124 >> v1:172.21.15.29:6789/0 conn(0x7ff7d40ccfc0 legacy=0x7ff7d40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.689+0000 7ff7d9282640 1 -- 172.21.15.29:0/1271635124 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d4160af0 con 0x7ff7d40b0b20 2023-10-16T15:55:45.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.692+0000 7ff7d9282640 1 --2- 172.21.15.29:0/1271635124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d40b0b20 0x7ff7d415d3f0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40097a0 tx=0x7ff7c402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d786431a012ec56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:45.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.693+0000 7ff7d8a81640 1 -- 172.21.15.29:0/1271635124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4031780 con 0x7ff7d40b0b20 2023-10-16T15:55:45.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.693+0000 7ff7d8a81640 1 -- 172.21.15.29:0/1271635124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c40318e0 con 0x7ff7d40b0b20 2023-10-16T15:55:45.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.693+0000 7ff7d8a81640 1 -- 172.21.15.29:0/1271635124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4031c70 con 0x7ff7d40b0b20 2023-10-16T15:55:45.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 -- 172.21.15.29:0/1271635124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d40b0b20 msgr2=0x7ff7d415d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/1271635124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d40b0b20 0x7ff7d415d3f0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40097a0 tx=0x7ff7c402b820 comp rx=0 tx=0).stop 2023-10-16T15:55:45.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 -- 172.21.15.29:0/1271635124 shutdown_connections 2023-10-16T15:55:45.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/1271635124 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d41615d0 0x7ff7d415d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/1271635124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d40b0b20 0x7ff7d415d3f0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 -- 172.21.15.29:0/1271635124 >> 172.21.15.29:0/1271635124 conn(0x7ff7d40cb410 msgr2=0x7ff7d40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:45.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 -- 172.21.15.29:0/1271635124 shutdown_connections 2023-10-16T15:55:45.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.694+0000 7ff7dbce7640 1 -- 172.21.15.29:0/1271635124 wait complete. 2023-10-16T15:55:45.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.696+0000 7ff7dbce7640 1 Processor -- start 2023-10-16T15:55:45.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.696+0000 7ff7dbce7640 1 -- start start 2023-10-16T15:55:45.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.697+0000 7ff7dbce7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d40ccfc0 0x7ff7d4159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.697+0000 7ff7dbce7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 0x7ff7d41569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.697+0000 7ff7dbce7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7d40b2650 0x7ff7d40b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.697+0000 7ff7dbce7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40b40c0 con 0x7ff7d41615d0 2023-10-16T15:55:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.697+0000 7ff7dbce7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d4156f10 con 0x7ff7d40b2650 2023-10-16T15:55:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.697+0000 7ff7dbce7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40b3de0 con 0x7ff7d40ccfc0 2023-10-16T15:55:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 0x7ff7d41569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 0x7ff7d41569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54022/0 (socket says 172.21.15.29:54022) 2023-10-16T15:55:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 -- 172.21.15.29:0/3831310825 learned_addr learned my addr 172.21.15.29:0/3831310825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9a83640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d40ccfc0 0x7ff7d4159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7da284640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7d40b2650 0x7ff7d40b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 -- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d40ccfc0 msgr2=0x7ff7d4159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d40ccfc0 0x7ff7d4159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 -- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7d40b2650 msgr2=0x7ff7d40b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7d40b2650 0x7ff7d40b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 -- 172.21.15.29:0/3831310825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c4009400 con 0x7ff7d41615d0 2023-10-16T15:55:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7da284640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7d40b2650 0x7ff7d40b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:45.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.698+0000 7ff7d9282640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 0x7ff7d41569d0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7ff7c402bdb0 tx=0x7ff7c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:45.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.699+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4004230 con 0x7ff7d41615d0 2023-10-16T15:55:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.699+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c4004390 con 0x7ff7d41615d0 2023-10-16T15:55:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.699+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4003850 con 0x7ff7d41615d0 2023-10-16T15:55:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.699+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d40c35b0 con 0x7ff7d41615d0 2023-10-16T15:55:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.699+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d40c3b70 con 0x7ff7d41615d0 2023-10-16T15:55:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.700+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d40c3410 con 0x7ff7d41615d0 2023-10-16T15:55:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.701+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff7c4043070 con 0x7ff7d41615d0 2023-10-16T15:55:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.702+0000 7ff7ca7fc640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff7b806b0b0 0x7ff7b806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.702+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7ff7c40ab6b0 con 0x7ff7d41615d0 2023-10-16T15:55:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.703+0000 7ff7d9a83640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff7b806b0b0 0x7ff7b806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.704+0000 7ff7d9a83640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff7b806b0b0 0x7ff7b806d570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0005880 tx=0x7ff7d0009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.707+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7d40c3410 con 0x7ff7d41615d0 2023-10-16T15:55:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.983+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff7d40c3410 con 0x7ff7d41615d0 2023-10-16T15:55:45.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.988+0000 7ff7ca7fc640 1 -- 172.21.15.29:0/3831310825 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff7d40c3410 con 0x7ff7d41615d0 2023-10-16T15:55:45.990 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-16T15:55:45.990 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQB1RS1l3VdnJhAAxxe609vO9oTmtGXzcxD3XA== 2023-10-16T15:55:45.990 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-16T15:55:45.991 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-16T15:55:45.991 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T15:55:45.991 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-16T15:55:45.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff7b806b0b0 msgr2=0x7ff7b806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff7b806b0b0 0x7ff7b806d570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0005880 tx=0x7ff7d0009040 comp rx=0 tx=0).stop 2023-10-16T15:55:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 msgr2=0x7ff7d41569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 0x7ff7d41569d0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7ff7c402bdb0 tx=0x7ff7c40046c0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 shutdown_connections 2023-10-16T15:55:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7ff7b806b0b0 0x7ff7b806d570 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7d41615d0 0x7ff7d41569d0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff7d40b2650 0x7ff7d40b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 --2- 172.21.15.29:0/3831310825 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff7d40ccfc0 0x7ff7d4159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 >> 172.21.15.29:0/3831310825 conn(0x7ff7d40cb410 msgr2=0x7ff7d40afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 shutdown_connections 2023-10-16T15:55:45.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:45.991+0000 7ff7dbce7640 1 -- 172.21.15.29:0/3831310825 wait complete. 2023-10-16T15:55:46.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:55:46.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.227+0000 7f421d1ff640 1 Processor -- start 2023-10-16T15:55:46.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.228+0000 7f421d1ff640 1 -- start start 2023-10-16T15:55:46.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.229+0000 7f421d1ff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f421815cd70 0x7f42180d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.229+0000 7f421d1ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f42180cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.229+0000 7f421d1ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421815d150 con 0x7f42180d0be0 2023-10-16T15:55:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.229+0000 7f421d1ff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42180d1180 con 0x7f421815cd70 2023-10-16T15:55:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.229+0000 7f421d1ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180d1460 con 0x7f42180b8480 2023-10-16T15:55:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.230+0000 7f421c9fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42180d0be0 0x7f42180b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60660/0 (socket says 172.21.15.29:60660) 2023-10-16T15:55:46.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.230+0000 7f421c9fe640 1 -- 172.21.15.29:0/85813796 learned_addr learned my addr 172.21.15.29:0/85813796 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.230+0000 7f42177fe640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f42180cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.230+0000 7f4217fff640 1 --2- 172.21.15.29:0/85813796 >> v2:172.21.15.170:3300/0 conn(0x7f421815cd70 0x7f42180d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.230+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075867286 0 0) 0x7f421815d150 con 0x7f42180d0be0 2023-10-16T15:55:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.230+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4200003520 con 0x7f42180d0be0 2023-10-16T15:55:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3134810657 0 0) 0x7f4200003520 con 0x7f42180d0be0 2023-10-16T15:55:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 msgr2=0x7f42180cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f42180cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 >> v2:172.21.15.170:3300/0 conn(0x7f421815cd70 msgr2=0x7f42180d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 --2- 172.21.15.29:0/85813796 >> v2:172.21.15.170:3300/0 conn(0x7f421815cd70 0x7f42180d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180d1740 con 0x7f42180d0be0 2023-10-16T15:55:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f4210001670 con 0x7f42180d0be0 2023-10-16T15:55:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 >> v1:172.21.15.29:6789/0 conn(0x7f42180d0be0 legacy=0x7f42180b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4200003970 0x7f4200003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42000047c0 0x7f4200004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f4200005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180d1180 con 0x7f4200003970 2023-10-16T15:55:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180d1460 con 0x7f42000047c0 2023-10-16T15:55:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420000b980 con 0x7f42180b8480 2023-10-16T15:55:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.231+0000 7f4217fff640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4200003970 0x7f4200003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f421c9fe640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f4200005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f42177fe640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42000047c0 0x7f4200004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f4217fff640 1 -- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 msgr2=0x7f4200005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f4217fff640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f4200005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f4217fff640 1 -- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42000047c0 msgr2=0x7f4200004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f4217fff640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42000047c0 0x7f4200004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f4217fff640 1 -- 172.21.15.29:0/85813796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200003520 con 0x7f4200003970 2023-10-16T15:55:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f42177fe640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42000047c0 0x7f4200004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.232+0000 7f421c9fe640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f42180b8480 0x7f4200005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.233+0000 7f4217fff640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4200003970 0x7f4200003d90 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f4208009fc0 tx=0x7f42000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555803a9355229a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.233+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420802cc50 con 0x7f4200003970 2023-10-16T15:55:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.233+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420800f040 con 0x7f4200003970 2023-10-16T15:55:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.233+0000 7f4216ffd640 1 -- 172.21.15.29:0/85813796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208034440 con 0x7f4200003970 2023-10-16T15:55:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421d1ff640 1 -- 172.21.15.29:0/85813796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4200003970 msgr2=0x7f4200003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421d1ff640 1 --2- 172.21.15.29:0/85813796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4200003970 0x7f4200003d90 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f4208009fc0 tx=0x7f42000057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421c9fe640 1 -- 172.21.15.29:0/85813796 reap_dead start 2023-10-16T15:55:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421d1ff640 1 -- 172.21.15.29:0/85813796 shutdown_connections 2023-10-16T15:55:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421d1ff640 1 -- 172.21.15.29:0/85813796 >> 172.21.15.29:0/85813796 conn(0x7f42180cf570 msgr2=0x7f42180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421d1ff640 1 -- 172.21.15.29:0/85813796 shutdown_connections 2023-10-16T15:55:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.234+0000 7f421d1ff640 1 -- 172.21.15.29:0/85813796 wait complete. 2023-10-16T15:55:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.236+0000 7f421d1ff640 1 Processor -- start 2023-10-16T15:55:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.236+0000 7f421d1ff640 1 -- start start 2023-10-16T15:55:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421d1ff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4200003970 0x7f4218154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421d1ff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421815d340 0x7f42181f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421d1ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 0x7f421815f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421d1ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421815fab0 con 0x7f421815f0c0 2023-10-16T15:55:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421d1ff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218154f30 con 0x7f4200003970 2023-10-16T15:55:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421d1ff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421815e410 con 0x7f421815d340 2023-10-16T15:55:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421c9fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 0x7f421815f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421c9fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 0x7f421815f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54040/0 (socket says 172.21.15.29:54040) 2023-10-16T15:55:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f421c9fe640 1 -- 172.21.15.29:0/2571251414 learned_addr learned my addr 172.21.15.29:0/2571251414 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.237+0000 7f4217fff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4200003970 0x7f4218154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f42177fe640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421815d340 0x7f42181f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421c9fe640 1 -- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421815d340 msgr2=0x7f42181f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421c9fe640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421815d340 0x7f42181f2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421c9fe640 1 -- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4200003970 msgr2=0x7f4218154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421c9fe640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4200003970 0x7f4218154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421c9fe640 1 -- 172.21.15.29:0/2571251414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42181548a0 con 0x7f421815f0c0 2023-10-16T15:55:46.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f42177fe640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421815d340 0x7f42181f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:46.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f4217fff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4200003970 0x7f4218154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:46.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421c9fe640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 0x7f421815f570 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f421808b0b0 tx=0x7f421000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:46.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42100074e0 con 0x7f421815f0c0 2023-10-16T15:55:46.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4210007640 con 0x7f421815f0c0 2023-10-16T15:55:46.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210004a20 con 0x7f421815f0c0 2023-10-16T15:55:46.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42181f3230 con 0x7f421815f0c0 2023-10-16T15:55:46.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.238+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421815dae0 con 0x7f421815f0c0 2023-10-16T15:55:46.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.240+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421804e9b0 con 0x7f421815f0c0 2023-10-16T15:55:46.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.241+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4210004b80 con 0x7f421815f0c0 2023-10-16T15:55:46.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.242+0000 7f4214ff9640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f420406b0b0 0x7f420406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:46.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.242+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f42100898d0 con 0x7f421815f0c0 2023-10-16T15:55:46.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.242+0000 7f4217fff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f420406b0b0 0x7f420406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:46.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.243+0000 7f4217fff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f420406b0b0 0x7f420406d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4208036000 tx=0x7f420802e000 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:46.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.246+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4210054420 con 0x7f421815f0c0 2023-10-16T15:55:46.324 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-16T15:55:46.322+0000 7fcd85b0c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:55:46.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.484+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f421815e550 con 0x7f421815f0c0 2023-10-16T15:55:46.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.486+0000 7f4214ff9640 1 -- 172.21.15.29:0/2571251414 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f4210057a40 con 0x7f421815f0c0 2023-10-16T15:55:46.488 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:46.488 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:55:39.826547+0000","modified":"2023-10-16T15:55:42.906111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12345},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12345":{"gid":12345,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2425185007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2425185007},{"type":"v1","addr":"172.21.15.29:6835","nonce":2425185007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T15:55:46.488 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-16T15:55:46.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.488+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f420406b0b0 msgr2=0x7f420406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:46.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.488+0000 7f421d1ff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f420406b0b0 0x7f420406d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4208036000 tx=0x7f420802e000 comp rx=0 tx=0).stop 2023-10-16T15:55:46.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.488+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 msgr2=0x7f421815f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:46.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.488+0000 7f421d1ff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 0x7f421815f570 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f421808b0b0 tx=0x7f421000d9d0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 shutdown_connections 2023-10-16T15:55:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f420406b0b0 0x7f420406d570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f421815f0c0 0x7f421815f570 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f421815d340 0x7f42181f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 --2- 172.21.15.29:0/2571251414 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4200003970 0x7f4218154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:46.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 >> 172.21.15.29:0/2571251414 conn(0x7f42180cf570 msgr2=0x7f42180d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 shutdown_connections 2023-10-16T15:55:46.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:46.489+0000 7f421d1ff640 1 -- 172.21.15.29:0/2571251414 wait complete. 2023-10-16T15:55:46.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 141, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T15:55:39.826547+0000', 'modified': '2023-10-16T15:55:42.906111+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12345}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12345': {'gid': 12345, 'name': 'a', 'rank': 0, 'incarnation': 137, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6835/2425185007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 2425185007}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2425185007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T15:55:46.913 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T15:55:46.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T15:55:47.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.147+0000 7f705ad1a640 1 Processor -- start 2023-10-16T15:55:47.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.147+0000 7f705ad1a640 1 -- start start 2023-10-16T15:55:47.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.149+0000 7f705ad1a640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f70540ad970 0x7f70540cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.149+0000 7f705ad1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705415b9d0 con 0x7f70540ccfb0 2023-10-16T15:55:47.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.149+0000 7f705ad1a640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705415bcb0 con 0x7f705406c040 2023-10-16T15:55:47.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.149+0000 7f705ad1a640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705415bf90 con 0x7f70540ad970 2023-10-16T15:55:47.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.150+0000 7f705a519640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70540ccfb0 0x7f70540cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60672/0 (socket says 172.21.15.29:60672) 2023-10-16T15:55:47.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.150+0000 7f705a519640 1 -- 172.21.15.29:0/2027560340 learned_addr learned my addr 172.21.15.29:0/2027560340 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:47.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.150+0000 7f7059517640 1 --2- 172.21.15.29:0/2027560340 >> v2:172.21.15.170:3300/0 conn(0x7f70540ad970 0x7f70540cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.150+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588151817 0 0) 0x7f705415b9d0 con 0x7f70540ccfb0 2023-10-16T15:55:47.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.150+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7040003520 con 0x7f70540ccfb0 2023-10-16T15:55:47.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131345633 0 0) 0x7f7040003520 con 0x7f70540ccfb0 2023-10-16T15:55:47.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 >> v2:172.21.15.170:3300/0 conn(0x7f70540ad970 msgr2=0x7f70540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 --2- 172.21.15.29:0/2027560340 >> v2:172.21.15.170:3300/0 conn(0x7f70540ad970 0x7f70540cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 >> v1:172.21.15.170:6789/0 conn(0x7f705406c040 legacy=0x7f705406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540cc270 con 0x7f70540ccfb0 2023-10-16T15:55:47.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f704c0019f0 con 0x7f70540ccfb0 2023-10-16T15:55:47.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 >> v1:172.21.15.29:6789/0 conn(0x7f70540ccfb0 legacy=0x7f70540cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040003970 0x7f7040003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70400047c0 0x7f7040004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7040005180 0x7f704000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705415bf90 con 0x7f70400047c0 2023-10-16T15:55:47.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704000c020 con 0x7f7040005180 2023-10-16T15:55:47.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704000c1e0 con 0x7f7040003970 2023-10-16T15:55:47.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.151+0000 7f7059517640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70400047c0 0x7f7040004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f705a519640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7040005180 0x7f704000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f7059517640 1 -- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040003970 msgr2=0x7f7040003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f7059517640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7040003970 0x7f7040003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f7059517640 1 -- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7040005180 msgr2=0x7f704000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f7059517640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7040005180 0x7f704000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f7059517640 1 -- 172.21.15.29:0/2027560340 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040003520 con 0x7f70400047c0 2023-10-16T15:55:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f705a519640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7040005180 0x7f704000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.152+0000 7f705a519640 1 -- 172.21.15.29:0/2027560340 reap_dead start 2023-10-16T15:55:47.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.153+0000 7f7059517640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70400047c0 0x7f7040004c40 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f7044009a90 tx=0x7f70400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6813388beb7faf78 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.153+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704402c680 con 0x7f70400047c0 2023-10-16T15:55:47.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.153+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704402c7e0 con 0x7f70400047c0 2023-10-16T15:55:47.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.153+0000 7f7058d16640 1 -- 172.21.15.29:0/2027560340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704402cb90 con 0x7f70400047c0 2023-10-16T15:55:47.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 -- 172.21.15.29:0/2027560340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70400047c0 msgr2=0x7f7040004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70400047c0 0x7f7040004c40 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f7044009a90 tx=0x7f70400057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 -- 172.21.15.29:0/2027560340 shutdown_connections 2023-10-16T15:55:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 --2- 172.21.15.29:0/2027560340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70400047c0 0x7f7040004c40 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 -- 172.21.15.29:0/2027560340 >> 172.21.15.29:0/2027560340 conn(0x7f70540ce6a0 msgr2=0x7f70540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 -- 172.21.15.29:0/2027560340 shutdown_connections 2023-10-16T15:55:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.154+0000 7f705ad1a640 1 -- 172.21.15.29:0/2027560340 wait complete. 2023-10-16T15:55:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.156+0000 7f705ad1a640 1 Processor -- start 2023-10-16T15:55:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.156+0000 7f705ad1a640 1 -- start start 2023-10-16T15:55:47.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705ad1a640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70400047c0 0x7f705415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705ad1a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70541fa940 0x7f705415e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705ad1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 0x7f70540c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705ad1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705415eb40 con 0x7f705415e6c0 2023-10-16T15:55:47.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705ad1a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70541fa4b0 con 0x7f70541fa940 2023-10-16T15:55:47.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705ad1a640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705415f020 con 0x7f70400047c0 2023-10-16T15:55:47.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 0x7f70540c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 0x7f70540c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54064/0 (socket says 172.21.15.29:54064) 2023-10-16T15:55:47.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.157+0000 7f705a519640 1 -- 172.21.15.29:0/3525075694 learned_addr learned my addr 172.21.15.29:0/3525075694 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f7059d18640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70400047c0 0x7f705415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f7059517640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70541fa940 0x7f705415e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f705a519640 1 -- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70400047c0 msgr2=0x7f705415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f705a519640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70400047c0 0x7f705415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f705a519640 1 -- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70541fa940 msgr2=0x7f705415e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f705a519640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70541fa940 0x7f705415e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f705a519640 1 -- 172.21.15.29:0/3525075694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705415b0b0 con 0x7f705415e6c0 2023-10-16T15:55:47.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f7059517640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70541fa940 0x7f705415e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f705a519640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 0x7f70540c7890 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f705406d2e0 tx=0x7f704c00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f704a7fc640 1 -- 172.21.15.29:0/3525075694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c00b870 con 0x7f705415e6c0 2023-10-16T15:55:47.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f704a7fc640 1 -- 172.21.15.29:0/3525075694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c00b9f0 con 0x7f705415e6c0 2023-10-16T15:55:47.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.158+0000 7f704a7fc640 1 -- 172.21.15.29:0/3525075694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c00bda0 con 0x7f705415e6c0 2023-10-16T15:55:47.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.159+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705415ed40 con 0x7f705415e6c0 2023-10-16T15:55:47.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.159+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70541fa140 con 0x7f705415e6c0 2023-10-16T15:55:47.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.161+0000 7f704a7fc640 1 -- 172.21.15.29:0/3525075694 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f704c007110 con 0x7f705415e6c0 2023-10-16T15:55:47.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.162+0000 7f704a7fc640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f703806b0b0 0x7f703806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.162+0000 7f704a7fc640 1 -- 172.21.15.29:0/3525075694 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f704c089f40 con 0x7f705415e6c0 2023-10-16T15:55:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.163+0000 7f705ebe6740 1 Processor -- start 2023-10-16T15:55:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.163+0000 7f7059d18640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f703806b0b0 0x7f703806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.163+0000 7f705ebe6740 1 -- start start 2023-10-16T15:55:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943170 0x55bb28916070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f7059d18640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f703806b0b0 0x7f703806d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7050001650 tx=0x7f7050009040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb28945f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb289165b0 0x55bb28946440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f705ebe6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb288367a0 con 0x55bb28943550 2023-10-16T15:55:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb28888740 con 0x55bb289165b0 2023-10-16T15:55:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb28885e30 con 0x55bb28943170 2023-10-16T15:55:47.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f7059517640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb28945f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f7059517640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb28945f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54068/0 (socket says 172.21.15.29:54068) 2023-10-16T15:55:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f7059517640 1 -- 172.21.15.29:0/3342979676 learned_addr learned my addr 172.21.15.29:0/3342979676 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f705a519640 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb289165b0 0x55bb28946440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.164+0000 7f7059d18640 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943170 0x55bb28916070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f7059517640 1 -- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943170 msgr2=0x55bb28916070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f7059517640 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943170 0x55bb28916070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f7059517640 1 -- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb289165b0 msgr2=0x55bb28946440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f7059517640 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb289165b0 0x55bb28946440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f7059517640 1 -- 172.21.15.29:0/3342979676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb28845910 con 0x55bb28943550 2023-10-16T15:55:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f7059517640 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb28945f00 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f704402c220 tx=0x7f704402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2ec45efe0525e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f70377fe640 1 -- 172.21.15.29:0/3342979676 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704402cc10 con 0x55bb28943550 2023-10-16T15:55:47.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f70377fe640 1 -- 172.21.15.29:0/3342979676 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704402cd70 con 0x55bb28943550 2023-10-16T15:55:47.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.165+0000 7f70377fe640 1 -- 172.21.15.29:0/3342979676 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044034640 con 0x55bb28943550 2023-10-16T15:55:47.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 -- 172.21.15.29:0/3342979676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 msgr2=0x55bb28945f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb28945f00 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f704402c220 tx=0x7f704402e000 comp rx=0 tx=0).stop 2023-10-16T15:55:47.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 -- 172.21.15.29:0/3342979676 shutdown_connections 2023-10-16T15:55:47.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb28945f00 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb289165b0 0x55bb28946440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 --2- 172.21.15.29:0/3342979676 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943170 0x55bb28916070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 -- 172.21.15.29:0/3342979676 >> 172.21.15.29:0/3342979676 conn(0x55bb28918380 msgr2=0x55bb28960f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 -- 172.21.15.29:0/3342979676 shutdown_connections 2023-10-16T15:55:47.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.166+0000 7f705ebe6740 1 -- 172.21.15.29:0/3342979676 wait complete. 2023-10-16T15:55:47.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.167+0000 7f705ebe6740 1 -- start start 2023-10-16T15:55:47.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.167+0000 7f705ebe6740 1 Processor -- start 2023-10-16T15:55:47.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.168+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.168+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943550 0x55bb289494b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.168+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 0x55bb289499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.168+0000 7f705ebe6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb28881720 con 0x55bb2893e5f0 2023-10-16T15:55:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.168+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb2881f970 con 0x55bb28943170 2023-10-16T15:55:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.168+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb2881f610 con 0x55bb28943550 2023-10-16T15:55:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705ebe6740 10 client.0 _ll_register_callbacks cb 0x55bb288419d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:55:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 0x55bb289499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 0x55bb289499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54070/0 (socket says 172.21.15.29:54070) 2023-10-16T15:55:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 -- 172.21.15.29:0/328578962 learned_addr learned my addr 172.21.15.29:0/328578962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f7059d18640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f7059517640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943550 0x55bb289494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 -- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943550 msgr2=0x55bb289494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943550 0x55bb289494b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 -- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 msgr2=0x55bb2893c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f705a519640 1 -- 172.21.15.29:0/328578962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7040003520 con 0x55bb2893e5f0 2023-10-16T15:55:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.169+0000 7f7059517640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943550 0x55bb289494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f705a519640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 0x55bb289499f0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f704c058770 tx=0x7f704c0587e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43bbbaa8a4f3470 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f7034ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:55:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f7034ff9640 1 -- 172.21.15.29:0/328578962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c060ae0 con 0x55bb2893e5f0 2023-10-16T15:55:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f7034ff9640 1 -- 172.21.15.29:0/328578962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f704c060650 con 0x55bb2893e5f0 2023-10-16T15:55:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb288a8160 con 0x55bb2893e5f0 2023-10-16T15:55:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f7034ff9640 1 -- 172.21.15.29:0/328578962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) v2 ==== 24+0+0 (secure 0 0 0) 0x7f704c060810 con 0x55bb2893e5f0 2023-10-16T15:55:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f705ebe6740 10 client.12489 fetch_fsmap learned FSMap version 141 2023-10-16T15:55:47.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=2+}) v3 -- 0x55bb2887fda0 con 0x55bb2893e5f0 2023-10-16T15:55:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f7034ff9640 1 -- 172.21.15.29:0/328578962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bb288a8160 con 0x55bb2893e5f0 2023-10-16T15:55:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.170+0000 7f7034ff9640 1 -- 172.21.15.29:0/328578962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 141) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f704c088250 con 0x55bb2893e5f0 2023-10-16T15:55:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f705ebe6740 10 client.12489 fetch_fsmap finished waiting for FSMap version 141 2023-10-16T15:55:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f705ebe6740 10 client.12489 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T15:55:47.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f705ebe6740 20 client.12489 populate_metadata read hostname 'smithi029' 2023-10-16T15:55:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f705ebe6740 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb2892c8c0 0x55bb28944820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f705ebe6740 4 client.12489 mds_command: new command op to 12345 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T15:55:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 --> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb28853cc0 con 0x55bb2892c8c0 2023-10-16T15:55:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f7059d18640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb2892c8c0 0x55bb28944820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f7059d18640 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb2892c8c0 0x55bb28944820 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e1a3dbb412e423c server_cookie=3bba94a1be012df in_seq=0 out_seq=0 2023-10-16T15:55:47.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.171+0000 7f7034ff9640 10 client.12489 ms_handle_connect on v2:172.21.15.29:6833/2425185007 2023-10-16T15:55:47.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.172+0000 7f7034ff9640 1 -- 172.21.15.29:0/328578962 <== mds.0 v2:172.21.15.29:6833/2425185007 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bb28853cc0 con 0x55bb2892c8c0 2023-10-16T15:55:47.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.172+0000 7f7034ff9640 10 client.12489 handle_command_reply: tid=0 2023-10-16T15:55:47.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.172+0000 7f705ebe6740 1 client.12489 shutdown 2023-10-16T15:55:47.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 msgr2=0x55bb289499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 0x55bb289499f0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f704c058770 tx=0x7f704c0587e0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 shutdown_connections 2023-10-16T15:55:47.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb2892c8c0 0x55bb28944820 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb2893e5f0 0x55bb289499f0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb28943550 0x55bb289494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 --2- 172.21.15.29:0/328578962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 >> 172.21.15.29:0/328578962 conn(0x55bb289165b0 msgr2=0x55bb28964410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f7034ff9640 0 client.12489 ms_handle_reset on v2:172.21.15.29:6833/2425185007 2023-10-16T15:55:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 shutdown_connections 2023-10-16T15:55:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 1 -- 172.21.15.29:0/328578962 wait complete. 2023-10-16T15:55:47.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.173+0000 7f705ebe6740 20 client.12489 trim_cache size 0 max 16384 2023-10-16T15:55:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.190+0000 7f705ebe6740 1 Processor -- start 2023-10-16T15:55:47.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.190+0000 7f705ebe6740 1 -- start start 2023-10-16T15:55:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb28964d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb2893df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705ebe6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb285e5f20 con 0x55bb28943550 2023-10-16T15:55:47.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb28929910 con 0x55bb28943170 2023-10-16T15:55:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb289118a0 con 0x55bb2893e5f0 2023-10-16T15:55:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb2893df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb2893df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54082/0 (socket says 172.21.15.29:54082) 2023-10-16T15:55:47.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.191+0000 7f705a519640 1 -- 172.21.15.29:0/4064040562 learned_addr learned my addr 172.21.15.29:0/4064040562 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f7059517640 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f7059d18640 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb28964d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f705a519640 1 -- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 msgr2=0x55bb28964d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f705a519640 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb28964d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f705a519640 1 -- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 msgr2=0x55bb2893da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f705a519640 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f705a519640 1 -- 172.21.15.29:0/4064040562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb2881eb80 con 0x55bb28943550 2023-10-16T15:55:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f7059d18640 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb28964d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f705a519640 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb2893df70 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f704c056b30 tx=0x7f704c05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b2280063fb2ef9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f7037fff640 1 -- 172.21.15.29:0/4064040562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c077170 con 0x55bb28943550 2023-10-16T15:55:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f7037fff640 1 -- 172.21.15.29:0/4064040562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c060ae0 con 0x55bb28943550 2023-10-16T15:55:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.192+0000 7f7037fff640 1 -- 172.21.15.29:0/4064040562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c060650 con 0x55bb28943550 2023-10-16T15:55:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 -- 172.21.15.29:0/4064040562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 msgr2=0x55bb2893df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb2893df70 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f704c056b30 tx=0x7f704c05c000 comp rx=0 tx=0).stop 2023-10-16T15:55:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 -- 172.21.15.29:0/4064040562 shutdown_connections 2023-10-16T15:55:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb2893df70 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb2893da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 --2- 172.21.15.29:0/4064040562 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb28964d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 -- 172.21.15.29:0/4064040562 >> 172.21.15.29:0/4064040562 conn(0x55bb289165b0 msgr2=0x55bb28929190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 -- 172.21.15.29:0/4064040562 shutdown_connections 2023-10-16T15:55:47.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.193+0000 7f705ebe6740 1 -- 172.21.15.29:0/4064040562 wait complete. 2023-10-16T15:55:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.194+0000 7f705ebe6740 1 -- start start 2023-10-16T15:55:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.194+0000 7f705ebe6740 1 Processor -- start 2023-10-16T15:55:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb289452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb289c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb289c9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb289c9c80 con 0x55bb28943550 2023-10-16T15:55:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb28972d50 con 0x55bb28943170 2023-10-16T15:55:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb2893b3b0 con 0x55bb2893e5f0 2023-10-16T15:55:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705ebe6740 10 client.0 _ll_register_callbacks cb 0x7f704c00bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T15:55:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb289c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705a519640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb289c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54094/0 (socket says 172.21.15.29:54094) 2023-10-16T15:55:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.195+0000 7f705a519640 1 -- 172.21.15.29:0/2092629746 learned_addr learned my addr 172.21.15.29:0/2092629746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7059517640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb289c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705a519640 1 -- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 msgr2=0x55bb289452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7059d18640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb289452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705a519640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb289452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705a519640 1 -- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 msgr2=0x55bb289c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705a519640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb289c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705a519640 1 -- 172.21.15.29:0/2092629746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb2887fda0 con 0x55bb28943550 2023-10-16T15:55:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7059d18640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb289452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705a519640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb289c9430 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f704c058770 tx=0x7f704c001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1e9db9499037c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7034ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-16T15:55:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7034ff9640 1 -- 172.21.15.29:0/2092629746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c059d10 con 0x55bb28943550 2023-10-16T15:55:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7034ff9640 1 -- 172.21.15.29:0/2092629746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f704c074b80 con 0x55bb28943550 2023-10-16T15:55:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f704402cd70 con 0x55bb28943550 2023-10-16T15:55:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f7034ff9640 1 -- 172.21.15.29:0/2092629746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) v2 ==== 24+0+0 (secure 0 0 0) 0x7f704c059e70 con 0x55bb28943550 2023-10-16T15:55:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705ebe6740 10 client.12495 fetch_fsmap learned FSMap version 141 2023-10-16T15:55:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.196+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=2+}) v3 -- 0x55bb28844170 con 0x55bb28943550 2023-10-16T15:55:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f7034ff9640 1 -- 172.21.15.29:0/2092629746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704402cd70 con 0x55bb28943550 2023-10-16T15:55:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f7034ff9640 1 -- 172.21.15.29:0/2092629746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 141) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f704c0545d0 con 0x55bb28943550 2023-10-16T15:55:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f705ebe6740 10 client.12495 fetch_fsmap finished waiting for FSMap version 141 2023-10-16T15:55:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f705ebe6740 10 client.12495 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T15:55:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f705ebe6740 20 client.12495 populate_metadata read hostname 'smithi029' 2023-10-16T15:55:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f705ebe6740 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb28949810 0x55bb2893bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f705ebe6740 4 client.12495 mds_command: new command op to 12345 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T15:55:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 --> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb2887a700 con 0x55bb28949810 2023-10-16T15:55:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.197+0000 7f7059d18640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb28949810 0x55bb2893bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f7059d18640 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb28949810 0x55bb2893bce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb61d41cb9d4f4c8 server_cookie=4bdf9f47dd27373d in_seq=0 out_seq=0 2023-10-16T15:55:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f7034ff9640 10 client.12495 ms_handle_connect on v2:172.21.15.29:6833/2425185007 2023-10-16T15:55:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f7034ff9640 1 -- 172.21.15.29:0/2092629746 <== mds.0 v2:172.21.15.29:6833/2425185007 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55bb2887a700 con 0x55bb28949810 2023-10-16T15:55:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f7034ff9640 10 client.12495 handle_command_reply: tid=0 2023-10-16T15:55:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 client.12495 shutdown 2023-10-16T15:55:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 msgr2=0x55bb289c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb289c9430 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f704c058770 tx=0x7f704c001540 comp rx=0 tx=0).stop 2023-10-16T15:55:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 shutdown_connections 2023-10-16T15:55:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:6833/2425185007,v1:172.21.15.29:6835/2425185007] conn(0x55bb28949810 0x55bb2893bce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb28943550 0x55bb289c9430 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55bb28943170 0x55bb289c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 --2- 172.21.15.29:0/2092629746 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x55bb2893e5f0 0x55bb289452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f7034ff9640 0 client.12495 ms_handle_reset on v2:172.21.15.29:6833/2425185007 2023-10-16T15:55:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.198+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 >> 172.21.15.29:0/2092629746 conn(0x55bb289165b0 msgr2=0x55bb28964b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 shutdown_connections 2023-10-16T15:55:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ebe6740 1 -- 172.21.15.29:0/2092629746 wait complete. 2023-10-16T15:55:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ebe6740 20 client.12495 trim_cache size 0 max 16384 2023-10-16T15:55:47.213 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:55:47.213 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "aff38114-d596-4938-a779-5e1b7478d87b", 2023-10-16T15:55:47.214 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-16T15:55:47.214 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 12345, 2023-10-16T15:55:47.214 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-16T15:55:47.214 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-16T15:55:47.214 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-16T15:55:47.215 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8366959420000004, 2023-10-16T15:55:47.215 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 141, 2023-10-16T15:55:47.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 266, 2023-10-16T15:55:47.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 266, 2023-10-16T15:55:47.215 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.60575124 2023-10-16T15:55:47.216 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:55:47.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f703806b0b0 msgr2=0x7f703806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f703806b0b0 0x7f703806d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7050001650 tx=0x7f7050009040 comp rx=0 tx=0).stop 2023-10-16T15:55:47.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 msgr2=0x7f70540c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 0x7f70540c7890 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f705406d2e0 tx=0x7f704c00b0d0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 shutdown_connections 2023-10-16T15:55:47.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f703806b0b0 0x7f703806d570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f705415e6c0 0x7f70540c7890 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f70541fa940 0x7f705415e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 --2- 172.21.15.29:0/3525075694 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f70400047c0 0x7f705415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.199+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 >> 172.21.15.29:0/3525075694 conn(0x7f70540ce6a0 msgr2=0x7f70540afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.200+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 shutdown_connections 2023-10-16T15:55:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.200+0000 7f705ad1a640 1 -- 172.21.15.29:0/3525075694 wait complete. 2023-10-16T15:55:47.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T15:55:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.449+0000 7fd116fd0640 1 Processor -- start 2023-10-16T15:55:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.449+0000 7fd116fd0640 1 -- start start 2023-10-16T15:55:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.450+0000 7fd116fd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd11006c800 0x7fd11015c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.450+0000 7fd116fd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd11015ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd116fd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1100aff30 0x7fd1100cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd116fd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11015d3f0 con 0x7fd11006c800 2023-10-16T15:55:47.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd116fd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1100cd060 con 0x7fd1100aff30 2023-10-16T15:55:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd116fd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100cdbe0 con 0x7fd110160a40 2023-10-16T15:55:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd11006c800 0x7fd11015c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd11006c800 0x7fd11015c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54102/0 (socket says 172.21.15.29:54102) 2023-10-16T15:55:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 -- 172.21.15.29:0/315181646 learned_addr learned my addr 172.21.15.29:0/315181646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd107fff640 1 --2- 172.21.15.29:0/315181646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd11015ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 -- 172.21.15.29:0/315181646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 msgr2=0x7fd11015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd11556d640 1 --2- 172.21.15.29:0/315181646 >> v2:172.21.15.170:3300/0 conn(0x7fd1100aff30 0x7fd1100cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 --2- 172.21.15.29:0/315181646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd11015ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 -- 172.21.15.29:0/315181646 >> v2:172.21.15.170:3300/0 conn(0x7fd1100aff30 msgr2=0x7fd1100cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 --2- 172.21.15.29:0/315181646 >> v2:172.21.15.170:3300/0 conn(0x7fd1100aff30 0x7fd1100cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.451+0000 7fd114d6c640 1 -- 172.21.15.29:0/315181646 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100cd2b0 con 0x7fd11006c800 2023-10-16T15:55:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.453+0000 7fd114d6c640 1 --2- 172.21.15.29:0/315181646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd11015c970 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd1100cfbd0 tx=0x7fd10c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd90a87272229fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.453+0000 7fd1077fe640 1 -- 172.21.15.29:0/315181646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c02ca70 con 0x7fd11006c800 2023-10-16T15:55:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.453+0000 7fd1077fe640 1 -- 172.21.15.29:0/315181646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c02cbd0 con 0x7fd11006c800 2023-10-16T15:55:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.453+0000 7fd1077fe640 1 -- 172.21.15.29:0/315181646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c0343f0 con 0x7fd11006c800 2023-10-16T15:55:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 -- 172.21.15.29:0/315181646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 msgr2=0x7fd11015c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 --2- 172.21.15.29:0/315181646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd11015c970 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd1100cfbd0 tx=0x7fd10c02bed0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 -- 172.21.15.29:0/315181646 shutdown_connections 2023-10-16T15:55:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 --2- 172.21.15.29:0/315181646 >> v2:172.21.15.170:3300/0 conn(0x7fd1100aff30 0x7fd1100cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 --2- 172.21.15.29:0/315181646 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd11015ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 --2- 172.21.15.29:0/315181646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd11015c970 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 -- 172.21.15.29:0/315181646 >> 172.21.15.29:0/315181646 conn(0x7fd11006d700 msgr2=0x7fd11006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 -- 172.21.15.29:0/315181646 shutdown_connections 2023-10-16T15:55:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.454+0000 7fd116fd0640 1 -- 172.21.15.29:0/315181646 wait complete. 2023-10-16T15:55:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.456+0000 7fd116fd0640 1 Processor -- start 2023-10-16T15:55:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.456+0000 7fd116fd0640 1 -- start start 2023-10-16T15:55:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd116fd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd110156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd116fd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1100aff30 0x7fd1100c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd116fd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd1100c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd116fd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b32e0 con 0x7fd11006c800 2023-10-16T15:55:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd116fd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100c5890 con 0x7fd1100aff30 2023-10-16T15:55:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd116fd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b2a40 con 0x7fd110160a40 2023-10-16T15:55:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd114d6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd110156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd114d6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd110156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54116/0 (socket says 172.21.15.29:54116) 2023-10-16T15:55:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.457+0000 7fd114d6c640 1 -- 172.21.15.29:0/1330876551 learned_addr learned my addr 172.21.15.29:0/1330876551 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd114d6c640 1 -- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 msgr2=0x7fd1100c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd11556d640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd1100c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd107fff640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1100aff30 0x7fd1100c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd114d6c640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd1100c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd114d6c640 1 -- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1100aff30 msgr2=0x7fd1100c45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd114d6c640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1100aff30 0x7fd1100c45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd114d6c640 1 -- 172.21.15.29:0/1330876551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c009b30 con 0x7fd11006c800 2023-10-16T15:55:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd11556d640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd1100c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd107fff640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1100aff30 0x7fd1100c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd114d6c640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd110156150 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fd10c000f80 tx=0x7fd10c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c03e450 con 0x7fd11006c800 2023-10-16T15:55:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c03e5f0 con 0x7fd11006c800 2023-10-16T15:55:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c03e9a0 con 0x7fd11006c800 2023-10-16T15:55:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.458+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1100b2d20 con 0x7fd11006c800 2023-10-16T15:55:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.459+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1100c8c90 con 0x7fd11006c800 2023-10-16T15:55:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.460+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11006f610 con 0x7fd11006c800 2023-10-16T15:55:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.461+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd10c03eb00 con 0x7fd11006c800 2023-10-16T15:55:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.462+0000 7fd1057fa640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd0fc06b0e0 0x7fd0fc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.462+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7fd10c0ac940 con 0x7fd11006c800 2023-10-16T15:55:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.462+0000 7fd107fff640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd0fc06b0e0 0x7fd0fc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.464+0000 7fd107fff640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd0fc06b0e0 0x7fd0fc06d5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd1100b6a20 tx=0x7fd100008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:47.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.466+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd10c0776f0 con 0x7fd11006c800 2023-10-16T15:55:47.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.702+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd11004e160 con 0x7fd11006c800 2023-10-16T15:55:47.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.703+0000 7fd1057fa640 1 -- 172.21.15.29:0/1330876551 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v141) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd10c03edb0 con 0x7fd11006c800 2023-10-16T15:55:47.705 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T15:55:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd0fc06b0e0 msgr2=0x7fd0fc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd0fc06b0e0 0x7fd0fc06d5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd1100b6a20 tx=0x7fd100008040 comp rx=0 tx=0).stop 2023-10-16T15:55:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 msgr2=0x7fd110156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:47.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd110156150 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fd10c000f80 tx=0x7fd10c004a40 comp rx=0 tx=0).stop 2023-10-16T15:55:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 shutdown_connections 2023-10-16T15:55:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7fd0fc06b0e0 0x7fd0fc06d5a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd110160a40 0x7fd1100c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1100aff30 0x7fd1100c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 --2- 172.21.15.29:0/1330876551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11006c800 0x7fd110156150 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 >> 172.21.15.29:0/1330876551 conn(0x7fd11006d700 msgr2=0x7fd11006dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 shutdown_connections 2023-10-16T15:55:47.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:47.706+0000 7fd116fd0640 1 -- 172.21.15.29:0/1330876551 wait complete. 2023-10-16T15:55:47.728 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T15:55:47.728 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T15:55:47.728 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-16T15:55:47.729 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T15:55:47.729 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T15:55:47.729 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T15:55:47.730 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T15:55:47.730 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T15:55:47.730 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:55:47.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:47.731 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-16T15:55:47.790 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T15:55:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T15:55:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T15:55:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-16T15:55:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.792 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:55:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:55:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-16T15:55:47.792 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:55:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:55:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-16T15:55:47.793 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T15:55:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-16T15:55:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-16T15:55:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T15:55:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36377sec preferred_lft 36377sec 2023-10-16T15:55:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-16T15:55:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.794 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T15:55:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-16T15:55:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-16T15:55:47.795 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T15:55:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff 2023-10-16T15:55:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T15:55:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::5c5c:9fff:fe8c:f8f7/64 scope link 2023-10-16T15:55:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.796 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:55:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 56:2f:58:6c:38:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:55:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::542f:58ff:fe6c:38f3/64 scope link 2023-10-16T15:55:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.796 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T15:55:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 9a:23:75:24:f2:ca brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:55:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::9823:75ff:fe24:f2ca/64 scope link 2023-10-16T15:55:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-16T15:55:47.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:47.798 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T15:55:47.897 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-16T15:55:47.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:47.902 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-16T15:55:47.963 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-16T15:55:47.963 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T15:55:47.963 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T15:55:47.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T15:55:47.964 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T15:55:47.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:47.965 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-16T15:55:47.965 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-16T15:55:47.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T15:55:47.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T15:55:47.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T15:55:47.967 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-16T15:55:48.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:48.172 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:55:48.226 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:55:48.226 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:55:48.227 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:55:48.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:48.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-16T15:55:48.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:55:48.332 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T15:55:48.397 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T15:55:48.397 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T15:55:48.397 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:55:48.452 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T15:55:48.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:48.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:55:48.601 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:55:48.601 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:55:48.601 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:55:48.623 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:55:48.623 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:55:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:55:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:55:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:55:48.650 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T15:55:48.651 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T15:55:48.651 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-16T15:55:48.651 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.170:3300,172.21.15.170:3301" 2023-10-16T15:55:48.651 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T15:55:48.651 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790" 2023-10-16T15:55:48.651 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.170:6789,172.21.15.170:6790:/ 2023-10-16T15:55:48.652 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=aff38114-d596-4938-a779-5e1b7478d87b" will pass to kernel 2023-10-16T15:55:48.652 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T15:55:48.652 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-16T15:55:48.653 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T15:55:48.653 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T15:55:48.653 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T15:55:48.656 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T15:55:48.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:48.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T15:55:48.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:55:48.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:55:48.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:48.750 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T15:55:48.806 INFO:teuthology.orchestra.run.smithi029.stdout:client9787 2023-10-16T15:55:48.807 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:48.807 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:48.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T15:55:49.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.124+0000 7f6154fae640 1 Processor -- start 2023-10-16T15:55:49.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.124+0000 7f6154fae640 1 -- start start 2023-10-16T15:55:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f6154fae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f6154fae640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61500cdd60 0x7f6150071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f6154fae640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61500ce1c0 con 0x7f6150071fa0 2023-10-16T15:55:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f6154fae640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61500d0500 con 0x7f61500cdd60 2023-10-16T15:55:49.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f6154fae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61500d12f0 con 0x7f615015cc80 2023-10-16T15:55:49.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f614ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6150071fa0 0x7f61501607e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60676/0 (socket says 172.21.15.29:60676) 2023-10-16T15:55:49.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.125+0000 7f614ffff640 1 -- 172.21.15.29:0/2039489148 learned_addr learned my addr 172.21.15.29:0/2039489148 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614effd640 1 --2- 172.21.15.29:0/2039489148 >> v2:172.21.15.29:3300/0 conn(0x7f61500cdd60 0x7f6150071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614f7fe640 1 --2- 172.21.15.29:0/2039489148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614e7fc640 1 -- 172.21.15.29:0/2039489148 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069296948 0 0) 0x7f61500ce1c0 con 0x7f6150071fa0 2023-10-16T15:55:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614effd640 1 -- 172.21.15.29:0/2039489148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 msgr2=0x7f61500cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:49.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614effd640 1 --2- 172.21.15.29:0/2039489148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614effd640 1 -- 172.21.15.29:0/2039489148 >> v1:172.21.15.29:6789/0 conn(0x7f6150071fa0 legacy=0x7f61501607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.126+0000 7f614effd640 1 -- 172.21.15.29:0/2039489148 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6150072410 con 0x7f61500cdd60 2023-10-16T15:55:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.128+0000 7f614effd640 1 --2- 172.21.15.29:0/2039489148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61500cdd60 0x7f6150071a60 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f61380097a0 tx=0x7f613802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e6df78fb18cf33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.128+0000 7f614e7fc640 1 -- 172.21.15.29:0/2039489148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138031830 con 0x7f61500cdd60 2023-10-16T15:55:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.128+0000 7f614e7fc640 1 -- 172.21.15.29:0/2039489148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138031990 con 0x7f61500cdd60 2023-10-16T15:55:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.128+0000 7f614e7fc640 1 -- 172.21.15.29:0/2039489148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138031d20 con 0x7f61500cdd60 2023-10-16T15:55:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.128+0000 7f6154fae640 1 -- 172.21.15.29:0/2039489148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61500cdd60 msgr2=0x7f6150071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.128+0000 7f6154fae640 1 --2- 172.21.15.29:0/2039489148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61500cdd60 0x7f6150071a60 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f61380097a0 tx=0x7f613802b8d0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.129+0000 7f6154fae640 1 -- 172.21.15.29:0/2039489148 shutdown_connections 2023-10-16T15:55:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.129+0000 7f6154fae640 1 --2- 172.21.15.29:0/2039489148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61500cdd60 0x7f6150071a60 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.129+0000 7f6154fae640 1 --2- 172.21.15.29:0/2039489148 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.129+0000 7f6154fae640 1 -- 172.21.15.29:0/2039489148 >> 172.21.15.29:0/2039489148 conn(0x7f6150065ed0 msgr2=0x7f61500662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.129+0000 7f6154fae640 1 -- 172.21.15.29:0/2039489148 shutdown_connections 2023-10-16T15:55:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.129+0000 7f6154fae640 1 -- 172.21.15.29:0/2039489148 wait complete. 2023-10-16T15:55:49.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.130+0000 7f6154fae640 1 Processor -- start 2023-10-16T15:55:49.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.131+0000 7f6154fae640 1 -- start start 2023-10-16T15:55:49.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.131+0000 7f6154fae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 0x7f6150158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f6154fae640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61500cdd60 0x7f6150157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f6154fae640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f6154fae640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500b3500 con 0x7f6150071fa0 2023-10-16T15:55:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f6154fae640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500b37c0 con 0x7f61500cdd60 2023-10-16T15:55:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f6154fae640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61500b3aa0 con 0x7f615015cc80 2023-10-16T15:55:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 0x7f6150158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 0x7f6150158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54120/0 (socket says 172.21.15.29:54120) 2023-10-16T15:55:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 -- 172.21.15.29:0/2597036367 learned_addr learned my addr 172.21.15.29:0/2597036367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614effd640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61500cdd60 0x7f6150157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614ffff640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 -- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 msgr2=0x7f61500b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 -- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61500cdd60 msgr2=0x7f6150157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61500cdd60 0x7f6150157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.132+0000 7f614f7fe640 1 -- 172.21.15.29:0/2597036367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138009400 con 0x7f6150071fa0 2023-10-16T15:55:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.133+0000 7f614f7fe640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 0x7f6150158160 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f6140007e40 tx=0x7f614000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.133+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140014620 con 0x7f6150071fa0 2023-10-16T15:55:49.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.133+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614000ee90 con 0x7f6150071fa0 2023-10-16T15:55:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.133+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61500b3d80 con 0x7f6150071fa0 2023-10-16T15:55:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.133+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61400149c0 con 0x7f6150071fa0 2023-10-16T15:55:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.133+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61500b4340 con 0x7f6150071fa0 2023-10-16T15:55:49.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.134+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615004e9b0 con 0x7f6150071fa0 2023-10-16T15:55:49.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.135+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6140014b20 con 0x7f6150071fa0 2023-10-16T15:55:49.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.136+0000 7f6147fff640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f613006b0b0 0x7f613006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.136+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f614008a3a0 con 0x7f6150071fa0 2023-10-16T15:55:49.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.137+0000 7f614effd640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f613006b0b0 0x7f613006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.138+0000 7f614effd640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f613006b0b0 0x7f613006d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f6138008030 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:49.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.140+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f61400550d0 con 0x7f6150071fa0 2023-10-16T15:55:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.357+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 --> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f61500d09c0 con 0x7f613006b0b0 2023-10-16T15:55:49.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.362+0000 7f6147fff640 1 -- 172.21.15.29:0/2597036367 <== mgr.9472 v2:172.21.15.170:6832/3469082932 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6138031b00 con 0x7f613006b0b0 2023-10-16T15:55:49.363 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-16T15:55:49.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-16T15:55:49.364 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-16T15:55:49.364 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-16T15:55:49.364 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-16T15:55:49.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.363+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f613006b0b0 msgr2=0x7f613006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.363+0000 7f6154fae640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f613006b0b0 0x7f613006d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f6138008030 comp rx=0 tx=0).stop 2023-10-16T15:55:49.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 msgr2=0x7f6150158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 0x7f6150158160 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f6140007e40 tx=0x7f614000e4b0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 shutdown_connections 2023-10-16T15:55:49.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f613006b0b0 0x7f613006d570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f615015cc80 0x7f61500b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61500cdd60 0x7f6150157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 --2- 172.21.15.29:0/2597036367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6150071fa0 0x7f6150158160 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 >> 172.21.15.29:0/2597036367 conn(0x7f6150065ed0 msgr2=0x7f61500662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 shutdown_connections 2023-10-16T15:55:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.364+0000 7f6154fae640 1 -- 172.21.15.29:0/2597036367 wait complete. 2023-10-16T15:55:49.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T15:55:49.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.617+0000 7f987d773640 1 Processor -- start 2023-10-16T15:55:49.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.618+0000 7f987d773640 1 -- start start 2023-10-16T15:55:49.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.619+0000 7f987d773640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f98781573e0 0x7f98780ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.619+0000 7f987d773640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9878159e70 0x7f987815a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f987d773640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98780ca3a0 con 0x7f9878070240 2023-10-16T15:55:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f987d773640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98780cad70 con 0x7f9878159e70 2023-10-16T15:55:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f987d773640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98780c9d70 con 0x7f98781573e0 2023-10-16T15:55:49.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f98777fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9878159e70 0x7f987815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f98777fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9878159e70 0x7f987815a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54132/0 (socket says 172.21.15.29:54132) 2023-10-16T15:55:49.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f98777fe640 1 -- 172.21.15.29:0/4102900293 learned_addr learned my addr 172.21.15.29:0/4102900293 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:49.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f987cf72640 1 --1- 172.21.15.29:0/4102900293 >> v1:172.21.15.29:6789/0 conn(0x7f9878070240 0x7f987806bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60690/0 (socket says 172.21.15.29:60690) 2023-10-16T15:55:49.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.620+0000 7f9877fff640 1 --2- 172.21.15.29:0/4102900293 >> v2:172.21.15.170:3300/0 conn(0x7f98781573e0 0x7f98780ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.621+0000 7f98777fe640 1 -- 172.21.15.29:0/4102900293 >> v2:172.21.15.170:3300/0 conn(0x7f98781573e0 msgr2=0x7f98780ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:49.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.621+0000 7f98777fe640 1 --2- 172.21.15.29:0/4102900293 >> v2:172.21.15.170:3300/0 conn(0x7f98781573e0 0x7f98780ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.621+0000 7f98777fe640 1 -- 172.21.15.29:0/4102900293 >> v1:172.21.15.29:6789/0 conn(0x7f9878070240 legacy=0x7f987806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.621+0000 7f98777fe640 1 -- 172.21.15.29:0/4102900293 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98781577c0 con 0x7f9878159e70 2023-10-16T15:55:49.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.622+0000 7f98777fe640 1 --2- 172.21.15.29:0/4102900293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878159e70 0x7f987815a320 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f98780cd0a0 tx=0x7f986c02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80be36bc8d009797 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:49.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.622+0000 7f9876ffd640 1 -- 172.21.15.29:0/4102900293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c036440 con 0x7f9878159e70 2023-10-16T15:55:49.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.622+0000 7f9876ffd640 1 -- 172.21.15.29:0/4102900293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c0365a0 con 0x7f9878159e70 2023-10-16T15:55:49.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.622+0000 7f9876ffd640 1 -- 172.21.15.29:0/4102900293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c036930 con 0x7f9878159e70 2023-10-16T15:55:49.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 -- 172.21.15.29:0/4102900293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878159e70 msgr2=0x7f987815a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 --2- 172.21.15.29:0/4102900293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878159e70 0x7f987815a320 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f98780cd0a0 tx=0x7f986c02e180 comp rx=0 tx=0).stop 2023-10-16T15:55:49.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 -- 172.21.15.29:0/4102900293 shutdown_connections 2023-10-16T15:55:49.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 --2- 172.21.15.29:0/4102900293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878159e70 0x7f987815a320 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 --2- 172.21.15.29:0/4102900293 >> v2:172.21.15.170:3300/0 conn(0x7f98781573e0 0x7f98780ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 -- 172.21.15.29:0/4102900293 >> 172.21.15.29:0/4102900293 conn(0x7f98780cc230 msgr2=0x7f98780cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:49.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 -- 172.21.15.29:0/4102900293 shutdown_connections 2023-10-16T15:55:49.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.623+0000 7f987d773640 1 -- 172.21.15.29:0/4102900293 wait complete. 2023-10-16T15:55:49.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.625+0000 7f987d773640 1 Processor -- start 2023-10-16T15:55:49.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.625+0000 7f987d773640 1 -- start start 2023-10-16T15:55:49.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.626+0000 7f987d773640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 0x7f9878156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.626+0000 7f987d773640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98781573e0 0x7f98780ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.626+0000 7f987d773640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98780adc30 0x7f98780c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.626+0000 7f987d773640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98780ae0e0 con 0x7f9878070240 2023-10-16T15:55:49.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.626+0000 7f987d773640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9878156b80 con 0x7f98781573e0 2023-10-16T15:55:49.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.626+0000 7f987d773640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98780c33f0 con 0x7f98780adc30 2023-10-16T15:55:49.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 0x7f9878156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 0x7f9878156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54146/0 (socket says 172.21.15.29:54146) 2023-10-16T15:55:49.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 -- 172.21.15.29:0/2895067187 learned_addr learned my addr 172.21.15.29:0/2895067187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:49.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f987cf72640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98780adc30 0x7f98780c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f98777fe640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98781573e0 0x7f98780ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 -- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98780adc30 msgr2=0x7f98780c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98780adc30 0x7f98780c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 -- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98781573e0 msgr2=0x7f98780ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98781573e0 0x7f98780ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.627+0000 7f9877fff640 1 -- 172.21.15.29:0/2895067187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c00b770 con 0x7f9878070240 2023-10-16T15:55:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.628+0000 7f9877fff640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 0x7f9878156640 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f986800e9b0 tx=0x7f986800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.628+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868009670 con 0x7f9878070240 2023-10-16T15:55:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.628+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986800c9a0 con 0x7f9878070240 2023-10-16T15:55:49.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.628+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986800cd50 con 0x7f9878070240 2023-10-16T15:55:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.628+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98780c36d0 con 0x7f9878070240 2023-10-16T15:55:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.628+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98780c3c90 con 0x7f9878070240 2023-10-16T15:55:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.629+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987804e9b0 con 0x7f9878070240 2023-10-16T15:55:49.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.630+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9868013470 con 0x7f9878070240 2023-10-16T15:55:49.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.631+0000 7f9874ff9640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f986406b0b0 0x7f986406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:49.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.631+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f9868089d50 con 0x7f9878070240 2023-10-16T15:55:49.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.632+0000 7f98777fe640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f986406b0b0 0x7f986406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:49.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.633+0000 7f98777fe640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f986406b0b0 0x7f986406d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f986c038000 tx=0x7f986c008040 comp rx=0 tx=0).ready entity=mgr.9472 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:49.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.636+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9868054a40 con 0x7f9878070240 2023-10-16T15:55:49.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:49.882+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f98780b5730 con 0x7f9878070240 2023-10-16T15:55:50.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.464+0000 7f9874ff9640 1 -- 172.21.15.29:0/2895067187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v147) v1 ==== 92+0+0 (secure 0 0 0) 0x7f98680170a0 con 0x7f9878070240 2023-10-16T15:55:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.467+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f986406b0b0 msgr2=0x7f986406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.467+0000 7f987d773640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f986406b0b0 0x7f986406d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f986c038000 tx=0x7f986c008040 comp rx=0 tx=0).stop 2023-10-16T15:55:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.467+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 msgr2=0x7f9878156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.467+0000 7f987d773640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 0x7f9878156640 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f986800e9b0 tx=0x7f986800ee70 comp rx=0 tx=0).stop 2023-10-16T15:55:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.468+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 shutdown_connections 2023-10-16T15:55:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.468+0000 7f987d773640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f986406b0b0 0x7f986406d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.468+0000 7f987d773640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f98780adc30 0x7f98780c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.468+0000 7f987d773640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98781573e0 0x7f98780ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.468+0000 7f987d773640 1 --2- 172.21.15.29:0/2895067187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9878070240 0x7f9878156640 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.468+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 >> 172.21.15.29:0/2895067187 conn(0x7f98780cc230 msgr2=0x7f9878158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.469+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 shutdown_connections 2023-10-16T15:55:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.469+0000 7f987d773640 1 -- 172.21.15.29:0/2895067187 wait complete. 2023-10-16T15:55:50.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T15:55:50.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.723+0000 7f6a20dd0640 1 Processor -- start 2023-10-16T15:55:50.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.723+0000 7f6a20dd0640 1 -- start start 2023-10-16T15:55:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.724+0000 7f6a20dd0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6a1c15ad70 0x7f6a1c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.724+0000 7f6a20dd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a1c0caac0 0x7f6a1c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:50.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.724+0000 7f6a20dd0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1c0cb030 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.724+0000 7f6a20dd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c0c91c0 con 0x7f6a1c0caac0 2023-10-16T15:55:50.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.724+0000 7f6a20dd0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c0c94a0 con 0x7f6a1c15ad70 2023-10-16T15:55:50.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.725+0000 7f6a1ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a1c0caac0 0x7f6a1c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:50.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.725+0000 7f6a1ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a1c0caac0 0x7f6a1c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54180/0 (socket says 172.21.15.29:54180) 2023-10-16T15:55:50.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.725+0000 7f6a1ad9d640 1 -- 172.21.15.29:0/1774627291 learned_addr learned my addr 172.21.15.29:0/1774627291 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:50.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.725+0000 7f6a1a59c640 1 --2- 172.21.15.29:0/1774627291 >> v2:172.21.15.170:3300/0 conn(0x7f6a1c15ad70 0x7f6a1c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:50.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.725+0000 7f6a1ad9d640 1 -- 172.21.15.29:0/1774627291 >> v2:172.21.15.170:3300/0 conn(0x7f6a1c15ad70 msgr2=0x7f6a1c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:50.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.725+0000 7f6a1ad9d640 1 --2- 172.21.15.29:0/1774627291 >> v2:172.21.15.170:3300/0 conn(0x7f6a1c15ad70 0x7f6a1c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.726+0000 7f6a1ad9d640 1 -- 172.21.15.29:0/1774627291 >> v1:172.21.15.170:6789/0 conn(0x7f6a1c0c9ca0 legacy=0x7f6a1c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.726+0000 7f6a1ad9d640 1 -- 172.21.15.29:0/1774627291 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c0c9780 con 0x7f6a1c0caac0 2023-10-16T15:55:50.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.727+0000 7f6a1ad9d640 1 --2- 172.21.15.29:0/1774627291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0caac0 0x7f6a1c06a2e0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0cf780 tx=0x7f6a1402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4a87d1db0e880f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:50.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a19d9b640 1 -- 172.21.15.29:0/1774627291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1402cb20 con 0x7f6a1c0caac0 2023-10-16T15:55:50.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a19d9b640 1 -- 172.21.15.29:0/1774627291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1402cc80 con 0x7f6a1c0caac0 2023-10-16T15:55:50.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a19d9b640 1 -- 172.21.15.29:0/1774627291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a140343f0 con 0x7f6a1c0caac0 2023-10-16T15:55:50.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1774627291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0caac0 msgr2=0x7f6a1c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1774627291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0caac0 0x7f6a1c06a2e0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0cf780 tx=0x7f6a1402bf80 comp rx=0 tx=0).stop 2023-10-16T15:55:50.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1774627291 shutdown_connections 2023-10-16T15:55:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1774627291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0caac0 0x7f6a1c06a2e0 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1774627291 >> v2:172.21.15.170:3300/0 conn(0x7f6a1c15ad70 0x7f6a1c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.728+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1774627291 >> 172.21.15.29:0/1774627291 conn(0x7f6a1c0ce9b0 msgr2=0x7f6a1c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.729+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1774627291 shutdown_connections 2023-10-16T15:55:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.729+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1774627291 wait complete. 2023-10-16T15:55:50.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.730+0000 7f6a20dd0640 1 Processor -- start 2023-10-16T15:55:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.731+0000 7f6a20dd0640 1 -- start start 2023-10-16T15:55:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.731+0000 7f6a20dd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 0x7f6a1c14deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a20dd0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a1c0caac0 0x7f6a1c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a20dd0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a1c15ad70 0x7f6a1c0b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a20dd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c0c69e0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a20dd0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c15e0e0 con 0x7f6a1c0caac0 2023-10-16T15:55:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a20dd0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c0c5a80 con 0x7f6a1c15ad70 2023-10-16T15:55:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 0x7f6a1c14deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 0x7f6a1c14deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54186/0 (socket says 172.21.15.29:54186) 2023-10-16T15:55:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 -- 172.21.15.29:0/1557651545 learned_addr learned my addr 172.21.15.29:0/1557651545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1ad9d640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a1c15ad70 0x7f6a1c0b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 -- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a1c15ad70 msgr2=0x7f6a1c0b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a1c15ad70 0x7f6a1c0b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 -- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a1c0caac0 msgr2=0x7f6a1c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a1c0caac0 0x7f6a1c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.732+0000 7f6a1a59c640 1 -- 172.21.15.29:0/1557651545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14009b30 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.733+0000 7f6a1a59c640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 0x7f6a1c14deb0 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00d5f0 tx=0x7f6a0c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.733+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0125f0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.733+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c012750 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.733+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1c0c5d60 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.733+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c012ae0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.733+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1c0c6320 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.734+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1c0ad4b0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.735+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f6a0c012c40 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.736+0000 7f6a137fe640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 0x7f6a0006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.736+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f6a0c089fe0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.737+0000 7f6a11d9b640 1 -- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 msgr2=0x7f6a0006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.737+0000 7f6a11d9b640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 0x7f6a0006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.745+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a1c0ad4b0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.938+0000 7f6a11d9b640 1 -- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 msgr2=0x7f6a0006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.938+0000 7f6a11d9b640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 0x7f6a0006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.939+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6a1c0ad4b0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.946+0000 7f6a137fe640 1 -- 172.21.15.29:0/1557651545 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6a1c0ad4b0 con 0x7f6a1c0c9ca0 2023-10-16T15:55:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 msgr2=0x7f6a0006d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 0x7f6a0006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 msgr2=0x7f6a1c14deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:50.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 0x7f6a1c14deb0 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00d5f0 tx=0x7f6a0c00dab0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 shutdown_connections 2023-10-16T15:55:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f6a0006b100 0x7f6a0006d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6a1c15ad70 0x7f6a1c0b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a1c0caac0 0x7f6a1c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 --2- 172.21.15.29:0/1557651545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a1c0c9ca0 0x7f6a1c14deb0 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 >> 172.21.15.29:0/1557651545 conn(0x7f6a1c0ce9b0 msgr2=0x7f6a1c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 shutdown_connections 2023-10-16T15:55:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:50.949+0000 7f6a20dd0640 1 -- 172.21.15.29:0/1557651545 wait complete. 2023-10-16T15:55:50.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T15:55:51.002 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.000+0000 7fb59124f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:51.055 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.053+0000 7fb59124f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:51.059 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.057+0000 7f44c1214040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:51.120 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.118+0000 7f44c1214040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.192+0000 7f3056409640 1 Processor -- start 2023-10-16T15:55:51.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.193+0000 7f3056409640 1 -- start start 2023-10-16T15:55:51.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.194+0000 7f3056409640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f305015ad70 0x7f3050069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.194+0000 7f3056409640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30500c9ca0 0x7f305006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.194+0000 7f3056409640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500cb030 con 0x7f30500ca3b0 2023-10-16T15:55:51.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.194+0000 7f3056409640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30500c91c0 con 0x7f30500c9ca0 2023-10-16T15:55:51.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.194+0000 7f3056409640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30500c94a0 con 0x7f305015ad70 2023-10-16T15:55:51.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30500c9ca0 0x7f305006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30500c9ca0 0x7f305006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54198/0 (socket says 172.21.15.29:54198) 2023-10-16T15:55:51.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 -- 172.21.15.29:0/1685177276 learned_addr learned my addr 172.21.15.29:0/1685177276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:51.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f30549a6640 1 --1- 172.21.15.29:0/1685177276 >> v1:172.21.15.29:6789/0 conn(0x7f30500ca3b0 0x7f30500ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60700/0 (socket says 172.21.15.29:60700) 2023-10-16T15:55:51.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f304ffff640 1 --2- 172.21.15.29:0/1685177276 >> v2:172.21.15.170:3300/0 conn(0x7f305015ad70 0x7f3050069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 -- 172.21.15.29:0/1685177276 >> v2:172.21.15.170:3300/0 conn(0x7f305015ad70 msgr2=0x7f3050069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:51.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 --2- 172.21.15.29:0/1685177276 >> v2:172.21.15.170:3300/0 conn(0x7f305015ad70 0x7f3050069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 -- 172.21.15.29:0/1685177276 >> v1:172.21.15.29:6789/0 conn(0x7f30500ca3b0 legacy=0x7f30500ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.195+0000 7f3047fff640 1 -- 172.21.15.29:0/1685177276 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500c9780 con 0x7f30500c9ca0 2023-10-16T15:55:51.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.196+0000 7f3047fff640 1 --2- 172.21.15.29:0/1685177276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305006a2e0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f30500cf780 tx=0x7f303802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510587c440142602 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:51.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.197+0000 7f304f7fe640 1 -- 172.21.15.29:0/1685177276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038030860 con 0x7f30500c9ca0 2023-10-16T15:55:51.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.197+0000 7f304f7fe640 1 -- 172.21.15.29:0/1685177276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30380309c0 con 0x7f30500c9ca0 2023-10-16T15:55:51.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.197+0000 7f304f7fe640 1 -- 172.21.15.29:0/1685177276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038030d50 con 0x7f30500c9ca0 2023-10-16T15:55:51.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 -- 172.21.15.29:0/1685177276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 msgr2=0x7f305006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 --2- 172.21.15.29:0/1685177276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305006a2e0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f30500cf780 tx=0x7f303802b9a0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 -- 172.21.15.29:0/1685177276 shutdown_connections 2023-10-16T15:55:51.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 --2- 172.21.15.29:0/1685177276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305006a2e0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 --2- 172.21.15.29:0/1685177276 >> v2:172.21.15.170:3300/0 conn(0x7f305015ad70 0x7f3050069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 -- 172.21.15.29:0/1685177276 >> 172.21.15.29:0/1685177276 conn(0x7f30500ce9b0 msgr2=0x7f30500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:51.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 -- 172.21.15.29:0/1685177276 shutdown_connections 2023-10-16T15:55:51.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.198+0000 7f3056409640 1 -- 172.21.15.29:0/1685177276 wait complete. 2023-10-16T15:55:51.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.200+0000 7f3056409640 1 Processor -- start 2023-10-16T15:55:51.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.200+0000 7f3056409640 1 -- start start 2023-10-16T15:55:51.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3056409640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3056409640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30500ca3b0 0x7f305015c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3056409640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f305015ad70 0x7f305015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3056409640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305015b7c0 con 0x7f30500c9ca0 2023-10-16T15:55:51.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3056409640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305015cf50 con 0x7f305015ad70 2023-10-16T15:55:51.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3056409640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30500c5b90 con 0x7f30500ca3b0 2023-10-16T15:55:51.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54204/0 (socket says 172.21.15.29:54204) 2023-10-16T15:55:51.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 -- 172.21.15.29:0/164320717 learned_addr learned my addr 172.21.15.29:0/164320717 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f3047fff640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30500ca3b0 0x7f305015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 -- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30500ca3b0 msgr2=0x7f305015c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30500ca3b0 0x7f305015c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 -- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f305015ad70 msgr2=0x7f305015c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f30549a6640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f305015ad70 0x7f305015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f305015ad70 0x7f305015c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.201+0000 7f304ffff640 1 -- 172.21.15.29:0/164320717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30380094d0 con 0x7f30500c9ca0 2023-10-16T15:55:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.202+0000 7f304ffff640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305015d5f0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f304000e870 tx=0x7f304000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.202+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400094f0 con 0x7f30500c9ca0 2023-10-16T15:55:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.202+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304000c940 con 0x7f30500c9ca0 2023-10-16T15:55:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.202+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304000ccf0 con 0x7f30500c9ca0 2023-10-16T15:55:51.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.202+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30500c5e70 con 0x7f30500c9ca0 2023-10-16T15:55:51.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.202+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30500c6430 con 0x7f30500c9ca0 2023-10-16T15:55:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.204+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30500c65c0 con 0x7f30500c9ca0 2023-10-16T15:55:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.205+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f3040013470 con 0x7f30500c9ca0 2023-10-16T15:55:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.205+0000 7f304d7fa640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 0x7f303006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.206+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f3040089ba0 con 0x7f30500c9ca0 2023-10-16T15:55:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.206+0000 7f3047fff640 1 -- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 msgr2=0x7f303006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.206+0000 7f3047fff640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 0x7f303006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:51.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.210+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30400548a0 con 0x7f30500c9ca0 2023-10-16T15:55:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.402+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f30500cb220 con 0x7f30500c9ca0 2023-10-16T15:55:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.407+0000 7f304d7fa640 1 -- 172.21.15.29:0/164320717 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f30400170c0 con 0x7f30500c9ca0 2023-10-16T15:55:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.407+0000 7f3047fff640 1 -- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 msgr2=0x7f303006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.407+0000 7f3047fff640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 0x7f303006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 msgr2=0x7f303006d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 0x7f303006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 msgr2=0x7f305015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305015d5f0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f304000e870 tx=0x7f304000ed30 comp rx=0 tx=0).stop 2023-10-16T15:55:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 shutdown_connections 2023-10-16T15:55:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f303006b130 0x7f303006d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f305015ad70 0x7f305015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f30500ca3b0 0x7f305015c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 --2- 172.21.15.29:0/164320717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30500c9ca0 0x7f305015d5f0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 >> 172.21.15.29:0/164320717 conn(0x7f30500ce9b0 msgr2=0x7f30500acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 shutdown_connections 2023-10-16T15:55:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.410+0000 7f3056409640 1 -- 172.21.15.29:0/164320717 wait complete. 2023-10-16T15:55:51.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:51.429 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T15:55:51.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T15:55:51.503 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.502+0000 7fb59124f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:51.556 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.554+0000 7fb59124f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:51.606 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.604+0000 7fb59124f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:51.609 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.607+0000 7f44c1214040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:51.664 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.662+0000 7f44c1214040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:51.667 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.665+0000 7fb59124f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.705+0000 7f2b944e1640 1 Processor -- start 2023-10-16T15:55:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.705+0000 7f2b944e1640 1 -- start start 2023-10-16T15:55:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b944e1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b8c15d410 0x7f2b8c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b944e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c0cd260 0x7f2b8c0cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b944e1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c0b5d90 con 0x7f2b8c15d030 2023-10-16T15:55:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b944e1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c0d2b50 con 0x7f2b8c15d410 2023-10-16T15:55:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b944e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0cdbf0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b9227d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2b8c15d030 0x7f2b8c161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60702/0 (socket says 172.21.15.29:60702) 2023-10-16T15:55:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b9227d640 1 -- 172.21.15.29:0/165243575 learned_addr learned my addr 172.21.15.29:0/165243575 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/165243575 >> v2:172.21.15.29:3300/0 conn(0x7f2b8c15d410 0x7f2b8c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.707+0000 7f2b92a7e640 1 --2- 172.21.15.29:0/165243575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c0cd260 0x7f2b8c0cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.708+0000 7f2b9127b640 1 -- 172.21.15.29:0/165243575 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278729096 0 0) 0x7f2b8c0b5d90 con 0x7f2b8c15d030 2023-10-16T15:55:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.708+0000 7f2b9127b640 1 -- 172.21.15.29:0/165243575 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78003520 con 0x7f2b8c15d030 2023-10-16T15:55:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.708+0000 7f2b91a7c640 1 -- 172.21.15.29:0/165243575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c0cd260 msgr2=0x7f2b8c0cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.708+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/165243575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c0cd260 0x7f2b8c0cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.708+0000 7f2b91a7c640 1 -- 172.21.15.29:0/165243575 >> v1:172.21.15.29:6789/0 conn(0x7f2b8c15d030 legacy=0x7f2b8c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.708+0000 7f2b91a7c640 1 -- 172.21.15.29:0/165243575 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c160fc0 con 0x7f2b8c15d410 2023-10-16T15:55:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.709+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/165243575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c15d410 0x7f2b8c0d1630 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0097a0 tx=0x7f2b7c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4c3d84fd862a41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.709+0000 7f2b9127b640 1 -- 172.21.15.29:0/165243575 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c030800 con 0x7f2b8c15d410 2023-10-16T15:55:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.709+0000 7f2b9127b640 1 -- 172.21.15.29:0/165243575 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c030960 con 0x7f2b8c15d410 2023-10-16T15:55:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.709+0000 7f2b9127b640 1 -- 172.21.15.29:0/165243575 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c030cd0 con 0x7f2b8c15d410 2023-10-16T15:55:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 -- 172.21.15.29:0/165243575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c15d410 msgr2=0x7f2b8c0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 --2- 172.21.15.29:0/165243575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c15d410 0x7f2b8c0d1630 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0097a0 tx=0x7f2b7c02b940 comp rx=0 tx=0).stop 2023-10-16T15:55:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 -- 172.21.15.29:0/165243575 shutdown_connections 2023-10-16T15:55:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 --2- 172.21.15.29:0/165243575 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c0cd260 0x7f2b8c0cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 --2- 172.21.15.29:0/165243575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c15d410 0x7f2b8c0d1630 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 -- 172.21.15.29:0/165243575 >> 172.21.15.29:0/165243575 conn(0x7f2b8c0cf720 msgr2=0x7f2b8c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 -- 172.21.15.29:0/165243575 shutdown_connections 2023-10-16T15:55:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.710+0000 7f2b944e1640 1 -- 172.21.15.29:0/165243575 wait complete. 2023-10-16T15:55:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.712+0000 7f2b944e1640 1 Processor -- start 2023-10-16T15:55:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.712+0000 7f2b944e1640 1 -- start start 2023-10-16T15:55:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b944e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c0cd260 0x7f2b8c0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b944e1640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c15d030 0x7f2b8c150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b944e1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b8c15d410 0x7f2b8c0b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b944e1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c15e0c0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b944e1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c15d820 con 0x7f2b8c15d410 2023-10-16T15:55:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b944e1640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c15db00 con 0x7f2b8c15d030 2023-10-16T15:55:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b9227d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c0cd260 0x7f2b8c0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b9227d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c0cd260 0x7f2b8c0cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54216/0 (socket says 172.21.15.29:54216) 2023-10-16T15:55:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b9227d640 1 -- 172.21.15.29:0/4092384329 learned_addr learned my addr 172.21.15.29:0/4092384329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c15d030 0x7f2b8c150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.713+0000 7f2b9227d640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c15d030 msgr2=0x7f2b8c150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b9227d640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c15d030 0x7f2b8c150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b9227d640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b8c15d410 msgr2=0x7f2b8c0b66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b9227d640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2b8c15d410 0x7f2b8c0b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b9227d640 1 -- 172.21.15.29:0/4092384329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c009400 con 0x7f2b8c0cd260 2023-10-16T15:55:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f2b8c15d030 0x7f2b8c150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b9227d640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c0cd260 0x7f2b8c0cc180 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f2b880010e0 tx=0x7f2b8800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88005ce0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88005e60 con 0x7f2b8c0cd260 2023-10-16T15:55:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b880072f0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c15dde0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.714+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c0cb6c0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.722 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.714+0000 7fb59124f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.716+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c0cb3e0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.717+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f2b88007450 con 0x7f2b8c0cd260 2023-10-16T15:55:51.723 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.718+0000 7f44c1214040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.718+0000 7f2b82ffd640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 0x7f2b7006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.718+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5780+0+0 (secure 0 0 0) 0x7f2b88089110 con 0x7f2b8c0cd260 2023-10-16T15:55:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.718+0000 7f2b91a7c640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 msgr2=0x7f2b7006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.718+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 0x7f2b7006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.722+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b8c0cb3e0 con 0x7f2b8c0cd260 2023-10-16T15:55:51.784 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.782+0000 7f44c1214040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:51.826 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.824+0000 7fb59124f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:55:51.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.834+0000 7f44c1214040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:51.888 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.887+0000 7fb59124f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.919+0000 7f2b91a7c640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 msgr2=0x7f2b7006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.919+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 0x7f2b7006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:51.938 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.936+0000 7fb59124f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:55:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:51.936+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 --> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b8c0d2290 con 0x7f2b7006b130 2023-10-16T15:55:51.955 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:51.953+0000 7f44c1214040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:55:51.996 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:51.994+0000 7fb59124f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:55:52.021 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.019+0000 7f44c1214040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:52.046 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.044+0000 7fb59124f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:55:52.072 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.070+0000 7f44c1214040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:55:52.131 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.129+0000 7f44c1214040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:55:52.184 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.182+0000 7f44c1214040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:55:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:52.320+0000 7f2b91a7c640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 msgr2=0x7f2b7006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:52.320+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 0x7f2b7006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T15:55:52.431 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.429+0000 7fb59124f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:55:52.583 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.581+0000 7f44c1214040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:55:52.591 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.589+0000 7fb59124f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:55:52.702 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.700+0000 7fb59124f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:55:52.752 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.750+0000 7f44c1214040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:55:52.752 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.750+0000 7fb59124f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:55:52.858 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.857+0000 7fb59124f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:52.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.862+0000 7f44c1214040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:55:52.917 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:52.915+0000 7f44c1214040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:55:52.966 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:52.964+0000 7fb59124f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:55:53.030 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:53.028+0000 7f44c1214040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:53.122 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:53.120+0000 7fb59124f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:53.121+0000 7f2b91a7c640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 msgr2=0x7f2b7006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/3469082932 2023-10-16T15:55:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:53.121+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 0x7f2b7006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T15:55:53.141 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:53.139+0000 7f44c1214040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:55:53.171 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:53.170+0000 7fb59124f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:53.290 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:53.288+0000 7fb59124f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:55:53.307 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:53.304+0000 7f44c1214040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:55:53.358 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:53.357+0000 7f44c1214040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:55:53.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:53.482+0000 7f44c1214040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:55:54.123 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.121+0000 7fb59124f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:55:54.352 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:54.351+0000 7f44c1214040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:55:54.416 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.414+0000 7fb59124f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:55:54.473 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.472+0000 7fb59124f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:55:54.587 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.585+0000 7fb59124f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:55:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:54.606+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 148) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f2b88054150 con 0x7f2b8c0cd260 2023-10-16T15:55:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:54.607+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 msgr2=0x7f2b7006d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:54.607+0000 7f2b82ffd640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/3469082932,v1:172.21.15.170:6833/3469082932] conn(0x7f2b7006b130 0x7f2b7006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:54.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:54.671+0000 7f44c1214040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:55:54.735 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:54.734+0000 7f44c1214040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:55:54.746 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.743+0000 7fb50dde3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.746 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb50dde3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.746 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb50dde3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.746 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb50dde3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.747 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb50dde3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.747 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb510e29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.748 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb510e29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.748 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb510e29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.748 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb510e29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.748 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:54.744+0000 7fb510e29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:55:54.857 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:54.856+0000 7f44c1214040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:55:55.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.617+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 149) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2b880874c0 con 0x7f2b8c0cd260 2023-10-16T15:55:55.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.618+0000 7f2b82ffd640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f2b70035890 0x7f2b70075110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:55.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.618+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 --> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b700283c0 con 0x7f2b70035890 2023-10-16T15:55:55.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.623+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f2b70035890 0x7f2b70075110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:55.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.627+0000 7f2b91a7c640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f2b70035890 0x7f2b70075110 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c036000 tx=0x7f2b7c008030 comp rx=0 tx=0).ready entity=mgr.9805 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:55.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.770+0000 7f2b82ffd640 1 -- 172.21.15.29:0/4092384329 <== mgr.9805 v2:172.21.15.170:6832/1364949379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2b700283c0 con 0x7f2b70035890 2023-10-16T15:55:55.771 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-16T15:55:55.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.773+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f2b70035890 msgr2=0x7f2b70075110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:55.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.773+0000 7f2b944e1640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f2b70035890 0x7f2b70075110 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c036000 tx=0x7f2b7c008030 comp rx=0 tx=0).stop 2023-10-16T15:55:55.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.773+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c0cd260 msgr2=0x7f2b8c0cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:55.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.773+0000 7f2b944e1640 1 --2- 172.21.15.29:0/4092384329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b8c0cd260 0x7f2b8c0cc180 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f2b880010e0 tx=0x7f2b8800c960 comp rx=0 tx=0).stop 2023-10-16T15:55:55.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.773+0000 7f2b92a7e640 1 -- 172.21.15.29:0/4092384329 reap_dead start 2023-10-16T15:55:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.774+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 shutdown_connections 2023-10-16T15:55:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.774+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 >> 172.21.15.29:0/4092384329 conn(0x7f2b8c0cf720 msgr2=0x7f2b8c15c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.774+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 shutdown_connections 2023-10-16T15:55:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:55.774+0000 7f2b944e1640 1 -- 172.21.15.29:0/4092384329 wait complete. 2023-10-16T15:55:55.796 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T15:55:55.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T15:55:56.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.030+0000 7fe7ed035640 1 Processor -- start 2023-10-16T15:55:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.030+0000 7fe7ed035640 1 -- start start 2023-10-16T15:55:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.031+0000 7fe7ed035640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e806c040 0x7fe7e806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.031+0000 7fe7ed035640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe7e80ad970 0x7fe7e815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.031+0000 7fe7ed035640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7e815ebc0 0x7fe7e815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.031+0000 7fe7ed035640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e806c960 con 0x7fe7e815ebc0 2023-10-16T15:55:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.031+0000 7fe7ed035640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e815b9d0 con 0x7fe7e80ad970 2023-10-16T15:55:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.031+0000 7fe7ed035640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e815bcb0 con 0x7fe7e806c040 2023-10-16T15:55:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7e815ebc0 0x7fe7e815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7e815ebc0 0x7fe7e815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42802/0 (socket says 172.21.15.29:42802) 2023-10-16T15:55:56.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 -- 172.21.15.29:0/4104380182 learned_addr learned my addr 172.21.15.29:0/4104380182 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:56.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7e7fff640 1 --2- 172.21.15.29:0/4104380182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e806c040 0x7fe7e806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7dffff640 1 --2- 172.21.15.29:0/4104380182 >> v2:172.21.15.170:3300/0 conn(0x7fe7e80ad970 0x7fe7e815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 -- 172.21.15.29:0/4104380182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e806c040 msgr2=0x7fe7e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:56.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 --2- 172.21.15.29:0/4104380182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e806c040 0x7fe7e806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 -- 172.21.15.29:0/4104380182 >> v2:172.21.15.170:3300/0 conn(0x7fe7e80ad970 msgr2=0x7fe7e815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:56.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 --2- 172.21.15.29:0/4104380182 >> v2:172.21.15.170:3300/0 conn(0x7fe7e80ad970 0x7fe7e815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.032+0000 7fe7ec834640 1 -- 172.21.15.29:0/4104380182 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e815bf90 con 0x7fe7e815ebc0 2023-10-16T15:55:56.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.034+0000 7fe7ec834640 1 --2- 172.21.15.29:0/4104380182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e815ebc0 0x7fe7e815f050 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009e50 tx=0x7fe7e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2aaaca6dbb2d644 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:56.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.034+0000 7fe7e77fe640 1 -- 172.21.15.29:0/4104380182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e002ca80 con 0x7fe7e815ebc0 2023-10-16T15:55:56.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.034+0000 7fe7e77fe640 1 -- 172.21.15.29:0/4104380182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e002cbe0 con 0x7fe7e815ebc0 2023-10-16T15:55:56.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.034+0000 7fe7e77fe640 1 -- 172.21.15.29:0/4104380182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00343f0 con 0x7fe7e815ebc0 2023-10-16T15:55:56.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 -- 172.21.15.29:0/4104380182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e815ebc0 msgr2=0x7fe7e815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 --2- 172.21.15.29:0/4104380182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e815ebc0 0x7fe7e815f050 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009e50 tx=0x7fe7e002bee0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 -- 172.21.15.29:0/4104380182 shutdown_connections 2023-10-16T15:55:56.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 --2- 172.21.15.29:0/4104380182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e815ebc0 0x7fe7e815f050 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 --2- 172.21.15.29:0/4104380182 >> v2:172.21.15.170:3300/0 conn(0x7fe7e80ad970 0x7fe7e815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 --2- 172.21.15.29:0/4104380182 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e806c040 0x7fe7e806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 -- 172.21.15.29:0/4104380182 >> 172.21.15.29:0/4104380182 conn(0x7fe7e80ce6a0 msgr2=0x7fe7e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:56.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 -- 172.21.15.29:0/4104380182 shutdown_connections 2023-10-16T15:55:56.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.035+0000 7fe7ed035640 1 -- 172.21.15.29:0/4104380182 wait complete. 2023-10-16T15:55:56.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.037+0000 7fe7ed035640 1 Processor -- start 2023-10-16T15:55:56.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.037+0000 7fe7ed035640 1 -- start start 2023-10-16T15:55:56.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ed035640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7e806c040 0x7fe7e80b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ed035640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 0x7fe7e80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ed035640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e815ebc0 0x7fe7e80acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ed035640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f5b60 con 0x7fe7e80ad970 2023-10-16T15:55:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ed035640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f32f0 con 0x7fe7e806c040 2023-10-16T15:55:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ed035640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f5d70 con 0x7fe7e815ebc0 2023-10-16T15:55:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 0x7fe7e80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 0x7fe7e80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42806/0 (socket says 172.21.15.29:42806) 2023-10-16T15:55:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7dffff640 1 -- 172.21.15.29:0/1067168998 learned_addr learned my addr 172.21.15.29:0/1067168998 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7e7fff640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7e806c040 0x7fe7e80b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.038+0000 7fe7ec834640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e815ebc0 0x7fe7e80acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7dffff640 1 -- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e815ebc0 msgr2=0x7fe7e80acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7dffff640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e815ebc0 0x7fe7e80acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7dffff640 1 -- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7e806c040 msgr2=0x7fe7e80b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7dffff640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7e806c040 0x7fe7e80b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7dffff640 1 -- 172.21.15.29:0/1067168998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0009b30 con 0x7fe7e80ad970 2023-10-16T15:55:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7dffff640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 0x7fe7e80ac890 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000d5f0 tx=0x7fe7d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00125f0 con 0x7fe7e80ad970 2023-10-16T15:55:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d0012750 con 0x7fe7e80ad970 2023-10-16T15:55:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e81f6050 con 0x7fe7e80ad970 2023-10-16T15:55:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0012b00 con 0x7fe7e80ad970 2023-10-16T15:55:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.039+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e81f6850 con 0x7fe7e80ad970 2023-10-16T15:55:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.041+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e806b330 con 0x7fe7e80ad970 2023-10-16T15:55:56.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.042+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 149) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe7d0012c60 con 0x7fe7e80ad970 2023-10-16T15:55:56.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.043+0000 7fe7e57fa640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fe7c806ad10 0x7fe7c806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.043+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fe7d0089960 con 0x7fe7e80ad970 2023-10-16T15:55:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.043+0000 7fe7e7fff640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fe7c806ad10 0x7fe7c806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.044+0000 7fe7e7fff640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fe7c806ad10 0x7fe7c806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8006fd0 tx=0x7fe7d8008040 comp rx=0 tx=0).ready entity=mgr.9805 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.047+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe7d0054b50 con 0x7fe7e80ad970 2023-10-16T15:55:56.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.265+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 --> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7e80cc550 con 0x7fe7c806ad10 2023-10-16T15:55:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.410+0000 7fe7e57fa640 1 -- 172.21.15.29:0/1067168998 <== mgr.9805 v2:172.21.15.170:6832/1364949379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fe7d800f860 con 0x7fe7c806ad10 2023-10-16T15:55:56.411 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path snap_test_dir1 2023-10-16T15:55:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fe7c806ad10 msgr2=0x7fe7c806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fe7c806ad10 0x7fe7c806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8006fd0 tx=0x7fe7d8008040 comp rx=0 tx=0).stop 2023-10-16T15:55:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 msgr2=0x7fe7e80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 0x7fe7e80ac890 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000d5f0 tx=0x7fe7d000dab0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 shutdown_connections 2023-10-16T15:55:56.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fe7c806ad10 0x7fe7c806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e80ad970 0x7fe7e80ac890 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7e815ebc0 0x7fe7e80acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 --2- 172.21.15.29:0/1067168998 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7e806c040 0x7fe7e80b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 >> 172.21.15.29:0/1067168998 conn(0x7fe7e80ce6a0 msgr2=0x7fe7e815a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:56.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 shutdown_connections 2023-10-16T15:55:56.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.414+0000 7fe7ed035640 1 -- 172.21.15.29:0/1067168998 wait complete. 2023-10-16T15:55:56.439 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T15:55:56.443 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-15_56_00 in ~4s... 2023-10-16T15:55:56.444 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T15:55:56.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T15:55:56.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.684+0000 7f6cad456640 1 Processor -- start 2023-10-16T15:55:56.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.684+0000 7f6cad456640 1 -- start start 2023-10-16T15:55:56.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6cad456640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ca815d330 0x7f6ca81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6cad456640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca80661c0 0x7f6ca80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6cad456640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca80d0890 con 0x7f6ca806c7d0 2023-10-16T15:55:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6cad456640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca806b1e0 con 0x7f6ca815d330 2023-10-16T15:55:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6cad456640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80d05b0 con 0x7f6ca80661c0 2023-10-16T15:55:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6ca6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ca815d330 0x7f6ca81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6ca6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ca815d330 0x7f6ca81618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42818/0 (socket says 172.21.15.29:42818) 2023-10-16T15:55:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.686+0000 7f6ca6ffd640 1 -- 172.21.15.29:0/991897839 learned_addr learned my addr 172.21.15.29:0/991897839 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.687+0000 7f6ca6ffd640 1 -- 172.21.15.29:0/991897839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca80661c0 msgr2=0x7f6ca80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.687+0000 7f6ca6ffd640 1 --2- 172.21.15.29:0/991897839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca80661c0 0x7f6ca80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.687+0000 7f6ca6ffd640 1 -- 172.21.15.29:0/991897839 >> v1:172.21.15.170:6789/0 conn(0x7f6ca806c7d0 legacy=0x7f6ca8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:56.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.687+0000 7f6ca6ffd640 1 -- 172.21.15.29:0/991897839 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca8160dd0 con 0x7f6ca815d330 2023-10-16T15:55:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.688+0000 7f6ca6ffd640 1 --2- 172.21.15.29:0/991897839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca815d330 0x7f6ca81618b0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009870 tx=0x7f6c9802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d2b129c7921d78 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.688+0000 7f6ca5ffb640 1 -- 172.21.15.29:0/991897839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c980317c0 con 0x7f6ca815d330 2023-10-16T15:55:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.688+0000 7f6ca5ffb640 1 -- 172.21.15.29:0/991897839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98031920 con 0x7f6ca815d330 2023-10-16T15:55:56.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.688+0000 7f6ca5ffb640 1 -- 172.21.15.29:0/991897839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98031cb0 con 0x7f6ca815d330 2023-10-16T15:55:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 -- 172.21.15.29:0/991897839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca815d330 msgr2=0x7f6ca81618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 --2- 172.21.15.29:0/991897839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca815d330 0x7f6ca81618b0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009870 tx=0x7f6c9802b860 comp rx=0 tx=0).stop 2023-10-16T15:55:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 -- 172.21.15.29:0/991897839 shutdown_connections 2023-10-16T15:55:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 --2- 172.21.15.29:0/991897839 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca80661c0 0x7f6ca80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 --2- 172.21.15.29:0/991897839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca815d330 0x7f6ca81618b0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 -- 172.21.15.29:0/991897839 >> 172.21.15.29:0/991897839 conn(0x7f6ca80d0ab0 msgr2=0x7f6ca80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 -- 172.21.15.29:0/991897839 shutdown_connections 2023-10-16T15:55:56.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.689+0000 7f6cad456640 1 -- 172.21.15.29:0/991897839 wait complete. 2023-10-16T15:55:56.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.691+0000 7f6cad456640 1 Processor -- start 2023-10-16T15:55:56.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.691+0000 7f6cad456640 1 -- start start 2023-10-16T15:55:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6cad456640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ca80661c0 0x7f6ca8151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6cad456640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 0x7f6ca80c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6cad456640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca815d330 0x7f6ca80c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6cad456640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80c9440 con 0x7f6ca806c7d0 2023-10-16T15:55:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6cad456640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca8152370 con 0x7f6ca80661c0 2023-10-16T15:55:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6cad456640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80c72b0 con 0x7f6ca815d330 2023-10-16T15:55:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.692+0000 7f6ca67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 0x7f6ca80c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 0x7f6ca80c5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42830/0 (socket says 172.21.15.29:42830) 2023-10-16T15:55:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 -- 172.21.15.29:0/3053086731 learned_addr learned my addr 172.21.15.29:0/3053086731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:56.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca6ffd640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ca80661c0 0x7f6ca8151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca77fe640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca815d330 0x7f6ca80c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 -- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca815d330 msgr2=0x7f6ca80c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca815d330 0x7f6ca80c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 -- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ca80661c0 msgr2=0x7f6ca8151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ca80661c0 0x7f6ca8151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 -- 172.21.15.29:0/3053086731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980094d0 con 0x7f6ca806c7d0 2023-10-16T15:55:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca6ffd640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ca80661c0 0x7f6ca8151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca77fe640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca815d330 0x7f6ca80c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.693+0000 7f6ca67fc640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 0x7f6ca80c5f00 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c00b3e0 tx=0x7f6c9c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.694+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c011830 con 0x7f6ca806c7d0 2023-10-16T15:55:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.694+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c0119b0 con 0x7f6ca806c7d0 2023-10-16T15:55:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.694+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c011d60 con 0x7f6ca806c7d0 2023-10-16T15:55:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.694+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca80c88c0 con 0x7f6ca806c7d0 2023-10-16T15:55:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.694+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca80c8e80 con 0x7f6ca806c7d0 2023-10-16T15:55:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.695+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca8159a20 con 0x7f6ca806c7d0 2023-10-16T15:55:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.696+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6c9c01a400 con 0x7f6ca806c7d0 2023-10-16T15:55:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.698+0000 7f6c8f7fe640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f6c9406afe0 0x7f6c9406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:56.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.698+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f6c9c089bf0 con 0x7f6ca806c7d0 2023-10-16T15:55:56.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.698+0000 7f6ca6ffd640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f6c9406afe0 0x7f6c9406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:56.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.699+0000 7f6ca6ffd640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f6c9406afe0 0x7f6c9406d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c98036000 tx=0x7f6c9803d040 comp rx=0 tx=0).ready entity=mgr.9805 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:56.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.702+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ca8159a20 con 0x7f6ca806c7d0 2023-10-16T15:55:56.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.918+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 --> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ca8161390 con 0x7f6c9406afe0 2023-10-16T15:55:56.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.935+0000 7f6c8f7fe640 1 -- 172.21.15.29:0/3053086731 <== mgr.9805 v2:172.21.15.170:6832/1364949379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f6c98031c10 con 0x7f6c9406afe0 2023-10-16T15:55:56.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:56.936 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T15:55:56.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.937+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f6c9406afe0 msgr2=0x7f6c9406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f6c9406afe0 0x7f6c9406d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c98036000 tx=0x7f6c9803d040 comp rx=0 tx=0).stop 2023-10-16T15:55:56.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 msgr2=0x7f6ca80c5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:56.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 0x7f6ca80c5f00 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c00b3e0 tx=0x7f6c9c00b8a0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 shutdown_connections 2023-10-16T15:55:56.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f6c9406afe0 0x7f6c9406d4a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca806c7d0 0x7f6ca80c5f00 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ca815d330 0x7f6ca80c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 --2- 172.21.15.29:0/3053086731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ca80661c0 0x7f6ca8151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:56.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 >> 172.21.15.29:0/3053086731 conn(0x7f6ca80d0ab0 msgr2=0x7f6ca815e630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:56.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 shutdown_connections 2023-10-16T15:55:56.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:56.938+0000 7f6cad456640 1 -- 172.21.15.29:0/3053086731 wait complete. 2023-10-16T15:55:56.960 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T15:55:56.960 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T15:55:56.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T15:55:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.190+0000 7f6200839640 1 Processor -- start 2023-10-16T15:55:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.191+0000 7f6200839640 1 -- start start 2023-10-16T15:55:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.192+0000 7f6200839640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61fc0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.192+0000 7f6200839640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0ac890 con 0x7f61fc160b90 2023-10-16T15:55:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.192+0000 7f6200839640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc161650 con 0x7f61fc06a4b0 2023-10-16T15:55:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.192+0000 7f6200839640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc15d530 con 0x7f61fc15d150 2023-10-16T15:55:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.192+0000 7f61f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61fc160b90 0x7f61fc069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47706/0 (socket says 172.21.15.29:47706) 2023-10-16T15:55:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f3fff640 1 -- 172.21.15.29:0/3314107899 learned_addr learned my addr 172.21.15.29:0/3314107899 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61fa59c640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61fc0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915405506 0 0) 0x7f61fc0ac890 con 0x7f61fc160b90 2023-10-16T15:55:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e8003520 con 0x7f61fc160b90 2023-10-16T15:55:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3423958073 0 0) 0x7f61e8003520 con 0x7f61fc160b90 2023-10-16T15:55:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 msgr2=0x7f61fc0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61fc0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 >> v1:172.21.15.170:6789/0 conn(0x7f61fc06a4b0 legacy=0x7f61fc0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.193+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc161930 con 0x7f61fc160b90 2023-10-16T15:55:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f61e4001670 con 0x7f61fc160b90 2023-10-16T15:55:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 >> v1:172.21.15.29:6789/0 conn(0x7f61fc160b90 legacy=0x7f61fc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e80039a0 0x7f61e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8004d00 0x7f61e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc15d530 con 0x7f61e8004d00 2023-10-16T15:55:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e800ba30 con 0x7f61e80039a0 2023-10-16T15:55:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e800bbf0 con 0x7f61fc15d150 2023-10-16T15:55:57.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61fad9d640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8004d00 0x7f61e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61fa59c640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e80039a0 0x7f61e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61f3fff640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.194+0000 7f61fad9d640 1 -- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 msgr2=0x7f61e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.195+0000 7f61fad9d640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.195+0000 7f61fad9d640 1 -- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e80039a0 msgr2=0x7f61e8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.195+0000 7f61fad9d640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e80039a0 0x7f61e8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.195+0000 7f61fad9d640 1 -- 172.21.15.29:0/3314107899 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e8003520 con 0x7f61e8004d00 2023-10-16T15:55:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.195+0000 7f61f3fff640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15d150 0x7f61e80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.195+0000 7f61fa59c640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e80039a0 0x7f61e8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:55:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.196+0000 7f61fad9d640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8004d00 0x7f61e8005180 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f61f4001e20 tx=0x7f61e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557acbb50bbddf99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.196+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4003b70 con 0x7f61e8004d00 2023-10-16T15:55:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.196+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4003cd0 con 0x7f61e8004d00 2023-10-16T15:55:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.196+0000 7f61f9d9b640 1 -- 172.21.15.29:0/3314107899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4005310 con 0x7f61e8004d00 2023-10-16T15:55:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f6200839640 1 -- 172.21.15.29:0/3314107899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8004d00 msgr2=0x7f61e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f6200839640 1 --2- 172.21.15.29:0/3314107899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8004d00 0x7f61e8005180 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f61f4001e20 tx=0x7f61e80057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f61fad9d640 1 -- 172.21.15.29:0/3314107899 reap_dead start 2023-10-16T15:55:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f6200839640 1 -- 172.21.15.29:0/3314107899 shutdown_connections 2023-10-16T15:55:57.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f6200839640 1 -- 172.21.15.29:0/3314107899 >> 172.21.15.29:0/3314107899 conn(0x7f61fc0ce9b0 msgr2=0x7f61fc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:57.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f6200839640 1 -- 172.21.15.29:0/3314107899 shutdown_connections 2023-10-16T15:55:57.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.197+0000 7f6200839640 1 -- 172.21.15.29:0/3314107899 wait complete. 2023-10-16T15:55:57.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.199+0000 7f6200839640 1 Processor -- start 2023-10-16T15:55:57.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.199+0000 7f6200839640 1 -- start start 2023-10-16T15:55:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f6200839640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e8004d00 0x7f61fc15d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f6200839640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 0x7f61fc152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f6200839640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15a340 0x7f61fc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f6200839640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc160500 con 0x7f61fc1f2700 2023-10-16T15:55:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f6200839640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc0c4010 con 0x7f61e8004d00 2023-10-16T15:55:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f6200839640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc0c3720 con 0x7f61fc15a340 2023-10-16T15:55:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 0x7f61fc152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 0x7f61fc152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42838/0 (socket says 172.21.15.29:42838) 2023-10-16T15:55:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f61f3fff640 1 -- 172.21.15.29:0/599937576 learned_addr learned my addr 172.21.15.29:0/599937576 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.200+0000 7f61fa59c640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e8004d00 0x7f61fc15d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f3fff640 1 -- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15a340 msgr2=0x7f61fc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f3fff640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15a340 0x7f61fc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f3fff640 1 -- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e8004d00 msgr2=0x7f61fc15d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f3fff640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e8004d00 0x7f61fc15d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f3fff640 1 -- 172.21.15.29:0/599937576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0c39c0 con 0x7f61fc1f2700 2023-10-16T15:55:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f3fff640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 0x7f61fc152f90 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f61e400d860 tx=0x7f61e400dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e40074e0 con 0x7f61fc1f2700 2023-10-16T15:55:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e4007640 con 0x7f61fc1f2700 2023-10-16T15:55:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e40049e0 con 0x7f61fc1f2700 2023-10-16T15:55:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc0c3ca0 con 0x7f61fc1f2700 2023-10-16T15:55:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.201+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc1534d0 con 0x7f61fc1f2700 2023-10-16T15:55:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.203+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc04e160 con 0x7f61fc1f2700 2023-10-16T15:55:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.204+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f61e4004b40 con 0x7f61fc1f2700 2023-10-16T15:55:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.205+0000 7f61f2ffd640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f61dc06afe0 0x7f61dc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.205+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f61e4089490 con 0x7f61fc1f2700 2023-10-16T15:55:57.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.205+0000 7f61fa59c640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f61dc06afe0 0x7f61dc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.207+0000 7f61fa59c640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f61dc06afe0 0x7f61dc06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009a90 tx=0x7f61ec008040 comp rx=0 tx=0).ready entity=mgr.9805 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:57.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.209+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f61e4054120 con 0x7f61fc1f2700 2023-10-16T15:55:57.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.432+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 --> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f61fc0ace30 con 0x7f61dc06afe0 2023-10-16T15:55:57.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.603+0000 7f61f2ffd640 1 -- 172.21.15.29:0/599937576 <== mgr.9805 v2:172.21.15.170:6832/1364949379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f61ec013a20 con 0x7f61dc06afe0 2023-10-16T15:55:57.605 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-16T15:55:57.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f61dc06afe0 msgr2=0x7f61dc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f61dc06afe0 0x7f61dc06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009a90 tx=0x7f61ec008040 comp rx=0 tx=0).stop 2023-10-16T15:55:57.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 msgr2=0x7f61fc152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 0x7f61fc152f90 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f61e400d860 tx=0x7f61e400dd20 comp rx=0 tx=0).stop 2023-10-16T15:55:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 shutdown_connections 2023-10-16T15:55:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f61dc06afe0 0x7f61dc06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61fc1f2700 0x7f61fc152f90 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f61fc15a340 0x7f61fc0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 --2- 172.21.15.29:0/599937576 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f61e8004d00 0x7f61fc15d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 >> 172.21.15.29:0/599937576 conn(0x7f61fc0ce9b0 msgr2=0x7f61fc15bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 shutdown_connections 2023-10-16T15:55:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.607+0000 7f6200839640 1 -- 172.21.15.29:0/599937576 wait complete. 2023-10-16T15:55:57.633 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T15:55:57.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:57.634 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T15:55:57.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:55:57.692 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T15:55:57.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T15:55:57.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.977+0000 7f06bc835640 1 Processor -- start 2023-10-16T15:55:57.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.977+0000 7f06bc835640 1 -- start start 2023-10-16T15:55:57.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.979+0000 7f06bc835640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f06b815c9f0 0x7f06b80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.979+0000 7f06bc835640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06b80c9230 0x7f06b80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.979+0000 7f06bc835640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b8160a90 con 0x7f06b8070f20 2023-10-16T15:55:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.979+0000 7f06bc835640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b80cdec0 con 0x7f06b80c9230 2023-10-16T15:55:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.979+0000 7f06bc835640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b80cd900 con 0x7f06b815c9f0 2023-10-16T15:55:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.979+0000 7f06affff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06b80c9230 0x7f06b80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06affff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f06b80c9230 0x7f06b80709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42844/0 (socket says 172.21.15.29:42844) 2023-10-16T15:55:57.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06affff640 1 -- 172.21.15.29:0/3090240099 learned_addr learned my addr 172.21.15.29:0/3090240099 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06b77fe640 1 --2- 172.21.15.29:0/3090240099 >> v2:172.21.15.170:3300/0 conn(0x7f06b815c9f0 0x7f06b80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06affff640 1 -- 172.21.15.29:0/3090240099 >> v2:172.21.15.170:3300/0 conn(0x7f06b815c9f0 msgr2=0x7f06b80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06affff640 1 --2- 172.21.15.29:0/3090240099 >> v2:172.21.15.170:3300/0 conn(0x7f06b815c9f0 0x7f06b80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06affff640 1 -- 172.21.15.29:0/3090240099 >> v1:172.21.15.29:6789/0 conn(0x7f06b8070f20 legacy=0x7f06b80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.980+0000 7f06affff640 1 -- 172.21.15.29:0/3090240099 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80cdbe0 con 0x7f06b80c9230 2023-10-16T15:55:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.981+0000 7f06affff640 1 --2- 172.21.15.29:0/3090240099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b80709e0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f06b80ad4e0 tx=0x7f06a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1d4eca69cdd955 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06b6ffd640 1 -- 172.21.15.29:0/3090240099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0030860 con 0x7f06b80c9230 2023-10-16T15:55:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06b6ffd640 1 -- 172.21.15.29:0/3090240099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a00309c0 con 0x7f06b80c9230 2023-10-16T15:55:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06b6ffd640 1 -- 172.21.15.29:0/3090240099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0030d50 con 0x7f06b80c9230 2023-10-16T15:55:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06bc835640 1 -- 172.21.15.29:0/3090240099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 msgr2=0x7f06b80709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06bc835640 1 --2- 172.21.15.29:0/3090240099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b80709e0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f06b80ad4e0 tx=0x7f06a002b9a0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06bc835640 1 -- 172.21.15.29:0/3090240099 shutdown_connections 2023-10-16T15:55:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06bc835640 1 --2- 172.21.15.29:0/3090240099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b80709e0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06bc835640 1 --2- 172.21.15.29:0/3090240099 >> v2:172.21.15.170:3300/0 conn(0x7f06b815c9f0 0x7f06b80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.982+0000 7f06bc835640 1 -- 172.21.15.29:0/3090240099 >> 172.21.15.29:0/3090240099 conn(0x7f06b80ce6a0 msgr2=0x7f06b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.983+0000 7f06bc835640 1 -- 172.21.15.29:0/3090240099 shutdown_connections 2023-10-16T15:55:57.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.983+0000 7f06bc835640 1 -- 172.21.15.29:0/3090240099 wait complete. 2023-10-16T15:55:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.984+0000 7f06bc835640 1 Processor -- start 2023-10-16T15:55:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.985+0000 7f06bc835640 1 -- start start 2023-10-16T15:55:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06bc835640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06b8070f20 0x7f06b8151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06bc835640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b8150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06bc835640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06b815c9f0 0x7f06b806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06bc835640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c3230 con 0x7f06b80c9230 2023-10-16T15:55:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06bc835640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c26b0 con 0x7f06b815c9f0 2023-10-16T15:55:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06bc835640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c2990 con 0x7f06b8070f20 2023-10-16T15:55:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b8150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b8150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42852/0 (socket says 172.21.15.29:42852) 2023-10-16T15:55:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 -- 172.21.15.29:0/2477542735 learned_addr learned my addr 172.21.15.29:0/2477542735 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06b77fe640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06b8070f20 0x7f06b8151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06b7fff640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06b815c9f0 0x7f06b806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 -- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06b8070f20 msgr2=0x7f06b8151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06b8070f20 0x7f06b8151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 -- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06b815c9f0 msgr2=0x7f06b806b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06b815c9f0 0x7f06b806b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06affff640 1 -- 172.21.15.29:0/2477542735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00094d0 con 0x7f06b80c9230 2023-10-16T15:55:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.986+0000 7f06b77fe640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06b8070f20 0x7f06b8151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.987+0000 7f06affff640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b8150b60 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f06a002b9e0 tx=0x7f06a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.987+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0039d60 con 0x7f06b80c9230 2023-10-16T15:55:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.987+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a0043400 con 0x7f06b80c9230 2023-10-16T15:55:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.987+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00437b0 con 0x7f06b80c9230 2023-10-16T15:55:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.987+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b80c2c70 con 0x7f06b80c9230 2023-10-16T15:55:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.987+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b81510a0 con 0x7f06b80c9230 2023-10-16T15:55:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.989+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b804e9b0 con 0x7f06b80c9230 2023-10-16T15:55:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.989+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 150) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f06a0043910 con 0x7f06b80c9230 2023-10-16T15:55:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.990+0000 7f06b4ff9640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f069806afe0 0x7f069806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.991+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f06a003e070 con 0x7f06b80c9230 2023-10-16T15:55:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.991+0000 7f06b77fe640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f069806afe0 0x7f069806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.992+0000 7f06b77fe640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f069806afe0 0x7f069806d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f06a8006fd0 tx=0x7f06a8008040 comp rx=0 tx=0).ready entity=mgr.9805 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:57.995+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f06a0077940 con 0x7f06b80c9230 2023-10-16T15:55:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.239+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f06b80adac0 con 0x7f06b80c9230 2023-10-16T15:55:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.682+0000 7f06b4ff9640 1 -- 172.21.15.29:0/2477542735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v151) v1 ==== 93+0+0 (secure 0 0 0) 0x7f06a00340a0 con 0x7f06b80c9230 2023-10-16T15:55:58.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.685+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f069806afe0 msgr2=0x7f069806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.685+0000 7f06bc835640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f069806afe0 0x7f069806d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f06a8006fd0 tx=0x7f06a8008040 comp rx=0 tx=0).stop 2023-10-16T15:55:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.685+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 msgr2=0x7f06b8150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b8150b60 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f06a002b9e0 tx=0x7f06a0004730 comp rx=0 tx=0).stop 2023-10-16T15:55:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 shutdown_connections 2023-10-16T15:55:58.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f069806afe0 0x7f069806d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:58.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06b80c9230 0x7f06b8150b60 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:58.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f06b815c9f0 0x7f06b806b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:58.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 --2- 172.21.15.29:0/2477542735 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f06b8070f20 0x7f06b8151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:58.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.686+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 >> 172.21.15.29:0/2477542735 conn(0x7f06b80ce6a0 msgr2=0x7f06b80c15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:58.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.687+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 shutdown_connections 2023-10-16T15:55:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:58.687+0000 7f06bc835640 1 -- 172.21.15.29:0/2477542735 wait complete. 2023-10-16T15:55:58.711 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T15:55:58.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:55:58.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T15:55:58.924 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T15:55:58.924 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:58.924 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:58.981 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:55:58.982 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:55:59.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:55:59.215 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.213+0000 7f577862e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:59.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:59.256+0000 7f56c76b0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T15:55:59.267 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.265+0000 7f577862e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.287+0000 7f1a2b4c2640 1 Processor -- start 2023-10-16T15:55:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.288+0000 7f1a2b4c2640 1 -- start start 2023-10-16T15:55:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.289+0000 7f1a2b4c2640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1a2415d110 0x7f1a240caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.289+0000 7f1a2b4c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a24160ce0 0x7f1a2406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.289+0000 7f1a2b4c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a2406a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.289+0000 7f1a2b4c2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a240cdfe0 con 0x7f1a24160ce0 2023-10-16T15:55:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.289+0000 7f1a2b4c2640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2406a710 con 0x7f1a2415d110 2023-10-16T15:55:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.289+0000 7f1a2b4c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a24161250 con 0x7f1a24069da0 2023-10-16T15:55:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a24160ce0 0x7f1a2406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a24160ce0 0x7f1a2406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42884/0 (socket says 172.21.15.29:42884) 2023-10-16T15:55:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 -- 172.21.15.29:0/2953860830 learned_addr learned my addr 172.21.15.29:0/2953860830 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a29a5f640 1 --2- 172.21.15.29:0/2953860830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a2406a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a2925e640 1 --2- 172.21.15.29:0/2953860830 >> v2:172.21.15.170:3300/0 conn(0x7f1a2415d110 0x7f1a240caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 -- 172.21.15.29:0/2953860830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 msgr2=0x7f1a2406a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 --2- 172.21.15.29:0/2953860830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a2406a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 -- 172.21.15.29:0/2953860830 >> v2:172.21.15.170:3300/0 conn(0x7f1a2415d110 msgr2=0x7f1a240caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 --2- 172.21.15.29:0/2953860830 >> v2:172.21.15.170:3300/0 conn(0x7f1a2415d110 0x7f1a240caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a28a5d640 1 -- 172.21.15.29:0/2953860830 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a24161530 con 0x7f1a24160ce0 2023-10-16T15:55:59.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.290+0000 7f1a2925e640 1 --2- 172.21.15.29:0/2953860830 >> v2:172.21.15.170:3300/0 conn(0x7f1a2415d110 0x7f1a240caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:59.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.291+0000 7f1a28a5d640 1 --2- 172.21.15.29:0/2953860830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406c5d0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f1a240cd5f0 tx=0x7f1a1402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2e5db40b5bf79a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:59.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a1bfff640 1 -- 172.21.15.29:0/2953860830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14031740 con 0x7f1a24160ce0 2023-10-16T15:55:59.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a1bfff640 1 -- 172.21.15.29:0/2953860830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a140318a0 con 0x7f1a24160ce0 2023-10-16T15:55:59.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a1bfff640 1 -- 172.21.15.29:0/2953860830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14031c30 con 0x7f1a24160ce0 2023-10-16T15:55:59.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/2953860830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 msgr2=0x7f1a2406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/2953860830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406c5d0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f1a240cd5f0 tx=0x7f1a1402b7e0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/2953860830 shutdown_connections 2023-10-16T15:55:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/2953860830 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a2406a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/2953860830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406c5d0 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/2953860830 >> v2:172.21.15.170:3300/0 conn(0x7f1a2415d110 0x7f1a240caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.292+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/2953860830 >> 172.21.15.29:0/2953860830 conn(0x7f1a240b1560 msgr2=0x7f1a240b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:59.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.293+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/2953860830 shutdown_connections 2023-10-16T15:55:59.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.293+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/2953860830 wait complete. 2023-10-16T15:55:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.294+0000 7f1a2b4c2640 1 Processor -- start 2023-10-16T15:55:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.294+0000 7f1a2b4c2640 1 -- start start 2023-10-16T15:55:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.295+0000 7f1a2b4c2640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a24155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.295+0000 7f1a2b4c2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2415d110 0x7f1a2415f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.295+0000 7f1a2b4c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.295+0000 7f1a2b4c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a241564b0 con 0x7f1a24160ce0 2023-10-16T15:55:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.295+0000 7f1a2b4c2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2415e1e0 con 0x7f1a2415d110 2023-10-16T15:55:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.295+0000 7f1a2b4c2640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2415e4c0 con 0x7f1a24069da0 2023-10-16T15:55:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42892/0 (socket says 172.21.15.29:42892) 2023-10-16T15:55:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 -- 172.21.15.29:0/1942661615 learned_addr learned my addr 172.21.15.29:0/1942661615 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a2925e640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a24155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a28a5d640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2415d110 0x7f1a2415f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 -- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 msgr2=0x7f1a24155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a24155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 -- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2415d110 msgr2=0x7f1a2415f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2415d110 0x7f1a2415f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a29a5f640 1 -- 172.21.15.29:0/1942661615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a140094d0 con 0x7f1a24160ce0 2023-10-16T15:55:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.296+0000 7f1a28a5d640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2415d110 0x7f1a2415f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.297+0000 7f1a29a5f640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406def0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c007d20 tx=0x7f1a1c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.297+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c014420 con 0x7f1a24160ce0 2023-10-16T15:55:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.297+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c0145a0 con 0x7f1a24160ce0 2023-10-16T15:55:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.297+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c014950 con 0x7f1a24160ce0 2023-10-16T15:55:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.297+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2415e7a0 con 0x7f1a24160ce0 2023-10-16T15:55:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.297+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2415ed60 con 0x7f1a24160ce0 2023-10-16T15:55:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.298+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a240cf4a0 con 0x7f1a24160ce0 2023-10-16T15:55:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.299+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f1a1c014ab0 con 0x7f1a24160ce0 2023-10-16T15:55:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.300+0000 7f1a19ffb640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 0x7f1a0806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.300+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f1a1c08a420 con 0x7f1a24160ce0 2023-10-16T15:55:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.300+0000 7f1a2925e640 1 -- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 msgr2=0x7f1a0806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:55:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.300+0000 7f1a2925e640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 0x7f1a0806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.308+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a1c055040 con 0x7f1a24160ce0 2023-10-16T15:55:59.322 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:59.321+0000 7f56c76b0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T15:55:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.501+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a240b59f0 con 0x7f1a24160ce0 2023-10-16T15:55:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.502+0000 7f1a2925e640 1 -- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 msgr2=0x7f1a0806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:55:59.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.502+0000 7f1a2925e640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 0x7f1a0806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:55:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.503+0000 7f1a19ffb640 1 -- 172.21.15.29:0/1942661615 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+17267 (secure 0 0 0) 0x7f1a1c058660 con 0x7f1a24160ce0 2023-10-16T15:55:59.505 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:55:59.505 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":267,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:54.590749+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2485822714":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/831181775":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/1047565771":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2742286180":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3423446827":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3865201075":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6833/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/763385128":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2667017960":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:55:59.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 msgr2=0x7f1a0806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:59.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 0x7f1a0806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 msgr2=0x7f1a2406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406def0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c007d20 tx=0x7f1a1c00e4b0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 shutdown_connections 2023-10-16T15:55:59.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f1a0806b020 0x7f1a0806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a24160ce0 0x7f1a2406def0 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1a2415d110 0x7f1a2415f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 --2- 172.21.15.29:0/1942661615 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1a24069da0 0x7f1a24155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 >> 172.21.15.29:0/1942661615 conn(0x7f1a240b1560 msgr2=0x7f1a240afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 shutdown_connections 2023-10-16T15:55:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.506+0000 7f1a2b4c2640 1 -- 172.21.15.29:0/1942661615 wait complete. 2023-10-16T15:55:59.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:55:37.838394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:55:38.855075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '266', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:55:59.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:55:59.711 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.709+0000 7f577862e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:59.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.736+0000 7f8903961640 1 Processor -- start 2023-10-16T15:55:59.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.736+0000 7f8903961640 1 -- start start 2023-10-16T15:55:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.738+0000 7f8903961640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f88fc0ae0d0 0x7f88fc0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.738+0000 7f8903961640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc15b2c0 con 0x7f88fc15aee0 2023-10-16T15:55:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.738+0000 7f8903961640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc0ae5d0 con 0x7f88fc0ce820 2023-10-16T15:55:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.738+0000 7f8903961640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc0ae8b0 con 0x7f88fc0ae0d0 2023-10-16T15:55:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.738+0000 7f89016fd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f88fc15aee0 0x7f88fc0b3920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47734/0 (socket says 172.21.15.29:47734) 2023-10-16T15:55:59.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.738+0000 7f89016fd640 1 -- 172.21.15.29:0/3030557956 learned_addr learned my addr 172.21.15.29:0/3030557956 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:55:59.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915814460 0 0) 0x7f88fc15b2c0 con 0x7f88fc15aee0 2023-10-16T15:55:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f8900efc640 1 --2- 172.21.15.29:0/3030557956 >> v2:172.21.15.170:3300/0 conn(0x7f88fc0ae0d0 0x7f88fc0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e8003520 con 0x7f88fc15aee0 2023-10-16T15:55:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877741704 0 0) 0x7f88e8003520 con 0x7f88fc15aee0 2023-10-16T15:55:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 >> v2:172.21.15.170:3300/0 conn(0x7f88fc0ae0d0 msgr2=0x7f88fc0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:59.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 --2- 172.21.15.29:0/3030557956 >> v2:172.21.15.170:3300/0 conn(0x7f88fc0ae0d0 0x7f88fc0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 >> v1:172.21.15.170:6789/0 conn(0x7f88fc0ce820 legacy=0x7f88fc0af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0aeb90 con 0x7f88fc15aee0 2023-10-16T15:55:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f88f80017f0 con 0x7f88fc15aee0 2023-10-16T15:55:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 >> v1:172.21.15.29:6789/0 conn(0x7f88fc15aee0 legacy=0x7f88fc0b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88e80047c0 0x7f88e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88e80051d0 0x7f88e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc0ae8b0 con 0x7f88e80039c0 2023-10-16T15:55:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e800c0e0 con 0x7f88e80051d0 2023-10-16T15:55:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.739+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e800c2a0 con 0x7f88e80047c0 2023-10-16T15:55:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f89016fd640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f8901efe640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88e80051d0 0x7f88e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f8900efc640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88e80047c0 0x7f88e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f89016fd640 1 -- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88e80047c0 msgr2=0x7f88e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f89016fd640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88e80047c0 0x7f88e8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f89016fd640 1 -- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88e80051d0 msgr2=0x7f88e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:55:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f89016fd640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88e80051d0 0x7f88e800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f89016fd640 1 -- 172.21.15.29:0/3030557956 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e8003520 con 0x7f88e80039c0 2023-10-16T15:55:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f8901efe640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88e80051d0 0x7f88e800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.740+0000 7f8901efe640 1 -- 172.21.15.29:0/3030557956 reap_dead start 2023-10-16T15:55:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.741+0000 7f89016fd640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88e8003dc0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c0e0 tx=0x7f88e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f761497a4e8b35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.741+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8007270 con 0x7f88e80039c0 2023-10-16T15:55:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f80073d0 con 0x7f88e80039c0 2023-10-16T15:55:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f88effff640 1 -- 172.21.15.29:0/3030557956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f802c810 con 0x7f88e80039c0 2023-10-16T15:55:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 -- 172.21.15.29:0/3030557956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 msgr2=0x7f88e8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88e8003dc0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c0e0 tx=0x7f88e80057f0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 -- 172.21.15.29:0/3030557956 shutdown_connections 2023-10-16T15:55:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 --2- 172.21.15.29:0/3030557956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88e8003dc0 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 -- 172.21.15.29:0/3030557956 >> 172.21.15.29:0/3030557956 conn(0x7f88fc0d0ad0 msgr2=0x7f88fc0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 -- 172.21.15.29:0/3030557956 shutdown_connections 2023-10-16T15:55:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.742+0000 7f8903961640 1 -- 172.21.15.29:0/3030557956 wait complete. 2023-10-16T15:55:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.744+0000 7f8903961640 1 Processor -- start 2023-10-16T15:55:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.744+0000 7f8903961640 1 -- start start 2023-10-16T15:55:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f8903961640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88fc15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f8903961640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88fc0ca750 0x7f88fc1556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f8903961640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88fc155c10 0x7f88fc1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f8903961640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc15c990 con 0x7f88e80039c0 2023-10-16T15:55:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f8903961640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc1f1a80 con 0x7f88fc0ca750 2023-10-16T15:55:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f8903961640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc1ee0e0 con 0x7f88fc155c10 2023-10-16T15:55:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f89016fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88fc15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f89016fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88fc15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42908/0 (socket says 172.21.15.29:42908) 2023-10-16T15:55:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.745+0000 7f89016fd640 1 -- 172.21.15.29:0/941213641 learned_addr learned my addr 172.21.15.29:0/941213641 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:55:59.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f8901efe640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88fc155c10 0x7f88fc1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f8900efc640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88fc0ca750 0x7f88fc1556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:55:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f89016fd640 1 -- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88fc155c10 msgr2=0x7f88fc1f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f89016fd640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88fc155c10 0x7f88fc1f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f89016fd640 1 -- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88fc0ca750 msgr2=0x7f88fc1556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f89016fd640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88fc0ca750 0x7f88fc1556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f89016fd640 1 -- 172.21.15.29:0/941213641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0b0f30 con 0x7f88e80039c0 2023-10-16T15:55:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f8900efc640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88fc0ca750 0x7f88fc1556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:55:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.746+0000 7f89016fd640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88fc15c450 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f88f802c3b0 tx=0x7f88f802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:55:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.747+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f803c040 con 0x7f88e80039c0 2023-10-16T15:55:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.747+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f802cc20 con 0x7f88e80039c0 2023-10-16T15:55:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.747+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80057a0 con 0x7f88e80039c0 2023-10-16T15:55:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.747+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc0b1210 con 0x7f88e80039c0 2023-10-16T15:55:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.747+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc1f2450 con 0x7f88e80039c0 2023-10-16T15:55:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.748+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc1f1850 con 0x7f88e80039c0 2023-10-16T15:55:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.749+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f88f8032070 con 0x7f88e80039c0 2023-10-16T15:55:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.750+0000 7f88edffb640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f88e006b020 0x7f88e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:55:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.750+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f88f80aaf80 con 0x7f88e80039c0 2023-10-16T15:55:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.750+0000 7f8900efc640 1 -- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f88e006b020 msgr2=0x7f88e006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:55:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.750+0000 7f8900efc640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f88e006b020 0x7f88e006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:55:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.754+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88f80ddca0 con 0x7f88e80039c0 2023-10-16T15:55:59.763 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.761+0000 7f577862e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:59.807 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:59.806+0000 7f56c76b0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T15:55:59.812 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.811+0000 7f577862e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:59.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:59.861+0000 7f56c76b0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T15:55:59.872 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.870+0000 7f577862e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:55:59.915 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:59.914+0000 7f56c76b0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T15:55:59.921 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:55:59.919+0000 7f577862e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:55:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.946+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88fc04e160 con 0x7f88e80039c0 2023-10-16T15:55:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.946+0000 7f88edffb640 1 -- 172.21.15.29:0/941213641 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v267) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88fc1f1850 con 0x7f88e80039c0 2023-10-16T15:55:59.947 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:55:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f88e006b020 msgr2=0x7f88e006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:55:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f88e006b020 0x7f88e006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 msgr2=0x7f88fc15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:55:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88fc15c450 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f88f802c3b0 tx=0x7f88f802e070 comp rx=0 tx=0).stop 2023-10-16T15:55:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 shutdown_connections 2023-10-16T15:55:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f88e006b020 0x7f88e006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f88fc155c10 0x7f88fc1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88fc0ca750 0x7f88fc1556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 --2- 172.21.15.29:0/941213641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88e80039c0 0x7f88fc15c450 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:55:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 >> 172.21.15.29:0/941213641 conn(0x7f88fc0d0ad0 msgr2=0x7f88fc0b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:55:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 shutdown_connections 2023-10-16T15:55:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:55:59.949+0000 7f8903961640 1 -- 172.21.15.29:0/941213641 wait complete. 2023-10-16T15:55:59.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:55:59.983 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:55:59.982+0000 7f56c76b0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T15:56:00.030 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.028+0000 7f577862e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:56:00.035 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.035+0000 7f56c76b0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T15:56:00.091 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.089+0000 7f577862e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:56:00.140 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.139+0000 7f577862e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:56:00.154 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.153+0000 7f56c76b0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T15:56:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.183+0000 7fadc8831640 1 Processor -- start 2023-10-16T15:56:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.184+0000 7fadc8831640 1 -- start start 2023-10-16T15:56:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc8831640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fadc40cded0 0x7fadc40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc8831640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc415d150 con 0x7fadc40b8480 2023-10-16T15:56:00.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc8831640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc40d1180 con 0x7fadc415cd70 2023-10-16T15:56:00.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc8831640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc40d1460 con 0x7fadc40cded0 2023-10-16T15:56:00.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fadc40b8480 0x7fadc40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47742/0 (socket says 172.21.15.29:47742) 2023-10-16T15:56:00.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc259c640 1 -- 172.21.15.29:0/3856120895 learned_addr learned my addr 172.21.15.29:0/3856120895 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:00.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.185+0000 7fadc359e640 1 --2- 172.21.15.29:0/3856120895 >> v2:172.21.15.29:3300/0 conn(0x7fadc40cded0 0x7fadc40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.186+0000 7fadc1d9b640 1 -- 172.21.15.29:0/3856120895 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331273931 0 0) 0x7fadc415d150 con 0x7fadc40b8480 2023-10-16T15:56:00.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.186+0000 7fadc1d9b640 1 -- 172.21.15.29:0/3856120895 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac003520 con 0x7fadc40b8480 2023-10-16T15:56:00.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.186+0000 7fadc359e640 1 -- 172.21.15.29:0/3856120895 >> v1:172.21.15.170:6789/0 conn(0x7fadc415cd70 legacy=0x7fadc40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.186+0000 7fadc359e640 1 -- 172.21.15.29:0/3856120895 >> v1:172.21.15.29:6789/0 conn(0x7fadc40b8480 legacy=0x7fadc40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.186+0000 7fadc359e640 1 -- 172.21.15.29:0/3856120895 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc40d1740 con 0x7fadc40cded0 2023-10-16T15:56:00.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.188+0000 7fadc359e640 1 --2- 172.21.15.29:0/3856120895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40cded0 0x7fadc40b5c90 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009d90 tx=0x7fadbc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bef5cb34fa1672a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:00.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.188+0000 7fadc1d9b640 1 -- 172.21.15.29:0/3856120895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc02cd00 con 0x7fadc40cded0 2023-10-16T15:56:00.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.188+0000 7fadc1d9b640 1 -- 172.21.15.29:0/3856120895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc02ce60 con 0x7fadc40cded0 2023-10-16T15:56:00.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.188+0000 7fadc1d9b640 1 -- 172.21.15.29:0/3856120895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc034640 con 0x7fadc40cded0 2023-10-16T15:56:00.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 -- 172.21.15.29:0/3856120895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40cded0 msgr2=0x7fadc40b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 --2- 172.21.15.29:0/3856120895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40cded0 0x7fadc40b5c90 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009d90 tx=0x7fadbc02bf50 comp rx=0 tx=0).stop 2023-10-16T15:56:00.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 -- 172.21.15.29:0/3856120895 shutdown_connections 2023-10-16T15:56:00.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 --2- 172.21.15.29:0/3856120895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40cded0 0x7fadc40b5c90 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 -- 172.21.15.29:0/3856120895 >> 172.21.15.29:0/3856120895 conn(0x7fadc40cf570 msgr2=0x7fadc40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:00.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 -- 172.21.15.29:0/3856120895 shutdown_connections 2023-10-16T15:56:00.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.189+0000 7fadc8831640 1 -- 172.21.15.29:0/3856120895 wait complete. 2023-10-16T15:56:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.191+0000 7fadc8831640 1 Processor -- start 2023-10-16T15:56:00.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.191+0000 7fadc8831640 1 -- start start 2023-10-16T15:56:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc8831640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 0x7fadc415ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc8831640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc40cded0 0x7fadc41f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc8831640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc415cd70 0x7fadc41e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc8831640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc41f56f0 con 0x7fadc40b8480 2023-10-16T15:56:00.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc8831640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc41e6810 con 0x7fadc415cd70 2023-10-16T15:56:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc8831640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc41e6af0 con 0x7fadc40cded0 2023-10-16T15:56:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 0x7fadc415ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 0x7fadc415ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42930/0 (socket says 172.21.15.29:42930) 2023-10-16T15:56:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 -- 172.21.15.29:0/1346731665 learned_addr learned my addr 172.21.15.29:0/1346731665 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:00.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc259c640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc40cded0 0x7fadc41f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 -- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc40cded0 msgr2=0x7fadc41f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc40cded0 0x7fadc41f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 -- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc415cd70 msgr2=0x7fadc41e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc415cd70 0x7fadc41e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.192+0000 7fadc2d9d640 1 -- 172.21.15.29:0/1346731665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc009a70 con 0x7fadc40b8480 2023-10-16T15:56:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.193+0000 7fadc2d9d640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 0x7fadc415ffb0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fadb400c400 tx=0x7fadb400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.193+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb40052d0 con 0x7fadc40b8480 2023-10-16T15:56:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.193+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb4005430 con 0x7fadc40b8480 2023-10-16T15:56:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.193+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4003830 con 0x7fadc40b8480 2023-10-16T15:56:00.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.193+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc41e6dd0 con 0x7fadc40b8480 2023-10-16T15:56:00.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.193+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc40c9ea0 con 0x7fadc40b8480 2023-10-16T15:56:00.199 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.196+0000 7f577862e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:56:00.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.195+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc40742a0 con 0x7fadc40b8480 2023-10-16T15:56:00.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.195+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fadb4003990 con 0x7fadc40b8480 2023-10-16T15:56:00.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.196+0000 7fadab7fe640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 0x7fadb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.196+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fadb4088ff0 con 0x7fadc40b8480 2023-10-16T15:56:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.196+0000 7fadc259c640 1 -- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 msgr2=0x7fadb006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.196+0000 7fadc259c640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 0x7fadb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:56:00.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.200+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadb4053c70 con 0x7fadc40b8480 2023-10-16T15:56:00.240 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.239+0000 7f56c76b0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T15:56:00.248 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.246+0000 7f577862e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:56:00.304 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.303+0000 7f56c76b0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T15:56:00.366 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.365+0000 7f56c76b0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T15:56:00.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.397+0000 7fadc259c640 1 -- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 msgr2=0x7fadb006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:00.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.397+0000 7fadc259c640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 0x7fadb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:56:00.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.421+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fadc404e9b0 con 0x7fadc40b8480 2023-10-16T15:56:00.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.422+0000 7fadab7fe640 1 -- 172.21.15.29:0/1346731665 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v267) v1 ==== 104+0+11 (secure 0 0 0) 0x7fadb4057290 con 0x7fadc40b8480 2023-10-16T15:56:00.423 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:56:00.424 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.423+0000 7f56c76b0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T15:56:00.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 msgr2=0x7fadb006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:56:00.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 0x7fadb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 msgr2=0x7fadc415ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 0x7fadc415ffb0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fadb400c400 tx=0x7fadb400c8c0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 shutdown_connections 2023-10-16T15:56:00.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fadb006b020 0x7fadb006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fadc415cd70 0x7fadc41e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fadc40cded0 0x7fadc41f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 --2- 172.21.15.29:0/1346731665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc40b8480 0x7fadc415ffb0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.424+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 >> 172.21.15.29:0/1346731665 conn(0x7fadc40cf570 msgr2=0x7fadc40b7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:00.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.425+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 shutdown_connections 2023-10-16T15:56:00.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.425+0000 7fadc8831640 1 -- 172.21.15.29:0/1346731665 wait complete. 2023-10-16T15:56:00.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:56:00.627 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.625+0000 7f577862e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:56:00.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.697+0000 7f386e810640 1 Processor -- start 2023-10-16T15:56:00.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.697+0000 7f386e810640 1 -- start start 2023-10-16T15:56:00.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.698+0000 7f386e810640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f386815d330 0x7f38681618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.698+0000 7f386e810640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38680d0890 con 0x7f386806c7d0 2023-10-16T15:56:00.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f386e810640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38680d05b0 con 0x7f38680661c0 2023-10-16T15:56:00.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f386e810640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386806b1e0 con 0x7f386815d330 2023-10-16T15:56:00.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f3867fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f386815d330 0x7f38681618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f3867fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f386815d330 0x7f38681618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42934/0 (socket says 172.21.15.29:42934) 2023-10-16T15:56:00.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f3867fff640 1 -- 172.21.15.29:0/4236895872 learned_addr learned my addr 172.21.15.29:0/4236895872 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:00.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f3867fff640 1 -- 172.21.15.29:0/4236895872 >> v1:172.21.15.170:6789/0 conn(0x7f38680661c0 legacy=0x7f38680ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:00.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f3867fff640 1 -- 172.21.15.29:0/4236895872 >> v1:172.21.15.29:6789/0 conn(0x7f386806c7d0 legacy=0x7f3868065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.699+0000 7f3867fff640 1 -- 172.21.15.29:0/4236895872 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868160dd0 con 0x7f386815d330 2023-10-16T15:56:00.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.700+0000 7f3867fff640 1 --2- 172.21.15.29:0/4236895872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386815d330 0x7f38681618b0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f3858009870 tx=0x7f385802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610a634a5dd13d2c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:00.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.701+0000 7f3866ffd640 1 -- 172.21.15.29:0/4236895872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3858031870 con 0x7f386815d330 2023-10-16T15:56:00.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.701+0000 7f3866ffd640 1 -- 172.21.15.29:0/4236895872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38580319d0 con 0x7f386815d330 2023-10-16T15:56:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.701+0000 7f3866ffd640 1 -- 172.21.15.29:0/4236895872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3858031d60 con 0x7f386815d330 2023-10-16T15:56:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.701+0000 7f386e810640 1 -- 172.21.15.29:0/4236895872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386815d330 msgr2=0x7f38681618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.701+0000 7f386e810640 1 --2- 172.21.15.29:0/4236895872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386815d330 0x7f38681618b0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f3858009870 tx=0x7f385802b910 comp rx=0 tx=0).stop 2023-10-16T15:56:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.702+0000 7f386e810640 1 -- 172.21.15.29:0/4236895872 shutdown_connections 2023-10-16T15:56:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.702+0000 7f386e810640 1 --2- 172.21.15.29:0/4236895872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386815d330 0x7f38681618b0 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.702+0000 7f386e810640 1 -- 172.21.15.29:0/4236895872 >> 172.21.15.29:0/4236895872 conn(0x7f38680d0ab0 msgr2=0x7f38680d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:00.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.702+0000 7f386e810640 1 -- 172.21.15.29:0/4236895872 shutdown_connections 2023-10-16T15:56:00.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.702+0000 7f386e810640 1 -- 172.21.15.29:0/4236895872 wait complete. 2023-10-16T15:56:00.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.703+0000 7f386e810640 1 Processor -- start 2023-10-16T15:56:00.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 -- start start 2023-10-16T15:56:00.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38680661c0 0x7f3868157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 0x7f38680b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f386815d330 0x7f38680b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386815be80 con 0x7f386806c7d0 2023-10-16T15:56:00.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38680ad5b0 con 0x7f38680661c0 2023-10-16T15:56:00.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.704+0000 7f386e810640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386815b270 con 0x7f386815d330 2023-10-16T15:56:00.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 0x7f38680b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 0x7f38680b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42950/0 (socket says 172.21.15.29:42950) 2023-10-16T15:56:00.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 -- 172.21.15.29:0/434973447 learned_addr learned my addr 172.21.15.29:0/434973447 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:00.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f3867fff640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38680661c0 0x7f3868157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f386cdad640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f386815d330 0x7f38680b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:00.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 -- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f386815d330 msgr2=0x7f38680b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f386815d330 0x7f38680b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 -- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38680661c0 msgr2=0x7f3868157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38680661c0 0x7f3868157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.705+0000 7f38677fe640 1 -- 172.21.15.29:0/434973447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38580094d0 con 0x7f386806c7d0 2023-10-16T15:56:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.706+0000 7f38677fe640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 0x7f38680b1100 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f385c00c480 tx=0x7f385c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.706+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c0052d0 con 0x7f386806c7d0 2023-10-16T15:56:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.706+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c005430 con 0x7f386806c7d0 2023-10-16T15:56:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.706+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c012870 con 0x7f386806c7d0 2023-10-16T15:56:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.706+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386815b550 con 0x7f386806c7d0 2023-10-16T15:56:00.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.706+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386815bb10 con 0x7f386806c7d0 2023-10-16T15:56:00.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.707+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386804e160 con 0x7f386806c7d0 2023-10-16T15:56:00.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.709+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f385c0129d0 con 0x7f386806c7d0 2023-10-16T15:56:00.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.710+0000 7f3864ff9640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 0x7f385406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:00.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.710+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f385c0896e0 con 0x7f386806c7d0 2023-10-16T15:56:00.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.710+0000 7f3867fff640 1 -- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 msgr2=0x7f385406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:00.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.710+0000 7f3867fff640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 0x7f385406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:56:00.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.714+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f385c054300 con 0x7f386806c7d0 2023-10-16T15:56:00.787 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.785+0000 7f577862e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:56:00.846 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:00.845+0000 7f56c76b0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T15:56:00.893 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.892+0000 7f577862e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:56:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.909+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f386806f3a0 con 0x7f386806c7d0 2023-10-16T15:56:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.909+0000 7f3864ff9640 1 -- 172.21.15.29:0/434973447 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v267) v1 ==== 100+0+11 (secure 0 0 0) 0x7f385c057920 con 0x7f386806c7d0 2023-10-16T15:56:00.911 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:56:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.911+0000 7f3867fff640 1 -- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 msgr2=0x7f385406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.911+0000 7f3867fff640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 0x7f385406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:56:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 msgr2=0x7f385406d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:56:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 0x7f385406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 msgr2=0x7f38680b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 0x7f38680b1100 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f385c00c480 tx=0x7f385c00c940 comp rx=0 tx=0).stop 2023-10-16T15:56:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 shutdown_connections 2023-10-16T15:56:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f385406b050 0x7f385406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f386806c7d0 0x7f38680b1100 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f386815d330 0x7f38680b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 --2- 172.21.15.29:0/434973447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f38680661c0 0x7f3868157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 >> 172.21.15.29:0/434973447 conn(0x7f38680d0ab0 msgr2=0x7f3868157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 shutdown_connections 2023-10-16T15:56:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:00.912+0000 7f386e810640 1 -- 172.21.15.29:0/434973447 wait complete. 2023-10-16T15:56:00.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T15:56:00.942 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:00.941+0000 7f577862e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:56:01.016 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.016+0000 7f56c76b0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T15:56:01.049 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:01.047+0000 7f577862e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:56:01.132 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.131+0000 7f56c76b0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T15:56:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.138+0000 7f63d0765640 1 Processor -- start 2023-10-16T15:56:01.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.139+0000 7f63d0765640 1 -- start start 2023-10-16T15:56:01.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63d0765640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c815d330 0x7f63c81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63d0765640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f63c80661c0 0x7f63c80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63d0765640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80d0890 con 0x7f63c806c7d0 2023-10-16T15:56:01.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63d0765640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c806b1e0 con 0x7f63c815d330 2023-10-16T15:56:01.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63d0765640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c80d05b0 con 0x7f63c80661c0 2023-10-16T15:56:01.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63ce501640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c815d330 0x7f63c81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63ce501640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c815d330 0x7f63c81618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42956/0 (socket says 172.21.15.29:42956) 2023-10-16T15:56:01.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.140+0000 7f63ce501640 1 -- 172.21.15.29:0/294471483 learned_addr learned my addr 172.21.15.29:0/294471483 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:01.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.141+0000 7f63ce501640 1 -- 172.21.15.29:0/294471483 >> v2:172.21.15.170:3300/0 conn(0x7f63c80661c0 msgr2=0x7f63c80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:01.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.141+0000 7f63ce501640 1 --2- 172.21.15.29:0/294471483 >> v2:172.21.15.170:3300/0 conn(0x7f63c80661c0 0x7f63c80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.141+0000 7f63ce501640 1 -- 172.21.15.29:0/294471483 >> v1:172.21.15.170:6789/0 conn(0x7f63c806c7d0 legacy=0x7f63c8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:01.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.141+0000 7f63ce501640 1 -- 172.21.15.29:0/294471483 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c8160dd0 con 0x7f63c815d330 2023-10-16T15:56:01.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.142+0000 7f63ce501640 1 --2- 172.21.15.29:0/294471483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c81618b0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f63c806f340 tx=0x7f63c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54487bc7747791d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:01.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.142+0000 7f63cd4ff640 1 -- 172.21.15.29:0/294471483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c40317e0 con 0x7f63c815d330 2023-10-16T15:56:01.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.142+0000 7f63cd4ff640 1 -- 172.21.15.29:0/294471483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4031940 con 0x7f63c815d330 2023-10-16T15:56:01.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.142+0000 7f63cd4ff640 1 -- 172.21.15.29:0/294471483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4031cd0 con 0x7f63c815d330 2023-10-16T15:56:01.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 -- 172.21.15.29:0/294471483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 msgr2=0x7f63c81618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 --2- 172.21.15.29:0/294471483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c81618b0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f63c806f340 tx=0x7f63c402b880 comp rx=0 tx=0).stop 2023-10-16T15:56:01.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 -- 172.21.15.29:0/294471483 shutdown_connections 2023-10-16T15:56:01.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 --2- 172.21.15.29:0/294471483 >> v2:172.21.15.170:3300/0 conn(0x7f63c80661c0 0x7f63c80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 --2- 172.21.15.29:0/294471483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c81618b0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 -- 172.21.15.29:0/294471483 >> 172.21.15.29:0/294471483 conn(0x7f63c80d0ab0 msgr2=0x7f63c80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:01.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 -- 172.21.15.29:0/294471483 shutdown_connections 2023-10-16T15:56:01.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.143+0000 7f63d0765640 1 -- 172.21.15.29:0/294471483 wait complete. 2023-10-16T15:56:01.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.145+0000 7f63d0765640 1 Processor -- start 2023-10-16T15:56:01.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.145+0000 7f63d0765640 1 -- start start 2023-10-16T15:56:01.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63d0765640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63c80661c0 0x7f63c80c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63d0765640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63c806c7d0 0x7f63c8155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63d0765640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c80c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63d0765640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80c6740 con 0x7f63c815d330 2023-10-16T15:56:01.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63d0765640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8155540 con 0x7f63c806c7d0 2023-10-16T15:56:01.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63d0765640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80c8030 con 0x7f63c80661c0 2023-10-16T15:56:01.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63ced02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c80c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63ced02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c80c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42972/0 (socket says 172.21.15.29:42972) 2023-10-16T15:56:01.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63ced02640 1 -- 172.21.15.29:0/2075747093 learned_addr learned my addr 172.21.15.29:0/2075747093 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:01.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63ce501640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63c80661c0 0x7f63c80c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.146+0000 7f63cdd00640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63c806c7d0 0x7f63c8155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63ced02640 1 -- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63c80661c0 msgr2=0x7f63c80c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63ced02640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63c80661c0 0x7f63c80c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63ced02640 1 -- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63c806c7d0 msgr2=0x7f63c8155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63ced02640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63c806c7d0 0x7f63c8155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63ced02640 1 -- 172.21.15.29:0/2075747093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40094d0 con 0x7f63c815d330 2023-10-16T15:56:01.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63ced02640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c80c7af0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b9a0 tx=0x7f63c000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:01.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000cba0 con 0x7f63c815d330 2023-10-16T15:56:01.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c000cd20 con 0x7f63c815d330 2023-10-16T15:56:01.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c80c6930 con 0x7f63c815d330 2023-10-16T15:56:01.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0014680 con 0x7f63c815d330 2023-10-16T15:56:01.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.147+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c80adfc0 con 0x7f63c815d330 2023-10-16T15:56:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.149+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c804e160 con 0x7f63c815d330 2023-10-16T15:56:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.151+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f63c001d3b0 con 0x7f63c815d330 2023-10-16T15:56:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.152+0000 7f63beffd640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 0x7f63ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.152+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f63c0089aa0 con 0x7f63c815d330 2023-10-16T15:56:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.152+0000 7f63ce501640 1 -- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 msgr2=0x7f63ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:01.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.152+0000 7f63ce501640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 0x7f63ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:56:01.157 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:01.155+0000 7f577862e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:56:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.156+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63c00546d0 con 0x7f63c815d330 2023-10-16T15:56:01.187 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.187+0000 7f56c76b0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T15:56:01.302 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.301+0000 7f56c76b0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:56:01.312 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:01.310+0000 7f577862e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:56:01.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.353+0000 7f63ce501640 1 -- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 msgr2=0x7f63ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:01.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.353+0000 7f63ce501640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 0x7f63ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:56:01.360 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:01.359+0000 7f577862e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:56:01.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.368+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f63c8155690 con 0x7f63c815d330 2023-10-16T15:56:01.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.368+0000 7f63beffd640 1 -- 172.21.15.29:0/2075747093 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 11 entries v267) v1 ==== 81+0+653 (secure 0 0 0) 0x7f63c0019020 con 0x7f63c815d330 2023-10-16T15:56:01.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2485822714 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/831181775 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/1047565771 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6832/3469082932 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.370 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2742286180 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.371 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3423446827 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.371 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3865201075 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.371 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:6833/3469082932 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.371 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/763385128 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.371 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/2667017960 2023-10-17T15:55:54.590651+0000 2023-10-16T15:56:01.372 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.170:0/3985010689 2023-10-16T16:55:37.652115+0000 2023-10-16T15:56:01.372 INFO:teuthology.orchestra.run.smithi029.stderr:listed 11 entries 2023-10-16T15:56:01.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 msgr2=0x7f63ac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:56:01.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 0x7f63ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 msgr2=0x7f63c80c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c80c7af0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b9a0 tx=0x7f63c000be60 comp rx=0 tx=0).stop 2023-10-16T15:56:01.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 shutdown_connections 2023-10-16T15:56:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f63ac06b020 0x7f63ac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c815d330 0x7f63c80c7af0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63c806c7d0 0x7f63c8155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 --2- 172.21.15.29:0/2075747093 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f63c80661c0 0x7f63c80c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 >> 172.21.15.29:0/2075747093 conn(0x7f63c80d0ab0 msgr2=0x7f63c815f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.371+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 shutdown_connections 2023-10-16T15:56:01.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.372+0000 7f63d0765640 1 -- 172.21.15.29:0/2075747093 wait complete. 2023-10-16T15:56:01.391 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T15:56:01.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:01.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T15:56:01.418 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.417+0000 7f56c76b0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T15:56:01.479 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:01.477+0000 7f577862e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:56:01.495 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:56:01.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:01.496 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T15:56:01.587 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.586+0000 7f56c76b0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T15:56:01.589 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:56:01.589 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:01.640 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.639+0000 7f56c76b0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T15:56:01.646 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:56:01.646 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:56:01.646 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:01.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:56:01.703 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:56:01.704 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:01.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:56:01.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:56:01.767 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:01.766+0000 7f56c76b0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T15:56:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.979+0000 7fde9937c640 1 Processor -- start 2023-10-16T15:56:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.979+0000 7fde9937c640 1 -- start start 2023-10-16T15:56:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde9937c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde9406a4b0 0x7fde940cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde9937c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde94160f30 0x7fde940cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde9937c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde941614f0 con 0x7fde940ae220 2023-10-16T15:56:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde9937c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde940cf760 con 0x7fde9406a4b0 2023-10-16T15:56:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde9937c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde941617d0 con 0x7fde94160f30 2023-10-16T15:56:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde8a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde9406a4b0 0x7fde940cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde92ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fde940ae220 0x7fde94069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47764/0 (socket says 172.21.15.29:47764) 2023-10-16T15:56:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde92ffd640 1 -- 172.21.15.29:0/3513479370 learned_addr learned my addr 172.21.15.29:0/3513479370 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.981+0000 7fde8a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde9406a4b0 0x7fde940cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42982/0 (socket says 172.21.15.29:42982) 2023-10-16T15:56:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.982+0000 7fde937fe640 1 --2- 172.21.15.29:0/3513479370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde94160f30 0x7fde940cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.982+0000 7fde8a7fc640 1 -- 172.21.15.29:0/3513479370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde94160f30 msgr2=0x7fde940cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.982+0000 7fde8a7fc640 1 --2- 172.21.15.29:0/3513479370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde94160f30 0x7fde940cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.982+0000 7fde8a7fc640 1 -- 172.21.15.29:0/3513479370 >> v1:172.21.15.29:6789/0 conn(0x7fde940ae220 legacy=0x7fde94069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.982+0000 7fde8a7fc640 1 -- 172.21.15.29:0/3513479370 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94161ab0 con 0x7fde9406a4b0 2023-10-16T15:56:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.983+0000 7fde8a7fc640 1 --2- 172.21.15.29:0/3513479370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde9406a4b0 0x7fde940cabd0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0097a0 tx=0x7fde7c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8b89b347ce1e53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.983+0000 7fde927fc640 1 -- 172.21.15.29:0/3513479370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c02d040 con 0x7fde9406a4b0 2023-10-16T15:56:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.983+0000 7fde927fc640 1 -- 172.21.15.29:0/3513479370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c031810 con 0x7fde9406a4b0 2023-10-16T15:56:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde927fc640 1 -- 172.21.15.29:0/3513479370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c031ba0 con 0x7fde9406a4b0 2023-10-16T15:56:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 -- 172.21.15.29:0/3513479370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde9406a4b0 msgr2=0x7fde940cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 --2- 172.21.15.29:0/3513479370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde9406a4b0 0x7fde940cabd0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0097a0 tx=0x7fde7c02b8b0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 -- 172.21.15.29:0/3513479370 shutdown_connections 2023-10-16T15:56:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 --2- 172.21.15.29:0/3513479370 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde94160f30 0x7fde940cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 --2- 172.21.15.29:0/3513479370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde9406a4b0 0x7fde940cabd0 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 -- 172.21.15.29:0/3513479370 >> 172.21.15.29:0/3513479370 conn(0x7fde940cfb70 msgr2=0x7fde940cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 -- 172.21.15.29:0/3513479370 shutdown_connections 2023-10-16T15:56:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.984+0000 7fde9937c640 1 -- 172.21.15.29:0/3513479370 wait complete. 2023-10-16T15:56:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.986+0000 7fde9937c640 1 Processor -- start 2023-10-16T15:56:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.986+0000 7fde9937c640 1 -- start start 2023-10-16T15:56:01.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde9937c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde9406a4b0 0x7fde941f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde9937c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde940ae220 0x7fde940b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde9937c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 0x7fde940acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde9937c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde941f5c40 con 0x7fde94160f30 2023-10-16T15:56:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde9937c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940b42b0 con 0x7fde940ae220 2023-10-16T15:56:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde9937c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde940b2b20 con 0x7fde9406a4b0 2023-10-16T15:56:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 0x7fde940acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 0x7fde940acfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42986/0 (socket says 172.21.15.29:42986) 2023-10-16T15:56:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde937fe640 1 -- 172.21.15.29:0/4044230792 learned_addr learned my addr 172.21.15.29:0/4044230792 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.987+0000 7fde92ffd640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde9406a4b0 0x7fde941f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde8a7fc640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde940ae220 0x7fde940b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde937fe640 1 -- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde9406a4b0 msgr2=0x7fde941f5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde937fe640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde9406a4b0 0x7fde941f5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde937fe640 1 -- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde940ae220 msgr2=0x7fde940b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde937fe640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde940ae220 0x7fde940b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde937fe640 1 -- 172.21.15.29:0/4044230792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c009400 con 0x7fde94160f30 2023-10-16T15:56:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde937fe640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 0x7fde940acfa0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00e930 tx=0x7fde8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c009690 con 0x7fde94160f30 2023-10-16T15:56:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c00c980 con 0x7fde94160f30 2023-10-16T15:56:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c00cd30 con 0x7fde94160f30 2023-10-16T15:56:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde940b2e00 con 0x7fde94160f30 2023-10-16T15:56:01.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.988+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde940b33c0 con 0x7fde94160f30 2023-10-16T15:56:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.990+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9404e160 con 0x7fde94160f30 2023-10-16T15:56:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.991+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fde8c013470 con 0x7fde94160f30 2023-10-16T15:56:01.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.992+0000 7fde8bfff640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 0x7fde7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:01.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.992+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fde8c089c00 con 0x7fde94160f30 2023-10-16T15:56:01.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.992+0000 7fde92ffd640 1 -- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 msgr2=0x7fde7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:01.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:01.992+0000 7fde92ffd640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 0x7fde7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:56:02.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.000+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde8c090080 con 0x7fde94160f30 2023-10-16T15:56:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.193+0000 7fde92ffd640 1 -- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 msgr2=0x7fde7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.193+0000 7fde92ffd640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 0x7fde7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:56:02.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.230+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde940b5af0 con 0x7fde94160f30 2023-10-16T15:56:02.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.231+0000 7fde8bfff640 1 -- 172.21.15.29:0/4044230792 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fde8c017090 con 0x7fde94160f30 2023-10-16T15:56:02.232 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:02.232 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:55:39.826547+0000","modified":"2023-10-16T15:55:42.906111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12345},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12345":{"gid":12345,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2425185007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2425185007},{"type":"v1","addr":"172.21.15.29:6835","nonce":2425185007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T15:56:02.232 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-16T15:56:02.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 msgr2=0x7fde7406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:56:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 0x7fde7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 msgr2=0x7fde940acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 0x7fde940acfa0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00e930 tx=0x7fde8c00edf0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 shutdown_connections 2023-10-16T15:56:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7fde7406b020 0x7fde7406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde94160f30 0x7fde940acfa0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde940ae220 0x7fde940b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 --2- 172.21.15.29:0/4044230792 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fde9406a4b0 0x7fde941f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 >> 172.21.15.29:0/4044230792 conn(0x7fde940cfb70 msgr2=0x7fde940c60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 shutdown_connections 2023-10-16T15:56:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.233+0000 7fde9937c640 1 -- 172.21.15.29:0/4044230792 wait complete. 2023-10-16T15:56:02.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:56:02.316 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.314+0000 7f577862e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:56:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.505+0000 7f330f905640 1 Processor -- start 2023-10-16T15:56:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.506+0000 7f330f905640 1 -- start start 2023-10-16T15:56:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330f905640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3308159b20 0x7f330806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330f905640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f33080b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330f905640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f330806c120 0x7f33080b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330f905640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330806f7f0 con 0x7f3308159b20 2023-10-16T15:56:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330f905640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330806c5e0 con 0x7f330806c120 2023-10-16T15:56:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330f905640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330806c8c0 con 0x7f3308159f00 2023-10-16T15:56:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.507+0000 7f330d6a1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3308159b20 0x7f330806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3308159b20 0x7f330806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42994/0 (socket says 172.21.15.29:42994) 2023-10-16T15:56:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 -- 172.21.15.29:0/2071766966 learned_addr learned my addr 172.21.15.29:0/2071766966 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330cea0640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f33080b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330dea2640 1 --2- 172.21.15.29:0/2071766966 >> v2:172.21.15.170:3300/0 conn(0x7f330806c120 0x7f33080b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 -- 172.21.15.29:0/2071766966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 msgr2=0x7f33080b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f33080b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 -- 172.21.15.29:0/2071766966 >> v2:172.21.15.170:3300/0 conn(0x7f330806c120 msgr2=0x7f33080b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 --2- 172.21.15.29:0/2071766966 >> v2:172.21.15.170:3300/0 conn(0x7f330806c120 0x7f33080b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330d6a1640 1 -- 172.21.15.29:0/2071766966 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330806cba0 con 0x7f3308159b20 2023-10-16T15:56:02.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.508+0000 7f330cea0640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f33080b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:56:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.509+0000 7f330d6a1640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f330806bbe0 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f33080c9270 tx=0x7f330402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d504cd050b039a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.509+0000 7f32fffff640 1 -- 172.21.15.29:0/2071766966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304031870 con 0x7f3308159b20 2023-10-16T15:56:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.509+0000 7f32fffff640 1 -- 172.21.15.29:0/2071766966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33040319d0 con 0x7f3308159b20 2023-10-16T15:56:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.509+0000 7f32fffff640 1 -- 172.21.15.29:0/2071766966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304031d60 con 0x7f3308159b20 2023-10-16T15:56:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 -- 172.21.15.29:0/2071766966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 msgr2=0x7f330806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f330806bbe0 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f33080c9270 tx=0x7f330402b910 comp rx=0 tx=0).stop 2023-10-16T15:56:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 -- 172.21.15.29:0/2071766966 shutdown_connections 2023-10-16T15:56:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 --2- 172.21.15.29:0/2071766966 >> v2:172.21.15.170:3300/0 conn(0x7f330806c120 0x7f33080b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f33080b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 --2- 172.21.15.29:0/2071766966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f330806bbe0 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 -- 172.21.15.29:0/2071766966 >> 172.21.15.29:0/2071766966 conn(0x7f33080cb410 msgr2=0x7f33080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 -- 172.21.15.29:0/2071766966 shutdown_connections 2023-10-16T15:56:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.510+0000 7f330f905640 1 -- 172.21.15.29:0/2071766966 wait complete. 2023-10-16T15:56:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.512+0000 7f330f905640 1 Processor -- start 2023-10-16T15:56:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.512+0000 7f330f905640 1 -- start start 2023-10-16T15:56:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.513+0000 7f330f905640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f330806c120 0x7f3308151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.513+0000 7f330f905640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f3308155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.513+0000 7f330f905640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f3308156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.513+0000 7f330f905640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3308151700 con 0x7f3308159b20 2023-10-16T15:56:02.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.513+0000 7f330f905640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33081519e0 con 0x7f330806c120 2023-10-16T15:56:02.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.513+0000 7f330f905640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3308151cc0 con 0x7f3308159f00 2023-10-16T15:56:02.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f3308155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f3308155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42998/0 (socket says 172.21.15.29:42998) 2023-10-16T15:56:02.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 -- 172.21.15.29:0/1220241429 learned_addr learned my addr 172.21.15.29:0/1220241429 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:02.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330dea2640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f3308156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330d6a1640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f330806c120 0x7f3308151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 -- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 msgr2=0x7f3308156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f3308156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 -- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f330806c120 msgr2=0x7f3308151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f330806c120 0x7f3308151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 -- 172.21.15.29:0/1220241429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33040094d0 con 0x7f3308159b20 2023-10-16T15:56:02.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.514+0000 7f330cea0640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f3308155f30 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f32f800e810 tx=0x7f32f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:02.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.515+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8004230 con 0x7f3308159b20 2023-10-16T15:56:02.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.515+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f8004390 con 0x7f3308159b20 2023-10-16T15:56:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.515+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f800c910 con 0x7f3308159b20 2023-10-16T15:56:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.515+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3308151fa0 con 0x7f3308159b20 2023-10-16T15:56:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.515+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3308158940 con 0x7f3308159b20 2023-10-16T15:56:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.516+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330804e160 con 0x7f3308159b20 2023-10-16T15:56:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.517+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f32f800ca70 con 0x7f3308159b20 2023-10-16T15:56:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.518+0000 7f32fdffb640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 0x7f32ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.518+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f32f80896c0 con 0x7f3308159b20 2023-10-16T15:56:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.518+0000 7f330d6a1640 1 -- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 msgr2=0x7f32ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.518+0000 7f330d6a1640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 0x7f32ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T15:56:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.522+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32f8054290 con 0x7f3308159b20 2023-10-16T15:56:02.613 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.612+0000 7f577862e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:56:02.651 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:02.650+0000 7f56c76b0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T15:56:02.670 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.668+0000 7f577862e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:56:02.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.716+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3308071160 con 0x7f3308159b20 2023-10-16T15:56:02.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.718+0000 7f32fdffb640 1 -- 172.21.15.29:0/1220241429 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+17267 (secure 0 0 0) 0x7f32f8006e10 con 0x7f3308159b20 2023-10-16T15:56:02.719 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:02.720 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":267,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:55:54.590749+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2485822714":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/831181775":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/1047565771":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2742286180":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3423446827":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3865201075":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6833/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/763385128":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2667017960":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:56:02.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.719+0000 7f330d6a1640 1 -- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 msgr2=0x7f32ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.170:6832/1364949379 2023-10-16T15:56:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.719+0000 7f330d6a1640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 0x7f32ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T15:56:02.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 msgr2=0x7f32ec06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T15:56:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 0x7f32ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 msgr2=0x7f3308155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f3308155f30 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f32f800e810 tx=0x7f32f800ecd0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 shutdown_connections 2023-10-16T15:56:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:6832/1364949379,v1:172.21.15.170:6833/1364949379] conn(0x7f32ec06b020 0x7f32ec06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3308159b20 0x7f3308155f30 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3308159f00 0x7f3308156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 --2- 172.21.15.29:0/1220241429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f330806c120 0x7f3308151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.720+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 >> 172.21.15.29:0/1220241429 conn(0x7f33080cb410 msgr2=0x7f33080cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:02.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.721+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 shutdown_connections 2023-10-16T15:56:02.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.721+0000 7f330f905640 1 -- 172.21.15.29:0/1220241429 wait complete. 2023-10-16T15:56:02.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T14:16:17.885864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T15:55:37.838394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T15:55:38.855075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '266', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T15:56:02.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T15:56:02.784 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.782+0000 7f577862e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:56:02.874 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.873+0000 7f56fa22e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.875 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.873+0000 7f56fa22e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.875 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.873+0000 7f56fa22e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.875 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.873+0000 7f56fa22e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.875 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.873+0000 7f56fa22e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.876 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.874+0000 7f56fd234640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.876 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.874+0000 7f56fd234640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.876 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.874+0000 7f56fd234640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.876 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.874+0000 7f56fd234640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.877 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-16T15:56:02.874+0000 7f56fd234640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T15:56:02.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.946+0000 7f549b59e640 1 Processor -- start 2023-10-16T15:56:02.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.946+0000 7f549b59e640 1 -- start start 2023-10-16T15:56:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.948+0000 7f549b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f549c0b1ca0 0x7f549c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.948+0000 7f549b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549c0d0970 con 0x7f549c0cfc40 2023-10-16T15:56:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.948+0000 7f549b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549c0d0ea0 con 0x7f549c0b2080 2023-10-16T15:56:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.948+0000 7f549b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549c0d0bc0 con 0x7f549c0b1ca0 2023-10-16T15:56:02.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.948+0000 7f549ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f549c0cfc40 0x7f549c0b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47768/0 (socket says 172.21.15.29:47768) 2023-10-16T15:56:02.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.948+0000 7f549ad9d640 1 -- 172.21.15.29:0/453772011 learned_addr learned my addr 172.21.15.29:0/453772011 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:02.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.949+0000 7f549a59c640 1 --2- 172.21.15.29:0/453772011 >> v2:172.21.15.170:3300/0 conn(0x7f549c0b1ca0 0x7f549c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.949+0000 7f5499d9b640 1 -- 172.21.15.29:0/453772011 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7719705 0 0) 0x7f549c0d0970 con 0x7f549c0cfc40 2023-10-16T15:56:02.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.949+0000 7f5499d9b640 1 -- 172.21.15.29:0/453772011 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5488003520 con 0x7f549c0cfc40 2023-10-16T15:56:02.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.949+0000 7f549a59c640 1 -- 172.21.15.29:0/453772011 >> v1:172.21.15.170:6789/0 conn(0x7f549c0b2080 legacy=0x7f549c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.949+0000 7f549a59c640 1 -- 172.21.15.29:0/453772011 >> v1:172.21.15.29:6789/0 conn(0x7f549c0cfc40 legacy=0x7f549c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.949+0000 7f549a59c640 1 -- 172.21.15.29:0/453772011 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c0cd940 con 0x7f549c0b1ca0 2023-10-16T15:56:02.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.950+0000 7f549a59c640 1 --2- 172.21.15.29:0/453772011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0b1ca0 0x7f549c0d36b0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f548c0097a0 tx=0x7f548c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258ff76b0739efc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:02.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.951+0000 7f5499d9b640 1 -- 172.21.15.29:0/453772011 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c02f810 con 0x7f549c0b1ca0 2023-10-16T15:56:02.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.951+0000 7f5499d9b640 1 -- 172.21.15.29:0/453772011 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c02f970 con 0x7f549c0b1ca0 2023-10-16T15:56:02.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.951+0000 7f5499d9b640 1 -- 172.21.15.29:0/453772011 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c02fd00 con 0x7f549c0b1ca0 2023-10-16T15:56:02.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.951+0000 7f549b59e640 1 -- 172.21.15.29:0/453772011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0b1ca0 msgr2=0x7f549c0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.951+0000 7f549b59e640 1 --2- 172.21.15.29:0/453772011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0b1ca0 0x7f549c0d36b0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f548c0097a0 tx=0x7f548c02b9f0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.952+0000 7f549b59e640 1 -- 172.21.15.29:0/453772011 shutdown_connections 2023-10-16T15:56:02.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.952+0000 7f549b59e640 1 --2- 172.21.15.29:0/453772011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0b1ca0 0x7f549c0d36b0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.952+0000 7f549b59e640 1 -- 172.21.15.29:0/453772011 >> 172.21.15.29:0/453772011 conn(0x7f549c0d2b90 msgr2=0x7f549c0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:02.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.952+0000 7f549b59e640 1 -- 172.21.15.29:0/453772011 shutdown_connections 2023-10-16T15:56:02.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.952+0000 7f549b59e640 1 -- 172.21.15.29:0/453772011 wait complete. 2023-10-16T15:56:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.954+0000 7f549b59e640 1 Processor -- start 2023-10-16T15:56:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.954+0000 7f549b59e640 1 -- start start 2023-10-16T15:56:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f549b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f549c0b1ca0 0x7f549c14d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f549b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 0x7f549c1f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f549b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0cfc40 0x7f549c15a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f549b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c15a920 con 0x7f549c0b2080 2023-10-16T15:56:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f549b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c1f4a00 con 0x7f549c0cfc40 2023-10-16T15:56:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f549b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549c1f4ce0 con 0x7f549c0b1ca0 2023-10-16T15:56:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f5493fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 0x7f549c1f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f5493fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 0x7f549c1f4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43006/0 (socket says 172.21.15.29:43006) 2023-10-16T15:56:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.955+0000 7f5493fff640 1 -- 172.21.15.29:0/2235769352 learned_addr learned my addr 172.21.15.29:0/2235769352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f549a59c640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f549c0b1ca0 0x7f549c14d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f549ad9d640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0cfc40 0x7f549c15a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5493fff640 1 -- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f549c0b1ca0 msgr2=0x7f549c14d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5493fff640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f549c0b1ca0 0x7f549c14d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5493fff640 1 -- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0cfc40 msgr2=0x7f549c15a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5493fff640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0cfc40 0x7f549c15a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5493fff640 1 -- 172.21.15.29:0/2235769352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c009400 con 0x7f549c0b2080 2023-10-16T15:56:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f549ad9d640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0cfc40 0x7f549c15a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:56:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5493fff640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 0x7f549c1f4370 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f548400c400 tx=0x7f548400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54840052d0 con 0x7f549c0b2080 2023-10-16T15:56:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5484005430 con 0x7f549c0b2080 2023-10-16T15:56:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.956+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484003830 con 0x7f549c0b2080 2023-10-16T15:56:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.957+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c1f4fc0 con 0x7f549c0b2080 2023-10-16T15:56:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.957+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c1f5580 con 0x7f549c0b2080 2023-10-16T15:56:02.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.958+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c1f57a0 con 0x7f549c0b2080 2023-10-16T15:56:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.959+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f5484003990 con 0x7f549c0b2080 2023-10-16T15:56:02.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.960+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f5484088a70 con 0x7f549c0b2080 2023-10-16T15:56:02.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:02.965+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5484053b00 con 0x7f549c0b2080 2023-10-16T15:56:02.980 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:02.979+0000 7f56c76b0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T15:56:03.040 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:03.039+0000 7f56c76b0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T15:56:03.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.158+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f549c04e9b0 con 0x7f549c0b2080 2023-10-16T15:56:03.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.159+0000 7f5492ffd640 1 -- 172.21.15.29:0/2235769352 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v268) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5484017020 con 0x7f549c0b2080 2023-10-16T15:56:03.160 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-16T15:56:03.161 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-16T15:56:03.161+0000 7f56c76b0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T15:56:03.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 msgr2=0x7f549c1f4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 0x7f549c1f4370 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f548400c400 tx=0x7f548400c8c0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 shutdown_connections 2023-10-16T15:56:03.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549c0b2080 0x7f549c1f4370 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f549c0cfc40 0x7f549c15a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 --2- 172.21.15.29:0/2235769352 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f549c0b1ca0 0x7f549c14d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 >> 172.21.15.29:0/2235769352 conn(0x7f549c0d2b90 msgr2=0x7f549c0d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:03.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.162+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 shutdown_connections 2023-10-16T15:56:03.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.163+0000 7f549b59e640 1 -- 172.21.15.29:0/2235769352 wait complete. 2023-10-16T15:56:03.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T15:56:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.400+0000 7fce7d78e640 1 Processor -- start 2023-10-16T15:56:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.400+0000 7fce7d78e640 1 -- start start 2023-10-16T15:56:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.402+0000 7fce7d78e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fce7815c9f0 0x7fce780c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.402+0000 7fce7d78e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.402+0000 7fce7d78e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce78160a90 con 0x7fce780c9230 2023-10-16T15:56:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.402+0000 7fce7d78e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce780cdec0 con 0x7fce7815c9f0 2023-10-16T15:56:03.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.402+0000 7fce7d78e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce780cd900 con 0x7fce7815dde0 2023-10-16T15:56:03.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce7cf8d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce77fff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fce7815c9f0 0x7fce780c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce6ffff640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fce780c9230 0x7fce780709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:37866/0 (socket says 172.21.15.29:37866) 2023-10-16T15:56:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce6ffff640 1 -- 172.21.15.29:0/1393563519 learned_addr learned my addr 172.21.15.29:0/1393563519 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce77fff640 1 -- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 msgr2=0x7fce780ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce77fff640 1 --2- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce77fff640 1 -- 172.21.15.29:0/1393563519 >> v1:172.21.15.170:6789/0 conn(0x7fce780c9230 legacy=0x7fce780709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.403+0000 7fce77fff640 1 -- 172.21.15.29:0/1393563519 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780cdbe0 con 0x7fce7815c9f0 2023-10-16T15:56:03.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.405+0000 7fce77fff640 1 --2- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce7815c9f0 0x7fce780c8cf0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fce680097a0 tx=0x7fce6802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0aaca4e76e86f31 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:03.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.405+0000 7fce777fe640 1 -- 172.21.15.29:0/1393563519 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6802d040 con 0x7fce7815c9f0 2023-10-16T15:56:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.405+0000 7fce777fe640 1 -- 172.21.15.29:0/1393563519 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce68031810 con 0x7fce7815c9f0 2023-10-16T15:56:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.405+0000 7fce777fe640 1 -- 172.21.15.29:0/1393563519 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68031ba0 con 0x7fce7815c9f0 2023-10-16T15:56:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 -- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce7815c9f0 msgr2=0x7fce780c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 --2- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce7815c9f0 0x7fce780c8cf0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fce680097a0 tx=0x7fce6802b8b0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 -- 172.21.15.29:0/1393563519 shutdown_connections 2023-10-16T15:56:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 --2- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 --2- 172.21.15.29:0/1393563519 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce7815c9f0 0x7fce780c8cf0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 -- 172.21.15.29:0/1393563519 >> 172.21.15.29:0/1393563519 conn(0x7fce780ce6a0 msgr2=0x7fce780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 -- 172.21.15.29:0/1393563519 shutdown_connections 2023-10-16T15:56:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.406+0000 7fce7d78e640 1 -- 172.21.15.29:0/1393563519 wait complete. 2023-10-16T15:56:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.408+0000 7fce7d78e640 1 Processor -- start 2023-10-16T15:56:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.408+0000 7fce7d78e640 1 -- start start 2023-10-16T15:56:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce7d78e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce780c9230 0x7fce78157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce7d78e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 0x7fce780c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce7d78e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce7d78e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce780c3360 con 0x7fce7815c9f0 2023-10-16T15:56:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce7d78e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce780c3640 con 0x7fce780c9230 2023-10-16T15:56:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce7d78e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce780c3920 con 0x7fce7815dde0 2023-10-16T15:56:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 0x7fce780c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 0x7fce780c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43008/0 (socket says 172.21.15.29:43008) 2023-10-16T15:56:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.409+0000 7fce6ffff640 1 -- 172.21.15.29:0/4236537301 learned_addr learned my addr 172.21.15.29:0/4236537301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce7cf8d640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce77fff640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce780c9230 0x7fce78157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce6ffff640 1 -- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 msgr2=0x7fce780c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce6ffff640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce6ffff640 1 -- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce780c9230 msgr2=0x7fce78157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce6ffff640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce780c9230 0x7fce78157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce6ffff640 1 -- 172.21.15.29:0/4236537301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68009400 con 0x7fce7815c9f0 2023-10-16T15:56:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce6ffff640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 0x7fce780c5770 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7fce6000c4a0 tx=0x7fce6000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60005ce0 con 0x7fce7815c9f0 2023-10-16T15:56:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.410+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce60005e40 con 0x7fce7815c9f0 2023-10-16T15:56:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.411+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60007270 con 0x7fce7815c9f0 2023-10-16T15:56:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.411+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce780c3c00 con 0x7fce7815c9f0 2023-10-16T15:56:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.411+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce780c5cb0 con 0x7fce7815c9f0 2023-10-16T15:56:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.412+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce780c4120 con 0x7fce7815c9f0 2023-10-16T15:56:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.413+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fce600073d0 con 0x7fce7815c9f0 2023-10-16T15:56:03.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.414+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7fce60088900 con 0x7fce7815c9f0 2023-10-16T15:56:03.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.418+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce780c4120 con 0x7fce7815c9f0 2023-10-16T15:56:03.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.610+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce780c5e40 con 0x7fce7815c9f0 2023-10-16T15:56:03.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.611+0000 7fce757fa640 1 -- 172.21.15.29:0/4236537301 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v268) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce60017020 con 0x7fce7815c9f0 2023-10-16T15:56:03.612 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-16T15:56:03.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 msgr2=0x7fce780c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 0x7fce780c5770 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7fce6000c4a0 tx=0x7fce6000c960 comp rx=0 tx=0).stop 2023-10-16T15:56:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 shutdown_connections 2023-10-16T15:56:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce7815c9f0 0x7fce780c5770 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fce7815dde0 0x7fce780c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 --2- 172.21.15.29:0/4236537301 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce780c9230 0x7fce78157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 >> 172.21.15.29:0/4236537301 conn(0x7fce780ce6a0 msgr2=0x7fce780cd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.613+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 shutdown_connections 2023-10-16T15:56:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.614+0000 7fce7d78e640 1 -- 172.21.15.29:0/4236537301 wait complete. 2023-10-16T15:56:03.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T15:56:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.866+0000 7f8b57fff640 1 Processor -- start 2023-10-16T15:56:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.866+0000 7f8b57fff640 1 -- start start 2023-10-16T15:56:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.868+0000 7f8b57fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b58159950 0x7f8b5815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.868+0000 7f8b57fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b58070450 0x7f8b580708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.868+0000 7f8b57fff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b58070df0 con 0x7f8b5806c100 2023-10-16T15:56:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.868+0000 7f8b57fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5815e490 con 0x7f8b58159950 2023-10-16T15:56:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.868+0000 7f8b57fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5815aef0 con 0x7f8b58070450 2023-10-16T15:56:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b58159950 0x7f8b5815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b58159950 0x7f8b5815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43010/0 (socket says 172.21.15.29:43010) 2023-10-16T15:56:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 -- 172.21.15.29:0/2811425882 learned_addr learned my addr 172.21.15.29:0/2811425882 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b4ffff640 1 --2- 172.21.15.29:0/2811425882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b58070450 0x7f8b580708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 -- 172.21.15.29:0/2811425882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b58070450 msgr2=0x7f8b580708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 --2- 172.21.15.29:0/2811425882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b58070450 0x7f8b580708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 -- 172.21.15.29:0/2811425882 >> v1:172.21.15.170:6789/0 conn(0x7f8b5806c100 legacy=0x7f8b5806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.869+0000 7f8b56ffd640 1 -- 172.21.15.29:0/2811425882 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5806cc80 con 0x7f8b58159950 2023-10-16T15:56:03.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.870+0000 7f8b56ffd640 1 --2- 172.21.15.29:0/2811425882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b5815a9b0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800bb10 tx=0x7f8b4802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb3cf75ad26d010 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:03.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b567fc640 1 -- 172.21.15.29:0/2811425882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4802ee70 con 0x7f8b58159950 2023-10-16T15:56:03.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b567fc640 1 -- 172.21.15.29:0/2811425882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b48036440 con 0x7f8b58159950 2023-10-16T15:56:03.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b567fc640 1 -- 172.21.15.29:0/2811425882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480367d0 con 0x7f8b58159950 2023-10-16T15:56:03.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b57fff640 1 -- 172.21.15.29:0/2811425882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 msgr2=0x7f8b5815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b57fff640 1 --2- 172.21.15.29:0/2811425882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b5815a9b0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800bb10 tx=0x7f8b4802e0c0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b57fff640 1 -- 172.21.15.29:0/2811425882 shutdown_connections 2023-10-16T15:56:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b57fff640 1 --2- 172.21.15.29:0/2811425882 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b58070450 0x7f8b580708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b57fff640 1 --2- 172.21.15.29:0/2811425882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b5815a9b0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.871+0000 7f8b57fff640 1 -- 172.21.15.29:0/2811425882 >> 172.21.15.29:0/2811425882 conn(0x7f8b580ce6a0 msgr2=0x7f8b580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.872+0000 7f8b57fff640 1 -- 172.21.15.29:0/2811425882 shutdown_connections 2023-10-16T15:56:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.872+0000 7f8b57fff640 1 -- 172.21.15.29:0/2811425882 wait complete. 2023-10-16T15:56:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.873+0000 7f8b57fff640 1 Processor -- start 2023-10-16T15:56:03.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.873+0000 7f8b57fff640 1 -- start start 2023-10-16T15:56:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.874+0000 7f8b57fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5806c100 0x7f8b58158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.874+0000 7f8b57fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b58070450 0x7f8b580c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.874+0000 7f8b57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b580ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.874+0000 7f8b57fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580c3310 con 0x7f8b58159950 2023-10-16T15:56:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.874+0000 7f8b57fff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b58159120 con 0x7f8b58070450 2023-10-16T15:56:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.874+0000 7f8b57fff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b580aee10 con 0x7f8b5806c100 2023-10-16T15:56:03.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b580ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b580ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43014/0 (socket says 172.21.15.29:43014) 2023-10-16T15:56:03.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 -- 172.21.15.29:0/802394768 learned_addr learned my addr 172.21.15.29:0/802394768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:03.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b4ffff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b58070450 0x7f8b580c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 -- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5806c100 msgr2=0x7f8b58158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5806c100 0x7f8b58158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 -- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b58070450 msgr2=0x7f8b580c4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b58070450 0x7f8b580c4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:03.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 -- 172.21.15.29:0/802394768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4800b770 con 0x7f8b58159950 2023-10-16T15:56:03.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.875+0000 7f8b577fe640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b580ae070 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c6b0 tx=0x7f8b5000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:03.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.876+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50003b70 con 0x7f8b58159950 2023-10-16T15:56:03.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.876+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50003cf0 con 0x7f8b58159950 2023-10-16T15:56:03.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.876+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50003ea0 con 0x7f8b58159950 2023-10-16T15:56:03.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.876+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b580aeb30 con 0x7f8b58159950 2023-10-16T15:56:03.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.876+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b580c3820 con 0x7f8b58159950 2023-10-16T15:56:03.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.877+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5804e9b0 con 0x7f8b58159950 2023-10-16T15:56:03.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.878+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8b50005310 con 0x7f8b58159950 2023-10-16T15:56:03.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.879+0000 7f8b4f7fe640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f8b3806acc0 0x7f8b3806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:03.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.879+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f8b50089430 con 0x7f8b58159950 2023-10-16T15:56:03.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.880+0000 7f8b56ffd640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f8b3806acc0 0x7f8b3806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:03.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.881+0000 7f8b56ffd640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f8b3806acc0 0x7f8b3806d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8b48038000 tx=0x7f8b48030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:03.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:03.883+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b500543d0 con 0x7f8b58159950 2023-10-16T15:56:04.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.076+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8b5806c890 con 0x7f8b58159950 2023-10-16T15:56:04.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.077+0000 7f8b4f7fe640 1 -- 172.21.15.29:0/802394768 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v268) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8b500579f0 con 0x7f8b58159950 2023-10-16T15:56:04.078 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-16T15:56:04.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.079+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f8b3806acc0 msgr2=0x7f8b3806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.079+0000 7f8b57fff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f8b3806acc0 0x7f8b3806d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8b48038000 tx=0x7f8b48030000 comp rx=0 tx=0).stop 2023-10-16T15:56:04.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.079+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 msgr2=0x7f8b580ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.079+0000 7f8b57fff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b580ae070 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c6b0 tx=0x7f8b5000cb70 comp rx=0 tx=0).stop 2023-10-16T15:56:04.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 shutdown_connections 2023-10-16T15:56:04.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f8b3806acc0 0x7f8b3806d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b58159950 0x7f8b580ae070 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8b58070450 0x7f8b580c4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 --2- 172.21.15.29:0/802394768 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8b5806c100 0x7f8b58158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 >> 172.21.15.29:0/802394768 conn(0x7f8b580ce6a0 msgr2=0x7f8b5815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:04.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 shutdown_connections 2023-10-16T15:56:04.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.080+0000 7f8b57fff640 1 -- 172.21.15.29:0/802394768 wait complete. 2023-10-16T15:56:04.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:56:04.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.319+0000 7fe74f59e640 1 Processor -- start 2023-10-16T15:56:04.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.319+0000 7fe74f59e640 1 -- start start 2023-10-16T15:56:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe75015c9f0 0x7fe7500c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7500c9230 0x7fe7500709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe750070f20 0x7fe7500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7500cdec0 con 0x7fe75015c9f0 2023-10-16T15:56:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe750160a90 con 0x7fe750070f20 2023-10-16T15:56:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500cd900 con 0x7fe7500c9230 2023-10-16T15:56:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.321+0000 7fe74e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe75015c9f0 0x7fe7500c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe75015c9f0 0x7fe7500c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51380/0 (socket says 172.21.15.29:51380) 2023-10-16T15:56:04.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 -- 172.21.15.29:0/2370561238 learned_addr learned my addr 172.21.15.29:0/2370561238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:04.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe747fff640 1 --2- 172.21.15.29:0/2370561238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7500c9230 0x7fe7500709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74ed9d640 1 --2- 172.21.15.29:0/2370561238 >> v2:172.21.15.170:3300/0 conn(0x7fe750070f20 0x7fe7500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 -- 172.21.15.29:0/2370561238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7500c9230 msgr2=0x7fe7500709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 --2- 172.21.15.29:0/2370561238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7500c9230 0x7fe7500709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 -- 172.21.15.29:0/2370561238 >> v2:172.21.15.170:3300/0 conn(0x7fe750070f20 msgr2=0x7fe7500ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 --2- 172.21.15.29:0/2370561238 >> v2:172.21.15.170:3300/0 conn(0x7fe750070f20 0x7fe7500ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.322+0000 7fe74e59c640 1 -- 172.21.15.29:0/2370561238 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500cdbe0 con 0x7fe75015c9f0 2023-10-16T15:56:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.324+0000 7fe74e59c640 1 --2- 172.21.15.29:0/2370561238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75015c9f0 0x7fe7500c8cf0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7fe7500c8c80 tx=0x7fe74002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cb31adf52bdd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.324+0000 7fe74dd9b640 1 -- 172.21.15.29:0/2370561238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740031870 con 0x7fe75015c9f0 2023-10-16T15:56:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.324+0000 7fe74dd9b640 1 -- 172.21.15.29:0/2370561238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7400319d0 con 0x7fe75015c9f0 2023-10-16T15:56:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.324+0000 7fe74dd9b640 1 -- 172.21.15.29:0/2370561238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740031d60 con 0x7fe75015c9f0 2023-10-16T15:56:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 -- 172.21.15.29:0/2370561238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75015c9f0 msgr2=0x7fe7500c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 --2- 172.21.15.29:0/2370561238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75015c9f0 0x7fe7500c8cf0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7fe7500c8c80 tx=0x7fe74002b910 comp rx=0 tx=0).stop 2023-10-16T15:56:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 -- 172.21.15.29:0/2370561238 shutdown_connections 2023-10-16T15:56:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 --2- 172.21.15.29:0/2370561238 >> v2:172.21.15.170:3300/0 conn(0x7fe750070f20 0x7fe7500ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 --2- 172.21.15.29:0/2370561238 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe7500c9230 0x7fe7500709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 --2- 172.21.15.29:0/2370561238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe75015c9f0 0x7fe7500c8cf0 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 -- 172.21.15.29:0/2370561238 >> 172.21.15.29:0/2370561238 conn(0x7fe7500ce6a0 msgr2=0x7fe7500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 -- 172.21.15.29:0/2370561238 shutdown_connections 2023-10-16T15:56:04.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.325+0000 7fe74f59e640 1 -- 172.21.15.29:0/2370561238 wait complete. 2023-10-16T15:56:04.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.327+0000 7fe74f59e640 1 Processor -- start 2023-10-16T15:56:04.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.327+0000 7fe74f59e640 1 -- start start 2023-10-16T15:56:04.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 0x7fe7500c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7500c9230 0x7fe75014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe75015c9f0 0x7fe75014ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c4830 con 0x7fe750070f20 2023-10-16T15:56:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c4b10 con 0x7fe7500c9230 2023-10-16T15:56:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c4df0 con 0x7fe75015c9f0 2023-10-16T15:56:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 0x7fe7500c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 0x7fe7500c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51388/0 (socket says 172.21.15.29:51388) 2023-10-16T15:56:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 -- 172.21.15.29:0/1048863049 learned_addr learned my addr 172.21.15.29:0/1048863049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 -- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe75015c9f0 msgr2=0x7fe75014ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe747fff640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7500c9230 0x7fe75014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe75015c9f0 0x7fe75014ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 -- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7500c9230 msgr2=0x7fe75014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7500c9230 0x7fe75014a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.329+0000 7fe74e59c640 1 -- 172.21.15.29:0/1048863049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400094d0 con 0x7fe750070f20 2023-10-16T15:56:04.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.330+0000 7fe74e59c640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 0x7fe7500c8030 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fe740000c00 tx=0x7fe7400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.330+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74003abe0 con 0x7fe750070f20 2023-10-16T15:56:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.330+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74003ad40 con 0x7fe750070f20 2023-10-16T15:56:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.330+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740044610 con 0x7fe750070f20 2023-10-16T15:56:04.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.330+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7500c50d0 con 0x7fe750070f20 2023-10-16T15:56:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.330+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7500c3ac0 con 0x7fe750070f20 2023-10-16T15:56:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.331+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75004e9b0 con 0x7fe750070f20 2023-10-16T15:56:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.332+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe74004d450 con 0x7fe750070f20 2023-10-16T15:56:04.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.333+0000 7fe746ffd640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe73006acc0 0x7fe73006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.333+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7fe74002d040 con 0x7fe750070f20 2023-10-16T15:56:04.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.334+0000 7fe747fff640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe73006acc0 0x7fe73006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.335+0000 7fe747fff640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe73006acc0 0x7fe73006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7501f5800 tx=0x7fe738008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:04.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.339+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe7400ade60 con 0x7fe750070f20 2023-10-16T15:56:04.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.577+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7500adac0 con 0x7fe750070f20 2023-10-16T15:56:04.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.578+0000 7fe746ffd640 1 -- 172.21.15.29:0/1048863049 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fe740035020 con 0x7fe750070f20 2023-10-16T15:56:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:04.580 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:55:39.826547+0000","modified":"2023-10-16T15:55:42.906111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12345},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12345":{"gid":12345,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2425185007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2425185007},{"type":"v1","addr":"172.21.15.29:6835","nonce":2425185007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T15:56:04.580 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-16T15:56:04.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe73006acc0 msgr2=0x7fe73006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe73006acc0 0x7fe73006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7501f5800 tx=0x7fe738008040 comp rx=0 tx=0).stop 2023-10-16T15:56:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 msgr2=0x7fe7500c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 0x7fe7500c8030 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fe740000c00 tx=0x7fe7400046c0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 shutdown_connections 2023-10-16T15:56:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe73006acc0 0x7fe73006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe75015c9f0 0x7fe75014ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe7500c9230 0x7fe75014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 --2- 172.21.15.29:0/1048863049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe750070f20 0x7fe7500c8030 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 >> 172.21.15.29:0/1048863049 conn(0x7fe7500ce6a0 msgr2=0x7fe75006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:04.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 shutdown_connections 2023-10-16T15:56:04.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.581+0000 7fe74f59e640 1 -- 172.21.15.29:0/1048863049 wait complete. 2023-10-16T15:56:04.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:56:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.832+0000 7fd15f7f5640 1 Processor -- start 2023-10-16T15:56:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.833+0000 7fd15f7f5640 1 -- start start 2023-10-16T15:56:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.834+0000 7fd15f7f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1580cd0c0 0x7fd1581615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.834+0000 7fd15f7f5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1580b0410 0x7fd1580b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.834+0000 7fd15f7f5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15815e3e0 con 0x7fd15815d3f0 2023-10-16T15:56:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.834+0000 7fd15f7f5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1580b0de0 con 0x7fd1580cd0c0 2023-10-16T15:56:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.834+0000 7fd15f7f5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1580cd4a0 con 0x7fd1580b0410 2023-10-16T15:56:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.834+0000 7fd15d591640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1580cd0c0 0x7fd1581615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15d591640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1580cd0c0 0x7fd1581615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51390/0 (socket says 172.21.15.29:51390) 2023-10-16T15:56:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15d591640 1 -- 172.21.15.29:0/2611768888 learned_addr learned my addr 172.21.15.29:0/2611768888 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15cd90640 1 --2- 172.21.15.29:0/2611768888 >> v2:172.21.15.170:3300/0 conn(0x7fd1580b0410 0x7fd1580b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15d591640 1 -- 172.21.15.29:0/2611768888 >> v2:172.21.15.170:3300/0 conn(0x7fd1580b0410 msgr2=0x7fd1580b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15d591640 1 --2- 172.21.15.29:0/2611768888 >> v2:172.21.15.170:3300/0 conn(0x7fd1580b0410 0x7fd1580b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15d591640 1 -- 172.21.15.29:0/2611768888 >> v1:172.21.15.170:6789/0 conn(0x7fd15815d3f0 legacy=0x7fd15815d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.835+0000 7fd15d591640 1 -- 172.21.15.29:0/2611768888 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15815e100 con 0x7fd1580cd0c0 2023-10-16T15:56:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.836+0000 7fd15d591640 1 --2- 172.21.15.29:0/2611768888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580cd0c0 0x7fd1581615f0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fd1580cdad0 tx=0x7fd15402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f1546b9ff6c95a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd14ffff640 1 -- 172.21.15.29:0/2611768888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540317e0 con 0x7fd1580cd0c0 2023-10-16T15:56:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd14ffff640 1 -- 172.21.15.29:0/2611768888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd154031940 con 0x7fd1580cd0c0 2023-10-16T15:56:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd14ffff640 1 -- 172.21.15.29:0/2611768888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154031cd0 con 0x7fd1580cd0c0 2023-10-16T15:56:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd15f7f5640 1 -- 172.21.15.29:0/2611768888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580cd0c0 msgr2=0x7fd1581615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/2611768888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580cd0c0 0x7fd1581615f0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fd1580cdad0 tx=0x7fd15402b880 comp rx=0 tx=0).stop 2023-10-16T15:56:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd15f7f5640 1 -- 172.21.15.29:0/2611768888 shutdown_connections 2023-10-16T15:56:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/2611768888 >> v2:172.21.15.170:3300/0 conn(0x7fd1580b0410 0x7fd1580b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/2611768888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580cd0c0 0x7fd1581615f0 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.837+0000 7fd15f7f5640 1 -- 172.21.15.29:0/2611768888 >> 172.21.15.29:0/2611768888 conn(0x7fd1580cb410 msgr2=0x7fd1580cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.838+0000 7fd15f7f5640 1 -- 172.21.15.29:0/2611768888 shutdown_connections 2023-10-16T15:56:04.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.838+0000 7fd15f7f5640 1 -- 172.21.15.29:0/2611768888 wait complete. 2023-10-16T15:56:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.839+0000 7fd15f7f5640 1 Processor -- start 2023-10-16T15:56:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.839+0000 7fd15f7f5640 1 -- start start 2023-10-16T15:56:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.840+0000 7fd15f7f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 0x7fd158155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.840+0000 7fd15f7f5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1580cd0c0 0x7fd1580c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.840+0000 7fd15f7f5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd15815d3f0 0x7fd1580c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.840+0000 7fd15f7f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580c5370 con 0x7fd1580b0410 2023-10-16T15:56:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.840+0000 7fd15f7f5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580c6da0 con 0x7fd1580cd0c0 2023-10-16T15:56:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.840+0000 7fd15f7f5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580c47f0 con 0x7fd15815d3f0 2023-10-16T15:56:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 0x7fd158155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 0x7fd158155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51394/0 (socket says 172.21.15.29:51394) 2023-10-16T15:56:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 -- 172.21.15.29:0/3398546886 learned_addr learned my addr 172.21.15.29:0/3398546886 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 -- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd15815d3f0 msgr2=0x7fd1580c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15cd90640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1580cd0c0 0x7fd1580c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd15815d3f0 0x7fd1580c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 -- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1580cd0c0 msgr2=0x7fd1580c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1580cd0c0 0x7fd1580c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 -- 172.21.15.29:0/3398546886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1540094d0 con 0x7fd1580b0410 2023-10-16T15:56:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.841+0000 7fd15d591640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 0x7fd158155d70 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7fd154000c00 tx=0x7fd1540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.842+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154032880 con 0x7fd1580b0410 2023-10-16T15:56:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.842+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1540329e0 con 0x7fd1580b0410 2023-10-16T15:56:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.842+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154032d90 con 0x7fd1580b0410 2023-10-16T15:56:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.842+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1580c4ad0 con 0x7fd1580b0410 2023-10-16T15:56:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.842+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1580c5090 con 0x7fd1580b0410 2023-10-16T15:56:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.843+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15804e160 con 0x7fd1580b0410 2023-10-16T15:56:04.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.844+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd1540453f0 con 0x7fd1580b0410 2023-10-16T15:56:04.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.845+0000 7fd14dffb640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd13c06af50 0x7fd13c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:04.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.845+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7fd1540abde0 con 0x7fd1580b0410 2023-10-16T15:56:04.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.845+0000 7fd15cd90640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd13c06af50 0x7fd13c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:04.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.847+0000 7fd15cd90640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd13c06af50 0x7fd13c06d410 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd15806d530 tx=0x7fd148008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:04.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:04.849+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd154076a40 con 0x7fd1580b0410 2023-10-16T15:56:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.049+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1580b4910 con 0x7fd1580b0410 2023-10-16T15:56:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.052+0000 7fd14dffb640 1 -- 172.21.15.29:0/3398546886 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+17836 (secure 0 0 0) 0x7fd15407a060 con 0x7fd1580b0410 2023-10-16T15:56:05.053 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:05.054 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":268,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:56:02.787237+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/567930087":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/936087252":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/2036144584":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/4135571630":"2023-10-17T15:56:02.787159+0000","172.21.15.170:6833/1364949379":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/187921439":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/2485822714":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/965201049":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/831181775":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/1364949379":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/1047565771":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2742286180":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3423446827":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/1983218799":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/3865201075":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6833/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/763385128":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2667017960":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:56:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd13c06af50 msgr2=0x7fd13c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd13c06af50 0x7fd13c06d410 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd15806d530 tx=0x7fd148008040 comp rx=0 tx=0).stop 2023-10-16T15:56:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 msgr2=0x7fd158155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 0x7fd158155d70 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7fd154000c00 tx=0x7fd1540046c0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 shutdown_connections 2023-10-16T15:56:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd13c06af50 0x7fd13c06d410 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd15815d3f0 0x7fd1580c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd1580cd0c0 0x7fd1580c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 --2- 172.21.15.29:0/3398546886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1580b0410 0x7fd158155d70 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 >> 172.21.15.29:0/3398546886 conn(0x7fd1580cb410 msgr2=0x7fd158156d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 shutdown_connections 2023-10-16T15:56:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.055+0000 7fd15f7f5640 1 -- 172.21.15.29:0/3398546886 wait complete. 2023-10-16T15:56:05.075 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:56:05.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:56:05.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.301+0000 7f596f59e640 1 Processor -- start 2023-10-16T15:56:05.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.301+0000 7f596f59e640 1 -- start start 2023-10-16T15:56:05.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.302+0000 7f596f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f597015c9f0 0x7f59700c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596f59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f597015dde0 0x7f59700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596f59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5970160a90 con 0x7f59700c9230 2023-10-16T15:56:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59700cdec0 con 0x7f597015c9f0 2023-10-16T15:56:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596f59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59700cd900 con 0x7f597015dde0 2023-10-16T15:56:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f597015c9f0 0x7f59700c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f597015c9f0 0x7f59700c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51396/0 (socket says 172.21.15.29:51396) 2023-10-16T15:56:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 -- 172.21.15.29:0/893212868 learned_addr learned my addr 172.21.15.29:0/893212868 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 -- 172.21.15.29:0/893212868 >> v2:172.21.15.170:3300/0 conn(0x7f597015dde0 msgr2=0x7f59700ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 --2- 172.21.15.29:0/893212868 >> v2:172.21.15.170:3300/0 conn(0x7f597015dde0 0x7f59700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 -- 172.21.15.29:0/893212868 >> v1:172.21.15.170:6789/0 conn(0x7f59700c9230 legacy=0x7f59700709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.303+0000 7f596e59c640 1 -- 172.21.15.29:0/893212868 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59700cdbe0 con 0x7f597015c9f0 2023-10-16T15:56:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.304+0000 7f596e59c640 1 --2- 172.21.15.29:0/893212868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c8cf0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f59700b0120 tx=0x7f596002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d7bfde0f8d6ab8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.305+0000 7f596d59a640 1 -- 172.21.15.29:0/893212868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960031890 con 0x7f597015c9f0 2023-10-16T15:56:05.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.305+0000 7f596d59a640 1 -- 172.21.15.29:0/893212868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59600319f0 con 0x7f597015c9f0 2023-10-16T15:56:05.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.305+0000 7f596d59a640 1 -- 172.21.15.29:0/893212868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960031d80 con 0x7f597015c9f0 2023-10-16T15:56:05.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.305+0000 7f596f59e640 1 -- 172.21.15.29:0/893212868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 msgr2=0x7f59700c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.305+0000 7f596f59e640 1 --2- 172.21.15.29:0/893212868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c8cf0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f59700b0120 tx=0x7f596002b930 comp rx=0 tx=0).stop 2023-10-16T15:56:05.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.306+0000 7f596f59e640 1 -- 172.21.15.29:0/893212868 shutdown_connections 2023-10-16T15:56:05.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.306+0000 7f596f59e640 1 --2- 172.21.15.29:0/893212868 >> v2:172.21.15.170:3300/0 conn(0x7f597015dde0 0x7f59700ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.306+0000 7f596f59e640 1 --2- 172.21.15.29:0/893212868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c8cf0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.306+0000 7f596f59e640 1 -- 172.21.15.29:0/893212868 >> 172.21.15.29:0/893212868 conn(0x7f59700ce6a0 msgr2=0x7f59700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:05.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.306+0000 7f596f59e640 1 -- 172.21.15.29:0/893212868 shutdown_connections 2023-10-16T15:56:05.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.306+0000 7f596f59e640 1 -- 172.21.15.29:0/893212868 wait complete. 2023-10-16T15:56:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.308+0000 7f596f59e640 1 Processor -- start 2023-10-16T15:56:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.308+0000 7f596f59e640 1 -- start start 2023-10-16T15:56:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596f59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59700c9230 0x7f59701f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596f59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f597015dde0 0x7f597006b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59701f3350 con 0x7f597015c9f0 2023-10-16T15:56:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596f59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597006b090 con 0x7f59700c9230 2023-10-16T15:56:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596f59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597006ccc0 con 0x7f597015dde0 2023-10-16T15:56:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51398/0 (socket says 172.21.15.29:51398) 2023-10-16T15:56:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596dd9b640 1 -- 172.21.15.29:0/1670072685 learned_addr learned my addr 172.21.15.29:0/1670072685 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.309+0000 7f596e59c640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59700c9230 0x7f59701f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596ed9d640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f597015dde0 0x7f597006b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596dd9b640 1 -- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f597015dde0 msgr2=0x7f597006b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596dd9b640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f597015dde0 0x7f597006b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596dd9b640 1 -- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59700c9230 msgr2=0x7f59701f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596dd9b640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59700c9230 0x7f59701f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596dd9b640 1 -- 172.21.15.29:0/1670072685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59600094d0 con 0x7f597015c9f0 2023-10-16T15:56:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596dd9b640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c43d0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f596400c440 tx=0x7f596400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964005480 con 0x7f597015c9f0 2023-10-16T15:56:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59640035f0 con 0x7f597015c9f0 2023-10-16T15:56:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59640039c0 con 0x7f597015c9f0 2023-10-16T15:56:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.310+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59701f2830 con 0x7f597015c9f0 2023-10-16T15:56:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.311+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59701f2d90 con 0x7f597015c9f0 2023-10-16T15:56:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.312+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597006adf0 con 0x7f597015c9f0 2023-10-16T15:56:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.313+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5964003b20 con 0x7f597015c9f0 2023-10-16T15:56:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.314+0000 7f595effd640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f595006af00 0x7f595006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.314+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f5964088b20 con 0x7f597015c9f0 2023-10-16T15:56:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.314+0000 7f596e59c640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f595006af00 0x7f595006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.316+0000 7f596e59c640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f595006af00 0x7f595006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5960036000 tx=0x7f596003c040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.319+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f597006adf0 con 0x7f597015c9f0 2023-10-16T15:56:05.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.551+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f597006b1d0 con 0x7f597015c9f0 2023-10-16T15:56:05.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.552+0000 7f595effd640 1 -- 172.21.15.29:0/1670072685 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f597006b1d0 con 0x7f597015c9f0 2023-10-16T15:56:05.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:05.554 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:55:39.826547+0000","modified":"2023-10-16T15:55:42.906111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12345},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12345":{"gid":12345,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2425185007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2425185007},{"type":"v1","addr":"172.21.15.29:6835","nonce":2425185007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T15:56:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-16T15:56:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f595006af00 msgr2=0x7f595006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f595006af00 0x7f595006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5960036000 tx=0x7f596003c040 comp rx=0 tx=0).stop 2023-10-16T15:56:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 msgr2=0x7f59700c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c43d0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f596400c440 tx=0x7f596400c900 comp rx=0 tx=0).stop 2023-10-16T15:56:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 shutdown_connections 2023-10-16T15:56:05.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f595006af00 0x7f595006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f597015c9f0 0x7f59700c43d0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f597015dde0 0x7f597006b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 --2- 172.21.15.29:0/1670072685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59700c9230 0x7f59701f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 >> 172.21.15.29:0/1670072685 conn(0x7f59700ce6a0 msgr2=0x7f59700ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:05.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.555+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 shutdown_connections 2023-10-16T15:56:05.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.556+0000 7f596f59e640 1 -- 172.21.15.29:0/1670072685 wait complete. 2023-10-16T15:56:05.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:56:05.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.798+0000 7f3fbffff640 1 Processor -- start 2023-10-16T15:56:05.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.798+0000 7f3fbffff640 1 -- start start 2023-10-16T15:56:05.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.800+0000 7f3fbffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3fc015aa60 0x7f3fc00af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.800+0000 7f3fbffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc00cc9f0 0x7f3fc00b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.800+0000 7f3fbffff640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc00b3730 con 0x7f3fc00afd60 2023-10-16T15:56:05.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.800+0000 7f3fbffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc00b3980 con 0x7f3fc015aa60 2023-10-16T15:56:05.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.800+0000 7f3fbffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00b3c60 con 0x7f3fc00cc9f0 2023-10-16T15:56:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3fc015aa60 0x7f3fc00af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f3fc015aa60 0x7f3fc00af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:38974/0 (socket says 172.21.15.29:38974) 2023-10-16T15:56:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 -- 172.21.15.29:0/3839621480 learned_addr learned my addr 172.21.15.29:0/3839621480 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbf7fe640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc00cc9f0 0x7f3fc00b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 -- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc00cc9f0 msgr2=0x7f3fc00b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:05.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc00cc9f0 0x7f3fc00b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 -- 172.21.15.29:0/3839621480 >> v1:172.21.15.170:6789/0 conn(0x7f3fc00afd60 legacy=0x7f3fc00cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbeffd640 1 -- 172.21.15.29:0/3839621480 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc00b3f40 con 0x7f3fc015aa60 2023-10-16T15:56:05.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.801+0000 7f3fbf7fe640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc00cc9f0 0x7f3fc00b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T15:56:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.803+0000 7f3fbeffd640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc015aa60 0x7f3fc00af820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009870 tx=0x7f3fb002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b304cab3b31f3734 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.803+0000 7f3fbe7fc640 1 -- 172.21.15.29:0/3839621480 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0030840 con 0x7f3fc015aa60 2023-10-16T15:56:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.803+0000 7f3fbe7fc640 1 -- 172.21.15.29:0/3839621480 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb00309a0 con 0x7f3fc015aa60 2023-10-16T15:56:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.803+0000 7f3fbe7fc640 1 -- 172.21.15.29:0/3839621480 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0030d30 con 0x7f3fc015aa60 2023-10-16T15:56:05.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 -- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc015aa60 msgr2=0x7f3fc00af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc015aa60 0x7f3fc00af820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009870 tx=0x7f3fb002b980 comp rx=0 tx=0).stop 2023-10-16T15:56:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 -- 172.21.15.29:0/3839621480 shutdown_connections 2023-10-16T15:56:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc00cc9f0 0x7f3fc00b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 --2- 172.21.15.29:0/3839621480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc015aa60 0x7f3fc00af820 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 -- 172.21.15.29:0/3839621480 >> 172.21.15.29:0/3839621480 conn(0x7f3fc0065ed0 msgr2=0x7f3fc00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 -- 172.21.15.29:0/3839621480 shutdown_connections 2023-10-16T15:56:05.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.804+0000 7f3fbffff640 1 -- 172.21.15.29:0/3839621480 wait complete. 2023-10-16T15:56:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.807+0000 7f3fbffff640 1 Processor -- start 2023-10-16T15:56:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.807+0000 7f3fbffff640 1 -- start start 2023-10-16T15:56:05.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.808+0000 7f3fbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 0x7f3fc0158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.808+0000 7f3fbffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc00cc9f0 0x7f3fc00c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.808+0000 7f3fbffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc015aa60 0x7f3fc00c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.808+0000 7f3fbffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00c8ec0 con 0x7f3fc00afd60 2023-10-16T15:56:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.808+0000 7f3fbffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00c91a0 con 0x7f3fc00cc9f0 2023-10-16T15:56:05.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.808+0000 7f3fbffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00c9480 con 0x7f3fc015aa60 2023-10-16T15:56:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 0x7f3fc0158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 0x7f3fc0158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51414/0 (socket says 172.21.15.29:51414) 2023-10-16T15:56:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 -- 172.21.15.29:0/2605183447 learned_addr learned my addr 172.21.15.29:0/2605183447 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:05.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fb7fff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc00cc9f0 0x7f3fc00c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbf7fe640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc015aa60 0x7f3fc00c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 -- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc015aa60 msgr2=0x7f3fc00c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc015aa60 0x7f3fc00c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 -- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc00cc9f0 msgr2=0x7f3fc00c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc00cc9f0 0x7f3fc00c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.809+0000 7f3fbeffd640 1 -- 172.21.15.29:0/2605183447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8009a70 con 0x7f3fc00afd60 2023-10-16T15:56:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.810+0000 7f3fbeffd640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 0x7f3fc0158510 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000c00 tx=0x7f3fb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:05.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.810+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0039a90 con 0x7f3fc00afd60 2023-10-16T15:56:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.810+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb0039bf0 con 0x7f3fc00afd60 2023-10-16T15:56:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.810+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0043410 con 0x7f3fc00afd60 2023-10-16T15:56:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.810+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb00094d0 con 0x7f3fc00afd60 2023-10-16T15:56:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.811+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc015b1c0 con 0x7f3fc00afd60 2023-10-16T15:56:05.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.812+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc004e9b0 con 0x7f3fc00afd60 2023-10-16T15:56:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.813+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3fb002f390 con 0x7f3fc00afd60 2023-10-16T15:56:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.814+0000 7f3fbcff9640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f3fa006af00 0x7f3fa006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.814+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f3fb00abc90 con 0x7f3fc00afd60 2023-10-16T15:56:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.815+0000 7f3fb7fff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f3fa006af00 0x7f3fa006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:05.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.815+0000 7f3fb7fff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f3fa006af00 0x7f3fa006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8000f60 tx=0x7f3fa8009040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:05.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:05.818+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3fb00767c0 con 0x7f3fc00afd60 2023-10-16T15:56:06.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.052+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc00d1070 con 0x7f3fc00afd60 2023-10-16T15:56:06.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.053+0000 7f3fbcff9640 1 -- 172.21.15.29:0/2605183447 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f3fb0079de0 con 0x7f3fc00afd60 2023-10-16T15:56:06.054 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:06.054 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T15:55:39.826547+0000","modified":"2023-10-16T15:55:42.906111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12345},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12345":{"gid":12345,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6835/2425185007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2425185007},{"type":"v1","addr":"172.21.15.29:6835","nonce":2425185007}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T15:56:06.055 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-16T15:56:06.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f3fa006af00 msgr2=0x7f3fa006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f3fa006af00 0x7f3fa006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8000f60 tx=0x7f3fa8009040 comp rx=0 tx=0).stop 2023-10-16T15:56:06.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 msgr2=0x7f3fc0158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 0x7f3fc0158510 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000c00 tx=0x7f3fb0004730 comp rx=0 tx=0).stop 2023-10-16T15:56:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 shutdown_connections 2023-10-16T15:56:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f3fa006af00 0x7f3fa006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f3fc015aa60 0x7f3fc00c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fc00cc9f0 0x7f3fc00c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 --2- 172.21.15.29:0/2605183447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fc00afd60 0x7f3fc0158510 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 >> 172.21.15.29:0/2605183447 conn(0x7f3fc0065ed0 msgr2=0x7f3fc006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 shutdown_connections 2023-10-16T15:56:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.056+0000 7f3fbffff640 1 -- 172.21.15.29:0/2605183447 wait complete. 2023-10-16T15:56:06.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:56:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.303+0000 7f2972452640 1 Processor -- start 2023-10-16T15:56:06.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.303+0000 7f2972452640 1 -- start start 2023-10-16T15:56:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.304+0000 7f2972452640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f296c0ca0a0 0x7f296c06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.304+0000 7f2972452640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c15d910 con 0x7f296c159950 2023-10-16T15:56:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.304+0000 7f2972452640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c15dbf0 con 0x7f296c070b60 2023-10-16T15:56:06.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.304+0000 7f2972452640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296c15e490 con 0x7f296c0ca0a0 2023-10-16T15:56:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.304+0000 7f2971c51640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f296c0ca0a0 0x7f296c06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.305+0000 7f2971450640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f296c159950 0x7f296c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37468/0 (socket says 172.21.15.29:37468) 2023-10-16T15:56:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.305+0000 7f2971c51640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f296c0ca0a0 0x7f296c06c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51426/0 (socket says 172.21.15.29:51426) 2023-10-16T15:56:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.305+0000 7f2971450640 1 -- 172.21.15.29:0/1133895689 learned_addr learned my addr 172.21.15.29:0/1133895689 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:06.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.305+0000 7f2971c51640 1 -- 172.21.15.29:0/1133895689 >> v1:172.21.15.170:6789/0 conn(0x7f296c070b60 legacy=0x7f296c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.305+0000 7f2971c51640 1 -- 172.21.15.29:0/1133895689 >> v1:172.21.15.29:6789/0 conn(0x7f296c159950 legacy=0x7f296c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.305+0000 7f2971c51640 1 -- 172.21.15.29:0/1133895689 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c15ded0 con 0x7f296c0ca0a0 2023-10-16T15:56:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.306+0000 7f2971c51640 1 --2- 172.21.15.29:0/1133895689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c0ca0a0 0x7f296c06c810 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f2964009d90 tx=0x7f296402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a07432c5db2ccc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.306+0000 7f2963fff640 1 -- 172.21.15.29:0/1133895689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296402cbd0 con 0x7f296c0ca0a0 2023-10-16T15:56:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.306+0000 7f2963fff640 1 -- 172.21.15.29:0/1133895689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296402cd30 con 0x7f296c0ca0a0 2023-10-16T15:56:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2963fff640 1 -- 172.21.15.29:0/1133895689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296402cea0 con 0x7f296c0ca0a0 2023-10-16T15:56:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 -- 172.21.15.29:0/1133895689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c0ca0a0 msgr2=0x7f296c06c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 --2- 172.21.15.29:0/1133895689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c0ca0a0 0x7f296c06c810 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f2964009d90 tx=0x7f296402be20 comp rx=0 tx=0).stop 2023-10-16T15:56:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 -- 172.21.15.29:0/1133895689 shutdown_connections 2023-10-16T15:56:06.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 --2- 172.21.15.29:0/1133895689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c0ca0a0 0x7f296c06c810 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 -- 172.21.15.29:0/1133895689 >> 172.21.15.29:0/1133895689 conn(0x7f296c0ce6a0 msgr2=0x7f296c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 -- 172.21.15.29:0/1133895689 shutdown_connections 2023-10-16T15:56:06.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.307+0000 7f2972452640 1 -- 172.21.15.29:0/1133895689 wait complete. 2023-10-16T15:56:06.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.309+0000 7f2972452640 1 Processor -- start 2023-10-16T15:56:06.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.309+0000 7f2972452640 1 -- start start 2023-10-16T15:56:06.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2972452640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 0x7f296c0a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2972452640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f296c0ca0a0 0x7f296c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2972452640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f296c159950 0x7f296c158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2972452640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c7900 con 0x7f296c070b60 2023-10-16T15:56:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2972452640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c8740 con 0x7f296c159950 2023-10-16T15:56:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2972452640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c7b30 con 0x7f296c0ca0a0 2023-10-16T15:56:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2971450640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 0x7f296c0a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2971450640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 0x7f296c0a8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51442/0 (socket says 172.21.15.29:51442) 2023-10-16T15:56:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.310+0000 7f2971450640 1 -- 172.21.15.29:0/2315550044 learned_addr learned my addr 172.21.15.29:0/2315550044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971450640 1 -- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f296c0ca0a0 msgr2=0x7f296c0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2970c4f640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f296c0ca0a0 0x7f296c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971c51640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f296c159950 0x7f296c158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971450640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f296c0ca0a0 0x7f296c0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971450640 1 -- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f296c159950 msgr2=0x7f296c158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971450640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f296c159950 0x7f296c158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971450640 1 -- 172.21.15.29:0/2315550044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2964009a70 con 0x7f296c070b60 2023-10-16T15:56:06.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2970c4f640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f296c0ca0a0 0x7f296c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971c51640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f296c159950 0x7f296c158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2971450640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 0x7f296c0a8fc0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f296800c480 tx=0x7f296800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.311+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968005ce0 con 0x7f296c070b60 2023-10-16T15:56:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.312+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2968005e40 con 0x7f296c070b60 2023-10-16T15:56:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.312+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c0c7e10 con 0x7f296c070b60 2023-10-16T15:56:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.312+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29680072f0 con 0x7f296c070b60 2023-10-16T15:56:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.312+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c0c83d0 con 0x7f296c070b60 2023-10-16T15:56:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.313+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c04e9b0 con 0x7f296c070b60 2023-10-16T15:56:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.314+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2968007450 con 0x7f296c070b60 2023-10-16T15:56:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.315+0000 7f2961ffb640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f295006af30 0x7f295006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.315+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f2968088e40 con 0x7f296c070b60 2023-10-16T15:56:06.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.315+0000 7f2970c4f640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f295006af30 0x7f295006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.317+0000 7f2970c4f640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f295006af30 0x7f295006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f295c007fd0 tx=0x7f295c009040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:06.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.319+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2968053bc0 con 0x7f296c070b60 2023-10-16T15:56:06.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.513+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f296c0b4620 con 0x7f296c070b60 2023-10-16T15:56:06.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.515+0000 7f2961ffb640 1 -- 172.21.15.29:0/2315550044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+17836 (secure 0 0 0) 0x7f2968017020 con 0x7f296c070b60 2023-10-16T15:56:06.517 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:06.517 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":268,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:56:02.787237+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T15:55:37.838394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T15:55:38.855075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/567930087":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/936087252":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/2036144584":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/4135571630":"2023-10-17T15:56:02.787159+0000","172.21.15.170:6833/1364949379":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/187921439":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/2485822714":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/965201049":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/831181775":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/1364949379":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/1047565771":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2742286180":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3423446827":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/1983218799":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/3865201075":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6833/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/763385128":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2667017960":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:56:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f295006af30 msgr2=0x7f295006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f295006af30 0x7f295006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f295c007fd0 tx=0x7f295c009040 comp rx=0 tx=0).stop 2023-10-16T15:56:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 msgr2=0x7f296c0a8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 0x7f296c0a8fc0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f296800c480 tx=0x7f296800c940 comp rx=0 tx=0).stop 2023-10-16T15:56:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 shutdown_connections 2023-10-16T15:56:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f295006af30 0x7f295006d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f296c159950 0x7f296c158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f296c0ca0a0 0x7f296c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 --2- 172.21.15.29:0/2315550044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f296c070b60 0x7f296c0a8fc0 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 >> 172.21.15.29:0/2315550044 conn(0x7f296c0ce6a0 msgr2=0x7f296c070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:06.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 shutdown_connections 2023-10-16T15:56:06.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.518+0000 7f2972452640 1 -- 172.21.15.29:0/2315550044 wait complete. 2023-10-16T15:56:06.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T15:56:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.768+0000 7f5f37f24640 1 Processor -- start 2023-10-16T15:56:06.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.768+0000 7f5f37f24640 1 -- start start 2023-10-16T15:56:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f37f24640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f3015d150 0x7f5f300ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f37f24640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30160b90 0x7f5f30160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f37f24640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f300ac890 con 0x7f5f30069da0 2023-10-16T15:56:06.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f37f24640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3015d530 con 0x7f5f3015d150 2023-10-16T15:56:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f37f24640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f30161650 con 0x7f5f30160b90 2023-10-16T15:56:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f35cc0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f3015d150 0x7f5f300ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f35cc0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f3015d150 0x7f5f300ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51458/0 (socket says 172.21.15.29:51458) 2023-10-16T15:56:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.770+0000 7f5f35cc0640 1 -- 172.21.15.29:0/792979017 learned_addr learned my addr 172.21.15.29:0/792979017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:06.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.771+0000 7f5f354bf640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30160b90 0x7f5f30160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.771+0000 7f5f35cc0640 1 -- 172.21.15.29:0/792979017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30160b90 msgr2=0x7f5f30160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:06.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.771+0000 7f5f35cc0640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30160b90 0x7f5f30160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.771+0000 7f5f35cc0640 1 -- 172.21.15.29:0/792979017 >> v1:172.21.15.170:6789/0 conn(0x7f5f30069da0 legacy=0x7f5f3006a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:06.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.771+0000 7f5f35cc0640 1 -- 172.21.15.29:0/792979017 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30161930 con 0x7f5f3015d150 2023-10-16T15:56:06.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.771+0000 7f5f354bf640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30160b90 0x7f5f30160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:06.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.772+0000 7f5f35cc0640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f3015d150 0x7f5f300ad350 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009870 tx=0x7f5f2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e91918893cf97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:06.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.772+0000 7f5f34cbe640 1 -- 172.21.15.29:0/792979017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c031640 con 0x7f5f3015d150 2023-10-16T15:56:06.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.772+0000 7f5f34cbe640 1 -- 172.21.15.29:0/792979017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c0317a0 con 0x7f5f3015d150 2023-10-16T15:56:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.772+0000 7f5f34cbe640 1 -- 172.21.15.29:0/792979017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c031b30 con 0x7f5f3015d150 2023-10-16T15:56:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 -- 172.21.15.29:0/792979017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f3015d150 msgr2=0x7f5f300ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f3015d150 0x7f5f300ad350 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009870 tx=0x7f5f2c02b8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 -- 172.21.15.29:0/792979017 shutdown_connections 2023-10-16T15:56:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30160b90 0x7f5f30160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 --2- 172.21.15.29:0/792979017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f3015d150 0x7f5f300ad350 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 -- 172.21.15.29:0/792979017 >> 172.21.15.29:0/792979017 conn(0x7f5f300ce9b0 msgr2=0x7f5f300cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 -- 172.21.15.29:0/792979017 shutdown_connections 2023-10-16T15:56:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.773+0000 7f5f37f24640 1 -- 172.21.15.29:0/792979017 wait complete. 2023-10-16T15:56:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.775+0000 7f5f37f24640 1 Processor -- start 2023-10-16T15:56:06.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.775+0000 7f5f37f24640 1 -- start start 2023-10-16T15:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f37f24640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30069da0 0x7f5f300c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f37f24640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f3015d150 0x7f5f300c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f37f24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 0x7f5f300c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f37f24640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f30159970 con 0x7f5f30160b90 2023-10-16T15:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f37f24640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300ae9c0 con 0x7f5f3015d150 2023-10-16T15:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f37f24640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300c38d0 con 0x7f5f30069da0 2023-10-16T15:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f364c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 0x7f5f300c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f364c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 0x7f5f300c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51466/0 (socket says 172.21.15.29:51466) 2023-10-16T15:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.776+0000 7f5f364c1640 1 -- 172.21.15.29:0/1619559701 learned_addr learned my addr 172.21.15.29:0/1619559701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f35cc0640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30069da0 0x7f5f300c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f364c1640 1 -- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30069da0 msgr2=0x7f5f300c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f364c1640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30069da0 0x7f5f300c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f354bf640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f3015d150 0x7f5f300c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f364c1640 1 -- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f3015d150 msgr2=0x7f5f300c2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f364c1640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f3015d150 0x7f5f300c2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f364c1640 1 -- 172.21.15.29:0/1619559701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c0094d0 con 0x7f5f30160b90 2023-10-16T15:56:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f354bf640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f3015d150 0x7f5f300c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f364c1640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 0x7f5f300c4090 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c780 tx=0x7f5f2800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28004b60 con 0x7f5f30160b90 2023-10-16T15:56:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2800ce90 con 0x7f5f30160b90 2023-10-16T15:56:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800a850 con 0x7f5f30160b90 2023-10-16T15:56:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.777+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f300c35f0 con 0x7f5f30160b90 2023-10-16T15:56:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.778+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f300ae650 con 0x7f5f30160b90 2023-10-16T15:56:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.779+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f300aeb50 con 0x7f5f30160b90 2023-10-16T15:56:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.780+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5f2800a9b0 con 0x7f5f30160b90 2023-10-16T15:56:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.780+0000 7f5f267fc640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5f1406af00 0x7f5f1406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.781+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f5f28089c00 con 0x7f5f30160b90 2023-10-16T15:56:06.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.781+0000 7f5f35cc0640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5f1406af00 0x7f5f1406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:06.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.782+0000 7f5f35cc0640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5f1406af00 0x7f5f1406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c036000 tx=0x7f5f2c03c040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:06.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.785+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f280548d0 con 0x7f5f30160b90 2023-10-16T15:56:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:06.845+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f28054ca0 con 0x7f5f30160b90 2023-10-16T15:56:07.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.027+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5f3004e160 con 0x7f5f30160b90 2023-10-16T15:56:07.029 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-16T15:56:07.028+0000 7fd742dd2640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T15:56:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.032+0000 7f5f267fc640 1 -- 172.21.15.29:0/1619559701 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v142) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5f28072b00 con 0x7f5f30160b90 2023-10-16T15:56:07.033 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T15:56:07.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5f1406af00 msgr2=0x7f5f1406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:07.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5f1406af00 0x7f5f1406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c036000 tx=0x7f5f2c03c040 comp rx=0 tx=0).stop 2023-10-16T15:56:07.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 msgr2=0x7f5f300c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 0x7f5f300c4090 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c780 tx=0x7f5f2800cc40 comp rx=0 tx=0).stop 2023-10-16T15:56:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 shutdown_connections 2023-10-16T15:56:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5f1406af00 0x7f5f1406d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f30160b90 0x7f5f300c4090 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5f3015d150 0x7f5f300c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 --2- 172.21.15.29:0/1619559701 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5f30069da0 0x7f5f300c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.035+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 >> 172.21.15.29:0/1619559701 conn(0x7f5f300ce9b0 msgr2=0x7f5f300ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:07.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.036+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 shutdown_connections 2023-10-16T15:56:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.036+0000 7f5f37f24640 1 -- 172.21.15.29:0/1619559701 wait complete. 2023-10-16T15:56:07.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T15:56:07.071 INFO:tasks.ceph.mds.a.smithi029.stderr: -3925> 2023-10-16T15:55:46.322+0000 7fcd85b0c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T15:56:07.137 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-16T15:56:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.265+0000 7fa85f521640 1 Processor -- start 2023-10-16T15:56:07.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.265+0000 7fa85f521640 1 -- start start 2023-10-16T15:56:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85f521640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa858159950 0x7fa85815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85f521640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fa858070450 0x7fa8580708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85f521640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85815aef0 con 0x7fa85806c100 2023-10-16T15:56:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85f521640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa858070e20 con 0x7fa858159950 2023-10-16T15:56:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85f521640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85815e490 con 0x7fa858070450 2023-10-16T15:56:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85e51f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa858159950 0x7fa85815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85e51f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa858159950 0x7fa85815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51508/0 (socket says 172.21.15.29:51508) 2023-10-16T15:56:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85e51f640 1 -- 172.21.15.29:0/1302702662 learned_addr learned my addr 172.21.15.29:0/1302702662 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.266+0000 7fa85ed20640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa85806c100 0x7fa85806c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37478/0 (socket says 172.21.15.29:37478) 2023-10-16T15:56:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.267+0000 7fa85dd1e640 1 --2- 172.21.15.29:0/1302702662 >> v2:172.21.15.170:3300/0 conn(0x7fa858070450 0x7fa8580708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.267+0000 7fa85e51f640 1 -- 172.21.15.29:0/1302702662 >> v2:172.21.15.170:3300/0 conn(0x7fa858070450 msgr2=0x7fa8580708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.267+0000 7fa85e51f640 1 --2- 172.21.15.29:0/1302702662 >> v2:172.21.15.170:3300/0 conn(0x7fa858070450 0x7fa8580708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.267+0000 7fa85e51f640 1 -- 172.21.15.29:0/1302702662 >> v1:172.21.15.29:6789/0 conn(0x7fa85806c100 legacy=0x7fa85806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.267+0000 7fa85e51f640 1 -- 172.21.15.29:0/1302702662 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85815d910 con 0x7fa858159950 2023-10-16T15:56:07.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.268+0000 7fa85e51f640 1 --2- 172.21.15.29:0/1302702662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa85815a9b0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fa8580cab70 tx=0x7fa85402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afe180ef3c350bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:07.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.268+0000 7fa85d51d640 1 -- 172.21.15.29:0/1302702662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854031870 con 0x7fa858159950 2023-10-16T15:56:07.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.268+0000 7fa85d51d640 1 -- 172.21.15.29:0/1302702662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8540319d0 con 0x7fa858159950 2023-10-16T15:56:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.268+0000 7fa85d51d640 1 -- 172.21.15.29:0/1302702662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854031d60 con 0x7fa858159950 2023-10-16T15:56:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 -- 172.21.15.29:0/1302702662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 msgr2=0x7fa85815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 --2- 172.21.15.29:0/1302702662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa85815a9b0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fa8580cab70 tx=0x7fa85402b910 comp rx=0 tx=0).stop 2023-10-16T15:56:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 -- 172.21.15.29:0/1302702662 shutdown_connections 2023-10-16T15:56:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 --2- 172.21.15.29:0/1302702662 >> v2:172.21.15.170:3300/0 conn(0x7fa858070450 0x7fa8580708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 --2- 172.21.15.29:0/1302702662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa85815a9b0 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 -- 172.21.15.29:0/1302702662 >> 172.21.15.29:0/1302702662 conn(0x7fa8580ce6a0 msgr2=0x7fa8580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 -- 172.21.15.29:0/1302702662 shutdown_connections 2023-10-16T15:56:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.269+0000 7fa85f521640 1 -- 172.21.15.29:0/1302702662 wait complete. 2023-10-16T15:56:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.271+0000 7fa85f521640 1 Processor -- start 2023-10-16T15:56:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.271+0000 7fa85f521640 1 -- start start 2023-10-16T15:56:07.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85f521640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85806c100 0x7fa8580b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:07.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85f521640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa858070450 0x7fa8580b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85f521640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa8580b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85f521640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8580b41f0 con 0x7fa858159950 2023-10-16T15:56:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85f521640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8580b35e0 con 0x7fa858070450 2023-10-16T15:56:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85f521640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8580b38c0 con 0x7fa85806c100 2023-10-16T15:56:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85ed20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa8580b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85ed20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa8580b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51520/0 (socket says 172.21.15.29:51520) 2023-10-16T15:56:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.272+0000 7fa85ed20640 1 -- 172.21.15.29:0/2289997020 learned_addr learned my addr 172.21.15.29:0/2289997020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85e51f640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85806c100 0x7fa8580b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85dd1e640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa858070450 0x7fa8580b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85ed20640 1 -- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85806c100 msgr2=0x7fa8580b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85ed20640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85806c100 0x7fa8580b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85ed20640 1 -- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa858070450 msgr2=0x7fa8580b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85ed20640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa858070450 0x7fa8580b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85ed20640 1 -- 172.21.15.29:0/2289997020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8540094d0 con 0x7fa858159950 2023-10-16T15:56:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.273+0000 7fa85ed20640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa8580b12c0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c780 tx=0x7fa85000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.274+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85000ce90 con 0x7fa858159950 2023-10-16T15:56:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.274+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85000a640 con 0x7fa858159950 2023-10-16T15:56:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.274+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85000a9f0 con 0x7fa858159950 2023-10-16T15:56:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.274+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8580b3ba0 con 0x7fa858159950 2023-10-16T15:56:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.274+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8580b3220 con 0x7fa858159950 2023-10-16T15:56:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.275+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8580c5770 con 0x7fa858159950 2023-10-16T15:56:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.276+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa85000ab50 con 0x7fa858159950 2023-10-16T15:56:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.277+0000 7fa84effd640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fa83c06afe0 0x7fa83c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.277+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6683+0+0 (secure 0 0 0) 0x7fa850089100 con 0x7fa858159950 2023-10-16T15:56:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.278+0000 7fa85e51f640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fa83c06afe0 0x7fa83c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.279+0000 7fa85e51f640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fa83c06afe0 0x7fa83c06d4a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa854036000 tx=0x7fa85403c040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.281+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8580c5770 con 0x7fa858159950 2023-10-16T15:56:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:07.514+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa85815aef0 con 0x7fa858159950 2023-10-16T15:56:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.036+0000 7fa84effd640 1 -- 172.21.15.29:0/2289997020 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v143) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa85815aef0 con 0x7fa858159950 2023-10-16T15:56:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fa83c06afe0 msgr2=0x7fa83c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fa83c06afe0 0x7fa83c06d4a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa854036000 tx=0x7fa85403c040 comp rx=0 tx=0).stop 2023-10-16T15:56:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 msgr2=0x7fa8580b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa8580b12c0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c780 tx=0x7fa85000cc40 comp rx=0 tx=0).stop 2023-10-16T15:56:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 shutdown_connections 2023-10-16T15:56:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fa83c06afe0 0x7fa83c06d4a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa858159950 0x7fa8580b12c0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa858070450 0x7fa8580b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 --2- 172.21.15.29:0/2289997020 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fa85806c100 0x7fa8580b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 >> 172.21.15.29:0/2289997020 conn(0x7fa8580ce6a0 msgr2=0x7fa8580ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 shutdown_connections 2023-10-16T15:56:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.039+0000 7fa85f521640 1 -- 172.21.15.29:0/2289997020 wait complete. 2023-10-16T15:56:08.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T15:56:08.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.291+0000 7f6e83b97640 1 Processor -- start 2023-10-16T15:56:08.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.292+0000 7f6e83b97640 1 -- start start 2023-10-16T15:56:08.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.293+0000 7f6e83b97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e7c15cf60 0x7f6e7c06f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:08.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.293+0000 7f6e83b97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c06f9d0 0x7f6e7c0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:08.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.293+0000 7f6e83b97640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c0c9620 con 0x7f6e7c0c9ff0 2023-10-16T15:56:08.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.293+0000 7f6e83b97640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c15d340 con 0x7f6e7c15cf60 2023-10-16T15:56:08.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.293+0000 7f6e83b97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0cc860 con 0x7f6e7c06f9d0 2023-10-16T15:56:08.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e7c15cf60 0x7f6e7c06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:08.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e7c15cf60 0x7f6e7c06f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51522/0 (socket says 172.21.15.29:51522) 2023-10-16T15:56:08.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 -- 172.21.15.29:0/2887213623 learned_addr learned my addr 172.21.15.29:0/2887213623 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:08.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 -- 172.21.15.29:0/2887213623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c06f9d0 msgr2=0x7f6e7c0c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:08.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81132640 1 --2- 172.21.15.29:0/2887213623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c06f9d0 0x7f6e7c0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:08.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 --2- 172.21.15.29:0/2887213623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c06f9d0 0x7f6e7c0c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 -- 172.21.15.29:0/2887213623 >> v1:172.21.15.170:6789/0 conn(0x7f6e7c0c9ff0 legacy=0x7f6e7c0b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:08.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.294+0000 7f6e81933640 1 -- 172.21.15.29:0/2887213623 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c0ad4a0 con 0x7f6e7c15cf60 2023-10-16T15:56:08.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.295+0000 7f6e81933640 1 --2- 172.21.15.29:0/2887213623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06f490 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009e50 tx=0x7f6e7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fe80ca8f7a86cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:08.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.296+0000 7f6e80931640 1 -- 172.21.15.29:0/2887213623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7802ca70 con 0x7f6e7c15cf60 2023-10-16T15:56:08.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.296+0000 7f6e80931640 1 -- 172.21.15.29:0/2887213623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7802cbd0 con 0x7f6e7c15cf60 2023-10-16T15:56:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.296+0000 7f6e80931640 1 -- 172.21.15.29:0/2887213623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780343f0 con 0x7f6e7c15cf60 2023-10-16T15:56:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.296+0000 7f6e83b97640 1 -- 172.21.15.29:0/2887213623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 msgr2=0x7f6e7c06f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.296+0000 7f6e83b97640 1 --2- 172.21.15.29:0/2887213623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06f490 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009e50 tx=0x7f6e7802bed0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.297+0000 7f6e83b97640 1 -- 172.21.15.29:0/2887213623 shutdown_connections 2023-10-16T15:56:08.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.297+0000 7f6e83b97640 1 --2- 172.21.15.29:0/2887213623 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c06f9d0 0x7f6e7c0c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.297+0000 7f6e83b97640 1 --2- 172.21.15.29:0/2887213623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06f490 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.297+0000 7f6e83b97640 1 -- 172.21.15.29:0/2887213623 >> 172.21.15.29:0/2887213623 conn(0x7f6e7c0cb360 msgr2=0x7f6e7c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:08.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.297+0000 7f6e83b97640 1 -- 172.21.15.29:0/2887213623 shutdown_connections 2023-10-16T15:56:08.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.297+0000 7f6e83b97640 1 -- 172.21.15.29:0/2887213623 wait complete. 2023-10-16T15:56:08.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.298+0000 7f6e83b97640 1 Processor -- start 2023-10-16T15:56:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.299+0000 7f6e83b97640 1 -- start start 2023-10-16T15:56:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e83b97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e7c06f9d0 0x7f6e7c1548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e83b97640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c0c9ff0 0x7f6e7c157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e83b97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e83b97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c158190 con 0x7f6e7c15cf60 2023-10-16T15:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e83b97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c2c00 con 0x7f6e7c06f9d0 2023-10-16T15:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e83b97640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c157eb0 con 0x7f6e7c0c9ff0 2023-10-16T15:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51534/0 (socket says 172.21.15.29:51534) 2023-10-16T15:56:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 -- 172.21.15.29:0/733566569 learned_addr learned my addr 172.21.15.29:0/733566569 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e81132640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c0c9ff0 0x7f6e7c157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e81933640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e7c06f9d0 0x7f6e7c1548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 -- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c0c9ff0 msgr2=0x7f6e7c157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c0c9ff0 0x7f6e7c157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 -- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e7c06f9d0 msgr2=0x7f6e7c1548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e7c06f9d0 0x7f6e7c1548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.300+0000 7f6e82134640 1 -- 172.21.15.29:0/733566569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78009b30 con 0x7f6e7c15cf60 2023-10-16T15:56:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.301+0000 7f6e82134640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06fe90 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400b440 tx=0x7f6e7400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.301+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74011830 con 0x7f6e7c15cf60 2023-10-16T15:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.301+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e74011990 con 0x7f6e7c15cf60 2023-10-16T15:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.301+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c0c25b0 con 0x7f6e7c15cf60 2023-10-16T15:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.301+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74011d40 con 0x7f6e7c15cf60 2023-10-16T15:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.301+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c1550e0 con 0x7f6e7c15cf60 2023-10-16T15:56:08.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.303+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c0c2410 con 0x7f6e7c15cf60 2023-10-16T15:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.303+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e7401a400 con 0x7f6e7c15cf60 2023-10-16T15:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.304+0000 7f6e727fc640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6e6006af90 0x7f6e6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.304+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6683+0+0 (secure 0 0 0) 0x7f6e740898c0 con 0x7f6e7c15cf60 2023-10-16T15:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.305+0000 7f6e81933640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6e6006af90 0x7f6e6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.306+0000 7f6e81933640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6e6006af90 0x7f6e6006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e78035000 tx=0x7f6e78030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.309+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e7c0c2410 con 0x7f6e7c15cf60 2023-10-16T15:56:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:08.505+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6e7c0c2410 con 0x7f6e7c15cf60 2023-10-16T15:56:09.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.039+0000 7f6e727fc640 1 -- 172.21.15.29:0/733566569 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v270) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6e7c0c2410 con 0x7f6e7c15cf60 2023-10-16T15:56:09.041 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-16T15:56:09.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6e6006af90 msgr2=0x7f6e6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:09.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6e6006af90 0x7f6e6006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e78035000 tx=0x7f6e78030000 comp rx=0 tx=0).stop 2023-10-16T15:56:09.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 msgr2=0x7f6e7c06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:09.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06fe90 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400b440 tx=0x7f6e7400b900 comp rx=0 tx=0).stop 2023-10-16T15:56:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 shutdown_connections 2023-10-16T15:56:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6e6006af90 0x7f6e6006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e7c15cf60 0x7f6e7c06fe90 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6e7c0c9ff0 0x7f6e7c157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 --2- 172.21.15.29:0/733566569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e7c06f9d0 0x7f6e7c1548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.043+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 >> 172.21.15.29:0/733566569 conn(0x7f6e7c0cb360 msgr2=0x7f6e7c06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.045+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 shutdown_connections 2023-10-16T15:56:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.046+0000 7f6e83b97640 1 -- 172.21.15.29:0/733566569 wait complete. 2023-10-16T15:56:09.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T15:56:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.288+0000 7f6fbc032640 1 Processor -- start 2023-10-16T15:56:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.288+0000 7f6fbc032640 1 -- start start 2023-10-16T15:56:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.289+0000 7f6fbc032640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 0x7f6fb40b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.289+0000 7f6fbc032640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6fb40ca510 0x7f6fb40ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.289+0000 7f6fbc032640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb40b0d00 con 0x7f6fb415ac10 2023-10-16T15:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.289+0000 7f6fbc032640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb415f800 con 0x7f6fb40ca510 2023-10-16T15:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.289+0000 7f6fbc032640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb406ce40 con 0x7f6fb40b0360 2023-10-16T15:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.290+0000 7f6fb95cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 0x7f6fb40b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.290+0000 7f6fba5cf640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f6fb40ca510 0x7f6fb40ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.290+0000 7f6fb95cd640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 0x7f6fb40b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3301/0 says I am v2:172.21.15.29:54652/0 (socket says 172.21.15.29:54652) 2023-10-16T15:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.290+0000 7f6fb95cd640 1 -- 172.21.15.29:0/7198177 learned_addr learned my addr 172.21.15.29:0/7198177 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.291+0000 7f6fb95cd640 1 -- 172.21.15.29:0/7198177 >> v2:172.21.15.170:3300/0 conn(0x7f6fb40ca510 msgr2=0x7f6fb40ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.291+0000 7f6fb95cd640 1 --2- 172.21.15.29:0/7198177 >> v2:172.21.15.170:3300/0 conn(0x7f6fb40ca510 0x7f6fb40ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.291+0000 7f6fb95cd640 1 -- 172.21.15.29:0/7198177 >> v1:172.21.15.170:6789/0 conn(0x7f6fb415ac10 legacy=0x7f6fb406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.291+0000 7f6fb95cd640 1 -- 172.21.15.29:0/7198177 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb415b310 con 0x7f6fb40b0360 2023-10-16T15:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.292+0000 7f6fb95cd640 1 --2- 172.21.15.29:0/7198177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 0x7f6fb40b07c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009870 tx=0x7f6fa002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ba046d10093d697 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.292+0000 7f6fb8dcc640 1 -- 172.21.15.29:0/7198177 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0031640 con 0x7f6fb40b0360 2023-10-16T15:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.292+0000 7f6fb8dcc640 1 -- 172.21.15.29:0/7198177 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa00317a0 con 0x7f6fb40b0360 2023-10-16T15:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.292+0000 7f6fb8dcc640 1 -- 172.21.15.29:0/7198177 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0031b30 con 0x7f6fb40b0360 2023-10-16T15:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 -- 172.21.15.29:0/7198177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 msgr2=0x7f6fb40b07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 --2- 172.21.15.29:0/7198177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 0x7f6fb40b07c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009870 tx=0x7f6fa002b8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 -- 172.21.15.29:0/7198177 shutdown_connections 2023-10-16T15:56:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 --2- 172.21.15.29:0/7198177 >> v2:172.21.15.170:3300/0 conn(0x7f6fb40ca510 0x7f6fb40ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 --2- 172.21.15.29:0/7198177 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb40b0360 0x7f6fb40b07c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 -- 172.21.15.29:0/7198177 >> 172.21.15.29:0/7198177 conn(0x7f6fb40cb360 msgr2=0x7f6fb40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 -- 172.21.15.29:0/7198177 shutdown_connections 2023-10-16T15:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.293+0000 7f6fbc032640 1 -- 172.21.15.29:0/7198177 wait complete. 2023-10-16T15:56:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.295+0000 7f6fbc032640 1 Processor -- start 2023-10-16T15:56:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.295+0000 7f6fbc032640 1 -- start start 2023-10-16T15:56:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.296+0000 7f6fbc032640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 0x7f6fb40c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.296+0000 7f6fbc032640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fb40ca510 0x7f6fb40c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.296+0000 7f6fbc032640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb415ac10 0x7f6fb40cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.296+0000 7f6fbc032640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40cd130 con 0x7f6fb40b0360 2023-10-16T15:56:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.296+0000 7f6fbc032640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40cc4b0 con 0x7f6fb40ca510 2023-10-16T15:56:09.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.296+0000 7f6fbc032640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40cc790 con 0x7f6fb415ac10 2023-10-16T15:56:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 0x7f6fb40c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 0x7f6fb40c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51544/0 (socket says 172.21.15.29:51544) 2023-10-16T15:56:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 -- 172.21.15.29:0/2829674669 learned_addr learned my addr 172.21.15.29:0/2829674669 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:09.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fba5cf640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb415ac10 0x7f6fb40cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb95cd640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fb40ca510 0x7f6fb40c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 -- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb415ac10 msgr2=0x7f6fb40cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb415ac10 0x7f6fb40cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 -- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fb40ca510 msgr2=0x7f6fb40c4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:09.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fb40ca510 0x7f6fb40c4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.297+0000 7f6fb9dce640 1 -- 172.21.15.29:0/2829674669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa00094d0 con 0x7f6fb40b0360 2023-10-16T15:56:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.298+0000 7f6fb9dce640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 0x7f6fb40c6a10 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800c870 tx=0x7f6fa800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.298+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003b70 con 0x7f6fb40b0360 2023-10-16T15:56:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.298+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8003cf0 con 0x7f6fb40b0360 2023-10-16T15:56:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.298+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb40cca70 con 0x7f6fb40b0360 2023-10-16T15:56:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.298+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa800aa90 con 0x7f6fb40b0360 2023-10-16T15:56:09.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.298+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb40c5c40 con 0x7f6fb40b0360 2023-10-16T15:56:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.299+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb40cd270 con 0x7f6fb40b0360 2023-10-16T15:56:09.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.300+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6fa800abf0 con 0x7f6fb40b0360 2023-10-16T15:56:09.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.301+0000 7f6fb27fc640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6f9806af90 0x7f6f9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.301+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6fa8089890 con 0x7f6fb40b0360 2023-10-16T15:56:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.302+0000 7f6fb95cd640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6f9806af90 0x7f6f9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.303+0000 7f6fb95cd640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6f9806af90 0x7f6f9806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0036000 tx=0x7f6fa003c040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:09.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.305+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6fb40cd270 con 0x7f6fb40b0360 2023-10-16T15:56:09.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:09.501+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6fb40cd270 con 0x7f6fb40b0360 2023-10-16T15:56:10.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.069+0000 7f6fb27fc640 1 -- 172.21.15.29:0/2829674669 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v271) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6fb40cd270 con 0x7f6fb40b0360 2023-10-16T15:56:10.070 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-16T15:56:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.073+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6f9806af90 msgr2=0x7f6f9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.073+0000 7f6fbc032640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6f9806af90 0x7f6f9806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0036000 tx=0x7f6fa003c040 comp rx=0 tx=0).stop 2023-10-16T15:56:10.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.073+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 msgr2=0x7f6fb40c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.073+0000 7f6fbc032640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 0x7f6fb40c6a10 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800c870 tx=0x7f6fa800cd30 comp rx=0 tx=0).stop 2023-10-16T15:56:10.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.075+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 shutdown_connections 2023-10-16T15:56:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.075+0000 7f6fbc032640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6f9806af90 0x7f6f9806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.075+0000 7f6fbc032640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6fb415ac10 0x7f6fb40cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.075+0000 7f6fbc032640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6fb40ca510 0x7f6fb40c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.075+0000 7f6fbc032640 1 --2- 172.21.15.29:0/2829674669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fb40b0360 0x7f6fb40c6a10 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.075+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 >> 172.21.15.29:0/2829674669 conn(0x7f6fb40cb360 msgr2=0x7f6fb406bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.077+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 shutdown_connections 2023-10-16T15:56:10.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.077+0000 7f6fbc032640 1 -- 172.21.15.29:0/2829674669 wait complete. 2023-10-16T15:56:10.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T15:56:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.314+0000 7f7f6b59e640 1 Processor -- start 2023-10-16T15:56:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.314+0000 7f7f6b59e640 1 -- start start 2023-10-16T15:56:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 0x7f7f6c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6b59e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f7f6c0cda80 0x7f7f6c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c06c960 con 0x7f7f6c0ad970 2023-10-16T15:56:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6b59e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6c15b9d0 con 0x7f7f6c0cda80 2023-10-16T15:56:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c15bcb0 con 0x7f7f6c06c040 2023-10-16T15:56:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f63fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7f6c0ad970 0x7f7f6c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37490/0 (socket says 172.21.15.29:37490) 2023-10-16T15:56:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f63fff640 1 -- 172.21.15.29:0/3797210620 learned_addr learned my addr 172.21.15.29:0/3797210620 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6ad9d640 1 --2- 172.21.15.29:0/3797210620 >> v2:172.21.15.170:3300/0 conn(0x7f7f6c0cda80 0x7f7f6c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.316+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 0x7f7f6c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534041333 0 0) 0x7f7f6c06c960 con 0x7f7f6c0ad970 2023-10-16T15:56:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f58003520 con 0x7f7f6c0ad970 2023-10-16T15:56:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048506206 0 0) 0x7f7f58003520 con 0x7f7f6c0ad970 2023-10-16T15:56:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 msgr2=0x7f7f6c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 0x7f7f6c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 >> v2:172.21.15.170:3300/0 conn(0x7f7f6c0cda80 msgr2=0x7f7f6c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 --2- 172.21.15.29:0/3797210620 >> v2:172.21.15.170:3300/0 conn(0x7f7f6c0cda80 0x7f7f6c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c15bf90 con 0x7f7f6c0ad970 2023-10-16T15:56:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f7f54001670 con 0x7f7f6c0ad970 2023-10-16T15:56:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 >> v1:172.21.15.29:6789/0 conn(0x7f7f6c0ad970 legacy=0x7f7f6c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f58003970 0x7f7f58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 0x7f7f580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58004d00 0x7f7f58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c15b9d0 con 0x7f7f58003970 2023-10-16T15:56:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c15bcb0 con 0x7f7f58004d00 2023-10-16T15:56:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5800b980 con 0x7f7f6c06c040 2023-10-16T15:56:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.317+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f58003970 0x7f7f58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6a59c640 1 -- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 msgr2=0x7f7f580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f6c06c040 0x7f7f580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6ad9d640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58004d00 0x7f7f58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6a59c640 1 -- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58004d00 msgr2=0x7f7f58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58004d00 0x7f7f58005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6a59c640 1 -- 172.21.15.29:0/3797210620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f58003520 con 0x7f7f58003970 2023-10-16T15:56:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.318+0000 7f7f6ad9d640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58004d00 0x7f7f58005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.319+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f58003970 0x7f7f58003d90 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009ac0 tx=0x7f7f580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebe60e52e9f8f28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.319+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c02c710 con 0x7f7f58003970 2023-10-16T15:56:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.319+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c02c870 con 0x7f7f58003970 2023-10-16T15:56:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.319+0000 7f7f69d9b640 1 -- 172.21.15.29:0/3797210620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c02cc00 con 0x7f7f58003970 2023-10-16T15:56:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3797210620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f58003970 msgr2=0x7f7f58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3797210620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f58003970 0x7f7f58003d90 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009ac0 tx=0x7f7f580057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6ad9d640 1 -- 172.21.15.29:0/3797210620 reap_dead start 2023-10-16T15:56:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3797210620 shutdown_connections 2023-10-16T15:56:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3797210620 >> 172.21.15.29:0/3797210620 conn(0x7f7f6c0ce6a0 msgr2=0x7f7f6c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3797210620 shutdown_connections 2023-10-16T15:56:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.320+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3797210620 wait complete. 2023-10-16T15:56:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.322+0000 7f7f6b59e640 1 Processor -- start 2023-10-16T15:56:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.322+0000 7f7f6b59e640 1 -- start start 2023-10-16T15:56:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f58003970 0x7f7f6c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f6c0ae0c0 0x7f7f6c15d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 0x7f7f6c15d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0c5810 con 0x7f7f6c0ae540 2023-10-16T15:56:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0af740 con 0x7f7f6c0ae0c0 2023-10-16T15:56:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c1f2a60 con 0x7f7f58003970 2023-10-16T15:56:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 0x7f7f6c15d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 0x7f7f6c15d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51558/0 (socket says 172.21.15.29:51558) 2023-10-16T15:56:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.323+0000 7f7f6ad9d640 1 -- 172.21.15.29:0/3031062922 learned_addr learned my addr 172.21.15.29:0/3031062922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f63fff640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f6c0ae0c0 0x7f7f6c15d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f58003970 0x7f7f6c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6ad9d640 1 -- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f58003970 msgr2=0x7f7f6c0c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6ad9d640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f58003970 0x7f7f6c0c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6ad9d640 1 -- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f6c0ae0c0 msgr2=0x7f7f6c15d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6ad9d640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f6c0ae0c0 0x7f7f6c15d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6ad9d640 1 -- 172.21.15.29:0/3031062922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c1f2d40 con 0x7f7f6c0ae540 2023-10-16T15:56:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6ad9d640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 0x7f7f6c15d760 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c06d2e0 tx=0x7f7f6400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64016620 con 0x7f7f6c0ae540 2023-10-16T15:56:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f64016780 con 0x7f7f6c0ae540 2023-10-16T15:56:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64016b10 con 0x7f7f6c0ae540 2023-10-16T15:56:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.324+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c1f3020 con 0x7f7f6c0ae540 2023-10-16T15:56:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.325+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c0c5cf0 con 0x7f7f6c0ae540 2023-10-16T15:56:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.326+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c06b330 con 0x7f7f6c0ae540 2023-10-16T15:56:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.327+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f64016c70 con 0x7f7f6c0ae540 2023-10-16T15:56:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.328+0000 7f7f62ffd640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f7f4c06afc0 0x7f7f4c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.328+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f7f6408a670 con 0x7f7f6c0ae540 2023-10-16T15:56:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.329+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f7f4c06afc0 0x7f7f4c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.330+0000 7f7f6a59c640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f7f4c06afc0 0x7f7f4c06d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c036000 tx=0x7f7f5c030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.334+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f64055490 con 0x7f7f6c0ae540 2023-10-16T15:56:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.524+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7f6c04e9b0 con 0x7f7f6c0ae540 2023-10-16T15:56:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.532+0000 7f7f62ffd640 1 -- 172.21.15.29:0/3031062922 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7f64058ab0 con 0x7f7f6c0ae540 2023-10-16T15:56:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.534+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f7f4c06afc0 msgr2=0x7f7f4c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.534+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f7f4c06afc0 0x7f7f4c06d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c036000 tx=0x7f7f5c030000 comp rx=0 tx=0).stop 2023-10-16T15:56:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.534+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 msgr2=0x7f7f6c15d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.534+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 0x7f7f6c15d760 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c06d2e0 tx=0x7f7f6400f520 comp rx=0 tx=0).stop 2023-10-16T15:56:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 shutdown_connections 2023-10-16T15:56:10.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f7f4c06afc0 0x7f7f4c06d480 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c036000 tx=0x7f7f5c030000 comp rx=0 tx=0).stop 2023-10-16T15:56:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f6c0ae540 0x7f7f6c15d760 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f6c0ae0c0 0x7f7f6c15d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 --2- 172.21.15.29:0/3031062922 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f7f58003970 0x7f7f6c0c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 >> 172.21.15.29:0/3031062922 conn(0x7f7f6c0ce6a0 msgr2=0x7f7f6c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 shutdown_connections 2023-10-16T15:56:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.535+0000 7f7f6b59e640 1 -- 172.21.15.29:0/3031062922 wait complete. 2023-10-16T15:56:10.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T15:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.767+0000 7fbf8b59e640 1 Processor -- start 2023-10-16T15:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.767+0000 7fbf8b59e640 1 -- start start 2023-10-16T15:56:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.769+0000 7fbf8b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 0x7fbf8c070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.769+0000 7fbf8b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c0cd260 con 0x7fbf8c15d650 2023-10-16T15:56:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.769+0000 7fbf8b59e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c15de30 con 0x7fbf8c0c9de0 2023-10-16T15:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.769+0000 7fbf8b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c071470 con 0x7fbf8c159ad0 2023-10-16T15:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.769+0000 7fbf89d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf8c15d650 0x7fbf8c06aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37496/0 (socket says 172.21.15.29:37496) 2023-10-16T15:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.769+0000 7fbf89d9b640 1 -- 172.21.15.29:0/1371135402 learned_addr learned my addr 172.21.15.29:0/1371135402 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8a59c640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 0x7fbf8c070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198266402 0 0) 0x7fbf8c0cd260 con 0x7fbf8c15d650 2023-10-16T15:56:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf74003520 con 0x7fbf8c15d650 2023-10-16T15:56:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465382591 0 0) 0x7fbf74003520 con 0x7fbf8c15d650 2023-10-16T15:56:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 msgr2=0x7fbf8c070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 0x7fbf8c070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 >> v1:172.21.15.170:6789/0 conn(0x7fbf8c0c9de0 legacy=0x7fbf8c0cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c15e110 con 0x7fbf8c15d650 2023-10-16T15:56:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fbf80001670 con 0x7fbf8c15d650 2023-10-16T15:56:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 >> v1:172.21.15.29:6789/0 conn(0x7fbf8c15d650 legacy=0x7fbf8c06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740039a0 0x7fbf74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 0x7fbf740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.770+0000 7fbf8959a640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf74004d00 0x7fbf74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c071470 con 0x7fbf74004d00 2023-10-16T15:56:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7400ba60 con 0x7fbf740039a0 2023-10-16T15:56:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7400de30 con 0x7fbf8c159ad0 2023-10-16T15:56:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8ad9d640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf74004d00 0x7fbf74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8a59c640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740039a0 0x7fbf74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf89d9b640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 0x7fbf740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8ad9d640 1 -- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 msgr2=0x7fbf740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8ad9d640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c159ad0 0x7fbf740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8ad9d640 1 -- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740039a0 msgr2=0x7fbf74003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8ad9d640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf740039a0 0x7fbf74003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.771+0000 7fbf8ad9d640 1 -- 172.21.15.29:0/1371135402 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74003520 con 0x7fbf74004d00 2023-10-16T15:56:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.773+0000 7fbf8ad9d640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf74004d00 0x7fbf74005180 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7fbf84001e20 tx=0x7fbf740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73fb0f0aff793e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.773+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84003b70 con 0x7fbf74004d00 2023-10-16T15:56:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.773+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84003cd0 con 0x7fbf74004d00 2023-10-16T15:56:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.773+0000 7fbf8959a640 1 -- 172.21.15.29:0/1371135402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84005310 con 0x7fbf74004d00 2023-10-16T15:56:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8b59e640 1 -- 172.21.15.29:0/1371135402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf74004d00 msgr2=0x7fbf74005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/1371135402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf74004d00 0x7fbf74005180 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7fbf84001e20 tx=0x7fbf740057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8ad9d640 1 -- 172.21.15.29:0/1371135402 reap_dead start 2023-10-16T15:56:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8b59e640 1 -- 172.21.15.29:0/1371135402 shutdown_connections 2023-10-16T15:56:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8b59e640 1 -- 172.21.15.29:0/1371135402 >> 172.21.15.29:0/1371135402 conn(0x7fbf8c0b0170 msgr2=0x7fbf8c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8b59e640 1 -- 172.21.15.29:0/1371135402 shutdown_connections 2023-10-16T15:56:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.774+0000 7fbf8b59e640 1 -- 172.21.15.29:0/1371135402 wait complete. 2023-10-16T15:56:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.776+0000 7fbf8b59e640 1 Processor -- start 2023-10-16T15:56:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.776+0000 7fbf8b59e640 1 -- start start 2023-10-16T15:56:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8b59e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf74004d00 0x7fbf8c15cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 0x7fbf8c1f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8b59e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c0ac890 0x7fbf8c0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0ad280 con 0x7fbf8c15bd60 2023-10-16T15:56:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8b59e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0b3940 con 0x7fbf74004d00 2023-10-16T15:56:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8b59e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c15d320 con 0x7fbf8c0ac890 2023-10-16T15:56:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf89d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 0x7fbf8c1f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf89d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 0x7fbf8c1f1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51578/0 (socket says 172.21.15.29:51578) 2023-10-16T15:56:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf89d9b640 1 -- 172.21.15.29:0/4057816100 learned_addr learned my addr 172.21.15.29:0/4057816100 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.777+0000 7fbf8a59c640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf74004d00 0x7fbf8c15cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf8ad9d640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c0ac890 0x7fbf8c0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf89d9b640 1 -- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c0ac890 msgr2=0x7fbf8c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf89d9b640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c0ac890 0x7fbf8c0acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf89d9b640 1 -- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf74004d00 msgr2=0x7fbf8c15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf89d9b640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf74004d00 0x7fbf8c15cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf89d9b640 1 -- 172.21.15.29:0/4057816100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0b34b0 con 0x7fbf8c15bd60 2023-10-16T15:56:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf89d9b640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 0x7fbf8c1f1d00 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7fbf80001820 tx=0x7fbf8000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000c6c0 con 0x7fbf8c15bd60 2023-10-16T15:56:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8000c820 con 0x7fbf8c15bd60 2023-10-16T15:56:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000cbd0 con 0x7fbf8c15bd60 2023-10-16T15:56:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.778+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c1f2240 con 0x7fbf8c15bd60 2023-10-16T15:56:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.779+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c0b2b80 con 0x7fbf8c15bd60 2023-10-16T15:56:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.780+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c14d420 con 0x7fbf8c15bd60 2023-10-16T15:56:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.781+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf8001d460 con 0x7fbf8c15bd60 2023-10-16T15:56:10.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.782+0000 7fbf7affd640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fbf6c06af90 0x7fbf6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:10.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.782+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fbf80089990 con 0x7fbf8c15bd60 2023-10-16T15:56:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.782+0000 7fbf8a59c640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fbf6c06af90 0x7fbf6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.784+0000 7fbf8a59c640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fbf6c06af90 0x7fbf6c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009a10 tx=0x7fbf7c008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.786+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf8c14d420 con 0x7fbf8c15bd60 2023-10-16T15:56:10.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.979+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbf8c14d420 con 0x7fbf8c15bd60 2023-10-16T15:56:10.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.985+0000 7fbf7affd640 1 -- 172.21.15.29:0/4057816100 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v92) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbf8c14d420 con 0x7fbf8c15bd60 2023-10-16T15:56:10.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.987+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fbf6c06af90 msgr2=0x7fbf6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.987+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fbf6c06af90 0x7fbf6c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009a10 tx=0x7fbf7c008040 comp rx=0 tx=0).stop 2023-10-16T15:56:10.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.987+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 msgr2=0x7fbf8c1f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:10.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.987+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 0x7fbf8c1f1d00 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7fbf80001820 tx=0x7fbf8000eb90 comp rx=0 tx=0).stop 2023-10-16T15:56:10.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 shutdown_connections 2023-10-16T15:56:10.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fbf6c06af90 0x7fbf6c06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf8c15bd60 0x7fbf8c1f1d00 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fbf8c0ac890 0x7fbf8c0acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 --2- 172.21.15.29:0/4057816100 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbf74004d00 0x7fbf8c15cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:10.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 >> 172.21.15.29:0/4057816100 conn(0x7fbf8c0b0170 msgr2=0x7fbf8c156530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:10.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 shutdown_connections 2023-10-16T15:56:10.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:10.988+0000 7fbf8b59e640 1 -- 172.21.15.29:0/4057816100 wait complete. 2023-10-16T15:56:11.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T15:56:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.231+0000 7fb17a5d0640 1 Processor -- start 2023-10-16T15:56:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.231+0000 7fb17a5d0640 1 -- start start 2023-10-16T15:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb17a5d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 0x7fb1740d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb17a5d0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb1740b8480 0x7fb1740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb17a5d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17415d150 con 0x7fb1740d0be0 2023-10-16T15:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb17a5d0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1740d1180 con 0x7fb1740b8480 2023-10-16T15:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb17a5d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740d1460 con 0x7fb17415cd70 2023-10-16T15:56:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb179dcf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb1740d0be0 0x7fb1740b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37512/0 (socket says 172.21.15.29:37512) 2023-10-16T15:56:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.233+0000 7fb179dcf640 1 -- 172.21.15.29:0/2563671393 learned_addr learned my addr 172.21.15.29:0/2563671393 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb178dcd640 1 --2- 172.21.15.29:0/2563671393 >> v2:172.21.15.170:3300/0 conn(0x7fb1740b8480 0x7fb1740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 0x7fb1740d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458860279 0 0) 0x7fb17415d150 con 0x7fb1740d0be0 2023-10-16T15:56:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb160003520 con 0x7fb1740d0be0 2023-10-16T15:56:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856726795 0 0) 0x7fb160003520 con 0x7fb1740d0be0 2023-10-16T15:56:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 msgr2=0x7fb1740d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:11.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 0x7fb1740d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 >> v2:172.21.15.170:3300/0 conn(0x7fb1740b8480 msgr2=0x7fb1740cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2563671393 >> v2:172.21.15.170:3300/0 conn(0x7fb1740b8480 0x7fb1740cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1740d1740 con 0x7fb1740d0be0 2023-10-16T15:56:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fb16c001670 con 0x7fb1740d0be0 2023-10-16T15:56:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 >> v1:172.21.15.29:6789/0 conn(0x7fb1740d0be0 legacy=0x7fb1740b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb160003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1600047c0 0x7fb160004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 0x7fb160005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740d1180 con 0x7fb160003970 2023-10-16T15:56:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.234+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740d1460 con 0x7fb1600047c0 2023-10-16T15:56:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16000b980 con 0x7fb17415cd70 2023-10-16T15:56:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb160003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb178dcd640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1600047c0 0x7fb160004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb179dcf640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 0x7fb160005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb1795ce640 1 -- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 msgr2=0x7fb160005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415cd70 0x7fb160005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb1795ce640 1 -- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1600047c0 msgr2=0x7fb160004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:11.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1600047c0 0x7fb160004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.235+0000 7fb1795ce640 1 -- 172.21.15.29:0/2563671393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160003520 con 0x7fb160003970 2023-10-16T15:56:11.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.237+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb160003d90 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fb170009fc0 tx=0x7fb1600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bca602277ebf57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:11.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.237+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17002cc00 con 0x7fb160003970 2023-10-16T15:56:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.237+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17000f040 con 0x7fb160003970 2023-10-16T15:56:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.237+0000 7fb16bfff640 1 -- 172.21.15.29:0/2563671393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170034440 con 0x7fb160003970 2023-10-16T15:56:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.237+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2563671393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 msgr2=0x7fb160003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.237+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2563671393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb160003d90 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fb170009fc0 tx=0x7fb1600057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.238+0000 7fb179dcf640 1 -- 172.21.15.29:0/2563671393 reap_dead start 2023-10-16T15:56:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.238+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2563671393 shutdown_connections 2023-10-16T15:56:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.238+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2563671393 >> 172.21.15.29:0/2563671393 conn(0x7fb1740cf570 msgr2=0x7fb1740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.238+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2563671393 shutdown_connections 2023-10-16T15:56:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.238+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2563671393 wait complete. 2023-10-16T15:56:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.242+0000 7fb17a5d0640 1 Processor -- start 2023-10-16T15:56:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.243+0000 7fb17a5d0640 1 -- start start 2023-10-16T15:56:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb17a5d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb174151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb17a5d0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415e3b0 0x7fb17415fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb17a5d0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb17406bbd0 0x7fb17406c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb17a5d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17406c5a0 con 0x7fb160003970 2023-10-16T15:56:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb17a5d0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17415e920 con 0x7fb17406bbd0 2023-10-16T15:56:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb17a5d0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17415ec00 con 0x7fb17415e3b0 2023-10-16T15:56:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb1795ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb174151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb1795ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb174151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51606/0 (socket says 172.21.15.29:51606) 2023-10-16T15:56:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.245+0000 7fb1795ce640 1 -- 172.21.15.29:0/2859331834 learned_addr learned my addr 172.21.15.29:0/2859331834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb178dcd640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415e3b0 0x7fb17415fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb1795ce640 1 -- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415e3b0 msgr2=0x7fb17415fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415e3b0 0x7fb17415fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb1795ce640 1 -- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb17406bbd0 msgr2=0x7fb17406c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb17406bbd0 0x7fb17406c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb1795ce640 1 -- 172.21.15.29:0/2859331834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17415eee0 con 0x7fb160003970 2023-10-16T15:56:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb1795ce640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb174151f80 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fb17000e230 tx=0x7fb170009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170034bd0 con 0x7fb160003970 2023-10-16T15:56:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170034d30 con 0x7fb160003970 2023-10-16T15:56:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170034ea0 con 0x7fb160003970 2023-10-16T15:56:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.246+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17415f1c0 con 0x7fb160003970 2023-10-16T15:56:11.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.247+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb174160190 con 0x7fb160003970 2023-10-16T15:56:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.248+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17404e9b0 con 0x7fb160003970 2023-10-16T15:56:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.249+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb170040450 con 0x7fb160003970 2023-10-16T15:56:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.250+0000 7fb169ffb640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb15806af90 0x7fb15806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.250+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fb1700abfa0 con 0x7fb160003970 2023-10-16T15:56:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.250+0000 7fb178dcd640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb15806af90 0x7fb15806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.251+0000 7fb178dcd640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb15806af90 0x7fb15806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb1640098f0 tx=0x7fb164008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.257+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb1700deca0 con 0x7fb160003970 2023-10-16T15:56:11.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.445+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fb1740caa40 con 0x7fb160003970 2023-10-16T15:56:11.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.452+0000 7fb169ffb640 1 -- 172.21.15.29:0/2859331834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb170076ec0 con 0x7fb160003970 2023-10-16T15:56:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.454+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb15806af90 msgr2=0x7fb15806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.454+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb15806af90 0x7fb15806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb1640098f0 tx=0x7fb164008040 comp rx=0 tx=0).stop 2023-10-16T15:56:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.454+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 msgr2=0x7fb174151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.454+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb174151f80 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fb17000e230 tx=0x7fb170009750 comp rx=0 tx=0).stop 2023-10-16T15:56:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 shutdown_connections 2023-10-16T15:56:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb15806af90 0x7fb15806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb17406bbd0 0x7fb17406c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb17415e3b0 0x7fb17415fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 --2- 172.21.15.29:0/2859331834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb160003970 0x7fb174151f80 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 >> 172.21.15.29:0/2859331834 conn(0x7fb1740cf570 msgr2=0x7fb1740b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:11.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 shutdown_connections 2023-10-16T15:56:11.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.455+0000 7fb17a5d0640 1 -- 172.21.15.29:0/2859331834 wait complete. 2023-10-16T15:56:11.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T15:56:11.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.695+0000 7f6feffff640 1 Processor -- start 2023-10-16T15:56:11.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.696+0000 7f6feffff640 1 -- start start 2023-10-16T15:56:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff00cc270 0x7f6ff00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ff00acdd0 0x7f6ff006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00c8cf0 con 0x7f6ff015cdb0 2023-10-16T15:56:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff015d190 con 0x7f6ff00acdd0 2023-10-16T15:56:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00ad240 con 0x7f6ff00cc270 2023-10-16T15:56:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feeffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6ff015cdb0 0x7f6ff00c9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:37518/0 (socket says 172.21.15.29:37518) 2023-10-16T15:56:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6feeffd640 1 -- 172.21.15.29:0/879454508 learned_addr learned my addr 172.21.15.29:0/879454508 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6fef7fe640 1 --2- 172.21.15.29:0/879454508 >> v2:172.21.15.29:3300/0 conn(0x7f6ff00acdd0 0x7f6ff006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff00cc270 0x7f6ff00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6fef7fe640 1 -- 172.21.15.29:0/879454508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff00cc270 msgr2=0x7f6ff00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:11.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.698+0000 7f6fef7fe640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff00cc270 0x7f6ff00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.699+0000 7f6fef7fe640 1 -- 172.21.15.29:0/879454508 >> v1:172.21.15.29:6789/0 conn(0x7f6ff015cdb0 legacy=0x7f6ff00c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.699+0000 7f6fef7fe640 1 -- 172.21.15.29:0/879454508 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff00cc6d0 con 0x7f6ff00acdd0 2023-10-16T15:56:11.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.699+0000 7f6fee7fc640 1 -- 172.21.15.29:0/879454508 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226946948 0 0) 0x7f6ff00c8cf0 con 0x7f6ff015cdb0 2023-10-16T15:56:11.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.699+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff00cc270 0x7f6ff00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:11.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.700+0000 7f6fef7fe640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff006fea0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009dc0 tx=0x7f6fe802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a014e8c4792a460c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:11.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.700+0000 7f6fee7fc640 1 -- 172.21.15.29:0/879454508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe802cbb0 con 0x7f6ff00acdd0 2023-10-16T15:56:11.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.700+0000 7f6fee7fc640 1 -- 172.21.15.29:0/879454508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe802cd10 con 0x7f6ff00acdd0 2023-10-16T15:56:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.700+0000 7f6fee7fc640 1 -- 172.21.15.29:0/879454508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8034660 con 0x7f6ff00acdd0 2023-10-16T15:56:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 -- 172.21.15.29:0/879454508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 msgr2=0x7f6ff006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff006fea0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009dc0 tx=0x7f6fe802be00 comp rx=0 tx=0).stop 2023-10-16T15:56:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 -- 172.21.15.29:0/879454508 shutdown_connections 2023-10-16T15:56:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff006fea0 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 --2- 172.21.15.29:0/879454508 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff00cc270 0x7f6ff00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 -- 172.21.15.29:0/879454508 >> 172.21.15.29:0/879454508 conn(0x7f6ff00b0a90 msgr2=0x7f6ff00ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 -- 172.21.15.29:0/879454508 shutdown_connections 2023-10-16T15:56:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.701+0000 7f6feffff640 1 -- 172.21.15.29:0/879454508 wait complete. 2023-10-16T15:56:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.703+0000 7f6feffff640 1 Processor -- start 2023-10-16T15:56:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.703+0000 7f6feffff640 1 -- start start 2023-10-16T15:56:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff015dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ff00cc270 0x7f6ff006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff015cdb0 0x7f6ff00c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00c5190 con 0x7f6ff00acdd0 2023-10-16T15:56:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff006b6c0 con 0x7f6ff00cc270 2023-10-16T15:56:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff006b9a0 con 0x7f6ff015cdb0 2023-10-16T15:56:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff015dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff015dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51628/0 (socket says 172.21.15.29:51628) 2023-10-16T15:56:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.704+0000 7f6feeffd640 1 -- 172.21.15.29:0/2924880861 learned_addr learned my addr 172.21.15.29:0/2924880861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feeffd640 1 -- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff015cdb0 msgr2=0x7f6ff00c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6fef7fe640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff015cdb0 0x7f6ff00c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ff00cc270 0x7f6ff006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feeffd640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff015cdb0 0x7f6ff00c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feeffd640 1 -- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ff00cc270 msgr2=0x7f6ff006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feeffd640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ff00cc270 0x7f6ff006afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:11.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feeffd640 1 -- 172.21.15.29:0/2924880861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009a70 con 0x7f6ff00acdd0 2023-10-16T15:56:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feeffd640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff015dd20 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000c430 tx=0x7f6fe000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe00074e0 con 0x7f6ff00acdd0 2023-10-16T15:56:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe0007660 con 0x7f6ff00acdd0 2023-10-16T15:56:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0004a20 con 0x7f6ff00acdd0 2023-10-16T15:56:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.705+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff006bc80 con 0x7f6ff00acdd0 2023-10-16T15:56:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.706+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff00c2900 con 0x7f6ff00acdd0 2023-10-16T15:56:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.707+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff00c2020 con 0x7f6ff00acdd0 2023-10-16T15:56:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.708+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6fe0004b80 con 0x7f6ff00acdd0 2023-10-16T15:56:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.709+0000 7f6fecff9640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6fd006af90 0x7f6fd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.709+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f6fe0089640 con 0x7f6ff00acdd0 2023-10-16T15:56:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.710+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6fd006af90 0x7f6fd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.711+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6fd006af90 0x7f6fd006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ff01f5860 tx=0x7f6fd8008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:11.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.713+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6fe0054560 con 0x7f6ff00acdd0 2023-10-16T15:56:11.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:11.983+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f6ff006c1a0 con 0x7f6ff00acdd0 2023-10-16T15:56:12.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.074+0000 7f6fecff9640 1 -- 172.21.15.29:0/2924880861 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3320) v1 ==== 155+0+0 (secure 0 0 0) 0x7f6fe0057b80 con 0x7f6ff00acdd0 2023-10-16T15:56:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6fd006af90 msgr2=0x7f6fd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6fd006af90 0x7f6fd006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ff01f5860 tx=0x7f6fd8008040 comp rx=0 tx=0).stop 2023-10-16T15:56:12.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 msgr2=0x7f6ff015dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:12.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff015dd20 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000c430 tx=0x7f6fe000c8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:12.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 shutdown_connections 2023-10-16T15:56:12.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f6fd006af90 0x7f6fd006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:12.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f6ff015cdb0 0x7f6ff00c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ff00cc270 0x7f6ff006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 --2- 172.21.15.29:0/2924880861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ff00acdd0 0x7f6ff015dd20 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 >> 172.21.15.29:0/2924880861 conn(0x7f6ff00b0a90 msgr2=0x7f6ff015a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 shutdown_connections 2023-10-16T15:56:12.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:12.078+0000 7f6feffff640 1 -- 172.21.15.29:0/2924880861 wait complete. 2023-10-16T15:56:12.099 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-16T15:56:12.100 INFO:tasks.cephfs_test_runner: 2023-10-16T15:56:12.100 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T15:56:12.100 INFO:tasks.cephfs_test_runner:Ran 14 tests in 5943.010s 2023-10-16T15:56:12.100 INFO:tasks.cephfs_test_runner: 2023-10-16T15:56:12.101 INFO:tasks.cephfs_test_runner:OK 2023-10-16T15:56:12.101 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-16T15:56:12.111 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-16T15:56:12.125 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-16T15:56:12.126 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:56:12.126 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:12.187 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:56:12.187 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-16T15:56:12.187 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:12.244 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T15:56:12.244 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:56:12.245 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:12.307 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:56:12.307 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-16T15:56:12.308 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T15:56:12.362 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054303,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498832k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T15:56:12.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.363 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:56:12.426 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T15:56:12.426 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T15:56:12.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T15:56:12.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T15:56:12.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-16T15:56:12.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.781 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-16T15:56:12.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-16T15:56:12.967 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-16T15:56:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T15:56:12.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:12.972 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2023-10-16T15:56:13.030 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T15:56:13.030 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-16T15:56:13.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:13.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T15:56:13.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:13.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T15:56:13.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:13.225 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link delete ceph-brx 2023-10-16T15:56:13.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:13.346 DEBUG:teuthology.orchestra.run.smithi170:> ip netns list 2023-10-16T15:56:13.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T15:56:13.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo ip link delete ceph-brx 2023-10-16T15:56:13.542 INFO:teuthology.orchestra.run.smithi170.stderr:Cannot find device "ceph-brx" 2023-10-16T15:56:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T15:56:13.547 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-16T15:56:13.557 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T15:56:13.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:56:13.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.795+0000 7f5a1ffff640 1 Processor -- start 2023-10-16T15:56:13.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.795+0000 7f5a1ffff640 1 -- start start 2023-10-16T15:56:13.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a2006c040 0x7f5a2006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1ffff640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f5a200ad970 0x7f5a2015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a2015ebc0 0x7f5a2015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1ffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2006c960 con 0x7f5a2015ebc0 2023-10-16T15:56:13.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1ffff640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2015b9d0 con 0x7f5a200ad970 2023-10-16T15:56:13.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2015bcb0 con 0x7f5a2006c040 2023-10-16T15:56:13.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a2015ebc0 0x7f5a2015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a2015ebc0 0x7f5a2015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51632/0 (socket says 172.21.15.29:51632) 2023-10-16T15:56:13.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.797+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/2047454626 learned_addr learned my addr 172.21.15.29:0/2047454626 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:13.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1e7fc640 1 --2- 172.21.15.29:0/2047454626 >> v2:172.21.15.170:3300/0 conn(0x7f5a200ad970 0x7f5a2015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/2047454626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a2006c040 msgr2=0x7f5a2006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:13.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1effd640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a2006c040 0x7f5a2006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1f7fe640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a2006c040 0x7f5a2006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/2047454626 >> v2:172.21.15.170:3300/0 conn(0x7f5a200ad970 msgr2=0x7f5a2015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1f7fe640 1 --2- 172.21.15.29:0/2047454626 >> v2:172.21.15.170:3300/0 conn(0x7f5a200ad970 0x7f5a2015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/2047454626 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2015bf90 con 0x7f5a2015ebc0 2023-10-16T15:56:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1e7fc640 1 --2- 172.21.15.29:0/2047454626 >> v2:172.21.15.170:3300/0 conn(0x7f5a200ad970 0x7f5a2015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:13.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.798+0000 7f5a1effd640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a2006c040 0x7f5a2006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:13.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.799+0000 7f5a1f7fe640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a2015f050 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009e50 tx=0x7f5a1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b4410a3765a41a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:13.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.799+0000 7f5a1dffb640 1 -- 172.21.15.29:0/2047454626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1802ca80 con 0x7f5a2015ebc0 2023-10-16T15:56:13.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.799+0000 7f5a1dffb640 1 -- 172.21.15.29:0/2047454626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1802cbe0 con 0x7f5a2015ebc0 2023-10-16T15:56:13.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.799+0000 7f5a1dffb640 1 -- 172.21.15.29:0/2047454626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180343f0 con 0x7f5a2015ebc0 2023-10-16T15:56:13.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 -- 172.21.15.29:0/2047454626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 msgr2=0x7f5a2015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:13.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a2015f050 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009e50 tx=0x7f5a1802bee0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 -- 172.21.15.29:0/2047454626 shutdown_connections 2023-10-16T15:56:13.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a2015f050 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/2047454626 >> v2:172.21.15.170:3300/0 conn(0x7f5a200ad970 0x7f5a2015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/2047454626 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a2006c040 0x7f5a2006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 -- 172.21.15.29:0/2047454626 >> 172.21.15.29:0/2047454626 conn(0x7f5a200ce6a0 msgr2=0x7f5a200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:13.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 -- 172.21.15.29:0/2047454626 shutdown_connections 2023-10-16T15:56:13.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.800+0000 7f5a1ffff640 1 -- 172.21.15.29:0/2047454626 wait complete. 2023-10-16T15:56:13.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.803+0000 7f5a1ffff640 1 Processor -- start 2023-10-16T15:56:13.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.803+0000 7f5a1ffff640 1 -- start start 2023-10-16T15:56:13.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.804+0000 7f5a1ffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2006c040 0x7f5a20158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.804+0000 7f5a1ffff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a200ad970 0x7f5a20153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.804+0000 7f5a1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a200afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.804+0000 7f5a1ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a201f5850 con 0x7f5a2015ebc0 2023-10-16T15:56:13.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.804+0000 7f5a1ffff640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a20154020 con 0x7f5a2006c040 2023-10-16T15:56:13.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.804+0000 7f5a1ffff640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a201f5aa0 con 0x7f5a200ad970 2023-10-16T15:56:13.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a200afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a200afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51648/0 (socket says 172.21.15.29:51648) 2023-10-16T15:56:13.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/1780860044 learned_addr learned my addr 172.21.15.29:0/1780860044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:13.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1effd640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2006c040 0x7f5a20158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1e7fc640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a200ad970 0x7f5a20153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a200ad970 msgr2=0x7f5a20153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:13.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a200ad970 0x7f5a20153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2006c040 msgr2=0x7f5a20158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:13.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2006c040 0x7f5a20158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:13.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 -- 172.21.15.29:0/1780860044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18009b30 con 0x7f5a2015ebc0 2023-10-16T15:56:13.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1e7fc640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a200ad970 0x7f5a20153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:13.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.805+0000 7f5a1f7fe640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a200afe90 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009e50 tx=0x7f5a18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:13.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.806+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18040670 con 0x7f5a2015ebc0 2023-10-16T15:56:13.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.806+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a180407d0 con 0x7f5a2015ebc0 2023-10-16T15:56:13.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.806+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18040b80 con 0x7f5a2015ebc0 2023-10-16T15:56:13.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.806+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a201f5d80 con 0x7f5a2015ebc0 2023-10-16T15:56:13.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.806+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a201f6340 con 0x7f5a2015ebc0 2023-10-16T15:56:13.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.807+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2006b330 con 0x7f5a2015ebc0 2023-10-16T15:56:13.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.808+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a1802cd50 con 0x7f5a2015ebc0 2023-10-16T15:56:13.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.809+0000 7f5a077fe640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5a0c06af90 0x7f5a0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:13.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.809+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f5a180ac760 con 0x7f5a2015ebc0 2023-10-16T15:56:13.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.810+0000 7f5a1effd640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5a0c06af90 0x7f5a0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:13.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.811+0000 7f5a1effd640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5a0c06af90 0x7f5a0c06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000a980 tx=0x7f5a10008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:13.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:13.814+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a180775e0 con 0x7f5a2015ebc0 2023-10-16T15:56:14.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.002+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a200cc550 con 0x7f5a0c06af90 2023-10-16T15:56:14.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.010+0000 7f5a077fe640 1 -- 172.21.15.29:0/1780860044 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47882 (secure 0 0 0) 0x7f5a1000c050 con 0x7f5a0c06af90 2023-10-16T15:56:14.012 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:14.013 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:56:14.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5a0c06af90 msgr2=0x7f5a0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5a0c06af90 0x7f5a0c06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000a980 tx=0x7f5a10008040 comp rx=0 tx=0).stop 2023-10-16T15:56:14.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 msgr2=0x7f5a200afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a200afe90 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009e50 tx=0x7f5a18004a40 comp rx=0 tx=0).stop 2023-10-16T15:56:14.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 shutdown_connections 2023-10-16T15:56:14.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f5a0c06af90 0x7f5a0c06d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a2015ebc0 0x7f5a200afe90 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f5a200ad970 0x7f5a20153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 --2- 172.21.15.29:0/1780860044 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a2006c040 0x7f5a20158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 >> 172.21.15.29:0/1780860044 conn(0x7f5a200ce6a0 msgr2=0x7f5a20072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:14.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 shutdown_connections 2023-10-16T15:56:14.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.013+0000 7f5a1ffff640 1 -- 172.21.15.29:0/1780860044 wait complete. 2023-10-16T15:56:14.035 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-16T15:56:12.823153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457584,"kb_used_data":29072,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517136,"statfs":{"total":773094113280,"available":772625547264,"internally_reserved":0,"allocated":29769728,"data_stored":15983476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":438685241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668451,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-262,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001855"},"pg_stats":[{"pgid":"1.0","version":"268'2809","reported_seq":3921,"reported_epoch":268,"state":"active+clean","last_fresh":"2023-10-16T15:56:04.939424+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:56:04.939424+0000","last_peered":"2023-10-16T15:56:04.939424+0000","last_clean":"2023-10-16T15:56:04.939424+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:56:04.939424+0000","last_undegraded":"2023-10-16T15:56:04.939424+0000","last_fullsized":"2023-10-16T15:56:04.939424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576032,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57284,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314556,"statfs":{"total":96636764160,"available":96578105344,"internally_reserved":0,"allocated":3989504,"data_stored":2273185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0820000000000001,"15min":1.163},"min":{"1min":0.70099999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.72,"5min":1.794,"15min":1.8879999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.998,"5min":1.083,"15min":1.1339999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.486,"5min":1.796,"15min":1.796},"last":0.80200000000000005}]},{"osd":1,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.105,"15min":1.1559999999999999},"min":{"1min":0.66800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7170000000000001,"5min":1.76,"15min":1.8580000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7809999999999999,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.28}]},{"osd":2,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2969999999999999,"15min":1.3260000000000001},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.77,"5min":1.893,"15min":2.1269999999999998},"last":1.087},{"interface":"front","average":{"1min":1.127,"5min":1.2230000000000001,"15min":1.2829999999999999},"min":{"1min":0.82999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7110000000000001,"5min":1.909,"15min":1.95},"last":1.0409999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2629999999999999,"15min":1.327},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.823,"5min":1.843,"15min":1.869},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.387,"15min":1.4390000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8500000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.4199999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.385,"15min":1.385},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.907,"5min":1.9370000000000001,"15min":2.2610000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.337},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.667,"5min":1.8740000000000001,"15min":2.0409999999999999},"last":1.462}]},{"osd":5,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.456},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":2.1469999999999998},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.502,"15min":1.552},"min":{"1min":0.67900000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3540000000000001},"last":1.675}]},{"osd":6,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4430000000000001,"15min":1.4770000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9330000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.5609999999999999,"15min":1.5800000000000001},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.998,"5min":1.998,"15min":2.2599999999999998},"last":1.764}]}]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52548,"kb_used_data":3320,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319292,"statfs":{"total":96636764160,"available":96582955008,"internally_reserved":0,"allocated":3399680,"data_stored":1682907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.208,"15min":1.206},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9099999999999999,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.2130000000000001,"15min":1.2430000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":1.5309999999999999,"5min":1.905,"15min":1.905},"last":1.0289999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.72199999999999998},"max":{"1min":1.675,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2889999999999999,"15min":1.3160000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.252}]},{"osd":2,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2330000000000001,"15min":1.232},"min":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.611,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3859999999999999,"15min":1.3959999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8400000000000001,"5min":1.905,"15min":2.0339999999999998},"last":1.1599999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.361,"15min":1.391},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.9079999999999999,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.236,"5min":1.284,"15min":1.331},"min":{"1min":0.82799999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.5980000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.421,"15min":1.351},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.978,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.458,"15min":1.4119999999999999},"min":{"1min":0.81100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0830000000000002,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.371}]},{"osd":5,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3759999999999999,"15min":1.357},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.67300000000000004},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5980000000000001,"15min":1.54},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1059999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.5580000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.6419999999999999,"15min":1.6040000000000001},"min":{"1min":0.86699999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.4940000000000002},"last":1.764},{"interface":"front","average":{"1min":1.599,"5min":1.6399999999999999,"15min":1.5980000000000001},"min":{"1min":0.98099999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.677}]}]},{"osd":5,"up_from":13,"seq":55834576031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49060,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322780,"statfs":{"total":96636764160,"available":96586526720,"internally_reserved":0,"allocated":4087808,"data_stored":2352672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.093,"15min":1.139},"min":{"1min":0.64600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.228},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2190000000000001,"15min":1.2390000000000001},"min":{"1min":0.66400000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.885,"5min":2.738,"15min":2.738},"last":1.3160000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1859999999999999,"15min":1.216},"min":{"1min":0.71999999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.173,"5min":1.2150000000000001,"15min":1.2789999999999999},"min":{"1min":0.75,"5min":0.443,"15min":0.443},"max":{"1min":1.579,"5min":2.137,"15min":2.137},"last":1.708}]},{"osd":2,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1779999999999999,"15min":1.2350000000000001},"min":{"1min":0.51900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.621,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.339,"15min":1.381},"min":{"1min":0.72699999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.891,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.615}]},{"osd":3,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.29,"15min":1.3260000000000001},"min":{"1min":0.81399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5509999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2749999999999999,"15min":1.359},"min":{"1min":0.63400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.5149999999999999,"5min":1.966,"15min":1.9670000000000001},"last":1.5649999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.3460000000000001},"min":{"1min":0.91900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.623,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.4670000000000001,"15min":1.5029999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7250000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.7529999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4299999999999999,"15min":1.3919999999999999},"min":{"1min":0.90700000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.077,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.425,"15min":1.429},"min":{"1min":0.84399999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6950000000000001,"5min":2.0230000000000001,"15min":2.0609999999999999},"last":1.167}]},{"osd":7,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.498,"15min":1.5269999999999999},"min":{"1min":1.246,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8320000000000001,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.667},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6319999999999999,"15min":1.6339999999999999},"min":{"1min":1.284,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2029999999999998,"5min":3.109,"15min":3.109},"last":1.875}]}]},{"osd":4,"up_from":13,"seq":55834576032,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70924,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300916,"statfs":{"total":96636764160,"available":96564137984,"internally_reserved":0,"allocated":3735552,"data_stored":2011141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1279999999999999,"15min":1.1890000000000001},"min":{"1min":0.749,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.1419999999999999,"15min":1.1830000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.349}]},{"osd":1,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.629},"max":{"1min":1.6990000000000001,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.83999999999999997,"5min":0.75900000000000001,"15min":0.70199999999999996},"max":{"1min":1.6579999999999999,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.3939999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4099999999999999,"15min":1.452},"min":{"1min":0.91800000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.8929999999999998},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.452,"15min":1.4850000000000001},"min":{"1min":0.88900000000000001,"5min":0.82499999999999996,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.4610000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.383,"15min":1.377},"min":{"1min":0.91100000000000003,"5min":0.84699999999999998,"15min":0.79400000000000004},"max":{"1min":1.915,"5min":2.23,"15min":2.3650000000000002},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3540000000000001,"15min":1.3819999999999999},"min":{"1min":0.81999999999999995,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.7729999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3979999999999999,"15min":1.3500000000000001},"min":{"1min":0.89400000000000002,"5min":0.80200000000000005,"15min":0.754},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.196},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4490000000000001},"min":{"1min":1.056,"5min":0.88900000000000001,"15min":0.74399999999999999},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":1.6060000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.427,"15min":1.4079999999999999},"min":{"1min":0.91300000000000003,"5min":0.90500000000000003,"15min":0.75900000000000001},"max":{"1min":2.129,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.9019999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.597,"15min":1.538},"min":{"1min":1.119,"5min":1.0089999999999999,"15min":0.82499999999999996},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.6699999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.619,"15min":1.5880000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.83999999999999997},"max":{"1min":2.2189999999999999,"5min":2.399,"15min":2.399},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.6379999999999999,"15min":1.611},"min":{"1min":1.232,"5min":1.036,"15min":0.98899999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.972}]}]},{"osd":3,"up_from":13,"seq":55834576031,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":3424256,"data_stored":1696209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.91800000000000004,"15min":0.96299999999999997},"min":{"1min":0.58899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.369,"5min":1.48,"15min":2.0430000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.98599999999999999,"15min":1.042},"min":{"1min":0.58899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2210000000000001,"5min":1.722,"15min":2.5990000000000002},"last":1.3939999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":1.01},"min":{"1min":0.69599999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.375,"5min":1.776,"15min":2.589},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1279999999999999,"15min":1.196},"min":{"1min":0.65300000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.6060000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.097}]},{"osd":2,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2649999999999999,"15min":1.383},"min":{"1min":0.81100000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.629,"5min":2.0899999999999999,"15min":2.302},"last":1.772},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3740000000000001,"15min":1.478},"min":{"1min":0.76100000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":1.954,"15min":2.754},"last":1.5409999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.292,"15min":1.3480000000000001},"min":{"1min":0.92000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.958,"5min":2.0110000000000001,"15min":2.512},"last":1.698},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3400000000000001,"15min":1.407},"min":{"1min":0.92200000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.194,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.216}]},{"osd":5,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.51,"15min":1.552},"min":{"1min":0.95799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0710000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.5,"15min":1.556},"min":{"1min":1.0549999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.109,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.9259999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.4690000000000001,"15min":1.4990000000000001},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2090000000000001},"last":1.835},{"interface":"front","average":{"1min":1.532,"5min":1.5589999999999999,"15min":1.5900000000000001},"min":{"1min":1.022,"5min":0.749,"15min":0.749},"max":{"1min":2.1419999999999999,"5min":2.3029999999999999,"15min":2.706},"last":1.998}]},{"osd":7,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.575,"15min":1.6679999999999999},"min":{"1min":0.96299999999999997,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0619999999999998,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":2.145},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0910000000000002,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":2.085}]}]},{"osd":2,"up_from":13,"seq":55834576032,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56708,"kb_used_data":3320,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315132,"statfs":{"total":96636764160,"available":96578695168,"internally_reserved":0,"allocated":3399680,"data_stored":1682907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":0.92000000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.3080000000000001,"5min":1.726,"15min":1.76},"last":0.623},{"interface":"front","average":{"1min":0.879,"5min":0.97799999999999998,"15min":1.0780000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6739999999999999,"5min":1.804,"15min":1.8999999999999999},"last":0.68899999999999995}]},{"osd":1,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97599999999999998,"15min":1.0369999999999999},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1739999999999999,"5min":1.974,"15min":1.974},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0900000000000001,"15min":1.206},"min":{"1min":0.502,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4990000000000001,"5min":1.879,"15min":1.879},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.107,"15min":1.256},"min":{"1min":0.64100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3979999999999999,"5min":1.704,"15min":2.1509999999999998},"last":0.871},{"interface":"front","average":{"1min":1.099,"5min":1.272,"15min":1.3460000000000001},"min":{"1min":0.629,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.569,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.1970000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.4019999999999999,"15min":1.4610000000000001},"min":{"1min":0.79700000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0409999999999999,"5min":2.2669999999999999,"15min":2.335},"last":1.371},{"interface":"front","average":{"1min":1.266,"5min":1.3160000000000001,"15min":1.409},"min":{"1min":0.90400000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7909999999999999,"5min":1.901,"15min":2.1469999999999998},"last":1.073}]},{"osd":5,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.377,"15min":1.466},"min":{"1min":0.80100000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.008,"5min":2.3050000000000002,"15min":2.657},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.492,"15min":1.593},"min":{"1min":0.85799999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.9039999999999999,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.5289999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4299999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.881,"5min":1.9510000000000001,"15min":2.3849999999999998},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.5109999999999999,"15min":1.6060000000000001},"min":{"1min":0.91200000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.831,"5min":2.3759999999999999,"15min":2.7090000000000001},"last":1.4039999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.544,"15min":1.645},"min":{"1min":1.171,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.181,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.617,"15min":1.7150000000000001},"min":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0960000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.2869999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576032,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52868,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318972,"statfs":{"total":96636764160,"available":96582627328,"internally_reserved":0,"allocated":3727360,"data_stored":2010531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.90900000000000003,"15min":0.93999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.351,"5min":1.49,"15min":1.8420000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98899999999999999,"15min":1.113},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6639999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.1299999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.94199999999999995,"15min":1.0209999999999999},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.764,"5min":1.764,"15min":2.286},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.125,"15min":1.214},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7929999999999999,"15min":1.853},"last":1.359}]},{"osd":3,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.254,"15min":1.353},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.651,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":1.53},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1879999999999999,"15min":1.2929999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.532,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":1.405}]},{"osd":4,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.319,"15min":1.365},"min":{"1min":0.89200000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.915,"5min":2.327,"15min":2.3809999999999998},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.47,"15min":1.5109999999999999},"min":{"1min":0.874,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.8759999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.706}]},{"osd":5,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.52},"min":{"1min":0.86499999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9830000000000001,"5min":2.456,"15min":2.456},"last":2.3610000000000002},{"interface":"front","average":{"1min":1.252,"5min":1.4450000000000001,"15min":1.484},"min":{"1min":0.79200000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9490000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.5920000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.595,"15min":1.641},"min":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0489999999999999,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":2.0139999999999998},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5880000000000001,"15min":1.637},"min":{"1min":0.84199999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.343,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":2.2309999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.6499999999999999,"15min":1.7310000000000001},"min":{"1min":0.79900000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1379999999999999,"5min":2.734,"15min":3.2090000000000001},"last":2.4729999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.5820000000000001,"15min":1.6399999999999999},"min":{"1min":0.89000000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1110000000000002,"5min":2.452,"15min":3.1539999999999999},"last":1.754}]}]},{"osd":0,"up_from":13,"seq":55834576032,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61460,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310380,"statfs":{"total":96636764160,"available":96573829120,"internally_reserved":0,"allocated":4005888,"data_stored":2273924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88900000000000001,"15min":0.94299999999999995},"min":{"1min":0.47299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.468,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0620000000000001,"15min":1.1899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4630000000000001,"5min":2.036,"15min":2.036},"last":1.2569999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1439999999999999,"15min":1.248},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.712,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.9379999999999999},{"interface":"front","average":{"1min":1,"5min":1.083,"15min":1.2},"min":{"1min":0.54200000000000004,"5min":0.439,"15min":0.439},"max":{"1min":1.5600000000000001,"5min":1.7330000000000001,"15min":1.7849999999999999},"last":1.478}]},{"osd":3,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96299999999999997,"15min":1.018},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.208,"5min":1.6339999999999999,"15min":1.6799999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1319999999999999,"15min":1.238},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6579999999999999,"5min":1.841,"15min":2.0899999999999999},"last":1.1279999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.258,"15min":1.3200000000000001},"min":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.542,"5min":1.9890000000000001,"15min":2.137},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3660000000000001,"15min":1.4850000000000001},"min":{"1min":0.80800000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.113,"5min":2.113,"15min":2.452},"last":1.593}]},{"osd":5,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3799999999999999,"15min":1.4570000000000001},"min":{"1min":0.79300000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0550000000000002,"5min":2.1059999999999999,"15min":2.137},"last":1.861},{"interface":"front","average":{"1min":1.274,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.76800000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2,"5min":2,"15min":2.282},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5049999999999999,"15min":1.615},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.7010000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.502,"15min":1.5629999999999999},"min":{"1min":0.73899999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.982}]},{"osd":7,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.589,"15min":1.667},"min":{"1min":1.01,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1230000000000002,"5min":2.339,"15min":2.339},"last":2.101},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.516,"15min":1.5840000000000001},"min":{"1min":0.86099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.274},"last":1.4039999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:56:14.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:56:14.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.268+0000 7f776992e640 1 Processor -- start 2023-10-16T15:56:14.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.268+0000 7f776992e640 1 -- start start 2023-10-16T15:56:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.270+0000 7f776992e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f776406c040 0x7f776406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.270+0000 7f776992e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f77640cda80 0x7f776415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.270+0000 7f776992e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776406c960 con 0x7f77640ad970 2023-10-16T15:56:14.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.270+0000 7f776992e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776415b9d0 con 0x7f776406c040 2023-10-16T15:56:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.270+0000 7f776992e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776415bcb0 con 0x7f77640cda80 2023-10-16T15:56:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f776406c040 0x7f776406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f776406c040 0x7f776406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34248/0 (socket says 172.21.15.29:34248) 2023-10-16T15:56:14.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 -- 172.21.15.29:0/1820432453 learned_addr learned my addr 172.21.15.29:0/1820432453 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:14.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776912d640 1 --2- 172.21.15.29:0/1820432453 >> v2:172.21.15.170:3300/0 conn(0x7f77640cda80 0x7f776415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 -- 172.21.15.29:0/1820432453 >> v2:172.21.15.170:3300/0 conn(0x7f77640cda80 msgr2=0x7f776415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 --2- 172.21.15.29:0/1820432453 >> v2:172.21.15.170:3300/0 conn(0x7f77640cda80 0x7f776415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 -- 172.21.15.29:0/1820432453 >> v1:172.21.15.170:6789/0 conn(0x7f77640ad970 legacy=0x7f776415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.271+0000 7f776892c640 1 -- 172.21.15.29:0/1820432453 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776415bf90 con 0x7f776406c040 2023-10-16T15:56:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.273+0000 7f776892c640 1 --2- 172.21.15.29:0/1820432453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f776406c420 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f77640cf300 tx=0x7f775802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238417175c001960 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.273+0000 7f77637fe640 1 -- 172.21.15.29:0/1820432453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758031870 con 0x7f776406c040 2023-10-16T15:56:14.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.273+0000 7f77637fe640 1 -- 172.21.15.29:0/1820432453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77580319d0 con 0x7f776406c040 2023-10-16T15:56:14.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.273+0000 7f77637fe640 1 -- 172.21.15.29:0/1820432453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758031d60 con 0x7f776406c040 2023-10-16T15:56:14.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 -- 172.21.15.29:0/1820432453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 msgr2=0x7f776406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 --2- 172.21.15.29:0/1820432453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f776406c420 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f77640cf300 tx=0x7f775802b910 comp rx=0 tx=0).stop 2023-10-16T15:56:14.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 -- 172.21.15.29:0/1820432453 shutdown_connections 2023-10-16T15:56:14.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 --2- 172.21.15.29:0/1820432453 >> v2:172.21.15.170:3300/0 conn(0x7f77640cda80 0x7f776415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 --2- 172.21.15.29:0/1820432453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f776406c420 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 -- 172.21.15.29:0/1820432453 >> 172.21.15.29:0/1820432453 conn(0x7f77640ce6a0 msgr2=0x7f77640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 -- 172.21.15.29:0/1820432453 shutdown_connections 2023-10-16T15:56:14.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.274+0000 7f776992e640 1 -- 172.21.15.29:0/1820432453 wait complete. 2023-10-16T15:56:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.276+0000 7f776992e640 1 Processor -- start 2023-10-16T15:56:14.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.276+0000 7f776992e640 1 -- start start 2023-10-16T15:56:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776992e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f77641f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776992e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77640ad970 0x7f77640c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776992e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77640cda80 0x7f77640c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776992e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77640b1750 con 0x7f776406c040 2023-10-16T15:56:14.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776992e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77641f3520 con 0x7f77640cda80 2023-10-16T15:56:14.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776992e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77641f6030 con 0x7f77640ad970 2023-10-16T15:56:14.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776892c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f77641f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776892c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f77641f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34262/0 (socket says 172.21.15.29:34262) 2023-10-16T15:56:14.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.277+0000 7f776892c640 1 -- 172.21.15.29:0/1823814590 learned_addr learned my addr 172.21.15.29:0/1823814590 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:14.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776892c640 1 -- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77640ad970 msgr2=0x7f77640c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776892c640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77640ad970 0x7f77640c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776892c640 1 -- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77640cda80 msgr2=0x7f77640c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776892c640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77640cda80 0x7f77640c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776892c640 1 -- 172.21.15.29:0/1823814590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77580094d0 con 0x7f776406c040 2023-10-16T15:56:14.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776892c640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f77641f2fe0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f7758000c00 tx=0x7f77580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:14.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775803ad30 con 0x7f776406c040 2023-10-16T15:56:14.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775803ae90 con 0x7f776406c040 2023-10-16T15:56:14.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758044630 con 0x7f776406c040 2023-10-16T15:56:14.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.278+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77641f6310 con 0x7f776406c040 2023-10-16T15:56:14.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.279+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77641f68d0 con 0x7f776406c040 2023-10-16T15:56:14.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.280+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776406b330 con 0x7f776406c040 2023-10-16T15:56:14.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.281+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f775804d450 con 0x7f776406c040 2023-10-16T15:56:14.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.282+0000 7f77617fa640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f774806af90 0x7f774806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.282+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f775802d040 con 0x7f776406c040 2023-10-16T15:56:14.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.283+0000 7f7763fff640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f774806af90 0x7f774806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.284+0000 7f7763fff640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f774806af90 0x7f774806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77640c51b0 tx=0x7f77500079b0 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:14.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.286+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7758076da0 con 0x7f776406c040 2023-10-16T15:56:14.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.478+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77640cc550 con 0x7f774806af90 2023-10-16T15:56:14.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.484+0000 7f77617fa640 1 -- 172.21.15.29:0/1823814590 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47882 (secure 0 0 0) 0x7f7750014400 con 0x7f774806af90 2023-10-16T15:56:14.486 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:14.487 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:56:14.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f774806af90 msgr2=0x7f774806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f774806af90 0x7f774806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77640c51b0 tx=0x7f77500079b0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 msgr2=0x7f77641f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f77641f2fe0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f7758000c00 tx=0x7f77580046c0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 shutdown_connections 2023-10-16T15:56:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f774806af90 0x7f774806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f77640cda80 0x7f77640c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f77640ad970 0x7f77640c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 --2- 172.21.15.29:0/1823814590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776406c040 0x7f77641f2fe0 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 >> 172.21.15.29:0/1823814590 conn(0x7f77640ce6a0 msgr2=0x7f776415c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.487+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 shutdown_connections 2023-10-16T15:56:14.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.488+0000 7f776992e640 1 -- 172.21.15.29:0/1823814590 wait complete. 2023-10-16T15:56:14.509 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-16T15:56:12.823153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457584,"kb_used_data":29072,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517136,"statfs":{"total":773094113280,"available":772625547264,"internally_reserved":0,"allocated":29769728,"data_stored":15983476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":438685241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668451,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-262,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001855"},"pg_stats":[{"pgid":"1.0","version":"268'2809","reported_seq":3921,"reported_epoch":268,"state":"active+clean","last_fresh":"2023-10-16T15:56:04.939424+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:56:04.939424+0000","last_peered":"2023-10-16T15:56:04.939424+0000","last_clean":"2023-10-16T15:56:04.939424+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:56:04.939424+0000","last_undegraded":"2023-10-16T15:56:04.939424+0000","last_fullsized":"2023-10-16T15:56:04.939424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576032,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57284,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314556,"statfs":{"total":96636764160,"available":96578105344,"internally_reserved":0,"allocated":3989504,"data_stored":2273185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0820000000000001,"15min":1.163},"min":{"1min":0.70099999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.72,"5min":1.794,"15min":1.8879999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.998,"5min":1.083,"15min":1.1339999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.486,"5min":1.796,"15min":1.796},"last":0.80200000000000005}]},{"osd":1,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.105,"15min":1.1559999999999999},"min":{"1min":0.66800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7170000000000001,"5min":1.76,"15min":1.8580000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7809999999999999,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.28}]},{"osd":2,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2969999999999999,"15min":1.3260000000000001},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.77,"5min":1.893,"15min":2.1269999999999998},"last":1.087},{"interface":"front","average":{"1min":1.127,"5min":1.2230000000000001,"15min":1.2829999999999999},"min":{"1min":0.82999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7110000000000001,"5min":1.909,"15min":1.95},"last":1.0409999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2629999999999999,"15min":1.327},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.823,"5min":1.843,"15min":1.869},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.387,"15min":1.4390000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8500000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.4199999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.385,"15min":1.385},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.907,"5min":1.9370000000000001,"15min":2.2610000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.337},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.667,"5min":1.8740000000000001,"15min":2.0409999999999999},"last":1.462}]},{"osd":5,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.456},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":2.1469999999999998},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.502,"15min":1.552},"min":{"1min":0.67900000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3540000000000001},"last":1.675}]},{"osd":6,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4430000000000001,"15min":1.4770000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9330000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.5609999999999999,"15min":1.5800000000000001},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.998,"5min":1.998,"15min":2.2599999999999998},"last":1.764}]}]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52548,"kb_used_data":3320,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319292,"statfs":{"total":96636764160,"available":96582955008,"internally_reserved":0,"allocated":3399680,"data_stored":1682907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.208,"15min":1.206},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9099999999999999,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.2130000000000001,"15min":1.2430000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":1.5309999999999999,"5min":1.905,"15min":1.905},"last":1.0289999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.72199999999999998},"max":{"1min":1.675,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2889999999999999,"15min":1.3160000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.252}]},{"osd":2,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2330000000000001,"15min":1.232},"min":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.611,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3859999999999999,"15min":1.3959999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8400000000000001,"5min":1.905,"15min":2.0339999999999998},"last":1.1599999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.361,"15min":1.391},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.9079999999999999,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.236,"5min":1.284,"15min":1.331},"min":{"1min":0.82799999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.5980000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.421,"15min":1.351},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.978,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.458,"15min":1.4119999999999999},"min":{"1min":0.81100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0830000000000002,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.371}]},{"osd":5,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3759999999999999,"15min":1.357},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.67300000000000004},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5980000000000001,"15min":1.54},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1059999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.5580000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.6419999999999999,"15min":1.6040000000000001},"min":{"1min":0.86699999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.4940000000000002},"last":1.764},{"interface":"front","average":{"1min":1.599,"5min":1.6399999999999999,"15min":1.5980000000000001},"min":{"1min":0.98099999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.677}]}]},{"osd":5,"up_from":13,"seq":55834576031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49060,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322780,"statfs":{"total":96636764160,"available":96586526720,"internally_reserved":0,"allocated":4087808,"data_stored":2352672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.093,"15min":1.139},"min":{"1min":0.64600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.228},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2190000000000001,"15min":1.2390000000000001},"min":{"1min":0.66400000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.885,"5min":2.738,"15min":2.738},"last":1.3160000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1859999999999999,"15min":1.216},"min":{"1min":0.71999999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.173,"5min":1.2150000000000001,"15min":1.2789999999999999},"min":{"1min":0.75,"5min":0.443,"15min":0.443},"max":{"1min":1.579,"5min":2.137,"15min":2.137},"last":1.708}]},{"osd":2,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1779999999999999,"15min":1.2350000000000001},"min":{"1min":0.51900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.621,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.339,"15min":1.381},"min":{"1min":0.72699999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.891,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.615}]},{"osd":3,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.29,"15min":1.3260000000000001},"min":{"1min":0.81399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5509999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2749999999999999,"15min":1.359},"min":{"1min":0.63400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.5149999999999999,"5min":1.966,"15min":1.9670000000000001},"last":1.5649999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.3460000000000001},"min":{"1min":0.91900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.623,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.4670000000000001,"15min":1.5029999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7250000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.7529999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4299999999999999,"15min":1.3919999999999999},"min":{"1min":0.90700000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.077,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.425,"15min":1.429},"min":{"1min":0.84399999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6950000000000001,"5min":2.0230000000000001,"15min":2.0609999999999999},"last":1.167}]},{"osd":7,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.498,"15min":1.5269999999999999},"min":{"1min":1.246,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8320000000000001,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.667},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6319999999999999,"15min":1.6339999999999999},"min":{"1min":1.284,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2029999999999998,"5min":3.109,"15min":3.109},"last":1.875}]}]},{"osd":4,"up_from":13,"seq":55834576032,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70924,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300916,"statfs":{"total":96636764160,"available":96564137984,"internally_reserved":0,"allocated":3735552,"data_stored":2011141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1279999999999999,"15min":1.1890000000000001},"min":{"1min":0.749,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.1419999999999999,"15min":1.1830000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.349}]},{"osd":1,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.629},"max":{"1min":1.6990000000000001,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.83999999999999997,"5min":0.75900000000000001,"15min":0.70199999999999996},"max":{"1min":1.6579999999999999,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.3939999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4099999999999999,"15min":1.452},"min":{"1min":0.91800000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.8929999999999998},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.452,"15min":1.4850000000000001},"min":{"1min":0.88900000000000001,"5min":0.82499999999999996,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.4610000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.383,"15min":1.377},"min":{"1min":0.91100000000000003,"5min":0.84699999999999998,"15min":0.79400000000000004},"max":{"1min":1.915,"5min":2.23,"15min":2.3650000000000002},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3540000000000001,"15min":1.3819999999999999},"min":{"1min":0.81999999999999995,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.7729999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3979999999999999,"15min":1.3500000000000001},"min":{"1min":0.89400000000000002,"5min":0.80200000000000005,"15min":0.754},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.196},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4490000000000001},"min":{"1min":1.056,"5min":0.88900000000000001,"15min":0.74399999999999999},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":1.6060000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.427,"15min":1.4079999999999999},"min":{"1min":0.91300000000000003,"5min":0.90500000000000003,"15min":0.75900000000000001},"max":{"1min":2.129,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.9019999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.597,"15min":1.538},"min":{"1min":1.119,"5min":1.0089999999999999,"15min":0.82499999999999996},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.6699999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.619,"15min":1.5880000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.83999999999999997},"max":{"1min":2.2189999999999999,"5min":2.399,"15min":2.399},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.6379999999999999,"15min":1.611},"min":{"1min":1.232,"5min":1.036,"15min":0.98899999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.972}]}]},{"osd":3,"up_from":13,"seq":55834576031,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":3424256,"data_stored":1696209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.91800000000000004,"15min":0.96299999999999997},"min":{"1min":0.58899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.369,"5min":1.48,"15min":2.0430000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.98599999999999999,"15min":1.042},"min":{"1min":0.58899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2210000000000001,"5min":1.722,"15min":2.5990000000000002},"last":1.3939999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":1.01},"min":{"1min":0.69599999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.375,"5min":1.776,"15min":2.589},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1279999999999999,"15min":1.196},"min":{"1min":0.65300000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.6060000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.097}]},{"osd":2,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2649999999999999,"15min":1.383},"min":{"1min":0.81100000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.629,"5min":2.0899999999999999,"15min":2.302},"last":1.772},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3740000000000001,"15min":1.478},"min":{"1min":0.76100000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":1.954,"15min":2.754},"last":1.5409999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.292,"15min":1.3480000000000001},"min":{"1min":0.92000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.958,"5min":2.0110000000000001,"15min":2.512},"last":1.698},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3400000000000001,"15min":1.407},"min":{"1min":0.92200000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.194,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.216}]},{"osd":5,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.51,"15min":1.552},"min":{"1min":0.95799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0710000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.5,"15min":1.556},"min":{"1min":1.0549999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.109,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.9259999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.4690000000000001,"15min":1.4990000000000001},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2090000000000001},"last":1.835},{"interface":"front","average":{"1min":1.532,"5min":1.5589999999999999,"15min":1.5900000000000001},"min":{"1min":1.022,"5min":0.749,"15min":0.749},"max":{"1min":2.1419999999999999,"5min":2.3029999999999999,"15min":2.706},"last":1.998}]},{"osd":7,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.575,"15min":1.6679999999999999},"min":{"1min":0.96299999999999997,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0619999999999998,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":2.145},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0910000000000002,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":2.085}]}]},{"osd":2,"up_from":13,"seq":55834576032,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56708,"kb_used_data":3320,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315132,"statfs":{"total":96636764160,"available":96578695168,"internally_reserved":0,"allocated":3399680,"data_stored":1682907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":0.92000000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.3080000000000001,"5min":1.726,"15min":1.76},"last":0.623},{"interface":"front","average":{"1min":0.879,"5min":0.97799999999999998,"15min":1.0780000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6739999999999999,"5min":1.804,"15min":1.8999999999999999},"last":0.68899999999999995}]},{"osd":1,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97599999999999998,"15min":1.0369999999999999},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1739999999999999,"5min":1.974,"15min":1.974},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0900000000000001,"15min":1.206},"min":{"1min":0.502,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4990000000000001,"5min":1.879,"15min":1.879},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.107,"15min":1.256},"min":{"1min":0.64100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3979999999999999,"5min":1.704,"15min":2.1509999999999998},"last":0.871},{"interface":"front","average":{"1min":1.099,"5min":1.272,"15min":1.3460000000000001},"min":{"1min":0.629,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.569,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.1970000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.4019999999999999,"15min":1.4610000000000001},"min":{"1min":0.79700000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0409999999999999,"5min":2.2669999999999999,"15min":2.335},"last":1.371},{"interface":"front","average":{"1min":1.266,"5min":1.3160000000000001,"15min":1.409},"min":{"1min":0.90400000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7909999999999999,"5min":1.901,"15min":2.1469999999999998},"last":1.073}]},{"osd":5,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.377,"15min":1.466},"min":{"1min":0.80100000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.008,"5min":2.3050000000000002,"15min":2.657},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.492,"15min":1.593},"min":{"1min":0.85799999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.9039999999999999,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.5289999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4299999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.881,"5min":1.9510000000000001,"15min":2.3849999999999998},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.5109999999999999,"15min":1.6060000000000001},"min":{"1min":0.91200000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.831,"5min":2.3759999999999999,"15min":2.7090000000000001},"last":1.4039999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.544,"15min":1.645},"min":{"1min":1.171,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.181,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.617,"15min":1.7150000000000001},"min":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0960000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.2869999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576032,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52868,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318972,"statfs":{"total":96636764160,"available":96582627328,"internally_reserved":0,"allocated":3727360,"data_stored":2010531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.90900000000000003,"15min":0.93999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.351,"5min":1.49,"15min":1.8420000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98899999999999999,"15min":1.113},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6639999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.1299999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.94199999999999995,"15min":1.0209999999999999},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.764,"5min":1.764,"15min":2.286},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.125,"15min":1.214},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7929999999999999,"15min":1.853},"last":1.359}]},{"osd":3,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.254,"15min":1.353},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.651,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":1.53},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1879999999999999,"15min":1.2929999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.532,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":1.405}]},{"osd":4,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.319,"15min":1.365},"min":{"1min":0.89200000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.915,"5min":2.327,"15min":2.3809999999999998},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.47,"15min":1.5109999999999999},"min":{"1min":0.874,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.8759999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.706}]},{"osd":5,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.52},"min":{"1min":0.86499999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9830000000000001,"5min":2.456,"15min":2.456},"last":2.3610000000000002},{"interface":"front","average":{"1min":1.252,"5min":1.4450000000000001,"15min":1.484},"min":{"1min":0.79200000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9490000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.5920000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.595,"15min":1.641},"min":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0489999999999999,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":2.0139999999999998},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5880000000000001,"15min":1.637},"min":{"1min":0.84199999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.343,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":2.2309999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.6499999999999999,"15min":1.7310000000000001},"min":{"1min":0.79900000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1379999999999999,"5min":2.734,"15min":3.2090000000000001},"last":2.4729999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.5820000000000001,"15min":1.6399999999999999},"min":{"1min":0.89000000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1110000000000002,"5min":2.452,"15min":3.1539999999999999},"last":1.754}]}]},{"osd":0,"up_from":13,"seq":55834576032,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61460,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310380,"statfs":{"total":96636764160,"available":96573829120,"internally_reserved":0,"allocated":4005888,"data_stored":2273924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88900000000000001,"15min":0.94299999999999995},"min":{"1min":0.47299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.468,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0620000000000001,"15min":1.1899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4630000000000001,"5min":2.036,"15min":2.036},"last":1.2569999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1439999999999999,"15min":1.248},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.712,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.9379999999999999},{"interface":"front","average":{"1min":1,"5min":1.083,"15min":1.2},"min":{"1min":0.54200000000000004,"5min":0.439,"15min":0.439},"max":{"1min":1.5600000000000001,"5min":1.7330000000000001,"15min":1.7849999999999999},"last":1.478}]},{"osd":3,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96299999999999997,"15min":1.018},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.208,"5min":1.6339999999999999,"15min":1.6799999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1319999999999999,"15min":1.238},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6579999999999999,"5min":1.841,"15min":2.0899999999999999},"last":1.1279999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.258,"15min":1.3200000000000001},"min":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.542,"5min":1.9890000000000001,"15min":2.137},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3660000000000001,"15min":1.4850000000000001},"min":{"1min":0.80800000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.113,"5min":2.113,"15min":2.452},"last":1.593}]},{"osd":5,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3799999999999999,"15min":1.4570000000000001},"min":{"1min":0.79300000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0550000000000002,"5min":2.1059999999999999,"15min":2.137},"last":1.861},{"interface":"front","average":{"1min":1.274,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.76800000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2,"5min":2,"15min":2.282},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5049999999999999,"15min":1.615},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.7010000000000001},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.502,"15min":1.5629999999999999},"min":{"1min":0.73899999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.982}]},{"osd":7,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.589,"15min":1.667},"min":{"1min":1.01,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1230000000000002,"5min":2.339,"15min":2.339},"last":2.101},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.516,"15min":1.5840000000000001},"min":{"1min":0.86099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.274},"last":1.4039999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:56:14.511 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T15:56:14.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:56:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.740+0000 7fe162683640 1 Processor -- start 2023-10-16T15:56:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.741+0000 7fe162683640 1 -- start start 2023-10-16T15:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe162683640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cbe0 0x7fe15c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe162683640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe15c0b4c80 0x7fe15c0b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe162683640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c0d15a0 con 0x7fe15c15cfc0 2023-10-16T15:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe162683640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c0d1880 con 0x7fe15c0b4c80 2023-10-16T15:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe162683640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c0d1b60 con 0x7fe15c15cbe0 2023-10-16T15:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe161e82640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe15c0b4c80 0x7fe15c0b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe160e80640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe15c15cfc0 0x7fe15c0b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52204/0 (socket says 172.21.15.29:52204) 2023-10-16T15:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.742+0000 7fe160e80640 1 -- 172.21.15.29:0/4112993006 learned_addr learned my addr 172.21.15.29:0/4112993006 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.743+0000 7fe161681640 1 --2- 172.21.15.29:0/4112993006 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cbe0 0x7fe15c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.743+0000 7fe161e82640 1 -- 172.21.15.29:0/4112993006 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cbe0 msgr2=0x7fe15c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.743+0000 7fe161e82640 1 --2- 172.21.15.29:0/4112993006 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cbe0 0x7fe15c0d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.743+0000 7fe161e82640 1 -- 172.21.15.29:0/4112993006 >> v1:172.21.15.29:6789/0 conn(0x7fe15c15cfc0 legacy=0x7fe15c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.743+0000 7fe161e82640 1 -- 172.21.15.29:0/4112993006 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0d1e40 con 0x7fe15c0b4c80 2023-10-16T15:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.743+0000 7fe161681640 1 --2- 172.21.15.29:0/4112993006 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cbe0 0x7fe15c0d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:14.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.744+0000 7fe161e82640 1 --2- 172.21.15.29:0/4112993006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0b5c20 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0b53f0 tx=0x7fe15402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1110113a2775fbeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:14.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.744+0000 7fe153fff640 1 -- 172.21.15.29:0/4112993006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15402cc70 con 0x7fe15c0b4c80 2023-10-16T15:56:14.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.744+0000 7fe153fff640 1 -- 172.21.15.29:0/4112993006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15402cdd0 con 0x7fe15c0b4c80 2023-10-16T15:56:14.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.744+0000 7fe153fff640 1 -- 172.21.15.29:0/4112993006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154034660 con 0x7fe15c0b4c80 2023-10-16T15:56:14.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 -- 172.21.15.29:0/4112993006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 msgr2=0x7fe15c0b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 --2- 172.21.15.29:0/4112993006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0b5c20 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0b53f0 tx=0x7fe15402bec0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 -- 172.21.15.29:0/4112993006 shutdown_connections 2023-10-16T15:56:14.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 --2- 172.21.15.29:0/4112993006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0b5c20 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 --2- 172.21.15.29:0/4112993006 >> v2:172.21.15.170:3300/0 conn(0x7fe15c15cbe0 0x7fe15c0d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 -- 172.21.15.29:0/4112993006 >> 172.21.15.29:0/4112993006 conn(0x7fe15c0cf570 msgr2=0x7fe15c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:14.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 -- 172.21.15.29:0/4112993006 shutdown_connections 2023-10-16T15:56:14.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.745+0000 7fe162683640 1 -- 172.21.15.29:0/4112993006 wait complete. 2023-10-16T15:56:14.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.747+0000 7fe162683640 1 Processor -- start 2023-10-16T15:56:14.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.747+0000 7fe162683640 1 -- start start 2023-10-16T15:56:14.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe162683640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe162683640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c15cbe0 0x7fe15c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe162683640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cfc0 0x7fe15c15e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe162683640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c15f110 con 0x7fe15c0b4c80 2023-10-16T15:56:14.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe162683640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c15f3f0 con 0x7fe15c15cbe0 2023-10-16T15:56:14.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe162683640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c15f6d0 con 0x7fe15c15cfc0 2023-10-16T15:56:14.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe161681640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe161681640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34274/0 (socket says 172.21.15.29:34274) 2023-10-16T15:56:14.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.748+0000 7fe161681640 1 -- 172.21.15.29:0/2995445661 learned_addr learned my addr 172.21.15.29:0/2995445661 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:14.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe161681640 1 -- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cfc0 msgr2=0x7fe15c15e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe161681640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cfc0 0x7fe15c15e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe161681640 1 -- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c15cbe0 msgr2=0x7fe15c0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe161681640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c15cbe0 0x7fe15c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe161681640 1 -- 172.21.15.29:0/2995445661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154009a70 con 0x7fe15c0b4c80 2023-10-16T15:56:14.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe161681640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0c9bc0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c7f0 tx=0x7fe15800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:14.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158007a30 con 0x7fe15c0b4c80 2023-10-16T15:56:14.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe158007b90 con 0x7fe15c0b4c80 2023-10-16T15:56:14.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15800ce40 con 0x7fe15c0b4c80 2023-10-16T15:56:14.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.749+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c15f9b0 con 0x7fe15c0b4c80 2023-10-16T15:56:14.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.750+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c1f5570 con 0x7fe15c0b4c80 2023-10-16T15:56:14.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.751+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c1f3670 con 0x7fe15c0b4c80 2023-10-16T15:56:14.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.752+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe1580113f0 con 0x7fe15c0b4c80 2023-10-16T15:56:14.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.753+0000 7fe151ffb640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe14006afe0 0x7fe14006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:14.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.753+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fe1580899d0 con 0x7fe15c0b4c80 2023-10-16T15:56:14.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.754+0000 7fe160e80640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe14006afe0 0x7fe14006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:14.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.755+0000 7fe160e80640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe14006afe0 0x7fe14006d4a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe15c1553b0 tx=0x7fe14c007a40 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:14.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.757+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe15c1f3670 con 0x7fe15c0b4c80 2023-10-16T15:56:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.948+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe15c15f560 con 0x7fe14006afe0 2023-10-16T15:56:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.955+0000 7fe151ffb640 1 -- 172.21.15.29:0/2995445661 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47884 (secure 0 0 0) 0x7fe15c15f560 con 0x7fe14006afe0 2023-10-16T15:56:14.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:56:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.957+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe14006afe0 msgr2=0x7fe14006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.957+0000 7fe162683640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe14006afe0 0x7fe14006d4a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe15c1553b0 tx=0x7fe14c007a40 comp rx=0 tx=0).stop 2023-10-16T15:56:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.957+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 msgr2=0x7fe15c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.957+0000 7fe162683640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0c9bc0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c7f0 tx=0x7fe15800cbf0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 shutdown_connections 2023-10-16T15:56:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fe14006afe0 0x7fe14006d4a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fe15c15cfc0 0x7fe15c15e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c15cbe0 0x7fe15c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 --2- 172.21.15.29:0/2995445661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe15c0b4c80 0x7fe15c0c9bc0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:14.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 >> 172.21.15.29:0/2995445661 conn(0x7fe15c0cf570 msgr2=0x7fe15c0b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:14.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 shutdown_connections 2023-10-16T15:56:14.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:14.958+0000 7fe162683640 1 -- 172.21.15.29:0/2995445661 wait complete. 2023-10-16T15:56:14.980 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-16T15:56:14.823728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":456476,"kb_used_data":27944,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754518244,"statfs":{"total":773094113280,"available":772626681856,"internally_reserved":0,"allocated":28614656,"data_stored":14750024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":438685241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668451,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-262,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.787836"},"pg_stats":[{"pgid":"1.0","version":"268'2809","reported_seq":3923,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T15:56:09.046325+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:56:09.046325+0000","last_peered":"2023-10-16T15:56:09.046325+0000","last_clean":"2023-10-16T15:56:09.046325+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:56:09.046325+0000","last_undegraded":"2023-10-16T15:56:09.046325+0000","last_fullsized":"2023-10-16T15:56:09.046325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576033,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57316,"kb_used_data":3928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314524,"statfs":{"total":96636764160,"available":96578072576,"internally_reserved":0,"allocated":4022272,"data_stored":2286529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0820000000000001,"15min":1.163},"min":{"1min":0.70099999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.72,"5min":1.794,"15min":1.8879999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.083,"15min":1.1339999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.486,"5min":1.796,"15min":1.796},"last":0.88800000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.105,"15min":1.1559999999999999},"min":{"1min":0.66800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7170000000000001,"5min":1.76,"15min":1.8580000000000001},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7809999999999999,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2969999999999999,"15min":1.3260000000000001},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.77,"5min":1.893,"15min":2.1269999999999998},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.127,"5min":1.2230000000000001,"15min":1.2829999999999999},"min":{"1min":0.82999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7110000000000001,"5min":1.909,"15min":1.95},"last":1.276}]},{"osd":3,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2629999999999999,"15min":1.327},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.823,"5min":1.843,"15min":1.869},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.387,"15min":1.4390000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8500000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.151}]},{"osd":4,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.385,"15min":1.385},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.907,"5min":1.9370000000000001,"15min":2.2610000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.337},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.667,"5min":1.8740000000000001,"15min":2.0409999999999999},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.456},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":2.1469999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.502,"15min":1.552},"min":{"1min":0.67900000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3540000000000001},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4430000000000001,"15min":1.4770000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9330000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.587,"5min":1.5609999999999999,"15min":1.5800000000000001},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.998,"5min":1.998,"15min":2.2599999999999998},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52576,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.208,"15min":1.206},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9099999999999999,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.2130000000000001,"15min":1.2430000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":1.5309999999999999,"5min":1.905,"15min":1.905},"last":0.88700000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.72199999999999998},"max":{"1min":1.675,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2889999999999999,"15min":1.3160000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.2889999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2330000000000001,"15min":1.232},"min":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.611,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.464},{"interface":"front","average":{"1min":1.323,"5min":1.3859999999999999,"15min":1.3959999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8400000000000001,"5min":1.905,"15min":2.0339999999999998},"last":1.0349999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.361,"15min":1.391},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.9079999999999999,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.095},{"interface":"front","average":{"1min":1.236,"5min":1.284,"15min":1.331},"min":{"1min":0.82799999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.2210000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.421,"15min":1.351},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.978,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.53},{"interface":"front","average":{"1min":1.512,"5min":1.458,"15min":1.4119999999999999},"min":{"1min":0.81100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0830000000000002,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.431}]},{"osd":5,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3759999999999999,"15min":1.357},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.67300000000000004},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5980000000000001,"15min":1.54},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1059999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.5920000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.6419999999999999,"15min":1.6040000000000001},"min":{"1min":0.86699999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.4940000000000002},"last":1.504},{"interface":"front","average":{"1min":1.599,"5min":1.6399999999999999,"15min":1.5980000000000001},"min":{"1min":0.98099999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.5649999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48420,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323420,"statfs":{"total":96636764160,"available":96587182080,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.093,"15min":1.139},"min":{"1min":0.64600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2190000000000001,"15min":1.2390000000000001},"min":{"1min":0.66400000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.885,"5min":2.738,"15min":2.738},"last":1.0840000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1859999999999999,"15min":1.216},"min":{"1min":0.71999999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":0.749},{"interface":"front","average":{"1min":1.173,"5min":1.2150000000000001,"15min":1.2789999999999999},"min":{"1min":0.75,"5min":0.443,"15min":0.443},"max":{"1min":1.579,"5min":2.137,"15min":2.137},"last":0.84199999999999997}]},{"osd":2,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1779999999999999,"15min":1.2350000000000001},"min":{"1min":0.51900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.621,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.286,"5min":1.339,"15min":1.381},"min":{"1min":0.72699999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.891,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.268}]},{"osd":3,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.29,"15min":1.3260000000000001},"min":{"1min":0.81399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5509999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.194},{"interface":"front","average":{"1min":1.208,"5min":1.2749999999999999,"15min":1.359},"min":{"1min":0.63400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.5149999999999999,"5min":1.966,"15min":1.9670000000000001},"last":1.143}]},{"osd":4,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.3460000000000001},"min":{"1min":0.91900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.623,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.383},{"interface":"front","average":{"1min":1.365,"5min":1.4670000000000001,"15min":1.5029999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7250000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.49}]},{"osd":6,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4299999999999999,"15min":1.3919999999999999},"min":{"1min":0.90700000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.077,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.351},{"interface":"front","average":{"1min":1.347,"5min":1.425,"15min":1.429},"min":{"1min":0.84399999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6950000000000001,"5min":2.0230000000000001,"15min":2.0609999999999999},"last":1.536}]},{"osd":7,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.498,"15min":1.5269999999999999},"min":{"1min":1.246,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8320000000000001,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.319},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6319999999999999,"15min":1.6339999999999999},"min":{"1min":1.284,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2029999999999998,"5min":3.109,"15min":3.109},"last":1.4490000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70624,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301216,"statfs":{"total":96636764160,"available":96564445184,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1279999999999999,"15min":1.1890000000000001},"min":{"1min":0.749,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.161},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.1419999999999999,"15min":1.1830000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.90800000000000003}]},{"osd":1,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.629},"max":{"1min":1.6990000000000001,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.83999999999999997,"5min":0.75900000000000001,"15min":0.70199999999999996},"max":{"1min":1.6579999999999999,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":0.73899999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4099999999999999,"15min":1.452},"min":{"1min":0.91800000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.8929999999999998},"last":1.278},{"interface":"front","average":{"1min":1.419,"5min":1.452,"15min":1.4850000000000001},"min":{"1min":0.88900000000000001,"5min":0.82499999999999996,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.2370000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.383,"15min":1.377},"min":{"1min":0.91100000000000003,"5min":0.84699999999999998,"15min":0.79400000000000004},"max":{"1min":1.915,"5min":2.23,"15min":2.3650000000000002},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3540000000000001,"15min":1.3819999999999999},"min":{"1min":0.81999999999999995,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.492}]},{"osd":5,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3979999999999999,"15min":1.3500000000000001},"min":{"1min":0.89400000000000002,"5min":0.80200000000000005,"15min":0.754},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.046},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4490000000000001},"min":{"1min":1.056,"5min":0.88900000000000001,"15min":0.74399999999999999},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":1.526}]},{"osd":6,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.427,"15min":1.4079999999999999},"min":{"1min":0.91300000000000003,"5min":0.90500000000000003,"15min":0.75900000000000001},"max":{"1min":2.129,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.329},{"interface":"front","average":{"1min":1.621,"5min":1.597,"15min":1.538},"min":{"1min":1.119,"5min":1.0089999999999999,"15min":0.82499999999999996},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.4370000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.619,"15min":1.5880000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.83999999999999997},"max":{"1min":2.2189999999999999,"5min":2.399,"15min":2.399},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.6379999999999999,"15min":1.611},"min":{"1min":1.232,"5min":1.036,"15min":0.98899999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.583}]}]},{"osd":3,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56740,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.91800000000000004,"15min":0.96299999999999997},"min":{"1min":0.58899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.369,"5min":1.48,"15min":2.0430000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.98599999999999999,"15min":1.042},"min":{"1min":0.58899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2210000000000001,"5min":1.722,"15min":2.5990000000000002},"last":0.84899999999999998}]},{"osd":1,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":1.01},"min":{"1min":0.69599999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.375,"5min":1.776,"15min":2.589},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1279999999999999,"15min":1.196},"min":{"1min":0.65300000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.6060000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.077}]},{"osd":2,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2649999999999999,"15min":1.383},"min":{"1min":0.81100000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.629,"5min":2.0899999999999999,"15min":2.302},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3740000000000001,"15min":1.478},"min":{"1min":0.76100000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":1.954,"15min":2.754},"last":1.8600000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.292,"15min":1.3480000000000001},"min":{"1min":0.92000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.958,"5min":2.0110000000000001,"15min":2.512},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3400000000000001,"15min":1.407},"min":{"1min":0.92200000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.194,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.2909999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.51,"15min":1.552},"min":{"1min":0.95799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0710000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.962},{"interface":"front","average":{"1min":1.528,"5min":1.5,"15min":1.556},"min":{"1min":1.0549999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.109,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.3220000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.4690000000000001,"15min":1.4990000000000001},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2090000000000001},"last":2.2679999999999998},{"interface":"front","average":{"1min":1.532,"5min":1.5589999999999999,"15min":1.5900000000000001},"min":{"1min":1.022,"5min":0.749,"15min":0.749},"max":{"1min":2.1419999999999999,"5min":2.3029999999999999,"15min":2.706},"last":2.1869999999999998}]},{"osd":7,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.575,"15min":1.6679999999999999},"min":{"1min":0.96299999999999997,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0619999999999998,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":1.75},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0910000000000002,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":2.3620000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56740,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":0.92000000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.3080000000000001,"5min":1.726,"15min":1.76},"last":1.103},{"interface":"front","average":{"1min":0.879,"5min":0.97799999999999998,"15min":1.0780000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6739999999999999,"5min":1.804,"15min":1.8999999999999999},"last":1.6539999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97599999999999998,"15min":1.0369999999999999},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1739999999999999,"5min":1.974,"15min":1.974},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0900000000000001,"15min":1.206},"min":{"1min":0.502,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4990000000000001,"5min":1.879,"15min":1.879},"last":1.3029999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.107,"15min":1.256},"min":{"1min":0.64100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3979999999999999,"5min":1.704,"15min":2.1509999999999998},"last":1.149},{"interface":"front","average":{"1min":1.099,"5min":1.272,"15min":1.3460000000000001},"min":{"1min":0.629,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.569,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.522}]},{"osd":4,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.4019999999999999,"15min":1.4610000000000001},"min":{"1min":0.79700000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0409999999999999,"5min":2.2669999999999999,"15min":2.335},"last":1.78},{"interface":"front","average":{"1min":1.266,"5min":1.3160000000000001,"15min":1.409},"min":{"1min":0.90400000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7909999999999999,"5min":1.901,"15min":2.1469999999999998},"last":1.4830000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.377,"15min":1.466},"min":{"1min":0.80100000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.008,"5min":2.3050000000000002,"15min":2.657},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.492,"15min":1.593},"min":{"1min":0.85799999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.9039999999999999,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.619}]},{"osd":6,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4299999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.881,"5min":1.9510000000000001,"15min":2.3849999999999998},"last":1.369},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.5109999999999999,"15min":1.6060000000000001},"min":{"1min":0.91200000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.831,"5min":2.3759999999999999,"15min":2.7090000000000001},"last":1.4199999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.544,"15min":1.645},"min":{"1min":1.171,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.181,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.617,"15min":1.7150000000000001},"min":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0960000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.724}]}]},{"osd":1,"up_from":13,"seq":55834576033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52580,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.90900000000000003,"15min":0.93999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.351,"5min":1.49,"15min":1.8420000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98899999999999999,"15min":1.113},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6639999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.94199999999999995,"15min":1.0209999999999999},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.764,"5min":1.764,"15min":2.286},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.125,"15min":1.214},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7929999999999999,"15min":1.853},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.254,"15min":1.353},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.651,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1879999999999999,"15min":1.2929999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.532,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":0.84499999999999997}]},{"osd":4,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.319,"15min":1.365},"min":{"1min":0.89200000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.915,"5min":2.327,"15min":2.3809999999999998},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.47,"15min":1.5109999999999999},"min":{"1min":0.874,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.8759999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.93500000000000005}]},{"osd":5,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.52},"min":{"1min":0.86499999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9830000000000001,"5min":2.456,"15min":2.456},"last":1.222},{"interface":"front","average":{"1min":1.252,"5min":1.4450000000000001,"15min":1.484},"min":{"1min":0.79200000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9490000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":0.95199999999999996}]},{"osd":6,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.595,"15min":1.641},"min":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0489999999999999,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5880000000000001,"15min":1.637},"min":{"1min":0.84199999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.343,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.3149999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.6499999999999999,"15min":1.7310000000000001},"min":{"1min":0.79900000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1379999999999999,"5min":2.734,"15min":3.2090000000000001},"last":1.284},{"interface":"front","average":{"1min":1.381,"5min":1.5820000000000001,"15min":1.6399999999999999},"min":{"1min":0.89000000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1110000000000002,"5min":2.452,"15min":3.1539999999999999},"last":1.373}]}]},{"osd":0,"up_from":13,"seq":55834576033,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61480,"kb_used_data":3928,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310360,"statfs":{"total":96636764160,"available":96573808640,"internally_reserved":0,"allocated":4022272,"data_stored":2286529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88900000000000001,"15min":0.94299999999999995},"min":{"1min":0.47299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.468,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0620000000000001,"15min":1.1899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4630000000000001,"5min":2.036,"15min":2.036},"last":1.266}]},{"osd":2,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1439999999999999,"15min":1.248},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.712,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1,"5min":1.083,"15min":1.2},"min":{"1min":0.54200000000000004,"5min":0.439,"15min":0.439},"max":{"1min":1.5600000000000001,"5min":1.7330000000000001,"15min":1.7849999999999999},"last":1.1910000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96299999999999997,"15min":1.018},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.208,"5min":1.6339999999999999,"15min":1.6799999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1319999999999999,"15min":1.238},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6579999999999999,"5min":1.841,"15min":2.0899999999999999},"last":1.5129999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.258,"15min":1.3200000000000001},"min":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.542,"5min":1.9890000000000001,"15min":2.137},"last":1.859},{"interface":"front","average":{"1min":1.319,"5min":1.3660000000000001,"15min":1.4850000000000001},"min":{"1min":0.80800000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.113,"5min":2.113,"15min":2.452},"last":1.6879999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3799999999999999,"15min":1.4570000000000001},"min":{"1min":0.79300000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0550000000000002,"5min":2.1059999999999999,"15min":2.137},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.76800000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2,"5min":2,"15min":2.282},"last":1.774}]},{"osd":6,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5049999999999999,"15min":1.615},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.7010000000000001},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.502,"15min":1.5629999999999999},"min":{"1min":0.73899999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.7370000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.589,"15min":1.667},"min":{"1min":1.01,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1230000000000002,"5min":2.339,"15min":2.339},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.516,"15min":1.5840000000000001},"min":{"1min":0.86099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.274},"last":1.637}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:56:14.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T15:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.208+0000 7f8767e46640 1 Processor -- start 2023-10-16T15:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.208+0000 7f8767e46640 1 -- start start 2023-10-16T15:56:15.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8767e46640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f876015aaa0 0x7f876015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:15.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8767e46640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f87600ce390 0x7f8760070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:15.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8767e46640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876015ed60 con 0x7f876015ae80 2023-10-16T15:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8767e46640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876006b810 con 0x7f876015aaa0 2023-10-16T15:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8767e46640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87600707e0 con 0x7f87600ce390 2023-10-16T15:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8765be2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f876015aaa0 0x7f876015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8765be2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f876015aaa0 0x7f876015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34288/0 (socket says 172.21.15.29:34288) 2023-10-16T15:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.210+0000 7f8765be2640 1 -- 172.21.15.29:0/1382115879 learned_addr learned my addr 172.21.15.29:0/1382115879 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.211+0000 7f87663e3640 1 --2- 172.21.15.29:0/1382115879 >> v2:172.21.15.170:3300/0 conn(0x7f87600ce390 0x7f8760070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.211+0000 7f8765be2640 1 -- 172.21.15.29:0/1382115879 >> v2:172.21.15.170:3300/0 conn(0x7f87600ce390 msgr2=0x7f8760070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.211+0000 7f8765be2640 1 --2- 172.21.15.29:0/1382115879 >> v2:172.21.15.170:3300/0 conn(0x7f87600ce390 0x7f8760070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.211+0000 7f8765be2640 1 -- 172.21.15.29:0/1382115879 >> v1:172.21.15.170:6789/0 conn(0x7f876015ae80 legacy=0x7f87600cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:15.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.211+0000 7f8765be2640 1 -- 172.21.15.29:0/1382115879 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876015ef90 con 0x7f876015aaa0 2023-10-16T15:56:15.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.212+0000 7f8765be2640 1 --2- 172.21.15.29:0/1382115879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f876015e820 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f87600c2490 tx=0x7f875c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb63231285c5383 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:15.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.212+0000 7f8764be0640 1 -- 172.21.15.29:0/1382115879 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c02ca80 con 0x7f876015aaa0 2023-10-16T15:56:15.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.212+0000 7f8764be0640 1 -- 172.21.15.29:0/1382115879 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c02cbe0 con 0x7f876015aaa0 2023-10-16T15:56:15.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.212+0000 7f8764be0640 1 -- 172.21.15.29:0/1382115879 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0343f0 con 0x7f876015aaa0 2023-10-16T15:56:15.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 -- 172.21.15.29:0/1382115879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 msgr2=0x7f876015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:15.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 --2- 172.21.15.29:0/1382115879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f876015e820 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f87600c2490 tx=0x7f875c02bee0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 -- 172.21.15.29:0/1382115879 shutdown_connections 2023-10-16T15:56:15.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 --2- 172.21.15.29:0/1382115879 >> v2:172.21.15.170:3300/0 conn(0x7f87600ce390 0x7f8760070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 --2- 172.21.15.29:0/1382115879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f876015e820 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 -- 172.21.15.29:0/1382115879 >> 172.21.15.29:0/1382115879 conn(0x7f87600cb410 msgr2=0x7f87600cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 -- 172.21.15.29:0/1382115879 shutdown_connections 2023-10-16T15:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.213+0000 7f8767e46640 1 -- 172.21.15.29:0/1382115879 wait complete. 2023-10-16T15:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.215+0000 7f8767e46640 1 Processor -- start 2023-10-16T15:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.215+0000 7f8767e46640 1 -- start start 2023-10-16T15:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f8767e46640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87600ce390 0x7f87600b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f8767e46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f87601552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f8767e46640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876015ae80 0x7f8760153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f8767e46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87601ef160 con 0x7f876015aaa0 2023-10-16T15:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f8767e46640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87601ee4e0 con 0x7f876015ae80 2023-10-16T15:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f8767e46640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87601ee7c0 con 0x7f87600ce390 2023-10-16T15:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f87653e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f87601552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f87653e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f87601552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34292/0 (socket says 172.21.15.29:34292) 2023-10-16T15:56:15.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.216+0000 7f87653e1640 1 -- 172.21.15.29:0/3187278155 learned_addr learned my addr 172.21.15.29:0/3187278155 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:15.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f8765be2640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87600ce390 0x7f87600b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:15.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87663e3640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876015ae80 0x7f8760153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:15.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87653e1640 1 -- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87600ce390 msgr2=0x7f87600b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:15.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87653e1640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87600ce390 0x7f87600b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87653e1640 1 -- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876015ae80 msgr2=0x7f8760153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87653e1640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876015ae80 0x7f8760153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87653e1640 1 -- 172.21.15.29:0/3187278155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c009b30 con 0x7f876015aaa0 2023-10-16T15:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87653e1640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f87601552c0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f875000b3c0 tx=0x7f875000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.217+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750011830 con 0x7f876015aaa0 2023-10-16T15:56:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.218+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8750011990 con 0x7f876015aaa0 2023-10-16T15:56:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.218+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87601eeaa0 con 0x7f876015aaa0 2023-10-16T15:56:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.218+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750011d80 con 0x7f876015aaa0 2023-10-16T15:56:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.218+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87600b36e0 con 0x7f876015aaa0 2023-10-16T15:56:15.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.219+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87601ef2a0 con 0x7f876015aaa0 2023-10-16T15:56:15.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.220+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f875001b460 con 0x7f876015aaa0 2023-10-16T15:56:15.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.221+0000 7f87567fc640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f874406af90 0x7f874406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:15.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.221+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f8750089440 con 0x7f876015aaa0 2023-10-16T15:56:15.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.222+0000 7f8765be2640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f874406af90 0x7f874406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:15.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.223+0000 7f8765be2640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f874406af90 0x7f874406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f875c035000 tx=0x7f875c030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:15.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.225+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87601ef2a0 con 0x7f876015aaa0 2023-10-16T15:56:15.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.420+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87601ef2a0 con 0x7f876015aaa0 2023-10-16T15:56:15.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.422+0000 7f87567fc640 1 -- 172.21.15.29:0/3187278155 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+14306 (secure 0 0 0) 0x7f87601ef2a0 con 0x7f876015aaa0 2023-10-16T15:56:15.424 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:15.424 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":271,"fsid":"aff38114-d596-4938-a779-5e1b7478d87b","created":"2023-10-16T14:16:04.817388+0000","modified":"2023-10-16T15:56:10.062470+0000","last_up_change":"2023-10-16T14:16:17.590339+0000","last_in_change":"2023-10-16T14:16:08.697085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T14:16:17.885864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87107ae3-91be-4d1c-a15f-09a9db4e483a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6810","nonce":1553131287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6815","nonce":1553131287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6827","nonce":1553131287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":1553131287},{"type":"v1","addr":"172.21.15.29:6821","nonce":1553131287}]},"public_addr":"172.21.15.29:6810/1553131287","cluster_addr":"172.21.15.29:6815/1553131287","heartbeat_back_addr":"172.21.15.29:6827/1553131287","heartbeat_front_addr":"172.21.15.29:6821/1553131287","state":["exists","up"]},{"osd":1,"uuid":"096d82fa-2713-4df9-ba03-9533658a0d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6801","nonce":3296543673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6803","nonce":3296543673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6807","nonce":3296543673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3296543673},{"type":"v1","addr":"172.21.15.29:6805","nonce":3296543673}]},"public_addr":"172.21.15.29:6801/3296543673","cluster_addr":"172.21.15.29:6803/3296543673","heartbeat_back_addr":"172.21.15.29:6807/3296543673","heartbeat_front_addr":"172.21.15.29:6805/3296543673","state":["exists","up"]},{"osd":2,"uuid":"121bf1ec-267b-4a07-aec1-e63cd130ccea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6811","nonce":4164234701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6814","nonce":4164234701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6824","nonce":4164234701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":4164234701},{"type":"v1","addr":"172.21.15.29:6818","nonce":4164234701}]},"public_addr":"172.21.15.29:6811/4164234701","cluster_addr":"172.21.15.29:6814/4164234701","heartbeat_back_addr":"172.21.15.29:6824/4164234701","heartbeat_front_addr":"172.21.15.29:6818/4164234701","state":["exists","up"]},{"osd":3,"uuid":"71979c26-ad9d-4dd4-bb21-3f766e254965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6820","nonce":1457722234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6823","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6826","nonce":1457722234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6831","nonce":1457722234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1457722234},{"type":"v1","addr":"172.21.15.29:6829","nonce":1457722234}]},"public_addr":"172.21.15.29:6820/1457722234","cluster_addr":"172.21.15.29:6826/1457722234","heartbeat_back_addr":"172.21.15.29:6831/1457722234","heartbeat_front_addr":"172.21.15.29:6829/1457722234","state":["exists","up"]},{"osd":4,"uuid":"d2efc192-e765-4786-bd6f-141672cf5d53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6809","nonce":1474743223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6811","nonce":1474743223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6815","nonce":1474743223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":1474743223},{"type":"v1","addr":"172.21.15.170:6813","nonce":1474743223}]},"public_addr":"172.21.15.170:6809/1474743223","cluster_addr":"172.21.15.170:6811/1474743223","heartbeat_back_addr":"172.21.15.170:6815/1474743223","heartbeat_front_addr":"172.21.15.170:6813/1474743223","state":["exists","up"]},{"osd":5,"uuid":"6cf2f5c6-ec40-4efe-b2c2-7890a56f513f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6801","nonce":3780367638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6803","nonce":3780367638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6807","nonce":3780367638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3780367638},{"type":"v1","addr":"172.21.15.170:6805","nonce":3780367638}]},"public_addr":"172.21.15.170:6801/3780367638","cluster_addr":"172.21.15.170:6803/3780367638","heartbeat_back_addr":"172.21.15.170:6807/3780367638","heartbeat_front_addr":"172.21.15.170:6805/3780367638","state":["exists","up"]},{"osd":6,"uuid":"886d6f1f-a25b-4b98-a2c8-056eab1d62dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6817","nonce":690425284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6819","nonce":690425284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6825","nonce":690425284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":690425284},{"type":"v1","addr":"172.21.15.170:6822","nonce":690425284}]},"public_addr":"172.21.15.170:6817/690425284","cluster_addr":"172.21.15.170:6819/690425284","heartbeat_back_addr":"172.21.15.170:6825/690425284","heartbeat_front_addr":"172.21.15.170:6822/690425284","state":["exists","up"]},{"osd":7,"uuid":"324071bd-38bc-42a6-bde4-8b9546db1433","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6823","nonce":3840642819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6827","nonce":3840642819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6831","nonce":3840642819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3840642819},{"type":"v1","addr":"172.21.15.170:6829","nonce":3840642819}]},"public_addr":"172.21.15.170:6823/3840642819","cluster_addr":"172.21.15.170:6827/3840642819","heartbeat_back_addr":"172.21.15.170:6831/3840642819","heartbeat_front_addr":"172.21.15.170:6829/3840642819","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.277199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.227408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.066406+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.095857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:13.967687+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.188449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.004815+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T14:16:14.039123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/2425185007":"2023-10-17T15:56:07.028593+0000","172.21.15.29:6833/2425185007":"2023-10-17T15:56:07.028593+0000","172.21.15.170:0/567930087":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/936087252":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/2036144584":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/4135571630":"2023-10-17T15:56:02.787159+0000","172.21.15.170:6833/1364949379":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/187921439":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/2485822714":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/965201049":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/831181775":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/1364949379":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/1047565771":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6832/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2742286180":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3423446827":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/1983218799":"2023-10-17T15:56:02.787159+0000","172.21.15.170:0/3865201075":"2023-10-17T15:55:54.590651+0000","172.21.15.170:6833/3469082932":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/763385128":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/2667017960":"2023-10-17T15:55:54.590651+0000","172.21.15.170:0/3985010689":"2023-10-16T16:55:37.652115+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T15:56:15.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f874406af90 msgr2=0x7f874406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:15.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f874406af90 0x7f874406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f875c035000 tx=0x7f875c030000 comp rx=0 tx=0).stop 2023-10-16T15:56:15.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 msgr2=0x7f87601552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f87601552c0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f875000b3c0 tx=0x7f875000b880 comp rx=0 tx=0).stop 2023-10-16T15:56:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 shutdown_connections 2023-10-16T15:56:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f874406af90 0x7f874406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f876015aaa0 0x7f87601552c0 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f876015ae80 0x7f8760153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 --2- 172.21.15.29:0/3187278155 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f87600ce390 0x7f87600b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:15.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 >> 172.21.15.29:0/3187278155 conn(0x7f87600cb410 msgr2=0x7f87600cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:15.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 shutdown_connections 2023-10-16T15:56:15.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:15.425+0000 7f8767e46640 1 -- 172.21.15.29:0/3187278155 wait complete. 2023-10-16T15:56:16.445 INFO:tasks.ceph:Scrubbing osd.0 2023-10-16T15:56:16.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:16.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.691+0000 7ff9242b0640 1 Processor -- start 2023-10-16T15:56:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.691+0000 7ff9242b0640 1 -- start start 2023-10-16T15:56:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.692+0000 7ff9242b0640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff91c15acf0 0x7ff91c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.692+0000 7ff9242b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c06d3d0 0x7ff91c06d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.692+0000 7ff9242b0640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c070260 con 0x7ff91c070500 2023-10-16T15:56:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.692+0000 7ff9242b0640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91c06e2b0 con 0x7ff91c15acf0 2023-10-16T15:56:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.692+0000 7ff9242b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c06e590 con 0x7ff91c06d3d0 2023-10-16T15:56:16.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.693+0000 7ff92204c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff91c15acf0 0x7ff91c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.693+0000 7ff92284d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c06d3d0 0x7ff91c06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.693+0000 7ff92204c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7ff91c15acf0 0x7ff91c06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:34984/0 (socket says 172.21.15.29:34984) 2023-10-16T15:56:16.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.693+0000 7ff92204c640 1 -- 172.21.15.29:0/3223597343 learned_addr learned my addr 172.21.15.29:0/3223597343 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.694+0000 7ff92284d640 1 -- 172.21.15.29:0/3223597343 >> v2:172.21.15.170:3300/0 conn(0x7ff91c15acf0 msgr2=0x7ff91c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.694+0000 7ff92284d640 1 --2- 172.21.15.29:0/3223597343 >> v2:172.21.15.170:3300/0 conn(0x7ff91c15acf0 0x7ff91c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.694+0000 7ff92284d640 1 -- 172.21.15.29:0/3223597343 >> v1:172.21.15.170:6789/0 conn(0x7ff91c070500 legacy=0x7ff91c0ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.694+0000 7ff92284d640 1 -- 172.21.15.29:0/3223597343 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c06cbd0 con 0x7ff91c06d3d0 2023-10-16T15:56:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.695+0000 7ff92284d640 1 --2- 172.21.15.29:0/3223597343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c06d3d0 0x7ff91c06d860 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff914009d90 tx=0x7ff91402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7645f5f3fd3e75a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.696+0000 7ff92104a640 1 -- 172.21.15.29:0/3223597343 <== mon.2 v2:172.21.15.170:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91402cb30 con 0x7ff91c06d3d0 2023-10-16T15:56:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.696+0000 7ff92104a640 1 -- 172.21.15.29:0/3223597343 <== mon.2 v2:172.21.15.170:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91402cc90 con 0x7ff91c06d3d0 2023-10-16T15:56:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.696+0000 7ff92104a640 1 -- 172.21.15.29:0/3223597343 <== mon.2 v2:172.21.15.170:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914034440 con 0x7ff91c06d3d0 2023-10-16T15:56:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 -- 172.21.15.29:0/3223597343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c06d3d0 msgr2=0x7ff91c06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 --2- 172.21.15.29:0/3223597343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c06d3d0 0x7ff91c06d860 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff914009d90 tx=0x7ff91402bd80 comp rx=0 tx=0).stop 2023-10-16T15:56:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 -- 172.21.15.29:0/3223597343 shutdown_connections 2023-10-16T15:56:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 --2- 172.21.15.29:0/3223597343 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c06d3d0 0x7ff91c06d860 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 --2- 172.21.15.29:0/3223597343 >> v2:172.21.15.170:3300/0 conn(0x7ff91c15acf0 0x7ff91c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 -- 172.21.15.29:0/3223597343 >> 172.21.15.29:0/3223597343 conn(0x7ff91c0b0cc0 msgr2=0x7ff91c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:16.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 -- 172.21.15.29:0/3223597343 shutdown_connections 2023-10-16T15:56:16.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.697+0000 7ff9242b0640 1 -- 172.21.15.29:0/3223597343 wait complete. 2023-10-16T15:56:16.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.699+0000 7ff9242b0640 1 Processor -- start 2023-10-16T15:56:16.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.699+0000 7ff9242b0640 1 -- start start 2023-10-16T15:56:16.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.700+0000 7ff9242b0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff91c06d3d0 0x7ff91c0b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.700+0000 7ff9242b0640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c070500 0x7ff91c1f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.700+0000 7ff9242b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff91c15acf0 0x7ff91c1f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.700+0000 7ff9242b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0c5910 con 0x7ff91c15acf0 2023-10-16T15:56:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.700+0000 7ff9242b0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0b4a80 con 0x7ff91c06d3d0 2023-10-16T15:56:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.700+0000 7ff9242b0640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0b3e70 con 0x7ff91c070500 2023-10-16T15:56:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff91c15acf0 0x7ff91c1f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff91c15acf0 0x7ff91c1f6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34308/0 (socket says 172.21.15.29:34308) 2023-10-16T15:56:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 -- 172.21.15.29:0/1396912800 learned_addr learned my addr 172.21.15.29:0/1396912800 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92204c640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff91c06d3d0 0x7ff91c0b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92184b640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c070500 0x7ff91c1f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 -- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c070500 msgr2=0x7ff91c1f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7ff91c070500 0x7ff91c1f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 -- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff91c06d3d0 msgr2=0x7ff91c0b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff91c06d3d0 0x7ff91c0b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.701+0000 7ff92284d640 1 -- 172.21.15.29:0/1396912800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914009a70 con 0x7ff91c15acf0 2023-10-16T15:56:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.702+0000 7ff92284d640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff91c15acf0 0x7ff91c1f6250 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7ff914009eb0 tx=0x7ff914004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.702+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914040450 con 0x7ff91c15acf0 2023-10-16T15:56:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.702+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9140405d0 con 0x7ff91c15acf0 2023-10-16T15:56:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.702+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914040980 con 0x7ff91c15acf0 2023-10-16T15:56:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.702+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c0b4150 con 0x7ff91c15acf0 2023-10-16T15:56:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.702+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c0b4710 con 0x7ff91c15acf0 2023-10-16T15:56:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.703+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff91c06c880 con 0x7ff91c15acf0 2023-10-16T15:56:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.704+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff914040ae0 con 0x7ff91c15acf0 2023-10-16T15:56:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.705+0000 7ff912ffd640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7ff90006af90 0x7ff90006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.705+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7ff9140ac9e0 con 0x7ff91c15acf0 2023-10-16T15:56:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.706+0000 7ff912ffd640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7ff900071520 0x7ff900073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.706+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 --> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff900074010 con 0x7ff900071520 2023-10-16T15:56:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.706+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9140acd50 con 0x7ff91c15acf0 2023-10-16T15:56:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.706+0000 7ff92184b640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7ff900071520 0x7ff900073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.706+0000 7ff92204c640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7ff90006af90 0x7ff90006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.707+0000 7ff92184b640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7ff900071520 0x7ff900073940 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.707+0000 7ff92204c640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7ff90006af90 0x7ff90006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff918009ac0 tx=0x7ff918008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.708+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== osd.0 v2:172.21.15.29:6808/1553131287 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ff900074010 con 0x7ff900071520 2023-10-16T15:56:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.722+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 --> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff91c06ceb0 con 0x7ff900071520 2023-10-16T15:56:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff912ffd640 1 -- 172.21.15.29:0/1396912800 <== osd.0 v2:172.21.15.29:6808/1553131287 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff90c0050f0 con 0x7ff900071520 2023-10-16T15:56:16.724 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:16.724 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:16.724 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7ff900071520 msgr2=0x7ff900073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:6808/1553131287,v1:172.21.15.29:6810/1553131287] conn(0x7ff900071520 0x7ff900073940 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7ff90006af90 msgr2=0x7ff90006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7ff90006af90 0x7ff90006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff918009ac0 tx=0x7ff918008040 comp rx=0 tx=0).stop 2023-10-16T15:56:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff91c15acf0 msgr2=0x7ff91c1f6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 --2- 172.21.15.29:0/1396912800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff91c15acf0 0x7ff91c1f6250 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7ff914009eb0 tx=0x7ff914004a40 comp rx=0 tx=0).stop 2023-10-16T15:56:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff92284d640 1 -- 172.21.15.29:0/1396912800 reap_dead start 2023-10-16T15:56:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 shutdown_connections 2023-10-16T15:56:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.723+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 >> 172.21.15.29:0/1396912800 conn(0x7ff91c0b0cc0 msgr2=0x7ff91c0c9820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.724+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 shutdown_connections 2023-10-16T15:56:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.724+0000 7ff9242b0640 1 -- 172.21.15.29:0/1396912800 wait complete. 2023-10-16T15:56:16.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-16T15:56:16.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.965+0000 7f12b58ca640 1 Processor -- start 2023-10-16T15:56:16.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.965+0000 7f12b58ca640 1 -- start start 2023-10-16T15:56:16.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.967+0000 7f12b58ca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12b00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.967+0000 7f12b58ca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b015b9d0 con 0x7f12b006c040 2023-10-16T15:56:16.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.967+0000 7f12b58ca640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b015bcb0 con 0x7f12b00ccfb0 2023-10-16T15:56:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.967+0000 7f12b58ca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b015bf90 con 0x7f12b00ad970 2023-10-16T15:56:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.967+0000 7f12b48c8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f12b006c040 0x7f12b006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52214/0 (socket says 172.21.15.29:52214) 2023-10-16T15:56:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.967+0000 7f12b48c8640 1 -- 172.21.15.29:0/3327921819 learned_addr learned my addr 172.21.15.29:0/3327921819 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:16.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12affff640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12b00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10630036 0 0) 0x7f12b015b9d0 con 0x7f12b006c040 2023-10-16T15:56:16.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12b006c040 2023-10-16T15:56:16.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1830973876 0 0) 0x7f12a0003520 con 0x7f12b006c040 2023-10-16T15:56:16.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 msgr2=0x7f12b00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:16.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12b00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 >> v1:172.21.15.170:6789/0 conn(0x7f12b00ccfb0 legacy=0x7f12b00cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00cc270 con 0x7f12b006c040 2023-10-16T15:56:16.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f12a4001670 con 0x7f12b006c040 2023-10-16T15:56:16.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 >> v1:172.21.15.29:6789/0 conn(0x7f12b006c040 legacy=0x7f12b006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a00047c0 0x7f12a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b015bf90 con 0x7f12a00039a0 2023-10-16T15:56:16.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000bab0 con 0x7f12a00047c0 2023-10-16T15:56:16.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.968+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000bc70 con 0x7f12b00ad970 2023-10-16T15:56:16.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12affff640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a00047c0 0x7f12a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b50c9640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b48c8640 1 -- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 msgr2=0x7f12a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:16.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b48c8640 1 -- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a00047c0 msgr2=0x7f12a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:16.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12a00047c0 0x7f12a0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b48c8640 1 -- 172.21.15.29:0/3327921819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0003520 con 0x7f12a00039a0 2023-10-16T15:56:16.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.969+0000 7f12b50c9640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00ad970 0x7f12a00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:16.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.970+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12a0003dc0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c0e0 tx=0x7f12a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6daf92ae8d0d6c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4005ad0 con 0x7f12a00039a0 2023-10-16T15:56:16.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4005c30 con 0x7f12a00039a0 2023-10-16T15:56:16.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12af7fe640 1 -- 172.21.15.29:0/3327921819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a402c7f0 con 0x7f12a00039a0 2023-10-16T15:56:16.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b58ca640 1 -- 172.21.15.29:0/3327921819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 msgr2=0x7f12a0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3327921819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12a0003dc0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c0e0 tx=0x7f12a00057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b50c9640 1 -- 172.21.15.29:0/3327921819 reap_dead start 2023-10-16T15:56:16.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b58ca640 1 -- 172.21.15.29:0/3327921819 shutdown_connections 2023-10-16T15:56:16.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b58ca640 1 -- 172.21.15.29:0/3327921819 >> 172.21.15.29:0/3327921819 conn(0x7f12b00ce6a0 msgr2=0x7f12b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:16.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b58ca640 1 -- 172.21.15.29:0/3327921819 shutdown_connections 2023-10-16T15:56:16.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.971+0000 7f12b58ca640 1 -- 172.21.15.29:0/3327921819 wait complete. 2023-10-16T15:56:16.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.973+0000 7f12b58ca640 1 Processor -- start 2023-10-16T15:56:16.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.973+0000 7f12b58ca640 1 -- start start 2023-10-16T15:56:16.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.974+0000 7f12b58ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12b006f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.974+0000 7f12b58ca640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00b2410 0x7f12b014f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.974+0000 7f12b58ca640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b00b2890 0x7f12b00c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.974+0000 7f12b58ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00c9550 con 0x7f12a00039a0 2023-10-16T15:56:16.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.974+0000 7f12b58ca640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00b2dd0 con 0x7f12b00b2890 2023-10-16T15:56:16.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.974+0000 7f12b58ca640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00b30b0 con 0x7f12b00b2410 2023-10-16T15:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12b006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12b006f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34338/0 (socket says 172.21.15.29:34338) 2023-10-16T15:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 -- 172.21.15.29:0/3809991208 learned_addr learned my addr 172.21.15.29:0/3809991208 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12affff640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00b2410 0x7f12b014f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 -- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00b2410 msgr2=0x7f12b014f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00b2410 0x7f12b014f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 -- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b00b2890 msgr2=0x7f12b00c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b00b2890 0x7f12b00c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 -- 172.21.15.29:0/3809991208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00cdcd0 con 0x7f12a00039a0 2023-10-16T15:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12affff640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00b2410 0x7f12b014f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.975+0000 7f12b48c8640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12b006f970 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f12a402c390 tx=0x7f12a4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.976+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a403c040 con 0x7f12a00039a0 2023-10-16T15:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.976+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a402cc00 con 0x7f12a00039a0 2023-10-16T15:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.976+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4007070 con 0x7f12a00039a0 2023-10-16T15:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.976+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b00c9760 con 0x7f12a00039a0 2023-10-16T15:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.976+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b006feb0 con 0x7f12a00039a0 2023-10-16T15:56:16.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.977+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b0070090 con 0x7f12a00039a0 2023-10-16T15:56:16.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.978+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f12a4032070 con 0x7f12a00039a0 2023-10-16T15:56:16.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.979+0000 7f12ad7fa640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f129406af90 0x7f129406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:16.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.979+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f12a40aab10 con 0x7f12a00039a0 2023-10-16T15:56:16.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.980+0000 7f12affff640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f129406af90 0x7f129406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:16.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.981+0000 7f12affff640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f129406af90 0x7f129406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f12b00cac40 tx=0x7f129c008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:16.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:16.983+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12b0070090 con 0x7f12a00039a0 2023-10-16T15:56:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.178+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f12b00cdb50 con 0x7f129406af90 2023-10-16T15:56:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.179+0000 7f12ad7fa640 1 -- 172.21.15.29:0/3809991208 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f129c0095c0 con 0x7f129406af90 2023-10-16T15:56:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 0 to deep-scrub 2023-10-16T15:56:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f129406af90 msgr2=0x7f129406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f129406af90 0x7f129406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f12b00cac40 tx=0x7f129c008040 comp rx=0 tx=0).stop 2023-10-16T15:56:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 msgr2=0x7f12b006f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12b006f970 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f12a402c390 tx=0x7f12a4001930 comp rx=0 tx=0).stop 2023-10-16T15:56:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 shutdown_connections 2023-10-16T15:56:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f129406af90 0x7f129406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f12b00b2890 0x7f12b00c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f12b00b2410 0x7f12b014f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 --2- 172.21.15.29:0/3809991208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12a00039a0 0x7f12b006f970 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 >> 172.21.15.29:0/3809991208 conn(0x7f12b00ce6a0 msgr2=0x7f12b00c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 shutdown_connections 2023-10-16T15:56:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.182+0000 7f12b58ca640 1 -- 172.21.15.29:0/3809991208 wait complete. 2023-10-16T15:56:17.202 INFO:tasks.ceph:Scrubbing osd.1 2023-10-16T15:56:17.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:17.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.438+0000 7fd125e4f640 1 Processor -- start 2023-10-16T15:56:17.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.438+0000 7fd125e4f640 1 -- start start 2023-10-16T15:56:17.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.439+0000 7fd125e4f640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd1200cca70 0x7fd1200cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.439+0000 7fd125e4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.439+0000 7fd125e4f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd12006b630 0x7fd1200cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.439+0000 7fd125e4f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12006bbc0 con 0x7fd12006b630 2023-10-16T15:56:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.439+0000 7fd125e4f640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12006bea0 con 0x7fd1200cca70 2023-10-16T15:56:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.439+0000 7fd125e4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12006c180 con 0x7fd12006cc60 2023-10-16T15:56:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd12006b630 0x7fd1200cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd12006b630 0x7fd1200cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34344/0 (socket says 172.21.15.29:34344) 2023-10-16T15:56:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 -- 172.21.15.29:0/3361572353 learned_addr learned my addr 172.21.15.29:0/3361572353 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11effd640 1 --2- 172.21.15.29:0/3361572353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11f7fe640 1 --2- 172.21.15.29:0/3361572353 >> v2:172.21.15.170:3300/0 conn(0x7fd1200cca70 0x7fd1200cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 -- 172.21.15.29:0/3361572353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 msgr2=0x7fd1200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 --2- 172.21.15.29:0/3361572353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 -- 172.21.15.29:0/3361572353 >> v2:172.21.15.170:3300/0 conn(0x7fd1200cca70 msgr2=0x7fd1200cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 --2- 172.21.15.29:0/3361572353 >> v2:172.21.15.170:3300/0 conn(0x7fd1200cca70 0x7fd1200cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.440+0000 7fd11ffff640 1 -- 172.21.15.29:0/3361572353 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12006c460 con 0x7fd12006b630 2023-10-16T15:56:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.442+0000 7fd11ffff640 1 --2- 172.21.15.29:0/3361572353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd12006b630 0x7fd1200cf830 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fd1200b3670 tx=0x7fd11802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fe97c09140a869 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.442+0000 7fd11e7fc640 1 -- 172.21.15.29:0/3361572353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118031870 con 0x7fd12006b630 2023-10-16T15:56:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.442+0000 7fd11e7fc640 1 -- 172.21.15.29:0/3361572353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1180319d0 con 0x7fd12006b630 2023-10-16T15:56:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.442+0000 7fd11e7fc640 1 -- 172.21.15.29:0/3361572353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118031d60 con 0x7fd12006b630 2023-10-16T15:56:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.442+0000 7fd125e4f640 1 -- 172.21.15.29:0/3361572353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd12006b630 msgr2=0x7fd1200cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 --2- 172.21.15.29:0/3361572353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd12006b630 0x7fd1200cf830 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fd1200b3670 tx=0x7fd11802b910 comp rx=0 tx=0).stop 2023-10-16T15:56:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 -- 172.21.15.29:0/3361572353 shutdown_connections 2023-10-16T15:56:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 --2- 172.21.15.29:0/3361572353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd12006b630 0x7fd1200cf830 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 --2- 172.21.15.29:0/3361572353 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 --2- 172.21.15.29:0/3361572353 >> v2:172.21.15.170:3300/0 conn(0x7fd1200cca70 0x7fd1200cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 -- 172.21.15.29:0/3361572353 >> 172.21.15.29:0/3361572353 conn(0x7fd120099e30 msgr2=0x7fd1200d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 -- 172.21.15.29:0/3361572353 shutdown_connections 2023-10-16T15:56:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.443+0000 7fd125e4f640 1 -- 172.21.15.29:0/3361572353 wait complete. 2023-10-16T15:56:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.445+0000 7fd125e4f640 1 Processor -- start 2023-10-16T15:56:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.445+0000 7fd125e4f640 1 -- start start 2023-10-16T15:56:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd125e4f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd12006b630 0x7fd120150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd125e4f640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd125e4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1200cca70 0x7fd1201f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd125e4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201f3340 con 0x7fd1200cca70 2023-10-16T15:56:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd125e4f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200b61f0 con 0x7fd12006b630 2023-10-16T15:56:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd125e4f640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201f36f0 con 0x7fd12006cc60 2023-10-16T15:56:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1200cca70 0x7fd1201f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1200cca70 0x7fd1201f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34358/0 (socket says 172.21.15.29:34358) 2023-10-16T15:56:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 -- 172.21.15.29:0/1248211374 learned_addr learned my addr 172.21.15.29:0/1248211374 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11effd640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 -- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 msgr2=0x7fd1200b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd12006cc60 0x7fd1200b5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 -- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd12006b630 msgr2=0x7fd120150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11f7fe640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd12006b630 0x7fd120150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.446+0000 7fd11ffff640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd12006b630 0x7fd120150050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd11ffff640 1 -- 172.21.15.29:0/1248211374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1180094d0 con 0x7fd1200cca70 2023-10-16T15:56:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd11ffff640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1200cca70 0x7fd1201f2e00 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd1180090b0 tx=0x7fd1180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11803adf0 con 0x7fd1200cca70 2023-10-16T15:56:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1180443f0 con 0x7fd1200cca70 2023-10-16T15:56:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1180447a0 con 0x7fd1200cca70 2023-10-16T15:56:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1200b5f10 con 0x7fd1200cca70 2023-10-16T15:56:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.447+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1201f3c70 con 0x7fd1200cca70 2023-10-16T15:56:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.448+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1200cce50 con 0x7fd1200cca70 2023-10-16T15:56:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.450+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd118044900 con 0x7fd1200cca70 2023-10-16T15:56:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd107fff640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd10c06afe0 0x7fd10c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fd11802d040 con 0x7fd1200cca70 2023-10-16T15:56:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd107fff640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7fd10c071570 0x7fd10c073990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 --> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd10c074060 con 0x7fd10c071570 2023-10-16T15:56:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1180accd0 con 0x7fd1200cca70 2023-10-16T15:56:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd11effd640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7fd10c071570 0x7fd10c073990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.451+0000 7fd11f7fe640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd10c06afe0 0x7fd10c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.452+0000 7fd11effd640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7fd10c071570 0x7fd10c073990 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.453+0000 7fd11f7fe640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd10c06afe0 0x7fd10c06d4a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd110008ef0 tx=0x7fd110008440 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.453+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== osd.1 v2:172.21.15.29:6800/3296543673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd10c074060 con 0x7fd10c071570 2023-10-16T15:56:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.467+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 --> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd120160010 con 0x7fd10c071570 2023-10-16T15:56:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.467+0000 7fd107fff640 1 -- 172.21.15.29:0/1248211374 <== osd.1 v2:172.21.15.29:6800/3296543673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd120160010 con 0x7fd10c071570 2023-10-16T15:56:17.471 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:17.472 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7fd10c071570 msgr2=0x7fd10c073990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:6800/3296543673,v1:172.21.15.29:6801/3296543673] conn(0x7fd10c071570 0x7fd10c073990 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd10c06afe0 msgr2=0x7fd10c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd10c06afe0 0x7fd10c06d4a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd110008ef0 tx=0x7fd110008440 comp rx=0 tx=0).stop 2023-10-16T15:56:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1200cca70 msgr2=0x7fd1201f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 --2- 172.21.15.29:0/1248211374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1200cca70 0x7fd1201f2e00 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd1180090b0 tx=0x7fd1180046c0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd11ffff640 1 -- 172.21.15.29:0/1248211374 reap_dead start 2023-10-16T15:56:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 shutdown_connections 2023-10-16T15:56:17.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 >> 172.21.15.29:0/1248211374 conn(0x7fd120099e30 msgr2=0x7fd12006d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 shutdown_connections 2023-10-16T15:56:17.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.468+0000 7fd125e4f640 1 -- 172.21.15.29:0/1248211374 wait complete. 2023-10-16T15:56:17.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-16T15:56:17.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.712+0000 7fb27e4d7640 1 Processor -- start 2023-10-16T15:56:17.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.712+0000 7fb27e4d7640 1 -- start start 2023-10-16T15:56:17.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.713+0000 7fb27e4d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb27815c9f0 0x7fb2780c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.713+0000 7fb27e4d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb278160a90 con 0x7fb2780c9230 2023-10-16T15:56:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.713+0000 7fb27e4d7640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2780cd900 con 0x7fb27815dde0 2023-10-16T15:56:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.713+0000 7fb27e4d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2780cdec0 con 0x7fb27815c9f0 2023-10-16T15:56:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27d4d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb27815c9f0 0x7fb2780c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27ccd4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2780c9230 0x7fb2780709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52224/0 (socket says 172.21.15.29:52224) 2023-10-16T15:56:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27d4d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb27815c9f0 0x7fb2780c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34360/0 (socket says 172.21.15.29:34360) 2023-10-16T15:56:17.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27ccd4640 1 -- 172.21.15.29:0/866466997 learned_addr learned my addr 172.21.15.29:0/866466997 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:17.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27d4d5640 1 -- 172.21.15.29:0/866466997 >> v1:172.21.15.170:6789/0 conn(0x7fb27815dde0 legacy=0x7fb2780ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27d4d5640 1 -- 172.21.15.29:0/866466997 >> v1:172.21.15.29:6789/0 conn(0x7fb2780c9230 legacy=0x7fb2780709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.714+0000 7fb27d4d5640 1 -- 172.21.15.29:0/866466997 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2780cdbe0 con 0x7fb27815c9f0 2023-10-16T15:56:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.715+0000 7fb27d4d5640 1 --2- 172.21.15.29:0/866466997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815c9f0 0x7fb2780c8cf0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fb2740097a0 tx=0x7fb27402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2268d38396c865b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.716+0000 7fb26ffff640 1 -- 172.21.15.29:0/866466997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274031830 con 0x7fb27815c9f0 2023-10-16T15:56:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.716+0000 7fb26ffff640 1 -- 172.21.15.29:0/866466997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb274031990 con 0x7fb27815c9f0 2023-10-16T15:56:17.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.716+0000 7fb26ffff640 1 -- 172.21.15.29:0/866466997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274031d20 con 0x7fb27815c9f0 2023-10-16T15:56:17.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.716+0000 7fb27e4d7640 1 -- 172.21.15.29:0/866466997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815c9f0 msgr2=0x7fb2780c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.716+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/866466997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815c9f0 0x7fb2780c8cf0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fb2740097a0 tx=0x7fb27402b8d0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.717+0000 7fb27e4d7640 1 -- 172.21.15.29:0/866466997 shutdown_connections 2023-10-16T15:56:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.717+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/866466997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815c9f0 0x7fb2780c8cf0 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.717+0000 7fb27e4d7640 1 -- 172.21.15.29:0/866466997 >> 172.21.15.29:0/866466997 conn(0x7fb2780ce6a0 msgr2=0x7fb2780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.717+0000 7fb27e4d7640 1 -- 172.21.15.29:0/866466997 shutdown_connections 2023-10-16T15:56:17.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.717+0000 7fb27e4d7640 1 -- 172.21.15.29:0/866466997 wait complete. 2023-10-16T15:56:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.719+0000 7fb27e4d7640 1 Processor -- start 2023-10-16T15:56:17.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.719+0000 7fb27e4d7640 1 -- start start 2023-10-16T15:56:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27e4d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2780c9230 0x7fb2780c0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27e4d7640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb27815c9f0 0x7fb2780c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27e4d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 0x7fb278145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27e4d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb278159a50 con 0x7fb27815dde0 2023-10-16T15:56:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27e4d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb278158e40 con 0x7fb2780c9230 2023-10-16T15:56:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27e4d7640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb278159120 con 0x7fb27815c9f0 2023-10-16T15:56:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27dcd6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 0x7fb278145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27dcd6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 0x7fb278145680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34370/0 (socket says 172.21.15.29:34370) 2023-10-16T15:56:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27dcd6640 1 -- 172.21.15.29:0/4141072708 learned_addr learned my addr 172.21.15.29:0/4141072708 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27ccd4640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb27815c9f0 0x7fb2780c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.720+0000 7fb27d4d5640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2780c9230 0x7fb2780c0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27dcd6640 1 -- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb27815c9f0 msgr2=0x7fb2780c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27dcd6640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb27815c9f0 0x7fb2780c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27dcd6640 1 -- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2780c9230 msgr2=0x7fb2780c0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27dcd6640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2780c9230 0x7fb2780c0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27dcd6640 1 -- 172.21.15.29:0/4141072708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274009400 con 0x7fb27815dde0 2023-10-16T15:56:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27ccd4640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb27815c9f0 0x7fb2780c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27d4d5640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2780c9230 0x7fb2780c0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:17.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27dcd6640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 0x7fb278145680 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7fb27000be60 tx=0x7fb27000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270014440 con 0x7fb27815dde0 2023-10-16T15:56:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2700145a0 con 0x7fb27815dde0 2023-10-16T15:56:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270014990 con 0x7fb27815dde0 2023-10-16T15:56:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb278159400 con 0x7fb27815dde0 2023-10-16T15:56:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.721+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb278158ad0 con 0x7fb27815dde0 2023-10-16T15:56:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.722+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2780acfa0 con 0x7fb27815dde0 2023-10-16T15:56:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.724+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb27001d3b0 con 0x7fb27815dde0 2023-10-16T15:56:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.724+0000 7fb26dffb640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb25c06afc0 0x7fb25c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:17.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.725+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fb270089a20 con 0x7fb27815dde0 2023-10-16T15:56:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.725+0000 7fb27d4d5640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb25c06afc0 0x7fb25c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.726+0000 7fb27d4d5640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb25c06afc0 0x7fb25c06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb274036000 tx=0x7fb274008030 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.729+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb2700548a0 con 0x7fb27815dde0 2023-10-16T15:56:17.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.923+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2780c73e0 con 0x7fb25c06afc0 2023-10-16T15:56:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.924+0000 7fb26dffb640 1 -- 172.21.15.29:0/4141072708 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb2780c73e0 con 0x7fb25c06afc0 2023-10-16T15:56:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 1 to deep-scrub 2023-10-16T15:56:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb25c06afc0 msgr2=0x7fb25c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb25c06afc0 0x7fb25c06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb274036000 tx=0x7fb274008030 comp rx=0 tx=0).stop 2023-10-16T15:56:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 msgr2=0x7fb278145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 0x7fb278145680 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7fb27000be60 tx=0x7fb27000bf50 comp rx=0 tx=0).stop 2023-10-16T15:56:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 shutdown_connections 2023-10-16T15:56:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb25c06afc0 0x7fb25c06d480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb27815dde0 0x7fb278145680 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb27815c9f0 0x7fb2780c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 --2- 172.21.15.29:0/4141072708 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb2780c9230 0x7fb2780c0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 >> 172.21.15.29:0/4141072708 conn(0x7fb2780ce6a0 msgr2=0x7fb2780ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 shutdown_connections 2023-10-16T15:56:17.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:17.927+0000 7fb27e4d7640 1 -- 172.21.15.29:0/4141072708 wait complete. 2023-10-16T15:56:17.947 INFO:tasks.ceph:Scrubbing osd.2 2023-10-16T15:56:17.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:18.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.173+0000 7fac6013e640 1 Processor -- start 2023-10-16T15:56:18.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.173+0000 7fac6013e640 1 -- start start 2023-10-16T15:56:18.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.174+0000 7fac6013e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806cc60 0x7fac580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.174+0000 7fac6013e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac5806b630 0x7fac580cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.174+0000 7fac6013e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5806bbc0 con 0x7fac580cca70 2023-10-16T15:56:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.174+0000 7fac6013e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac580ce480 con 0x7fac5806b630 2023-10-16T15:56:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.174+0000 7fac6013e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5806bea0 con 0x7fac5806cc60 2023-10-16T15:56:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5deda640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fac580cca70 0x7fac580cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52230/0 (socket says 172.21.15.29:52230) 2023-10-16T15:56:18.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5e6db640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac5806b630 0x7fac580cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5deda640 1 -- 172.21.15.29:0/4084711457 learned_addr learned my addr 172.21.15.29:0/4084711457 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5d6d9640 1 --2- 172.21.15.29:0/4084711457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806cc60 0x7fac580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5e6db640 1 -- 172.21.15.29:0/4084711457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806cc60 msgr2=0x7fac580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5e6db640 1 --2- 172.21.15.29:0/4084711457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806cc60 0x7fac580cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5e6db640 1 -- 172.21.15.29:0/4084711457 >> v1:172.21.15.29:6789/0 conn(0x7fac580cca70 legacy=0x7fac580cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5e6db640 1 -- 172.21.15.29:0/4084711457 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5806c180 con 0x7fac5806b630 2023-10-16T15:56:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.175+0000 7fac5ced8640 1 -- 172.21.15.29:0/4084711457 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449519664 0 0) 0x7fac5806bbc0 con 0x7fac580cca70 2023-10-16T15:56:18.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac5e6db640 1 --2- 172.21.15.29:0/4084711457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5806b630 0x7fac580cf830 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fac50009d90 tx=0x7fac5002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed92d9b251dd0aed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac5ced8640 1 -- 172.21.15.29:0/4084711457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5002cc50 con 0x7fac5806b630 2023-10-16T15:56:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac5ced8640 1 -- 172.21.15.29:0/4084711457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5002cdb0 con 0x7fac5806b630 2023-10-16T15:56:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac5ced8640 1 -- 172.21.15.29:0/4084711457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50034660 con 0x7fac5806b630 2023-10-16T15:56:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac6013e640 1 -- 172.21.15.29:0/4084711457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5806b630 msgr2=0x7fac580cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac6013e640 1 --2- 172.21.15.29:0/4084711457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5806b630 0x7fac580cf830 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fac50009d90 tx=0x7fac5002bea0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac6013e640 1 -- 172.21.15.29:0/4084711457 shutdown_connections 2023-10-16T15:56:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac6013e640 1 --2- 172.21.15.29:0/4084711457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac5806b630 0x7fac580cf830 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac6013e640 1 --2- 172.21.15.29:0/4084711457 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806cc60 0x7fac580cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.177+0000 7fac6013e640 1 -- 172.21.15.29:0/4084711457 >> 172.21.15.29:0/4084711457 conn(0x7fac58099e30 msgr2=0x7fac580d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.178+0000 7fac6013e640 1 -- 172.21.15.29:0/4084711457 shutdown_connections 2023-10-16T15:56:18.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.178+0000 7fac6013e640 1 -- 172.21.15.29:0/4084711457 wait complete. 2023-10-16T15:56:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.179+0000 7fac6013e640 1 Processor -- start 2023-10-16T15:56:18.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.179+0000 7fac6013e640 1 -- start start 2023-10-16T15:56:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.180+0000 7fac6013e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806b630 0x7fac580af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.180+0000 7fac6013e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac5806cc60 0x7fac580c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.180+0000 7fac6013e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580cca70 0x7fac5815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.180+0000 7fac6013e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5815c840 con 0x7fac580cca70 2023-10-16T15:56:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.180+0000 7fac6013e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580afb90 con 0x7fac5806cc60 2023-10-16T15:56:18.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.180+0000 7fac6013e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580c8ba0 con 0x7fac5806b630 2023-10-16T15:56:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580cca70 0x7fac5815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580cca70 0x7fac5815be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34390/0 (socket says 172.21.15.29:34390) 2023-10-16T15:56:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 -- 172.21.15.29:0/946305209 learned_addr learned my addr 172.21.15.29:0/946305209 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5deda640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806b630 0x7fac580af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5d6d9640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac5806cc60 0x7fac580c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 -- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806b630 msgr2=0x7fac580af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5806b630 0x7fac580af540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 -- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac5806cc60 msgr2=0x7fac580c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac5806cc60 0x7fac580c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 -- 172.21.15.29:0/946305209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac50009a70 con 0x7fac580cca70 2023-10-16T15:56:18.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5d6d9640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac5806cc60 0x7fac580c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.181+0000 7fac5e6db640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580cca70 0x7fac5815be50 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7fac50009d90 tx=0x7fac50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.182+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50003b90 con 0x7fac580cca70 2023-10-16T15:56:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.182+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac50003cf0 con 0x7fac580cca70 2023-10-16T15:56:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.182+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50004270 con 0x7fac580cca70 2023-10-16T15:56:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.182+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac580c8e80 con 0x7fac580cca70 2023-10-16T15:56:18.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.182+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac580c9bc0 con 0x7fac580cca70 2023-10-16T15:56:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.183+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac5806c740 con 0x7fac580cca70 2023-10-16T15:56:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.184+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac5003b050 con 0x7fac580cca70 2023-10-16T15:56:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.185+0000 7fac567fc640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac4406af90 0x7fac4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.185+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fac500abdc0 con 0x7fac580cca70 2023-10-16T15:56:18.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.185+0000 7fac567fc640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7fac44071520 0x7fac44073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.185+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 --> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac44074010 con 0x7fac44071520 2023-10-16T15:56:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.185+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac500ae3a0 con 0x7fac580cca70 2023-10-16T15:56:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.186+0000 7fac5d6d9640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7fac44071520 0x7fac44073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.186+0000 7fac5deda640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac4406af90 0x7fac4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.186+0000 7fac5d6d9640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7fac44071520 0x7fac44073940 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.187+0000 7fac5deda640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac4406af90 0x7fac4406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fac48000f60 tx=0x7fac48009040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.187+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== osd.2 v2:172.21.15.29:6809/4164234701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fac44074010 con 0x7fac44071520 2023-10-16T15:56:18.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.202+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 --> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fac5806c460 con 0x7fac44071520 2023-10-16T15:56:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.202+0000 7fac567fc640 1 -- 172.21.15.29:0/946305209 <== osd.2 v2:172.21.15.29:6809/4164234701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fac4c009550 con 0x7fac44071520 2023-10-16T15:56:18.203 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:18.203 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7fac44071520 msgr2=0x7fac44073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:6809/4164234701,v1:172.21.15.29:6811/4164234701] conn(0x7fac44071520 0x7fac44073940 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac4406af90 msgr2=0x7fac4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac4406af90 0x7fac4406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fac48000f60 tx=0x7fac48009040 comp rx=0 tx=0).stop 2023-10-16T15:56:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580cca70 msgr2=0x7fac5815be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 --2- 172.21.15.29:0/946305209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580cca70 0x7fac5815be50 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7fac50009d90 tx=0x7fac50004a40 comp rx=0 tx=0).stop 2023-10-16T15:56:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac5e6db640 1 -- 172.21.15.29:0/946305209 reap_dead start 2023-10-16T15:56:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 shutdown_connections 2023-10-16T15:56:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 >> 172.21.15.29:0/946305209 conn(0x7fac58099e30 msgr2=0x7fac5806e960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 shutdown_connections 2023-10-16T15:56:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.203+0000 7fac6013e640 1 -- 172.21.15.29:0/946305209 wait complete. 2023-10-16T15:56:18.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-16T15:56:18.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.415+0000 7fc658525640 1 Processor -- start 2023-10-16T15:56:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.415+0000 7fc658525640 1 -- start start 2023-10-16T15:56:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc658525640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc650159ab0 0x7fc6500ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc658525640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc658525640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc650070360 con 0x7fc650159e90 2023-10-16T15:56:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc658525640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65006cb10 con 0x7fc650159ab0 2023-10-16T15:56:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc658525640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65015d890 con 0x7fc65006c140 2023-10-16T15:56:18.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc6562c1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc650159ab0 0x7fc6500ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc655ac0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc650159e90 0x7fc65006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52244/0 (socket says 172.21.15.29:52244) 2023-10-16T15:56:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.416+0000 7fc655ac0640 1 -- 172.21.15.29:0/1196366736 learned_addr learned my addr 172.21.15.29:0/1196366736 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.417+0000 7fc656ac2640 1 --2- 172.21.15.29:0/1196366736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.417+0000 7fc6562c1640 1 -- 172.21.15.29:0/1196366736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 msgr2=0x7fc65006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.417+0000 7fc6562c1640 1 --2- 172.21.15.29:0/1196366736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.417+0000 7fc6562c1640 1 -- 172.21.15.29:0/1196366736 >> v1:172.21.15.29:6789/0 conn(0x7fc650159e90 legacy=0x7fc65006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.417+0000 7fc6562c1640 1 -- 172.21.15.29:0/1196366736 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500cd1f0 con 0x7fc650159ab0 2023-10-16T15:56:18.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.417+0000 7fc6552bf640 1 -- 172.21.15.29:0/1196366736 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944518108 0 0) 0x7fc650070360 con 0x7fc650159e90 2023-10-16T15:56:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.418+0000 7fc6562c1640 1 --2- 172.21.15.29:0/1196366736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159ab0 0x7fc6500ce370 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0097a0 tx=0x7fc64c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb9c2431c1e8762 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc6552bf640 1 -- 172.21.15.29:0/1196366736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0316f0 con 0x7fc650159ab0 2023-10-16T15:56:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc6552bf640 1 -- 172.21.15.29:0/1196366736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c031850 con 0x7fc650159ab0 2023-10-16T15:56:18.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc6552bf640 1 -- 172.21.15.29:0/1196366736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c031be0 con 0x7fc650159ab0 2023-10-16T15:56:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 -- 172.21.15.29:0/1196366736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159ab0 msgr2=0x7fc6500ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 --2- 172.21.15.29:0/1196366736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159ab0 0x7fc6500ce370 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0097a0 tx=0x7fc64c02b790 comp rx=0 tx=0).stop 2023-10-16T15:56:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 -- 172.21.15.29:0/1196366736 shutdown_connections 2023-10-16T15:56:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 --2- 172.21.15.29:0/1196366736 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 --2- 172.21.15.29:0/1196366736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159ab0 0x7fc6500ce370 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 -- 172.21.15.29:0/1196366736 >> 172.21.15.29:0/1196366736 conn(0x7fc6500ce9b0 msgr2=0x7fc6500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 -- 172.21.15.29:0/1196366736 shutdown_connections 2023-10-16T15:56:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.419+0000 7fc658525640 1 -- 172.21.15.29:0/1196366736 wait complete. 2023-10-16T15:56:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.421+0000 7fc658525640 1 Processor -- start 2023-10-16T15:56:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.421+0000 7fc658525640 1 -- start start 2023-10-16T15:56:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.422+0000 7fc658525640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65015acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.422+0000 7fc658525640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc650159ab0 0x7fc650070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.422+0000 7fc658525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 0x7fc65014efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.422+0000 7fc658525640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65014ad00 con 0x7fc650159e90 2023-10-16T15:56:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.422+0000 7fc658525640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65015b600 con 0x7fc650159ab0 2023-10-16T15:56:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.422+0000 7fc658525640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65015b320 con 0x7fc65006c140 2023-10-16T15:56:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 0x7fc65014efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 0x7fc65014efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34408/0 (socket says 172.21.15.29:34408) 2023-10-16T15:56:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 -- 172.21.15.29:0/1165786420 learned_addr learned my addr 172.21.15.29:0/1165786420 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc6562c1640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65015acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc655ac0640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc650159ab0 0x7fc650070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 -- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 msgr2=0x7fc65015acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65015acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 -- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc650159ab0 msgr2=0x7fc650070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc650159ab0 0x7fc650070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.423+0000 7fc656ac2640 1 -- 172.21.15.29:0/1165786420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c009400 con 0x7fc650159e90 2023-10-16T15:56:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.424+0000 7fc656ac2640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 0x7fc65014efa0 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7fc648007e60 tx=0x7fc64800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.424+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648014420 con 0x7fc650159e90 2023-10-16T15:56:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.424+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6480145a0 con 0x7fc650159e90 2023-10-16T15:56:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.424+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648014950 con 0x7fc650159e90 2023-10-16T15:56:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.424+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65014e4c0 con 0x7fc650159e90 2023-10-16T15:56:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.424+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65014ea80 con 0x7fc650159e90 2023-10-16T15:56:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.425+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65004e160 con 0x7fc650159e90 2023-10-16T15:56:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.427+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc648014ab0 con 0x7fc650159e90 2023-10-16T15:56:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.427+0000 7fc646ffd640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc63406af90 0x7fc63406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.428+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fc64808a090 con 0x7fc650159e90 2023-10-16T15:56:18.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.428+0000 7fc6562c1640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc63406af90 0x7fc63406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.429+0000 7fc6562c1640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc63406af90 0x7fc63406d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc64c036000 tx=0x7fc64c008030 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.432+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc648054fb0 con 0x7fc650159e90 2023-10-16T15:56:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.627+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6500cd7b0 con 0x7fc63406af90 2023-10-16T15:56:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.628+0000 7fc646ffd640 1 -- 172.21.15.29:0/1165786420 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc64c042070 con 0x7fc63406af90 2023-10-16T15:56:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 2 to deep-scrub 2023-10-16T15:56:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc63406af90 msgr2=0x7fc63406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc63406af90 0x7fc63406d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc64c036000 tx=0x7fc64c008030 comp rx=0 tx=0).stop 2023-10-16T15:56:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 msgr2=0x7fc65014efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 0x7fc65014efa0 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7fc648007e60 tx=0x7fc64800e4b0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 shutdown_connections 2023-10-16T15:56:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc63406af90 0x7fc63406d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc650159e90 0x7fc65014efa0 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc650159ab0 0x7fc650070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 --2- 172.21.15.29:0/1165786420 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc65006c140 0x7fc65015acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 >> 172.21.15.29:0/1165786420 conn(0x7fc6500ce9b0 msgr2=0x7fc650158fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 shutdown_connections 2023-10-16T15:56:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.631+0000 7fc658525640 1 -- 172.21.15.29:0/1165786420 wait complete. 2023-10-16T15:56:18.649 INFO:tasks.ceph:Scrubbing osd.3 2023-10-16T15:56:18.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:18.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:18.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.852+0000 7fb974e61640 1 Processor -- start 2023-10-16T15:56:18.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.853+0000 7fb974e61640 1 -- start start 2023-10-16T15:56:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.854+0000 7fb974e61640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb97015cf60 0x7fb97006f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.854+0000 7fb974e61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9700c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.854+0000 7fb974e61640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700c9620 con 0x7fb9700c9ff0 2023-10-16T15:56:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.854+0000 7fb974e61640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97015d340 con 0x7fb97015cf60 2023-10-16T15:56:18.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.854+0000 7fb974e61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9700cc860 con 0x7fb97006f9d0 2023-10-16T15:56:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.855+0000 7fb96e59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb97015cf60 0x7fb97006f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.855+0000 7fb967fff640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9700c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.855+0000 7fb96ed9d640 1 --1- >> v1:172.21.15.170:6789/0 conn(0x7fb9700c9ff0 0x7fb9700b0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.170:6789/0 says I am v1:172.21.15.29:45762/0 (socket says 172.21.15.29:45762) 2023-10-16T15:56:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.855+0000 7fb96e59c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb97015cf60 0x7fb97006f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.29:35046/0 (socket says 172.21.15.29:35046) 2023-10-16T15:56:18.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.855+0000 7fb96ed9d640 1 -- 172.21.15.29:0/1032065250 learned_addr learned my addr 172.21.15.29:0/1032065250 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.856+0000 7fb96e59c640 1 -- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 msgr2=0x7fb9700c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.856+0000 7fb96e59c640 1 --2- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9700c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.856+0000 7fb96e59c640 1 -- 172.21.15.29:0/1032065250 >> v1:172.21.15.170:6789/0 conn(0x7fb9700c9ff0 legacy=0x7fb9700b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.856+0000 7fb96e59c640 1 -- 172.21.15.29:0/1032065250 --> v2:172.21.15.170:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700ad4a0 con 0x7fb97015cf60 2023-10-16T15:56:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.857+0000 7fb96e59c640 1 --2- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb97015cf60 0x7fb97006f490 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb9600097a0 tx=0x7fb96002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6735f89874066b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.857+0000 7fb96dd9b640 1 -- 172.21.15.29:0/1032065250 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96002d040 con 0x7fb97015cf60 2023-10-16T15:56:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.857+0000 7fb96dd9b640 1 -- 172.21.15.29:0/1032065250 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb960031810 con 0x7fb97015cf60 2023-10-16T15:56:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.857+0000 7fb96dd9b640 1 -- 172.21.15.29:0/1032065250 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960031ba0 con 0x7fb97015cf60 2023-10-16T15:56:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.858+0000 7fb974e61640 1 -- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb97015cf60 msgr2=0x7fb97006f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.858+0000 7fb974e61640 1 --2- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb97015cf60 0x7fb97006f490 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb9600097a0 tx=0x7fb96002b8b0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.858+0000 7fb974e61640 1 -- 172.21.15.29:0/1032065250 shutdown_connections 2023-10-16T15:56:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.858+0000 7fb974e61640 1 --2- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9700c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.858+0000 7fb974e61640 1 --2- 172.21.15.29:0/1032065250 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb97015cf60 0x7fb97006f490 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.858+0000 7fb974e61640 1 -- 172.21.15.29:0/1032065250 >> 172.21.15.29:0/1032065250 conn(0x7fb9700cb360 msgr2=0x7fb9700cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.859+0000 7fb974e61640 1 -- 172.21.15.29:0/1032065250 shutdown_connections 2023-10-16T15:56:18.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.859+0000 7fb974e61640 1 -- 172.21.15.29:0/1032065250 wait complete. 2023-10-16T15:56:18.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.861+0000 7fb974e61640 1 Processor -- start 2023-10-16T15:56:18.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.861+0000 7fb974e61640 1 -- start start 2023-10-16T15:56:18.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb974e61640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9701f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb974e61640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9700c9ff0 0x7fb9700c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb974e61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97015cf60 0x7fb9700c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb974e61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9700c7220 con 0x7fb97015cf60 2023-10-16T15:56:18.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb974e61640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970155690 con 0x7fb9700c9ff0 2023-10-16T15:56:18.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb974e61640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb970155970 con 0x7fb97006f9d0 2023-10-16T15:56:18.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb96ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97015cf60 0x7fb9700c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb96ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97015cf60 0x7fb9700c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34412/0 (socket says 172.21.15.29:34412) 2023-10-16T15:56:18.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb96ed9d640 1 -- 172.21.15.29:0/2665750645 learned_addr learned my addr 172.21.15.29:0/2665750645 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:18.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb96e59c640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9701f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.862+0000 7fb967fff640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9700c9ff0 0x7fb9700c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb96ed9d640 1 -- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 msgr2=0x7fb9701f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb96ed9d640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb97006f9d0 0x7fb9701f2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb96ed9d640 1 -- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9700c9ff0 msgr2=0x7fb9700c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb96ed9d640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9700c9ff0 0x7fb9700c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb96ed9d640 1 -- 172.21.15.29:0/2665750645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb960009400 con 0x7fb97015cf60 2023-10-16T15:56:18.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb96ed9d640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97015cf60 0x7fb9700c6ce0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fb96800ca80 tx=0x7fb96800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96800ac30 con 0x7fb97015cf60 2023-10-16T15:56:18.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96800ad90 con 0x7fb97015cf60 2023-10-16T15:56:18.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968012600 con 0x7fb97015cf60 2023-10-16T15:56:18.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.863+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb970155c50 con 0x7fb97015cf60 2023-10-16T15:56:18.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.864+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb970156210 con 0x7fb97015cf60 2023-10-16T15:56:18.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.865+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9700c92b0 con 0x7fb97015cf60 2023-10-16T15:56:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.866+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb968007070 con 0x7fb97015cf60 2023-10-16T15:56:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.867+0000 7fb966ffd640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb95006af90 0x7fb95006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.867+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fb968089540 con 0x7fb97015cf60 2023-10-16T15:56:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.867+0000 7fb966ffd640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7fb950071520 0x7fb950073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.867+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 --> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb950074010 con 0x7fb950071520 2023-10-16T15:56:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.867+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb96808bc00 con 0x7fb97015cf60 2023-10-16T15:56:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.868+0000 7fb96e59c640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb95006af90 0x7fb95006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.868+0000 7fb967fff640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7fb950071520 0x7fb950073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.868+0000 7fb967fff640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7fb950071520 0x7fb950073940 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.869+0000 7fb96e59c640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb95006af90 0x7fb95006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb960036000 tx=0x7fb96003c040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:18.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.869+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== osd.3 v2:172.21.15.29:6819/1457722234 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb950074010 con 0x7fb950071520 2023-10-16T15:56:18.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.887+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 --> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb9700c8fd0 con 0x7fb950071520 2023-10-16T15:56:18.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.888+0000 7fb966ffd640 1 -- 172.21.15.29:0/2665750645 <== osd.3 v2:172.21.15.29:6819/1457722234 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb958009b60 con 0x7fb950071520 2023-10-16T15:56:18.889 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:18.889 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:18.890 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:18.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.888+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7fb950071520 msgr2=0x7fb950073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.888+0000 7fb974e61640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:6819/1457722234,v1:172.21.15.29:6820/1457722234] conn(0x7fb950071520 0x7fb950073940 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:18.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb95006af90 msgr2=0x7fb95006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb95006af90 0x7fb95006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb960036000 tx=0x7fb96003c040 comp rx=0 tx=0).stop 2023-10-16T15:56:18.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97015cf60 msgr2=0x7fb9700c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:18.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 --2- 172.21.15.29:0/2665750645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97015cf60 0x7fb9700c6ce0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fb96800ca80 tx=0x7fb96800cf40 comp rx=0 tx=0).stop 2023-10-16T15:56:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb96ed9d640 1 -- 172.21.15.29:0/2665750645 reap_dead start 2023-10-16T15:56:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 shutdown_connections 2023-10-16T15:56:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 >> 172.21.15.29:0/2665750645 conn(0x7fb9700cb360 msgr2=0x7fb9700ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 shutdown_connections 2023-10-16T15:56:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:18.889+0000 7fb974e61640 1 -- 172.21.15.29:0/2665750645 wait complete. 2023-10-16T15:56:18.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-16T15:56:19.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.154+0000 7f117dcbb640 1 Processor -- start 2023-10-16T15:56:19.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.154+0000 7f117dcbb640 1 -- start start 2023-10-16T15:56:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.155+0000 7f117dcbb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f117806c040 0x7f117806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117dcbb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11780cda80 0x7f117815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117dcbb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117806c960 con 0x7f11780ad970 2023-10-16T15:56:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117dcbb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117815b9d0 con 0x7f11780cda80 2023-10-16T15:56:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117dcbb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117815bcb0 con 0x7f117806c040 2023-10-16T15:56:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117d4ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11780cda80 0x7f117815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117d4ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f11780cda80 0x7f117815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34418/0 (socket says 172.21.15.29:34418) 2023-10-16T15:56:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117d4ba640 1 -- 172.21.15.29:0/2482001912 learned_addr learned my addr 172.21.15.29:0/2482001912 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117d4ba640 1 -- 172.21.15.29:0/2482001912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f117806c040 msgr2=0x7f117806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.156+0000 7f117ccb9640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f117806c040 0x7f117806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.157+0000 7f117d4ba640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f117806c040 0x7f117806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.157+0000 7f117d4ba640 1 -- 172.21.15.29:0/2482001912 >> v1:172.21.15.170:6789/0 conn(0x7f11780ad970 legacy=0x7f117815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.157+0000 7f117d4ba640 1 -- 172.21.15.29:0/2482001912 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117815bf90 con 0x7f11780cda80 2023-10-16T15:56:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.157+0000 7f117ccb9640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f117806c040 0x7f117806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.158+0000 7f117d4ba640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11780cda80 0x7f117815ed90 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f1170009e50 tx=0x7f117002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d9aa0dd2f2c7b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.158+0000 7f11777fe640 1 -- 172.21.15.29:0/2482001912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117002ca70 con 0x7f11780cda80 2023-10-16T15:56:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.158+0000 7f11777fe640 1 -- 172.21.15.29:0/2482001912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117002cbd0 con 0x7f11780cda80 2023-10-16T15:56:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.158+0000 7f11777fe640 1 -- 172.21.15.29:0/2482001912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700343f0 con 0x7f11780cda80 2023-10-16T15:56:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 -- 172.21.15.29:0/2482001912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11780cda80 msgr2=0x7f117815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11780cda80 0x7f117815ed90 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f1170009e50 tx=0x7f117002bed0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 -- 172.21.15.29:0/2482001912 shutdown_connections 2023-10-16T15:56:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11780cda80 0x7f117815ed90 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 --2- 172.21.15.29:0/2482001912 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f117806c040 0x7f117806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 -- 172.21.15.29:0/2482001912 >> 172.21.15.29:0/2482001912 conn(0x7f11780ce6a0 msgr2=0x7f11780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:19.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 -- 172.21.15.29:0/2482001912 shutdown_connections 2023-10-16T15:56:19.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.159+0000 7f117dcbb640 1 -- 172.21.15.29:0/2482001912 wait complete. 2023-10-16T15:56:19.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.161+0000 7f117dcbb640 1 Processor -- start 2023-10-16T15:56:19.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.161+0000 7f117dcbb640 1 -- start start 2023-10-16T15:56:19.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117dcbb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 0x7f117814f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117dcbb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11780ad970 0x7f11781518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117dcbb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11780cda80 0x7f117815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117dcbb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178155730 con 0x7f117806c040 2023-10-16T15:56:19.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117dcbb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178151590 con 0x7f11780cda80 2023-10-16T15:56:19.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117dcbb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1178150fd0 con 0x7f11780ad970 2023-10-16T15:56:19.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117ccb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 0x7f117814f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117ccb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 0x7f117814f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34426/0 (socket says 172.21.15.29:34426) 2023-10-16T15:56:19.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.162+0000 7f117ccb9640 1 -- 172.21.15.29:0/1872211842 learned_addr learned my addr 172.21.15.29:0/1872211842 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:19.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117d4ba640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11780cda80 0x7f117815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f1177fff640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11780ad970 0x7f11781518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117ccb9640 1 -- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11780ad970 msgr2=0x7f11781518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117ccb9640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11780ad970 0x7f11781518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117ccb9640 1 -- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11780cda80 msgr2=0x7f117815e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117ccb9640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11780cda80 0x7f117815e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117ccb9640 1 -- 172.21.15.29:0/1872211842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1170009b30 con 0x7f117806c040 2023-10-16T15:56:19.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117d4ba640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11780cda80 0x7f117815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f1177fff640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11780ad970 0x7f11781518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f117ccb9640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 0x7f117814f800 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c820 tx=0x7f116c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.163+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c007a30 con 0x7f117806c040 2023-10-16T15:56:19.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.164+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c007b90 con 0x7f117806c040 2023-10-16T15:56:19.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.164+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00ce10 con 0x7f117806c040 2023-10-16T15:56:19.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.164+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11781512b0 con 0x7f117806c040 2023-10-16T15:56:19.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.164+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117814fd40 con 0x7f117806c040 2023-10-16T15:56:19.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.165+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117806b330 con 0x7f117806c040 2023-10-16T15:56:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.167+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f116c0113f0 con 0x7f117806c040 2023-10-16T15:56:19.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.168+0000 7f11757fa640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f115c06af90 0x7f115c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.168+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f116c089b20 con 0x7f117806c040 2023-10-16T15:56:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.168+0000 7f1177fff640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f115c06af90 0x7f115c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.170+0000 7f1177fff640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f115c06af90 0x7f115c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1178159ae0 tx=0x7f1164008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.173+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f116c054890 con 0x7f117806c040 2023-10-16T15:56:19.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.365+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f11780cc550 con 0x7f115c06af90 2023-10-16T15:56:19.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.366+0000 7f11757fa640 1 -- 172.21.15.29:0/1872211842 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f116400a420 con 0x7f115c06af90 2023-10-16T15:56:19.367 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 3 to deep-scrub 2023-10-16T15:56:19.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f115c06af90 msgr2=0x7f115c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f115c06af90 0x7f115c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1178159ae0 tx=0x7f1164008040 comp rx=0 tx=0).stop 2023-10-16T15:56:19.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 msgr2=0x7f117814f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 0x7f117814f800 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c820 tx=0x7f116c00cce0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 shutdown_connections 2023-10-16T15:56:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f115c06af90 0x7f115c06d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f11780cda80 0x7f117815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f11780ad970 0x7f11781518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 --2- 172.21.15.29:0/1872211842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f117806c040 0x7f117814f800 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 >> 172.21.15.29:0/1872211842 conn(0x7f11780ce6a0 msgr2=0x7f117815d880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:19.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 shutdown_connections 2023-10-16T15:56:19.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.369+0000 7f117dcbb640 1 -- 172.21.15.29:0/1872211842 wait complete. 2023-10-16T15:56:19.387 INFO:tasks.ceph:Scrubbing osd.4 2023-10-16T15:56:19.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:19.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.597+0000 7fc046f6e640 1 Processor -- start 2023-10-16T15:56:19.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.597+0000 7fc046f6e640 1 -- start start 2023-10-16T15:56:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc046f6e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0400cca70 0x7fc0400cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc046f6e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc04006cc60 0x7fc0400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc046f6e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fc04006b630 0x7fc0400cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc046f6e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04006bbc0 con 0x7fc0400cca70 2023-10-16T15:56:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc046f6e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04006bea0 con 0x7fc04006b630 2023-10-16T15:56:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc046f6e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04006c180 con 0x7fc04006cc60 2023-10-16T15:56:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc044d0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0400cca70 0x7fc0400cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc044d0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0400cca70 0x7fc0400cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34438/0 (socket says 172.21.15.29:34438) 2023-10-16T15:56:19.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.599+0000 7fc044d0a640 1 -- 172.21.15.29:0/1863407663 learned_addr learned my addr 172.21.15.29:0/1863407663 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:19.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc04006cc60 0x7fc0400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc044d0a640 1 -- 172.21.15.29:0/1863407663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc04006cc60 msgr2=0x7fc0400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc04550b640 1 --2- 172.21.15.29:0/1863407663 >> v2:172.21.15.170:3300/0 conn(0x7fc04006b630 0x7fc0400cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc044d0a640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc04006cc60 0x7fc0400cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc044d0a640 1 -- 172.21.15.29:0/1863407663 >> v2:172.21.15.170:3300/0 conn(0x7fc04006b630 msgr2=0x7fc0400cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc044d0a640 1 --2- 172.21.15.29:0/1863407663 >> v2:172.21.15.170:3300/0 conn(0x7fc04006b630 0x7fc0400cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc044d0a640 1 -- 172.21.15.29:0/1863407663 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04006c460 con 0x7fc0400cca70 2023-10-16T15:56:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc04006cc60 0x7fc0400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.600+0000 7fc04550b640 1 --2- 172.21.15.29:0/1863407663 >> v2:172.21.15.170:3300/0 conn(0x7fc04006b630 0x7fc0400cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.601+0000 7fc044d0a640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0400cca70 0x7fc0400cdd70 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fc0400cb0a0 tx=0x7fc03402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5f30b1d5708cc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.601+0000 7fc03f7fe640 1 -- 172.21.15.29:0/1863407663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034031870 con 0x7fc0400cca70 2023-10-16T15:56:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.601+0000 7fc03f7fe640 1 -- 172.21.15.29:0/1863407663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0340319d0 con 0x7fc0400cca70 2023-10-16T15:56:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.601+0000 7fc03f7fe640 1 -- 172.21.15.29:0/1863407663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034031d60 con 0x7fc0400cca70 2023-10-16T15:56:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 -- 172.21.15.29:0/1863407663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0400cca70 msgr2=0x7fc0400cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0400cca70 0x7fc0400cdd70 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fc0400cb0a0 tx=0x7fc03402b910 comp rx=0 tx=0).stop 2023-10-16T15:56:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 -- 172.21.15.29:0/1863407663 shutdown_connections 2023-10-16T15:56:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 --2- 172.21.15.29:0/1863407663 >> v2:172.21.15.170:3300/0 conn(0x7fc04006b630 0x7fc0400cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc04006cc60 0x7fc0400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 --2- 172.21.15.29:0/1863407663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0400cca70 0x7fc0400cdd70 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 -- 172.21.15.29:0/1863407663 >> 172.21.15.29:0/1863407663 conn(0x7fc040099e30 msgr2=0x7fc0400d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 -- 172.21.15.29:0/1863407663 shutdown_connections 2023-10-16T15:56:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.602+0000 7fc046f6e640 1 -- 172.21.15.29:0/1863407663 wait complete. 2023-10-16T15:56:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.604+0000 7fc046f6e640 1 Processor -- start 2023-10-16T15:56:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.604+0000 7fc046f6e640 1 -- start start 2023-10-16T15:56:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc046f6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc04006b630 0x7fc0400b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc046f6e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc04006cc60 0x7fc0400c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc046f6e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc0400cca70 0x7fc0400c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc046f6e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0400aeb40 con 0x7fc04006b630 2023-10-16T15:56:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc046f6e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0400af5f0 con 0x7fc04006cc60 2023-10-16T15:56:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc046f6e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0401f5ae0 con 0x7fc0400cca70 2023-10-16T15:56:19.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc044d0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc04006b630 0x7fc0400b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc044d0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc04006b630 0x7fc0400b62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34450/0 (socket says 172.21.15.29:34450) 2023-10-16T15:56:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc044d0a640 1 -- 172.21.15.29:0/4141930748 learned_addr learned my addr 172.21.15.29:0/4141930748 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc044d0a640 1 -- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc0400cca70 msgr2=0x7fc0400c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc044d0a640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fc0400cca70 0x7fc0400c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.605+0000 7fc044d0a640 1 -- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc04006cc60 msgr2=0x7fc0400c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc044d0a640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc04006cc60 0x7fc0400c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc044d0a640 1 -- 172.21.15.29:0/4141930748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340094d0 con 0x7fc04006b630 2023-10-16T15:56:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc044d0a640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc04006b630 0x7fc0400b62b0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fc034000c00 tx=0x7fc0340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03403ad10 con 0x7fc04006b630 2023-10-16T15:56:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03403ae70 con 0x7fc04006b630 2023-10-16T15:56:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034044610 con 0x7fc04006b630 2023-10-16T15:56:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0400aed50 con 0x7fc04006b630 2023-10-16T15:56:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.606+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0400af310 con 0x7fc04006b630 2023-10-16T15:56:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.607+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0400cce50 con 0x7fc04006b630 2023-10-16T15:56:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.608+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc03404d450 con 0x7fc04006b630 2023-10-16T15:56:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.609+0000 7fc03d7fa640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc02406af90 0x7fc02406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.609+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fc03402d040 con 0x7fc04006b630 2023-10-16T15:56:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.610+0000 7fc03d7fa640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7fc024071520 0x7fc024073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.610+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 --> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc024074010 con 0x7fc024071520 2023-10-16T15:56:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.610+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0340ae460 con 0x7fc04006b630 2023-10-16T15:56:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.610+0000 7fc03ffff640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc02406af90 0x7fc02406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.610+0000 7fc04550b640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7fc024071520 0x7fc024073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.611+0000 7fc03ffff640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc02406af90 0x7fc02406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc02c006fd0 tx=0x7fc02c008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.612+0000 7fc04550b640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7fc024071520 0x7fc024073940 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.614+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== osd.4 v2:172.21.15.170:6808/1474743223 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc024074010 con 0x7fc024071520 2023-10-16T15:56:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.627+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 --> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc04006c740 con 0x7fc024071520 2023-10-16T15:56:19.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.628+0000 7fc03d7fa640 1 -- 172.21.15.29:0/4141930748 <== osd.4 v2:172.21.15.170:6808/1474743223 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc038007920 con 0x7fc024071520 2023-10-16T15:56:19.629 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:19.629 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:19.630 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.628+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7fc024071520 msgr2=0x7fc024073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.628+0000 7fc046f6e640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6808/1474743223,v1:172.21.15.170:6809/1474743223] conn(0x7fc024071520 0x7fc024073940 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc02406af90 msgr2=0x7fc02406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fc02406af90 0x7fc02406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc02c006fd0 tx=0x7fc02c008040 comp rx=0 tx=0).stop 2023-10-16T15:56:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc04006b630 msgr2=0x7fc0400b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 --2- 172.21.15.29:0/4141930748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc04006b630 0x7fc0400b62b0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fc034000c00 tx=0x7fc0340046c0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc04550b640 1 -- 172.21.15.29:0/4141930748 reap_dead start 2023-10-16T15:56:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 shutdown_connections 2023-10-16T15:56:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 >> 172.21.15.29:0/4141930748 conn(0x7fc040099e30 msgr2=0x7fc04015f140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 shutdown_connections 2023-10-16T15:56:19.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.629+0000 7fc046f6e640 1 -- 172.21.15.29:0/4141930748 wait complete. 2023-10-16T15:56:19.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-16T15:56:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.841+0000 7f81c1597640 1 Processor -- start 2023-10-16T15:56:19.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.841+0000 7f81c1597640 1 -- start start 2023-10-16T15:56:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.843+0000 7f81c1597640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 0x7f81bc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.843+0000 7f81c1597640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81bc0cb030 con 0x7f81bc0caac0 2023-10-16T15:56:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.843+0000 7f81c1597640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81bc0c91c0 con 0x7f81bc0c9ca0 2023-10-16T15:56:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.843+0000 7f81c1597640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0c94a0 con 0x7f81bc15ad70 2023-10-16T15:56:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.843+0000 7f81bb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f81bc0caac0 0x7f81bc0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52258/0 (socket says 172.21.15.29:52258) 2023-10-16T15:56:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.843+0000 7f81bb7fe640 1 -- 172.21.15.29:0/1708188386 learned_addr learned my addr 172.21.15.29:0/1708188386 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81baffd640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 0x7f81bc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102956805 0 0) 0x7f81bc0cb030 con 0x7f81bc0caac0 2023-10-16T15:56:19.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4003520 con 0x7f81bc0caac0 2023-10-16T15:56:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635062417 0 0) 0x7f81a4003520 con 0x7f81bc0caac0 2023-10-16T15:56:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 msgr2=0x7f81bc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 0x7f81bc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 >> v1:172.21.15.170:6789/0 conn(0x7f81bc0c9ca0 legacy=0x7f81bc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0c9780 con 0x7f81bc0caac0 2023-10-16T15:56:19.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f81b40019f0 con 0x7f81bc0caac0 2023-10-16T15:56:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 >> v1:172.21.15.29:6789/0 conn(0x7f81bc0caac0 legacy=0x7f81bc0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a40039a0 0x7f81a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40047c0 0x7f81a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.844+0000 7f81b9ffb640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 0x7f81a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0c94a0 con 0x7f81a40039a0 2023-10-16T15:56:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a400ba60 con 0x7f81a40047c0 2023-10-16T15:56:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a400bc20 con 0x7f81bc15ad70 2023-10-16T15:56:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81baffd640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a40039a0 0x7f81a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81ba7fc640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40047c0 0x7f81a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81bb7fe640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 0x7f81a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81baffd640 1 -- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 msgr2=0x7f81a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81baffd640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc15ad70 0x7f81a4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81baffd640 1 -- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40047c0 msgr2=0x7f81a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81baffd640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40047c0 0x7f81a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81baffd640 1 -- 172.21.15.29:0/1708188386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4003520 con 0x7f81a40039a0 2023-10-16T15:56:19.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.845+0000 7f81ba7fc640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40047c0 0x7f81a4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.846+0000 7f81baffd640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a40039a0 0x7f81a4003dc0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009a90 tx=0x7f81a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ba565dcc10606 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac02c680 con 0x7f81a40039a0 2023-10-16T15:56:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac02c7e0 con 0x7f81a40039a0 2023-10-16T15:56:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81b9ffb640 1 -- 172.21.15.29:0/1708188386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac02cb70 con 0x7f81a40039a0 2023-10-16T15:56:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81c1597640 1 -- 172.21.15.29:0/1708188386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a40039a0 msgr2=0x7f81a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81c1597640 1 --2- 172.21.15.29:0/1708188386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81a40039a0 0x7f81a4003dc0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009a90 tx=0x7f81a40057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81bb7fe640 1 -- 172.21.15.29:0/1708188386 reap_dead start 2023-10-16T15:56:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81c1597640 1 -- 172.21.15.29:0/1708188386 shutdown_connections 2023-10-16T15:56:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81c1597640 1 -- 172.21.15.29:0/1708188386 >> 172.21.15.29:0/1708188386 conn(0x7f81bc0ce9b0 msgr2=0x7f81bc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81c1597640 1 -- 172.21.15.29:0/1708188386 shutdown_connections 2023-10-16T15:56:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.847+0000 7f81c1597640 1 -- 172.21.15.29:0/1708188386 wait complete. 2023-10-16T15:56:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.849+0000 7f81c1597640 1 Processor -- start 2023-10-16T15:56:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.849+0000 7f81c1597640 1 -- start start 2023-10-16T15:56:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81c1597640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40039a0 0x7f81bc158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81c1597640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 0x7f81bc0c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81c1597640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc0b2cc0 0x7f81bc0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81c1597640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0b3680 con 0x7f81bc0b0b40 2023-10-16T15:56:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81c1597640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0c4770 con 0x7f81a40039a0 2023-10-16T15:56:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81c1597640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0c4a50 con 0x7f81bc0b2cc0 2023-10-16T15:56:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81ba7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 0x7f81bc0c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81ba7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 0x7f81bc0c3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34470/0 (socket says 172.21.15.29:34470) 2023-10-16T15:56:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.850+0000 7f81ba7fc640 1 -- 172.21.15.29:0/2560162871 learned_addr learned my addr 172.21.15.29:0/2560162871 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81baffd640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40039a0 0x7f81bc158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81ba7fc640 1 -- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc0b2cc0 msgr2=0x7f81bc0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81bb7fe640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc0b2cc0 0x7f81bc0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81ba7fc640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc0b2cc0 0x7f81bc0b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81ba7fc640 1 -- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40039a0 msgr2=0x7f81bc158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81ba7fc640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40039a0 0x7f81bc158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81ba7fc640 1 -- 172.21.15.29:0/2560162871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0c4d30 con 0x7f81bc0b0b40 2023-10-16T15:56:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81baffd640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40039a0 0x7f81bc158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81ba7fc640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 0x7f81bc0c3e00 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f81b000d7a0 tx=0x7f81b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.851+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0017070 con 0x7f81bc0b0b40 2023-10-16T15:56:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.852+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b0005070 con 0x7f81bc0b0b40 2023-10-16T15:56:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.852+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0005420 con 0x7f81bc0b0b40 2023-10-16T15:56:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.852+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc0c5010 con 0x7f81bc0b0b40 2023-10-16T15:56:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.852+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc0c4520 con 0x7f81bc0b0b40 2023-10-16T15:56:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.853+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc0c2550 con 0x7f81bc0b0b40 2023-10-16T15:56:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.854+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f81b0007030 con 0x7f81bc0b0b40 2023-10-16T15:56:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.855+0000 7f81a37fe640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f81a806afc0 0x7f81a806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.855+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f81b0089190 con 0x7f81bc0b0b40 2023-10-16T15:56:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.855+0000 7f81baffd640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f81a806afc0 0x7f81a806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.857+0000 7f81baffd640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f81a806afc0 0x7f81a806d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81ac036000 tx=0x7f81ac030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:19.859+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f81bc0c2550 con 0x7f81bc0b0b40 2023-10-16T15:56:20.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.056+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f81bc04e160 con 0x7f81a806afc0 2023-10-16T15:56:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.057+0000 7f81a37fe640 1 -- 172.21.15.29:0/2560162871 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f81bc04e160 con 0x7f81a806afc0 2023-10-16T15:56:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 4 to deep-scrub 2023-10-16T15:56:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.060+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f81a806afc0 msgr2=0x7f81a806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.060+0000 7f81c1597640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f81a806afc0 0x7f81a806d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81ac036000 tx=0x7f81ac030000 comp rx=0 tx=0).stop 2023-10-16T15:56:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.060+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 msgr2=0x7f81bc0c3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.060+0000 7f81c1597640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 0x7f81bc0c3e00 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f81b000d7a0 tx=0x7f81b000dc60 comp rx=0 tx=0).stop 2023-10-16T15:56:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 shutdown_connections 2023-10-16T15:56:20.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f81a806afc0 0x7f81a806d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81bc0b0b40 0x7f81bc0c3e00 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f81bc0b2cc0 0x7f81bc0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 --2- 172.21.15.29:0/2560162871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f81a40039a0 0x7f81bc158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 >> 172.21.15.29:0/2560162871 conn(0x7f81bc0ce9b0 msgr2=0x7f81bc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 shutdown_connections 2023-10-16T15:56:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.061+0000 7f81c1597640 1 -- 172.21.15.29:0/2560162871 wait complete. 2023-10-16T15:56:20.080 INFO:tasks.ceph:Scrubbing osd.5 2023-10-16T15:56:20.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.307+0000 7fac5759e640 1 Processor -- start 2023-10-16T15:56:20.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.307+0000 7fac5759e640 1 -- start start 2023-10-16T15:56:20.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.308+0000 7fac5759e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fac5815cd70 0x7fac580d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.308+0000 7fac5759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac580b8480 0x7fac580cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac5759e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5815d150 con 0x7fac580d0be0 2023-10-16T15:56:20.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac5759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac580d1180 con 0x7fac580b8480 2023-10-16T15:56:20.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac5759e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac580d1460 con 0x7fac5815cd70 2023-10-16T15:56:20.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac580b8480 0x7fac580cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac580b8480 0x7fac580cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34484/0 (socket says 172.21.15.29:34484) 2023-10-16T15:56:20.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 -- 172.21.15.29:0/2358866941 learned_addr learned my addr 172.21.15.29:0/2358866941 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:20.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 -- 172.21.15.29:0/2358866941 >> v2:172.21.15.170:3300/0 conn(0x7fac5815cd70 msgr2=0x7fac580d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:20.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac5659c640 1 --2- 172.21.15.29:0/2358866941 >> v2:172.21.15.170:3300/0 conn(0x7fac5815cd70 0x7fac580d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 --2- 172.21.15.29:0/2358866941 >> v2:172.21.15.170:3300/0 conn(0x7fac5815cd70 0x7fac580d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 -- 172.21.15.29:0/2358866941 >> v1:172.21.15.170:6789/0 conn(0x7fac580d0be0 legacy=0x7fac580b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:20.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.309+0000 7fac4ffff640 1 -- 172.21.15.29:0/2358866941 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580d1740 con 0x7fac580b8480 2023-10-16T15:56:20.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.310+0000 7fac4ffff640 1 --2- 172.21.15.29:0/2358866941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac580cd7c0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fac580b56a0 tx=0x7fac4002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f515602d4868e32f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac55d9b640 1 -- 172.21.15.29:0/2358866941 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400317e0 con 0x7fac580b8480 2023-10-16T15:56:20.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac55d9b640 1 -- 172.21.15.29:0/2358866941 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac40031940 con 0x7fac580b8480 2023-10-16T15:56:20.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac55d9b640 1 -- 172.21.15.29:0/2358866941 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40031cd0 con 0x7fac580b8480 2023-10-16T15:56:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac5759e640 1 -- 172.21.15.29:0/2358866941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 msgr2=0x7fac580cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac5759e640 1 --2- 172.21.15.29:0/2358866941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac580cd7c0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fac580b56a0 tx=0x7fac4002b880 comp rx=0 tx=0).stop 2023-10-16T15:56:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac5759e640 1 -- 172.21.15.29:0/2358866941 shutdown_connections 2023-10-16T15:56:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac5759e640 1 --2- 172.21.15.29:0/2358866941 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac580cd7c0 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac5759e640 1 --2- 172.21.15.29:0/2358866941 >> v2:172.21.15.170:3300/0 conn(0x7fac5815cd70 0x7fac580d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.311+0000 7fac5759e640 1 -- 172.21.15.29:0/2358866941 >> 172.21.15.29:0/2358866941 conn(0x7fac580cf570 msgr2=0x7fac580cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.312+0000 7fac5759e640 1 -- 172.21.15.29:0/2358866941 shutdown_connections 2023-10-16T15:56:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.312+0000 7fac5759e640 1 -- 172.21.15.29:0/2358866941 wait complete. 2023-10-16T15:56:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.313+0000 7fac5759e640 1 Processor -- start 2023-10-16T15:56:20.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.313+0000 7fac5759e640 1 -- start start 2023-10-16T15:56:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.314+0000 7fac5759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac581555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.314+0000 7fac5759e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac580d0be0 0x7fac580ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.314+0000 7fac5759e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5815cd70 0x7fac5815ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.314+0000 7fac5759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580cc210 con 0x7fac580b8480 2023-10-16T15:56:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.314+0000 7fac5759e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580cbc50 con 0x7fac580d0be0 2023-10-16T15:56:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.314+0000 7fac5759e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580cbf30 con 0x7fac5815cd70 2023-10-16T15:56:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac581555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac581555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34494/0 (socket says 172.21.15.29:34494) 2023-10-16T15:56:20.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 -- 172.21.15.29:0/3512075516 learned_addr learned my addr 172.21.15.29:0/3512075516 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 -- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5815cd70 msgr2=0x7fac5815ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fac5815cd70 0x7fac5815ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 -- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac580d0be0 msgr2=0x7fac580ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac580d0be0 0x7fac580ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 -- 172.21.15.29:0/3512075516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac400094d0 con 0x7fac580b8480 2023-10-16T15:56:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.315+0000 7fac5659c640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac581555e0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c7f0 tx=0x7fac4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.316+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48004220 con 0x7fac580b8480 2023-10-16T15:56:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.316+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac480043a0 con 0x7fac580b8480 2023-10-16T15:56:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.316+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48007860 con 0x7fac580b8480 2023-10-16T15:56:20.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.316+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac580c9c60 con 0x7fac580b8480 2023-10-16T15:56:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.316+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac58155b20 con 0x7fac580b8480 2023-10-16T15:56:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.316+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac580d1d00 con 0x7fac580b8480 2023-10-16T15:56:20.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.318+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac48007bd0 con 0x7fac580b8480 2023-10-16T15:56:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.319+0000 7fac4effd640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac3806af90 0x7fac3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.319+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fac480898a0 con 0x7fac580b8480 2023-10-16T15:56:20.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.319+0000 7fac4effd640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7fac38071520 0x7fac38073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.319+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 --> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac38074010 con 0x7fac38071520 2023-10-16T15:56:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.319+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac4808e2c0 con 0x7fac580b8480 2023-10-16T15:56:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.319+0000 7fac4ffff640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac3806af90 0x7fac3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.320+0000 7fac56d9d640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7fac38071520 0x7fac38073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.321+0000 7fac56d9d640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7fac38071520 0x7fac38073940 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.321+0000 7fac4ffff640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac3806af90 0x7fac3806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac40036000 tx=0x7fac40030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.323+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== osd.5 v2:172.21.15.170:6800/3780367638 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fac38074010 con 0x7fac38071520 2023-10-16T15:56:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.336+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 --> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fac580d1a20 con 0x7fac38071520 2023-10-16T15:56:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.337+0000 7fac4effd640 1 -- 172.21.15.29:0/3512075516 <== osd.5 v2:172.21.15.170:6800/3780367638 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fac50001de0 con 0x7fac38071520 2023-10-16T15:56:20.339 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:20.339 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:20.339 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7fac38071520 msgr2=0x7fac38073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6800/3780367638,v1:172.21.15.170:6801/3780367638] conn(0x7fac38071520 0x7fac38073940 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac3806af90 msgr2=0x7fac3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fac3806af90 0x7fac3806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac40036000 tx=0x7fac40030000 comp rx=0 tx=0).stop 2023-10-16T15:56:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 msgr2=0x7fac581555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 --2- 172.21.15.29:0/3512075516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac580b8480 0x7fac581555e0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c7f0 tx=0x7fac4800ccb0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac56d9d640 1 -- 172.21.15.29:0/3512075516 reap_dead start 2023-10-16T15:56:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 shutdown_connections 2023-10-16T15:56:20.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 >> 172.21.15.29:0/3512075516 conn(0x7fac580cf570 msgr2=0x7fac580b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:20.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.338+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 shutdown_connections 2023-10-16T15:56:20.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.339+0000 7fac5759e640 1 -- 172.21.15.29:0/3512075516 wait complete. 2023-10-16T15:56:20.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-16T15:56:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.572+0000 7f431d213640 1 Processor -- start 2023-10-16T15:56:20.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.572+0000 7f431d213640 1 -- start start 2023-10-16T15:56:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f431d213640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f431806c040 0x7f431806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f431d213640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f431815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f431d213640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f431815ebc0 0x7f431815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f431d213640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431806c960 con 0x7f431806c040 2023-10-16T15:56:20.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f431d213640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431815b9d0 con 0x7f431815ebc0 2023-10-16T15:56:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f431d213640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431815bcb0 con 0x7f43180ad970 2023-10-16T15:56:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f4317fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f431806c040 0x7f431806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f4317fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f431806c040 0x7f431806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34510/0 (socket says 172.21.15.29:34510) 2023-10-16T15:56:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.574+0000 7f4317fff640 1 -- 172.21.15.29:0/4274963527 learned_addr learned my addr 172.21.15.29:0/4274963527 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.575+0000 7f4317fff640 1 -- 172.21.15.29:0/4274963527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 msgr2=0x7f431815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:20.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.575+0000 7f4317fff640 1 --2- 172.21.15.29:0/4274963527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f431815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.575+0000 7f4317fff640 1 -- 172.21.15.29:0/4274963527 >> v2:172.21.15.170:3300/0 conn(0x7f431815ebc0 msgr2=0x7f431815f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.575+0000 7f4317fff640 1 --2- 172.21.15.29:0/4274963527 >> v2:172.21.15.170:3300/0 conn(0x7f431815ebc0 0x7f431815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.575+0000 7f4317fff640 1 -- 172.21.15.29:0/4274963527 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431815bf90 con 0x7f431806c040 2023-10-16T15:56:20.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.576+0000 7f4317fff640 1 --2- 172.21.15.29:0/4274963527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c040 0x7f431806c420 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f4308009870 tx=0x7f430802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eaf3a66098bc372 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.576+0000 7f43177fe640 1 -- 172.21.15.29:0/4274963527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308031640 con 0x7f431806c040 2023-10-16T15:56:20.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.576+0000 7f43177fe640 1 -- 172.21.15.29:0/4274963527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43080317a0 con 0x7f431806c040 2023-10-16T15:56:20.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.576+0000 7f43177fe640 1 -- 172.21.15.29:0/4274963527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308031b30 con 0x7f431806c040 2023-10-16T15:56:20.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 -- 172.21.15.29:0/4274963527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c040 msgr2=0x7f431806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 --2- 172.21.15.29:0/4274963527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c040 0x7f431806c420 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f4308009870 tx=0x7f430802b8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 -- 172.21.15.29:0/4274963527 shutdown_connections 2023-10-16T15:56:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 --2- 172.21.15.29:0/4274963527 >> v2:172.21.15.170:3300/0 conn(0x7f431815ebc0 0x7f431815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 --2- 172.21.15.29:0/4274963527 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f431815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 --2- 172.21.15.29:0/4274963527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c040 0x7f431806c420 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 -- 172.21.15.29:0/4274963527 >> 172.21.15.29:0/4274963527 conn(0x7f43180ce6a0 msgr2=0x7f43180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:20.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 -- 172.21.15.29:0/4274963527 shutdown_connections 2023-10-16T15:56:20.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.577+0000 7f431d213640 1 -- 172.21.15.29:0/4274963527 wait complete. 2023-10-16T15:56:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.579+0000 7f431d213640 1 Processor -- start 2023-10-16T15:56:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.580+0000 7f431d213640 1 -- start start 2023-10-16T15:56:20.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431d213640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431806c040 0x7f431815aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431d213640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f43180b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431d213640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 0x7f43180b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431d213640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180af990 con 0x7f431815ebc0 2023-10-16T15:56:20.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431d213640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431806a150 con 0x7f431806c040 2023-10-16T15:56:20.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431d213640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43180afdb0 con 0x7f43180ad970 2023-10-16T15:56:20.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431ca12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 0x7f43180b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431ca12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 0x7f43180b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34522/0 (socket says 172.21.15.29:34522) 2023-10-16T15:56:20.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.581+0000 7f431ca12640 1 -- 172.21.15.29:0/400515014 learned_addr learned my addr 172.21.15.29:0/400515014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:20.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f4317fff640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431806c040 0x7f431815aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f430ffff640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f43180b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f431ca12640 1 -- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 msgr2=0x7f43180b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f431ca12640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f43180b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f431ca12640 1 -- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431806c040 msgr2=0x7f431815aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f431ca12640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431806c040 0x7f431815aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f431ca12640 1 -- 172.21.15.29:0/400515014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080094d0 con 0x7f431815ebc0 2023-10-16T15:56:20.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f431ca12640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 0x7f43180b0620 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f431000b970 tx=0x7f431000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431000cb40 con 0x7f431815ebc0 2023-10-16T15:56:20.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.582+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431000ccc0 con 0x7f431815ebc0 2023-10-16T15:56:20.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.583+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310014680 con 0x7f431815ebc0 2023-10-16T15:56:20.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.583+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43180aed10 con 0x7f431815ebc0 2023-10-16T15:56:20.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.583+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43180af2d0 con 0x7f431815ebc0 2023-10-16T15:56:20.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.584+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431806b330 con 0x7f431815ebc0 2023-10-16T15:56:20.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.585+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f431001d3b0 con 0x7f431815ebc0 2023-10-16T15:56:20.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.586+0000 7f43157fa640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f42f806af90 0x7f42f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:20.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.586+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f4310089790 con 0x7f431815ebc0 2023-10-16T15:56:20.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.587+0000 7f4317fff640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f42f806af90 0x7f42f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:20.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.588+0000 7f4317fff640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f42f806af90 0x7f42f806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4308036000 tx=0x7f430803c040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:20.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.590+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f43100546a0 con 0x7f431815ebc0 2023-10-16T15:56:20.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.782+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f43180cc550 con 0x7f42f806af90 2023-10-16T15:56:20.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.783+0000 7f43157fa640 1 -- 172.21.15.29:0/400515014 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4308031a90 con 0x7f42f806af90 2023-10-16T15:56:20.784 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 5 to deep-scrub 2023-10-16T15:56:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.785+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f42f806af90 msgr2=0x7f42f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.785+0000 7f431d213640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f42f806af90 0x7f42f806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4308036000 tx=0x7f430803c040 comp rx=0 tx=0).stop 2023-10-16T15:56:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.785+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 msgr2=0x7f43180b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.785+0000 7f431d213640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 0x7f43180b0620 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f431000b970 tx=0x7f431000be30 comp rx=0 tx=0).stop 2023-10-16T15:56:20.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 shutdown_connections 2023-10-16T15:56:20.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f42f806af90 0x7f42f806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431815ebc0 0x7f43180b0620 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f43180ad970 0x7f43180b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 --2- 172.21.15.29:0/400515014 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431806c040 0x7f431815aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:20.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 >> 172.21.15.29:0/400515014 conn(0x7f43180ce6a0 msgr2=0x7f43180c8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 shutdown_connections 2023-10-16T15:56:20.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:20.786+0000 7f431d213640 1 -- 172.21.15.29:0/400515014 wait complete. 2023-10-16T15:56:20.806 INFO:tasks.ceph:Scrubbing osd.6 2023-10-16T15:56:20.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:21.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.032+0000 7fb95b7c5640 1 Processor -- start 2023-10-16T15:56:21.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.032+0000 7fb95b7c5640 1 -- start start 2023-10-16T15:56:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.033+0000 7fb95b7c5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb95415af90 0x7fb9540ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb95b7c5640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fb9540cd320 0x7fb9540cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb95b7c5640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9540cb890 con 0x7fb9540cf3d0 2023-10-16T15:56:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb95b7c5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9540cef80 con 0x7fb95415af90 2023-10-16T15:56:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb95b7c5640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9540cbac0 con 0x7fb9540cd320 2023-10-16T15:56:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb95415af90 0x7fb9540ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb95415af90 0x7fb9540ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34530/0 (socket says 172.21.15.29:34530) 2023-10-16T15:56:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 -- 172.21.15.29:0/1625125477 learned_addr learned my addr 172.21.15.29:0/1625125477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb958d60640 1 --2- 172.21.15.29:0/1625125477 >> v2:172.21.15.170:3300/0 conn(0x7fb9540cd320 0x7fb9540cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 -- 172.21.15.29:0/1625125477 >> v2:172.21.15.170:3300/0 conn(0x7fb9540cd320 msgr2=0x7fb9540cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 --2- 172.21.15.29:0/1625125477 >> v2:172.21.15.170:3300/0 conn(0x7fb9540cd320 0x7fb9540cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 -- 172.21.15.29:0/1625125477 >> v1:172.21.15.170:6789/0 conn(0x7fb9540cf3d0 legacy=0x7fb9540afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.034+0000 7fb959561640 1 -- 172.21.15.29:0/1625125477 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9540cdcc0 con 0x7fb95415af90 2023-10-16T15:56:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.036+0000 7fb959561640 1 --2- 172.21.15.29:0/1625125477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb95415af90 0x7fb9540ccb60 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fb954070ea0 tx=0x7fb94402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df53ea067f20557 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.036+0000 7fb953fff640 1 -- 172.21.15.29:0/1625125477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94402ca80 con 0x7fb95415af90 2023-10-16T15:56:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.036+0000 7fb953fff640 1 -- 172.21.15.29:0/1625125477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94402cbe0 con 0x7fb95415af90 2023-10-16T15:56:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.036+0000 7fb953fff640 1 -- 172.21.15.29:0/1625125477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440343f0 con 0x7fb95415af90 2023-10-16T15:56:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 -- 172.21.15.29:0/1625125477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb95415af90 msgr2=0x7fb9540ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 --2- 172.21.15.29:0/1625125477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb95415af90 0x7fb9540ccb60 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fb954070ea0 tx=0x7fb94402bee0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 -- 172.21.15.29:0/1625125477 shutdown_connections 2023-10-16T15:56:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 --2- 172.21.15.29:0/1625125477 >> v2:172.21.15.170:3300/0 conn(0x7fb9540cd320 0x7fb9540cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 --2- 172.21.15.29:0/1625125477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb95415af90 0x7fb9540ccb60 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 -- 172.21.15.29:0/1625125477 >> 172.21.15.29:0/1625125477 conn(0x7fb954099e30 msgr2=0x7fb9540d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 -- 172.21.15.29:0/1625125477 shutdown_connections 2023-10-16T15:56:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.037+0000 7fb95b7c5640 1 -- 172.21.15.29:0/1625125477 wait complete. 2023-10-16T15:56:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.040+0000 7fb95b7c5640 1 Processor -- start 2023-10-16T15:56:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.040+0000 7fb95b7c5640 1 -- start start 2023-10-16T15:56:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.041+0000 7fb95b7c5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9540cd320 0x7fb9541552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.041+0000 7fb95b7c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9540cf3d0 0x7fb9541f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.041+0000 7fb95b7c5640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb95415af90 0x7fb95414acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.041+0000 7fb95b7c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9541f2b70 con 0x7fb9540cf3d0 2023-10-16T15:56:21.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.041+0000 7fb95b7c5640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540c8be0 con 0x7fb9540cd320 2023-10-16T15:56:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.041+0000 7fb95b7c5640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540c9760 con 0x7fb95415af90 2023-10-16T15:56:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9540cf3d0 0x7fb9541f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9540cf3d0 0x7fb9541f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34536/0 (socket says 172.21.15.29:34536) 2023-10-16T15:56:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 -- 172.21.15.29:0/3984643383 learned_addr learned my addr 172.21.15.29:0/3984643383 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb959561640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9540cd320 0x7fb9541552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 -- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb95415af90 msgr2=0x7fb95414acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fb95415af90 0x7fb95414acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 -- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9540cd320 msgr2=0x7fb9541552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb9540cd320 0x7fb9541552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 -- 172.21.15.29:0/3984643383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944009b30 con 0x7fb9540cf3d0 2023-10-16T15:56:21.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.042+0000 7fb958d60640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9540cf3d0 0x7fb9541f2630 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fb94800c820 tx=0x7fb94800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.043+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948004020 con 0x7fb9540cf3d0 2023-10-16T15:56:21.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.043+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb948004180 con 0x7fb9540cf3d0 2023-10-16T15:56:21.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.043+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948007830 con 0x7fb9540cf3d0 2023-10-16T15:56:21.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.043+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9540c8e30 con 0x7fb9540cf3d0 2023-10-16T15:56:21.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.043+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9540c93f0 con 0x7fb9540cf3d0 2023-10-16T15:56:21.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.044+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9540cc080 con 0x7fb9540cf3d0 2023-10-16T15:56:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.045+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb948007990 con 0x7fb9540cf3d0 2023-10-16T15:56:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.046+0000 7fb951ffb640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb94006af90 0x7fb94006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.046+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fb948089660 con 0x7fb9540cf3d0 2023-10-16T15:56:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.046+0000 7fb951ffb640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7fb940071520 0x7fb940073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.046+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 --> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb940074010 con 0x7fb940071520 2023-10-16T15:56:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.046+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb94808bc40 con 0x7fb9540cf3d0 2023-10-16T15:56:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.047+0000 7fb959561640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb94006af90 0x7fb94006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.047+0000 7fb959d62640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7fb940071520 0x7fb940073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.048+0000 7fb959561640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb94006af90 0x7fb94006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb944035000 tx=0x7fb944030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.048+0000 7fb959d62640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7fb940071520 0x7fb940073940 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.051+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== osd.6 v2:172.21.15.170:6816/690425284 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb940074010 con 0x7fb940071520 2023-10-16T15:56:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.065+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 --> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb9540cbda0 con 0x7fb940071520 2023-10-16T15:56:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.066+0000 7fb951ffb640 1 -- 172.21.15.29:0/3984643383 <== osd.6 v2:172.21.15.170:6816/690425284 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb94c009730 con 0x7fb940071520 2023-10-16T15:56:21.067 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:21.067 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:21.067 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.066+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7fb940071520 msgr2=0x7fb940073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.066+0000 7fb95b7c5640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6816/690425284,v1:172.21.15.170:6817/690425284] conn(0x7fb940071520 0x7fb940073940 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb94006af90 msgr2=0x7fb94006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fb94006af90 0x7fb94006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb944035000 tx=0x7fb944030000 comp rx=0 tx=0).stop 2023-10-16T15:56:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9540cf3d0 msgr2=0x7fb9541f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 --2- 172.21.15.29:0/3984643383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9540cf3d0 0x7fb9541f2630 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fb94800c820 tx=0x7fb94800cce0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb959d62640 1 -- 172.21.15.29:0/3984643383 reap_dead start 2023-10-16T15:56:21.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 shutdown_connections 2023-10-16T15:56:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 >> 172.21.15.29:0/3984643383 conn(0x7fb954099e30 msgr2=0x7fb9540b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 shutdown_connections 2023-10-16T15:56:21.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.067+0000 7fb95b7c5640 1 -- 172.21.15.29:0/3984643383 wait complete. 2023-10-16T15:56:21.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-16T15:56:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.316+0000 7fd79b7bb640 1 Processor -- start 2023-10-16T15:56:21.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.316+0000 7fd79b7bb640 1 -- start start 2023-10-16T15:56:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd79b7bb640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7fd79406b8c0 0x7fd79406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd79b7bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7940cd390 con 0x7fd79415d130 2023-10-16T15:56:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd79b7bb640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79406c1e0 con 0x7fd79406b1a0 2023-10-16T15:56:21.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd79b7bb640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7940cc2f0 con 0x7fd79406b8c0 2023-10-16T15:56:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd798d56640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd79415d130 0x7fd7940cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52272/0 (socket says 172.21.15.29:52272) 2023-10-16T15:56:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd798d56640 1 -- 172.21.15.29:0/7973071 learned_addr learned my addr 172.21.15.29:0/7973071 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd799557640 1 --2- 172.21.15.29:0/7973071 >> v2:172.21.15.170:3300/0 conn(0x7fd79406b8c0 0x7fd79406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028146878 0 0) 0x7fd7940cd390 con 0x7fd79415d130 2023-10-16T15:56:21.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.318+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd780003520 con 0x7fd79415d130 2023-10-16T15:56:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132249145 0 0) 0x7fd780003520 con 0x7fd79415d130 2023-10-16T15:56:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 >> v2:172.21.15.170:3300/0 conn(0x7fd79406b8c0 msgr2=0x7fd79406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 --2- 172.21.15.29:0/7973071 >> v2:172.21.15.170:3300/0 conn(0x7fd79406b8c0 0x7fd79406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 >> v1:172.21.15.170:6789/0 conn(0x7fd79406b1a0 legacy=0x7fd7940ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940cc5d0 con 0x7fd79415d130 2023-10-16T15:56:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7fd784001670 con 0x7fd79415d130 2023-10-16T15:56:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 >> v1:172.21.15.29:6789/0 conn(0x7fd79415d130 legacy=0x7fd7940cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd780003970 0x7fd780003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd780005180 0x7fd78000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940cc2f0 con 0x7fd7800047c0 2023-10-16T15:56:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78000c020 con 0x7fd780005180 2023-10-16T15:56:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78000c1e0 con 0x7fd780003970 2023-10-16T15:56:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.319+0000 7fd798d56640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd799d58640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd780005180 0x7fd78000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd799557640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd780003970 0x7fd780003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd798d56640 1 -- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd780003970 msgr2=0x7fd780003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd798d56640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd780003970 0x7fd780003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd798d56640 1 -- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd780005180 msgr2=0x7fd78000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd798d56640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd780005180 0x7fd78000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd798d56640 1 -- 172.21.15.29:0/7973071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780003520 con 0x7fd7800047c0 2023-10-16T15:56:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd799d58640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd780005180 0x7fd78000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.320+0000 7fd799d58640 1 -- 172.21.15.29:0/7973071 reap_dead start 2023-10-16T15:56:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.321+0000 7fd798d56640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c240 tx=0x7fd7800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f064b481057ebe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.321+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784005ad0 con 0x7fd7800047c0 2023-10-16T15:56:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.321+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd784005c30 con 0x7fd7800047c0 2023-10-16T15:56:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.321+0000 7fd78bfff640 1 -- 172.21.15.29:0/7973071 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78402caf0 con 0x7fd7800047c0 2023-10-16T15:56:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 -- 172.21.15.29:0/7973071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800047c0 msgr2=0x7fd780004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c240 tx=0x7fd7800057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 -- 172.21.15.29:0/7973071 shutdown_connections 2023-10-16T15:56:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/7973071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800047c0 0x7fd780004c40 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 -- 172.21.15.29:0/7973071 >> 172.21.15.29:0/7973071 conn(0x7fd7940d0ad0 msgr2=0x7fd7940d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 -- 172.21.15.29:0/7973071 shutdown_connections 2023-10-16T15:56:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.322+0000 7fd79b7bb640 1 -- 172.21.15.29:0/7973071 wait complete. 2023-10-16T15:56:21.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.324+0000 7fd79b7bb640 1 Processor -- start 2023-10-16T15:56:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.324+0000 7fd79b7bb640 1 -- start start 2023-10-16T15:56:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd79b7bb640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7800047c0 0x7fd79415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd79b7bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 0x7fd79409c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd79b7bb640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd79409cee0 0x7fd79409d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd79b7bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79415beb0 con 0x7fd79415b610 2023-10-16T15:56:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd79b7bb640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940c9720 con 0x7fd7800047c0 2023-10-16T15:56:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd79b7bb640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7941e75b0 con 0x7fd79409cee0 2023-10-16T15:56:21.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd798d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 0x7fd79409c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd798d56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 0x7fd79409c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34556/0 (socket says 172.21.15.29:34556) 2023-10-16T15:56:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.325+0000 7fd798d56640 1 -- 172.21.15.29:0/271084498 learned_addr learned my addr 172.21.15.29:0/271084498 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd799557640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7800047c0 0x7fd79415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd798d56640 1 -- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd79409cee0 msgr2=0x7fd79409d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd798d56640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd79409cee0 0x7fd79409d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd798d56640 1 -- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7800047c0 msgr2=0x7fd79415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd798d56640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7800047c0 0x7fd79415ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd798d56640 1 -- 172.21.15.29:0/271084498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7941e7890 con 0x7fd79415b610 2023-10-16T15:56:21.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd798d56640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 0x7fd79409c9a0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c240 tx=0x7fd784001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784044070 con 0x7fd79415b610 2023-10-16T15:56:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78403a050 con 0x7fd79415b610 2023-10-16T15:56:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840049e0 con 0x7fd79415b610 2023-10-16T15:56:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.326+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7940c93b0 con 0x7fd79415b610 2023-10-16T15:56:21.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.327+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7941e7b70 con 0x7fd79415b610 2023-10-16T15:56:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.328+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79406f630 con 0x7fd79415b610 2023-10-16T15:56:21.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.329+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd784032070 con 0x7fd79415b610 2023-10-16T15:56:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.330+0000 7fd789ffb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd77806afc0 0x7fd77806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.330+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7fd7840ab2e0 con 0x7fd79415b610 2023-10-16T15:56:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.331+0000 7fd799557640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd77806afc0 0x7fd77806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.332+0000 7fd799557640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd77806afc0 0x7fd77806d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd790009a80 tx=0x7fd790008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.334+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd784076320 con 0x7fd79415b610 2023-10-16T15:56:21.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.529+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7940cce70 con 0x7fd77806afc0 2023-10-16T15:56:21.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.530+0000 7fd789ffb640 1 -- 172.21.15.29:0/271084498 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd790005990 con 0x7fd77806afc0 2023-10-16T15:56:21.531 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 6 to deep-scrub 2023-10-16T15:56:21.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd77806afc0 msgr2=0x7fd77806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd77806afc0 0x7fd77806d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd790009a80 tx=0x7fd790008040 comp rx=0 tx=0).stop 2023-10-16T15:56:21.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 msgr2=0x7fd79409c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 0x7fd79409c9a0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fd78400c240 tx=0x7fd784001a00 comp rx=0 tx=0).stop 2023-10-16T15:56:21.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 shutdown_connections 2023-10-16T15:56:21.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7fd77806afc0 0x7fd77806d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd79415b610 0x7fd79409c9a0 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7fd79409cee0 0x7fd79409d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 --2- 172.21.15.29:0/271084498 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7800047c0 0x7fd79415ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 >> 172.21.15.29:0/271084498 conn(0x7fd7940d0ad0 msgr2=0x7fd7940af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:21.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 shutdown_connections 2023-10-16T15:56:21.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.533+0000 7fd79b7bb640 1 -- 172.21.15.29:0/271084498 wait complete. 2023-10-16T15:56:21.554 INFO:tasks.ceph:Scrubbing osd.7 2023-10-16T15:56:21.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-16T15:56:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.785+0000 7f9973d7c640 1 Processor -- start 2023-10-16T15:56:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.785+0000 7f9973d7c640 1 -- start start 2023-10-16T15:56:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.786+0000 7f9973d7c640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f996c15cf60 0x7f996c06f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.786+0000 7f9973d7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f996c06f9d0 0x7f996c0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.786+0000 7f9973d7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 0x7f996c0b0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.786+0000 7f9973d7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c15d340 con 0x7f996c06f9d0 2023-10-16T15:56:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.786+0000 7f9973d7c640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c0cc860 con 0x7f996c15cf60 2023-10-16T15:56:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.786+0000 7f9973d7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c9620 con 0x7f996c0c9ff0 2023-10-16T15:56:21.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f996c06f9d0 0x7f996c0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f996c06f9d0 0x7f996c0c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34562/0 (socket says 172.21.15.29:34562) 2023-10-16T15:56:21.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 -- 172.21.15.29:0/1733718132 learned_addr learned my addr 172.21.15.29:0/1733718132 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:21.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9972319640 1 --2- 172.21.15.29:0/1733718132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 0x7f996c0b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971b18640 1 --2- 172.21.15.29:0/1733718132 >> v2:172.21.15.170:3300/0 conn(0x7f996c15cf60 0x7f996c06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 -- 172.21.15.29:0/1733718132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 msgr2=0x7f996c0b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 --2- 172.21.15.29:0/1733718132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 0x7f996c0b0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 -- 172.21.15.29:0/1733718132 >> v2:172.21.15.170:3300/0 conn(0x7f996c15cf60 msgr2=0x7f996c06f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:21.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 --2- 172.21.15.29:0/1733718132 >> v2:172.21.15.170:3300/0 conn(0x7f996c15cf60 0x7f996c06f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.787+0000 7f9971317640 1 -- 172.21.15.29:0/1733718132 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c0ad4a0 con 0x7f996c06f9d0 2023-10-16T15:56:21.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.788+0000 7f9971317640 1 --2- 172.21.15.29:0/1733718132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c0c9ab0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f995c009870 tx=0x7f995c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13babe3e43870f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.789+0000 7f9970b16640 1 -- 172.21.15.29:0/1733718132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c031640 con 0x7f996c06f9d0 2023-10-16T15:56:21.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.789+0000 7f9970b16640 1 -- 172.21.15.29:0/1733718132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c0317a0 con 0x7f996c06f9d0 2023-10-16T15:56:21.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.789+0000 7f9970b16640 1 -- 172.21.15.29:0/1733718132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c031b30 con 0x7f996c06f9d0 2023-10-16T15:56:21.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.789+0000 7f9973d7c640 1 -- 172.21.15.29:0/1733718132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 msgr2=0x7f996c0c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 --2- 172.21.15.29:0/1733718132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c0c9ab0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f995c009870 tx=0x7f995c02b8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 -- 172.21.15.29:0/1733718132 shutdown_connections 2023-10-16T15:56:21.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 --2- 172.21.15.29:0/1733718132 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 0x7f996c0b0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 --2- 172.21.15.29:0/1733718132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c0c9ab0 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 --2- 172.21.15.29:0/1733718132 >> v2:172.21.15.170:3300/0 conn(0x7f996c15cf60 0x7f996c06f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 -- 172.21.15.29:0/1733718132 >> 172.21.15.29:0/1733718132 conn(0x7f996c0cb360 msgr2=0x7f996c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:21.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 -- 172.21.15.29:0/1733718132 shutdown_connections 2023-10-16T15:56:21.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.790+0000 7f9973d7c640 1 -- 172.21.15.29:0/1733718132 wait complete. 2023-10-16T15:56:21.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.792+0000 7f9973d7c640 1 Processor -- start 2023-10-16T15:56:21.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.792+0000 7f9973d7c640 1 -- start start 2023-10-16T15:56:21.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9973d7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c1f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9973d7c640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 0x7f996c154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9973d7c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c15cf60 0x7f996c0c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9973d7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c7060 con 0x7f996c06f9d0 2023-10-16T15:56:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9973d7c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0b2260 con 0x7f996c15cf60 2023-10-16T15:56:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9973d7c640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0b2540 con 0x7f996c0c9ff0 2023-10-16T15:56:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9971b18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c1f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9971b18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c1f1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34572/0 (socket says 172.21.15.29:34572) 2023-10-16T15:56:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.793+0000 7f9971b18640 1 -- 172.21.15.29:0/4133440561 learned_addr learned my addr 172.21.15.29:0/4133440561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9971b18640 1 -- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 msgr2=0x7f996c154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9971b18640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f996c0c9ff0 0x7f996c154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9971b18640 1 -- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c15cf60 msgr2=0x7f996c0c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9971b18640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f996c15cf60 0x7f996c0c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9971b18640 1 -- 172.21.15.29:0/4133440561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c0094d0 con 0x7f996c06f9d0 2023-10-16T15:56:21.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9971b18640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c1f1ec0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f996800d5f0 tx=0x7f996800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680125f0 con 0x7f996c06f9d0 2023-10-16T15:56:21.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9968012750 con 0x7f996c06f9d0 2023-10-16T15:56:21.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968012b00 con 0x7f996c06f9d0 2023-10-16T15:56:21.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c0c6c80 con 0x7f996c06f9d0 2023-10-16T15:56:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.794+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c0c68c0 con 0x7f996c06f9d0 2023-10-16T15:56:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.795+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f996c0c92b0 con 0x7f996c06f9d0 2023-10-16T15:56:21.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.797+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9968012c60 con 0x7f996c06f9d0 2023-10-16T15:56:21.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.797+0000 7f99627fc640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f995006af90 0x7f995006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.798+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f9968089c70 con 0x7f996c06f9d0 2023-10-16T15:56:21.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.798+0000 7f99627fc640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7f9950071520 0x7f9950073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:21.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.798+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 --> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9950074010 con 0x7f9950071520 2023-10-16T15:56:21.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.798+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=271) v2 ==== 24+0+0 (secure 0 0 0) 0x7f996808c330 con 0x7f996c06f9d0 2023-10-16T15:56:21.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.798+0000 7f9971317640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f995006af90 0x7f995006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.799+0000 7f9972319640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7f9950071520 0x7f9950073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.800+0000 7f9971317640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f995006af90 0x7f995006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f995c036000 tx=0x7f995c030000 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.800+0000 7f9972319640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7f9950071520 0x7f9950073940 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:21.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.802+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== osd.7 v2:172.21.15.170:6821/3840642819 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9950074010 con 0x7f9950071520 2023-10-16T15:56:21.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.818+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 --> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f996c0c8fd0 con 0x7f9950071520 2023-10-16T15:56:21.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.819+0000 7f99627fc640 1 -- 172.21.15.29:0/4133440561 <== osd.7 v2:172.21.15.170:6821/3840642819 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9964009a30 con 0x7f9950071520 2023-10-16T15:56:21.820 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-16T15:56:21.821 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T15:56:21.821 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-16T15:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7f9950071520 msgr2=0x7f9950073940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6821/3840642819,v1:172.21.15.170:6823/3840642819] conn(0x7f9950071520 0x7f9950073940 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f995006af90 msgr2=0x7f995006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f995006af90 0x7f995006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f995c036000 tx=0x7f995c030000 comp rx=0 tx=0).stop 2023-10-16T15:56:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 msgr2=0x7f996c1f1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 --2- 172.21.15.29:0/4133440561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f996c06f9d0 0x7f996c1f1ec0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f996800d5f0 tx=0x7f996800dab0 comp rx=0 tx=0).stop 2023-10-16T15:56:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9972319640 1 -- 172.21.15.29:0/4133440561 reap_dead start 2023-10-16T15:56:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 shutdown_connections 2023-10-16T15:56:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 >> 172.21.15.29:0/4133440561 conn(0x7f996c0cb360 msgr2=0x7f996c0ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:21.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.820+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 shutdown_connections 2023-10-16T15:56:21.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:21.821+0000 7f9973d7c640 1 -- 172.21.15.29:0/4133440561 wait complete. 2023-10-16T15:56:21.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-16T15:56:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.053+0000 7f833e34e640 1 Processor -- start 2023-10-16T15:56:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.053+0000 7f833e34e640 1 -- start start 2023-10-16T15:56:22.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833e34e640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f8338070b60 0x7f833815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833e34e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833815aef0 con 0x7f8338159950 2023-10-16T15:56:22.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833e34e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833815d910 con 0x7f833806c100 2023-10-16T15:56:22.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833e34e640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833815e490 con 0x7f8338070b60 2023-10-16T15:56:22.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833d34c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8338159950 0x7f8338070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52278/0 (socket says 172.21.15.29:52278) 2023-10-16T15:56:22.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833d34c640 1 -- 172.21.15.29:0/1661787455 learned_addr learned my addr 172.21.15.29:0/1661787455 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:22.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.055+0000 7f833cb4b640 1 --2- 172.21.15.29:0/1661787455 >> v2:172.21.15.170:3300/0 conn(0x7f8338070b60 0x7f833815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424782377 0 0) 0x7f8334001110 con 0x7f8338159950 2023-10-16T15:56:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8324003520 con 0x7f8338159950 2023-10-16T15:56:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090920510 0 0) 0x7f8324003520 con 0x7f8338159950 2023-10-16T15:56:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 >> v2:172.21.15.170:3300/0 conn(0x7f8338070b60 msgr2=0x7f833815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 --2- 172.21.15.29:0/1661787455 >> v2:172.21.15.170:3300/0 conn(0x7f8338070b60 0x7f833815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 >> v1:172.21.15.170:6789/0 conn(0x7f833806c100 legacy=0x7f833806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833815dbf0 con 0x7f8338159950 2023-10-16T15:56:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f8334001800 con 0x7f8338159950 2023-10-16T15:56:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 >> v1:172.21.15.29:6789/0 conn(0x7f8338159950 legacy=0x7f8338070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8324003970 0x7f8324003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83240047c0 0x7f8324004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8324005180 0x7f832400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833815e490 con 0x7f83240047c0 2023-10-16T15:56:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832400c120 con 0x7f8324003970 2023-10-16T15:56:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.056+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832400c2e0 con 0x7f8324005180 2023-10-16T15:56:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833cb4b640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83240047c0 0x7f8324004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833d34c640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8324003970 0x7f8324003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833db4d640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8324005180 0x7f832400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833cb4b640 1 -- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8324005180 msgr2=0x7f832400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833cb4b640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f8324005180 0x7f832400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833cb4b640 1 -- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8324003970 msgr2=0x7f8324003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833cb4b640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8324003970 0x7f8324003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833cb4b640 1 -- 172.21.15.29:0/1661787455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324003520 con 0x7f83240047c0 2023-10-16T15:56:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833db4d640 1 -- 172.21.15.29:0/1661787455 reap_dead start 2023-10-16T15:56:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.057+0000 7f833d34c640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8324003970 0x7f8324003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:56:22.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.059+0000 7f833cb4b640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83240047c0 0x7f8324004c40 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f8328009ac0 tx=0x7f83240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f84109470a98ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.059+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832802c6b0 con 0x7f83240047c0 2023-10-16T15:56:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.059+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832802c810 con 0x7f83240047c0 2023-10-16T15:56:22.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.059+0000 7f832ffff640 1 -- 172.21.15.29:0/1661787455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832802cba0 con 0x7f83240047c0 2023-10-16T15:56:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.059+0000 7f833e34e640 1 -- 172.21.15.29:0/1661787455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83240047c0 msgr2=0x7f8324004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.059+0000 7f833e34e640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83240047c0 0x7f8324004c40 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f8328009ac0 tx=0x7f83240057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.060+0000 7f833e34e640 1 -- 172.21.15.29:0/1661787455 shutdown_connections 2023-10-16T15:56:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.060+0000 7f833e34e640 1 --2- 172.21.15.29:0/1661787455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83240047c0 0x7f8324004c40 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.060+0000 7f833e34e640 1 -- 172.21.15.29:0/1661787455 >> 172.21.15.29:0/1661787455 conn(0x7f83380ce6a0 msgr2=0x7f83380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.060+0000 7f833e34e640 1 -- 172.21.15.29:0/1661787455 shutdown_connections 2023-10-16T15:56:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.060+0000 7f833e34e640 1 -- 172.21.15.29:0/1661787455 wait complete. 2023-10-16T15:56:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.062+0000 7f833e34e640 1 Processor -- start 2023-10-16T15:56:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.062+0000 7f833e34e640 1 -- start start 2023-10-16T15:56:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833e34e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240047c0 0x7f83380c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833e34e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83381584d0 0x7f8338157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833e34e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 0x7f83380cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833e34e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380ce170 con 0x7f83380cd780 2023-10-16T15:56:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833e34e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380c3d90 con 0x7f83381584d0 2023-10-16T15:56:22.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833e34e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380c4070 con 0x7f83240047c0 2023-10-16T15:56:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 0x7f83380cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 0x7f83380cdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34584/0 (socket says 172.21.15.29:34584) 2023-10-16T15:56:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 -- 172.21.15.29:0/1715594755 learned_addr learned my addr 172.21.15.29:0/1715594755 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 -- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240047c0 msgr2=0x7f83380c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833d34c640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240047c0 0x7f83380c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833cb4b640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83381584d0 0x7f8338157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240047c0 0x7f83380c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 -- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83381584d0 msgr2=0x7f8338157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83381584d0 0x7f8338157c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.063+0000 7f833db4d640 1 -- 172.21.15.29:0/1715594755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380b1390 con 0x7f83380cd780 2023-10-16T15:56:22.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.064+0000 7f833db4d640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 0x7f83380cdc30 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f8330007fd0 tx=0x7f833000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.064+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330003970 con 0x7f83380cd780 2023-10-16T15:56:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.064+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330003af0 con 0x7f83380cd780 2023-10-16T15:56:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.064+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330003ea0 con 0x7f83380cd780 2023-10-16T15:56:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.064+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83380b1670 con 0x7f83380cd780 2023-10-16T15:56:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.064+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83381581a0 con 0x7f83380cd780 2023-10-16T15:56:22.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.065+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83380b11f0 con 0x7f83380cd780 2023-10-16T15:56:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.066+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f83300050d0 con 0x7f83380cd780 2023-10-16T15:56:22.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.067+0000 7f832dffb640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f831c06afc0 0x7f831c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.067+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f8330089650 con 0x7f83380cd780 2023-10-16T15:56:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.068+0000 7f833d34c640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f831c06afc0 0x7f831c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.069+0000 7f833d34c640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f831c06afc0 0x7f831c06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8334001390 tx=0x7f8334000ed0 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:22.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.071+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83380b11f0 con 0x7f83380cd780 2023-10-16T15:56:22.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.263+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f83381588b0 con 0x7f831c06afc0 2023-10-16T15:56:22.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.265+0000 7f832dffb640 1 -- 172.21.15.29:0/1715594755 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83381588b0 con 0x7f831c06afc0 2023-10-16T15:56:22.266 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 7 to deep-scrub 2023-10-16T15:56:22.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.267+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f831c06afc0 msgr2=0x7f831c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.267+0000 7f833e34e640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f831c06afc0 0x7f831c06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8334001390 tx=0x7f8334000ed0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.267+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 msgr2=0x7f83380cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.267+0000 7f833e34e640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 0x7f83380cdc30 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f8330007fd0 tx=0x7f833000ec10 comp rx=0 tx=0).stop 2023-10-16T15:56:22.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 shutdown_connections 2023-10-16T15:56:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f831c06afc0 0x7f831c06d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380cd780 0x7f83380cdc30 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83381584d0 0x7f8338157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 --2- 172.21.15.29:0/1715594755 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f83240047c0 0x7f83380c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 >> 172.21.15.29:0/1715594755 conn(0x7f83380ce6a0 msgr2=0x7f83380ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 shutdown_connections 2023-10-16T15:56:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.268+0000 7f833e34e640 1 -- 172.21.15.29:0/1715594755 wait complete. 2023-10-16T15:56:22.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:56:22.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.517+0000 7f3751773640 1 Processor -- start 2023-10-16T15:56:22.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.517+0000 7f3751773640 1 -- start start 2023-10-16T15:56:22.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.518+0000 7f3751773640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f374c06c040 0x7f374c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.518+0000 7f3751773640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 0x7f374c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.518+0000 7f3751773640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374c06c960 con 0x7f374c15ebc0 2023-10-16T15:56:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.518+0000 7f3751773640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374c15b9d0 con 0x7f374c06c040 2023-10-16T15:56:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.518+0000 7f3751773640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c15bcb0 con 0x7f374c0ad970 2023-10-16T15:56:22.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.519+0000 7f3750f72640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f374c15ebc0 0x7f374c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52284/0 (socket says 172.21.15.29:52284) 2023-10-16T15:56:22.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.519+0000 7f3750f72640 1 -- 172.21.15.29:0/441837217 learned_addr learned my addr 172.21.15.29:0/441837217 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.519+0000 7f3743fff640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 0x7f374c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.519+0000 7f374bfff640 1 --2- 172.21.15.29:0/441837217 >> v2:172.21.15.170:3300/0 conn(0x7f374c06c040 0x7f374c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.519+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607638022 0 0) 0x7f374c06c960 con 0x7f374c15ebc0 2023-10-16T15:56:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.519+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3738003520 con 0x7f374c15ebc0 2023-10-16T15:56:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080643573 0 0) 0x7f3738003520 con 0x7f374c15ebc0 2023-10-16T15:56:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 msgr2=0x7f374c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 0x7f374c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 >> v2:172.21.15.170:3300/0 conn(0x7f374c06c040 msgr2=0x7f374c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 --2- 172.21.15.29:0/441837217 >> v2:172.21.15.170:3300/0 conn(0x7f374c06c040 0x7f374c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c15bf90 con 0x7f374c15ebc0 2023-10-16T15:56:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f37440019f0 con 0x7f374c15ebc0 2023-10-16T15:56:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 >> v1:172.21.15.29:6789/0 conn(0x7f374c15ebc0 legacy=0x7f374c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f3738003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37380047c0 0x7f3738004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 0x7f3738005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c15b9d0 con 0x7f3738003970 2023-10-16T15:56:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c15bcb0 con 0x7f37380047c0 2023-10-16T15:56:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373800b980 con 0x7f374c0ad970 2023-10-16T15:56:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.520+0000 7f374bfff640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f3738003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.521+0000 7f374bfff640 1 -- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 msgr2=0x7f3738005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.521+0000 7f374bfff640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0ad970 0x7f3738005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.521+0000 7f374bfff640 1 -- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37380047c0 msgr2=0x7f3738004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.521+0000 7f3743fff640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37380047c0 0x7f3738004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.521+0000 7f374bfff640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37380047c0 0x7f3738004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.521+0000 7f374bfff640 1 -- 172.21.15.29:0/441837217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738003520 con 0x7f3738003970 2023-10-16T15:56:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.522+0000 7f374bfff640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f3738003d90 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f373c009ac0 tx=0x7f37380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65f92f8bfd9014d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.522+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c02c710 con 0x7f3738003970 2023-10-16T15:56:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.522+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c02c870 con 0x7f3738003970 2023-10-16T15:56:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.522+0000 7f374b7fe640 1 -- 172.21.15.29:0/441837217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c02cc00 con 0x7f3738003970 2023-10-16T15:56:22.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3751773640 1 -- 172.21.15.29:0/441837217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 msgr2=0x7f3738003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3751773640 1 --2- 172.21.15.29:0/441837217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f3738003d90 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f373c009ac0 tx=0x7f37380057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3750f72640 1 -- 172.21.15.29:0/441837217 reap_dead start 2023-10-16T15:56:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3751773640 1 -- 172.21.15.29:0/441837217 shutdown_connections 2023-10-16T15:56:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3751773640 1 -- 172.21.15.29:0/441837217 >> 172.21.15.29:0/441837217 conn(0x7f374c0ce6a0 msgr2=0x7f374c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3751773640 1 -- 172.21.15.29:0/441837217 shutdown_connections 2023-10-16T15:56:22.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.523+0000 7f3751773640 1 -- 172.21.15.29:0/441837217 wait complete. 2023-10-16T15:56:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.525+0000 7f3751773640 1 Processor -- start 2023-10-16T15:56:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.525+0000 7f3751773640 1 -- start start 2023-10-16T15:56:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f3751773640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f374c150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f3751773640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f374c0acf40 0x7f374c0b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f3751773640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0b2e10 0x7f374c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f3751773640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c151110 con 0x7f3738003970 2023-10-16T15:56:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f3751773640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c0b3290 con 0x7f374c0acf40 2023-10-16T15:56:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f3751773640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c0b0580 con 0x7f374c0b2e10 2023-10-16T15:56:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f374bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f374c150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f374bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f374c150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34598/0 (socket says 172.21.15.29:34598) 2023-10-16T15:56:22.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f374bfff640 1 -- 172.21.15.29:0/376689853 learned_addr learned my addr 172.21.15.29:0/376689853 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f374bfff640 1 -- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0b2e10 msgr2=0x7f374c0c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.526+0000 7f374bfff640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0b2e10 0x7f374c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f374bfff640 1 -- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f374c0acf40 msgr2=0x7f374c0b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f374bfff640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f374c0acf40 0x7f374c0b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f374bfff640 1 -- 172.21.15.29:0/376689853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c0b0860 con 0x7f3738003970 2023-10-16T15:56:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f374bfff640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f374c150bd0 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f373c02c2b0 tx=0x7f373c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c003810 con 0x7f3738003970 2023-10-16T15:56:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c02de90 con 0x7f3738003970 2023-10-16T15:56:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374c0b0b40 con 0x7f3738003970 2023-10-16T15:56:22.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c003b70 con 0x7f3738003970 2023-10-16T15:56:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.527+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374c15d960 con 0x7f3738003970 2023-10-16T15:56:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.528+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374c06b330 con 0x7f3738003970 2023-10-16T15:56:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.529+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f373c003cd0 con 0x7f3738003970 2023-10-16T15:56:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.530+0000 7f37497fa640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f372c06af90 0x7f372c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:22.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.531+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f373c0ab680 con 0x7f3738003970 2023-10-16T15:56:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.531+0000 7f3743fff640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f372c06af90 0x7f372c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.532+0000 7f3743fff640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f372c06af90 0x7f372c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f374c0ace90 tx=0x7f3734008040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:22.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.535+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f373c076590 con 0x7f3738003970 2023-10-16T15:56:22.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.726+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f374c0cc550 con 0x7f372c06af90 2023-10-16T15:56:22.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.733+0000 7f37497fa640 1 -- 172.21.15.29:0/376689853 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47741 (secure 0 0 0) 0x7f37340095c0 con 0x7f372c06af90 2023-10-16T15:56:22.735 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:22.735 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:56:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f372c06af90 msgr2=0x7f372c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f372c06af90 0x7f372c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f374c0ace90 tx=0x7f3734008040 comp rx=0 tx=0).stop 2023-10-16T15:56:22.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 msgr2=0x7f374c150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f374c150bd0 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f373c02c2b0 tx=0x7f373c004730 comp rx=0 tx=0).stop 2023-10-16T15:56:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 shutdown_connections 2023-10-16T15:56:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f372c06af90 0x7f372c06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f374c0b2e10 0x7f374c0c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f374c0acf40 0x7f374c0b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 --2- 172.21.15.29:0/376689853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738003970 0x7f374c150bd0 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 >> 172.21.15.29:0/376689853 conn(0x7f374c0ce6a0 msgr2=0x7f374c0c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 shutdown_connections 2023-10-16T15:56:22.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:22.736+0000 7f3751773640 1 -- 172.21.15.29:0/376689853 wait complete. 2023-10-16T15:56:22.755 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-10-16T15:56:20.825718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":456476,"kb_used_data":27944,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754518244,"statfs":{"total":773094113280,"available":772626681856,"internally_reserved":0,"allocated":28614656,"data_stored":14750024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":438685241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.752751"},"pg_stats":[{"pgid":"1.0","version":"268'2809","reported_seq":3923,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-16T15:56:09.046325+0000","last_change":"2023-10-16T14:16:19.733775+0000","last_active":"2023-10-16T15:56:09.046325+0000","last_peered":"2023-10-16T15:56:09.046325+0000","last_clean":"2023-10-16T15:56:09.046325+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:56:09.046325+0000","last_undegraded":"2023-10-16T15:56:09.046325+0000","last_fullsized":"2023-10-16T15:56:09.046325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:16:18.594607+0000","last_clean_scrub_stamp":"2023-10-16T14:16:18.594607+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:17.339815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2809,"ondisk_log_size":2809,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57316,"kb_used_data":3928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314524,"statfs":{"total":96636764160,"available":96578072576,"internally_reserved":0,"allocated":4022272,"data_stored":2286529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0820000000000001,"15min":1.163},"min":{"1min":0.70099999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.72,"5min":1.794,"15min":1.8879999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.998,"5min":1.083,"15min":1.1339999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.486,"5min":1.796,"15min":1.796},"last":1.1299999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.105,"15min":1.1559999999999999},"min":{"1min":0.66800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7170000000000001,"5min":1.76,"15min":1.8580000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7809999999999999,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.5649999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2969999999999999,"15min":1.3260000000000001},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.77,"5min":1.893,"15min":2.1269999999999998},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.2230000000000001,"15min":1.2829999999999999},"min":{"1min":0.82999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7110000000000001,"5min":1.909,"15min":1.95},"last":1.2769999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2629999999999999,"15min":1.327},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.823,"5min":1.843,"15min":1.869},"last":1.054},{"interface":"front","average":{"1min":1.385,"5min":1.387,"15min":1.4390000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8500000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.782}]},{"osd":4,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.385,"15min":1.385},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.907,"5min":1.9370000000000001,"15min":2.2610000000000001},"last":1.177},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.337},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.667,"5min":1.8740000000000001,"15min":2.0409999999999999},"last":1.4930000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.456},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":2.1469999999999998},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.502,"15min":1.552},"min":{"1min":0.67900000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3540000000000001},"last":1.839}]},{"osd":6,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4430000000000001,"15min":1.4770000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9330000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.609},{"interface":"front","average":{"1min":1.587,"5min":1.5609999999999999,"15min":1.5800000000000001},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.998,"5min":1.998,"15min":2.2599999999999998},"last":1.712}]}]},{"osd":6,"up_from":13,"seq":55834576033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52576,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.208,"15min":1.206},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9099999999999999,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":1.21},{"interface":"front","average":{"1min":1.087,"5min":1.2130000000000001,"15min":1.2430000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":1.5309999999999999,"5min":1.905,"15min":1.905},"last":0.89900000000000002}]},{"osd":1,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.72199999999999998},"max":{"1min":1.675,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2889999999999999,"15min":1.3160000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.046}]},{"osd":2,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2330000000000001,"15min":1.232},"min":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.611,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.069},{"interface":"front","average":{"1min":1.323,"5min":1.3859999999999999,"15min":1.3959999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8400000000000001,"5min":1.905,"15min":2.0339999999999998},"last":0.997}]},{"osd":3,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.361,"15min":1.391},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.9079999999999999,"5min":1.9530000000000001,"15min":2.0459999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.284,"15min":1.331},"min":{"1min":0.82799999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.321}]},{"osd":4,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.421,"15min":1.351},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.978,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.458,"15min":1.4119999999999999},"min":{"1min":0.81100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0830000000000002,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.4239999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3759999999999999,"15min":1.357},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.67300000000000004},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.5980000000000001,"15min":1.54},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1059999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.575}]},{"osd":7,"last update":"Mon Oct 16 15:55:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.6419999999999999,"15min":1.6040000000000001},"min":{"1min":0.86699999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.4940000000000002},"last":1.274},{"interface":"front","average":{"1min":1.599,"5min":1.6399999999999999,"15min":1.5980000000000001},"min":{"1min":0.98099999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.4690000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48420,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323420,"statfs":{"total":96636764160,"available":96587182080,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.093,"15min":1.139},"min":{"1min":0.64600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2190000000000001,"15min":1.2390000000000001},"min":{"1min":0.66400000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.885,"5min":2.738,"15min":2.738},"last":1.1779999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1859999999999999,"15min":1.216},"min":{"1min":0.71999999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":1.278},{"interface":"front","average":{"1min":1.173,"5min":1.2150000000000001,"15min":1.2789999999999999},"min":{"1min":0.75,"5min":0.443,"15min":0.443},"max":{"1min":1.579,"5min":2.137,"15min":2.137},"last":1.343}]},{"osd":2,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1779999999999999,"15min":1.2350000000000001},"min":{"1min":0.51900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.621,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.286,"5min":1.339,"15min":1.381},"min":{"1min":0.72699999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.891,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.0429999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.29,"15min":1.3260000000000001},"min":{"1min":0.81399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5509999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2749999999999999,"15min":1.359},"min":{"1min":0.63400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.5149999999999999,"5min":1.966,"15min":1.9670000000000001},"last":1.222}]},{"osd":4,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.3460000000000001},"min":{"1min":0.91900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.623,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.4670000000000001,"15min":1.5029999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7250000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.647}]},{"osd":6,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4299999999999999,"15min":1.3919999999999999},"min":{"1min":0.90700000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.077,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.425,"15min":1.429},"min":{"1min":0.84399999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6950000000000001,"5min":2.0230000000000001,"15min":2.0609999999999999},"last":1.5169999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.498,"15min":1.5269999999999999},"min":{"1min":1.246,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8320000000000001,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6319999999999999,"15min":1.6339999999999999},"min":{"1min":1.284,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2029999999999998,"5min":3.109,"15min":3.109},"last":1.73}]}]},{"osd":4,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70624,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301216,"statfs":{"total":96636764160,"available":96564445184,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1279999999999999,"15min":1.1890000000000001},"min":{"1min":0.749,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.256},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.1419999999999999,"15min":1.1830000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.079}]},{"osd":1,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.629},"max":{"1min":1.6990000000000001,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.83999999999999997,"5min":0.75900000000000001,"15min":0.70199999999999996},"max":{"1min":1.6579999999999999,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.137}]},{"osd":2,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4099999999999999,"15min":1.452},"min":{"1min":0.91800000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.8929999999999998},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.452,"15min":1.4850000000000001},"min":{"1min":0.88900000000000001,"5min":0.82499999999999996,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.2929999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.383,"15min":1.377},"min":{"1min":0.91100000000000003,"5min":0.84699999999999998,"15min":0.79400000000000004},"max":{"1min":1.915,"5min":2.23,"15min":2.3650000000000002},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3540000000000001,"15min":1.3819999999999999},"min":{"1min":0.81999999999999995,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.218}]},{"osd":5,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3979999999999999,"15min":1.3500000000000001},"min":{"1min":0.89400000000000002,"5min":0.80200000000000005,"15min":0.754},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.325},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4490000000000001},"min":{"1min":1.056,"5min":0.88900000000000001,"15min":0.74399999999999999},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":1.5189999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.427,"15min":1.4079999999999999},"min":{"1min":0.91300000000000003,"5min":0.90500000000000003,"15min":0.75900000000000001},"max":{"1min":2.129,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.347},{"interface":"front","average":{"1min":1.621,"5min":1.597,"15min":1.538},"min":{"1min":1.119,"5min":1.0089999999999999,"15min":0.82499999999999996},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.4219999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.619,"15min":1.5880000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.83999999999999997},"max":{"1min":2.2189999999999999,"5min":2.399,"15min":2.399},"last":1.724},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.6379999999999999,"15min":1.611},"min":{"1min":1.232,"5min":1.036,"15min":0.98899999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.4390000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56740,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.91800000000000004,"15min":0.96299999999999997},"min":{"1min":0.58899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.369,"5min":1.48,"15min":2.0430000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.98599999999999999,"15min":1.042},"min":{"1min":0.58899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2210000000000001,"5min":1.722,"15min":2.5990000000000002},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":1.01},"min":{"1min":0.69599999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.375,"5min":1.776,"15min":2.589},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1279999999999999,"15min":1.196},"min":{"1min":0.65300000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.6060000000000001,"5min":1.748,"15min":2.5569999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2649999999999999,"15min":1.383},"min":{"1min":0.81100000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.629,"5min":2.0899999999999999,"15min":2.302},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3740000000000001,"15min":1.478},"min":{"1min":0.76100000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":1.954,"15min":2.754},"last":0.75800000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.292,"15min":1.3480000000000001},"min":{"1min":0.92000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.958,"5min":2.0110000000000001,"15min":2.512},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3400000000000001,"15min":1.407},"min":{"1min":0.92200000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.194,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":0.89100000000000001}]},{"osd":5,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.51,"15min":1.552},"min":{"1min":0.95799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0710000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.5,"15min":1.556},"min":{"1min":1.0549999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.109,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.0740000000000001}]},{"osd":6,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.4690000000000001,"15min":1.4990000000000001},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2090000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.532,"5min":1.5589999999999999,"15min":1.5900000000000001},"min":{"1min":1.022,"5min":0.749,"15min":0.749},"max":{"1min":2.1419999999999999,"5min":2.3029999999999999,"15min":2.706},"last":1.0269999999999999}]},{"osd":7,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.575,"15min":1.6679999999999999},"min":{"1min":0.96299999999999997,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0619999999999998,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0910000000000002,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.1599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56740,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":0.92000000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.3080000000000001,"5min":1.726,"15min":1.76},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.97799999999999998,"15min":1.0780000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6739999999999999,"5min":1.804,"15min":1.8999999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97599999999999998,"15min":1.0369999999999999},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1739999999999999,"5min":1.974,"15min":1.974},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0900000000000001,"15min":1.206},"min":{"1min":0.502,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4990000000000001,"5min":1.879,"15min":1.879},"last":1.3140000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.107,"15min":1.256},"min":{"1min":0.64100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3979999999999999,"5min":1.704,"15min":2.1509999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.099,"5min":1.272,"15min":1.3460000000000001},"min":{"1min":0.629,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.569,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.1970000000000001}]},{"osd":4,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.4019999999999999,"15min":1.4610000000000001},"min":{"1min":0.79700000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0409999999999999,"5min":2.2669999999999999,"15min":2.335},"last":1.395},{"interface":"front","average":{"1min":1.266,"5min":1.3160000000000001,"15min":1.409},"min":{"1min":0.90400000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7909999999999999,"5min":1.901,"15min":2.1469999999999998},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.377,"15min":1.466},"min":{"1min":0.80100000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.008,"5min":2.3050000000000002,"15min":2.657},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.492,"15min":1.593},"min":{"1min":0.85799999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.9039999999999999,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.454}]},{"osd":6,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4299999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.881,"5min":1.9510000000000001,"15min":2.3849999999999998},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.5109999999999999,"15min":1.6060000000000001},"min":{"1min":0.91200000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.831,"5min":2.3759999999999999,"15min":2.7090000000000001},"last":1.2390000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.544,"15min":1.645},"min":{"1min":1.171,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.181,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.095},{"interface":"front","average":{"1min":1.536,"5min":1.617,"15min":1.7150000000000001},"min":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0960000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.1160000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52580,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.90900000000000003,"15min":0.93999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.351,"5min":1.49,"15min":1.8420000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98899999999999999,"15min":1.113},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6639999999999999,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.94199999999999995,"15min":1.0209999999999999},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.764,"5min":1.764,"15min":2.286},"last":0.496},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.125,"15min":1.214},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7929999999999999,"15min":1.853},"last":0.621}]},{"osd":3,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.254,"15min":1.353},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.651,"5min":2.0449999999999999,"15min":2.0619999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1879999999999999,"15min":1.2929999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.532,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":0.438}]},{"osd":4,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.319,"15min":1.365},"min":{"1min":0.89200000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.915,"5min":2.327,"15min":2.3809999999999998},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.47,"15min":1.5109999999999999},"min":{"1min":0.874,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.8759999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.52},"min":{"1min":0.86499999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9830000000000001,"5min":2.456,"15min":2.456},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.252,"5min":1.4450000000000001,"15min":1.484},"min":{"1min":0.79200000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9490000000000001,"5min":2.1589999999999998,"15min":2.4689999999999999},"last":1.014}]},{"osd":6,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.595,"15min":1.641},"min":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0489999999999999,"5min":2.3220000000000001,"15min":2.5009999999999999},"last":1.157},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5880000000000001,"15min":1.637},"min":{"1min":0.84199999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.343,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Mon Oct 16 15:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.6499999999999999,"15min":1.7310000000000001},"min":{"1min":0.79900000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1379999999999999,"5min":2.734,"15min":3.2090000000000001},"last":1.111},{"interface":"front","average":{"1min":1.381,"5min":1.5820000000000001,"15min":1.6399999999999999},"min":{"1min":0.89000000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1110000000000002,"5min":2.452,"15min":3.1539999999999999},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834576034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61480,"kb_used_data":3928,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310360,"statfs":{"total":96636764160,"available":96573808640,"internally_reserved":0,"allocated":4022272,"data_stored":2286529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88900000000000001,"15min":0.94299999999999995},"min":{"1min":0.47299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.468,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.173},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0620000000000001,"15min":1.1899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4630000000000001,"5min":2.036,"15min":2.036},"last":1.091}]},{"osd":2,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1439999999999999,"15min":1.248},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.712,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.595},{"interface":"front","average":{"1min":1,"5min":1.083,"15min":1.2},"min":{"1min":0.54200000000000004,"5min":0.439,"15min":0.439},"max":{"1min":1.5600000000000001,"5min":1.7330000000000001,"15min":1.7849999999999999},"last":1.73}]},{"osd":3,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96299999999999997,"15min":1.018},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.208,"5min":1.6339999999999999,"15min":1.6799999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1319999999999999,"15min":1.238},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6579999999999999,"5min":1.841,"15min":2.0899999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.258,"15min":1.3200000000000001},"min":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.542,"5min":1.9890000000000001,"15min":2.137},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.319,"5min":1.3660000000000001,"15min":1.4850000000000001},"min":{"1min":0.80800000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.113,"5min":2.113,"15min":2.452},"last":1.141}]},{"osd":5,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3799999999999999,"15min":1.4570000000000001},"min":{"1min":0.79300000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0550000000000002,"5min":2.1059999999999999,"15min":2.137},"last":1.363},{"interface":"front","average":{"1min":1.274,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.76800000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2,"5min":2,"15min":2.282},"last":1.046}]},{"osd":6,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5049999999999999,"15min":1.615},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.7010000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.502,"15min":1.5629999999999999},"min":{"1min":0.73899999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.534}]},{"osd":7,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.589,"15min":1.667},"min":{"1min":1.01,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1230000000000002,"5min":2.339,"15min":2.339},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.516,"15min":1.5840000000000001},"min":{"1min":0.86099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.274},"last":1.1060000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:56:22.759 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-16T14:16:18.594607+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=14, tm_min=16, tm_sec=18, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=15, tm_min=56, tm_sec=15, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T15:56:22.759 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T15:56:42.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T15:56:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:42.998+0000 7f6706f2e640 1 Processor -- start 2023-10-16T15:56:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:42.999+0000 7f6706f2e640 1 -- start start 2023-10-16T15:56:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.000+0000 7f6706f2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67000ad970 0x7f67000cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.000+0000 7f6706f2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.000+0000 7f6706f2e640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670015b9d0 con 0x7f670006c040 2023-10-16T15:56:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.000+0000 7f6706f2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670015bcb0 con 0x7f67000ad970 2023-10-16T15:56:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.000+0000 7f6706f2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670015bf90 con 0x7f67000ccfb0 2023-10-16T15:56:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67000ad970 0x7f67000cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67000ad970 0x7f67000cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49250/0 (socket says 172.21.15.29:49250) 2023-10-16T15:56:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 -- 172.21.15.29:0/986825309 learned_addr learned my addr 172.21.15.29:0/986825309 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670672d640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 -- 172.21.15.29:0/986825309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 msgr2=0x7f67000cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 -- 172.21.15.29:0/986825309 >> v1:172.21.15.170:6789/0 conn(0x7f670006c040 legacy=0x7f670006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670572b640 1 -- 172.21.15.29:0/986825309 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000cc270 con 0x7f67000ad970 2023-10-16T15:56:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.001+0000 7f670672d640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.002+0000 7f670572b640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f67000cca70 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009870 tx=0x7f66f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d984982fba4af78 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.002+0000 7f6704f2a640 1 -- 172.21.15.29:0/986825309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0031640 con 0x7f67000ad970 2023-10-16T15:56:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6704f2a640 1 -- 172.21.15.29:0/986825309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f00317a0 con 0x7f67000ad970 2023-10-16T15:56:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6704f2a640 1 -- 172.21.15.29:0/986825309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0031b30 con 0x7f67000ad970 2023-10-16T15:56:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6706f2e640 1 -- 172.21.15.29:0/986825309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 msgr2=0x7f67000cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6706f2e640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f67000cca70 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009870 tx=0x7f66f002b8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6706f2e640 1 -- 172.21.15.29:0/986825309 shutdown_connections 2023-10-16T15:56:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6706f2e640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6706f2e640 1 --2- 172.21.15.29:0/986825309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f67000cca70 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.003+0000 7f6706f2e640 1 -- 172.21.15.29:0/986825309 >> 172.21.15.29:0/986825309 conn(0x7f67000ce6a0 msgr2=0x7f67000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.004+0000 7f6706f2e640 1 -- 172.21.15.29:0/986825309 shutdown_connections 2023-10-16T15:56:43.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.004+0000 7f6706f2e640 1 -- 172.21.15.29:0/986825309 wait complete. 2023-10-16T15:56:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.005+0000 7f6706f2e640 1 Processor -- start 2023-10-16T15:56:43.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.005+0000 7f6706f2e640 1 -- start start 2023-10-16T15:56:43.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.006+0000 7f6706f2e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f670006c040 0x7f67000c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.006+0000 7f6706f2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f670015aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.006+0000 7f6706f2e640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.006+0000 7f6706f2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67000c77d0 con 0x7f67000ad970 2023-10-16T15:56:43.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.006+0000 7f6706f2e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67000c26f0 con 0x7f670006c040 2023-10-16T15:56:43.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.006+0000 7f6706f2e640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67000c7210 con 0x7f67000ccfb0 2023-10-16T15:56:43.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f670015aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f670015aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49256/0 (socket says 172.21.15.29:49256) 2023-10-16T15:56:43.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 -- 172.21.15.29:0/2246155330 learned_addr learned my addr 172.21.15.29:0/2246155330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:43.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f6705f2c640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f670006c040 0x7f67000c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670672d640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 -- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 msgr2=0x7f67000c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 -- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f670006c040 msgr2=0x7f67000c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f670006c040 0x7f67000c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670572b640 1 -- 172.21.15.29:0/2246155330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00094d0 con 0x7f67000ad970 2023-10-16T15:56:43.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.007+0000 7f670672d640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:43.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.008+0000 7f670572b640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f670015aa40 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097c0 tx=0x7f66f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.008+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f003ad20 con 0x7f67000ad970 2023-10-16T15:56:43.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.008+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f003ae80 con 0x7f67000ad970 2023-10-16T15:56:43.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.008+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0044630 con 0x7f67000ad970 2023-10-16T15:56:43.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.008+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67000c74f0 con 0x7f67000ad970 2023-10-16T15:56:43.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.008+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67000c3e80 con 0x7f67000ad970 2023-10-16T15:56:43.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.009+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670006b330 con 0x7f67000ad970 2023-10-16T15:56:43.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.010+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f66f002f050 con 0x7f67000ad970 2023-10-16T15:56:43.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.012+0000 7f66f67fc640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f66e406af90 0x7f66e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.012+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f66f00aba00 con 0x7f67000ad970 2023-10-16T15:56:43.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.012+0000 7f6705f2c640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f66e406af90 0x7f66e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.013+0000 7f6705f2c640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f66e406af90 0x7f66e406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f66fc005520 tx=0x7f66fc009040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.017+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66f0076800 con 0x7f67000ad970 2023-10-16T15:56:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.207+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 --> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67000cc550 con 0x7f66e406af90 2023-10-16T15:56:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.214+0000 7f66f67fc640 1 -- 172.21.15.29:0/2246155330 <== mgr.8771 v2:172.21.15.170:6832/1181449968 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47615 (secure 0 0 0) 0x7f66fc0051c0 con 0x7f66e406af90 2023-10-16T15:56:43.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:43.217 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-16T15:56:43.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f66e406af90 msgr2=0x7f66e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f66e406af90 0x7f66e406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f66fc005520 tx=0x7f66fc009040 comp rx=0 tx=0).stop 2023-10-16T15:56:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 msgr2=0x7f670015aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f670015aa40 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097c0 tx=0x7f66f00046c0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 shutdown_connections 2023-10-16T15:56:43.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f66e406af90 0x7f66e406d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67000ad970 0x7f670015aa40 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f67000ccfb0 0x7f67000c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 --2- 172.21.15.29:0/2246155330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f670006c040 0x7f67000c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 >> 172.21.15.29:0/2246155330 conn(0x7f67000ce6a0 msgr2=0x7f67000ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 shutdown_connections 2023-10-16T15:56:43.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.217+0000 7f6706f2e640 1 -- 172.21.15.29:0/2246155330 wait complete. 2023-10-16T15:56:43.237 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T15:56:42.831922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":456464,"kb_used_data":27944,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754518256,"statfs":{"total":773094113280,"available":772626694144,"internally_reserved":0,"allocated":28614656,"data_stored":14750024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":438685241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003475"},"pg_stats":[{"pgid":"1.0","version":"271'2810","reported_seq":3933,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-16T15:56:23.048288+0000","last_change":"2023-10-16T15:56:23.048524+0000","last_active":"2023-10-16T15:56:23.048288+0000","last_peered":"2023-10-16T15:56:23.048288+0000","last_clean":"2023-10-16T15:56:23.048288+0000","last_became_active":"2023-10-16T14:16:19.731869+0000","last_became_peered":"2023-10-16T14:16:19.731869+0000","last_unstale":"2023-10-16T15:56:23.048288+0000","last_undegraded":"2023-10-16T15:56:23.048288+0000","last_fullsized":"2023-10-16T15:56:23.048288+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'2810","last_scrub_stamp":"2023-10-16T15:56:23.048090+0000","last_deep_scrub":"271'2810","last_deep_scrub_stamp":"2023-10-16T15:56:23.048090+0000","last_clean_scrub_stamp":"2023-10-16T15:56:23.048090+0000","objects_scrubbed":2,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:03.873272+0000","scrub_duration":0.014232010999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2679,"num_read_kb":5580,"num_write":685,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57312,"kb_used_data":3928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314528,"statfs":{"total":96636764160,"available":96578076672,"internally_reserved":0,"allocated":4022272,"data_stored":2286529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0820000000000001,"15min":1.163},"min":{"1min":0.70099999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.72,"5min":1.794,"15min":1.8879999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.998,"5min":1.083,"15min":1.1339999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.486,"5min":1.796,"15min":1.796},"last":1.4239999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.105,"15min":1.1559999999999999},"min":{"1min":0.66800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.7170000000000001,"5min":1.76,"15min":1.8580000000000001},"last":1.165},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7809999999999999,"5min":1.9330000000000001,"15min":2.0569999999999999},"last":1.393}]},{"osd":2,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2969999999999999,"15min":1.3260000000000001},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.77,"5min":1.893,"15min":2.1269999999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.2230000000000001,"15min":1.2829999999999999},"min":{"1min":0.82999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7110000000000001,"5min":1.909,"15min":1.95},"last":1.046}]},{"osd":3,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2629999999999999,"15min":1.327},"min":{"1min":0.81000000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.823,"5min":1.843,"15min":1.869},"last":1.258},{"interface":"front","average":{"1min":1.385,"5min":1.387,"15min":1.4390000000000001},"min":{"1min":0.68200000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.8500000000000001,"5min":1.8720000000000001,"15min":2.1779999999999999},"last":1.462}]},{"osd":4,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.385,"15min":1.385},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.907,"5min":1.9370000000000001,"15min":2.2610000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.337},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.667,"5min":1.8740000000000001,"15min":2.0409999999999999},"last":1.113}]},{"osd":5,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.456},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":2.1469999999999998},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.502,"15min":1.552},"min":{"1min":0.67900000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3540000000000001},"last":1.534}]},{"osd":6,"last update":"Mon Oct 16 15:55:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4430000000000001,"15min":1.4770000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9330000000000001,"5min":1.9810000000000001,"15min":2.2149999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.5609999999999999,"15min":1.5800000000000001},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.998,"5min":1.998,"15min":2.2599999999999998},"last":1.3340000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52576,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1859999999999999,"15min":1.196},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.877,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.202,"15min":1.24},"min":{"1min":0.77700000000000002,"5min":0.74199999999999999,"15min":0.70999999999999996},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":1.0049999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.256,"15min":1.278},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.0270000000000001,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2969999999999999,"15min":1.3109999999999999},"min":{"1min":0.76200000000000001,"5min":0.625,"15min":0.625},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.472}]},{"osd":2,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.218,"15min":1.228},"min":{"1min":0.83199999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.601,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.375,"15min":1.3939999999999999},"min":{"1min":0.997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.5820000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3660000000000001,"15min":1.385},"min":{"1min":0.84199999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.296,"15min":1.3320000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.6819999999999999,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.5249999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.419,"15min":1.3620000000000001},"min":{"1min":0.81899999999999995,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.234,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4590000000000001,"15min":1.4139999999999999},"min":{"1min":0.82299999999999995,"5min":0.81100000000000005,"15min":0.74399999999999999},"max":{"1min":3.093,"5min":3.093,"15min":3.093},"last":1.643}]},{"osd":5,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3799999999999999,"15min":1.363},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8049999999999999,"5min":2.379,"15min":2.379},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5940000000000001,"15min":1.5429999999999999},"min":{"1min":0.95299999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":1.7130000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:56:38 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.649,"15min":1.607},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":1.784},{"interface":"front","average":{"1min":1.556,"5min":1.6240000000000001,"15min":1.597},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.72299999999999998},"max":{"1min":2.105,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.6279999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48420,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323420,"statfs":{"total":96636764160,"available":96587182080,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.093,"15min":1.139},"min":{"1min":0.64600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.466},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2190000000000001,"15min":1.2390000000000001},"min":{"1min":0.66400000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.885,"5min":2.738,"15min":2.738},"last":1.2649999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1859999999999999,"15min":1.216},"min":{"1min":0.71999999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5,"5min":1.7849999999999999,"15min":1.8140000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.173,"5min":1.2150000000000001,"15min":1.2789999999999999},"min":{"1min":0.75,"5min":0.443,"15min":0.443},"max":{"1min":1.579,"5min":2.137,"15min":2.137},"last":1.5429999999999999}]},{"osd":2,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1779999999999999,"15min":1.2350000000000001},"min":{"1min":0.51900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.621,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.02},{"interface":"front","average":{"1min":1.286,"5min":1.339,"15min":1.381},"min":{"1min":0.72699999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.891,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.617}]},{"osd":3,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.29,"15min":1.3260000000000001},"min":{"1min":0.81399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5509999999999999,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.163},{"interface":"front","average":{"1min":1.208,"5min":1.2749999999999999,"15min":1.359},"min":{"1min":0.63400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.5149999999999999,"5min":1.966,"15min":1.9670000000000001},"last":1.1339999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.3460000000000001},"min":{"1min":0.91900000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.623,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.506},{"interface":"front","average":{"1min":1.365,"5min":1.4670000000000001,"15min":1.5029999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7250000000000001,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4299999999999999,"15min":1.3919999999999999},"min":{"1min":0.90700000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.077,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.425,"15min":1.429},"min":{"1min":0.84399999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6950000000000001,"5min":2.0230000000000001,"15min":2.0609999999999999},"last":1.3300000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.498,"15min":1.5269999999999999},"min":{"1min":1.246,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8320000000000001,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.377},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6319999999999999,"15min":1.6339999999999999},"min":{"1min":1.284,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2029999999999998,"5min":3.109,"15min":3.109},"last":1.6659999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70624,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301216,"statfs":{"total":96636764160,"available":96564445184,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1279999999999999,"15min":1.1890000000000001},"min":{"1min":0.749,"5min":0.67600000000000005,"15min":0.52700000000000002},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.1419999999999999,"15min":1.1830000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.63200000000000001},"max":{"1min":1.399,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.4430000000000001}]},{"osd":1,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.629},"max":{"1min":1.6990000000000001,"5min":2.1709999999999998,"15min":2.2160000000000002},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.83999999999999997,"5min":0.75900000000000001,"15min":0.70199999999999996},"max":{"1min":1.6579999999999999,"5min":2.0510000000000002,"15min":2.2989999999999999},"last":1.232}]},{"osd":2,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4099999999999999,"15min":1.452},"min":{"1min":0.91800000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.8929999999999998},"last":1.516},{"interface":"front","average":{"1min":1.419,"5min":1.452,"15min":1.4850000000000001},"min":{"1min":0.88900000000000001,"5min":0.82499999999999996,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.3370000000000002,"15min":2.4550000000000001},"last":1.5580000000000001}]},{"osd":3,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.383,"15min":1.377},"min":{"1min":0.91100000000000003,"5min":0.84699999999999998,"15min":0.79400000000000004},"max":{"1min":1.915,"5min":2.23,"15min":2.3650000000000002},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3540000000000001,"15min":1.3819999999999999},"min":{"1min":0.81999999999999995,"5min":0.79500000000000004,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1019999999999999,"15min":2.2970000000000002},"last":1.266}]},{"osd":5,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3979999999999999,"15min":1.3500000000000001},"min":{"1min":0.89400000000000002,"5min":0.80200000000000005,"15min":0.754},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":1.038},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.4490000000000001},"min":{"1min":1.056,"5min":0.88900000000000001,"15min":0.74399999999999999},"max":{"1min":2.7069999999999999,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.97099999999999997}]},{"osd":6,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.427,"15min":1.4079999999999999},"min":{"1min":0.91300000000000003,"5min":0.90500000000000003,"15min":0.75900000000000001},"max":{"1min":2.129,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.597,"15min":1.538},"min":{"1min":1.119,"5min":1.0089999999999999,"15min":0.82499999999999996},"max":{"1min":2.762,"5min":2.762,"15min":2.762},"last":1.321}]},{"osd":7,"last update":"Mon Oct 16 15:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.619,"15min":1.5880000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.83999999999999997},"max":{"1min":2.2189999999999999,"5min":2.399,"15min":2.399},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.6379999999999999,"15min":1.611},"min":{"1min":1.232,"5min":1.036,"15min":0.98899999999999999},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":1.4810000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56740,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":54655433},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.91800000000000004,"15min":0.96299999999999997},"min":{"1min":0.58899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.369,"5min":1.48,"15min":2.0430000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.98599999999999999,"15min":1.042},"min":{"1min":0.58899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2210000000000001,"5min":1.722,"15min":2.5990000000000002},"last":1.0629999999999999}]},{"osd":1,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":1.01},"min":{"1min":0.69599999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.375,"5min":1.776,"15min":2.589},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1279999999999999,"15min":1.196},"min":{"1min":0.65300000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.6060000000000001,"5min":1.748,"15min":2.5569999999999999},"last":1.377}]},{"osd":2,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2649999999999999,"15min":1.383},"min":{"1min":0.81100000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.629,"5min":2.0899999999999999,"15min":2.302},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3740000000000001,"15min":1.478},"min":{"1min":0.76100000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":1.954,"15min":2.754},"last":1.512}]},{"osd":4,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.292,"15min":1.3480000000000001},"min":{"1min":0.92000000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.958,"5min":2.0110000000000001,"15min":2.512},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3400000000000001,"15min":1.407},"min":{"1min":0.92200000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.194,"5min":2.2170000000000001,"15min":2.4780000000000002},"last":1.425}]},{"osd":5,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.51,"15min":1.552},"min":{"1min":0.95799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.0710000000000002,"5min":2.2909999999999999,"15min":2.4729999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.5,"15min":1.556},"min":{"1min":1.0549999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.109,"5min":2.3340000000000001,"15min":2.8170000000000002},"last":1.645}]},{"osd":6,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.4690000000000001,"15min":1.4990000000000001},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2090000000000001},"last":1.262},{"interface":"front","average":{"1min":1.532,"5min":1.5589999999999999,"15min":1.5900000000000001},"min":{"1min":1.022,"5min":0.749,"15min":0.749},"max":{"1min":2.1419999999999999,"5min":2.3029999999999999,"15min":2.706},"last":1.702}]},{"osd":7,"last update":"Mon Oct 16 15:55:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.575,"15min":1.6679999999999999},"min":{"1min":0.96299999999999997,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.0619999999999998,"5min":2.1970000000000001,"15min":2.8780000000000001},"last":1.746},{"interface":"front","average":{"1min":1.587,"5min":1.587,"15min":1.6299999999999999},"min":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0910000000000002,"5min":2.4350000000000001,"15min":2.6699999999999999},"last":1.5900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56740,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":0.92000000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.3080000000000001,"5min":1.726,"15min":1.76},"last":1.3},{"interface":"front","average":{"1min":0.879,"5min":0.97799999999999998,"15min":1.0780000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6739999999999999,"5min":1.804,"15min":1.8999999999999999},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.97599999999999998,"15min":1.0369999999999999},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1739999999999999,"5min":1.974,"15min":1.974},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0900000000000001,"15min":1.206},"min":{"1min":0.502,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4990000000000001,"5min":1.879,"15min":1.879},"last":1.0429999999999999}]},{"osd":3,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.107,"15min":1.256},"min":{"1min":0.64100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3979999999999999,"5min":1.704,"15min":2.1509999999999998},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.099,"5min":1.272,"15min":1.3460000000000001},"min":{"1min":0.629,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.569,"5min":2.1339999999999999,"15min":2.5390000000000001},"last":1.4419999999999999}]},{"osd":4,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.4019999999999999,"15min":1.4610000000000001},"min":{"1min":0.79700000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.0409999999999999,"5min":2.2669999999999999,"15min":2.335},"last":1.671},{"interface":"front","average":{"1min":1.266,"5min":1.3160000000000001,"15min":1.409},"min":{"1min":0.90400000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.7909999999999999,"5min":1.901,"15min":2.1469999999999998},"last":1.125}]},{"osd":5,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.377,"15min":1.466},"min":{"1min":0.80100000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.008,"5min":2.3050000000000002,"15min":2.657},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.492,"15min":1.593},"min":{"1min":0.85799999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.9039999999999999,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.5609999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4299999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.881,"5min":1.9510000000000001,"15min":2.3849999999999998},"last":1.5},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.5109999999999999,"15min":1.6060000000000001},"min":{"1min":0.91200000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.831,"5min":2.3759999999999999,"15min":2.7090000000000001},"last":1.3740000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.544,"15min":1.645},"min":{"1min":1.171,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.181,"5min":2.3170000000000002,"15min":2.4750000000000001},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.617,"15min":1.7150000000000001},"min":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0960000000000001,"5min":2.3140000000000001,"15min":2.4039999999999999},"last":1.6120000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52580,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3428352,"data_stored":1696161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.93999999999999995},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.462,"5min":1.49,"15min":1.8420000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.97999999999999998,"15min":1.0920000000000001},"min":{"1min":0.52000000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.708,"5min":1.9099999999999999,"15min":1.9299999999999999},"last":1.1180000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.91900000000000004,"15min":1.012},"min":{"1min":0.496,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.306,"5min":1.764,"15min":2.286},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.119,"15min":1.202},"min":{"1min":0.621,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.853},"last":1.615}]},{"osd":3,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2350000000000001,"15min":1.337},"min":{"1min":0.53300000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.7010000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.1200000000000001,"15min":1.266},"min":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.613,"5min":2.1400000000000001,"15min":2.7839999999999998},"last":1.728}]},{"osd":4,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.3009999999999999,"15min":1.357},"min":{"1min":0.873,"5min":0.71999999999999997,"15min":0.50700000000000001},"max":{"1min":1.694,"5min":2.327,"15min":2.3809999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.458,"15min":1.502},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.83699999999999997},"max":{"1min":2.044,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":1.7649999999999999}]},{"osd":5,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.4319999999999999,"15min":1.502},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.58099999999999996},"max":{"1min":2.3610000000000002,"5min":2.456,"15min":2.456},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.403,"15min":1.472},"min":{"1min":0.95199999999999996,"5min":0.79200000000000004,"15min":0.70599999999999996},"max":{"1min":1.7769999999999999,"5min":2.093,"15min":2.4689999999999999},"last":1.651}]},{"osd":6,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.603,"15min":1.6379999999999999},"min":{"1min":1.024,"5min":0.84099999999999997,"15min":0.78000000000000003},"max":{"1min":2.605,"5min":2.605,"15min":2.605},"last":2.4220000000000002},{"interface":"front","average":{"1min":1.383,"5min":1.534,"15min":1.6160000000000001},"min":{"1min":0.84399999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.2309999999999999,"5min":2.6099999999999999,"15min":3.0430000000000001},"last":1.8160000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:56:28 2023","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.615,"15min":1.712},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.68500000000000005},"max":{"1min":2.4729999999999999,"5min":2.734,"15min":3.2090000000000001},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.5309999999999999,"15min":1.6240000000000001},"min":{"1min":0.877,"5min":0.877,"15min":0.66800000000000004},"max":{"1min":2.0129999999999999,"5min":2.452,"15min":3.1539999999999999},"last":1.786}]}]},{"osd":0,"up_from":13,"seq":55834576038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61472,"kb_used_data":3928,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310368,"statfs":{"total":96636764160,"available":96573816832,"internally_reserved":0,"allocated":4022272,"data_stored":2286529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88900000000000001,"15min":0.94299999999999995},"min":{"1min":0.47299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.468,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.0620000000000001,"15min":1.1899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4630000000000001,"5min":2.036,"15min":2.036},"last":1.3140000000000001}]},{"osd":2,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1439999999999999,"15min":1.248},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.712,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1,"5min":1.083,"15min":1.2},"min":{"1min":0.54200000000000004,"5min":0.439,"15min":0.439},"max":{"1min":1.5600000000000001,"5min":1.7330000000000001,"15min":1.7849999999999999},"last":1.341}]},{"osd":3,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96299999999999997,"15min":1.018},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.208,"5min":1.6339999999999999,"15min":1.6799999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1319999999999999,"15min":1.238},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6579999999999999,"5min":1.841,"15min":2.0899999999999999},"last":1.048}]},{"osd":4,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.258,"15min":1.3200000000000001},"min":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.542,"5min":1.9890000000000001,"15min":2.137},"last":1.274},{"interface":"front","average":{"1min":1.319,"5min":1.3660000000000001,"15min":1.4850000000000001},"min":{"1min":0.80800000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.113,"5min":2.113,"15min":2.452},"last":1.393}]},{"osd":5,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.3799999999999999,"15min":1.4570000000000001},"min":{"1min":0.79300000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0550000000000002,"5min":2.1059999999999999,"15min":2.137},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.3149999999999999,"15min":1.403},"min":{"1min":0.76800000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2,"5min":2,"15min":2.282},"last":1.4239999999999999}]},{"osd":6,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.5049999999999999,"15min":1.615},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":2.3940000000000001,"5min":2.3940000000000001,"15min":2.7010000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.502,"15min":1.5629999999999999},"min":{"1min":0.73899999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":2.4710000000000001},"last":1.4810000000000001}]},{"osd":7,"last update":"Mon Oct 16 15:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.589,"15min":1.667},"min":{"1min":1.01,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1230000000000002,"5min":2.339,"15min":2.339},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.516,"15min":1.5840000000000001},"min":{"1min":0.86099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.274},"last":1.514}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T15:56:43.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-16T15:56:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.446+0000 7f1c30432640 1 Processor -- start 2023-10-16T15:56:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.446+0000 7f1c30432640 1 -- start start 2023-10-16T15:56:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.447+0000 7f1c30432640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2806a4b0 0x7f1c28160980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c30432640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c2806ea40 0x7f1c280ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c30432640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c281618b0 con 0x7f1c2815ce00 2023-10-16T15:56:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c30432640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280cdc60 con 0x7f1c2806ea40 2023-10-16T15:56:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c30432640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28161010 con 0x7f1c2806a4b0 2023-10-16T15:56:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c2e9cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c2806ea40 0x7f1c280ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c2e9cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c2806ea40 0x7f1c280ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49260/0 (socket says 172.21.15.29:49260) 2023-10-16T15:56:43.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c2e9cf640 1 -- 172.21.15.29:0/2417521867 learned_addr learned my addr 172.21.15.29:0/2417521867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.448+0000 7f1c2d9cd640 1 --2- 172.21.15.29:0/2417521867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2806a4b0 0x7f1c28160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.449+0000 7f1c2e9cf640 1 -- 172.21.15.29:0/2417521867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2806a4b0 msgr2=0x7f1c28160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.449+0000 7f1c2e9cf640 1 --2- 172.21.15.29:0/2417521867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2806a4b0 0x7f1c28160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.449+0000 7f1c2e9cf640 1 -- 172.21.15.29:0/2417521867 >> v1:172.21.15.170:6789/0 conn(0x7f1c2815ce00 legacy=0x7f1c28069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.449+0000 7f1c2e9cf640 1 -- 172.21.15.29:0/2417521867 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c281612f0 con 0x7f1c2806ea40 2023-10-16T15:56:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.450+0000 7f1c2e9cf640 1 --2- 172.21.15.29:0/2417521867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c280ce080 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009870 tx=0x7f1c2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32126ca666ed524a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.450+0000 7f1c2d1cc640 1 -- 172.21.15.29:0/2417521867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20031640 con 0x7f1c2806ea40 2023-10-16T15:56:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.450+0000 7f1c2d1cc640 1 -- 172.21.15.29:0/2417521867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c200317a0 con 0x7f1c2806ea40 2023-10-16T15:56:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.450+0000 7f1c2d1cc640 1 -- 172.21.15.29:0/2417521867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20031b30 con 0x7f1c2806ea40 2023-10-16T15:56:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 -- 172.21.15.29:0/2417521867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 msgr2=0x7f1c280ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 --2- 172.21.15.29:0/2417521867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c280ce080 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009870 tx=0x7f1c2002b8f0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 -- 172.21.15.29:0/2417521867 shutdown_connections 2023-10-16T15:56:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 --2- 172.21.15.29:0/2417521867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c280ce080 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 --2- 172.21.15.29:0/2417521867 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2806a4b0 0x7f1c28160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 -- 172.21.15.29:0/2417521867 >> 172.21.15.29:0/2417521867 conn(0x7f1c280cc460 msgr2=0x7f1c280cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 -- 172.21.15.29:0/2417521867 shutdown_connections 2023-10-16T15:56:43.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.451+0000 7f1c30432640 1 -- 172.21.15.29:0/2417521867 wait complete. 2023-10-16T15:56:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.453+0000 7f1c30432640 1 Processor -- start 2023-10-16T15:56:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.453+0000 7f1c30432640 1 -- start start 2023-10-16T15:56:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c30432640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2806a4b0 0x7f1c281f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c30432640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c2815d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c30432640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2815ce00 0x7f1c2815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c30432640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281604b0 con 0x7f1c2806ea40 2023-10-16T15:56:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c30432640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281f39d0 con 0x7f1c2806a4b0 2023-10-16T15:56:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c30432640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2815db60 con 0x7f1c2815ce00 2023-10-16T15:56:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c2d9cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c2815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c2d9cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c2815d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49272/0 (socket says 172.21.15.29:49272) 2023-10-16T15:56:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.454+0000 7f1c2d9cd640 1 -- 172.21.15.29:0/2492819217 learned_addr learned my addr 172.21.15.29:0/2492819217 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2e1ce640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2806a4b0 0x7f1c281f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2e9cf640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2815ce00 0x7f1c2815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2d9cd640 1 -- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2815ce00 msgr2=0x7f1c2815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2d9cd640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2815ce00 0x7f1c2815fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2d9cd640 1 -- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2806a4b0 msgr2=0x7f1c281f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2d9cd640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2806a4b0 0x7f1c281f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2d9cd640 1 -- 172.21.15.29:0/2492819217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200094d0 con 0x7f1c2806ea40 2023-10-16T15:56:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2e1ce640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2806a4b0 0x7f1c281f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2e9cf640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2815ce00 0x7f1c2815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T15:56:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c2d9cd640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c2815d620 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00d620 tx=0x7f1c1c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0125f0 con 0x7f1c2806ea40 2023-10-16T15:56:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c012750 con 0x7f1c2806ea40 2023-10-16T15:56:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c012b00 con 0x7f1c2806ea40 2023-10-16T15:56:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.455+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2815ff90 con 0x7f1c2806ea40 2023-10-16T15:56:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.456+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c280aca30 con 0x7f1c2806ea40 2023-10-16T15:56:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.457+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2804e160 con 0x7f1c2806ea40 2023-10-16T15:56:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.458+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c1c012c60 con 0x7f1c2806ea40 2023-10-16T15:56:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.459+0000 7f1c26ffd640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1406afc0 0x7f1c1406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.459+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f1c1c089b00 con 0x7f1c2806ea40 2023-10-16T15:56:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.459+0000 7f1c2e1ce640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1406afc0 0x7f1c1406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.460+0000 7f1c2e1ce640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1406afc0 0x7f1c1406d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c180012e0 tx=0x7f1c18009040 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.463+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c1c08be60 con 0x7f1c2806ea40 2023-10-16T15:56:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.655+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f1c2815ff90 con 0x7f1c2806ea40 2023-10-16T15:56:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.663+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v94)=0 v94) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1c2815ff90 con 0x7f1c2806ea40 2023-10-16T15:56:43.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.664+0000 7f1c26ffd640 1 -- 172.21.15.29:0/2492819217 <== mon.0 v2:172.21.15.29:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1c1c054980 con 0x7f1c2806ea40 2023-10-16T15:56:43.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.666+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1406afc0 msgr2=0x7f1c1406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.666+0000 7f1c30432640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1406afc0 0x7f1c1406d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c180012e0 tx=0x7f1c18009040 comp rx=0 tx=0).stop 2023-10-16T15:56:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.666+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 msgr2=0x7f1c2815d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.666+0000 7f1c30432640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c2815d620 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00d620 tx=0x7f1c1c00dae0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 shutdown_connections 2023-10-16T15:56:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1406afc0 0x7f1c1406d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2806ea40 0x7f1c2815d620 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c2815ce00 0x7f1c2815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 --2- 172.21.15.29:0/2492819217 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2806a4b0 0x7f1c281f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 >> 172.21.15.29:0/2492819217 conn(0x7f1c280cc460 msgr2=0x7f1c280ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 shutdown_connections 2023-10-16T15:56:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.667+0000 7f1c30432640 1 -- 172.21.15.29:0/2492819217 wait complete. 2023-10-16T15:56:43.687 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T15:56:43.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T15:56:43.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.902+0000 7f1c33f1d640 1 Processor -- start 2023-10-16T15:56:43.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.902+0000 7f1c33f1d640 1 -- start start 2023-10-16T15:56:43.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c33f1d640 1 --2- >> v2:172.21.15.170:3300/0 conn(0x7f1c2c0b0b20 0x7f1c2c1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c33f1d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c15e3e0 con 0x7f1c2c0cd0c0 2023-10-16T15:56:43.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c33f1d640 1 -- --> v1:172.21.15.170:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0cd4a0 con 0x7f1c2c15d3f0 2023-10-16T15:56:43.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c33f1d640 1 -- --> v2:172.21.15.170:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2c15e100 con 0x7f1c2c0b0b20 2023-10-16T15:56:43.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c31cb9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1c2c0cd0c0 0x7f1c2c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35800/0 (socket says 172.21.15.29:35800) 2023-10-16T15:56:43.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c31cb9640 1 -- 172.21.15.29:0/984932418 learned_addr learned my addr 172.21.15.29:0/984932418 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-16T15:56:43.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.904+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485837419 0 0) 0x7f1c2c15e3e0 con 0x7f1c2c0cd0c0 2023-10-16T15:56:43.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c18003520 con 0x7f1c2c0cd0c0 2023-10-16T15:56:43.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c314b8640 1 --2- 172.21.15.29:0/984932418 >> v2:172.21.15.170:3300/0 conn(0x7f1c2c0b0b20 0x7f1c2c1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154180331 0 0) 0x7f1c18003520 con 0x7f1c2c0cd0c0 2023-10-16T15:56:43.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 >> v2:172.21.15.170:3300/0 conn(0x7f1c2c0b0b20 msgr2=0x7f1c2c1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:43.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 --2- 172.21.15.29:0/984932418 >> v2:172.21.15.170:3300/0 conn(0x7f1c2c0b0b20 0x7f1c2c1615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 >> v1:172.21.15.170:6789/0 conn(0x7f1c2c15d3f0 legacy=0x7f1c2c15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c160b10 con 0x7f1c2c0cd0c0 2023-10-16T15:56:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1172980871 0 0) 0x7f1c28001670 con 0x7f1c2c0cd0c0 2023-10-16T15:56:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 >> v1:172.21.15.29:6789/0 conn(0x7f1c2c0cd0c0 legacy=0x7f1c2c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c180039c0 0x7f1c18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180047c0 0x7f1c18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180052a0 0x7f1c1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c15e100 con 0x7f1c180039c0 2023-10-16T15:56:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1800c1c0 con 0x7f1c180052a0 2023-10-16T15:56:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1800c380 con 0x7f1c180047c0 2023-10-16T15:56:43.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.905+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c180039c0 0x7f1c18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c324ba640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180052a0 0x7f1c1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c314b8640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180047c0 0x7f1c18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c31cb9640 1 -- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180047c0 msgr2=0x7f1c18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180047c0 0x7f1c18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c31cb9640 1 -- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180052a0 msgr2=0x7f1c1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T15:56:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180052a0 0x7f1c1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c31cb9640 1 -- 172.21.15.29:0/984932418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c18003520 con 0x7f1c180039c0 2023-10-16T15:56:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c324ba640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c180052a0 0x7f1c1800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T15:56:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.906+0000 7f1c324ba640 1 -- 172.21.15.29:0/984932418 reap_dead start 2023-10-16T15:56:43.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.907+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c180039c0 0x7f1c18003de0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800c0e0 tx=0x7f1c180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d4456a4ce2f200 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28005ad0 con 0x7f1c180039c0 2023-10-16T15:56:43.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1c28005c30 con 0x7f1c180039c0 2023-10-16T15:56:43.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c30cb7640 1 -- 172.21.15.29:0/984932418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2802ca70 con 0x7f1c180039c0 2023-10-16T15:56:43.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 -- 172.21.15.29:0/984932418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c180039c0 msgr2=0x7f1c18003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c180039c0 0x7f1c18003de0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800c0e0 tx=0x7f1c180057f0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 -- 172.21.15.29:0/984932418 shutdown_connections 2023-10-16T15:56:43.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/984932418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c180039c0 0x7f1c18003de0 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 -- 172.21.15.29:0/984932418 >> 172.21.15.29:0/984932418 conn(0x7f1c2c0cb410 msgr2=0x7f1c2c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:43.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 -- 172.21.15.29:0/984932418 shutdown_connections 2023-10-16T15:56:43.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.908+0000 7f1c33f1d640 1 -- 172.21.15.29:0/984932418 wait complete. 2023-10-16T15:56:43.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.910+0000 7f1c33f1d640 1 Processor -- start 2023-10-16T15:56:43.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.910+0000 7f1c33f1d640 1 -- start start 2023-10-16T15:56:43.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c33f1d640 1 --2- >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180039c0 0x7f1c2c1f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c33f1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 0x7f1c2c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c33f1d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2c1f5df0 0x7f1c2c1f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c33f1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0c87d0 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c33f1d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1f3830 con 0x7f1c2c1f5df0 2023-10-16T15:56:43.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c33f1d640 1 -- --> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1f67b0 con 0x7f1c180039c0 2023-10-16T15:56:43.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c314b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 0x7f1c2c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.911+0000 7f1c314b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 0x7f1c2c15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51874/0 (socket says 172.21.15.29:51874) 2023-10-16T15:56:43.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 -- 172.21.15.29:0/665748748 learned_addr learned my addr 172.21.15.29:0/665748748 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-16T15:56:43.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180039c0 0x7f1c2c1f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c324ba640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2c1f5df0 0x7f1c2c1f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 -- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180039c0 msgr2=0x7f1c2c1f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180039c0 0x7f1c2c1f32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 -- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2c1f5df0 msgr2=0x7f1c2c1f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2c1f5df0 0x7f1c2c1f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 -- 172.21.15.29:0/665748748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0c7b50 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.912+0000 7f1c314b8640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 0x7f1c2c15fe60 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00f620 tx=0x7f1c1c00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.913+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c014870 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.913+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1c1c0149d0 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.913+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c014d80 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.913+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c0c7e30 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.913+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c0c83f0 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.914+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c04e160 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.915+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c1c00d490 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.916+0000 7f1c227fc640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1006afc0 0x7f1c1006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T15:56:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.916+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5831+0+0 (secure 0 0 0) 0x7f1c1c08a1d0 con 0x7f1c2c0c73c0 2023-10-16T15:56:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.917+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1006afc0 0x7f1c1006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T15:56:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.918+0000 7f1c31cb9640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1006afc0 0x7f1c1006d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c28034000 tx=0x7f1c28009030 comp rx=0 tx=0).ready entity=mgr.8771 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T15:56:43.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:43.921+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c1c0550f0 con 0x7f1c2c0c73c0 2023-10-16T15:56:44.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.156+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c2c0b4910 con 0x7f1c2c0c73c0 2023-10-16T15:56:44.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.157+0000 7f1c227fc640 1 -- 172.21.15.29:0/665748748 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+3708 (secure 0 0 0) 0x7f1c1c019e90 con 0x7f1c2c0c73c0 2023-10-16T15:56:44.159 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-16T15:56:44.159 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":143,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/845443941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":845443941},{"type":"v1","addr":"172.21.15.170:6835","nonce":845443941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":135},{"gid":9712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6834/2049743874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2049743874},{"type":"v1","addr":"172.21.15.29:6834","nonce":2049743874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9718,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/1894368152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1894368152},{"type":"v1","addr":"172.21.15.170:6837","nonce":1894368152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":12738,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3797189783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3797189783},{"type":"v1","addr":"172.21.15.29:6835","nonce":3797189783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[]} 2023-10-16T15:56:44.159 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 143 2023-10-16T15:56:44.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1006afc0 msgr2=0x7f1c1006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1006afc0 0x7f1c1006d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c28034000 tx=0x7f1c28009030 comp rx=0 tx=0).stop 2023-10-16T15:56:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 msgr2=0x7f1c2c15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T15:56:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 0x7f1c2c15fe60 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00f620 tx=0x7f1c1c00fae0 comp rx=0 tx=0).stop 2023-10-16T15:56:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 shutdown_connections 2023-10-16T15:56:44.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:6832/1181449968,v1:172.21.15.170:6833/1181449968] conn(0x7f1c1006afc0 0x7f1c1006d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c2c0c73c0 0x7f1c2c15fe60 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c2c1f5df0 0x7f1c2c1f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 --2- 172.21.15.29:0/665748748 >> [v2:172.21.15.170:3301/0,v1:172.21.15.170:6790/0] conn(0x7f1c180039c0 0x7f1c2c1f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T15:56:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.159+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 >> 172.21.15.29:0/665748748 conn(0x7f1c2c0cb410 msgr2=0x7f1c2c15b110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T15:56:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.160+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 shutdown_connections 2023-10-16T15:56:44.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-16T15:56:44.160+0000 7f1c33f1d640 1 -- 172.21.15.29:0/665748748 wait complete. 2023-10-16T15:56:44.181 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-16T15:56:44.181 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-16T15:56:44.181 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T15:56:44.182 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.200 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T15:56:44.200 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T15:56:44.201 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.272 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T15:56:44.272 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T15:56:44.273 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.305 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T15:56:44.305 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T15:56:44.306 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.372 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T15:56:44.373 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-16T15:56:44.373 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-16T15:56:44.374 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.462 INFO:tasks.ceph.osd.0:Stopped 2023-10-16T15:56:44.463 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-16T15:56:44.463 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.552 INFO:tasks.ceph.osd.1:Stopped 2023-10-16T15:56:44.553 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-16T15:56:44.553 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.623 INFO:tasks.ceph.osd.2:Stopped 2023-10-16T15:56:44.623 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-16T15:56:44.624 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.677 INFO:tasks.ceph.osd.3:Stopped 2023-10-16T15:56:44.678 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-16T15:56:44.678 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.758 INFO:tasks.ceph.osd.4:Stopped 2023-10-16T15:56:44.758 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-16T15:56:44.759 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.840 INFO:tasks.ceph.osd.5:Stopped 2023-10-16T15:56:44.840 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-16T15:56:44.841 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.920 INFO:tasks.ceph.osd.6:Stopped 2023-10-16T15:56:44.921 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-16T15:56:44.921 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:44.982 INFO:tasks.ceph.osd.7:Stopped 2023-10-16T15:56:44.982 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-16T15:56:44.982 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T15:56:44.983 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:48.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:48.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:49.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2023-10-16T15:56:51.084 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T15:56:51.085 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T15:56:51.085 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:51.171 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T15:56:51.171 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-16T15:56:51.171 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-16T15:56:51.172 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:51.243 INFO:tasks.ceph.mon.a:Stopped 2023-10-16T15:56:51.243 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-16T15:56:51.243 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:51.299 INFO:tasks.ceph.mon.b:Stopped 2023-10-16T15:56:51.299 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-16T15:56:51.300 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T15:56:51.359 INFO:tasks.ceph.mon.c:Stopped 2023-10-16T15:56:51.360 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-16T15:56:51.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T15:56:51.475 INFO:teuthology.orchestra.run.smithi029.stdout:2023-10-16T14:31:33.181255+0000 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-10-16T15:56:51.476 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-16T15:56:51.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T15:56:51.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T15:56:51.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T15:56:51.735 INFO:teuthology.orchestra.run.smithi029.stdout:2023-10-16T14:31:33.181255+0000 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-10-16T15:56:51.736 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:56:51.737 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-16T15:56:51.982 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:56:51.983 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-16T15:56:52.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:56:52.157 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-16T15:56:52.332 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:56:52.332 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-16T15:56:52.522 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T15:56:52.523 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-16T15:56:52.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T15:56:52.864 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-16T15:56:53.022 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T15:56:53.022 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-16T15:56:53.182 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T15:56:53.182 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-16T15:56:53.393 INFO:tasks.ceph:Archiving mon data... 2023-10-16T15:56:53.394 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/data/mon.a.tgz 2023-10-16T15:56:53.395 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-10-16T15:56:53.456 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.Bzu1aHqHh3 2023-10-16T15:56:53.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Bzu1aHqHh3 2023-10-16T15:56:54.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.Bzu1aHqHh3 2023-10-16T15:56:54.355 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.Bzu1aHqHh3 is 4MB 2023-10-16T15:56:54.524 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.Bzu1aHqHh3 2023-10-16T15:56:54.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2023-10-16T15:56:54.590 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/data/mon.b.tgz 2023-10-16T15:56:54.590 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-16T15:56:54.651 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.mqT7mqbwjD 2023-10-16T15:56:54.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.mqT7mqbwjD 2023-10-16T15:56:55.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.mqT7mqbwjD 2023-10-16T15:56:55.484 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.mqT7mqbwjD is 4MB 2023-10-16T15:56:55.650 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.mqT7mqbwjD 2023-10-16T15:56:55.711 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/data/mon.c.tgz 2023-10-16T15:56:55.712 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-16T15:56:55.764 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.jWOWYIedOW 2023-10-16T15:56:55.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.jWOWYIedOW 2023-10-16T15:56:56.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.jWOWYIedOW 2023-10-16T15:56:56.579 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.jWOWYIedOW is 4MB 2023-10-16T15:56:56.745 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.jWOWYIedOW 2023-10-16T15:56:56.806 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-16T15:56:56.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T15:56:56.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T15:56:56.918 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-16T15:56:56.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T15:56:56.927 DEBUG:teuthology.orchestra.run.smithi170:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T15:56:57.030 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T15:56:57.034 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T15:56:57.047 INFO:tasks.ceph:Archiving crash dumps... 2023-10-16T15:56:57.049 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/remote/smithi029/crash 2023-10-16T15:56:57.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T15:56:57.139 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/remote/smithi170/crash 2023-10-16T15:56:57.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T15:56:57.242 INFO:tasks.ceph:Shutting down logrotate 2023-10-16T15:56:57.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:57.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:57.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:57.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T15:56:57.591 INFO:tasks.ceph:Compressing logs... 2023-10-16T15:56:57.591 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T15:56:57.600 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T15:56:57.694 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-16T15:56:57.695 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87485.log 2023-10-16T15:56:57.696 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87725.log 2023-10-16T15:56:57.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87485.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87485.log.gz 2023-10-16T15:56:57.698 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T15:56:57.698 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134075.log 2023-10-16T15:56:57.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87725.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-16T15:56:57.699 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87725.log.gz 2023-10-16T15:56:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91696.log 2023-10-16T15:56:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131603.log 2023-10-16T15:56:57.701 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.134075.log.gz 2023-10-16T15:56:57.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87645.log 2023-10-16T15:56:57.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-16T15:56:57.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87645.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87645.log.gz 2023-10-16T15:56:57.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.92297.log 2023-10-16T15:56:57.704 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91696.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91696.log.gz 2023-10-16T15:56:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148124.log 2023-10-16T15:56:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131603.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.131603.log.gz 2023-10-16T15:56:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125362.log 2023-10-16T15:56:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134742.log 2023-10-16T15:56:57.707 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.148124.log.gz 2023-10-16T15:56:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125362.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125362.log.gz 2023-10-16T15:56:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115514.log 2023-10-16T15:56:57.707 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log/var/log/ceph/ceph-osd.4.log: 2023-10-16T15:56:57.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.1.92297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.92297.log.gz 2023-10-16T15:56:57.708 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-16T15:56:57.709 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.7.log: -- /var/log/ceph/ceph-client.admin.87445.log 2023-10-16T15:56:57.710 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144850.log 2023-10-16T15:56:57.710 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134742.log.gz 2023-10-16T15:56:57.710 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115514.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115514.log.gz 2023-10-16T15:56:57.710 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139337.log 2023-10-16T15:56:57.711 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144850.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144850.log.gz 2023-10-16T15:56:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145944.log 2023-10-16T15:56:57.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87685.log 2023-10-16T15:56:57.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87445.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87445.log.gz 2023-10-16T15:56:57.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87605.log 2023-10-16T15:56:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139337.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139337.log.gz 2023-10-16T15:56:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157116.log 2023-10-16T15:56:57.715 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91178.log 2023-10-16T15:56:57.716 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145944.log.gz 2023-10-16T15:56:57.716 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109687.log 2023-10-16T15:56:57.716 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.157116.log.gz 2023-10-16T15:56:57.717 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87685.log.gz 2023-10-16T15:56:57.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.92684.log 2023-10-16T15:56:57.717 INFO:teuthology.orchestra.run.smithi170.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T15:56:57.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87605.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87605.log.gz 2023-10-16T15:56:57.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.3.92684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.92684.log.gz 2023-10-16T15:56:57.718 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-16T15:56:57.720 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87525.log 2023-10-16T15:56:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91178.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91178.log.gz 2023-10-16T15:56:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97948.log 2023-10-16T15:56:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140467.log 2023-10-16T15:56:57.722 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109687.log.gz 2023-10-16T15:56:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153472.log 2023-10-16T15:56:57.722 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97948.log.gz 2023-10-16T15:56:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91906.log 2023-10-16T15:56:57.723 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.140467.log.gz 2023-10-16T15:56:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140414.log 2023-10-16T15:56:57.723 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.153472.log.gz 2023-10-16T15:56:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109290.log 2023-10-16T15:56:57.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T15:56:57.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87525.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87525.log.gz 2023-10-16T15:56:57.726 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91906.log.gz 2023-10-16T15:56:57.726 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147893.log 2023-10-16T15:56:57.726 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.140414.log.gz 2023-10-16T15:56:57.726 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109290.log: gzip -5 88.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.109290.log.gz -- 2023-10-16T15:56:57.727 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.95551.log 2023-10-16T15:56:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112669.log 2023-10-16T15:56:57.727 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147893.log.gz 2023-10-16T15:56:57.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.92688.log 2023-10-16T15:56:57.729 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-16T15:56:57.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.3.92688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.92688.log.gz 2023-10-16T15:56:57.729 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87565.log 2023-10-16T15:56:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97768.log 2023-10-16T15:56:57.730 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95551.log.gz 2023-10-16T15:56:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145734.log 2023-10-16T15:56:57.731 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112669.log.gz 2023-10-16T15:56:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152727.log 2023-10-16T15:56:57.731 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97768.log.gz 2023-10-16T15:56:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119176.log 2023-10-16T15:56:57.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.87565.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87565.log.gz 2023-10-16T15:56:57.733 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145734.log.gz 2023-10-16T15:56:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150987.log 2023-10-16T15:56:57.733 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152727.log.gz 2023-10-16T15:56:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105430.log 2023-10-16T15:56:57.734 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119176.log.gz 2023-10-16T15:56:57.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.92288.log 2023-10-16T15:56:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109855.log 2023-10-16T15:56:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105430.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.150987.log.gz 2023-10-16T15:56:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-16T15:56:57.736 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105430.log.gz 2023-10-16T15:56:57.737 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-16T15:56:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145394.log 2023-10-16T15:56:57.738 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109855.log.gz 2023-10-16T15:56:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125831.log 2023-10-16T15:56:57.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.1.92288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.92288.log.gz 2023-10-16T15:56:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145394.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145394.log.gz 2023-10-16T15:56:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122778.log 2023-10-16T15:56:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125831.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125831.log.gz 2023-10-16T15:56:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143595.log 2023-10-16T15:56:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106499.log 2023-10-16T15:56:57.742 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122778.log.gz 2023-10-16T15:56:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143595.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143595.log.gz 2023-10-16T15:56:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104421.log 2023-10-16T15:56:57.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-16T15:56:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106499.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106499.log.gz 2023-10-16T15:56:57.745 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144934.log 2023-10-16T15:56:57.745 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106139.log 2023-10-16T15:56:57.746 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104421.log.gz 2023-10-16T15:56:57.746 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139745.log 2023-10-16T15:56:57.747 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.144934.log.gz 2023-10-16T15:56:57.748 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92632.log 2023-10-16T15:56:57.748 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106139.log.gz 2023-10-16T15:56:57.750 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96094.log 2023-10-16T15:56:57.750 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.139745.log.gz 2023-10-16T15:56:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92632.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92632.log.gz 2023-10-16T15:56:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.133031.log 2023-10-16T15:56:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122386.log 2023-10-16T15:56:57.752 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96094.log.gz 2023-10-16T15:56:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.133031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.133031.log.gz 2023-10-16T15:56:57.754 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93228.log 2023-10-16T15:56:57.755 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122386.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.122386.log.gz -5 2023-10-16T15:56:57.755 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.97060.log 2023-10-16T15:56:57.756 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139703.log 2023-10-16T15:56:57.757 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93228.log.gz 2023-10-16T15:56:57.757 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115472.log 2023-10-16T15:56:57.758 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97060.log.gz 2023-10-16T15:56:57.759 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.c.log: 94.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T15:56:57.759 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96226.log 2023-10-16T15:56:57.760 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139703.log.gz 2023-10-16T15:56:57.760 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87254.log 2023-10-16T15:56:57.760 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115472.log.gz 2023-10-16T15:56:57.761 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96226.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96226.log.gz 2023-10-16T15:56:57.762 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97810.log 2023-10-16T15:56:57.763 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101046.log 2023-10-16T15:56:57.763 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87254.log.gz 2023-10-16T15:56:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124744.log 2023-10-16T15:56:57.764 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97810.log.gz 2023-10-16T15:56:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101046.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101046.log.gz 2023-10-16T15:56:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105687.log 2023-10-16T15:56:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121063.log 2023-10-16T15:56:57.767 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124744.log.gz 2023-10-16T15:56:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140355.log 2023-10-16T15:56:57.768 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105687.log.gz 2023-10-16T15:56:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121063.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121063.log.gz 2023-10-16T15:56:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115178.log 2023-10-16T15:56:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110468.log 2023-10-16T15:56:57.770 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140355.log.gz 2023-10-16T15:56:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143805.log 2023-10-16T15:56:57.771 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115178.log.gz 2023-10-16T15:56:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110468.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110468.log.gz 2023-10-16T15:56:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126575.log 2023-10-16T15:56:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143805.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143805.log.gz 2023-10-16T15:56:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148082.log 2023-10-16T15:56:57.775 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126389.log 2023-10-16T15:56:57.775 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126575.log.gz 2023-10-16T15:56:57.776 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102503.log 2023-10-16T15:56:57.776 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.148082.log.gz 2023-10-16T15:56:57.777 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126389.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.126389.log.gz 2023-10-16T15:56:57.777 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136857.log 2023-10-16T15:56:57.778 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.94482.log 2023-10-16T15:56:57.779 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102503.log.gz 2023-10-16T15:56:57.779 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142925.log 2023-10-16T15:56:57.780 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.136857.log.gz 2023-10-16T15:56:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.2.94482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.94482.log.gz 2023-10-16T15:56:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145520.log 2023-10-16T15:56:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91524.log 2023-10-16T15:56:57.782 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142925.log.gz 2023-10-16T15:56:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130597.log 2023-10-16T15:56:57.783 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145520.log.gz 2023-10-16T15:56:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91524.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91524.log.gz 2023-10-16T15:56:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112021.log 2023-10-16T15:56:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137736.log 2023-10-16T15:56:57.786 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130597.log.gz 2023-10-16T15:56:57.787 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146974.log 2023-10-16T15:56:57.787 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112021.log.gz 2023-10-16T15:56:57.788 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132690.log 2023-10-16T15:56:57.788 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.137736.log.gz 2023-10-16T15:56:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146974.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146974.log.gz 2023-10-16T15:56:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139787.log 2023-10-16T15:56:57.790 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122262.log 2023-10-16T15:56:57.791 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.132690.log.gz 2023-10-16T15:56:57.791 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144247.log 2023-10-16T15:56:57.792 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.139787.log.gz 2023-10-16T15:56:57.793 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136899.log 2023-10-16T15:56:57.793 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122262.log.gz 2023-10-16T15:56:57.794 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144247.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.144247.log.gz 2023-10-16T15:56:57.794 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115220.log 2023-10-16T15:56:57.795 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98454.log 2023-10-16T15:56:57.795 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136899.log.gz 2023-10-16T15:56:57.796 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137956.log 2023-10-16T15:56:57.797 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.115220.log.gz 2023-10-16T15:56:57.797 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130639.log 2023-10-16T15:56:57.798 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98454.log.gz 2023-10-16T15:56:57.799 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137956.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137956.log.gz 2023-10-16T15:56:57.799 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87383.log 2023-10-16T15:56:57.800 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152433.log 2023-10-16T15:56:57.800 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130639.log.gz 2023-10-16T15:56:57.801 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143019.log 2023-10-16T15:56:57.801 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87383.log.gz 2023-10-16T15:56:57.802 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135347.log 2023-10-16T15:56:57.802 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152433.log.gz 2023-10-16T15:56:57.803 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143019.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143019.log.gz 2023-10-16T15:56:57.804 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146542.log 2023-10-16T15:56:57.805 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132302.log 2023-10-16T15:56:57.805 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.135347.log.gz 2023-10-16T15:56:57.806 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92116.log 2023-10-16T15:56:57.806 INFO:teuthology.orchestra.run.smithi170.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-16T15:56:57.807 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146542.log.gz 2023-10-16T15:56:57.807 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132302.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.132302.log.gz 2023-10-16T15:56:57.807 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145608.log 2023-10-16T15:56:57.808 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156792.log 2023-10-16T15:56:57.808 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92116.log.gz 2023-10-16T15:56:57.809 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149485.log 2023-10-16T15:56:57.810 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145608.log.gz 2023-10-16T15:56:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135209.log 2023-10-16T15:56:57.811 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156792.log.gz 2023-10-16T15:56:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149485.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.149485.log.gz 2023-10-16T15:56:57.812 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106265.log 2023-10-16T15:56:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104294.log 2023-10-16T15:56:57.813 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135209.log.gz 2023-10-16T15:56:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116549.log 2023-10-16T15:56:57.815 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106265.log.gz 2023-10-16T15:56:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126473.log 2023-10-16T15:56:57.816 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.104294.log.gz 2023-10-16T15:56:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116549.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.116549.log.gz 2023-10-16T15:56:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101663.log 2023-10-16T15:56:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126473.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.126473.log.gz /var/log/ceph/ceph-client.admin.117459.log 2023-10-16T15:56:57.818 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135167.log 2023-10-16T15:56:57.819 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101663.log.gz 2023-10-16T15:56:57.820 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151619.log 2023-10-16T15:56:57.820 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117459.log.gz 2023-10-16T15:56:57.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135167.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135167.log.gz 2023-10-16T15:56:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112837.log 2023-10-16T15:56:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148628.log 2023-10-16T15:56:57.823 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151619.log.gz 2023-10-16T15:56:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131477.log 2023-10-16T15:56:57.824 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112837.log.gz 2023-10-16T15:56:57.825 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148628.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148628.log.gz 2023-10-16T15:56:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152144.log 2023-10-16T15:56:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138429.log 2023-10-16T15:56:57.827 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.131477.log.gz 2023-10-16T15:56:57.828 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96976.log 2023-10-16T15:56:57.828 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152144.log.gz 2023-10-16T15:56:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138429.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.138429.log.gz 2023-10-16T15:56:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112291.log 2023-10-16T15:56:57.830 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133906.log 2023-10-16T15:56:57.831 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96976.log.gz 2023-10-16T15:56:57.831 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148502.log 2023-10-16T15:56:57.832 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112291.log.gz 2023-10-16T15:56:57.833 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133906.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133906.log.gz 2023-10-16T15:56:57.833 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157158.log 2023-10-16T15:56:57.834 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148502.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80928.log 2023-10-16T15:56:57.834 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.148502.log.gz 2023-10-16T15:56:57.835 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146668.log 2023-10-16T15:56:57.835 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157158.log.gz 2023-10-16T15:56:57.836 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.80928.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80928.log.gz 2023-10-16T15:56:57.836 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150607.log 2023-10-16T15:56:57.837 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105094.log 2023-10-16T15:56:57.838 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.146668.log.gz 2023-10-16T15:56:57.838 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120178.log 2023-10-16T15:56:57.839 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.150607.log.gz 2023-10-16T15:56:57.839 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105094.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105094.log.gz 2023-10-16T15:56:57.840 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131122.log 2023-10-16T15:56:57.841 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99910.log 2023-10-16T15:56:57.841 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.120178.log.gz 2023-10-16T15:56:57.842 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.106686.log 2023-10-16T15:56:57.842 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131122.log.gz 2023-10-16T15:56:57.843 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99910.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99910.log.gz 2023-10-16T15:56:57.843 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147499.log 2023-10-16T15:56:57.844 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.106686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.106686.log.gz 2023-10-16T15:56:57.845 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147331.log 2023-10-16T15:56:57.846 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131350.log 2023-10-16T15:56:57.846 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147499.log.gz 2023-10-16T15:56:57.847 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144418.log 2023-10-16T15:56:57.847 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.147331.log.gz 2023-10-16T15:56:57.848 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131350.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131350.log.gz 2023-10-16T15:56:57.848 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147184.log 2023-10-16T15:56:57.849 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113476.log 2023-10-16T15:56:57.850 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.144418.log.gz 2023-10-16T15:56:57.850 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149179.log 2023-10-16T15:56:57.851 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.147184.log.gz 2023-10-16T15:56:57.852 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121691.log 2023-10-16T15:56:57.852 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113476.log.gz 2023-10-16T15:56:57.853 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149179.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.109500.log -- replaced with /var/log/ceph/ceph-client.admin.149179.log.gz 2023-10-16T15:56:57.853 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:57.854 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121691.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.121691.log.gz 2023-10-16T15:56:57.854 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.120979.log 2023-10-16T15:56:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111874.log 2023-10-16T15:56:57.856 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109500.log.gz 2023-10-16T15:56:57.856 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94825.log 2023-10-16T15:56:57.857 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120979.log.gz 2023-10-16T15:56:57.857 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111874.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111874.log.gz 2023-10-16T15:56:57.858 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109416.log 2023-10-16T15:56:57.859 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121859.log 2023-10-16T15:56:57.859 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94825.log.gz 2023-10-16T15:56:57.860 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143198.log 2023-10-16T15:56:57.860 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109416.log.gz 2023-10-16T15:56:57.861 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121859.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121859.log.gz 2023-10-16T15:56:57.861 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135297.log 2023-10-16T15:56:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155340.log 2023-10-16T15:56:57.863 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143198.log.gz 2023-10-16T15:56:57.864 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101105.log 2023-10-16T15:56:57.864 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135297.log.gz 2023-10-16T15:56:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155340.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.155340.log.gz 2023-10-16T15:56:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92850.log 2023-10-16T15:56:57.866 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111040.log 2023-10-16T15:56:57.867 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101105.log.gz 2023-10-16T15:56:57.868 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132045.log 2023-10-16T15:56:57.868 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92850.log.gz 2023-10-16T15:56:57.868 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111040.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111040.log.gz 2023-10-16T15:56:57.869 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96268.log 2023-10-16T15:56:57.870 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93396.log 2023-10-16T15:56:57.871 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.132045.log.gz 2023-10-16T15:56:57.871 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-16T15:56:57.872 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96268.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96268.log.gz 2023-10-16T15:56:57.872 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123980.log 2023-10-16T15:56:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93702.log 2023-10-16T15:56:57.873 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93396.log.gz 2023-10-16T15:56:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.116970.log 2023-10-16T15:56:57.874 INFO:teuthology.orchestra.run.smithi029.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.123980.log.gz 2023-10-16T15:56:57.875 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93702.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93702.log.gz 2023-10-16T15:56:57.875 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102783.log 2023-10-16T15:56:57.876 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.116970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.116970.log.gz 2023-10-16T15:56:57.876 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155571.log 2023-10-16T15:56:57.878 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96808.log 2023-10-16T15:56:57.878 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102783.log.gz 2023-10-16T15:56:57.879 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130905.log 2023-10-16T15:56:57.879 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.155571.log.gz 2023-10-16T15:56:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96808.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96808.log.gz 2023-10-16T15:56:57.880 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143324.log 2023-10-16T15:56:57.881 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146626.log 2023-10-16T15:56:57.881 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130905.log.gz 2023-10-16T15:56:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153251.log 2023-10-16T15:56:57.883 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.143324.log.gz 2023-10-16T15:56:57.884 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146626.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146626.log.gz 2023-10-16T15:56:57.884 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143450.log 2023-10-16T15:56:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153251.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153251.log.gz 2023-10-16T15:56:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143679.log 2023-10-16T15:56:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150259.log 2023-10-16T15:56:57.887 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143450.log.gz 2023-10-16T15:56:57.887 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116149.log 2023-10-16T15:56:57.888 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143679.log.gz 2023-10-16T15:56:57.888 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150259.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150259.log.gz 2023-10-16T15:56:57.889 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99658.log 2023-10-16T15:56:57.890 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104863.log 2023-10-16T15:56:57.890 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116149.log.gz 2023-10-16T15:56:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119694.log 2023-10-16T15:56:57.891 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99658.log.gz 2023-10-16T15:56:57.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104863.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.104863.log.gz 2023-10-16T15:56:57.893 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120070.log 2023-10-16T15:56:57.894 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98094.log 2023-10-16T15:56:57.894 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119694.log.gz 2023-10-16T15:56:57.895 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115598.log 2023-10-16T15:56:57.896 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120070.log.gz 2023-10-16T15:56:57.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98094.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98094.log.gz 2023-10-16T15:56:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135433.log 2023-10-16T15:56:57.898 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134033.log 2023-10-16T15:56:57.898 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.115598.log.gz 2023-10-16T15:56:57.899 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146070.log 2023-10-16T15:56:57.899 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.135433.log.gz 2023-10-16T15:56:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134033.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134033.log.gz 2023-10-16T15:56:57.900 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123329.log 2023-10-16T15:56:57.901 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150735.log 2023-10-16T15:56:57.901 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146070.log.gz 2023-10-16T15:56:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.123329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.123329.log.gz 2023-10-16T15:56:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91948.log 2023-10-16T15:56:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124021.log 2023-10-16T15:56:57.904 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.150735.log.gz 2023-10-16T15:56:57.905 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135391.log 2023-10-16T15:56:57.905 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91948.log.gz 2023-10-16T15:56:57.906 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124021.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.124021.log.gz 2023-10-16T15:56:57.906 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100869.log 2023-10-16T15:56:57.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122470.log 2023-10-16T15:56:57.907 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.135391.log.gz 2023-10-16T15:56:57.908 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102223.log 2023-10-16T15:56:57.909 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100869.log.gz 2023-10-16T15:56:57.909 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142545.log 2023-10-16T15:56:57.910 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122470.log.gz 2023-10-16T15:56:57.910 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102223.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102223.log.gz 2023-10-16T15:56:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92627.log 2023-10-16T15:56:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155865.log 2023-10-16T15:56:57.912 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.142545.log.gz 2023-10-16T15:56:57.913 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133949.log 2023-10-16T15:56:57.913 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92627.log.gz 2023-10-16T15:56:57.914 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155865.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.155865.log.gz -5 2023-10-16T15:56:57.914 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.151450.log 2023-10-16T15:56:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133949.log: gzip -5 --verbose 88.0% -- /var/log/ceph/ceph-client.admin.151661.log -- replaced with /var/log/ceph/ceph-client.admin.133949.log.gz 2023-10-16T15:56:57.916 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143510.log 2023-10-16T15:56:57.917 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151450.log.gz 2023-10-16T15:56:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99532.log 2023-10-16T15:56:57.918 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151661.log.gz 2023-10-16T15:56:57.919 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121252.log 2023-10-16T15:56:57.919 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143510.log.gz 2023-10-16T15:56:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99532.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99532.log.gz 2023-10-16T15:56:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123448.log 2023-10-16T15:56:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154026.log 2023-10-16T15:56:57.922 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121252.log.gz 2023-10-16T15:56:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116237.log 2023-10-16T15:56:57.923 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123448.log.gz 2023-10-16T15:56:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154026.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154026.log.gz 2023-10-16T15:56:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119568.log 2023-10-16T15:56:57.925 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104516.log 2023-10-16T15:56:57.926 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116237.log.gz 2023-10-16T15:56:57.926 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119568.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119568.log.gz 2023-10-16T15:56:57.927 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149571.log 2023-10-16T15:56:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104516.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104516.log.gz 2023-10-16T15:56:57.928 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104378.log 2023-10-16T15:56:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96556.log 2023-10-16T15:56:57.930 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149571.log.gz 2023-10-16T15:56:57.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104378.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104378.log.gz 2023-10-16T15:56:57.931 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95341.log 2023-10-16T15:56:57.932 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96556.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.96556.log.gz 2023-10-16T15:56:57.932 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126305.log 2023-10-16T15:56:57.933 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95341.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95341.log.gz 2023-10-16T15:56:57.933 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104905.log 2023-10-16T15:56:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126305.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126305.log.gz 2023-10-16T15:56:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96052.log 2023-10-16T15:56:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153982.log 2023-10-16T15:56:57.936 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104905.log.gz 2023-10-16T15:56:57.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143949.log 2023-10-16T15:56:57.937 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96052.log.gz 2023-10-16T15:56:57.938 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148250.log 2023-10-16T15:56:57.938 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.153982.log.gz 2023-10-16T15:56:57.939 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96850.log 2023-10-16T15:56:57.939 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143949.log.gz 2023-10-16T15:56:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148250.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.148250.log.gz 2023-10-16T15:56:57.940 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115304.log 2023-10-16T15:56:57.941 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106767.log 2023-10-16T15:56:57.941 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96850.log.gz 2023-10-16T15:56:57.942 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114926.log 2023-10-16T15:56:57.943 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115304.log.gz 2023-10-16T15:56:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106767.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106767.log.gz 2023-10-16T15:56:57.944 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91864.log 2023-10-16T15:56:57.945 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156835.log 2023-10-16T15:56:57.945 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114926.log.gz 2023-10-16T15:56:57.946 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132214.log 2023-10-16T15:56:57.946 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91864.log.gz 2023-10-16T15:56:57.947 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134700.log 2023-10-16T15:56:57.947 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.156835.log.gz 2023-10-16T15:56:57.948 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135040.log 2023-10-16T15:56:57.948 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.132214.log.gz 2023-10-16T15:56:57.949 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134700.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.134700.log.gz -5 2023-10-16T15:56:57.949 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.150343.log 2023-10-16T15:56:57.950 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135040.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.135040.log.gz 2023-10-16T15:56:57.951 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124162.log 2023-10-16T15:56:57.951 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122182.log 2023-10-16T15:56:57.952 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150343.log.gz 2023-10-16T15:56:57.952 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137778.log 2023-10-16T15:56:57.953 INFO:teuthology.orchestra.run.smithi029.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.124162.log.gz 2023-10-16T15:56:57.954 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96934.log 2023-10-16T15:56:57.954 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122182.log.gz 2023-10-16T15:56:57.955 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137778.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.137778.log.gz 2023-10-16T15:56:57.955 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97308.log 2023-10-16T15:56:57.956 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115346.log 2023-10-16T15:56:57.956 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96934.log.gz 2023-10-16T15:56:57.957 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119314.log 2023-10-16T15:56:57.957 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97308.log.gz 2023-10-16T15:56:57.958 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126125.log 2023-10-16T15:56:57.958 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115346.log.gz 2023-10-16T15:56:57.959 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119314.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119314.log.gz 2023-10-16T15:56:57.959 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101341.log 2023-10-16T15:56:57.960 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97558.log 2023-10-16T15:56:57.961 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126125.log.gz 2023-10-16T15:56:57.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115094.log 2023-10-16T15:56:57.962 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101341.log.gz 2023-10-16T15:56:57.963 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145818.log 2023-10-16T15:56:57.963 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.97558.log.gz 2023-10-16T15:56:57.964 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114450.log 2023-10-16T15:56:57.964 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115094.log.gz 2023-10-16T15:56:57.965 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118103.log 2023-10-16T15:56:57.965 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145818.log.gz 2023-10-16T15:56:57.966 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113176.log 2023-10-16T15:56:57.966 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114450.log.gz 2023-10-16T15:56:57.967 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120683.log 2023-10-16T15:56:57.967 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118103.log.gz 2023-10-16T15:56:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113176.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113176.log.gz 2023-10-16T15:56:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97474.log 2023-10-16T15:56:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124839.log 2023-10-16T15:56:57.970 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120683.log.gz 2023-10-16T15:56:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148586.log 2023-10-16T15:56:57.971 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97474.log.gz 2023-10-16T15:56:57.972 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150819.log 2023-10-16T15:56:57.972 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124839.log.gz 2023-10-16T15:56:57.973 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114611.log 2023-10-16T15:56:57.973 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148586.log.gz 2023-10-16T15:56:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150819.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.150819.log.gz 2023-10-16T15:56:57.974 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145902.log 2023-10-16T15:56:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149781.log 2023-10-16T15:56:57.976 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114611.log.gz 2023-10-16T15:56:57.976 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116107.log 2023-10-16T15:56:57.977 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145902.log.gz 2023-10-16T15:56:57.977 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149781.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149781.log.gz 2023-10-16T15:56:57.978 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121588.log 2023-10-16T15:56:57.979 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108688.log 2023-10-16T15:56:57.979 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116107.log.gz 2023-10-16T15:56:57.980 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92296.log 2023-10-16T15:56:57.980 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.121588.log.gz 2023-10-16T15:56:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.133027.log 2023-10-16T15:56:57.982 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108688.log.gz 2023-10-16T15:56:57.982 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92296.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.92296.log.gz 2023-10-16T15:56:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148040.log 2023-10-16T15:56:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.133027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.133027.log.gz 2023-10-16T15:56:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123685.log 2023-10-16T15:56:57.985 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131687.log 2023-10-16T15:56:57.985 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.148040.log.gz 2023-10-16T15:56:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111231.log 2023-10-16T15:56:57.986 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123685.log.gz 2023-10-16T15:56:57.987 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135643.log 2023-10-16T15:56:57.987 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.131687.log.gz 2023-10-16T15:56:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.111231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.111231.log.gz 2023-10-16T15:56:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95806.log 2023-10-16T15:56:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109897.log 2023-10-16T15:56:57.990 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135643.log.gz 2023-10-16T15:56:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111706.log 2023-10-16T15:56:57.991 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95806.log.gz 2023-10-16T15:56:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109897.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109897.log.gz 2023-10-16T15:56:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93270.log 2023-10-16T15:56:57.993 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134303.log 2023-10-16T15:56:57.993 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111706.log.gz 2023-10-16T15:56:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148418.log 2023-10-16T15:56:57.994 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93270.log.gz 2023-10-16T15:56:57.995 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134303.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.134303.log.gz 2023-10-16T15:56:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135082.log 2023-10-16T15:56:57.996 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138261.log 2023-10-16T15:56:57.997 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148418.log.gz 2023-10-16T15:56:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112207.log 2023-10-16T15:56:57.998 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135082.log.gz 2023-10-16T15:56:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132258.log 2023-10-16T15:56:57.999 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138261.log.gz 2023-10-16T15:56:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112207.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112207.log.gz 2023-10-16T15:56:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113560.log 2023-10-16T15:56:58.001 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110848.log 2023-10-16T15:56:58.002 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.132258.log.gz 2023-10-16T15:56:58.002 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156582.log 2023-10-16T15:56:58.003 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113560.log.gz 2023-10-16T15:56:58.004 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110848.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110848.log.gz 2023-10-16T15:56:58.004 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121943.log 2023-10-16T15:56:58.005 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99195.log 2023-10-16T15:56:58.005 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156582.log.gz 2023-10-16T15:56:58.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149823.log 2023-10-16T15:56:58.006 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121943.log.gz 2023-10-16T15:56:58.007 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99195.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99195.log.gz 2023-10-16T15:56:58.007 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153293.log 2023-10-16T15:56:58.008 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133258.log 2023-10-16T15:56:58.008 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.149823.log.gz 2023-10-16T15:56:58.009 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.145187.log 2023-10-16T15:56:58.009 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.153293.log.gz 2023-10-16T15:56:58.010 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118355.log 2023-10-16T15:56:58.011 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133258.log.gz 2023-10-16T15:56:58.011 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.145187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.145187.log.gz 2023-10-16T15:56:58.012 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134616.log 2023-10-16T15:56:58.013 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117891.log 2023-10-16T15:56:58.013 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118355.log.gz 2023-10-16T15:56:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134616.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134616.log.gz -5 2023-10-16T15:56:58.014 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.132522.log 2023-10-16T15:56:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117891.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117891.log.gz 2023-10-16T15:56:58.016 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119736.log 2023-10-16T15:56:58.017 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123072.log 2023-10-16T15:56:58.017 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.132522.log.gz 2023-10-16T15:56:58.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149443.log 2023-10-16T15:56:58.018 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.119736.log.gz 2023-10-16T15:56:58.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123072.log: gzip 90.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.123072.log.gz --verbose 2023-10-16T15:56:58.019 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.146172.log 2023-10-16T15:56:58.020 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143721.log 2023-10-16T15:56:58.021 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149443.log.gz 2023-10-16T15:56:58.022 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146172.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.146172.log.gz 2023-10-16T15:56:58.022 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.109771.log 2023-10-16T15:56:58.023 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143721.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.143721.log.gz 2023-10-16T15:56:58.023 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116465.log 2023-10-16T15:56:58.024 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109771.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109771.log.gz 2023-10-16T15:56:58.025 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110424.log 2023-10-16T15:56:58.026 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123556.log 2023-10-16T15:56:58.026 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116465.log.gz 2023-10-16T15:56:58.027 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110424.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110424.log.gz 2023-10-16T15:56:58.027 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94783.log 2023-10-16T15:56:58.028 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109645.log 2023-10-16T15:56:58.028 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.123556.log.gz 2023-10-16T15:56:58.029 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94783.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94783.log.gz 2023-10-16T15:56:58.029 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117375.log 2023-10-16T15:56:58.030 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109645.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109645.log.gz 2023-10-16T15:56:58.031 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148166.log 2023-10-16T15:56:58.032 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117375.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.117375.log.gz 2023-10-16T15:56:58.032 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.145191.log 2023-10-16T15:56:58.034 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148166.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.148166.log.gz 2023-10-16T15:56:58.034 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150693.log 2023-10-16T15:56:58.034 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.145191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.145191.log.gz 2023-10-16T15:56:58.035 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143282.log 2023-10-16T15:56:58.036 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153377.log 2023-10-16T15:56:58.036 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150693.log.gz 2023-10-16T15:56:58.037 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143282.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.143282.log.gz 2023-10-16T15:56:58.038 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118439.log 2023-10-16T15:56:58.038 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153377.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.153377.log.gz 2023-10-16T15:56:58.039 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145352.log 2023-10-16T15:56:58.040 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118439.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118439.log.gz 2023-10-16T15:56:58.040 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125751.log 2023-10-16T15:56:58.041 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145352.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145352.log.gz 2023-10-16T15:56:58.042 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139559.log 2023-10-16T15:56:58.043 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125751.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.125751.log.gz 2023-10-16T15:56:58.043 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98220.log 2023-10-16T15:56:58.044 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139559.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139559.log.gz 2023-10-16T15:56:58.045 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116321.log 2023-10-16T15:56:58.046 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113602.log 2023-10-16T15:56:58.046 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98220.log.gz 2023-10-16T15:56:58.047 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115640.log 2023-10-16T15:56:58.047 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116321.log.gz 2023-10-16T15:56:58.048 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113602.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113602.log.gz 2023-10-16T15:56:58.048 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116023.log 2023-10-16T15:56:58.049 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139209.log 2023-10-16T15:56:58.050 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115640.log.gz 2023-10-16T15:56:58.050 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116023.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116023.log.gz 2023-10-16T15:56:58.051 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127255.log 2023-10-16T15:56:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139209.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.139209.log.gz 2023-10-16T15:56:58.052 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147016.log 2023-10-16T15:56:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127255.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127255.log.gz 2023-10-16T15:56:58.054 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134117.log 2023-10-16T15:56:58.055 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147016.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147016.log.gz 2023-10-16T15:56:58.055 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117662.log 2023-10-16T15:56:58.056 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121546.log 2023-10-16T15:56:58.057 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134117.log.gz 2023-10-16T15:56:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117662.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117662.log.gz 2023-10-16T15:56:58.058 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154714.log 2023-10-16T15:56:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121546.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121546.log.gz 2023-10-16T15:56:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99313.log 2023-10-16T15:56:58.060 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154714.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154714.log.gz 2023-10-16T15:56:58.060 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104653.log 2023-10-16T15:56:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147980.log 2023-10-16T15:56:58.062 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99313.log.gz 2023-10-16T15:56:58.063 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104653.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104653.log.gz 2023-10-16T15:56:58.063 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111392.log 2023-10-16T15:56:58.064 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91274.log 2023-10-16T15:56:58.065 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.147980.log.gz 2023-10-16T15:56:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111392.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111392.log.gz 2023-10-16T15:56:58.066 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153802.log 2023-10-16T15:56:58.067 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91274.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.91274.log.gz --verbose 2023-10-16T15:56:58.067 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.122512.log 2023-10-16T15:56:58.068 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153802.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.153802.log.gz 2023-10-16T15:56:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135125.log 2023-10-16T15:56:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145000.log 2023-10-16T15:56:58.070 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122512.log.gz 2023-10-16T15:56:58.071 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135125.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135125.log.gz 2023-10-16T15:56:58.071 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149739.log 2023-10-16T15:56:58.072 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145000.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145000.log.gz 2023-10-16T15:56:58.072 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111227.log 2023-10-16T15:56:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125530.log 2023-10-16T15:56:58.074 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149739.log.gz 2023-10-16T15:56:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.111227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.111227.log.gz 2023-10-16T15:56:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96472.log 2023-10-16T15:56:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153552.log 2023-10-16T15:56:58.077 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125530.log.gz 2023-10-16T15:56:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96472.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96472.log.gz 2023-10-16T15:56:58.078 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125091.log 2023-10-16T15:56:58.079 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153552.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.153552.log.gz 2023-10-16T15:56:58.079 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154194.log 2023-10-16T15:56:58.080 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108891.log 2023-10-16T15:56:58.081 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125091.log.gz 2023-10-16T15:56:58.081 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154194.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154194.log.gz 2023-10-16T15:56:58.082 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110974.log 2023-10-16T15:56:58.083 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108891.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108891.log.gz 2023-10-16T15:56:58.083 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122344.log 2023-10-16T15:56:58.084 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122045.log 2023-10-16T15:56:58.085 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122344.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.110974.log.gz 2023-10-16T15:56:58.086 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122344.log.gz 2023-10-16T15:56:58.086 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138093.log 2023-10-16T15:56:58.087 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122045.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122045.log.gz 2023-10-16T15:56:58.087 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138219.log 2023-10-16T15:56:58.088 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116633.log 2023-10-16T15:56:58.089 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138093.log.gz 2023-10-16T15:56:58.090 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138219.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.138219.log.gz 2023-10-16T15:56:58.090 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103063.log 2023-10-16T15:56:58.091 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116633.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116633.log.gz 2023-10-16T15:56:58.091 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111832.log 2023-10-16T15:56:58.092 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146322.log 2023-10-16T15:56:58.093 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103063.log.gz 2023-10-16T15:56:58.093 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111832.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111832.log.gz 2023-10-16T15:56:58.094 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114737.log 2023-10-16T15:56:58.095 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146322.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.146322.log.gz 2023-10-16T15:56:58.095 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131290.log 2023-10-16T15:56:58.096 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123138.log 2023-10-16T15:56:58.097 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114737.log.gz 2023-10-16T15:56:58.098 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131290.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.131290.log.gz 2023-10-16T15:56:58.098 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101383.log 2023-10-16T15:56:58.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123138.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.123138.log.gz 2023-10-16T15:56:58.099 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96598.log 2023-10-16T15:56:58.100 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106859.log 2023-10-16T15:56:58.101 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101383.log.gz 2023-10-16T15:56:58.101 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96598.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96598.log.gz 2023-10-16T15:56:58.102 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151534.log 2023-10-16T15:56:58.102 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106859.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106859.log.gz 2023-10-16T15:56:58.103 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116065.log 2023-10-16T15:56:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134784.log 2023-10-16T15:56:58.105 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151534.log.gz 2023-10-16T15:56:58.106 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116065.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116065.log.gz 2023-10-16T15:56:58.106 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105605.log 2023-10-16T15:56:58.107 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134784.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.134784.log.gz 2023-10-16T15:56:58.107 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140247.log 2023-10-16T15:56:58.108 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97102.log 2023-10-16T15:56:58.108 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105605.log.gz 2023-10-16T15:56:58.109 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140247.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140247.log.gz 2023-10-16T15:56:58.110 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92630.log 2023-10-16T15:56:58.110 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97102.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97102.log.gz 2023-10-16T15:56:58.111 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133150.log 2023-10-16T15:56:58.112 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157074.log 2023-10-16T15:56:58.113 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92630.log.gz 2023-10-16T15:56:58.113 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133150.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133150.log.gz 2023-10-16T15:56:58.114 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133592.log 2023-10-16T15:56:58.114 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157074.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157074.log.gz 2023-10-16T15:56:58.115 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155256.log 2023-10-16T15:56:58.116 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133592.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133592.log.gz 2023-10-16T15:56:58.116 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105993.log 2023-10-16T15:56:58.117 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97018.log 2023-10-16T15:56:58.118 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.155256.log.gz 2023-10-16T15:56:58.119 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105993.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105993.log.gz 2023-10-16T15:56:58.119 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153335.log 2023-10-16T15:56:58.120 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97018.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97018.log.gz 2023-10-16T15:56:58.121 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155382.log 2023-10-16T15:56:58.122 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125235.log 2023-10-16T15:56:58.122 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153335.log.gz 2023-10-16T15:56:58.122 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155382.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155382.log.gz 2023-10-16T15:56:58.123 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124203.log 2023-10-16T15:56:58.124 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138963.log 2023-10-16T15:56:58.125 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125235.log.gz 2023-10-16T15:56:58.126 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124203.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.124203.log.gz 2023-10-16T15:56:58.126 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96514.log 2023-10-16T15:56:58.127 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138963.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138963.log.gz 2023-10-16T15:56:58.127 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131519.log 2023-10-16T15:56:58.129 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96514.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96514.log.gzgzip 2023-10-16T15:56:58.129 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.114831.log 2023-10-16T15:56:58.130 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131519.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131519.log.gz 2023-10-16T15:56:58.130 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130723.log 2023-10-16T15:56:58.131 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137126.log 2023-10-16T15:56:58.132 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114831.log.gz 2023-10-16T15:56:58.133 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130723.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130723.log.gz 2023-10-16T15:56:58.133 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145436.log 2023-10-16T15:56:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137126.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.137126.log.gz 2023-10-16T15:56:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150385.log 2023-10-16T15:56:58.136 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115899.log 2023-10-16T15:56:58.136 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.145436.log.gz 2023-10-16T15:56:58.136 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150385.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150385.log.gz 2023-10-16T15:56:58.137 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142799.log 2023-10-16T15:56:58.138 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115899.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115899.log.gz 2023-10-16T15:56:58.139 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134363.log 2023-10-16T15:56:58.140 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91397.log 2023-10-16T15:56:58.140 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142799.log.gz 2023-10-16T15:56:58.141 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134363.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.134363.log.gz 2023-10-16T15:56:58.141 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92622.log 2023-10-16T15:56:58.143 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115430.log 2023-10-16T15:56:58.143 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91397.log.gz 2023-10-16T15:56:58.144 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92622.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92622.log.gz 2023-10-16T15:56:58.144 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95215.log 2023-10-16T15:56:58.145 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115430.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.115430.log.gz 2023-10-16T15:56:58.145 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139043.log 2023-10-16T15:56:58.146 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95215.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95215.log.gz 2023-10-16T15:56:58.147 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115262.log 2023-10-16T15:56:58.148 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138490.log 2023-10-16T15:56:58.148 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.139043.log.gz 2023-10-16T15:56:58.149 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115262.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.115262.log.gz 2023-10-16T15:56:58.149 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157451.log 2023-10-16T15:56:58.150 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138490.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138490.log.gz 2023-10-16T15:56:58.151 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121294.log 2023-10-16T15:56:58.152 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157451.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.157451.log.gz 2023-10-16T15:56:58.152 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114296.log 2023-10-16T15:56:58.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153940.log 2023-10-16T15:56:58.153 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121294.log.gz 2023-10-16T15:56:58.154 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114296.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114296.log.gz 2023-10-16T15:56:58.155 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109999.log 2023-10-16T15:56:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153940.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153940.log.gz 2023-10-16T15:56:58.156 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143070.log 2023-10-16T15:56:58.157 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109999.log: gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.109999.log.gz -- 2023-10-16T15:56:58.157 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.157284.log 2023-10-16T15:56:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143070.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.143070.log.gz 2023-10-16T15:56:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150199.log 2023-10-16T15:56:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157284.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157284.log.gz 2023-10-16T15:56:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92846.log 2023-10-16T15:56:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150199.log: gzip 88.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.150199.log.gz -- 2023-10-16T15:56:58.161 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.110256.log 2023-10-16T15:56:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92846.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.92846.log.gz 2023-10-16T15:56:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125404.log 2023-10-16T15:56:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110256.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110256.log.gz 2023-10-16T15:56:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142757.log 2023-10-16T15:56:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125404.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.125404.log.gz 2023-10-16T15:56:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.114111.log 2023-10-16T15:56:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142757.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142757.log.gz 2023-10-16T15:56:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130996.log 2023-10-16T15:56:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.114111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.114111.log.gz 2023-10-16T15:56:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105262.log 2023-10-16T15:56:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130996.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130996.log.gz 2023-10-16T15:56:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148670.log 2023-10-16T15:56:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105262.log: gzip -5 87.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.105262.log.gz -- 2023-10-16T15:56:58.170 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.93480.log 2023-10-16T15:56:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148670.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148670.log.gz 2023-10-16T15:56:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147667.log 2023-10-16T15:56:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93480.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93480.log.gz 2023-10-16T15:56:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131393.log 2023-10-16T15:56:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147667.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.147667.log.gz 2023-10-16T15:56:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T15:56:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131393.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.131393.log.gz 2023-10-16T15:56:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148712.log 2023-10-16T15:56:58.176 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120895.log 2023-10-16T15:56:58.178 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146214.log 2023-10-16T15:56:58.178 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148712.log.gz 2023-10-16T15:56:58.179 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120895.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.120895.log.gz 2023-10-16T15:56:58.179 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92298.log 2023-10-16T15:56:58.180 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100667.log 2023-10-16T15:56:58.181 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.146214.log.gz 2023-10-16T15:56:58.181 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110680.log 2023-10-16T15:56:58.182 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92298.log.gz 2023-10-16T15:56:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100667.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100667.log.gz 2023-10-16T15:56:58.183 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117296.log 2023-10-16T15:56:58.183 INFO:teuthology.orchestra.run.smithi029.stderr: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T15:56:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110680.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110680.log.gz 2023-10-16T15:56:58.184 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134406.log 2023-10-16T15:56:58.185 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117296.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117296.log.gz 2023-10-16T15:56:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117935.log 2023-10-16T15:56:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134406.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134406.log.gz 2023-10-16T15:56:58.187 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103870.log 2023-10-16T15:56:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117935.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117935.log.gz 2023-10-16T15:56:58.188 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150513.log 2023-10-16T15:56:58.189 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103870.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.0.114107.log -- replaced with /var/log/ceph/ceph-client.admin.103870.log.gz 2023-10-16T15:56:58.189 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.190 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156178.log 2023-10-16T15:56:58.191 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150513.log.gz 2023-10-16T15:56:58.191 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.114107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.114107.log.gz 2023-10-16T15:56:58.192 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153082.log 2023-10-16T15:56:58.193 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156178.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.155136.log -- replaced with /var/log/ceph/ceph-client.admin.156178.log.gz 2023-10-16T15:56:58.193 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.194 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139829.log 2023-10-16T15:56:58.194 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153082.log.gz 2023-10-16T15:56:58.195 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155136.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155136.log.gz 2023-10-16T15:56:58.196 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139431.log 2023-10-16T15:56:58.197 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139829.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.139829.log.gz 2023-10-16T15:56:58.197 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149055.log 2023-10-16T15:56:58.198 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139431.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.139431.log.gz 2023-10-16T15:56:58.198 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153890.log 2023-10-16T15:56:58.199 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149055.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149055.log.gz 2023-10-16T15:56:58.199 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.95130.log 2023-10-16T15:56:58.200 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132564.log 2023-10-16T15:56:58.201 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153890.log.gz 2023-10-16T15:56:58.202 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95130.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95130.log.gz 2023-10-16T15:56:58.202 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148796.log 2023-10-16T15:56:58.203 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132564.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.132564.log.gz 2023-10-16T15:56:58.203 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.116717.log 2023-10-16T15:56:58.204 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148796.log: gzip -5 89.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.148796.log.gz /var/log/ceph/ceph-client.admin.156708.log 2023-10-16T15:56:58.205 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.205 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125572.log 2023-10-16T15:56:58.206 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.116717.log.gz 2023-10-16T15:56:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156708.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.156708.log.gz 2023-10-16T15:56:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138135.log 2023-10-16T15:56:58.208 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125572.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125572.log.gz 2023-10-16T15:56:58.208 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151203.log 2023-10-16T15:56:58.209 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98034.log 2023-10-16T15:56:58.210 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138135.log.gz 2023-10-16T15:56:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151203.log: gzip -5 88.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.151203.log.gz -- 2023-10-16T15:56:58.211 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.148334.log 2023-10-16T15:56:58.212 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98034.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98034.log.gz 2023-10-16T15:56:58.212 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109332.log 2023-10-16T15:56:58.213 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140509.log 2023-10-16T15:56:58.213 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.148334.log.gz 2023-10-16T15:56:58.214 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109332.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109332.log.gz 2023-10-16T15:56:58.214 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104252.log 2023-10-16T15:56:58.216 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140509.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.140509.log.gz 2023-10-16T15:56:58.216 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132480.log 2023-10-16T15:56:58.217 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104252.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104252.log.gz 2023-10-16T15:56:58.217 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154278.log 2023-10-16T15:56:58.218 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132480.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.132480.log.gz 2023-10-16T15:56:58.219 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.110638.log 2023-10-16T15:56:58.220 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152811.log 2023-10-16T15:56:58.220 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154278.log.gz 2023-10-16T15:56:58.221 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110638.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110638.log.gz 2023-10-16T15:56:58.221 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151808.log 2023-10-16T15:56:58.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152811.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152811.log.gz 2023-10-16T15:56:58.222 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132438.log 2023-10-16T15:56:58.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152998.log 2023-10-16T15:56:58.224 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151808.log.gz 2023-10-16T15:56:58.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132438.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.132438.log.gz 2023-10-16T15:56:58.225 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140205.log 2023-10-16T15:56:58.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131248.log 2023-10-16T15:56:58.226 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152998.log.gz 2023-10-16T15:56:58.227 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140205.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.140205.log.gz 2023-10-16T15:56:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124482.log 2023-10-16T15:56:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131248.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.131248.log.gz -5 2023-10-16T15:56:58.229 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.133317.log 2023-10-16T15:56:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124482.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124482.log.gz 2023-10-16T15:56:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118397.log 2023-10-16T15:56:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147373.log 2023-10-16T15:56:58.231 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.133317.log.gz 2023-10-16T15:56:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118397.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118397.log.gz 2023-10-16T15:56:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138700.log 2023-10-16T15:56:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147373.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147373.log.gz 2023-10-16T15:56:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115682.log 2023-10-16T15:56:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138700.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138700.log.gz 2023-10-16T15:56:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94657.log 2023-10-16T15:56:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119008.log 88.0% 2023-10-16T15:56:58.237 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.115682.log.gz 2023-10-16T15:56:58.237 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112585.log 2023-10-16T15:56:58.238 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94657.log.gz 2023-10-16T15:56:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119008.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119008.log.gz 2023-10-16T15:56:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124608.log 2023-10-16T15:56:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112585.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112585.log.gz 2023-10-16T15:56:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94615.log 2023-10-16T15:56:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121336.log 2023-10-16T15:56:58.242 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124608.log.gz 2023-10-16T15:56:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94615.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94615.log.gz 2023-10-16T15:56:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98948.log 2023-10-16T15:56:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121336.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121336.log.gz 2023-10-16T15:56:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152895.log 2023-10-16T15:56:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98948.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.120332.log -- replaced with /var/log/ceph/ceph-client.admin.98948.log.gz 2023-10-16T15:56:58.245 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.246 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152895.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.114569.log 2023-10-16T15:56:58.246 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.152895.log.gz 2023-10-16T15:56:58.247 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120332.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.137820.log -- replaced with /var/log/ceph/ceph-client.admin.120332.log.gz 2023-10-16T15:56:58.248 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.248 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.150118.log 2023-10-16T15:56:58.249 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114569.log.gz 2023-10-16T15:56:58.250 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137820.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137820.log.gz 2023-10-16T15:56:58.250 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97431.log 2023-10-16T15:56:58.251 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.150118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.150118.log.gz 2023-10-16T15:56:58.251 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149221.log 2023-10-16T15:56:58.252 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97431.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97431.log.gz 2023-10-16T15:56:58.252 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144554.log 2023-10-16T15:56:58.254 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149221.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149221.log.gz 2023-10-16T15:56:58.254 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130859.log 2023-10-16T15:56:58.255 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144554.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144554.log.gz 2023-10-16T15:56:58.255 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136211.log 2023-10-16T15:56:58.256 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130859.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130859.log.gz 2023-10-16T15:56:58.256 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104569.log 2023-10-16T15:56:58.257 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136211.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136211.log.gz 2023-10-16T15:56:58.257 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139381.log 2023-10-16T15:56:58.258 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104569.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104569.log.gzgzip 2023-10-16T15:56:58.259 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91654.log 2023-10-16T15:56:58.260 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139381.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139381.log.gz 2023-10-16T15:56:58.260 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156666.log 2023-10-16T15:56:58.261 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91654.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91654.log.gzgzip 2023-10-16T15:56:58.261 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.117977.log 2023-10-16T15:56:58.262 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156666.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156666.log.gz 2023-10-16T15:56:58.262 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91738.log 2023-10-16T15:56:58.263 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117977.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.117977.log.gz 2023-10-16T15:56:58.264 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138616.log 2023-10-16T15:56:58.265 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121021.log 2023-10-16T15:56:58.265 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91738.log.gz 2023-10-16T15:56:58.266 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138616.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.138616.log.gz 2023-10-16T15:56:58.266 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143907.log 2023-10-16T15:56:58.267 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121021.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121021.log.gz 2023-10-16T15:56:58.267 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135475.log 2023-10-16T15:56:58.268 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143907.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143907.log.gz 2023-10-16T15:56:58.269 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.126912.log 2023-10-16T15:56:58.270 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135475.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135475.log.gz 2023-10-16T15:56:58.270 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139251.log 2023-10-16T15:56:58.271 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.126912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.126912.log.gz 2023-10-16T15:56:58.271 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156540.log 2023-10-16T15:56:58.272 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99136.log 2023-10-16T15:56:58.273 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139251.log.gz 2023-10-16T15:56:58.273 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156540.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156540.log.gz 2023-10-16T15:56:58.274 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105388.log 2023-10-16T15:56:58.275 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99136.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99136.log.gz 2023-10-16T15:56:58.275 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156750.log 2023-10-16T15:56:58.276 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105388.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105388.log.gz 2023-10-16T15:56:58.276 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157031.log 2023-10-16T15:56:58.277 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156750.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156750.log.gz 2023-10-16T15:56:58.277 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92294.log 2023-10-16T15:56:58.278 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157031.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.157031.log.gz 2023-10-16T15:56:58.279 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157369.log 2023-10-16T15:56:58.280 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97516.log 88.5% 2023-10-16T15:56:58.280 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92294.log.gz 2023-10-16T15:56:58.281 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145564.log 2023-10-16T15:56:58.281 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157369.log.gz 2023-10-16T15:56:58.282 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97516.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97516.log.gz 2023-10-16T15:56:58.282 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101164.log 2023-10-16T15:56:58.283 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145564.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145564.log.gz 2023-10-16T15:56:58.284 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105813.log 2023-10-16T15:56:58.285 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101164.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101164.log.gz 2023-10-16T15:56:58.285 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139125.log 2023-10-16T15:56:58.286 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105813.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105813.log.gz 2023-10-16T15:56:58.286 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122087.log 2023-10-16T15:56:58.287 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153208.log 2023-10-16T15:56:58.287 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139125.log.gz 2023-10-16T15:56:58.288 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122087.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122087.log.gz 2023-10-16T15:56:58.288 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145986.log 2023-10-16T15:56:58.290 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153208.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153208.log.gz 2023-10-16T15:56:58.290 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143156.log 2023-10-16T15:56:58.291 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145986.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145986.log.gz -5 2023-10-16T15:56:58.291 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.147541.log 2023-10-16T15:56:58.292 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143156.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.143156.log.gz 2023-10-16T15:56:58.292 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131645.log 2023-10-16T15:56:58.293 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135253.log 2023-10-16T15:56:58.293 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.147541.log.gz 2023-10-16T15:56:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131645.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131645.log.gz 2023-10-16T15:56:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150649.log 2023-10-16T15:56:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138345.log 2023-10-16T15:56:58.296 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.135253.log.gz 2023-10-16T15:56:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150649.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150649.log.gz 2023-10-16T15:56:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149655.log 2023-10-16T15:56:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138345.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138345.log.gz -5 2023-10-16T15:56:58.298 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.154428.log 2023-10-16T15:56:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151095.log 2023-10-16T15:56:58.300 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149655.log.gz 2023-10-16T15:56:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154428.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.154428.log.gz 2023-10-16T15:56:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100499.log 2023-10-16T15:56:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151095.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151095.log.gz 2023-10-16T15:56:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131946.log 2023-10-16T15:56:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100499.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100499.log.gz 2023-10-16T15:56:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132840.log 2023-10-16T15:56:58.304 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131946.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131946.log.gz 2023-10-16T15:56:58.305 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144290.log 2023-10-16T15:56:58.306 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114968.log 2023-10-16T15:56:58.307 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.132840.log.gz 2023-10-16T15:56:58.307 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144290.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144290.log.gz 2023-10-16T15:56:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144892.log 2023-10-16T15:56:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114968.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114968.log.gz 2023-10-16T15:56:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144682.log 2023-10-16T15:56:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119802.log 2023-10-16T15:56:58.310 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144892.log.gz 2023-10-16T15:56:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133760.log 2023-10-16T15:56:58.311 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144682.log.gz 2023-10-16T15:56:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131561.log 2023-10-16T15:56:58.313 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119802.log.gz 2023-10-16T15:56:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133760.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.133760.log.gz 2023-10-16T15:56:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157409.log 2023-10-16T15:56:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131561.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.131561.log.gz 2023-10-16T15:56:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98722.log 2023-10-16T15:56:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112627.log 2023-10-16T15:56:58.316 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.157409.log.gz 2023-10-16T15:56:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98722.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98722.log.gz 2023-10-16T15:56:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.126916.log 2023-10-16T15:56:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.150122.log 2023-10-16T15:56:58.319 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112627.log.gz 2023-10-16T15:56:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.126916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.126916.log.gz 2023-10-16T15:56:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150903.log 2023-10-16T15:56:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.150122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.150122.log.gz 2023-10-16T15:56:58.321 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92032.log 2023-10-16T15:56:58.322 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154362.log 2023-10-16T15:56:58.323 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150903.log.gz 2023-10-16T15:56:58.323 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92032.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92032.log.gz 2023-10-16T15:56:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100987.log 2023-10-16T15:56:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146280.log 2023-10-16T15:56:58.325 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.154362.log.gz 2023-10-16T15:56:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115010.log 2023-10-16T15:56:58.326 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100987.log.gz 2023-10-16T15:56:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146280.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146280.log.gz 2023-10-16T15:56:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96352.log 2023-10-16T15:56:58.328 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115010.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115010.log.gz 2023-10-16T15:56:58.329 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105855.log 2023-10-16T15:56:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138051.log 2023-10-16T15:56:58.330 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96352.log.gz 2023-10-16T15:56:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105855.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105855.log.gz 2023-10-16T15:56:58.331 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154923.log 2023-10-16T15:56:58.332 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135835.log 2023-10-16T15:56:58.332 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138051.log.gz 2023-10-16T15:56:58.333 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154923.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154923.log.gz 2023-10-16T15:56:58.334 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121420.log 2023-10-16T15:56:58.335 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138742.log 2023-10-16T15:56:58.335 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.135835.log.gz 2023-10-16T15:56:58.336 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121420.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121420.log.gz 2023-10-16T15:56:58.336 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100625.log 2023-10-16T15:56:58.337 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138742.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138742.log.gz 2023-10-16T15:56:58.337 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146028.log 2023-10-16T15:56:58.338 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100625.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100625.log.gz 2023-10-16T15:56:58.338 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113392.log 2023-10-16T15:56:58.339 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146028.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.146028.log.gz 2023-10-16T15:56:58.340 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98346.log 2023-10-16T15:56:58.341 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95081.log 2023-10-16T15:56:58.341 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113392.log.gz 2023-10-16T15:56:58.342 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93354.log 2023-10-16T15:56:58.342 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98346.log.gz 2023-10-16T15:56:58.343 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95081.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95081.log.gz 2023-10-16T15:56:58.343 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151262.log 2023-10-16T15:56:58.344 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93354.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93354.log.gz 2023-10-16T15:56:58.344 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151850.log 2023-10-16T15:56:58.345 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95467.log 2023-10-16T15:56:58.346 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151262.log.gz 2023-10-16T15:56:58.347 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132648.log 2023-10-16T15:56:58.347 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151850.log.gz 2023-10-16T15:56:58.348 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95467.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95467.log.gz 2023-10-16T15:56:58.348 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126169.log 2023-10-16T15:56:58.349 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132648.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.132648.log.gz 2023-10-16T15:56:58.349 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110174.log 2023-10-16T15:56:58.350 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99574.log 2023-10-16T15:56:58.351 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126169.log.gz 2023-10-16T15:56:58.352 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110174.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110174.log.gz 2023-10-16T15:56:58.352 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156498.log 2023-10-16T15:56:58.353 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99574.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99574.log.gz 2023-10-16T15:56:58.353 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157200.log 2023-10-16T15:56:58.355 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144374.log 2023-10-16T15:56:58.355 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.156498.log.gz 2023-10-16T15:56:58.356 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157200.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.157200.log.gz 2023-10-16T15:56:58.356 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-16T15:56:58.357 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111517.log 2023-10-16T15:56:58.357 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144374.log.gz 2023-10-16T15:56:58.358 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120937.log 2023-10-16T15:56:58.359 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111517.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111517.log.gz 2023-10-16T15:56:58.360 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149931.log 2023-10-16T15:56:58.361 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132732.log 2023-10-16T15:56:58.361 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120937.log.gz 2023-10-16T15:56:58.361 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149931.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149931.log.gz 2023-10-16T15:56:58.362 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146384.log 2023-10-16T15:56:58.364 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155060.log 2023-10-16T15:56:58.364 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.132732.log.gz 2023-10-16T15:56:58.365 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146384.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146384.log.gz 2023-10-16T15:56:58.365 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131771.log 2023-10-16T15:56:58.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105346.log 2023-10-16T15:56:58.366 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155060.log.gz 2023-10-16T15:56:58.367 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98872.log 2023-10-16T15:56:58.368 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.131771.log.gz 2023-10-16T15:56:58.368 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105346.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105346.log.gz 2023-10-16T15:56:58.369 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99018.log 2023-10-16T15:56:58.370 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98872.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98872.log.gz 2023-10-16T15:56:58.370 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.118649.log 2023-10-16T15:56:58.371 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117093.log 2023-10-16T15:56:58.371 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99018.log.gz 2023-10-16T15:56:58.372 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118649.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118649.log.gz 2023-10-16T15:56:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156945.log 2023-10-16T15:56:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116783.log 2023-10-16T15:56:58.374 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117093.log.gz 2023-10-16T15:56:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121462.log 2023-10-16T15:56:58.375 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156945.log.gz 2023-10-16T15:56:58.376 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116783.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116783.log.gz 2023-10-16T15:56:58.377 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133676.log 2023-10-16T15:56:58.378 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132774.log 2023-10-16T15:56:58.378 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121462.log.gz 2023-10-16T15:56:58.379 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138658.log 2023-10-16T15:56:58.379 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.133676.log.gz 2023-10-16T15:56:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121378.log 2023-10-16T15:56:58.380 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.132774.log.gz 2023-10-16T15:56:58.381 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108849.log 2023-10-16T15:56:58.381 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.138658.log.gz 2023-10-16T15:56:58.382 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121378.log: gzip -5 --verbose 89.0% -- -- replaced with /var/log/ceph/ceph-client.admin.121378.log.gz /var/log/ceph/ceph-client.admin.132087.log 2023-10-16T15:56:58.383 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.384 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116423.log 2023-10-16T15:56:58.384 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108849.log.gz 2023-10-16T15:56:58.385 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132087.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.132087.log.gz 2023-10-16T15:56:58.385 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.133414.log 2023-10-16T15:56:58.386 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126261.log 2023-10-16T15:56:58.386 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116423.log.gz 2023-10-16T15:56:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133414.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.133414.log.gz 2023-10-16T15:56:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136103.log 2023-10-16T15:56:58.389 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134159.log 2023-10-16T15:56:58.389 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126261.log.gz 2023-10-16T15:56:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136103.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.136103.log.gz --verbose 2023-10-16T15:56:58.390 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.0.140128.log 2023-10-16T15:56:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134159.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134159.log.gz 2023-10-16T15:56:58.391 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113348.log 2023-10-16T15:56:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.140128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.140128.log.gz 2023-10-16T15:56:58.392 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104737.log 2023-10-16T15:56:58.393 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130765.log 88.7% 2023-10-16T15:56:58.394 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.113348.log.gz 2023-10-16T15:56:58.395 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125320.log 2023-10-16T15:56:58.395 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104737.log.gz 2023-10-16T15:56:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130765.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130765.log.gz 2023-10-16T15:56:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125446.log 2023-10-16T15:56:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154976.log 2023-10-16T15:56:58.398 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.125320.log.gz 2023-10-16T15:56:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125446.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.125446.log.gz 2023-10-16T15:56:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131038.log 2023-10-16T15:56:58.400 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124524.log 2023-10-16T15:56:58.401 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154976.log.gz 2023-10-16T15:56:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131038.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.131038.log.gz 2023-10-16T15:56:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154110.log 2023-10-16T15:56:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127035.log 2023-10-16T15:56:58.403 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124524.log.gz 2023-10-16T15:56:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154110.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154110.log.gz 2023-10-16T15:56:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92289.log 2023-10-16T15:56:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149613.log 2023-10-16T15:56:58.406 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127035.log.gz 2023-10-16T15:56:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92289.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92289.log.gz 2023-10-16T15:56:58.407 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133108.log 2023-10-16T15:56:58.408 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155518.log 2023-10-16T15:56:58.408 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.149613.log.gz 2023-10-16T15:56:58.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133108.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.133108.log.gz 2023-10-16T15:56:58.409 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100751.log 2023-10-16T15:56:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112974.log 2023-10-16T15:56:58.411 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155518.log.gz 2023-10-16T15:56:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100751.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100751.log.gz 2023-10-16T15:56:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152372.log 2023-10-16T15:56:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95764.log 2023-10-16T15:56:58.414 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112974.log.gz 2023-10-16T15:56:58.414 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152372.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152372.log.gz 2023-10-16T15:56:58.415 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123744.log 2023-10-16T15:56:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116193.log 2023-10-16T15:56:58.416 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95764.log.gz 2023-10-16T15:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120112.log 2023-10-16T15:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123744.log.gz 2023-10-16T15:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116193.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116193.log.gz 2023-10-16T15:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152186.log 2023-10-16T15:56:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126521.log 2023-10-16T15:56:58.420 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.120112.log.gz 2023-10-16T15:56:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152186.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152186.log.gz 2023-10-16T15:56:58.421 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145650.log 2023-10-16T15:56:58.422 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126521.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.117201.log -- replaced with /var/log/ceph/ceph-client.admin.126521.log.gz 2023-10-16T15:56:58.422 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.423 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145650.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145650.log.gzgzip 2023-10-16T15:56:58.423 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.117746.log 2023-10-16T15:56:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155298.log 2023-10-16T15:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117201.log.gz 2023-10-16T15:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117746.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117746.log.gz 2023-10-16T15:56:58.426 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110512.log 2023-10-16T15:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119526.log 2023-10-16T15:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.155298.log.gz 2023-10-16T15:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110512.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110512.log.gz 2023-10-16T15:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147709.log 2023-10-16T15:56:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107029.log 2023-10-16T15:56:58.430 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119526.log.gz 2023-10-16T15:56:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147709.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147709.log.gz 2023-10-16T15:56:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155823.log 2023-10-16T15:56:58.432 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93312.log 2023-10-16T15:56:58.432 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107029.log.gz 2023-10-16T15:56:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155823.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155823.log.gz 2023-10-16T15:56:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147457.log 2023-10-16T15:56:58.434 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99784.log 2023-10-16T15:56:58.435 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93312.log.gz 2023-10-16T15:56:58.435 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147457.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147457.log.gz 2023-10-16T15:56:58.436 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124965.log 2023-10-16T15:56:58.437 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106223.log 2023-10-16T15:56:58.437 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99784.log.gz 2023-10-16T15:56:58.438 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124965.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124965.log.gz 2023-10-16T15:56:58.438 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134490.log 2023-10-16T15:56:58.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94993.log 2023-10-16T15:56:58.439 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106223.log.gz 2023-10-16T15:56:58.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134490.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134490.log.gz 2023-10-16T15:56:58.440 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138387.log 2023-10-16T15:56:58.442 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100928.log 2023-10-16T15:56:58.442 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94993.log.gz 2023-10-16T15:56:58.443 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138387.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.138387.log.gz 2023-10-16T15:56:58.443 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118709.log 2023-10-16T15:56:58.444 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155655.log 2023-10-16T15:56:58.444 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5%/var/log/ceph/ceph-client.admin.118709.log: -- replaced with /var/log/ceph/ceph-client.admin.100928.log.gz 2023-10-16T15:56:58.445 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.118709.log.gz 2023-10-16T15:56:58.445 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146855.log 2023-10-16T15:56:58.447 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155655.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155655.log.gz 2023-10-16T15:56:58.447 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143763.log 2023-10-16T15:56:58.448 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146855.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.146855.log.gz 2023-10-16T15:56:58.448 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97684.log 2023-10-16T15:56:58.449 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127297.log 2023-10-16T15:56:58.449 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.143763.log.gz 2023-10-16T15:56:58.450 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97684.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97684.log.gz 2023-10-16T15:56:58.450 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99077.log 2023-10-16T15:56:58.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139293.log 2023-10-16T15:56:58.452 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127297.log.gz 2023-10-16T15:56:58.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99077.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99077.log.gz 2023-10-16T15:56:58.453 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93041.log 2023-10-16T15:56:58.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133854.log 2023-10-16T15:56:58.454 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139293.log.gz 2023-10-16T15:56:58.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93041.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93041.log.gz 2023-10-16T15:56:58.455 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136026.log 2023-10-16T15:56:58.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147625.log 2023-10-16T15:56:58.457 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133854.log.gz 2023-10-16T15:56:58.457 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.136026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136026.log.gz 2023-10-16T15:56:58.458 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137185.log 2023-10-16T15:56:58.459 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155949.log 2023-10-16T15:56:58.459 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.147625.log.gz 2023-10-16T15:56:58.460 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137185.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137185.log.gz 2023-10-16T15:56:58.460 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111748.log 2023-10-16T15:56:58.461 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136329.log 2023-10-16T15:56:58.462 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155949.log.gz 2023-10-16T15:56:58.462 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111748.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111748.log.gz 2023-10-16T15:56:58.462 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152601.log 2023-10-16T15:56:58.464 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114877.log 2023-10-16T15:56:58.464 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.136329.log.gz 2023-10-16T15:56:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152601.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152601.log.gz 2023-10-16T15:56:58.465 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98178.log 2023-10-16T15:56:58.466 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126347.log 2023-10-16T15:56:58.466 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114877.log.gz 2023-10-16T15:56:58.467 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98178.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98178.log.gz 2023-10-16T15:56:58.467 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100709.log 2023-10-16T15:56:58.469 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150557.log 2023-10-16T15:56:58.469 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.126347.log.gz 2023-10-16T15:56:58.470 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100709.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100709.log.gz 2023-10-16T15:56:58.470 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155781.log 2023-10-16T15:56:58.471 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105009.log 2023-10-16T15:56:58.471 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.150557.log.gz 2023-10-16T15:56:58.472 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155781.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155781.log.gz 2023-10-16T15:56:58.472 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108807.log 2023-10-16T15:56:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131729.log 2023-10-16T15:56:58.474 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105009.log.gz 2023-10-16T15:56:58.474 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108807.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108807.log.gzgzip 2023-10-16T15:56:58.475 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.144808.log 2023-10-16T15:56:58.476 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111586.log 2023-10-16T15:56:58.476 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.131729.log.gz 2023-10-16T15:56:58.477 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133634.log 2023-10-16T15:56:58.477 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144808.log.gz 2023-10-16T15:56:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151053.log 2023-10-16T15:56:58.478 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111586.log.gz 2023-10-16T15:56:58.479 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133634.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.133634.log.gz 2023-10-16T15:56:58.480 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97144.log 2023-10-16T15:56:58.481 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151053.log: gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.151053.log.gz --verbose 2023-10-16T15:56:58.481 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.150777.log 2023-10-16T15:56:58.482 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149393.log 2023-10-16T15:56:58.483 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97144.log.gz 2023-10-16T15:56:58.483 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150777.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150777.log.gz 2023-10-16T15:56:58.484 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124650.log 2023-10-16T15:56:58.485 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94867.log 2023-10-16T15:56:58.485 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149393.log.gz 2023-10-16T15:56:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124650.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124650.log.gz 2023-10-16T15:56:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104779.log 2023-10-16T15:56:58.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131206.log 2023-10-16T15:56:58.488 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94867.log.gz 2023-10-16T15:56:58.489 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118187.log 2023-10-16T15:56:58.489 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104779.log.gz 2023-10-16T15:56:58.490 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131206.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.131206.log.gz 2023-10-16T15:56:58.490 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153124.log 2023-10-16T15:56:58.491 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98820.log 2023-10-16T15:56:58.492 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118187.log.gz 2023-10-16T15:56:58.492 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118313.log 2023-10-16T15:56:58.493 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153124.log.gz 2023-10-16T15:56:58.494 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98820.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98820.log.gz 2023-10-16T15:56:58.494 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150301.log 2023-10-16T15:56:58.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95968.log 2023-10-16T15:56:58.495 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.118313.log.gz 2023-10-16T15:56:58.496 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150301.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.150301.log.gz 2023-10-16T15:56:58.496 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101282.log 2023-10-16T15:56:58.497 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149529.log 2023-10-16T15:56:58.497 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95968.log.gz 2023-10-16T15:56:58.498 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101282.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101282.log.gz 2023-10-16T15:56:58.498 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156220.log 2023-10-16T15:56:58.499 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100541.log 2023-10-16T15:56:58.500 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.149529.log.gz 2023-10-16T15:56:58.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156220.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156220.log.gz 2023-10-16T15:56:58.501 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105136.log 2023-10-16T15:56:58.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152643.log 2023-10-16T15:56:58.502 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100541.log.gz 2023-10-16T15:56:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105136.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105136.log.gz 2023-10-16T15:56:58.503 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123921.log 2023-10-16T15:56:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122736.log 2023-10-16T15:56:58.505 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152643.log.gz 2023-10-16T15:56:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123921.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123921.log.gz 2023-10-16T15:56:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139167.log 2023-10-16T15:56:58.507 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147938.log 2023-10-16T15:56:58.507 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122736.log.gz 2023-10-16T15:56:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139167.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.139167.log.gz 2023-10-16T15:56:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105052.log 2023-10-16T15:56:58.509 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150469.log 2023-10-16T15:56:58.510 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147938.log.gz 2023-10-16T15:56:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105052.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105052.log.gz 2023-10-16T15:56:58.511 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151892.log 2023-10-16T15:56:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150469.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.150469.log.gz 2023-10-16T15:56:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121157.log 2023-10-16T15:56:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92956.log 2023-10-16T15:56:58.513 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151892.log.gz 2023-10-16T15:56:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110764.log 2023-10-16T15:56:58.514 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121157.log.gz 2023-10-16T15:56:58.515 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92956.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92956.log.gz 2023-10-16T15:56:58.515 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112375.log 2023-10-16T15:56:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125278.log 2023-10-16T15:56:58.517 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110764.log.gz 2023-10-16T15:56:58.518 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106917.log 2023-10-16T15:56:58.518 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112375.log.gz 2023-10-16T15:56:58.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99826.log 2023-10-16T15:56:58.519 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.125278.log.gz 2023-10-16T15:56:58.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92282.log 2023-10-16T15:56:58.520 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106917.log.gz 2023-10-16T15:56:58.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98641.log 2023-10-16T15:56:58.522 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99826.log.gz 2023-10-16T15:56:58.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92282.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-16T15:56:58.523 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92282.log.gz 2023-10-16T15:56:58.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.98641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.98641.log.gz 2023-10-16T15:56:58.524 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152018.log 2023-10-16T15:56:58.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156136.log 2023-10-16T15:56:58.526 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152018.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152018.log.gz 2023-10-16T15:56:58.526 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113770.log 2023-10-16T15:56:58.527 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91097.log 2023-10-16T15:56:58.528 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156136.log: /var/log/ceph/ceph-client.admin.113770.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156136.log.gz 2023-10-16T15:56:58.529 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113770.log.gz 2023-10-16T15:56:58.529 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139871.log 2023-10-16T15:56:58.530 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145776.log 2023-10-16T15:56:58.531 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91097.log.gz 2023-10-16T15:56:58.531 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139871.log: gzip 90.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.139871.log.gz --verbose 2023-10-16T15:56:58.531 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.127101.log 2023-10-16T15:56:58.532 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121648.log 2023-10-16T15:56:58.533 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145776.log.gz 2023-10-16T15:56:58.534 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127101.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127101.log.gz 2023-10-16T15:56:58.534 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146130.log 2023-10-16T15:56:58.535 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123325.log 2023-10-16T15:56:58.535 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121648.log.gz 2023-10-16T15:56:58.536 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146130.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146130.log.gz 2023-10-16T15:56:58.536 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135769.log 2023-10-16T15:56:58.538 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.123325.log: -- /var/log/ceph/ceph-client.admin.112501.log 2023-10-16T15:56:58.538 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.123325.log.gz 2023-10-16T15:56:58.539 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137862.log 2023-10-16T15:56:58.539 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.135769.log.gz 2023-10-16T15:56:58.540 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148460.log 2023-10-16T15:56:58.541 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112501.log.gz 2023-10-16T15:56:58.541 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137862.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137862.log.gz 2023-10-16T15:56:58.542 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95036.log 2023-10-16T15:56:58.543 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142638.log 2023-10-16T15:56:58.544 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148460.log: /var/log/ceph/ceph-client.admin.95036.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148460.log.gz 2023-10-16T15:56:58.544 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95036.log.gz 2023-10-16T15:56:58.544 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116363.log 2023-10-16T15:56:58.545 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111790.log 2023-10-16T15:56:58.546 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142638.log.gz 2023-10-16T15:56:58.546 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116363.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116363.log.gz 2023-10-16T15:56:58.547 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105525.log 2023-10-16T15:56:58.548 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92297.log 2023-10-16T15:56:58.549 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111790.log: /var/log/ceph/ceph-client.admin.105525.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111790.log.gz 2023-10-16T15:56:58.549 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105525.log.gz 2023-10-16T15:56:58.549 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97990.log 2023-10-16T15:56:58.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118061.log 2023-10-16T15:56:58.551 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92297.log.gz 2023-10-16T15:56:58.551 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97990.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97990.log.gz 2023-10-16T15:56:58.552 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110298.log 2023-10-16T15:56:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126725.log 2023-10-16T15:56:58.553 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118061.log: /var/log/ceph/ceph-client.admin.110298.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118061.log.gz 2023-10-16T15:56:58.554 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110298.log.gz 2023-10-16T15:56:58.554 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133991.log 2023-10-16T15:56:58.555 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111308.log 2023-10-16T15:56:58.555 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.126725.log.gz 2023-10-16T15:56:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133991.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.133991.log.gz 2023-10-16T15:56:58.557 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145268.log 2023-10-16T15:56:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148544.log 2023-10-16T15:56:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111308.log: /var/log/ceph/ceph-client.admin.145268.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111308.log.gz 2023-10-16T15:56:58.559 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145268.log.gz 2023-10-16T15:56:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144766.log 2023-10-16T15:56:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156052.log 2023-10-16T15:56:58.560 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.148544.log.gz 2023-10-16T15:56:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144766.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.144766.log.gz 2023-10-16T15:56:58.562 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112249.log 2023-10-16T15:56:58.563 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.156052.log: -- /var/log/ceph/ceph-client.admin.138868.log 2023-10-16T15:56:58.563 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156052.log.gz 2023-10-16T15:56:58.564 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112249.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112249.log.gz 2023-10-16T15:56:58.564 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104133.log 2023-10-16T15:56:58.565 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117620.log 2023-10-16T15:56:58.566 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.138868.log.gz 2023-10-16T15:56:58.566 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104133.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104133.log.gz 2023-10-16T15:56:58.566 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104336.log 2023-10-16T15:56:58.567 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.117620.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.105771.log 2023-10-16T15:56:58.568 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117620.log.gz 2023-10-16T15:56:58.568 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104336.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104336.log.gz 2023-10-16T15:56:58.569 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139619.log 2023-10-16T15:56:58.570 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107071.log 2023-10-16T15:56:58.570 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105771.log.gz 2023-10-16T15:56:58.571 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139619.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139619.log.gz 2023-10-16T15:56:58.572 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109729.log 2023-10-16T15:56:58.573 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.107071.log: -- /var/log/ceph/ceph-client.admin.118481.log 2023-10-16T15:56:58.573 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.107071.log.gz 2023-10-16T15:56:58.574 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109729.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109729.log.gz 2023-10-16T15:56:58.574 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143114.log 2023-10-16T15:56:58.575 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106809.log 2023-10-16T15:56:58.576 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118481.log.gz 2023-10-16T15:56:58.576 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143114.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.143114.log.gz 2023-10-16T15:56:58.576 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93564.log 2023-10-16T15:56:58.577 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109955.log 2023-10-16T15:56:58.578 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106809.log.gz 2023-10-16T15:56:58.579 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132606.log 2023-10-16T15:56:58.579 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.93564.log.gz 2023-10-16T15:56:58.580 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95299.log 2023-10-16T15:56:58.580 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109955.log.gz 2023-10-16T15:56:58.581 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132606.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.132606.log.gz 2023-10-16T15:56:58.582 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148880.log 2023-10-16T15:56:58.583 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143865.log 2023-10-16T15:56:58.584 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95299.log.gz 2023-10-16T15:56:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148880.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.148880.log.gz 2023-10-16T15:56:58.584 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156009.log 2023-10-16T15:56:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97854.log 2023-10-16T15:56:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143865.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143865.log.gz 2023-10-16T15:56:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156009.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.156009.log.gz 2023-10-16T15:56:58.587 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92998.log 2023-10-16T15:56:58.587 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108975.log 2023-10-16T15:56:58.588 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97854.log.gz 2023-10-16T15:56:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92998.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92998.log.gz 2023-10-16T15:56:58.589 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118565.log 2023-10-16T15:56:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.108975.log: /var/log/ceph/ceph-client.admin.147835.log 2023-10-16T15:56:58.591 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.108975.log.gz 2023-10-16T15:56:58.591 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118565.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118565.log.gz 2023-10-16T15:56:58.592 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114188.log 2023-10-16T15:56:58.592 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.147835.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.100457.log 2023-10-16T15:56:58.593 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147835.log.gz 2023-10-16T15:56:58.594 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114188.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114188.log.gz 2023-10-16T15:56:58.594 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143637.log 2023-10-16T15:56:58.595 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105729.log 2023-10-16T15:56:58.595 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100457.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.100457.log.gz 2023-10-16T15:56:58.596 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143637.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.143637.log.gz 2023-10-16T15:56:58.596 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134448.log 2023-10-16T15:56:58.597 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109374.log 2023-10-16T15:56:58.598 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105729.log.gz 2023-10-16T15:56:58.598 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134448.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134448.log.gz 2023-10-16T15:56:58.599 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112753.log 2023-10-16T15:56:58.600 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133718.log 2023-10-16T15:56:58.600 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109374.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109374.log.gz 2023-10-16T15:56:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112753.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112753.log.gz 2023-10-16T15:56:58.601 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96010.log 2023-10-16T15:56:58.602 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112879.log 2023-10-16T15:56:58.603 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.133718.log.gz 2023-10-16T15:56:58.603 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96010.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96010.log.gz 2023-10-16T15:56:58.604 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119652.log 2023-10-16T15:56:58.605 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.112879.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.113262.log 2023-10-16T15:56:58.605 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112879.log.gz 2023-10-16T15:56:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119652.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119652.log.gz 2023-10-16T15:56:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133216.log 2023-10-16T15:56:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152685.log 2023-10-16T15:56:58.608 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113262.log.gz 2023-10-16T15:56:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133216.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.133216.log.gz 2023-10-16T15:56:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114380.log 2023-10-16T15:56:58.610 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153718.log 2023-10-16T15:56:58.610 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.152685.log.gz 2023-10-16T15:56:58.611 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114380.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114380.log.gz 2023-10-16T15:56:58.611 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144332.log 2023-10-16T15:56:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135727.log 2023-10-16T15:56:58.613 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153718.log.gz 2023-10-16T15:56:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144332.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.144332.log.gz 2023-10-16T15:56:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125175.log 2023-10-16T15:56:58.614 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118523.log 2023-10-16T15:56:58.615 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135727.log.gz 2023-10-16T15:56:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125175.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125175.log.gz 2023-10-16T15:56:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118926.log 2023-10-16T15:56:58.617 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134959.log 2023-10-16T15:56:58.617 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118523.log.gz 2023-10-16T15:56:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118926.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118926.log.gz 2023-10-16T15:56:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152559.log 2023-10-16T15:56:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132352.log 2023-10-16T15:56:58.620 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134959.log.gz 2023-10-16T15:56:58.620 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152559.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152559.log.gz 2023-10-16T15:56:58.621 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106097.log 2023-10-16T15:56:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138000.log 2023-10-16T15:56:58.622 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.132352.log.gz 2023-10-16T15:56:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106097.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106097.log.gz 2023-10-16T15:56:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94053.log 2023-10-16T15:56:58.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104611.log 2023-10-16T15:56:58.625 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.138000.log.gz 2023-10-16T15:56:58.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.94053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94053.log.gz 2023-10-16T15:56:58.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135685.log 2023-10-16T15:56:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130681.log 2023-10-16T15:56:58.627 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104611.log.gz 2023-10-16T15:56:58.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135685.log.gz 2023-10-16T15:56:58.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113812.log 2023-10-16T15:56:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.130681.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.155907.log 2023-10-16T15:56:58.630 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130681.log.gz 2023-10-16T15:56:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113812.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113812.log.gz 2023-10-16T15:56:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113518.log 2023-10-16T15:56:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155907.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.155907.log.gz 2023-10-16T15:56:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119264.log 2023-10-16T15:56:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153040.log 2023-10-16T15:56:58.634 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113518.log.gz 2023-10-16T15:56:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119264.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119264.log.gz 2023-10-16T15:56:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137575.log 2023-10-16T15:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153040.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.153040.log.gz 2023-10-16T15:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112417.log 2023-10-16T15:56:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108578.log 2023-10-16T15:56:58.638 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137575.log.gz 2023-10-16T15:56:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112417.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112417.log.gz 2023-10-16T15:56:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149263.log 2023-10-16T15:56:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108578.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108578.log.gz 2023-10-16T15:56:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110596.log 2023-10-16T15:56:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149263.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.149263.log.gz 2023-10-16T15:56:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109248.log 2023-10-16T15:56:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110596.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110596.log.gz 2023-10-16T15:56:58.644 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156262.log 2023-10-16T15:56:58.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113728.log 2023-10-16T15:56:58.645 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109248.log.gz 2023-10-16T15:56:58.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156262.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156262.log.gz 2023-10-16T15:56:58.647 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157326.log 2023-10-16T15:56:58.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113728.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113728.log.gz 2023-10-16T15:56:58.648 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124795.log 2023-10-16T15:56:58.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106181.log 2023-10-16T15:56:58.649 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.157326.log.gz 2023-10-16T15:56:58.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124795.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124795.log.gz 2023-10-16T15:56:58.651 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157242.log 2023-10-16T15:56:58.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106181.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106181.log.gz 2023-10-16T15:56:58.652 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106037.log 2023-10-16T15:56:58.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109164.log 2023-10-16T15:56:58.654 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.157242.log.gz 2023-10-16T15:56:58.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106037.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106037.log.gz 2023-10-16T15:56:58.655 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155424.log 2023-10-16T15:56:58.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109164.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109164.log.gz 2023-10-16T15:56:58.656 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99742.log 2023-10-16T15:56:58.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106349.log 2023-10-16T15:56:58.658 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155424.log.gz 2023-10-16T15:56:58.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115136.log 2023-10-16T15:56:58.659 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99742.log.gz 2023-10-16T15:56:58.661 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113644.log 2023-10-16T15:56:58.661 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106349.log.gz 2023-10-16T15:56:58.661 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115136.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115136.log.gz 2023-10-16T15:56:58.662 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126659.log 2023-10-16T15:56:58.663 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132130.log 2023-10-16T15:56:58.663 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113644.log.gz 2023-10-16T15:56:58.665 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119050.log 2023-10-16T15:56:58.665 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.126659.log.gz 2023-10-16T15:56:58.666 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132130.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.132130.log.gz 2023-10-16T15:56:58.666 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116279.log 2023-10-16T15:56:58.667 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129466.log 2023-10-16T15:56:58.667 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119050.log.gz 2023-10-16T15:56:58.669 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116279.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116279.log.gz 2023-10-16T15:56:58.669 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131080.log 2023-10-16T15:56:58.670 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129466.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129466.log.gz 2023-10-16T15:56:58.670 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121985.log 2023-10-16T15:56:58.671 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147583.log 2023-10-16T15:56:58.672 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.131080.log.gz 2023-10-16T15:56:58.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121985.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121985.log.gz 2023-10-16T15:56:58.673 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105304.log 2023-10-16T15:56:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147583.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147583.log.gz 2023-10-16T15:56:58.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118229.log 2023-10-16T15:56:58.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105304.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105304.log.gz 2023-10-16T15:56:58.676 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110932.log 2023-10-16T15:56:58.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98304.log 2023-10-16T15:56:58.677 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118229.log.gz/var/log/ceph/ceph-client.admin.110932.log: 2023-10-16T15:56:58.678 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110932.log.gz 2023-10-16T15:56:58.679 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136022.log 2023-10-16T15:56:58.679 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98304.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98304.log.gz 2023-10-16T15:56:58.680 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-16T15:56:58.681 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.136022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100415.log 2023-10-16T15:56:58.681 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136022.log.gz 2023-10-16T15:56:58.683 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152955.log 2023-10-16T15:56:58.684 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119484.log 2023-10-16T15:56:58.684 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100415.log.gz 2023-10-16T15:56:58.686 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154236.log 2023-10-16T15:56:58.686 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.152955.log.gz 2023-10-16T15:56:58.687 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91780.log 2023-10-16T15:56:58.687 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119484.log.gz 2023-10-16T15:56:58.689 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96724.log 2023-10-16T15:56:58.689 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154236.log.gz 2023-10-16T15:56:58.690 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.140124.log 2023-10-16T15:56:58.690 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91780.log.gz 2023-10-16T15:56:58.692 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134261.log 2023-10-16T15:56:58.692 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96724.log.gz 2023-10-16T15:56:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.140124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.140124.log.gz 2023-10-16T15:56:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110890.log 2023-10-16T15:56:58.694 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109603.log 2023-10-16T15:56:58.694 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134261.log.gz 2023-10-16T15:56:58.696 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148754.log 2023-10-16T15:56:58.696 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110890.log.gz 2023-10-16T15:56:58.698 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117051.log 2023-10-16T15:56:58.698 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109603.log.gz 2023-10-16T15:56:58.699 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131866.log 2023-10-16T15:56:58.699 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148754.log.gz 2023-10-16T15:56:58.700 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146430.log 2023-10-16T15:56:58.700 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117051.log.gz 2023-10-16T15:56:58.702 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144724.log 2023-10-16T15:56:58.702 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.131866.log.gz 2023-10-16T15:56:58.703 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126219.log 2023-10-16T15:56:58.703 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146430.log.gz 2023-10-16T15:56:58.704 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144724.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.144724.log.gz 2023-10-16T15:56:58.705 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.154864.log 2023-10-16T15:56:58.706 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110554.log 2023-10-16T15:56:58.706 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126219.log.gz 2023-10-16T15:56:58.708 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110382.log 2023-10-16T15:56:58.708 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.154864.log.gz 2023-10-16T15:56:58.709 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145310.log 2023-10-16T15:56:58.709 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110554.log.gz 2023-10-16T15:56:58.710 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120776.log 2023-10-16T15:56:58.711 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110382.log.gz 2023-10-16T15:56:58.712 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125007.log 2023-10-16T15:56:58.712 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145310.log.gz 2023-10-16T15:56:58.713 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112795.log 2023-10-16T15:56:58.714 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120776.log.gz 2023-10-16T15:56:58.715 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120220.log 2023-10-16T15:56:58.716 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125007.log.gz 2023-10-16T15:56:58.716 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151934.log 2023-10-16T15:56:58.717 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112795.log.gz 2023-10-16T15:56:58.718 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134658.log 2023-10-16T15:56:58.718 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120220.log.gz 2023-10-16T15:56:58.719 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91355.log 2023-10-16T15:56:58.719 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151934.log.gz 2023-10-16T15:56:58.721 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147058.log 2023-10-16T15:56:58.722 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134658.log.gz 2023-10-16T15:56:58.722 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134219.log 2023-10-16T15:56:58.723 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91355.log.gz 2023-10-16T15:56:58.724 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131164.log 2023-10-16T15:56:58.724 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.147058.log.gz 2023-10-16T15:56:58.725 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113054.log 2023-10-16T15:56:58.726 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134219.log.gz 2023-10-16T15:56:58.727 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123406.log 2023-10-16T15:56:58.727 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.131164.log.gz 2023-10-16T15:56:58.728 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123862.log 2023-10-16T15:56:58.729 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113054.log.gz 2023-10-16T15:56:58.729 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86999.log 2023-10-16T15:56:58.730 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123406.log.gz 2023-10-16T15:56:58.732 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135601.log 2023-10-16T15:56:58.732 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123862.log.gz 2023-10-16T15:56:58.733 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.86999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109560.log 2023-10-16T15:56:58.733 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.86999.log.gz 2023-10-16T15:56:58.734 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135601.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95848.log 2023-10-16T15:56:58.734 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.135601.log.gz 2023-10-16T15:56:58.736 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156456.log 2023-10-16T15:56:58.736 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109560.log.gz 2023-10-16T15:56:58.737 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148208.log 2023-10-16T15:56:58.737 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95848.log.gz 2023-10-16T15:56:58.738 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.154615.log 2023-10-16T15:56:58.738 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.156456.log.gz 2023-10-16T15:56:58.740 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111968.log 2023-10-16T15:56:58.740 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.148208.log.gz 2023-10-16T15:56:58.741 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.154615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.154615.log.gz 2023-10-16T15:56:58.742 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150861.log 2023-10-16T15:56:58.743 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147226.log 2023-10-16T15:56:58.743 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111968.log.gz 2023-10-16T15:56:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139515.log 2023-10-16T15:56:58.744 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150861.log.gz 2023-10-16T15:56:58.746 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110722.log 2023-10-16T15:56:58.747 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147226.log.gz 2023-10-16T15:56:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138177.log 2023-10-16T15:56:58.748 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139515.log.gz 2023-10-16T15:56:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147415.log 2023-10-16T15:56:58.749 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110722.log.gz 2023-10-16T15:56:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153632.log 2023-10-16T15:56:58.750 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.138177.log.gz 2023-10-16T15:56:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115724.log 2023-10-16T15:56:58.752 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147415.log.gz 2023-10-16T15:56:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110094.log 2023-10-16T15:56:58.753 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.153632.log.gz 2023-10-16T15:56:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121504.log 2023-10-16T15:56:58.755 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115724.log.gz 2023-10-16T15:56:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149697.log 2023-10-16T15:56:58.756 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110094.log.gz 2023-10-16T15:56:58.757 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125912.log 2023-10-16T15:56:58.757 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121504.log.gz 2023-10-16T15:56:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147793.log 2023-10-16T15:56:58.759 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.149697.log.gz 2023-10-16T15:56:58.760 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130477.log 2023-10-16T15:56:58.761 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125912.log.gz 2023-10-16T15:56:58.762 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112123.log 2023-10-16T15:56:58.763 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147793.log.gz 2023-10-16T15:56:58.764 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144462.log 2023-10-16T15:56:58.764 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130477.log.gz 2023-10-16T15:56:58.765 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112123.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.112123.log.gz -5 2023-10-16T15:56:58.765 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.117159.log 2023-10-16T15:56:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116591.log 2023-10-16T15:56:58.766 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.144462.log.gz 2023-10-16T15:56:58.767 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96682.log 2023-10-16T15:56:58.768 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.117159.log.gz 2023-10-16T15:56:58.769 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152853.log 2023-10-16T15:56:58.769 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116591.log.gz 2023-10-16T15:56:58.771 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119092.log 2023-10-16T15:56:58.771 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96682.log.gz 2023-10-16T15:56:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94044.log 2023-10-16T15:56:58.772 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152853.log.gz 2023-10-16T15:56:58.774 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134532.log 2023-10-16T15:56:58.774 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119092.log.gz 2023-10-16T15:56:58.774 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.94044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94044.log.gz 2023-10-16T15:56:58.775 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93522.log 2023-10-16T15:56:58.776 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106976.log 2023-10-16T15:56:58.776 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.134532.log.gz 2023-10-16T15:56:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101223.log 2023-10-16T15:56:58.778 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93522.log.gz 2023-10-16T15:56:58.779 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137694.log 2023-10-16T15:56:58.780 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.106976.log.gz 2023-10-16T15:56:58.781 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109112.log 2023-10-16T15:56:58.781 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101223.log.gz 2023-10-16T15:56:58.782 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139661.log 2023-10-16T15:56:58.782 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137694.log.gz 2023-10-16T15:56:58.784 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109112.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.109112.log.gz -5 2023-10-16T15:56:58.784 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.118751.log 2023-10-16T15:56:58.785 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94909.log 2023-10-16T15:56:58.785 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139661.log.gz 2023-10-16T15:56:58.787 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112165.log 2023-10-16T15:56:58.787 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.118751.log.gz 2023-10-16T15:56:58.789 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146489.log 2023-10-16T15:56:58.789 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94909.log.gz 2023-10-16T15:56:58.789 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142883.log 2023-10-16T15:56:58.790 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112165.log.gz 2023-10-16T15:56:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126039.log 2023-10-16T15:56:58.792 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146489.log.gz 2023-10-16T15:56:58.793 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119356.log 2023-10-16T15:56:58.793 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.142883.log.gz 2023-10-16T15:56:58.794 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134879.log 2023-10-16T15:56:58.794 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126039.log.gz 2023-10-16T15:56:58.795 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99616.log 2023-10-16T15:56:58.796 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.119356.log.gz 2023-10-16T15:56:58.797 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114338.log 2023-10-16T15:56:58.797 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134879.log.gz 2023-10-16T15:56:58.798 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122988.log 2023-10-16T15:56:58.798 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99616.log.gz 2023-10-16T15:56:58.799 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92074.log 2023-10-16T15:56:58.800 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114338.log.gz 2023-10-16T15:56:58.802 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122556.log 2023-10-16T15:56:58.802 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122988.log.gz 2023-10-16T15:56:58.803 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120279.log 2023-10-16T15:56:58.803 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92074.log.gz 2023-10-16T15:56:58.805 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91822.log 2023-10-16T15:56:58.805 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122556.log.gz 2023-10-16T15:56:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143408.log 2023-10-16T15:56:58.806 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120279.log.gz 2023-10-16T15:56:58.807 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91822.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-16T15:56:58.807 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91822.log.gz 2023-10-16T15:56:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94182.log 2023-10-16T15:56:58.809 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143408.log.gz 2023-10-16T15:56:58.810 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125049.log 2023-10-16T15:56:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95722.log 2023-10-16T15:56:58.812 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94182.log.gz 2023-10-16T15:56:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125049.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147751.log 2023-10-16T15:56:58.813 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.125049.log.gz 2023-10-16T15:56:58.814 INFO:teuthology.orchestra.run.smithi029.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-16T15:56:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95722.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95722.log.gz 2023-10-16T15:56:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122650.log 2023-10-16T15:56:58.816 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144640.log 2023-10-16T15:56:58.816 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147751.log.gz 2023-10-16T15:56:58.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122650.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122650.log.gz 2023-10-16T15:56:58.817 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97228.log 2023-10-16T15:56:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144640.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144640.log.gz 2023-10-16T15:56:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155697.log 2023-10-16T15:56:58.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97228.log: gzip 88.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.97228.log.gz -- 2023-10-16T15:56:58.821 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.124923.log 2023-10-16T15:56:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155697.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155697.log.gz 2023-10-16T15:56:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117417.log 2023-10-16T15:56:58.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97726.log 2023-10-16T15:56:58.823 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124923.log.gz 2023-10-16T15:56:58.825 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96136.log 2023-10-16T15:56:58.826 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117417.log.gz 2023-10-16T15:56:58.827 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92634.log 2023-10-16T15:56:58.827 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97726.log.gz 2023-10-16T15:56:58.828 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96136.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96136.log.gz 2023-10-16T15:56:58.828 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91566.log 2023-10-16T15:56:58.829 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156094.log 2023-10-16T15:56:58.829 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92634.log.gz 2023-10-16T15:56:58.830 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91566.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91566.log.gz 2023-10-16T15:56:58.830 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156880.log 2023-10-16T15:56:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156094.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.156094.log.gz 2023-10-16T15:56:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151703.log 2023-10-16T15:56:58.833 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156880.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.156880.log.gz 2023-10-16T15:56:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98764.log 2023-10-16T15:56:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151703.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.151703.log.gz 2023-10-16T15:56:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119610.log 2023-10-16T15:56:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98764.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98764.log.gz 2023-10-16T15:56:58.837 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122946.log 2023-10-16T15:56:58.838 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119610.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119610.log.gz 2023-10-16T15:56:58.838 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155739.log 2023-10-16T15:56:58.839 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122946.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122946.log.gz 2023-10-16T15:56:58.840 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.94486.log 2023-10-16T15:56:58.841 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155739.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.155739.log.gz 2023-10-16T15:56:58.841 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153846.log 2023-10-16T15:56:58.843 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.2.94486.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.151976.log 2023-10-16T15:56:58.843 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.2.94486.log.gz 2023-10-16T15:56:58.844 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153846.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153846.log.gz 2023-10-16T15:56:58.844 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96310.log 2023-10-16T15:56:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151976.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151976.log.gz 2023-10-16T15:56:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103326.log 2023-10-16T15:56:58.847 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96310.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.99490.log 2023-10-16T15:56:58.847 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96310.log.gz 2023-10-16T15:56:58.849 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95172.log 2023-10-16T15:56:58.849 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103326.log.gz 2023-10-16T15:56:58.850 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154152.log 2023-10-16T15:56:58.850 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99490.log.gz 2023-10-16T15:56:58.852 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97898.log 2023-10-16T15:56:58.852 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95172.log.gz 2023-10-16T15:56:58.853 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133371.log 2023-10-16T15:56:58.854 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154152.log.gz 2023-10-16T15:56:58.855 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152312.log 2023-10-16T15:56:58.855 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97898.log.gz 2023-10-16T15:56:58.856 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151161.log 2023-10-16T15:56:58.857 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.133371.log.gz 2023-10-16T15:56:58.858 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105178.log 2023-10-16T15:56:58.858 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.152312.log.gz 2023-10-16T15:56:58.860 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118145.log 2023-10-16T15:56:58.860 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151161.log.gz 2023-10-16T15:56:58.861 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105178.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.155018.log 2023-10-16T15:56:58.861 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105178.log.gz 2023-10-16T15:56:58.862 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118145.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.118145.log.gz -5 2023-10-16T15:56:58.863 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.123615.log 2023-10-16T15:56:58.864 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155018.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155018.log.gz 2023-10-16T15:56:58.864 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153674.log 2023-10-16T15:56:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123615.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123615.log.gz 2023-10-16T15:56:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95383.log 2023-10-16T15:56:58.867 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153674.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153674.log.gz 2023-10-16T15:56:58.868 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96640.log 2023-10-16T15:56:58.869 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95383.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95383.log.gz 2023-10-16T15:56:58.869 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109458.log 2023-10-16T15:56:58.870 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96640.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96640.log.gz 2023-10-16T15:56:58.871 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109813.log 2023-10-16T15:56:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109458.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109458.log.gz 2023-10-16T15:56:58.872 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137227.log 2023-10-16T15:56:58.874 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109813.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109813.log.gz 2023-10-16T15:56:58.874 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94699.log 2023-10-16T15:56:58.875 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137227.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137227.log.gz 2023-10-16T15:56:58.875 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110340.log 2023-10-16T15:56:58.877 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94699.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94699.log.gz 2023-10-16T15:56:58.877 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93084.log 2023-10-16T15:56:58.878 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110340.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110340.log.gz 2023-10-16T15:56:58.878 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147100.log 2023-10-16T15:56:58.880 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93084.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93084.log.gz 2023-10-16T15:56:58.880 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97389.log 2023-10-16T15:56:58.881 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147100.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147100.log.gz 2023-10-16T15:56:58.882 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144598.log 2023-10-16T15:56:58.883 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97389.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97389.log.gz 2023-10-16T15:56:58.883 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125656.log 2023-10-16T15:56:58.884 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144598.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.144598.log.gz 2023-10-16T15:56:58.885 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145478.log 2023-10-16T15:56:58.886 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125656.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125656.log.gz 2023-10-16T15:56:58.886 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T15:56:58.888 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145478.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145478.log.gz 2023-10-16T15:56:58.888 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122820.log 2023-10-16T15:56:58.889 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.119993.log 2023-10-16T15:56:58.891 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96394.log 2023-10-16T15:56:58.891 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122820.log.gz 2023-10-16T15:56:58.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.119993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.119993.log.gz 2023-10-16T15:56:58.893 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143553.log 2023-10-16T15:56:58.894 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94951.log 2023-10-16T15:56:58.894 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96394.log.gz 2023-10-16T15:56:58.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108933.log 2023-10-16T15:56:58.896 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143553.log.gz 2023-10-16T15:56:58.897 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122692.log 2023-10-16T15:56:58.897 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94951.log.gz 2023-10-16T15:56:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108933.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108933.log.gz 2023-10-16T15:56:58.899 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149865.log 2023-10-16T15:56:58.899 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122692.log: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T15:56:58.900 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.122692.log.gz 2023-10-16T15:56:58.900 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122904.log 2023-10-16T15:56:58.901 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122600.log 2023-10-16T15:56:58.902 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122904.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.149865.log.gz 2023-10-16T15:56:58.902 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122904.log.gz 2023-10-16T15:56:58.903 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91610.log 2023-10-16T15:56:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122600.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.122600.log.gz 2023-10-16T15:56:58.904 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124062.log 2023-10-16T15:56:58.906 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91610.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.91610.log.gz /var/log/ceph/ceph-client.admin.121733.log 2023-10-16T15:56:58.906 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100810.log 2023-10-16T15:56:58.907 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.124062.log.gz 2023-10-16T15:56:58.909 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119220.log 2023-10-16T15:56:58.909 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121733.log.gz 2023-10-16T15:56:58.911 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127143.log 2023-10-16T15:56:58.911 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100810.log.gz 2023-10-16T15:56:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119220.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-16T15:56:58.914 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.119220.log.gz 2023-10-16T15:56:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131435.log 2023-10-16T15:56:58.914 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127143.log.gz 2023-10-16T15:56:58.915 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155613.log 2023-10-16T15:56:58.918 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.131435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95890.log 2023-10-16T15:56:58.918 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.131435.log.gz 2023-10-16T15:56:58.919 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155613.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.155613.log.gz 2023-10-16T15:56:58.919 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142841.log 2023-10-16T15:56:58.920 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113686.log 2023-10-16T15:56:58.920 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95890.log.gz 2023-10-16T15:56:58.921 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114653.log 2023-10-16T15:56:58.921 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142841.log.gz 2023-10-16T15:56:58.923 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103607.log 2023-10-16T15:56:58.923 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113686.log.gz 2023-10-16T15:56:58.924 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111350.log 2023-10-16T15:56:58.925 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114653.log.gz 2023-10-16T15:56:58.925 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123803.log 2023-10-16T15:56:58.926 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103607.log.gz 2023-10-16T15:56:58.927 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135517.log 2023-10-16T15:56:58.928 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111350.log.gz 2023-10-16T15:56:58.929 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139473.log 2023-10-16T15:56:58.929 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123803.log.gz 2023-10-16T15:56:58.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116507.log 2023-10-16T15:56:58.930 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135517.log.gz 2023-10-16T15:56:58.931 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139473.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.139473.log.gz /var/log/ceph/ceph-client.admin.114695.log 2023-10-16T15:56:58.932 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:56:58.933 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116507.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.109069.log 2023-10-16T15:56:58.933 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116507.log.gz 2023-10-16T15:56:58.934 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114695.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-16T15:56:58.935 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114695.log.gz 2023-10-16T15:56:58.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125488.log 2023-10-16T15:56:58.936 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109069.log.gz 2023-10-16T15:56:58.938 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123030.log 2023-10-16T15:56:58.939 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125488.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.125488.log.gz 2023-10-16T15:56:58.939 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116675.log 2023-10-16T15:56:58.940 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123030.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123030.log.gz 2023-10-16T15:56:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144205.log 2023-10-16T15:56:58.942 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116675.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116675.log.gz 2023-10-16T15:56:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146796.log 2023-10-16T15:56:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144205.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.144205.log.gz 2023-10-16T15:56:58.945 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106307.log 2023-10-16T15:56:58.946 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146796.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.146796.log.gz 2023-10-16T15:56:58.947 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91990.log 2023-10-16T15:56:58.948 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106307.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106307.log.gz 2023-10-16T15:56:58.949 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118846.log 2023-10-16T15:56:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91990.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91990.log.gz 2023-10-16T15:56:58.951 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150427.log 2023-10-16T15:56:58.952 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118846.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118846.log.gz 2023-10-16T15:56:58.953 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104821.log 2023-10-16T15:56:58.955 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150427.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.150427.log.gz 2023-10-16T15:56:58.955 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118019.log 2023-10-16T15:56:58.956 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104821.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104821.log.gz 2023-10-16T15:56:58.957 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112459.log 2023-10-16T15:56:58.959 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118019.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118019.log.gz 2023-10-16T15:56:58.960 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101943.log 2023-10-16T15:56:58.960 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112459.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112459.log.gz 2023-10-16T15:56:58.961 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138574.log 2023-10-16T15:56:58.963 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101943.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101943.log.gz 2023-10-16T15:56:58.964 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113304.log 2023-10-16T15:56:58.964 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138574.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138574.log.gz 2023-10-16T15:56:58.965 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106391.log 2023-10-16T15:56:58.967 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113304.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113304.log.gz 2023-10-16T15:56:58.967 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104695.log 2023-10-16T15:56:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106391.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106391.log.gz 2023-10-16T15:56:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119400.log 2023-10-16T15:56:58.971 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104695.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104695.log.gz 2023-10-16T15:56:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125954.log 2023-10-16T15:56:58.973 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119400.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119400.log.gz 2023-10-16T15:56:58.974 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138532.log 2023-10-16T15:56:58.974 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125954.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125954.log.gz 2023-10-16T15:56:58.975 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105220.log 2023-10-16T15:56:58.977 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138532.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.138532.log.gz 2023-10-16T15:56:58.978 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143240.log 2023-10-16T15:56:58.979 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105220.log.gz 2023-10-16T15:56:58.980 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148376.log 2023-10-16T15:56:58.981 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143240.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143240.log.gz 2023-10-16T15:56:58.982 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92633.log 2023-10-16T15:56:58.983 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148376.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148376.log.gz 2023-10-16T15:56:58.984 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139937.log 2023-10-16T15:56:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92633.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92633.log.gz 2023-10-16T15:56:58.986 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111458.log 2023-10-16T15:56:58.987 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139937.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.139937.log.gz 2023-10-16T15:56:58.988 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91482.log 2023-10-16T15:56:58.990 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111458.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111458.log.gz 2023-10-16T15:56:58.990 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.91482.log: /var/log/ceph/ceph-client.admin.93438.log 2023-10-16T15:56:58.991 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91482.log.gz 2023-10-16T15:56:58.992 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144044.log 2023-10-16T15:56:58.993 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93438.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93438.log.gz 2023-10-16T15:56:58.994 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106433.log 2023-10-16T15:56:58.995 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144044.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144044.log.gz 2023-10-16T15:56:58.996 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152060.log 2023-10-16T15:56:58.998 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106433.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.106433.log.gz 2023-10-16T15:56:58.999 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119134.log 2023-10-16T15:56:58.999 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152060.log.gz 2023-10-16T15:56:59.000 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152270.log 2023-10-16T15:56:59.002 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119134.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119134.log.gz 2023-10-16T15:56:59.003 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124881.log 2023-10-16T15:56:59.003 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152270.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.152270.log.gz 2023-10-16T15:56:59.004 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130954.log 2023-10-16T15:56:59.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124881.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124881.log.gz 2023-10-16T15:56:59.007 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93168.log 2023-10-16T15:56:59.007 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.130954.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130954.log.gz 2023-10-16T15:56:59.009 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115556.log 2023-10-16T15:56:59.010 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93168.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.93168.log.gz 2023-10-16T15:56:59.010 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.119989.log 2023-10-16T15:56:59.011 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115556.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115556.log.gz 2023-10-16T15:56:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118607.log 2023-10-16T15:56:59.013 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.119989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.119989.log.gz 2023-10-16T15:56:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100373.log 2023-10-16T15:56:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118607.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118607.log.gz 2023-10-16T15:56:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92914.log 2023-10-16T15:56:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100373.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100373.log.gz 2023-10-16T15:56:59.018 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145692.log 2023-10-16T15:56:59.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92914.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.92914.log.gz 2023-10-16T15:56:59.020 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147142.log 2023-10-16T15:56:59.021 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145692.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145692.log.gz 2023-10-16T15:56:59.022 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95425.log 2023-10-16T15:56:59.023 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147142.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147142.log.gz 2023-10-16T15:56:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153760.log 2023-10-16T15:56:59.025 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95425.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95425.log.gz 2023-10-16T15:56:59.026 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115388.log 2023-10-16T15:56:59.027 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153760.log.gz 2023-10-16T15:56:59.028 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140289.log 2023-10-16T15:56:59.029 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115388.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115388.log.gz 2023-10-16T15:56:59.030 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119442.log 2023-10-16T15:56:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140289.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.140289.log.gz 2023-10-16T15:56:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125133.log 2023-10-16T15:56:59.033 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119442.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119442.log.gz 2023-10-16T15:56:59.034 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99372.log 2023-10-16T15:56:59.035 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125133.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125133.log.gz 2023-10-16T15:56:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129424.log 2023-10-16T15:56:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99372.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99372.log.gz 2023-10-16T15:56:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123514.log 2023-10-16T15:56:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129424.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129424.log.gz 2023-10-16T15:56:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94741.log 2023-10-16T15:56:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123514.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123514.log.gz 2023-10-16T15:56:59.042 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96766.log 2023-10-16T15:56:59.043 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94741.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94741.log.gz 2023-10-16T15:56:59.044 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.106690.log 2023-10-16T15:56:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96766.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96766.log.gz 2023-10-16T15:56:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112333.log 2023-10-16T15:56:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.106690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.106690.log.gz 2023-10-16T15:56:59.048 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144124.log 2023-10-16T15:56:59.050 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112333.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112333.log.gz 2023-10-16T15:56:59.051 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.144124.log: -- /var/log/ceph/ceph-client.admin.121817.log 2023-10-16T15:56:59.051 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144124.log.gz 2023-10-16T15:56:59.052 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109206.log 2023-10-16T15:56:59.053 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121817.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121817.log.gz 2023-10-16T15:56:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96892.log 2023-10-16T15:56:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109206.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109206.log.gz 2023-10-16T15:56:59.057 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.96892.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.146584.log 2023-10-16T15:56:59.057 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96892.log.gz 2023-10-16T15:56:59.058 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122862.log 2023-10-16T15:56:59.060 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146584.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146584.log.gz 2023-10-16T15:56:59.060 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148975.log 2023-10-16T15:56:59.062 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122862.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.122862.log.gz 2023-10-16T15:56:59.063 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95509.log 2023-10-16T15:56:59.063 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148975.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148975.log.gz 2023-10-16T15:56:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124363.log 2023-10-16T15:56:59.066 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95509.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95509.log.gz 2023-10-16T15:56:59.066 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99868.log 2023-10-16T15:56:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124363.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124363.log.gz 2023-10-16T15:56:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126993.log 2023-10-16T15:56:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99868.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99868.log.gz 2023-10-16T15:56:59.071 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134574.log 2023-10-16T15:56:59.071 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126993.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126993.log.gz 2023-10-16T15:56:59.072 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154068.log 2023-10-16T15:56:59.074 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134574.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134574.log.gz 2023-10-16T15:56:59.075 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126081.log 2023-10-16T15:56:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154068.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154068.log.gz 2023-10-16T15:56:59.077 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113854.log 2023-10-16T15:56:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126081.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126081.log.gz 2023-10-16T15:56:59.079 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121901.log 2023-10-16T15:56:59.080 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113854.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.113854.log.gz 2023-10-16T15:56:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.121901.log: /var/log/ceph/ceph-client.admin.105899.log 2023-10-16T15:56:59.082 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121901.log.gz 2023-10-16T15:56:59.083 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126617.log 2023-10-16T15:56:59.084 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105899.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105899.log.gz 2023-10-16T15:56:59.085 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127202.log 2023-10-16T15:56:59.086 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126617.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126617.log.gz 2023-10-16T15:56:59.087 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132172.log 2023-10-16T15:56:59.088 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127202.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127202.log.gz 2023-10-16T15:56:59.088 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117704.log 2023-10-16T15:56:59.090 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132172.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.132172.log.gz 2023-10-16T15:56:59.091 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117243.log 2023-10-16T15:56:59.092 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117704.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117704.log.gz 2023-10-16T15:56:59.092 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146754.log 2023-10-16T15:56:59.094 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117243.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.117243.log.gz 2023-10-16T15:56:59.095 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113434.log 2023-10-16T15:56:59.096 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146754.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146754.log.gz 2023-10-16T15:56:59.097 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143366.log 2023-10-16T15:56:59.098 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113434.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113434.log.gz 2023-10-16T15:56:59.099 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99431.log 2023-10-16T15:56:59.100 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143366.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143366.log.gz 2023-10-16T15:56:59.101 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124566.log 2023-10-16T15:56:59.102 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99431.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99431.log.gz 2023-10-16T15:56:59.103 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133473.log 2023-10-16T15:56:59.104 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124566.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124566.log.gz 2023-10-16T15:56:59.105 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117578.log 2023-10-16T15:56:59.106 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133473.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133473.log.gz 2023-10-16T15:56:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151492.log 2023-10-16T15:56:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117578.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117578.log.gz 2023-10-16T15:56:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92284.log 2023-10-16T15:56:59.110 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151492.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.151492.log.gz 2023-10-16T15:56:59.111 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154816.log 2023-10-16T15:56:59.112 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92284.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92284.log.gz 2023-10-16T15:56:59.113 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122428.log 2023-10-16T15:56:59.114 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154816.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154816.log.gz 2023-10-16T15:56:59.115 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113134.log 2023-10-16T15:56:59.116 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122428.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122428.log.gz 2023-10-16T15:56:59.117 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98388.log 2023-10-16T15:56:59.118 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113134.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.113134.log.gz 2023-10-16T15:56:59.119 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138784.log/var/log/ceph/ceph-client.admin.98388.log: 2023-10-16T15:56:59.120 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.98388.log.gz 2023-10-16T15:56:59.121 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112711.log 2023-10-16T15:56:59.122 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138784.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138784.log.gz 2023-10-16T15:56:59.123 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99700.log 2023-10-16T15:56:59.124 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112711.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112711.log.gz 2023-10-16T15:56:59.125 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146712.log 2023-10-16T15:56:59.126 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99700.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99700.log.gz 2023-10-16T15:56:59.126 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156989.log 2023-10-16T15:56:59.127 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-16T15:56:59.128 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146712.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.146712.log.gz 2023-10-16T15:56:59.129 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92280.log 2023-10-16T15:56:59.130 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156989.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156989.log.gz 2023-10-16T15:56:59.131 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124116.log 2023-10-16T15:56:59.132 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92280.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92280.log.gz 2023-10-16T15:56:59.133 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104948.log 2023-10-16T15:56:59.134 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124116.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124116.log.gz 2023-10-16T15:56:59.135 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110806.log 2023-10-16T15:56:59.136 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104948.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104948.log.gz 2023-10-16T15:56:59.136 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149305.log 2023-10-16T15:56:59.138 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110806.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110806.log.gz 2023-10-16T15:56:59.138 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153166.log 2023-10-16T15:56:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149305.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149305.log.gz 2023-10-16T15:56:59.140 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91440.log 2023-10-16T15:56:59.141 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153166.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.153166.log.gz 2023-10-16T15:56:59.142 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.116974.log 2023-10-16T15:56:59.144 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91440.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91440.log.gz 2023-10-16T15:56:59.144 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118271.log 2023-10-16T15:56:59.145 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.116974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.116974.log.gz 2023-10-16T15:56:59.146 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151331.log 2023-10-16T15:56:59.147 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118271.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118271.log.gz 2023-10-16T15:56:59.148 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125614.log 2023-10-16T15:56:59.149 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151331.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.151331.log.gz 2023-10-16T15:56:59.150 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115052.log 2023-10-16T15:56:59.152 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125614.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.125614.log.gz 2023-10-16T15:56:59.152 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113920.log 2023-10-16T15:56:59.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115052.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.115052.log.gz 2023-10-16T15:56:59.154 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149137.log 2023-10-16T15:56:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113920.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113920.log.gz 2023-10-16T15:56:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145860.log 2023-10-16T15:56:59.157 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149137.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149137.log.gz 2023-10-16T15:56:59.158 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126431.log 2023-10-16T15:56:59.159 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.145860.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145860.log.gz 2023-10-16T15:56:59.160 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136253.log 2023-10-16T15:56:59.161 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126431.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126431.log.gz 2023-10-16T15:56:59.162 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92616.log 2023-10-16T15:56:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136253.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.136253.log.gz 2023-10-16T15:56:59.164 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99254.log 2023-10-16T15:56:59.165 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92616.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92616.log.gz 2023-10-16T15:56:59.166 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136145.log 2023-10-16T15:56:59.167 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99254.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99254.log.gz 2023-10-16T15:56:59.168 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156624.log 2023-10-16T15:56:59.169 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136145.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.136145.log.gz 2023-10-16T15:56:59.169 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154320.log 2023-10-16T15:56:59.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156624.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.156624.log.gz 2023-10-16T15:56:59.172 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154756.log 2023-10-16T15:56:59.173 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154320.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.154320.log.gz 2023-10-16T15:56:59.174 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98645.log 2023-10-16T15:56:59.174 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154756.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154756.log.gz 2023-10-16T15:56:59.175 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93126.log 2023-10-16T15:56:59.176 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.98645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.98645.log.gz 2023-10-16T15:56:59.177 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152517.log 2023-10-16T15:56:59.178 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93126.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93126.log.gz 2023-10-16T15:56:59.179 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114230.log 2023-10-16T15:56:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152517.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.152517.log.gz 2023-10-16T15:56:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100583.log 2023-10-16T15:56:59.182 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114230.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114230.log.gz 2023-10-16T15:56:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117840.log 2023-10-16T15:56:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100583.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100583.log.gz 2023-10-16T15:56:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95257.log 2023-10-16T15:56:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117840.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117840.log.gz 2023-10-16T15:56:59.187 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136400.log 2023-10-16T15:56:59.188 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95257.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95257.log.gz 2023-10-16T15:56:59.189 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125997.log 2023-10-16T15:56:59.191 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136400.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136400.log.gz 2023-10-16T15:56:59.191 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150945.log 2023-10-16T15:56:59.192 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125997.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125997.log.gz 2023-10-16T15:56:59.193 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148292.log 2023-10-16T15:56:59.194 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150945.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150945.log.gz 2023-10-16T15:56:59.195 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95593.log 2023-10-16T15:56:59.195 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148292.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.148292.log.gz 2023-10-16T15:56:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148838.log 2023-10-16T15:56:59.198 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95593.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95593.log.gz 2023-10-16T15:56:59.198 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115819.log 2023-10-16T15:56:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148838.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.148838.log.gz 2023-10-16T15:56:59.200 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97600.log 2023-10-16T15:56:59.201 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115819.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115819.log.gz 2023-10-16T15:56:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152475.log 2023-10-16T15:56:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97600.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97600.log.gz 2023-10-16T15:56:59.204 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-16T15:56:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152475.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152475.log.gz 2023-10-16T15:56:59.206 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144512.log 2023-10-16T15:56:59.209 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.144512.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144512.log.gz 2023-10-16T15:56:59.209 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112543.log 2023-10-16T15:56:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98136.log 2023-10-16T15:56:59.213 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112543.log.gz 2023-10-16T15:56:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149349.log 2023-10-16T15:56:59.217 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98136.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98136.log.gz 2023-10-16T15:56:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121210.log 2023-10-16T15:56:59.221 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149349.log.gz 2023-10-16T15:56:59.224 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.154619.log/var/log/ceph/ceph-client.admin.121210.log: 2023-10-16T15:56:59.225 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121210.log.gz 2023-10-16T15:56:59.227 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97642.log 2023-10-16T15:56:59.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.154619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.154619.log.gz 2023-10-16T15:56:59.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121775.log 2023-10-16T15:56:59.233 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.97642.log.gz -5 2023-10-16T15:56:59.234 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.92619.log 2023-10-16T15:56:59.236 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121775.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121775.log.gz 2023-10-16T15:56:59.237 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112063.log 2023-10-16T15:56:59.241 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112063.log: /var/log/ceph/ceph-client.admin.92619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152228.log 2023-10-16T15:56:59.242 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92619.log.gz 2023-10-16T15:56:59.244 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105943.log 2023-10-16T15:56:59.244 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.112063.log.gz 2023-10-16T15:56:59.247 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152228.log: 87.8%/var/log/ceph/ceph-client.admin.105943.log: -- replaced with /var/log/ceph/ceph-client.admin.152228.log.gz 2023-10-16T15:56:59.248 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105943.log.gz 2023-10-16T15:56:59.249 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151576.log 2023-10-16T15:56:59.252 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151576.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.151576.log.gz 2023-10-16T15:56:59.252 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132394.log 2023-10-16T15:56:59.255 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152102.log 2023-10-16T15:56:59.256 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.132394.log.gz 2023-10-16T15:56:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135559.log 2023-10-16T15:56:59.263 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152102.log: 89.2%/var/log/ceph/ceph-client.admin.135559.log: -- replaced with /var/log/ceph/ceph-client.admin.152102.log.gzgzip 2023-10-16T15:56:59.263 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.115981.log 2023-10-16T15:56:59.264 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135559.log.gz 2023-10-16T15:56:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138303.log 2023-10-16T15:56:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115981.log: /var/log/ceph/ceph-client.admin.138303.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138826.log 2023-10-16T15:56:59.270 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.115981.log.gz 2023-10-16T15:56:59.271 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.138303.log.gz 2023-10-16T15:56:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152769.log 2023-10-16T15:56:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152769.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.138826.log.gz 2023-10-16T15:56:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98262.log 2023-10-16T15:56:59.279 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.152769.log.gz 2023-10-16T15:56:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113220.log 2023-10-16T15:56:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113220.log: /var/log/ceph/ceph-client.admin.98262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97186.log 2023-10-16T15:56:59.286 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.98262.log.gz 2023-10-16T15:56:59.287 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.113220.log.gz 2023-10-16T15:56:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97186.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.97186.log.gz 2023-10-16T15:56:59.383 INFO:teuthology.orchestra.run.smithi029.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-16T15:57:00.116 INFO:teuthology.orchestra.run.smithi170.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-16T15:57:00.216 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-16T15:57:00.907 INFO:teuthology.orchestra.run.smithi170.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-16T15:57:01.240 INFO:teuthology.orchestra.run.smithi029.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-16T15:57:01.270 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-16T15:57:01.521 INFO:teuthology.orchestra.run.smithi029.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-16T15:57:01.632 INFO:teuthology.orchestra.run.smithi029.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-16T15:57:01.858 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-16T15:57:01.923 INFO:teuthology.orchestra.run.smithi170.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-16T15:57:01.926 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-16T15:57:01.926 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m4.272s 2023-10-16T15:57:01.926 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m19.963s 2023-10-16T15:57:01.926 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m1.777s 2023-10-16T15:57:02.418 INFO:teuthology.orchestra.run.smithi029.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-16T15:57:02.477 INFO:teuthology.orchestra.run.smithi029.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-16T15:57:03.556 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-16T15:57:03.562 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-16T15:57:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m5.910s 2023-10-16T15:57:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m20.742s 2023-10-16T15:57:03.563 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m4.572s 2023-10-16T15:57:03.563 INFO:tasks.ceph:Archiving logs... 2023-10-16T15:57:03.563 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/remote/smithi029/log 2023-10-16T15:57:03.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T15:57:06.791 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/remote/smithi170/log 2023-10-16T15:57:06.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T15:57:09.026 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-16T15:57:09.035 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-16T15:57:09.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T15:57:09.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T15:57:09.202 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-16T15:57:09.211 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-16T15:57:09.211 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T15:57:09.218 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T15:57:09.270 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-10-16T15:57:09.276 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2023-10-16T15:57:09.282 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T15:57:09.282 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-10-16T15:57:09.282 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 +1549us[+1549us] +/- 81ms 2023-10-16T15:57:09.282 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -732us[ -752us] +/- 51ms 2023-10-16T15:57:09.283 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -548us[ -548us] +/- 90ms 2023-10-16T15:57:09.283 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T15:57:09.286 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T15:57:09.286 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-10-16T15:57:09.286 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 169 +1536us[+1517us] +/- 79ms 2023-10-16T15:57:09.286 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 7 377 101 -711us[ -731us] +/- 50ms 2023-10-16T15:57:09.287 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -627us[ -627us] +/- 90ms 2023-10-16T15:57:09.287 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T15:57:09.288 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-16T15:57:09.296 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-16T15:57:09.297 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-16T15:57:09.305 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T15:57:09.503 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T15:57:09.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T15:57:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T15:57:09.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T15:57:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T15:57:09.971 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-16T15:57:09.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-16T15:57:09.990 INFO:teuthology.task.internal:Duration was 6872.977388 seconds 2023-10-16T15:57:09.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-16T15:57:10.000 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-16T15:57:10.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T15:57:10.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T15:57:10.180 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T15:57:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T15:57:10.502 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-16T15:57:10.502 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-10-16T15:57:10.503 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T15:57:10.573 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-10-16T15:57:10.573 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T15:57:10.652 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-16T15:57:10.653 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T15:57:10.662 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T15:57:10.761 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-16T15:57:10.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T15:57:10.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T15:57:11.042 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-16T15:57:11.052 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-16T15:57:11.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T15:57:11.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T15:57:11.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-16T15:57:11.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T15:57:11.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T15:57:11.393 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-10-16T15:57:11.396 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-10-16T15:57:11.450 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T15:57:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T15:57:11.501 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T15:57:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T15:57:11.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-16T15:57:11.567 INFO:teuthology.task.internal:Transferring archived files... 2023-10-16T15:57:11.569 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/remote/smithi029 2023-10-16T15:57:11.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T15:57:11.687 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037/remote/smithi170 2023-10-16T15:57:11.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T15:57:11.798 INFO:teuthology.task.internal:Removing archive directory... 2023-10-16T15:57:11.798 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T15:57:11.807 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T15:57:11.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-16T15:57:11.876 INFO:teuthology.task.internal:Not uploading archives. 2023-10-16T15:57:11.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-16T15:57:11.885 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-16T15:57:11.886 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T15:57:11.895 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T15:57:11.951 INFO:teuthology.orchestra.run.smithi029.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 15:57 /home/ubuntu/cephtest 2023-10-16T15:57:11.957 INFO:teuthology.orchestra.run.smithi170.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 15:57 /home/ubuntu/cephtest 2023-10-16T15:57:11.960 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-16T15:57:11.970 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-16T15:57:11.996 INFO:teuthology.nuke:Checking targets against current locks 2023-10-16T15:57:12.024 DEBUG:teuthology.nuke:shortname: smithi029 2023-10-16T15:57:12.025 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T15:57:12.048 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 13:53:02.388477', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLovRJaWv6bFJ6tOwzKzcWOli1s8H6A6heFn15YRWNmDHD1HBnGtgWuQuDz9pU15qe12OyQtW/A9zhZwnw3C9bg='} 2023-10-16T15:57:12.051 DEBUG:teuthology.nuke:shortname: smithi170 2023-10-16T15:57:12.051 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T15:57:12.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_12:17:50-fs-rishabh-2023oct13-testing-default-smithi/7429037', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 13:53:02.386821', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOBtmsYnh/Ohhod1nwajWDpTORU7ptsmilduWzGZaP1RVkfWpkWMiYu1Q5wefsop6hhXLZARMmrAzXyTGKqqYw='} 2023-10-16T15:57:12.100 INFO:teuthology.orchestra.console.smithi029:Power off 2023-10-16T15:57:12.101 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T15:57:12.123 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-10-16T15:57:12.123 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:12.138 INFO:teuthology.orchestra.console.smithi170:Power off 2023-10-16T15:57:12.138 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T15:57:12.147 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-16T15:57:12.159 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-10-16T15:57:12.159 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:12.180 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-16T15:57:16.148 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:16.182 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:16.258 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-16T15:57:16.291 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-16T15:57:20.260 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:20.293 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:20.373 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-16T15:57:20.403 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-10-16T15:57:20.503 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-10-16T15:57:24.377 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T15:57:24.487 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-10-16T15:57:24.588 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-10-16T15:57:24.693 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 client.1-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6872.977388143539 failure_reason: '"2023-10-16T14:31:33.181255+0000 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_rishabh@teuthology success: false 2023-10-16T15:57:24.693 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T15:57:24.769 INFO:teuthology.run:FAIL